2024-01-26T23:15:57.623 INFO:root:teuthology version: 0.0.1.dev251+gc856e0f 2024-01-26T23:15:57.623 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} --owner scheduled_yuriw@teuthology --verbose --name yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi/7533360 -- /home/teuthworker/archive/yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi/7533360/orig.config.yaml 2024-01-26T23:15:57.667 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-26T23:15:57.746 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi/7533360 branch: wip-yuri4-testing-2024-01-25-1331-quincy description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} email: yweinste@redhat.com first_in_suite: false job_id: '7533360' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-yuri4-testing-2024-01-25-1331-quincy ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 7495007658df82356749387b98425d4a7fcef9b6 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 7495007658df82356749387b98425d4a7fcef9b6 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2024-01-25-1331-quincy sha1: 7495007658df82356749387b98425d4a7fcef9b6 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 1285 sha1: 7495007658df82356749387b98425d4a7fcef9b6 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri4-testing-2024-01-25-1331-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7495007658df82356749387b98425d4a7fcef9b6/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7495007658df82356749387b98425d4a7fcef9b6 targets: smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH0jCpB2oSBYuBJFz8r7h5gvoxbVdD5G78u+0MHqsdwmoDh9tqhFT/LoUCId1h189cSnuoFeCa4UY2kBN3lkbp0= smithi106.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGNrzIpyQQzXce3S622+VCGLhT0dDQ6JzjjnSWBTu1pGvGlS9KyptTAX9xnumt/OLUGHnhziceBhmNU9cWNCBCE= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: c856e0f9244dd29a50697123b480d63f16a4ed81 timestamp: 2024-01-26_01:07:29 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-26T23:15:57.746 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7495007658df82356749387b98425d4a7fcef9b6/qa; will attempt to use it 2024-01-26T23:15:57.747 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7495007658df82356749387b98425d4a7fcef9b6/qa/tasks 2024-01-26T23:15:57.747 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-26T23:15:57.748 INFO:teuthology.task.internal:Checking packages... 2024-01-26T23:15:57.772 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '7495007658df82356749387b98425d4a7fcef9b6' 2024-01-26T23:15:57.772 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-26T23:15:57.772 INFO:teuthology.packaging:ref: None 2024-01-26T23:15:57.772 INFO:teuthology.packaging:tag: None 2024-01-26T23:15:57.772 INFO:teuthology.packaging:branch: wip-yuri4-testing-2024-01-25-1331-quincy 2024-01-26T23:15:57.773 INFO:teuthology.packaging:sha1: 7495007658df82356749387b98425d4a7fcef9b6 2024-01-26T23:15:57.773 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2024-01-25-1331-quincy 2024-01-26T23:15:58.145 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-472.g74950076 2024-01-26T23:15:58.147 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-26T23:15:58.154 INFO:teuthology.task.internal:no buildpackages task found 2024-01-26T23:15:58.154 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-26T23:15:58.171 INFO:teuthology.task.internal:Saving configuration 2024-01-26T23:15:58.186 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-26T23:15:58.194 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-26T23:15:58.219 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi/7533360', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-26 23:10:49.696299', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH0jCpB2oSBYuBJFz8r7h5gvoxbVdD5G78u+0MHqsdwmoDh9tqhFT/LoUCId1h189cSnuoFeCa4UY2kBN3lkbp0='} 2024-01-26T23:15:58.244 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi/7533360', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-26 23:10:49.697695', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGNrzIpyQQzXce3S622+VCGLhT0dDQ6JzjjnSWBTu1pGvGlS9KyptTAX9xnumt/OLUGHnhziceBhmNU9cWNCBCE='} 2024-01-26T23:15:58.244 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-26T23:15:58.251 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-01-26T23:15:58.251 INFO:teuthology.task.internal:roles: ubuntu@smithi106.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-01-26T23:15:58.251 INFO:teuthology.run_tasks:Running task console_log... 2024-01-26T23:15:58.322 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fedca8b2160>, signals=[15]) 2024-01-26T23:15:58.323 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-26T23:15:58.330 INFO:teuthology.task.internal:Opening connections... 2024-01-26T23:15:58.330 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:15:58.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-26T23:15:58.408 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:15:58.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-26T23:15:58.479 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-26T23:15:58.487 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2024-01-26T23:15:58.508 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2024-01-26T23:15:58.509 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2024-01-26T23:15:58.569 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="Red Hat Enterprise Linux" 2024-01-26T23:15:58.569 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="8.6 (Ootpa)" 2024-01-26T23:15:58.569 INFO:teuthology.orchestra.run.smithi046.stdout:ID="rhel" 2024-01-26T23:15:58.569 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="fedora" 2024-01-26T23:15:58.570 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="8.6" 2024-01-26T23:15:58.570 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el8" 2024-01-26T23:15:58.570 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-26T23:15:58.570 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2024-01-26T23:15:58.570 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-26T23:15:58.570 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://www.redhat.com/" 2024-01-26T23:15:58.570 INFO:teuthology.orchestra.run.smithi046.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-26T23:15:58.570 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-26T23:15:58.570 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:15:58.570 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-26T23:15:58.570 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-26T23:15:58.570 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-26T23:15:58.570 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-26T23:15:58.571 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2024-01-26T23:15:58.595 DEBUG:teuthology.orchestra.run.smithi106:> uname -m 2024-01-26T23:15:58.616 INFO:teuthology.orchestra.run.smithi106.stdout:x86_64 2024-01-26T23:15:58.616 DEBUG:teuthology.orchestra.run.smithi106:> cat /etc/os-release 2024-01-26T23:15:58.675 INFO:teuthology.orchestra.run.smithi106.stdout:NAME="Red Hat Enterprise Linux" 2024-01-26T23:15:58.676 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION="8.6 (Ootpa)" 2024-01-26T23:15:58.676 INFO:teuthology.orchestra.run.smithi106.stdout:ID="rhel" 2024-01-26T23:15:58.676 INFO:teuthology.orchestra.run.smithi106.stdout:ID_LIKE="fedora" 2024-01-26T23:15:58.676 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_ID="8.6" 2024-01-26T23:15:58.676 INFO:teuthology.orchestra.run.smithi106.stdout:PLATFORM_ID="platform:el8" 2024-01-26T23:15:58.676 INFO:teuthology.orchestra.run.smithi106.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-26T23:15:58.676 INFO:teuthology.orchestra.run.smithi106.stdout:ANSI_COLOR="0;31" 2024-01-26T23:15:58.676 INFO:teuthology.orchestra.run.smithi106.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-26T23:15:58.676 INFO:teuthology.orchestra.run.smithi106.stdout:HOME_URL="https://www.redhat.com/" 2024-01-26T23:15:58.676 INFO:teuthology.orchestra.run.smithi106.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-26T23:15:58.676 INFO:teuthology.orchestra.run.smithi106.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-26T23:15:58.676 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:15:58.676 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-26T23:15:58.676 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-26T23:15:58.677 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-26T23:15:58.677 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-26T23:15:58.677 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server 2024-01-26T23:15:58.695 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-26T23:15:58.705 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-26T23:15:58.713 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-26T23:15:58.713 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2024-01-26T23:15:58.715 DEBUG:teuthology.orchestra.run.smithi106:> test '!' -e /home/ubuntu/cephtest 2024-01-26T23:15:58.735 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-26T23:15:58.743 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-26T23:15:58.743 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2024-01-26T23:15:58.776 DEBUG:teuthology.orchestra.run.smithi106:> test -z $(ls -A /var/lib/ceph) 2024-01-26T23:15:58.813 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-26T23:15:58.868 INFO:teuthology.run_tasks:Running task kernel... 2024-01-26T23:15:58.883 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-26T23:15:58.883 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-26T23:15:58.884 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-26T23:15:58.884 DEBUG:teuthology.orchestra.run.smithi106:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-26T23:15:58.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:15:58.903 DEBUG:teuthology.orchestra.run.smithi106:> uname -r 2024-01-26T23:15:58.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:15:58.904 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-01-26T23:15:58.964 INFO:teuthology.orchestra.run.smithi106.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-26T23:15:58.964 INFO:teuthology.task.kernel:Running kernel on smithi106: 4.18.0-513.9.1.el8_9.x86_64 2024-01-26T23:15:58.964 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum install -y kernel 2024-01-26T23:15:58.965 INFO:teuthology.orchestra.run.smithi046.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-26T23:15:58.966 INFO:teuthology.task.kernel:Running kernel on smithi046: 4.18.0-513.9.1.el8_9.x86_64 2024-01-26T23:15:58.966 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum install -y kernel 2024-01-26T23:16:00.548 INFO:teuthology.orchestra.run.smithi106.stdout:Updating Subscription Management repositories. 2024-01-26T23:16:00.548 INFO:teuthology.orchestra.run.smithi106.stdout:Unable to read consumer identity 2024-01-26T23:16:00.619 INFO:teuthology.orchestra.run.smithi046.stdout:Updating Subscription Management repositories. 2024-01-26T23:16:00.619 INFO:teuthology.orchestra.run.smithi046.stdout:Unable to read consumer identity 2024-01-26T23:16:00.812 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:16:00.812 INFO:teuthology.orchestra.run.smithi106.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-26T23:16:00.812 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:16:00.892 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:16:00.892 INFO:teuthology.orchestra.run.smithi046.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-26T23:16:00.892 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:16:02.607 INFO:teuthology.orchestra.run.smithi046.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-01-26T23:16:02.632 INFO:teuthology.orchestra.run.smithi106.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-01-26T23:16:03.985 INFO:teuthology.orchestra.run.smithi106.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-01-26T23:16:04.315 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 9.7 MB/s | 16 MB 00:01 2024-01-26T23:16:07.600 INFO:teuthology.orchestra.run.smithi106.stdout:lab-extras 407 kB/s | 24 kB 00:00 2024-01-26T23:16:08.361 INFO:teuthology.orchestra.run.smithi106.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-26T23:16:08.361 INFO:teuthology.orchestra.run.smithi106.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-26T23:16:08.415 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 532 kB/s | 24 kB 00:00 2024-01-26T23:16:08.427 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-26T23:16:08.429 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-01-26T23:16:08.429 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-26T23:16:08.572 DEBUG:teuthology.orchestra.run.smithi106:> echo no | sudo yum reinstall kernel || true 2024-01-26T23:16:08.907 INFO:teuthology.orchestra.run.smithi106.stdout:Updating Subscription Management repositories. 2024-01-26T23:16:08.907 INFO:teuthology.orchestra.run.smithi106.stdout:Unable to read consumer identity 2024-01-26T23:16:09.153 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:16:09.153 INFO:teuthology.orchestra.run.smithi106.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-26T23:16:09.153 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:16:09.210 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-26T23:16:09.210 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-26T23:16:09.311 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-26T23:16:09.313 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-26T23:16:09.313 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-26T23:16:09.464 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:02 ago on Fri 26 Jan 2024 11:16:07 PM UTC. 2024-01-26T23:16:09.491 DEBUG:teuthology.orchestra.run.smithi046:> echo no | sudo yum reinstall kernel || true 2024-01-26T23:16:09.542 INFO:teuthology.orchestra.run.smithi106.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-26T23:16:09.558 INFO:teuthology.orchestra.run.smithi106.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-26T23:16:09.596 INFO:teuthology.orchestra.run.smithi106.stderr:Error: No packages marked for reinstall. 2024-01-26T23:16:09.643 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum reinstall -y kernel || true 2024-01-26T23:16:09.924 INFO:teuthology.orchestra.run.smithi046.stdout:Updating Subscription Management repositories. 2024-01-26T23:16:09.924 INFO:teuthology.orchestra.run.smithi046.stdout:Unable to read consumer identity 2024-01-26T23:16:09.976 INFO:teuthology.orchestra.run.smithi106.stdout:Updating Subscription Management repositories. 2024-01-26T23:16:09.976 INFO:teuthology.orchestra.run.smithi106.stdout:Unable to read consumer identity 2024-01-26T23:16:10.139 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:16:10.139 INFO:teuthology.orchestra.run.smithi046.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-26T23:16:10.139 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:16:10.200 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:16:10.200 INFO:teuthology.orchestra.run.smithi106.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-26T23:16:10.200 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:16:10.502 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:03 ago on Fri 26 Jan 2024 11:16:07 PM UTC. 2024-01-26T23:16:10.503 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:02 ago on Fri 26 Jan 2024 11:16:08 PM UTC. 2024-01-26T23:16:10.569 INFO:teuthology.orchestra.run.smithi106.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-26T23:16:10.569 INFO:teuthology.orchestra.run.smithi106.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-26T23:16:10.584 INFO:teuthology.orchestra.run.smithi046.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-26T23:16:10.592 INFO:teuthology.orchestra.run.smithi046.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-26T23:16:10.602 INFO:teuthology.orchestra.run.smithi106.stderr:Error: No packages marked for reinstall. 2024-01-26T23:16:10.649 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q kernel | sort -rV | head -n 1 2024-01-26T23:16:10.655 INFO:teuthology.orchestra.run.smithi046.stderr:Error: No packages marked for reinstall. 2024-01-26T23:16:10.713 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum reinstall -y kernel || true 2024-01-26T23:16:10.808 INFO:teuthology.orchestra.run.smithi106.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-26T23:16:10.809 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-26T23:16:10.809 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-26T23:16:10.809 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-01-26T23:16:10.809 DEBUG:teuthology.orchestra.run.smithi106:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-26T23:16:10.840 INFO:teuthology.orchestra.run.smithi106.stdout:ttyS1 2024-01-26T23:16:10.853 DEBUG:teuthology.parallel:result is None 2024-01-26T23:16:11.096 INFO:teuthology.orchestra.run.smithi046.stdout:Updating Subscription Management repositories. 2024-01-26T23:16:11.097 INFO:teuthology.orchestra.run.smithi046.stdout:Unable to read consumer identity 2024-01-26T23:16:11.333 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:16:11.333 INFO:teuthology.orchestra.run.smithi046.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-26T23:16:11.333 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:16:11.692 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:03 ago on Fri 26 Jan 2024 11:16:08 PM UTC. 2024-01-26T23:16:11.764 INFO:teuthology.orchestra.run.smithi046.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-26T23:16:11.764 INFO:teuthology.orchestra.run.smithi046.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-26T23:16:11.810 INFO:teuthology.orchestra.run.smithi046.stderr:Error: No packages marked for reinstall. 2024-01-26T23:16:11.864 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2024-01-26T23:16:12.048 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-26T23:16:12.048 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-26T23:16:12.048 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-26T23:16:12.048 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-26T23:16:12.048 DEBUG:teuthology.orchestra.run.smithi046:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-26T23:16:12.087 INFO:teuthology.orchestra.run.smithi046.stdout:ttyS1 2024-01-26T23:16:12.101 DEBUG:teuthology.parallel:result is None 2024-01-26T23:16:12.101 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-26T23:16:12.110 INFO:teuthology.task.internal:Creating test directory... 2024-01-26T23:16:12.110 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-26T23:16:12.144 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-26T23:16:12.164 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-26T23:16:12.172 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-26T23:16:12.179 INFO:teuthology.task.internal:Creating archive directory... 2024-01-26T23:16:12.179 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-26T23:16:12.204 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-26T23:16:12.235 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-26T23:16:12.242 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-26T23:16:12.242 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-26T23:16:12.273 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-26T23:16:12.311 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-26T23:16:12.318 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-26T23:16:12.325 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-26T23:16:12.331 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-26T23:16:12.335 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-26T23:16:12.342 INFO:teuthology.task.internal:Configuring sudo... 2024-01-26T23:16:12.342 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-26T23:16:12.369 DEBUG:teuthology.orchestra.run.smithi106:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-26T23:16:12.402 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-26T23:16:12.410 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-26T23:16:12.410 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-26T23:16:12.444 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-26T23:16:12.464 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-26T23:16:12.554 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-26T23:16:12.606 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-26T23:16:12.695 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-26T23:16:12.744 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:16:12.744 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-26T23:16:12.818 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-26T23:16:12.867 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-26T23:16:12.931 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-26T23:16:12.981 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-26T23:16:13.028 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:16:13.028 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-26T23:16:13.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2024-01-26T23:16:13.101 DEBUG:teuthology.orchestra.run.smithi106:> sudo service rsyslog restart 2024-01-26T23:16:13.154 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-26T23:16:13.186 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-26T23:16:13.579 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-26T23:16:13.590 INFO:teuthology.task.internal:Starting timer... 2024-01-26T23:16:13.591 INFO:teuthology.run_tasks:Running task pcp... 2024-01-26T23:16:13.601 INFO:teuthology.run_tasks:Running task selinux... 2024-01-26T23:16:13.610 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-26T23:16:13.610 DEBUG:teuthology.orchestra.run.smithi046:> sudo service auditd rotate 2024-01-26T23:16:13.720 INFO:teuthology.orchestra.run.smithi046.stdout:Rotating logs: 2024-01-26T23:16:13.723 DEBUG:teuthology.orchestra.run.smithi106:> sudo service auditd rotate 2024-01-26T23:16:13.814 INFO:teuthology.orchestra.run.smithi106.stdout:Rotating logs: 2024-01-26T23:16:13.815 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-26T23:16:13.815 DEBUG:teuthology.orchestra.run.smithi046:> /usr/sbin/getenforce 2024-01-26T23:16:13.842 INFO:teuthology.orchestra.run.smithi046.stdout:Permissive 2024-01-26T23:16:13.843 DEBUG:teuthology.orchestra.run.smithi106:> /usr/sbin/getenforce 2024-01-26T23:16:13.862 INFO:teuthology.orchestra.run.smithi106.stdout:Permissive 2024-01-26T23:16:13.862 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi046.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi106.front.sepia.ceph.com': 'permissive'} 2024-01-26T23:16:13.863 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-26T23:16:13.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:16:13.920 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-26T23:16:13.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:16:13.948 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-26T23:16:13.948 DEBUG:teuthology.orchestra.run.smithi046:> sudo /usr/sbin/setenforce permissive 2024-01-26T23:16:13.993 DEBUG:teuthology.orchestra.run.smithi106:> sudo /usr/sbin/setenforce permissive 2024-01-26T23:16:14.019 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-26T23:16:14.031 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-26T23:16:14.047 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-26T23:16:14.048 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi046.front.sepia.ceph.com,smithi106.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-26T23:25:34.338 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi046.front.sepia.ceph.com'), Remote(name='ubuntu@smithi106.front.sepia.ceph.com')] 2024-01-26T23:25:34.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-01-26T23:25:34.341 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-26T23:25:34.421 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-01-26T23:25:34.501 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-01-26T23:25:34.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-01-26T23:25:34.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-26T23:25:34.584 DEBUG:teuthology.orchestra.run.smithi106:> true 2024-01-26T23:25:34.658 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-01-26T23:25:34.659 INFO:teuthology.run_tasks:Running task clock... 2024-01-26T23:25:34.676 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-26T23:25:34.676 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-26T23:25:34.676 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-26T23:25:34.679 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-26T23:25:34.679 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-26T23:25:34.717 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-26T23:25:34.734 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-26T23:25:34.735 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-26T23:25:34.749 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-26T23:25:34.818 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: ntpd: command not found 2024-01-26T23:25:34.828 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: ntpd: command not found 2024-01-26T23:25:34.830 INFO:teuthology.orchestra.run.smithi106.stdout:506 Cannot talk to daemon 2024-01-26T23:25:34.843 INFO:teuthology.orchestra.run.smithi046.stdout:506 Cannot talk to daemon 2024-01-26T23:25:34.845 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-26T23:25:34.860 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-26T23:25:34.861 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-26T23:25:34.877 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-26T23:25:34.933 INFO:teuthology.orchestra.run.smithi106.stderr:bash: ntpq: command not found 2024-01-26T23:25:34.936 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-26T23:25:34.936 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-01-26T23:25:34.936 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-26T23:25:34.936 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-26T23:25:34.936 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-26T23:25:34.936 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-26T23:25:34.953 INFO:teuthology.orchestra.run.smithi046.stderr:bash: ntpq: command not found 2024-01-26T23:25:34.958 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-26T23:25:34.958 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-01-26T23:25:34.958 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-26T23:25:34.958 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-26T23:25:34.958 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-26T23:25:34.959 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-26T23:25:34.959 INFO:teuthology.run_tasks:Running task install... 2024-01-26T23:25:34.980 DEBUG:teuthology.task.install:project ceph 2024-01-26T23:25:34.981 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7495007658df82356749387b98425d4a7fcef9b6'}} 2024-01-26T23:25:34.981 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '7495007658df82356749387b98425d4a7fcef9b6'} 2024-01-26T23:25:34.981 INFO:teuthology.task.install:Using flavor: default 2024-01-26T23:25:34.987 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-26T23:25:34.987 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-01-26T23:25:34.988 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7495007658df82356749387b98425d4a7fcef9b6', 'tag': None, 'wait_for_package': False} 2024-01-26T23:25:34.988 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7495007658df82356749387b98425d4a7fcef9b6 2024-01-26T23:25:34.990 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7495007658df82356749387b98425d4a7fcef9b6', 'tag': None, 'wait_for_package': False} 2024-01-26T23:25:34.990 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7495007658df82356749387b98425d4a7fcef9b6 2024-01-26T23:25:35.141 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-01-25-1331-quincy/7495007658df82356749387b98425d4a7fcef9b6/centos/8/flavors/default/ 2024-01-26T23:25:35.141 INFO:teuthology.task.install.rpm:Package version is 17.2.7-472.g74950076 2024-01-26T23:25:35.346 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-01-25-1331-quincy/7495007658df82356749387b98425d4a7fcef9b6/centos/8/flavors/default/ 2024-01-26T23:25:35.346 INFO:teuthology.task.install.rpm:Package version is 17.2.7-472.g74950076 2024-01-26T23:25:35.432 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-01-25-1331-quincy/7495007658df82356749387b98425d4a7fcef9b6/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-01-25-1331-quincy/7495007658df82356749387b98425d4a7fcef9b6/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-01-25-1331-quincy/7495007658df82356749387b98425d4a7fcef9b6/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-26T23:25:35.432 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:25:35.432 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-26T23:25:35.464 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2024-01-26T23:25:35.464 DEBUG:teuthology.orchestra.run.smithi106:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7495007658df82356749387b98425d4a7fcef9b6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-26T23:25:35.534 DEBUG:teuthology.orchestra.run.smithi106:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-26T23:25:35.615 DEBUG:teuthology.orchestra.run.smithi106:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-26T23:25:35.643 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-01-25-1331-quincy/7495007658df82356749387b98425d4a7fcef9b6/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-01-25-1331-quincy/7495007658df82356749387b98425d4a7fcef9b6/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-01-25-1331-quincy/7495007658df82356749387b98425d4a7fcef9b6/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-26T23:25:35.643 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:25:35.643 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-26T23:25:35.679 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2024-01-26T23:25:35.679 DEBUG:teuthology.orchestra.run.smithi046:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7495007658df82356749387b98425d4a7fcef9b6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-26T23:25:35.682 INFO:teuthology.orchestra.run.smithi106.stdout:check_obsoletes = 1 2024-01-26T23:25:35.684 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum clean all 2024-01-26T23:25:35.755 DEBUG:teuthology.orchestra.run.smithi046:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-26T23:25:35.844 DEBUG:teuthology.orchestra.run.smithi046:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-26T23:25:35.876 INFO:teuthology.orchestra.run.smithi046.stdout:check_obsoletes = 1 2024-01-26T23:25:35.878 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum clean all 2024-01-26T23:25:36.061 INFO:teuthology.orchestra.run.smithi106.stdout:Updating Subscription Management repositories. 2024-01-26T23:25:36.312 INFO:teuthology.orchestra.run.smithi046.stdout:Updating Subscription Management repositories. 2024-01-26T23:25:37.296 INFO:teuthology.orchestra.run.smithi106.stdout:57 files removed 2024-01-26T23:25:37.352 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2024-01-26T23:25:37.577 INFO:teuthology.orchestra.run.smithi046.stdout:57 files removed 2024-01-26T23:25:37.633 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2024-01-26T23:25:37.685 INFO:teuthology.orchestra.run.smithi106.stdout:Updating Subscription Management repositories. 2024-01-26T23:25:38.016 INFO:teuthology.orchestra.run.smithi046.stdout:Updating Subscription Management repositories. 2024-01-26T23:25:39.611 INFO:teuthology.orchestra.run.smithi106.stdout:ceph packages for x86_64 189 kB/s | 78 kB 00:00 2024-01-26T23:25:40.036 INFO:teuthology.orchestra.run.smithi106.stdout:ceph noarch packages 24 kB/s | 9.2 kB 00:00 2024-01-26T23:25:40.281 INFO:teuthology.orchestra.run.smithi046.stdout:ceph packages for x86_64 160 kB/s | 78 kB 00:00 2024-01-26T23:25:40.370 INFO:teuthology.orchestra.run.smithi106.stdout:ceph source packages 4.7 kB/s | 1.5 kB 00:00 2024-01-26T23:25:40.589 INFO:teuthology.orchestra.run.smithi106.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2024-01-26T23:25:40.707 INFO:teuthology.orchestra.run.smithi046.stdout:ceph noarch packages 24 kB/s | 9.2 kB 00:00 2024-01-26T23:25:41.157 INFO:teuthology.orchestra.run.smithi046.stdout:ceph source packages 3.4 kB/s | 1.5 kB 00:00 2024-01-26T23:25:41.374 INFO:teuthology.orchestra.run.smithi046.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2024-01-26T23:25:42.572 INFO:teuthology.orchestra.run.smithi106.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 16 MB 00:01 2024-01-26T23:25:43.588 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 7.4 MB/s | 16 MB 00:02 2024-01-26T23:25:46.189 INFO:teuthology.orchestra.run.smithi106.stdout:lab-extras 296 kB/s | 24 kB 00:00 2024-01-26T23:25:47.507 INFO:teuthology.orchestra.run.smithi106.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 47 MB/s | 59 MB 00:01 2024-01-26T23:25:47.633 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 459 kB/s | 24 kB 00:00 2024-01-26T23:25:49.227 INFO:teuthology.orchestra.run.smithi046.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 43 MB/s | 65 MB 00:01 2024-01-26T23:25:58.758 INFO:teuthology.orchestra.run.smithi106.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 49 MB/s | 65 MB 00:01 2024-01-26T23:26:01.061 INFO:teuthology.orchestra.run.smithi046.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 40 MB/s | 59 MB 00:01 2024-01-26T23:26:09.033 INFO:teuthology.orchestra.run.smithi106.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 14 MB/s | 8.8 MB 00:00 2024-01-26T23:26:12.862 INFO:teuthology.orchestra.run.smithi046.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 13 MB/s | 8.8 MB 00:00 2024-01-26T23:26:15.955 INFO:teuthology.orchestra.run.smithi106.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-26T23:26:15.956 INFO:teuthology.orchestra.run.smithi106.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-26T23:26:15.974 INFO:teuthology.orchestra.run.smithi106.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-01-26T23:26:15.975 INFO:teuthology.orchestra.run.smithi106.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-01-26T23:26:15.977 INFO:teuthology.orchestra.run.smithi106.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-01-26T23:26:15.978 INFO:teuthology.orchestra.run.smithi106.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-01-26T23:26:15.981 INFO:teuthology.orchestra.run.smithi106.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-01-26T23:26:15.985 INFO:teuthology.orchestra.run.smithi106.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-01-26T23:26:15.986 INFO:teuthology.orchestra.run.smithi106.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-01-26T23:26:16.189 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-26T23:26:16.203 INFO:teuthology.orchestra.run.smithi106.stdout:=========================================================================================================================== 2024-01-26T23:26:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-01-26T23:26:16.203 INFO:teuthology.orchestra.run.smithi106.stdout:=========================================================================================================================== 2024-01-26T23:26:16.203 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-01-26T23:26:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-01-26T23:26:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-01-26T23:26:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: ceph x86_64 2:17.2.7-472.g74950076.el8 ceph 6.4 k 2024-01-26T23:26:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base x86_64 2:17.2.7-472.g74950076.el8 ceph 5.2 M 2024-01-26T23:26:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse x86_64 2:17.2.7-472.g74950076.el8 ceph 877 k 2024-01-26T23:26:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-472.g74950076.el8 ceph 162 k 2024-01-26T23:26:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr x86_64 2:17.2.7-472.g74950076.el8 ceph 1.6 M 2024-01-26T23:26:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-cephadm noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 121 k 2024-01-26T23:26:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-dashboard noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 1.7 M 2024-01-26T23:26:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 7.4 M 2024-01-26T23:26:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-rook noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 47 k 2024-01-26T23:26:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw x86_64 2:17.2.7-472.g74950076.el8 ceph 17 M 2024-01-26T23:26:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test x86_64 2:17.2.7-472.g74950076.el8 ceph 52 M 2024-01-26T23:26:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-volume noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 267 k 2024-01-26T23:26:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: cephadm noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 80 k 2024-01-26T23:26:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-mirror x86_64 2:17.2.7-472.g74950076.el8 ceph 255 k 2024-01-26T23:26:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-top noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 18 k 2024-01-26T23:26:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-01-26T23:26:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-01-26T23:26:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-01-26T23:26:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs-devel x86_64 2:17.2.7-472.g74950076.el8 ceph 25 k 2024-01-26T23:26:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2 x86_64 2:17.2.7-472.g74950076.el8 ceph 797 k 2024-01-26T23:26:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel x86_64 2:17.2.7-472.g74950076.el8 ceph 130 k 2024-01-26T23:26:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-01-26T23:26:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-01-26T23:26:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-01-26T23:26:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs x86_64 2:17.2.7-472.g74950076.el8 ceph 190 k 2024-01-26T23:26:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados x86_64 2:17.2.7-472.g74950076.el8 ceph 370 k 2024-01-26T23:26:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd x86_64 2:17.2.7-472.g74950076.el8 ceph 359 k 2024-01-26T23:26:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw x86_64 2:17.2.7-472.g74950076.el8 ceph 106 k 2024-01-26T23:26:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse x86_64 2:17.2.7-472.g74950076.el8 ceph 90 k 2024-01-26T23:26:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror x86_64 2:17.2.7-472.g74950076.el8 ceph 3.5 M 2024-01-26T23:26:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd x86_64 2:17.2.7-472.g74950076.el8 ceph 175 k 2024-01-26T23:26:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-01-26T23:26:16.206 INFO:teuthology.orchestra.run.smithi106.stdout:Upgrading: 2024-01-26T23:26:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-01-26T23:26:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-01-26T23:26:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-01-26T23:26:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-26T23:26:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-01-26T23:26:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-01-26T23:26:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: librados2 x86_64 2:17.2.7-472.g74950076.el8 ceph 3.5 M 2024-01-26T23:26:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: librbd1 x86_64 2:17.2.7-472.g74950076.el8 ceph 3.6 M 2024-01-26T23:26:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-01-26T23:26:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-01-26T23:26:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-01-26T23:26:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-01-26T23:26:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-01-26T23:26:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-01-26T23:26:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-01-26T23:26:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-01-26T23:26:16.207 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-01-26T23:26:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-01-26T23:26:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common x86_64 2:17.2.7-472.g74950076.el8 ceph 23 M 2024-01-26T23:26:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-grafana-dashboards noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 23 k 2024-01-26T23:26:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds x86_64 2:17.2.7-472.g74950076.el8 ceph 2.3 M 2024-01-26T23:26:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-modules-core noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 239 k 2024-01-26T23:26:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon x86_64 2:17.2.7-472.g74950076.el8 ceph 3.9 M 2024-01-26T23:26:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd x86_64 2:17.2.7-472.g74950076.el8 ceph 17 M 2024-01-26T23:26:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-prometheus-alerts noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 14 k 2024-01-26T23:26:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-selinux x86_64 2:17.2.7-472.g74950076.el8 ceph 24 k 2024-01-26T23:26:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-01-26T23:26:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-26T23:26:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-01-26T23:26:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-01-26T23:26:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-01-26T23:26:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: libcephsqlite x86_64 2:17.2.7-472.g74950076.el8 ceph 178 k 2024-01-26T23:26:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-01-26T23:26:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-01-26T23:26:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-26T23:26:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-01-26T23:26:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2024-01-26T23:26:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: libradosstriper1 x86_64 2:17.2.7-472.g74950076.el8 ceph 506 k 2024-01-26T23:26:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-01-26T23:26:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: librgw2 x86_64 2:17.2.7-472.g74950076.el8 ceph 5.6 M 2024-01-26T23:26:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-01-26T23:26:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-01-26T23:26:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-26T23:26:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-01-26T23:26:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-01-26T23:26:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-01-26T23:26:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2024-01-26T23:26:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-01-26T23:26:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-01-26T23:26:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-01-26T23:26:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-01-26T23:26:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-26T23:26:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-01-26T23:26:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-01-26T23:26:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-26T23:26:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-26T23:26:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-26T23:26:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-argparse x86_64 2:17.2.7-472.g74950076.el8 ceph 45 k 2024-01-26T23:26:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-common x86_64 2:17.2.7-472.g74950076.el8 ceph 115 k 2024-01-26T23:26:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-26T23:26:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-01-26T23:26:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-26T23:26:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-26T23:26:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2024-01-26T23:26:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-26T23:26:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-26T23:26:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-26T23:26:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-01-26T23:26:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-01-26T23:26:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-01-26T23:26:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-26T23:26:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-26T23:26:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-26T23:26:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-01-26T23:26:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-26T23:26:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-26T23:26:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-01-26T23:26:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-26T23:26:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-01-26T23:26:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-01-26T23:26:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-26T23:26:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-26T23:26:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-01-26T23:26:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-01-26T23:26:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-01-26T23:26:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-01-26T23:26:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-01-26T23:26:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-01-26T23:26:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-26T23:26:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-26T23:26:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-26T23:26:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-26T23:26:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-01-26T23:26:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-26T23:26:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-26T23:26:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-26T23:26:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-26T23:26:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-26T23:26:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-26T23:26:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-26T23:26:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-26T23:26:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-01-26T23:26:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-26T23:26:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-26T23:26:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-01-26T23:26:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-01-26T23:26:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-01-26T23:26:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:26:16.214 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-26T23:26:16.215 INFO:teuthology.orchestra.run.smithi106.stdout:=========================================================================================================================== 2024-01-26T23:26:16.215 INFO:teuthology.orchestra.run.smithi106.stdout:Install 124 Packages 2024-01-26T23:26:16.215 INFO:teuthology.orchestra.run.smithi106.stdout:Upgrade 16 Packages 2024-01-26T23:26:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:26:16.215 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 212 M 2024-01-26T23:26:16.216 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-26T23:26:16.501 INFO:teuthology.orchestra.run.smithi106.stdout:(1/140): ceph-17.2.7-472.g74950076.el8.x86_64.r 30 kB/s | 6.4 kB 00:00 2024-01-26T23:26:17.726 INFO:teuthology.orchestra.run.smithi106.stdout:(2/140): ceph-fuse-17.2.7-472.g74950076.el8.x86 716 kB/s | 877 kB 00:01 2024-01-26T23:26:17.985 INFO:teuthology.orchestra.run.smithi106.stdout:(3/140): ceph-immutable-object-cache-17.2.7-472 627 kB/s | 162 kB 00:00 2024-01-26T23:26:19.260 INFO:teuthology.orchestra.run.smithi106.stdout:(4/140): ceph-base-17.2.7-472.g74950076.el8.x86 1.7 MB/s | 5.2 MB 00:02 2024-01-26T23:26:19.759 INFO:teuthology.orchestra.run.smithi106.stdout:(5/140): ceph-mgr-17.2.7-472.g74950076.el8.x86_ 3.2 MB/s | 1.6 MB 00:00 2024-01-26T23:26:20.698 INFO:teuthology.orchestra.run.smithi046.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-26T23:26:20.699 INFO:teuthology.orchestra.run.smithi046.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-26T23:26:20.719 INFO:teuthology.orchestra.run.smithi046.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-01-26T23:26:20.721 INFO:teuthology.orchestra.run.smithi046.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-01-26T23:26:20.722 INFO:teuthology.orchestra.run.smithi046.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-01-26T23:26:20.724 INFO:teuthology.orchestra.run.smithi046.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-01-26T23:26:20.727 INFO:teuthology.orchestra.run.smithi046.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-01-26T23:26:20.731 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-01-26T23:26:20.733 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-01-26T23:26:20.957 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-26T23:26:20.972 INFO:teuthology.orchestra.run.smithi046.stdout:=========================================================================================================================== 2024-01-26T23:26:20.972 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-26T23:26:20.972 INFO:teuthology.orchestra.run.smithi046.stdout:=========================================================================================================================== 2024-01-26T23:26:20.972 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-26T23:26:20.972 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-01-26T23:26:20.972 INFO:teuthology.orchestra.run.smithi046.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-01-26T23:26:20.973 INFO:teuthology.orchestra.run.smithi046.stdout: ceph x86_64 2:17.2.7-472.g74950076.el8 ceph 6.4 k 2024-01-26T23:26:20.973 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base x86_64 2:17.2.7-472.g74950076.el8 ceph 5.2 M 2024-01-26T23:26:20.973 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse x86_64 2:17.2.7-472.g74950076.el8 ceph 877 k 2024-01-26T23:26:20.973 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-472.g74950076.el8 ceph 162 k 2024-01-26T23:26:20.973 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr x86_64 2:17.2.7-472.g74950076.el8 ceph 1.6 M 2024-01-26T23:26:20.973 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 121 k 2024-01-26T23:26:20.973 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 1.7 M 2024-01-26T23:26:20.973 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 7.4 M 2024-01-26T23:26:20.973 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 47 k 2024-01-26T23:26:20.973 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw x86_64 2:17.2.7-472.g74950076.el8 ceph 17 M 2024-01-26T23:26:20.973 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test x86_64 2:17.2.7-472.g74950076.el8 ceph 52 M 2024-01-26T23:26:20.973 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 267 k 2024-01-26T23:26:20.973 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 80 k 2024-01-26T23:26:20.974 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror x86_64 2:17.2.7-472.g74950076.el8 ceph 255 k 2024-01-26T23:26:20.974 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 18 k 2024-01-26T23:26:20.974 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-01-26T23:26:20.974 INFO:teuthology.orchestra.run.smithi046.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-01-26T23:26:20.974 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-01-26T23:26:20.974 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel x86_64 2:17.2.7-472.g74950076.el8 ceph 25 k 2024-01-26T23:26:20.974 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2 x86_64 2:17.2.7-472.g74950076.el8 ceph 797 k 2024-01-26T23:26:20.974 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel x86_64 2:17.2.7-472.g74950076.el8 ceph 130 k 2024-01-26T23:26:20.974 INFO:teuthology.orchestra.run.smithi046.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-01-26T23:26:20.974 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-01-26T23:26:20.974 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-01-26T23:26:20.974 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs x86_64 2:17.2.7-472.g74950076.el8 ceph 190 k 2024-01-26T23:26:20.974 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados x86_64 2:17.2.7-472.g74950076.el8 ceph 370 k 2024-01-26T23:26:20.975 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd x86_64 2:17.2.7-472.g74950076.el8 ceph 359 k 2024-01-26T23:26:20.975 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw x86_64 2:17.2.7-472.g74950076.el8 ceph 106 k 2024-01-26T23:26:20.975 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse x86_64 2:17.2.7-472.g74950076.el8 ceph 90 k 2024-01-26T23:26:20.975 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror x86_64 2:17.2.7-472.g74950076.el8 ceph 3.5 M 2024-01-26T23:26:20.975 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd x86_64 2:17.2.7-472.g74950076.el8 ceph 175 k 2024-01-26T23:26:20.975 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-01-26T23:26:20.975 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrading: 2024-01-26T23:26:20.975 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-01-26T23:26:20.975 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-01-26T23:26:20.975 INFO:teuthology.orchestra.run.smithi046.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-01-26T23:26:20.976 INFO:teuthology.orchestra.run.smithi046.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-26T23:26:20.976 INFO:teuthology.orchestra.run.smithi046.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-01-26T23:26:20.976 INFO:teuthology.orchestra.run.smithi046.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-01-26T23:26:20.976 INFO:teuthology.orchestra.run.smithi046.stdout: librados2 x86_64 2:17.2.7-472.g74950076.el8 ceph 3.5 M 2024-01-26T23:26:20.976 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1 x86_64 2:17.2.7-472.g74950076.el8 ceph 3.6 M 2024-01-26T23:26:20.976 INFO:teuthology.orchestra.run.smithi046.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-01-26T23:26:20.976 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-01-26T23:26:20.976 INFO:teuthology.orchestra.run.smithi046.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-01-26T23:26:20.976 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-01-26T23:26:20.976 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-01-26T23:26:20.976 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-01-26T23:26:20.976 INFO:teuthology.orchestra.run.smithi046.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-01-26T23:26:20.976 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-01-26T23:26:20.977 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-26T23:26:20.977 INFO:teuthology.orchestra.run.smithi046.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-01-26T23:26:20.977 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common x86_64 2:17.2.7-472.g74950076.el8 ceph 23 M 2024-01-26T23:26:20.977 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 23 k 2024-01-26T23:26:20.977 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds x86_64 2:17.2.7-472.g74950076.el8 ceph 2.3 M 2024-01-26T23:26:20.977 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 239 k 2024-01-26T23:26:20.977 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon x86_64 2:17.2.7-472.g74950076.el8 ceph 3.9 M 2024-01-26T23:26:20.977 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd x86_64 2:17.2.7-472.g74950076.el8 ceph 17 M 2024-01-26T23:26:20.977 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts noarch 2:17.2.7-472.g74950076.el8 ceph-noarch 14 k 2024-01-26T23:26:20.977 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux x86_64 2:17.2.7-472.g74950076.el8 ceph 24 k 2024-01-26T23:26:20.977 INFO:teuthology.orchestra.run.smithi046.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-01-26T23:26:20.977 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-26T23:26:20.977 INFO:teuthology.orchestra.run.smithi046.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-01-26T23:26:20.977 INFO:teuthology.orchestra.run.smithi046.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-01-26T23:26:20.978 INFO:teuthology.orchestra.run.smithi046.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-01-26T23:26:20.978 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite x86_64 2:17.2.7-472.g74950076.el8 ceph 178 k 2024-01-26T23:26:20.978 INFO:teuthology.orchestra.run.smithi046.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-01-26T23:26:20.978 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-01-26T23:26:20.978 INFO:teuthology.orchestra.run.smithi046.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-26T23:26:20.978 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-01-26T23:26:20.978 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2024-01-26T23:26:20.978 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1 x86_64 2:17.2.7-472.g74950076.el8 ceph 506 k 2024-01-26T23:26:20.978 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-01-26T23:26:20.978 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2 x86_64 2:17.2.7-472.g74950076.el8 ceph 5.6 M 2024-01-26T23:26:20.978 INFO:teuthology.orchestra.run.smithi046.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-01-26T23:26:20.978 INFO:teuthology.orchestra.run.smithi046.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-01-26T23:26:20.978 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-26T23:26:20.978 INFO:teuthology.orchestra.run.smithi046.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-01-26T23:26:20.978 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-01-26T23:26:20.978 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-01-26T23:26:20.979 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2024-01-26T23:26:20.979 INFO:teuthology.orchestra.run.smithi046.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-01-26T23:26:20.979 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-01-26T23:26:20.979 INFO:teuthology.orchestra.run.smithi046.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-01-26T23:26:20.979 INFO:teuthology.orchestra.run.smithi046.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-01-26T23:26:20.979 INFO:teuthology.orchestra.run.smithi046.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-26T23:26:20.979 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-01-26T23:26:20.979 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-01-26T23:26:20.979 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-26T23:26:20.979 INFO:teuthology.orchestra.run.smithi046.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-26T23:26:20.979 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-26T23:26:20.979 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse x86_64 2:17.2.7-472.g74950076.el8 ceph 45 k 2024-01-26T23:26:20.979 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common x86_64 2:17.2.7-472.g74950076.el8 ceph 115 k 2024-01-26T23:26:20.979 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-26T23:26:20.979 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-01-26T23:26:20.980 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-26T23:26:20.980 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-26T23:26:20.980 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2024-01-26T23:26:20.980 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-26T23:26:20.980 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-26T23:26:20.980 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-26T23:26:20.980 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-01-26T23:26:20.980 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-01-26T23:26:20.980 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-01-26T23:26:20.980 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-26T23:26:20.980 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-26T23:26:20.980 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-26T23:26:20.980 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-01-26T23:26:20.981 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-26T23:26:20.981 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-26T23:26:20.981 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-01-26T23:26:20.981 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-26T23:26:20.981 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-01-26T23:26:20.981 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-01-26T23:26:20.981 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-26T23:26:20.981 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-26T23:26:20.981 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-01-26T23:26:20.981 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-01-26T23:26:20.981 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-01-26T23:26:20.981 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-01-26T23:26:20.981 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-01-26T23:26:20.981 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-01-26T23:26:20.981 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-26T23:26:20.982 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-26T23:26:20.982 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-26T23:26:20.982 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-26T23:26:20.982 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-01-26T23:26:20.982 INFO:teuthology.orchestra.run.smithi046.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-26T23:26:20.982 INFO:teuthology.orchestra.run.smithi046.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-26T23:26:20.982 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-26T23:26:20.982 INFO:teuthology.orchestra.run.smithi046.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-26T23:26:20.982 INFO:teuthology.orchestra.run.smithi046.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-26T23:26:20.982 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-26T23:26:20.982 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-26T23:26:20.982 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-26T23:26:20.982 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-01-26T23:26:20.983 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-26T23:26:20.983 INFO:teuthology.orchestra.run.smithi046.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-26T23:26:20.983 INFO:teuthology.orchestra.run.smithi046.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-01-26T23:26:20.983 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-01-26T23:26:20.983 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-01-26T23:26:20.983 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:26:20.983 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-26T23:26:20.983 INFO:teuthology.orchestra.run.smithi046.stdout:=========================================================================================================================== 2024-01-26T23:26:20.983 INFO:teuthology.orchestra.run.smithi046.stdout:Install 124 Packages 2024-01-26T23:26:20.983 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrade 16 Packages 2024-01-26T23:26:20.983 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:26:20.984 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 212 M 2024-01-26T23:26:20.984 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-26T23:26:21.052 INFO:teuthology.orchestra.run.smithi106.stdout:(6/140): ceph-mon-17.2.7-472.g74950076.el8.x86_ 3.0 MB/s | 3.9 MB 00:01 2024-01-26T23:26:21.194 INFO:teuthology.orchestra.run.smithi106.stdout:(7/140): ceph-mds-17.2.7-472.g74950076.el8.x86_ 737 kB/s | 2.3 MB 00:03 2024-01-26T23:26:21.497 INFO:teuthology.orchestra.run.smithi046.stdout:(1/140): ceph-17.2.7-472.g74950076.el8.x86_64.r 17 kB/s | 6.4 kB 00:00 2024-01-26T23:26:22.867 INFO:teuthology.orchestra.run.smithi046.stdout:(2/140): ceph-fuse-17.2.7-472.g74950076.el8.x86 640 kB/s | 877 kB 00:01 2024-01-26T23:26:23.063 INFO:teuthology.orchestra.run.smithi046.stdout:(3/140): ceph-immutable-object-cache-17.2.7-472 826 kB/s | 162 kB 00:00 2024-01-26T23:26:23.447 INFO:teuthology.orchestra.run.smithi046.stdout:(4/140): ceph-base-17.2.7-472.g74950076.el8.x86 2.2 MB/s | 5.2 MB 00:02 2024-01-26T23:26:24.211 INFO:teuthology.orchestra.run.smithi046.stdout:(5/140): ceph-mgr-17.2.7-472.g74950076.el8.x86_ 2.1 MB/s | 1.6 MB 00:00 2024-01-26T23:26:25.402 INFO:teuthology.orchestra.run.smithi106.stdout:(8/140): ceph-osd-17.2.7-472.g74950076.el8.x86_ 3.8 MB/s | 17 MB 00:04 2024-01-26T23:26:25.495 INFO:teuthology.orchestra.run.smithi106.stdout:(9/140): ceph-selinux-17.2.7-472.g74950076.el8. 267 kB/s | 24 kB 00:00 2024-01-26T23:26:25.614 INFO:teuthology.orchestra.run.smithi046.stdout:(6/140): ceph-mds-17.2.7-472.g74950076.el8.x86_ 927 kB/s | 2.3 MB 00:02 2024-01-26T23:26:25.848 INFO:teuthology.orchestra.run.smithi046.stdout:(7/140): ceph-mon-17.2.7-472.g74950076.el8.x86_ 2.4 MB/s | 3.9 MB 00:01 2024-01-26T23:26:28.527 INFO:teuthology.orchestra.run.smithi106.stdout:(10/140): ceph-common-17.2.7-472.g74950076.el8. 1.9 MB/s | 23 MB 00:12 2024-01-26T23:26:28.711 INFO:teuthology.orchestra.run.smithi106.stdout:(11/140): cephfs-mirror-17.2.7-472.g74950076.el 1.4 MB/s | 255 kB 00:00 2024-01-26T23:26:28.803 INFO:teuthology.orchestra.run.smithi106.stdout:(12/140): libcephfs-devel-17.2.7-472.g74950076. 271 kB/s | 25 kB 00:00 2024-01-26T23:26:29.144 INFO:teuthology.orchestra.run.smithi106.stdout:(13/140): libcephfs2-17.2.7-472.g74950076.el8.x 2.3 MB/s | 797 kB 00:00 2024-01-26T23:26:29.320 INFO:teuthology.orchestra.run.smithi106.stdout:(14/140): libcephsqlite-17.2.7-472.g74950076.el 1.0 MB/s | 178 kB 00:00 2024-01-26T23:26:29.437 INFO:teuthology.orchestra.run.smithi106.stdout:(15/140): librados-devel-17.2.7-472.g74950076.e 1.1 MB/s | 130 kB 00:00 2024-01-26T23:26:29.762 INFO:teuthology.orchestra.run.smithi106.stdout:(16/140): libradosstriper1-17.2.7-472.g74950076 1.5 MB/s | 506 kB 00:00 2024-01-26T23:26:31.879 INFO:teuthology.orchestra.run.smithi106.stdout:(17/140): librgw2-17.2.7-472.g74950076.el8.x86_ 2.6 MB/s | 5.6 MB 00:02 2024-01-26T23:26:31.970 INFO:teuthology.orchestra.run.smithi106.stdout:(18/140): python3-ceph-argparse-17.2.7-472.g749 490 kB/s | 45 kB 00:00 2024-01-26T23:26:32.062 INFO:teuthology.orchestra.run.smithi106.stdout:(19/140): python3-ceph-common-17.2.7-472.g74950 1.2 MB/s | 115 kB 00:00 2024-01-26T23:26:32.213 INFO:teuthology.orchestra.run.smithi106.stdout:(20/140): python3-cephfs-17.2.7-472.g74950076.e 1.2 MB/s | 190 kB 00:00 2024-01-26T23:26:32.256 INFO:teuthology.orchestra.run.smithi046.stdout:(8/140): ceph-radosgw-17.2.7-472.g74950076.el8. 2.6 MB/s | 17 MB 00:06 2024-01-26T23:26:32.348 INFO:teuthology.orchestra.run.smithi046.stdout:(9/140): ceph-selinux-17.2.7-472.g74950076.el8. 267 kB/s | 24 kB 00:00 2024-01-26T23:26:32.430 INFO:teuthology.orchestra.run.smithi106.stdout:(21/140): python3-rados-17.2.7-472.g74950076.el 1.7 MB/s | 370 kB 00:00 2024-01-26T23:26:32.671 INFO:teuthology.orchestra.run.smithi106.stdout:(22/140): python3-rbd-17.2.7-472.g74950076.el8. 1.5 MB/s | 359 kB 00:00 2024-01-26T23:26:32.772 INFO:teuthology.orchestra.run.smithi106.stdout:(23/140): python3-rgw-17.2.7-472.g74950076.el8. 1.0 MB/s | 106 kB 00:00 2024-01-26T23:26:32.881 INFO:teuthology.orchestra.run.smithi106.stdout:(24/140): rbd-fuse-17.2.7-472.g74950076.el8.x86 832 kB/s | 90 kB 00:00 2024-01-26T23:26:33.465 INFO:teuthology.orchestra.run.smithi046.stdout:(10/140): ceph-common-17.2.7-472.g74950076.el8. 1.9 MB/s | 23 MB 00:12 2024-01-26T23:26:33.698 INFO:teuthology.orchestra.run.smithi046.stdout:(11/140): cephfs-mirror-17.2.7-472.g74950076.el 1.1 MB/s | 255 kB 00:00 2024-01-26T23:26:33.815 INFO:teuthology.orchestra.run.smithi046.stdout:(12/140): libcephfs-devel-17.2.7-472.g74950076. 213 kB/s | 25 kB 00:00 2024-01-26T23:26:33.981 INFO:teuthology.orchestra.run.smithi106.stdout:(25/140): ceph-radosgw-17.2.7-472.g74950076.el8 1.3 MB/s | 17 MB 00:12 2024-01-26T23:26:34.149 INFO:teuthology.orchestra.run.smithi106.stdout:(26/140): rbd-nbd-17.2.7-472.g74950076.el8.x86_ 1.0 MB/s | 175 kB 00:00 2024-01-26T23:26:34.223 INFO:teuthology.orchestra.run.smithi106.stdout:(27/140): rbd-mirror-17.2.7-472.g74950076.el8.x 2.6 MB/s | 3.5 MB 00:01 2024-01-26T23:26:34.248 INFO:teuthology.orchestra.run.smithi106.stdout:(28/140): ceph-grafana-dashboards-17.2.7-472.g7 235 kB/s | 23 kB 00:00 2024-01-26T23:26:34.266 INFO:teuthology.orchestra.run.smithi046.stdout:(13/140): libcephfs2-17.2.7-472.g74950076.el8.x 1.7 MB/s | 797 kB 00:00 2024-01-26T23:26:34.299 INFO:teuthology.orchestra.run.smithi106.stdout:(29/140): ceph-mgr-cephadm-17.2.7-472.g74950076 1.6 MB/s | 121 kB 00:00 2024-01-26T23:26:34.708 INFO:teuthology.orchestra.run.smithi046.stdout:(14/140): libcephsqlite-17.2.7-472.g74950076.el 404 kB/s | 178 kB 00:00 2024-01-26T23:26:34.892 INFO:teuthology.orchestra.run.smithi046.stdout:(15/140): librados-devel-17.2.7-472.g74950076.e 709 kB/s | 130 kB 00:00 2024-01-26T23:26:35.299 INFO:teuthology.orchestra.run.smithi106.stdout:(30/140): ceph-mgr-dashboard-17.2.7-472.g749500 1.6 MB/s | 1.7 MB 00:01 2024-01-26T23:26:35.333 INFO:teuthology.orchestra.run.smithi046.stdout:(16/140): libradosstriper1-17.2.7-472.g74950076 1.1 MB/s | 506 kB 00:00 2024-01-26T23:26:35.458 INFO:teuthology.orchestra.run.smithi106.stdout:(31/140): ceph-mgr-modules-core-17.2.7-472.g749 1.5 MB/s | 239 kB 00:00 2024-01-26T23:26:35.558 INFO:teuthology.orchestra.run.smithi106.stdout:(32/140): ceph-mgr-rook-17.2.7-472.g74950076.el 475 kB/s | 47 kB 00:00 2024-01-26T23:26:35.650 INFO:teuthology.orchestra.run.smithi106.stdout:(33/140): ceph-prometheus-alerts-17.2.7-472.g74 158 kB/s | 14 kB 00:00 2024-01-26T23:26:35.834 INFO:teuthology.orchestra.run.smithi106.stdout:(34/140): ceph-volume-17.2.7-472.g74950076.el8. 1.4 MB/s | 267 kB 00:00 2024-01-26T23:26:35.934 INFO:teuthology.orchestra.run.smithi106.stdout:(35/140): cephadm-17.2.7-472.g74950076.el8.noar 805 kB/s | 80 kB 00:00 2024-01-26T23:26:36.026 INFO:teuthology.orchestra.run.smithi106.stdout:(36/140): cephfs-top-17.2.7-472.g74950076.el8.n 197 kB/s | 18 kB 00:00 2024-01-26T23:26:36.234 INFO:teuthology.orchestra.run.smithi106.stdout:(37/140): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2024-01-26T23:26:36.276 INFO:teuthology.orchestra.run.smithi106.stdout:(38/140): liboath-2.6.2-3.el8.x86_64.rpm 1.4 MB/s | 59 kB 00:00 2024-01-26T23:26:36.327 INFO:teuthology.orchestra.run.smithi106.stdout:(39/140): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2024-01-26T23:26:36.402 INFO:teuthology.orchestra.run.smithi106.stdout:(40/140): python3-asyncssh-2.7.0-2.el8.noarch.r 6.2 MB/s | 476 kB 00:00 2024-01-26T23:26:36.485 INFO:teuthology.orchestra.run.smithi106.stdout:(41/140): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 524 kB/s | 44 kB 00:00 2024-01-26T23:26:36.561 INFO:teuthology.orchestra.run.smithi106.stdout:(42/140): python3-beautifulsoup4-4.6.3-2.el8.1. 2.4 MB/s | 185 kB 00:00 2024-01-26T23:26:36.603 INFO:teuthology.orchestra.run.smithi106.stdout:(43/140): python3-cachetools-3.1.1-4.el8.noarch 795 kB/s | 33 kB 00:00 2024-01-26T23:26:36.670 INFO:teuthology.orchestra.run.smithi106.stdout:(44/140): python3-certifi-2018.10.15-7.el8.noar 244 kB/s | 16 kB 00:00 2024-01-26T23:26:36.770 INFO:teuthology.orchestra.run.smithi106.stdout:(45/140): python3-cheroot-8.5.2-1.el8.noarch.rp 1.7 MB/s | 173 kB 00:00 2024-01-26T23:26:36.820 INFO:teuthology.orchestra.run.smithi106.stdout:(46/140): python3-cherrypy-18.4.0-1.el8.noarch. 7.5 MB/s | 384 kB 00:00 2024-01-26T23:26:36.862 INFO:teuthology.orchestra.run.smithi106.stdout:(47/140): python3-google-auth-1.1.1-10.el8.noar 1.9 MB/s | 81 kB 00:00 2024-01-26T23:26:36.929 INFO:teuthology.orchestra.run.smithi106.stdout:(48/140): python3-jaraco-6.2-6.el8.noarch.rpm 167 kB/s | 11 kB 00:00 2024-01-26T23:26:36.988 INFO:teuthology.orchestra.run.smithi106.stdout:(49/140): python3-jaraco-functools-2.0-4.el8.no 301 kB/s | 18 kB 00:00 2024-01-26T23:26:37.096 INFO:teuthology.orchestra.run.smithi106.stdout:(50/140): python3-kubernetes-11.0.0-6.el8.noarc 11 MB/s | 1.2 MB 00:00 2024-01-26T23:26:37.194 INFO:teuthology.orchestra.run.smithi106.stdout:(51/140): python3-logutils-0.3.5-11.el8.noarch. 500 kB/s | 49 kB 00:00 2024-01-26T23:26:37.554 INFO:teuthology.orchestra.run.smithi106.stdout:(52/140): python3-more-itertools-7.2.0-3.el8.no 163 kB/s | 59 kB 00:00 2024-01-26T23:26:37.606 INFO:teuthology.orchestra.run.smithi106.stdout:(53/140): python3-natsort-7.1.1-2.el8.noarch.rp 1.1 MB/s | 60 kB 00:00 2024-01-26T23:26:37.656 INFO:teuthology.orchestra.run.smithi106.stdout:(54/140): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2024-01-26T23:26:37.756 INFO:teuthology.orchestra.run.smithi106.stdout:(55/140): python3-portend-2.6-1.el8.noarch.rpm 156 kB/s | 16 kB 00:00 2024-01-26T23:26:37.849 INFO:teuthology.orchestra.run.smithi106.stdout:(56/140): python3-repoze-lru-0.7-6.el8.noarch.r 367 kB/s | 34 kB 00:00 2024-01-26T23:26:37.899 INFO:teuthology.orchestra.run.smithi106.stdout:(57/140): python3-routes-2.4.1-12.el8.noarch.rp 3.9 MB/s | 196 kB 00:00 2024-01-26T23:26:37.949 INFO:teuthology.orchestra.run.smithi106.stdout:(58/140): python3-rsa-4.9-2.el8.noarch.rpm 1.3 MB/s | 64 kB 00:00 2024-01-26T23:26:38.016 INFO:teuthology.orchestra.run.smithi106.stdout:(59/140): python3-simplegeneric-0.8.1-17.el8.no 281 kB/s | 19 kB 00:00 2024-01-26T23:26:38.124 INFO:teuthology.orchestra.run.smithi106.stdout:(60/140): ceph-mgr-diskprediction-local-17.2.7- 1.9 MB/s | 7.4 MB 00:03 2024-01-26T23:26:38.150 INFO:teuthology.orchestra.run.smithi106.stdout:(61/140): python3-singledispatch-3.4.0.3-18.el8 179 kB/s | 24 kB 00:00 2024-01-26T23:26:38.225 INFO:teuthology.orchestra.run.smithi106.stdout:(62/140): python3-trustme-0.6.0-4.el8.noarch.rp 391 kB/s | 27 kB 00:00 2024-01-26T23:26:38.259 INFO:teuthology.orchestra.run.smithi106.stdout:(63/140): python3-tempora-1.14.1-5.el8.noarch.r 218 kB/s | 29 kB 00:00 2024-01-26T23:26:38.301 INFO:teuthology.orchestra.run.smithi106.stdout:(64/140): python3-waitress-1.4.3-1.el8.noarch.r 3.3 MB/s | 255 kB 00:00 2024-01-26T23:26:38.343 INFO:teuthology.orchestra.run.smithi106.stdout:(65/140): python3-websocket-client-0.56.0-5.el8 1.4 MB/s | 61 kB 00:00 2024-01-26T23:26:38.411 INFO:teuthology.orchestra.run.smithi106.stdout:(66/140): python3-webob-1.8.5-1.el8.1.noarch.rp 1.6 MB/s | 251 kB 00:00 2024-01-26T23:26:38.435 INFO:teuthology.orchestra.run.smithi106.stdout:(67/140): python3-webtest-2.0.33-1.el8.noarch.r 929 kB/s | 85 kB 00:00 2024-01-26T23:26:38.477 INFO:teuthology.orchestra.run.smithi106.stdout:(68/140): python3-zc-lockfile-2.0-2.el8.noarch. 343 kB/s | 23 kB 00:00 2024-01-26T23:26:38.502 INFO:teuthology.orchestra.run.smithi106.stdout:(69/140): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2024-01-26T23:26:38.561 INFO:teuthology.orchestra.run.smithi106.stdout:(70/140): thrift-0.13.0-2.el8.x86_64.rpm 14 MB/s | 1.7 MB 00:00 2024-01-26T23:26:38.720 INFO:teuthology.orchestra.run.smithi106.stdout:(71/140): python3-prettytable-0.7.2-14.el8.noar 278 kB/s | 44 kB 00:00 2024-01-26T23:26:38.753 INFO:teuthology.orchestra.run.smithi106.stdout:(72/140): python3-jsonpointer-1.10-11.el8.noarc 80 kB/s | 20 kB 00:00 2024-01-26T23:26:38.820 INFO:teuthology.orchestra.run.smithi106.stdout:(73/140): python3-jsonpatch-1.21-2.el8.noarch.r 398 kB/s | 27 kB 00:00 2024-01-26T23:26:38.950 INFO:teuthology.orchestra.run.smithi046.stdout:(17/140): librgw2-17.2.7-472.g74950076.el8.x86_ 1.5 MB/s | 5.6 MB 00:03 2024-01-26T23:26:38.978 INFO:teuthology.orchestra.run.smithi106.stdout:(74/140): python3-pyasn1-modules-0.3.7-6.el8.no 425 kB/s | 110 kB 00:00 2024-01-26T23:26:39.037 INFO:teuthology.orchestra.run.smithi106.stdout:(75/140): python3-werkzeug-0.12.2-4.el8.noarch. 7.7 MB/s | 457 kB 00:00 2024-01-26T23:26:39.054 INFO:teuthology.orchestra.run.smithi106.stdout:(76/140): python3-pyasn1-0.3.7-6.el8.noarch.rpm 540 kB/s | 126 kB 00:00 2024-01-26T23:26:39.163 INFO:teuthology.orchestra.run.smithi106.stdout:(77/140): bison-3.0.4-10.el8.x86_64.rpm 5.4 MB/s | 688 kB 00:00 2024-01-26T23:26:39.192 INFO:teuthology.orchestra.run.smithi046.stdout:(18/140): ceph-osd-17.2.7-472.g74950076.el8.x86 1.2 MB/s | 17 MB 00:13 2024-01-26T23:26:39.217 INFO:teuthology.orchestra.run.smithi046.stdout:(19/140): python3-ceph-argparse-17.2.7-472.g749 167 kB/s | 45 kB 00:00 2024-01-26T23:26:39.221 INFO:teuthology.orchestra.run.smithi106.stdout:(78/140): flex-2.6.1-9.el8.x86_64.rpm 1.9 MB/s | 320 kB 00:00 2024-01-26T23:26:39.301 INFO:teuthology.orchestra.run.smithi046.stdout:(20/140): python3-ceph-common-17.2.7-472.g74950 1.0 MB/s | 115 kB 00:00 2024-01-26T23:26:39.393 INFO:teuthology.orchestra.run.smithi046.stdout:(21/140): python3-cephfs-17.2.7-472.g74950076.e 1.1 MB/s | 190 kB 00:00 2024-01-26T23:26:39.430 INFO:teuthology.orchestra.run.smithi106.stdout:(79/140): python3-markupsafe-0.23-19.el8.x86_64 186 kB/s | 39 kB 00:00 2024-01-26T23:26:39.505 INFO:teuthology.orchestra.run.smithi106.stdout:(80/140): libtool-2.4.6-25.el8.x86_64.rpm 2.0 MB/s | 709 kB 00:00 2024-01-26T23:26:39.584 INFO:teuthology.orchestra.run.smithi046.stdout:(22/140): python3-rados-17.2.7-472.g74950076.el 1.3 MB/s | 370 kB 00:00 2024-01-26T23:26:39.606 INFO:teuthology.orchestra.run.smithi106.stdout:(81/140): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2024-01-26T23:26:39.635 INFO:teuthology.orchestra.run.smithi046.stdout:(23/140): python3-rbd-17.2.7-472.g74950076.el8. 1.4 MB/s | 359 kB 00:00 2024-01-26T23:26:39.681 INFO:teuthology.orchestra.run.smithi106.stdout:(82/140): oniguruma-6.8.2-2.el8.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-01-26T23:26:39.710 INFO:teuthology.orchestra.run.smithi046.stdout:(24/140): python3-rgw-17.2.7-472.g74950076.el8. 849 kB/s | 106 kB 00:00 2024-01-26T23:26:39.736 INFO:teuthology.orchestra.run.smithi046.stdout:(25/140): rbd-fuse-17.2.7-472.g74950076.el8.x86 900 kB/s | 90 kB 00:00 2024-01-26T23:26:39.765 INFO:teuthology.orchestra.run.smithi106.stdout:(83/140): python3-pyOpenSSL-19.0.0-1.el8.noarch 653 kB/s | 103 kB 00:00 2024-01-26T23:26:39.902 INFO:teuthology.orchestra.run.smithi046.stdout:(26/140): rbd-nbd-17.2.7-472.g74950076.el8.x86_ 1.0 MB/s | 175 kB 00:00 2024-01-26T23:26:40.011 INFO:teuthology.orchestra.run.smithi046.stdout:(27/140): ceph-grafana-dashboards-17.2.7-472.g7 215 kB/s | 23 kB 00:00 2024-01-26T23:26:40.021 INFO:teuthology.orchestra.run.smithi106.stdout:(84/140): python3-jinja2-2.10.1-3.el8.noarch.rp 2.1 MB/s | 538 kB 00:00 2024-01-26T23:26:40.095 INFO:teuthology.orchestra.run.smithi046.stdout:(28/140): ceph-mgr-cephadm-17.2.7-472.g74950076 1.4 MB/s | 121 kB 00:00 2024-01-26T23:26:40.123 INFO:teuthology.orchestra.run.smithi106.stdout:(85/140): python3-babel-2.5.1-7.el8.noarch.rpm 11 MB/s | 4.8 MB 00:00 2024-01-26T23:26:40.308 INFO:teuthology.orchestra.run.smithi106.stdout:(86/140): python3-numpy-1.14.3-10.el8.x86_64.rp 13 MB/s | 3.7 MB 00:00 2024-01-26T23:26:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:(87/140): socat-1.7.4.1-1.el8.x86_64.rpm 1.5 MB/s | 323 kB 00:00 2024-01-26T23:26:40.458 INFO:teuthology.orchestra.run.smithi106.stdout:(88/140): librdkafka-0.11.4-3.el8.x86_64.rpm 2.8 MB/s | 354 kB 00:00 2024-01-26T23:26:40.483 INFO:teuthology.orchestra.run.smithi106.stdout:(89/140): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.3 MB/s | 225 kB 00:00 2024-01-26T23:26:40.658 INFO:teuthology.orchestra.run.smithi106.stdout:(90/140): python3-mako-1.0.6-14.el8.noarch.rpm 897 kB/s | 157 kB 00:00 2024-01-26T23:26:40.925 INFO:teuthology.orchestra.run.smithi106.stdout:(91/140): openblas-threads-0.3.15-6.el8.x86_64. 11 MB/s | 5.2 MB 00:00 2024-01-26T23:26:41.042 INFO:teuthology.orchestra.run.smithi106.stdout:(92/140): openblas-0.3.15-6.el8.x86_64.rpm 13 MB/s | 5.0 MB 00:00 2024-01-26T23:26:41.092 INFO:teuthology.orchestra.run.smithi106.stdout:(93/140): automake-1.16.1-8.el8.noarch.rpm 4.2 MB/s | 714 kB 00:00 2024-01-26T23:26:41.120 INFO:teuthology.orchestra.run.smithi046.stdout:(29/140): ceph-mgr-dashboard-17.2.7-472.g749500 1.7 MB/s | 1.7 MB 00:01 2024-01-26T23:26:41.184 INFO:teuthology.orchestra.run.smithi106.stdout:(94/140): jq-1.6-7.el8.x86_64.rpm 1.4 MB/s | 202 kB 00:00 2024-01-26T23:26:41.293 INFO:teuthology.orchestra.run.smithi106.stdout:(95/140): python3-pytz-2017.2-11.el8.noarch.rpm 271 kB/s | 54 kB 00:00 2024-01-26T23:26:41.602 INFO:teuthology.orchestra.run.smithi106.stdout:(96/140): libverto-devel-0.3.0-5.el8.x86_64.rpm 58 kB/s | 18 kB 00:00 2024-01-26T23:26:41.777 INFO:teuthology.orchestra.run.smithi106.stdout:(97/140): python3-scipy-1.0.0-21.module+el8.9.0 23 MB/s | 14 MB 00:00 2024-01-26T23:26:41.835 INFO:teuthology.orchestra.run.smithi106.stdout:(98/140): python3-jwt-1.6.1-2.el8.noarch.rpm 182 kB/s | 43 kB 00:00 2024-01-26T23:26:41.956 INFO:teuthology.orchestra.run.smithi106.stdout:(99/140): python3-pycparser-2.14-14.el8.noarch. 910 kB/s | 109 kB 00:00 2024-01-26T23:26:42.003 INFO:teuthology.orchestra.run.smithi106.stdout:(100/140): python3-oauthlib-2.1.0-1.el8.noarch. 688 kB/s | 155 kB 00:00 2024-01-26T23:26:42.211 INFO:teuthology.orchestra.run.smithi106.stdout:(101/140): python3-requests-oauthlib-1.0.0-1.el 167 kB/s | 43 kB 00:00 2024-01-26T23:26:42.261 INFO:teuthology.orchestra.run.smithi106.stdout:(102/140): libaio-devel-0.3.112-1.el8.x86_64.rp 72 kB/s | 19 kB 00:00 2024-01-26T23:26:42.270 INFO:teuthology.orchestra.run.smithi046.stdout:(30/140): rbd-mirror-17.2.7-472.g74950076.el8.x 1.4 MB/s | 3.5 MB 00:02 2024-01-26T23:26:42.329 INFO:teuthology.orchestra.run.smithi106.stdout:(103/140): libzstd-devel-1.4.4-1.el8.x86_64.rpm 375 kB/s | 44 kB 00:00 2024-01-26T23:26:42.570 INFO:teuthology.orchestra.run.smithi106.stdout:(104/140): pcre2-devel-10.32-2.el8.x86_64.rpm 1.9 MB/s | 605 kB 00:00 2024-01-26T23:26:42.579 INFO:teuthology.orchestra.run.smithi046.stdout:(31/140): ceph-mgr-modules-core-17.2.7-472.g749 775 kB/s | 239 kB 00:00 2024-01-26T23:26:42.595 INFO:teuthology.orchestra.run.smithi106.stdout:(105/140): pcre2-utf16-10.32-2.el8.x86_64.rpm 858 kB/s | 229 kB 00:00 2024-01-26T23:26:42.804 INFO:teuthology.orchestra.run.smithi046.stdout:(32/140): ceph-mgr-rook-17.2.7-472.g74950076.el 210 kB/s | 47 kB 00:00 2024-01-26T23:26:42.821 INFO:teuthology.orchestra.run.smithi106.stdout:(106/140): pcre2-utf32-10.32-2.el8.x86_64.rpm 881 kB/s | 220 kB 00:00 2024-01-26T23:26:42.937 INFO:teuthology.orchestra.run.smithi046.stdout:(33/140): ceph-prometheus-alerts-17.2.7-472.g74 109 kB/s | 14 kB 00:00 2024-01-26T23:26:42.938 INFO:teuthology.orchestra.run.smithi106.stdout:(107/140): libselinux-devel-2.9-5.el8.x86_64.rp 584 kB/s | 200 kB 00:00 2024-01-26T23:26:43.114 INFO:teuthology.orchestra.run.smithi106.stdout:(108/140): libsepol-devel-2.9-3.el8.x86_64.rpm 498 kB/s | 87 kB 00:00 2024-01-26T23:26:43.131 INFO:teuthology.orchestra.run.smithi106.stdout:(109/140): keyutils-libs-devel-1.5.10-9.el8.x86 155 kB/s | 48 kB 00:00 2024-01-26T23:26:43.213 INFO:teuthology.orchestra.run.smithi046.stdout:(34/140): ceph-volume-17.2.7-472.g74950076.el8. 972 kB/s | 267 kB 00:00 2024-01-26T23:26:43.321 INFO:teuthology.orchestra.run.smithi046.stdout:(35/140): cephadm-17.2.7-472.g74950076.el8.noar 742 kB/s | 80 kB 00:00 2024-01-26T23:26:43.374 INFO:teuthology.orchestra.run.smithi106.stdout:(110/140): libcom_err-devel-1.45.6-4.el8.x86_64 159 kB/s | 39 kB 00:00 2024-01-26T23:26:43.406 INFO:teuthology.orchestra.run.smithi106.stdout:(111/140): zlib-devel-1.2.11-18.el8_5.x86_64.rp 197 kB/s | 58 kB 00:00 2024-01-26T23:26:43.449 INFO:teuthology.orchestra.run.smithi046.stdout:(36/140): cephfs-top-17.2.7-472.g74950076.el8.n 142 kB/s | 18 kB 00:00 2024-01-26T23:26:43.582 INFO:teuthology.orchestra.run.smithi106.stdout:(112/140): fuse-2.9.7-17.el8.x86_64.rpm 400 kB/s | 83 kB 00:00 2024-01-26T23:26:43.673 INFO:teuthology.orchestra.run.smithi106.stdout:(113/140): libgfortran-8.5.0-20.el8.x86_64.rpm 2.4 MB/s | 645 kB 00:00 2024-01-26T23:26:43.941 INFO:teuthology.orchestra.run.smithi106.stdout:(114/140): libquadmath-8.5.0-20.el8.x86_64.rpm 481 kB/s | 172 kB 00:00 2024-01-26T23:26:44.041 INFO:teuthology.orchestra.run.smithi106.stdout:(115/140): libuuid-devel-2.32.1-43.el8.x86_64.r 272 kB/s | 100 kB 00:00 2024-01-26T23:26:44.066 INFO:teuthology.orchestra.run.smithi106.stdout:(116/140): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.9 MB/s | 238 kB 00:00 2024-01-26T23:26:44.166 INFO:teuthology.orchestra.run.smithi106.stdout:(117/140): python3-cryptography-3.2.1-6.el8.x86 4.4 MB/s | 559 kB 00:00 2024-01-26T23:26:44.251 INFO:teuthology.orchestra.run.smithi046.stdout:(37/140): gperftools-libs-2.7-9.el8.x86_64.rpm 382 kB/s | 306 kB 00:00 2024-01-26T23:26:44.335 INFO:teuthology.orchestra.run.smithi106.stdout:(118/140): xfsprogs-devel-5.0.0-12.el8.x86_64.r 272 kB/s | 73 kB 00:00 2024-01-26T23:26:44.350 INFO:teuthology.orchestra.run.smithi106.stdout:(119/140): elfutils-libelf-devel-0.189-3.el8.x8 336 kB/s | 62 kB 00:00 2024-01-26T23:26:44.356 INFO:teuthology.orchestra.run.smithi046.stdout:(38/140): liboath-2.6.2-3.el8.x86_64.rpm 562 kB/s | 59 kB 00:00 2024-01-26T23:26:44.473 INFO:teuthology.orchestra.run.smithi046.stdout:(39/140): libunwind-1.3.1-3.el8.x86_64.rpm 644 kB/s | 75 kB 00:00 2024-01-26T23:26:44.492 INFO:teuthology.orchestra.run.smithi106.stdout:(120/140): librabbitmq-0.9.0-4.el8.x86_64.rpm 296 kB/s | 47 kB 00:00 2024-01-26T23:26:44.584 INFO:teuthology.orchestra.run.smithi106.stdout:(121/140): krb5-devel-1.18.2-26.el8_9.x86_64.rp 2.3 MB/s | 562 kB 00:00 2024-01-26T23:26:45.098 INFO:teuthology.orchestra.run.smithi046.stdout:(40/140): python3-asyncssh-2.7.0-2.el8.noarch.r 761 kB/s | 476 kB 00:00 2024-01-26T23:26:45.134 INFO:teuthology.orchestra.run.smithi106.stdout:(122/140): ceph-test-17.2.7-472.g74950076.el8.x 2.6 MB/s | 52 MB 00:19 2024-01-26T23:26:45.159 INFO:teuthology.orchestra.run.smithi106.stdout:(123/140): NetworkManager-initscripts-updown-1. 250 kB/s | 144 kB 00:00 2024-01-26T23:26:45.218 INFO:teuthology.orchestra.run.smithi106.stdout:(124/140): openssl-devel-1.1.1k-12.el8_9.x86_64 3.2 MB/s | 2.3 MB 00:00 2024-01-26T23:26:45.240 INFO:teuthology.orchestra.run.smithi046.stdout:(41/140): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 308 kB/s | 44 kB 00:00 2024-01-26T23:26:45.390 INFO:teuthology.orchestra.run.smithi046.stdout:(42/140): python3-beautifulsoup4-4.6.3-2.el8.1. 1.2 MB/s | 185 kB 00:00 2024-01-26T23:26:45.460 INFO:teuthology.orchestra.run.smithi106.stdout:(125/140): elfutils-libelf-0.189-3.el8.x86_64.r 960 kB/s | 232 kB 00:00 2024-01-26T23:26:45.483 INFO:teuthology.orchestra.run.smithi046.stdout:(43/140): ceph-mgr-diskprediction-local-17.2.7- 1.7 MB/s | 7.4 MB 00:04 2024-01-26T23:26:45.508 INFO:teuthology.orchestra.run.smithi046.stdout:(44/140): python3-cachetools-3.1.1-4.el8.noarch 282 kB/s | 33 kB 00:00 2024-01-26T23:26:45.642 INFO:teuthology.orchestra.run.smithi046.stdout:(45/140): python3-cheroot-8.5.2-1.el8.noarch.rp 1.3 MB/s | 173 kB 00:00 2024-01-26T23:26:45.694 INFO:teuthology.orchestra.run.smithi106.stdout:(126/140): elfutils-libs-0.189-3.el8.x86_64.rpm 1.3 MB/s | 303 kB 00:00 2024-01-26T23:26:45.892 INFO:teuthology.orchestra.run.smithi046.stdout:(46/140): python3-cherrypy-18.4.0-1.el8.noarch. 1.5 MB/s | 384 kB 00:00 2024-01-26T23:26:46.019 INFO:teuthology.orchestra.run.smithi106.stdout:(127/140): libmount-2.32.1-43.el8.x86_64.rpm 728 kB/s | 237 kB 00:00 2024-01-26T23:26:46.227 INFO:teuthology.orchestra.run.smithi106.stdout:(128/140): libsmartcols-2.32.1-43.el8.x86_64.rp 862 kB/s | 179 kB 00:00 2024-01-26T23:26:46.475 INFO:teuthology.orchestra.run.smithi046.stdout:(47/140): ceph-test-17.2.7-472.g74950076.el8.x8 3.7 MB/s | 52 MB 00:14 2024-01-26T23:26:46.500 INFO:teuthology.orchestra.run.smithi046.stdout:(48/140): python3-certifi-2018.10.15-7.el8.noar 16 kB/s | 16 kB 00:01 2024-01-26T23:26:46.502 INFO:teuthology.orchestra.run.smithi106.stdout:(129/140): libuuid-2.32.1-43.el8.x86_64.rpm 360 kB/s | 99 kB 00:00 2024-01-26T23:26:46.517 INFO:teuthology.orchestra.run.smithi046.stdout:(49/140): python3-google-auth-1.1.1-10.el8.noar 130 kB/s | 81 kB 00:00 2024-01-26T23:26:46.734 INFO:teuthology.orchestra.run.smithi046.stdout:(50/140): python3-jaraco-functools-2.0-4.el8.no 75 kB/s | 18 kB 00:00 2024-01-26T23:26:46.803 INFO:teuthology.orchestra.run.smithi106.stdout:(130/140): librbd1-17.2.7-472.g74950076.el8.x86 2.2 MB/s | 3.6 MB 00:01 2024-01-26T23:26:46.862 INFO:teuthology.orchestra.run.smithi106.stdout:(131/140): util-linux-2.32.1-43.el8.x86_64.rpm 6.9 MB/s | 2.5 MB 00:00 2024-01-26T23:26:46.876 INFO:teuthology.orchestra.run.smithi046.stdout:(51/140): python3-kubernetes-11.0.0-6.el8.noarc 3.3 MB/s | 1.2 MB 00:00 2024-01-26T23:26:46.893 INFO:teuthology.orchestra.run.smithi046.stdout:(52/140): python3-jaraco-6.2-6.el8.noarch.rpm 27 kB/s | 11 kB 00:00 2024-01-26T23:26:46.962 INFO:teuthology.orchestra.run.smithi106.stdout:(132/140): librados2-17.2.7-472.g74950076.el8.x 1.9 MB/s | 3.5 MB 00:01 2024-01-26T23:26:46.985 INFO:teuthology.orchestra.run.smithi046.stdout:(53/140): python3-logutils-0.3.5-11.el8.noarch. 194 kB/s | 49 kB 00:00 2024-01-26T23:26:47.054 INFO:teuthology.orchestra.run.smithi106.stdout:(133/140): util-linux-user-2.32.1-43.el8.x86_64 410 kB/s | 103 kB 00:00 2024-01-26T23:26:47.104 INFO:teuthology.orchestra.run.smithi106.stdout:(134/140): iproute-6.2.0-5.el8_9.x86_64.rpm 3.4 MB/s | 854 kB 00:00 2024-01-26T23:26:47.135 INFO:teuthology.orchestra.run.smithi046.stdout:(54/140): python3-more-itertools-7.2.0-3.el8.no 227 kB/s | 59 kB 00:00 2024-01-26T23:26:47.152 INFO:teuthology.orchestra.run.smithi046.stdout:(55/140): python3-natsort-7.1.1-2.el8.noarch.rp 233 kB/s | 60 kB 00:00 2024-01-26T23:26:47.261 INFO:teuthology.orchestra.run.smithi046.stdout:(56/140): python3-portend-2.6-1.el8.noarch.rpm 124 kB/s | 16 kB 00:00 2024-01-26T23:26:47.278 INFO:teuthology.orchestra.run.smithi046.stdout:(57/140): python3-repoze-lru-0.7-6.el8.noarch.r 268 kB/s | 34 kB 00:00 2024-01-26T23:26:47.303 INFO:teuthology.orchestra.run.smithi106.stdout:(135/140): xfsprogs-5.0.0-12.el8.x86_64.rpm 5.4 MB/s | 1.1 MB 00:00 2024-01-26T23:26:47.338 INFO:teuthology.orchestra.run.smithi106.stdout:(136/140): libblkid-2.32.1-43.el8.x86_64.rpm 587 kB/s | 220 kB 00:00 2024-01-26T23:26:47.386 INFO:teuthology.orchestra.run.smithi046.stdout:(58/140): python3-routes-2.4.1-12.el8.noarch.rp 1.5 MB/s | 196 kB 00:00 2024-01-26T23:26:47.405 INFO:teuthology.orchestra.run.smithi106.stdout:(137/140): libfdisk-2.32.1-43.el8.x86_64.rpm 723 kB/s | 254 kB 00:00 2024-01-26T23:26:47.534 INFO:teuthology.orchestra.run.smithi046.stdout:(59/140): python3-rsa-4.9-2.el8.noarch.rpm 251 kB/s | 64 kB 00:00 2024-01-26T23:26:47.547 INFO:teuthology.orchestra.run.smithi106.stdout:(138/140): openssl-1.1.1k-12.el8_9.x86_64.rpm 2.8 MB/s | 711 kB 00:00 2024-01-26T23:26:47.571 INFO:teuthology.orchestra.run.smithi046.stdout:(60/140): python3-pecan-1.3.2-9.el8.noarch.rpm 483 kB/s | 283 kB 00:00 2024-01-26T23:26:47.596 INFO:teuthology.orchestra.run.smithi046.stdout:(61/140): python3-simplegeneric-0.8.1-17.el8.no 90 kB/s | 19 kB 00:00 2024-01-26T23:26:47.638 INFO:teuthology.orchestra.run.smithi046.stdout:(62/140): python3-singledispatch-3.4.0.3-18.el8 232 kB/s | 24 kB 00:00 2024-01-26T23:26:47.638 INFO:teuthology.orchestra.run.smithi106.stdout:(139/140): openssl-libs-1.1.1k-12.el8_9.x86_64. 4.9 MB/s | 1.5 MB 00:00 2024-01-26T23:26:47.671 INFO:teuthology.orchestra.run.smithi046.stdout:(63/140): python3-tempora-1.14.1-5.el8.noarch.r 291 kB/s | 29 kB 00:00 2024-01-26T23:26:47.705 INFO:teuthology.orchestra.run.smithi046.stdout:(64/140): python3-trustme-0.6.0-4.el8.noarch.rp 253 kB/s | 27 kB 00:00 2024-01-26T23:26:47.764 INFO:teuthology.orchestra.run.smithi106.stdout:(140/140): xfsdump-3.1.8-7.el8_9.x86_64.rpm 929 kB/s | 333 kB 00:00 2024-01-26T23:26:47.769 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-26T23:26:47.769 INFO:teuthology.orchestra.run.smithi106.stdout:Total 6.7 MB/s | 212 MB 00:31 2024-01-26T23:26:47.813 INFO:teuthology.orchestra.run.smithi046.stdout:(65/140): python3-webob-1.8.5-1.el8.1.noarch.rp 1.7 MB/s | 251 kB 00:00 2024-01-26T23:26:47.839 INFO:teuthology.orchestra.run.smithi046.stdout:(66/140): python3-websocket-client-0.56.0-5.el8 458 kB/s | 61 kB 00:00 2024-01-26T23:26:47.914 INFO:teuthology.orchestra.run.smithi046.stdout:(67/140): python3-webtest-2.0.33-1.el8.noarch.r 846 kB/s | 85 kB 00:00 2024-01-26T23:26:47.940 INFO:teuthology.orchestra.run.smithi046.stdout:(68/140): python3-zc-lockfile-2.0-2.el8.noarch. 228 kB/s | 23 kB 00:00 2024-01-26T23:26:47.957 INFO:teuthology.orchestra.run.smithi046.stdout:(69/140): xmlstarlet-1.6.1-20.el8.x86_64.rpm 4.1 MB/s | 70 kB 00:00 2024-01-26T23:26:47.982 INFO:teuthology.orchestra.run.smithi046.stdout:(70/140): python3-waitress-1.4.3-1.el8.noarch.r 743 kB/s | 255 kB 00:00 2024-01-26T23:26:48.132 INFO:teuthology.orchestra.run.smithi046.stdout:(71/140): libverto-devel-0.3.0-5.el8.x86_64.rpm 102 kB/s | 18 kB 00:00 2024-01-26T23:26:48.158 INFO:teuthology.orchestra.run.smithi046.stdout:(72/140): python3-jwt-1.6.1-2.el8.noarch.rpm 243 kB/s | 43 kB 00:00 2024-01-26T23:26:48.199 INFO:teuthology.orchestra.run.smithi046.stdout:(73/140): thrift-0.13.0-2.el8.x86_64.rpm 6.1 MB/s | 1.7 MB 00:00 2024-01-26T23:26:48.216 INFO:teuthology.orchestra.run.smithi046.stdout:(74/140): python3-oauthlib-2.1.0-1.el8.noarch.r 1.8 MB/s | 155 kB 00:00 2024-01-26T23:26:48.283 INFO:teuthology.orchestra.run.smithi046.stdout:(75/140): python3-pycparser-2.14-14.el8.noarch. 870 kB/s | 109 kB 00:00 2024-01-26T23:26:48.367 INFO:teuthology.orchestra.run.smithi046.stdout:(76/140): python3-requests-oauthlib-1.0.0-1.el8 256 kB/s | 43 kB 00:00 2024-01-26T23:26:48.392 INFO:teuthology.orchestra.run.smithi046.stdout:(77/140): libaio-devel-0.3.112-1.el8.x86_64.rpm 112 kB/s | 19 kB 00:00 2024-01-26T23:26:48.418 INFO:teuthology.orchestra.run.smithi046.stdout:(78/140): libzstd-devel-1.4.4-1.el8.x86_64.rpm 327 kB/s | 44 kB 00:00 2024-01-26T23:26:48.501 INFO:teuthology.orchestra.run.smithi046.stdout:(79/140): pcre2-devel-10.32-2.el8.x86_64.rpm 4.4 MB/s | 605 kB 00:00 2024-01-26T23:26:48.526 INFO:teuthology.orchestra.run.smithi046.stdout:(80/140): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.7 MB/s | 229 kB 00:00 2024-01-26T23:26:48.568 INFO:teuthology.orchestra.run.smithi046.stdout:(81/140): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.4 MB/s | 220 kB 00:00 2024-01-26T23:26:48.627 INFO:teuthology.orchestra.run.smithi046.stdout:(82/140): libselinux-devel-2.9-5.el8.x86_64.rpm 1.6 MB/s | 200 kB 00:00 2024-01-26T23:26:48.686 INFO:teuthology.orchestra.run.smithi046.stdout:(83/140): keyutils-libs-devel-1.5.10-9.el8.x86_ 303 kB/s | 48 kB 00:00 2024-01-26T23:26:48.711 INFO:teuthology.orchestra.run.smithi046.stdout:(84/140): libsepol-devel-2.9-3.el8.x86_64.rpm 612 kB/s | 87 kB 00:00 2024-01-26T23:26:48.726 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-26T23:26:48.786 INFO:teuthology.orchestra.run.smithi046.stdout:(85/140): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 363 kB/s | 58 kB 00:00 2024-01-26T23:26:48.820 INFO:teuthology.orchestra.run.smithi046.stdout:(86/140): libcom_err-devel-1.45.6-4.el8.x86_64. 288 kB/s | 39 kB 00:00 2024-01-26T23:26:48.895 INFO:teuthology.orchestra.run.smithi046.stdout:(87/140): fuse-2.9.7-17.el8.x86_64.rpm 452 kB/s | 83 kB 00:00 2024-01-26T23:26:48.937 INFO:teuthology.orchestra.run.smithi046.stdout:(88/140): libgfortran-8.5.0-20.el8.x86_64.rpm 4.2 MB/s | 645 kB 00:00 2024-01-26T23:26:48.962 INFO:teuthology.orchestra.run.smithi046.stdout:(89/140): libquadmath-8.5.0-20.el8.x86_64.rpm 1.2 MB/s | 172 kB 00:00 2024-01-26T23:26:48.996 INFO:teuthology.orchestra.run.smithi046.stdout:(90/140): libuuid-devel-2.32.1-43.el8.x86_64.rp 993 kB/s | 100 kB 00:00 2024-01-26T23:26:49.054 INFO:teuthology.orchestra.run.smithi046.stdout:(91/140): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.0 MB/s | 238 kB 00:00 2024-01-26T23:26:49.071 INFO:teuthology.orchestra.run.smithi046.stdout:(92/140): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 967 kB/s | 73 kB 00:00 2024-01-26T23:26:49.118 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-26T23:26:49.118 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-26T23:26:49.125 INFO:teuthology.orchestra.run.smithi046.stdout:(93/140): elfutils-libelf-devel-0.189-3.el8.x86 879 kB/s | 62 kB 00:00 2024-01-26T23:26:49.155 INFO:teuthology.orchestra.run.smithi046.stdout:(94/140): librabbitmq-0.9.0-4.el8.x86_64.rpm 562 kB/s | 47 kB 00:00 2024-01-26T23:26:49.197 INFO:teuthology.orchestra.run.smithi046.stdout:(95/140): python3-cryptography-3.2.1-6.el8.x86_ 2.3 MB/s | 559 kB 00:00 2024-01-26T23:26:49.264 INFO:teuthology.orchestra.run.smithi046.stdout:(96/140): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 4.0 MB/s | 562 kB 00:00 2024-01-26T23:26:49.340 INFO:teuthology.orchestra.run.smithi046.stdout:(97/140): openssl-devel-1.1.1k-12.el8_9.x86_64. 13 MB/s | 2.3 MB 00:00 2024-01-26T23:26:49.396 INFO:teuthology.orchestra.run.smithi046.stdout:(98/140): NetworkManager-initscripts-updown-1.4 727 kB/s | 144 kB 00:00 2024-01-26T23:26:49.415 INFO:teuthology.orchestra.run.smithi046.stdout:(99/140): python3-jsonpointer-1.10-11.el8.noarc 133 kB/s | 20 kB 00:00 2024-01-26T23:26:49.449 INFO:teuthology.orchestra.run.smithi046.stdout:(100/140): python3-prettytable-0.7.2-14.el8.noa 403 kB/s | 44 kB 00:00 2024-01-26T23:26:49.507 INFO:teuthology.orchestra.run.smithi046.stdout:(101/140): python3-pyasn1-modules-0.3.7-6.el8.n 988 kB/s | 110 kB 00:00 2024-01-26T23:26:49.566 INFO:teuthology.orchestra.run.smithi046.stdout:(102/140): python3-jsonpatch-1.21-2.el8.noarch. 176 kB/s | 27 kB 00:00 2024-01-26T23:26:49.600 INFO:teuthology.orchestra.run.smithi046.stdout:(103/140): python3-pyasn1-0.3.7-6.el8.noarch.rp 840 kB/s | 126 kB 00:00 2024-01-26T23:26:49.657 INFO:teuthology.orchestra.run.smithi046.stdout:(104/140): python3-werkzeug-0.12.2-4.el8.noarch 3.0 MB/s | 457 kB 00:00 2024-01-26T23:26:49.709 INFO:teuthology.orchestra.run.smithi046.stdout:(105/140): bison-3.0.4-10.el8.x86_64.rpm 4.7 MB/s | 688 kB 00:00 2024-01-26T23:26:49.742 INFO:teuthology.orchestra.run.smithi046.stdout:(106/140): flex-2.6.1-9.el8.x86_64.rpm 2.2 MB/s | 320 kB 00:00 2024-01-26T23:26:49.776 INFO:teuthology.orchestra.run.smithi046.stdout:(107/140): libtool-2.4.6-25.el8.x86_64.rpm 5.9 MB/s | 709 kB 00:00 2024-01-26T23:26:49.834 INFO:teuthology.orchestra.run.smithi046.stdout:(108/140): python3-markupsafe-0.23-19.el8.x86_6 310 kB/s | 39 kB 00:00 2024-01-26T23:26:49.885 INFO:teuthology.orchestra.run.smithi046.stdout:(109/140): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.8 MB/s | 259 kB 00:00 2024-01-26T23:26:49.952 INFO:teuthology.orchestra.run.smithi046.stdout:(110/140): oniguruma-6.8.2-2.el8.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-01-26T23:26:49.977 INFO:teuthology.orchestra.run.smithi046.stdout:(111/140): python3-pyOpenSSL-19.0.0-1.el8.noarc 728 kB/s | 103 kB 00:00 2024-01-26T23:26:50.102 INFO:teuthology.orchestra.run.smithi046.stdout:(112/140): python3-babel-2.5.1-7.el8.noarch.rpm 22 MB/s | 4.8 MB 00:00 2024-01-26T23:26:50.136 INFO:teuthology.orchestra.run.smithi046.stdout:(113/140): python3-jinja2-2.10.1-3.el8.noarch.r 2.9 MB/s | 538 kB 00:00 2024-01-26T23:26:50.236 INFO:teuthology.orchestra.run.smithi046.stdout:(114/140): python3-numpy-1.14.3-10.el8.x86_64.r 14 MB/s | 3.7 MB 00:00 2024-01-26T23:26:50.261 INFO:teuthology.orchestra.run.smithi046.stdout:(115/140): socat-1.7.4.1-1.el8.x86_64.rpm 2.0 MB/s | 323 kB 00:00 2024-01-26T23:26:50.290 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-26T23:26:50.300 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-26T23:26:50.311 INFO:teuthology.orchestra.run.smithi046.stdout:(116/140): python3-numpy-f2py-1.14.3-10.el8.x86 1.3 MB/s | 225 kB 00:00 2024-01-26T23:26:50.345 INFO:teuthology.orchestra.run.smithi046.stdout:(117/140): librdkafka-0.11.4-3.el8.x86_64.rpm 3.2 MB/s | 354 kB 00:00 2024-01-26T23:26:50.412 INFO:teuthology.orchestra.run.smithi046.stdout:(118/140): python3-mako-1.0.6-14.el8.noarch.rpm 1.5 MB/s | 157 kB 00:00 2024-01-26T23:26:50.546 INFO:teuthology.orchestra.run.smithi046.stdout:(119/140): openblas-threads-0.3.15-6.el8.x86_64 18 MB/s | 5.2 MB 00:00 2024-01-26T23:26:50.588 INFO:teuthology.orchestra.run.smithi046.stdout:(120/140): automake-1.16.1-8.el8.noarch.rpm 4.0 MB/s | 714 kB 00:00 2024-01-26T23:26:50.679 INFO:teuthology.orchestra.run.smithi046.stdout:(121/140): jq-1.6-7.el8.x86_64.rpm 1.5 MB/s | 202 kB 00:00 2024-01-26T23:26:50.730 INFO:teuthology.orchestra.run.smithi046.stdout:(122/140): python3-pytz-2017.2-11.el8.noarch.rp 382 kB/s | 54 kB 00:00 2024-01-26T23:26:50.839 INFO:teuthology.orchestra.run.smithi046.stdout:(123/140): openblas-0.3.15-6.el8.x86_64.rpm 10 MB/s | 5.0 MB 00:00 2024-01-26T23:26:51.114 INFO:teuthology.orchestra.run.smithi046.stdout:(124/140): python3-scipy-1.0.0-21.module+el8.9. 32 MB/s | 14 MB 00:00 2024-01-26T23:26:51.252 INFO:teuthology.orchestra.run.smithi046.stdout:(125/140): elfutils-libelf-0.189-3.el8.x86_64.r 1.6 MB/s | 232 kB 00:00 2024-01-26T23:26:51.414 INFO:teuthology.orchestra.run.smithi046.stdout:(126/140): elfutils-libs-0.189-3.el8.x86_64.rpm 1.8 MB/s | 303 kB 00:00 2024-01-26T23:26:51.556 INFO:teuthology.orchestra.run.smithi046.stdout:(127/140): libmount-2.32.1-43.el8.x86_64.rpm 1.6 MB/s | 237 kB 00:00 2024-01-26T23:26:51.673 INFO:teuthology.orchestra.run.smithi046.stdout:(128/140): libsmartcols-2.32.1-43.el8.x86_64.rp 1.5 MB/s | 179 kB 00:00 2024-01-26T23:26:51.799 INFO:teuthology.orchestra.run.smithi046.stdout:(129/140): libuuid-2.32.1-43.el8.x86_64.rpm 793 kB/s | 99 kB 00:00 2024-01-26T23:26:51.982 INFO:teuthology.orchestra.run.smithi046.stdout:(130/140): util-linux-2.32.1-43.el8.x86_64.rpm 14 MB/s | 2.5 MB 00:00 2024-01-26T23:26:52.133 INFO:teuthology.orchestra.run.smithi046.stdout:(131/140): util-linux-user-2.32.1-43.el8.x86_64 684 kB/s | 103 kB 00:00 2024-01-26T23:26:52.216 INFO:teuthology.orchestra.run.smithi046.stdout:(132/140): librbd1-17.2.7-472.g74950076.el8.x86 2.6 MB/s | 3.6 MB 00:01 2024-01-26T23:26:52.266 INFO:teuthology.orchestra.run.smithi046.stdout:(133/140): iproute-6.2.0-5.el8_9.x86_64.rpm 6.3 MB/s | 854 kB 00:00 2024-01-26T23:26:52.334 INFO:teuthology.orchestra.run.smithi046.stdout:(134/140): libblkid-2.32.1-43.el8.x86_64.rpm 1.8 MB/s | 220 kB 00:00 2024-01-26T23:26:52.359 INFO:teuthology.orchestra.run.smithi046.stdout:(135/140): libfdisk-2.32.1-43.el8.x86_64.rpm 2.7 MB/s | 254 kB 00:00 2024-01-26T23:26:52.468 INFO:teuthology.orchestra.run.smithi046.stdout:(136/140): openssl-1.1.1k-12.el8_9.x86_64.rpm 6.4 MB/s | 711 kB 00:00 2024-01-26T23:26:52.509 INFO:teuthology.orchestra.run.smithi046.stdout:(137/140): xfsprogs-5.0.0-12.el8.x86_64.rpm 6.1 MB/s | 1.1 MB 00:00 2024-01-26T23:26:52.601 INFO:teuthology.orchestra.run.smithi046.stdout:(138/140): openssl-libs-1.1.1k-12.el8_9.x86_64. 11 MB/s | 1.5 MB 00:00 2024-01-26T23:26:52.635 INFO:teuthology.orchestra.run.smithi046.stdout:(139/140): xfsdump-3.1.8-7.el8_9.x86_64.rpm 2.6 MB/s | 333 kB 00:00 2024-01-26T23:26:54.202 INFO:teuthology.orchestra.run.smithi046.stdout:(140/140): librados2-17.2.7-472.g74950076.el8.x 1.0 MB/s | 3.5 MB 00:03 2024-01-26T23:26:54.207 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-26T23:26:54.207 INFO:teuthology.orchestra.run.smithi046.stdout:Total 6.4 MB/s | 212 MB 00:33 2024-01-26T23:26:55.238 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-26T23:26:55.686 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-26T23:26:55.686 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-26T23:26:55.911 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-26T23:26:56.046 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-01-26T23:26:56.329 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/156 2024-01-26T23:26:56.398 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/156 2024-01-26T23:26:56.556 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 2/156 2024-01-26T23:26:56.614 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 2/156 2024-01-26T23:26:56.789 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 3/156 2024-01-26T23:26:56.875 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 3/156 2024-01-26T23:26:56.963 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-26T23:26:56.975 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-26T23:26:57.183 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/156 2024-01-26T23:26:57.436 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : thrift-0.13.0-2.el8.x86_64 5/156 2024-01-26T23:26:57.622 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/156 2024-01-26T23:26:57.717 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/156 2024-01-26T23:26:57.931 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/156 2024-01-26T23:26:58.138 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/156 2024-01-26T23:26:58.551 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/156 2024-01-26T23:26:58.797 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : librados2-2:17.2.7-472.g74950076.el8.x86_64 9/156 2024-01-26T23:26:59.192 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados2-2:17.2.7-472.g74950076.el8.x86_64 9/156 2024-01-26T23:26:59.405 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : librbd1-2:17.2.7-472.g74950076.el8.x86_64 10/156 2024-01-26T23:26:59.535 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librbd1-2:17.2.7-472.g74950076.el8.x86_64 10/156 2024-01-26T23:26:59.739 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephfs2-2:17.2.7-472.g74950076.el8.x86_64 11/156 2024-01-26T23:26:59.837 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephfs2-2:17.2.7-472.g74950076.el8.x86_64 11/156 2024-01-26T23:27:00.509 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rados-2:17.2.7-472.g74950076.el8.x86_64 12/156 2024-01-26T23:27:00.655 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librgw2-2:17.2.7-472.g74950076.el8.x86_64 13/156 2024-01-26T23:27:00.839 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librgw2-2:17.2.7-472.g74950076.el8.x86_64 13/156 2024-01-26T23:27:01.121 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 14/156 2024-01-26T23:27:01.212 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 14/156 2024-01-26T23:27:01.451 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/156 2024-01-26T23:27:01.667 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/156 2024-01-26T23:27:01.837 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/156 2024-01-26T23:27:02.039 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephsqlite-2:17.2.7-472.g74950076.el8.x86_64 18/156 2024-01-26T23:27:02.147 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephsqlite-2:17.2.7-472.g74950076.el8.x86_64 18/156 2024-01-26T23:27:02.305 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libradosstriper1-2:17.2.7-472.g74950076.el8.x86_ 19/156 2024-01-26T23:27:02.381 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libradosstriper1-2:17.2.7-472.g74950076.el8.x86_ 19/156 2024-01-26T23:27:02.563 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 20/156 2024-01-26T23:27:02.694 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 20/156 2024-01-26T23:27:02.736 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-26T23:27:02.885 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-01-26T23:27:02.888 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 21/156 2024-01-26T23:27:03.154 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/156 2024-01-26T23:27:03.259 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/156 2024-01-26T23:27:03.415 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 2/156 2024-01-26T23:27:03.496 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 2/156 2024-01-26T23:27:03.519 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 21/156 2024-01-26T23:27:03.715 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 3/156 2024-01-26T23:27:03.721 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 22/156 2024-01-26T23:27:03.832 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 3/156 2024-01-26T23:27:04.159 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/156 2024-01-26T23:27:04.314 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 22/156 2024-01-26T23:27:04.362 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : thrift-0.13.0-2.el8.x86_64 5/156 2024-01-26T23:27:04.530 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openblas-0.3.15-6.el8.x86_64 23/156 2024-01-26T23:27:04.575 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 6/156 2024-01-26T23:27:04.764 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/156 2024-01-26T23:27:04.899 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/156 2024-01-26T23:27:05.089 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/156 2024-01-26T23:27:05.212 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 23/156 2024-01-26T23:27:05.533 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/156 2024-01-26T23:27:05.552 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 24/156 2024-01-26T23:27:05.718 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 25/156 2024-01-26T23:27:05.764 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librados2-2:17.2.7-472.g74950076.el8.x86_64 9/156 2024-01-26T23:27:05.925 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 26/156 2024-01-26T23:27:06.203 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:17.2.7-472.g74950076.el8.x86_64 9/156 2024-01-26T23:27:06.226 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 27/156 2024-01-26T23:27:06.372 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librbd1-2:17.2.7-472.g74950076.el8.x86_64 10/156 2024-01-26T23:27:06.536 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 28/156 2024-01-26T23:27:06.537 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:17.2.7-472.g74950076.el8.x86_64 10/156 2024-01-26T23:27:06.689 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs2-2:17.2.7-472.g74950076.el8.x86_64 11/156 2024-01-26T23:27:06.815 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 29/156 2024-01-26T23:27:06.831 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs2-2:17.2.7-472.g74950076.el8.x86_64 11/156 2024-01-26T23:27:07.042 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ceph-common-2:17.2.7-472.g74950076.el8.x 30/156 2024-01-26T23:27:07.181 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ceph-argparse-2:17.2.7-472.g74950076.el8 31/156 2024-01-26T23:27:07.451 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cephfs-2:17.2.7-472.g74950076.el8.x86_64 32/156 2024-01-26T23:27:07.565 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rados-2:17.2.7-472.g74950076.el8.x86_64 12/156 2024-01-26T23:27:07.839 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librgw2-2:17.2.7-472.g74950076.el8.x86_64 13/156 2024-01-26T23:27:08.035 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librgw2-2:17.2.7-472.g74950076.el8.x86_64 13/156 2024-01-26T23:27:08.289 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 14/156 2024-01-26T23:27:08.388 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 14/156 2024-01-26T23:27:08.419 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 33/156 2024-01-26T23:27:08.645 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/156 2024-01-26T23:27:08.836 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/156 2024-01-26T23:27:09.015 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/156 2024-01-26T23:27:09.052 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 34/156 2024-01-26T23:27:09.194 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephsqlite-2:17.2.7-472.g74950076.el8.x86_64 18/156 2024-01-26T23:27:09.306 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephsqlite-2:17.2.7-472.g74950076.el8.x86_64 18/156 2024-01-26T23:27:09.324 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 35/156 2024-01-26T23:27:09.456 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libradosstriper1-2:17.2.7-472.g74950076.el8.x86_ 19/156 2024-01-26T23:27:09.540 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libradosstriper1-2:17.2.7-472.g74950076.el8.x86_ 19/156 2024-01-26T23:27:09.544 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 36/156 2024-01-26T23:27:09.720 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/156 2024-01-26T23:27:09.995 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 21/156 2024-01-26T23:27:10.254 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/156 2024-01-26T23:27:10.397 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 23/156 2024-01-26T23:27:10.581 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 23/156 2024-01-26T23:27:10.705 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 24/156 2024-01-26T23:27:11.273 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 36/156 2024-01-26T23:27:11.425 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 24/156 2024-01-26T23:27:11.622 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 25/156 2024-01-26T23:27:11.638 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 37/156 2024-01-26T23:27:11.907 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rsa-4.9-2.el8.noarch 38/156 2024-01-26T23:27:12.115 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 39/156 2024-01-26T23:27:12.287 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 25/156 2024-01-26T23:27:12.405 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rgw-2:17.2.7-472.g74950076.el8.x86_64 40/156 2024-01-26T23:27:12.522 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-0.3.15-6.el8.x86_64 26/156 2024-01-26T23:27:12.625 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rbd-2:17.2.7-472.g74950076.el8.x86_64 41/156 2024-01-26T23:27:12.818 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librados-devel-2:17.2.7-472.g74950076.el8.x86_64 42/156 2024-01-26T23:27:13.045 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/156 2024-01-26T23:27:13.117 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/156 2024-01-26T23:27:13.291 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 26/156 2024-01-26T23:27:13.354 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/156 2024-01-26T23:27:13.409 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/156 2024-01-26T23:27:13.588 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 27/156 2024-01-26T23:27:13.666 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/156 2024-01-26T23:27:13.822 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 28/156 2024-01-26T23:27:13.947 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/156 2024-01-26T23:27:14.132 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 29/156 2024-01-26T23:27:14.229 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/156 2024-01-26T23:27:14.365 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-common-2:17.2.7-472.g74950076.el8.x 30/156 2024-01-26T23:27:14.518 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-argparse-2:17.2.7-472.g74950076.el8 31/156 2024-01-26T23:27:14.706 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/156 2024-01-26T23:27:14.813 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cephfs-2:17.2.7-472.g74950076.el8.x86_64 32/156 2024-01-26T23:27:15.043 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 33/156 2024-01-26T23:27:15.045 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/156 2024-01-26T23:27:15.062 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/156 2024-01-26T23:27:15.307 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : fuse-2.9.7-17.el8.x86_64 49/156 2024-01-26T23:27:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 50/156 2024-01-26T23:27:15.699 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 51/156 2024-01-26T23:27:15.906 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 52/156 2024-01-26T23:27:16.140 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 53/156 2024-01-26T23:27:16.461 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 54/156 2024-01-26T23:27:16.781 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 55/156 2024-01-26T23:27:17.023 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 33/156 2024-01-26T23:27:17.033 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 56/156 2024-01-26T23:27:17.191 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 57/156 2024-01-26T23:27:17.403 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 34/156 2024-01-26T23:27:17.413 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 58/156 2024-01-26T23:27:17.786 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 59/156 2024-01-26T23:27:18.105 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 60/156 2024-01-26T23:27:18.321 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 61/156 2024-01-26T23:27:18.471 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 35/156 2024-01-26T23:27:18.614 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 62/156 2024-01-26T23:27:18.820 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 36/156 2024-01-26T23:27:18.848 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 63/156 2024-01-26T23:27:19.179 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 37/156 2024-01-26T23:27:19.192 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 64/156 2024-01-26T23:27:19.427 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 65/156 2024-01-26T23:27:19.433 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rsa-4.9-2.el8.noarch 38/156 2024-01-26T23:27:19.593 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 39/156 2024-01-26T23:27:19.620 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 66/156 2024-01-26T23:27:19.863 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 67/156 2024-01-26T23:27:19.869 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rgw-2:17.2.7-472.g74950076.el8.x86_64 40/156 2024-01-26T23:27:20.085 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 68/156 2024-01-26T23:27:20.103 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rbd-2:17.2.7-472.g74950076.el8.x86_64 41/156 2024-01-26T23:27:20.338 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librados-devel-2:17.2.7-472.g74950076.el8.x86_64 42/156 2024-01-26T23:27:20.425 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 69/156 2024-01-26T23:27:20.563 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/156 2024-01-26T23:27:20.649 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/156 2024-01-26T23:27:20.689 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : automake-1.16.1-8.el8.noarch 70/156 2024-01-26T23:27:20.844 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 71/156 2024-01-26T23:27:20.887 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/156 2024-01-26T23:27:20.944 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 71/156 2024-01-26T23:27:20.958 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/156 2024-01-26T23:27:21.177 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : jq-1.6-7.el8.x86_64 72/156 2024-01-26T23:27:21.197 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/156 2024-01-26T23:27:21.334 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 73/156 2024-01-26T23:27:21.461 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/156 2024-01-26T23:27:21.552 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 74/156 2024-01-26T23:27:21.700 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/156 2024-01-26T23:27:21.771 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 75/156 2024-01-26T23:27:21.936 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 76/156 2024-01-26T23:27:22.167 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 77/156 2024-01-26T23:27:22.169 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/156 2024-01-26T23:27:22.449 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 78/156 2024-01-26T23:27:22.496 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/156 2024-01-26T23:27:22.606 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/156 2024-01-26T23:27:22.685 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 79/156 2024-01-26T23:27:22.827 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : automake-1.16.1-8.el8.noarch 49/156 2024-01-26T23:27:22.924 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 80/156 2024-01-26T23:27:23.020 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 50/156 2024-01-26T23:27:23.105 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 81/156 2024-01-26T23:27:23.107 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 50/156 2024-01-26T23:27:23.330 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : jq-1.6-7.el8.x86_64 51/156 2024-01-26T23:27:23.353 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 82/156 2024-01-26T23:27:23.529 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 52/156 2024-01-26T23:27:23.614 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 83/156 2024-01-26T23:27:23.763 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 53/156 2024-01-26T23:27:23.846 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 84/156 2024-01-26T23:27:23.940 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 54/156 2024-01-26T23:27:24.064 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 85/156 2024-01-26T23:27:24.176 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : fuse-2.9.7-17.el8.x86_64 55/156 2024-01-26T23:27:24.241 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 86/156 2024-01-26T23:27:24.414 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 56/156 2024-01-26T23:27:24.529 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 87/156 2024-01-26T23:27:24.611 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 57/156 2024-01-26T23:27:24.790 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 88/156 2024-01-26T23:27:24.834 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 58/156 2024-01-26T23:27:24.983 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 89/156 2024-01-26T23:27:25.035 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 59/156 2024-01-26T23:27:25.253 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-portend-2.6-1.el8.noarch 90/156 2024-01-26T23:27:25.378 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 60/156 2024-01-26T23:27:25.509 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 91/156 2024-01-26T23:27:25.734 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 92/156 2024-01-26T23:27:25.834 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 61/156 2024-01-26T23:27:25.951 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 93/156 2024-01-26T23:27:26.076 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 62/156 2024-01-26T23:27:26.289 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 63/156 2024-01-26T23:27:26.522 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 64/156 2024-01-26T23:27:27.027 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 94/156 2024-01-26T23:27:27.071 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 65/156 2024-01-26T23:27:27.439 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 95/156 2024-01-26T23:27:27.477 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 66/156 2024-01-26T23:27:27.684 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 96/156 2024-01-26T23:27:27.710 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 67/156 2024-01-26T23:27:27.979 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 68/156 2024-01-26T23:27:28.011 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 97/156 2024-01-26T23:27:28.288 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 69/156 2024-01-26T23:27:28.344 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 98/156 2024-01-26T23:27:28.584 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 99/156 2024-01-26T23:27:28.655 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 70/156 2024-01-26T23:27:28.810 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 100/156 2024-01-26T23:27:29.005 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 71/156 2024-01-26T23:27:29.164 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 72/156 2024-01-26T23:27:29.364 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 73/156 2024-01-26T23:27:29.610 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 74/156 2024-01-26T23:27:29.873 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 75/156 2024-01-26T23:27:30.071 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 76/156 2024-01-26T23:27:30.267 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 77/156 2024-01-26T23:27:30.542 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 78/156 2024-01-26T23:27:30.753 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 79/156 2024-01-26T23:27:30.966 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 80/156 2024-01-26T23:27:31.182 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 81/156 2024-01-26T23:27:31.424 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 82/156 2024-01-26T23:27:31.660 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 83/156 2024-01-26T23:27:31.866 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 84/156 2024-01-26T23:27:32.056 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 85/156 2024-01-26T23:27:32.259 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 86/156 2024-01-26T23:27:32.504 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 87/156 2024-01-26T23:27:32.764 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 88/156 2024-01-26T23:27:32.943 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 89/156 2024-01-26T23:27:32.992 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-common-2:17.2.7-472.g74950076.el8.x86_64 101/156 2024-01-26T23:27:33.252 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-common-2:17.2.7-472.g74950076.el8.x86_64 101/156 2024-01-26T23:27:33.257 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-portend-2.6-1.el8.noarch 90/156 2024-01-26T23:27:33.470 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 91/156 2024-01-26T23:27:33.695 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 92/156 2024-01-26T23:27:33.812 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-common-2:17.2.7-472.g74950076.el8.x86_64 101/156 2024-01-26T23:27:33.940 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 93/156 2024-01-26T23:27:34.142 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-base-2:17.2.7-472.g74950076.el8.x86_64 102/156 2024-01-26T23:27:34.231 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-base-2:17.2.7-472.g74950076.el8.x86_64 102/156 2024-01-26T23:27:34.418 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-selinux-2:17.2.7-472.g74950076.el8.x86_64 103/156 2024-01-26T23:27:35.144 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 94/156 2024-01-26T23:27:35.439 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 95/156 2024-01-26T23:27:35.666 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 96/156 2024-01-26T23:27:36.050 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 97/156 2024-01-26T23:27:36.355 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 98/156 2024-01-26T23:27:36.565 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 99/156 2024-01-26T23:27:36.728 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 100/156 2024-01-26T23:27:42.599 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:17.2.7-472.g74950076.el8.x86_64 101/156 2024-01-26T23:27:42.861 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-common-2:17.2.7-472.g74950076.el8.x86_64 101/156 2024-01-26T23:27:43.485 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:17.2.7-472.g74950076.el8.x86_64 101/156 2024-01-26T23:27:43.791 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-base-2:17.2.7-472.g74950076.el8.x86_64 102/156 2024-01-26T23:27:43.871 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-base-2:17.2.7-472.g74950076.el8.x86_64 102/156 2024-01-26T23:27:44.068 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-selinux-2:17.2.7-472.g74950076.el8.x86_64 103/156 2024-01-26T23:27:53.609 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-selinux-2:17.2.7-472.g74950076.el8.x86_64 103/156 2024-01-26T23:27:53.609 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /sys 2024-01-26T23:27:53.609 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /proc 2024-01-26T23:27:53.609 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /dev 2024-01-26T23:27:53.609 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /run 2024-01-26T23:27:53.609 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /mnt 2024-01-26T23:27:53.610 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /var/tmp 2024-01-26T23:27:53.610 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /home 2024-01-26T23:27:53.610 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /tmp 2024-01-26T23:27:53.610 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /dev 2024-01-26T23:27:53.610 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:27:54.032 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-volume-2:17.2.7-472.g74950076.el8.noarch 104/156 2024-01-26T23:27:55.505 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-volume-2:17.2.7-472.g74950076.el8.noarch 104/156 2024-01-26T23:27:55.780 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-osd-2:17.2.7-472.g74950076.el8.x86_64 105/156 2024-01-26T23:27:56.003 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-osd-2:17.2.7-472.g74950076.el8.x86_64 105/156 2024-01-26T23:27:56.190 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mds-2:17.2.7-472.g74950076.el8.x86_64 106/156 2024-01-26T23:27:56.579 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mds-2:17.2.7-472.g74950076.el8.x86_64 106/156 2024-01-26T23:27:56.748 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mon-2:17.2.7-472.g74950076.el8.x86_64 107/156 2024-01-26T23:27:56.765 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mon-2:17.2.7-472.g74950076.el8.x86_64 107/156 2024-01-26T23:27:58.608 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: cephadm-2:17.2.7-472.g74950076.el8.noarch 108/156 2024-01-26T23:27:58.774 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephadm-2:17.2.7-472.g74950076.el8.noarch 108/156 2024-01-26T23:27:58.920 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-472.g74950076.el 109/156 2024-01-26T23:27:59.158 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-472.g74950076.e 110/156 2024-01-26T23:27:59.431 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-472.g74950076.el8.noar 111/156 2024-01-26T23:27:59.646 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-472.g74950076.el8.noar 111/156 2024-01-26T23:27:59.931 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-472.g74950076.el8.no 112/156 2024-01-26T23:28:00.898 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-472.g74950076.el8.no 112/156 2024-01-26T23:28:01.105 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-472.g7495 113/156 2024-01-26T23:28:01.198 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-472.g7495 113/156 2024-01-26T23:28:01.584 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-472.g74950076.el8 114/156 2024-01-26T23:28:01.796 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-2:17.2.7-472.g74950076.el8.x86_64 115/156 2024-01-26T23:28:01.833 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-2:17.2.7-472.g74950076.el8.x86_64 115/156 2024-01-26T23:28:02.005 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-rook-2:17.2.7-472.g74950076.el8.noarch 116/156 2024-01-26T23:28:02.017 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-472.g74950076.el8.noarch 116/156 2024-01-26T23:28:03.593 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-2:17.2.7-472.g74950076.el8.x86_64 117/156 2024-01-26T23:28:03.796 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-radosgw-2:17.2.7-472.g74950076.el8.x86_64 118/156 2024-01-26T23:28:03.921 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-472.g74950076.el8.x86_64 118/156 2024-01-26T23:28:04.089 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-472.g749500 119/156 2024-01-26T23:28:04.132 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-472.g749500 119/156 2024-01-26T23:28:04.298 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephfs-mirror-2:17.2.7-472.g74950076.el8.x86_64 120/156 2024-01-26T23:28:04.646 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: cephfs-mirror-2:17.2.7-472.g74950076.el8.x86_64 120/156 2024-01-26T23:28:04.855 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-mirror-2:17.2.7-472.g74950076.el8.x86_64 121/156 2024-01-26T23:28:06.409 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-selinux-2:17.2.7-472.g74950076.el8.x86_64 103/156 2024-01-26T23:28:06.409 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /sys 2024-01-26T23:28:06.410 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /proc 2024-01-26T23:28:06.410 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /dev 2024-01-26T23:28:06.410 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /run 2024-01-26T23:28:06.410 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /mnt 2024-01-26T23:28:06.410 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /var/tmp 2024-01-26T23:28:06.410 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /home 2024-01-26T23:28:06.410 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /tmp 2024-01-26T23:28:06.410 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /dev 2024-01-26T23:28:06.410 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:06.791 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-volume-2:17.2.7-472.g74950076.el8.noarch 104/156 2024-01-26T23:28:08.424 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-volume-2:17.2.7-472.g74950076.el8.noarch 104/156 2024-01-26T23:28:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-osd-2:17.2.7-472.g74950076.el8.x86_64 105/156 2024-01-26T23:28:08.868 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-osd-2:17.2.7-472.g74950076.el8.x86_64 105/156 2024-01-26T23:28:09.074 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mds-2:17.2.7-472.g74950076.el8.x86_64 106/156 2024-01-26T23:28:09.507 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mds-2:17.2.7-472.g74950076.el8.x86_64 106/156 2024-01-26T23:28:09.657 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mon-2:17.2.7-472.g74950076.el8.x86_64 107/156 2024-01-26T23:28:09.679 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mon-2:17.2.7-472.g74950076.el8.x86_64 107/156 2024-01-26T23:28:09.746 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-mirror-2:17.2.7-472.g74950076.el8.x86_64 121/156 2024-01-26T23:28:10.132 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-test-2:17.2.7-472.g74950076.el8.x86_64 122/156 2024-01-26T23:28:10.279 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libtool-2.4.6-25.el8.x86_64 123/156 2024-01-26T23:28:10.962 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 123/156 2024-01-26T23:28:11.284 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 124/156 2024-01-26T23:28:11.572 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 125/156 2024-01-26T23:28:11.666 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:17.2.7-472.g74950076.el8.noarch 108/156 2024-01-26T23:28:11.793 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-fuse-2:17.2.7-472.g74950076.el8.x86_64 126/156 2024-01-26T23:28:11.841 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephadm-2:17.2.7-472.g74950076.el8.noarch 108/156 2024-01-26T23:28:12.006 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-472.g74950076.el 109/156 2024-01-26T23:28:12.057 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 127/156 2024-01-26T23:28:12.212 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-472.g74950076.e 110/156 2024-01-26T23:28:12.244 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 128/156 2024-01-26T23:28:12.451 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-472.g74950076.el8.noar 111/156 2024-01-26T23:28:12.460 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephfs-devel-2:17.2.7-472.g74950076.el8.x86_6 129/156 2024-01-26T23:28:12.681 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-472.g74950076.el8.noar 111/156 2024-01-26T23:28:12.837 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 130/156 2024-01-26T23:28:12.920 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-472.g74950076.el8.no 112/156 2024-01-26T23:28:13.228 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 131/156 2024-01-26T23:28:13.450 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 132/156 2024-01-26T23:28:13.628 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephfs-top-2:17.2.7-472.g74950076.el8.noarch 133/156 2024-01-26T23:28:13.870 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-fuse-2:17.2.7-472.g74950076.el8.x86_64 134/156 2024-01-26T23:28:13.995 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-472.g74950076.el8.no 112/156 2024-01-26T23:28:14.189 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-472.g7495 113/156 2024-01-26T23:28:14.189 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-nbd-2:17.2.7-472.g74950076.el8.x86_64 135/156 2024-01-26T23:28:14.302 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-472.g7495 113/156 2024-01-26T23:28:14.472 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 136/156 2024-01-26T23:28:14.687 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 137/156 2024-01-26T23:28:14.691 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-472.g74950076.el8 114/156 2024-01-26T23:28:14.696 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 137/156 2024-01-26T23:28:14.837 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-2:17.2.7-472.g74950076.el8.x86_64 115/156 2024-01-26T23:28:14.881 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-2:17.2.7-472.g74950076.el8.x86_64 115/156 2024-01-26T23:28:14.898 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 138/156 2024-01-26T23:28:15.031 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-rook-2:17.2.7-472.g74950076.el8.noarch 116/156 2024-01-26T23:28:15.046 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-472.g74950076.el8.noarch 116/156 2024-01-26T23:28:15.062 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : flex-2.6.1-9.el8.x86_64 139/156 2024-01-26T23:28:15.137 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 139/156 2024-01-26T23:28:15.354 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : bison-3.0.4-10.el8.x86_64 140/156 2024-01-26T23:28:15.366 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 140/156 2024-01-26T23:28:15.581 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 141/156 2024-01-26T23:28:15.814 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 142/156 2024-01-26T23:28:15.957 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 143/156 2024-01-26T23:28:15.957 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 144/156 2024-01-26T23:28:16.290 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 144/156 2024-01-26T23:28:16.290 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 145/156 2024-01-26T23:28:16.515 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 145/156 2024-01-26T23:28:16.516 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 146/156 2024-01-26T23:28:16.784 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 146/156 2024-01-26T23:28:16.823 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-2:17.2.7-472.g74950076.el8.x86_64 117/156 2024-01-26T23:28:17.039 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 147/156 2024-01-26T23:28:17.045 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-radosgw-2:17.2.7-472.g74950076.el8.x86_64 118/156 2024-01-26T23:28:17.143 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-472.g74950076.el8.x86_64 118/156 2024-01-26T23:28:17.264 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 148/156 2024-01-26T23:28:17.306 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-472.g749500 119/156 2024-01-26T23:28:17.357 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-472.g749500 119/156 2024-01-26T23:28:17.489 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-mirror-2:17.2.7-472.g74950076.el8.x86_64 120/156 2024-01-26T23:28:17.547 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 149/156 2024-01-26T23:28:17.548 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 150/156 2024-01-26T23:28:17.814 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 150/156 2024-01-26T23:28:17.814 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 151/156 2024-01-26T23:28:17.877 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephfs-mirror-2:17.2.7-472.g74950076.el8.x86_64 120/156 2024-01-26T23:28:18.039 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-mirror-2:17.2.7-472.g74950076.el8.x86_64 121/156 2024-01-26T23:28:18.081 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 151/156 2024-01-26T23:28:18.081 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 152/156 2024-01-26T23:28:18.364 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 152/156 2024-01-26T23:28:18.364 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 153/156 2024-01-26T23:28:18.639 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 153/156 2024-01-26T23:28:18.639 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 154/156 2024-01-26T23:28:18.897 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 154/156 2024-01-26T23:28:19.064 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 155/156 2024-01-26T23:28:19.064 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 156/156 2024-01-26T23:28:19.974 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 156/156 2024-01-26T23:28:19.975 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-2:17.2.7-472.g74950076.el8.x86_64 1/156 2024-01-26T23:28:19.975 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-base-2:17.2.7-472.g74950076.el8.x86_64 2/156 2024-01-26T23:28:19.975 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-common-2:17.2.7-472.g74950076.el8.x86_64 3/156 2024-01-26T23:28:19.975 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-fuse-2:17.2.7-472.g74950076.el8.x86_64 4/156 2024-01-26T23:28:19.975 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-472.g749500 5/156 2024-01-26T23:28:19.975 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mds-2:17.2.7-472.g74950076.el8.x86_64 6/156 2024-01-26T23:28:19.975 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-2:17.2.7-472.g74950076.el8.x86_64 7/156 2024-01-26T23:28:19.975 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mon-2:17.2.7-472.g74950076.el8.x86_64 8/156 2024-01-26T23:28:19.975 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-osd-2:17.2.7-472.g74950076.el8.x86_64 9/156 2024-01-26T23:28:19.975 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-radosgw-2:17.2.7-472.g74950076.el8.x86_64 10/156 2024-01-26T23:28:19.976 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-selinux-2:17.2.7-472.g74950076.el8.x86_64 11/156 2024-01-26T23:28:19.976 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-test-2:17.2.7-472.g74950076.el8.x86_64 12/156 2024-01-26T23:28:19.976 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephfs-mirror-2:17.2.7-472.g74950076.el8.x86_64 13/156 2024-01-26T23:28:19.976 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephfs-devel-2:17.2.7-472.g74950076.el8.x86_6 14/156 2024-01-26T23:28:19.976 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephfs2-2:17.2.7-472.g74950076.el8.x86_64 15/156 2024-01-26T23:28:19.976 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephsqlite-2:17.2.7-472.g74950076.el8.x86_64 16/156 2024-01-26T23:28:19.976 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados-devel-2:17.2.7-472.g74950076.el8.x86_64 17/156 2024-01-26T23:28:19.976 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libradosstriper1-2:17.2.7-472.g74950076.el8.x86_ 18/156 2024-01-26T23:28:19.976 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librgw2-2:17.2.7-472.g74950076.el8.x86_64 19/156 2024-01-26T23:28:19.976 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ceph-argparse-2:17.2.7-472.g74950076.el8 20/156 2024-01-26T23:28:19.976 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ceph-common-2:17.2.7-472.g74950076.el8.x 21/156 2024-01-26T23:28:19.976 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cephfs-2:17.2.7-472.g74950076.el8.x86_64 22/156 2024-01-26T23:28:19.977 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rados-2:17.2.7-472.g74950076.el8.x86_64 23/156 2024-01-26T23:28:19.977 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rbd-2:17.2.7-472.g74950076.el8.x86_64 24/156 2024-01-26T23:28:19.977 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rgw-2:17.2.7-472.g74950076.el8.x86_64 25/156 2024-01-26T23:28:19.977 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-fuse-2:17.2.7-472.g74950076.el8.x86_64 26/156 2024-01-26T23:28:19.977 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-mirror-2:17.2.7-472.g74950076.el8.x86_64 27/156 2024-01-26T23:28:19.977 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-nbd-2:17.2.7-472.g74950076.el8.x86_64 28/156 2024-01-26T23:28:19.989 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-472.g74950076.e 29/156 2024-01-26T23:28:19.989 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-472.g74950076.el8.noar 30/156 2024-01-26T23:28:19.990 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-472.g74950076.el8.no 31/156 2024-01-26T23:28:19.990 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-472.g7495 32/156 2024-01-26T23:28:19.990 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-472.g74950076.el8 33/156 2024-01-26T23:28:19.990 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-rook-2:17.2.7-472.g74950076.el8.noarch 34/156 2024-01-26T23:28:19.990 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-472.g74950076.el 35/156 2024-01-26T23:28:19.990 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-volume-2:17.2.7-472.g74950076.el8.noarch 36/156 2024-01-26T23:28:19.990 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephadm-2:17.2.7-472.g74950076.el8.noarch 37/156 2024-01-26T23:28:19.990 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephfs-top-2:17.2.7-472.g74950076.el8.noarch 38/156 2024-01-26T23:28:19.990 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/156 2024-01-26T23:28:19.990 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/156 2024-01-26T23:28:19.990 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/156 2024-01-26T23:28:19.990 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/156 2024-01-26T23:28:19.991 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/156 2024-01-26T23:28:19.991 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/156 2024-01-26T23:28:19.991 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/156 2024-01-26T23:28:19.991 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/156 2024-01-26T23:28:19.991 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/156 2024-01-26T23:28:19.991 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/156 2024-01-26T23:28:19.991 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/156 2024-01-26T23:28:19.991 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/156 2024-01-26T23:28:19.991 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/156 2024-01-26T23:28:19.991 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/156 2024-01-26T23:28:19.991 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/156 2024-01-26T23:28:19.992 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/156 2024-01-26T23:28:19.992 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/156 2024-01-26T23:28:19.992 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/156 2024-01-26T23:28:19.992 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/156 2024-01-26T23:28:19.992 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/156 2024-01-26T23:28:19.992 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/156 2024-01-26T23:28:19.992 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/156 2024-01-26T23:28:19.992 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/156 2024-01-26T23:28:19.992 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/156 2024-01-26T23:28:19.992 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/156 2024-01-26T23:28:19.992 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/156 2024-01-26T23:28:19.993 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/156 2024-01-26T23:28:19.993 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/156 2024-01-26T23:28:19.993 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/156 2024-01-26T23:28:19.993 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/156 2024-01-26T23:28:19.993 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/156 2024-01-26T23:28:19.993 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/156 2024-01-26T23:28:19.993 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/156 2024-01-26T23:28:19.993 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 72/156 2024-01-26T23:28:19.993 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/156 2024-01-26T23:28:19.993 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 74/156 2024-01-26T23:28:19.993 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 75/156 2024-01-26T23:28:19.993 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 76/156 2024-01-26T23:28:19.994 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/156 2024-01-26T23:28:19.994 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/156 2024-01-26T23:28:19.994 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : flex-2.6.1-9.el8.x86_64 79/156 2024-01-26T23:28:19.994 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 80/156 2024-01-26T23:28:19.994 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 81/156 2024-01-26T23:28:19.994 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 82/156 2024-01-26T23:28:19.994 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 83/156 2024-01-26T23:28:19.994 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 84/156 2024-01-26T23:28:19.994 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 85/156 2024-01-26T23:28:19.994 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 86/156 2024-01-26T23:28:19.994 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/156 2024-01-26T23:28:19.995 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 88/156 2024-01-26T23:28:19.995 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/156 2024-01-26T23:28:19.995 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 90/156 2024-01-26T23:28:19.995 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 91/156 2024-01-26T23:28:19.995 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 92/156 2024-01-26T23:28:19.995 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 93/156 2024-01-26T23:28:19.995 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : automake-1.16.1-8.el8.noarch 94/156 2024-01-26T23:28:19.995 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : jq-1.6-7.el8.x86_64 95/156 2024-01-26T23:28:19.995 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 96/156 2024-01-26T23:28:19.995 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 97/156 2024-01-26T23:28:19.996 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 98/156 2024-01-26T23:28:19.996 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 99/156 2024-01-26T23:28:19.996 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 100/156 2024-01-26T23:28:19.996 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 101/156 2024-01-26T23:28:19.996 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 102/156 2024-01-26T23:28:19.996 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 103/156 2024-01-26T23:28:19.996 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 104/156 2024-01-26T23:28:19.996 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 105/156 2024-01-26T23:28:19.996 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 106/156 2024-01-26T23:28:19.996 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 107/156 2024-01-26T23:28:19.996 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 108/156 2024-01-26T23:28:19.996 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 109/156 2024-01-26T23:28:19.997 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 110/156 2024-01-26T23:28:19.997 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 111/156 2024-01-26T23:28:19.997 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 112/156 2024-01-26T23:28:19.997 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 113/156 2024-01-26T23:28:19.997 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 114/156 2024-01-26T23:28:19.997 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 115/156 2024-01-26T23:28:19.997 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 116/156 2024-01-26T23:28:19.997 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 117/156 2024-01-26T23:28:19.997 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 118/156 2024-01-26T23:28:19.997 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 119/156 2024-01-26T23:28:19.998 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 120/156 2024-01-26T23:28:19.998 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 121/156 2024-01-26T23:28:19.998 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 122/156 2024-01-26T23:28:19.998 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 123/156 2024-01-26T23:28:19.998 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 124/156 2024-01-26T23:28:19.998 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados2-2:17.2.7-472.g74950076.el8.x86_64 125/156 2024-01-26T23:28:19.998 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 126/156 2024-01-26T23:28:19.998 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librbd1-2:17.2.7-472.g74950076.el8.x86_64 127/156 2024-01-26T23:28:19.998 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 128/156 2024-01-26T23:28:19.998 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 129/156 2024-01-26T23:28:19.998 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 130/156 2024-01-26T23:28:19.999 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 131/156 2024-01-26T23:28:19.999 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 132/156 2024-01-26T23:28:19.999 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 133/156 2024-01-26T23:28:19.999 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 134/156 2024-01-26T23:28:19.999 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 135/156 2024-01-26T23:28:19.999 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 136/156 2024-01-26T23:28:19.999 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 137/156 2024-01-26T23:28:19.999 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 138/156 2024-01-26T23:28:19.999 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 139/156 2024-01-26T23:28:19.999 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 140/156 2024-01-26T23:28:19.999 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 141/156 2024-01-26T23:28:19.999 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 142/156 2024-01-26T23:28:20.000 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 143/156 2024-01-26T23:28:20.000 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 144/156 2024-01-26T23:28:20.000 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 145/156 2024-01-26T23:28:20.000 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 146/156 2024-01-26T23:28:20.000 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 147/156 2024-01-26T23:28:20.000 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 148/156 2024-01-26T23:28:20.000 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 149/156 2024-01-26T23:28:20.000 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 150/156 2024-01-26T23:28:20.000 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 151/156 2024-01-26T23:28:20.000 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 152/156 2024-01-26T23:28:20.001 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 153/156 2024-01-26T23:28:20.001 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 154/156 2024-01-26T23:28:20.001 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 155/156 2024-01-26T23:28:23.466 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-mirror-2:17.2.7-472.g74950076.el8.x86_64 121/156 2024-01-26T23:28:24.555 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-test-2:17.2.7-472.g74950076.el8.x86_64 122/156 2024-01-26T23:28:24.852 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 123/156 2024-01-26T23:28:25.131 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 124/156 2024-01-26T23:28:25.375 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-fuse-2:17.2.7-472.g74950076.el8.x86_64 125/156 2024-01-26T23:28:25.579 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libtool-2.4.6-25.el8.x86_64 126/156 2024-01-26T23:28:25.598 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 126/156 2024-01-26T23:28:25.793 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 127/156 2024-01-26T23:28:25.992 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 128/156 2024-01-26T23:28:26.247 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs-devel-2:17.2.7-472.g74950076.el8.x86_6 129/156 2024-01-26T23:28:26.606 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 130/156 2024-01-26T23:28:26.958 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 131/156 2024-01-26T23:28:27.193 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 132/156 2024-01-26T23:28:27.355 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-top-2:17.2.7-472.g74950076.el8.noarch 133/156 2024-01-26T23:28:27.573 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-fuse-2:17.2.7-472.g74950076.el8.x86_64 134/156 2024-01-26T23:28:27.884 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-nbd-2:17.2.7-472.g74950076.el8.x86_64 135/156 2024-01-26T23:28:28.188 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 136/156 2024-01-26T23:28:28.387 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flex-2.6.1-9.el8.x86_64 137/156 2024-01-26T23:28:28.475 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 137/156 2024-01-26T23:28:28.729 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : bison-3.0.4-10.el8.x86_64 138/156 2024-01-26T23:28:28.746 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 138/156 2024-01-26T23:28:28.971 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 139/156 2024-01-26T23:28:28.983 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 139/156 2024-01-26T23:28:29.165 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 140/156 2024-01-26T23:28:29.315 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 141/156 2024-01-26T23:28:29.515 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 142/156 2024-01-26T23:28:29.700 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 143/156 2024-01-26T23:28:29.700 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 144/156 2024-01-26T23:28:29.974 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 144/156 2024-01-26T23:28:29.974 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 145/156 2024-01-26T23:28:30.208 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 145/156 2024-01-26T23:28:30.209 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 146/156 2024-01-26T23:28:30.478 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 146/156 2024-01-26T23:28:30.721 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 147/156 2024-01-26T23:28:30.901 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 148/156 2024-01-26T23:28:31.131 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 149/156 2024-01-26T23:28:31.131 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 150/156 2024-01-26T23:28:31.407 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 150/156 2024-01-26T23:28:31.407 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 151/156 2024-01-26T23:28:31.674 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 151/156 2024-01-26T23:28:31.674 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 152/156 2024-01-26T23:28:31.932 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 152/156 2024-01-26T23:28:31.932 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 153/156 2024-01-26T23:28:32.173 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 153/156 2024-01-26T23:28:32.173 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 154/156 2024-01-26T23:28:32.388 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 154/156 2024-01-26T23:28:32.557 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 155/156 2024-01-26T23:28:32.557 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 156/156 2024-01-26T23:28:33.576 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 156/156 2024-01-26T23:28:33.576 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-2:17.2.7-472.g74950076.el8.x86_64 1/156 2024-01-26T23:28:33.576 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-base-2:17.2.7-472.g74950076.el8.x86_64 2/156 2024-01-26T23:28:33.576 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-common-2:17.2.7-472.g74950076.el8.x86_64 3/156 2024-01-26T23:28:33.576 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-fuse-2:17.2.7-472.g74950076.el8.x86_64 4/156 2024-01-26T23:28:33.576 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-472.g749500 5/156 2024-01-26T23:28:33.576 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mds-2:17.2.7-472.g74950076.el8.x86_64 6/156 2024-01-26T23:28:33.576 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-2:17.2.7-472.g74950076.el8.x86_64 7/156 2024-01-26T23:28:33.577 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mon-2:17.2.7-472.g74950076.el8.x86_64 8/156 2024-01-26T23:28:33.577 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-osd-2:17.2.7-472.g74950076.el8.x86_64 9/156 2024-01-26T23:28:33.577 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-radosgw-2:17.2.7-472.g74950076.el8.x86_64 10/156 2024-01-26T23:28:33.577 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-selinux-2:17.2.7-472.g74950076.el8.x86_64 11/156 2024-01-26T23:28:33.577 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-test-2:17.2.7-472.g74950076.el8.x86_64 12/156 2024-01-26T23:28:33.577 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-mirror-2:17.2.7-472.g74950076.el8.x86_64 13/156 2024-01-26T23:28:33.577 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs-devel-2:17.2.7-472.g74950076.el8.x86_6 14/156 2024-01-26T23:28:33.577 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs2-2:17.2.7-472.g74950076.el8.x86_64 15/156 2024-01-26T23:28:33.577 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephsqlite-2:17.2.7-472.g74950076.el8.x86_64 16/156 2024-01-26T23:28:33.577 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados-devel-2:17.2.7-472.g74950076.el8.x86_64 17/156 2024-01-26T23:28:33.577 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libradosstriper1-2:17.2.7-472.g74950076.el8.x86_ 18/156 2024-01-26T23:28:33.577 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librgw2-2:17.2.7-472.g74950076.el8.x86_64 19/156 2024-01-26T23:28:33.577 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-argparse-2:17.2.7-472.g74950076.el8 20/156 2024-01-26T23:28:33.577 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-common-2:17.2.7-472.g74950076.el8.x 21/156 2024-01-26T23:28:33.577 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cephfs-2:17.2.7-472.g74950076.el8.x86_64 22/156 2024-01-26T23:28:33.577 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rados-2:17.2.7-472.g74950076.el8.x86_64 23/156 2024-01-26T23:28:33.578 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rbd-2:17.2.7-472.g74950076.el8.x86_64 24/156 2024-01-26T23:28:33.578 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rgw-2:17.2.7-472.g74950076.el8.x86_64 25/156 2024-01-26T23:28:33.578 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-fuse-2:17.2.7-472.g74950076.el8.x86_64 26/156 2024-01-26T23:28:33.578 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-mirror-2:17.2.7-472.g74950076.el8.x86_64 27/156 2024-01-26T23:28:33.578 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-nbd-2:17.2.7-472.g74950076.el8.x86_64 28/156 2024-01-26T23:28:33.578 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-472.g74950076.e 29/156 2024-01-26T23:28:33.578 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-472.g74950076.el8.noar 30/156 2024-01-26T23:28:33.578 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-472.g74950076.el8.no 31/156 2024-01-26T23:28:33.578 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-472.g7495 32/156 2024-01-26T23:28:33.578 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-472.g74950076.el8 33/156 2024-01-26T23:28:33.578 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-rook-2:17.2.7-472.g74950076.el8.noarch 34/156 2024-01-26T23:28:33.578 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-472.g74950076.el 35/156 2024-01-26T23:28:33.578 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-volume-2:17.2.7-472.g74950076.el8.noarch 36/156 2024-01-26T23:28:33.578 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephadm-2:17.2.7-472.g74950076.el8.noarch 37/156 2024-01-26T23:28:33.579 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-top-2:17.2.7-472.g74950076.el8.noarch 38/156 2024-01-26T23:28:33.579 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/156 2024-01-26T23:28:33.579 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/156 2024-01-26T23:28:33.579 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/156 2024-01-26T23:28:33.579 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/156 2024-01-26T23:28:33.579 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/156 2024-01-26T23:28:33.579 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/156 2024-01-26T23:28:33.579 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/156 2024-01-26T23:28:33.579 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/156 2024-01-26T23:28:33.579 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/156 2024-01-26T23:28:33.579 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/156 2024-01-26T23:28:33.579 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/156 2024-01-26T23:28:33.579 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/156 2024-01-26T23:28:33.579 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/156 2024-01-26T23:28:33.580 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/156 2024-01-26T23:28:33.580 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/156 2024-01-26T23:28:33.580 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/156 2024-01-26T23:28:33.580 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/156 2024-01-26T23:28:33.580 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/156 2024-01-26T23:28:33.580 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/156 2024-01-26T23:28:33.580 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/156 2024-01-26T23:28:33.580 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/156 2024-01-26T23:28:33.580 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/156 2024-01-26T23:28:33.580 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/156 2024-01-26T23:28:33.580 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/156 2024-01-26T23:28:33.580 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/156 2024-01-26T23:28:33.580 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/156 2024-01-26T23:28:33.580 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/156 2024-01-26T23:28:33.580 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/156 2024-01-26T23:28:33.581 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/156 2024-01-26T23:28:33.581 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/156 2024-01-26T23:28:33.581 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/156 2024-01-26T23:28:33.581 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/156 2024-01-26T23:28:33.581 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/156 2024-01-26T23:28:33.581 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 72/156 2024-01-26T23:28:33.581 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/156 2024-01-26T23:28:33.581 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/156 2024-01-26T23:28:33.581 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 75/156 2024-01-26T23:28:33.581 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 76/156 2024-01-26T23:28:33.581 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 77/156 2024-01-26T23:28:33.581 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 78/156 2024-01-26T23:28:33.581 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 79/156 2024-01-26T23:28:33.581 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/156 2024-01-26T23:28:33.582 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 81/156 2024-01-26T23:28:33.582 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 82/156 2024-01-26T23:28:33.582 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 83/156 2024-01-26T23:28:33.582 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 84/156 2024-01-26T23:28:33.582 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 85/156 2024-01-26T23:28:33.582 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 86/156 2024-01-26T23:28:33.582 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 87/156 2024-01-26T23:28:33.582 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 88/156 2024-01-26T23:28:33.582 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 89/156 2024-01-26T23:28:33.582 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 90/156 2024-01-26T23:28:33.582 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 91/156 2024-01-26T23:28:33.582 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 92/156 2024-01-26T23:28:33.582 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 93/156 2024-01-26T23:28:33.582 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 94/156 2024-01-26T23:28:33.583 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 95/156 2024-01-26T23:28:33.583 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 96/156 2024-01-26T23:28:33.583 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 97/156 2024-01-26T23:28:33.583 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 98/156 2024-01-26T23:28:33.583 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 99/156 2024-01-26T23:28:33.583 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 100/156 2024-01-26T23:28:33.583 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 101/156 2024-01-26T23:28:33.583 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 102/156 2024-01-26T23:28:33.583 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 103/156 2024-01-26T23:28:33.583 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 104/156 2024-01-26T23:28:33.583 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : bison-3.0.4-10.el8.x86_64 105/156 2024-01-26T23:28:33.583 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flex-2.6.1-9.el8.x86_64 106/156 2024-01-26T23:28:33.583 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 107/156 2024-01-26T23:28:33.583 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 108/156 2024-01-26T23:28:33.584 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 109/156 2024-01-26T23:28:33.584 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 110/156 2024-01-26T23:28:33.584 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 111/156 2024-01-26T23:28:33.584 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 112/156 2024-01-26T23:28:33.584 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 113/156 2024-01-26T23:28:33.584 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 114/156 2024-01-26T23:28:33.584 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 115/156 2024-01-26T23:28:33.584 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 116/156 2024-01-26T23:28:33.584 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 117/156 2024-01-26T23:28:33.584 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 118/156 2024-01-26T23:28:33.584 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 119/156 2024-01-26T23:28:33.584 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 120/156 2024-01-26T23:28:33.584 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : automake-1.16.1-8.el8.noarch 121/156 2024-01-26T23:28:33.585 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : jq-1.6-7.el8.x86_64 122/156 2024-01-26T23:28:33.585 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 123/156 2024-01-26T23:28:33.585 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 124/156 2024-01-26T23:28:33.585 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:17.2.7-472.g74950076.el8.x86_64 125/156 2024-01-26T23:28:33.585 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 126/156 2024-01-26T23:28:33.585 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:17.2.7-472.g74950076.el8.x86_64 127/156 2024-01-26T23:28:33.585 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 128/156 2024-01-26T23:28:33.585 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 129/156 2024-01-26T23:28:33.585 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 130/156 2024-01-26T23:28:33.585 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 131/156 2024-01-26T23:28:33.585 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 132/156 2024-01-26T23:28:33.585 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 133/156 2024-01-26T23:28:33.585 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 134/156 2024-01-26T23:28:33.585 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 135/156 2024-01-26T23:28:33.585 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 136/156 2024-01-26T23:28:33.586 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 137/156 2024-01-26T23:28:33.586 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 138/156 2024-01-26T23:28:33.586 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 139/156 2024-01-26T23:28:33.586 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 140/156 2024-01-26T23:28:33.586 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 141/156 2024-01-26T23:28:33.586 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 142/156 2024-01-26T23:28:33.586 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 143/156 2024-01-26T23:28:33.586 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 144/156 2024-01-26T23:28:33.586 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 145/156 2024-01-26T23:28:33.586 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 146/156 2024-01-26T23:28:33.586 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 147/156 2024-01-26T23:28:33.586 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 148/156 2024-01-26T23:28:33.586 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 149/156 2024-01-26T23:28:33.586 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 150/156 2024-01-26T23:28:33.586 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 151/156 2024-01-26T23:28:33.587 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 152/156 2024-01-26T23:28:33.587 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 153/156 2024-01-26T23:28:33.587 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 154/156 2024-01-26T23:28:33.587 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 155/156 2024-01-26T23:28:35.195 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 156/156 2024-01-26T23:28:35.195 INFO:teuthology.orchestra.run.smithi106.stdout:Installed products updated. 2024-01-26T23:28:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:36.466 INFO:teuthology.orchestra.run.smithi106.stdout:Upgraded: 2024-01-26T23:28:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-01-26T23:28:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-01-26T23:28:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-01-26T23:28:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: libblkid-2.32.1-43.el8.x86_64 2024-01-26T23:28:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-01-26T23:28:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: libmount-2.32.1-43.el8.x86_64 2024-01-26T23:28:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: librados2-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: librbd1-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-01-26T23:28:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: libuuid-2.32.1-43.el8.x86_64 2024-01-26T23:28:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-01-26T23:28:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-01-26T23:28:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-2.32.1-43.el8.x86_64 2024-01-26T23:28:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-01-26T23:28:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-01-26T23:28:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-01-26T23:28:36.468 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-26T23:28:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-01-26T23:28:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: automake-1.16.1-8.el8.noarch 2024-01-26T23:28:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: bison-3.0.4-10.el8.x86_64 2024-01-26T23:28:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-grafana-dashboards-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-immutable-object-cache-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-cephadm-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-dashboard-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-diskprediction-local-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-modules-core-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-rook-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-prometheus-alerts-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-selinux-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-volume-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: cephadm-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-mirror-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-top-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-01-26T23:28:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: flex-2.6.1-9.el8.x86_64 2024-01-26T23:28:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: fuse-2.9.7-17.el8.x86_64 2024-01-26T23:28:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-26T23:28:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: jq-1.6-7.el8.x86_64 2024-01-26T23:28:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-01-26T23:28:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-01-26T23:28:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-01-26T23:28:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs-devel-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: libcephsqlite-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-01-26T23:28:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-01-26T23:28:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-26T23:28:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-01-26T23:28:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-26T23:28:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: libradosstriper1-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-01-26T23:28:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: librgw2-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-01-26T23:28:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-01-26T23:28:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: libtool-2.4.6-25.el8.x86_64 2024-01-26T23:28:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-26T23:28:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-01-26T23:28:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-01-26T23:28:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-01-26T23:28:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-26T23:28:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-26T23:28:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-0.3.15-6.el8.x86_64 2024-01-26T23:28:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-01-26T23:28:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-01-26T23:28:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-01-26T23:28:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-01-26T23:28:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-01-26T23:28:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-01-26T23:28:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-26T23:28:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-26T23:28:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-26T23:28:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-26T23:28:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-argparse-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-common-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-26T23:28:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-26T23:28:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-26T23:28:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-26T23:28:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2024-01-26T23:28:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-26T23:28:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-26T23:28:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-26T23:28:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-26T23:28:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-26T23:28:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-26T23:28:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-26T23:28:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-26T23:28:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-26T23:28:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-26T23:28:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-26T23:28:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-26T23:28:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-01-26T23:28:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-26T23:28:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-26T23:28:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-26T23:28:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-26T23:28:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: python3-portend-2.6-1.el8.noarch 2024-01-26T23:28:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-26T23:28:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-26T23:28:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-26T23:28:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-26T23:28:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-26T23:28:36.475 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-26T23:28:36.475 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.475 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.475 INFO:teuthology.orchestra.run.smithi106.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-26T23:28:36.475 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-26T23:28:36.475 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.475 INFO:teuthology.orchestra.run.smithi106.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-26T23:28:36.475 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-26T23:28:36.475 INFO:teuthology.orchestra.run.smithi106.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-01-26T23:28:36.475 INFO:teuthology.orchestra.run.smithi106.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-26T23:28:36.475 INFO:teuthology.orchestra.run.smithi106.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-26T23:28:36.475 INFO:teuthology.orchestra.run.smithi106.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-26T23:28:36.475 INFO:teuthology.orchestra.run.smithi106.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-26T23:28:36.475 INFO:teuthology.orchestra.run.smithi106.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-26T23:28:36.476 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-26T23:28:36.476 INFO:teuthology.orchestra.run.smithi106.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-26T23:28:36.476 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-26T23:28:36.476 INFO:teuthology.orchestra.run.smithi106.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-26T23:28:36.476 INFO:teuthology.orchestra.run.smithi106.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-26T23:28:36.476 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.476 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.476 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:36.476 INFO:teuthology.orchestra.run.smithi106.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-26T23:28:36.476 INFO:teuthology.orchestra.run.smithi106.stdout: thrift-0.13.0-2.el8.x86_64 2024-01-26T23:28:36.476 INFO:teuthology.orchestra.run.smithi106.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-01-26T23:28:36.476 INFO:teuthology.orchestra.run.smithi106.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-26T23:28:36.477 INFO:teuthology.orchestra.run.smithi106.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-01-26T23:28:36.477 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:36.477 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-26T23:28:36.676 DEBUG:teuthology.parallel:result is None 2024-01-26T23:28:48.296 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 156/156 2024-01-26T23:28:48.296 INFO:teuthology.orchestra.run.smithi046.stdout:Installed products updated. 2024-01-26T23:28:49.776 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:49.776 INFO:teuthology.orchestra.run.smithi046.stdout:Upgraded: 2024-01-26T23:28:49.776 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-01-26T23:28:49.776 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-01-26T23:28:49.777 INFO:teuthology.orchestra.run.smithi046.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-01-26T23:28:49.777 INFO:teuthology.orchestra.run.smithi046.stdout: libblkid-2.32.1-43.el8.x86_64 2024-01-26T23:28:49.777 INFO:teuthology.orchestra.run.smithi046.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-01-26T23:28:49.777 INFO:teuthology.orchestra.run.smithi046.stdout: libmount-2.32.1-43.el8.x86_64 2024-01-26T23:28:49.777 INFO:teuthology.orchestra.run.smithi046.stdout: librados2-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.777 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.777 INFO:teuthology.orchestra.run.smithi046.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-01-26T23:28:49.777 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-2.32.1-43.el8.x86_64 2024-01-26T23:28:49.777 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-01-26T23:28:49.777 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-01-26T23:28:49.777 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-2.32.1-43.el8.x86_64 2024-01-26T23:28:49.777 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-01-26T23:28:49.777 INFO:teuthology.orchestra.run.smithi046.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-01-26T23:28:49.777 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-01-26T23:28:49.778 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-26T23:28:49.778 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-01-26T23:28:49.778 INFO:teuthology.orchestra.run.smithi046.stdout: automake-1.16.1-8.el8.noarch 2024-01-26T23:28:49.778 INFO:teuthology.orchestra.run.smithi046.stdout: bison-3.0.4-10.el8.x86_64 2024-01-26T23:28:49.778 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.778 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.778 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.778 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.778 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:49.778 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.778 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.778 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.778 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:49.779 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:49.779 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:49.779 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:49.779 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:49.779 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.779 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.779 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:49.779 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.779 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.779 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.779 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:49.779 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:49.779 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.779 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top-2:17.2.7-472.g74950076.el8.noarch 2024-01-26T23:28:49.780 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-01-26T23:28:49.780 INFO:teuthology.orchestra.run.smithi046.stdout: flex-2.6.1-9.el8.x86_64 2024-01-26T23:28:49.780 INFO:teuthology.orchestra.run.smithi046.stdout: fuse-2.9.7-17.el8.x86_64 2024-01-26T23:28:49.780 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-26T23:28:49.780 INFO:teuthology.orchestra.run.smithi046.stdout: jq-1.6-7.el8.x86_64 2024-01-26T23:28:49.780 INFO:teuthology.orchestra.run.smithi046.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-01-26T23:28:49.780 INFO:teuthology.orchestra.run.smithi046.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-01-26T23:28:49.780 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-01-26T23:28:49.780 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.780 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.780 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.780 INFO:teuthology.orchestra.run.smithi046.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-01-26T23:28:49.781 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-01-26T23:28:49.781 INFO:teuthology.orchestra.run.smithi046.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-26T23:28:49.781 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-01-26T23:28:49.781 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-26T23:28:49.781 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.781 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.781 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-01-26T23:28:49.781 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.781 INFO:teuthology.orchestra.run.smithi046.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-01-26T23:28:49.781 INFO:teuthology.orchestra.run.smithi046.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-01-26T23:28:49.781 INFO:teuthology.orchestra.run.smithi046.stdout: libtool-2.4.6-25.el8.x86_64 2024-01-26T23:28:49.781 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-26T23:28:49.781 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-01-26T23:28:49.782 INFO:teuthology.orchestra.run.smithi046.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-01-26T23:28:49.782 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-01-26T23:28:49.782 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-26T23:28:49.782 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-26T23:28:49.782 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-0.3.15-6.el8.x86_64 2024-01-26T23:28:49.782 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-01-26T23:28:49.782 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-01-26T23:28:49.782 INFO:teuthology.orchestra.run.smithi046.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-01-26T23:28:49.782 INFO:teuthology.orchestra.run.smithi046.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-01-26T23:28:49.782 INFO:teuthology.orchestra.run.smithi046.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-01-26T23:28:49.782 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-01-26T23:28:49.782 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-26T23:28:49.782 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-26T23:28:49.782 INFO:teuthology.orchestra.run.smithi046.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-26T23:28:49.783 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-26T23:28:49.783 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.783 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.783 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.783 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-26T23:28:49.783 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-26T23:28:49.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-26T23:28:49.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-26T23:28:49.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2024-01-26T23:28:49.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-26T23:28:49.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-26T23:28:49.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-26T23:28:49.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-26T23:28:49.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-26T23:28:49.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-26T23:28:49.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-26T23:28:49.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-26T23:28:49.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-26T23:28:49.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-26T23:28:49.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-26T23:28:49.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-26T23:28:49.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-01-26T23:28:49.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-26T23:28:49.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-26T23:28:49.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-26T23:28:49.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-26T23:28:49.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend-2.6-1.el8.noarch 2024-01-26T23:28:49.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-26T23:28:49.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-26T23:28:49.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-26T23:28:49.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-26T23:28:49.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-26T23:28:49.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-26T23:28:49.788 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.788 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.788 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-26T23:28:49.788 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-26T23:28:49.788 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.788 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-26T23:28:49.788 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-26T23:28:49.789 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-01-26T23:28:49.789 INFO:teuthology.orchestra.run.smithi046.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-26T23:28:49.789 INFO:teuthology.orchestra.run.smithi046.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-26T23:28:49.789 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-26T23:28:49.789 INFO:teuthology.orchestra.run.smithi046.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-26T23:28:49.789 INFO:teuthology.orchestra.run.smithi046.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-26T23:28:49.790 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-26T23:28:49.790 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-26T23:28:49.790 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-26T23:28:49.790 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-26T23:28:49.790 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-26T23:28:49.790 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.790 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.790 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd-2:17.2.7-472.g74950076.el8.x86_64 2024-01-26T23:28:49.791 INFO:teuthology.orchestra.run.smithi046.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-26T23:28:49.791 INFO:teuthology.orchestra.run.smithi046.stdout: thrift-0.13.0-2.el8.x86_64 2024-01-26T23:28:49.791 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-01-26T23:28:49.791 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-26T23:28:49.792 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-01-26T23:28:49.792 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:49.792 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-26T23:28:50.003 DEBUG:teuthology.parallel:result is None 2024-01-26T23:28:50.004 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7495007658df82356749387b98425d4a7fcef9b6 2024-01-26T23:28:50.155 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-26T23:28:50.183 INFO:teuthology.orchestra.run.smithi046.stdout:17.2.7-472.g74950076.el8 2024-01-26T23:28:50.184 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-472.g74950076.el8 2024-01-26T23:28:50.184 INFO:teuthology.task.install:The correct ceph version 17.2.7-472.g74950076 is installed. 2024-01-26T23:28:50.186 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7495007658df82356749387b98425d4a7fcef9b6 2024-01-26T23:28:50.434 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-26T23:28:50.463 INFO:teuthology.orchestra.run.smithi106.stdout:17.2.7-472.g74950076.el8 2024-01-26T23:28:50.463 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-472.g74950076.el8 2024-01-26T23:28:50.463 INFO:teuthology.task.install:The correct ceph version 17.2.7-472.g74950076 is installed. 2024-01-26T23:28:50.465 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-26T23:28:50.465 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:28:50.465 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-26T23:28:50.517 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:28:50.517 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-26T23:28:50.567 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-26T23:28:50.567 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:28:50.567 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2024-01-26T23:28:50.601 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-26T23:28:50.676 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:28:50.677 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/daemon-helper 2024-01-26T23:28:50.708 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-26T23:28:50.779 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-26T23:28:50.779 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:28:50.779 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-26T23:28:50.813 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-26T23:28:50.887 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:28:50.887 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-26T23:28:50.916 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-26T23:28:50.985 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-26T23:28:50.985 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:28:50.985 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2024-01-26T23:28:51.020 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-26T23:28:51.094 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:28:51.094 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/stdin-killer 2024-01-26T23:28:51.122 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-26T23:28:51.189 INFO:teuthology.run_tasks:Running task ceph... 2024-01-26T23:28:51.254 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-26T23:28:51.254 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /var/log/ceph 2024-01-26T23:28:51.257 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 777 /var/log/ceph 2024-01-26T23:28:51.288 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-26T23:28:51.288 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-26T23:28:51.331 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-26T23:28:51.364 INFO:tasks.ceph:Creating extra log directories... 2024-01-26T23:28:51.364 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-26T23:28:51.406 DEBUG:teuthology.orchestra.run.smithi106:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-26T23:28:51.444 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-01-26T23:28:51.445 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-01-26T23:28:51.445 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-01-26T23:28:51.445 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:28:51.445 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-26T23:28:51.500 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-26T23:28:51.628 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-26T23:28:51.662 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:28:51.662 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-26T23:28:51.679 DEBUG:teuthology.orchestra.run.smithi106:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-26T23:28:51.794 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-26T23:28:51.826 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-26T23:28:51.826 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-26T23:28:51.827 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi/7533360', 'branch': 'wip-yuri4-testing-2024-01-25-1331-quincy', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7533360', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2024-01-25-1331-quincy'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '7495007658df82356749387b98425d4a7fcef9b6'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7495007658df82356749387b98425d4a7fcef9b6'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2024-01-25-1331-quincy', 'sha1': '7495007658df82356749387b98425d4a7fcef9b6'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 1285, 'sha1': '7495007658df82356749387b98425d4a7fcef9b6', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'fs', 'suite_branch': 'wip-yuri4-testing-2024-01-25-1331-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_7495007658df82356749387b98425d4a7fcef9b6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '7495007658df82356749387b98425d4a7fcef9b6', 'targets': {'smithi046.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH0jCpB2oSBYuBJFz8r7h5gvoxbVdD5G78u+0MHqsdwmoDh9tqhFT/LoUCId1h189cSnuoFeCa4UY2kBN3lkbp0=', 'smithi106.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGNrzIpyQQzXce3S622+VCGLhT0dDQ6JzjjnSWBTu1pGvGlS9KyptTAX9xnumt/OLUGHnhziceBhmNU9cWNCBCE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '7495007658df82356749387b98425d4a7fcef9b6'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '7495007658df82356749387b98425d4a7fcef9b6', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'c856e0f9244dd29a50697123b480d63f16a4ed81', 'timestamp': '2024-01-26_01:07:29', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-26T23:28:51.827 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-26T23:28:51.830 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-26T23:28:51.878 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/run/ceph 2024-01-26T23:28:51.891 DEBUG:teuthology.orchestra.run.smithi106:> sudo install -d -m0777 -- /var/run/ceph 2024-01-26T23:28:51.945 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:28:51.945 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2024-01-26T23:28:51.984 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-26T23:28:51.985 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2024-01-26T23:28:52.047 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-26T23:28:52.047 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-26T23:28:52.047 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 6h/6d Inode: 165703 Links: 1 2024-01-26T23:28:52.047 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-26T23:28:52.047 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-01-26T23:28:52.047 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-01-26 23:28:33.127203578 +0000 2024-01-26T23:28:52.047 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-01-26 23:25:03.677272158 +0000 2024-01-26T23:28:52.047 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-01-26 23:25:03.677272158 +0000 2024-01-26T23:28:52.047 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2024-01-26T23:28:52.048 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-26T23:28:52.120 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-01-26T23:28:52.120 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-01-26T23:28:52.121 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000141838 s, 3.6 MB/s 2024-01-26T23:28:52.122 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-26T23:28:52.186 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2024-01-26T23:28:52.247 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-26T23:28:52.248 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-26T23:28:52.248 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 6h/6d Inode: 176324 Links: 1 2024-01-26T23:28:52.248 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-26T23:28:52.248 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-01-26T23:28:52.248 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-01-26 23:28:33.127203578 +0000 2024-01-26T23:28:52.248 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-01-26 23:25:04.036268768 +0000 2024-01-26T23:28:52.248 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-01-26 23:25:04.036268768 +0000 2024-01-26T23:28:52.248 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2024-01-26T23:28:52.248 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-26T23:28:52.320 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-01-26T23:28:52.320 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-01-26T23:28:52.320 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000121344 s, 4.2 MB/s 2024-01-26T23:28:52.324 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-26T23:28:52.387 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2024-01-26T23:28:52.449 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-26T23:28:52.450 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-26T23:28:52.450 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 6h/6d Inode: 175822 Links: 1 2024-01-26T23:28:52.450 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-26T23:28:52.450 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-01-26T23:28:52.450 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-01-26 23:28:33.127203578 +0000 2024-01-26T23:28:52.450 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-01-26 23:25:04.410265235 +0000 2024-01-26T23:28:52.450 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-01-26 23:25:04.410265235 +0000 2024-01-26T23:28:52.450 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2024-01-26T23:28:52.450 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-26T23:28:52.522 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-01-26T23:28:52.522 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-01-26T23:28:52.522 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000128249 s, 4.0 MB/s 2024-01-26T23:28:52.523 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-26T23:28:52.587 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2024-01-26T23:28:52.649 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-26T23:28:52.649 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-26T23:28:52.650 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 6h/6d Inode: 178379 Links: 1 2024-01-26T23:28:52.650 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-26T23:28:52.650 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-01-26T23:28:52.650 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-01-26 23:28:33.127203578 +0000 2024-01-26T23:28:52.650 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-01-26 23:25:04.781261732 +0000 2024-01-26T23:28:52.650 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-01-26 23:25:04.781261732 +0000 2024-01-26T23:28:52.650 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2024-01-26T23:28:52.650 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-26T23:28:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-01-26T23:28:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-01-26T23:28:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000121279 s, 4.2 MB/s 2024-01-26T23:28:52.724 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-26T23:28:52.788 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-26T23:28:52.788 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:28:52.788 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/scratch_devs of=/dev/stdout 2024-01-26T23:28:52.809 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-26T23:28:52.810 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_1 2024-01-26T23:28:52.870 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-26T23:28:52.870 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-26T23:28:52.870 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 6h/6d Inode: 176778 Links: 1 2024-01-26T23:28:52.870 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-26T23:28:52.870 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-01-26T23:28:52.870 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-01-26 23:28:19.627261830 +0000 2024-01-26T23:28:52.870 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-01-26 23:24:41.207791122 +0000 2024-01-26T23:28:52.870 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-01-26 23:24:41.207791122 +0000 2024-01-26T23:28:52.870 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: - 2024-01-26T23:28:52.871 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-26T23:28:52.944 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-01-26T23:28:52.944 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-01-26T23:28:52.944 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000259033 s, 2.0 MB/s 2024-01-26T23:28:52.944 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-26T23:28:53.007 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_2 2024-01-26T23:28:53.068 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-26T23:28:53.068 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-26T23:28:53.068 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 6h/6d Inode: 177889 Links: 1 2024-01-26T23:28:53.068 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-26T23:28:53.068 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-01-26T23:28:53.069 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-01-26 23:28:19.627261830 +0000 2024-01-26T23:28:53.069 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-01-26 23:24:41.557782732 +0000 2024-01-26T23:28:53.069 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-01-26 23:24:41.557782732 +0000 2024-01-26T23:28:53.069 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: - 2024-01-26T23:28:53.069 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-26T23:28:53.139 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-01-26T23:28:53.139 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-01-26T23:28:53.140 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000276983 s, 1.8 MB/s 2024-01-26T23:28:53.141 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-26T23:28:53.202 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_3 2024-01-26T23:28:53.262 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-26T23:28:53.262 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-26T23:28:53.262 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 6h/6d Inode: 182322 Links: 1 2024-01-26T23:28:53.262 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-26T23:28:53.262 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-01-26T23:28:53.262 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-01-26 23:28:19.627261830 +0000 2024-01-26T23:28:53.262 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-01-26 23:24:41.921774007 +0000 2024-01-26T23:28:53.262 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-01-26 23:24:41.921774007 +0000 2024-01-26T23:28:53.262 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: - 2024-01-26T23:28:53.263 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-26T23:28:53.331 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-01-26T23:28:53.331 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-01-26T23:28:53.331 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000198272 s, 2.6 MB/s 2024-01-26T23:28:53.333 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-26T23:28:53.393 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_4 2024-01-26T23:28:53.454 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-26T23:28:53.454 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-26T23:28:53.454 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 6h/6d Inode: 176947 Links: 1 2024-01-26T23:28:53.454 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-26T23:28:53.454 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-01-26T23:28:53.455 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-01-26 23:28:19.627261830 +0000 2024-01-26T23:28:53.455 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-01-26 23:24:42.282765354 +0000 2024-01-26T23:28:53.455 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-01-26 23:24:42.282765354 +0000 2024-01-26T23:28:53.455 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: - 2024-01-26T23:28:53.456 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-26T23:28:53.522 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-01-26T23:28:53.522 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-01-26T23:28:53.522 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000256685 s, 2.0 MB/s 2024-01-26T23:28:53.523 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-26T23:28:53.583 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-26T23:28:53.583 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi106.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-26T23:28:53.584 INFO:tasks.ceph:Generating config... 2024-01-26T23:28:53.585 INFO:tasks.ceph:[client] client mount timeout = 600 2024-01-26T23:28:53.585 INFO:tasks.ceph:[client] debug client = 20 2024-01-26T23:28:53.585 INFO:tasks.ceph:[client] debug ms = 1 2024-01-26T23:28:53.585 INFO:tasks.ceph:[client] pid file = 2024-01-26T23:28:53.586 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-01-26T23:28:53.586 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-01-26T23:28:53.586 INFO:tasks.ceph:[mds] debug mds = 20 2024-01-26T23:28:53.586 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-01-26T23:28:53.586 INFO:tasks.ceph:[mds] debug ms = 1 2024-01-26T23:28:53.587 INFO:tasks.ceph:[mds] mds debug frag = True 2024-01-26T23:28:53.587 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-01-26T23:28:53.587 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-01-26T23:28:53.587 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-01-26T23:28:53.587 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-01-26T23:28:53.587 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-01-26T23:28:53.587 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-01-26T23:28:53.587 INFO:tasks.ceph:[mgr] debug client = 20 2024-01-26T23:28:53.587 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-01-26T23:28:53.587 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-26T23:28:53.587 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-26T23:28:53.587 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-26T23:28:53.587 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-26T23:28:53.587 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-26T23:28:53.587 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-01-26T23:28:53.588 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-26T23:28:53.588 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-26T23:28:53.588 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-01-26T23:28:53.588 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-01-26T23:28:53.588 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-26T23:28:53.588 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-26T23:28:53.588 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-01-26T23:28:53.588 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-26T23:28:53.588 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-26T23:28:53.588 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-26T23:28:53.589 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-26T23:28:53.589 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-26T23:28:53.589 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-01-26T23:28:53.589 INFO:tasks.ceph:Setting up mon.a... 2024-01-26T23:28:53.589 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-26T23:28:53.655 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.keyring 2024-01-26T23:28:53.658 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-26T23:28:53.709 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-26T23:28:53.741 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.46'), ('mon.b', '172.21.15.106'), ('mon.c', '[v2:172.21.15.106:3301,v1:172.21.15.106:6790]')] 2024-01-26T23:28:53.741 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.46,172.21.15.106,[v2:172.21.15.106:3301,v1:172.21.15.106:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-01-26T23:28:53.742 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:28:53.742 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-26T23:28:53.803 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.46 --add b 172.21.15.106 --addv c '[v2:172.21.15.106:3301,v1:172.21.15.106:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-01-26T23:28:53.896 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-26T23:28:53.896 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: generated fsid 62f24da2-f257-4a3f-ab5a-77a1100d6f1e 2024-01-26T23:28:53.897 INFO:teuthology.orchestra.run.smithi046.stdout:setting min_mon_release = octopus 2024-01-26T23:28:53.897 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 0 2024-01-26T23:28:53.897 INFO:teuthology.orchestra.run.smithi046.stdout:fsid 62f24da2-f257-4a3f-ab5a-77a1100d6f1e 2024-01-26T23:28:53.897 INFO:teuthology.orchestra.run.smithi046.stdout:last_changed 2024-01-26T23:28:53.896352+0000 2024-01-26T23:28:53.897 INFO:teuthology.orchestra.run.smithi046.stdout:created 2024-01-26T23:28:53.896352+0000 2024-01-26T23:28:53.897 INFO:teuthology.orchestra.run.smithi046.stdout:min_mon_release 15 (octopus) 2024-01-26T23:28:53.897 INFO:teuthology.orchestra.run.smithi046.stdout:election_strategy: 1 2024-01-26T23:28:53.897 INFO:teuthology.orchestra.run.smithi046.stdout:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-01-26T23:28:53.897 INFO:teuthology.orchestra.run.smithi046.stdout:1: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.b 2024-01-26T23:28:53.897 INFO:teuthology.orchestra.run.smithi046.stdout:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-01-26T23:28:53.897 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-01-26T23:28:53.898 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-26T23:28:53.918 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 62f24da2-f257-4a3f-ab5a-77a1100d6f1e... 2024-01-26T23:28:53.919 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-26T23:28:53.961 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-26T23:28:54.016 INFO:teuthology.orchestra.run.smithi106.stdout:[global] 2024-01-26T23:28:54.016 INFO:teuthology.orchestra.run.smithi106.stdout: chdir = "" 2024-01-26T23:28:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-26T23:28:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: auth supported = cephx 2024-01-26T23:28:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: filestore xattr use omap = true 2024-01-26T23:28:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: mon clock drift allowed = 1.000 2024-01-26T23:28:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: osd crush chooseleaf type = 0 2024-01-26T23:28:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: auth debug = true 2024-01-26T23:28:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: ms die on old message = true 2024-01-26T23:28:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: ms die on bug = true 2024-01-26T23:28:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-26T23:28:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: mon pg warn max object skew = 0 2024-01-26T23:28:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: # disable pg_autoscaler by default for new pools 2024-01-26T23:28:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-26T23:28:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: osd pool default size = 2 2024-01-26T23:28:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd allow primary affinity = true 2024-01-26T23:28:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd allow pg remap = true 2024-01-26T23:28:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on legacy crush tunables = false 2024-01-26T23:28:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on crush straw calc version zero = false 2024-01-26T23:28:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on no sortbitwise = false 2024-01-26T23:28:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on osd down out interval zero = false 2024-01-26T23:28:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on too few osds = false 2024-01-26T23:28:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-26T23:28:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-26T23:28:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: mon_allow_pool_size_one = true 2024-01-26T23:28:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-26T23:28:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: osd default data pool replay window = 5 2024-01-26T23:28:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: mon allow pool delete = true 2024-01-26T23:28:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: mon cluster log file level = debug 2024-01-26T23:28:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: debug asserts on shutdown = true 2024-01-26T23:28:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: mon health detail to clog = false 2024-01-26T23:28:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: mon host = "172.21.15.46,172.21.15.106,[v2:172.21.15.106:3301,v1:172.21.15.106:6790]" 2024-01-26T23:28:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: fsid = 62f24da2-f257-4a3f-ab5a-77a1100d6f1e 2024-01-26T23:28:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.022 INFO:teuthology.orchestra.run.smithi106.stdout:[osd] 2024-01-26T23:28:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: osd journal size = 100 2024-01-26T23:28:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: osd scrub load threshold = 5.0 2024-01-26T23:28:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: osd scrub max interval = 600 2024-01-26T23:28:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: osd mclock profile = high_recovery_ops 2024-01-26T23:28:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: osd recover clone overlap = true 2024-01-26T23:28:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: osd recovery max chunk = 1048576 2024-01-26T23:28:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug shutdown = true 2024-01-26T23:28:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug op order = true 2024-01-26T23:28:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug verify stray on activate = true 2024-01-26T23:28:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: osd open classes on start = true 2024-01-26T23:28:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug pg log writeout = true 2024-01-26T23:28:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: osd deep scrub update digest min age = 30 2024-01-26T23:28:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: osd map max advance = 10 2024-01-26T23:28:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: journal zero on create = true 2024-01-26T23:28:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: filestore ondisk finisher threads = 3 2024-01-26T23:28:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: filestore apply finisher threads = 3 2024-01-26T23:28:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: bdev debug aio = true 2024-01-26T23:28:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug misdirected ops = true 2024-01-26T23:28:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: bluestore block size = 96636764160 2024-01-26T23:28:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: bluestore fsck on mount = True 2024-01-26T23:28:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: debug bluefs = 20 2024-01-26T23:28:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: debug bluestore = 20 2024-01-26T23:28:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-01-26T23:28:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: debug osd = 20 2024-01-26T23:28:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: debug rocksdb = 10 2024-01-26T23:28:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd backfillfull_ratio = 0.85 2024-01-26T23:28:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd full ratio = 0.9 2024-01-26T23:28:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd nearfull ratio = 0.8 2024-01-26T23:28:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: osd failsafe full ratio = 0.95 2024-01-26T23:28:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: osd objectstore = bluestore 2024-01-26T23:28:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: osd op complaint time = 180 2024-01-26T23:28:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.025 INFO:teuthology.orchestra.run.smithi106.stdout:[mgr] 2024-01-26T23:28:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-01-26T23:28:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: debug mgr = 20 2024-01-26T23:28:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: debug mon = 20 2024-01-26T23:28:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: debug auth = 20 2024-01-26T23:28:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: mon reweight min pgs per osd = 4 2024-01-26T23:28:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: mon reweight min bytes per osd = 10 2024-01-26T23:28:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: mgr/telemetry/nag = false 2024-01-26T23:28:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: debug client = 20 2024-01-26T23:28:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: debug finisher = 20 2024-01-26T23:28:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.026 INFO:teuthology.orchestra.run.smithi106.stdout:[mon] 2024-01-26T23:28:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-01-26T23:28:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: debug mon = 20 2024-01-26T23:28:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: debug paxos = 20 2024-01-26T23:28:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: debug auth = 20 2024-01-26T23:28:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: mon data avail warn = 5 2024-01-26T23:28:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: mon mgr mkfs grace = 240 2024-01-26T23:28:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: mon reweight min pgs per osd = 4 2024-01-26T23:28:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd reporter subtree level = osd 2024-01-26T23:28:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd prime pg temp = true 2024-01-26T23:28:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: mon reweight min bytes per osd = 10 2024-01-26T23:28:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-26T23:28:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: auth mon ticket ttl = 660 # 11m 2024-01-26T23:28:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: auth service ticket ttl = 240 # 4m 2024-01-26T23:28:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: # don't complain about insecure global_id in the test suite 2024-01-26T23:28:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-26T23:28:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-26T23:28:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: # 1m isn't quite enough 2024-01-26T23:28:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: mon_down_mkfs_grace = 2m 2024-01-26T23:28:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_filestore_osds = false 2024-01-26T23:28:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: mon op complaint time = 120 2024-01-26T23:28:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-26T23:28:54.028 INFO:teuthology.orchestra.run.smithi106.stdout:[client] 2024-01-26T23:28:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: rgw cache enabled = true 2024-01-26T23:28:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: rgw enable ops log = true 2024-01-26T23:28:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: rgw enable usage log = true 2024-01-26T23:28:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-26T23:28:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-26T23:28:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: client mount timeout = 600 2024-01-26T23:28:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: debug client = 20 2024-01-26T23:28:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-01-26T23:28:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: pid file = "" 2024-01-26T23:28:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: rados mon op timeout = 900 2024-01-26T23:28:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: rados osd op timeout = 900 2024-01-26T23:28:54.029 INFO:teuthology.orchestra.run.smithi106.stdout:[mon.a] 2024-01-26T23:28:54.029 INFO:teuthology.orchestra.run.smithi106.stdout:[mon.b] 2024-01-26T23:28:54.029 INFO:teuthology.orchestra.run.smithi106.stdout:[mon.c] 2024-01-26T23:28:54.029 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.a] 2024-01-26T23:28:54.029 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.c] 2024-01-26T23:28:54.029 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.b] 2024-01-26T23:28:54.029 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.d] 2024-01-26T23:28:54.029 INFO:teuthology.orchestra.run.smithi106.stdout:[mds] 2024-01-26T23:28:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: debug mds = 20 2024-01-26T23:28:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: debug mds balancer = 20 2024-01-26T23:28:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-01-26T23:28:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: mds debug frag = True 2024-01-26T23:28:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: mds debug scatterstat = True 2024-01-26T23:28:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: mds op complaint time = 180 2024-01-26T23:28:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: mds verify scatter = True 2024-01-26T23:28:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: osd op complaint time = 180 2024-01-26T23:28:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: rados mon op timeout = 900 2024-01-26T23:28:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: rados osd op timeout = 900 2024-01-26T23:28:54.031 INFO:teuthology.orchestra.run.smithi046.stdout:[global] 2024-01-26T23:28:54.032 INFO:teuthology.orchestra.run.smithi046.stdout: chdir = "" 2024-01-26T23:28:54.032 INFO:teuthology.orchestra.run.smithi046.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-26T23:28:54.032 INFO:teuthology.orchestra.run.smithi046.stdout: auth supported = cephx 2024-01-26T23:28:54.032 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.032 INFO:teuthology.orchestra.run.smithi046.stdout: filestore xattr use omap = true 2024-01-26T23:28:54.032 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.032 INFO:teuthology.orchestra.run.smithi046.stdout: mon clock drift allowed = 1.000 2024-01-26T23:28:54.032 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.032 INFO:teuthology.orchestra.run.smithi046.stdout: osd crush chooseleaf type = 0 2024-01-26T23:28:54.032 INFO:teuthology.orchestra.run.smithi046.stdout: auth debug = true 2024-01-26T23:28:54.032 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.032 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on old message = true 2024-01-26T23:28:54.032 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on bug = true 2024-01-26T23:28:54.032 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.033 INFO:teuthology.orchestra.run.smithi046.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-26T23:28:54.033 INFO:teuthology.orchestra.run.smithi046.stdout: mon pg warn max object skew = 0 2024-01-26T23:28:54.033 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.033 INFO:teuthology.orchestra.run.smithi046.stdout: # disable pg_autoscaler by default for new pools 2024-01-26T23:28:54.033 INFO:teuthology.orchestra.run.smithi046.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-26T23:28:54.033 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.033 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default size = 2 2024-01-26T23:28:54.033 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.033 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow primary affinity = true 2024-01-26T23:28:54.033 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow pg remap = true 2024-01-26T23:28:54.033 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on legacy crush tunables = false 2024-01-26T23:28:54.033 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on crush straw calc version zero = false 2024-01-26T23:28:54.033 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on no sortbitwise = false 2024-01-26T23:28:54.033 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on osd down out interval zero = false 2024-01-26T23:28:54.034 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on too few osds = false 2024-01-26T23:28:54.034 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-26T23:28:54.034 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-26T23:28:54.034 INFO:teuthology.orchestra.run.smithi046.stdout: mon_allow_pool_size_one = true 2024-01-26T23:28:54.034 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.034 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-26T23:28:54.034 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.034 INFO:teuthology.orchestra.run.smithi046.stdout: osd default data pool replay window = 5 2024-01-26T23:28:54.034 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.034 INFO:teuthology.orchestra.run.smithi046.stdout: mon allow pool delete = true 2024-01-26T23:28:54.034 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.034 INFO:teuthology.orchestra.run.smithi046.stdout: mon cluster log file level = debug 2024-01-26T23:28:54.034 INFO:teuthology.orchestra.run.smithi046.stdout: debug asserts on shutdown = true 2024-01-26T23:28:54.034 INFO:teuthology.orchestra.run.smithi046.stdout: mon health detail to clog = false 2024-01-26T23:28:54.035 INFO:teuthology.orchestra.run.smithi046.stdout: mon host = "172.21.15.46,172.21.15.106,[v2:172.21.15.106:3301,v1:172.21.15.106:6790]" 2024-01-26T23:28:54.035 INFO:teuthology.orchestra.run.smithi046.stdout: fsid = 62f24da2-f257-4a3f-ab5a-77a1100d6f1e 2024-01-26T23:28:54.035 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.035 INFO:teuthology.orchestra.run.smithi046.stdout:[osd] 2024-01-26T23:28:54.035 INFO:teuthology.orchestra.run.smithi046.stdout: osd journal size = 100 2024-01-26T23:28:54.035 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.035 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub load threshold = 5.0 2024-01-26T23:28:54.035 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub max interval = 600 2024-01-26T23:28:54.035 INFO:teuthology.orchestra.run.smithi046.stdout: osd mclock profile = high_recovery_ops 2024-01-26T23:28:54.035 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.035 INFO:teuthology.orchestra.run.smithi046.stdout: osd recover clone overlap = true 2024-01-26T23:28:54.035 INFO:teuthology.orchestra.run.smithi046.stdout: osd recovery max chunk = 1048576 2024-01-26T23:28:54.035 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.035 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug shutdown = true 2024-01-26T23:28:54.036 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug op order = true 2024-01-26T23:28:54.036 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify stray on activate = true 2024-01-26T23:28:54.036 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.036 INFO:teuthology.orchestra.run.smithi046.stdout: osd open classes on start = true 2024-01-26T23:28:54.037 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug pg log writeout = true 2024-01-26T23:28:54.037 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.037 INFO:teuthology.orchestra.run.smithi046.stdout: osd deep scrub update digest min age = 30 2024-01-26T23:28:54.037 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.038 INFO:teuthology.orchestra.run.smithi046.stdout: osd map max advance = 10 2024-01-26T23:28:54.038 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.038 INFO:teuthology.orchestra.run.smithi046.stdout: journal zero on create = true 2024-01-26T23:28:54.038 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.038 INFO:teuthology.orchestra.run.smithi046.stdout: filestore ondisk finisher threads = 3 2024-01-26T23:28:54.038 INFO:teuthology.orchestra.run.smithi046.stdout: filestore apply finisher threads = 3 2024-01-26T23:28:54.038 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.038 INFO:teuthology.orchestra.run.smithi046.stdout: bdev debug aio = true 2024-01-26T23:28:54.038 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug misdirected ops = true 2024-01-26T23:28:54.038 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore block size = 96636764160 2024-01-26T23:28:54.038 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore fsck on mount = True 2024-01-26T23:28:54.038 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluefs = 20 2024-01-26T23:28:54.038 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluestore = 20 2024-01-26T23:28:54.038 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-01-26T23:28:54.038 INFO:teuthology.orchestra.run.smithi046.stdout: debug osd = 20 2024-01-26T23:28:54.039 INFO:teuthology.orchestra.run.smithi046.stdout: debug rocksdb = 10 2024-01-26T23:28:54.039 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd backfillfull_ratio = 0.85 2024-01-26T23:28:54.039 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd full ratio = 0.9 2024-01-26T23:28:54.039 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd nearfull ratio = 0.8 2024-01-26T23:28:54.039 INFO:teuthology.orchestra.run.smithi046.stdout: osd failsafe full ratio = 0.95 2024-01-26T23:28:54.039 INFO:teuthology.orchestra.run.smithi046.stdout: osd objectstore = bluestore 2024-01-26T23:28:54.039 INFO:teuthology.orchestra.run.smithi046.stdout: osd op complaint time = 180 2024-01-26T23:28:54.039 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.039 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr] 2024-01-26T23:28:54.039 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-01-26T23:28:54.039 INFO:teuthology.orchestra.run.smithi046.stdout: debug mgr = 20 2024-01-26T23:28:54.039 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2024-01-26T23:28:54.039 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2024-01-26T23:28:54.039 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2024-01-26T23:28:54.039 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2024-01-26T23:28:54.040 INFO:teuthology.orchestra.run.smithi046.stdout: mgr/telemetry/nag = false 2024-01-26T23:28:54.040 INFO:teuthology.orchestra.run.smithi046.stdout: debug client = 20 2024-01-26T23:28:54.040 INFO:teuthology.orchestra.run.smithi046.stdout: debug finisher = 20 2024-01-26T23:28:54.040 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.040 INFO:teuthology.orchestra.run.smithi046.stdout:[mon] 2024-01-26T23:28:54.040 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-01-26T23:28:54.040 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2024-01-26T23:28:54.040 INFO:teuthology.orchestra.run.smithi046.stdout: debug paxos = 20 2024-01-26T23:28:54.040 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2024-01-26T23:28:54.040 INFO:teuthology.orchestra.run.smithi046.stdout: mon data avail warn = 5 2024-01-26T23:28:54.040 INFO:teuthology.orchestra.run.smithi046.stdout: mon mgr mkfs grace = 240 2024-01-26T23:28:54.040 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2024-01-26T23:28:54.040 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd reporter subtree level = osd 2024-01-26T23:28:54.040 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd prime pg temp = true 2024-01-26T23:28:54.041 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2024-01-26T23:28:54.041 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.041 INFO:teuthology.orchestra.run.smithi046.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-26T23:28:54.041 INFO:teuthology.orchestra.run.smithi046.stdout: auth mon ticket ttl = 660 # 11m 2024-01-26T23:28:54.041 INFO:teuthology.orchestra.run.smithi046.stdout: auth service ticket ttl = 240 # 4m 2024-01-26T23:28:54.041 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.041 INFO:teuthology.orchestra.run.smithi046.stdout: # don't complain about insecure global_id in the test suite 2024-01-26T23:28:54.041 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-26T23:28:54.041 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-26T23:28:54.041 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.041 INFO:teuthology.orchestra.run.smithi046.stdout: # 1m isn't quite enough 2024-01-26T23:28:54.041 INFO:teuthology.orchestra.run.smithi046.stdout: mon_down_mkfs_grace = 2m 2024-01-26T23:28:54.041 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.041 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_filestore_osds = false 2024-01-26T23:28:54.041 INFO:teuthology.orchestra.run.smithi046.stdout: mon op complaint time = 120 2024-01-26T23:28:54.042 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:28:54.042 INFO:teuthology.orchestra.run.smithi046.stdout:[client] 2024-01-26T23:28:54.042 INFO:teuthology.orchestra.run.smithi046.stdout: rgw cache enabled = true 2024-01-26T23:28:54.042 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable ops log = true 2024-01-26T23:28:54.042 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable usage log = true 2024-01-26T23:28:54.042 INFO:teuthology.orchestra.run.smithi046.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-26T23:28:54.042 INFO:teuthology.orchestra.run.smithi046.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-26T23:28:54.042 INFO:teuthology.orchestra.run.smithi046.stdout: client mount timeout = 600 2024-01-26T23:28:54.042 INFO:teuthology.orchestra.run.smithi046.stdout: debug client = 20 2024-01-26T23:28:54.042 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-01-26T23:28:54.042 INFO:teuthology.orchestra.run.smithi046.stdout: pid file = "" 2024-01-26T23:28:54.042 INFO:teuthology.orchestra.run.smithi046.stdout: rados mon op timeout = 900 2024-01-26T23:28:54.042 INFO:teuthology.orchestra.run.smithi046.stdout: rados osd op timeout = 900 2024-01-26T23:28:54.042 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.a] 2024-01-26T23:28:54.043 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.b] 2024-01-26T23:28:54.043 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.c] 2024-01-26T23:28:54.043 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.a] 2024-01-26T23:28:54.043 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.c] 2024-01-26T23:28:54.043 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.b] 2024-01-26T23:28:54.043 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.d] 2024-01-26T23:28:54.043 INFO:teuthology.orchestra.run.smithi046.stdout:[mds] 2024-01-26T23:28:54.043 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds = 20 2024-01-26T23:28:54.043 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds balancer = 20 2024-01-26T23:28:54.043 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-01-26T23:28:54.043 INFO:teuthology.orchestra.run.smithi046.stdout: mds debug frag = True 2024-01-26T23:28:54.043 INFO:teuthology.orchestra.run.smithi046.stdout: mds debug scatterstat = True 2024-01-26T23:28:54.043 INFO:teuthology.orchestra.run.smithi046.stdout: mds op complaint time = 180 2024-01-26T23:28:54.043 INFO:teuthology.orchestra.run.smithi046.stdout: mds verify scatter = True 2024-01-26T23:28:54.043 INFO:teuthology.orchestra.run.smithi046.stdout: osd op complaint time = 180 2024-01-26T23:28:54.044 INFO:teuthology.orchestra.run.smithi046.stdout: rados mon op timeout = 900 2024-01-26T23:28:54.044 INFO:teuthology.orchestra.run.smithi046.stdout: rados osd op timeout = 900 2024-01-26T23:28:54.044 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-26T23:28:54.044 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-26T23:28:54.137 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-26T23:28:54.137 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:28:54.137 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-26T23:28:54.157 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:28:54.157 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-26T23:28:54.219 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:28:54.219 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:28:54.219 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-26T23:28:54.219 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-26T23:28:54.306 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:28:54.306 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-26T23:28:54.368 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:28:54.369 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:28:54.369 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-26T23:28:54.369 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-26T23:28:54.414 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:28:54.414 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-26T23:28:54.474 INFO:tasks.ceph:Setting up mon nodes... 2024-01-26T23:28:54.475 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-26T23:28:54.475 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-01-26T23:28:54.538 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-01-26T23:28:54.541 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-26T23:28:54.610 INFO:teuthology.orchestra.run.smithi106.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-26T23:28:54.612 INFO:tasks.ceph:Setting up mds nodes... 2024-01-26T23:28:54.612 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-01-26T23:28:54.676 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-01-26T23:28:54.678 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-01-26T23:28:54.710 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-01-26T23:28:54.812 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-01-26T23:28:54.815 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-01-26T23:28:54.845 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-01-26T23:28:54.903 INFO:teuthology.orchestra.run.smithi106.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-01-26T23:28:54.905 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-01-26T23:28:54.933 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-01-26T23:28:55.025 INFO:teuthology.orchestra.run.smithi106.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-01-26T23:28:55.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-01-26T23:28:55.053 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-26T23:28:55.053 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-26T23:28:55.104 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-26T23:28:55.119 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-01-26T23:28:55.167 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-01-26T23:28:55.182 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-26T23:28:55.222 INFO:teuthology.orchestra.run.smithi106.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-01-26T23:28:55.235 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-01-26T23:28:55.315 INFO:teuthology.orchestra.run.smithi106.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-01-26T23:28:55.328 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-26T23:28:55.328 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi106.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-26T23:28:55.328 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-26T23:28:55.363 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-26T23:28:55.363 INFO:tasks.ceph:role: osd.0 2024-01-26T23:28:55.363 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:28:55.363 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-26T23:28:55.437 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-26T23:28:55.437 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-26T23:28:55.437 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-26T23:28:55.437 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-26T23:28:55.437 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-26T23:28:55.437 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-01-26T23:28:55.437 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-26T23:28:55.438 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-26T23:28:55.438 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-26T23:28:55.438 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-26T23:28:55.460 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-01-26T23:28:55.503 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-01-26T23:28:55.503 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-26T23:28:55.592 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-26T23:28:55.629 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-26T23:28:55.704 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-26T23:28:55.704 INFO:tasks.ceph:role: osd.1 2024-01-26T23:28:55.704 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:28:55.704 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-26T23:28:55.779 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-26T23:28:55.779 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-26T23:28:55.779 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-26T23:28:55.779 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-26T23:28:55.779 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-26T23:28:55.779 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-01-26T23:28:55.779 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-26T23:28:55.779 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-26T23:28:55.779 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-26T23:28:55.779 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-26T23:28:55.801 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-01-26T23:28:55.803 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-01-26T23:28:55.804 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-26T23:28:55.893 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-26T23:28:55.931 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-26T23:28:56.006 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-26T23:28:56.006 INFO:tasks.ceph:role: osd.2 2024-01-26T23:28:56.006 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:28:56.006 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-26T23:28:56.080 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-26T23:28:56.081 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-26T23:28:56.081 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-26T23:28:56.081 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-26T23:28:56.081 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-26T23:28:56.081 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-01-26T23:28:56.081 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-26T23:28:56.081 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-26T23:28:56.081 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-26T23:28:56.081 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-26T23:28:56.104 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-01-26T23:28:56.106 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-01-26T23:28:56.106 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-26T23:28:56.194 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-26T23:28:56.232 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-01-26T23:28:56.308 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-26T23:28:56.308 INFO:tasks.ceph:role: osd.3 2024-01-26T23:28:56.308 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:28:56.308 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-26T23:28:56.383 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-26T23:28:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-26T23:28:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-26T23:28:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-26T23:28:56.383 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-26T23:28:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-01-26T23:28:56.383 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-26T23:28:56.383 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-26T23:28:56.383 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-26T23:28:56.383 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-26T23:28:56.405 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-01-26T23:28:56.407 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-01-26T23:28:56.408 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-01-26T23:28:56.496 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-01-26T23:28:56.533 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-26T23:28:56.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:28:56.628+0000 7fb170bb3540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-26T23:28:56.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:28:56.628+0000 7fb170bb3540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-26T23:28:56.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:28:56.628+0000 7fb170bb3540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-26T23:28:56.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:28:56.628+0000 7fb170bb3540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-26T23:28:56.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:28:56.628+0000 7fb170bb3540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-26T23:28:56.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:28:56.629+0000 7fb170bb3540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-26T23:28:57.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:28:57.162+0000 7fb170bb3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:28:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:28:57.923+0000 7fb170bb3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:28:58.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:28:58.691+0000 7fb170bb3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:28:59.494 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-26T23:28:59.529 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-26T23:28:59.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:28:59.623+0000 7f137815c540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-26T23:28:59.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:28:59.623+0000 7f137815c540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-26T23:28:59.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:28:59.623+0000 7f137815c540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-26T23:28:59.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:28:59.623+0000 7f137815c540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-26T23:28:59.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:28:59.623+0000 7f137815c540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-26T23:28:59.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:28:59.624+0000 7f137815c540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-26T23:29:00.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:00.151+0000 7f137815c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:00.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:00.922+0000 7f137815c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:01.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:01.684+0000 7f137815c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:02.483 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-26T23:29:02.518 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-26T23:29:02.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:02.612+0000 7f36cb152540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-26T23:29:02.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:02.612+0000 7f36cb152540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-26T23:29:02.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:02.612+0000 7f36cb152540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-26T23:29:02.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:02.612+0000 7f36cb152540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-26T23:29:02.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:02.612+0000 7f36cb152540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-26T23:29:02.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:02.613+0000 7f36cb152540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-26T23:29:03.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:03.141+0000 7f36cb152540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:03.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:03.902+0000 7f36cb152540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:04.665+0000 7f36cb152540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:05.461 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-26T23:29:05.496 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-26T23:29:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:05.590+0000 7f8327fb8540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-01-26T23:29:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:05.590+0000 7f8327fb8540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-01-26T23:29:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:05.590+0000 7f8327fb8540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-26T23:29:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:05.590+0000 7f8327fb8540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-26T23:29:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:05.590+0000 7f8327fb8540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-26T23:29:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:05.590+0000 7f8327fb8540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-01-26T23:29:06.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:06.115+0000 7f8327fb8540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:06.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:06.876+0000 7f8327fb8540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:07.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:07.642+0000 7f8327fb8540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:08.446 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-01-26T23:29:08.480 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-01-26T23:29:08.517 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-26T23:29:08.518 INFO:tasks.ceph:role: osd.4 2024-01-26T23:29:08.518 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:29:08.518 DEBUG:teuthology.orchestra.run.smithi106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-26T23:29:08.598 INFO:teuthology.orchestra.run.smithi106.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-26T23:29:08.598 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-26T23:29:08.598 INFO:teuthology.orchestra.run.smithi106.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-26T23:29:08.598 INFO:teuthology.orchestra.run.smithi106.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-26T23:29:08.598 INFO:teuthology.orchestra.run.smithi106.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-26T23:29:08.598 INFO:teuthology.orchestra.run.smithi106.stdout: = sunit=0 swidth=0 blks 2024-01-26T23:29:08.598 INFO:teuthology.orchestra.run.smithi106.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-26T23:29:08.598 INFO:teuthology.orchestra.run.smithi106.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-26T23:29:08.598 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-26T23:29:08.599 INFO:teuthology.orchestra.run.smithi106.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-26T23:29:09.032 INFO:teuthology.orchestra.run.smithi106.stdout:Discarding blocks...Done. 2024-01-26T23:29:09.034 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi106.front.sepia.ceph.com -o noatime 2024-01-26T23:29:09.035 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-01-26T23:29:09.219 DEBUG:teuthology.orchestra.run.smithi106:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-01-26T23:29:09.256 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-01-26T23:29:09.325 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-26T23:29:09.325 INFO:tasks.ceph:role: osd.5 2024-01-26T23:29:09.325 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:29:09.326 DEBUG:teuthology.orchestra.run.smithi106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-26T23:29:09.401 INFO:teuthology.orchestra.run.smithi106.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-26T23:29:09.402 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-26T23:29:09.402 INFO:teuthology.orchestra.run.smithi106.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-26T23:29:09.402 INFO:teuthology.orchestra.run.smithi106.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-26T23:29:09.402 INFO:teuthology.orchestra.run.smithi106.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-26T23:29:09.403 INFO:teuthology.orchestra.run.smithi106.stdout: = sunit=0 swidth=0 blks 2024-01-26T23:29:09.403 INFO:teuthology.orchestra.run.smithi106.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-26T23:29:09.403 INFO:teuthology.orchestra.run.smithi106.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-26T23:29:09.403 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-26T23:29:09.403 INFO:teuthology.orchestra.run.smithi106.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-26T23:29:09.821 INFO:teuthology.orchestra.run.smithi106.stdout:Discarding blocks...Done. 2024-01-26T23:29:09.822 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi106.front.sepia.ceph.com -o noatime 2024-01-26T23:29:09.823 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-01-26T23:29:09.929 DEBUG:teuthology.orchestra.run.smithi106:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-01-26T23:29:09.960 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-01-26T23:29:10.027 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-26T23:29:10.028 INFO:tasks.ceph:role: osd.6 2024-01-26T23:29:10.028 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:29:10.028 DEBUG:teuthology.orchestra.run.smithi106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-26T23:29:10.103 INFO:teuthology.orchestra.run.smithi106.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-26T23:29:10.103 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-26T23:29:10.103 INFO:teuthology.orchestra.run.smithi106.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-26T23:29:10.103 INFO:teuthology.orchestra.run.smithi106.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-26T23:29:10.103 INFO:teuthology.orchestra.run.smithi106.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-26T23:29:10.103 INFO:teuthology.orchestra.run.smithi106.stdout: = sunit=0 swidth=0 blks 2024-01-26T23:29:10.103 INFO:teuthology.orchestra.run.smithi106.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-26T23:29:10.103 INFO:teuthology.orchestra.run.smithi106.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-26T23:29:10.103 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-26T23:29:10.103 INFO:teuthology.orchestra.run.smithi106.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-26T23:29:10.554 INFO:teuthology.orchestra.run.smithi106.stdout:Discarding blocks...Done. 2024-01-26T23:29:10.556 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi106.front.sepia.ceph.com -o noatime 2024-01-26T23:29:10.556 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-01-26T23:29:10.727 DEBUG:teuthology.orchestra.run.smithi106:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-01-26T23:29:10.764 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-01-26T23:29:10.834 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-26T23:29:10.834 INFO:tasks.ceph:role: osd.7 2024-01-26T23:29:10.834 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:29:10.835 DEBUG:teuthology.orchestra.run.smithi106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-26T23:29:10.911 INFO:teuthology.orchestra.run.smithi106.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-26T23:29:10.911 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-26T23:29:10.911 INFO:teuthology.orchestra.run.smithi106.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-26T23:29:10.911 INFO:teuthology.orchestra.run.smithi106.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-26T23:29:10.911 INFO:teuthology.orchestra.run.smithi106.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-26T23:29:10.911 INFO:teuthology.orchestra.run.smithi106.stdout: = sunit=0 swidth=0 blks 2024-01-26T23:29:10.911 INFO:teuthology.orchestra.run.smithi106.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-26T23:29:10.911 INFO:teuthology.orchestra.run.smithi106.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-26T23:29:10.911 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-26T23:29:10.911 INFO:teuthology.orchestra.run.smithi106.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-26T23:29:11.349 INFO:teuthology.orchestra.run.smithi106.stdout:Discarding blocks...Done. 2024-01-26T23:29:11.350 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi106.front.sepia.ceph.com -o noatime 2024-01-26T23:29:11.351 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-01-26T23:29:11.529 DEBUG:teuthology.orchestra.run.smithi106:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-01-26T23:29:11.567 DEBUG:teuthology.orchestra.run.smithi106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-26T23:29:11.658 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:11.657+0000 7f13f2e38540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-01-26T23:29:11.658 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:11.657+0000 7f13f2e38540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-01-26T23:29:11.658 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:11.657+0000 7f13f2e38540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-26T23:29:11.658 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:11.657+0000 7f13f2e38540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-26T23:29:11.658 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:11.657+0000 7f13f2e38540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-26T23:29:11.658 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:11.658+0000 7f13f2e38540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-01-26T23:29:12.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:12.185+0000 7f13f2e38540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:12.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:12.953+0000 7f13f2e38540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:13.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:13.715+0000 7f13f2e38540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:14.512 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-01-26T23:29:14.543 DEBUG:teuthology.orchestra.run.smithi106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-26T23:29:14.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:14.630+0000 7f8c5ba3e540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-01-26T23:29:14.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:14.630+0000 7f8c5ba3e540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-01-26T23:29:14.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:14.631+0000 7f8c5ba3e540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-26T23:29:14.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:14.631+0000 7f8c5ba3e540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-26T23:29:14.632 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:14.631+0000 7f8c5ba3e540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-26T23:29:14.633 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:14.631+0000 7f8c5ba3e540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-01-26T23:29:15.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:15.155+0000 7f8c5ba3e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:15.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:15.921+0000 7f8c5ba3e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:16.689 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:16.687+0000 7f8c5ba3e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:17.494 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-01-26T23:29:17.526 DEBUG:teuthology.orchestra.run.smithi106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-26T23:29:17.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:17.618+0000 7f86e5869540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-01-26T23:29:17.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:17.618+0000 7f86e5869540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-01-26T23:29:17.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:17.618+0000 7f86e5869540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-26T23:29:17.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:17.618+0000 7f86e5869540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-26T23:29:17.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:17.618+0000 7f86e5869540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-26T23:29:17.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:17.619+0000 7f86e5869540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-01-26T23:29:18.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:18.148+0000 7f86e5869540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:18.919 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:18.917+0000 7f86e5869540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:19.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:19.680+0000 7f86e5869540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:20.482 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-01-26T23:29:20.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-26T23:29:20.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:20.600+0000 7fcb2d5db540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-01-26T23:29:20.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:20.601+0000 7fcb2d5db540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-01-26T23:29:20.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:20.601+0000 7fcb2d5db540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-26T23:29:20.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:20.601+0000 7fcb2d5db540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-26T23:29:20.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:20.601+0000 7fcb2d5db540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-26T23:29:20.603 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:20.601+0000 7fcb2d5db540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-01-26T23:29:21.126 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:21.124+0000 7fcb2d5db540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:21.828 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:29:21.831 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:29:21.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:21.904+0000 7fcb2d5db540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:22.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:22.670+0000 7fcb2d5db540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-26T23:29:23.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-01-26T23:29:23.538 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-26T23:29:23.538 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:29:23.538 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-01-26T23:29:23.573 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:29:23.573 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-01-26T23:29:23.647 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:29:23.647 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-01-26T23:29:23.722 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:29:23.722 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-26T23:29:23.797 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:29:23.797 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-26T23:29:23.871 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:29:23.871 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-26T23:29:23.946 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:29:23.946 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-01-26T23:29:24.020 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:29:24.021 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-26T23:29:24.053 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:29:24.053 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-01-26T23:29:24.124 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:29:24.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-01-26T23:29:24.195 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:29:24.195 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-01-26T23:29:24.264 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:29:24.264 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-01-26T23:29:24.333 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:29:24.333 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-01-26T23:29:24.401 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:29:24.401 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-01-26T23:29:24.469 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:29:24.469 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-26T23:29:24.491 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:29:24.491 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-01-26T23:29:24.552 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:29:24.552 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-01-26T23:29:24.569 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:29:24.569 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-01-26T23:29:24.626 INFO:tasks.ceph:Adding keys to all mons... 2024-01-26T23:29:24.626 DEBUG:teuthology.orchestra.run.smithi046:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-26T23:29:24.629 DEBUG:teuthology.orchestra.run.smithi106:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-26T23:29:24.669 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr.y] 2024-01-26T23:29:24.670 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2QLRlI9EbIBAACTpzAO+8A/VFJZcT/WWBng== 2024-01-26T23:29:24.670 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.a] 2024-01-26T23:29:24.670 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2QLRlM6dKKBAAPpeWeDZWrFgc/4zuxKCvHw== 2024-01-26T23:29:24.670 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.c] 2024-01-26T23:29:24.670 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2QLRlMZxoMBAAG5kyfm+LLG86y/p2A8BfDQ== 2024-01-26T23:29:24.670 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.0] 2024-01-26T23:29:24.670 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA4QLRlJ2OKJRAAWVhxEdQkUW20294w/KLL6g== 2024-01-26T23:29:24.670 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.1] 2024-01-26T23:29:24.670 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA7QLRlE2tAJRAAzNlTiCr6a2eWiabRy1SqSw== 2024-01-26T23:29:24.670 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.2] 2024-01-26T23:29:24.670 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA+QLRlyySaJBAA5072ba5HUNi2dYE3D02M/Q== 2024-01-26T23:29:24.670 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.3] 2024-01-26T23:29:24.670 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBBQLRl9+tAIxAApZk0btXIpQPWSb3BYUYsFQ== 2024-01-26T23:29:24.670 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr.x] 2024-01-26T23:29:24.671 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2QLRlhMNSJBAA38QU6DrCvFjO+u9zQ8bkEA== 2024-01-26T23:29:24.671 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.b] 2024-01-26T23:29:24.671 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2QLRlmzHRNRAAflWYZ/p4AXeMy54tvLpugA== 2024-01-26T23:29:24.671 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.d] 2024-01-26T23:29:24.671 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA3QLRl5jx/ARAA3cwtLAOWQ/n2NG+xwEZ3QQ== 2024-01-26T23:29:24.671 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.4] 2024-01-26T23:29:24.671 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBHQLRlurUtJxAAT9fwKrAWl62VOXwShg7NJg== 2024-01-26T23:29:24.671 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.5] 2024-01-26T23:29:24.671 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBKQLRl7aqoJRAAXu9fykWELsW3SwnahoIOLA== 2024-01-26T23:29:24.671 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.6] 2024-01-26T23:29:24.671 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBNQLRlWf7pJBAAopmFLIKb8s32YZs46VwbGg== 2024-01-26T23:29:24.671 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.7] 2024-01-26T23:29:24.671 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBQQLRlNGfhIxAAmCgmOB3QAGfSIYHBb5BUZw== 2024-01-26T23:29:24.671 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-01-26T23:29:24.671 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA3QLRleO8wBhAA1018AcO3KcyOZxgB0Itxaw== 2024-01-26T23:29:24.672 INFO:teuthology.orchestra.run.smithi046.stdout:[client.2] 2024-01-26T23:29:24.672 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA3QLRl6YT1CRAApXlN4j20xzwda0tDMy94Eg== 2024-01-26T23:29:24.672 INFO:teuthology.orchestra.run.smithi046.stdout:[client.1] 2024-01-26T23:29:24.672 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA3QLRlsiZEDRAAjXSamHe2bZcDqVsqXqIcGg== 2024-01-26T23:29:24.672 INFO:teuthology.orchestra.run.smithi046.stdout:[client.3] 2024-01-26T23:29:24.672 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA3QLRlHvDQEhAAxMhgWJsZC5/6Af5zFV6XIA== 2024-01-26T23:29:24.692 INFO:teuthology.orchestra.run.smithi106.stdout:[mgr.y] 2024-01-26T23:29:24.692 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQA2QLRlI9EbIBAACTpzAO+8A/VFJZcT/WWBng== 2024-01-26T23:29:24.692 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.a] 2024-01-26T23:29:24.692 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQA2QLRlM6dKKBAAPpeWeDZWrFgc/4zuxKCvHw== 2024-01-26T23:29:24.692 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.c] 2024-01-26T23:29:24.692 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQA2QLRlMZxoMBAAG5kyfm+LLG86y/p2A8BfDQ== 2024-01-26T23:29:24.692 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.0] 2024-01-26T23:29:24.692 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQA4QLRlJ2OKJRAAWVhxEdQkUW20294w/KLL6g== 2024-01-26T23:29:24.692 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.1] 2024-01-26T23:29:24.692 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQA7QLRlE2tAJRAAzNlTiCr6a2eWiabRy1SqSw== 2024-01-26T23:29:24.692 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.2] 2024-01-26T23:29:24.693 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQA+QLRlyySaJBAA5072ba5HUNi2dYE3D02M/Q== 2024-01-26T23:29:24.693 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.3] 2024-01-26T23:29:24.693 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQBBQLRl9+tAIxAApZk0btXIpQPWSb3BYUYsFQ== 2024-01-26T23:29:24.693 INFO:teuthology.orchestra.run.smithi106.stdout:[mgr.x] 2024-01-26T23:29:24.693 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQA2QLRlhMNSJBAA38QU6DrCvFjO+u9zQ8bkEA== 2024-01-26T23:29:24.693 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.b] 2024-01-26T23:29:24.693 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQA2QLRlmzHRNRAAflWYZ/p4AXeMy54tvLpugA== 2024-01-26T23:29:24.693 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.d] 2024-01-26T23:29:24.693 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQA3QLRl5jx/ARAA3cwtLAOWQ/n2NG+xwEZ3QQ== 2024-01-26T23:29:24.693 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.4] 2024-01-26T23:29:24.693 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQBHQLRlurUtJxAAT9fwKrAWl62VOXwShg7NJg== 2024-01-26T23:29:24.693 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.5] 2024-01-26T23:29:24.693 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQBKQLRl7aqoJRAAXu9fykWELsW3SwnahoIOLA== 2024-01-26T23:29:24.693 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.6] 2024-01-26T23:29:24.693 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQBNQLRlWf7pJBAAopmFLIKb8s32YZs46VwbGg== 2024-01-26T23:29:24.694 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.7] 2024-01-26T23:29:24.694 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQBQQLRlNGfhIxAAmCgmOB3QAGfSIYHBb5BUZw== 2024-01-26T23:29:24.694 INFO:teuthology.orchestra.run.smithi106.stdout:[client.0] 2024-01-26T23:29:24.694 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQA3QLRleO8wBhAA1018AcO3KcyOZxgB0Itxaw== 2024-01-26T23:29:24.694 INFO:teuthology.orchestra.run.smithi106.stdout:[client.2] 2024-01-26T23:29:24.694 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQA3QLRl6YT1CRAApXlN4j20xzwda0tDMy94Eg== 2024-01-26T23:29:24.694 INFO:teuthology.orchestra.run.smithi106.stdout:[client.1] 2024-01-26T23:29:24.694 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQA3QLRlsiZEDRAAjXSamHe2bZcDqVsqXqIcGg== 2024-01-26T23:29:24.694 INFO:teuthology.orchestra.run.smithi106.stdout:[client.3] 2024-01-26T23:29:24.694 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQA3QLRlHvDQEhAAxMhgWJsZC5/6Af5zFV6XIA== 2024-01-26T23:29:24.695 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-26T23:29:24.716 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-26T23:29:24.784 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-26T23:29:24.787 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-26T23:29:24.846 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-26T23:29:24.848 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-26T23:29:24.906 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-26T23:29:24.908 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-26T23:29:24.962 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-26T23:29:24.965 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-26T23:29:25.019 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-26T23:29:25.021 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-26T23:29:25.074 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-26T23:29:25.076 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-26T23:29:25.129 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-26T23:29:25.131 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-26T23:29:25.184 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-26T23:29:25.187 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-26T23:29:25.243 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-26T23:29:25.245 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-26T23:29:25.301 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-26T23:29:25.303 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-26T23:29:25.357 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-26T23:29:25.359 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-26T23:29:25.412 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-26T23:29:25.415 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-26T23:29:25.468 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-26T23:29:25.470 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-26T23:29:25.523 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-26T23:29:25.525 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-26T23:29:25.581 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-26T23:29:25.583 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-26T23:29:25.640 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-26T23:29:25.643 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-26T23:29:25.699 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-26T23:29:25.701 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-26T23:29:25.757 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-26T23:29:25.757 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-26T23:29:25.787 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-26T23:29:25.890 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-26T23:29:25.921 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-01-26T23:29:25.948 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-26T23:29:26.043 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-01-26T23:29:26.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-01-26T23:29:26.136 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-26T23:29:26.231 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-01-26T23:29:26.257 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-26T23:29:26.260 DEBUG:teuthology.orchestra.run.smithi106:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-26T23:29:26.314 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-26T23:29:26.315 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-26T23:29:26.315 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-26T23:29:26.321 INFO:tasks.ceph.mon.a:Started 2024-01-26T23:29:26.321 INFO:tasks.ceph.mon.b:Restarting daemon 2024-01-26T23:29:26.321 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-01-26T23:29:26.357 INFO:tasks.ceph.mon.b:Started 2024-01-26T23:29:26.357 INFO:tasks.ceph.mon.c:Restarting daemon 2024-01-26T23:29:26.357 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-01-26T23:29:26.359 INFO:tasks.ceph.mon.c:Started 2024-01-26T23:29:26.359 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-26T23:29:26.359 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-01-26T23:29:26.359 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-01-26T23:29:26.362 INFO:tasks.ceph.mgr.y:Started 2024-01-26T23:29:26.362 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-26T23:29:26.362 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-26T23:29:26.365 INFO:tasks.ceph.mgr.x:Started 2024-01-26T23:29:26.365 INFO:tasks.ceph:Setting crush tunables to default 2024-01-26T23:29:26.365 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd crush tunables default 2024-01-26T23:29:26.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.502+0000 7f1b550b9700 1 Processor -- start 2024-01-26T23:29:26.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.502+0000 7f1b550b9700 1 -- start start 2024-01-26T23:29:26.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.503+0000 7f1b550b9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1b500c6700 0x7f1b500cf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:26.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.503+0000 7f1b550b9700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b500cce40 con 0x7f1b500cd360 2024-01-26T23:29:26.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.503+0000 7f1b550b9700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b500cc230 con 0x7f1b500cd730 2024-01-26T23:29:26.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.503+0000 7f1b550b9700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b500cdb90 con 0x7f1b500c6700 2024-01-26T23:29:26.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.504+0000 7f1b4ed9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1b500cd360 0x7f1b500cef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57334/0 (socket says 172.21.15.46:57334) 2024-01-26T23:29:26.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.504+0000 7f1b4ed9d700 1 -- 172.21.15.46:0/465246352 learned_addr learned my addr 172.21.15.46:0/465246352 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:26.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.504+0000 7f1b4e59c700 1 --2- 172.21.15.46:0/465246352 >> v2:172.21.15.106:3300/0 conn(0x7f1b500c6700 0x7f1b500cf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:26.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.504+0000 7f1b4e59c700 1 -- 172.21.15.46:0/465246352 >> v2:172.21.15.106:3300/0 conn(0x7f1b500c6700 msgr2=0x7f1b500cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-26T23:29:26.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.504+0000 7f1b4e59c700 1 -- 172.21.15.46:0/465246352 >> v2:172.21.15.106:3300/0 conn(0x7f1b500c6700 msgr2=0x7f1b500cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-26T23:29:26.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.504+0000 7f1b4e59c700 1 --2- 172.21.15.46:0/465246352 >> v2:172.21.15.106:3300/0 conn(0x7f1b500c6700 0x7f1b500cf680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-26T23:29:26.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.504+0000 7f1b4e59c700 1 --2- 172.21.15.46:0/465246352 >> v2:172.21.15.106:3300/0 conn(0x7f1b500c6700 0x7f1b500cf680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:29:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.697+0000 7f1b4dd9b700 1 -- 172.21.15.46:0/465246352 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459986842 0 0) 0x7f1b500cce40 con 0x7f1b500cd360 2024-01-26T23:29:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.697+0000 7f1b4dd9b700 1 -- 172.21.15.46:0/465246352 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b38003570 con 0x7f1b500cd360 2024-01-26T23:29:26.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.698+0000 7f1b4dd9b700 1 -- 172.21.15.46:0/465246352 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4269451015 0 0) 0x7f1b38003570 con 0x7f1b500cd360 2024-01-26T23:29:26.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.698+0000 7f1b4dd9b700 1 -- 172.21.15.46:0/465246352 >> v2:172.21.15.106:3300/0 conn(0x7f1b500c6700 msgr2=0x7f1b500cf680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-26T23:29:26.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.698+0000 7f1b4dd9b700 1 --2- 172.21.15.46:0/465246352 >> v2:172.21.15.106:3300/0 conn(0x7f1b500c6700 0x7f1b500cf680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:26.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.698+0000 7f1b4dd9b700 1 -- 172.21.15.46:0/465246352 >> v1:172.21.15.106:6789/0 conn(0x7f1b500cd730 legacy=0x7f1b50162f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:26.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.698+0000 7f1b4dd9b700 1 -- 172.21.15.46:0/465246352 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b500cc510 con 0x7f1b500cd360 2024-01-26T23:29:26.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.698+0000 7f1b4dd9b700 1 -- 172.21.15.46:0/465246352 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f1b400015f0 con 0x7f1b500cd360 2024-01-26T23:29:26.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.698+0000 7f1b4dd9b700 1 -- 172.21.15.46:0/465246352 >> v1:172.21.15.46:6789/0 conn(0x7f1b500cd360 legacy=0x7f1b500cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:26.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.698+0000 7f1b4dd9b700 1 --2- 172.21.15.46:0/465246352 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b38003ab0 0x7f1b38003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:26.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.698+0000 7f1b4dd9b700 1 --2- 172.21.15.46:0/465246352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b38004810 0x7f1b38004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:26.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.698+0000 7f1b4dd9b700 1 --2- 172.21.15.46:0/465246352 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b380053c0 0x7f1b3800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.698+0000 7f1b4dd9b700 1 -- 172.21.15.46:0/465246352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3800c2d0 con 0x7f1b38004810 2024-01-26T23:29:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.698+0000 7f1b4dd9b700 1 -- 172.21.15.46:0/465246352 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3800c440 con 0x7f1b38003ab0 2024-01-26T23:29:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.698+0000 7f1b4dd9b700 1 -- 172.21.15.46:0/465246352 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3800c600 con 0x7f1b380053c0 2024-01-26T23:29:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.698+0000 7f1b4ed9d700 1 --2- 172.21.15.46:0/465246352 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b38003ab0 0x7f1b38003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.698+0000 7f1b4e59c700 1 --2- 172.21.15.46:0/465246352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b38004810 0x7f1b38004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.698+0000 7f1b4f59e700 1 --2- 172.21.15.46:0/465246352 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b380053c0 0x7f1b3800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.699+0000 7f1b4e59c700 1 -- 172.21.15.46:0/465246352 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b380053c0 msgr2=0x7f1b3800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.699+0000 7f1b4e59c700 1 --2- 172.21.15.46:0/465246352 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b380053c0 0x7f1b3800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.699+0000 7f1b4e59c700 1 -- 172.21.15.46:0/465246352 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b38003ab0 msgr2=0x7f1b38003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.699+0000 7f1b4e59c700 1 --2- 172.21.15.46:0/465246352 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b38003ab0 0x7f1b38003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.699+0000 7f1b4e59c700 1 -- 172.21.15.46:0/465246352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b38003570 con 0x7f1b38004810 2024-01-26T23:29:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.699+0000 7f1b4f59e700 1 -- 172.21.15.46:0/465246352 reap_dead start 2024-01-26T23:29:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.699+0000 7f1b4ed9d700 1 --2- 172.21.15.46:0/465246352 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b38003ab0 0x7f1b38003ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:29:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.699+0000 7f1b4e59c700 1 --2- 172.21.15.46:0/465246352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b38004810 0x7f1b38004c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400bb40 tx=0x7f1b44008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=636d087588da40c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.699+0000 7f1b4dd9b700 1 -- 172.21.15.46:0/465246352 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b44018070 con 0x7f1b38004810 2024-01-26T23:29:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.699+0000 7f1b4dd9b700 1 -- 172.21.15.46:0/465246352 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b4400d5d0 con 0x7f1b38004810 2024-01-26T23:29:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.699+0000 7f1b4dd9b700 1 -- 172.21.15.46:0/465246352 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4400d990 con 0x7f1b38004810 2024-01-26T23:29:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.700+0000 7f1b550b9700 1 -- 172.21.15.46:0/465246352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b38004810 msgr2=0x7f1b38004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.700+0000 7f1b550b9700 1 --2- 172.21.15.46:0/465246352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b38004810 0x7f1b38004c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400bb40 tx=0x7f1b44008040 comp rx=0 tx=0).stop 2024-01-26T23:29:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.700+0000 7f1b550b9700 1 -- 172.21.15.46:0/465246352 shutdown_connections 2024-01-26T23:29:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.700+0000 7f1b550b9700 1 --2- 172.21.15.46:0/465246352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b38004810 0x7f1b38004c80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.700+0000 7f1b550b9700 1 -- 172.21.15.46:0/465246352 >> 172.21.15.46:0/465246352 conn(0x7f1b50099ae0 msgr2=0x7f1b5006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.700+0000 7f1b550b9700 1 -- 172.21.15.46:0/465246352 shutdown_connections 2024-01-26T23:29:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.700+0000 7f1b550b9700 1 -- 172.21.15.46:0/465246352 wait complete. 2024-01-26T23:29:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.702+0000 7f1b550b9700 1 Processor -- start 2024-01-26T23:29:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.702+0000 7f1b550b9700 1 -- start start 2024-01-26T23:29:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.703+0000 7f1b550b9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b38004810 0x7f1b501578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.703+0000 7f1b550b9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b500c8ef0 0x7f1b500c9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.703+0000 7f1b550b9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b500c6bc0 0x7f1b500c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.703+0000 7f1b550b9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b500c98a0 con 0x7f1b38004810 2024-01-26T23:29:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.703+0000 7f1b550b9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b500c7580 con 0x7f1b500c8ef0 2024-01-26T23:29:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.703+0000 7f1b550b9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5016d710 con 0x7f1b500c6bc0 2024-01-26T23:29:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.703+0000 7f1b4ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b38004810 0x7f1b501578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.703+0000 7f1b4ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b38004810 0x7f1b501578d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46912/0 (socket says 172.21.15.46:46912) 2024-01-26T23:29:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.703+0000 7f1b4ed9d700 1 -- 172.21.15.46:0/2463100656 learned_addr learned my addr 172.21.15.46:0/2463100656 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.703+0000 7f1b4e59c700 1 --2- 172.21.15.46:0/2463100656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b500c8ef0 0x7f1b500c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.704+0000 7f1b4f59e700 1 --2- 172.21.15.46:0/2463100656 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b500c6bc0 0x7f1b500c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.704+0000 7f1b4ed9d700 1 -- 172.21.15.46:0/2463100656 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b500c6bc0 msgr2=0x7f1b500c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.704+0000 7f1b4ed9d700 1 --2- 172.21.15.46:0/2463100656 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b500c6bc0 0x7f1b500c7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.704+0000 7f1b4ed9d700 1 -- 172.21.15.46:0/2463100656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b500c8ef0 msgr2=0x7f1b500c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.704+0000 7f1b4ed9d700 1 --2- 172.21.15.46:0/2463100656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b500c8ef0 0x7f1b500c9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.704+0000 7f1b4ed9d700 1 -- 172.21.15.46:0/2463100656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5016ce50 con 0x7f1b38004810 2024-01-26T23:29:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.704+0000 7f1b4e59c700 1 --2- 172.21.15.46:0/2463100656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b500c8ef0 0x7f1b500c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:29:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.704+0000 7f1b4ed9d700 1 --2- 172.21.15.46:0/2463100656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b38004810 0x7f1b501578d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1b4000fee0 tx=0x7f1b40009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.704+0000 7f1b36ffd700 1 -- 172.21.15.46:0/2463100656 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4000d920 con 0x7f1b38004810 2024-01-26T23:29:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.705+0000 7f1b550b9700 1 -- 172.21.15.46:0/2463100656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b501589d0 con 0x7f1b38004810 2024-01-26T23:29:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.705+0000 7f1b550b9700 1 -- 172.21.15.46:0/2463100656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5016d3a0 con 0x7f1b38004810 2024-01-26T23:29:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.706+0000 7f1b550b9700 1 -- 172.21.15.46:0/2463100656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b5004c710 con 0x7f1b38004810 2024-01-26T23:29:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.707+0000 7f1b36ffd700 1 -- 172.21.15.46:0/2463100656 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b4000dc90 con 0x7f1b38004810 2024-01-26T23:29:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.707+0000 7f1b36ffd700 1 -- 172.21.15.46:0/2463100656 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4000de00 con 0x7f1b38004810 2024-01-26T23:29:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.707+0000 7f1b36ffd700 1 -- 172.21.15.46:0/2463100656 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1b400070d0 con 0x7f1b38004810 2024-01-26T23:29:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.707+0000 7f1b36ffd700 1 -- 172.21.15.46:0/2463100656 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1b40005ca0 con 0x7f1b38004810 2024-01-26T23:29:26.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.710+0000 7f1b36ffd700 1 -- 172.21.15.46:0/2463100656 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f1b40007380 con 0x7f1b38004810 2024-01-26T23:29:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.800+0000 7f1b550b9700 1 -- 172.21.15.46:0/2463100656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f1b50068330 con 0x7f1b38004810 2024-01-26T23:29:26.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.803+0000 7f1b36ffd700 1 -- 172.21.15.46:0/2463100656 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1b4002b090 con 0x7f1b38004810 2024-01-26T23:29:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.804+0000 7f1b550b9700 1 -- 172.21.15.46:0/2463100656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b38004810 msgr2=0x7f1b501578d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.804+0000 7f1b550b9700 1 --2- 172.21.15.46:0/2463100656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b38004810 0x7f1b501578d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1b4000fee0 tx=0x7f1b40009040 comp rx=0 tx=0).stop 2024-01-26T23:29:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.805+0000 7f1b550b9700 1 -- 172.21.15.46:0/2463100656 shutdown_connections 2024-01-26T23:29:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.805+0000 7f1b550b9700 1 --2- 172.21.15.46:0/2463100656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b38004810 0x7f1b501578d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.805+0000 7f1b550b9700 1 --2- 172.21.15.46:0/2463100656 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b500c6bc0 0x7f1b500c7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.805+0000 7f1b550b9700 1 --2- 172.21.15.46:0/2463100656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b500c8ef0 0x7f1b500c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.805+0000 7f1b550b9700 1 -- 172.21.15.46:0/2463100656 >> 172.21.15.46:0/2463100656 conn(0x7f1b50099ae0 msgr2=0x7f1b5015b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.805+0000 7f1b550b9700 1 -- 172.21.15.46:0/2463100656 shutdown_connections 2024-01-26T23:29:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:26.805+0000 7f1b550b9700 1 -- 172.21.15.46:0/2463100656 wait complete. 2024-01-26T23:29:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:adjusted tunables profile to default 2024-01-26T23:29:26.825 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-26T23:29:26.825 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:29:26.826 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-26T23:29:26.856 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:29:26.857 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-26T23:29:26.926 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:29:26.927 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-26T23:29:26.996 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-26T23:29:26.996 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-01-26T23:29:27.051 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:27.050+0000 7efc42e4f1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:29:27.065 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:29:27.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-01-26T23:29:27.092 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:29:27.092 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-01-26T23:29:27.116 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:27.115+0000 7f507f3191c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:29:27.131 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:27.130+0000 7efc42e4f1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:29:27.159 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:29:27.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-01-26T23:29:27.204 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:27.202+0000 7f507f3191c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:29:27.208 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:27.207+0000 7efc42e4f1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:29:27.225 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-26T23:29:27.225 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-01-26T23:29:27.279 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:27.278+0000 7efc42e4f1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:29:27.284 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:27.283+0000 7f507f3191c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:29:27.291 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd new 66210791-18e9-4245-bbc8-2071c29bc2cf 0 2024-01-26T23:29:27.362 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:27.360+0000 7f507f3191c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:29:27.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.448+0000 7f8de2ae1700 1 Processor -- start 2024-01-26T23:29:27.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.448+0000 7f8de2ae1700 1 -- start start 2024-01-26T23:29:27.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.449+0000 7f8de2ae1700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8ddc06b4e0 0x7f8ddc06a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:27.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.449+0000 7f8de2ae1700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc0a9a80 con 0x7f8ddc068ed0 2024-01-26T23:29:27.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.449+0000 7f8de2ae1700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc15a680 con 0x7f8ddc068b00 2024-01-26T23:29:27.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.449+0000 7f8de2ae1700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ddc069320 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.449+0000 7f8de107e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8ddc06b4e0 0x7f8ddc06a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:27.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.449+0000 7f8de087d700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f8ddc068b00 0x7f8ddc159ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:36998/0 (socket says 172.21.15.106:36998) 2024-01-26T23:29:27.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.449+0000 7f8de087d700 1 -- 172.21.15.106:0/192291552 learned_addr learned my addr 172.21.15.106:0/192291552 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-01-26T23:29:27.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.449+0000 7f8de107e700 1 -- 172.21.15.106:0/192291552 >> v1:172.21.15.106:6789/0 conn(0x7f8ddc068b00 legacy=0x7f8ddc159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:27.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.449+0000 7f8de107e700 1 -- 172.21.15.106:0/192291552 >> v1:172.21.15.46:6789/0 conn(0x7f8ddc068ed0 legacy=0x7f8ddc06add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:27.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.449+0000 7f8de107e700 1 -- 172.21.15.106:0/192291552 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc15a960 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.450+0000 7f8de107e700 1 --2- 172.21.15.106:0/192291552 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ddc06b4e0 0x7f8ddc06a660 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4009e60 tx=0x7f8dd4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=457453d69a880bc server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.450+0000 7f8ddb7fe700 1 -- 172.21.15.106:0/192291552 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd400e070 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.450+0000 7f8ddb7fe700 1 -- 172.21.15.106:0/192291552 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd400bc30 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.450+0000 7f8ddb7fe700 1 -- 172.21.15.106:0/192291552 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4012430 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.450+0000 7f8de2ae1700 1 -- 172.21.15.106:0/192291552 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ddc06b4e0 msgr2=0x7f8ddc06a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.450+0000 7f8de2ae1700 1 --2- 172.21.15.106:0/192291552 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ddc06b4e0 0x7f8ddc06a660 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4009e60 tx=0x7f8dd4009780 comp rx=0 tx=0).stop 2024-01-26T23:29:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.450+0000 7f8de2ae1700 1 -- 172.21.15.106:0/192291552 shutdown_connections 2024-01-26T23:29:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.450+0000 7f8de2ae1700 1 --2- 172.21.15.106:0/192291552 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ddc06b4e0 0x7f8ddc06a660 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.450+0000 7f8de2ae1700 1 -- 172.21.15.106:0/192291552 >> 172.21.15.106:0/192291552 conn(0x7f8ddc0ca100 msgr2=0x7f8ddc06e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.450+0000 7f8de2ae1700 1 -- 172.21.15.106:0/192291552 shutdown_connections 2024-01-26T23:29:27.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.450+0000 7f8de2ae1700 1 -- 172.21.15.106:0/192291552 wait complete. 2024-01-26T23:29:27.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.451+0000 7f8de2ae1700 1 Processor -- start 2024-01-26T23:29:27.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.452+0000 7f8de2ae1700 1 -- start start 2024-01-26T23:29:27.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.452+0000 7f8de2ae1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ddc068b00 0x7f8ddc0b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:27.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.452+0000 7f8de2ae1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ddc068ed0 0x7f8ddc0ada70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:27.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.452+0000 7f8de2ae1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ddc06b4e0 0x7f8ddc14be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:27.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.452+0000 7f8de2ae1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc0b1340 con 0x7f8ddc068ed0 2024-01-26T23:29:27.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.452+0000 7f8de2ae1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc0b2140 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.452+0000 7f8de2ae1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc0b1530 con 0x7f8ddc068b00 2024-01-26T23:29:27.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8de087d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ddc068b00 0x7f8ddc0b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8de107e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ddc06b4e0 0x7f8ddc14be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8de087d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ddc068b00 0x7f8ddc0b25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:43888/0 (socket says 172.21.15.106:43888) 2024-01-26T23:29:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8de087d700 1 -- 172.21.15.106:0/3373076311 learned_addr learned my addr 172.21.15.106:0/3373076311 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-26T23:29:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8ddbfff700 1 --2- 172.21.15.106:0/3373076311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ddc068ed0 0x7f8ddc0ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8de107e700 1 -- 172.21.15.106:0/3373076311 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ddc068b00 msgr2=0x7f8ddc0b25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8de107e700 1 --2- 172.21.15.106:0/3373076311 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ddc068b00 0x7f8ddc0b25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8de107e700 1 -- 172.21.15.106:0/3373076311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ddc068ed0 msgr2=0x7f8ddc0ada70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8de107e700 1 --2- 172.21.15.106:0/3373076311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ddc068ed0 0x7f8ddc0ada70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8de107e700 1 -- 172.21.15.106:0/3373076311 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd4009a20 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8de107e700 1 --2- 172.21.15.106:0/3373076311 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ddc06b4e0 0x7f8ddc14be90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4016000 tx=0x7f8dd4019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8dd8ff9700 1 -- 172.21.15.106:0/3373076311 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4012e10 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8dd8ff9700 1 -- 172.21.15.106:0/3373076311 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd401e630 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8dd8ff9700 1 -- 172.21.15.106:0/3373076311 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd401e9f0 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8de2ae1700 1 -- 172.21.15.106:0/3373076311 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ddc0b1810 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8de2ae1700 1 -- 172.21.15.106:0/3373076311 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ddc0b1e50 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8dd8ff9700 1 -- 172.21.15.106:0/3373076311 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f8dd4024070 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.453+0000 7f8dd8ff9700 1 -- 172.21.15.106:0/3373076311 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8dd4027770 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.454+0000 7f8de2ae1700 1 -- 172.21.15.106:0/3373076311 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ddc04c710 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.458 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:27.456+0000 7efc42e4f1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:29:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.456+0000 7f8dd8ff9700 1 -- 172.21.15.106:0/3373076311 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f8dd402c020 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.541 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.539+0000 7f8de2ae1700 1 -- 172.21.15.106:0/3373076311 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd new", "uuid": "66210791-18e9-4245-bbc8-2071c29bc2cf", "id": 0} v 0) v1 -- 0x7f8ddc06edf0 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.542+0000 7f8dd8ff9700 1 -- 172.21.15.106:0/3373076311 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "66210791-18e9-4245-bbc8-2071c29bc2cf", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8dd401ede0 con 0x7f8ddc06b4e0 2024-01-26T23:29:27.544 INFO:teuthology.orchestra.run.smithi106.stdout:0 2024-01-26T23:29:27.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.543+0000 7f8de2ae1700 1 -- 172.21.15.106:0/3373076311 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ddc06b4e0 msgr2=0x7f8ddc14be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:27.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.543+0000 7f8de2ae1700 1 --2- 172.21.15.106:0/3373076311 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ddc06b4e0 0x7f8ddc14be90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4016000 tx=0x7f8dd4019000 comp rx=0 tx=0).stop 2024-01-26T23:29:27.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.543+0000 7f8de2ae1700 1 -- 172.21.15.106:0/3373076311 shutdown_connections 2024-01-26T23:29:27.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.543+0000 7f8de2ae1700 1 --2- 172.21.15.106:0/3373076311 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ddc06b4e0 0x7f8ddc14be90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.543+0000 7f8de2ae1700 1 --2- 172.21.15.106:0/3373076311 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ddc068b00 0x7f8ddc0b25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.543+0000 7f8de2ae1700 1 --2- 172.21.15.106:0/3373076311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ddc068ed0 0x7f8ddc0ada70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.543+0000 7f8de2ae1700 1 -- 172.21.15.106:0/3373076311 >> 172.21.15.106:0/3373076311 conn(0x7f8ddc0ca100 msgr2=0x7f8ddc157020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:27.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.543+0000 7f8de2ae1700 1 -- 172.21.15.106:0/3373076311 shutdown_connections 2024-01-26T23:29:27.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.543+0000 7f8de2ae1700 1 -- 172.21.15.106:0/3373076311 wait complete. 2024-01-26T23:29:27.551 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:27.550+0000 7f507f3191c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:29:27.556 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd new a6bdcf0b-37dd-44d3-bff9-6e7639924cd9 1 2024-01-26T23:29:27.611 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:27.610+0000 7efc42e4f1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:29:27.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.672+0000 7f125a46f700 1 Processor -- start 2024-01-26T23:29:27.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.673+0000 7f125a46f700 1 -- start start 2024-01-26T23:29:27.676 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.673+0000 7f125a46f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12540ab6e0 0x7f125406a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:27.677 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.673+0000 7f125a46f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1254068940 0x7f1254068d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:27.677 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.673+0000 7f125a46f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12540692c0 con 0x7f12540ab310 2024-01-26T23:29:27.677 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.673+0000 7f125a46f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12540c7540 con 0x7f12540ab6e0 2024-01-26T23:29:27.677 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.673+0000 7f125a46f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12540c7820 con 0x7f1254068940 2024-01-26T23:29:27.677 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.673+0000 7f1253fff700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f12540ab310 0x7f12540a9850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:37006/0 (socket says 172.21.15.106:37006) 2024-01-26T23:29:27.677 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.673+0000 7f1253fff700 1 -- 172.21.15.106:0/2008086081 learned_addr learned my addr 172.21.15.106:0/2008086081 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-01-26T23:29:27.677 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.673+0000 7f1258a0c700 1 --2- 172.21.15.106:0/2008086081 >> v2:172.21.15.106:3300/0 conn(0x7f1254068940 0x7f1254068d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:27.677 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.674+0000 7f12537fe700 1 --2- 172.21.15.106:0/2008086081 >> v2:172.21.15.46:3300/0 conn(0x7f12540ab6e0 0x7f125406a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:27.677 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.674+0000 7f1252ffd700 1 -- 172.21.15.106:0/2008086081 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707583963 0 0) 0x7f12540692c0 con 0x7f12540ab310 2024-01-26T23:29:27.677 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.674+0000 7f1252ffd700 1 -- 172.21.15.106:0/2008086081 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f123c003570 con 0x7f12540ab310 2024-01-26T23:29:27.677 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.674+0000 7f1258a0c700 1 -- 172.21.15.106:0/2008086081 >> v2:172.21.15.46:3300/0 conn(0x7f12540ab6e0 msgr2=0x7f125406a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:27.677 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.674+0000 7f1258a0c700 1 --2- 172.21.15.106:0/2008086081 >> v2:172.21.15.46:3300/0 conn(0x7f12540ab6e0 0x7f125406a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.677 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.674+0000 7f1258a0c700 1 -- 172.21.15.106:0/2008086081 >> v1:172.21.15.106:6789/0 conn(0x7f12540ab310 legacy=0x7f12540a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:27.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.674+0000 7f1258a0c700 1 -- 172.21.15.106:0/2008086081 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12540c7b00 con 0x7f1254068940 2024-01-26T23:29:27.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.674+0000 7f1258a0c700 1 --2- 172.21.15.106:0/2008086081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1254068940 0x7f1254068d80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f125415e530 tx=0x7f124c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75057f01a3963b36 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:27.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.674+0000 7f1252ffd700 1 -- 172.21.15.106:0/2008086081 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124c00e070 con 0x7f1254068940 2024-01-26T23:29:27.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.674+0000 7f1252ffd700 1 -- 172.21.15.106:0/2008086081 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f124c00bab0 con 0x7f1254068940 2024-01-26T23:29:27.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.674+0000 7f1252ffd700 1 -- 172.21.15.106:0/2008086081 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124c00be70 con 0x7f1254068940 2024-01-26T23:29:27.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.675+0000 7f125a46f700 1 -- 172.21.15.106:0/2008086081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1254068940 msgr2=0x7f1254068d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:27.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.675+0000 7f125a46f700 1 --2- 172.21.15.106:0/2008086081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1254068940 0x7f1254068d80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f125415e530 tx=0x7f124c009780 comp rx=0 tx=0).stop 2024-01-26T23:29:27.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.675+0000 7f125a46f700 1 -- 172.21.15.106:0/2008086081 shutdown_connections 2024-01-26T23:29:27.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.675+0000 7f125a46f700 1 --2- 172.21.15.106:0/2008086081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1254068940 0x7f1254068d80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.675+0000 7f125a46f700 1 --2- 172.21.15.106:0/2008086081 >> v2:172.21.15.46:3300/0 conn(0x7f12540ab6e0 0x7f125406a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.675+0000 7f125a46f700 1 -- 172.21.15.106:0/2008086081 >> 172.21.15.106:0/2008086081 conn(0x7f12540ca100 msgr2=0x7f125406e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:27.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.675+0000 7f125a46f700 1 -- 172.21.15.106:0/2008086081 shutdown_connections 2024-01-26T23:29:27.678 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.675+0000 7f125a46f700 1 -- 172.21.15.106:0/2008086081 wait complete. 2024-01-26T23:29:27.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.676+0000 7f125a46f700 1 Processor -- start 2024-01-26T23:29:27.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.676+0000 7f125a46f700 1 -- start start 2024-01-26T23:29:27.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.677+0000 7f125a46f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1254068940 0x7f12541591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:27.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.677+0000 7f125a46f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12540ab310 0x7f12540c5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:27.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.677+0000 7f125a46f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12540ab6e0 0x7f12540c6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:27.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.677+0000 7f125a46f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12540c3d90 con 0x7f1254068940 2024-01-26T23:29:27.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.677+0000 7f125a46f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12541596f0 con 0x7f12540ab6e0 2024-01-26T23:29:27.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.677+0000 7f125a46f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12540ad4b0 con 0x7f12540ab310 2024-01-26T23:29:27.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.677+0000 7f1258a0c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12540ab6e0 0x7f12540c6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:27.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.677+0000 7f12537fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12540ab310 0x7f12540c5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:27.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.677+0000 7f12537fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12540ab310 0x7f12540c5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:43900/0 (socket says 172.21.15.106:43900) 2024-01-26T23:29:27.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.677+0000 7f1258a0c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12540ab6e0 0x7f12540c6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:49252/0 (socket says 172.21.15.106:49252) 2024-01-26T23:29:27.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.677+0000 7f12537fe700 1 -- 172.21.15.106:0/1922613675 learned_addr learned my addr 172.21.15.106:0/1922613675 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-26T23:29:27.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.678+0000 7f1253fff700 1 --2- 172.21.15.106:0/1922613675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1254068940 0x7f12541591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:27.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.678+0000 7f1258a0c700 1 -- 172.21.15.106:0/1922613675 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12540ab310 msgr2=0x7f12540c5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:27.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.678+0000 7f1258a0c700 1 --2- 172.21.15.106:0/1922613675 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12540ab310 0x7f12540c5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.678+0000 7f1258a0c700 1 -- 172.21.15.106:0/1922613675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1254068940 msgr2=0x7f12541591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:27.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.678+0000 7f1258a0c700 1 --2- 172.21.15.106:0/1922613675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1254068940 0x7f12541591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.678+0000 7f1258a0c700 1 -- 172.21.15.106:0/1922613675 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124c009a20 con 0x7f12540ab6e0 2024-01-26T23:29:27.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.678+0000 7f12537fe700 1 --2- 172.21.15.106:0/1922613675 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12540ab310 0x7f12540c5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:29:27.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.678+0000 7f1258a0c700 1 --2- 172.21.15.106:0/1922613675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12540ab6e0 0x7f12540c6310 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f124c016000 tx=0x7f124c019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:27.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.678+0000 7f123bfff700 1 -- 172.21.15.106:0/1922613675 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124c012d00 con 0x7f12540ab6e0 2024-01-26T23:29:27.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.678+0000 7f123bfff700 1 -- 172.21.15.106:0/1922613675 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f124c012e80 con 0x7f12540ab6e0 2024-01-26T23:29:27.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.678+0000 7f125a46f700 1 -- 172.21.15.106:0/1922613675 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12540acc10 con 0x7f12540ab6e0 2024-01-26T23:29:27.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.678+0000 7f123bfff700 1 -- 172.21.15.106:0/1922613675 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124c0276c0 con 0x7f12540ab6e0 2024-01-26T23:29:27.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.678+0000 7f125a46f700 1 -- 172.21.15.106:0/1922613675 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12540ad1d0 con 0x7f12540ab6e0 2024-01-26T23:29:27.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.678+0000 7f123bfff700 1 -- 172.21.15.106:0/1922613675 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f124c024070 con 0x7f12540ab6e0 2024-01-26T23:29:27.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.678+0000 7f123bfff700 1 -- 172.21.15.106:0/1922613675 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f124c01e830 con 0x7f12540ab6e0 2024-01-26T23:29:27.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.679+0000 7f125a46f700 1 -- 172.21.15.106:0/1922613675 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12540c3f40 con 0x7f12540ab6e0 2024-01-26T23:29:27.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.681+0000 7f123bfff700 1 -- 172.21.15.106:0/1922613675 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f124c0466d0 con 0x7f12540ab6e0 2024-01-26T23:29:27.684 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:27.683+0000 7efc42e4f1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:29:27.715 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:27.713+0000 7f507f3191c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:29:27.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.765+0000 7f125a46f700 1 -- 172.21.15.106:0/1922613675 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a6bdcf0b-37dd-44d3-bff9-6e7639924cd9", "id": 1} v 0) v1 -- 0x7f125404c710 con 0x7f12540ab6e0 2024-01-26T23:29:27.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.768+0000 7f123bfff700 1 -- 172.21.15.106:0/1922613675 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a6bdcf0b-37dd-44d3-bff9-6e7639924cd9", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f124c035030 con 0x7f12540ab6e0 2024-01-26T23:29:27.770 INFO:teuthology.orchestra.run.smithi106.stdout:1 2024-01-26T23:29:27.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.769+0000 7f125a46f700 1 -- 172.21.15.106:0/1922613675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12540ab6e0 msgr2=0x7f12540c6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:27.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.769+0000 7f125a46f700 1 --2- 172.21.15.106:0/1922613675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12540ab6e0 0x7f12540c6310 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f124c016000 tx=0x7f124c019000 comp rx=0 tx=0).stop 2024-01-26T23:29:27.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.769+0000 7f125a46f700 1 -- 172.21.15.106:0/1922613675 shutdown_connections 2024-01-26T23:29:27.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.769+0000 7f125a46f700 1 --2- 172.21.15.106:0/1922613675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1254068940 0x7f12541591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.769+0000 7f125a46f700 1 --2- 172.21.15.106:0/1922613675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12540ab6e0 0x7f12540c6310 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.769+0000 7f125a46f700 1 --2- 172.21.15.106:0/1922613675 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12540ab310 0x7f12540c5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.772 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.769+0000 7f125a46f700 1 -- 172.21.15.106:0/1922613675 >> 172.21.15.106:0/1922613675 conn(0x7f12540ca100 msgr2=0x7f12540a9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:27.772 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.769+0000 7f125a46f700 1 -- 172.21.15.106:0/1922613675 shutdown_connections 2024-01-26T23:29:27.772 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.769+0000 7f125a46f700 1 -- 172.21.15.106:0/1922613675 wait complete. 2024-01-26T23:29:27.782 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd new c27617ee-7d44-4220-9fb0-c2739304afc3 2 2024-01-26T23:29:27.792 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:27.791+0000 7f507f3191c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:29:27.824 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:27.822+0000 7efc42e4f1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:29:27.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.899+0000 7f3008fd5700 1 Processor -- start 2024-01-26T23:29:27.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.899+0000 7f3008fd5700 1 -- start start 2024-01-26T23:29:27.901 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:27.900+0000 7efc42e4f1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:29:27.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.900+0000 7f3008fd5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3004068b00 0x7f3004068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:27.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.900+0000 7f3008fd5700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f300406a660 0x7f3004159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:27.903 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.900+0000 7f3008fd5700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30040a9a80 con 0x7f300406add0 2024-01-26T23:29:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.900+0000 7f3008fd5700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f300415a3e0 con 0x7f300406a660 2024-01-26T23:29:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.900+0000 7f3008fd5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f300415a680 con 0x7f3004068b00 2024-01-26T23:29:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.900+0000 7f300259c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3004068b00 0x7f3004068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.900+0000 7f300259c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3004068b00 0x7f3004068ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:43916/0 (socket says 172.21.15.106:43916) 2024-01-26T23:29:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.900+0000 7f300259c700 1 -- 172.21.15.106:0/950960291 learned_addr learned my addr 172.21.15.106:0/950960291 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-26T23:29:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.900+0000 7f3001d9b700 1 --2- 172.21.15.106:0/950960291 >> v2:172.21.15.46:3300/0 conn(0x7f300406a660 0x7f3004159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.901+0000 7f300259c700 1 -- 172.21.15.106:0/950960291 >> v2:172.21.15.46:3300/0 conn(0x7f300406a660 msgr2=0x7f3004159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.901+0000 7f300259c700 1 --2- 172.21.15.106:0/950960291 >> v2:172.21.15.46:3300/0 conn(0x7f300406a660 0x7f3004159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.901+0000 7f300259c700 1 -- 172.21.15.106:0/950960291 >> v1:172.21.15.46:6789/0 conn(0x7f300406add0 legacy=0x7f300415d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.901+0000 7f300259c700 1 -- 172.21.15.106:0/950960291 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300406aad0 con 0x7f3004068b00 2024-01-26T23:29:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.901+0000 7f300259c700 1 --2- 172.21.15.106:0/950960291 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3004068b00 0x7f3004068ed0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009990 tx=0x7f2ff4009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95ca2513bd6a58f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.901+0000 7f300159a700 1 -- 172.21.15.106:0/950960291 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4005c10 con 0x7f3004068b00 2024-01-26T23:29:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.901+0000 7f300159a700 1 -- 172.21.15.106:0/950960291 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff4005d70 con 0x7f3004068b00 2024-01-26T23:29:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.901+0000 7f300159a700 1 -- 172.21.15.106:0/950960291 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff400b840 con 0x7f3004068b00 2024-01-26T23:29:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.901+0000 7f3008fd5700 1 -- 172.21.15.106:0/950960291 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3004068b00 msgr2=0x7f3004068ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.901+0000 7f3008fd5700 1 --2- 172.21.15.106:0/950960291 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3004068b00 0x7f3004068ed0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009990 tx=0x7f2ff4009220 comp rx=0 tx=0).stop 2024-01-26T23:29:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.901+0000 7f3008fd5700 1 -- 172.21.15.106:0/950960291 shutdown_connections 2024-01-26T23:29:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.901+0000 7f3008fd5700 1 --2- 172.21.15.106:0/950960291 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3004068b00 0x7f3004068ed0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.901+0000 7f3008fd5700 1 --2- 172.21.15.106:0/950960291 >> v2:172.21.15.46:3300/0 conn(0x7f300406a660 0x7f3004159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.901+0000 7f3008fd5700 1 -- 172.21.15.106:0/950960291 >> 172.21.15.106:0/950960291 conn(0x7f30040ca100 msgr2=0x7f300406e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.902+0000 7f3008fd5700 1 -- 172.21.15.106:0/950960291 shutdown_connections 2024-01-26T23:29:27.905 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.902+0000 7f3008fd5700 1 -- 172.21.15.106:0/950960291 wait complete. 2024-01-26T23:29:27.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.903+0000 7f3008fd5700 1 Processor -- start 2024-01-26T23:29:27.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.903+0000 7f3008fd5700 1 -- start start 2024-01-26T23:29:27.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.903+0000 7f3008fd5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3004068b00 0x7f3004167840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:27.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.903+0000 7f3008fd5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f300406a660 0x7f300416a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:27.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.903+0000 7f3008fd5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f300406add0 0x7f3004169510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:27.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.903+0000 7f3008fd5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30040ae720 con 0x7f300406add0 2024-01-26T23:29:27.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.903+0000 7f3008fd5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300416bf30 con 0x7f3004068b00 2024-01-26T23:29:27.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.903+0000 7f3008fd5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3004167d80 con 0x7f300406a660 2024-01-26T23:29:27.908 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.903+0000 7f3001d9b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f300406a660 0x7f300416a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:27.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.903+0000 7f300259c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3004068b00 0x7f3004167840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:27.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.903+0000 7f3001d9b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f300406a660 0x7f300416a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:43918/0 (socket says 172.21.15.106:43918) 2024-01-26T23:29:27.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.903+0000 7f3001d9b700 1 -- 172.21.15.106:0/1212312314 learned_addr learned my addr 172.21.15.106:0/1212312314 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-26T23:29:27.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.903+0000 7f3002d9d700 1 --2- 172.21.15.106:0/1212312314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f300406add0 0x7f3004169510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:27.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.904+0000 7f3001d9b700 1 -- 172.21.15.106:0/1212312314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3004068b00 msgr2=0x7f3004167840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:27.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.904+0000 7f3001d9b700 1 --2- 172.21.15.106:0/1212312314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3004068b00 0x7f3004167840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.904+0000 7f3001d9b700 1 -- 172.21.15.106:0/1212312314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f300406add0 msgr2=0x7f3004169510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:27.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.904+0000 7f3001d9b700 1 --2- 172.21.15.106:0/1212312314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f300406add0 0x7f3004169510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:27.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.904+0000 7f3001d9b700 1 -- 172.21.15.106:0/1212312314 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff4009480 con 0x7f300406a660 2024-01-26T23:29:27.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.904+0000 7f300259c700 1 --2- 172.21.15.106:0/1212312314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3004068b00 0x7f3004167840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:29:27.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.904+0000 7f3001d9b700 1 --2- 172.21.15.106:0/1212312314 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f300406a660 0x7f300416a650 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800d890 tx=0x7f2ff8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:27.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.904+0000 7f2ff27fc700 1 -- 172.21.15.106:0/1212312314 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff80125f0 con 0x7f300406a660 2024-01-26T23:29:27.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.904+0000 7f2ff27fc700 1 -- 172.21.15.106:0/1212312314 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff8012750 con 0x7f300406a660 2024-01-26T23:29:27.909 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.904+0000 7f2ff27fc700 1 -- 172.21.15.106:0/1212312314 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8012b10 con 0x7f300406a660 2024-01-26T23:29:27.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.904+0000 7f3008fd5700 1 -- 172.21.15.106:0/1212312314 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300416ab90 con 0x7f300406a660 2024-01-26T23:29:27.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.904+0000 7f3008fd5700 1 -- 172.21.15.106:0/1212312314 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300416b320 con 0x7f300406a660 2024-01-26T23:29:27.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.904+0000 7f2ff27fc700 1 -- 172.21.15.106:0/1212312314 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f2ff801e460 con 0x7f300406a660 2024-01-26T23:29:27.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.904+0000 7f2ff27fc700 1 -- 172.21.15.106:0/1212312314 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f2ff8014430 con 0x7f300406a660 2024-01-26T23:29:27.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.905+0000 7f3008fd5700 1 -- 172.21.15.106:0/1212312314 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300404c710 con 0x7f300406a660 2024-01-26T23:29:27.910 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.907+0000 7f2ff27fc700 1 -- 172.21.15.106:0/1212312314 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f2ff801e710 con 0x7f300406a660 2024-01-26T23:29:27.942 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:27.941+0000 7f507f3191c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:29:27.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.994+0000 7f3008fd5700 1 -- 172.21.15.106:0/1212312314 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c27617ee-7d44-4220-9fb0-c2739304afc3", "id": 2} v 0) v1 -- 0x7f300406edf0 con 0x7f300406a660 2024-01-26T23:29:27.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.997+0000 7f2ff27fc700 1 -- 172.21.15.106:0/1212312314 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c27617ee-7d44-4220-9fb0-c2739304afc3", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2ff802d0b0 con 0x7f300406a660 2024-01-26T23:29:27.999 INFO:teuthology.orchestra.run.smithi106.stdout:2 2024-01-26T23:29:28.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.998+0000 7f3008fd5700 1 -- 172.21.15.106:0/1212312314 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f300406a660 msgr2=0x7f300416a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.998+0000 7f3008fd5700 1 --2- 172.21.15.106:0/1212312314 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f300406a660 0x7f300416a650 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800d890 tx=0x7f2ff8008040 comp rx=0 tx=0).stop 2024-01-26T23:29:28.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.999+0000 7f3008fd5700 1 -- 172.21.15.106:0/1212312314 shutdown_connections 2024-01-26T23:29:28.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.999+0000 7f3008fd5700 1 --2- 172.21.15.106:0/1212312314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f300406add0 0x7f3004169510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.000 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.999+0000 7f3008fd5700 1 --2- 172.21.15.106:0/1212312314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3004068b00 0x7f3004167840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.999+0000 7f3008fd5700 1 --2- 172.21.15.106:0/1212312314 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f300406a660 0x7f300416a650 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.999+0000 7f3008fd5700 1 -- 172.21.15.106:0/1212312314 >> 172.21.15.106:0/1212312314 conn(0x7f30040ca100 msgr2=0x7f30041570b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:28.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.999+0000 7f3008fd5700 1 -- 172.21.15.106:0/1212312314 shutdown_connections 2024-01-26T23:29:28.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:27.999+0000 7f3008fd5700 1 -- 172.21.15.106:0/1212312314 wait complete. 2024-01-26T23:29:28.015 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd new 83b0aff3-46e9-4da0-9b6f-4e1088814137 3 2024-01-26T23:29:28.026 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:28.024+0000 7f507f3191c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:29:28.072 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:28.071+0000 7efc42e4f1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:29:28.177 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.176+0000 7f8359e3a700 1 Processor -- start 2024-01-26T23:29:28.178 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.176+0000 7f8359e3a700 1 -- start start 2024-01-26T23:29:28.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.177+0000 7f8359e3a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8354157750 0x7f8354157b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.177+0000 7f8359e3a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f835406b3c0 0x7f83540af080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.177+0000 7f8359e3a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8354158f00 con 0x7f83540af5c0 2024-01-26T23:29:28.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.177+0000 7f8359e3a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83540afa20 con 0x7f8354157750 2024-01-26T23:29:28.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.177+0000 7f8359e3a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8354158060 con 0x7f835406b3c0 2024-01-26T23:29:28.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.177+0000 7f8352ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f835406b3c0 0x7f83540af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.177+0000 7f8352ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f835406b3c0 0x7f83540af080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:43934/0 (socket says 172.21.15.106:43934) 2024-01-26T23:29:28.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.177+0000 7f8352ffd700 1 -- 172.21.15.106:0/2592593559 learned_addr learned my addr 172.21.15.106:0/2592593559 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-26T23:29:28.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.177+0000 7f83537fe700 1 --2- 172.21.15.106:0/2592593559 >> v2:172.21.15.46:3300/0 conn(0x7f8354157750 0x7f8354157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.177+0000 7f8352ffd700 1 -- 172.21.15.106:0/2592593559 >> v2:172.21.15.46:3300/0 conn(0x7f8354157750 msgr2=0x7f8354157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:28.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.177+0000 7f8352ffd700 1 --2- 172.21.15.106:0/2592593559 >> v2:172.21.15.46:3300/0 conn(0x7f8354157750 0x7f8354157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.177+0000 7f8352ffd700 1 -- 172.21.15.106:0/2592593559 >> v1:172.21.15.46:6789/0 conn(0x7f83540af5c0 legacy=0x7f8354067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.177+0000 7f8352ffd700 1 -- 172.21.15.106:0/2592593559 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835406fe90 con 0x7f835406b3c0 2024-01-26T23:29:28.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.177+0000 7f83537fe700 1 --2- 172.21.15.106:0/2592593559 >> v2:172.21.15.46:3300/0 conn(0x7f8354157750 0x7f8354157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:29:28.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.178+0000 7f8352ffd700 1 --2- 172.21.15.106:0/2592593559 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f835406b3c0 0x7f83540af080 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f83480099b0 tx=0x7f8348009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b72f95984d6a73e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:28.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.178+0000 7f83527fc700 1 -- 172.21.15.106:0/2592593559 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834800b860 con 0x7f835406b3c0 2024-01-26T23:29:28.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.178+0000 7f83527fc700 1 -- 172.21.15.106:0/2592593559 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834800b9c0 con 0x7f835406b3c0 2024-01-26T23:29:28.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.178+0000 7f83527fc700 1 -- 172.21.15.106:0/2592593559 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834800bd80 con 0x7f835406b3c0 2024-01-26T23:29:28.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.178+0000 7f8359e3a700 1 -- 172.21.15.106:0/2592593559 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f835406b3c0 msgr2=0x7f83540af080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.178+0000 7f8359e3a700 1 --2- 172.21.15.106:0/2592593559 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f835406b3c0 0x7f83540af080 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f83480099b0 tx=0x7f8348009250 comp rx=0 tx=0).stop 2024-01-26T23:29:28.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.178+0000 7f8359e3a700 1 -- 172.21.15.106:0/2592593559 shutdown_connections 2024-01-26T23:29:28.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.178+0000 7f8359e3a700 1 --2- 172.21.15.106:0/2592593559 >> v2:172.21.15.46:3300/0 conn(0x7f8354157750 0x7f8354157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.178+0000 7f8359e3a700 1 --2- 172.21.15.106:0/2592593559 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f835406b3c0 0x7f83540af080 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.178+0000 7f8359e3a700 1 -- 172.21.15.106:0/2592593559 >> 172.21.15.106:0/2592593559 conn(0x7f83540cdf80 msgr2=0x7f835406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:28.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.178+0000 7f8359e3a700 1 -- 172.21.15.106:0/2592593559 shutdown_connections 2024-01-26T23:29:28.182 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.178+0000 7f8359e3a700 1 -- 172.21.15.106:0/2592593559 wait complete. 2024-01-26T23:29:28.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.180+0000 7f8359e3a700 1 Processor -- start 2024-01-26T23:29:28.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.180+0000 7f8359e3a700 1 -- start start 2024-01-26T23:29:28.184 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.180+0000 7f8359e3a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f835406b3c0 0x7f8354154400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.180+0000 7f8359e3a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83540af5c0 0x7f8354160cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.180+0000 7f8359e3a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8354157750 0x7f8354161200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.180+0000 7f8359e3a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835415eb90 con 0x7f835406b3c0 2024-01-26T23:29:28.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.180+0000 7f8359e3a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835415df80 con 0x7f83540af5c0 2024-01-26T23:29:28.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.180+0000 7f8359e3a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835415e260 con 0x7f8354157750 2024-01-26T23:29:28.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f8352ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83540af5c0 0x7f8354160cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f8353fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8354157750 0x7f8354161200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f8353fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8354157750 0x7f8354161200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:43948/0 (socket says 172.21.15.106:43948) 2024-01-26T23:29:28.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f8353fff700 1 -- 172.21.15.106:0/1912309228 learned_addr learned my addr 172.21.15.106:0/1912309228 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-26T23:29:28.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f8352ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83540af5c0 0x7f8354160cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:49270/0 (socket says 172.21.15.106:49270) 2024-01-26T23:29:28.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f83537fe700 1 --2- 172.21.15.106:0/1912309228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f835406b3c0 0x7f8354154400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f8353fff700 1 -- 172.21.15.106:0/1912309228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83540af5c0 msgr2=0x7f8354160cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f8353fff700 1 --2- 172.21.15.106:0/1912309228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83540af5c0 0x7f8354160cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.185 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f8353fff700 1 -- 172.21.15.106:0/1912309228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f835406b3c0 msgr2=0x7f8354154400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f8353fff700 1 --2- 172.21.15.106:0/1912309228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f835406b3c0 0x7f8354154400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f8353fff700 1 -- 172.21.15.106:0/1912309228 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83480094f0 con 0x7f8354157750 2024-01-26T23:29:28.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f8352ffd700 1 --2- 172.21.15.106:0/1912309228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83540af5c0 0x7f8354160cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:29:28.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f8353fff700 1 --2- 172.21.15.106:0/1912309228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8354157750 0x7f8354161200 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f834c00c8e0 tx=0x7f834c001710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:28.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f833b7fe700 1 -- 172.21.15.106:0/1912309228 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834c012630 con 0x7f8354157750 2024-01-26T23:29:28.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f833b7fe700 1 -- 172.21.15.106:0/1912309228 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834c012790 con 0x7f8354157750 2024-01-26T23:29:28.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f8359e3a700 1 -- 172.21.15.106:0/1912309228 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f835415e540 con 0x7f8354157750 2024-01-26T23:29:28.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f833b7fe700 1 -- 172.21.15.106:0/1912309228 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834c012b50 con 0x7f8354157750 2024-01-26T23:29:28.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f8359e3a700 1 -- 172.21.15.106:0/1912309228 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8354154940 con 0x7f8354157750 2024-01-26T23:29:28.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f833b7fe700 1 -- 172.21.15.106:0/1912309228 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f834c01b400 con 0x7f8354157750 2024-01-26T23:29:28.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.181+0000 7f833b7fe700 1 -- 172.21.15.106:0/1912309228 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f834c01a840 con 0x7f8354157750 2024-01-26T23:29:28.186 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.182+0000 7f8359e3a700 1 -- 172.21.15.106:0/1912309228 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83540b14d0 con 0x7f8354157750 2024-01-26T23:29:28.187 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.184+0000 7f833b7fe700 1 -- 172.21.15.106:0/1912309228 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f834c01b890 con 0x7f8354157750 2024-01-26T23:29:28.202 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:28.201+0000 7efc42e4f1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:29:28.214 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:28.212+0000 7f507f3191c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:29:28.269 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.267+0000 7f8359e3a700 1 -- 172.21.15.106:0/1912309228 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd new", "uuid": "83b0aff3-46e9-4da0-9b6f-4e1088814137", "id": 3} v 0) v1 -- 0x7f835415ed00 con 0x7f8354157750 2024-01-26T23:29:28.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.271+0000 7f833b7fe700 1 -- 172.21.15.106:0/1912309228 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "83b0aff3-46e9-4da0-9b6f-4e1088814137", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f834c01bb10 con 0x7f8354157750 2024-01-26T23:29:28.272 INFO:teuthology.orchestra.run.smithi106.stdout:3 2024-01-26T23:29:28.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.272+0000 7f8359e3a700 1 -- 172.21.15.106:0/1912309228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8354157750 msgr2=0x7f8354161200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.272+0000 7f8359e3a700 1 --2- 172.21.15.106:0/1912309228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8354157750 0x7f8354161200 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f834c00c8e0 tx=0x7f834c001710 comp rx=0 tx=0).stop 2024-01-26T23:29:28.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.272+0000 7f8359e3a700 1 -- 172.21.15.106:0/1912309228 shutdown_connections 2024-01-26T23:29:28.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.272+0000 7f8359e3a700 1 --2- 172.21.15.106:0/1912309228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f835406b3c0 0x7f8354154400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.272+0000 7f8359e3a700 1 --2- 172.21.15.106:0/1912309228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8354157750 0x7f8354161200 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.272+0000 7f8359e3a700 1 --2- 172.21.15.106:0/1912309228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83540af5c0 0x7f8354160cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.272+0000 7f8359e3a700 1 -- 172.21.15.106:0/1912309228 >> 172.21.15.106:0/1912309228 conn(0x7f83540cdf80 msgr2=0x7f835406ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:28.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.272+0000 7f8359e3a700 1 -- 172.21.15.106:0/1912309228 shutdown_connections 2024-01-26T23:29:28.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.272+0000 7f8359e3a700 1 -- 172.21.15.106:0/1912309228 wait complete. 2024-01-26T23:29:28.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd new 372407c3-89ba-4f74-ab8a-b15a924b188d 4 2024-01-26T23:29:28.356 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:28.354+0000 7f507f3191c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:29:28.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.441+0000 7fe80d2f4700 1 Processor -- start 2024-01-26T23:29:28.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.441+0000 7fe80d2f4700 1 -- start start 2024-01-26T23:29:28.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.442+0000 7fe80d2f4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe808067ec0 0x7fe8080c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.442+0000 7fe80d2f4700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe8080c7fe0 0x7fe808157200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.442+0000 7fe80d2f4700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe80806c160 con 0x7fe808067af0 2024-01-26T23:29:28.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.442+0000 7fe80d2f4700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe808068330 con 0x7fe8080c7fe0 2024-01-26T23:29:28.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.442+0000 7fe80d2f4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80806ca00 con 0x7fe808067ec0 2024-01-26T23:29:28.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.442+0000 7fe8077fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe8080c7fe0 0x7fe808157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.442+0000 7fe8067fc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe808067ec0 0x7fe8080c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.442+0000 7fe8077fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe8080c7fe0 0x7fe808157200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:49284/0 (socket says 172.21.15.106:49284) 2024-01-26T23:29:28.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.442+0000 7fe8077fe700 1 -- 172.21.15.106:0/1799598063 learned_addr learned my addr 172.21.15.106:0/1799598063 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-26T23:29:28.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.442+0000 7fe8067fc700 1 --2- 172.21.15.106:0/1799598063 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe808067ec0 0x7fe8080c7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:43952/0 (socket says 172.21.15.106:43952) 2024-01-26T23:29:28.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.443+0000 7fe8077fe700 1 -- 172.21.15.106:0/1799598063 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe808067ec0 msgr2=0x7fe8080c7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:28.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.443+0000 7fe8077fe700 1 --2- 172.21.15.106:0/1799598063 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe808067ec0 0x7fe8080c7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.443+0000 7fe8077fe700 1 -- 172.21.15.106:0/1799598063 >> v1:172.21.15.46:6789/0 conn(0x7fe808067af0 legacy=0x7fe80806ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.443+0000 7fe8077fe700 1 -- 172.21.15.106:0/1799598063 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8080c84f0 con 0x7fe8080c7fe0 2024-01-26T23:29:28.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.443+0000 7fe8077fe700 1 --2- 172.21.15.106:0/1799598063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8080c7fe0 0x7fe808157200 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe800009f20 tx=0x7fe800009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=488f4af44cc7d78e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:28.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.443+0000 7fe805ffb700 1 -- 172.21.15.106:0/1799598063 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80000e070 con 0x7fe8080c7fe0 2024-01-26T23:29:28.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.443+0000 7fe805ffb700 1 -- 172.21.15.106:0/1799598063 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80000b9d0 con 0x7fe8080c7fe0 2024-01-26T23:29:28.446 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.443+0000 7fe805ffb700 1 -- 172.21.15.106:0/1799598063 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8000092e0 con 0x7fe8080c7fe0 2024-01-26T23:29:28.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.443+0000 7fe80d2f4700 1 -- 172.21.15.106:0/1799598063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8080c7fe0 msgr2=0x7fe808157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.443+0000 7fe80d2f4700 1 --2- 172.21.15.106:0/1799598063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8080c7fe0 0x7fe808157200 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe800009f20 tx=0x7fe800009840 comp rx=0 tx=0).stop 2024-01-26T23:29:28.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.443+0000 7fe80d2f4700 1 -- 172.21.15.106:0/1799598063 shutdown_connections 2024-01-26T23:29:28.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.443+0000 7fe80d2f4700 1 --2- 172.21.15.106:0/1799598063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8080c7fe0 0x7fe808157200 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.443+0000 7fe80d2f4700 1 --2- 172.21.15.106:0/1799598063 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe808067ec0 0x7fe8080c7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.443+0000 7fe80d2f4700 1 -- 172.21.15.106:0/1799598063 >> 172.21.15.106:0/1799598063 conn(0x7fe8080ca9a0 msgr2=0x7fe808068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:28.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.444+0000 7fe80d2f4700 1 -- 172.21.15.106:0/1799598063 shutdown_connections 2024-01-26T23:29:28.447 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.444+0000 7fe80d2f4700 1 -- 172.21.15.106:0/1799598063 wait complete. 2024-01-26T23:29:28.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.445+0000 7fe80d2f4700 1 Processor -- start 2024-01-26T23:29:28.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.445+0000 7fe80d2f4700 1 -- start start 2024-01-26T23:29:28.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.445+0000 7fe80d2f4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe808067af0 0x7fe8080cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.445+0000 7fe80d2f4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe808067ec0 0x7fe8080c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.445+0000 7fe80d2f4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8080c7fe0 0x7fe808169e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe80d2f4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8080cc020 con 0x7fe808067af0 2024-01-26T23:29:28.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe80d2f4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8080cb410 con 0x7fe808067ec0 2024-01-26T23:29:28.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe80d2f4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8080cb6f0 con 0x7fe8080c7fe0 2024-01-26T23:29:28.450 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe8067fc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe808067ec0 0x7fe8080c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe8077fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8080c7fe0 0x7fe808169e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe8067fc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe808067ec0 0x7fe8080c74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:49294/0 (socket says 172.21.15.106:49294) 2024-01-26T23:29:28.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe8077fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8080c7fe0 0x7fe808169e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:43964/0 (socket says 172.21.15.106:43964) 2024-01-26T23:29:28.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe8067fc700 1 -- 172.21.15.106:0/3610007572 learned_addr learned my addr 172.21.15.106:0/3610007572 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-26T23:29:28.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe806ffd700 1 --2- 172.21.15.106:0/3610007572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe808067af0 0x7fe8080cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe8067fc700 1 -- 172.21.15.106:0/3610007572 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8080c7fe0 msgr2=0x7fe808169e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe8067fc700 1 --2- 172.21.15.106:0/3610007572 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8080c7fe0 0x7fe808169e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe8067fc700 1 -- 172.21.15.106:0/3610007572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe808067af0 msgr2=0x7fe8080cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe8067fc700 1 --2- 172.21.15.106:0/3610007572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe808067af0 0x7fe8080cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe8067fc700 1 -- 172.21.15.106:0/3610007572 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe800009ae0 con 0x7fe808067ec0 2024-01-26T23:29:28.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe8077fe700 1 --2- 172.21.15.106:0/3610007572 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8080c7fe0 0x7fe808169e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:29:28.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe8067fc700 1 --2- 172.21.15.106:0/3610007572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe808067ec0 0x7fe8080c74d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc00ec80 tx=0x7fe7fc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:28.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe7eeffd700 1 -- 172.21.15.106:0/3610007572 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc00c850 con 0x7fe808067ec0 2024-01-26T23:29:28.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe7eeffd700 1 -- 172.21.15.106:0/3610007572 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7fc00c9b0 con 0x7fe808067ec0 2024-01-26T23:29:28.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe7eeffd700 1 -- 172.21.15.106:0/3610007572 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc00cd50 con 0x7fe808067ec0 2024-01-26T23:29:28.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe80d2f4700 1 -- 172.21.15.106:0/3610007572 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8080cb9d0 con 0x7fe808067ec0 2024-01-26T23:29:28.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.446+0000 7fe80d2f4700 1 -- 172.21.15.106:0/3610007572 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8081545c0 con 0x7fe808067ec0 2024-01-26T23:29:28.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.447+0000 7fe7eeffd700 1 -- 172.21.15.106:0/3610007572 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe7fc005c80 con 0x7fe808067ec0 2024-01-26T23:29:28.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.447+0000 7fe7eeffd700 1 -- 172.21.15.106:0/3610007572 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7fe7fc013b30 con 0x7fe808067ec0 2024-01-26T23:29:28.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.448+0000 7fe80d2f4700 1 -- 172.21.15.106:0/3610007572 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8080cc9f0 con 0x7fe808067ec0 2024-01-26T23:29:28.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.450+0000 7fe7eeffd700 1 -- 172.21.15.106:0/3610007572 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe7fc021060 con 0x7fe808067ec0 2024-01-26T23:29:28.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.556+0000 7fe80d2f4700 1 -- 172.21.15.106:0/3610007572 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd new", "uuid": "372407c3-89ba-4f74-ab8a-b15a924b188d", "id": 4} v 0) v1 -- 0x7fe80804c710 con 0x7fe808067ec0 2024-01-26T23:29:28.561 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.560+0000 7fe7eeffd700 1 -- 172.21.15.106:0/3610007572 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "372407c3-89ba-4f74-ab8a-b15a924b188d", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe7fc017070 con 0x7fe808067ec0 2024-01-26T23:29:28.561 INFO:teuthology.orchestra.run.smithi106.stdout:4 2024-01-26T23:29:28.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.561+0000 7fe80d2f4700 1 -- 172.21.15.106:0/3610007572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe808067ec0 msgr2=0x7fe8080c74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.561+0000 7fe80d2f4700 1 --2- 172.21.15.106:0/3610007572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe808067ec0 0x7fe8080c74d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc00ec80 tx=0x7fe7fc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.561+0000 7fe80d2f4700 1 -- 172.21.15.106:0/3610007572 shutdown_connections 2024-01-26T23:29:28.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.561+0000 7fe80d2f4700 1 --2- 172.21.15.106:0/3610007572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe808067af0 0x7fe8080cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.561+0000 7fe80d2f4700 1 --2- 172.21.15.106:0/3610007572 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8080c7fe0 0x7fe808169e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.561+0000 7fe80d2f4700 1 --2- 172.21.15.106:0/3610007572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe808067ec0 0x7fe8080c74d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.561+0000 7fe80d2f4700 1 -- 172.21.15.106:0/3610007572 >> 172.21.15.106:0/3610007572 conn(0x7fe8080ca9a0 msgr2=0x7fe80806eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:28.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.561+0000 7fe80d2f4700 1 -- 172.21.15.106:0/3610007572 shutdown_connections 2024-01-26T23:29:28.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.561+0000 7fe80d2f4700 1 -- 172.21.15.106:0/3610007572 wait complete. 2024-01-26T23:29:28.574 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd new 647f327a-33e0-4b2c-a84b-d2636d18a18e 5 2024-01-26T23:29:28.616 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:28.615+0000 7efc42e4f1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:29:28.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.689+0000 7f9f4ff07700 1 Processor -- start 2024-01-26T23:29:28.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.690+0000 7f9f4ff07700 1 -- start start 2024-01-26T23:29:28.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.690+0000 7f9f4ff07700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f48068940 0x7f9f48068d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.690+0000 7f9f4ff07700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9f480c8c50 0x7f9f480c7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.690+0000 7f9f4ff07700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9f480c76a0 0x7f9f480af990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.690+0000 7f9f4ff07700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f480c87c0 con 0x7f9f480c76a0 2024-01-26T23:29:28.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.690+0000 7f9f4ff07700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f48069250 con 0x7f9f480c8c50 2024-01-26T23:29:28.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.690+0000 7f9f4ff07700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f480c7bb0 con 0x7f9f48068940 2024-01-26T23:29:28.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.691+0000 7f9f4dca3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f48068940 0x7f9f48068d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.691+0000 7f9f4d4a2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9f480c8c50 0x7f9f480c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.691+0000 7f9f4d4a2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9f480c8c50 0x7f9f480c7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:49302/0 (socket says 172.21.15.106:49302) 2024-01-26T23:29:28.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.691+0000 7f9f4dca3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f48068940 0x7f9f48068d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:43968/0 (socket says 172.21.15.106:43968) 2024-01-26T23:29:28.694 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.691+0000 7f9f4d4a2700 1 -- 172.21.15.106:0/2346427601 learned_addr learned my addr 172.21.15.106:0/2346427601 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-26T23:29:28.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.691+0000 7f9f4e4a4700 1 --2- 172.21.15.106:0/2346427601 >> v2:172.21.15.46:3300/0 conn(0x7f9f480c76a0 0x7f9f480af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.691+0000 7f9f4dca3700 1 -- 172.21.15.106:0/2346427601 >> v2:172.21.15.106:3300/0 conn(0x7f9f480c8c50 msgr2=0x7f9f480c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:28.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.691+0000 7f9f4dca3700 1 --2- 172.21.15.106:0/2346427601 >> v2:172.21.15.106:3300/0 conn(0x7f9f480c8c50 0x7f9f480c7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.691+0000 7f9f4dca3700 1 -- 172.21.15.106:0/2346427601 >> v2:172.21.15.46:3300/0 conn(0x7f9f480c76a0 msgr2=0x7f9f480af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:28.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.691+0000 7f9f4dca3700 1 --2- 172.21.15.106:0/2346427601 >> v2:172.21.15.46:3300/0 conn(0x7f9f480c76a0 0x7f9f480af990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.691+0000 7f9f4dca3700 1 -- 172.21.15.106:0/2346427601 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f480c7e90 con 0x7f9f48068940 2024-01-26T23:29:28.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.691+0000 7f9f4d4a2700 1 --2- 172.21.15.106:0/2346427601 >> v2:172.21.15.106:3300/0 conn(0x7f9f480c8c50 0x7f9f480c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:29:28.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.691+0000 7f9f4dca3700 1 --2- 172.21.15.106:0/2346427601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f48068940 0x7f9f48068d10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9f4806dc80 tx=0x7f9f44009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4358d7fdee6e193 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:28.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.691+0000 7f9f4cca1700 1 -- 172.21.15.106:0/2346427601 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f44005c10 con 0x7f9f48068940 2024-01-26T23:29:28.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.691+0000 7f9f4cca1700 1 -- 172.21.15.106:0/2346427601 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f44005d70 con 0x7f9f48068940 2024-01-26T23:29:28.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.691+0000 7f9f4cca1700 1 -- 172.21.15.106:0/2346427601 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f44006260 con 0x7f9f48068940 2024-01-26T23:29:28.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.692+0000 7f9f4ff07700 1 -- 172.21.15.106:0/2346427601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f48068940 msgr2=0x7f9f48068d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.692+0000 7f9f4ff07700 1 --2- 172.21.15.106:0/2346427601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f48068940 0x7f9f48068d10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9f4806dc80 tx=0x7f9f44009120 comp rx=0 tx=0).stop 2024-01-26T23:29:28.695 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.692+0000 7f9f4ff07700 1 -- 172.21.15.106:0/2346427601 shutdown_connections 2024-01-26T23:29:28.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.692+0000 7f9f4ff07700 1 --2- 172.21.15.106:0/2346427601 >> v2:172.21.15.46:3300/0 conn(0x7f9f480c76a0 0x7f9f480af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.692+0000 7f9f4ff07700 1 --2- 172.21.15.106:0/2346427601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f48068940 0x7f9f48068d10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.692+0000 7f9f4ff07700 1 --2- 172.21.15.106:0/2346427601 >> v2:172.21.15.106:3300/0 conn(0x7f9f480c8c50 0x7f9f480c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.692+0000 7f9f4ff07700 1 -- 172.21.15.106:0/2346427601 >> 172.21.15.106:0/2346427601 conn(0x7f9f480ca100 msgr2=0x7f9f4806e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:28.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.692+0000 7f9f4ff07700 1 -- 172.21.15.106:0/2346427601 shutdown_connections 2024-01-26T23:29:28.696 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.692+0000 7f9f4ff07700 1 -- 172.21.15.106:0/2346427601 wait complete. 2024-01-26T23:29:28.698 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.693+0000 7f9f4ff07700 1 Processor -- start 2024-01-26T23:29:28.698 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.693+0000 7f9f4ff07700 1 -- start start 2024-01-26T23:29:28.698 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.694+0000 7f9f4ff07700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f48068940 0x7f9f48158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.698 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.694+0000 7f9f4ff07700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f480c76a0 0x7f9f480c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.698 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.694+0000 7f9f4ff07700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f480c8c50 0x7f9f480c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.698 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.694+0000 7f9f4ff07700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4806b9c0 con 0x7f9f480c76a0 2024-01-26T23:29:28.698 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.694+0000 7f9f4ff07700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f48158fd0 con 0x7f9f48068940 2024-01-26T23:29:28.698 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.694+0000 7f9f4ff07700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f480c0750 con 0x7f9f480c8c50 2024-01-26T23:29:28.698 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.694+0000 7f9f4dca3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f48068940 0x7f9f48158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.698 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.694+0000 7f9f4dca3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f48068940 0x7f9f48158a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:49304/0 (socket says 172.21.15.106:49304) 2024-01-26T23:29:28.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.694+0000 7f9f4dca3700 1 -- 172.21.15.106:0/1943194933 learned_addr learned my addr 172.21.15.106:0/1943194933 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-26T23:29:28.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.694+0000 7f9f4e4a4700 1 --2- 172.21.15.106:0/1943194933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f480c8c50 0x7f9f480c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.694+0000 7f9f4dca3700 1 -- 172.21.15.106:0/1943194933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f480c8c50 msgr2=0x7f9f480c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.694+0000 7f9f4dca3700 1 --2- 172.21.15.106:0/1943194933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f480c8c50 0x7f9f480c5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.694+0000 7f9f4dca3700 1 -- 172.21.15.106:0/1943194933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f480c76a0 msgr2=0x7f9f480c50f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:29:28.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.694+0000 7f9f4dca3700 1 --2- 172.21.15.106:0/1943194933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f480c76a0 0x7f9f480c50f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.694+0000 7f9f4dca3700 1 -- 172.21.15.106:0/1943194933 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f380094f0 con 0x7f9f48068940 2024-01-26T23:29:28.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.694+0000 7f9f4e4a4700 1 --2- 172.21.15.106:0/1943194933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f480c8c50 0x7f9f480c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:29:28.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.695+0000 7f9f4dca3700 1 --2- 172.21.15.106:0/1943194933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f48068940 0x7f9f48158a90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9f480c7cf0 tx=0x7f9f44016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:28.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.695+0000 7f9f3dffb700 1 -- 172.21.15.106:0/1943194933 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4401d780 con 0x7f9f48068940 2024-01-26T23:29:28.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.695+0000 7f9f3dffb700 1 -- 172.21.15.106:0/1943194933 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4401d900 con 0x7f9f48068940 2024-01-26T23:29:28.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.695+0000 7f9f4ff07700 1 -- 172.21.15.106:0/1943194933 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f440093c0 con 0x7f9f48068940 2024-01-26T23:29:28.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.695+0000 7f9f3dffb700 1 -- 172.21.15.106:0/1943194933 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4401dcc0 con 0x7f9f48068940 2024-01-26T23:29:28.699 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.695+0000 7f9f4ff07700 1 -- 172.21.15.106:0/1943194933 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f480c0d10 con 0x7f9f48068940 2024-01-26T23:29:28.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.695+0000 7f9f3dffb700 1 -- 172.21.15.106:0/1943194933 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9f4401b910 con 0x7f9f48068940 2024-01-26T23:29:28.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.696+0000 7f9f3dffb700 1 -- 172.21.15.106:0/1943194933 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f9f440269c0 con 0x7f9f48068940 2024-01-26T23:29:28.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.696+0000 7f9f4ff07700 1 -- 172.21.15.106:0/1943194933 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f4804c710 con 0x7f9f48068940 2024-01-26T23:29:28.700 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.698+0000 7f9f3dffb700 1 -- 172.21.15.106:0/1943194933 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9f4401bd50 con 0x7f9f48068940 2024-01-26T23:29:28.769 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:28.767+0000 7f507f3191c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:29:28.804 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.802+0000 7f9f4ff07700 1 -- 172.21.15.106:0/1943194933 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd new", "uuid": "647f327a-33e0-4b2c-a84b-d2636d18a18e", "id": 5} v 0) v1 -- 0x7f9f480c72b0 con 0x7f9f48068940 2024-01-26T23:29:28.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.808+0000 7f9f3dffb700 1 -- 172.21.15.106:0/1943194933 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "647f327a-33e0-4b2c-a84b-d2636d18a18e", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9f44026bd0 con 0x7f9f48068940 2024-01-26T23:29:28.810 INFO:teuthology.orchestra.run.smithi106.stdout:5 2024-01-26T23:29:28.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.809+0000 7f9f4ff07700 1 -- 172.21.15.106:0/1943194933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f48068940 msgr2=0x7f9f48158a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.809+0000 7f9f4ff07700 1 --2- 172.21.15.106:0/1943194933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f48068940 0x7f9f48158a90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9f480c7cf0 tx=0x7f9f44016000 comp rx=0 tx=0).stop 2024-01-26T23:29:28.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.809+0000 7f9f4ff07700 1 -- 172.21.15.106:0/1943194933 shutdown_connections 2024-01-26T23:29:28.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.809+0000 7f9f4ff07700 1 --2- 172.21.15.106:0/1943194933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f480c8c50 0x7f9f480c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.809+0000 7f9f4ff07700 1 --2- 172.21.15.106:0/1943194933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f48068940 0x7f9f48158a90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.809+0000 7f9f4ff07700 1 --2- 172.21.15.106:0/1943194933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f480c76a0 0x7f9f480c50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.809+0000 7f9f4ff07700 1 -- 172.21.15.106:0/1943194933 >> 172.21.15.106:0/1943194933 conn(0x7f9f480ca100 msgr2=0x7f9f4815bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:28.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.809+0000 7f9f4ff07700 1 -- 172.21.15.106:0/1943194933 shutdown_connections 2024-01-26T23:29:28.811 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.809+0000 7f9f4ff07700 1 -- 172.21.15.106:0/1943194933 wait complete. 2024-01-26T23:29:28.822 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd new d11ca954-7718-40e9-8579-083963a480f8 6 2024-01-26T23:29:28.856 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:28.855+0000 7efc42e4f1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:29:28.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.975+0000 7f0387337700 1 Processor -- start 2024-01-26T23:29:28.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.975+0000 7f0387337700 1 -- start start 2024-01-26T23:29:28.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.976+0000 7f0387337700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0380067af0 0x7f0380067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.976+0000 7f0387337700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f03801597e0 0x7f0380159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.976+0000 7f0387337700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f038006fdb0 con 0x7f038006ebe0 2024-01-26T23:29:28.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.976+0000 7f0387337700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f038006f1a0 con 0x7f03801597e0 2024-01-26T23:29:28.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.976+0000 7f0387337700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f038006f480 con 0x7f0380067af0 2024-01-26T23:29:28.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.976+0000 7f03850d3700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0380067af0 0x7f0380067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.976+0000 7f03848d2700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f038006ebe0 0x7f03800ad8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:37010/0 (socket says 172.21.15.106:37010) 2024-01-26T23:29:28.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.976+0000 7f03848d2700 1 -- 172.21.15.106:0/2081265818 learned_addr learned my addr 172.21.15.106:0/2081265818 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-01-26T23:29:28.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.976+0000 7f03858d4700 1 --2- 172.21.15.106:0/2081265818 >> v2:172.21.15.46:3300/0 conn(0x7f03801597e0 0x7f0380159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.976+0000 7f03850d3700 1 -- 172.21.15.106:0/2081265818 >> v2:172.21.15.46:3300/0 conn(0x7f03801597e0 msgr2=0x7f0380159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:28.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.976+0000 7f03850d3700 1 --2- 172.21.15.106:0/2081265818 >> v2:172.21.15.46:3300/0 conn(0x7f03801597e0 0x7f0380159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.976+0000 7f03850d3700 1 -- 172.21.15.106:0/2081265818 >> v1:172.21.15.106:6789/0 conn(0x7f038006ebe0 legacy=0x7f03800ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.976+0000 7f03850d3700 1 -- 172.21.15.106:0/2081265818 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038006f760 con 0x7f0380067af0 2024-01-26T23:29:28.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.976+0000 7f0377fff700 1 -- 172.21.15.106:0/2081265818 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857482300 0 0) 0x7f038006fdb0 con 0x7f038006ebe0 2024-01-26T23:29:28.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.976+0000 7f03858d4700 1 --2- 172.21.15.106:0/2081265818 >> v2:172.21.15.46:3300/0 conn(0x7f03801597e0 0x7f0380159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:29:28.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.977+0000 7f03850d3700 1 --2- 172.21.15.106:0/2081265818 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0380067af0 0x7f0380067ec0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0380064050 tx=0x7f037c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e1732c891b1e99f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:28.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.977+0000 7f0377fff700 1 -- 172.21.15.106:0/2081265818 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c00b6b0 con 0x7f0380067af0 2024-01-26T23:29:28.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.977+0000 7f0377fff700 1 -- 172.21.15.106:0/2081265818 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037c00b810 con 0x7f0380067af0 2024-01-26T23:29:28.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.977+0000 7f0377fff700 1 -- 172.21.15.106:0/2081265818 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c00bbb0 con 0x7f0380067af0 2024-01-26T23:29:28.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.977+0000 7f0387337700 1 -- 172.21.15.106:0/2081265818 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0380067af0 msgr2=0x7f0380067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.977+0000 7f0387337700 1 --2- 172.21.15.106:0/2081265818 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0380067af0 0x7f0380067ec0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0380064050 tx=0x7f037c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.977+0000 7f0387337700 1 -- 172.21.15.106:0/2081265818 shutdown_connections 2024-01-26T23:29:28.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.977+0000 7f0387337700 1 --2- 172.21.15.106:0/2081265818 >> v2:172.21.15.46:3300/0 conn(0x7f03801597e0 0x7f0380159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.977+0000 7f0387337700 1 --2- 172.21.15.106:0/2081265818 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0380067af0 0x7f0380067ec0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.977+0000 7f0387337700 1 -- 172.21.15.106:0/2081265818 >> 172.21.15.106:0/2081265818 conn(0x7f03800cdf80 msgr2=0x7f038006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:28.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.978+0000 7f0387337700 1 -- 172.21.15.106:0/2081265818 shutdown_connections 2024-01-26T23:29:28.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.978+0000 7f0387337700 1 -- 172.21.15.106:0/2081265818 wait complete. 2024-01-26T23:29:28.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.979+0000 7f0387337700 1 Processor -- start 2024-01-26T23:29:28.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.979+0000 7f0387337700 1 -- start start 2024-01-26T23:29:28.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.980+0000 7f0387337700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0380067af0 0x7f03801569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.980+0000 7f0387337700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f038006ebe0 0x7f0380152c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.980+0000 7f0387337700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03801597e0 0x7f038015c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:28.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.980+0000 7f0387337700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0380157a50 con 0x7f03801597e0 2024-01-26T23:29:28.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.980+0000 7f0387337700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0380159db0 con 0x7f0380067af0 2024-01-26T23:29:28.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.980+0000 7f0387337700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0380156f10 con 0x7f038006ebe0 2024-01-26T23:29:28.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.980+0000 7f03848d2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f038006ebe0 0x7f0380152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.980+0000 7f03850d3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0380067af0 0x7f03801569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.980+0000 7f03848d2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f038006ebe0 0x7f0380152c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:43990/0 (socket says 172.21.15.106:43990) 2024-01-26T23:29:28.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.980+0000 7f03850d3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0380067af0 0x7f03801569d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:49314/0 (socket says 172.21.15.106:49314) 2024-01-26T23:29:28.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.980+0000 7f03848d2700 1 -- 172.21.15.106:0/965868262 learned_addr learned my addr 172.21.15.106:0/965868262 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-26T23:29:28.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.980+0000 7f03858d4700 1 --2- 172.21.15.106:0/965868262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03801597e0 0x7f038015c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:28.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.981+0000 7f03850d3700 1 -- 172.21.15.106:0/965868262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f038006ebe0 msgr2=0x7f0380152c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.981+0000 7f03850d3700 1 --2- 172.21.15.106:0/965868262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f038006ebe0 0x7f0380152c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.981+0000 7f03850d3700 1 -- 172.21.15.106:0/965868262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03801597e0 msgr2=0x7f038015c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:28.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.981+0000 7f03850d3700 1 --2- 172.21.15.106:0/965868262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03801597e0 0x7f038015c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:28.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.981+0000 7f03850d3700 1 -- 172.21.15.106:0/965868262 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037c009420 con 0x7f0380067af0 2024-01-26T23:29:28.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.981+0000 7f03850d3700 1 --2- 172.21.15.106:0/965868262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0380067af0 0x7f03801569d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f037c016000 tx=0x7f037c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:28.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.981+0000 7f03757fa700 1 -- 172.21.15.106:0/965868262 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c006af0 con 0x7f0380067af0 2024-01-26T23:29:28.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.981+0000 7f03757fa700 1 -- 172.21.15.106:0/965868262 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037c000bc0 con 0x7f0380067af0 2024-01-26T23:29:28.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.981+0000 7f0387337700 1 -- 172.21.15.106:0/965868262 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03801571b0 con 0x7f0380067af0 2024-01-26T23:29:28.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.981+0000 7f03757fa700 1 -- 172.21.15.106:0/965868262 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c026430 con 0x7f0380067af0 2024-01-26T23:29:28.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.981+0000 7f0387337700 1 -- 172.21.15.106:0/965868262 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0380157770 con 0x7f0380067af0 2024-01-26T23:29:28.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.981+0000 7f03757fa700 1 -- 172.21.15.106:0/965868262 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f037c023030 con 0x7f0380067af0 2024-01-26T23:29:28.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.981+0000 7f03757fa700 1 -- 172.21.15.106:0/965868262 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f037c011a20 con 0x7f0380067af0 2024-01-26T23:29:28.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.982+0000 7f0387337700 1 -- 172.21.15.106:0/965868262 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f038004c710 con 0x7f0380067af0 2024-01-26T23:29:28.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:28.984+0000 7f03757fa700 1 -- 172.21.15.106:0/965868262 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f037c015020 con 0x7f0380067af0 2024-01-26T23:29:29.008 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:29.006+0000 7f507f3191c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:29:29.070 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.068+0000 7f0387337700 1 -- 172.21.15.106:0/965868262 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d11ca954-7718-40e9-8579-083963a480f8", "id": 6} v 0) v1 -- 0x7f038006a580 con 0x7f0380067af0 2024-01-26T23:29:29.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.074+0000 7f03757fa700 1 -- 172.21.15.106:0/965868262 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d11ca954-7718-40e9-8579-083963a480f8", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f037c026cc0 con 0x7f0380067af0 2024-01-26T23:29:29.076 INFO:teuthology.orchestra.run.smithi106.stdout:6 2024-01-26T23:29:29.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.075+0000 7f0387337700 1 -- 172.21.15.106:0/965868262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0380067af0 msgr2=0x7f03801569d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:29.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.075+0000 7f0387337700 1 --2- 172.21.15.106:0/965868262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0380067af0 0x7f03801569d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f037c016000 tx=0x7f037c017000 comp rx=0 tx=0).stop 2024-01-26T23:29:29.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.075+0000 7f0387337700 1 -- 172.21.15.106:0/965868262 shutdown_connections 2024-01-26T23:29:29.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.075+0000 7f0387337700 1 --2- 172.21.15.106:0/965868262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03801597e0 0x7f038015c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:29.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.075+0000 7f0387337700 1 --2- 172.21.15.106:0/965868262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0380067af0 0x7f03801569d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:29.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.075+0000 7f0387337700 1 --2- 172.21.15.106:0/965868262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f038006ebe0 0x7f0380152c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:29.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.075+0000 7f0387337700 1 -- 172.21.15.106:0/965868262 >> 172.21.15.106:0/965868262 conn(0x7f03800cdf80 msgr2=0x7f03800b2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:29.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.075+0000 7f0387337700 1 -- 172.21.15.106:0/965868262 shutdown_connections 2024-01-26T23:29:29.078 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.075+0000 7f0387337700 1 -- 172.21.15.106:0/965868262 wait complete. 2024-01-26T23:29:29.088 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd new dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724 7 2024-01-26T23:29:29.094 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:29.093+0000 7efc42e4f1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:29:29.177 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:29.176+0000 7efc42e4f1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:29:29.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.204+0000 7f974f63a700 1 Processor -- start 2024-01-26T23:29:29.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.204+0000 7f974f63a700 1 -- start start 2024-01-26T23:29:29.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.205+0000 7f974f63a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9748068940 0x7f9748068d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:29.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.205+0000 7f974f63a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97480c87c0 con 0x7f97480c8c50 2024-01-26T23:29:29.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.205+0000 7f974f63a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9748069250 con 0x7f97480c90c0 2024-01-26T23:29:29.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.205+0000 7f974f63a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97480c7bb0 con 0x7f9748068940 2024-01-26T23:29:29.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.205+0000 7f974dbd7700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f97480c90c0 0x7f97480af990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:37024/0 (socket says 172.21.15.106:37024) 2024-01-26T23:29:29.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.205+0000 7f974dbd7700 1 -- 172.21.15.106:0/4064606438 learned_addr learned my addr 172.21.15.106:0/4064606438 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-01-26T23:29:29.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.205+0000 7f974d3d6700 1 --2- 172.21.15.106:0/4064606438 >> v2:172.21.15.46:3300/0 conn(0x7f9748068940 0x7f9748068d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:29.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.205+0000 7f973ffff700 1 -- 172.21.15.106:0/4064606438 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006997585 0 0) 0x7f9748069250 con 0x7f97480c90c0 2024-01-26T23:29:29.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.205+0000 7f973ffff700 1 -- 172.21.15.106:0/4064606438 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9734003570 con 0x7f97480c90c0 2024-01-26T23:29:29.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.206+0000 7f973ffff700 1 -- 172.21.15.106:0/4064606438 <== mon.1 v1:172.21.15.106:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1060374320 0 0) 0x7f9734003570 con 0x7f97480c90c0 2024-01-26T23:29:29.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.206+0000 7f974d3d6700 1 -- 172.21.15.106:0/4064606438 >> v1:172.21.15.106:6789/0 conn(0x7f97480c90c0 legacy=0x7f97480af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:29.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.206+0000 7f974d3d6700 1 -- 172.21.15.106:0/4064606438 >> v1:172.21.15.46:6789/0 conn(0x7f97480c8c50 legacy=0x7f97480c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:29.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.206+0000 7f974d3d6700 1 -- 172.21.15.106:0/4064606438 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97480c7e90 con 0x7f9748068940 2024-01-26T23:29:29.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.206+0000 7f974d3d6700 1 --2- 172.21.15.106:0/4064606438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9748068940 0x7f9748068d10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9738009850 tx=0x7f9738009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c815a422ea7da2 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:29.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.206+0000 7f973ffff700 1 -- 172.21.15.106:0/4064606438 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97400017f0 con 0x7f9748068940 2024-01-26T23:29:29.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.206+0000 7f973ffff700 1 -- 172.21.15.106:0/4064606438 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9738011420 con 0x7f9748068940 2024-01-26T23:29:29.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.206+0000 7f973ffff700 1 -- 172.21.15.106:0/4064606438 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97380117c0 con 0x7f9748068940 2024-01-26T23:29:29.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.206+0000 7f974f63a700 1 -- 172.21.15.106:0/4064606438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9748068940 msgr2=0x7f9748068d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:29.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.206+0000 7f974f63a700 1 --2- 172.21.15.106:0/4064606438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9748068940 0x7f9748068d10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9738009850 tx=0x7f9738009120 comp rx=0 tx=0).stop 2024-01-26T23:29:29.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.207+0000 7f974f63a700 1 -- 172.21.15.106:0/4064606438 shutdown_connections 2024-01-26T23:29:29.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.207+0000 7f974f63a700 1 --2- 172.21.15.106:0/4064606438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9748068940 0x7f9748068d10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:29.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.207+0000 7f974f63a700 1 -- 172.21.15.106:0/4064606438 >> 172.21.15.106:0/4064606438 conn(0x7f97480ca100 msgr2=0x7f974806e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:29.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.207+0000 7f974f63a700 1 -- 172.21.15.106:0/4064606438 shutdown_connections 2024-01-26T23:29:29.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.207+0000 7f974f63a700 1 -- 172.21.15.106:0/4064606438 wait complete. 2024-01-26T23:29:29.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.208+0000 7f974f63a700 1 Processor -- start 2024-01-26T23:29:29.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.208+0000 7f974f63a700 1 -- start start 2024-01-26T23:29:29.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974f63a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9748068940 0x7f974815aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:29.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974f63a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97480c8c50 0x7f97480c20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:29.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974f63a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97480c90c0 0x7f97480c2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:29.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974f63a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97480c38f0 con 0x7f9748068940 2024-01-26T23:29:29.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974f63a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974815b000 con 0x7f97480c8c50 2024-01-26T23:29:29.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974f63a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974806bcc0 con 0x7f97480c90c0 2024-01-26T23:29:29.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974cbd5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97480c8c50 0x7f97480c20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:29.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974dbd7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97480c90c0 0x7f97480c2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:29.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974dbd7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97480c90c0 0x7f97480c2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:43998/0 (socket says 172.21.15.106:43998) 2024-01-26T23:29:29.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974cbd5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97480c8c50 0x7f97480c20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:49326/0 (socket says 172.21.15.106:49326) 2024-01-26T23:29:29.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974dbd7700 1 -- 172.21.15.106:0/4062521720 learned_addr learned my addr 172.21.15.106:0/4062521720 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-26T23:29:29.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974d3d6700 1 --2- 172.21.15.106:0/4062521720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9748068940 0x7f974815aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:29.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974cbd5700 1 -- 172.21.15.106:0/4062521720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97480c90c0 msgr2=0x7f97480c2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:29.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974cbd5700 1 --2- 172.21.15.106:0/4062521720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97480c90c0 0x7f97480c2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:29.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974cbd5700 1 -- 172.21.15.106:0/4062521720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9748068940 msgr2=0x7f974815aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:29.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974cbd5700 1 --2- 172.21.15.106:0/4062521720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9748068940 0x7f974815aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:29.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974cbd5700 1 -- 172.21.15.106:0/4062521720 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97380093c0 con 0x7f97480c8c50 2024-01-26T23:29:29.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974dbd7700 1 --2- 172.21.15.106:0/4062521720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97480c90c0 0x7f97480c2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:29:29.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f974cbd5700 1 --2- 172.21.15.106:0/4062521720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97480c8c50 0x7f97480c20f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f974400c590 tx=0x7f9744007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:29.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.209+0000 7f973d7fa700 1 -- 172.21.15.106:0/4062521720 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9744011590 con 0x7f97480c8c50 2024-01-26T23:29:29.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.210+0000 7f973d7fa700 1 -- 172.21.15.106:0/4062521720 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9744011710 con 0x7f97480c8c50 2024-01-26T23:29:29.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.210+0000 7f973d7fa700 1 -- 172.21.15.106:0/4062521720 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9744011ad0 con 0x7f97480c8c50 2024-01-26T23:29:29.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.210+0000 7f974f63a700 1 -- 172.21.15.106:0/4062521720 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974806b140 con 0x7f97480c8c50 2024-01-26T23:29:29.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.210+0000 7f974f63a700 1 -- 172.21.15.106:0/4062521720 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974806b700 con 0x7f97480c8c50 2024-01-26T23:29:29.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.210+0000 7f973d7fa700 1 -- 172.21.15.106:0/4062521720 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f974401c6c0 con 0x7f97480c8c50 2024-01-26T23:29:29.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.210+0000 7f973d7fa700 1 -- 172.21.15.106:0/4062521720 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7f9744012a90 con 0x7f97480c8c50 2024-01-26T23:29:29.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.210+0000 7f974f63a700 1 -- 172.21.15.106:0/4062521720 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974804c710 con 0x7f97480c8c50 2024-01-26T23:29:29.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.213+0000 7f973d7fa700 1 -- 172.21.15.106:0/4062521720 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9744017070 con 0x7f97480c8c50 2024-01-26T23:29:29.267 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:29.265+0000 7f507f3191c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:29:29.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.295+0000 7f974f63a700 1 -- 172.21.15.106:0/4062521720 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd new", "uuid": "dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724", "id": 7} v 0) v1 -- 0x7f97480c6360 con 0x7f97480c8c50 2024-01-26T23:29:29.304 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.303+0000 7f973d7fa700 1 -- 172.21.15.106:0/4062521720 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f974401cba0 con 0x7f97480c8c50 2024-01-26T23:29:29.304 INFO:teuthology.orchestra.run.smithi106.stdout:7 2024-01-26T23:29:29.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.304+0000 7f974f63a700 1 -- 172.21.15.106:0/4062521720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97480c8c50 msgr2=0x7f97480c20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:29.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.304+0000 7f974f63a700 1 --2- 172.21.15.106:0/4062521720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97480c8c50 0x7f97480c20f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f974400c590 tx=0x7f9744007ec0 comp rx=0 tx=0).stop 2024-01-26T23:29:29.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.304+0000 7f974f63a700 1 -- 172.21.15.106:0/4062521720 shutdown_connections 2024-01-26T23:29:29.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.304+0000 7f974f63a700 1 --2- 172.21.15.106:0/4062521720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9748068940 0x7f974815aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:29.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.304+0000 7f974f63a700 1 --2- 172.21.15.106:0/4062521720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97480c90c0 0x7f97480c2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:29.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.304+0000 7f974f63a700 1 --2- 172.21.15.106:0/4062521720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97480c8c50 0x7f97480c20f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:29.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.304+0000 7f974f63a700 1 -- 172.21.15.106:0/4062521720 >> 172.21.15.106:0/4062521720 conn(0x7f97480ca100 msgr2=0x7f97480af970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:29.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.304+0000 7f974f63a700 1 -- 172.21.15.106:0/4062521720 shutdown_connections 2024-01-26T23:29:29.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-26T23:29:29.304+0000 7f974f63a700 1 -- 172.21.15.106:0/4062521720 wait complete. 2024-01-26T23:29:29.317 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-26T23:29:29.317 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-01-26T23:29:29.319 INFO:tasks.ceph.osd.0:Started 2024-01-26T23:29:29.319 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-26T23:29:29.319 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-01-26T23:29:29.322 INFO:tasks.ceph.osd.1:Started 2024-01-26T23:29:29.322 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-26T23:29:29.322 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-01-26T23:29:29.324 INFO:tasks.ceph.osd.2:Started 2024-01-26T23:29:29.324 INFO:tasks.ceph.osd.3:Restarting daemon 2024-01-26T23:29:29.324 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-01-26T23:29:29.326 INFO:tasks.ceph.osd.3:Started 2024-01-26T23:29:29.327 INFO:tasks.ceph.osd.4:Restarting daemon 2024-01-26T23:29:29.327 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-01-26T23:29:29.359 INFO:tasks.ceph.osd.4:Started 2024-01-26T23:29:29.359 INFO:tasks.ceph.osd.5:Restarting daemon 2024-01-26T23:29:29.360 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-01-26T23:29:29.361 INFO:tasks.ceph.osd.5:Started 2024-01-26T23:29:29.362 INFO:tasks.ceph.osd.6:Restarting daemon 2024-01-26T23:29:29.362 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-01-26T23:29:29.364 INFO:tasks.ceph.osd.6:Started 2024-01-26T23:29:29.364 INFO:tasks.ceph.osd.7:Restarting daemon 2024-01-26T23:29:29.364 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-01-26T23:29:29.364 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:29.362+0000 7f507f3191c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:29:29.366 INFO:tasks.ceph.osd.7:Started 2024-01-26T23:29:29.367 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:29:29.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.522+0000 7f4a7d6f1700 1 Processor -- start 2024-01-26T23:29:29.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.522+0000 7f4a7d6f1700 1 -- start start 2024-01-26T23:29:29.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.523+0000 7f4a7d6f1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a780cbdf0 0x7f4a780ae610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:29.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.523+0000 7f4a7d6f1700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a780ae180 con 0x7f4a780cc1c0 2024-01-26T23:29:29.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.523+0000 7f4a7d6f1700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a78156c60 con 0x7f4a780ad070 2024-01-26T23:29:29.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.523+0000 7f4a7d6f1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a780aeb50 con 0x7f4a780cbdf0 2024-01-26T23:29:29.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.523+0000 7f4a767fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4a780cc1c0 0x7f4a780b0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57354/0 (socket says 172.21.15.46:57354) 2024-01-26T23:29:29.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.523+0000 7f4a767fc700 1 -- 172.21.15.46:0/1771642059 learned_addr learned my addr 172.21.15.46:0/1771642059 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:29.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.523+0000 7f4a76ffd700 1 --2- 172.21.15.46:0/1771642059 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a780cbdf0 0x7f4a780ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:29.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.523+0000 7f4a75ffb700 1 -- 172.21.15.46:0/1771642059 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185664440 0 0) 0x7f4a780ae180 con 0x7f4a780cc1c0 2024-01-26T23:29:29.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.523+0000 7f4a75ffb700 1 -- 172.21.15.46:0/1771642059 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a60003570 con 0x7f4a780cc1c0 2024-01-26T23:29:29.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.523+0000 7f4a75ffb700 1 -- 172.21.15.46:0/1771642059 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735349013 0 0) 0x7f4a78156c60 con 0x7f4a780ad070 2024-01-26T23:29:29.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.523+0000 7f4a75ffb700 1 -- 172.21.15.46:0/1771642059 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a780ae180 con 0x7f4a780ad070 2024-01-26T23:29:29.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.523+0000 7f4a76ffd700 1 -- 172.21.15.46:0/1771642059 >> v1:172.21.15.106:6789/0 conn(0x7f4a780ad070 legacy=0x7f4a780c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:29.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.523+0000 7f4a76ffd700 1 -- 172.21.15.46:0/1771642059 >> v1:172.21.15.46:6789/0 conn(0x7f4a780cc1c0 legacy=0x7f4a780b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:29.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.523+0000 7f4a76ffd700 1 -- 172.21.15.46:0/1771642059 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a780ad570 con 0x7f4a780cbdf0 2024-01-26T23:29:29.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.524+0000 7f4a76ffd700 1 --2- 172.21.15.46:0/1771642059 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a780cbdf0 0x7f4a780ae610 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009910 tx=0x7f4a680091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40f6fa8aa99a94c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:29.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.524+0000 7f4a75ffb700 1 -- 172.21.15.46:0/1771642059 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a68011630 con 0x7f4a780cbdf0 2024-01-26T23:29:29.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.524+0000 7f4a75ffb700 1 -- 172.21.15.46:0/1771642059 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a68011790 con 0x7f4a780cbdf0 2024-01-26T23:29:29.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.524+0000 7f4a75ffb700 1 -- 172.21.15.46:0/1771642059 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a68011b30 con 0x7f4a780cbdf0 2024-01-26T23:29:29.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.524+0000 7f4a7d6f1700 1 -- 172.21.15.46:0/1771642059 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a780cbdf0 msgr2=0x7f4a780ae610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:29.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.524+0000 7f4a7d6f1700 1 --2- 172.21.15.46:0/1771642059 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a780cbdf0 0x7f4a780ae610 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009910 tx=0x7f4a680091a0 comp rx=0 tx=0).stop 2024-01-26T23:29:29.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.525+0000 7f4a7d6f1700 1 -- 172.21.15.46:0/1771642059 shutdown_connections 2024-01-26T23:29:29.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.525+0000 7f4a7d6f1700 1 --2- 172.21.15.46:0/1771642059 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a780cbdf0 0x7f4a780ae610 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.525+0000 7f4a7d6f1700 1 -- 172.21.15.46:0/1771642059 >> 172.21.15.46:0/1771642059 conn(0x7f4a78068400 msgr2=0x7f4a780cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.525+0000 7f4a7d6f1700 1 -- 172.21.15.46:0/1771642059 shutdown_connections 2024-01-26T23:29:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.525+0000 7f4a7d6f1700 1 -- 172.21.15.46:0/1771642059 wait complete. 2024-01-26T23:29:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.526+0000 7f4a7d6f1700 1 Processor -- start 2024-01-26T23:29:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.526+0000 7f4a7d6f1700 1 -- start start 2024-01-26T23:29:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.527+0000 7f4a7d6f1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a780ad070 0x7f4a781531a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.527+0000 7f4a7d6f1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a780cbdf0 0x7f4a780cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.527+0000 7f4a7d6f1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a780a7440 0x7f4a780ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.527+0000 7f4a7d6f1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7816c350 con 0x7f4a780ad070 2024-01-26T23:29:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.527+0000 7f4a7d6f1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7816b780 con 0x7f4a780a7440 2024-01-26T23:29:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.527+0000 7f4a7d6f1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7816ba20 con 0x7f4a780cbdf0 2024-01-26T23:29:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.527+0000 7f4a76ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a780ad070 0x7f4a781531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.527+0000 7f4a76ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a780ad070 0x7f4a781531a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46946/0 (socket says 172.21.15.46:46946) 2024-01-26T23:29:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.527+0000 7f4a76ffd700 1 -- 172.21.15.46:0/582687767 learned_addr learned my addr 172.21.15.46:0/582687767 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.527+0000 7f4a767fc700 1 --2- 172.21.15.46:0/582687767 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a780cbdf0 0x7f4a780cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.527+0000 7f4a777fe700 1 --2- 172.21.15.46:0/582687767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a780a7440 0x7f4a780ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.527+0000 7f4a76ffd700 1 -- 172.21.15.46:0/582687767 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a780cbdf0 msgr2=0x7f4a780cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.527+0000 7f4a76ffd700 1 --2- 172.21.15.46:0/582687767 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a780cbdf0 0x7f4a780cae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.527+0000 7f4a76ffd700 1 -- 172.21.15.46:0/582687767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a780a7440 msgr2=0x7f4a780ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.527+0000 7f4a76ffd700 1 --2- 172.21.15.46:0/582687767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a780a7440 0x7f4a780ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.527+0000 7f4a76ffd700 1 -- 172.21.15.46:0/582687767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a68009420 con 0x7f4a780ad070 2024-01-26T23:29:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.527+0000 7f4a76ffd700 1 --2- 172.21.15.46:0/582687767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a780ad070 0x7f4a781531a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009f00 tx=0x7f4a68017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.528+0000 7f4a5effd700 1 -- 172.21.15.46:0/582687767 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6801f070 con 0x7f4a780ad070 2024-01-26T23:29:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.528+0000 7f4a5effd700 1 -- 172.21.15.46:0/582687767 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a68002f70 con 0x7f4a780ad070 2024-01-26T23:29:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.528+0000 7f4a7d6f1700 1 -- 172.21.15.46:0/582687767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a7816bd00 con 0x7f4a780ad070 2024-01-26T23:29:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.528+0000 7f4a5effd700 1 -- 172.21.15.46:0/582687767 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a68011900 con 0x7f4a780ad070 2024-01-26T23:29:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.528+0000 7f4a7d6f1700 1 -- 172.21.15.46:0/582687767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a781536e0 con 0x7f4a780ad070 2024-01-26T23:29:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.528+0000 7f4a5effd700 1 -- 172.21.15.46:0/582687767 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4a68023070 con 0x7f4a780ad070 2024-01-26T23:29:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.528+0000 7f4a5effd700 1 -- 172.21.15.46:0/582687767 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f4a68011a60 con 0x7f4a780ad070 2024-01-26T23:29:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.529+0000 7f4a7d6f1700 1 -- 172.21.15.46:0/582687767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a7804c710 con 0x7f4a780ad070 2024-01-26T23:29:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.531+0000 7f4a5effd700 1 -- 172.21.15.46:0/582687767 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f4a6802a070 con 0x7f4a780ad070 2024-01-26T23:29:29.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.605+0000 7f4a7d6f1700 1 -- 172.21.15.46:0/582687767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a780ad850 con 0x7f4a780ad070 2024-01-26T23:29:29.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.606+0000 7f4a5effd700 1 -- 172.21.15.46:0/582687767 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f4a68015020 con 0x7f4a780ad070 2024-01-26T23:29:29.607 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:29:29.608 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":10,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:29.296944+0000","last_up_change":"0.000000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:29:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.606+0000 7f4a7d6f1700 1 -- 172.21.15.46:0/582687767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a780ad070 msgr2=0x7f4a781531a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.606+0000 7f4a7d6f1700 1 --2- 172.21.15.46:0/582687767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a780ad070 0x7f4a781531a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009f00 tx=0x7f4a68017000 comp rx=0 tx=0).stop 2024-01-26T23:29:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.607+0000 7f4a7d6f1700 1 -- 172.21.15.46:0/582687767 shutdown_connections 2024-01-26T23:29:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.607+0000 7f4a7d6f1700 1 --2- 172.21.15.46:0/582687767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a780ad070 0x7f4a781531a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.607+0000 7f4a7d6f1700 1 --2- 172.21.15.46:0/582687767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a780a7440 0x7f4a780ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.607+0000 7f4a7d6f1700 1 --2- 172.21.15.46:0/582687767 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a780cbdf0 0x7f4a780cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.607+0000 7f4a7d6f1700 1 -- 172.21.15.46:0/582687767 >> 172.21.15.46:0/582687767 conn(0x7f4a78068400 msgr2=0x7f4a7806b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.607+0000 7f4a7d6f1700 1 -- 172.21.15.46:0/582687767 shutdown_connections 2024-01-26T23:29:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:29.607+0000 7f4a7d6f1700 1 -- 172.21.15.46:0/582687767 wait complete. 2024-01-26T23:29:29.622 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-26T23:29:29.622 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-26T23:29:29.877 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:29.875+0000 7efc42e4f1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:29:29.934 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-01-26T23:29:29.932+0000 7f767f0a0540 -1 Falling back to public interface 2024-01-26T23:29:29.955 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:29.953+0000 7efc42e4f1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:29:29.958 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-01-26T23:29:29.957+0000 7f6bff23b540 -1 Falling back to public interface 2024-01-26T23:29:29.959 INFO:tasks.ceph.osd.5.smithi106.stderr:2024-01-26T23:29:29.958+0000 7f8ef2382540 -1 Falling back to public interface 2024-01-26T23:29:29.960 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-01-26T23:29:29.958+0000 7eff807a7540 -1 Falling back to public interface 2024-01-26T23:29:29.961 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-01-26T23:29:29.960+0000 7f9ab52cf540 -1 Falling back to public interface 2024-01-26T23:29:29.962 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-01-26T23:29:29.961+0000 7eff7abc7700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6822/0: (98) Address already in use 2024-01-26T23:29:29.963 INFO:tasks.ceph.osd.4.smithi106.stderr:2024-01-26T23:29:29.962+0000 7f984fe19540 -1 Falling back to public interface 2024-01-26T23:29:29.974 INFO:tasks.ceph.osd.6.smithi106.stderr:2024-01-26T23:29:29.973+0000 7f53465f1540 -1 Falling back to public interface 2024-01-26T23:29:29.983 INFO:tasks.ceph.osd.7.smithi106.stderr:2024-01-26T23:29:29.982+0000 7efca0f40540 -1 Falling back to public interface 2024-01-26T23:29:30.089 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:30.088+0000 7efc42e4f1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:29:30.163 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:30.161+0000 7f507f3191c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:29:30.253 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:30.251+0000 7f507f3191c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:29:30.401 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:30.400+0000 7f507f3191c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:29:30.428 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-26T23:29:30.550 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.548+0000 7f5c5b1a3700 1 Processor -- start 2024-01-26T23:29:30.550 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.548+0000 7f5c5b1a3700 1 -- start start 2024-01-26T23:29:30.552 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.549+0000 7f5c5b1a3700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5c540c8a90 0x7f5c540c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:30.553 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.549+0000 7f5c5b1a3700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c540cc300 con 0x7f5c541519d0 2024-01-26T23:29:30.553 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.549+0000 7f5c5b1a3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c540ccba0 con 0x7f5c54151da0 2024-01-26T23:29:30.553 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.549+0000 7f5c5b1a3700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c54158a00 con 0x7f5c540c8a90 2024-01-26T23:29:30.553 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.549+0000 7f5c58f3f700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5c541519d0 0x7f5c540cd030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34582/0 (socket says 172.21.15.46:34582) 2024-01-26T23:29:30.553 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.549+0000 7f5c58f3f700 1 -- 172.21.15.46:0/2823614376 learned_addr learned my addr 172.21.15.46:0/2823614376 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:30.553 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.549+0000 7f5c59740700 1 --2- 172.21.15.46:0/2823614376 >> v2:172.21.15.106:3300/0 conn(0x7f5c540c8a90 0x7f5c540c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:30.553 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.549+0000 7f5c537fe700 1 -- 172.21.15.46:0/2823614376 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983412287 0 0) 0x7f5c540cc300 con 0x7f5c541519d0 2024-01-26T23:29:30.553 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.549+0000 7f5c537fe700 1 -- 172.21.15.46:0/2823614376 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c44003570 con 0x7f5c541519d0 2024-01-26T23:29:30.553 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.549+0000 7f5c537fe700 1 -- 172.21.15.46:0/2823614376 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437155000 0 0) 0x7f5c540ccba0 con 0x7f5c54151da0 2024-01-26T23:29:30.553 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.549+0000 7f5c537fe700 1 -- 172.21.15.46:0/2823614376 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c540cc300 con 0x7f5c54151da0 2024-01-26T23:29:30.554 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.549+0000 7f5c59740700 1 -- 172.21.15.46:0/2823614376 >> v1:172.21.15.106:6789/0 conn(0x7f5c54151da0 legacy=0x7f5c5406bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:30.554 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.549+0000 7f5c59740700 1 -- 172.21.15.46:0/2823614376 >> v1:172.21.15.46:6789/0 conn(0x7f5c541519d0 legacy=0x7f5c540cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:30.554 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.550+0000 7f5c59740700 1 -- 172.21.15.46:0/2823614376 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c540c9450 con 0x7f5c540c8a90 2024-01-26T23:29:30.554 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.550+0000 7f5c59740700 1 --2- 172.21.15.46:0/2823614376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c540c8a90 0x7f5c540c8f10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009e60 tx=0x7f5c4c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7e2f184c933a942 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:30.554 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.550+0000 7f5c537fe700 1 -- 172.21.15.46:0/2823614376 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c00e070 con 0x7f5c540c8a90 2024-01-26T23:29:30.554 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.550+0000 7f5c537fe700 1 -- 172.21.15.46:0/2823614376 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4c00bab0 con 0x7f5c540c8a90 2024-01-26T23:29:30.554 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.550+0000 7f5c537fe700 1 -- 172.21.15.46:0/2823614376 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c00be50 con 0x7f5c540c8a90 2024-01-26T23:29:30.554 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.550+0000 7f5c5b1a3700 1 -- 172.21.15.46:0/2823614376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c540c8a90 msgr2=0x7f5c540c8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:30.554 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.551+0000 7f5c5b1a3700 1 --2- 172.21.15.46:0/2823614376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c540c8a90 0x7f5c540c8f10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009e60 tx=0x7f5c4c009780 comp rx=0 tx=0).stop 2024-01-26T23:29:30.554 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.551+0000 7f5c5b1a3700 1 -- 172.21.15.46:0/2823614376 shutdown_connections 2024-01-26T23:29:30.554 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.551+0000 7f5c5b1a3700 1 --2- 172.21.15.46:0/2823614376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c540c8a90 0x7f5c540c8f10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:30.554 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.551+0000 7f5c5b1a3700 1 -- 172.21.15.46:0/2823614376 >> 172.21.15.46:0/2823614376 conn(0x7f5c540ce450 msgr2=0x7f5c540ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:30.554 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.551+0000 7f5c5b1a3700 1 -- 172.21.15.46:0/2823614376 shutdown_connections 2024-01-26T23:29:30.554 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.551+0000 7f5c5b1a3700 1 -- 172.21.15.46:0/2823614376 wait complete. 2024-01-26T23:29:30.557 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.552+0000 7f5c5b1a3700 1 Processor -- start 2024-01-26T23:29:30.557 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.552+0000 7f5c5b1a3700 1 -- start start 2024-01-26T23:29:30.557 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.553+0000 7f5c5b1a3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5c540c8a90 0x7f5c540c5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:30.558 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.553+0000 7f5c5b1a3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c541519d0 0x7f5c54146de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:30.558 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.553+0000 7f5c5b1a3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c540c31f0 0x7f5c540c3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:30.558 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.553+0000 7f5c5b1a3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c540c3bd0 con 0x7f5c540c31f0 2024-01-26T23:29:30.558 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.553+0000 7f5c5b1a3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c540699e0 con 0x7f5c541519d0 2024-01-26T23:29:30.558 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.553+0000 7f5c5b1a3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c540c5a40 con 0x7f5c540c8a90 2024-01-26T23:29:30.558 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.553+0000 7f5c59740700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c540c31f0 0x7f5c540c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:30.558 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.553+0000 7f5c59740700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c540c31f0 0x7f5c540c3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37224/0 (socket says 172.21.15.46:37224) 2024-01-26T23:29:30.558 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.553+0000 7f5c59740700 1 -- 172.21.15.46:0/139998179 learned_addr learned my addr 172.21.15.46:0/139998179 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:30.559 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.553+0000 7f5c58f3f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5c540c8a90 0x7f5c540c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:30.559 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.553+0000 7f5c53fff700 1 --2- 172.21.15.46:0/139998179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c541519d0 0x7f5c54146de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:30.559 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.553+0000 7f5c59740700 1 -- 172.21.15.46:0/139998179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5c540c8a90 msgr2=0x7f5c540c5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:30.559 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.553+0000 7f5c59740700 1 --2- 172.21.15.46:0/139998179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5c540c8a90 0x7f5c540c5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:30.559 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.553+0000 7f5c59740700 1 -- 172.21.15.46:0/139998179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c541519d0 msgr2=0x7f5c54146de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:30.559 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.553+0000 7f5c59740700 1 --2- 172.21.15.46:0/139998179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c541519d0 0x7f5c54146de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:30.559 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.553+0000 7f5c59740700 1 -- 172.21.15.46:0/139998179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c009a20 con 0x7f5c540c31f0 2024-01-26T23:29:30.559 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.553+0000 7f5c58f3f700 1 --2- 172.21.15.46:0/139998179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5c540c8a90 0x7f5c540c5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:29:30.560 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.554+0000 7f5c59740700 1 --2- 172.21.15.46:0/139998179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c540c31f0 0x7f5c540c3690 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c00b610 tx=0x7f5c4c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:30.560 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.554+0000 7f5c50ff9700 1 -- 172.21.15.46:0/139998179 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c012d30 con 0x7f5c540c31f0 2024-01-26T23:29:30.560 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.554+0000 7f5c50ff9700 1 -- 172.21.15.46:0/139998179 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4c012e90 con 0x7f5c540c31f0 2024-01-26T23:29:30.560 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.554+0000 7f5c5b1a3700 1 -- 172.21.15.46:0/139998179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c540690b0 con 0x7f5c540c31f0 2024-01-26T23:29:30.560 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.554+0000 7f5c50ff9700 1 -- 172.21.15.46:0/139998179 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c024630 con 0x7f5c540c31f0 2024-01-26T23:29:30.560 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.554+0000 7f5c5b1a3700 1 -- 172.21.15.46:0/139998179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c54069670 con 0x7f5c540c31f0 2024-01-26T23:29:30.560 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.554+0000 7f5c50ff9700 1 -- 172.21.15.46:0/139998179 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f5c4c024a60 con 0x7f5c540c31f0 2024-01-26T23:29:30.560 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.554+0000 7f5c50ff9700 1 -- 172.21.15.46:0/139998179 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f5c4c009800 con 0x7f5c540c31f0 2024-01-26T23:29:30.561 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.555+0000 7f5c5b1a3700 1 -- 172.21.15.46:0/139998179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c54069b90 con 0x7f5c540c31f0 2024-01-26T23:29:30.561 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.559+0000 7f5c50ff9700 1 -- 172.21.15.46:0/139998179 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f5c4c024d10 con 0x7f5c540c31f0 2024-01-26T23:29:30.635 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.633+0000 7f5c5b1a3700 1 -- 172.21.15.46:0/139998179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c540713b0 con 0x7f5c540c31f0 2024-01-26T23:29:30.636 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.634+0000 7f5c50ff9700 1 -- 172.21.15.46:0/139998179 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f5c4c036450 con 0x7f5c540c31f0 2024-01-26T23:29:30.636 INFO:teuthology.misc.health.smithi046.stdout: 2024-01-26T23:29:30.636 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:29.296944+0000","last_up_change":"0.000000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:29:30.637 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.635+0000 7f5c5b1a3700 1 -- 172.21.15.46:0/139998179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c540c31f0 msgr2=0x7f5c540c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:30.637 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.635+0000 7f5c5b1a3700 1 --2- 172.21.15.46:0/139998179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c540c31f0 0x7f5c540c3690 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c00b610 tx=0x7f5c4c019000 comp rx=0 tx=0).stop 2024-01-26T23:29:30.638 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.635+0000 7f5c5b1a3700 1 -- 172.21.15.46:0/139998179 shutdown_connections 2024-01-26T23:29:30.638 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.635+0000 7f5c5b1a3700 1 --2- 172.21.15.46:0/139998179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c540c31f0 0x7f5c540c3690 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:30.638 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.635+0000 7f5c5b1a3700 1 --2- 172.21.15.46:0/139998179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5c540c8a90 0x7f5c540c5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:30.638 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.635+0000 7f5c5b1a3700 1 --2- 172.21.15.46:0/139998179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c541519d0 0x7f5c54146de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:30.638 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.635+0000 7f5c5b1a3700 1 -- 172.21.15.46:0/139998179 >> 172.21.15.46:0/139998179 conn(0x7f5c540ce450 msgr2=0x7f5c54159020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:30.638 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.636+0000 7f5c5b1a3700 1 -- 172.21.15.46:0/139998179 shutdown_connections 2024-01-26T23:29:30.638 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:30.636+0000 7f5c5b1a3700 1 -- 172.21.15.46:0/139998179 wait complete. 2024-01-26T23:29:30.641 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:30.640+0000 7efc42e4f1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:29:30.650 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-26T23:29:30.804 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:30.803+0000 7efc42e4f1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:29:30.965 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:30.964+0000 7efc42e4f1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:29:31.061 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:31.059+0000 7f507f3191c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:29:31.248 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:31.247+0000 7f507f3191c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:29:31.430 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:31.428+0000 7f507f3191c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:29:31.753 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:31.752+0000 7efc42e4f1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:29:31.971 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:31.969+0000 7efc42e4f1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:29:32.042 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:32.040+0000 7efc42e4f1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:29:32.201 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:32.200+0000 7efc42e4f1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:29:32.324 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:32.322+0000 7f507f3191c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:29:32.577 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:32.575+0000 7f507f3191c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:29:32.663 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:32.662+0000 7f507f3191c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:29:32.882 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:29:32.880+0000 7f507f3191c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:29:33.228 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:33.227+0000 7efc01906700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-01-26T23:29:33.228 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:29:33.227+0000 7efc01906700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-01-26T23:29:33.378 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-01-26T23:29:33.376+0000 7f767f0a0540 -1 osd.1 0 log_to_monitors true 2024-01-26T23:29:33.471 INFO:tasks.ceph.osd.4.smithi106.stderr:2024-01-26T23:29:33.470+0000 7f984fe19540 -1 osd.4 0 log_to_monitors true 2024-01-26T23:29:33.477 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-01-26T23:29:33.476+0000 7eff807a7540 -1 osd.3 0 log_to_monitors true 2024-01-26T23:29:33.482 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-01-26T23:29:33.481+0000 7f6bff23b540 -1 osd.2 0 log_to_monitors true 2024-01-26T23:29:33.495 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-01-26T23:29:33.494+0000 7f9ab52cf540 -1 osd.0 0 log_to_monitors true 2024-01-26T23:29:33.501 INFO:tasks.ceph.osd.5.smithi106.stderr:2024-01-26T23:29:33.500+0000 7f8ef2382540 -1 osd.5 0 log_to_monitors true 2024-01-26T23:29:33.514 INFO:tasks.ceph.osd.6.smithi106.stderr:2024-01-26T23:29:33.513+0000 7f53465f1540 -1 osd.6 0 log_to_monitors true 2024-01-26T23:29:33.540 INFO:tasks.ceph.osd.7.smithi106.stderr:2024-01-26T23:29:33.539+0000 7efca0f40540 -1 osd.7 0 log_to_monitors true 2024-01-26T23:29:36.581 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-01-26T23:29:36.575+0000 7f9aabee8700 -1 osd.0 0 waiting for initial osdmap 2024-01-26T23:29:36.584 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-01-26T23:29:36.582+0000 7f9aa4cdf700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-26T23:29:36.595 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-01-26T23:29:36.593+0000 7eff773c0700 -1 osd.3 0 waiting for initial osdmap 2024-01-26T23:29:36.604 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-01-26T23:29:36.602+0000 7eff701b7700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-26T23:29:36.627 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-01-26T23:29:36.625+0000 7f6bf7657700 -1 osd.2 0 waiting for initial osdmap 2024-01-26T23:29:36.631 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-01-26T23:29:36.630+0000 7f6beec4b700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-26T23:29:36.646 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-01-26T23:29:36.644+0000 7f7675cb9700 -1 osd.1 0 waiting for initial osdmap 2024-01-26T23:29:36.649 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-01-26T23:29:36.648+0000 7f766eab0700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-26T23:29:37.232 INFO:tasks.ceph.osd.7.smithi106.stderr:2024-01-26T23:29:37.229+0000 7efc9935c700 -1 osd.7 0 waiting for initial osdmap 2024-01-26T23:29:37.248 INFO:tasks.ceph.osd.7.smithi106.stderr:2024-01-26T23:29:37.247+0000 7efc90950700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-26T23:29:37.274 INFO:tasks.ceph.osd.6.smithi106.stderr:2024-01-26T23:29:37.270+0000 7f533d20a700 -1 osd.6 0 waiting for initial osdmap 2024-01-26T23:29:37.279 INFO:tasks.ceph.osd.6.smithi106.stderr:2024-01-26T23:29:37.278+0000 7f5336001700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-26T23:29:37.282 INFO:tasks.ceph.osd.5.smithi106.stderr:2024-01-26T23:29:37.281+0000 7f8ee8f9b700 -1 osd.5 0 waiting for initial osdmap 2024-01-26T23:29:37.286 INFO:tasks.ceph.osd.5.smithi106.stderr:2024-01-26T23:29:37.285+0000 7f8ee1d92700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-26T23:29:37.303 INFO:tasks.ceph.osd.4.smithi106.stderr:2024-01-26T23:29:37.302+0000 7f9846a32700 -1 osd.4 0 waiting for initial osdmap 2024-01-26T23:29:37.308 INFO:tasks.ceph.osd.4.smithi106.stderr:2024-01-26T23:29:37.307+0000 7f983f829700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-26T23:29:37.455 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-26T23:29:37.591 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.589+0000 7f0f7de14700 1 Processor -- start 2024-01-26T23:29:37.591 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.589+0000 7f0f7de14700 1 -- start start 2024-01-26T23:29:37.593 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.590+0000 7f0f7de14700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0f78151da0 0x7f0f7806bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:37.593 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.590+0000 7f0f7de14700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f7806c1f0 0x7f0f780c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:37.594 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.590+0000 7f0f7de14700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f780cc300 con 0x7f0f781519d0 2024-01-26T23:29:37.594 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.590+0000 7f0f7de14700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f780ccba0 con 0x7f0f78151da0 2024-01-26T23:29:37.594 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.590+0000 7f0f7de14700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f78158a00 con 0x7f0f7806c1f0 2024-01-26T23:29:37.594 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.590+0000 7f0f77fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f7806c1f0 0x7f0f780c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:37.594 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.590+0000 7f0f76ffd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0f78151da0 0x7f0f7806bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:37.594 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.590+0000 7f0f777fe700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f0f781519d0 0x7f0f780cd030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:49352/0 (socket says 172.21.15.46:49352) 2024-01-26T23:29:37.594 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.590+0000 7f0f777fe700 1 -- 172.21.15.46:0/4198067284 learned_addr learned my addr 172.21.15.46:0/4198067284 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:37.594 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.590+0000 7f0f77fff700 1 -- 172.21.15.46:0/4198067284 >> v2:172.21.15.106:3300/0 conn(0x7f0f78151da0 msgr2=0x7f0f7806bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:37.594 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.590+0000 7f0f77fff700 1 --2- 172.21.15.46:0/4198067284 >> v2:172.21.15.106:3300/0 conn(0x7f0f78151da0 0x7f0f7806bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:37.594 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.590+0000 7f0f767fc700 1 -- 172.21.15.46:0/4198067284 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192681211 0 0) 0x7f0f780cc300 con 0x7f0f781519d0 2024-01-26T23:29:37.594 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.590+0000 7f0f77fff700 1 -- 172.21.15.46:0/4198067284 >> v1:172.21.15.106:6789/0 conn(0x7f0f781519d0 legacy=0x7f0f780cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:37.594 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.590+0000 7f0f77fff700 1 -- 172.21.15.46:0/4198067284 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f7806c820 con 0x7f0f7806c1f0 2024-01-26T23:29:37.594 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.590+0000 7f0f76ffd700 1 --2- 172.21.15.46:0/4198067284 >> v2:172.21.15.106:3300/0 conn(0x7f0f78151da0 0x7f0f7806bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:29:37.594 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.591+0000 7f0f77fff700 1 --2- 172.21.15.46:0/4198067284 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f7806c1f0 0x7f0f780c8a90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009e60 tx=0x7f0f70009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cefa1dfdd1a90a38 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:37.595 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.591+0000 7f0f767fc700 1 -- 172.21.15.46:0/4198067284 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7000e070 con 0x7f0f7806c1f0 2024-01-26T23:29:37.595 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.591+0000 7f0f767fc700 1 -- 172.21.15.46:0/4198067284 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7000bb60 con 0x7f0f7806c1f0 2024-01-26T23:29:37.595 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.591+0000 7f0f767fc700 1 -- 172.21.15.46:0/4198067284 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70012430 con 0x7f0f7806c1f0 2024-01-26T23:29:37.595 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.591+0000 7f0f7de14700 1 -- 172.21.15.46:0/4198067284 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f7806c1f0 msgr2=0x7f0f780c8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:37.595 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.591+0000 7f0f7de14700 1 --2- 172.21.15.46:0/4198067284 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f7806c1f0 0x7f0f780c8a90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009e60 tx=0x7f0f70009780 comp rx=0 tx=0).stop 2024-01-26T23:29:37.595 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.591+0000 7f0f7de14700 1 -- 172.21.15.46:0/4198067284 shutdown_connections 2024-01-26T23:29:37.595 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.591+0000 7f0f7de14700 1 --2- 172.21.15.46:0/4198067284 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f7806c1f0 0x7f0f780c8a90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:37.595 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.591+0000 7f0f7de14700 1 --2- 172.21.15.46:0/4198067284 >> v2:172.21.15.106:3300/0 conn(0x7f0f78151da0 0x7f0f7806bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:37.595 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.591+0000 7f0f7de14700 1 -- 172.21.15.46:0/4198067284 >> 172.21.15.46:0/4198067284 conn(0x7f0f780ce450 msgr2=0x7f0f780ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:37.595 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.591+0000 7f0f7de14700 1 -- 172.21.15.46:0/4198067284 shutdown_connections 2024-01-26T23:29:37.595 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.591+0000 7f0f7de14700 1 -- 172.21.15.46:0/4198067284 wait complete. 2024-01-26T23:29:37.598 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.593+0000 7f0f7de14700 1 Processor -- start 2024-01-26T23:29:37.598 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.593+0000 7f0f7de14700 1 -- start start 2024-01-26T23:29:37.598 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f7de14700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f7806c1f0 0x7f0f780b1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:37.598 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f7de14700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f781519d0 0x7f0f78156810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:37.598 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f7de14700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f7814b000 0x7f0f78156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:37.598 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f7de14700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7814b490 con 0x7f0f7814b000 2024-01-26T23:29:37.599 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f7de14700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7814b770 con 0x7f0f7806c1f0 2024-01-26T23:29:37.599 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f7de14700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7814ba50 con 0x7f0f781519d0 2024-01-26T23:29:37.599 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f77fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f7814b000 0x7f0f78156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:37.599 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f77fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f7814b000 0x7f0f78156d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37260/0 (socket says 172.21.15.46:37260) 2024-01-26T23:29:37.599 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f77fff700 1 -- 172.21.15.46:0/980180463 learned_addr learned my addr 172.21.15.46:0/980180463 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:37.599 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f76ffd700 1 --2- 172.21.15.46:0/980180463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f781519d0 0x7f0f78156810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:37.599 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f777fe700 1 --2- 172.21.15.46:0/980180463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f7806c1f0 0x7f0f780b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:37.599 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f76ffd700 1 -- 172.21.15.46:0/980180463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f7806c1f0 msgr2=0x7f0f780b1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:37.599 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f76ffd700 1 --2- 172.21.15.46:0/980180463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f7806c1f0 0x7f0f780b1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:37.599 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f76ffd700 1 -- 172.21.15.46:0/980180463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f7814b000 msgr2=0x7f0f78156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:37.599 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f76ffd700 1 --2- 172.21.15.46:0/980180463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f7814b000 0x7f0f78156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:37.599 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f76ffd700 1 -- 172.21.15.46:0/980180463 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f68009420 con 0x7f0f781519d0 2024-01-26T23:29:37.599 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f77fff700 1 --2- 172.21.15.46:0/980180463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f7814b000 0x7f0f78156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:29:37.600 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.594+0000 7f0f76ffd700 1 --2- 172.21.15.46:0/980180463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f781519d0 0x7f0f78156810 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0f6800ec50 tx=0x7f0f680091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:37.600 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.595+0000 7f0f6f7fe700 1 -- 172.21.15.46:0/980180463 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6800c850 con 0x7f0f781519d0 2024-01-26T23:29:37.600 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.595+0000 7f0f6f7fe700 1 -- 172.21.15.46:0/980180463 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6800c9b0 con 0x7f0f781519d0 2024-01-26T23:29:37.600 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.595+0000 7f0f6f7fe700 1 -- 172.21.15.46:0/980180463 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6800cd50 con 0x7f0f781519d0 2024-01-26T23:29:37.600 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.595+0000 7f0f7de14700 1 -- 172.21.15.46:0/980180463 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f70009a20 con 0x7f0f781519d0 2024-01-26T23:29:37.600 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.595+0000 7f0f7de14700 1 -- 172.21.15.46:0/980180463 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f7814c010 con 0x7f0f781519d0 2024-01-26T23:29:37.600 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.596+0000 7f0f7de14700 1 -- 172.21.15.46:0/980180463 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f780713b0 con 0x7f0f781519d0 2024-01-26T23:29:37.600 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.596+0000 7f0f6f7fe700 1 -- 172.21.15.46:0/980180463 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 4) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f0f68013680 con 0x7f0f781519d0 2024-01-26T23:29:37.600 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.596+0000 7f0f6f7fe700 1 --2- 172.21.15.46:0/980180463 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0f58066ea0 0x7f0f58069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:37.600 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.597+0000 7f0f6f7fe700 1 -- 172.21.15.46:0/980180463 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4467+0+0 (secure 0 0 0) 0x7f0f68084150 con 0x7f0f781519d0 2024-01-26T23:29:37.600 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.597+0000 7f0f777fe700 1 --2- 172.21.15.46:0/980180463 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0f58066ea0 0x7f0f58069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:37.601 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.597+0000 7f0f777fe700 1 --2- 172.21.15.46:0/980180463 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0f58066ea0 0x7f0f58069350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0f60001230 tx=0x7f0f60007d20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:37.602 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.601+0000 7f0f6f7fe700 1 -- 172.21.15.46:0/980180463 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0f68051970 con 0x7f0f781519d0 2024-01-26T23:29:37.810 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.808+0000 7f0f7de14700 1 -- 172.21.15.46:0/980180463 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f780c3280 con 0x7f0f781519d0 2024-01-26T23:29:37.811 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.810+0000 7f0f6f7fe700 1 -- 172.21.15.46:0/980180463 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10396 (secure 0 0 0) 0x7f0f68054f90 con 0x7f0f781519d0 2024-01-26T23:29:37.811 INFO:teuthology.misc.health.smithi046.stdout: 2024-01-26T23:29:37.811 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":13,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:37.506695+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:29:37.813 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.811+0000 7f0f7de14700 1 -- 172.21.15.46:0/980180463 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0f58066ea0 msgr2=0x7f0f58069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:37.813 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.811+0000 7f0f7de14700 1 --2- 172.21.15.46:0/980180463 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0f58066ea0 0x7f0f58069350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0f60001230 tx=0x7f0f60007d20 comp rx=0 tx=0).stop 2024-01-26T23:29:37.814 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.811+0000 7f0f7de14700 1 -- 172.21.15.46:0/980180463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f781519d0 msgr2=0x7f0f78156810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:37.814 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.811+0000 7f0f7de14700 1 --2- 172.21.15.46:0/980180463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f781519d0 0x7f0f78156810 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0f6800ec50 tx=0x7f0f680091a0 comp rx=0 tx=0).stop 2024-01-26T23:29:37.814 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.811+0000 7f0f7de14700 1 -- 172.21.15.46:0/980180463 shutdown_connections 2024-01-26T23:29:37.814 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.811+0000 7f0f7de14700 1 --2- 172.21.15.46:0/980180463 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0f58066ea0 0x7f0f58069350 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:37.814 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.811+0000 7f0f7de14700 1 --2- 172.21.15.46:0/980180463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f7814b000 0x7f0f78156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:37.814 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.811+0000 7f0f7de14700 1 --2- 172.21.15.46:0/980180463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f7806c1f0 0x7f0f780b1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:37.814 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.811+0000 7f0f7de14700 1 --2- 172.21.15.46:0/980180463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f781519d0 0x7f0f78156810 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:37.814 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.811+0000 7f0f7de14700 1 -- 172.21.15.46:0/980180463 >> 172.21.15.46:0/980180463 conn(0x7f0f780ce450 msgr2=0x7f0f7814d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:37.814 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.811+0000 7f0f7de14700 1 -- 172.21.15.46:0/980180463 shutdown_connections 2024-01-26T23:29:37.814 INFO:teuthology.misc.health.smithi046.stderr:2024-01-26T23:29:37.811+0000 7f0f7de14700 1 -- 172.21.15.46:0/980180463 wait complete. 2024-01-26T23:29:37.826 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-01-26T23:29:37.826 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-26T23:29:37.826 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-26T23:29:37.826 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-26T23:29:37.829 INFO:tasks.ceph.mds.a:Started 2024-01-26T23:29:37.829 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-26T23:29:37.829 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-26T23:29:37.831 INFO:tasks.ceph.mds.c:Started 2024-01-26T23:29:37.831 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-26T23:29:37.831 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-26T23:29:37.833 INFO:tasks.ceph.mds.b:Started 2024-01-26T23:29:37.833 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-26T23:29:37.833 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-26T23:29:37.835 INFO:tasks.ceph.mds.d:Started 2024-01-26T23:29:37.835 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-01-26T23:29:37.836 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:29:37.909 INFO:tasks.ceph.mds.b.smithi106.stdout:starting mds.b at 2024-01-26T23:29:37.909 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-01-26T23:29:37.914 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-26T23:29:37.918 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-26T23:29:37.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.974+0000 7f6de3089700 1 Processor -- start 2024-01-26T23:29:37.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.974+0000 7f6de3089700 1 -- start start 2024-01-26T23:29:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.975+0000 7f6de3089700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6ddc159b30 0x7f6ddc067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.975+0000 7f6de3089700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ddc15abc0 con 0x7f6ddc15afc0 2024-01-26T23:29:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.975+0000 7f6de3089700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ddc0b2e90 con 0x7f6ddc159760 2024-01-26T23:29:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.975+0000 7f6de3089700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ddc159fb0 con 0x7f6ddc159b30 2024-01-26T23:29:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.975+0000 7f6ddbfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6ddc159b30 0x7f6ddc067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.975+0000 7f6de1626700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6ddc15afc0 0x7f6ddc06c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34604/0 (socket says 172.21.15.46:34604) 2024-01-26T23:29:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.975+0000 7f6de1626700 1 -- 172.21.15.46:0/780459051 learned_addr learned my addr 172.21.15.46:0/780459051 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.975+0000 7f6ddbfff700 1 -- 172.21.15.46:0/780459051 >> v1:172.21.15.106:6789/0 conn(0x7f6ddc159760 legacy=0x7f6ddc0b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.975+0000 7f6ddbfff700 1 -- 172.21.15.46:0/780459051 >> v1:172.21.15.46:6789/0 conn(0x7f6ddc15afc0 legacy=0x7f6ddc06c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.975+0000 7f6ddb7fe700 1 -- 172.21.15.46:0/780459051 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075537847 0 0) 0x7f6ddc15abc0 con 0x7f6ddc15afc0 2024-01-26T23:29:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.975+0000 7f6ddbfff700 1 -- 172.21.15.46:0/780459051 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc15a290 con 0x7f6ddc159b30 2024-01-26T23:29:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.975+0000 7f6ddbfff700 1 --2- 172.21.15.46:0/780459051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc159b30 0x7f6ddc067930 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80098e0 tx=0x7f6dc80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f16e951b6286e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.976+0000 7f6ddb7fe700 1 -- 172.21.15.46:0/780459051 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc8009e80 con 0x7f6ddc159b30 2024-01-26T23:29:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.976+0000 7f6ddb7fe700 1 -- 172.21.15.46:0/780459051 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc800b860 con 0x7f6ddc159b30 2024-01-26T23:29:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.976+0000 7f6ddb7fe700 1 -- 172.21.15.46:0/780459051 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc800bbe0 con 0x7f6ddc159b30 2024-01-26T23:29:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.976+0000 7f6de3089700 1 -- 172.21.15.46:0/780459051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc159b30 msgr2=0x7f6ddc067930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.976+0000 7f6de3089700 1 --2- 172.21.15.46:0/780459051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc159b30 0x7f6ddc067930 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80098e0 tx=0x7f6dc80091a0 comp rx=0 tx=0).stop 2024-01-26T23:29:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.976+0000 7f6de3089700 1 -- 172.21.15.46:0/780459051 shutdown_connections 2024-01-26T23:29:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.976+0000 7f6de3089700 1 --2- 172.21.15.46:0/780459051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc159b30 0x7f6ddc067930 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.976+0000 7f6de3089700 1 -- 172.21.15.46:0/780459051 >> 172.21.15.46:0/780459051 conn(0x7f6ddc06b630 msgr2=0x7f6ddc0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.976+0000 7f6de3089700 1 -- 172.21.15.46:0/780459051 shutdown_connections 2024-01-26T23:29:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.976+0000 7f6de3089700 1 -- 172.21.15.46:0/780459051 wait complete. 2024-01-26T23:29:37.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.977+0000 7f6de3089700 1 Processor -- start 2024-01-26T23:29:37.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.978+0000 7f6de3089700 1 -- start start 2024-01-26T23:29:37.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.978+0000 7f6de3089700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ddc159760 0x7f6ddc15db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.978+0000 7f6de3089700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc15afc0 0x7f6ddc15e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.978+0000 7f6de3089700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ddc144090 0x7f6ddc144c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.978+0000 7f6de3089700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc15e6a0 con 0x7f6ddc15afc0 2024-01-26T23:29:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.978+0000 7f6de3089700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc15e890 con 0x7f6ddc159760 2024-01-26T23:29:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.978+0000 7f6de3089700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc15eb70 con 0x7f6ddc144090 2024-01-26T23:29:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.979+0000 7f6ddbfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc15afc0 0x7f6ddc15e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.979+0000 7f6ddbfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc15afc0 0x7f6ddc15e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37304/0 (socket says 172.21.15.46:37304) 2024-01-26T23:29:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.979+0000 7f6ddbfff700 1 -- 172.21.15.46:0/2485181759 learned_addr learned my addr 172.21.15.46:0/2485181759 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.979+0000 7f6de1626700 1 --2- 172.21.15.46:0/2485181759 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ddc144090 0x7f6ddc144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.979+0000 7f6de0e25700 1 --2- 172.21.15.46:0/2485181759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ddc159760 0x7f6ddc15db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.979+0000 7f6ddbfff700 1 -- 172.21.15.46:0/2485181759 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ddc144090 msgr2=0x7f6ddc144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.979+0000 7f6ddbfff700 1 --2- 172.21.15.46:0/2485181759 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ddc144090 0x7f6ddc144c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.979+0000 7f6ddbfff700 1 -- 172.21.15.46:0/2485181759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ddc159760 msgr2=0x7f6ddc15db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.979+0000 7f6ddbfff700 1 --2- 172.21.15.46:0/2485181759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ddc159760 0x7f6ddc15db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.979+0000 7f6ddbfff700 1 -- 172.21.15.46:0/2485181759 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc8009420 con 0x7f6ddc15afc0 2024-01-26T23:29:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.979+0000 7f6ddbfff700 1 --2- 172.21.15.46:0/2485181759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc15afc0 0x7f6ddc15e160 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc0ce590 tx=0x7f6dc8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.979+0000 7f6dd8ff9700 1 -- 172.21.15.46:0/2485181759 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc800bbe0 con 0x7f6ddc15afc0 2024-01-26T23:29:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.979+0000 7f6dd8ff9700 1 -- 172.21.15.46:0/2485181759 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc8000d10 con 0x7f6ddc15afc0 2024-01-26T23:29:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.979+0000 7f6de3089700 1 -- 172.21.15.46:0/2485181759 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ddc15ee50 con 0x7f6ddc15afc0 2024-01-26T23:29:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.979+0000 7f6dd8ff9700 1 -- 172.21.15.46:0/2485181759 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc8011650 con 0x7f6ddc15afc0 2024-01-26T23:29:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.979+0000 7f6de3089700 1 -- 172.21.15.46:0/2485181759 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ddc15f410 con 0x7f6ddc15afc0 2024-01-26T23:29:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.981+0000 7f6de3089700 1 -- 172.21.15.46:0/2485181759 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ddc15f600 con 0x7f6ddc15afc0 2024-01-26T23:29:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.981+0000 7f6dd8ff9700 1 -- 172.21.15.46:0/2485181759 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f6dc8011a40 con 0x7f6ddc15afc0 2024-01-26T23:29:37.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.981+0000 7f6dd8ff9700 1 --2- 172.21.15.46:0/2485181759 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6dc0066ea0 0x7f6dc0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:37.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.981+0000 7f6dd8ff9700 1 -- 172.21.15.46:0/2485181759 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4467+0+0 (secure 0 0 0) 0x7f6dc8085880 con 0x7f6ddc15afc0 2024-01-26T23:29:37.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.982+0000 7f6de0e25700 1 --2- 172.21.15.46:0/2485181759 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6dc0066ea0 0x7f6dc0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:37.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.982+0000 7f6de0e25700 1 --2- 172.21.15.46:0/2485181759 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6dc0066ea0 0x7f6dc0069350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0003a80 tx=0x7f6dd0007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:37.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:37.986+0000 7f6dd8ff9700 1 -- 172.21.15.46:0/2485181759 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6dc80530a0 con 0x7f6ddc15afc0 2024-01-26T23:29:38.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.191+0000 7f6de3089700 1 -- 172.21.15.46:0/2485181759 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ddc04c710 con 0x7f6ddc15afc0 2024-01-26T23:29:38.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.192+0000 7f6dd8ff9700 1 -- 172.21.15.46:0/2485181759 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10396 (secure 0 0 0) 0x7f6dc80566c0 con 0x7f6ddc15afc0 2024-01-26T23:29:38.194 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:29:38.194 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":13,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:37.506695+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:29:38.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.194+0000 7f6de3089700 1 -- 172.21.15.46:0/2485181759 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6dc0066ea0 msgr2=0x7f6dc0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:38.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.194+0000 7f6de3089700 1 --2- 172.21.15.46:0/2485181759 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6dc0066ea0 0x7f6dc0069350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0003a80 tx=0x7f6dd0007ec0 comp rx=0 tx=0).stop 2024-01-26T23:29:38.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.194+0000 7f6de3089700 1 -- 172.21.15.46:0/2485181759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc15afc0 msgr2=0x7f6ddc15e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:38.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.194+0000 7f6de3089700 1 --2- 172.21.15.46:0/2485181759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc15afc0 0x7f6ddc15e160 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc0ce590 tx=0x7f6dc8017000 comp rx=0 tx=0).stop 2024-01-26T23:29:38.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.194+0000 7f6de3089700 1 -- 172.21.15.46:0/2485181759 shutdown_connections 2024-01-26T23:29:38.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.194+0000 7f6de3089700 1 --2- 172.21.15.46:0/2485181759 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6dc0066ea0 0x7f6dc0069350 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:38.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.194+0000 7f6de3089700 1 --2- 172.21.15.46:0/2485181759 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ddc144090 0x7f6ddc144c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:38.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.194+0000 7f6de3089700 1 --2- 172.21.15.46:0/2485181759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ddc159760 0x7f6ddc15db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:38.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.194+0000 7f6de3089700 1 --2- 172.21.15.46:0/2485181759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc15afc0 0x7f6ddc15e160 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:38.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.195+0000 7f6de3089700 1 -- 172.21.15.46:0/2485181759 >> 172.21.15.46:0/2485181759 conn(0x7f6ddc06b630 msgr2=0x7f6ddc06eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:38.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.195+0000 7f6de3089700 1 -- 172.21.15.46:0/2485181759 shutdown_connections 2024-01-26T23:29:38.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.195+0000 7f6de3089700 1 -- 172.21.15.46:0/2485181759 wait complete. 2024-01-26T23:29:38.210 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-01-26T23:29:38.210 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:29:38.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.343+0000 7f662a73d700 1 Processor -- start 2024-01-26T23:29:38.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.343+0000 7f662a73d700 1 -- start start 2024-01-26T23:29:38.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.343+0000 7f662a73d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66240b2160 0x7f662415a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:38.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.344+0000 7f662a73d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f662415da10 0x7f662415de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:38.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.344+0000 7f662a73d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66240a9b90 con 0x7f66240b2530 2024-01-26T23:29:38.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.344+0000 7f662a73d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66240aa210 con 0x7f662415da10 2024-01-26T23:29:38.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.344+0000 7f662a73d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662415d580 con 0x7f66240b2160 2024-01-26T23:29:38.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.344+0000 7f6628cda700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f662415da10 0x7f662415de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:38.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.344+0000 7f66237fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f66240b2530 0x7f66240abe50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34616/0 (socket says 172.21.15.46:34616) 2024-01-26T23:29:38.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.344+0000 7f66237fe700 1 -- 172.21.15.46:0/2675754655 learned_addr learned my addr 172.21.15.46:0/2675754655 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:38.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.344+0000 7f6623fff700 1 --2- 172.21.15.46:0/2675754655 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66240b2160 0x7f662415a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:38.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.344+0000 7f6628cda700 1 -- 172.21.15.46:0/2675754655 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66240b2160 msgr2=0x7f662415a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:38.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.344+0000 7f6628cda700 1 --2- 172.21.15.46:0/2675754655 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66240b2160 0x7f662415a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:38.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.344+0000 7f6628cda700 1 -- 172.21.15.46:0/2675754655 >> v1:172.21.15.46:6789/0 conn(0x7f66240b2530 legacy=0x7f66240abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:38.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.344+0000 7f6628cda700 1 -- 172.21.15.46:0/2675754655 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66240aa460 con 0x7f662415da10 2024-01-26T23:29:38.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.344+0000 7f6628cda700 1 --2- 172.21.15.46:0/2675754655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f662415da10 0x7f662415de90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f661c009e60 tx=0x7f661c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f157fab048ba9237 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:38.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.344+0000 7f6622ffd700 1 -- 172.21.15.46:0/2675754655 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c00e070 con 0x7f662415da10 2024-01-26T23:29:38.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.344+0000 7f6622ffd700 1 -- 172.21.15.46:0/2675754655 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661c00bb60 con 0x7f662415da10 2024-01-26T23:29:38.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.344+0000 7f6622ffd700 1 -- 172.21.15.46:0/2675754655 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c012430 con 0x7f662415da10 2024-01-26T23:29:38.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.345+0000 7f662a73d700 1 -- 172.21.15.46:0/2675754655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f662415da10 msgr2=0x7f662415de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:38.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.345+0000 7f662a73d700 1 --2- 172.21.15.46:0/2675754655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f662415da10 0x7f662415de90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f661c009e60 tx=0x7f661c009780 comp rx=0 tx=0).stop 2024-01-26T23:29:38.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.345+0000 7f662a73d700 1 -- 172.21.15.46:0/2675754655 shutdown_connections 2024-01-26T23:29:38.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.345+0000 7f662a73d700 1 --2- 172.21.15.46:0/2675754655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f662415da10 0x7f662415de90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:38.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.345+0000 7f662a73d700 1 --2- 172.21.15.46:0/2675754655 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66240b2160 0x7f662415a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:38.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.345+0000 7f662a73d700 1 -- 172.21.15.46:0/2675754655 >> 172.21.15.46:0/2675754655 conn(0x7f66240ccbc0 msgr2=0x7f66240ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:38.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.345+0000 7f662a73d700 1 -- 172.21.15.46:0/2675754655 shutdown_connections 2024-01-26T23:29:38.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.345+0000 7f662a73d700 1 -- 172.21.15.46:0/2675754655 wait complete. 2024-01-26T23:29:38.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.346+0000 7f662a73d700 1 Processor -- start 2024-01-26T23:29:38.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.347+0000 7f662a73d700 1 -- start start 2024-01-26T23:29:38.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.347+0000 7f662a73d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66240b2160 0x7f66240af010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:38.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.347+0000 7f662a73d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66240b2530 0x7f6624159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:38.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.347+0000 7f662a73d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f662415da10 0x7f662414fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:38.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.347+0000 7f662a73d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66240aeb80 con 0x7f662415da10 2024-01-26T23:29:38.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.347+0000 7f662a73d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66240af550 con 0x7f66240b2530 2024-01-26T23:29:38.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.347+0000 7f662a73d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66240adf70 con 0x7f66240b2160 2024-01-26T23:29:38.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.347+0000 7f6628cda700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f662415da10 0x7f662414fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:38.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.347+0000 7f6623fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66240b2160 0x7f66240af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:38.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.347+0000 7f6628cda700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f662415da10 0x7f662414fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37326/0 (socket says 172.21.15.46:37326) 2024-01-26T23:29:38.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.347+0000 7f6628cda700 1 -- 172.21.15.46:0/2017172226 learned_addr learned my addr 172.21.15.46:0/2017172226 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:38.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.347+0000 7f66237fe700 1 --2- 172.21.15.46:0/2017172226 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66240b2530 0x7f6624159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:38.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.348+0000 7f6628cda700 1 -- 172.21.15.46:0/2017172226 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66240b2160 msgr2=0x7f66240af010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:38.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.348+0000 7f6628cda700 1 --2- 172.21.15.46:0/2017172226 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66240b2160 0x7f66240af010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:38.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.348+0000 7f6628cda700 1 -- 172.21.15.46:0/2017172226 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66240b2530 msgr2=0x7f6624159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:38.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.348+0000 7f6628cda700 1 --2- 172.21.15.46:0/2017172226 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66240b2530 0x7f6624159b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:38.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.348+0000 7f6628cda700 1 -- 172.21.15.46:0/2017172226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c009a20 con 0x7f662415da10 2024-01-26T23:29:38.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.348+0000 7f6628cda700 1 --2- 172.21.15.46:0/2017172226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f662415da10 0x7f662414fd70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f661c00b6c0 tx=0x7f661c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:38.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.348+0000 7f660bfff700 1 -- 172.21.15.46:0/2017172226 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c025440 con 0x7f662415da10 2024-01-26T23:29:38.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.348+0000 7f660bfff700 1 -- 172.21.15.46:0/2017172226 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661c0255c0 con 0x7f662415da10 2024-01-26T23:29:38.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.348+0000 7f662a73d700 1 -- 172.21.15.46:0/2017172226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66240ae250 con 0x7f662415da10 2024-01-26T23:29:38.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.348+0000 7f660bfff700 1 -- 172.21.15.46:0/2017172226 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c025980 con 0x7f662415da10 2024-01-26T23:29:38.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.348+0000 7f662a73d700 1 -- 172.21.15.46:0/2017172226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66240ae810 con 0x7f662415da10 2024-01-26T23:29:38.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.349+0000 7f662a73d700 1 -- 172.21.15.46:0/2017172226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66240aecf0 con 0x7f662415da10 2024-01-26T23:29:38.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.350+0000 7f660bfff700 1 -- 172.21.15.46:0/2017172226 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f661c003050 con 0x7f662415da10 2024-01-26T23:29:38.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.350+0000 7f660bfff700 1 --2- 172.21.15.46:0/2017172226 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6610066ea0 0x7f6610069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:38.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.350+0000 7f660bfff700 1 -- 172.21.15.46:0/2017172226 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4467+0+0 (secure 0 0 0) 0x7f661c085f70 con 0x7f662415da10 2024-01-26T23:29:38.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.350+0000 7f6623fff700 1 --2- 172.21.15.46:0/2017172226 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6610066ea0 0x7f6610069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:38.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.351+0000 7f6623fff700 1 --2- 172.21.15.46:0/2017172226 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6610066ea0 0x7f6610069350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6614009db0 tx=0x7f66140091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:38.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.355+0000 7f660bfff700 1 -- 172.21.15.46:0/2017172226 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f661c053790 con 0x7f662415da10 2024-01-26T23:29:38.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.608+0000 7f662a73d700 1 -- 172.21.15.46:0/2017172226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f662404c710 con 0x7f662415da10 2024-01-26T23:29:38.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.608+0000 7f660bfff700 1 -- 172.21.15.46:0/2017172226 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1259 (secure 0 0 0) 0x7f661c056db0 con 0x7f662415da10 2024-01-26T23:29:38.611 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:29:38.611 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3319220815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3319220815},{"type":"v1","addr":"172.21.15.106:6836","nonce":3319220815}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2024-01-26T23:29:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.610+0000 7f662a73d700 1 -- 172.21.15.46:0/2017172226 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6610066ea0 msgr2=0x7f6610069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.610+0000 7f662a73d700 1 --2- 172.21.15.46:0/2017172226 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6610066ea0 0x7f6610069350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6614009db0 tx=0x7f66140091a0 comp rx=0 tx=0).stop 2024-01-26T23:29:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.610+0000 7f662a73d700 1 -- 172.21.15.46:0/2017172226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f662415da10 msgr2=0x7f662414fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.610+0000 7f662a73d700 1 --2- 172.21.15.46:0/2017172226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f662415da10 0x7f662414fd70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f661c00b6c0 tx=0x7f661c019000 comp rx=0 tx=0).stop 2024-01-26T23:29:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.611+0000 7f662a73d700 1 -- 172.21.15.46:0/2017172226 shutdown_connections 2024-01-26T23:29:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.611+0000 7f662a73d700 1 --2- 172.21.15.46:0/2017172226 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6610066ea0 0x7f6610069350 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.611+0000 7f662a73d700 1 --2- 172.21.15.46:0/2017172226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f662415da10 0x7f662414fd70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.611+0000 7f662a73d700 1 --2- 172.21.15.46:0/2017172226 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66240b2160 0x7f66240af010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.611+0000 7f662a73d700 1 --2- 172.21.15.46:0/2017172226 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66240b2530 0x7f6624159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.611+0000 7f662a73d700 1 -- 172.21.15.46:0/2017172226 >> 172.21.15.46:0/2017172226 conn(0x7f66240ccbc0 msgr2=0x7f66240affa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.611+0000 7f662a73d700 1 -- 172.21.15.46:0/2017172226 shutdown_connections 2024-01-26T23:29:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:38.611+0000 7f662a73d700 1 -- 172.21.15.46:0/2017172226 wait complete. 2024-01-26T23:29:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 2 2024-01-26T23:29:40.626 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:29:40.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.761+0000 7ff61fab3700 1 Processor -- start 2024-01-26T23:29:40.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.761+0000 7ff61fab3700 1 -- start start 2024-01-26T23:29:40.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.762+0000 7ff61fab3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff618157340 0x7ff61815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.762+0000 7ff61fab3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6180ae520 0x7ff6180b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.762+0000 7ff61fab3700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff618158ae0 con 0x7ff6180b3fb0 2024-01-26T23:29:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.762+0000 7ff61fab3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6180b4430 con 0x7ff618157340 2024-01-26T23:29:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.762+0000 7ff61fab3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61815b650 con 0x7ff6180ae520 2024-01-26T23:29:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.762+0000 7ff61d84f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff618157340 0x7ff61815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.762+0000 7ff61e050700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff6180b3fb0 0x7ff618157710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49584/0 (socket says 172.21.15.46:49584) 2024-01-26T23:29:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.762+0000 7ff61d84f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff618157340 0x7ff61815bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37096/0 (socket says 172.21.15.46:37096) 2024-01-26T23:29:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.762+0000 7ff61e050700 1 -- 172.21.15.46:0/3668021007 learned_addr learned my addr 172.21.15.46:0/3668021007 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.762+0000 7ff61d04e700 1 --2- 172.21.15.46:0/3668021007 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6180ae520 0x7ff6180b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.763+0000 7ff61d84f700 1 -- 172.21.15.46:0/3668021007 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6180ae520 msgr2=0x7ff6180b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.763+0000 7ff61d84f700 1 --2- 172.21.15.46:0/3668021007 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6180ae520 0x7ff6180b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.763+0000 7ff61d84f700 1 -- 172.21.15.46:0/3668021007 >> v1:172.21.15.46:6789/0 conn(0x7ff6180b3fb0 legacy=0x7ff618157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.763+0000 7ff61d84f700 1 -- 172.21.15.46:0/3668021007 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61815c020 con 0x7ff618157340 2024-01-26T23:29:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.763+0000 7ff61d84f700 1 --2- 172.21.15.46:0/3668021007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff618157340 0x7ff61815bae0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff614009e60 tx=0x7ff614009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa97f4b5b229b1cb server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.763+0000 7ff61c84d700 1 -- 172.21.15.46:0/3668021007 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61400e070 con 0x7ff618157340 2024-01-26T23:29:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.763+0000 7ff61c84d700 1 -- 172.21.15.46:0/3668021007 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61400bb60 con 0x7ff618157340 2024-01-26T23:29:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.763+0000 7ff61c84d700 1 -- 172.21.15.46:0/3668021007 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614012430 con 0x7ff618157340 2024-01-26T23:29:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.764+0000 7ff61fab3700 1 -- 172.21.15.46:0/3668021007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff618157340 msgr2=0x7ff61815bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.764+0000 7ff61fab3700 1 --2- 172.21.15.46:0/3668021007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff618157340 0x7ff61815bae0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff614009e60 tx=0x7ff614009780 comp rx=0 tx=0).stop 2024-01-26T23:29:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.764+0000 7ff61fab3700 1 -- 172.21.15.46:0/3668021007 shutdown_connections 2024-01-26T23:29:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.764+0000 7ff61fab3700 1 --2- 172.21.15.46:0/3668021007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff618157340 0x7ff61815bae0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.764+0000 7ff61fab3700 1 --2- 172.21.15.46:0/3668021007 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6180ae520 0x7ff6180b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.764+0000 7ff61fab3700 1 -- 172.21.15.46:0/3668021007 >> 172.21.15.46:0/3668021007 conn(0x7ff6180ce530 msgr2=0x7ff6180ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.764+0000 7ff61fab3700 1 -- 172.21.15.46:0/3668021007 shutdown_connections 2024-01-26T23:29:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.764+0000 7ff61fab3700 1 -- 172.21.15.46:0/3668021007 wait complete. 2024-01-26T23:29:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.765+0000 7ff61fab3700 1 Processor -- start 2024-01-26T23:29:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.765+0000 7ff61fab3700 1 -- start start 2024-01-26T23:29:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.766+0000 7ff61fab3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6180ae520 0x7ff6180b3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.766+0000 7ff61fab3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6180b3fb0 0x7ff618156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.766+0000 7ff61fab3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff618157340 0x7ff61815d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.766+0000 7ff61fab3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6181676b0 con 0x7ff6180b3fb0 2024-01-26T23:29:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.766+0000 7ff61fab3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff618157010 con 0x7ff6180ae520 2024-01-26T23:29:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.766+0000 7ff61fab3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6180b1770 con 0x7ff618157340 2024-01-26T23:29:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.766+0000 7ff61d04e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6180b3fb0 0x7ff618156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.766+0000 7ff61d04e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6180b3fb0 0x7ff618156ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37112/0 (socket says 172.21.15.46:37112) 2024-01-26T23:29:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.766+0000 7ff61d04e700 1 -- 172.21.15.46:0/1027153021 learned_addr learned my addr 172.21.15.46:0/1027153021 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.766+0000 7ff61e050700 1 --2- 172.21.15.46:0/1027153021 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff618157340 0x7ff61815d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.766+0000 7ff61d84f700 1 --2- 172.21.15.46:0/1027153021 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6180ae520 0x7ff6180b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.766+0000 7ff61d04e700 1 -- 172.21.15.46:0/1027153021 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff618157340 msgr2=0x7ff61815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.766+0000 7ff61d04e700 1 --2- 172.21.15.46:0/1027153021 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff618157340 0x7ff61815d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.766+0000 7ff61d04e700 1 -- 172.21.15.46:0/1027153021 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6180ae520 msgr2=0x7ff6180b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.766+0000 7ff61d04e700 1 --2- 172.21.15.46:0/1027153021 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6180ae520 0x7ff6180b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.766+0000 7ff61d04e700 1 -- 172.21.15.46:0/1027153021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff614009a20 con 0x7ff6180b3fb0 2024-01-26T23:29:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.766+0000 7ff61d04e700 1 --2- 172.21.15.46:0/1027153021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6180b3fb0 0x7ff618156ad0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff60800ec80 tx=0x7ff6080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.767+0000 7ff60dffb700 1 -- 172.21.15.46:0/1027153021 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60800c850 con 0x7ff6180b3fb0 2024-01-26T23:29:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.767+0000 7ff60dffb700 1 -- 172.21.15.46:0/1027153021 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60800c9b0 con 0x7ff6180b3fb0 2024-01-26T23:29:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.767+0000 7ff61fab3700 1 -- 172.21.15.46:0/1027153021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6180b1a50 con 0x7ff6180b3fb0 2024-01-26T23:29:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.767+0000 7ff60dffb700 1 -- 172.21.15.46:0/1027153021 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60800cd50 con 0x7ff6180b3fb0 2024-01-26T23:29:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.767+0000 7ff61fab3700 1 -- 172.21.15.46:0/1027153021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6180b2010 con 0x7ff6180b3fb0 2024-01-26T23:29:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.768+0000 7ff60dffb700 1 -- 172.21.15.46:0/1027153021 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 84686+0+0 (secure 0 0 0) 0x7ff608013680 con 0x7ff6180b3fb0 2024-01-26T23:29:40.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.769+0000 7ff61fab3700 1 -- 172.21.15.46:0/1027153021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff61804c710 con 0x7ff6180b3fb0 2024-01-26T23:29:40.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.769+0000 7ff60dffb700 1 --2- 172.21.15.46:0/1027153021 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5fc066f20 0x7ff5fc0693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:40.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.769+0000 7ff60dffb700 1 -- 172.21.15.46:0/1027153021 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7ff608084380 con 0x7ff6180b3fb0 2024-01-26T23:29:40.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.769+0000 7ff61d84f700 1 --2- 172.21.15.46:0/1027153021 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5fc066f20 0x7ff5fc0693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:40.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.769+0000 7ff61d84f700 1 --2- 172.21.15.46:0/1027153021 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5fc066f20 0x7ff5fc0693d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff614009e60 tx=0x7ff614019000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:40.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:40.774+0000 7ff60dffb700 1 -- 172.21.15.46:0/1027153021 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff608051980 con 0x7ff6180b3fb0 2024-01-26T23:29:41.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.029+0000 7ff61fab3700 1 -- 172.21.15.46:0/1027153021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff618158520 con 0x7ff6180b3fb0 2024-01-26T23:29:41.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.030+0000 7ff60dffb700 1 -- 172.21.15.46:0/1027153021 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3524 (secure 0 0 0) 0x7ff608017de0 con 0x7ff6180b3fb0 2024-01-26T23:29:41.032 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:29:41.032 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3409861270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3409861270},{"type":"v1","addr":"172.21.15.46:6833","nonce":3409861270}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3319220815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3319220815},{"type":"v1","addr":"172.21.15.106:6836","nonce":3319220815}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4252,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/529198432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":529198432},{"type":"v1","addr":"172.21.15.106:6837","nonce":529198432}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/688086235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":688086235},{"type":"v1","addr":"172.21.15.46:6835","nonce":688086235}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-01-26T23:29:41.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.032+0000 7ff61fab3700 1 -- 172.21.15.46:0/1027153021 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5fc066f20 msgr2=0x7ff5fc0693d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.032+0000 7ff61fab3700 1 --2- 172.21.15.46:0/1027153021 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5fc066f20 0x7ff5fc0693d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff614009e60 tx=0x7ff614019000 comp rx=0 tx=0).stop 2024-01-26T23:29:41.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.032+0000 7ff61fab3700 1 -- 172.21.15.46:0/1027153021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6180b3fb0 msgr2=0x7ff618156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.032+0000 7ff61fab3700 1 --2- 172.21.15.46:0/1027153021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6180b3fb0 0x7ff618156ad0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff60800ec80 tx=0x7ff6080091a0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.032+0000 7ff61fab3700 1 -- 172.21.15.46:0/1027153021 shutdown_connections 2024-01-26T23:29:41.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.032+0000 7ff61fab3700 1 --2- 172.21.15.46:0/1027153021 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5fc066f20 0x7ff5fc0693d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.032+0000 7ff61fab3700 1 --2- 172.21.15.46:0/1027153021 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff618157340 0x7ff61815d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.032+0000 7ff61fab3700 1 --2- 172.21.15.46:0/1027153021 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6180ae520 0x7ff6180b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.032+0000 7ff61fab3700 1 --2- 172.21.15.46:0/1027153021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6180b3fb0 0x7ff618156ad0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.032+0000 7ff61fab3700 1 -- 172.21.15.46:0/1027153021 >> 172.21.15.46:0/1027153021 conn(0x7ff6180ce530 msgr2=0x7ff618157710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:41.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.032+0000 7ff61fab3700 1 -- 172.21.15.46:0/1027153021 shutdown_connections 2024-01-26T23:29:41.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.032+0000 7ff61fab3700 1 -- 172.21.15.46:0/1027153021 wait complete. 2024-01-26T23:29:41.036 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 3 2024-01-26T23:29:41.048 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:29:41.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.180+0000 7f976c4f5700 1 Processor -- start 2024-01-26T23:29:41.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.180+0000 7f976c4f5700 1 -- start start 2024-01-26T23:29:41.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.181+0000 7f976c4f5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97640acf30 0x7f97640c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.181+0000 7f976c4f5700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97640c4230 con 0x7f97640ad770 2024-01-26T23:29:41.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.181+0000 7f976c4f5700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97640cbf10 con 0x7f97640ad300 2024-01-26T23:29:41.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.181+0000 7f976c4f5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9764157be0 con 0x7f97640acf30 2024-01-26T23:29:41.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.181+0000 7f976aa92700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f97640ad770 0x7f97640ab1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49596/0 (socket says 172.21.15.46:49596) 2024-01-26T23:29:41.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.181+0000 7f976aa92700 1 -- 172.21.15.46:0/1761405420 learned_addr learned my addr 172.21.15.46:0/1761405420 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:41.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.181+0000 7f976a291700 1 --2- 172.21.15.46:0/1761405420 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97640acf30 0x7f97640c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.181+0000 7f976928f700 1 -- 172.21.15.46:0/1761405420 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371156585 0 0) 0x7f97640c4230 con 0x7f97640ad770 2024-01-26T23:29:41.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.181+0000 7f976928f700 1 -- 172.21.15.46:0/1761405420 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9750003570 con 0x7f97640ad770 2024-01-26T23:29:41.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.181+0000 7f976928f700 1 -- 172.21.15.46:0/1761405420 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158271458 0 0) 0x7f97640cbf10 con 0x7f97640ad300 2024-01-26T23:29:41.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.181+0000 7f976a291700 1 -- 172.21.15.46:0/1761405420 >> v1:172.21.15.106:6789/0 conn(0x7f97640ad300 legacy=0x7f97640ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.181+0000 7f976a291700 1 -- 172.21.15.46:0/1761405420 >> v1:172.21.15.46:6789/0 conn(0x7f97640ad770 legacy=0x7f97640ab1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.181+0000 7f976a291700 1 -- 172.21.15.46:0/1761405420 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97640c4460 con 0x7f97640acf30 2024-01-26T23:29:41.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.182+0000 7f976a291700 1 --2- 172.21.15.46:0/1761405420 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97640acf30 0x7f97640c3cf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9760009910 tx=0x7f97600091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0944acecd27fd18 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:41.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.182+0000 7f976928f700 1 -- 172.21.15.46:0/1761405420 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976000b680 con 0x7f97640acf30 2024-01-26T23:29:41.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.182+0000 7f976928f700 1 -- 172.21.15.46:0/1761405420 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976000b7e0 con 0x7f97640acf30 2024-01-26T23:29:41.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.182+0000 7f976928f700 1 -- 172.21.15.46:0/1761405420 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976000bb60 con 0x7f97640acf30 2024-01-26T23:29:41.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.182+0000 7f976c4f5700 1 -- 172.21.15.46:0/1761405420 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97640acf30 msgr2=0x7f97640c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.182+0000 7f976c4f5700 1 --2- 172.21.15.46:0/1761405420 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97640acf30 0x7f97640c3cf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9760009910 tx=0x7f97600091a0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.182+0000 7f976c4f5700 1 -- 172.21.15.46:0/1761405420 shutdown_connections 2024-01-26T23:29:41.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.182+0000 7f976c4f5700 1 --2- 172.21.15.46:0/1761405420 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97640acf30 0x7f97640c3cf0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.182+0000 7f976c4f5700 1 -- 172.21.15.46:0/1761405420 >> 172.21.15.46:0/1761405420 conn(0x7f9764097820 msgr2=0x7f9764068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:41.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.182+0000 7f976c4f5700 1 -- 172.21.15.46:0/1761405420 shutdown_connections 2024-01-26T23:29:41.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.183+0000 7f976c4f5700 1 -- 172.21.15.46:0/1761405420 wait complete. 2024-01-26T23:29:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.184+0000 7f976c4f5700 1 Processor -- start 2024-01-26T23:29:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.184+0000 7f976c4f5700 1 -- start start 2024-01-26T23:29:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.184+0000 7f976c4f5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97640acf30 0x7f97640b3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.184+0000 7f976c4f5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97640ad300 0x7f97641520a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.184+0000 7f976c4f5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9764169680 0x7f9764068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.184+0000 7f976c4f5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9764153c90 con 0x7f9764169680 2024-01-26T23:29:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.184+0000 7f976c4f5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9764169f40 con 0x7f97640acf30 2024-01-26T23:29:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.184+0000 7f976c4f5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976416a220 con 0x7f97640ad300 2024-01-26T23:29:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.185+0000 7f976aa92700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9764169680 0x7f9764068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.185+0000 7f976aa92700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9764169680 0x7f9764068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37128/0 (socket says 172.21.15.46:37128) 2024-01-26T23:29:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.185+0000 7f976aa92700 1 -- 172.21.15.46:0/1253098112 learned_addr learned my addr 172.21.15.46:0/1253098112 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.185+0000 7f976a291700 1 --2- 172.21.15.46:0/1253098112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97640acf30 0x7f97640b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.185+0000 7f9769a90700 1 --2- 172.21.15.46:0/1253098112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97640ad300 0x7f97641520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.185+0000 7f976aa92700 1 -- 172.21.15.46:0/1253098112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97640ad300 msgr2=0x7f97641520a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.185+0000 7f976aa92700 1 --2- 172.21.15.46:0/1253098112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97640ad300 0x7f97641520a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.185+0000 7f976aa92700 1 -- 172.21.15.46:0/1253098112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97640acf30 msgr2=0x7f97640b3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.185+0000 7f976aa92700 1 --2- 172.21.15.46:0/1253098112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97640acf30 0x7f97640b3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.185+0000 7f976aa92700 1 -- 172.21.15.46:0/1253098112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9760009420 con 0x7f9764169680 2024-01-26T23:29:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.185+0000 7f976aa92700 1 --2- 172.21.15.46:0/1253098112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9764169680 0x7f9764068a40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f975c001460 tx=0x7f975c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.185+0000 7f975a7fc700 1 -- 172.21.15.46:0/1253098112 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c00a870 con 0x7f9764169680 2024-01-26T23:29:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.185+0000 7f975a7fc700 1 -- 172.21.15.46:0/1253098112 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975c00a9d0 con 0x7f9764169680 2024-01-26T23:29:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.185+0000 7f976c4f5700 1 -- 172.21.15.46:0/1253098112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976416a500 con 0x7f9764169680 2024-01-26T23:29:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.185+0000 7f975a7fc700 1 -- 172.21.15.46:0/1253098112 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c00ad50 con 0x7f9764169680 2024-01-26T23:29:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.185+0000 7f976c4f5700 1 -- 172.21.15.46:0/1253098112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97641525e0 con 0x7f9764169680 2024-01-26T23:29:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.186+0000 7f976c4f5700 1 -- 172.21.15.46:0/1253098112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9764153e00 con 0x7f9764169680 2024-01-26T23:29:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.187+0000 7f975a7fc700 1 -- 172.21.15.46:0/1253098112 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f975c007300 con 0x7f9764169680 2024-01-26T23:29:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.187+0000 7f975a7fc700 1 --2- 172.21.15.46:0/1253098112 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9748066ea0 0x7f9748069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.187+0000 7f975a7fc700 1 -- 172.21.15.46:0/1253098112 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f975c0843f0 con 0x7f9764169680 2024-01-26T23:29:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.187+0000 7f976a291700 1 --2- 172.21.15.46:0/1253098112 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9748066ea0 0x7f9748069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.188+0000 7f976a291700 1 --2- 172.21.15.46:0/1253098112 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9748066ea0 0x7f9748069350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f97600093f0 tx=0x7f9760017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.192+0000 7f975a7fc700 1 -- 172.21.15.46:0/1253098112 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f975c051a80 con 0x7f9764169680 2024-01-26T23:29:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.399+0000 7f976c4f5700 1 -- 172.21.15.46:0/1253098112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f976404c710 con 0x7f9764169680 2024-01-26T23:29:41.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.400+0000 7f975a7fc700 1 -- 172.21.15.46:0/1253098112 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12127 (secure 0 0 0) 0x7f975c0550a0 con 0x7f9764169680 2024-01-26T23:29:41.402 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:29:41.402 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":16,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:40.511634+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:29:41.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.403+0000 7f976c4f5700 1 -- 172.21.15.46:0/1253098112 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9748066ea0 msgr2=0x7f9748069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.403+0000 7f976c4f5700 1 --2- 172.21.15.46:0/1253098112 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9748066ea0 0x7f9748069350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f97600093f0 tx=0x7f9760017000 comp rx=0 tx=0).stop 2024-01-26T23:29:41.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.403+0000 7f976c4f5700 1 -- 172.21.15.46:0/1253098112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9764169680 msgr2=0x7f9764068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.403+0000 7f976c4f5700 1 --2- 172.21.15.46:0/1253098112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9764169680 0x7f9764068a40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f975c001460 tx=0x7f975c001a40 comp rx=0 tx=0).stop 2024-01-26T23:29:41.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.403+0000 7f976c4f5700 1 -- 172.21.15.46:0/1253098112 shutdown_connections 2024-01-26T23:29:41.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.403+0000 7f976c4f5700 1 --2- 172.21.15.46:0/1253098112 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9748066ea0 0x7f9748069350 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.403+0000 7f976c4f5700 1 --2- 172.21.15.46:0/1253098112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9764169680 0x7f9764068a40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.403+0000 7f976c4f5700 1 --2- 172.21.15.46:0/1253098112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97640acf30 0x7f97640b3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.403+0000 7f976c4f5700 1 --2- 172.21.15.46:0/1253098112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97640ad300 0x7f97641520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.403+0000 7f976c4f5700 1 -- 172.21.15.46:0/1253098112 >> 172.21.15.46:0/1253098112 conn(0x7f9764097820 msgr2=0x7f976406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:41.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.403+0000 7f976c4f5700 1 -- 172.21.15.46:0/1253098112 shutdown_connections 2024-01-26T23:29:41.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.403+0000 7f976c4f5700 1 -- 172.21.15.46:0/1253098112 wait complete. 2024-01-26T23:29:41.418 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-26T23:29:41.418 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:29:41.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.556+0000 7f5754eb1700 1 Processor -- start 2024-01-26T23:29:41.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.556+0000 7f5754eb1700 1 -- start start 2024-01-26T23:29:41.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.557+0000 7f5754eb1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5750068b00 0x7f5750068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.557+0000 7f5754eb1700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f575006a660 0x7f5750159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.557+0000 7f5754eb1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f575006add0 0x7f575006b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.557+0000 7f5754eb1700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57500a9a80 con 0x7f5750068b00 2024-01-26T23:29:41.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.557+0000 7f5754eb1700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575006b790 con 0x7f575006a660 2024-01-26T23:29:41.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.557+0000 7f5754eb1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575006aab0 con 0x7f575006add0 2024-01-26T23:29:41.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.557+0000 7f574e59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5750068b00 0x7f5750068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.557+0000 7f574e59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5750068b00 0x7f5750068ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37140/0 (socket says 172.21.15.46:37140) 2024-01-26T23:29:41.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.557+0000 7f574e59c700 1 -- 172.21.15.46:0/2021509798 learned_addr learned my addr 172.21.15.46:0/2021509798 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:41.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.557+0000 7f574dd9b700 1 --2- 172.21.15.46:0/2021509798 >> v2:172.21.15.106:3300/0 conn(0x7f575006a660 0x7f5750159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.557+0000 7f574ed9d700 1 --2- 172.21.15.46:0/2021509798 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f575006add0 0x7f575006b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.557+0000 7f574e59c700 1 -- 172.21.15.46:0/2021509798 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f575006add0 msgr2=0x7f575006b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.557+0000 7f574e59c700 1 --2- 172.21.15.46:0/2021509798 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f575006add0 0x7f575006b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.557+0000 7f574e59c700 1 -- 172.21.15.46:0/2021509798 >> v2:172.21.15.106:3300/0 conn(0x7f575006a660 msgr2=0x7f5750159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.557+0000 7f574e59c700 1 --2- 172.21.15.46:0/2021509798 >> v2:172.21.15.106:3300/0 conn(0x7f575006a660 0x7f5750159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.557+0000 7f574e59c700 1 -- 172.21.15.46:0/2021509798 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575015a3e0 con 0x7f5750068b00 2024-01-26T23:29:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.558+0000 7f574e59c700 1 --2- 172.21.15.46:0/2021509798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5750068b00 0x7f5750068ed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5750063ae0 tx=0x7f5740009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfe04093b7e138ca server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.558+0000 7f574d59a700 1 -- 172.21.15.46:0/2021509798 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740005c10 con 0x7f5750068b00 2024-01-26T23:29:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.558+0000 7f574d59a700 1 -- 172.21.15.46:0/2021509798 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5740005d70 con 0x7f5750068b00 2024-01-26T23:29:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.558+0000 7f574d59a700 1 -- 172.21.15.46:0/2021509798 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740006260 con 0x7f5750068b00 2024-01-26T23:29:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.558+0000 7f5754eb1700 1 -- 172.21.15.46:0/2021509798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5750068b00 msgr2=0x7f5750068ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.558+0000 7f5754eb1700 1 --2- 172.21.15.46:0/2021509798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5750068b00 0x7f5750068ed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5750063ae0 tx=0x7f5740009220 comp rx=0 tx=0).stop 2024-01-26T23:29:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.559+0000 7f5754eb1700 1 -- 172.21.15.46:0/2021509798 shutdown_connections 2024-01-26T23:29:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.559+0000 7f5754eb1700 1 --2- 172.21.15.46:0/2021509798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5750068b00 0x7f5750068ed0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.559+0000 7f5754eb1700 1 --2- 172.21.15.46:0/2021509798 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f575006add0 0x7f575006b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.559+0000 7f5754eb1700 1 --2- 172.21.15.46:0/2021509798 >> v2:172.21.15.106:3300/0 conn(0x7f575006a660 0x7f5750159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.559+0000 7f5754eb1700 1 -- 172.21.15.46:0/2021509798 >> 172.21.15.46:0/2021509798 conn(0x7f57500ca100 msgr2=0x7f575006e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:41.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.559+0000 7f5754eb1700 1 -- 172.21.15.46:0/2021509798 shutdown_connections 2024-01-26T23:29:41.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.559+0000 7f5754eb1700 1 -- 172.21.15.46:0/2021509798 wait complete. 2024-01-26T23:29:41.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.560+0000 7f5754eb1700 1 Processor -- start 2024-01-26T23:29:41.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.560+0000 7f5754eb1700 1 -- start start 2024-01-26T23:29:41.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f5754eb1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5750068b00 0x7f5750155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f5754eb1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f575006a660 0x7f57501584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f5754eb1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f575006add0 0x7f5750158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f5754eb1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57500b0d30 con 0x7f5750068b00 2024-01-26T23:29:41.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f5754eb1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5750156590 con 0x7f575006add0 2024-01-26T23:29:41.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f5754eb1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57500b0720 con 0x7f575006a660 2024-01-26T23:29:41.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f574e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5750068b00 0x7f5750155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f574e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5750068b00 0x7f5750155a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37154/0 (socket says 172.21.15.46:37154) 2024-01-26T23:29:41.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f574e59c700 1 -- 172.21.15.46:0/1464913092 learned_addr learned my addr 172.21.15.46:0/1464913092 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:41.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f574dd9b700 1 --2- 172.21.15.46:0/1464913092 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f575006a660 0x7f57501584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f574ed9d700 1 --2- 172.21.15.46:0/1464913092 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f575006add0 0x7f5750158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f574e59c700 1 -- 172.21.15.46:0/1464913092 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f575006a660 msgr2=0x7f57501584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f574e59c700 1 --2- 172.21.15.46:0/1464913092 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f575006a660 0x7f57501584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f574e59c700 1 -- 172.21.15.46:0/1464913092 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f575006add0 msgr2=0x7f5750158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f574e59c700 1 --2- 172.21.15.46:0/1464913092 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f575006add0 0x7f5750158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f574e59c700 1 -- 172.21.15.46:0/1464913092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5740009480 con 0x7f5750068b00 2024-01-26T23:29:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f574e59c700 1 --2- 172.21.15.46:0/1464913092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5750068b00 0x7f5750155a80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5740009f50 tx=0x7f5740016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f573e7fc700 1 -- 172.21.15.46:0/1464913092 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740011850 con 0x7f5750068b00 2024-01-26T23:29:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.561+0000 7f573e7fc700 1 -- 172.21.15.46:0/1464913092 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57400119b0 con 0x7f5750068b00 2024-01-26T23:29:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.562+0000 7f5754eb1700 1 -- 172.21.15.46:0/1464913092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57501590b0 con 0x7f5750068b00 2024-01-26T23:29:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.562+0000 7f573e7fc700 1 -- 172.21.15.46:0/1464913092 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740011d70 con 0x7f5750068b00 2024-01-26T23:29:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.562+0000 7f5754eb1700 1 -- 172.21.15.46:0/1464913092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57500b09c0 con 0x7f5750068b00 2024-01-26T23:29:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.562+0000 7f5754eb1700 1 -- 172.21.15.46:0/1464913092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575004c710 con 0x7f5750068b00 2024-01-26T23:29:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.563+0000 7f573e7fc700 1 -- 172.21.15.46:0/1464913092 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f574001b640 con 0x7f5750068b00 2024-01-26T23:29:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.563+0000 7f573e7fc700 1 --2- 172.21.15.46:0/1464913092 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5730066ea0 0x7f5730069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.564+0000 7f573e7fc700 1 -- 172.21.15.46:0/1464913092 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f57400858c0 con 0x7f5750068b00 2024-01-26T23:29:41.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.564+0000 7f574dd9b700 1 --2- 172.21.15.46:0/1464913092 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5730066ea0 0x7f5730069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.564+0000 7f574dd9b700 1 --2- 172.21.15.46:0/1464913092 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5730066ea0 0x7f5730069350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5744009d10 tx=0x7f57440091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:41.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.568+0000 7f573e7fc700 1 -- 172.21.15.46:0/1464913092 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5740052f40 con 0x7f5750068b00 2024-01-26T23:29:41.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.772+0000 7f5754eb1700 1 -- 172.21.15.46:0/1464913092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f575006edf0 con 0x7f5750068b00 2024-01-26T23:29:41.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.773+0000 7f573e7fc700 1 -- 172.21.15.46:0/1464913092 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f574002a0a0 con 0x7f5750068b00 2024-01-26T23:29:41.775 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:29:41.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.775+0000 7f5754eb1700 1 -- 172.21.15.46:0/1464913092 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5730066ea0 msgr2=0x7f5730069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.775+0000 7f5754eb1700 1 --2- 172.21.15.46:0/1464913092 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5730066ea0 0x7f5730069350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5744009d10 tx=0x7f57440091a0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.775+0000 7f5754eb1700 1 -- 172.21.15.46:0/1464913092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5750068b00 msgr2=0x7f5750155a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.775+0000 7f5754eb1700 1 --2- 172.21.15.46:0/1464913092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5750068b00 0x7f5750155a80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5740009f50 tx=0x7f5740016000 comp rx=0 tx=0).stop 2024-01-26T23:29:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.775+0000 7f5754eb1700 1 -- 172.21.15.46:0/1464913092 shutdown_connections 2024-01-26T23:29:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.775+0000 7f5754eb1700 1 --2- 172.21.15.46:0/1464913092 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5730066ea0 0x7f5730069350 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.775+0000 7f5754eb1700 1 --2- 172.21.15.46:0/1464913092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5750068b00 0x7f5750155a80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.775+0000 7f5754eb1700 1 --2- 172.21.15.46:0/1464913092 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f575006add0 0x7f5750158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.775+0000 7f5754eb1700 1 --2- 172.21.15.46:0/1464913092 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f575006a660 0x7f57501584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.775+0000 7f5754eb1700 1 -- 172.21.15.46:0/1464913092 >> 172.21.15.46:0/1464913092 conn(0x7f57500ca100 msgr2=0x7f575015c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.775+0000 7f5754eb1700 1 -- 172.21.15.46:0/1464913092 shutdown_connections 2024-01-26T23:29:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.775+0000 7f5754eb1700 1 -- 172.21.15.46:0/1464913092 wait complete. 2024-01-26T23:29:41.790 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-26T23:29:41.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.919+0000 7f132a278700 1 Processor -- start 2024-01-26T23:29:41.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.920+0000 7f132a278700 1 -- start start 2024-01-26T23:29:41.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.920+0000 7f132a278700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f13240b0a20 0x7f13240cae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.920+0000 7f132a278700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13240c4410 0x7f1324161300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.920+0000 7f132a278700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13240cb840 con 0x7f13240cb3d0 2024-01-26T23:29:41.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.920+0000 7f132a278700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1324157c00 con 0x7f13240c4410 2024-01-26T23:29:41.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.920+0000 7f132a278700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13240b0df0 con 0x7f13240b0a20 2024-01-26T23:29:41.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.920+0000 7f1328815700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13240c4410 0x7f1324161300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.921+0000 7f13237fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f13240cb3d0 0x7f13240ccc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49600/0 (socket says 172.21.15.46:49600) 2024-01-26T23:29:41.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.921+0000 7f13237fe700 1 -- 172.21.15.46:0/897777913 learned_addr learned my addr 172.21.15.46:0/897777913 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:41.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.921+0000 7f1323fff700 1 --2- 172.21.15.46:0/897777913 >> v2:172.21.15.106:3300/0 conn(0x7f13240b0a20 0x7f13240cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.921+0000 7f1328815700 1 -- 172.21.15.46:0/897777913 >> v2:172.21.15.106:3300/0 conn(0x7f13240b0a20 msgr2=0x7f13240cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:41.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.921+0000 7f1328815700 1 --2- 172.21.15.46:0/897777913 >> v2:172.21.15.106:3300/0 conn(0x7f13240b0a20 0x7f13240cae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.921+0000 7f1328815700 1 -- 172.21.15.46:0/897777913 >> v1:172.21.15.46:6789/0 conn(0x7f13240cb3d0 legacy=0x7f13240ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.921+0000 7f1328815700 1 -- 172.21.15.46:0/897777913 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13240c9df0 con 0x7f13240c4410 2024-01-26T23:29:41.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.921+0000 7f1322ffd700 1 -- 172.21.15.46:0/897777913 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3241518615 0 0) 0x7f13240cb840 con 0x7f13240cb3d0 2024-01-26T23:29:41.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.921+0000 7f1328815700 1 --2- 172.21.15.46:0/897777913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240c4410 0x7f1324161300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f132406f060 tx=0x7f131c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90143b5800398ec5 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:41.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.921+0000 7f1322ffd700 1 -- 172.21.15.46:0/897777913 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c00e070 con 0x7f13240c4410 2024-01-26T23:29:41.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.921+0000 7f1322ffd700 1 -- 172.21.15.46:0/897777913 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131c00bc30 con 0x7f13240c4410 2024-01-26T23:29:41.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.921+0000 7f1322ffd700 1 -- 172.21.15.46:0/897777913 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c012430 con 0x7f13240c4410 2024-01-26T23:29:41.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.922+0000 7f132a278700 1 -- 172.21.15.46:0/897777913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240c4410 msgr2=0x7f1324161300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.922+0000 7f132a278700 1 --2- 172.21.15.46:0/897777913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240c4410 0x7f1324161300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f132406f060 tx=0x7f131c009780 comp rx=0 tx=0).stop 2024-01-26T23:29:41.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.922+0000 7f132a278700 1 -- 172.21.15.46:0/897777913 shutdown_connections 2024-01-26T23:29:41.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.922+0000 7f132a278700 1 --2- 172.21.15.46:0/897777913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240c4410 0x7f1324161300 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.922+0000 7f132a278700 1 --2- 172.21.15.46:0/897777913 >> v2:172.21.15.106:3300/0 conn(0x7f13240b0a20 0x7f13240cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.922+0000 7f132a278700 1 -- 172.21.15.46:0/897777913 >> 172.21.15.46:0/897777913 conn(0x7f1324097820 msgr2=0x7f1324068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:41.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.922+0000 7f132a278700 1 -- 172.21.15.46:0/897777913 shutdown_connections 2024-01-26T23:29:41.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.922+0000 7f132a278700 1 -- 172.21.15.46:0/897777913 wait complete. 2024-01-26T23:29:41.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.923+0000 7f132a278700 1 Processor -- start 2024-01-26T23:29:41.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.923+0000 7f132a278700 1 -- start start 2024-01-26T23:29:41.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.924+0000 7f132a278700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13240b0a20 0x7f132416a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.924+0000 7f132a278700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240c4410 0x7f1324167d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.924+0000 7f132a278700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13240cb3d0 0x7f1324168390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.924+0000 7f132a278700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13240b2e60 con 0x7f13240c4410 2024-01-26T23:29:41.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.924+0000 7f132a278700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f132416b790 con 0x7f13240cb3d0 2024-01-26T23:29:41.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.924+0000 7f132a278700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13241688d0 con 0x7f13240b0a20 2024-01-26T23:29:41.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.924+0000 7f13237fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240c4410 0x7f1324167d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.924+0000 7f13237fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240c4410 0x7f1324167d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37180/0 (socket says 172.21.15.46:37180) 2024-01-26T23:29:41.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.924+0000 7f13237fe700 1 -- 172.21.15.46:0/3561028679 learned_addr learned my addr 172.21.15.46:0/3561028679 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:41.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.924+0000 7f1323fff700 1 --2- 172.21.15.46:0/3561028679 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13240b0a20 0x7f132416a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.924+0000 7f1328815700 1 --2- 172.21.15.46:0/3561028679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13240cb3d0 0x7f1324168390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.925+0000 7f13237fe700 1 -- 172.21.15.46:0/3561028679 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13240b0a20 msgr2=0x7f132416a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.925+0000 7f13237fe700 1 --2- 172.21.15.46:0/3561028679 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13240b0a20 0x7f132416a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.925+0000 7f13237fe700 1 -- 172.21.15.46:0/3561028679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13240cb3d0 msgr2=0x7f1324168390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:41.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.925+0000 7f13237fe700 1 --2- 172.21.15.46:0/3561028679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13240cb3d0 0x7f1324168390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:41.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.925+0000 7f13237fe700 1 -- 172.21.15.46:0/3561028679 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c009a20 con 0x7f13240c4410 2024-01-26T23:29:41.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.925+0000 7f13237fe700 1 --2- 172.21.15.46:0/3561028679 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240c4410 0x7f1324167d00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f130c00c7e0 tx=0x7f130c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:41.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.925+0000 7f131bfff700 1 -- 172.21.15.46:0/3561028679 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c005b30 con 0x7f13240c4410 2024-01-26T23:29:41.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.925+0000 7f131bfff700 1 -- 172.21.15.46:0/3561028679 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130c005c90 con 0x7f13240c4410 2024-01-26T23:29:41.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.925+0000 7f132a278700 1 -- 172.21.15.46:0/3561028679 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f132416aef0 con 0x7f13240c4410 2024-01-26T23:29:41.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.925+0000 7f131bfff700 1 -- 172.21.15.46:0/3561028679 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c007300 con 0x7f13240c4410 2024-01-26T23:29:41.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.925+0000 7f132a278700 1 -- 172.21.15.46:0/3561028679 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f132416b4b0 con 0x7f13240c4410 2024-01-26T23:29:41.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.926+0000 7f132a278700 1 -- 172.21.15.46:0/3561028679 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f132404c710 con 0x7f13240c4410 2024-01-26T23:29:41.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.926+0000 7f131bfff700 1 -- 172.21.15.46:0/3561028679 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f130c01b3b0 con 0x7f13240c4410 2024-01-26T23:29:41.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.927+0000 7f131bfff700 1 --2- 172.21.15.46:0/3561028679 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1304066ea0 0x7f1304069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:41.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.927+0000 7f131bfff700 1 -- 172.21.15.46:0/3561028679 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f130c0834f0 con 0x7f13240c4410 2024-01-26T23:29:41.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.927+0000 7f1323fff700 1 --2- 172.21.15.46:0/3561028679 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1304066ea0 0x7f1304069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:41.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.928+0000 7f1323fff700 1 --2- 172.21.15.46:0/3561028679 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1304066ea0 0x7f1304069350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f13140061f0 tx=0x7f1314008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:41.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:41.931+0000 7f131bfff700 1 -- 172.21.15.46:0/3561028679 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f130c050b90 con 0x7f13240c4410 2024-01-26T23:29:42.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.142+0000 7f132a278700 1 -- 172.21.15.46:0/3561028679 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f13240ca0a0 con 0x7f13240c4410 2024-01-26T23:29:42.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.142+0000 7f131bfff700 1 -- 172.21.15.46:0/3561028679 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f130c017060 con 0x7f13240c4410 2024-01-26T23:29:42.144 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:29:42.144 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-26T23:29:42.144 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:29:42.144 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-01-26T23:29:42.145 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-01-26T23:29:42.145 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:29:42.145 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-26T23:29:42.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.144+0000 7f132a278700 1 -- 172.21.15.46:0/3561028679 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1304066ea0 msgr2=0x7f1304069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:42.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.144+0000 7f132a278700 1 --2- 172.21.15.46:0/3561028679 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1304066ea0 0x7f1304069350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f13140061f0 tx=0x7f1314008040 comp rx=0 tx=0).stop 2024-01-26T23:29:42.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.144+0000 7f132a278700 1 -- 172.21.15.46:0/3561028679 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240c4410 msgr2=0x7f1324167d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:42.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.144+0000 7f132a278700 1 --2- 172.21.15.46:0/3561028679 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240c4410 0x7f1324167d00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f130c00c7e0 tx=0x7f130c007df0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.144+0000 7f132a278700 1 -- 172.21.15.46:0/3561028679 shutdown_connections 2024-01-26T23:29:42.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.144+0000 7f132a278700 1 --2- 172.21.15.46:0/3561028679 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1304066ea0 0x7f1304069350 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.144+0000 7f132a278700 1 --2- 172.21.15.46:0/3561028679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13240cb3d0 0x7f1324168390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.144+0000 7f132a278700 1 --2- 172.21.15.46:0/3561028679 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13240b0a20 0x7f132416a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.144+0000 7f132a278700 1 --2- 172.21.15.46:0/3561028679 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240c4410 0x7f1324167d00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.144+0000 7f132a278700 1 -- 172.21.15.46:0/3561028679 >> 172.21.15.46:0/3561028679 conn(0x7f1324097820 msgr2=0x7f132406c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:42.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.144+0000 7f132a278700 1 -- 172.21.15.46:0/3561028679 shutdown_connections 2024-01-26T23:29:42.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.144+0000 7f132a278700 1 -- 172.21.15.46:0/3561028679 wait complete. 2024-01-26T23:29:42.159 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-26T23:29:42.159 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-26T23:29:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.292+0000 7effa897b700 1 Processor -- start 2024-01-26T23:29:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.292+0000 7effa897b700 1 -- start start 2024-01-26T23:29:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.293+0000 7effa897b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa4156b00 0x7effa415ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.293+0000 7effa897b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa4157000 con 0x7effa40c9980 2024-01-26T23:29:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.293+0000 7effa897b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa406f3e0 con 0x7effa40c9d50 2024-01-26T23:29:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.293+0000 7effa897b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa406f6c0 con 0x7effa4156b00 2024-01-26T23:29:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.293+0000 7effa259c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7effa40c9980 0x7effa406eb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49602/0 (socket says 172.21.15.46:49602) 2024-01-26T23:29:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.293+0000 7effa259c700 1 -- 172.21.15.46:0/70519035 learned_addr learned my addr 172.21.15.46:0/70519035 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.293+0000 7eff99d9b700 1 --2- 172.21.15.46:0/70519035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa4156b00 0x7effa415ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.293+0000 7effa1d9b700 1 -- 172.21.15.46:0/70519035 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216031749 0 0) 0x7effa4157000 con 0x7effa40c9980 2024-01-26T23:29:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.293+0000 7effa1d9b700 1 -- 172.21.15.46:0/70519035 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff90003570 con 0x7effa40c9980 2024-01-26T23:29:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.293+0000 7effa1d9b700 1 -- 172.21.15.46:0/70519035 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2684205851 0 0) 0x7eff90003570 con 0x7effa40c9980 2024-01-26T23:29:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.294+0000 7eff99d9b700 1 -- 172.21.15.46:0/70519035 >> v1:172.21.15.106:6789/0 conn(0x7effa40c9d50 legacy=0x7effa40b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.294+0000 7eff99d9b700 1 -- 172.21.15.46:0/70519035 >> v1:172.21.15.46:6789/0 conn(0x7effa40c9980 legacy=0x7effa406eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.294+0000 7eff99d9b700 1 -- 172.21.15.46:0/70519035 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa406f9a0 con 0x7effa4156b00 2024-01-26T23:29:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.294+0000 7eff99d9b700 1 --2- 172.21.15.46:0/70519035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa4156b00 0x7effa415ae10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7eff8c009910 tx=0x7eff8c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ce04d83c08163d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:42.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.294+0000 7effa1d9b700 1 -- 172.21.15.46:0/70519035 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff94001470 con 0x7effa4156b00 2024-01-26T23:29:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.294+0000 7effa1d9b700 1 -- 172.21.15.46:0/70519035 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8c011420 con 0x7effa4156b00 2024-01-26T23:29:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.294+0000 7effa1d9b700 1 -- 172.21.15.46:0/70519035 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0117c0 con 0x7effa4156b00 2024-01-26T23:29:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.294+0000 7effa897b700 1 -- 172.21.15.46:0/70519035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa4156b00 msgr2=0x7effa415ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.294+0000 7effa897b700 1 --2- 172.21.15.46:0/70519035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa4156b00 0x7effa415ae10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7eff8c009910 tx=0x7eff8c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.295+0000 7effa897b700 1 -- 172.21.15.46:0/70519035 shutdown_connections 2024-01-26T23:29:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.295+0000 7effa897b700 1 --2- 172.21.15.46:0/70519035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa4156b00 0x7effa415ae10 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.295+0000 7effa897b700 1 -- 172.21.15.46:0/70519035 >> 172.21.15.46:0/70519035 conn(0x7effa4069510 msgr2=0x7effa40aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.295+0000 7effa897b700 1 -- 172.21.15.46:0/70519035 shutdown_connections 2024-01-26T23:29:42.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.295+0000 7effa897b700 1 -- 172.21.15.46:0/70519035 wait complete. 2024-01-26T23:29:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.296+0000 7effa897b700 1 Processor -- start 2024-01-26T23:29:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.296+0000 7effa897b700 1 -- start start 2024-01-26T23:29:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7effa897b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effa40c9980 0x7effa4067b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7effa897b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effa40c9d50 0x7effa40c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7effa897b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa4156b00 0x7effa4144d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7effa897b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa40c7bf0 con 0x7effa40c9980 2024-01-26T23:29:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7effa897b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa40c6fe0 con 0x7effa40c9d50 2024-01-26T23:29:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7effa897b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa40c72c0 con 0x7effa4156b00 2024-01-26T23:29:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7effa259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effa40c9980 0x7effa4067b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:42.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7effa259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effa40c9980 0x7effa4067b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37188/0 (socket says 172.21.15.46:37188) 2024-01-26T23:29:42.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7effa259c700 1 -- 172.21.15.46:0/2972184406 learned_addr learned my addr 172.21.15.46:0/2972184406 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:42.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7eff99d9b700 1 --2- 172.21.15.46:0/2972184406 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effa40c9d50 0x7effa40c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:42.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7effa2d9d700 1 --2- 172.21.15.46:0/2972184406 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa4156b00 0x7effa4144d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:42.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7effa259c700 1 -- 172.21.15.46:0/2972184406 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa4156b00 msgr2=0x7effa4144d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:42.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7effa259c700 1 --2- 172.21.15.46:0/2972184406 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa4156b00 0x7effa4144d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7effa259c700 1 -- 172.21.15.46:0/2972184406 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effa40c9d50 msgr2=0x7effa40c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:42.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7effa259c700 1 --2- 172.21.15.46:0/2972184406 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effa40c9d50 0x7effa40c6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7effa259c700 1 -- 172.21.15.46:0/2972184406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff8c009420 con 0x7effa40c9980 2024-01-26T23:29:42.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7eff99d9b700 1 --2- 172.21.15.46:0/2972184406 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effa40c9d50 0x7effa40c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:29:42.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.297+0000 7effa259c700 1 --2- 172.21.15.46:0/2972184406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effa40c9980 0x7effa4067b80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7eff9400cb90 tx=0x7eff94001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:42.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.298+0000 7eff9affd700 1 -- 172.21.15.46:0/2972184406 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff94001d40 con 0x7effa40c9980 2024-01-26T23:29:42.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.298+0000 7eff9affd700 1 -- 172.21.15.46:0/2972184406 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff94007360 con 0x7effa40c9980 2024-01-26T23:29:42.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.298+0000 7effa897b700 1 -- 172.21.15.46:0/2972184406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa40c75a0 con 0x7effa40c9980 2024-01-26T23:29:42.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.298+0000 7eff9affd700 1 -- 172.21.15.46:0/2972184406 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff94004830 con 0x7effa40c9980 2024-01-26T23:29:42.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.298+0000 7effa897b700 1 -- 172.21.15.46:0/2972184406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa4145950 con 0x7effa40c9980 2024-01-26T23:29:42.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.299+0000 7effa897b700 1 -- 172.21.15.46:0/2972184406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa404c710 con 0x7effa40c9980 2024-01-26T23:29:42.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.299+0000 7eff9affd700 1 -- 172.21.15.46:0/2972184406 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 84686+0+0 (secure 0 0 0) 0x7eff94004ba0 con 0x7effa40c9980 2024-01-26T23:29:42.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.299+0000 7eff9affd700 1 --2- 172.21.15.46:0/2972184406 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff84066ea0 0x7eff84069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:42.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.299+0000 7eff9affd700 1 -- 172.21.15.46:0/2972184406 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7eff940839b0 con 0x7effa40c9980 2024-01-26T23:29:42.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.300+0000 7eff99d9b700 1 --2- 172.21.15.46:0/2972184406 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff84066ea0 0x7eff84069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:42.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.300+0000 7eff99d9b700 1 --2- 172.21.15.46:0/2972184406 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff84066ea0 0x7eff84069350 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0093f0 tx=0x7eff8c017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:42.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.304+0000 7eff9affd700 1 -- 172.21.15.46:0/2972184406 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7eff94051030 con 0x7effa40c9980 2024-01-26T23:29:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.507+0000 7effa897b700 1 -- 172.21.15.46:0/2972184406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7effa40c95a0 con 0x7effa40c9980 2024-01-26T23:29:42.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.548+0000 7eff9affd700 1 -- 172.21.15.46:0/2972184406 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7eff94051400 con 0x7effa40c9980 2024-01-26T23:29:42.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.592+0000 7eff9affd700 1 -- 172.21.15.46:0/2972184406 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7eff9405bee0 con 0x7effa40c9980 2024-01-26T23:29:42.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.596+0000 7effa897b700 1 -- 172.21.15.46:0/2972184406 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff84066ea0 msgr2=0x7eff84069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:42.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.596+0000 7effa897b700 1 --2- 172.21.15.46:0/2972184406 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff84066ea0 0x7eff84069350 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0093f0 tx=0x7eff8c017000 comp rx=0 tx=0).stop 2024-01-26T23:29:42.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.596+0000 7effa897b700 1 -- 172.21.15.46:0/2972184406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effa40c9980 msgr2=0x7effa4067b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:42.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.596+0000 7effa897b700 1 --2- 172.21.15.46:0/2972184406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effa40c9980 0x7effa4067b80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7eff9400cb90 tx=0x7eff94001ab0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.597+0000 7effa897b700 1 -- 172.21.15.46:0/2972184406 shutdown_connections 2024-01-26T23:29:42.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.597+0000 7effa897b700 1 --2- 172.21.15.46:0/2972184406 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff84066ea0 0x7eff84069350 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.597+0000 7effa897b700 1 --2- 172.21.15.46:0/2972184406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effa40c9980 0x7effa4067b80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.597+0000 7effa897b700 1 --2- 172.21.15.46:0/2972184406 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa4156b00 0x7effa4144d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.597+0000 7effa897b700 1 --2- 172.21.15.46:0/2972184406 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effa40c9d50 0x7effa40c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.597+0000 7effa897b700 1 -- 172.21.15.46:0/2972184406 >> 172.21.15.46:0/2972184406 conn(0x7effa4069510 msgr2=0x7effa41594a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:42.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.597+0000 7effa897b700 1 -- 172.21.15.46:0/2972184406 shutdown_connections 2024-01-26T23:29:42.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.597+0000 7effa897b700 1 -- 172.21.15.46:0/2972184406 wait complete. 2024-01-26T23:29:42.601 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-01-26T23:29:42.614 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-01-26T23:29:42.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.753+0000 7fb7de2d8700 1 Processor -- start 2024-01-26T23:29:42.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.753+0000 7fb7de2d8700 1 -- start start 2024-01-26T23:29:42.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.753+0000 7fb7de2d8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb7d8159750 0x7fb7d815df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:42.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.754+0000 7fb7de2d8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d815ab30 con 0x7fb7d80b3a10 2024-01-26T23:29:42.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.754+0000 7fb7de2d8700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d80b2510 con 0x7fb7d8069cd0 2024-01-26T23:29:42.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.754+0000 7fb7de2d8700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7d8069840 con 0x7fb7d8159750 2024-01-26T23:29:42.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.754+0000 7fb7d7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb7d8159750 0x7fb7d815df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:42.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.754+0000 7fb7dc875700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb7d80b3a10 0x7fb7d80b3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49618/0 (socket says 172.21.15.46:49618) 2024-01-26T23:29:42.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.754+0000 7fb7dc875700 1 -- 172.21.15.46:0/94195560 learned_addr learned my addr 172.21.15.46:0/94195560 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:42.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.754+0000 7fb7d7fff700 1 -- 172.21.15.46:0/94195560 >> v1:172.21.15.106:6789/0 conn(0x7fb7d8069cd0 legacy=0x7fb7d80b1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:42.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.754+0000 7fb7d7fff700 1 -- 172.21.15.46:0/94195560 >> v1:172.21.15.46:6789/0 conn(0x7fb7d80b3a10 legacy=0x7fb7d80b3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:42.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.754+0000 7fb7d7fff700 1 -- 172.21.15.46:0/94195560 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d806a140 con 0x7fb7d8159750 2024-01-26T23:29:42.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.754+0000 7fb7d6ffd700 1 -- 172.21.15.46:0/94195560 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583281628 0 0) 0x7fb7d0001490 con 0x7fb7d80b3a10 2024-01-26T23:29:42.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.754+0000 7fb7d7fff700 1 --2- 172.21.15.46:0/94195560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d8159750 0x7fb7d815df50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80098e0 tx=0x7fb7c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3940797b0acdcc94 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:42.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.754+0000 7fb7d6ffd700 1 -- 172.21.15.46:0/94195560 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8009e80 con 0x7fb7d8159750 2024-01-26T23:29:42.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.754+0000 7fb7d6ffd700 1 -- 172.21.15.46:0/94195560 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c800b860 con 0x7fb7d8159750 2024-01-26T23:29:42.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.754+0000 7fb7d6ffd700 1 -- 172.21.15.46:0/94195560 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c800bbe0 con 0x7fb7d8159750 2024-01-26T23:29:42.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.755+0000 7fb7de2d8700 1 -- 172.21.15.46:0/94195560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d8159750 msgr2=0x7fb7d815df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:42.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.755+0000 7fb7de2d8700 1 --2- 172.21.15.46:0/94195560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d8159750 0x7fb7d815df50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80098e0 tx=0x7fb7c80091a0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.755+0000 7fb7de2d8700 1 -- 172.21.15.46:0/94195560 shutdown_connections 2024-01-26T23:29:42.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.755+0000 7fb7de2d8700 1 --2- 172.21.15.46:0/94195560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d8159750 0x7fb7d815df50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.755+0000 7fb7de2d8700 1 -- 172.21.15.46:0/94195560 >> 172.21.15.46:0/94195560 conn(0x7fb7d8067930 msgr2=0x7fb7d8067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:42.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.755+0000 7fb7de2d8700 1 -- 172.21.15.46:0/94195560 shutdown_connections 2024-01-26T23:29:42.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.755+0000 7fb7de2d8700 1 -- 172.21.15.46:0/94195560 wait complete. 2024-01-26T23:29:42.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.756+0000 7fb7de2d8700 1 Processor -- start 2024-01-26T23:29:42.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.757+0000 7fb7de2d8700 1 -- start start 2024-01-26T23:29:42.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.757+0000 7fb7de2d8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7d8069cd0 0x7fb7d816a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:42.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.757+0000 7fb7de2d8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d80b3a10 0x7fb7d81460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:42.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.757+0000 7fb7de2d8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7d8159750 0x7fb7d815acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:42.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.757+0000 7fb7de2d8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d80af750 con 0x7fb7d80b3a10 2024-01-26T23:29:42.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.757+0000 7fb7de2d8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d8169e50 con 0x7fb7d8069cd0 2024-01-26T23:29:42.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.757+0000 7fb7de2d8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d815e0c0 con 0x7fb7d8159750 2024-01-26T23:29:42.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.757+0000 7fb7d77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d80b3a10 0x7fb7d81460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:42.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.757+0000 7fb7d77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d80b3a10 0x7fb7d81460e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37202/0 (socket says 172.21.15.46:37202) 2024-01-26T23:29:42.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.757+0000 7fb7d77fe700 1 -- 172.21.15.46:0/3666906203 learned_addr learned my addr 172.21.15.46:0/3666906203 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:42.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.757+0000 7fb7d7fff700 1 --2- 172.21.15.46:0/3666906203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7d8069cd0 0x7fb7d816a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:42.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.757+0000 7fb7dc875700 1 --2- 172.21.15.46:0/3666906203 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7d8159750 0x7fb7d815acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:42.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.758+0000 7fb7d77fe700 1 -- 172.21.15.46:0/3666906203 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7d8159750 msgr2=0x7fb7d815acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:42.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.758+0000 7fb7d77fe700 1 --2- 172.21.15.46:0/3666906203 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7d8159750 0x7fb7d815acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.758+0000 7fb7d77fe700 1 -- 172.21.15.46:0/3666906203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7d8069cd0 msgr2=0x7fb7d816a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:42.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.758+0000 7fb7d77fe700 1 --2- 172.21.15.46:0/3666906203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7d8069cd0 0x7fb7d816a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:42.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.758+0000 7fb7d77fe700 1 -- 172.21.15.46:0/3666906203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c8009420 con 0x7fb7d80b3a10 2024-01-26T23:29:42.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.758+0000 7fb7dc875700 1 --2- 172.21.15.46:0/3666906203 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7d8159750 0x7fb7d815acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:29:42.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.758+0000 7fb7d77fe700 1 --2- 172.21.15.46:0/3666906203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d80b3a10 0x7fb7d81460e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00c5c0 tx=0x7fb7cc007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:42.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.758+0000 7fb7bffff700 1 -- 172.21.15.46:0/3666906203 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc011590 con 0x7fb7d80b3a10 2024-01-26T23:29:42.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.758+0000 7fb7bffff700 1 -- 172.21.15.46:0/3666906203 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7cc011710 con 0x7fb7d80b3a10 2024-01-26T23:29:42.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.758+0000 7fb7de2d8700 1 -- 172.21.15.46:0/3666906203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d80aeeb0 con 0x7fb7d80b3a10 2024-01-26T23:29:42.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.758+0000 7fb7bffff700 1 -- 172.21.15.46:0/3666906203 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc011ad0 con 0x7fb7d80b3a10 2024-01-26T23:29:42.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.758+0000 7fb7de2d8700 1 -- 172.21.15.46:0/3666906203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d80af240 con 0x7fb7d80b3a10 2024-01-26T23:29:42.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.759+0000 7fb7de2d8700 1 -- 172.21.15.46:0/3666906203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d804c710 con 0x7fb7d80b3a10 2024-01-26T23:29:42.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.759+0000 7fb7bffff700 1 -- 172.21.15.46:0/3666906203 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb7cc01a360 con 0x7fb7d80b3a10 2024-01-26T23:29:42.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.760+0000 7fb7bffff700 1 --2- 172.21.15.46:0/3666906203 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb7c4066f30 0x7fb7c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:42.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.760+0000 7fb7bffff700 1 -- 172.21.15.46:0/3666906203 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5223+0+0 (secure 0 0 0) 0x7fb7cc083210 con 0x7fb7d80b3a10 2024-01-26T23:29:42.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.760+0000 7fb7d7fff700 1 --2- 172.21.15.46:0/3666906203 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb7c4066f30 0x7fb7c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:42.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.761+0000 7fb7d7fff700 1 --2- 172.21.15.46:0/3666906203 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb7c4066f30 0x7fb7c40693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80093f0 tx=0x7fb7c8017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:42.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.765+0000 7fb7bffff700 1 -- 172.21.15.46:0/3666906203 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb7cc050660 con 0x7fb7d80b3a10 2024-01-26T23:29:42.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:42.970+0000 7fb7de2d8700 1 -- 172.21.15.46:0/3666906203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7fb7d80cad90 con 0x7fb7d80b3a10 2024-01-26T23:29:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.594+0000 7fb7bffff700 1 -- 172.21.15.46:0/3666906203 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) v1 ==== 171+0+0 (secure 0 0 0) 0x7fb7cc053c80 con 0x7fb7d80b3a10 2024-01-26T23:29:43.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.601+0000 7fb7de2d8700 1 -- 172.21.15.46:0/3666906203 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb7c4066f30 msgr2=0x7fb7c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:43.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.601+0000 7fb7de2d8700 1 --2- 172.21.15.46:0/3666906203 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb7c4066f30 0x7fb7c40693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80093f0 tx=0x7fb7c8017000 comp rx=0 tx=0).stop 2024-01-26T23:29:43.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.601+0000 7fb7de2d8700 1 -- 172.21.15.46:0/3666906203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d80b3a10 msgr2=0x7fb7d81460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:43.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.601+0000 7fb7de2d8700 1 --2- 172.21.15.46:0/3666906203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d80b3a10 0x7fb7d81460e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00c5c0 tx=0x7fb7cc007f50 comp rx=0 tx=0).stop 2024-01-26T23:29:43.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.601+0000 7fb7de2d8700 1 -- 172.21.15.46:0/3666906203 shutdown_connections 2024-01-26T23:29:43.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.601+0000 7fb7de2d8700 1 --2- 172.21.15.46:0/3666906203 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb7c4066f30 0x7fb7c40693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:43.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.601+0000 7fb7de2d8700 1 --2- 172.21.15.46:0/3666906203 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7d8159750 0x7fb7d815acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:43.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.601+0000 7fb7de2d8700 1 --2- 172.21.15.46:0/3666906203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7d8069cd0 0x7fb7d816a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:43.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.601+0000 7fb7de2d8700 1 --2- 172.21.15.46:0/3666906203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d80b3a10 0x7fb7d81460e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:43.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.601+0000 7fb7de2d8700 1 -- 172.21.15.46:0/3666906203 >> 172.21.15.46:0/3666906203 conn(0x7fb7d8067930 msgr2=0x7fb7d81543b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:43.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.602+0000 7fb7de2d8700 1 -- 172.21.15.46:0/3666906203 shutdown_connections 2024-01-26T23:29:43.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.604+0000 7fb7de2d8700 1 -- 172.21.15.46:0/3666906203 wait complete. 2024-01-26T23:29:43.608 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-01-26T23:29:43.629 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-26T23:29:43.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.765+0000 7fc4c31d9700 1 Processor -- start 2024-01-26T23:29:43.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.765+0000 7fc4c31d9700 1 -- start start 2024-01-26T23:29:43.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.766+0000 7fc4c31d9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc4bc157490 0x7fc4bc0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:43.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.766+0000 7fc4c31d9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4bc157860 0x7fc4bc0cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:43.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.766+0000 7fc4c31d9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc4bc06d6a0 0x7fc4bc06dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:43.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.766+0000 7fc4c31d9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4bc0cdc60 con 0x7fc4bc157490 2024-01-26T23:29:43.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.766+0000 7fc4c31d9700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4bc0cea80 con 0x7fc4bc06d6a0 2024-01-26T23:29:43.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.766+0000 7fc4c31d9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc06dff0 con 0x7fc4bc157860 2024-01-26T23:29:43.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.766+0000 7fc4c0f75700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc4bc157490 0x7fc4bc0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:43.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.766+0000 7fc4c0f75700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc4bc157490 0x7fc4bc0cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37214/0 (socket says 172.21.15.46:37214) 2024-01-26T23:29:43.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.766+0000 7fc4c0f75700 1 -- 172.21.15.46:0/2508966195 learned_addr learned my addr 172.21.15.46:0/2508966195 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:43.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.766+0000 7fc4bbfff700 1 --2- 172.21.15.46:0/2508966195 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4bc157860 0x7fc4bc0cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:43.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.766+0000 7fc4c1776700 1 --2- 172.21.15.46:0/2508966195 >> v2:172.21.15.106:3300/0 conn(0x7fc4bc06d6a0 0x7fc4bc06dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:43.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.766+0000 7fc4c0f75700 1 -- 172.21.15.46:0/2508966195 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4bc157860 msgr2=0x7fc4bc0cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:43.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.766+0000 7fc4c0f75700 1 --2- 172.21.15.46:0/2508966195 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4bc157860 0x7fc4bc0cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.767+0000 7fc4c0f75700 1 -- 172.21.15.46:0/2508966195 >> v2:172.21.15.106:3300/0 conn(0x7fc4bc06d6a0 msgr2=0x7fc4bc06dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.767+0000 7fc4c0f75700 1 --2- 172.21.15.46:0/2508966195 >> v2:172.21.15.106:3300/0 conn(0x7fc4bc06d6a0 0x7fc4bc06dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.767+0000 7fc4c0f75700 1 -- 172.21.15.46:0/2508966195 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc0cde70 con 0x7fc4bc157490 2024-01-26T23:29:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.767+0000 7fc4c1776700 1 --2- 172.21.15.46:0/2508966195 >> v2:172.21.15.106:3300/0 conn(0x7fc4bc06d6a0 0x7fc4bc06dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:29:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.767+0000 7fc4c0f75700 1 --2- 172.21.15.46:0/2508966195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4bc157490 0x7fc4bc0cc600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc06f970 tx=0x7fc4b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e790709fdd2ed server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.767+0000 7fc4bb7fe700 1 -- 172.21.15.46:0/2508966195 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b000b860 con 0x7fc4bc157490 2024-01-26T23:29:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.767+0000 7fc4bb7fe700 1 -- 172.21.15.46:0/2508966195 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b000b9c0 con 0x7fc4bc157490 2024-01-26T23:29:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.767+0000 7fc4bb7fe700 1 -- 172.21.15.46:0/2508966195 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b000bd60 con 0x7fc4bc157490 2024-01-26T23:29:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.767+0000 7fc4c31d9700 1 -- 172.21.15.46:0/2508966195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4bc157490 msgr2=0x7fc4bc0cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.767+0000 7fc4c31d9700 1 --2- 172.21.15.46:0/2508966195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4bc157490 0x7fc4bc0cc600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc06f970 tx=0x7fc4b0009250 comp rx=0 tx=0).stop 2024-01-26T23:29:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.767+0000 7fc4c31d9700 1 -- 172.21.15.46:0/2508966195 shutdown_connections 2024-01-26T23:29:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.767+0000 7fc4c31d9700 1 --2- 172.21.15.46:0/2508966195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4bc157490 0x7fc4bc0cc600 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.767+0000 7fc4c31d9700 1 --2- 172.21.15.46:0/2508966195 >> v2:172.21.15.106:3300/0 conn(0x7fc4bc06d6a0 0x7fc4bc06dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.767+0000 7fc4c31d9700 1 --2- 172.21.15.46:0/2508966195 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4bc157860 0x7fc4bc0cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:43.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.767+0000 7fc4c31d9700 1 -- 172.21.15.46:0/2508966195 >> 172.21.15.46:0/2508966195 conn(0x7fc4bc067930 msgr2=0x7fc4bc067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:43.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.768+0000 7fc4c31d9700 1 -- 172.21.15.46:0/2508966195 shutdown_connections 2024-01-26T23:29:43.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.768+0000 7fc4c31d9700 1 -- 172.21.15.46:0/2508966195 wait complete. 2024-01-26T23:29:43.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.769+0000 7fc4c31d9700 1 Processor -- start 2024-01-26T23:29:43.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.769+0000 7fc4c31d9700 1 -- start start 2024-01-26T23:29:43.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4c31d9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4bc06d6a0 0x7fc4bc169960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:43.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4c31d9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4bc157490 0x7fc4bc151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:43.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4c31d9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4bc157860 0x7fc4bc16ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:43.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4c31d9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc1512b0 con 0x7fc4bc157860 2024-01-26T23:29:43.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4c31d9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc150fa0 con 0x7fc4bc157490 2024-01-26T23:29:43.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4c31d9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc169ea0 con 0x7fc4bc06d6a0 2024-01-26T23:29:43.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4c1776700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4bc157860 0x7fc4bc16ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:43.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4c1776700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4bc157860 0x7fc4bc16ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37222/0 (socket says 172.21.15.46:37222) 2024-01-26T23:29:43.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4c1776700 1 -- 172.21.15.46:0/3994715390 learned_addr learned my addr 172.21.15.46:0/3994715390 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:43.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4c0f75700 1 --2- 172.21.15.46:0/3994715390 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4bc06d6a0 0x7fc4bc169960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:43.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4bbfff700 1 --2- 172.21.15.46:0/3994715390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4bc157490 0x7fc4bc151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:43.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4c1776700 1 -- 172.21.15.46:0/3994715390 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4bc06d6a0 msgr2=0x7fc4bc169960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:43.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4c1776700 1 --2- 172.21.15.46:0/3994715390 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4bc06d6a0 0x7fc4bc169960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:43.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4c1776700 1 -- 172.21.15.46:0/3994715390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4bc157490 msgr2=0x7fc4bc151740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:43.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4c1776700 1 --2- 172.21.15.46:0/3994715390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4bc157490 0x7fc4bc151740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:43.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4c1776700 1 -- 172.21.15.46:0/3994715390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b00094f0 con 0x7fc4bc157860 2024-01-26T23:29:43.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4c1776700 1 --2- 172.21.15.46:0/3994715390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4bc157860 0x7fc4bc16ab40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400d180 tx=0x7fc4b4009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:43.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4b8ff9700 1 -- 172.21.15.46:0/3994715390 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b400dcc0 con 0x7fc4bc157860 2024-01-26T23:29:43.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4b8ff9700 1 -- 172.21.15.46:0/3994715390 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b400f040 con 0x7fc4bc157860 2024-01-26T23:29:43.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4b8ff9700 1 -- 172.21.15.46:0/3994715390 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4015690 con 0x7fc4bc157860 2024-01-26T23:29:43.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.770+0000 7fc4c31d9700 1 -- 172.21.15.46:0/3994715390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4bc16a150 con 0x7fc4bc157860 2024-01-26T23:29:43.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.771+0000 7fc4c31d9700 1 -- 172.21.15.46:0/3994715390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4bc16a710 con 0x7fc4bc157860 2024-01-26T23:29:43.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.772+0000 7fc4c31d9700 1 -- 172.21.15.46:0/3994715390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4bc16a930 con 0x7fc4bc157860 2024-01-26T23:29:43.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.772+0000 7fc4b8ff9700 1 -- 172.21.15.46:0/3994715390 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc4b401e3b0 con 0x7fc4bc157860 2024-01-26T23:29:43.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.772+0000 7fc4b8ff9700 1 --2- 172.21.15.46:0/3994715390 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc4a0066f30 0x7fc4a00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:43.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.772+0000 7fc4b8ff9700 1 -- 172.21.15.46:0/3994715390 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5605+0+0 (secure 0 0 0) 0x7fc4b40849e0 con 0x7fc4bc157860 2024-01-26T23:29:43.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.773+0000 7fc4c0f75700 1 --2- 172.21.15.46:0/3994715390 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc4a0066f30 0x7fc4a00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:43.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.773+0000 7fc4c0f75700 1 --2- 172.21.15.46:0/3994715390 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc4a0066f30 0x7fc4a00693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc06f970 tx=0x7fc4b0017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:43.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:43.777+0000 7fc4b8ff9700 1 -- 172.21.15.46:0/3994715390 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc4b4051cb0 con 0x7fc4bc157860 2024-01-26T23:29:44.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.029+0000 7fc4c31d9700 1 -- 172.21.15.46:0/3994715390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc4bc04c710 con 0x7fc4bc157860 2024-01-26T23:29:44.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.604+0000 7fc4b8ff9700 1 -- 172.21.15.46:0/3994715390 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fc4b401a0c0 con 0x7fc4bc157860 2024-01-26T23:29:44.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.618+0000 7fc4c31d9700 1 -- 172.21.15.46:0/3994715390 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc4a0066f30 msgr2=0x7fc4a00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:44.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.618+0000 7fc4c31d9700 1 --2- 172.21.15.46:0/3994715390 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc4a0066f30 0x7fc4a00693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc06f970 tx=0x7fc4b0017000 comp rx=0 tx=0).stop 2024-01-26T23:29:44.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.618+0000 7fc4c31d9700 1 -- 172.21.15.46:0/3994715390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4bc157860 msgr2=0x7fc4bc16ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:44.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.618+0000 7fc4c31d9700 1 --2- 172.21.15.46:0/3994715390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4bc157860 0x7fc4bc16ab40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400d180 tx=0x7fc4b4009a50 comp rx=0 tx=0).stop 2024-01-26T23:29:44.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.626+0000 7fc4c31d9700 1 -- 172.21.15.46:0/3994715390 shutdown_connections 2024-01-26T23:29:44.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.626+0000 7fc4c31d9700 1 --2- 172.21.15.46:0/3994715390 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc4a0066f30 0x7fc4a00693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:44.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.626+0000 7fc4c31d9700 1 --2- 172.21.15.46:0/3994715390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4bc157860 0x7fc4bc16ab40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:44.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.626+0000 7fc4c31d9700 1 --2- 172.21.15.46:0/3994715390 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4bc06d6a0 0x7fc4bc169960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:44.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.626+0000 7fc4c31d9700 1 --2- 172.21.15.46:0/3994715390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4bc157490 0x7fc4bc151740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:44.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.626+0000 7fc4c31d9700 1 -- 172.21.15.46:0/3994715390 >> 172.21.15.46:0/3994715390 conn(0x7fc4bc067930 msgr2=0x7fc4bc069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:44.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.627+0000 7fc4c31d9700 1 -- 172.21.15.46:0/3994715390 shutdown_connections 2024-01-26T23:29:44.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.627+0000 7fc4c31d9700 1 -- 172.21.15.46:0/3994715390 wait complete. 2024-01-26T23:29:44.630 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 2 and data pool 3 2024-01-26T23:29:44.646 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-01-26T23:29:44.646 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-01-26T23:29:44.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.783+0000 7fbd08bb9700 1 Processor -- start 2024-01-26T23:29:44.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.783+0000 7fbd08bb9700 1 -- start start 2024-01-26T23:29:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.784+0000 7fbd08bb9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd04159630 0x7fbd04159a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.784+0000 7fbd08bb9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd040af570 0x7fbd040b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.784+0000 7fbd08bb9700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0415ab40 con 0x7fbd04067930 2024-01-26T23:29:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.784+0000 7fbd08bb9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd04159f40 con 0x7fbd04159630 2024-01-26T23:29:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.784+0000 7fbd08bb9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0415a210 con 0x7fbd040af570 2024-01-26T23:29:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.784+0000 7fbd0259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd04159630 0x7fbd04159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.785+0000 7fbd0259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd04159630 0x7fbd04159a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37236/0 (socket says 172.21.15.46:37236) 2024-01-26T23:29:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.785+0000 7fbd0259c700 1 -- 172.21.15.46:0/101198577 learned_addr learned my addr 172.21.15.46:0/101198577 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.785+0000 7fbd01d9b700 1 --2- 172.21.15.46:0/101198577 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd040af570 0x7fbd040b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.785+0000 7fbd0259c700 1 -- 172.21.15.46:0/101198577 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd040af570 msgr2=0x7fbd040b2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.785+0000 7fbd0259c700 1 --2- 172.21.15.46:0/101198577 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd040af570 0x7fbd040b2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.785+0000 7fbd0259c700 1 -- 172.21.15.46:0/101198577 >> v1:172.21.15.106:6789/0 conn(0x7fbd04067930 legacy=0x7fbd040cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.785+0000 7fbd0159a700 1 -- 172.21.15.46:0/101198577 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936408782 0 0) 0x7fbd0415ab40 con 0x7fbd04067930 2024-01-26T23:29:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.785+0000 7fbd0259c700 1 -- 172.21.15.46:0/101198577 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0415a4f0 con 0x7fbd04159630 2024-01-26T23:29:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.785+0000 7fbd0259c700 1 --2- 172.21.15.46:0/101198577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04159630 0x7fbd04159a00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80099b0 tx=0x7fbcf8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e412fba024435855 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.785+0000 7fbd0159a700 1 -- 172.21.15.46:0/101198577 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8009ea0 con 0x7fbd04159630 2024-01-26T23:29:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.785+0000 7fbd0159a700 1 -- 172.21.15.46:0/101198577 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcf800b860 con 0x7fbd04159630 2024-01-26T23:29:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.785+0000 7fbd0159a700 1 -- 172.21.15.46:0/101198577 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf800bc00 con 0x7fbd04159630 2024-01-26T23:29:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.786+0000 7fbd08bb9700 1 -- 172.21.15.46:0/101198577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04159630 msgr2=0x7fbd04159a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.786+0000 7fbd08bb9700 1 --2- 172.21.15.46:0/101198577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04159630 0x7fbd04159a00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80099b0 tx=0x7fbcf8009250 comp rx=0 tx=0).stop 2024-01-26T23:29:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.786+0000 7fbd08bb9700 1 -- 172.21.15.46:0/101198577 shutdown_connections 2024-01-26T23:29:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.786+0000 7fbd08bb9700 1 --2- 172.21.15.46:0/101198577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04159630 0x7fbd04159a00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.786+0000 7fbd08bb9700 1 --2- 172.21.15.46:0/101198577 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd040af570 0x7fbd040b2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.786+0000 7fbd08bb9700 1 -- 172.21.15.46:0/101198577 >> 172.21.15.46:0/101198577 conn(0x7fbd04069510 msgr2=0x7fbd040aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.786+0000 7fbd08bb9700 1 -- 172.21.15.46:0/101198577 shutdown_connections 2024-01-26T23:29:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.786+0000 7fbd08bb9700 1 -- 172.21.15.46:0/101198577 wait complete. 2024-01-26T23:29:44.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.787+0000 7fbd08bb9700 1 Processor -- start 2024-01-26T23:29:44.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.787+0000 7fbd08bb9700 1 -- start start 2024-01-26T23:29:44.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.788+0000 7fbd08bb9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd04067930 0x7fbd0415b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:44.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.788+0000 7fbd08bb9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd040af570 0x7fbd04158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:44.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.788+0000 7fbd08bb9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04159630 0x7fbd0413d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:44.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.788+0000 7fbd08bb9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd04158fc0 con 0x7fbd04159630 2024-01-26T23:29:44.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.788+0000 7fbd08bb9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0415beb0 con 0x7fbd040af570 2024-01-26T23:29:44.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.788+0000 7fbd08bb9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd040ab700 con 0x7fbd04067930 2024-01-26T23:29:44.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.788+0000 7fbd02d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04159630 0x7fbd0413d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:44.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.788+0000 7fbd02d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04159630 0x7fbd0413d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37254/0 (socket says 172.21.15.46:37254) 2024-01-26T23:29:44.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.788+0000 7fbd02d9d700 1 -- 172.21.15.46:0/3868239474 learned_addr learned my addr 172.21.15.46:0/3868239474 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:44.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.788+0000 7fbd01d9b700 1 --2- 172.21.15.46:0/3868239474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd040af570 0x7fbd04158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:44.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.788+0000 7fbd0259c700 1 --2- 172.21.15.46:0/3868239474 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd04067930 0x7fbd0415b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:44.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.788+0000 7fbd02d9d700 1 -- 172.21.15.46:0/3868239474 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd04067930 msgr2=0x7fbd0415b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:44.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.788+0000 7fbd02d9d700 1 --2- 172.21.15.46:0/3868239474 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd04067930 0x7fbd0415b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:44.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.788+0000 7fbd02d9d700 1 -- 172.21.15.46:0/3868239474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd040af570 msgr2=0x7fbd04158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:44.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.788+0000 7fbd02d9d700 1 --2- 172.21.15.46:0/3868239474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd040af570 0x7fbd04158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:44.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.788+0000 7fbd02d9d700 1 -- 172.21.15.46:0/3868239474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf80094f0 con 0x7fbd04159630 2024-01-26T23:29:44.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.789+0000 7fbd02d9d700 1 --2- 172.21.15.46:0/3868239474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04159630 0x7fbd0413d7f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00cb60 tx=0x7fbcfc001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:44.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.789+0000 7fbcf27fc700 1 -- 172.21.15.46:0/3868239474 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc00a770 con 0x7fbd04159630 2024-01-26T23:29:44.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.789+0000 7fbcf27fc700 1 -- 172.21.15.46:0/3868239474 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcfc00a8d0 con 0x7fbd04159630 2024-01-26T23:29:44.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.789+0000 7fbd08bb9700 1 -- 172.21.15.46:0/3868239474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd040aadd0 con 0x7fbd04159630 2024-01-26T23:29:44.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.789+0000 7fbcf27fc700 1 -- 172.21.15.46:0/3868239474 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc00ac90 con 0x7fbd04159630 2024-01-26T23:29:44.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.789+0000 7fbd08bb9700 1 -- 172.21.15.46:0/3868239474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd040ab390 con 0x7fbd04159630 2024-01-26T23:29:44.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.790+0000 7fbd08bb9700 1 -- 172.21.15.46:0/3868239474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd0404c710 con 0x7fbd04159630 2024-01-26T23:29:44.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.790+0000 7fbcf27fc700 1 -- 172.21.15.46:0/3868239474 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbcfc012630 con 0x7fbd04159630 2024-01-26T23:29:44.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.791+0000 7fbcf27fc700 1 --2- 172.21.15.46:0/3868239474 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbce4066f30 0x7fbce40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:44.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.791+0000 7fbcf27fc700 1 -- 172.21.15.46:0/3868239474 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fbcfc084950 con 0x7fbd04159630 2024-01-26T23:29:44.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.791+0000 7fbd0259c700 1 --2- 172.21.15.46:0/3868239474 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbce4066f30 0x7fbce40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:44.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.791+0000 7fbd0259c700 1 --2- 172.21.15.46:0/3868239474 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbce4066f30 0x7fbce40693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80099b0 tx=0x7fbcf8017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:44.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:44.795+0000 7fbcf27fc700 1 -- 172.21.15.46:0/3868239474 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbcfc051bd0 con 0x7fbd04159630 2024-01-26T23:29:45.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.013+0000 7fbd08bb9700 1 -- 172.21.15.46:0/3868239474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fbd040ab840 con 0x7fbd04159630 2024-01-26T23:29:45.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.607+0000 7fbcf27fc700 1 -- 172.21.15.46:0/3868239474 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) v1 ==== 157+0+0 (secure 0 0 0) 0x7fbcfc0551f0 con 0x7fbd04159630 2024-01-26T23:29:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.611+0000 7fbd08bb9700 1 -- 172.21.15.46:0/3868239474 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbce4066f30 msgr2=0x7fbce40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.611+0000 7fbd08bb9700 1 --2- 172.21.15.46:0/3868239474 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbce4066f30 0x7fbce40693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80099b0 tx=0x7fbcf8017000 comp rx=0 tx=0).stop 2024-01-26T23:29:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.611+0000 7fbd08bb9700 1 -- 172.21.15.46:0/3868239474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04159630 msgr2=0x7fbd0413d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.611+0000 7fbd08bb9700 1 --2- 172.21.15.46:0/3868239474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04159630 0x7fbd0413d7f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00cb60 tx=0x7fbcfc001770 comp rx=0 tx=0).stop 2024-01-26T23:29:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.611+0000 7fbd08bb9700 1 -- 172.21.15.46:0/3868239474 shutdown_connections 2024-01-26T23:29:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.611+0000 7fbd08bb9700 1 --2- 172.21.15.46:0/3868239474 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbce4066f30 0x7fbce40693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.611+0000 7fbd08bb9700 1 --2- 172.21.15.46:0/3868239474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04159630 0x7fbd0413d7f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.611+0000 7fbd08bb9700 1 --2- 172.21.15.46:0/3868239474 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd04067930 0x7fbd0415b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.611+0000 7fbd08bb9700 1 --2- 172.21.15.46:0/3868239474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd040af570 0x7fbd04158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.611+0000 7fbd08bb9700 1 -- 172.21.15.46:0/3868239474 >> 172.21.15.46:0/3868239474 conn(0x7fbd04069510 msgr2=0x7fbd0406abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.611+0000 7fbd08bb9700 1 -- 172.21.15.46:0/3868239474 shutdown_connections 2024-01-26T23:29:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.611+0000 7fbd08bb9700 1 -- 172.21.15.46:0/3868239474 wait complete. 2024-01-26T23:29:45.628 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-01-26T23:29:45.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.770+0000 7f73adb69700 1 Processor -- start 2024-01-26T23:29:45.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.770+0000 7f73adb69700 1 -- start start 2024-01-26T23:29:45.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.771+0000 7f73adb69700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f73a80ccb20 0x7f73a8157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:45.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.771+0000 7f73adb69700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73a806beb0 0x7f73a806c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:45.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.771+0000 7f73adb69700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a80cdbe0 con 0x7f73a81574c0 2024-01-26T23:29:45.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.771+0000 7f73adb69700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73a80b2880 con 0x7f73a806beb0 2024-01-26T23:29:45.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.771+0000 7f73adb69700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73a806c800 con 0x7f73a80ccb20 2024-01-26T23:29:45.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.771+0000 7f73a7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73a806beb0 0x7f73a806c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:45.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.771+0000 7f73a7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73a806beb0 0x7f73a806c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37260/0 (socket says 172.21.15.46:37260) 2024-01-26T23:29:45.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.772+0000 7f73a7fff700 1 -- 172.21.15.46:0/4060513738 learned_addr learned my addr 172.21.15.46:0/4060513738 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.772+0000 7f73a6ffd700 1 --2- 172.21.15.46:0/4060513738 >> v2:172.21.15.106:3300/0 conn(0x7f73a80ccb20 0x7f73a8157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.772+0000 7f73a7fff700 1 -- 172.21.15.46:0/4060513738 >> v2:172.21.15.106:3300/0 conn(0x7f73a80ccb20 msgr2=0x7f73a8157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.772+0000 7f73a7fff700 1 --2- 172.21.15.46:0/4060513738 >> v2:172.21.15.106:3300/0 conn(0x7f73a80ccb20 0x7f73a8157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.772+0000 7f73a7fff700 1 -- 172.21.15.46:0/4060513738 >> v1:172.21.15.106:6789/0 conn(0x7f73a81574c0 legacy=0x7f73a80cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.772+0000 7f73a7fff700 1 -- 172.21.15.46:0/4060513738 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a80ccfd0 con 0x7f73a806beb0 2024-01-26T23:29:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.772+0000 7f73a6ffd700 1 --2- 172.21.15.46:0/4060513738 >> v2:172.21.15.106:3300/0 conn(0x7f73a80ccb20 0x7f73a8157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:29:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.772+0000 7f73a7fff700 1 --2- 172.21.15.46:0/4060513738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73a806beb0 0x7f73a806c2c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f73a8064380 tx=0x7f73a0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c677f27ac06da5 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.772+0000 7f73a67fc700 1 -- 172.21.15.46:0/4060513738 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a000e070 con 0x7f73a806beb0 2024-01-26T23:29:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.772+0000 7f73a67fc700 1 -- 172.21.15.46:0/4060513738 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a000ba10 con 0x7f73a806beb0 2024-01-26T23:29:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.772+0000 7f73a67fc700 1 -- 172.21.15.46:0/4060513738 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a00092e0 con 0x7f73a806beb0 2024-01-26T23:29:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.773+0000 7f73adb69700 1 -- 172.21.15.46:0/4060513738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73a806beb0 msgr2=0x7f73a806c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.773+0000 7f73adb69700 1 --2- 172.21.15.46:0/4060513738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73a806beb0 0x7f73a806c2c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f73a8064380 tx=0x7f73a0009840 comp rx=0 tx=0).stop 2024-01-26T23:29:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.773+0000 7f73adb69700 1 -- 172.21.15.46:0/4060513738 shutdown_connections 2024-01-26T23:29:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.773+0000 7f73adb69700 1 --2- 172.21.15.46:0/4060513738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73a806beb0 0x7f73a806c2c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:45.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.773+0000 7f73adb69700 1 --2- 172.21.15.46:0/4060513738 >> v2:172.21.15.106:3300/0 conn(0x7f73a80ccb20 0x7f73a8157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:45.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.773+0000 7f73adb69700 1 -- 172.21.15.46:0/4060513738 >> 172.21.15.46:0/4060513738 conn(0x7f73a80ce5f0 msgr2=0x7f73a80ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:45.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.773+0000 7f73adb69700 1 -- 172.21.15.46:0/4060513738 shutdown_connections 2024-01-26T23:29:45.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.773+0000 7f73adb69700 1 -- 172.21.15.46:0/4060513738 wait complete. 2024-01-26T23:29:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.774+0000 7f73adb69700 1 Processor -- start 2024-01-26T23:29:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.774+0000 7f73adb69700 1 -- start start 2024-01-26T23:29:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.775+0000 7f73adb69700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73a806beb0 0x7f73a8150d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.775+0000 7f73adb69700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73a80ccb20 0x7f73a80b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.775+0000 7f73adb69700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73a81574c0 0x7f73a816a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.775+0000 7f73adb69700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a8158f20 con 0x7f73a81574c0 2024-01-26T23:29:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.775+0000 7f73adb69700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a81518b0 con 0x7f73a80ccb20 2024-01-26T23:29:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.775+0000 7f73adb69700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a81589a0 con 0x7f73a806beb0 2024-01-26T23:29:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.775+0000 7f73a7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73a81574c0 0x7f73a816a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.775+0000 7f73a7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73a81574c0 0x7f73a816a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37276/0 (socket says 172.21.15.46:37276) 2024-01-26T23:29:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.775+0000 7f73a7fff700 1 -- 172.21.15.46:0/1453657735 learned_addr learned my addr 172.21.15.46:0/1453657735 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.775+0000 7f73a77fe700 1 --2- 172.21.15.46:0/1453657735 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73a806beb0 0x7f73a8150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.775+0000 7f73a6ffd700 1 --2- 172.21.15.46:0/1453657735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73a80ccb20 0x7f73a80b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.775+0000 7f73a7fff700 1 -- 172.21.15.46:0/1453657735 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73a806beb0 msgr2=0x7f73a8150d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.775+0000 7f73a7fff700 1 --2- 172.21.15.46:0/1453657735 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73a806beb0 0x7f73a8150d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.775+0000 7f73a7fff700 1 -- 172.21.15.46:0/1453657735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73a80ccb20 msgr2=0x7f73a80b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.775+0000 7f73a7fff700 1 --2- 172.21.15.46:0/1453657735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73a80ccb20 0x7f73a80b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.775+0000 7f73a7fff700 1 -- 172.21.15.46:0/1453657735 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a0009ae0 con 0x7f73a81574c0 2024-01-26T23:29:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.775+0000 7f73a6ffd700 1 --2- 172.21.15.46:0/1453657735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73a80ccb20 0x7f73a80b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:29:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.776+0000 7f73a7fff700 1 --2- 172.21.15.46:0/1453657735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73a81574c0 0x7f73a816a800 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f73a8064c90 tx=0x7f73a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.776+0000 7f738f7fe700 1 -- 172.21.15.46:0/1453657735 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a00094e0 con 0x7f73a81574c0 2024-01-26T23:29:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.776+0000 7f738f7fe700 1 -- 172.21.15.46:0/1453657735 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a00032f0 con 0x7f73a81574c0 2024-01-26T23:29:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.776+0000 7f73adb69700 1 -- 172.21.15.46:0/1453657735 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a8158c40 con 0x7f73a81574c0 2024-01-26T23:29:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.776+0000 7f738f7fe700 1 -- 172.21.15.46:0/1453657735 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a001d420 con 0x7f73a81574c0 2024-01-26T23:29:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.776+0000 7f73adb69700 1 -- 172.21.15.46:0/1453657735 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a80b06e0 con 0x7f73a81574c0 2024-01-26T23:29:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.777+0000 7f73adb69700 1 -- 172.21.15.46:0/1453657735 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a81513a0 con 0x7f73a81574c0 2024-01-26T23:29:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.777+0000 7f738f7fe700 1 -- 172.21.15.46:0/1453657735 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f73a001d810 con 0x7f73a81574c0 2024-01-26T23:29:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.778+0000 7f738f7fe700 1 --2- 172.21.15.46:0/1453657735 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7394066f30 0x7f73940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.778+0000 7f738f7fe700 1 -- 172.21.15.46:0/1453657735 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f73a0085dc0 con 0x7f73a81574c0 2024-01-26T23:29:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.778+0000 7f73a77fe700 1 --2- 172.21.15.46:0/1453657735 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7394066f30 0x7f73940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.779+0000 7f73a77fe700 1 --2- 172.21.15.46:0/1453657735 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7394066f30 0x7f73940693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7398009af0 tx=0x7f7398009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.782+0000 7f738f7fe700 1 -- 172.21.15.46:0/1453657735 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f73a81513a0 con 0x7f73a81574c0 2024-01-26T23:29:45.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:45.984+0000 7f73adb69700 1 -- 172.21.15.46:0/1453657735 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f73a81513a0 con 0x7f73a81574c0 2024-01-26T23:29:47.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.617+0000 7f738f7fe700 1 -- 172.21.15.46:0/1453657735 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) v1 ==== 228+0+0 (secure 0 0 0) 0x7f73a81513a0 con 0x7f73a81574c0 2024-01-26T23:29:47.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.624+0000 7f73adb69700 1 -- 172.21.15.46:0/1453657735 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7394066f30 msgr2=0x7f73940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:47.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.624+0000 7f73adb69700 1 --2- 172.21.15.46:0/1453657735 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7394066f30 0x7f73940693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7398009af0 tx=0x7f7398009060 comp rx=0 tx=0).stop 2024-01-26T23:29:47.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.624+0000 7f73adb69700 1 -- 172.21.15.46:0/1453657735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73a81574c0 msgr2=0x7f73a816a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:47.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.624+0000 7f73adb69700 1 --2- 172.21.15.46:0/1453657735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73a81574c0 0x7f73a816a800 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f73a8064c90 tx=0x7f73a0017000 comp rx=0 tx=0).stop 2024-01-26T23:29:47.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.625+0000 7f73adb69700 1 -- 172.21.15.46:0/1453657735 shutdown_connections 2024-01-26T23:29:47.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.625+0000 7f73adb69700 1 --2- 172.21.15.46:0/1453657735 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7394066f30 0x7f73940693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:47.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.625+0000 7f73adb69700 1 --2- 172.21.15.46:0/1453657735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73a81574c0 0x7f73a816a800 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:47.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.625+0000 7f73adb69700 1 --2- 172.21.15.46:0/1453657735 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73a806beb0 0x7f73a8150d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:47.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.625+0000 7f73adb69700 1 --2- 172.21.15.46:0/1453657735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73a80ccb20 0x7f73a80b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:47.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.625+0000 7f73adb69700 1 -- 172.21.15.46:0/1453657735 >> 172.21.15.46:0/1453657735 conn(0x7f73a80ce5f0 msgr2=0x7f73a81550e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:47.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.628+0000 7f73adb69700 1 -- 172.21.15.46:0/1453657735 shutdown_connections 2024-01-26T23:29:47.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.628+0000 7f73adb69700 1 -- 172.21.15.46:0/1453657735 wait complete. 2024-01-26T23:29:47.631 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data_ec' created 2024-01-26T23:29:47.645 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-01-26T23:29:47.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.783+0000 7f3adde6d700 1 Processor -- start 2024-01-26T23:29:47.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.783+0000 7f3adde6d700 1 -- start start 2024-01-26T23:29:47.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.783+0000 7f3adde6d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3ad80ca0e0 0x7f3ad80ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:47.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.783+0000 7f3adde6d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ad806c0b0 0x7f3ad80afe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:47.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.783+0000 7f3adde6d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad8158010 con 0x7f3ad806c520 2024-01-26T23:29:47.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.783+0000 7f3adde6d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ad80cada0 con 0x7f3ad806c0b0 2024-01-26T23:29:47.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.783+0000 7f3adde6d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ad81582f0 con 0x7f3ad80ca0e0 2024-01-26T23:29:47.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.784+0000 7f3ad6ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ad806c0b0 0x7f3ad80afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:47.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.784+0000 7f3ad6ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ad806c0b0 0x7f3ad80afe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37278/0 (socket says 172.21.15.46:37278) 2024-01-26T23:29:47.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.784+0000 7f3ad6ffd700 1 -- 172.21.15.46:0/530414068 learned_addr learned my addr 172.21.15.46:0/530414068 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:47.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.784+0000 7f3ad77fe700 1 --2- 172.21.15.46:0/530414068 >> v2:172.21.15.106:3300/0 conn(0x7f3ad80ca0e0 0x7f3ad80ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:47.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.784+0000 7f3ad6ffd700 1 -- 172.21.15.46:0/530414068 >> v2:172.21.15.106:3300/0 conn(0x7f3ad80ca0e0 msgr2=0x7f3ad80ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:47.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.784+0000 7f3ad6ffd700 1 --2- 172.21.15.46:0/530414068 >> v2:172.21.15.106:3300/0 conn(0x7f3ad80ca0e0 0x7f3ad80ca4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:47.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.784+0000 7f3ad6ffd700 1 -- 172.21.15.46:0/530414068 >> v1:172.21.15.106:6789/0 conn(0x7f3ad806c520 legacy=0x7f3ad80b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:47.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.784+0000 7f3ad6ffd700 1 -- 172.21.15.46:0/530414068 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad81585d0 con 0x7f3ad806c0b0 2024-01-26T23:29:47.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.784+0000 7f3ad77fe700 1 --2- 172.21.15.46:0/530414068 >> v2:172.21.15.106:3300/0 conn(0x7f3ad80ca0e0 0x7f3ad80ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:29:47.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.784+0000 7f3ad6ffd700 1 --2- 172.21.15.46:0/530414068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad806c0b0 0x7f3ad80afe40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3ad806b850 tx=0x7f3acc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa2a1208b76e3ed server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:47.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.784+0000 7f3ad67fc700 1 -- 172.21.15.46:0/530414068 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3acc00b860 con 0x7f3ad806c0b0 2024-01-26T23:29:47.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.784+0000 7f3ad67fc700 1 -- 172.21.15.46:0/530414068 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3acc00b9c0 con 0x7f3ad806c0b0 2024-01-26T23:29:47.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.784+0000 7f3ad67fc700 1 -- 172.21.15.46:0/530414068 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3acc00bd60 con 0x7f3ad806c0b0 2024-01-26T23:29:47.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.785+0000 7f3adde6d700 1 -- 172.21.15.46:0/530414068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad806c0b0 msgr2=0x7f3ad80afe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:47.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.785+0000 7f3adde6d700 1 --2- 172.21.15.46:0/530414068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad806c0b0 0x7f3ad80afe40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3ad806b850 tx=0x7f3acc009250 comp rx=0 tx=0).stop 2024-01-26T23:29:47.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.785+0000 7f3adde6d700 1 -- 172.21.15.46:0/530414068 shutdown_connections 2024-01-26T23:29:47.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.785+0000 7f3adde6d700 1 --2- 172.21.15.46:0/530414068 >> v2:172.21.15.106:3300/0 conn(0x7f3ad80ca0e0 0x7f3ad80ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:47.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.785+0000 7f3adde6d700 1 --2- 172.21.15.46:0/530414068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad806c0b0 0x7f3ad80afe40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:47.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.785+0000 7f3adde6d700 1 -- 172.21.15.46:0/530414068 >> 172.21.15.46:0/530414068 conn(0x7f3ad80ce530 msgr2=0x7f3ad80ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:47.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.785+0000 7f3adde6d700 1 -- 172.21.15.46:0/530414068 shutdown_connections 2024-01-26T23:29:47.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.785+0000 7f3adde6d700 1 -- 172.21.15.46:0/530414068 wait complete. 2024-01-26T23:29:47.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.787+0000 7f3adde6d700 1 Processor -- start 2024-01-26T23:29:47.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.787+0000 7f3adde6d700 1 -- start start 2024-01-26T23:29:47.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.788+0000 7f3adde6d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ad806c0b0 0x7f3ad81500f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:47.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.788+0000 7f3adde6d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad806c520 0x7f3ad8068ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:47.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.788+0000 7f3adde6d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ad80ca0e0 0x7f3ad80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:47.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.788+0000 7f3adde6d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad8150630 con 0x7f3ad806c520 2024-01-26T23:29:47.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.788+0000 7f3adde6d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad806aab0 con 0x7f3ad806c0b0 2024-01-26T23:29:47.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.788+0000 7f3adde6d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad806a620 con 0x7f3ad80ca0e0 2024-01-26T23:29:47.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.788+0000 7f3ad6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad806c520 0x7f3ad8068ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:47.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.788+0000 7f3ad6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad806c520 0x7f3ad8068ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37288/0 (socket says 172.21.15.46:37288) 2024-01-26T23:29:47.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.788+0000 7f3ad6ffd700 1 -- 172.21.15.46:0/1646983342 learned_addr learned my addr 172.21.15.46:0/1646983342 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:47.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.788+0000 7f3ad77fe700 1 --2- 172.21.15.46:0/1646983342 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ad806c0b0 0x7f3ad81500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:47.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.788+0000 7f3ad7fff700 1 --2- 172.21.15.46:0/1646983342 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ad80ca0e0 0x7f3ad80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:47.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.788+0000 7f3ad6ffd700 1 -- 172.21.15.46:0/1646983342 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ad80ca0e0 msgr2=0x7f3ad80693e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:47.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.788+0000 7f3ad6ffd700 1 --2- 172.21.15.46:0/1646983342 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ad80ca0e0 0x7f3ad80693e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:47.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.788+0000 7f3ad6ffd700 1 -- 172.21.15.46:0/1646983342 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ad806c0b0 msgr2=0x7f3ad81500f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:47.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.788+0000 7f3ad6ffd700 1 --2- 172.21.15.46:0/1646983342 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ad806c0b0 0x7f3ad81500f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:47.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.788+0000 7f3ad6ffd700 1 -- 172.21.15.46:0/1646983342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc0094f0 con 0x7f3ad806c520 2024-01-26T23:29:47.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.789+0000 7f3ad6ffd700 1 --2- 172.21.15.46:0/1646983342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad806c520 0x7f3ad8068ea0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3acc009290 tx=0x7f3acc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:47.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.789+0000 7f3abf7fe700 1 -- 172.21.15.46:0/1646983342 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3acc026480 con 0x7f3ad806c520 2024-01-26T23:29:47.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.789+0000 7f3abf7fe700 1 -- 172.21.15.46:0/1646983342 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3acc01f070 con 0x7f3ad806c520 2024-01-26T23:29:47.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.789+0000 7f3adde6d700 1 -- 172.21.15.46:0/1646983342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad815bb80 con 0x7f3ad806c520 2024-01-26T23:29:47.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.789+0000 7f3abf7fe700 1 -- 172.21.15.46:0/1646983342 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3acc026850 con 0x7f3ad806c520 2024-01-26T23:29:47.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.789+0000 7f3adde6d700 1 -- 172.21.15.46:0/1646983342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad8069cf0 con 0x7f3ad806c520 2024-01-26T23:29:47.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.790+0000 7f3adde6d700 1 -- 172.21.15.46:0/1646983342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad806a7b0 con 0x7f3ad806c520 2024-01-26T23:29:47.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.790+0000 7f3abf7fe700 1 -- 172.21.15.46:0/1646983342 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3acc01a430 con 0x7f3ad806c520 2024-01-26T23:29:47.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.791+0000 7f3abf7fe700 1 --2- 172.21.15.46:0/1646983342 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3ac4066f30 0x7f3ac40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:47.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.791+0000 7f3abf7fe700 1 -- 172.21.15.46:0/1646983342 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6424+0+0 (secure 0 0 0) 0x7f3acc021070 con 0x7f3ad806c520 2024-01-26T23:29:47.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.791+0000 7f3ad77fe700 1 --2- 172.21.15.46:0/1646983342 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3ac4066f30 0x7f3ac40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:47.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.792+0000 7f3ad77fe700 1 --2- 172.21.15.46:0/1646983342 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3ac4066f30 0x7f3ac40693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8006f00 tx=0x7f3ac8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:47.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.796+0000 7f3abf7fe700 1 -- 172.21.15.46:0/1646983342 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3acc053580 con 0x7f3ad806c520 2024-01-26T23:29:48.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:47.998+0000 7f3adde6d700 1 -- 172.21.15.46:0/1646983342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f3ad804c710 con 0x7f3ad806c520 2024-01-26T23:29:48.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.625+0000 7f3abf7fe700 1 -- 172.21.15.46:0/1646983342 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3acc056ba0 con 0x7f3ad806c520 2024-01-26T23:29:48.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.638+0000 7f3adde6d700 1 -- 172.21.15.46:0/1646983342 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3ac4066f30 msgr2=0x7f3ac40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:48.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.638+0000 7f3adde6d700 1 --2- 172.21.15.46:0/1646983342 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3ac4066f30 0x7f3ac40693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8006f00 tx=0x7f3ac8008040 comp rx=0 tx=0).stop 2024-01-26T23:29:48.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.638+0000 7f3adde6d700 1 -- 172.21.15.46:0/1646983342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad806c520 msgr2=0x7f3ad8068ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:48.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.638+0000 7f3adde6d700 1 --2- 172.21.15.46:0/1646983342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad806c520 0x7f3ad8068ea0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3acc009290 tx=0x7f3acc017000 comp rx=0 tx=0).stop 2024-01-26T23:29:48.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.639+0000 7f3adde6d700 1 -- 172.21.15.46:0/1646983342 shutdown_connections 2024-01-26T23:29:48.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.639+0000 7f3adde6d700 1 --2- 172.21.15.46:0/1646983342 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3ac4066f30 0x7f3ac40693e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:48.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.639+0000 7f3adde6d700 1 --2- 172.21.15.46:0/1646983342 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ad80ca0e0 0x7f3ad80693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:48.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.639+0000 7f3adde6d700 1 --2- 172.21.15.46:0/1646983342 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ad806c0b0 0x7f3ad81500f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:48.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.639+0000 7f3adde6d700 1 --2- 172.21.15.46:0/1646983342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad806c520 0x7f3ad8068ea0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:48.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.639+0000 7f3adde6d700 1 -- 172.21.15.46:0/1646983342 >> 172.21.15.46:0/1646983342 conn(0x7f3ad80ce530 msgr2=0x7f3ad80cca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.641+0000 7f3adde6d700 1 -- 172.21.15.46:0/1646983342 shutdown_connections 2024-01-26T23:29:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.641+0000 7f3adde6d700 1 -- 172.21.15.46:0/1646983342 wait complete. 2024-01-26T23:29:48.650 INFO:teuthology.orchestra.run.smithi046.stderr:set pool 4 allow_ec_overwrites to true 2024-01-26T23:29:48.665 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-01-26T23:29:48.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.805+0000 7f421a0f0700 1 Processor -- start 2024-01-26T23:29:48.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.805+0000 7f421a0f0700 1 -- start start 2024-01-26T23:29:48.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.806+0000 7f421a0f0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4214141200 0x7f42140c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:48.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.806+0000 7f421a0f0700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42140ccc00 con 0x7f42140caf30 2024-01-26T23:29:48.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.806+0000 7f421a0f0700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42140cc310 con 0x7f42140c8da0 2024-01-26T23:29:48.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.806+0000 7f421a0f0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42140cbbf0 con 0x7f4214141200 2024-01-26T23:29:48.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.806+0000 7f4213fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f42140caf30 0x7f42140cb370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49634/0 (socket says 172.21.15.46:49634) 2024-01-26T23:29:48.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.806+0000 7f4213fff700 1 -- 172.21.15.46:0/2289138836 learned_addr learned my addr 172.21.15.46:0/2289138836 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:48.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.806+0000 7f4212ffd700 1 --2- 172.21.15.46:0/2289138836 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4214141200 0x7f42140c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:48.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.806+0000 7f42127fc700 1 -- 172.21.15.46:0/2289138836 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009587318 0 0) 0x7f42140ccc00 con 0x7f42140caf30 2024-01-26T23:29:48.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.806+0000 7f42127fc700 1 -- 172.21.15.46:0/2289138836 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41fc003570 con 0x7f42140caf30 2024-01-26T23:29:48.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.806+0000 7f42127fc700 1 -- 172.21.15.46:0/2289138836 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187514955 0 0) 0x7f42140cc310 con 0x7f42140c8da0 2024-01-26T23:29:48.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.806+0000 7f42127fc700 1 -- 172.21.15.46:0/2289138836 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42140ccc00 con 0x7f42140c8da0 2024-01-26T23:29:48.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.806+0000 7f4212ffd700 1 -- 172.21.15.46:0/2289138836 >> v1:172.21.15.106:6789/0 conn(0x7f42140c8da0 legacy=0x7f4214140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:48.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.806+0000 7f4212ffd700 1 -- 172.21.15.46:0/2289138836 >> v1:172.21.15.46:6789/0 conn(0x7f42140caf30 legacy=0x7f42140cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:48.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.806+0000 7f42127fc700 1 -- 172.21.15.46:0/2289138836 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3433845816 0 0) 0x7f41fc003570 con 0x7f42140caf30 2024-01-26T23:29:48.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.806+0000 7f4212ffd700 1 -- 172.21.15.46:0/2289138836 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42140cc5b0 con 0x7f4214141200 2024-01-26T23:29:48.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.807+0000 7f4212ffd700 1 --2- 172.21.15.46:0/2289138836 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4214141200 0x7f42140c9170 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4208009910 tx=0x7f42080091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a652d592773e6ba5 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:48.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.807+0000 7f42127fc700 1 -- 172.21.15.46:0/2289138836 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c0017f0 con 0x7f4214141200 2024-01-26T23:29:48.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.807+0000 7f42127fc700 1 -- 172.21.15.46:0/2289138836 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4208011420 con 0x7f4214141200 2024-01-26T23:29:48.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.807+0000 7f42127fc700 1 -- 172.21.15.46:0/2289138836 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42080117c0 con 0x7f4214141200 2024-01-26T23:29:48.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.807+0000 7f421a0f0700 1 -- 172.21.15.46:0/2289138836 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4214141200 msgr2=0x7f42140c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:48.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.807+0000 7f421a0f0700 1 --2- 172.21.15.46:0/2289138836 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4214141200 0x7f42140c9170 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4208009910 tx=0x7f42080091a0 comp rx=0 tx=0).stop 2024-01-26T23:29:48.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.807+0000 7f421a0f0700 1 -- 172.21.15.46:0/2289138836 shutdown_connections 2024-01-26T23:29:48.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.807+0000 7f421a0f0700 1 --2- 172.21.15.46:0/2289138836 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4214141200 0x7f42140c9170 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:48.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.807+0000 7f421a0f0700 1 -- 172.21.15.46:0/2289138836 >> 172.21.15.46:0/2289138836 conn(0x7f421406bd90 msgr2=0x7f421406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:48.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.807+0000 7f421a0f0700 1 -- 172.21.15.46:0/2289138836 shutdown_connections 2024-01-26T23:29:48.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.807+0000 7f421a0f0700 1 -- 172.21.15.46:0/2289138836 wait complete. 2024-01-26T23:29:48.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.809+0000 7f421a0f0700 1 Processor -- start 2024-01-26T23:29:48.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.809+0000 7f421a0f0700 1 -- start start 2024-01-26T23:29:48.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.809+0000 7f421a0f0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42140c8da0 0x7f4214142110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:48.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.809+0000 7f421a0f0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42140caf30 0x7f42140b3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:48.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.809+0000 7f421a0f0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4214141200 0x7f421413cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:48.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.809+0000 7f421a0f0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42140b3a50 con 0x7f42140c8da0 2024-01-26T23:29:48.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.809+0000 7f421a0f0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42141427d0 con 0x7f42140caf30 2024-01-26T23:29:48.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.809+0000 7f421a0f0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4214142ab0 con 0x7f4214141200 2024-01-26T23:29:48.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.810+0000 7f42137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42140c8da0 0x7f4214142110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:48.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.810+0000 7f42137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42140c8da0 0x7f4214142110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37290/0 (socket says 172.21.15.46:37290) 2024-01-26T23:29:48.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.810+0000 7f42137fe700 1 -- 172.21.15.46:0/3131304663 learned_addr learned my addr 172.21.15.46:0/3131304663 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:48.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.810+0000 7f4213fff700 1 --2- 172.21.15.46:0/3131304663 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4214141200 0x7f421413cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:48.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.810+0000 7f4212ffd700 1 --2- 172.21.15.46:0/3131304663 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42140caf30 0x7f42140b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:48.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.810+0000 7f42137fe700 1 -- 172.21.15.46:0/3131304663 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4214141200 msgr2=0x7f421413cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:48.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.810+0000 7f42137fe700 1 --2- 172.21.15.46:0/3131304663 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4214141200 0x7f421413cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:48.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.810+0000 7f42137fe700 1 -- 172.21.15.46:0/3131304663 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42140caf30 msgr2=0x7f42140b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:48.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.810+0000 7f42137fe700 1 --2- 172.21.15.46:0/3131304663 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42140caf30 0x7f42140b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:48.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.810+0000 7f42137fe700 1 -- 172.21.15.46:0/3131304663 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4208009420 con 0x7f42140c8da0 2024-01-26T23:29:48.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.810+0000 7f4212ffd700 1 --2- 172.21.15.46:0/3131304663 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42140caf30 0x7f42140b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:29:48.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.810+0000 7f42137fe700 1 --2- 172.21.15.46:0/3131304663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42140c8da0 0x7f4214142110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f420400c790 tx=0x7f4204007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:48.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.810+0000 7f41fb7fe700 1 -- 172.21.15.46:0/3131304663 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4204010510 con 0x7f42140c8da0 2024-01-26T23:29:48.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.810+0000 7f41fb7fe700 1 -- 172.21.15.46:0/3131304663 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4204010670 con 0x7f42140c8da0 2024-01-26T23:29:48.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.810+0000 7f421a0f0700 1 -- 172.21.15.46:0/3131304663 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4214142d90 con 0x7f42140c8da0 2024-01-26T23:29:48.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.810+0000 7f41fb7fe700 1 -- 172.21.15.46:0/3131304663 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4204010a30 con 0x7f42140c8da0 2024-01-26T23:29:48.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.810+0000 7f421a0f0700 1 -- 172.21.15.46:0/3131304663 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4214143350 con 0x7f42140c8da0 2024-01-26T23:29:48.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.812+0000 7f421a0f0700 1 -- 172.21.15.46:0/3131304663 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f421404c710 con 0x7f42140c8da0 2024-01-26T23:29:48.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.812+0000 7f41fb7fe700 1 -- 172.21.15.46:0/3131304663 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f420401b3b0 con 0x7f42140c8da0 2024-01-26T23:29:48.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.812+0000 7f41fb7fe700 1 --2- 172.21.15.46:0/3131304663 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4200066f30 0x7f42000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:48.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.812+0000 7f41fb7fe700 1 -- 172.21.15.46:0/3131304663 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6424+0+0 (secure 0 0 0) 0x7f42040840c0 con 0x7f42140c8da0 2024-01-26T23:29:48.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.813+0000 7f4212ffd700 1 --2- 172.21.15.46:0/3131304663 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4200066f30 0x7f42000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:48.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.813+0000 7f4212ffd700 1 --2- 172.21.15.46:0/3131304663 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4200066f30 0x7f42000693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f42080030c0 tx=0x7f4208016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:48.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:48.817+0000 7f41fb7fe700 1 -- 172.21.15.46:0/3131304663 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4204051060 con 0x7f42140c8da0 2024-01-26T23:29:48.922 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-01-26T23:29:48.920+0000 7f95b1ce4700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-26T23:29:49.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.073+0000 7f421a0f0700 1 -- 172.21.15.46:0/3131304663 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f4214143540 con 0x7f42140c8da0 2024-01-26T23:29:49.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.635+0000 7f41fb7fe700 1 -- 172.21.15.46:0/3131304663 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7f4214143540 con 0x7f42140c8da0 2024-01-26T23:29:49.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.638+0000 7f421a0f0700 1 -- 172.21.15.46:0/3131304663 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4200066f30 msgr2=0x7f42000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.638+0000 7f421a0f0700 1 --2- 172.21.15.46:0/3131304663 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4200066f30 0x7f42000693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f42080030c0 tx=0x7f4208016000 comp rx=0 tx=0).stop 2024-01-26T23:29:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.638+0000 7f421a0f0700 1 -- 172.21.15.46:0/3131304663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42140c8da0 msgr2=0x7f4214142110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.638+0000 7f421a0f0700 1 --2- 172.21.15.46:0/3131304663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42140c8da0 0x7f4214142110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f420400c790 tx=0x7f4204007e10 comp rx=0 tx=0).stop 2024-01-26T23:29:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.639+0000 7f421a0f0700 1 -- 172.21.15.46:0/3131304663 shutdown_connections 2024-01-26T23:29:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.639+0000 7f421a0f0700 1 --2- 172.21.15.46:0/3131304663 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4200066f30 0x7f42000693e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.639+0000 7f421a0f0700 1 --2- 172.21.15.46:0/3131304663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42140c8da0 0x7f4214142110 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.639+0000 7f421a0f0700 1 --2- 172.21.15.46:0/3131304663 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4214141200 0x7f421413cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.639+0000 7f421a0f0700 1 --2- 172.21.15.46:0/3131304663 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42140caf30 0x7f42140b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.639+0000 7f421a0f0700 1 -- 172.21.15.46:0/3131304663 >> 172.21.15.46:0/3131304663 conn(0x7f421406bd90 msgr2=0x7f42140c4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.639+0000 7f421a0f0700 1 -- 172.21.15.46:0/3131304663 shutdown_connections 2024-01-26T23:29:49.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.639+0000 7f421a0f0700 1 -- 172.21.15.46:0/3131304663 wait complete. 2024-01-26T23:29:49.642 INFO:teuthology.orchestra.run.smithi046.stderr:added data pool 4 to fsmap 2024-01-26T23:29:49.656 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:29:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.789+0000 7fbb35b46700 1 Processor -- start 2024-01-26T23:29:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.789+0000 7fbb35b46700 1 -- start start 2024-01-26T23:29:49.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.790+0000 7fbb35b46700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbb301561d0 0x7fbb300ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:49.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.790+0000 7fbb35b46700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbb300b0280 0x7fbb301565a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.790+0000 7fbb35b46700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb300b0760 con 0x7fbb3006eb70 2024-01-26T23:29:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.790+0000 7fbb35b46700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb300b0a40 con 0x7fbb300b0280 2024-01-26T23:29:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.790+0000 7fbb35b46700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb300b0d20 con 0x7fbb301561d0 2024-01-26T23:29:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.790+0000 7fbb2effd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbb300b0280 0x7fbb301565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.790+0000 7fbb2ffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbb3006eb70 0x7fbb3006c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49642/0 (socket says 172.21.15.46:49642) 2024-01-26T23:29:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.790+0000 7fbb2effd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbb300b0280 0x7fbb301565a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37306/0 (socket says 172.21.15.46:37306) 2024-01-26T23:29:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.790+0000 7fbb2ffff700 1 -- 172.21.15.46:0/318496442 learned_addr learned my addr 172.21.15.46:0/318496442 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.791+0000 7fbb2f7fe700 1 --2- 172.21.15.46:0/318496442 >> v2:172.21.15.106:3300/0 conn(0x7fbb301561d0 0x7fbb300ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.791+0000 7fbb2effd700 1 -- 172.21.15.46:0/318496442 >> v2:172.21.15.106:3300/0 conn(0x7fbb301561d0 msgr2=0x7fbb300ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.791+0000 7fbb2effd700 1 --2- 172.21.15.46:0/318496442 >> v2:172.21.15.106:3300/0 conn(0x7fbb301561d0 0x7fbb300ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.791+0000 7fbb2effd700 1 -- 172.21.15.46:0/318496442 >> v1:172.21.15.46:6789/0 conn(0x7fbb3006eb70 legacy=0x7fbb3006c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.791+0000 7fbb2effd700 1 -- 172.21.15.46:0/318496442 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb300b1000 con 0x7fbb300b0280 2024-01-26T23:29:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.791+0000 7fbb2e7fc700 1 -- 172.21.15.46:0/318496442 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554602459 0 0) 0x7fbb300b0760 con 0x7fbb3006eb70 2024-01-26T23:29:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.791+0000 7fbb2effd700 1 --2- 172.21.15.46:0/318496442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb300b0280 0x7fbb301565a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbb3006bb50 tx=0x7fbb180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf1d5799428f7d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.791+0000 7fbb2e7fc700 1 -- 172.21.15.46:0/318496442 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1800b6b0 con 0x7fbb300b0280 2024-01-26T23:29:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.791+0000 7fbb2e7fc700 1 -- 172.21.15.46:0/318496442 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1800b810 con 0x7fbb300b0280 2024-01-26T23:29:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.791+0000 7fbb2e7fc700 1 -- 172.21.15.46:0/318496442 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1800bbb0 con 0x7fbb300b0280 2024-01-26T23:29:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.792+0000 7fbb35b46700 1 -- 172.21.15.46:0/318496442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb300b0280 msgr2=0x7fbb301565a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.792+0000 7fbb35b46700 1 --2- 172.21.15.46:0/318496442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb300b0280 0x7fbb301565a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbb3006bb50 tx=0x7fbb180091a0 comp rx=0 tx=0).stop 2024-01-26T23:29:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.792+0000 7fbb35b46700 1 -- 172.21.15.46:0/318496442 shutdown_connections 2024-01-26T23:29:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.792+0000 7fbb35b46700 1 --2- 172.21.15.46:0/318496442 >> v2:172.21.15.106:3300/0 conn(0x7fbb301561d0 0x7fbb300ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.792+0000 7fbb35b46700 1 --2- 172.21.15.46:0/318496442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb300b0280 0x7fbb301565a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.792+0000 7fbb35b46700 1 -- 172.21.15.46:0/318496442 >> 172.21.15.46:0/318496442 conn(0x7fbb3006b630 msgr2=0x7fbb300aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.792+0000 7fbb35b46700 1 -- 172.21.15.46:0/318496442 shutdown_connections 2024-01-26T23:29:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.792+0000 7fbb35b46700 1 -- 172.21.15.46:0/318496442 wait complete. 2024-01-26T23:29:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.793+0000 7fbb35b46700 1 Processor -- start 2024-01-26T23:29:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.793+0000 7fbb35b46700 1 -- start start 2024-01-26T23:29:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.794+0000 7fbb35b46700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb3006eb70 0x7fbb30152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.794+0000 7fbb35b46700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb300b0280 0x7fbb301543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.794+0000 7fbb35b46700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb301561d0 0x7fbb30154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.794+0000 7fbb35b46700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb300a6d80 con 0x7fbb301561d0 2024-01-26T23:29:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.794+0000 7fbb35b46700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb301591e0 con 0x7fbb300b0280 2024-01-26T23:29:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.794+0000 7fbb35b46700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb300a64e0 con 0x7fbb3006eb70 2024-01-26T23:29:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.794+0000 7fbb2ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb301561d0 0x7fbb30154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.794+0000 7fbb2ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb301561d0 0x7fbb30154930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37308/0 (socket says 172.21.15.46:37308) 2024-01-26T23:29:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.794+0000 7fbb2ffff700 1 -- 172.21.15.46:0/2500038339 learned_addr learned my addr 172.21.15.46:0/2500038339 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.794+0000 7fbb2effd700 1 --2- 172.21.15.46:0/2500038339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb300b0280 0x7fbb301543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.794+0000 7fbb2f7fe700 1 --2- 172.21.15.46:0/2500038339 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb3006eb70 0x7fbb30152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.794+0000 7fbb2ffff700 1 -- 172.21.15.46:0/2500038339 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb3006eb70 msgr2=0x7fbb30152390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.794+0000 7fbb2ffff700 1 --2- 172.21.15.46:0/2500038339 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb3006eb70 0x7fbb30152390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.794+0000 7fbb2ffff700 1 -- 172.21.15.46:0/2500038339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb300b0280 msgr2=0x7fbb301543f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.794+0000 7fbb2ffff700 1 --2- 172.21.15.46:0/2500038339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb300b0280 0x7fbb301543f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.794+0000 7fbb2ffff700 1 -- 172.21.15.46:0/2500038339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb18009420 con 0x7fbb301561d0 2024-01-26T23:29:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.795+0000 7fbb2ffff700 1 --2- 172.21.15.46:0/2500038339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb301561d0 0x7fbb30154930 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800cc90 tx=0x7fbb28001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.795+0000 7fbb277fe700 1 -- 172.21.15.46:0/2500038339 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2800cea0 con 0x7fbb301561d0 2024-01-26T23:29:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.795+0000 7fbb277fe700 1 -- 172.21.15.46:0/2500038339 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2800ab70 con 0x7fbb301561d0 2024-01-26T23:29:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.795+0000 7fbb35b46700 1 -- 172.21.15.46:0/2500038339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb300a67c0 con 0x7fbb301561d0 2024-01-26T23:29:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.795+0000 7fbb277fe700 1 -- 172.21.15.46:0/2500038339 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb280070f0 con 0x7fbb301561d0 2024-01-26T23:29:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.795+0000 7fbb35b46700 1 -- 172.21.15.46:0/2500038339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb300a5760 con 0x7fbb301561d0 2024-01-26T23:29:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.796+0000 7fbb35b46700 1 -- 172.21.15.46:0/2500038339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb30158fb0 con 0x7fbb301561d0 2024-01-26T23:29:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.796+0000 7fbb277fe700 1 -- 172.21.15.46:0/2500038339 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbb280074a0 con 0x7fbb301561d0 2024-01-26T23:29:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.797+0000 7fbb277fe700 1 --2- 172.21.15.46:0/2500038339 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbb10066f30 0x7fbb100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.797+0000 7fbb277fe700 1 -- 172.21.15.46:0/2500038339 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fbb28084ca0 con 0x7fbb301561d0 2024-01-26T23:29:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.797+0000 7fbb2f7fe700 1 --2- 172.21.15.46:0/2500038339 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbb10066f30 0x7fbb100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.797+0000 7fbb2f7fe700 1 --2- 172.21.15.46:0/2500038339 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbb10066f30 0x7fbb100693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbb20006fd0 tx=0x7fbb20008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:49.801+0000 7fbb277fe700 1 -- 172.21.15.46:0/2500038339 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbb30158fb0 con 0x7fbb301561d0 2024-01-26T23:29:50.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.051+0000 7fbb35b46700 1 -- 172.21.15.46:0/2500038339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb30158fb0 con 0x7fbb301561d0 2024-01-26T23:29:50.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.051+0000 7fbb277fe700 1 -- 172.21.15.46:0/2500038339 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4593 (secure 0 0 0) 0x7fbb30158fb0 con 0x7fbb301561d0 2024-01-26T23:29:50.053 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:29:50.053 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3409861270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3409861270},{"type":"v1","addr":"172.21.15.46:6833","nonce":3409861270}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3319220815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3319220815},{"type":"v1","addr":"172.21.15.106:6836","nonce":3319220815}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4252,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/529198432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":529198432},{"type":"v1","addr":"172.21.15.106:6837","nonce":529198432}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:29:44.031963+0000","modified":"2024-01-26T23:29:49.624317+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/688086235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":688086235},{"type":"v1","addr":"172.21.15.46:6835","nonce":688086235}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-01-26T23:29:50.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.054+0000 7fbb35b46700 1 -- 172.21.15.46:0/2500038339 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbb10066f30 msgr2=0x7fbb100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.054+0000 7fbb35b46700 1 --2- 172.21.15.46:0/2500038339 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbb10066f30 0x7fbb100693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbb20006fd0 tx=0x7fbb20008040 comp rx=0 tx=0).stop 2024-01-26T23:29:50.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.054+0000 7fbb35b46700 1 -- 172.21.15.46:0/2500038339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb301561d0 msgr2=0x7fbb30154930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.054+0000 7fbb35b46700 1 --2- 172.21.15.46:0/2500038339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb301561d0 0x7fbb30154930 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800cc90 tx=0x7fbb28001a80 comp rx=0 tx=0).stop 2024-01-26T23:29:50.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.054+0000 7fbb35b46700 1 -- 172.21.15.46:0/2500038339 shutdown_connections 2024-01-26T23:29:50.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.054+0000 7fbb35b46700 1 --2- 172.21.15.46:0/2500038339 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbb10066f30 0x7fbb100693e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.054+0000 7fbb35b46700 1 --2- 172.21.15.46:0/2500038339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb301561d0 0x7fbb30154930 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.054+0000 7fbb35b46700 1 --2- 172.21.15.46:0/2500038339 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb3006eb70 0x7fbb30152390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.054+0000 7fbb35b46700 1 --2- 172.21.15.46:0/2500038339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb300b0280 0x7fbb301543f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.054+0000 7fbb35b46700 1 -- 172.21.15.46:0/2500038339 >> 172.21.15.46:0/2500038339 conn(0x7fbb3006b630 msgr2=0x7fbb30152b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:50.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.054+0000 7fbb35b46700 1 -- 172.21.15.46:0/2500038339 shutdown_connections 2024-01-26T23:29:50.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.054+0000 7fbb35b46700 1 -- 172.21.15.46:0/2500038339 wait complete. 2024-01-26T23:29:50.057 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 7 2024-01-26T23:29:50.069 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:29:50.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.199+0000 7faaaf3cf700 1 Processor -- start 2024-01-26T23:29:50.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.199+0000 7faaaf3cf700 1 -- start start 2024-01-26T23:29:50.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.200+0000 7faaaf3cf700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faaa80cb4c0 0x7faaa8159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.200+0000 7faaaf3cf700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa80c97f0 con 0x7faaa80c4720 2024-01-26T23:29:50.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.200+0000 7faaaf3cf700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa80c8c20 con 0x7faaa81596e0 2024-01-26T23:29:50.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.200+0000 7faaaf3cf700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa80ce0d0 con 0x7faaa80cb4c0 2024-01-26T23:29:50.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.200+0000 7faaad96c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faaa80c4720 0x7faaa80cd9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49656/0 (socket says 172.21.15.46:49656) 2024-01-26T23:29:50.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.200+0000 7faaad96c700 1 -- 172.21.15.46:0/2785865622 learned_addr learned my addr 172.21.15.46:0/2785865622 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:50.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.200+0000 7faaac96a700 1 --2- 172.21.15.46:0/2785865622 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faaa80cb4c0 0x7faaa8159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.200+0000 7faa9bfff700 1 -- 172.21.15.46:0/2785865622 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345031805 0 0) 0x7faaa80c97f0 con 0x7faaa80c4720 2024-01-26T23:29:50.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.200+0000 7faa9bfff700 1 -- 172.21.15.46:0/2785865622 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa94003570 con 0x7faaa80c4720 2024-01-26T23:29:50.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.201+0000 7faa9bfff700 1 -- 172.21.15.46:0/2785865622 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882833818 0 0) 0x7faaa80c8c20 con 0x7faaa81596e0 2024-01-26T23:29:50.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.201+0000 7faaac96a700 1 -- 172.21.15.46:0/2785865622 >> v1:172.21.15.106:6789/0 conn(0x7faaa81596e0 legacy=0x7faaa80cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.201+0000 7faaac96a700 1 -- 172.21.15.46:0/2785865622 >> v1:172.21.15.46:6789/0 conn(0x7faaa80c4720 legacy=0x7faaa80cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.201+0000 7faaac96a700 1 -- 172.21.15.46:0/2785865622 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa80c8ec0 con 0x7faaa80cb4c0 2024-01-26T23:29:50.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.201+0000 7faaac96a700 1 --2- 172.21.15.46:0/2785865622 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faaa80cb4c0 0x7faaa8159ab0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faa9c009910 tx=0x7faa9c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ba53401275e3b5b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:50.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.201+0000 7faa9bfff700 1 -- 172.21.15.46:0/2785865622 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9c00b680 con 0x7faaa80cb4c0 2024-01-26T23:29:50.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.201+0000 7faa9bfff700 1 -- 172.21.15.46:0/2785865622 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9c00b7e0 con 0x7faaa80cb4c0 2024-01-26T23:29:50.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.201+0000 7faa9bfff700 1 -- 172.21.15.46:0/2785865622 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9c00bba0 con 0x7faaa80cb4c0 2024-01-26T23:29:50.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.202+0000 7faaaf3cf700 1 -- 172.21.15.46:0/2785865622 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faaa80cb4c0 msgr2=0x7faaa8159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.202+0000 7faaaf3cf700 1 --2- 172.21.15.46:0/2785865622 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faaa80cb4c0 0x7faaa8159ab0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faa9c009910 tx=0x7faa9c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.202+0000 7faaaf3cf700 1 -- 172.21.15.46:0/2785865622 shutdown_connections 2024-01-26T23:29:50.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.202+0000 7faaaf3cf700 1 --2- 172.21.15.46:0/2785865622 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faaa80cb4c0 0x7faaa8159ab0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.202+0000 7faaaf3cf700 1 -- 172.21.15.46:0/2785865622 >> 172.21.15.46:0/2785865622 conn(0x7faaa8097820 msgr2=0x7faaa8068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:50.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.202+0000 7faaaf3cf700 1 -- 172.21.15.46:0/2785865622 shutdown_connections 2024-01-26T23:29:50.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.202+0000 7faaaf3cf700 1 -- 172.21.15.46:0/2785865622 wait complete. 2024-01-26T23:29:50.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.203+0000 7faaaf3cf700 1 Processor -- start 2024-01-26T23:29:50.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.203+0000 7faaaf3cf700 1 -- start start 2024-01-26T23:29:50.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.204+0000 7faaaf3cf700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faaa80c4720 0x7faaa8147440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.204+0000 7faaaf3cf700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faaa80cb4c0 0x7faaa816cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.204+0000 7faaaf3cf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa81596e0 0x7faaa816d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.204+0000 7faaaf3cf700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa8158860 con 0x7faaa81596e0 2024-01-26T23:29:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.204+0000 7faaaf3cf700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa8147980 con 0x7faaa80c4720 2024-01-26T23:29:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.204+0000 7faaaf3cf700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa816b610 con 0x7faaa80cb4c0 2024-01-26T23:29:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.204+0000 7faaad96c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa81596e0 0x7faaa816d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.204+0000 7faaad96c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa81596e0 0x7faaa816d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37320/0 (socket says 172.21.15.46:37320) 2024-01-26T23:29:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.204+0000 7faaad96c700 1 -- 172.21.15.46:0/3513654266 learned_addr learned my addr 172.21.15.46:0/3513654266 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.204+0000 7faaac96a700 1 --2- 172.21.15.46:0/3513654266 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faaa80cb4c0 0x7faaa816cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.204+0000 7faaad16b700 1 --2- 172.21.15.46:0/3513654266 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faaa80c4720 0x7faaa8147440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.204+0000 7faaad96c700 1 -- 172.21.15.46:0/3513654266 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faaa80cb4c0 msgr2=0x7faaa816cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.204+0000 7faaad96c700 1 --2- 172.21.15.46:0/3513654266 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faaa80cb4c0 0x7faaa816cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.204+0000 7faaad96c700 1 -- 172.21.15.46:0/3513654266 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faaa80c4720 msgr2=0x7faaa8147440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.204+0000 7faaad96c700 1 --2- 172.21.15.46:0/3513654266 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faaa80c4720 0x7faaa8147440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.205+0000 7faaad96c700 1 -- 172.21.15.46:0/3513654266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c009420 con 0x7faaa81596e0 2024-01-26T23:29:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.205+0000 7faaad96c700 1 --2- 172.21.15.46:0/3513654266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa81596e0 0x7faaa816d4e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faaa0001460 tx=0x7faaa00019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.205+0000 7faa997fa700 1 -- 172.21.15.46:0/3513654266 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa000a8b0 con 0x7faaa81596e0 2024-01-26T23:29:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.205+0000 7faa997fa700 1 -- 172.21.15.46:0/3513654266 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa000aa10 con 0x7faaa81596e0 2024-01-26T23:29:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.205+0000 7faaaf3cf700 1 -- 172.21.15.46:0/3513654266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa816b8f0 con 0x7faaa81596e0 2024-01-26T23:29:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.205+0000 7faa997fa700 1 -- 172.21.15.46:0/3513654266 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa000ad90 con 0x7faaa81596e0 2024-01-26T23:29:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.205+0000 7faaaf3cf700 1 -- 172.21.15.46:0/3513654266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa80b1380 con 0x7faaa81596e0 2024-01-26T23:29:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.206+0000 7faaaf3cf700 1 -- 172.21.15.46:0/3513654266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa804c710 con 0x7faaa81596e0 2024-01-26T23:29:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.206+0000 7faa997fa700 1 -- 172.21.15.46:0/3513654266 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7faaa0007300 con 0x7faaa81596e0 2024-01-26T23:29:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.207+0000 7faa997fa700 1 --2- 172.21.15.46:0/3513654266 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa8c066f30 0x7faa8c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.207+0000 7faa997fa700 1 -- 172.21.15.46:0/3513654266 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7faaa0084bb0 con 0x7faaa81596e0 2024-01-26T23:29:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.207+0000 7faaad16b700 1 --2- 172.21.15.46:0/3513654266 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa8c066f30 0x7faa8c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.207+0000 7faaad16b700 1 --2- 172.21.15.46:0/3513654266 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa8c066f30 0x7faa8c0693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faaa4007fd0 tx=0x7faaa4007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:50.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.211+0000 7faa997fa700 1 -- 172.21.15.46:0/3513654266 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faaa0051b10 con 0x7faaa81596e0 2024-01-26T23:29:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.417+0000 7faaaf3cf700 1 -- 172.21.15.46:0/3513654266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faaa806e4b0 con 0x7faaa81596e0 2024-01-26T23:29:50.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.418+0000 7faa997fa700 1 -- 172.21.15.46:0/3513654266 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7faaa0055130 con 0x7faaa81596e0 2024-01-26T23:29:50.420 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:29:50.420 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:49.075104+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:29:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.420+0000 7faaaf3cf700 1 -- 172.21.15.46:0/3513654266 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa8c066f30 msgr2=0x7faa8c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.420+0000 7faaaf3cf700 1 --2- 172.21.15.46:0/3513654266 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa8c066f30 0x7faa8c0693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faaa4007fd0 tx=0x7faaa4007d30 comp rx=0 tx=0).stop 2024-01-26T23:29:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.420+0000 7faaaf3cf700 1 -- 172.21.15.46:0/3513654266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa81596e0 msgr2=0x7faaa816d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.420+0000 7faaaf3cf700 1 --2- 172.21.15.46:0/3513654266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa81596e0 0x7faaa816d4e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faaa0001460 tx=0x7faaa00019e0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.420+0000 7faaaf3cf700 1 -- 172.21.15.46:0/3513654266 shutdown_connections 2024-01-26T23:29:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.420+0000 7faaaf3cf700 1 --2- 172.21.15.46:0/3513654266 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa8c066f30 0x7faa8c0693e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.420+0000 7faaaf3cf700 1 --2- 172.21.15.46:0/3513654266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa81596e0 0x7faaa816d4e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.420+0000 7faaaf3cf700 1 --2- 172.21.15.46:0/3513654266 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faaa80c4720 0x7faaa8147440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.420+0000 7faaaf3cf700 1 --2- 172.21.15.46:0/3513654266 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faaa80cb4c0 0x7faaa816cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.420+0000 7faaaf3cf700 1 -- 172.21.15.46:0/3513654266 >> 172.21.15.46:0/3513654266 conn(0x7faaa8097820 msgr2=0x7faaa80cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.420+0000 7faaaf3cf700 1 -- 172.21.15.46:0/3513654266 shutdown_connections 2024-01-26T23:29:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.420+0000 7faaaf3cf700 1 -- 172.21.15.46:0/3513654266 wait complete. 2024-01-26T23:29:50.436 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:29:50.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.566+0000 7fbd09d17700 1 Processor -- start 2024-01-26T23:29:50.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.566+0000 7fbd09d17700 1 -- start start 2024-01-26T23:29:50.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.566+0000 7fbd09d17700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd04150720 0x7fbd041575a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.566+0000 7fbd09d17700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbd04150af0 0x7fbd04067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.566+0000 7fbd09d17700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd0406e2a0 0x7fbd0406e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.566+0000 7fbd09d17700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd0406ec60 con 0x7fbd04150720 2024-01-26T23:29:50.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.566+0000 7fbd09d17700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd04068d20 con 0x7fbd04150af0 2024-01-26T23:29:50.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.566+0000 7fbd09d17700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd04067e70 con 0x7fbd0406e2a0 2024-01-26T23:29:50.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.567+0000 7fbd037fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd04150720 0x7fbd041575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.567+0000 7fbd037fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd04150720 0x7fbd041575a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52150/0 (socket says 172.21.15.46:52150) 2024-01-26T23:29:50.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.567+0000 7fbd037fe700 1 -- 172.21.15.46:0/285404126 learned_addr learned my addr 172.21.15.46:0/285404126 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:50.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.567+0000 7fbd03fff700 1 --2- 172.21.15.46:0/285404126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd0406e2a0 0x7fbd0406e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.567+0000 7fbd02ffd700 1 --2- 172.21.15.46:0/285404126 >> v2:172.21.15.106:3300/0 conn(0x7fbd04150af0 0x7fbd04067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.567+0000 7fbd037fe700 1 -- 172.21.15.46:0/285404126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd0406e2a0 msgr2=0x7fbd0406e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:50.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.567+0000 7fbd037fe700 1 --2- 172.21.15.46:0/285404126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd0406e2a0 0x7fbd0406e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.567+0000 7fbd037fe700 1 -- 172.21.15.46:0/285404126 >> v2:172.21.15.106:3300/0 conn(0x7fbd04150af0 msgr2=0x7fbd04067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:50.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.567+0000 7fbd037fe700 1 --2- 172.21.15.46:0/285404126 >> v2:172.21.15.106:3300/0 conn(0x7fbd04150af0 0x7fbd04067930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.567+0000 7fbd037fe700 1 -- 172.21.15.46:0/285404126 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd04068110 con 0x7fbd04150720 2024-01-26T23:29:50.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.567+0000 7fbd02ffd700 1 --2- 172.21.15.46:0/285404126 >> v2:172.21.15.106:3300/0 conn(0x7fbd04150af0 0x7fbd04067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:29:50.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.567+0000 7fbd037fe700 1 --2- 172.21.15.46:0/285404126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04150720 0x7fbd041575a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbcf40099b0 tx=0x7fbcf4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed76267819da4c66 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:50.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.567+0000 7fbd027fc700 1 -- 172.21.15.46:0/285404126 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4009ea0 con 0x7fbd04150720 2024-01-26T23:29:50.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.567+0000 7fbd027fc700 1 -- 172.21.15.46:0/285404126 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcf400b820 con 0x7fbd04150720 2024-01-26T23:29:50.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.567+0000 7fbd027fc700 1 -- 172.21.15.46:0/285404126 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf400bbc0 con 0x7fbd04150720 2024-01-26T23:29:50.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.568+0000 7fbd09d17700 1 -- 172.21.15.46:0/285404126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04150720 msgr2=0x7fbd041575a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.568+0000 7fbd09d17700 1 --2- 172.21.15.46:0/285404126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04150720 0x7fbd041575a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbcf40099b0 tx=0x7fbcf4009250 comp rx=0 tx=0).stop 2024-01-26T23:29:50.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.568+0000 7fbd09d17700 1 -- 172.21.15.46:0/285404126 shutdown_connections 2024-01-26T23:29:50.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.568+0000 7fbd09d17700 1 --2- 172.21.15.46:0/285404126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04150720 0x7fbd041575a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.568+0000 7fbd09d17700 1 --2- 172.21.15.46:0/285404126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd0406e2a0 0x7fbd0406e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.568+0000 7fbd09d17700 1 --2- 172.21.15.46:0/285404126 >> v2:172.21.15.106:3300/0 conn(0x7fbd04150af0 0x7fbd04067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.568+0000 7fbd09d17700 1 -- 172.21.15.46:0/285404126 >> 172.21.15.46:0/285404126 conn(0x7fbd040ccbc0 msgr2=0x7fbd040ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:50.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.568+0000 7fbd09d17700 1 -- 172.21.15.46:0/285404126 shutdown_connections 2024-01-26T23:29:50.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.568+0000 7fbd09d17700 1 -- 172.21.15.46:0/285404126 wait complete. 2024-01-26T23:29:50.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.569+0000 7fbd09d17700 1 Processor -- start 2024-01-26T23:29:50.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.570+0000 7fbd09d17700 1 -- start start 2024-01-26T23:29:50.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.570+0000 7fbd09d17700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd0406e2a0 0x7fbd041415a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.570+0000 7fbd09d17700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04150720 0x7fbd040ab010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.570+0000 7fbd09d17700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd04150af0 0x7fbd040ab550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.570+0000 7fbd09d17700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd040a69e0 con 0x7fbd04150720 2024-01-26T23:29:50.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.570+0000 7fbd09d17700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd040abbd0 con 0x7fbd0406e2a0 2024-01-26T23:29:50.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.570+0000 7fbd09d17700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd040abeb0 con 0x7fbd04150af0 2024-01-26T23:29:50.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.570+0000 7fbd02ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04150720 0x7fbd040ab010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.570+0000 7fbd03fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd04150af0 0x7fbd040ab550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.570+0000 7fbd037fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd0406e2a0 0x7fbd041415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.570+0000 7fbd02ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04150720 0x7fbd040ab010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52164/0 (socket says 172.21.15.46:52164) 2024-01-26T23:29:50.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.570+0000 7fbd03fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd04150af0 0x7fbd040ab550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:34702/0 (socket says 172.21.15.46:34702) 2024-01-26T23:29:50.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.570+0000 7fbd02ffd700 1 -- 172.21.15.46:0/152065941 learned_addr learned my addr 172.21.15.46:0/152065941 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:50.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.571+0000 7fbd02ffd700 1 -- 172.21.15.46:0/152065941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd04150af0 msgr2=0x7fbd040ab550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.571+0000 7fbd02ffd700 1 --2- 172.21.15.46:0/152065941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd04150af0 0x7fbd040ab550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.571+0000 7fbd02ffd700 1 -- 172.21.15.46:0/152065941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd0406e2a0 msgr2=0x7fbd041415a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.571+0000 7fbd02ffd700 1 --2- 172.21.15.46:0/152065941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd0406e2a0 0x7fbd041415a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.571+0000 7fbd02ffd700 1 -- 172.21.15.46:0/152065941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf40094f0 con 0x7fbd04150720 2024-01-26T23:29:50.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.571+0000 7fbd02ffd700 1 --2- 172.21.15.46:0/152065941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04150720 0x7fbd040ab010 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbcf800d860 tx=0x7fbcf8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:50.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.571+0000 7fbceb7fe700 1 -- 172.21.15.46:0/152065941 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf80125f0 con 0x7fbd04150720 2024-01-26T23:29:50.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.571+0000 7fbceb7fe700 1 -- 172.21.15.46:0/152065941 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcf8012750 con 0x7fbd04150720 2024-01-26T23:29:50.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.571+0000 7fbceb7fe700 1 -- 172.21.15.46:0/152065941 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8012af0 con 0x7fbd04150720 2024-01-26T23:29:50.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.571+0000 7fbd09d17700 1 -- 172.21.15.46:0/152065941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd040ac190 con 0x7fbd04150720 2024-01-26T23:29:50.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.571+0000 7fbd09d17700 1 -- 172.21.15.46:0/152065941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd04141ae0 con 0x7fbd04150720 2024-01-26T23:29:50.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.572+0000 7fbd09d17700 1 -- 172.21.15.46:0/152065941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd040a6b50 con 0x7fbd04150720 2024-01-26T23:29:50.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.573+0000 7fbceb7fe700 1 -- 172.21.15.46:0/152065941 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbcf8013480 con 0x7fbd04150720 2024-01-26T23:29:50.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.573+0000 7fbceb7fe700 1 --2- 172.21.15.46:0/152065941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbcf0066f30 0x7fbcf00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.573+0000 7fbceb7fe700 1 -- 172.21.15.46:0/152065941 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fbcf80849c0 con 0x7fbd04150720 2024-01-26T23:29:50.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.573+0000 7fbd037fe700 1 --2- 172.21.15.46:0/152065941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbcf0066f30 0x7fbcf00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.574+0000 7fbd037fe700 1 --2- 172.21.15.46:0/152065941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbcf0066f30 0x7fbcf00693e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbcf40099b0 tx=0x7fbcf4017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:50.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.578+0000 7fbceb7fe700 1 -- 172.21.15.46:0/152065941 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbcf8051940 con 0x7fbd04150720 2024-01-26T23:29:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.786+0000 7fbd09d17700 1 -- 172.21.15.46:0/152065941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd0404c710 con 0x7fbd04150720 2024-01-26T23:29:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.787+0000 7fbceb7fe700 1 -- 172.21.15.46:0/152065941 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7fbcf8054f60 con 0x7fbd04150720 2024-01-26T23:29:50.789 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:29:50.789 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:49.075104+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:29:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.789+0000 7fbd09d17700 1 -- 172.21.15.46:0/152065941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbcf0066f30 msgr2=0x7fbcf00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.789+0000 7fbd09d17700 1 --2- 172.21.15.46:0/152065941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbcf0066f30 0x7fbcf00693e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbcf40099b0 tx=0x7fbcf4017000 comp rx=0 tx=0).stop 2024-01-26T23:29:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.789+0000 7fbd09d17700 1 -- 172.21.15.46:0/152065941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04150720 msgr2=0x7fbd040ab010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.789+0000 7fbd09d17700 1 --2- 172.21.15.46:0/152065941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04150720 0x7fbd040ab010 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbcf800d860 tx=0x7fbcf8008040 comp rx=0 tx=0).stop 2024-01-26T23:29:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.789+0000 7fbd09d17700 1 -- 172.21.15.46:0/152065941 shutdown_connections 2024-01-26T23:29:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.789+0000 7fbd09d17700 1 --2- 172.21.15.46:0/152065941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbcf0066f30 0x7fbcf00693e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.789+0000 7fbd09d17700 1 --2- 172.21.15.46:0/152065941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd04150af0 0x7fbd040ab550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.789+0000 7fbd09d17700 1 --2- 172.21.15.46:0/152065941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd0406e2a0 0x7fbd041415a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.789+0000 7fbd09d17700 1 --2- 172.21.15.46:0/152065941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd04150720 0x7fbd040ab010 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.789+0000 7fbd09d17700 1 -- 172.21.15.46:0/152065941 >> 172.21.15.46:0/152065941 conn(0x7fbd040ccbc0 msgr2=0x7fbd040aef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.789+0000 7fbd09d17700 1 -- 172.21.15.46:0/152065941 shutdown_connections 2024-01-26T23:29:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.789+0000 7fbd09d17700 1 -- 172.21.15.46:0/152065941 wait complete. 2024-01-26T23:29:50.817 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:29:50.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.948+0000 7fd49d9fe700 1 Processor -- start 2024-01-26T23:29:50.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.948+0000 7fd49d9fe700 1 -- start start 2024-01-26T23:29:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.948+0000 7fd49d9fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd4980b2b40 0x7fd4980ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.948+0000 7fd49d9fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd4980ae960 0x7fd4980ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.949+0000 7fd49d9fe700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49815a0b0 con 0x7fd4980aedd0 2024-01-26T23:29:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.949+0000 7fd49d9fe700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd498099fc0 con 0x7fd4980ae960 2024-01-26T23:29:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.949+0000 7fd49d9fe700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4980ae4d0 con 0x7fd4980b2b40 2024-01-26T23:29:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.949+0000 7fd4967fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd4980ae960 0x7fd4980ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.949+0000 7fd4977fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd4980aedd0 0x7fd4980cb5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51132/0 (socket says 172.21.15.46:51132) 2024-01-26T23:29:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.949+0000 7fd4977fe700 1 -- 172.21.15.46:0/1094296735 learned_addr learned my addr 172.21.15.46:0/1094296735 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.949+0000 7fd496ffd700 1 --2- 172.21.15.46:0/1094296735 >> v2:172.21.15.106:3300/0 conn(0x7fd4980b2b40 0x7fd4980ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.949+0000 7fd4967fc700 1 -- 172.21.15.46:0/1094296735 >> v2:172.21.15.106:3300/0 conn(0x7fd4980b2b40 msgr2=0x7fd4980ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.949+0000 7fd4967fc700 1 --2- 172.21.15.46:0/1094296735 >> v2:172.21.15.106:3300/0 conn(0x7fd4980b2b40 0x7fd4980ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.949+0000 7fd4967fc700 1 -- 172.21.15.46:0/1094296735 >> v1:172.21.15.46:6789/0 conn(0x7fd4980aedd0 legacy=0x7fd4980cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.949+0000 7fd495ffb700 1 -- 172.21.15.46:0/1094296735 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549198962 0 0) 0x7fd49815a0b0 con 0x7fd4980aedd0 2024-01-26T23:29:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.949+0000 7fd4967fc700 1 -- 172.21.15.46:0/1094296735 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4980ad8c0 con 0x7fd4980ae960 2024-01-26T23:29:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.949+0000 7fd4967fc700 1 --2- 172.21.15.46:0/1094296735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4980ae960 0x7fd4980ced50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd498088850 tx=0x7fd48c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=625dc4ea6ae2226 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.949+0000 7fd495ffb700 1 -- 172.21.15.46:0/1094296735 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c00b6b0 con 0x7fd4980ae960 2024-01-26T23:29:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.949+0000 7fd495ffb700 1 -- 172.21.15.46:0/1094296735 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48c00b810 con 0x7fd4980ae960 2024-01-26T23:29:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.950+0000 7fd495ffb700 1 -- 172.21.15.46:0/1094296735 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c00bbb0 con 0x7fd4980ae960 2024-01-26T23:29:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.950+0000 7fd49d9fe700 1 -- 172.21.15.46:0/1094296735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4980ae960 msgr2=0x7fd4980ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.950+0000 7fd49d9fe700 1 --2- 172.21.15.46:0/1094296735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4980ae960 0x7fd4980ced50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd498088850 tx=0x7fd48c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.950+0000 7fd49d9fe700 1 -- 172.21.15.46:0/1094296735 shutdown_connections 2024-01-26T23:29:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.950+0000 7fd49d9fe700 1 --2- 172.21.15.46:0/1094296735 >> v2:172.21.15.106:3300/0 conn(0x7fd4980b2b40 0x7fd4980ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.950+0000 7fd49d9fe700 1 --2- 172.21.15.46:0/1094296735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4980ae960 0x7fd4980ced50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.950+0000 7fd49d9fe700 1 -- 172.21.15.46:0/1094296735 >> 172.21.15.46:0/1094296735 conn(0x7fd498099a30 msgr2=0x7fd49806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.950+0000 7fd49d9fe700 1 -- 172.21.15.46:0/1094296735 shutdown_connections 2024-01-26T23:29:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.950+0000 7fd49d9fe700 1 -- 172.21.15.46:0/1094296735 wait complete. 2024-01-26T23:29:50.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.951+0000 7fd49d9fe700 1 Processor -- start 2024-01-26T23:29:50.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.952+0000 7fd49d9fe700 1 -- start start 2024-01-26T23:29:50.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.952+0000 7fd49d9fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4980ae960 0x7fd4980af290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.952+0000 7fd49d9fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd4980b2b40 0x7fd4980afb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.952+0000 7fd49d9fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd498156250 0x7fd49816cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.952+0000 7fd49d9fe700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980b0040 con 0x7fd498156250 2024-01-26T23:29:50.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.952+0000 7fd49d9fe700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980b4180 con 0x7fd4980ae960 2024-01-26T23:29:50.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.952+0000 7fd49d9fe700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980b1960 con 0x7fd4980b2b40 2024-01-26T23:29:50.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.952+0000 7fd4977fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd498156250 0x7fd49816cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.952+0000 7fd4977fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd498156250 0x7fd49816cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52186/0 (socket says 172.21.15.46:52186) 2024-01-26T23:29:50.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.953+0000 7fd4977fe700 1 -- 172.21.15.46:0/1487278587 learned_addr learned my addr 172.21.15.46:0/1487278587 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:50.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.953+0000 7fd496ffd700 1 --2- 172.21.15.46:0/1487278587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4980ae960 0x7fd4980af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.953+0000 7fd4967fc700 1 --2- 172.21.15.46:0/1487278587 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd4980b2b40 0x7fd4980afb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.953+0000 7fd4977fe700 1 -- 172.21.15.46:0/1487278587 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd4980b2b40 msgr2=0x7fd4980afb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.953+0000 7fd4977fe700 1 --2- 172.21.15.46:0/1487278587 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd4980b2b40 0x7fd4980afb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.953+0000 7fd4977fe700 1 -- 172.21.15.46:0/1487278587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4980ae960 msgr2=0x7fd4980af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:50.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.953+0000 7fd4977fe700 1 --2- 172.21.15.46:0/1487278587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4980ae960 0x7fd4980af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:50.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.953+0000 7fd4977fe700 1 -- 172.21.15.46:0/1487278587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48c009420 con 0x7fd498156250 2024-01-26T23:29:50.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.953+0000 7fd4977fe700 1 --2- 172.21.15.46:0/1487278587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd498156250 0x7fd49816cd70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd49000ccc0 tx=0x7fd490001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:50.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.953+0000 7fd47effd700 1 -- 172.21.15.46:0/1487278587 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49000a820 con 0x7fd498156250 2024-01-26T23:29:50.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.953+0000 7fd47effd700 1 -- 172.21.15.46:0/1487278587 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49000a980 con 0x7fd498156250 2024-01-26T23:29:50.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.953+0000 7fd49d9fe700 1 -- 172.21.15.46:0/1487278587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4980b1c40 con 0x7fd498156250 2024-01-26T23:29:50.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.953+0000 7fd47effd700 1 -- 172.21.15.46:0/1487278587 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49000ad20 con 0x7fd498156250 2024-01-26T23:29:50.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.953+0000 7fd49d9fe700 1 -- 172.21.15.46:0/1487278587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4980b0280 con 0x7fd498156250 2024-01-26T23:29:50.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.954+0000 7fd49d9fe700 1 -- 172.21.15.46:0/1487278587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4980639f0 con 0x7fd498156250 2024-01-26T23:29:50.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.955+0000 7fd47effd700 1 -- 172.21.15.46:0/1487278587 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd490007300 con 0x7fd498156250 2024-01-26T23:29:50.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.955+0000 7fd47effd700 1 --2- 172.21.15.46:0/1487278587 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd484066f30 0x7fd4840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:50.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.955+0000 7fd47effd700 1 -- 172.21.15.46:0/1487278587 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fd490084b10 con 0x7fd498156250 2024-01-26T23:29:50.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.955+0000 7fd496ffd700 1 --2- 172.21.15.46:0/1487278587 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd484066f30 0x7fd4840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:50.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.956+0000 7fd496ffd700 1 --2- 172.21.15.46:0/1487278587 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd484066f30 0x7fd4840693e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd488006aa0 tx=0x7fd488008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:50.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:50.959+0000 7fd47effd700 1 -- 172.21.15.46:0/1487278587 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd490051b10 con 0x7fd498156250 2024-01-26T23:29:51.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.213+0000 7fd49d9fe700 1 -- 172.21.15.46:0/1487278587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd49804c710 con 0x7fd498156250 2024-01-26T23:29:51.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.213+0000 7fd47effd700 1 -- 172.21.15.46:0/1487278587 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4593 (secure 0 0 0) 0x7fd490055130 con 0x7fd498156250 2024-01-26T23:29:51.215 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:29:51.215 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3409861270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3409861270},{"type":"v1","addr":"172.21.15.46:6833","nonce":3409861270}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3319220815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3319220815},{"type":"v1","addr":"172.21.15.106:6836","nonce":3319220815}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4252,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/529198432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":529198432},{"type":"v1","addr":"172.21.15.106:6837","nonce":529198432}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:29:44.031963+0000","modified":"2024-01-26T23:29:49.624317+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/688086235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":688086235},{"type":"v1","addr":"172.21.15.46:6835","nonce":688086235}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-01-26T23:29:51.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.216+0000 7fd49d9fe700 1 -- 172.21.15.46:0/1487278587 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd484066f30 msgr2=0x7fd4840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:51.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.216+0000 7fd49d9fe700 1 --2- 172.21.15.46:0/1487278587 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd484066f30 0x7fd4840693e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd488006aa0 tx=0x7fd488008040 comp rx=0 tx=0).stop 2024-01-26T23:29:51.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.216+0000 7fd49d9fe700 1 -- 172.21.15.46:0/1487278587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd498156250 msgr2=0x7fd49816cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:51.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.216+0000 7fd49d9fe700 1 --2- 172.21.15.46:0/1487278587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd498156250 0x7fd49816cd70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd49000ccc0 tx=0x7fd490001a20 comp rx=0 tx=0).stop 2024-01-26T23:29:51.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.216+0000 7fd49d9fe700 1 -- 172.21.15.46:0/1487278587 shutdown_connections 2024-01-26T23:29:51.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.216+0000 7fd49d9fe700 1 --2- 172.21.15.46:0/1487278587 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd484066f30 0x7fd4840693e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:51.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.216+0000 7fd49d9fe700 1 --2- 172.21.15.46:0/1487278587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd498156250 0x7fd49816cd70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:51.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.216+0000 7fd49d9fe700 1 --2- 172.21.15.46:0/1487278587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4980ae960 0x7fd4980af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:51.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.216+0000 7fd49d9fe700 1 --2- 172.21.15.46:0/1487278587 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd4980b2b40 0x7fd4980afb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:51.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.216+0000 7fd49d9fe700 1 -- 172.21.15.46:0/1487278587 >> 172.21.15.46:0/1487278587 conn(0x7fd498099a30 msgr2=0x7fd498068910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:51.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.216+0000 7fd49d9fe700 1 -- 172.21.15.46:0/1487278587 shutdown_connections 2024-01-26T23:29:51.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.216+0000 7fd49d9fe700 1 -- 172.21.15.46:0/1487278587 wait complete. 2024-01-26T23:29:51.219 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 7 2024-01-26T23:29:51.231 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:29:51.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.361+0000 7fdb622b4700 1 Processor -- start 2024-01-26T23:29:51.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.361+0000 7fdb622b4700 1 -- start start 2024-01-26T23:29:51.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.362+0000 7fdb622b4700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdb5c068040 0x7fdb5c164430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.362+0000 7fdb622b4700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c15abc0 con 0x7fdb5c159b30 2024-01-26T23:29:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.362+0000 7fdb622b4700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c0b2e90 con 0x7fdb5c159760 2024-01-26T23:29:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.362+0000 7fdb622b4700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb5c159fb0 con 0x7fdb5c068040 2024-01-26T23:29:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.362+0000 7fdb5b7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdb5c159b30 0x7fdb5c067930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51138/0 (socket says 172.21.15.46:51138) 2024-01-26T23:29:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.362+0000 7fdb5b7fe700 1 -- 172.21.15.46:0/2670471247 learned_addr learned my addr 172.21.15.46:0/2670471247 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.362+0000 7fdb60851700 1 --2- 172.21.15.46:0/2670471247 >> v2:172.21.15.106:3300/0 conn(0x7fdb5c068040 0x7fdb5c164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.362+0000 7fdb5affd700 1 -- 172.21.15.46:0/2670471247 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778490164 0 0) 0x7fdb5c15abc0 con 0x7fdb5c159b30 2024-01-26T23:29:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.362+0000 7fdb5affd700 1 -- 172.21.15.46:0/2670471247 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb48003570 con 0x7fdb5c159b30 2024-01-26T23:29:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.362+0000 7fdb60851700 1 -- 172.21.15.46:0/2670471247 >> v1:172.21.15.106:6789/0 conn(0x7fdb5c159760 legacy=0x7fdb5c0b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.362+0000 7fdb60851700 1 -- 172.21.15.46:0/2670471247 >> v1:172.21.15.46:6789/0 conn(0x7fdb5c159b30 legacy=0x7fdb5c067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.362+0000 7fdb5affd700 1 -- 172.21.15.46:0/2670471247 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787533871 0 0) 0x7fdb5c0b2e90 con 0x7fdb5c159760 2024-01-26T23:29:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.362+0000 7fdb60851700 1 -- 172.21.15.46:0/2670471247 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c15a290 con 0x7fdb5c068040 2024-01-26T23:29:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.363+0000 7fdb60851700 1 --2- 172.21.15.46:0/2670471247 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb5c068040 0x7fdb5c164430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009e60 tx=0x7fdb54009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6b2d43132492158 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.363+0000 7fdb5affd700 1 -- 172.21.15.46:0/2670471247 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5400e070 con 0x7fdb5c068040 2024-01-26T23:29:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.363+0000 7fdb5affd700 1 -- 172.21.15.46:0/2670471247 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5400bb60 con 0x7fdb5c068040 2024-01-26T23:29:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.363+0000 7fdb5affd700 1 -- 172.21.15.46:0/2670471247 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb54012430 con 0x7fdb5c068040 2024-01-26T23:29:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.363+0000 7fdb622b4700 1 -- 172.21.15.46:0/2670471247 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb5c068040 msgr2=0x7fdb5c164430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.363+0000 7fdb622b4700 1 --2- 172.21.15.46:0/2670471247 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb5c068040 0x7fdb5c164430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009e60 tx=0x7fdb54009780 comp rx=0 tx=0).stop 2024-01-26T23:29:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.363+0000 7fdb622b4700 1 -- 172.21.15.46:0/2670471247 shutdown_connections 2024-01-26T23:29:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.363+0000 7fdb622b4700 1 --2- 172.21.15.46:0/2670471247 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb5c068040 0x7fdb5c164430 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.363+0000 7fdb622b4700 1 -- 172.21.15.46:0/2670471247 >> 172.21.15.46:0/2670471247 conn(0x7fdb5c06b630 msgr2=0x7fdb5c0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.363+0000 7fdb622b4700 1 -- 172.21.15.46:0/2670471247 shutdown_connections 2024-01-26T23:29:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.364+0000 7fdb622b4700 1 -- 172.21.15.46:0/2670471247 wait complete. 2024-01-26T23:29:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.365+0000 7fdb622b4700 1 Processor -- start 2024-01-26T23:29:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.365+0000 7fdb622b4700 1 -- start start 2024-01-26T23:29:51.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb622b4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb5c068040 0x7fdb5c16cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:51.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb622b4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c159760 0x7fdb5c16d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:51.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb622b4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb5c14dbc0 0x7fdb5c14e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:51.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb622b4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c16d920 con 0x7fdb5c159760 2024-01-26T23:29:51.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb622b4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c15d720 con 0x7fdb5c14dbc0 2024-01-26T23:29:51.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb622b4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c15da00 con 0x7fdb5c068040 2024-01-26T23:29:51.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb5b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c159760 0x7fdb5c16d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:51.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb5b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c159760 0x7fdb5c16d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52196/0 (socket says 172.21.15.46:52196) 2024-01-26T23:29:51.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb5b7fe700 1 -- 172.21.15.46:0/2055495294 learned_addr learned my addr 172.21.15.46:0/2055495294 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb5bfff700 1 --2- 172.21.15.46:0/2055495294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb5c068040 0x7fdb5c16cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb60851700 1 --2- 172.21.15.46:0/2055495294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb5c14dbc0 0x7fdb5c14e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb5b7fe700 1 -- 172.21.15.46:0/2055495294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb5c068040 msgr2=0x7fdb5c16cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb5b7fe700 1 --2- 172.21.15.46:0/2055495294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb5c068040 0x7fdb5c16cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb5b7fe700 1 -- 172.21.15.46:0/2055495294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb5c14dbc0 msgr2=0x7fdb5c14e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb5b7fe700 1 --2- 172.21.15.46:0/2055495294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb5c14dbc0 0x7fdb5c14e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb5b7fe700 1 -- 172.21.15.46:0/2055495294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb54009a20 con 0x7fdb5c159760 2024-01-26T23:29:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb5bfff700 1 --2- 172.21.15.46:0/2055495294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb5c068040 0x7fdb5c16cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:29:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb5b7fe700 1 --2- 172.21.15.46:0/2055495294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c159760 0x7fdb5c16d2d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdb440011e0 tx=0x7fdb44007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.366+0000 7fdb53fff700 1 -- 172.21.15.46:0/2055495294 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb44005b30 con 0x7fdb5c159760 2024-01-26T23:29:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.367+0000 7fdb53fff700 1 -- 172.21.15.46:0/2055495294 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb44005c90 con 0x7fdb5c159760 2024-01-26T23:29:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.367+0000 7fdb622b4700 1 -- 172.21.15.46:0/2055495294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c15dce0 con 0x7fdb5c159760 2024-01-26T23:29:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.367+0000 7fdb53fff700 1 -- 172.21.15.46:0/2055495294 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb44007320 con 0x7fdb5c159760 2024-01-26T23:29:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.367+0000 7fdb622b4700 1 -- 172.21.15.46:0/2055495294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c14e580 con 0x7fdb5c159760 2024-01-26T23:29:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.368+0000 7fdb622b4700 1 -- 172.21.15.46:0/2055495294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5c15df20 con 0x7fdb5c159760 2024-01-26T23:29:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.368+0000 7fdb53fff700 1 -- 172.21.15.46:0/2055495294 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fdb44007690 con 0x7fdb5c159760 2024-01-26T23:29:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.369+0000 7fdb53fff700 1 --2- 172.21.15.46:0/2055495294 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdb3c066f30 0x7fdb3c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:51.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.369+0000 7fdb53fff700 1 -- 172.21.15.46:0/2055495294 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fdb44083c30 con 0x7fdb5c159760 2024-01-26T23:29:51.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.369+0000 7fdb5bfff700 1 --2- 172.21.15.46:0/2055495294 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdb3c066f30 0x7fdb3c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:51.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.369+0000 7fdb5bfff700 1 --2- 172.21.15.46:0/2055495294 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdb3c066f30 0x7fdb3c0693e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c14f320 tx=0x7fdb4c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:51.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.373+0000 7fdb53fff700 1 -- 172.21.15.46:0/2055495294 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdb5c15df20 con 0x7fdb5c159760 2024-01-26T23:29:51.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.580+0000 7fdb622b4700 1 -- 172.21.15.46:0/2055495294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb5c15df20 con 0x7fdb5c159760 2024-01-26T23:29:51.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.581+0000 7fdb53fff700 1 -- 172.21.15.46:0/2055495294 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7fdb5c15df20 con 0x7fdb5c159760 2024-01-26T23:29:51.583 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:29:51.583 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:49.075104+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:29:51.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.583+0000 7fdb622b4700 1 -- 172.21.15.46:0/2055495294 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdb3c066f30 msgr2=0x7fdb3c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:51.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.583+0000 7fdb622b4700 1 --2- 172.21.15.46:0/2055495294 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdb3c066f30 0x7fdb3c0693e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c14f320 tx=0x7fdb4c009040 comp rx=0 tx=0).stop 2024-01-26T23:29:51.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.583+0000 7fdb622b4700 1 -- 172.21.15.46:0/2055495294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c159760 msgr2=0x7fdb5c16d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:51.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.583+0000 7fdb622b4700 1 --2- 172.21.15.46:0/2055495294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c159760 0x7fdb5c16d2d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdb440011e0 tx=0x7fdb44007df0 comp rx=0 tx=0).stop 2024-01-26T23:29:51.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.583+0000 7fdb622b4700 1 -- 172.21.15.46:0/2055495294 shutdown_connections 2024-01-26T23:29:51.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.583+0000 7fdb622b4700 1 --2- 172.21.15.46:0/2055495294 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdb3c066f30 0x7fdb3c0693e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:51.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.583+0000 7fdb622b4700 1 --2- 172.21.15.46:0/2055495294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb5c14dbc0 0x7fdb5c14e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:51.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.583+0000 7fdb622b4700 1 --2- 172.21.15.46:0/2055495294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb5c068040 0x7fdb5c16cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:51.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.583+0000 7fdb622b4700 1 --2- 172.21.15.46:0/2055495294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c159760 0x7fdb5c16d2d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:51.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.583+0000 7fdb622b4700 1 -- 172.21.15.46:0/2055495294 >> 172.21.15.46:0/2055495294 conn(0x7fdb5c06b630 msgr2=0x7fdb5c0ad280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:51.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.583+0000 7fdb622b4700 1 -- 172.21.15.46:0/2055495294 shutdown_connections 2024-01-26T23:29:51.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:51.583+0000 7fdb622b4700 1 -- 172.21.15.46:0/2055495294 wait complete. 2024-01-26T23:29:51.921 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:29:51.924 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:29:52.001 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-26T23:29:44.031963+0000', 'modified': '2024-01-26T23:29:49.624317+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4254}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4254': {'gid': 4254, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6835/688086235', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6834', 'nonce': 688086235}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 688086235}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2024-01-26T23:29:52.001 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-26T23:29:52.002 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-26T23:29:52.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.141+0000 7f252eb4f700 1 Processor -- start 2024-01-26T23:29:52.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.141+0000 7f252eb4f700 1 -- start start 2024-01-26T23:29:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.142+0000 7f252eb4f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f25280b1fc0 0x7f25280b2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.142+0000 7f252eb4f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25280cac60 con 0x7f2528068310 2024-01-26T23:29:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.142+0000 7f252eb4f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2528068dc0 con 0x7f2528157690 2024-01-26T23:29:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.142+0000 7f252eb4f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25280690a0 con 0x7f25280b1fc0 2024-01-26T23:29:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.142+0000 7f2527fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2528068310 0x7f2528156f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51144/0 (socket says 172.21.15.46:51144) 2024-01-26T23:29:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.142+0000 7f2527fff700 1 -- 172.21.15.46:0/2402560454 learned_addr learned my addr 172.21.15.46:0/2402560454 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.142+0000 7f252c8eb700 1 --2- 172.21.15.46:0/2402560454 >> v2:172.21.15.106:3300/0 conn(0x7f25280b1fc0 0x7f25280b2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.143+0000 7f25277fe700 1 -- 172.21.15.46:0/2402560454 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2217562197 0 0) 0x7f25280cac60 con 0x7f2528068310 2024-01-26T23:29:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.143+0000 7f25277fe700 1 -- 172.21.15.46:0/2402560454 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2518003570 con 0x7f2528068310 2024-01-26T23:29:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.143+0000 7f252c8eb700 1 -- 172.21.15.46:0/2402560454 >> v1:172.21.15.106:6789/0 conn(0x7f2528157690 legacy=0x7f25280cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.143+0000 7f252c8eb700 1 -- 172.21.15.46:0/2402560454 >> v1:172.21.15.46:6789/0 conn(0x7f2528068310 legacy=0x7f2528156f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.143+0000 7f252c8eb700 1 -- 172.21.15.46:0/2402560454 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25280caee0 con 0x7f25280b1fc0 2024-01-26T23:29:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.143+0000 7f252c8eb700 1 --2- 172.21.15.46:0/2402560454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25280b1fc0 0x7f25280b2390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f251c0098e0 tx=0x7f251c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91490cd8e0b3224c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.143+0000 7f25277fe700 1 -- 172.21.15.46:0/2402560454 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c00b640 con 0x7f25280b1fc0 2024-01-26T23:29:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.143+0000 7f25277fe700 1 -- 172.21.15.46:0/2402560454 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251c00b7a0 con 0x7f25280b1fc0 2024-01-26T23:29:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.143+0000 7f25277fe700 1 -- 172.21.15.46:0/2402560454 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c00bb40 con 0x7f25280b1fc0 2024-01-26T23:29:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.144+0000 7f252eb4f700 1 -- 172.21.15.46:0/2402560454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25280b1fc0 msgr2=0x7f25280b2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.144+0000 7f252eb4f700 1 --2- 172.21.15.46:0/2402560454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25280b1fc0 0x7f25280b2390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f251c0098e0 tx=0x7f251c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.144+0000 7f252eb4f700 1 -- 172.21.15.46:0/2402560454 shutdown_connections 2024-01-26T23:29:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.144+0000 7f252eb4f700 1 --2- 172.21.15.46:0/2402560454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25280b1fc0 0x7f25280b2390 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.144+0000 7f252eb4f700 1 -- 172.21.15.46:0/2402560454 >> 172.21.15.46:0/2402560454 conn(0x7f25280ccbc0 msgr2=0x7f25280ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.144+0000 7f252eb4f700 1 -- 172.21.15.46:0/2402560454 shutdown_connections 2024-01-26T23:29:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.144+0000 7f252eb4f700 1 -- 172.21.15.46:0/2402560454 wait complete. 2024-01-26T23:29:52.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.145+0000 7f252eb4f700 1 Processor -- start 2024-01-26T23:29:52.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.145+0000 7f252eb4f700 1 -- start start 2024-01-26T23:29:52.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.146+0000 7f252eb4f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2528068310 0x7f252815b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.146+0000 7f252eb4f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25280b1fc0 0x7f252814f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.146+0000 7f252eb4f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2528157690 0x7f252813f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.146+0000 7f252eb4f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252815af80 con 0x7f2528157690 2024-01-26T23:29:52.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.146+0000 7f252eb4f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252815a370 con 0x7f2528068310 2024-01-26T23:29:52.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.146+0000 7f252eb4f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f252815a650 con 0x7f25280b1fc0 2024-01-26T23:29:52.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.146+0000 7f252d0ec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2528157690 0x7f252813f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.146+0000 7f252d0ec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2528157690 0x7f252813f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52208/0 (socket says 172.21.15.46:52208) 2024-01-26T23:29:52.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.146+0000 7f252d0ec700 1 -- 172.21.15.46:0/1392596020 learned_addr learned my addr 172.21.15.46:0/1392596020 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:52.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.146+0000 7f2527fff700 1 --2- 172.21.15.46:0/1392596020 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25280b1fc0 0x7f252814f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.146+0000 7f252c8eb700 1 --2- 172.21.15.46:0/1392596020 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2528068310 0x7f252815b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.147+0000 7f252d0ec700 1 -- 172.21.15.46:0/1392596020 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25280b1fc0 msgr2=0x7f252814f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:52.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.147+0000 7f252d0ec700 1 --2- 172.21.15.46:0/1392596020 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25280b1fc0 0x7f252814f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.147+0000 7f252d0ec700 1 -- 172.21.15.46:0/1392596020 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2528068310 msgr2=0x7f252815b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:52.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.147+0000 7f252d0ec700 1 --2- 172.21.15.46:0/1392596020 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2528068310 0x7f252815b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.147+0000 7f252d0ec700 1 -- 172.21.15.46:0/1392596020 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251c009420 con 0x7f2528157690 2024-01-26T23:29:52.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.147+0000 7f252d0ec700 1 --2- 172.21.15.46:0/1392596020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2528157690 0x7f252813f290 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f252000c840 tx=0x7f2520001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:52.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.147+0000 7f2524ff9700 1 -- 172.21.15.46:0/1392596020 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2520012630 con 0x7f2528157690 2024-01-26T23:29:52.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.147+0000 7f252eb4f700 1 -- 172.21.15.46:0/1392596020 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f252815a930 con 0x7f2528157690 2024-01-26T23:29:52.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.147+0000 7f2524ff9700 1 -- 172.21.15.46:0/1392596020 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25200127b0 con 0x7f2528157690 2024-01-26T23:29:52.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.147+0000 7f2524ff9700 1 -- 172.21.15.46:0/1392596020 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2520012b70 con 0x7f2528157690 2024-01-26T23:29:52.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.147+0000 7f252eb4f700 1 -- 172.21.15.46:0/1392596020 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f252814fae0 con 0x7f2528157690 2024-01-26T23:29:52.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.148+0000 7f2524ff9700 1 -- 172.21.15.46:0/1392596020 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2520003030 con 0x7f2528157690 2024-01-26T23:29:52.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.149+0000 7f2524ff9700 1 --2- 172.21.15.46:0/1392596020 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f250c066f30 0x7f250c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.149+0000 7f2524ff9700 1 -- 172.21.15.46:0/1392596020 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f2520084890 con 0x7f2528157690 2024-01-26T23:29:52.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.149+0000 7f252c8eb700 1 --2- 172.21.15.46:0/1392596020 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f250c066f30 0x7f250c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.150+0000 7f252c8eb700 1 --2- 172.21.15.46:0/1392596020 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f250c066f30 0x7f250c0693e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f251c0093f0 tx=0x7f251c017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.151+0000 7f2537e52b80 1 Processor -- start 2024-01-26T23:29:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.151+0000 7f2537e52b80 1 -- start start 2024-01-26T23:29:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f2537e52b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d35010 0x559517d353e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f2537e52b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d7d360 0x559517d5f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f2537e52b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d345d0 0x559517d5fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f2537e52b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x559517c389f0 con 0x559517d7d360 2024-01-26T23:29:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f2537e52b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x559517c8a130 con 0x559517d345d0 2024-01-26T23:29:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f2537e52b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x559517c87930 con 0x559517d35010 2024-01-26T23:29:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f2527fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d7d360 0x559517d5f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f2527fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d7d360 0x559517d5f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52210/0 (socket says 172.21.15.46:52210) 2024-01-26T23:29:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f2527fff700 1 -- 172.21.15.46:0/279669230 learned_addr learned my addr 172.21.15.46:0/279669230 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f252c8eb700 1 --2- 172.21.15.46:0/279669230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d35010 0x559517d353e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f252d0ec700 1 --2- 172.21.15.46:0/279669230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d345d0 0x559517d5fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f2527fff700 1 -- 172.21.15.46:0/279669230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d35010 msgr2=0x559517d353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f2527fff700 1 --2- 172.21.15.46:0/279669230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d35010 0x559517d353e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f2527fff700 1 -- 172.21.15.46:0/279669230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d345d0 msgr2=0x559517d5fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f2527fff700 1 --2- 172.21.15.46:0/279669230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d345d0 0x559517d5fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f2527fff700 1 -- 172.21.15.46:0/279669230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559517ac2b10 con 0x559517d7d360 2024-01-26T23:29:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f252d0ec700 1 --2- 172.21.15.46:0/279669230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d345d0 0x559517d5fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:29:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.152+0000 7f2527fff700 1 --2- 172.21.15.46:0/279669230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d7d360 0x559517d5f840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f25140016e0 tx=0x7f2514009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c562b7db92a1d2e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.153+0000 7f2511ffb700 1 -- 172.21.15.46:0/279669230 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2514005b10 con 0x559517d7d360 2024-01-26T23:29:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.153+0000 7f2511ffb700 1 -- 172.21.15.46:0/279669230 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2514005c90 con 0x559517d7d360 2024-01-26T23:29:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.153+0000 7f2511ffb700 1 -- 172.21.15.46:0/279669230 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2514007320 con 0x559517d7d360 2024-01-26T23:29:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.153+0000 7f2537e52b80 1 -- 172.21.15.46:0/279669230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d7d360 msgr2=0x559517d5f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.153+0000 7f2537e52b80 1 --2- 172.21.15.46:0/279669230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d7d360 0x559517d5f840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f25140016e0 tx=0x7f2514009040 comp rx=0 tx=0).stop 2024-01-26T23:29:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.153+0000 7f2537e52b80 1 -- 172.21.15.46:0/279669230 shutdown_connections 2024-01-26T23:29:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.153+0000 7f2537e52b80 1 --2- 172.21.15.46:0/279669230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d345d0 0x559517d5fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.153+0000 7f2537e52b80 1 --2- 172.21.15.46:0/279669230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d35010 0x559517d353e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.153+0000 7f2537e52b80 1 --2- 172.21.15.46:0/279669230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d7d360 0x559517d5f840 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.153+0000 7f2537e52b80 1 -- 172.21.15.46:0/279669230 >> 172.21.15.46:0/279669230 conn(0x559517d34a70 msgr2=0x559517d7d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.153+0000 7f2537e52b80 1 -- 172.21.15.46:0/279669230 shutdown_connections 2024-01-26T23:29:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.153+0000 7f2537e52b80 1 -- 172.21.15.46:0/279669230 wait complete. 2024-01-26T23:29:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.154+0000 7f2537e52b80 1 -- start start 2024-01-26T23:29:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.154+0000 7f2537e52b80 1 Processor -- start 2024-01-26T23:29:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.155+0000 7f2537e52b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d345d0 0x559517d5a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.155+0000 7f2537e52b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d35010 0x559517d60800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.155+0000 7f2537e52b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d7d360 0x559517d60d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.155+0000 7f2537e52b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x559517c83720 con 0x559517d345d0 2024-01-26T23:29:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.155+0000 7f2537e52b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x559517c2f1b0 con 0x559517d7d360 2024-01-26T23:29:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.155+0000 7f2537e52b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x559517c2ee50 con 0x559517d35010 2024-01-26T23:29:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.155+0000 7f2537e52b80 10 client.0 _ll_register_callbacks cb 0x559517a917b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:29:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.155+0000 7f252c8eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d345d0 0x559517d5a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.155+0000 7f252c8eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d345d0 0x559517d5a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52218/0 (socket says 172.21.15.46:52218) 2024-01-26T23:29:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.155+0000 7f252c8eb700 1 -- 172.21.15.46:0/4275111851 learned_addr learned my addr 172.21.15.46:0/4275111851 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.155+0000 7f252d0ec700 1 --2- 172.21.15.46:0/4275111851 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d7d360 0x559517d60d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.155+0000 7f2527fff700 1 --2- 172.21.15.46:0/4275111851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d35010 0x559517d60800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.155+0000 7f252c8eb700 1 -- 172.21.15.46:0/4275111851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d35010 msgr2=0x559517d60800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.155+0000 7f252c8eb700 1 --2- 172.21.15.46:0/4275111851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d35010 0x559517d60800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.155+0000 7f252c8eb700 1 -- 172.21.15.46:0/4275111851 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d7d360 msgr2=0x559517d60d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.155+0000 7f252c8eb700 1 --2- 172.21.15.46:0/4275111851 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d7d360 0x559517d60d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.155+0000 7f252c8eb700 1 -- 172.21.15.46:0/4275111851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f251400f040 con 0x559517d345d0 2024-01-26T23:29:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.156+0000 7f252c8eb700 1 --2- 172.21.15.46:0/4275111851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d345d0 0x559517d5a4f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f25280b2c40 tx=0x7f251c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da797426cd8ef8dd server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.156+0000 7f2506ffd700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:29:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.156+0000 7f2506ffd700 1 -- 172.21.15.46:0/4275111851 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c02f070 con 0x559517d345d0 2024-01-26T23:29:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.156+0000 7f2506ffd700 1 -- 172.21.15.46:0/4275111851 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f251c026430 con 0x559517d345d0 2024-01-26T23:29:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.156+0000 7f2537e52b80 1 -- 172.21.15.46:0/4275111851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559517d28190 con 0x559517d345d0 2024-01-26T23:29:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.156+0000 7f2506ffd700 1 -- 172.21.15.46:0/4275111851 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f251c020070 con 0x559517d345d0 2024-01-26T23:29:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.156+0000 7f2537e52b80 10 client.4404 fetch_fsmap learned FSMap version 7 2024-01-26T23:29:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.156+0000 7f2537e52b80 1 -- 172.21.15.46:0/4275111851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x559517c51290 con 0x559517d345d0 2024-01-26T23:29:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.156+0000 7f2506ffd700 1 -- 172.21.15.46:0/4275111851 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559517d28190 con 0x559517d345d0 2024-01-26T23:29:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.156+0000 7f2506ffd700 1 -- 172.21.15.46:0/4275111851 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 7) v1 ==== 3108+0+0 (secure 0 0 0) 0x7f251c026800 con 0x559517d345d0 2024-01-26T23:29:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.156+0000 7f2537e52b80 10 client.4404 fetch_fsmap finished waiting for FSMap version 7 2024-01-26T23:29:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.156+0000 7f2537e52b80 10 client.4404 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-26T23:29:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.156+0000 7f2537e52b80 20 client.4404 populate_metadata read hostname 'smithi046' 2024-01-26T23:29:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.156+0000 7f2537e52b80 1 --2- 172.21.15.46:0/4275111851 >> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] conn(0x559517d62930 0x559517d62d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.156+0000 7f2537e52b80 4 client.4404 mds_command: new command op to 4254 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-26T23:29:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.156+0000 7f2537e52b80 1 -- 172.21.15.46:0/4275111851 --> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559517c25ee0 con 0x559517d62930 2024-01-26T23:29:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.157+0000 7f2527fff700 1 --2- 172.21.15.46:0/4275111851 >> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] conn(0x559517d62930 0x559517d62d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.157+0000 7f2527fff700 1 --2- 172.21.15.46:0/4275111851 >> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] conn(0x559517d62930 0x559517d62d00 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2ffe9e292ac3e0 server_cookie=c5128a8af44487d1 in_seq=0 out_seq=0 2024-01-26T23:29:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.157+0000 7f2506ffd700 10 client.4404 ms_handle_connect on v2:172.21.15.46:6834/688086235 2024-01-26T23:29:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.157+0000 7f2506ffd700 1 -- 172.21.15.46:0/4275111851 <== mds.0 v2:172.21.15.46:6834/688086235 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x559517c25ee0 con 0x559517d62930 2024-01-26T23:29:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.157+0000 7f2506ffd700 10 client.4404 handle_command_reply: tid=0 2024-01-26T23:29:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.157+0000 7f2537e52b80 1 client.4404 shutdown 2024-01-26T23:29:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.158+0000 7f2537e52b80 1 -- 172.21.15.46:0/4275111851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d345d0 msgr2=0x559517d5a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.158+0000 7f2537e52b80 1 --2- 172.21.15.46:0/4275111851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d345d0 0x559517d5a4f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f25280b2c40 tx=0x7f251c020000 comp rx=0 tx=0).stop 2024-01-26T23:29:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.158+0000 7f2537e52b80 1 -- 172.21.15.46:0/4275111851 shutdown_connections 2024-01-26T23:29:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.158+0000 7f2537e52b80 1 --2- 172.21.15.46:0/4275111851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d345d0 0x559517d5a4f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.158+0000 7f2537e52b80 1 --2- 172.21.15.46:0/4275111851 >> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] conn(0x559517d62930 0x559517d62d00 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.158+0000 7f2537e52b80 1 --2- 172.21.15.46:0/4275111851 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d7d360 0x559517d60d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.158+0000 7f2537e52b80 1 --2- 172.21.15.46:0/4275111851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d35010 0x559517d60800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.158+0000 7f2506ffd700 0 client.4404 ms_handle_reset on v2:172.21.15.46:6834/688086235 2024-01-26T23:29:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.158+0000 7f2537e52b80 1 -- 172.21.15.46:0/4275111851 >> 172.21.15.46:0/4275111851 conn(0x559517d34a70 msgr2=0x559517d2fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.158+0000 7f2537e52b80 1 -- 172.21.15.46:0/4275111851 shutdown_connections 2024-01-26T23:29:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.158+0000 7f2537e52b80 1 -- 172.21.15.46:0/4275111851 wait complete. 2024-01-26T23:29:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.158+0000 7f2537e52b80 20 client.4404 trim_cache size 0 max 16384 2024-01-26T23:29:52.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.176+0000 7f2537e52b80 1 Processor -- start 2024-01-26T23:29:52.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.176+0000 7f2537e52b80 1 -- start start 2024-01-26T23:29:52.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f2537e52b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d345d0 0x559517d73db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f2537e52b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d35010 0x559517d78e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f2537e52b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d7d360 0x559517d79370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f2537e52b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x559517b7be40 con 0x559517d345d0 2024-01-26T23:29:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f2537e52b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x559517d309a0 con 0x559517d7d360 2024-01-26T23:29:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f2537e52b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x559517d31a90 con 0x559517d35010 2024-01-26T23:29:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f252c8eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d345d0 0x559517d73db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f252c8eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d345d0 0x559517d73db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52220/0 (socket says 172.21.15.46:52220) 2024-01-26T23:29:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f252c8eb700 1 -- 172.21.15.46:0/2755224397 learned_addr learned my addr 172.21.15.46:0/2755224397 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f2527fff700 1 --2- 172.21.15.46:0/2755224397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d35010 0x559517d78e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f252d0ec700 1 --2- 172.21.15.46:0/2755224397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d7d360 0x559517d79370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f252c8eb700 1 -- 172.21.15.46:0/2755224397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d35010 msgr2=0x559517d78e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f252c8eb700 1 --2- 172.21.15.46:0/2755224397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d35010 0x559517d78e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f252c8eb700 1 -- 172.21.15.46:0/2755224397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d7d360 msgr2=0x559517d79370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f252c8eb700 1 --2- 172.21.15.46:0/2755224397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d7d360 0x559517d79370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f252c8eb700 1 -- 172.21.15.46:0/2755224397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559517b1ac80 con 0x559517d345d0 2024-01-26T23:29:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f2527fff700 1 --2- 172.21.15.46:0/2755224397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d35010 0x559517d78e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:29:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f252c8eb700 1 --2- 172.21.15.46:0/2755224397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d345d0 0x559517d73db0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f251c00b140 tx=0x7f251c00b050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94ae7ee2c5613aac server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f25127fc700 1 -- 172.21.15.46:0/2755224397 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c02f040 con 0x559517d345d0 2024-01-26T23:29:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f25127fc700 1 -- 172.21.15.46:0/2755224397 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251c026430 con 0x559517d345d0 2024-01-26T23:29:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.177+0000 7f25127fc700 1 -- 172.21.15.46:0/2755224397 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c020040 con 0x559517d345d0 2024-01-26T23:29:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.178+0000 7f2537e52b80 1 -- 172.21.15.46:0/2755224397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d345d0 msgr2=0x559517d73db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.178+0000 7f2537e52b80 1 --2- 172.21.15.46:0/2755224397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d345d0 0x559517d73db0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f251c00b140 tx=0x7f251c00b050 comp rx=0 tx=0).stop 2024-01-26T23:29:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.178+0000 7f2537e52b80 1 -- 172.21.15.46:0/2755224397 shutdown_connections 2024-01-26T23:29:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.178+0000 7f2537e52b80 1 --2- 172.21.15.46:0/2755224397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d345d0 0x559517d73db0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.178+0000 7f2537e52b80 1 --2- 172.21.15.46:0/2755224397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d7d360 0x559517d79370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.178+0000 7f2537e52b80 1 --2- 172.21.15.46:0/2755224397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d35010 0x559517d78e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.178+0000 7f2537e52b80 1 -- 172.21.15.46:0/2755224397 >> 172.21.15.46:0/2755224397 conn(0x559517d34a70 msgr2=0x559517d88640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.178+0000 7f2537e52b80 1 -- 172.21.15.46:0/2755224397 shutdown_connections 2024-01-26T23:29:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.178+0000 7f2537e52b80 1 -- 172.21.15.46:0/2755224397 wait complete. 2024-01-26T23:29:52.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.179+0000 7f2537e52b80 1 -- start start 2024-01-26T23:29:52.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.179+0000 7f2537e52b80 1 Processor -- start 2024-01-26T23:29:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2537e52b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d345d0 0x559517dde3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2537e52b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d35010 0x559517d32e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2537e52b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d7d360 0x559517d333a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2537e52b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x559517dded60 con 0x559517d35010 2024-01-26T23:29:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2537e52b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x559517d5bfb0 con 0x559517d345d0 2024-01-26T23:29:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2537e52b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x559517d596d0 con 0x559517d7d360 2024-01-26T23:29:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2537e52b80 10 client.0 _ll_register_callbacks cb 0x559517a917b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:29:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2527fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d35010 0x559517d32e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2527fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d35010 0x559517d32e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52230/0 (socket says 172.21.15.46:52230) 2024-01-26T23:29:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2527fff700 1 -- 172.21.15.46:0/2035782672 learned_addr learned my addr 172.21.15.46:0/2035782672 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f252d0ec700 1 --2- 172.21.15.46:0/2035782672 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d7d360 0x559517d333a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f252c8eb700 1 --2- 172.21.15.46:0/2035782672 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d345d0 0x559517dde3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2527fff700 1 -- 172.21.15.46:0/2035782672 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d7d360 msgr2=0x559517d333a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2527fff700 1 --2- 172.21.15.46:0/2035782672 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d7d360 0x559517d333a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2527fff700 1 -- 172.21.15.46:0/2035782672 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d345d0 msgr2=0x559517dde3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2527fff700 1 --2- 172.21.15.46:0/2035782672 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d345d0 0x559517dde3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2527fff700 1 -- 172.21.15.46:0/2035782672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559517c51290 con 0x559517d35010 2024-01-26T23:29:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2527fff700 1 --2- 172.21.15.46:0/2035782672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d35010 0x559517d32e60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f25140058d0 tx=0x7f2514028310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffc55ce3ed98998 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2506ffd700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:29:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2506ffd700 1 -- 172.21.15.46:0/2035782672 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2514027070 con 0x559517d35010 2024-01-26T23:29:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.180+0000 7f2506ffd700 1 -- 172.21.15.46:0/2035782672 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f2514007840 con 0x559517d35010 2024-01-26T23:29:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.181+0000 7f2537e52b80 1 -- 172.21.15.46:0/2035782672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2514005c90 con 0x559517d35010 2024-01-26T23:29:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.181+0000 7f2506ffd700 1 -- 172.21.15.46:0/2035782672 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f25140072e0 con 0x559517d35010 2024-01-26T23:29:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.181+0000 7f2537e52b80 10 client.4410 fetch_fsmap learned FSMap version 7 2024-01-26T23:29:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.181+0000 7f2537e52b80 1 -- 172.21.15.46:0/2035782672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x559517ac0fe0 con 0x559517d35010 2024-01-26T23:29:52.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.181+0000 7f2506ffd700 1 -- 172.21.15.46:0/2035782672 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2514005c90 con 0x559517d35010 2024-01-26T23:29:52.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.181+0000 7f2506ffd700 1 -- 172.21.15.46:0/2035782672 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 7) v1 ==== 3108+0+0 (secure 0 0 0) 0x7f2514001710 con 0x559517d35010 2024-01-26T23:29:52.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.181+0000 7f2537e52b80 10 client.4410 fetch_fsmap finished waiting for FSMap version 7 2024-01-26T23:29:52.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.181+0000 7f2537e52b80 10 client.4410 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-26T23:29:52.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.181+0000 7f2537e52b80 20 client.4410 populate_metadata read hostname 'smithi046' 2024-01-26T23:29:52.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.181+0000 7f2537e52b80 1 --2- 172.21.15.46:0/2035782672 >> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] conn(0x559517d60b20 0x559517d60ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.181+0000 7f2537e52b80 4 client.4410 mds_command: new command op to 4254 tid=0[{"prefix": "status"}] 2024-01-26T23:29:52.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.181+0000 7f2537e52b80 1 -- 172.21.15.46:0/2035782672 --> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559517c25c00 con 0x559517d60b20 2024-01-26T23:29:52.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.181+0000 7f252c8eb700 1 --2- 172.21.15.46:0/2035782672 >> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] conn(0x559517d60b20 0x559517d60ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.181+0000 7f2506ffd700 10 client.4410 ms_handle_connect on v2:172.21.15.46:6834/688086235 2024-01-26T23:29:52.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.181+0000 7f252c8eb700 1 --2- 172.21.15.46:0/2035782672 >> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] conn(0x559517d60b20 0x559517d60ef0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50588f238ad9e3c6 server_cookie=e3a8e6973cd5a84e in_seq=0 out_seq=0 2024-01-26T23:29:52.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.182+0000 7f2506ffd700 1 -- 172.21.15.46:0/2035782672 <== mds.0 v2:172.21.15.46:6834/688086235 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x559517c25c00 con 0x559517d60b20 2024-01-26T23:29:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.182+0000 7f2506ffd700 10 client.4410 handle_command_reply: tid=0 2024-01-26T23:29:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.182+0000 7f2537e52b80 1 client.4410 shutdown 2024-01-26T23:29:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.182+0000 7f2537e52b80 1 -- 172.21.15.46:0/2035782672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d35010 msgr2=0x559517d32e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.182+0000 7f2537e52b80 1 --2- 172.21.15.46:0/2035782672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d35010 0x559517d32e60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f25140058d0 tx=0x7f2514028310 comp rx=0 tx=0).stop 2024-01-26T23:29:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.182+0000 7f2537e52b80 1 -- 172.21.15.46:0/2035782672 shutdown_connections 2024-01-26T23:29:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.182+0000 7f2537e52b80 1 --2- 172.21.15.46:0/2035782672 >> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] conn(0x559517d60b20 0x559517d60ef0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.182+0000 7f2537e52b80 1 --2- 172.21.15.46:0/2035782672 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559517d7d360 0x559517d333a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.182+0000 7f2537e52b80 1 --2- 172.21.15.46:0/2035782672 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559517d345d0 0x559517dde3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.182+0000 7f2506ffd700 0 client.4410 ms_handle_reset on v2:172.21.15.46:6834/688086235 2024-01-26T23:29:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.182+0000 7f2537e52b80 1 --2- 172.21.15.46:0/2035782672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559517d35010 0x559517d32e60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.182+0000 7f2537e52b80 1 -- 172.21.15.46:0/2035782672 >> 172.21.15.46:0/2035782672 conn(0x559517d34a70 msgr2=0x559517d73db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.182+0000 7f2537e52b80 1 -- 172.21.15.46:0/2035782672 shutdown_connections 2024-01-26T23:29:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.182+0000 7f2537e52b80 1 -- 172.21.15.46:0/2035782672 wait complete. 2024-01-26T23:29:52.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.182+0000 7f2537e52b80 20 client.4410 trim_cache size 0 max 16384 2024-01-26T23:29:52.191 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:29:52.191 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-26T23:29:52.191 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-26T23:29:52.191 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4254, 2024-01-26T23:29:52.191 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-26T23:29:52.191 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-26T23:29:52.191 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-01-26T23:29:52.191 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 7.5812897359999996, 2024-01-26T23:29:52.191 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 7, 2024-01-26T23:29:52.191 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 24, 2024-01-26T23:29:52.192 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 24, 2024-01-26T23:29:52.192 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 14.263032339 2024-01-26T23:29:52.192 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:29:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.183+0000 7f252eb4f700 1 -- 172.21.15.46:0/1392596020 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f250c066f30 msgr2=0x7f250c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.183+0000 7f252eb4f700 1 --2- 172.21.15.46:0/1392596020 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f250c066f30 0x7f250c0693e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f251c0093f0 tx=0x7f251c017000 comp rx=0 tx=0).stop 2024-01-26T23:29:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.183+0000 7f252eb4f700 1 -- 172.21.15.46:0/1392596020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2528157690 msgr2=0x7f252813f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.183+0000 7f252eb4f700 1 --2- 172.21.15.46:0/1392596020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2528157690 0x7f252813f290 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f252000c840 tx=0x7f2520001670 comp rx=0 tx=0).stop 2024-01-26T23:29:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.183+0000 7f252eb4f700 1 -- 172.21.15.46:0/1392596020 shutdown_connections 2024-01-26T23:29:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.183+0000 7f252eb4f700 1 --2- 172.21.15.46:0/1392596020 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f250c066f30 0x7f250c0693e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.183+0000 7f252eb4f700 1 --2- 172.21.15.46:0/1392596020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2528157690 0x7f252813f290 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.183+0000 7f252eb4f700 1 --2- 172.21.15.46:0/1392596020 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2528068310 0x7f252815b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.183+0000 7f252eb4f700 1 --2- 172.21.15.46:0/1392596020 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25280b1fc0 0x7f252814f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.183+0000 7f252eb4f700 1 -- 172.21.15.46:0/1392596020 >> 172.21.15.46:0/1392596020 conn(0x7f25280ccbc0 msgr2=0x7f2528156090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.183+0000 7f252eb4f700 1 -- 172.21.15.46:0/1392596020 shutdown_connections 2024-01-26T23:29:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.183+0000 7f252eb4f700 1 -- 172.21.15.46:0/1392596020 wait complete. 2024-01-26T23:29:52.198 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-26T23:29:52.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.327+0000 7f2aa2f85700 1 Processor -- start 2024-01-26T23:29:52.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.328+0000 7f2aa2f85700 1 -- start start 2024-01-26T23:29:52.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.328+0000 7f2aa2f85700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a9c0ae520 0x7f2a9c0b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.328+0000 7f2aa2f85700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2a9c0b3fb0 0x7f2a9c157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.328+0000 7f2aa2f85700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9c158ae0 con 0x7f2a9c157340 2024-01-26T23:29:52.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.328+0000 7f2aa2f85700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a9c15b650 con 0x7f2a9c0ae520 2024-01-26T23:29:52.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.328+0000 7f2aa2f85700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a9c0b43c0 con 0x7f2a9c0b3fb0 2024-01-26T23:29:52.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.329+0000 7f2a9bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a9c0ae520 0x7f2a9c0b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.329+0000 7f2aa0d21700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2a9c157340 0x7f2a9c15bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51148/0 (socket says 172.21.15.46:51148) 2024-01-26T23:29:52.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.329+0000 7f2a9bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a9c0ae520 0x7f2a9c0b3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52232/0 (socket says 172.21.15.46:52232) 2024-01-26T23:29:52.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.329+0000 7f2aa0d21700 1 -- 172.21.15.46:0/3269263424 learned_addr learned my addr 172.21.15.46:0/3269263424 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:29:52.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.329+0000 7f2aa1522700 1 --2- 172.21.15.46:0/3269263424 >> v2:172.21.15.106:3300/0 conn(0x7f2a9c0b3fb0 0x7f2a9c157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.329+0000 7f2a9bfff700 1 -- 172.21.15.46:0/3269263424 >> v2:172.21.15.106:3300/0 conn(0x7f2a9c0b3fb0 msgr2=0x7f2a9c157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:29:52.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.329+0000 7f2a9bfff700 1 --2- 172.21.15.46:0/3269263424 >> v2:172.21.15.106:3300/0 conn(0x7f2a9c0b3fb0 0x7f2a9c157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.329+0000 7f2a9bfff700 1 -- 172.21.15.46:0/3269263424 >> v1:172.21.15.46:6789/0 conn(0x7f2a9c157340 legacy=0x7f2a9c15bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:52.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.329+0000 7f2a9bfff700 1 -- 172.21.15.46:0/3269263424 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c158520 con 0x7f2a9c0ae520 2024-01-26T23:29:52.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.329+0000 7f2a9b7fe700 1 -- 172.21.15.46:0/3269263424 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1781779263 0 0) 0x7f2a9c158ae0 con 0x7f2a9c157340 2024-01-26T23:29:52.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.329+0000 7f2a9bfff700 1 --2- 172.21.15.46:0/3269263424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0ae520 0x7f2a9c0b3a70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0caeb0 tx=0x7f2a840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3dd35d890820f5b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:52.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.329+0000 7f2a9b7fe700 1 -- 172.21.15.46:0/3269263424 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a84011630 con 0x7f2a9c0ae520 2024-01-26T23:29:52.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.329+0000 7f2a9b7fe700 1 -- 172.21.15.46:0/3269263424 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a84011790 con 0x7f2a9c0ae520 2024-01-26T23:29:52.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.329+0000 7f2a9b7fe700 1 -- 172.21.15.46:0/3269263424 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a84011b30 con 0x7f2a9c0ae520 2024-01-26T23:29:52.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.330+0000 7f2aa2f85700 1 -- 172.21.15.46:0/3269263424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0ae520 msgr2=0x7f2a9c0b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:52.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.330+0000 7f2aa2f85700 1 --2- 172.21.15.46:0/3269263424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0ae520 0x7f2a9c0b3a70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0caeb0 tx=0x7f2a840091a0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.330+0000 7f2aa2f85700 1 -- 172.21.15.46:0/3269263424 shutdown_connections 2024-01-26T23:29:52.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.330+0000 7f2aa2f85700 1 --2- 172.21.15.46:0/3269263424 >> v2:172.21.15.106:3300/0 conn(0x7f2a9c0b3fb0 0x7f2a9c157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.330+0000 7f2aa2f85700 1 --2- 172.21.15.46:0/3269263424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0ae520 0x7f2a9c0b3a70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.330+0000 7f2aa2f85700 1 -- 172.21.15.46:0/3269263424 >> 172.21.15.46:0/3269263424 conn(0x7f2a9c0ce530 msgr2=0x7f2a9c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:52.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.330+0000 7f2aa2f85700 1 -- 172.21.15.46:0/3269263424 shutdown_connections 2024-01-26T23:29:52.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.330+0000 7f2aa2f85700 1 -- 172.21.15.46:0/3269263424 wait complete. 2024-01-26T23:29:52.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.331+0000 7f2aa2f85700 1 Processor -- start 2024-01-26T23:29:52.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.331+0000 7f2aa2f85700 1 -- start start 2024-01-26T23:29:52.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.332+0000 7f2aa2f85700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0ae520 0x7f2a9c069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.332+0000 7f2aa2f85700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a9c0b3fb0 0x7f2a9c169ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.332+0000 7f2aa2f85700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a9c157340 0x7f2a9c0b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.332+0000 7f2aa2f85700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c16ac90 con 0x7f2a9c0ae520 2024-01-26T23:29:52.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.332+0000 7f2aa2f85700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c16a1e0 con 0x7f2a9c157340 2024-01-26T23:29:52.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.332+0000 7f2aa2f85700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c16a3f0 con 0x7f2a9c0b3fb0 2024-01-26T23:29:52.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.332+0000 7f2aa0d21700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0ae520 0x7f2a9c069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.332+0000 7f2aa0d21700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0ae520 0x7f2a9c069e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52248/0 (socket says 172.21.15.46:52248) 2024-01-26T23:29:52.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.332+0000 7f2aa0d21700 1 -- 172.21.15.46:0/2788132979 learned_addr learned my addr 172.21.15.46:0/2788132979 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:29:52.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.332+0000 7f2aa1522700 1 --2- 172.21.15.46:0/2788132979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a9c157340 0x7f2a9c0b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.332+0000 7f2a9bfff700 1 --2- 172.21.15.46:0/2788132979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a9c0b3fb0 0x7f2a9c169ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.333+0000 7f2aa0d21700 1 -- 172.21.15.46:0/2788132979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a9c0b3fb0 msgr2=0x7f2a9c169ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:52.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.333+0000 7f2aa0d21700 1 --2- 172.21.15.46:0/2788132979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a9c0b3fb0 0x7f2a9c169ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.333+0000 7f2aa0d21700 1 -- 172.21.15.46:0/2788132979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a9c157340 msgr2=0x7f2a9c0b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:52.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.333+0000 7f2aa0d21700 1 --2- 172.21.15.46:0/2788132979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a9c157340 0x7f2a9c0b31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.333+0000 7f2aa0d21700 1 -- 172.21.15.46:0/2788132979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a84009420 con 0x7f2a9c0ae520 2024-01-26T23:29:52.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.333+0000 7f2aa0d21700 1 --2- 172.21.15.46:0/2788132979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0ae520 0x7f2a9c069e30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00c7e0 tx=0x7f2a8c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:52.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.333+0000 7f2a98ff9700 1 -- 172.21.15.46:0/2788132979 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c005b30 con 0x7f2a9c0ae520 2024-01-26T23:29:52.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.333+0000 7f2a98ff9700 1 -- 172.21.15.46:0/2788132979 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8c005c90 con 0x7f2a9c0ae520 2024-01-26T23:29:52.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.333+0000 7f2aa2f85700 1 -- 172.21.15.46:0/2788132979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a9c16a6d0 con 0x7f2a9c0ae520 2024-01-26T23:29:52.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.333+0000 7f2a98ff9700 1 -- 172.21.15.46:0/2788132979 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c007300 con 0x7f2a9c0ae520 2024-01-26T23:29:52.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.333+0000 7f2aa2f85700 1 -- 172.21.15.46:0/2788132979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a9c16d420 con 0x7f2a9c0ae520 2024-01-26T23:29:52.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.334+0000 7f2aa2f85700 1 -- 172.21.15.46:0/2788132979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a9c16d280 con 0x7f2a9c0ae520 2024-01-26T23:29:52.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.334+0000 7f2a98ff9700 1 -- 172.21.15.46:0/2788132979 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2a8c01b3b0 con 0x7f2a9c0ae520 2024-01-26T23:29:52.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.335+0000 7f2a98ff9700 1 --2- 172.21.15.46:0/2788132979 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a7c066f00 0x7f2a7c0693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:29:52.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.335+0000 7f2a98ff9700 1 -- 172.21.15.46:0/2788132979 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f2a8c083d40 con 0x7f2a9c0ae520 2024-01-26T23:29:52.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.335+0000 7f2a9bfff700 1 --2- 172.21.15.46:0/2788132979 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a7c066f00 0x7f2a7c0693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:29:52.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.335+0000 7f2a9bfff700 1 --2- 172.21.15.46:0/2788132979 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a7c066f00 0x7f2a7c0693b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2a840093f0 tx=0x7f2a84017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:29:52.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.339+0000 7f2a98ff9700 1 -- 172.21.15.46:0/2788132979 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2a9c16d280 con 0x7f2a9c0ae520 2024-01-26T23:29:52.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.594+0000 7f2aa2f85700 1 -- 172.21.15.46:0/2788132979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2a9c16d280 con 0x7f2a9c0ae520 2024-01-26T23:29:52.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.594+0000 7f2a98ff9700 1 -- 172.21.15.46:0/2788132979 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7f2a9c16d280 con 0x7f2a9c0ae520 2024-01-26T23:29:52.596 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-01-26T23:29:52.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.596+0000 7f2aa2f85700 1 -- 172.21.15.46:0/2788132979 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a7c066f00 msgr2=0x7f2a7c0693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:52.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.596+0000 7f2aa2f85700 1 --2- 172.21.15.46:0/2788132979 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a7c066f00 0x7f2a7c0693b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2a840093f0 tx=0x7f2a84017000 comp rx=0 tx=0).stop 2024-01-26T23:29:52.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.596+0000 7f2aa2f85700 1 -- 172.21.15.46:0/2788132979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0ae520 msgr2=0x7f2a9c069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:29:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.596+0000 7f2aa2f85700 1 --2- 172.21.15.46:0/2788132979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0ae520 0x7f2a9c069e30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00c7e0 tx=0x7f2a8c007df0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.596+0000 7f2aa2f85700 1 -- 172.21.15.46:0/2788132979 shutdown_connections 2024-01-26T23:29:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.596+0000 7f2aa2f85700 1 --2- 172.21.15.46:0/2788132979 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a7c066f00 0x7f2a7c0693b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.596+0000 7f2aa2f85700 1 --2- 172.21.15.46:0/2788132979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0ae520 0x7f2a9c069e30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.596+0000 7f2aa2f85700 1 --2- 172.21.15.46:0/2788132979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a9c157340 0x7f2a9c0b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.596+0000 7f2aa2f85700 1 --2- 172.21.15.46:0/2788132979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a9c0b3fb0 0x7f2a9c169ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:29:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.596+0000 7f2aa2f85700 1 -- 172.21.15.46:0/2788132979 >> 172.21.15.46:0/2788132979 conn(0x7f2a9c0ce530 msgr2=0x7f2a9c156a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:29:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.596+0000 7f2aa2f85700 1 -- 172.21.15.46:0/2788132979 shutdown_connections 2024-01-26T23:29:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:29:52.596+0000 7f2aa2f85700 1 -- 172.21.15.46:0/2788132979 wait complete. 2024-01-26T23:29:52.611 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-26T23:29:52.612 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-26T23:29:52.612 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-26T23:29:52.612 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-01-26T23:29:52.612 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-01-26T23:29:52.612 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-26T23:29:52.612 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-26T23:29:52.612 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-26T23:29:52.612 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-01-26T23:29:52.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:29:52.612 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-26T23:29:52.632 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-26T23:29:52.632 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-26T23:29:52.633 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-26T23:29:52.633 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:29:52.633 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-26T23:29:52.633 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:29:52.633 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:29:52.633 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:29:52.633 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-26T23:29:52.633 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:29:52.633 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:29:52.633 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-26T23:29:52.633 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-26T23:29:52.634 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:29:52.634 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-26T23:29:52.634 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-26T23:29:52.634 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42338sec preferred_lft 42338sec 2024-01-26T23:29:52.634 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-26T23:29:52.634 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:29:52.634 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:29:52.634 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:29:52.634 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-26T23:29:52.635 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-26T23:29:52.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:29:52.635 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:29:52.635 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:29:52.635 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add name ceph-brx type bridge 2024-01-26T23:29:52.635 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr flush dev ceph-brx 2024-01-26T23:29:52.635 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set ceph-brx up 2024-01-26T23:29:52.635 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-26T23:29:52.635 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:29:52.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:29:52.828 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-26T23:29:52.861 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-26T23:29:52.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:29:52.863 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-26T23:29:52.939 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-26T23:29:52.939 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-26T23:29:52.939 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-26T23:29:52.939 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-26T23:29:52.939 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-26T23:29:52.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:29:52.940 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:29:52.940 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:29:52.940 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-26T23:29:52.940 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-26T23:29:52.940 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-26T23:29:52.940 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:29:53.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:29:53.241 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-26T23:29:53.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:29:53.263 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list-id 2024-01-26T23:29:53.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:29:53.324 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:29:53.325 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:29:53.325 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-01-26T23:29:53.325 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-01-26T23:29:53.325 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:29:53.436 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-01-26T23:29:53.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:29:53.436 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:29:53.436 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:29:53.436 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-01-26T23:29:53.437 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-26T23:29:53.437 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-01-26T23:29:53.437 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-01-26T23:29:53.437 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-01-26T23:29:53.437 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:29:53.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:29:53.700 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:29:53.700 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:29:53.700 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-01-26T23:29:53.700 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set dev brx.0 master ceph-brx 2024-01-26T23:29:53.700 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:29:53.814 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-01-26T23:29:53.814 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:29:53.814 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-01-26T23:29:53.833 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-01-26T23:29:53.833 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:29:53.834 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-01-26T23:29:53.894 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:29:53.970 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:29:53.970 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:29:54.045 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:29:54.046 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:29:54.047 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:29:54.047 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:29:54.109 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-26T23:29:54.109 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-01-26T23:29:54.152 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:29:54.187 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:29:54.188 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:29:54.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi046.stderr:2024-01-26T23:29:54.258+0000 7fd54c507540 -1 init, newargv = 0x55c16a801730 newargc=15 2024-01-26T23:29:54.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi046.stderr:ceph-fuse[50720]: starting ceph client 2024-01-26T23:29:54.261 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:29:54.262 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:29:54.263 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:29:54.263 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:29:54.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi046.stderr:ceph-fuse[50720]: starting fuse 2024-01-26T23:29:54.286 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-26T23:29:54.286 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-26T23:29:54.286 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-26T23:29:54.286 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-26T23:29:54.286 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-26T23:29:54.287 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-26T23:29:54.287 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-26T23:29:54.287 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:29:54.287 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-26T23:29:54.287 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-01-26T23:29:54.287 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-26T23:29:54.287 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-01-26T23:29:54.287 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:29:54.287 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-26T23:29:54.287 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-26T23:29:54.287 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-26T23:29:54.287 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-26T23:29:54.287 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:29:54.287 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-26T23:29:54.287 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-26T23:29:54.287 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-26T23:29:54.288 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-26T23:29:54.288 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-26T23:29:54.288 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-26T23:29:54.288 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-26T23:29:54.288 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-26T23:29:54.288 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:29:54.288 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.admin")) 2024-01-26T23:29:54.288 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-26T23:29:54.424 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.admin.50720.asok 2024-01-26T23:29:54.606 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.50720.asok 2024-01-26T23:29:54.606 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:29:54.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.50720.asok status 2024-01-26T23:29:54.781 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:29:54.781 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-26T23:29:54.781 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-26T23:29:54.781 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-26T23:29:54.781 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "admin", 2024-01-26T23:29:54.781 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-26T23:29:54.781 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-01-26T23:29:54.782 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "50720", 2024-01-26T23:29:54.782 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-26T23:29:54.782 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:29:54.782 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-26T23:29:54.782 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-26T23:29:54.782 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4425, 2024-01-26T23:29:54.782 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-26T23:29:54.782 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-26T23:29:54.782 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-26T23:29:54.782 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4425 2024-01-26T23:29:54.782 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:29:54.782 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:29:54.782 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:29:54.782 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:29:54.783 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2743477527 2024-01-26T23:29:54.783 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:29:54.783 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:29:54.783 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:29:54.783 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:29:54.783 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:29:54.783 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2743477527 2024-01-26T23:29:54.783 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:29:54.783 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.4425 192.168.0.1:0/2743477527", 2024-01-26T23:29:54.783 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/2743477527", 2024-01-26T23:29:54.783 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-26T23:29:54.783 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 7, 2024-01-26T23:29:54.783 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 24, 2024-01-26T23:29:54.783 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-26T23:29:54.783 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-26T23:29:54.784 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-26T23:29:54.784 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:29:54.792 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:29:54.792 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-26T23:29:54.792 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-01-26T23:29:54.812 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-26T23:29:54.812 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-01-26T23:29:54.812 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:29:54.812 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-01-26T23:29:54.886 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:29:54.886 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-01-26T23:29:54.996 INFO:teuthology.orchestra.run.smithi046.stdout:# file: . 2024-01-26T23:29:54.996 INFO:teuthology.orchestra.run.smithi046.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-01-26T23:29:54.996 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:29:54.996 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-26T23:29:54.996 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-26T23:29:54.996 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-01-26T23:29:55.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi046.stderr:ceph-fuse[50720]: fuse finished with error 0 and tester_r 0 2024-01-26T23:29:55.048 INFO:teuthology.orchestra.run:waiting for 900 2024-01-26T23:30:01.150 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:30:01.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:01.150 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-01-26T23:30:01.178 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:01.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.362+0000 7f6d549b8700 1 Processor -- start 2024-01-26T23:30:01.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.362+0000 7f6d549b8700 1 -- start start 2024-01-26T23:30:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.363+0000 7f6d549b8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6d500cbdf0 0x7f6d500ae610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.363+0000 7f6d549b8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d500ad070 0x7f6d500c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.363+0000 7f6d549b8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d500ae180 con 0x7f6d500cc1c0 2024-01-26T23:30:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.363+0000 7f6d549b8700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d500aeb50 con 0x7f6d500cbdf0 2024-01-26T23:30:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.363+0000 7f6d549b8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d50156c60 con 0x7f6d500ad070 2024-01-26T23:30:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.363+0000 7f6d4e59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6d500cbdf0 0x7f6d500ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.363+0000 7f6d4dd9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6d500cc1c0 0x7f6d500b0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44356/0 (socket says 172.21.15.46:44356) 2024-01-26T23:30:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.363+0000 7f6d4dd9b700 1 -- 172.21.15.46:0/1592347262 learned_addr learned my addr 172.21.15.46:0/1592347262 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.363+0000 7f6d4ed9d700 1 --2- 172.21.15.46:0/1592347262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d500ad070 0x7f6d500c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.363+0000 7f6d4e59c700 1 -- 172.21.15.46:0/1592347262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d500ad070 msgr2=0x7f6d500c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.363+0000 7f6d4e59c700 1 --2- 172.21.15.46:0/1592347262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d500ad070 0x7f6d500c98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.364+0000 7f6d4e59c700 1 -- 172.21.15.46:0/1592347262 >> v1:172.21.15.46:6789/0 conn(0x7f6d500cc1c0 legacy=0x7f6d500b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.364+0000 7f6d4e59c700 1 -- 172.21.15.46:0/1592347262 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d500ad570 con 0x7f6d500cbdf0 2024-01-26T23:30:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.363+0000 7f6d4d59a700 1 -- 172.21.15.46:0/1592347262 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214527578 0 0) 0x7f6d500ae180 con 0x7f6d500cc1c0 2024-01-26T23:30:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.364+0000 7f6d4e59c700 1 --2- 172.21.15.46:0/1592347262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d500cbdf0 0x7f6d500ae610 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6d400098e0 tx=0x7f6d400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea8378db1119066 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:01.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.364+0000 7f6d4d59a700 1 -- 172.21.15.46:0/1592347262 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4000b640 con 0x7f6d500cbdf0 2024-01-26T23:30:01.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.364+0000 7f6d4d59a700 1 -- 172.21.15.46:0/1592347262 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4000b7a0 con 0x7f6d500cbdf0 2024-01-26T23:30:01.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.364+0000 7f6d4d59a700 1 -- 172.21.15.46:0/1592347262 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4000bb40 con 0x7f6d500cbdf0 2024-01-26T23:30:01.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.364+0000 7f6d549b8700 1 -- 172.21.15.46:0/1592347262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d500cbdf0 msgr2=0x7f6d500ae610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:01.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.364+0000 7f6d549b8700 1 --2- 172.21.15.46:0/1592347262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d500cbdf0 0x7f6d500ae610 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6d400098e0 tx=0x7f6d400091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.365+0000 7f6d549b8700 1 -- 172.21.15.46:0/1592347262 shutdown_connections 2024-01-26T23:30:01.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.365+0000 7f6d549b8700 1 --2- 172.21.15.46:0/1592347262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d500ad070 0x7f6d500c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.365+0000 7f6d549b8700 1 --2- 172.21.15.46:0/1592347262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d500cbdf0 0x7f6d500ae610 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.365+0000 7f6d549b8700 1 -- 172.21.15.46:0/1592347262 >> 172.21.15.46:0/1592347262 conn(0x7f6d50068400 msgr2=0x7f6d500cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:01.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.365+0000 7f6d549b8700 1 -- 172.21.15.46:0/1592347262 shutdown_connections 2024-01-26T23:30:01.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.365+0000 7f6d549b8700 1 -- 172.21.15.46:0/1592347262 wait complete. 2024-01-26T23:30:01.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.366+0000 7f6d549b8700 1 Processor -- start 2024-01-26T23:30:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.366+0000 7f6d549b8700 1 -- start start 2024-01-26T23:30:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.366+0000 7f6d549b8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d500ad070 0x7f6d501542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d549b8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d500cbdf0 0x7f6d500ab210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d549b8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d500ab750 0x7f6d500a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d549b8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d500abbf0 con 0x7f6d500ad070 2024-01-26T23:30:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d549b8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d500a74a0 con 0x7f6d500ab750 2024-01-26T23:30:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d549b8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d500a7780 con 0x7f6d500cbdf0 2024-01-26T23:30:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d4e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d500ad070 0x7f6d501542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d4e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d500ad070 0x7f6d501542a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33630/0 (socket says 172.21.15.46:33630) 2024-01-26T23:30:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d4e59c700 1 -- 172.21.15.46:0/1418738806 learned_addr learned my addr 172.21.15.46:0/1418738806 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d4dd9b700 1 --2- 172.21.15.46:0/1418738806 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d500cbdf0 0x7f6d500ab210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d4ed9d700 1 --2- 172.21.15.46:0/1418738806 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d500ab750 0x7f6d500a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d4e59c700 1 -- 172.21.15.46:0/1418738806 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d500cbdf0 msgr2=0x7f6d500ab210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d4e59c700 1 --2- 172.21.15.46:0/1418738806 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d500cbdf0 0x7f6d500ab210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d4e59c700 1 -- 172.21.15.46:0/1418738806 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d500ab750 msgr2=0x7f6d500a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d4e59c700 1 --2- 172.21.15.46:0/1418738806 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d500ab750 0x7f6d500a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d4e59c700 1 -- 172.21.15.46:0/1418738806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d40009420 con 0x7f6d500ad070 2024-01-26T23:30:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d4e59c700 1 --2- 172.21.15.46:0/1418738806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d500ad070 0x7f6d501542a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6d40009f80 tx=0x7f6d40017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d3e7fc700 1 -- 172.21.15.46:0/1418738806 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4000bb20 con 0x7f6d500ad070 2024-01-26T23:30:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d3e7fc700 1 -- 172.21.15.46:0/1418738806 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d40011420 con 0x7f6d500ad070 2024-01-26T23:30:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d3e7fc700 1 -- 172.21.15.46:0/1418738806 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d40011800 con 0x7f6d500ad070 2024-01-26T23:30:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d549b8700 1 -- 172.21.15.46:0/1418738806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d500a7a60 con 0x7f6d500ad070 2024-01-26T23:30:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.367+0000 7f6d549b8700 1 -- 172.21.15.46:0/1418738806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d500a6cb0 con 0x7f6d500ad070 2024-01-26T23:30:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.368+0000 7f6d549b8700 1 -- 172.21.15.46:0/1418738806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d5004c710 con 0x7f6d500ad070 2024-01-26T23:30:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.369+0000 7f6d3e7fc700 1 -- 172.21.15.46:0/1418738806 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6d40011c30 con 0x7f6d500ad070 2024-01-26T23:30:01.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.369+0000 7f6d3e7fc700 1 --2- 172.21.15.46:0/1418738806 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6d30066f30 0x7f6d300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:01.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.370+0000 7f6d3e7fc700 1 -- 172.21.15.46:0/1418738806 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f6d40086370 con 0x7f6d500ad070 2024-01-26T23:30:01.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.370+0000 7f6d4dd9b700 1 --2- 172.21.15.46:0/1418738806 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6d30066f30 0x7f6d300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:01.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.370+0000 7f6d4dd9b700 1 --2- 172.21.15.46:0/1418738806 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6d30066f30 0x7f6d300693e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6d500ac170 tx=0x7f6d44009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:01.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.374+0000 7f6d3e7fc700 1 -- 172.21.15.46:0/1418738806 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6d400532f0 con 0x7f6d500ad070 2024-01-26T23:30:01.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.581+0000 7f6d549b8700 1 -- 172.21.15.46:0/1418738806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d500a7f00 con 0x7f6d500ad070 2024-01-26T23:30:01.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.582+0000 7f6d3e7fc700 1 -- 172.21.15.46:0/1418738806 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7f6d40056910 con 0x7f6d500ad070 2024-01-26T23:30:01.583 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:01.583 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:49.075104+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.584+0000 7f6d549b8700 1 -- 172.21.15.46:0/1418738806 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6d30066f30 msgr2=0x7f6d300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.584+0000 7f6d549b8700 1 --2- 172.21.15.46:0/1418738806 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6d30066f30 0x7f6d300693e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6d500ac170 tx=0x7f6d44009040 comp rx=0 tx=0).stop 2024-01-26T23:30:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.584+0000 7f6d549b8700 1 -- 172.21.15.46:0/1418738806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d500ad070 msgr2=0x7f6d501542a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.584+0000 7f6d549b8700 1 --2- 172.21.15.46:0/1418738806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d500ad070 0x7f6d501542a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6d40009f80 tx=0x7f6d40017000 comp rx=0 tx=0).stop 2024-01-26T23:30:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.584+0000 7f6d549b8700 1 -- 172.21.15.46:0/1418738806 shutdown_connections 2024-01-26T23:30:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.584+0000 7f6d549b8700 1 --2- 172.21.15.46:0/1418738806 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6d30066f30 0x7f6d300693e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.584+0000 7f6d549b8700 1 --2- 172.21.15.46:0/1418738806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d500ad070 0x7f6d501542a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.584+0000 7f6d549b8700 1 --2- 172.21.15.46:0/1418738806 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d500ab750 0x7f6d500a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.584+0000 7f6d549b8700 1 --2- 172.21.15.46:0/1418738806 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d500cbdf0 0x7f6d500ab210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.584+0000 7f6d549b8700 1 -- 172.21.15.46:0/1418738806 >> 172.21.15.46:0/1418738806 conn(0x7f6d50068400 msgr2=0x7f6d500b13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:01.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.585+0000 7f6d549b8700 1 -- 172.21.15.46:0/1418738806 shutdown_connections 2024-01-26T23:30:01.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.585+0000 7f6d549b8700 1 -- 172.21.15.46:0/1418738806 wait complete. 2024-01-26T23:30:01.599 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.731+0000 7fcdeddf2700 1 Processor -- start 2024-01-26T23:30:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.731+0000 7fcdeddf2700 1 -- start start 2024-01-26T23:30:01.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.732+0000 7fcdeddf2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fcde80cc1c0 0x7fcde80ae610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.732+0000 7fcdeddf2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcde80ad070 0x7fcde80c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.732+0000 7fcdeddf2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde80ae180 con 0x7fcde80cbdf0 2024-01-26T23:30:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.732+0000 7fcdeddf2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcde80aeb50 con 0x7fcde80ad070 2024-01-26T23:30:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.732+0000 7fcdeddf2700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcde8156c60 con 0x7fcde80cc1c0 2024-01-26T23:30:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.732+0000 7fcde7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcde80ad070 0x7fcde80c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.732+0000 7fcde7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcde80ad070 0x7fcde80c98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33640/0 (socket says 172.21.15.46:33640) 2024-01-26T23:30:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.732+0000 7fcde7fff700 1 -- 172.21.15.46:0/431933633 learned_addr learned my addr 172.21.15.46:0/431933633 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.732+0000 7fcde6ffd700 1 --2- 172.21.15.46:0/431933633 >> v2:172.21.15.106:3300/0 conn(0x7fcde80cc1c0 0x7fcde80ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.732+0000 7fcde7fff700 1 -- 172.21.15.46:0/431933633 >> v2:172.21.15.106:3300/0 conn(0x7fcde80cc1c0 msgr2=0x7fcde80ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.732+0000 7fcde7fff700 1 --2- 172.21.15.46:0/431933633 >> v2:172.21.15.106:3300/0 conn(0x7fcde80cc1c0 0x7fcde80ae610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.732+0000 7fcde7fff700 1 -- 172.21.15.46:0/431933633 >> v1:172.21.15.106:6789/0 conn(0x7fcde80cbdf0 legacy=0x7fcde80b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.732+0000 7fcde7fff700 1 -- 172.21.15.46:0/431933633 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde80ad570 con 0x7fcde80ad070 2024-01-26T23:30:01.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.733+0000 7fcde7fff700 1 --2- 172.21.15.46:0/431933633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde80ad070 0x7fcde80c98b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcde806f030 tx=0x7fcde0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28649204a3cb2f88 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:01.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.733+0000 7fcde67fc700 1 -- 172.21.15.46:0/431933633 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde0009e70 con 0x7fcde80ad070 2024-01-26T23:30:01.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.733+0000 7fcde67fc700 1 -- 172.21.15.46:0/431933633 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde000b860 con 0x7fcde80ad070 2024-01-26T23:30:01.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.733+0000 7fcde67fc700 1 -- 172.21.15.46:0/431933633 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde000bc00 con 0x7fcde80ad070 2024-01-26T23:30:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.733+0000 7fcdeddf2700 1 -- 172.21.15.46:0/431933633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde80ad070 msgr2=0x7fcde80c98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.733+0000 7fcdeddf2700 1 --2- 172.21.15.46:0/431933633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde80ad070 0x7fcde80c98b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcde806f030 tx=0x7fcde0009250 comp rx=0 tx=0).stop 2024-01-26T23:30:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.733+0000 7fcdeddf2700 1 -- 172.21.15.46:0/431933633 shutdown_connections 2024-01-26T23:30:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.733+0000 7fcdeddf2700 1 --2- 172.21.15.46:0/431933633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde80ad070 0x7fcde80c98b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.733+0000 7fcdeddf2700 1 --2- 172.21.15.46:0/431933633 >> v2:172.21.15.106:3300/0 conn(0x7fcde80cc1c0 0x7fcde80ae610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.733+0000 7fcdeddf2700 1 -- 172.21.15.46:0/431933633 >> 172.21.15.46:0/431933633 conn(0x7fcde8068400 msgr2=0x7fcde80cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.733+0000 7fcdeddf2700 1 -- 172.21.15.46:0/431933633 shutdown_connections 2024-01-26T23:30:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.734+0000 7fcdeddf2700 1 -- 172.21.15.46:0/431933633 wait complete. 2024-01-26T23:30:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.735+0000 7fcdeddf2700 1 Processor -- start 2024-01-26T23:30:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.735+0000 7fcdeddf2700 1 -- start start 2024-01-26T23:30:01.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.736+0000 7fcdeddf2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcde80ad070 0x7fcde80a91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:01.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.736+0000 7fcdeddf2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde80cbdf0 0x7fcde80ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:01.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.736+0000 7fcdeddf2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcde80c1340 0x7fcde814b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:01.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.736+0000 7fcdeddf2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80c1b50 con 0x7fcde80cbdf0 2024-01-26T23:30:01.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.736+0000 7fcdeddf2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80a9710 con 0x7fcde80ad070 2024-01-26T23:30:01.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.736+0000 7fcdeddf2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80c1870 con 0x7fcde80c1340 2024-01-26T23:30:01.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.737+0000 7fcde6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde80cbdf0 0x7fcde80ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:01.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.737+0000 7fcde6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde80cbdf0 0x7fcde80ac1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33644/0 (socket says 172.21.15.46:33644) 2024-01-26T23:30:01.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.737+0000 7fcde6ffd700 1 -- 172.21.15.46:0/2998325916 learned_addr learned my addr 172.21.15.46:0/2998325916 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:01.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.737+0000 7fcde77fe700 1 --2- 172.21.15.46:0/2998325916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcde80ad070 0x7fcde80a91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:01.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.737+0000 7fcde7fff700 1 --2- 172.21.15.46:0/2998325916 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcde80c1340 0x7fcde814b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:01.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.737+0000 7fcde6ffd700 1 -- 172.21.15.46:0/2998325916 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcde80c1340 msgr2=0x7fcde814b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:01.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.737+0000 7fcde6ffd700 1 --2- 172.21.15.46:0/2998325916 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcde80c1340 0x7fcde814b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.737+0000 7fcde6ffd700 1 -- 172.21.15.46:0/2998325916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcde80ad070 msgr2=0x7fcde80a91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:01.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.737+0000 7fcde6ffd700 1 --2- 172.21.15.46:0/2998325916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcde80ad070 0x7fcde80a91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.737+0000 7fcde6ffd700 1 -- 172.21.15.46:0/2998325916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde00094f0 con 0x7fcde80cbdf0 2024-01-26T23:30:01.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.737+0000 7fcde6ffd700 1 --2- 172.21.15.46:0/2998325916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde80cbdf0 0x7fcde80ac1b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcddc00e930 tx=0x7fcddc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:01.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.737+0000 7fcdcf7fe700 1 -- 172.21.15.46:0/2998325916 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc013680 con 0x7fcde80cbdf0 2024-01-26T23:30:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.737+0000 7fcdcf7fe700 1 -- 172.21.15.46:0/2998325916 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcddc0137e0 con 0x7fcde80cbdf0 2024-01-26T23:30:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.737+0000 7fcdeddf2700 1 -- 172.21.15.46:0/2998325916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde80a9b00 con 0x7fcde80cbdf0 2024-01-26T23:30:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.737+0000 7fcdcf7fe700 1 -- 172.21.15.46:0/2998325916 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc013b80 con 0x7fcde80cbdf0 2024-01-26T23:30:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.737+0000 7fcdeddf2700 1 -- 172.21.15.46:0/2998325916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde80aa0c0 con 0x7fcde80cbdf0 2024-01-26T23:30:01.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.738+0000 7fcdeddf2700 1 -- 172.21.15.46:0/2998325916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde80a9960 con 0x7fcde80cbdf0 2024-01-26T23:30:01.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.739+0000 7fcdcf7fe700 1 -- 172.21.15.46:0/2998325916 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fcddc00c3e0 con 0x7fcde80cbdf0 2024-01-26T23:30:01.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.739+0000 7fcdcf7fe700 1 --2- 172.21.15.46:0/2998325916 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcdd4066f30 0x7fcdd40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:01.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.739+0000 7fcdcf7fe700 1 -- 172.21.15.46:0/2998325916 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fcddc084cb0 con 0x7fcde80cbdf0 2024-01-26T23:30:01.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.739+0000 7fcde77fe700 1 --2- 172.21.15.46:0/2998325916 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcdd4066f30 0x7fcdd40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:01.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.740+0000 7fcde77fe700 1 --2- 172.21.15.46:0/2998325916 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcdd4066f30 0x7fcdd40693e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8007fd0 tx=0x7fcdd80016f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:01.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.744+0000 7fcdcf7fe700 1 -- 172.21.15.46:0/2998325916 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcde80a9960 con 0x7fcde80cbdf0 2024-01-26T23:30:01.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.950+0000 7fcdeddf2700 1 -- 172.21.15.46:0/2998325916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcde80a9960 con 0x7fcde80cbdf0 2024-01-26T23:30:01.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.951+0000 7fcdcf7fe700 1 -- 172.21.15.46:0/2998325916 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7fcde80a9960 con 0x7fcde80cbdf0 2024-01-26T23:30:01.952 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:01.952 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:49.075104+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:01.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.953+0000 7fcdeddf2700 1 -- 172.21.15.46:0/2998325916 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcdd4066f30 msgr2=0x7fcdd40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:01.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.953+0000 7fcdeddf2700 1 --2- 172.21.15.46:0/2998325916 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcdd4066f30 0x7fcdd40693e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8007fd0 tx=0x7fcdd80016f0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.953+0000 7fcdeddf2700 1 -- 172.21.15.46:0/2998325916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde80cbdf0 msgr2=0x7fcde80ac1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:01.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.953+0000 7fcdeddf2700 1 --2- 172.21.15.46:0/2998325916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde80cbdf0 0x7fcde80ac1b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcddc00e930 tx=0x7fcddc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.953+0000 7fcdeddf2700 1 -- 172.21.15.46:0/2998325916 shutdown_connections 2024-01-26T23:30:01.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.953+0000 7fcdeddf2700 1 --2- 172.21.15.46:0/2998325916 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcdd4066f30 0x7fcdd40693e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.953+0000 7fcdeddf2700 1 --2- 172.21.15.46:0/2998325916 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcde80c1340 0x7fcde814b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.953+0000 7fcdeddf2700 1 --2- 172.21.15.46:0/2998325916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcde80ad070 0x7fcde80a91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.953+0000 7fcdeddf2700 1 --2- 172.21.15.46:0/2998325916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde80cbdf0 0x7fcde80ac1b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:01.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.953+0000 7fcdeddf2700 1 -- 172.21.15.46:0/2998325916 >> 172.21.15.46:0/2998325916 conn(0x7fcde8068400 msgr2=0x7fcde8158e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:01.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.953+0000 7fcdeddf2700 1 -- 172.21.15.46:0/2998325916 shutdown_connections 2024-01-26T23:30:01.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:01.953+0000 7fcdeddf2700 1 -- 172.21.15.46:0/2998325916 wait complete. 2024-01-26T23:30:01.967 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-26T23:30:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.102+0000 7f2d7d5c6700 1 Processor -- start 2024-01-26T23:30:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.102+0000 7f2d7d5c6700 1 -- start start 2024-01-26T23:30:02.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.103+0000 7f2d7d5c6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d78159da0 0x7f2d780ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:02.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.103+0000 7f2d7d5c6700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7815a200 con 0x7f2d7815d8c0 2024-01-26T23:30:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.103+0000 7f2d7d5c6700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d780b1a60 con 0x7f2d781599d0 2024-01-26T23:30:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.103+0000 7f2d7d5c6700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d780b11c0 con 0x7f2d78159da0 2024-01-26T23:30:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.103+0000 7f2d767fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d78159da0 0x7f2d780ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.103+0000 7f2d777fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2d7815d8c0 0x7f2d780b3b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44368/0 (socket says 172.21.15.46:44368) 2024-01-26T23:30:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.103+0000 7f2d777fe700 1 -- 172.21.15.46:0/763220432 learned_addr learned my addr 172.21.15.46:0/763220432 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.103+0000 7f2d767fc700 1 -- 172.21.15.46:0/763220432 >> v1:172.21.15.106:6789/0 conn(0x7f2d781599d0 legacy=0x7f2d780b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.103+0000 7f2d767fc700 1 -- 172.21.15.46:0/763220432 >> v1:172.21.15.46:6789/0 conn(0x7f2d7815d8c0 legacy=0x7f2d780b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.103+0000 7f2d767fc700 1 -- 172.21.15.46:0/763220432 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d780b14a0 con 0x7f2d78159da0 2024-01-26T23:30:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.103+0000 7f2d75ffb700 1 -- 172.21.15.46:0/763220432 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287238422 0 0) 0x7f2d7815a200 con 0x7f2d7815d8c0 2024-01-26T23:30:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.104+0000 7f2d767fc700 1 --2- 172.21.15.46:0/763220432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d78159da0 0x7f2d780ab1c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2d600098e0 tx=0x7f2d600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad63045f49de9f80 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.104+0000 7f2d75ffb700 1 -- 172.21.15.46:0/763220432 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d60009e80 con 0x7f2d78159da0 2024-01-26T23:30:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.104+0000 7f2d75ffb700 1 -- 172.21.15.46:0/763220432 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6000b860 con 0x7f2d78159da0 2024-01-26T23:30:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.104+0000 7f2d75ffb700 1 -- 172.21.15.46:0/763220432 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6000bbe0 con 0x7f2d78159da0 2024-01-26T23:30:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.104+0000 7f2d7d5c6700 1 -- 172.21.15.46:0/763220432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d78159da0 msgr2=0x7f2d780ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.104+0000 7f2d7d5c6700 1 --2- 172.21.15.46:0/763220432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d78159da0 0x7f2d780ab1c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2d600098e0 tx=0x7f2d600091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.104+0000 7f2d7d5c6700 1 -- 172.21.15.46:0/763220432 shutdown_connections 2024-01-26T23:30:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.104+0000 7f2d7d5c6700 1 --2- 172.21.15.46:0/763220432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d78159da0 0x7f2d780ab1c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.104+0000 7f2d7d5c6700 1 -- 172.21.15.46:0/763220432 >> 172.21.15.46:0/763220432 conn(0x7f2d7806f840 msgr2=0x7f2d78087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.104+0000 7f2d7d5c6700 1 -- 172.21.15.46:0/763220432 shutdown_connections 2024-01-26T23:30:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.104+0000 7f2d7d5c6700 1 -- 172.21.15.46:0/763220432 wait complete. 2024-01-26T23:30:02.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.106+0000 7f2d7d5c6700 1 Processor -- start 2024-01-26T23:30:02.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.106+0000 7f2d7d5c6700 1 -- start start 2024-01-26T23:30:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d7d5c6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d781599d0 0x7f2d7815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d7d5c6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2d78159da0 0x7f2d780ac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d7d5c6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2d7815d8c0 0x7f2d780aba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d7d5c6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d780af620 con 0x7f2d781599d0 2024-01-26T23:30:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d7d5c6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7815bef0 con 0x7f2d7815d8c0 2024-01-26T23:30:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d7d5c6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7815c1d0 con 0x7f2d78159da0 2024-01-26T23:30:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d76ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d781599d0 0x7f2d7815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d76ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d781599d0 0x7f2d7815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33654/0 (socket says 172.21.15.46:33654) 2024-01-26T23:30:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d76ffd700 1 -- 172.21.15.46:0/344405877 learned_addr learned my addr 172.21.15.46:0/344405877 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d767fc700 1 --2- 172.21.15.46:0/344405877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2d78159da0 0x7f2d780ac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d777fe700 1 --2- 172.21.15.46:0/344405877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2d7815d8c0 0x7f2d780aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d76ffd700 1 -- 172.21.15.46:0/344405877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2d78159da0 msgr2=0x7f2d780ac6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d76ffd700 1 --2- 172.21.15.46:0/344405877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2d78159da0 0x7f2d780ac6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d76ffd700 1 -- 172.21.15.46:0/344405877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2d7815d8c0 msgr2=0x7f2d780aba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d76ffd700 1 --2- 172.21.15.46:0/344405877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2d7815d8c0 0x7f2d780aba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d76ffd700 1 -- 172.21.15.46:0/344405877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d60009420 con 0x7f2d781599d0 2024-01-26T23:30:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d76ffd700 1 --2- 172.21.15.46:0/344405877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d781599d0 0x7f2d7815cf90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800c7c0 tx=0x7f2d68007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d6effd700 1 -- 172.21.15.46:0/344405877 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d68011590 con 0x7f2d781599d0 2024-01-26T23:30:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d6effd700 1 -- 172.21.15.46:0/344405877 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d68011710 con 0x7f2d781599d0 2024-01-26T23:30:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d7d5c6700 1 -- 172.21.15.46:0/344405877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7815c4b0 con 0x7f2d781599d0 2024-01-26T23:30:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.107+0000 7f2d6effd700 1 -- 172.21.15.46:0/344405877 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d68011ad0 con 0x7f2d781599d0 2024-01-26T23:30:02.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.108+0000 7f2d7d5c6700 1 -- 172.21.15.46:0/344405877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7815ca70 con 0x7f2d781599d0 2024-01-26T23:30:02.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.109+0000 7f2d7d5c6700 1 -- 172.21.15.46:0/344405877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7804c710 con 0x7f2d781599d0 2024-01-26T23:30:02.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.109+0000 7f2d6effd700 1 -- 172.21.15.46:0/344405877 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2d6801b3b0 con 0x7f2d781599d0 2024-01-26T23:30:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.109+0000 7f2d6effd700 1 --2- 172.21.15.46:0/344405877 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d58066f30 0x7f2d580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.110+0000 7f2d6effd700 1 -- 172.21.15.46:0/344405877 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f2d68083dd0 con 0x7f2d781599d0 2024-01-26T23:30:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.110+0000 7f2d767fc700 1 --2- 172.21.15.46:0/344405877 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d58066f30 0x7f2d580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.110+0000 7f2d767fc700 1 --2- 172.21.15.46:0/344405877 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d58066f30 0x7f2d580693e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2d600093f0 tx=0x7f2d60017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.114+0000 7f2d6effd700 1 -- 172.21.15.46:0/344405877 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2d68050d60 con 0x7f2d781599d0 2024-01-26T23:30:02.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.365+0000 7f2d7d5c6700 1 -- 172.21.15.46:0/344405877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2d780b1780 con 0x7f2d781599d0 2024-01-26T23:30:02.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.653+0000 7f2d6effd700 1 -- 172.21.15.46:0/344405877 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2d68017070 con 0x7f2d781599d0 2024-01-26T23:30:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.656+0000 7f2d7d5c6700 1 -- 172.21.15.46:0/344405877 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d58066f30 msgr2=0x7f2d580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.656+0000 7f2d7d5c6700 1 --2- 172.21.15.46:0/344405877 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d58066f30 0x7f2d580693e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2d600093f0 tx=0x7f2d60017000 comp rx=0 tx=0).stop 2024-01-26T23:30:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.656+0000 7f2d7d5c6700 1 -- 172.21.15.46:0/344405877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d781599d0 msgr2=0x7f2d7815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.656+0000 7f2d7d5c6700 1 --2- 172.21.15.46:0/344405877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d781599d0 0x7f2d7815cf90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800c7c0 tx=0x7f2d68007f70 comp rx=0 tx=0).stop 2024-01-26T23:30:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.656+0000 7f2d7d5c6700 1 -- 172.21.15.46:0/344405877 shutdown_connections 2024-01-26T23:30:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.656+0000 7f2d7d5c6700 1 --2- 172.21.15.46:0/344405877 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d58066f30 0x7f2d580693e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.656+0000 7f2d7d5c6700 1 --2- 172.21.15.46:0/344405877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d781599d0 0x7f2d7815cf90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:02.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.656+0000 7f2d7d5c6700 1 --2- 172.21.15.46:0/344405877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2d7815d8c0 0x7f2d780aba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:02.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.656+0000 7f2d7d5c6700 1 --2- 172.21.15.46:0/344405877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2d78159da0 0x7f2d780ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:02.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.656+0000 7f2d7d5c6700 1 -- 172.21.15.46:0/344405877 >> 172.21.15.46:0/344405877 conn(0x7f2d7806f840 msgr2=0x7f2d78069770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:02.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.656+0000 7f2d7d5c6700 1 -- 172.21.15.46:0/344405877 shutdown_connections 2024-01-26T23:30:02.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.656+0000 7f2d7d5c6700 1 -- 172.21.15.46:0/344405877 wait complete. 2024-01-26T23:30:02.672 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-26T23:30:02.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.809+0000 7f8f717c1700 1 Processor -- start 2024-01-26T23:30:02.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.809+0000 7f8f717c1700 1 -- start start 2024-01-26T23:30:02.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.810+0000 7f8f717c1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f6c0c93a0 0x7f8f6c0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:02.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.810+0000 7f8f717c1700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c15e390 con 0x7f8f6c067d50 2024-01-26T23:30:02.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.810+0000 7f8f717c1700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c15d810 con 0x7f8f6c0681c0 2024-01-26T23:30:02.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.810+0000 7f8f717c1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c15daf0 con 0x7f8f6c0c93a0 2024-01-26T23:30:02.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.810+0000 7f8f6a7fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8f6c067d50 0x7f8f6c159220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44384/0 (socket says 172.21.15.46:44384) 2024-01-26T23:30:02.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.810+0000 7f8f6a7fc700 1 -- 172.21.15.46:0/1630913284 learned_addr learned my addr 172.21.15.46:0/1630913284 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:02.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.810+0000 7f8f6affd700 1 --2- 172.21.15.46:0/1630913284 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f6c0c93a0 0x7f8f6c0c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:02.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f69ffb700 1 -- 172.21.15.46:0/1630913284 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175339014 0 0) 0x7f8f6c15e390 con 0x7f8f6c067d50 2024-01-26T23:30:02.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f69ffb700 1 -- 172.21.15.46:0/1630913284 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f58003570 con 0x7f8f6c067d50 2024-01-26T23:30:02.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f69ffb700 1 -- 172.21.15.46:0/1630913284 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782872732 0 0) 0x7f8f6c15d810 con 0x7f8f6c0681c0 2024-01-26T23:30:02.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f69ffb700 1 -- 172.21.15.46:0/1630913284 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f6c15e390 con 0x7f8f6c0681c0 2024-01-26T23:30:02.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f69ffb700 1 -- 172.21.15.46:0/1630913284 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 750627073 0 0) 0x7f8f58003570 con 0x7f8f6c067d50 2024-01-26T23:30:02.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f69ffb700 1 -- 172.21.15.46:0/1630913284 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f6c0c93a0 msgr2=0x7f8f6c0c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:02.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f69ffb700 1 --2- 172.21.15.46:0/1630913284 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f6c0c93a0 0x7f8f6c0c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:02.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f69ffb700 1 -- 172.21.15.46:0/1630913284 >> v1:172.21.15.106:6789/0 conn(0x7f8f6c0681c0 legacy=0x7f8f6c0c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:02.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f69ffb700 1 -- 172.21.15.46:0/1630913284 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c15ddd0 con 0x7f8f6c067d50 2024-01-26T23:30:02.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f69ffb700 1 -- 172.21.15.46:0/1630913284 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f8f5c001470 con 0x7f8f6c067d50 2024-01-26T23:30:02.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f6affd700 1 --2- 172.21.15.46:0/1630913284 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f6c0c93a0 0x7f8f6c0c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:02.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f69ffb700 1 -- 172.21.15.46:0/1630913284 >> v1:172.21.15.46:6789/0 conn(0x7f8f6c067d50 legacy=0x7f8f6c159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:02.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f69ffb700 1 --2- 172.21.15.46:0/1630913284 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f58003c20 0x7f8f58004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:02.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f69ffb700 1 --2- 172.21.15.46:0/1630913284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f58004810 0x7f8f58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:02.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f69ffb700 1 --2- 172.21.15.46:0/1630913284 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f6c0c93a0 0x7f8f58005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:02.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f69ffb700 1 -- 172.21.15.46:0/1630913284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c15daf0 con 0x7f8f58004810 2024-01-26T23:30:02.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f69ffb700 1 -- 172.21.15.46:0/1630913284 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5800bbf0 con 0x7f8f58003c20 2024-01-26T23:30:02.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f69ffb700 1 -- 172.21.15.46:0/1630913284 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5800bd80 con 0x7f8f6c0c93a0 2024-01-26T23:30:02.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f6a7fc700 1 --2- 172.21.15.46:0/1630913284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f58004810 0x7f8f58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:02.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f6b7fe700 1 --2- 172.21.15.46:0/1630913284 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f6c0c93a0 0x7f8f58005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:02.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f6affd700 1 --2- 172.21.15.46:0/1630913284 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f58003c20 0x7f8f58004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:02.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f6a7fc700 1 -- 172.21.15.46:0/1630913284 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f6c0c93a0 msgr2=0x7f8f58005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:02.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.811+0000 7f8f6a7fc700 1 --2- 172.21.15.46:0/1630913284 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f6c0c93a0 0x7f8f58005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:02.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.812+0000 7f8f6a7fc700 1 -- 172.21.15.46:0/1630913284 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f58003c20 msgr2=0x7f8f58004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:02.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.812+0000 7f8f6a7fc700 1 --2- 172.21.15.46:0/1630913284 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f58003c20 0x7f8f58004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:02.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.812+0000 7f8f6a7fc700 1 -- 172.21.15.46:0/1630913284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f54009420 con 0x7f8f58004810 2024-01-26T23:30:02.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.812+0000 7f8f6a7fc700 1 --2- 172.21.15.46:0/1630913284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f58004810 0x7f8f58004c80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c00c2d0 tx=0x7f8f5c001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc76fd615e382d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:02.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.812+0000 7f8f69ffb700 1 -- 172.21.15.46:0/1630913284 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c00cb80 con 0x7f8f58004810 2024-01-26T23:30:02.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.812+0000 7f8f69ffb700 1 -- 172.21.15.46:0/1630913284 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5c00cce0 con 0x7f8f58004810 2024-01-26T23:30:02.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.812+0000 7f8f69ffb700 1 -- 172.21.15.46:0/1630913284 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c005b50 con 0x7f8f58004810 2024-01-26T23:30:02.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.812+0000 7f8f717c1700 1 -- 172.21.15.46:0/1630913284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f58004810 msgr2=0x7f8f58004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:02.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.812+0000 7f8f717c1700 1 --2- 172.21.15.46:0/1630913284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f58004810 0x7f8f58004c80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c00c2d0 tx=0x7f8f5c001e10 comp rx=0 tx=0).stop 2024-01-26T23:30:02.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.812+0000 7f8f6b7fe700 1 -- 172.21.15.46:0/1630913284 reap_dead start 2024-01-26T23:30:02.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.812+0000 7f8f717c1700 1 -- 172.21.15.46:0/1630913284 shutdown_connections 2024-01-26T23:30:02.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.812+0000 7f8f717c1700 1 -- 172.21.15.46:0/1630913284 >> 172.21.15.46:0/1630913284 conn(0x7f8f6c0ca9a0 msgr2=0x7f8f6c068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:02.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.813+0000 7f8f717c1700 1 -- 172.21.15.46:0/1630913284 shutdown_connections 2024-01-26T23:30:02.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.813+0000 7f8f717c1700 1 -- 172.21.15.46:0/1630913284 wait complete. 2024-01-26T23:30:02.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.814+0000 7f8f717c1700 1 Processor -- start 2024-01-26T23:30:02.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.814+0000 7f8f717c1700 1 -- start start 2024-01-26T23:30:02.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.815+0000 7f8f717c1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f58004810 0x7f8f6c154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:02.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.815+0000 7f8f717c1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f6c0c5f30 0x7f8f6c0c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:02.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.815+0000 7f8f717c1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f6c0c63a0 0x7f8f6c0c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:02.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.815+0000 7f8f717c1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c15a0a0 con 0x7f8f6c0c63a0 2024-01-26T23:30:02.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.815+0000 7f8f717c1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c0c68a0 con 0x7f8f6c0c5f30 2024-01-26T23:30:02.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.815+0000 7f8f717c1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c0c6b80 con 0x7f8f58004810 2024-01-26T23:30:02.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.815+0000 7f8f6b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f6c0c63a0 0x7f8f6c0c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:02.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.815+0000 7f8f6b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f6c0c63a0 0x7f8f6c0c83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33684/0 (socket says 172.21.15.46:33684) 2024-01-26T23:30:02.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.815+0000 7f8f6b7fe700 1 -- 172.21.15.46:0/1922744757 learned_addr learned my addr 172.21.15.46:0/1922744757 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:02.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.815+0000 7f8f6affd700 1 --2- 172.21.15.46:0/1922744757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f58004810 0x7f8f6c154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:02.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.815+0000 7f8f6a7fc700 1 --2- 172.21.15.46:0/1922744757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f6c0c5f30 0x7f8f6c0c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:02.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.815+0000 7f8f6b7fe700 1 -- 172.21.15.46:0/1922744757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f58004810 msgr2=0x7f8f6c154320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:02.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.815+0000 7f8f6b7fe700 1 --2- 172.21.15.46:0/1922744757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f58004810 0x7f8f6c154320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:02.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.815+0000 7f8f6b7fe700 1 -- 172.21.15.46:0/1922744757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f6c0c5f30 msgr2=0x7f8f6c0c7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:02.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.815+0000 7f8f6b7fe700 1 --2- 172.21.15.46:0/1922744757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f6c0c5f30 0x7f8f6c0c7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:02.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.815+0000 7f8f6b7fe700 1 -- 172.21.15.46:0/1922744757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c15a340 con 0x7f8f6c0c63a0 2024-01-26T23:30:02.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.815+0000 7f8f6b7fe700 1 --2- 172.21.15.46:0/1922744757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f6c0c63a0 0x7f8f6c0c83c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c06e380 tx=0x7f8f640017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:02.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.816+0000 7f8f62ffd700 1 -- 172.21.15.46:0/1922744757 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f64018070 con 0x7f8f6c0c63a0 2024-01-26T23:30:02.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.816+0000 7f8f62ffd700 1 -- 172.21.15.46:0/1922744757 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f64006ba0 con 0x7f8f6c0c63a0 2024-01-26T23:30:02.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.816+0000 7f8f717c1700 1 -- 172.21.15.46:0/1922744757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6c0c6e60 con 0x7f8f6c0c63a0 2024-01-26T23:30:02.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.816+0000 7f8f62ffd700 1 -- 172.21.15.46:0/1922744757 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6400c7b0 con 0x7f8f6c0c63a0 2024-01-26T23:30:02.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.816+0000 7f8f717c1700 1 -- 172.21.15.46:0/1922744757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6c155770 con 0x7f8f6c0c63a0 2024-01-26T23:30:02.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.817+0000 7f8f717c1700 1 -- 172.21.15.46:0/1922744757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f6c159f00 con 0x7f8f6c0c63a0 2024-01-26T23:30:02.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.817+0000 7f8f62ffd700 1 -- 172.21.15.46:0/1922744757 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8f6400cb60 con 0x7f8f6c0c63a0 2024-01-26T23:30:02.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.817+0000 7f8f62ffd700 1 --2- 172.21.15.46:0/1922744757 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8f4c066f30 0x7f8f4c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:02.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.818+0000 7f8f62ffd700 1 -- 172.21.15.46:0/1922744757 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f8f64084580 con 0x7f8f6c0c63a0 2024-01-26T23:30:02.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.818+0000 7f8f6affd700 1 --2- 172.21.15.46:0/1922744757 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8f4c066f30 0x7f8f4c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:02.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.818+0000 7f8f6affd700 1 --2- 172.21.15.46:0/1922744757 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8f4c066f30 0x7f8f4c0693e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8f54006f00 tx=0x7f8f540091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:02.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:02.822+0000 7f8f62ffd700 1 -- 172.21.15.46:0/1922744757 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8f6c159f00 con 0x7f8f6c0c63a0 2024-01-26T23:30:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.075+0000 7f8f717c1700 1 -- 172.21.15.46:0/1922744757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8f6c159f00 con 0x7f8f6c0c63a0 2024-01-26T23:30:03.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.658+0000 7f8f62ffd700 1 -- 172.21.15.46:0/1922744757 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8f6c159f00 con 0x7f8f6c0c63a0 2024-01-26T23:30:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.661+0000 7f8f717c1700 1 -- 172.21.15.46:0/1922744757 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8f4c066f30 msgr2=0x7f8f4c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.661+0000 7f8f717c1700 1 --2- 172.21.15.46:0/1922744757 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8f4c066f30 0x7f8f4c0693e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8f54006f00 tx=0x7f8f540091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.661+0000 7f8f717c1700 1 -- 172.21.15.46:0/1922744757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f6c0c63a0 msgr2=0x7f8f6c0c83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.661+0000 7f8f717c1700 1 --2- 172.21.15.46:0/1922744757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f6c0c63a0 0x7f8f6c0c83c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c06e380 tx=0x7f8f640017a0 comp rx=0 tx=0).stop 2024-01-26T23:30:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.661+0000 7f8f717c1700 1 -- 172.21.15.46:0/1922744757 shutdown_connections 2024-01-26T23:30:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.661+0000 7f8f717c1700 1 --2- 172.21.15.46:0/1922744757 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8f4c066f30 0x7f8f4c0693e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.661+0000 7f8f717c1700 1 --2- 172.21.15.46:0/1922744757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f6c0c63a0 0x7f8f6c0c83c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.661+0000 7f8f717c1700 1 --2- 172.21.15.46:0/1922744757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8f58004810 0x7f8f6c154320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:03.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.661+0000 7f8f717c1700 1 --2- 172.21.15.46:0/1922744757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f6c0c5f30 0x7f8f6c0c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:03.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.661+0000 7f8f717c1700 1 -- 172.21.15.46:0/1922744757 >> 172.21.15.46:0/1922744757 conn(0x7f8f6c0ca9a0 msgr2=0x7f8f6c158440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:03.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.661+0000 7f8f717c1700 1 -- 172.21.15.46:0/1922744757 shutdown_connections 2024-01-26T23:30:03.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.661+0000 7f8f717c1700 1 -- 172.21.15.46:0/1922744757 wait complete. 2024-01-26T23:30:03.677 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:30:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.812+0000 7fc666ea2700 1 Processor -- start 2024-01-26T23:30:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.812+0000 7fc666ea2700 1 -- start start 2024-01-26T23:30:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.813+0000 7fc666ea2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6601596e0 0x7fc660159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.813+0000 7fc666ea2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc6600c4720 0x7fc660068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.813+0000 7fc666ea2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc6600cadb0 0x7fc6600cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.813+0000 7fc666ea2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6600cbf90 con 0x7fc6600cadb0 2024-01-26T23:30:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.813+0000 7fc666ea2700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6600c97f0 con 0x7fc6600c4720 2024-01-26T23:30:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.813+0000 7fc666ea2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600cb730 con 0x7fc6601596e0 2024-01-26T23:30:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.813+0000 7fc66543f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc6600cadb0 0x7fc6600cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.813+0000 7fc66543f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc6600cadb0 0x7fc6600cb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33690/0 (socket says 172.21.15.46:33690) 2024-01-26T23:30:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.813+0000 7fc66543f700 1 -- 172.21.15.46:0/2843863701 learned_addr learned my addr 172.21.15.46:0/2843863701 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.813+0000 7fc664c3e700 1 --2- 172.21.15.46:0/2843863701 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6601596e0 0x7fc660159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.813+0000 7fc65ffff700 1 --2- 172.21.15.46:0/2843863701 >> v2:172.21.15.106:3300/0 conn(0x7fc6600c4720 0x7fc660068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.813+0000 7fc66543f700 1 -- 172.21.15.46:0/2843863701 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6601596e0 msgr2=0x7fc660159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:03.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.813+0000 7fc66543f700 1 --2- 172.21.15.46:0/2843863701 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6601596e0 0x7fc660159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:03.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.813+0000 7fc66543f700 1 -- 172.21.15.46:0/2843863701 >> v2:172.21.15.106:3300/0 conn(0x7fc6600c4720 msgr2=0x7fc660068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:03.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.813+0000 7fc66543f700 1 --2- 172.21.15.46:0/2843863701 >> v2:172.21.15.106:3300/0 conn(0x7fc6600c4720 0x7fc660068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:03.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.813+0000 7fc66543f700 1 -- 172.21.15.46:0/2843863701 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6600c8c20 con 0x7fc6600cadb0 2024-01-26T23:30:03.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.813+0000 7fc65ffff700 1 --2- 172.21.15.46:0/2843863701 >> v2:172.21.15.106:3300/0 conn(0x7fc6600c4720 0x7fc660068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:03.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.814+0000 7fc66543f700 1 --2- 172.21.15.46:0/2843863701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6600cadb0 0x7fc6600cb1f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc66006ef80 tx=0x7fc658009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b3f01a41e0e04a7 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:03.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.814+0000 7fc65f7fe700 1 -- 172.21.15.46:0/2843863701 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65800e070 con 0x7fc6600cadb0 2024-01-26T23:30:03.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.814+0000 7fc65f7fe700 1 -- 172.21.15.46:0/2843863701 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65800ba20 con 0x7fc6600cadb0 2024-01-26T23:30:03.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.814+0000 7fc65f7fe700 1 -- 172.21.15.46:0/2843863701 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6580092e0 con 0x7fc6600cadb0 2024-01-26T23:30:03.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.814+0000 7fc666ea2700 1 -- 172.21.15.46:0/2843863701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6600cadb0 msgr2=0x7fc6600cb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:03.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.814+0000 7fc666ea2700 1 --2- 172.21.15.46:0/2843863701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6600cadb0 0x7fc6600cb1f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc66006ef80 tx=0x7fc658009840 comp rx=0 tx=0).stop 2024-01-26T23:30:03.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.814+0000 7fc666ea2700 1 -- 172.21.15.46:0/2843863701 shutdown_connections 2024-01-26T23:30:03.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.814+0000 7fc666ea2700 1 --2- 172.21.15.46:0/2843863701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6600cadb0 0x7fc6600cb1f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:03.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.814+0000 7fc666ea2700 1 --2- 172.21.15.46:0/2843863701 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6601596e0 0x7fc660159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:03.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.814+0000 7fc666ea2700 1 --2- 172.21.15.46:0/2843863701 >> v2:172.21.15.106:3300/0 conn(0x7fc6600c4720 0x7fc660068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:03.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.814+0000 7fc666ea2700 1 -- 172.21.15.46:0/2843863701 >> 172.21.15.46:0/2843863701 conn(0x7fc660097820 msgr2=0x7fc660068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:03.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.814+0000 7fc666ea2700 1 -- 172.21.15.46:0/2843863701 shutdown_connections 2024-01-26T23:30:03.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.814+0000 7fc666ea2700 1 -- 172.21.15.46:0/2843863701 wait complete. 2024-01-26T23:30:03.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.816+0000 7fc666ea2700 1 Processor -- start 2024-01-26T23:30:03.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.816+0000 7fc666ea2700 1 -- start start 2024-01-26T23:30:03.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc666ea2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6600c4720 0x7fc66015fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:03.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc666ea2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6600cadb0 0x7fc6600b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:03.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc666ea2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6601596e0 0x7fc6600b1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:03.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc666ea2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600c7540 con 0x7fc6600cadb0 2024-01-26T23:30:03.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc666ea2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600c7820 con 0x7fc6600c4720 2024-01-26T23:30:03.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc666ea2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600c7b00 con 0x7fc6601596e0 2024-01-26T23:30:03.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc65ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6600cadb0 0x7fc6600b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:03.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc65ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6600cadb0 0x7fc6600b12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33700/0 (socket says 172.21.15.46:33700) 2024-01-26T23:30:03.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc65ffff700 1 -- 172.21.15.46:0/3257327721 learned_addr learned my addr 172.21.15.46:0/3257327721 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:03.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc66543f700 1 --2- 172.21.15.46:0/3257327721 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6601596e0 0x7fc6600b1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:03.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc664c3e700 1 --2- 172.21.15.46:0/3257327721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6600c4720 0x7fc66015fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:03.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc65ffff700 1 -- 172.21.15.46:0/3257327721 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6601596e0 msgr2=0x7fc6600b1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:03.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc65ffff700 1 --2- 172.21.15.46:0/3257327721 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6601596e0 0x7fc6600b1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:03.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc65ffff700 1 -- 172.21.15.46:0/3257327721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6600c4720 msgr2=0x7fc66015fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:03.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc65ffff700 1 --2- 172.21.15.46:0/3257327721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6600c4720 0x7fc66015fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:03.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc65ffff700 1 -- 172.21.15.46:0/3257327721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc658009ae0 con 0x7fc6600cadb0 2024-01-26T23:30:03.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc664c3e700 1 --2- 172.21.15.46:0/3257327721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6600c4720 0x7fc66015fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:30:03.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc65ffff700 1 --2- 172.21.15.46:0/3257327721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6600cadb0 0x7fc6600b12e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00e8d0 tx=0x7fc64c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:03.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc65cff9700 1 -- 172.21.15.46:0/3257327721 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c00ee70 con 0x7fc6600cadb0 2024-01-26T23:30:03.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc65cff9700 1 -- 172.21.15.46:0/3257327721 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64c00c5e0 con 0x7fc6600cadb0 2024-01-26T23:30:03.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc666ea2700 1 -- 172.21.15.46:0/3257327721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6600c7de0 con 0x7fc6600cadb0 2024-01-26T23:30:03.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.817+0000 7fc65cff9700 1 -- 172.21.15.46:0/3257327721 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c00c980 con 0x7fc6600cadb0 2024-01-26T23:30:03.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.818+0000 7fc666ea2700 1 -- 172.21.15.46:0/3257327721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc660160210 con 0x7fc6600cadb0 2024-01-26T23:30:03.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.818+0000 7fc666ea2700 1 -- 172.21.15.46:0/3257327721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc66016aa10 con 0x7fc6600cadb0 2024-01-26T23:30:03.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.819+0000 7fc65cff9700 1 -- 172.21.15.46:0/3257327721 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc64c013410 con 0x7fc6600cadb0 2024-01-26T23:30:03.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.819+0000 7fc65cff9700 1 --2- 172.21.15.46:0/3257327721 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc644066f30 0x7fc6440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:03.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.819+0000 7fc65cff9700 1 -- 172.21.15.46:0/3257327721 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fc64c084610 con 0x7fc6600cadb0 2024-01-26T23:30:03.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.820+0000 7fc664c3e700 1 --2- 172.21.15.46:0/3257327721 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc644066f30 0x7fc6440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:03.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.820+0000 7fc664c3e700 1 --2- 172.21.15.46:0/3257327721 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc644066f30 0x7fc6440693e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc654006f00 tx=0x7fc654008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:03.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:03.824+0000 7fc65cff9700 1 -- 172.21.15.46:0/3257327721 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc66016aa10 con 0x7fc6600cadb0 2024-01-26T23:30:04.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.076+0000 7fc666ea2700 1 -- 172.21.15.46:0/3257327721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc66016aa10 con 0x7fc6600cadb0 2024-01-26T23:30:04.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.077+0000 7fc65cff9700 1 -- 172.21.15.46:0/3257327721 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4593 (secure 0 0 0) 0x7fc66016aa10 con 0x7fc6600cadb0 2024-01-26T23:30:04.077 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:04.078 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3409861270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3409861270},{"type":"v1","addr":"172.21.15.46:6833","nonce":3409861270}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3319220815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3319220815},{"type":"v1","addr":"172.21.15.106:6836","nonce":3319220815}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4252,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/529198432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":529198432},{"type":"v1","addr":"172.21.15.106:6837","nonce":529198432}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:29:44.031963+0000","modified":"2024-01-26T23:30:03.652307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/688086235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":688086235},{"type":"v1","addr":"172.21.15.46:6835","nonce":688086235}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2024-01-26T23:30:04.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.079+0000 7fc666ea2700 1 -- 172.21.15.46:0/3257327721 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc644066f30 msgr2=0x7fc6440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:04.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.079+0000 7fc666ea2700 1 --2- 172.21.15.46:0/3257327721 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc644066f30 0x7fc6440693e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc654006f00 tx=0x7fc654008040 comp rx=0 tx=0).stop 2024-01-26T23:30:04.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.079+0000 7fc666ea2700 1 -- 172.21.15.46:0/3257327721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6600cadb0 msgr2=0x7fc6600b12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:04.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.079+0000 7fc666ea2700 1 --2- 172.21.15.46:0/3257327721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6600cadb0 0x7fc6600b12e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00e8d0 tx=0x7fc64c008040 comp rx=0 tx=0).stop 2024-01-26T23:30:04.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.079+0000 7fc666ea2700 1 -- 172.21.15.46:0/3257327721 shutdown_connections 2024-01-26T23:30:04.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.079+0000 7fc666ea2700 1 --2- 172.21.15.46:0/3257327721 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc644066f30 0x7fc6440693e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.079+0000 7fc666ea2700 1 --2- 172.21.15.46:0/3257327721 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6601596e0 0x7fc6600b1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.079+0000 7fc666ea2700 1 --2- 172.21.15.46:0/3257327721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6600c4720 0x7fc66015fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.079+0000 7fc666ea2700 1 --2- 172.21.15.46:0/3257327721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6600cadb0 0x7fc6600b12e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.079+0000 7fc666ea2700 1 -- 172.21.15.46:0/3257327721 >> 172.21.15.46:0/3257327721 conn(0x7fc660097820 msgr2=0x7fc66006a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:04.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.079+0000 7fc666ea2700 1 -- 172.21.15.46:0/3257327721 shutdown_connections 2024-01-26T23:30:04.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.079+0000 7fc666ea2700 1 -- 172.21.15.46:0/3257327721 wait complete. 2024-01-26T23:30:04.081 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 9 2024-01-26T23:30:04.093 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:04.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.225+0000 7f18f591c700 1 Processor -- start 2024-01-26T23:30:04.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.225+0000 7f18f591c700 1 -- start start 2024-01-26T23:30:04.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.226+0000 7f18f591c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18f0159960 0x7f18f00cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.226+0000 7f18f591c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f18f0159d30 0x7f18f00ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.226+0000 7f18f591c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f006d760 con 0x7f18f0067930 2024-01-26T23:30:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.226+0000 7f18f591c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f006f820 con 0x7f18f0159d30 2024-01-26T23:30:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.226+0000 7f18f591c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f00c9f00 con 0x7f18f0159960 2024-01-26T23:30:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.226+0000 7f18ef7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f18f0067930 0x7f18f015d740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44396/0 (socket says 172.21.15.46:44396) 2024-01-26T23:30:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.226+0000 7f18ef7fe700 1 -- 172.21.15.46:0/3492213934 learned_addr learned my addr 172.21.15.46:0/3492213934 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.226+0000 7f18e7fff700 1 --2- 172.21.15.46:0/3492213934 >> v2:172.21.15.106:3300/0 conn(0x7f18f0159d30 0x7f18f00ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.226+0000 7f18eeffd700 1 --2- 172.21.15.46:0/3492213934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18f0159960 0x7f18f00cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.226+0000 7f18ee7fc700 1 -- 172.21.15.46:0/3492213934 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491257144 0 0) 0x7f18f006d760 con 0x7f18f0067930 2024-01-26T23:30:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.227+0000 7f18ee7fc700 1 -- 172.21.15.46:0/3492213934 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18dc003570 con 0x7f18f0067930 2024-01-26T23:30:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.227+0000 7f18eeffd700 1 -- 172.21.15.46:0/3492213934 >> v2:172.21.15.106:3300/0 conn(0x7f18f0159d30 msgr2=0x7f18f00ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.227+0000 7f18eeffd700 1 --2- 172.21.15.46:0/3492213934 >> v2:172.21.15.106:3300/0 conn(0x7f18f0159d30 0x7f18f00ce1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.227+0000 7f18eeffd700 1 -- 172.21.15.46:0/3492213934 >> v1:172.21.15.46:6789/0 conn(0x7f18f0067930 legacy=0x7f18f015d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.227+0000 7f18eeffd700 1 -- 172.21.15.46:0/3492213934 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f006fcb0 con 0x7f18f0159960 2024-01-26T23:30:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.227+0000 7f18eeffd700 1 --2- 172.21.15.46:0/3492213934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18f0159960 0x7f18f00cbc30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f18e00098e0 tx=0x7f18e00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6939edff8f8a3e89 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.227+0000 7f18ee7fc700 1 -- 172.21.15.46:0/3492213934 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e0011630 con 0x7f18f0159960 2024-01-26T23:30:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.227+0000 7f18ee7fc700 1 -- 172.21.15.46:0/3492213934 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e0011790 con 0x7f18f0159960 2024-01-26T23:30:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.227+0000 7f18ee7fc700 1 -- 172.21.15.46:0/3492213934 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e0011b30 con 0x7f18f0159960 2024-01-26T23:30:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.228+0000 7f18f591c700 1 -- 172.21.15.46:0/3492213934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18f0159960 msgr2=0x7f18f00cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.228+0000 7f18f591c700 1 --2- 172.21.15.46:0/3492213934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18f0159960 0x7f18f00cbc30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f18e00098e0 tx=0x7f18e00091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.228+0000 7f18f591c700 1 -- 172.21.15.46:0/3492213934 shutdown_connections 2024-01-26T23:30:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.228+0000 7f18f591c700 1 --2- 172.21.15.46:0/3492213934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18f0159960 0x7f18f00cbc30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.228+0000 7f18f591c700 1 --2- 172.21.15.46:0/3492213934 >> v2:172.21.15.106:3300/0 conn(0x7f18f0159d30 0x7f18f00ce1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.228+0000 7f18f591c700 1 -- 172.21.15.46:0/3492213934 >> 172.21.15.46:0/3492213934 conn(0x7f18f006b850 msgr2=0x7f18f00cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.228+0000 7f18f591c700 1 -- 172.21.15.46:0/3492213934 shutdown_connections 2024-01-26T23:30:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.228+0000 7f18f591c700 1 -- 172.21.15.46:0/3492213934 wait complete. 2024-01-26T23:30:04.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.229+0000 7f18f591c700 1 Processor -- start 2024-01-26T23:30:04.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.229+0000 7f18f591c700 1 -- start start 2024-01-26T23:30:04.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.230+0000 7f18f591c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18f0067930 0x7f18f01555e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:04.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.230+0000 7f18f591c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18f0159960 0x7f18f0169af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:04.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.230+0000 7f18f591c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18f0159d30 0x7f18f01470a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:04.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.230+0000 7f18f591c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f00c7960 con 0x7f18f0159960 2024-01-26T23:30:04.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18f591c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f016d0b0 con 0x7f18f0159d30 2024-01-26T23:30:04.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18f591c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f016a320 con 0x7f18f0067930 2024-01-26T23:30:04.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18e7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18f0159960 0x7f18f0169af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:04.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18e7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18f0159960 0x7f18f0169af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33716/0 (socket says 172.21.15.46:33716) 2024-01-26T23:30:04.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18e7fff700 1 -- 172.21.15.46:0/1614120035 learned_addr learned my addr 172.21.15.46:0/1614120035 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:04.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18eeffd700 1 --2- 172.21.15.46:0/1614120035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18f0067930 0x7f18f01555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:04.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18ef7fe700 1 --2- 172.21.15.46:0/1614120035 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18f0159d30 0x7f18f01470a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:04.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18e7fff700 1 -- 172.21.15.46:0/1614120035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18f0067930 msgr2=0x7f18f01555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:04.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18e7fff700 1 --2- 172.21.15.46:0/1614120035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18f0067930 0x7f18f01555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18e7fff700 1 -- 172.21.15.46:0/1614120035 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18f0159d30 msgr2=0x7f18f01470a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:04.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18e7fff700 1 --2- 172.21.15.46:0/1614120035 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18f0159d30 0x7f18f01470a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18e7fff700 1 -- 172.21.15.46:0/1614120035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e0009420 con 0x7f18f0159960 2024-01-26T23:30:04.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18e7fff700 1 --2- 172.21.15.46:0/1614120035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18f0159960 0x7f18f0169af0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f18d800ec80 tx=0x7f18d80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:04.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18e6ffd700 1 -- 172.21.15.46:0/1614120035 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d800c850 con 0x7f18f0159960 2024-01-26T23:30:04.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18e6ffd700 1 -- 172.21.15.46:0/1614120035 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d800c9b0 con 0x7f18f0159960 2024-01-26T23:30:04.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18f591c700 1 -- 172.21.15.46:0/1614120035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f016a600 con 0x7f18f0159960 2024-01-26T23:30:04.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18e6ffd700 1 -- 172.21.15.46:0/1614120035 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d800cd50 con 0x7f18f0159960 2024-01-26T23:30:04.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.231+0000 7f18f591c700 1 -- 172.21.15.46:0/1614120035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f016d300 con 0x7f18f0159960 2024-01-26T23:30:04.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.232+0000 7f18f591c700 1 -- 172.21.15.46:0/1614120035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f016d670 con 0x7f18f0159960 2024-01-26T23:30:04.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.233+0000 7f18e6ffd700 1 -- 172.21.15.46:0/1614120035 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f18d8013680 con 0x7f18f0159960 2024-01-26T23:30:04.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.233+0000 7f18e6ffd700 1 --2- 172.21.15.46:0/1614120035 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f18d0066f30 0x7f18d00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:04.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.233+0000 7f18e6ffd700 1 -- 172.21.15.46:0/1614120035 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f18d8084a50 con 0x7f18f0159960 2024-01-26T23:30:04.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.234+0000 7f18eeffd700 1 --2- 172.21.15.46:0/1614120035 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f18d0066f30 0x7f18d00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:04.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.234+0000 7f18eeffd700 1 --2- 172.21.15.46:0/1614120035 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f18d0066f30 0x7f18d00693e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f18e00091e0 tx=0x7f18e0017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:04.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.238+0000 7f18e6ffd700 1 -- 172.21.15.46:0/1614120035 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f18d80519d0 con 0x7f18f0159960 2024-01-26T23:30:04.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.442+0000 7f18f591c700 1 -- 172.21.15.46:0/1614120035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18f004c710 con 0x7f18f0159960 2024-01-26T23:30:04.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.443+0000 7f18e6ffd700 1 -- 172.21.15.46:0/1614120035 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7f18d8054ff0 con 0x7f18f0159960 2024-01-26T23:30:04.444 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:04.444 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:49.075104+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:04.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.445+0000 7f18f591c700 1 -- 172.21.15.46:0/1614120035 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f18d0066f30 msgr2=0x7f18d00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:04.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.445+0000 7f18f591c700 1 --2- 172.21.15.46:0/1614120035 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f18d0066f30 0x7f18d00693e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f18e00091e0 tx=0x7f18e0017000 comp rx=0 tx=0).stop 2024-01-26T23:30:04.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.445+0000 7f18f591c700 1 -- 172.21.15.46:0/1614120035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18f0159960 msgr2=0x7f18f0169af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:04.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.445+0000 7f18f591c700 1 --2- 172.21.15.46:0/1614120035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18f0159960 0x7f18f0169af0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f18d800ec80 tx=0x7f18d80091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.445+0000 7f18f591c700 1 -- 172.21.15.46:0/1614120035 shutdown_connections 2024-01-26T23:30:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.445+0000 7f18f591c700 1 --2- 172.21.15.46:0/1614120035 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f18d0066f30 0x7f18d00693e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.445+0000 7f18f591c700 1 --2- 172.21.15.46:0/1614120035 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18f0159d30 0x7f18f01470a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.445+0000 7f18f591c700 1 --2- 172.21.15.46:0/1614120035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18f0067930 0x7f18f01555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.445+0000 7f18f591c700 1 --2- 172.21.15.46:0/1614120035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18f0159960 0x7f18f0169af0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.445+0000 7f18f591c700 1 -- 172.21.15.46:0/1614120035 >> 172.21.15.46:0/1614120035 conn(0x7f18f006b850 msgr2=0x7f18f0156940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.445+0000 7f18f591c700 1 -- 172.21.15.46:0/1614120035 shutdown_connections 2024-01-26T23:30:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.445+0000 7f18f591c700 1 -- 172.21.15.46:0/1614120035 wait complete. 2024-01-26T23:30:04.460 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-26T23:30:04.461 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:30:04.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.592+0000 7f5f7a749700 1 Processor -- start 2024-01-26T23:30:04.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.592+0000 7f5f7a749700 1 -- start start 2024-01-26T23:30:04.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.593+0000 7f5f7a749700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5f741573f0 0x7f5f741577c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:04.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.593+0000 7f5f7a749700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5f740b2570 0x7f5f741612e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:04.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.593+0000 7f5f7a749700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f740aedf0 con 0x7f5f740b40a0 2024-01-26T23:30:04.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.593+0000 7f5f7a749700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f740b3510 con 0x7f5f740b2570 2024-01-26T23:30:04.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.593+0000 7f5f7a749700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f740af0d0 con 0x7f5f741573f0 2024-01-26T23:30:04.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.593+0000 7f5f78ce6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5f740b2570 0x7f5f741612e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:04.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.593+0000 7f5f737fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5f740b40a0 0x7f5f740b1e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44406/0 (socket says 172.21.15.46:44406) 2024-01-26T23:30:04.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.593+0000 7f5f737fe700 1 -- 172.21.15.46:0/1364702322 learned_addr learned my addr 172.21.15.46:0/1364702322 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:04.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.593+0000 7f5f73fff700 1 --2- 172.21.15.46:0/1364702322 >> v2:172.21.15.106:3300/0 conn(0x7f5f741573f0 0x7f5f741577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:04.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.593+0000 7f5f78ce6700 1 -- 172.21.15.46:0/1364702322 >> v2:172.21.15.106:3300/0 conn(0x7f5f741573f0 msgr2=0x7f5f741577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:04.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.594+0000 7f5f78ce6700 1 --2- 172.21.15.46:0/1364702322 >> v2:172.21.15.106:3300/0 conn(0x7f5f741573f0 0x7f5f741577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.594+0000 7f5f78ce6700 1 -- 172.21.15.46:0/1364702322 >> v1:172.21.15.46:6789/0 conn(0x7f5f740b40a0 legacy=0x7f5f740b1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:04.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.594+0000 7f5f78ce6700 1 -- 172.21.15.46:0/1364702322 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f740af3b0 con 0x7f5f740b2570 2024-01-26T23:30:04.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.594+0000 7f5f72ffd700 1 -- 172.21.15.46:0/1364702322 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663475168 0 0) 0x7f5f740aedf0 con 0x7f5f740b40a0 2024-01-26T23:30:04.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.594+0000 7f5f78ce6700 1 --2- 172.21.15.46:0/1364702322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f740b2570 0x7f5f741612e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5f740649a0 tx=0x7f5f6c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8206d21313bb22 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:04.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.594+0000 7f5f72ffd700 1 -- 172.21.15.46:0/1364702322 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c00e070 con 0x7f5f740b2570 2024-01-26T23:30:04.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.594+0000 7f5f72ffd700 1 -- 172.21.15.46:0/1364702322 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6c00bb60 con 0x7f5f740b2570 2024-01-26T23:30:04.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.594+0000 7f5f72ffd700 1 -- 172.21.15.46:0/1364702322 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c012430 con 0x7f5f740b2570 2024-01-26T23:30:04.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.594+0000 7f5f7a749700 1 -- 172.21.15.46:0/1364702322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f740b2570 msgr2=0x7f5f741612e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:04.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.595+0000 7f5f7a749700 1 --2- 172.21.15.46:0/1364702322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f740b2570 0x7f5f741612e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5f740649a0 tx=0x7f5f6c009780 comp rx=0 tx=0).stop 2024-01-26T23:30:04.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.595+0000 7f5f7a749700 1 -- 172.21.15.46:0/1364702322 shutdown_connections 2024-01-26T23:30:04.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.595+0000 7f5f7a749700 1 --2- 172.21.15.46:0/1364702322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f740b2570 0x7f5f741612e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.595+0000 7f5f7a749700 1 --2- 172.21.15.46:0/1364702322 >> v2:172.21.15.106:3300/0 conn(0x7f5f741573f0 0x7f5f741577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.595+0000 7f5f7a749700 1 -- 172.21.15.46:0/1364702322 >> 172.21.15.46:0/1364702322 conn(0x7f5f74087f60 msgr2=0x7f5f74088330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:04.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.595+0000 7f5f7a749700 1 -- 172.21.15.46:0/1364702322 shutdown_connections 2024-01-26T23:30:04.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.595+0000 7f5f7a749700 1 -- 172.21.15.46:0/1364702322 wait complete. 2024-01-26T23:30:04.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.596+0000 7f5f7a749700 1 Processor -- start 2024-01-26T23:30:04.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.596+0000 7f5f7a749700 1 -- start start 2024-01-26T23:30:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.597+0000 7f5f7a749700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f740b2570 0x7f5f741565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.597+0000 7f5f7a749700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f740b40a0 0x7f5f740aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.597+0000 7f5f7a749700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f741573f0 0x7f5f7414a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.597+0000 7f5f7a749700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f74156af0 con 0x7f5f741573f0 2024-01-26T23:30:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.597+0000 7f5f7a749700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f740ac500 con 0x7f5f740b40a0 2024-01-26T23:30:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.597+0000 7f5f7a749700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f740ab8f0 con 0x7f5f740b2570 2024-01-26T23:30:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.597+0000 7f5f78ce6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f741573f0 0x7f5f7414a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.597+0000 7f5f73fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f740b2570 0x7f5f741565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.597+0000 7f5f78ce6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f741573f0 0x7f5f7414a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33734/0 (socket says 172.21.15.46:33734) 2024-01-26T23:30:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.597+0000 7f5f78ce6700 1 -- 172.21.15.46:0/3062059841 learned_addr learned my addr 172.21.15.46:0/3062059841 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:04.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.597+0000 7f5f737fe700 1 --2- 172.21.15.46:0/3062059841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f740b40a0 0x7f5f740aaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:04.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.597+0000 7f5f78ce6700 1 -- 172.21.15.46:0/3062059841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f740b2570 msgr2=0x7f5f741565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:04.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.597+0000 7f5f78ce6700 1 --2- 172.21.15.46:0/3062059841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f740b2570 0x7f5f741565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.597+0000 7f5f78ce6700 1 -- 172.21.15.46:0/3062059841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f740b40a0 msgr2=0x7f5f740aaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:04.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.597+0000 7f5f78ce6700 1 --2- 172.21.15.46:0/3062059841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f740b40a0 0x7f5f740aaf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.597+0000 7f5f78ce6700 1 -- 172.21.15.46:0/3062059841 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f6c009a20 con 0x7f5f741573f0 2024-01-26T23:30:04.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.597+0000 7f5f73fff700 1 --2- 172.21.15.46:0/3062059841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f740b2570 0x7f5f741565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:04.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.598+0000 7f5f78ce6700 1 --2- 172.21.15.46:0/3062059841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f741573f0 0x7f5f7414a820 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5f7406e9e0 tx=0x7f5f6c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:04.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.598+0000 7f5f5bfff700 1 -- 172.21.15.46:0/3062059841 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c027470 con 0x7f5f741573f0 2024-01-26T23:30:04.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.598+0000 7f5f5bfff700 1 -- 172.21.15.46:0/3062059841 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6c0275d0 con 0x7f5f741573f0 2024-01-26T23:30:04.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.598+0000 7f5f7a749700 1 -- 172.21.15.46:0/3062059841 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f740abbd0 con 0x7f5f741573f0 2024-01-26T23:30:04.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.598+0000 7f5f5bfff700 1 -- 172.21.15.46:0/3062059841 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c027990 con 0x7f5f741573f0 2024-01-26T23:30:04.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.598+0000 7f5f7a749700 1 -- 172.21.15.46:0/3062059841 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f740ac190 con 0x7f5f741573f0 2024-01-26T23:30:04.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.599+0000 7f5f7a749700 1 -- 172.21.15.46:0/3062059841 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f740ac6b0 con 0x7f5f741573f0 2024-01-26T23:30:04.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.599+0000 7f5f5bfff700 1 -- 172.21.15.46:0/3062059841 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5f6c024070 con 0x7f5f741573f0 2024-01-26T23:30:04.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.600+0000 7f5f5bfff700 1 --2- 172.21.15.46:0/3062059841 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5f60066f30 0x7f5f600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:04.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.600+0000 7f5f5bfff700 1 -- 172.21.15.46:0/3062059841 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f5f6c087290 con 0x7f5f741573f0 2024-01-26T23:30:04.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.600+0000 7f5f73fff700 1 --2- 172.21.15.46:0/3062059841 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5f60066f30 0x7f5f600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:04.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.601+0000 7f5f73fff700 1 --2- 172.21.15.46:0/3062059841 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5f60066f30 0x7f5f600693e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5f640113e0 tx=0x7f5f640091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:04.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.604+0000 7f5f5bfff700 1 -- 172.21.15.46:0/3062059841 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5f740ac6b0 con 0x7f5f741573f0 2024-01-26T23:30:04.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.815+0000 7f5f7a749700 1 -- 172.21.15.46:0/3062059841 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5f7404c7b0 con 0x7f5f60066f30 2024-01-26T23:30:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.826+0000 7f5f5bfff700 1 -- 172.21.15.46:0/3062059841 <== mgr.4102 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345239 (secure 0 0 0) 0x7f5f7404c7b0 con 0x7f5f60066f30 2024-01-26T23:30:04.831 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:04.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.829+0000 7f5f7a749700 1 -- 172.21.15.46:0/3062059841 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5f60066f30 msgr2=0x7f5f600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:04.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.829+0000 7f5f7a749700 1 --2- 172.21.15.46:0/3062059841 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5f60066f30 0x7f5f600693e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5f640113e0 tx=0x7f5f640091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.829+0000 7f5f7a749700 1 -- 172.21.15.46:0/3062059841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f741573f0 msgr2=0x7f5f7414a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.829+0000 7f5f7a749700 1 --2- 172.21.15.46:0/3062059841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f741573f0 0x7f5f7414a820 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5f7406e9e0 tx=0x7f5f6c019000 comp rx=0 tx=0).stop 2024-01-26T23:30:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.829+0000 7f5f7a749700 1 -- 172.21.15.46:0/3062059841 shutdown_connections 2024-01-26T23:30:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.829+0000 7f5f7a749700 1 --2- 172.21.15.46:0/3062059841 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5f60066f30 0x7f5f600693e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.829+0000 7f5f7a749700 1 --2- 172.21.15.46:0/3062059841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f741573f0 0x7f5f7414a820 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.829+0000 7f5f7a749700 1 --2- 172.21.15.46:0/3062059841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f740b2570 0x7f5f741565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.829+0000 7f5f7a749700 1 --2- 172.21.15.46:0/3062059841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f740b40a0 0x7f5f740aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.829+0000 7f5f7a749700 1 -- 172.21.15.46:0/3062059841 >> 172.21.15.46:0/3062059841 conn(0x7f5f74087f60 msgr2=0x7f5f7406c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.830+0000 7f5f7a749700 1 -- 172.21.15.46:0/3062059841 shutdown_connections 2024-01-26T23:30:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:04.830+0000 7f5f7a749700 1 -- 172.21.15.46:0/3062059841 wait complete. 2024-01-26T23:30:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:30:04.844 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-01-26T23:30:04.229026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594851,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381512,"kb_used_data":3208,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593208,"statfs":{"total":773094113280,"available":770655444992,"internally_reserved":0,"allocated":3284992,"data_stored":1985606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2177,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003050"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690995+0000","last_change":"2024-01-26T23:29:44.615781+0000","last_active":"2024-01-26T23:29:48.690995+0000","last_peered":"2024-01-26T23:29:48.690995+0000","last_clean":"2024-01-26T23:29:48.690995+0000","last_became_active":"2024-01-26T23:29:44.615522+0000","last_became_peered":"2024-01-26T23:29:44.615522+0000","last_unstale":"2024-01-26T23:29:48.690995+0000","last_undegraded":"2024-01-26T23:29:48.690995+0000","last_fullsized":"2024-01-26T23:29:48.690995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:03.997854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.627221+0000","last_change":"2024-01-26T23:29:44.628874+0000","last_active":"2024-01-26T23:29:48.627221+0000","last_peered":"2024-01-26T23:29:48.627221+0000","last_clean":"2024-01-26T23:29:48.627221+0000","last_became_active":"2024-01-26T23:29:44.628581+0000","last_became_peered":"2024-01-26T23:29:44.628581+0000","last_unstale":"2024-01-26T23:29:48.627221+0000","last_undegraded":"2024-01-26T23:29:48.627221+0000","last_fullsized":"2024-01-26T23:29:48.627221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:14.555745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631189+0000","last_change":"2024-01-26T23:29:44.601978+0000","last_active":"2024-01-26T23:29:48.631189+0000","last_peered":"2024-01-26T23:29:48.631189+0000","last_clean":"2024-01-26T23:29:48.631189+0000","last_became_active":"2024-01-26T23:29:44.601539+0000","last_became_peered":"2024-01-26T23:29:44.601539+0000","last_unstale":"2024-01-26T23:29:48.631189+0000","last_undegraded":"2024-01-26T23:29:48.631189+0000","last_fullsized":"2024-01-26T23:29:48.631189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:35:11.068882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636104+0000","last_change":"2024-01-26T23:29:44.632221+0000","last_active":"2024-01-26T23:29:48.636104+0000","last_peered":"2024-01-26T23:29:48.636104+0000","last_clean":"2024-01-26T23:29:48.636104+0000","last_became_active":"2024-01-26T23:29:44.631611+0000","last_became_peered":"2024-01-26T23:29:44.631611+0000","last_unstale":"2024-01-26T23:29:48.636104+0000","last_undegraded":"2024-01-26T23:29:48.636104+0000","last_fullsized":"2024-01-26T23:29:48.636104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:55:14.606640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690150+0000","last_change":"2024-01-26T23:29:44.616723+0000","last_active":"2024-01-26T23:29:48.690150+0000","last_peered":"2024-01-26T23:29:48.690150+0000","last_clean":"2024-01-26T23:29:48.690150+0000","last_became_active":"2024-01-26T23:29:44.615737+0000","last_became_peered":"2024-01-26T23:29:44.615737+0000","last_unstale":"2024-01-26T23:29:48.690150+0000","last_undegraded":"2024-01-26T23:29:48.690150+0000","last_fullsized":"2024-01-26T23:29:48.690150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635997+0000","last_change":"2024-01-26T23:29:44.622983+0000","last_active":"2024-01-26T23:29:48.635997+0000","last_peered":"2024-01-26T23:29:48.635997+0000","last_clean":"2024-01-26T23:29:48.635997+0000","last_became_active":"2024-01-26T23:29:44.622721+0000","last_became_peered":"2024-01-26T23:29:44.622721+0000","last_unstale":"2024-01-26T23:29:48.635997+0000","last_undegraded":"2024-01-26T23:29:48.635997+0000","last_fullsized":"2024-01-26T23:29:48.635997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690538+0000","last_change":"2024-01-26T23:29:44.620570+0000","last_active":"2024-01-26T23:29:48.690538+0000","last_peered":"2024-01-26T23:29:48.690538+0000","last_clean":"2024-01-26T23:29:48.690538+0000","last_became_active":"2024-01-26T23:29:44.620303+0000","last_became_peered":"2024-01-26T23:29:44.620303+0000","last_unstale":"2024-01-26T23:29:48.690538+0000","last_undegraded":"2024-01-26T23:29:48.690538+0000","last_fullsized":"2024-01-26T23:29:48.690538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:55:14.606640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636953+0000","last_change":"2024-01-26T23:29:44.630873+0000","last_active":"2024-01-26T23:29:48.636953+0000","last_peered":"2024-01-26T23:29:48.636953+0000","last_clean":"2024-01-26T23:29:48.636953+0000","last_became_active":"2024-01-26T23:29:44.630579+0000","last_became_peered":"2024-01-26T23:29:44.630579+0000","last_unstale":"2024-01-26T23:29:48.636953+0000","last_undegraded":"2024-01-26T23:29:48.636953+0000","last_fullsized":"2024-01-26T23:29:48.636953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:48:24.085774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690821+0000","last_change":"2024-01-26T23:29:44.626396+0000","last_active":"2024-01-26T23:29:48.690821+0000","last_peered":"2024-01-26T23:29:48.690821+0000","last_clean":"2024-01-26T23:29:48.690821+0000","last_became_active":"2024-01-26T23:29:44.622150+0000","last_became_peered":"2024-01-26T23:29:44.622150+0000","last_unstale":"2024-01-26T23:29:48.690821+0000","last_undegraded":"2024-01-26T23:29:48.690821+0000","last_fullsized":"2024-01-26T23:29:48.690821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:57:15.558584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637882+0000","last_change":"2024-01-26T23:29:44.618861+0000","last_active":"2024-01-26T23:29:48.637882+0000","last_peered":"2024-01-26T23:29:48.637882+0000","last_clean":"2024-01-26T23:29:48.637882+0000","last_became_active":"2024-01-26T23:29:44.618575+0000","last_became_peered":"2024-01-26T23:29:44.618575+0000","last_unstale":"2024-01-26T23:29:48.637882+0000","last_undegraded":"2024-01-26T23:29:48.637882+0000","last_fullsized":"2024-01-26T23:29:48.637882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:32.912164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637940+0000","last_change":"2024-01-26T23:29:44.617693+0000","last_active":"2024-01-26T23:29:48.637940+0000","last_peered":"2024-01-26T23:29:48.637940+0000","last_clean":"2024-01-26T23:29:48.637940+0000","last_became_active":"2024-01-26T23:29:44.617393+0000","last_became_peered":"2024-01-26T23:29:44.617393+0000","last_unstale":"2024-01-26T23:29:48.637940+0000","last_undegraded":"2024-01-26T23:29:48.637940+0000","last_fullsized":"2024-01-26T23:29:48.637940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633424+0000","last_change":"2024-01-26T23:29:44.614095+0000","last_active":"2024-01-26T23:29:48.633424+0000","last_peered":"2024-01-26T23:29:48.633424+0000","last_clean":"2024-01-26T23:29:48.633424+0000","last_became_active":"2024-01-26T23:29:44.613758+0000","last_became_peered":"2024-01-26T23:29:44.613758+0000","last_unstale":"2024-01-26T23:29:48.633424+0000","last_undegraded":"2024-01-26T23:29:48.633424+0000","last_fullsized":"2024-01-26T23:29:48.633424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:22.392135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636817+0000","last_change":"2024-01-26T23:29:44.624158+0000","last_active":"2024-01-26T23:29:48.636817+0000","last_peered":"2024-01-26T23:29:48.636817+0000","last_clean":"2024-01-26T23:29:48.636817+0000","last_became_active":"2024-01-26T23:29:44.623876+0000","last_became_peered":"2024-01-26T23:29:44.623876+0000","last_unstale":"2024-01-26T23:29:48.636817+0000","last_undegraded":"2024-01-26T23:29:48.636817+0000","last_fullsized":"2024-01-26T23:29:48.636817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631396+0000","last_change":"2024-01-26T23:29:44.618895+0000","last_active":"2024-01-26T23:29:48.631396+0000","last_peered":"2024-01-26T23:29:48.631396+0000","last_clean":"2024-01-26T23:29:48.631396+0000","last_became_active":"2024-01-26T23:29:44.618505+0000","last_became_peered":"2024-01-26T23:29:44.618505+0000","last_unstale":"2024-01-26T23:29:48.631396+0000","last_undegraded":"2024-01-26T23:29:48.631396+0000","last_fullsized":"2024-01-26T23:29:48.631396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:59:47.867303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.625861+0000","last_change":"2024-01-26T23:29:44.612783+0000","last_active":"2024-01-26T23:29:48.625861+0000","last_peered":"2024-01-26T23:29:48.625861+0000","last_clean":"2024-01-26T23:29:48.625861+0000","last_became_active":"2024-01-26T23:29:44.612451+0000","last_became_peered":"2024-01-26T23:29:44.612451+0000","last_unstale":"2024-01-26T23:29:48.625861+0000","last_undegraded":"2024-01-26T23:29:48.625861+0000","last_fullsized":"2024-01-26T23:29:48.625861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:12.500308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632657+0000","last_change":"2024-01-26T23:29:44.630662+0000","last_active":"2024-01-26T23:29:48.632657+0000","last_peered":"2024-01-26T23:29:48.632657+0000","last_clean":"2024-01-26T23:29:48.632657+0000","last_became_active":"2024-01-26T23:29:44.630412+0000","last_became_peered":"2024-01-26T23:29:44.630412+0000","last_unstale":"2024-01-26T23:29:48.632657+0000","last_undegraded":"2024-01-26T23:29:48.632657+0000","last_fullsized":"2024-01-26T23:29:48.632657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636581+0000","last_change":"2024-01-26T23:29:44.624186+0000","last_active":"2024-01-26T23:29:48.636581+0000","last_peered":"2024-01-26T23:29:48.636581+0000","last_clean":"2024-01-26T23:29:48.636581+0000","last_became_active":"2024-01-26T23:29:44.623729+0000","last_became_peered":"2024-01-26T23:29:44.623729+0000","last_unstale":"2024-01-26T23:29:48.636581+0000","last_undegraded":"2024-01-26T23:29:48.636581+0000","last_fullsized":"2024-01-26T23:29:48.636581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:20.138786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633462+0000","last_change":"2024-01-26T23:29:44.615801+0000","last_active":"2024-01-26T23:29:48.633462+0000","last_peered":"2024-01-26T23:29:48.633462+0000","last_clean":"2024-01-26T23:29:48.633462+0000","last_became_active":"2024-01-26T23:29:44.614126+0000","last_became_peered":"2024-01-26T23:29:44.614126+0000","last_unstale":"2024-01-26T23:29:48.633462+0000","last_undegraded":"2024-01-26T23:29:48.633462+0000","last_fullsized":"2024-01-26T23:29:48.633462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:22.392135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690472+0000","last_change":"2024-01-26T23:29:44.620718+0000","last_active":"2024-01-26T23:29:48.690472+0000","last_peered":"2024-01-26T23:29:48.690472+0000","last_clean":"2024-01-26T23:29:48.690472+0000","last_became_active":"2024-01-26T23:29:44.620452+0000","last_became_peered":"2024-01-26T23:29:44.620452+0000","last_unstale":"2024-01-26T23:29:48.690472+0000","last_undegraded":"2024-01-26T23:29:48.690472+0000","last_fullsized":"2024-01-26T23:29:48.690472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:01.636855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.697979+0000","last_change":"2024-01-26T23:29:48.697204+0000","last_active":"2024-01-26T23:29:48.697979+0000","last_peered":"2024-01-26T23:29:48.697979+0000","last_clean":"2024-01-26T23:29:48.697979+0000","last_became_active":"2024-01-26T23:29:48.696727+0000","last_became_peered":"2024-01-26T23:29:48.696727+0000","last_unstale":"2024-01-26T23:29:48.697979+0000","last_undegraded":"2024-01-26T23:29:48.697979+0000","last_fullsized":"2024-01-26T23:29:48.697979+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:59.257622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633767+0000","last_change":"2024-01-26T23:29:43.611980+0000","last_active":"2024-01-26T23:29:48.633767+0000","last_peered":"2024-01-26T23:29:48.633767+0000","last_clean":"2024-01-26T23:29:48.633767+0000","last_became_active":"2024-01-26T23:29:43.611649+0000","last_became_peered":"2024-01-26T23:29:43.611649+0000","last_unstale":"2024-01-26T23:29:48.633767+0000","last_undegraded":"2024-01-26T23:29:48.633767+0000","last_fullsized":"2024-01-26T23:29:48.633767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:31.317009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637215+0000","last_change":"2024-01-26T23:29:44.617315+0000","last_active":"2024-01-26T23:29:48.637215+0000","last_peered":"2024-01-26T23:29:48.637215+0000","last_clean":"2024-01-26T23:29:48.637215+0000","last_became_active":"2024-01-26T23:29:44.617020+0000","last_became_peered":"2024-01-26T23:29:44.617020+0000","last_unstale":"2024-01-26T23:29:48.637215+0000","last_undegraded":"2024-01-26T23:29:48.637215+0000","last_fullsized":"2024-01-26T23:29:48.637215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:21:47.569396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652507+0000","last_change":"2024-01-26T23:29:48.647630+0000","last_active":"2024-01-26T23:29:48.652507+0000","last_peered":"2024-01-26T23:29:48.652507+0000","last_clean":"2024-01-26T23:29:48.652507+0000","last_became_active":"2024-01-26T23:29:48.646094+0000","last_became_peered":"2024-01-26T23:29:48.646094+0000","last_unstale":"2024-01-26T23:29:48.652507+0000","last_undegraded":"2024-01-26T23:29:48.652507+0000","last_fullsized":"2024-01-26T23:29:48.652507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:20.380384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637332+0000","last_change":"2024-01-26T23:29:43.616917+0000","last_active":"2024-01-26T23:29:48.637332+0000","last_peered":"2024-01-26T23:29:48.637332+0000","last_clean":"2024-01-26T23:29:48.637332+0000","last_became_active":"2024-01-26T23:29:43.616606+0000","last_became_peered":"2024-01-26T23:29:43.616606+0000","last_unstale":"2024-01-26T23:29:48.637332+0000","last_undegraded":"2024-01-26T23:29:48.637332+0000","last_fullsized":"2024-01-26T23:29:48.637332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:54:53.198685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633652+0000","last_change":"2024-01-26T23:29:44.626223+0000","last_active":"2024-01-26T23:29:48.633652+0000","last_peered":"2024-01-26T23:29:48.633652+0000","last_clean":"2024-01-26T23:29:48.633652+0000","last_became_active":"2024-01-26T23:29:44.625830+0000","last_became_peered":"2024-01-26T23:29:44.625830+0000","last_unstale":"2024-01-26T23:29:48.633652+0000","last_undegraded":"2024-01-26T23:29:48.633652+0000","last_fullsized":"2024-01-26T23:29:48.633652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652505+0000","last_change":"2024-01-26T23:29:48.647450+0000","last_active":"2024-01-26T23:29:48.652505+0000","last_peered":"2024-01-26T23:29:48.652505+0000","last_clean":"2024-01-26T23:29:48.652505+0000","last_became_active":"2024-01-26T23:29:48.646159+0000","last_became_peered":"2024-01-26T23:29:48.646159+0000","last_unstale":"2024-01-26T23:29:48.652505+0000","last_undegraded":"2024-01-26T23:29:48.652505+0000","last_fullsized":"2024-01-26T23:29:48.652505+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:13:54.119091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690197+0000","last_change":"2024-01-26T23:29:43.624480+0000","last_active":"2024-01-26T23:29:48.690197+0000","last_peered":"2024-01-26T23:29:48.690197+0000","last_clean":"2024-01-26T23:29:48.690197+0000","last_became_active":"2024-01-26T23:29:43.624225+0000","last_became_peered":"2024-01-26T23:29:43.624225+0000","last_unstale":"2024-01-26T23:29:48.690197+0000","last_undegraded":"2024-01-26T23:29:48.690197+0000","last_fullsized":"2024-01-26T23:29:48.690197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:21.390599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636199+0000","last_change":"2024-01-26T23:29:44.621996+0000","last_active":"2024-01-26T23:29:48.636199+0000","last_peered":"2024-01-26T23:29:48.636199+0000","last_clean":"2024-01-26T23:29:48.636199+0000","last_became_active":"2024-01-26T23:29:44.621750+0000","last_became_peered":"2024-01-26T23:29:44.621750+0000","last_unstale":"2024-01-26T23:29:48.636199+0000","last_undegraded":"2024-01-26T23:29:48.636199+0000","last_fullsized":"2024-01-26T23:29:48.636199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:32.912164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652475+0000","last_change":"2024-01-26T23:29:48.647439+0000","last_active":"2024-01-26T23:29:48.652475+0000","last_peered":"2024-01-26T23:29:48.652475+0000","last_clean":"2024-01-26T23:29:48.652475+0000","last_became_active":"2024-01-26T23:29:48.647004+0000","last_became_peered":"2024-01-26T23:29:48.647004+0000","last_unstale":"2024-01-26T23:29:48.652475+0000","last_undegraded":"2024-01-26T23:29:48.652475+0000","last_fullsized":"2024-01-26T23:29:48.652475+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:29:51.471531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690167+0000","last_change":"2024-01-26T23:29:43.623440+0000","last_active":"2024-01-26T23:29:48.690167+0000","last_peered":"2024-01-26T23:29:48.690167+0000","last_clean":"2024-01-26T23:29:48.690167+0000","last_became_active":"2024-01-26T23:29:43.623105+0000","last_became_peered":"2024-01-26T23:29:43.623105+0000","last_unstale":"2024-01-26T23:29:48.690167+0000","last_undegraded":"2024-01-26T23:29:48.690167+0000","last_fullsized":"2024-01-26T23:29:48.690167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:37:22.647734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636762+0000","last_change":"2024-01-26T23:29:44.622275+0000","last_active":"2024-01-26T23:29:48.636762+0000","last_peered":"2024-01-26T23:29:48.636762+0000","last_clean":"2024-01-26T23:29:48.636762+0000","last_became_active":"2024-01-26T23:29:44.621687+0000","last_became_peered":"2024-01-26T23:29:44.621687+0000","last_unstale":"2024-01-26T23:29:48.636762+0000","last_undegraded":"2024-01-26T23:29:48.636762+0000","last_fullsized":"2024-01-26T23:29:48.636762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:12.500308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.653689+0000","last_change":"2024-01-26T23:29:48.652656+0000","last_active":"2024-01-26T23:29:48.653689+0000","last_peered":"2024-01-26T23:29:48.653689+0000","last_clean":"2024-01-26T23:29:48.653689+0000","last_became_active":"2024-01-26T23:29:48.652264+0000","last_became_peered":"2024-01-26T23:29:48.652264+0000","last_unstale":"2024-01-26T23:29:48.653689+0000","last_undegraded":"2024-01-26T23:29:48.653689+0000","last_fullsized":"2024-01-26T23:29:48.653689+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:35:23.165011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636924+0000","last_change":"2024-01-26T23:29:43.620505+0000","last_active":"2024-01-26T23:29:48.636924+0000","last_peered":"2024-01-26T23:29:48.636924+0000","last_clean":"2024-01-26T23:29:48.636924+0000","last_became_active":"2024-01-26T23:29:43.620202+0000","last_became_peered":"2024-01-26T23:29:43.620202+0000","last_unstale":"2024-01-26T23:29:48.636924+0000","last_undegraded":"2024-01-26T23:29:48.636924+0000","last_fullsized":"2024-01-26T23:29:48.636924+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:31.317009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633949+0000","last_change":"2024-01-26T23:29:44.615753+0000","last_active":"2024-01-26T23:29:48.633949+0000","last_peered":"2024-01-26T23:29:48.633949+0000","last_clean":"2024-01-26T23:29:48.633949+0000","last_became_active":"2024-01-26T23:29:44.615469+0000","last_became_peered":"2024-01-26T23:29:44.615469+0000","last_unstale":"2024-01-26T23:29:48.633949+0000","last_undegraded":"2024-01-26T23:29:48.633949+0000","last_fullsized":"2024-01-26T23:29:48.633949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:37:23.649270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.651391+0000","last_change":"2024-01-26T23:29:48.646032+0000","last_active":"2024-01-26T23:29:48.651391+0000","last_peered":"2024-01-26T23:29:48.651391+0000","last_clean":"2024-01-26T23:29:48.651391+0000","last_became_active":"2024-01-26T23:29:48.645627+0000","last_became_peered":"2024-01-26T23:29:48.645627+0000","last_unstale":"2024-01-26T23:29:48.651391+0000","last_undegraded":"2024-01-26T23:29:48.651391+0000","last_fullsized":"2024-01-26T23:29:48.651391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:42.473075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632782+0000","last_change":"2024-01-26T23:29:43.614220+0000","last_active":"2024-01-26T23:29:48.632782+0000","last_peered":"2024-01-26T23:29:48.632782+0000","last_clean":"2024-01-26T23:29:48.632782+0000","last_became_active":"2024-01-26T23:29:43.613886+0000","last_became_peered":"2024-01-26T23:29:43.613886+0000","last_unstale":"2024-01-26T23:29:48.632782+0000","last_undegraded":"2024-01-26T23:29:48.632782+0000","last_fullsized":"2024-01-26T23:29:48.632782+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:09:46.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632076+0000","last_change":"2024-01-26T23:29:44.614249+0000","last_active":"2024-01-26T23:29:48.632076+0000","last_peered":"2024-01-26T23:29:48.632076+0000","last_clean":"2024-01-26T23:29:48.632076+0000","last_became_active":"2024-01-26T23:29:44.613949+0000","last_became_peered":"2024-01-26T23:29:44.613949+0000","last_unstale":"2024-01-26T23:29:48.632076+0000","last_undegraded":"2024-01-26T23:29:48.632076+0000","last_fullsized":"2024-01-26T23:29:48.632076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:14.555745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.651517+0000","last_change":"2024-01-26T23:29:48.647625+0000","last_active":"2024-01-26T23:29:48.651517+0000","last_peered":"2024-01-26T23:29:48.651517+0000","last_clean":"2024-01-26T23:29:48.651517+0000","last_became_active":"2024-01-26T23:29:48.647273+0000","last_became_peered":"2024-01-26T23:29:48.647273+0000","last_unstale":"2024-01-26T23:29:48.651517+0000","last_undegraded":"2024-01-26T23:29:48.651517+0000","last_fullsized":"2024-01-26T23:29:48.651517+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:52:44.817581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636975+0000","last_change":"2024-01-26T23:29:43.616939+0000","last_active":"2024-01-26T23:29:48.636975+0000","last_peered":"2024-01-26T23:29:48.636975+0000","last_clean":"2024-01-26T23:29:48.636975+0000","last_became_active":"2024-01-26T23:29:43.614953+0000","last_became_peered":"2024-01-26T23:29:43.614953+0000","last_unstale":"2024-01-26T23:29:48.636975+0000","last_undegraded":"2024-01-26T23:29:48.636975+0000","last_fullsized":"2024-01-26T23:29:48.636975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:36.035429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636683+0000","last_change":"2024-01-26T23:29:44.621908+0000","last_active":"2024-01-26T23:29:48.636683+0000","last_peered":"2024-01-26T23:29:48.636683+0000","last_clean":"2024-01-26T23:29:48.636683+0000","last_became_active":"2024-01-26T23:29:44.621706+0000","last_became_peered":"2024-01-26T23:29:44.621706+0000","last_unstale":"2024-01-26T23:29:48.636683+0000","last_undegraded":"2024-01-26T23:29:48.636683+0000","last_fullsized":"2024-01-26T23:29:48.636683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696535+0000","last_change":"2024-01-26T23:29:48.695257+0000","last_active":"2024-01-26T23:29:48.696535+0000","last_peered":"2024-01-26T23:29:48.696535+0000","last_clean":"2024-01-26T23:29:48.696535+0000","last_became_active":"2024-01-26T23:29:48.694835+0000","last_became_peered":"2024-01-26T23:29:48.694835+0000","last_unstale":"2024-01-26T23:29:48.696535+0000","last_undegraded":"2024-01-26T23:29:48.696535+0000","last_fullsized":"2024-01-26T23:29:48.696535+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:53:55.279661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632759+0000","last_change":"2024-01-26T23:29:43.617477+0000","last_active":"2024-01-26T23:29:48.632759+0000","last_peered":"2024-01-26T23:29:48.632759+0000","last_clean":"2024-01-26T23:29:48.632759+0000","last_became_active":"2024-01-26T23:29:43.617014+0000","last_became_peered":"2024-01-26T23:29:43.617014+0000","last_unstale":"2024-01-26T23:29:48.632759+0000","last_undegraded":"2024-01-26T23:29:48.632759+0000","last_fullsized":"2024-01-26T23:29:48.632759+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:32.417441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626943+0000","last_change":"2024-01-26T23:29:44.619133+0000","last_active":"2024-01-26T23:29:48.626943+0000","last_peered":"2024-01-26T23:29:48.626943+0000","last_clean":"2024-01-26T23:29:48.626943+0000","last_became_active":"2024-01-26T23:29:44.618827+0000","last_became_peered":"2024-01-26T23:29:44.618827+0000","last_unstale":"2024-01-26T23:29:48.626943+0000","last_undegraded":"2024-01-26T23:29:48.626943+0000","last_fullsized":"2024-01-26T23:29:48.626943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696639+0000","last_change":"2024-01-26T23:29:48.693971+0000","last_active":"2024-01-26T23:29:48.696639+0000","last_peered":"2024-01-26T23:29:48.696639+0000","last_clean":"2024-01-26T23:29:48.696639+0000","last_became_active":"2024-01-26T23:29:48.693563+0000","last_became_peered":"2024-01-26T23:29:48.693563+0000","last_unstale":"2024-01-26T23:29:48.696639+0000","last_undegraded":"2024-01-26T23:29:48.696639+0000","last_fullsized":"2024-01-26T23:29:48.696639+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:42:42.888059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.638779+0000","last_change":"2024-01-26T23:29:43.622313+0000","last_active":"2024-01-26T23:29:48.638779+0000","last_peered":"2024-01-26T23:29:48.638779+0000","last_clean":"2024-01-26T23:29:48.638779+0000","last_became_active":"2024-01-26T23:29:43.622008+0000","last_became_peered":"2024-01-26T23:29:43.622008+0000","last_unstale":"2024-01-26T23:29:48.638779+0000","last_undegraded":"2024-01-26T23:29:48.638779+0000","last_fullsized":"2024-01-26T23:29:48.638779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:37:22.647734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.634360+0000","last_change":"2024-01-26T23:29:44.619813+0000","last_active":"2024-01-26T23:29:48.634360+0000","last_peered":"2024-01-26T23:29:48.634360+0000","last_clean":"2024-01-26T23:29:48.634360+0000","last_became_active":"2024-01-26T23:29:44.619466+0000","last_became_peered":"2024-01-26T23:29:44.619466+0000","last_unstale":"2024-01-26T23:29:48.634360+0000","last_undegraded":"2024-01-26T23:29:48.634360+0000","last_fullsized":"2024-01-26T23:29:48.634360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:48:24.085774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.653660+0000","last_change":"2024-01-26T23:29:48.652676+0000","last_active":"2024-01-26T23:29:48.653660+0000","last_peered":"2024-01-26T23:29:48.653660+0000","last_clean":"2024-01-26T23:29:48.653660+0000","last_became_active":"2024-01-26T23:29:48.652134+0000","last_became_peered":"2024-01-26T23:29:48.652134+0000","last_unstale":"2024-01-26T23:29:48.653660+0000","last_undegraded":"2024-01-26T23:29:48.653660+0000","last_fullsized":"2024-01-26T23:29:48.653660+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:27.746532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.634232+0000","last_change":"2024-01-26T23:29:43.620804+0000","last_active":"2024-01-26T23:29:48.634232+0000","last_peered":"2024-01-26T23:29:48.634232+0000","last_clean":"2024-01-26T23:29:48.634232+0000","last_became_active":"2024-01-26T23:29:43.620530+0000","last_became_peered":"2024-01-26T23:29:43.620530+0000","last_unstale":"2024-01-26T23:29:48.634232+0000","last_undegraded":"2024-01-26T23:29:48.634232+0000","last_fullsized":"2024-01-26T23:29:48.634232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:44:34.698447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.634272+0000","last_change":"2024-01-26T23:29:44.626105+0000","last_active":"2024-01-26T23:29:48.634272+0000","last_peered":"2024-01-26T23:29:48.634272+0000","last_clean":"2024-01-26T23:29:48.634272+0000","last_became_active":"2024-01-26T23:29:44.625500+0000","last_became_peered":"2024-01-26T23:29:44.625500+0000","last_unstale":"2024-01-26T23:29:48.634272+0000","last_undegraded":"2024-01-26T23:29:48.634272+0000","last_fullsized":"2024-01-26T23:29:48.634272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.644865+0000","last_change":"2024-01-26T23:29:48.643281+0000","last_active":"2024-01-26T23:29:48.644865+0000","last_peered":"2024-01-26T23:29:48.644865+0000","last_clean":"2024-01-26T23:29:48.644865+0000","last_became_active":"2024-01-26T23:29:48.642907+0000","last_became_peered":"2024-01-26T23:29:48.642907+0000","last_unstale":"2024-01-26T23:29:48.644865+0000","last_undegraded":"2024-01-26T23:29:48.644865+0000","last_fullsized":"2024-01-26T23:29:48.644865+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:24:23.994084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626216+0000","last_change":"2024-01-26T23:29:43.621999+0000","last_active":"2024-01-26T23:29:48.626216+0000","last_peered":"2024-01-26T23:29:48.626216+0000","last_clean":"2024-01-26T23:29:48.626216+0000","last_became_active":"2024-01-26T23:29:43.621214+0000","last_became_peered":"2024-01-26T23:29:43.621214+0000","last_unstale":"2024-01-26T23:29:48.626216+0000","last_undegraded":"2024-01-26T23:29:48.626216+0000","last_fullsized":"2024-01-26T23:29:48.626216+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:37:22.647734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633819+0000","last_change":"2024-01-26T23:29:44.608941+0000","last_active":"2024-01-26T23:29:48.633819+0000","last_peered":"2024-01-26T23:29:48.633819+0000","last_clean":"2024-01-26T23:29:48.633819+0000","last_became_active":"2024-01-26T23:29:44.608664+0000","last_became_peered":"2024-01-26T23:29:44.608664+0000","last_unstale":"2024-01-26T23:29:48.633819+0000","last_undegraded":"2024-01-26T23:29:48.633819+0000","last_fullsized":"2024-01-26T23:29:48.633819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:47:10.097930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.651674+0000","last_change":"2024-01-26T23:29:48.643876+0000","last_active":"2024-01-26T23:29:48.651674+0000","last_peered":"2024-01-26T23:29:48.651674+0000","last_clean":"2024-01-26T23:29:48.651674+0000","last_became_active":"2024-01-26T23:29:48.641104+0000","last_became_peered":"2024-01-26T23:29:48.641104+0000","last_unstale":"2024-01-26T23:29:48.651674+0000","last_undegraded":"2024-01-26T23:29:48.651674+0000","last_fullsized":"2024-01-26T23:29:48.651674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:29:51.471531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-01-26T23:29:54.992720+0000","last_change":"2024-01-26T23:29:43.622211+0000","last_active":"2024-01-26T23:29:54.992720+0000","last_peered":"2024-01-26T23:29:54.992720+0000","last_clean":"2024-01-26T23:29:54.992720+0000","last_became_active":"2024-01-26T23:29:43.621919+0000","last_became_peered":"2024-01-26T23:29:43.621919+0000","last_unstale":"2024-01-26T23:29:54.992720+0000","last_undegraded":"2024-01-26T23:29:54.992720+0000","last_fullsized":"2024-01-26T23:29:54.992720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:45.384500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3713,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690317+0000","last_change":"2024-01-26T23:29:44.618740+0000","last_active":"2024-01-26T23:29:48.690317+0000","last_peered":"2024-01-26T23:29:48.690317+0000","last_clean":"2024-01-26T23:29:48.690317+0000","last_became_active":"2024-01-26T23:29:44.618482+0000","last_became_peered":"2024-01-26T23:29:44.618482+0000","last_unstale":"2024-01-26T23:29:48.690317+0000","last_undegraded":"2024-01-26T23:29:48.690317+0000","last_fullsized":"2024-01-26T23:29:48.690317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:03.314483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695102+0000","last_change":"2024-01-26T23:29:48.691860+0000","last_active":"2024-01-26T23:29:48.695102+0000","last_peered":"2024-01-26T23:29:48.695102+0000","last_clean":"2024-01-26T23:29:48.695102+0000","last_became_active":"2024-01-26T23:29:48.691440+0000","last_became_peered":"2024-01-26T23:29:48.691440+0000","last_unstale":"2024-01-26T23:29:48.695102+0000","last_undegraded":"2024-01-26T23:29:48.695102+0000","last_fullsized":"2024-01-26T23:29:48.695102+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:20.246494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632364+0000","last_change":"2024-01-26T23:29:43.613477+0000","last_active":"2024-01-26T23:29:48.632364+0000","last_peered":"2024-01-26T23:29:48.632364+0000","last_clean":"2024-01-26T23:29:48.632364+0000","last_became_active":"2024-01-26T23:29:43.613228+0000","last_became_peered":"2024-01-26T23:29:43.613228+0000","last_unstale":"2024-01-26T23:29:48.632364+0000","last_undegraded":"2024-01-26T23:29:48.632364+0000","last_fullsized":"2024-01-26T23:29:48.632364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:54:53.198685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631483+0000","last_change":"2024-01-26T23:29:44.613649+0000","last_active":"2024-01-26T23:29:48.631483+0000","last_peered":"2024-01-26T23:29:48.631483+0000","last_clean":"2024-01-26T23:29:48.631483+0000","last_became_active":"2024-01-26T23:29:44.613278+0000","last_became_peered":"2024-01-26T23:29:44.613278+0000","last_unstale":"2024-01-26T23:29:48.631483+0000","last_undegraded":"2024-01-26T23:29:48.631483+0000","last_fullsized":"2024-01-26T23:29:48.631483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:38.452904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696548+0000","last_change":"2024-01-26T23:29:48.691601+0000","last_active":"2024-01-26T23:29:48.696548+0000","last_peered":"2024-01-26T23:29:48.696548+0000","last_clean":"2024-01-26T23:29:48.696548+0000","last_became_active":"2024-01-26T23:29:48.691169+0000","last_became_peered":"2024-01-26T23:29:48.691169+0000","last_unstale":"2024-01-26T23:29:48.696548+0000","last_undegraded":"2024-01-26T23:29:48.696548+0000","last_fullsized":"2024-01-26T23:29:48.696548+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:09.701112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635763+0000","last_change":"2024-01-26T23:29:43.622165+0000","last_active":"2024-01-26T23:29:48.635763+0000","last_peered":"2024-01-26T23:29:48.635763+0000","last_clean":"2024-01-26T23:29:48.635763+0000","last_became_active":"2024-01-26T23:29:43.621876+0000","last_became_peered":"2024-01-26T23:29:43.621876+0000","last_unstale":"2024-01-26T23:29:48.635763+0000","last_undegraded":"2024-01-26T23:29:48.635763+0000","last_fullsized":"2024-01-26T23:29:48.635763+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:30.053494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690214+0000","last_change":"2024-01-26T23:29:44.615999+0000","last_active":"2024-01-26T23:29:48.690214+0000","last_peered":"2024-01-26T23:29:48.690214+0000","last_clean":"2024-01-26T23:29:48.690214+0000","last_became_active":"2024-01-26T23:29:44.615376+0000","last_became_peered":"2024-01-26T23:29:44.615376+0000","last_unstale":"2024-01-26T23:29:48.690214+0000","last_undegraded":"2024-01-26T23:29:48.690214+0000","last_fullsized":"2024-01-26T23:29:48.690214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:32.912164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696674+0000","last_change":"2024-01-26T23:29:48.694846+0000","last_active":"2024-01-26T23:29:48.696674+0000","last_peered":"2024-01-26T23:29:48.696674+0000","last_clean":"2024-01-26T23:29:48.696674+0000","last_became_active":"2024-01-26T23:29:48.694439+0000","last_became_peered":"2024-01-26T23:29:48.694439+0000","last_unstale":"2024-01-26T23:29:48.696674+0000","last_undegraded":"2024-01-26T23:29:48.696674+0000","last_fullsized":"2024-01-26T23:29:48.696674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:39:25.688252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632550+0000","last_change":"2024-01-26T23:29:43.613829+0000","last_active":"2024-01-26T23:29:48.632550+0000","last_peered":"2024-01-26T23:29:48.632550+0000","last_clean":"2024-01-26T23:29:48.632550+0000","last_became_active":"2024-01-26T23:29:43.613486+0000","last_became_peered":"2024-01-26T23:29:43.613486+0000","last_unstale":"2024-01-26T23:29:48.632550+0000","last_undegraded":"2024-01-26T23:29:48.632550+0000","last_fullsized":"2024-01-26T23:29:48.632550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:44:34.698447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633466+0000","last_change":"2024-01-26T23:29:44.614404+0000","last_active":"2024-01-26T23:29:48.633466+0000","last_peered":"2024-01-26T23:29:48.633466+0000","last_clean":"2024-01-26T23:29:48.633466+0000","last_became_active":"2024-01-26T23:29:44.614127+0000","last_became_peered":"2024-01-26T23:29:44.614127+0000","last_unstale":"2024-01-26T23:29:48.633466+0000","last_undegraded":"2024-01-26T23:29:48.633466+0000","last_fullsized":"2024-01-26T23:29:48.633466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:35:11.068882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.653735+0000","last_change":"2024-01-26T23:29:48.647303+0000","last_active":"2024-01-26T23:29:48.653735+0000","last_peered":"2024-01-26T23:29:48.653735+0000","last_clean":"2024-01-26T23:29:48.653735+0000","last_became_active":"2024-01-26T23:29:48.646893+0000","last_became_peered":"2024-01-26T23:29:48.646893+0000","last_unstale":"2024-01-26T23:29:48.653735+0000","last_undegraded":"2024-01-26T23:29:48.653735+0000","last_fullsized":"2024-01-26T23:29:48.653735+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:49.464111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631121+0000","last_change":"2024-01-26T23:29:43.610693+0000","last_active":"2024-01-26T23:29:48.631121+0000","last_peered":"2024-01-26T23:29:48.631121+0000","last_clean":"2024-01-26T23:29:48.631121+0000","last_became_active":"2024-01-26T23:29:43.610370+0000","last_became_peered":"2024-01-26T23:29:43.610370+0000","last_unstale":"2024-01-26T23:29:48.631121+0000","last_undegraded":"2024-01-26T23:29:48.631121+0000","last_fullsized":"2024-01-26T23:29:48.631121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:09:46.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.692648+0000","last_change":"2024-01-26T23:29:44.611702+0000","last_active":"2024-01-26T23:29:48.692648+0000","last_peered":"2024-01-26T23:29:48.692648+0000","last_clean":"2024-01-26T23:29:48.692648+0000","last_became_active":"2024-01-26T23:29:44.611378+0000","last_became_peered":"2024-01-26T23:29:44.611378+0000","last_unstale":"2024-01-26T23:29:48.692648+0000","last_undegraded":"2024-01-26T23:29:48.692648+0000","last_fullsized":"2024-01-26T23:29:48.692648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:38:52.041592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.644666+0000","last_change":"2024-01-26T23:29:48.643205+0000","last_active":"2024-01-26T23:29:48.644666+0000","last_peered":"2024-01-26T23:29:48.644666+0000","last_clean":"2024-01-26T23:29:48.644666+0000","last_became_active":"2024-01-26T23:29:48.642819+0000","last_became_peered":"2024-01-26T23:29:48.642819+0000","last_unstale":"2024-01-26T23:29:48.644666+0000","last_undegraded":"2024-01-26T23:29:48.644666+0000","last_fullsized":"2024-01-26T23:29:48.644666+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:51.210803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633091+0000","last_change":"2024-01-26T23:29:43.612609+0000","last_active":"2024-01-26T23:29:48.633091+0000","last_peered":"2024-01-26T23:29:48.633091+0000","last_clean":"2024-01-26T23:29:48.633091+0000","last_became_active":"2024-01-26T23:29:43.612318+0000","last_became_peered":"2024-01-26T23:29:43.612318+0000","last_unstale":"2024-01-26T23:29:48.633091+0000","last_undegraded":"2024-01-26T23:29:48.633091+0000","last_fullsized":"2024-01-26T23:29:48.633091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:31.317009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690843+0000","last_change":"2024-01-26T23:29:44.616632+0000","last_active":"2024-01-26T23:29:48.690843+0000","last_peered":"2024-01-26T23:29:48.690843+0000","last_clean":"2024-01-26T23:29:48.690843+0000","last_became_active":"2024-01-26T23:29:44.616280+0000","last_became_peered":"2024-01-26T23:29:44.616280+0000","last_unstale":"2024-01-26T23:29:48.690843+0000","last_undegraded":"2024-01-26T23:29:48.690843+0000","last_fullsized":"2024-01-26T23:29:48.690843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:48:24.085774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.654458+0000","last_change":"2024-01-26T23:29:48.644846+0000","last_active":"2024-01-26T23:29:48.654458+0000","last_peered":"2024-01-26T23:29:48.654458+0000","last_clean":"2024-01-26T23:29:48.654458+0000","last_became_active":"2024-01-26T23:29:48.644359+0000","last_became_peered":"2024-01-26T23:29:48.644359+0000","last_unstale":"2024-01-26T23:29:48.654458+0000","last_undegraded":"2024-01-26T23:29:48.654458+0000","last_fullsized":"2024-01-26T23:29:48.654458+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:59.257622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626696+0000","last_change":"2024-01-26T23:29:43.605084+0000","last_active":"2024-01-26T23:29:48.626696+0000","last_peered":"2024-01-26T23:29:48.626696+0000","last_clean":"2024-01-26T23:29:48.626696+0000","last_became_active":"2024-01-26T23:29:43.604761+0000","last_became_peered":"2024-01-26T23:29:43.604761+0000","last_unstale":"2024-01-26T23:29:48.626696+0000","last_undegraded":"2024-01-26T23:29:48.626696+0000","last_fullsized":"2024-01-26T23:29:48.626696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:44:34.698447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636936+0000","last_change":"2024-01-26T23:29:44.632161+0000","last_active":"2024-01-26T23:29:48.636936+0000","last_peered":"2024-01-26T23:29:48.636936+0000","last_clean":"2024-01-26T23:29:48.636936+0000","last_became_active":"2024-01-26T23:29:44.631945+0000","last_became_peered":"2024-01-26T23:29:44.631945+0000","last_unstale":"2024-01-26T23:29:48.636936+0000","last_undegraded":"2024-01-26T23:29:48.636936+0000","last_fullsized":"2024-01-26T23:29:48.636936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:03.314483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.654064+0000","last_change":"2024-01-26T23:29:48.647375+0000","last_active":"2024-01-26T23:29:48.654064+0000","last_peered":"2024-01-26T23:29:48.654064+0000","last_clean":"2024-01-26T23:29:48.654064+0000","last_became_active":"2024-01-26T23:29:48.646978+0000","last_became_peered":"2024-01-26T23:29:48.646978+0000","last_unstale":"2024-01-26T23:29:48.654064+0000","last_undegraded":"2024-01-26T23:29:48.654064+0000","last_fullsized":"2024-01-26T23:29:48.654064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:20.246494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635696+0000","last_change":"2024-01-26T23:29:43.620942+0000","last_active":"2024-01-26T23:29:48.635696+0000","last_peered":"2024-01-26T23:29:48.635696+0000","last_clean":"2024-01-26T23:29:48.635696+0000","last_became_active":"2024-01-26T23:29:43.620644+0000","last_became_peered":"2024-01-26T23:29:43.620644+0000","last_unstale":"2024-01-26T23:29:48.635696+0000","last_undegraded":"2024-01-26T23:29:48.635696+0000","last_fullsized":"2024-01-26T23:29:48.635696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:45.384500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.634352+0000","last_change":"2024-01-26T23:29:44.605909+0000","last_active":"2024-01-26T23:29:48.634352+0000","last_peered":"2024-01-26T23:29:48.634352+0000","last_clean":"2024-01-26T23:29:48.634352+0000","last_became_active":"2024-01-26T23:29:44.605652+0000","last_became_peered":"2024-01-26T23:29:44.605652+0000","last_unstale":"2024-01-26T23:29:48.634352+0000","last_undegraded":"2024-01-26T23:29:48.634352+0000","last_fullsized":"2024-01-26T23:29:48.634352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:31.055030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.654078+0000","last_change":"2024-01-26T23:29:48.647396+0000","last_active":"2024-01-26T23:29:48.654078+0000","last_peered":"2024-01-26T23:29:48.654078+0000","last_clean":"2024-01-26T23:29:48.654078+0000","last_became_active":"2024-01-26T23:29:48.646950+0000","last_became_peered":"2024-01-26T23:29:48.646950+0000","last_unstale":"2024-01-26T23:29:48.654078+0000","last_undegraded":"2024-01-26T23:29:48.654078+0000","last_fullsized":"2024-01-26T23:29:48.654078+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:51.210803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.625864+0000","last_change":"2024-01-26T23:29:44.617670+0000","last_active":"2024-01-26T23:29:48.625864+0000","last_peered":"2024-01-26T23:29:48.625864+0000","last_clean":"2024-01-26T23:29:48.625864+0000","last_became_active":"2024-01-26T23:29:44.616510+0000","last_became_peered":"2024-01-26T23:29:44.616510+0000","last_unstale":"2024-01-26T23:29:48.625864+0000","last_undegraded":"2024-01-26T23:29:48.625864+0000","last_fullsized":"2024-01-26T23:29:48.625864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:12:04.973733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631229+0000","last_change":"2024-01-26T23:29:43.618892+0000","last_active":"2024-01-26T23:29:48.631229+0000","last_peered":"2024-01-26T23:29:48.631229+0000","last_clean":"2024-01-26T23:29:48.631229+0000","last_became_active":"2024-01-26T23:29:43.618446+0000","last_became_peered":"2024-01-26T23:29:43.618446+0000","last_unstale":"2024-01-26T23:29:48.631229+0000","last_undegraded":"2024-01-26T23:29:48.631229+0000","last_fullsized":"2024-01-26T23:29:48.631229+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:36.035429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.694860+0000","last_change":"2024-01-26T23:29:48.691567+0000","last_active":"2024-01-26T23:29:48.694860+0000","last_peered":"2024-01-26T23:29:48.694860+0000","last_clean":"2024-01-26T23:29:48.694860+0000","last_became_active":"2024-01-26T23:29:48.691153+0000","last_became_peered":"2024-01-26T23:29:48.691153+0000","last_unstale":"2024-01-26T23:29:48.694860+0000","last_undegraded":"2024-01-26T23:29:48.694860+0000","last_fullsized":"2024-01-26T23:29:48.694860+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:49.464111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637181+0000","last_change":"2024-01-26T23:29:44.620961+0000","last_active":"2024-01-26T23:29:48.637181+0000","last_peered":"2024-01-26T23:29:48.637181+0000","last_clean":"2024-01-26T23:29:48.637181+0000","last_became_active":"2024-01-26T23:29:44.620701+0000","last_became_peered":"2024-01-26T23:29:44.620701+0000","last_unstale":"2024-01-26T23:29:48.637181+0000","last_undegraded":"2024-01-26T23:29:48.637181+0000","last_fullsized":"2024-01-26T23:29:48.637181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:32.912164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635054+0000","last_change":"2024-01-26T23:29:43.620894+0000","last_active":"2024-01-26T23:29:48.635054+0000","last_peered":"2024-01-26T23:29:48.635054+0000","last_clean":"2024-01-26T23:29:48.635054+0000","last_became_active":"2024-01-26T23:29:43.620603+0000","last_became_peered":"2024-01-26T23:29:43.620603+0000","last_unstale":"2024-01-26T23:29:48.635054+0000","last_undegraded":"2024-01-26T23:29:48.635054+0000","last_fullsized":"2024-01-26T23:29:48.635054+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:36.035429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.689568+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:29:48.689568+0000","last_peered":"2024-01-26T23:29:48.689568+0000","last_clean":"2024-01-26T23:29:48.689568+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:29:48.689568+0000","last_undegraded":"2024-01-26T23:29:48.689568+0000","last_fullsized":"2024-01-26T23:29:48.689568+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696515+0000","last_change":"2024-01-26T23:29:48.692933+0000","last_active":"2024-01-26T23:29:48.696515+0000","last_peered":"2024-01-26T23:29:48.696515+0000","last_clean":"2024-01-26T23:29:48.696515+0000","last_became_active":"2024-01-26T23:29:48.692576+0000","last_became_peered":"2024-01-26T23:29:48.692576+0000","last_unstale":"2024-01-26T23:29:48.696515+0000","last_undegraded":"2024-01-26T23:29:48.696515+0000","last_fullsized":"2024-01-26T23:29:48.696515+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:24:23.994084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631114+0000","last_change":"2024-01-26T23:29:44.601856+0000","last_active":"2024-01-26T23:29:48.631114+0000","last_peered":"2024-01-26T23:29:48.631114+0000","last_clean":"2024-01-26T23:29:48.631114+0000","last_became_active":"2024-01-26T23:29:44.601367+0000","last_became_peered":"2024-01-26T23:29:44.601367+0000","last_unstale":"2024-01-26T23:29:48.631114+0000","last_undegraded":"2024-01-26T23:29:48.631114+0000","last_fullsized":"2024-01-26T23:29:48.631114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:21:47.569396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.689624+0000","last_change":"2024-01-26T23:29:43.623719+0000","last_active":"2024-01-26T23:29:48.689624+0000","last_peered":"2024-01-26T23:29:48.689624+0000","last_clean":"2024-01-26T23:29:48.689624+0000","last_became_active":"2024-01-26T23:29:43.623493+0000","last_became_peered":"2024-01-26T23:29:43.623493+0000","last_unstale":"2024-01-26T23:29:48.689624+0000","last_undegraded":"2024-01-26T23:29:48.689624+0000","last_fullsized":"2024-01-26T23:29:48.689624+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:23:25.049333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695757+0000","last_change":"2024-01-26T23:29:48.691446+0000","last_active":"2024-01-26T23:29:48.695757+0000","last_peered":"2024-01-26T23:29:48.695757+0000","last_clean":"2024-01-26T23:29:48.695757+0000","last_became_active":"2024-01-26T23:29:48.691005+0000","last_became_peered":"2024-01-26T23:29:48.691005+0000","last_unstale":"2024-01-26T23:29:48.695757+0000","last_undegraded":"2024-01-26T23:29:48.695757+0000","last_fullsized":"2024-01-26T23:29:48.695757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:51.210803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690165+0000","last_change":"2024-01-26T23:29:43.624304+0000","last_active":"2024-01-26T23:29:48.690165+0000","last_peered":"2024-01-26T23:29:48.690165+0000","last_clean":"2024-01-26T23:29:48.690165+0000","last_became_active":"2024-01-26T23:29:43.624031+0000","last_became_peered":"2024-01-26T23:29:43.624031+0000","last_unstale":"2024-01-26T23:29:48.690165+0000","last_undegraded":"2024-01-26T23:29:48.690165+0000","last_fullsized":"2024-01-26T23:29:48.690165+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:32.417441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626845+0000","last_change":"2024-01-26T23:29:44.615706+0000","last_active":"2024-01-26T23:29:48.626845+0000","last_peered":"2024-01-26T23:29:48.626845+0000","last_clean":"2024-01-26T23:29:48.626845+0000","last_became_active":"2024-01-26T23:29:44.615377+0000","last_became_peered":"2024-01-26T23:29:44.615377+0000","last_unstale":"2024-01-26T23:29:48.626845+0000","last_undegraded":"2024-01-26T23:29:48.626845+0000","last_fullsized":"2024-01-26T23:29:48.626845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:38.452904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696514+0000","last_change":"2024-01-26T23:29:48.691661+0000","last_active":"2024-01-26T23:29:48.696514+0000","last_peered":"2024-01-26T23:29:48.696514+0000","last_clean":"2024-01-26T23:29:48.696514+0000","last_became_active":"2024-01-26T23:29:48.691233+0000","last_became_peered":"2024-01-26T23:29:48.691233+0000","last_unstale":"2024-01-26T23:29:48.696514+0000","last_undegraded":"2024-01-26T23:29:48.696514+0000","last_fullsized":"2024-01-26T23:29:48.696514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:22:12.593080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626042+0000","last_change":"2024-01-26T23:29:43.614006+0000","last_active":"2024-01-26T23:29:48.626042+0000","last_peered":"2024-01-26T23:29:48.626042+0000","last_clean":"2024-01-26T23:29:48.626042+0000","last_became_active":"2024-01-26T23:29:43.613688+0000","last_became_peered":"2024-01-26T23:29:43.613688+0000","last_unstale":"2024-01-26T23:29:48.626042+0000","last_undegraded":"2024-01-26T23:29:48.626042+0000","last_fullsized":"2024-01-26T23:29:48.626042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:30.053494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.639703+0000","last_change":"2024-01-26T23:29:44.618360+0000","last_active":"2024-01-26T23:29:48.639703+0000","last_peered":"2024-01-26T23:29:48.639703+0000","last_clean":"2024-01-26T23:29:48.639703+0000","last_became_active":"2024-01-26T23:29:44.618061+0000","last_became_peered":"2024-01-26T23:29:44.618061+0000","last_unstale":"2024-01-26T23:29:48.639703+0000","last_undegraded":"2024-01-26T23:29:48.639703+0000","last_fullsized":"2024-01-26T23:29:48.639703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:26:26.776664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695399+0000","last_change":"2024-01-26T23:29:48.694453+0000","last_active":"2024-01-26T23:29:48.695399+0000","last_peered":"2024-01-26T23:29:48.695399+0000","last_clean":"2024-01-26T23:29:48.695399+0000","last_became_active":"2024-01-26T23:29:48.693791+0000","last_became_peered":"2024-01-26T23:29:48.693791+0000","last_unstale":"2024-01-26T23:29:48.695399+0000","last_undegraded":"2024-01-26T23:29:48.695399+0000","last_fullsized":"2024-01-26T23:29:48.695399+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:20.246494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.638726+0000","last_change":"2024-01-26T23:29:43.619037+0000","last_active":"2024-01-26T23:29:48.638726+0000","last_peered":"2024-01-26T23:29:48.638726+0000","last_clean":"2024-01-26T23:29:48.638726+0000","last_became_active":"2024-01-26T23:29:43.618822+0000","last_became_peered":"2024-01-26T23:29:43.618822+0000","last_unstale":"2024-01-26T23:29:48.638726+0000","last_undegraded":"2024-01-26T23:29:48.638726+0000","last_fullsized":"2024-01-26T23:29:48.638726+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:30.053494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633637+0000","last_change":"2024-01-26T23:29:44.615872+0000","last_active":"2024-01-26T23:29:48.633637+0000","last_peered":"2024-01-26T23:29:48.633637+0000","last_clean":"2024-01-26T23:29:48.633637+0000","last_became_active":"2024-01-26T23:29:44.615547+0000","last_became_peered":"2024-01-26T23:29:44.615547+0000","last_unstale":"2024-01-26T23:29:48.633637+0000","last_undegraded":"2024-01-26T23:29:48.633637+0000","last_fullsized":"2024-01-26T23:29:48.633637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:04:39.582710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652459+0000","last_change":"2024-01-26T23:29:48.646587+0000","last_active":"2024-01-26T23:29:48.652459+0000","last_peered":"2024-01-26T23:29:48.652459+0000","last_clean":"2024-01-26T23:29:48.652459+0000","last_became_active":"2024-01-26T23:29:48.646117+0000","last_became_peered":"2024-01-26T23:29:48.646117+0000","last_unstale":"2024-01-26T23:29:48.652459+0000","last_undegraded":"2024-01-26T23:29:48.652459+0000","last_fullsized":"2024-01-26T23:29:48.652459+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:53:55.279661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.689527+0000","last_change":"2024-01-26T23:29:43.623735+0000","last_active":"2024-01-26T23:29:48.689527+0000","last_peered":"2024-01-26T23:29:48.689527+0000","last_clean":"2024-01-26T23:29:48.689527+0000","last_became_active":"2024-01-26T23:29:43.623048+0000","last_became_peered":"2024-01-26T23:29:43.623048+0000","last_unstale":"2024-01-26T23:29:48.689527+0000","last_undegraded":"2024-01-26T23:29:48.689527+0000","last_fullsized":"2024-01-26T23:29:48.689527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:47:09.096394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632551+0000","last_change":"2024-01-26T23:29:44.602462+0000","last_active":"2024-01-26T23:29:48.632551+0000","last_peered":"2024-01-26T23:29:48.632551+0000","last_clean":"2024-01-26T23:29:48.632551+0000","last_became_active":"2024-01-26T23:29:44.602007+0000","last_became_peered":"2024-01-26T23:29:44.602007+0000","last_unstale":"2024-01-26T23:29:48.632551+0000","last_undegraded":"2024-01-26T23:29:48.632551+0000","last_fullsized":"2024-01-26T23:29:48.632551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:23:26.050869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695910+0000","last_change":"2024-01-26T23:29:48.691917+0000","last_active":"2024-01-26T23:29:48.695910+0000","last_peered":"2024-01-26T23:29:48.695910+0000","last_clean":"2024-01-26T23:29:48.695910+0000","last_became_active":"2024-01-26T23:29:48.691441+0000","last_became_peered":"2024-01-26T23:29:48.691441+0000","last_unstale":"2024-01-26T23:29:48.695910+0000","last_undegraded":"2024-01-26T23:29:48.695910+0000","last_fullsized":"2024-01-26T23:29:48.695910+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:52:44.817581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.625872+0000","last_change":"2024-01-26T23:29:43.619776+0000","last_active":"2024-01-26T23:29:48.625872+0000","last_peered":"2024-01-26T23:29:48.625872+0000","last_clean":"2024-01-26T23:29:48.625872+0000","last_became_active":"2024-01-26T23:29:43.619374+0000","last_became_peered":"2024-01-26T23:29:43.619374+0000","last_unstale":"2024-01-26T23:29:48.625872+0000","last_undegraded":"2024-01-26T23:29:48.625872+0000","last_fullsized":"2024-01-26T23:29:48.625872+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:32.417441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636819+0000","last_change":"2024-01-26T23:29:44.621339+0000","last_active":"2024-01-26T23:29:48.636819+0000","last_peered":"2024-01-26T23:29:48.636819+0000","last_clean":"2024-01-26T23:29:48.636819+0000","last_became_active":"2024-01-26T23:29:44.621012+0000","last_became_peered":"2024-01-26T23:29:44.621012+0000","last_unstale":"2024-01-26T23:29:48.636819+0000","last_undegraded":"2024-01-26T23:29:48.636819+0000","last_fullsized":"2024-01-26T23:29:48.636819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:20.138786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652798+0000","last_change":"2024-01-26T23:29:48.650063+0000","last_active":"2024-01-26T23:29:48.652798+0000","last_peered":"2024-01-26T23:29:48.652798+0000","last_clean":"2024-01-26T23:29:48.652798+0000","last_became_active":"2024-01-26T23:29:48.649732+0000","last_became_peered":"2024-01-26T23:29:48.649732+0000","last_unstale":"2024-01-26T23:29:48.652798+0000","last_undegraded":"2024-01-26T23:29:48.652798+0000","last_fullsized":"2024-01-26T23:29:48.652798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:22:12.593080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626014+0000","last_change":"2024-01-26T23:29:43.618706+0000","last_active":"2024-01-26T23:29:48.626014+0000","last_peered":"2024-01-26T23:29:48.626014+0000","last_clean":"2024-01-26T23:29:48.626014+0000","last_became_active":"2024-01-26T23:29:43.618353+0000","last_became_peered":"2024-01-26T23:29:43.618353+0000","last_unstale":"2024-01-26T23:29:48.626014+0000","last_undegraded":"2024-01-26T23:29:48.626014+0000","last_fullsized":"2024-01-26T23:29:48.626014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:45.384500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636855+0000","last_change":"2024-01-26T23:29:44.633055+0000","last_active":"2024-01-26T23:29:48.636855+0000","last_peered":"2024-01-26T23:29:48.636855+0000","last_clean":"2024-01-26T23:29:48.636855+0000","last_became_active":"2024-01-26T23:29:44.631874+0000","last_became_peered":"2024-01-26T23:29:44.631874+0000","last_unstale":"2024-01-26T23:29:48.636855+0000","last_undegraded":"2024-01-26T23:29:48.636855+0000","last_fullsized":"2024-01-26T23:29:48.636855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:01.636855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652530+0000","last_change":"2024-01-26T23:29:48.648195+0000","last_active":"2024-01-26T23:29:48.652530+0000","last_peered":"2024-01-26T23:29:48.652530+0000","last_clean":"2024-01-26T23:29:48.652530+0000","last_became_active":"2024-01-26T23:29:48.647780+0000","last_became_peered":"2024-01-26T23:29:48.647780+0000","last_unstale":"2024-01-26T23:29:48.652530+0000","last_undegraded":"2024-01-26T23:29:48.652530+0000","last_fullsized":"2024-01-26T23:29:48.652530+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:52:44.817581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690272+0000","last_change":"2024-01-26T23:29:43.623737+0000","last_active":"2024-01-26T23:29:48.690272+0000","last_peered":"2024-01-26T23:29:48.690272+0000","last_clean":"2024-01-26T23:29:48.690272+0000","last_became_active":"2024-01-26T23:29:43.623439+0000","last_became_peered":"2024-01-26T23:29:43.623439+0000","last_unstale":"2024-01-26T23:29:48.690272+0000","last_undegraded":"2024-01-26T23:29:48.690272+0000","last_fullsized":"2024-01-26T23:29:48.690272+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:45.384500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632872+0000","last_change":"2024-01-26T23:29:44.613570+0000","last_active":"2024-01-26T23:29:48.632872+0000","last_peered":"2024-01-26T23:29:48.632872+0000","last_clean":"2024-01-26T23:29:48.632872+0000","last_became_active":"2024-01-26T23:29:44.613282+0000","last_became_peered":"2024-01-26T23:29:44.613282+0000","last_unstale":"2024-01-26T23:29:48.632872+0000","last_undegraded":"2024-01-26T23:29:48.632872+0000","last_fullsized":"2024-01-26T23:29:48.632872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:12.500308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696078+0000","last_change":"2024-01-26T23:29:48.691625+0000","last_active":"2024-01-26T23:29:48.696078+0000","last_peered":"2024-01-26T23:29:48.696078+0000","last_clean":"2024-01-26T23:29:48.696078+0000","last_became_active":"2024-01-26T23:29:48.691242+0000","last_became_peered":"2024-01-26T23:29:48.691242+0000","last_unstale":"2024-01-26T23:29:48.696078+0000","last_undegraded":"2024-01-26T23:29:48.696078+0000","last_fullsized":"2024-01-26T23:29:48.696078+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:02.358147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633000+0000","last_change":"2024-01-26T23:29:43.622471+0000","last_active":"2024-01-26T23:29:48.633000+0000","last_peered":"2024-01-26T23:29:48.633000+0000","last_clean":"2024-01-26T23:29:48.633000+0000","last_became_active":"2024-01-26T23:29:43.622106+0000","last_became_peered":"2024-01-26T23:29:43.622106+0000","last_unstale":"2024-01-26T23:29:48.633000+0000","last_undegraded":"2024-01-26T23:29:48.633000+0000","last_fullsized":"2024-01-26T23:29:48.633000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:32.417441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690284+0000","last_change":"2024-01-26T23:29:44.623102+0000","last_active":"2024-01-26T23:29:48.690284+0000","last_peered":"2024-01-26T23:29:48.690284+0000","last_clean":"2024-01-26T23:29:48.690284+0000","last_became_active":"2024-01-26T23:29:44.622834+0000","last_became_peered":"2024-01-26T23:29:44.622834+0000","last_unstale":"2024-01-26T23:29:48.690284+0000","last_undegraded":"2024-01-26T23:29:48.690284+0000","last_fullsized":"2024-01-26T23:29:48.690284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:18:55.543376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.651007+0000","last_change":"2024-01-26T23:29:48.644283+0000","last_active":"2024-01-26T23:29:48.651007+0000","last_peered":"2024-01-26T23:29:48.651007+0000","last_clean":"2024-01-26T23:29:48.651007+0000","last_became_active":"2024-01-26T23:29:48.643862+0000","last_became_peered":"2024-01-26T23:29:48.643862+0000","last_unstale":"2024-01-26T23:29:48.651007+0000","last_undegraded":"2024-01-26T23:29:48.651007+0000","last_fullsized":"2024-01-26T23:29:48.651007+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:29:51.471531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626158+0000","last_change":"2024-01-26T23:29:43.613316+0000","last_active":"2024-01-26T23:29:48.626158+0000","last_peered":"2024-01-26T23:29:48.626158+0000","last_clean":"2024-01-26T23:29:48.626158+0000","last_became_active":"2024-01-26T23:29:43.613054+0000","last_became_peered":"2024-01-26T23:29:43.613054+0000","last_unstale":"2024-01-26T23:29:48.626158+0000","last_undegraded":"2024-01-26T23:29:48.626158+0000","last_fullsized":"2024-01-26T23:29:48.626158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:09:46.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636441+0000","last_change":"2024-01-26T23:29:44.631866+0000","last_active":"2024-01-26T23:29:48.636441+0000","last_peered":"2024-01-26T23:29:48.636441+0000","last_clean":"2024-01-26T23:29:48.636441+0000","last_became_active":"2024-01-26T23:29:44.631581+0000","last_became_peered":"2024-01-26T23:29:44.631581+0000","last_unstale":"2024-01-26T23:29:48.636441+0000","last_undegraded":"2024-01-26T23:29:48.636441+0000","last_fullsized":"2024-01-26T23:29:48.636441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:38:52.041592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695113+0000","last_change":"2024-01-26T23:29:48.691846+0000","last_active":"2024-01-26T23:29:48.695113+0000","last_peered":"2024-01-26T23:29:48.695113+0000","last_clean":"2024-01-26T23:29:48.695113+0000","last_became_active":"2024-01-26T23:29:48.691410+0000","last_became_peered":"2024-01-26T23:29:48.691410+0000","last_unstale":"2024-01-26T23:29:48.695113+0000","last_undegraded":"2024-01-26T23:29:48.695113+0000","last_fullsized":"2024-01-26T23:29:48.695113+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:49:28.002810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633357+0000","last_change":"2024-01-26T23:29:43.618244+0000","last_active":"2024-01-26T23:29:48.633357+0000","last_peered":"2024-01-26T23:29:48.633357+0000","last_clean":"2024-01-26T23:29:48.633357+0000","last_became_active":"2024-01-26T23:29:43.617875+0000","last_became_peered":"2024-01-26T23:29:43.617875+0000","last_unstale":"2024-01-26T23:29:48.633357+0000","last_undegraded":"2024-01-26T23:29:48.633357+0000","last_fullsized":"2024-01-26T23:29:48.633357+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:44:34.698447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690732+0000","last_change":"2024-01-26T23:29:44.618128+0000","last_active":"2024-01-26T23:29:48.690732+0000","last_peered":"2024-01-26T23:29:48.690732+0000","last_clean":"2024-01-26T23:29:48.690732+0000","last_became_active":"2024-01-26T23:29:44.617849+0000","last_became_peered":"2024-01-26T23:29:44.617849+0000","last_unstale":"2024-01-26T23:29:48.690732+0000","last_undegraded":"2024-01-26T23:29:48.690732+0000","last_fullsized":"2024-01-26T23:29:48.690732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:20.138786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633526+0000","last_change":"2024-01-26T23:29:44.615453+0000","last_active":"2024-01-26T23:29:48.633526+0000","last_peered":"2024-01-26T23:29:48.633526+0000","last_clean":"2024-01-26T23:29:48.633526+0000","last_became_active":"2024-01-26T23:29:44.615201+0000","last_became_peered":"2024-01-26T23:29:44.615201+0000","last_unstale":"2024-01-26T23:29:48.633526+0000","last_undegraded":"2024-01-26T23:29:48.633526+0000","last_fullsized":"2024-01-26T23:29:48.633526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:38.452904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631336+0000","last_change":"2024-01-26T23:29:44.619769+0000","last_active":"2024-01-26T23:29:48.631336+0000","last_peered":"2024-01-26T23:29:48.631336+0000","last_clean":"2024-01-26T23:29:48.631336+0000","last_became_active":"2024-01-26T23:29:44.619471+0000","last_became_peered":"2024-01-26T23:29:44.619471+0000","last_unstale":"2024-01-26T23:29:48.631336+0000","last_undegraded":"2024-01-26T23:29:48.631336+0000","last_fullsized":"2024-01-26T23:29:48.631336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:47:10.097930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635849+0000","last_change":"2024-01-26T23:29:44.633598+0000","last_active":"2024-01-26T23:29:48.635849+0000","last_peered":"2024-01-26T23:29:48.635849+0000","last_clean":"2024-01-26T23:29:48.635849+0000","last_became_active":"2024-01-26T23:29:44.633320+0000","last_became_peered":"2024-01-26T23:29:44.633320+0000","last_unstale":"2024-01-26T23:29:48.635849+0000","last_undegraded":"2024-01-26T23:29:48.635849+0000","last_fullsized":"2024-01-26T23:29:48.635849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:03.997854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632174+0000","last_change":"2024-01-26T23:29:44.624890+0000","last_active":"2024-01-26T23:29:48.632174+0000","last_peered":"2024-01-26T23:29:48.632174+0000","last_clean":"2024-01-26T23:29:48.632174+0000","last_became_active":"2024-01-26T23:29:44.624615+0000","last_became_peered":"2024-01-26T23:29:44.624615+0000","last_unstale":"2024-01-26T23:29:48.632174+0000","last_undegraded":"2024-01-26T23:29:48.632174+0000","last_fullsized":"2024-01-26T23:29:48.632174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:12:04.973733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637534+0000","last_change":"2024-01-26T23:29:44.618261+0000","last_active":"2024-01-26T23:29:48.637534+0000","last_peered":"2024-01-26T23:29:48.637534+0000","last_clean":"2024-01-26T23:29:48.637534+0000","last_became_active":"2024-01-26T23:29:44.616559+0000","last_became_peered":"2024-01-26T23:29:44.616559+0000","last_unstale":"2024-01-26T23:29:48.637534+0000","last_undegraded":"2024-01-26T23:29:48.637534+0000","last_fullsized":"2024-01-26T23:29:48.637534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:12:04.973733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626013+0000","last_change":"2024-01-26T23:29:44.619408+0000","last_active":"2024-01-26T23:29:48.626013+0000","last_peered":"2024-01-26T23:29:48.626013+0000","last_clean":"2024-01-26T23:29:48.626013+0000","last_became_active":"2024-01-26T23:29:44.619093+0000","last_became_peered":"2024-01-26T23:29:44.619093+0000","last_unstale":"2024-01-26T23:29:48.626013+0000","last_undegraded":"2024-01-26T23:29:48.626013+0000","last_fullsized":"2024-01-26T23:29:48.626013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:35:11.068882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.638008+0000","last_change":"2024-01-26T23:29:44.614999+0000","last_active":"2024-01-26T23:29:48.638008+0000","last_peered":"2024-01-26T23:29:48.638008+0000","last_clean":"2024-01-26T23:29:48.638008+0000","last_became_active":"2024-01-26T23:29:44.614579+0000","last_became_peered":"2024-01-26T23:29:44.614579+0000","last_unstale":"2024-01-26T23:29:48.638008+0000","last_undegraded":"2024-01-26T23:29:48.638008+0000","last_fullsized":"2024-01-26T23:29:48.638008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.639791+0000","last_change":"2024-01-26T23:29:44.612377+0000","last_active":"2024-01-26T23:29:48.639791+0000","last_peered":"2024-01-26T23:29:48.639791+0000","last_clean":"2024-01-26T23:29:48.639791+0000","last_became_active":"2024-01-26T23:29:44.612073+0000","last_became_peered":"2024-01-26T23:29:44.612073+0000","last_unstale":"2024-01-26T23:29:48.639791+0000","last_undegraded":"2024-01-26T23:29:48.639791+0000","last_fullsized":"2024-01-26T23:29:48.639791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:59:47.867303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632846+0000","last_change":"2024-01-26T23:29:44.633571+0000","last_active":"2024-01-26T23:29:48.632846+0000","last_peered":"2024-01-26T23:29:48.632846+0000","last_clean":"2024-01-26T23:29:48.632846+0000","last_became_active":"2024-01-26T23:29:44.633312+0000","last_became_peered":"2024-01-26T23:29:44.633312+0000","last_unstale":"2024-01-26T23:29:48.632846+0000","last_undegraded":"2024-01-26T23:29:48.632846+0000","last_fullsized":"2024-01-26T23:29:48.632846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:22.113035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637064+0000","last_change":"2024-01-26T23:29:44.617840+0000","last_active":"2024-01-26T23:29:48.637064+0000","last_peered":"2024-01-26T23:29:48.637064+0000","last_clean":"2024-01-26T23:29:48.637064+0000","last_became_active":"2024-01-26T23:29:44.611989+0000","last_became_peered":"2024-01-26T23:29:44.611989+0000","last_unstale":"2024-01-26T23:29:48.637064+0000","last_undegraded":"2024-01-26T23:29:48.637064+0000","last_fullsized":"2024-01-26T23:29:48.637064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:14.555745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635947+0000","last_change":"2024-01-26T23:29:44.617326+0000","last_active":"2024-01-26T23:29:48.635947+0000","last_peered":"2024-01-26T23:29:48.635947+0000","last_clean":"2024-01-26T23:29:48.635947+0000","last_became_active":"2024-01-26T23:29:44.617063+0000","last_became_peered":"2024-01-26T23:29:44.617063+0000","last_unstale":"2024-01-26T23:29:48.635947+0000","last_undegraded":"2024-01-26T23:29:48.635947+0000","last_fullsized":"2024-01-26T23:29:48.635947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:01.636855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632268+0000","last_change":"2024-01-26T23:29:44.613645+0000","last_active":"2024-01-26T23:29:48.632268+0000","last_peered":"2024-01-26T23:29:48.632268+0000","last_clean":"2024-01-26T23:29:48.632268+0000","last_became_active":"2024-01-26T23:29:44.613290+0000","last_became_peered":"2024-01-26T23:29:44.613290+0000","last_unstale":"2024-01-26T23:29:48.632268+0000","last_undegraded":"2024-01-26T23:29:48.632268+0000","last_fullsized":"2024-01-26T23:29:48.632268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:12.500308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637188+0000","last_change":"2024-01-26T23:29:44.609934+0000","last_active":"2024-01-26T23:29:48.637188+0000","last_peered":"2024-01-26T23:29:48.637188+0000","last_clean":"2024-01-26T23:29:48.637188+0000","last_became_active":"2024-01-26T23:29:44.609682+0000","last_became_peered":"2024-01-26T23:29:44.609682+0000","last_unstale":"2024-01-26T23:29:48.637188+0000","last_undegraded":"2024-01-26T23:29:48.637188+0000","last_fullsized":"2024-01-26T23:29:48.637188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:12:04.973733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4483,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298120,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073720,"statfs":{"total":96636764160,"available":96331489280,"internally_reserved":0,"allocated":851968,"data_stored":690410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":103201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297544,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074296,"statfs":{"total":96636764160,"available":96332079104,"internally_reserved":0,"allocated":262144,"data_stored":99648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298128,"kb_used_data":840,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073712,"statfs":{"total":96636764160,"available":96331481088,"internally_reserved":0,"allocated":860160,"data_stored":690466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297536,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074304,"statfs":{"total":96636764160,"available":96332087296,"internally_reserved":0,"allocated":253952,"data_stored":99488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.221}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":103201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":99614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297540,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074300,"statfs":{"total":96636764160,"available":96332083200,"internally_reserved":0,"allocated":258048,"data_stored":99578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:30:04.850 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:30:05.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.018+0000 7ff3cd00e700 1 Processor -- start 2024-01-26T23:30:05.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.018+0000 7ff3cd00e700 1 -- start start 2024-01-26T23:30:05.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.019+0000 7ff3cd00e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff3c8159540 0x7ff3c8159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.019+0000 7ff3cd00e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3c80b2b20 0x7ff3c80b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.019+0000 7ff3cd00e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c806d4b0 con 0x7ff3c8067930 2024-01-26T23:30:05.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.019+0000 7ff3cd00e700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c815d4a0 con 0x7ff3c80b2b20 2024-01-26T23:30:05.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.019+0000 7ff3cd00e700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c815d780 con 0x7ff3c8159540 2024-01-26T23:30:05.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.019+0000 7ff3c659c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3c80b2b20 0x7ff3c80b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.019+0000 7ff3c659c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3c80b2b20 0x7ff3c80b2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33746/0 (socket says 172.21.15.46:33746) 2024-01-26T23:30:05.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.019+0000 7ff3c659c700 1 -- 172.21.15.46:0/3101145284 learned_addr learned my addr 172.21.15.46:0/3101145284 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:05.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.019+0000 7ff3c6d9d700 1 --2- 172.21.15.46:0/3101145284 >> v2:172.21.15.106:3300/0 conn(0x7ff3c8159540 0x7ff3c8159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.020+0000 7ff3c659c700 1 -- 172.21.15.46:0/3101145284 >> v2:172.21.15.106:3300/0 conn(0x7ff3c8159540 msgr2=0x7ff3c8159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:05.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.020+0000 7ff3c659c700 1 --2- 172.21.15.46:0/3101145284 >> v2:172.21.15.106:3300/0 conn(0x7ff3c8159540 0x7ff3c8159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.020+0000 7ff3c659c700 1 -- 172.21.15.46:0/3101145284 >> v1:172.21.15.106:6789/0 conn(0x7ff3c8067930 legacy=0x7ff3c806eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:05.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.020+0000 7ff3c659c700 1 -- 172.21.15.46:0/3101145284 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c815da60 con 0x7ff3c80b2b20 2024-01-26T23:30:05.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.020+0000 7ff3c659c700 1 --2- 172.21.15.46:0/3101145284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c80b2b20 0x7ff3c80b2fa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff3c806e750 tx=0x7ff3b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d8e94be8191e2e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:05.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.020+0000 7ff3c5d9b700 1 -- 172.21.15.46:0/3101145284 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8009e70 con 0x7ff3c80b2b20 2024-01-26T23:30:05.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.020+0000 7ff3c5d9b700 1 -- 172.21.15.46:0/3101145284 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b800b860 con 0x7ff3c80b2b20 2024-01-26T23:30:05.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.020+0000 7ff3c5d9b700 1 -- 172.21.15.46:0/3101145284 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b800bc00 con 0x7ff3c80b2b20 2024-01-26T23:30:05.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.021+0000 7ff3cd00e700 1 -- 172.21.15.46:0/3101145284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c80b2b20 msgr2=0x7ff3c80b2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:05.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.021+0000 7ff3cd00e700 1 --2- 172.21.15.46:0/3101145284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c80b2b20 0x7ff3c80b2fa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff3c806e750 tx=0x7ff3b8009250 comp rx=0 tx=0).stop 2024-01-26T23:30:05.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.021+0000 7ff3cd00e700 1 -- 172.21.15.46:0/3101145284 shutdown_connections 2024-01-26T23:30:05.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.021+0000 7ff3cd00e700 1 --2- 172.21.15.46:0/3101145284 >> v2:172.21.15.106:3300/0 conn(0x7ff3c8159540 0x7ff3c8159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.021+0000 7ff3cd00e700 1 --2- 172.21.15.46:0/3101145284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c80b2b20 0x7ff3c80b2fa0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.021+0000 7ff3cd00e700 1 -- 172.21.15.46:0/3101145284 >> 172.21.15.46:0/3101145284 conn(0x7ff3c806b630 msgr2=0x7ff3c80aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:05.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.021+0000 7ff3cd00e700 1 -- 172.21.15.46:0/3101145284 shutdown_connections 2024-01-26T23:30:05.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.021+0000 7ff3cd00e700 1 -- 172.21.15.46:0/3101145284 wait complete. 2024-01-26T23:30:05.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.022+0000 7ff3cd00e700 1 Processor -- start 2024-01-26T23:30:05.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.022+0000 7ff3cd00e700 1 -- start start 2024-01-26T23:30:05.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.023+0000 7ff3cd00e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c8067930 0x7ff3c816d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.023+0000 7ff3cd00e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3c80b2b20 0x7ff3c8140370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.023+0000 7ff3cd00e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3c8159540 0x7ff3c81408b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.023+0000 7ff3cd00e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c813b6e0 con 0x7ff3c8067930 2024-01-26T23:30:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.023+0000 7ff3cd00e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c813b9c0 con 0x7ff3c80b2b20 2024-01-26T23:30:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.023+0000 7ff3cd00e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c813bca0 con 0x7ff3c8159540 2024-01-26T23:30:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.023+0000 7ff3c6d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c8067930 0x7ff3c816d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.023+0000 7ff3c6d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c8067930 0x7ff3c816d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33756/0 (socket says 172.21.15.46:33756) 2024-01-26T23:30:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.023+0000 7ff3c6d9d700 1 -- 172.21.15.46:0/3104370678 learned_addr learned my addr 172.21.15.46:0/3104370678 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.023+0000 7ff3c659c700 1 --2- 172.21.15.46:0/3104370678 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3c80b2b20 0x7ff3c8140370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.023+0000 7ff3c759e700 1 --2- 172.21.15.46:0/3104370678 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3c8159540 0x7ff3c81408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.023+0000 7ff3c6d9d700 1 -- 172.21.15.46:0/3104370678 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3c8159540 msgr2=0x7ff3c81408b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:05.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.023+0000 7ff3c6d9d700 1 --2- 172.21.15.46:0/3104370678 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3c8159540 0x7ff3c81408b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.023+0000 7ff3c6d9d700 1 -- 172.21.15.46:0/3104370678 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3c80b2b20 msgr2=0x7ff3c8140370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:05.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.023+0000 7ff3c6d9d700 1 --2- 172.21.15.46:0/3104370678 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3c80b2b20 0x7ff3c8140370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.023+0000 7ff3c6d9d700 1 -- 172.21.15.46:0/3104370678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b80094f0 con 0x7ff3c8067930 2024-01-26T23:30:05.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.024+0000 7ff3c6d9d700 1 --2- 172.21.15.46:0/3104370678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c8067930 0x7ff3c816d490 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff3b000e930 tx=0x7ff3b00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:05.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.024+0000 7ff3beffd700 1 -- 172.21.15.46:0/3104370678 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0013680 con 0x7ff3c8067930 2024-01-26T23:30:05.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.024+0000 7ff3beffd700 1 -- 172.21.15.46:0/3104370678 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b00137e0 con 0x7ff3c8067930 2024-01-26T23:30:05.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.024+0000 7ff3cd00e700 1 -- 172.21.15.46:0/3104370678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c813bf80 con 0x7ff3c8067930 2024-01-26T23:30:05.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.024+0000 7ff3beffd700 1 -- 172.21.15.46:0/3104370678 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0013ba0 con 0x7ff3c8067930 2024-01-26T23:30:05.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.024+0000 7ff3cd00e700 1 -- 172.21.15.46:0/3104370678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c813dd60 con 0x7ff3c8067930 2024-01-26T23:30:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.025+0000 7ff3cd00e700 1 -- 172.21.15.46:0/3104370678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c804c710 con 0x7ff3c8067930 2024-01-26T23:30:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.025+0000 7ff3beffd700 1 -- 172.21.15.46:0/3104370678 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff3b000c3e0 con 0x7ff3c8067930 2024-01-26T23:30:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.026+0000 7ff3beffd700 1 --2- 172.21.15.46:0/3104370678 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff3a8066f30 0x7ff3a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.026+0000 7ff3beffd700 1 -- 172.21.15.46:0/3104370678 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7ff3b0084820 con 0x7ff3c8067930 2024-01-26T23:30:05.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.026+0000 7ff3c659c700 1 --2- 172.21.15.46:0/3104370678 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff3a8066f30 0x7ff3a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.027+0000 7ff3c659c700 1 --2- 172.21.15.46:0/3104370678 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff3a8066f30 0x7ff3a80693e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff3c806e750 tx=0x7ff3b8017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:05.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.030+0000 7ff3beffd700 1 -- 172.21.15.46:0/3104370678 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff3b00517a0 con 0x7ff3c8067930 2024-01-26T23:30:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.231+0000 7ff3cd00e700 1 -- 172.21.15.46:0/3104370678 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff3c813bb30 con 0x7ff3a8066f30 2024-01-26T23:30:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.241+0000 7ff3beffd700 1 -- 172.21.15.46:0/3104370678 <== mgr.4102 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345239 (secure 0 0 0) 0x7ff3c813bb30 con 0x7ff3a8066f30 2024-01-26T23:30:05.246 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.245+0000 7ff3cd00e700 1 -- 172.21.15.46:0/3104370678 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff3a8066f30 msgr2=0x7ff3a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.245+0000 7ff3cd00e700 1 --2- 172.21.15.46:0/3104370678 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff3a8066f30 0x7ff3a80693e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff3c806e750 tx=0x7ff3b8017000 comp rx=0 tx=0).stop 2024-01-26T23:30:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.245+0000 7ff3cd00e700 1 -- 172.21.15.46:0/3104370678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c8067930 msgr2=0x7ff3c816d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.245+0000 7ff3cd00e700 1 --2- 172.21.15.46:0/3104370678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c8067930 0x7ff3c816d490 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff3b000e930 tx=0x7ff3b00091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.245+0000 7ff3cd00e700 1 -- 172.21.15.46:0/3104370678 shutdown_connections 2024-01-26T23:30:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.245+0000 7ff3cd00e700 1 --2- 172.21.15.46:0/3104370678 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff3a8066f30 0x7ff3a80693e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.245+0000 7ff3cd00e700 1 --2- 172.21.15.46:0/3104370678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c8067930 0x7ff3c816d490 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.245+0000 7ff3cd00e700 1 --2- 172.21.15.46:0/3104370678 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3c8159540 0x7ff3c81408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.245+0000 7ff3cd00e700 1 --2- 172.21.15.46:0/3104370678 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3c80b2b20 0x7ff3c8140370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.245+0000 7ff3cd00e700 1 -- 172.21.15.46:0/3104370678 >> 172.21.15.46:0/3104370678 conn(0x7ff3c806b630 msgr2=0x7ff3c81588a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.245+0000 7ff3cd00e700 1 -- 172.21.15.46:0/3104370678 shutdown_connections 2024-01-26T23:30:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.245+0000 7ff3cd00e700 1 -- 172.21.15.46:0/3104370678 wait complete. 2024-01-26T23:30:05.250 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:30:05.260 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-01-26T23:30:04.229026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594851,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381512,"kb_used_data":3208,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593208,"statfs":{"total":773094113280,"available":770655444992,"internally_reserved":0,"allocated":3284992,"data_stored":1985606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2177,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003050"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690995+0000","last_change":"2024-01-26T23:29:44.615781+0000","last_active":"2024-01-26T23:29:48.690995+0000","last_peered":"2024-01-26T23:29:48.690995+0000","last_clean":"2024-01-26T23:29:48.690995+0000","last_became_active":"2024-01-26T23:29:44.615522+0000","last_became_peered":"2024-01-26T23:29:44.615522+0000","last_unstale":"2024-01-26T23:29:48.690995+0000","last_undegraded":"2024-01-26T23:29:48.690995+0000","last_fullsized":"2024-01-26T23:29:48.690995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:03.997854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.627221+0000","last_change":"2024-01-26T23:29:44.628874+0000","last_active":"2024-01-26T23:29:48.627221+0000","last_peered":"2024-01-26T23:29:48.627221+0000","last_clean":"2024-01-26T23:29:48.627221+0000","last_became_active":"2024-01-26T23:29:44.628581+0000","last_became_peered":"2024-01-26T23:29:44.628581+0000","last_unstale":"2024-01-26T23:29:48.627221+0000","last_undegraded":"2024-01-26T23:29:48.627221+0000","last_fullsized":"2024-01-26T23:29:48.627221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:14.555745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631189+0000","last_change":"2024-01-26T23:29:44.601978+0000","last_active":"2024-01-26T23:29:48.631189+0000","last_peered":"2024-01-26T23:29:48.631189+0000","last_clean":"2024-01-26T23:29:48.631189+0000","last_became_active":"2024-01-26T23:29:44.601539+0000","last_became_peered":"2024-01-26T23:29:44.601539+0000","last_unstale":"2024-01-26T23:29:48.631189+0000","last_undegraded":"2024-01-26T23:29:48.631189+0000","last_fullsized":"2024-01-26T23:29:48.631189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:35:11.068882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636104+0000","last_change":"2024-01-26T23:29:44.632221+0000","last_active":"2024-01-26T23:29:48.636104+0000","last_peered":"2024-01-26T23:29:48.636104+0000","last_clean":"2024-01-26T23:29:48.636104+0000","last_became_active":"2024-01-26T23:29:44.631611+0000","last_became_peered":"2024-01-26T23:29:44.631611+0000","last_unstale":"2024-01-26T23:29:48.636104+0000","last_undegraded":"2024-01-26T23:29:48.636104+0000","last_fullsized":"2024-01-26T23:29:48.636104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:55:14.606640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690150+0000","last_change":"2024-01-26T23:29:44.616723+0000","last_active":"2024-01-26T23:29:48.690150+0000","last_peered":"2024-01-26T23:29:48.690150+0000","last_clean":"2024-01-26T23:29:48.690150+0000","last_became_active":"2024-01-26T23:29:44.615737+0000","last_became_peered":"2024-01-26T23:29:44.615737+0000","last_unstale":"2024-01-26T23:29:48.690150+0000","last_undegraded":"2024-01-26T23:29:48.690150+0000","last_fullsized":"2024-01-26T23:29:48.690150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635997+0000","last_change":"2024-01-26T23:29:44.622983+0000","last_active":"2024-01-26T23:29:48.635997+0000","last_peered":"2024-01-26T23:29:48.635997+0000","last_clean":"2024-01-26T23:29:48.635997+0000","last_became_active":"2024-01-26T23:29:44.622721+0000","last_became_peered":"2024-01-26T23:29:44.622721+0000","last_unstale":"2024-01-26T23:29:48.635997+0000","last_undegraded":"2024-01-26T23:29:48.635997+0000","last_fullsized":"2024-01-26T23:29:48.635997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690538+0000","last_change":"2024-01-26T23:29:44.620570+0000","last_active":"2024-01-26T23:29:48.690538+0000","last_peered":"2024-01-26T23:29:48.690538+0000","last_clean":"2024-01-26T23:29:48.690538+0000","last_became_active":"2024-01-26T23:29:44.620303+0000","last_became_peered":"2024-01-26T23:29:44.620303+0000","last_unstale":"2024-01-26T23:29:48.690538+0000","last_undegraded":"2024-01-26T23:29:48.690538+0000","last_fullsized":"2024-01-26T23:29:48.690538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:55:14.606640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636953+0000","last_change":"2024-01-26T23:29:44.630873+0000","last_active":"2024-01-26T23:29:48.636953+0000","last_peered":"2024-01-26T23:29:48.636953+0000","last_clean":"2024-01-26T23:29:48.636953+0000","last_became_active":"2024-01-26T23:29:44.630579+0000","last_became_peered":"2024-01-26T23:29:44.630579+0000","last_unstale":"2024-01-26T23:29:48.636953+0000","last_undegraded":"2024-01-26T23:29:48.636953+0000","last_fullsized":"2024-01-26T23:29:48.636953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:48:24.085774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690821+0000","last_change":"2024-01-26T23:29:44.626396+0000","last_active":"2024-01-26T23:29:48.690821+0000","last_peered":"2024-01-26T23:29:48.690821+0000","last_clean":"2024-01-26T23:29:48.690821+0000","last_became_active":"2024-01-26T23:29:44.622150+0000","last_became_peered":"2024-01-26T23:29:44.622150+0000","last_unstale":"2024-01-26T23:29:48.690821+0000","last_undegraded":"2024-01-26T23:29:48.690821+0000","last_fullsized":"2024-01-26T23:29:48.690821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:57:15.558584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637882+0000","last_change":"2024-01-26T23:29:44.618861+0000","last_active":"2024-01-26T23:29:48.637882+0000","last_peered":"2024-01-26T23:29:48.637882+0000","last_clean":"2024-01-26T23:29:48.637882+0000","last_became_active":"2024-01-26T23:29:44.618575+0000","last_became_peered":"2024-01-26T23:29:44.618575+0000","last_unstale":"2024-01-26T23:29:48.637882+0000","last_undegraded":"2024-01-26T23:29:48.637882+0000","last_fullsized":"2024-01-26T23:29:48.637882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:32.912164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637940+0000","last_change":"2024-01-26T23:29:44.617693+0000","last_active":"2024-01-26T23:29:48.637940+0000","last_peered":"2024-01-26T23:29:48.637940+0000","last_clean":"2024-01-26T23:29:48.637940+0000","last_became_active":"2024-01-26T23:29:44.617393+0000","last_became_peered":"2024-01-26T23:29:44.617393+0000","last_unstale":"2024-01-26T23:29:48.637940+0000","last_undegraded":"2024-01-26T23:29:48.637940+0000","last_fullsized":"2024-01-26T23:29:48.637940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633424+0000","last_change":"2024-01-26T23:29:44.614095+0000","last_active":"2024-01-26T23:29:48.633424+0000","last_peered":"2024-01-26T23:29:48.633424+0000","last_clean":"2024-01-26T23:29:48.633424+0000","last_became_active":"2024-01-26T23:29:44.613758+0000","last_became_peered":"2024-01-26T23:29:44.613758+0000","last_unstale":"2024-01-26T23:29:48.633424+0000","last_undegraded":"2024-01-26T23:29:48.633424+0000","last_fullsized":"2024-01-26T23:29:48.633424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:22.392135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636817+0000","last_change":"2024-01-26T23:29:44.624158+0000","last_active":"2024-01-26T23:29:48.636817+0000","last_peered":"2024-01-26T23:29:48.636817+0000","last_clean":"2024-01-26T23:29:48.636817+0000","last_became_active":"2024-01-26T23:29:44.623876+0000","last_became_peered":"2024-01-26T23:29:44.623876+0000","last_unstale":"2024-01-26T23:29:48.636817+0000","last_undegraded":"2024-01-26T23:29:48.636817+0000","last_fullsized":"2024-01-26T23:29:48.636817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631396+0000","last_change":"2024-01-26T23:29:44.618895+0000","last_active":"2024-01-26T23:29:48.631396+0000","last_peered":"2024-01-26T23:29:48.631396+0000","last_clean":"2024-01-26T23:29:48.631396+0000","last_became_active":"2024-01-26T23:29:44.618505+0000","last_became_peered":"2024-01-26T23:29:44.618505+0000","last_unstale":"2024-01-26T23:29:48.631396+0000","last_undegraded":"2024-01-26T23:29:48.631396+0000","last_fullsized":"2024-01-26T23:29:48.631396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:59:47.867303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.625861+0000","last_change":"2024-01-26T23:29:44.612783+0000","last_active":"2024-01-26T23:29:48.625861+0000","last_peered":"2024-01-26T23:29:48.625861+0000","last_clean":"2024-01-26T23:29:48.625861+0000","last_became_active":"2024-01-26T23:29:44.612451+0000","last_became_peered":"2024-01-26T23:29:44.612451+0000","last_unstale":"2024-01-26T23:29:48.625861+0000","last_undegraded":"2024-01-26T23:29:48.625861+0000","last_fullsized":"2024-01-26T23:29:48.625861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:12.500308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632657+0000","last_change":"2024-01-26T23:29:44.630662+0000","last_active":"2024-01-26T23:29:48.632657+0000","last_peered":"2024-01-26T23:29:48.632657+0000","last_clean":"2024-01-26T23:29:48.632657+0000","last_became_active":"2024-01-26T23:29:44.630412+0000","last_became_peered":"2024-01-26T23:29:44.630412+0000","last_unstale":"2024-01-26T23:29:48.632657+0000","last_undegraded":"2024-01-26T23:29:48.632657+0000","last_fullsized":"2024-01-26T23:29:48.632657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636581+0000","last_change":"2024-01-26T23:29:44.624186+0000","last_active":"2024-01-26T23:29:48.636581+0000","last_peered":"2024-01-26T23:29:48.636581+0000","last_clean":"2024-01-26T23:29:48.636581+0000","last_became_active":"2024-01-26T23:29:44.623729+0000","last_became_peered":"2024-01-26T23:29:44.623729+0000","last_unstale":"2024-01-26T23:29:48.636581+0000","last_undegraded":"2024-01-26T23:29:48.636581+0000","last_fullsized":"2024-01-26T23:29:48.636581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:20.138786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633462+0000","last_change":"2024-01-26T23:29:44.615801+0000","last_active":"2024-01-26T23:29:48.633462+0000","last_peered":"2024-01-26T23:29:48.633462+0000","last_clean":"2024-01-26T23:29:48.633462+0000","last_became_active":"2024-01-26T23:29:44.614126+0000","last_became_peered":"2024-01-26T23:29:44.614126+0000","last_unstale":"2024-01-26T23:29:48.633462+0000","last_undegraded":"2024-01-26T23:29:48.633462+0000","last_fullsized":"2024-01-26T23:29:48.633462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:22.392135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690472+0000","last_change":"2024-01-26T23:29:44.620718+0000","last_active":"2024-01-26T23:29:48.690472+0000","last_peered":"2024-01-26T23:29:48.690472+0000","last_clean":"2024-01-26T23:29:48.690472+0000","last_became_active":"2024-01-26T23:29:44.620452+0000","last_became_peered":"2024-01-26T23:29:44.620452+0000","last_unstale":"2024-01-26T23:29:48.690472+0000","last_undegraded":"2024-01-26T23:29:48.690472+0000","last_fullsized":"2024-01-26T23:29:48.690472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:01.636855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.697979+0000","last_change":"2024-01-26T23:29:48.697204+0000","last_active":"2024-01-26T23:29:48.697979+0000","last_peered":"2024-01-26T23:29:48.697979+0000","last_clean":"2024-01-26T23:29:48.697979+0000","last_became_active":"2024-01-26T23:29:48.696727+0000","last_became_peered":"2024-01-26T23:29:48.696727+0000","last_unstale":"2024-01-26T23:29:48.697979+0000","last_undegraded":"2024-01-26T23:29:48.697979+0000","last_fullsized":"2024-01-26T23:29:48.697979+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:59.257622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633767+0000","last_change":"2024-01-26T23:29:43.611980+0000","last_active":"2024-01-26T23:29:48.633767+0000","last_peered":"2024-01-26T23:29:48.633767+0000","last_clean":"2024-01-26T23:29:48.633767+0000","last_became_active":"2024-01-26T23:29:43.611649+0000","last_became_peered":"2024-01-26T23:29:43.611649+0000","last_unstale":"2024-01-26T23:29:48.633767+0000","last_undegraded":"2024-01-26T23:29:48.633767+0000","last_fullsized":"2024-01-26T23:29:48.633767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:31.317009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637215+0000","last_change":"2024-01-26T23:29:44.617315+0000","last_active":"2024-01-26T23:29:48.637215+0000","last_peered":"2024-01-26T23:29:48.637215+0000","last_clean":"2024-01-26T23:29:48.637215+0000","last_became_active":"2024-01-26T23:29:44.617020+0000","last_became_peered":"2024-01-26T23:29:44.617020+0000","last_unstale":"2024-01-26T23:29:48.637215+0000","last_undegraded":"2024-01-26T23:29:48.637215+0000","last_fullsized":"2024-01-26T23:29:48.637215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:21:47.569396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652507+0000","last_change":"2024-01-26T23:29:48.647630+0000","last_active":"2024-01-26T23:29:48.652507+0000","last_peered":"2024-01-26T23:29:48.652507+0000","last_clean":"2024-01-26T23:29:48.652507+0000","last_became_active":"2024-01-26T23:29:48.646094+0000","last_became_peered":"2024-01-26T23:29:48.646094+0000","last_unstale":"2024-01-26T23:29:48.652507+0000","last_undegraded":"2024-01-26T23:29:48.652507+0000","last_fullsized":"2024-01-26T23:29:48.652507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:20.380384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637332+0000","last_change":"2024-01-26T23:29:43.616917+0000","last_active":"2024-01-26T23:29:48.637332+0000","last_peered":"2024-01-26T23:29:48.637332+0000","last_clean":"2024-01-26T23:29:48.637332+0000","last_became_active":"2024-01-26T23:29:43.616606+0000","last_became_peered":"2024-01-26T23:29:43.616606+0000","last_unstale":"2024-01-26T23:29:48.637332+0000","last_undegraded":"2024-01-26T23:29:48.637332+0000","last_fullsized":"2024-01-26T23:29:48.637332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:54:53.198685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633652+0000","last_change":"2024-01-26T23:29:44.626223+0000","last_active":"2024-01-26T23:29:48.633652+0000","last_peered":"2024-01-26T23:29:48.633652+0000","last_clean":"2024-01-26T23:29:48.633652+0000","last_became_active":"2024-01-26T23:29:44.625830+0000","last_became_peered":"2024-01-26T23:29:44.625830+0000","last_unstale":"2024-01-26T23:29:48.633652+0000","last_undegraded":"2024-01-26T23:29:48.633652+0000","last_fullsized":"2024-01-26T23:29:48.633652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652505+0000","last_change":"2024-01-26T23:29:48.647450+0000","last_active":"2024-01-26T23:29:48.652505+0000","last_peered":"2024-01-26T23:29:48.652505+0000","last_clean":"2024-01-26T23:29:48.652505+0000","last_became_active":"2024-01-26T23:29:48.646159+0000","last_became_peered":"2024-01-26T23:29:48.646159+0000","last_unstale":"2024-01-26T23:29:48.652505+0000","last_undegraded":"2024-01-26T23:29:48.652505+0000","last_fullsized":"2024-01-26T23:29:48.652505+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:13:54.119091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690197+0000","last_change":"2024-01-26T23:29:43.624480+0000","last_active":"2024-01-26T23:29:48.690197+0000","last_peered":"2024-01-26T23:29:48.690197+0000","last_clean":"2024-01-26T23:29:48.690197+0000","last_became_active":"2024-01-26T23:29:43.624225+0000","last_became_peered":"2024-01-26T23:29:43.624225+0000","last_unstale":"2024-01-26T23:29:48.690197+0000","last_undegraded":"2024-01-26T23:29:48.690197+0000","last_fullsized":"2024-01-26T23:29:48.690197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:21.390599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636199+0000","last_change":"2024-01-26T23:29:44.621996+0000","last_active":"2024-01-26T23:29:48.636199+0000","last_peered":"2024-01-26T23:29:48.636199+0000","last_clean":"2024-01-26T23:29:48.636199+0000","last_became_active":"2024-01-26T23:29:44.621750+0000","last_became_peered":"2024-01-26T23:29:44.621750+0000","last_unstale":"2024-01-26T23:29:48.636199+0000","last_undegraded":"2024-01-26T23:29:48.636199+0000","last_fullsized":"2024-01-26T23:29:48.636199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:32.912164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652475+0000","last_change":"2024-01-26T23:29:48.647439+0000","last_active":"2024-01-26T23:29:48.652475+0000","last_peered":"2024-01-26T23:29:48.652475+0000","last_clean":"2024-01-26T23:29:48.652475+0000","last_became_active":"2024-01-26T23:29:48.647004+0000","last_became_peered":"2024-01-26T23:29:48.647004+0000","last_unstale":"2024-01-26T23:29:48.652475+0000","last_undegraded":"2024-01-26T23:29:48.652475+0000","last_fullsized":"2024-01-26T23:29:48.652475+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:29:51.471531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690167+0000","last_change":"2024-01-26T23:29:43.623440+0000","last_active":"2024-01-26T23:29:48.690167+0000","last_peered":"2024-01-26T23:29:48.690167+0000","last_clean":"2024-01-26T23:29:48.690167+0000","last_became_active":"2024-01-26T23:29:43.623105+0000","last_became_peered":"2024-01-26T23:29:43.623105+0000","last_unstale":"2024-01-26T23:29:48.690167+0000","last_undegraded":"2024-01-26T23:29:48.690167+0000","last_fullsized":"2024-01-26T23:29:48.690167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:37:22.647734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636762+0000","last_change":"2024-01-26T23:29:44.622275+0000","last_active":"2024-01-26T23:29:48.636762+0000","last_peered":"2024-01-26T23:29:48.636762+0000","last_clean":"2024-01-26T23:29:48.636762+0000","last_became_active":"2024-01-26T23:29:44.621687+0000","last_became_peered":"2024-01-26T23:29:44.621687+0000","last_unstale":"2024-01-26T23:29:48.636762+0000","last_undegraded":"2024-01-26T23:29:48.636762+0000","last_fullsized":"2024-01-26T23:29:48.636762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:12.500308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.653689+0000","last_change":"2024-01-26T23:29:48.652656+0000","last_active":"2024-01-26T23:29:48.653689+0000","last_peered":"2024-01-26T23:29:48.653689+0000","last_clean":"2024-01-26T23:29:48.653689+0000","last_became_active":"2024-01-26T23:29:48.652264+0000","last_became_peered":"2024-01-26T23:29:48.652264+0000","last_unstale":"2024-01-26T23:29:48.653689+0000","last_undegraded":"2024-01-26T23:29:48.653689+0000","last_fullsized":"2024-01-26T23:29:48.653689+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:35:23.165011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636924+0000","last_change":"2024-01-26T23:29:43.620505+0000","last_active":"2024-01-26T23:29:48.636924+0000","last_peered":"2024-01-26T23:29:48.636924+0000","last_clean":"2024-01-26T23:29:48.636924+0000","last_became_active":"2024-01-26T23:29:43.620202+0000","last_became_peered":"2024-01-26T23:29:43.620202+0000","last_unstale":"2024-01-26T23:29:48.636924+0000","last_undegraded":"2024-01-26T23:29:48.636924+0000","last_fullsized":"2024-01-26T23:29:48.636924+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:31.317009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633949+0000","last_change":"2024-01-26T23:29:44.615753+0000","last_active":"2024-01-26T23:29:48.633949+0000","last_peered":"2024-01-26T23:29:48.633949+0000","last_clean":"2024-01-26T23:29:48.633949+0000","last_became_active":"2024-01-26T23:29:44.615469+0000","last_became_peered":"2024-01-26T23:29:44.615469+0000","last_unstale":"2024-01-26T23:29:48.633949+0000","last_undegraded":"2024-01-26T23:29:48.633949+0000","last_fullsized":"2024-01-26T23:29:48.633949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:37:23.649270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.651391+0000","last_change":"2024-01-26T23:29:48.646032+0000","last_active":"2024-01-26T23:29:48.651391+0000","last_peered":"2024-01-26T23:29:48.651391+0000","last_clean":"2024-01-26T23:29:48.651391+0000","last_became_active":"2024-01-26T23:29:48.645627+0000","last_became_peered":"2024-01-26T23:29:48.645627+0000","last_unstale":"2024-01-26T23:29:48.651391+0000","last_undegraded":"2024-01-26T23:29:48.651391+0000","last_fullsized":"2024-01-26T23:29:48.651391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:42.473075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632782+0000","last_change":"2024-01-26T23:29:43.614220+0000","last_active":"2024-01-26T23:29:48.632782+0000","last_peered":"2024-01-26T23:29:48.632782+0000","last_clean":"2024-01-26T23:29:48.632782+0000","last_became_active":"2024-01-26T23:29:43.613886+0000","last_became_peered":"2024-01-26T23:29:43.613886+0000","last_unstale":"2024-01-26T23:29:48.632782+0000","last_undegraded":"2024-01-26T23:29:48.632782+0000","last_fullsized":"2024-01-26T23:29:48.632782+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:09:46.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632076+0000","last_change":"2024-01-26T23:29:44.614249+0000","last_active":"2024-01-26T23:29:48.632076+0000","last_peered":"2024-01-26T23:29:48.632076+0000","last_clean":"2024-01-26T23:29:48.632076+0000","last_became_active":"2024-01-26T23:29:44.613949+0000","last_became_peered":"2024-01-26T23:29:44.613949+0000","last_unstale":"2024-01-26T23:29:48.632076+0000","last_undegraded":"2024-01-26T23:29:48.632076+0000","last_fullsized":"2024-01-26T23:29:48.632076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:14.555745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.651517+0000","last_change":"2024-01-26T23:29:48.647625+0000","last_active":"2024-01-26T23:29:48.651517+0000","last_peered":"2024-01-26T23:29:48.651517+0000","last_clean":"2024-01-26T23:29:48.651517+0000","last_became_active":"2024-01-26T23:29:48.647273+0000","last_became_peered":"2024-01-26T23:29:48.647273+0000","last_unstale":"2024-01-26T23:29:48.651517+0000","last_undegraded":"2024-01-26T23:29:48.651517+0000","last_fullsized":"2024-01-26T23:29:48.651517+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:52:44.817581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636975+0000","last_change":"2024-01-26T23:29:43.616939+0000","last_active":"2024-01-26T23:29:48.636975+0000","last_peered":"2024-01-26T23:29:48.636975+0000","last_clean":"2024-01-26T23:29:48.636975+0000","last_became_active":"2024-01-26T23:29:43.614953+0000","last_became_peered":"2024-01-26T23:29:43.614953+0000","last_unstale":"2024-01-26T23:29:48.636975+0000","last_undegraded":"2024-01-26T23:29:48.636975+0000","last_fullsized":"2024-01-26T23:29:48.636975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:36.035429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636683+0000","last_change":"2024-01-26T23:29:44.621908+0000","last_active":"2024-01-26T23:29:48.636683+0000","last_peered":"2024-01-26T23:29:48.636683+0000","last_clean":"2024-01-26T23:29:48.636683+0000","last_became_active":"2024-01-26T23:29:44.621706+0000","last_became_peered":"2024-01-26T23:29:44.621706+0000","last_unstale":"2024-01-26T23:29:48.636683+0000","last_undegraded":"2024-01-26T23:29:48.636683+0000","last_fullsized":"2024-01-26T23:29:48.636683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696535+0000","last_change":"2024-01-26T23:29:48.695257+0000","last_active":"2024-01-26T23:29:48.696535+0000","last_peered":"2024-01-26T23:29:48.696535+0000","last_clean":"2024-01-26T23:29:48.696535+0000","last_became_active":"2024-01-26T23:29:48.694835+0000","last_became_peered":"2024-01-26T23:29:48.694835+0000","last_unstale":"2024-01-26T23:29:48.696535+0000","last_undegraded":"2024-01-26T23:29:48.696535+0000","last_fullsized":"2024-01-26T23:29:48.696535+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:53:55.279661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632759+0000","last_change":"2024-01-26T23:29:43.617477+0000","last_active":"2024-01-26T23:29:48.632759+0000","last_peered":"2024-01-26T23:29:48.632759+0000","last_clean":"2024-01-26T23:29:48.632759+0000","last_became_active":"2024-01-26T23:29:43.617014+0000","last_became_peered":"2024-01-26T23:29:43.617014+0000","last_unstale":"2024-01-26T23:29:48.632759+0000","last_undegraded":"2024-01-26T23:29:48.632759+0000","last_fullsized":"2024-01-26T23:29:48.632759+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:32.417441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626943+0000","last_change":"2024-01-26T23:29:44.619133+0000","last_active":"2024-01-26T23:29:48.626943+0000","last_peered":"2024-01-26T23:29:48.626943+0000","last_clean":"2024-01-26T23:29:48.626943+0000","last_became_active":"2024-01-26T23:29:44.618827+0000","last_became_peered":"2024-01-26T23:29:44.618827+0000","last_unstale":"2024-01-26T23:29:48.626943+0000","last_undegraded":"2024-01-26T23:29:48.626943+0000","last_fullsized":"2024-01-26T23:29:48.626943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696639+0000","last_change":"2024-01-26T23:29:48.693971+0000","last_active":"2024-01-26T23:29:48.696639+0000","last_peered":"2024-01-26T23:29:48.696639+0000","last_clean":"2024-01-26T23:29:48.696639+0000","last_became_active":"2024-01-26T23:29:48.693563+0000","last_became_peered":"2024-01-26T23:29:48.693563+0000","last_unstale":"2024-01-26T23:29:48.696639+0000","last_undegraded":"2024-01-26T23:29:48.696639+0000","last_fullsized":"2024-01-26T23:29:48.696639+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:42:42.888059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.638779+0000","last_change":"2024-01-26T23:29:43.622313+0000","last_active":"2024-01-26T23:29:48.638779+0000","last_peered":"2024-01-26T23:29:48.638779+0000","last_clean":"2024-01-26T23:29:48.638779+0000","last_became_active":"2024-01-26T23:29:43.622008+0000","last_became_peered":"2024-01-26T23:29:43.622008+0000","last_unstale":"2024-01-26T23:29:48.638779+0000","last_undegraded":"2024-01-26T23:29:48.638779+0000","last_fullsized":"2024-01-26T23:29:48.638779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:37:22.647734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.634360+0000","last_change":"2024-01-26T23:29:44.619813+0000","last_active":"2024-01-26T23:29:48.634360+0000","last_peered":"2024-01-26T23:29:48.634360+0000","last_clean":"2024-01-26T23:29:48.634360+0000","last_became_active":"2024-01-26T23:29:44.619466+0000","last_became_peered":"2024-01-26T23:29:44.619466+0000","last_unstale":"2024-01-26T23:29:48.634360+0000","last_undegraded":"2024-01-26T23:29:48.634360+0000","last_fullsized":"2024-01-26T23:29:48.634360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:48:24.085774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.653660+0000","last_change":"2024-01-26T23:29:48.652676+0000","last_active":"2024-01-26T23:29:48.653660+0000","last_peered":"2024-01-26T23:29:48.653660+0000","last_clean":"2024-01-26T23:29:48.653660+0000","last_became_active":"2024-01-26T23:29:48.652134+0000","last_became_peered":"2024-01-26T23:29:48.652134+0000","last_unstale":"2024-01-26T23:29:48.653660+0000","last_undegraded":"2024-01-26T23:29:48.653660+0000","last_fullsized":"2024-01-26T23:29:48.653660+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:27.746532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.634232+0000","last_change":"2024-01-26T23:29:43.620804+0000","last_active":"2024-01-26T23:29:48.634232+0000","last_peered":"2024-01-26T23:29:48.634232+0000","last_clean":"2024-01-26T23:29:48.634232+0000","last_became_active":"2024-01-26T23:29:43.620530+0000","last_became_peered":"2024-01-26T23:29:43.620530+0000","last_unstale":"2024-01-26T23:29:48.634232+0000","last_undegraded":"2024-01-26T23:29:48.634232+0000","last_fullsized":"2024-01-26T23:29:48.634232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:44:34.698447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.634272+0000","last_change":"2024-01-26T23:29:44.626105+0000","last_active":"2024-01-26T23:29:48.634272+0000","last_peered":"2024-01-26T23:29:48.634272+0000","last_clean":"2024-01-26T23:29:48.634272+0000","last_became_active":"2024-01-26T23:29:44.625500+0000","last_became_peered":"2024-01-26T23:29:44.625500+0000","last_unstale":"2024-01-26T23:29:48.634272+0000","last_undegraded":"2024-01-26T23:29:48.634272+0000","last_fullsized":"2024-01-26T23:29:48.634272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.644865+0000","last_change":"2024-01-26T23:29:48.643281+0000","last_active":"2024-01-26T23:29:48.644865+0000","last_peered":"2024-01-26T23:29:48.644865+0000","last_clean":"2024-01-26T23:29:48.644865+0000","last_became_active":"2024-01-26T23:29:48.642907+0000","last_became_peered":"2024-01-26T23:29:48.642907+0000","last_unstale":"2024-01-26T23:29:48.644865+0000","last_undegraded":"2024-01-26T23:29:48.644865+0000","last_fullsized":"2024-01-26T23:29:48.644865+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:24:23.994084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626216+0000","last_change":"2024-01-26T23:29:43.621999+0000","last_active":"2024-01-26T23:29:48.626216+0000","last_peered":"2024-01-26T23:29:48.626216+0000","last_clean":"2024-01-26T23:29:48.626216+0000","last_became_active":"2024-01-26T23:29:43.621214+0000","last_became_peered":"2024-01-26T23:29:43.621214+0000","last_unstale":"2024-01-26T23:29:48.626216+0000","last_undegraded":"2024-01-26T23:29:48.626216+0000","last_fullsized":"2024-01-26T23:29:48.626216+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:37:22.647734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633819+0000","last_change":"2024-01-26T23:29:44.608941+0000","last_active":"2024-01-26T23:29:48.633819+0000","last_peered":"2024-01-26T23:29:48.633819+0000","last_clean":"2024-01-26T23:29:48.633819+0000","last_became_active":"2024-01-26T23:29:44.608664+0000","last_became_peered":"2024-01-26T23:29:44.608664+0000","last_unstale":"2024-01-26T23:29:48.633819+0000","last_undegraded":"2024-01-26T23:29:48.633819+0000","last_fullsized":"2024-01-26T23:29:48.633819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:47:10.097930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.651674+0000","last_change":"2024-01-26T23:29:48.643876+0000","last_active":"2024-01-26T23:29:48.651674+0000","last_peered":"2024-01-26T23:29:48.651674+0000","last_clean":"2024-01-26T23:29:48.651674+0000","last_became_active":"2024-01-26T23:29:48.641104+0000","last_became_peered":"2024-01-26T23:29:48.641104+0000","last_unstale":"2024-01-26T23:29:48.651674+0000","last_undegraded":"2024-01-26T23:29:48.651674+0000","last_fullsized":"2024-01-26T23:29:48.651674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:29:51.471531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-01-26T23:29:54.992720+0000","last_change":"2024-01-26T23:29:43.622211+0000","last_active":"2024-01-26T23:29:54.992720+0000","last_peered":"2024-01-26T23:29:54.992720+0000","last_clean":"2024-01-26T23:29:54.992720+0000","last_became_active":"2024-01-26T23:29:43.621919+0000","last_became_peered":"2024-01-26T23:29:43.621919+0000","last_unstale":"2024-01-26T23:29:54.992720+0000","last_undegraded":"2024-01-26T23:29:54.992720+0000","last_fullsized":"2024-01-26T23:29:54.992720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:45.384500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3713,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690317+0000","last_change":"2024-01-26T23:29:44.618740+0000","last_active":"2024-01-26T23:29:48.690317+0000","last_peered":"2024-01-26T23:29:48.690317+0000","last_clean":"2024-01-26T23:29:48.690317+0000","last_became_active":"2024-01-26T23:29:44.618482+0000","last_became_peered":"2024-01-26T23:29:44.618482+0000","last_unstale":"2024-01-26T23:29:48.690317+0000","last_undegraded":"2024-01-26T23:29:48.690317+0000","last_fullsized":"2024-01-26T23:29:48.690317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:03.314483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695102+0000","last_change":"2024-01-26T23:29:48.691860+0000","last_active":"2024-01-26T23:29:48.695102+0000","last_peered":"2024-01-26T23:29:48.695102+0000","last_clean":"2024-01-26T23:29:48.695102+0000","last_became_active":"2024-01-26T23:29:48.691440+0000","last_became_peered":"2024-01-26T23:29:48.691440+0000","last_unstale":"2024-01-26T23:29:48.695102+0000","last_undegraded":"2024-01-26T23:29:48.695102+0000","last_fullsized":"2024-01-26T23:29:48.695102+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:20.246494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632364+0000","last_change":"2024-01-26T23:29:43.613477+0000","last_active":"2024-01-26T23:29:48.632364+0000","last_peered":"2024-01-26T23:29:48.632364+0000","last_clean":"2024-01-26T23:29:48.632364+0000","last_became_active":"2024-01-26T23:29:43.613228+0000","last_became_peered":"2024-01-26T23:29:43.613228+0000","last_unstale":"2024-01-26T23:29:48.632364+0000","last_undegraded":"2024-01-26T23:29:48.632364+0000","last_fullsized":"2024-01-26T23:29:48.632364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:54:53.198685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631483+0000","last_change":"2024-01-26T23:29:44.613649+0000","last_active":"2024-01-26T23:29:48.631483+0000","last_peered":"2024-01-26T23:29:48.631483+0000","last_clean":"2024-01-26T23:29:48.631483+0000","last_became_active":"2024-01-26T23:29:44.613278+0000","last_became_peered":"2024-01-26T23:29:44.613278+0000","last_unstale":"2024-01-26T23:29:48.631483+0000","last_undegraded":"2024-01-26T23:29:48.631483+0000","last_fullsized":"2024-01-26T23:29:48.631483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:38.452904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696548+0000","last_change":"2024-01-26T23:29:48.691601+0000","last_active":"2024-01-26T23:29:48.696548+0000","last_peered":"2024-01-26T23:29:48.696548+0000","last_clean":"2024-01-26T23:29:48.696548+0000","last_became_active":"2024-01-26T23:29:48.691169+0000","last_became_peered":"2024-01-26T23:29:48.691169+0000","last_unstale":"2024-01-26T23:29:48.696548+0000","last_undegraded":"2024-01-26T23:29:48.696548+0000","last_fullsized":"2024-01-26T23:29:48.696548+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:09.701112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635763+0000","last_change":"2024-01-26T23:29:43.622165+0000","last_active":"2024-01-26T23:29:48.635763+0000","last_peered":"2024-01-26T23:29:48.635763+0000","last_clean":"2024-01-26T23:29:48.635763+0000","last_became_active":"2024-01-26T23:29:43.621876+0000","last_became_peered":"2024-01-26T23:29:43.621876+0000","last_unstale":"2024-01-26T23:29:48.635763+0000","last_undegraded":"2024-01-26T23:29:48.635763+0000","last_fullsized":"2024-01-26T23:29:48.635763+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:30.053494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690214+0000","last_change":"2024-01-26T23:29:44.615999+0000","last_active":"2024-01-26T23:29:48.690214+0000","last_peered":"2024-01-26T23:29:48.690214+0000","last_clean":"2024-01-26T23:29:48.690214+0000","last_became_active":"2024-01-26T23:29:44.615376+0000","last_became_peered":"2024-01-26T23:29:44.615376+0000","last_unstale":"2024-01-26T23:29:48.690214+0000","last_undegraded":"2024-01-26T23:29:48.690214+0000","last_fullsized":"2024-01-26T23:29:48.690214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:32.912164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696674+0000","last_change":"2024-01-26T23:29:48.694846+0000","last_active":"2024-01-26T23:29:48.696674+0000","last_peered":"2024-01-26T23:29:48.696674+0000","last_clean":"2024-01-26T23:29:48.696674+0000","last_became_active":"2024-01-26T23:29:48.694439+0000","last_became_peered":"2024-01-26T23:29:48.694439+0000","last_unstale":"2024-01-26T23:29:48.696674+0000","last_undegraded":"2024-01-26T23:29:48.696674+0000","last_fullsized":"2024-01-26T23:29:48.696674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:39:25.688252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632550+0000","last_change":"2024-01-26T23:29:43.613829+0000","last_active":"2024-01-26T23:29:48.632550+0000","last_peered":"2024-01-26T23:29:48.632550+0000","last_clean":"2024-01-26T23:29:48.632550+0000","last_became_active":"2024-01-26T23:29:43.613486+0000","last_became_peered":"2024-01-26T23:29:43.613486+0000","last_unstale":"2024-01-26T23:29:48.632550+0000","last_undegraded":"2024-01-26T23:29:48.632550+0000","last_fullsized":"2024-01-26T23:29:48.632550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:44:34.698447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633466+0000","last_change":"2024-01-26T23:29:44.614404+0000","last_active":"2024-01-26T23:29:48.633466+0000","last_peered":"2024-01-26T23:29:48.633466+0000","last_clean":"2024-01-26T23:29:48.633466+0000","last_became_active":"2024-01-26T23:29:44.614127+0000","last_became_peered":"2024-01-26T23:29:44.614127+0000","last_unstale":"2024-01-26T23:29:48.633466+0000","last_undegraded":"2024-01-26T23:29:48.633466+0000","last_fullsized":"2024-01-26T23:29:48.633466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:35:11.068882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.653735+0000","last_change":"2024-01-26T23:29:48.647303+0000","last_active":"2024-01-26T23:29:48.653735+0000","last_peered":"2024-01-26T23:29:48.653735+0000","last_clean":"2024-01-26T23:29:48.653735+0000","last_became_active":"2024-01-26T23:29:48.646893+0000","last_became_peered":"2024-01-26T23:29:48.646893+0000","last_unstale":"2024-01-26T23:29:48.653735+0000","last_undegraded":"2024-01-26T23:29:48.653735+0000","last_fullsized":"2024-01-26T23:29:48.653735+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:49.464111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631121+0000","last_change":"2024-01-26T23:29:43.610693+0000","last_active":"2024-01-26T23:29:48.631121+0000","last_peered":"2024-01-26T23:29:48.631121+0000","last_clean":"2024-01-26T23:29:48.631121+0000","last_became_active":"2024-01-26T23:29:43.610370+0000","last_became_peered":"2024-01-26T23:29:43.610370+0000","last_unstale":"2024-01-26T23:29:48.631121+0000","last_undegraded":"2024-01-26T23:29:48.631121+0000","last_fullsized":"2024-01-26T23:29:48.631121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:09:46.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.692648+0000","last_change":"2024-01-26T23:29:44.611702+0000","last_active":"2024-01-26T23:29:48.692648+0000","last_peered":"2024-01-26T23:29:48.692648+0000","last_clean":"2024-01-26T23:29:48.692648+0000","last_became_active":"2024-01-26T23:29:44.611378+0000","last_became_peered":"2024-01-26T23:29:44.611378+0000","last_unstale":"2024-01-26T23:29:48.692648+0000","last_undegraded":"2024-01-26T23:29:48.692648+0000","last_fullsized":"2024-01-26T23:29:48.692648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:38:52.041592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.644666+0000","last_change":"2024-01-26T23:29:48.643205+0000","last_active":"2024-01-26T23:29:48.644666+0000","last_peered":"2024-01-26T23:29:48.644666+0000","last_clean":"2024-01-26T23:29:48.644666+0000","last_became_active":"2024-01-26T23:29:48.642819+0000","last_became_peered":"2024-01-26T23:29:48.642819+0000","last_unstale":"2024-01-26T23:29:48.644666+0000","last_undegraded":"2024-01-26T23:29:48.644666+0000","last_fullsized":"2024-01-26T23:29:48.644666+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:51.210803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633091+0000","last_change":"2024-01-26T23:29:43.612609+0000","last_active":"2024-01-26T23:29:48.633091+0000","last_peered":"2024-01-26T23:29:48.633091+0000","last_clean":"2024-01-26T23:29:48.633091+0000","last_became_active":"2024-01-26T23:29:43.612318+0000","last_became_peered":"2024-01-26T23:29:43.612318+0000","last_unstale":"2024-01-26T23:29:48.633091+0000","last_undegraded":"2024-01-26T23:29:48.633091+0000","last_fullsized":"2024-01-26T23:29:48.633091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:31.317009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690843+0000","last_change":"2024-01-26T23:29:44.616632+0000","last_active":"2024-01-26T23:29:48.690843+0000","last_peered":"2024-01-26T23:29:48.690843+0000","last_clean":"2024-01-26T23:29:48.690843+0000","last_became_active":"2024-01-26T23:29:44.616280+0000","last_became_peered":"2024-01-26T23:29:44.616280+0000","last_unstale":"2024-01-26T23:29:48.690843+0000","last_undegraded":"2024-01-26T23:29:48.690843+0000","last_fullsized":"2024-01-26T23:29:48.690843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:48:24.085774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.654458+0000","last_change":"2024-01-26T23:29:48.644846+0000","last_active":"2024-01-26T23:29:48.654458+0000","last_peered":"2024-01-26T23:29:48.654458+0000","last_clean":"2024-01-26T23:29:48.654458+0000","last_became_active":"2024-01-26T23:29:48.644359+0000","last_became_peered":"2024-01-26T23:29:48.644359+0000","last_unstale":"2024-01-26T23:29:48.654458+0000","last_undegraded":"2024-01-26T23:29:48.654458+0000","last_fullsized":"2024-01-26T23:29:48.654458+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:59.257622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626696+0000","last_change":"2024-01-26T23:29:43.605084+0000","last_active":"2024-01-26T23:29:48.626696+0000","last_peered":"2024-01-26T23:29:48.626696+0000","last_clean":"2024-01-26T23:29:48.626696+0000","last_became_active":"2024-01-26T23:29:43.604761+0000","last_became_peered":"2024-01-26T23:29:43.604761+0000","last_unstale":"2024-01-26T23:29:48.626696+0000","last_undegraded":"2024-01-26T23:29:48.626696+0000","last_fullsized":"2024-01-26T23:29:48.626696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:44:34.698447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636936+0000","last_change":"2024-01-26T23:29:44.632161+0000","last_active":"2024-01-26T23:29:48.636936+0000","last_peered":"2024-01-26T23:29:48.636936+0000","last_clean":"2024-01-26T23:29:48.636936+0000","last_became_active":"2024-01-26T23:29:44.631945+0000","last_became_peered":"2024-01-26T23:29:44.631945+0000","last_unstale":"2024-01-26T23:29:48.636936+0000","last_undegraded":"2024-01-26T23:29:48.636936+0000","last_fullsized":"2024-01-26T23:29:48.636936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:03.314483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.654064+0000","last_change":"2024-01-26T23:29:48.647375+0000","last_active":"2024-01-26T23:29:48.654064+0000","last_peered":"2024-01-26T23:29:48.654064+0000","last_clean":"2024-01-26T23:29:48.654064+0000","last_became_active":"2024-01-26T23:29:48.646978+0000","last_became_peered":"2024-01-26T23:29:48.646978+0000","last_unstale":"2024-01-26T23:29:48.654064+0000","last_undegraded":"2024-01-26T23:29:48.654064+0000","last_fullsized":"2024-01-26T23:29:48.654064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:20.246494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635696+0000","last_change":"2024-01-26T23:29:43.620942+0000","last_active":"2024-01-26T23:29:48.635696+0000","last_peered":"2024-01-26T23:29:48.635696+0000","last_clean":"2024-01-26T23:29:48.635696+0000","last_became_active":"2024-01-26T23:29:43.620644+0000","last_became_peered":"2024-01-26T23:29:43.620644+0000","last_unstale":"2024-01-26T23:29:48.635696+0000","last_undegraded":"2024-01-26T23:29:48.635696+0000","last_fullsized":"2024-01-26T23:29:48.635696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:45.384500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.634352+0000","last_change":"2024-01-26T23:29:44.605909+0000","last_active":"2024-01-26T23:29:48.634352+0000","last_peered":"2024-01-26T23:29:48.634352+0000","last_clean":"2024-01-26T23:29:48.634352+0000","last_became_active":"2024-01-26T23:29:44.605652+0000","last_became_peered":"2024-01-26T23:29:44.605652+0000","last_unstale":"2024-01-26T23:29:48.634352+0000","last_undegraded":"2024-01-26T23:29:48.634352+0000","last_fullsized":"2024-01-26T23:29:48.634352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:31.055030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.654078+0000","last_change":"2024-01-26T23:29:48.647396+0000","last_active":"2024-01-26T23:29:48.654078+0000","last_peered":"2024-01-26T23:29:48.654078+0000","last_clean":"2024-01-26T23:29:48.654078+0000","last_became_active":"2024-01-26T23:29:48.646950+0000","last_became_peered":"2024-01-26T23:29:48.646950+0000","last_unstale":"2024-01-26T23:29:48.654078+0000","last_undegraded":"2024-01-26T23:29:48.654078+0000","last_fullsized":"2024-01-26T23:29:48.654078+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:51.210803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.625864+0000","last_change":"2024-01-26T23:29:44.617670+0000","last_active":"2024-01-26T23:29:48.625864+0000","last_peered":"2024-01-26T23:29:48.625864+0000","last_clean":"2024-01-26T23:29:48.625864+0000","last_became_active":"2024-01-26T23:29:44.616510+0000","last_became_peered":"2024-01-26T23:29:44.616510+0000","last_unstale":"2024-01-26T23:29:48.625864+0000","last_undegraded":"2024-01-26T23:29:48.625864+0000","last_fullsized":"2024-01-26T23:29:48.625864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:12:04.973733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631229+0000","last_change":"2024-01-26T23:29:43.618892+0000","last_active":"2024-01-26T23:29:48.631229+0000","last_peered":"2024-01-26T23:29:48.631229+0000","last_clean":"2024-01-26T23:29:48.631229+0000","last_became_active":"2024-01-26T23:29:43.618446+0000","last_became_peered":"2024-01-26T23:29:43.618446+0000","last_unstale":"2024-01-26T23:29:48.631229+0000","last_undegraded":"2024-01-26T23:29:48.631229+0000","last_fullsized":"2024-01-26T23:29:48.631229+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:36.035429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.694860+0000","last_change":"2024-01-26T23:29:48.691567+0000","last_active":"2024-01-26T23:29:48.694860+0000","last_peered":"2024-01-26T23:29:48.694860+0000","last_clean":"2024-01-26T23:29:48.694860+0000","last_became_active":"2024-01-26T23:29:48.691153+0000","last_became_peered":"2024-01-26T23:29:48.691153+0000","last_unstale":"2024-01-26T23:29:48.694860+0000","last_undegraded":"2024-01-26T23:29:48.694860+0000","last_fullsized":"2024-01-26T23:29:48.694860+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:49.464111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637181+0000","last_change":"2024-01-26T23:29:44.620961+0000","last_active":"2024-01-26T23:29:48.637181+0000","last_peered":"2024-01-26T23:29:48.637181+0000","last_clean":"2024-01-26T23:29:48.637181+0000","last_became_active":"2024-01-26T23:29:44.620701+0000","last_became_peered":"2024-01-26T23:29:44.620701+0000","last_unstale":"2024-01-26T23:29:48.637181+0000","last_undegraded":"2024-01-26T23:29:48.637181+0000","last_fullsized":"2024-01-26T23:29:48.637181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:32.912164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635054+0000","last_change":"2024-01-26T23:29:43.620894+0000","last_active":"2024-01-26T23:29:48.635054+0000","last_peered":"2024-01-26T23:29:48.635054+0000","last_clean":"2024-01-26T23:29:48.635054+0000","last_became_active":"2024-01-26T23:29:43.620603+0000","last_became_peered":"2024-01-26T23:29:43.620603+0000","last_unstale":"2024-01-26T23:29:48.635054+0000","last_undegraded":"2024-01-26T23:29:48.635054+0000","last_fullsized":"2024-01-26T23:29:48.635054+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:36.035429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.689568+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:29:48.689568+0000","last_peered":"2024-01-26T23:29:48.689568+0000","last_clean":"2024-01-26T23:29:48.689568+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:29:48.689568+0000","last_undegraded":"2024-01-26T23:29:48.689568+0000","last_fullsized":"2024-01-26T23:29:48.689568+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696515+0000","last_change":"2024-01-26T23:29:48.692933+0000","last_active":"2024-01-26T23:29:48.696515+0000","last_peered":"2024-01-26T23:29:48.696515+0000","last_clean":"2024-01-26T23:29:48.696515+0000","last_became_active":"2024-01-26T23:29:48.692576+0000","last_became_peered":"2024-01-26T23:29:48.692576+0000","last_unstale":"2024-01-26T23:29:48.696515+0000","last_undegraded":"2024-01-26T23:29:48.696515+0000","last_fullsized":"2024-01-26T23:29:48.696515+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:24:23.994084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631114+0000","last_change":"2024-01-26T23:29:44.601856+0000","last_active":"2024-01-26T23:29:48.631114+0000","last_peered":"2024-01-26T23:29:48.631114+0000","last_clean":"2024-01-26T23:29:48.631114+0000","last_became_active":"2024-01-26T23:29:44.601367+0000","last_became_peered":"2024-01-26T23:29:44.601367+0000","last_unstale":"2024-01-26T23:29:48.631114+0000","last_undegraded":"2024-01-26T23:29:48.631114+0000","last_fullsized":"2024-01-26T23:29:48.631114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:21:47.569396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.689624+0000","last_change":"2024-01-26T23:29:43.623719+0000","last_active":"2024-01-26T23:29:48.689624+0000","last_peered":"2024-01-26T23:29:48.689624+0000","last_clean":"2024-01-26T23:29:48.689624+0000","last_became_active":"2024-01-26T23:29:43.623493+0000","last_became_peered":"2024-01-26T23:29:43.623493+0000","last_unstale":"2024-01-26T23:29:48.689624+0000","last_undegraded":"2024-01-26T23:29:48.689624+0000","last_fullsized":"2024-01-26T23:29:48.689624+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:23:25.049333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695757+0000","last_change":"2024-01-26T23:29:48.691446+0000","last_active":"2024-01-26T23:29:48.695757+0000","last_peered":"2024-01-26T23:29:48.695757+0000","last_clean":"2024-01-26T23:29:48.695757+0000","last_became_active":"2024-01-26T23:29:48.691005+0000","last_became_peered":"2024-01-26T23:29:48.691005+0000","last_unstale":"2024-01-26T23:29:48.695757+0000","last_undegraded":"2024-01-26T23:29:48.695757+0000","last_fullsized":"2024-01-26T23:29:48.695757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:51.210803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690165+0000","last_change":"2024-01-26T23:29:43.624304+0000","last_active":"2024-01-26T23:29:48.690165+0000","last_peered":"2024-01-26T23:29:48.690165+0000","last_clean":"2024-01-26T23:29:48.690165+0000","last_became_active":"2024-01-26T23:29:43.624031+0000","last_became_peered":"2024-01-26T23:29:43.624031+0000","last_unstale":"2024-01-26T23:29:48.690165+0000","last_undegraded":"2024-01-26T23:29:48.690165+0000","last_fullsized":"2024-01-26T23:29:48.690165+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:32.417441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626845+0000","last_change":"2024-01-26T23:29:44.615706+0000","last_active":"2024-01-26T23:29:48.626845+0000","last_peered":"2024-01-26T23:29:48.626845+0000","last_clean":"2024-01-26T23:29:48.626845+0000","last_became_active":"2024-01-26T23:29:44.615377+0000","last_became_peered":"2024-01-26T23:29:44.615377+0000","last_unstale":"2024-01-26T23:29:48.626845+0000","last_undegraded":"2024-01-26T23:29:48.626845+0000","last_fullsized":"2024-01-26T23:29:48.626845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:38.452904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696514+0000","last_change":"2024-01-26T23:29:48.691661+0000","last_active":"2024-01-26T23:29:48.696514+0000","last_peered":"2024-01-26T23:29:48.696514+0000","last_clean":"2024-01-26T23:29:48.696514+0000","last_became_active":"2024-01-26T23:29:48.691233+0000","last_became_peered":"2024-01-26T23:29:48.691233+0000","last_unstale":"2024-01-26T23:29:48.696514+0000","last_undegraded":"2024-01-26T23:29:48.696514+0000","last_fullsized":"2024-01-26T23:29:48.696514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:22:12.593080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626042+0000","last_change":"2024-01-26T23:29:43.614006+0000","last_active":"2024-01-26T23:29:48.626042+0000","last_peered":"2024-01-26T23:29:48.626042+0000","last_clean":"2024-01-26T23:29:48.626042+0000","last_became_active":"2024-01-26T23:29:43.613688+0000","last_became_peered":"2024-01-26T23:29:43.613688+0000","last_unstale":"2024-01-26T23:29:48.626042+0000","last_undegraded":"2024-01-26T23:29:48.626042+0000","last_fullsized":"2024-01-26T23:29:48.626042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:30.053494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.639703+0000","last_change":"2024-01-26T23:29:44.618360+0000","last_active":"2024-01-26T23:29:48.639703+0000","last_peered":"2024-01-26T23:29:48.639703+0000","last_clean":"2024-01-26T23:29:48.639703+0000","last_became_active":"2024-01-26T23:29:44.618061+0000","last_became_peered":"2024-01-26T23:29:44.618061+0000","last_unstale":"2024-01-26T23:29:48.639703+0000","last_undegraded":"2024-01-26T23:29:48.639703+0000","last_fullsized":"2024-01-26T23:29:48.639703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:26:26.776664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695399+0000","last_change":"2024-01-26T23:29:48.694453+0000","last_active":"2024-01-26T23:29:48.695399+0000","last_peered":"2024-01-26T23:29:48.695399+0000","last_clean":"2024-01-26T23:29:48.695399+0000","last_became_active":"2024-01-26T23:29:48.693791+0000","last_became_peered":"2024-01-26T23:29:48.693791+0000","last_unstale":"2024-01-26T23:29:48.695399+0000","last_undegraded":"2024-01-26T23:29:48.695399+0000","last_fullsized":"2024-01-26T23:29:48.695399+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:20.246494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.638726+0000","last_change":"2024-01-26T23:29:43.619037+0000","last_active":"2024-01-26T23:29:48.638726+0000","last_peered":"2024-01-26T23:29:48.638726+0000","last_clean":"2024-01-26T23:29:48.638726+0000","last_became_active":"2024-01-26T23:29:43.618822+0000","last_became_peered":"2024-01-26T23:29:43.618822+0000","last_unstale":"2024-01-26T23:29:48.638726+0000","last_undegraded":"2024-01-26T23:29:48.638726+0000","last_fullsized":"2024-01-26T23:29:48.638726+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:30.053494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633637+0000","last_change":"2024-01-26T23:29:44.615872+0000","last_active":"2024-01-26T23:29:48.633637+0000","last_peered":"2024-01-26T23:29:48.633637+0000","last_clean":"2024-01-26T23:29:48.633637+0000","last_became_active":"2024-01-26T23:29:44.615547+0000","last_became_peered":"2024-01-26T23:29:44.615547+0000","last_unstale":"2024-01-26T23:29:48.633637+0000","last_undegraded":"2024-01-26T23:29:48.633637+0000","last_fullsized":"2024-01-26T23:29:48.633637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:04:39.582710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652459+0000","last_change":"2024-01-26T23:29:48.646587+0000","last_active":"2024-01-26T23:29:48.652459+0000","last_peered":"2024-01-26T23:29:48.652459+0000","last_clean":"2024-01-26T23:29:48.652459+0000","last_became_active":"2024-01-26T23:29:48.646117+0000","last_became_peered":"2024-01-26T23:29:48.646117+0000","last_unstale":"2024-01-26T23:29:48.652459+0000","last_undegraded":"2024-01-26T23:29:48.652459+0000","last_fullsized":"2024-01-26T23:29:48.652459+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:53:55.279661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.689527+0000","last_change":"2024-01-26T23:29:43.623735+0000","last_active":"2024-01-26T23:29:48.689527+0000","last_peered":"2024-01-26T23:29:48.689527+0000","last_clean":"2024-01-26T23:29:48.689527+0000","last_became_active":"2024-01-26T23:29:43.623048+0000","last_became_peered":"2024-01-26T23:29:43.623048+0000","last_unstale":"2024-01-26T23:29:48.689527+0000","last_undegraded":"2024-01-26T23:29:48.689527+0000","last_fullsized":"2024-01-26T23:29:48.689527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:47:09.096394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632551+0000","last_change":"2024-01-26T23:29:44.602462+0000","last_active":"2024-01-26T23:29:48.632551+0000","last_peered":"2024-01-26T23:29:48.632551+0000","last_clean":"2024-01-26T23:29:48.632551+0000","last_became_active":"2024-01-26T23:29:44.602007+0000","last_became_peered":"2024-01-26T23:29:44.602007+0000","last_unstale":"2024-01-26T23:29:48.632551+0000","last_undegraded":"2024-01-26T23:29:48.632551+0000","last_fullsized":"2024-01-26T23:29:48.632551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:23:26.050869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695910+0000","last_change":"2024-01-26T23:29:48.691917+0000","last_active":"2024-01-26T23:29:48.695910+0000","last_peered":"2024-01-26T23:29:48.695910+0000","last_clean":"2024-01-26T23:29:48.695910+0000","last_became_active":"2024-01-26T23:29:48.691441+0000","last_became_peered":"2024-01-26T23:29:48.691441+0000","last_unstale":"2024-01-26T23:29:48.695910+0000","last_undegraded":"2024-01-26T23:29:48.695910+0000","last_fullsized":"2024-01-26T23:29:48.695910+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:52:44.817581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.625872+0000","last_change":"2024-01-26T23:29:43.619776+0000","last_active":"2024-01-26T23:29:48.625872+0000","last_peered":"2024-01-26T23:29:48.625872+0000","last_clean":"2024-01-26T23:29:48.625872+0000","last_became_active":"2024-01-26T23:29:43.619374+0000","last_became_peered":"2024-01-26T23:29:43.619374+0000","last_unstale":"2024-01-26T23:29:48.625872+0000","last_undegraded":"2024-01-26T23:29:48.625872+0000","last_fullsized":"2024-01-26T23:29:48.625872+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:32.417441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636819+0000","last_change":"2024-01-26T23:29:44.621339+0000","last_active":"2024-01-26T23:29:48.636819+0000","last_peered":"2024-01-26T23:29:48.636819+0000","last_clean":"2024-01-26T23:29:48.636819+0000","last_became_active":"2024-01-26T23:29:44.621012+0000","last_became_peered":"2024-01-26T23:29:44.621012+0000","last_unstale":"2024-01-26T23:29:48.636819+0000","last_undegraded":"2024-01-26T23:29:48.636819+0000","last_fullsized":"2024-01-26T23:29:48.636819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:20.138786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652798+0000","last_change":"2024-01-26T23:29:48.650063+0000","last_active":"2024-01-26T23:29:48.652798+0000","last_peered":"2024-01-26T23:29:48.652798+0000","last_clean":"2024-01-26T23:29:48.652798+0000","last_became_active":"2024-01-26T23:29:48.649732+0000","last_became_peered":"2024-01-26T23:29:48.649732+0000","last_unstale":"2024-01-26T23:29:48.652798+0000","last_undegraded":"2024-01-26T23:29:48.652798+0000","last_fullsized":"2024-01-26T23:29:48.652798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:22:12.593080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626014+0000","last_change":"2024-01-26T23:29:43.618706+0000","last_active":"2024-01-26T23:29:48.626014+0000","last_peered":"2024-01-26T23:29:48.626014+0000","last_clean":"2024-01-26T23:29:48.626014+0000","last_became_active":"2024-01-26T23:29:43.618353+0000","last_became_peered":"2024-01-26T23:29:43.618353+0000","last_unstale":"2024-01-26T23:29:48.626014+0000","last_undegraded":"2024-01-26T23:29:48.626014+0000","last_fullsized":"2024-01-26T23:29:48.626014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:45.384500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636855+0000","last_change":"2024-01-26T23:29:44.633055+0000","last_active":"2024-01-26T23:29:48.636855+0000","last_peered":"2024-01-26T23:29:48.636855+0000","last_clean":"2024-01-26T23:29:48.636855+0000","last_became_active":"2024-01-26T23:29:44.631874+0000","last_became_peered":"2024-01-26T23:29:44.631874+0000","last_unstale":"2024-01-26T23:29:48.636855+0000","last_undegraded":"2024-01-26T23:29:48.636855+0000","last_fullsized":"2024-01-26T23:29:48.636855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:01.636855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652530+0000","last_change":"2024-01-26T23:29:48.648195+0000","last_active":"2024-01-26T23:29:48.652530+0000","last_peered":"2024-01-26T23:29:48.652530+0000","last_clean":"2024-01-26T23:29:48.652530+0000","last_became_active":"2024-01-26T23:29:48.647780+0000","last_became_peered":"2024-01-26T23:29:48.647780+0000","last_unstale":"2024-01-26T23:29:48.652530+0000","last_undegraded":"2024-01-26T23:29:48.652530+0000","last_fullsized":"2024-01-26T23:29:48.652530+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:52:44.817581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690272+0000","last_change":"2024-01-26T23:29:43.623737+0000","last_active":"2024-01-26T23:29:48.690272+0000","last_peered":"2024-01-26T23:29:48.690272+0000","last_clean":"2024-01-26T23:29:48.690272+0000","last_became_active":"2024-01-26T23:29:43.623439+0000","last_became_peered":"2024-01-26T23:29:43.623439+0000","last_unstale":"2024-01-26T23:29:48.690272+0000","last_undegraded":"2024-01-26T23:29:48.690272+0000","last_fullsized":"2024-01-26T23:29:48.690272+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:45.384500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632872+0000","last_change":"2024-01-26T23:29:44.613570+0000","last_active":"2024-01-26T23:29:48.632872+0000","last_peered":"2024-01-26T23:29:48.632872+0000","last_clean":"2024-01-26T23:29:48.632872+0000","last_became_active":"2024-01-26T23:29:44.613282+0000","last_became_peered":"2024-01-26T23:29:44.613282+0000","last_unstale":"2024-01-26T23:29:48.632872+0000","last_undegraded":"2024-01-26T23:29:48.632872+0000","last_fullsized":"2024-01-26T23:29:48.632872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:12.500308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696078+0000","last_change":"2024-01-26T23:29:48.691625+0000","last_active":"2024-01-26T23:29:48.696078+0000","last_peered":"2024-01-26T23:29:48.696078+0000","last_clean":"2024-01-26T23:29:48.696078+0000","last_became_active":"2024-01-26T23:29:48.691242+0000","last_became_peered":"2024-01-26T23:29:48.691242+0000","last_unstale":"2024-01-26T23:29:48.696078+0000","last_undegraded":"2024-01-26T23:29:48.696078+0000","last_fullsized":"2024-01-26T23:29:48.696078+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:02.358147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633000+0000","last_change":"2024-01-26T23:29:43.622471+0000","last_active":"2024-01-26T23:29:48.633000+0000","last_peered":"2024-01-26T23:29:48.633000+0000","last_clean":"2024-01-26T23:29:48.633000+0000","last_became_active":"2024-01-26T23:29:43.622106+0000","last_became_peered":"2024-01-26T23:29:43.622106+0000","last_unstale":"2024-01-26T23:29:48.633000+0000","last_undegraded":"2024-01-26T23:29:48.633000+0000","last_fullsized":"2024-01-26T23:29:48.633000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:32.417441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690284+0000","last_change":"2024-01-26T23:29:44.623102+0000","last_active":"2024-01-26T23:29:48.690284+0000","last_peered":"2024-01-26T23:29:48.690284+0000","last_clean":"2024-01-26T23:29:48.690284+0000","last_became_active":"2024-01-26T23:29:44.622834+0000","last_became_peered":"2024-01-26T23:29:44.622834+0000","last_unstale":"2024-01-26T23:29:48.690284+0000","last_undegraded":"2024-01-26T23:29:48.690284+0000","last_fullsized":"2024-01-26T23:29:48.690284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:18:55.543376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.651007+0000","last_change":"2024-01-26T23:29:48.644283+0000","last_active":"2024-01-26T23:29:48.651007+0000","last_peered":"2024-01-26T23:29:48.651007+0000","last_clean":"2024-01-26T23:29:48.651007+0000","last_became_active":"2024-01-26T23:29:48.643862+0000","last_became_peered":"2024-01-26T23:29:48.643862+0000","last_unstale":"2024-01-26T23:29:48.651007+0000","last_undegraded":"2024-01-26T23:29:48.651007+0000","last_fullsized":"2024-01-26T23:29:48.651007+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:29:51.471531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626158+0000","last_change":"2024-01-26T23:29:43.613316+0000","last_active":"2024-01-26T23:29:48.626158+0000","last_peered":"2024-01-26T23:29:48.626158+0000","last_clean":"2024-01-26T23:29:48.626158+0000","last_became_active":"2024-01-26T23:29:43.613054+0000","last_became_peered":"2024-01-26T23:29:43.613054+0000","last_unstale":"2024-01-26T23:29:48.626158+0000","last_undegraded":"2024-01-26T23:29:48.626158+0000","last_fullsized":"2024-01-26T23:29:48.626158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:09:46.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636441+0000","last_change":"2024-01-26T23:29:44.631866+0000","last_active":"2024-01-26T23:29:48.636441+0000","last_peered":"2024-01-26T23:29:48.636441+0000","last_clean":"2024-01-26T23:29:48.636441+0000","last_became_active":"2024-01-26T23:29:44.631581+0000","last_became_peered":"2024-01-26T23:29:44.631581+0000","last_unstale":"2024-01-26T23:29:48.636441+0000","last_undegraded":"2024-01-26T23:29:48.636441+0000","last_fullsized":"2024-01-26T23:29:48.636441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:38:52.041592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695113+0000","last_change":"2024-01-26T23:29:48.691846+0000","last_active":"2024-01-26T23:29:48.695113+0000","last_peered":"2024-01-26T23:29:48.695113+0000","last_clean":"2024-01-26T23:29:48.695113+0000","last_became_active":"2024-01-26T23:29:48.691410+0000","last_became_peered":"2024-01-26T23:29:48.691410+0000","last_unstale":"2024-01-26T23:29:48.695113+0000","last_undegraded":"2024-01-26T23:29:48.695113+0000","last_fullsized":"2024-01-26T23:29:48.695113+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:49:28.002810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633357+0000","last_change":"2024-01-26T23:29:43.618244+0000","last_active":"2024-01-26T23:29:48.633357+0000","last_peered":"2024-01-26T23:29:48.633357+0000","last_clean":"2024-01-26T23:29:48.633357+0000","last_became_active":"2024-01-26T23:29:43.617875+0000","last_became_peered":"2024-01-26T23:29:43.617875+0000","last_unstale":"2024-01-26T23:29:48.633357+0000","last_undegraded":"2024-01-26T23:29:48.633357+0000","last_fullsized":"2024-01-26T23:29:48.633357+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:44:34.698447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690732+0000","last_change":"2024-01-26T23:29:44.618128+0000","last_active":"2024-01-26T23:29:48.690732+0000","last_peered":"2024-01-26T23:29:48.690732+0000","last_clean":"2024-01-26T23:29:48.690732+0000","last_became_active":"2024-01-26T23:29:44.617849+0000","last_became_peered":"2024-01-26T23:29:44.617849+0000","last_unstale":"2024-01-26T23:29:48.690732+0000","last_undegraded":"2024-01-26T23:29:48.690732+0000","last_fullsized":"2024-01-26T23:29:48.690732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:20.138786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633526+0000","last_change":"2024-01-26T23:29:44.615453+0000","last_active":"2024-01-26T23:29:48.633526+0000","last_peered":"2024-01-26T23:29:48.633526+0000","last_clean":"2024-01-26T23:29:48.633526+0000","last_became_active":"2024-01-26T23:29:44.615201+0000","last_became_peered":"2024-01-26T23:29:44.615201+0000","last_unstale":"2024-01-26T23:29:48.633526+0000","last_undegraded":"2024-01-26T23:29:48.633526+0000","last_fullsized":"2024-01-26T23:29:48.633526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:38.452904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631336+0000","last_change":"2024-01-26T23:29:44.619769+0000","last_active":"2024-01-26T23:29:48.631336+0000","last_peered":"2024-01-26T23:29:48.631336+0000","last_clean":"2024-01-26T23:29:48.631336+0000","last_became_active":"2024-01-26T23:29:44.619471+0000","last_became_peered":"2024-01-26T23:29:44.619471+0000","last_unstale":"2024-01-26T23:29:48.631336+0000","last_undegraded":"2024-01-26T23:29:48.631336+0000","last_fullsized":"2024-01-26T23:29:48.631336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:47:10.097930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635849+0000","last_change":"2024-01-26T23:29:44.633598+0000","last_active":"2024-01-26T23:29:48.635849+0000","last_peered":"2024-01-26T23:29:48.635849+0000","last_clean":"2024-01-26T23:29:48.635849+0000","last_became_active":"2024-01-26T23:29:44.633320+0000","last_became_peered":"2024-01-26T23:29:44.633320+0000","last_unstale":"2024-01-26T23:29:48.635849+0000","last_undegraded":"2024-01-26T23:29:48.635849+0000","last_fullsized":"2024-01-26T23:29:48.635849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:03.997854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632174+0000","last_change":"2024-01-26T23:29:44.624890+0000","last_active":"2024-01-26T23:29:48.632174+0000","last_peered":"2024-01-26T23:29:48.632174+0000","last_clean":"2024-01-26T23:29:48.632174+0000","last_became_active":"2024-01-26T23:29:44.624615+0000","last_became_peered":"2024-01-26T23:29:44.624615+0000","last_unstale":"2024-01-26T23:29:48.632174+0000","last_undegraded":"2024-01-26T23:29:48.632174+0000","last_fullsized":"2024-01-26T23:29:48.632174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:12:04.973733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637534+0000","last_change":"2024-01-26T23:29:44.618261+0000","last_active":"2024-01-26T23:29:48.637534+0000","last_peered":"2024-01-26T23:29:48.637534+0000","last_clean":"2024-01-26T23:29:48.637534+0000","last_became_active":"2024-01-26T23:29:44.616559+0000","last_became_peered":"2024-01-26T23:29:44.616559+0000","last_unstale":"2024-01-26T23:29:48.637534+0000","last_undegraded":"2024-01-26T23:29:48.637534+0000","last_fullsized":"2024-01-26T23:29:48.637534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:12:04.973733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626013+0000","last_change":"2024-01-26T23:29:44.619408+0000","last_active":"2024-01-26T23:29:48.626013+0000","last_peered":"2024-01-26T23:29:48.626013+0000","last_clean":"2024-01-26T23:29:48.626013+0000","last_became_active":"2024-01-26T23:29:44.619093+0000","last_became_peered":"2024-01-26T23:29:44.619093+0000","last_unstale":"2024-01-26T23:29:48.626013+0000","last_undegraded":"2024-01-26T23:29:48.626013+0000","last_fullsized":"2024-01-26T23:29:48.626013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:35:11.068882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.638008+0000","last_change":"2024-01-26T23:29:44.614999+0000","last_active":"2024-01-26T23:29:48.638008+0000","last_peered":"2024-01-26T23:29:48.638008+0000","last_clean":"2024-01-26T23:29:48.638008+0000","last_became_active":"2024-01-26T23:29:44.614579+0000","last_became_peered":"2024-01-26T23:29:44.614579+0000","last_unstale":"2024-01-26T23:29:48.638008+0000","last_undegraded":"2024-01-26T23:29:48.638008+0000","last_fullsized":"2024-01-26T23:29:48.638008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.639791+0000","last_change":"2024-01-26T23:29:44.612377+0000","last_active":"2024-01-26T23:29:48.639791+0000","last_peered":"2024-01-26T23:29:48.639791+0000","last_clean":"2024-01-26T23:29:48.639791+0000","last_became_active":"2024-01-26T23:29:44.612073+0000","last_became_peered":"2024-01-26T23:29:44.612073+0000","last_unstale":"2024-01-26T23:29:48.639791+0000","last_undegraded":"2024-01-26T23:29:48.639791+0000","last_fullsized":"2024-01-26T23:29:48.639791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:59:47.867303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632846+0000","last_change":"2024-01-26T23:29:44.633571+0000","last_active":"2024-01-26T23:29:48.632846+0000","last_peered":"2024-01-26T23:29:48.632846+0000","last_clean":"2024-01-26T23:29:48.632846+0000","last_became_active":"2024-01-26T23:29:44.633312+0000","last_became_peered":"2024-01-26T23:29:44.633312+0000","last_unstale":"2024-01-26T23:29:48.632846+0000","last_undegraded":"2024-01-26T23:29:48.632846+0000","last_fullsized":"2024-01-26T23:29:48.632846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:22.113035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637064+0000","last_change":"2024-01-26T23:29:44.617840+0000","last_active":"2024-01-26T23:29:48.637064+0000","last_peered":"2024-01-26T23:29:48.637064+0000","last_clean":"2024-01-26T23:29:48.637064+0000","last_became_active":"2024-01-26T23:29:44.611989+0000","last_became_peered":"2024-01-26T23:29:44.611989+0000","last_unstale":"2024-01-26T23:29:48.637064+0000","last_undegraded":"2024-01-26T23:29:48.637064+0000","last_fullsized":"2024-01-26T23:29:48.637064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:14.555745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635947+0000","last_change":"2024-01-26T23:29:44.617326+0000","last_active":"2024-01-26T23:29:48.635947+0000","last_peered":"2024-01-26T23:29:48.635947+0000","last_clean":"2024-01-26T23:29:48.635947+0000","last_became_active":"2024-01-26T23:29:44.617063+0000","last_became_peered":"2024-01-26T23:29:44.617063+0000","last_unstale":"2024-01-26T23:29:48.635947+0000","last_undegraded":"2024-01-26T23:29:48.635947+0000","last_fullsized":"2024-01-26T23:29:48.635947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:01.636855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632268+0000","last_change":"2024-01-26T23:29:44.613645+0000","last_active":"2024-01-26T23:29:48.632268+0000","last_peered":"2024-01-26T23:29:48.632268+0000","last_clean":"2024-01-26T23:29:48.632268+0000","last_became_active":"2024-01-26T23:29:44.613290+0000","last_became_peered":"2024-01-26T23:29:44.613290+0000","last_unstale":"2024-01-26T23:29:48.632268+0000","last_undegraded":"2024-01-26T23:29:48.632268+0000","last_fullsized":"2024-01-26T23:29:48.632268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:12.500308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637188+0000","last_change":"2024-01-26T23:29:44.609934+0000","last_active":"2024-01-26T23:29:48.637188+0000","last_peered":"2024-01-26T23:29:48.637188+0000","last_clean":"2024-01-26T23:29:48.637188+0000","last_became_active":"2024-01-26T23:29:44.609682+0000","last_became_peered":"2024-01-26T23:29:44.609682+0000","last_unstale":"2024-01-26T23:29:48.637188+0000","last_undegraded":"2024-01-26T23:29:48.637188+0000","last_fullsized":"2024-01-26T23:29:48.637188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:12:04.973733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4483,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298120,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073720,"statfs":{"total":96636764160,"available":96331489280,"internally_reserved":0,"allocated":851968,"data_stored":690410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":103201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297544,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074296,"statfs":{"total":96636764160,"available":96332079104,"internally_reserved":0,"allocated":262144,"data_stored":99648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298128,"kb_used_data":840,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073712,"statfs":{"total":96636764160,"available":96331481088,"internally_reserved":0,"allocated":860160,"data_stored":690466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297536,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074304,"statfs":{"total":96636764160,"available":96332087296,"internally_reserved":0,"allocated":253952,"data_stored":99488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.221}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":103201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":99614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297540,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074300,"statfs":{"total":96636764160,"available":96332083200,"internally_reserved":0,"allocated":258048,"data_stored":99578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:30:05.265 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-26T23:30:05.266 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-26T23:30:05.266 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-26T23:30:05.266 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-26T23:30:05.266 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-26T23:30:05.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.435+0000 7efd56f1f700 1 Processor -- start 2024-01-26T23:30:05.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.435+0000 7efd56f1f700 1 -- start start 2024-01-26T23:30:05.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.436+0000 7efd56f1f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd501596e0 0x7efd50159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.436+0000 7efd56f1f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efd500cb4c0 0x7efd50068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.436+0000 7efd56f1f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd500c97f0 con 0x7efd500c4720 2024-01-26T23:30:05.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.436+0000 7efd56f1f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd500cbf90 con 0x7efd500cb4c0 2024-01-26T23:30:05.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.436+0000 7efd56f1f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd500c8c20 con 0x7efd501596e0 2024-01-26T23:30:05.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.436+0000 7efd554bc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efd500cb4c0 0x7efd50068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.436+0000 7efd4ffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efd500c4720 0x7efd500cadb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44418/0 (socket says 172.21.15.46:44418) 2024-01-26T23:30:05.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.436+0000 7efd4ffff700 1 -- 172.21.15.46:0/648041633 learned_addr learned my addr 172.21.15.46:0/648041633 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:05.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.436+0000 7efd54cbb700 1 --2- 172.21.15.46:0/648041633 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd501596e0 0x7efd50159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.437+0000 7efd554bc700 1 -- 172.21.15.46:0/648041633 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd501596e0 msgr2=0x7efd50159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:05.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.437+0000 7efd554bc700 1 --2- 172.21.15.46:0/648041633 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd501596e0 0x7efd50159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.437+0000 7efd554bc700 1 -- 172.21.15.46:0/648041633 >> v1:172.21.15.46:6789/0 conn(0x7efd500c4720 legacy=0x7efd500cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:05.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.437+0000 7efd554bc700 1 -- 172.21.15.46:0/648041633 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd500c8ec0 con 0x7efd500cb4c0 2024-01-26T23:30:05.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.437+0000 7efd4f7fe700 1 -- 172.21.15.46:0/648041633 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193146150 0 0) 0x7efd500c97f0 con 0x7efd500c4720 2024-01-26T23:30:05.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.437+0000 7efd54cbb700 1 --2- 172.21.15.46:0/648041633 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd501596e0 0x7efd50159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:05.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.437+0000 7efd554bc700 1 --2- 172.21.15.46:0/648041633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd500cb4c0 0x7efd50068a40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efd48009e60 tx=0x7efd48009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dce80794970b671 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:05.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.437+0000 7efd4f7fe700 1 -- 172.21.15.46:0/648041633 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4800e070 con 0x7efd500cb4c0 2024-01-26T23:30:05.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.437+0000 7efd4f7fe700 1 -- 172.21.15.46:0/648041633 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd4800bc30 con 0x7efd500cb4c0 2024-01-26T23:30:05.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.437+0000 7efd4f7fe700 1 -- 172.21.15.46:0/648041633 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd48012430 con 0x7efd500cb4c0 2024-01-26T23:30:05.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.438+0000 7efd56f1f700 1 -- 172.21.15.46:0/648041633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd500cb4c0 msgr2=0x7efd50068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:05.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.438+0000 7efd56f1f700 1 --2- 172.21.15.46:0/648041633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd500cb4c0 0x7efd50068a40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efd48009e60 tx=0x7efd48009780 comp rx=0 tx=0).stop 2024-01-26T23:30:05.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.438+0000 7efd56f1f700 1 -- 172.21.15.46:0/648041633 shutdown_connections 2024-01-26T23:30:05.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.438+0000 7efd56f1f700 1 --2- 172.21.15.46:0/648041633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd500cb4c0 0x7efd50068a40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.438+0000 7efd56f1f700 1 --2- 172.21.15.46:0/648041633 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd501596e0 0x7efd50159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.438+0000 7efd56f1f700 1 -- 172.21.15.46:0/648041633 >> 172.21.15.46:0/648041633 conn(0x7efd50097820 msgr2=0x7efd50068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:05.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.438+0000 7efd56f1f700 1 -- 172.21.15.46:0/648041633 shutdown_connections 2024-01-26T23:30:05.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.438+0000 7efd56f1f700 1 -- 172.21.15.46:0/648041633 wait complete. 2024-01-26T23:30:05.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.439+0000 7efd56f1f700 1 Processor -- start 2024-01-26T23:30:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.439+0000 7efd56f1f700 1 -- start start 2024-01-26T23:30:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd56f1f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd500c4720 0x7efd50154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd56f1f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd500cb4c0 0x7efd500b1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd56f1f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd501596e0 0x7efd500c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd56f1f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd500b1a20 con 0x7efd500c4720 2024-01-26T23:30:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd56f1f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd500c85b0 con 0x7efd500cb4c0 2024-01-26T23:30:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd56f1f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd500b1410 con 0x7efd501596e0 2024-01-26T23:30:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd54cbb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd500c4720 0x7efd50154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd54cbb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd500c4720 0x7efd50154250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33788/0 (socket says 172.21.15.46:33788) 2024-01-26T23:30:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd54cbb700 1 -- 172.21.15.46:0/784107584 learned_addr learned my addr 172.21.15.46:0/784107584 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd554bc700 1 --2- 172.21.15.46:0/784107584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd501596e0 0x7efd500c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd4ffff700 1 --2- 172.21.15.46:0/784107584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd500cb4c0 0x7efd500b1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd54cbb700 1 -- 172.21.15.46:0/784107584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd501596e0 msgr2=0x7efd500c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd54cbb700 1 --2- 172.21.15.46:0/784107584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd501596e0 0x7efd500c79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd54cbb700 1 -- 172.21.15.46:0/784107584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd500cb4c0 msgr2=0x7efd500b1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd54cbb700 1 --2- 172.21.15.46:0/784107584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd500cb4c0 0x7efd500b1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd54cbb700 1 -- 172.21.15.46:0/784107584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd48009a20 con 0x7efd500c4720 2024-01-26T23:30:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd554bc700 1 --2- 172.21.15.46:0/784107584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd501596e0 0x7efd500c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.440+0000 7efd54cbb700 1 --2- 172.21.15.46:0/784107584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd500c4720 0x7efd50154250 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efd4000d860 tx=0x7efd40008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.441+0000 7efd4cff9700 1 -- 172.21.15.46:0/784107584 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd400125f0 con 0x7efd500c4720 2024-01-26T23:30:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.441+0000 7efd4cff9700 1 -- 172.21.15.46:0/784107584 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd40012750 con 0x7efd500c4720 2024-01-26T23:30:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.441+0000 7efd56f1f700 1 -- 172.21.15.46:0/784107584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd50154790 con 0x7efd500c4720 2024-01-26T23:30:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.441+0000 7efd4cff9700 1 -- 172.21.15.46:0/784107584 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd40012ad0 con 0x7efd500c4720 2024-01-26T23:30:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.441+0000 7efd56f1f700 1 -- 172.21.15.46:0/784107584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd500b16b0 con 0x7efd500c4720 2024-01-26T23:30:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.442+0000 7efd56f1f700 1 -- 172.21.15.46:0/784107584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd5004c710 con 0x7efd500c4720 2024-01-26T23:30:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.442+0000 7efd4cff9700 1 -- 172.21.15.46:0/784107584 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7efd40014430 con 0x7efd500c4720 2024-01-26T23:30:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.443+0000 7efd4cff9700 1 --2- 172.21.15.46:0/784107584 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd30066f30 0x7efd300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.443+0000 7efd4cff9700 1 -- 172.21.15.46:0/784107584 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7efd40084ca0 con 0x7efd500c4720 2024-01-26T23:30:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.443+0000 7efd4ffff700 1 --2- 172.21.15.46:0/784107584 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd30066f30 0x7efd300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.443+0000 7efd4ffff700 1 --2- 172.21.15.46:0/784107584 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd30066f30 0x7efd300693e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efd500c96e0 tx=0x7efd38009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.447+0000 7efd4cff9700 1 -- 172.21.15.46:0/784107584 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efd400556b0 con 0x7efd500c4720 2024-01-26T23:30:05.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.711+0000 7efd56f1f700 1 -- 172.21.15.46:0/784107584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7efd500c91a0 con 0x7efd500c4720 2024-01-26T23:30:05.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.714+0000 7efd4cff9700 1 -- 172.21.15.46:0/784107584 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+163707 (secure 0 0 0) 0x7efd40055240 con 0x7efd500c4720 2024-01-26T23:30:05.717 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:05.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.717+0000 7efd56f1f700 1 -- 172.21.15.46:0/784107584 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd30066f30 msgr2=0x7efd300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:05.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.717+0000 7efd56f1f700 1 --2- 172.21.15.46:0/784107584 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd30066f30 0x7efd300693e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efd500c96e0 tx=0x7efd38009040 comp rx=0 tx=0).stop 2024-01-26T23:30:05.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.717+0000 7efd56f1f700 1 -- 172.21.15.46:0/784107584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd500c4720 msgr2=0x7efd50154250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:05.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.717+0000 7efd56f1f700 1 --2- 172.21.15.46:0/784107584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd500c4720 0x7efd50154250 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efd4000d860 tx=0x7efd40008040 comp rx=0 tx=0).stop 2024-01-26T23:30:05.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.717+0000 7efd56f1f700 1 -- 172.21.15.46:0/784107584 shutdown_connections 2024-01-26T23:30:05.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.717+0000 7efd56f1f700 1 --2- 172.21.15.46:0/784107584 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd30066f30 0x7efd300693e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.717+0000 7efd56f1f700 1 --2- 172.21.15.46:0/784107584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd500c4720 0x7efd50154250 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.717+0000 7efd56f1f700 1 --2- 172.21.15.46:0/784107584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd501596e0 0x7efd500c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.717+0000 7efd56f1f700 1 --2- 172.21.15.46:0/784107584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd500cb4c0 0x7efd500b1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.717+0000 7efd56f1f700 1 -- 172.21.15.46:0/784107584 >> 172.21.15.46:0/784107584 conn(0x7efd50097820 msgr2=0x7efd5006c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:05.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.717+0000 7efd56f1f700 1 -- 172.21.15.46:0/784107584 shutdown_connections 2024-01-26T23:30:05.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.717+0000 7efd56f1f700 1 -- 172.21.15.46:0/784107584 wait complete. 2024-01-26T23:30:05.732 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":43388},{"type":"v1","addr":"172.21.15.106:6833","nonce":43388}]},"active_addr":"172.21.15.106:6833/43388","active_change":"2024-01-26T23:29:32.214710+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4108,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":927283051}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":3967117552}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":3492442819}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":2294180308}]}]} 2024-01-26T23:30:05.736 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-26T23:30:05.736 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-26T23:30:05.736 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:05.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.868+0000 7f1fb1c8d700 1 Processor -- start 2024-01-26T23:30:05.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.868+0000 7f1fb1c8d700 1 -- start start 2024-01-26T23:30:05.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.869+0000 7f1fb1c8d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1fac1561d0 0x7f1fac0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.869+0000 7f1fb1c8d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1fac06cbb0 0x7f1fac1565a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.869+0000 7f1fb1c8d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fac0b0760 con 0x7f1fac0b0280 2024-01-26T23:30:05.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.869+0000 7f1fb1c8d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fac0b0a40 con 0x7f1fac06cbb0 2024-01-26T23:30:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.869+0000 7f1fb1c8d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fac0b0d20 con 0x7f1fac1561d0 2024-01-26T23:30:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.869+0000 7f1fabfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1fac06cbb0 0x7f1fac1565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.869+0000 7f1faaffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1fac0b0280 0x7f1fac06c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44434/0 (socket says 172.21.15.46:44434) 2024-01-26T23:30:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.869+0000 7f1fabfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1fac06cbb0 0x7f1fac1565a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33792/0 (socket says 172.21.15.46:33792) 2024-01-26T23:30:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.869+0000 7f1faaffd700 1 -- 172.21.15.46:0/1926244305 learned_addr learned my addr 172.21.15.46:0/1926244305 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.869+0000 7f1fab7fe700 1 --2- 172.21.15.46:0/1926244305 >> v2:172.21.15.106:3300/0 conn(0x7f1fac1561d0 0x7f1fac0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.870+0000 7f1fabfff700 1 -- 172.21.15.46:0/1926244305 >> v2:172.21.15.106:3300/0 conn(0x7f1fac1561d0 msgr2=0x7f1fac0ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.870+0000 7f1fabfff700 1 --2- 172.21.15.46:0/1926244305 >> v2:172.21.15.106:3300/0 conn(0x7f1fac1561d0 0x7f1fac0ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.870+0000 7f1fabfff700 1 -- 172.21.15.46:0/1926244305 >> v1:172.21.15.46:6789/0 conn(0x7f1fac0b0280 legacy=0x7f1fac06c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.870+0000 7f1fabfff700 1 -- 172.21.15.46:0/1926244305 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fac0b1000 con 0x7f1fac06cbb0 2024-01-26T23:30:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.870+0000 7f1faa7fc700 1 -- 172.21.15.46:0/1926244305 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889915206 0 0) 0x7f1fac0b0760 con 0x7f1fac0b0280 2024-01-26T23:30:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.870+0000 7f1fabfff700 1 --2- 172.21.15.46:0/1926244305 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fac06cbb0 0x7f1fac1565a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1fac06bb50 tx=0x7f1fa4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad50342d23e4daf9 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.870+0000 7f1faa7fc700 1 -- 172.21.15.46:0/1926244305 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa400e070 con 0x7f1fac06cbb0 2024-01-26T23:30:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.870+0000 7f1faa7fc700 1 -- 172.21.15.46:0/1926244305 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa400bb60 con 0x7f1fac06cbb0 2024-01-26T23:30:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.870+0000 7f1faa7fc700 1 -- 172.21.15.46:0/1926244305 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4012430 con 0x7f1fac06cbb0 2024-01-26T23:30:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.870+0000 7f1fb1c8d700 1 -- 172.21.15.46:0/1926244305 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fac06cbb0 msgr2=0x7f1fac1565a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.870+0000 7f1fb1c8d700 1 --2- 172.21.15.46:0/1926244305 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fac06cbb0 0x7f1fac1565a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1fac06bb50 tx=0x7f1fa4009780 comp rx=0 tx=0).stop 2024-01-26T23:30:05.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.871+0000 7f1fb1c8d700 1 -- 172.21.15.46:0/1926244305 shutdown_connections 2024-01-26T23:30:05.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.871+0000 7f1fb1c8d700 1 --2- 172.21.15.46:0/1926244305 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fac06cbb0 0x7f1fac1565a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.871+0000 7f1fb1c8d700 1 --2- 172.21.15.46:0/1926244305 >> v2:172.21.15.106:3300/0 conn(0x7f1fac1561d0 0x7f1fac0ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.871+0000 7f1fb1c8d700 1 -- 172.21.15.46:0/1926244305 >> 172.21.15.46:0/1926244305 conn(0x7f1fac06b630 msgr2=0x7f1fac0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:05.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.871+0000 7f1fb1c8d700 1 -- 172.21.15.46:0/1926244305 shutdown_connections 2024-01-26T23:30:05.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.871+0000 7f1fb1c8d700 1 -- 172.21.15.46:0/1926244305 wait complete. 2024-01-26T23:30:05.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.872+0000 7f1fb1c8d700 1 Processor -- start 2024-01-26T23:30:05.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.873+0000 7f1fb1c8d700 1 -- start start 2024-01-26T23:30:05.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.873+0000 7f1fb1c8d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1fac06cbb0 0x7f1fac141dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.873+0000 7f1fb1c8d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1fac0b0280 0x7f1fac169e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.873+0000 7f1fb1c8d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fac1561d0 0x7f1fac16a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1fb1c8d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac16db20 con 0x7f1fac1561d0 2024-01-26T23:30:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1fb1c8d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac16a9d0 con 0x7f1fac06cbb0 2024-01-26T23:30:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1fb1c8d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac16acb0 con 0x7f1fac0b0280 2024-01-26T23:30:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1fabfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fac1561d0 0x7f1fac16a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1fabfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fac1561d0 0x7f1fac16a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33802/0 (socket says 172.21.15.46:33802) 2024-01-26T23:30:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1fabfff700 1 -- 172.21.15.46:0/846568795 learned_addr learned my addr 172.21.15.46:0/846568795 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1fab7fe700 1 --2- 172.21.15.46:0/846568795 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1fac06cbb0 0x7f1fac141dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1faaffd700 1 --2- 172.21.15.46:0/846568795 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1fac0b0280 0x7f1fac169e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1fabfff700 1 -- 172.21.15.46:0/846568795 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1fac0b0280 msgr2=0x7f1fac169e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1fabfff700 1 --2- 172.21.15.46:0/846568795 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1fac0b0280 0x7f1fac169e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1fabfff700 1 -- 172.21.15.46:0/846568795 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1fac06cbb0 msgr2=0x7f1fac141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1fabfff700 1 --2- 172.21.15.46:0/846568795 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1fac06cbb0 0x7f1fac141dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1fabfff700 1 -- 172.21.15.46:0/846568795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa4009a20 con 0x7f1fac1561d0 2024-01-26T23:30:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1fab7fe700 1 --2- 172.21.15.46:0/846568795 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1fac06cbb0 0x7f1fac141dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1fabfff700 1 --2- 172.21.15.46:0/846568795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fac1561d0 0x7f1fac16a370 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4013000 tx=0x7f1fa4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1f937fe700 1 -- 172.21.15.46:0/846568795 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4007000 con 0x7f1fac1561d0 2024-01-26T23:30:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1f937fe700 1 -- 172.21.15.46:0/846568795 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa40263f0 con 0x7f1fac1561d0 2024-01-26T23:30:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1fb1c8d700 1 -- 172.21.15.46:0/846568795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fac16d4d0 con 0x7f1fac1561d0 2024-01-26T23:30:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.874+0000 7f1f937fe700 1 -- 172.21.15.46:0/846568795 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa40267b0 con 0x7f1fac1561d0 2024-01-26T23:30:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.875+0000 7f1fb1c8d700 1 -- 172.21.15.46:0/846568795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fac14f890 con 0x7f1fac1561d0 2024-01-26T23:30:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.876+0000 7f1fb1c8d700 1 -- 172.21.15.46:0/846568795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fac06fa80 con 0x7f1fac1561d0 2024-01-26T23:30:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.876+0000 7f1f937fe700 1 -- 172.21.15.46:0/846568795 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1fa402e3f0 con 0x7f1fac1561d0 2024-01-26T23:30:05.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.876+0000 7f1f937fe700 1 --2- 172.21.15.46:0/846568795 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1f98066f30 0x7f1f980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:05.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.876+0000 7f1f937fe700 1 -- 172.21.15.46:0/846568795 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f1fa40861e0 con 0x7f1fac1561d0 2024-01-26T23:30:05.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.877+0000 7f1fab7fe700 1 --2- 172.21.15.46:0/846568795 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1f98066f30 0x7f1f980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:05.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.877+0000 7f1fab7fe700 1 --2- 172.21.15.46:0/846568795 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1f98066f30 0x7f1f980693e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0113e0 tx=0x7f1f9c0091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:05.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:05.881+0000 7f1f937fe700 1 -- 172.21.15.46:0/846568795 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1fa4053160 con 0x7f1fac1561d0 2024-01-26T23:30:06.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.084+0000 7f1fb1c8d700 1 -- 172.21.15.46:0/846568795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fac04c710 con 0x7f1fac1561d0 2024-01-26T23:30:06.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.085+0000 7f1f937fe700 1 -- 172.21.15.46:0/846568795 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7f1fa4056780 con 0x7f1fac1561d0 2024-01-26T23:30:06.086 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:06.086 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:49.075104+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:06.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.087+0000 7f1fb1c8d700 1 -- 172.21.15.46:0/846568795 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1f98066f30 msgr2=0x7f1f980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.087+0000 7f1fb1c8d700 1 --2- 172.21.15.46:0/846568795 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1f98066f30 0x7f1f980693e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0113e0 tx=0x7f1f9c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.087+0000 7f1fb1c8d700 1 -- 172.21.15.46:0/846568795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fac1561d0 msgr2=0x7f1fac16a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.087+0000 7f1fb1c8d700 1 --2- 172.21.15.46:0/846568795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fac1561d0 0x7f1fac16a370 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4013000 tx=0x7f1fa4019000 comp rx=0 tx=0).stop 2024-01-26T23:30:06.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.087+0000 7f1fb1c8d700 1 -- 172.21.15.46:0/846568795 shutdown_connections 2024-01-26T23:30:06.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.087+0000 7f1fb1c8d700 1 --2- 172.21.15.46:0/846568795 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1f98066f30 0x7f1f980693e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.087+0000 7f1fb1c8d700 1 --2- 172.21.15.46:0/846568795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fac1561d0 0x7f1fac16a370 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.087+0000 7f1fb1c8d700 1 --2- 172.21.15.46:0/846568795 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1fac06cbb0 0x7f1fac141dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.087+0000 7f1fb1c8d700 1 --2- 172.21.15.46:0/846568795 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1fac0b0280 0x7f1fac169e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.087+0000 7f1fb1c8d700 1 -- 172.21.15.46:0/846568795 >> 172.21.15.46:0/846568795 conn(0x7f1fac06b630 msgr2=0x7f1fac155120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.087+0000 7f1fb1c8d700 1 -- 172.21.15.46:0/846568795 shutdown_connections 2024-01-26T23:30:06.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.087+0000 7f1fb1c8d700 1 -- 172.21.15.46:0/846568795 wait complete. 2024-01-26T23:30:06.102 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-26T23:30:06.102 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:06.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.234+0000 7fb9c6635700 1 Processor -- start 2024-01-26T23:30:06.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.234+0000 7fb9c6635700 1 -- start start 2024-01-26T23:30:06.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.235+0000 7fb9c6635700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9c00acf30 0x7fb9c00c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.235+0000 7fb9c6635700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c00c4230 con 0x7fb9c00ad770 2024-01-26T23:30:06.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.235+0000 7fb9c6635700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c00cbf10 con 0x7fb9c00ad300 2024-01-26T23:30:06.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.235+0000 7fb9c6635700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c0157be0 con 0x7fb9c00acf30 2024-01-26T23:30:06.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.235+0000 7fb9c4bd2700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb9c00ad770 0x7fb9c00ab1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44450/0 (socket says 172.21.15.46:44450) 2024-01-26T23:30:06.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.235+0000 7fb9c4bd2700 1 -- 172.21.15.46:0/3614115964 learned_addr learned my addr 172.21.15.46:0/3614115964 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:06.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.235+0000 7fb9bffff700 1 --2- 172.21.15.46:0/3614115964 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9c00acf30 0x7fb9c00c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.235+0000 7fb9beffd700 1 -- 172.21.15.46:0/3614115964 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734113845 0 0) 0x7fb9c00c4230 con 0x7fb9c00ad770 2024-01-26T23:30:06.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.235+0000 7fb9beffd700 1 -- 172.21.15.46:0/3614115964 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a8003570 con 0x7fb9c00ad770 2024-01-26T23:30:06.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9beffd700 1 -- 172.21.15.46:0/3614115964 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 730605756 0 0) 0x7fb9a8003570 con 0x7fb9c00ad770 2024-01-26T23:30:06.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9beffd700 1 -- 172.21.15.46:0/3614115964 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9c00acf30 msgr2=0x7fb9c00c3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:06.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9beffd700 1 --2- 172.21.15.46:0/3614115964 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9c00acf30 0x7fb9c00c3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9beffd700 1 -- 172.21.15.46:0/3614115964 >> v1:172.21.15.106:6789/0 conn(0x7fb9c00ad300 legacy=0x7fb9c00ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9beffd700 1 -- 172.21.15.46:0/3614115964 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c00c4460 con 0x7fb9c00ad770 2024-01-26T23:30:06.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9beffd700 1 -- 172.21.15.46:0/3614115964 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fb9b80017f0 con 0x7fb9c00ad770 2024-01-26T23:30:06.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9bffff700 1 --2- 172.21.15.46:0/3614115964 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9c00acf30 0x7fb9c00c3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:06.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9beffd700 1 -- 172.21.15.46:0/3614115964 >> v1:172.21.15.46:6789/0 conn(0x7fb9c00ad770 legacy=0x7fb9c00ab1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9beffd700 1 --2- 172.21.15.46:0/3614115964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a8003a90 0x7fb9a8003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9beffd700 1 --2- 172.21.15.46:0/3614115964 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9a8004810 0x7fb9a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9beffd700 1 --2- 172.21.15.46:0/3614115964 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9c00acf30 0x7fb9a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9beffd700 1 -- 172.21.15.46:0/3614115964 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c0157be0 con 0x7fb9a8003a90 2024-01-26T23:30:06.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9beffd700 1 -- 172.21.15.46:0/3614115964 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a800bb10 con 0x7fb9a8004810 2024-01-26T23:30:06.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9beffd700 1 -- 172.21.15.46:0/3614115964 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a800bc80 con 0x7fb9c00acf30 2024-01-26T23:30:06.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9bffff700 1 --2- 172.21.15.46:0/3614115964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a8003a90 0x7fb9a8003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9bf7fe700 1 --2- 172.21.15.46:0/3614115964 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9a8004810 0x7fb9a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9c4bd2700 1 --2- 172.21.15.46:0/3614115964 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9c00acf30 0x7fb9a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9bffff700 1 -- 172.21.15.46:0/3614115964 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9c00acf30 msgr2=0x7fb9a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:06.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9bffff700 1 --2- 172.21.15.46:0/3614115964 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9c00acf30 0x7fb9a80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9bffff700 1 -- 172.21.15.46:0/3614115964 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9a8004810 msgr2=0x7fb9a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:06.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9bffff700 1 --2- 172.21.15.46:0/3614115964 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9a8004810 0x7fb9a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9bffff700 1 -- 172.21.15.46:0/3614115964 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b0009420 con 0x7fb9a8003a90 2024-01-26T23:30:06.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.236+0000 7fb9bffff700 1 --2- 172.21.15.46:0/3614115964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a8003a90 0x7fb9a8003e60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009df0 tx=0x7fb9b00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0e937765ace1c5a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.237+0000 7fb9beffd700 1 -- 172.21.15.46:0/3614115964 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b0009740 con 0x7fb9a8003a90 2024-01-26T23:30:06.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.237+0000 7fb9beffd700 1 -- 172.21.15.46:0/3614115964 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b000e8c0 con 0x7fb9a8003a90 2024-01-26T23:30:06.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.237+0000 7fb9beffd700 1 -- 172.21.15.46:0/3614115964 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b000ec80 con 0x7fb9a8003a90 2024-01-26T23:30:06.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.237+0000 7fb9c6635700 1 -- 172.21.15.46:0/3614115964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a8003a90 msgr2=0x7fb9a8003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.237+0000 7fb9c6635700 1 --2- 172.21.15.46:0/3614115964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a8003a90 0x7fb9a8003e60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009df0 tx=0x7fb9b00091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.237+0000 7fb9c4bd2700 1 -- 172.21.15.46:0/3614115964 reap_dead start 2024-01-26T23:30:06.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.237+0000 7fb9c6635700 1 -- 172.21.15.46:0/3614115964 shutdown_connections 2024-01-26T23:30:06.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.237+0000 7fb9c6635700 1 -- 172.21.15.46:0/3614115964 >> 172.21.15.46:0/3614115964 conn(0x7fb9c0097820 msgr2=0x7fb9c0068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.237+0000 7fb9c6635700 1 -- 172.21.15.46:0/3614115964 shutdown_connections 2024-01-26T23:30:06.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.237+0000 7fb9c6635700 1 -- 172.21.15.46:0/3614115964 wait complete. 2024-01-26T23:30:06.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.238+0000 7fb9c6635700 1 Processor -- start 2024-01-26T23:30:06.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.239+0000 7fb9c6635700 1 -- start start 2024-01-26T23:30:06.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.239+0000 7fb9c6635700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a8003a90 0x7fb9c0168ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.239+0000 7fb9c6635700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9c0140bc0 0x7fb9c00c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.239+0000 7fb9c6635700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9c00c82b0 0x7fb9c016b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.239+0000 7fb9c6635700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c00c9aa0 con 0x7fb9a8003a90 2024-01-26T23:30:06.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.239+0000 7fb9c6635700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c00c5a00 con 0x7fb9c00c82b0 2024-01-26T23:30:06.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.239+0000 7fb9c6635700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c00c5ce0 con 0x7fb9c0140bc0 2024-01-26T23:30:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.240+0000 7fb9bffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a8003a90 0x7fb9c0168ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.240+0000 7fb9bffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a8003a90 0x7fb9c0168ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33824/0 (socket says 172.21.15.46:33824) 2024-01-26T23:30:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.240+0000 7fb9bffff700 1 -- 172.21.15.46:0/823754392 learned_addr learned my addr 172.21.15.46:0/823754392 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.240+0000 7fb9bf7fe700 1 --2- 172.21.15.46:0/823754392 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9c0140bc0 0x7fb9c00c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.240+0000 7fb9c4bd2700 1 --2- 172.21.15.46:0/823754392 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9c00c82b0 0x7fb9c016b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.240+0000 7fb9bffff700 1 -- 172.21.15.46:0/823754392 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9c0140bc0 msgr2=0x7fb9c00c7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.240+0000 7fb9bffff700 1 --2- 172.21.15.46:0/823754392 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9c0140bc0 0x7fb9c00c7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.240+0000 7fb9bffff700 1 -- 172.21.15.46:0/823754392 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9c00c82b0 msgr2=0x7fb9c016b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.240+0000 7fb9bffff700 1 --2- 172.21.15.46:0/823754392 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9c00c82b0 0x7fb9c016b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.240+0000 7fb9bffff700 1 -- 172.21.15.46:0/823754392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c00c9450 con 0x7fb9a8003a90 2024-01-26T23:30:06.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.240+0000 7fb9c4bd2700 1 --2- 172.21.15.46:0/823754392 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9c00c82b0 0x7fb9c016b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:30:06.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.240+0000 7fb9bffff700 1 --2- 172.21.15.46:0/823754392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a8003a90 0x7fb9c0168ce0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8004730 tx=0x7fb9b001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.240+0000 7fb9a7fff700 1 -- 172.21.15.46:0/823754392 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b0022070 con 0x7fb9a8003a90 2024-01-26T23:30:06.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.240+0000 7fb9a7fff700 1 -- 172.21.15.46:0/823754392 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b0000d10 con 0x7fb9a8003a90 2024-01-26T23:30:06.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.240+0000 7fb9c6635700 1 -- 172.21.15.46:0/823754392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c00c9aa0 con 0x7fb9a8003a90 2024-01-26T23:30:06.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.240+0000 7fb9a7fff700 1 -- 172.21.15.46:0/823754392 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b000ec60 con 0x7fb9a8003a90 2024-01-26T23:30:06.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.240+0000 7fb9c6635700 1 -- 172.21.15.46:0/823754392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c00c55f0 con 0x7fb9a8003a90 2024-01-26T23:30:06.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.241+0000 7fb9c6635700 1 -- 172.21.15.46:0/823754392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c004c710 con 0x7fb9a8003a90 2024-01-26T23:30:06.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.242+0000 7fb9a7fff700 1 -- 172.21.15.46:0/823754392 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb9b0026070 con 0x7fb9a8003a90 2024-01-26T23:30:06.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.242+0000 7fb9a7fff700 1 --2- 172.21.15.46:0/823754392 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb9ac066f30 0x7fb9ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.242+0000 7fb9a7fff700 1 -- 172.21.15.46:0/823754392 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fb9b0086120 con 0x7fb9a8003a90 2024-01-26T23:30:06.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.242+0000 7fb9bf7fe700 1 --2- 172.21.15.46:0/823754392 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb9ac066f30 0x7fb9ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.243+0000 7fb9bf7fe700 1 --2- 172.21.15.46:0/823754392 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb9ac066f30 0x7fb9ac0693e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00b1030 tx=0x7fb9b4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.247+0000 7fb9a7fff700 1 -- 172.21.15.46:0/823754392 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb9b00530a0 con 0x7fb9a8003a90 2024-01-26T23:30:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.453+0000 7fb9c6635700 1 -- 172.21.15.46:0/823754392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9c00c4740 con 0x7fb9a8003a90 2024-01-26T23:30:06.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.453+0000 7fb9a7fff700 1 -- 172.21.15.46:0/823754392 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7fb9b00566c0 con 0x7fb9a8003a90 2024-01-26T23:30:06.454 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:06.455 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:49.075104+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.456+0000 7fb9c6635700 1 -- 172.21.15.46:0/823754392 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb9ac066f30 msgr2=0x7fb9ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.456+0000 7fb9c6635700 1 --2- 172.21.15.46:0/823754392 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb9ac066f30 0x7fb9ac0693e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00b1030 tx=0x7fb9b4009040 comp rx=0 tx=0).stop 2024-01-26T23:30:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.456+0000 7fb9c6635700 1 -- 172.21.15.46:0/823754392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a8003a90 msgr2=0x7fb9c0168ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.456+0000 7fb9c6635700 1 --2- 172.21.15.46:0/823754392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a8003a90 0x7fb9c0168ce0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8004730 tx=0x7fb9b001a000 comp rx=0 tx=0).stop 2024-01-26T23:30:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.456+0000 7fb9c6635700 1 -- 172.21.15.46:0/823754392 shutdown_connections 2024-01-26T23:30:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.456+0000 7fb9c6635700 1 --2- 172.21.15.46:0/823754392 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb9ac066f30 0x7fb9ac0693e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.456+0000 7fb9c6635700 1 --2- 172.21.15.46:0/823754392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a8003a90 0x7fb9c0168ce0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.456+0000 7fb9c6635700 1 --2- 172.21.15.46:0/823754392 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9c00c82b0 0x7fb9c016b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.456+0000 7fb9c6635700 1 --2- 172.21.15.46:0/823754392 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9c0140bc0 0x7fb9c00c7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.456+0000 7fb9c6635700 1 -- 172.21.15.46:0/823754392 >> 172.21.15.46:0/823754392 conn(0x7fb9c0097820 msgr2=0x7fb9c006be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.456+0000 7fb9c6635700 1 -- 172.21.15.46:0/823754392 shutdown_connections 2024-01-26T23:30:06.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.456+0000 7fb9c6635700 1 -- 172.21.15.46:0/823754392 wait complete. 2024-01-26T23:30:06.470 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-26T23:30:06.471 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-26T23:30:06.471 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-26T23:30:06.472 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-01-26T23:30:06.472 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-01-26T23:30:06.472 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-01-26T23:30:06.473 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-01-26T23:30:06.473 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-01-26T23:30:06.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.707+0000 7f38adf9a700 1 Processor -- start 2024-01-26T23:30:06.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.708+0000 7fc31bdb3700 1 Processor -- start 2024-01-26T23:30:06.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.709+0000 7f90b3a1e700 1 Processor -- start 2024-01-26T23:30:06.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.709+0000 7f90b3a1e700 1 -- start start 2024-01-26T23:30:06.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.709+0000 7fc31bdb3700 1 -- start start 2024-01-26T23:30:06.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.706+0000 7f043258f700 1 Processor -- start 2024-01-26T23:30:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.706+0000 7f043258f700 1 -- start start 2024-01-26T23:30:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.708+0000 7f043258f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f042c166520 0x7f042c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.709+0000 7f043258f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f042c0b2200 0x7f042c16a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7f043258f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f042c1668f0 0x7f042c16a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f043258f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f042c0acbe0 con 0x7f042c166520 2024-01-26T23:30:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f043258f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f042c0abfd0 con 0x7f042c1668f0 2024-01-26T23:30:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f043258f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042c0ac2b0 con 0x7f042c0b2200 2024-01-26T23:30:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f042bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f042c166520 0x7f042c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f042b7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f042c0b2200 0x7f042c16a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7f90b3a1e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f90ac06bde0 0x7f90ac153710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f90b3a1e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90ac156050 0x7f90ac156470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f90b3a1e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ac0b0860 con 0x7f90ac153340 2024-01-26T23:30:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f90b3a1e700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90ac0b0b40 con 0x7f90ac156050 2024-01-26T23:30:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f90b3a1e700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90ac157220 con 0x7f90ac06bde0 2024-01-26T23:30:06.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f042b7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f042c0b2200 0x7f042c16a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:57332/0 (socket says 172.21.15.46:57332) 2024-01-26T23:30:06.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f042b7fe700 1 -- 172.21.15.46:0/137429419 learned_addr learned my addr 172.21.15.46:0/137429419 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:06.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f0430b2c700 1 --2- 172.21.15.46:0/137429419 >> v2:172.21.15.106:3300/0 conn(0x7f042c1668f0 0x7f042c16a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f90b0fb9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f90ac06bde0 0x7f90ac153710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f90b0fb9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f90ac06bde0 0x7f90ac153710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56430/0 (socket says 172.21.15.46:56430) 2024-01-26T23:30:06.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f90b0fb9700 1 -- 172.21.15.46:0/667208686 learned_addr learned my addr 172.21.15.46:0/667208686 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:06.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f042b7fe700 1 -- 172.21.15.46:0/137429419 >> v2:172.21.15.106:3300/0 conn(0x7f042c1668f0 msgr2=0x7f042c16a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:06.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f042b7fe700 1 --2- 172.21.15.46:0/137429419 >> v2:172.21.15.106:3300/0 conn(0x7f042c1668f0 0x7f042c16a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7f042b7fe700 1 -- 172.21.15.46:0/137429419 >> v2:172.21.15.46:3300/0 conn(0x7f042c166520 msgr2=0x7f042c0b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:06.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7f042b7fe700 1 --2- 172.21.15.46:0/137429419 >> v2:172.21.15.46:3300/0 conn(0x7f042c166520 0x7f042c0b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7f042b7fe700 1 -- 172.21.15.46:0/137429419 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042c0ac590 con 0x7f042c0b2200 2024-01-26T23:30:06.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7f90b0fb9700 1 -- 172.21.15.46:0/667208686 >> v2:172.21.15.46:3300/0 conn(0x7f90ac156050 msgr2=0x7f90ac156470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:06.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7f90b0fb9700 1 --2- 172.21.15.46:0/667208686 >> v2:172.21.15.46:3300/0 conn(0x7f90ac156050 0x7f90ac156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7f90b0fb9700 1 -- 172.21.15.46:0/667208686 >> v1:172.21.15.106:6789/0 conn(0x7f90ac153340 legacy=0x7f90ac0b0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:06.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7f90b0fb9700 1 -- 172.21.15.46:0/667208686 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ac157500 con 0x7f90ac06bde0 2024-01-26T23:30:06.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7f042b7fe700 1 --2- 172.21.15.46:0/137429419 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f042c0b2200 0x7f042c16a2e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f042c0af0c0 tx=0x7f0420009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b86e8b5be1526f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7fc31bdb3700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc31415b9e0 0x7fc3140b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7fc31bdb3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3140b00b0 0x7fc314160650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7fc31bdb3700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc31415c050 con 0x7fc31415ee60 2024-01-26T23:30:06.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7fc31bdb3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3140b15b0 con 0x7fc3140b00b0 2024-01-26T23:30:06.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7fc31bdb3700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3141601c0 con 0x7fc31415b9e0 2024-01-26T23:30:06.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7fc319b4f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc31415b9e0 0x7fc3140b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7fc319b4f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc31415b9e0 0x7fc3140b1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56406/0 (socket says 172.21.15.46:56406) 2024-01-26T23:30:06.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7fc319b4f700 1 -- 172.21.15.46:0/1902573938 learned_addr learned my addr 172.21.15.46:0/1902573938 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:06.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.711+0000 7fc31a350700 1 --2- 172.21.15.46:0/1902573938 >> v2:172.21.15.46:3300/0 conn(0x7fc3140b00b0 0x7fc314160650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7fc319b4f700 1 -- 172.21.15.46:0/1902573938 >> v2:172.21.15.46:3300/0 conn(0x7fc3140b00b0 msgr2=0x7fc314160650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:06.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7fc318b4d700 1 -- 172.21.15.46:0/1902573938 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749876242 0 0) 0x7fc31415c050 con 0x7fc31415ee60 2024-01-26T23:30:06.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7fc319b4f700 1 --2- 172.21.15.46:0/1902573938 >> v2:172.21.15.46:3300/0 conn(0x7fc3140b00b0 0x7fc314160650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7fc319b4f700 1 -- 172.21.15.46:0/1902573938 >> v1:172.21.15.46:6789/0 conn(0x7fc31415ee60 legacy=0x7fc3140af9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7fc319b4f700 1 -- 172.21.15.46:0/1902573938 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3140b2460 con 0x7fc31415b9e0 2024-01-26T23:30:06.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7f042affd700 1 -- 172.21.15.46:0/137429419 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0420009e70 con 0x7f042c0b2200 2024-01-26T23:30:06.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7f042affd700 1 -- 172.21.15.46:0/137429419 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042000b860 con 0x7f042c0b2200 2024-01-26T23:30:06.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7f042affd700 1 -- 172.21.15.46:0/137429419 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042000bc00 con 0x7f042c0b2200 2024-01-26T23:30:06.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7fc319b4f700 1 --2- 172.21.15.46:0/1902573938 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc31415b9e0 0x7fc3140b1d80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc314068fc0 tx=0x7fc310009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74993f141f9537a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.713+0000 7fc318b4d700 1 -- 172.21.15.46:0/1902573938 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc310009e70 con 0x7fc31415b9e0 2024-01-26T23:30:06.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.713+0000 7fc318b4d700 1 -- 172.21.15.46:0/1902573938 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc31000b860 con 0x7fc31415b9e0 2024-01-26T23:30:06.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.713+0000 7fc318b4d700 1 -- 172.21.15.46:0/1902573938 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31000bc00 con 0x7fc31415b9e0 2024-01-26T23:30:06.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7f90b0fb9700 1 --2- 172.21.15.46:0/667208686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90ac06bde0 0x7f90ac153710 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f90ac0cc570 tx=0x7f90a8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a42a92be3d685a81 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7fc31bdb3700 1 -- 172.21.15.46:0/1902573938 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc31415b9e0 msgr2=0x7fc3140b1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7fc31bdb3700 1 --2- 172.21.15.46:0/1902573938 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc31415b9e0 0x7fc3140b1d80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc314068fc0 tx=0x7fc310009250 comp rx=0 tx=0).stop 2024-01-26T23:30:06.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7fc31bdb3700 1 -- 172.21.15.46:0/1902573938 shutdown_connections 2024-01-26T23:30:06.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7fc31bdb3700 1 --2- 172.21.15.46:0/1902573938 >> v2:172.21.15.46:3300/0 conn(0x7fc3140b00b0 0x7fc314160650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7fc31bdb3700 1 --2- 172.21.15.46:0/1902573938 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc31415b9e0 0x7fc3140b1d80 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc314068fc0 tx=0x7fc310009250 comp rx=0 tx=0).stop 2024-01-26T23:30:06.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7fc31bdb3700 1 -- 172.21.15.46:0/1902573938 >> 172.21.15.46:0/1902573938 conn(0x7fc314068400 msgr2=0x7fc3140cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7fc31bdb3700 1 -- 172.21.15.46:0/1902573938 shutdown_connections 2024-01-26T23:30:06.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.713+0000 7f90a3fff700 1 -- 172.21.15.46:0/667208686 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a800b820 con 0x7f90ac06bde0 2024-01-26T23:30:06.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.713+0000 7f90a3fff700 1 -- 172.21.15.46:0/667208686 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a800b980 con 0x7f90ac06bde0 2024-01-26T23:30:06.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.713+0000 7f90a3fff700 1 -- 172.21.15.46:0/667208686 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a800bd20 con 0x7f90ac06bde0 2024-01-26T23:30:06.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7f90b3a1e700 1 -- 172.21.15.46:0/667208686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90ac06bde0 msgr2=0x7f90ac153710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7f90b3a1e700 1 --2- 172.21.15.46:0/667208686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90ac06bde0 0x7f90ac153710 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f90ac0cc570 tx=0x7f90a8009250 comp rx=0 tx=0).stop 2024-01-26T23:30:06.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.713+0000 7f043258f700 1 -- 172.21.15.46:0/137429419 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f042c0b2200 msgr2=0x7f042c16a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.713+0000 7f043258f700 1 --2- 172.21.15.46:0/137429419 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f042c0b2200 0x7f042c16a2e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f042c0af0c0 tx=0x7f0420009250 comp rx=0 tx=0).stop 2024-01-26T23:30:06.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7f90b3a1e700 1 -- 172.21.15.46:0/667208686 shutdown_connections 2024-01-26T23:30:06.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7f90b3a1e700 1 --2- 172.21.15.46:0/667208686 >> v2:172.21.15.46:3300/0 conn(0x7f90ac156050 0x7f90ac156470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.715+0000 7f90b3a1e700 1 --2- 172.21.15.46:0/667208686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90ac06bde0 0x7f90ac153710 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.715+0000 7f90b3a1e700 1 -- 172.21.15.46:0/667208686 >> 172.21.15.46:0/667208686 conn(0x7f90ac068400 msgr2=0x7f90ac0cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.715+0000 7f043258f700 1 -- 172.21.15.46:0/137429419 shutdown_connections 2024-01-26T23:30:06.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.715+0000 7f043258f700 1 --2- 172.21.15.46:0/137429419 >> v2:172.21.15.46:3300/0 conn(0x7f042c166520 0x7f042c0b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.715+0000 7f043258f700 1 --2- 172.21.15.46:0/137429419 >> v2:172.21.15.106:3300/0 conn(0x7f042c1668f0 0x7f042c16a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.715+0000 7f043258f700 1 --2- 172.21.15.46:0/137429419 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f042c0b2200 0x7f042c16a2e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.715+0000 7f043258f700 1 -- 172.21.15.46:0/137429419 >> 172.21.15.46:0/137429419 conn(0x7f042c068400 msgr2=0x7f042c0cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.715+0000 7f043258f700 1 -- 172.21.15.46:0/137429419 shutdown_connections 2024-01-26T23:30:06.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.715+0000 7f043258f700 1 -- 172.21.15.46:0/137429419 wait complete. 2024-01-26T23:30:06.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.708+0000 7f38adf9a700 1 -- start start 2024-01-26T23:30:06.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7f38adf9a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f38a8156170 0x7f38a80b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7f38adf9a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f38a80afbd0 0x7f38a80b0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7f38adf9a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a8157d50 con 0x7f38a8156540 2024-01-26T23:30:06.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7f38adf9a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a80af920 con 0x7f38a80afbd0 2024-01-26T23:30:06.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7f38adf9a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a8157140 con 0x7f38a8156170 2024-01-26T23:30:06.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7f389effd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f38a80afbd0 0x7f38a80b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7f389effd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f38a80afbd0 0x7f38a80b0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33844/0 (socket says 172.21.15.46:33844) 2024-01-26T23:30:06.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7f389effd700 1 -- 172.21.15.46:0/3995109034 learned_addr learned my addr 172.21.15.46:0/3995109034 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:06.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7f38a7fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f38a8156540 0x7f38a8159630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44456/0 (socket says 172.21.15.46:44456) 2024-01-26T23:30:06.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7f38a77fe700 1 --2- 172.21.15.46:0/3995109034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f38a8156170 0x7f38a80b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7f389effd700 1 -- 172.21.15.46:0/3995109034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f38a8156170 msgr2=0x7f38a80b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:06.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7f389effd700 1 --2- 172.21.15.46:0/3995109034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f38a8156170 0x7f38a80b0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7f389effd700 1 -- 172.21.15.46:0/3995109034 >> v1:172.21.15.46:6789/0 conn(0x7f38a8156540 legacy=0x7f38a8159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.710+0000 7f389effd700 1 -- 172.21.15.46:0/3995109034 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a8157420 con 0x7f38a80afbd0 2024-01-26T23:30:06.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.712+0000 7f389effd700 1 --2- 172.21.15.46:0/3995109034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38a80afbd0 0x7f38a80b0670 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f38980099b0 tx=0x7f3898009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f1bed6e28944a22 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.713+0000 7f38a6ffd700 1 -- 172.21.15.46:0/3995109034 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389800b860 con 0x7f38a80afbd0 2024-01-26T23:30:06.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.713+0000 7f38a6ffd700 1 -- 172.21.15.46:0/3995109034 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389800b9c0 con 0x7f38a80afbd0 2024-01-26T23:30:06.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.713+0000 7f38a6ffd700 1 -- 172.21.15.46:0/3995109034 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389800bd60 con 0x7f38a80afbd0 2024-01-26T23:30:06.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7f38adf9a700 1 -- 172.21.15.46:0/3995109034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38a80afbd0 msgr2=0x7f38a80b0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7f38adf9a700 1 --2- 172.21.15.46:0/3995109034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38a80afbd0 0x7f38a80b0670 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f38980099b0 tx=0x7f3898009250 comp rx=0 tx=0).stop 2024-01-26T23:30:06.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7f38adf9a700 1 -- 172.21.15.46:0/3995109034 shutdown_connections 2024-01-26T23:30:06.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7f38adf9a700 1 --2- 172.21.15.46:0/3995109034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f38a8156170 0x7f38a80b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7f38adf9a700 1 --2- 172.21.15.46:0/3995109034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38a80afbd0 0x7f38a80b0670 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7f38adf9a700 1 -- 172.21.15.46:0/3995109034 >> 172.21.15.46:0/3995109034 conn(0x7f38a8068400 msgr2=0x7f38a80cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.715+0000 7f38adf9a700 1 -- 172.21.15.46:0/3995109034 shutdown_connections 2024-01-26T23:30:06.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.716+0000 7f90b3a1e700 1 -- 172.21.15.46:0/667208686 shutdown_connections 2024-01-26T23:30:06.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.714+0000 7fc31bdb3700 1 -- 172.21.15.46:0/1902573938 wait complete. 2024-01-26T23:30:06.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.716+0000 7fc31bdb3700 1 Processor -- start 2024-01-26T23:30:06.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.717+0000 7f043258f700 1 Processor -- start 2024-01-26T23:30:06.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.716+0000 7fe7d71f5700 1 Processor -- start 2024-01-26T23:30:06.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.716+0000 7fe7d71f5700 1 -- start start 2024-01-26T23:30:06.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.717+0000 7fe7d71f5700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe7d015b700 0x7fe7d015bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.717+0000 7fe7d71f5700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe7d00af9a0 0x7fe7d00afe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.717+0000 7fe7d71f5700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d00b0340 con 0x7fe7d00b1cc0 2024-01-26T23:30:06.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.717+0000 7fe7d71f5700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d00b2160 con 0x7fe7d00af9a0 2024-01-26T23:30:06.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.717+0000 7fe7d71f5700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d015feb0 con 0x7fe7d015b700 2024-01-26T23:30:06.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7fe7d5792700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe7d00af9a0 0x7fe7d00afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7fe7d5792700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe7d00af9a0 0x7fe7d00afe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33864/0 (socket says 172.21.15.46:33864) 2024-01-26T23:30:06.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7fe7d5792700 1 -- 172.21.15.46:0/1740475239 learned_addr learned my addr 172.21.15.46:0/1740475239 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:06.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.716+0000 7f90b3a1e700 1 -- 172.21.15.46:0/667208686 wait complete. 2024-01-26T23:30:06.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7f90b3a1e700 1 Processor -- start 2024-01-26T23:30:06.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.717+0000 7f043258f700 1 -- start start 2024-01-26T23:30:06.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7f043258f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f042c0b2200 0x7f042c0a6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7f043258f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f042c166520 0x7f042c0aa0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7f043258f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f042c1668f0 0x7f042c0868b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7f043258f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042c0a9cb0 con 0x7f042c0b2200 2024-01-26T23:30:06.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7f043258f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042c0a7630 con 0x7f042c166520 2024-01-26T23:30:06.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7f043258f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042c0a9350 con 0x7f042c1668f0 2024-01-26T23:30:06.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7f042b7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f042c166520 0x7f042c0aa0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7f042b7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f042c166520 0x7f042c0aa0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56456/0 (socket says 172.21.15.46:56456) 2024-01-26T23:30:06.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7f042b7fe700 1 -- 172.21.15.46:0/1335999664 learned_addr learned my addr 172.21.15.46:0/1335999664 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:06.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7f042b7fe700 1 -- 172.21.15.46:0/1335999664 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f042c1668f0 msgr2=0x7f042c0868b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7f042b7fe700 1 --2- 172.21.15.46:0/1335999664 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f042c1668f0 0x7f042c0868b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7f042b7fe700 1 -- 172.21.15.46:0/1335999664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f042c0b2200 msgr2=0x7f042c0a6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7f042b7fe700 1 --2- 172.21.15.46:0/1335999664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f042c0b2200 0x7f042c0a6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7f042b7fe700 1 -- 172.21.15.46:0/1335999664 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04200094f0 con 0x7f042c166520 2024-01-26T23:30:06.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7fe7d5792700 1 -- 172.21.15.46:0/1740475239 >> v2:172.21.15.106:3300/0 conn(0x7fe7d015b700 msgr2=0x7fe7d015bad0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-26T23:30:06.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7fe7d5792700 1 --2- 172.21.15.46:0/1740475239 >> v2:172.21.15.106:3300/0 conn(0x7fe7d015b700 0x7fe7d015bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7fe7d5792700 1 -- 172.21.15.46:0/1740475239 >> v1:172.21.15.46:6789/0 conn(0x7fe7d00b1cc0 legacy=0x7fe7d015ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-26T23:30:06.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7fe7d5792700 1 -- 172.21.15.46:0/1740475239 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d015f5c0 con 0x7fe7d00af9a0 2024-01-26T23:30:06.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7f042b7fe700 1 --2- 172.21.15.46:0/1335999664 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f042c166520 0x7f042c0aa0d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0420009290 tx=0x7f0420018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7f305fe80700 1 Processor -- start 2024-01-26T23:30:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7f90b3a1e700 1 -- start start 2024-01-26T23:30:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7f90b3a1e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90ac06bde0 0x7f90ac0c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7f90b3a1e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90ac153340 0x7f90ac0cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7f90b3a1e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac156050 0x7f90ac0c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7f90b3a1e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac0c7a40 con 0x7f90ac156050 2024-01-26T23:30:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7f90b3a1e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac0c7d20 con 0x7f90ac153340 2024-01-26T23:30:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7f90b3a1e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac0c8000 con 0x7f90ac06bde0 2024-01-26T23:30:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7fe7d5792700 1 --2- 172.21.15.46:0/1740475239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7d00af9a0 0x7fe7d00afe00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe7d006bca0 tx=0x7fe7c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55043893bc64d8b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7f38adf9a700 1 -- 172.21.15.46:0/3995109034 wait complete. 2024-01-26T23:30:06.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7f90b1fbb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac156050 0x7f90ac0c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7f90b1fbb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac156050 0x7f90ac0c46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33874/0 (socket says 172.21.15.46:33874) 2024-01-26T23:30:06.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7f90b1fbb700 1 -- 172.21.15.46:0/2863638460 learned_addr learned my addr 172.21.15.46:0/2863638460 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:06.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7f38adf9a700 1 Processor -- start 2024-01-26T23:30:06.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7f305fe80700 1 -- start start 2024-01-26T23:30:06.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.717+0000 7fc31bdb3700 1 -- start start 2024-01-26T23:30:06.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7fc31bdb3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3140b00b0 0x7fc314176350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7fc31bdb3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc31415ee60 0x7fc31415cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7f0413fff700 1 -- 172.21.15.46:0/1335999664 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042000bbe0 con 0x7f042c166520 2024-01-26T23:30:06.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7f0413fff700 1 -- 172.21.15.46:0/1335999664 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0420021450 con 0x7f042c166520 2024-01-26T23:30:06.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7f0413fff700 1 -- 172.21.15.46:0/1335999664 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0420021810 con 0x7f042c166520 2024-01-26T23:30:06.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7fc31bdb3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3141502e0 0x7fc31415d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7fc31bdb3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc314159630 con 0x7fc3140b00b0 2024-01-26T23:30:06.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7fc31bdb3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc314158d00 con 0x7fc3141502e0 2024-01-26T23:30:06.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7fc31bdb3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc314158fe0 con 0x7fc31415ee60 2024-01-26T23:30:06.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7fc31934e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc31415ee60 0x7fc31415cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7fc31934e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc31415ee60 0x7fc31415cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:57350/0 (socket says 172.21.15.46:57350) 2024-01-26T23:30:06.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7fc31934e700 1 -- 172.21.15.46:0/3959720760 learned_addr learned my addr 172.21.15.46:0/3959720760 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:06.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.718+0000 7fc31a350700 1 --2- 172.21.15.46:0/3959720760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3141502e0 0x7fc31415d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.719+0000 7fc31934e700 1 -- 172.21.15.46:0/3959720760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3141502e0 msgr2=0x7fc31415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7fc31934e700 1 --2- 172.21.15.46:0/3959720760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3141502e0 0x7fc31415d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7fc31934e700 1 -- 172.21.15.46:0/3959720760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3140b00b0 msgr2=0x7fc314176350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:30:06.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7fc31934e700 1 --2- 172.21.15.46:0/3959720760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3140b00b0 0x7fc314176350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7fc31934e700 1 -- 172.21.15.46:0/3959720760 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3100094f0 con 0x7fc31415ee60 2024-01-26T23:30:06.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7fc31a350700 1 --2- 172.21.15.46:0/3959720760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3141502e0 0x7fc31415d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7f90b0fb9700 1 --2- 172.21.15.46:0/2863638460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90ac153340 0x7f90ac0cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7f90b17ba700 1 --2- 172.21.15.46:0/2863638460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90ac06bde0 0x7f90ac0c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7f90b1fbb700 1 -- 172.21.15.46:0/2863638460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90ac06bde0 msgr2=0x7f90ac0c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7f90b1fbb700 1 --2- 172.21.15.46:0/2863638460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90ac06bde0 0x7f90ac0c70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7f90b1fbb700 1 -- 172.21.15.46:0/2863638460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90ac153340 msgr2=0x7f90ac0cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7f90b1fbb700 1 --2- 172.21.15.46:0/2863638460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90ac153340 0x7f90ac0cacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7f90b1fbb700 1 -- 172.21.15.46:0/2863638460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a80094f0 con 0x7f90ac156050 2024-01-26T23:30:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.721+0000 7f90b1fbb700 1 --2- 172.21.15.46:0/2863638460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac156050 0x7f90ac0c46a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f90a400bc30 tx=0x7f90a4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.721+0000 7f38adf9a700 1 -- start start 2024-01-26T23:30:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.721+0000 7fe7cf7fe700 1 -- 172.21.15.46:0/1740475239 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c000b7e0 con 0x7fe7d00af9a0 2024-01-26T23:30:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.721+0000 7fe7cf7fe700 1 -- 172.21.15.46:0/1740475239 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c000b940 con 0x7fe7d00af9a0 2024-01-26T23:30:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.721+0000 7fe7cf7fe700 1 -- 172.21.15.46:0/1740475239 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c000bce0 con 0x7fe7d00af9a0 2024-01-26T23:30:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7f043258f700 1 -- 172.21.15.46:0/1335999664 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f042c0a95f0 con 0x7f042c166520 2024-01-26T23:30:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.720+0000 7f043258f700 1 -- 172.21.15.46:0/1335999664 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f042c17fd40 con 0x7f042c166520 2024-01-26T23:30:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.721+0000 7f305fe80700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f305806c3c0 0x7f30581617a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.721+0000 7f305fe80700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f30580c1ab0 0x7f3058161ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.721+0000 7f305fe80700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3058162700 con 0x7f30580c16e0 2024-01-26T23:30:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.721+0000 7f305fe80700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30580b0e40 con 0x7f305806c3c0 2024-01-26T23:30:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.721+0000 7f305fe80700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3058162220 con 0x7f30580c1ab0 2024-01-26T23:30:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.721+0000 7fc31934e700 1 --2- 172.21.15.46:0/3959720760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc31415ee60 0x7fc31415cd80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc30400c5b0 tx=0x7fc304007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.721+0000 7f64c180f700 1 Processor -- start 2024-01-26T23:30:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.721+0000 7f043258f700 1 -- 172.21.15.46:0/1335999664 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f042c04c710 con 0x7f042c166520 2024-01-26T23:30:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.721+0000 7f90a17fa700 1 -- 172.21.15.46:0/2863638460 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a40092e0 con 0x7f90ac156050 2024-01-26T23:30:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7f90a17fa700 1 -- 172.21.15.46:0/2863638460 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a400c7a0 con 0x7f90ac156050 2024-01-26T23:30:06.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7fc309ffb700 1 -- 172.21.15.46:0/3959720760 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc304011590 con 0x7fc31415ee60 2024-01-26T23:30:06.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7fc309ffb700 1 -- 172.21.15.46:0/3959720760 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc304011710 con 0x7fc31415ee60 2024-01-26T23:30:06.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7fc309ffb700 1 -- 172.21.15.46:0/3959720760 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc304011ad0 con 0x7fc31415ee60 2024-01-26T23:30:06.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7f90a17fa700 1 -- 172.21.15.46:0/2863638460 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a400cb80 con 0x7f90ac156050 2024-01-26T23:30:06.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7fc31bdb3700 1 -- 172.21.15.46:0/3959720760 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3141592c0 con 0x7fc31415ee60 2024-01-26T23:30:06.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7fc31bdb3700 1 -- 172.21.15.46:0/3959720760 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3141589b0 con 0x7fc31415ee60 2024-01-26T23:30:06.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7f90b3a1e700 1 -- 172.21.15.46:0/2863638460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ac0c82e0 con 0x7f90ac156050 2024-01-26T23:30:06.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7f90b3a1e700 1 -- 172.21.15.46:0/2863638460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90ac0cb210 con 0x7f90ac156050 2024-01-26T23:30:06.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7f305e41d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f30580c1ab0 0x7f3058161ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7f305d41b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f305806c3c0 0x7f30581617a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7f305d41b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f305806c3c0 0x7f30581617a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33898/0 (socket says 172.21.15.46:33898) 2024-01-26T23:30:06.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7f305e41d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f30580c1ab0 0x7f3058161ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56468/0 (socket says 172.21.15.46:56468) 2024-01-26T23:30:06.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7f305d41b700 1 -- 172.21.15.46:0/2002896051 learned_addr learned my addr 172.21.15.46:0/2002896051 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.723+0000 7f305d41b700 1 -- 172.21.15.46:0/2002896051 >> v2:172.21.15.106:3300/0 conn(0x7f30580c1ab0 msgr2=0x7f3058161ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.723+0000 7f305d41b700 1 --2- 172.21.15.46:0/2002896051 >> v2:172.21.15.106:3300/0 conn(0x7f30580c1ab0 0x7f3058161ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.723+0000 7f305d41b700 1 -- 172.21.15.46:0/2002896051 >> v1:172.21.15.46:6789/0 conn(0x7f30580c16e0 legacy=0x7f305806bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.723+0000 7f305d41b700 1 -- 172.21.15.46:0/2002896051 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30580691c0 con 0x7f305806c3c0 2024-01-26T23:30:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7f64c180f700 1 -- start start 2024-01-26T23:30:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.723+0000 7f64c180f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f64bc156170 0x7f64bc0b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.723+0000 7f64c180f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f64bc0afbd0 0x7f64bc0b0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.723+0000 7f64c180f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64bc156540 0x7f64bc1569c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.723+0000 7f64c180f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64bc156f00 con 0x7f64bc156170 2024-01-26T23:30:06.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.723+0000 7f64c180f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64bc157d50 con 0x7f64bc0afbd0 2024-01-26T23:30:06.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.723+0000 7f64c180f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc0af920 con 0x7f64bc156540 2024-01-26T23:30:06.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.723+0000 7fe7d71f5700 1 -- 172.21.15.46:0/1740475239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7d00af9a0 msgr2=0x7fe7d00afe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.723+0000 7fe7d71f5700 1 --2- 172.21.15.46:0/1740475239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7d00af9a0 0x7fe7d00afe00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe7d006bca0 tx=0x7fe7c0009250 comp rx=0 tx=0).stop 2024-01-26T23:30:06.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.723+0000 7f305d41b700 1 --2- 172.21.15.46:0/2002896051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f305806c3c0 0x7f30581617a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f30580b4610 tx=0x7f3048009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa3a6eac3b8d2a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.723+0000 7fc31bdb3700 1 -- 172.21.15.46:0/3959720760 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc2f4000fd0 con 0x7fc31415ee60 2024-01-26T23:30:06.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7f38adf9a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38a80afbd0 0x7f38a80cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7f38adf9a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38a8156170 0x7f38a8086560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7f38adf9a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f38a8156540 0x7f38a8141c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7f38adf9a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80cac50 con 0x7f38a80afbd0 2024-01-26T23:30:06.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7f38adf9a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80ca320 con 0x7f38a8156170 2024-01-26T23:30:06.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.722+0000 7f38adf9a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80ca600 con 0x7f38a8156540 2024-01-26T23:30:06.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.723+0000 7f389effd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38a8156170 0x7f38a8086560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.723+0000 7f389effd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38a8156170 0x7f38a8086560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56470/0 (socket says 172.21.15.46:56470) 2024-01-26T23:30:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.723+0000 7f389effd700 1 -- 172.21.15.46:0/3806616831 learned_addr learned my addr 172.21.15.46:0/3806616831 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f389effd700 1 -- 172.21.15.46:0/3806616831 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f38a8156540 msgr2=0x7f38a8141c60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-26T23:30:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f389effd700 1 --2- 172.21.15.46:0/3806616831 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f38a8156540 0x7f38a8141c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f389effd700 1 -- 172.21.15.46:0/3806616831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38a80afbd0 msgr2=0x7f38a80cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f389effd700 1 --2- 172.21.15.46:0/3806616831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38a80afbd0 0x7f38a80cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f389effd700 1 -- 172.21.15.46:0/3806616831 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38980094f0 con 0x7f38a8156170 2024-01-26T23:30:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f90b3a1e700 1 -- 172.21.15.46:0/2863638460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f908c000fd0 con 0x7f90ac156050 2024-01-26T23:30:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f389effd700 1 --2- 172.21.15.46:0/3806616831 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38a8156170 0x7f38a8086560 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3898009290 tx=0x7f3898017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f64ba7fc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f64bc0afbd0 0x7f64bc0b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f64ba7fc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f64bc0afbd0 0x7f64bc0b0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56476/0 (socket says 172.21.15.46:56476) 2024-01-26T23:30:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f64ba7fc700 1 -- 172.21.15.46:0/1120688179 learned_addr learned my addr 172.21.15.46:0/1120688179 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f64bb7fe700 1 --2- 172.21.15.46:0/1120688179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64bc156540 0x7f64bc1569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f305cc1a700 1 -- 172.21.15.46:0/2002896051 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304800b820 con 0x7f305806c3c0 2024-01-26T23:30:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f305cc1a700 1 -- 172.21.15.46:0/2002896051 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304800b980 con 0x7f305806c3c0 2024-01-26T23:30:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f305cc1a700 1 -- 172.21.15.46:0/2002896051 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304800bd40 con 0x7f305806c3c0 2024-01-26T23:30:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f389ffff700 1 -- 172.21.15.46:0/3806616831 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3898024440 con 0x7f38a8156170 2024-01-26T23:30:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f389ffff700 1 -- 172.21.15.46:0/3806616831 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38980245a0 con 0x7f38a8156170 2024-01-26T23:30:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f389ffff700 1 -- 172.21.15.46:0/3806616831 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3898024960 con 0x7f38a8156170 2024-01-26T23:30:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7fe7d71f5700 1 -- 172.21.15.46:0/1740475239 shutdown_connections 2024-01-26T23:30:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7fe7d71f5700 1 --2- 172.21.15.46:0/1740475239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7d00af9a0 0x7fe7d00afe00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7fe7d71f5700 1 --2- 172.21.15.46:0/1740475239 >> v2:172.21.15.106:3300/0 conn(0x7fe7d015b700 0x7fe7d015bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7fe7d71f5700 1 -- 172.21.15.46:0/1740475239 >> 172.21.15.46:0/1740475239 conn(0x7fe7d0068400 msgr2=0x7fe7d00cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.725+0000 7fe7d71f5700 1 -- 172.21.15.46:0/1740475239 shutdown_connections 2024-01-26T23:30:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.726+0000 7fe7d71f5700 1 -- 172.21.15.46:0/1740475239 wait complete. 2024-01-26T23:30:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.728+0000 7f90a17fa700 1 -- 172.21.15.46:0/2863638460 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f90a4012070 con 0x7f90ac156050 2024-01-26T23:30:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.728+0000 7f90a17fa700 1 --2- 172.21.15.46:0/2863638460 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9090067000 0x7f90900694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.728+0000 7f90a17fa700 1 -- 172.21.15.46:0/2863638460 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f90a4084c60 con 0x7f90ac156050 2024-01-26T23:30:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.728+0000 7f90b17ba700 1 --2- 172.21.15.46:0/2863638460 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9090067000 0x7f90900694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.728+0000 7f90a17fa700 1 --2- 172.21.15.46:0/2863638460 >> [v2:172.21.15.46:6800/43676,v1:172.21.15.46:6801/43676] conn(0x7f9090070630 0x7f9090072a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.728+0000 7f90a17fa700 1 -- 172.21.15.46:0/2863638460 --> [v2:172.21.15.46:6800/43676,v1:172.21.15.46:6801/43676] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9090073130 con 0x7f9090070630 2024-01-26T23:30:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.728+0000 7f90a17fa700 1 -- 172.21.15.46:0/2863638460 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90a4085000 con 0x7f90ac156050 2024-01-26T23:30:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.729+0000 7f90b17ba700 1 --2- 172.21.15.46:0/2863638460 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9090067000 0x7f90900694b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f909c0027e0 tx=0x7f909c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f0413fff700 1 -- 172.21.15.46:0/1335999664 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f04200113d0 con 0x7f042c166520 2024-01-26T23:30:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f0413fff700 1 --2- 172.21.15.46:0/1335999664 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0418067080 0x7f0418069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.725+0000 7f0413fff700 1 -- 172.21.15.46:0/1335999664 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f0420086420 con 0x7f042c166520 2024-01-26T23:30:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.725+0000 7f0413fff700 1 --2- 172.21.15.46:0/1335999664 >> [v2:172.21.15.46:6808/43680,v1:172.21.15.46:6809/43680] conn(0x7f04180706b0 0x7f0418072ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.725+0000 7f0413fff700 1 -- 172.21.15.46:0/1335999664 --> [v2:172.21.15.46:6808/43680,v1:172.21.15.46:6809/43680] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04180731b0 con 0x7f04180706b0 2024-01-26T23:30:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.725+0000 7f0413fff700 1 -- 172.21.15.46:0/1335999664 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0420086750 con 0x7f042c166520 2024-01-26T23:30:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.725+0000 7f0430b2c700 1 --2- 172.21.15.46:0/1335999664 >> [v2:172.21.15.46:6808/43680,v1:172.21.15.46:6809/43680] conn(0x7f04180706b0 0x7f0418072ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.725+0000 7f042bfff700 1 --2- 172.21.15.46:0/1335999664 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0418067080 0x7f0418069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.726+0000 7f042bfff700 1 --2- 172.21.15.46:0/1335999664 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0418067080 0x7f0418069530 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f042c165ae0 tx=0x7f041c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.726+0000 7f0430b2c700 1 --2- 172.21.15.46:0/1335999664 >> [v2:172.21.15.46:6808/43680,v1:172.21.15.46:6809/43680] conn(0x7f04180706b0 0x7f0418072ae0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.727+0000 7f0413fff700 1 -- 172.21.15.46:0/1335999664 <== osd.2 v2:172.21.15.46:6808/43680 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f04180731b0 con 0x7f04180706b0 2024-01-26T23:30:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.725+0000 7f305fe80700 1 -- 172.21.15.46:0/2002896051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f305806c3c0 msgr2=0x7f30581617a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.725+0000 7f305fe80700 1 --2- 172.21.15.46:0/2002896051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f305806c3c0 0x7f30581617a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f30580b4610 tx=0x7f3048009250 comp rx=0 tx=0).stop 2024-01-26T23:30:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.726+0000 7f305fe80700 1 -- 172.21.15.46:0/2002896051 shutdown_connections 2024-01-26T23:30:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.726+0000 7f305fe80700 1 --2- 172.21.15.46:0/2002896051 >> v2:172.21.15.106:3300/0 conn(0x7f30580c1ab0 0x7f3058161ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.726+0000 7f305fe80700 1 --2- 172.21.15.46:0/2002896051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f305806c3c0 0x7f30581617a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.726+0000 7f305fe80700 1 -- 172.21.15.46:0/2002896051 >> 172.21.15.46:0/2002896051 conn(0x7f30580ce530 msgr2=0x7f30580ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.726+0000 7f305fe80700 1 -- 172.21.15.46:0/2002896051 shutdown_connections 2024-01-26T23:30:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.726+0000 7f305fe80700 1 -- 172.21.15.46:0/2002896051 wait complete. 2024-01-26T23:30:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.728+0000 7f305fe80700 1 Processor -- start 2024-01-26T23:30:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.728+0000 7f305fe80700 1 -- start start 2024-01-26T23:30:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7fc309ffb700 1 -- 172.21.15.46:0/3959720760 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc30401b3b0 con 0x7fc31415ee60 2024-01-26T23:30:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.725+0000 7fc309ffb700 1 --2- 172.21.15.46:0/3959720760 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc2f8067080 0x7fc2f8069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.725+0000 7fc309ffb700 1 -- 172.21.15.46:0/3959720760 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fc304083ca0 con 0x7fc31415ee60 2024-01-26T23:30:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.725+0000 7fc309ffb700 1 --2- 172.21.15.46:0/3959720760 >> [v2:172.21.15.106:6808/44050,v1:172.21.15.106:6809/44050] conn(0x7fc2f80706b0 0x7fc2f8072ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.725+0000 7fc309ffb700 1 -- 172.21.15.46:0/3959720760 --> [v2:172.21.15.106:6808/44050,v1:172.21.15.106:6809/44050] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc2f80731b0 con 0x7fc2f80706b0 2024-01-26T23:30:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.725+0000 7fc309ffb700 1 -- 172.21.15.46:0/3959720760 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc304086240 con 0x7fc31415ee60 2024-01-26T23:30:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.725+0000 7fc319b4f700 1 --2- 172.21.15.46:0/3959720760 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc2f8067080 0x7fc2f8069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f38adf9a700 1 -- 172.21.15.46:0/3806616831 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a80ca8e0 con 0x7f38a8156170 2024-01-26T23:30:06.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f38adf9a700 1 -- 172.21.15.46:0/3806616831 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a80cb620 con 0x7f38a8156170 2024-01-26T23:30:06.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.725+0000 7f38adf9a700 1 -- 172.21.15.46:0/3806616831 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f38a80ca490 con 0x7f38a8156170 2024-01-26T23:30:06.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.726+0000 7f389ffff700 1 -- 172.21.15.46:0/3806616831 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f389801a420 con 0x7f38a8156170 2024-01-26T23:30:06.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.727+0000 7f389ffff700 1 --2- 172.21.15.46:0/3806616831 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3888067080 0x7f3888069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.727+0000 7f389ffff700 1 -- 172.21.15.46:0/3806616831 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f3898086dc0 con 0x7f38a8156170 2024-01-26T23:30:06.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.727+0000 7f389ffff700 1 --2- 172.21.15.46:0/3806616831 >> [v2:172.21.15.106:6816/44080,v1:172.21.15.106:6817/44080] conn(0x7f38880706b0 0x7f3888072ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.727+0000 7f389ffff700 1 -- 172.21.15.46:0/3806616831 --> [v2:172.21.15.106:6816/44080,v1:172.21.15.106:6817/44080] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f38880731b0 con 0x7f38880706b0 2024-01-26T23:30:06.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.727+0000 7f389ffff700 1 -- 172.21.15.46:0/3806616831 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38980870f0 con 0x7f38a8156170 2024-01-26T23:30:06.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.727+0000 7f38a77fe700 1 --2- 172.21.15.46:0/3806616831 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3888067080 0x7f3888069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.727+0000 7f38a7fff700 1 --2- 172.21.15.46:0/3806616831 >> [v2:172.21.15.106:6816/44080,v1:172.21.15.106:6817/44080] conn(0x7f38880706b0 0x7f3888072ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.728+0000 7f38a77fe700 1 --2- 172.21.15.46:0/3806616831 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3888067080 0x7f3888069530 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f38a80c5da0 tx=0x7f3890008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.728+0000 7f38a7fff700 1 --2- 172.21.15.46:0/3806616831 >> [v2:172.21.15.106:6816/44080,v1:172.21.15.106:6817/44080] conn(0x7f38880706b0 0x7f3888072ae0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.729+0000 7f389ffff700 1 -- 172.21.15.46:0/3806616831 <== osd.6 v2:172.21.15.106:6816/44080 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f38880731b0 con 0x7f38880706b0 2024-01-26T23:30:06.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f64ba7fc700 1 -- 172.21.15.46:0/1120688179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64bc156540 msgr2=0x7f64bc1569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:06.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f64ba7fc700 1 --2- 172.21.15.46:0/1120688179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64bc156540 0x7f64bc1569c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f64ba7fc700 1 -- 172.21.15.46:0/1120688179 >> v2:172.21.15.46:3300/0 conn(0x7f64bc156170 msgr2=0x7f64bc0b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:06.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f64ba7fc700 1 --2- 172.21.15.46:0/1120688179 >> v2:172.21.15.46:3300/0 conn(0x7f64bc156170 0x7f64bc0b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.724+0000 7f64ba7fc700 1 -- 172.21.15.46:0/1120688179 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64bc157140 con 0x7f64bc0afbd0 2024-01-26T23:30:06.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.725+0000 7f64ba7fc700 1 --2- 172.21.15.46:0/1120688179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64bc0afbd0 0x7f64bc0b0670 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f64bc0cd170 tx=0x7f64b00082d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d60bd344ec8accfe server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.726+0000 7f64b9ffb700 1 -- 172.21.15.46:0/1120688179 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b0010070 con 0x7f64bc0afbd0 2024-01-26T23:30:06.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.726+0000 7f64b9ffb700 1 -- 172.21.15.46:0/1120688179 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b000db00 con 0x7f64bc0afbd0 2024-01-26T23:30:06.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.726+0000 7f64b9ffb700 1 -- 172.21.15.46:0/1120688179 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b000dea0 con 0x7f64bc0afbd0 2024-01-26T23:30:06.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.727+0000 7f64c180f700 1 -- 172.21.15.46:0/1120688179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64bc0afbd0 msgr2=0x7f64bc0b0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.727+0000 7f64c180f700 1 --2- 172.21.15.46:0/1120688179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64bc0afbd0 0x7f64bc0b0670 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f64bc0cd170 tx=0x7f64b00082d0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.727+0000 7f64c180f700 1 -- 172.21.15.46:0/1120688179 shutdown_connections 2024-01-26T23:30:06.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.727+0000 7f64c180f700 1 --2- 172.21.15.46:0/1120688179 >> v2:172.21.15.46:3300/0 conn(0x7f64bc156170 0x7f64bc0b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.727+0000 7f64c180f700 1 --2- 172.21.15.46:0/1120688179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64bc156540 0x7f64bc1569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.727+0000 7f64c180f700 1 --2- 172.21.15.46:0/1120688179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64bc0afbd0 0x7f64bc0b0670 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.727+0000 7f64c180f700 1 -- 172.21.15.46:0/1120688179 >> 172.21.15.46:0/1120688179 conn(0x7f64bc068400 msgr2=0x7f64bc0cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.727+0000 7f64c180f700 1 -- 172.21.15.46:0/1120688179 shutdown_connections 2024-01-26T23:30:06.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.727+0000 7f64c180f700 1 -- 172.21.15.46:0/1120688179 wait complete. 2024-01-26T23:30:06.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.729+0000 7f64c180f700 1 Processor -- start 2024-01-26T23:30:06.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.728+0000 7fe7d71f5700 1 Processor -- start 2024-01-26T23:30:06.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.730+0000 7f64c180f700 1 -- start start 2024-01-26T23:30:06.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.730+0000 7f90b0fb9700 1 --2- 172.21.15.46:0/2863638460 >> [v2:172.21.15.46:6800/43676,v1:172.21.15.46:6801/43676] conn(0x7f9090070630 0x7f9090072a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.730+0000 7f305fe80700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f305806c3c0 0x7f30581579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.730+0000 7f305fe80700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30580c16e0 0x7f305816dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.730+0000 7f305fe80700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f30580c1ab0 0x7f30580c1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.730+0000 7f305fe80700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305816e620 con 0x7f305806c3c0 2024-01-26T23:30:06.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.730+0000 7f305fe80700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305816e900 con 0x7f30580c16e0 2024-01-26T23:30:06.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.730+0000 7f305fe80700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f305816ebe0 con 0x7f30580c1ab0 2024-01-26T23:30:06.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.730+0000 7f305e41d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f30580c1ab0 0x7f30580c1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.730+0000 7f305e41d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f30580c1ab0 0x7f30580c1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:57410/0 (socket says 172.21.15.46:57410) 2024-01-26T23:30:06.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.730+0000 7f305e41d700 1 -- 172.21.15.46:0/2969887179 learned_addr learned my addr 172.21.15.46:0/2969887179 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:06.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.730+0000 7f305d41b700 1 --2- 172.21.15.46:0/2969887179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30580c16e0 0x7f305816dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.731+0000 7f90b0fb9700 1 --2- 172.21.15.46:0/2863638460 >> [v2:172.21.15.46:6800/43676,v1:172.21.15.46:6801/43676] conn(0x7f9090070630 0x7f9090072a60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.731+0000 7f64c180f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64bc0afbd0 0x7f64bc14d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.731+0000 7f64c180f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64bc156170 0x7f64bc0cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.731+0000 7f64c180f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64bc156540 0x7f64bc0cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.731+0000 7f64c180f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc14d710 con 0x7f64bc0afbd0 2024-01-26T23:30:06.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.731+0000 7f64c180f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc16af00 con 0x7f64bc156170 2024-01-26T23:30:06.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.731+0000 7f64c180f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc16d6e0 con 0x7f64bc156540 2024-01-26T23:30:06.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.730+0000 7fe7d71f5700 1 -- start start 2024-01-26T23:30:06.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.730+0000 7fe7d71f5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7d00af9a0 0x7fe7d01442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.730+0000 7fe7d71f5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7d00b1cc0 0x7fe7d0159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.731+0000 7fe7d71f5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7d015b700 0x7fe7d015a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.731+0000 7fe7d71f5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d01603e0 con 0x7fe7d015b700 2024-01-26T23:30:06.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.731+0000 7fe7d71f5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d0160680 con 0x7fe7d00b1cc0 2024-01-26T23:30:06.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.731+0000 7fe7d71f5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d0160960 con 0x7fe7d00af9a0 2024-01-26T23:30:06.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.732+0000 7f90a17fa700 1 -- 172.21.15.46:0/2863638460 <== osd.1 v2:172.21.15.46:6800/43676 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f9090073130 con 0x7f9090070630 2024-01-26T23:30:06.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.732+0000 7fc319b4f700 1 --2- 172.21.15.46:0/3959720760 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc2f8067080 0x7fc2f8069530 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc314068fc0 tx=0x7fc310017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.732+0000 7f64ba7fc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64bc156170 0x7f64bc0cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.732+0000 7f64ba7fc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64bc156170 0x7f64bc0cb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56506/0 (socket says 172.21.15.46:56506) 2024-01-26T23:30:06.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.732+0000 7f64ba7fc700 1 -- 172.21.15.46:0/3759365295 learned_addr learned my addr 172.21.15.46:0/3759365295 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:06.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.732+0000 7f64bb7fe700 1 --2- 172.21.15.46:0/3759365295 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64bc156540 0x7f64bc0cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.732+0000 7f305d41b700 1 -- 172.21.15.46:0/2969887179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f30580c1ab0 msgr2=0x7f30580c1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.732+0000 7f305d41b700 1 --2- 172.21.15.46:0/2969887179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f30580c1ab0 0x7f30580c1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.732+0000 7f305d41b700 1 -- 172.21.15.46:0/2969887179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f305806c3c0 msgr2=0x7f30581579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.732+0000 7f305d41b700 1 --2- 172.21.15.46:0/2969887179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f305806c3c0 0x7f30581579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.732+0000 7f305d41b700 1 -- 172.21.15.46:0/2969887179 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30480094f0 con 0x7f30580c16e0 2024-01-26T23:30:06.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.732+0000 7f305d41b700 1 --2- 172.21.15.46:0/2969887179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30580c16e0 0x7f305816dd10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f305806f980 tx=0x7f3048016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.733+0000 7fe7d4f91700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7d00af9a0 0x7fe7d01442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.733+0000 7fe7d4f91700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7d00af9a0 0x7fe7d01442b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:57426/0 (socket says 172.21.15.46:57426) 2024-01-26T23:30:06.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.733+0000 7fe7d4f91700 1 -- 172.21.15.46:0/2584368859 learned_addr learned my addr 172.21.15.46:0/2584368859 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:06.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.733+0000 7fe7d4f91700 1 -- 172.21.15.46:0/2584368859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7d00b1cc0 msgr2=0x7fe7d0159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.733+0000 7f304dffb700 1 -- 172.21.15.46:0/2969887179 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3048011b90 con 0x7f30580c16e0 2024-01-26T23:30:06.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.733+0000 7f304dffb700 1 -- 172.21.15.46:0/2969887179 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3048020070 con 0x7f30580c16e0 2024-01-26T23:30:06.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.733+0000 7f305fe80700 1 -- 172.21.15.46:0/2969887179 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30580b28b0 con 0x7f30580c16e0 2024-01-26T23:30:06.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.733+0000 7f304dffb700 1 -- 172.21.15.46:0/2969887179 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304801b450 con 0x7f30580c16e0 2024-01-26T23:30:06.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.733+0000 7f305fe80700 1 -- 172.21.15.46:0/2969887179 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f305816e250 con 0x7f30580c16e0 2024-01-26T23:30:06.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.733+0000 7fe7d4f91700 1 --2- 172.21.15.46:0/2584368859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7d00b1cc0 0x7fe7d0159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.733+0000 7fe7d4f91700 1 -- 172.21.15.46:0/2584368859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7d015b700 msgr2=0x7fe7d015a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.733+0000 7fe7d4f91700 1 --2- 172.21.15.46:0/2584368859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7d015b700 0x7fe7d015a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.733+0000 7fe7d4f91700 1 -- 172.21.15.46:0/2584368859 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c00094f0 con 0x7fe7d00af9a0 2024-01-26T23:30:06.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.733+0000 7fd58b684700 1 Processor -- start 2024-01-26T23:30:06.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.733+0000 7fe7d4f91700 1 --2- 172.21.15.46:0/2584368859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7d00af9a0 0x7fe7d01442b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe7c400b5e0 tx=0x7fe7c40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.734+0000 7fe7ccff9700 1 -- 172.21.15.46:0/2584368859 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c400f510 con 0x7fe7d00af9a0 2024-01-26T23:30:06.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.734+0000 7fe7ccff9700 1 -- 172.21.15.46:0/2584368859 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c400f690 con 0x7fe7d00af9a0 2024-01-26T23:30:06.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.734+0000 7fe7ccff9700 1 -- 172.21.15.46:0/2584368859 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c400fa50 con 0x7fe7d00af9a0 2024-01-26T23:30:06.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.734+0000 7fe7d71f5700 1 -- 172.21.15.46:0/2584368859 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d0160c40 con 0x7fe7d00af9a0 2024-01-26T23:30:06.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.734+0000 7fe7d71f5700 1 -- 172.21.15.46:0/2584368859 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d01447f0 con 0x7fe7d00af9a0 2024-01-26T23:30:06.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.735+0000 7fe7d71f5700 1 -- 172.21.15.46:0/2584368859 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe7d004c710 con 0x7fe7d00af9a0 2024-01-26T23:30:06.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.734+0000 7fd58b684700 1 -- start start 2024-01-26T23:30:06.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.735+0000 7fd58b684700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd5840b08c0 0x7fd584068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.735+0000 7fd58b684700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5840cb4c0 0x7fd5840c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.735+0000 7fd58b684700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5840c98c0 con 0x7fd5840cd9d0 2024-01-26T23:30:06.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.735+0000 7fd58b684700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5840c8cb0 con 0x7fd5840b08c0 2024-01-26T23:30:06.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.735+0000 7fd58b684700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5840c8f90 con 0x7fd5840cb4c0 2024-01-26T23:30:06.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.736+0000 7fd588c1f700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fd5840cd9d0 0x7fd5840cadb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:51734/0 (socket says 172.21.15.46:51734) 2024-01-26T23:30:06.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.736+0000 7fd588c1f700 1 -- 172.21.15.46:0/1284591513 learned_addr learned my addr 172.21.15.46:0/1284591513 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:06.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.736+0000 7fc31a350700 1 --2- 172.21.15.46:0/3959720760 >> [v2:172.21.15.106:6808/44050,v1:172.21.15.106:6809/44050] conn(0x7fc2f80706b0 0x7fc2f8072ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.732+0000 7f64ba7fc700 1 -- 172.21.15.46:0/3759365295 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64bc156540 msgr2=0x7f64bc0cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.732+0000 7f64ba7fc700 1 --2- 172.21.15.46:0/3759365295 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64bc156540 0x7f64bc0cb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.732+0000 7f64ba7fc700 1 -- 172.21.15.46:0/3759365295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64bc0afbd0 msgr2=0x7f64bc14d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.732+0000 7f64ba7fc700 1 --2- 172.21.15.46:0/3759365295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64bc0afbd0 0x7f64bc14d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.732+0000 7f64ba7fc700 1 -- 172.21.15.46:0/3759365295 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b0008570 con 0x7f64bc156170 2024-01-26T23:30:06.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.735+0000 7f64ba7fc700 1 --2- 172.21.15.46:0/3759365295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64bc156170 0x7f64bc0cb110 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f64bc06d7a0 tx=0x7f64b001c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.736+0000 7fd589c21700 1 --2- 172.21.15.46:0/1284591513 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5840cb4c0 0x7fd5840c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.736+0000 7fe7ccff9700 1 -- 172.21.15.46:0/2584368859 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe7c401a400 con 0x7fe7d00af9a0 2024-01-26T23:30:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.737+0000 7fe7ccff9700 1 --2- 172.21.15.46:0/2584368859 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe7bc066f30 0x7fe7bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.737+0000 7fe7ccff9700 1 -- 172.21.15.46:0/2584368859 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fe7c4084400 con 0x7fe7d00af9a0 2024-01-26T23:30:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.737+0000 7fe7ccff9700 1 --2- 172.21.15.46:0/2584368859 >> [v2:172.21.15.46:6819/43682,v1:172.21.15.46:6821/43682] conn(0x7fe7bc070560 0x7fe7bc072990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.737+0000 7fe7ccff9700 1 -- 172.21.15.46:0/2584368859 --> [v2:172.21.15.46:6819/43682,v1:172.21.15.46:6821/43682] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe7bc073060 con 0x7fe7bc070560 2024-01-26T23:30:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.737+0000 7fe7ccff9700 1 -- 172.21.15.46:0/2584368859 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7c4086900 con 0x7fe7d00af9a0 2024-01-26T23:30:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.738+0000 7f304dffb700 1 -- 172.21.15.46:0/2969887179 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3048022030 con 0x7f30580c16e0 2024-01-26T23:30:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.738+0000 7f304dffb700 1 --2- 172.21.15.46:0/2969887179 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f303c066fb0 0x7f303c069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.738+0000 7f304dffb700 1 -- 172.21.15.46:0/2969887179 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f30480866c0 con 0x7f30580c16e0 2024-01-26T23:30:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.738+0000 7f305dc1c700 1 --2- 172.21.15.46:0/2969887179 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f303c066fb0 0x7f303c069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.738+0000 7fe7d5792700 1 --2- 172.21.15.46:0/2584368859 >> [v2:172.21.15.46:6819/43682,v1:172.21.15.46:6821/43682] conn(0x7fe7bc070560 0x7fe7bc072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.738+0000 7fe7cffff700 1 --2- 172.21.15.46:0/2584368859 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe7bc066f30 0x7fe7bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.739+0000 7fd589c21700 1 -- 172.21.15.46:0/1284591513 >> v2:172.21.15.106:3300/0 conn(0x7fd5840b08c0 msgr2=0x7fd584068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.739+0000 7fd589c21700 1 --2- 172.21.15.46:0/1284591513 >> v2:172.21.15.106:3300/0 conn(0x7fd5840b08c0 0x7fd584068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.739+0000 7fd589c21700 1 -- 172.21.15.46:0/1284591513 >> v1:172.21.15.106:6789/0 conn(0x7fd5840cd9d0 legacy=0x7fd5840cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.739+0000 7fd589c21700 1 -- 172.21.15.46:0/1284591513 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5840c9270 con 0x7fd5840cb4c0 2024-01-26T23:30:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.739+0000 7fe7d5792700 1 --2- 172.21.15.46:0/2584368859 >> [v2:172.21.15.46:6819/43682,v1:172.21.15.46:6821/43682] conn(0x7fe7bc070560 0x7fe7bc072990 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.739+0000 7f305dc1c700 1 --2- 172.21.15.46:0/2969887179 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f303c066fb0 0x7f303c069460 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3054007fd0 tx=0x7f3054007b20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.739+0000 7fe7cffff700 1 --2- 172.21.15.46:0/2584368859 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe7bc066f30 0x7fe7bc0693e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe7d006a890 tx=0x7fe7c8009350 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.739+0000 7fd589c21700 1 --2- 172.21.15.46:0/1284591513 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5840cb4c0 0x7fd5840c9d50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009e60 tx=0x7fd57c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efb216cb6b288154 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.740+0000 7f64a2ffd700 1 -- 172.21.15.46:0/3759365295 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b0016ab0 con 0x7f64bc156170 2024-01-26T23:30:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.740+0000 7f64a2ffd700 1 -- 172.21.15.46:0/3759365295 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b0016c10 con 0x7f64bc156170 2024-01-26T23:30:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.740+0000 7f64a2ffd700 1 -- 172.21.15.46:0/3759365295 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b000b0c0 con 0x7f64bc156170 2024-01-26T23:30:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.741+0000 7f64c180f700 1 -- 172.21.15.46:0/3759365295 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64bc16d9c0 con 0x7f64bc156170 2024-01-26T23:30:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.741+0000 7f64c180f700 1 -- 172.21.15.46:0/3759365295 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64bc16df80 con 0x7f64bc156170 2024-01-26T23:30:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.742+0000 7f64c180f700 1 -- 172.21.15.46:0/3759365295 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f64bc04c710 con 0x7f64bc156170 2024-01-26T23:30:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.742+0000 7fe7ccff9700 1 -- 172.21.15.46:0/2584368859 <== osd.0 v2:172.21.15.46:6819/43682 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fe7bc073060 con 0x7fe7bc070560 2024-01-26T23:30:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.742+0000 7f305fe80700 1 --2- 172.21.15.46:0/2969887179 >> [v2:172.21.15.46:6815/43691,v1:172.21.15.46:6817/43691] conn(0x7f30380015f0 0x7f3038003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.742+0000 7f305fe80700 1 -- 172.21.15.46:0/2969887179 --> [v2:172.21.15.46:6815/43691,v1:172.21.15.46:6817/43691] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3038006b90 con 0x7f30380015f0 2024-01-26T23:30:06.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.742+0000 7fd57bfff700 1 -- 172.21.15.46:0/1284591513 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c00e070 con 0x7fd5840cb4c0 2024-01-26T23:30:06.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.742+0000 7fd57bfff700 1 -- 172.21.15.46:0/1284591513 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57c00bb30 con 0x7fd5840cb4c0 2024-01-26T23:30:06.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.742+0000 7fd57bfff700 1 -- 172.21.15.46:0/1284591513 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c00bed0 con 0x7fd5840cb4c0 2024-01-26T23:30:06.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.743+0000 7fd58b684700 1 -- 172.21.15.46:0/1284591513 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5840cb4c0 msgr2=0x7fd5840c9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.743+0000 7fd58b684700 1 --2- 172.21.15.46:0/1284591513 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5840cb4c0 0x7fd5840c9d50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009e60 tx=0x7fd57c009780 comp rx=0 tx=0).stop 2024-01-26T23:30:06.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.744+0000 7fc31a350700 1 --2- 172.21.15.46:0/3959720760 >> [v2:172.21.15.106:6808/44050,v1:172.21.15.106:6809/44050] conn(0x7fc2f80706b0 0x7fc2f8072ae0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.745+0000 7f64a2ffd700 1 -- 172.21.15.46:0/3759365295 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f64b000b4f0 con 0x7f64bc156170 2024-01-26T23:30:06.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.745+0000 7f305e41d700 1 --2- 172.21.15.46:0/2969887179 >> [v2:172.21.15.46:6815/43691,v1:172.21.15.46:6817/43691] conn(0x7f30380015f0 0x7f3038003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.745+0000 7f64a2ffd700 1 --2- 172.21.15.46:0/3759365295 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f64a8067000 0x7f64a80694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.745+0000 7f64a2ffd700 1 -- 172.21.15.46:0/3759365295 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f64b0086b20 con 0x7f64bc156170 2024-01-26T23:30:06.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.745+0000 7f64a2ffd700 1 --2- 172.21.15.46:0/3759365295 >> [v2:172.21.15.106:6824/44081,v1:172.21.15.106:6825/44081] conn(0x7f64a8070630 0x7f64a8072a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.745+0000 7f64a2ffd700 1 -- 172.21.15.46:0/3759365295 --> [v2:172.21.15.106:6824/44081,v1:172.21.15.106:6825/44081] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f64a8073130 con 0x7f64a8070630 2024-01-26T23:30:06.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.745+0000 7f64a2ffd700 1 -- 172.21.15.46:0/3759365295 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64b0086e10 con 0x7f64bc156170 2024-01-26T23:30:06.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.746+0000 7f305e41d700 1 --2- 172.21.15.46:0/2969887179 >> [v2:172.21.15.46:6815/43691,v1:172.21.15.46:6817/43691] conn(0x7f30380015f0 0x7f3038003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.746+0000 7fd58b684700 1 -- 172.21.15.46:0/1284591513 shutdown_connections 2024-01-26T23:30:06.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.746+0000 7fd58b684700 1 --2- 172.21.15.46:0/1284591513 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5840cb4c0 0x7fd5840c9d50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.746+0000 7fd58b684700 1 --2- 172.21.15.46:0/1284591513 >> v2:172.21.15.106:3300/0 conn(0x7fd5840b08c0 0x7fd584068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.746+0000 7fd58b684700 1 -- 172.21.15.46:0/1284591513 >> 172.21.15.46:0/1284591513 conn(0x7fd584097820 msgr2=0x7fd584068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.746+0000 7f64bb7fe700 1 --2- 172.21.15.46:0/3759365295 >> [v2:172.21.15.106:6824/44081,v1:172.21.15.106:6825/44081] conn(0x7f64a8070630 0x7f64a8072a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.746+0000 7f64baffd700 1 --2- 172.21.15.46:0/3759365295 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f64a8067000 0x7f64a80694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.746+0000 7fd58b684700 1 -- 172.21.15.46:0/1284591513 shutdown_connections 2024-01-26T23:30:06.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.746+0000 7f64baffd700 1 --2- 172.21.15.46:0/3759365295 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f64a8067000 0x7f64a80694b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f64ac006fd0 tx=0x7f64ac006db0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.748+0000 7fc309ffb700 1 -- 172.21.15.46:0/3959720760 <== osd.4 v2:172.21.15.106:6808/44050 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fc2f80731b0 con 0x7fc2f80706b0 2024-01-26T23:30:06.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.746+0000 7fd58b684700 1 -- 172.21.15.46:0/1284591513 wait complete. 2024-01-26T23:30:06.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.748+0000 7fd58b684700 1 Processor -- start 2024-01-26T23:30:06.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.748+0000 7fd58b684700 1 -- start start 2024-01-26T23:30:06.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.750+0000 7fd58b684700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5840b08c0 0x7fd584156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.750+0000 7fd58b684700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5840cb4c0 0x7fd584149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.750+0000 7fd58b684700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5840cd9d0 0x7fd584152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.750+0000 7fd58b684700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd584155670 con 0x7fd5840cb4c0 2024-01-26T23:30:06.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.750+0000 7fd58b684700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58414dec0 con 0x7fd5840b08c0 2024-01-26T23:30:06.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.750+0000 7fd58b684700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58414e1a0 con 0x7fd5840cd9d0 2024-01-26T23:30:06.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.750+0000 7fd589420700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5840b08c0 0x7fd584156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.750+0000 7fd589420700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5840b08c0 0x7fd584156ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56526/0 (socket says 172.21.15.46:56526) 2024-01-26T23:30:06.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.750+0000 7fd589420700 1 -- 172.21.15.46:0/1102339853 learned_addr learned my addr 172.21.15.46:0/1102339853 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:06.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.750+0000 7fd588c1f700 1 --2- 172.21.15.46:0/1102339853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5840cb4c0 0x7fd584149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.750+0000 7fd589420700 1 -- 172.21.15.46:0/1102339853 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5840cd9d0 msgr2=0x7fd584152830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.750+0000 7fd589420700 1 --2- 172.21.15.46:0/1102339853 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5840cd9d0 0x7fd584152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.750+0000 7fd589420700 1 -- 172.21.15.46:0/1102339853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5840cb4c0 msgr2=0x7fd584149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.750+0000 7fd589420700 1 --2- 172.21.15.46:0/1102339853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5840cb4c0 0x7fd584149ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.750+0000 7fd589420700 1 -- 172.21.15.46:0/1102339853 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c009a20 con 0x7fd5840b08c0 2024-01-26T23:30:06.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.750+0000 7fd588c1f700 1 --2- 172.21.15.46:0/1102339853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5840cb4c0 0x7fd584149ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:30:06.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.750+0000 7fd589420700 1 --2- 172.21.15.46:0/1102339853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5840b08c0 0x7fd584156ba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd58000b810 tx=0x7fd580009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.752+0000 7f304dffb700 1 -- 172.21.15.46:0/2969887179 <== osd.3 v2:172.21.15.46:6815/43691 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f3038006b90 con 0x7f30380015f0 2024-01-26T23:30:06.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.754+0000 7f90b3a1e700 1 -- 172.21.15.46:0/2863638460 --> [v2:172.21.15.46:6800/43676,v1:172.21.15.46:6801/43676] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f908c002d30 con 0x7f9090070630 2024-01-26T23:30:06.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.752+0000 7fd5797fa700 1 -- 172.21.15.46:0/1102339853 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58000f5f0 con 0x7fd5840b08c0 2024-01-26T23:30:06.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.752+0000 7fd5797fa700 1 -- 172.21.15.46:0/1102339853 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58000f770 con 0x7fd5840b08c0 2024-01-26T23:30:06.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.752+0000 7fd5797fa700 1 -- 172.21.15.46:0/1102339853 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58000fb30 con 0x7fd5840b08c0 2024-01-26T23:30:06.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.752+0000 7f64bb7fe700 1 --2- 172.21.15.46:0/3759365295 >> [v2:172.21.15.106:6824/44081,v1:172.21.15.106:6825/44081] conn(0x7f64a8070630 0x7f64a8072a60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.753+0000 7f64a2ffd700 1 -- 172.21.15.46:0/3759365295 <== osd.7 v2:172.21.15.106:6824/44081 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f64a8073130 con 0x7f64a8070630 2024-01-26T23:30:06.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.758+0000 7f38adf9a700 1 -- 172.21.15.46:0/3806616831 --> [v2:172.21.15.106:6816/44080,v1:172.21.15.106:6817/44080] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f38a806ed00 con 0x7f38880706b0 2024-01-26T23:30:06.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.763+0000 7f389ffff700 1 -- 172.21.15.46:0/3806616831 <== osd.6 v2:172.21.15.106:6816/44080 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f38a806ed00 con 0x7f38880706b0 2024-01-26T23:30:06.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.763+0000 7f90a17fa700 1 -- 172.21.15.46:0/2863638460 <== osd.1 v2:172.21.15.46:6800/43676 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f908c002d30 con 0x7f9090070630 2024-01-26T23:30:06.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.752+0000 7fd58b684700 1 -- 172.21.15.46:0/1102339853 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd58414e480 con 0x7fd5840b08c0 2024-01-26T23:30:06.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.752+0000 7fd58b684700 1 -- 172.21.15.46:0/1102339853 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd58414db70 con 0x7fd5840b08c0 2024-01-26T23:30:06.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.753+0000 7fd58b684700 1 -- 172.21.15.46:0/1102339853 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd58404c710 con 0x7fd5840b08c0 2024-01-26T23:30:06.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.758+0000 7fd5797fa700 1 -- 172.21.15.46:0/1102339853 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd580010480 con 0x7fd5840b08c0 2024-01-26T23:30:06.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.758+0000 7fd5797fa700 1 --2- 172.21.15.46:0/1102339853 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd568066f30 0x7fd5680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.758+0000 7fd5797fa700 1 -- 172.21.15.46:0/1102339853 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fd580084800 con 0x7fd5840b08c0 2024-01-26T23:30:06.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.759+0000 7fd5797fa700 1 --2- 172.21.15.46:0/1102339853 >> [v2:172.21.15.106:6800/44051,v1:172.21.15.106:6801/44051] conn(0x7fd5680704e0 0x7fd568072910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:06.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.759+0000 7fd5797fa700 1 -- 172.21.15.46:0/1102339853 --> [v2:172.21.15.106:6800/44051,v1:172.21.15.106:6801/44051] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd568072fe0 con 0x7fd5680704e0 2024-01-26T23:30:06.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.763+0000 7f38adf9a700 1 -- 172.21.15.46:0/3806616831 >> [v2:172.21.15.106:6816/44080,v1:172.21.15.106:6817/44080] conn(0x7f38880706b0 msgr2=0x7f3888072ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.763+0000 7f38adf9a700 1 --2- 172.21.15.46:0/3806616831 >> [v2:172.21.15.106:6816/44080,v1:172.21.15.106:6817/44080] conn(0x7f38880706b0 0x7f3888072ae0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.763+0000 7f38adf9a700 1 -- 172.21.15.46:0/3806616831 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3888067080 msgr2=0x7f3888069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.763+0000 7f38adf9a700 1 --2- 172.21.15.46:0/3806616831 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3888067080 0x7f3888069530 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f38a80c5da0 tx=0x7f3890008040 comp rx=0 tx=0).stop 2024-01-26T23:30:06.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.763+0000 7f38adf9a700 1 -- 172.21.15.46:0/3806616831 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38a8156170 msgr2=0x7f38a8086560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.763+0000 7f38adf9a700 1 --2- 172.21.15.46:0/3806616831 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38a8156170 0x7f38a8086560 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3898009290 tx=0x7f3898017000 comp rx=0 tx=0).stop 2024-01-26T23:30:06.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.764+0000 7f38a7fff700 1 -- 172.21.15.46:0/3806616831 reap_dead start 2024-01-26T23:30:06.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.764+0000 7f38adf9a700 1 -- 172.21.15.46:0/3806616831 shutdown_connections 2024-01-26T23:30:06.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.764+0000 7f38adf9a700 1 -- 172.21.15.46:0/3806616831 >> 172.21.15.46:0/3806616831 conn(0x7f38a8068400 msgr2=0x7f38a8159b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.764+0000 7fd589c21700 1 --2- 172.21.15.46:0/1102339853 >> [v2:172.21.15.106:6800/44051,v1:172.21.15.106:6801/44051] conn(0x7fd5680704e0 0x7fd568072910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.764+0000 7fd588c1f700 1 --2- 172.21.15.46:0/1102339853 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd568066f30 0x7fd5680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:06.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.764+0000 7fd5797fa700 1 -- 172.21.15.46:0/1102339853 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd580051760 con 0x7fd5840b08c0 2024-01-26T23:30:06.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.764+0000 7f90b3a1e700 1 -- 172.21.15.46:0/2863638460 >> [v2:172.21.15.46:6800/43676,v1:172.21.15.46:6801/43676] conn(0x7f9090070630 msgr2=0x7f9090072a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.764+0000 7f90b3a1e700 1 --2- 172.21.15.46:0/2863638460 >> [v2:172.21.15.46:6800/43676,v1:172.21.15.46:6801/43676] conn(0x7f9090070630 0x7f9090072a60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.765+0000 7f90b3a1e700 1 -- 172.21.15.46:0/2863638460 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9090067000 msgr2=0x7f90900694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.765+0000 7f90b3a1e700 1 --2- 172.21.15.46:0/2863638460 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9090067000 0x7f90900694b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f909c0027e0 tx=0x7f909c009040 comp rx=0 tx=0).stop 2024-01-26T23:30:06.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.765+0000 7f90b3a1e700 1 -- 172.21.15.46:0/2863638460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac156050 msgr2=0x7f90ac0c46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.765+0000 7f90b3a1e700 1 --2- 172.21.15.46:0/2863638460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac156050 0x7f90ac0c46a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f90a400bc30 tx=0x7f90a4009840 comp rx=0 tx=0).stop 2024-01-26T23:30:06.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.765+0000 7f90b1fbb700 1 -- 172.21.15.46:0/2863638460 reap_dead start 2024-01-26T23:30:06.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.765+0000 7f90b3a1e700 1 -- 172.21.15.46:0/2863638460 shutdown_connections 2024-01-26T23:30:06.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.765+0000 7f90b3a1e700 1 -- 172.21.15.46:0/2863638460 >> 172.21.15.46:0/2863638460 conn(0x7f90ac068400 msgr2=0x7f90ac156470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.764+0000 7f043258f700 1 -- 172.21.15.46:0/1335999664 --> [v2:172.21.15.46:6808/43680,v1:172.21.15.46:6809/43680] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f042c0ac870 con 0x7f04180706b0 2024-01-26T23:30:06.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.765+0000 7fd589c21700 1 --2- 172.21.15.46:0/1102339853 >> [v2:172.21.15.106:6800/44051,v1:172.21.15.106:6801/44051] conn(0x7fd5680704e0 0x7fd568072910 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.765+0000 7f38adf9a700 1 -- 172.21.15.46:0/3806616831 shutdown_connections 2024-01-26T23:30:06.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.765+0000 7f90b3a1e700 1 -- 172.21.15.46:0/2863638460 shutdown_connections 2024-01-26T23:30:06.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.765+0000 7f90b3a1e700 1 -- 172.21.15.46:0/2863638460 wait complete. 2024-01-26T23:30:06.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.765+0000 7f38adf9a700 1 -- 172.21.15.46:0/3806616831 wait complete. 2024-01-26T23:30:06.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.765+0000 7f0413fff700 1 -- 172.21.15.46:0/1335999664 <== osd.2 v2:172.21.15.46:6808/43680 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f042c0ac870 con 0x7f04180706b0 2024-01-26T23:30:06.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.766+0000 7f043258f700 1 -- 172.21.15.46:0/1335999664 >> [v2:172.21.15.46:6808/43680,v1:172.21.15.46:6809/43680] conn(0x7f04180706b0 msgr2=0x7f0418072ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.766+0000 7f043258f700 1 --2- 172.21.15.46:0/1335999664 >> [v2:172.21.15.46:6808/43680,v1:172.21.15.46:6809/43680] conn(0x7f04180706b0 0x7f0418072ae0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.766+0000 7fd588c1f700 1 --2- 172.21.15.46:0/1102339853 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd568066f30 0x7fd5680693e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd5740058d0 tx=0x7fd574009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:06.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.766+0000 7fd5797fa700 1 -- 172.21.15.46:0/1102339853 <== osd.5 v2:172.21.15.106:6800/44051 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fd568072fe0 con 0x7fd5680704e0 2024-01-26T23:30:06.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.768+0000 7f043258f700 1 -- 172.21.15.46:0/1335999664 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0418067080 msgr2=0x7f0418069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.768+0000 7f043258f700 1 --2- 172.21.15.46:0/1335999664 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0418067080 0x7f0418069530 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f042c165ae0 tx=0x7f041c008040 comp rx=0 tx=0).stop 2024-01-26T23:30:06.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.768+0000 7f043258f700 1 -- 172.21.15.46:0/1335999664 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f042c166520 msgr2=0x7f042c0aa0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.768+0000 7f043258f700 1 --2- 172.21.15.46:0/1335999664 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f042c166520 0x7f042c0aa0d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0420009290 tx=0x7f0420018000 comp rx=0 tx=0).stop 2024-01-26T23:30:06.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.768+0000 7f0430b2c700 1 -- 172.21.15.46:0/1335999664 reap_dead start 2024-01-26T23:30:06.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.769+0000 7f043258f700 1 -- 172.21.15.46:0/1335999664 shutdown_connections 2024-01-26T23:30:06.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.769+0000 7f043258f700 1 -- 172.21.15.46:0/1335999664 >> 172.21.15.46:0/1335999664 conn(0x7f042c068400 msgr2=0x7f042c166d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.771+0000 7f305fe80700 1 -- 172.21.15.46:0/2969887179 --> [v2:172.21.15.46:6815/43691,v1:172.21.15.46:6817/43691] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3038005c80 con 0x7f30380015f0 2024-01-26T23:30:06.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.770+0000 7fe7d71f5700 1 -- 172.21.15.46:0/2584368859 --> [v2:172.21.15.46:6819/43682,v1:172.21.15.46:6821/43682] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe7d00b2380 con 0x7fe7bc070560 2024-01-26T23:30:06.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.775+0000 7f304dffb700 1 -- 172.21.15.46:0/2969887179 <== osd.3 v2:172.21.15.46:6815/43691 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3038005c80 con 0x7f30380015f0 2024-01-26T23:30:06.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.775+0000 7f043258f700 1 -- 172.21.15.46:0/1335999664 shutdown_connections 2024-01-26T23:30:06.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.775+0000 7f043258f700 1 -- 172.21.15.46:0/1335999664 wait complete. 2024-01-26T23:30:06.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.780+0000 7fe7ccff9700 1 -- 172.21.15.46:0/2584368859 <== osd.0 v2:172.21.15.46:6819/43682 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe7d00b2380 con 0x7fe7bc070560 2024-01-26T23:30:06.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.780+0000 7fc31bdb3700 1 -- 172.21.15.46:0/3959720760 --> [v2:172.21.15.106:6808/44050,v1:172.21.15.106:6809/44050] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc2f4002d30 con 0x7fc2f80706b0 2024-01-26T23:30:06.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.781+0000 7fc309ffb700 1 -- 172.21.15.46:0/3959720760 <== osd.4 v2:172.21.15.106:6808/44050 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc2f4002d30 con 0x7fc2f80706b0 2024-01-26T23:30:06.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.781+0000 7fe7d71f5700 1 -- 172.21.15.46:0/2584368859 >> [v2:172.21.15.46:6819/43682,v1:172.21.15.46:6821/43682] conn(0x7fe7bc070560 msgr2=0x7fe7bc072990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.781+0000 7fe7d71f5700 1 --2- 172.21.15.46:0/2584368859 >> [v2:172.21.15.46:6819/43682,v1:172.21.15.46:6821/43682] conn(0x7fe7bc070560 0x7fe7bc072990 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.781+0000 7fe7d71f5700 1 -- 172.21.15.46:0/2584368859 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe7bc066f30 msgr2=0x7fe7bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.781+0000 7fe7d71f5700 1 --2- 172.21.15.46:0/2584368859 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe7bc066f30 0x7fe7bc0693e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe7d006a890 tx=0x7fe7c8009350 comp rx=0 tx=0).stop 2024-01-26T23:30:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.781+0000 7fe7d71f5700 1 -- 172.21.15.46:0/2584368859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7d00af9a0 msgr2=0x7fe7d01442b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.781+0000 7fe7d71f5700 1 --2- 172.21.15.46:0/2584368859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7d00af9a0 0x7fe7d01442b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe7c400b5e0 tx=0x7fe7c40090f0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.782+0000 7f305fe80700 1 -- 172.21.15.46:0/2969887179 >> [v2:172.21.15.46:6815/43691,v1:172.21.15.46:6817/43691] conn(0x7f30380015f0 msgr2=0x7f3038003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.782+0000 7f305fe80700 1 --2- 172.21.15.46:0/2969887179 >> [v2:172.21.15.46:6815/43691,v1:172.21.15.46:6817/43691] conn(0x7f30380015f0 0x7f3038003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.782+0000 7f305fe80700 1 -- 172.21.15.46:0/2969887179 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f303c066fb0 msgr2=0x7f303c069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.782+0000 7f305fe80700 1 --2- 172.21.15.46:0/2969887179 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f303c066fb0 0x7f303c069460 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3054007fd0 tx=0x7f3054007b20 comp rx=0 tx=0).stop 2024-01-26T23:30:06.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.782+0000 7f305fe80700 1 -- 172.21.15.46:0/2969887179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30580c16e0 msgr2=0x7f305816dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.782+0000 7f305fe80700 1 --2- 172.21.15.46:0/2969887179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30580c16e0 0x7f305816dd10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f305806f980 tx=0x7f3048016000 comp rx=0 tx=0).stop 2024-01-26T23:30:06.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.782+0000 7fc31bdb3700 1 -- 172.21.15.46:0/3959720760 >> [v2:172.21.15.106:6808/44050,v1:172.21.15.106:6809/44050] conn(0x7fc2f80706b0 msgr2=0x7fc2f8072ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.782+0000 7fc31bdb3700 1 --2- 172.21.15.46:0/3959720760 >> [v2:172.21.15.106:6808/44050,v1:172.21.15.106:6809/44050] conn(0x7fc2f80706b0 0x7fc2f8072ae0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.782+0000 7f305e41d700 1 -- 172.21.15.46:0/2969887179 reap_dead start 2024-01-26T23:30:06.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.783+0000 7fe7d5792700 1 -- 172.21.15.46:0/2584368859 reap_dead start 2024-01-26T23:30:06.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.783+0000 7fe7d71f5700 1 -- 172.21.15.46:0/2584368859 shutdown_connections 2024-01-26T23:30:06.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.783+0000 7fe7d71f5700 1 -- 172.21.15.46:0/2584368859 >> 172.21.15.46:0/2584368859 conn(0x7fe7d0068400 msgr2=0x7fe7d0147840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.783+0000 7fe7d71f5700 1 -- 172.21.15.46:0/2584368859 shutdown_connections 2024-01-26T23:30:06.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.783+0000 7fe7d71f5700 1 -- 172.21.15.46:0/2584368859 wait complete. 2024-01-26T23:30:06.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.783+0000 7f305fe80700 1 -- 172.21.15.46:0/2969887179 shutdown_connections 2024-01-26T23:30:06.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.783+0000 7f305fe80700 1 -- 172.21.15.46:0/2969887179 >> 172.21.15.46:0/2969887179 conn(0x7f30580ce530 msgr2=0x7f30580c8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.784+0000 7f305fe80700 1 -- 172.21.15.46:0/2969887179 shutdown_connections 2024-01-26T23:30:06.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.784+0000 7f305fe80700 1 -- 172.21.15.46:0/2969887179 wait complete. 2024-01-26T23:30:06.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.784+0000 7fc31bdb3700 1 -- 172.21.15.46:0/3959720760 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc2f8067080 msgr2=0x7fc2f8069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.784+0000 7fc31bdb3700 1 --2- 172.21.15.46:0/3959720760 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc2f8067080 0x7fc2f8069530 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc314068fc0 tx=0x7fc310017000 comp rx=0 tx=0).stop 2024-01-26T23:30:06.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.784+0000 7fc31bdb3700 1 -- 172.21.15.46:0/3959720760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc31415ee60 msgr2=0x7fc31415cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.784+0000 7fc31bdb3700 1 --2- 172.21.15.46:0/3959720760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc31415ee60 0x7fc31415cd80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc30400c5b0 tx=0x7fc304007f60 comp rx=0 tx=0).stop 2024-01-26T23:30:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.784+0000 7fc31a350700 1 -- 172.21.15.46:0/3959720760 reap_dead start 2024-01-26T23:30:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.785+0000 7fc31bdb3700 1 -- 172.21.15.46:0/3959720760 shutdown_connections 2024-01-26T23:30:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.785+0000 7fc31bdb3700 1 -- 172.21.15.46:0/3959720760 >> 172.21.15.46:0/3959720760 conn(0x7fc314068400 msgr2=0x7fc3140ac5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.785+0000 7fc31bdb3700 1 -- 172.21.15.46:0/3959720760 shutdown_connections 2024-01-26T23:30:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.785+0000 7fc31bdb3700 1 -- 172.21.15.46:0/3959720760 wait complete. 2024-01-26T23:30:06.797 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-01-26T23:30:06.797 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-26T23:30:06.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.787+0000 7f64c180f700 1 -- 172.21.15.46:0/3759365295 --> [v2:172.21.15.106:6824/44081,v1:172.21.15.106:6825/44081] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f64bc157420 con 0x7f64a8070630 2024-01-26T23:30:06.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.787+0000 7f64a2ffd700 1 -- 172.21.15.46:0/3759365295 <== osd.7 v2:172.21.15.106:6824/44081 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f64bc157420 con 0x7f64a8070630 2024-01-26T23:30:06.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.788+0000 7fd58b684700 1 -- 172.21.15.46:0/1102339853 --> [v2:172.21.15.106:6800/44051,v1:172.21.15.106:6801/44051] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd5840c9550 con 0x7fd5680704e0 2024-01-26T23:30:06.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.788+0000 7fd5797fa700 1 -- 172.21.15.46:0/1102339853 <== osd.5 v2:172.21.15.106:6800/44051 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd5840c9550 con 0x7fd5680704e0 2024-01-26T23:30:06.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.788+0000 7f64c180f700 1 -- 172.21.15.46:0/3759365295 >> [v2:172.21.15.106:6824/44081,v1:172.21.15.106:6825/44081] conn(0x7f64a8070630 msgr2=0x7f64a8072a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.788+0000 7f64c180f700 1 --2- 172.21.15.46:0/3759365295 >> [v2:172.21.15.106:6824/44081,v1:172.21.15.106:6825/44081] conn(0x7f64a8070630 0x7f64a8072a60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.799 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-01-26T23:30:06.800 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-26T23:30:06.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.789+0000 7f64c180f700 1 -- 172.21.15.46:0/3759365295 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f64a8067000 msgr2=0x7f64a80694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.789+0000 7f64c180f700 1 --2- 172.21.15.46:0/3759365295 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f64a8067000 0x7f64a80694b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f64ac006fd0 tx=0x7f64ac006db0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.789+0000 7f64c180f700 1 -- 172.21.15.46:0/3759365295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64bc156170 msgr2=0x7f64bc0cb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.789+0000 7f64c180f700 1 --2- 172.21.15.46:0/3759365295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64bc156170 0x7f64bc0cb110 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f64bc06d7a0 tx=0x7f64b001c000 comp rx=0 tx=0).stop 2024-01-26T23:30:06.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.789+0000 7f64bb7fe700 1 -- 172.21.15.46:0/3759365295 reap_dead start 2024-01-26T23:30:06.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.789+0000 7f64c180f700 1 -- 172.21.15.46:0/3759365295 shutdown_connections 2024-01-26T23:30:06.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.789+0000 7f64c180f700 1 -- 172.21.15.46:0/3759365295 >> 172.21.15.46:0/3759365295 conn(0x7f64bc068400 msgr2=0x7f64bc155070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.793+0000 7f64c180f700 1 -- 172.21.15.46:0/3759365295 shutdown_connections 2024-01-26T23:30:06.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.793+0000 7f64c180f700 1 -- 172.21.15.46:0/3759365295 wait complete. 2024-01-26T23:30:06.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.794+0000 7fd58b684700 1 -- 172.21.15.46:0/1102339853 >> [v2:172.21.15.106:6800/44051,v1:172.21.15.106:6801/44051] conn(0x7fd5680704e0 msgr2=0x7fd568072910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.794+0000 7fd58b684700 1 --2- 172.21.15.46:0/1102339853 >> [v2:172.21.15.106:6800/44051,v1:172.21.15.106:6801/44051] conn(0x7fd5680704e0 0x7fd568072910 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:06.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.794+0000 7fd58b684700 1 -- 172.21.15.46:0/1102339853 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd568066f30 msgr2=0x7fd5680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.794+0000 7fd58b684700 1 --2- 172.21.15.46:0/1102339853 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd568066f30 0x7fd5680693e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd5740058d0 tx=0x7fd574009040 comp rx=0 tx=0).stop 2024-01-26T23:30:06.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.794+0000 7fd58b684700 1 -- 172.21.15.46:0/1102339853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5840b08c0 msgr2=0x7fd584156ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:06.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.794+0000 7fd58b684700 1 --2- 172.21.15.46:0/1102339853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5840b08c0 0x7fd584156ba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd58000b810 tx=0x7fd580009250 comp rx=0 tx=0).stop 2024-01-26T23:30:06.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.794+0000 7fd589c21700 1 -- 172.21.15.46:0/1102339853 reap_dead start 2024-01-26T23:30:06.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.796+0000 7fd58b684700 1 -- 172.21.15.46:0/1102339853 shutdown_connections 2024-01-26T23:30:06.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.796+0000 7fd58b684700 1 -- 172.21.15.46:0/1102339853 >> 172.21.15.46:0/1102339853 conn(0x7fd584097820 msgr2=0x7fd5840690a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:06.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.796+0000 7fd58b684700 1 -- 172.21.15.46:0/1102339853 shutdown_connections 2024-01-26T23:30:06.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:06.796+0000 7fd58b684700 1 -- 172.21.15.46:0/1102339853 wait complete. 2024-01-26T23:30:06.804 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2024-01-26T23:30:06.804 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-26T23:30:06.808 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2024-01-26T23:30:06.809 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2024-01-26T23:30:06.809 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-26T23:30:06.810 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-26T23:30:06.812 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-01-26T23:30:06.813 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-26T23:30:06.820 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-01-26T23:30:06.821 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-26T23:30:06.826 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-01-26T23:30:06.826 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-26T23:30:07.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.028+0000 7fcec8b1c700 1 Processor -- start 2024-01-26T23:30:07.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.029+0000 7fcec8b1c700 1 -- start start 2024-01-26T23:30:07.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.030+0000 7fcec8b1c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec4158d10 0x7fcec4156760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.030+0000 7fcec8b1c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec40b0860 con 0x7fcec4153340 2024-01-26T23:30:07.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.030+0000 7fcec8b1c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec40b0b40 con 0x7fcec4153710 2024-01-26T23:30:07.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.030+0000 7fcec8b1c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec4157220 con 0x7fcec4158d10 2024-01-26T23:30:07.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.031+0000 7fcec259c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcec4153340 0x7fcec40b0080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44482/0 (socket says 172.21.15.46:44482) 2024-01-26T23:30:07.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.031+0000 7fcec259c700 1 -- 172.21.15.46:0/1885906736 learned_addr learned my addr 172.21.15.46:0/1885906736 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:07.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.032+0000 7fcec2d9d700 1 --2- 172.21.15.46:0/1885906736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec4158d10 0x7fcec4156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.032+0000 7fcec159a700 1 -- 172.21.15.46:0/1885906736 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231008560 0 0) 0x7fcec40b0860 con 0x7fcec4153340 2024-01-26T23:30:07.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.032+0000 7fcec159a700 1 -- 172.21.15.46:0/1885906736 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceac003570 con 0x7fcec4153340 2024-01-26T23:30:07.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.032+0000 7fcec2d9d700 1 -- 172.21.15.46:0/1885906736 >> v1:172.21.15.106:6789/0 conn(0x7fcec4153710 legacy=0x7fcec4156050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.032+0000 7fcec2d9d700 1 -- 172.21.15.46:0/1885906736 >> v1:172.21.15.46:6789/0 conn(0x7fcec4153340 legacy=0x7fcec40b0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.032+0000 7fcec2d9d700 1 -- 172.21.15.46:0/1885906736 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec4157500 con 0x7fcec4158d10 2024-01-26T23:30:07.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.033+0000 7fcec2d9d700 1 --2- 172.21.15.46:0/1885906736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec4158d10 0x7fcec4156760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcebc009c00 tx=0x7fcebc009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a14eea980314a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.033+0000 7fcec159a700 1 -- 172.21.15.46:0/1885906736 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc010740 con 0x7fcec4158d10 2024-01-26T23:30:07.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.034+0000 7fcec159a700 1 -- 172.21.15.46:0/1885906736 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcebc0108a0 con 0x7fcec4158d10 2024-01-26T23:30:07.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.034+0000 7fcec159a700 1 -- 172.21.15.46:0/1885906736 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc010c60 con 0x7fcec4158d10 2024-01-26T23:30:07.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.035+0000 7fcec8b1c700 1 -- 172.21.15.46:0/1885906736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec4158d10 msgr2=0x7fcec4156760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.035+0000 7fcec8b1c700 1 --2- 172.21.15.46:0/1885906736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec4158d10 0x7fcec4156760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcebc009c00 tx=0x7fcebc009c30 comp rx=0 tx=0).stop 2024-01-26T23:30:07.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.035+0000 7f3a9b396700 1 Processor -- start 2024-01-26T23:30:07.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.035+0000 7fcec8b1c700 1 -- 172.21.15.46:0/1885906736 shutdown_connections 2024-01-26T23:30:07.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.035+0000 7fcec8b1c700 1 --2- 172.21.15.46:0/1885906736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec4158d10 0x7fcec4156760 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.035+0000 7fcec8b1c700 1 -- 172.21.15.46:0/1885906736 >> 172.21.15.46:0/1885906736 conn(0x7fcec4068400 msgr2=0x7fcec40cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:07.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.035+0000 7fcec8b1c700 1 -- 172.21.15.46:0/1885906736 shutdown_connections 2024-01-26T23:30:07.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.035+0000 7f3a9b396700 1 -- start start 2024-01-26T23:30:07.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.035+0000 7fcec8b1c700 1 -- 172.21.15.46:0/1885906736 wait complete. 2024-01-26T23:30:07.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.037+0000 7f3a9b396700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3a94156170 0x7f3a940b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.037+0000 7f3a9b396700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a94157d50 con 0x7f3a940afbd0 2024-01-26T23:30:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.037+0000 7f3a9b396700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a940af920 con 0x7f3a94156c50 2024-01-26T23:30:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.037+0000 7f3a9b396700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a94157140 con 0x7f3a94156170 2024-01-26T23:30:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.037+0000 7f3a99132700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3a94156170 0x7f3a940b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.037+0000 7f3a99132700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3a94156170 0x7f3a940b0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56530/0 (socket says 172.21.15.46:56530) 2024-01-26T23:30:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.037+0000 7f3a99132700 1 -- 172.21.15.46:0/2347811298 learned_addr learned my addr 172.21.15.46:0/2347811298 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.037+0000 7fcec8b1c700 1 Processor -- start 2024-01-26T23:30:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.038+0000 7fb480ded700 1 Processor -- start 2024-01-26T23:30:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.038+0000 7f3a99132700 1 -- 172.21.15.46:0/2347811298 >> v1:172.21.15.106:6789/0 conn(0x7f3a94156c50 legacy=0x7f3a941581e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-26T23:30:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.038+0000 7f3a99132700 1 -- 172.21.15.46:0/2347811298 >> v1:172.21.15.46:6789/0 conn(0x7f3a940afbd0 legacy=0x7f3a94156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.038+0000 7f3a99132700 1 -- 172.21.15.46:0/2347811298 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a94157420 con 0x7f3a94156170 2024-01-26T23:30:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.038+0000 7f4d57e24700 1 Processor -- start 2024-01-26T23:30:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.038+0000 7f3a99132700 1 --2- 172.21.15.46:0/2347811298 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a94156170 0x7f3a940b0130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00d1a0 tx=0x7f3a8c007690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=635dcde8397ceb3b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.039+0000 7f3a8bfff700 1 -- 172.21.15.46:0/2347811298 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c010070 con 0x7f3a94156170 2024-01-26T23:30:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.039+0000 7f3a8bfff700 1 -- 172.21.15.46:0/2347811298 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8c00dcb0 con 0x7f3a94156170 2024-01-26T23:30:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.039+0000 7f3a8bfff700 1 -- 172.21.15.46:0/2347811298 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c007d90 con 0x7f3a94156170 2024-01-26T23:30:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.039+0000 7f4d57e24700 1 -- start start 2024-01-26T23:30:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.038+0000 7fcec8b1c700 1 -- start start 2024-01-26T23:30:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.039+0000 7fcec8b1c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcec4153340 0x7fcec414e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.039+0000 7fcec8b1c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcec4153710 0x7fcec416e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.039+0000 7fcec8b1c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec4158d10 0x7fcec41624c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.039+0000 7fcec8b1c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec416f3e0 con 0x7fcec4153340 2024-01-26T23:30:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.039+0000 7fcec8b1c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec416ee20 con 0x7fcec4153710 2024-01-26T23:30:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.039+0000 7fcec8b1c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec416f100 con 0x7fcec4158d10 2024-01-26T23:30:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.039+0000 7fcec259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcec4153340 0x7fcec414e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.039+0000 7fcec259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcec4153340 0x7fcec414e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33964/0 (socket says 172.21.15.46:33964) 2024-01-26T23:30:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.039+0000 7fcec259c700 1 -- 172.21.15.46:0/2854027775 learned_addr learned my addr 172.21.15.46:0/2854027775 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.039+0000 7fcec2d9d700 1 --2- 172.21.15.46:0/2854027775 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec4158d10 0x7fcec41624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.039+0000 7fcec1d9b700 1 --2- 172.21.15.46:0/2854027775 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcec4153710 0x7fcec416e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.040+0000 7f3a9b396700 1 -- 172.21.15.46:0/2347811298 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a94156170 msgr2=0x7f3a940b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.040+0000 7f3a9b396700 1 --2- 172.21.15.46:0/2347811298 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a94156170 0x7f3a940b0130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00d1a0 tx=0x7f3a8c007690 comp rx=0 tx=0).stop 2024-01-26T23:30:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.040+0000 7fcec259c700 1 -- 172.21.15.46:0/2854027775 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec4158d10 msgr2=0x7fcec41624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.040+0000 7fcec259c700 1 --2- 172.21.15.46:0/2854027775 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec4158d10 0x7fcec41624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.040+0000 7fcec259c700 1 -- 172.21.15.46:0/2854027775 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcec4153710 msgr2=0x7fcec416e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.040+0000 7fcec259c700 1 --2- 172.21.15.46:0/2854027775 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcec4153710 0x7fcec416e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.040+0000 7fcec259c700 1 -- 172.21.15.46:0/2854027775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc009790 con 0x7fcec4153340 2024-01-26T23:30:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.040+0000 7f3a9b396700 1 -- 172.21.15.46:0/2347811298 shutdown_connections 2024-01-26T23:30:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.040+0000 7f3a9b396700 1 --2- 172.21.15.46:0/2347811298 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a94156170 0x7f3a940b0130 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.041+0000 7f3a9b396700 1 -- 172.21.15.46:0/2347811298 >> 172.21.15.46:0/2347811298 conn(0x7f3a94068400 msgr2=0x7f3a940cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.041+0000 7fcec259c700 1 --2- 172.21.15.46:0/2854027775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcec4153340 0x7fcec414e510 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fceb800c810 tx=0x7fceb8007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.041+0000 7f4d57e24700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4d50156170 0x7f4d500b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.041+0000 7f4d57e24700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d50156c50 0x7f4d500b0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.041+0000 7f4d57e24700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d50157d50 con 0x7f4d500afbd0 2024-01-26T23:30:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.041+0000 7f4d57e24700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d500af920 con 0x7f4d50156170 2024-01-26T23:30:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.041+0000 7f4d57e24700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d50157140 con 0x7f4d50156c50 2024-01-26T23:30:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.041+0000 7f3a9b396700 1 -- 172.21.15.46:0/2347811298 shutdown_connections 2024-01-26T23:30:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.041+0000 7f4d55bc0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4d50156170 0x7f4d500b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.041+0000 7f4d55bc0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4d50156170 0x7f4d500b0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33976/0 (socket says 172.21.15.46:33976) 2024-01-26T23:30:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.041+0000 7f4d55bc0700 1 -- 172.21.15.46:0/133625036 learned_addr learned my addr 172.21.15.46:0/133625036 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.041+0000 7f4d563c1700 1 --2- 172.21.15.46:0/133625036 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d50156c50 0x7f4d500b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.041+0000 7f3a9b396700 1 -- 172.21.15.46:0/2347811298 wait complete. 2024-01-26T23:30:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.041+0000 7f4d55bc0700 1 -- 172.21.15.46:0/133625036 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d50156c50 msgr2=0x7f4d500b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.041+0000 7f4d55bc0700 1 --2- 172.21.15.46:0/133625036 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d50156c50 0x7f4d500b0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.041+0000 7f4d55bc0700 1 -- 172.21.15.46:0/133625036 >> v1:172.21.15.46:6789/0 conn(0x7f4d500afbd0 legacy=0x7f4d50156540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-26T23:30:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.041+0000 7f4d55bc0700 1 -- 172.21.15.46:0/133625036 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d50157420 con 0x7f4d50156170 2024-01-26T23:30:07.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.042+0000 7fceb27fc700 1 -- 172.21.15.46:0/2854027775 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb8005c00 con 0x7fcec4153340 2024-01-26T23:30:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.042+0000 7fceb27fc700 1 -- 172.21.15.46:0/2854027775 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb8005d60 con 0x7fcec4153340 2024-01-26T23:30:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.042+0000 7fceb27fc700 1 -- 172.21.15.46:0/2854027775 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb8007320 con 0x7fcec4153340 2024-01-26T23:30:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.042+0000 7f4d55bc0700 1 --2- 172.21.15.46:0/133625036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d50156170 0x7f4d500b0130 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4d48007a70 tx=0x7f4d4800d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c90a50ec662f7a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.042+0000 7fcec8b1c700 1 -- 172.21.15.46:0/2854027775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec4161cc0 con 0x7fcec4153340 2024-01-26T23:30:07.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.042+0000 7fcec8b1c700 1 -- 172.21.15.46:0/2854027775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec414ea50 con 0x7fcec4153340 2024-01-26T23:30:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.043+0000 7f3a9b396700 1 Processor -- start 2024-01-26T23:30:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.043+0000 7f4d54bbe700 1 -- 172.21.15.46:0/133625036 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4800cc40 con 0x7f4d50156170 2024-01-26T23:30:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.043+0000 7f4d54bbe700 1 -- 172.21.15.46:0/133625036 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4800cda0 con 0x7f4d50156170 2024-01-26T23:30:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.043+0000 7f4d54bbe700 1 -- 172.21.15.46:0/133625036 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4800a360 con 0x7f4d50156170 2024-01-26T23:30:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.043+0000 7f3a9b396700 1 -- start start 2024-01-26T23:30:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.044+0000 7f4d57e24700 1 -- 172.21.15.46:0/133625036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d50156170 msgr2=0x7f4d500b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.044+0000 7f4d57e24700 1 --2- 172.21.15.46:0/133625036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d50156170 0x7f4d500b0130 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4d48007a70 tx=0x7f4d4800d040 comp rx=0 tx=0).stop 2024-01-26T23:30:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.044+0000 7f3a9b396700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a940afbd0 0x7f3a9414bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.044+0000 7f3a9b396700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a94156170 0x7f3a9406b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.044+0000 7f3a9b396700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a94156c50 0x7f3a940adc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.044+0000 7f3a9b396700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a940c2260 con 0x7f3a94156c50 2024-01-26T23:30:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.044+0000 7f3a9b396700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9416ed70 con 0x7f3a940afbd0 2024-01-26T23:30:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.045+0000 7f3a9b396700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9416e160 con 0x7f3a94156170 2024-01-26T23:30:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.045+0000 7f3a99933700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a94156c50 0x7f3a940adc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.045+0000 7f3a99933700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a94156c50 0x7f3a940adc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33980/0 (socket says 172.21.15.46:33980) 2024-01-26T23:30:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.045+0000 7f3a99933700 1 -- 172.21.15.46:0/2027974761 learned_addr learned my addr 172.21.15.46:0/2027974761 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.045+0000 7f3a99132700 1 --2- 172.21.15.46:0/2027974761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a940afbd0 0x7f3a9414bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.045+0000 7f3a98931700 1 --2- 172.21.15.46:0/2027974761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a94156170 0x7f3a9406b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.045+0000 7f4d57e24700 1 -- 172.21.15.46:0/133625036 shutdown_connections 2024-01-26T23:30:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.045+0000 7f4d57e24700 1 --2- 172.21.15.46:0/133625036 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d50156c50 0x7f4d500b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.045+0000 7f4d57e24700 1 --2- 172.21.15.46:0/133625036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d50156170 0x7f4d500b0130 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.045+0000 7f4d57e24700 1 -- 172.21.15.46:0/133625036 >> 172.21.15.46:0/133625036 conn(0x7f4d50068400 msgr2=0x7f4d500cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.045+0000 7fceb27fc700 1 -- 172.21.15.46:0/2854027775 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fceb801b3b0 con 0x7fcec4153340 2024-01-26T23:30:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.045+0000 7fcec8b1c700 1 -- 172.21.15.46:0/2854027775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec404c710 con 0x7fcec4153340 2024-01-26T23:30:07.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.046+0000 7f3a98931700 1 -- 172.21.15.46:0/2027974761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a940afbd0 msgr2=0x7f3a9414bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.046+0000 7f3a98931700 1 --2- 172.21.15.46:0/2027974761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a940afbd0 0x7f3a9414bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.046+0000 7f3a98931700 1 -- 172.21.15.46:0/2027974761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a94156c50 msgr2=0x7f3a940adc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.046+0000 7f3a98931700 1 --2- 172.21.15.46:0/2027974761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a94156c50 0x7f3a940adc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.046+0000 7f3a98931700 1 -- 172.21.15.46:0/2027974761 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8c00be70 con 0x7f3a94156170 2024-01-26T23:30:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.046+0000 7f3a99132700 1 --2- 172.21.15.46:0/2027974761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a940afbd0 0x7f3a9414bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.046+0000 7f3a99933700 1 --2- 172.21.15.46:0/2027974761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a94156c50 0x7f3a940adc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.046+0000 7f3a98931700 1 --2- 172.21.15.46:0/2027974761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a94156170 0x7f3a9406b010 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400c5c0 tx=0x7f3a84007ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.046+0000 7fceb27fc700 1 --2- 172.21.15.46:0/2854027775 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fceac067500 0x7fceac0699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.046+0000 7fceb27fc700 1 -- 172.21.15.46:0/2854027775 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fceb8083d50 con 0x7fcec4153340 2024-01-26T23:30:07.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.046+0000 7f3a897fa700 1 -- 172.21.15.46:0/2027974761 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a84011590 con 0x7f3a94156170 2024-01-26T23:30:07.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.046+0000 7f3a897fa700 1 -- 172.21.15.46:0/2027974761 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a84011710 con 0x7f3a94156170 2024-01-26T23:30:07.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.046+0000 7f3a897fa700 1 -- 172.21.15.46:0/2027974761 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a84011ad0 con 0x7f3a94156170 2024-01-26T23:30:07.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.046+0000 7f3a9b396700 1 -- 172.21.15.46:0/2027974761 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a9416e440 con 0x7f3a94156170 2024-01-26T23:30:07.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.046+0000 7f3a9b396700 1 -- 172.21.15.46:0/2027974761 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a9416ea80 con 0x7f3a94156170 2024-01-26T23:30:07.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.047+0000 7f4d57e24700 1 -- 172.21.15.46:0/133625036 shutdown_connections 2024-01-26T23:30:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.046+0000 7fb480ded700 1 -- start start 2024-01-26T23:30:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.047+0000 7fb480ded700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb47c156170 0x7fb47c0b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.047+0000 7fb480ded700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb47c0afbd0 0x7fb47c0b0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.047+0000 7fb480ded700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47c157d50 con 0x7fb47c06ed00 2024-01-26T23:30:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.047+0000 7fb480ded700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c0af920 con 0x7fb47c156170 2024-01-26T23:30:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.047+0000 7fb480ded700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c157140 con 0x7fb47c0afbd0 2024-01-26T23:30:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.047+0000 7f8b0f1e1700 1 Processor -- start 2024-01-26T23:30:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.047+0000 7f4d57e24700 1 -- 172.21.15.46:0/133625036 wait complete. 2024-01-26T23:30:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.048+0000 7f8b0f1e1700 1 -- start start 2024-01-26T23:30:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.048+0000 7fb471d9b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb47c0afbd0 0x7fb47c0b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.048+0000 7fb471d9b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb47c0afbd0 0x7fb47c0b0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:57484/0 (socket says 172.21.15.46:57484) 2024-01-26T23:30:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.048+0000 7fb471d9b700 1 -- 172.21.15.46:0/1742434117 learned_addr learned my addr 172.21.15.46:0/1742434117 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.048+0000 7f4d57e24700 1 Processor -- start 2024-01-26T23:30:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.048+0000 7f3a9b396700 1 -- 172.21.15.46:0/2027974761 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a940c9f90 con 0x7f3a94156170 2024-01-26T23:30:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.049+0000 7f3a897fa700 1 -- 172.21.15.46:0/2027974761 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3a8401b3b0 con 0x7f3a94156170 2024-01-26T23:30:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.049+0000 7f4d57e24700 1 -- start start 2024-01-26T23:30:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.049+0000 7f3a897fa700 1 --2- 172.21.15.46:0/2027974761 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a78066fd0 0x7f3a78069480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.050+0000 7f3a897fa700 1 -- 172.21.15.46:0/2027974761 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f3a84083c20 con 0x7f3a94156170 2024-01-26T23:30:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.050+0000 7f4d57e24700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d50156170 0x7f4d500ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.050+0000 7f4d57e24700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d50156c50 0x7f4d500c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.050+0000 7f4d57e24700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d5016e750 0x7f4d500c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.050+0000 7f4d57e24700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d500cb160 con 0x7f4d5016e750 2024-01-26T23:30:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.050+0000 7f4d57e24700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d500c9f40 con 0x7f4d50156170 2024-01-26T23:30:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.050+0000 7f4d57e24700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d500c96a0 con 0x7f4d50156c50 2024-01-26T23:30:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.050+0000 7f4d55bc0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d50156170 0x7f4d500ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.050+0000 7f4d55bc0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d50156170 0x7f4d500ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56556/0 (socket says 172.21.15.46:56556) 2024-01-26T23:30:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.050+0000 7f4d55bc0700 1 -- 172.21.15.46:0/3868967549 learned_addr learned my addr 172.21.15.46:0/3868967549 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.051+0000 7f4d553bf700 1 --2- 172.21.15.46:0/3868967549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d50156c50 0x7f4d500c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.049+0000 7fb479d9b700 1 -- 172.21.15.46:0/1742434117 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268239182 0 0) 0x7fb47c157d50 con 0x7fb47c06ed00 2024-01-26T23:30:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.049+0000 7fb479d9b700 1 -- 172.21.15.46:0/1742434117 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb468003570 con 0x7fb47c06ed00 2024-01-26T23:30:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.049+0000 7fb47a59c700 1 --2- 172.21.15.46:0/1742434117 >> v2:172.21.15.46:3300/0 conn(0x7fb47c156170 0x7fb47c0b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.050+0000 7fb47a59c700 1 -- 172.21.15.46:0/1742434117 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb47c0afbd0 msgr2=0x7fb47c0b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.050+0000 7fb47a59c700 1 --2- 172.21.15.46:0/1742434117 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb47c0afbd0 0x7fb47c0b0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.050+0000 7fb47a59c700 1 -- 172.21.15.46:0/1742434117 >> v1:172.21.15.106:6789/0 conn(0x7fb47c06ed00 legacy=0x7fb47c159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.050+0000 7fb47a59c700 1 -- 172.21.15.46:0/1742434117 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c157420 con 0x7fb47c156170 2024-01-26T23:30:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.050+0000 7fb47a59c700 1 --2- 172.21.15.46:0/1742434117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb47c156170 0x7fb47c0b0130 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0098e0 tx=0x7fb46c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e1ab3363d96cf server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.051+0000 7f4d55bc0700 1 -- 172.21.15.46:0/3868967549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d50156c50 msgr2=0x7f4d500c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.051+0000 7f4d55bc0700 1 --2- 172.21.15.46:0/3868967549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d50156c50 0x7f4d500c8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.051+0000 7f4d55bc0700 1 -- 172.21.15.46:0/3868967549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d5016e750 msgr2=0x7f4d500c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.051+0000 7f4d55bc0700 1 --2- 172.21.15.46:0/3868967549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d5016e750 0x7f4d500c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.051+0000 7f4d55bc0700 1 -- 172.21.15.46:0/3868967549 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4800be70 con 0x7f4d50156170 2024-01-26T23:30:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.051+0000 7f4d553bf700 1 --2- 172.21.15.46:0/3868967549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d50156c50 0x7f4d500c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.051+0000 7f4d55bc0700 1 --2- 172.21.15.46:0/3868967549 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d50156170 0x7f4d500ca830 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4d48007260 tx=0x7f4d48015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.051+0000 7f4d45ffb700 1 -- 172.21.15.46:0/3868967549 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d480149c0 con 0x7f4d50156170 2024-01-26T23:30:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.052+0000 7f4d45ffb700 1 -- 172.21.15.46:0/3868967549 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d48014b40 con 0x7f4d50156170 2024-01-26T23:30:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.052+0000 7f4d45ffb700 1 -- 172.21.15.46:0/3868967549 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4801f420 con 0x7f4d50156170 2024-01-26T23:30:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.052+0000 7f4d57e24700 1 -- 172.21.15.46:0/3868967549 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d500c9980 con 0x7f4d50156170 2024-01-26T23:30:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.052+0000 7f4d57e24700 1 -- 172.21.15.46:0/3868967549 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d5016ebd0 con 0x7f4d50156170 2024-01-26T23:30:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.052+0000 7f3a99132700 1 --2- 172.21.15.46:0/2027974761 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a78066fd0 0x7f3a78069480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.053+0000 7f3a99132700 1 --2- 172.21.15.46:0/2027974761 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a78066fd0 0x7f3a78069480 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c007260 tx=0x7f3a8c01a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.053+0000 7f4d57e24700 1 -- 172.21.15.46:0/3868967549 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d500caeb0 con 0x7f4d50156170 2024-01-26T23:30:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.054+0000 7f8b0f1e1700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8b080afbd0 0x7f8b081566c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.054+0000 7f8b0f1e1700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b08157d50 con 0x7f8b0806ed00 2024-01-26T23:30:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.051+0000 7fcec1d9b700 1 --2- 172.21.15.46:0/2854027775 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fceac067500 0x7fceac0699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.052+0000 7fcec1d9b700 1 --2- 172.21.15.46:0/2854027775 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fceac067500 0x7fceac0699b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fceb4000f80 tx=0x7fceb4007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.054+0000 7fb479d9b700 1 -- 172.21.15.46:0/1742434117 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c011630 con 0x7fb47c156170 2024-01-26T23:30:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.054+0000 7fb479d9b700 1 -- 172.21.15.46:0/1742434117 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46c011790 con 0x7fb47c156170 2024-01-26T23:30:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.054+0000 7fb479d9b700 1 -- 172.21.15.46:0/1742434117 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c011b50 con 0x7fb47c156170 2024-01-26T23:30:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.055+0000 7fceb27fc700 1 -- 172.21.15.46:0/2854027775 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fceb8017080 con 0x7fcec4153340 2024-01-26T23:30:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.055+0000 7f3a897fa700 1 -- 172.21.15.46:0/2027974761 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3a940c9f90 con 0x7f3a94156170 2024-01-26T23:30:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.055+0000 7fb480ded700 1 -- 172.21.15.46:0/1742434117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb47c156170 msgr2=0x7fb47c0b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.055+0000 7fb480ded700 1 --2- 172.21.15.46:0/1742434117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb47c156170 0x7fb47c0b0130 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0098e0 tx=0x7fb46c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.054+0000 7f8b0cf7d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8b0806ed00 0x7f8b080b0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44492/0 (socket says 172.21.15.46:44492) 2024-01-26T23:30:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.054+0000 7f8b0cf7d700 1 -- 172.21.15.46:0/4062283295 learned_addr learned my addr 172.21.15.46:0/4062283295 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.055+0000 7f8b0d77e700 1 --2- 172.21.15.46:0/4062283295 >> v2:172.21.15.106:3300/0 conn(0x7f8b080afbd0 0x7f8b081566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.055+0000 7f8b0f1e1700 1 -- 172.21.15.46:0/4062283295 --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b08157140 con 0x7f8b080b0840 2024-01-26T23:30:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.055+0000 7f8b0f1e1700 1 -- 172.21.15.46:0/4062283295 --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b0815c9c0 con 0x7f8b080afbd0 2024-01-26T23:30:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.055+0000 7f8b0d77e700 1 -- 172.21.15.46:0/4062283295 >> v1:172.21.15.106:6789/0 conn(0x7f8b080b0840 legacy=0x7f8b08155fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-26T23:30:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.055+0000 7f8b0d77e700 1 -- 172.21.15.46:0/4062283295 >> v1:172.21.15.46:6789/0 conn(0x7f8b0806ed00 legacy=0x7f8b080b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.055+0000 7f8b0d77e700 1 -- 172.21.15.46:0/4062283295 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b08157420 con 0x7f8b080afbd0 2024-01-26T23:30:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.055+0000 7f4d45ffb700 1 -- 172.21.15.46:0/3868967549 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4d4801f7d0 con 0x7f4d50156170 2024-01-26T23:30:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.056+0000 7f8b0d77e700 1 --2- 172.21.15.46:0/4062283295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b080afbd0 0x7f8b081566c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8af80098e0 tx=0x7f8af80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8df4198e71a5abd8 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.056+0000 7f4d45ffb700 1 --2- 172.21.15.46:0/3868967549 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4d34067050 0x7f4d34069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.056+0000 7f4d45ffb700 1 -- 172.21.15.46:0/3868967549 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f4d48087160 con 0x7f4d50156170 2024-01-26T23:30:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.057+0000 7fb480ded700 1 -- 172.21.15.46:0/1742434117 shutdown_connections 2024-01-26T23:30:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.057+0000 7fb480ded700 1 --2- 172.21.15.46:0/1742434117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb47c156170 0x7fb47c0b0130 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.057+0000 7fb480ded700 1 --2- 172.21.15.46:0/1742434117 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb47c0afbd0 0x7fb47c0b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.057+0000 7fb480ded700 1 -- 172.21.15.46:0/1742434117 >> 172.21.15.46:0/1742434117 conn(0x7fb47c068400 msgr2=0x7fb47c0cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.057+0000 7fb480ded700 1 -- 172.21.15.46:0/1742434117 shutdown_connections 2024-01-26T23:30:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.057+0000 7fb480ded700 1 -- 172.21.15.46:0/1742434117 wait complete. 2024-01-26T23:30:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.058+0000 7f85a31bd700 1 Processor -- start 2024-01-26T23:30:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.058+0000 7f85a31bd700 1 -- start start 2024-01-26T23:30:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.059+0000 7fb480ded700 1 Processor -- start 2024-01-26T23:30:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.060+0000 7f4d553bf700 1 --2- 172.21.15.46:0/3868967549 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4d34067050 0x7f4d34069500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.059+0000 7f85a31bd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f859c15ce40 0x7f859c0d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.059+0000 7f85a31bd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f859c15d210 0x7f859c0cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.059+0000 7f85a31bd700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f859c0cff70 con 0x7f859c0d1390 2024-01-26T23:30:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.059+0000 7f85a31bd700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f859c0cf400 con 0x7f859c15ce40 2024-01-26T23:30:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.059+0000 7f85a31bd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f859c0cf660 con 0x7f859c15d210 2024-01-26T23:30:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.060+0000 7f85a0f59700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f859c15ce40 0x7f859c0d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.060+0000 7f85a0f59700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f859c15ce40 0x7f859c0d0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34008/0 (socket says 172.21.15.46:34008) 2024-01-26T23:30:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.060+0000 7f85a0f59700 1 -- 172.21.15.46:0/969972912 learned_addr learned my addr 172.21.15.46:0/969972912 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.060+0000 7f8b077fe700 1 -- 172.21.15.46:0/4062283295 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af8009e80 con 0x7f8b080afbd0 2024-01-26T23:30:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.060+0000 7f8b077fe700 1 -- 172.21.15.46:0/4062283295 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8af800b820 con 0x7f8b080afbd0 2024-01-26T23:30:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.060+0000 7f8b077fe700 1 -- 172.21.15.46:0/4062283295 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af800bba0 con 0x7f8b080afbd0 2024-01-26T23:30:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.059+0000 7fb480ded700 1 -- start start 2024-01-26T23:30:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.060+0000 7fb480ded700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb47c06ed00 0x7fb47c0c2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.060+0000 7fb480ded700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb47c0afbd0 0x7fb47c0c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.060+0000 7fb480ded700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb47c156170 0x7fb47c0c1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.060+0000 7fb480ded700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0ae950 con 0x7fb47c156170 2024-01-26T23:30:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.060+0000 7fb480ded700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c16e1f0 con 0x7fb47c0afbd0 2024-01-26T23:30:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.060+0000 7fb480ded700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c16e400 con 0x7fb47c06ed00 2024-01-26T23:30:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.060+0000 7f4d45ffb700 1 -- 172.21.15.46:0/3868967549 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4d480540c0 con 0x7f4d50156170 2024-01-26T23:30:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.061+0000 7f859bfff700 1 --2- 172.21.15.46:0/969972912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f859c15d210 0x7f859c0cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.061+0000 7fb47ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb47c156170 0x7fb47c0c1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.061+0000 7fb47ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb47c156170 0x7fb47c0c1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34010/0 (socket says 172.21.15.46:34010) 2024-01-26T23:30:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.061+0000 7fb47ad9d700 1 -- 172.21.15.46:0/1482191102 learned_addr learned my addr 172.21.15.46:0/1482191102 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.061+0000 7f8b0f1e1700 1 -- 172.21.15.46:0/4062283295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b080afbd0 msgr2=0x7f8b081566c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.061+0000 7f8b0f1e1700 1 --2- 172.21.15.46:0/4062283295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b080afbd0 0x7f8b081566c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8af80098e0 tx=0x7f8af80091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.062+0000 7f4d553bf700 1 --2- 172.21.15.46:0/3868967549 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4d34067050 0x7f4d34069500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4d40006fd0 tx=0x7f4d40008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.061+0000 7f859bfff700 1 -- 172.21.15.46:0/969972912 >> v2:172.21.15.46:3300/0 conn(0x7f859c15ce40 msgr2=0x7f859c0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.061+0000 7f859bfff700 1 --2- 172.21.15.46:0/969972912 >> v2:172.21.15.46:3300/0 conn(0x7f859c15ce40 0x7f859c0d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.061+0000 7f859bfff700 1 -- 172.21.15.46:0/969972912 >> v1:172.21.15.46:6789/0 conn(0x7f859c0d1390 legacy=0x7f859c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.061+0000 7f859bfff700 1 -- 172.21.15.46:0/969972912 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f859c0d0500 con 0x7f859c15d210 2024-01-26T23:30:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.062+0000 7f859bfff700 1 --2- 172.21.15.46:0/969972912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f859c15d210 0x7f859c0cbab0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f85880098e0 tx=0x7f85880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2cb5dfc7fe4c933 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.062+0000 7f8b0f1e1700 1 -- 172.21.15.46:0/4062283295 shutdown_connections 2024-01-26T23:30:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.062+0000 7f8b0f1e1700 1 --2- 172.21.15.46:0/4062283295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b080afbd0 0x7f8b081566c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.062+0000 7f8b0f1e1700 1 -- 172.21.15.46:0/4062283295 >> 172.21.15.46:0/4062283295 conn(0x7f8b08068400 msgr2=0x7f8b080cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.062+0000 7f8b0f1e1700 1 -- 172.21.15.46:0/4062283295 shutdown_connections 2024-01-26T23:30:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.062+0000 7fb47ad9d700 1 -- 172.21.15.46:0/1482191102 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb47c06ed00 msgr2=0x7fb47c0c2460 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-26T23:30:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.062+0000 7fb47ad9d700 1 --2- 172.21.15.46:0/1482191102 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb47c06ed00 0x7fb47c0c2460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.062+0000 7fb47ad9d700 1 -- 172.21.15.46:0/1482191102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb47c0afbd0 msgr2=0x7fb47c0c29a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-26T23:30:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.062+0000 7fb47ad9d700 1 --2- 172.21.15.46:0/1482191102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb47c0afbd0 0x7fb47c0c29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.062+0000 7fb47ad9d700 1 -- 172.21.15.46:0/1482191102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c009420 con 0x7fb47c156170 2024-01-26T23:30:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.062+0000 7fb47ad9d700 1 --2- 172.21.15.46:0/1482191102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb47c156170 0x7fb47c0c1340 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb47400c8b0 tx=0x7fb4740016c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.063+0000 7fb472ffd700 1 -- 172.21.15.46:0/1482191102 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb474012630 con 0x7fb47c156170 2024-01-26T23:30:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.063+0000 7fb472ffd700 1 -- 172.21.15.46:0/1482191102 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb474012790 con 0x7fb47c156170 2024-01-26T23:30:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.063+0000 7fb472ffd700 1 -- 172.21.15.46:0/1482191102 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb474012b30 con 0x7fb47c156170 2024-01-26T23:30:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.064+0000 7f859b7fe700 1 -- 172.21.15.46:0/969972912 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8588011630 con 0x7f859c15d210 2024-01-26T23:30:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.064+0000 7f859b7fe700 1 -- 172.21.15.46:0/969972912 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8588011790 con 0x7f859c15d210 2024-01-26T23:30:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.064+0000 7f859b7fe700 1 -- 172.21.15.46:0/969972912 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8588011b30 con 0x7f859c15d210 2024-01-26T23:30:07.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.062+0000 7f8b0f1e1700 1 -- 172.21.15.46:0/4062283295 wait complete. 2024-01-26T23:30:07.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.064+0000 7f8b0f1e1700 1 Processor -- start 2024-01-26T23:30:07.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.065+0000 7f8b0f1e1700 1 -- start start 2024-01-26T23:30:07.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.066+0000 7f8b0f1e1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b0806ed00 0x7f8b0813bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.066+0000 7f8b0f1e1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b080afbd0 0x7f8b0813c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.066+0000 7f8b0f1e1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b080b0840 0x7f8b080c1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.066+0000 7f8b0f1e1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b08140080 con 0x7f8b0806ed00 2024-01-26T23:30:07.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.066+0000 7f8b0f1e1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b08140eb0 con 0x7f8b080afbd0 2024-01-26T23:30:07.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.066+0000 7f8b0f1e1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0813c010 con 0x7f8b080b0840 2024-01-26T23:30:07.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.063+0000 7fb480ded700 1 -- 172.21.15.46:0/1482191102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c16e6e0 con 0x7fb47c156170 2024-01-26T23:30:07.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.063+0000 7fb480ded700 1 -- 172.21.15.46:0/1482191102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c16eca0 con 0x7fb47c156170 2024-01-26T23:30:07.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.066+0000 7f8b0d77e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b080b0840 0x7f8b080c1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.066+0000 7f8b0d77e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b080b0840 0x7f8b080c1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:57526/0 (socket says 172.21.15.46:57526) 2024-01-26T23:30:07.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.066+0000 7f8b0d77e700 1 -- 172.21.15.46:0/1722957148 learned_addr learned my addr 172.21.15.46:0/1722957148 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:07.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.066+0000 7f8b0cf7d700 1 --2- 172.21.15.46:0/1722957148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b0806ed00 0x7f8b0813bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.067+0000 7f8b0d77e700 1 -- 172.21.15.46:0/1722957148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b080afbd0 msgr2=0x7f8b0813c2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-26T23:30:07.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.067+0000 7f8b0d77e700 1 --2- 172.21.15.46:0/1722957148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b080afbd0 0x7f8b0813c2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.067+0000 7f8b0d77e700 1 -- 172.21.15.46:0/1722957148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b0806ed00 msgr2=0x7f8b0813bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.067+0000 7f8b0d77e700 1 --2- 172.21.15.46:0/1722957148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b0806ed00 0x7f8b0813bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.067+0000 7f8b0d77e700 1 -- 172.21.15.46:0/1722957148 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af8009420 con 0x7f8b080b0840 2024-01-26T23:30:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.067+0000 7f8b0d77e700 1 --2- 172.21.15.46:0/1722957148 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b080b0840 0x7f8b080c1340 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8b0806bcd0 tx=0x7f8af8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.067+0000 7fb472ffd700 1 -- 172.21.15.46:0/1482191102 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb474003030 con 0x7fb47c156170 2024-01-26T23:30:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.068+0000 7f85a31bd700 1 -- 172.21.15.46:0/969972912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f859c15d210 msgr2=0x7f859c0cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.068+0000 7f85a31bd700 1 --2- 172.21.15.46:0/969972912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f859c15d210 0x7f859c0cbab0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f85880098e0 tx=0x7f85880091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.068+0000 7f85a31bd700 1 -- 172.21.15.46:0/969972912 shutdown_connections 2024-01-26T23:30:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.068+0000 7f85a31bd700 1 --2- 172.21.15.46:0/969972912 >> v2:172.21.15.46:3300/0 conn(0x7f859c15ce40 0x7f859c0d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.068+0000 7f85a31bd700 1 --2- 172.21.15.46:0/969972912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f859c15d210 0x7f859c0cbab0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.068+0000 7f85a31bd700 1 -- 172.21.15.46:0/969972912 >> 172.21.15.46:0/969972912 conn(0x7f859c06f840 msgr2=0x7f859c0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.068+0000 7fb472ffd700 1 --2- 172.21.15.46:0/1482191102 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb45c067000 0x7fb45c0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.068+0000 7fb472ffd700 1 -- 172.21.15.46:0/1482191102 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fb474084850 con 0x7fb47c156170 2024-01-26T23:30:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.068+0000 7fb47a59c700 1 --2- 172.21.15.46:0/1482191102 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb45c067000 0x7fb45c0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.069+0000 7f8b04ff9700 1 -- 172.21.15.46:0/1722957148 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af801c660 con 0x7f8b080b0840 2024-01-26T23:30:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.069+0000 7f8b04ff9700 1 -- 172.21.15.46:0/1722957148 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8af801c7e0 con 0x7f8b080b0840 2024-01-26T23:30:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.069+0000 7f8b0f1e1700 1 -- 172.21.15.46:0/1722957148 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b081402a0 con 0x7f8b080b0840 2024-01-26T23:30:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.069+0000 7f8b0f1e1700 1 -- 172.21.15.46:0/1722957148 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b08140860 con 0x7f8b080b0840 2024-01-26T23:30:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.071+0000 7f8b04ff9700 1 -- 172.21.15.46:0/1722957148 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af801cba0 con 0x7f8b080b0840 2024-01-26T23:30:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.071+0000 7f8b04ff9700 1 -- 172.21.15.46:0/1722957148 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8af801b470 con 0x7f8b080b0840 2024-01-26T23:30:07.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.072+0000 7f85a31bd700 1 -- 172.21.15.46:0/969972912 shutdown_connections 2024-01-26T23:30:07.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.072+0000 7f85a31bd700 1 -- 172.21.15.46:0/969972912 wait complete. 2024-01-26T23:30:07.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.073+0000 7f85a31bd700 1 Processor -- start 2024-01-26T23:30:07.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.072+0000 7f8b04ff9700 1 --2- 172.21.15.46:0/1722957148 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8af4067000 0x7f8af40694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.072+0000 7f8b04ff9700 1 -- 172.21.15.46:0/1722957148 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f8af80866e0 con 0x7f8b080b0840 2024-01-26T23:30:07.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.072+0000 7f8b0cf7d700 1 --2- 172.21.15.46:0/1722957148 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8af4067000 0x7f8af40694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.072+0000 7f8b0f1e1700 1 -- 172.21.15.46:0/1722957148 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae80052b0 con 0x7f8b080b0840 2024-01-26T23:30:07.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.075+0000 7f8b0cf7d700 1 --2- 172.21.15.46:0/1722957148 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8af4067000 0x7f8af40694b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000e160 tx=0x7f8b00009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.074+0000 7f85a31bd700 1 -- start start 2024-01-26T23:30:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.075+0000 7f85a31bd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f859c0d1390 0x7f859c09a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.075+0000 7f85a31bd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f859c15ce40 0x7f859c0c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.075+0000 7f85a31bd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f859c15d210 0x7f859c16eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.075+0000 7f85a31bd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859c0c7880 con 0x7f859c15ce40 2024-01-26T23:30:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.075+0000 7f85a31bd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859c0c7b60 con 0x7f859c0d1390 2024-01-26T23:30:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.075+0000 7f85a31bd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f859c0c7e40 con 0x7f859c15d210 2024-01-26T23:30:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.076+0000 7f85a175a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f859c15d210 0x7f859c16eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.076+0000 7f85a175a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f859c15d210 0x7f859c16eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:57540/0 (socket says 172.21.15.46:57540) 2024-01-26T23:30:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.076+0000 7f85a175a700 1 -- 172.21.15.46:0/2536186716 learned_addr learned my addr 172.21.15.46:0/2536186716 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.076+0000 7f85a0f59700 1 --2- 172.21.15.46:0/2536186716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f859c0d1390 0x7f859c09a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.076+0000 7f859bfff700 1 --2- 172.21.15.46:0/2536186716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f859c15ce40 0x7f859c0c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.076+0000 7f859bfff700 1 -- 172.21.15.46:0/2536186716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f859c15d210 msgr2=0x7f859c16eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.076+0000 7f859bfff700 1 --2- 172.21.15.46:0/2536186716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f859c15d210 0x7f859c16eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.076+0000 7f859bfff700 1 -- 172.21.15.46:0/2536186716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f859c0d1390 msgr2=0x7f859c09a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.076+0000 7f859bfff700 1 --2- 172.21.15.46:0/2536186716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f859c0d1390 0x7f859c09a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.076+0000 7f859bfff700 1 -- 172.21.15.46:0/2536186716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8588009420 con 0x7f859c15ce40 2024-01-26T23:30:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.077+0000 7f859bfff700 1 --2- 172.21.15.46:0/2536186716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f859c15ce40 0x7f859c0c9de0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8588015000 tx=0x7f8588016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.077+0000 7f8598ff9700 1 -- 172.21.15.46:0/2536186716 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858801bd40 con 0x7f859c15ce40 2024-01-26T23:30:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.077+0000 7f8598ff9700 1 -- 172.21.15.46:0/2536186716 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f858801c450 con 0x7f859c15ce40 2024-01-26T23:30:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.077+0000 7f8598ff9700 1 -- 172.21.15.46:0/2536186716 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858801c810 con 0x7f859c15ce40 2024-01-26T23:30:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.077+0000 7f85a31bd700 1 -- 172.21.15.46:0/2536186716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f859c0c8120 con 0x7f859c15ce40 2024-01-26T23:30:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.078+0000 7f85a31bd700 1 -- 172.21.15.46:0/2536186716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f859c09a6e0 con 0x7f859c15ce40 2024-01-26T23:30:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.079+0000 7f85a31bd700 1 -- 172.21.15.46:0/2536186716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f859c08ad10 con 0x7f859c15ce40 2024-01-26T23:30:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.079+0000 7fb480ded700 1 -- 172.21.15.46:0/1482191102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4600052b0 con 0x7fb47c156170 2024-01-26T23:30:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.082+0000 7f8b04ff9700 1 -- 172.21.15.46:0/1722957148 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8af80536e0 con 0x7f8b080b0840 2024-01-26T23:30:07.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.085+0000 7f8a6570f700 1 Processor -- start 2024-01-26T23:30:07.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.085+0000 7fb47a59c700 1 --2- 172.21.15.46:0/1482191102 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb45c067000 0x7fb45c0694b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0098e0 tx=0x7fb46c016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.085+0000 7f8a6570f700 1 -- start start 2024-01-26T23:30:07.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.086+0000 7f8a6570f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a6006ed00 0x7f8a600b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.086+0000 7f8a6570f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8a600b0670 0x7f8a60159d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.087+0000 7f8a6570f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a600b0ad0 con 0x7f8a600afbd0 2024-01-26T23:30:07.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.087+0000 7f8a6570f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a60157d50 con 0x7f8a600b0670 2024-01-26T23:30:07.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.087+0000 7f8a6570f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a60157140 con 0x7f8a6006ed00 2024-01-26T23:30:07.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.087+0000 7f8a5f7fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8a600b0670 0x7f8a60159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.087+0000 7f8a5f7fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8a600b0670 0x7f8a60159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56604/0 (socket says 172.21.15.46:56604) 2024-01-26T23:30:07.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.087+0000 7f8a5f7fe700 1 -- 172.21.15.46:0/3186359843 learned_addr learned my addr 172.21.15.46:0/3186359843 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:07.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.087+0000 7f8a5effd700 1 --2- 172.21.15.46:0/3186359843 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a6006ed00 0x7f8a600b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.088+0000 7f8a5f7fe700 1 -- 172.21.15.46:0/3186359843 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a6006ed00 msgr2=0x7f8a600b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:07.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.088+0000 7f8a5f7fe700 1 --2- 172.21.15.46:0/3186359843 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a6006ed00 0x7f8a600b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.088+0000 7f8a5f7fe700 1 -- 172.21.15.46:0/3186359843 >> v1:172.21.15.46:6789/0 conn(0x7f8a600afbd0 legacy=0x7f8a60159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.088+0000 7f8a5f7fe700 1 -- 172.21.15.46:0/3186359843 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a60157420 con 0x7f8a600b0670 2024-01-26T23:30:07.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.088+0000 7f8a5f7fe700 1 --2- 172.21.15.46:0/3186359843 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a600b0670 0x7f8a60159d40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8a540099b0 tx=0x7f8a54009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f6fc1084617950b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.088+0000 7f8a5dffb700 1 -- 172.21.15.46:0/3186359843 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5400b860 con 0x7f8a600b0670 2024-01-26T23:30:07.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.088+0000 7f8a5dffb700 1 -- 172.21.15.46:0/3186359843 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a5400b9c0 con 0x7f8a600b0670 2024-01-26T23:30:07.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.088+0000 7f8a5dffb700 1 -- 172.21.15.46:0/3186359843 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5400bd80 con 0x7f8a600b0670 2024-01-26T23:30:07.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.089+0000 7f8a6570f700 1 -- 172.21.15.46:0/3186359843 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a600b0670 msgr2=0x7f8a60159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.089+0000 7f8a6570f700 1 --2- 172.21.15.46:0/3186359843 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a600b0670 0x7f8a60159d40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8a540099b0 tx=0x7f8a54009250 comp rx=0 tx=0).stop 2024-01-26T23:30:07.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.090+0000 7f8a6570f700 1 -- 172.21.15.46:0/3186359843 shutdown_connections 2024-01-26T23:30:07.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.090+0000 7f8a6570f700 1 --2- 172.21.15.46:0/3186359843 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a600b0670 0x7f8a60159d40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.090+0000 7f8a6570f700 1 --2- 172.21.15.46:0/3186359843 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a6006ed00 0x7f8a600b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.090+0000 7f8a6570f700 1 -- 172.21.15.46:0/3186359843 >> 172.21.15.46:0/3186359843 conn(0x7f8a60068400 msgr2=0x7f8a600cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:07.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.090+0000 7f8a6570f700 1 -- 172.21.15.46:0/3186359843 shutdown_connections 2024-01-26T23:30:07.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.090+0000 7f8a6570f700 1 -- 172.21.15.46:0/3186359843 wait complete. 2024-01-26T23:30:07.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.092+0000 7f8a6570f700 1 Processor -- start 2024-01-26T23:30:07.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.092+0000 7f8a6570f700 1 -- start start 2024-01-26T23:30:07.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.093+0000 7f8a6570f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a6006ed00 0x7f8a600c41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.093+0000 7f8a6570f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a600afbd0 0x7f8a600c4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.094+0000 7f8a6570f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a600b0670 0x7f8a6016ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.094+0000 7f8a6570f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a600c4eb0 con 0x7f8a600b0670 2024-01-26T23:30:07.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.094+0000 7f8a6570f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a600c50e0 con 0x7f8a6006ed00 2024-01-26T23:30:07.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.094+0000 7f8a6570f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a600c53c0 con 0x7f8a600afbd0 2024-01-26T23:30:07.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.094+0000 7f8a5f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a600b0670 0x7f8a6016ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.094+0000 7f8a5f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a600b0670 0x7f8a6016ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34022/0 (socket says 172.21.15.46:34022) 2024-01-26T23:30:07.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.094+0000 7f8a5f7fe700 1 -- 172.21.15.46:0/1143566239 learned_addr learned my addr 172.21.15.46:0/1143566239 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:07.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.095+0000 7f8a5effd700 1 --2- 172.21.15.46:0/1143566239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a6006ed00 0x7f8a600c41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.095+0000 7f8a5e7fc700 1 --2- 172.21.15.46:0/1143566239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a600afbd0 0x7f8a600c4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.095+0000 7f8a5f7fe700 1 -- 172.21.15.46:0/1143566239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a600afbd0 msgr2=0x7f8a600c4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.095+0000 7f8a5f7fe700 1 --2- 172.21.15.46:0/1143566239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a600afbd0 0x7f8a600c4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.095+0000 7f8a5f7fe700 1 -- 172.21.15.46:0/1143566239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a6006ed00 msgr2=0x7f8a600c41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.095+0000 7f8a5f7fe700 1 --2- 172.21.15.46:0/1143566239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a6006ed00 0x7f8a600c41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.095+0000 7f8a5f7fe700 1 -- 172.21.15.46:0/1143566239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a540094f0 con 0x7f8a600b0670 2024-01-26T23:30:07.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.095+0000 7f8a5effd700 1 --2- 172.21.15.46:0/1143566239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a6006ed00 0x7f8a600c41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:30:07.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.095+0000 7f8a5f7fe700 1 --2- 172.21.15.46:0/1143566239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a600b0670 0x7f8a6016ac80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8a6006bcd0 tx=0x7f8a54017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.096+0000 7f8598ff9700 1 -- 172.21.15.46:0/2536186716 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f858801cc40 con 0x7f859c15ce40 2024-01-26T23:30:07.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.097+0000 7f8598ff9700 1 --2- 172.21.15.46:0/2536186716 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8580066fb0 0x7f8580069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.097+0000 7f8598ff9700 1 -- 172.21.15.46:0/2536186716 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f8588086680 con 0x7f859c15ce40 2024-01-26T23:30:07.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.097+0000 7f8598ff9700 1 -- 172.21.15.46:0/2536186716 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8588086b10 con 0x7f859c15ce40 2024-01-26T23:30:07.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.099+0000 7f85a0f59700 1 --2- 172.21.15.46:0/2536186716 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8580066fb0 0x7f8580069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.100+0000 7f85a0f59700 1 --2- 172.21.15.46:0/2536186716 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8580066fb0 0x7f8580069460 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8590009da0 tx=0x7f8590009220 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.106+0000 7f8a46ffd700 1 -- 172.21.15.46:0/1143566239 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5401f070 con 0x7f8a600b0670 2024-01-26T23:30:07.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.106+0000 7fb472ffd700 1 -- 172.21.15.46:0/1482191102 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb4740517b0 con 0x7fb47c156170 2024-01-26T23:30:07.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.106+0000 7f8a46ffd700 1 -- 172.21.15.46:0/1143566239 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a54006b10 con 0x7f8a600b0670 2024-01-26T23:30:07.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.106+0000 7f8a46ffd700 1 -- 172.21.15.46:0/1143566239 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5400b860 con 0x7f8a600b0670 2024-01-26T23:30:07.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.106+0000 7f8a6570f700 1 -- 172.21.15.46:0/1143566239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a600c56a0 con 0x7f8a600b0670 2024-01-26T23:30:07.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.106+0000 7f8a6570f700 1 -- 172.21.15.46:0/1143566239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a600c5c60 con 0x7f8a600b0670 2024-01-26T23:30:07.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.108+0000 7f8a6570f700 1 -- 172.21.15.46:0/1143566239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a400052b0 con 0x7f8a600b0670 2024-01-26T23:30:07.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.111+0000 7f8a46ffd700 1 -- 172.21.15.46:0/1143566239 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8a54023070 con 0x7f8a600b0670 2024-01-26T23:30:07.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.111+0000 7f8a46ffd700 1 --2- 172.21.15.46:0/1143566239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8a4c067080 0x7f8a4c069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.111+0000 7f8a46ffd700 1 -- 172.21.15.46:0/1143566239 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f8a54086160 con 0x7f8a600b0670 2024-01-26T23:30:07.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.116+0000 7f8a5effd700 1 --2- 172.21.15.46:0/1143566239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8a4c067080 0x7f8a4c069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.121+0000 7f8a46ffd700 1 -- 172.21.15.46:0/1143566239 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8a54053140 con 0x7f8a600b0670 2024-01-26T23:30:07.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.135+0000 7f8a5effd700 1 --2- 172.21.15.46:0/1143566239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8a4c067080 0x7f8a4c069530 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8a600cb420 tx=0x7f8a5800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.136+0000 7f2416ba8700 1 Processor -- start 2024-01-26T23:30:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.136+0000 7f2416ba8700 1 -- start start 2024-01-26T23:30:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.137+0000 7f2416ba8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2410156170 0x7f24100b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.137+0000 7f2416ba8700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f24100afbd0 0x7f24100b0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.137+0000 7f2416ba8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2410156540 0x7f24101569c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.137+0000 7f2416ba8700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2410156f00 con 0x7f2410156170 2024-01-26T23:30:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.137+0000 7f2416ba8700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2410157d50 con 0x7f24100afbd0 2024-01-26T23:30:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.137+0000 7f2416ba8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24100af920 con 0x7f2410156540 2024-01-26T23:30:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.138+0000 7f2415145700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2410156540 0x7f24101569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.138+0000 7f2415145700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2410156540 0x7f24101569c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:57556/0 (socket says 172.21.15.46:57556) 2024-01-26T23:30:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.138+0000 7f2415145700 1 -- 172.21.15.46:0/633408009 learned_addr learned my addr 172.21.15.46:0/633408009 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.138+0000 7f240ffff700 1 --2- 172.21.15.46:0/633408009 >> v2:172.21.15.106:3300/0 conn(0x7f24100afbd0 0x7f24100b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.139+0000 7f2415145700 1 -- 172.21.15.46:0/633408009 >> v2:172.21.15.106:3300/0 conn(0x7f24100afbd0 msgr2=0x7f24100b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.139+0000 7f2415145700 1 --2- 172.21.15.46:0/633408009 >> v2:172.21.15.106:3300/0 conn(0x7f24100afbd0 0x7f24100b0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.139+0000 7f2415145700 1 -- 172.21.15.46:0/633408009 >> v2:172.21.15.46:3300/0 conn(0x7f2410156170 msgr2=0x7f24100b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.139+0000 7f2415145700 1 --2- 172.21.15.46:0/633408009 >> v2:172.21.15.46:3300/0 conn(0x7f2410156170 0x7f24100b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.139+0000 7f2415145700 1 -- 172.21.15.46:0/633408009 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2410157140 con 0x7f2410156540 2024-01-26T23:30:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.140+0000 7f2415145700 1 --2- 172.21.15.46:0/633408009 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2410156540 0x7f24101569c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f24100cd170 tx=0x7f2408007690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=319a1de533d9895f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.140+0000 7f240f7fe700 1 -- 172.21.15.46:0/633408009 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240800de90 con 0x7f2410156540 2024-01-26T23:30:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.140+0000 7f240f7fe700 1 -- 172.21.15.46:0/633408009 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2408007b40 con 0x7f2410156540 2024-01-26T23:30:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.140+0000 7f240f7fe700 1 -- 172.21.15.46:0/633408009 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24080124d0 con 0x7f2410156540 2024-01-26T23:30:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.141+0000 7f2416ba8700 1 -- 172.21.15.46:0/633408009 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2410156540 msgr2=0x7f24101569c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.141+0000 7f2416ba8700 1 --2- 172.21.15.46:0/633408009 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2410156540 0x7f24101569c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f24100cd170 tx=0x7f2408007690 comp rx=0 tx=0).stop 2024-01-26T23:30:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.142+0000 7f2416ba8700 1 -- 172.21.15.46:0/633408009 shutdown_connections 2024-01-26T23:30:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.142+0000 7f2416ba8700 1 --2- 172.21.15.46:0/633408009 >> v2:172.21.15.46:3300/0 conn(0x7f2410156170 0x7f24100b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.142+0000 7f2416ba8700 1 --2- 172.21.15.46:0/633408009 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2410156540 0x7f24101569c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.142+0000 7f2416ba8700 1 --2- 172.21.15.46:0/633408009 >> v2:172.21.15.106:3300/0 conn(0x7f24100afbd0 0x7f24100b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.142+0000 7f2416ba8700 1 -- 172.21.15.46:0/633408009 >> 172.21.15.46:0/633408009 conn(0x7f2410068400 msgr2=0x7f24100cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:07.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.142+0000 7f2416ba8700 1 -- 172.21.15.46:0/633408009 shutdown_connections 2024-01-26T23:30:07.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.142+0000 7f2416ba8700 1 -- 172.21.15.46:0/633408009 wait complete. 2024-01-26T23:30:07.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.144+0000 7f2416ba8700 1 Processor -- start 2024-01-26T23:30:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.145+0000 7f2416ba8700 1 -- start start 2024-01-26T23:30:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.146+0000 7f2416ba8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24100afbd0 0x7f2410176120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.146+0000 7f2416ba8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2410156170 0x7f24100cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.146+0000 7f2416ba8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2410156540 0x7f24100c2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.147+0000 7f2416ba8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2410176660 con 0x7f24100afbd0 2024-01-26T23:30:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.147+0000 7f2416ba8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24100c90f0 con 0x7f2410156170 2024-01-26T23:30:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.147+0000 7f2416ba8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24100cb6c0 con 0x7f2410156540 2024-01-26T23:30:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.147+0000 7f2414944700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24100afbd0 0x7f2410176120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.147+0000 7f2414944700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24100afbd0 0x7f2410176120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34046/0 (socket says 172.21.15.46:34046) 2024-01-26T23:30:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.147+0000 7f2414944700 1 -- 172.21.15.46:0/2073941319 learned_addr learned my addr 172.21.15.46:0/2073941319 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.148+0000 7f240ffff700 1 --2- 172.21.15.46:0/2073941319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2410156170 0x7f24100cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.148+0000 7f2415145700 1 --2- 172.21.15.46:0/2073941319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2410156540 0x7f24100c2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.148+0000 7f2414944700 1 -- 172.21.15.46:0/2073941319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2410156540 msgr2=0x7f24100c2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.148+0000 7f2414944700 1 --2- 172.21.15.46:0/2073941319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2410156540 0x7f24100c2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.148+0000 7f2414944700 1 -- 172.21.15.46:0/2073941319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2410156170 msgr2=0x7f24100cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.148+0000 7f2414944700 1 --2- 172.21.15.46:0/2073941319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2410156170 0x7f24100cabc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.148+0000 7f2414944700 1 -- 172.21.15.46:0/2073941319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240800be70 con 0x7f24100afbd0 2024-01-26T23:30:07.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.148+0000 7f2414944700 1 --2- 172.21.15.46:0/2073941319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24100afbd0 0x7f2410176120 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f240400b810 tx=0x7f2404009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.150+0000 7f240cff9700 1 -- 172.21.15.46:0/2073941319 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240400f5f0 con 0x7f24100afbd0 2024-01-26T23:30:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.150+0000 7f240cff9700 1 -- 172.21.15.46:0/2073941319 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240400f750 con 0x7f24100afbd0 2024-01-26T23:30:07.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.150+0000 7f240cff9700 1 -- 172.21.15.46:0/2073941319 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240400faf0 con 0x7f24100afbd0 2024-01-26T23:30:07.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.150+0000 7f2416ba8700 1 -- 172.21.15.46:0/2073941319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24100cb9a0 con 0x7f24100afbd0 2024-01-26T23:30:07.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.150+0000 7f2416ba8700 1 -- 172.21.15.46:0/2073941319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24100cbf60 con 0x7f24100afbd0 2024-01-26T23:30:07.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.154+0000 7f240cff9700 1 -- 172.21.15.46:0/2073941319 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2404010480 con 0x7f24100afbd0 2024-01-26T23:30:07.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.156+0000 7f240cff9700 1 --2- 172.21.15.46:0/2073941319 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f23fc067080 0x7f23fc069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:07.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.157+0000 7f240cff9700 1 -- 172.21.15.46:0/2073941319 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f2404084770 con 0x7f24100afbd0 2024-01-26T23:30:07.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.157+0000 7f240ffff700 1 --2- 172.21.15.46:0/2073941319 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f23fc067080 0x7f23fc069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:07.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.158+0000 7f2416ba8700 1 -- 172.21.15.46:0/2073941319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f00052b0 con 0x7f24100afbd0 2024-01-26T23:30:07.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.167+0000 7f240cff9700 1 -- 172.21.15.46:0/2073941319 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f24040551e0 con 0x7f24100afbd0 2024-01-26T23:30:07.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.201+0000 7f240ffff700 1 --2- 172.21.15.46:0/2073941319 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f23fc067080 0x7f23fc069530 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f24100c26c0 tx=0x7f2400008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:07.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.469+0000 7fcec8b1c700 1 -- 172.21.15.46:0/2854027775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fcec41577e0 con 0x7fcec4153340 2024-01-26T23:30:07.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.470+0000 7f4d57e24700 1 -- 172.21.15.46:0/3868967549 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f4d500caeb0 con 0x7f4d50156170 2024-01-26T23:30:07.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.476+0000 7f4d45ffb700 1 -- 172.21.15.46:0/3868967549 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4d480576e0 con 0x7f4d50156170 2024-01-26T23:30:07.479 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2024-01-26T23:30:07.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.478+0000 7fceb27fc700 1 -- 172.21.15.46:0/2854027775 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fceb8050cb0 con 0x7fcec4153340 2024-01-26T23:30:07.482 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2024-01-26T23:30:07.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.482+0000 7f4d57e24700 1 -- 172.21.15.46:0/3868967549 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4d34067050 msgr2=0x7f4d34069500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.482+0000 7f4d57e24700 1 --2- 172.21.15.46:0/3868967549 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4d34067050 0x7f4d34069500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4d40006fd0 tx=0x7f4d40008040 comp rx=0 tx=0).stop 2024-01-26T23:30:07.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.482+0000 7f4d57e24700 1 -- 172.21.15.46:0/3868967549 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d50156170 msgr2=0x7f4d500ca830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.482+0000 7f4d57e24700 1 --2- 172.21.15.46:0/3868967549 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d50156170 0x7f4d500ca830 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4d48007260 tx=0x7f4d48015000 comp rx=0 tx=0).stop 2024-01-26T23:30:07.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.482+0000 7f4d57e24700 1 -- 172.21.15.46:0/3868967549 shutdown_connections 2024-01-26T23:30:07.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.482+0000 7f4d57e24700 1 --2- 172.21.15.46:0/3868967549 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4d34067050 0x7f4d34069500 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.482+0000 7f4d57e24700 1 --2- 172.21.15.46:0/3868967549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d5016e750 0x7f4d500c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.482+0000 7f4d57e24700 1 --2- 172.21.15.46:0/3868967549 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d50156170 0x7f4d500ca830 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.482+0000 7f4d57e24700 1 --2- 172.21.15.46:0/3868967549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d50156c50 0x7f4d500c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.482+0000 7f4d57e24700 1 -- 172.21.15.46:0/3868967549 >> 172.21.15.46:0/3868967549 conn(0x7f4d50068400 msgr2=0x7f4d50159920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.482+0000 7fcec8b1c700 1 -- 172.21.15.46:0/2854027775 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fceac067500 msgr2=0x7fceac0699b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.482+0000 7fcec8b1c700 1 --2- 172.21.15.46:0/2854027775 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fceac067500 0x7fceac0699b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fceb4000f80 tx=0x7fceb4007ec0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.482+0000 7fcec8b1c700 1 -- 172.21.15.46:0/2854027775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcec4153340 msgr2=0x7fcec414e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.482+0000 7fcec8b1c700 1 --2- 172.21.15.46:0/2854027775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcec4153340 0x7fcec414e510 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fceb800c810 tx=0x7fceb8007d90 comp rx=0 tx=0).stop 2024-01-26T23:30:07.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.483+0000 7fcec8b1c700 1 -- 172.21.15.46:0/2854027775 shutdown_connections 2024-01-26T23:30:07.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.483+0000 7fcec8b1c700 1 --2- 172.21.15.46:0/2854027775 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fceac067500 0x7fceac0699b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.483+0000 7fcec8b1c700 1 --2- 172.21.15.46:0/2854027775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcec4153340 0x7fcec414e510 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.483+0000 7fcec8b1c700 1 --2- 172.21.15.46:0/2854027775 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec4158d10 0x7fcec41624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.483+0000 7fcec8b1c700 1 --2- 172.21.15.46:0/2854027775 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcec4153710 0x7fcec416e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.483+0000 7fcec8b1c700 1 -- 172.21.15.46:0/2854027775 >> 172.21.15.46:0/2854027775 conn(0x7fcec4068400 msgr2=0x7fcec40adc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:07.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.483+0000 7fcec8b1c700 1 -- 172.21.15.46:0/2854027775 shutdown_connections 2024-01-26T23:30:07.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.483+0000 7fcec8b1c700 1 -- 172.21.15.46:0/2854027775 wait complete. 2024-01-26T23:30:07.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.483+0000 7f4d57e24700 1 -- 172.21.15.46:0/3868967549 shutdown_connections 2024-01-26T23:30:07.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.483+0000 7f4d57e24700 1 -- 172.21.15.46:0/3868967549 wait complete. 2024-01-26T23:30:07.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.483+0000 7f3a9b396700 1 -- 172.21.15.46:0/2027974761 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3a940c9f90 con 0x7f3a94156170 2024-01-26T23:30:07.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.484+0000 7f3a897fa700 1 -- 172.21.15.46:0/2027974761 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3a940c9f90 con 0x7f3a94156170 2024-01-26T23:30:07.495 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2024-01-26T23:30:07.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.495+0000 7f3a9b396700 1 -- 172.21.15.46:0/2027974761 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a78066fd0 msgr2=0x7f3a78069480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.495+0000 7f3a9b396700 1 --2- 172.21.15.46:0/2027974761 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a78066fd0 0x7f3a78069480 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c007260 tx=0x7f3a8c01a000 comp rx=0 tx=0).stop 2024-01-26T23:30:07.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.495+0000 7f3a9b396700 1 -- 172.21.15.46:0/2027974761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a94156170 msgr2=0x7f3a9406b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.495+0000 7f3a9b396700 1 --2- 172.21.15.46:0/2027974761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a94156170 0x7f3a9406b010 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400c5c0 tx=0x7f3a84007ec0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.495+0000 7f3a9b396700 1 -- 172.21.15.46:0/2027974761 shutdown_connections 2024-01-26T23:30:07.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.495+0000 7f3a9b396700 1 --2- 172.21.15.46:0/2027974761 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a78066fd0 0x7f3a78069480 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c007260 tx=0x7f3a8c01a000 comp rx=0 tx=0).stop 2024-01-26T23:30:07.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.495+0000 7f3a9b396700 1 --2- 172.21.15.46:0/2027974761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a94156c50 0x7f3a940adc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.495+0000 7f3a9b396700 1 --2- 172.21.15.46:0/2027974761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a940afbd0 0x7f3a9414bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.495+0000 7f3a9b396700 1 --2- 172.21.15.46:0/2027974761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a94156170 0x7f3a9406b010 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.496+0000 7f3a9b396700 1 -- 172.21.15.46:0/2027974761 >> 172.21.15.46:0/2027974761 conn(0x7f3a94068400 msgr2=0x7f3a94151ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:07.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.496+0000 7f3a9b396700 1 -- 172.21.15.46:0/2027974761 shutdown_connections 2024-01-26T23:30:07.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.497+0000 7f3a9b396700 1 -- 172.21.15.46:0/2027974761 wait complete. 2024-01-26T23:30:07.507 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.6 2024-01-26T23:30:07.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.512+0000 7f8b0f1e1700 1 -- 172.21.15.46:0/1722957148 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f8ae8005830 con 0x7f8b080b0840 2024-01-26T23:30:07.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.513+0000 7f8b04ff9700 1 -- 172.21.15.46:0/1722957148 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8af802a030 con 0x7f8b080b0840 2024-01-26T23:30:07.514 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.2 2024-01-26T23:30:07.515 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.1 2024-01-26T23:30:07.515 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2024-01-26T23:30:07.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.515+0000 7fb480ded700 1 -- 172.21.15.46:0/1482191102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fb460005560 con 0x7fb47c156170 2024-01-26T23:30:07.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.515+0000 7f8b0f1e1700 1 -- 172.21.15.46:0/1722957148 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8af4067000 msgr2=0x7f8af40694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.515+0000 7f8b0f1e1700 1 --2- 172.21.15.46:0/1722957148 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8af4067000 0x7f8af40694b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000e160 tx=0x7f8b00009040 comp rx=0 tx=0).stop 2024-01-26T23:30:07.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.515+0000 7f8b0f1e1700 1 -- 172.21.15.46:0/1722957148 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b080b0840 msgr2=0x7f8b080c1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.515+0000 7f8b0f1e1700 1 --2- 172.21.15.46:0/1722957148 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b080b0840 0x7f8b080c1340 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8b0806bcd0 tx=0x7f8af8016000 comp rx=0 tx=0).stop 2024-01-26T23:30:07.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.516+0000 7f8b0f1e1700 1 -- 172.21.15.46:0/1722957148 shutdown_connections 2024-01-26T23:30:07.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.516+0000 7f8b0f1e1700 1 --2- 172.21.15.46:0/1722957148 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8af4067000 0x7f8af40694b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.516+0000 7f8b0f1e1700 1 --2- 172.21.15.46:0/1722957148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b0806ed00 0x7f8b0813bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.516+0000 7f8b0f1e1700 1 --2- 172.21.15.46:0/1722957148 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b080b0840 0x7f8b080c1340 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.516+0000 7f8b0f1e1700 1 --2- 172.21.15.46:0/1722957148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b080afbd0 0x7f8b0813c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.516+0000 7f8b0f1e1700 1 -- 172.21.15.46:0/1722957148 >> 172.21.15.46:0/1722957148 conn(0x7f8b08068400 msgr2=0x7f8b0815ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:07.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.516+0000 7f8b0f1e1700 1 -- 172.21.15.46:0/1722957148 shutdown_connections 2024-01-26T23:30:07.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.516+0000 7f8b0f1e1700 1 -- 172.21.15.46:0/1722957148 wait complete. 2024-01-26T23:30:07.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.516+0000 7fb472ffd700 1 -- 172.21.15.46:0/1482191102 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb474054dd0 con 0x7fb47c156170 2024-01-26T23:30:07.518 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2024-01-26T23:30:07.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.518+0000 7fb480ded700 1 -- 172.21.15.46:0/1482191102 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb45c067000 msgr2=0x7fb45c0694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.518+0000 7fb480ded700 1 --2- 172.21.15.46:0/1482191102 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb45c067000 0x7fb45c0694b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0098e0 tx=0x7fb46c016000 comp rx=0 tx=0).stop 2024-01-26T23:30:07.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.518+0000 7fb480ded700 1 -- 172.21.15.46:0/1482191102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb47c156170 msgr2=0x7fb47c0c1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.518+0000 7fb480ded700 1 --2- 172.21.15.46:0/1482191102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb47c156170 0x7fb47c0c1340 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb47400c8b0 tx=0x7fb4740016c0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.519+0000 7fb480ded700 1 -- 172.21.15.46:0/1482191102 shutdown_connections 2024-01-26T23:30:07.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.519+0000 7fb480ded700 1 --2- 172.21.15.46:0/1482191102 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb45c067000 0x7fb45c0694b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.519+0000 7fb480ded700 1 --2- 172.21.15.46:0/1482191102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb47c156170 0x7fb47c0c1340 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.519+0000 7fb480ded700 1 --2- 172.21.15.46:0/1482191102 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb47c06ed00 0x7fb47c0c2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.519+0000 7fb480ded700 1 --2- 172.21.15.46:0/1482191102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb47c0afbd0 0x7fb47c0c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.519+0000 7fb480ded700 1 -- 172.21.15.46:0/1482191102 >> 172.21.15.46:0/1482191102 conn(0x7fb47c068400 msgr2=0x7fb47c069c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:07.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.519+0000 7f85a31bd700 1 -- 172.21.15.46:0/2536186716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f859c04c710 con 0x7f859c15ce40 2024-01-26T23:30:07.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.522+0000 7f8598ff9700 1 -- 172.21.15.46:0/2536186716 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f85880535e0 con 0x7f859c15ce40 2024-01-26T23:30:07.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.522+0000 7fb480ded700 1 -- 172.21.15.46:0/1482191102 shutdown_connections 2024-01-26T23:30:07.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.522+0000 7fb480ded700 1 -- 172.21.15.46:0/1482191102 wait complete. 2024-01-26T23:30:07.523 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2024-01-26T23:30:07.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.524+0000 7f85a31bd700 1 -- 172.21.15.46:0/2536186716 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8580066fb0 msgr2=0x7f8580069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.524+0000 7f85a31bd700 1 --2- 172.21.15.46:0/2536186716 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8580066fb0 0x7f8580069460 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8590009da0 tx=0x7f8590009220 comp rx=0 tx=0).stop 2024-01-26T23:30:07.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.524+0000 7f85a31bd700 1 -- 172.21.15.46:0/2536186716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f859c15ce40 msgr2=0x7f859c0c9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.524+0000 7f85a31bd700 1 --2- 172.21.15.46:0/2536186716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f859c15ce40 0x7f859c0c9de0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8588015000 tx=0x7f8588016000 comp rx=0 tx=0).stop 2024-01-26T23:30:07.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.528+0000 7f85a31bd700 1 -- 172.21.15.46:0/2536186716 shutdown_connections 2024-01-26T23:30:07.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.528+0000 7f85a31bd700 1 --2- 172.21.15.46:0/2536186716 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8580066fb0 0x7f8580069460 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.528+0000 7f85a31bd700 1 --2- 172.21.15.46:0/2536186716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f859c15d210 0x7f859c16eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.528+0000 7f85a31bd700 1 --2- 172.21.15.46:0/2536186716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f859c0d1390 0x7f859c09a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.528+0000 7f85a31bd700 1 --2- 172.21.15.46:0/2536186716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f859c15ce40 0x7f859c0c9de0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.528+0000 7f85a31bd700 1 -- 172.21.15.46:0/2536186716 >> 172.21.15.46:0/2536186716 conn(0x7f859c06f840 msgr2=0x7f859c0b5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:07.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.528+0000 7f85a31bd700 1 -- 172.21.15.46:0/2536186716 shutdown_connections 2024-01-26T23:30:07.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.528+0000 7f85a31bd700 1 -- 172.21.15.46:0/2536186716 wait complete. 2024-01-26T23:30:07.535 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.0 2024-01-26T23:30:07.539 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.3 2024-01-26T23:30:07.545 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.5 2024-01-26T23:30:07.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.546+0000 7f8a6570f700 1 -- 172.21.15.46:0/1143566239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f8a40005830 con 0x7f8a600b0670 2024-01-26T23:30:07.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.546+0000 7f8a46ffd700 1 -- 172.21.15.46:0/1143566239 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8a54056760 con 0x7f8a600b0670 2024-01-26T23:30:07.547 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2024-01-26T23:30:07.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.549+0000 7f8a6570f700 1 -- 172.21.15.46:0/1143566239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8a4c067080 msgr2=0x7f8a4c069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.549+0000 7f8a6570f700 1 --2- 172.21.15.46:0/1143566239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8a4c067080 0x7f8a4c069530 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8a600cb420 tx=0x7f8a5800b040 comp rx=0 tx=0).stop 2024-01-26T23:30:07.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.549+0000 7f8a6570f700 1 -- 172.21.15.46:0/1143566239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a600b0670 msgr2=0x7f8a6016ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.549+0000 7f8a6570f700 1 --2- 172.21.15.46:0/1143566239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a600b0670 0x7f8a6016ac80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8a6006bcd0 tx=0x7f8a54017000 comp rx=0 tx=0).stop 2024-01-26T23:30:07.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.550+0000 7f8a6570f700 1 -- 172.21.15.46:0/1143566239 shutdown_connections 2024-01-26T23:30:07.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.550+0000 7f8a6570f700 1 --2- 172.21.15.46:0/1143566239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8a4c067080 0x7f8a4c069530 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.550+0000 7f8a6570f700 1 --2- 172.21.15.46:0/1143566239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a600b0670 0x7f8a6016ac80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.550+0000 7f8a6570f700 1 --2- 172.21.15.46:0/1143566239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a6006ed00 0x7f8a600c41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.550+0000 7f8a6570f700 1 --2- 172.21.15.46:0/1143566239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a600afbd0 0x7f8a600c4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.550+0000 7f8a6570f700 1 -- 172.21.15.46:0/1143566239 >> 172.21.15.46:0/1143566239 conn(0x7f8a60068400 msgr2=0x7f8a600cc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:07.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.550+0000 7f8a6570f700 1 -- 172.21.15.46:0/1143566239 shutdown_connections 2024-01-26T23:30:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.550+0000 7f8a6570f700 1 -- 172.21.15.46:0/1143566239 wait complete. 2024-01-26T23:30:07.569 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.4 2024-01-26T23:30:07.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.580+0000 7f2416ba8700 1 -- 172.21.15.46:0/2073941319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f23f0005560 con 0x7f24100afbd0 2024-01-26T23:30:07.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.580+0000 7f240cff9700 1 -- 172.21.15.46:0/2073941319 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2404054d70 con 0x7f24100afbd0 2024-01-26T23:30:07.581 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2024-01-26T23:30:07.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.583+0000 7f2416ba8700 1 -- 172.21.15.46:0/2073941319 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f23fc067080 msgr2=0x7f23fc069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.583+0000 7f2416ba8700 1 --2- 172.21.15.46:0/2073941319 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f23fc067080 0x7f23fc069530 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f24100c26c0 tx=0x7f2400008040 comp rx=0 tx=0).stop 2024-01-26T23:30:07.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.583+0000 7f2416ba8700 1 -- 172.21.15.46:0/2073941319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24100afbd0 msgr2=0x7f2410176120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:07.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.583+0000 7f2416ba8700 1 --2- 172.21.15.46:0/2073941319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24100afbd0 0x7f2410176120 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f240400b810 tx=0x7f2404009250 comp rx=0 tx=0).stop 2024-01-26T23:30:07.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.583+0000 7f2416ba8700 1 -- 172.21.15.46:0/2073941319 shutdown_connections 2024-01-26T23:30:07.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.583+0000 7f2416ba8700 1 --2- 172.21.15.46:0/2073941319 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f23fc067080 0x7f23fc069530 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.583+0000 7f2416ba8700 1 --2- 172.21.15.46:0/2073941319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24100afbd0 0x7f2410176120 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.583+0000 7f2416ba8700 1 --2- 172.21.15.46:0/2073941319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2410156540 0x7f24100c2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.583+0000 7f2416ba8700 1 --2- 172.21.15.46:0/2073941319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2410156170 0x7f24100cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:07.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.583+0000 7f2416ba8700 1 -- 172.21.15.46:0/2073941319 >> 172.21.15.46:0/2073941319 conn(0x7f2410068400 msgr2=0x7f24101581e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:07.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.583+0000 7f2416ba8700 1 -- 172.21.15.46:0/2073941319 shutdown_connections 2024-01-26T23:30:07.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:07.583+0000 7f2416ba8700 1 -- 172.21.15.46:0/2073941319 wait complete. 2024-01-26T23:30:07.597 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.7 2024-01-26T23:30:08.508 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-26T23:30:08.515 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-26T23:30:08.516 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-26T23:30:08.536 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-26T23:30:08.540 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-26T23:30:08.547 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-26T23:30:08.570 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-26T23:30:08.598 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-26T23:30:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.715+0000 7ff031dbd700 1 Processor -- start 2024-01-26T23:30:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.715+0000 7ff031dbd700 1 -- start start 2024-01-26T23:30:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.717+0000 7ff031dbd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff02c166520 0x7ff02c0b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.717+0000 7ff031dbd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff02c0b2200 0x7ff02c0af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.717+0000 7ff031dbd700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff02c0acbe0 con 0x7ff02c1668f0 2024-01-26T23:30:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.717+0000 7ff031dbd700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff02c0abfd0 con 0x7ff02c166520 2024-01-26T23:30:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.717+0000 7ff031dbd700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff02c0ac2b0 con 0x7ff02c0b2200 2024-01-26T23:30:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.717+0000 7ff02bfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff02c1668f0 0x7ff02c16a2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44506/0 (socket says 172.21.15.46:44506) 2024-01-26T23:30:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.717+0000 7ff02bfff700 1 -- 172.21.15.46:0/3581654808 learned_addr learned my addr 172.21.15.46:0/3581654808 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.718+0000 7ff02a7fc700 1 -- 172.21.15.46:0/3581654808 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405484411 0 0) 0x7ff02c0acbe0 con 0x7ff02c1668f0 2024-01-26T23:30:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.718+0000 7ff02a7fc700 1 -- 172.21.15.46:0/3581654808 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff014003570 con 0x7ff02c1668f0 2024-01-26T23:30:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.719+0000 7ff02b7fe700 1 --2- 172.21.15.46:0/3581654808 >> v2:172.21.15.46:3300/0 conn(0x7ff02c166520 0x7ff02c0b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.720+0000 7ff02b7fe700 1 -- 172.21.15.46:0/3581654808 >> v2:172.21.15.106:3300/0 conn(0x7ff02c0b2200 msgr2=0x7ff02c0af9a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-26T23:30:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.720+0000 7ff02b7fe700 1 --2- 172.21.15.46:0/3581654808 >> v2:172.21.15.106:3300/0 conn(0x7ff02c0b2200 0x7ff02c0af9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.720+0000 7ff02b7fe700 1 -- 172.21.15.46:0/3581654808 >> v1:172.21.15.46:6789/0 conn(0x7ff02c1668f0 legacy=0x7ff02c16a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.720+0000 7ff02b7fe700 1 -- 172.21.15.46:0/3581654808 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff02c0ac590 con 0x7ff02c166520 2024-01-26T23:30:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.720+0000 7ff02b7fe700 1 --2- 172.21.15.46:0/3581654808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02c166520 0x7ff02c0b1cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff02c06f190 tx=0x7ff01c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab078891cb12ab2e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.721+0000 7ff02a7fc700 1 -- 172.21.15.46:0/3581654808 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01c00c150 con 0x7ff02c166520 2024-01-26T23:30:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.721+0000 7ff02a7fc700 1 -- 172.21.15.46:0/3581654808 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff01c00c2b0 con 0x7ff02c166520 2024-01-26T23:30:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.721+0000 7ff02a7fc700 1 -- 172.21.15.46:0/3581654808 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01c007a50 con 0x7ff02c166520 2024-01-26T23:30:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.722+0000 7ff031dbd700 1 -- 172.21.15.46:0/3581654808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02c166520 msgr2=0x7ff02c0b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.722+0000 7ff031dbd700 1 --2- 172.21.15.46:0/3581654808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02c166520 0x7ff02c0b1cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff02c06f190 tx=0x7ff01c005e40 comp rx=0 tx=0).stop 2024-01-26T23:30:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.723+0000 7ff031dbd700 1 -- 172.21.15.46:0/3581654808 shutdown_connections 2024-01-26T23:30:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.723+0000 7ff031dbd700 1 --2- 172.21.15.46:0/3581654808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02c166520 0x7ff02c0b1cc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.723+0000 7ff031dbd700 1 --2- 172.21.15.46:0/3581654808 >> v2:172.21.15.106:3300/0 conn(0x7ff02c0b2200 0x7ff02c0af9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.723+0000 7ff031dbd700 1 -- 172.21.15.46:0/3581654808 >> 172.21.15.46:0/3581654808 conn(0x7ff02c068400 msgr2=0x7ff02c0cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:08.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.724+0000 7ff031dbd700 1 -- 172.21.15.46:0/3581654808 shutdown_connections 2024-01-26T23:30:08.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.724+0000 7ff031dbd700 1 -- 172.21.15.46:0/3581654808 wait complete. 2024-01-26T23:30:08.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.726+0000 7ff031dbd700 1 Processor -- start 2024-01-26T23:30:08.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.726+0000 7ff031dbd700 1 -- start start 2024-01-26T23:30:08.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.727+0000 7ff031dbd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02c166520 0x7ff02c16c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.727+0000 7ff031dbd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff02c1668f0 0x7ff02c17aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.728+0000 7ff031dbd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff02c1575f0 0x7ff02c086240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.728+0000 7ff031dbd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02c157ae0 con 0x7ff02c166520 2024-01-26T23:30:08.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.728+0000 7ff031dbd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02c167370 con 0x7ff02c1575f0 2024-01-26T23:30:08.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.728+0000 7ff031dbd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff02c167650 con 0x7ff02c1668f0 2024-01-26T23:30:08.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.728+0000 7ff02b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02c166520 0x7ff02c16c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.728+0000 7ff02b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02c166520 0x7ff02c16c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34068/0 (socket says 172.21.15.46:34068) 2024-01-26T23:30:08.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.728+0000 7ff02b7fe700 1 -- 172.21.15.46:0/4082350573 learned_addr learned my addr 172.21.15.46:0/4082350573 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:08.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.729+0000 7ff02bfff700 1 --2- 172.21.15.46:0/4082350573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff02c1575f0 0x7ff02c086240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.729+0000 7ff02b7fe700 1 -- 172.21.15.46:0/4082350573 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff02c1668f0 msgr2=0x7ff02c17aa00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-26T23:30:08.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.729+0000 7ff02b7fe700 1 --2- 172.21.15.46:0/4082350573 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff02c1668f0 0x7ff02c17aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.729+0000 7ff02b7fe700 1 -- 172.21.15.46:0/4082350573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff02c1575f0 msgr2=0x7ff02c086240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.729+0000 7ff02b7fe700 1 --2- 172.21.15.46:0/4082350573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff02c1575f0 0x7ff02c086240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.729+0000 7ff02b7fe700 1 -- 172.21.15.46:0/4082350573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01c006100 con 0x7ff02c166520 2024-01-26T23:30:08.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.730+0000 7ff02b7fe700 1 --2- 172.21.15.46:0/4082350573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02c166520 0x7ff02c16c360 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff01c01b000 tx=0x7ff01c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.731+0000 7ff01b7fe700 1 -- 172.21.15.46:0/4082350573 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01c00c420 con 0x7ff02c166520 2024-01-26T23:30:08.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.731+0000 7ff01b7fe700 1 -- 172.21.15.46:0/4082350573 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff01c000bc0 con 0x7ff02c166520 2024-01-26T23:30:08.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.731+0000 7ff01b7fe700 1 -- 172.21.15.46:0/4082350573 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01c01a4a0 con 0x7ff02c166520 2024-01-26T23:30:08.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.731+0000 7ff031dbd700 1 -- 172.21.15.46:0/4082350573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff02c167930 con 0x7ff02c166520 2024-01-26T23:30:08.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.731+0000 7ff031dbd700 1 -- 172.21.15.46:0/4082350573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff02c16c8a0 con 0x7ff02c166520 2024-01-26T23:30:08.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.737+0000 7ff01b7fe700 1 -- 172.21.15.46:0/4082350573 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff01c01a850 con 0x7ff02c166520 2024-01-26T23:30:08.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.738+0000 7ff01b7fe700 1 --2- 172.21.15.46:0/4082350573 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff014067520 0x7ff0140699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.738+0000 7ff01b7fe700 1 -- 172.21.15.46:0/4082350573 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7ff01c011070 con 0x7ff02c166520 2024-01-26T23:30:08.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.741+0000 7ff02affd700 1 --2- 172.21.15.46:0/4082350573 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff014067520 0x7ff0140699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.741+0000 7ff031dbd700 1 -- 172.21.15.46:0/4082350573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0080052b0 con 0x7ff02c166520 2024-01-26T23:30:08.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.745+0000 7ff02affd700 1 --2- 172.21.15.46:0/4082350573 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff014067520 0x7ff0140699d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0aebf0 tx=0x7ff024009370 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.757+0000 7ff01b7fe700 1 -- 172.21.15.46:0/4082350573 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff01c0529d0 con 0x7ff02c166520 2024-01-26T23:30:08.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.786+0000 7fc5662dd700 1 Processor -- start 2024-01-26T23:30:08.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.786+0000 7fc5662dd700 1 -- start start 2024-01-26T23:30:08.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.787+0000 7fc5662dd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc56015dad0 0x7fc560067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.787+0000 7fc5662dd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc560067e70 0x7fc5600b0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.787+0000 7fc5662dd700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc56006cfe0 con 0x7fc5600b0e70 2024-01-26T23:30:08.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.787+0000 7fc5662dd700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5600b16a0 con 0x7fc56015dad0 2024-01-26T23:30:08.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.787+0000 7fc5662dd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc560068310 con 0x7fc560067e70 2024-01-26T23:30:08.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.788+0000 7fc55ffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc56015dad0 0x7fc560067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.788+0000 7fc55ffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc56015dad0 0x7fc560067930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34076/0 (socket says 172.21.15.46:34076) 2024-01-26T23:30:08.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.788+0000 7fc55ffff700 1 -- 172.21.15.46:0/3519829732 learned_addr learned my addr 172.21.15.46:0/3519829732 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:08.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.788+0000 7fc55f7fe700 1 --2- 172.21.15.46:0/3519829732 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc560067e70 0x7fc5600b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.788+0000 7fc55ffff700 1 -- 172.21.15.46:0/3519829732 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc560067e70 msgr2=0x7fc5600b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:08.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.788+0000 7fc55ffff700 1 --2- 172.21.15.46:0/3519829732 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc560067e70 0x7fc5600b0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.788+0000 7fc55ffff700 1 -- 172.21.15.46:0/3519829732 >> v1:172.21.15.46:6789/0 conn(0x7fc5600b0e70 legacy=0x7fc56006eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:08.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.788+0000 7fc55ffff700 1 -- 172.21.15.46:0/3519829732 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56006c6b0 con 0x7fc56015dad0 2024-01-26T23:30:08.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.788+0000 7fc55f7fe700 1 --2- 172.21.15.46:0/3519829732 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc560067e70 0x7fc5600b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:08.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.789+0000 7fc55ffff700 1 --2- 172.21.15.46:0/3519829732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc56015dad0 0x7fc560067930 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc5540099e0 tx=0x7fc554009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ddf916faa0ccb15 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.790+0000 7f9da1f14700 1 Processor -- start 2024-01-26T23:30:08.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.790+0000 7f9da1f14700 1 -- start start 2024-01-26T23:30:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.791+0000 7fe20226b700 1 Processor -- start 2024-01-26T23:30:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.791+0000 7fb8c2959700 1 Processor -- start 2024-01-26T23:30:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.791+0000 7fb8c2959700 1 -- start start 2024-01-26T23:30:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.792+0000 7f9da1f14700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d9c15b700 0x7f9d9c15bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.792+0000 7f9da1f14700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d9c0b1cc0 0x7f9d9c15ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.792+0000 7f9da1f14700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9d9c0af9a0 0x7f9d9c0afe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.792+0000 7f9da1f14700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d9c0b0360 con 0x7f9d9c15b700 2024-01-26T23:30:08.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.792+0000 7f9da1f14700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d9c15feb0 con 0x7f9d9c0af9a0 2024-01-26T23:30:08.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.792+0000 7f9da1f14700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0b2130 con 0x7f9d9c0b1cc0 2024-01-26T23:30:08.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.792+0000 7f9d9bfff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9d9c0af9a0 0x7f9d9c0afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.792+0000 7f9d9affd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d9c0b1cc0 0x7f9d9c15ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.792+0000 7f9d9affd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d9c0b1cc0 0x7f9d9c15ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:57576/0 (socket says 172.21.15.46:57576) 2024-01-26T23:30:08.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.792+0000 7f9d9bfff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9d9c0af9a0 0x7f9d9c0afe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56670/0 (socket says 172.21.15.46:56670) 2024-01-26T23:30:08.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.792+0000 7f9d9affd700 1 -- 172.21.15.46:0/438920780 learned_addr learned my addr 172.21.15.46:0/438920780 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:08.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.792+0000 7fb8c2959700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8bc0a9840 0x7fb8bc0aade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fb8c2959700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb8bc159bd0 0x7fb8bc0ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fb8c2959700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8bc0aa950 con 0x7fb8bc159800 2024-01-26T23:30:08.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fb8c2959700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8bc0ace00 con 0x7fb8bc159bd0 2024-01-26T23:30:08.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fb8c2959700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0a9d40 con 0x7fb8bc0a9840 2024-01-26T23:30:08.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7f9d9affd700 1 -- 172.21.15.46:0/438920780 >> v2:172.21.15.106:3300/0 conn(0x7f9d9c0af9a0 msgr2=0x7f9d9c0afe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:08.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7f9d9affd700 1 --2- 172.21.15.46:0/438920780 >> v2:172.21.15.106:3300/0 conn(0x7f9d9c0af9a0 0x7f9d9c0afe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7f9d9affd700 1 -- 172.21.15.46:0/438920780 >> v2:172.21.15.46:3300/0 conn(0x7f9d9c15b700 msgr2=0x7f9d9c15bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:08.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7f9d9affd700 1 --2- 172.21.15.46:0/438920780 >> v2:172.21.15.46:3300/0 conn(0x7f9d9c15b700 0x7f9d9c15bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7f9d9affd700 1 -- 172.21.15.46:0/438920780 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c15f5c0 con 0x7f9d9c0b1cc0 2024-01-26T23:30:08.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.791+0000 7fe20226b700 1 -- start start 2024-01-26T23:30:08.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fe20226b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe1fc068040 0x7fe1fc0b0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fe20226b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe1fc0b0e70 0x7fe1fc15dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fe20226b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1fc06cfe0 con 0x7fe1fc15dad0 2024-01-26T23:30:08.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fe20226b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1fc0b16a0 con 0x7fe1fc068040 2024-01-26T23:30:08.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fe20226b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1fc06c4a0 con 0x7fe1fc0b0e70 2024-01-26T23:30:08.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fb8c0ef6700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb8bc159bd0 0x7fb8bc0ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fb8c0ef6700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb8bc159bd0 0x7fb8bc0ad230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56682/0 (socket says 172.21.15.46:56682) 2024-01-26T23:30:08.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fb8c0ef6700 1 -- 172.21.15.46:0/2790029953 learned_addr learned my addr 172.21.15.46:0/2790029953 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:08.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7f9d9affd700 1 --2- 172.21.15.46:0/438920780 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d9c0b1cc0 0x7f9d9c15ee20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c071bf0 tx=0x7f9d90009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edba1a79735b999b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fc55effd700 1 -- 172.21.15.46:0/3519829732 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55400b7e0 con 0x7fc56015dad0 2024-01-26T23:30:08.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fe1fbfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe1fc15dad0 0x7fe1fc067930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44522/0 (socket says 172.21.15.46:44522) 2024-01-26T23:30:08.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fe1fbfff700 1 -- 172.21.15.46:0/1560510157 learned_addr learned my addr 172.21.15.46:0/1560510157 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:08.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fc55effd700 1 -- 172.21.15.46:0/3519829732 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55400b940 con 0x7fc56015dad0 2024-01-26T23:30:08.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fc55effd700 1 -- 172.21.15.46:0/3519829732 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55400bce0 con 0x7fc56015dad0 2024-01-26T23:30:08.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fe200808700 1 --2- 172.21.15.46:0/1560510157 >> v2:172.21.15.106:3300/0 conn(0x7fe1fc0b0e70 0x7fe1fc15dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.794+0000 7f9d9a7fc700 1 -- 172.21.15.46:0/438920780 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d90009e70 con 0x7f9d9c0b1cc0 2024-01-26T23:30:08.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.793+0000 7fb8bb7fe700 1 --2- 172.21.15.46:0/2790029953 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8bc0a9840 0x7fb8bc0aade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.794+0000 7fb8bb7fe700 1 -- 172.21.15.46:0/2790029953 >> v2:172.21.15.106:3300/0 conn(0x7fb8bc159bd0 msgr2=0x7fb8bc0ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:08.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.794+0000 7fb8bb7fe700 1 --2- 172.21.15.46:0/2790029953 >> v2:172.21.15.106:3300/0 conn(0x7fb8bc159bd0 0x7fb8bc0ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.794+0000 7fb8bb7fe700 1 -- 172.21.15.46:0/2790029953 >> v1:172.21.15.46:6789/0 conn(0x7fb8bc159800 legacy=0x7fb8bc0ac6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:08.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.794+0000 7fb8bb7fe700 1 -- 172.21.15.46:0/2790029953 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc0aa020 con 0x7fb8bc0a9840 2024-01-26T23:30:08.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.794+0000 7f9d9a7fc700 1 -- 172.21.15.46:0/438920780 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9000b860 con 0x7f9d9c0b1cc0 2024-01-26T23:30:08.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.794+0000 7f9d9a7fc700 1 -- 172.21.15.46:0/438920780 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9000bc00 con 0x7f9d9c0b1cc0 2024-01-26T23:30:08.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.794+0000 7fe1fb7fe700 1 --2- 172.21.15.46:0/1560510157 >> v2:172.21.15.46:3300/0 conn(0x7fe1fc068040 0x7fe1fc0b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.794+0000 7fb8bb7fe700 1 --2- 172.21.15.46:0/2790029953 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8bc0a9840 0x7fb8bc0aade0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009e60 tx=0x7fb8b4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aea1e0ac50c5f7af server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.794+0000 7fc5662dd700 1 -- 172.21.15.46:0/3519829732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc56015dad0 msgr2=0x7fc560067930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.794+0000 7fc5662dd700 1 --2- 172.21.15.46:0/3519829732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc56015dad0 0x7fc560067930 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc5540099e0 tx=0x7fc554009250 comp rx=0 tx=0).stop 2024-01-26T23:30:08.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7fc5662dd700 1 -- 172.21.15.46:0/3519829732 shutdown_connections 2024-01-26T23:30:08.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7fc5662dd700 1 --2- 172.21.15.46:0/3519829732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc56015dad0 0x7fc560067930 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7fc5662dd700 1 --2- 172.21.15.46:0/3519829732 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc560067e70 0x7fc5600b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7fc5662dd700 1 -- 172.21.15.46:0/3519829732 >> 172.21.15.46:0/3519829732 conn(0x7fc56006b630 msgr2=0x7fc5600aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:08.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7f9da1f14700 1 -- 172.21.15.46:0/438920780 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d9c0b1cc0 msgr2=0x7f9d9c15ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7f9da1f14700 1 --2- 172.21.15.46:0/438920780 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d9c0b1cc0 0x7f9d9c15ee20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c071bf0 tx=0x7f9d90009250 comp rx=0 tx=0).stop 2024-01-26T23:30:08.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.794+0000 7fe1fb7fe700 1 -- 172.21.15.46:0/1560510157 >> v2:172.21.15.106:3300/0 conn(0x7fe1fc0b0e70 msgr2=0x7fe1fc15dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:08.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.794+0000 7fe1fb7fe700 1 --2- 172.21.15.46:0/1560510157 >> v2:172.21.15.106:3300/0 conn(0x7fe1fc0b0e70 0x7fe1fc15dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7fe1fb7fe700 1 -- 172.21.15.46:0/1560510157 >> v1:172.21.15.46:6789/0 conn(0x7fe1fc15dad0 legacy=0x7fe1fc067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7fe1fb7fe700 1 -- 172.21.15.46:0/1560510157 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1fc06c6b0 con 0x7fe1fc068040 2024-01-26T23:30:08.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7fc5662dd700 1 -- 172.21.15.46:0/3519829732 shutdown_connections 2024-01-26T23:30:08.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7f9da1f14700 1 -- 172.21.15.46:0/438920780 shutdown_connections 2024-01-26T23:30:08.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7f9da1f14700 1 --2- 172.21.15.46:0/438920780 >> v2:172.21.15.46:3300/0 conn(0x7f9d9c15b700 0x7f9d9c15bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7f9da1f14700 1 --2- 172.21.15.46:0/438920780 >> v2:172.21.15.106:3300/0 conn(0x7f9d9c0af9a0 0x7f9d9c0afe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7f9da1f14700 1 --2- 172.21.15.46:0/438920780 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d9c0b1cc0 0x7f9d9c15ee20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7f9da1f14700 1 -- 172.21.15.46:0/438920780 >> 172.21.15.46:0/438920780 conn(0x7f9d9c068400 msgr2=0x7f9d9c0cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:08.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7fc5662dd700 1 -- 172.21.15.46:0/3519829732 wait complete. 2024-01-26T23:30:08.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7f9da1f14700 1 -- 172.21.15.46:0/438920780 shutdown_connections 2024-01-26T23:30:08.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7f9da1f14700 1 -- 172.21.15.46:0/438920780 wait complete. 2024-01-26T23:30:08.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.795+0000 7fe1fb7fe700 1 --2- 172.21.15.46:0/1560510157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1fc068040 0x7fe1fc0b0930 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc0b1d10 tx=0x7fe1f00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269cea1d752a34f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.796+0000 7fe1faffd700 1 -- 172.21.15.46:0/1560510157 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0011630 con 0x7fe1fc068040 2024-01-26T23:30:08.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.796+0000 7fe1faffd700 1 -- 172.21.15.46:0/1560510157 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f0011790 con 0x7fe1fc068040 2024-01-26T23:30:08.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.796+0000 7fe1faffd700 1 -- 172.21.15.46:0/1560510157 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0011b30 con 0x7fe1fc068040 2024-01-26T23:30:08.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.796+0000 7fb8baffd700 1 -- 172.21.15.46:0/2790029953 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b400e070 con 0x7fb8bc0a9840 2024-01-26T23:30:08.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.796+0000 7fb8baffd700 1 -- 172.21.15.46:0/2790029953 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b400bb20 con 0x7fb8bc0a9840 2024-01-26T23:30:08.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.796+0000 7fb8baffd700 1 -- 172.21.15.46:0/2790029953 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4012430 con 0x7fb8bc0a9840 2024-01-26T23:30:08.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.797+0000 7fc5662dd700 1 Processor -- start 2024-01-26T23:30:08.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.797+0000 7fb8c2959700 1 -- 172.21.15.46:0/2790029953 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8bc0a9840 msgr2=0x7fb8bc0aade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.797+0000 7fb8c2959700 1 --2- 172.21.15.46:0/2790029953 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8bc0a9840 0x7fb8bc0aade0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009e60 tx=0x7fb8b4009780 comp rx=0 tx=0).stop 2024-01-26T23:30:08.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.797+0000 7f42fe9a8700 1 Processor -- start 2024-01-26T23:30:08.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.797+0000 7f42fe9a8700 1 -- start start 2024-01-26T23:30:08.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.798+0000 7fe20226b700 1 -- 172.21.15.46:0/1560510157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1fc068040 msgr2=0x7fe1fc0b0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.798+0000 7fe20226b700 1 --2- 172.21.15.46:0/1560510157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1fc068040 0x7fe1fc0b0930 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc0b1d10 tx=0x7fe1f00091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.798+0000 7f42fe9a8700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f42f80b2b40 0x7f42f80ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.798+0000 7f42fe9a8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f80aedd0 0x7f42f80cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.798+0000 7f42fe9a8700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f815a0b0 con 0x7f42f80ae960 2024-01-26T23:30:08.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.798+0000 7f42fe9a8700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f8099fc0 con 0x7f42f80b2b40 2024-01-26T23:30:08.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.798+0000 7f42fe9a8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f80ae4d0 con 0x7f42f80aedd0 2024-01-26T23:30:08.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.797+0000 7f9da1f14700 1 Processor -- start 2024-01-26T23:30:08.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7f42fcf45700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f80aedd0 0x7f42f80cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7f42fcf45700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f80aedd0 0x7f42f80cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:57600/0 (socket says 172.21.15.46:57600) 2024-01-26T23:30:08.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7f42fcf45700 1 -- 172.21.15.46:0/91384229 learned_addr learned my addr 172.21.15.46:0/91384229 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:08.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.797+0000 7fc5662dd700 1 -- start start 2024-01-26T23:30:08.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7fc5662dd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc560067e70 0x7fc560158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7fc5662dd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5600b0e70 0x7fc560160780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7fc5662dd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc56015dad0 0x7fc560160cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7fc5662dd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5600cab60 con 0x7fc560067e70 2024-01-26T23:30:08.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7fc5662dd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5600c9f50 con 0x7fc5600b0e70 2024-01-26T23:30:08.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7fc5662dd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5600ca230 con 0x7fc56015dad0 2024-01-26T23:30:08.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7f42fcf45700 1 -- 172.21.15.46:0/91384229 >> v2:172.21.15.106:3300/0 conn(0x7f42f80b2b40 msgr2=0x7f42f80ad1c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-26T23:30:08.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7f42fcf45700 1 --2- 172.21.15.46:0/91384229 >> v2:172.21.15.106:3300/0 conn(0x7f42f80b2b40 0x7f42f80ad1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7f42fcf45700 1 -- 172.21.15.46:0/91384229 >> v1:172.21.15.106:6789/0 conn(0x7f42f80ae960 legacy=0x7f42f80ced50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-26T23:30:08.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7f42fcf45700 1 -- 172.21.15.46:0/91384229 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f80ad8c0 con 0x7f42f80aedd0 2024-01-26T23:30:08.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7fc55f7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5600b0e70 0x7fc560160780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7fc55f7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5600b0e70 0x7fc560160780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56700/0 (socket says 172.21.15.46:56700) 2024-01-26T23:30:08.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7fc55f7fe700 1 -- 172.21.15.46:0/3023354000 learned_addr learned my addr 172.21.15.46:0/3023354000 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:08.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7f42fcf45700 1 --2- 172.21.15.46:0/91384229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f80aedd0 0x7f42f80cb5e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f42f0009f20 tx=0x7f42f0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=631395e2a95de8d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7fc55ffff700 1 --2- 172.21.15.46:0/3023354000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc560067e70 0x7fc560158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7fc56487a700 1 --2- 172.21.15.46:0/3023354000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc56015dad0 0x7fc560160cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7fc55f7fe700 1 -- 172.21.15.46:0/3023354000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc56015dad0 msgr2=0x7fc560160cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7fc55f7fe700 1 --2- 172.21.15.46:0/3023354000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc56015dad0 0x7fc560160cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7fc55f7fe700 1 -- 172.21.15.46:0/3023354000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc560067e70 msgr2=0x7fc560158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7fc55f7fe700 1 --2- 172.21.15.46:0/3023354000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc560067e70 0x7fc560158b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7fc55f7fe700 1 -- 172.21.15.46:0/3023354000 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5540094f0 con 0x7fc5600b0e70 2024-01-26T23:30:08.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7fc55ffff700 1 --2- 172.21.15.46:0/3023354000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc560067e70 0x7fc560158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:08.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7f42f6ffd700 1 -- 172.21.15.46:0/91384229 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f000e070 con 0x7f42f80aedd0 2024-01-26T23:30:08.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7f42f6ffd700 1 -- 172.21.15.46:0/91384229 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f000ba40 con 0x7f42f80aedd0 2024-01-26T23:30:08.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7f42f6ffd700 1 -- 172.21.15.46:0/91384229 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f00092e0 con 0x7f42f80aedd0 2024-01-26T23:30:08.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.799+0000 7f9da1f14700 1 -- start start 2024-01-26T23:30:08.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7f9da1f14700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d9c0af9a0 0x7f9d9c163a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7f9da1f14700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d9c0b1cc0 0x7f9d9c0a64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7f9da1f14700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c15b700 0x7f9d9c16e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7f9da1f14700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c1635a0 con 0x7f9d9c15b700 2024-01-26T23:30:08.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7f9da1f14700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c162990 con 0x7f9d9c0af9a0 2024-01-26T23:30:08.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7f9da1f14700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c162c70 con 0x7f9d9c0b1cc0 2024-01-26T23:30:08.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7f9d9bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c15b700 0x7f9d9c16e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7f9d9bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c15b700 0x7f9d9c16e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34094/0 (socket says 172.21.15.46:34094) 2024-01-26T23:30:08.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7f9d9bfff700 1 -- 172.21.15.46:0/1869657807 learned_addr learned my addr 172.21.15.46:0/1869657807 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:08.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.801+0000 7f9d9bfff700 1 -- 172.21.15.46:0/1869657807 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d9c0b1cc0 msgr2=0x7f9d9c0a64f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-26T23:30:08.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.801+0000 7f9d9bfff700 1 --2- 172.21.15.46:0/1869657807 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d9c0b1cc0 0x7f9d9c0a64f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.801+0000 7f9d9bfff700 1 -- 172.21.15.46:0/1869657807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d9c0af9a0 msgr2=0x7f9d9c163a30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-26T23:30:08.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.801+0000 7f9d9bfff700 1 --2- 172.21.15.46:0/1869657807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d9c0af9a0 0x7f9d9c163a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.801+0000 7f9d9bfff700 1 -- 172.21.15.46:0/1869657807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d900094f0 con 0x7f9d9c15b700 2024-01-26T23:30:08.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.801+0000 7f9d9bfff700 1 --2- 172.21.15.46:0/1869657807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c15b700 0x7f9d9c16e670 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00ea50 tx=0x7f9d8c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.802+0000 7f9d837fe700 1 -- 172.21.15.46:0/1869657807 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c00c850 con 0x7f9d9c15b700 2024-01-26T23:30:08.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.802+0000 7f9d837fe700 1 -- 172.21.15.46:0/1869657807 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c00c9b0 con 0x7f9d9c15b700 2024-01-26T23:30:08.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.802+0000 7f9d837fe700 1 -- 172.21.15.46:0/1869657807 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c00cd70 con 0x7f9d9c15b700 2024-01-26T23:30:08.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.802+0000 7fe20226b700 1 -- 172.21.15.46:0/1560510157 shutdown_connections 2024-01-26T23:30:08.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.802+0000 7fe20226b700 1 --2- 172.21.15.46:0/1560510157 >> v2:172.21.15.106:3300/0 conn(0x7fe1fc0b0e70 0x7fe1fc15dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.802+0000 7fe20226b700 1 --2- 172.21.15.46:0/1560510157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1fc068040 0x7fe1fc0b0930 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.802+0000 7fe20226b700 1 -- 172.21.15.46:0/1560510157 >> 172.21.15.46:0/1560510157 conn(0x7fe1fc06b630 msgr2=0x7fe1fc0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:08.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.802+0000 7f9da1f14700 1 -- 172.21.15.46:0/1869657807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d9c162f50 con 0x7f9d9c15b700 2024-01-26T23:30:08.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.802+0000 7f9da1f14700 1 -- 172.21.15.46:0/1869657807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9c0a6a30 con 0x7f9d9c15b700 2024-01-26T23:30:08.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.803+0000 7f42fe9a8700 1 -- 172.21.15.46:0/91384229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f80aedd0 msgr2=0x7f42f80cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.803+0000 7f42fe9a8700 1 --2- 172.21.15.46:0/91384229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f80aedd0 0x7f42f80cb5e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f42f0009f20 tx=0x7f42f0009840 comp rx=0 tx=0).stop 2024-01-26T23:30:08.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7fc55f7fe700 1 --2- 172.21.15.46:0/3023354000 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5600b0e70 0x7fc560160780 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc55000e8d0 tx=0x7fc550008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7fc547fff700 1 -- 172.21.15.46:0/3023354000 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55000ee70 con 0x7fc5600b0e70 2024-01-26T23:30:08.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7fc547fff700 1 -- 172.21.15.46:0/3023354000 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55000c5e0 con 0x7fc5600b0e70 2024-01-26T23:30:08.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7fc547fff700 1 -- 172.21.15.46:0/3023354000 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55000c980 con 0x7fc5600b0e70 2024-01-26T23:30:08.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7fc5662dd700 1 -- 172.21.15.46:0/3023354000 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5600ca510 con 0x7fc5600b0e70 2024-01-26T23:30:08.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.800+0000 7fc5662dd700 1 -- 172.21.15.46:0/3023354000 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc560159070 con 0x7fc5600b0e70 2024-01-26T23:30:08.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.802+0000 7fc547fff700 1 -- 172.21.15.46:0/3023354000 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc550013410 con 0x7fc5600b0e70 2024-01-26T23:30:08.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.803+0000 7fc547fff700 1 --2- 172.21.15.46:0/3023354000 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc54c066fb0 0x7fc54c069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.803+0000 7fc547fff700 1 -- 172.21.15.46:0/3023354000 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fc5500846d0 con 0x7fc5600b0e70 2024-01-26T23:30:08.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.803+0000 7fc5662dd700 1 -- 172.21.15.46:0/3023354000 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc56004c710 con 0x7fc5600b0e70 2024-01-26T23:30:08.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.803+0000 7f42fe9a8700 1 -- 172.21.15.46:0/91384229 shutdown_connections 2024-01-26T23:30:08.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.803+0000 7f42fe9a8700 1 --2- 172.21.15.46:0/91384229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f80aedd0 0x7f42f80cb5e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.803+0000 7f42fe9a8700 1 --2- 172.21.15.46:0/91384229 >> v2:172.21.15.106:3300/0 conn(0x7f42f80b2b40 0x7f42f80ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.803+0000 7f42fe9a8700 1 -- 172.21.15.46:0/91384229 >> 172.21.15.46:0/91384229 conn(0x7f42f8099a30 msgr2=0x7f42f806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:08.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.804+0000 7f42fe9a8700 1 -- 172.21.15.46:0/91384229 shutdown_connections 2024-01-26T23:30:08.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.804+0000 7fe20226b700 1 -- 172.21.15.46:0/1560510157 shutdown_connections 2024-01-26T23:30:08.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.805+0000 7f9d837fe700 1 -- 172.21.15.46:0/1869657807 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9d8c013680 con 0x7f9d9c15b700 2024-01-26T23:30:08.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.805+0000 7f9d837fe700 1 --2- 172.21.15.46:0/1869657807 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d88066f30 0x7f9d880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.804+0000 7fc55ffff700 1 --2- 172.21.15.46:0/3023354000 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc54c066fb0 0x7fc54c069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.805+0000 7fc55ffff700 1 --2- 172.21.15.46:0/3023354000 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc54c066fb0 0x7fc54c069460 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc5540099e0 tx=0x7fc554016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.806+0000 7f9d837fe700 1 -- 172.21.15.46:0/1869657807 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f9d8c084c30 con 0x7f9d9c15b700 2024-01-26T23:30:08.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.804+0000 7f42fe9a8700 1 -- 172.21.15.46:0/91384229 wait complete. 2024-01-26T23:30:08.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.806+0000 7f42fe9a8700 1 Processor -- start 2024-01-26T23:30:08.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.805+0000 7fe20226b700 1 -- 172.21.15.46:0/1560510157 wait complete. 2024-01-26T23:30:08.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.807+0000 7fb8c2959700 1 -- 172.21.15.46:0/2790029953 shutdown_connections 2024-01-26T23:30:08.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.807+0000 7fb8c2959700 1 --2- 172.21.15.46:0/2790029953 >> v2:172.21.15.106:3300/0 conn(0x7fb8bc159bd0 0x7fb8bc0ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.807+0000 7fb8c2959700 1 --2- 172.21.15.46:0/2790029953 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8bc0a9840 0x7fb8bc0aade0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.807+0000 7fe20226b700 1 Processor -- start 2024-01-26T23:30:08.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.807+0000 7fb8c2959700 1 -- 172.21.15.46:0/2790029953 >> 172.21.15.46:0/2790029953 conn(0x7fb8bc068400 msgr2=0x7fb8bc0cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:08.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.807+0000 7fe20226b700 1 -- start start 2024-01-26T23:30:08.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.809+0000 7fc547fff700 1 -- 172.21.15.46:0/3023354000 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc550055040 con 0x7fc5600b0e70 2024-01-26T23:30:08.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.808+0000 7fe20226b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1fc068040 0x7fe1fc0cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.808+0000 7fe20226b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe1fc0b0e70 0x7fe1fc13ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.808+0000 7fe20226b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1fc15dad0 0x7fe1fc14db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.808+0000 7fe20226b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc13d810 con 0x7fe1fc068040 2024-01-26T23:30:08.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.808+0000 7fe20226b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc0cb730 con 0x7fe1fc15dad0 2024-01-26T23:30:08.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.808+0000 7fe20226b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc13da20 con 0x7fe1fc0b0e70 2024-01-26T23:30:08.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.809+0000 7fe1fbfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1fc068040 0x7fe1fc0cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.809+0000 7fe1fbfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1fc068040 0x7fe1fc0cb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34108/0 (socket says 172.21.15.46:34108) 2024-01-26T23:30:08.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.809+0000 7fe1fbfff700 1 -- 172.21.15.46:0/3081585256 learned_addr learned my addr 172.21.15.46:0/3081585256 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:08.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.809+0000 7fe1fbfff700 1 -- 172.21.15.46:0/3081585256 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe1fc0b0e70 msgr2=0x7fe1fc13ed30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-26T23:30:08.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.809+0000 7fe1fbfff700 1 --2- 172.21.15.46:0/3081585256 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe1fc0b0e70 0x7fe1fc13ed30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.809+0000 7fe1fbfff700 1 -- 172.21.15.46:0/3081585256 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1fc15dad0 msgr2=0x7fe1fc14db20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-26T23:30:08.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.809+0000 7fe1fbfff700 1 --2- 172.21.15.46:0/3081585256 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1fc15dad0 0x7fe1fc14db20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.809+0000 7fe1fbfff700 1 -- 172.21.15.46:0/3081585256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f0009420 con 0x7fe1fc068040 2024-01-26T23:30:08.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.806+0000 7f42fe9a8700 1 -- start start 2024-01-26T23:30:08.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.807+0000 7f42fe9a8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42f80ae960 0x7f42f81589e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.807+0000 7f42fe9a8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f80b2b40 0x7f42f80b5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.807+0000 7f42fe9a8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f80b1310 0x7f42f80b1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.807+0000 7f42fe9a8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f80b5260 con 0x7f42f80b2b40 2024-01-26T23:30:08.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.807+0000 7f42fe9a8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f80b1cb0 con 0x7f42f80ae960 2024-01-26T23:30:08.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.807+0000 7f42fe9a8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f80b4dd0 con 0x7f42f80b1310 2024-01-26T23:30:08.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.808+0000 7f42fcf45700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f80b1310 0x7f42f80b1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.808+0000 7f42fcf45700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f80b1310 0x7f42f80b1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:57638/0 (socket says 172.21.15.46:57638) 2024-01-26T23:30:08.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.808+0000 7f42fcf45700 1 -- 172.21.15.46:0/3798103400 learned_addr learned my addr 172.21.15.46:0/3798103400 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:08.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.808+0000 7f42fcf45700 1 -- 172.21.15.46:0/3798103400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42f80ae960 msgr2=0x7f42f81589e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-26T23:30:08.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.808+0000 7f42fcf45700 1 --2- 172.21.15.46:0/3798103400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42f80ae960 0x7f42f81589e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.808+0000 7f42fcf45700 1 -- 172.21.15.46:0/3798103400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f80b2b40 msgr2=0x7f42f80b5d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:30:08.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.808+0000 7f42fcf45700 1 --2- 172.21.15.46:0/3798103400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f80b2b40 0x7f42f80b5d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.808+0000 7f42fcf45700 1 -- 172.21.15.46:0/3798103400 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f0009ae0 con 0x7f42f80b1310 2024-01-26T23:30:08.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.808+0000 7f42fcf45700 1 --2- 172.21.15.46:0/3798103400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f80b1310 0x7f42f80b1770 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f42f80ae610 tx=0x7f42f0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.807+0000 7fb8c2959700 1 -- 172.21.15.46:0/2790029953 shutdown_connections 2024-01-26T23:30:08.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.808+0000 7fb8c2959700 1 -- 172.21.15.46:0/2790029953 wait complete. 2024-01-26T23:30:08.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.809+0000 7fb8c2959700 1 Processor -- start 2024-01-26T23:30:08.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.810+0000 7fe1fbfff700 1 --2- 172.21.15.46:0/3081585256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1fc068040 0x7fe1fc0cb1f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00c7c0 tx=0x7fe1ec007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.810+0000 7fe1e3fff700 1 -- 172.21.15.46:0/3081585256 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec005b30 con 0x7fe1fc068040 2024-01-26T23:30:08.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.810+0000 7fe1e3fff700 1 -- 172.21.15.46:0/3081585256 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ec005cb0 con 0x7fe1fc068040 2024-01-26T23:30:08.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.810+0000 7fe1e3fff700 1 -- 172.21.15.46:0/3081585256 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec004a80 con 0x7fe1fc068040 2024-01-26T23:30:08.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.811+0000 7f9da1f14700 1 -- 172.21.15.46:0/1869657807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9c04c710 con 0x7f9d9c15b700 2024-01-26T23:30:08.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.811+0000 7f9d9b7fe700 1 --2- 172.21.15.46:0/1869657807 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d88066f30 0x7f9d880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.811+0000 7f9d9b7fe700 1 --2- 172.21.15.46:0/1869657807 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d88066f30 0x7f9d880693e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9d9400acf0 tx=0x7f9d94009370 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.812+0000 7f42dffff700 1 -- 172.21.15.46:0/3798103400 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f0009300 con 0x7f42f80b1310 2024-01-26T23:30:08.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.812+0000 7f42dffff700 1 -- 172.21.15.46:0/3798103400 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f0012e60 con 0x7f42f80b1310 2024-01-26T23:30:08.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.812+0000 7f42dffff700 1 -- 172.21.15.46:0/3798103400 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f001e680 con 0x7f42f80b1310 2024-01-26T23:30:08.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.813+0000 7f42fe9a8700 1 -- 172.21.15.46:0/3798103400 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f80b3f20 con 0x7f42f80b1310 2024-01-26T23:30:08.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.813+0000 7f42fe9a8700 1 -- 172.21.15.46:0/3798103400 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f80b44a0 con 0x7f42f80b1310 2024-01-26T23:30:08.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.814+0000 7f42fe9a8700 1 -- 172.21.15.46:0/3798103400 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f804c710 con 0x7f42f80b1310 2024-01-26T23:30:08.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.815+0000 7f42dffff700 1 -- 172.21.15.46:0/3798103400 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f42f001ea30 con 0x7f42f80b1310 2024-01-26T23:30:08.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.816+0000 7f42dffff700 1 --2- 172.21.15.46:0/3798103400 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f42e4067080 0x7f42e4069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.816+0000 7f42dffff700 1 -- 172.21.15.46:0/3798103400 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f42f0086510 con 0x7f42f80b1310 2024-01-26T23:30:08.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.816+0000 7f42f7fff700 1 --2- 172.21.15.46:0/3798103400 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f42e4067080 0x7f42e4069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.814+0000 7fb8c2959700 1 -- start start 2024-01-26T23:30:08.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.815+0000 7fb8c2959700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8bc0a9840 0x7fb8bc13afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.815+0000 7fb8c2959700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8bc159800 0x7fb8bc142c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.815+0000 7fb8c2959700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8bc159bd0 0x7fb8bc1483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.815+0000 7fb8c2959700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc141bc0 con 0x7fb8bc159bd0 2024-01-26T23:30:08.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.815+0000 7fb8c2959700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc141ea0 con 0x7fb8bc0a9840 2024-01-26T23:30:08.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.815+0000 7fb8c2959700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc142180 con 0x7fb8bc159800 2024-01-26T23:30:08.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.815+0000 7fb8bb7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8bc159800 0x7fb8bc142c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.815+0000 7fb8bb7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8bc159800 0x7fb8bc142c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:57658/0 (socket says 172.21.15.46:57658) 2024-01-26T23:30:08.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.815+0000 7fb8bb7fe700 1 -- 172.21.15.46:0/625948241 learned_addr learned my addr 172.21.15.46:0/625948241 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:08.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.816+0000 7fb8bb7fe700 1 -- 172.21.15.46:0/625948241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8bc0a9840 msgr2=0x7fb8bc13afc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:30:08.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.816+0000 7fb8bb7fe700 1 --2- 172.21.15.46:0/625948241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8bc0a9840 0x7fb8bc13afc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.816+0000 7fb8bb7fe700 1 -- 172.21.15.46:0/625948241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8bc159bd0 msgr2=0x7fb8bc1483b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:30:08.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.816+0000 7fb8bb7fe700 1 --2- 172.21.15.46:0/625948241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8bc159bd0 0x7fb8bc1483b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.816+0000 7fb8bb7fe700 1 -- 172.21.15.46:0/625948241 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b4009a20 con 0x7fb8bc159800 2024-01-26T23:30:08.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.816+0000 7fb8bb7fe700 1 --2- 172.21.15.46:0/625948241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8bc159800 0x7fb8bc142c60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb8b40097c0 tx=0x7fb8b4018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.817+0000 7f42f7fff700 1 --2- 172.21.15.46:0/3798103400 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f42e4067080 0x7f42e4069530 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f42e8009740 tx=0x7f42e8006c50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.817+0000 7fe20226b700 1 -- 172.21.15.46:0/3081585256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1fc13dd00 con 0x7fe1fc068040 2024-01-26T23:30:08.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.817+0000 7fe20226b700 1 -- 172.21.15.46:0/3081585256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1fc13e2c0 con 0x7fe1fc068040 2024-01-26T23:30:08.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.818+0000 7fe20226b700 1 -- 172.21.15.46:0/3081585256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1fc13e4e0 con 0x7fe1fc068040 2024-01-26T23:30:08.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.820+0000 7f42dffff700 1 -- 172.21.15.46:0/3798103400 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f42f0053470 con 0x7f42f80b1310 2024-01-26T23:30:08.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.820+0000 7fb8abfff700 1 -- 172.21.15.46:0/625948241 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4012d50 con 0x7fb8bc159800 2024-01-26T23:30:08.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.820+0000 7fb8abfff700 1 -- 172.21.15.46:0/625948241 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b4027470 con 0x7fb8bc159800 2024-01-26T23:30:08.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.820+0000 7fb8abfff700 1 -- 172.21.15.46:0/625948241 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4027830 con 0x7fb8bc159800 2024-01-26T23:30:08.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.820+0000 7fb8c2959700 1 -- 172.21.15.46:0/625948241 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8bc142460 con 0x7fb8bc159800 2024-01-26T23:30:08.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.821+0000 7fb8c2959700 1 -- 172.21.15.46:0/625948241 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8bc13b500 con 0x7fb8bc159800 2024-01-26T23:30:08.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.822+0000 7f9d837fe700 1 -- 172.21.15.46:0/1869657807 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9d8c051bb0 con 0x7f9d9c15b700 2024-01-26T23:30:08.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.823+0000 7fb8c2959700 1 -- 172.21.15.46:0/625948241 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8bc04c710 con 0x7fb8bc159800 2024-01-26T23:30:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.827+0000 7fe1e3fff700 1 -- 172.21.15.46:0/3081585256 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe1ec0070f0 con 0x7fe1fc068040 2024-01-26T23:30:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.827+0000 7fe1e3fff700 1 --2- 172.21.15.46:0/3081585256 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe1e8066f30 0x7fe1e80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.828+0000 7fe1e3fff700 1 -- 172.21.15.46:0/3081585256 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fe1ec084180 con 0x7fe1fc068040 2024-01-26T23:30:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.832+0000 7fe1fb7fe700 1 --2- 172.21.15.46:0/3081585256 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe1e8066f30 0x7fe1e80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.832+0000 7fe1fb7fe700 1 --2- 172.21.15.46:0/3081585256 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe1e8066f30 0x7fe1e80693e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc068440 tx=0x7fe1f0017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.831+0000 7fb8abfff700 1 -- 172.21.15.46:0/625948241 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb8b4024070 con 0x7fb8bc159800 2024-01-26T23:30:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.831+0000 7fb8abfff700 1 --2- 172.21.15.46:0/625948241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8a0067050 0x7fb8a0069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.831+0000 7fb8abfff700 1 -- 172.21.15.46:0/625948241 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fb8b4086fd0 con 0x7fb8bc159800 2024-01-26T23:30:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.832+0000 7fb8abfff700 1 -- 172.21.15.46:0/625948241 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb8b408b5f0 con 0x7fb8bc159800 2024-01-26T23:30:08.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.837+0000 7fb8bbfff700 1 --2- 172.21.15.46:0/625948241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8a0067050 0x7fb8a0069500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.838+0000 7fe1e3fff700 1 -- 172.21.15.46:0/3081585256 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe1ec051160 con 0x7fe1fc068040 2024-01-26T23:30:08.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.856+0000 7fb8bbfff700 1 --2- 172.21.15.46:0/625948241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8a0067050 0x7fb8a0069500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0007fd0 tx=0x7fb8b0007a90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.857+0000 7f752a40d700 1 Processor -- start 2024-01-26T23:30:08.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.858+0000 7f752a40d700 1 -- start start 2024-01-26T23:30:08.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.859+0000 7f752a40d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7524156170 0x7f75240b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.859+0000 7f752a40d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f75240afbd0 0x7f75240b0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.859+0000 7f752a40d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7524157d50 con 0x7f7524156540 2024-01-26T23:30:08.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.859+0000 7f752a40d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75240af920 con 0x7f7524156170 2024-01-26T23:30:08.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.859+0000 7f752a40d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7524157140 con 0x7f75240afbd0 2024-01-26T23:30:08.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.860+0000 7f75289aa700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7524156540 0x7f7524156970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44538/0 (socket says 172.21.15.46:44538) 2024-01-26T23:30:08.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.860+0000 7f75289aa700 1 -- 172.21.15.46:0/1860872688 learned_addr learned my addr 172.21.15.46:0/1860872688 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:08.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.861+0000 7f7523fff700 1 --2- 172.21.15.46:0/1860872688 >> v2:172.21.15.46:3300/0 conn(0x7f7524156170 0x7f75240b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.861+0000 7f7522ffd700 1 -- 172.21.15.46:0/1860872688 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235850894 0 0) 0x7f7524157d50 con 0x7f7524156540 2024-01-26T23:30:08.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.861+0000 7f75237fe700 1 --2- 172.21.15.46:0/1860872688 >> v2:172.21.15.106:3300/0 conn(0x7f75240afbd0 0x7f75240b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.861+0000 7f7523fff700 1 -- 172.21.15.46:0/1860872688 >> v2:172.21.15.106:3300/0 conn(0x7f75240afbd0 msgr2=0x7f75240b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:08.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.861+0000 7f7523fff700 1 --2- 172.21.15.46:0/1860872688 >> v2:172.21.15.106:3300/0 conn(0x7f75240afbd0 0x7f75240b0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.861+0000 7f7523fff700 1 -- 172.21.15.46:0/1860872688 >> v1:172.21.15.46:6789/0 conn(0x7f7524156540 legacy=0x7f7524156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.861+0000 7f7523fff700 1 -- 172.21.15.46:0/1860872688 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7524157420 con 0x7f7524156170 2024-01-26T23:30:08.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.862+0000 7f7523fff700 1 --2- 172.21.15.46:0/1860872688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7524156170 0x7f75240b0130 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f75240cd170 tx=0x7f751c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca271a4471e2134 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.863+0000 7f7522ffd700 1 -- 172.21.15.46:0/1860872688 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751c00cc70 con 0x7f7524156170 2024-01-26T23:30:08.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.863+0000 7f7522ffd700 1 -- 172.21.15.46:0/1860872688 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751c00cdd0 con 0x7f7524156170 2024-01-26T23:30:08.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.863+0000 7f7522ffd700 1 -- 172.21.15.46:0/1860872688 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751c011650 con 0x7f7524156170 2024-01-26T23:30:08.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.864+0000 7f752a40d700 1 -- 172.21.15.46:0/1860872688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7524156170 msgr2=0x7f75240b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.864+0000 7f752a40d700 1 --2- 172.21.15.46:0/1860872688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7524156170 0x7f75240b0130 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f75240cd170 tx=0x7f751c009780 comp rx=0 tx=0).stop 2024-01-26T23:30:08.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.864+0000 7f752a40d700 1 -- 172.21.15.46:0/1860872688 shutdown_connections 2024-01-26T23:30:08.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.864+0000 7f752a40d700 1 --2- 172.21.15.46:0/1860872688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7524156170 0x7f75240b0130 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.864+0000 7f752a40d700 1 --2- 172.21.15.46:0/1860872688 >> v2:172.21.15.106:3300/0 conn(0x7f75240afbd0 0x7f75240b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.864+0000 7f752a40d700 1 -- 172.21.15.46:0/1860872688 >> 172.21.15.46:0/1860872688 conn(0x7f7524068400 msgr2=0x7f75240cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:08.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.864+0000 7f752a40d700 1 -- 172.21.15.46:0/1860872688 shutdown_connections 2024-01-26T23:30:08.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.864+0000 7f752a40d700 1 -- 172.21.15.46:0/1860872688 wait complete. 2024-01-26T23:30:08.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.866+0000 7f752a40d700 1 Processor -- start 2024-01-26T23:30:08.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.867+0000 7f752a40d700 1 -- start start 2024-01-26T23:30:08.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.868+0000 7f752a40d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75240afbd0 0x7f7524145150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.868+0000 7f752a40d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7524156170 0x7f75240c8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.868+0000 7f752a40d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7524156540 0x7f75240c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.868+0000 7f752a40d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75241494c0 con 0x7f7524156540 2024-01-26T23:30:08.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.868+0000 7f752a40d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75241488b0 con 0x7f75240afbd0 2024-01-26T23:30:08.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.868+0000 7f752a40d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7524148b90 con 0x7f7524156170 2024-01-26T23:30:08.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.869+0000 7f75237fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7524156170 0x7f75240c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.869+0000 7f7523fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75240afbd0 0x7f7524145150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.869+0000 7f75289aa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7524156540 0x7f75240c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.869+0000 7f75289aa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7524156540 0x7f75240c8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34126/0 (socket says 172.21.15.46:34126) 2024-01-26T23:30:08.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.869+0000 7f75289aa700 1 -- 172.21.15.46:0/862650700 learned_addr learned my addr 172.21.15.46:0/862650700 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:08.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.870+0000 7f75237fe700 1 -- 172.21.15.46:0/862650700 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75240afbd0 msgr2=0x7f7524145150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.870+0000 7f75237fe700 1 --2- 172.21.15.46:0/862650700 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75240afbd0 0x7f7524145150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.870+0000 7f75237fe700 1 -- 172.21.15.46:0/862650700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7524156540 msgr2=0x7f75240c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.870+0000 7f75237fe700 1 --2- 172.21.15.46:0/862650700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7524156540 0x7f75240c8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.870+0000 7f75237fe700 1 -- 172.21.15.46:0/862650700 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751c00d040 con 0x7f7524156170 2024-01-26T23:30:08.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.871+0000 7f75237fe700 1 --2- 172.21.15.46:0/862650700 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7524156170 0x7f75240c8800 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f751800c880 tx=0x7f7518008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.871+0000 7f7513fff700 1 -- 172.21.15.46:0/862650700 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7518011600 con 0x7f7524156170 2024-01-26T23:30:08.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.871+0000 7f7513fff700 1 -- 172.21.15.46:0/862650700 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7518011760 con 0x7f7524156170 2024-01-26T23:30:08.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.871+0000 7f7513fff700 1 -- 172.21.15.46:0/862650700 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7518011b00 con 0x7f7524156170 2024-01-26T23:30:08.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.871+0000 7f752a40d700 1 -- 172.21.15.46:0/862650700 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7524148e70 con 0x7f7524156170 2024-01-26T23:30:08.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.871+0000 7f752a40d700 1 -- 172.21.15.46:0/862650700 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7524145690 con 0x7f7524156170 2024-01-26T23:30:08.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.875+0000 7f7513fff700 1 -- 172.21.15.46:0/862650700 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f75180193b0 con 0x7f7524156170 2024-01-26T23:30:08.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.875+0000 7f7513fff700 1 --2- 172.21.15.46:0/862650700 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f750c067190 0x7f750c069640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.876+0000 7f7513fff700 1 -- 172.21.15.46:0/862650700 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f7518084640 con 0x7f7524156170 2024-01-26T23:30:08.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.877+0000 7f7523fff700 1 --2- 172.21.15.46:0/862650700 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f750c067190 0x7f750c069640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.878+0000 7f752a40d700 1 -- 172.21.15.46:0/862650700 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75000052b0 con 0x7f7524156170 2024-01-26T23:30:08.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.878+0000 7f7523fff700 1 --2- 172.21.15.46:0/862650700 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f750c067190 0x7f750c069640 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f751c00c130 tx=0x7f751c01a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.883+0000 7f7513fff700 1 -- 172.21.15.46:0/862650700 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7518055030 con 0x7f7524156170 2024-01-26T23:30:08.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.894+0000 7ff5a4bab700 1 Processor -- start 2024-01-26T23:30:08.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.894+0000 7ff5a4bab700 1 -- start start 2024-01-26T23:30:08.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.896+0000 7ff5a4bab700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff5a006ed00 0x7ff5a00b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.896+0000 7ff5a4bab700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a00b0ad0 con 0x7ff5a00b0670 2024-01-26T23:30:08.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.896+0000 7ff5a4bab700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a0157d50 con 0x7ff5a00afbd0 2024-01-26T23:30:08.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.896+0000 7ff5a4bab700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a015a450 con 0x7ff5a006ed00 2024-01-26T23:30:08.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.897+0000 7ff59dd9b700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7ff5a00afbd0 0x7ff5a0159630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:51748/0 (socket says 172.21.15.46:51748) 2024-01-26T23:30:08.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.897+0000 7ff59dd9b700 1 -- 172.21.15.46:0/129202669 learned_addr learned my addr 172.21.15.46:0/129202669 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:08.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.897+0000 7ff59e59c700 1 --2- 172.21.15.46:0/129202669 >> v2:172.21.15.46:3300/0 conn(0x7ff5a006ed00 0x7ff5a00b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.898+0000 7ff59e59c700 1 -- 172.21.15.46:0/129202669 >> v1:172.21.15.106:6789/0 conn(0x7ff5a00afbd0 legacy=0x7ff5a0159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.898+0000 7ff59e59c700 1 -- 172.21.15.46:0/129202669 >> v1:172.21.15.46:6789/0 conn(0x7ff5a00b0670 legacy=0x7ff5a0159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.898+0000 7ff59e59c700 1 -- 172.21.15.46:0/129202669 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a015c9c0 con 0x7ff5a006ed00 2024-01-26T23:30:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.898+0000 7ff59e59c700 1 --2- 172.21.15.46:0/129202669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a006ed00 0x7ff5a00b0130 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff5940098e0 tx=0x7ff5940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196ed3d3106e8fbb server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.905+0000 7ff59d59a700 1 -- 172.21.15.46:0/129202669 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff594009e80 con 0x7ff5a006ed00 2024-01-26T23:30:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.906+0000 7ff59d59a700 1 -- 172.21.15.46:0/129202669 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59400b760 con 0x7ff5a006ed00 2024-01-26T23:30:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.906+0000 7ff59d59a700 1 -- 172.21.15.46:0/129202669 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59400bb20 con 0x7ff5a006ed00 2024-01-26T23:30:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.907+0000 7ff5a4bab700 1 -- 172.21.15.46:0/129202669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a006ed00 msgr2=0x7ff5a00b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.907+0000 7ff5a4bab700 1 --2- 172.21.15.46:0/129202669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a006ed00 0x7ff5a00b0130 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff5940098e0 tx=0x7ff5940091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.907+0000 7ff5a4bab700 1 -- 172.21.15.46:0/129202669 shutdown_connections 2024-01-26T23:30:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.907+0000 7ff5a4bab700 1 --2- 172.21.15.46:0/129202669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a006ed00 0x7ff5a00b0130 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.907+0000 7ff5a4bab700 1 -- 172.21.15.46:0/129202669 >> 172.21.15.46:0/129202669 conn(0x7ff5a0068400 msgr2=0x7ff5a00cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.908+0000 7ff5a4bab700 1 -- 172.21.15.46:0/129202669 shutdown_connections 2024-01-26T23:30:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.908+0000 7ff5a4bab700 1 -- 172.21.15.46:0/129202669 wait complete. 2024-01-26T23:30:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.910+0000 7ff5a4bab700 1 Processor -- start 2024-01-26T23:30:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.910+0000 7ff5a4bab700 1 -- start start 2024-01-26T23:30:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.911+0000 7ff5a4bab700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5a006ed00 0x7ff5a01885d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.911+0000 7ff5a4bab700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5a00afbd0 0x7ff5a015f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.911+0000 7ff5a4bab700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a00b0670 0x7ff5a01615f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.911+0000 7ff5a4bab700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a0160a50 con 0x7ff5a00b0670 2024-01-26T23:30:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.911+0000 7ff5a4bab700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a015f780 con 0x7ff5a00afbd0 2024-01-26T23:30:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.911+0000 7ff5a4bab700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a015f9b0 con 0x7ff5a006ed00 2024-01-26T23:30:08.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.912+0000 7ff59ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a00b0670 0x7ff5a01615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.912+0000 7ff59ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a00b0670 0x7ff5a01615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34148/0 (socket says 172.21.15.46:34148) 2024-01-26T23:30:08.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.912+0000 7ff59ed9d700 1 -- 172.21.15.46:0/1775953063 learned_addr learned my addr 172.21.15.46:0/1775953063 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:08.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.913+0000 7ff59dd9b700 1 --2- 172.21.15.46:0/1775953063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5a00afbd0 0x7ff5a015f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.913+0000 7ff59ed9d700 1 -- 172.21.15.46:0/1775953063 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5a006ed00 msgr2=0x7ff5a01885d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:30:08.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.913+0000 7ff59ed9d700 1 --2- 172.21.15.46:0/1775953063 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5a006ed00 0x7ff5a01885d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.913+0000 7ff59ed9d700 1 -- 172.21.15.46:0/1775953063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5a00afbd0 msgr2=0x7ff5a015f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:08.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.913+0000 7ff59ed9d700 1 --2- 172.21.15.46:0/1775953063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5a00afbd0 0x7ff5a015f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:08.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.913+0000 7ff59ed9d700 1 -- 172.21.15.46:0/1775953063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff594009420 con 0x7ff5a00b0670 2024-01-26T23:30:08.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.913+0000 7ff59ed9d700 1 --2- 172.21.15.46:0/1775953063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a00b0670 0x7ff5a01615f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff59000c760 tx=0x7ff590007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.917+0000 7ff58e7fc700 1 -- 172.21.15.46:0/1775953063 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff590012630 con 0x7ff5a00b0670 2024-01-26T23:30:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.917+0000 7ff58e7fc700 1 -- 172.21.15.46:0/1775953063 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff590012790 con 0x7ff5a00b0670 2024-01-26T23:30:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.917+0000 7ff58e7fc700 1 -- 172.21.15.46:0/1775953063 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff590012b50 con 0x7ff5a00b0670 2024-01-26T23:30:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.917+0000 7ff5a4bab700 1 -- 172.21.15.46:0/1775953063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a015fc90 con 0x7ff5a00b0670 2024-01-26T23:30:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.917+0000 7ff5a4bab700 1 -- 172.21.15.46:0/1775953063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a0160250 con 0x7ff5a00b0670 2024-01-26T23:30:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.921+0000 7ff58e7fc700 1 -- 172.21.15.46:0/1775953063 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff59001b400 con 0x7ff5a00b0670 2024-01-26T23:30:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.922+0000 7ff58e7fc700 1 --2- 172.21.15.46:0/1775953063 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff580067080 0x7ff580069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.922+0000 7ff58e7fc700 1 -- 172.21.15.46:0/1775953063 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7ff5900843b0 con 0x7ff5a00b0670 2024-01-26T23:30:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.927+0000 7ff5a4bab700 1 -- 172.21.15.46:0/1775953063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5840052b0 con 0x7ff5a00b0670 2024-01-26T23:30:08.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.932+0000 7ff59e59c700 1 --2- 172.21.15.46:0/1775953063 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff580067080 0x7ff580069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:08.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.937+0000 7ff58e7fc700 1 -- 172.21.15.46:0/1775953063 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff590051310 con 0x7ff5a00b0670 2024-01-26T23:30:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:08.971+0000 7ff59e59c700 1 --2- 172.21.15.46:0/1775953063 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff580067080 0x7ff580069530 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff5940098e0 tx=0x7ff594016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:09.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.169+0000 7ff031dbd700 1 -- 172.21.15.46:0/4082350573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7ff008005560 con 0x7ff02c166520 2024-01-26T23:30:09.198 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-01-26T23:30:09.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.187+0000 7ff01b7fe700 1 -- 172.21.15.46:0/4082350573 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff01c028090 con 0x7ff02c166520 2024-01-26T23:30:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.199+0000 7ff031dbd700 1 -- 172.21.15.46:0/4082350573 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff014067520 msgr2=0x7ff0140699d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.199+0000 7ff031dbd700 1 --2- 172.21.15.46:0/4082350573 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff014067520 0x7ff0140699d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0aebf0 tx=0x7ff024009370 comp rx=0 tx=0).stop 2024-01-26T23:30:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.199+0000 7ff031dbd700 1 -- 172.21.15.46:0/4082350573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02c166520 msgr2=0x7ff02c16c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.199+0000 7ff031dbd700 1 --2- 172.21.15.46:0/4082350573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02c166520 0x7ff02c16c360 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff01c01b000 tx=0x7ff01c01c000 comp rx=0 tx=0).stop 2024-01-26T23:30:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.199+0000 7ff031dbd700 1 -- 172.21.15.46:0/4082350573 shutdown_connections 2024-01-26T23:30:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.199+0000 7ff031dbd700 1 --2- 172.21.15.46:0/4082350573 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff014067520 0x7ff0140699d0 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0aebf0 tx=0x7ff024009370 comp rx=0 tx=0).stop 2024-01-26T23:30:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.199+0000 7ff031dbd700 1 --2- 172.21.15.46:0/4082350573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02c166520 0x7ff02c16c360 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.199+0000 7ff031dbd700 1 --2- 172.21.15.46:0/4082350573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff02c1575f0 0x7ff02c086240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.199+0000 7ff031dbd700 1 --2- 172.21.15.46:0/4082350573 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff02c1668f0 0x7ff02c17aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.199+0000 7ff031dbd700 1 -- 172.21.15.46:0/4082350573 >> 172.21.15.46:0/4082350573 conn(0x7ff02c068400 msgr2=0x7ff02c15da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:09.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.200+0000 7ff031dbd700 1 -- 172.21.15.46:0/4082350573 shutdown_connections 2024-01-26T23:30:09.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.200+0000 7ff031dbd700 1 -- 172.21.15.46:0/4082350573 wait complete. 2024-01-26T23:30:09.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.214+0000 7fc5662dd700 1 -- 172.21.15.46:0/3023354000 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc5600b0630 con 0x7fc5600b0e70 2024-01-26T23:30:09.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.215+0000 7fc547fff700 1 -- 172.21.15.46:0/3023354000 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc550020090 con 0x7fc5600b0e70 2024-01-26T23:30:09.217 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2024-01-26T23:30:09.220 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2024-01-26T23:30:09.221 DEBUG:teuthology.parallel:result is None 2024-01-26T23:30:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.219+0000 7fc5662dd700 1 -- 172.21.15.46:0/3023354000 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc54c066fb0 msgr2=0x7fc54c069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.219+0000 7fc5662dd700 1 --2- 172.21.15.46:0/3023354000 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc54c066fb0 0x7fc54c069460 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc5540099e0 tx=0x7fc554016000 comp rx=0 tx=0).stop 2024-01-26T23:30:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.219+0000 7fc5662dd700 1 -- 172.21.15.46:0/3023354000 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5600b0e70 msgr2=0x7fc560160780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.219+0000 7fc5662dd700 1 --2- 172.21.15.46:0/3023354000 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5600b0e70 0x7fc560160780 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc55000e8d0 tx=0x7fc550008040 comp rx=0 tx=0).stop 2024-01-26T23:30:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.220+0000 7fc5662dd700 1 -- 172.21.15.46:0/3023354000 shutdown_connections 2024-01-26T23:30:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.220+0000 7fc5662dd700 1 --2- 172.21.15.46:0/3023354000 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc54c066fb0 0x7fc54c069460 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.220+0000 7fc5662dd700 1 --2- 172.21.15.46:0/3023354000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc560067e70 0x7fc560158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.220+0000 7fc5662dd700 1 --2- 172.21.15.46:0/3023354000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc56015dad0 0x7fc560160cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.220+0000 7fc5662dd700 1 --2- 172.21.15.46:0/3023354000 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5600b0e70 0x7fc560160780 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.220+0000 7fc5662dd700 1 -- 172.21.15.46:0/3023354000 >> 172.21.15.46:0/3023354000 conn(0x7fc56006b630 msgr2=0x7fc56015f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:09.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.220+0000 7fc5662dd700 1 -- 172.21.15.46:0/3023354000 shutdown_connections 2024-01-26T23:30:09.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.220+0000 7fc5662dd700 1 -- 172.21.15.46:0/3023354000 wait complete. 2024-01-26T23:30:09.234 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2024-01-26T23:30:09.235 DEBUG:teuthology.parallel:result is None 2024-01-26T23:30:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.248+0000 7f42fe9a8700 1 -- 172.21.15.46:0/3798103400 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f42f80cdad0 con 0x7f42f80b1310 2024-01-26T23:30:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.249+0000 7f42dffff700 1 -- 172.21.15.46:0/3798103400 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f42f0056a90 con 0x7f42f80b1310 2024-01-26T23:30:09.249 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2024-01-26T23:30:09.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.251+0000 7f9da1f14700 1 -- 172.21.15.46:0/1869657807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9d9c0b2380 con 0x7f9d9c15b700 2024-01-26T23:30:09.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.252+0000 7f9d837fe700 1 -- 172.21.15.46:0/1869657807 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9d8c017090 con 0x7f9d9c15b700 2024-01-26T23:30:09.252 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-01-26T23:30:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f42fe9a8700 1 -- 172.21.15.46:0/3798103400 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f42e4067080 msgr2=0x7f42e4069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f42fe9a8700 1 --2- 172.21.15.46:0/3798103400 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f42e4067080 0x7f42e4069530 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f42e8009740 tx=0x7f42e8006c50 comp rx=0 tx=0).stop 2024-01-26T23:30:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f42fe9a8700 1 -- 172.21.15.46:0/3798103400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f80b1310 msgr2=0x7f42f80b1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f42fe9a8700 1 --2- 172.21.15.46:0/3798103400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f80b1310 0x7f42f80b1770 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f42f80ae610 tx=0x7f42f0017000 comp rx=0 tx=0).stop 2024-01-26T23:30:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f9da1f14700 1 -- 172.21.15.46:0/1869657807 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d88066f30 msgr2=0x7f9d880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f9da1f14700 1 --2- 172.21.15.46:0/1869657807 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d88066f30 0x7f9d880693e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9d9400acf0 tx=0x7f9d94009370 comp rx=0 tx=0).stop 2024-01-26T23:30:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f9da1f14700 1 -- 172.21.15.46:0/1869657807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c15b700 msgr2=0x7f9d9c16e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f9da1f14700 1 --2- 172.21.15.46:0/1869657807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c15b700 0x7f9d9c16e670 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00ea50 tx=0x7f9d8c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f42fe9a8700 1 -- 172.21.15.46:0/3798103400 shutdown_connections 2024-01-26T23:30:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f42fe9a8700 1 --2- 172.21.15.46:0/3798103400 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f42e4067080 0x7f42e4069530 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f42fe9a8700 1 --2- 172.21.15.46:0/3798103400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f80b1310 0x7f42f80b1770 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f42fe9a8700 1 --2- 172.21.15.46:0/3798103400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42f80ae960 0x7f42f81589e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f42fe9a8700 1 --2- 172.21.15.46:0/3798103400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f80b2b40 0x7f42f80b5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f42fe9a8700 1 -- 172.21.15.46:0/3798103400 >> 172.21.15.46:0/3798103400 conn(0x7f42f8099a30 msgr2=0x7f42f80c5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f9da1f14700 1 -- 172.21.15.46:0/1869657807 shutdown_connections 2024-01-26T23:30:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f9da1f14700 1 --2- 172.21.15.46:0/1869657807 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d88066f30 0x7f9d880693e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f9da1f14700 1 --2- 172.21.15.46:0/1869657807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c15b700 0x7f9d9c16e670 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f9da1f14700 1 --2- 172.21.15.46:0/1869657807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d9c0af9a0 0x7f9d9c163a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f9da1f14700 1 --2- 172.21.15.46:0/1869657807 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d9c0b1cc0 0x7f9d9c0a64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f9da1f14700 1 -- 172.21.15.46:0/1869657807 >> 172.21.15.46:0/1869657807 conn(0x7f9d9c068400 msgr2=0x7f9d9c159e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.255+0000 7f9da1f14700 1 -- 172.21.15.46:0/1869657807 shutdown_connections 2024-01-26T23:30:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.256+0000 7f9da1f14700 1 -- 172.21.15.46:0/1869657807 wait complete. 2024-01-26T23:30:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.256+0000 7f42fe9a8700 1 -- 172.21.15.46:0/3798103400 shutdown_connections 2024-01-26T23:30:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.256+0000 7f42fe9a8700 1 -- 172.21.15.46:0/3798103400 wait complete. 2024-01-26T23:30:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.263+0000 7fb8c2959700 1 -- 172.21.15.46:0/625948241 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fb8bc0aa300 con 0x7fb8bc159800 2024-01-26T23:30:09.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.263+0000 7fb8abfff700 1 -- 172.21.15.46:0/625948241 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb8b4053f30 con 0x7fb8bc159800 2024-01-26T23:30:09.265 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-01-26T23:30:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.267+0000 7fb8c2959700 1 -- 172.21.15.46:0/625948241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8a0067050 msgr2=0x7fb8a0069500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.267+0000 7fb8c2959700 1 --2- 172.21.15.46:0/625948241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8a0067050 0x7fb8a0069500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0007fd0 tx=0x7fb8b0007a90 comp rx=0 tx=0).stop 2024-01-26T23:30:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.267+0000 7fb8c2959700 1 -- 172.21.15.46:0/625948241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8bc159800 msgr2=0x7fb8bc142c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.267+0000 7fb8c2959700 1 --2- 172.21.15.46:0/625948241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8bc159800 0x7fb8bc142c60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb8b40097c0 tx=0x7fb8b4018000 comp rx=0 tx=0).stop 2024-01-26T23:30:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.267+0000 7fb8c2959700 1 -- 172.21.15.46:0/625948241 shutdown_connections 2024-01-26T23:30:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.267+0000 7fb8c2959700 1 --2- 172.21.15.46:0/625948241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8a0067050 0x7fb8a0069500 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.267+0000 7fb8c2959700 1 --2- 172.21.15.46:0/625948241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8bc159bd0 0x7fb8bc1483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.267+0000 7fb8c2959700 1 --2- 172.21.15.46:0/625948241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8bc0a9840 0x7fb8bc13afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.267+0000 7fb8c2959700 1 --2- 172.21.15.46:0/625948241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8bc159800 0x7fb8bc142c60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.267+0000 7fb8c2959700 1 -- 172.21.15.46:0/625948241 >> 172.21.15.46:0/625948241 conn(0x7fb8bc068400 msgr2=0x7fb8bc131920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.267+0000 7fb8c2959700 1 -- 172.21.15.46:0/625948241 shutdown_connections 2024-01-26T23:30:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.267+0000 7fb8c2959700 1 -- 172.21.15.46:0/625948241 wait complete. 2024-01-26T23:30:09.274 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.1 2024-01-26T23:30:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.273+0000 7fe20226b700 1 -- 172.21.15.46:0/3081585256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe1fc04c710 con 0x7fe1fc068040 2024-01-26T23:30:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.274+0000 7fe1e3fff700 1 -- 172.21.15.46:0/3081585256 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe1ec018090 con 0x7fe1fc068040 2024-01-26T23:30:09.275 DEBUG:teuthology.parallel:result is None 2024-01-26T23:30:09.275 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2024-01-26T23:30:09.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.278+0000 7fe20226b700 1 -- 172.21.15.46:0/3081585256 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe1e8066f30 msgr2=0x7fe1e80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.278+0000 7fe20226b700 1 --2- 172.21.15.46:0/3081585256 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe1e8066f30 0x7fe1e80693e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc068440 tx=0x7fe1f0017000 comp rx=0 tx=0).stop 2024-01-26T23:30:09.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.278+0000 7fe20226b700 1 -- 172.21.15.46:0/3081585256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1fc068040 msgr2=0x7fe1fc0cb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.278+0000 7fe20226b700 1 --2- 172.21.15.46:0/3081585256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1fc068040 0x7fe1fc0cb1f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00c7c0 tx=0x7fe1ec007f70 comp rx=0 tx=0).stop 2024-01-26T23:30:09.280 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2024-01-26T23:30:09.280 DEBUG:teuthology.parallel:result is None 2024-01-26T23:30:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.281+0000 7fe20226b700 1 -- 172.21.15.46:0/3081585256 shutdown_connections 2024-01-26T23:30:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.281+0000 7fe20226b700 1 --2- 172.21.15.46:0/3081585256 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe1e8066f30 0x7fe1e80693e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.281+0000 7fe20226b700 1 --2- 172.21.15.46:0/3081585256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1fc068040 0x7fe1fc0cb1f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.281+0000 7fe20226b700 1 --2- 172.21.15.46:0/3081585256 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1fc15dad0 0x7fe1fc14db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.281+0000 7fe20226b700 1 --2- 172.21.15.46:0/3081585256 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe1fc0b0e70 0x7fe1fc13ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.281+0000 7fe20226b700 1 -- 172.21.15.46:0/3081585256 >> 172.21.15.46:0/3081585256 conn(0x7fe1fc06b630 msgr2=0x7fe1fc0ccc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:09.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.281+0000 7fe20226b700 1 -- 172.21.15.46:0/3081585256 shutdown_connections 2024-01-26T23:30:09.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.281+0000 7fe20226b700 1 -- 172.21.15.46:0/3081585256 wait complete. 2024-01-26T23:30:09.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.288+0000 7f752a40d700 1 -- 172.21.15.46:0/862650700 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f7500005560 con 0x7f7524156170 2024-01-26T23:30:09.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.288+0000 7f7513fff700 1 -- 172.21.15.46:0/862650700 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7518015090 con 0x7f7524156170 2024-01-26T23:30:09.289 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-01-26T23:30:09.290 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.5 2024-01-26T23:30:09.290 DEBUG:teuthology.parallel:result is None 2024-01-26T23:30:09.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.291+0000 7f752a40d700 1 -- 172.21.15.46:0/862650700 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f750c067190 msgr2=0x7f750c069640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.291+0000 7f752a40d700 1 --2- 172.21.15.46:0/862650700 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f750c067190 0x7f750c069640 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f751c00c130 tx=0x7f751c01a000 comp rx=0 tx=0).stop 2024-01-26T23:30:09.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.291+0000 7f752a40d700 1 -- 172.21.15.46:0/862650700 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7524156170 msgr2=0x7f75240c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.291+0000 7f752a40d700 1 --2- 172.21.15.46:0/862650700 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7524156170 0x7f75240c8800 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f751800c880 tx=0x7f7518008040 comp rx=0 tx=0).stop 2024-01-26T23:30:09.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.291+0000 7f752a40d700 1 -- 172.21.15.46:0/862650700 shutdown_connections 2024-01-26T23:30:09.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.291+0000 7f752a40d700 1 --2- 172.21.15.46:0/862650700 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f750c067190 0x7f750c069640 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.291+0000 7f752a40d700 1 --2- 172.21.15.46:0/862650700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7524156540 0x7f75240c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.291+0000 7f752a40d700 1 --2- 172.21.15.46:0/862650700 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75240afbd0 0x7f7524145150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.291+0000 7f752a40d700 1 --2- 172.21.15.46:0/862650700 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7524156170 0x7f75240c8800 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.291+0000 7f752a40d700 1 -- 172.21.15.46:0/862650700 >> 172.21.15.46:0/862650700 conn(0x7f7524068400 msgr2=0x7f7524069c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:09.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.291+0000 7f752a40d700 1 -- 172.21.15.46:0/862650700 shutdown_connections 2024-01-26T23:30:09.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.291+0000 7f752a40d700 1 -- 172.21.15.46:0/862650700 wait complete. 2024-01-26T23:30:09.299 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.0 2024-01-26T23:30:09.299 DEBUG:teuthology.parallel:result is None 2024-01-26T23:30:09.306 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.7 2024-01-26T23:30:09.306 DEBUG:teuthology.parallel:result is None 2024-01-26T23:30:09.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.315+0000 7ff5a4bab700 1 -- 172.21.15.46:0/1775953063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff584005560 con 0x7ff5a00b0670 2024-01-26T23:30:09.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.315+0000 7ff58e7fc700 1 -- 172.21.15.46:0/1775953063 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff590054930 con 0x7ff5a00b0670 2024-01-26T23:30:09.316 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-01-26T23:30:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.317+0000 7ff5a4bab700 1 -- 172.21.15.46:0/1775953063 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff580067080 msgr2=0x7ff580069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.317+0000 7ff5a4bab700 1 --2- 172.21.15.46:0/1775953063 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff580067080 0x7ff580069530 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff5940098e0 tx=0x7ff594016000 comp rx=0 tx=0).stop 2024-01-26T23:30:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.318+0000 7ff5a4bab700 1 -- 172.21.15.46:0/1775953063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a00b0670 msgr2=0x7ff5a01615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.318+0000 7ff5a4bab700 1 --2- 172.21.15.46:0/1775953063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a00b0670 0x7ff5a01615f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff59000c760 tx=0x7ff590007f30 comp rx=0 tx=0).stop 2024-01-26T23:30:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.318+0000 7ff5a4bab700 1 -- 172.21.15.46:0/1775953063 shutdown_connections 2024-01-26T23:30:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.318+0000 7ff5a4bab700 1 --2- 172.21.15.46:0/1775953063 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff580067080 0x7ff580069530 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.318+0000 7ff5a4bab700 1 --2- 172.21.15.46:0/1775953063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a00b0670 0x7ff5a01615f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.318+0000 7ff5a4bab700 1 --2- 172.21.15.46:0/1775953063 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5a006ed00 0x7ff5a01885d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.318+0000 7ff5a4bab700 1 --2- 172.21.15.46:0/1775953063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5a00afbd0 0x7ff5a015f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.318+0000 7ff5a4bab700 1 -- 172.21.15.46:0/1775953063 >> 172.21.15.46:0/1775953063 conn(0x7ff5a0068400 msgr2=0x7ff5a0158b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.318+0000 7ff5a4bab700 1 -- 172.21.15.46:0/1775953063 shutdown_connections 2024-01-26T23:30:09.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.318+0000 7ff5a4bab700 1 -- 172.21.15.46:0/1775953063 wait complete. 2024-01-26T23:30:09.332 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.4 2024-01-26T23:30:09.332 DEBUG:teuthology.parallel:result is None 2024-01-26T23:30:09.332 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-26T23:30:09.332 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:30:09.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.506+0000 7f4afc977700 1 Processor -- start 2024-01-26T23:30:09.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.506+0000 7f4afc977700 1 -- start start 2024-01-26T23:30:09.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.507+0000 7f4afc977700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4af8155af0 0x7f4af806b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:09.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.507+0000 7f4afc977700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4af80cba30 0x7f4af80c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:09.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.507+0000 7f4afc977700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af80c7e10 con 0x7f4af80c8a60 2024-01-26T23:30:09.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.507+0000 7f4afc977700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4af806a680 con 0x7f4af8155af0 2024-01-26T23:30:09.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.507+0000 7f4afc977700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af806a960 con 0x7f4af80cba30 2024-01-26T23:30:09.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.507+0000 7f4af659c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4af8155af0 0x7f4af806b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:09.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.507+0000 7f4af6d9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4af80cba30 0x7f4af80c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:09.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.507+0000 7f4af5d9b700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f4af80c8a60 0x7f4af80cb320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:51750/0 (socket says 172.21.15.46:51750) 2024-01-26T23:30:09.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.507+0000 7f4af6d9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4af80cba30 0x7f4af80c8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:57674/0 (socket says 172.21.15.46:57674) 2024-01-26T23:30:09.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.507+0000 7f4af659c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4af8155af0 0x7f4af806b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56760/0 (socket says 172.21.15.46:56760) 2024-01-26T23:30:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.507+0000 7f4af5d9b700 1 -- 172.21.15.46:0/950757196 learned_addr learned my addr 172.21.15.46:0/950757196 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.508+0000 7f4af6d9d700 1 -- 172.21.15.46:0/950757196 >> v2:172.21.15.106:3300/0 conn(0x7f4af8155af0 msgr2=0x7f4af806b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.508+0000 7f4af6d9d700 1 --2- 172.21.15.46:0/950757196 >> v2:172.21.15.106:3300/0 conn(0x7f4af8155af0 0x7f4af806b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.508+0000 7f4af6d9d700 1 -- 172.21.15.46:0/950757196 >> v1:172.21.15.106:6789/0 conn(0x7f4af80c8a60 legacy=0x7f4af80cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.508+0000 7f4af6d9d700 1 -- 172.21.15.46:0/950757196 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af806ac40 con 0x7f4af80cba30 2024-01-26T23:30:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.508+0000 7f4af659c700 1 --2- 172.21.15.46:0/950757196 >> v2:172.21.15.106:3300/0 conn(0x7f4af8155af0 0x7f4af806b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.508+0000 7f4af6d9d700 1 --2- 172.21.15.46:0/950757196 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4af80cba30 0x7f4af80c8ed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4af0009e60 tx=0x7f4af0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59b4d42076d35582 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.508+0000 7f4af559a700 1 -- 172.21.15.46:0/950757196 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af000e070 con 0x7f4af80cba30 2024-01-26T23:30:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.508+0000 7f4af559a700 1 -- 172.21.15.46:0/950757196 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af000bb60 con 0x7f4af80cba30 2024-01-26T23:30:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.508+0000 7f4af559a700 1 -- 172.21.15.46:0/950757196 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af0012430 con 0x7f4af80cba30 2024-01-26T23:30:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.509+0000 7f4afc977700 1 -- 172.21.15.46:0/950757196 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4af80cba30 msgr2=0x7f4af80c8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.509+0000 7f4afc977700 1 --2- 172.21.15.46:0/950757196 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4af80cba30 0x7f4af80c8ed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4af0009e60 tx=0x7f4af0009780 comp rx=0 tx=0).stop 2024-01-26T23:30:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.509+0000 7f4afc977700 1 -- 172.21.15.46:0/950757196 shutdown_connections 2024-01-26T23:30:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.509+0000 7f4afc977700 1 --2- 172.21.15.46:0/950757196 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4af80cba30 0x7f4af80c8ed0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.509+0000 7f4afc977700 1 --2- 172.21.15.46:0/950757196 >> v2:172.21.15.106:3300/0 conn(0x7f4af8155af0 0x7f4af806b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.509+0000 7f4afc977700 1 -- 172.21.15.46:0/950757196 >> 172.21.15.46:0/950757196 conn(0x7f4af80cd6e0 msgr2=0x7f4af80c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.509+0000 7f4afc977700 1 -- 172.21.15.46:0/950757196 shutdown_connections 2024-01-26T23:30:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.509+0000 7f4afc977700 1 -- 172.21.15.46:0/950757196 wait complete. 2024-01-26T23:30:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.510+0000 7f4afc977700 1 Processor -- start 2024-01-26T23:30:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.510+0000 7f4afc977700 1 -- start start 2024-01-26T23:30:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.511+0000 7f4afc977700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af80c8a60 0x7f4af81500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.511+0000 7f4afc977700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4af80cba30 0x7f4af8154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.511+0000 7f4afc977700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4af8155af0 0x7f4af815bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.511+0000 7f4afc977700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af8154ec0 con 0x7f4af80c8a60 2024-01-26T23:30:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.511+0000 7f4afc977700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af80b29a0 con 0x7f4af8155af0 2024-01-26T23:30:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.511+0000 7f4afc977700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af8154be0 con 0x7f4af80cba30 2024-01-26T23:30:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.511+0000 7f4af659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af80c8a60 0x7f4af81500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.511+0000 7f4af659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af80c8a60 0x7f4af81500e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34150/0 (socket says 172.21.15.46:34150) 2024-01-26T23:30:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.511+0000 7f4af659c700 1 -- 172.21.15.46:0/1796769978 learned_addr learned my addr 172.21.15.46:0/1796769978 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.511+0000 7f4af5d9b700 1 --2- 172.21.15.46:0/1796769978 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4af80cba30 0x7f4af8154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.511+0000 7f4af6d9d700 1 --2- 172.21.15.46:0/1796769978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4af8155af0 0x7f4af815bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.511+0000 7f4af659c700 1 -- 172.21.15.46:0/1796769978 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4af80cba30 msgr2=0x7f4af8154190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.511+0000 7f4af659c700 1 --2- 172.21.15.46:0/1796769978 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4af80cba30 0x7f4af8154190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.511+0000 7f4af659c700 1 -- 172.21.15.46:0/1796769978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4af8155af0 msgr2=0x7f4af815bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.511+0000 7f4af659c700 1 --2- 172.21.15.46:0/1796769978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4af8155af0 0x7f4af815bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.511+0000 7f4af659c700 1 -- 172.21.15.46:0/1796769978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae800a5d0 con 0x7f4af80c8a60 2024-01-26T23:30:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.511+0000 7f4af6d9d700 1 --2- 172.21.15.46:0/1796769978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4af8155af0 0x7f4af815bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.512+0000 7f4af659c700 1 --2- 172.21.15.46:0/1796769978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af80c8a60 0x7f4af81500e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800ea10 tx=0x7f4ae800a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.512+0000 7f4ae67fc700 1 -- 172.21.15.46:0/1796769978 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae800c950 con 0x7f4af80c8a60 2024-01-26T23:30:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.512+0000 7f4ae67fc700 1 -- 172.21.15.46:0/1796769978 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae800cab0 con 0x7f4af80c8a60 2024-01-26T23:30:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.512+0000 7f4afc977700 1 -- 172.21.15.46:0/1796769978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af0009a20 con 0x7f4af80c8a60 2024-01-26T23:30:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.512+0000 7f4ae67fc700 1 -- 172.21.15.46:0/1796769978 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae800ce30 con 0x7f4af80c8a60 2024-01-26T23:30:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.512+0000 7f4afc977700 1 -- 172.21.15.46:0/1796769978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af80c1e30 con 0x7f4af80c8a60 2024-01-26T23:30:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.513+0000 7f4afc977700 1 -- 172.21.15.46:0/1796769978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af80c19b0 con 0x7f4af80c8a60 2024-01-26T23:30:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.513+0000 7f4ae67fc700 1 -- 172.21.15.46:0/1796769978 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4ae80113f0 con 0x7f4af80c8a60 2024-01-26T23:30:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.514+0000 7f4ae67fc700 1 --2- 172.21.15.46:0/1796769978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4ad8066f30 0x7f4ad80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.514+0000 7f4ae67fc700 1 -- 172.21.15.46:0/1796769978 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f4ae8015030 con 0x7f4af80c8a60 2024-01-26T23:30:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.514+0000 7f4af5d9b700 1 --2- 172.21.15.46:0/1796769978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4ad8066f30 0x7f4ad80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.515+0000 7f4af5d9b700 1 --2- 172.21.15.46:0/1796769978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4ad8066f30 0x7f4ad80693e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4af8152300 tx=0x7f4aec007eb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:09.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.518+0000 7f4ae67fc700 1 -- 172.21.15.46:0/1796769978 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4af80c19b0 con 0x7f4af80c8a60 2024-01-26T23:30:09.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.716+0000 7f4afc977700 1 -- 172.21.15.46:0/1796769978 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4af80c19b0 con 0x7f4ad8066f30 2024-01-26T23:30:09.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.726+0000 7f4ae67fc700 1 -- 172.21.15.46:0/1796769978 <== mgr.4102 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345254 (secure 0 0 0) 0x7f4af80c19b0 con 0x7f4ad8066f30 2024-01-26T23:30:09.731 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:09.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.730+0000 7f4afc977700 1 -- 172.21.15.46:0/1796769978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4ad8066f30 msgr2=0x7f4ad80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.730+0000 7f4afc977700 1 --2- 172.21.15.46:0/1796769978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4ad8066f30 0x7f4ad80693e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4af8152300 tx=0x7f4aec007eb0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.730+0000 7f4afc977700 1 -- 172.21.15.46:0/1796769978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af80c8a60 msgr2=0x7f4af81500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.730+0000 7f4afc977700 1 --2- 172.21.15.46:0/1796769978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af80c8a60 0x7f4af81500e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800ea10 tx=0x7f4ae800a330 comp rx=0 tx=0).stop 2024-01-26T23:30:09.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.730+0000 7f4afc977700 1 -- 172.21.15.46:0/1796769978 shutdown_connections 2024-01-26T23:30:09.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.730+0000 7f4afc977700 1 --2- 172.21.15.46:0/1796769978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4ad8066f30 0x7f4ad80693e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.730+0000 7f4afc977700 1 --2- 172.21.15.46:0/1796769978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af80c8a60 0x7f4af81500e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.730+0000 7f4afc977700 1 --2- 172.21.15.46:0/1796769978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4af8155af0 0x7f4af815bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.730+0000 7f4afc977700 1 --2- 172.21.15.46:0/1796769978 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4af80cba30 0x7f4af8154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.730+0000 7f4afc977700 1 -- 172.21.15.46:0/1796769978 >> 172.21.15.46:0/1796769978 conn(0x7f4af80cd6e0 msgr2=0x7f4af8152550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:09.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.730+0000 7f4afc977700 1 -- 172.21.15.46:0/1796769978 shutdown_connections 2024-01-26T23:30:09.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.730+0000 7f4afc977700 1 -- 172.21.15.46:0/1796769978 wait complete. 2024-01-26T23:30:09.735 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:30:09.744 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-01-26T23:30:08.230030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594851,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381512,"kb_used_data":3208,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593208,"statfs":{"total":773094113280,"available":770655444992,"internally_reserved":0,"allocated":3284992,"data_stored":1985606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2177,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002799"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690995+0000","last_change":"2024-01-26T23:29:44.615781+0000","last_active":"2024-01-26T23:29:48.690995+0000","last_peered":"2024-01-26T23:29:48.690995+0000","last_clean":"2024-01-26T23:29:48.690995+0000","last_became_active":"2024-01-26T23:29:44.615522+0000","last_became_peered":"2024-01-26T23:29:44.615522+0000","last_unstale":"2024-01-26T23:29:48.690995+0000","last_undegraded":"2024-01-26T23:29:48.690995+0000","last_fullsized":"2024-01-26T23:29:48.690995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:03.997854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.627221+0000","last_change":"2024-01-26T23:29:44.628874+0000","last_active":"2024-01-26T23:29:48.627221+0000","last_peered":"2024-01-26T23:29:48.627221+0000","last_clean":"2024-01-26T23:29:48.627221+0000","last_became_active":"2024-01-26T23:29:44.628581+0000","last_became_peered":"2024-01-26T23:29:44.628581+0000","last_unstale":"2024-01-26T23:29:48.627221+0000","last_undegraded":"2024-01-26T23:29:48.627221+0000","last_fullsized":"2024-01-26T23:29:48.627221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:14.555745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631189+0000","last_change":"2024-01-26T23:29:44.601978+0000","last_active":"2024-01-26T23:29:48.631189+0000","last_peered":"2024-01-26T23:29:48.631189+0000","last_clean":"2024-01-26T23:29:48.631189+0000","last_became_active":"2024-01-26T23:29:44.601539+0000","last_became_peered":"2024-01-26T23:29:44.601539+0000","last_unstale":"2024-01-26T23:29:48.631189+0000","last_undegraded":"2024-01-26T23:29:48.631189+0000","last_fullsized":"2024-01-26T23:29:48.631189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:35:11.068882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636104+0000","last_change":"2024-01-26T23:29:44.632221+0000","last_active":"2024-01-26T23:29:48.636104+0000","last_peered":"2024-01-26T23:29:48.636104+0000","last_clean":"2024-01-26T23:29:48.636104+0000","last_became_active":"2024-01-26T23:29:44.631611+0000","last_became_peered":"2024-01-26T23:29:44.631611+0000","last_unstale":"2024-01-26T23:29:48.636104+0000","last_undegraded":"2024-01-26T23:29:48.636104+0000","last_fullsized":"2024-01-26T23:29:48.636104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:55:14.606640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690150+0000","last_change":"2024-01-26T23:29:44.616723+0000","last_active":"2024-01-26T23:29:48.690150+0000","last_peered":"2024-01-26T23:29:48.690150+0000","last_clean":"2024-01-26T23:29:48.690150+0000","last_became_active":"2024-01-26T23:29:44.615737+0000","last_became_peered":"2024-01-26T23:29:44.615737+0000","last_unstale":"2024-01-26T23:29:48.690150+0000","last_undegraded":"2024-01-26T23:29:48.690150+0000","last_fullsized":"2024-01-26T23:29:48.690150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635997+0000","last_change":"2024-01-26T23:29:44.622983+0000","last_active":"2024-01-26T23:29:48.635997+0000","last_peered":"2024-01-26T23:29:48.635997+0000","last_clean":"2024-01-26T23:29:48.635997+0000","last_became_active":"2024-01-26T23:29:44.622721+0000","last_became_peered":"2024-01-26T23:29:44.622721+0000","last_unstale":"2024-01-26T23:29:48.635997+0000","last_undegraded":"2024-01-26T23:29:48.635997+0000","last_fullsized":"2024-01-26T23:29:48.635997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690538+0000","last_change":"2024-01-26T23:29:44.620570+0000","last_active":"2024-01-26T23:29:48.690538+0000","last_peered":"2024-01-26T23:29:48.690538+0000","last_clean":"2024-01-26T23:29:48.690538+0000","last_became_active":"2024-01-26T23:29:44.620303+0000","last_became_peered":"2024-01-26T23:29:44.620303+0000","last_unstale":"2024-01-26T23:29:48.690538+0000","last_undegraded":"2024-01-26T23:29:48.690538+0000","last_fullsized":"2024-01-26T23:29:48.690538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:55:14.606640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636953+0000","last_change":"2024-01-26T23:29:44.630873+0000","last_active":"2024-01-26T23:29:48.636953+0000","last_peered":"2024-01-26T23:29:48.636953+0000","last_clean":"2024-01-26T23:29:48.636953+0000","last_became_active":"2024-01-26T23:29:44.630579+0000","last_became_peered":"2024-01-26T23:29:44.630579+0000","last_unstale":"2024-01-26T23:29:48.636953+0000","last_undegraded":"2024-01-26T23:29:48.636953+0000","last_fullsized":"2024-01-26T23:29:48.636953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:48:24.085774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690821+0000","last_change":"2024-01-26T23:29:44.626396+0000","last_active":"2024-01-26T23:29:48.690821+0000","last_peered":"2024-01-26T23:29:48.690821+0000","last_clean":"2024-01-26T23:29:48.690821+0000","last_became_active":"2024-01-26T23:29:44.622150+0000","last_became_peered":"2024-01-26T23:29:44.622150+0000","last_unstale":"2024-01-26T23:29:48.690821+0000","last_undegraded":"2024-01-26T23:29:48.690821+0000","last_fullsized":"2024-01-26T23:29:48.690821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:57:15.558584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637882+0000","last_change":"2024-01-26T23:29:44.618861+0000","last_active":"2024-01-26T23:29:48.637882+0000","last_peered":"2024-01-26T23:29:48.637882+0000","last_clean":"2024-01-26T23:29:48.637882+0000","last_became_active":"2024-01-26T23:29:44.618575+0000","last_became_peered":"2024-01-26T23:29:44.618575+0000","last_unstale":"2024-01-26T23:29:48.637882+0000","last_undegraded":"2024-01-26T23:29:48.637882+0000","last_fullsized":"2024-01-26T23:29:48.637882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:32.912164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637940+0000","last_change":"2024-01-26T23:29:44.617693+0000","last_active":"2024-01-26T23:29:48.637940+0000","last_peered":"2024-01-26T23:29:48.637940+0000","last_clean":"2024-01-26T23:29:48.637940+0000","last_became_active":"2024-01-26T23:29:44.617393+0000","last_became_peered":"2024-01-26T23:29:44.617393+0000","last_unstale":"2024-01-26T23:29:48.637940+0000","last_undegraded":"2024-01-26T23:29:48.637940+0000","last_fullsized":"2024-01-26T23:29:48.637940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633424+0000","last_change":"2024-01-26T23:29:44.614095+0000","last_active":"2024-01-26T23:29:48.633424+0000","last_peered":"2024-01-26T23:29:48.633424+0000","last_clean":"2024-01-26T23:29:48.633424+0000","last_became_active":"2024-01-26T23:29:44.613758+0000","last_became_peered":"2024-01-26T23:29:44.613758+0000","last_unstale":"2024-01-26T23:29:48.633424+0000","last_undegraded":"2024-01-26T23:29:48.633424+0000","last_fullsized":"2024-01-26T23:29:48.633424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:22.392135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636817+0000","last_change":"2024-01-26T23:29:44.624158+0000","last_active":"2024-01-26T23:29:48.636817+0000","last_peered":"2024-01-26T23:29:48.636817+0000","last_clean":"2024-01-26T23:29:48.636817+0000","last_became_active":"2024-01-26T23:29:44.623876+0000","last_became_peered":"2024-01-26T23:29:44.623876+0000","last_unstale":"2024-01-26T23:29:48.636817+0000","last_undegraded":"2024-01-26T23:29:48.636817+0000","last_fullsized":"2024-01-26T23:29:48.636817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631396+0000","last_change":"2024-01-26T23:29:44.618895+0000","last_active":"2024-01-26T23:29:48.631396+0000","last_peered":"2024-01-26T23:29:48.631396+0000","last_clean":"2024-01-26T23:29:48.631396+0000","last_became_active":"2024-01-26T23:29:44.618505+0000","last_became_peered":"2024-01-26T23:29:44.618505+0000","last_unstale":"2024-01-26T23:29:48.631396+0000","last_undegraded":"2024-01-26T23:29:48.631396+0000","last_fullsized":"2024-01-26T23:29:48.631396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:59:47.867303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.625861+0000","last_change":"2024-01-26T23:29:44.612783+0000","last_active":"2024-01-26T23:29:48.625861+0000","last_peered":"2024-01-26T23:29:48.625861+0000","last_clean":"2024-01-26T23:29:48.625861+0000","last_became_active":"2024-01-26T23:29:44.612451+0000","last_became_peered":"2024-01-26T23:29:44.612451+0000","last_unstale":"2024-01-26T23:29:48.625861+0000","last_undegraded":"2024-01-26T23:29:48.625861+0000","last_fullsized":"2024-01-26T23:29:48.625861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:12.500308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632657+0000","last_change":"2024-01-26T23:29:44.630662+0000","last_active":"2024-01-26T23:29:48.632657+0000","last_peered":"2024-01-26T23:29:48.632657+0000","last_clean":"2024-01-26T23:29:48.632657+0000","last_became_active":"2024-01-26T23:29:44.630412+0000","last_became_peered":"2024-01-26T23:29:44.630412+0000","last_unstale":"2024-01-26T23:29:48.632657+0000","last_undegraded":"2024-01-26T23:29:48.632657+0000","last_fullsized":"2024-01-26T23:29:48.632657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636581+0000","last_change":"2024-01-26T23:29:44.624186+0000","last_active":"2024-01-26T23:29:48.636581+0000","last_peered":"2024-01-26T23:29:48.636581+0000","last_clean":"2024-01-26T23:29:48.636581+0000","last_became_active":"2024-01-26T23:29:44.623729+0000","last_became_peered":"2024-01-26T23:29:44.623729+0000","last_unstale":"2024-01-26T23:29:48.636581+0000","last_undegraded":"2024-01-26T23:29:48.636581+0000","last_fullsized":"2024-01-26T23:29:48.636581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:20.138786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633462+0000","last_change":"2024-01-26T23:29:44.615801+0000","last_active":"2024-01-26T23:29:48.633462+0000","last_peered":"2024-01-26T23:29:48.633462+0000","last_clean":"2024-01-26T23:29:48.633462+0000","last_became_active":"2024-01-26T23:29:44.614126+0000","last_became_peered":"2024-01-26T23:29:44.614126+0000","last_unstale":"2024-01-26T23:29:48.633462+0000","last_undegraded":"2024-01-26T23:29:48.633462+0000","last_fullsized":"2024-01-26T23:29:48.633462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:22.392135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690472+0000","last_change":"2024-01-26T23:29:44.620718+0000","last_active":"2024-01-26T23:29:48.690472+0000","last_peered":"2024-01-26T23:29:48.690472+0000","last_clean":"2024-01-26T23:29:48.690472+0000","last_became_active":"2024-01-26T23:29:44.620452+0000","last_became_peered":"2024-01-26T23:29:44.620452+0000","last_unstale":"2024-01-26T23:29:48.690472+0000","last_undegraded":"2024-01-26T23:29:48.690472+0000","last_fullsized":"2024-01-26T23:29:48.690472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:01.636855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.697979+0000","last_change":"2024-01-26T23:29:48.697204+0000","last_active":"2024-01-26T23:29:48.697979+0000","last_peered":"2024-01-26T23:29:48.697979+0000","last_clean":"2024-01-26T23:29:48.697979+0000","last_became_active":"2024-01-26T23:29:48.696727+0000","last_became_peered":"2024-01-26T23:29:48.696727+0000","last_unstale":"2024-01-26T23:29:48.697979+0000","last_undegraded":"2024-01-26T23:29:48.697979+0000","last_fullsized":"2024-01-26T23:29:48.697979+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:59.257622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633767+0000","last_change":"2024-01-26T23:29:43.611980+0000","last_active":"2024-01-26T23:29:48.633767+0000","last_peered":"2024-01-26T23:29:48.633767+0000","last_clean":"2024-01-26T23:29:48.633767+0000","last_became_active":"2024-01-26T23:29:43.611649+0000","last_became_peered":"2024-01-26T23:29:43.611649+0000","last_unstale":"2024-01-26T23:29:48.633767+0000","last_undegraded":"2024-01-26T23:29:48.633767+0000","last_fullsized":"2024-01-26T23:29:48.633767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:31.317009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637215+0000","last_change":"2024-01-26T23:29:44.617315+0000","last_active":"2024-01-26T23:29:48.637215+0000","last_peered":"2024-01-26T23:29:48.637215+0000","last_clean":"2024-01-26T23:29:48.637215+0000","last_became_active":"2024-01-26T23:29:44.617020+0000","last_became_peered":"2024-01-26T23:29:44.617020+0000","last_unstale":"2024-01-26T23:29:48.637215+0000","last_undegraded":"2024-01-26T23:29:48.637215+0000","last_fullsized":"2024-01-26T23:29:48.637215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:21:47.569396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652507+0000","last_change":"2024-01-26T23:29:48.647630+0000","last_active":"2024-01-26T23:29:48.652507+0000","last_peered":"2024-01-26T23:29:48.652507+0000","last_clean":"2024-01-26T23:29:48.652507+0000","last_became_active":"2024-01-26T23:29:48.646094+0000","last_became_peered":"2024-01-26T23:29:48.646094+0000","last_unstale":"2024-01-26T23:29:48.652507+0000","last_undegraded":"2024-01-26T23:29:48.652507+0000","last_fullsized":"2024-01-26T23:29:48.652507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:20.380384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637332+0000","last_change":"2024-01-26T23:29:43.616917+0000","last_active":"2024-01-26T23:29:48.637332+0000","last_peered":"2024-01-26T23:29:48.637332+0000","last_clean":"2024-01-26T23:29:48.637332+0000","last_became_active":"2024-01-26T23:29:43.616606+0000","last_became_peered":"2024-01-26T23:29:43.616606+0000","last_unstale":"2024-01-26T23:29:48.637332+0000","last_undegraded":"2024-01-26T23:29:48.637332+0000","last_fullsized":"2024-01-26T23:29:48.637332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:54:53.198685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633652+0000","last_change":"2024-01-26T23:29:44.626223+0000","last_active":"2024-01-26T23:29:48.633652+0000","last_peered":"2024-01-26T23:29:48.633652+0000","last_clean":"2024-01-26T23:29:48.633652+0000","last_became_active":"2024-01-26T23:29:44.625830+0000","last_became_peered":"2024-01-26T23:29:44.625830+0000","last_unstale":"2024-01-26T23:29:48.633652+0000","last_undegraded":"2024-01-26T23:29:48.633652+0000","last_fullsized":"2024-01-26T23:29:48.633652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652505+0000","last_change":"2024-01-26T23:29:48.647450+0000","last_active":"2024-01-26T23:29:48.652505+0000","last_peered":"2024-01-26T23:29:48.652505+0000","last_clean":"2024-01-26T23:29:48.652505+0000","last_became_active":"2024-01-26T23:29:48.646159+0000","last_became_peered":"2024-01-26T23:29:48.646159+0000","last_unstale":"2024-01-26T23:29:48.652505+0000","last_undegraded":"2024-01-26T23:29:48.652505+0000","last_fullsized":"2024-01-26T23:29:48.652505+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:13:54.119091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690197+0000","last_change":"2024-01-26T23:29:43.624480+0000","last_active":"2024-01-26T23:29:48.690197+0000","last_peered":"2024-01-26T23:29:48.690197+0000","last_clean":"2024-01-26T23:29:48.690197+0000","last_became_active":"2024-01-26T23:29:43.624225+0000","last_became_peered":"2024-01-26T23:29:43.624225+0000","last_unstale":"2024-01-26T23:29:48.690197+0000","last_undegraded":"2024-01-26T23:29:48.690197+0000","last_fullsized":"2024-01-26T23:29:48.690197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:21.390599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636199+0000","last_change":"2024-01-26T23:29:44.621996+0000","last_active":"2024-01-26T23:29:48.636199+0000","last_peered":"2024-01-26T23:29:48.636199+0000","last_clean":"2024-01-26T23:29:48.636199+0000","last_became_active":"2024-01-26T23:29:44.621750+0000","last_became_peered":"2024-01-26T23:29:44.621750+0000","last_unstale":"2024-01-26T23:29:48.636199+0000","last_undegraded":"2024-01-26T23:29:48.636199+0000","last_fullsized":"2024-01-26T23:29:48.636199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:32.912164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652475+0000","last_change":"2024-01-26T23:29:48.647439+0000","last_active":"2024-01-26T23:29:48.652475+0000","last_peered":"2024-01-26T23:29:48.652475+0000","last_clean":"2024-01-26T23:29:48.652475+0000","last_became_active":"2024-01-26T23:29:48.647004+0000","last_became_peered":"2024-01-26T23:29:48.647004+0000","last_unstale":"2024-01-26T23:29:48.652475+0000","last_undegraded":"2024-01-26T23:29:48.652475+0000","last_fullsized":"2024-01-26T23:29:48.652475+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:29:51.471531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690167+0000","last_change":"2024-01-26T23:29:43.623440+0000","last_active":"2024-01-26T23:29:48.690167+0000","last_peered":"2024-01-26T23:29:48.690167+0000","last_clean":"2024-01-26T23:29:48.690167+0000","last_became_active":"2024-01-26T23:29:43.623105+0000","last_became_peered":"2024-01-26T23:29:43.623105+0000","last_unstale":"2024-01-26T23:29:48.690167+0000","last_undegraded":"2024-01-26T23:29:48.690167+0000","last_fullsized":"2024-01-26T23:29:48.690167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:37:22.647734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636762+0000","last_change":"2024-01-26T23:29:44.622275+0000","last_active":"2024-01-26T23:29:48.636762+0000","last_peered":"2024-01-26T23:29:48.636762+0000","last_clean":"2024-01-26T23:29:48.636762+0000","last_became_active":"2024-01-26T23:29:44.621687+0000","last_became_peered":"2024-01-26T23:29:44.621687+0000","last_unstale":"2024-01-26T23:29:48.636762+0000","last_undegraded":"2024-01-26T23:29:48.636762+0000","last_fullsized":"2024-01-26T23:29:48.636762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:12.500308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.653689+0000","last_change":"2024-01-26T23:29:48.652656+0000","last_active":"2024-01-26T23:29:48.653689+0000","last_peered":"2024-01-26T23:29:48.653689+0000","last_clean":"2024-01-26T23:29:48.653689+0000","last_became_active":"2024-01-26T23:29:48.652264+0000","last_became_peered":"2024-01-26T23:29:48.652264+0000","last_unstale":"2024-01-26T23:29:48.653689+0000","last_undegraded":"2024-01-26T23:29:48.653689+0000","last_fullsized":"2024-01-26T23:29:48.653689+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:35:23.165011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636924+0000","last_change":"2024-01-26T23:29:43.620505+0000","last_active":"2024-01-26T23:29:48.636924+0000","last_peered":"2024-01-26T23:29:48.636924+0000","last_clean":"2024-01-26T23:29:48.636924+0000","last_became_active":"2024-01-26T23:29:43.620202+0000","last_became_peered":"2024-01-26T23:29:43.620202+0000","last_unstale":"2024-01-26T23:29:48.636924+0000","last_undegraded":"2024-01-26T23:29:48.636924+0000","last_fullsized":"2024-01-26T23:29:48.636924+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:31.317009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633949+0000","last_change":"2024-01-26T23:29:44.615753+0000","last_active":"2024-01-26T23:29:48.633949+0000","last_peered":"2024-01-26T23:29:48.633949+0000","last_clean":"2024-01-26T23:29:48.633949+0000","last_became_active":"2024-01-26T23:29:44.615469+0000","last_became_peered":"2024-01-26T23:29:44.615469+0000","last_unstale":"2024-01-26T23:29:48.633949+0000","last_undegraded":"2024-01-26T23:29:48.633949+0000","last_fullsized":"2024-01-26T23:29:48.633949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:37:23.649270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.651391+0000","last_change":"2024-01-26T23:29:48.646032+0000","last_active":"2024-01-26T23:29:48.651391+0000","last_peered":"2024-01-26T23:29:48.651391+0000","last_clean":"2024-01-26T23:29:48.651391+0000","last_became_active":"2024-01-26T23:29:48.645627+0000","last_became_peered":"2024-01-26T23:29:48.645627+0000","last_unstale":"2024-01-26T23:29:48.651391+0000","last_undegraded":"2024-01-26T23:29:48.651391+0000","last_fullsized":"2024-01-26T23:29:48.651391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:42.473075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632782+0000","last_change":"2024-01-26T23:29:43.614220+0000","last_active":"2024-01-26T23:29:48.632782+0000","last_peered":"2024-01-26T23:29:48.632782+0000","last_clean":"2024-01-26T23:29:48.632782+0000","last_became_active":"2024-01-26T23:29:43.613886+0000","last_became_peered":"2024-01-26T23:29:43.613886+0000","last_unstale":"2024-01-26T23:29:48.632782+0000","last_undegraded":"2024-01-26T23:29:48.632782+0000","last_fullsized":"2024-01-26T23:29:48.632782+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:09:46.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632076+0000","last_change":"2024-01-26T23:29:44.614249+0000","last_active":"2024-01-26T23:29:48.632076+0000","last_peered":"2024-01-26T23:29:48.632076+0000","last_clean":"2024-01-26T23:29:48.632076+0000","last_became_active":"2024-01-26T23:29:44.613949+0000","last_became_peered":"2024-01-26T23:29:44.613949+0000","last_unstale":"2024-01-26T23:29:48.632076+0000","last_undegraded":"2024-01-26T23:29:48.632076+0000","last_fullsized":"2024-01-26T23:29:48.632076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:14.555745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.651517+0000","last_change":"2024-01-26T23:29:48.647625+0000","last_active":"2024-01-26T23:29:48.651517+0000","last_peered":"2024-01-26T23:29:48.651517+0000","last_clean":"2024-01-26T23:29:48.651517+0000","last_became_active":"2024-01-26T23:29:48.647273+0000","last_became_peered":"2024-01-26T23:29:48.647273+0000","last_unstale":"2024-01-26T23:29:48.651517+0000","last_undegraded":"2024-01-26T23:29:48.651517+0000","last_fullsized":"2024-01-26T23:29:48.651517+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:52:44.817581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636975+0000","last_change":"2024-01-26T23:29:43.616939+0000","last_active":"2024-01-26T23:29:48.636975+0000","last_peered":"2024-01-26T23:29:48.636975+0000","last_clean":"2024-01-26T23:29:48.636975+0000","last_became_active":"2024-01-26T23:29:43.614953+0000","last_became_peered":"2024-01-26T23:29:43.614953+0000","last_unstale":"2024-01-26T23:29:48.636975+0000","last_undegraded":"2024-01-26T23:29:48.636975+0000","last_fullsized":"2024-01-26T23:29:48.636975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:36.035429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636683+0000","last_change":"2024-01-26T23:29:44.621908+0000","last_active":"2024-01-26T23:29:48.636683+0000","last_peered":"2024-01-26T23:29:48.636683+0000","last_clean":"2024-01-26T23:29:48.636683+0000","last_became_active":"2024-01-26T23:29:44.621706+0000","last_became_peered":"2024-01-26T23:29:44.621706+0000","last_unstale":"2024-01-26T23:29:48.636683+0000","last_undegraded":"2024-01-26T23:29:48.636683+0000","last_fullsized":"2024-01-26T23:29:48.636683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696535+0000","last_change":"2024-01-26T23:29:48.695257+0000","last_active":"2024-01-26T23:29:48.696535+0000","last_peered":"2024-01-26T23:29:48.696535+0000","last_clean":"2024-01-26T23:29:48.696535+0000","last_became_active":"2024-01-26T23:29:48.694835+0000","last_became_peered":"2024-01-26T23:29:48.694835+0000","last_unstale":"2024-01-26T23:29:48.696535+0000","last_undegraded":"2024-01-26T23:29:48.696535+0000","last_fullsized":"2024-01-26T23:29:48.696535+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:53:55.279661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632759+0000","last_change":"2024-01-26T23:29:43.617477+0000","last_active":"2024-01-26T23:29:48.632759+0000","last_peered":"2024-01-26T23:29:48.632759+0000","last_clean":"2024-01-26T23:29:48.632759+0000","last_became_active":"2024-01-26T23:29:43.617014+0000","last_became_peered":"2024-01-26T23:29:43.617014+0000","last_unstale":"2024-01-26T23:29:48.632759+0000","last_undegraded":"2024-01-26T23:29:48.632759+0000","last_fullsized":"2024-01-26T23:29:48.632759+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:32.417441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626943+0000","last_change":"2024-01-26T23:29:44.619133+0000","last_active":"2024-01-26T23:29:48.626943+0000","last_peered":"2024-01-26T23:29:48.626943+0000","last_clean":"2024-01-26T23:29:48.626943+0000","last_became_active":"2024-01-26T23:29:44.618827+0000","last_became_peered":"2024-01-26T23:29:44.618827+0000","last_unstale":"2024-01-26T23:29:48.626943+0000","last_undegraded":"2024-01-26T23:29:48.626943+0000","last_fullsized":"2024-01-26T23:29:48.626943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696639+0000","last_change":"2024-01-26T23:29:48.693971+0000","last_active":"2024-01-26T23:29:48.696639+0000","last_peered":"2024-01-26T23:29:48.696639+0000","last_clean":"2024-01-26T23:29:48.696639+0000","last_became_active":"2024-01-26T23:29:48.693563+0000","last_became_peered":"2024-01-26T23:29:48.693563+0000","last_unstale":"2024-01-26T23:29:48.696639+0000","last_undegraded":"2024-01-26T23:29:48.696639+0000","last_fullsized":"2024-01-26T23:29:48.696639+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:42:42.888059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.638779+0000","last_change":"2024-01-26T23:29:43.622313+0000","last_active":"2024-01-26T23:29:48.638779+0000","last_peered":"2024-01-26T23:29:48.638779+0000","last_clean":"2024-01-26T23:29:48.638779+0000","last_became_active":"2024-01-26T23:29:43.622008+0000","last_became_peered":"2024-01-26T23:29:43.622008+0000","last_unstale":"2024-01-26T23:29:48.638779+0000","last_undegraded":"2024-01-26T23:29:48.638779+0000","last_fullsized":"2024-01-26T23:29:48.638779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:37:22.647734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.634360+0000","last_change":"2024-01-26T23:29:44.619813+0000","last_active":"2024-01-26T23:29:48.634360+0000","last_peered":"2024-01-26T23:29:48.634360+0000","last_clean":"2024-01-26T23:29:48.634360+0000","last_became_active":"2024-01-26T23:29:44.619466+0000","last_became_peered":"2024-01-26T23:29:44.619466+0000","last_unstale":"2024-01-26T23:29:48.634360+0000","last_undegraded":"2024-01-26T23:29:48.634360+0000","last_fullsized":"2024-01-26T23:29:48.634360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:48:24.085774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.653660+0000","last_change":"2024-01-26T23:29:48.652676+0000","last_active":"2024-01-26T23:29:48.653660+0000","last_peered":"2024-01-26T23:29:48.653660+0000","last_clean":"2024-01-26T23:29:48.653660+0000","last_became_active":"2024-01-26T23:29:48.652134+0000","last_became_peered":"2024-01-26T23:29:48.652134+0000","last_unstale":"2024-01-26T23:29:48.653660+0000","last_undegraded":"2024-01-26T23:29:48.653660+0000","last_fullsized":"2024-01-26T23:29:48.653660+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:27.746532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.634232+0000","last_change":"2024-01-26T23:29:43.620804+0000","last_active":"2024-01-26T23:29:48.634232+0000","last_peered":"2024-01-26T23:29:48.634232+0000","last_clean":"2024-01-26T23:29:48.634232+0000","last_became_active":"2024-01-26T23:29:43.620530+0000","last_became_peered":"2024-01-26T23:29:43.620530+0000","last_unstale":"2024-01-26T23:29:48.634232+0000","last_undegraded":"2024-01-26T23:29:48.634232+0000","last_fullsized":"2024-01-26T23:29:48.634232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:44:34.698447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.634272+0000","last_change":"2024-01-26T23:29:44.626105+0000","last_active":"2024-01-26T23:29:48.634272+0000","last_peered":"2024-01-26T23:29:48.634272+0000","last_clean":"2024-01-26T23:29:48.634272+0000","last_became_active":"2024-01-26T23:29:44.625500+0000","last_became_peered":"2024-01-26T23:29:44.625500+0000","last_unstale":"2024-01-26T23:29:48.634272+0000","last_undegraded":"2024-01-26T23:29:48.634272+0000","last_fullsized":"2024-01-26T23:29:48.634272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.644865+0000","last_change":"2024-01-26T23:29:48.643281+0000","last_active":"2024-01-26T23:29:48.644865+0000","last_peered":"2024-01-26T23:29:48.644865+0000","last_clean":"2024-01-26T23:29:48.644865+0000","last_became_active":"2024-01-26T23:29:48.642907+0000","last_became_peered":"2024-01-26T23:29:48.642907+0000","last_unstale":"2024-01-26T23:29:48.644865+0000","last_undegraded":"2024-01-26T23:29:48.644865+0000","last_fullsized":"2024-01-26T23:29:48.644865+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:24:23.994084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626216+0000","last_change":"2024-01-26T23:29:43.621999+0000","last_active":"2024-01-26T23:29:48.626216+0000","last_peered":"2024-01-26T23:29:48.626216+0000","last_clean":"2024-01-26T23:29:48.626216+0000","last_became_active":"2024-01-26T23:29:43.621214+0000","last_became_peered":"2024-01-26T23:29:43.621214+0000","last_unstale":"2024-01-26T23:29:48.626216+0000","last_undegraded":"2024-01-26T23:29:48.626216+0000","last_fullsized":"2024-01-26T23:29:48.626216+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:37:22.647734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633819+0000","last_change":"2024-01-26T23:29:44.608941+0000","last_active":"2024-01-26T23:29:48.633819+0000","last_peered":"2024-01-26T23:29:48.633819+0000","last_clean":"2024-01-26T23:29:48.633819+0000","last_became_active":"2024-01-26T23:29:44.608664+0000","last_became_peered":"2024-01-26T23:29:44.608664+0000","last_unstale":"2024-01-26T23:29:48.633819+0000","last_undegraded":"2024-01-26T23:29:48.633819+0000","last_fullsized":"2024-01-26T23:29:48.633819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:47:10.097930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.651674+0000","last_change":"2024-01-26T23:29:48.643876+0000","last_active":"2024-01-26T23:29:48.651674+0000","last_peered":"2024-01-26T23:29:48.651674+0000","last_clean":"2024-01-26T23:29:48.651674+0000","last_became_active":"2024-01-26T23:29:48.641104+0000","last_became_peered":"2024-01-26T23:29:48.641104+0000","last_unstale":"2024-01-26T23:29:48.651674+0000","last_undegraded":"2024-01-26T23:29:48.651674+0000","last_fullsized":"2024-01-26T23:29:48.651674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:29:51.471531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-01-26T23:29:54.992720+0000","last_change":"2024-01-26T23:29:43.622211+0000","last_active":"2024-01-26T23:29:54.992720+0000","last_peered":"2024-01-26T23:29:54.992720+0000","last_clean":"2024-01-26T23:29:54.992720+0000","last_became_active":"2024-01-26T23:29:43.621919+0000","last_became_peered":"2024-01-26T23:29:43.621919+0000","last_unstale":"2024-01-26T23:29:54.992720+0000","last_undegraded":"2024-01-26T23:29:54.992720+0000","last_fullsized":"2024-01-26T23:29:54.992720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:45.384500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3713,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690317+0000","last_change":"2024-01-26T23:29:44.618740+0000","last_active":"2024-01-26T23:29:48.690317+0000","last_peered":"2024-01-26T23:29:48.690317+0000","last_clean":"2024-01-26T23:29:48.690317+0000","last_became_active":"2024-01-26T23:29:44.618482+0000","last_became_peered":"2024-01-26T23:29:44.618482+0000","last_unstale":"2024-01-26T23:29:48.690317+0000","last_undegraded":"2024-01-26T23:29:48.690317+0000","last_fullsized":"2024-01-26T23:29:48.690317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:03.314483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695102+0000","last_change":"2024-01-26T23:29:48.691860+0000","last_active":"2024-01-26T23:29:48.695102+0000","last_peered":"2024-01-26T23:29:48.695102+0000","last_clean":"2024-01-26T23:29:48.695102+0000","last_became_active":"2024-01-26T23:29:48.691440+0000","last_became_peered":"2024-01-26T23:29:48.691440+0000","last_unstale":"2024-01-26T23:29:48.695102+0000","last_undegraded":"2024-01-26T23:29:48.695102+0000","last_fullsized":"2024-01-26T23:29:48.695102+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:20.246494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632364+0000","last_change":"2024-01-26T23:29:43.613477+0000","last_active":"2024-01-26T23:29:48.632364+0000","last_peered":"2024-01-26T23:29:48.632364+0000","last_clean":"2024-01-26T23:29:48.632364+0000","last_became_active":"2024-01-26T23:29:43.613228+0000","last_became_peered":"2024-01-26T23:29:43.613228+0000","last_unstale":"2024-01-26T23:29:48.632364+0000","last_undegraded":"2024-01-26T23:29:48.632364+0000","last_fullsized":"2024-01-26T23:29:48.632364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:54:53.198685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631483+0000","last_change":"2024-01-26T23:29:44.613649+0000","last_active":"2024-01-26T23:29:48.631483+0000","last_peered":"2024-01-26T23:29:48.631483+0000","last_clean":"2024-01-26T23:29:48.631483+0000","last_became_active":"2024-01-26T23:29:44.613278+0000","last_became_peered":"2024-01-26T23:29:44.613278+0000","last_unstale":"2024-01-26T23:29:48.631483+0000","last_undegraded":"2024-01-26T23:29:48.631483+0000","last_fullsized":"2024-01-26T23:29:48.631483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:38.452904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696548+0000","last_change":"2024-01-26T23:29:48.691601+0000","last_active":"2024-01-26T23:29:48.696548+0000","last_peered":"2024-01-26T23:29:48.696548+0000","last_clean":"2024-01-26T23:29:48.696548+0000","last_became_active":"2024-01-26T23:29:48.691169+0000","last_became_peered":"2024-01-26T23:29:48.691169+0000","last_unstale":"2024-01-26T23:29:48.696548+0000","last_undegraded":"2024-01-26T23:29:48.696548+0000","last_fullsized":"2024-01-26T23:29:48.696548+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:09.701112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635763+0000","last_change":"2024-01-26T23:29:43.622165+0000","last_active":"2024-01-26T23:29:48.635763+0000","last_peered":"2024-01-26T23:29:48.635763+0000","last_clean":"2024-01-26T23:29:48.635763+0000","last_became_active":"2024-01-26T23:29:43.621876+0000","last_became_peered":"2024-01-26T23:29:43.621876+0000","last_unstale":"2024-01-26T23:29:48.635763+0000","last_undegraded":"2024-01-26T23:29:48.635763+0000","last_fullsized":"2024-01-26T23:29:48.635763+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:30.053494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690214+0000","last_change":"2024-01-26T23:29:44.615999+0000","last_active":"2024-01-26T23:29:48.690214+0000","last_peered":"2024-01-26T23:29:48.690214+0000","last_clean":"2024-01-26T23:29:48.690214+0000","last_became_active":"2024-01-26T23:29:44.615376+0000","last_became_peered":"2024-01-26T23:29:44.615376+0000","last_unstale":"2024-01-26T23:29:48.690214+0000","last_undegraded":"2024-01-26T23:29:48.690214+0000","last_fullsized":"2024-01-26T23:29:48.690214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:32.912164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696674+0000","last_change":"2024-01-26T23:29:48.694846+0000","last_active":"2024-01-26T23:29:48.696674+0000","last_peered":"2024-01-26T23:29:48.696674+0000","last_clean":"2024-01-26T23:29:48.696674+0000","last_became_active":"2024-01-26T23:29:48.694439+0000","last_became_peered":"2024-01-26T23:29:48.694439+0000","last_unstale":"2024-01-26T23:29:48.696674+0000","last_undegraded":"2024-01-26T23:29:48.696674+0000","last_fullsized":"2024-01-26T23:29:48.696674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:39:25.688252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632550+0000","last_change":"2024-01-26T23:29:43.613829+0000","last_active":"2024-01-26T23:29:48.632550+0000","last_peered":"2024-01-26T23:29:48.632550+0000","last_clean":"2024-01-26T23:29:48.632550+0000","last_became_active":"2024-01-26T23:29:43.613486+0000","last_became_peered":"2024-01-26T23:29:43.613486+0000","last_unstale":"2024-01-26T23:29:48.632550+0000","last_undegraded":"2024-01-26T23:29:48.632550+0000","last_fullsized":"2024-01-26T23:29:48.632550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:44:34.698447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633466+0000","last_change":"2024-01-26T23:29:44.614404+0000","last_active":"2024-01-26T23:29:48.633466+0000","last_peered":"2024-01-26T23:29:48.633466+0000","last_clean":"2024-01-26T23:29:48.633466+0000","last_became_active":"2024-01-26T23:29:44.614127+0000","last_became_peered":"2024-01-26T23:29:44.614127+0000","last_unstale":"2024-01-26T23:29:48.633466+0000","last_undegraded":"2024-01-26T23:29:48.633466+0000","last_fullsized":"2024-01-26T23:29:48.633466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:35:11.068882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.653735+0000","last_change":"2024-01-26T23:29:48.647303+0000","last_active":"2024-01-26T23:29:48.653735+0000","last_peered":"2024-01-26T23:29:48.653735+0000","last_clean":"2024-01-26T23:29:48.653735+0000","last_became_active":"2024-01-26T23:29:48.646893+0000","last_became_peered":"2024-01-26T23:29:48.646893+0000","last_unstale":"2024-01-26T23:29:48.653735+0000","last_undegraded":"2024-01-26T23:29:48.653735+0000","last_fullsized":"2024-01-26T23:29:48.653735+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:49.464111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631121+0000","last_change":"2024-01-26T23:29:43.610693+0000","last_active":"2024-01-26T23:29:48.631121+0000","last_peered":"2024-01-26T23:29:48.631121+0000","last_clean":"2024-01-26T23:29:48.631121+0000","last_became_active":"2024-01-26T23:29:43.610370+0000","last_became_peered":"2024-01-26T23:29:43.610370+0000","last_unstale":"2024-01-26T23:29:48.631121+0000","last_undegraded":"2024-01-26T23:29:48.631121+0000","last_fullsized":"2024-01-26T23:29:48.631121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:09:46.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.692648+0000","last_change":"2024-01-26T23:29:44.611702+0000","last_active":"2024-01-26T23:29:48.692648+0000","last_peered":"2024-01-26T23:29:48.692648+0000","last_clean":"2024-01-26T23:29:48.692648+0000","last_became_active":"2024-01-26T23:29:44.611378+0000","last_became_peered":"2024-01-26T23:29:44.611378+0000","last_unstale":"2024-01-26T23:29:48.692648+0000","last_undegraded":"2024-01-26T23:29:48.692648+0000","last_fullsized":"2024-01-26T23:29:48.692648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:38:52.041592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.644666+0000","last_change":"2024-01-26T23:29:48.643205+0000","last_active":"2024-01-26T23:29:48.644666+0000","last_peered":"2024-01-26T23:29:48.644666+0000","last_clean":"2024-01-26T23:29:48.644666+0000","last_became_active":"2024-01-26T23:29:48.642819+0000","last_became_peered":"2024-01-26T23:29:48.642819+0000","last_unstale":"2024-01-26T23:29:48.644666+0000","last_undegraded":"2024-01-26T23:29:48.644666+0000","last_fullsized":"2024-01-26T23:29:48.644666+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:51.210803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633091+0000","last_change":"2024-01-26T23:29:43.612609+0000","last_active":"2024-01-26T23:29:48.633091+0000","last_peered":"2024-01-26T23:29:48.633091+0000","last_clean":"2024-01-26T23:29:48.633091+0000","last_became_active":"2024-01-26T23:29:43.612318+0000","last_became_peered":"2024-01-26T23:29:43.612318+0000","last_unstale":"2024-01-26T23:29:48.633091+0000","last_undegraded":"2024-01-26T23:29:48.633091+0000","last_fullsized":"2024-01-26T23:29:48.633091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:31.317009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690843+0000","last_change":"2024-01-26T23:29:44.616632+0000","last_active":"2024-01-26T23:29:48.690843+0000","last_peered":"2024-01-26T23:29:48.690843+0000","last_clean":"2024-01-26T23:29:48.690843+0000","last_became_active":"2024-01-26T23:29:44.616280+0000","last_became_peered":"2024-01-26T23:29:44.616280+0000","last_unstale":"2024-01-26T23:29:48.690843+0000","last_undegraded":"2024-01-26T23:29:48.690843+0000","last_fullsized":"2024-01-26T23:29:48.690843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:48:24.085774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.654458+0000","last_change":"2024-01-26T23:29:48.644846+0000","last_active":"2024-01-26T23:29:48.654458+0000","last_peered":"2024-01-26T23:29:48.654458+0000","last_clean":"2024-01-26T23:29:48.654458+0000","last_became_active":"2024-01-26T23:29:48.644359+0000","last_became_peered":"2024-01-26T23:29:48.644359+0000","last_unstale":"2024-01-26T23:29:48.654458+0000","last_undegraded":"2024-01-26T23:29:48.654458+0000","last_fullsized":"2024-01-26T23:29:48.654458+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:59.257622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626696+0000","last_change":"2024-01-26T23:29:43.605084+0000","last_active":"2024-01-26T23:29:48.626696+0000","last_peered":"2024-01-26T23:29:48.626696+0000","last_clean":"2024-01-26T23:29:48.626696+0000","last_became_active":"2024-01-26T23:29:43.604761+0000","last_became_peered":"2024-01-26T23:29:43.604761+0000","last_unstale":"2024-01-26T23:29:48.626696+0000","last_undegraded":"2024-01-26T23:29:48.626696+0000","last_fullsized":"2024-01-26T23:29:48.626696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:44:34.698447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636936+0000","last_change":"2024-01-26T23:29:44.632161+0000","last_active":"2024-01-26T23:29:48.636936+0000","last_peered":"2024-01-26T23:29:48.636936+0000","last_clean":"2024-01-26T23:29:48.636936+0000","last_became_active":"2024-01-26T23:29:44.631945+0000","last_became_peered":"2024-01-26T23:29:44.631945+0000","last_unstale":"2024-01-26T23:29:48.636936+0000","last_undegraded":"2024-01-26T23:29:48.636936+0000","last_fullsized":"2024-01-26T23:29:48.636936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:03.314483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.654064+0000","last_change":"2024-01-26T23:29:48.647375+0000","last_active":"2024-01-26T23:29:48.654064+0000","last_peered":"2024-01-26T23:29:48.654064+0000","last_clean":"2024-01-26T23:29:48.654064+0000","last_became_active":"2024-01-26T23:29:48.646978+0000","last_became_peered":"2024-01-26T23:29:48.646978+0000","last_unstale":"2024-01-26T23:29:48.654064+0000","last_undegraded":"2024-01-26T23:29:48.654064+0000","last_fullsized":"2024-01-26T23:29:48.654064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:20.246494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635696+0000","last_change":"2024-01-26T23:29:43.620942+0000","last_active":"2024-01-26T23:29:48.635696+0000","last_peered":"2024-01-26T23:29:48.635696+0000","last_clean":"2024-01-26T23:29:48.635696+0000","last_became_active":"2024-01-26T23:29:43.620644+0000","last_became_peered":"2024-01-26T23:29:43.620644+0000","last_unstale":"2024-01-26T23:29:48.635696+0000","last_undegraded":"2024-01-26T23:29:48.635696+0000","last_fullsized":"2024-01-26T23:29:48.635696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:45.384500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.634352+0000","last_change":"2024-01-26T23:29:44.605909+0000","last_active":"2024-01-26T23:29:48.634352+0000","last_peered":"2024-01-26T23:29:48.634352+0000","last_clean":"2024-01-26T23:29:48.634352+0000","last_became_active":"2024-01-26T23:29:44.605652+0000","last_became_peered":"2024-01-26T23:29:44.605652+0000","last_unstale":"2024-01-26T23:29:48.634352+0000","last_undegraded":"2024-01-26T23:29:48.634352+0000","last_fullsized":"2024-01-26T23:29:48.634352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:31.055030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.654078+0000","last_change":"2024-01-26T23:29:48.647396+0000","last_active":"2024-01-26T23:29:48.654078+0000","last_peered":"2024-01-26T23:29:48.654078+0000","last_clean":"2024-01-26T23:29:48.654078+0000","last_became_active":"2024-01-26T23:29:48.646950+0000","last_became_peered":"2024-01-26T23:29:48.646950+0000","last_unstale":"2024-01-26T23:29:48.654078+0000","last_undegraded":"2024-01-26T23:29:48.654078+0000","last_fullsized":"2024-01-26T23:29:48.654078+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:51.210803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.625864+0000","last_change":"2024-01-26T23:29:44.617670+0000","last_active":"2024-01-26T23:29:48.625864+0000","last_peered":"2024-01-26T23:29:48.625864+0000","last_clean":"2024-01-26T23:29:48.625864+0000","last_became_active":"2024-01-26T23:29:44.616510+0000","last_became_peered":"2024-01-26T23:29:44.616510+0000","last_unstale":"2024-01-26T23:29:48.625864+0000","last_undegraded":"2024-01-26T23:29:48.625864+0000","last_fullsized":"2024-01-26T23:29:48.625864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:12:04.973733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631229+0000","last_change":"2024-01-26T23:29:43.618892+0000","last_active":"2024-01-26T23:29:48.631229+0000","last_peered":"2024-01-26T23:29:48.631229+0000","last_clean":"2024-01-26T23:29:48.631229+0000","last_became_active":"2024-01-26T23:29:43.618446+0000","last_became_peered":"2024-01-26T23:29:43.618446+0000","last_unstale":"2024-01-26T23:29:48.631229+0000","last_undegraded":"2024-01-26T23:29:48.631229+0000","last_fullsized":"2024-01-26T23:29:48.631229+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:36.035429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.694860+0000","last_change":"2024-01-26T23:29:48.691567+0000","last_active":"2024-01-26T23:29:48.694860+0000","last_peered":"2024-01-26T23:29:48.694860+0000","last_clean":"2024-01-26T23:29:48.694860+0000","last_became_active":"2024-01-26T23:29:48.691153+0000","last_became_peered":"2024-01-26T23:29:48.691153+0000","last_unstale":"2024-01-26T23:29:48.694860+0000","last_undegraded":"2024-01-26T23:29:48.694860+0000","last_fullsized":"2024-01-26T23:29:48.694860+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:49.464111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637181+0000","last_change":"2024-01-26T23:29:44.620961+0000","last_active":"2024-01-26T23:29:48.637181+0000","last_peered":"2024-01-26T23:29:48.637181+0000","last_clean":"2024-01-26T23:29:48.637181+0000","last_became_active":"2024-01-26T23:29:44.620701+0000","last_became_peered":"2024-01-26T23:29:44.620701+0000","last_unstale":"2024-01-26T23:29:48.637181+0000","last_undegraded":"2024-01-26T23:29:48.637181+0000","last_fullsized":"2024-01-26T23:29:48.637181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:32.912164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635054+0000","last_change":"2024-01-26T23:29:43.620894+0000","last_active":"2024-01-26T23:29:48.635054+0000","last_peered":"2024-01-26T23:29:48.635054+0000","last_clean":"2024-01-26T23:29:48.635054+0000","last_became_active":"2024-01-26T23:29:43.620603+0000","last_became_peered":"2024-01-26T23:29:43.620603+0000","last_unstale":"2024-01-26T23:29:48.635054+0000","last_undegraded":"2024-01-26T23:29:48.635054+0000","last_fullsized":"2024-01-26T23:29:48.635054+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:36.035429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.689568+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:29:48.689568+0000","last_peered":"2024-01-26T23:29:48.689568+0000","last_clean":"2024-01-26T23:29:48.689568+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:29:48.689568+0000","last_undegraded":"2024-01-26T23:29:48.689568+0000","last_fullsized":"2024-01-26T23:29:48.689568+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696515+0000","last_change":"2024-01-26T23:29:48.692933+0000","last_active":"2024-01-26T23:29:48.696515+0000","last_peered":"2024-01-26T23:29:48.696515+0000","last_clean":"2024-01-26T23:29:48.696515+0000","last_became_active":"2024-01-26T23:29:48.692576+0000","last_became_peered":"2024-01-26T23:29:48.692576+0000","last_unstale":"2024-01-26T23:29:48.696515+0000","last_undegraded":"2024-01-26T23:29:48.696515+0000","last_fullsized":"2024-01-26T23:29:48.696515+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:24:23.994084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631114+0000","last_change":"2024-01-26T23:29:44.601856+0000","last_active":"2024-01-26T23:29:48.631114+0000","last_peered":"2024-01-26T23:29:48.631114+0000","last_clean":"2024-01-26T23:29:48.631114+0000","last_became_active":"2024-01-26T23:29:44.601367+0000","last_became_peered":"2024-01-26T23:29:44.601367+0000","last_unstale":"2024-01-26T23:29:48.631114+0000","last_undegraded":"2024-01-26T23:29:48.631114+0000","last_fullsized":"2024-01-26T23:29:48.631114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:21:47.569396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.689624+0000","last_change":"2024-01-26T23:29:43.623719+0000","last_active":"2024-01-26T23:29:48.689624+0000","last_peered":"2024-01-26T23:29:48.689624+0000","last_clean":"2024-01-26T23:29:48.689624+0000","last_became_active":"2024-01-26T23:29:43.623493+0000","last_became_peered":"2024-01-26T23:29:43.623493+0000","last_unstale":"2024-01-26T23:29:48.689624+0000","last_undegraded":"2024-01-26T23:29:48.689624+0000","last_fullsized":"2024-01-26T23:29:48.689624+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:23:25.049333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695757+0000","last_change":"2024-01-26T23:29:48.691446+0000","last_active":"2024-01-26T23:29:48.695757+0000","last_peered":"2024-01-26T23:29:48.695757+0000","last_clean":"2024-01-26T23:29:48.695757+0000","last_became_active":"2024-01-26T23:29:48.691005+0000","last_became_peered":"2024-01-26T23:29:48.691005+0000","last_unstale":"2024-01-26T23:29:48.695757+0000","last_undegraded":"2024-01-26T23:29:48.695757+0000","last_fullsized":"2024-01-26T23:29:48.695757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:51.210803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690165+0000","last_change":"2024-01-26T23:29:43.624304+0000","last_active":"2024-01-26T23:29:48.690165+0000","last_peered":"2024-01-26T23:29:48.690165+0000","last_clean":"2024-01-26T23:29:48.690165+0000","last_became_active":"2024-01-26T23:29:43.624031+0000","last_became_peered":"2024-01-26T23:29:43.624031+0000","last_unstale":"2024-01-26T23:29:48.690165+0000","last_undegraded":"2024-01-26T23:29:48.690165+0000","last_fullsized":"2024-01-26T23:29:48.690165+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:32.417441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626845+0000","last_change":"2024-01-26T23:29:44.615706+0000","last_active":"2024-01-26T23:29:48.626845+0000","last_peered":"2024-01-26T23:29:48.626845+0000","last_clean":"2024-01-26T23:29:48.626845+0000","last_became_active":"2024-01-26T23:29:44.615377+0000","last_became_peered":"2024-01-26T23:29:44.615377+0000","last_unstale":"2024-01-26T23:29:48.626845+0000","last_undegraded":"2024-01-26T23:29:48.626845+0000","last_fullsized":"2024-01-26T23:29:48.626845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:38.452904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696514+0000","last_change":"2024-01-26T23:29:48.691661+0000","last_active":"2024-01-26T23:29:48.696514+0000","last_peered":"2024-01-26T23:29:48.696514+0000","last_clean":"2024-01-26T23:29:48.696514+0000","last_became_active":"2024-01-26T23:29:48.691233+0000","last_became_peered":"2024-01-26T23:29:48.691233+0000","last_unstale":"2024-01-26T23:29:48.696514+0000","last_undegraded":"2024-01-26T23:29:48.696514+0000","last_fullsized":"2024-01-26T23:29:48.696514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:22:12.593080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626042+0000","last_change":"2024-01-26T23:29:43.614006+0000","last_active":"2024-01-26T23:29:48.626042+0000","last_peered":"2024-01-26T23:29:48.626042+0000","last_clean":"2024-01-26T23:29:48.626042+0000","last_became_active":"2024-01-26T23:29:43.613688+0000","last_became_peered":"2024-01-26T23:29:43.613688+0000","last_unstale":"2024-01-26T23:29:48.626042+0000","last_undegraded":"2024-01-26T23:29:48.626042+0000","last_fullsized":"2024-01-26T23:29:48.626042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:30.053494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.639703+0000","last_change":"2024-01-26T23:29:44.618360+0000","last_active":"2024-01-26T23:29:48.639703+0000","last_peered":"2024-01-26T23:29:48.639703+0000","last_clean":"2024-01-26T23:29:48.639703+0000","last_became_active":"2024-01-26T23:29:44.618061+0000","last_became_peered":"2024-01-26T23:29:44.618061+0000","last_unstale":"2024-01-26T23:29:48.639703+0000","last_undegraded":"2024-01-26T23:29:48.639703+0000","last_fullsized":"2024-01-26T23:29:48.639703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:26:26.776664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695399+0000","last_change":"2024-01-26T23:29:48.694453+0000","last_active":"2024-01-26T23:29:48.695399+0000","last_peered":"2024-01-26T23:29:48.695399+0000","last_clean":"2024-01-26T23:29:48.695399+0000","last_became_active":"2024-01-26T23:29:48.693791+0000","last_became_peered":"2024-01-26T23:29:48.693791+0000","last_unstale":"2024-01-26T23:29:48.695399+0000","last_undegraded":"2024-01-26T23:29:48.695399+0000","last_fullsized":"2024-01-26T23:29:48.695399+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:20.246494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.638726+0000","last_change":"2024-01-26T23:29:43.619037+0000","last_active":"2024-01-26T23:29:48.638726+0000","last_peered":"2024-01-26T23:29:48.638726+0000","last_clean":"2024-01-26T23:29:48.638726+0000","last_became_active":"2024-01-26T23:29:43.618822+0000","last_became_peered":"2024-01-26T23:29:43.618822+0000","last_unstale":"2024-01-26T23:29:48.638726+0000","last_undegraded":"2024-01-26T23:29:48.638726+0000","last_fullsized":"2024-01-26T23:29:48.638726+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:30.053494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633637+0000","last_change":"2024-01-26T23:29:44.615872+0000","last_active":"2024-01-26T23:29:48.633637+0000","last_peered":"2024-01-26T23:29:48.633637+0000","last_clean":"2024-01-26T23:29:48.633637+0000","last_became_active":"2024-01-26T23:29:44.615547+0000","last_became_peered":"2024-01-26T23:29:44.615547+0000","last_unstale":"2024-01-26T23:29:48.633637+0000","last_undegraded":"2024-01-26T23:29:48.633637+0000","last_fullsized":"2024-01-26T23:29:48.633637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:04:39.582710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652459+0000","last_change":"2024-01-26T23:29:48.646587+0000","last_active":"2024-01-26T23:29:48.652459+0000","last_peered":"2024-01-26T23:29:48.652459+0000","last_clean":"2024-01-26T23:29:48.652459+0000","last_became_active":"2024-01-26T23:29:48.646117+0000","last_became_peered":"2024-01-26T23:29:48.646117+0000","last_unstale":"2024-01-26T23:29:48.652459+0000","last_undegraded":"2024-01-26T23:29:48.652459+0000","last_fullsized":"2024-01-26T23:29:48.652459+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:53:55.279661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.689527+0000","last_change":"2024-01-26T23:29:43.623735+0000","last_active":"2024-01-26T23:29:48.689527+0000","last_peered":"2024-01-26T23:29:48.689527+0000","last_clean":"2024-01-26T23:29:48.689527+0000","last_became_active":"2024-01-26T23:29:43.623048+0000","last_became_peered":"2024-01-26T23:29:43.623048+0000","last_unstale":"2024-01-26T23:29:48.689527+0000","last_undegraded":"2024-01-26T23:29:48.689527+0000","last_fullsized":"2024-01-26T23:29:48.689527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:47:09.096394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632551+0000","last_change":"2024-01-26T23:29:44.602462+0000","last_active":"2024-01-26T23:29:48.632551+0000","last_peered":"2024-01-26T23:29:48.632551+0000","last_clean":"2024-01-26T23:29:48.632551+0000","last_became_active":"2024-01-26T23:29:44.602007+0000","last_became_peered":"2024-01-26T23:29:44.602007+0000","last_unstale":"2024-01-26T23:29:48.632551+0000","last_undegraded":"2024-01-26T23:29:48.632551+0000","last_fullsized":"2024-01-26T23:29:48.632551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:23:26.050869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695910+0000","last_change":"2024-01-26T23:29:48.691917+0000","last_active":"2024-01-26T23:29:48.695910+0000","last_peered":"2024-01-26T23:29:48.695910+0000","last_clean":"2024-01-26T23:29:48.695910+0000","last_became_active":"2024-01-26T23:29:48.691441+0000","last_became_peered":"2024-01-26T23:29:48.691441+0000","last_unstale":"2024-01-26T23:29:48.695910+0000","last_undegraded":"2024-01-26T23:29:48.695910+0000","last_fullsized":"2024-01-26T23:29:48.695910+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:52:44.817581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.625872+0000","last_change":"2024-01-26T23:29:43.619776+0000","last_active":"2024-01-26T23:29:48.625872+0000","last_peered":"2024-01-26T23:29:48.625872+0000","last_clean":"2024-01-26T23:29:48.625872+0000","last_became_active":"2024-01-26T23:29:43.619374+0000","last_became_peered":"2024-01-26T23:29:43.619374+0000","last_unstale":"2024-01-26T23:29:48.625872+0000","last_undegraded":"2024-01-26T23:29:48.625872+0000","last_fullsized":"2024-01-26T23:29:48.625872+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:32.417441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636819+0000","last_change":"2024-01-26T23:29:44.621339+0000","last_active":"2024-01-26T23:29:48.636819+0000","last_peered":"2024-01-26T23:29:48.636819+0000","last_clean":"2024-01-26T23:29:48.636819+0000","last_became_active":"2024-01-26T23:29:44.621012+0000","last_became_peered":"2024-01-26T23:29:44.621012+0000","last_unstale":"2024-01-26T23:29:48.636819+0000","last_undegraded":"2024-01-26T23:29:48.636819+0000","last_fullsized":"2024-01-26T23:29:48.636819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:20.138786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652798+0000","last_change":"2024-01-26T23:29:48.650063+0000","last_active":"2024-01-26T23:29:48.652798+0000","last_peered":"2024-01-26T23:29:48.652798+0000","last_clean":"2024-01-26T23:29:48.652798+0000","last_became_active":"2024-01-26T23:29:48.649732+0000","last_became_peered":"2024-01-26T23:29:48.649732+0000","last_unstale":"2024-01-26T23:29:48.652798+0000","last_undegraded":"2024-01-26T23:29:48.652798+0000","last_fullsized":"2024-01-26T23:29:48.652798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:22:12.593080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626014+0000","last_change":"2024-01-26T23:29:43.618706+0000","last_active":"2024-01-26T23:29:48.626014+0000","last_peered":"2024-01-26T23:29:48.626014+0000","last_clean":"2024-01-26T23:29:48.626014+0000","last_became_active":"2024-01-26T23:29:43.618353+0000","last_became_peered":"2024-01-26T23:29:43.618353+0000","last_unstale":"2024-01-26T23:29:48.626014+0000","last_undegraded":"2024-01-26T23:29:48.626014+0000","last_fullsized":"2024-01-26T23:29:48.626014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:45.384500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636855+0000","last_change":"2024-01-26T23:29:44.633055+0000","last_active":"2024-01-26T23:29:48.636855+0000","last_peered":"2024-01-26T23:29:48.636855+0000","last_clean":"2024-01-26T23:29:48.636855+0000","last_became_active":"2024-01-26T23:29:44.631874+0000","last_became_peered":"2024-01-26T23:29:44.631874+0000","last_unstale":"2024-01-26T23:29:48.636855+0000","last_undegraded":"2024-01-26T23:29:48.636855+0000","last_fullsized":"2024-01-26T23:29:48.636855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:01.636855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652530+0000","last_change":"2024-01-26T23:29:48.648195+0000","last_active":"2024-01-26T23:29:48.652530+0000","last_peered":"2024-01-26T23:29:48.652530+0000","last_clean":"2024-01-26T23:29:48.652530+0000","last_became_active":"2024-01-26T23:29:48.647780+0000","last_became_peered":"2024-01-26T23:29:48.647780+0000","last_unstale":"2024-01-26T23:29:48.652530+0000","last_undegraded":"2024-01-26T23:29:48.652530+0000","last_fullsized":"2024-01-26T23:29:48.652530+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:52:44.817581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690272+0000","last_change":"2024-01-26T23:29:43.623737+0000","last_active":"2024-01-26T23:29:48.690272+0000","last_peered":"2024-01-26T23:29:48.690272+0000","last_clean":"2024-01-26T23:29:48.690272+0000","last_became_active":"2024-01-26T23:29:43.623439+0000","last_became_peered":"2024-01-26T23:29:43.623439+0000","last_unstale":"2024-01-26T23:29:48.690272+0000","last_undegraded":"2024-01-26T23:29:48.690272+0000","last_fullsized":"2024-01-26T23:29:48.690272+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:45.384500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632872+0000","last_change":"2024-01-26T23:29:44.613570+0000","last_active":"2024-01-26T23:29:48.632872+0000","last_peered":"2024-01-26T23:29:48.632872+0000","last_clean":"2024-01-26T23:29:48.632872+0000","last_became_active":"2024-01-26T23:29:44.613282+0000","last_became_peered":"2024-01-26T23:29:44.613282+0000","last_unstale":"2024-01-26T23:29:48.632872+0000","last_undegraded":"2024-01-26T23:29:48.632872+0000","last_fullsized":"2024-01-26T23:29:48.632872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:12.500308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696078+0000","last_change":"2024-01-26T23:29:48.691625+0000","last_active":"2024-01-26T23:29:48.696078+0000","last_peered":"2024-01-26T23:29:48.696078+0000","last_clean":"2024-01-26T23:29:48.696078+0000","last_became_active":"2024-01-26T23:29:48.691242+0000","last_became_peered":"2024-01-26T23:29:48.691242+0000","last_unstale":"2024-01-26T23:29:48.696078+0000","last_undegraded":"2024-01-26T23:29:48.696078+0000","last_fullsized":"2024-01-26T23:29:48.696078+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:02.358147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633000+0000","last_change":"2024-01-26T23:29:43.622471+0000","last_active":"2024-01-26T23:29:48.633000+0000","last_peered":"2024-01-26T23:29:48.633000+0000","last_clean":"2024-01-26T23:29:48.633000+0000","last_became_active":"2024-01-26T23:29:43.622106+0000","last_became_peered":"2024-01-26T23:29:43.622106+0000","last_unstale":"2024-01-26T23:29:48.633000+0000","last_undegraded":"2024-01-26T23:29:48.633000+0000","last_fullsized":"2024-01-26T23:29:48.633000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:32.417441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690284+0000","last_change":"2024-01-26T23:29:44.623102+0000","last_active":"2024-01-26T23:29:48.690284+0000","last_peered":"2024-01-26T23:29:48.690284+0000","last_clean":"2024-01-26T23:29:48.690284+0000","last_became_active":"2024-01-26T23:29:44.622834+0000","last_became_peered":"2024-01-26T23:29:44.622834+0000","last_unstale":"2024-01-26T23:29:48.690284+0000","last_undegraded":"2024-01-26T23:29:48.690284+0000","last_fullsized":"2024-01-26T23:29:48.690284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:18:55.543376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.651007+0000","last_change":"2024-01-26T23:29:48.644283+0000","last_active":"2024-01-26T23:29:48.651007+0000","last_peered":"2024-01-26T23:29:48.651007+0000","last_clean":"2024-01-26T23:29:48.651007+0000","last_became_active":"2024-01-26T23:29:48.643862+0000","last_became_peered":"2024-01-26T23:29:48.643862+0000","last_unstale":"2024-01-26T23:29:48.651007+0000","last_undegraded":"2024-01-26T23:29:48.651007+0000","last_fullsized":"2024-01-26T23:29:48.651007+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:29:51.471531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626158+0000","last_change":"2024-01-26T23:29:43.613316+0000","last_active":"2024-01-26T23:29:48.626158+0000","last_peered":"2024-01-26T23:29:48.626158+0000","last_clean":"2024-01-26T23:29:48.626158+0000","last_became_active":"2024-01-26T23:29:43.613054+0000","last_became_peered":"2024-01-26T23:29:43.613054+0000","last_unstale":"2024-01-26T23:29:48.626158+0000","last_undegraded":"2024-01-26T23:29:48.626158+0000","last_fullsized":"2024-01-26T23:29:48.626158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:09:46.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636441+0000","last_change":"2024-01-26T23:29:44.631866+0000","last_active":"2024-01-26T23:29:48.636441+0000","last_peered":"2024-01-26T23:29:48.636441+0000","last_clean":"2024-01-26T23:29:48.636441+0000","last_became_active":"2024-01-26T23:29:44.631581+0000","last_became_peered":"2024-01-26T23:29:44.631581+0000","last_unstale":"2024-01-26T23:29:48.636441+0000","last_undegraded":"2024-01-26T23:29:48.636441+0000","last_fullsized":"2024-01-26T23:29:48.636441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:38:52.041592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695113+0000","last_change":"2024-01-26T23:29:48.691846+0000","last_active":"2024-01-26T23:29:48.695113+0000","last_peered":"2024-01-26T23:29:48.695113+0000","last_clean":"2024-01-26T23:29:48.695113+0000","last_became_active":"2024-01-26T23:29:48.691410+0000","last_became_peered":"2024-01-26T23:29:48.691410+0000","last_unstale":"2024-01-26T23:29:48.695113+0000","last_undegraded":"2024-01-26T23:29:48.695113+0000","last_fullsized":"2024-01-26T23:29:48.695113+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:49:28.002810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633357+0000","last_change":"2024-01-26T23:29:43.618244+0000","last_active":"2024-01-26T23:29:48.633357+0000","last_peered":"2024-01-26T23:29:48.633357+0000","last_clean":"2024-01-26T23:29:48.633357+0000","last_became_active":"2024-01-26T23:29:43.617875+0000","last_became_peered":"2024-01-26T23:29:43.617875+0000","last_unstale":"2024-01-26T23:29:48.633357+0000","last_undegraded":"2024-01-26T23:29:48.633357+0000","last_fullsized":"2024-01-26T23:29:48.633357+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:44:34.698447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690732+0000","last_change":"2024-01-26T23:29:44.618128+0000","last_active":"2024-01-26T23:29:48.690732+0000","last_peered":"2024-01-26T23:29:48.690732+0000","last_clean":"2024-01-26T23:29:48.690732+0000","last_became_active":"2024-01-26T23:29:44.617849+0000","last_became_peered":"2024-01-26T23:29:44.617849+0000","last_unstale":"2024-01-26T23:29:48.690732+0000","last_undegraded":"2024-01-26T23:29:48.690732+0000","last_fullsized":"2024-01-26T23:29:48.690732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:20.138786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633526+0000","last_change":"2024-01-26T23:29:44.615453+0000","last_active":"2024-01-26T23:29:48.633526+0000","last_peered":"2024-01-26T23:29:48.633526+0000","last_clean":"2024-01-26T23:29:48.633526+0000","last_became_active":"2024-01-26T23:29:44.615201+0000","last_became_peered":"2024-01-26T23:29:44.615201+0000","last_unstale":"2024-01-26T23:29:48.633526+0000","last_undegraded":"2024-01-26T23:29:48.633526+0000","last_fullsized":"2024-01-26T23:29:48.633526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:38.452904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631336+0000","last_change":"2024-01-26T23:29:44.619769+0000","last_active":"2024-01-26T23:29:48.631336+0000","last_peered":"2024-01-26T23:29:48.631336+0000","last_clean":"2024-01-26T23:29:48.631336+0000","last_became_active":"2024-01-26T23:29:44.619471+0000","last_became_peered":"2024-01-26T23:29:44.619471+0000","last_unstale":"2024-01-26T23:29:48.631336+0000","last_undegraded":"2024-01-26T23:29:48.631336+0000","last_fullsized":"2024-01-26T23:29:48.631336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:47:10.097930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635849+0000","last_change":"2024-01-26T23:29:44.633598+0000","last_active":"2024-01-26T23:29:48.635849+0000","last_peered":"2024-01-26T23:29:48.635849+0000","last_clean":"2024-01-26T23:29:48.635849+0000","last_became_active":"2024-01-26T23:29:44.633320+0000","last_became_peered":"2024-01-26T23:29:44.633320+0000","last_unstale":"2024-01-26T23:29:48.635849+0000","last_undegraded":"2024-01-26T23:29:48.635849+0000","last_fullsized":"2024-01-26T23:29:48.635849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:03.997854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632174+0000","last_change":"2024-01-26T23:29:44.624890+0000","last_active":"2024-01-26T23:29:48.632174+0000","last_peered":"2024-01-26T23:29:48.632174+0000","last_clean":"2024-01-26T23:29:48.632174+0000","last_became_active":"2024-01-26T23:29:44.624615+0000","last_became_peered":"2024-01-26T23:29:44.624615+0000","last_unstale":"2024-01-26T23:29:48.632174+0000","last_undegraded":"2024-01-26T23:29:48.632174+0000","last_fullsized":"2024-01-26T23:29:48.632174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:12:04.973733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637534+0000","last_change":"2024-01-26T23:29:44.618261+0000","last_active":"2024-01-26T23:29:48.637534+0000","last_peered":"2024-01-26T23:29:48.637534+0000","last_clean":"2024-01-26T23:29:48.637534+0000","last_became_active":"2024-01-26T23:29:44.616559+0000","last_became_peered":"2024-01-26T23:29:44.616559+0000","last_unstale":"2024-01-26T23:29:48.637534+0000","last_undegraded":"2024-01-26T23:29:48.637534+0000","last_fullsized":"2024-01-26T23:29:48.637534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:12:04.973733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626013+0000","last_change":"2024-01-26T23:29:44.619408+0000","last_active":"2024-01-26T23:29:48.626013+0000","last_peered":"2024-01-26T23:29:48.626013+0000","last_clean":"2024-01-26T23:29:48.626013+0000","last_became_active":"2024-01-26T23:29:44.619093+0000","last_became_peered":"2024-01-26T23:29:44.619093+0000","last_unstale":"2024-01-26T23:29:48.626013+0000","last_undegraded":"2024-01-26T23:29:48.626013+0000","last_fullsized":"2024-01-26T23:29:48.626013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:35:11.068882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.638008+0000","last_change":"2024-01-26T23:29:44.614999+0000","last_active":"2024-01-26T23:29:48.638008+0000","last_peered":"2024-01-26T23:29:48.638008+0000","last_clean":"2024-01-26T23:29:48.638008+0000","last_became_active":"2024-01-26T23:29:44.614579+0000","last_became_peered":"2024-01-26T23:29:44.614579+0000","last_unstale":"2024-01-26T23:29:48.638008+0000","last_undegraded":"2024-01-26T23:29:48.638008+0000","last_fullsized":"2024-01-26T23:29:48.638008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.639791+0000","last_change":"2024-01-26T23:29:44.612377+0000","last_active":"2024-01-26T23:29:48.639791+0000","last_peered":"2024-01-26T23:29:48.639791+0000","last_clean":"2024-01-26T23:29:48.639791+0000","last_became_active":"2024-01-26T23:29:44.612073+0000","last_became_peered":"2024-01-26T23:29:44.612073+0000","last_unstale":"2024-01-26T23:29:48.639791+0000","last_undegraded":"2024-01-26T23:29:48.639791+0000","last_fullsized":"2024-01-26T23:29:48.639791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:59:47.867303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632846+0000","last_change":"2024-01-26T23:29:44.633571+0000","last_active":"2024-01-26T23:29:48.632846+0000","last_peered":"2024-01-26T23:29:48.632846+0000","last_clean":"2024-01-26T23:29:48.632846+0000","last_became_active":"2024-01-26T23:29:44.633312+0000","last_became_peered":"2024-01-26T23:29:44.633312+0000","last_unstale":"2024-01-26T23:29:48.632846+0000","last_undegraded":"2024-01-26T23:29:48.632846+0000","last_fullsized":"2024-01-26T23:29:48.632846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:22.113035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637064+0000","last_change":"2024-01-26T23:29:44.617840+0000","last_active":"2024-01-26T23:29:48.637064+0000","last_peered":"2024-01-26T23:29:48.637064+0000","last_clean":"2024-01-26T23:29:48.637064+0000","last_became_active":"2024-01-26T23:29:44.611989+0000","last_became_peered":"2024-01-26T23:29:44.611989+0000","last_unstale":"2024-01-26T23:29:48.637064+0000","last_undegraded":"2024-01-26T23:29:48.637064+0000","last_fullsized":"2024-01-26T23:29:48.637064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:14.555745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635947+0000","last_change":"2024-01-26T23:29:44.617326+0000","last_active":"2024-01-26T23:29:48.635947+0000","last_peered":"2024-01-26T23:29:48.635947+0000","last_clean":"2024-01-26T23:29:48.635947+0000","last_became_active":"2024-01-26T23:29:44.617063+0000","last_became_peered":"2024-01-26T23:29:44.617063+0000","last_unstale":"2024-01-26T23:29:48.635947+0000","last_undegraded":"2024-01-26T23:29:48.635947+0000","last_fullsized":"2024-01-26T23:29:48.635947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:01.636855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632268+0000","last_change":"2024-01-26T23:29:44.613645+0000","last_active":"2024-01-26T23:29:48.632268+0000","last_peered":"2024-01-26T23:29:48.632268+0000","last_clean":"2024-01-26T23:29:48.632268+0000","last_became_active":"2024-01-26T23:29:44.613290+0000","last_became_peered":"2024-01-26T23:29:44.613290+0000","last_unstale":"2024-01-26T23:29:48.632268+0000","last_undegraded":"2024-01-26T23:29:48.632268+0000","last_fullsized":"2024-01-26T23:29:48.632268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:12.500308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637188+0000","last_change":"2024-01-26T23:29:44.609934+0000","last_active":"2024-01-26T23:29:48.637188+0000","last_peered":"2024-01-26T23:29:48.637188+0000","last_clean":"2024-01-26T23:29:48.637188+0000","last_became_active":"2024-01-26T23:29:44.609682+0000","last_became_peered":"2024-01-26T23:29:44.609682+0000","last_unstale":"2024-01-26T23:29:48.637188+0000","last_undegraded":"2024-01-26T23:29:48.637188+0000","last_fullsized":"2024-01-26T23:29:48.637188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:12:04.973733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4483,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298120,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073720,"statfs":{"total":96636764160,"available":96331489280,"internally_reserved":0,"allocated":851968,"data_stored":690410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":103201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297544,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074296,"statfs":{"total":96636764160,"available":96332079104,"internally_reserved":0,"allocated":262144,"data_stored":99648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298128,"kb_used_data":840,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073712,"statfs":{"total":96636764160,"available":96331481088,"internally_reserved":0,"allocated":860160,"data_stored":690466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.214}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297536,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074304,"statfs":{"total":96636764160,"available":96332087296,"internally_reserved":0,"allocated":253952,"data_stored":99488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":103201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":99614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297540,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074300,"statfs":{"total":96636764160,"available":96332083200,"internally_reserved":0,"allocated":258048,"data_stored":99578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:30:09.749 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:30:09.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.917+0000 7f8a13be8700 1 Processor -- start 2024-01-26T23:30:09.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.917+0000 7f8a13be8700 1 -- start start 2024-01-26T23:30:09.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.918+0000 7f8a13be8700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8a0c0cb4c0 0x7f8a0c159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:09.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.918+0000 7f8a13be8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0c0c97f0 con 0x7f8a0c1596e0 2024-01-26T23:30:09.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.918+0000 7f8a13be8700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0c0c8c20 con 0x7f8a0c0c4720 2024-01-26T23:30:09.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.918+0000 7f8a13be8700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a0c0ce0d0 con 0x7f8a0c0cb4c0 2024-01-26T23:30:09.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.918+0000 7f8a11984700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8a0c1596e0 0x7f8a0c0cadb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44546/0 (socket says 172.21.15.46:44546) 2024-01-26T23:30:09.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.918+0000 7f8a11984700 1 -- 172.21.15.46:0/161515477 learned_addr learned my addr 172.21.15.46:0/161515477 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:09.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.919+0000 7f8a11183700 1 --2- 172.21.15.46:0/161515477 >> v2:172.21.15.106:3300/0 conn(0x7f8a0c0cb4c0 0x7f8a0c159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.919+0000 7f8a10982700 1 -- 172.21.15.46:0/161515477 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433544282 0 0) 0x7f8a0c0c97f0 con 0x7f8a0c1596e0 2024-01-26T23:30:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.919+0000 7f8a10982700 1 -- 172.21.15.46:0/161515477 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89f4003570 con 0x7f8a0c1596e0 2024-01-26T23:30:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.919+0000 7f8a10982700 1 -- 172.21.15.46:0/161515477 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1603000113 0 0) 0x7f89f4003570 con 0x7f8a0c1596e0 2024-01-26T23:30:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.919+0000 7f8a11183700 1 -- 172.21.15.46:0/161515477 >> v1:172.21.15.106:6789/0 conn(0x7f8a0c0c4720 legacy=0x7f8a0c0cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.919+0000 7f8a11183700 1 -- 172.21.15.46:0/161515477 >> v1:172.21.15.46:6789/0 conn(0x7f8a0c1596e0 legacy=0x7f8a0c0cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.919+0000 7f8a11183700 1 -- 172.21.15.46:0/161515477 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0c0c8ec0 con 0x7f8a0c0cb4c0 2024-01-26T23:30:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.919+0000 7f8a11183700 1 --2- 172.21.15.46:0/161515477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a0c0cb4c0 0x7f8a0c159ab0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8a000098e0 tx=0x7f8a000091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28bcaae151fb351f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.919+0000 7f8a10982700 1 -- 172.21.15.46:0/161515477 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc0017f0 con 0x7f8a0c0cb4c0 2024-01-26T23:30:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.919+0000 7f8a10982700 1 -- 172.21.15.46:0/161515477 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a00011420 con 0x7f8a0c0cb4c0 2024-01-26T23:30:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.919+0000 7f8a10982700 1 -- 172.21.15.46:0/161515477 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a000117c0 con 0x7f8a0c0cb4c0 2024-01-26T23:30:09.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.920+0000 7f8a13be8700 1 -- 172.21.15.46:0/161515477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a0c0cb4c0 msgr2=0x7f8a0c159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.920+0000 7f8a13be8700 1 --2- 172.21.15.46:0/161515477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a0c0cb4c0 0x7f8a0c159ab0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8a000098e0 tx=0x7f8a000091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.920+0000 7f8a13be8700 1 -- 172.21.15.46:0/161515477 shutdown_connections 2024-01-26T23:30:09.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.920+0000 7f8a13be8700 1 --2- 172.21.15.46:0/161515477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a0c0cb4c0 0x7f8a0c159ab0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.920+0000 7f8a13be8700 1 -- 172.21.15.46:0/161515477 >> 172.21.15.46:0/161515477 conn(0x7f8a0c097820 msgr2=0x7f8a0c068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:09.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.920+0000 7f8a13be8700 1 -- 172.21.15.46:0/161515477 shutdown_connections 2024-01-26T23:30:09.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.920+0000 7f8a13be8700 1 -- 172.21.15.46:0/161515477 wait complete. 2024-01-26T23:30:09.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.921+0000 7f8a13be8700 1 Processor -- start 2024-01-26T23:30:09.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.921+0000 7f8a13be8700 1 -- start start 2024-01-26T23:30:09.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.922+0000 7f8a13be8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a0c0c4720 0x7f8a0c147280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:09.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.922+0000 7f8a13be8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c0cb4c0 0x7f8a0c0b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:09.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.922+0000 7f8a13be8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a0c1596e0 0x7f8a0c0b3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:09.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.922+0000 7f8a13be8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c0b1f30 con 0x7f8a0c0cb4c0 2024-01-26T23:30:09.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.922+0000 7f8a13be8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c0b2210 con 0x7f8a0c0c4720 2024-01-26T23:30:09.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.922+0000 7f8a13be8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c0b24f0 con 0x7f8a0c1596e0 2024-01-26T23:30:09.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.922+0000 7f8a11183700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c0cb4c0 0x7f8a0c0b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:09.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.922+0000 7f8a11183700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c0cb4c0 0x7f8a0c0b3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34164/0 (socket says 172.21.15.46:34164) 2024-01-26T23:30:09.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.922+0000 7f8a11183700 1 -- 172.21.15.46:0/606587162 learned_addr learned my addr 172.21.15.46:0/606587162 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:09.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.922+0000 7f8a11984700 1 --2- 172.21.15.46:0/606587162 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a0c0c4720 0x7f8a0c147280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:09.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.922+0000 7f8a12185700 1 --2- 172.21.15.46:0/606587162 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a0c1596e0 0x7f8a0c0b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:09.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.923+0000 7f8a11183700 1 -- 172.21.15.46:0/606587162 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a0c1596e0 msgr2=0x7f8a0c0b3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.923+0000 7f8a11183700 1 --2- 172.21.15.46:0/606587162 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a0c1596e0 0x7f8a0c0b3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.923+0000 7f8a11183700 1 -- 172.21.15.46:0/606587162 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a0c0c4720 msgr2=0x7f8a0c147280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:09.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.923+0000 7f8a11183700 1 --2- 172.21.15.46:0/606587162 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a0c0c4720 0x7f8a0c147280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:09.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.923+0000 7f8a11183700 1 -- 172.21.15.46:0/606587162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a00009420 con 0x7f8a0c0cb4c0 2024-01-26T23:30:09.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.923+0000 7f8a12185700 1 --2- 172.21.15.46:0/606587162 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a0c1596e0 0x7f8a0c0b3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:09.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.923+0000 7f8a11183700 1 --2- 172.21.15.46:0/606587162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c0cb4c0 0x7f8a0c0b3130 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8a000091e0 tx=0x7f8a00016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:09.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.923+0000 7f8a09ffb700 1 -- 172.21.15.46:0/606587162 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0001cbe0 con 0x7f8a0c0cb4c0 2024-01-26T23:30:09.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.923+0000 7f8a09ffb700 1 -- 172.21.15.46:0/606587162 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0001cd60 con 0x7f8a0c0cb4c0 2024-01-26T23:30:09.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.923+0000 7f8a13be8700 1 -- 172.21.15.46:0/606587162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a0c0b27d0 con 0x7f8a0c0cb4c0 2024-01-26T23:30:09.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.923+0000 7f8a09ffb700 1 -- 172.21.15.46:0/606587162 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0001a6a0 con 0x7f8a0c0cb4c0 2024-01-26T23:30:09.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.923+0000 7f8a13be8700 1 -- 172.21.15.46:0/606587162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a0c0b12e0 con 0x7f8a0c0cb4c0 2024-01-26T23:30:09.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.924+0000 7f8a13be8700 1 -- 172.21.15.46:0/606587162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a0c0c5680 con 0x7f8a0c0cb4c0 2024-01-26T23:30:09.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.925+0000 7f8a09ffb700 1 -- 172.21.15.46:0/606587162 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8a0001aa50 con 0x7f8a0c0cb4c0 2024-01-26T23:30:09.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.925+0000 7f8a09ffb700 1 --2- 172.21.15.46:0/606587162 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f89f8066f30 0x7f89f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:09.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.925+0000 7f8a09ffb700 1 -- 172.21.15.46:0/606587162 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f8a00086490 con 0x7f8a0c0cb4c0 2024-01-26T23:30:09.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.925+0000 7f8a11984700 1 --2- 172.21.15.46:0/606587162 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f89f8066f30 0x7f89f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:09.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.926+0000 7f8a11984700 1 --2- 172.21.15.46:0/606587162 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f89f8066f30 0x7f89f80693e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f89fc001560 tx=0x7f89fc000f40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:09.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:09.930+0000 7f8a09ffb700 1 -- 172.21.15.46:0/606587162 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8a0c0c5680 con 0x7f8a0c0cb4c0 2024-01-26T23:30:10.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.132+0000 7f8a13be8700 1 -- 172.21.15.46:0/606587162 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a0c0c5680 con 0x7f89f8066f30 2024-01-26T23:30:10.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.142+0000 7f8a09ffb700 1 -- 172.21.15.46:0/606587162 <== mgr.4102 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345254 (secure 0 0 0) 0x7f8a0c0c5680 con 0x7f89f8066f30 2024-01-26T23:30:10.148 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:10.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.147+0000 7f8a13be8700 1 -- 172.21.15.46:0/606587162 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f89f8066f30 msgr2=0x7f89f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:10.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.147+0000 7f8a13be8700 1 --2- 172.21.15.46:0/606587162 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f89f8066f30 0x7f89f80693e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f89fc001560 tx=0x7f89fc000f40 comp rx=0 tx=0).stop 2024-01-26T23:30:10.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.147+0000 7f8a13be8700 1 -- 172.21.15.46:0/606587162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c0cb4c0 msgr2=0x7f8a0c0b3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:10.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.147+0000 7f8a13be8700 1 --2- 172.21.15.46:0/606587162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c0cb4c0 0x7f8a0c0b3130 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8a000091e0 tx=0x7f8a00016000 comp rx=0 tx=0).stop 2024-01-26T23:30:10.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.147+0000 7f8a13be8700 1 -- 172.21.15.46:0/606587162 shutdown_connections 2024-01-26T23:30:10.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.147+0000 7f8a13be8700 1 --2- 172.21.15.46:0/606587162 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f89f8066f30 0x7f89f80693e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.147+0000 7f8a13be8700 1 --2- 172.21.15.46:0/606587162 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a0c1596e0 0x7f8a0c0b3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.147+0000 7f8a13be8700 1 --2- 172.21.15.46:0/606587162 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a0c0c4720 0x7f8a0c147280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.147+0000 7f8a13be8700 1 --2- 172.21.15.46:0/606587162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c0cb4c0 0x7f8a0c0b3130 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.147+0000 7f8a13be8700 1 -- 172.21.15.46:0/606587162 >> 172.21.15.46:0/606587162 conn(0x7f8a0c097820 msgr2=0x7f8a0c06caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:10.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.147+0000 7f8a13be8700 1 -- 172.21.15.46:0/606587162 shutdown_connections 2024-01-26T23:30:10.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.147+0000 7f8a13be8700 1 -- 172.21.15.46:0/606587162 wait complete. 2024-01-26T23:30:10.151 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:30:10.161 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-01-26T23:30:08.230030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594851,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381512,"kb_used_data":3208,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593208,"statfs":{"total":773094113280,"available":770655444992,"internally_reserved":0,"allocated":3284992,"data_stored":1985606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2177,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002799"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690995+0000","last_change":"2024-01-26T23:29:44.615781+0000","last_active":"2024-01-26T23:29:48.690995+0000","last_peered":"2024-01-26T23:29:48.690995+0000","last_clean":"2024-01-26T23:29:48.690995+0000","last_became_active":"2024-01-26T23:29:44.615522+0000","last_became_peered":"2024-01-26T23:29:44.615522+0000","last_unstale":"2024-01-26T23:29:48.690995+0000","last_undegraded":"2024-01-26T23:29:48.690995+0000","last_fullsized":"2024-01-26T23:29:48.690995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:03.997854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.627221+0000","last_change":"2024-01-26T23:29:44.628874+0000","last_active":"2024-01-26T23:29:48.627221+0000","last_peered":"2024-01-26T23:29:48.627221+0000","last_clean":"2024-01-26T23:29:48.627221+0000","last_became_active":"2024-01-26T23:29:44.628581+0000","last_became_peered":"2024-01-26T23:29:44.628581+0000","last_unstale":"2024-01-26T23:29:48.627221+0000","last_undegraded":"2024-01-26T23:29:48.627221+0000","last_fullsized":"2024-01-26T23:29:48.627221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:14.555745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631189+0000","last_change":"2024-01-26T23:29:44.601978+0000","last_active":"2024-01-26T23:29:48.631189+0000","last_peered":"2024-01-26T23:29:48.631189+0000","last_clean":"2024-01-26T23:29:48.631189+0000","last_became_active":"2024-01-26T23:29:44.601539+0000","last_became_peered":"2024-01-26T23:29:44.601539+0000","last_unstale":"2024-01-26T23:29:48.631189+0000","last_undegraded":"2024-01-26T23:29:48.631189+0000","last_fullsized":"2024-01-26T23:29:48.631189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:35:11.068882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636104+0000","last_change":"2024-01-26T23:29:44.632221+0000","last_active":"2024-01-26T23:29:48.636104+0000","last_peered":"2024-01-26T23:29:48.636104+0000","last_clean":"2024-01-26T23:29:48.636104+0000","last_became_active":"2024-01-26T23:29:44.631611+0000","last_became_peered":"2024-01-26T23:29:44.631611+0000","last_unstale":"2024-01-26T23:29:48.636104+0000","last_undegraded":"2024-01-26T23:29:48.636104+0000","last_fullsized":"2024-01-26T23:29:48.636104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:55:14.606640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690150+0000","last_change":"2024-01-26T23:29:44.616723+0000","last_active":"2024-01-26T23:29:48.690150+0000","last_peered":"2024-01-26T23:29:48.690150+0000","last_clean":"2024-01-26T23:29:48.690150+0000","last_became_active":"2024-01-26T23:29:44.615737+0000","last_became_peered":"2024-01-26T23:29:44.615737+0000","last_unstale":"2024-01-26T23:29:48.690150+0000","last_undegraded":"2024-01-26T23:29:48.690150+0000","last_fullsized":"2024-01-26T23:29:48.690150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635997+0000","last_change":"2024-01-26T23:29:44.622983+0000","last_active":"2024-01-26T23:29:48.635997+0000","last_peered":"2024-01-26T23:29:48.635997+0000","last_clean":"2024-01-26T23:29:48.635997+0000","last_became_active":"2024-01-26T23:29:44.622721+0000","last_became_peered":"2024-01-26T23:29:44.622721+0000","last_unstale":"2024-01-26T23:29:48.635997+0000","last_undegraded":"2024-01-26T23:29:48.635997+0000","last_fullsized":"2024-01-26T23:29:48.635997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690538+0000","last_change":"2024-01-26T23:29:44.620570+0000","last_active":"2024-01-26T23:29:48.690538+0000","last_peered":"2024-01-26T23:29:48.690538+0000","last_clean":"2024-01-26T23:29:48.690538+0000","last_became_active":"2024-01-26T23:29:44.620303+0000","last_became_peered":"2024-01-26T23:29:44.620303+0000","last_unstale":"2024-01-26T23:29:48.690538+0000","last_undegraded":"2024-01-26T23:29:48.690538+0000","last_fullsized":"2024-01-26T23:29:48.690538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:55:14.606640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636953+0000","last_change":"2024-01-26T23:29:44.630873+0000","last_active":"2024-01-26T23:29:48.636953+0000","last_peered":"2024-01-26T23:29:48.636953+0000","last_clean":"2024-01-26T23:29:48.636953+0000","last_became_active":"2024-01-26T23:29:44.630579+0000","last_became_peered":"2024-01-26T23:29:44.630579+0000","last_unstale":"2024-01-26T23:29:48.636953+0000","last_undegraded":"2024-01-26T23:29:48.636953+0000","last_fullsized":"2024-01-26T23:29:48.636953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:48:24.085774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690821+0000","last_change":"2024-01-26T23:29:44.626396+0000","last_active":"2024-01-26T23:29:48.690821+0000","last_peered":"2024-01-26T23:29:48.690821+0000","last_clean":"2024-01-26T23:29:48.690821+0000","last_became_active":"2024-01-26T23:29:44.622150+0000","last_became_peered":"2024-01-26T23:29:44.622150+0000","last_unstale":"2024-01-26T23:29:48.690821+0000","last_undegraded":"2024-01-26T23:29:48.690821+0000","last_fullsized":"2024-01-26T23:29:48.690821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:57:15.558584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637882+0000","last_change":"2024-01-26T23:29:44.618861+0000","last_active":"2024-01-26T23:29:48.637882+0000","last_peered":"2024-01-26T23:29:48.637882+0000","last_clean":"2024-01-26T23:29:48.637882+0000","last_became_active":"2024-01-26T23:29:44.618575+0000","last_became_peered":"2024-01-26T23:29:44.618575+0000","last_unstale":"2024-01-26T23:29:48.637882+0000","last_undegraded":"2024-01-26T23:29:48.637882+0000","last_fullsized":"2024-01-26T23:29:48.637882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:32.912164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637940+0000","last_change":"2024-01-26T23:29:44.617693+0000","last_active":"2024-01-26T23:29:48.637940+0000","last_peered":"2024-01-26T23:29:48.637940+0000","last_clean":"2024-01-26T23:29:48.637940+0000","last_became_active":"2024-01-26T23:29:44.617393+0000","last_became_peered":"2024-01-26T23:29:44.617393+0000","last_unstale":"2024-01-26T23:29:48.637940+0000","last_undegraded":"2024-01-26T23:29:48.637940+0000","last_fullsized":"2024-01-26T23:29:48.637940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633424+0000","last_change":"2024-01-26T23:29:44.614095+0000","last_active":"2024-01-26T23:29:48.633424+0000","last_peered":"2024-01-26T23:29:48.633424+0000","last_clean":"2024-01-26T23:29:48.633424+0000","last_became_active":"2024-01-26T23:29:44.613758+0000","last_became_peered":"2024-01-26T23:29:44.613758+0000","last_unstale":"2024-01-26T23:29:48.633424+0000","last_undegraded":"2024-01-26T23:29:48.633424+0000","last_fullsized":"2024-01-26T23:29:48.633424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:22.392135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636817+0000","last_change":"2024-01-26T23:29:44.624158+0000","last_active":"2024-01-26T23:29:48.636817+0000","last_peered":"2024-01-26T23:29:48.636817+0000","last_clean":"2024-01-26T23:29:48.636817+0000","last_became_active":"2024-01-26T23:29:44.623876+0000","last_became_peered":"2024-01-26T23:29:44.623876+0000","last_unstale":"2024-01-26T23:29:48.636817+0000","last_undegraded":"2024-01-26T23:29:48.636817+0000","last_fullsized":"2024-01-26T23:29:48.636817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631396+0000","last_change":"2024-01-26T23:29:44.618895+0000","last_active":"2024-01-26T23:29:48.631396+0000","last_peered":"2024-01-26T23:29:48.631396+0000","last_clean":"2024-01-26T23:29:48.631396+0000","last_became_active":"2024-01-26T23:29:44.618505+0000","last_became_peered":"2024-01-26T23:29:44.618505+0000","last_unstale":"2024-01-26T23:29:48.631396+0000","last_undegraded":"2024-01-26T23:29:48.631396+0000","last_fullsized":"2024-01-26T23:29:48.631396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:59:47.867303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.625861+0000","last_change":"2024-01-26T23:29:44.612783+0000","last_active":"2024-01-26T23:29:48.625861+0000","last_peered":"2024-01-26T23:29:48.625861+0000","last_clean":"2024-01-26T23:29:48.625861+0000","last_became_active":"2024-01-26T23:29:44.612451+0000","last_became_peered":"2024-01-26T23:29:44.612451+0000","last_unstale":"2024-01-26T23:29:48.625861+0000","last_undegraded":"2024-01-26T23:29:48.625861+0000","last_fullsized":"2024-01-26T23:29:48.625861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:12.500308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632657+0000","last_change":"2024-01-26T23:29:44.630662+0000","last_active":"2024-01-26T23:29:48.632657+0000","last_peered":"2024-01-26T23:29:48.632657+0000","last_clean":"2024-01-26T23:29:48.632657+0000","last_became_active":"2024-01-26T23:29:44.630412+0000","last_became_peered":"2024-01-26T23:29:44.630412+0000","last_unstale":"2024-01-26T23:29:48.632657+0000","last_undegraded":"2024-01-26T23:29:48.632657+0000","last_fullsized":"2024-01-26T23:29:48.632657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636581+0000","last_change":"2024-01-26T23:29:44.624186+0000","last_active":"2024-01-26T23:29:48.636581+0000","last_peered":"2024-01-26T23:29:48.636581+0000","last_clean":"2024-01-26T23:29:48.636581+0000","last_became_active":"2024-01-26T23:29:44.623729+0000","last_became_peered":"2024-01-26T23:29:44.623729+0000","last_unstale":"2024-01-26T23:29:48.636581+0000","last_undegraded":"2024-01-26T23:29:48.636581+0000","last_fullsized":"2024-01-26T23:29:48.636581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:20.138786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633462+0000","last_change":"2024-01-26T23:29:44.615801+0000","last_active":"2024-01-26T23:29:48.633462+0000","last_peered":"2024-01-26T23:29:48.633462+0000","last_clean":"2024-01-26T23:29:48.633462+0000","last_became_active":"2024-01-26T23:29:44.614126+0000","last_became_peered":"2024-01-26T23:29:44.614126+0000","last_unstale":"2024-01-26T23:29:48.633462+0000","last_undegraded":"2024-01-26T23:29:48.633462+0000","last_fullsized":"2024-01-26T23:29:48.633462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:22.392135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690472+0000","last_change":"2024-01-26T23:29:44.620718+0000","last_active":"2024-01-26T23:29:48.690472+0000","last_peered":"2024-01-26T23:29:48.690472+0000","last_clean":"2024-01-26T23:29:48.690472+0000","last_became_active":"2024-01-26T23:29:44.620452+0000","last_became_peered":"2024-01-26T23:29:44.620452+0000","last_unstale":"2024-01-26T23:29:48.690472+0000","last_undegraded":"2024-01-26T23:29:48.690472+0000","last_fullsized":"2024-01-26T23:29:48.690472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:01.636855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.697979+0000","last_change":"2024-01-26T23:29:48.697204+0000","last_active":"2024-01-26T23:29:48.697979+0000","last_peered":"2024-01-26T23:29:48.697979+0000","last_clean":"2024-01-26T23:29:48.697979+0000","last_became_active":"2024-01-26T23:29:48.696727+0000","last_became_peered":"2024-01-26T23:29:48.696727+0000","last_unstale":"2024-01-26T23:29:48.697979+0000","last_undegraded":"2024-01-26T23:29:48.697979+0000","last_fullsized":"2024-01-26T23:29:48.697979+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:59.257622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633767+0000","last_change":"2024-01-26T23:29:43.611980+0000","last_active":"2024-01-26T23:29:48.633767+0000","last_peered":"2024-01-26T23:29:48.633767+0000","last_clean":"2024-01-26T23:29:48.633767+0000","last_became_active":"2024-01-26T23:29:43.611649+0000","last_became_peered":"2024-01-26T23:29:43.611649+0000","last_unstale":"2024-01-26T23:29:48.633767+0000","last_undegraded":"2024-01-26T23:29:48.633767+0000","last_fullsized":"2024-01-26T23:29:48.633767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:31.317009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637215+0000","last_change":"2024-01-26T23:29:44.617315+0000","last_active":"2024-01-26T23:29:48.637215+0000","last_peered":"2024-01-26T23:29:48.637215+0000","last_clean":"2024-01-26T23:29:48.637215+0000","last_became_active":"2024-01-26T23:29:44.617020+0000","last_became_peered":"2024-01-26T23:29:44.617020+0000","last_unstale":"2024-01-26T23:29:48.637215+0000","last_undegraded":"2024-01-26T23:29:48.637215+0000","last_fullsized":"2024-01-26T23:29:48.637215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:21:47.569396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652507+0000","last_change":"2024-01-26T23:29:48.647630+0000","last_active":"2024-01-26T23:29:48.652507+0000","last_peered":"2024-01-26T23:29:48.652507+0000","last_clean":"2024-01-26T23:29:48.652507+0000","last_became_active":"2024-01-26T23:29:48.646094+0000","last_became_peered":"2024-01-26T23:29:48.646094+0000","last_unstale":"2024-01-26T23:29:48.652507+0000","last_undegraded":"2024-01-26T23:29:48.652507+0000","last_fullsized":"2024-01-26T23:29:48.652507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:20.380384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637332+0000","last_change":"2024-01-26T23:29:43.616917+0000","last_active":"2024-01-26T23:29:48.637332+0000","last_peered":"2024-01-26T23:29:48.637332+0000","last_clean":"2024-01-26T23:29:48.637332+0000","last_became_active":"2024-01-26T23:29:43.616606+0000","last_became_peered":"2024-01-26T23:29:43.616606+0000","last_unstale":"2024-01-26T23:29:48.637332+0000","last_undegraded":"2024-01-26T23:29:48.637332+0000","last_fullsized":"2024-01-26T23:29:48.637332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:54:53.198685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633652+0000","last_change":"2024-01-26T23:29:44.626223+0000","last_active":"2024-01-26T23:29:48.633652+0000","last_peered":"2024-01-26T23:29:48.633652+0000","last_clean":"2024-01-26T23:29:48.633652+0000","last_became_active":"2024-01-26T23:29:44.625830+0000","last_became_peered":"2024-01-26T23:29:44.625830+0000","last_unstale":"2024-01-26T23:29:48.633652+0000","last_undegraded":"2024-01-26T23:29:48.633652+0000","last_fullsized":"2024-01-26T23:29:48.633652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652505+0000","last_change":"2024-01-26T23:29:48.647450+0000","last_active":"2024-01-26T23:29:48.652505+0000","last_peered":"2024-01-26T23:29:48.652505+0000","last_clean":"2024-01-26T23:29:48.652505+0000","last_became_active":"2024-01-26T23:29:48.646159+0000","last_became_peered":"2024-01-26T23:29:48.646159+0000","last_unstale":"2024-01-26T23:29:48.652505+0000","last_undegraded":"2024-01-26T23:29:48.652505+0000","last_fullsized":"2024-01-26T23:29:48.652505+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:13:54.119091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690197+0000","last_change":"2024-01-26T23:29:43.624480+0000","last_active":"2024-01-26T23:29:48.690197+0000","last_peered":"2024-01-26T23:29:48.690197+0000","last_clean":"2024-01-26T23:29:48.690197+0000","last_became_active":"2024-01-26T23:29:43.624225+0000","last_became_peered":"2024-01-26T23:29:43.624225+0000","last_unstale":"2024-01-26T23:29:48.690197+0000","last_undegraded":"2024-01-26T23:29:48.690197+0000","last_fullsized":"2024-01-26T23:29:48.690197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:21.390599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636199+0000","last_change":"2024-01-26T23:29:44.621996+0000","last_active":"2024-01-26T23:29:48.636199+0000","last_peered":"2024-01-26T23:29:48.636199+0000","last_clean":"2024-01-26T23:29:48.636199+0000","last_became_active":"2024-01-26T23:29:44.621750+0000","last_became_peered":"2024-01-26T23:29:44.621750+0000","last_unstale":"2024-01-26T23:29:48.636199+0000","last_undegraded":"2024-01-26T23:29:48.636199+0000","last_fullsized":"2024-01-26T23:29:48.636199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:32.912164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652475+0000","last_change":"2024-01-26T23:29:48.647439+0000","last_active":"2024-01-26T23:29:48.652475+0000","last_peered":"2024-01-26T23:29:48.652475+0000","last_clean":"2024-01-26T23:29:48.652475+0000","last_became_active":"2024-01-26T23:29:48.647004+0000","last_became_peered":"2024-01-26T23:29:48.647004+0000","last_unstale":"2024-01-26T23:29:48.652475+0000","last_undegraded":"2024-01-26T23:29:48.652475+0000","last_fullsized":"2024-01-26T23:29:48.652475+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:29:51.471531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690167+0000","last_change":"2024-01-26T23:29:43.623440+0000","last_active":"2024-01-26T23:29:48.690167+0000","last_peered":"2024-01-26T23:29:48.690167+0000","last_clean":"2024-01-26T23:29:48.690167+0000","last_became_active":"2024-01-26T23:29:43.623105+0000","last_became_peered":"2024-01-26T23:29:43.623105+0000","last_unstale":"2024-01-26T23:29:48.690167+0000","last_undegraded":"2024-01-26T23:29:48.690167+0000","last_fullsized":"2024-01-26T23:29:48.690167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:37:22.647734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636762+0000","last_change":"2024-01-26T23:29:44.622275+0000","last_active":"2024-01-26T23:29:48.636762+0000","last_peered":"2024-01-26T23:29:48.636762+0000","last_clean":"2024-01-26T23:29:48.636762+0000","last_became_active":"2024-01-26T23:29:44.621687+0000","last_became_peered":"2024-01-26T23:29:44.621687+0000","last_unstale":"2024-01-26T23:29:48.636762+0000","last_undegraded":"2024-01-26T23:29:48.636762+0000","last_fullsized":"2024-01-26T23:29:48.636762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:12.500308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.653689+0000","last_change":"2024-01-26T23:29:48.652656+0000","last_active":"2024-01-26T23:29:48.653689+0000","last_peered":"2024-01-26T23:29:48.653689+0000","last_clean":"2024-01-26T23:29:48.653689+0000","last_became_active":"2024-01-26T23:29:48.652264+0000","last_became_peered":"2024-01-26T23:29:48.652264+0000","last_unstale":"2024-01-26T23:29:48.653689+0000","last_undegraded":"2024-01-26T23:29:48.653689+0000","last_fullsized":"2024-01-26T23:29:48.653689+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:35:23.165011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636924+0000","last_change":"2024-01-26T23:29:43.620505+0000","last_active":"2024-01-26T23:29:48.636924+0000","last_peered":"2024-01-26T23:29:48.636924+0000","last_clean":"2024-01-26T23:29:48.636924+0000","last_became_active":"2024-01-26T23:29:43.620202+0000","last_became_peered":"2024-01-26T23:29:43.620202+0000","last_unstale":"2024-01-26T23:29:48.636924+0000","last_undegraded":"2024-01-26T23:29:48.636924+0000","last_fullsized":"2024-01-26T23:29:48.636924+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:31.317009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633949+0000","last_change":"2024-01-26T23:29:44.615753+0000","last_active":"2024-01-26T23:29:48.633949+0000","last_peered":"2024-01-26T23:29:48.633949+0000","last_clean":"2024-01-26T23:29:48.633949+0000","last_became_active":"2024-01-26T23:29:44.615469+0000","last_became_peered":"2024-01-26T23:29:44.615469+0000","last_unstale":"2024-01-26T23:29:48.633949+0000","last_undegraded":"2024-01-26T23:29:48.633949+0000","last_fullsized":"2024-01-26T23:29:48.633949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:37:23.649270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.651391+0000","last_change":"2024-01-26T23:29:48.646032+0000","last_active":"2024-01-26T23:29:48.651391+0000","last_peered":"2024-01-26T23:29:48.651391+0000","last_clean":"2024-01-26T23:29:48.651391+0000","last_became_active":"2024-01-26T23:29:48.645627+0000","last_became_peered":"2024-01-26T23:29:48.645627+0000","last_unstale":"2024-01-26T23:29:48.651391+0000","last_undegraded":"2024-01-26T23:29:48.651391+0000","last_fullsized":"2024-01-26T23:29:48.651391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:42.473075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632782+0000","last_change":"2024-01-26T23:29:43.614220+0000","last_active":"2024-01-26T23:29:48.632782+0000","last_peered":"2024-01-26T23:29:48.632782+0000","last_clean":"2024-01-26T23:29:48.632782+0000","last_became_active":"2024-01-26T23:29:43.613886+0000","last_became_peered":"2024-01-26T23:29:43.613886+0000","last_unstale":"2024-01-26T23:29:48.632782+0000","last_undegraded":"2024-01-26T23:29:48.632782+0000","last_fullsized":"2024-01-26T23:29:48.632782+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:09:46.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632076+0000","last_change":"2024-01-26T23:29:44.614249+0000","last_active":"2024-01-26T23:29:48.632076+0000","last_peered":"2024-01-26T23:29:48.632076+0000","last_clean":"2024-01-26T23:29:48.632076+0000","last_became_active":"2024-01-26T23:29:44.613949+0000","last_became_peered":"2024-01-26T23:29:44.613949+0000","last_unstale":"2024-01-26T23:29:48.632076+0000","last_undegraded":"2024-01-26T23:29:48.632076+0000","last_fullsized":"2024-01-26T23:29:48.632076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:14.555745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.651517+0000","last_change":"2024-01-26T23:29:48.647625+0000","last_active":"2024-01-26T23:29:48.651517+0000","last_peered":"2024-01-26T23:29:48.651517+0000","last_clean":"2024-01-26T23:29:48.651517+0000","last_became_active":"2024-01-26T23:29:48.647273+0000","last_became_peered":"2024-01-26T23:29:48.647273+0000","last_unstale":"2024-01-26T23:29:48.651517+0000","last_undegraded":"2024-01-26T23:29:48.651517+0000","last_fullsized":"2024-01-26T23:29:48.651517+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:52:44.817581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636975+0000","last_change":"2024-01-26T23:29:43.616939+0000","last_active":"2024-01-26T23:29:48.636975+0000","last_peered":"2024-01-26T23:29:48.636975+0000","last_clean":"2024-01-26T23:29:48.636975+0000","last_became_active":"2024-01-26T23:29:43.614953+0000","last_became_peered":"2024-01-26T23:29:43.614953+0000","last_unstale":"2024-01-26T23:29:48.636975+0000","last_undegraded":"2024-01-26T23:29:48.636975+0000","last_fullsized":"2024-01-26T23:29:48.636975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:36.035429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636683+0000","last_change":"2024-01-26T23:29:44.621908+0000","last_active":"2024-01-26T23:29:48.636683+0000","last_peered":"2024-01-26T23:29:48.636683+0000","last_clean":"2024-01-26T23:29:48.636683+0000","last_became_active":"2024-01-26T23:29:44.621706+0000","last_became_peered":"2024-01-26T23:29:44.621706+0000","last_unstale":"2024-01-26T23:29:48.636683+0000","last_undegraded":"2024-01-26T23:29:48.636683+0000","last_fullsized":"2024-01-26T23:29:48.636683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.192248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696535+0000","last_change":"2024-01-26T23:29:48.695257+0000","last_active":"2024-01-26T23:29:48.696535+0000","last_peered":"2024-01-26T23:29:48.696535+0000","last_clean":"2024-01-26T23:29:48.696535+0000","last_became_active":"2024-01-26T23:29:48.694835+0000","last_became_peered":"2024-01-26T23:29:48.694835+0000","last_unstale":"2024-01-26T23:29:48.696535+0000","last_undegraded":"2024-01-26T23:29:48.696535+0000","last_fullsized":"2024-01-26T23:29:48.696535+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:53:55.279661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632759+0000","last_change":"2024-01-26T23:29:43.617477+0000","last_active":"2024-01-26T23:29:48.632759+0000","last_peered":"2024-01-26T23:29:48.632759+0000","last_clean":"2024-01-26T23:29:48.632759+0000","last_became_active":"2024-01-26T23:29:43.617014+0000","last_became_peered":"2024-01-26T23:29:43.617014+0000","last_unstale":"2024-01-26T23:29:48.632759+0000","last_undegraded":"2024-01-26T23:29:48.632759+0000","last_fullsized":"2024-01-26T23:29:48.632759+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:32.417441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626943+0000","last_change":"2024-01-26T23:29:44.619133+0000","last_active":"2024-01-26T23:29:48.626943+0000","last_peered":"2024-01-26T23:29:48.626943+0000","last_clean":"2024-01-26T23:29:48.626943+0000","last_became_active":"2024-01-26T23:29:44.618827+0000","last_became_peered":"2024-01-26T23:29:44.618827+0000","last_unstale":"2024-01-26T23:29:48.626943+0000","last_undegraded":"2024-01-26T23:29:48.626943+0000","last_fullsized":"2024-01-26T23:29:48.626943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696639+0000","last_change":"2024-01-26T23:29:48.693971+0000","last_active":"2024-01-26T23:29:48.696639+0000","last_peered":"2024-01-26T23:29:48.696639+0000","last_clean":"2024-01-26T23:29:48.696639+0000","last_became_active":"2024-01-26T23:29:48.693563+0000","last_became_peered":"2024-01-26T23:29:48.693563+0000","last_unstale":"2024-01-26T23:29:48.696639+0000","last_undegraded":"2024-01-26T23:29:48.696639+0000","last_fullsized":"2024-01-26T23:29:48.696639+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:42:42.888059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.638779+0000","last_change":"2024-01-26T23:29:43.622313+0000","last_active":"2024-01-26T23:29:48.638779+0000","last_peered":"2024-01-26T23:29:48.638779+0000","last_clean":"2024-01-26T23:29:48.638779+0000","last_became_active":"2024-01-26T23:29:43.622008+0000","last_became_peered":"2024-01-26T23:29:43.622008+0000","last_unstale":"2024-01-26T23:29:48.638779+0000","last_undegraded":"2024-01-26T23:29:48.638779+0000","last_fullsized":"2024-01-26T23:29:48.638779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:37:22.647734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.634360+0000","last_change":"2024-01-26T23:29:44.619813+0000","last_active":"2024-01-26T23:29:48.634360+0000","last_peered":"2024-01-26T23:29:48.634360+0000","last_clean":"2024-01-26T23:29:48.634360+0000","last_became_active":"2024-01-26T23:29:44.619466+0000","last_became_peered":"2024-01-26T23:29:44.619466+0000","last_unstale":"2024-01-26T23:29:48.634360+0000","last_undegraded":"2024-01-26T23:29:48.634360+0000","last_fullsized":"2024-01-26T23:29:48.634360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:48:24.085774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.653660+0000","last_change":"2024-01-26T23:29:48.652676+0000","last_active":"2024-01-26T23:29:48.653660+0000","last_peered":"2024-01-26T23:29:48.653660+0000","last_clean":"2024-01-26T23:29:48.653660+0000","last_became_active":"2024-01-26T23:29:48.652134+0000","last_became_peered":"2024-01-26T23:29:48.652134+0000","last_unstale":"2024-01-26T23:29:48.653660+0000","last_undegraded":"2024-01-26T23:29:48.653660+0000","last_fullsized":"2024-01-26T23:29:48.653660+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:27.746532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.634232+0000","last_change":"2024-01-26T23:29:43.620804+0000","last_active":"2024-01-26T23:29:48.634232+0000","last_peered":"2024-01-26T23:29:48.634232+0000","last_clean":"2024-01-26T23:29:48.634232+0000","last_became_active":"2024-01-26T23:29:43.620530+0000","last_became_peered":"2024-01-26T23:29:43.620530+0000","last_unstale":"2024-01-26T23:29:48.634232+0000","last_undegraded":"2024-01-26T23:29:48.634232+0000","last_fullsized":"2024-01-26T23:29:48.634232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:44:34.698447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.634272+0000","last_change":"2024-01-26T23:29:44.626105+0000","last_active":"2024-01-26T23:29:48.634272+0000","last_peered":"2024-01-26T23:29:48.634272+0000","last_clean":"2024-01-26T23:29:48.634272+0000","last_became_active":"2024-01-26T23:29:44.625500+0000","last_became_peered":"2024-01-26T23:29:44.625500+0000","last_unstale":"2024-01-26T23:29:48.634272+0000","last_undegraded":"2024-01-26T23:29:48.634272+0000","last_fullsized":"2024-01-26T23:29:48.634272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.644865+0000","last_change":"2024-01-26T23:29:48.643281+0000","last_active":"2024-01-26T23:29:48.644865+0000","last_peered":"2024-01-26T23:29:48.644865+0000","last_clean":"2024-01-26T23:29:48.644865+0000","last_became_active":"2024-01-26T23:29:48.642907+0000","last_became_peered":"2024-01-26T23:29:48.642907+0000","last_unstale":"2024-01-26T23:29:48.644865+0000","last_undegraded":"2024-01-26T23:29:48.644865+0000","last_fullsized":"2024-01-26T23:29:48.644865+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:24:23.994084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626216+0000","last_change":"2024-01-26T23:29:43.621999+0000","last_active":"2024-01-26T23:29:48.626216+0000","last_peered":"2024-01-26T23:29:48.626216+0000","last_clean":"2024-01-26T23:29:48.626216+0000","last_became_active":"2024-01-26T23:29:43.621214+0000","last_became_peered":"2024-01-26T23:29:43.621214+0000","last_unstale":"2024-01-26T23:29:48.626216+0000","last_undegraded":"2024-01-26T23:29:48.626216+0000","last_fullsized":"2024-01-26T23:29:48.626216+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:37:22.647734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633819+0000","last_change":"2024-01-26T23:29:44.608941+0000","last_active":"2024-01-26T23:29:48.633819+0000","last_peered":"2024-01-26T23:29:48.633819+0000","last_clean":"2024-01-26T23:29:48.633819+0000","last_became_active":"2024-01-26T23:29:44.608664+0000","last_became_peered":"2024-01-26T23:29:44.608664+0000","last_unstale":"2024-01-26T23:29:48.633819+0000","last_undegraded":"2024-01-26T23:29:48.633819+0000","last_fullsized":"2024-01-26T23:29:48.633819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:47:10.097930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.651674+0000","last_change":"2024-01-26T23:29:48.643876+0000","last_active":"2024-01-26T23:29:48.651674+0000","last_peered":"2024-01-26T23:29:48.651674+0000","last_clean":"2024-01-26T23:29:48.651674+0000","last_became_active":"2024-01-26T23:29:48.641104+0000","last_became_peered":"2024-01-26T23:29:48.641104+0000","last_unstale":"2024-01-26T23:29:48.651674+0000","last_undegraded":"2024-01-26T23:29:48.651674+0000","last_fullsized":"2024-01-26T23:29:48.651674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:29:51.471531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-01-26T23:29:54.992720+0000","last_change":"2024-01-26T23:29:43.622211+0000","last_active":"2024-01-26T23:29:54.992720+0000","last_peered":"2024-01-26T23:29:54.992720+0000","last_clean":"2024-01-26T23:29:54.992720+0000","last_became_active":"2024-01-26T23:29:43.621919+0000","last_became_peered":"2024-01-26T23:29:43.621919+0000","last_unstale":"2024-01-26T23:29:54.992720+0000","last_undegraded":"2024-01-26T23:29:54.992720+0000","last_fullsized":"2024-01-26T23:29:54.992720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:45.384500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3713,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690317+0000","last_change":"2024-01-26T23:29:44.618740+0000","last_active":"2024-01-26T23:29:48.690317+0000","last_peered":"2024-01-26T23:29:48.690317+0000","last_clean":"2024-01-26T23:29:48.690317+0000","last_became_active":"2024-01-26T23:29:44.618482+0000","last_became_peered":"2024-01-26T23:29:44.618482+0000","last_unstale":"2024-01-26T23:29:48.690317+0000","last_undegraded":"2024-01-26T23:29:48.690317+0000","last_fullsized":"2024-01-26T23:29:48.690317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:03.314483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695102+0000","last_change":"2024-01-26T23:29:48.691860+0000","last_active":"2024-01-26T23:29:48.695102+0000","last_peered":"2024-01-26T23:29:48.695102+0000","last_clean":"2024-01-26T23:29:48.695102+0000","last_became_active":"2024-01-26T23:29:48.691440+0000","last_became_peered":"2024-01-26T23:29:48.691440+0000","last_unstale":"2024-01-26T23:29:48.695102+0000","last_undegraded":"2024-01-26T23:29:48.695102+0000","last_fullsized":"2024-01-26T23:29:48.695102+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:20.246494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632364+0000","last_change":"2024-01-26T23:29:43.613477+0000","last_active":"2024-01-26T23:29:48.632364+0000","last_peered":"2024-01-26T23:29:48.632364+0000","last_clean":"2024-01-26T23:29:48.632364+0000","last_became_active":"2024-01-26T23:29:43.613228+0000","last_became_peered":"2024-01-26T23:29:43.613228+0000","last_unstale":"2024-01-26T23:29:48.632364+0000","last_undegraded":"2024-01-26T23:29:48.632364+0000","last_fullsized":"2024-01-26T23:29:48.632364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:54:53.198685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631483+0000","last_change":"2024-01-26T23:29:44.613649+0000","last_active":"2024-01-26T23:29:48.631483+0000","last_peered":"2024-01-26T23:29:48.631483+0000","last_clean":"2024-01-26T23:29:48.631483+0000","last_became_active":"2024-01-26T23:29:44.613278+0000","last_became_peered":"2024-01-26T23:29:44.613278+0000","last_unstale":"2024-01-26T23:29:48.631483+0000","last_undegraded":"2024-01-26T23:29:48.631483+0000","last_fullsized":"2024-01-26T23:29:48.631483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:38.452904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696548+0000","last_change":"2024-01-26T23:29:48.691601+0000","last_active":"2024-01-26T23:29:48.696548+0000","last_peered":"2024-01-26T23:29:48.696548+0000","last_clean":"2024-01-26T23:29:48.696548+0000","last_became_active":"2024-01-26T23:29:48.691169+0000","last_became_peered":"2024-01-26T23:29:48.691169+0000","last_unstale":"2024-01-26T23:29:48.696548+0000","last_undegraded":"2024-01-26T23:29:48.696548+0000","last_fullsized":"2024-01-26T23:29:48.696548+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:09.701112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635763+0000","last_change":"2024-01-26T23:29:43.622165+0000","last_active":"2024-01-26T23:29:48.635763+0000","last_peered":"2024-01-26T23:29:48.635763+0000","last_clean":"2024-01-26T23:29:48.635763+0000","last_became_active":"2024-01-26T23:29:43.621876+0000","last_became_peered":"2024-01-26T23:29:43.621876+0000","last_unstale":"2024-01-26T23:29:48.635763+0000","last_undegraded":"2024-01-26T23:29:48.635763+0000","last_fullsized":"2024-01-26T23:29:48.635763+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:30.053494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690214+0000","last_change":"2024-01-26T23:29:44.615999+0000","last_active":"2024-01-26T23:29:48.690214+0000","last_peered":"2024-01-26T23:29:48.690214+0000","last_clean":"2024-01-26T23:29:48.690214+0000","last_became_active":"2024-01-26T23:29:44.615376+0000","last_became_peered":"2024-01-26T23:29:44.615376+0000","last_unstale":"2024-01-26T23:29:48.690214+0000","last_undegraded":"2024-01-26T23:29:48.690214+0000","last_fullsized":"2024-01-26T23:29:48.690214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:32.912164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696674+0000","last_change":"2024-01-26T23:29:48.694846+0000","last_active":"2024-01-26T23:29:48.696674+0000","last_peered":"2024-01-26T23:29:48.696674+0000","last_clean":"2024-01-26T23:29:48.696674+0000","last_became_active":"2024-01-26T23:29:48.694439+0000","last_became_peered":"2024-01-26T23:29:48.694439+0000","last_unstale":"2024-01-26T23:29:48.696674+0000","last_undegraded":"2024-01-26T23:29:48.696674+0000","last_fullsized":"2024-01-26T23:29:48.696674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:39:25.688252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632550+0000","last_change":"2024-01-26T23:29:43.613829+0000","last_active":"2024-01-26T23:29:48.632550+0000","last_peered":"2024-01-26T23:29:48.632550+0000","last_clean":"2024-01-26T23:29:48.632550+0000","last_became_active":"2024-01-26T23:29:43.613486+0000","last_became_peered":"2024-01-26T23:29:43.613486+0000","last_unstale":"2024-01-26T23:29:48.632550+0000","last_undegraded":"2024-01-26T23:29:48.632550+0000","last_fullsized":"2024-01-26T23:29:48.632550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:44:34.698447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633466+0000","last_change":"2024-01-26T23:29:44.614404+0000","last_active":"2024-01-26T23:29:48.633466+0000","last_peered":"2024-01-26T23:29:48.633466+0000","last_clean":"2024-01-26T23:29:48.633466+0000","last_became_active":"2024-01-26T23:29:44.614127+0000","last_became_peered":"2024-01-26T23:29:44.614127+0000","last_unstale":"2024-01-26T23:29:48.633466+0000","last_undegraded":"2024-01-26T23:29:48.633466+0000","last_fullsized":"2024-01-26T23:29:48.633466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:35:11.068882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.653735+0000","last_change":"2024-01-26T23:29:48.647303+0000","last_active":"2024-01-26T23:29:48.653735+0000","last_peered":"2024-01-26T23:29:48.653735+0000","last_clean":"2024-01-26T23:29:48.653735+0000","last_became_active":"2024-01-26T23:29:48.646893+0000","last_became_peered":"2024-01-26T23:29:48.646893+0000","last_unstale":"2024-01-26T23:29:48.653735+0000","last_undegraded":"2024-01-26T23:29:48.653735+0000","last_fullsized":"2024-01-26T23:29:48.653735+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:49.464111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631121+0000","last_change":"2024-01-26T23:29:43.610693+0000","last_active":"2024-01-26T23:29:48.631121+0000","last_peered":"2024-01-26T23:29:48.631121+0000","last_clean":"2024-01-26T23:29:48.631121+0000","last_became_active":"2024-01-26T23:29:43.610370+0000","last_became_peered":"2024-01-26T23:29:43.610370+0000","last_unstale":"2024-01-26T23:29:48.631121+0000","last_undegraded":"2024-01-26T23:29:48.631121+0000","last_fullsized":"2024-01-26T23:29:48.631121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:09:46.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.692648+0000","last_change":"2024-01-26T23:29:44.611702+0000","last_active":"2024-01-26T23:29:48.692648+0000","last_peered":"2024-01-26T23:29:48.692648+0000","last_clean":"2024-01-26T23:29:48.692648+0000","last_became_active":"2024-01-26T23:29:44.611378+0000","last_became_peered":"2024-01-26T23:29:44.611378+0000","last_unstale":"2024-01-26T23:29:48.692648+0000","last_undegraded":"2024-01-26T23:29:48.692648+0000","last_fullsized":"2024-01-26T23:29:48.692648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:38:52.041592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.644666+0000","last_change":"2024-01-26T23:29:48.643205+0000","last_active":"2024-01-26T23:29:48.644666+0000","last_peered":"2024-01-26T23:29:48.644666+0000","last_clean":"2024-01-26T23:29:48.644666+0000","last_became_active":"2024-01-26T23:29:48.642819+0000","last_became_peered":"2024-01-26T23:29:48.642819+0000","last_unstale":"2024-01-26T23:29:48.644666+0000","last_undegraded":"2024-01-26T23:29:48.644666+0000","last_fullsized":"2024-01-26T23:29:48.644666+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:51.210803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633091+0000","last_change":"2024-01-26T23:29:43.612609+0000","last_active":"2024-01-26T23:29:48.633091+0000","last_peered":"2024-01-26T23:29:48.633091+0000","last_clean":"2024-01-26T23:29:48.633091+0000","last_became_active":"2024-01-26T23:29:43.612318+0000","last_became_peered":"2024-01-26T23:29:43.612318+0000","last_unstale":"2024-01-26T23:29:48.633091+0000","last_undegraded":"2024-01-26T23:29:48.633091+0000","last_fullsized":"2024-01-26T23:29:48.633091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:31.317009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690843+0000","last_change":"2024-01-26T23:29:44.616632+0000","last_active":"2024-01-26T23:29:48.690843+0000","last_peered":"2024-01-26T23:29:48.690843+0000","last_clean":"2024-01-26T23:29:48.690843+0000","last_became_active":"2024-01-26T23:29:44.616280+0000","last_became_peered":"2024-01-26T23:29:44.616280+0000","last_unstale":"2024-01-26T23:29:48.690843+0000","last_undegraded":"2024-01-26T23:29:48.690843+0000","last_fullsized":"2024-01-26T23:29:48.690843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:48:24.085774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.654458+0000","last_change":"2024-01-26T23:29:48.644846+0000","last_active":"2024-01-26T23:29:48.654458+0000","last_peered":"2024-01-26T23:29:48.654458+0000","last_clean":"2024-01-26T23:29:48.654458+0000","last_became_active":"2024-01-26T23:29:48.644359+0000","last_became_peered":"2024-01-26T23:29:48.644359+0000","last_unstale":"2024-01-26T23:29:48.654458+0000","last_undegraded":"2024-01-26T23:29:48.654458+0000","last_fullsized":"2024-01-26T23:29:48.654458+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:59.257622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626696+0000","last_change":"2024-01-26T23:29:43.605084+0000","last_active":"2024-01-26T23:29:48.626696+0000","last_peered":"2024-01-26T23:29:48.626696+0000","last_clean":"2024-01-26T23:29:48.626696+0000","last_became_active":"2024-01-26T23:29:43.604761+0000","last_became_peered":"2024-01-26T23:29:43.604761+0000","last_unstale":"2024-01-26T23:29:48.626696+0000","last_undegraded":"2024-01-26T23:29:48.626696+0000","last_fullsized":"2024-01-26T23:29:48.626696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:44:34.698447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636936+0000","last_change":"2024-01-26T23:29:44.632161+0000","last_active":"2024-01-26T23:29:48.636936+0000","last_peered":"2024-01-26T23:29:48.636936+0000","last_clean":"2024-01-26T23:29:48.636936+0000","last_became_active":"2024-01-26T23:29:44.631945+0000","last_became_peered":"2024-01-26T23:29:44.631945+0000","last_unstale":"2024-01-26T23:29:48.636936+0000","last_undegraded":"2024-01-26T23:29:48.636936+0000","last_fullsized":"2024-01-26T23:29:48.636936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:03.314483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.654064+0000","last_change":"2024-01-26T23:29:48.647375+0000","last_active":"2024-01-26T23:29:48.654064+0000","last_peered":"2024-01-26T23:29:48.654064+0000","last_clean":"2024-01-26T23:29:48.654064+0000","last_became_active":"2024-01-26T23:29:48.646978+0000","last_became_peered":"2024-01-26T23:29:48.646978+0000","last_unstale":"2024-01-26T23:29:48.654064+0000","last_undegraded":"2024-01-26T23:29:48.654064+0000","last_fullsized":"2024-01-26T23:29:48.654064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:20.246494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635696+0000","last_change":"2024-01-26T23:29:43.620942+0000","last_active":"2024-01-26T23:29:48.635696+0000","last_peered":"2024-01-26T23:29:48.635696+0000","last_clean":"2024-01-26T23:29:48.635696+0000","last_became_active":"2024-01-26T23:29:43.620644+0000","last_became_peered":"2024-01-26T23:29:43.620644+0000","last_unstale":"2024-01-26T23:29:48.635696+0000","last_undegraded":"2024-01-26T23:29:48.635696+0000","last_fullsized":"2024-01-26T23:29:48.635696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:45.384500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.634352+0000","last_change":"2024-01-26T23:29:44.605909+0000","last_active":"2024-01-26T23:29:48.634352+0000","last_peered":"2024-01-26T23:29:48.634352+0000","last_clean":"2024-01-26T23:29:48.634352+0000","last_became_active":"2024-01-26T23:29:44.605652+0000","last_became_peered":"2024-01-26T23:29:44.605652+0000","last_unstale":"2024-01-26T23:29:48.634352+0000","last_undegraded":"2024-01-26T23:29:48.634352+0000","last_fullsized":"2024-01-26T23:29:48.634352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:31.055030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.654078+0000","last_change":"2024-01-26T23:29:48.647396+0000","last_active":"2024-01-26T23:29:48.654078+0000","last_peered":"2024-01-26T23:29:48.654078+0000","last_clean":"2024-01-26T23:29:48.654078+0000","last_became_active":"2024-01-26T23:29:48.646950+0000","last_became_peered":"2024-01-26T23:29:48.646950+0000","last_unstale":"2024-01-26T23:29:48.654078+0000","last_undegraded":"2024-01-26T23:29:48.654078+0000","last_fullsized":"2024-01-26T23:29:48.654078+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:51.210803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.625864+0000","last_change":"2024-01-26T23:29:44.617670+0000","last_active":"2024-01-26T23:29:48.625864+0000","last_peered":"2024-01-26T23:29:48.625864+0000","last_clean":"2024-01-26T23:29:48.625864+0000","last_became_active":"2024-01-26T23:29:44.616510+0000","last_became_peered":"2024-01-26T23:29:44.616510+0000","last_unstale":"2024-01-26T23:29:48.625864+0000","last_undegraded":"2024-01-26T23:29:48.625864+0000","last_fullsized":"2024-01-26T23:29:48.625864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:12:04.973733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631229+0000","last_change":"2024-01-26T23:29:43.618892+0000","last_active":"2024-01-26T23:29:48.631229+0000","last_peered":"2024-01-26T23:29:48.631229+0000","last_clean":"2024-01-26T23:29:48.631229+0000","last_became_active":"2024-01-26T23:29:43.618446+0000","last_became_peered":"2024-01-26T23:29:43.618446+0000","last_unstale":"2024-01-26T23:29:48.631229+0000","last_undegraded":"2024-01-26T23:29:48.631229+0000","last_fullsized":"2024-01-26T23:29:48.631229+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:36.035429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.694860+0000","last_change":"2024-01-26T23:29:48.691567+0000","last_active":"2024-01-26T23:29:48.694860+0000","last_peered":"2024-01-26T23:29:48.694860+0000","last_clean":"2024-01-26T23:29:48.694860+0000","last_became_active":"2024-01-26T23:29:48.691153+0000","last_became_peered":"2024-01-26T23:29:48.691153+0000","last_unstale":"2024-01-26T23:29:48.694860+0000","last_undegraded":"2024-01-26T23:29:48.694860+0000","last_fullsized":"2024-01-26T23:29:48.694860+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:49.464111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637181+0000","last_change":"2024-01-26T23:29:44.620961+0000","last_active":"2024-01-26T23:29:48.637181+0000","last_peered":"2024-01-26T23:29:48.637181+0000","last_clean":"2024-01-26T23:29:48.637181+0000","last_became_active":"2024-01-26T23:29:44.620701+0000","last_became_peered":"2024-01-26T23:29:44.620701+0000","last_unstale":"2024-01-26T23:29:48.637181+0000","last_undegraded":"2024-01-26T23:29:48.637181+0000","last_fullsized":"2024-01-26T23:29:48.637181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:32.912164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635054+0000","last_change":"2024-01-26T23:29:43.620894+0000","last_active":"2024-01-26T23:29:48.635054+0000","last_peered":"2024-01-26T23:29:48.635054+0000","last_clean":"2024-01-26T23:29:48.635054+0000","last_became_active":"2024-01-26T23:29:43.620603+0000","last_became_peered":"2024-01-26T23:29:43.620603+0000","last_unstale":"2024-01-26T23:29:48.635054+0000","last_undegraded":"2024-01-26T23:29:48.635054+0000","last_fullsized":"2024-01-26T23:29:48.635054+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:36.035429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.689568+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:29:48.689568+0000","last_peered":"2024-01-26T23:29:48.689568+0000","last_clean":"2024-01-26T23:29:48.689568+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:29:48.689568+0000","last_undegraded":"2024-01-26T23:29:48.689568+0000","last_fullsized":"2024-01-26T23:29:48.689568+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696515+0000","last_change":"2024-01-26T23:29:48.692933+0000","last_active":"2024-01-26T23:29:48.696515+0000","last_peered":"2024-01-26T23:29:48.696515+0000","last_clean":"2024-01-26T23:29:48.696515+0000","last_became_active":"2024-01-26T23:29:48.692576+0000","last_became_peered":"2024-01-26T23:29:48.692576+0000","last_unstale":"2024-01-26T23:29:48.696515+0000","last_undegraded":"2024-01-26T23:29:48.696515+0000","last_fullsized":"2024-01-26T23:29:48.696515+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:24:23.994084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631114+0000","last_change":"2024-01-26T23:29:44.601856+0000","last_active":"2024-01-26T23:29:48.631114+0000","last_peered":"2024-01-26T23:29:48.631114+0000","last_clean":"2024-01-26T23:29:48.631114+0000","last_became_active":"2024-01-26T23:29:44.601367+0000","last_became_peered":"2024-01-26T23:29:44.601367+0000","last_unstale":"2024-01-26T23:29:48.631114+0000","last_undegraded":"2024-01-26T23:29:48.631114+0000","last_fullsized":"2024-01-26T23:29:48.631114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:21:47.569396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.689624+0000","last_change":"2024-01-26T23:29:43.623719+0000","last_active":"2024-01-26T23:29:48.689624+0000","last_peered":"2024-01-26T23:29:48.689624+0000","last_clean":"2024-01-26T23:29:48.689624+0000","last_became_active":"2024-01-26T23:29:43.623493+0000","last_became_peered":"2024-01-26T23:29:43.623493+0000","last_unstale":"2024-01-26T23:29:48.689624+0000","last_undegraded":"2024-01-26T23:29:48.689624+0000","last_fullsized":"2024-01-26T23:29:48.689624+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:23:25.049333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695757+0000","last_change":"2024-01-26T23:29:48.691446+0000","last_active":"2024-01-26T23:29:48.695757+0000","last_peered":"2024-01-26T23:29:48.695757+0000","last_clean":"2024-01-26T23:29:48.695757+0000","last_became_active":"2024-01-26T23:29:48.691005+0000","last_became_peered":"2024-01-26T23:29:48.691005+0000","last_unstale":"2024-01-26T23:29:48.695757+0000","last_undegraded":"2024-01-26T23:29:48.695757+0000","last_fullsized":"2024-01-26T23:29:48.695757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:51.210803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690165+0000","last_change":"2024-01-26T23:29:43.624304+0000","last_active":"2024-01-26T23:29:48.690165+0000","last_peered":"2024-01-26T23:29:48.690165+0000","last_clean":"2024-01-26T23:29:48.690165+0000","last_became_active":"2024-01-26T23:29:43.624031+0000","last_became_peered":"2024-01-26T23:29:43.624031+0000","last_unstale":"2024-01-26T23:29:48.690165+0000","last_undegraded":"2024-01-26T23:29:48.690165+0000","last_fullsized":"2024-01-26T23:29:48.690165+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:32.417441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626845+0000","last_change":"2024-01-26T23:29:44.615706+0000","last_active":"2024-01-26T23:29:48.626845+0000","last_peered":"2024-01-26T23:29:48.626845+0000","last_clean":"2024-01-26T23:29:48.626845+0000","last_became_active":"2024-01-26T23:29:44.615377+0000","last_became_peered":"2024-01-26T23:29:44.615377+0000","last_unstale":"2024-01-26T23:29:48.626845+0000","last_undegraded":"2024-01-26T23:29:48.626845+0000","last_fullsized":"2024-01-26T23:29:48.626845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:38.452904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696514+0000","last_change":"2024-01-26T23:29:48.691661+0000","last_active":"2024-01-26T23:29:48.696514+0000","last_peered":"2024-01-26T23:29:48.696514+0000","last_clean":"2024-01-26T23:29:48.696514+0000","last_became_active":"2024-01-26T23:29:48.691233+0000","last_became_peered":"2024-01-26T23:29:48.691233+0000","last_unstale":"2024-01-26T23:29:48.696514+0000","last_undegraded":"2024-01-26T23:29:48.696514+0000","last_fullsized":"2024-01-26T23:29:48.696514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:22:12.593080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626042+0000","last_change":"2024-01-26T23:29:43.614006+0000","last_active":"2024-01-26T23:29:48.626042+0000","last_peered":"2024-01-26T23:29:48.626042+0000","last_clean":"2024-01-26T23:29:48.626042+0000","last_became_active":"2024-01-26T23:29:43.613688+0000","last_became_peered":"2024-01-26T23:29:43.613688+0000","last_unstale":"2024-01-26T23:29:48.626042+0000","last_undegraded":"2024-01-26T23:29:48.626042+0000","last_fullsized":"2024-01-26T23:29:48.626042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:30.053494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.639703+0000","last_change":"2024-01-26T23:29:44.618360+0000","last_active":"2024-01-26T23:29:48.639703+0000","last_peered":"2024-01-26T23:29:48.639703+0000","last_clean":"2024-01-26T23:29:48.639703+0000","last_became_active":"2024-01-26T23:29:44.618061+0000","last_became_peered":"2024-01-26T23:29:44.618061+0000","last_unstale":"2024-01-26T23:29:48.639703+0000","last_undegraded":"2024-01-26T23:29:48.639703+0000","last_fullsized":"2024-01-26T23:29:48.639703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:26:26.776664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695399+0000","last_change":"2024-01-26T23:29:48.694453+0000","last_active":"2024-01-26T23:29:48.695399+0000","last_peered":"2024-01-26T23:29:48.695399+0000","last_clean":"2024-01-26T23:29:48.695399+0000","last_became_active":"2024-01-26T23:29:48.693791+0000","last_became_peered":"2024-01-26T23:29:48.693791+0000","last_unstale":"2024-01-26T23:29:48.695399+0000","last_undegraded":"2024-01-26T23:29:48.695399+0000","last_fullsized":"2024-01-26T23:29:48.695399+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:20.246494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.638726+0000","last_change":"2024-01-26T23:29:43.619037+0000","last_active":"2024-01-26T23:29:48.638726+0000","last_peered":"2024-01-26T23:29:48.638726+0000","last_clean":"2024-01-26T23:29:48.638726+0000","last_became_active":"2024-01-26T23:29:43.618822+0000","last_became_peered":"2024-01-26T23:29:43.618822+0000","last_unstale":"2024-01-26T23:29:48.638726+0000","last_undegraded":"2024-01-26T23:29:48.638726+0000","last_fullsized":"2024-01-26T23:29:48.638726+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:30.053494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633637+0000","last_change":"2024-01-26T23:29:44.615872+0000","last_active":"2024-01-26T23:29:48.633637+0000","last_peered":"2024-01-26T23:29:48.633637+0000","last_clean":"2024-01-26T23:29:48.633637+0000","last_became_active":"2024-01-26T23:29:44.615547+0000","last_became_peered":"2024-01-26T23:29:44.615547+0000","last_unstale":"2024-01-26T23:29:48.633637+0000","last_undegraded":"2024-01-26T23:29:48.633637+0000","last_fullsized":"2024-01-26T23:29:48.633637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:04:39.582710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652459+0000","last_change":"2024-01-26T23:29:48.646587+0000","last_active":"2024-01-26T23:29:48.652459+0000","last_peered":"2024-01-26T23:29:48.652459+0000","last_clean":"2024-01-26T23:29:48.652459+0000","last_became_active":"2024-01-26T23:29:48.646117+0000","last_became_peered":"2024-01-26T23:29:48.646117+0000","last_unstale":"2024-01-26T23:29:48.652459+0000","last_undegraded":"2024-01-26T23:29:48.652459+0000","last_fullsized":"2024-01-26T23:29:48.652459+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:53:55.279661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.689527+0000","last_change":"2024-01-26T23:29:43.623735+0000","last_active":"2024-01-26T23:29:48.689527+0000","last_peered":"2024-01-26T23:29:48.689527+0000","last_clean":"2024-01-26T23:29:48.689527+0000","last_became_active":"2024-01-26T23:29:43.623048+0000","last_became_peered":"2024-01-26T23:29:43.623048+0000","last_unstale":"2024-01-26T23:29:48.689527+0000","last_undegraded":"2024-01-26T23:29:48.689527+0000","last_fullsized":"2024-01-26T23:29:48.689527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:47:09.096394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632551+0000","last_change":"2024-01-26T23:29:44.602462+0000","last_active":"2024-01-26T23:29:48.632551+0000","last_peered":"2024-01-26T23:29:48.632551+0000","last_clean":"2024-01-26T23:29:48.632551+0000","last_became_active":"2024-01-26T23:29:44.602007+0000","last_became_peered":"2024-01-26T23:29:44.602007+0000","last_unstale":"2024-01-26T23:29:48.632551+0000","last_undegraded":"2024-01-26T23:29:48.632551+0000","last_fullsized":"2024-01-26T23:29:48.632551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:23:26.050869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695910+0000","last_change":"2024-01-26T23:29:48.691917+0000","last_active":"2024-01-26T23:29:48.695910+0000","last_peered":"2024-01-26T23:29:48.695910+0000","last_clean":"2024-01-26T23:29:48.695910+0000","last_became_active":"2024-01-26T23:29:48.691441+0000","last_became_peered":"2024-01-26T23:29:48.691441+0000","last_unstale":"2024-01-26T23:29:48.695910+0000","last_undegraded":"2024-01-26T23:29:48.695910+0000","last_fullsized":"2024-01-26T23:29:48.695910+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:52:44.817581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.625872+0000","last_change":"2024-01-26T23:29:43.619776+0000","last_active":"2024-01-26T23:29:48.625872+0000","last_peered":"2024-01-26T23:29:48.625872+0000","last_clean":"2024-01-26T23:29:48.625872+0000","last_became_active":"2024-01-26T23:29:43.619374+0000","last_became_peered":"2024-01-26T23:29:43.619374+0000","last_unstale":"2024-01-26T23:29:48.625872+0000","last_undegraded":"2024-01-26T23:29:48.625872+0000","last_fullsized":"2024-01-26T23:29:48.625872+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:32.417441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636819+0000","last_change":"2024-01-26T23:29:44.621339+0000","last_active":"2024-01-26T23:29:48.636819+0000","last_peered":"2024-01-26T23:29:48.636819+0000","last_clean":"2024-01-26T23:29:48.636819+0000","last_became_active":"2024-01-26T23:29:44.621012+0000","last_became_peered":"2024-01-26T23:29:44.621012+0000","last_unstale":"2024-01-26T23:29:48.636819+0000","last_undegraded":"2024-01-26T23:29:48.636819+0000","last_fullsized":"2024-01-26T23:29:48.636819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:20.138786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652798+0000","last_change":"2024-01-26T23:29:48.650063+0000","last_active":"2024-01-26T23:29:48.652798+0000","last_peered":"2024-01-26T23:29:48.652798+0000","last_clean":"2024-01-26T23:29:48.652798+0000","last_became_active":"2024-01-26T23:29:48.649732+0000","last_became_peered":"2024-01-26T23:29:48.649732+0000","last_unstale":"2024-01-26T23:29:48.652798+0000","last_undegraded":"2024-01-26T23:29:48.652798+0000","last_fullsized":"2024-01-26T23:29:48.652798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:22:12.593080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626014+0000","last_change":"2024-01-26T23:29:43.618706+0000","last_active":"2024-01-26T23:29:48.626014+0000","last_peered":"2024-01-26T23:29:48.626014+0000","last_clean":"2024-01-26T23:29:48.626014+0000","last_became_active":"2024-01-26T23:29:43.618353+0000","last_became_peered":"2024-01-26T23:29:43.618353+0000","last_unstale":"2024-01-26T23:29:48.626014+0000","last_undegraded":"2024-01-26T23:29:48.626014+0000","last_fullsized":"2024-01-26T23:29:48.626014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:45.384500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636855+0000","last_change":"2024-01-26T23:29:44.633055+0000","last_active":"2024-01-26T23:29:48.636855+0000","last_peered":"2024-01-26T23:29:48.636855+0000","last_clean":"2024-01-26T23:29:48.636855+0000","last_became_active":"2024-01-26T23:29:44.631874+0000","last_became_peered":"2024-01-26T23:29:44.631874+0000","last_unstale":"2024-01-26T23:29:48.636855+0000","last_undegraded":"2024-01-26T23:29:48.636855+0000","last_fullsized":"2024-01-26T23:29:48.636855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:01.636855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.652530+0000","last_change":"2024-01-26T23:29:48.648195+0000","last_active":"2024-01-26T23:29:48.652530+0000","last_peered":"2024-01-26T23:29:48.652530+0000","last_clean":"2024-01-26T23:29:48.652530+0000","last_became_active":"2024-01-26T23:29:48.647780+0000","last_became_peered":"2024-01-26T23:29:48.647780+0000","last_unstale":"2024-01-26T23:29:48.652530+0000","last_undegraded":"2024-01-26T23:29:48.652530+0000","last_fullsized":"2024-01-26T23:29:48.652530+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:52:44.817581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690272+0000","last_change":"2024-01-26T23:29:43.623737+0000","last_active":"2024-01-26T23:29:48.690272+0000","last_peered":"2024-01-26T23:29:48.690272+0000","last_clean":"2024-01-26T23:29:48.690272+0000","last_became_active":"2024-01-26T23:29:43.623439+0000","last_became_peered":"2024-01-26T23:29:43.623439+0000","last_unstale":"2024-01-26T23:29:48.690272+0000","last_undegraded":"2024-01-26T23:29:48.690272+0000","last_fullsized":"2024-01-26T23:29:48.690272+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:45.384500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632872+0000","last_change":"2024-01-26T23:29:44.613570+0000","last_active":"2024-01-26T23:29:48.632872+0000","last_peered":"2024-01-26T23:29:48.632872+0000","last_clean":"2024-01-26T23:29:48.632872+0000","last_became_active":"2024-01-26T23:29:44.613282+0000","last_became_peered":"2024-01-26T23:29:44.613282+0000","last_unstale":"2024-01-26T23:29:48.632872+0000","last_undegraded":"2024-01-26T23:29:48.632872+0000","last_fullsized":"2024-01-26T23:29:48.632872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:12.500308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.696078+0000","last_change":"2024-01-26T23:29:48.691625+0000","last_active":"2024-01-26T23:29:48.696078+0000","last_peered":"2024-01-26T23:29:48.696078+0000","last_clean":"2024-01-26T23:29:48.696078+0000","last_became_active":"2024-01-26T23:29:48.691242+0000","last_became_peered":"2024-01-26T23:29:48.691242+0000","last_unstale":"2024-01-26T23:29:48.696078+0000","last_undegraded":"2024-01-26T23:29:48.696078+0000","last_fullsized":"2024-01-26T23:29:48.696078+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:02.358147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633000+0000","last_change":"2024-01-26T23:29:43.622471+0000","last_active":"2024-01-26T23:29:48.633000+0000","last_peered":"2024-01-26T23:29:48.633000+0000","last_clean":"2024-01-26T23:29:48.633000+0000","last_became_active":"2024-01-26T23:29:43.622106+0000","last_became_peered":"2024-01-26T23:29:43.622106+0000","last_unstale":"2024-01-26T23:29:48.633000+0000","last_undegraded":"2024-01-26T23:29:48.633000+0000","last_fullsized":"2024-01-26T23:29:48.633000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:32.417441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690284+0000","last_change":"2024-01-26T23:29:44.623102+0000","last_active":"2024-01-26T23:29:48.690284+0000","last_peered":"2024-01-26T23:29:48.690284+0000","last_clean":"2024-01-26T23:29:48.690284+0000","last_became_active":"2024-01-26T23:29:44.622834+0000","last_became_peered":"2024-01-26T23:29:44.622834+0000","last_unstale":"2024-01-26T23:29:48.690284+0000","last_undegraded":"2024-01-26T23:29:48.690284+0000","last_fullsized":"2024-01-26T23:29:48.690284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:18:55.543376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.651007+0000","last_change":"2024-01-26T23:29:48.644283+0000","last_active":"2024-01-26T23:29:48.651007+0000","last_peered":"2024-01-26T23:29:48.651007+0000","last_clean":"2024-01-26T23:29:48.651007+0000","last_became_active":"2024-01-26T23:29:48.643862+0000","last_became_peered":"2024-01-26T23:29:48.643862+0000","last_unstale":"2024-01-26T23:29:48.651007+0000","last_undegraded":"2024-01-26T23:29:48.651007+0000","last_fullsized":"2024-01-26T23:29:48.651007+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:29:51.471531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626158+0000","last_change":"2024-01-26T23:29:43.613316+0000","last_active":"2024-01-26T23:29:48.626158+0000","last_peered":"2024-01-26T23:29:48.626158+0000","last_clean":"2024-01-26T23:29:48.626158+0000","last_became_active":"2024-01-26T23:29:43.613054+0000","last_became_peered":"2024-01-26T23:29:43.613054+0000","last_unstale":"2024-01-26T23:29:48.626158+0000","last_undegraded":"2024-01-26T23:29:48.626158+0000","last_fullsized":"2024-01-26T23:29:48.626158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:09:46.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.636441+0000","last_change":"2024-01-26T23:29:44.631866+0000","last_active":"2024-01-26T23:29:48.636441+0000","last_peered":"2024-01-26T23:29:48.636441+0000","last_clean":"2024-01-26T23:29:48.636441+0000","last_became_active":"2024-01-26T23:29:44.631581+0000","last_became_peered":"2024-01-26T23:29:44.631581+0000","last_unstale":"2024-01-26T23:29:48.636441+0000","last_undegraded":"2024-01-26T23:29:48.636441+0000","last_fullsized":"2024-01-26T23:29:48.636441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:38:52.041592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.695113+0000","last_change":"2024-01-26T23:29:48.691846+0000","last_active":"2024-01-26T23:29:48.695113+0000","last_peered":"2024-01-26T23:29:48.695113+0000","last_clean":"2024-01-26T23:29:48.695113+0000","last_became_active":"2024-01-26T23:29:48.691410+0000","last_became_peered":"2024-01-26T23:29:48.691410+0000","last_unstale":"2024-01-26T23:29:48.695113+0000","last_undegraded":"2024-01-26T23:29:48.695113+0000","last_fullsized":"2024-01-26T23:29:48.695113+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:47.611606+0000","last_clean_scrub_stamp":"2024-01-26T23:29:47.611606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:49:28.002810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633357+0000","last_change":"2024-01-26T23:29:43.618244+0000","last_active":"2024-01-26T23:29:48.633357+0000","last_peered":"2024-01-26T23:29:48.633357+0000","last_clean":"2024-01-26T23:29:48.633357+0000","last_became_active":"2024-01-26T23:29:43.617875+0000","last_became_peered":"2024-01-26T23:29:43.617875+0000","last_unstale":"2024-01-26T23:29:48.633357+0000","last_undegraded":"2024-01-26T23:29:48.633357+0000","last_fullsized":"2024-01-26T23:29:48.633357+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:42.589899+0000","last_clean_scrub_stamp":"2024-01-26T23:29:42.589899+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:44:34.698447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.690732+0000","last_change":"2024-01-26T23:29:44.618128+0000","last_active":"2024-01-26T23:29:48.690732+0000","last_peered":"2024-01-26T23:29:48.690732+0000","last_clean":"2024-01-26T23:29:48.690732+0000","last_became_active":"2024-01-26T23:29:44.617849+0000","last_became_peered":"2024-01-26T23:29:44.617849+0000","last_unstale":"2024-01-26T23:29:48.690732+0000","last_undegraded":"2024-01-26T23:29:48.690732+0000","last_fullsized":"2024-01-26T23:29:48.690732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:20.138786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.633526+0000","last_change":"2024-01-26T23:29:44.615453+0000","last_active":"2024-01-26T23:29:48.633526+0000","last_peered":"2024-01-26T23:29:48.633526+0000","last_clean":"2024-01-26T23:29:48.633526+0000","last_became_active":"2024-01-26T23:29:44.615201+0000","last_became_peered":"2024-01-26T23:29:44.615201+0000","last_unstale":"2024-01-26T23:29:48.633526+0000","last_undegraded":"2024-01-26T23:29:48.633526+0000","last_fullsized":"2024-01-26T23:29:48.633526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:38.452904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.631336+0000","last_change":"2024-01-26T23:29:44.619769+0000","last_active":"2024-01-26T23:29:48.631336+0000","last_peered":"2024-01-26T23:29:48.631336+0000","last_clean":"2024-01-26T23:29:48.631336+0000","last_became_active":"2024-01-26T23:29:44.619471+0000","last_became_peered":"2024-01-26T23:29:44.619471+0000","last_unstale":"2024-01-26T23:29:48.631336+0000","last_undegraded":"2024-01-26T23:29:48.631336+0000","last_fullsized":"2024-01-26T23:29:48.631336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:47:10.097930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635849+0000","last_change":"2024-01-26T23:29:44.633598+0000","last_active":"2024-01-26T23:29:48.635849+0000","last_peered":"2024-01-26T23:29:48.635849+0000","last_clean":"2024-01-26T23:29:48.635849+0000","last_became_active":"2024-01-26T23:29:44.633320+0000","last_became_peered":"2024-01-26T23:29:44.633320+0000","last_unstale":"2024-01-26T23:29:48.635849+0000","last_undegraded":"2024-01-26T23:29:48.635849+0000","last_fullsized":"2024-01-26T23:29:48.635849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:03.997854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632174+0000","last_change":"2024-01-26T23:29:44.624890+0000","last_active":"2024-01-26T23:29:48.632174+0000","last_peered":"2024-01-26T23:29:48.632174+0000","last_clean":"2024-01-26T23:29:48.632174+0000","last_became_active":"2024-01-26T23:29:44.624615+0000","last_became_peered":"2024-01-26T23:29:44.624615+0000","last_unstale":"2024-01-26T23:29:48.632174+0000","last_undegraded":"2024-01-26T23:29:48.632174+0000","last_fullsized":"2024-01-26T23:29:48.632174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:12:04.973733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637534+0000","last_change":"2024-01-26T23:29:44.618261+0000","last_active":"2024-01-26T23:29:48.637534+0000","last_peered":"2024-01-26T23:29:48.637534+0000","last_clean":"2024-01-26T23:29:48.637534+0000","last_became_active":"2024-01-26T23:29:44.616559+0000","last_became_peered":"2024-01-26T23:29:44.616559+0000","last_unstale":"2024-01-26T23:29:48.637534+0000","last_undegraded":"2024-01-26T23:29:48.637534+0000","last_fullsized":"2024-01-26T23:29:48.637534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:12:04.973733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.626013+0000","last_change":"2024-01-26T23:29:44.619408+0000","last_active":"2024-01-26T23:29:48.626013+0000","last_peered":"2024-01-26T23:29:48.626013+0000","last_clean":"2024-01-26T23:29:48.626013+0000","last_became_active":"2024-01-26T23:29:44.619093+0000","last_became_peered":"2024-01-26T23:29:44.619093+0000","last_unstale":"2024-01-26T23:29:48.626013+0000","last_undegraded":"2024-01-26T23:29:48.626013+0000","last_fullsized":"2024-01-26T23:29:48.626013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:35:11.068882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.638008+0000","last_change":"2024-01-26T23:29:44.614999+0000","last_active":"2024-01-26T23:29:48.638008+0000","last_peered":"2024-01-26T23:29:48.638008+0000","last_clean":"2024-01-26T23:29:48.638008+0000","last_became_active":"2024-01-26T23:29:44.614579+0000","last_became_peered":"2024-01-26T23:29:44.614579+0000","last_unstale":"2024-01-26T23:29:48.638008+0000","last_undegraded":"2024-01-26T23:29:48.638008+0000","last_fullsized":"2024-01-26T23:29:48.638008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:33:36.090025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.639791+0000","last_change":"2024-01-26T23:29:44.612377+0000","last_active":"2024-01-26T23:29:48.639791+0000","last_peered":"2024-01-26T23:29:48.639791+0000","last_clean":"2024-01-26T23:29:48.639791+0000","last_became_active":"2024-01-26T23:29:44.612073+0000","last_became_peered":"2024-01-26T23:29:44.612073+0000","last_unstale":"2024-01-26T23:29:48.639791+0000","last_undegraded":"2024-01-26T23:29:48.639791+0000","last_fullsized":"2024-01-26T23:29:48.639791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:59:47.867303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632846+0000","last_change":"2024-01-26T23:29:44.633571+0000","last_active":"2024-01-26T23:29:48.632846+0000","last_peered":"2024-01-26T23:29:48.632846+0000","last_clean":"2024-01-26T23:29:48.632846+0000","last_became_active":"2024-01-26T23:29:44.633312+0000","last_became_peered":"2024-01-26T23:29:44.633312+0000","last_unstale":"2024-01-26T23:29:48.632846+0000","last_undegraded":"2024-01-26T23:29:48.632846+0000","last_fullsized":"2024-01-26T23:29:48.632846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:22.113035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637064+0000","last_change":"2024-01-26T23:29:44.617840+0000","last_active":"2024-01-26T23:29:48.637064+0000","last_peered":"2024-01-26T23:29:48.637064+0000","last_clean":"2024-01-26T23:29:48.637064+0000","last_became_active":"2024-01-26T23:29:44.611989+0000","last_became_peered":"2024-01-26T23:29:44.611989+0000","last_unstale":"2024-01-26T23:29:48.637064+0000","last_undegraded":"2024-01-26T23:29:48.637064+0000","last_fullsized":"2024-01-26T23:29:48.637064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:14.555745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.635947+0000","last_change":"2024-01-26T23:29:44.617326+0000","last_active":"2024-01-26T23:29:48.635947+0000","last_peered":"2024-01-26T23:29:48.635947+0000","last_clean":"2024-01-26T23:29:48.635947+0000","last_became_active":"2024-01-26T23:29:44.617063+0000","last_became_peered":"2024-01-26T23:29:44.617063+0000","last_unstale":"2024-01-26T23:29:48.635947+0000","last_undegraded":"2024-01-26T23:29:48.635947+0000","last_fullsized":"2024-01-26T23:29:48.635947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:01.636855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.632268+0000","last_change":"2024-01-26T23:29:44.613645+0000","last_active":"2024-01-26T23:29:48.632268+0000","last_peered":"2024-01-26T23:29:48.632268+0000","last_clean":"2024-01-26T23:29:48.632268+0000","last_became_active":"2024-01-26T23:29:44.613290+0000","last_became_peered":"2024-01-26T23:29:44.613290+0000","last_unstale":"2024-01-26T23:29:48.632268+0000","last_undegraded":"2024-01-26T23:29:48.632268+0000","last_fullsized":"2024-01-26T23:29:48.632268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:12.500308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-01-26T23:29:48.637188+0000","last_change":"2024-01-26T23:29:44.609934+0000","last_active":"2024-01-26T23:29:48.637188+0000","last_peered":"2024-01-26T23:29:48.637188+0000","last_clean":"2024-01-26T23:29:48.637188+0000","last_became_active":"2024-01-26T23:29:44.609682+0000","last_became_peered":"2024-01-26T23:29:44.609682+0000","last_unstale":"2024-01-26T23:29:48.637188+0000","last_undegraded":"2024-01-26T23:29:48.637188+0000","last_fullsized":"2024-01-26T23:29:48.637188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:43.591435+0000","last_clean_scrub_stamp":"2024-01-26T23:29:43.591435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:12:04.973733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4483,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298120,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073720,"statfs":{"total":96636764160,"available":96331489280,"internally_reserved":0,"allocated":851968,"data_stored":690410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":103201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297544,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074296,"statfs":{"total":96636764160,"available":96332079104,"internally_reserved":0,"allocated":262144,"data_stored":99648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298128,"kb_used_data":840,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073712,"statfs":{"total":96636764160,"available":96331481088,"internally_reserved":0,"allocated":860160,"data_stored":690466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.214}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297536,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074304,"statfs":{"total":96636764160,"available":96332087296,"internally_reserved":0,"allocated":253952,"data_stored":99488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":103201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":99614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297540,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074300,"statfs":{"total":96636764160,"available":96332083200,"internally_reserved":0,"allocated":258048,"data_stored":99578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:30:10.166 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-26T23:30:10.166 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-26T23:30:10.166 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-26T23:30:10.166 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-26T23:30:10.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.334+0000 7f40ce3c5700 1 Processor -- start 2024-01-26T23:30:10.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.334+0000 7f40ce3c5700 1 -- start start 2024-01-26T23:30:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.335+0000 7f40ce3c5700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f40c80acf30 0x7f40c80c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.335+0000 7f40ce3c5700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f40c80ad780 0x7f40c80ab1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.335+0000 7f40ce3c5700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c80c4230 con 0x7f40c80ad300 2024-01-26T23:30:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.335+0000 7f40ce3c5700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c8157be0 con 0x7f40c80ad780 2024-01-26T23:30:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.335+0000 7f40ce3c5700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c80cbf10 con 0x7f40c80acf30 2024-01-26T23:30:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.335+0000 7f40cc962700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f40c80ad780 0x7f40c80ab1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.336+0000 7f40cc962700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f40c80ad780 0x7f40c80ab1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40274/0 (socket says 172.21.15.46:40274) 2024-01-26T23:30:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.336+0000 7f40cc962700 1 -- 172.21.15.46:0/1586577794 learned_addr learned my addr 172.21.15.46:0/1586577794 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.336+0000 7f40c7fff700 1 --2- 172.21.15.46:0/1586577794 >> v2:172.21.15.106:3300/0 conn(0x7f40c80acf30 0x7f40c80c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.336+0000 7f40cc962700 1 -- 172.21.15.46:0/1586577794 >> v2:172.21.15.106:3300/0 conn(0x7f40c80acf30 msgr2=0x7f40c80c3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.336+0000 7f40cc962700 1 --2- 172.21.15.46:0/1586577794 >> v2:172.21.15.106:3300/0 conn(0x7f40c80acf30 0x7f40c80c3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.336+0000 7f40cc962700 1 -- 172.21.15.46:0/1586577794 >> v1:172.21.15.106:6789/0 conn(0x7f40c80ad300 legacy=0x7f40c80ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.336+0000 7f40cc962700 1 -- 172.21.15.46:0/1586577794 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c80c4460 con 0x7f40c80ad780 2024-01-26T23:30:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.336+0000 7f40cc962700 1 --2- 172.21.15.46:0/1586577794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c80ad780 0x7f40c80ab1b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f40c806fa70 tx=0x7f40c0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c1137f16c9d2bf0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.336+0000 7f40c6ffd700 1 -- 172.21.15.46:0/1586577794 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c000e070 con 0x7f40c80ad780 2024-01-26T23:30:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.336+0000 7f40c6ffd700 1 -- 172.21.15.46:0/1586577794 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c000ba20 con 0x7f40c80ad780 2024-01-26T23:30:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.336+0000 7f40c6ffd700 1 -- 172.21.15.46:0/1586577794 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c00092e0 con 0x7f40c80ad780 2024-01-26T23:30:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.337+0000 7f40ce3c5700 1 -- 172.21.15.46:0/1586577794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c80ad780 msgr2=0x7f40c80ab1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.337+0000 7f40ce3c5700 1 --2- 172.21.15.46:0/1586577794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c80ad780 0x7f40c80ab1b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f40c806fa70 tx=0x7f40c0009840 comp rx=0 tx=0).stop 2024-01-26T23:30:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.337+0000 7f40ce3c5700 1 -- 172.21.15.46:0/1586577794 shutdown_connections 2024-01-26T23:30:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.337+0000 7f40ce3c5700 1 --2- 172.21.15.46:0/1586577794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c80ad780 0x7f40c80ab1b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.337+0000 7f40ce3c5700 1 --2- 172.21.15.46:0/1586577794 >> v2:172.21.15.106:3300/0 conn(0x7f40c80acf30 0x7f40c80c3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.337+0000 7f40ce3c5700 1 -- 172.21.15.46:0/1586577794 >> 172.21.15.46:0/1586577794 conn(0x7f40c8097820 msgr2=0x7f40c8068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.337+0000 7f40ce3c5700 1 -- 172.21.15.46:0/1586577794 shutdown_connections 2024-01-26T23:30:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.337+0000 7f40ce3c5700 1 -- 172.21.15.46:0/1586577794 wait complete. 2024-01-26T23:30:10.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.338+0000 7f40ce3c5700 1 Processor -- start 2024-01-26T23:30:10.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.338+0000 7f40ce3c5700 1 -- start start 2024-01-26T23:30:10.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.339+0000 7f40ce3c5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c80acf30 0x7f40c80c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:10.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.339+0000 7f40ce3c5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40c80ad300 0x7f40c8151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:10.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.339+0000 7f40ce3c5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40c80ad780 0x7f40c80caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:10.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.339+0000 7f40ce3c5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80c6d60 con 0x7f40c80acf30 2024-01-26T23:30:10.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.339+0000 7f40ce3c5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80caf80 con 0x7f40c80ad780 2024-01-26T23:30:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.339+0000 7f40ce3c5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80ca2c0 con 0x7f40c80ad300 2024-01-26T23:30:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.339+0000 7f40c7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c80acf30 0x7f40c80c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.339+0000 7f40c7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c80acf30 0x7f40c80c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40276/0 (socket says 172.21.15.46:40276) 2024-01-26T23:30:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.339+0000 7f40c7fff700 1 -- 172.21.15.46:0/3579429684 learned_addr learned my addr 172.21.15.46:0/3579429684 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.339+0000 7f40cc962700 1 --2- 172.21.15.46:0/3579429684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40c80ad780 0x7f40c80caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.339+0000 7f40c77fe700 1 --2- 172.21.15.46:0/3579429684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40c80ad300 0x7f40c8151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.340+0000 7f40c7fff700 1 -- 172.21.15.46:0/3579429684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40c80ad300 msgr2=0x7f40c8151220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.340+0000 7f40c7fff700 1 --2- 172.21.15.46:0/3579429684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40c80ad300 0x7f40c8151220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.340+0000 7f40c7fff700 1 -- 172.21.15.46:0/3579429684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40c80ad780 msgr2=0x7f40c80caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.340+0000 7f40c7fff700 1 --2- 172.21.15.46:0/3579429684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40c80ad780 0x7f40c80caa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.340+0000 7f40c7fff700 1 -- 172.21.15.46:0/3579429684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c0009ae0 con 0x7f40c80acf30 2024-01-26T23:30:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.340+0000 7f40cc962700 1 --2- 172.21.15.46:0/3579429684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40c80ad780 0x7f40c80caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:30:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.340+0000 7f40c7fff700 1 --2- 172.21.15.46:0/3579429684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c80acf30 0x7f40c80c93b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f40b800ec80 tx=0x7f40b80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.340+0000 7f40bffff700 1 -- 172.21.15.46:0/3579429684 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b800c9d0 con 0x7f40c80acf30 2024-01-26T23:30:10.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.340+0000 7f40bffff700 1 -- 172.21.15.46:0/3579429684 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b800cb30 con 0x7f40c80acf30 2024-01-26T23:30:10.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.340+0000 7f40ce3c5700 1 -- 172.21.15.46:0/3579429684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c80c9a20 con 0x7f40c80acf30 2024-01-26T23:30:10.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.340+0000 7f40bffff700 1 -- 172.21.15.46:0/3579429684 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b8013470 con 0x7f40c80acf30 2024-01-26T23:30:10.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.340+0000 7f40ce3c5700 1 -- 172.21.15.46:0/3579429684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c80c9fe0 con 0x7f40c80acf30 2024-01-26T23:30:10.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.341+0000 7f40ce3c5700 1 -- 172.21.15.46:0/3579429684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c80c6b30 con 0x7f40c80acf30 2024-01-26T23:30:10.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.341+0000 7f40bffff700 1 -- 172.21.15.46:0/3579429684 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f40b8013820 con 0x7f40c80acf30 2024-01-26T23:30:10.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.342+0000 7f40bffff700 1 --2- 172.21.15.46:0/3579429684 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f40a8066f30 0x7f40a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:10.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.342+0000 7f40bffff700 1 -- 172.21.15.46:0/3579429684 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f40b8084bb0 con 0x7f40c80acf30 2024-01-26T23:30:10.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.342+0000 7f40c77fe700 1 --2- 172.21.15.46:0/3579429684 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f40a8066f30 0x7f40a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:10.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.343+0000 7f40c77fe700 1 --2- 172.21.15.46:0/3579429684 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f40a8066f30 0x7f40a80693e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f40c80abfa0 tx=0x7f40b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:10.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.346+0000 7f40bffff700 1 -- 172.21.15.46:0/3579429684 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f40b8051b30 con 0x7f40c80acf30 2024-01-26T23:30:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.593+0000 7f40ce3c5700 1 -- 172.21.15.46:0/3579429684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f40c804c710 con 0x7f40c80acf30 2024-01-26T23:30:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.593+0000 7f40bffff700 1 -- 172.21.15.46:0/3579429684 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f40b8055150 con 0x7f40c80acf30 2024-01-26T23:30:10.594 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:10.594 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-26T23:30:10.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.596+0000 7f40ce3c5700 1 -- 172.21.15.46:0/3579429684 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f40a8066f30 msgr2=0x7f40a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:10.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.596+0000 7f40ce3c5700 1 --2- 172.21.15.46:0/3579429684 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f40a8066f30 0x7f40a80693e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f40c80abfa0 tx=0x7f40b0009040 comp rx=0 tx=0).stop 2024-01-26T23:30:10.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.596+0000 7f40ce3c5700 1 -- 172.21.15.46:0/3579429684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c80acf30 msgr2=0x7f40c80c93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:10.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.596+0000 7f40ce3c5700 1 --2- 172.21.15.46:0/3579429684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c80acf30 0x7f40c80c93b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f40b800ec80 tx=0x7f40b80091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.596+0000 7f40ce3c5700 1 -- 172.21.15.46:0/3579429684 shutdown_connections 2024-01-26T23:30:10.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.596+0000 7f40ce3c5700 1 --2- 172.21.15.46:0/3579429684 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f40a8066f30 0x7f40a80693e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.596+0000 7f40ce3c5700 1 --2- 172.21.15.46:0/3579429684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c80acf30 0x7f40c80c93b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.596+0000 7f40ce3c5700 1 --2- 172.21.15.46:0/3579429684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40c80ad780 0x7f40c80caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.596+0000 7f40ce3c5700 1 --2- 172.21.15.46:0/3579429684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40c80ad300 0x7f40c8151220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.596+0000 7f40ce3c5700 1 -- 172.21.15.46:0/3579429684 >> 172.21.15.46:0/3579429684 conn(0x7f40c8097820 msgr2=0x7f40c806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:10.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.596+0000 7f40ce3c5700 1 -- 172.21.15.46:0/3579429684 shutdown_connections 2024-01-26T23:30:10.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.596+0000 7f40ce3c5700 1 -- 172.21.15.46:0/3579429684 wait complete. 2024-01-26T23:30:10.610 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-26T23:30:10.610 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:10.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.743+0000 7f52758cc700 1 Processor -- start 2024-01-26T23:30:10.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.743+0000 7f52758cc700 1 -- start start 2024-01-26T23:30:10.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.744+0000 7f52758cc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f52700cf460 0x7f52700ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:10.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.744+0000 7f52758cc700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527015a0b0 con 0x7f52700b2b40 2024-01-26T23:30:10.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.744+0000 7f52758cc700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52700ae4d0 con 0x7f52700c6520 2024-01-26T23:30:10.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.744+0000 7f52758cc700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52700af070 con 0x7f52700cf460 2024-01-26T23:30:10.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.744+0000 7f526effd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f52700b2b40 0x7f52700ae960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50402/0 (socket says 172.21.15.46:50402) 2024-01-26T23:30:10.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.744+0000 7f526effd700 1 -- 172.21.15.46:0/3948337077 learned_addr learned my addr 172.21.15.46:0/3948337077 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:10.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.744+0000 7f526f7fe700 1 --2- 172.21.15.46:0/3948337077 >> v2:172.21.15.106:3300/0 conn(0x7f52700cf460 0x7f52700ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:10.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.744+0000 7f526dffb700 1 -- 172.21.15.46:0/3948337077 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162808404 0 0) 0x7f527015a0b0 con 0x7f52700b2b40 2024-01-26T23:30:10.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.744+0000 7f526dffb700 1 -- 172.21.15.46:0/3948337077 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f525c003570 con 0x7f52700b2b40 2024-01-26T23:30:10.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.745+0000 7f526dffb700 1 -- 172.21.15.46:0/3948337077 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498556851 0 0) 0x7f52700ae4d0 con 0x7f52700c6520 2024-01-26T23:30:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.745+0000 7f526dffb700 1 -- 172.21.15.46:0/3948337077 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f527015a0b0 con 0x7f52700c6520 2024-01-26T23:30:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.745+0000 7f526dffb700 1 -- 172.21.15.46:0/3948337077 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1784634603 0 0) 0x7f525c003570 con 0x7f52700b2b40 2024-01-26T23:30:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.745+0000 7f526f7fe700 1 -- 172.21.15.46:0/3948337077 >> v1:172.21.15.106:6789/0 conn(0x7f52700c6520 legacy=0x7f52700ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.745+0000 7f526f7fe700 1 -- 172.21.15.46:0/3948337077 >> v1:172.21.15.46:6789/0 conn(0x7f52700b2b40 legacy=0x7f52700ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.745+0000 7f526f7fe700 1 -- 172.21.15.46:0/3948337077 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52700ad8c0 con 0x7f52700cf460 2024-01-26T23:30:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.745+0000 7f526f7fe700 1 --2- 172.21.15.46:0/3948337077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52700cf460 0x7f52700ad1c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5268009e60 tx=0x7f5268009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ad76d6e7f464fa server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.745+0000 7f526dffb700 1 -- 172.21.15.46:0/3948337077 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526800e070 con 0x7f52700cf460 2024-01-26T23:30:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.745+0000 7f526dffb700 1 -- 172.21.15.46:0/3948337077 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526800b7a0 con 0x7f52700cf460 2024-01-26T23:30:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.745+0000 7f526dffb700 1 -- 172.21.15.46:0/3948337077 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52600015f0 con 0x7f52700cf460 2024-01-26T23:30:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.746+0000 7f52758cc700 1 -- 172.21.15.46:0/3948337077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52700cf460 msgr2=0x7f52700ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.746+0000 7f52758cc700 1 --2- 172.21.15.46:0/3948337077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52700cf460 0x7f52700ad1c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5268009e60 tx=0x7f5268009780 comp rx=0 tx=0).stop 2024-01-26T23:30:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.746+0000 7f52758cc700 1 -- 172.21.15.46:0/3948337077 shutdown_connections 2024-01-26T23:30:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.746+0000 7f52758cc700 1 --2- 172.21.15.46:0/3948337077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52700cf460 0x7f52700ad1c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.746+0000 7f52758cc700 1 -- 172.21.15.46:0/3948337077 >> 172.21.15.46:0/3948337077 conn(0x7f5270099a30 msgr2=0x7f527006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:10.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.746+0000 7f52758cc700 1 -- 172.21.15.46:0/3948337077 shutdown_connections 2024-01-26T23:30:10.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.746+0000 7f52758cc700 1 -- 172.21.15.46:0/3948337077 wait complete. 2024-01-26T23:30:10.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.747+0000 7f52758cc700 1 Processor -- start 2024-01-26T23:30:10.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.747+0000 7f52758cc700 1 -- start start 2024-01-26T23:30:10.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.748+0000 7f52758cc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52700b2b40 0x7f52700afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:10.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.748+0000 7f52758cc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52700c6520 0x7f52700b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:10.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.748+0000 7f52758cc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52700cf460 0x7f52701516a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:10.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.748+0000 7f52758cc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52700b41c0 con 0x7f52700c6520 2024-01-26T23:30:10.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.748+0000 7f52758cc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52700b4fc0 con 0x7f52700cf460 2024-01-26T23:30:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.748+0000 7f52758cc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5270151be0 con 0x7f52700b2b40 2024-01-26T23:30:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.748+0000 7f526e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52700c6520 0x7f52700b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.748+0000 7f526e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52700c6520 0x7f52700b3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40284/0 (socket says 172.21.15.46:40284) 2024-01-26T23:30:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.748+0000 7f526e7fc700 1 -- 172.21.15.46:0/1910234329 learned_addr learned my addr 172.21.15.46:0/1910234329 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.748+0000 7f526effd700 1 --2- 172.21.15.46:0/1910234329 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52700b2b40 0x7f52700afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.748+0000 7f526f7fe700 1 --2- 172.21.15.46:0/1910234329 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52700cf460 0x7f52701516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.748+0000 7f526e7fc700 1 -- 172.21.15.46:0/1910234329 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52700b2b40 msgr2=0x7f52700afd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.748+0000 7f526e7fc700 1 --2- 172.21.15.46:0/1910234329 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52700b2b40 0x7f52700afd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.748+0000 7f526e7fc700 1 -- 172.21.15.46:0/1910234329 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52700cf460 msgr2=0x7f52701516a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.748+0000 7f526e7fc700 1 --2- 172.21.15.46:0/1910234329 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52700cf460 0x7f52701516a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.748+0000 7f526e7fc700 1 -- 172.21.15.46:0/1910234329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5268009a20 con 0x7f52700c6520 2024-01-26T23:30:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.748+0000 7f526f7fe700 1 --2- 172.21.15.46:0/1910234329 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52700cf460 0x7f52701516a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.749+0000 7f526e7fc700 1 --2- 172.21.15.46:0/1910234329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52700c6520 0x7f52700b3c80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f525800c7c0 tx=0x7f5258007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.749+0000 7f5266ffd700 1 -- 172.21.15.46:0/1910234329 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258010510 con 0x7f52700c6520 2024-01-26T23:30:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.749+0000 7f5266ffd700 1 -- 172.21.15.46:0/1910234329 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5258010670 con 0x7f52700c6520 2024-01-26T23:30:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.749+0000 7f52758cc700 1 -- 172.21.15.46:0/1910234329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52700b43b0 con 0x7f52700c6520 2024-01-26T23:30:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.749+0000 7f5266ffd700 1 -- 172.21.15.46:0/1910234329 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258010a30 con 0x7f52700c6520 2024-01-26T23:30:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.749+0000 7f52758cc700 1 -- 172.21.15.46:0/1910234329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52700b4970 con 0x7f52700c6520 2024-01-26T23:30:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.750+0000 7f52758cc700 1 -- 172.21.15.46:0/1910234329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52700cdb30 con 0x7f52700c6520 2024-01-26T23:30:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.750+0000 7f5266ffd700 1 -- 172.21.15.46:0/1910234329 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f525801b3b0 con 0x7f52700c6520 2024-01-26T23:30:10.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.751+0000 7f5266ffd700 1 --2- 172.21.15.46:0/1910234329 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5250066f30 0x7f52500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:10.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.751+0000 7f5266ffd700 1 -- 172.21.15.46:0/1910234329 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f52580840c0 con 0x7f52700c6520 2024-01-26T23:30:10.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.751+0000 7f526effd700 1 --2- 172.21.15.46:0/1910234329 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5250066f30 0x7f52500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:10.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.751+0000 7f526effd700 1 --2- 172.21.15.46:0/1910234329 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5250066f30 0x7f52500693e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5260006d80 tx=0x7f52600012c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:10.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.755+0000 7f5266ffd700 1 -- 172.21.15.46:0/1910234329 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5258051040 con 0x7f52700c6520 2024-01-26T23:30:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.958+0000 7f52758cc700 1 -- 172.21.15.46:0/1910234329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f527004c710 con 0x7f52700c6520 2024-01-26T23:30:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.959+0000 7f5266ffd700 1 -- 172.21.15.46:0/1910234329 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7f5258054660 con 0x7f52700c6520 2024-01-26T23:30:10.960 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:10.960 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:49.075104+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:10.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.961+0000 7f52758cc700 1 -- 172.21.15.46:0/1910234329 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5250066f30 msgr2=0x7f52500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:10.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.961+0000 7f52758cc700 1 --2- 172.21.15.46:0/1910234329 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5250066f30 0x7f52500693e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5260006d80 tx=0x7f52600012c0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.961+0000 7f52758cc700 1 -- 172.21.15.46:0/1910234329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52700c6520 msgr2=0x7f52700b3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:10.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.961+0000 7f52758cc700 1 --2- 172.21.15.46:0/1910234329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52700c6520 0x7f52700b3c80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f525800c7c0 tx=0x7f5258007dc0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.961+0000 7f52758cc700 1 -- 172.21.15.46:0/1910234329 shutdown_connections 2024-01-26T23:30:10.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.961+0000 7f52758cc700 1 --2- 172.21.15.46:0/1910234329 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5250066f30 0x7f52500693e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.961+0000 7f52758cc700 1 --2- 172.21.15.46:0/1910234329 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52700cf460 0x7f52701516a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.961+0000 7f52758cc700 1 --2- 172.21.15.46:0/1910234329 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52700b2b40 0x7f52700afd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.961+0000 7f52758cc700 1 --2- 172.21.15.46:0/1910234329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52700c6520 0x7f52700b3c80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:10.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.961+0000 7f52758cc700 1 -- 172.21.15.46:0/1910234329 >> 172.21.15.46:0/1910234329 conn(0x7f5270099a30 msgr2=0x7f52700ae960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:10.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.961+0000 7f52758cc700 1 -- 172.21.15.46:0/1910234329 shutdown_connections 2024-01-26T23:30:10.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:10.961+0000 7f52758cc700 1 -- 172.21.15.46:0/1910234329 wait complete. 2024-01-26T23:30:10.976 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-26T23:29:42.539759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-26T23:29:43.002669+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-01-26T23:29:46.646864+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-26T23:30:10.976 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:30:11.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.110+0000 7f1b918de700 1 Processor -- start 2024-01-26T23:30:11.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.111+0000 7f1b918de700 1 -- start start 2024-01-26T23:30:11.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.111+0000 7f1b918de700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b8c1599c0 0x7f1b8c0ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:11.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.111+0000 7f1b918de700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c0c8f70 con 0x7f1b8c0caa60 2024-01-26T23:30:11.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.111+0000 7f1b918de700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c15a9c0 con 0x7f1b8c068c50 2024-01-26T23:30:11.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.111+0000 7f1b918de700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c15aca0 con 0x7f1b8c1599c0 2024-01-26T23:30:11.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.112+0000 7f1b8b7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1b8c0caa60 0x7f1b8c159d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50406/0 (socket says 172.21.15.46:50406) 2024-01-26T23:30:11.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.112+0000 7f1b8b7fe700 1 -- 172.21.15.46:0/387483229 learned_addr learned my addr 172.21.15.46:0/387483229 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:11.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.112+0000 7f1b8affd700 1 --2- 172.21.15.46:0/387483229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b8c1599c0 0x7f1b8c0ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:11.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.112+0000 7f1b89ffb700 1 -- 172.21.15.46:0/387483229 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424198691 0 0) 0x7f1b8c0c8f70 con 0x7f1b8c0caa60 2024-01-26T23:30:11.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.112+0000 7f1b89ffb700 1 -- 172.21.15.46:0/387483229 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b74003570 con 0x7f1b8c0caa60 2024-01-26T23:30:11.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.112+0000 7f1b89ffb700 1 -- 172.21.15.46:0/387483229 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512854631 0 0) 0x7f1b8c15a9c0 con 0x7f1b8c068c50 2024-01-26T23:30:11.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.112+0000 7f1b8affd700 1 -- 172.21.15.46:0/387483229 >> v1:172.21.15.106:6789/0 conn(0x7f1b8c068c50 legacy=0x7f1b8c0ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.112+0000 7f1b8affd700 1 -- 172.21.15.46:0/387483229 >> v1:172.21.15.46:6789/0 conn(0x7f1b8c0caa60 legacy=0x7f1b8c159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.112+0000 7f1b8affd700 1 -- 172.21.15.46:0/387483229 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c15af80 con 0x7f1b8c1599c0 2024-01-26T23:30:11.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.112+0000 7f1b8affd700 1 --2- 172.21.15.46:0/387483229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b8c1599c0 0x7f1b8c0ccc50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c009910 tx=0x7f1b7c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95a6650867cef301 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:11.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.112+0000 7f1b89ffb700 1 -- 172.21.15.46:0/387483229 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c00b680 con 0x7f1b8c1599c0 2024-01-26T23:30:11.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.113+0000 7f1b89ffb700 1 -- 172.21.15.46:0/387483229 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7c00b7e0 con 0x7f1b8c1599c0 2024-01-26T23:30:11.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.113+0000 7f1b89ffb700 1 -- 172.21.15.46:0/387483229 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c00bba0 con 0x7f1b8c1599c0 2024-01-26T23:30:11.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.113+0000 7f1b918de700 1 -- 172.21.15.46:0/387483229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b8c1599c0 msgr2=0x7f1b8c0ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.113+0000 7f1b918de700 1 --2- 172.21.15.46:0/387483229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b8c1599c0 0x7f1b8c0ccc50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c009910 tx=0x7f1b7c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.113+0000 7f1b918de700 1 -- 172.21.15.46:0/387483229 shutdown_connections 2024-01-26T23:30:11.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.113+0000 7f1b918de700 1 --2- 172.21.15.46:0/387483229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b8c1599c0 0x7f1b8c0ccc50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.113+0000 7f1b918de700 1 -- 172.21.15.46:0/387483229 >> 172.21.15.46:0/387483229 conn(0x7f1b8c0ce570 msgr2=0x7f1b8c0ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:11.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.113+0000 7f1b918de700 1 -- 172.21.15.46:0/387483229 shutdown_connections 2024-01-26T23:30:11.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.113+0000 7f1b918de700 1 -- 172.21.15.46:0/387483229 wait complete. 2024-01-26T23:30:11.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.114+0000 7f1b918de700 1 Processor -- start 2024-01-26T23:30:11.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.115+0000 7f1b918de700 1 -- start start 2024-01-26T23:30:11.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.115+0000 7f1b918de700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b8c068c50 0x7f1b8c153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:11.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.115+0000 7f1b918de700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b8c0caa60 0x7f1b8c06bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:11.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.115+0000 7f1b918de700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b8c1599c0 0x7f1b8c06c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:11.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.115+0000 7f1b918de700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c143d10 con 0x7f1b8c1599c0 2024-01-26T23:30:11.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.115+0000 7f1b918de700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c0b1660 con 0x7f1b8c068c50 2024-01-26T23:30:11.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.115+0000 7f1b918de700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c0b1940 con 0x7f1b8c0caa60 2024-01-26T23:30:11.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.115+0000 7f1b8b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b8c1599c0 0x7f1b8c06c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:11.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.116+0000 7f1b8affd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b8c068c50 0x7f1b8c153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:11.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.116+0000 7f1b8b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b8c1599c0 0x7f1b8c06c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40296/0 (socket says 172.21.15.46:40296) 2024-01-26T23:30:11.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.116+0000 7f1b8a7fc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b8c0caa60 0x7f1b8c06bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:11.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.116+0000 7f1b8b7fe700 1 -- 172.21.15.46:0/1775756124 learned_addr learned my addr 172.21.15.46:0/1775756124 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:11.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.116+0000 7f1b8b7fe700 1 -- 172.21.15.46:0/1775756124 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b8c0caa60 msgr2=0x7f1b8c06bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.116+0000 7f1b8b7fe700 1 --2- 172.21.15.46:0/1775756124 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b8c0caa60 0x7f1b8c06bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.116+0000 7f1b8b7fe700 1 -- 172.21.15.46:0/1775756124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b8c068c50 msgr2=0x7f1b8c153e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.116+0000 7f1b8b7fe700 1 --2- 172.21.15.46:0/1775756124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b8c068c50 0x7f1b8c153e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.116+0000 7f1b8b7fe700 1 -- 172.21.15.46:0/1775756124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c009420 con 0x7f1b8c1599c0 2024-01-26T23:30:11.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.116+0000 7f1b8a7fc700 1 --2- 172.21.15.46:0/1775756124 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b8c0caa60 0x7f1b8c06bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:11.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.116+0000 7f1b8affd700 1 --2- 172.21.15.46:0/1775756124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b8c068c50 0x7f1b8c153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:11.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.116+0000 7f1b8b7fe700 1 --2- 172.21.15.46:0/1775756124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b8c1599c0 0x7f1b8c06c330 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400cc40 tx=0x7f1b84001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:11.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.116+0000 7f1b72ffd700 1 -- 172.21.15.46:0/1775756124 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8400a8b0 con 0x7f1b8c1599c0 2024-01-26T23:30:11.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.116+0000 7f1b72ffd700 1 -- 172.21.15.46:0/1775756124 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8400aa10 con 0x7f1b8c1599c0 2024-01-26T23:30:11.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.116+0000 7f1b918de700 1 -- 172.21.15.46:0/1775756124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8c0b1c20 con 0x7f1b8c1599c0 2024-01-26T23:30:11.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.116+0000 7f1b72ffd700 1 -- 172.21.15.46:0/1775756124 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8400ad90 con 0x7f1b8c1599c0 2024-01-26T23:30:11.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.116+0000 7f1b918de700 1 -- 172.21.15.46:0/1775756124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8c0b21e0 con 0x7f1b8c1599c0 2024-01-26T23:30:11.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.117+0000 7f1b918de700 1 -- 172.21.15.46:0/1775756124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8c070270 con 0x7f1b8c1599c0 2024-01-26T23:30:11.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.118+0000 7f1b72ffd700 1 -- 172.21.15.46:0/1775756124 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1b84007300 con 0x7f1b8c1599c0 2024-01-26T23:30:11.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.118+0000 7f1b72ffd700 1 --2- 172.21.15.46:0/1775756124 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1b78066f30 0x7f1b780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:11.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.118+0000 7f1b72ffd700 1 -- 172.21.15.46:0/1775756124 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f1b84084cd0 con 0x7f1b8c1599c0 2024-01-26T23:30:11.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.118+0000 7f1b8affd700 1 --2- 172.21.15.46:0/1775756124 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1b78066f30 0x7f1b780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:11.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.119+0000 7f1b8affd700 1 --2- 172.21.15.46:0/1775756124 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1b78066f30 0x7f1b780693e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0093f0 tx=0x7f1b7c017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:11.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.123+0000 7f1b72ffd700 1 -- 172.21.15.46:0/1775756124 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1b84051b20 con 0x7f1b8c1599c0 2024-01-26T23:30:11.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.326+0000 7f1b918de700 1 -- 172.21.15.46:0/1775756124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1b8c04c710 con 0x7f1b8c1599c0 2024-01-26T23:30:11.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.327+0000 7f1b72ffd700 1 -- 172.21.15.46:0/1775756124 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1b84055140 con 0x7f1b8c1599c0 2024-01-26T23:30:11.328 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:30:11.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.329+0000 7f1b918de700 1 -- 172.21.15.46:0/1775756124 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1b78066f30 msgr2=0x7f1b780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.329+0000 7f1b918de700 1 --2- 172.21.15.46:0/1775756124 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1b78066f30 0x7f1b780693e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0093f0 tx=0x7f1b7c017000 comp rx=0 tx=0).stop 2024-01-26T23:30:11.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.329+0000 7f1b918de700 1 -- 172.21.15.46:0/1775756124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b8c1599c0 msgr2=0x7f1b8c06c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.329+0000 7f1b918de700 1 --2- 172.21.15.46:0/1775756124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b8c1599c0 0x7f1b8c06c330 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400cc40 tx=0x7f1b84001aa0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.329+0000 7f1b918de700 1 -- 172.21.15.46:0/1775756124 shutdown_connections 2024-01-26T23:30:11.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.329+0000 7f1b918de700 1 --2- 172.21.15.46:0/1775756124 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1b78066f30 0x7f1b780693e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.329+0000 7f1b918de700 1 --2- 172.21.15.46:0/1775756124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b8c1599c0 0x7f1b8c06c330 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.329+0000 7f1b918de700 1 --2- 172.21.15.46:0/1775756124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b8c068c50 0x7f1b8c153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.329+0000 7f1b918de700 1 --2- 172.21.15.46:0/1775756124 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b8c0caa60 0x7f1b8c06bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.329+0000 7f1b918de700 1 -- 172.21.15.46:0/1775756124 >> 172.21.15.46:0/1775756124 conn(0x7f1b8c0ce570 msgr2=0x7f1b8c069140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:11.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.330+0000 7f1b918de700 1 -- 172.21.15.46:0/1775756124 shutdown_connections 2024-01-26T23:30:11.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.330+0000 7f1b918de700 1 -- 172.21.15.46:0/1775756124 wait complete. 2024-01-26T23:30:11.344 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-26T23:30:11.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.478+0000 7f6bf232c700 1 Processor -- start 2024-01-26T23:30:11.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.478+0000 7f6bf232c700 1 -- start start 2024-01-26T23:30:11.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.479+0000 7f6bf232c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6bec159510 0x7f6bec1598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:11.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.479+0000 7f6bf232c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bec0ca690 0x7f6bec0cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:11.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.479+0000 7f6bf232c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bec069260 con 0x7f6bec0ca220 2024-01-26T23:30:11.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.479+0000 7f6bf232c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bec15d670 con 0x7f6bec159510 2024-01-26T23:30:11.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.479+0000 7f6bf232c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec0c9750 con 0x7f6bec0ca690 2024-01-26T23:30:11.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.479+0000 7f6bf08c9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bec0ca690 0x7f6bec0cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:11.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.479+0000 7f6bf08c9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bec0ca690 0x7f6bec0cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:37288/0 (socket says 172.21.15.46:37288) 2024-01-26T23:30:11.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.479+0000 7f6bf08c9700 1 -- 172.21.15.46:0/2333769023 learned_addr learned my addr 172.21.15.46:0/2333769023 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:11.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.479+0000 7f6bebfff700 1 --2- 172.21.15.46:0/2333769023 >> v2:172.21.15.106:3300/0 conn(0x7f6bec159510 0x7f6bec1598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:11.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.479+0000 7f6beb7fe700 1 --1- 172.21.15.46:0/2333769023 >> v1:172.21.15.106:6789/0 conn(0x7f6bec0ca220 0x7f6bec0c8be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:59440/0 (socket says 172.21.15.46:59440) 2024-01-26T23:30:11.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.480+0000 7f6bf08c9700 1 -- 172.21.15.46:0/2333769023 >> v2:172.21.15.106:3300/0 conn(0x7f6bec159510 msgr2=0x7f6bec1598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:11.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.480+0000 7f6bf08c9700 1 --2- 172.21.15.46:0/2333769023 >> v2:172.21.15.106:3300/0 conn(0x7f6bec159510 0x7f6bec1598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.480+0000 7f6bf08c9700 1 -- 172.21.15.46:0/2333769023 >> v1:172.21.15.106:6789/0 conn(0x7f6bec0ca220 legacy=0x7f6bec0c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.480+0000 7f6bf08c9700 1 -- 172.21.15.46:0/2333769023 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bec0c9a30 con 0x7f6bec0ca690 2024-01-26T23:30:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.480+0000 7f6beaffd700 1 -- 172.21.15.46:0/2333769023 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815926742 0 0) 0x7f6bec069260 con 0x7f6bec0ca220 2024-01-26T23:30:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.480+0000 7f6bf08c9700 1 --2- 172.21.15.46:0/2333769023 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bec0ca690 0x7f6bec0cc380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6be4009f20 tx=0x7f6be4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e87611b19d48c03 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.480+0000 7f6beaffd700 1 -- 172.21.15.46:0/2333769023 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be400e070 con 0x7f6bec0ca690 2024-01-26T23:30:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.480+0000 7f6beaffd700 1 -- 172.21.15.46:0/2333769023 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be400b9d0 con 0x7f6bec0ca690 2024-01-26T23:30:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.480+0000 7f6beaffd700 1 -- 172.21.15.46:0/2333769023 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be40092e0 con 0x7f6bec0ca690 2024-01-26T23:30:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.481+0000 7f6bf232c700 1 -- 172.21.15.46:0/2333769023 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bec0ca690 msgr2=0x7f6bec0cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.481+0000 7f6bf232c700 1 --2- 172.21.15.46:0/2333769023 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bec0ca690 0x7f6bec0cc380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6be4009f20 tx=0x7f6be4009840 comp rx=0 tx=0).stop 2024-01-26T23:30:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.481+0000 7f6bf232c700 1 -- 172.21.15.46:0/2333769023 shutdown_connections 2024-01-26T23:30:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.481+0000 7f6bf232c700 1 --2- 172.21.15.46:0/2333769023 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bec0ca690 0x7f6bec0cc380 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.481+0000 7f6bf232c700 1 --2- 172.21.15.46:0/2333769023 >> v2:172.21.15.106:3300/0 conn(0x7f6bec159510 0x7f6bec1598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.481+0000 7f6bf232c700 1 -- 172.21.15.46:0/2333769023 >> 172.21.15.46:0/2333769023 conn(0x7f6bec06bd90 msgr2=0x7f6bec06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.481+0000 7f6bf232c700 1 -- 172.21.15.46:0/2333769023 shutdown_connections 2024-01-26T23:30:11.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.481+0000 7f6bf232c700 1 -- 172.21.15.46:0/2333769023 wait complete. 2024-01-26T23:30:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.482+0000 7f6bf232c700 1 Processor -- start 2024-01-26T23:30:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.482+0000 7f6bf232c700 1 -- start start 2024-01-26T23:30:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.483+0000 7f6bf232c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bec0ca220 0x7f6bec154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.483+0000 7f6bf232c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6bec159510 0x7f6bec156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.483+0000 7f6bf232c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bec157f30 0x7f6bec0b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.483+0000 7f6bf232c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec1577f0 con 0x7f6bec0ca220 2024-01-26T23:30:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.483+0000 7f6bf232c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec0b1750 con 0x7f6bec159510 2024-01-26T23:30:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.483+0000 7f6bf232c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec157230 con 0x7f6bec157f30 2024-01-26T23:30:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.483+0000 7f6bebfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bec0ca220 0x7f6bec154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.483+0000 7f6bebfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bec0ca220 0x7f6bec154120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40312/0 (socket says 172.21.15.46:40312) 2024-01-26T23:30:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.483+0000 7f6bebfff700 1 -- 172.21.15.46:0/565788187 learned_addr learned my addr 172.21.15.46:0/565788187 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.483+0000 7f6bf08c9700 1 --2- 172.21.15.46:0/565788187 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bec157f30 0x7f6bec0b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.483+0000 7f6beb7fe700 1 --2- 172.21.15.46:0/565788187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6bec159510 0x7f6bec156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.484+0000 7f6bebfff700 1 -- 172.21.15.46:0/565788187 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bec157f30 msgr2=0x7f6bec0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.484+0000 7f6bebfff700 1 --2- 172.21.15.46:0/565788187 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bec157f30 0x7f6bec0b1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.484+0000 7f6bebfff700 1 -- 172.21.15.46:0/565788187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6bec159510 msgr2=0x7f6bec156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.484+0000 7f6bebfff700 1 --2- 172.21.15.46:0/565788187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6bec159510 0x7f6bec156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.484+0000 7f6bebfff700 1 -- 172.21.15.46:0/565788187 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be4009ae0 con 0x7f6bec0ca220 2024-01-26T23:30:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.484+0000 7f6bf08c9700 1 --2- 172.21.15.46:0/565788187 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bec157f30 0x7f6bec0b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.484+0000 7f6bebfff700 1 --2- 172.21.15.46:0/565788187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bec0ca220 0x7f6bec154120 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc00e960 tx=0x7f6bdc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.484+0000 7f6be3fff700 1 -- 172.21.15.46:0/565788187 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc013680 con 0x7f6bec0ca220 2024-01-26T23:30:11.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.484+0000 7f6bf232c700 1 -- 172.21.15.46:0/565788187 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bec157510 con 0x7f6bec0ca220 2024-01-26T23:30:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.484+0000 7f6be3fff700 1 -- 172.21.15.46:0/565788187 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bdc0137e0 con 0x7f6bec0ca220 2024-01-26T23:30:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.484+0000 7f6be3fff700 1 -- 172.21.15.46:0/565788187 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc013b80 con 0x7f6bec0ca220 2024-01-26T23:30:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.484+0000 7f6bf232c700 1 -- 172.21.15.46:0/565788187 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bec154660 con 0x7f6bec0ca220 2024-01-26T23:30:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.485+0000 7f6bf232c700 1 -- 172.21.15.46:0/565788187 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bec04c710 con 0x7f6bec0ca220 2024-01-26T23:30:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.486+0000 7f6be3fff700 1 -- 172.21.15.46:0/565788187 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6bdc0123d0 con 0x7f6bec0ca220 2024-01-26T23:30:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.486+0000 7f6be3fff700 1 --2- 172.21.15.46:0/565788187 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6bcc066f30 0x7f6bcc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.486+0000 7f6be3fff700 1 -- 172.21.15.46:0/565788187 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f6bdc084960 con 0x7f6bec0ca220 2024-01-26T23:30:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.486+0000 7f6beb7fe700 1 --2- 172.21.15.46:0/565788187 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6bcc066f30 0x7f6bcc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.487+0000 7f6beb7fe700 1 --2- 172.21.15.46:0/565788187 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6bcc066f30 0x7f6bcc0693e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6bec159c40 tx=0x7f6bd4007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.491+0000 7f6be3fff700 1 -- 172.21.15.46:0/565788187 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6bdc0518e0 con 0x7f6bec0ca220 2024-01-26T23:30:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.693+0000 7f6bf232c700 1 -- 172.21.15.46:0/565788187 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6bec0c9d10 con 0x7f6bec0ca220 2024-01-26T23:30:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.694+0000 7f6be3fff700 1 -- 172.21.15.46:0/565788187 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6bdc054f00 con 0x7f6bec0ca220 2024-01-26T23:30:11.695 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:30:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.696+0000 7f6bf232c700 1 -- 172.21.15.46:0/565788187 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6bcc066f30 msgr2=0x7f6bcc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.696+0000 7f6bf232c700 1 --2- 172.21.15.46:0/565788187 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6bcc066f30 0x7f6bcc0693e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6bec159c40 tx=0x7f6bd4007d50 comp rx=0 tx=0).stop 2024-01-26T23:30:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.696+0000 7f6bf232c700 1 -- 172.21.15.46:0/565788187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bec0ca220 msgr2=0x7f6bec154120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.696+0000 7f6bf232c700 1 --2- 172.21.15.46:0/565788187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bec0ca220 0x7f6bec154120 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc00e960 tx=0x7f6bdc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.696+0000 7f6bf232c700 1 -- 172.21.15.46:0/565788187 shutdown_connections 2024-01-26T23:30:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.696+0000 7f6bf232c700 1 --2- 172.21.15.46:0/565788187 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6bcc066f30 0x7f6bcc0693e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.696+0000 7f6bf232c700 1 --2- 172.21.15.46:0/565788187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bec0ca220 0x7f6bec154120 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.696+0000 7f6bf232c700 1 --2- 172.21.15.46:0/565788187 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bec157f30 0x7f6bec0b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.696+0000 7f6bf232c700 1 --2- 172.21.15.46:0/565788187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6bec159510 0x7f6bec156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.696+0000 7f6bf232c700 1 -- 172.21.15.46:0/565788187 >> 172.21.15.46:0/565788187 conn(0x7f6bec06bd90 msgr2=0x7f6bec06c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.696+0000 7f6bf232c700 1 -- 172.21.15.46:0/565788187 shutdown_connections 2024-01-26T23:30:11.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.696+0000 7f6bf232c700 1 -- 172.21.15.46:0/565788187 wait complete. 2024-01-26T23:30:11.710 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-26T23:30:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.844+0000 7f57febe3700 1 Processor -- start 2024-01-26T23:30:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.844+0000 7f57febe3700 1 -- start start 2024-01-26T23:30:11.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.845+0000 7f57febe3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57f806f2a0 0x7f57f806c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:11.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.845+0000 7f57febe3700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f80c8b90 con 0x7f57f8156940 2024-01-26T23:30:11.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.845+0000 7f57febe3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f80c8e50 con 0x7f57f8156570 2024-01-26T23:30:11.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.845+0000 7f57febe3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f806d0b0 con 0x7f57f806f2a0 2024-01-26T23:30:11.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.845+0000 7f57f7fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f57f8156940 0x7f57f806eb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50414/0 (socket says 172.21.15.46:50414) 2024-01-26T23:30:11.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.845+0000 7f57f7fff700 1 -- 172.21.15.46:0/3986358233 learned_addr learned my addr 172.21.15.46:0/3986358233 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:11.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.845+0000 7f57fd180700 1 --2- 172.21.15.46:0/3986358233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57f806f2a0 0x7f57f806c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:11.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.846+0000 7f57f77fe700 1 -- 172.21.15.46:0/3986358233 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994546670 0 0) 0x7f57f80c8b90 con 0x7f57f8156940 2024-01-26T23:30:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.846+0000 7f57f77fe700 1 -- 172.21.15.46:0/3986358233 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e8003570 con 0x7f57f8156940 2024-01-26T23:30:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.846+0000 7f57f77fe700 1 -- 172.21.15.46:0/3986358233 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827398149 0 0) 0x7f57f80c8e50 con 0x7f57f8156570 2024-01-26T23:30:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.846+0000 7f57f77fe700 1 -- 172.21.15.46:0/3986358233 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57f80c8b90 con 0x7f57f8156570 2024-01-26T23:30:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.846+0000 7f57f77fe700 1 -- 172.21.15.46:0/3986358233 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 142950657 0 0) 0x7f57e8003570 con 0x7f57f8156940 2024-01-26T23:30:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.846+0000 7f57fd180700 1 -- 172.21.15.46:0/3986358233 >> v1:172.21.15.106:6789/0 conn(0x7f57f8156570 legacy=0x7f57f80c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.846+0000 7f57fd180700 1 -- 172.21.15.46:0/3986358233 >> v1:172.21.15.46:6789/0 conn(0x7f57f8156940 legacy=0x7f57f806eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.846+0000 7f57fd180700 1 -- 172.21.15.46:0/3986358233 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f80c4c40 con 0x7f57f806f2a0 2024-01-26T23:30:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.846+0000 7f57fd180700 1 --2- 172.21.15.46:0/3986358233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57f806f2a0 0x7f57f806c110 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f57f0009e90 tx=0x7f57f0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b11823808b30c13 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.846+0000 7f57f77fe700 1 -- 172.21.15.46:0/3986358233 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f000e070 con 0x7f57f806f2a0 2024-01-26T23:30:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.846+0000 7f57f77fe700 1 -- 172.21.15.46:0/3986358233 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f000b870 con 0x7f57f806f2a0 2024-01-26T23:30:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.846+0000 7f57f77fe700 1 -- 172.21.15.46:0/3986358233 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e4001470 con 0x7f57f806f2a0 2024-01-26T23:30:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.847+0000 7f57febe3700 1 -- 172.21.15.46:0/3986358233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57f806f2a0 msgr2=0x7f57f806c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.847+0000 7f57febe3700 1 --2- 172.21.15.46:0/3986358233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57f806f2a0 0x7f57f806c110 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f57f0009e90 tx=0x7f57f0009780 comp rx=0 tx=0).stop 2024-01-26T23:30:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.847+0000 7f57febe3700 1 -- 172.21.15.46:0/3986358233 shutdown_connections 2024-01-26T23:30:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.847+0000 7f57febe3700 1 --2- 172.21.15.46:0/3986358233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57f806f2a0 0x7f57f806c110 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.847+0000 7f57febe3700 1 -- 172.21.15.46:0/3986358233 >> 172.21.15.46:0/3986358233 conn(0x7f57f80cdf50 msgr2=0x7f57f80c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.847+0000 7f57febe3700 1 -- 172.21.15.46:0/3986358233 shutdown_connections 2024-01-26T23:30:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.847+0000 7f57febe3700 1 -- 172.21.15.46:0/3986358233 wait complete. 2024-01-26T23:30:11.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.849+0000 7f57febe3700 1 Processor -- start 2024-01-26T23:30:11.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.849+0000 7f57febe3700 1 -- start start 2024-01-26T23:30:11.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57febe3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57f806f2a0 0x7f57f816c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:11.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57febe3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57f8156570 0x7f57f8156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:11.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57febe3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57f8156940 0x7f57f80c3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57febe3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f816a310 con 0x7f57f8156570 2024-01-26T23:30:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57febe3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f816c9a0 con 0x7f57f806f2a0 2024-01-26T23:30:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57febe3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f8169700 con 0x7f57f8156940 2024-01-26T23:30:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57f7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57f8156570 0x7f57f8156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57f7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57f8156570 0x7f57f8156e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40316/0 (socket says 172.21.15.46:40316) 2024-01-26T23:30:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57f7fff700 1 -- 172.21.15.46:0/17968113 learned_addr learned my addr 172.21.15.46:0/17968113 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57fd180700 1 --2- 172.21.15.46:0/17968113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57f8156940 0x7f57f80c3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57fc97f700 1 --2- 172.21.15.46:0/17968113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57f806f2a0 0x7f57f816c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57f7fff700 1 -- 172.21.15.46:0/17968113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57f8156940 msgr2=0x7f57f80c3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57f7fff700 1 --2- 172.21.15.46:0/17968113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57f8156940 0x7f57f80c3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57f7fff700 1 -- 172.21.15.46:0/17968113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57f806f2a0 msgr2=0x7f57f816c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57f7fff700 1 --2- 172.21.15.46:0/17968113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57f806f2a0 0x7f57f816c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57f7fff700 1 -- 172.21.15.46:0/17968113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f0009a20 con 0x7f57f8156570 2024-01-26T23:30:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57fc97f700 1 --2- 172.21.15.46:0/17968113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57f806f2a0 0x7f57f816c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.850+0000 7f57f7fff700 1 --2- 172.21.15.46:0/17968113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57f8156570 0x7f57f8156e30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f57e400cba0 tx=0x7f57e4001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.851+0000 7f57f4ff9700 1 -- 172.21.15.46:0/17968113 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e4007e90 con 0x7f57f8156570 2024-01-26T23:30:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.851+0000 7f57f4ff9700 1 -- 172.21.15.46:0/17968113 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e4007360 con 0x7f57f8156570 2024-01-26T23:30:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.851+0000 7f57febe3700 1 -- 172.21.15.46:0/17968113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f81699e0 con 0x7f57f8156570 2024-01-26T23:30:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.851+0000 7f57f4ff9700 1 -- 172.21.15.46:0/17968113 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e4004830 con 0x7f57f8156570 2024-01-26T23:30:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.851+0000 7f57febe3700 1 -- 172.21.15.46:0/17968113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f8169fa0 con 0x7f57f8156570 2024-01-26T23:30:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.852+0000 7f57febe3700 1 -- 172.21.15.46:0/17968113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f816a4c0 con 0x7f57f8156570 2024-01-26T23:30:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.852+0000 7f57f4ff9700 1 -- 172.21.15.46:0/17968113 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f57e4004ba0 con 0x7f57f8156570 2024-01-26T23:30:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.853+0000 7f57f4ff9700 1 --2- 172.21.15.46:0/17968113 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57dc066f30 0x7f57dc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.853+0000 7f57f4ff9700 1 -- 172.21.15.46:0/17968113 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f57e4084060 con 0x7f57f8156570 2024-01-26T23:30:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.853+0000 7f57fc97f700 1 --2- 172.21.15.46:0/17968113 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57dc066f30 0x7f57dc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.853+0000 7f57fc97f700 1 --2- 172.21.15.46:0/17968113 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57dc066f30 0x7f57dc0693e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f57f80cb340 tx=0x7f57ec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:11.857+0000 7f57f4ff9700 1 -- 172.21.15.46:0/17968113 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f57f816a4c0 con 0x7f57f8156570 2024-01-26T23:30:12.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.062+0000 7f57febe3700 1 -- 172.21.15.46:0/17968113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f57f816a4c0 con 0x7f57f8156570 2024-01-26T23:30:12.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.063+0000 7f57f4ff9700 1 -- 172.21.15.46:0/17968113 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f57e4018020 con 0x7f57f8156570 2024-01-26T23:30:12.064 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-26T23:30:12.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.065+0000 7f57febe3700 1 -- 172.21.15.46:0/17968113 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57dc066f30 msgr2=0x7f57dc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:12.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.065+0000 7f57febe3700 1 --2- 172.21.15.46:0/17968113 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57dc066f30 0x7f57dc0693e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f57f80cb340 tx=0x7f57ec009040 comp rx=0 tx=0).stop 2024-01-26T23:30:12.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.065+0000 7f57febe3700 1 -- 172.21.15.46:0/17968113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57f8156570 msgr2=0x7f57f8156e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:12.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.065+0000 7f57febe3700 1 --2- 172.21.15.46:0/17968113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57f8156570 0x7f57f8156e30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f57e400cba0 tx=0x7f57e4001b50 comp rx=0 tx=0).stop 2024-01-26T23:30:12.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.065+0000 7f57febe3700 1 -- 172.21.15.46:0/17968113 shutdown_connections 2024-01-26T23:30:12.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.065+0000 7f57febe3700 1 --2- 172.21.15.46:0/17968113 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57dc066f30 0x7f57dc0693e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.065+0000 7f57febe3700 1 --2- 172.21.15.46:0/17968113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57f8156940 0x7f57f80c3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.065+0000 7f57febe3700 1 --2- 172.21.15.46:0/17968113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57f806f2a0 0x7f57f816c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.065+0000 7f57febe3700 1 --2- 172.21.15.46:0/17968113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57f8156570 0x7f57f8156e30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.065+0000 7f57febe3700 1 -- 172.21.15.46:0/17968113 >> 172.21.15.46:0/17968113 conn(0x7f57f80cdf50 msgr2=0x7f57f80cb890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.065+0000 7f57febe3700 1 -- 172.21.15.46:0/17968113 shutdown_connections 2024-01-26T23:30:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.065+0000 7f57febe3700 1 -- 172.21.15.46:0/17968113 wait complete. 2024-01-26T23:30:12.079 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-01-26T23:30:12.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.214+0000 7fe5d0ee3700 1 Processor -- start 2024-01-26T23:30:12.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.214+0000 7fe5d0ee3700 1 -- start start 2024-01-26T23:30:12.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.215+0000 7fe5d0ee3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5cc0c8a30 0x7fe5cc06b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:12.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.215+0000 7fe5d0ee3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5cc0c8e80 0x7fe5cc068940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:12.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.215+0000 7fe5d0ee3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5cc15a720 con 0x7fe5cc0c8660 2024-01-26T23:30:12.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.215+0000 7fe5d0ee3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5cc06c870 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.215+0000 7fe5d0ee3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc06c2b0 con 0x7fe5cc0c8a30 2024-01-26T23:30:12.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.215+0000 7fe5cad9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5cc0c8e80 0x7fe5cc068940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:12.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.215+0000 7fe5cad9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5cc0c8e80 0x7fe5cc068940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40328/0 (socket says 172.21.15.46:40328) 2024-01-26T23:30:12.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.215+0000 7fe5cad9d700 1 -- 172.21.15.46:0/3082315873 learned_addr learned my addr 172.21.15.46:0/3082315873 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:12.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.215+0000 7fe5c9d9b700 1 --2- 172.21.15.46:0/3082315873 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5cc0c8a30 0x7fe5cc06b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:12.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.215+0000 7fe5cad9d700 1 -- 172.21.15.46:0/3082315873 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5cc0c8a30 msgr2=0x7fe5cc06b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:12.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.215+0000 7fe5cad9d700 1 --2- 172.21.15.46:0/3082315873 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5cc0c8a30 0x7fe5cc06b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.215+0000 7fe5cad9d700 1 -- 172.21.15.46:0/3082315873 >> v1:172.21.15.106:6789/0 conn(0x7fe5cc0c8660 legacy=0x7fe5cc15abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:12.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.215+0000 7fe5cad9d700 1 -- 172.21.15.46:0/3082315873 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5cc06c590 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.215+0000 7fe5cad9d700 1 --2- 172.21.15.46:0/3082315873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5cc0c8e80 0x7fe5cc068940 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400d1a0 tx=0x7fe5c400ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d92b19c8d4aabe5a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:12.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.216+0000 7fe5c959a700 1 -- 172.21.15.46:0/3082315873 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c4010070 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.216+0000 7fe5c959a700 1 -- 172.21.15.46:0/3082315873 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c400de10 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.216+0000 7fe5c959a700 1 -- 172.21.15.46:0/3082315873 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c400b320 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.216+0000 7fe5d0ee3700 1 -- 172.21.15.46:0/3082315873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5cc0c8e80 msgr2=0x7fe5cc068940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:12.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.216+0000 7fe5d0ee3700 1 --2- 172.21.15.46:0/3082315873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5cc0c8e80 0x7fe5cc068940 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400d1a0 tx=0x7fe5c400ba50 comp rx=0 tx=0).stop 2024-01-26T23:30:12.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.216+0000 7fe5d0ee3700 1 -- 172.21.15.46:0/3082315873 shutdown_connections 2024-01-26T23:30:12.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.216+0000 7fe5d0ee3700 1 --2- 172.21.15.46:0/3082315873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5cc0c8e80 0x7fe5cc068940 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.216+0000 7fe5d0ee3700 1 --2- 172.21.15.46:0/3082315873 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5cc0c8a30 0x7fe5cc06b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.216+0000 7fe5d0ee3700 1 -- 172.21.15.46:0/3082315873 >> 172.21.15.46:0/3082315873 conn(0x7fe5cc0ca100 msgr2=0x7fe5cc06e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:12.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.216+0000 7fe5d0ee3700 1 -- 172.21.15.46:0/3082315873 shutdown_connections 2024-01-26T23:30:12.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.216+0000 7fe5d0ee3700 1 -- 172.21.15.46:0/3082315873 wait complete. 2024-01-26T23:30:12.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.218+0000 7fe5d0ee3700 1 Processor -- start 2024-01-26T23:30:12.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.218+0000 7fe5d0ee3700 1 -- start start 2024-01-26T23:30:12.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.219+0000 7fe5d0ee3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5cc0c8660 0x7fe5cc0c0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:12.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.219+0000 7fe5d0ee3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5cc0c8a30 0x7fe5cc0c1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:12.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.219+0000 7fe5d0ee3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5cc0c8e80 0x7fe5cc165170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:12.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.219+0000 7fe5d0ee3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc0c1b80 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.219+0000 7fe5d0ee3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc0c1e60 con 0x7fe5cc0c8660 2024-01-26T23:30:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.219+0000 7fe5d0ee3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc0c2140 con 0x7fe5cc0c8a30 2024-01-26T23:30:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.219+0000 7fe5cad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5cc0c8e80 0x7fe5cc165170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.219+0000 7fe5cad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5cc0c8e80 0x7fe5cc165170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40332/0 (socket says 172.21.15.46:40332) 2024-01-26T23:30:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.219+0000 7fe5cad9d700 1 -- 172.21.15.46:0/3798848971 learned_addr learned my addr 172.21.15.46:0/3798848971 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.219+0000 7fe5c9d9b700 1 --2- 172.21.15.46:0/3798848971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5cc0c8a30 0x7fe5cc0c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.219+0000 7fe5ca59c700 1 --2- 172.21.15.46:0/3798848971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5cc0c8660 0x7fe5cc0c0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.219+0000 7fe5cad9d700 1 -- 172.21.15.46:0/3798848971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5cc0c8a30 msgr2=0x7fe5cc0c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.219+0000 7fe5cad9d700 1 --2- 172.21.15.46:0/3798848971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5cc0c8a30 0x7fe5cc0c1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.219+0000 7fe5cad9d700 1 -- 172.21.15.46:0/3798848971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5cc0c8660 msgr2=0x7fe5cc0c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.220+0000 7fe5cad9d700 1 --2- 172.21.15.46:0/3798848971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5cc0c8660 0x7fe5cc0c0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.220+0000 7fe5cad9d700 1 -- 172.21.15.46:0/3798848971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c400bcf0 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.220+0000 7fe5cad9d700 1 --2- 172.21.15.46:0/3798848971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5cc0c8e80 0x7fe5cc165170 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4014000 tx=0x7fe5c401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.220+0000 7fe5ba7fc700 1 -- 172.21.15.46:0/3798848971 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c400ac90 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.220+0000 7fe5ba7fc700 1 -- 172.21.15.46:0/3798848971 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c400adf0 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.220+0000 7fe5d0ee3700 1 -- 172.21.15.46:0/3798848971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5cc0afa10 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.220+0000 7fe5ba7fc700 1 -- 172.21.15.46:0/3798848971 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c401e660 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.220+0000 7fe5d0ee3700 1 -- 172.21.15.46:0/3798848971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5cc1543e0 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.221+0000 7fe5d0ee3700 1 -- 172.21.15.46:0/3798848971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5cc04c710 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.221+0000 7fe5ba7fc700 1 -- 172.21.15.46:0/3798848971 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe5c4003040 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.222+0000 7fe5ba7fc700 1 --2- 172.21.15.46:0/3798848971 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe5ac066f30 0x7fe5ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:12.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.222+0000 7fe5ba7fc700 1 -- 172.21.15.46:0/3798848971 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fe5c4088190 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.222+0000 7fe5ca59c700 1 --2- 172.21.15.46:0/3798848971 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe5ac066f30 0x7fe5ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:12.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.223+0000 7fe5ca59c700 1 --2- 172.21.15.46:0/3798848971 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe5ac066f30 0x7fe5ac0693e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc000c00 tx=0x7fe5bc007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:12.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.226+0000 7fe5ba7fc700 1 -- 172.21.15.46:0/3798848971 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe5c4055110 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.433+0000 7fe5d0ee3700 1 -- 172.21.15.46:0/3798848971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe5cc15a3b0 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.433+0000 7fe5ba7fc700 1 -- 172.21.15.46:0/3798848971 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe5c4058730 con 0x7fe5cc0c8e80 2024-01-26T23:30:12.434 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:30:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.436+0000 7fe5d0ee3700 1 -- 172.21.15.46:0/3798848971 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe5ac066f30 msgr2=0x7fe5ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.436+0000 7fe5d0ee3700 1 --2- 172.21.15.46:0/3798848971 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe5ac066f30 0x7fe5ac0693e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc000c00 tx=0x7fe5bc007e70 comp rx=0 tx=0).stop 2024-01-26T23:30:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.436+0000 7fe5d0ee3700 1 -- 172.21.15.46:0/3798848971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5cc0c8e80 msgr2=0x7fe5cc165170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.436+0000 7fe5d0ee3700 1 --2- 172.21.15.46:0/3798848971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5cc0c8e80 0x7fe5cc165170 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4014000 tx=0x7fe5c401a000 comp rx=0 tx=0).stop 2024-01-26T23:30:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.436+0000 7fe5d0ee3700 1 -- 172.21.15.46:0/3798848971 shutdown_connections 2024-01-26T23:30:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.436+0000 7fe5d0ee3700 1 --2- 172.21.15.46:0/3798848971 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe5ac066f30 0x7fe5ac0693e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.436+0000 7fe5d0ee3700 1 --2- 172.21.15.46:0/3798848971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5cc0c8e80 0x7fe5cc165170 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.436+0000 7fe5d0ee3700 1 --2- 172.21.15.46:0/3798848971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5cc0c8660 0x7fe5cc0c0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.436+0000 7fe5d0ee3700 1 --2- 172.21.15.46:0/3798848971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5cc0c8a30 0x7fe5cc0c1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.436+0000 7fe5d0ee3700 1 -- 172.21.15.46:0/3798848971 >> 172.21.15.46:0/3798848971 conn(0x7fe5cc0ca100 msgr2=0x7fe5cc156160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.436+0000 7fe5d0ee3700 1 -- 172.21.15.46:0/3798848971 shutdown_connections 2024-01-26T23:30:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.436+0000 7fe5d0ee3700 1 -- 172.21.15.46:0/3798848971 wait complete. 2024-01-26T23:30:12.450 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:30:12.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.581+0000 7f1d73c0c700 1 Processor -- start 2024-01-26T23:30:12.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.582+0000 7f1d73c0c700 1 -- start start 2024-01-26T23:30:12.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.583+0000 7f1d73c0c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1d6c0ccff0 0x7f1d6c068340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.583+0000 7f1d73c0c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d6c0b2590 0x7f1d6c0b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.583+0000 7f1d73c0c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6c0b2f10 con 0x7f1d6c0ccc20 2024-01-26T23:30:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.583+0000 7f1d73c0c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6c158020 con 0x7f1d6c0b2590 2024-01-26T23:30:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.583+0000 7f1d73c0c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6c158300 con 0x7f1d6c0ccff0 2024-01-26T23:30:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.583+0000 7f1d721a9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d6c0b2590 0x7f1d6c0b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.583+0000 7f1d719a8700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1d6c0ccc20 0x7f1d6c06eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50418/0 (socket says 172.21.15.46:50418) 2024-01-26T23:30:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.583+0000 7f1d719a8700 1 -- 172.21.15.46:0/2338468384 learned_addr learned my addr 172.21.15.46:0/2338468384 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.583+0000 7f1d711a7700 1 --2- 172.21.15.46:0/2338468384 >> v2:172.21.15.106:3300/0 conn(0x7f1d6c0ccff0 0x7f1d6c068340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.583+0000 7f1d721a9700 1 -- 172.21.15.46:0/2338468384 >> v2:172.21.15.106:3300/0 conn(0x7f1d6c0ccff0 msgr2=0x7f1d6c068340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.583+0000 7f1d721a9700 1 --2- 172.21.15.46:0/2338468384 >> v2:172.21.15.106:3300/0 conn(0x7f1d6c0ccff0 0x7f1d6c068340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.583+0000 7f1d721a9700 1 -- 172.21.15.46:0/2338468384 >> v1:172.21.15.46:6789/0 conn(0x7f1d6c0ccc20 legacy=0x7f1d6c06eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.583+0000 7f1d709a6700 1 -- 172.21.15.46:0/2338468384 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772645652 0 0) 0x7f1d6c0b2f10 con 0x7f1d6c0ccc20 2024-01-26T23:30:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.583+0000 7f1d721a9700 1 -- 172.21.15.46:0/2338468384 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c1585e0 con 0x7f1d6c0b2590 2024-01-26T23:30:12.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.583+0000 7f1d721a9700 1 --2- 172.21.15.46:0/2338468384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d6c0b2590 0x7f1d6c0b29d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c158ba0 tx=0x7f1d64009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9049767f4d21fc server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:12.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.584+0000 7f1d709a6700 1 -- 172.21.15.46:0/2338468384 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6400e070 con 0x7f1d6c0b2590 2024-01-26T23:30:12.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.584+0000 7f1d709a6700 1 -- 172.21.15.46:0/2338468384 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6400bb60 con 0x7f1d6c0b2590 2024-01-26T23:30:12.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.584+0000 7f1d709a6700 1 -- 172.21.15.46:0/2338468384 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d64012430 con 0x7f1d6c0b2590 2024-01-26T23:30:12.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.584+0000 7f1d73c0c700 1 -- 172.21.15.46:0/2338468384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d6c0b2590 msgr2=0x7f1d6c0b29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:12.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.584+0000 7f1d73c0c700 1 --2- 172.21.15.46:0/2338468384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d6c0b2590 0x7f1d6c0b29d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c158ba0 tx=0x7f1d64009780 comp rx=0 tx=0).stop 2024-01-26T23:30:12.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.584+0000 7f1d73c0c700 1 -- 172.21.15.46:0/2338468384 shutdown_connections 2024-01-26T23:30:12.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.584+0000 7f1d73c0c700 1 --2- 172.21.15.46:0/2338468384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d6c0b2590 0x7f1d6c0b29d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.584+0000 7f1d73c0c700 1 --2- 172.21.15.46:0/2338468384 >> v2:172.21.15.106:3300/0 conn(0x7f1d6c0ccff0 0x7f1d6c068340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.584+0000 7f1d73c0c700 1 -- 172.21.15.46:0/2338468384 >> 172.21.15.46:0/2338468384 conn(0x7f1d6c06b630 msgr2=0x7f1d6c0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:12.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.584+0000 7f1d73c0c700 1 -- 172.21.15.46:0/2338468384 shutdown_connections 2024-01-26T23:30:12.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.584+0000 7f1d73c0c700 1 -- 172.21.15.46:0/2338468384 wait complete. 2024-01-26T23:30:12.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.586+0000 7f1d73c0c700 1 Processor -- start 2024-01-26T23:30:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.586+0000 7f1d73c0c700 1 -- start start 2024-01-26T23:30:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.586+0000 7f1d73c0c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d6c0b2590 0x7f1d6c0c7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.586+0000 7f1d73c0c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d6c0ccc20 0x7f1d6c0a5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.586+0000 7f1d73c0c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d6c0ccff0 0x7f1d6c1692f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.586+0000 7f1d73c0c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c0c7d00 con 0x7f1d6c0b2590 2024-01-26T23:30:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.586+0000 7f1d73c0c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c0b1820 con 0x7f1d6c0ccc20 2024-01-26T23:30:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.586+0000 7f1d73c0c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c0b1cb0 con 0x7f1d6c0ccff0 2024-01-26T23:30:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.586+0000 7f1d719a8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d6c0b2590 0x7f1d6c0c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.587+0000 7f1d719a8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d6c0b2590 0x7f1d6c0c7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40342/0 (socket says 172.21.15.46:40342) 2024-01-26T23:30:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.587+0000 7f1d719a8700 1 -- 172.21.15.46:0/432904259 learned_addr learned my addr 172.21.15.46:0/432904259 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.587+0000 7f1d711a7700 1 --2- 172.21.15.46:0/432904259 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d6c0ccc20 0x7f1d6c0a5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.587+0000 7f1d721a9700 1 --2- 172.21.15.46:0/432904259 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d6c0ccff0 0x7f1d6c1692f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:12.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.587+0000 7f1d719a8700 1 -- 172.21.15.46:0/432904259 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d6c0ccff0 msgr2=0x7f1d6c1692f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:12.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.587+0000 7f1d719a8700 1 --2- 172.21.15.46:0/432904259 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d6c0ccff0 0x7f1d6c1692f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.587+0000 7f1d719a8700 1 -- 172.21.15.46:0/432904259 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d6c0ccc20 msgr2=0x7f1d6c0a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:12.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.587+0000 7f1d719a8700 1 --2- 172.21.15.46:0/432904259 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d6c0ccc20 0x7f1d6c0a5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.587+0000 7f1d719a8700 1 -- 172.21.15.46:0/432904259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d64009a20 con 0x7f1d6c0b2590 2024-01-26T23:30:12.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.587+0000 7f1d719a8700 1 --2- 172.21.15.46:0/432904259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d6c0b2590 0x7f1d6c0c7600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1d6800c7e0 tx=0x7f1d68007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:12.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.587+0000 7f1d5dffb700 1 -- 172.21.15.46:0/432904259 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d68005b30 con 0x7f1d6c0b2590 2024-01-26T23:30:12.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.587+0000 7f1d5dffb700 1 -- 172.21.15.46:0/432904259 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d68005c90 con 0x7f1d6c0b2590 2024-01-26T23:30:12.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.587+0000 7f1d73c0c700 1 -- 172.21.15.46:0/432904259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6c0b0f30 con 0x7f1d6c0b2590 2024-01-26T23:30:12.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.587+0000 7f1d5dffb700 1 -- 172.21.15.46:0/432904259 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d68007300 con 0x7f1d6c0b2590 2024-01-26T23:30:12.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.588+0000 7f1d73c0c700 1 -- 172.21.15.46:0/432904259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6c0b11d0 con 0x7f1d6c0b2590 2024-01-26T23:30:12.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.589+0000 7f1d73c0c700 1 -- 172.21.15.46:0/432904259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d6c0b1b00 con 0x7f1d6c0b2590 2024-01-26T23:30:12.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.589+0000 7f1d5dffb700 1 -- 172.21.15.46:0/432904259 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1d680076b0 con 0x7f1d6c0b2590 2024-01-26T23:30:12.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.589+0000 7f1d5dffb700 1 --2- 172.21.15.46:0/432904259 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1d50066f30 0x7f1d500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:12.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.589+0000 7f1d5dffb700 1 -- 172.21.15.46:0/432904259 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f1d68084350 con 0x7f1d6c0b2590 2024-01-26T23:30:12.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.590+0000 7f1d711a7700 1 --2- 172.21.15.46:0/432904259 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1d50066f30 0x7f1d500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:12.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.590+0000 7f1d711a7700 1 --2- 172.21.15.46:0/432904259 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1d50066f30 0x7f1d500693e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c15e320 tx=0x7f1d60008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:12.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.594+0000 7f1d5dffb700 1 -- 172.21.15.46:0/432904259 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1d68051350 con 0x7f1d6c0b2590 2024-01-26T23:30:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.852+0000 7f1d73c0c700 1 -- 172.21.15.46:0/432904259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d6c04c710 con 0x7f1d6c0b2590 2024-01-26T23:30:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.852+0000 7f1d5dffb700 1 -- 172.21.15.46:0/432904259 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4593 (secure 0 0 0) 0x7f1d68054970 con 0x7f1d6c0b2590 2024-01-26T23:30:12.854 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:12.854 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3409861270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3409861270},{"type":"v1","addr":"172.21.15.46:6833","nonce":3409861270}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3319220815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3319220815},{"type":"v1","addr":"172.21.15.106:6836","nonce":3319220815}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4252,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/529198432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":529198432},{"type":"v1","addr":"172.21.15.106:6837","nonce":529198432}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:29:44.031963+0000","modified":"2024-01-26T23:30:03.652307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/688086235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":688086235},{"type":"v1","addr":"172.21.15.46:6835","nonce":688086235}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2024-01-26T23:30:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.854+0000 7f1d73c0c700 1 -- 172.21.15.46:0/432904259 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1d50066f30 msgr2=0x7f1d500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.854+0000 7f1d73c0c700 1 --2- 172.21.15.46:0/432904259 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1d50066f30 0x7f1d500693e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c15e320 tx=0x7f1d60008040 comp rx=0 tx=0).stop 2024-01-26T23:30:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.854+0000 7f1d73c0c700 1 -- 172.21.15.46:0/432904259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d6c0b2590 msgr2=0x7f1d6c0c7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.854+0000 7f1d73c0c700 1 --2- 172.21.15.46:0/432904259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d6c0b2590 0x7f1d6c0c7600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1d6800c7e0 tx=0x7f1d68007df0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.855+0000 7f1d73c0c700 1 -- 172.21.15.46:0/432904259 shutdown_connections 2024-01-26T23:30:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.855+0000 7f1d73c0c700 1 --2- 172.21.15.46:0/432904259 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1d50066f30 0x7f1d500693e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.855+0000 7f1d73c0c700 1 --2- 172.21.15.46:0/432904259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d6c0b2590 0x7f1d6c0c7600 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.855+0000 7f1d73c0c700 1 --2- 172.21.15.46:0/432904259 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d6c0ccff0 0x7f1d6c1692f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.855+0000 7f1d73c0c700 1 --2- 172.21.15.46:0/432904259 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d6c0ccc20 0x7f1d6c0a5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.855+0000 7f1d73c0c700 1 -- 172.21.15.46:0/432904259 >> 172.21.15.46:0/432904259 conn(0x7f1d6c06b630 msgr2=0x7f1d6c0cb7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.855+0000 7f1d73c0c700 1 -- 172.21.15.46:0/432904259 shutdown_connections 2024-01-26T23:30:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:12.855+0000 7f1d73c0c700 1 -- 172.21.15.46:0/432904259 wait complete. 2024-01-26T23:30:12.857 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 9 2024-01-26T23:30:12.869 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.000+0000 7f19880b6700 1 Processor -- start 2024-01-26T23:30:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.001+0000 7f19880b6700 1 -- start start 2024-01-26T23:30:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.001+0000 7f19880b6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f19801561d0 0x7f19800ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.001+0000 7f19880b6700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f19800b0280 0x7f19801565a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.002+0000 7f19880b6700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19800b0760 con 0x7f198006eb70 2024-01-26T23:30:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.002+0000 7f19880b6700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19800b0a40 con 0x7f19801561d0 2024-01-26T23:30:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.002+0000 7f19880b6700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19800b0d20 con 0x7f19800b0280 2024-01-26T23:30:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.002+0000 7f1985e52700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f19801561d0 0x7f19800ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.002+0000 7f1986653700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f198006eb70 0x7f198006c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50430/0 (socket says 172.21.15.46:50430) 2024-01-26T23:30:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.002+0000 7f1986653700 1 -- 172.21.15.46:0/1320258133 learned_addr learned my addr 172.21.15.46:0/1320258133 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.002+0000 7f1985651700 1 --2- 172.21.15.46:0/1320258133 >> v2:172.21.15.106:3300/0 conn(0x7f19800b0280 0x7f19801565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.002+0000 7f1985e52700 1 -- 172.21.15.46:0/1320258133 >> v2:172.21.15.106:3300/0 conn(0x7f19800b0280 msgr2=0x7f19801565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.002+0000 7f1985e52700 1 --2- 172.21.15.46:0/1320258133 >> v2:172.21.15.106:3300/0 conn(0x7f19800b0280 0x7f19801565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.002+0000 7f1985e52700 1 -- 172.21.15.46:0/1320258133 >> v1:172.21.15.46:6789/0 conn(0x7f198006eb70 legacy=0x7f198006c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.002+0000 7f1985e52700 1 -- 172.21.15.46:0/1320258133 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19800b1000 con 0x7f19801561d0 2024-01-26T23:30:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.002+0000 7f1984e50700 1 -- 172.21.15.46:0/1320258133 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724938222 0 0) 0x7f19800b0760 con 0x7f198006eb70 2024-01-26T23:30:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.002+0000 7f1985e52700 1 --2- 172.21.15.46:0/1320258133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19801561d0 0x7f19800ca5f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f198006d720 tx=0x7f19700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4d46dc659d241f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.002+0000 7f1984e50700 1 -- 172.21.15.46:0/1320258133 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970011630 con 0x7f19801561d0 2024-01-26T23:30:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.003+0000 7f1984e50700 1 -- 172.21.15.46:0/1320258133 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1970011790 con 0x7f19801561d0 2024-01-26T23:30:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.003+0000 7f1984e50700 1 -- 172.21.15.46:0/1320258133 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970011b30 con 0x7f19801561d0 2024-01-26T23:30:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.003+0000 7f19880b6700 1 -- 172.21.15.46:0/1320258133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19801561d0 msgr2=0x7f19800ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.003+0000 7f19880b6700 1 --2- 172.21.15.46:0/1320258133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19801561d0 0x7f19800ca5f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f198006d720 tx=0x7f19700091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.003+0000 7f19880b6700 1 -- 172.21.15.46:0/1320258133 shutdown_connections 2024-01-26T23:30:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.003+0000 7f19880b6700 1 --2- 172.21.15.46:0/1320258133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19801561d0 0x7f19800ca5f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.003+0000 7f19880b6700 1 --2- 172.21.15.46:0/1320258133 >> v2:172.21.15.106:3300/0 conn(0x7f19800b0280 0x7f19801565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.003+0000 7f19880b6700 1 -- 172.21.15.46:0/1320258133 >> 172.21.15.46:0/1320258133 conn(0x7f198006b630 msgr2=0x7f19800aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.003+0000 7f19880b6700 1 -- 172.21.15.46:0/1320258133 shutdown_connections 2024-01-26T23:30:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.003+0000 7f19880b6700 1 -- 172.21.15.46:0/1320258133 wait complete. 2024-01-26T23:30:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.004+0000 7f19880b6700 1 Processor -- start 2024-01-26T23:30:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.004+0000 7f19880b6700 1 -- start start 2024-01-26T23:30:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.005+0000 7f19880b6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f198006eb70 0x7f1980145f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.005+0000 7f19880b6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19800b0280 0x7f19800c2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.005+0000 7f19880b6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19801561d0 0x7f19800c3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.005+0000 7f19880b6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198015ab10 con 0x7f19801561d0 2024-01-26T23:30:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.005+0000 7f19880b6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198015a680 con 0x7f19800b0280 2024-01-26T23:30:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.005+0000 7f19880b6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1980159a70 con 0x7f198006eb70 2024-01-26T23:30:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.005+0000 7f1986653700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19801561d0 0x7f19800c3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.005+0000 7f1986653700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19801561d0 0x7f19800c3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40356/0 (socket says 172.21.15.46:40356) 2024-01-26T23:30:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.005+0000 7f1986653700 1 -- 172.21.15.46:0/976820283 learned_addr learned my addr 172.21.15.46:0/976820283 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.005+0000 7f1985651700 1 --2- 172.21.15.46:0/976820283 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19800b0280 0x7f19800c2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.005+0000 7f1985e52700 1 --2- 172.21.15.46:0/976820283 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f198006eb70 0x7f1980145f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.006+0000 7f1986653700 1 -- 172.21.15.46:0/976820283 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f198006eb70 msgr2=0x7f1980145f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.006+0000 7f1986653700 1 --2- 172.21.15.46:0/976820283 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f198006eb70 0x7f1980145f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.006+0000 7f1986653700 1 -- 172.21.15.46:0/976820283 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19800b0280 msgr2=0x7f19800c2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.006+0000 7f1986653700 1 --2- 172.21.15.46:0/976820283 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19800b0280 0x7f19800c2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.006+0000 7f1986653700 1 -- 172.21.15.46:0/976820283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1970009420 con 0x7f19801561d0 2024-01-26T23:30:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.006+0000 7f1986653700 1 --2- 172.21.15.46:0/976820283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19801561d0 0x7f19800c3170 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f197800cc90 tx=0x7f1978001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.006+0000 7f1975ffb700 1 -- 172.21.15.46:0/976820283 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197800a8b0 con 0x7f19801561d0 2024-01-26T23:30:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.006+0000 7f1975ffb700 1 -- 172.21.15.46:0/976820283 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f197800aa10 con 0x7f19801561d0 2024-01-26T23:30:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.006+0000 7f1975ffb700 1 -- 172.21.15.46:0/976820283 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197800add0 con 0x7f19801561d0 2024-01-26T23:30:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.006+0000 7f19880b6700 1 -- 172.21.15.46:0/976820283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1980159d50 con 0x7f19801561d0 2024-01-26T23:30:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.006+0000 7f19880b6700 1 -- 172.21.15.46:0/976820283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f198015a310 con 0x7f19801561d0 2024-01-26T23:30:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.007+0000 7f19880b6700 1 -- 172.21.15.46:0/976820283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f198015a810 con 0x7f19801561d0 2024-01-26T23:30:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.008+0000 7f1975ffb700 1 -- 172.21.15.46:0/976820283 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1978007300 con 0x7f19801561d0 2024-01-26T23:30:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.008+0000 7f1975ffb700 1 --2- 172.21.15.46:0/976820283 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1964066f30 0x7f19640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.008+0000 7f1975ffb700 1 -- 172.21.15.46:0/976820283 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f1978084b00 con 0x7f19801561d0 2024-01-26T23:30:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.008+0000 7f1985e52700 1 --2- 172.21.15.46:0/976820283 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1964066f30 0x7f19640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.009+0000 7f1985e52700 1 --2- 172.21.15.46:0/976820283 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1964066f30 0x7f19640693e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f19700093f0 tx=0x7f1970017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.013+0000 7f1975ffb700 1 -- 172.21.15.46:0/976820283 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1978051b00 con 0x7f19801561d0 2024-01-26T23:30:13.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.219+0000 7f19880b6700 1 -- 172.21.15.46:0/976820283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f198004c710 con 0x7f19801561d0 2024-01-26T23:30:13.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.220+0000 7f1975ffb700 1 -- 172.21.15.46:0/976820283 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7f1978055120 con 0x7f19801561d0 2024-01-26T23:30:13.221 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:13.221 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:49.075104+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:13.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.222+0000 7f19880b6700 1 -- 172.21.15.46:0/976820283 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1964066f30 msgr2=0x7f19640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.222+0000 7f19880b6700 1 --2- 172.21.15.46:0/976820283 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1964066f30 0x7f19640693e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f19700093f0 tx=0x7f1970017000 comp rx=0 tx=0).stop 2024-01-26T23:30:13.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.222+0000 7f19880b6700 1 -- 172.21.15.46:0/976820283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19801561d0 msgr2=0x7f19800c3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.222+0000 7f19880b6700 1 --2- 172.21.15.46:0/976820283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19801561d0 0x7f19800c3170 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f197800cc90 tx=0x7f1978001a80 comp rx=0 tx=0).stop 2024-01-26T23:30:13.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.222+0000 7f19880b6700 1 -- 172.21.15.46:0/976820283 shutdown_connections 2024-01-26T23:30:13.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.222+0000 7f19880b6700 1 --2- 172.21.15.46:0/976820283 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1964066f30 0x7f19640693e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.222+0000 7f19880b6700 1 --2- 172.21.15.46:0/976820283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19801561d0 0x7f19800c3170 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.222+0000 7f19880b6700 1 --2- 172.21.15.46:0/976820283 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f198006eb70 0x7f1980145f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.222+0000 7f19880b6700 1 --2- 172.21.15.46:0/976820283 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19800b0280 0x7f19800c2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.222+0000 7f19880b6700 1 -- 172.21.15.46:0/976820283 >> 172.21.15.46:0/976820283 conn(0x7f198006b630 msgr2=0x7f1980150c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:13.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.222+0000 7f19880b6700 1 -- 172.21.15.46:0/976820283 shutdown_connections 2024-01-26T23:30:13.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.222+0000 7f19880b6700 1 -- 172.21.15.46:0/976820283 wait complete. 2024-01-26T23:30:13.237 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-26T23:29:42.539759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-26T23:29:43.002669+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-01-26T23:29:46.646864+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-26T23:30:13.238 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:30:13.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.372+0000 7fbf6ca90700 1 Processor -- start 2024-01-26T23:30:13.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.372+0000 7fbf6ca90700 1 -- start start 2024-01-26T23:30:13.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.373+0000 7fbf6ca90700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbf68157860 0x7fbf680cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:13.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.373+0000 7fbf6ca90700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf680cea80 con 0x7fbf6806ddb0 2024-01-26T23:30:13.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.373+0000 7fbf6ca90700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf680cde70 con 0x7fbf68157490 2024-01-26T23:30:13.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.373+0000 7fbf6ca90700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf680cdc60 con 0x7fbf68157860 2024-01-26T23:30:13.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.373+0000 7fbf5dd9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbf68157860 0x7fbf680cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:13.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.373+0000 7fbf66d9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbf6806ddb0 0x7fbf680b1ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50444/0 (socket says 172.21.15.46:50444) 2024-01-26T23:30:13.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.373+0000 7fbf66d9d700 1 -- 172.21.15.46:0/2105538449 learned_addr learned my addr 172.21.15.46:0/2105538449 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:13.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.373+0000 7fbf5dd9b700 1 -- 172.21.15.46:0/2105538449 >> v1:172.21.15.106:6789/0 conn(0x7fbf68157490 legacy=0x7fbf6806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.373+0000 7fbf5dd9b700 1 -- 172.21.15.46:0/2105538449 >> v1:172.21.15.46:6789/0 conn(0x7fbf6806ddb0 legacy=0x7fbf680b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.373+0000 7fbf5dd9b700 1 -- 172.21.15.46:0/2105538449 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf680ce150 con 0x7fbf68157860 2024-01-26T23:30:13.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.373+0000 7fbf65d9b700 1 -- 172.21.15.46:0/2105538449 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364072077 0 0) 0x7fbf680cea80 con 0x7fbf6806ddb0 2024-01-26T23:30:13.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.374+0000 7fbf5dd9b700 1 --2- 172.21.15.46:0/2105538449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf68157860 0x7fbf680cc600 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbf500098e0 tx=0x7fbf500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a8d281e688271b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:13.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.374+0000 7fbf65d9b700 1 -- 172.21.15.46:0/2105538449 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50009e80 con 0x7fbf68157860 2024-01-26T23:30:13.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.374+0000 7fbf65d9b700 1 -- 172.21.15.46:0/2105538449 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5000b860 con 0x7fbf68157860 2024-01-26T23:30:13.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.374+0000 7fbf65d9b700 1 -- 172.21.15.46:0/2105538449 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5000bbe0 con 0x7fbf68157860 2024-01-26T23:30:13.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.374+0000 7fbf6ca90700 1 -- 172.21.15.46:0/2105538449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf68157860 msgr2=0x7fbf680cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.374+0000 7fbf6ca90700 1 --2- 172.21.15.46:0/2105538449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf68157860 0x7fbf680cc600 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbf500098e0 tx=0x7fbf500091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.374+0000 7fbf6ca90700 1 -- 172.21.15.46:0/2105538449 shutdown_connections 2024-01-26T23:30:13.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.374+0000 7fbf6ca90700 1 --2- 172.21.15.46:0/2105538449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf68157860 0x7fbf680cc600 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.374+0000 7fbf6ca90700 1 -- 172.21.15.46:0/2105538449 >> 172.21.15.46:0/2105538449 conn(0x7fbf68067930 msgr2=0x7fbf68067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:13.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.374+0000 7fbf6ca90700 1 -- 172.21.15.46:0/2105538449 shutdown_connections 2024-01-26T23:30:13.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.374+0000 7fbf6ca90700 1 -- 172.21.15.46:0/2105538449 wait complete. 2024-01-26T23:30:13.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.376+0000 7fbf6ca90700 1 Processor -- start 2024-01-26T23:30:13.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.376+0000 7fbf6ca90700 1 -- start start 2024-01-26T23:30:13.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf6ca90700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf6806ddb0 0x7fbf680c3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:13.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf6ca90700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf68157490 0x7fbf68167d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf6ca90700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf68157860 0x7fbf68168290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf6ca90700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf68151970 con 0x7fbf68157860 2024-01-26T23:30:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf6ca90700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf680c3890 con 0x7fbf68157490 2024-01-26T23:30:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf6ca90700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf680af5c0 con 0x7fbf6806ddb0 2024-01-26T23:30:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf66d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf68157860 0x7fbf68168290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf66d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf68157860 0x7fbf68168290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40376/0 (socket says 172.21.15.46:40376) 2024-01-26T23:30:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf66d9d700 1 -- 172.21.15.46:0/1276941703 learned_addr learned my addr 172.21.15.46:0/1276941703 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf6659c700 1 --2- 172.21.15.46:0/1276941703 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf6806ddb0 0x7fbf680c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf5dd9b700 1 --2- 172.21.15.46:0/1276941703 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf68157490 0x7fbf68167d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf66d9d700 1 -- 172.21.15.46:0/1276941703 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf6806ddb0 msgr2=0x7fbf680c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf66d9d700 1 --2- 172.21.15.46:0/1276941703 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf6806ddb0 0x7fbf680c3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf66d9d700 1 -- 172.21.15.46:0/1276941703 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf68157490 msgr2=0x7fbf68167d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf66d9d700 1 --2- 172.21.15.46:0/1276941703 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf68157490 0x7fbf68167d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf66d9d700 1 -- 172.21.15.46:0/1276941703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf50009420 con 0x7fbf68157860 2024-01-26T23:30:13.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf66d9d700 1 --2- 172.21.15.46:0/1276941703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf68157860 0x7fbf68168290 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbf6000ccc0 tx=0x7fbf60001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:13.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf5effd700 1 -- 172.21.15.46:0/1276941703 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6000ab10 con 0x7fbf68157860 2024-01-26T23:30:13.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.377+0000 7fbf5effd700 1 -- 172.21.15.46:0/1276941703 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6000ac70 con 0x7fbf68157860 2024-01-26T23:30:13.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.378+0000 7fbf6ca90700 1 -- 172.21.15.46:0/1276941703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf680af8a0 con 0x7fbf68157860 2024-01-26T23:30:13.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.378+0000 7fbf5effd700 1 -- 172.21.15.46:0/1276941703 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf600070f0 con 0x7fbf68157860 2024-01-26T23:30:13.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.378+0000 7fbf6ca90700 1 -- 172.21.15.46:0/1276941703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf680afe60 con 0x7fbf68157860 2024-01-26T23:30:13.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.378+0000 7fbf6ca90700 1 -- 172.21.15.46:0/1276941703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf6804c710 con 0x7fbf68157860 2024-01-26T23:30:13.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.379+0000 7fbf5effd700 1 -- 172.21.15.46:0/1276941703 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbf60007520 con 0x7fbf68157860 2024-01-26T23:30:13.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.379+0000 7fbf5effd700 1 --2- 172.21.15.46:0/1276941703 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbf48066f30 0x7fbf480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:13.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.380+0000 7fbf5effd700 1 -- 172.21.15.46:0/1276941703 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fbf60084ce0 con 0x7fbf68157860 2024-01-26T23:30:13.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.380+0000 7fbf6659c700 1 --2- 172.21.15.46:0/1276941703 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbf48066f30 0x7fbf480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:13.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.380+0000 7fbf6659c700 1 --2- 172.21.15.46:0/1276941703 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbf48066f30 0x7fbf480693e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbf58003860 tx=0x7fbf58009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:13.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.384+0000 7fbf5effd700 1 -- 172.21.15.46:0/1276941703 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbf60051ce0 con 0x7fbf68157860 2024-01-26T23:30:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.586+0000 7fbf6ca90700 1 -- 172.21.15.46:0/1276941703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbf680cac50 con 0x7fbf68157860 2024-01-26T23:30:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.587+0000 7fbf5effd700 1 -- 172.21.15.46:0/1276941703 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbf60055300 con 0x7fbf68157860 2024-01-26T23:30:13.587 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:30:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.589+0000 7fbf6ca90700 1 -- 172.21.15.46:0/1276941703 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbf48066f30 msgr2=0x7fbf480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.589+0000 7fbf6ca90700 1 --2- 172.21.15.46:0/1276941703 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbf48066f30 0x7fbf480693e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbf58003860 tx=0x7fbf58009040 comp rx=0 tx=0).stop 2024-01-26T23:30:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.589+0000 7fbf6ca90700 1 -- 172.21.15.46:0/1276941703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf68157860 msgr2=0x7fbf68168290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.589+0000 7fbf6ca90700 1 --2- 172.21.15.46:0/1276941703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf68157860 0x7fbf68168290 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbf6000ccc0 tx=0x7fbf60001a20 comp rx=0 tx=0).stop 2024-01-26T23:30:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.589+0000 7fbf6ca90700 1 -- 172.21.15.46:0/1276941703 shutdown_connections 2024-01-26T23:30:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.589+0000 7fbf6ca90700 1 --2- 172.21.15.46:0/1276941703 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbf48066f30 0x7fbf480693e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.589+0000 7fbf6ca90700 1 --2- 172.21.15.46:0/1276941703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf68157860 0x7fbf68168290 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.589+0000 7fbf6ca90700 1 --2- 172.21.15.46:0/1276941703 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf6806ddb0 0x7fbf680c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.589+0000 7fbf6ca90700 1 --2- 172.21.15.46:0/1276941703 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf68157490 0x7fbf68167d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.589+0000 7fbf6ca90700 1 -- 172.21.15.46:0/1276941703 >> 172.21.15.46:0/1276941703 conn(0x7fbf68067930 msgr2=0x7fbf68153170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.589+0000 7fbf6ca90700 1 -- 172.21.15.46:0/1276941703 shutdown_connections 2024-01-26T23:30:13.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.589+0000 7fbf6ca90700 1 -- 172.21.15.46:0/1276941703 wait complete. 2024-01-26T23:30:13.603 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-26T23:30:13.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.737+0000 7f49d4833700 1 Processor -- start 2024-01-26T23:30:13.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.737+0000 7f49d4833700 1 -- start start 2024-01-26T23:30:13.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.738+0000 7f49d4833700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49d0068b00 0x7f49d0068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:13.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.738+0000 7f49d4833700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f49d006add0 0x7f49d006b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:13.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.738+0000 7f49d4833700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d00a9a80 con 0x7f49d006a660 2024-01-26T23:30:13.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.738+0000 7f49d4833700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d006b7b0 con 0x7f49d006add0 2024-01-26T23:30:13.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.738+0000 7f49d4833700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d015a680 con 0x7f49d0068b00 2024-01-26T23:30:13.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.739+0000 7f49cdd9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f49d006a660 0x7f49d0159ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50460/0 (socket says 172.21.15.46:50460) 2024-01-26T23:30:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.739+0000 7f49cdd9b700 1 -- 172.21.15.46:0/4250092018 learned_addr learned my addr 172.21.15.46:0/4250092018 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.739+0000 7f49ce59c700 1 --2- 172.21.15.46:0/4250092018 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49d0068b00 0x7f49d0068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.739+0000 7f49ced9d700 1 --2- 172.21.15.46:0/4250092018 >> v2:172.21.15.106:3300/0 conn(0x7f49d006add0 0x7f49d006b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.739+0000 7f49cd59a700 1 -- 172.21.15.46:0/4250092018 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997351137 0 0) 0x7f49d00a9a80 con 0x7f49d006a660 2024-01-26T23:30:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.739+0000 7f49cd59a700 1 -- 172.21.15.46:0/4250092018 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49bc003570 con 0x7f49d006a660 2024-01-26T23:30:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.739+0000 7f49ce59c700 1 -- 172.21.15.46:0/4250092018 >> v2:172.21.15.106:3300/0 conn(0x7f49d006add0 msgr2=0x7f49d006b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.739+0000 7f49ce59c700 1 --2- 172.21.15.46:0/4250092018 >> v2:172.21.15.106:3300/0 conn(0x7f49d006add0 0x7f49d006b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.739+0000 7f49ce59c700 1 -- 172.21.15.46:0/4250092018 >> v1:172.21.15.46:6789/0 conn(0x7f49d006a660 legacy=0x7f49d0159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.739+0000 7f49ce59c700 1 -- 172.21.15.46:0/4250092018 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d006aad0 con 0x7f49d0068b00 2024-01-26T23:30:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.739+0000 7f49ced9d700 1 --2- 172.21.15.46:0/4250092018 >> v2:172.21.15.106:3300/0 conn(0x7f49d006add0 0x7f49d006b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.739+0000 7f49ce59c700 1 --2- 172.21.15.46:0/4250092018 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49d0068b00 0x7f49d0068ed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f49c00098d0 tx=0x7f49c0009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a98550ba38f1cff server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.740+0000 7f49cd59a700 1 -- 172.21.15.46:0/4250092018 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0005c10 con 0x7f49d0068b00 2024-01-26T23:30:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.740+0000 7f49cd59a700 1 -- 172.21.15.46:0/4250092018 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c0005d70 con 0x7f49d0068b00 2024-01-26T23:30:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.740+0000 7f49cd59a700 1 -- 172.21.15.46:0/4250092018 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0006280 con 0x7f49d0068b00 2024-01-26T23:30:13.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.740+0000 7f49d4833700 1 -- 172.21.15.46:0/4250092018 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49d0068b00 msgr2=0x7f49d0068ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.740+0000 7f49d4833700 1 --2- 172.21.15.46:0/4250092018 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49d0068b00 0x7f49d0068ed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f49c00098d0 tx=0x7f49c0009120 comp rx=0 tx=0).stop 2024-01-26T23:30:13.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.740+0000 7f49d4833700 1 -- 172.21.15.46:0/4250092018 shutdown_connections 2024-01-26T23:30:13.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.740+0000 7f49d4833700 1 --2- 172.21.15.46:0/4250092018 >> v2:172.21.15.106:3300/0 conn(0x7f49d006add0 0x7f49d006b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.740+0000 7f49d4833700 1 --2- 172.21.15.46:0/4250092018 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49d0068b00 0x7f49d0068ed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.740+0000 7f49d4833700 1 -- 172.21.15.46:0/4250092018 >> 172.21.15.46:0/4250092018 conn(0x7f49d00ca100 msgr2=0x7f49d006e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:13.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.740+0000 7f49d4833700 1 -- 172.21.15.46:0/4250092018 shutdown_connections 2024-01-26T23:30:13.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.740+0000 7f49d4833700 1 -- 172.21.15.46:0/4250092018 wait complete. 2024-01-26T23:30:13.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.741+0000 7f49d4833700 1 Processor -- start 2024-01-26T23:30:13.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.742+0000 7f49d4833700 1 -- start start 2024-01-26T23:30:13.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.742+0000 7f49d4833700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d0068b00 0x7f49d00c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.742+0000 7f49d4833700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49d006a660 0x7f49d00aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.742+0000 7f49d4833700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49d006add0 0x7f49d00ac300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.742+0000 7f49d4833700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d0157730 con 0x7f49d0068b00 2024-01-26T23:30:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.742+0000 7f49d4833700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d00b1310 con 0x7f49d006a660 2024-01-26T23:30:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.742+0000 7f49d4833700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d0157190 con 0x7f49d006add0 2024-01-26T23:30:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.742+0000 7f49ce59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d0068b00 0x7f49d00c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.742+0000 7f49ce59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d0068b00 0x7f49d00c3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40390/0 (socket says 172.21.15.46:40390) 2024-01-26T23:30:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.742+0000 7f49ce59c700 1 -- 172.21.15.46:0/3285099817 learned_addr learned my addr 172.21.15.46:0/3285099817 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.743+0000 7f49ced9d700 1 --2- 172.21.15.46:0/3285099817 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49d006add0 0x7f49d00ac300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.743+0000 7f49cdd9b700 1 --2- 172.21.15.46:0/3285099817 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49d006a660 0x7f49d00aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.743+0000 7f49ce59c700 1 -- 172.21.15.46:0/3285099817 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49d006add0 msgr2=0x7f49d00ac300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.743+0000 7f49ce59c700 1 --2- 172.21.15.46:0/3285099817 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49d006add0 0x7f49d00ac300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.743+0000 7f49ce59c700 1 -- 172.21.15.46:0/3285099817 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49d006a660 msgr2=0x7f49d00aeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.743+0000 7f49ce59c700 1 --2- 172.21.15.46:0/3285099817 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49d006a660 0x7f49d00aeae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.743+0000 7f49ce59c700 1 -- 172.21.15.46:0/3285099817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c8009a20 con 0x7f49d0068b00 2024-01-26T23:30:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.743+0000 7f49cdd9b700 1 --2- 172.21.15.46:0/3285099817 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49d006a660 0x7f49d00aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.743+0000 7f49ce59c700 1 --2- 172.21.15.46:0/3285099817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d0068b00 0x7f49d00c3bd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f49c0006360 tx=0x7f49c0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.743+0000 7f49c67fc700 1 -- 172.21.15.46:0/3285099817 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c001d980 con 0x7f49d0068b00 2024-01-26T23:30:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.743+0000 7f49c67fc700 1 -- 172.21.15.46:0/3285099817 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c001dae0 con 0x7f49d0068b00 2024-01-26T23:30:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.743+0000 7f49c67fc700 1 -- 172.21.15.46:0/3285099817 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c001dea0 con 0x7f49d0068b00 2024-01-26T23:30:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.743+0000 7f49d4833700 1 -- 172.21.15.46:0/3285099817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c00093c0 con 0x7f49d0068b00 2024-01-26T23:30:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.743+0000 7f49d4833700 1 -- 172.21.15.46:0/3285099817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d00b09e0 con 0x7f49d0068b00 2024-01-26T23:30:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.744+0000 7f49d4833700 1 -- 172.21.15.46:0/3285099817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d004c710 con 0x7f49d0068b00 2024-01-26T23:30:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.745+0000 7f49c67fc700 1 -- 172.21.15.46:0/3285099817 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f49c001b690 con 0x7f49d0068b00 2024-01-26T23:30:13.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.745+0000 7f49c67fc700 1 --2- 172.21.15.46:0/3285099817 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f49b0066f30 0x7f49b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:13.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.745+0000 7f49c67fc700 1 -- 172.21.15.46:0/3285099817 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f49c0086470 con 0x7f49d0068b00 2024-01-26T23:30:13.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.745+0000 7f49cdd9b700 1 --2- 172.21.15.46:0/3285099817 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f49b0066f30 0x7f49b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:13.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.746+0000 7f49cdd9b700 1 --2- 172.21.15.46:0/3285099817 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f49b0066f30 0x7f49b00693e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f49d0157990 tx=0x7f49b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:13.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.750+0000 7f49c67fc700 1 -- 172.21.15.46:0/3285099817 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f49c00534b0 con 0x7f49d0068b00 2024-01-26T23:30:13.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.953+0000 7f49d4833700 1 -- 172.21.15.46:0/3285099817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f49d006edf0 con 0x7f49d0068b00 2024-01-26T23:30:13.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.953+0000 7f49c67fc700 1 -- 172.21.15.46:0/3285099817 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f49c002a020 con 0x7f49d0068b00 2024-01-26T23:30:13.954 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:30:13.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.955+0000 7f49d4833700 1 -- 172.21.15.46:0/3285099817 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f49b0066f30 msgr2=0x7f49b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.955+0000 7f49d4833700 1 --2- 172.21.15.46:0/3285099817 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f49b0066f30 0x7f49b00693e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f49d0157990 tx=0x7f49b8009040 comp rx=0 tx=0).stop 2024-01-26T23:30:13.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.955+0000 7f49d4833700 1 -- 172.21.15.46:0/3285099817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d0068b00 msgr2=0x7f49d00c3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:13.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.955+0000 7f49d4833700 1 --2- 172.21.15.46:0/3285099817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d0068b00 0x7f49d00c3bd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f49c0006360 tx=0x7f49c0016000 comp rx=0 tx=0).stop 2024-01-26T23:30:13.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.955+0000 7f49d4833700 1 -- 172.21.15.46:0/3285099817 shutdown_connections 2024-01-26T23:30:13.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.955+0000 7f49d4833700 1 --2- 172.21.15.46:0/3285099817 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f49b0066f30 0x7f49b00693e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.955+0000 7f49d4833700 1 --2- 172.21.15.46:0/3285099817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49d0068b00 0x7f49d00c3bd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.955+0000 7f49d4833700 1 --2- 172.21.15.46:0/3285099817 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49d006add0 0x7f49d00ac300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.955+0000 7f49d4833700 1 --2- 172.21.15.46:0/3285099817 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49d006a660 0x7f49d00aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:13.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.955+0000 7f49d4833700 1 -- 172.21.15.46:0/3285099817 >> 172.21.15.46:0/3285099817 conn(0x7f49d00ca100 msgr2=0x7f49d00cbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:13.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.956+0000 7f49d4833700 1 -- 172.21.15.46:0/3285099817 shutdown_connections 2024-01-26T23:30:13.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:13.956+0000 7f49d4833700 1 -- 172.21.15.46:0/3285099817 wait complete. 2024-01-26T23:30:13.970 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-26T23:30:14.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.105+0000 7fc474fab700 1 Processor -- start 2024-01-26T23:30:14.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.105+0000 7fc474fab700 1 -- start start 2024-01-26T23:30:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.106+0000 7fc474fab700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc470153c40 0x7fc47006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.106+0000 7fc474fab700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47015ad60 con 0x7fc470154010 2024-01-26T23:30:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.106+0000 7fc474fab700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47015a150 con 0x7fc4700b1e80 2024-01-26T23:30:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.106+0000 7fc474fab700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4700b1d20 con 0x7fc470153c40 2024-01-26T23:30:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.107+0000 7fc46dd9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc470154010 0x7fc47006dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50474/0 (socket says 172.21.15.46:50474) 2024-01-26T23:30:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.107+0000 7fc46dd9b700 1 -- 172.21.15.46:0/343309372 learned_addr learned my addr 172.21.15.46:0/343309372 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:14.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.107+0000 7fc46e59c700 1 --2- 172.21.15.46:0/343309372 >> v2:172.21.15.106:3300/0 conn(0x7fc470153c40 0x7fc47006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.107+0000 7fc46d59a700 1 -- 172.21.15.46:0/343309372 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516324603 0 0) 0x7fc47015ad60 con 0x7fc470154010 2024-01-26T23:30:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.107+0000 7fc46d59a700 1 -- 172.21.15.46:0/343309372 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc45c003570 con 0x7fc470154010 2024-01-26T23:30:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.107+0000 7fc46e59c700 1 -- 172.21.15.46:0/343309372 >> v1:172.21.15.106:6789/0 conn(0x7fc4700b1e80 legacy=0x7fc4700692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.107+0000 7fc46e59c700 1 -- 172.21.15.46:0/343309372 >> v1:172.21.15.46:6789/0 conn(0x7fc470154010 legacy=0x7fc47006dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.107+0000 7fc46e59c700 1 -- 172.21.15.46:0/343309372 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47015a430 con 0x7fc470153c40 2024-01-26T23:30:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.107+0000 7fc46e59c700 1 --2- 172.21.15.46:0/343309372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc470153c40 0x7fc47006d6a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc4600098e0 tx=0x7fc4600091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10961795197462bc server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.107+0000 7fc46d59a700 1 -- 172.21.15.46:0/343309372 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46000b640 con 0x7fc470153c40 2024-01-26T23:30:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.108+0000 7fc46d59a700 1 -- 172.21.15.46:0/343309372 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46000b7a0 con 0x7fc470153c40 2024-01-26T23:30:14.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.108+0000 7fc46d59a700 1 -- 172.21.15.46:0/343309372 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46000bb40 con 0x7fc470153c40 2024-01-26T23:30:14.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.108+0000 7fc474fab700 1 -- 172.21.15.46:0/343309372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc470153c40 msgr2=0x7fc47006d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:14.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.108+0000 7fc474fab700 1 --2- 172.21.15.46:0/343309372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc470153c40 0x7fc47006d6a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc4600098e0 tx=0x7fc4600091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.108+0000 7fc474fab700 1 -- 172.21.15.46:0/343309372 shutdown_connections 2024-01-26T23:30:14.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.108+0000 7fc474fab700 1 --2- 172.21.15.46:0/343309372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc470153c40 0x7fc47006d6a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.108+0000 7fc474fab700 1 -- 172.21.15.46:0/343309372 >> 172.21.15.46:0/343309372 conn(0x7fc470067930 msgr2=0x7fc470067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:14.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.108+0000 7fc474fab700 1 -- 172.21.15.46:0/343309372 shutdown_connections 2024-01-26T23:30:14.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.108+0000 7fc474fab700 1 -- 172.21.15.46:0/343309372 wait complete. 2024-01-26T23:30:14.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.109+0000 7fc474fab700 1 Processor -- start 2024-01-26T23:30:14.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.109+0000 7fc474fab700 1 -- start start 2024-01-26T23:30:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.110+0000 7fc474fab700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4700b1e80 0x7fc4700ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.110+0000 7fc474fab700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc470153c40 0x7fc4700c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.110+0000 7fc474fab700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc470154010 0x7fc470150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.110+0000 7fc474fab700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4700c8660 con 0x7fc470153c40 2024-01-26T23:30:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.110+0000 7fc474fab700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc470154460 con 0x7fc470154010 2024-01-26T23:30:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.110+0000 7fc474fab700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4701511f0 con 0x7fc4700b1e80 2024-01-26T23:30:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.110+0000 7fc46dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc470153c40 0x7fc4700c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.110+0000 7fc46dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc470153c40 0x7fc4700c8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40392/0 (socket says 172.21.15.46:40392) 2024-01-26T23:30:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.110+0000 7fc46dd9b700 1 -- 172.21.15.46:0/185931925 learned_addr learned my addr 172.21.15.46:0/185931925 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.110+0000 7fc46e59c700 1 --2- 172.21.15.46:0/185931925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4700b1e80 0x7fc4700ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:14.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.110+0000 7fc46ed9d700 1 --2- 172.21.15.46:0/185931925 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc470154010 0x7fc470150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.111+0000 7fc46dd9b700 1 -- 172.21.15.46:0/185931925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4700b1e80 msgr2=0x7fc4700ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.111+0000 7fc46dd9b700 1 --2- 172.21.15.46:0/185931925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4700b1e80 0x7fc4700ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.111+0000 7fc46dd9b700 1 -- 172.21.15.46:0/185931925 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc470154010 msgr2=0x7fc470150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.111+0000 7fc46dd9b700 1 --2- 172.21.15.46:0/185931925 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc470154010 0x7fc470150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.111+0000 7fc46dd9b700 1 -- 172.21.15.46:0/185931925 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc460009420 con 0x7fc470153c40 2024-01-26T23:30:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.111+0000 7fc46dd9b700 1 --2- 172.21.15.46:0/185931925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc470153c40 0x7fc4700c8120 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc45800c7c0 tx=0x7fc458007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.111+0000 7fc4667fc700 1 -- 172.21.15.46:0/185931925 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458005cd0 con 0x7fc470153c40 2024-01-26T23:30:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.111+0000 7fc4667fc700 1 -- 172.21.15.46:0/185931925 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc458005e30 con 0x7fc470153c40 2024-01-26T23:30:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.111+0000 7fc474fab700 1 -- 172.21.15.46:0/185931925 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4700c8850 con 0x7fc470153c40 2024-01-26T23:30:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.111+0000 7fc4667fc700 1 -- 172.21.15.46:0/185931925 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458007320 con 0x7fc470153c40 2024-01-26T23:30:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.111+0000 7fc474fab700 1 -- 172.21.15.46:0/185931925 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4700c8e10 con 0x7fc470153c40 2024-01-26T23:30:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.112+0000 7fc474fab700 1 -- 172.21.15.46:0/185931925 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47004c710 con 0x7fc470153c40 2024-01-26T23:30:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.113+0000 7fc4667fc700 1 -- 172.21.15.46:0/185931925 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc45801b3b0 con 0x7fc470153c40 2024-01-26T23:30:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.113+0000 7fc4667fc700 1 --2- 172.21.15.46:0/185931925 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc450066f60 0x7fc450069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.113+0000 7fc4667fc700 1 -- 172.21.15.46:0/185931925 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fc458083e90 con 0x7fc470153c40 2024-01-26T23:30:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.113+0000 7fc46e59c700 1 --2- 172.21.15.46:0/185931925 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc450066f60 0x7fc450069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.114+0000 7fc46e59c700 1 --2- 172.21.15.46:0/185931925 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc450066f60 0x7fc450069410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc4600093f0 tx=0x7fc460017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.118+0000 7fc4667fc700 1 -- 172.21.15.46:0/185931925 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc458050e10 con 0x7fc470153c40 2024-01-26T23:30:14.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.323+0000 7fc474fab700 1 -- 172.21.15.46:0/185931925 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc47015a710 con 0x7fc470153c40 2024-01-26T23:30:14.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.324+0000 7fc4667fc700 1 -- 172.21.15.46:0/185931925 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc458017070 con 0x7fc470153c40 2024-01-26T23:30:14.325 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-26T23:30:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.326+0000 7fc474fab700 1 -- 172.21.15.46:0/185931925 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc450066f60 msgr2=0x7fc450069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.326+0000 7fc474fab700 1 --2- 172.21.15.46:0/185931925 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc450066f60 0x7fc450069410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc4600093f0 tx=0x7fc460017000 comp rx=0 tx=0).stop 2024-01-26T23:30:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.326+0000 7fc474fab700 1 -- 172.21.15.46:0/185931925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc470153c40 msgr2=0x7fc4700c8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.326+0000 7fc474fab700 1 --2- 172.21.15.46:0/185931925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc470153c40 0x7fc4700c8120 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc45800c7c0 tx=0x7fc458007e00 comp rx=0 tx=0).stop 2024-01-26T23:30:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.326+0000 7fc474fab700 1 -- 172.21.15.46:0/185931925 shutdown_connections 2024-01-26T23:30:14.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.326+0000 7fc474fab700 1 --2- 172.21.15.46:0/185931925 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc450066f60 0x7fc450069410 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.326+0000 7fc474fab700 1 --2- 172.21.15.46:0/185931925 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc470154010 0x7fc470150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.326+0000 7fc474fab700 1 --2- 172.21.15.46:0/185931925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4700b1e80 0x7fc4700ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.326+0000 7fc474fab700 1 --2- 172.21.15.46:0/185931925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc470153c40 0x7fc4700c8120 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.326+0000 7fc474fab700 1 -- 172.21.15.46:0/185931925 >> 172.21.15.46:0/185931925 conn(0x7fc470067930 msgr2=0x7fc4700b0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:14.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.326+0000 7fc474fab700 1 -- 172.21.15.46:0/185931925 shutdown_connections 2024-01-26T23:30:14.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.326+0000 7fc474fab700 1 -- 172.21.15.46:0/185931925 wait complete. 2024-01-26T23:30:14.340 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-01-26T23:30:14.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.474+0000 7fcd364c9700 1 Processor -- start 2024-01-26T23:30:14.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.474+0000 7fcd364c9700 1 -- start start 2024-01-26T23:30:14.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.475+0000 7fcd364c9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcd30067ec0 0x7fcd300c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:14.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.475+0000 7fcd364c9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcd300c7fe0 0x7fcd30157200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:14.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.475+0000 7fcd364c9700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3006c160 con 0x7fcd30067af0 2024-01-26T23:30:14.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.475+0000 7fcd364c9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd30068330 con 0x7fcd30067ec0 2024-01-26T23:30:14.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.475+0000 7fcd364c9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3006ca00 con 0x7fcd300c7fe0 2024-01-26T23:30:14.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.475+0000 7fcd2f7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcd30067ec0 0x7fcd300c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:14.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.475+0000 7fcd2f7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcd30067ec0 0x7fcd300c7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40394/0 (socket says 172.21.15.46:40394) 2024-01-26T23:30:14.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.475+0000 7fcd2f7fe700 1 -- 172.21.15.46:0/3329866439 learned_addr learned my addr 172.21.15.46:0/3329866439 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:14.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.475+0000 7fcd34a66700 1 --2- 172.21.15.46:0/3329866439 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcd300c7fe0 0x7fcd30157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:14.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.475+0000 7fcd2f7fe700 1 -- 172.21.15.46:0/3329866439 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcd300c7fe0 msgr2=0x7fcd30157200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:14.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.475+0000 7fcd2f7fe700 1 --2- 172.21.15.46:0/3329866439 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcd300c7fe0 0x7fcd30157200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.475+0000 7fcd2f7fe700 1 -- 172.21.15.46:0/3329866439 >> v1:172.21.15.106:6789/0 conn(0x7fcd30067af0 legacy=0x7fcd3006ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:14.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.475+0000 7fcd2f7fe700 1 -- 172.21.15.46:0/3329866439 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd300c84f0 con 0x7fcd30067ec0 2024-01-26T23:30:14.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.476+0000 7fcd2f7fe700 1 --2- 172.21.15.46:0/3329866439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd30067ec0 0x7fcd300c7aa0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcd240099b0 tx=0x7fcd24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47add89fae1b8cba server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:14.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.476+0000 7fcd2effd700 1 -- 172.21.15.46:0/3329866439 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd24009ea0 con 0x7fcd30067ec0 2024-01-26T23:30:14.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.476+0000 7fcd2effd700 1 -- 172.21.15.46:0/3329866439 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2400b860 con 0x7fcd30067ec0 2024-01-26T23:30:14.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.476+0000 7fcd2effd700 1 -- 172.21.15.46:0/3329866439 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2400bc00 con 0x7fcd30067ec0 2024-01-26T23:30:14.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.476+0000 7fcd364c9700 1 -- 172.21.15.46:0/3329866439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd30067ec0 msgr2=0x7fcd300c7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:14.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.476+0000 7fcd364c9700 1 --2- 172.21.15.46:0/3329866439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd30067ec0 0x7fcd300c7aa0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcd240099b0 tx=0x7fcd24009250 comp rx=0 tx=0).stop 2024-01-26T23:30:14.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.476+0000 7fcd364c9700 1 -- 172.21.15.46:0/3329866439 shutdown_connections 2024-01-26T23:30:14.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.476+0000 7fcd364c9700 1 --2- 172.21.15.46:0/3329866439 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcd300c7fe0 0x7fcd30157200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.476+0000 7fcd364c9700 1 --2- 172.21.15.46:0/3329866439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd30067ec0 0x7fcd300c7aa0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.476+0000 7fcd364c9700 1 -- 172.21.15.46:0/3329866439 >> 172.21.15.46:0/3329866439 conn(0x7fcd300ca9a0 msgr2=0x7fcd30068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:14.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.476+0000 7fcd364c9700 1 -- 172.21.15.46:0/3329866439 shutdown_connections 2024-01-26T23:30:14.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.476+0000 7fcd364c9700 1 -- 172.21.15.46:0/3329866439 wait complete. 2024-01-26T23:30:14.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.478+0000 7fcd364c9700 1 Processor -- start 2024-01-26T23:30:14.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.478+0000 7fcd364c9700 1 -- start start 2024-01-26T23:30:14.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.479+0000 7fcd364c9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd30067af0 0x7fcd30153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:14.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.479+0000 7fcd364c9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd30067ec0 0x7fcd300cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:14.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.479+0000 7fcd364c9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcd300c7fe0 0x7fcd3006eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:14.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.479+0000 7fcd364c9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd30169ca0 con 0x7fcd30067af0 2024-01-26T23:30:14.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.479+0000 7fcd364c9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3016ab00 con 0x7fcd30067ec0 2024-01-26T23:30:14.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.479+0000 7fcd364c9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd30153850 con 0x7fcd300c7fe0 2024-01-26T23:30:14.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.479+0000 7fcd2ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd30067af0 0x7fcd30153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:14.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.479+0000 7fcd2ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd30067af0 0x7fcd30153310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40402/0 (socket says 172.21.15.46:40402) 2024-01-26T23:30:14.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.479+0000 7fcd2ffff700 1 -- 172.21.15.46:0/3188157809 learned_addr learned my addr 172.21.15.46:0/3188157809 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:14.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.479+0000 7fcd34a66700 1 --2- 172.21.15.46:0/3188157809 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcd300c7fe0 0x7fcd3006eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:14.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.479+0000 7fcd2f7fe700 1 --2- 172.21.15.46:0/3188157809 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd30067ec0 0x7fcd300cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:14.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.479+0000 7fcd2ffff700 1 -- 172.21.15.46:0/3188157809 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcd300c7fe0 msgr2=0x7fcd3006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:14.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.479+0000 7fcd2ffff700 1 --2- 172.21.15.46:0/3188157809 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcd300c7fe0 0x7fcd3006eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.479+0000 7fcd2ffff700 1 -- 172.21.15.46:0/3188157809 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd30067ec0 msgr2=0x7fcd300cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:14.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.479+0000 7fcd2ffff700 1 --2- 172.21.15.46:0/3188157809 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd30067ec0 0x7fcd300cc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.479+0000 7fcd2ffff700 1 -- 172.21.15.46:0/3188157809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd240094f0 con 0x7fcd30067af0 2024-01-26T23:30:14.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.479+0000 7fcd2ffff700 1 --2- 172.21.15.46:0/3188157809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd30067af0 0x7fcd30153310 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcd2000c5c0 tx=0x7fcd2000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:14.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.480+0000 7fcd17fff700 1 -- 172.21.15.46:0/3188157809 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd20011590 con 0x7fcd30067af0 2024-01-26T23:30:14.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.480+0000 7fcd17fff700 1 -- 172.21.15.46:0/3188157809 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd20011710 con 0x7fcd30067af0 2024-01-26T23:30:14.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.480+0000 7fcd17fff700 1 -- 172.21.15.46:0/3188157809 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd20011ad0 con 0x7fcd30067af0 2024-01-26T23:30:14.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.480+0000 7fcd364c9700 1 -- 172.21.15.46:0/3188157809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd30169ef0 con 0x7fcd30067af0 2024-01-26T23:30:14.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.480+0000 7fcd364c9700 1 -- 172.21.15.46:0/3188157809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd3016a4b0 con 0x7fcd30067af0 2024-01-26T23:30:14.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.481+0000 7fcd364c9700 1 -- 172.21.15.46:0/3188157809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd3006d330 con 0x7fcd30067af0 2024-01-26T23:30:14.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.481+0000 7fcd17fff700 1 -- 172.21.15.46:0/3188157809 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fcd2001b3b0 con 0x7fcd30067af0 2024-01-26T23:30:14.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.482+0000 7fcd17fff700 1 --2- 172.21.15.46:0/3188157809 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcd1c066f30 0x7fcd1c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:14.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.482+0000 7fcd17fff700 1 -- 172.21.15.46:0/3188157809 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fcd20083d40 con 0x7fcd30067af0 2024-01-26T23:30:14.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.482+0000 7fcd2f7fe700 1 --2- 172.21.15.46:0/3188157809 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcd1c066f30 0x7fcd1c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:14.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.482+0000 7fcd2f7fe700 1 --2- 172.21.15.46:0/3188157809 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcd1c066f30 0x7fcd1c0693e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcd240099b0 tx=0x7fcd24017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:14.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.486+0000 7fcd17fff700 1 -- 172.21.15.46:0/3188157809 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcd20050c20 con 0x7fcd30067af0 2024-01-26T23:30:14.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.690+0000 7fcd364c9700 1 -- 172.21.15.46:0/3188157809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fcd3004c710 con 0x7fcd30067af0 2024-01-26T23:30:14.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.691+0000 7fcd17fff700 1 -- 172.21.15.46:0/3188157809 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7fcd20017070 con 0x7fcd30067af0 2024-01-26T23:30:14.692 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:30:14.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.693+0000 7fcd364c9700 1 -- 172.21.15.46:0/3188157809 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcd1c066f30 msgr2=0x7fcd1c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:14.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.693+0000 7fcd364c9700 1 --2- 172.21.15.46:0/3188157809 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcd1c066f30 0x7fcd1c0693e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcd240099b0 tx=0x7fcd24017000 comp rx=0 tx=0).stop 2024-01-26T23:30:14.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.693+0000 7fcd364c9700 1 -- 172.21.15.46:0/3188157809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd30067af0 msgr2=0x7fcd30153310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:14.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.693+0000 7fcd364c9700 1 --2- 172.21.15.46:0/3188157809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd30067af0 0x7fcd30153310 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcd2000c5c0 tx=0x7fcd2000c8d0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.693+0000 7fcd364c9700 1 -- 172.21.15.46:0/3188157809 shutdown_connections 2024-01-26T23:30:14.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.693+0000 7fcd364c9700 1 --2- 172.21.15.46:0/3188157809 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcd1c066f30 0x7fcd1c0693e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.693+0000 7fcd364c9700 1 --2- 172.21.15.46:0/3188157809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd30067af0 0x7fcd30153310 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.693+0000 7fcd364c9700 1 --2- 172.21.15.46:0/3188157809 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcd300c7fe0 0x7fcd3006eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.693+0000 7fcd364c9700 1 --2- 172.21.15.46:0/3188157809 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd30067ec0 0x7fcd300cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.693+0000 7fcd364c9700 1 -- 172.21.15.46:0/3188157809 >> 172.21.15.46:0/3188157809 conn(0x7fcd300ca9a0 msgr2=0x7fcd300b2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:14.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.693+0000 7fcd364c9700 1 -- 172.21.15.46:0/3188157809 shutdown_connections 2024-01-26T23:30:14.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.693+0000 7fcd364c9700 1 -- 172.21.15.46:0/3188157809 wait complete. 2024-01-26T23:30:14.708 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:30:14.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.839+0000 7f0c617f1700 1 Processor -- start 2024-01-26T23:30:14.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.839+0000 7f0c617f1700 1 -- start start 2024-01-26T23:30:14.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.840+0000 7f0c617f1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c5c068620 0x7f0c5c0c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:14.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.840+0000 7f0c617f1700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0c5c0cccd0 0x7f0c5c0c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.840+0000 7f0c617f1700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c5c068190 con 0x7f0c5c0c8dc0 2024-01-26T23:30:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.840+0000 7f0c617f1700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c5c067930 con 0x7f0c5c068620 2024-01-26T23:30:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.840+0000 7f0c617f1700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c5c0c40c0 con 0x7f0c5c0cccd0 2024-01-26T23:30:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.840+0000 7f0c5a7fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c5c068620 0x7f0c5c0c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.840+0000 7f0c5affd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0c5c0c8dc0 0x7f0c5c0cc5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50476/0 (socket says 172.21.15.46:50476) 2024-01-26T23:30:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.840+0000 7f0c5affd700 1 -- 172.21.15.46:0/2907836519 learned_addr learned my addr 172.21.15.46:0/2907836519 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.840+0000 7f0c5b7fe700 1 --2- 172.21.15.46:0/2907836519 >> v2:172.21.15.106:3300/0 conn(0x7f0c5c0cccd0 0x7f0c5c0c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.841+0000 7f0c5a7fc700 1 -- 172.21.15.46:0/2907836519 >> v2:172.21.15.106:3300/0 conn(0x7f0c5c0cccd0 msgr2=0x7f0c5c0c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.841+0000 7f0c5a7fc700 1 --2- 172.21.15.46:0/2907836519 >> v2:172.21.15.106:3300/0 conn(0x7f0c5c0cccd0 0x7f0c5c0c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.841+0000 7f0c5a7fc700 1 -- 172.21.15.46:0/2907836519 >> v1:172.21.15.46:6789/0 conn(0x7f0c5c0c8dc0 legacy=0x7f0c5c0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.841+0000 7f0c5a7fc700 1 -- 172.21.15.46:0/2907836519 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5c067b40 con 0x7f0c5c068620 2024-01-26T23:30:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.841+0000 7f0c59ffb700 1 -- 172.21.15.46:0/2907836519 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806361511 0 0) 0x7f0c5c068190 con 0x7f0c5c0c8dc0 2024-01-26T23:30:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.841+0000 7f0c5b7fe700 1 --2- 172.21.15.46:0/2907836519 >> v2:172.21.15.106:3300/0 conn(0x7f0c5c0cccd0 0x7f0c5c0c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.841+0000 7f0c5a7fc700 1 --2- 172.21.15.46:0/2907836519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c5c068620 0x7f0c5c0c3a00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c06a570 tx=0x7f0c500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3a7466ee7679c7f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.841+0000 7f0c59ffb700 1 -- 172.21.15.46:0/2907836519 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5000b6b0 con 0x7f0c5c068620 2024-01-26T23:30:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.841+0000 7f0c59ffb700 1 -- 172.21.15.46:0/2907836519 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5000b810 con 0x7f0c5c068620 2024-01-26T23:30:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.841+0000 7f0c59ffb700 1 -- 172.21.15.46:0/2907836519 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5000bbb0 con 0x7f0c5c068620 2024-01-26T23:30:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.842+0000 7f0c617f1700 1 -- 172.21.15.46:0/2907836519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c5c068620 msgr2=0x7f0c5c0c3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.842+0000 7f0c617f1700 1 --2- 172.21.15.46:0/2907836519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c5c068620 0x7f0c5c0c3a00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c06a570 tx=0x7f0c500091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.842+0000 7f0c617f1700 1 -- 172.21.15.46:0/2907836519 shutdown_connections 2024-01-26T23:30:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.842+0000 7f0c617f1700 1 --2- 172.21.15.46:0/2907836519 >> v2:172.21.15.106:3300/0 conn(0x7f0c5c0cccd0 0x7f0c5c0c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.842+0000 7f0c617f1700 1 --2- 172.21.15.46:0/2907836519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c5c068620 0x7f0c5c0c3a00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.842+0000 7f0c617f1700 1 -- 172.21.15.46:0/2907836519 >> 172.21.15.46:0/2907836519 conn(0x7f0c5c069220 msgr2=0x7f0c5c0cd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:14.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.842+0000 7f0c617f1700 1 -- 172.21.15.46:0/2907836519 shutdown_connections 2024-01-26T23:30:14.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.842+0000 7f0c617f1700 1 -- 172.21.15.46:0/2907836519 wait complete. 2024-01-26T23:30:14.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.843+0000 7f0c617f1700 1 Processor -- start 2024-01-26T23:30:14.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.843+0000 7f0c617f1700 1 -- start start 2024-01-26T23:30:14.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c617f1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c5c068620 0x7f0c5c16d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:14.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c617f1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c5c0c8dc0 0x7f0c5c0adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:14.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c617f1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c5c0cccd0 0x7f0c5c0c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c617f1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c16df40 con 0x7f0c5c0cccd0 2024-01-26T23:30:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c617f1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c16d6a0 con 0x7f0c5c0c8dc0 2024-01-26T23:30:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c617f1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c16d980 con 0x7f0c5c068620 2024-01-26T23:30:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c5b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c5c0cccd0 0x7f0c5c0c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c5b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c5c0cccd0 0x7f0c5c0c6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40414/0 (socket says 172.21.15.46:40414) 2024-01-26T23:30:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c5b7fe700 1 -- 172.21.15.46:0/3609150929 learned_addr learned my addr 172.21.15.46:0/3609150929 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c5affd700 1 --2- 172.21.15.46:0/3609150929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c5c068620 0x7f0c5c16d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c5a7fc700 1 --2- 172.21.15.46:0/3609150929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c5c0c8dc0 0x7f0c5c0adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c5b7fe700 1 -- 172.21.15.46:0/3609150929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c5c068620 msgr2=0x7f0c5c16d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c5b7fe700 1 --2- 172.21.15.46:0/3609150929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c5c068620 0x7f0c5c16d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c5b7fe700 1 -- 172.21.15.46:0/3609150929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c5c0c8dc0 msgr2=0x7f0c5c0adc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c5b7fe700 1 --2- 172.21.15.46:0/3609150929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c5c0c8dc0 0x7f0c5c0adc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c5b7fe700 1 -- 172.21.15.46:0/3609150929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c50009420 con 0x7f0c5c0cccd0 2024-01-26T23:30:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c5a7fc700 1 --2- 172.21.15.46:0/3609150929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c5c0c8dc0 0x7f0c5c0adc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:30:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.844+0000 7f0c5b7fe700 1 --2- 172.21.15.46:0/3609150929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c5c0cccd0 0x7f0c5c0c6f00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0c5400d0d0 tx=0x7f0c540099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.845+0000 7f0c42ffd700 1 -- 172.21.15.46:0/3609150929 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5400dd70 con 0x7f0c5c0cccd0 2024-01-26T23:30:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.845+0000 7f0c42ffd700 1 -- 172.21.15.46:0/3609150929 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5400f040 con 0x7f0c5c0cccd0 2024-01-26T23:30:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.845+0000 7f0c617f1700 1 -- 172.21.15.46:0/3609150929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c5c16dc60 con 0x7f0c5c0cccd0 2024-01-26T23:30:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.845+0000 7f0c42ffd700 1 -- 172.21.15.46:0/3609150929 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c54015690 con 0x7f0c5c0cccd0 2024-01-26T23:30:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.845+0000 7f0c617f1700 1 -- 172.21.15.46:0/3609150929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c5c1516f0 con 0x7f0c5c0cccd0 2024-01-26T23:30:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.846+0000 7f0c617f1700 1 -- 172.21.15.46:0/3609150929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c5c04c710 con 0x7f0c5c0cccd0 2024-01-26T23:30:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.846+0000 7f0c42ffd700 1 -- 172.21.15.46:0/3609150929 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0c5401e3b0 con 0x7f0c5c0cccd0 2024-01-26T23:30:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.847+0000 7f0c42ffd700 1 --2- 172.21.15.46:0/3609150929 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c48066f30 0x7f0c480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.847+0000 7f0c42ffd700 1 -- 172.21.15.46:0/3609150929 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f0c54084db0 con 0x7f0c5c0cccd0 2024-01-26T23:30:14.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.847+0000 7f0c5affd700 1 --2- 172.21.15.46:0/3609150929 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c48066f30 0x7f0c480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:14.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.848+0000 7f0c5affd700 1 --2- 172.21.15.46:0/3609150929 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c48066f30 0x7f0c480693e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c0017d0 tx=0x7f0c4c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:14.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:14.851+0000 7f0c42ffd700 1 -- 172.21.15.46:0/3609150929 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0c54051d30 con 0x7f0c5c0cccd0 2024-01-26T23:30:15.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.099+0000 7f0c617f1700 1 -- 172.21.15.46:0/3609150929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c5c0b2380 con 0x7f0c5c0cccd0 2024-01-26T23:30:15.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.099+0000 7f0c42ffd700 1 -- 172.21.15.46:0/3609150929 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4593 (secure 0 0 0) 0x7f0c5401a090 con 0x7f0c5c0cccd0 2024-01-26T23:30:15.100 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:15.100 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3409861270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3409861270},{"type":"v1","addr":"172.21.15.46:6833","nonce":3409861270}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3319220815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3319220815},{"type":"v1","addr":"172.21.15.106:6836","nonce":3319220815}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4252,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/529198432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":529198432},{"type":"v1","addr":"172.21.15.106:6837","nonce":529198432}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:29:44.031963+0000","modified":"2024-01-26T23:30:03.652307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/688086235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":688086235},{"type":"v1","addr":"172.21.15.46:6835","nonce":688086235}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2024-01-26T23:30:15.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.101+0000 7f0c617f1700 1 -- 172.21.15.46:0/3609150929 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c48066f30 msgr2=0x7f0c480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.101+0000 7f0c617f1700 1 --2- 172.21.15.46:0/3609150929 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c48066f30 0x7f0c480693e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c0017d0 tx=0x7f0c4c009040 comp rx=0 tx=0).stop 2024-01-26T23:30:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.101+0000 7f0c617f1700 1 -- 172.21.15.46:0/3609150929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c5c0cccd0 msgr2=0x7f0c5c0c6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.101+0000 7f0c617f1700 1 --2- 172.21.15.46:0/3609150929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c5c0cccd0 0x7f0c5c0c6f00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0c5400d0d0 tx=0x7f0c540099f0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.101+0000 7f0c617f1700 1 -- 172.21.15.46:0/3609150929 shutdown_connections 2024-01-26T23:30:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.101+0000 7f0c617f1700 1 --2- 172.21.15.46:0/3609150929 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c48066f30 0x7f0c480693e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.101+0000 7f0c617f1700 1 --2- 172.21.15.46:0/3609150929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c5c0cccd0 0x7f0c5c0c6f00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.101+0000 7f0c617f1700 1 --2- 172.21.15.46:0/3609150929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c5c068620 0x7f0c5c16d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.101+0000 7f0c617f1700 1 --2- 172.21.15.46:0/3609150929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c5c0c8dc0 0x7f0c5c0adc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.101+0000 7f0c617f1700 1 -- 172.21.15.46:0/3609150929 >> 172.21.15.46:0/3609150929 conn(0x7f0c5c069220 msgr2=0x7f0c5c0ac060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.102+0000 7f0c617f1700 1 -- 172.21.15.46:0/3609150929 shutdown_connections 2024-01-26T23:30:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.102+0000 7f0c617f1700 1 -- 172.21.15.46:0/3609150929 wait complete. 2024-01-26T23:30:15.104 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 9 2024-01-26T23:30:15.116 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:15.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.248+0000 7f50a87ca700 1 Processor -- start 2024-01-26T23:30:15.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.248+0000 7f50a87ca700 1 -- start start 2024-01-26T23:30:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.249+0000 7f50a87ca700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f50a0159770 0x7f50a006a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.249+0000 7f50a87ca700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50a006b0a0 0x7f50a0068940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.249+0000 7f50a87ca700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a006b970 con 0x7f50a0159b40 2024-01-26T23:30:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.249+0000 7f50a87ca700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50a00ca5c0 con 0x7f50a0159770 2024-01-26T23:30:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.249+0000 7f50a87ca700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a00a9ce0 con 0x7f50a006b0a0 2024-01-26T23:30:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.249+0000 7f50a6d67700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50a006b0a0 0x7f50a0068940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.249+0000 7f50a6566700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f50a0159770 0x7f50a006a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.249+0000 7f50a6566700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f50a0159770 0x7f50a006a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:41116/0 (socket says 172.21.15.46:41116) 2024-01-26T23:30:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.249+0000 7f50a6566700 1 -- 172.21.15.46:0/182255066 learned_addr learned my addr 172.21.15.46:0/182255066 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.249+0000 7f50a6d67700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50a006b0a0 0x7f50a0068940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:37432/0 (socket says 172.21.15.46:37432) 2024-01-26T23:30:15.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.249+0000 7f50a6d67700 1 -- 172.21.15.46:0/182255066 >> v2:172.21.15.106:3300/0 conn(0x7f50a0159770 msgr2=0x7f50a006a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:15.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.249+0000 7f50a6d67700 1 --2- 172.21.15.46:0/182255066 >> v2:172.21.15.106:3300/0 conn(0x7f50a0159770 0x7f50a006a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.249+0000 7f50a6d67700 1 -- 172.21.15.46:0/182255066 >> v1:172.21.15.106:6789/0 conn(0x7f50a0159b40 legacy=0x7f50a00aa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:15.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.249+0000 7f50a6d67700 1 -- 172.21.15.46:0/182255066 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a00ca040 con 0x7f50a006b0a0 2024-01-26T23:30:15.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.250+0000 7f50a6566700 1 --2- 172.21.15.46:0/182255066 >> v2:172.21.15.106:3300/0 conn(0x7f50a0159770 0x7f50a006a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:15.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.250+0000 7f50a6d67700 1 --2- 172.21.15.46:0/182255066 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50a006b0a0 0x7f50a0068940 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5098009e60 tx=0x7f5098009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=246c30726410ff9c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:15.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.250+0000 7f50a5564700 1 -- 172.21.15.46:0/182255066 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509800e070 con 0x7f50a006b0a0 2024-01-26T23:30:15.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.250+0000 7f50a5564700 1 -- 172.21.15.46:0/182255066 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509800bb60 con 0x7f50a006b0a0 2024-01-26T23:30:15.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.250+0000 7f50a5564700 1 -- 172.21.15.46:0/182255066 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5098012430 con 0x7f50a006b0a0 2024-01-26T23:30:15.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.250+0000 7f50a87ca700 1 -- 172.21.15.46:0/182255066 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50a006b0a0 msgr2=0x7f50a0068940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:15.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.250+0000 7f50a87ca700 1 --2- 172.21.15.46:0/182255066 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50a006b0a0 0x7f50a0068940 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5098009e60 tx=0x7f5098009780 comp rx=0 tx=0).stop 2024-01-26T23:30:15.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.251+0000 7f50a87ca700 1 -- 172.21.15.46:0/182255066 shutdown_connections 2024-01-26T23:30:15.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.251+0000 7f50a87ca700 1 --2- 172.21.15.46:0/182255066 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50a006b0a0 0x7f50a0068940 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.251+0000 7f50a87ca700 1 --2- 172.21.15.46:0/182255066 >> v2:172.21.15.106:3300/0 conn(0x7f50a0159770 0x7f50a006a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.251+0000 7f50a87ca700 1 -- 172.21.15.46:0/182255066 >> 172.21.15.46:0/182255066 conn(0x7f50a00cd6e0 msgr2=0x7f50a00c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:15.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.251+0000 7f50a87ca700 1 -- 172.21.15.46:0/182255066 shutdown_connections 2024-01-26T23:30:15.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.251+0000 7f50a87ca700 1 -- 172.21.15.46:0/182255066 wait complete. 2024-01-26T23:30:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.252+0000 7f50a87ca700 1 Processor -- start 2024-01-26T23:30:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.252+0000 7f50a87ca700 1 -- start start 2024-01-26T23:30:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a87ca700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50a006b0a0 0x7f50a015c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a87ca700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50a0159770 0x7f50a0158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a87ca700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50a0159b40 0x7f50a0169f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a87ca700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a016a770 con 0x7f50a0159770 2024-01-26T23:30:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a87ca700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a016a4d0 con 0x7f50a006b0a0 2024-01-26T23:30:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a87ca700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a00afb50 con 0x7f50a0159b40 2024-01-26T23:30:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a5d65700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50a0159770 0x7f50a0158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a5d65700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50a0159770 0x7f50a0158a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40428/0 (socket says 172.21.15.46:40428) 2024-01-26T23:30:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a5d65700 1 -- 172.21.15.46:0/2766243569 learned_addr learned my addr 172.21.15.46:0/2766243569 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a6d67700 1 --2- 172.21.15.46:0/2766243569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50a0159b40 0x7f50a0169f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a6566700 1 --2- 172.21.15.46:0/2766243569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50a006b0a0 0x7f50a015c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a5d65700 1 -- 172.21.15.46:0/2766243569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50a0159b40 msgr2=0x7f50a0169f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a5d65700 1 --2- 172.21.15.46:0/2766243569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50a0159b40 0x7f50a0169f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a5d65700 1 -- 172.21.15.46:0/2766243569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50a006b0a0 msgr2=0x7f50a015c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a5d65700 1 --2- 172.21.15.46:0/2766243569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50a006b0a0 0x7f50a015c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a5d65700 1 -- 172.21.15.46:0/2766243569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50900094f0 con 0x7f50a0159770 2024-01-26T23:30:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a6566700 1 --2- 172.21.15.46:0/2766243569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50a006b0a0 0x7f50a015c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a5d65700 1 --2- 172.21.15.46:0/2766243569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50a0159770 0x7f50a0158a20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f509c00c590 tx=0x7f509c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50967fc700 1 -- 172.21.15.46:0/2766243569 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509c011590 con 0x7f50a0159770 2024-01-26T23:30:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50967fc700 1 -- 172.21.15.46:0/2766243569 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509c011710 con 0x7f50a0159770 2024-01-26T23:30:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.253+0000 7f50a87ca700 1 -- 172.21.15.46:0/2766243569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5098009a20 con 0x7f50a0159770 2024-01-26T23:30:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.254+0000 7f50967fc700 1 -- 172.21.15.46:0/2766243569 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509c011ad0 con 0x7f50a0159770 2024-01-26T23:30:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.254+0000 7f50a87ca700 1 -- 172.21.15.46:0/2766243569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a00b0110 con 0x7f50a0159770 2024-01-26T23:30:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.254+0000 7f50a87ca700 1 -- 172.21.15.46:0/2766243569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a004c710 con 0x7f50a0159770 2024-01-26T23:30:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.255+0000 7f50967fc700 1 -- 172.21.15.46:0/2766243569 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f509c01b3b0 con 0x7f50a0159770 2024-01-26T23:30:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.255+0000 7f50967fc700 1 --2- 172.21.15.46:0/2766243569 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5084066f30 0x7f50840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.255+0000 7f50967fc700 1 -- 172.21.15.46:0/2766243569 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f509c083d30 con 0x7f50a0159770 2024-01-26T23:30:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.256+0000 7f50a6566700 1 --2- 172.21.15.46:0/2766243569 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5084066f30 0x7f50840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.256+0000 7f50a6566700 1 --2- 172.21.15.46:0/2766243569 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5084066f30 0x7f50840693e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5090006d40 tx=0x7f5090008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.260+0000 7f50967fc700 1 -- 172.21.15.46:0/2766243569 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f509c050cb0 con 0x7f50a0159770 2024-01-26T23:30:15.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.467+0000 7f50a87ca700 1 -- 172.21.15.46:0/2766243569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50a00cbf60 con 0x7f50a0159770 2024-01-26T23:30:15.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.468+0000 7f50967fc700 1 -- 172.21.15.46:0/2766243569 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7f509c017070 con 0x7f50a0159770 2024-01-26T23:30:15.468 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:15.469 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:49.075104+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:15.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.470+0000 7f50a87ca700 1 -- 172.21.15.46:0/2766243569 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5084066f30 msgr2=0x7f50840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:15.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.470+0000 7f50a87ca700 1 --2- 172.21.15.46:0/2766243569 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5084066f30 0x7f50840693e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5090006d40 tx=0x7f5090008040 comp rx=0 tx=0).stop 2024-01-26T23:30:15.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.470+0000 7f50a87ca700 1 -- 172.21.15.46:0/2766243569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50a0159770 msgr2=0x7f50a0158a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:15.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.470+0000 7f50a87ca700 1 --2- 172.21.15.46:0/2766243569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50a0159770 0x7f50a0158a20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f509c00c590 tx=0x7f509c00c8a0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.470+0000 7f50a87ca700 1 -- 172.21.15.46:0/2766243569 shutdown_connections 2024-01-26T23:30:15.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.470+0000 7f50a87ca700 1 --2- 172.21.15.46:0/2766243569 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5084066f30 0x7f50840693e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.470+0000 7f50a87ca700 1 --2- 172.21.15.46:0/2766243569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50a0159b40 0x7f50a0169f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.470+0000 7f50a87ca700 1 --2- 172.21.15.46:0/2766243569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50a006b0a0 0x7f50a015c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.470+0000 7f50a87ca700 1 --2- 172.21.15.46:0/2766243569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50a0159770 0x7f50a0158a20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.470+0000 7f50a87ca700 1 -- 172.21.15.46:0/2766243569 >> 172.21.15.46:0/2766243569 conn(0x7f50a00cd6e0 msgr2=0x7f50a0068720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:15.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.470+0000 7f50a87ca700 1 -- 172.21.15.46:0/2766243569 shutdown_connections 2024-01-26T23:30:15.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.470+0000 7f50a87ca700 1 -- 172.21.15.46:0/2766243569 wait complete. 2024-01-26T23:30:15.484 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:30:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.616+0000 7ff35455f700 1 Processor -- start 2024-01-26T23:30:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.616+0000 7ff35455f700 1 -- start start 2024-01-26T23:30:15.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.617+0000 7ff35455f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff34c159760 0x7ff34c0b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:15.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.617+0000 7ff35455f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff34c068040 0x7ff34c0b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:15.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.617+0000 7ff35455f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34c15abc0 con 0x7ff34c159b30 2024-01-26T23:30:15.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.617+0000 7ff35455f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff34c0b2e90 con 0x7ff34c159760 2024-01-26T23:30:15.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.617+0000 7ff35455f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c159fb0 con 0x7ff34c068040 2024-01-26T23:30:15.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.617+0000 7ff352afc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff34c068040 0x7ff34c0b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:15.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.617+0000 7ff3522fb700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff34c159760 0x7ff34c0b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:15.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.618+0000 7ff352afc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff34c068040 0x7ff34c0b11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:37450/0 (socket says 172.21.15.46:37450) 2024-01-26T23:30:15.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.618+0000 7ff352afc700 1 -- 172.21.15.46:0/3326570534 learned_addr learned my addr 172.21.15.46:0/3326570534 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:15.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.618+0000 7ff3522fb700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff34c159760 0x7ff34c0b0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:41136/0 (socket says 172.21.15.46:41136) 2024-01-26T23:30:15.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.618+0000 7ff351afa700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7ff34c159b30 0x7ff34c067930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:59506/0 (socket says 172.21.15.46:59506) 2024-01-26T23:30:15.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.618+0000 7ff352afc700 1 -- 172.21.15.46:0/3326570534 >> v2:172.21.15.106:3300/0 conn(0x7ff34c159760 msgr2=0x7ff34c0b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:15.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.618+0000 7ff352afc700 1 --2- 172.21.15.46:0/3326570534 >> v2:172.21.15.106:3300/0 conn(0x7ff34c159760 0x7ff34c0b0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.618+0000 7ff352afc700 1 -- 172.21.15.46:0/3326570534 >> v1:172.21.15.106:6789/0 conn(0x7ff34c159b30 legacy=0x7ff34c067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:15.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.618+0000 7ff352afc700 1 -- 172.21.15.46:0/3326570534 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c15a290 con 0x7ff34c068040 2024-01-26T23:30:15.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.618+0000 7ff352afc700 1 --2- 172.21.15.46:0/3326570534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff34c068040 0x7ff34c0b11b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff344009f20 tx=0x7ff344009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9908854a18adcde8 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:15.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.618+0000 7ff3512f9700 1 -- 172.21.15.46:0/3326570534 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34400e070 con 0x7ff34c068040 2024-01-26T23:30:15.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.618+0000 7ff3512f9700 1 -- 172.21.15.46:0/3326570534 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34400b9d0 con 0x7ff34c068040 2024-01-26T23:30:15.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.618+0000 7ff3512f9700 1 -- 172.21.15.46:0/3326570534 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3440092e0 con 0x7ff34c068040 2024-01-26T23:30:15.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.619+0000 7ff35455f700 1 -- 172.21.15.46:0/3326570534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff34c068040 msgr2=0x7ff34c0b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:15.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.619+0000 7ff35455f700 1 --2- 172.21.15.46:0/3326570534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff34c068040 0x7ff34c0b11b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff344009f20 tx=0x7ff344009840 comp rx=0 tx=0).stop 2024-01-26T23:30:15.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.619+0000 7ff35455f700 1 -- 172.21.15.46:0/3326570534 shutdown_connections 2024-01-26T23:30:15.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.619+0000 7ff35455f700 1 --2- 172.21.15.46:0/3326570534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff34c068040 0x7ff34c0b11b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.619+0000 7ff35455f700 1 --2- 172.21.15.46:0/3326570534 >> v2:172.21.15.106:3300/0 conn(0x7ff34c159760 0x7ff34c0b0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.619+0000 7ff35455f700 1 -- 172.21.15.46:0/3326570534 >> 172.21.15.46:0/3326570534 conn(0x7ff34c06b630 msgr2=0x7ff34c0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:15.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.619+0000 7ff35455f700 1 -- 172.21.15.46:0/3326570534 shutdown_connections 2024-01-26T23:30:15.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.619+0000 7ff35455f700 1 -- 172.21.15.46:0/3326570534 wait complete. 2024-01-26T23:30:15.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.620+0000 7ff35455f700 1 Processor -- start 2024-01-26T23:30:15.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.621+0000 7ff35455f700 1 -- start start 2024-01-26T23:30:15.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.621+0000 7ff35455f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff34c068040 0x7ff34c0ac550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:15.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.621+0000 7ff35455f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff34c159760 0x7ff34c146fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:15.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.621+0000 7ff35455f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff34c0abaa0 0x7ff34c169700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:15.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.621+0000 7ff35455f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c154b40 con 0x7ff34c159760 2024-01-26T23:30:15.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.621+0000 7ff35455f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c0ab610 con 0x7ff34c068040 2024-01-26T23:30:15.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.621+0000 7ff35455f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c0aaa00 con 0x7ff34c0abaa0 2024-01-26T23:30:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.622+0000 7ff351afa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff34c159760 0x7ff34c146fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.622+0000 7ff352afc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff34c0abaa0 0x7ff34c169700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.622+0000 7ff351afa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff34c159760 0x7ff34c146fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40440/0 (socket says 172.21.15.46:40440) 2024-01-26T23:30:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.622+0000 7ff351afa700 1 -- 172.21.15.46:0/4038896565 learned_addr learned my addr 172.21.15.46:0/4038896565 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.622+0000 7ff3522fb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff34c068040 0x7ff34c0ac550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.622+0000 7ff351afa700 1 -- 172.21.15.46:0/4038896565 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff34c0abaa0 msgr2=0x7ff34c169700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.622+0000 7ff351afa700 1 --2- 172.21.15.46:0/4038896565 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff34c0abaa0 0x7ff34c169700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.622+0000 7ff351afa700 1 -- 172.21.15.46:0/4038896565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff34c068040 msgr2=0x7ff34c0ac550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.622+0000 7ff351afa700 1 --2- 172.21.15.46:0/4038896565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff34c068040 0x7ff34c0ac550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.622+0000 7ff351afa700 1 -- 172.21.15.46:0/4038896565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff344009ae0 con 0x7ff34c159760 2024-01-26T23:30:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.622+0000 7ff351afa700 1 --2- 172.21.15.46:0/4038896565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff34c159760 0x7ff34c146fb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00c560 tx=0x7ff33c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.622+0000 7ff3427fc700 1 -- 172.21.15.46:0/4038896565 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c005a70 con 0x7ff34c159760 2024-01-26T23:30:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.622+0000 7ff3427fc700 1 -- 172.21.15.46:0/4038896565 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33c005bd0 con 0x7ff34c159760 2024-01-26T23:30:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.622+0000 7ff3427fc700 1 -- 172.21.15.46:0/4038896565 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c0070d0 con 0x7ff34c159760 2024-01-26T23:30:15.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.623+0000 7ff35455f700 1 -- 172.21.15.46:0/4038896565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff34c0aace0 con 0x7ff34c159760 2024-01-26T23:30:15.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.623+0000 7ff35455f700 1 -- 172.21.15.46:0/4038896565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff34c0ab2a0 con 0x7ff34c159760 2024-01-26T23:30:15.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.624+0000 7ff35455f700 1 -- 172.21.15.46:0/4038896565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34c0b3300 con 0x7ff34c159760 2024-01-26T23:30:15.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.624+0000 7ff3427fc700 1 -- 172.21.15.46:0/4038896565 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff33c007480 con 0x7ff34c159760 2024-01-26T23:30:15.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.625+0000 7ff3427fc700 1 --2- 172.21.15.46:0/4038896565 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff330066f30 0x7ff3300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:15.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.625+0000 7ff3427fc700 1 -- 172.21.15.46:0/4038896565 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7ff33c083b80 con 0x7ff34c159760 2024-01-26T23:30:15.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.625+0000 7ff3522fb700 1 --2- 172.21.15.46:0/4038896565 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff330066f30 0x7ff3300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:15.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.625+0000 7ff3522fb700 1 --2- 172.21.15.46:0/4038896565 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff330066f30 0x7ff3300693e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff348009db0 tx=0x7ff3480091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.629+0000 7ff3427fc700 1 -- 172.21.15.46:0/4038896565 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff33c050b00 con 0x7ff34c159760 2024-01-26T23:30:15.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.886+0000 7ff35455f700 1 -- 172.21.15.46:0/4038896565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff34c064c40 con 0x7ff34c159760 2024-01-26T23:30:15.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.887+0000 7ff3427fc700 1 -- 172.21.15.46:0/4038896565 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4593 (secure 0 0 0) 0x7ff33c017020 con 0x7ff34c159760 2024-01-26T23:30:15.888 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:15.888 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3409861270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3409861270},{"type":"v1","addr":"172.21.15.46:6833","nonce":3409861270}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3319220815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3319220815},{"type":"v1","addr":"172.21.15.106:6836","nonce":3319220815}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4252,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/529198432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":529198432},{"type":"v1","addr":"172.21.15.106:6837","nonce":529198432}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:29:44.031963+0000","modified":"2024-01-26T23:30:03.652307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/688086235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":688086235},{"type":"v1","addr":"172.21.15.46:6835","nonce":688086235}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2024-01-26T23:30:15.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.889+0000 7ff35455f700 1 -- 172.21.15.46:0/4038896565 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff330066f30 msgr2=0x7ff3300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:15.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.889+0000 7ff35455f700 1 --2- 172.21.15.46:0/4038896565 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff330066f30 0x7ff3300693e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff348009db0 tx=0x7ff3480091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.889+0000 7ff35455f700 1 -- 172.21.15.46:0/4038896565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff34c159760 msgr2=0x7ff34c146fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:15.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.889+0000 7ff35455f700 1 --2- 172.21.15.46:0/4038896565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff34c159760 0x7ff34c146fb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00c560 tx=0x7ff33c007f70 comp rx=0 tx=0).stop 2024-01-26T23:30:15.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.889+0000 7ff35455f700 1 -- 172.21.15.46:0/4038896565 shutdown_connections 2024-01-26T23:30:15.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.889+0000 7ff35455f700 1 --2- 172.21.15.46:0/4038896565 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff330066f30 0x7ff3300693e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.889+0000 7ff35455f700 1 --2- 172.21.15.46:0/4038896565 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff34c0abaa0 0x7ff34c169700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.889+0000 7ff35455f700 1 --2- 172.21.15.46:0/4038896565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff34c068040 0x7ff34c0ac550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.889+0000 7ff35455f700 1 --2- 172.21.15.46:0/4038896565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff34c159760 0x7ff34c146fb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:15.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.889+0000 7ff35455f700 1 -- 172.21.15.46:0/4038896565 >> 172.21.15.46:0/4038896565 conn(0x7ff34c06b630 msgr2=0x7ff34c15af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:15.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.889+0000 7ff35455f700 1 -- 172.21.15.46:0/4038896565 shutdown_connections 2024-01-26T23:30:15.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:15.889+0000 7ff35455f700 1 -- 172.21.15.46:0/4038896565 wait complete. 2024-01-26T23:30:15.891 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 9 2024-01-26T23:30:16.306 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-26T23:29:44.031963+0000', 'modified': '2024-01-26T23:30:03.652307+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4254}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4254': {'gid': 4254, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6835/688086235', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6834', 'nonce': 688086235}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 688086235}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-26T23:30:16.306 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-26T23:30:16.306 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-26T23:30:16.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.447+0000 7f818b23f700 1 Processor -- start 2024-01-26T23:30:16.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.448+0000 7f818b23f700 1 -- start start 2024-01-26T23:30:16.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.448+0000 7f818b23f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f818406cbb0 0x7f81840ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.448+0000 7f818b23f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f81840b0280 0x7f81841565a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.448+0000 7f818b23f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81840b0760 con 0x7f81841561d0 2024-01-26T23:30:16.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.448+0000 7f818b23f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81840b0a40 con 0x7f81840b0280 2024-01-26T23:30:16.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.448+0000 7f818b23f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81840b0d20 con 0x7f818406cbb0 2024-01-26T23:30:16.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.449+0000 7f8188fdb700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f81841561d0 0x7f818406c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50484/0 (socket says 172.21.15.46:50484) 2024-01-26T23:30:16.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.449+0000 7f8188fdb700 1 -- 172.21.15.46:0/637448814 learned_addr learned my addr 172.21.15.46:0/637448814 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:16.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.449+0000 7f81897dc700 1 --2- 172.21.15.46:0/637448814 >> v2:172.21.15.106:3300/0 conn(0x7f81840b0280 0x7f81841565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.449+0000 7f8183fff700 1 --2- 172.21.15.46:0/637448814 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f818406cbb0 0x7f81840ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.449+0000 7f81837fe700 1 -- 172.21.15.46:0/637448814 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650365189 0 0) 0x7f81840b0760 con 0x7f81841561d0 2024-01-26T23:30:16.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.449+0000 7f81837fe700 1 -- 172.21.15.46:0/637448814 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8174003570 con 0x7f81841561d0 2024-01-26T23:30:16.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.449+0000 7f81897dc700 1 -- 172.21.15.46:0/637448814 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f818406cbb0 msgr2=0x7f81840ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:16.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.449+0000 7f81897dc700 1 --2- 172.21.15.46:0/637448814 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f818406cbb0 0x7f81840ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.449+0000 7f81897dc700 1 -- 172.21.15.46:0/637448814 >> v1:172.21.15.46:6789/0 conn(0x7f81841561d0 legacy=0x7f818406c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:16.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.449+0000 7f81897dc700 1 -- 172.21.15.46:0/637448814 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81840b1000 con 0x7f81840b0280 2024-01-26T23:30:16.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.449+0000 7f8183fff700 1 --2- 172.21.15.46:0/637448814 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f818406cbb0 0x7f81840ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:16.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.449+0000 7f81897dc700 1 --2- 172.21.15.46:0/637448814 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81840b0280 0x7f81841565a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f817c009e60 tx=0x7f817c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8a1a61a76bc4d53 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:16.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.449+0000 7f81837fe700 1 -- 172.21.15.46:0/637448814 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c00bab0 con 0x7f81840b0280 2024-01-26T23:30:16.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.449+0000 7f81837fe700 1 -- 172.21.15.46:0/637448814 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817c00bc10 con 0x7f81840b0280 2024-01-26T23:30:16.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.449+0000 7f81837fe700 1 -- 172.21.15.46:0/637448814 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c011460 con 0x7f81840b0280 2024-01-26T23:30:16.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.450+0000 7f818b23f700 1 -- 172.21.15.46:0/637448814 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81840b0280 msgr2=0x7f81841565a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:16.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.450+0000 7f818b23f700 1 --2- 172.21.15.46:0/637448814 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81840b0280 0x7f81841565a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f817c009e60 tx=0x7f817c009780 comp rx=0 tx=0).stop 2024-01-26T23:30:16.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.450+0000 7f818b23f700 1 -- 172.21.15.46:0/637448814 shutdown_connections 2024-01-26T23:30:16.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.450+0000 7f818b23f700 1 --2- 172.21.15.46:0/637448814 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81840b0280 0x7f81841565a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.450+0000 7f818b23f700 1 --2- 172.21.15.46:0/637448814 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f818406cbb0 0x7f81840ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.450+0000 7f818b23f700 1 -- 172.21.15.46:0/637448814 >> 172.21.15.46:0/637448814 conn(0x7f818406b630 msgr2=0x7f81840aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:16.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.450+0000 7f818b23f700 1 -- 172.21.15.46:0/637448814 shutdown_connections 2024-01-26T23:30:16.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.450+0000 7f818b23f700 1 -- 172.21.15.46:0/637448814 wait complete. 2024-01-26T23:30:16.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.451+0000 7f818b23f700 1 Processor -- start 2024-01-26T23:30:16.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.452+0000 7f818b23f700 1 -- start start 2024-01-26T23:30:16.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.452+0000 7f818b23f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f818406cbb0 0x7f8184140e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.452+0000 7f818b23f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81840b0280 0x7f8184165360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.452+0000 7f818b23f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81841561d0 0x7f81841524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.452+0000 7f818b23f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8184157b80 con 0x7f818406cbb0 2024-01-26T23:30:16.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.452+0000 7f818b23f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8184156c90 con 0x7f81840b0280 2024-01-26T23:30:16.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.452+0000 7f818b23f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8184156f70 con 0x7f81841561d0 2024-01-26T23:30:16.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.453+0000 7f8188fdb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f818406cbb0 0x7f8184140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.453+0000 7f8188fdb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f818406cbb0 0x7f8184140e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40448/0 (socket says 172.21.15.46:40448) 2024-01-26T23:30:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.453+0000 7f8188fdb700 1 -- 172.21.15.46:0/1204543396 learned_addr learned my addr 172.21.15.46:0/1204543396 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.453+0000 7f8183fff700 1 --2- 172.21.15.46:0/1204543396 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81840b0280 0x7f8184165360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.453+0000 7f81897dc700 1 --2- 172.21.15.46:0/1204543396 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81841561d0 0x7f81841524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.453+0000 7f8188fdb700 1 -- 172.21.15.46:0/1204543396 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81841561d0 msgr2=0x7f81841524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.453+0000 7f8188fdb700 1 --2- 172.21.15.46:0/1204543396 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81841561d0 0x7f81841524e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.453+0000 7f8188fdb700 1 -- 172.21.15.46:0/1204543396 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81840b0280 msgr2=0x7f8184165360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.453+0000 7f8188fdb700 1 --2- 172.21.15.46:0/1204543396 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81840b0280 0x7f8184165360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.453+0000 7f8188fdb700 1 -- 172.21.15.46:0/1204543396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8170009420 con 0x7f818406cbb0 2024-01-26T23:30:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.453+0000 7f8188fdb700 1 --2- 172.21.15.46:0/1204543396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f818406cbb0 0x7f8184140e70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f817800c590 tx=0x7f8178007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.453+0000 7f8180ff9700 1 -- 172.21.15.46:0/1204543396 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8178005b30 con 0x7f818406cbb0 2024-01-26T23:30:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.453+0000 7f8180ff9700 1 -- 172.21.15.46:0/1204543396 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8178005cb0 con 0x7f818406cbb0 2024-01-26T23:30:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.453+0000 7f818b23f700 1 -- 172.21.15.46:0/1204543396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f817c009a20 con 0x7f818406cbb0 2024-01-26T23:30:16.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.453+0000 7f8180ff9700 1 -- 172.21.15.46:0/1204543396 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8178007320 con 0x7f818406cbb0 2024-01-26T23:30:16.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.453+0000 7f818b23f700 1 -- 172.21.15.46:0/1204543396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8184157530 con 0x7f818406cbb0 2024-01-26T23:30:16.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.455+0000 7f8180ff9700 1 -- 172.21.15.46:0/1204543396 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f817801b3b0 con 0x7f818406cbb0 2024-01-26T23:30:16.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.455+0000 7f8180ff9700 1 --2- 172.21.15.46:0/1204543396 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8168066f30 0x7f81680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.455+0000 7f8180ff9700 1 -- 172.21.15.46:0/1204543396 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f8178083c50 con 0x7f818406cbb0 2024-01-26T23:30:16.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.456+0000 7f8183fff700 1 --2- 172.21.15.46:0/1204543396 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8168066f30 0x7f81680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.456+0000 7f8183fff700 1 --2- 172.21.15.46:0/1204543396 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8168066f30 0x7f81680693e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f81840c26e0 tx=0x7f81700091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:16.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.458+0000 7f8194542b80 1 Processor -- start 2024-01-26T23:30:16.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.458+0000 7f8194542b80 1 -- start start 2024-01-26T23:30:16.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.458+0000 7f8194542b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c03f0010 0x5622c03f03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.458+0000 7f8194542b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c0438360 0x5622c041a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.458+0000 7f8194542b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 0x5622c041ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.458+0000 7f8194542b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c02f39f0 con 0x5622c03f0010 2024-01-26T23:30:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.458+0000 7f8194542b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c0345130 con 0x5622c0438360 2024-01-26T23:30:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.458+0000 7f8194542b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5622c0342930 con 0x5622c03ef5d0 2024-01-26T23:30:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.459+0000 7f8188fdb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c03f0010 0x5622c03f03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.459+0000 7f8188fdb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c03f0010 0x5622c03f03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40450/0 (socket says 172.21.15.46:40450) 2024-01-26T23:30:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.459+0000 7f8188fdb700 1 -- 172.21.15.46:0/3480618560 learned_addr learned my addr 172.21.15.46:0/3480618560 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.459+0000 7f81897dc700 1 --2- 172.21.15.46:0/3480618560 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 0x5622c041ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.459+0000 7f8183fff700 1 --2- 172.21.15.46:0/3480618560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c0438360 0x5622c041a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.459+0000 7f8188fdb700 1 -- 172.21.15.46:0/3480618560 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 msgr2=0x5622c041ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.459+0000 7f8188fdb700 1 --2- 172.21.15.46:0/3480618560 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 0x5622c041ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.459+0000 7f8188fdb700 1 -- 172.21.15.46:0/3480618560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c0438360 msgr2=0x5622c041a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.459+0000 7f8188fdb700 1 --2- 172.21.15.46:0/3480618560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c0438360 0x5622c041a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.459+0000 7f8188fdb700 1 -- 172.21.15.46:0/3480618560 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622c017db10 con 0x5622c03f0010 2024-01-26T23:30:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.459+0000 7f8188fdb700 1 --2- 172.21.15.46:0/3480618560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c03f0010 0x5622c03f03e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8178081e50 tx=0x7f8178081de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=352ccd5a4c096581 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.459+0000 7f816dffb700 1 -- 172.21.15.46:0/3480618560 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8178054890 con 0x5622c03f0010 2024-01-26T23:30:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.459+0000 7f816dffb700 1 -- 172.21.15.46:0/3480618560 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8178069070 con 0x5622c03f0010 2024-01-26T23:30:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.459+0000 7f816dffb700 1 -- 172.21.15.46:0/3480618560 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8178057ae0 con 0x5622c03f0010 2024-01-26T23:30:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.460+0000 7f8194542b80 1 -- 172.21.15.46:0/3480618560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c03f0010 msgr2=0x5622c03f03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.460+0000 7f8194542b80 1 --2- 172.21.15.46:0/3480618560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c03f0010 0x5622c03f03e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8178081e50 tx=0x7f8178081de0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.460+0000 7f8194542b80 1 -- 172.21.15.46:0/3480618560 shutdown_connections 2024-01-26T23:30:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.460+0000 7f8194542b80 1 --2- 172.21.15.46:0/3480618560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c03f0010 0x5622c03f03e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.460+0000 7f8194542b80 1 --2- 172.21.15.46:0/3480618560 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 0x5622c041ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.460+0000 7f8194542b80 1 --2- 172.21.15.46:0/3480618560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c0438360 0x5622c041a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.460+0000 7f8194542b80 1 -- 172.21.15.46:0/3480618560 >> 172.21.15.46:0/3480618560 conn(0x5622c03efa70 msgr2=0x5622c0438820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.460+0000 7f8194542b80 1 -- 172.21.15.46:0/3480618560 shutdown_connections 2024-01-26T23:30:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.460+0000 7f8194542b80 1 -- 172.21.15.46:0/3480618560 wait complete. 2024-01-26T23:30:16.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.461+0000 7f8194542b80 1 -- start start 2024-01-26T23:30:16.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.461+0000 7f8194542b80 1 Processor -- start 2024-01-26T23:30:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8194542b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 0x5622c04451e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8194542b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c03f0010 0x5622c042f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8194542b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c0438360 0x5622c042e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8194542b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c033e720 con 0x5622c03f0010 2024-01-26T23:30:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8194542b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c02ea1b0 con 0x5622c0438360 2024-01-26T23:30:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8194542b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5622c02e9e50 con 0x5622c03ef5d0 2024-01-26T23:30:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8194542b80 10 client.0 _ll_register_callbacks cb 0x5622c014c7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:30:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8183fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c03f0010 0x5622c042f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8183fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c03f0010 0x5622c042f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40458/0 (socket says 172.21.15.46:40458) 2024-01-26T23:30:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8183fff700 1 -- 172.21.15.46:0/1955840860 learned_addr learned my addr 172.21.15.46:0/1955840860 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8188fdb700 1 --2- 172.21.15.46:0/1955840860 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 0x5622c04451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f81897dc700 1 --2- 172.21.15.46:0/1955840860 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c0438360 0x5622c042e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8183fff700 1 -- 172.21.15.46:0/1955840860 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 msgr2=0x5622c04451e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8183fff700 1 --2- 172.21.15.46:0/1955840860 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 0x5622c04451e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8183fff700 1 -- 172.21.15.46:0/1955840860 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c0438360 msgr2=0x5622c042e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8183fff700 1 --2- 172.21.15.46:0/1955840860 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c0438360 0x5622c042e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8183fff700 1 -- 172.21.15.46:0/1955840860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8184157530 con 0x5622c03f0010 2024-01-26T23:30:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f81897dc700 1 --2- 172.21.15.46:0/1955840860 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c0438360 0x5622c042e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:30:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8183fff700 1 --2- 172.21.15.46:0/1955840860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c03f0010 0x5622c042f150 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8184150190 tx=0x7f8170016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b33b897dc693fd76 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.462+0000 7f8162ffd700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:30:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.463+0000 7f8162ffd700 1 -- 172.21.15.46:0/1955840860 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8170011ed0 con 0x5622c03f0010 2024-01-26T23:30:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.463+0000 7f8162ffd700 1 -- 172.21.15.46:0/1955840860 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f8170013b90 con 0x5622c03f0010 2024-01-26T23:30:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.463+0000 7f8194542b80 1 -- 172.21.15.46:0/1955840860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5622c03e3190 con 0x5622c03f0010 2024-01-26T23:30:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.463+0000 7f8162ffd700 1 -- 172.21.15.46:0/1955840860 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8170023470 con 0x5622c03f0010 2024-01-26T23:30:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.463+0000 7f8194542b80 10 client.4725 fetch_fsmap learned FSMap version 9 2024-01-26T23:30:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.463+0000 7f8194542b80 1 -- 172.21.15.46:0/1955840860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x5622c030c290 con 0x5622c03f0010 2024-01-26T23:30:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.463+0000 7f8162ffd700 1 -- 172.21.15.46:0/1955840860 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5622c03e3190 con 0x5622c03f0010 2024-01-26T23:30:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.463+0000 7f8162ffd700 1 -- 172.21.15.46:0/1955840860 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 9) v1 ==== 3108+0+0 (secure 0 0 0) 0x7f81700235d0 con 0x5622c03f0010 2024-01-26T23:30:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.463+0000 7f8194542b80 10 client.4725 fetch_fsmap finished waiting for FSMap version 9 2024-01-26T23:30:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.463+0000 7f8194542b80 10 client.4725 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-26T23:30:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.463+0000 7f8194542b80 20 client.4725 populate_metadata read hostname 'smithi046' 2024-01-26T23:30:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.463+0000 7f8194542b80 1 --2- 172.21.15.46:0/1955840860 >> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] conn(0x5622c041b4c0 0x5622c041b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.463+0000 7f8194542b80 4 client.4725 mds_command: new command op to 4254 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-26T23:30:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.463+0000 7f8194542b80 1 -- 172.21.15.46:0/1955840860 --> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5622c02e0ee0 con 0x5622c041b4c0 2024-01-26T23:30:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.463+0000 7f8188fdb700 1 --2- 172.21.15.46:0/1955840860 >> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] conn(0x5622c041b4c0 0x5622c041b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.464+0000 7f8188fdb700 1 --2- 172.21.15.46:0/1955840860 >> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] conn(0x5622c041b4c0 0x5622c041b890 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b189f46ae9f62b3b server_cookie=27d992bfdafd1607 in_seq=0 out_seq=0 2024-01-26T23:30:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.464+0000 7f8162ffd700 10 client.4725 ms_handle_connect on v2:172.21.15.46:6834/688086235 2024-01-26T23:30:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.464+0000 7f8162ffd700 1 -- 172.21.15.46:0/1955840860 <== mds.0 v2:172.21.15.46:6834/688086235 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x5622c02e0ee0 con 0x5622c041b4c0 2024-01-26T23:30:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.464+0000 7f8162ffd700 10 client.4725 handle_command_reply: tid=0 2024-01-26T23:30:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.464+0000 7f8194542b80 1 client.4725 shutdown 2024-01-26T23:30:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.464+0000 7f8194542b80 1 -- 172.21.15.46:0/1955840860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c03f0010 msgr2=0x5622c042f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.464+0000 7f8194542b80 1 --2- 172.21.15.46:0/1955840860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c03f0010 0x5622c042f150 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8184150190 tx=0x7f8170016000 comp rx=0 tx=0).stop 2024-01-26T23:30:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.465+0000 7f8194542b80 1 -- 172.21.15.46:0/1955840860 shutdown_connections 2024-01-26T23:30:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.465+0000 7f8194542b80 1 --2- 172.21.15.46:0/1955840860 >> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] conn(0x5622c041b4c0 0x5622c041b890 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.465+0000 7f8194542b80 1 --2- 172.21.15.46:0/1955840860 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c0438360 0x5622c042e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.465+0000 7f8194542b80 1 --2- 172.21.15.46:0/1955840860 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 0x5622c04451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.465+0000 7f8194542b80 1 --2- 172.21.15.46:0/1955840860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c03f0010 0x5622c042f150 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.465+0000 7f8162ffd700 0 client.4725 ms_handle_reset on v2:172.21.15.46:6834/688086235 2024-01-26T23:30:16.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.465+0000 7f8194542b80 1 -- 172.21.15.46:0/1955840860 >> 172.21.15.46:0/1955840860 conn(0x5622c03efa70 msgr2=0x5622c03ead00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:16.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.465+0000 7f8194542b80 1 -- 172.21.15.46:0/1955840860 shutdown_connections 2024-01-26T23:30:16.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.465+0000 7f8194542b80 1 -- 172.21.15.46:0/1955840860 wait complete. 2024-01-26T23:30:16.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.465+0000 7f8194542b80 20 client.4725 trim_cache size 0 max 16384 2024-01-26T23:30:16.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.483+0000 7f8194542b80 1 Processor -- start 2024-01-26T23:30:16.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.483+0000 7f8194542b80 1 -- start start 2024-01-26T23:30:16.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f8194542b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 0x5622c0419db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f8194542b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c03f0010 0x5622c0438730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f8194542b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c0438360 0x5622c0438c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f8194542b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c0236e40 con 0x5622c0438360 2024-01-26T23:30:16.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f8194542b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c03cd020 con 0x5622c03f0010 2024-01-26T23:30:16.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f8194542b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5622c041bf90 con 0x5622c03ef5d0 2024-01-26T23:30:16.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f81897dc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c0438360 0x5622c0438c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f81897dc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c0438360 0x5622c0438c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40466/0 (socket says 172.21.15.46:40466) 2024-01-26T23:30:16.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f81897dc700 1 -- 172.21.15.46:0/1014404384 learned_addr learned my addr 172.21.15.46:0/1014404384 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:16.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f8188fdb700 1 --2- 172.21.15.46:0/1014404384 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 0x5622c0419db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f8183fff700 1 --2- 172.21.15.46:0/1014404384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c03f0010 0x5622c0438730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f81897dc700 1 -- 172.21.15.46:0/1014404384 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 msgr2=0x5622c0419db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:16.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f81897dc700 1 --2- 172.21.15.46:0/1014404384 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 0x5622c0419db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f81897dc700 1 -- 172.21.15.46:0/1014404384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c03f0010 msgr2=0x5622c0438730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:16.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f81897dc700 1 --2- 172.21.15.46:0/1014404384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c03f0010 0x5622c0438730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f81897dc700 1 -- 172.21.15.46:0/1014404384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622c01d5c80 con 0x5622c0438360 2024-01-26T23:30:16.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f81897dc700 1 --2- 172.21.15.46:0/1014404384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c0438360 0x5622c0438c70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f81840ce590 tx=0x7f817c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac518a018939d02 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:16.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.484+0000 7f816e7fc700 1 -- 172.21.15.46:0/1014404384 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c00e070 con 0x5622c0438360 2024-01-26T23:30:16.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.485+0000 7f816e7fc700 1 -- 172.21.15.46:0/1014404384 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817c0032f0 con 0x5622c0438360 2024-01-26T23:30:16.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.485+0000 7f816e7fc700 1 -- 172.21.15.46:0/1014404384 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c0117d0 con 0x5622c0438360 2024-01-26T23:30:16.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.485+0000 7f8194542b80 1 -- 172.21.15.46:0/1014404384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c0438360 msgr2=0x5622c0438c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:16.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.485+0000 7f8194542b80 1 --2- 172.21.15.46:0/1014404384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c0438360 0x5622c0438c70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f81840ce590 tx=0x7f817c01b000 comp rx=0 tx=0).stop 2024-01-26T23:30:16.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.485+0000 7f8194542b80 1 -- 172.21.15.46:0/1014404384 shutdown_connections 2024-01-26T23:30:16.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.485+0000 7f8194542b80 1 --2- 172.21.15.46:0/1014404384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c0438360 0x5622c0438c70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.485+0000 7f8194542b80 1 --2- 172.21.15.46:0/1014404384 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 0x5622c0419db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.485+0000 7f8194542b80 1 --2- 172.21.15.46:0/1014404384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c03f0010 0x5622c0438730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.485+0000 7f8194542b80 1 -- 172.21.15.46:0/1014404384 >> 172.21.15.46:0/1014404384 conn(0x5622c03efa70 msgr2=0x5622c042e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:16.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.485+0000 7f8194542b80 1 -- 172.21.15.46:0/1014404384 shutdown_connections 2024-01-26T23:30:16.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.485+0000 7f8194542b80 1 -- 172.21.15.46:0/1014404384 wait complete. 2024-01-26T23:30:16.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.486+0000 7f8194542b80 1 -- start start 2024-01-26T23:30:16.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.486+0000 7f8194542b80 1 Processor -- start 2024-01-26T23:30:16.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.487+0000 7f8194542b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 0x5622c0498f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.487+0000 7f8194542b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c03f0010 0x5622c03e9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.487+0000 7f8194542b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c0438360 0x5622c03ea270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.487+0000 7f8194542b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c04317f0 con 0x5622c0438360 2024-01-26T23:30:16.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.487+0000 7f8194542b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c0451360 con 0x5622c03f0010 2024-01-26T23:30:16.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.487+0000 7f8194542b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5622c0455d80 con 0x5622c03ef5d0 2024-01-26T23:30:16.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.487+0000 7f8194542b80 10 client.0 _ll_register_callbacks cb 0x5622c014c7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:30:16.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.487+0000 7f8188fdb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 0x5622c0498f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.487+0000 7f81897dc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c0438360 0x5622c03ea270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.487+0000 7f8188fdb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 0x5622c0498f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:37518/0 (socket says 172.21.15.46:37518) 2024-01-26T23:30:16.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.487+0000 7f8188fdb700 1 -- 172.21.15.46:0/4025978387 learned_addr learned my addr 172.21.15.46:0/4025978387 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:16.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.487+0000 7f8183fff700 1 --2- 172.21.15.46:0/4025978387 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c03f0010 0x5622c03e9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.487+0000 7f81897dc700 1 -- 172.21.15.46:0/4025978387 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 msgr2=0x5622c0498f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:16.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.487+0000 7f81897dc700 1 --2- 172.21.15.46:0/4025978387 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 0x5622c0498f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.487+0000 7f81897dc700 1 -- 172.21.15.46:0/4025978387 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c03f0010 msgr2=0x5622c03e9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:16.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.487+0000 7f81897dc700 1 --2- 172.21.15.46:0/4025978387 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c03f0010 0x5622c03e9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.487+0000 7f81897dc700 1 -- 172.21.15.46:0/4025978387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f817c01f040 con 0x5622c0438360 2024-01-26T23:30:16.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.488+0000 7f81897dc700 1 --2- 172.21.15.46:0/4025978387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c0438360 0x5622c03ea270 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f817c026000 tx=0x7f817c024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3696e4287e5f6880 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:16.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.488+0000 7f8162ffd700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:30:16.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.488+0000 7f8162ffd700 1 -- 172.21.15.46:0/4025978387 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c00e040 con 0x5622c0438360 2024-01-26T23:30:16.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.488+0000 7f8162ffd700 1 -- 172.21.15.46:0/4025978387 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f817c011d90 con 0x5622c0438360 2024-01-26T23:30:16.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.488+0000 7f8194542b80 1 -- 172.21.15.46:0/4025978387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8178069070 con 0x5622c0438360 2024-01-26T23:30:16.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.488+0000 7f8162ffd700 1 -- 172.21.15.46:0/4025978387 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f817c025640 con 0x5622c0438360 2024-01-26T23:30:16.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.488+0000 7f8194542b80 10 client.4731 fetch_fsmap learned FSMap version 9 2024-01-26T23:30:16.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.488+0000 7f8194542b80 1 -- 172.21.15.46:0/4025978387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x5622c017bfe0 con 0x5622c0438360 2024-01-26T23:30:16.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.488+0000 7f8162ffd700 1 -- 172.21.15.46:0/4025978387 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8178069070 con 0x5622c0438360 2024-01-26T23:30:16.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.489+0000 7f8162ffd700 1 -- 172.21.15.46:0/4025978387 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 9) v1 ==== 3108+0+0 (secure 0 0 0) 0x7f817c017020 con 0x5622c0438360 2024-01-26T23:30:16.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.489+0000 7f8194542b80 10 client.4731 fetch_fsmap finished waiting for FSMap version 9 2024-01-26T23:30:16.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.489+0000 7f8194542b80 10 client.4731 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-26T23:30:16.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.489+0000 7f8194542b80 20 client.4731 populate_metadata read hostname 'smithi046' 2024-01-26T23:30:16.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.489+0000 7f8194542b80 1 --2- 172.21.15.46:0/4025978387 >> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] conn(0x5622c049cb70 0x5622c049cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.489+0000 7f8194542b80 4 client.4731 mds_command: new command op to 4254 tid=0[{"prefix": "status"}] 2024-01-26T23:30:16.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.489+0000 7f8194542b80 1 -- 172.21.15.46:0/4025978387 --> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5622c02e0c00 con 0x5622c049cb70 2024-01-26T23:30:16.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.489+0000 7f8188fdb700 1 --2- 172.21.15.46:0/4025978387 >> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] conn(0x5622c049cb70 0x5622c049cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.489+0000 7f8162ffd700 10 client.4731 ms_handle_connect on v2:172.21.15.46:6834/688086235 2024-01-26T23:30:16.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.489+0000 7f8188fdb700 1 --2- 172.21.15.46:0/4025978387 >> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] conn(0x5622c049cb70 0x5622c049cf40 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2f257b57cf2a3d5 server_cookie=2bbe981baceeba9e in_seq=0 out_seq=0 2024-01-26T23:30:16.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.489+0000 7f8162ffd700 1 -- 172.21.15.46:0/4025978387 <== mds.0 v2:172.21.15.46:6834/688086235 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x5622c02e0c00 con 0x5622c049cb70 2024-01-26T23:30:16.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.489+0000 7f8162ffd700 10 client.4731 handle_command_reply: tid=0 2024-01-26T23:30:16.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.489+0000 7f8194542b80 1 client.4731 shutdown 2024-01-26T23:30:16.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f8194542b80 1 -- 172.21.15.46:0/4025978387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c0438360 msgr2=0x5622c03ea270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:16.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f8194542b80 1 --2- 172.21.15.46:0/4025978387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c0438360 0x5622c03ea270 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f817c026000 tx=0x7f817c024000 comp rx=0 tx=0).stop 2024-01-26T23:30:16.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f8194542b80 1 -- 172.21.15.46:0/4025978387 shutdown_connections 2024-01-26T23:30:16.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f8194542b80 1 --2- 172.21.15.46:0/4025978387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622c0438360 0x5622c03ea270 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f8194542b80 1 --2- 172.21.15.46:0/4025978387 >> [v2:172.21.15.46:6834/688086235,v1:172.21.15.46:6835/688086235] conn(0x5622c049cb70 0x5622c049cf40 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f8194542b80 1 --2- 172.21.15.46:0/4025978387 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622c03ef5d0 0x5622c0498f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f8194542b80 1 --2- 172.21.15.46:0/4025978387 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622c03f0010 0x5622c03e9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f8162ffd700 0 client.4731 ms_handle_reset on v2:172.21.15.46:6834/688086235 2024-01-26T23:30:16.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f8194542b80 1 -- 172.21.15.46:0/4025978387 >> 172.21.15.46:0/4025978387 conn(0x5622c03efa70 msgr2=0x5622c042fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:16.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f8194542b80 1 -- 172.21.15.46:0/4025978387 shutdown_connections 2024-01-26T23:30:16.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f8194542b80 1 -- 172.21.15.46:0/4025978387 wait complete. 2024-01-26T23:30:16.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f8194542b80 20 client.4731 trim_cache size 0 max 16384 2024-01-26T23:30:16.497 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:30:16.497 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-26T23:30:16.497 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-26T23:30:16.497 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4254, 2024-01-26T23:30:16.497 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-26T23:30:16.497 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-26T23:30:16.497 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-01-26T23:30:16.497 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 31.887970189000001, 2024-01-26T23:30:16.497 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 9, 2024-01-26T23:30:16.497 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 24, 2024-01-26T23:30:16.497 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 24, 2024-01-26T23:30:16.497 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 38.569712885999998 2024-01-26T23:30:16.497 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:30:16.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f818b23f700 1 -- 172.21.15.46:0/1204543396 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8168066f30 msgr2=0x7f81680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:16.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f818b23f700 1 --2- 172.21.15.46:0/1204543396 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8168066f30 0x7f81680693e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f81840c26e0 tx=0x7f81700091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f818b23f700 1 -- 172.21.15.46:0/1204543396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f818406cbb0 msgr2=0x7f8184140e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:16.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f818b23f700 1 --2- 172.21.15.46:0/1204543396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f818406cbb0 0x7f8184140e70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f817800c590 tx=0x7f8178007e00 comp rx=0 tx=0).stop 2024-01-26T23:30:16.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f818b23f700 1 -- 172.21.15.46:0/1204543396 shutdown_connections 2024-01-26T23:30:16.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f818b23f700 1 --2- 172.21.15.46:0/1204543396 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8168066f30 0x7f81680693e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.490+0000 7f818b23f700 1 --2- 172.21.15.46:0/1204543396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f818406cbb0 0x7f8184140e70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.491+0000 7f818b23f700 1 --2- 172.21.15.46:0/1204543396 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81841561d0 0x7f81841524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.491+0000 7f818b23f700 1 --2- 172.21.15.46:0/1204543396 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81840b0280 0x7f8184165360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.491+0000 7f818b23f700 1 -- 172.21.15.46:0/1204543396 >> 172.21.15.46:0/1204543396 conn(0x7f818406b630 msgr2=0x7f8184067930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:16.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.491+0000 7f818b23f700 1 -- 172.21.15.46:0/1204543396 shutdown_connections 2024-01-26T23:30:16.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.491+0000 7f818b23f700 1 -- 172.21.15.46:0/1204543396 wait complete. 2024-01-26T23:30:16.505 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-01-26T23:30:16.516 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-01-26T23:30:16.516 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-01-26T23:30:16.516 INFO:tasks.ceph_fuse:client.0 config is {} 2024-01-26T23:30:16.517 INFO:tasks.ceph_fuse:client.2 config is {} 2024-01-26T23:30:16.517 INFO:tasks.ceph_fuse:client.1 config is {} 2024-01-26T23:30:16.517 INFO:tasks.ceph_fuse:client.3 config is {} 2024-01-26T23:30:16.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:16.517 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-26T23:30:16.536 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-01-26T23:30:16.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:16.537 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-01-26T23:30:16.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:16.617 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link delete ceph-brx 2024-01-26T23:30:16.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:16.727 DEBUG:teuthology.orchestra.run.smithi106:> ip netns list 2024-01-26T23:30:16.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:16.748 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link delete ceph-brx 2024-01-26T23:30:16.823 INFO:teuthology.orchestra.run.smithi106.stderr:Cannot find device "ceph-brx" 2024-01-26T23:30:16.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:30:16.826 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-01-26T23:30:16.826 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-26T23:30:16.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.970+0000 7fea6a787700 1 Processor -- start 2024-01-26T23:30:16.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.970+0000 7fea6a787700 1 -- start start 2024-01-26T23:30:16.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.971+0000 7fea6a787700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fea64159590 0x7fea6415d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.971+0000 7fea6a787700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea6415d8f0 0x7fea6406bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.971+0000 7fea6a787700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fea64159960 0x7fea6406c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.971+0000 7fea6a787700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea6415e350 con 0x7fea64159960 2024-01-26T23:30:16.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.971+0000 7fea6a787700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea640c97c0 con 0x7fea64159590 2024-01-26T23:30:16.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.971+0000 7fea6a787700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6415dd90 con 0x7fea6415d8f0 2024-01-26T23:30:16.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.971+0000 7fea68d24700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fea64159960 0x7fea6406c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.971+0000 7fea68d24700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fea64159960 0x7fea6406c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40478/0 (socket says 172.21.15.46:40478) 2024-01-26T23:30:16.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.971+0000 7fea68d24700 1 -- 172.21.15.46:0/3352551242 learned_addr learned my addr 172.21.15.46:0/3352551242 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:16.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.971+0000 7fea637fe700 1 --2- 172.21.15.46:0/3352551242 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea6415d8f0 0x7fea6406bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.971+0000 7fea63fff700 1 --2- 172.21.15.46:0/3352551242 >> v2:172.21.15.106:3300/0 conn(0x7fea64159590 0x7fea6415d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.971+0000 7fea68d24700 1 -- 172.21.15.46:0/3352551242 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea6415d8f0 msgr2=0x7fea6406bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:16.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.971+0000 7fea68d24700 1 --2- 172.21.15.46:0/3352551242 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea6415d8f0 0x7fea6406bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.971+0000 7fea68d24700 1 -- 172.21.15.46:0/3352551242 >> v2:172.21.15.106:3300/0 conn(0x7fea64159590 msgr2=0x7fea6415d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:16.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.971+0000 7fea68d24700 1 --2- 172.21.15.46:0/3352551242 >> v2:172.21.15.106:3300/0 conn(0x7fea64159590 0x7fea6415d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.971+0000 7fea68d24700 1 -- 172.21.15.46:0/3352551242 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6415e070 con 0x7fea64159960 2024-01-26T23:30:16.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.971+0000 7fea68d24700 1 --2- 172.21.15.46:0/3352551242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea64159960 0x7fea6406c1f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fea640c1f60 tx=0x7fea5c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d9b2e1cb775f478 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:16.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.972+0000 7fea62ffd700 1 -- 172.21.15.46:0/3352551242 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c00e070 con 0x7fea64159960 2024-01-26T23:30:16.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.972+0000 7fea62ffd700 1 -- 172.21.15.46:0/3352551242 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5c00ba40 con 0x7fea64159960 2024-01-26T23:30:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.972+0000 7fea62ffd700 1 -- 172.21.15.46:0/3352551242 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c0092e0 con 0x7fea64159960 2024-01-26T23:30:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.972+0000 7fea6a787700 1 -- 172.21.15.46:0/3352551242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea64159960 msgr2=0x7fea6406c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.972+0000 7fea6a787700 1 --2- 172.21.15.46:0/3352551242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea64159960 0x7fea6406c1f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fea640c1f60 tx=0x7fea5c009840 comp rx=0 tx=0).stop 2024-01-26T23:30:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.972+0000 7fea6a787700 1 -- 172.21.15.46:0/3352551242 shutdown_connections 2024-01-26T23:30:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.972+0000 7fea6a787700 1 --2- 172.21.15.46:0/3352551242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea64159960 0x7fea6406c1f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.972+0000 7fea6a787700 1 --2- 172.21.15.46:0/3352551242 >> v2:172.21.15.106:3300/0 conn(0x7fea64159590 0x7fea6415d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.972+0000 7fea6a787700 1 --2- 172.21.15.46:0/3352551242 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea6415d8f0 0x7fea6406bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.972+0000 7fea6a787700 1 -- 172.21.15.46:0/3352551242 >> 172.21.15.46:0/3352551242 conn(0x7fea640ce530 msgr2=0x7fea640ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.972+0000 7fea6a787700 1 -- 172.21.15.46:0/3352551242 shutdown_connections 2024-01-26T23:30:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.972+0000 7fea6a787700 1 -- 172.21.15.46:0/3352551242 wait complete. 2024-01-26T23:30:16.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.973+0000 7fea6a787700 1 Processor -- start 2024-01-26T23:30:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.974+0000 7fea6a787700 1 -- start start 2024-01-26T23:30:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.974+0000 7fea6a787700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea64159590 0x7fea640c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea6a787700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea64159960 0x7fea64149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea6a787700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea6415d8f0 0x7fea641499b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea6a787700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea640b12e0 con 0x7fea64159590 2024-01-26T23:30:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea6a787700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea64149ef0 con 0x7fea64159960 2024-01-26T23:30:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea6a787700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6416ad20 con 0x7fea6415d8f0 2024-01-26T23:30:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea63fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea64159590 0x7fea640c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea63fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea64159590 0x7fea640c6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40484/0 (socket says 172.21.15.46:40484) 2024-01-26T23:30:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea63fff700 1 -- 172.21.15.46:0/2405080431 learned_addr learned my addr 172.21.15.46:0/2405080431 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea68d24700 1 --2- 172.21.15.46:0/2405080431 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea6415d8f0 0x7fea641499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea637fe700 1 --2- 172.21.15.46:0/2405080431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea64159960 0x7fea64149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea63fff700 1 -- 172.21.15.46:0/2405080431 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea6415d8f0 msgr2=0x7fea641499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:16.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea63fff700 1 --2- 172.21.15.46:0/2405080431 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea6415d8f0 0x7fea641499b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea63fff700 1 -- 172.21.15.46:0/2405080431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea64159960 msgr2=0x7fea64149470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:16.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea63fff700 1 --2- 172.21.15.46:0/2405080431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea64159960 0x7fea64149470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:16.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea63fff700 1 -- 172.21.15.46:0/2405080431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c009ae0 con 0x7fea64159590 2024-01-26T23:30:16.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea63fff700 1 --2- 172.21.15.46:0/2405080431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea64159590 0x7fea640c6d90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fea5400d890 tx=0x7fea54008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:16.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea4bfff700 1 -- 172.21.15.46:0/2405080431 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea540125f0 con 0x7fea64159590 2024-01-26T23:30:16.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea4bfff700 1 -- 172.21.15.46:0/2405080431 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea54012750 con 0x7fea64159590 2024-01-26T23:30:16.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.975+0000 7fea4bfff700 1 -- 172.21.15.46:0/2405080431 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea54012af0 con 0x7fea64159590 2024-01-26T23:30:16.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.976+0000 7fea6a787700 1 -- 172.21.15.46:0/2405080431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea640b14f0 con 0x7fea64159590 2024-01-26T23:30:16.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.976+0000 7fea6a787700 1 -- 172.21.15.46:0/2405080431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea640b1b30 con 0x7fea64159590 2024-01-26T23:30:16.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.977+0000 7fea6a787700 1 -- 172.21.15.46:0/2405080431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea640b1e20 con 0x7fea64159590 2024-01-26T23:30:16.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.977+0000 7fea4bfff700 1 -- 172.21.15.46:0/2405080431 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fea5401e460 con 0x7fea64159590 2024-01-26T23:30:16.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.977+0000 7fea4bfff700 1 --2- 172.21.15.46:0/2405080431 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fea50066f30 0x7fea500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:16.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.977+0000 7fea4bfff700 1 -- 172.21.15.46:0/2405080431 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fea54084d90 con 0x7fea64159590 2024-01-26T23:30:16.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.978+0000 7fea637fe700 1 --2- 172.21.15.46:0/2405080431 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fea50066f30 0x7fea500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:16.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.978+0000 7fea637fe700 1 --2- 172.21.15.46:0/2405080431 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fea50066f30 0x7fea500693e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fea640c8690 tx=0x7fea58008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:16.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:16.982+0000 7fea4bfff700 1 -- 172.21.15.46:0/2405080431 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fea54051d10 con 0x7fea64159590 2024-01-26T23:30:17.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:17.235+0000 7fea6a787700 1 -- 172.21.15.46:0/2405080431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fea6404c710 con 0x7fea64159590 2024-01-26T23:30:17.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:17.235+0000 7fea4bfff700 1 -- 172.21.15.46:0/2405080431 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7fea54055330 con 0x7fea64159590 2024-01-26T23:30:17.236 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-01-26T23:30:17.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:17.238+0000 7fea6a787700 1 -- 172.21.15.46:0/2405080431 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fea50066f30 msgr2=0x7fea500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:17.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:17.238+0000 7fea6a787700 1 --2- 172.21.15.46:0/2405080431 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fea50066f30 0x7fea500693e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fea640c8690 tx=0x7fea58008040 comp rx=0 tx=0).stop 2024-01-26T23:30:17.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:17.238+0000 7fea6a787700 1 -- 172.21.15.46:0/2405080431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea64159590 msgr2=0x7fea640c6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:17.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:17.238+0000 7fea6a787700 1 --2- 172.21.15.46:0/2405080431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea64159590 0x7fea640c6d90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fea5400d890 tx=0x7fea54008040 comp rx=0 tx=0).stop 2024-01-26T23:30:17.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:17.238+0000 7fea6a787700 1 -- 172.21.15.46:0/2405080431 shutdown_connections 2024-01-26T23:30:17.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:17.238+0000 7fea6a787700 1 --2- 172.21.15.46:0/2405080431 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fea50066f30 0x7fea500693e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:17.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:17.238+0000 7fea6a787700 1 --2- 172.21.15.46:0/2405080431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea64159590 0x7fea640c6d90 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:17.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:17.238+0000 7fea6a787700 1 --2- 172.21.15.46:0/2405080431 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea6415d8f0 0x7fea641499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:17.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:17.238+0000 7fea6a787700 1 --2- 172.21.15.46:0/2405080431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea64159960 0x7fea64149470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:17.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:17.238+0000 7fea6a787700 1 -- 172.21.15.46:0/2405080431 >> 172.21.15.46:0/2405080431 conn(0x7fea640ce530 msgr2=0x7fea641589b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:17.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:17.238+0000 7fea6a787700 1 -- 172.21.15.46:0/2405080431 shutdown_connections 2024-01-26T23:30:17.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:17.238+0000 7fea6a787700 1 -- 172.21.15.46:0/2405080431 wait complete. 2024-01-26T23:30:17.252 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-26T23:30:17.252 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-26T23:30:17.252 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-26T23:30:17.252 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-26T23:30:17.252 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-26T23:30:17.252 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-26T23:30:17.252 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-26T23:30:17.252 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-26T23:30:17.253 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:30:17.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:17.253 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-26T23:30:17.272 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-26T23:30:17.272 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-26T23:30:17.272 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-26T23:30:17.272 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:17.272 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-26T23:30:17.272 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:17.272 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:30:17.273 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:17.273 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-26T23:30:17.273 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:30:17.273 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:17.273 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-26T23:30:17.273 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-26T23:30:17.273 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:17.273 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-26T23:30:17.273 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-26T23:30:17.273 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42313sec preferred_lft 42313sec 2024-01-26T23:30:17.273 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-26T23:30:17.273 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:17.273 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:30:17.273 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:17.274 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-26T23:30:17.274 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-26T23:30:17.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:17.274 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:30:17.274 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:30:17.274 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add name ceph-brx type bridge 2024-01-26T23:30:17.274 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr flush dev ceph-brx 2024-01-26T23:30:17.274 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set ceph-brx up 2024-01-26T23:30:17.274 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-26T23:30:17.274 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:30:17.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:17.408 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-26T23:30:17.439 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-26T23:30:17.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:17.441 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-26T23:30:17.503 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-26T23:30:17.503 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-26T23:30:17.503 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-26T23:30:17.503 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-26T23:30:17.504 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-26T23:30:17.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:17.504 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:30:17.504 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:30:17.504 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-26T23:30:17.504 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-26T23:30:17.504 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-26T23:30:17.504 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:30:17.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:17.629 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-26T23:30:17.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:17.650 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list-id 2024-01-26T23:30:17.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:17.713 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:30:17.713 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:30:17.713 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-26T23:30:17.713 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-01-26T23:30:17.713 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:30:17.824 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-01-26T23:30:17.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:17.824 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:30:17.824 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:30:17.824 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-01-26T23:30:17.824 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-26T23:30:17.824 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-01-26T23:30:17.824 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-01-26T23:30:17.824 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-01-26T23:30:17.824 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:30:18.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:18.034 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:30:18.034 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:30:18.034 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-01-26T23:30:18.034 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set dev brx.0 master ceph-brx 2024-01-26T23:30:18.034 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:30:18.145 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-26T23:30:18.145 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:30:18.145 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-26T23:30:18.164 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:30:18.165 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:30:18.165 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:30:18.225 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:30:18.301 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:30:18.301 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:30:18.376 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:30:18.377 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:30:18.378 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:18.378 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:30:18.441 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-26T23:30:18.441 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-26T23:30:18.484 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:30:18.516 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:30:18.517 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:30:18.584 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-01-26T23:30:18.583+0000 7f2170df5540 -1 init, newargv = 0x560e61572100 newargc=15 2024-01-26T23:30:18.584 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[54063]: starting ceph client 2024-01-26T23:30:18.588 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:30:18.589 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[54063]: starting fuse 2024-01-26T23:30:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:30:18.590 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:18.590 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:30:18.650 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-26T23:30:18.650 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-26T23:30:18.650 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-26T23:30:18.650 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-26T23:30:18.650 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-26T23:30:18.650 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-26T23:30:18.651 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-26T23:30:18.651 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:30:18.651 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-26T23:30:18.651 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-26T23:30:18.651 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-26T23:30:18.651 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-26T23:30:18.651 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:30:18.651 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-26T23:30:18.651 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-26T23:30:18.651 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-26T23:30:18.651 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-26T23:30:18.651 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:30:18.651 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-26T23:30:18.651 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-26T23:30:18.651 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-26T23:30:18.651 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-26T23:30:18.651 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-26T23:30:18.652 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-26T23:30:18.652 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-26T23:30:18.652 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-26T23:30:18.652 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:30:18.652 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-01-26T23:30:18.652 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-26T23:30:18.792 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.54063.asok 2024-01-26T23:30:18.973 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.54063.asok 2024-01-26T23:30:18.974 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:18.974 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.54063.asok status 2024-01-26T23:30:19.149 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:30:19.150 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-26T23:30:19.150 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-26T23:30:19.150 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-26T23:30:19.150 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-01-26T23:30:19.150 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-26T23:30:19.150 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-26T23:30:19.150 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "54063", 2024-01-26T23:30:19.150 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-26T23:30:19.150 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:19.150 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-26T23:30:19.150 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-26T23:30:19.150 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4743, 2024-01-26T23:30:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-26T23:30:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-26T23:30:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-26T23:30:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4743 2024-01-26T23:30:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:30:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:30:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:30:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2987245562 2024-01-26T23:30:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:30:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:30:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:30:19.152 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2987245562 2024-01-26T23:30:19.152 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:19.152 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.4743 192.168.0.1:0/2987245562", 2024-01-26T23:30:19.152 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/2987245562", 2024-01-26T23:30:19.152 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-26T23:30:19.152 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 9, 2024-01-26T23:30:19.152 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 24, 2024-01-26T23:30:19.152 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-26T23:30:19.152 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-26T23:30:19.153 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-26T23:30:19.153 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:30:19.161 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-26T23:30:19.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.291+0000 7fbd78af7700 1 Processor -- start 2024-01-26T23:30:19.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.291+0000 7fbd78af7700 1 -- start start 2024-01-26T23:30:19.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.292+0000 7fbd78af7700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbd74159760 0x7fbd740b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:19.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.292+0000 7fbd78af7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740b3060 0x7fbd740b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:19.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.292+0000 7fbd78af7700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd7406f940 con 0x7fbd74159b30 2024-01-26T23:30:19.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.292+0000 7fbd78af7700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd7406ed30 con 0x7fbd74159760 2024-01-26T23:30:19.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.292+0000 7fbd78af7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7406f010 con 0x7fbd740b3060 2024-01-26T23:30:19.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.292+0000 7fbd72d9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbd74159b30 0x7fbd7415d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50498/0 (socket says 172.21.15.46:50498) 2024-01-26T23:30:19.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.292+0000 7fbd72d9d700 1 -- 172.21.15.46:0/790959525 learned_addr learned my addr 172.21.15.46:0/790959525 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:19.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.292+0000 7fbd69d9b700 1 --2- 172.21.15.46:0/790959525 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740b3060 0x7fbd740b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:19.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.292+0000 7fbd7259c700 1 --2- 172.21.15.46:0/790959525 >> v2:172.21.15.106:3300/0 conn(0x7fbd74159760 0x7fbd740b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:19.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.292+0000 7fbd71d9b700 1 -- 172.21.15.46:0/790959525 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464581859 0 0) 0x7fbd7406f940 con 0x7fbd74159b30 2024-01-26T23:30:19.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.292+0000 7fbd71d9b700 1 -- 172.21.15.46:0/790959525 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd60003570 con 0x7fbd74159b30 2024-01-26T23:30:19.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd71d9b700 1 -- 172.21.15.46:0/790959525 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2317008087 0 0) 0x7fbd60003570 con 0x7fbd74159b30 2024-01-26T23:30:19.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd71d9b700 1 -- 172.21.15.46:0/790959525 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740b3060 msgr2=0x7fbd740b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:19.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd71d9b700 1 --2- 172.21.15.46:0/790959525 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740b3060 0x7fbd740b0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:19.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd71d9b700 1 -- 172.21.15.46:0/790959525 >> v2:172.21.15.106:3300/0 conn(0x7fbd74159760 msgr2=0x7fbd740b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:19.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd71d9b700 1 --2- 172.21.15.46:0/790959525 >> v2:172.21.15.106:3300/0 conn(0x7fbd74159760 0x7fbd740b2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:19.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd71d9b700 1 -- 172.21.15.46:0/790959525 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7406f2f0 con 0x7fbd74159b30 2024-01-26T23:30:19.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd71d9b700 1 -- 172.21.15.46:0/790959525 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fbd6c0017f0 con 0x7fbd74159b30 2024-01-26T23:30:19.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd69d9b700 1 --2- 172.21.15.46:0/790959525 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740b3060 0x7fbd740b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:19.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd7259c700 1 --2- 172.21.15.46:0/790959525 >> v2:172.21.15.106:3300/0 conn(0x7fbd74159760 0x7fbd740b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:19.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd71d9b700 1 -- 172.21.15.46:0/790959525 >> v1:172.21.15.46:6789/0 conn(0x7fbd74159b30 legacy=0x7fbd7415d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:19.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd71d9b700 1 --2- 172.21.15.46:0/790959525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd60003a60 0x7fbd60003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:19.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd71d9b700 1 --2- 172.21.15.46:0/790959525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd60004810 0x7fbd60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:19.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd71d9b700 1 --2- 172.21.15.46:0/790959525 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740b3060 0x7fbd600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:19.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd71d9b700 1 -- 172.21.15.46:0/790959525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7406ed30 con 0x7fbd60003a60 2024-01-26T23:30:19.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd71d9b700 1 -- 172.21.15.46:0/790959525 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7406f010 con 0x7fbd60004810 2024-01-26T23:30:19.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd71d9b700 1 -- 172.21.15.46:0/790959525 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6000bab0 con 0x7fbd740b3060 2024-01-26T23:30:19.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd7259c700 1 --2- 172.21.15.46:0/790959525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd60003a60 0x7fbd60003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:19.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd69d9b700 1 --2- 172.21.15.46:0/790959525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd60004810 0x7fbd60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:19.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd72d9d700 1 --2- 172.21.15.46:0/790959525 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740b3060 0x7fbd600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:19.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd7259c700 1 -- 172.21.15.46:0/790959525 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740b3060 msgr2=0x7fbd600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:19.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd7259c700 1 --2- 172.21.15.46:0/790959525 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740b3060 0x7fbd600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:19.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd7259c700 1 -- 172.21.15.46:0/790959525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd60004810 msgr2=0x7fbd60004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:19.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd7259c700 1 --2- 172.21.15.46:0/790959525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd60004810 0x7fbd60004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:19.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd7259c700 1 -- 172.21.15.46:0/790959525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c009420 con 0x7fbd60003a60 2024-01-26T23:30:19.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd69d9b700 1 --2- 172.21.15.46:0/790959525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd60004810 0x7fbd60004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:19.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd7259c700 1 --2- 172.21.15.46:0/790959525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd60003a60 0x7fbd60003e70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbd64009df0 tx=0x7fbd640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9df4c94debf1d82 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:19.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.293+0000 7fbd71d9b700 1 -- 172.21.15.46:0/790959525 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6400e920 con 0x7fbd60003a60 2024-01-26T23:30:19.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.294+0000 7fbd71d9b700 1 -- 172.21.15.46:0/790959525 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6400ea80 con 0x7fbd60003a60 2024-01-26T23:30:19.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.294+0000 7fbd71d9b700 1 -- 172.21.15.46:0/790959525 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6400ee20 con 0x7fbd60003a60 2024-01-26T23:30:19.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.294+0000 7fbd78af7700 1 -- 172.21.15.46:0/790959525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd60003a60 msgr2=0x7fbd60003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:19.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.294+0000 7fbd78af7700 1 --2- 172.21.15.46:0/790959525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd60003a60 0x7fbd60003e70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbd64009df0 tx=0x7fbd640091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:19.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.294+0000 7fbd72d9d700 1 -- 172.21.15.46:0/790959525 reap_dead start 2024-01-26T23:30:19.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.294+0000 7fbd78af7700 1 -- 172.21.15.46:0/790959525 shutdown_connections 2024-01-26T23:30:19.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.294+0000 7fbd78af7700 1 -- 172.21.15.46:0/790959525 >> 172.21.15.46:0/790959525 conn(0x7fbd74069510 msgr2=0x7fbd740aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:19.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.294+0000 7fbd78af7700 1 -- 172.21.15.46:0/790959525 shutdown_connections 2024-01-26T23:30:19.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.294+0000 7fbd78af7700 1 -- 172.21.15.46:0/790959525 wait complete. 2024-01-26T23:30:19.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.295+0000 7fbd78af7700 1 Processor -- start 2024-01-26T23:30:19.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.296+0000 7fbd78af7700 1 -- start start 2024-01-26T23:30:19.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.296+0000 7fbd78af7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd60003a60 0x7fbd7414e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:19.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.296+0000 7fbd78af7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd7415a2a0 0x7fbd7415c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:19.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.296+0000 7fbd78af7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd74150f10 0x7fbd7415cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:19.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.296+0000 7fbd78af7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7415e500 con 0x7fbd60003a60 2024-01-26T23:30:19.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.296+0000 7fbd78af7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd741514d0 con 0x7fbd7415a2a0 2024-01-26T23:30:19.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.296+0000 7fbd78af7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd741517b0 con 0x7fbd74150f10 2024-01-26T23:30:19.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.296+0000 7fbd7259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd60003a60 0x7fbd7414e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:19.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.296+0000 7fbd7259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd60003a60 0x7fbd7414e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40508/0 (socket says 172.21.15.46:40508) 2024-01-26T23:30:19.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.296+0000 7fbd72d9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd74150f10 0x7fbd7415cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:19.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.296+0000 7fbd7259c700 1 -- 172.21.15.46:0/2046184329 learned_addr learned my addr 172.21.15.46:0/2046184329 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:19.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.296+0000 7fbd69d9b700 1 --2- 172.21.15.46:0/2046184329 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd7415a2a0 0x7fbd7415c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:19.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.297+0000 7fbd7259c700 1 -- 172.21.15.46:0/2046184329 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd74150f10 msgr2=0x7fbd7415cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:19.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.297+0000 7fbd7259c700 1 --2- 172.21.15.46:0/2046184329 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd74150f10 0x7fbd7415cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:19.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.297+0000 7fbd7259c700 1 -- 172.21.15.46:0/2046184329 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd7415a2a0 msgr2=0x7fbd7415c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:19.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.297+0000 7fbd7259c700 1 --2- 172.21.15.46:0/2046184329 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd7415a2a0 0x7fbd7415c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:19.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.297+0000 7fbd7259c700 1 -- 172.21.15.46:0/2046184329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd74151a90 con 0x7fbd60003a60 2024-01-26T23:30:19.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.297+0000 7fbd72d9d700 1 --2- 172.21.15.46:0/2046184329 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd74150f10 0x7fbd7415cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:19.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.297+0000 7fbd7259c700 1 --2- 172.21.15.46:0/2046184329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd60003a60 0x7fbd7414e090 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbd60004760 tx=0x7fbd6401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:19.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.297+0000 7fbd6affd700 1 -- 172.21.15.46:0/2046184329 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd64022070 con 0x7fbd60003a60 2024-01-26T23:30:19.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.297+0000 7fbd6affd700 1 -- 172.21.15.46:0/2046184329 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd64013bb0 con 0x7fbd60003a60 2024-01-26T23:30:19.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.297+0000 7fbd78af7700 1 -- 172.21.15.46:0/2046184329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd74151d70 con 0x7fbd60003a60 2024-01-26T23:30:19.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.297+0000 7fbd6affd700 1 -- 172.21.15.46:0/2046184329 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6401c450 con 0x7fbd60003a60 2024-01-26T23:30:19.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.297+0000 7fbd78af7700 1 -- 172.21.15.46:0/2046184329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd7414ede0 con 0x7fbd60003a60 2024-01-26T23:30:19.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.298+0000 7fbd78af7700 1 -- 172.21.15.46:0/2046184329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd7415e7e0 con 0x7fbd60003a60 2024-01-26T23:30:19.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.299+0000 7fbd6affd700 1 -- 172.21.15.46:0/2046184329 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbd64026070 con 0x7fbd60003a60 2024-01-26T23:30:19.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.299+0000 7fbd6affd700 1 --2- 172.21.15.46:0/2046184329 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbd54066f30 0x7fbd540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:19.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.299+0000 7fbd6affd700 1 -- 172.21.15.46:0/2046184329 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fbd64085e70 con 0x7fbd60003a60 2024-01-26T23:30:19.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.299+0000 7fbd69d9b700 1 --2- 172.21.15.46:0/2046184329 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbd54066f30 0x7fbd540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:19.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.300+0000 7fbd69d9b700 1 --2- 172.21.15.46:0/2046184329 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbd54066f30 0x7fbd540693e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbd741605a0 tx=0x7fbd5c0091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:19.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.304+0000 7fbd6affd700 1 -- 172.21.15.46:0/2046184329 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbd64052df0 con 0x7fbd60003a60 2024-01-26T23:30:19.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.558+0000 7fbd78af7700 1 -- 172.21.15.46:0/2046184329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbd7404c710 con 0x7fbd60003a60 2024-01-26T23:30:19.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.558+0000 7fbd6affd700 1 -- 172.21.15.46:0/2046184329 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7fbd64017070 con 0x7fbd60003a60 2024-01-26T23:30:19.559 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-01-26T23:30:19.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.560+0000 7fbd78af7700 1 -- 172.21.15.46:0/2046184329 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbd54066f30 msgr2=0x7fbd540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:19.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.560+0000 7fbd78af7700 1 --2- 172.21.15.46:0/2046184329 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbd54066f30 0x7fbd540693e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbd741605a0 tx=0x7fbd5c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:19.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.561+0000 7fbd78af7700 1 -- 172.21.15.46:0/2046184329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd60003a60 msgr2=0x7fbd7414e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:19.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.561+0000 7fbd78af7700 1 --2- 172.21.15.46:0/2046184329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd60003a60 0x7fbd7414e090 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbd60004760 tx=0x7fbd6401a000 comp rx=0 tx=0).stop 2024-01-26T23:30:19.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.561+0000 7fbd78af7700 1 -- 172.21.15.46:0/2046184329 shutdown_connections 2024-01-26T23:30:19.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.561+0000 7fbd78af7700 1 --2- 172.21.15.46:0/2046184329 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbd54066f30 0x7fbd540693e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:19.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.561+0000 7fbd78af7700 1 --2- 172.21.15.46:0/2046184329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd60003a60 0x7fbd7414e090 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:19.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.561+0000 7fbd78af7700 1 --2- 172.21.15.46:0/2046184329 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd74150f10 0x7fbd7415cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:19.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.561+0000 7fbd78af7700 1 --2- 172.21.15.46:0/2046184329 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd7415a2a0 0x7fbd7415c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:19.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.561+0000 7fbd78af7700 1 -- 172.21.15.46:0/2046184329 >> 172.21.15.46:0/2046184329 conn(0x7fbd74069510 msgr2=0x7fbd741557b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:19.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.561+0000 7fbd78af7700 1 -- 172.21.15.46:0/2046184329 shutdown_connections 2024-01-26T23:30:19.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:19.561+0000 7fbd78af7700 1 -- 172.21.15.46:0/2046184329 wait complete. 2024-01-26T23:30:19.575 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-26T23:30:19.575 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-26T23:30:19.575 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-26T23:30:19.575 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-01-26T23:30:19.575 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-26T23:30:19.575 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-26T23:30:19.575 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-26T23:30:19.575 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-26T23:30:19.575 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-01-26T23:30:19.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:19.575 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-26T23:30:19.595 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-26T23:30:19.596 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-26T23:30:19.596 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-26T23:30:19.596 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:19.596 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-26T23:30:19.596 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:19.597 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:30:19.597 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:19.597 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-26T23:30:19.597 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:30:19.597 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:19.597 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-26T23:30:19.597 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-26T23:30:19.597 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:19.598 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-26T23:30:19.598 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-26T23:30:19.598 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42311sec preferred_lft 42311sec 2024-01-26T23:30:19.598 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-26T23:30:19.598 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:19.598 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:30:19.598 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:19.598 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-26T23:30:19.599 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-26T23:30:19.599 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 76:ef:90:4c:9d:6f brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:19.599 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-26T23:30:19.599 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:19.599 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::cce6:a7ff:fe4f:38d/64 scope link 2024-01-26T23:30:19.599 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:19.599 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:30:19.599 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 76:ef:90:4c:9d:6f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-26T23:30:19.600 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::74ef:90ff:fe4c:9d6f/64 scope link tentative 2024-01-26T23:30:19.600 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:19.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:19.600 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-26T23:30:19.666 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-26T23:30:19.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:19.668 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-26T23:30:19.733 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-26T23:30:19.733 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-26T23:30:19.733 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-26T23:30:19.733 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-26T23:30:19.733 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-26T23:30:19.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:19.734 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:30:19.734 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:30:19.734 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-26T23:30:19.734 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-26T23:30:19.734 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-26T23:30:19.734 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:30:19.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:19.858 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-26T23:30:19.878 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-26T23:30:19.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:19.879 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list-id 2024-01-26T23:30:19.940 INFO:teuthology.orchestra.run.smithi046.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-01-26T23:30:19.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:19.941 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:30:19.941 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:30:19.941 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-26T23:30:19.941 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-01-26T23:30:19.941 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:30:20.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:20.051 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-26T23:30:20.095 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-26T23:30:20.095 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-26T23:30:20.095 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-26T23:30:20.095 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:20.095 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-26T23:30:20.095 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:20.095 INFO:teuthology.orchestra.run.smithi046.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-26T23:30:20.095 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether a6:aa:2a:06:ee:fb brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-26T23:30:20.095 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-26T23:30:20.095 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:20.095 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::a4aa:2aff:fe06:eefb/64 scope link 2024-01-26T23:30:20.095 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:20.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:20.107 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-26T23:30:20.193 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-26T23:30:20.193 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-26T23:30:20.194 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-26T23:30:20.194 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:20.194 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-26T23:30:20.194 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:20.194 INFO:teuthology.orchestra.run.smithi046.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-26T23:30:20.194 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether a6:aa:2a:06:ee:fb brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-26T23:30:20.194 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-26T23:30:20.194 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:20.194 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::a4aa:2aff:fe06:eefb/64 scope link 2024-01-26T23:30:20.194 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:20.206 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-01-26T23:30:20.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:20.206 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:30:20.207 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:30:20.207 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-01-26T23:30:20.207 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-26T23:30:20.207 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-01-26T23:30:20.207 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-01-26T23:30:20.207 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-01-26T23:30:20.207 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:30:20.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:20.439 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:30:20.439 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:30:20.439 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.1 up 2024-01-26T23:30:20.439 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set dev brx.1 master ceph-brx 2024-01-26T23:30:20.439 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:30:20.552 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-01-26T23:30:20.552 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:30:20.552 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-01-26T23:30:20.571 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-01-26T23:30:20.572 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:30:20.572 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-01-26T23:30:20.632 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:30:20.709 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:30:20.710 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:30:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:30:20.785 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:30:20.786 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:20.786 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:30:20.847 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-26T23:30:20.848 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [47] 2024-01-26T23:30:20.848 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-01-26T23:30:20.890 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:30:20.924 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:30:20.924 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:30:20.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi046.stderr:2024-01-26T23:30:20.991+0000 7f07bb1b8540 -1 init, newargv = 0x55fad90170e0 newargc=15 2024-01-26T23:30:20.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi046.stderr:ceph-fuse[54719]: starting ceph client 2024-01-26T23:30:20.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi046.stderr:ceph-fuse[54719]: starting fuse 2024-01-26T23:30:20.998 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:30:20.999 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:30:21.000 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:21.000 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:30:21.060 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-26T23:30:21.060 INFO:teuthology.orchestra.run.smithi046.stdout:48 2024-01-26T23:30:21.061 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47, 48] 2024-01-26T23:30:21.061 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-26T23:30:21.061 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-26T23:30:21.061 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-26T23:30:21.061 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-26T23:30:21.061 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-26T23:30:21.061 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:30:21.061 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-26T23:30:21.061 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-01-26T23:30:21.061 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-26T23:30:21.061 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-01-26T23:30:21.061 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:30:21.062 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-26T23:30:21.062 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-26T23:30:21.062 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-26T23:30:21.062 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-26T23:30:21.062 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:30:21.062 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-26T23:30:21.062 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-26T23:30:21.062 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-26T23:30:21.062 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-26T23:30:21.062 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-26T23:30:21.062 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-26T23:30:21.062 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-26T23:30:21.062 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-26T23:30:21.062 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:30:21.063 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.2")) 2024-01-26T23:30:21.063 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-26T23:30:21.201 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.2.54719.asok 2024-01-26T23:30:21.382 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.54719.asok 2024-01-26T23:30:21.382 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:21.383 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.54719.asok status 2024-01-26T23:30:21.557 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:30:21.558 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-26T23:30:21.558 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-26T23:30:21.558 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-26T23:30:21.558 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "2", 2024-01-26T23:30:21.558 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-26T23:30:21.558 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-01-26T23:30:21.558 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "54719", 2024-01-26T23:30:21.558 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-26T23:30:21.558 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:21.558 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-26T23:30:21.558 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-26T23:30:21.559 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4711, 2024-01-26T23:30:21.559 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-26T23:30:21.559 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-26T23:30:21.559 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-26T23:30:21.559 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4711 2024-01-26T23:30:21.559 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:21.559 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:30:21.559 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:30:21.559 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:0", 2024-01-26T23:30:21.559 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 98139593 2024-01-26T23:30:21.559 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:21.559 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:21.560 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:30:21.560 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:30:21.560 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:0", 2024-01-26T23:30:21.560 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 98139593 2024-01-26T23:30:21.560 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:21.560 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.4711 172.21.15.46:0/98139593", 2024-01-26T23:30:21.560 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "172.21.15.46:0/98139593", 2024-01-26T23:30:21.560 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-26T23:30:21.560 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 9, 2024-01-26T23:30:21.560 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 24, 2024-01-26T23:30:21.560 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-26T23:30:21.560 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-26T23:30:21.561 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-26T23:30:21.561 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:30:21.569 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-26T23:30:21.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.700+0000 7f5fbace6700 1 Processor -- start 2024-01-26T23:30:21.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.700+0000 7f5fbace6700 1 -- start start 2024-01-26T23:30:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.700+0000 7f5fbace6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5fb40b2b40 0x7f5fb40ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.700+0000 7f5fbace6700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb415a0b0 con 0x7f5fb40aedd0 2024-01-26T23:30:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.700+0000 7f5fbace6700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb40ae4d0 con 0x7f5fb40ae960 2024-01-26T23:30:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.700+0000 7f5fbace6700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fb4099fc0 con 0x7f5fb40b2b40 2024-01-26T23:30:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.701+0000 7f5fb8a82700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5fb40b2b40 0x7f5fb40ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.701+0000 7f5fb9283700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5fb40aedd0 0x7f5fb40cb5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41964/0 (socket says 172.21.15.46:41964) 2024-01-26T23:30:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.701+0000 7f5fb9283700 1 -- 172.21.15.46:0/867405130 learned_addr learned my addr 172.21.15.46:0/867405130 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:21.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.701+0000 7f5fb8a82700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5fb40b2b40 0x7f5fb40ad1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54318/0 (socket says 172.21.15.46:54318) 2024-01-26T23:30:21.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.701+0000 7f5fb8a82700 1 -- 172.21.15.46:0/867405130 >> v1:172.21.15.106:6789/0 conn(0x7f5fb40ae960 legacy=0x7f5fb40ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:21.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.701+0000 7f5fb8a82700 1 -- 172.21.15.46:0/867405130 >> v1:172.21.15.46:6789/0 conn(0x7f5fb40aedd0 legacy=0x7f5fb40cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:21.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.701+0000 7f5fb8a82700 1 -- 172.21.15.46:0/867405130 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb40ad8c0 con 0x7f5fb40b2b40 2024-01-26T23:30:21.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.701+0000 7f5fb37fe700 1 -- 172.21.15.46:0/867405130 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546537440 0 0) 0x7f5fb415a0b0 con 0x7f5fb40aedd0 2024-01-26T23:30:21.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.701+0000 7f5fb8a82700 1 --2- 172.21.15.46:0/867405130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40b2b40 0x7f5fb40ad1c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5fa40098d0 tx=0x7f5fa4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f27a6daabebd93 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:21.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.701+0000 7f5fb37fe700 1 -- 172.21.15.46:0/867405130 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4009e70 con 0x7f5fb40b2b40 2024-01-26T23:30:21.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.701+0000 7f5fb37fe700 1 -- 172.21.15.46:0/867405130 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa4005c10 con 0x7f5fb40b2b40 2024-01-26T23:30:21.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.701+0000 7f5fb37fe700 1 -- 172.21.15.46:0/867405130 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa400b610 con 0x7f5fb40b2b40 2024-01-26T23:30:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.702+0000 7f5fbace6700 1 -- 172.21.15.46:0/867405130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40b2b40 msgr2=0x7f5fb40ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.702+0000 7f5fbace6700 1 --2- 172.21.15.46:0/867405130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40b2b40 0x7f5fb40ad1c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5fa40098d0 tx=0x7f5fa4009120 comp rx=0 tx=0).stop 2024-01-26T23:30:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.702+0000 7f5fbace6700 1 -- 172.21.15.46:0/867405130 shutdown_connections 2024-01-26T23:30:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.702+0000 7f5fbace6700 1 --2- 172.21.15.46:0/867405130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40b2b40 0x7f5fb40ad1c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.702+0000 7f5fbace6700 1 -- 172.21.15.46:0/867405130 >> 172.21.15.46:0/867405130 conn(0x7f5fb4099a30 msgr2=0x7f5fb406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.702+0000 7f5fbace6700 1 -- 172.21.15.46:0/867405130 shutdown_connections 2024-01-26T23:30:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.702+0000 7f5fbace6700 1 -- 172.21.15.46:0/867405130 wait complete. 2024-01-26T23:30:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.703+0000 7f5fbace6700 1 Processor -- start 2024-01-26T23:30:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.703+0000 7f5fbace6700 1 -- start start 2024-01-26T23:30:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.704+0000 7f5fbace6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40ae960 0x7f5fb40c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:21.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.704+0000 7f5fbace6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5fb40b2b40 0x7f5fb415b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.704+0000 7f5fbace6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5fb415b5d0 0x7f5fb40c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.704+0000 7f5fbace6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb416e440 con 0x7f5fb40ae960 2024-01-26T23:30:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.704+0000 7f5fbace6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb40c7d60 con 0x7f5fb415b5d0 2024-01-26T23:30:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.704+0000 7f5fbace6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb416d570 con 0x7f5fb40b2b40 2024-01-26T23:30:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.704+0000 7f5fb8a82700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40ae960 0x7f5fb40c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.704+0000 7f5fb8a82700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40ae960 0x7f5fb40c7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54320/0 (socket says 172.21.15.46:54320) 2024-01-26T23:30:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.704+0000 7f5fb8a82700 1 -- 172.21.15.46:0/2515178939 learned_addr learned my addr 172.21.15.46:0/2515178939 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.705+0000 7f5fb3fff700 1 --2- 172.21.15.46:0/2515178939 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5fb40b2b40 0x7f5fb415b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.705+0000 7f5fb9283700 1 --2- 172.21.15.46:0/2515178939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5fb415b5d0 0x7f5fb40c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.705+0000 7f5fb8a82700 1 -- 172.21.15.46:0/2515178939 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5fb40b2b40 msgr2=0x7f5fb415b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.705+0000 7f5fb8a82700 1 --2- 172.21.15.46:0/2515178939 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5fb40b2b40 0x7f5fb415b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.705+0000 7f5fb8a82700 1 -- 172.21.15.46:0/2515178939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5fb415b5d0 msgr2=0x7f5fb40c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:21.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.705+0000 7f5fb8a82700 1 --2- 172.21.15.46:0/2515178939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5fb415b5d0 0x7f5fb40c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.705+0000 7f5fb8a82700 1 -- 172.21.15.46:0/2515178939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa40093c0 con 0x7f5fb40ae960 2024-01-26T23:30:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.705+0000 7f5fb9283700 1 --2- 172.21.15.46:0/2515178939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5fb415b5d0 0x7f5fb40c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.705+0000 7f5fb3fff700 1 --2- 172.21.15.46:0/2515178939 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5fb40b2b40 0x7f5fb415b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.705+0000 7f5fb8a82700 1 --2- 172.21.15.46:0/2515178939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40ae960 0x7f5fb40c7820 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009160 tx=0x7f5fa4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.705+0000 7f5fb0ff9700 1 -- 172.21.15.46:0/2515178939 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4022070 con 0x7f5fb40ae960 2024-01-26T23:30:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.705+0000 7f5fb0ff9700 1 -- 172.21.15.46:0/2515178939 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa40065f0 con 0x7f5fb40ae960 2024-01-26T23:30:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.705+0000 7f5fbace6700 1 -- 172.21.15.46:0/2515178939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb416d830 con 0x7f5fb40ae960 2024-01-26T23:30:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.705+0000 7f5fb0ff9700 1 -- 172.21.15.46:0/2515178939 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa401c450 con 0x7f5fb40ae960 2024-01-26T23:30:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.705+0000 7f5fbace6700 1 -- 172.21.15.46:0/2515178939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb416ddf0 con 0x7f5fb40ae960 2024-01-26T23:30:21.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.706+0000 7f5fbace6700 1 -- 172.21.15.46:0/2515178939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb4169fe0 con 0x7f5fb40ae960 2024-01-26T23:30:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.707+0000 7f5fb0ff9700 1 -- 172.21.15.46:0/2515178939 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5fa401c840 con 0x7f5fb40ae960 2024-01-26T23:30:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.707+0000 7f5fb0ff9700 1 --2- 172.21.15.46:0/2515178939 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5fa0066f30 0x7f5fa00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.707+0000 7f5fb0ff9700 1 -- 172.21.15.46:0/2515178939 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f5fa4086240 con 0x7f5fb40ae960 2024-01-26T23:30:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.707+0000 7f5fb3fff700 1 --2- 172.21.15.46:0/2515178939 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5fa0066f30 0x7f5fa00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.708+0000 7f5fb3fff700 1 --2- 172.21.15.46:0/2515178939 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5fa0066f30 0x7f5fa00693e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8007fd0 tx=0x7f5fa8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.712+0000 7f5fb0ff9700 1 -- 172.21.15.46:0/2515178939 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5fb4169fe0 con 0x7f5fb40ae960 2024-01-26T23:30:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.964+0000 7f5fbace6700 1 -- 172.21.15.46:0/2515178939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5fb404c710 con 0x7f5fb40ae960 2024-01-26T23:30:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.965+0000 7f5fb0ff9700 1 -- 172.21.15.46:0/2515178939 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f5fa40531c0 con 0x7f5fb40ae960 2024-01-26T23:30:21.966 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-01-26T23:30:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.967+0000 7f5fbace6700 1 -- 172.21.15.46:0/2515178939 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5fa0066f30 msgr2=0x7f5fa00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.967+0000 7f5fbace6700 1 --2- 172.21.15.46:0/2515178939 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5fa0066f30 0x7f5fa00693e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8007fd0 tx=0x7f5fa8009040 comp rx=0 tx=0).stop 2024-01-26T23:30:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.967+0000 7f5fbace6700 1 -- 172.21.15.46:0/2515178939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40ae960 msgr2=0x7f5fb40c7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.967+0000 7f5fbace6700 1 --2- 172.21.15.46:0/2515178939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40ae960 0x7f5fb40c7820 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009160 tx=0x7f5fa4015000 comp rx=0 tx=0).stop 2024-01-26T23:30:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.967+0000 7f5fbace6700 1 -- 172.21.15.46:0/2515178939 shutdown_connections 2024-01-26T23:30:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.967+0000 7f5fbace6700 1 --2- 172.21.15.46:0/2515178939 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5fa0066f30 0x7f5fa00693e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.967+0000 7f5fbace6700 1 --2- 172.21.15.46:0/2515178939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40ae960 0x7f5fb40c7820 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.967+0000 7f5fbace6700 1 --2- 172.21.15.46:0/2515178939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5fb415b5d0 0x7f5fb40c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.967+0000 7f5fbace6700 1 --2- 172.21.15.46:0/2515178939 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5fb40b2b40 0x7f5fb415b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.967+0000 7f5fbace6700 1 -- 172.21.15.46:0/2515178939 >> 172.21.15.46:0/2515178939 conn(0x7f5fb4099a30 msgr2=0x7f5fb40c61e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.967+0000 7f5fbace6700 1 -- 172.21.15.46:0/2515178939 shutdown_connections 2024-01-26T23:30:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:21.967+0000 7f5fbace6700 1 -- 172.21.15.46:0/2515178939 wait complete. 2024-01-26T23:30:21.981 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-26T23:30:21.981 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-26T23:30:21.981 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi106.front.sepia.ceph.com 2024-01-26T23:30:21.981 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-26T23:30:21.981 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-26T23:30:21.981 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-26T23:30:21.981 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-26T23:30:21.981 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-26T23:30:21.981 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-26T23:30:21.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:21.982 DEBUG:teuthology.orchestra.run.smithi106:> ip addr 2024-01-26T23:30:21.990 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:30:21.993 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:30:22.005 INFO:teuthology.orchestra.run.smithi106.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-26T23:30:22.005 INFO:teuthology.orchestra.run.smithi106.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-26T23:30:22.005 INFO:teuthology.orchestra.run.smithi106.stdout: inet 127.0.0.1/8 scope host lo 2024-01-26T23:30:22.005 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:22.005 INFO:teuthology.orchestra.run.smithi106.stdout: inet6 ::1/128 scope host 2024-01-26T23:30:22.005 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:22.005 INFO:teuthology.orchestra.run.smithi106.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:30:22.005 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 0c:c4:7a:d8:a1:2a brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:22.006 INFO:teuthology.orchestra.run.smithi106.stdout: altname enp7s0f0 2024-01-26T23:30:22.006 INFO:teuthology.orchestra.run.smithi106.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:30:22.006 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 0c:c4:7a:d8:a1:2b brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:22.006 INFO:teuthology.orchestra.run.smithi106.stdout: altname enp7s0f1 2024-01-26T23:30:22.006 INFO:teuthology.orchestra.run.smithi106.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:30:22.006 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 0c:c4:7a:8f:ce:6c brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:22.006 INFO:teuthology.orchestra.run.smithi106.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-26T23:30:22.006 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 0c:c4:7a:8f:ce:6d brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:22.006 INFO:teuthology.orchestra.run.smithi106.stdout: inet 172.21.15.106/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-26T23:30:22.006 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft 42307sec preferred_lft 42307sec 2024-01-26T23:30:22.006 INFO:teuthology.orchestra.run.smithi106.stdout: inet6 fe80::aeb9:b4d5:ff93:2978/64 scope link noprefixroute 2024-01-26T23:30:22.006 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:22.007 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-26T23:30:22.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:22.007 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-01-26T23:30:22.007 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-01-26T23:30:22.007 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link add name ceph-brx type bridge 2024-01-26T23:30:22.007 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip addr flush dev ceph-brx 2024-01-26T23:30:22.007 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link set ceph-brx up 2024-01-26T23:30:22.007 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-26T23:30:22.007 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-01-26T23:30:22.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:22.217 DEBUG:teuthology.orchestra.run.smithi106:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-26T23:30:22.244 INFO:teuthology.orchestra.run.smithi106.stdout:1 2024-01-26T23:30:22.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:22.246 DEBUG:teuthology.orchestra.run.smithi106:> route 2024-01-26T23:30:22.316 INFO:teuthology.orchestra.run.smithi106.stdout:Kernel IP routing table 2024-01-26T23:30:22.316 INFO:teuthology.orchestra.run.smithi106.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-26T23:30:22.316 INFO:teuthology.orchestra.run.smithi106.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-26T23:30:22.316 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-26T23:30:22.316 INFO:teuthology.orchestra.run.smithi106.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-26T23:30:22.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:22.317 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-01-26T23:30:22.317 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-01-26T23:30:22.317 DEBUG:teuthology.orchestra.run.smithi106:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-26T23:30:22.317 DEBUG:teuthology.orchestra.run.smithi106:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-26T23:30:22.317 DEBUG:teuthology.orchestra.run.smithi106:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-26T23:30:22.317 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-01-26T23:30:22.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:22.544 DEBUG:teuthology.orchestra.run.smithi106:> ip netns list 2024-01-26T23:30:22.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:22.561 DEBUG:teuthology.orchestra.run.smithi106:> ip netns list-id 2024-01-26T23:30:22.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:22.618 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-01-26T23:30:22.618 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-01-26T23:30:22.618 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-26T23:30:22.618 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-01-26T23:30:22.618 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-01-26T23:30:22.713 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-01-26T23:30:22.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:22.714 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-01-26T23:30:22.714 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-01-26T23:30:22.714 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-01-26T23:30:22.714 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-26T23:30:22.714 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-01-26T23:30:22.714 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-01-26T23:30:22.714 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-01-26T23:30:22.714 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-01-26T23:30:22.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:22.913 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-01-26T23:30:22.913 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-01-26T23:30:22.913 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link set brx.0 up 2024-01-26T23:30:22.913 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link set dev brx.0 master ceph-brx 2024-01-26T23:30:22.913 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-01-26T23:30:23.011 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-26T23:30:23.011 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:30:23.011 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-26T23:30:23.027 INFO:teuthology.orchestra.run.smithi106.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-26T23:30:23.027 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:30:23.027 DEBUG:teuthology.orchestra.run.smithi106:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-26T23:30:23.084 DEBUG:teuthology.orchestra.run.smithi106:> sudo modprobe fuse 2024-01-26T23:30:23.151 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:30:23.151 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:30:23.217 INFO:teuthology.orchestra.run.smithi106.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:30:23.218 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:30:23.219 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:23.219 DEBUG:teuthology.orchestra.run.smithi106:> ls /sys/fs/fuse/connections 2024-01-26T23:30:23.276 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-26T23:30:23.276 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-26T23:30:23.319 DEBUG:teuthology.orchestra.run.smithi106:> sudo modprobe fuse 2024-01-26T23:30:23.347 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:30:23.347 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:30:23.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi106.stderr:2024-01-26T23:30:23.403+0000 7f8b16cda540 -1 init, newargv = 0x560633172640 newargc=15 2024-01-26T23:30:23.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi106.stderr:ceph-fuse[51047]: starting ceph client 2024-01-26T23:30:23.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi106.stderr:ceph-fuse[51047]: starting fuse 2024-01-26T23:30:23.417 INFO:teuthology.orchestra.run.smithi106.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:30:23.418 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:30:23.418 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:23.419 DEBUG:teuthology.orchestra.run.smithi106:> ls /sys/fs/fuse/connections 2024-01-26T23:30:23.476 INFO:teuthology.orchestra.run.smithi106.stdout:47 2024-01-26T23:30:23.477 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-26T23:30:23.477 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-26T23:30:23.477 DEBUG:teuthology.orchestra.run.smithi106:> import glob 2024-01-26T23:30:23.477 DEBUG:teuthology.orchestra.run.smithi106:> import re 2024-01-26T23:30:23.477 DEBUG:teuthology.orchestra.run.smithi106:> import os 2024-01-26T23:30:23.477 DEBUG:teuthology.orchestra.run.smithi106:> import subprocess 2024-01-26T23:30:23.477 DEBUG:teuthology.orchestra.run.smithi106:> 2024-01-26T23:30:23.477 DEBUG:teuthology.orchestra.run.smithi106:> def _find_admin_socket(client_name): 2024-01-26T23:30:23.477 DEBUG:teuthology.orchestra.run.smithi106:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-26T23:30:23.477 DEBUG:teuthology.orchestra.run.smithi106:> files = glob.glob(asok_path) 2024-01-26T23:30:23.477 DEBUG:teuthology.orchestra.run.smithi106:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-26T23:30:23.478 DEBUG:teuthology.orchestra.run.smithi106:> 2024-01-26T23:30:23.478 DEBUG:teuthology.orchestra.run.smithi106:> # Given a non-glob path, it better be there 2024-01-26T23:30:23.478 DEBUG:teuthology.orchestra.run.smithi106:> if "*" not in asok_path: 2024-01-26T23:30:23.478 DEBUG:teuthology.orchestra.run.smithi106:> assert(len(files) == 1) 2024-01-26T23:30:23.478 DEBUG:teuthology.orchestra.run.smithi106:> return files[0] 2024-01-26T23:30:23.478 DEBUG:teuthology.orchestra.run.smithi106:> 2024-01-26T23:30:23.478 DEBUG:teuthology.orchestra.run.smithi106:> for f in files: 2024-01-26T23:30:23.478 DEBUG:teuthology.orchestra.run.smithi106:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-26T23:30:23.478 DEBUG:teuthology.orchestra.run.smithi106:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-26T23:30:23.478 DEBUG:teuthology.orchestra.run.smithi106:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-26T23:30:23.478 DEBUG:teuthology.orchestra.run.smithi106:> contents = proc_f.read() 2024-01-26T23:30:23.478 DEBUG:teuthology.orchestra.run.smithi106:> if mountpoint in contents: 2024-01-26T23:30:23.478 DEBUG:teuthology.orchestra.run.smithi106:> return f 2024-01-26T23:30:23.478 DEBUG:teuthology.orchestra.run.smithi106:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-26T23:30:23.478 DEBUG:teuthology.orchestra.run.smithi106:> 2024-01-26T23:30:23.478 DEBUG:teuthology.orchestra.run.smithi106:> print(_find_admin_socket("client.1")) 2024-01-26T23:30:23.478 DEBUG:teuthology.orchestra.run.smithi106:> ' 2024-01-26T23:30:23.599 INFO:teuthology.orchestra.run.smithi106.stdout:/var/run/ceph/ceph-client.1.51047.asok 2024-01-26T23:30:23.782 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.51047.asok 2024-01-26T23:30:23.782 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:23.782 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.51047.asok status 2024-01-26T23:30:23.935 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-01-26T23:30:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "metadata": { 2024-01-26T23:30:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-26T23:30:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-26T23:30:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "entity_id": "1", 2024-01-26T23:30:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "hostname": "smithi106", 2024-01-26T23:30:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-26T23:30:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "pid": "51047", 2024-01-26T23:30:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "root": "/" 2024-01-26T23:30:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-01-26T23:30:23.935 INFO:teuthology.orchestra.run.smithi106.stdout: "dentry_count": 0, 2024-01-26T23:30:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "dentry_pinned_count": 0, 2024-01-26T23:30:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "id": 4720, 2024-01-26T23:30:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "inst": { 2024-01-26T23:30:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "name": { 2024-01-26T23:30:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "client", 2024-01-26T23:30:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "num": 4720 2024-01-26T23:30:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-01-26T23:30:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": { 2024-01-26T23:30:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-01-26T23:30:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:30:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2886715530 2024-01-26T23:30:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-01-26T23:30:23.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-01-26T23:30:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": { 2024-01-26T23:30:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-01-26T23:30:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:30:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2886715530 2024-01-26T23:30:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-01-26T23:30:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "inst_str": "client.4720 192.168.0.1:0/2886715530", 2024-01-26T23:30:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "addr_str": "192.168.0.1:0/2886715530", 2024-01-26T23:30:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "inode_count": 1, 2024-01-26T23:30:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_epoch": 9, 2024-01-26T23:30:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_epoch": 24, 2024-01-26T23:30:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_epoch_barrier": 0, 2024-01-26T23:30:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "blocklisted": false, 2024-01-26T23:30:23.937 INFO:teuthology.orchestra.run.smithi106.stdout: "fs_name": "cephfs" 2024-01-26T23:30:23.937 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-01-26T23:30:23.945 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-26T23:30:24.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.085+0000 7f9c35cb1700 1 Processor -- start 2024-01-26T23:30:24.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.086+0000 7f9c35cb1700 1 -- start start 2024-01-26T23:30:24.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.086+0000 7f9c35cb1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c300acf30 0x7f9c300c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:24.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.086+0000 7f9c35cb1700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9c300ad300 0x7f9c300ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:24.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.086+0000 7f9c35cb1700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c300c4230 con 0x7f9c300ad770 2024-01-26T23:30:24.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.086+0000 7f9c35cb1700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c30157be0 con 0x7f9c300ad300 2024-01-26T23:30:24.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.086+0000 7f9c35cb1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c300cbf10 con 0x7f9c300acf30 2024-01-26T23:30:24.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.087+0000 7f9c2f7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c300acf30 0x7f9c300c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:24.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.087+0000 7f9c2f7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c300acf30 0x7f9c300c3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:47348/0 (socket says 172.21.15.46:47348) 2024-01-26T23:30:24.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.087+0000 7f9c2effd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9c300ad300 0x7f9c300ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:24.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.087+0000 7f9c2f7fe700 1 -- 172.21.15.46:0/3037547434 learned_addr learned my addr 172.21.15.46:0/3037547434 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:24.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.087+0000 7f9c2f7fe700 1 -- 172.21.15.46:0/3037547434 >> v2:172.21.15.106:3300/0 conn(0x7f9c300ad300 msgr2=0x7f9c300ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:24.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.087+0000 7f9c2f7fe700 1 --2- 172.21.15.46:0/3037547434 >> v2:172.21.15.106:3300/0 conn(0x7f9c300ad300 0x7f9c300ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:24.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.087+0000 7f9c2f7fe700 1 -- 172.21.15.46:0/3037547434 >> v1:172.21.15.106:6789/0 conn(0x7f9c300ad770 legacy=0x7f9c3015b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:24.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.087+0000 7f9c2f7fe700 1 -- 172.21.15.46:0/3037547434 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c300c4460 con 0x7f9c300acf30 2024-01-26T23:30:24.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.087+0000 7f9c2e7fc700 1 -- 172.21.15.46:0/3037547434 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315404810 0 0) 0x7f9c300c4230 con 0x7f9c300ad770 2024-01-26T23:30:24.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.087+0000 7f9c2effd700 1 --2- 172.21.15.46:0/3037547434 >> v2:172.21.15.106:3300/0 conn(0x7f9c300ad300 0x7f9c300ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:24.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.088+0000 7f9c2f7fe700 1 --2- 172.21.15.46:0/3037547434 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c300acf30 0x7f9c300c3cf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c180099b0 tx=0x7f9c18009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd77a83effb1aebf server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:24.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.088+0000 7f9c2e7fc700 1 -- 172.21.15.46:0/3037547434 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c18009ea0 con 0x7f9c300acf30 2024-01-26T23:30:24.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.088+0000 7f9c2e7fc700 1 -- 172.21.15.46:0/3037547434 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1800b860 con 0x7f9c300acf30 2024-01-26T23:30:24.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.088+0000 7f9c2e7fc700 1 -- 172.21.15.46:0/3037547434 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1800bc00 con 0x7f9c300acf30 2024-01-26T23:30:24.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.088+0000 7f9c35cb1700 1 -- 172.21.15.46:0/3037547434 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c300acf30 msgr2=0x7f9c300c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:24.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.088+0000 7f9c35cb1700 1 --2- 172.21.15.46:0/3037547434 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c300acf30 0x7f9c300c3cf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c180099b0 tx=0x7f9c18009250 comp rx=0 tx=0).stop 2024-01-26T23:30:24.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.088+0000 7f9c35cb1700 1 -- 172.21.15.46:0/3037547434 shutdown_connections 2024-01-26T23:30:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.088+0000 7f9c35cb1700 1 --2- 172.21.15.46:0/3037547434 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c300acf30 0x7f9c300c3cf0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.088+0000 7f9c35cb1700 1 --2- 172.21.15.46:0/3037547434 >> v2:172.21.15.106:3300/0 conn(0x7f9c300ad300 0x7f9c300ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.088+0000 7f9c35cb1700 1 -- 172.21.15.46:0/3037547434 >> 172.21.15.46:0/3037547434 conn(0x7f9c30097820 msgr2=0x7f9c30068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.088+0000 7f9c35cb1700 1 -- 172.21.15.46:0/3037547434 shutdown_connections 2024-01-26T23:30:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.088+0000 7f9c35cb1700 1 -- 172.21.15.46:0/3037547434 wait complete. 2024-01-26T23:30:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.090+0000 7f9c35cb1700 1 Processor -- start 2024-01-26T23:30:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.090+0000 7f9c35cb1700 1 -- start start 2024-01-26T23:30:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.091+0000 7f9c35cb1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c300acf30 0x7f9c301688a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.091+0000 7f9c35cb1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c300ad300 0x7f9c30141010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.091+0000 7f9c35cb1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c300b12e0 0x7f9c300b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.091+0000 7f9c35cb1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c30141550 con 0x7f9c300ad300 2024-01-26T23:30:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.091+0000 7f9c35cb1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c300b1ca0 con 0x7f9c300acf30 2024-01-26T23:30:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.091+0000 7f9c35cb1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c30168410 con 0x7f9c300b12e0 2024-01-26T23:30:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.091+0000 7f9c2effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c300ad300 0x7f9c30141010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.091+0000 7f9c2effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c300ad300 0x7f9c30141010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54324/0 (socket says 172.21.15.46:54324) 2024-01-26T23:30:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.091+0000 7f9c2effd700 1 -- 172.21.15.46:0/1133538811 learned_addr learned my addr 172.21.15.46:0/1133538811 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.091+0000 7f9c2f7fe700 1 --2- 172.21.15.46:0/1133538811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c300acf30 0x7f9c301688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.091+0000 7f9c2ffff700 1 --2- 172.21.15.46:0/1133538811 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c300b12e0 0x7f9c300b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.091+0000 7f9c2effd700 1 -- 172.21.15.46:0/1133538811 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c300b12e0 msgr2=0x7f9c300b1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.091+0000 7f9c2effd700 1 --2- 172.21.15.46:0/1133538811 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c300b12e0 0x7f9c300b1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.091+0000 7f9c2effd700 1 -- 172.21.15.46:0/1133538811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c300acf30 msgr2=0x7f9c301688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.091+0000 7f9c2effd700 1 --2- 172.21.15.46:0/1133538811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c300acf30 0x7f9c301688a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.091+0000 7f9c2effd700 1 -- 172.21.15.46:0/1133538811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c20009420 con 0x7f9c300ad300 2024-01-26T23:30:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.091+0000 7f9c2effd700 1 --2- 172.21.15.46:0/1133538811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c300ad300 0x7f9c30141010 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9c2000ecb0 tx=0x7f9c200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:24.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.092+0000 7f9c277fe700 1 -- 172.21.15.46:0/1133538811 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2000c920 con 0x7f9c300ad300 2024-01-26T23:30:24.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.092+0000 7f9c277fe700 1 -- 172.21.15.46:0/1133538811 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2000ca80 con 0x7f9c300ad300 2024-01-26T23:30:24.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.092+0000 7f9c277fe700 1 -- 172.21.15.46:0/1133538811 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2000ce00 con 0x7f9c300ad300 2024-01-26T23:30:24.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.092+0000 7f9c35cb1700 1 -- 172.21.15.46:0/1133538811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c180094f0 con 0x7f9c300ad300 2024-01-26T23:30:24.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.092+0000 7f9c35cb1700 1 -- 172.21.15.46:0/1133538811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c300c9660 con 0x7f9c300ad300 2024-01-26T23:30:24.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.093+0000 7f9c35cb1700 1 -- 172.21.15.46:0/1133538811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c30168580 con 0x7f9c300ad300 2024-01-26T23:30:24.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.093+0000 7f9c277fe700 1 -- 172.21.15.46:0/1133538811 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9c20013680 con 0x7f9c300ad300 2024-01-26T23:30:24.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.094+0000 7f9c277fe700 1 --2- 172.21.15.46:0/1133538811 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9c10066f00 0x7f9c100693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:24.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.094+0000 7f9c277fe700 1 -- 172.21.15.46:0/1133538811 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f9c20084ac0 con 0x7f9c300ad300 2024-01-26T23:30:24.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.094+0000 7f9c2f7fe700 1 --2- 172.21.15.46:0/1133538811 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9c10066f00 0x7f9c100693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:24.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.094+0000 7f9c2f7fe700 1 --2- 172.21.15.46:0/1133538811 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9c10066f00 0x7f9c100693b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9c180099b0 tx=0x7f9c18017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:24.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.098+0000 7f9c277fe700 1 -- 172.21.15.46:0/1133538811 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9c20051a40 con 0x7f9c300ad300 2024-01-26T23:30:24.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.350+0000 7f9c35cb1700 1 -- 172.21.15.46:0/1133538811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9c3004c710 con 0x7f9c300ad300 2024-01-26T23:30:24.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.350+0000 7f9c277fe700 1 -- 172.21.15.46:0/1133538811 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f9c20055060 con 0x7f9c300ad300 2024-01-26T23:30:24.351 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-01-26T23:30:24.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.352+0000 7f9c35cb1700 1 -- 172.21.15.46:0/1133538811 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9c10066f00 msgr2=0x7f9c100693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:24.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.352+0000 7f9c35cb1700 1 --2- 172.21.15.46:0/1133538811 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9c10066f00 0x7f9c100693b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9c180099b0 tx=0x7f9c18017000 comp rx=0 tx=0).stop 2024-01-26T23:30:24.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.352+0000 7f9c35cb1700 1 -- 172.21.15.46:0/1133538811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c300ad300 msgr2=0x7f9c30141010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:24.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.352+0000 7f9c35cb1700 1 --2- 172.21.15.46:0/1133538811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c300ad300 0x7f9c30141010 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9c2000ecb0 tx=0x7f9c200091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.352+0000 7f9c35cb1700 1 -- 172.21.15.46:0/1133538811 shutdown_connections 2024-01-26T23:30:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.352+0000 7f9c35cb1700 1 --2- 172.21.15.46:0/1133538811 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9c10066f00 0x7f9c100693b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.352+0000 7f9c35cb1700 1 --2- 172.21.15.46:0/1133538811 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c300b12e0 0x7f9c300b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.352+0000 7f9c35cb1700 1 --2- 172.21.15.46:0/1133538811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c300acf30 0x7f9c301688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.352+0000 7f9c35cb1700 1 --2- 172.21.15.46:0/1133538811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c300ad300 0x7f9c30141010 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.352+0000 7f9c35cb1700 1 -- 172.21.15.46:0/1133538811 >> 172.21.15.46:0/1133538811 conn(0x7f9c30097820 msgr2=0x7f9c3006d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.352+0000 7f9c35cb1700 1 -- 172.21.15.46:0/1133538811 shutdown_connections 2024-01-26T23:30:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:24.352+0000 7f9c35cb1700 1 -- 172.21.15.46:0/1133538811 wait complete. 2024-01-26T23:30:24.366 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-26T23:30:24.366 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-26T23:30:24.367 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi106.front.sepia.ceph.com 2024-01-26T23:30:24.367 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-01-26T23:30:24.367 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-26T23:30:24.367 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-26T23:30:24.367 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-26T23:30:24.367 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-26T23:30:24.367 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-01-26T23:30:24.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:24.367 DEBUG:teuthology.orchestra.run.smithi106:> ip addr 2024-01-26T23:30:24.386 INFO:teuthology.orchestra.run.smithi106.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-26T23:30:24.386 INFO:teuthology.orchestra.run.smithi106.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-26T23:30:24.386 INFO:teuthology.orchestra.run.smithi106.stdout: inet 127.0.0.1/8 scope host lo 2024-01-26T23:30:24.387 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:24.387 INFO:teuthology.orchestra.run.smithi106.stdout: inet6 ::1/128 scope host 2024-01-26T23:30:24.387 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:24.387 INFO:teuthology.orchestra.run.smithi106.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:30:24.387 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 0c:c4:7a:d8:a1:2a brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:24.387 INFO:teuthology.orchestra.run.smithi106.stdout: altname enp7s0f0 2024-01-26T23:30:24.387 INFO:teuthology.orchestra.run.smithi106.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:30:24.387 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 0c:c4:7a:d8:a1:2b brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:24.387 INFO:teuthology.orchestra.run.smithi106.stdout: altname enp7s0f1 2024-01-26T23:30:24.387 INFO:teuthology.orchestra.run.smithi106.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:30:24.387 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 0c:c4:7a:8f:ce:6c brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:24.387 INFO:teuthology.orchestra.run.smithi106.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-26T23:30:24.387 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 0c:c4:7a:8f:ce:6d brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:24.387 INFO:teuthology.orchestra.run.smithi106.stdout: inet 172.21.15.106/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-26T23:30:24.387 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft 42305sec preferred_lft 42305sec 2024-01-26T23:30:24.388 INFO:teuthology.orchestra.run.smithi106.stdout: inet6 fe80::aeb9:b4d5:ff93:2978/64 scope link noprefixroute 2024-01-26T23:30:24.388 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:24.388 INFO:teuthology.orchestra.run.smithi106.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-26T23:30:24.388 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 8a:a5:9c:bd:71:60 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:24.388 INFO:teuthology.orchestra.run.smithi106.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-26T23:30:24.388 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:24.388 INFO:teuthology.orchestra.run.smithi106.stdout: inet6 fe80::70f6:9fff:fe37:9111/64 scope link 2024-01-26T23:30:24.388 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:24.388 INFO:teuthology.orchestra.run.smithi106.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:30:24.388 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether 8a:a5:9c:bd:71:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-26T23:30:24.388 INFO:teuthology.orchestra.run.smithi106.stdout: inet6 fe80::88a5:9cff:febd:7160/64 scope link tentative 2024-01-26T23:30:24.388 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:24.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:24.389 DEBUG:teuthology.orchestra.run.smithi106:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-26T23:30:24.455 INFO:teuthology.orchestra.run.smithi106.stdout:1 2024-01-26T23:30:24.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:24.456 DEBUG:teuthology.orchestra.run.smithi106:> route 2024-01-26T23:30:24.515 INFO:teuthology.orchestra.run.smithi106.stdout:Kernel IP routing table 2024-01-26T23:30:24.515 INFO:teuthology.orchestra.run.smithi106.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-26T23:30:24.515 INFO:teuthology.orchestra.run.smithi106.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-26T23:30:24.515 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-26T23:30:24.516 INFO:teuthology.orchestra.run.smithi106.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-26T23:30:24.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:24.516 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-01-26T23:30:24.516 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-01-26T23:30:24.516 DEBUG:teuthology.orchestra.run.smithi106:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-26T23:30:24.516 DEBUG:teuthology.orchestra.run.smithi106:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-26T23:30:24.516 DEBUG:teuthology.orchestra.run.smithi106:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-26T23:30:24.516 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-01-26T23:30:24.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:24.623 DEBUG:teuthology.orchestra.run.smithi106:> ip netns list 2024-01-26T23:30:24.639 INFO:teuthology.orchestra.run.smithi106.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-26T23:30:24.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:24.640 DEBUG:teuthology.orchestra.run.smithi106:> ip netns list-id 2024-01-26T23:30:24.696 INFO:teuthology.orchestra.run.smithi106.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-01-26T23:30:24.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:24.697 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-01-26T23:30:24.697 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-01-26T23:30:24.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-26T23:30:24.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-01-26T23:30:24.697 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-01-26T23:30:24.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:24.790 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-26T23:30:24.829 INFO:teuthology.orchestra.run.smithi106.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-26T23:30:24.829 INFO:teuthology.orchestra.run.smithi106.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-26T23:30:24.829 INFO:teuthology.orchestra.run.smithi106.stdout: inet 127.0.0.1/8 scope host lo 2024-01-26T23:30:24.829 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:24.830 INFO:teuthology.orchestra.run.smithi106.stdout: inet6 ::1/128 scope host 2024-01-26T23:30:24.830 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:24.830 INFO:teuthology.orchestra.run.smithi106.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-26T23:30:24.830 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether ae:0f:29:68:6a:52 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-26T23:30:24.830 INFO:teuthology.orchestra.run.smithi106.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-26T23:30:24.830 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:24.830 INFO:teuthology.orchestra.run.smithi106.stdout: inet6 fe80::ac0f:29ff:fe68:6a52/64 scope link 2024-01-26T23:30:24.830 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:24.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:24.842 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-26T23:30:24.929 INFO:teuthology.orchestra.run.smithi106.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-26T23:30:24.929 INFO:teuthology.orchestra.run.smithi106.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-26T23:30:24.929 INFO:teuthology.orchestra.run.smithi106.stdout: inet 127.0.0.1/8 scope host lo 2024-01-26T23:30:24.929 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:24.930 INFO:teuthology.orchestra.run.smithi106.stdout: inet6 ::1/128 scope host 2024-01-26T23:30:24.930 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:24.930 INFO:teuthology.orchestra.run.smithi106.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-26T23:30:24.930 INFO:teuthology.orchestra.run.smithi106.stdout: link/ether ae:0f:29:68:6a:52 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-26T23:30:24.930 INFO:teuthology.orchestra.run.smithi106.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-26T23:30:24.930 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:24.930 INFO:teuthology.orchestra.run.smithi106.stdout: inet6 fe80::ac0f:29ff:fe68:6a52/64 scope link 2024-01-26T23:30:24.930 INFO:teuthology.orchestra.run.smithi106.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:24.936 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-01-26T23:30:24.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:24.937 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-01-26T23:30:24.937 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-01-26T23:30:24.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-01-26T23:30:24.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-26T23:30:24.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-01-26T23:30:24.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-01-26T23:30:24.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-01-26T23:30:24.937 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-01-26T23:30:25.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:25.144 DEBUG:teuthology.orchestra.run.smithi106:> (cd / && exec bash -c ' 2024-01-26T23:30:25.144 DEBUG:teuthology.orchestra.run.smithi106:> set -e 2024-01-26T23:30:25.144 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link set brx.1 up 2024-01-26T23:30:25.144 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link set dev brx.1 master ceph-brx 2024-01-26T23:30:25.144 DEBUG:teuthology.orchestra.run.smithi106:> ') 2024-01-26T23:30:25.241 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-01-26T23:30:25.241 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:30:25.242 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-01-26T23:30:25.258 INFO:teuthology.orchestra.run.smithi106.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-01-26T23:30:25.258 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:30:25.258 DEBUG:teuthology.orchestra.run.smithi106:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-01-26T23:30:25.315 DEBUG:teuthology.orchestra.run.smithi106:> sudo modprobe fuse 2024-01-26T23:30:25.383 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:30:25.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:30:25.449 INFO:teuthology.orchestra.run.smithi106.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:30:25.450 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:30:25.451 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:25.451 DEBUG:teuthology.orchestra.run.smithi106:> ls /sys/fs/fuse/connections 2024-01-26T23:30:25.508 INFO:teuthology.orchestra.run.smithi106.stdout:47 2024-01-26T23:30:25.509 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [47] 2024-01-26T23:30:25.509 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-01-26T23:30:25.551 DEBUG:teuthology.orchestra.run.smithi106:> sudo modprobe fuse 2024-01-26T23:30:25.578 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:30:25.578 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:30:25.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi106.stderr:2024-01-26T23:30:25.634+0000 7fee0c8a3540 -1 init, newargv = 0x56037d805bb0 newargc=15 2024-01-26T23:30:25.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi106.stderr:ceph-fuse[51656]: starting ceph client 2024-01-26T23:30:25.640 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi106.stderr:ceph-fuse[51656]: starting fuse 2024-01-26T23:30:25.646 INFO:teuthology.orchestra.run.smithi106.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:30:25.647 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:30:25.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:25.647 DEBUG:teuthology.orchestra.run.smithi106:> ls /sys/fs/fuse/connections 2024-01-26T23:30:25.704 INFO:teuthology.orchestra.run.smithi106.stdout:47 2024-01-26T23:30:25.704 INFO:teuthology.orchestra.run.smithi106.stdout:48 2024-01-26T23:30:25.704 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47, 48] 2024-01-26T23:30:25.705 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-26T23:30:25.705 DEBUG:teuthology.orchestra.run.smithi106:> import glob 2024-01-26T23:30:25.705 DEBUG:teuthology.orchestra.run.smithi106:> import re 2024-01-26T23:30:25.705 DEBUG:teuthology.orchestra.run.smithi106:> import os 2024-01-26T23:30:25.705 DEBUG:teuthology.orchestra.run.smithi106:> import subprocess 2024-01-26T23:30:25.705 DEBUG:teuthology.orchestra.run.smithi106:> 2024-01-26T23:30:25.705 DEBUG:teuthology.orchestra.run.smithi106:> def _find_admin_socket(client_name): 2024-01-26T23:30:25.705 DEBUG:teuthology.orchestra.run.smithi106:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-01-26T23:30:25.705 DEBUG:teuthology.orchestra.run.smithi106:> files = glob.glob(asok_path) 2024-01-26T23:30:25.705 DEBUG:teuthology.orchestra.run.smithi106:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-01-26T23:30:25.705 DEBUG:teuthology.orchestra.run.smithi106:> 2024-01-26T23:30:25.705 DEBUG:teuthology.orchestra.run.smithi106:> # Given a non-glob path, it better be there 2024-01-26T23:30:25.705 DEBUG:teuthology.orchestra.run.smithi106:> if "*" not in asok_path: 2024-01-26T23:30:25.705 DEBUG:teuthology.orchestra.run.smithi106:> assert(len(files) == 1) 2024-01-26T23:30:25.706 DEBUG:teuthology.orchestra.run.smithi106:> return files[0] 2024-01-26T23:30:25.706 DEBUG:teuthology.orchestra.run.smithi106:> 2024-01-26T23:30:25.706 DEBUG:teuthology.orchestra.run.smithi106:> for f in files: 2024-01-26T23:30:25.706 DEBUG:teuthology.orchestra.run.smithi106:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-26T23:30:25.706 DEBUG:teuthology.orchestra.run.smithi106:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-26T23:30:25.706 DEBUG:teuthology.orchestra.run.smithi106:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-26T23:30:25.706 DEBUG:teuthology.orchestra.run.smithi106:> contents = proc_f.read() 2024-01-26T23:30:25.706 DEBUG:teuthology.orchestra.run.smithi106:> if mountpoint in contents: 2024-01-26T23:30:25.706 DEBUG:teuthology.orchestra.run.smithi106:> return f 2024-01-26T23:30:25.706 DEBUG:teuthology.orchestra.run.smithi106:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-26T23:30:25.706 DEBUG:teuthology.orchestra.run.smithi106:> 2024-01-26T23:30:25.706 DEBUG:teuthology.orchestra.run.smithi106:> print(_find_admin_socket("client.3")) 2024-01-26T23:30:25.706 DEBUG:teuthology.orchestra.run.smithi106:> ' 2024-01-26T23:30:25.827 INFO:teuthology.orchestra.run.smithi106.stdout:/var/run/ceph/ceph-client.3.51656.asok 2024-01-26T23:30:26.009 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.51656.asok 2024-01-26T23:30:26.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:26.010 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.51656.asok status 2024-01-26T23:30:26.162 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-01-26T23:30:26.163 INFO:teuthology.orchestra.run.smithi106.stdout: "metadata": { 2024-01-26T23:30:26.163 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-26T23:30:26.163 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-26T23:30:26.163 INFO:teuthology.orchestra.run.smithi106.stdout: "entity_id": "3", 2024-01-26T23:30:26.163 INFO:teuthology.orchestra.run.smithi106.stdout: "hostname": "smithi106", 2024-01-26T23:30:26.163 INFO:teuthology.orchestra.run.smithi106.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-01-26T23:30:26.163 INFO:teuthology.orchestra.run.smithi106.stdout: "pid": "51656", 2024-01-26T23:30:26.163 INFO:teuthology.orchestra.run.smithi106.stdout: "root": "/" 2024-01-26T23:30:26.163 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-01-26T23:30:26.163 INFO:teuthology.orchestra.run.smithi106.stdout: "dentry_count": 0, 2024-01-26T23:30:26.163 INFO:teuthology.orchestra.run.smithi106.stdout: "dentry_pinned_count": 0, 2024-01-26T23:30:26.163 INFO:teuthology.orchestra.run.smithi106.stdout: "id": 4735, 2024-01-26T23:30:26.163 INFO:teuthology.orchestra.run.smithi106.stdout: "inst": { 2024-01-26T23:30:26.164 INFO:teuthology.orchestra.run.smithi106.stdout: "name": { 2024-01-26T23:30:26.164 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "client", 2024-01-26T23:30:26.164 INFO:teuthology.orchestra.run.smithi106.stdout: "num": 4735 2024-01-26T23:30:26.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-01-26T23:30:26.164 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": { 2024-01-26T23:30:26.164 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-01-26T23:30:26.164 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "192.168.0.2:0", 2024-01-26T23:30:26.164 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1633284217 2024-01-26T23:30:26.164 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-01-26T23:30:26.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-01-26T23:30:26.164 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": { 2024-01-26T23:30:26.164 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-01-26T23:30:26.164 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "192.168.0.2:0", 2024-01-26T23:30:26.164 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1633284217 2024-01-26T23:30:26.165 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-01-26T23:30:26.165 INFO:teuthology.orchestra.run.smithi106.stdout: "inst_str": "client.4735 192.168.0.2:0/1633284217", 2024-01-26T23:30:26.165 INFO:teuthology.orchestra.run.smithi106.stdout: "addr_str": "192.168.0.2:0/1633284217", 2024-01-26T23:30:26.165 INFO:teuthology.orchestra.run.smithi106.stdout: "inode_count": 1, 2024-01-26T23:30:26.165 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_epoch": 9, 2024-01-26T23:30:26.165 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_epoch": 24, 2024-01-26T23:30:26.165 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_epoch_barrier": 0, 2024-01-26T23:30:26.165 INFO:teuthology.orchestra.run.smithi106.stdout: "blocklisted": false, 2024-01-26T23:30:26.165 INFO:teuthology.orchestra.run.smithi106.stdout: "fs_name": "cephfs" 2024-01-26T23:30:26.165 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-01-26T23:30:26.172 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:26.172 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-26T23:30:26.172 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-26T23:30:26.195 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-26T23:30:26.196 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-26T23:30:26.196 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:26.196 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:30:26.271 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:26.271 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-26T23:30:26.271 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.2 2024-01-26T23:30:26.335 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-26T23:30:26.335 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-01-26T23:30:26.335 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:26.335 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-01-26T23:30:26.411 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:26.411 DEBUG:teuthology.orchestra.run.smithi106:> stat --file-system '--printf=%T 2024-01-26T23:30:26.411 DEBUG:teuthology.orchestra.run.smithi106:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-26T23:30:26.429 INFO:teuthology.orchestra.run.smithi106.stdout:fuseblk 2024-01-26T23:30:26.429 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-26T23:30:26.429 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:26.430 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-26T23:30:26.496 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:26.496 DEBUG:teuthology.orchestra.run.smithi106:> stat --file-system '--printf=%T 2024-01-26T23:30:26.496 DEBUG:teuthology.orchestra.run.smithi106:> ' -- /home/ubuntu/cephtest/mnt.3 2024-01-26T23:30:26.553 INFO:teuthology.orchestra.run.smithi106.stdout:fuseblk 2024-01-26T23:30:26.554 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-01-26T23:30:26.554 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:30:26.554 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-01-26T23:30:26.620 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-01-26T23:30:26.654 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:26.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.794+0000 7f37acab2700 1 Processor -- start 2024-01-26T23:30:26.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.794+0000 7f37acab2700 1 -- start start 2024-01-26T23:30:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.795+0000 7f37acab2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f37a80cd250 0x7f37a8162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.795+0000 7f37acab2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a80cbf10 con 0x7f37a80acf30 2024-01-26T23:30:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.795+0000 7f37acab2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a80c4460 con 0x7f37a80c3cf0 2024-01-26T23:30:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.795+0000 7f37acab2700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a80ada10 con 0x7f37a80cd250 2024-01-26T23:30:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.795+0000 7f37a659c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f37a80acf30 0x7f37a80ad300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41974/0 (socket says 172.21.15.46:41974) 2024-01-26T23:30:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.795+0000 7f37a659c700 1 -- 172.21.15.46:0/1105166755 learned_addr learned my addr 172.21.15.46:0/1105166755 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.795+0000 7f37a6d9d700 1 --2- 172.21.15.46:0/1105166755 >> v2:172.21.15.106:3300/0 conn(0x7f37a80cd250 0x7f37a8162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a559a700 1 -- 172.21.15.46:0/1105166755 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864793015 0 0) 0x7f37a80cbf10 con 0x7f37a80acf30 2024-01-26T23:30:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a559a700 1 -- 172.21.15.46:0/1105166755 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3794003570 con 0x7f37a80acf30 2024-01-26T23:30:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a559a700 1 -- 172.21.15.46:0/1105166755 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1152933349 0 0) 0x7f3794003570 con 0x7f37a80acf30 2024-01-26T23:30:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a559a700 1 -- 172.21.15.46:0/1105166755 >> v2:172.21.15.106:3300/0 conn(0x7f37a80cd250 msgr2=0x7f37a8162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a559a700 1 --2- 172.21.15.46:0/1105166755 >> v2:172.21.15.106:3300/0 conn(0x7f37a80cd250 0x7f37a8162360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a559a700 1 -- 172.21.15.46:0/1105166755 >> v1:172.21.15.106:6789/0 conn(0x7f37a80c3cf0 legacy=0x7f37a80ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a559a700 1 -- 172.21.15.46:0/1105166755 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a80c4740 con 0x7f37a80acf30 2024-01-26T23:30:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a559a700 1 -- 172.21.15.46:0/1105166755 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f3790001470 con 0x7f37a80acf30 2024-01-26T23:30:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a559a700 1 -- 172.21.15.46:0/1105166755 >> v1:172.21.15.46:6789/0 conn(0x7f37a80acf30 legacy=0x7f37a80ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a559a700 1 --2- 172.21.15.46:0/1105166755 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3794003a60 0x7f3794003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a559a700 1 --2- 172.21.15.46:0/1105166755 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3794004810 0x7f3794004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a559a700 1 --2- 172.21.15.46:0/1105166755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37940051c0 0x7f379400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a559a700 1 -- 172.21.15.46:0/1105166755 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a80ada10 con 0x7f37940051c0 2024-01-26T23:30:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a559a700 1 -- 172.21.15.46:0/1105166755 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379400c160 con 0x7f3794004810 2024-01-26T23:30:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a559a700 1 -- 172.21.15.46:0/1105166755 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379400c320 con 0x7f3794003a60 2024-01-26T23:30:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a6d9d700 1 --2- 172.21.15.46:0/1105166755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37940051c0 0x7f379400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a659c700 1 --2- 172.21.15.46:0/1105166755 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3794003a60 0x7f3794003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.796+0000 7f37a5d9b700 1 --2- 172.21.15.46:0/1105166755 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3794004810 0x7f3794004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.797+0000 7f37a6d9d700 1 -- 172.21.15.46:0/1105166755 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3794003a60 msgr2=0x7f3794003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.797+0000 7f37a6d9d700 1 --2- 172.21.15.46:0/1105166755 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3794003a60 0x7f3794003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.797+0000 7f37a6d9d700 1 -- 172.21.15.46:0/1105166755 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3794004810 msgr2=0x7f3794004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.797+0000 7f37a6d9d700 1 --2- 172.21.15.46:0/1105166755 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3794004810 0x7f3794004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.797+0000 7f37a6d9d700 1 -- 172.21.15.46:0/1105166755 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3794003570 con 0x7f37940051c0 2024-01-26T23:30:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.797+0000 7f37a5d9b700 1 --2- 172.21.15.46:0/1105166755 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3794004810 0x7f3794004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.797+0000 7f37a6d9d700 1 -- 172.21.15.46:0/1105166755 reap_dead start 2024-01-26T23:30:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.797+0000 7f37a6d9d700 1 --2- 172.21.15.46:0/1105166755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37940051c0 0x7f379400ba60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f37a000e210 tx=0x7f37a000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abfaff20a71e1e7e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.797+0000 7f37a559a700 1 -- 172.21.15.46:0/1105166755 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a000eac0 con 0x7f37940051c0 2024-01-26T23:30:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.797+0000 7f37a559a700 1 -- 172.21.15.46:0/1105166755 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a000f040 con 0x7f37940051c0 2024-01-26T23:30:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.797+0000 7f37a559a700 1 -- 172.21.15.46:0/1105166755 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a000ee70 con 0x7f37940051c0 2024-01-26T23:30:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.797+0000 7f37acab2700 1 -- 172.21.15.46:0/1105166755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37940051c0 msgr2=0x7f379400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.797+0000 7f37acab2700 1 --2- 172.21.15.46:0/1105166755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37940051c0 0x7f379400ba60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f37a000e210 tx=0x7f37a000b040 comp rx=0 tx=0).stop 2024-01-26T23:30:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.797+0000 7f37acab2700 1 -- 172.21.15.46:0/1105166755 shutdown_connections 2024-01-26T23:30:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.798+0000 7f37acab2700 1 --2- 172.21.15.46:0/1105166755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37940051c0 0x7f379400ba60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.798+0000 7f37acab2700 1 -- 172.21.15.46:0/1105166755 >> 172.21.15.46:0/1105166755 conn(0x7f37a8097820 msgr2=0x7f37a8068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.798+0000 7f37acab2700 1 -- 172.21.15.46:0/1105166755 shutdown_connections 2024-01-26T23:30:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.798+0000 7f37acab2700 1 -- 172.21.15.46:0/1105166755 wait complete. 2024-01-26T23:30:26.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.799+0000 7f37acab2700 1 Processor -- start 2024-01-26T23:30:26.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.799+0000 7f37acab2700 1 -- start start 2024-01-26T23:30:26.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.799+0000 7f37acab2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f37940051c0 0x7f37a80ae770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:26.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.799+0000 7f37acab2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37a80aecb0 0x7f37a806d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:26.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.799+0000 7f37acab2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37a806d540 0x7f37a80b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f37acab2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a8156980 con 0x7f37a806d540 2024-01-26T23:30:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f37acab2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a8158190 con 0x7f37a80aecb0 2024-01-26T23:30:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f37acab2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a806d9b0 con 0x7f37940051c0 2024-01-26T23:30:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f37a6d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37a806d540 0x7f37a80b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f37a6d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37a806d540 0x7f37a80b12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54344/0 (socket says 172.21.15.46:54344) 2024-01-26T23:30:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f37a6d9d700 1 -- 172.21.15.46:0/3033406206 learned_addr learned my addr 172.21.15.46:0/3033406206 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f37a5d9b700 1 --2- 172.21.15.46:0/3033406206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37a80aecb0 0x7f37a806d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f37a659c700 1 --2- 172.21.15.46:0/3033406206 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f37940051c0 0x7f37a80ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f37a6d9d700 1 -- 172.21.15.46:0/3033406206 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f37940051c0 msgr2=0x7f37a80ae770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f37a6d9d700 1 --2- 172.21.15.46:0/3033406206 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f37940051c0 0x7f37a80ae770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f37a6d9d700 1 -- 172.21.15.46:0/3033406206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37a80aecb0 msgr2=0x7f37a806d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f37a6d9d700 1 --2- 172.21.15.46:0/3033406206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37a80aecb0 0x7f37a806d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f37a6d9d700 1 -- 172.21.15.46:0/3033406206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a80af290 con 0x7f37a806d540 2024-01-26T23:30:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f37a5d9b700 1 --2- 172.21.15.46:0/3033406206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37a80aecb0 0x7f37a806d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:30:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f37a6d9d700 1 --2- 172.21.15.46:0/3033406206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37a806d540 0x7f37a80b12e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f37a80add90 tx=0x7f37a0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f379e7fc700 1 -- 172.21.15.46:0/3033406206 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a001d660 con 0x7f37a806d540 2024-01-26T23:30:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f379e7fc700 1 -- 172.21.15.46:0/3033406206 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a0022070 con 0x7f37a806d540 2024-01-26T23:30:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f37acab2700 1 -- 172.21.15.46:0/3033406206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a80af570 con 0x7f37a806d540 2024-01-26T23:30:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f379e7fc700 1 -- 172.21.15.46:0/3033406206 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a001da10 con 0x7f37a806d540 2024-01-26T23:30:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.800+0000 7f37acab2700 1 -- 172.21.15.46:0/3033406206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a80b2e80 con 0x7f37a806d540 2024-01-26T23:30:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.801+0000 7f37acab2700 1 -- 172.21.15.46:0/3033406206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a8157f60 con 0x7f37a806d540 2024-01-26T23:30:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.802+0000 7f379e7fc700 1 -- 172.21.15.46:0/3033406206 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f37a0003020 con 0x7f37a806d540 2024-01-26T23:30:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.802+0000 7f379e7fc700 1 --2- 172.21.15.46:0/3033406206 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3788066f30 0x7f37880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.802+0000 7f379e7fc700 1 -- 172.21.15.46:0/3033406206 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f37a0024070 con 0x7f37a806d540 2024-01-26T23:30:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.802+0000 7f37a659c700 1 --2- 172.21.15.46:0/3033406206 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3788066f30 0x7f37880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.803+0000 7f37a659c700 1 --2- 172.21.15.46:0/3033406206 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3788066f30 0x7f37880693e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3790001050 tx=0x7f3790009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:26.807+0000 7f379e7fc700 1 -- 172.21.15.46:0/3033406206 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f37a8157f60 con 0x7f37a806d540 2024-01-26T23:30:27.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.011+0000 7f37acab2700 1 -- 172.21.15.46:0/3033406206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37a8158300 con 0x7f37a806d540 2024-01-26T23:30:27.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.012+0000 7f379e7fc700 1 -- 172.21.15.46:0/3033406206 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7f37a8158300 con 0x7f37a806d540 2024-01-26T23:30:27.013 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:27.013 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:49.075104+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.015+0000 7f37acab2700 1 -- 172.21.15.46:0/3033406206 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3788066f30 msgr2=0x7f37880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.015+0000 7f37acab2700 1 --2- 172.21.15.46:0/3033406206 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3788066f30 0x7f37880693e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3790001050 tx=0x7f3790009040 comp rx=0 tx=0).stop 2024-01-26T23:30:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.015+0000 7f37acab2700 1 -- 172.21.15.46:0/3033406206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37a806d540 msgr2=0x7f37a80b12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.015+0000 7f37acab2700 1 --2- 172.21.15.46:0/3033406206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37a806d540 0x7f37a80b12e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f37a80add90 tx=0x7f37a0019000 comp rx=0 tx=0).stop 2024-01-26T23:30:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.015+0000 7f37acab2700 1 -- 172.21.15.46:0/3033406206 shutdown_connections 2024-01-26T23:30:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.015+0000 7f37acab2700 1 --2- 172.21.15.46:0/3033406206 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3788066f30 0x7f37880693e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.015+0000 7f37acab2700 1 --2- 172.21.15.46:0/3033406206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37a806d540 0x7f37a80b12e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.015+0000 7f37acab2700 1 --2- 172.21.15.46:0/3033406206 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f37940051c0 0x7f37a80ae770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.015+0000 7f37acab2700 1 --2- 172.21.15.46:0/3033406206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37a80aecb0 0x7f37a806d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.015+0000 7f37acab2700 1 -- 172.21.15.46:0/3033406206 >> 172.21.15.46:0/3033406206 conn(0x7f37a8097820 msgr2=0x7f37a8157160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.015+0000 7f37acab2700 1 -- 172.21.15.46:0/3033406206 shutdown_connections 2024-01-26T23:30:27.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.015+0000 7f37acab2700 1 -- 172.21.15.46:0/3033406206 wait complete. 2024-01-26T23:30:27.030 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-26T23:29:42.539759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-26T23:29:43.002669+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-01-26T23:29:46.646864+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-26T23:30:27.030 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:30:27.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.164+0000 7fbb745c4700 1 Processor -- start 2024-01-26T23:30:27.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.164+0000 7fbb745c4700 1 -- start start 2024-01-26T23:30:27.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.165+0000 7fbb745c4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbb6c1599d0 0x7fbb6c159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.165+0000 7fbb745c4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb6c15d8c0 0x7fbb6c15dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.165+0000 7fbb745c4700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6c0ab620 con 0x7fbb6c0ab1c0 2024-01-26T23:30:27.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.165+0000 7fbb745c4700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb6c0b1a60 con 0x7fbb6c1599d0 2024-01-26T23:30:27.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.165+0000 7fbb745c4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c15e230 con 0x7fbb6c15d8c0 2024-01-26T23:30:27.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.165+0000 7fbb71b5f700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbb6c0ab1c0 0x7fbb6c0b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41990/0 (socket says 172.21.15.46:41990) 2024-01-26T23:30:27.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.165+0000 7fbb72360700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbb6c1599d0 0x7fbb6c159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.165+0000 7fbb71b5f700 1 -- 172.21.15.46:0/1796292089 learned_addr learned my addr 172.21.15.46:0/1796292089 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:27.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.165+0000 7fbb72b61700 1 --2- 172.21.15.46:0/1796292089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb6c15d8c0 0x7fbb6c15dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.165+0000 7fbb7135e700 1 -- 172.21.15.46:0/1796292089 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762448364 0 0) 0x7fbb5c001110 con 0x7fbb6c0ab1c0 2024-01-26T23:30:27.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.165+0000 7fbb7135e700 1 -- 172.21.15.46:0/1796292089 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb58003570 con 0x7fbb6c0ab1c0 2024-01-26T23:30:27.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.165+0000 7fbb72360700 1 -- 172.21.15.46:0/1796292089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb6c15d8c0 msgr2=0x7fbb6c15dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:27.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.165+0000 7fbb72360700 1 --2- 172.21.15.46:0/1796292089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb6c15d8c0 0x7fbb6c15dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.166+0000 7fbb72360700 1 -- 172.21.15.46:0/1796292089 >> v1:172.21.15.46:6789/0 conn(0x7fbb6c0ab1c0 legacy=0x7fbb6c0b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.166+0000 7fbb72360700 1 -- 172.21.15.46:0/1796292089 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c0b11c0 con 0x7fbb6c1599d0 2024-01-26T23:30:27.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.166+0000 7fbb72360700 1 --2- 172.21.15.46:0/1796292089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb6c1599d0 0x7fbb6c159da0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbb680098d0 tx=0x7fbb68009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7971be244f2f68de server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:27.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.166+0000 7fbb7135e700 1 -- 172.21.15.46:0/1796292089 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb68005c10 con 0x7fbb6c1599d0 2024-01-26T23:30:27.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.166+0000 7fbb7135e700 1 -- 172.21.15.46:0/1796292089 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb68005d70 con 0x7fbb6c1599d0 2024-01-26T23:30:27.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.166+0000 7fbb7135e700 1 -- 172.21.15.46:0/1796292089 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb68006260 con 0x7fbb6c1599d0 2024-01-26T23:30:27.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.166+0000 7fbb745c4700 1 -- 172.21.15.46:0/1796292089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb6c1599d0 msgr2=0x7fbb6c159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.166+0000 7fbb745c4700 1 --2- 172.21.15.46:0/1796292089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb6c1599d0 0x7fbb6c159da0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbb680098d0 tx=0x7fbb68009120 comp rx=0 tx=0).stop 2024-01-26T23:30:27.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.167+0000 7fbb745c4700 1 -- 172.21.15.46:0/1796292089 shutdown_connections 2024-01-26T23:30:27.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.167+0000 7fbb745c4700 1 --2- 172.21.15.46:0/1796292089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb6c15d8c0 0x7fbb6c15dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.167+0000 7fbb745c4700 1 --2- 172.21.15.46:0/1796292089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb6c1599d0 0x7fbb6c159da0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.167+0000 7fbb745c4700 1 -- 172.21.15.46:0/1796292089 >> 172.21.15.46:0/1796292089 conn(0x7fbb6c06f840 msgr2=0x7fbb6c087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:27.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.167+0000 7fbb745c4700 1 -- 172.21.15.46:0/1796292089 shutdown_connections 2024-01-26T23:30:27.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.167+0000 7fbb745c4700 1 -- 172.21.15.46:0/1796292089 wait complete. 2024-01-26T23:30:27.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.168+0000 7fbb745c4700 1 Processor -- start 2024-01-26T23:30:27.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.168+0000 7fbb745c4700 1 -- start start 2024-01-26T23:30:27.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.168+0000 7fbb745c4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb6c0ab1c0 0x7fbb6c0ac390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb745c4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb6c1599d0 0x7fbb6c09da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb745c4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb6c15d8c0 0x7fbb6c09df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb745c4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c0af1c0 con 0x7fbb6c0ab1c0 2024-01-26T23:30:27.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb745c4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c0af4a0 con 0x7fbb6c15d8c0 2024-01-26T23:30:27.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb745c4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c09a3b0 con 0x7fbb6c1599d0 2024-01-26T23:30:27.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb72360700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb6c0ab1c0 0x7fbb6c0ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb72360700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb6c0ab1c0 0x7fbb6c0ac390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54360/0 (socket says 172.21.15.46:54360) 2024-01-26T23:30:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb72360700 1 -- 172.21.15.46:0/3003080517 learned_addr learned my addr 172.21.15.46:0/3003080517 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb71b5f700 1 --2- 172.21.15.46:0/3003080517 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb6c1599d0 0x7fbb6c09da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb72b61700 1 --2- 172.21.15.46:0/3003080517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb6c15d8c0 0x7fbb6c09df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb72360700 1 -- 172.21.15.46:0/3003080517 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb6c1599d0 msgr2=0x7fbb6c09da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb72360700 1 --2- 172.21.15.46:0/3003080517 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb6c1599d0 0x7fbb6c09da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb72360700 1 -- 172.21.15.46:0/3003080517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb6c15d8c0 msgr2=0x7fbb6c09df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb72360700 1 --2- 172.21.15.46:0/3003080517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb6c15d8c0 0x7fbb6c09df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb72360700 1 -- 172.21.15.46:0/3003080517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb680093c0 con 0x7fbb6c0ab1c0 2024-01-26T23:30:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb71b5f700 1 --2- 172.21.15.46:0/3003080517 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb6c1599d0 0x7fbb6c09da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb72360700 1 --2- 172.21.15.46:0/3003080517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb6c0ab1c0 0x7fbb6c0ac390 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009ec0 tx=0x7fbb68016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb627fc700 1 -- 172.21.15.46:0/3003080517 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6801d910 con 0x7fbb6c0ab1c0 2024-01-26T23:30:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb627fc700 1 -- 172.21.15.46:0/3003080517 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6801da70 con 0x7fbb6c0ab1c0 2024-01-26T23:30:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb745c4700 1 -- 172.21.15.46:0/3003080517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb6c09a690 con 0x7fbb6c0ab1c0 2024-01-26T23:30:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb627fc700 1 -- 172.21.15.46:0/3003080517 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6801de30 con 0x7fbb6c0ab1c0 2024-01-26T23:30:27.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.169+0000 7fbb745c4700 1 -- 172.21.15.46:0/3003080517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb6c0ac8d0 con 0x7fbb6c0ab1c0 2024-01-26T23:30:27.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.170+0000 7fbb745c4700 1 -- 172.21.15.46:0/3003080517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb6c09a880 con 0x7fbb6c0ab1c0 2024-01-26T23:30:27.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.171+0000 7fbb627fc700 1 -- 172.21.15.46:0/3003080517 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbb6801b690 con 0x7fbb6c0ab1c0 2024-01-26T23:30:27.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.171+0000 7fbb627fc700 1 --2- 172.21.15.46:0/3003080517 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbb50066f30 0x7fbb500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.171+0000 7fbb627fc700 1 -- 172.21.15.46:0/3003080517 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fbb680867e0 con 0x7fbb6c0ab1c0 2024-01-26T23:30:27.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.172+0000 7fbb71b5f700 1 --2- 172.21.15.46:0/3003080517 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbb50066f30 0x7fbb500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.172+0000 7fbb71b5f700 1 --2- 172.21.15.46:0/3003080517 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbb50066f30 0x7fbb500693e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c069de0 tx=0x7fbb5c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:27.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.176+0000 7fbb627fc700 1 -- 172.21.15.46:0/3003080517 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbb680536c0 con 0x7fbb6c0ab1c0 2024-01-26T23:30:27.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.380+0000 7fbb745c4700 1 -- 172.21.15.46:0/3003080517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbb6c04c710 con 0x7fbb6c0ab1c0 2024-01-26T23:30:27.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.381+0000 7fbb627fc700 1 -- 172.21.15.46:0/3003080517 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbb68056ce0 con 0x7fbb6c0ab1c0 2024-01-26T23:30:27.382 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:30:27.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.383+0000 7fbb745c4700 1 -- 172.21.15.46:0/3003080517 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbb50066f30 msgr2=0x7fbb500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.383+0000 7fbb745c4700 1 --2- 172.21.15.46:0/3003080517 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbb50066f30 0x7fbb500693e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c069de0 tx=0x7fbb5c009040 comp rx=0 tx=0).stop 2024-01-26T23:30:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.383+0000 7fbb745c4700 1 -- 172.21.15.46:0/3003080517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb6c0ab1c0 msgr2=0x7fbb6c0ac390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.383+0000 7fbb745c4700 1 --2- 172.21.15.46:0/3003080517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb6c0ab1c0 0x7fbb6c0ac390 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009ec0 tx=0x7fbb68016000 comp rx=0 tx=0).stop 2024-01-26T23:30:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.383+0000 7fbb745c4700 1 -- 172.21.15.46:0/3003080517 shutdown_connections 2024-01-26T23:30:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.383+0000 7fbb745c4700 1 --2- 172.21.15.46:0/3003080517 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbb50066f30 0x7fbb500693e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.383+0000 7fbb745c4700 1 --2- 172.21.15.46:0/3003080517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb6c0ab1c0 0x7fbb6c0ac390 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.383+0000 7fbb745c4700 1 --2- 172.21.15.46:0/3003080517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb6c15d8c0 0x7fbb6c09df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.383+0000 7fbb745c4700 1 --2- 172.21.15.46:0/3003080517 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb6c1599d0 0x7fbb6c09da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.383+0000 7fbb745c4700 1 -- 172.21.15.46:0/3003080517 >> 172.21.15.46:0/3003080517 conn(0x7fbb6c06f840 msgr2=0x7fbb6c0b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.383+0000 7fbb745c4700 1 -- 172.21.15.46:0/3003080517 shutdown_connections 2024-01-26T23:30:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.384+0000 7fbb745c4700 1 -- 172.21.15.46:0/3003080517 wait complete. 2024-01-26T23:30:27.397 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-26T23:30:27.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.533+0000 7f49e630d700 1 Processor -- start 2024-01-26T23:30:27.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.533+0000 7f49e630d700 1 -- start start 2024-01-26T23:30:27.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.534+0000 7f49e630d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f49e0153c40 0x7f49e006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.534+0000 7f49e630d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49e00b1e80 0x7f49e00692e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.534+0000 7f49e630d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e015ad60 con 0x7f49e0154010 2024-01-26T23:30:27.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.534+0000 7f49e630d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e00b1d20 con 0x7f49e0153c40 2024-01-26T23:30:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.534+0000 7f49e630d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e015a150 con 0x7f49e00b1e80 2024-01-26T23:30:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.534+0000 7f49dffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f49e0153c40 0x7f49e006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.534+0000 7f49df7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f49e0154010 0x7f49e006dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41996/0 (socket says 172.21.15.46:41996) 2024-01-26T23:30:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.534+0000 7f49df7fe700 1 -- 172.21.15.46:0/2912490878 learned_addr learned my addr 172.21.15.46:0/2912490878 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.534+0000 7f49e48aa700 1 --2- 172.21.15.46:0/2912490878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49e00b1e80 0x7f49e00692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.534+0000 7f49dffff700 1 -- 172.21.15.46:0/2912490878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49e00b1e80 msgr2=0x7f49e00692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.534+0000 7f49dffff700 1 --2- 172.21.15.46:0/2912490878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49e00b1e80 0x7f49e00692e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.534+0000 7f49dffff700 1 -- 172.21.15.46:0/2912490878 >> v1:172.21.15.46:6789/0 conn(0x7f49e0154010 legacy=0x7f49e006dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.534+0000 7f49dffff700 1 -- 172.21.15.46:0/2912490878 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e015a430 con 0x7f49e0153c40 2024-01-26T23:30:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.534+0000 7f49deffd700 1 -- 172.21.15.46:0/2912490878 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790865203 0 0) 0x7f49e015ad60 con 0x7f49e0154010 2024-01-26T23:30:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.534+0000 7f49dffff700 1 --2- 172.21.15.46:0/2912490878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0153c40 0x7f49e006d6a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f49d00098e0 tx=0x7f49d00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45ba12075f041924 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.535+0000 7f49deffd700 1 -- 172.21.15.46:0/2912490878 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0009e80 con 0x7f49e0153c40 2024-01-26T23:30:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.535+0000 7f49deffd700 1 -- 172.21.15.46:0/2912490878 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d000b860 con 0x7f49e0153c40 2024-01-26T23:30:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.535+0000 7f49deffd700 1 -- 172.21.15.46:0/2912490878 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d000bbe0 con 0x7f49e0153c40 2024-01-26T23:30:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.535+0000 7f49e630d700 1 -- 172.21.15.46:0/2912490878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0153c40 msgr2=0x7f49e006d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.535+0000 7f49e630d700 1 --2- 172.21.15.46:0/2912490878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0153c40 0x7f49e006d6a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f49d00098e0 tx=0x7f49d00091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.535+0000 7f49e630d700 1 -- 172.21.15.46:0/2912490878 shutdown_connections 2024-01-26T23:30:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.535+0000 7f49e630d700 1 --2- 172.21.15.46:0/2912490878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49e00b1e80 0x7f49e00692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.535+0000 7f49e630d700 1 --2- 172.21.15.46:0/2912490878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0153c40 0x7f49e006d6a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.535+0000 7f49e630d700 1 -- 172.21.15.46:0/2912490878 >> 172.21.15.46:0/2912490878 conn(0x7f49e0067930 msgr2=0x7f49e0067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.535+0000 7f49e630d700 1 -- 172.21.15.46:0/2912490878 shutdown_connections 2024-01-26T23:30:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.535+0000 7f49e630d700 1 -- 172.21.15.46:0/2912490878 wait complete. 2024-01-26T23:30:27.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.536+0000 7f49e630d700 1 Processor -- start 2024-01-26T23:30:27.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.537+0000 7f49e630d700 1 -- start start 2024-01-26T23:30:27.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.537+0000 7f49e630d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49e00b1e80 0x7f49e013fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.537+0000 7f49e630d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49e0153c40 0x7f49e0150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.537+0000 7f49e630d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0154010 0x7f49e0150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.537+0000 7f49e630d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e00af600 con 0x7f49e0154010 2024-01-26T23:30:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.537+0000 7f49e630d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e0140530 con 0x7f49e0153c40 2024-01-26T23:30:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.537+0000 7f49e630d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e00ae9f0 con 0x7f49e00b1e80 2024-01-26T23:30:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.538+0000 7f49e48aa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0154010 0x7f49e0150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.538+0000 7f49e48aa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0154010 0x7f49e0150870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54370/0 (socket says 172.21.15.46:54370) 2024-01-26T23:30:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.538+0000 7f49e48aa700 1 -- 172.21.15.46:0/3710107165 learned_addr learned my addr 172.21.15.46:0/3710107165 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.538+0000 7f49dffff700 1 --2- 172.21.15.46:0/3710107165 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49e00b1e80 0x7f49e013fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.538+0000 7f49df7fe700 1 --2- 172.21.15.46:0/3710107165 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49e0153c40 0x7f49e0150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.538+0000 7f49e48aa700 1 -- 172.21.15.46:0/3710107165 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49e00b1e80 msgr2=0x7f49e013fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.538+0000 7f49e48aa700 1 --2- 172.21.15.46:0/3710107165 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49e00b1e80 0x7f49e013fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.538+0000 7f49e48aa700 1 -- 172.21.15.46:0/3710107165 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49e0153c40 msgr2=0x7f49e0150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.538+0000 7f49e48aa700 1 --2- 172.21.15.46:0/3710107165 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49e0153c40 0x7f49e0150330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.538+0000 7f49e48aa700 1 -- 172.21.15.46:0/3710107165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d0009420 con 0x7f49e0154010 2024-01-26T23:30:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.538+0000 7f49df7fe700 1 --2- 172.21.15.46:0/3710107165 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49e0153c40 0x7f49e0150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:30:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.538+0000 7f49e48aa700 1 --2- 172.21.15.46:0/3710107165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0154010 0x7f49e0150870 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009fa0 tx=0x7f49d8009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.538+0000 7f49c7fff700 1 -- 172.21.15.46:0/3710107165 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d800ed20 con 0x7f49e0154010 2024-01-26T23:30:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.538+0000 7f49c7fff700 1 -- 172.21.15.46:0/3710107165 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d800ee80 con 0x7f49e0154010 2024-01-26T23:30:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.538+0000 7f49e630d700 1 -- 172.21.15.46:0/3710107165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e00aecd0 con 0x7f49e0154010 2024-01-26T23:30:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.538+0000 7f49c7fff700 1 -- 172.21.15.46:0/3710107165 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d8014670 con 0x7f49e0154010 2024-01-26T23:30:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.538+0000 7f49e630d700 1 -- 172.21.15.46:0/3710107165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e00af290 con 0x7f49e0154010 2024-01-26T23:30:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.539+0000 7f49e630d700 1 -- 172.21.15.46:0/3710107165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e004c710 con 0x7f49e0154010 2024-01-26T23:30:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.540+0000 7f49c7fff700 1 -- 172.21.15.46:0/3710107165 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f49d80149e0 con 0x7f49e0154010 2024-01-26T23:30:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.540+0000 7f49c7fff700 1 --2- 172.21.15.46:0/3710107165 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f49cc066f30 0x7f49cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.540+0000 7f49c7fff700 1 -- 172.21.15.46:0/3710107165 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f49d8084fc0 con 0x7f49e0154010 2024-01-26T23:30:27.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.541+0000 7f49dffff700 1 --2- 172.21.15.46:0/3710107165 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f49cc066f30 0x7f49cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.541+0000 7f49dffff700 1 --2- 172.21.15.46:0/3710107165 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f49cc066f30 0x7f49cc0693e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f49d00091e0 tx=0x7f49d0017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:27.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.545+0000 7f49c7fff700 1 -- 172.21.15.46:0/3710107165 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f49d8051f40 con 0x7f49e0154010 2024-01-26T23:30:27.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.748+0000 7f49e630d700 1 -- 172.21.15.46:0/3710107165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f49e00af740 con 0x7f49e0154010 2024-01-26T23:30:27.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.748+0000 7f49c7fff700 1 -- 172.21.15.46:0/3710107165 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f49d8055560 con 0x7f49e0154010 2024-01-26T23:30:27.749 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:30:27.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.751+0000 7f49e630d700 1 -- 172.21.15.46:0/3710107165 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f49cc066f30 msgr2=0x7f49cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.751+0000 7f49e630d700 1 --2- 172.21.15.46:0/3710107165 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f49cc066f30 0x7f49cc0693e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f49d00091e0 tx=0x7f49d0017000 comp rx=0 tx=0).stop 2024-01-26T23:30:27.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.751+0000 7f49e630d700 1 -- 172.21.15.46:0/3710107165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0154010 msgr2=0x7f49e0150870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.751+0000 7f49e630d700 1 --2- 172.21.15.46:0/3710107165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0154010 0x7f49e0150870 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009fa0 tx=0x7f49d8009900 comp rx=0 tx=0).stop 2024-01-26T23:30:27.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.751+0000 7f49e630d700 1 -- 172.21.15.46:0/3710107165 shutdown_connections 2024-01-26T23:30:27.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.751+0000 7f49e630d700 1 --2- 172.21.15.46:0/3710107165 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f49cc066f30 0x7f49cc0693e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.751+0000 7f49e630d700 1 --2- 172.21.15.46:0/3710107165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0154010 0x7f49e0150870 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.751+0000 7f49e630d700 1 --2- 172.21.15.46:0/3710107165 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49e00b1e80 0x7f49e013fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.751+0000 7f49e630d700 1 --2- 172.21.15.46:0/3710107165 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49e0153c40 0x7f49e0150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.751+0000 7f49e630d700 1 -- 172.21.15.46:0/3710107165 >> 172.21.15.46:0/3710107165 conn(0x7f49e0067930 msgr2=0x7f49e00692e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:27.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.751+0000 7f49e630d700 1 -- 172.21.15.46:0/3710107165 shutdown_connections 2024-01-26T23:30:27.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.751+0000 7f49e630d700 1 -- 172.21.15.46:0/3710107165 wait complete. 2024-01-26T23:30:27.765 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-26T23:30:27.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.898+0000 7f6cfbaa1700 1 Processor -- start 2024-01-26T23:30:27.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.898+0000 7f6cfbaa1700 1 -- start start 2024-01-26T23:30:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.899+0000 7f6cfbaa1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6cf41597d0 0x7f6cf406ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.899+0000 7f6cfbaa1700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6cf415ad40 0x7f6cf406f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.899+0000 7f6cfbaa1700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf40cbfb0 con 0x7f6cf4159ba0 2024-01-26T23:30:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.899+0000 7f6cfbaa1700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf415a8b0 con 0x7f6cf415ad40 2024-01-26T23:30:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.899+0000 7f6cfbaa1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf40cbcd0 con 0x7f6cf41597d0 2024-01-26T23:30:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.900+0000 7f6cf983d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6cf41597d0 0x7f6cf406ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.900+0000 7f6cf903c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6cf415ad40 0x7f6cf406f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.900+0000 7f6cf983d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6cf41597d0 0x7f6cf406ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:47428/0 (socket says 172.21.15.46:47428) 2024-01-26T23:30:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.900+0000 7f6cf903c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6cf415ad40 0x7f6cf406f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:45008/0 (socket says 172.21.15.46:45008) 2024-01-26T23:30:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.900+0000 7f6cf983d700 1 -- 172.21.15.46:0/1838686865 learned_addr learned my addr 172.21.15.46:0/1838686865 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.900+0000 7f6cf983d700 1 -- 172.21.15.46:0/1838686865 >> v2:172.21.15.106:3300/0 conn(0x7f6cf415ad40 msgr2=0x7f6cf406f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.900+0000 7f6cf983d700 1 --2- 172.21.15.46:0/1838686865 >> v2:172.21.15.106:3300/0 conn(0x7f6cf415ad40 0x7f6cf406f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.900+0000 7f6cf983d700 1 -- 172.21.15.46:0/1838686865 >> v1:172.21.15.106:6789/0 conn(0x7f6cf4159ba0 legacy=0x7f6cf40cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.900+0000 7f6cf983d700 1 -- 172.21.15.46:0/1838686865 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf415a260 con 0x7f6cf41597d0 2024-01-26T23:30:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.900+0000 7f6cf903c700 1 --2- 172.21.15.46:0/1838686865 >> v2:172.21.15.106:3300/0 conn(0x7f6cf415ad40 0x7f6cf406f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.900+0000 7f6cf983d700 1 --2- 172.21.15.46:0/1838686865 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6cf41597d0 0x7f6cf406ec50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800bc50 tx=0x7f6ce800b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e4aac7e77dbff62 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.901+0000 7f6cf883b700 1 -- 172.21.15.46:0/1838686865 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce800dbf0 con 0x7f6cf41597d0 2024-01-26T23:30:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.901+0000 7f6cf883b700 1 -- 172.21.15.46:0/1838686865 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce800dd50 con 0x7f6cf41597d0 2024-01-26T23:30:27.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.901+0000 7f6cf883b700 1 -- 172.21.15.46:0/1838686865 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce80196d0 con 0x7f6cf41597d0 2024-01-26T23:30:27.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.901+0000 7f6cfbaa1700 1 -- 172.21.15.46:0/1838686865 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6cf41597d0 msgr2=0x7f6cf406ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.901+0000 7f6cfbaa1700 1 --2- 172.21.15.46:0/1838686865 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6cf41597d0 0x7f6cf406ec50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800bc50 tx=0x7f6ce800b4f0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.901+0000 7f6cfbaa1700 1 -- 172.21.15.46:0/1838686865 shutdown_connections 2024-01-26T23:30:27.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.901+0000 7f6cfbaa1700 1 --2- 172.21.15.46:0/1838686865 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6cf41597d0 0x7f6cf406ec50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.901+0000 7f6cfbaa1700 1 --2- 172.21.15.46:0/1838686865 >> v2:172.21.15.106:3300/0 conn(0x7f6cf415ad40 0x7f6cf406f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.901+0000 7f6cfbaa1700 1 -- 172.21.15.46:0/1838686865 >> 172.21.15.46:0/1838686865 conn(0x7f6cf40ca9a0 msgr2=0x7f6cf4068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.901+0000 7f6cfbaa1700 1 -- 172.21.15.46:0/1838686865 shutdown_connections 2024-01-26T23:30:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.901+0000 7f6cfbaa1700 1 -- 172.21.15.46:0/1838686865 wait complete. 2024-01-26T23:30:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.902+0000 7f6cfbaa1700 1 Processor -- start 2024-01-26T23:30:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.903+0000 7f6cfbaa1700 1 -- start start 2024-01-26T23:30:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.903+0000 7f6cfbaa1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6cf41597d0 0x7f6cf414f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.903+0000 7f6cfbaa1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6cf4159ba0 0x7f6cf40af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.903+0000 7f6cfbaa1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf415ad40 0x7f6cf4168d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.903+0000 7f6cfbaa1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf40b2d10 con 0x7f6cf415ad40 2024-01-26T23:30:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.903+0000 7f6cfbaa1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf40b31a0 con 0x7f6cf41597d0 2024-01-26T23:30:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.903+0000 7f6cfbaa1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf414f810 con 0x7f6cf4159ba0 2024-01-26T23:30:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.903+0000 7f6cfa03e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf415ad40 0x7f6cf4168d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.903+0000 7f6cf983d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6cf41597d0 0x7f6cf414f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.903+0000 7f6cf903c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6cf4159ba0 0x7f6cf40af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.903+0000 7f6cf983d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6cf41597d0 0x7f6cf414f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:45018/0 (socket says 172.21.15.46:45018) 2024-01-26T23:30:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.903+0000 7f6cfa03e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf415ad40 0x7f6cf4168d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54378/0 (socket says 172.21.15.46:54378) 2024-01-26T23:30:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.903+0000 7f6cf983d700 1 -- 172.21.15.46:0/157519673 learned_addr learned my addr 172.21.15.46:0/157519673 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.904+0000 7f6cf983d700 1 -- 172.21.15.46:0/157519673 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6cf4159ba0 msgr2=0x7f6cf40af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.904+0000 7f6cf983d700 1 --2- 172.21.15.46:0/157519673 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6cf4159ba0 0x7f6cf40af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.904+0000 7f6cf983d700 1 -- 172.21.15.46:0/157519673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf415ad40 msgr2=0x7f6cf4168d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.904+0000 7f6cf983d700 1 --2- 172.21.15.46:0/157519673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf415ad40 0x7f6cf4168d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.904+0000 7f6cf983d700 1 -- 172.21.15.46:0/157519673 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf0009420 con 0x7f6cf41597d0 2024-01-26T23:30:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.904+0000 7f6cfa03e700 1 --2- 172.21.15.46:0/157519673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf415ad40 0x7f6cf4168d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.904+0000 7f6cf983d700 1 --2- 172.21.15.46:0/157519673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6cf41597d0 0x7f6cf414f2d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6cf40cbe10 tx=0x7f6ce801b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.904+0000 7f6ce5ffb700 1 -- 172.21.15.46:0/157519673 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce800dbf0 con 0x7f6cf41597d0 2024-01-26T23:30:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.904+0000 7f6ce5ffb700 1 -- 172.21.15.46:0/157519673 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce800dec0 con 0x7f6cf41597d0 2024-01-26T23:30:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.904+0000 7f6cfbaa1700 1 -- 172.21.15.46:0/157519673 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce800b790 con 0x7f6cf41597d0 2024-01-26T23:30:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.904+0000 7f6ce5ffb700 1 -- 172.21.15.46:0/157519673 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce80108d0 con 0x7f6cf41597d0 2024-01-26T23:30:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.905+0000 7f6cfbaa1700 1 -- 172.21.15.46:0/157519673 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf40b16d0 con 0x7f6cf41597d0 2024-01-26T23:30:27.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.905+0000 7f6cfbaa1700 1 -- 172.21.15.46:0/157519673 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf404c710 con 0x7f6cf41597d0 2024-01-26T23:30:27.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.906+0000 7f6ce5ffb700 1 -- 172.21.15.46:0/157519673 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6ce8041b80 con 0x7f6cf41597d0 2024-01-26T23:30:27.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.906+0000 7f6ce5ffb700 1 --2- 172.21.15.46:0/157519673 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6cd8066f30 0x7f6cd80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:27.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.907+0000 7f6ce5ffb700 1 -- 172.21.15.46:0/157519673 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f6ce8014070 con 0x7f6cf41597d0 2024-01-26T23:30:27.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.907+0000 7f6cf903c700 1 --2- 172.21.15.46:0/157519673 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6cd8066f30 0x7f6cd80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:27.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.907+0000 7f6cf903c700 1 --2- 172.21.15.46:0/157519673 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6cd8066f30 0x7f6cd80693e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6cf41524b0 tx=0x7f6cf00091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:27.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:27.911+0000 7f6ce5ffb700 1 -- 172.21.15.46:0/157519673 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6ce8053600 con 0x7f6cf41597d0 2024-01-26T23:30:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.114+0000 7f6cfbaa1700 1 -- 172.21.15.46:0/157519673 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6cf40b2ff0 con 0x7f6cf41597d0 2024-01-26T23:30:28.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.115+0000 7f6ce5ffb700 1 -- 172.21.15.46:0/157519673 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6ce8056c20 con 0x7f6cf41597d0 2024-01-26T23:30:28.116 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-26T23:30:28.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.117+0000 7f6cfbaa1700 1 -- 172.21.15.46:0/157519673 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6cd8066f30 msgr2=0x7f6cd80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:28.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.117+0000 7f6cfbaa1700 1 --2- 172.21.15.46:0/157519673 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6cd8066f30 0x7f6cd80693e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6cf41524b0 tx=0x7f6cf00091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.117+0000 7f6cfbaa1700 1 -- 172.21.15.46:0/157519673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6cf41597d0 msgr2=0x7f6cf414f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.117+0000 7f6cfbaa1700 1 --2- 172.21.15.46:0/157519673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6cf41597d0 0x7f6cf414f2d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6cf40cbe10 tx=0x7f6ce801b000 comp rx=0 tx=0).stop 2024-01-26T23:30:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.117+0000 7f6cfbaa1700 1 -- 172.21.15.46:0/157519673 shutdown_connections 2024-01-26T23:30:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.117+0000 7f6cfbaa1700 1 --2- 172.21.15.46:0/157519673 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6cd8066f30 0x7f6cd80693e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.117+0000 7f6cfbaa1700 1 --2- 172.21.15.46:0/157519673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf415ad40 0x7f6cf4168d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.117+0000 7f6cfbaa1700 1 --2- 172.21.15.46:0/157519673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6cf41597d0 0x7f6cf414f2d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.117+0000 7f6cfbaa1700 1 --2- 172.21.15.46:0/157519673 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6cf4159ba0 0x7f6cf40af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.117+0000 7f6cfbaa1700 1 -- 172.21.15.46:0/157519673 >> 172.21.15.46:0/157519673 conn(0x7f6cf40ca9a0 msgr2=0x7f6cf406c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.117+0000 7f6cfbaa1700 1 -- 172.21.15.46:0/157519673 shutdown_connections 2024-01-26T23:30:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.117+0000 7f6cfbaa1700 1 -- 172.21.15.46:0/157519673 wait complete. 2024-01-26T23:30:28.131 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-01-26T23:30:28.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.265+0000 7f378788c700 1 Processor -- start 2024-01-26T23:30:28.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.265+0000 7f378788c700 1 -- start start 2024-01-26T23:30:28.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.266+0000 7f378788c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3780067af0 0x7f3780067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:28.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.266+0000 7f378788c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378006fdb0 con 0x7f37801597e0 2024-01-26T23:30:28.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.266+0000 7f378788c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378006f1a0 con 0x7f378006ebe0 2024-01-26T23:30:28.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.266+0000 7f378788c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378006f480 con 0x7f3780067af0 2024-01-26T23:30:28.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.267+0000 7f3785e29700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f37801597e0 0x7f3780159c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42004/0 (socket says 172.21.15.46:42004) 2024-01-26T23:30:28.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.267+0000 7f3785e29700 1 -- 172.21.15.46:0/2476763651 learned_addr learned my addr 172.21.15.46:0/2476763651 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:28.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.267+0000 7f3785628700 1 --2- 172.21.15.46:0/2476763651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3780067af0 0x7f3780067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:28.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.267+0000 7f3777fff700 1 -- 172.21.15.46:0/2476763651 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034255695 0 0) 0x7f378006fdb0 con 0x7f37801597e0 2024-01-26T23:30:28.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.267+0000 7f3777fff700 1 -- 172.21.15.46:0/2476763651 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f376c003570 con 0x7f37801597e0 2024-01-26T23:30:28.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.267+0000 7f3777fff700 1 -- 172.21.15.46:0/2476763651 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956314140 0 0) 0x7f378006f1a0 con 0x7f378006ebe0 2024-01-26T23:30:28.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.267+0000 7f3777fff700 1 -- 172.21.15.46:0/2476763651 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f378006fdb0 con 0x7f378006ebe0 2024-01-26T23:30:28.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.267+0000 7f3785628700 1 -- 172.21.15.46:0/2476763651 >> v1:172.21.15.106:6789/0 conn(0x7f378006ebe0 legacy=0x7f37800ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:28.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.267+0000 7f3785628700 1 -- 172.21.15.46:0/2476763651 >> v1:172.21.15.46:6789/0 conn(0x7f37801597e0 legacy=0x7f3780159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:28.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.267+0000 7f3785628700 1 -- 172.21.15.46:0/2476763651 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378006f760 con 0x7f3780067af0 2024-01-26T23:30:28.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.267+0000 7f3785628700 1 --2- 172.21.15.46:0/2476763651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3780067af0 0x7f3780067ec0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3770009910 tx=0x7f37700091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b69efc2124fc8d0d server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:28.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.267+0000 7f3777fff700 1 -- 172.21.15.46:0/2476763651 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3770011630 con 0x7f3780067af0 2024-01-26T23:30:28.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.268+0000 7f3777fff700 1 -- 172.21.15.46:0/2476763651 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3770011790 con 0x7f3780067af0 2024-01-26T23:30:28.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.268+0000 7f3777fff700 1 -- 172.21.15.46:0/2476763651 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3770011b30 con 0x7f3780067af0 2024-01-26T23:30:28.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.268+0000 7f378788c700 1 -- 172.21.15.46:0/2476763651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3780067af0 msgr2=0x7f3780067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:28.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.268+0000 7f378788c700 1 --2- 172.21.15.46:0/2476763651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3780067af0 0x7f3780067ec0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3770009910 tx=0x7f37700091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.268+0000 7f378788c700 1 -- 172.21.15.46:0/2476763651 shutdown_connections 2024-01-26T23:30:28.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.268+0000 7f378788c700 1 --2- 172.21.15.46:0/2476763651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3780067af0 0x7f3780067ec0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.268+0000 7f378788c700 1 -- 172.21.15.46:0/2476763651 >> 172.21.15.46:0/2476763651 conn(0x7f37800cdf80 msgr2=0x7f378006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:28.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.268+0000 7f378788c700 1 -- 172.21.15.46:0/2476763651 shutdown_connections 2024-01-26T23:30:28.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.268+0000 7f378788c700 1 -- 172.21.15.46:0/2476763651 wait complete. 2024-01-26T23:30:28.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.269+0000 7f378788c700 1 Processor -- start 2024-01-26T23:30:28.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.270+0000 7f378788c700 1 -- start start 2024-01-26T23:30:28.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.270+0000 7f378788c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3780067af0 0x7f37801587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:28.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.270+0000 7f378788c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f378006ebe0 0x7f378015b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:28.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.270+0000 7f378788c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f37801597e0 0x7f378015bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:28.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.270+0000 7f378788c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378015c290 con 0x7f3780067af0 2024-01-26T23:30:28.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.270+0000 7f378788c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378015c570 con 0x7f378006ebe0 2024-01-26T23:30:28.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.270+0000 7f378788c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378015c850 con 0x7f37801597e0 2024-01-26T23:30:28.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.271+0000 7f3785628700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3780067af0 0x7f37801587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:28.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.271+0000 7f3785628700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3780067af0 0x7f37801587e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54384/0 (socket says 172.21.15.46:54384) 2024-01-26T23:30:28.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.271+0000 7f3785628700 1 -- 172.21.15.46:0/4014816477 learned_addr learned my addr 172.21.15.46:0/4014816477 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:28.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.271+0000 7f3785e29700 1 --2- 172.21.15.46:0/4014816477 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f37801597e0 0x7f378015bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:28.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.271+0000 7f3784e27700 1 --2- 172.21.15.46:0/4014816477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f378006ebe0 0x7f378015b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:28.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.271+0000 7f3785628700 1 -- 172.21.15.46:0/4014816477 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f37801597e0 msgr2=0x7f378015bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:28.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.271+0000 7f3785628700 1 --2- 172.21.15.46:0/4014816477 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f37801597e0 0x7f378015bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.271+0000 7f3785628700 1 -- 172.21.15.46:0/4014816477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f378006ebe0 msgr2=0x7f378015b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:28.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.271+0000 7f3785628700 1 --2- 172.21.15.46:0/4014816477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f378006ebe0 0x7f378015b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.271+0000 7f3785628700 1 -- 172.21.15.46:0/4014816477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3770009420 con 0x7f3780067af0 2024-01-26T23:30:28.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.271+0000 7f3785628700 1 --2- 172.21.15.46:0/4014816477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3780067af0 0x7f37801587e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3770009f00 tx=0x7f3770017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:28.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.271+0000 7f37757fa700 1 -- 172.21.15.46:0/4014816477 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377001f070 con 0x7f3780067af0 2024-01-26T23:30:28.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.271+0000 7f37757fa700 1 -- 172.21.15.46:0/4014816477 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3770002f70 con 0x7f3780067af0 2024-01-26T23:30:28.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.271+0000 7f37757fa700 1 -- 172.21.15.46:0/4014816477 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3770006af0 con 0x7f3780067af0 2024-01-26T23:30:28.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.271+0000 7f378788c700 1 -- 172.21.15.46:0/4014816477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f378015cb30 con 0x7f3780067af0 2024-01-26T23:30:28.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.271+0000 7f378788c700 1 -- 172.21.15.46:0/4014816477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37800cbab0 con 0x7f3780067af0 2024-01-26T23:30:28.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.272+0000 7f378788c700 1 -- 172.21.15.46:0/4014816477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f378004c710 con 0x7f3780067af0 2024-01-26T23:30:28.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.273+0000 7f37757fa700 1 -- 172.21.15.46:0/4014816477 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3770023070 con 0x7f3780067af0 2024-01-26T23:30:28.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.273+0000 7f37757fa700 1 --2- 172.21.15.46:0/4014816477 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3764066f30 0x7f37640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:28.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.273+0000 7f37757fa700 1 -- 172.21.15.46:0/4014816477 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f37700858e0 con 0x7f3780067af0 2024-01-26T23:30:28.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.274+0000 7f3784e27700 1 --2- 172.21.15.46:0/4014816477 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3764066f30 0x7f37640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:28.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.274+0000 7f3784e27700 1 --2- 172.21.15.46:0/4014816477 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3764066f30 0x7f37640693e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f377c007fd0 tx=0x7f377c007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:28.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.278+0000 7f37757fa700 1 -- 172.21.15.46:0/4014816477 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3770052860 con 0x7f3780067af0 2024-01-26T23:30:28.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.479+0000 7f378788c700 1 -- 172.21.15.46:0/4014816477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f378016ad40 con 0x7f3780067af0 2024-01-26T23:30:28.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.479+0000 7f37757fa700 1 -- 172.21.15.46:0/4014816477 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f377002a0a0 con 0x7f3780067af0 2024-01-26T23:30:28.480 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:30:28.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.481+0000 7f378788c700 1 -- 172.21.15.46:0/4014816477 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3764066f30 msgr2=0x7f37640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:28.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.481+0000 7f378788c700 1 --2- 172.21.15.46:0/4014816477 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3764066f30 0x7f37640693e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f377c007fd0 tx=0x7f377c007d50 comp rx=0 tx=0).stop 2024-01-26T23:30:28.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.481+0000 7f378788c700 1 -- 172.21.15.46:0/4014816477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3780067af0 msgr2=0x7f37801587e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:28.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.481+0000 7f378788c700 1 --2- 172.21.15.46:0/4014816477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3780067af0 0x7f37801587e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3770009f00 tx=0x7f3770017000 comp rx=0 tx=0).stop 2024-01-26T23:30:28.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.481+0000 7f378788c700 1 -- 172.21.15.46:0/4014816477 shutdown_connections 2024-01-26T23:30:28.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.481+0000 7f378788c700 1 --2- 172.21.15.46:0/4014816477 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3764066f30 0x7f37640693e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.481+0000 7f378788c700 1 --2- 172.21.15.46:0/4014816477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3780067af0 0x7f37801587e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.481+0000 7f378788c700 1 --2- 172.21.15.46:0/4014816477 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f37801597e0 0x7f378015bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.481+0000 7f378788c700 1 --2- 172.21.15.46:0/4014816477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f378006ebe0 0x7f378015b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.481+0000 7f378788c700 1 -- 172.21.15.46:0/4014816477 >> 172.21.15.46:0/4014816477 conn(0x7f37800cdf80 msgr2=0x7f37800cca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:28.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.481+0000 7f378788c700 1 -- 172.21.15.46:0/4014816477 shutdown_connections 2024-01-26T23:30:28.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.481+0000 7f378788c700 1 -- 172.21.15.46:0/4014816477 wait complete. 2024-01-26T23:30:28.495 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:28.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.625+0000 7fa021e9b700 1 Processor -- start 2024-01-26T23:30:28.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.625+0000 7fa021e9b700 1 -- start start 2024-01-26T23:30:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.626+0000 7fa021e9b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa01c0ccc20 0x7fa01c068340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.626+0000 7fa021e9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa01c0ccff0 0x7fa01c06eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.626+0000 7fa021e9b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa01c0b2590 0x7fa01c0b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.626+0000 7fa021e9b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa01c0b2f10 con 0x7fa01c0ccff0 2024-01-26T23:30:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.626+0000 7fa021e9b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa01c158020 con 0x7fa01c0ccc20 2024-01-26T23:30:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.626+0000 7fa021e9b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c158300 con 0x7fa01c0b2590 2024-01-26T23:30:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.626+0000 7fa01affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa01c0ccff0 0x7fa01c06eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.626+0000 7fa01affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa01c0ccff0 0x7fa01c06eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54398/0 (socket says 172.21.15.46:54398) 2024-01-26T23:30:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.626+0000 7fa01affd700 1 -- 172.21.15.46:0/2030220727 learned_addr learned my addr 172.21.15.46:0/2030220727 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.626+0000 7fa01bfff700 1 --2- 172.21.15.46:0/2030220727 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa01c0b2590 0x7fa01c0b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.626+0000 7fa01b7fe700 1 --2- 172.21.15.46:0/2030220727 >> v2:172.21.15.106:3300/0 conn(0x7fa01c0ccc20 0x7fa01c068340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.627+0000 7fa01affd700 1 -- 172.21.15.46:0/2030220727 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa01c0b2590 msgr2=0x7fa01c0b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.627+0000 7fa01affd700 1 --2- 172.21.15.46:0/2030220727 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa01c0b2590 0x7fa01c0b29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.627+0000 7fa01affd700 1 -- 172.21.15.46:0/2030220727 >> v2:172.21.15.106:3300/0 conn(0x7fa01c0ccc20 msgr2=0x7fa01c068340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.627+0000 7fa01affd700 1 --2- 172.21.15.46:0/2030220727 >> v2:172.21.15.106:3300/0 conn(0x7fa01c0ccc20 0x7fa01c068340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.627+0000 7fa01affd700 1 -- 172.21.15.46:0/2030220727 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c1585e0 con 0x7fa01c0ccff0 2024-01-26T23:30:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.627+0000 7fa01affd700 1 --2- 172.21.15.46:0/2030220727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c0ccff0 0x7fa01c06eb70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa01c06e840 tx=0x7fa010009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df85b5fe08d3c34 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.627+0000 7fa01a7fc700 1 -- 172.21.15.46:0/2030220727 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01000b860 con 0x7fa01c0ccff0 2024-01-26T23:30:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.627+0000 7fa01a7fc700 1 -- 172.21.15.46:0/2030220727 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa01000b9c0 con 0x7fa01c0ccff0 2024-01-26T23:30:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.627+0000 7fa01a7fc700 1 -- 172.21.15.46:0/2030220727 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01000bd80 con 0x7fa01c0ccff0 2024-01-26T23:30:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.628+0000 7fa021e9b700 1 -- 172.21.15.46:0/2030220727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c0ccff0 msgr2=0x7fa01c06eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.628+0000 7fa021e9b700 1 --2- 172.21.15.46:0/2030220727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c0ccff0 0x7fa01c06eb70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa01c06e840 tx=0x7fa010009250 comp rx=0 tx=0).stop 2024-01-26T23:30:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.628+0000 7fa021e9b700 1 -- 172.21.15.46:0/2030220727 shutdown_connections 2024-01-26T23:30:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.628+0000 7fa021e9b700 1 --2- 172.21.15.46:0/2030220727 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa01c0b2590 0x7fa01c0b29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.628+0000 7fa021e9b700 1 --2- 172.21.15.46:0/2030220727 >> v2:172.21.15.106:3300/0 conn(0x7fa01c0ccc20 0x7fa01c068340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.628+0000 7fa021e9b700 1 --2- 172.21.15.46:0/2030220727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c0ccff0 0x7fa01c06eb70 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.628+0000 7fa021e9b700 1 -- 172.21.15.46:0/2030220727 >> 172.21.15.46:0/2030220727 conn(0x7fa01c06b630 msgr2=0x7fa01c0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.628+0000 7fa021e9b700 1 -- 172.21.15.46:0/2030220727 shutdown_connections 2024-01-26T23:30:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.628+0000 7fa021e9b700 1 -- 172.21.15.46:0/2030220727 wait complete. 2024-01-26T23:30:28.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.629+0000 7fa021e9b700 1 Processor -- start 2024-01-26T23:30:28.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.629+0000 7fa021e9b700 1 -- start start 2024-01-26T23:30:28.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.630+0000 7fa021e9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c0b2590 0x7fa01c14e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:28.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.630+0000 7fa021e9b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa01c0ccc20 0x7fa01c0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:28.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.630+0000 7fa021e9b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa01c0ccff0 0x7fa01c0c7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:28.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.630+0000 7fa021e9b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c0b0cb0 con 0x7fa01c0b2590 2024-01-26T23:30:28.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.630+0000 7fa021e9b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c0b0f90 con 0x7fa01c0ccc20 2024-01-26T23:30:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.630+0000 7fa021e9b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c0b1270 con 0x7fa01c0ccff0 2024-01-26T23:30:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.630+0000 7fa01b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c0b2590 0x7fa01c14e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.630+0000 7fa01b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c0b2590 0x7fa01c14e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54410/0 (socket says 172.21.15.46:54410) 2024-01-26T23:30:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.630+0000 7fa01b7fe700 1 -- 172.21.15.46:0/2711482307 learned_addr learned my addr 172.21.15.46:0/2711482307 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.630+0000 7fa01bfff700 1 --2- 172.21.15.46:0/2711482307 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa01c0ccff0 0x7fa01c0c7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.630+0000 7fa01affd700 1 --2- 172.21.15.46:0/2711482307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa01c0ccc20 0x7fa01c0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.630+0000 7fa01b7fe700 1 -- 172.21.15.46:0/2711482307 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa01c0ccff0 msgr2=0x7fa01c0c7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.630+0000 7fa01b7fe700 1 --2- 172.21.15.46:0/2711482307 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa01c0ccff0 0x7fa01c0c7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.630+0000 7fa01b7fe700 1 -- 172.21.15.46:0/2711482307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa01c0ccc20 msgr2=0x7fa01c0b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.630+0000 7fa01b7fe700 1 --2- 172.21.15.46:0/2711482307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa01c0ccc20 0x7fa01c0b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.630+0000 7fa01b7fe700 1 -- 172.21.15.46:0/2711482307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0100094f0 con 0x7fa01c0b2590 2024-01-26T23:30:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.630+0000 7fa01affd700 1 --2- 172.21.15.46:0/2711482307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa01c0ccc20 0x7fa01c0b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.631+0000 7fa01b7fe700 1 --2- 172.21.15.46:0/2711482307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c0b2590 0x7fa01c14e930 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00d890 tx=0x7fa00c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.631+0000 7fa0037fe700 1 -- 172.21.15.46:0/2711482307 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c0125f0 con 0x7fa01c0b2590 2024-01-26T23:30:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.631+0000 7fa0037fe700 1 -- 172.21.15.46:0/2711482307 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00c012750 con 0x7fa01c0b2590 2024-01-26T23:30:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.631+0000 7fa021e9b700 1 -- 172.21.15.46:0/2711482307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa01c0b1550 con 0x7fa01c0b2590 2024-01-26T23:30:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.631+0000 7fa0037fe700 1 -- 172.21.15.46:0/2711482307 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c012ad0 con 0x7fa01c0b2590 2024-01-26T23:30:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.631+0000 7fa021e9b700 1 -- 172.21.15.46:0/2711482307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa01c14ee70 con 0x7fa01c0b2590 2024-01-26T23:30:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.632+0000 7fa021e9b700 1 -- 172.21.15.46:0/2711482307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01c04c710 con 0x7fa01c0b2590 2024-01-26T23:30:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.632+0000 7fa0037fe700 1 -- 172.21.15.46:0/2711482307 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa00c01e460 con 0x7fa01c0b2590 2024-01-26T23:30:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.632+0000 7fa0037fe700 1 --2- 172.21.15.46:0/2711482307 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa008066f30 0x7fa0080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.633+0000 7fa0037fe700 1 -- 172.21.15.46:0/2711482307 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fa00c084e10 con 0x7fa01c0b2590 2024-01-26T23:30:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.633+0000 7fa01affd700 1 --2- 172.21.15.46:0/2711482307 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa008066f30 0x7fa0080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.633+0000 7fa01affd700 1 --2- 172.21.15.46:0/2711482307 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa008066f30 0x7fa0080693e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0b0350 tx=0x7fa010017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:28.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.637+0000 7fa0037fe700 1 -- 172.21.15.46:0/2711482307 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa00c055820 con 0x7fa01c0b2590 2024-01-26T23:30:28.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.844+0000 7fa021e9b700 1 -- 172.21.15.46:0/2711482307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa01c1588c0 con 0x7fa01c0b2590 2024-01-26T23:30:28.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.845+0000 7fa0037fe700 1 -- 172.21.15.46:0/2711482307 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7fa00c0553b0 con 0x7fa01c0b2590 2024-01-26T23:30:28.847 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:28.847 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:49.075104+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:28.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.847+0000 7fa021e9b700 1 -- 172.21.15.46:0/2711482307 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa008066f30 msgr2=0x7fa0080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:28.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.847+0000 7fa021e9b700 1 --2- 172.21.15.46:0/2711482307 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa008066f30 0x7fa0080693e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0b0350 tx=0x7fa010017000 comp rx=0 tx=0).stop 2024-01-26T23:30:28.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.847+0000 7fa021e9b700 1 -- 172.21.15.46:0/2711482307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c0b2590 msgr2=0x7fa01c14e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:28.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.847+0000 7fa021e9b700 1 --2- 172.21.15.46:0/2711482307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c0b2590 0x7fa01c14e930 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00d890 tx=0x7fa00c008040 comp rx=0 tx=0).stop 2024-01-26T23:30:28.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.847+0000 7fa021e9b700 1 -- 172.21.15.46:0/2711482307 shutdown_connections 2024-01-26T23:30:28.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.847+0000 7fa021e9b700 1 --2- 172.21.15.46:0/2711482307 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa008066f30 0x7fa0080693e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.847+0000 7fa021e9b700 1 --2- 172.21.15.46:0/2711482307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c0b2590 0x7fa01c14e930 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.847+0000 7fa021e9b700 1 --2- 172.21.15.46:0/2711482307 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa01c0ccff0 0x7fa01c0c7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.847+0000 7fa021e9b700 1 --2- 172.21.15.46:0/2711482307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa01c0ccc20 0x7fa01c0b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.847+0000 7fa021e9b700 1 -- 172.21.15.46:0/2711482307 >> 172.21.15.46:0/2711482307 conn(0x7fa01c06b630 msgr2=0x7fa01c15a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:28.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.847+0000 7fa021e9b700 1 -- 172.21.15.46:0/2711482307 shutdown_connections 2024-01-26T23:30:28.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.847+0000 7fa021e9b700 1 -- 172.21.15.46:0/2711482307 wait complete. 2024-01-26T23:30:28.861 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-26T23:29:42.539759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-26T23:29:43.002669+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-01-26T23:29:46.646864+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-26T23:30:28.862 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:30:28.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.994+0000 7f2d2b26b700 1 Processor -- start 2024-01-26T23:30:28.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.995+0000 7f2d2b26b700 1 -- start start 2024-01-26T23:30:28.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.996+0000 7f2d2b26b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2d241598e0 0x7f2d240c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:28.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.996+0000 7f2d2b26b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d240c9120 0x7f2d240cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:28.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.996+0000 7f2d2b26b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d24069260 con 0x7f2d24159510 2024-01-26T23:30:28.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.996+0000 7f2d2b26b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d240ca930 con 0x7f2d240c9120 2024-01-26T23:30:28.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.996+0000 7f2d2b26b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d24159d50 con 0x7f2d241598e0 2024-01-26T23:30:28.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.996+0000 7f2d29007700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2d24159510 0x7f2d240ca220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42008/0 (socket says 172.21.15.46:42008) 2024-01-26T23:30:28.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.996+0000 7f2d29007700 1 -- 172.21.15.46:0/1051029975 learned_addr learned my addr 172.21.15.46:0/1051029975 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:28.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.996+0000 7f2d29808700 1 --2- 172.21.15.46:0/1051029975 >> v2:172.21.15.46:3300/0 conn(0x7f2d240c9120 0x7f2d240cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:28.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.996+0000 7f2d28806700 1 --2- 172.21.15.46:0/1051029975 >> v2:172.21.15.106:3300/0 conn(0x7f2d241598e0 0x7f2d240c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:28.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.996+0000 7f2d1bfff700 1 -- 172.21.15.46:0/1051029975 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834668337 0 0) 0x7f2d24069260 con 0x7f2d24159510 2024-01-26T23:30:28.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.996+0000 7f2d1bfff700 1 -- 172.21.15.46:0/1051029975 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d10003570 con 0x7f2d24159510 2024-01-26T23:30:28.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.996+0000 7f2d29808700 1 -- 172.21.15.46:0/1051029975 >> v2:172.21.15.106:3300/0 conn(0x7f2d241598e0 msgr2=0x7f2d240c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:28.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.996+0000 7f2d29808700 1 --2- 172.21.15.46:0/1051029975 >> v2:172.21.15.106:3300/0 conn(0x7f2d241598e0 0x7f2d240c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:28.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.996+0000 7f2d29808700 1 -- 172.21.15.46:0/1051029975 >> v1:172.21.15.46:6789/0 conn(0x7f2d24159510 legacy=0x7f2d240ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:28.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.996+0000 7f2d29808700 1 -- 172.21.15.46:0/1051029975 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2415d670 con 0x7f2d240c9120 2024-01-26T23:30:28.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.996+0000 7f2d29808700 1 --2- 172.21.15.46:0/1051029975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d240c9120 0x7f2d240cc380 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2d2406f730 tx=0x7f2d1c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0bd2daba5115cad server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:28.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.997+0000 7f2d1bfff700 1 -- 172.21.15.46:0/1051029975 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c00e070 con 0x7f2d240c9120 2024-01-26T23:30:28.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.997+0000 7f2d1bfff700 1 -- 172.21.15.46:0/1051029975 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c00bab0 con 0x7f2d240c9120 2024-01-26T23:30:28.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.997+0000 7f2d1bfff700 1 -- 172.21.15.46:0/1051029975 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c00be30 con 0x7f2d240c9120 2024-01-26T23:30:29.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.997+0000 7f2d2b26b700 1 -- 172.21.15.46:0/1051029975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d240c9120 msgr2=0x7f2d240cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.997+0000 7f2d2b26b700 1 --2- 172.21.15.46:0/1051029975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d240c9120 0x7f2d240cc380 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2d2406f730 tx=0x7f2d1c009780 comp rx=0 tx=0).stop 2024-01-26T23:30:29.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.997+0000 7f2d2b26b700 1 -- 172.21.15.46:0/1051029975 shutdown_connections 2024-01-26T23:30:29.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.997+0000 7f2d2b26b700 1 --2- 172.21.15.46:0/1051029975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d240c9120 0x7f2d240cc380 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.997+0000 7f2d2b26b700 1 --2- 172.21.15.46:0/1051029975 >> v2:172.21.15.106:3300/0 conn(0x7f2d241598e0 0x7f2d240c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.997+0000 7f2d2b26b700 1 -- 172.21.15.46:0/1051029975 >> 172.21.15.46:0/1051029975 conn(0x7f2d2406bd90 msgr2=0x7f2d2406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:29.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.997+0000 7f2d2b26b700 1 -- 172.21.15.46:0/1051029975 shutdown_connections 2024-01-26T23:30:29.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.997+0000 7f2d2b26b700 1 -- 172.21.15.46:0/1051029975 wait complete. 2024-01-26T23:30:29.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.998+0000 7f2d2b26b700 1 Processor -- start 2024-01-26T23:30:29.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.998+0000 7f2d2b26b700 1 -- start start 2024-01-26T23:30:29.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.999+0000 7f2d2b26b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2d240c9120 0x7f2d2406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:29.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.999+0000 7f2d2b26b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2d24159510 0x7f2d2406d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:29.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.999+0000 7f2d2b26b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d241598e0 0x7f2d240b1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:29.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.999+0000 7f2d2b26b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d241696d0 con 0x7f2d241598e0 2024-01-26T23:30:29.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.999+0000 7f2d2b26b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d240b3b00 con 0x7f2d240c9120 2024-01-26T23:30:29.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:28.999+0000 7f2d2b26b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d240c5ca0 con 0x7f2d24159510 2024-01-26T23:30:29.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.000+0000 7f2d29808700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d241598e0 0x7f2d240b1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:29.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.000+0000 7f2d29808700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d241598e0 0x7f2d240b1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54430/0 (socket says 172.21.15.46:54430) 2024-01-26T23:30:29.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.000+0000 7f2d29808700 1 -- 172.21.15.46:0/2193714378 learned_addr learned my addr 172.21.15.46:0/2193714378 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:29.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.000+0000 7f2d29007700 1 --2- 172.21.15.46:0/2193714378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2d240c9120 0x7f2d2406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:29.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.000+0000 7f2d28806700 1 --2- 172.21.15.46:0/2193714378 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2d24159510 0x7f2d2406d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:29.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.000+0000 7f2d29808700 1 -- 172.21.15.46:0/2193714378 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2d24159510 msgr2=0x7f2d2406d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.000+0000 7f2d29808700 1 --2- 172.21.15.46:0/2193714378 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2d24159510 0x7f2d2406d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.000+0000 7f2d29808700 1 -- 172.21.15.46:0/2193714378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2d240c9120 msgr2=0x7f2d2406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.000+0000 7f2d29808700 1 --2- 172.21.15.46:0/2193714378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2d240c9120 0x7f2d2406d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.000+0000 7f2d29808700 1 -- 172.21.15.46:0/2193714378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1c009a20 con 0x7f2d241598e0 2024-01-26T23:30:29.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.000+0000 7f2d28806700 1 --2- 172.21.15.46:0/2193714378 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2d24159510 0x7f2d2406d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:29.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.000+0000 7f2d29808700 1 --2- 172.21.15.46:0/2193714378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d241598e0 0x7f2d240b1f50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0097c0 tx=0x7f2d1c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:29.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.000+0000 7f2d197fa700 1 -- 172.21.15.46:0/2193714378 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c012c20 con 0x7f2d241598e0 2024-01-26T23:30:29.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.000+0000 7f2d197fa700 1 -- 172.21.15.46:0/2193714378 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c021070 con 0x7f2d241598e0 2024-01-26T23:30:29.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.000+0000 7f2d2b26b700 1 -- 172.21.15.46:0/2193714378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d240b3d50 con 0x7f2d241598e0 2024-01-26T23:30:29.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.000+0000 7f2d197fa700 1 -- 172.21.15.46:0/2193714378 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c026430 con 0x7f2d241598e0 2024-01-26T23:30:29.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.000+0000 7f2d2b26b700 1 -- 172.21.15.46:0/2193714378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d240c5650 con 0x7f2d241598e0 2024-01-26T23:30:29.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.001+0000 7f2d2b26b700 1 -- 172.21.15.46:0/2193714378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d240c5e10 con 0x7f2d241598e0 2024-01-26T23:30:29.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.002+0000 7f2d197fa700 1 -- 172.21.15.46:0/2193714378 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2d1c023030 con 0x7f2d241598e0 2024-01-26T23:30:29.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.002+0000 7f2d197fa700 1 --2- 172.21.15.46:0/2193714378 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d08066f30 0x7f2d080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:29.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.002+0000 7f2d197fa700 1 -- 172.21.15.46:0/2193714378 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f2d1c086800 con 0x7f2d241598e0 2024-01-26T23:30:29.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.003+0000 7f2d29007700 1 --2- 172.21.15.46:0/2193714378 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d08066f30 0x7f2d080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:29.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.003+0000 7f2d29007700 1 --2- 172.21.15.46:0/2193714378 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d08066f30 0x7f2d080693e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2d20007fd0 tx=0x7f2d20009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:29.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.007+0000 7f2d197fa700 1 -- 172.21.15.46:0/2193714378 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2d1c053780 con 0x7f2d241598e0 2024-01-26T23:30:29.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.209+0000 7f2d2b26b700 1 -- 172.21.15.46:0/2193714378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2d240b1d50 con 0x7f2d241598e0 2024-01-26T23:30:29.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.209+0000 7f2d197fa700 1 -- 172.21.15.46:0/2193714378 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2d1c056da0 con 0x7f2d241598e0 2024-01-26T23:30:29.210 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:30:29.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.211+0000 7f2d2b26b700 1 -- 172.21.15.46:0/2193714378 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d08066f30 msgr2=0x7f2d080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.211+0000 7f2d2b26b700 1 --2- 172.21.15.46:0/2193714378 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d08066f30 0x7f2d080693e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2d20007fd0 tx=0x7f2d20009040 comp rx=0 tx=0).stop 2024-01-26T23:30:29.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.211+0000 7f2d2b26b700 1 -- 172.21.15.46:0/2193714378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d241598e0 msgr2=0x7f2d240b1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.211+0000 7f2d2b26b700 1 --2- 172.21.15.46:0/2193714378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d241598e0 0x7f2d240b1f50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0097c0 tx=0x7f2d1c019000 comp rx=0 tx=0).stop 2024-01-26T23:30:29.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.212+0000 7f2d2b26b700 1 -- 172.21.15.46:0/2193714378 shutdown_connections 2024-01-26T23:30:29.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.212+0000 7f2d2b26b700 1 --2- 172.21.15.46:0/2193714378 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d08066f30 0x7f2d080693e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.212+0000 7f2d2b26b700 1 --2- 172.21.15.46:0/2193714378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d241598e0 0x7f2d240b1f50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.212+0000 7f2d2b26b700 1 --2- 172.21.15.46:0/2193714378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2d240c9120 0x7f2d2406d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.212+0000 7f2d2b26b700 1 --2- 172.21.15.46:0/2193714378 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2d24159510 0x7f2d2406d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.212+0000 7f2d2b26b700 1 -- 172.21.15.46:0/2193714378 >> 172.21.15.46:0/2193714378 conn(0x7f2d2406bd90 msgr2=0x7f2d240ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:29.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.212+0000 7f2d2b26b700 1 -- 172.21.15.46:0/2193714378 shutdown_connections 2024-01-26T23:30:29.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.212+0000 7f2d2b26b700 1 -- 172.21.15.46:0/2193714378 wait complete. 2024-01-26T23:30:29.226 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-26T23:30:29.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.359+0000 7f568bf59700 1 Processor -- start 2024-01-26T23:30:29.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.359+0000 7f568bf59700 1 -- start start 2024-01-26T23:30:29.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.360+0000 7f568bf59700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f568406f360 0x7f56840cb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:29.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.360+0000 7f568bf59700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56840cbfb0 con 0x7f568415ad40 2024-01-26T23:30:29.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.360+0000 7f568bf59700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f568415a8b0 con 0x7f56841597d0 2024-01-26T23:30:29.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.360+0000 7f568bf59700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56840cbcd0 con 0x7f568406f360 2024-01-26T23:30:29.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.360+0000 7f568a4f6700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f568415ad40 0x7f56840c8aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42014/0 (socket says 172.21.15.46:42014) 2024-01-26T23:30:29.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.360+0000 7f568a4f6700 1 -- 172.21.15.46:0/2619171286 learned_addr learned my addr 172.21.15.46:0/2619171286 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:29.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.360+0000 7f56894f4700 1 --2- 172.21.15.46:0/2619171286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f568406f360 0x7f56840cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:29.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.361+0000 7f5688cf3700 1 -- 172.21.15.46:0/2619171286 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312318968 0 0) 0x7f56840cbfb0 con 0x7f568415ad40 2024-01-26T23:30:29.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.361+0000 7f5688cf3700 1 -- 172.21.15.46:0/2619171286 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5670003570 con 0x7f568415ad40 2024-01-26T23:30:29.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.361+0000 7f5688cf3700 1 -- 172.21.15.46:0/2619171286 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453562782 0 0) 0x7f568415a8b0 con 0x7f56841597d0 2024-01-26T23:30:29.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.361+0000 7f5688cf3700 1 -- 172.21.15.46:0/2619171286 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56840cbfb0 con 0x7f56841597d0 2024-01-26T23:30:29.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.361+0000 7f56894f4700 1 -- 172.21.15.46:0/2619171286 >> v1:172.21.15.106:6789/0 conn(0x7f56841597d0 legacy=0x7f568406ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.361+0000 7f5688cf3700 1 -- 172.21.15.46:0/2619171286 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1188752310 0 0) 0x7f5670003570 con 0x7f568415ad40 2024-01-26T23:30:29.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.361+0000 7f56894f4700 1 -- 172.21.15.46:0/2619171286 >> v1:172.21.15.46:6789/0 conn(0x7f568415ad40 legacy=0x7f56840c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.361+0000 7f56894f4700 1 -- 172.21.15.46:0/2619171286 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568415a260 con 0x7f568406f360 2024-01-26T23:30:29.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.361+0000 7f56894f4700 1 --2- 172.21.15.46:0/2619171286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f568406f360 0x7f56840cb030 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5678009910 tx=0x7f56780091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c03b71eb7c7d8499 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:29.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.361+0000 7f5688cf3700 1 -- 172.21.15.46:0/2619171286 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567c0017f0 con 0x7f568406f360 2024-01-26T23:30:29.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.361+0000 7f5688cf3700 1 -- 172.21.15.46:0/2619171286 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5678011420 con 0x7f568406f360 2024-01-26T23:30:29.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.361+0000 7f5688cf3700 1 -- 172.21.15.46:0/2619171286 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56780117a0 con 0x7f568406f360 2024-01-26T23:30:29.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.362+0000 7f568bf59700 1 -- 172.21.15.46:0/2619171286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f568406f360 msgr2=0x7f56840cb030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.362+0000 7f568bf59700 1 --2- 172.21.15.46:0/2619171286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f568406f360 0x7f56840cb030 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5678009910 tx=0x7f56780091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.362+0000 7f568bf59700 1 -- 172.21.15.46:0/2619171286 shutdown_connections 2024-01-26T23:30:29.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.362+0000 7f568bf59700 1 --2- 172.21.15.46:0/2619171286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f568406f360 0x7f56840cb030 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.362+0000 7f568bf59700 1 -- 172.21.15.46:0/2619171286 >> 172.21.15.46:0/2619171286 conn(0x7f56840ca9a0 msgr2=0x7f5684068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:29.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.362+0000 7f568bf59700 1 -- 172.21.15.46:0/2619171286 shutdown_connections 2024-01-26T23:30:29.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.362+0000 7f568bf59700 1 -- 172.21.15.46:0/2619171286 wait complete. 2024-01-26T23:30:29.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.363+0000 7f568bf59700 1 Processor -- start 2024-01-26T23:30:29.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.363+0000 7f568bf59700 1 -- start start 2024-01-26T23:30:29.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.364+0000 7f568bf59700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f568406f360 0x7f568414b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:29.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.364+0000 7f568bf59700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56841597d0 0x7f56840b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:29.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.364+0000 7f568bf59700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f568415ad40 0x7f568415c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:29.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.364+0000 7f568bf59700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56840ab330 con 0x7f56841597d0 2024-01-26T23:30:29.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.364+0000 7f568bf59700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56840adbb0 con 0x7f568406f360 2024-01-26T23:30:29.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.364+0000 7f568bf59700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56840b2ff0 con 0x7f568415ad40 2024-01-26T23:30:29.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.364+0000 7f56894f4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56841597d0 0x7f56840b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:29.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.364+0000 7f56894f4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56841597d0 0x7f56840b04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54432/0 (socket says 172.21.15.46:54432) 2024-01-26T23:30:29.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.364+0000 7f56894f4700 1 -- 172.21.15.46:0/231777241 learned_addr learned my addr 172.21.15.46:0/231777241 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:29.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.364+0000 7f568a4f6700 1 --2- 172.21.15.46:0/231777241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f568415ad40 0x7f568415c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:29.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.364+0000 7f5689cf5700 1 --2- 172.21.15.46:0/231777241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f568406f360 0x7f568414b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:29.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.364+0000 7f56894f4700 1 -- 172.21.15.46:0/231777241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f568415ad40 msgr2=0x7f568415c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.364+0000 7f56894f4700 1 --2- 172.21.15.46:0/231777241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f568415ad40 0x7f568415c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.364+0000 7f56894f4700 1 -- 172.21.15.46:0/231777241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f568406f360 msgr2=0x7f568414b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.364+0000 7f56894f4700 1 --2- 172.21.15.46:0/231777241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f568406f360 0x7f568414b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.364+0000 7f56894f4700 1 -- 172.21.15.46:0/231777241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5678009420 con 0x7f56841597d0 2024-01-26T23:30:29.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.365+0000 7f5689cf5700 1 --2- 172.21.15.46:0/231777241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f568406f360 0x7f568414b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:29.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.365+0000 7f56894f4700 1 --2- 172.21.15.46:0/231777241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56841597d0 0x7f56840b04e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f568406e820 tx=0x7f5678016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:29.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.365+0000 7f5675ffb700 1 -- 172.21.15.46:0/231777241 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5678006af0 con 0x7f56841597d0 2024-01-26T23:30:29.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.365+0000 7f5675ffb700 1 -- 172.21.15.46:0/231777241 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5678000c40 con 0x7f56841597d0 2024-01-26T23:30:29.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.365+0000 7f568bf59700 1 -- 172.21.15.46:0/231777241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56840ab050 con 0x7f56841597d0 2024-01-26T23:30:29.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.365+0000 7f5675ffb700 1 -- 172.21.15.46:0/231777241 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567801cea0 con 0x7f56841597d0 2024-01-26T23:30:29.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.365+0000 7f568bf59700 1 -- 172.21.15.46:0/231777241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56840ad840 con 0x7f56841597d0 2024-01-26T23:30:29.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.366+0000 7f568bf59700 1 -- 172.21.15.46:0/231777241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56840add20 con 0x7f56841597d0 2024-01-26T23:30:29.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.367+0000 7f5675ffb700 1 -- 172.21.15.46:0/231777241 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f567801a660 con 0x7f56841597d0 2024-01-26T23:30:29.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.367+0000 7f5675ffb700 1 --2- 172.21.15.46:0/231777241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5668066f30 0x7f56680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:29.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.367+0000 7f5675ffb700 1 -- 172.21.15.46:0/231777241 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f5678086120 con 0x7f56841597d0 2024-01-26T23:30:29.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.367+0000 7f5689cf5700 1 --2- 172.21.15.46:0/231777241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5668066f30 0x7f56680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:29.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.368+0000 7f5689cf5700 1 --2- 172.21.15.46:0/231777241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5668066f30 0x7f56680693e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5680000c00 tx=0x7f5680009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:29.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.372+0000 7f5675ffb700 1 -- 172.21.15.46:0/231777241 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f56780530a0 con 0x7f56841597d0 2024-01-26T23:30:29.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.579+0000 7f568bf59700 1 -- 172.21.15.46:0/231777241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f568404c710 con 0x7f56841597d0 2024-01-26T23:30:29.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.580+0000 7f5675ffb700 1 -- 172.21.15.46:0/231777241 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f56780566c0 con 0x7f56841597d0 2024-01-26T23:30:29.581 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:30:29.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.582+0000 7f568bf59700 1 -- 172.21.15.46:0/231777241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5668066f30 msgr2=0x7f56680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.582+0000 7f568bf59700 1 --2- 172.21.15.46:0/231777241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5668066f30 0x7f56680693e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5680000c00 tx=0x7f5680009040 comp rx=0 tx=0).stop 2024-01-26T23:30:29.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.582+0000 7f568bf59700 1 -- 172.21.15.46:0/231777241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56841597d0 msgr2=0x7f56840b04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.582+0000 7f568bf59700 1 --2- 172.21.15.46:0/231777241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56841597d0 0x7f56840b04e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f568406e820 tx=0x7f5678016000 comp rx=0 tx=0).stop 2024-01-26T23:30:29.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.582+0000 7f568bf59700 1 -- 172.21.15.46:0/231777241 shutdown_connections 2024-01-26T23:30:29.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.582+0000 7f568bf59700 1 --2- 172.21.15.46:0/231777241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5668066f30 0x7f56680693e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.582+0000 7f568bf59700 1 --2- 172.21.15.46:0/231777241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f568415ad40 0x7f568415c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.582+0000 7f568bf59700 1 --2- 172.21.15.46:0/231777241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f568406f360 0x7f568414b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.582+0000 7f568bf59700 1 --2- 172.21.15.46:0/231777241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56841597d0 0x7f56840b04e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.582+0000 7f568bf59700 1 -- 172.21.15.46:0/231777241 >> 172.21.15.46:0/231777241 conn(0x7f56840ca9a0 msgr2=0x7f56840c39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:29.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.582+0000 7f568bf59700 1 -- 172.21.15.46:0/231777241 shutdown_connections 2024-01-26T23:30:29.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.582+0000 7f568bf59700 1 -- 172.21.15.46:0/231777241 wait complete. 2024-01-26T23:30:29.597 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-26T23:30:29.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.731+0000 7f1932628700 1 Processor -- start 2024-01-26T23:30:29.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.731+0000 7f1932628700 1 -- start start 2024-01-26T23:30:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.732+0000 7f1932628700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192c157750 0x7f192c157b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.732+0000 7f1932628700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f192c0af790 0x7f192c0b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.732+0000 7f1932628700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f192c158f00 con 0x7f192c06b3c0 2024-01-26T23:30:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.732+0000 7f1932628700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f192c158060 con 0x7f192c0af790 2024-01-26T23:30:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.732+0000 7f1932628700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f192c06fe90 con 0x7f192c157750 2024-01-26T23:30:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.732+0000 7f192b7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f192c06b3c0 0x7f192c0af080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42016/0 (socket says 172.21.15.46:42016) 2024-01-26T23:30:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.732+0000 7f192b7fe700 1 -- 172.21.15.46:0/1339650183 learned_addr learned my addr 172.21.15.46:0/1339650183 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.732+0000 7f1930bc5700 1 --2- 172.21.15.46:0/1339650183 >> v2:172.21.15.106:3300/0 conn(0x7f192c0af790 0x7f192c0b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.732+0000 7f192bfff700 1 --2- 172.21.15.46:0/1339650183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192c157750 0x7f192c157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.732+0000 7f192affd700 1 -- 172.21.15.46:0/1339650183 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869887875 0 0) 0x7f192c158f00 con 0x7f192c06b3c0 2024-01-26T23:30:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.733+0000 7f192affd700 1 -- 172.21.15.46:0/1339650183 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1914003570 con 0x7f192c06b3c0 2024-01-26T23:30:29.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.733+0000 7f192bfff700 1 -- 172.21.15.46:0/1339650183 >> v2:172.21.15.106:3300/0 conn(0x7f192c0af790 msgr2=0x7f192c0b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:29.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.733+0000 7f192bfff700 1 --2- 172.21.15.46:0/1339650183 >> v2:172.21.15.106:3300/0 conn(0x7f192c0af790 0x7f192c0b1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.733+0000 7f192bfff700 1 -- 172.21.15.46:0/1339650183 >> v1:172.21.15.46:6789/0 conn(0x7f192c06b3c0 legacy=0x7f192c0af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.733+0000 7f192affd700 1 -- 172.21.15.46:0/1339650183 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 208814675 0 0) 0x7f1914003570 con 0x7f192c06b3c0 2024-01-26T23:30:29.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.733+0000 7f192bfff700 1 -- 172.21.15.46:0/1339650183 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f192c1582f0 con 0x7f192c157750 2024-01-26T23:30:29.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.733+0000 7f1930bc5700 1 --2- 172.21.15.46:0/1339650183 >> v2:172.21.15.106:3300/0 conn(0x7f192c0af790 0x7f192c0b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:29.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.733+0000 7f192bfff700 1 --2- 172.21.15.46:0/1339650183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192c157750 0x7f192c157b20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f191c0098e0 tx=0x7f191c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=534b72c52f8b57fb server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:29.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.733+0000 7f192affd700 1 -- 172.21.15.46:0/1339650183 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191c011630 con 0x7f192c157750 2024-01-26T23:30:29.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.733+0000 7f192affd700 1 -- 172.21.15.46:0/1339650183 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f191c011790 con 0x7f192c157750 2024-01-26T23:30:29.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.733+0000 7f192affd700 1 -- 172.21.15.46:0/1339650183 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191c011b30 con 0x7f192c157750 2024-01-26T23:30:29.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.734+0000 7f1932628700 1 -- 172.21.15.46:0/1339650183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192c157750 msgr2=0x7f192c157b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.734+0000 7f1932628700 1 --2- 172.21.15.46:0/1339650183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192c157750 0x7f192c157b20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f191c0098e0 tx=0x7f191c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.734+0000 7f1932628700 1 -- 172.21.15.46:0/1339650183 shutdown_connections 2024-01-26T23:30:29.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.734+0000 7f1932628700 1 --2- 172.21.15.46:0/1339650183 >> v2:172.21.15.106:3300/0 conn(0x7f192c0af790 0x7f192c0b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.734+0000 7f1932628700 1 --2- 172.21.15.46:0/1339650183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192c157750 0x7f192c157b20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.734+0000 7f1932628700 1 -- 172.21.15.46:0/1339650183 >> 172.21.15.46:0/1339650183 conn(0x7f192c0cdf80 msgr2=0x7f192c06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:29.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.734+0000 7f1932628700 1 -- 172.21.15.46:0/1339650183 shutdown_connections 2024-01-26T23:30:29.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.734+0000 7f1932628700 1 -- 172.21.15.46:0/1339650183 wait complete. 2024-01-26T23:30:29.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.735+0000 7f1932628700 1 Processor -- start 2024-01-26T23:30:29.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.735+0000 7f1932628700 1 -- start start 2024-01-26T23:30:29.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.736+0000 7f1932628700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192c06b3c0 0x7f192c0ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:29.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.736+0000 7f1932628700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f192c0af790 0x7f192c169a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:29.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.736+0000 7f1932628700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f192c157750 0x7f192c169fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.736+0000 7f1932628700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192c0ae430 con 0x7f192c157750 2024-01-26T23:30:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.736+0000 7f1932628700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192c0aebd0 con 0x7f192c0af790 2024-01-26T23:30:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.736+0000 7f1932628700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f192c15b500 con 0x7f192c06b3c0 2024-01-26T23:30:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.736+0000 7f1930bc5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f192c157750 0x7f192c169fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.736+0000 7f1930bc5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f192c157750 0x7f192c169fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54446/0 (socket says 172.21.15.46:54446) 2024-01-26T23:30:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.736+0000 7f1930bc5700 1 -- 172.21.15.46:0/4053448967 learned_addr learned my addr 172.21.15.46:0/4053448967 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.736+0000 7f192b7fe700 1 --2- 172.21.15.46:0/4053448967 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f192c0af790 0x7f192c169a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.736+0000 7f192bfff700 1 --2- 172.21.15.46:0/4053448967 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192c06b3c0 0x7f192c0ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.737+0000 7f1930bc5700 1 -- 172.21.15.46:0/4053448967 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192c06b3c0 msgr2=0x7f192c0ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.737+0000 7f1930bc5700 1 --2- 172.21.15.46:0/4053448967 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192c06b3c0 0x7f192c0ca660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.737+0000 7f1930bc5700 1 -- 172.21.15.46:0/4053448967 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f192c0af790 msgr2=0x7f192c169a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.737+0000 7f1930bc5700 1 --2- 172.21.15.46:0/4053448967 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f192c0af790 0x7f192c169a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.737+0000 7f1930bc5700 1 -- 172.21.15.46:0/4053448967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1924009a20 con 0x7f192c157750 2024-01-26T23:30:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.737+0000 7f1930bc5700 1 --2- 172.21.15.46:0/4053448967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f192c157750 0x7f192c169fc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1924009f60 tx=0x7f192400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.737+0000 7f1913fff700 1 -- 172.21.15.46:0/4053448967 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192400cb10 con 0x7f192c157750 2024-01-26T23:30:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.737+0000 7f1913fff700 1 -- 172.21.15.46:0/4053448967 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192400f040 con 0x7f192c157750 2024-01-26T23:30:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.737+0000 7f1932628700 1 -- 172.21.15.46:0/4053448967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191c009420 con 0x7f192c157750 2024-01-26T23:30:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.737+0000 7f1913fff700 1 -- 172.21.15.46:0/4053448967 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1924014480 con 0x7f192c157750 2024-01-26T23:30:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.737+0000 7f1932628700 1 -- 172.21.15.46:0/4053448967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f192c0ae8f0 con 0x7f192c157750 2024-01-26T23:30:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.738+0000 7f1932628700 1 -- 172.21.15.46:0/4053448967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f192c15aff0 con 0x7f192c157750 2024-01-26T23:30:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.739+0000 7f1913fff700 1 -- 172.21.15.46:0/4053448967 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1924003020 con 0x7f192c157750 2024-01-26T23:30:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.739+0000 7f1913fff700 1 --2- 172.21.15.46:0/4053448967 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1918066f30 0x7f19180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.739+0000 7f1913fff700 1 -- 172.21.15.46:0/4053448967 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f1924085010 con 0x7f192c157750 2024-01-26T23:30:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.739+0000 7f192bfff700 1 --2- 172.21.15.46:0/4053448967 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1918066f30 0x7f19180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.740+0000 7f192bfff700 1 --2- 172.21.15.46:0/4053448967 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1918066f30 0x7f19180693e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f191c0091e0 tx=0x7f191c017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:29.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.744+0000 7f1913fff700 1 -- 172.21.15.46:0/4053448967 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f192c15aff0 con 0x7f192c157750 2024-01-26T23:30:29.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.950+0000 7f1932628700 1 -- 172.21.15.46:0/4053448967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f192c15aff0 con 0x7f192c157750 2024-01-26T23:30:29.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.950+0000 7f1913fff700 1 -- 172.21.15.46:0/4053448967 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f192c15aff0 con 0x7f192c157750 2024-01-26T23:30:29.951 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-26T23:30:29.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.953+0000 7f1932628700 1 -- 172.21.15.46:0/4053448967 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1918066f30 msgr2=0x7f19180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.953+0000 7f1932628700 1 --2- 172.21.15.46:0/4053448967 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1918066f30 0x7f19180693e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f191c0091e0 tx=0x7f191c017000 comp rx=0 tx=0).stop 2024-01-26T23:30:29.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.953+0000 7f1932628700 1 -- 172.21.15.46:0/4053448967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f192c157750 msgr2=0x7f192c169fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:29.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.953+0000 7f1932628700 1 --2- 172.21.15.46:0/4053448967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f192c157750 0x7f192c169fc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1924009f60 tx=0x7f192400b040 comp rx=0 tx=0).stop 2024-01-26T23:30:29.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.953+0000 7f1932628700 1 -- 172.21.15.46:0/4053448967 shutdown_connections 2024-01-26T23:30:29.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.953+0000 7f1932628700 1 --2- 172.21.15.46:0/4053448967 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1918066f30 0x7f19180693e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.953+0000 7f1932628700 1 --2- 172.21.15.46:0/4053448967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f192c157750 0x7f192c169fc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.953+0000 7f1932628700 1 --2- 172.21.15.46:0/4053448967 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192c06b3c0 0x7f192c0ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.953+0000 7f1932628700 1 --2- 172.21.15.46:0/4053448967 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f192c0af790 0x7f192c169a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:29.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.953+0000 7f1932628700 1 -- 172.21.15.46:0/4053448967 >> 172.21.15.46:0/4053448967 conn(0x7f192c0cdf80 msgr2=0x7f192c0cb5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:29.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.953+0000 7f1932628700 1 -- 172.21.15.46:0/4053448967 shutdown_connections 2024-01-26T23:30:29.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:29.953+0000 7f1932628700 1 -- 172.21.15.46:0/4053448967 wait complete. 2024-01-26T23:30:29.967 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-01-26T23:30:30.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.100+0000 7fed4c8f2700 1 Processor -- start 2024-01-26T23:30:30.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.101+0000 7fed4c8f2700 1 -- start start 2024-01-26T23:30:30.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.102+0000 7fed4c8f2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fed48068b00 0x7fed48068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:30.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.102+0000 7fed4c8f2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed4806add0 0x7fed4806b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:30.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.102+0000 7fed4c8f2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed480a9a80 con 0x7fed4806a660 2024-01-26T23:30:30.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.102+0000 7fed4c8f2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed4806b7b0 con 0x7fed48068b00 2024-01-26T23:30:30.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.102+0000 7fed4c8f2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4806aab0 con 0x7fed4806add0 2024-01-26T23:30:30.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.102+0000 7fed4659c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fed48068b00 0x7fed48068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:30.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.102+0000 7fed4659c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fed48068b00 0x7fed48068ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54448/0 (socket says 172.21.15.46:54448) 2024-01-26T23:30:30.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.102+0000 7fed4659c700 1 -- 172.21.15.46:0/2848065942 learned_addr learned my addr 172.21.15.46:0/2848065942 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:30.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.102+0000 7fed46d9d700 1 --2- 172.21.15.46:0/2848065942 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed4806add0 0x7fed4806b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:30.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.102+0000 7fed4659c700 1 -- 172.21.15.46:0/2848065942 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed4806add0 msgr2=0x7fed4806b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:30.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.102+0000 7fed4659c700 1 --2- 172.21.15.46:0/2848065942 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed4806add0 0x7fed4806b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.102+0000 7fed4659c700 1 -- 172.21.15.46:0/2848065942 >> v1:172.21.15.106:6789/0 conn(0x7fed4806a660 legacy=0x7fed48159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:30.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.102+0000 7fed4659c700 1 -- 172.21.15.46:0/2848065942 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4815a680 con 0x7fed48068b00 2024-01-26T23:30:30.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.102+0000 7fed4659c700 1 --2- 172.21.15.46:0/2848065942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed48068b00 0x7fed48068ed0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fed38009990 tx=0x7fed38009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50db6ef9a9391a28 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:30.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.102+0000 7fed4559a700 1 -- 172.21.15.46:0/2848065942 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed38009e80 con 0x7fed48068b00 2024-01-26T23:30:30.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.103+0000 7fed4559a700 1 -- 172.21.15.46:0/2848065942 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed38005c10 con 0x7fed48068b00 2024-01-26T23:30:30.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.103+0000 7fed4559a700 1 -- 172.21.15.46:0/2848065942 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3800b650 con 0x7fed48068b00 2024-01-26T23:30:30.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.103+0000 7fed4c8f2700 1 -- 172.21.15.46:0/2848065942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed48068b00 msgr2=0x7fed48068ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:30.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.103+0000 7fed4c8f2700 1 --2- 172.21.15.46:0/2848065942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed48068b00 0x7fed48068ed0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fed38009990 tx=0x7fed38009220 comp rx=0 tx=0).stop 2024-01-26T23:30:30.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.103+0000 7fed4c8f2700 1 -- 172.21.15.46:0/2848065942 shutdown_connections 2024-01-26T23:30:30.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.103+0000 7fed4c8f2700 1 --2- 172.21.15.46:0/2848065942 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed4806add0 0x7fed4806b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.103+0000 7fed4c8f2700 1 --2- 172.21.15.46:0/2848065942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed48068b00 0x7fed48068ed0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.103+0000 7fed4c8f2700 1 -- 172.21.15.46:0/2848065942 >> 172.21.15.46:0/2848065942 conn(0x7fed480ca100 msgr2=0x7fed4806e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:30.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.103+0000 7fed4c8f2700 1 -- 172.21.15.46:0/2848065942 shutdown_connections 2024-01-26T23:30:30.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.103+0000 7fed4c8f2700 1 -- 172.21.15.46:0/2848065942 wait complete. 2024-01-26T23:30:30.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.104+0000 7fed4c8f2700 1 Processor -- start 2024-01-26T23:30:30.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.104+0000 7fed4c8f2700 1 -- start start 2024-01-26T23:30:30.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.105+0000 7fed4c8f2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed48068b00 0x7fed480af100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:30.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.105+0000 7fed4c8f2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed4806a660 0x7fed480ac420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:30.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.105+0000 7fed4c8f2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4806add0 0x7fed480b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:30.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.105+0000 7fed4c8f2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed480abf90 con 0x7fed4806add0 2024-01-26T23:30:30.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.105+0000 7fed4c8f2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed480c3930 con 0x7fed4806a660 2024-01-26T23:30:30.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.105+0000 7fed4c8f2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4815dd50 con 0x7fed48068b00 2024-01-26T23:30:30.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.105+0000 7fed46d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4806add0 0x7fed480b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:30.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.105+0000 7fed46d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4806add0 0x7fed480b1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54458/0 (socket says 172.21.15.46:54458) 2024-01-26T23:30:30.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.105+0000 7fed46d9d700 1 -- 172.21.15.46:0/3062209200 learned_addr learned my addr 172.21.15.46:0/3062209200 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:30.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.105+0000 7fed45d9b700 1 --2- 172.21.15.46:0/3062209200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed4806a660 0x7fed480ac420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:30.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.105+0000 7fed4659c700 1 --2- 172.21.15.46:0/3062209200 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed48068b00 0x7fed480af100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:30.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.106+0000 7fed46d9d700 1 -- 172.21.15.46:0/3062209200 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed48068b00 msgr2=0x7fed480af100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:30.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.106+0000 7fed46d9d700 1 --2- 172.21.15.46:0/3062209200 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed48068b00 0x7fed480af100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.106+0000 7fed46d9d700 1 -- 172.21.15.46:0/3062209200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed4806a660 msgr2=0x7fed480ac420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:30.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.106+0000 7fed46d9d700 1 --2- 172.21.15.46:0/3062209200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed4806a660 0x7fed480ac420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.106+0000 7fed46d9d700 1 -- 172.21.15.46:0/3062209200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed38009480 con 0x7fed4806add0 2024-01-26T23:30:30.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.106+0000 7fed46d9d700 1 --2- 172.21.15.46:0/3062209200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4806add0 0x7fed480b1f30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fed40009e90 tx=0x7fed4000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:30.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.106+0000 7fed367fc700 1 -- 172.21.15.46:0/3062209200 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4000cb70 con 0x7fed4806add0 2024-01-26T23:30:30.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.106+0000 7fed367fc700 1 -- 172.21.15.46:0/3062209200 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4000f040 con 0x7fed4806add0 2024-01-26T23:30:30.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.106+0000 7fed4c8f2700 1 -- 172.21.15.46:0/3062209200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed4815e030 con 0x7fed4806add0 2024-01-26T23:30:30.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.106+0000 7fed367fc700 1 -- 172.21.15.46:0/3062209200 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40014480 con 0x7fed4806add0 2024-01-26T23:30:30.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.106+0000 7fed4c8f2700 1 -- 172.21.15.46:0/3062209200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed480abc20 con 0x7fed4806add0 2024-01-26T23:30:30.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.107+0000 7fed4c8f2700 1 -- 172.21.15.46:0/3062209200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed4804c710 con 0x7fed4806add0 2024-01-26T23:30:30.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.108+0000 7fed367fc700 1 -- 172.21.15.46:0/3062209200 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fed40014830 con 0x7fed4806add0 2024-01-26T23:30:30.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.108+0000 7fed367fc700 1 --2- 172.21.15.46:0/3062209200 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fed28066f30 0x7fed280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:30.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.108+0000 7fed367fc700 1 -- 172.21.15.46:0/3062209200 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fed40085520 con 0x7fed4806add0 2024-01-26T23:30:30.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.108+0000 7fed4659c700 1 --2- 172.21.15.46:0/3062209200 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fed28066f30 0x7fed280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:30.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.109+0000 7fed4659c700 1 --2- 172.21.15.46:0/3062209200 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fed28066f30 0x7fed280693e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fed38009260 tx=0x7fed38016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:30.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.113+0000 7fed367fc700 1 -- 172.21.15.46:0/3062209200 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fed400524a0 con 0x7fed4806add0 2024-01-26T23:30:30.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.317+0000 7fed4c8f2700 1 -- 172.21.15.46:0/3062209200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fed4815ac30 con 0x7fed4806add0 2024-01-26T23:30:30.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.318+0000 7fed367fc700 1 -- 172.21.15.46:0/3062209200 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7fed40055ac0 con 0x7fed4806add0 2024-01-26T23:30:30.319 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:30:30.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.320+0000 7fed4c8f2700 1 -- 172.21.15.46:0/3062209200 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fed28066f30 msgr2=0x7fed280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:30.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.320+0000 7fed4c8f2700 1 --2- 172.21.15.46:0/3062209200 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fed28066f30 0x7fed280693e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fed38009260 tx=0x7fed38016000 comp rx=0 tx=0).stop 2024-01-26T23:30:30.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.320+0000 7fed4c8f2700 1 -- 172.21.15.46:0/3062209200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4806add0 msgr2=0x7fed480b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:30.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.320+0000 7fed4c8f2700 1 --2- 172.21.15.46:0/3062209200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4806add0 0x7fed480b1f30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fed40009e90 tx=0x7fed4000b040 comp rx=0 tx=0).stop 2024-01-26T23:30:30.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.320+0000 7fed4c8f2700 1 -- 172.21.15.46:0/3062209200 shutdown_connections 2024-01-26T23:30:30.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.320+0000 7fed4c8f2700 1 --2- 172.21.15.46:0/3062209200 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fed28066f30 0x7fed280693e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.320+0000 7fed4c8f2700 1 --2- 172.21.15.46:0/3062209200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4806add0 0x7fed480b1f30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.320+0000 7fed4c8f2700 1 --2- 172.21.15.46:0/3062209200 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed48068b00 0x7fed480af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.320+0000 7fed4c8f2700 1 --2- 172.21.15.46:0/3062209200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed4806a660 0x7fed480ac420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.320+0000 7fed4c8f2700 1 -- 172.21.15.46:0/3062209200 >> 172.21.15.46:0/3062209200 conn(0x7fed480ca100 msgr2=0x7fed480cbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:30.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.320+0000 7fed4c8f2700 1 -- 172.21.15.46:0/3062209200 shutdown_connections 2024-01-26T23:30:30.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.320+0000 7fed4c8f2700 1 -- 172.21.15.46:0/3062209200 wait complete. 2024-01-26T23:30:30.334 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:30.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.465+0000 7f6bb1b5e700 1 Processor -- start 2024-01-26T23:30:30.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.465+0000 7f6bb1b5e700 1 -- start start 2024-01-26T23:30:30.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.466+0000 7f6bb1b5e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bac0b2530 0x7f6bac15a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:30.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.466+0000 7f6bb1b5e700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bac0a9b90 con 0x7f6bac0b2160 2024-01-26T23:30:30.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.466+0000 7f6bb1b5e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bac0aa210 con 0x7f6bac15da10 2024-01-26T23:30:30.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.466+0000 7f6bb1b5e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac15d580 con 0x7f6bac0b2530 2024-01-26T23:30:30.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.466+0000 7f6bab7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6bac0b2160 0x7f6bac0abe50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59048/0 (socket says 172.21.15.46:59048) 2024-01-26T23:30:30.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.466+0000 7f6bab7fe700 1 -- 172.21.15.46:0/4150460750 learned_addr learned my addr 172.21.15.46:0/4150460750 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:30.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.466+0000 7f6baaffd700 1 --2- 172.21.15.46:0/4150460750 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bac0b2530 0x7f6bac15a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:30.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.467+0000 7f6baaffd700 1 -- 172.21.15.46:0/4150460750 >> v1:172.21.15.106:6789/0 conn(0x7f6bac15da10 legacy=0x7f6bac06a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:30.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.467+0000 7f6baaffd700 1 -- 172.21.15.46:0/4150460750 >> v1:172.21.15.46:6789/0 conn(0x7f6bac0b2160 legacy=0x7f6bac0abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:30.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.467+0000 7f6baaffd700 1 -- 172.21.15.46:0/4150460750 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bac0aa460 con 0x7f6bac0b2530 2024-01-26T23:30:30.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.467+0000 7f6baaffd700 1 --2- 172.21.15.46:0/4150460750 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bac0b2530 0x7f6bac15a7a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0009910 tx=0x7f6ba00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c339c44cfe33bff server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.468+0000 7f6baa7fc700 1 -- 172.21.15.46:0/4150460750 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba000b860 con 0x7f6bac0b2530 2024-01-26T23:30:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.468+0000 7f6baa7fc700 1 -- 172.21.15.46:0/4150460750 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba000b9c0 con 0x7f6bac0b2530 2024-01-26T23:30:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.468+0000 7f6baa7fc700 1 -- 172.21.15.46:0/4150460750 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba000bd60 con 0x7f6bac0b2530 2024-01-26T23:30:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.469+0000 7f6bb1b5e700 1 -- 172.21.15.46:0/4150460750 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bac0b2530 msgr2=0x7f6bac15a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.469+0000 7f6bb1b5e700 1 --2- 172.21.15.46:0/4150460750 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bac0b2530 0x7f6bac15a7a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0009910 tx=0x7f6ba00091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.469+0000 7f6bb1b5e700 1 -- 172.21.15.46:0/4150460750 shutdown_connections 2024-01-26T23:30:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.469+0000 7f6bb1b5e700 1 --2- 172.21.15.46:0/4150460750 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bac0b2530 0x7f6bac15a7a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.469+0000 7f6bb1b5e700 1 -- 172.21.15.46:0/4150460750 >> 172.21.15.46:0/4150460750 conn(0x7f6bac0ccbc0 msgr2=0x7f6bac0ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.469+0000 7f6bb1b5e700 1 -- 172.21.15.46:0/4150460750 shutdown_connections 2024-01-26T23:30:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.469+0000 7f6bb1b5e700 1 -- 172.21.15.46:0/4150460750 wait complete. 2024-01-26T23:30:30.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.470+0000 7f6bb1b5e700 1 Processor -- start 2024-01-26T23:30:30.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.470+0000 7f6bb1b5e700 1 -- start start 2024-01-26T23:30:30.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.471+0000 7f6bb1b5e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6bac0b2160 0x7f6bac14a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:30.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.471+0000 7f6bb1b5e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bac0b2530 0x7f6bac0ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.471+0000 7f6bb1b5e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bac15da10 0x7f6bac068900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.471+0000 7f6bb1b5e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac14b150 con 0x7f6bac15da10 2024-01-26T23:30:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.471+0000 7f6bb1b5e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac0aea80 con 0x7f6bac0b2160 2024-01-26T23:30:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.471+0000 7f6bb1b5e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac0aed60 con 0x7f6bac0b2530 2024-01-26T23:30:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.471+0000 7f6babfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bac15da10 0x7f6bac068900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.471+0000 7f6baaffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bac0b2530 0x7f6bac0ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.471+0000 7f6babfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bac15da10 0x7f6bac068900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60720/0 (socket says 172.21.15.46:60720) 2024-01-26T23:30:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.471+0000 7f6babfff700 1 -- 172.21.15.46:0/1080188620 learned_addr learned my addr 172.21.15.46:0/1080188620 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.471+0000 7f6bab7fe700 1 --2- 172.21.15.46:0/1080188620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6bac0b2160 0x7f6bac14a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.472+0000 7f6babfff700 1 -- 172.21.15.46:0/1080188620 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bac0b2530 msgr2=0x7f6bac0ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.472+0000 7f6babfff700 1 --2- 172.21.15.46:0/1080188620 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bac0b2530 0x7f6bac0ae180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.472+0000 7f6babfff700 1 -- 172.21.15.46:0/1080188620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6bac0b2160 msgr2=0x7f6bac14a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:30.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.472+0000 7f6babfff700 1 --2- 172.21.15.46:0/1080188620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6bac0b2160 0x7f6bac14a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.472+0000 7f6babfff700 1 -- 172.21.15.46:0/1080188620 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba0009420 con 0x7f6bac15da10 2024-01-26T23:30:30.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.472+0000 7f6babfff700 1 --2- 172.21.15.46:0/1080188620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bac15da10 0x7f6bac068900 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6ba400c840 tx=0x7f6ba4001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:30.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.472+0000 7f6b937fe700 1 -- 172.21.15.46:0/1080188620 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4012630 con 0x7f6bac15da10 2024-01-26T23:30:30.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.472+0000 7f6b937fe700 1 -- 172.21.15.46:0/1080188620 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba4012790 con 0x7f6bac15da10 2024-01-26T23:30:30.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.472+0000 7f6bb1b5e700 1 -- 172.21.15.46:0/1080188620 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bac0af040 con 0x7f6bac15da10 2024-01-26T23:30:30.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.472+0000 7f6b937fe700 1 -- 172.21.15.46:0/1080188620 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4012b50 con 0x7f6bac15da10 2024-01-26T23:30:30.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.472+0000 7f6bb1b5e700 1 -- 172.21.15.46:0/1080188620 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bac14ad10 con 0x7f6bac15da10 2024-01-26T23:30:30.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.473+0000 7f6bb1b5e700 1 -- 172.21.15.46:0/1080188620 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bac04c710 con 0x7f6bac15da10 2024-01-26T23:30:30.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.474+0000 7f6b937fe700 1 -- 172.21.15.46:0/1080188620 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6ba4003030 con 0x7f6bac15da10 2024-01-26T23:30:30.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.474+0000 7f6b937fe700 1 --2- 172.21.15.46:0/1080188620 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6b98066f30 0x7f6b980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:30.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.474+0000 7f6b937fe700 1 -- 172.21.15.46:0/1080188620 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f6ba40845d0 con 0x7f6bac15da10 2024-01-26T23:30:30.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.474+0000 7f6bab7fe700 1 --2- 172.21.15.46:0/1080188620 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6b98066f30 0x7f6b980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:30.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.475+0000 7f6bab7fe700 1 --2- 172.21.15.46:0/1080188620 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6b98066f30 0x7f6b980693e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c007130 tx=0x7f6b9c007eb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:30.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.479+0000 7f6b937fe700 1 -- 172.21.15.46:0/1080188620 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6ba4051550 con 0x7f6bac15da10 2024-01-26T23:30:30.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.687+0000 7f6bb1b5e700 1 -- 172.21.15.46:0/1080188620 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bac0aa740 con 0x7f6bac15da10 2024-01-26T23:30:30.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.688+0000 7f6b937fe700 1 -- 172.21.15.46:0/1080188620 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7f6ba4054b70 con 0x7f6bac15da10 2024-01-26T23:30:30.689 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:30.690 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:29:49.075104+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.691+0000 7f6bb1b5e700 1 -- 172.21.15.46:0/1080188620 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6b98066f30 msgr2=0x7f6b980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.691+0000 7f6bb1b5e700 1 --2- 172.21.15.46:0/1080188620 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6b98066f30 0x7f6b980693e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c007130 tx=0x7f6b9c007eb0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.691+0000 7f6bb1b5e700 1 -- 172.21.15.46:0/1080188620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bac15da10 msgr2=0x7f6bac068900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.691+0000 7f6bb1b5e700 1 --2- 172.21.15.46:0/1080188620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bac15da10 0x7f6bac068900 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6ba400c840 tx=0x7f6ba4001670 comp rx=0 tx=0).stop 2024-01-26T23:30:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.691+0000 7f6bb1b5e700 1 -- 172.21.15.46:0/1080188620 shutdown_connections 2024-01-26T23:30:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.691+0000 7f6bb1b5e700 1 --2- 172.21.15.46:0/1080188620 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6b98066f30 0x7f6b980693e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.691+0000 7f6bb1b5e700 1 --2- 172.21.15.46:0/1080188620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bac15da10 0x7f6bac068900 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.691+0000 7f6bb1b5e700 1 --2- 172.21.15.46:0/1080188620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6bac0b2160 0x7f6bac14a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.691+0000 7f6bb1b5e700 1 --2- 172.21.15.46:0/1080188620 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6bac0b2530 0x7f6bac0ae180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.691+0000 7f6bb1b5e700 1 -- 172.21.15.46:0/1080188620 >> 172.21.15.46:0/1080188620 conn(0x7f6bac0ccbc0 msgr2=0x7f6bac0b1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.691+0000 7f6bb1b5e700 1 -- 172.21.15.46:0/1080188620 shutdown_connections 2024-01-26T23:30:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.691+0000 7f6bb1b5e700 1 -- 172.21.15.46:0/1080188620 wait complete. 2024-01-26T23:30:30.705 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-26T23:29:42.539759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-26T23:29:43.002669+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-01-26T23:29:46.646864+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-26T23:30:30.706 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:30:30.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.839+0000 7fcb1672e700 1 Processor -- start 2024-01-26T23:30:30.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.839+0000 7fcb1672e700 1 -- start start 2024-01-26T23:30:30.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.840+0000 7fcb1672e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcb10068b00 0x7fcb10068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:30.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.840+0000 7fcb1672e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fcb1006a660 0x7fcb10159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:30.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.840+0000 7fcb1672e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb100a9a80 con 0x7fcb1006add0 2024-01-26T23:30:30.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.840+0000 7fcb1672e700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb1006aab0 con 0x7fcb10068b00 2024-01-26T23:30:30.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.840+0000 7fcb1672e700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb1015a3e0 con 0x7fcb1006a660 2024-01-26T23:30:30.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.840+0000 7fcb0ffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcb10068b00 0x7fcb10068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:30.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.840+0000 7fcb0ffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcb10068b00 0x7fcb10068ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60726/0 (socket says 172.21.15.46:60726) 2024-01-26T23:30:30.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.840+0000 7fcb0ffff700 1 -- 172.21.15.46:0/3988881293 learned_addr learned my addr 172.21.15.46:0/3988881293 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:30.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.840+0000 7fcb0f7fe700 1 --2- 172.21.15.46:0/3988881293 >> v2:172.21.15.106:3300/0 conn(0x7fcb1006a660 0x7fcb10159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:30.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.840+0000 7fcb0ffff700 1 -- 172.21.15.46:0/3988881293 >> v2:172.21.15.106:3300/0 conn(0x7fcb1006a660 msgr2=0x7fcb10159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:30.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.840+0000 7fcb0ffff700 1 --2- 172.21.15.46:0/3988881293 >> v2:172.21.15.106:3300/0 conn(0x7fcb1006a660 0x7fcb10159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.840+0000 7fcb0ffff700 1 -- 172.21.15.46:0/3988881293 >> v1:172.21.15.106:6789/0 conn(0x7fcb1006add0 legacy=0x7fcb1006b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:30.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.840+0000 7fcb0ffff700 1 -- 172.21.15.46:0/3988881293 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb1015a680 con 0x7fcb10068b00 2024-01-26T23:30:30.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.841+0000 7fcb0ffff700 1 --2- 172.21.15.46:0/3988881293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb10068b00 0x7fcb10068ed0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcb10063ae0 tx=0x7fcb00009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa5363bbe83c4ec server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:30.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.841+0000 7fcb0effd700 1 -- 172.21.15.46:0/3988881293 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb00005c10 con 0x7fcb10068b00 2024-01-26T23:30:30.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.841+0000 7fcb0effd700 1 -- 172.21.15.46:0/3988881293 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb00005d70 con 0x7fcb10068b00 2024-01-26T23:30:30.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.841+0000 7fcb0effd700 1 -- 172.21.15.46:0/3988881293 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb00006260 con 0x7fcb10068b00 2024-01-26T23:30:30.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.841+0000 7fcb1672e700 1 -- 172.21.15.46:0/3988881293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb10068b00 msgr2=0x7fcb10068ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:30.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.841+0000 7fcb1672e700 1 --2- 172.21.15.46:0/3988881293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb10068b00 0x7fcb10068ed0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcb10063ae0 tx=0x7fcb00009220 comp rx=0 tx=0).stop 2024-01-26T23:30:30.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.841+0000 7fcb1672e700 1 -- 172.21.15.46:0/3988881293 shutdown_connections 2024-01-26T23:30:30.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.841+0000 7fcb1672e700 1 --2- 172.21.15.46:0/3988881293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb10068b00 0x7fcb10068ed0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.841+0000 7fcb1672e700 1 --2- 172.21.15.46:0/3988881293 >> v2:172.21.15.106:3300/0 conn(0x7fcb1006a660 0x7fcb10159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.841+0000 7fcb1672e700 1 -- 172.21.15.46:0/3988881293 >> 172.21.15.46:0/3988881293 conn(0x7fcb100ca100 msgr2=0x7fcb1006e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:30.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.841+0000 7fcb1672e700 1 -- 172.21.15.46:0/3988881293 shutdown_connections 2024-01-26T23:30:30.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.841+0000 7fcb1672e700 1 -- 172.21.15.46:0/3988881293 wait complete. 2024-01-26T23:30:30.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.843+0000 7fcb1672e700 1 Processor -- start 2024-01-26T23:30:30.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.843+0000 7fcb1672e700 1 -- start start 2024-01-26T23:30:30.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb1672e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb10068b00 0x7fcb10156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb1672e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb1006a660 0x7fcb10169a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb1672e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb1006add0 0x7fcb100cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb1672e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100afce0 con 0x7fcb1006add0 2024-01-26T23:30:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb1672e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1016a3a0 con 0x7fcb1006a660 2024-01-26T23:30:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb1672e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1016a680 con 0x7fcb10068b00 2024-01-26T23:30:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb14ccb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb1006add0 0x7fcb100cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb14ccb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb1006add0 0x7fcb100cb270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60740/0 (socket says 172.21.15.46:60740) 2024-01-26T23:30:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb14ccb700 1 -- 172.21.15.46:0/2852265815 learned_addr learned my addr 172.21.15.46:0/2852265815 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb0f7fe700 1 --2- 172.21.15.46:0/2852265815 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb1006a660 0x7fcb10169a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb0ffff700 1 --2- 172.21.15.46:0/2852265815 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb10068b00 0x7fcb10156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb14ccb700 1 -- 172.21.15.46:0/2852265815 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb10068b00 msgr2=0x7fcb10156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb14ccb700 1 --2- 172.21.15.46:0/2852265815 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb10068b00 0x7fcb10156ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb14ccb700 1 -- 172.21.15.46:0/2852265815 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb1006a660 msgr2=0x7fcb10169a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:30.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb14ccb700 1 --2- 172.21.15.46:0/2852265815 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb1006a660 0x7fcb10169a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:30.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb14ccb700 1 -- 172.21.15.46:0/2852265815 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb00009480 con 0x7fcb1006add0 2024-01-26T23:30:30.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb0f7fe700 1 --2- 172.21.15.46:0/2852265815 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb1006a660 0x7fcb10169a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:30.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.844+0000 7fcb14ccb700 1 --2- 172.21.15.46:0/2852265815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb1006add0 0x7fcb100cb270 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcb0800c8e0 tx=0x7fcb08001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:30.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.845+0000 7fcb07fff700 1 -- 172.21.15.46:0/2852265815 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb08012630 con 0x7fcb1006add0 2024-01-26T23:30:30.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.845+0000 7fcb07fff700 1 -- 172.21.15.46:0/2852265815 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb08012790 con 0x7fcb1006add0 2024-01-26T23:30:30.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.845+0000 7fcb07fff700 1 -- 172.21.15.46:0/2852265815 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb08012b50 con 0x7fcb1006add0 2024-01-26T23:30:30.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.845+0000 7fcb1672e700 1 -- 172.21.15.46:0/2852265815 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb1016a960 con 0x7fcb1006add0 2024-01-26T23:30:30.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.845+0000 7fcb1672e700 1 -- 172.21.15.46:0/2852265815 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb101571e0 con 0x7fcb1006add0 2024-01-26T23:30:30.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.846+0000 7fcb1672e700 1 -- 172.21.15.46:0/2852265815 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1004c710 con 0x7fcb1006add0 2024-01-26T23:30:30.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.846+0000 7fcb07fff700 1 -- 172.21.15.46:0/2852265815 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fcb0800a3b0 con 0x7fcb1006add0 2024-01-26T23:30:30.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.846+0000 7fcb07fff700 1 --2- 172.21.15.46:0/2852265815 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcaf0066f30 0x7fcaf00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:30.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.847+0000 7fcb07fff700 1 -- 172.21.15.46:0/2852265815 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fcb08084af0 con 0x7fcb1006add0 2024-01-26T23:30:30.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.847+0000 7fcb0ffff700 1 --2- 172.21.15.46:0/2852265815 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcaf0066f30 0x7fcaf00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:30.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.847+0000 7fcb0ffff700 1 --2- 172.21.15.46:0/2852265815 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcaf0066f30 0x7fcaf00693e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcb100b16b0 tx=0x7fcb00016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:30.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:30.851+0000 7fcb07fff700 1 -- 172.21.15.46:0/2852265815 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcb08051a70 con 0x7fcb1006add0 2024-01-26T23:30:31.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.054+0000 7fcb1672e700 1 -- 172.21.15.46:0/2852265815 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcb1006edf0 con 0x7fcb1006add0 2024-01-26T23:30:31.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.055+0000 7fcb07fff700 1 -- 172.21.15.46:0/2852265815 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcb08055090 con 0x7fcb1006add0 2024-01-26T23:30:31.056 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:30:31.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.057+0000 7fcb1672e700 1 -- 172.21.15.46:0/2852265815 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcaf0066f30 msgr2=0x7fcaf00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.057+0000 7fcb1672e700 1 --2- 172.21.15.46:0/2852265815 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcaf0066f30 0x7fcaf00693e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcb100b16b0 tx=0x7fcb00016000 comp rx=0 tx=0).stop 2024-01-26T23:30:31.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.057+0000 7fcb1672e700 1 -- 172.21.15.46:0/2852265815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb1006add0 msgr2=0x7fcb100cb270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.057+0000 7fcb1672e700 1 --2- 172.21.15.46:0/2852265815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb1006add0 0x7fcb100cb270 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcb0800c8e0 tx=0x7fcb08001710 comp rx=0 tx=0).stop 2024-01-26T23:30:31.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.057+0000 7fcb1672e700 1 -- 172.21.15.46:0/2852265815 shutdown_connections 2024-01-26T23:30:31.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.057+0000 7fcb1672e700 1 --2- 172.21.15.46:0/2852265815 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcaf0066f30 0x7fcaf00693e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.057+0000 7fcb1672e700 1 --2- 172.21.15.46:0/2852265815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb1006add0 0x7fcb100cb270 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.057+0000 7fcb1672e700 1 --2- 172.21.15.46:0/2852265815 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb10068b00 0x7fcb10156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.057+0000 7fcb1672e700 1 --2- 172.21.15.46:0/2852265815 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb1006a660 0x7fcb10169a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.057+0000 7fcb1672e700 1 -- 172.21.15.46:0/2852265815 >> 172.21.15.46:0/2852265815 conn(0x7fcb100ca100 msgr2=0x7fcb10159ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:31.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.057+0000 7fcb1672e700 1 -- 172.21.15.46:0/2852265815 shutdown_connections 2024-01-26T23:30:31.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.057+0000 7fcb1672e700 1 -- 172.21.15.46:0/2852265815 wait complete. 2024-01-26T23:30:31.071 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-26T23:30:31.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.204+0000 7f4ef4de5700 1 Processor -- start 2024-01-26T23:30:31.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.204+0000 7f4ef4de5700 1 -- start start 2024-01-26T23:30:31.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.205+0000 7f4ef4de5700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4ef0159510 0x7f4ef01598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:31.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.205+0000 7f4ef4de5700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef0069260 con 0x7f4ef00ca220 2024-01-26T23:30:31.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.205+0000 7f4ef4de5700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef015d670 con 0x7f4ef00ca690 2024-01-26T23:30:31.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.205+0000 7f4ef4de5700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef00c9750 con 0x7f4ef0159510 2024-01-26T23:30:31.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.205+0000 7f4eedd9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4ef00ca220 0x7f4ef00c8be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59060/0 (socket says 172.21.15.46:59060) 2024-01-26T23:30:31.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.205+0000 7f4eedd9b700 1 -- 172.21.15.46:0/227660211 learned_addr learned my addr 172.21.15.46:0/227660211 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:31.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.205+0000 7f4eee59c700 1 --2- 172.21.15.46:0/227660211 >> v2:172.21.15.106:3300/0 conn(0x7f4ef0159510 0x7f4ef01598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:31.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.206+0000 7f4eed59a700 1 -- 172.21.15.46:0/227660211 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877127257 0 0) 0x7f4ef0069260 con 0x7f4ef00ca220 2024-01-26T23:30:31.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.206+0000 7f4eed59a700 1 -- 172.21.15.46:0/227660211 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ed8003570 con 0x7f4ef00ca220 2024-01-26T23:30:31.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.206+0000 7f4eee59c700 1 -- 172.21.15.46:0/227660211 >> v1:172.21.15.106:6789/0 conn(0x7f4ef00ca690 legacy=0x7f4ef00cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.206+0000 7f4eee59c700 1 -- 172.21.15.46:0/227660211 >> v1:172.21.15.46:6789/0 conn(0x7f4ef00ca220 legacy=0x7f4ef00c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.206+0000 7f4eed59a700 1 -- 172.21.15.46:0/227660211 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967143555 0 0) 0x7f4ef015d670 con 0x7f4ef00ca690 2024-01-26T23:30:31.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.206+0000 7f4eee59c700 1 -- 172.21.15.46:0/227660211 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef00c9a30 con 0x7f4ef0159510 2024-01-26T23:30:31.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.206+0000 7f4eee59c700 1 --2- 172.21.15.46:0/227660211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef0159510 0x7f4ef01598e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4ee00098e0 tx=0x7f4ee00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b16255df9860f43f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:31.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.206+0000 7f4eed59a700 1 -- 172.21.15.46:0/227660211 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee000b640 con 0x7f4ef0159510 2024-01-26T23:30:31.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.206+0000 7f4eed59a700 1 -- 172.21.15.46:0/227660211 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee000b7a0 con 0x7f4ef0159510 2024-01-26T23:30:31.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.206+0000 7f4eed59a700 1 -- 172.21.15.46:0/227660211 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee000bb40 con 0x7f4ef0159510 2024-01-26T23:30:31.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.207+0000 7f4ef4de5700 1 -- 172.21.15.46:0/227660211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef0159510 msgr2=0x7f4ef01598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.207+0000 7f4ef4de5700 1 --2- 172.21.15.46:0/227660211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef0159510 0x7f4ef01598e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4ee00098e0 tx=0x7f4ee00091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.207+0000 7f4ef4de5700 1 -- 172.21.15.46:0/227660211 shutdown_connections 2024-01-26T23:30:31.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.207+0000 7f4ef4de5700 1 --2- 172.21.15.46:0/227660211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef0159510 0x7f4ef01598e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.207+0000 7f4ef4de5700 1 -- 172.21.15.46:0/227660211 >> 172.21.15.46:0/227660211 conn(0x7f4ef006bd90 msgr2=0x7f4ef006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:31.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.207+0000 7f4ef4de5700 1 -- 172.21.15.46:0/227660211 shutdown_connections 2024-01-26T23:30:31.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.207+0000 7f4ef4de5700 1 -- 172.21.15.46:0/227660211 wait complete. 2024-01-26T23:30:31.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.208+0000 7f4ef4de5700 1 Processor -- start 2024-01-26T23:30:31.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.208+0000 7f4ef4de5700 1 -- start start 2024-01-26T23:30:31.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.209+0000 7f4ef4de5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ef00ca220 0x7f4ef00c4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:31.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.209+0000 7f4ef4de5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef0159510 0x7f4ef0151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:31.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.209+0000 7f4ef4de5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ef006df80 0x7f4ef006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:31.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.209+0000 7f4ef4de5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef01407b0 con 0x7f4ef00ca220 2024-01-26T23:30:31.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.209+0000 7f4ef4de5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef00b21f0 con 0x7f4ef0159510 2024-01-26T23:30:31.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.209+0000 7f4ef4de5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef00b1340 con 0x7f4ef006df80 2024-01-26T23:30:31.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.209+0000 7f4eee59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ef00ca220 0x7f4ef00c4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:31.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.209+0000 7f4eee59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ef00ca220 0x7f4ef00c4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60750/0 (socket says 172.21.15.46:60750) 2024-01-26T23:30:31.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.209+0000 7f4eee59c700 1 -- 172.21.15.46:0/2755643176 learned_addr learned my addr 172.21.15.46:0/2755643176 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:31.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.209+0000 7f4eedd9b700 1 --2- 172.21.15.46:0/2755643176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef0159510 0x7f4ef0151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:31.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.209+0000 7f4eeed9d700 1 --2- 172.21.15.46:0/2755643176 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ef006df80 0x7f4ef006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:31.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.209+0000 7f4eee59c700 1 -- 172.21.15.46:0/2755643176 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ef006df80 msgr2=0x7f4ef006d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.209+0000 7f4eee59c700 1 --2- 172.21.15.46:0/2755643176 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ef006df80 0x7f4ef006d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.209+0000 7f4eee59c700 1 -- 172.21.15.46:0/2755643176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef0159510 msgr2=0x7f4ef0151860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.209+0000 7f4eee59c700 1 --2- 172.21.15.46:0/2755643176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef0159510 0x7f4ef0151860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.209+0000 7f4eee59c700 1 -- 172.21.15.46:0/2755643176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee0009420 con 0x7f4ef00ca220 2024-01-26T23:30:31.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.209+0000 7f4eee59c700 1 --2- 172.21.15.46:0/2755643176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ef00ca220 0x7f4ef00c4cb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4ef006f5a0 tx=0x7f4ee0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:31.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.210+0000 7f4ede7fc700 1 -- 172.21.15.46:0/2755643176 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee001f070 con 0x7f4ef00ca220 2024-01-26T23:30:31.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.210+0000 7f4ede7fc700 1 -- 172.21.15.46:0/2755643176 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee000bb20 con 0x7f4ef00ca220 2024-01-26T23:30:31.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.210+0000 7f4ef4de5700 1 -- 172.21.15.46:0/2755643176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef00b15e0 con 0x7f4ef00ca220 2024-01-26T23:30:31.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.210+0000 7f4ede7fc700 1 -- 172.21.15.46:0/2755643176 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0011670 con 0x7f4ef00ca220 2024-01-26T23:30:31.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.210+0000 7f4ef4de5700 1 -- 172.21.15.46:0/2755643176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef00b1ba0 con 0x7f4ef00ca220 2024-01-26T23:30:31.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.211+0000 7f4ef4de5700 1 -- 172.21.15.46:0/2755643176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef004c710 con 0x7f4ef00ca220 2024-01-26T23:30:31.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.211+0000 7f4ede7fc700 1 -- 172.21.15.46:0/2755643176 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4ee0011a60 con 0x7f4ef00ca220 2024-01-26T23:30:31.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.212+0000 7f4ede7fc700 1 --2- 172.21.15.46:0/2755643176 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4ed0066f30 0x7f4ed00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:31.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.212+0000 7f4ede7fc700 1 -- 172.21.15.46:0/2755643176 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f4ee0085c60 con 0x7f4ef00ca220 2024-01-26T23:30:31.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.212+0000 7f4eedd9b700 1 --2- 172.21.15.46:0/2755643176 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4ed0066f30 0x7f4ed00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:31.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.212+0000 7f4eedd9b700 1 --2- 172.21.15.46:0/2755643176 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4ed0066f30 0x7f4ed00693e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4ef00c51f0 tx=0x7f4ee4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:31.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.216+0000 7f4ede7fc700 1 -- 172.21.15.46:0/2755643176 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4ee0052be0 con 0x7f4ef00ca220 2024-01-26T23:30:31.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.419+0000 7f4ef4de5700 1 -- 172.21.15.46:0/2755643176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4ef00c9d10 con 0x7f4ef00ca220 2024-01-26T23:30:31.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.420+0000 7f4ede7fc700 1 -- 172.21.15.46:0/2755643176 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4ee0028020 con 0x7f4ef00ca220 2024-01-26T23:30:31.421 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:30:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.422+0000 7f4ef4de5700 1 -- 172.21.15.46:0/2755643176 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4ed0066f30 msgr2=0x7f4ed00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.422+0000 7f4ef4de5700 1 --2- 172.21.15.46:0/2755643176 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4ed0066f30 0x7f4ed00693e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4ef00c51f0 tx=0x7f4ee4009040 comp rx=0 tx=0).stop 2024-01-26T23:30:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.422+0000 7f4ef4de5700 1 -- 172.21.15.46:0/2755643176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ef00ca220 msgr2=0x7f4ef00c4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.422+0000 7f4ef4de5700 1 --2- 172.21.15.46:0/2755643176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ef00ca220 0x7f4ef00c4cb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4ef006f5a0 tx=0x7f4ee0017000 comp rx=0 tx=0).stop 2024-01-26T23:30:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.422+0000 7f4ef4de5700 1 -- 172.21.15.46:0/2755643176 shutdown_connections 2024-01-26T23:30:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.422+0000 7f4ef4de5700 1 --2- 172.21.15.46:0/2755643176 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4ed0066f30 0x7f4ed00693e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.422+0000 7f4ef4de5700 1 --2- 172.21.15.46:0/2755643176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ef00ca220 0x7f4ef00c4cb0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.422+0000 7f4ef4de5700 1 --2- 172.21.15.46:0/2755643176 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ef006df80 0x7f4ef006d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.422+0000 7f4ef4de5700 1 --2- 172.21.15.46:0/2755643176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ef0159510 0x7f4ef0151860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.422+0000 7f4ef4de5700 1 -- 172.21.15.46:0/2755643176 >> 172.21.15.46:0/2755643176 conn(0x7f4ef006bd90 msgr2=0x7f4ef0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.422+0000 7f4ef4de5700 1 -- 172.21.15.46:0/2755643176 shutdown_connections 2024-01-26T23:30:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.422+0000 7f4ef4de5700 1 -- 172.21.15.46:0/2755643176 wait complete. 2024-01-26T23:30:31.436 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-26T23:30:31.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.571+0000 7f41e8a97700 1 Processor -- start 2024-01-26T23:30:31.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.571+0000 7f41e8a97700 1 -- start start 2024-01-26T23:30:31.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.572+0000 7f41e8a97700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41e40c9770 0x7f41e4159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:31.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.572+0000 7f41e8a97700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e415e390 con 0x7f41e4159760 2024-01-26T23:30:31.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.572+0000 7f41e8a97700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e4068460 con 0x7f41e40c93a0 2024-01-26T23:30:31.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.572+0000 7f41e8a97700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e40c9be0 con 0x7f41e40c9770 2024-01-26T23:30:31.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.572+0000 7f41e2d9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f41e4159760 0x7f41e40c7e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59074/0 (socket says 172.21.15.46:59074) 2024-01-26T23:30:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.572+0000 7f41e2d9d700 1 -- 172.21.15.46:0/2125527661 learned_addr learned my addr 172.21.15.46:0/2125527661 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.573+0000 7f41e1d9b700 1 --2- 172.21.15.46:0/2125527661 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41e40c9770 0x7f41e4159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.573+0000 7f41e159a700 1 -- 172.21.15.46:0/2125527661 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361423903 0 0) 0x7f41e415e390 con 0x7f41e4159760 2024-01-26T23:30:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.573+0000 7f41e159a700 1 -- 172.21.15.46:0/2125527661 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41d0003570 con 0x7f41e4159760 2024-01-26T23:30:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.573+0000 7f41e159a700 1 -- 172.21.15.46:0/2125527661 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336649775 0 0) 0x7f41e4068460 con 0x7f41e40c93a0 2024-01-26T23:30:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.573+0000 7f41e159a700 1 -- 172.21.15.46:0/2125527661 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41e415e390 con 0x7f41e40c93a0 2024-01-26T23:30:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.573+0000 7f41e159a700 1 -- 172.21.15.46:0/2125527661 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 233222694 0 0) 0x7f41d0003570 con 0x7f41e4159760 2024-01-26T23:30:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.573+0000 7f41e1d9b700 1 -- 172.21.15.46:0/2125527661 >> v1:172.21.15.106:6789/0 conn(0x7f41e40c93a0 legacy=0x7f41e4067d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.573+0000 7f41e1d9b700 1 -- 172.21.15.46:0/2125527661 >> v1:172.21.15.46:6789/0 conn(0x7f41e4159760 legacy=0x7f41e40c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.573+0000 7f41e1d9b700 1 -- 172.21.15.46:0/2125527661 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e415d810 con 0x7f41e40c9770 2024-01-26T23:30:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.573+0000 7f41e1d9b700 1 --2- 172.21.15.46:0/2125527661 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41e40c9770 0x7f41e4159220 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f41d4009910 tx=0x7f41d40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c37b40f893e34a56 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.573+0000 7f41e159a700 1 -- 172.21.15.46:0/2125527661 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc0017f0 con 0x7f41e40c9770 2024-01-26T23:30:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.573+0000 7f41e159a700 1 -- 172.21.15.46:0/2125527661 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d4011420 con 0x7f41e40c9770 2024-01-26T23:30:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.573+0000 7f41e159a700 1 -- 172.21.15.46:0/2125527661 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d40117c0 con 0x7f41e40c9770 2024-01-26T23:30:31.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.574+0000 7f41e8a97700 1 -- 172.21.15.46:0/2125527661 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41e40c9770 msgr2=0x7f41e4159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.574+0000 7f41e8a97700 1 --2- 172.21.15.46:0/2125527661 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41e40c9770 0x7f41e4159220 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f41d4009910 tx=0x7f41d40091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.574+0000 7f41e8a97700 1 -- 172.21.15.46:0/2125527661 shutdown_connections 2024-01-26T23:30:31.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.574+0000 7f41e8a97700 1 --2- 172.21.15.46:0/2125527661 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41e40c9770 0x7f41e4159220 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.574+0000 7f41e8a97700 1 -- 172.21.15.46:0/2125527661 >> 172.21.15.46:0/2125527661 conn(0x7f41e40ca9a0 msgr2=0x7f41e4068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:31.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.574+0000 7f41e8a97700 1 -- 172.21.15.46:0/2125527661 shutdown_connections 2024-01-26T23:30:31.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.574+0000 7f41e8a97700 1 -- 172.21.15.46:0/2125527661 wait complete. 2024-01-26T23:30:31.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.575+0000 7f41e8a97700 1 Processor -- start 2024-01-26T23:30:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.575+0000 7f41e8a97700 1 -- start start 2024-01-26T23:30:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.576+0000 7f41e8a97700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41e40c93a0 0x7f41e40b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.576+0000 7f41e8a97700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41e40c9770 0x7f41e4149060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.576+0000 7f41e8a97700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41e4159760 0x7f41e4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.576+0000 7f41e8a97700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e4159bf0 con 0x7f41e4159760 2024-01-26T23:30:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.576+0000 7f41e8a97700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e40b17b0 con 0x7f41e40c93a0 2024-01-26T23:30:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.576+0000 7f41e8a97700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e40b1a90 con 0x7f41e40c9770 2024-01-26T23:30:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.576+0000 7f41e2d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41e4159760 0x7f41e4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.576+0000 7f41e2d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41e4159760 0x7f41e4155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60752/0 (socket says 172.21.15.46:60752) 2024-01-26T23:30:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.576+0000 7f41e2d9d700 1 -- 172.21.15.46:0/1697114525 learned_addr learned my addr 172.21.15.46:0/1697114525 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.576+0000 7f41e1d9b700 1 --2- 172.21.15.46:0/1697114525 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41e40c9770 0x7f41e4149060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.576+0000 7f41e259c700 1 --2- 172.21.15.46:0/1697114525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41e40c93a0 0x7f41e40b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.577+0000 7f41e2d9d700 1 -- 172.21.15.46:0/1697114525 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41e40c9770 msgr2=0x7f41e4149060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.577+0000 7f41e2d9d700 1 --2- 172.21.15.46:0/1697114525 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41e40c9770 0x7f41e4149060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.577+0000 7f41e2d9d700 1 -- 172.21.15.46:0/1697114525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41e40c93a0 msgr2=0x7f41e40b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.577+0000 7f41e2d9d700 1 --2- 172.21.15.46:0/1697114525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41e40c93a0 0x7f41e40b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.577+0000 7f41e2d9d700 1 -- 172.21.15.46:0/1697114525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d4009420 con 0x7f41e4159760 2024-01-26T23:30:31.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.577+0000 7f41e2d9d700 1 --2- 172.21.15.46:0/1697114525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41e4159760 0x7f41e4155c10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f41dc00cc10 tx=0x7f41dc007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:31.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.577+0000 7f41da7fc700 1 -- 172.21.15.46:0/1697114525 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc004dc0 con 0x7f41e4159760 2024-01-26T23:30:31.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.577+0000 7f41e8a97700 1 -- 172.21.15.46:0/1697114525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e40b1d70 con 0x7f41e4159760 2024-01-26T23:30:31.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.577+0000 7f41da7fc700 1 -- 172.21.15.46:0/1697114525 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41dc00a5c0 con 0x7f41e4159760 2024-01-26T23:30:31.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.577+0000 7f41da7fc700 1 -- 172.21.15.46:0/1697114525 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc00a980 con 0x7f41e4159760 2024-01-26T23:30:31.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.577+0000 7f41e8a97700 1 -- 172.21.15.46:0/1697114525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e4156150 con 0x7f41e4159760 2024-01-26T23:30:31.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.578+0000 7f41e8a97700 1 -- 172.21.15.46:0/1697114525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e4159d60 con 0x7f41e4159760 2024-01-26T23:30:31.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.579+0000 7f41da7fc700 1 -- 172.21.15.46:0/1697114525 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f41dc007150 con 0x7f41e4159760 2024-01-26T23:30:31.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.579+0000 7f41da7fc700 1 --2- 172.21.15.46:0/1697114525 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f41c4066f30 0x7f41c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:31.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.579+0000 7f41da7fc700 1 -- 172.21.15.46:0/1697114525 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f41dc083f40 con 0x7f41e4159760 2024-01-26T23:30:31.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.579+0000 7f41e259c700 1 --2- 172.21.15.46:0/1697114525 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f41c4066f30 0x7f41c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:31.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.580+0000 7f41e259c700 1 --2- 172.21.15.46:0/1697114525 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f41c4066f30 0x7f41c40693e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f41cc007fd0 tx=0x7f41cc007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:31.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.584+0000 7f41da7fc700 1 -- 172.21.15.46:0/1697114525 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f41e4159d60 con 0x7f41e4159760 2024-01-26T23:30:31.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.790+0000 7f41e8a97700 1 -- 172.21.15.46:0/1697114525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f41e4159d30 con 0x7f41e4159760 2024-01-26T23:30:31.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.790+0000 7f41da7fc700 1 -- 172.21.15.46:0/1697114525 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f41dc018020 con 0x7f41e4159760 2024-01-26T23:30:31.791 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-26T23:30:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.793+0000 7f41e8a97700 1 -- 172.21.15.46:0/1697114525 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f41c4066f30 msgr2=0x7f41c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.793+0000 7f41e8a97700 1 --2- 172.21.15.46:0/1697114525 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f41c4066f30 0x7f41c40693e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f41cc007fd0 tx=0x7f41cc007d50 comp rx=0 tx=0).stop 2024-01-26T23:30:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.793+0000 7f41e8a97700 1 -- 172.21.15.46:0/1697114525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41e4159760 msgr2=0x7f41e4155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.793+0000 7f41e8a97700 1 --2- 172.21.15.46:0/1697114525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41e4159760 0x7f41e4155c10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f41dc00cc10 tx=0x7f41dc007d20 comp rx=0 tx=0).stop 2024-01-26T23:30:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.793+0000 7f41e8a97700 1 -- 172.21.15.46:0/1697114525 shutdown_connections 2024-01-26T23:30:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.793+0000 7f41e8a97700 1 --2- 172.21.15.46:0/1697114525 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f41c4066f30 0x7f41c40693e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.793+0000 7f41e8a97700 1 --2- 172.21.15.46:0/1697114525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41e4159760 0x7f41e4155c10 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.793+0000 7f41e8a97700 1 --2- 172.21.15.46:0/1697114525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41e40c93a0 0x7f41e40b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.793+0000 7f41e8a97700 1 --2- 172.21.15.46:0/1697114525 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41e40c9770 0x7f41e4149060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.793+0000 7f41e8a97700 1 -- 172.21.15.46:0/1697114525 >> 172.21.15.46:0/1697114525 conn(0x7f41e40ca9a0 msgr2=0x7f41e4154dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:31.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.793+0000 7f41e8a97700 1 -- 172.21.15.46:0/1697114525 shutdown_connections 2024-01-26T23:30:31.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.793+0000 7f41e8a97700 1 -- 172.21.15.46:0/1697114525 wait complete. 2024-01-26T23:30:31.808 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-01-26T23:30:31.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.942+0000 7ff4d3156700 1 Processor -- start 2024-01-26T23:30:31.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.942+0000 7ff4d3156700 1 -- start start 2024-01-26T23:30:31.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.942+0000 7ff4d3156700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4cc0cd2c0 0x7ff4cc0cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:31.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.942+0000 7ff4d3156700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4cc0cce30 con 0x7ff4cc159aa0 2024-01-26T23:30:31.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.942+0000 7ff4d3156700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4cc0c9970 con 0x7ff4cc159e70 2024-01-26T23:30:31.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.942+0000 7ff4d3156700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc0c9c50 con 0x7ff4cc0cd2c0 2024-01-26T23:30:31.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.943+0000 7ff4d0ef2700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff4cc159aa0 0x7ff4cc0c7c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59080/0 (socket says 172.21.15.46:59080) 2024-01-26T23:30:31.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.943+0000 7ff4d0ef2700 1 -- 172.21.15.46:0/3737597742 learned_addr learned my addr 172.21.15.46:0/3737597742 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:31.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.943+0000 7ff4cbfff700 1 --2- 172.21.15.46:0/3737597742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4cc0cd2c0 0x7ff4cc0cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:31.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.943+0000 7ff4cb7fe700 1 -- 172.21.15.46:0/3737597742 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880417908 0 0) 0x7ff4cc0cce30 con 0x7ff4cc159aa0 2024-01-26T23:30:31.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.943+0000 7ff4cb7fe700 1 -- 172.21.15.46:0/3737597742 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b8003570 con 0x7ff4cc159aa0 2024-01-26T23:30:31.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.943+0000 7ff4cb7fe700 1 -- 172.21.15.46:0/3737597742 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810781714 0 0) 0x7ff4cc0c9970 con 0x7ff4cc159e70 2024-01-26T23:30:31.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.943+0000 7ff4cbfff700 1 -- 172.21.15.46:0/3737597742 >> v1:172.21.15.106:6789/0 conn(0x7ff4cc159e70 legacy=0x7ff4cc067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.943+0000 7ff4cbfff700 1 -- 172.21.15.46:0/3737597742 >> v1:172.21.15.46:6789/0 conn(0x7ff4cc159aa0 legacy=0x7ff4cc0c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.943+0000 7ff4cbfff700 1 -- 172.21.15.46:0/3737597742 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4cc0c9620 con 0x7ff4cc0cd2c0 2024-01-26T23:30:31.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.943+0000 7ff4cbfff700 1 --2- 172.21.15.46:0/3737597742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4cc0cd2c0 0x7ff4cc0cdff0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009910 tx=0x7ff4b40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7c612afdad3ee83 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:31.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.944+0000 7ff4cb7fe700 1 -- 172.21.15.46:0/3737597742 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b400b680 con 0x7ff4cc0cd2c0 2024-01-26T23:30:31.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.944+0000 7ff4cb7fe700 1 -- 172.21.15.46:0/3737597742 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b400b7e0 con 0x7ff4cc0cd2c0 2024-01-26T23:30:31.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.944+0000 7ff4cb7fe700 1 -- 172.21.15.46:0/3737597742 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b400bba0 con 0x7ff4cc0cd2c0 2024-01-26T23:30:31.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.944+0000 7ff4d3156700 1 -- 172.21.15.46:0/3737597742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4cc0cd2c0 msgr2=0x7ff4cc0cdff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.944+0000 7ff4d3156700 1 --2- 172.21.15.46:0/3737597742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4cc0cd2c0 0x7ff4cc0cdff0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009910 tx=0x7ff4b40091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.944+0000 7ff4d3156700 1 -- 172.21.15.46:0/3737597742 shutdown_connections 2024-01-26T23:30:31.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.944+0000 7ff4d3156700 1 --2- 172.21.15.46:0/3737597742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4cc0cd2c0 0x7ff4cc0cdff0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.944+0000 7ff4d3156700 1 -- 172.21.15.46:0/3737597742 >> 172.21.15.46:0/3737597742 conn(0x7ff4cc069220 msgr2=0x7ff4cc0cd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:31.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.944+0000 7ff4d3156700 1 -- 172.21.15.46:0/3737597742 shutdown_connections 2024-01-26T23:30:31.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.944+0000 7ff4d3156700 1 -- 172.21.15.46:0/3737597742 wait complete. 2024-01-26T23:30:31.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.945+0000 7ff4d3156700 1 Processor -- start 2024-01-26T23:30:31.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.946+0000 7ff4d3156700 1 -- start start 2024-01-26T23:30:31.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.946+0000 7ff4d3156700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4cc0cd2c0 0x7ff4cc0c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:31.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.946+0000 7ff4d3156700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4cc159aa0 0x7ff4cc16d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:31.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.946+0000 7ff4d3156700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4cc159e70 0x7ff4cc1567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:31.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.946+0000 7ff4d3156700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc0af890 con 0x7ff4cc159e70 2024-01-26T23:30:31.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.946+0000 7ff4d3156700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc16db60 con 0x7ff4cc0cd2c0 2024-01-26T23:30:31.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.946+0000 7ff4d3156700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc0ace90 con 0x7ff4cc159aa0 2024-01-26T23:30:31.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.946+0000 7ff4d16f3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4cc159e70 0x7ff4cc1567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:31.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.947+0000 7ff4d16f3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4cc159e70 0x7ff4cc1567a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60766/0 (socket says 172.21.15.46:60766) 2024-01-26T23:30:31.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.947+0000 7ff4d16f3700 1 -- 172.21.15.46:0/3844213537 learned_addr learned my addr 172.21.15.46:0/3844213537 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:31.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.947+0000 7ff4d0ef2700 1 --2- 172.21.15.46:0/3844213537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4cc0cd2c0 0x7ff4cc0c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:31.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.947+0000 7ff4cbfff700 1 --2- 172.21.15.46:0/3844213537 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4cc159aa0 0x7ff4cc16d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:31.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.947+0000 7ff4d16f3700 1 -- 172.21.15.46:0/3844213537 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4cc159aa0 msgr2=0x7ff4cc16d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.947+0000 7ff4d16f3700 1 --2- 172.21.15.46:0/3844213537 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4cc159aa0 0x7ff4cc16d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.947+0000 7ff4d16f3700 1 -- 172.21.15.46:0/3844213537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4cc0cd2c0 msgr2=0x7ff4cc0c3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:31.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.947+0000 7ff4d16f3700 1 --2- 172.21.15.46:0/3844213537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4cc0cd2c0 0x7ff4cc0c3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:31.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.947+0000 7ff4d16f3700 1 -- 172.21.15.46:0/3844213537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b4009420 con 0x7ff4cc159e70 2024-01-26T23:30:31.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.947+0000 7ff4d16f3700 1 --2- 172.21.15.46:0/3844213537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4cc159e70 0x7ff4cc1567a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff4c400cb60 tx=0x7ff4c4001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:31.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.947+0000 7ff4c8ff9700 1 -- 172.21.15.46:0/3844213537 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c40040d0 con 0x7ff4cc159e70 2024-01-26T23:30:31.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.947+0000 7ff4c8ff9700 1 -- 172.21.15.46:0/3844213537 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c400a7a0 con 0x7ff4cc159e70 2024-01-26T23:30:31.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.947+0000 7ff4d3156700 1 -- 172.21.15.46:0/3844213537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4cc16dd90 con 0x7ff4cc159e70 2024-01-26T23:30:31.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.947+0000 7ff4c8ff9700 1 -- 172.21.15.46:0/3844213537 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c400ab60 con 0x7ff4cc159e70 2024-01-26T23:30:31.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.947+0000 7ff4d3156700 1 -- 172.21.15.46:0/3844213537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4cc16e350 con 0x7ff4cc159e70 2024-01-26T23:30:31.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.948+0000 7ff4d3156700 1 -- 172.21.15.46:0/3844213537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4cc0acc60 con 0x7ff4cc159e70 2024-01-26T23:30:31.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.949+0000 7ff4c8ff9700 1 -- 172.21.15.46:0/3844213537 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff4c4012420 con 0x7ff4cc159e70 2024-01-26T23:30:31.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.949+0000 7ff4c8ff9700 1 --2- 172.21.15.46:0/3844213537 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff4ac066f30 0x7ff4ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:31.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.949+0000 7ff4c8ff9700 1 -- 172.21.15.46:0/3844213537 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7ff4c4084580 con 0x7ff4cc159e70 2024-01-26T23:30:31.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.949+0000 7ff4d0ef2700 1 --2- 172.21.15.46:0/3844213537 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff4ac066f30 0x7ff4ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:31.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.950+0000 7ff4d0ef2700 1 --2- 172.21.15.46:0/3844213537 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff4ac066f30 0x7ff4ac0693e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc000c00 tx=0x7ff4bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:31.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:31.954+0000 7ff4c8ff9700 1 -- 172.21.15.46:0/3844213537 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff4cc0acc60 con 0x7ff4cc159e70 2024-01-26T23:30:32.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.160+0000 7ff4d3156700 1 -- 172.21.15.46:0/3844213537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff4cc0acc60 con 0x7ff4cc159e70 2024-01-26T23:30:32.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.160+0000 7ff4c8ff9700 1 -- 172.21.15.46:0/3844213537 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff4cc0acc60 con 0x7ff4cc159e70 2024-01-26T23:30:32.161 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:30:32.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.162+0000 7ff4d3156700 1 -- 172.21.15.46:0/3844213537 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff4ac066f30 msgr2=0x7ff4ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:32.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.162+0000 7ff4d3156700 1 --2- 172.21.15.46:0/3844213537 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff4ac066f30 0x7ff4ac0693e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc000c00 tx=0x7ff4bc009040 comp rx=0 tx=0).stop 2024-01-26T23:30:32.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.162+0000 7ff4d3156700 1 -- 172.21.15.46:0/3844213537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4cc159e70 msgr2=0x7ff4cc1567a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:32.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.162+0000 7ff4d3156700 1 --2- 172.21.15.46:0/3844213537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4cc159e70 0x7ff4cc1567a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff4c400cb60 tx=0x7ff4c4001770 comp rx=0 tx=0).stop 2024-01-26T23:30:32.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.162+0000 7ff4d3156700 1 -- 172.21.15.46:0/3844213537 shutdown_connections 2024-01-26T23:30:32.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.162+0000 7ff4d3156700 1 --2- 172.21.15.46:0/3844213537 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff4ac066f30 0x7ff4ac0693e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.162+0000 7ff4d3156700 1 --2- 172.21.15.46:0/3844213537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4cc159e70 0x7ff4cc1567a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.162+0000 7ff4d3156700 1 --2- 172.21.15.46:0/3844213537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4cc0cd2c0 0x7ff4cc0c3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.162+0000 7ff4d3156700 1 --2- 172.21.15.46:0/3844213537 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4cc159aa0 0x7ff4cc16d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.162+0000 7ff4d3156700 1 -- 172.21.15.46:0/3844213537 >> 172.21.15.46:0/3844213537 conn(0x7ff4cc069220 msgr2=0x7ff4cc158a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:32.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.162+0000 7ff4d3156700 1 -- 172.21.15.46:0/3844213537 shutdown_connections 2024-01-26T23:30:32.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.162+0000 7ff4d3156700 1 -- 172.21.15.46:0/3844213537 wait complete. 2024-01-26T23:30:32.177 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:32.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.309+0000 7f3ca5a02700 1 Processor -- start 2024-01-26T23:30:32.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.309+0000 7f3ca5a02700 1 -- start start 2024-01-26T23:30:32.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.310+0000 7f3ca5a02700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3ca00c6520 0x7f3ca00ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.310+0000 7f3ca5a02700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ca00ced50 0x7f3ca00cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.310+0000 7f3ca5a02700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca015a0b0 con 0x7f3ca00b2b40 2024-01-26T23:30:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.310+0000 7f3ca5a02700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ca00af070 con 0x7f3ca00c6520 2024-01-26T23:30:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.310+0000 7f3ca5a02700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca0099fc0 con 0x7f3ca00ced50 2024-01-26T23:30:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.310+0000 7f3c9effd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3ca00b2b40 0x7f3ca00ae960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59092/0 (socket says 172.21.15.46:59092) 2024-01-26T23:30:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.310+0000 7f3c9effd700 1 -- 172.21.15.46:0/1243096408 learned_addr learned my addr 172.21.15.46:0/1243096408 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.310+0000 7f3c9e7fc700 1 --2- 172.21.15.46:0/1243096408 >> v2:172.21.15.106:3300/0 conn(0x7f3ca00c6520 0x7f3ca00ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.310+0000 7f3c9f7fe700 1 --2- 172.21.15.46:0/1243096408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ca00ced50 0x7f3ca00cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.310+0000 7f3c9dffb700 1 -- 172.21.15.46:0/1243096408 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773405149 0 0) 0x7f3ca015a0b0 con 0x7f3ca00b2b40 2024-01-26T23:30:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.310+0000 7f3c9dffb700 1 -- 172.21.15.46:0/1243096408 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c88003570 con 0x7f3ca00b2b40 2024-01-26T23:30:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.310+0000 7f3c9dffb700 1 -- 172.21.15.46:0/1243096408 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2267846365 0 0) 0x7f3c88003570 con 0x7f3ca00b2b40 2024-01-26T23:30:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.310+0000 7f3c9dffb700 1 -- 172.21.15.46:0/1243096408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ca00ced50 msgr2=0x7f3ca00cf1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9dffb700 1 --2- 172.21.15.46:0/1243096408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ca00ced50 0x7f3ca00cf1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9dffb700 1 -- 172.21.15.46:0/1243096408 >> v2:172.21.15.106:3300/0 conn(0x7f3ca00c6520 msgr2=0x7f3ca00ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9dffb700 1 --2- 172.21.15.46:0/1243096408 >> v2:172.21.15.106:3300/0 conn(0x7f3ca00c6520 0x7f3ca00ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9dffb700 1 -- 172.21.15.46:0/1243096408 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca00ae4d0 con 0x7f3ca00b2b40 2024-01-26T23:30:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9dffb700 1 -- 172.21.15.46:0/1243096408 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f3c900015f0 con 0x7f3ca00b2b40 2024-01-26T23:30:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9dffb700 1 -- 172.21.15.46:0/1243096408 >> v1:172.21.15.46:6789/0 conn(0x7f3ca00b2b40 legacy=0x7f3ca00ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9dffb700 1 --2- 172.21.15.46:0/1243096408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003ae0 0x7f3c88003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9dffb700 1 --2- 172.21.15.46:0/1243096408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c88004810 0x7f3c88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9dffb700 1 --2- 172.21.15.46:0/1243096408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ca00ced50 0x7f3c880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9dffb700 1 -- 172.21.15.46:0/1243096408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca00af070 con 0x7f3c88003ae0 2024-01-26T23:30:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9dffb700 1 -- 172.21.15.46:0/1243096408 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8800ba90 con 0x7f3c88004810 2024-01-26T23:30:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9dffb700 1 -- 172.21.15.46:0/1243096408 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8800bc50 con 0x7f3ca00ced50 2024-01-26T23:30:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9effd700 1 --2- 172.21.15.46:0/1243096408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003ae0 0x7f3c88003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9e7fc700 1 --2- 172.21.15.46:0/1243096408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c88004810 0x7f3c88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9f7fe700 1 --2- 172.21.15.46:0/1243096408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ca00ced50 0x7f3c880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9effd700 1 -- 172.21.15.46:0/1243096408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ca00ced50 msgr2=0x7f3c880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9effd700 1 --2- 172.21.15.46:0/1243096408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ca00ced50 0x7f3c880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9effd700 1 -- 172.21.15.46:0/1243096408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c88004810 msgr2=0x7f3c88004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9effd700 1 --2- 172.21.15.46:0/1243096408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c88004810 0x7f3c88004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9effd700 1 -- 172.21.15.46:0/1243096408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c88003570 con 0x7f3c88003ae0 2024-01-26T23:30:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9effd700 1 --2- 172.21.15.46:0/1243096408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003ae0 0x7f3c88003ef0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000c2a0 tx=0x7f3c900078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41bfe11a31a8a98e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9dffb700 1 -- 172.21.15.46:0/1243096408 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9000cb70 con 0x7f3c88003ae0 2024-01-26T23:30:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9dffb700 1 -- 172.21.15.46:0/1243096408 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9000ccd0 con 0x7f3c88003ae0 2024-01-26T23:30:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.311+0000 7f3c9dffb700 1 -- 172.21.15.46:0/1243096408 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c90007320 con 0x7f3c88003ae0 2024-01-26T23:30:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.312+0000 7f3ca5a02700 1 -- 172.21.15.46:0/1243096408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003ae0 msgr2=0x7f3c88003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.312+0000 7f3ca5a02700 1 --2- 172.21.15.46:0/1243096408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003ae0 0x7f3c88003ef0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000c2a0 tx=0x7f3c900078a0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.312+0000 7f3c9f7fe700 1 -- 172.21.15.46:0/1243096408 reap_dead start 2024-01-26T23:30:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.312+0000 7f3ca5a02700 1 -- 172.21.15.46:0/1243096408 shutdown_connections 2024-01-26T23:30:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.312+0000 7f3ca5a02700 1 -- 172.21.15.46:0/1243096408 >> 172.21.15.46:0/1243096408 conn(0x7f3ca0099a30 msgr2=0x7f3ca006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:32.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.312+0000 7f3ca5a02700 1 -- 172.21.15.46:0/1243096408 shutdown_connections 2024-01-26T23:30:32.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.312+0000 7f3ca5a02700 1 -- 172.21.15.46:0/1243096408 wait complete. 2024-01-26T23:30:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.313+0000 7f3ca5a02700 1 Processor -- start 2024-01-26T23:30:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.313+0000 7f3ca5a02700 1 -- start start 2024-01-26T23:30:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.314+0000 7f3ca5a02700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003ae0 0x7f3ca01572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.314+0000 7f3ca5a02700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ca00cc800 0x7f3ca00ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.314+0000 7f3ca5a02700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca00c9ce0 0x7f3ca00ca150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.314+0000 7f3ca5a02700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca00c82b0 con 0x7f3c88003ae0 2024-01-26T23:30:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.314+0000 7f3ca5a02700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca0157810 con 0x7f3ca00c9ce0 2024-01-26T23:30:32.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.314+0000 7f3ca5a02700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca00b0540 con 0x7f3ca00cc800 2024-01-26T23:30:32.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.314+0000 7f3c9effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003ae0 0x7f3ca01572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:32.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.314+0000 7f3c9effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003ae0 0x7f3ca01572d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60780/0 (socket says 172.21.15.46:60780) 2024-01-26T23:30:32.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.314+0000 7f3c9effd700 1 -- 172.21.15.46:0/2856744374 learned_addr learned my addr 172.21.15.46:0/2856744374 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:32.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.314+0000 7f3c9f7fe700 1 --2- 172.21.15.46:0/2856744374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca00c9ce0 0x7f3ca00ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:32.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.314+0000 7f3c9e7fc700 1 --2- 172.21.15.46:0/2856744374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ca00cc800 0x7f3ca00ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:32.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.315+0000 7f3c9effd700 1 -- 172.21.15.46:0/2856744374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ca00cc800 msgr2=0x7f3ca00ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:32.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.315+0000 7f3c9effd700 1 --2- 172.21.15.46:0/2856744374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ca00cc800 0x7f3ca00ccc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.315+0000 7f3c9effd700 1 -- 172.21.15.46:0/2856744374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca00c9ce0 msgr2=0x7f3ca00ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:32.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.315+0000 7f3c9effd700 1 --2- 172.21.15.46:0/2856744374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca00c9ce0 0x7f3ca00ca150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.315+0000 7f3c9effd700 1 -- 172.21.15.46:0/2856744374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca00cd1b0 con 0x7f3c88003ae0 2024-01-26T23:30:32.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.315+0000 7f3c9effd700 1 --2- 172.21.15.46:0/2856744374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003ae0 0x7f3ca01572d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800ba60 tx=0x7f3c90012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:32.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.315+0000 7f3c86ffd700 1 -- 172.21.15.46:0/2856744374 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9000cb70 con 0x7f3c88003ae0 2024-01-26T23:30:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.315+0000 7f3c86ffd700 1 -- 172.21.15.46:0/2856744374 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c90002c70 con 0x7f3c88003ae0 2024-01-26T23:30:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.315+0000 7f3ca5a02700 1 -- 172.21.15.46:0/2856744374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca00cd460 con 0x7f3c88003ae0 2024-01-26T23:30:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.315+0000 7f3c86ffd700 1 -- 172.21.15.46:0/2856744374 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c90005c50 con 0x7f3c88003ae0 2024-01-26T23:30:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.315+0000 7f3ca5a02700 1 -- 172.21.15.46:0/2856744374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca00b01d0 con 0x7f3c88003ae0 2024-01-26T23:30:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.316+0000 7f3ca5a02700 1 -- 172.21.15.46:0/2856744374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca004c710 con 0x7f3c88003ae0 2024-01-26T23:30:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.317+0000 7f3c86ffd700 1 -- 172.21.15.46:0/2856744374 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3c90019030 con 0x7f3c88003ae0 2024-01-26T23:30:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.317+0000 7f3c86ffd700 1 --2- 172.21.15.46:0/2856744374 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3c8c066f30 0x7f3c8c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.317+0000 7f3c86ffd700 1 -- 172.21.15.46:0/2856744374 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f3c90085e10 con 0x7f3c88003ae0 2024-01-26T23:30:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.317+0000 7f3c9e7fc700 1 --2- 172.21.15.46:0/2856744374 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3c8c066f30 0x7f3c8c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.318+0000 7f3c9e7fc700 1 --2- 172.21.15.46:0/2856744374 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3c8c066f30 0x7f3c8c0693e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3ca00afde0 tx=0x7f3c940091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.321+0000 7f3c86ffd700 1 -- 172.21.15.46:0/2856744374 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3c90052d90 con 0x7f3c88003ae0 2024-01-26T23:30:32.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.533+0000 7f3ca5a02700 1 -- 172.21.15.46:0/2856744374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ca00682c0 con 0x7f3c88003ae0 2024-01-26T23:30:32.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.534+0000 7f3c86ffd700 1 -- 172.21.15.46:0/2856744374 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17518 (secure 0 0 0) 0x7f3c900563b0 con 0x7f3c88003ae0 2024-01-26T23:30:32.535 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:32.535 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:30:32.466712+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:32.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.537+0000 7f3ca5a02700 1 -- 172.21.15.46:0/2856744374 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3c8c066f30 msgr2=0x7f3c8c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:32.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.537+0000 7f3ca5a02700 1 --2- 172.21.15.46:0/2856744374 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3c8c066f30 0x7f3c8c0693e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3ca00afde0 tx=0x7f3c940091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.537+0000 7f3ca5a02700 1 -- 172.21.15.46:0/2856744374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003ae0 msgr2=0x7f3ca01572d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:32.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.537+0000 7f3ca5a02700 1 --2- 172.21.15.46:0/2856744374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003ae0 0x7f3ca01572d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800ba60 tx=0x7f3c90012000 comp rx=0 tx=0).stop 2024-01-26T23:30:32.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.537+0000 7f3ca5a02700 1 -- 172.21.15.46:0/2856744374 shutdown_connections 2024-01-26T23:30:32.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.537+0000 7f3ca5a02700 1 --2- 172.21.15.46:0/2856744374 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3c8c066f30 0x7f3c8c0693e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.537+0000 7f3ca5a02700 1 --2- 172.21.15.46:0/2856744374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003ae0 0x7f3ca01572d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.537+0000 7f3ca5a02700 1 --2- 172.21.15.46:0/2856744374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca00c9ce0 0x7f3ca00ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.537+0000 7f3ca5a02700 1 --2- 172.21.15.46:0/2856744374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ca00cc800 0x7f3ca00ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.537+0000 7f3ca5a02700 1 -- 172.21.15.46:0/2856744374 >> 172.21.15.46:0/2856744374 conn(0x7f3ca0099a30 msgr2=0x7f3ca0159040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:32.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.537+0000 7f3ca5a02700 1 -- 172.21.15.46:0/2856744374 shutdown_connections 2024-01-26T23:30:32.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.537+0000 7f3ca5a02700 1 -- 172.21.15.46:0/2856744374 wait complete. 2024-01-26T23:30:32.551 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-26T23:29:42.539759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-26T23:29:43.002669+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-01-26T23:29:46.646864+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-26T23:30:32.551 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:30:32.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.687+0000 7fd36fa22700 1 Processor -- start 2024-01-26T23:30:32.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.687+0000 7fd36fa22700 1 -- start start 2024-01-26T23:30:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.688+0000 7fd36fa22700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3680b3ae0 0x7fd3680b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.688+0000 7fd36fa22700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3680b2060 0x7fd36815e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.688+0000 7fd36fa22700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36815f180 con 0x7fd3680cec50 2024-01-26T23:30:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.688+0000 7fd36fa22700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3680cf0d0 con 0x7fd3680b3ae0 2024-01-26T23:30:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.688+0000 7fd36fa22700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36815e8e0 con 0x7fd3680b2060 2024-01-26T23:30:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.688+0000 7fd36d7be700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3680b3ae0 0x7fd3680b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.688+0000 7fd36d7be700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3680b3ae0 0x7fd3680b3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60792/0 (socket says 172.21.15.46:60792) 2024-01-26T23:30:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.689+0000 7fd36d7be700 1 -- 172.21.15.46:0/3420952050 learned_addr learned my addr 172.21.15.46:0/3420952050 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.689+0000 7fd36dfbf700 1 --2- 172.21.15.46:0/3420952050 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3680b2060 0x7fd36815e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.689+0000 7fd36d7be700 1 -- 172.21.15.46:0/3420952050 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3680b2060 msgr2=0x7fd36815e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.689+0000 7fd36d7be700 1 --2- 172.21.15.46:0/3420952050 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3680b2060 0x7fd36815e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.689+0000 7fd36d7be700 1 -- 172.21.15.46:0/3420952050 >> v1:172.21.15.106:6789/0 conn(0x7fd3680cec50 legacy=0x7fd3680b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.689+0000 7fd36d7be700 1 -- 172.21.15.46:0/3420952050 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36815ebc0 con 0x7fd3680b3ae0 2024-01-26T23:30:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.689+0000 7fd36d7be700 1 --2- 172.21.15.46:0/3420952050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3680b3ae0 0x7fd3680b3eb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd3640099b0 tx=0x7fd364009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=334d50386466d1c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.689+0000 7fd35bfff700 1 -- 172.21.15.46:0/3420952050 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36400b860 con 0x7fd3680b3ae0 2024-01-26T23:30:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.689+0000 7fd35bfff700 1 -- 172.21.15.46:0/3420952050 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36400b9c0 con 0x7fd3680b3ae0 2024-01-26T23:30:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.689+0000 7fd35bfff700 1 -- 172.21.15.46:0/3420952050 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36400bd80 con 0x7fd3680b3ae0 2024-01-26T23:30:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.690+0000 7fd36fa22700 1 -- 172.21.15.46:0/3420952050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3680b3ae0 msgr2=0x7fd3680b3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.690+0000 7fd36fa22700 1 --2- 172.21.15.46:0/3420952050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3680b3ae0 0x7fd3680b3eb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd3640099b0 tx=0x7fd364009250 comp rx=0 tx=0).stop 2024-01-26T23:30:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.690+0000 7fd36fa22700 1 -- 172.21.15.46:0/3420952050 shutdown_connections 2024-01-26T23:30:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.690+0000 7fd36fa22700 1 --2- 172.21.15.46:0/3420952050 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3680b2060 0x7fd36815e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.690+0000 7fd36fa22700 1 --2- 172.21.15.46:0/3420952050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3680b3ae0 0x7fd3680b3eb0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.690+0000 7fd36fa22700 1 -- 172.21.15.46:0/3420952050 >> 172.21.15.46:0/3420952050 conn(0x7fd368067930 msgr2=0x7fd368067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.690+0000 7fd36fa22700 1 -- 172.21.15.46:0/3420952050 shutdown_connections 2024-01-26T23:30:32.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.690+0000 7fd36fa22700 1 -- 172.21.15.46:0/3420952050 wait complete. 2024-01-26T23:30:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.691+0000 7fd36fa22700 1 Processor -- start 2024-01-26T23:30:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.691+0000 7fd36fa22700 1 -- start start 2024-01-26T23:30:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.692+0000 7fd36fa22700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3680b2060 0x7fd36816e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.692+0000 7fd36fa22700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3680b3ae0 0x7fd3681626a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.692+0000 7fd36fa22700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3680cec50 0x7fd3681541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.692+0000 7fd36fa22700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd368163b50 con 0x7fd3680b3ae0 2024-01-26T23:30:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.692+0000 7fd36fa22700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd368162ed0 con 0x7fd3680cec50 2024-01-26T23:30:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.692+0000 7fd36fa22700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3681631b0 con 0x7fd3680b2060 2024-01-26T23:30:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.692+0000 7fd36cfbd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3680b3ae0 0x7fd3681626a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.692+0000 7fd36cfbd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3680b3ae0 0x7fd3681626a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60808/0 (socket says 172.21.15.46:60808) 2024-01-26T23:30:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.692+0000 7fd36cfbd700 1 -- 172.21.15.46:0/854604429 learned_addr learned my addr 172.21.15.46:0/854604429 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.692+0000 7fd36d7be700 1 --2- 172.21.15.46:0/854604429 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3680b2060 0x7fd36816e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.692+0000 7fd36dfbf700 1 --2- 172.21.15.46:0/854604429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3680cec50 0x7fd3681541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.693+0000 7fd36cfbd700 1 -- 172.21.15.46:0/854604429 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3680b2060 msgr2=0x7fd36816e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.693+0000 7fd36cfbd700 1 --2- 172.21.15.46:0/854604429 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3680b2060 0x7fd36816e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.693+0000 7fd36cfbd700 1 -- 172.21.15.46:0/854604429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3680cec50 msgr2=0x7fd3681541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.693+0000 7fd36cfbd700 1 --2- 172.21.15.46:0/854604429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3680cec50 0x7fd3681541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.693+0000 7fd36cfbd700 1 -- 172.21.15.46:0/854604429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3640094f0 con 0x7fd3680b3ae0 2024-01-26T23:30:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.693+0000 7fd36dfbf700 1 --2- 172.21.15.46:0/854604429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3680cec50 0x7fd3681541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:30:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.693+0000 7fd36cfbd700 1 --2- 172.21.15.46:0/854604429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3680b3ae0 0x7fd3681626a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00c4e0 tx=0x7fd35c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.693+0000 7fd3597fa700 1 -- 172.21.15.46:0/854604429 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c011590 con 0x7fd3680b3ae0 2024-01-26T23:30:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.693+0000 7fd3597fa700 1 -- 172.21.15.46:0/854604429 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35c011710 con 0x7fd3680b3ae0 2024-01-26T23:30:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.693+0000 7fd36fa22700 1 -- 172.21.15.46:0/854604429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd368163490 con 0x7fd3680b3ae0 2024-01-26T23:30:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.693+0000 7fd3597fa700 1 -- 172.21.15.46:0/854604429 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c011ad0 con 0x7fd3680b3ae0 2024-01-26T23:30:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.693+0000 7fd36fa22700 1 -- 172.21.15.46:0/854604429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd368162be0 con 0x7fd3680b3ae0 2024-01-26T23:30:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.694+0000 7fd36fa22700 1 -- 172.21.15.46:0/854604429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd368163c90 con 0x7fd3680b3ae0 2024-01-26T23:30:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.694+0000 7fd3597fa700 1 -- 172.21.15.46:0/854604429 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd35c01b3b0 con 0x7fd3680b3ae0 2024-01-26T23:30:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.695+0000 7fd3597fa700 1 --2- 172.21.15.46:0/854604429 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd34c066f30 0x7fd34c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.695+0000 7fd3597fa700 1 -- 172.21.15.46:0/854604429 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6659+0+0 (secure 0 0 0) 0x7fd35c083ec0 con 0x7fd3680b3ae0 2024-01-26T23:30:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.695+0000 7fd36d7be700 1 --2- 172.21.15.46:0/854604429 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd34c066f30 0x7fd34c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:32.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.696+0000 7fd36d7be700 1 --2- 172.21.15.46:0/854604429 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd34c066f30 0x7fd34c0693e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd3640099b0 tx=0x7fd364017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:32.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.700+0000 7fd3597fa700 1 -- 172.21.15.46:0/854604429 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd368163c90 con 0x7fd3680b3ae0 2024-01-26T23:30:32.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.905+0000 7fd36fa22700 1 -- 172.21.15.46:0/854604429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd368163c90 con 0x7fd3680b3ae0 2024-01-26T23:30:32.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.905+0000 7fd3597fa700 1 -- 172.21.15.46:0/854604429 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd368163c90 con 0x7fd3680b3ae0 2024-01-26T23:30:32.906 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:30:32.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.907+0000 7fd36fa22700 1 -- 172.21.15.46:0/854604429 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd34c066f30 msgr2=0x7fd34c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:32.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.907+0000 7fd36fa22700 1 --2- 172.21.15.46:0/854604429 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd34c066f30 0x7fd34c0693e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd3640099b0 tx=0x7fd364017000 comp rx=0 tx=0).stop 2024-01-26T23:30:32.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.907+0000 7fd36fa22700 1 -- 172.21.15.46:0/854604429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3680b3ae0 msgr2=0x7fd3681626a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:32.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.907+0000 7fd36fa22700 1 --2- 172.21.15.46:0/854604429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3680b3ae0 0x7fd3681626a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00c4e0 tx=0x7fd35c00c8a0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.908+0000 7fd36fa22700 1 -- 172.21.15.46:0/854604429 shutdown_connections 2024-01-26T23:30:32.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.908+0000 7fd36fa22700 1 --2- 172.21.15.46:0/854604429 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd34c066f30 0x7fd34c0693e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.908+0000 7fd36fa22700 1 --2- 172.21.15.46:0/854604429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3680cec50 0x7fd3681541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.908+0000 7fd36fa22700 1 --2- 172.21.15.46:0/854604429 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3680b2060 0x7fd36816e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.908+0000 7fd36fa22700 1 --2- 172.21.15.46:0/854604429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3680b3ae0 0x7fd3681626a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:32.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.908+0000 7fd36fa22700 1 -- 172.21.15.46:0/854604429 >> 172.21.15.46:0/854604429 conn(0x7fd368067930 msgr2=0x7fd368068f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:32.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.908+0000 7fd36fa22700 1 -- 172.21.15.46:0/854604429 shutdown_connections 2024-01-26T23:30:32.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:32.908+0000 7fd36fa22700 1 -- 172.21.15.46:0/854604429 wait complete. 2024-01-26T23:30:32.922 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-26T23:30:33.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.055+0000 7f62afe90700 1 Processor -- start 2024-01-26T23:30:33.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.055+0000 7f62afe90700 1 -- start start 2024-01-26T23:30:33.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.056+0000 7f62afe90700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f62a8068940 0x7f62a8068d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.056+0000 7f62afe90700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a80c8c50 0x7f62a80c7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.056+0000 7f62afe90700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a80c87c0 con 0x7f62a80c76a0 2024-01-26T23:30:33.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.056+0000 7f62afe90700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a8069250 con 0x7f62a8068940 2024-01-26T23:30:33.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.056+0000 7f62afe90700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a80c7bb0 con 0x7f62a80c8c50 2024-01-26T23:30:33.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.056+0000 7f62ad42b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a80c8c50 0x7f62a80c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.056+0000 7f62adc2c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f62a8068940 0x7f62a8068d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.057+0000 7f62adc2c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f62a8068940 0x7f62a8068d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:59680/0 (socket says 172.21.15.46:59680) 2024-01-26T23:30:33.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.057+0000 7f62adc2c700 1 -- 172.21.15.46:0/1452152287 learned_addr learned my addr 172.21.15.46:0/1452152287 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:33.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.057+0000 7f62ad42b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a80c8c50 0x7f62a80c7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:35286/0 (socket says 172.21.15.46:35286) 2024-01-26T23:30:33.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.057+0000 7f62ae42d700 1 --1- 172.21.15.46:0/1452152287 >> v1:172.21.15.106:6789/0 conn(0x7f62a80c76a0 0x7f62a80af990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:33358/0 (socket says 172.21.15.46:33358) 2024-01-26T23:30:33.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.057+0000 7f62adc2c700 1 -- 172.21.15.46:0/1452152287 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a80c8c50 msgr2=0x7f62a80c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:33.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.057+0000 7f62adc2c700 1 --2- 172.21.15.46:0/1452152287 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a80c8c50 0x7f62a80c7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.057+0000 7f62adc2c700 1 -- 172.21.15.46:0/1452152287 >> v1:172.21.15.106:6789/0 conn(0x7f62a80c76a0 legacy=0x7f62a80af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:33.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.057+0000 7f62adc2c700 1 -- 172.21.15.46:0/1452152287 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a80c7e90 con 0x7f62a8068940 2024-01-26T23:30:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.057+0000 7f62acc2a700 1 -- 172.21.15.46:0/1452152287 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797643744 0 0) 0x7f62a80c87c0 con 0x7f62a80c76a0 2024-01-26T23:30:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.057+0000 7f62adc2c700 1 --2- 172.21.15.46:0/1452152287 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62a8068940 0x7f62a8068d10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f62a4009990 tx=0x7f62a4009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89227968c819e9ea server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.057+0000 7f62acc2a700 1 -- 172.21.15.46:0/1452152287 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a4009e80 con 0x7f62a8068940 2024-01-26T23:30:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.057+0000 7f62acc2a700 1 -- 172.21.15.46:0/1452152287 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a4005c10 con 0x7f62a8068940 2024-01-26T23:30:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.057+0000 7f62acc2a700 1 -- 172.21.15.46:0/1452152287 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a400b650 con 0x7f62a8068940 2024-01-26T23:30:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.058+0000 7f62afe90700 1 -- 172.21.15.46:0/1452152287 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62a8068940 msgr2=0x7f62a8068d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.058+0000 7f62afe90700 1 --2- 172.21.15.46:0/1452152287 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62a8068940 0x7f62a8068d10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f62a4009990 tx=0x7f62a4009220 comp rx=0 tx=0).stop 2024-01-26T23:30:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.058+0000 7f62afe90700 1 -- 172.21.15.46:0/1452152287 shutdown_connections 2024-01-26T23:30:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.058+0000 7f62afe90700 1 --2- 172.21.15.46:0/1452152287 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62a8068940 0x7f62a8068d10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.058+0000 7f62afe90700 1 --2- 172.21.15.46:0/1452152287 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a80c8c50 0x7f62a80c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.058+0000 7f62afe90700 1 -- 172.21.15.46:0/1452152287 >> 172.21.15.46:0/1452152287 conn(0x7f62a80ca100 msgr2=0x7f62a806e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.058+0000 7f62afe90700 1 -- 172.21.15.46:0/1452152287 shutdown_connections 2024-01-26T23:30:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.058+0000 7f62afe90700 1 -- 172.21.15.46:0/1452152287 wait complete. 2024-01-26T23:30:33.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.059+0000 7f62afe90700 1 Processor -- start 2024-01-26T23:30:33.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.059+0000 7f62afe90700 1 -- start start 2024-01-26T23:30:33.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.060+0000 7f62afe90700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a8068940 0x7f62a80ac2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.060+0000 7f62afe90700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62a80c76a0 0x7f62a815a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.060+0000 7f62afe90700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62a80c8c50 0x7f62a815a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.060+0000 7f62afe90700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a815ac20 con 0x7f62a80c8c50 2024-01-26T23:30:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.060+0000 7f62afe90700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a80c5bc0 con 0x7f62a80c76a0 2024-01-26T23:30:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.060+0000 7f62afe90700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a80c5ea0 con 0x7f62a8068940 2024-01-26T23:30:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.060+0000 7f62ae42d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62a80c8c50 0x7f62a815a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.060+0000 7f62ae42d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62a80c8c50 0x7f62a815a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60820/0 (socket says 172.21.15.46:60820) 2024-01-26T23:30:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.060+0000 7f62ae42d700 1 -- 172.21.15.46:0/2445631988 learned_addr learned my addr 172.21.15.46:0/2445631988 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.060+0000 7f62adc2c700 1 --2- 172.21.15.46:0/2445631988 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a8068940 0x7f62a80ac2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.060+0000 7f62ad42b700 1 --2- 172.21.15.46:0/2445631988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62a80c76a0 0x7f62a815a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.061+0000 7f62ae42d700 1 -- 172.21.15.46:0/2445631988 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a8068940 msgr2=0x7f62a80ac2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.061+0000 7f62ae42d700 1 --2- 172.21.15.46:0/2445631988 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a8068940 0x7f62a80ac2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.061+0000 7f62ae42d700 1 -- 172.21.15.46:0/2445631988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62a80c76a0 msgr2=0x7f62a815a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.061+0000 7f62ae42d700 1 --2- 172.21.15.46:0/2445631988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62a80c76a0 0x7f62a815a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.061+0000 7f62ae42d700 1 -- 172.21.15.46:0/2445631988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a4009480 con 0x7f62a80c8c50 2024-01-26T23:30:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.061+0000 7f62ae42d700 1 --2- 172.21.15.46:0/2445631988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62a80c8c50 0x7f62a815a6e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f62a000cd80 tx=0x7f62a0001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.061+0000 7f629dffb700 1 -- 172.21.15.46:0/2445631988 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a0005df0 con 0x7f62a80c8c50 2024-01-26T23:30:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.061+0000 7f629dffb700 1 -- 172.21.15.46:0/2445631988 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a000a900 con 0x7f62a80c8c50 2024-01-26T23:30:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.061+0000 7f62afe90700 1 -- 172.21.15.46:0/2445631988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a80c6180 con 0x7f62a80c8c50 2024-01-26T23:30:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.061+0000 7f629dffb700 1 -- 172.21.15.46:0/2445631988 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a000acc0 con 0x7f62a80c8c50 2024-01-26T23:30:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.061+0000 7f62afe90700 1 -- 172.21.15.46:0/2445631988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a80c6740 con 0x7f62a80c8c50 2024-01-26T23:30:33.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.062+0000 7f62afe90700 1 -- 172.21.15.46:0/2445631988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a80c1640 con 0x7f62a80c8c50 2024-01-26T23:30:33.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.063+0000 7f629dffb700 1 -- 172.21.15.46:0/2445631988 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f62a0007300 con 0x7f62a80c8c50 2024-01-26T23:30:33.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.063+0000 7f629dffb700 1 --2- 172.21.15.46:0/2445631988 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f628c066f30 0x7f628c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.063+0000 7f629dffb700 1 -- 172.21.15.46:0/2445631988 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f62a0084b10 con 0x7f62a80c8c50 2024-01-26T23:30:33.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.063+0000 7f62adc2c700 1 --2- 172.21.15.46:0/2445631988 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f628c066f30 0x7f628c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.064+0000 7f62adc2c700 1 --2- 172.21.15.46:0/2445631988 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f628c066f30 0x7f628c0693e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f62a4009260 tx=0x7f62a4016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:33.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.068+0000 7f629dffb700 1 -- 172.21.15.46:0/2445631988 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f62a0051a40 con 0x7f62a80c8c50 2024-01-26T23:30:33.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.272+0000 7f62afe90700 1 -- 172.21.15.46:0/2445631988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f62a804c710 con 0x7f62a80c8c50 2024-01-26T23:30:33.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.272+0000 7f629dffb700 1 -- 172.21.15.46:0/2445631988 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f62a0055060 con 0x7f62a80c8c50 2024-01-26T23:30:33.273 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:30:33.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.275+0000 7f62afe90700 1 -- 172.21.15.46:0/2445631988 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f628c066f30 msgr2=0x7f628c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:33.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.275+0000 7f62afe90700 1 --2- 172.21.15.46:0/2445631988 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f628c066f30 0x7f628c0693e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f62a4009260 tx=0x7f62a4016000 comp rx=0 tx=0).stop 2024-01-26T23:30:33.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.275+0000 7f62afe90700 1 -- 172.21.15.46:0/2445631988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62a80c8c50 msgr2=0x7f62a815a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:33.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.275+0000 7f62afe90700 1 --2- 172.21.15.46:0/2445631988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62a80c8c50 0x7f62a815a6e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f62a000cd80 tx=0x7f62a0001ae0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.275+0000 7f62afe90700 1 -- 172.21.15.46:0/2445631988 shutdown_connections 2024-01-26T23:30:33.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.275+0000 7f62afe90700 1 --2- 172.21.15.46:0/2445631988 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f628c066f30 0x7f628c0693e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.275+0000 7f62afe90700 1 --2- 172.21.15.46:0/2445631988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62a80c8c50 0x7f62a815a6e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.275+0000 7f62afe90700 1 --2- 172.21.15.46:0/2445631988 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62a8068940 0x7f62a80ac2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.275+0000 7f62afe90700 1 --2- 172.21.15.46:0/2445631988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62a80c76a0 0x7f62a815a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.275+0000 7f62afe90700 1 -- 172.21.15.46:0/2445631988 >> 172.21.15.46:0/2445631988 conn(0x7f62a80ca100 msgr2=0x7f62a80b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:33.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.275+0000 7f62afe90700 1 -- 172.21.15.46:0/2445631988 shutdown_connections 2024-01-26T23:30:33.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.275+0000 7f62afe90700 1 -- 172.21.15.46:0/2445631988 wait complete. 2024-01-26T23:30:33.289 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-26T23:30:33.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.423+0000 7fda66f1c700 1 Processor -- start 2024-01-26T23:30:33.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.423+0000 7fda66f1c700 1 -- start start 2024-01-26T23:30:33.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.424+0000 7fda66f1c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda60157490 0x7fda600cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.424+0000 7fda66f1c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fda60157860 0x7fda600cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.424+0000 7fda66f1c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda6006d6a0 0x7fda6006dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.424+0000 7fda66f1c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda600cdc60 con 0x7fda6006d6a0 2024-01-26T23:30:33.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.424+0000 7fda66f1c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda600cea80 con 0x7fda60157860 2024-01-26T23:30:33.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.424+0000 7fda66f1c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6006dff0 con 0x7fda60157490 2024-01-26T23:30:33.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.424+0000 7fda654b9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda6006d6a0 0x7fda6006dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.424+0000 7fda654b9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda6006d6a0 0x7fda6006dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60836/0 (socket says 172.21.15.46:60836) 2024-01-26T23:30:33.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.424+0000 7fda654b9700 1 -- 172.21.15.46:0/154892311 learned_addr learned my addr 172.21.15.46:0/154892311 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:33.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.424+0000 7fda64cb8700 1 --2- 172.21.15.46:0/154892311 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda60157490 0x7fda600cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.424+0000 7fda5ffff700 1 --2- 172.21.15.46:0/154892311 >> v2:172.21.15.106:3300/0 conn(0x7fda60157860 0x7fda600cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.424+0000 7fda654b9700 1 -- 172.21.15.46:0/154892311 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda60157490 msgr2=0x7fda600cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:33.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.424+0000 7fda654b9700 1 --2- 172.21.15.46:0/154892311 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda60157490 0x7fda600cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.424+0000 7fda654b9700 1 -- 172.21.15.46:0/154892311 >> v2:172.21.15.106:3300/0 conn(0x7fda60157860 msgr2=0x7fda600cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:33.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.424+0000 7fda654b9700 1 --2- 172.21.15.46:0/154892311 >> v2:172.21.15.106:3300/0 conn(0x7fda60157860 0x7fda600cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.424+0000 7fda654b9700 1 -- 172.21.15.46:0/154892311 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda600cde70 con 0x7fda6006d6a0 2024-01-26T23:30:33.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.424+0000 7fda64cb8700 1 --2- 172.21.15.46:0/154892311 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda60157490 0x7fda600cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:33.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.425+0000 7fda654b9700 1 --2- 172.21.15.46:0/154892311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6006d6a0 0x7fda6006dab0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fda6006cfe0 tx=0x7fda58009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c05383e654bf49d server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:33.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.425+0000 7fda5f7fe700 1 -- 172.21.15.46:0/154892311 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5800e070 con 0x7fda6006d6a0 2024-01-26T23:30:33.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.425+0000 7fda5f7fe700 1 -- 172.21.15.46:0/154892311 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5800ba50 con 0x7fda6006d6a0 2024-01-26T23:30:33.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.425+0000 7fda5f7fe700 1 -- 172.21.15.46:0/154892311 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda580092e0 con 0x7fda6006d6a0 2024-01-26T23:30:33.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.425+0000 7fda66f1c700 1 -- 172.21.15.46:0/154892311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6006d6a0 msgr2=0x7fda6006dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:33.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.425+0000 7fda66f1c700 1 --2- 172.21.15.46:0/154892311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6006d6a0 0x7fda6006dab0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fda6006cfe0 tx=0x7fda58009840 comp rx=0 tx=0).stop 2024-01-26T23:30:33.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.425+0000 7fda66f1c700 1 -- 172.21.15.46:0/154892311 shutdown_connections 2024-01-26T23:30:33.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.425+0000 7fda66f1c700 1 --2- 172.21.15.46:0/154892311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6006d6a0 0x7fda6006dab0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.425+0000 7fda66f1c700 1 --2- 172.21.15.46:0/154892311 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda60157490 0x7fda600cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.425+0000 7fda66f1c700 1 --2- 172.21.15.46:0/154892311 >> v2:172.21.15.106:3300/0 conn(0x7fda60157860 0x7fda600cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.425+0000 7fda66f1c700 1 -- 172.21.15.46:0/154892311 >> 172.21.15.46:0/154892311 conn(0x7fda60067930 msgr2=0x7fda60067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:33.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.425+0000 7fda66f1c700 1 -- 172.21.15.46:0/154892311 shutdown_connections 2024-01-26T23:30:33.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.425+0000 7fda66f1c700 1 -- 172.21.15.46:0/154892311 wait complete. 2024-01-26T23:30:33.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.427+0000 7fda66f1c700 1 Processor -- start 2024-01-26T23:30:33.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.427+0000 7fda66f1c700 1 -- start start 2024-01-26T23:30:33.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda66f1c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6006d6a0 0x7fda600b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda66f1c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda60157490 0x7fda60167a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda66f1c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda60157860 0x7fda6014c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda66f1c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6014c700 con 0x7fda6006d6a0 2024-01-26T23:30:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda66f1c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda600aeba0 con 0x7fda60157490 2024-01-26T23:30:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda66f1c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda600af160 con 0x7fda60157860 2024-01-26T23:30:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda64cb8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6006d6a0 0x7fda600b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda64cb8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6006d6a0 0x7fda600b0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60844/0 (socket says 172.21.15.46:60844) 2024-01-26T23:30:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda64cb8700 1 -- 172.21.15.46:0/1788749553 learned_addr learned my addr 172.21.15.46:0/1788749553 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda654b9700 1 --2- 172.21.15.46:0/1788749553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda60157860 0x7fda6014c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda5ffff700 1 --2- 172.21.15.46:0/1788749553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda60157490 0x7fda60167a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda64cb8700 1 -- 172.21.15.46:0/1788749553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda60157860 msgr2=0x7fda6014c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda64cb8700 1 --2- 172.21.15.46:0/1788749553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda60157860 0x7fda6014c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda64cb8700 1 -- 172.21.15.46:0/1788749553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda60157490 msgr2=0x7fda60167a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda64cb8700 1 --2- 172.21.15.46:0/1788749553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda60157490 0x7fda60167a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda64cb8700 1 -- 172.21.15.46:0/1788749553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda58009ae0 con 0x7fda6006d6a0 2024-01-26T23:30:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda64cb8700 1 --2- 172.21.15.46:0/1788749553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6006d6a0 0x7fda600b0bd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fda5400e900 tx=0x7fda54008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda5cff9700 1 -- 172.21.15.46:0/1788749553 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5400eea0 con 0x7fda6006d6a0 2024-01-26T23:30:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda5cff9700 1 -- 172.21.15.46:0/1788749553 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5400c5e0 con 0x7fda6006d6a0 2024-01-26T23:30:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda66f1c700 1 -- 172.21.15.46:0/1788749553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda60157d90 con 0x7fda6006d6a0 2024-01-26T23:30:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda5cff9700 1 -- 172.21.15.46:0/1788749553 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5400c980 con 0x7fda6006d6a0 2024-01-26T23:30:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.428+0000 7fda66f1c700 1 -- 172.21.15.46:0/1788749553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda600aedf0 con 0x7fda6006d6a0 2024-01-26T23:30:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.430+0000 7fda66f1c700 1 -- 172.21.15.46:0/1788749553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6015bf80 con 0x7fda6006d6a0 2024-01-26T23:30:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.430+0000 7fda5cff9700 1 -- 172.21.15.46:0/1788749553 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fda54013410 con 0x7fda6006d6a0 2024-01-26T23:30:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.430+0000 7fda5cff9700 1 --2- 172.21.15.46:0/1788749553 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fda44066f30 0x7fda440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.431+0000 7fda5cff9700 1 -- 172.21.15.46:0/1788749553 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6659+0+0 (secure 0 0 0) 0x7fda54084660 con 0x7fda6006d6a0 2024-01-26T23:30:33.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.431+0000 7fda5ffff700 1 --2- 172.21.15.46:0/1788749553 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fda44066f30 0x7fda440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.431+0000 7fda5ffff700 1 --2- 172.21.15.46:0/1788749553 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fda44066f30 0x7fda440693e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fda600b1110 tx=0x7fda4c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:33.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.435+0000 7fda5cff9700 1 -- 172.21.15.46:0/1788749553 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fda54051510 con 0x7fda6006d6a0 2024-01-26T23:30:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.647+0000 7fda66f1c700 1 -- 172.21.15.46:0/1788749553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fda6004c710 con 0x7fda6006d6a0 2024-01-26T23:30:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.648+0000 7fda5cff9700 1 -- 172.21.15.46:0/1788749553 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7fda54054b30 con 0x7fda6006d6a0 2024-01-26T23:30:33.649 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-26T23:30:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.650+0000 7fda66f1c700 1 -- 172.21.15.46:0/1788749553 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fda44066f30 msgr2=0x7fda440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.650+0000 7fda66f1c700 1 --2- 172.21.15.46:0/1788749553 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fda44066f30 0x7fda440693e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fda600b1110 tx=0x7fda4c008040 comp rx=0 tx=0).stop 2024-01-26T23:30:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.650+0000 7fda66f1c700 1 -- 172.21.15.46:0/1788749553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6006d6a0 msgr2=0x7fda600b0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.650+0000 7fda66f1c700 1 --2- 172.21.15.46:0/1788749553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6006d6a0 0x7fda600b0bd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fda5400e900 tx=0x7fda54008040 comp rx=0 tx=0).stop 2024-01-26T23:30:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.650+0000 7fda66f1c700 1 -- 172.21.15.46:0/1788749553 shutdown_connections 2024-01-26T23:30:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.650+0000 7fda66f1c700 1 --2- 172.21.15.46:0/1788749553 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fda44066f30 0x7fda440693e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.650+0000 7fda66f1c700 1 --2- 172.21.15.46:0/1788749553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6006d6a0 0x7fda600b0bd0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.650+0000 7fda66f1c700 1 --2- 172.21.15.46:0/1788749553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda60157860 0x7fda6014c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.650+0000 7fda66f1c700 1 --2- 172.21.15.46:0/1788749553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda60157490 0x7fda60167a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.650+0000 7fda66f1c700 1 -- 172.21.15.46:0/1788749553 >> 172.21.15.46:0/1788749553 conn(0x7fda60067930 msgr2=0x7fda600b2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.650+0000 7fda66f1c700 1 -- 172.21.15.46:0/1788749553 shutdown_connections 2024-01-26T23:30:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.650+0000 7fda66f1c700 1 -- 172.21.15.46:0/1788749553 wait complete. 2024-01-26T23:30:33.664 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-01-26T23:30:33.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.800+0000 7f09b2649700 1 Processor -- start 2024-01-26T23:30:33.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.800+0000 7f09b2649700 1 -- start start 2024-01-26T23:30:33.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.801+0000 7f09b2649700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f09ac068b00 0x7f09ac068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.801+0000 7f09b2649700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f09ac06a660 0x7f09ac159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.801+0000 7f09b2649700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ac0a9a80 con 0x7f09ac06add0 2024-01-26T23:30:33.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.801+0000 7f09b2649700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ac15a3e0 con 0x7f09ac06a660 2024-01-26T23:30:33.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.801+0000 7f09b2649700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac15a680 con 0x7f09ac068b00 2024-01-26T23:30:33.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.802+0000 7f09abfff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f09ac068b00 0x7f09ac068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.802+0000 7f09ab7fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f09ac06a660 0x7f09ac159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.802+0000 7f09abfff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f09ac068b00 0x7f09ac068ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:35356/0 (socket says 172.21.15.46:35356) 2024-01-26T23:30:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.802+0000 7f09b0be6700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f09ac06add0 0x7f09ac15d4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:33370/0 (socket says 172.21.15.46:33370) 2024-01-26T23:30:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.802+0000 7f09abfff700 1 -- 172.21.15.46:0/277790782 learned_addr learned my addr 172.21.15.46:0/277790782 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.802+0000 7f09aaffd700 1 -- 172.21.15.46:0/277790782 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658351787 0 0) 0x7f09ac0a9a80 con 0x7f09ac06add0 2024-01-26T23:30:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.802+0000 7f09aaffd700 1 -- 172.21.15.46:0/277790782 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0994003570 con 0x7f09ac06add0 2024-01-26T23:30:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.802+0000 7f09abfff700 1 -- 172.21.15.46:0/277790782 >> v2:172.21.15.106:3300/0 conn(0x7f09ac06a660 msgr2=0x7f09ac159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.802+0000 7f09abfff700 1 --2- 172.21.15.46:0/277790782 >> v2:172.21.15.106:3300/0 conn(0x7f09ac06a660 0x7f09ac159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.802+0000 7f09abfff700 1 -- 172.21.15.46:0/277790782 >> v1:172.21.15.106:6789/0 conn(0x7f09ac06add0 legacy=0x7f09ac15d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.802+0000 7f09abfff700 1 -- 172.21.15.46:0/277790782 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac06aad0 con 0x7f09ac068b00 2024-01-26T23:30:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.802+0000 7f09ab7fe700 1 --2- 172.21.15.46:0/277790782 >> v2:172.21.15.106:3300/0 conn(0x7f09ac06a660 0x7f09ac159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.802+0000 7f09abfff700 1 --2- 172.21.15.46:0/277790782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f09ac068b00 0x7f09ac068ed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f099c009990 tx=0x7f099c009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91061b4b8bbde6a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.803+0000 7f09aaffd700 1 -- 172.21.15.46:0/277790782 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c005c10 con 0x7f09ac068b00 2024-01-26T23:30:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.803+0000 7f09aaffd700 1 -- 172.21.15.46:0/277790782 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099c005d70 con 0x7f09ac068b00 2024-01-26T23:30:33.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.803+0000 7f09aaffd700 1 -- 172.21.15.46:0/277790782 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c006260 con 0x7f09ac068b00 2024-01-26T23:30:33.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.803+0000 7f09b2649700 1 -- 172.21.15.46:0/277790782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f09ac068b00 msgr2=0x7f09ac068ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:33.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.803+0000 7f09b2649700 1 --2- 172.21.15.46:0/277790782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f09ac068b00 0x7f09ac068ed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f099c009990 tx=0x7f099c009220 comp rx=0 tx=0).stop 2024-01-26T23:30:33.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.803+0000 7f09b2649700 1 -- 172.21.15.46:0/277790782 shutdown_connections 2024-01-26T23:30:33.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.803+0000 7f09b2649700 1 --2- 172.21.15.46:0/277790782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f09ac068b00 0x7f09ac068ed0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.803+0000 7f09b2649700 1 --2- 172.21.15.46:0/277790782 >> v2:172.21.15.106:3300/0 conn(0x7f09ac06a660 0x7f09ac159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.803+0000 7f09b2649700 1 -- 172.21.15.46:0/277790782 >> 172.21.15.46:0/277790782 conn(0x7f09ac0ca100 msgr2=0x7f09ac06e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:33.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.803+0000 7f09b2649700 1 -- 172.21.15.46:0/277790782 shutdown_connections 2024-01-26T23:30:33.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.803+0000 7f09b2649700 1 -- 172.21.15.46:0/277790782 wait complete. 2024-01-26T23:30:33.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.804+0000 7f09b2649700 1 Processor -- start 2024-01-26T23:30:33.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.805+0000 7f09b2649700 1 -- start start 2024-01-26T23:30:33.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.805+0000 7f09b2649700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09ac068b00 0x7f09ac0c3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.805+0000 7f09b2649700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f09ac06a660 0x7f09ac0abdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.805+0000 7f09b2649700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f09ac06add0 0x7f09ac0ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.805+0000 7f09b2649700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac0c3540 con 0x7f09ac068b00 2024-01-26T23:30:33.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.805+0000 7f09b2649700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac1452a0 con 0x7f09ac06a660 2024-01-26T23:30:33.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.805+0000 7f09b2649700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac157620 con 0x7f09ac06add0 2024-01-26T23:30:33.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.806+0000 7f09abfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09ac068b00 0x7f09ac0c3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.806+0000 7f09abfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09ac068b00 0x7f09ac0c3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60856/0 (socket says 172.21.15.46:60856) 2024-01-26T23:30:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.806+0000 7f09abfff700 1 -- 172.21.15.46:0/3393700451 learned_addr learned my addr 172.21.15.46:0/3393700451 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.806+0000 7f09ab7fe700 1 --2- 172.21.15.46:0/3393700451 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f09ac06a660 0x7f09ac0abdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.806+0000 7f09b0be6700 1 --2- 172.21.15.46:0/3393700451 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f09ac06add0 0x7f09ac0ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.806+0000 7f09abfff700 1 -- 172.21.15.46:0/3393700451 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f09ac06add0 msgr2=0x7f09ac0ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.806+0000 7f09abfff700 1 --2- 172.21.15.46:0/3393700451 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f09ac06add0 0x7f09ac0ac2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.806+0000 7f09abfff700 1 -- 172.21.15.46:0/3393700451 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f09ac06a660 msgr2=0x7f09ac0abdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.806+0000 7f09abfff700 1 --2- 172.21.15.46:0/3393700451 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f09ac06a660 0x7f09ac0abdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.806+0000 7f09abfff700 1 -- 172.21.15.46:0/3393700451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a0009420 con 0x7f09ac068b00 2024-01-26T23:30:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.806+0000 7f09abfff700 1 --2- 172.21.15.46:0/3393700451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09ac068b00 0x7f09ac0c3000 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f099c00a010 tx=0x7f099c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.806+0000 7f0993fff700 1 -- 172.21.15.46:0/3393700451 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c011640 con 0x7f09ac068b00 2024-01-26T23:30:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.806+0000 7f0993fff700 1 -- 172.21.15.46:0/3393700451 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099c020070 con 0x7f09ac068b00 2024-01-26T23:30:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.806+0000 7f09b2649700 1 -- 172.21.15.46:0/3393700451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f099c009480 con 0x7f09ac068b00 2024-01-26T23:30:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.806+0000 7f0993fff700 1 -- 172.21.15.46:0/3393700451 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c011a30 con 0x7f09ac068b00 2024-01-26T23:30:33.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.806+0000 7f09b2649700 1 -- 172.21.15.46:0/3393700451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ac0acbe0 con 0x7f09ac068b00 2024-01-26T23:30:33.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.807+0000 7f09b2649700 1 -- 172.21.15.46:0/3393700451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ac04c710 con 0x7f09ac068b00 2024-01-26T23:30:33.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.808+0000 7f0993fff700 1 -- 172.21.15.46:0/3393700451 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f099c022030 con 0x7f09ac068b00 2024-01-26T23:30:33.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.808+0000 7f0993fff700 1 --2- 172.21.15.46:0/3393700451 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0998066f30 0x7f09980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:33.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.808+0000 7f0993fff700 1 -- 172.21.15.46:0/3393700451 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f099c086130 con 0x7f09ac068b00 2024-01-26T23:30:33.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.809+0000 7f09ab7fe700 1 --2- 172.21.15.46:0/3393700451 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0998066f30 0x7f09980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:33.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.809+0000 7f09ab7fe700 1 --2- 172.21.15.46:0/3393700451 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0998066f30 0x7f09980693e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f09ac16ccd0 tx=0x7f09a00091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:33.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:33.813+0000 7f0993fff700 1 -- 172.21.15.46:0/3393700451 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f099c052fe0 con 0x7f09ac068b00 2024-01-26T23:30:34.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.017+0000 7f09b2649700 1 -- 172.21.15.46:0/3393700451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f09ac06edf0 con 0x7f09ac068b00 2024-01-26T23:30:34.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.017+0000 7f0993fff700 1 -- 172.21.15.46:0/3393700451 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f099c02a020 con 0x7f09ac068b00 2024-01-26T23:30:34.018 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:30:34.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.019+0000 7f09b2649700 1 -- 172.21.15.46:0/3393700451 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0998066f30 msgr2=0x7f09980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:34.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.019+0000 7f09b2649700 1 --2- 172.21.15.46:0/3393700451 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0998066f30 0x7f09980693e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f09ac16ccd0 tx=0x7f09a00091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:34.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.019+0000 7f09b2649700 1 -- 172.21.15.46:0/3393700451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09ac068b00 msgr2=0x7f09ac0c3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.019+0000 7f09b2649700 1 --2- 172.21.15.46:0/3393700451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09ac068b00 0x7f09ac0c3000 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f099c00a010 tx=0x7f099c016000 comp rx=0 tx=0).stop 2024-01-26T23:30:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.019+0000 7f09b2649700 1 -- 172.21.15.46:0/3393700451 shutdown_connections 2024-01-26T23:30:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.019+0000 7f09b2649700 1 --2- 172.21.15.46:0/3393700451 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0998066f30 0x7f09980693e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.019+0000 7f09b2649700 1 --2- 172.21.15.46:0/3393700451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09ac068b00 0x7f09ac0c3000 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.019+0000 7f09b2649700 1 --2- 172.21.15.46:0/3393700451 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f09ac06add0 0x7f09ac0ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.019+0000 7f09b2649700 1 --2- 172.21.15.46:0/3393700451 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f09ac06a660 0x7f09ac0abdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.019+0000 7f09b2649700 1 -- 172.21.15.46:0/3393700451 >> 172.21.15.46:0/3393700451 conn(0x7f09ac0ca100 msgr2=0x7f09ac0a9f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.019+0000 7f09b2649700 1 -- 172.21.15.46:0/3393700451 shutdown_connections 2024-01-26T23:30:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.019+0000 7f09b2649700 1 -- 172.21.15.46:0/3393700451 wait complete. 2024-01-26T23:30:34.059 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2024-01-26T23:30:34.059 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-26T23:30:34.059 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-01-26T23:30:34.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.192+0000 7f0a9d5b3700 1 Processor -- start 2024-01-26T23:30:34.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.192+0000 7f0a9d5b3700 1 -- start start 2024-01-26T23:30:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.193+0000 7f0a9d5b3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a980cbdf0 0x7f0a980ae610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.193+0000 7f0a9d5b3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a980ad070 0x7f0a980c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.193+0000 7f0a9d5b3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a980ae180 con 0x7f0a980cc1c0 2024-01-26T23:30:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.193+0000 7f0a9d5b3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a980aeb50 con 0x7f0a980ad070 2024-01-26T23:30:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.193+0000 7f0a9d5b3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a98156c60 con 0x7f0a980cbdf0 2024-01-26T23:30:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.193+0000 7f0a977fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a980ad070 0x7f0a980c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.193+0000 7f0a977fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a980ad070 0x7f0a980c98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60860/0 (socket says 172.21.15.46:60860) 2024-01-26T23:30:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.193+0000 7f0a977fe700 1 -- 172.21.15.46:0/1529911647 learned_addr learned my addr 172.21.15.46:0/1529911647 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.193+0000 7f0a96ffd700 1 --2- 172.21.15.46:0/1529911647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a980cbdf0 0x7f0a980ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.193+0000 7f0a977fe700 1 -- 172.21.15.46:0/1529911647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a980cbdf0 msgr2=0x7f0a980ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.193+0000 7f0a977fe700 1 --2- 172.21.15.46:0/1529911647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a980cbdf0 0x7f0a980ae610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.193+0000 7f0a977fe700 1 -- 172.21.15.46:0/1529911647 >> v1:172.21.15.106:6789/0 conn(0x7f0a980cc1c0 legacy=0x7f0a980b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.193+0000 7f0a977fe700 1 -- 172.21.15.46:0/1529911647 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a980ad570 con 0x7f0a980ad070 2024-01-26T23:30:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.194+0000 7f0a977fe700 1 --2- 172.21.15.46:0/1529911647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980ad070 0x7f0a980c98b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000d1a0 tx=0x7f0a9000ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbc54e6a7e2c9c34 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.194+0000 7f0a95ffb700 1 -- 172.21.15.46:0/1529911647 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a90010070 con 0x7f0a980ad070 2024-01-26T23:30:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.194+0000 7f0a95ffb700 1 -- 172.21.15.46:0/1529911647 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9000de80 con 0x7f0a980ad070 2024-01-26T23:30:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.194+0000 7f0a95ffb700 1 -- 172.21.15.46:0/1529911647 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9000b320 con 0x7f0a980ad070 2024-01-26T23:30:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.194+0000 7f0a9d5b3700 1 -- 172.21.15.46:0/1529911647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980ad070 msgr2=0x7f0a980c98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.194+0000 7f0a9d5b3700 1 --2- 172.21.15.46:0/1529911647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980ad070 0x7f0a980c98b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000d1a0 tx=0x7f0a9000ba50 comp rx=0 tx=0).stop 2024-01-26T23:30:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.194+0000 7f0a9d5b3700 1 -- 172.21.15.46:0/1529911647 shutdown_connections 2024-01-26T23:30:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.194+0000 7f0a9d5b3700 1 --2- 172.21.15.46:0/1529911647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980ad070 0x7f0a980c98b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:34.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.194+0000 7f0a9d5b3700 1 --2- 172.21.15.46:0/1529911647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a980cbdf0 0x7f0a980ae610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:34.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.194+0000 7f0a9d5b3700 1 -- 172.21.15.46:0/1529911647 >> 172.21.15.46:0/1529911647 conn(0x7f0a98068400 msgr2=0x7f0a980cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:34.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.194+0000 7f0a9d5b3700 1 -- 172.21.15.46:0/1529911647 shutdown_connections 2024-01-26T23:30:34.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.194+0000 7f0a9d5b3700 1 -- 172.21.15.46:0/1529911647 wait complete. 2024-01-26T23:30:34.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.196+0000 7f0a9d5b3700 1 Processor -- start 2024-01-26T23:30:34.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.196+0000 7f0a9d5b3700 1 -- start start 2024-01-26T23:30:34.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.196+0000 7f0a9d5b3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a980ad070 0x7f0a9816a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:34.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a9d5b3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980cbdf0 0x7f0a9815a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:34.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a9d5b3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a98141080 0x7f0a981414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:34.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a9d5b3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a98141a20 con 0x7f0a980cbdf0 2024-01-26T23:30:34.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a9d5b3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a980aad00 con 0x7f0a98141080 2024-01-26T23:30:34.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a9d5b3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a980a9e50 con 0x7f0a980ad070 2024-01-26T23:30:34.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a967fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980cbdf0 0x7f0a9815a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:34.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a967fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980cbdf0 0x7f0a9815a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60868/0 (socket says 172.21.15.46:60868) 2024-01-26T23:30:34.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a967fc700 1 -- 172.21.15.46:0/3417413912 learned_addr learned my addr 172.21.15.46:0/3417413912 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:34.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a977fe700 1 --2- 172.21.15.46:0/3417413912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a98141080 0x7f0a981414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:34.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a96ffd700 1 --2- 172.21.15.46:0/3417413912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a980ad070 0x7f0a9816a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:34.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a967fc700 1 -- 172.21.15.46:0/3417413912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a980ad070 msgr2=0x7f0a9816a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:34.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a967fc700 1 --2- 172.21.15.46:0/3417413912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a980ad070 0x7f0a9816a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:34.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a967fc700 1 -- 172.21.15.46:0/3417413912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a98141080 msgr2=0x7f0a981414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:34.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a967fc700 1 --2- 172.21.15.46:0/3417413912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a98141080 0x7f0a981414e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:34.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a967fc700 1 -- 172.21.15.46:0/3417413912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9000bcf0 con 0x7f0a980cbdf0 2024-01-26T23:30:34.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a967fc700 1 --2- 172.21.15.46:0/3417413912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980cbdf0 0x7f0a9815a550 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00b5e0 tx=0x7f0a8c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:34.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a7effd700 1 -- 172.21.15.46:0/3417413912 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c00f510 con 0x7f0a980cbdf0 2024-01-26T23:30:34.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a7effd700 1 -- 172.21.15.46:0/3417413912 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8c00f670 con 0x7f0a980cbdf0 2024-01-26T23:30:34.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a9d5b3700 1 -- 172.21.15.46:0/3417413912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a980aa0f0 con 0x7f0a980cbdf0 2024-01-26T23:30:34.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.197+0000 7f0a7effd700 1 -- 172.21.15.46:0/3417413912 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c00fa30 con 0x7f0a980cbdf0 2024-01-26T23:30:34.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.198+0000 7f0a9d5b3700 1 -- 172.21.15.46:0/3417413912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a980aa6b0 con 0x7f0a980cbdf0 2024-01-26T23:30:34.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.199+0000 7f0a9d5b3700 1 -- 172.21.15.46:0/3417413912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a980aae70 con 0x7f0a980cbdf0 2024-01-26T23:30:34.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.199+0000 7f0a7effd700 1 -- 172.21.15.46:0/3417413912 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0a8c01a400 con 0x7f0a980cbdf0 2024-01-26T23:30:34.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.199+0000 7f0a7effd700 1 --2- 172.21.15.46:0/3417413912 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0a84066f30 0x7f0a840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:34.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.200+0000 7f0a7effd700 1 -- 172.21.15.46:0/3417413912 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f0a8c084480 con 0x7f0a980cbdf0 2024-01-26T23:30:34.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.200+0000 7f0a96ffd700 1 --2- 172.21.15.46:0/3417413912 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0a84066f30 0x7f0a840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:34.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.200+0000 7f0a96ffd700 1 --2- 172.21.15.46:0/3417413912 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0a84066f30 0x7f0a840693e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0a88006fd0 tx=0x7f0a88008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:34.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.204+0000 7f0a7effd700 1 -- 172.21.15.46:0/3417413912 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a8c051310 con 0x7f0a980cbdf0 2024-01-26T23:30:34.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:34.495+0000 7f0a9d5b3700 1 -- 172.21.15.46:0/3417413912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f0a9804c710 con 0x7f0a980cbdf0 2024-01-26T23:30:35.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.484+0000 7f0a7effd700 1 -- 172.21.15.46:0/3417413912 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v63) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0a8c054930 con 0x7f0a980cbdf0 2024-01-26T23:30:35.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.487+0000 7f0a9d5b3700 1 -- 172.21.15.46:0/3417413912 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0a84066f30 msgr2=0x7f0a840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:35.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.487+0000 7f0a9d5b3700 1 --2- 172.21.15.46:0/3417413912 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0a84066f30 0x7f0a840693e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0a88006fd0 tx=0x7f0a88008040 comp rx=0 tx=0).stop 2024-01-26T23:30:35.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.487+0000 7f0a9d5b3700 1 -- 172.21.15.46:0/3417413912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980cbdf0 msgr2=0x7f0a9815a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:35.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.487+0000 7f0a9d5b3700 1 --2- 172.21.15.46:0/3417413912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980cbdf0 0x7f0a9815a550 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00b5e0 tx=0x7f0a8c009120 comp rx=0 tx=0).stop 2024-01-26T23:30:35.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.487+0000 7f0a9d5b3700 1 -- 172.21.15.46:0/3417413912 shutdown_connections 2024-01-26T23:30:35.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.487+0000 7f0a9d5b3700 1 --2- 172.21.15.46:0/3417413912 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0a84066f30 0x7f0a840693e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:35.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.487+0000 7f0a9d5b3700 1 --2- 172.21.15.46:0/3417413912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a98141080 0x7f0a981414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.487+0000 7f0a9d5b3700 1 --2- 172.21.15.46:0/3417413912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a980ad070 0x7f0a9816a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.487+0000 7f0a9d5b3700 1 --2- 172.21.15.46:0/3417413912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980cbdf0 0x7f0a9815a550 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.487+0000 7f0a9d5b3700 1 -- 172.21.15.46:0/3417413912 >> 172.21.15.46:0/3417413912 conn(0x7f0a98068400 msgr2=0x7f0a980b13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.487+0000 7f0a9d5b3700 1 -- 172.21.15.46:0/3417413912 shutdown_connections 2024-01-26T23:30:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.487+0000 7f0a9d5b3700 1 -- 172.21.15.46:0/3417413912 wait complete. 2024-01-26T23:30:35.503 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-26T23:30:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.641+0000 7efe173af700 1 Processor -- start 2024-01-26T23:30:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.641+0000 7efe173af700 1 -- start start 2024-01-26T23:30:35.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.642+0000 7efe173af700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efe10159930 0x7efe101647d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:35.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.642+0000 7efe173af700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1015e390 con 0x7efe100c9770 2024-01-26T23:30:35.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.643+0000 7efe173af700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe10068460 con 0x7efe100c93a0 2024-01-26T23:30:35.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.643+0000 7efe173af700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe10159d30 con 0x7efe10159930 2024-01-26T23:30:35.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.643+0000 7efe1594c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efe10159930 0x7efe101647d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:35.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.643+0000 7efe1494a700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efe100c9770 0x7efe10159220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59094/0 (socket says 172.21.15.46:59094) 2024-01-26T23:30:35.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.643+0000 7efe1494a700 1 -- 172.21.15.46:0/3758946068 learned_addr learned my addr 172.21.15.46:0/3758946068 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:35.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.643+0000 7efe1594c700 1 -- 172.21.15.46:0/3758946068 >> v1:172.21.15.106:6789/0 conn(0x7efe100c93a0 legacy=0x7efe10067d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:35.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.643+0000 7efe1594c700 1 -- 172.21.15.46:0/3758946068 >> v1:172.21.15.46:6789/0 conn(0x7efe100c9770 legacy=0x7efe10159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:35.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.643+0000 7efe1594c700 1 -- 172.21.15.46:0/3758946068 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1015d810 con 0x7efe10159930 2024-01-26T23:30:35.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.643+0000 7efe1594c700 1 --2- 172.21.15.46:0/3758946068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe10159930 0x7efe101647d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efe08009e60 tx=0x7efe08009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96dcd2681b82705a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:35.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.643+0000 7efe07fff700 1 -- 172.21.15.46:0/3758946068 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0800e070 con 0x7efe10159930 2024-01-26T23:30:35.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.643+0000 7efe07fff700 1 -- 172.21.15.46:0/3758946068 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0800bc30 con 0x7efe10159930 2024-01-26T23:30:35.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.643+0000 7efe07fff700 1 -- 172.21.15.46:0/3758946068 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe08012430 con 0x7efe10159930 2024-01-26T23:30:35.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.644+0000 7efe173af700 1 -- 172.21.15.46:0/3758946068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe10159930 msgr2=0x7efe101647d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:35.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.644+0000 7efe173af700 1 --2- 172.21.15.46:0/3758946068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe10159930 0x7efe101647d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efe08009e60 tx=0x7efe08009780 comp rx=0 tx=0).stop 2024-01-26T23:30:35.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.644+0000 7efe173af700 1 -- 172.21.15.46:0/3758946068 shutdown_connections 2024-01-26T23:30:35.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.644+0000 7efe173af700 1 --2- 172.21.15.46:0/3758946068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe10159930 0x7efe101647d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:35.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.644+0000 7efe173af700 1 -- 172.21.15.46:0/3758946068 >> 172.21.15.46:0/3758946068 conn(0x7efe100ca9a0 msgr2=0x7efe10068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:35.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.644+0000 7efe173af700 1 -- 172.21.15.46:0/3758946068 shutdown_connections 2024-01-26T23:30:35.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.644+0000 7efe173af700 1 -- 172.21.15.46:0/3758946068 wait complete. 2024-01-26T23:30:35.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.646+0000 7efe173af700 1 Processor -- start 2024-01-26T23:30:35.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.646+0000 7efe173af700 1 -- start start 2024-01-26T23:30:35.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.647+0000 7efe173af700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe100c93a0 0x7efe1015ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:35.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.647+0000 7efe173af700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe100c9770 0x7efe100b2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:35.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.647+0000 7efe173af700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe10159930 0x7efe1016d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:35.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.647+0000 7efe173af700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1015b4a0 con 0x7efe100c93a0 2024-01-26T23:30:35.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.647+0000 7efe173af700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe101561d0 con 0x7efe10159930 2024-01-26T23:30:35.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.647+0000 7efe173af700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1015d3d0 con 0x7efe100c9770 2024-01-26T23:30:35.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.647+0000 7efe1514b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe100c93a0 0x7efe1015ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:35.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.647+0000 7efe1514b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe100c93a0 0x7efe1015ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60880/0 (socket says 172.21.15.46:60880) 2024-01-26T23:30:35.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.647+0000 7efe1514b700 1 -- 172.21.15.46:0/3943830403 learned_addr learned my addr 172.21.15.46:0/3943830403 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:35.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.647+0000 7efe1494a700 1 --2- 172.21.15.46:0/3943830403 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe100c9770 0x7efe100b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:35.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.647+0000 7efe1594c700 1 --2- 172.21.15.46:0/3943830403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe10159930 0x7efe1016d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:35.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.648+0000 7efe1514b700 1 -- 172.21.15.46:0/3943830403 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe100c9770 msgr2=0x7efe100b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:35.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.648+0000 7efe1514b700 1 --2- 172.21.15.46:0/3943830403 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe100c9770 0x7efe100b2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:35.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.648+0000 7efe1514b700 1 -- 172.21.15.46:0/3943830403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe10159930 msgr2=0x7efe1016d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:35.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.648+0000 7efe1514b700 1 --2- 172.21.15.46:0/3943830403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe10159930 0x7efe1016d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:35.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.648+0000 7efe1514b700 1 -- 172.21.15.46:0/3943830403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe08009a20 con 0x7efe100c93a0 2024-01-26T23:30:35.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.648+0000 7efe1514b700 1 --2- 172.21.15.46:0/3943830403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe100c93a0 0x7efe1015ce90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efe0c00c510 tx=0x7efe0c00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:35.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.648+0000 7efe057fa700 1 -- 172.21.15.46:0/3943830403 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c011590 con 0x7efe100c93a0 2024-01-26T23:30:35.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.648+0000 7efe057fa700 1 -- 172.21.15.46:0/3943830403 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0c011710 con 0x7efe100c93a0 2024-01-26T23:30:35.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.648+0000 7efe173af700 1 -- 172.21.15.46:0/3943830403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe1015a890 con 0x7efe100c93a0 2024-01-26T23:30:35.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.648+0000 7efe057fa700 1 -- 172.21.15.46:0/3943830403 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c011ad0 con 0x7efe100c93a0 2024-01-26T23:30:35.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.648+0000 7efe173af700 1 -- 172.21.15.46:0/3943830403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe1015ae50 con 0x7efe100c93a0 2024-01-26T23:30:35.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.649+0000 7efe173af700 1 -- 172.21.15.46:0/3943830403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe1004c710 con 0x7efe100c93a0 2024-01-26T23:30:35.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.649+0000 7efe057fa700 1 -- 172.21.15.46:0/3943830403 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7efe0c01b3b0 con 0x7efe100c93a0 2024-01-26T23:30:35.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.650+0000 7efe057fa700 1 --2- 172.21.15.46:0/3943830403 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efdf4066f30 0x7efdf40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:35.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.650+0000 7efe057fa700 1 -- 172.21.15.46:0/3943830403 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7efe0c083c60 con 0x7efe100c93a0 2024-01-26T23:30:35.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.650+0000 7efe1494a700 1 --2- 172.21.15.46:0/3943830403 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efdf4066f30 0x7efdf40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:35.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.651+0000 7efe1494a700 1 --2- 172.21.15.46:0/3943830403 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efdf4066f30 0x7efdf40693e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efe100afd40 tx=0x7efe00009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:35.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.654+0000 7efe057fa700 1 -- 172.21.15.46:0/3943830403 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efe0c050b10 con 0x7efe100c93a0 2024-01-26T23:30:35.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.861+0000 7efe173af700 1 -- 172.21.15.46:0/3943830403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7efe1006f6e0 con 0x7efe100c93a0 2024-01-26T23:30:35.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.865+0000 7efe057fa700 1 -- 172.21.15.46:0/3943830403 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7efe0c017070 con 0x7efe100c93a0 2024-01-26T23:30:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.867+0000 7efe173af700 1 -- 172.21.15.46:0/3943830403 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efdf4066f30 msgr2=0x7efdf40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.867+0000 7efe173af700 1 --2- 172.21.15.46:0/3943830403 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efdf4066f30 0x7efdf40693e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efe100afd40 tx=0x7efe00009040 comp rx=0 tx=0).stop 2024-01-26T23:30:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.867+0000 7efe173af700 1 -- 172.21.15.46:0/3943830403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe100c93a0 msgr2=0x7efe1015ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.867+0000 7efe173af700 1 --2- 172.21.15.46:0/3943830403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe100c93a0 0x7efe1015ce90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efe0c00c510 tx=0x7efe0c00c820 comp rx=0 tx=0).stop 2024-01-26T23:30:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.867+0000 7efe173af700 1 -- 172.21.15.46:0/3943830403 shutdown_connections 2024-01-26T23:30:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.867+0000 7efe173af700 1 --2- 172.21.15.46:0/3943830403 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efdf4066f30 0x7efdf40693e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.867+0000 7efe173af700 1 --2- 172.21.15.46:0/3943830403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe100c93a0 0x7efe1015ce90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:35.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.867+0000 7efe173af700 1 --2- 172.21.15.46:0/3943830403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe10159930 0x7efe1016d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:35.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.867+0000 7efe173af700 1 --2- 172.21.15.46:0/3943830403 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe100c9770 0x7efe100b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:35.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.867+0000 7efe173af700 1 -- 172.21.15.46:0/3943830403 >> 172.21.15.46:0/3943830403 conn(0x7efe100ca9a0 msgr2=0x7efe10154140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:35.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.867+0000 7efe173af700 1 -- 172.21.15.46:0/3943830403 shutdown_connections 2024-01-26T23:30:35.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:35.867+0000 7efe173af700 1 -- 172.21.15.46:0/3943830403 wait complete. 2024-01-26T23:30:35.881 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-26T23:30:35.882 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:30:35.926 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:30:35.971 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-26T23:30:35.971 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-26T23:30:35.971 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-26T23:30:35.971 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-26T23:30:36.011 INFO:teuthology.orchestra.run:waiting for 900 2024-01-26T23:30:36.012 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:30:36.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:36.012 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-26T23:30:36.074 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi106.front.sepia.ceph.com... 2024-01-26T23:30:36.074 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-26T23:30:36.074 DEBUG:teuthology.orchestra.run.smithi106:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-26T23:30:36.115 INFO:teuthology.orchestra.run:waiting for 900 2024-01-26T23:30:36.116 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:30:36.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:36.116 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-26T23:30:36.187 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-26T23:30:36.187 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-26T23:30:36.187 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-01-26T23:30:36.235 INFO:teuthology.orchestra.run:waiting for 900 2024-01-26T23:30:36.236 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:30:36.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:36.236 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-26T23:30:36.298 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi106.front.sepia.ceph.com... 2024-01-26T23:30:36.298 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-26T23:30:36.298 DEBUG:teuthology.orchestra.run.smithi106:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-01-26T23:30:36.343 INFO:teuthology.orchestra.run:waiting for 900 2024-01-26T23:30:36.344 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:30:36.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:36.344 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-26T23:30:36.404 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-26T23:30:36.404 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:30:36.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.545+0000 7f10cb374700 1 Processor -- start 2024-01-26T23:30:36.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.546+0000 7f10cb374700 1 -- start start 2024-01-26T23:30:36.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10cb374700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10c415a7a0 0x7f10c4068940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:36.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10cb374700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c40a9b90 con 0x7f10c40b2160 2024-01-26T23:30:36.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10cb374700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c40aa210 con 0x7f10c40b2530 2024-01-26T23:30:36.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10cb374700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c415d580 con 0x7f10c415a7a0 2024-01-26T23:30:36.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10c9110700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f10c40b2160 0x7f10c40abe50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59104/0 (socket says 172.21.15.46:59104) 2024-01-26T23:30:36.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10c9110700 1 -- 172.21.15.46:0/2552742387 learned_addr learned my addr 172.21.15.46:0/2552742387 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:36.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10c9911700 1 --2- 172.21.15.46:0/2552742387 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10c415a7a0 0x7f10c4068940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:36.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10b7fff700 1 -- 172.21.15.46:0/2552742387 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366355514 0 0) 0x7f10c40a9b90 con 0x7f10c40b2160 2024-01-26T23:30:36.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10b7fff700 1 -- 172.21.15.46:0/2552742387 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b0003570 con 0x7f10c40b2160 2024-01-26T23:30:36.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10b7fff700 1 -- 172.21.15.46:0/2552742387 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701515894 0 0) 0x7f10c40aa210 con 0x7f10c40b2530 2024-01-26T23:30:36.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10b7fff700 1 -- 172.21.15.46:0/2552742387 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c40a9b90 con 0x7f10c40b2530 2024-01-26T23:30:36.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10b7fff700 1 -- 172.21.15.46:0/2552742387 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2930456183 0 0) 0x7f10b0003570 con 0x7f10c40b2160 2024-01-26T23:30:36.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10b7fff700 1 -- 172.21.15.46:0/2552742387 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10c415a7a0 msgr2=0x7f10c4068940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:36.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10b7fff700 1 --2- 172.21.15.46:0/2552742387 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10c415a7a0 0x7f10c4068940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:36.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10b7fff700 1 -- 172.21.15.46:0/2552742387 >> v1:172.21.15.106:6789/0 conn(0x7f10c40b2530 legacy=0x7f10c415da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:36.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10b7fff700 1 -- 172.21.15.46:0/2552742387 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c40aa460 con 0x7f10c40b2160 2024-01-26T23:30:36.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10b7fff700 1 -- 172.21.15.46:0/2552742387 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f10c0001470 con 0x7f10c40b2160 2024-01-26T23:30:36.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10b7fff700 1 -- 172.21.15.46:0/2552742387 >> v1:172.21.15.46:6789/0 conn(0x7f10c40b2160 legacy=0x7f10c40abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:36.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10b7fff700 1 --2- 172.21.15.46:0/2552742387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10b0003c20 0x7f10b0004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:36.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10b7fff700 1 --2- 172.21.15.46:0/2552742387 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10b0004810 0x7f10b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:36.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10b7fff700 1 --2- 172.21.15.46:0/2552742387 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10c415a7a0 0x7f10b0005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:36.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10b7fff700 1 -- 172.21.15.46:0/2552742387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c415d580 con 0x7f10b0003c20 2024-01-26T23:30:36.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10b7fff700 1 -- 172.21.15.46:0/2552742387 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b000bbf0 con 0x7f10b0004810 2024-01-26T23:30:36.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.547+0000 7f10b7fff700 1 -- 172.21.15.46:0/2552742387 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b000bdb0 con 0x7f10c415a7a0 2024-01-26T23:30:36.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.548+0000 7f10c9110700 1 --2- 172.21.15.46:0/2552742387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10b0003c20 0x7f10b0004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:36.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.548+0000 7f10c890f700 1 --2- 172.21.15.46:0/2552742387 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10b0004810 0x7f10b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:36.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.548+0000 7f10c9911700 1 --2- 172.21.15.46:0/2552742387 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10c415a7a0 0x7f10b0005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:36.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.548+0000 7f10c9110700 1 -- 172.21.15.46:0/2552742387 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10c415a7a0 msgr2=0x7f10b0005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:36.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.548+0000 7f10c9110700 1 --2- 172.21.15.46:0/2552742387 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10c415a7a0 0x7f10b0005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:36.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.548+0000 7f10c9110700 1 -- 172.21.15.46:0/2552742387 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10b0004810 msgr2=0x7f10b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:36.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.548+0000 7f10c9110700 1 --2- 172.21.15.46:0/2552742387 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10b0004810 0x7f10b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:36.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.548+0000 7f10c9110700 1 -- 172.21.15.46:0/2552742387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b0003570 con 0x7f10b0003c20 2024-01-26T23:30:36.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.548+0000 7f10c9110700 1 --2- 172.21.15.46:0/2552742387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10b0003c20 0x7f10b0004010 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f10c000c2a0 tx=0x7f10c4159d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd52a7de25ce8345 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:36.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.548+0000 7f10b7fff700 1 -- 172.21.15.46:0/2552742387 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c000cbd0 con 0x7f10b0003c20 2024-01-26T23:30:36.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.548+0000 7f10b7fff700 1 -- 172.21.15.46:0/2552742387 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c000cd30 con 0x7f10b0003c20 2024-01-26T23:30:36.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.548+0000 7f10b7fff700 1 -- 172.21.15.46:0/2552742387 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c000cea0 con 0x7f10b0003c20 2024-01-26T23:30:36.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.549+0000 7f10cb374700 1 -- 172.21.15.46:0/2552742387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10b0003c20 msgr2=0x7f10b0004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:36.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.549+0000 7f10cb374700 1 --2- 172.21.15.46:0/2552742387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10b0003c20 0x7f10b0004010 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f10c000c2a0 tx=0x7f10c4159d70 comp rx=0 tx=0).stop 2024-01-26T23:30:36.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.549+0000 7f10c9911700 1 -- 172.21.15.46:0/2552742387 reap_dead start 2024-01-26T23:30:36.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.549+0000 7f10cb374700 1 -- 172.21.15.46:0/2552742387 shutdown_connections 2024-01-26T23:30:36.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.549+0000 7f10cb374700 1 -- 172.21.15.46:0/2552742387 >> 172.21.15.46:0/2552742387 conn(0x7f10c40ccbc0 msgr2=0x7f10c40ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:36.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.549+0000 7f10cb374700 1 -- 172.21.15.46:0/2552742387 shutdown_connections 2024-01-26T23:30:36.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.549+0000 7f10cb374700 1 -- 172.21.15.46:0/2552742387 wait complete. 2024-01-26T23:30:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.550+0000 7f10cb374700 1 Processor -- start 2024-01-26T23:30:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.550+0000 7f10cb374700 1 -- start start 2024-01-26T23:30:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.551+0000 7f10cb374700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10b0003c20 0x7f10c414e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.551+0000 7f10cb374700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10c414d040 0x7f10c414b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.551+0000 7f10cb374700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10c40ad820 0x7f10c40adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.551+0000 7f10cb374700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c414c030 con 0x7f10c414d040 2024-01-26T23:30:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.551+0000 7f10cb374700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c414cbb0 con 0x7f10c40ad820 2024-01-26T23:30:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.551+0000 7f10cb374700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c414ec70 con 0x7f10b0003c20 2024-01-26T23:30:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.551+0000 7f10c890f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10c414d040 0x7f10c414b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.551+0000 7f10c890f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10c414d040 0x7f10c414b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60890/0 (socket says 172.21.15.46:60890) 2024-01-26T23:30:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.551+0000 7f10c890f700 1 -- 172.21.15.46:0/1118443739 learned_addr learned my addr 172.21.15.46:0/1118443739 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.551+0000 7f10c9110700 1 --2- 172.21.15.46:0/1118443739 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10b0003c20 0x7f10c414e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.551+0000 7f10c9911700 1 --2- 172.21.15.46:0/1118443739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10c40ad820 0x7f10c40adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.552+0000 7f10c890f700 1 -- 172.21.15.46:0/1118443739 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10b0003c20 msgr2=0x7f10c414e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.552+0000 7f10c890f700 1 --2- 172.21.15.46:0/1118443739 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10b0003c20 0x7f10c414e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.552+0000 7f10c890f700 1 -- 172.21.15.46:0/1118443739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10c40ad820 msgr2=0x7f10c40adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.552+0000 7f10c890f700 1 --2- 172.21.15.46:0/1118443739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10c40ad820 0x7f10c40adc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.552+0000 7f10c890f700 1 -- 172.21.15.46:0/1118443739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c414c280 con 0x7f10c414d040 2024-01-26T23:30:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.552+0000 7f10c890f700 1 --2- 172.21.15.46:0/1118443739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10c414d040 0x7f10c414b530 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f10b8002c30 tx=0x7f10b8007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.552+0000 7f10b57fa700 1 -- 172.21.15.46:0/1118443739 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b8006a30 con 0x7f10c414d040 2024-01-26T23:30:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.552+0000 7f10b57fa700 1 -- 172.21.15.46:0/1118443739 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b8006bd0 con 0x7f10c414d040 2024-01-26T23:30:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.552+0000 7f10cb374700 1 -- 172.21.15.46:0/1118443739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c414c560 con 0x7f10c414d040 2024-01-26T23:30:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.552+0000 7f10b57fa700 1 -- 172.21.15.46:0/1118443739 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b8007030 con 0x7f10c414d040 2024-01-26T23:30:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.552+0000 7f10cb374700 1 -- 172.21.15.46:0/1118443739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c414ba70 con 0x7f10c414d040 2024-01-26T23:30:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.553+0000 7f10cb374700 1 -- 172.21.15.46:0/1118443739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c414cd40 con 0x7f10c414d040 2024-01-26T23:30:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.554+0000 7f10b57fa700 1 -- 172.21.15.46:0/1118443739 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f10b80073a0 con 0x7f10c414d040 2024-01-26T23:30:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.554+0000 7f10b57fa700 1 --2- 172.21.15.46:0/1118443739 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f10a8066f30 0x7f10a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:36.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.554+0000 7f10b57fa700 1 -- 172.21.15.46:0/1118443739 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f10b8084c90 con 0x7f10c414d040 2024-01-26T23:30:36.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.554+0000 7f10c9110700 1 --2- 172.21.15.46:0/1118443739 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f10a8066f30 0x7f10a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:36.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.555+0000 7f10c9110700 1 --2- 172.21.15.46:0/1118443739 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f10a8066f30 0x7f10a80693e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f10c0001ed0 tx=0x7f10c0015000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:36.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.559+0000 7f10b57fa700 1 -- 172.21.15.46:0/1118443739 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f10b8051b40 con 0x7f10c414d040 2024-01-26T23:30:36.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.813+0000 7f10cb374700 1 -- 172.21.15.46:0/1118443739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10c404c710 con 0x7f10c414d040 2024-01-26T23:30:36.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.814+0000 7f10b57fa700 1 -- 172.21.15.46:0/1118443739 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4593 (secure 0 0 0) 0x7f10b8055160 con 0x7f10c414d040 2024-01-26T23:30:36.815 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:36.815 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3409861270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3409861270},{"type":"v1","addr":"172.21.15.46:6833","nonce":3409861270}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3319220815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3319220815},{"type":"v1","addr":"172.21.15.106:6836","nonce":3319220815}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4252,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/529198432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":529198432},{"type":"v1","addr":"172.21.15.106:6837","nonce":529198432}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:29:44.031963+0000","modified":"2024-01-26T23:30:03.652307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/688086235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":688086235},{"type":"v1","addr":"172.21.15.46:6835","nonce":688086235}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2024-01-26T23:30:36.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.816+0000 7f10cb374700 1 -- 172.21.15.46:0/1118443739 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f10a8066f30 msgr2=0x7f10a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:36.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.816+0000 7f10cb374700 1 --2- 172.21.15.46:0/1118443739 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f10a8066f30 0x7f10a80693e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f10c0001ed0 tx=0x7f10c0015000 comp rx=0 tx=0).stop 2024-01-26T23:30:36.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.816+0000 7f10cb374700 1 -- 172.21.15.46:0/1118443739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10c414d040 msgr2=0x7f10c414b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:36.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.816+0000 7f10cb374700 1 --2- 172.21.15.46:0/1118443739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10c414d040 0x7f10c414b530 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f10b8002c30 tx=0x7f10b8007d30 comp rx=0 tx=0).stop 2024-01-26T23:30:36.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.816+0000 7f10cb374700 1 -- 172.21.15.46:0/1118443739 shutdown_connections 2024-01-26T23:30:36.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.816+0000 7f10cb374700 1 --2- 172.21.15.46:0/1118443739 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f10a8066f30 0x7f10a80693e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:36.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.816+0000 7f10cb374700 1 --2- 172.21.15.46:0/1118443739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10c40ad820 0x7f10c40adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:36.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.816+0000 7f10cb374700 1 --2- 172.21.15.46:0/1118443739 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10b0003c20 0x7f10c414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:36.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.816+0000 7f10cb374700 1 --2- 172.21.15.46:0/1118443739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10c414d040 0x7f10c414b530 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:36.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.816+0000 7f10cb374700 1 -- 172.21.15.46:0/1118443739 >> 172.21.15.46:0/1118443739 conn(0x7f10c40ccbc0 msgr2=0x7f10c40b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:36.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.816+0000 7f10cb374700 1 -- 172.21.15.46:0/1118443739 shutdown_connections 2024-01-26T23:30:36.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.816+0000 7f10cb374700 1 -- 172.21.15.46:0/1118443739 wait complete. 2024-01-26T23:30:36.818 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 9 2024-01-26T23:30:36.831 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:36.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.962+0000 7f11389e4700 1 Processor -- start 2024-01-26T23:30:36.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.962+0000 7f11389e4700 1 -- start start 2024-01-26T23:30:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.963+0000 7f11389e4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1134159760 0x7f11340b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.963+0000 7f11389e4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1134068040 0x7f11340b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.963+0000 7f11389e4700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113415abc0 con 0x7f1134159b30 2024-01-26T23:30:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.963+0000 7f11389e4700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11340b2e90 con 0x7f1134068040 2024-01-26T23:30:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.963+0000 7f11389e4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1134159fb0 con 0x7f1134159760 2024-01-26T23:30:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.964+0000 7f1132d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1134068040 0x7f11340b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.964+0000 7f1131d9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1134159b30 0x7f1134067930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59120/0 (socket says 172.21.15.46:59120) 2024-01-26T23:30:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.964+0000 7f1131d9b700 1 -- 172.21.15.46:0/468962833 learned_addr learned my addr 172.21.15.46:0/468962833 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.964+0000 7f113259c700 1 --2- 172.21.15.46:0/468962833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1134159760 0x7f11340b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.964+0000 7f1132d9d700 1 -- 172.21.15.46:0/468962833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1134159760 msgr2=0x7f11340b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.964+0000 7f1132d9d700 1 --2- 172.21.15.46:0/468962833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1134159760 0x7f11340b0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.964+0000 7f1132d9d700 1 -- 172.21.15.46:0/468962833 >> v1:172.21.15.46:6789/0 conn(0x7f1134159b30 legacy=0x7f1134067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.964+0000 7f1132d9d700 1 -- 172.21.15.46:0/468962833 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113415a290 con 0x7f1134068040 2024-01-26T23:30:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.964+0000 7f113259c700 1 --2- 172.21.15.46:0/468962833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1134159760 0x7f11340b0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:30:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.964+0000 7f1132d9d700 1 --2- 172.21.15.46:0/468962833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134068040 0x7f11340b11b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f112c009e60 tx=0x7f112c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701de64e358037cb server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.964+0000 7f113159a700 1 -- 172.21.15.46:0/468962833 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c00e070 con 0x7f1134068040 2024-01-26T23:30:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.964+0000 7f113159a700 1 -- 172.21.15.46:0/468962833 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c00bc30 con 0x7f1134068040 2024-01-26T23:30:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.964+0000 7f113159a700 1 -- 172.21.15.46:0/468962833 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c012430 con 0x7f1134068040 2024-01-26T23:30:36.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.965+0000 7f11389e4700 1 -- 172.21.15.46:0/468962833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134068040 msgr2=0x7f11340b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:36.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.965+0000 7f11389e4700 1 --2- 172.21.15.46:0/468962833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134068040 0x7f11340b11b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f112c009e60 tx=0x7f112c009780 comp rx=0 tx=0).stop 2024-01-26T23:30:36.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.965+0000 7f11389e4700 1 -- 172.21.15.46:0/468962833 shutdown_connections 2024-01-26T23:30:36.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.965+0000 7f11389e4700 1 --2- 172.21.15.46:0/468962833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134068040 0x7f11340b11b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:36.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.965+0000 7f11389e4700 1 --2- 172.21.15.46:0/468962833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1134159760 0x7f11340b0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:36.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.965+0000 7f11389e4700 1 -- 172.21.15.46:0/468962833 >> 172.21.15.46:0/468962833 conn(0x7f113406b630 msgr2=0x7f11340aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:36.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.965+0000 7f11389e4700 1 -- 172.21.15.46:0/468962833 shutdown_connections 2024-01-26T23:30:36.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.965+0000 7f11389e4700 1 -- 172.21.15.46:0/468962833 wait complete. 2024-01-26T23:30:36.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.966+0000 7f11389e4700 1 Processor -- start 2024-01-26T23:30:36.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.966+0000 7f11389e4700 1 -- start start 2024-01-26T23:30:36.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f11389e4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134068040 0x7f113415d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f11389e4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1134159760 0x7f113415eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f11389e4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11340ab2b0 0x7f11341695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f11389e4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113415e810 con 0x7f1134068040 2024-01-26T23:30:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f11389e4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113415dc00 con 0x7f11340ab2b0 2024-01-26T23:30:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f11389e4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113415dee0 con 0x7f1134159760 2024-01-26T23:30:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f113259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134068040 0x7f113415d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f113259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134068040 0x7f113415d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60906/0 (socket says 172.21.15.46:60906) 2024-01-26T23:30:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f113259c700 1 -- 172.21.15.46:0/2840172288 learned_addr learned my addr 172.21.15.46:0/2840172288 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f1131d9b700 1 --2- 172.21.15.46:0/2840172288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1134159760 0x7f113415eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f1132d9d700 1 --2- 172.21.15.46:0/2840172288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11340ab2b0 0x7f11341695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f113259c700 1 -- 172.21.15.46:0/2840172288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1134159760 msgr2=0x7f113415eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f113259c700 1 --2- 172.21.15.46:0/2840172288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1134159760 0x7f113415eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f113259c700 1 -- 172.21.15.46:0/2840172288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11340ab2b0 msgr2=0x7f11341695c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f113259c700 1 --2- 172.21.15.46:0/2840172288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11340ab2b0 0x7f11341695c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f113259c700 1 -- 172.21.15.46:0/2840172288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c009a20 con 0x7f1134068040 2024-01-26T23:30:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f1131d9b700 1 --2- 172.21.15.46:0/2840172288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1134159760 0x7f113415eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:30:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.967+0000 7f1132d9d700 1 --2- 172.21.15.46:0/2840172288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11340ab2b0 0x7f11341695c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:30:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.968+0000 7f113259c700 1 --2- 172.21.15.46:0/2840172288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134068040 0x7f113415d620 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f112400c830 tx=0x7f1124008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.968+0000 7f112a7fc700 1 -- 172.21.15.46:0/2840172288 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1124011600 con 0x7f1134068040 2024-01-26T23:30:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.968+0000 7f112a7fc700 1 -- 172.21.15.46:0/2840172288 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1124011760 con 0x7f1134068040 2024-01-26T23:30:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.968+0000 7f11389e4700 1 -- 172.21.15.46:0/2840172288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113415e1c0 con 0x7f1134068040 2024-01-26T23:30:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.968+0000 7f112a7fc700 1 -- 172.21.15.46:0/2840172288 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1124011b20 con 0x7f1134068040 2024-01-26T23:30:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.968+0000 7f11389e4700 1 -- 172.21.15.46:0/2840172288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11340ab750 con 0x7f1134068040 2024-01-26T23:30:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.969+0000 7f11389e4700 1 -- 172.21.15.46:0/2840172288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113404c710 con 0x7f1134068040 2024-01-26T23:30:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.969+0000 7f112a7fc700 1 -- 172.21.15.46:0/2840172288 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f112400a3e0 con 0x7f1134068040 2024-01-26T23:30:36.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.970+0000 7f112a7fc700 1 --2- 172.21.15.46:0/2840172288 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1114066f30 0x7f11140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:36.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.970+0000 7f112a7fc700 1 -- 172.21.15.46:0/2840172288 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f1124084d80 con 0x7f1134068040 2024-01-26T23:30:36.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.970+0000 7f1131d9b700 1 --2- 172.21.15.46:0/2840172288 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1114066f30 0x7f11140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:36.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.971+0000 7f1131d9b700 1 --2- 172.21.15.46:0/2840172288 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1114066f30 0x7f11140693e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f113415fc60 tx=0x7f111c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:36.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:36.975+0000 7f112a7fc700 1 -- 172.21.15.46:0/2840172288 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1124051c30 con 0x7f1134068040 2024-01-26T23:30:37.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.180+0000 7f11389e4700 1 -- 172.21.15.46:0/2840172288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f113415a570 con 0x7f1134068040 2024-01-26T23:30:37.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.181+0000 7f112a7fc700 1 -- 172.21.15.46:0/2840172288 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17518 (secure 0 0 0) 0x7f1124015090 con 0x7f1134068040 2024-01-26T23:30:37.182 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:37.182 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":26,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:30:33.471772+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:37.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.183+0000 7f11389e4700 1 -- 172.21.15.46:0/2840172288 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1114066f30 msgr2=0x7f11140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:37.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.183+0000 7f11389e4700 1 --2- 172.21.15.46:0/2840172288 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1114066f30 0x7f11140693e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f113415fc60 tx=0x7f111c009040 comp rx=0 tx=0).stop 2024-01-26T23:30:37.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.183+0000 7f11389e4700 1 -- 172.21.15.46:0/2840172288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134068040 msgr2=0x7f113415d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:37.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.183+0000 7f11389e4700 1 --2- 172.21.15.46:0/2840172288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134068040 0x7f113415d620 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f112400c830 tx=0x7f1124008040 comp rx=0 tx=0).stop 2024-01-26T23:30:37.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.183+0000 7f11389e4700 1 -- 172.21.15.46:0/2840172288 shutdown_connections 2024-01-26T23:30:37.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.183+0000 7f11389e4700 1 --2- 172.21.15.46:0/2840172288 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1114066f30 0x7f11140693e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.183+0000 7f11389e4700 1 --2- 172.21.15.46:0/2840172288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134068040 0x7f113415d620 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.183+0000 7f11389e4700 1 --2- 172.21.15.46:0/2840172288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11340ab2b0 0x7f11341695c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.183+0000 7f11389e4700 1 --2- 172.21.15.46:0/2840172288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1134159760 0x7f113415eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.183+0000 7f11389e4700 1 -- 172.21.15.46:0/2840172288 >> 172.21.15.46:0/2840172288 conn(0x7f113406b630 msgr2=0x7f1134157970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:37.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.184+0000 7f11389e4700 1 -- 172.21.15.46:0/2840172288 shutdown_connections 2024-01-26T23:30:37.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.184+0000 7f11389e4700 1 -- 172.21.15.46:0/2840172288 wait complete. 2024-01-26T23:30:37.199 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-26T23:29:42.539759+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-26T23:29:43.002669+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-01-26T23:29:46.646864+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-26T23:30:37.199 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:30:37.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.334+0000 7f4a6b5e7700 1 Processor -- start 2024-01-26T23:30:37.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.334+0000 7f4a6b5e7700 1 -- start start 2024-01-26T23:30:37.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.335+0000 7f4a6b5e7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a640cd2c0 0x7f4a640cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:37.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.335+0000 7f4a6b5e7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4a64159e70 0x7f4a64067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:37.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.335+0000 7f4a6b5e7700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a640cce30 con 0x7f4a64159aa0 2024-01-26T23:30:37.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.335+0000 7f4a6b5e7700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a640c9970 con 0x7f4a64159e70 2024-01-26T23:30:37.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.335+0000 7f4a6b5e7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a640c9c50 con 0x7f4a640cd2c0 2024-01-26T23:30:37.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.335+0000 7f4a69b84700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4a64159e70 0x7f4a64067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:37.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.335+0000 7f4a69b84700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4a64159e70 0x7f4a64067930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60910/0 (socket says 172.21.15.46:60910) 2024-01-26T23:30:37.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.335+0000 7f4a69b84700 1 -- 172.21.15.46:0/4238406500 learned_addr learned my addr 172.21.15.46:0/4238406500 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:37.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.335+0000 7f4a68b82700 1 --2- 172.21.15.46:0/4238406500 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a640cd2c0 0x7f4a640cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:37.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.336+0000 7f4a69b84700 1 -- 172.21.15.46:0/4238406500 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a640cd2c0 msgr2=0x7f4a640cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:37.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.336+0000 7f4a69b84700 1 --2- 172.21.15.46:0/4238406500 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a640cd2c0 0x7f4a640cdff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.336+0000 7f4a69b84700 1 -- 172.21.15.46:0/4238406500 >> v1:172.21.15.106:6789/0 conn(0x7f4a64159aa0 legacy=0x7f4a640c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:37.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.336+0000 7f4a69b84700 1 -- 172.21.15.46:0/4238406500 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a640c9620 con 0x7f4a64159e70 2024-01-26T23:30:37.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.336+0000 7f4a69b84700 1 --2- 172.21.15.46:0/4238406500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a64159e70 0x7f4a64067930 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009f20 tx=0x7f4a5c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d8de5b60791ded server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:37.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.336+0000 7f4a5bfff700 1 -- 172.21.15.46:0/4238406500 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c00e070 con 0x7f4a64159e70 2024-01-26T23:30:37.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.336+0000 7f4a5bfff700 1 -- 172.21.15.46:0/4238406500 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5c00ba80 con 0x7f4a64159e70 2024-01-26T23:30:37.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.336+0000 7f4a5bfff700 1 -- 172.21.15.46:0/4238406500 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c0092e0 con 0x7f4a64159e70 2024-01-26T23:30:37.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.337+0000 7f4a6b5e7700 1 -- 172.21.15.46:0/4238406500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a64159e70 msgr2=0x7f4a64067930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:37.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.337+0000 7f4a6b5e7700 1 --2- 172.21.15.46:0/4238406500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a64159e70 0x7f4a64067930 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009f20 tx=0x7f4a5c009840 comp rx=0 tx=0).stop 2024-01-26T23:30:37.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.337+0000 7f4a6b5e7700 1 -- 172.21.15.46:0/4238406500 shutdown_connections 2024-01-26T23:30:37.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.337+0000 7f4a6b5e7700 1 --2- 172.21.15.46:0/4238406500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a64159e70 0x7f4a64067930 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.337+0000 7f4a6b5e7700 1 --2- 172.21.15.46:0/4238406500 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a640cd2c0 0x7f4a640cdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.337+0000 7f4a6b5e7700 1 -- 172.21.15.46:0/4238406500 >> 172.21.15.46:0/4238406500 conn(0x7f4a64069220 msgr2=0x7f4a640cd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:37.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.337+0000 7f4a6b5e7700 1 -- 172.21.15.46:0/4238406500 shutdown_connections 2024-01-26T23:30:37.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.337+0000 7f4a6b5e7700 1 -- 172.21.15.46:0/4238406500 wait complete. 2024-01-26T23:30:37.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.338+0000 7f4a6b5e7700 1 Processor -- start 2024-01-26T23:30:37.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.338+0000 7f4a6b5e7700 1 -- start start 2024-01-26T23:30:37.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.339+0000 7f4a6b5e7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a640cd2c0 0x7f4a641580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:37.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.339+0000 7f4a6b5e7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a64159aa0 0x7f4a640b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:37.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.339+0000 7f4a6b5e7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a64159e70 0x7f4a6406c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:37.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.339+0000 7f4a6b5e7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a640ae1e0 con 0x7f4a64159e70 2024-01-26T23:30:37.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.339+0000 7f4a6b5e7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a64158620 con 0x7f4a64159aa0 2024-01-26T23:30:37.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.339+0000 7f4a6b5e7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a640c1130 con 0x7f4a640cd2c0 2024-01-26T23:30:37.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.339+0000 7f4a69b84700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a64159e70 0x7f4a6406c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:37.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.339+0000 7f4a69b84700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a64159e70 0x7f4a6406c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60916/0 (socket says 172.21.15.46:60916) 2024-01-26T23:30:37.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.339+0000 7f4a69b84700 1 -- 172.21.15.46:0/4051540929 learned_addr learned my addr 172.21.15.46:0/4051540929 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:37.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.339+0000 7f4a69383700 1 --2- 172.21.15.46:0/4051540929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a640cd2c0 0x7f4a641580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:37.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.339+0000 7f4a68b82700 1 --2- 172.21.15.46:0/4051540929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a64159aa0 0x7f4a640b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:37.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.339+0000 7f4a69b84700 1 -- 172.21.15.46:0/4051540929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a640cd2c0 msgr2=0x7f4a641580e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:37.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.339+0000 7f4a69b84700 1 --2- 172.21.15.46:0/4051540929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a640cd2c0 0x7f4a641580e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.339+0000 7f4a69b84700 1 -- 172.21.15.46:0/4051540929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a64159aa0 msgr2=0x7f4a640b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:37.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.339+0000 7f4a69b84700 1 --2- 172.21.15.46:0/4051540929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a64159aa0 0x7f4a640b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.339+0000 7f4a69b84700 1 -- 172.21.15.46:0/4051540929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c009ae0 con 0x7f4a64159e70 2024-01-26T23:30:37.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.340+0000 7f4a69b84700 1 --2- 172.21.15.46:0/4051540929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a64159e70 0x7f4a6406c2c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009990 tx=0x7f4a5c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:37.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.340+0000 7f4a597fa700 1 -- 172.21.15.46:0/4051540929 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c0092e0 con 0x7f4a64159e70 2024-01-26T23:30:37.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.340+0000 7f4a597fa700 1 -- 172.21.15.46:0/4051540929 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5c007d40 con 0x7f4a64159e70 2024-01-26T23:30:37.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.340+0000 7f4a6b5e7700 1 -- 172.21.15.46:0/4051540929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a640b1a70 con 0x7f4a64159e70 2024-01-26T23:30:37.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.340+0000 7f4a597fa700 1 -- 172.21.15.46:0/4051540929 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c01c640 con 0x7f4a64159e70 2024-01-26T23:30:37.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.340+0000 7f4a6b5e7700 1 -- 172.21.15.46:0/4051540929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a640b1140 con 0x7f4a64159e70 2024-01-26T23:30:37.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.341+0000 7f4a6b5e7700 1 -- 172.21.15.46:0/4051540929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a640b1c00 con 0x7f4a64159e70 2024-01-26T23:30:37.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.341+0000 7f4a597fa700 1 -- 172.21.15.46:0/4051540929 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4a5c003050 con 0x7f4a64159e70 2024-01-26T23:30:37.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.342+0000 7f4a597fa700 1 --2- 172.21.15.46:0/4051540929 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4a48066f30 0x7f4a480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:37.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.342+0000 7f4a597fa700 1 -- 172.21.15.46:0/4051540929 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f4a5c0862b0 con 0x7f4a64159e70 2024-01-26T23:30:37.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.342+0000 7f4a69383700 1 --2- 172.21.15.46:0/4051540929 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4a48066f30 0x7f4a480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:37.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.343+0000 7f4a69383700 1 --2- 172.21.15.46:0/4051540929 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4a48066f30 0x7f4a480693e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4a640cbda0 tx=0x7f4a60009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:37.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.346+0000 7f4a597fa700 1 -- 172.21.15.46:0/4051540929 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4a5c053160 con 0x7f4a64159e70 2024-01-26T23:30:37.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.551+0000 7f4a6b5e7700 1 -- 172.21.15.46:0/4051540929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4a6404c710 con 0x7f4a64159e70 2024-01-26T23:30:37.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.552+0000 7f4a597fa700 1 -- 172.21.15.46:0/4051540929 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4a5c056780 con 0x7f4a64159e70 2024-01-26T23:30:37.552 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:30:37.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.554+0000 7f4a6b5e7700 1 -- 172.21.15.46:0/4051540929 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4a48066f30 msgr2=0x7f4a480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:37.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.554+0000 7f4a6b5e7700 1 --2- 172.21.15.46:0/4051540929 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4a48066f30 0x7f4a480693e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4a640cbda0 tx=0x7f4a60009040 comp rx=0 tx=0).stop 2024-01-26T23:30:37.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.554+0000 7f4a6b5e7700 1 -- 172.21.15.46:0/4051540929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a64159e70 msgr2=0x7f4a6406c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:37.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.554+0000 7f4a6b5e7700 1 --2- 172.21.15.46:0/4051540929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a64159e70 0x7f4a6406c2c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009990 tx=0x7f4a5c018000 comp rx=0 tx=0).stop 2024-01-26T23:30:37.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.554+0000 7f4a6b5e7700 1 -- 172.21.15.46:0/4051540929 shutdown_connections 2024-01-26T23:30:37.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.554+0000 7f4a6b5e7700 1 --2- 172.21.15.46:0/4051540929 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4a48066f30 0x7f4a480693e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.554+0000 7f4a6b5e7700 1 --2- 172.21.15.46:0/4051540929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a64159e70 0x7f4a6406c2c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.554+0000 7f4a6b5e7700 1 --2- 172.21.15.46:0/4051540929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a640cd2c0 0x7f4a641580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.554+0000 7f4a6b5e7700 1 --2- 172.21.15.46:0/4051540929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a64159aa0 0x7f4a640b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.554+0000 7f4a6b5e7700 1 -- 172.21.15.46:0/4051540929 >> 172.21.15.46:0/4051540929 conn(0x7f4a64069220 msgr2=0x7f4a641588a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:37.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.554+0000 7f4a6b5e7700 1 -- 172.21.15.46:0/4051540929 shutdown_connections 2024-01-26T23:30:37.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.554+0000 7f4a6b5e7700 1 -- 172.21.15.46:0/4051540929 wait complete. 2024-01-26T23:30:37.568 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-26T23:30:37.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.702+0000 7f7903d95700 1 Processor -- start 2024-01-26T23:30:37.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.702+0000 7f7903d95700 1 -- start start 2024-01-26T23:30:37.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.703+0000 7f7903d95700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78fc15cd40 0x7f78fc0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:37.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.703+0000 7f7903d95700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f78fc15d110 0x7f78fc0cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:37.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.703+0000 7f7903d95700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc06caa0 con 0x7f78fc067930 2024-01-26T23:30:37.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.703+0000 7f7903d95700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78fc0c7ac0 con 0x7f78fc15d110 2024-01-26T23:30:37.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.703+0000 7f7903d95700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0ca8b0 con 0x7f78fc15cd40 2024-01-26T23:30:37.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.704+0000 7f7901b31700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78fc15cd40 0x7f78fc0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:37.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.704+0000 7f7901330700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f78fc15d110 0x7f78fc0cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:37.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.704+0000 7f7901330700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f78fc15d110 0x7f78fc0cb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:59938/0 (socket says 172.21.15.46:59938) 2024-01-26T23:30:37.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.704+0000 7f7901330700 1 -- 172.21.15.46:0/896103385 learned_addr learned my addr 172.21.15.46:0/896103385 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:37.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.704+0000 7f7902332700 1 --1- 172.21.15.46:0/896103385 >> v1:172.21.15.106:6789/0 conn(0x7f78fc067930 0x7f78fc15d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:33416/0 (socket says 172.21.15.46:33416) 2024-01-26T23:30:37.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.704+0000 7f7901330700 1 -- 172.21.15.46:0/896103385 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78fc15cd40 msgr2=0x7f78fc0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:37.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.704+0000 7f7901330700 1 --2- 172.21.15.46:0/896103385 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78fc15cd40 0x7f78fc0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.704+0000 7f7901330700 1 -- 172.21.15.46:0/896103385 >> v1:172.21.15.106:6789/0 conn(0x7f78fc067930 legacy=0x7f78fc15d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:37.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.704+0000 7f7901330700 1 -- 172.21.15.46:0/896103385 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc06c1b0 con 0x7f78fc15d110 2024-01-26T23:30:37.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.704+0000 7f7901330700 1 --2- 172.21.15.46:0/896103385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78fc15d110 0x7f78fc0cb7a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f78f80099b0 tx=0x7f78f8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ef0ef7dfc347ee0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.704+0000 7f7900b2f700 1 -- 172.21.15.46:0/896103385 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f8009ea0 con 0x7f78fc15d110 2024-01-26T23:30:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.705+0000 7f7900b2f700 1 -- 172.21.15.46:0/896103385 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f800b860 con 0x7f78fc15d110 2024-01-26T23:30:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.705+0000 7f7900b2f700 1 -- 172.21.15.46:0/896103385 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f800bc00 con 0x7f78fc15d110 2024-01-26T23:30:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.705+0000 7f7903d95700 1 -- 172.21.15.46:0/896103385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78fc15d110 msgr2=0x7f78fc0cb7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.705+0000 7f7903d95700 1 --2- 172.21.15.46:0/896103385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78fc15d110 0x7f78fc0cb7a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f78f80099b0 tx=0x7f78f8009250 comp rx=0 tx=0).stop 2024-01-26T23:30:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.705+0000 7f7903d95700 1 -- 172.21.15.46:0/896103385 shutdown_connections 2024-01-26T23:30:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.705+0000 7f7903d95700 1 --2- 172.21.15.46:0/896103385 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78fc15cd40 0x7f78fc0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.705+0000 7f7903d95700 1 --2- 172.21.15.46:0/896103385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78fc15d110 0x7f78fc0cb7a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.705+0000 7f7903d95700 1 -- 172.21.15.46:0/896103385 >> 172.21.15.46:0/896103385 conn(0x7f78fc0cdf50 msgr2=0x7f78fc0c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.705+0000 7f7903d95700 1 -- 172.21.15.46:0/896103385 shutdown_connections 2024-01-26T23:30:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.705+0000 7f7903d95700 1 -- 172.21.15.46:0/896103385 wait complete. 2024-01-26T23:30:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.706+0000 7f7903d95700 1 Processor -- start 2024-01-26T23:30:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.706+0000 7f7903d95700 1 -- start start 2024-01-26T23:30:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.707+0000 7f7903d95700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78fc067930 0x7f78fc154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.707+0000 7f7903d95700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78fc15cd40 0x7f78fc1584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.707+0000 7f7903d95700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78fc15d110 0x7f78fc0acd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.707+0000 7f7903d95700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc14f8d0 con 0x7f78fc15cd40 2024-01-26T23:30:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.707+0000 7f7903d95700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc1541c0 con 0x7f78fc15d110 2024-01-26T23:30:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.707+0000 7f7903d95700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc1532d0 con 0x7f78fc067930 2024-01-26T23:30:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.707+0000 7f7901330700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78fc15cd40 0x7f78fc1584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.707+0000 7f7901b31700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78fc067930 0x7f78fc154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.707+0000 7f7901330700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78fc15cd40 0x7f78fc1584f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60930/0 (socket says 172.21.15.46:60930) 2024-01-26T23:30:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.707+0000 7f7901330700 1 -- 172.21.15.46:0/306095901 learned_addr learned my addr 172.21.15.46:0/306095901 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.707+0000 7f7902332700 1 --2- 172.21.15.46:0/306095901 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78fc15d110 0x7f78fc0acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.708+0000 7f7902332700 1 -- 172.21.15.46:0/306095901 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78fc067930 msgr2=0x7f78fc154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.708+0000 7f7902332700 1 --2- 172.21.15.46:0/306095901 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78fc067930 0x7f78fc154650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.708+0000 7f7902332700 1 -- 172.21.15.46:0/306095901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78fc15cd40 msgr2=0x7f78fc1584f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.708+0000 7f7902332700 1 --2- 172.21.15.46:0/306095901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78fc15cd40 0x7f78fc1584f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.708+0000 7f7902332700 1 -- 172.21.15.46:0/306095901 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f80094f0 con 0x7f78fc15d110 2024-01-26T23:30:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.708+0000 7f7901330700 1 --2- 172.21.15.46:0/306095901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78fc15cd40 0x7f78fc1584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.708+0000 7f7902332700 1 --2- 172.21.15.46:0/306095901 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78fc15d110 0x7f78fc0acd00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f78f400cc40 tx=0x7f78f4001a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.708+0000 7f78f1ffb700 1 -- 172.21.15.46:0/306095901 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f4005df0 con 0x7f78fc15d110 2024-01-26T23:30:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.708+0000 7f78f1ffb700 1 -- 172.21.15.46:0/306095901 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f400a6a0 con 0x7f78fc15d110 2024-01-26T23:30:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.708+0000 7f78f1ffb700 1 -- 172.21.15.46:0/306095901 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f400aa60 con 0x7f78fc15d110 2024-01-26T23:30:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.708+0000 7f7903d95700 1 -- 172.21.15.46:0/306095901 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78fc1535b0 con 0x7f78fc15d110 2024-01-26T23:30:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.708+0000 7f7903d95700 1 -- 172.21.15.46:0/306095901 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78fc153b70 con 0x7f78fc15d110 2024-01-26T23:30:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.709+0000 7f7903d95700 1 -- 172.21.15.46:0/306095901 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78fc154350 con 0x7f78fc15d110 2024-01-26T23:30:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.710+0000 7f78f1ffb700 1 -- 172.21.15.46:0/306095901 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f78f40070f0 con 0x7f78fc15d110 2024-01-26T23:30:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.710+0000 7f78f1ffb700 1 --2- 172.21.15.46:0/306095901 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f78e0066f30 0x7f78e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.710+0000 7f78f1ffb700 1 -- 172.21.15.46:0/306095901 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f78f40848a0 con 0x7f78fc15d110 2024-01-26T23:30:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.710+0000 7f7901b31700 1 --2- 172.21.15.46:0/306095901 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f78e0066f30 0x7f78e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:37.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.711+0000 7f7901b31700 1 --2- 172.21.15.46:0/306095901 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f78e0066f30 0x7f78e00693e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f78fc155050 tx=0x7f78ec0091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.715+0000 7f78f1ffb700 1 -- 172.21.15.46:0/306095901 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f78f4051800 con 0x7f78fc15d110 2024-01-26T23:30:37.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.918+0000 7f7903d95700 1 -- 172.21.15.46:0/306095901 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f78fc04c710 con 0x7f78fc15d110 2024-01-26T23:30:37.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.919+0000 7f78f1ffb700 1 -- 172.21.15.46:0/306095901 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f78f4054e20 con 0x7f78fc15d110 2024-01-26T23:30:37.920 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:30:37.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.921+0000 7f7903d95700 1 -- 172.21.15.46:0/306095901 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f78e0066f30 msgr2=0x7f78e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:37.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.921+0000 7f7903d95700 1 --2- 172.21.15.46:0/306095901 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f78e0066f30 0x7f78e00693e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f78fc155050 tx=0x7f78ec0091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.921+0000 7f7903d95700 1 -- 172.21.15.46:0/306095901 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78fc15d110 msgr2=0x7f78fc0acd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:37.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.921+0000 7f7903d95700 1 --2- 172.21.15.46:0/306095901 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78fc15d110 0x7f78fc0acd00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f78f400cc40 tx=0x7f78f4001a50 comp rx=0 tx=0).stop 2024-01-26T23:30:37.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.921+0000 7f7903d95700 1 -- 172.21.15.46:0/306095901 shutdown_connections 2024-01-26T23:30:37.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.921+0000 7f7903d95700 1 --2- 172.21.15.46:0/306095901 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f78e0066f30 0x7f78e00693e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.921+0000 7f7903d95700 1 --2- 172.21.15.46:0/306095901 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78fc15d110 0x7f78fc0acd00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.921+0000 7f7903d95700 1 --2- 172.21.15.46:0/306095901 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78fc067930 0x7f78fc154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.921+0000 7f7903d95700 1 --2- 172.21.15.46:0/306095901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78fc15cd40 0x7f78fc1584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:37.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.921+0000 7f7903d95700 1 -- 172.21.15.46:0/306095901 >> 172.21.15.46:0/306095901 conn(0x7f78fc0cdf50 msgr2=0x7f78fc0c9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:37.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.921+0000 7f7903d95700 1 -- 172.21.15.46:0/306095901 shutdown_connections 2024-01-26T23:30:37.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:37.921+0000 7f7903d95700 1 -- 172.21.15.46:0/306095901 wait complete. 2024-01-26T23:30:37.936 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-26T23:30:38.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.070+0000 7f2ebc869700 1 Processor -- start 2024-01-26T23:30:38.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.070+0000 7f2ebc869700 1 -- start start 2024-01-26T23:30:38.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.071+0000 7f2ebc869700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2eb8157490 0x7f2eb80cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.071+0000 7f2ebc869700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2eb8157860 0x7f2eb80cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.071+0000 7f2ebc869700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2eb806d6a0 0x7f2eb806dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.071+0000 7f2ebc869700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb80cdc60 con 0x7f2eb8157860 2024-01-26T23:30:38.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.071+0000 7f2ebc869700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb80cea80 con 0x7f2eb806d6a0 2024-01-26T23:30:38.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.071+0000 7f2ebc869700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb806dff0 con 0x7f2eb8157490 2024-01-26T23:30:38.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.071+0000 7f2eb5d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2eb8157860 0x7f2eb80cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.071+0000 7f2eb5d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2eb8157860 0x7f2eb80cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60940/0 (socket says 172.21.15.46:60940) 2024-01-26T23:30:38.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.071+0000 7f2eb5d9b700 1 -- 172.21.15.46:0/512133333 learned_addr learned my addr 172.21.15.46:0/512133333 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:38.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.071+0000 7f2eb6d9d700 1 --2- 172.21.15.46:0/512133333 >> v2:172.21.15.106:3300/0 conn(0x7f2eb806d6a0 0x7f2eb806dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.071+0000 7f2eb659c700 1 --2- 172.21.15.46:0/512133333 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2eb8157490 0x7f2eb80cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.071+0000 7f2eb5d9b700 1 -- 172.21.15.46:0/512133333 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2eb8157490 msgr2=0x7f2eb80cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:38.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.071+0000 7f2eb5d9b700 1 --2- 172.21.15.46:0/512133333 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2eb8157490 0x7f2eb80cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.071+0000 7f2eb5d9b700 1 -- 172.21.15.46:0/512133333 >> v2:172.21.15.106:3300/0 conn(0x7f2eb806d6a0 msgr2=0x7f2eb806dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:38.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.071+0000 7f2eb5d9b700 1 --2- 172.21.15.46:0/512133333 >> v2:172.21.15.106:3300/0 conn(0x7f2eb806d6a0 0x7f2eb806dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.071+0000 7f2eb5d9b700 1 -- 172.21.15.46:0/512133333 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb80cde70 con 0x7f2eb8157860 2024-01-26T23:30:38.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.072+0000 7f2eb5d9b700 1 --2- 172.21.15.46:0/512133333 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eb8157860 0x7f2eb80cd620 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2eb806f970 tx=0x7f2eac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8fb6f0aa2848bb6 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:38.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.072+0000 7f2eb559a700 1 -- 172.21.15.46:0/512133333 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac00b860 con 0x7f2eb8157860 2024-01-26T23:30:38.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.072+0000 7f2eb559a700 1 -- 172.21.15.46:0/512133333 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eac00b9c0 con 0x7f2eb8157860 2024-01-26T23:30:38.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.072+0000 7f2eb559a700 1 -- 172.21.15.46:0/512133333 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac00bd60 con 0x7f2eb8157860 2024-01-26T23:30:38.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.072+0000 7f2ebc869700 1 -- 172.21.15.46:0/512133333 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eb8157860 msgr2=0x7f2eb80cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:38.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.072+0000 7f2ebc869700 1 --2- 172.21.15.46:0/512133333 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eb8157860 0x7f2eb80cd620 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2eb806f970 tx=0x7f2eac009250 comp rx=0 tx=0).stop 2024-01-26T23:30:38.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.072+0000 7f2ebc869700 1 -- 172.21.15.46:0/512133333 shutdown_connections 2024-01-26T23:30:38.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.072+0000 7f2ebc869700 1 --2- 172.21.15.46:0/512133333 >> v2:172.21.15.106:3300/0 conn(0x7f2eb806d6a0 0x7f2eb806dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.072+0000 7f2ebc869700 1 --2- 172.21.15.46:0/512133333 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2eb8157490 0x7f2eb80cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.072+0000 7f2ebc869700 1 --2- 172.21.15.46:0/512133333 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eb8157860 0x7f2eb80cd620 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.072+0000 7f2ebc869700 1 -- 172.21.15.46:0/512133333 >> 172.21.15.46:0/512133333 conn(0x7f2eb8067930 msgr2=0x7f2eb8067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:38.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.073+0000 7f2ebc869700 1 -- 172.21.15.46:0/512133333 shutdown_connections 2024-01-26T23:30:38.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.073+0000 7f2ebc869700 1 -- 172.21.15.46:0/512133333 wait complete. 2024-01-26T23:30:38.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.074+0000 7f2ebc869700 1 Processor -- start 2024-01-26T23:30:38.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.074+0000 7f2ebc869700 1 -- start start 2024-01-26T23:30:38.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.075+0000 7f2ebc869700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2eb806d6a0 0x7f2eb814c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.075+0000 7f2ebc869700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eb8157490 0x7f2eb80adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.075+0000 7f2ebc869700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2eb8157860 0x7f2eb80ae4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.075+0000 7f2ebc869700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb815b520 con 0x7f2eb8157490 2024-01-26T23:30:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.075+0000 7f2ebc869700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb80af760 con 0x7f2eb8157860 2024-01-26T23:30:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.075+0000 7f2ebc869700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb80aeb50 con 0x7f2eb806d6a0 2024-01-26T23:30:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.075+0000 7f2eb5d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eb8157490 0x7f2eb80adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.075+0000 7f2eb5d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eb8157490 0x7f2eb80adf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60948/0 (socket says 172.21.15.46:60948) 2024-01-26T23:30:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.075+0000 7f2eb5d9b700 1 -- 172.21.15.46:0/1142950984 learned_addr learned my addr 172.21.15.46:0/1142950984 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.076+0000 7f2eb6d9d700 1 --2- 172.21.15.46:0/1142950984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2eb8157860 0x7f2eb80ae4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.076+0000 7f2eb659c700 1 --2- 172.21.15.46:0/1142950984 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2eb806d6a0 0x7f2eb814c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.076+0000 7f2eb5d9b700 1 -- 172.21.15.46:0/1142950984 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2eb806d6a0 msgr2=0x7f2eb814c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.076+0000 7f2eb5d9b700 1 --2- 172.21.15.46:0/1142950984 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2eb806d6a0 0x7f2eb814c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.076+0000 7f2eb5d9b700 1 -- 172.21.15.46:0/1142950984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2eb8157860 msgr2=0x7f2eb80ae4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:38.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.076+0000 7f2eb5d9b700 1 --2- 172.21.15.46:0/1142950984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2eb8157860 0x7f2eb80ae4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.076+0000 7f2eb5d9b700 1 -- 172.21.15.46:0/1142950984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eac0094f0 con 0x7f2eb8157490 2024-01-26T23:30:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.076+0000 7f2eb659c700 1 --2- 172.21.15.46:0/1142950984 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2eb806d6a0 0x7f2eb814c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.076+0000 7f2eb5d9b700 1 --2- 172.21.15.46:0/1142950984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eb8157490 0x7f2eb80adf70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2eb806d060 tx=0x7f2eac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.076+0000 7f2ea67fc700 1 -- 172.21.15.46:0/1142950984 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac011e70 con 0x7f2eb8157490 2024-01-26T23:30:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.076+0000 7f2ea67fc700 1 -- 172.21.15.46:0/1142950984 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eac024440 con 0x7f2eb8157490 2024-01-26T23:30:38.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.076+0000 7f2ebc869700 1 -- 172.21.15.46:0/1142950984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb80aee30 con 0x7f2eb8157490 2024-01-26T23:30:38.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.076+0000 7f2ea67fc700 1 -- 172.21.15.46:0/1142950984 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac024800 con 0x7f2eb8157490 2024-01-26T23:30:38.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.076+0000 7f2ebc869700 1 -- 172.21.15.46:0/1142950984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb80af3f0 con 0x7f2eb8157490 2024-01-26T23:30:38.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.077+0000 7f2ebc869700 1 -- 172.21.15.46:0/1142950984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb80af910 con 0x7f2eb8157490 2024-01-26T23:30:38.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.078+0000 7f2ea67fc700 1 -- 172.21.15.46:0/1142950984 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2eac01a420 con 0x7f2eb8157490 2024-01-26T23:30:38.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.078+0000 7f2ea67fc700 1 --2- 172.21.15.46:0/1142950984 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2e98066f30 0x7f2e980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.078+0000 7f2ea67fc700 1 -- 172.21.15.46:0/1142950984 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f2eac086790 con 0x7f2eb8157490 2024-01-26T23:30:38.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.078+0000 7f2eb659c700 1 --2- 172.21.15.46:0/1142950984 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2e98066f30 0x7f2e980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.079+0000 7f2eb659c700 1 --2- 172.21.15.46:0/1142950984 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2e98066f30 0x7f2e980693e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800ab70 tx=0x7f2ea8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:38.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.083+0000 7f2ea67fc700 1 -- 172.21.15.46:0/1142950984 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2eb80af910 con 0x7f2eb8157490 2024-01-26T23:30:38.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.287+0000 7f2ebc869700 1 -- 172.21.15.46:0/1142950984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2eb80ce390 con 0x7f2eb8157490 2024-01-26T23:30:38.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.288+0000 7f2ea67fc700 1 -- 172.21.15.46:0/1142950984 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2eac053640 con 0x7f2eb8157490 2024-01-26T23:30:38.290 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-26T23:30:38.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.290+0000 7f2ebc869700 1 -- 172.21.15.46:0/1142950984 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2e98066f30 msgr2=0x7f2e980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:38.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.290+0000 7f2ebc869700 1 --2- 172.21.15.46:0/1142950984 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2e98066f30 0x7f2e980693e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800ab70 tx=0x7f2ea8008040 comp rx=0 tx=0).stop 2024-01-26T23:30:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.290+0000 7f2ebc869700 1 -- 172.21.15.46:0/1142950984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eb8157490 msgr2=0x7f2eb80adf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.290+0000 7f2ebc869700 1 --2- 172.21.15.46:0/1142950984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eb8157490 0x7f2eb80adf70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2eb806d060 tx=0x7f2eac017000 comp rx=0 tx=0).stop 2024-01-26T23:30:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.290+0000 7f2ebc869700 1 -- 172.21.15.46:0/1142950984 shutdown_connections 2024-01-26T23:30:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.290+0000 7f2ebc869700 1 --2- 172.21.15.46:0/1142950984 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2e98066f30 0x7f2e980693e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.290+0000 7f2ebc869700 1 --2- 172.21.15.46:0/1142950984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2eb8157860 0x7f2eb80ae4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.290+0000 7f2ebc869700 1 --2- 172.21.15.46:0/1142950984 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2eb806d6a0 0x7f2eb814c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.290+0000 7f2ebc869700 1 --2- 172.21.15.46:0/1142950984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eb8157490 0x7f2eb80adf70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.290+0000 7f2ebc869700 1 -- 172.21.15.46:0/1142950984 >> 172.21.15.46:0/1142950984 conn(0x7f2eb8067930 msgr2=0x7f2eb8153b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.290+0000 7f2ebc869700 1 -- 172.21.15.46:0/1142950984 shutdown_connections 2024-01-26T23:30:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.290+0000 7f2ebc869700 1 -- 172.21.15.46:0/1142950984 wait complete. 2024-01-26T23:30:38.306 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-01-26T23:30:38.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.438+0000 7f36c372a700 1 Processor -- start 2024-01-26T23:30:38.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.438+0000 7f36c372a700 1 -- start start 2024-01-26T23:30:38.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.439+0000 7f36c372a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc15df10 0x7f36bc0aa130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.439+0000 7f36c372a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f36bc0cdff0 0x7f36bc067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.439+0000 7f36c372a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc0c8f00 con 0x7f36bc15e2e0 2024-01-26T23:30:38.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.439+0000 7f36c372a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36bc068160 con 0x7f36bc0cdff0 2024-01-26T23:30:38.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.439+0000 7f36c372a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc068440 con 0x7f36bc15df10 2024-01-26T23:30:38.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.439+0000 7f36c1cc7700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f36bc15e2e0 0x7f36bc0cc4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59136/0 (socket says 172.21.15.46:59136) 2024-01-26T23:30:38.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.439+0000 7f36c1cc7700 1 -- 172.21.15.46:0/3133858086 learned_addr learned my addr 172.21.15.46:0/3133858086 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:38.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.439+0000 7f36c14c6700 1 --2- 172.21.15.46:0/3133858086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc15df10 0x7f36bc0aa130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.439+0000 7f36c0cc5700 1 --2- 172.21.15.46:0/3133858086 >> v2:172.21.15.106:3300/0 conn(0x7f36bc0cdff0 0x7f36bc067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.440+0000 7f36b3fff700 1 -- 172.21.15.46:0/3133858086 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166654972 0 0) 0x7f36bc0c8f00 con 0x7f36bc15e2e0 2024-01-26T23:30:38.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.440+0000 7f36b3fff700 1 -- 172.21.15.46:0/3133858086 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a8003570 con 0x7f36bc15e2e0 2024-01-26T23:30:38.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.440+0000 7f36c14c6700 1 -- 172.21.15.46:0/3133858086 >> v2:172.21.15.106:3300/0 conn(0x7f36bc0cdff0 msgr2=0x7f36bc067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:38.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.440+0000 7f36c14c6700 1 --2- 172.21.15.46:0/3133858086 >> v2:172.21.15.106:3300/0 conn(0x7f36bc0cdff0 0x7f36bc067930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.440+0000 7f36c14c6700 1 -- 172.21.15.46:0/3133858086 >> v1:172.21.15.46:6789/0 conn(0x7f36bc15e2e0 legacy=0x7f36bc0cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:38.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.440+0000 7f36c14c6700 1 -- 172.21.15.46:0/3133858086 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36bc068720 con 0x7f36bc15df10 2024-01-26T23:30:38.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.440+0000 7f36c0cc5700 1 --2- 172.21.15.46:0/3133858086 >> v2:172.21.15.106:3300/0 conn(0x7f36bc0cdff0 0x7f36bc067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:38.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.440+0000 7f36c14c6700 1 --2- 172.21.15.46:0/3133858086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc15df10 0x7f36bc0aa130 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f36b80098d0 tx=0x7f36b8009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5afe8be20f2a7943 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:38.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.440+0000 7f36b3fff700 1 -- 172.21.15.46:0/3133858086 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b8005c10 con 0x7f36bc15df10 2024-01-26T23:30:38.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.440+0000 7f36b3fff700 1 -- 172.21.15.46:0/3133858086 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b8005d70 con 0x7f36bc15df10 2024-01-26T23:30:38.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.440+0000 7f36b3fff700 1 -- 172.21.15.46:0/3133858086 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b8006280 con 0x7f36bc15df10 2024-01-26T23:30:38.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.441+0000 7f36c372a700 1 -- 172.21.15.46:0/3133858086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc15df10 msgr2=0x7f36bc0aa130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:38.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.441+0000 7f36c372a700 1 --2- 172.21.15.46:0/3133858086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc15df10 0x7f36bc0aa130 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f36b80098d0 tx=0x7f36b8009120 comp rx=0 tx=0).stop 2024-01-26T23:30:38.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.441+0000 7f36c372a700 1 -- 172.21.15.46:0/3133858086 shutdown_connections 2024-01-26T23:30:38.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.441+0000 7f36c372a700 1 --2- 172.21.15.46:0/3133858086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc15df10 0x7f36bc0aa130 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.441+0000 7f36c372a700 1 --2- 172.21.15.46:0/3133858086 >> v2:172.21.15.106:3300/0 conn(0x7f36bc0cdff0 0x7f36bc067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.441+0000 7f36c372a700 1 -- 172.21.15.46:0/3133858086 >> 172.21.15.46:0/3133858086 conn(0x7f36bc069220 msgr2=0x7f36bc0cd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:38.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.441+0000 7f36c372a700 1 -- 172.21.15.46:0/3133858086 shutdown_connections 2024-01-26T23:30:38.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.441+0000 7f36c372a700 1 -- 172.21.15.46:0/3133858086 wait complete. 2024-01-26T23:30:38.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.442+0000 7f36c372a700 1 Processor -- start 2024-01-26T23:30:38.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.442+0000 7f36c372a700 1 -- start start 2024-01-26T23:30:38.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.443+0000 7f36c372a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36bc0cdff0 0x7f36bc0c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.443+0000 7f36c372a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc15df10 0x7f36bc0af120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.443+0000 7f36c372a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36bc15e2e0 0x7f36bc1582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.443+0000 7f36c372a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc0af660 con 0x7f36bc15e2e0 2024-01-26T23:30:38.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.443+0000 7f36c372a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc0c74e0 con 0x7f36bc0cdff0 2024-01-26T23:30:38.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.443+0000 7f36c372a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc0af8b0 con 0x7f36bc15df10 2024-01-26T23:30:38.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.443+0000 7f36c1cc7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36bc15e2e0 0x7f36bc1582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.443+0000 7f36c1cc7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36bc15e2e0 0x7f36bc1582a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60952/0 (socket says 172.21.15.46:60952) 2024-01-26T23:30:38.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.443+0000 7f36c1cc7700 1 -- 172.21.15.46:0/492819434 learned_addr learned my addr 172.21.15.46:0/492819434 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:38.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.444+0000 7f36c14c6700 1 --2- 172.21.15.46:0/492819434 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36bc0cdff0 0x7f36bc0c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.444+0000 7f36c0cc5700 1 --2- 172.21.15.46:0/492819434 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc15df10 0x7f36bc0af120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.444+0000 7f36c1cc7700 1 -- 172.21.15.46:0/492819434 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc15df10 msgr2=0x7f36bc0af120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:38.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.444+0000 7f36c1cc7700 1 --2- 172.21.15.46:0/492819434 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc15df10 0x7f36bc0af120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.444+0000 7f36c1cc7700 1 -- 172.21.15.46:0/492819434 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36bc0cdff0 msgr2=0x7f36bc0c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:38.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.444+0000 7f36c1cc7700 1 --2- 172.21.15.46:0/492819434 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36bc0cdff0 0x7f36bc0c6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.444+0000 7f36c1cc7700 1 -- 172.21.15.46:0/492819434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ac009420 con 0x7f36bc15e2e0 2024-01-26T23:30:38.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.444+0000 7f36c1cc7700 1 --2- 172.21.15.46:0/492819434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36bc15e2e0 0x7f36bc1582a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f36b4001460 tx=0x7f36b4001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:38.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.444+0000 7f36b17fa700 1 -- 172.21.15.46:0/492819434 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b400a8a0 con 0x7f36bc15e2e0 2024-01-26T23:30:38.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.444+0000 7f36b17fa700 1 -- 172.21.15.46:0/492819434 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b400aa00 con 0x7f36bc15e2e0 2024-01-26T23:30:38.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.444+0000 7f36c372a700 1 -- 172.21.15.46:0/492819434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b80093c0 con 0x7f36bc15e2e0 2024-01-26T23:30:38.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.444+0000 7f36b17fa700 1 -- 172.21.15.46:0/492819434 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b400ad80 con 0x7f36bc15e2e0 2024-01-26T23:30:38.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.444+0000 7f36c372a700 1 -- 172.21.15.46:0/492819434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36bc0c62e0 con 0x7f36bc15e2e0 2024-01-26T23:30:38.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.445+0000 7f36c372a700 1 -- 172.21.15.46:0/492819434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36bc04c710 con 0x7f36bc15e2e0 2024-01-26T23:30:38.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.446+0000 7f36b17fa700 1 -- 172.21.15.46:0/492819434 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f36b4007300 con 0x7f36bc15e2e0 2024-01-26T23:30:38.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.446+0000 7f36b17fa700 1 --2- 172.21.15.46:0/492819434 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36a0066f30 0x7f36a00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.446+0000 7f36b17fa700 1 -- 172.21.15.46:0/492819434 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f36b4084d10 con 0x7f36bc15e2e0 2024-01-26T23:30:38.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.447+0000 7f36c14c6700 1 --2- 172.21.15.46:0/492819434 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36a0066f30 0x7f36a00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.447+0000 7f36c14c6700 1 --2- 172.21.15.46:0/492819434 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36a0066f30 0x7f36a00693e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f36b8009340 tx=0x7f36b8016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:38.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.451+0000 7f36b17fa700 1 -- 172.21.15.46:0/492819434 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f36b408a020 con 0x7f36bc15e2e0 2024-01-26T23:30:38.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.655+0000 7f36c372a700 1 -- 172.21.15.46:0/492819434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f36bc0aa670 con 0x7f36bc15e2e0 2024-01-26T23:30:38.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.656+0000 7f36b17fa700 1 -- 172.21.15.46:0/492819434 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f36b4051b20 con 0x7f36bc15e2e0 2024-01-26T23:30:38.658 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:30:38.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.658+0000 7f36c372a700 1 -- 172.21.15.46:0/492819434 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36a0066f30 msgr2=0x7f36a00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:38.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.658+0000 7f36c372a700 1 --2- 172.21.15.46:0/492819434 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36a0066f30 0x7f36a00693e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f36b8009340 tx=0x7f36b8016000 comp rx=0 tx=0).stop 2024-01-26T23:30:38.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.658+0000 7f36c372a700 1 -- 172.21.15.46:0/492819434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36bc15e2e0 msgr2=0x7f36bc1582a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:38.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.658+0000 7f36c372a700 1 --2- 172.21.15.46:0/492819434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36bc15e2e0 0x7f36bc1582a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f36b4001460 tx=0x7f36b4001a40 comp rx=0 tx=0).stop 2024-01-26T23:30:38.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.658+0000 7f36c372a700 1 -- 172.21.15.46:0/492819434 shutdown_connections 2024-01-26T23:30:38.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.658+0000 7f36c372a700 1 --2- 172.21.15.46:0/492819434 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36a0066f30 0x7f36a00693e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.658+0000 7f36c372a700 1 --2- 172.21.15.46:0/492819434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36bc15e2e0 0x7f36bc1582a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.658+0000 7f36c372a700 1 --2- 172.21.15.46:0/492819434 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36bc0cdff0 0x7f36bc0c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.658+0000 7f36c372a700 1 --2- 172.21.15.46:0/492819434 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc15df10 0x7f36bc0af120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.658+0000 7f36c372a700 1 -- 172.21.15.46:0/492819434 >> 172.21.15.46:0/492819434 conn(0x7f36bc069220 msgr2=0x7f36bc0c9390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:38.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.658+0000 7f36c372a700 1 -- 172.21.15.46:0/492819434 shutdown_connections 2024-01-26T23:30:38.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.658+0000 7f36c372a700 1 -- 172.21.15.46:0/492819434 wait complete. 2024-01-26T23:30:38.674 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:30:38.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.805+0000 7fb83373c700 1 Processor -- start 2024-01-26T23:30:38.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.805+0000 7fb83373c700 1 -- start start 2024-01-26T23:30:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.806+0000 7fb83373c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb82c154290 0x7fb82c0b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.806+0000 7fb83373c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb82c0b2100 0x7fb82c0b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.806+0000 7fb83373c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c0b2560 con 0x7fb82c153ec0 2024-01-26T23:30:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.806+0000 7fb83373c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82c15b140 con 0x7fb82c0b2100 2024-01-26T23:30:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.806+0000 7fb83373c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82c15a5c0 con 0x7fb82c154290 2024-01-26T23:30:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.806+0000 7fb831cd9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb82c0b2100 0x7fb82c0b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.806+0000 7fb831cd9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb82c0b2100 0x7fb82c0b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60956/0 (socket says 172.21.15.46:60956) 2024-01-26T23:30:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.806+0000 7fb831cd9700 1 -- 172.21.15.46:0/1899183995 learned_addr learned my addr 172.21.15.46:0/1899183995 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:38.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.806+0000 7fb830cd7700 1 --2- 172.21.15.46:0/1899183995 >> v2:172.21.15.106:3300/0 conn(0x7fb82c154290 0x7fb82c0b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.806+0000 7fb831cd9700 1 -- 172.21.15.46:0/1899183995 >> v2:172.21.15.106:3300/0 conn(0x7fb82c154290 msgr2=0x7fb82c0b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:38.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.806+0000 7fb831cd9700 1 --2- 172.21.15.46:0/1899183995 >> v2:172.21.15.106:3300/0 conn(0x7fb82c154290 0x7fb82c0b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.806+0000 7fb831cd9700 1 -- 172.21.15.46:0/1899183995 >> v1:172.21.15.106:6789/0 conn(0x7fb82c153ec0 legacy=0x7fb82c157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:38.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.806+0000 7fb831cd9700 1 -- 172.21.15.46:0/1899183995 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c15a8a0 con 0x7fb82c0b2100 2024-01-26T23:30:38.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.806+0000 7fb831cd9700 1 --2- 172.21.15.46:0/1899183995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c0b2100 0x7fb82c0b04f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb82c06e2c0 tx=0x7fb82400ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fcf6364f1d12521 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:38.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.807+0000 7fb81ffff700 1 -- 172.21.15.46:0/1899183995 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824010070 con 0x7fb82c0b2100 2024-01-26T23:30:38.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.807+0000 7fb81ffff700 1 -- 172.21.15.46:0/1899183995 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82400de50 con 0x7fb82c0b2100 2024-01-26T23:30:38.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.807+0000 7fb81ffff700 1 -- 172.21.15.46:0/1899183995 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82400b320 con 0x7fb82c0b2100 2024-01-26T23:30:38.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.807+0000 7fb83373c700 1 -- 172.21.15.46:0/1899183995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c0b2100 msgr2=0x7fb82c0b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:38.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.807+0000 7fb83373c700 1 --2- 172.21.15.46:0/1899183995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c0b2100 0x7fb82c0b04f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb82c06e2c0 tx=0x7fb82400ba50 comp rx=0 tx=0).stop 2024-01-26T23:30:38.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.807+0000 7fb83373c700 1 -- 172.21.15.46:0/1899183995 shutdown_connections 2024-01-26T23:30:38.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.807+0000 7fb83373c700 1 --2- 172.21.15.46:0/1899183995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c0b2100 0x7fb82c0b04f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.807+0000 7fb83373c700 1 --2- 172.21.15.46:0/1899183995 >> v2:172.21.15.106:3300/0 conn(0x7fb82c154290 0x7fb82c0b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.807+0000 7fb83373c700 1 -- 172.21.15.46:0/1899183995 >> 172.21.15.46:0/1899183995 conn(0x7fb82c069510 msgr2=0x7fb82c0aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:38.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.807+0000 7fb83373c700 1 -- 172.21.15.46:0/1899183995 shutdown_connections 2024-01-26T23:30:38.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.807+0000 7fb83373c700 1 -- 172.21.15.46:0/1899183995 wait complete. 2024-01-26T23:30:38.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.808+0000 7fb83373c700 1 Processor -- start 2024-01-26T23:30:38.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.809+0000 7fb83373c700 1 -- start start 2024-01-26T23:30:38.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.809+0000 7fb83373c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb82c0b2100 0x7fb82c153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.809+0000 7fb83373c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c153ec0 0x7fb82c0a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.809+0000 7fb83373c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb82c154290 0x7fb82c1582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.809+0000 7fb83373c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c153580 con 0x7fb82c153ec0 2024-01-26T23:30:38.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.809+0000 7fb83373c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c0cd070 con 0x7fb82c0b2100 2024-01-26T23:30:38.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.809+0000 7fb83373c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c165830 con 0x7fb82c154290 2024-01-26T23:30:38.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.810+0000 7fb830cd7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c153ec0 0x7fb82c0a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.810+0000 7fb830cd7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c153ec0 0x7fb82c0a5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60964/0 (socket says 172.21.15.46:60964) 2024-01-26T23:30:38.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.810+0000 7fb830cd7700 1 -- 172.21.15.46:0/1870782239 learned_addr learned my addr 172.21.15.46:0/1870782239 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:38.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.810+0000 7fb8314d8700 1 --2- 172.21.15.46:0/1870782239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb82c0b2100 0x7fb82c153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.810+0000 7fb831cd9700 1 --2- 172.21.15.46:0/1870782239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb82c154290 0x7fb82c1582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.810+0000 7fb830cd7700 1 -- 172.21.15.46:0/1870782239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb82c154290 msgr2=0x7fb82c1582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.810+0000 7fb830cd7700 1 --2- 172.21.15.46:0/1870782239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb82c154290 0x7fb82c1582d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.810+0000 7fb830cd7700 1 -- 172.21.15.46:0/1870782239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb82c0b2100 msgr2=0x7fb82c153040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.810+0000 7fb830cd7700 1 --2- 172.21.15.46:0/1870782239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb82c0b2100 0x7fb82c153040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.810+0000 7fb830cd7700 1 -- 172.21.15.46:0/1870782239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82400bcf0 con 0x7fb82c153ec0 2024-01-26T23:30:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.810+0000 7fb830cd7700 1 --2- 172.21.15.46:0/1870782239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c153ec0 0x7fb82c0a5970 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb82800d8c0 tx=0x7fb828008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.810+0000 7fb81d7fa700 1 -- 172.21.15.46:0/1870782239 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8280125f0 con 0x7fb82c153ec0 2024-01-26T23:30:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.810+0000 7fb81d7fa700 1 -- 172.21.15.46:0/1870782239 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb828012750 con 0x7fb82c153ec0 2024-01-26T23:30:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.810+0000 7fb83373c700 1 -- 172.21.15.46:0/1870782239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb82c0cc7d0 con 0x7fb82c153ec0 2024-01-26T23:30:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.810+0000 7fb81d7fa700 1 -- 172.21.15.46:0/1870782239 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb828012af0 con 0x7fb82c153ec0 2024-01-26T23:30:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.810+0000 7fb83373c700 1 -- 172.21.15.46:0/1870782239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb82c0ccd90 con 0x7fb82c153ec0 2024-01-26T23:30:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.811+0000 7fb83373c700 1 -- 172.21.15.46:0/1870782239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb82c06e3e0 con 0x7fb82c153ec0 2024-01-26T23:30:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.812+0000 7fb81d7fa700 1 -- 172.21.15.46:0/1870782239 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb82801e460 con 0x7fb82c153ec0 2024-01-26T23:30:38.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.812+0000 7fb81d7fa700 1 --2- 172.21.15.46:0/1870782239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb810066f30 0x7fb8100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:38.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.813+0000 7fb81d7fa700 1 -- 172.21.15.46:0/1870782239 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7fb828084f10 con 0x7fb82c153ec0 2024-01-26T23:30:38.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.813+0000 7fb8314d8700 1 --2- 172.21.15.46:0/1870782239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb810066f30 0x7fb8100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:38.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.813+0000 7fb8314d8700 1 --2- 172.21.15.46:0/1870782239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb810066f30 0x7fb8100693e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb820003a30 tx=0x7fb820007ea0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:38.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:38.817+0000 7fb81d7fa700 1 -- 172.21.15.46:0/1870782239 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb828051d20 con 0x7fb82c153ec0 2024-01-26T23:30:39.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.070+0000 7fb83373c700 1 -- 172.21.15.46:0/1870782239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb82c1655d0 con 0x7fb82c153ec0 2024-01-26T23:30:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.070+0000 7fb81d7fa700 1 -- 172.21.15.46:0/1870782239 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4593 (secure 0 0 0) 0x7fb828055340 con 0x7fb82c153ec0 2024-01-26T23:30:39.072 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:39.072 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3409861270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3409861270},{"type":"v1","addr":"172.21.15.46:6833","nonce":3409861270}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3319220815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3319220815},{"type":"v1","addr":"172.21.15.106:6836","nonce":3319220815}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4252,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/529198432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":529198432},{"type":"v1","addr":"172.21.15.106:6837","nonce":529198432}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:29:44.031963+0000","modified":"2024-01-26T23:30:03.652307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/688086235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":688086235},{"type":"v1","addr":"172.21.15.46:6835","nonce":688086235}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2024-01-26T23:30:39.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.072+0000 7fb83373c700 1 -- 172.21.15.46:0/1870782239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb810066f30 msgr2=0x7fb8100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:39.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.072+0000 7fb83373c700 1 --2- 172.21.15.46:0/1870782239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb810066f30 0x7fb8100693e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb820003a30 tx=0x7fb820007ea0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.072+0000 7fb83373c700 1 -- 172.21.15.46:0/1870782239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c153ec0 msgr2=0x7fb82c0a5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.072+0000 7fb83373c700 1 --2- 172.21.15.46:0/1870782239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c153ec0 0x7fb82c0a5970 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb82800d8c0 tx=0x7fb828008040 comp rx=0 tx=0).stop 2024-01-26T23:30:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.072+0000 7fb83373c700 1 -- 172.21.15.46:0/1870782239 shutdown_connections 2024-01-26T23:30:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.072+0000 7fb83373c700 1 --2- 172.21.15.46:0/1870782239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb810066f30 0x7fb8100693e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.072+0000 7fb83373c700 1 --2- 172.21.15.46:0/1870782239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb82c154290 0x7fb82c1582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.072+0000 7fb83373c700 1 --2- 172.21.15.46:0/1870782239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb82c0b2100 0x7fb82c153040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.072+0000 7fb83373c700 1 --2- 172.21.15.46:0/1870782239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c153ec0 0x7fb82c0a5970 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.072+0000 7fb83373c700 1 -- 172.21.15.46:0/1870782239 >> 172.21.15.46:0/1870782239 conn(0x7fb82c069510 msgr2=0x7fb82c06eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.072+0000 7fb83373c700 1 -- 172.21.15.46:0/1870782239 shutdown_connections 2024-01-26T23:30:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.072+0000 7fb83373c700 1 -- 172.21.15.46:0/1870782239 wait complete. 2024-01-26T23:30:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 9 2024-01-26T23:30:39.088 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.217+0000 7fbd79541700 1 Processor -- start 2024-01-26T23:30:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.218+0000 7fbd79541700 1 -- start start 2024-01-26T23:30:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.218+0000 7fbd79541700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740af850 0x7fbd74159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.218+0000 7fbd79541700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbd7415a000 0x7fbd740afc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.218+0000 7fbd79541700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd7415a470 con 0x7fbd740b0160 2024-01-26T23:30:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.218+0000 7fbd79541700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd741509d0 con 0x7fbd7415a000 2024-01-26T23:30:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.218+0000 7fbd79541700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd74157820 con 0x7fbd740af850 2024-01-26T23:30:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.219+0000 7fbd737fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbd740b0160 0x7fbd74156470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59138/0 (socket says 172.21.15.46:59138) 2024-01-26T23:30:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.219+0000 7fbd737fe700 1 -- 172.21.15.46:0/1798066425 learned_addr learned my addr 172.21.15.46:0/1798066425 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.219+0000 7fbd727fc700 1 --2- 172.21.15.46:0/1798066425 >> v2:172.21.15.106:3300/0 conn(0x7fbd7415a000 0x7fbd740afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.219+0000 7fbd72ffd700 1 --2- 172.21.15.46:0/1798066425 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740af850 0x7fbd74159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.219+0000 7fbd71ffb700 1 -- 172.21.15.46:0/1798066425 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619241563 0 0) 0x7fbd7415a470 con 0x7fbd740b0160 2024-01-26T23:30:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.219+0000 7fbd71ffb700 1 -- 172.21.15.46:0/1798066425 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd5c003570 con 0x7fbd740b0160 2024-01-26T23:30:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.219+0000 7fbd727fc700 1 -- 172.21.15.46:0/1798066425 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740af850 msgr2=0x7fbd74159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.219+0000 7fbd727fc700 1 --2- 172.21.15.46:0/1798066425 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740af850 0x7fbd74159ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.219+0000 7fbd727fc700 1 -- 172.21.15.46:0/1798066425 >> v1:172.21.15.46:6789/0 conn(0x7fbd740b0160 legacy=0x7fbd74156470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.219+0000 7fbd727fc700 1 -- 172.21.15.46:0/1798066425 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd740b05e0 con 0x7fbd7415a000 2024-01-26T23:30:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.219+0000 7fbd727fc700 1 --2- 172.21.15.46:0/1798066425 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd7415a000 0x7fbd740afc20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbd6800aa90 tx=0x7fbd6800a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6a9995556807fe5 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.220+0000 7fbd71ffb700 1 -- 172.21.15.46:0/1798066425 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6800ae70 con 0x7fbd7415a000 2024-01-26T23:30:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.220+0000 7fbd71ffb700 1 -- 172.21.15.46:0/1798066425 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6800c970 con 0x7fbd7415a000 2024-01-26T23:30:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.220+0000 7fbd71ffb700 1 -- 172.21.15.46:0/1798066425 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6800cd10 con 0x7fbd7415a000 2024-01-26T23:30:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.220+0000 7fbd79541700 1 -- 172.21.15.46:0/1798066425 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd7415a000 msgr2=0x7fbd740afc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.220+0000 7fbd79541700 1 --2- 172.21.15.46:0/1798066425 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd7415a000 0x7fbd740afc20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbd6800aa90 tx=0x7fbd6800a330 comp rx=0 tx=0).stop 2024-01-26T23:30:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.220+0000 7fbd79541700 1 -- 172.21.15.46:0/1798066425 shutdown_connections 2024-01-26T23:30:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.220+0000 7fbd79541700 1 --2- 172.21.15.46:0/1798066425 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740af850 0x7fbd74159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.220+0000 7fbd79541700 1 --2- 172.21.15.46:0/1798066425 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd7415a000 0x7fbd740afc20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.220+0000 7fbd79541700 1 -- 172.21.15.46:0/1798066425 >> 172.21.15.46:0/1798066425 conn(0x7fbd74068400 msgr2=0x7fbd740cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.220+0000 7fbd79541700 1 -- 172.21.15.46:0/1798066425 shutdown_connections 2024-01-26T23:30:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.220+0000 7fbd79541700 1 -- 172.21.15.46:0/1798066425 wait complete. 2024-01-26T23:30:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.222+0000 7fbd79541700 1 Processor -- start 2024-01-26T23:30:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.222+0000 7fbd79541700 1 -- start start 2024-01-26T23:30:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.222+0000 7fbd79541700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd740af850 0x7fbd74150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.222+0000 7fbd79541700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740b0160 0x7fbd7414f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.222+0000 7fbd79541700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd7415a000 0x7fbd7413eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.222+0000 7fbd79541700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd740c9b00 con 0x7fbd7415a000 2024-01-26T23:30:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.222+0000 7fbd79541700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd74151470 con 0x7fbd740af850 2024-01-26T23:30:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.222+0000 7fbd79541700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd74151750 con 0x7fbd740b0160 2024-01-26T23:30:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.223+0000 7fbd737fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd7415a000 0x7fbd7413eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.223+0000 7fbd737fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd7415a000 0x7fbd7413eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60970/0 (socket says 172.21.15.46:60970) 2024-01-26T23:30:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.223+0000 7fbd737fe700 1 -- 172.21.15.46:0/2298635910 learned_addr learned my addr 172.21.15.46:0/2298635910 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.223+0000 7fbd72ffd700 1 --2- 172.21.15.46:0/2298635910 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd740af850 0x7fbd74150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.223+0000 7fbd727fc700 1 --2- 172.21.15.46:0/2298635910 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740b0160 0x7fbd7414f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.223+0000 7fbd737fe700 1 -- 172.21.15.46:0/2298635910 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740b0160 msgr2=0x7fbd7414f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.223+0000 7fbd737fe700 1 --2- 172.21.15.46:0/2298635910 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740b0160 0x7fbd7414f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.223+0000 7fbd737fe700 1 -- 172.21.15.46:0/2298635910 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd740af850 msgr2=0x7fbd74150b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.223+0000 7fbd737fe700 1 --2- 172.21.15.46:0/2298635910 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd740af850 0x7fbd74150b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.223+0000 7fbd737fe700 1 -- 172.21.15.46:0/2298635910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6800a5d0 con 0x7fbd7415a000 2024-01-26T23:30:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.223+0000 7fbd72ffd700 1 --2- 172.21.15.46:0/2298635910 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd740af850 0x7fbd74150b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.223+0000 7fbd737fe700 1 --2- 172.21.15.46:0/2298635910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd7415a000 0x7fbd7413eb80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c001460 tx=0x7fbd6c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.223+0000 7fbd5affd700 1 -- 172.21.15.46:0/2298635910 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c00a950 con 0x7fbd7415a000 2024-01-26T23:30:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.223+0000 7fbd5affd700 1 -- 172.21.15.46:0/2298635910 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6c00aab0 con 0x7fbd7415a000 2024-01-26T23:30:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.223+0000 7fbd79541700 1 -- 172.21.15.46:0/2298635910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd74151a30 con 0x7fbd7415a000 2024-01-26T23:30:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.223+0000 7fbd5affd700 1 -- 172.21.15.46:0/2298635910 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c00ae30 con 0x7fbd7415a000 2024-01-26T23:30:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.223+0000 7fbd79541700 1 -- 172.21.15.46:0/2298635910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd741510b0 con 0x7fbd7415a000 2024-01-26T23:30:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.224+0000 7fbd79541700 1 -- 172.21.15.46:0/2298635910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd7404c710 con 0x7fbd7415a000 2024-01-26T23:30:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.225+0000 7fbd5affd700 1 -- 172.21.15.46:0/2298635910 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbd6c007300 con 0x7fbd7415a000 2024-01-26T23:30:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.225+0000 7fbd5affd700 1 --2- 172.21.15.46:0/2298635910 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbd60066f30 0x7fbd600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.225+0000 7fbd5affd700 1 -- 172.21.15.46:0/2298635910 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7fbd6c084da0 con 0x7fbd7415a000 2024-01-26T23:30:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.226+0000 7fbd72ffd700 1 --2- 172.21.15.46:0/2298635910 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbd60066f30 0x7fbd600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.226+0000 7fbd72ffd700 1 --2- 172.21.15.46:0/2298635910 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbd60066f30 0x7fbd600693e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbd64009ce0 tx=0x7fbd64008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.230+0000 7fbd5affd700 1 -- 172.21.15.46:0/2298635910 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbd6c051c30 con 0x7fbd7415a000 2024-01-26T23:30:39.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.436+0000 7fbd79541700 1 -- 172.21.15.46:0/2298635910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd740c8810 con 0x7fbd7415a000 2024-01-26T23:30:39.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.437+0000 7fbd5affd700 1 -- 172.21.15.46:0/2298635910 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17518 (secure 0 0 0) 0x7fbd6c055250 con 0x7fbd7415a000 2024-01-26T23:30:39.438 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:39.439 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":26,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:30:33.471772+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:39.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.439+0000 7fbd79541700 1 -- 172.21.15.46:0/2298635910 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbd60066f30 msgr2=0x7fbd600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.439+0000 7fbd79541700 1 --2- 172.21.15.46:0/2298635910 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbd60066f30 0x7fbd600693e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbd64009ce0 tx=0x7fbd64008040 comp rx=0 tx=0).stop 2024-01-26T23:30:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.439+0000 7fbd79541700 1 -- 172.21.15.46:0/2298635910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd7415a000 msgr2=0x7fbd7413eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.439+0000 7fbd79541700 1 --2- 172.21.15.46:0/2298635910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd7415a000 0x7fbd7413eb80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c001460 tx=0x7fbd6c001a40 comp rx=0 tx=0).stop 2024-01-26T23:30:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.439+0000 7fbd79541700 1 -- 172.21.15.46:0/2298635910 shutdown_connections 2024-01-26T23:30:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.439+0000 7fbd79541700 1 --2- 172.21.15.46:0/2298635910 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbd60066f30 0x7fbd600693e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.439+0000 7fbd79541700 1 --2- 172.21.15.46:0/2298635910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd7415a000 0x7fbd7413eb80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.439+0000 7fbd79541700 1 --2- 172.21.15.46:0/2298635910 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd740af850 0x7fbd74150b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.439+0000 7fbd79541700 1 --2- 172.21.15.46:0/2298635910 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd740b0160 0x7fbd7414f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.439+0000 7fbd79541700 1 -- 172.21.15.46:0/2298635910 >> 172.21.15.46:0/2298635910 conn(0x7fbd74068400 msgr2=0x7fbd741550a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.439+0000 7fbd79541700 1 -- 172.21.15.46:0/2298635910 shutdown_connections 2024-01-26T23:30:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.439+0000 7fbd79541700 1 -- 172.21.15.46:0/2298635910 wait complete. 2024-01-26T23:30:39.455 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-26T23:30:39.455 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:30:39.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.586+0000 7f340d145700 1 Processor -- start 2024-01-26T23:30:39.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.586+0000 7f340d145700 1 -- start start 2024-01-26T23:30:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.587+0000 7f340d145700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f34081561d0 0x7f34080ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.587+0000 7f340d145700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34080b0760 con 0x7f340806cbb0 2024-01-26T23:30:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.587+0000 7f340d145700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34080b0a40 con 0x7f34080b0280 2024-01-26T23:30:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.587+0000 7f340d145700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34080b0d20 con 0x7f34081561d0 2024-01-26T23:30:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.587+0000 7f340759e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f340806cbb0 0x7f34080b1800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59142/0 (socket says 172.21.15.46:59142) 2024-01-26T23:30:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.587+0000 7f340759e700 1 -- 172.21.15.46:0/2976350345 learned_addr learned my addr 172.21.15.46:0/2976350345 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.587+0000 7f3406d9d700 1 --2- 172.21.15.46:0/2976350345 >> v2:172.21.15.106:3300/0 conn(0x7f34081561d0 0x7f34080ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.588+0000 7f3405d9b700 1 -- 172.21.15.46:0/2976350345 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796164187 0 0) 0x7f34080b0760 con 0x7f340806cbb0 2024-01-26T23:30:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.588+0000 7f3405d9b700 1 -- 172.21.15.46:0/2976350345 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f0003570 con 0x7f340806cbb0 2024-01-26T23:30:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.588+0000 7f3406d9d700 1 -- 172.21.15.46:0/2976350345 >> v1:172.21.15.106:6789/0 conn(0x7f34080b0280 legacy=0x7f340806c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.588+0000 7f3405d9b700 1 -- 172.21.15.46:0/2976350345 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79490958 0 0) 0x7f34080b0a40 con 0x7f34080b0280 2024-01-26T23:30:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.588+0000 7f3406d9d700 1 -- 172.21.15.46:0/2976350345 >> v1:172.21.15.46:6789/0 conn(0x7f340806cbb0 legacy=0x7f34080b1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.588+0000 7f3406d9d700 1 -- 172.21.15.46:0/2976350345 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34080b1000 con 0x7f34081561d0 2024-01-26T23:30:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.588+0000 7f3406d9d700 1 --2- 172.21.15.46:0/2976350345 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34081561d0 0x7f34080ca5f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f33f80098e0 tx=0x7f33f80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdcdb573a1539f6a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.588+0000 7f3405d9b700 1 -- 172.21.15.46:0/2976350345 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f800b640 con 0x7f34081561d0 2024-01-26T23:30:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.588+0000 7f3405d9b700 1 -- 172.21.15.46:0/2976350345 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f800b7a0 con 0x7f34081561d0 2024-01-26T23:30:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.588+0000 7f3405d9b700 1 -- 172.21.15.46:0/2976350345 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f800bb40 con 0x7f34081561d0 2024-01-26T23:30:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.589+0000 7f340d145700 1 -- 172.21.15.46:0/2976350345 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34081561d0 msgr2=0x7f34080ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.589+0000 7f340d145700 1 --2- 172.21.15.46:0/2976350345 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34081561d0 0x7f34080ca5f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f33f80098e0 tx=0x7f33f80091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.589+0000 7f340d145700 1 -- 172.21.15.46:0/2976350345 shutdown_connections 2024-01-26T23:30:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.589+0000 7f340d145700 1 --2- 172.21.15.46:0/2976350345 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34081561d0 0x7f34080ca5f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.589+0000 7f340d145700 1 -- 172.21.15.46:0/2976350345 >> 172.21.15.46:0/2976350345 conn(0x7f340806b630 msgr2=0x7f34080aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.589+0000 7f340d145700 1 -- 172.21.15.46:0/2976350345 shutdown_connections 2024-01-26T23:30:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.589+0000 7f340d145700 1 -- 172.21.15.46:0/2976350345 wait complete. 2024-01-26T23:30:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.591+0000 7f340d145700 1 Processor -- start 2024-01-26T23:30:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.591+0000 7f340d145700 1 -- start start 2024-01-26T23:30:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f340d145700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f340806cbb0 0x7f34080c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f340d145700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34080b0280 0x7f3408159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f340d145700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34081561d0 0x7f34080adc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f340d145700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340815a7e0 con 0x7f340806cbb0 2024-01-26T23:30:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f340d145700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3408159bd0 con 0x7f34080b0280 2024-01-26T23:30:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f340d145700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3408159eb0 con 0x7f34081561d0 2024-01-26T23:30:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f3406d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f340806cbb0 0x7f34080c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f3406d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f340806cbb0 0x7f34080c8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60986/0 (socket says 172.21.15.46:60986) 2024-01-26T23:30:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f3406d9d700 1 -- 172.21.15.46:0/760597323 learned_addr learned my addr 172.21.15.46:0/760597323 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f340759e700 1 --2- 172.21.15.46:0/760597323 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34081561d0 0x7f34080adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f340659c700 1 --2- 172.21.15.46:0/760597323 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34080b0280 0x7f3408159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f3406d9d700 1 -- 172.21.15.46:0/760597323 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34081561d0 msgr2=0x7f34080adc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f3406d9d700 1 --2- 172.21.15.46:0/760597323 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34081561d0 0x7f34080adc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f3406d9d700 1 -- 172.21.15.46:0/760597323 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34080b0280 msgr2=0x7f3408159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f3406d9d700 1 --2- 172.21.15.46:0/760597323 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34080b0280 0x7f3408159130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f3406d9d700 1 -- 172.21.15.46:0/760597323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f8009420 con 0x7f340806cbb0 2024-01-26T23:30:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f340659c700 1 --2- 172.21.15.46:0/760597323 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34080b0280 0x7f3408159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:30:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.592+0000 7f3406d9d700 1 --2- 172.21.15.46:0/760597323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f340806cbb0 0x7f34080c8b10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f33f80093f0 tx=0x7f33f8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.593+0000 7f33eeffd700 1 -- 172.21.15.46:0/760597323 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f801f070 con 0x7f340806cbb0 2024-01-26T23:30:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.593+0000 7f33eeffd700 1 -- 172.21.15.46:0/760597323 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f8000bc0 con 0x7f340806cbb0 2024-01-26T23:30:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.593+0000 7f340d145700 1 -- 172.21.15.46:0/760597323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f340815a190 con 0x7f340806cbb0 2024-01-26T23:30:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.593+0000 7f33eeffd700 1 -- 172.21.15.46:0/760597323 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f800bb40 con 0x7f340806cbb0 2024-01-26T23:30:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.593+0000 7f340d145700 1 -- 172.21.15.46:0/760597323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34080c9050 con 0x7f340806cbb0 2024-01-26T23:30:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.594+0000 7f340d145700 1 -- 172.21.15.46:0/760597323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f340804c710 con 0x7f340806cbb0 2024-01-26T23:30:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.594+0000 7f33eeffd700 1 -- 172.21.15.46:0/760597323 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f33f8023070 con 0x7f340806cbb0 2024-01-26T23:30:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.595+0000 7f33eeffd700 1 --2- 172.21.15.46:0/760597323 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f33f4066f30 0x7f33f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.595+0000 7f33eeffd700 1 -- 172.21.15.46:0/760597323 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f33f8085ad0 con 0x7f340806cbb0 2024-01-26T23:30:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.595+0000 7f340659c700 1 --2- 172.21.15.46:0/760597323 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f33f4066f30 0x7f33f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.595+0000 7f340659c700 1 --2- 172.21.15.46:0/760597323 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f33f4066f30 0x7f33f40693e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f33fc0011e0 tx=0x7f33fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.599+0000 7f33eeffd700 1 -- 172.21.15.46:0/760597323 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f33f8052a30 con 0x7f340806cbb0 2024-01-26T23:30:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.850+0000 7f340d145700 1 -- 172.21.15.46:0/760597323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34080b12e0 con 0x7f340806cbb0 2024-01-26T23:30:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.851+0000 7f33eeffd700 1 -- 172.21.15.46:0/760597323 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4593 (secure 0 0 0) 0x7f33f8015090 con 0x7f340806cbb0 2024-01-26T23:30:39.853 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:39.853 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3409861270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3409861270},{"type":"v1","addr":"172.21.15.46:6833","nonce":3409861270}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3319220815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3319220815},{"type":"v1","addr":"172.21.15.106:6836","nonce":3319220815}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4252,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/529198432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":529198432},{"type":"v1","addr":"172.21.15.106:6837","nonce":529198432}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:29:44.031963+0000","modified":"2024-01-26T23:30:03.652307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/688086235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":688086235},{"type":"v1","addr":"172.21.15.46:6835","nonce":688086235}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2024-01-26T23:30:39.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.853+0000 7f340d145700 1 -- 172.21.15.46:0/760597323 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f33f4066f30 msgr2=0x7f33f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:39.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.853+0000 7f340d145700 1 --2- 172.21.15.46:0/760597323 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f33f4066f30 0x7f33f40693e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f33fc0011e0 tx=0x7f33fc009040 comp rx=0 tx=0).stop 2024-01-26T23:30:39.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.853+0000 7f340d145700 1 -- 172.21.15.46:0/760597323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f340806cbb0 msgr2=0x7f34080c8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:39.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.853+0000 7f340d145700 1 --2- 172.21.15.46:0/760597323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f340806cbb0 0x7f34080c8b10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f33f80093f0 tx=0x7f33f8017000 comp rx=0 tx=0).stop 2024-01-26T23:30:39.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.853+0000 7f340d145700 1 -- 172.21.15.46:0/760597323 shutdown_connections 2024-01-26T23:30:39.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.853+0000 7f340d145700 1 --2- 172.21.15.46:0/760597323 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f33f4066f30 0x7f33f40693e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.853+0000 7f340d145700 1 --2- 172.21.15.46:0/760597323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f340806cbb0 0x7f34080c8b10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.853+0000 7f340d145700 1 --2- 172.21.15.46:0/760597323 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34081561d0 0x7f34080adc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.853+0000 7f340d145700 1 --2- 172.21.15.46:0/760597323 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34080b0280 0x7f3408159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:39.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.853+0000 7f340d145700 1 -- 172.21.15.46:0/760597323 >> 172.21.15.46:0/760597323 conn(0x7f340806b630 msgr2=0x7f3408154da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:39.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.853+0000 7f340d145700 1 -- 172.21.15.46:0/760597323 shutdown_connections 2024-01-26T23:30:39.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.853+0000 7f340d145700 1 -- 172.21.15.46:0/760597323 wait complete. 2024-01-26T23:30:39.856 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 9 2024-01-26T23:30:39.869 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:30:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.998+0000 7f48b5cff700 1 Processor -- start 2024-01-26T23:30:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.998+0000 7f48b5cff700 1 -- start start 2024-01-26T23:30:40.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.999+0000 7f48b5cff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48b00b2b40 0x7f48b00ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:40.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.999+0000 7f48b5cff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48b00aedd0 0x7f48b00cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:40.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.999+0000 7f48b5cff700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b015a0b0 con 0x7f48b00ae960 2024-01-26T23:30:40.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.999+0000 7f48b5cff700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b0099fc0 con 0x7f48b00aedd0 2024-01-26T23:30:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.999+0000 7f48b5cff700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00ae4d0 con 0x7f48b00b2b40 2024-01-26T23:30:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.999+0000 7f48affff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48b00aedd0 0x7f48b00cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.999+0000 7f48aeffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f48b00ae960 0x7f48b00ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59148/0 (socket says 172.21.15.46:59148) 2024-01-26T23:30:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.999+0000 7f48aeffd700 1 -- 172.21.15.46:0/594668423 learned_addr learned my addr 172.21.15.46:0/594668423 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.999+0000 7f48af7fe700 1 --2- 172.21.15.46:0/594668423 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48b00b2b40 0x7f48b00ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.999+0000 7f48ae7fc700 1 -- 172.21.15.46:0/594668423 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182639482 0 0) 0x7f48b015a0b0 con 0x7f48b00ae960 2024-01-26T23:30:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.999+0000 7f48ae7fc700 1 -- 172.21.15.46:0/594668423 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4898003570 con 0x7f48b00ae960 2024-01-26T23:30:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.999+0000 7f48affff700 1 -- 172.21.15.46:0/594668423 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48b00b2b40 msgr2=0x7f48b00ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.999+0000 7f48affff700 1 --2- 172.21.15.46:0/594668423 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48b00b2b40 0x7f48b00ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.999+0000 7f48affff700 1 -- 172.21.15.46:0/594668423 >> v1:172.21.15.46:6789/0 conn(0x7f48b00ae960 legacy=0x7f48b00ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:39.999+0000 7f48affff700 1 -- 172.21.15.46:0/594668423 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b00ad8c0 con 0x7f48b00aedd0 2024-01-26T23:30:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.000+0000 7f48affff700 1 --2- 172.21.15.46:0/594668423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b00aedd0 0x7f48b00cb5e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f48a8009e60 tx=0x7f48a8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce286f0dc887a801 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.000+0000 7f48ae7fc700 1 -- 172.21.15.46:0/594668423 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a800e070 con 0x7f48b00aedd0 2024-01-26T23:30:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.000+0000 7f48ae7fc700 1 -- 172.21.15.46:0/594668423 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a800bab0 con 0x7f48b00aedd0 2024-01-26T23:30:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.000+0000 7f48ae7fc700 1 -- 172.21.15.46:0/594668423 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a800be70 con 0x7f48b00aedd0 2024-01-26T23:30:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.000+0000 7f48b5cff700 1 -- 172.21.15.46:0/594668423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b00aedd0 msgr2=0x7f48b00cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.000+0000 7f48b5cff700 1 --2- 172.21.15.46:0/594668423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b00aedd0 0x7f48b00cb5e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f48a8009e60 tx=0x7f48a8009780 comp rx=0 tx=0).stop 2024-01-26T23:30:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.000+0000 7f48b5cff700 1 -- 172.21.15.46:0/594668423 shutdown_connections 2024-01-26T23:30:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.000+0000 7f48b5cff700 1 --2- 172.21.15.46:0/594668423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b00aedd0 0x7f48b00cb5e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.000+0000 7f48b5cff700 1 --2- 172.21.15.46:0/594668423 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48b00b2b40 0x7f48b00ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.000+0000 7f48b5cff700 1 -- 172.21.15.46:0/594668423 >> 172.21.15.46:0/594668423 conn(0x7f48b0099a30 msgr2=0x7f48b006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.001+0000 7f48b5cff700 1 -- 172.21.15.46:0/594668423 shutdown_connections 2024-01-26T23:30:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.001+0000 7f48b5cff700 1 -- 172.21.15.46:0/594668423 wait complete. 2024-01-26T23:30:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.002+0000 7f48b5cff700 1 Processor -- start 2024-01-26T23:30:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.002+0000 7f48b5cff700 1 -- start start 2024-01-26T23:30:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.003+0000 7f48b5cff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48b00ae960 0x7f48b00b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.003+0000 7f48b5cff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48b00b2b40 0x7f48b009d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.003+0000 7f48b5cff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b00b5330 0x7f48b009d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.003+0000 7f48b5cff700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00b40a0 con 0x7f48b00b5330 2024-01-26T23:30:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.003+0000 7f48b5cff700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b009de50 con 0x7f48b00b2b40 2024-01-26T23:30:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.003+0000 7f48b5cff700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00b4ea0 con 0x7f48b00ae960 2024-01-26T23:30:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.003+0000 7f48affff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b00b5330 0x7f48b009d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.003+0000 7f48affff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b00b5330 0x7f48b009d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32768/0 (socket says 172.21.15.46:32768) 2024-01-26T23:30:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.003+0000 7f48affff700 1 -- 172.21.15.46:0/1153753039 learned_addr learned my addr 172.21.15.46:0/1153753039 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.003+0000 7f48aeffd700 1 --2- 172.21.15.46:0/1153753039 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48b00b2b40 0x7f48b009d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.003+0000 7f48af7fe700 1 --2- 172.21.15.46:0/1153753039 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48b00ae960 0x7f48b00b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.003+0000 7f48affff700 1 -- 172.21.15.46:0/1153753039 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48b00ae960 msgr2=0x7f48b00b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.003+0000 7f48affff700 1 --2- 172.21.15.46:0/1153753039 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48b00ae960 0x7f48b00b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.003+0000 7f48affff700 1 -- 172.21.15.46:0/1153753039 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48b00b2b40 msgr2=0x7f48b009d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.003+0000 7f48affff700 1 --2- 172.21.15.46:0/1153753039 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48b00b2b40 0x7f48b009d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.003+0000 7f48affff700 1 -- 172.21.15.46:0/1153753039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a8009a20 con 0x7f48b00b5330 2024-01-26T23:30:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.003+0000 7f48affff700 1 --2- 172.21.15.46:0/1153753039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b00b5330 0x7f48b009d910 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f48b0088e10 tx=0x7f48a8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.004+0000 7f48977fe700 1 -- 172.21.15.46:0/1153753039 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a8012dd0 con 0x7f48b00b5330 2024-01-26T23:30:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.004+0000 7f48977fe700 1 -- 172.21.15.46:0/1153753039 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a8027470 con 0x7f48b00b5330 2024-01-26T23:30:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.004+0000 7f48b5cff700 1 -- 172.21.15.46:0/1153753039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b00b4290 con 0x7f48b00b5330 2024-01-26T23:30:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.004+0000 7f48977fe700 1 -- 172.21.15.46:0/1153753039 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a8027830 con 0x7f48b00b5330 2024-01-26T23:30:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.004+0000 7f48b5cff700 1 -- 172.21.15.46:0/1153753039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b00b4850 con 0x7f48b00b5330 2024-01-26T23:30:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.005+0000 7f48b5cff700 1 -- 172.21.15.46:0/1153753039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b004c710 con 0x7f48b00b5330 2024-01-26T23:30:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.005+0000 7f48977fe700 1 -- 172.21.15.46:0/1153753039 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f48a8024070 con 0x7f48b00b5330 2024-01-26T23:30:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.006+0000 7f48977fe700 1 --2- 172.21.15.46:0/1153753039 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f489c066f30 0x7f489c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.006+0000 7f48977fe700 1 -- 172.21.15.46:0/1153753039 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f48a8087200 con 0x7f48b00b5330 2024-01-26T23:30:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.006+0000 7f48af7fe700 1 --2- 172.21.15.46:0/1153753039 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f489c066f30 0x7f489c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.006+0000 7f48af7fe700 1 --2- 172.21.15.46:0/1153753039 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f489c066f30 0x7f489c0693e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f48a0009da0 tx=0x7f48a0009120 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.010+0000 7f48977fe700 1 -- 172.21.15.46:0/1153753039 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f48a8054160 con 0x7f48b00b5330 2024-01-26T23:30:40.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.267+0000 7f48b5cff700 1 -- 172.21.15.46:0/1153753039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48b00682c0 con 0x7f48b00b5330 2024-01-26T23:30:40.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.268+0000 7f48977fe700 1 -- 172.21.15.46:0/1153753039 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4593 (secure 0 0 0) 0x7f48a8057780 con 0x7f48b00b5330 2024-01-26T23:30:40.270 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:40.270 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3409861270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3409861270},{"type":"v1","addr":"172.21.15.46:6833","nonce":3409861270}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3319220815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3319220815},{"type":"v1","addr":"172.21.15.106:6836","nonce":3319220815}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4252,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/529198432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":529198432},{"type":"v1","addr":"172.21.15.106:6837","nonce":529198432}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:29:44.031963+0000","modified":"2024-01-26T23:30:03.652307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/688086235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":688086235},{"type":"v1","addr":"172.21.15.46:6835","nonce":688086235}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2024-01-26T23:30:40.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.270+0000 7f48b5cff700 1 -- 172.21.15.46:0/1153753039 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f489c066f30 msgr2=0x7f489c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:40.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.270+0000 7f48b5cff700 1 --2- 172.21.15.46:0/1153753039 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f489c066f30 0x7f489c0693e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f48a0009da0 tx=0x7f48a0009120 comp rx=0 tx=0).stop 2024-01-26T23:30:40.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.270+0000 7f48b5cff700 1 -- 172.21.15.46:0/1153753039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b00b5330 msgr2=0x7f48b009d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:40.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.270+0000 7f48b5cff700 1 --2- 172.21.15.46:0/1153753039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b00b5330 0x7f48b009d910 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f48b0088e10 tx=0x7f48a8019000 comp rx=0 tx=0).stop 2024-01-26T23:30:40.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.270+0000 7f48b5cff700 1 -- 172.21.15.46:0/1153753039 shutdown_connections 2024-01-26T23:30:40.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.270+0000 7f48b5cff700 1 --2- 172.21.15.46:0/1153753039 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f489c066f30 0x7f489c0693e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.270+0000 7f48b5cff700 1 --2- 172.21.15.46:0/1153753039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b00b5330 0x7f48b009d910 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.270+0000 7f48b5cff700 1 --2- 172.21.15.46:0/1153753039 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48b00ae960 0x7f48b00b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.270+0000 7f48b5cff700 1 --2- 172.21.15.46:0/1153753039 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48b00b2b40 0x7f48b009d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.270+0000 7f48b5cff700 1 -- 172.21.15.46:0/1153753039 >> 172.21.15.46:0/1153753039 conn(0x7f48b0099a30 msgr2=0x7f48b0159140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:40.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.270+0000 7f48b5cff700 1 -- 172.21.15.46:0/1153753039 shutdown_connections 2024-01-26T23:30:40.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.270+0000 7f48b5cff700 1 -- 172.21.15.46:0/1153753039 wait complete. 2024-01-26T23:30:40.273 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 9 2024-01-26T23:30:40.286 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:40.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.415+0000 7f2fbb9c9700 1 Processor -- start 2024-01-26T23:30:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.416+0000 7f2fbb9c9700 1 -- start start 2024-01-26T23:30:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.416+0000 7f2fbb9c9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fb406cbb0 0x7f2fb40ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.416+0000 7f2fbb9c9700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb40b0760 con 0x7f2fb41561d0 2024-01-26T23:30:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.416+0000 7f2fbb9c9700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb40b0a40 con 0x7f2fb40b0280 2024-01-26T23:30:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.417+0000 7f2fbb9c9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb40b0d20 con 0x7f2fb406cbb0 2024-01-26T23:30:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.417+0000 7f2fb9765700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2fb41561d0 0x7f2fb406c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54234/0 (socket says 172.21.15.46:54234) 2024-01-26T23:30:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.417+0000 7f2fb9765700 1 -- 172.21.15.46:0/1390536882 learned_addr learned my addr 172.21.15.46:0/1390536882 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.417+0000 7f2fb8f64700 1 --2- 172.21.15.46:0/1390536882 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fb406cbb0 0x7f2fb40ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.417+0000 7f2fabfff700 1 -- 172.21.15.46:0/1390536882 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026330893 0 0) 0x7f2fb40b0760 con 0x7f2fb41561d0 2024-01-26T23:30:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.417+0000 7f2fabfff700 1 -- 172.21.15.46:0/1390536882 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa0003570 con 0x7f2fb41561d0 2024-01-26T23:30:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.417+0000 7f2fabfff700 1 -- 172.21.15.46:0/1390536882 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346521065 0 0) 0x7f2fb40b0a40 con 0x7f2fb40b0280 2024-01-26T23:30:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.417+0000 7f2fabfff700 1 -- 172.21.15.46:0/1390536882 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb40b0760 con 0x7f2fb40b0280 2024-01-26T23:30:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.417+0000 7f2fb8f64700 1 -- 172.21.15.46:0/1390536882 >> v1:172.21.15.106:6789/0 conn(0x7f2fb40b0280 legacy=0x7f2fb40b1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.417+0000 7f2fb8f64700 1 -- 172.21.15.46:0/1390536882 >> v1:172.21.15.46:6789/0 conn(0x7f2fb41561d0 legacy=0x7f2fb406c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.417+0000 7f2fb8f64700 1 -- 172.21.15.46:0/1390536882 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb40b1000 con 0x7f2fb406cbb0 2024-01-26T23:30:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.417+0000 7f2fb8f64700 1 --2- 172.21.15.46:0/1390536882 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fb406cbb0 0x7f2fb40ca5f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009910 tx=0x7f2fb00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4761c18ba84da4a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.418+0000 7f2fabfff700 1 -- 172.21.15.46:0/1390536882 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0011630 con 0x7f2fb406cbb0 2024-01-26T23:30:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.418+0000 7f2fabfff700 1 -- 172.21.15.46:0/1390536882 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb0011790 con 0x7f2fb406cbb0 2024-01-26T23:30:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.418+0000 7f2fabfff700 1 -- 172.21.15.46:0/1390536882 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0011b30 con 0x7f2fb406cbb0 2024-01-26T23:30:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.418+0000 7f2fbb9c9700 1 -- 172.21.15.46:0/1390536882 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fb406cbb0 msgr2=0x7f2fb40ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.418+0000 7f2fbb9c9700 1 --2- 172.21.15.46:0/1390536882 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fb406cbb0 0x7f2fb40ca5f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009910 tx=0x7f2fb00091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.418+0000 7f2fbb9c9700 1 -- 172.21.15.46:0/1390536882 shutdown_connections 2024-01-26T23:30:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.418+0000 7f2fbb9c9700 1 --2- 172.21.15.46:0/1390536882 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fb406cbb0 0x7f2fb40ca5f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.418+0000 7f2fbb9c9700 1 -- 172.21.15.46:0/1390536882 >> 172.21.15.46:0/1390536882 conn(0x7f2fb406b630 msgr2=0x7f2fb40aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.418+0000 7f2fbb9c9700 1 -- 172.21.15.46:0/1390536882 shutdown_connections 2024-01-26T23:30:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.418+0000 7f2fbb9c9700 1 -- 172.21.15.46:0/1390536882 wait complete. 2024-01-26T23:30:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.420+0000 7f2fbb9c9700 1 Processor -- start 2024-01-26T23:30:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.420+0000 7f2fbb9c9700 1 -- start start 2024-01-26T23:30:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.421+0000 7f2fbb9c9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fb406cbb0 0x7f2fb414c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.421+0000 7f2fbb9c9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fb40b0280 0x7f2fb414ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.421+0000 7f2fbb9c9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fb41561d0 0x7f2fb414cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.421+0000 7f2fbb9c9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb40ae610 con 0x7f2fb40b0280 2024-01-26T23:30:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.421+0000 7f2fbb9c9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb414dac0 con 0x7f2fb41561d0 2024-01-26T23:30:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.421+0000 7f2fbb9c9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb414cbf0 con 0x7f2fb406cbb0 2024-01-26T23:30:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.421+0000 7f2fb8f64700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fb40b0280 0x7f2fb414ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.421+0000 7f2fb9765700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fb406cbb0 0x7f2fb414c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.421+0000 7f2fb8f64700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fb40b0280 0x7f2fb414ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47710/0 (socket says 172.21.15.46:47710) 2024-01-26T23:30:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.421+0000 7f2fb8f64700 1 -- 172.21.15.46:0/1750626874 learned_addr learned my addr 172.21.15.46:0/1750626874 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.422+0000 7f2fb9f66700 1 --2- 172.21.15.46:0/1750626874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fb41561d0 0x7f2fb414cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.422+0000 7f2fb8f64700 1 -- 172.21.15.46:0/1750626874 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fb406cbb0 msgr2=0x7f2fb414c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.422+0000 7f2fb8f64700 1 --2- 172.21.15.46:0/1750626874 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fb406cbb0 0x7f2fb414c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.422+0000 7f2fb8f64700 1 -- 172.21.15.46:0/1750626874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fb41561d0 msgr2=0x7f2fb414cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.422+0000 7f2fb8f64700 1 --2- 172.21.15.46:0/1750626874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fb41561d0 0x7f2fb414cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.422+0000 7f2fb8f64700 1 -- 172.21.15.46:0/1750626874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb0009420 con 0x7f2fb40b0280 2024-01-26T23:30:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.422+0000 7f2fb8f64700 1 --2- 172.21.15.46:0/1750626874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fb40b0280 0x7f2fb414ae80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009f00 tx=0x7f2fb0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.422+0000 7f2fa97fa700 1 -- 172.21.15.46:0/1750626874 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb001f070 con 0x7f2fb40b0280 2024-01-26T23:30:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.422+0000 7f2fa97fa700 1 -- 172.21.15.46:0/1750626874 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb0002f70 con 0x7f2fb40b0280 2024-01-26T23:30:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.422+0000 7f2fbb9c9700 1 -- 172.21.15.46:0/1750626874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb414b8f0 con 0x7f2fb40b0280 2024-01-26T23:30:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.422+0000 7f2fa97fa700 1 -- 172.21.15.46:0/1750626874 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0006af0 con 0x7f2fb40b0280 2024-01-26T23:30:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.422+0000 7f2fbb9c9700 1 -- 172.21.15.46:0/1750626874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb414beb0 con 0x7f2fb40b0280 2024-01-26T23:30:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.423+0000 7f2fbb9c9700 1 -- 172.21.15.46:0/1750626874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb404c710 con 0x7f2fb40b0280 2024-01-26T23:30:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.424+0000 7f2fa97fa700 1 -- 172.21.15.46:0/1750626874 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2fb0023070 con 0x7f2fb40b0280 2024-01-26T23:30:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.424+0000 7f2fa97fa700 1 --2- 172.21.15.46:0/1750626874 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2f98066f30 0x7f2f980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.424+0000 7f2fa97fa700 1 -- 172.21.15.46:0/1750626874 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f2fb0085a70 con 0x7f2fb40b0280 2024-01-26T23:30:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.425+0000 7f2fb9765700 1 --2- 172.21.15.46:0/1750626874 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2f98066f30 0x7f2f980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.425+0000 7f2fb9765700 1 --2- 172.21.15.46:0/1750626874 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2f98066f30 0x7f2f980693e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4007fd0 tx=0x7f2fa4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:40.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.429+0000 7f2fa97fa700 1 -- 172.21.15.46:0/1750626874 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2fb0052920 con 0x7f2fb40b0280 2024-01-26T23:30:40.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.641+0000 7f2fbb9c9700 1 -- 172.21.15.46:0/1750626874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fb40b12e0 con 0x7f2fb40b0280 2024-01-26T23:30:40.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.642+0000 7f2fa97fa700 1 -- 172.21.15.46:0/1750626874 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17518 (secure 0 0 0) 0x7f2fb0015070 con 0x7f2fb40b0280 2024-01-26T23:30:40.644 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:40.644 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":26,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:30:33.471772+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:29:42.539759+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-26T23:29:43.002669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-26T23:29:46.646864+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:40.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.644+0000 7f2fbb9c9700 1 -- 172.21.15.46:0/1750626874 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2f98066f30 msgr2=0x7f2f980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:40.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.644+0000 7f2fbb9c9700 1 --2- 172.21.15.46:0/1750626874 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2f98066f30 0x7f2f980693e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4007fd0 tx=0x7f2fa4009040 comp rx=0 tx=0).stop 2024-01-26T23:30:40.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.644+0000 7f2fbb9c9700 1 -- 172.21.15.46:0/1750626874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fb40b0280 msgr2=0x7f2fb414ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:40.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.644+0000 7f2fbb9c9700 1 --2- 172.21.15.46:0/1750626874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fb40b0280 0x7f2fb414ae80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009f00 tx=0x7f2fb0017000 comp rx=0 tx=0).stop 2024-01-26T23:30:40.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.644+0000 7f2fbb9c9700 1 -- 172.21.15.46:0/1750626874 shutdown_connections 2024-01-26T23:30:40.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.644+0000 7f2fbb9c9700 1 --2- 172.21.15.46:0/1750626874 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2f98066f30 0x7f2f980693e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.644+0000 7f2fbb9c9700 1 --2- 172.21.15.46:0/1750626874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fb41561d0 0x7f2fb414cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.644+0000 7f2fbb9c9700 1 --2- 172.21.15.46:0/1750626874 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fb406cbb0 0x7f2fb414c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.644+0000 7f2fbb9c9700 1 --2- 172.21.15.46:0/1750626874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fb40b0280 0x7f2fb414ae80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.644+0000 7f2fbb9c9700 1 -- 172.21.15.46:0/1750626874 >> 172.21.15.46:0/1750626874 conn(0x7f2fb406b630 msgr2=0x7f2fb40ca6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:40.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.644+0000 7f2fbb9c9700 1 -- 172.21.15.46:0/1750626874 shutdown_connections 2024-01-26T23:30:40.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.644+0000 7f2fbb9c9700 1 -- 172.21.15.46:0/1750626874 wait complete. 2024-01-26T23:30:40.660 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-26T23:30:40.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.791+0000 7f3a515c9700 1 Processor -- start 2024-01-26T23:30:40.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.791+0000 7f3a515c9700 1 -- start start 2024-01-26T23:30:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.792+0000 7f3a515c9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3a4c0c6520 0x7f3a4c0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.792+0000 7f3a515c9700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4c15a0b0 con 0x7f3a4c0b2b40 2024-01-26T23:30:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.792+0000 7f3a515c9700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4c0ae4d0 con 0x7f3a4c0ced50 2024-01-26T23:30:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.792+0000 7f3a515c9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a4c0af070 con 0x7f3a4c0c6520 2024-01-26T23:30:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.792+0000 7f3a4a7fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3a4c0c6520 0x7f3a4c0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.792+0000 7f3a4affd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3a4c0b2b40 0x7f3a4c0ae960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54246/0 (socket says 172.21.15.46:54246) 2024-01-26T23:30:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.792+0000 7f3a4affd700 1 -- 172.21.15.46:0/2152865770 learned_addr learned my addr 172.21.15.46:0/2152865770 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.792+0000 7f3a4a7fc700 1 -- 172.21.15.46:0/2152865770 >> v1:172.21.15.106:6789/0 conn(0x7f3a4c0ced50 legacy=0x7f3a4c0cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.792+0000 7f3a4a7fc700 1 -- 172.21.15.46:0/2152865770 >> v1:172.21.15.46:6789/0 conn(0x7f3a4c0b2b40 legacy=0x7f3a4c0ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.792+0000 7f3a4a7fc700 1 -- 172.21.15.46:0/2152865770 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4c0ad8c0 con 0x7f3a4c0c6520 2024-01-26T23:30:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.793+0000 7f3a4a7fc700 1 --2- 172.21.15.46:0/2152865770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a4c0c6520 0x7f3a4c0ad1c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3a400098e0 tx=0x7f3a400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d914640daf1b8535 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.793+0000 7f3a49ffb700 1 -- 172.21.15.46:0/2152865770 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40009e80 con 0x7f3a4c0c6520 2024-01-26T23:30:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.793+0000 7f3a49ffb700 1 -- 172.21.15.46:0/2152865770 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a4000b860 con 0x7f3a4c0c6520 2024-01-26T23:30:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.793+0000 7f3a49ffb700 1 -- 172.21.15.46:0/2152865770 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4000bbe0 con 0x7f3a4c0c6520 2024-01-26T23:30:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.793+0000 7f3a515c9700 1 -- 172.21.15.46:0/2152865770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a4c0c6520 msgr2=0x7f3a4c0ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.793+0000 7f3a515c9700 1 --2- 172.21.15.46:0/2152865770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a4c0c6520 0x7f3a4c0ad1c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3a400098e0 tx=0x7f3a400091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.793+0000 7f3a515c9700 1 -- 172.21.15.46:0/2152865770 shutdown_connections 2024-01-26T23:30:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.793+0000 7f3a515c9700 1 --2- 172.21.15.46:0/2152865770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a4c0c6520 0x7f3a4c0ad1c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.793+0000 7f3a515c9700 1 -- 172.21.15.46:0/2152865770 >> 172.21.15.46:0/2152865770 conn(0x7f3a4c099a30 msgr2=0x7f3a4c06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.793+0000 7f3a515c9700 1 -- 172.21.15.46:0/2152865770 shutdown_connections 2024-01-26T23:30:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.793+0000 7f3a515c9700 1 -- 172.21.15.46:0/2152865770 wait complete. 2024-01-26T23:30:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.795+0000 7f3a515c9700 1 Processor -- start 2024-01-26T23:30:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.795+0000 7f3a515c9700 1 -- start start 2024-01-26T23:30:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.796+0000 7f3a515c9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a4c0b2b40 0x7f3a4c0b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.796+0000 7f3a515c9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a4c0c6520 0x7f3a4c1528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.796+0000 7f3a515c9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a4c0ced50 0x7f3a4c156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.796+0000 7f3a515c9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c0b4410 con 0x7f3a4c0ced50 2024-01-26T23:30:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.796+0000 7f3a515c9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c152e30 con 0x7f3a4c0b2b40 2024-01-26T23:30:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.796+0000 7f3a515c9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c157030 con 0x7f3a4c0c6520 2024-01-26T23:30:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.796+0000 7f3a4b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a4c0ced50 0x7f3a4c156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.796+0000 7f3a4b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a4c0ced50 0x7f3a4c156af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47726/0 (socket says 172.21.15.46:47726) 2024-01-26T23:30:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.796+0000 7f3a4b7fe700 1 -- 172.21.15.46:0/584754774 learned_addr learned my addr 172.21.15.46:0/584754774 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.796+0000 7f3a4a7fc700 1 --2- 172.21.15.46:0/584754774 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a4c0c6520 0x7f3a4c1528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.796+0000 7f3a4affd700 1 --2- 172.21.15.46:0/584754774 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a4c0b2b40 0x7f3a4c0b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.796+0000 7f3a4b7fe700 1 -- 172.21.15.46:0/584754774 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a4c0c6520 msgr2=0x7f3a4c1528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.796+0000 7f3a4b7fe700 1 --2- 172.21.15.46:0/584754774 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a4c0c6520 0x7f3a4c1528f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.796+0000 7f3a4b7fe700 1 -- 172.21.15.46:0/584754774 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a4c0b2b40 msgr2=0x7f3a4c0b3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.796+0000 7f3a4b7fe700 1 --2- 172.21.15.46:0/584754774 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a4c0b2b40 0x7f3a4c0b3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.796+0000 7f3a4b7fe700 1 -- 172.21.15.46:0/584754774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a40009420 con 0x7f3a4c0ced50 2024-01-26T23:30:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.796+0000 7f3a4b7fe700 1 --2- 172.21.15.46:0/584754774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a4c0ced50 0x7f3a4c156af0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400bc30 tx=0x7f3a44009820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.797+0000 7f3a32ffd700 1 -- 172.21.15.46:0/584754774 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4400c840 con 0x7f3a4c0ced50 2024-01-26T23:30:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.797+0000 7f3a32ffd700 1 -- 172.21.15.46:0/584754774 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a4400c9e0 con 0x7f3a4c0ced50 2024-01-26T23:30:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.797+0000 7f3a515c9700 1 -- 172.21.15.46:0/584754774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4c16dbe0 con 0x7f3a4c0ced50 2024-01-26T23:30:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.797+0000 7f3a32ffd700 1 -- 172.21.15.46:0/584754774 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4400cda0 con 0x7f3a4c0ced50 2024-01-26T23:30:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.797+0000 7f3a515c9700 1 -- 172.21.15.46:0/584754774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4c16e1a0 con 0x7f3a4c0ced50 2024-01-26T23:30:40.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.798+0000 7f3a515c9700 1 -- 172.21.15.46:0/584754774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a4c04c710 con 0x7f3a4c0ced50 2024-01-26T23:30:40.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.798+0000 7f3a32ffd700 1 -- 172.21.15.46:0/584754774 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3a44014620 con 0x7f3a4c0ced50 2024-01-26T23:30:40.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.799+0000 7f3a32ffd700 1 --2- 172.21.15.46:0/584754774 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a38066f30 0x7f3a380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:40.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.799+0000 7f3a32ffd700 1 -- 172.21.15.46:0/584754774 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f3a44084830 con 0x7f3a4c0ced50 2024-01-26T23:30:40.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.799+0000 7f3a4affd700 1 --2- 172.21.15.46:0/584754774 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a38066f30 0x7f3a380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:40.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.799+0000 7f3a4affd700 1 --2- 172.21.15.46:0/584754774 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a38066f30 0x7f3a380693e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c007fd0 tx=0x7f3a3c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:40.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:40.803+0000 7f3a32ffd700 1 -- 172.21.15.46:0/584754774 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3a4404c5a0 con 0x7f3a4c0ced50 2024-01-26T23:30:41.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.055+0000 7f3a515c9700 1 -- 172.21.15.46:0/584754774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3a4c0682c0 con 0x7f3a4c0ced50 2024-01-26T23:30:41.057 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-26T23:30:41.056+0000 7f33ffe86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-26T23:30:41.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.060+0000 7f3a32ffd700 1 -- 172.21.15.46:0/584754774 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3a44055170 con 0x7f3a4c0ced50 2024-01-26T23:30:41.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.063+0000 7f3a515c9700 1 -- 172.21.15.46:0/584754774 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a38066f30 msgr2=0x7f3a380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:41.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.063+0000 7f3a515c9700 1 --2- 172.21.15.46:0/584754774 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a38066f30 0x7f3a380693e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c007fd0 tx=0x7f3a3c009040 comp rx=0 tx=0).stop 2024-01-26T23:30:41.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.063+0000 7f3a515c9700 1 -- 172.21.15.46:0/584754774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a4c0ced50 msgr2=0x7f3a4c156af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:41.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.063+0000 7f3a515c9700 1 --2- 172.21.15.46:0/584754774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a4c0ced50 0x7f3a4c156af0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400bc30 tx=0x7f3a44009820 comp rx=0 tx=0).stop 2024-01-26T23:30:41.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.063+0000 7f3a515c9700 1 -- 172.21.15.46:0/584754774 shutdown_connections 2024-01-26T23:30:41.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.063+0000 7f3a515c9700 1 --2- 172.21.15.46:0/584754774 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a38066f30 0x7f3a380693e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:41.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.063+0000 7f3a515c9700 1 --2- 172.21.15.46:0/584754774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a4c0ced50 0x7f3a4c156af0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:41.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.063+0000 7f3a515c9700 1 --2- 172.21.15.46:0/584754774 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a4c0b2b40 0x7f3a4c0b3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:41.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.063+0000 7f3a515c9700 1 --2- 172.21.15.46:0/584754774 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a4c0c6520 0x7f3a4c1528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:41.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.063+0000 7f3a515c9700 1 -- 172.21.15.46:0/584754774 >> 172.21.15.46:0/584754774 conn(0x7f3a4c099a30 msgr2=0x7f3a4c1556e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:41.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.063+0000 7f3a515c9700 1 -- 172.21.15.46:0/584754774 shutdown_connections 2024-01-26T23:30:41.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.063+0000 7f3a515c9700 1 -- 172.21.15.46:0/584754774 wait complete. 2024-01-26T23:30:41.066 INFO:tasks.ceph.mds.c.smithi046.stderr: -4363> 2024-01-26T23:29:48.920+0000 7f95b1ce4700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-26T23:30:41.066 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-26T23:30:41.079 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-26T23:30:41.093 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-26T23:30:41.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.211+0000 7f0a9b92a700 1 Processor -- start 2024-01-26T23:30:41.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.211+0000 7f0a9b92a700 1 -- start start 2024-01-26T23:30:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.212+0000 7f0a9b92a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0a940c9770 0x7f0a94159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.212+0000 7f0a9b92a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9415e390 con 0x7f0a94159760 2024-01-26T23:30:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.212+0000 7f0a9b92a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a94068460 con 0x7f0a940c93a0 2024-01-26T23:30:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.212+0000 7f0a9b92a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a9415d810 con 0x7f0a940c9770 2024-01-26T23:30:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.212+0000 7f0a99ec7700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0a94159760 0x7f0a940c7e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54256/0 (socket says 172.21.15.46:54256) 2024-01-26T23:30:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.212+0000 7f0a99ec7700 1 -- 172.21.15.46:0/1897604738 learned_addr learned my addr 172.21.15.46:0/1897604738 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.213+0000 7f0a98ec5700 1 --2- 172.21.15.46:0/1897604738 >> v2:172.21.15.106:3300/0 conn(0x7f0a940c9770 0x7f0a94159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.213+0000 7f0a87fff700 1 -- 172.21.15.46:0/1897604738 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 150005654 0 0) 0x7f0a9415e390 con 0x7f0a94159760 2024-01-26T23:30:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.213+0000 7f0a87fff700 1 -- 172.21.15.46:0/1897604738 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a80003570 con 0x7f0a94159760 2024-01-26T23:30:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.213+0000 7f0a87fff700 1 -- 172.21.15.46:0/1897604738 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911422077 0 0) 0x7f0a94068460 con 0x7f0a940c93a0 2024-01-26T23:30:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.213+0000 7f0a87fff700 1 -- 172.21.15.46:0/1897604738 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a9415e390 con 0x7f0a940c93a0 2024-01-26T23:30:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.213+0000 7f0a98ec5700 1 -- 172.21.15.46:0/1897604738 >> v1:172.21.15.106:6789/0 conn(0x7f0a940c93a0 legacy=0x7f0a94067d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.213+0000 7f0a98ec5700 1 -- 172.21.15.46:0/1897604738 >> v1:172.21.15.46:6789/0 conn(0x7f0a94159760 legacy=0x7f0a940c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.213+0000 7f0a98ec5700 1 -- 172.21.15.46:0/1897604738 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9415daf0 con 0x7f0a940c9770 2024-01-26T23:30:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.213+0000 7f0a98ec5700 1 --2- 172.21.15.46:0/1897604738 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a940c9770 0x7f0a94159220 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0a900098e0 tx=0x7f0a900091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8ddc98b6e416fa4 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.213+0000 7f0a87fff700 1 -- 172.21.15.46:0/1897604738 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a90011630 con 0x7f0a940c9770 2024-01-26T23:30:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.213+0000 7f0a87fff700 1 -- 172.21.15.46:0/1897604738 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a90011790 con 0x7f0a940c9770 2024-01-26T23:30:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.213+0000 7f0a87fff700 1 -- 172.21.15.46:0/1897604738 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a90011b30 con 0x7f0a940c9770 2024-01-26T23:30:41.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.214+0000 7f0a9b92a700 1 -- 172.21.15.46:0/1897604738 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a940c9770 msgr2=0x7f0a94159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:41.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.214+0000 7f0a9b92a700 1 --2- 172.21.15.46:0/1897604738 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a940c9770 0x7f0a94159220 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0a900098e0 tx=0x7f0a900091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:41.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.214+0000 7f0a9b92a700 1 -- 172.21.15.46:0/1897604738 shutdown_connections 2024-01-26T23:30:41.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.214+0000 7f0a9b92a700 1 --2- 172.21.15.46:0/1897604738 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a940c9770 0x7f0a94159220 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:41.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.214+0000 7f0a9b92a700 1 -- 172.21.15.46:0/1897604738 >> 172.21.15.46:0/1897604738 conn(0x7f0a940ca9a0 msgr2=0x7f0a94068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:41.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.214+0000 7f0a9b92a700 1 -- 172.21.15.46:0/1897604738 shutdown_connections 2024-01-26T23:30:41.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.214+0000 7f0a9b92a700 1 -- 172.21.15.46:0/1897604738 wait complete. 2024-01-26T23:30:41.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.215+0000 7f0a9b92a700 1 Processor -- start 2024-01-26T23:30:41.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.215+0000 7f0a9b92a700 1 -- start start 2024-01-26T23:30:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.216+0000 7f0a9b92a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a940c93a0 0x7f0a94154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.216+0000 7f0a9b92a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a940c9770 0x7f0a9415a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.216+0000 7f0a9b92a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a94159760 0x7f0a9406c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.216+0000 7f0a9b92a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a940c4510 con 0x7f0a940c9770 2024-01-26T23:30:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.216+0000 7f0a9b92a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a940c76b0 con 0x7f0a940c93a0 2024-01-26T23:30:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.216+0000 7f0a9b92a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a940c4760 con 0x7f0a94159760 2024-01-26T23:30:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.216+0000 7f0a98ec5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a940c9770 0x7f0a9415a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.216+0000 7f0a98ec5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a940c9770 0x7f0a9415a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47748/0 (socket says 172.21.15.46:47748) 2024-01-26T23:30:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.216+0000 7f0a98ec5700 1 -- 172.21.15.46:0/2455976267 learned_addr learned my addr 172.21.15.46:0/2455976267 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.216+0000 7f0a996c6700 1 --2- 172.21.15.46:0/2455976267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a940c93a0 0x7f0a94154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.216+0000 7f0a99ec7700 1 --2- 172.21.15.46:0/2455976267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a94159760 0x7f0a9406c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.217+0000 7f0a98ec5700 1 -- 172.21.15.46:0/2455976267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a94159760 msgr2=0x7f0a9406c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.217+0000 7f0a98ec5700 1 --2- 172.21.15.46:0/2455976267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a94159760 0x7f0a9406c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.217+0000 7f0a98ec5700 1 -- 172.21.15.46:0/2455976267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a940c93a0 msgr2=0x7f0a94154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.217+0000 7f0a98ec5700 1 --2- 172.21.15.46:0/2455976267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a940c93a0 0x7f0a94154fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.217+0000 7f0a98ec5700 1 -- 172.21.15.46:0/2455976267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a90009420 con 0x7f0a940c9770 2024-01-26T23:30:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.217+0000 7f0a98ec5700 1 --2- 172.21.15.46:0/2455976267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a940c9770 0x7f0a9415a550 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0a9406e820 tx=0x7f0a90017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.217+0000 7f0a857fa700 1 -- 172.21.15.46:0/2455976267 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9001f070 con 0x7f0a940c9770 2024-01-26T23:30:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.217+0000 7f0a857fa700 1 -- 172.21.15.46:0/2455976267 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a90002f70 con 0x7f0a940c9770 2024-01-26T23:30:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.217+0000 7f0a857fa700 1 -- 172.21.15.46:0/2455976267 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a90006af0 con 0x7f0a940c9770 2024-01-26T23:30:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.217+0000 7f0a9b92a700 1 -- 172.21.15.46:0/2455976267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a940c4a40 con 0x7f0a940c9770 2024-01-26T23:30:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.217+0000 7f0a9b92a700 1 -- 172.21.15.46:0/2455976267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a940c70f0 con 0x7f0a940c9770 2024-01-26T23:30:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.218+0000 7f0a9b92a700 1 -- 172.21.15.46:0/2455976267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a9404c710 con 0x7f0a940c9770 2024-01-26T23:30:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.219+0000 7f0a857fa700 1 -- 172.21.15.46:0/2455976267 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0a90023070 con 0x7f0a940c9770 2024-01-26T23:30:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.219+0000 7f0a857fa700 1 --2- 172.21.15.46:0/2455976267 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0a78066f30 0x7f0a780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.219+0000 7f0a857fa700 1 -- 172.21.15.46:0/2455976267 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f0a90085a00 con 0x7f0a940c9770 2024-01-26T23:30:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.219+0000 7f0a996c6700 1 --2- 172.21.15.46:0/2455976267 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0a78066f30 0x7f0a780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.220+0000 7f0a996c6700 1 --2- 172.21.15.46:0/2455976267 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0a78066f30 0x7f0a780693e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0a9406d5e0 tx=0x7f0a88007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:41.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.224+0000 7f0a857fa700 1 -- 172.21.15.46:0/2455976267 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a90052860 con 0x7f0a940c9770 2024-01-26T23:30:41.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:41.479+0000 7f0a9b92a700 1 -- 172.21.15.46:0/2455976267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0a94155510 con 0x7f0a940c9770 2024-01-26T23:30:42.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.061+0000 7f0a857fa700 1 -- 172.21.15.46:0/2455976267 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0a900150a0 con 0x7f0a940c9770 2024-01-26T23:30:42.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.064+0000 7f0a9b92a700 1 -- 172.21.15.46:0/2455976267 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0a78066f30 msgr2=0x7f0a780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:42.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.064+0000 7f0a9b92a700 1 --2- 172.21.15.46:0/2455976267 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0a78066f30 0x7f0a780693e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0a9406d5e0 tx=0x7f0a88007d50 comp rx=0 tx=0).stop 2024-01-26T23:30:42.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.064+0000 7f0a9b92a700 1 -- 172.21.15.46:0/2455976267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a940c9770 msgr2=0x7f0a9415a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:42.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.064+0000 7f0a9b92a700 1 --2- 172.21.15.46:0/2455976267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a940c9770 0x7f0a9415a550 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0a9406e820 tx=0x7f0a90017000 comp rx=0 tx=0).stop 2024-01-26T23:30:42.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.064+0000 7f0a9b92a700 1 -- 172.21.15.46:0/2455976267 shutdown_connections 2024-01-26T23:30:42.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.064+0000 7f0a9b92a700 1 --2- 172.21.15.46:0/2455976267 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0a78066f30 0x7f0a780693e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:42.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.064+0000 7f0a9b92a700 1 --2- 172.21.15.46:0/2455976267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a94159760 0x7f0a9406c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:42.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.064+0000 7f0a9b92a700 1 --2- 172.21.15.46:0/2455976267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a940c93a0 0x7f0a94154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:42.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.064+0000 7f0a9b92a700 1 --2- 172.21.15.46:0/2455976267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a940c9770 0x7f0a9415a550 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:42.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.064+0000 7f0a9b92a700 1 -- 172.21.15.46:0/2455976267 >> 172.21.15.46:0/2455976267 conn(0x7f0a940ca9a0 msgr2=0x7f0a940b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:42.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.064+0000 7f0a9b92a700 1 -- 172.21.15.46:0/2455976267 shutdown_connections 2024-01-26T23:30:42.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.064+0000 7f0a9b92a700 1 -- 172.21.15.46:0/2455976267 wait complete. 2024-01-26T23:30:42.081 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-26T23:30:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.264+0000 7f3bfc241700 1 Processor -- start 2024-01-26T23:30:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.264+0000 7f3bfc241700 1 -- start start 2024-01-26T23:30:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.265+0000 7f3bfc241700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3bf4156b00 0x7f3bf415ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.265+0000 7f3bfc241700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf4157000 con 0x7f3bf40c9d50 2024-01-26T23:30:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.265+0000 7f3bfc241700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf406f3e0 con 0x7f3bf40c9980 2024-01-26T23:30:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.265+0000 7f3bfc241700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf406f6c0 con 0x7f3bf4156b00 2024-01-26T23:30:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.265+0000 7f3bfa7de700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3bf40c9d50 0x7f3bf40b1aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54262/0 (socket says 172.21.15.46:54262) 2024-01-26T23:30:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.265+0000 7f3bfa7de700 1 -- 172.21.15.46:0/2897193454 learned_addr learned my addr 172.21.15.46:0/2897193454 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.265+0000 7f3bf97dc700 1 --2- 172.21.15.46:0/2897193454 >> v2:172.21.15.106:3300/0 conn(0x7f3bf4156b00 0x7f3bf415ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.265+0000 7f3bf8fdb700 1 -- 172.21.15.46:0/2897193454 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942212337 0 0) 0x7f3bf4157000 con 0x7f3bf40c9d50 2024-01-26T23:30:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.265+0000 7f3bf8fdb700 1 -- 172.21.15.46:0/2897193454 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be0003570 con 0x7f3bf40c9d50 2024-01-26T23:30:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.265+0000 7f3bf8fdb700 1 -- 172.21.15.46:0/2897193454 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286300850 0 0) 0x7f3bf406f3e0 con 0x7f3bf40c9980 2024-01-26T23:30:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.265+0000 7f3bf8fdb700 1 -- 172.21.15.46:0/2897193454 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bf4157000 con 0x7f3bf40c9980 2024-01-26T23:30:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.265+0000 7f3bf8fdb700 1 -- 172.21.15.46:0/2897193454 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3971082165 0 0) 0x7f3be0003570 con 0x7f3bf40c9d50 2024-01-26T23:30:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.266+0000 7f3bf97dc700 1 -- 172.21.15.46:0/2897193454 >> v1:172.21.15.106:6789/0 conn(0x7f3bf40c9980 legacy=0x7f3bf406eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.266+0000 7f3bf97dc700 1 -- 172.21.15.46:0/2897193454 >> v1:172.21.15.46:6789/0 conn(0x7f3bf40c9d50 legacy=0x7f3bf40b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.266+0000 7f3bf97dc700 1 -- 172.21.15.46:0/2897193454 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf406f9a0 con 0x7f3bf4156b00 2024-01-26T23:30:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.266+0000 7f3bf97dc700 1 --2- 172.21.15.46:0/2897193454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bf4156b00 0x7f3bf415ae10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3be40098e0 tx=0x7f3be40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=137318b304e91258 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.266+0000 7f3bf8fdb700 1 -- 172.21.15.46:0/2897193454 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec0017f0 con 0x7f3bf4156b00 2024-01-26T23:30:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.266+0000 7f3bf8fdb700 1 -- 172.21.15.46:0/2897193454 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be4011420 con 0x7f3bf4156b00 2024-01-26T23:30:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.266+0000 7f3bf8fdb700 1 -- 172.21.15.46:0/2897193454 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be40117c0 con 0x7f3bf4156b00 2024-01-26T23:30:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.267+0000 7f3bfc241700 1 -- 172.21.15.46:0/2897193454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bf4156b00 msgr2=0x7f3bf415ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.267+0000 7f3bfc241700 1 --2- 172.21.15.46:0/2897193454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bf4156b00 0x7f3bf415ae10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3be40098e0 tx=0x7f3be40091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.267+0000 7f3bfc241700 1 -- 172.21.15.46:0/2897193454 shutdown_connections 2024-01-26T23:30:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.267+0000 7f3bfc241700 1 --2- 172.21.15.46:0/2897193454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bf4156b00 0x7f3bf415ae10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.267+0000 7f3bfc241700 1 -- 172.21.15.46:0/2897193454 >> 172.21.15.46:0/2897193454 conn(0x7f3bf4069510 msgr2=0x7f3bf40aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.267+0000 7f3bfc241700 1 -- 172.21.15.46:0/2897193454 shutdown_connections 2024-01-26T23:30:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.267+0000 7f3bfc241700 1 -- 172.21.15.46:0/2897193454 wait complete. 2024-01-26T23:30:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.268+0000 7f3bfc241700 1 Processor -- start 2024-01-26T23:30:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.268+0000 7f3bfc241700 1 -- start start 2024-01-26T23:30:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.269+0000 7f3bfc241700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bf40c9980 0x7f3bf41451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.269+0000 7f3bfc241700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3bf40c9d50 0x7f3bf40ab9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.269+0000 7f3bfc241700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bf4156b00 0x7f3bf40b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.269+0000 7f3bfc241700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf4146d50 con 0x7f3bf40c9980 2024-01-26T23:30:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.269+0000 7f3bfc241700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf4145700 con 0x7f3bf4156b00 2024-01-26T23:30:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.269+0000 7f3bfc241700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf4146140 con 0x7f3bf40c9d50 2024-01-26T23:30:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.269+0000 7f3bf9fdd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bf40c9980 0x7f3bf41451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.269+0000 7f3bf9fdd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bf40c9980 0x7f3bf41451c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47762/0 (socket says 172.21.15.46:47762) 2024-01-26T23:30:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.269+0000 7f3bf9fdd700 1 -- 172.21.15.46:0/2108343456 learned_addr learned my addr 172.21.15.46:0/2108343456 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.269+0000 7f3bfa7de700 1 --2- 172.21.15.46:0/2108343456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bf4156b00 0x7f3bf40b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.269+0000 7f3bf9fdd700 1 -- 172.21.15.46:0/2108343456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3bf40c9d50 msgr2=0x7f3bf40ab9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.269+0000 7f3bf9fdd700 1 --2- 172.21.15.46:0/2108343456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3bf40c9d50 0x7f3bf40ab9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:42.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.269+0000 7f3bf9fdd700 1 -- 172.21.15.46:0/2108343456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bf4156b00 msgr2=0x7f3bf40b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:42.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.269+0000 7f3bf9fdd700 1 --2- 172.21.15.46:0/2108343456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bf4156b00 0x7f3bf40b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:42.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.269+0000 7f3bf9fdd700 1 -- 172.21.15.46:0/2108343456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be4009420 con 0x7f3bf40c9980 2024-01-26T23:30:42.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.269+0000 7f3bf9fdd700 1 --2- 172.21.15.46:0/2108343456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bf40c9980 0x7f3bf41451c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000c790 tx=0x7f3bf0007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:42.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.270+0000 7f3be9ffb700 1 -- 172.21.15.46:0/2108343456 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0010510 con 0x7f3bf40c9980 2024-01-26T23:30:42.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.270+0000 7f3be9ffb700 1 -- 172.21.15.46:0/2108343456 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf0010670 con 0x7f3bf40c9980 2024-01-26T23:30:42.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.270+0000 7f3bfc241700 1 -- 172.21.15.46:0/2108343456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf4146420 con 0x7f3bf40c9980 2024-01-26T23:30:42.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.270+0000 7f3be9ffb700 1 -- 172.21.15.46:0/2108343456 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0010a30 con 0x7f3bf40c9980 2024-01-26T23:30:42.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.270+0000 7f3bfc241700 1 -- 172.21.15.46:0/2108343456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf41469e0 con 0x7f3bf40c9980 2024-01-26T23:30:42.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.271+0000 7f3bfc241700 1 -- 172.21.15.46:0/2108343456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf40b2ec0 con 0x7f3bf40c9980 2024-01-26T23:30:42.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.271+0000 7f3be9ffb700 1 -- 172.21.15.46:0/2108343456 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3bf001b400 con 0x7f3bf40c9980 2024-01-26T23:30:42.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.271+0000 7f3be9ffb700 1 --2- 172.21.15.46:0/2108343456 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3bd8066f30 0x7f3bd80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:42.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.272+0000 7f3be9ffb700 1 -- 172.21.15.46:0/2108343456 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f3bf0084790 con 0x7f3bf40c9980 2024-01-26T23:30:42.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.272+0000 7f3bf97dc700 1 --2- 172.21.15.46:0/2108343456 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3bd8066f30 0x7f3bd80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:42.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.272+0000 7f3bf97dc700 1 --2- 172.21.15.46:0/2108343456 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3bd8066f30 0x7f3bd80693e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3bf406fef0 tx=0x7f3be4016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:42.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.276+0000 7f3be9ffb700 1 -- 172.21.15.46:0/2108343456 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3bf00515f0 con 0x7f3bf40c9980 2024-01-26T23:30:42.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:42.480+0000 7f3bfc241700 1 -- 172.21.15.46:0/2108343456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3bf404c710 con 0x7f3bf40c9980 2024-01-26T23:30:43.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.065+0000 7f3be9ffb700 1 -- 172.21.15.46:0/2108343456 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v28) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3bf0054c10 con 0x7f3bf40c9980 2024-01-26T23:30:43.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.073+0000 7f3bfc241700 1 -- 172.21.15.46:0/2108343456 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3bd8066f30 msgr2=0x7f3bd80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:43.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.073+0000 7f3bfc241700 1 --2- 172.21.15.46:0/2108343456 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3bd8066f30 0x7f3bd80693e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3bf406fef0 tx=0x7f3be4016000 comp rx=0 tx=0).stop 2024-01-26T23:30:43.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.073+0000 7f3bfc241700 1 -- 172.21.15.46:0/2108343456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bf40c9980 msgr2=0x7f3bf41451c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:43.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.073+0000 7f3bfc241700 1 --2- 172.21.15.46:0/2108343456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bf40c9980 0x7f3bf41451c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000c790 tx=0x7f3bf0007e10 comp rx=0 tx=0).stop 2024-01-26T23:30:43.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.073+0000 7f3bfc241700 1 -- 172.21.15.46:0/2108343456 shutdown_connections 2024-01-26T23:30:43.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.073+0000 7f3bfc241700 1 --2- 172.21.15.46:0/2108343456 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3bd8066f30 0x7f3bd80693e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:43.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.073+0000 7f3bfc241700 1 --2- 172.21.15.46:0/2108343456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bf40c9980 0x7f3bf41451c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:43.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.073+0000 7f3bfc241700 1 --2- 172.21.15.46:0/2108343456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bf4156b00 0x7f3bf40b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:43.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.073+0000 7f3bfc241700 1 --2- 172.21.15.46:0/2108343456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3bf40c9d50 0x7f3bf40ab9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:43.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.073+0000 7f3bfc241700 1 -- 172.21.15.46:0/2108343456 >> 172.21.15.46:0/2108343456 conn(0x7f3bf4069510 msgr2=0x7f3bf4155480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:43.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.073+0000 7f3bfc241700 1 -- 172.21.15.46:0/2108343456 shutdown_connections 2024-01-26T23:30:43.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.073+0000 7f3bfc241700 1 -- 172.21.15.46:0/2108343456 wait complete. 2024-01-26T23:30:43.076 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-01-26T23:30:43.091 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-26T23:30:43.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.232+0000 7f3a75231700 1 Processor -- start 2024-01-26T23:30:43.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.232+0000 7f3a75231700 1 -- start start 2024-01-26T23:30:43.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.233+0000 7f3a75231700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a700ce2a0 0x7f3a700cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:43.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.233+0000 7f3a75231700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a700cf450 con 0x7f3a700ad790 2024-01-26T23:30:43.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.233+0000 7f3a75231700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a700cb1f0 con 0x7f3a701599c0 2024-01-26T23:30:43.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.233+0000 7f3a75231700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a700cc330 con 0x7f3a700ce2a0 2024-01-26T23:30:43.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.233+0000 7f3a6f59e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3a700ad790 0x7f3a700adc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54264/0 (socket says 172.21.15.46:54264) 2024-01-26T23:30:43.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.233+0000 7f3a6f59e700 1 -- 172.21.15.46:0/8989248 learned_addr learned my addr 172.21.15.46:0/8989248 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:43.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.233+0000 7f3a6e59c700 1 --2- 172.21.15.46:0/8989248 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a700ce2a0 0x7f3a700cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:43.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.233+0000 7f3a6dd9b700 1 -- 172.21.15.46:0/8989248 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002495947 0 0) 0x7f3a700cf450 con 0x7f3a700ad790 2024-01-26T23:30:43.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.233+0000 7f3a6dd9b700 1 -- 172.21.15.46:0/8989248 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a5c003570 con 0x7f3a700ad790 2024-01-26T23:30:43.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.233+0000 7f3a6dd9b700 1 -- 172.21.15.46:0/8989248 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189784270 0 0) 0x7f3a700cb1f0 con 0x7f3a701599c0 2024-01-26T23:30:43.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.234+0000 7f3a6dd9b700 1 -- 172.21.15.46:0/8989248 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a700cf450 con 0x7f3a701599c0 2024-01-26T23:30:43.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.234+0000 7f3a6e59c700 1 -- 172.21.15.46:0/8989248 >> v1:172.21.15.106:6789/0 conn(0x7f3a701599c0 legacy=0x7f3a700cf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:43.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.234+0000 7f3a6e59c700 1 -- 172.21.15.46:0/8989248 >> v1:172.21.15.46:6789/0 conn(0x7f3a700ad790 legacy=0x7f3a700adc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:43.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.234+0000 7f3a6dd9b700 1 -- 172.21.15.46:0/8989248 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2438703444 0 0) 0x7f3a5c003570 con 0x7f3a700ad790 2024-01-26T23:30:43.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.234+0000 7f3a6e59c700 1 -- 172.21.15.46:0/8989248 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a700cfff0 con 0x7f3a700ce2a0 2024-01-26T23:30:43.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.234+0000 7f3a6e59c700 1 --2- 172.21.15.46:0/8989248 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a700ce2a0 0x7f3a700cbdf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009910 tx=0x7f3a600091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=236d4cc62be22adf server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:43.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.234+0000 7f3a6dd9b700 1 -- 172.21.15.46:0/8989248 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a680017f0 con 0x7f3a700ce2a0 2024-01-26T23:30:43.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.234+0000 7f3a6dd9b700 1 -- 172.21.15.46:0/8989248 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a60011420 con 0x7f3a700ce2a0 2024-01-26T23:30:43.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.234+0000 7f3a6dd9b700 1 -- 172.21.15.46:0/8989248 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a600117a0 con 0x7f3a700ce2a0 2024-01-26T23:30:43.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.235+0000 7f3a75231700 1 -- 172.21.15.46:0/8989248 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a700ce2a0 msgr2=0x7f3a700cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:43.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.235+0000 7f3a75231700 1 --2- 172.21.15.46:0/8989248 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a700ce2a0 0x7f3a700cbdf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009910 tx=0x7f3a600091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:43.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.235+0000 7f3a75231700 1 -- 172.21.15.46:0/8989248 shutdown_connections 2024-01-26T23:30:43.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.235+0000 7f3a75231700 1 --2- 172.21.15.46:0/8989248 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a700ce2a0 0x7f3a700cbdf0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:43.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.235+0000 7f3a75231700 1 -- 172.21.15.46:0/8989248 >> 172.21.15.46:0/8989248 conn(0x7f3a70099a50 msgr2=0x7f3a7006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:43.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.235+0000 7f3a75231700 1 -- 172.21.15.46:0/8989248 shutdown_connections 2024-01-26T23:30:43.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.235+0000 7f3a75231700 1 -- 172.21.15.46:0/8989248 wait complete. 2024-01-26T23:30:43.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.236+0000 7f3a75231700 1 Processor -- start 2024-01-26T23:30:43.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.236+0000 7f3a75231700 1 -- start start 2024-01-26T23:30:43.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.237+0000 7f3a75231700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a700ad790 0x7f3a700c5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:43.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.237+0000 7f3a75231700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a700ce2a0 0x7f3a70155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:43.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.237+0000 7f3a75231700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a701599c0 0x7f3a70152d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:43.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.237+0000 7f3a75231700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7015d9a0 con 0x7f3a701599c0 2024-01-26T23:30:43.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.237+0000 7f3a75231700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a70155d20 con 0x7f3a700ce2a0 2024-01-26T23:30:43.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.237+0000 7f3a75231700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7015d000 con 0x7f3a700ad790 2024-01-26T23:30:43.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.237+0000 7f3a6f59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a701599c0 0x7f3a70152d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:43.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.237+0000 7f3a6f59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a701599c0 0x7f3a70152d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47778/0 (socket says 172.21.15.46:47778) 2024-01-26T23:30:43.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.237+0000 7f3a6f59e700 1 -- 172.21.15.46:0/3883642298 learned_addr learned my addr 172.21.15.46:0/3883642298 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.237+0000 7f3a6e59c700 1 --2- 172.21.15.46:0/3883642298 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a700ce2a0 0x7f3a70155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.237+0000 7f3a6ed9d700 1 --2- 172.21.15.46:0/3883642298 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a700ad790 0x7f3a700c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.238+0000 7f3a6f59e700 1 -- 172.21.15.46:0/3883642298 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a700ad790 msgr2=0x7f3a700c5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.238+0000 7f3a6f59e700 1 --2- 172.21.15.46:0/3883642298 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a700ad790 0x7f3a700c5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.238+0000 7f3a6f59e700 1 -- 172.21.15.46:0/3883642298 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a700ce2a0 msgr2=0x7f3a70155420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.238+0000 7f3a6f59e700 1 --2- 172.21.15.46:0/3883642298 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a700ce2a0 0x7f3a70155420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.238+0000 7f3a6f59e700 1 -- 172.21.15.46:0/3883642298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a60009420 con 0x7f3a701599c0 2024-01-26T23:30:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.238+0000 7f3a6f59e700 1 --2- 172.21.15.46:0/3883642298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a701599c0 0x7f3a70152d10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3a68001560 tx=0x7f3a68007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.238+0000 7f3a66ffd700 1 -- 172.21.15.46:0/3883642298 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68004dc0 con 0x7f3a701599c0 2024-01-26T23:30:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.238+0000 7f3a66ffd700 1 -- 172.21.15.46:0/3883642298 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6800a640 con 0x7f3a701599c0 2024-01-26T23:30:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.238+0000 7f3a75231700 1 -- 172.21.15.46:0/3883642298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a7015d2e0 con 0x7f3a701599c0 2024-01-26T23:30:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.238+0000 7f3a66ffd700 1 -- 172.21.15.46:0/3883642298 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6800a9e0 con 0x7f3a701599c0 2024-01-26T23:30:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.238+0000 7f3a75231700 1 -- 172.21.15.46:0/3883642298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7015d670 con 0x7f3a701599c0 2024-01-26T23:30:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.239+0000 7f3a75231700 1 -- 172.21.15.46:0/3883642298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7004c710 con 0x7f3a701599c0 2024-01-26T23:30:43.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.240+0000 7f3a66ffd700 1 -- 172.21.15.46:0/3883642298 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3a68007150 con 0x7f3a701599c0 2024-01-26T23:30:43.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.240+0000 7f3a66ffd700 1 --2- 172.21.15.46:0/3883642298 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a50066f30 0x7f3a500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:43.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.240+0000 7f3a66ffd700 1 -- 172.21.15.46:0/3883642298 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6307+0+0 (secure 0 0 0) 0x7f3a680841e0 con 0x7f3a701599c0 2024-01-26T23:30:43.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.240+0000 7f3a6ed9d700 1 --2- 172.21.15.46:0/3883642298 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a50066f30 0x7f3a500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:43.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.241+0000 7f3a6ed9d700 1 --2- 172.21.15.46:0/3883642298 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a50066f30 0x7f3a500693e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3a58000c00 tx=0x7f3a58007d20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:43.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.245+0000 7f3a66ffd700 1 -- 172.21.15.46:0/3883642298 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3a68051270 con 0x7f3a701599c0 2024-01-26T23:30:43.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:43.453+0000 7f3a75231700 1 -- 172.21.15.46:0/3883642298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3a700ce840 con 0x7f3a701599c0 2024-01-26T23:30:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.073+0000 7f3a66ffd700 1 -- 172.21.15.46:0/3883642298 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v29) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3a68054890 con 0x7f3a701599c0 2024-01-26T23:30:44.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.079+0000 7f3a75231700 1 -- 172.21.15.46:0/3883642298 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a50066f30 msgr2=0x7f3a500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:44.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.079+0000 7f3a75231700 1 --2- 172.21.15.46:0/3883642298 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a50066f30 0x7f3a500693e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3a58000c00 tx=0x7f3a58007d20 comp rx=0 tx=0).stop 2024-01-26T23:30:44.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.079+0000 7f3a75231700 1 -- 172.21.15.46:0/3883642298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a701599c0 msgr2=0x7f3a70152d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:44.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.079+0000 7f3a75231700 1 --2- 172.21.15.46:0/3883642298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a701599c0 0x7f3a70152d10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3a68001560 tx=0x7f3a68007d20 comp rx=0 tx=0).stop 2024-01-26T23:30:44.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.079+0000 7f3a75231700 1 -- 172.21.15.46:0/3883642298 shutdown_connections 2024-01-26T23:30:44.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.079+0000 7f3a75231700 1 --2- 172.21.15.46:0/3883642298 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3a50066f30 0x7f3a500693e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:44.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.079+0000 7f3a75231700 1 --2- 172.21.15.46:0/3883642298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a701599c0 0x7f3a70152d10 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:44.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.079+0000 7f3a75231700 1 --2- 172.21.15.46:0/3883642298 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a700ad790 0x7f3a700c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:44.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.079+0000 7f3a75231700 1 --2- 172.21.15.46:0/3883642298 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a700ce2a0 0x7f3a70155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:44.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.079+0000 7f3a75231700 1 -- 172.21.15.46:0/3883642298 >> 172.21.15.46:0/3883642298 conn(0x7f3a70099a50 msgr2=0x7f3a7015f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:44.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.084+0000 7f3a75231700 1 -- 172.21.15.46:0/3883642298 shutdown_connections 2024-01-26T23:30:44.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.086+0000 7f3a75231700 1 -- 172.21.15.46:0/3883642298 wait complete. 2024-01-26T23:30:44.089 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-01-26T23:30:44.103 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-01-26T23:30:44.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.240+0000 7f2f4734d700 1 Processor -- start 2024-01-26T23:30:44.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.240+0000 7f2f4734d700 1 -- start start 2024-01-26T23:30:44.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.241+0000 7f2f4734d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2f40159b20 0x7f2f400ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:44.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.241+0000 7f2f4734d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f400691c0 con 0x7f2f400ae340 2024-01-26T23:30:44.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.241+0000 7f2f4734d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f4006a5e0 con 0x7f2f40159750 2024-01-26T23:30:44.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.241+0000 7f2f4734d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f4015dfd0 con 0x7f2f40159b20 2024-01-26T23:30:44.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.241+0000 7f2f458ea700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2f400ae340 0x7f2f4015d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54272/0 (socket says 172.21.15.46:54272) 2024-01-26T23:30:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.242+0000 7f2f458ea700 1 -- 172.21.15.46:0/1356379032 learned_addr learned my addr 172.21.15.46:0/1356379032 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.242+0000 7f2f448e8700 1 --2- 172.21.15.46:0/1356379032 >> v2:172.21.15.106:3300/0 conn(0x7f2f40159b20 0x7f2f400ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.242+0000 7f2f37fff700 1 -- 172.21.15.46:0/1356379032 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610253815 0 0) 0x7f2f400691c0 con 0x7f2f400ae340 2024-01-26T23:30:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.242+0000 7f2f37fff700 1 -- 172.21.15.46:0/1356379032 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f2c003570 con 0x7f2f400ae340 2024-01-26T23:30:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.242+0000 7f2f37fff700 1 -- 172.21.15.46:0/1356379032 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3045641885 0 0) 0x7f2f4006a5e0 con 0x7f2f40159750 2024-01-26T23:30:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.242+0000 7f2f37fff700 1 -- 172.21.15.46:0/1356379032 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f400691c0 con 0x7f2f40159750 2024-01-26T23:30:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.242+0000 7f2f37fff700 1 -- 172.21.15.46:0/1356379032 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1680205056 0 0) 0x7f2f2c003570 con 0x7f2f400ae340 2024-01-26T23:30:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.242+0000 7f2f448e8700 1 -- 172.21.15.46:0/1356379032 >> v1:172.21.15.106:6789/0 conn(0x7f2f40159750 legacy=0x7f2f40069650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.242+0000 7f2f448e8700 1 -- 172.21.15.46:0/1356379032 >> v1:172.21.15.46:6789/0 conn(0x7f2f400ae340 legacy=0x7f2f4015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.242+0000 7f2f448e8700 1 -- 172.21.15.46:0/1356379032 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f40068b70 con 0x7f2f40159b20 2024-01-26T23:30:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.242+0000 7f2f448e8700 1 --2- 172.21.15.46:0/1356379032 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f40159b20 0x7f2f400ade00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2f300098e0 tx=0x7f2f300091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8a010c726dd68ae server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.243+0000 7f2f37fff700 1 -- 172.21.15.46:0/1356379032 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f380017f0 con 0x7f2f40159b20 2024-01-26T23:30:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.243+0000 7f2f37fff700 1 -- 172.21.15.46:0/1356379032 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f30011420 con 0x7f2f40159b20 2024-01-26T23:30:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.243+0000 7f2f37fff700 1 -- 172.21.15.46:0/1356379032 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f300117c0 con 0x7f2f40159b20 2024-01-26T23:30:44.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.243+0000 7f2f4734d700 1 -- 172.21.15.46:0/1356379032 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f40159b20 msgr2=0x7f2f400ade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:44.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.243+0000 7f2f4734d700 1 --2- 172.21.15.46:0/1356379032 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f40159b20 0x7f2f400ade00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2f300098e0 tx=0x7f2f300091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:44.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.243+0000 7f2f4734d700 1 -- 172.21.15.46:0/1356379032 shutdown_connections 2024-01-26T23:30:44.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.243+0000 7f2f4734d700 1 --2- 172.21.15.46:0/1356379032 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f40159b20 0x7f2f400ade00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:44.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.243+0000 7f2f4734d700 1 -- 172.21.15.46:0/1356379032 >> 172.21.15.46:0/1356379032 conn(0x7f2f400ce530 msgr2=0x7f2f400ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:44.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.243+0000 7f2f4734d700 1 -- 172.21.15.46:0/1356379032 shutdown_connections 2024-01-26T23:30:44.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.243+0000 7f2f4734d700 1 -- 172.21.15.46:0/1356379032 wait complete. 2024-01-26T23:30:44.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.244+0000 7f2f4734d700 1 Processor -- start 2024-01-26T23:30:44.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.244+0000 7f2f4734d700 1 -- start start 2024-01-26T23:30:44.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.245+0000 7f2f4734d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f400ae340 0x7f2f40155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:44.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.245+0000 7f2f4734d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f40159750 0x7f2f400b2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:44.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.245+0000 7f2f4734d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2f40159b20 0x7f2f400b2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:44.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.245+0000 7f2f4734d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400b3610 con 0x7f2f400ae340 2024-01-26T23:30:44.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.245+0000 7f2f4734d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400b38f0 con 0x7f2f40159750 2024-01-26T23:30:44.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.245+0000 7f2f4734d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400b3bd0 con 0x7f2f40159b20 2024-01-26T23:30:44.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.246+0000 7f2f450e9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f400ae340 0x7f2f40155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:44.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.246+0000 7f2f450e9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f400ae340 0x7f2f40155910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47794/0 (socket says 172.21.15.46:47794) 2024-01-26T23:30:44.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.246+0000 7f2f450e9700 1 -- 172.21.15.46:0/70999273 learned_addr learned my addr 172.21.15.46:0/70999273 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:44.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.246+0000 7f2f458ea700 1 --2- 172.21.15.46:0/70999273 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2f40159b20 0x7f2f400b2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:44.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.246+0000 7f2f448e8700 1 --2- 172.21.15.46:0/70999273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f40159750 0x7f2f400b2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:44.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.246+0000 7f2f450e9700 1 -- 172.21.15.46:0/70999273 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2f40159b20 msgr2=0x7f2f400b2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:44.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.246+0000 7f2f450e9700 1 --2- 172.21.15.46:0/70999273 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2f40159b20 0x7f2f400b2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:44.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.246+0000 7f2f450e9700 1 -- 172.21.15.46:0/70999273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f40159750 msgr2=0x7f2f400b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:44.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.246+0000 7f2f450e9700 1 --2- 172.21.15.46:0/70999273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f40159750 0x7f2f400b2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:44.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.246+0000 7f2f450e9700 1 -- 172.21.15.46:0/70999273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f30009420 con 0x7f2f400ae340 2024-01-26T23:30:44.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.246+0000 7f2f450e9700 1 --2- 172.21.15.46:0/70999273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f400ae340 0x7f2f40155910 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c00c790 tx=0x7f2f3c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:44.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.246+0000 7f2f357fa700 1 -- 172.21.15.46:0/70999273 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c010510 con 0x7f2f400ae340 2024-01-26T23:30:44.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.246+0000 7f2f357fa700 1 -- 172.21.15.46:0/70999273 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3c010670 con 0x7f2f400ae340 2024-01-26T23:30:44.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.246+0000 7f2f4734d700 1 -- 172.21.15.46:0/70999273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f400b3eb0 con 0x7f2f400ae340 2024-01-26T23:30:44.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.246+0000 7f2f357fa700 1 -- 172.21.15.46:0/70999273 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c010a10 con 0x7f2f400ae340 2024-01-26T23:30:44.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.246+0000 7f2f4734d700 1 -- 172.21.15.46:0/70999273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f40155e50 con 0x7f2f400ae340 2024-01-26T23:30:44.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.247+0000 7f2f4734d700 1 -- 172.21.15.46:0/70999273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f400b4220 con 0x7f2f400ae340 2024-01-26T23:30:44.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.248+0000 7f2f357fa700 1 -- 172.21.15.46:0/70999273 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2f3c01b400 con 0x7f2f400ae340 2024-01-26T23:30:44.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.248+0000 7f2f357fa700 1 --2- 172.21.15.46:0/70999273 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2f24066f30 0x7f2f240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:44.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.248+0000 7f2f357fa700 1 -- 172.21.15.46:0/70999273 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5777+0+0 (secure 0 0 0) 0x7f2f3c084350 con 0x7f2f400ae340 2024-01-26T23:30:44.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.249+0000 7f2f448e8700 1 --2- 172.21.15.46:0/70999273 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2f24066f30 0x7f2f240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:44.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.249+0000 7f2f448e8700 1 --2- 172.21.15.46:0/70999273 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2f24066f30 0x7f2f240693e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2f40159f20 tx=0x7f2f30016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:44.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.253+0000 7f2f357fa700 1 -- 172.21.15.46:0/70999273 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2f400b4220 con 0x7f2f400ae340 2024-01-26T23:30:44.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:44.456+0000 7f2f4734d700 1 -- 172.21.15.46:0/70999273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2f400b4220 con 0x7f2f400ae340 2024-01-26T23:30:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.072+0000 7f2f357fa700 1 -- 172.21.15.46:0/70999273 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v30) v1 ==== 178+0+0 (secure 0 0 0) 0x7f2f400b4220 con 0x7f2f400ae340 2024-01-26T23:30:45.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.077+0000 7f2f4734d700 1 -- 172.21.15.46:0/70999273 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2f24066f30 msgr2=0x7f2f240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:45.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.077+0000 7f2f4734d700 1 --2- 172.21.15.46:0/70999273 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2f24066f30 0x7f2f240693e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2f40159f20 tx=0x7f2f30016000 comp rx=0 tx=0).stop 2024-01-26T23:30:45.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.077+0000 7f2f4734d700 1 -- 172.21.15.46:0/70999273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f400ae340 msgr2=0x7f2f40155910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:45.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.077+0000 7f2f4734d700 1 --2- 172.21.15.46:0/70999273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f400ae340 0x7f2f40155910 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c00c790 tx=0x7f2f3c007e10 comp rx=0 tx=0).stop 2024-01-26T23:30:45.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.077+0000 7f2f4734d700 1 -- 172.21.15.46:0/70999273 shutdown_connections 2024-01-26T23:30:45.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.077+0000 7f2f4734d700 1 --2- 172.21.15.46:0/70999273 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2f24066f30 0x7f2f240693e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:45.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.077+0000 7f2f4734d700 1 --2- 172.21.15.46:0/70999273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f400ae340 0x7f2f40155910 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:45.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.077+0000 7f2f4734d700 1 --2- 172.21.15.46:0/70999273 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2f40159b20 0x7f2f400b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:45.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.077+0000 7f2f4734d700 1 --2- 172.21.15.46:0/70999273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f40159750 0x7f2f400b2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:45.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.077+0000 7f2f4734d700 1 -- 172.21.15.46:0/70999273 >> 172.21.15.46:0/70999273 conn(0x7f2f400ce530 msgr2=0x7f2f4006bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:45.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.077+0000 7f2f4734d700 1 -- 172.21.15.46:0/70999273 shutdown_connections 2024-01-26T23:30:45.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.077+0000 7f2f4734d700 1 -- 172.21.15.46:0/70999273 wait complete. 2024-01-26T23:30:45.080 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data_ec' removed 2024-01-26T23:30:45.096 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-26T23:30:45.096 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-26T23:30:45.096 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-26T23:30:45.097 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:30:45.097 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-26T23:30:45.097 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-26T23:30:45.097 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-26T23:30:45.097 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:30:45.097 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-26T23:30:45.097 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-26T23:30:45.098 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-26T23:30:45.098 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:30:45.098 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-26T23:30:45.098 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-26T23:30:45.098 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-26T23:30:45.098 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:30:45.110 INFO:tasks.ceph.mds.b:Stopped 2024-01-26T23:30:45.110 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-26T23:30:45.111 INFO:tasks.ceph.mds.d:Stopped 2024-01-26T23:30:45.112 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-26T23:30:45.113 INFO:tasks.ceph.mds.a:Stopped 2024-01-26T23:30:45.113 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-26T23:30:45.113 INFO:tasks.ceph.mds.c:Stopped 2024-01-26T23:30:45.114 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-26T23:30:45.152 INFO:tasks.ceph.mds.b:Started 2024-01-26T23:30:45.153 DEBUG:teuthology.parallel:result is None 2024-01-26T23:30:45.154 INFO:tasks.ceph.mds.d:Started 2024-01-26T23:30:45.154 DEBUG:teuthology.parallel:result is None 2024-01-26T23:30:45.155 INFO:tasks.ceph.mds.a:Started 2024-01-26T23:30:45.155 DEBUG:teuthology.parallel:result is None 2024-01-26T23:30:45.156 INFO:tasks.ceph.mds.c:Started 2024-01-26T23:30:45.156 DEBUG:teuthology.parallel:result is None 2024-01-26T23:30:45.156 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-26T23:30:45.236 INFO:tasks.ceph.mds.b.smithi106.stdout:starting mds.b at 2024-01-26T23:30:45.236 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-01-26T23:30:45.245 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-26T23:30:45.246 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-26T23:30:45.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.298+0000 7fa97fd87700 1 Processor -- start 2024-01-26T23:30:45.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.298+0000 7fa97fd87700 1 -- start start 2024-01-26T23:30:45.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.299+0000 7fa97fd87700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa9780b2530 0x7fa97815a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:45.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.299+0000 7fa97fd87700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9780a9b90 con 0x7fa97815da10 2024-01-26T23:30:45.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.299+0000 7fa97fd87700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9780aa210 con 0x7fa9780b2160 2024-01-26T23:30:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.299+0000 7fa97fd87700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa97815d580 con 0x7fa9780b2530 2024-01-26T23:30:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.299+0000 7fa97e324700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa97815da10 0x7fa97806a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54276/0 (socket says 172.21.15.46:54276) 2024-01-26T23:30:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.299+0000 7fa97e324700 1 -- 172.21.15.46:0/317656617 learned_addr learned my addr 172.21.15.46:0/317656617 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.299+0000 7fa97d322700 1 --2- 172.21.15.46:0/317656617 >> v2:172.21.15.106:3300/0 conn(0x7fa9780b2530 0x7fa97815a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.300+0000 7fa97cb21700 1 -- 172.21.15.46:0/317656617 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830812850 0 0) 0x7fa9780a9b90 con 0x7fa97815da10 2024-01-26T23:30:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.300+0000 7fa97cb21700 1 -- 172.21.15.46:0/317656617 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa964003570 con 0x7fa97815da10 2024-01-26T23:30:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.300+0000 7fa97cb21700 1 -- 172.21.15.46:0/317656617 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424970885 0 0) 0x7fa9780aa210 con 0x7fa9780b2160 2024-01-26T23:30:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.300+0000 7fa97cb21700 1 -- 172.21.15.46:0/317656617 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9780a9b90 con 0x7fa9780b2160 2024-01-26T23:30:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.300+0000 7fa97d322700 1 -- 172.21.15.46:0/317656617 >> v1:172.21.15.106:6789/0 conn(0x7fa9780b2160 legacy=0x7fa9780abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.300+0000 7fa97d322700 1 -- 172.21.15.46:0/317656617 >> v1:172.21.15.46:6789/0 conn(0x7fa97815da10 legacy=0x7fa97806a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.300+0000 7fa97d322700 1 -- 172.21.15.46:0/317656617 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9780aa460 con 0x7fa9780b2530 2024-01-26T23:30:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.300+0000 7fa97d322700 1 --2- 172.21.15.46:0/317656617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9780b2530 0x7fa97815a7a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa9680098e0 tx=0x7fa9680091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c52b8a70469412a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.300+0000 7fa97cb21700 1 -- 172.21.15.46:0/317656617 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa968011630 con 0x7fa9780b2530 2024-01-26T23:30:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.300+0000 7fa97cb21700 1 -- 172.21.15.46:0/317656617 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa968011790 con 0x7fa9780b2530 2024-01-26T23:30:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.300+0000 7fa97cb21700 1 -- 172.21.15.46:0/317656617 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa968011b30 con 0x7fa9780b2530 2024-01-26T23:30:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.301+0000 7fa97fd87700 1 -- 172.21.15.46:0/317656617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9780b2530 msgr2=0x7fa97815a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.301+0000 7fa97fd87700 1 --2- 172.21.15.46:0/317656617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9780b2530 0x7fa97815a7a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa9680098e0 tx=0x7fa9680091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.301+0000 7fa97fd87700 1 -- 172.21.15.46:0/317656617 shutdown_connections 2024-01-26T23:30:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.301+0000 7fa97fd87700 1 --2- 172.21.15.46:0/317656617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9780b2530 0x7fa97815a7a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.301+0000 7fa97fd87700 1 -- 172.21.15.46:0/317656617 >> 172.21.15.46:0/317656617 conn(0x7fa9780ccbc0 msgr2=0x7fa9780ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.301+0000 7fa97fd87700 1 -- 172.21.15.46:0/317656617 shutdown_connections 2024-01-26T23:30:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.301+0000 7fa97fd87700 1 -- 172.21.15.46:0/317656617 wait complete. 2024-01-26T23:30:45.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.303+0000 7fa97fd87700 1 Processor -- start 2024-01-26T23:30:45.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.303+0000 7fa97fd87700 1 -- start start 2024-01-26T23:30:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.304+0000 7fa97fd87700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9780b2160 0x7fa97815c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.304+0000 7fa97fd87700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9780b2530 0x7fa978158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.304+0000 7fa97fd87700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa97815da10 0x7fa978158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.304+0000 7fa97fd87700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9781591d0 con 0x7fa9780b2160 2024-01-26T23:30:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.304+0000 7fa97fd87700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9780ae520 con 0x7fa9780b2530 2024-01-26T23:30:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.304+0000 7fa97fd87700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9780ad670 con 0x7fa97815da10 2024-01-26T23:30:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.304+0000 7fa97db23700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9780b2160 0x7fa97815c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.304+0000 7fa97db23700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9780b2160 0x7fa97815c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47810/0 (socket says 172.21.15.46:47810) 2024-01-26T23:30:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.304+0000 7fa97db23700 1 -- 172.21.15.46:0/3559760516 learned_addr learned my addr 172.21.15.46:0/3559760516 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.304+0000 7fa97e324700 1 --2- 172.21.15.46:0/3559760516 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa97815da10 0x7fa978158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.304+0000 7fa97d322700 1 --2- 172.21.15.46:0/3559760516 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9780b2530 0x7fa978158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.304+0000 7fa97db23700 1 -- 172.21.15.46:0/3559760516 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa97815da10 msgr2=0x7fa978158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.304+0000 7fa97db23700 1 --2- 172.21.15.46:0/3559760516 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa97815da10 0x7fa978158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.304+0000 7fa97db23700 1 -- 172.21.15.46:0/3559760516 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9780b2530 msgr2=0x7fa978158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.304+0000 7fa97db23700 1 --2- 172.21.15.46:0/3559760516 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9780b2530 0x7fa978158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.304+0000 7fa97db23700 1 -- 172.21.15.46:0/3559760516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa968009420 con 0x7fa9780b2160 2024-01-26T23:30:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.304+0000 7fa97db23700 1 --2- 172.21.15.46:0/3559760516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9780b2160 0x7fa97815c8c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa97400c8f0 tx=0x7fa974002770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.305+0000 7fa96dffb700 1 -- 172.21.15.46:0/3559760516 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa974012630 con 0x7fa9780b2160 2024-01-26T23:30:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.305+0000 7fa96dffb700 1 -- 172.21.15.46:0/3559760516 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9740127b0 con 0x7fa9780b2160 2024-01-26T23:30:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.305+0000 7fa96dffb700 1 -- 172.21.15.46:0/3559760516 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa974012b70 con 0x7fa9780b2160 2024-01-26T23:30:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.305+0000 7fa97fd87700 1 -- 172.21.15.46:0/3559760516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9780ad910 con 0x7fa9780b2160 2024-01-26T23:30:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.305+0000 7fa97fd87700 1 -- 172.21.15.46:0/3559760516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9780ae1b0 con 0x7fa9780b2160 2024-01-26T23:30:45.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.306+0000 7fa97fd87700 1 -- 172.21.15.46:0/3559760516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9780ae800 con 0x7fa9780b2160 2024-01-26T23:30:45.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.306+0000 7fa96dffb700 1 -- 172.21.15.46:0/3559760516 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa97401b400 con 0x7fa9780b2160 2024-01-26T23:30:45.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.307+0000 7fa96dffb700 1 --2- 172.21.15.46:0/3559760516 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa95c066f30 0x7fa95c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:45.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.307+0000 7fa96dffb700 1 -- 172.21.15.46:0/3559760516 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5259+0+0 (secure 0 0 0) 0x7fa974084650 con 0x7fa9780b2160 2024-01-26T23:30:45.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.307+0000 7fa97d322700 1 --2- 172.21.15.46:0/3559760516 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa95c066f30 0x7fa95c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:45.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.308+0000 7fa97d322700 1 --2- 172.21.15.46:0/3559760516 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa95c066f30 0x7fa95c0693e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa9680093f0 tx=0x7fa968017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:45.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.311+0000 7fa96dffb700 1 -- 172.21.15.46:0/3559760516 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa974051b30 con 0x7fa9780b2160 2024-01-26T23:30:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:45.541+0000 7fa97fd87700 1 -- 172.21.15.46:0/3559760516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa9780add10 con 0x7fa9780b2160 2024-01-26T23:30:46.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.081+0000 7fa96dffb700 1 -- 172.21.15.46:0/3559760516 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v32) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa974055150 con 0x7fa9780b2160 2024-01-26T23:30:46.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.085+0000 7fa97fd87700 1 -- 172.21.15.46:0/3559760516 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa95c066f30 msgr2=0x7fa95c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:46.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.085+0000 7fa97fd87700 1 --2- 172.21.15.46:0/3559760516 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa95c066f30 0x7fa95c0693e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa9680093f0 tx=0x7fa968017000 comp rx=0 tx=0).stop 2024-01-26T23:30:46.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.085+0000 7fa97fd87700 1 -- 172.21.15.46:0/3559760516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9780b2160 msgr2=0x7fa97815c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:46.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.085+0000 7fa97fd87700 1 --2- 172.21.15.46:0/3559760516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9780b2160 0x7fa97815c8c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa97400c8f0 tx=0x7fa974002770 comp rx=0 tx=0).stop 2024-01-26T23:30:46.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.085+0000 7fa97fd87700 1 -- 172.21.15.46:0/3559760516 shutdown_connections 2024-01-26T23:30:46.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.085+0000 7fa97fd87700 1 --2- 172.21.15.46:0/3559760516 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa95c066f30 0x7fa95c0693e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.085+0000 7fa97fd87700 1 --2- 172.21.15.46:0/3559760516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9780b2160 0x7fa97815c8c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.085+0000 7fa97fd87700 1 --2- 172.21.15.46:0/3559760516 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa97815da10 0x7fa978158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.085+0000 7fa97fd87700 1 --2- 172.21.15.46:0/3559760516 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9780b2530 0x7fa978158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.085+0000 7fa97fd87700 1 -- 172.21.15.46:0/3559760516 >> 172.21.15.46:0/3559760516 conn(0x7fa9780ccbc0 msgr2=0x7fa9780b04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:46.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.085+0000 7fa97fd87700 1 -- 172.21.15.46:0/3559760516 shutdown_connections 2024-01-26T23:30:46.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.085+0000 7fa97fd87700 1 -- 172.21.15.46:0/3559760516 wait complete. 2024-01-26T23:30:46.088 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-01-26T23:30:46.103 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-26T23:30:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.236+0000 7fe3f6d79700 1 Processor -- start 2024-01-26T23:30:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.236+0000 7fe3f6d79700 1 -- start start 2024-01-26T23:30:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3f6d79700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3f0153ec0 0x7fe3f0154290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3f6d79700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3f0157630 0x7fe3f00b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3f6d79700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe3f00b2100 0x7fe3f006eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3f6d79700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f00b2580 con 0x7fe3f0153ec0 2024-01-26T23:30:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3f6d79700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f015b140 con 0x7fe3f00b2100 2024-01-26T23:30:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3f6d79700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f015a5c0 con 0x7fe3f0157630 2024-01-26T23:30:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3f4b15700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3f0153ec0 0x7fe3f0154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3f4b15700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3f0153ec0 0x7fe3f0154290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47822/0 (socket says 172.21.15.46:47822) 2024-01-26T23:30:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3f4b15700 1 -- 172.21.15.46:0/1489588611 learned_addr learned my addr 172.21.15.46:0/1489588611 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3effff700 1 --2- 172.21.15.46:0/1489588611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3f0157630 0x7fe3f00b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3f5316700 1 --2- 172.21.15.46:0/1489588611 >> v2:172.21.15.106:3300/0 conn(0x7fe3f00b2100 0x7fe3f006eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:46.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3f4b15700 1 -- 172.21.15.46:0/1489588611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3f0157630 msgr2=0x7fe3f00b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:46.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3f4b15700 1 --2- 172.21.15.46:0/1489588611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3f0157630 0x7fe3f00b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3f4b15700 1 -- 172.21.15.46:0/1489588611 >> v2:172.21.15.106:3300/0 conn(0x7fe3f00b2100 msgr2=0x7fe3f006eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:46.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3f4b15700 1 --2- 172.21.15.46:0/1489588611 >> v2:172.21.15.106:3300/0 conn(0x7fe3f00b2100 0x7fe3f006eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3f4b15700 1 -- 172.21.15.46:0/1489588611 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f015a8a0 con 0x7fe3f0153ec0 2024-01-26T23:30:46.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3f5316700 1 --2- 172.21.15.46:0/1489588611 >> v2:172.21.15.106:3300/0 conn(0x7fe3f00b2100 0x7fe3f006eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:46.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3f4b15700 1 --2- 172.21.15.46:0/1489588611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f0153ec0 0x7fe3f0154290 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe3f01547d0 tx=0x7fe3e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf1853832fc73fb8 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:46.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.237+0000 7fe3ef7fe700 1 -- 172.21.15.46:0/1489588611 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e400b860 con 0x7fe3f0153ec0 2024-01-26T23:30:46.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.238+0000 7fe3ef7fe700 1 -- 172.21.15.46:0/1489588611 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e400b9c0 con 0x7fe3f0153ec0 2024-01-26T23:30:46.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.238+0000 7fe3ef7fe700 1 -- 172.21.15.46:0/1489588611 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e400bd60 con 0x7fe3f0153ec0 2024-01-26T23:30:46.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.238+0000 7fe3f6d79700 1 -- 172.21.15.46:0/1489588611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f0153ec0 msgr2=0x7fe3f0154290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:46.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.238+0000 7fe3f6d79700 1 --2- 172.21.15.46:0/1489588611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f0153ec0 0x7fe3f0154290 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe3f01547d0 tx=0x7fe3e4009250 comp rx=0 tx=0).stop 2024-01-26T23:30:46.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.238+0000 7fe3f6d79700 1 -- 172.21.15.46:0/1489588611 shutdown_connections 2024-01-26T23:30:46.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.238+0000 7fe3f6d79700 1 --2- 172.21.15.46:0/1489588611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f0153ec0 0x7fe3f0154290 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.238+0000 7fe3f6d79700 1 --2- 172.21.15.46:0/1489588611 >> v2:172.21.15.106:3300/0 conn(0x7fe3f00b2100 0x7fe3f006eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.238+0000 7fe3f6d79700 1 --2- 172.21.15.46:0/1489588611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3f0157630 0x7fe3f00b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.238+0000 7fe3f6d79700 1 -- 172.21.15.46:0/1489588611 >> 172.21.15.46:0/1489588611 conn(0x7fe3f0069510 msgr2=0x7fe3f00aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:46.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.238+0000 7fe3f6d79700 1 -- 172.21.15.46:0/1489588611 shutdown_connections 2024-01-26T23:30:46.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.238+0000 7fe3f6d79700 1 -- 172.21.15.46:0/1489588611 wait complete. 2024-01-26T23:30:46.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.239+0000 7fe3f6d79700 1 Processor -- start 2024-01-26T23:30:46.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.240+0000 7fe3f6d79700 1 -- start start 2024-01-26T23:30:46.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.240+0000 7fe3f6d79700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f00b2100 0x7fe3f01584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:46.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.240+0000 7fe3f6d79700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3f0153ec0 0x7fe3f0170ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:46.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3f6d79700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3f0157630 0x7fe3f016f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3f6d79700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f0152260 con 0x7fe3f00b2100 2024-01-26T23:30:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3f6d79700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f016f060 con 0x7fe3f0157630 2024-01-26T23:30:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3f6d79700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f0171400 con 0x7fe3f0153ec0 2024-01-26T23:30:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3f4b15700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f00b2100 0x7fe3f01584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3f4b15700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f00b2100 0x7fe3f01584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47838/0 (socket says 172.21.15.46:47838) 2024-01-26T23:30:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3f4b15700 1 -- 172.21.15.46:0/1758950353 learned_addr learned my addr 172.21.15.46:0/1758950353 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3f5316700 1 --2- 172.21.15.46:0/1758950353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3f0157630 0x7fe3f016f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3effff700 1 --2- 172.21.15.46:0/1758950353 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3f0153ec0 0x7fe3f0170ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3f4b15700 1 -- 172.21.15.46:0/1758950353 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3f0153ec0 msgr2=0x7fe3f0170ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3f4b15700 1 --2- 172.21.15.46:0/1758950353 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3f0153ec0 0x7fe3f0170ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3f4b15700 1 -- 172.21.15.46:0/1758950353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3f0157630 msgr2=0x7fe3f016f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3f4b15700 1 --2- 172.21.15.46:0/1758950353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3f0157630 0x7fe3f016f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3f4b15700 1 -- 172.21.15.46:0/1758950353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e40094f0 con 0x7fe3f00b2100 2024-01-26T23:30:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3f5316700 1 --2- 172.21.15.46:0/1758950353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3f0157630 0x7fe3f016f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3f4b15700 1 --2- 172.21.15.46:0/1758950353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f00b2100 0x7fe3f01584b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009290 tx=0x7fe3e4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3ecff9700 1 -- 172.21.15.46:0/1758950353 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4026480 con 0x7fe3f00b2100 2024-01-26T23:30:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3ecff9700 1 -- 172.21.15.46:0/1758950353 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e401f070 con 0x7fe3f00b2100 2024-01-26T23:30:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3f6d79700 1 -- 172.21.15.46:0/1758950353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f01596d0 con 0x7fe3f00b2100 2024-01-26T23:30:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.241+0000 7fe3ecff9700 1 -- 172.21.15.46:0/1758950353 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4026830 con 0x7fe3f00b2100 2024-01-26T23:30:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.242+0000 7fe3f6d79700 1 -- 172.21.15.46:0/1758950353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f016e4e0 con 0x7fe3f00b2100 2024-01-26T23:30:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.242+0000 7fe3f6d79700 1 -- 172.21.15.46:0/1758950353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f004c710 con 0x7fe3f00b2100 2024-01-26T23:30:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.243+0000 7fe3ecff9700 1 -- 172.21.15.46:0/1758950353 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe3e4026ba0 con 0x7fe3f00b2100 2024-01-26T23:30:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.243+0000 7fe3ecff9700 1 --2- 172.21.15.46:0/1758950353 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe3d4066f30 0x7fe3d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.243+0000 7fe3ecff9700 1 -- 172.21.15.46:0/1758950353 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7fe3e4021070 con 0x7fe3f00b2100 2024-01-26T23:30:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.244+0000 7fe3effff700 1 --2- 172.21.15.46:0/1758950353 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe3d4066f30 0x7fe3d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:46.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.244+0000 7fe3effff700 1 --2- 172.21.15.46:0/1758950353 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe3d4066f30 0x7fe3d40693e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc006fd0 tx=0x7fe3dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:46.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.248+0000 7fe3ecff9700 1 -- 172.21.15.46:0/1758950353 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe3e4053650 con 0x7fe3f00b2100 2024-01-26T23:30:46.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.533+0000 7fe3f6d79700 1 -- 172.21.15.46:0/1758950353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe3f00ce060 con 0x7fe3f00b2100 2024-01-26T23:30:46.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.534+0000 7fe3ecff9700 1 -- 172.21.15.46:0/1758950353 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fe3e4056c70 con 0x7fe3f00b2100 2024-01-26T23:30:46.536 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:46.536 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:30:46.536 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-01-26T23:30:46.536 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.536 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-01-26T23:30:46.536 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlM6dKKBAAPpeWeDZWrFgc/4zuxKCvHw==", 2024-01-26T23:30:46.536 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.536 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:30:46.536 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:30:46.536 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:30:46.536 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:30:46.536 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.536 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.537 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.537 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-01-26T23:30:46.537 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlmzHRNRAAflWYZ/p4AXeMy54tvLpugA==", 2024-01-26T23:30:46.537 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.537 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:30:46.537 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:30:46.537 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:30:46.537 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:30:46.537 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.537 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.537 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.537 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-01-26T23:30:46.537 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlMZxoMBAAG5kyfm+LLG86y/p2A8BfDQ==", 2024-01-26T23:30:46.537 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.538 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:30:46.538 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:30:46.538 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:30:46.538 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:30:46.538 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.538 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.538 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.538 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-01-26T23:30:46.538 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl5jx/ARAA3cwtLAOWQ/n2NG+xwEZ3QQ==", 2024-01-26T23:30:46.538 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.538 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:30:46.538 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:30:46.538 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:30:46.538 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:30:46.538 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.539 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.539 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.539 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-01-26T23:30:46.539 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4QLRlJ2OKJRAAWVhxEdQkUW20294w/KLL6g==", 2024-01-26T23:30:46.539 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.539 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:30:46.539 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:30:46.539 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:30:46.539 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.539 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.539 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.539 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-01-26T23:30:46.539 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7QLRlE2tAJRAAzNlTiCr6a2eWiabRy1SqSw==", 2024-01-26T23:30:46.539 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.539 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:30:46.540 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:30:46.540 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:30:46.540 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.540 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.540 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.540 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-01-26T23:30:46.540 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+QLRlyySaJBAA5072ba5HUNi2dYE3D02M/Q==", 2024-01-26T23:30:46.540 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.540 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:30:46.540 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:30:46.540 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:30:46.540 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.540 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.540 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.541 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-01-26T23:30:46.541 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBQLRl9+tAIxAApZk0btXIpQPWSb3BYUYsFQ==", 2024-01-26T23:30:46.541 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.541 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:30:46.541 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:30:46.541 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:30:46.541 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.541 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.541 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.541 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-01-26T23:30:46.541 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHQLRlurUtJxAAT9fwKrAWl62VOXwShg7NJg==", 2024-01-26T23:30:46.541 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.541 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:30:46.541 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:30:46.542 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:30:46.542 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.542 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.542 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.542 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-01-26T23:30:46.542 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKQLRl7aqoJRAAXu9fykWELsW3SwnahoIOLA==", 2024-01-26T23:30:46.542 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.542 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:30:46.542 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:30:46.542 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:30:46.542 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.542 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.542 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.542 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-01-26T23:30:46.542 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNQLRlWf7pJBAAopmFLIKb8s32YZs46VwbGg==", 2024-01-26T23:30:46.543 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.543 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:30:46.543 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:30:46.543 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:30:46.543 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.543 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.543 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.543 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-01-26T23:30:46.543 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQQLRlNGfhIxAAmCgmOB3QAGfSIYHBb5BUZw==", 2024-01-26T23:30:46.543 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.543 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:30:46.543 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:30:46.543 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:30:46.543 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.543 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.544 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.544 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-01-26T23:30:46.544 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRleO8wBhAA1018AcO3KcyOZxgB0Itxaw==", 2024-01-26T23:30:46.544 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.544 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:30:46.544 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:30:46.544 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:30:46.544 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:30:46.544 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.544 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.544 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.544 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-01-26T23:30:46.544 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlsiZEDRAAjXSamHe2bZcDqVsqXqIcGg==", 2024-01-26T23:30:46.544 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:30:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:30:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:30:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:30:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-01-26T23:30:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl6YT1CRAApXlN4j20xzwda0tDMy94Eg==", 2024-01-26T23:30:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:30:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:30:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:30:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:30:46.545 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-01-26T23:30:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlHvDQEhAAxMhgWJsZC5/6Af5zFV6XIA==", 2024-01-26T23:30:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:30:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:30:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:30:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:30:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-01-26T23:30:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlvMwDCBAAXnFyMduz7PfWdLWbjytIiQ==", 2024-01-26T23:30:46.546 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.547 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:30:46.547 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:30:46.547 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:30:46.547 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:30:46.547 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.547 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.547 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.547 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-mds", 2024-01-26T23:30:46.547 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBWQLRlEKwvKBAAJKfkOg/fkV1E3uD8hSB84g==", 2024-01-26T23:30:46.547 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.547 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-mds" 2024-01-26T23:30:46.547 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.547 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.547 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.547 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-mgr", 2024-01-26T23:30:46.548 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBWQLRlXr0vKBAAtB7bDz4GLXPb1kbolkW3HA==", 2024-01-26T23:30:46.548 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.548 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-mgr" 2024-01-26T23:30:46.548 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.548 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.548 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.548 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-osd", 2024-01-26T23:30:46.548 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBWQLRlHc4vKBAApyo2FcV5nZPV2G+m5xMWTw==", 2024-01-26T23:30:46.548 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.548 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-osd" 2024-01-26T23:30:46.548 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.548 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.548 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.548 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-rbd", 2024-01-26T23:30:46.549 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBWQLRludcvKBAAMXUrsCwQxOVekvMnSBghUg==", 2024-01-26T23:30:46.549 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.549 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-rbd" 2024-01-26T23:30:46.549 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.549 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.549 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.549 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-01-26T23:30:46.549 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBWQLRlg+EvKBAAqW+kgILF9pHF+Z1SuADbww==", 2024-01-26T23:30:46.549 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.549 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-01-26T23:30:46.549 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.549 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.549 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.549 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-rgw", 2024-01-26T23:30:46.549 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBWQLRl1eovKBAAnoGKmqCcWa0wwnJ5ehWE2Q==", 2024-01-26T23:30:46.550 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.550 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-rgw" 2024-01-26T23:30:46.550 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.550 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.550 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.550 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-01-26T23:30:46.550 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlhMNSJBAA38QU6DrCvFjO+u9zQ8bkEA==", 2024-01-26T23:30:46.550 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.550 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:30:46.550 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-26T23:30:46.550 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:30:46.550 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.550 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:30:46.550 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:46.550 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-01-26T23:30:46.551 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlI9EbIBAACTpzAO+8A/VFJZcT/WWBng==", 2024-01-26T23:30:46.551 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:30:46.551 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:30:46.551 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-26T23:30:46.551 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:30:46.551 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.551 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:46.551 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-01-26T23:30:46.551 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:30:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.536+0000 7fe3f6d79700 1 -- 172.21.15.46:0/1758950353 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe3d4066f30 msgr2=0x7fe3d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.536+0000 7fe3f6d79700 1 --2- 172.21.15.46:0/1758950353 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe3d4066f30 0x7fe3d40693e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc006fd0 tx=0x7fe3dc008040 comp rx=0 tx=0).stop 2024-01-26T23:30:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.536+0000 7fe3f6d79700 1 -- 172.21.15.46:0/1758950353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f00b2100 msgr2=0x7fe3f01584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.536+0000 7fe3f6d79700 1 --2- 172.21.15.46:0/1758950353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f00b2100 0x7fe3f01584b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009290 tx=0x7fe3e4017000 comp rx=0 tx=0).stop 2024-01-26T23:30:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.536+0000 7fe3f6d79700 1 -- 172.21.15.46:0/1758950353 shutdown_connections 2024-01-26T23:30:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.536+0000 7fe3f6d79700 1 --2- 172.21.15.46:0/1758950353 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe3d4066f30 0x7fe3d40693e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.536+0000 7fe3f6d79700 1 --2- 172.21.15.46:0/1758950353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f00b2100 0x7fe3f01584b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.536+0000 7fe3f6d79700 1 --2- 172.21.15.46:0/1758950353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3f0157630 0x7fe3f016f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.536+0000 7fe3f6d79700 1 --2- 172.21.15.46:0/1758950353 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3f0153ec0 0x7fe3f0170ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.536+0000 7fe3f6d79700 1 -- 172.21.15.46:0/1758950353 >> 172.21.15.46:0/1758950353 conn(0x7fe3f0069510 msgr2=0x7fe3f0154800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.536+0000 7fe3f6d79700 1 -- 172.21.15.46:0/1758950353 shutdown_connections 2024-01-26T23:30:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.536+0000 7fe3f6d79700 1 -- 172.21.15.46:0/1758950353 wait complete. 2024-01-26T23:30:46.554 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-01-26T23:30:46.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.684+0000 7f4cf8f5c700 1 Processor -- start 2024-01-26T23:30:46.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.684+0000 7f4cf8f5c700 1 -- start start 2024-01-26T23:30:46.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.685+0000 7f4cf8f5c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf4157340 0x7f4cf415bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:46.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.685+0000 7f4cf8f5c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4cf40b4180 0x7f4cf4157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:46.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.685+0000 7f4cf8f5c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cf4158ae0 con 0x7f4cf40ae520 2024-01-26T23:30:46.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.685+0000 7f4cf8f5c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cf415b650 con 0x7f4cf40b4180 2024-01-26T23:30:46.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.685+0000 7f4cf8f5c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf415c020 con 0x7f4cf4157340 2024-01-26T23:30:46.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.686+0000 7f4cf2d9d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4cf40b4180 0x7f4cf4157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:46.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.686+0000 7f4cf259c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf4157340 0x7f4cf415bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:46.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.686+0000 7f4cf1d9b700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f4cf40ae520 0x7f4cf40b3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:51946/0 (socket says 172.21.15.46:51946) 2024-01-26T23:30:46.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.686+0000 7f4cf1d9b700 1 -- 172.21.15.46:0/2640581411 learned_addr learned my addr 172.21.15.46:0/2640581411 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:46.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.686+0000 7f4cf2d9d700 1 -- 172.21.15.46:0/2640581411 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf4157340 msgr2=0x7f4cf415bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:46.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.686+0000 7f4cf2d9d700 1 --2- 172.21.15.46:0/2640581411 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf4157340 0x7f4cf415bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.686+0000 7f4cf2d9d700 1 -- 172.21.15.46:0/2640581411 >> v1:172.21.15.106:6789/0 conn(0x7f4cf40ae520 legacy=0x7f4cf40b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:46.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.686+0000 7f4cf2d9d700 1 -- 172.21.15.46:0/2640581411 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf4158520 con 0x7f4cf40b4180 2024-01-26T23:30:46.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.686+0000 7f4cf159a700 1 -- 172.21.15.46:0/2640581411 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700550376 0 0) 0x7f4cf4158ae0 con 0x7f4cf40ae520 2024-01-26T23:30:46.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.686+0000 7f4cf259c700 1 --2- 172.21.15.46:0/2640581411 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf4157340 0x7f4cf415bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:46.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.687+0000 7f4cf2d9d700 1 --2- 172.21.15.46:0/2640581411 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf40b4180 0x7f4cf4157710 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009e60 tx=0x7f4cec009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fbc521dea7775a7 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:46.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.687+0000 7f4cf159a700 1 -- 172.21.15.46:0/2640581411 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec00e070 con 0x7f4cf40b4180 2024-01-26T23:30:46.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.687+0000 7f4cf159a700 1 -- 172.21.15.46:0/2640581411 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cec00bb60 con 0x7f4cf40b4180 2024-01-26T23:30:46.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.687+0000 7f4cf159a700 1 -- 172.21.15.46:0/2640581411 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec012430 con 0x7f4cf40b4180 2024-01-26T23:30:46.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.688+0000 7f4cf8f5c700 1 -- 172.21.15.46:0/2640581411 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf40b4180 msgr2=0x7f4cf4157710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.688+0000 7f4cf8f5c700 1 --2- 172.21.15.46:0/2640581411 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf40b4180 0x7f4cf4157710 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009e60 tx=0x7f4cec009780 comp rx=0 tx=0).stop 2024-01-26T23:30:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.688+0000 7f4cf8f5c700 1 -- 172.21.15.46:0/2640581411 shutdown_connections 2024-01-26T23:30:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.688+0000 7f4cf8f5c700 1 --2- 172.21.15.46:0/2640581411 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf40b4180 0x7f4cf4157710 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.688+0000 7f4cf8f5c700 1 --2- 172.21.15.46:0/2640581411 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf4157340 0x7f4cf415bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.688+0000 7f4cf8f5c700 1 -- 172.21.15.46:0/2640581411 >> 172.21.15.46:0/2640581411 conn(0x7f4cf40ce530 msgr2=0x7f4cf40ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.688+0000 7f4cf8f5c700 1 -- 172.21.15.46:0/2640581411 shutdown_connections 2024-01-26T23:30:46.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.688+0000 7f4cf8f5c700 1 -- 172.21.15.46:0/2640581411 wait complete. 2024-01-26T23:30:46.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.689+0000 7f4cf8f5c700 1 Processor -- start 2024-01-26T23:30:46.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.689+0000 7f4cf8f5c700 1 -- start start 2024-01-26T23:30:46.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.690+0000 7f4cf8f5c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf40ae520 0x7f4cf4068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:46.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.690+0000 7f4cf8f5c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf40b4180 0x7f4cf4154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:46.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.690+0000 7f4cf8f5c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf4157340 0x7f4cf40c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:46.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.690+0000 7f4cf8f5c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf40b2b90 con 0x7f4cf40ae520 2024-01-26T23:30:46.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.690+0000 7f4cf8f5c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf4068f80 con 0x7f4cf4157340 2024-01-26T23:30:46.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.690+0000 7f4cf8f5c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf4154540 con 0x7f4cf40b4180 2024-01-26T23:30:46.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.690+0000 7f4cf259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf40ae520 0x7f4cf4068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:46.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.690+0000 7f4cf259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf40ae520 0x7f4cf4068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47840/0 (socket says 172.21.15.46:47840) 2024-01-26T23:30:46.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.690+0000 7f4cf259c700 1 -- 172.21.15.46:0/3803251810 learned_addr learned my addr 172.21.15.46:0/3803251810 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:46.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.690+0000 7f4cf1d9b700 1 --2- 172.21.15.46:0/3803251810 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf40b4180 0x7f4cf4154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:46.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.690+0000 7f4cf2d9d700 1 --2- 172.21.15.46:0/3803251810 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf4157340 0x7f4cf40c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:46.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.690+0000 7f4cf259c700 1 -- 172.21.15.46:0/3803251810 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf40b4180 msgr2=0x7f4cf4154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:46.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.690+0000 7f4cf259c700 1 --2- 172.21.15.46:0/3803251810 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf40b4180 0x7f4cf4154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.690+0000 7f4cf259c700 1 -- 172.21.15.46:0/3803251810 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf4157340 msgr2=0x7f4cf40c1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:46.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.690+0000 7f4cf259c700 1 --2- 172.21.15.46:0/3803251810 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf4157340 0x7f4cf40c1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.690+0000 7f4cf259c700 1 -- 172.21.15.46:0/3803251810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce4009420 con 0x7f4cf40ae520 2024-01-26T23:30:46.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.690+0000 7f4cf259c700 1 --2- 172.21.15.46:0/3803251810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf40ae520 0x7f4cf4068a40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4ce400ec80 tx=0x7f4ce40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:46.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.691+0000 7f4ce27fc700 1 -- 172.21.15.46:0/3803251810 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce400c9d0 con 0x7f4cf40ae520 2024-01-26T23:30:46.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.691+0000 7f4ce27fc700 1 -- 172.21.15.46:0/3803251810 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce400cb30 con 0x7f4cf40ae520 2024-01-26T23:30:46.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.691+0000 7f4ce27fc700 1 -- 172.21.15.46:0/3803251810 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce400ceb0 con 0x7f4cf40ae520 2024-01-26T23:30:46.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.691+0000 7f4cf8f5c700 1 -- 172.21.15.46:0/3803251810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cec009a20 con 0x7f4cf40ae520 2024-01-26T23:30:46.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.691+0000 7f4cf8f5c700 1 -- 172.21.15.46:0/3803251810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf4069500 con 0x7f4cf40ae520 2024-01-26T23:30:46.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.692+0000 7f4cf8f5c700 1 -- 172.21.15.46:0/3803251810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf404c710 con 0x7f4cf40ae520 2024-01-26T23:30:46.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.692+0000 7f4ce27fc700 1 -- 172.21.15.46:0/3803251810 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4ce4013680 con 0x7f4cf40ae520 2024-01-26T23:30:46.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.693+0000 7f4ce27fc700 1 --2- 172.21.15.46:0/3803251810 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4cd4066f30 0x7f4cd40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:46.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.693+0000 7f4ce27fc700 1 -- 172.21.15.46:0/3803251810 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f4ce4084760 con 0x7f4cf40ae520 2024-01-26T23:30:46.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.693+0000 7f4cf1d9b700 1 --2- 172.21.15.46:0/3803251810 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4cd4066f30 0x7f4cd40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:46.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.693+0000 7f4cf1d9b700 1 --2- 172.21.15.46:0/3803251810 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4cd4066f30 0x7f4cd40693e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4cf416d000 tx=0x7f4ce8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:46.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.697+0000 7f4ce27fc700 1 -- 172.21.15.46:0/3803251810 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4ce4051be0 con 0x7f4cf40ae520 2024-01-26T23:30:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.987+0000 7f4cf8f5c700 1 -- 172.21.15.46:0/3803251810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f4cf4158800 con 0x7f4cf40ae520 2024-01-26T23:30:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.990+0000 7f4ce27fc700 1 -- 172.21.15.46:0/3803251810 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f4ce4055200 con 0x7f4cf40ae520 2024-01-26T23:30:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.993+0000 7f4cf8f5c700 1 -- 172.21.15.46:0/3803251810 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4cd4066f30 msgr2=0x7f4cd40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.993+0000 7f4cf8f5c700 1 --2- 172.21.15.46:0/3803251810 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4cd4066f30 0x7f4cd40693e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4cf416d000 tx=0x7f4ce8009040 comp rx=0 tx=0).stop 2024-01-26T23:30:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.993+0000 7f4cf8f5c700 1 -- 172.21.15.46:0/3803251810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf40ae520 msgr2=0x7f4cf4068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.993+0000 7f4cf8f5c700 1 --2- 172.21.15.46:0/3803251810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf40ae520 0x7f4cf4068a40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4ce400ec80 tx=0x7f4ce40091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.993+0000 7f4cf8f5c700 1 -- 172.21.15.46:0/3803251810 shutdown_connections 2024-01-26T23:30:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.993+0000 7f4cf8f5c700 1 --2- 172.21.15.46:0/3803251810 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4cd4066f30 0x7f4cd40693e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.993+0000 7f4cf8f5c700 1 --2- 172.21.15.46:0/3803251810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf40ae520 0x7f4cf4068a40 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.993+0000 7f4cf8f5c700 1 --2- 172.21.15.46:0/3803251810 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf4157340 0x7f4cf40c1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.993+0000 7f4cf8f5c700 1 --2- 172.21.15.46:0/3803251810 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf40b4180 0x7f4cf4154000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.993+0000 7f4cf8f5c700 1 -- 172.21.15.46:0/3803251810 >> 172.21.15.46:0/3803251810 conn(0x7f4cf40ce530 msgr2=0x7f4cf406d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.993+0000 7f4cf8f5c700 1 -- 172.21.15.46:0/3803251810 shutdown_connections 2024-01-26T23:30:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:46.993+0000 7f4cf8f5c700 1 -- 172.21.15.46:0/3803251810 wait complete. 2024-01-26T23:30:47.008 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-01-26T23:30:47.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.140+0000 7fa857325700 1 Processor -- start 2024-01-26T23:30:47.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.140+0000 7fa857325700 1 -- start start 2024-01-26T23:30:47.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.141+0000 7fa857325700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa85006ab20 0x7fa8500b0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:47.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.141+0000 7fa857325700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8501597b0 con 0x7fa85014ffb0 2024-01-26T23:30:47.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.141+0000 7fa857325700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85015a330 con 0x7fa850150380 2024-01-26T23:30:47.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.141+0000 7fa857325700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa85015a7c0 con 0x7fa85006ab20 2024-01-26T23:30:47.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.142+0000 7fa8550c1700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa85014ffb0 0x7fa85006a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54284/0 (socket says 172.21.15.46:54284) 2024-01-26T23:30:47.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.142+0000 7fa8550c1700 1 -- 172.21.15.46:0/2043278153 learned_addr learned my addr 172.21.15.46:0/2043278153 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:47.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.142+0000 7fa8548c0700 1 --2- 172.21.15.46:0/2043278153 >> v2:172.21.15.106:3300/0 conn(0x7fa85006ab20 0x7fa8500b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:47.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.142+0000 7fa843fff700 1 -- 172.21.15.46:0/2043278153 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502907053 0 0) 0x7fa8501597b0 con 0x7fa85014ffb0 2024-01-26T23:30:47.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.142+0000 7fa843fff700 1 -- 172.21.15.46:0/2043278153 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa83c003570 con 0x7fa85014ffb0 2024-01-26T23:30:47.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.142+0000 7fa843fff700 1 -- 172.21.15.46:0/2043278153 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376726379 0 0) 0x7fa85015a330 con 0x7fa850150380 2024-01-26T23:30:47.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.142+0000 7fa843fff700 1 -- 172.21.15.46:0/2043278153 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8501597b0 con 0x7fa850150380 2024-01-26T23:30:47.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.142+0000 7fa8548c0700 1 -- 172.21.15.46:0/2043278153 >> v1:172.21.15.106:6789/0 conn(0x7fa850150380 legacy=0x7fa8500b0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:47.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.142+0000 7fa8548c0700 1 -- 172.21.15.46:0/2043278153 >> v1:172.21.15.46:6789/0 conn(0x7fa85014ffb0 legacy=0x7fa85006a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:47.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.142+0000 7fa8548c0700 1 -- 172.21.15.46:0/2043278153 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa850159a00 con 0x7fa85006ab20 2024-01-26T23:30:47.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.142+0000 7fa8548c0700 1 --2- 172.21.15.46:0/2043278153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa85006ab20 0x7fa8500b0890 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00aa90 tx=0x7fa84c00a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c213193ddf8d85a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:47.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.142+0000 7fa843fff700 1 -- 172.21.15.46:0/2043278153 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c00c970 con 0x7fa85006ab20 2024-01-26T23:30:47.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.143+0000 7fa843fff700 1 -- 172.21.15.46:0/2043278153 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa84c00cad0 con 0x7fa85006ab20 2024-01-26T23:30:47.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.143+0000 7fa843fff700 1 -- 172.21.15.46:0/2043278153 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c00ce70 con 0x7fa85006ab20 2024-01-26T23:30:47.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.143+0000 7fa857325700 1 -- 172.21.15.46:0/2043278153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa85006ab20 msgr2=0x7fa8500b0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:47.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.143+0000 7fa857325700 1 --2- 172.21.15.46:0/2043278153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa85006ab20 0x7fa8500b0890 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00aa90 tx=0x7fa84c00a330 comp rx=0 tx=0).stop 2024-01-26T23:30:47.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.143+0000 7fa857325700 1 -- 172.21.15.46:0/2043278153 shutdown_connections 2024-01-26T23:30:47.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.143+0000 7fa857325700 1 --2- 172.21.15.46:0/2043278153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa85006ab20 0x7fa8500b0890 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:47.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.143+0000 7fa857325700 1 -- 172.21.15.46:0/2043278153 >> 172.21.15.46:0/2043278153 conn(0x7fa8500ccbc0 msgr2=0x7fa8500ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:47.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.143+0000 7fa857325700 1 -- 172.21.15.46:0/2043278153 shutdown_connections 2024-01-26T23:30:47.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.143+0000 7fa857325700 1 -- 172.21.15.46:0/2043278153 wait complete. 2024-01-26T23:30:47.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.144+0000 7fa857325700 1 Processor -- start 2024-01-26T23:30:47.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.144+0000 7fa857325700 1 -- start start 2024-01-26T23:30:47.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.145+0000 7fa857325700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa85006ab20 0x7fa85014df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:47.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.145+0000 7fa857325700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa85014ffb0 0x7fa850169910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:47.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.145+0000 7fa857325700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa850150380 0x7fa850169e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:47.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.145+0000 7fa857325700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85016ae50 con 0x7fa85014ffb0 2024-01-26T23:30:47.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.145+0000 7fa857325700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85014e8a0 con 0x7fa850150380 2024-01-26T23:30:47.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.145+0000 7fa857325700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa85014eb80 con 0x7fa85006ab20 2024-01-26T23:30:47.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.146+0000 7fa8548c0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa85014ffb0 0x7fa850169910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:47.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.146+0000 7fa8548c0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa85014ffb0 0x7fa850169910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47852/0 (socket says 172.21.15.46:47852) 2024-01-26T23:30:47.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.146+0000 7fa8548c0700 1 -- 172.21.15.46:0/1332356318 learned_addr learned my addr 172.21.15.46:0/1332356318 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:47.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.146+0000 7fa8550c1700 1 --2- 172.21.15.46:0/1332356318 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa85006ab20 0x7fa85014df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:47.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.146+0000 7fa8558c2700 1 --2- 172.21.15.46:0/1332356318 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa850150380 0x7fa850169e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:47.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.146+0000 7fa8548c0700 1 -- 172.21.15.46:0/1332356318 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa85006ab20 msgr2=0x7fa85014df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.146+0000 7fa8548c0700 1 --2- 172.21.15.46:0/1332356318 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa85006ab20 0x7fa85014df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.146+0000 7fa8548c0700 1 -- 172.21.15.46:0/1332356318 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa850150380 msgr2=0x7fa850169e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.146+0000 7fa8548c0700 1 --2- 172.21.15.46:0/1332356318 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa850150380 0x7fa850169e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.146+0000 7fa8548c0700 1 -- 172.21.15.46:0/1332356318 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84c00a5d0 con 0x7fa85014ffb0 2024-01-26T23:30:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.146+0000 7fa8548c0700 1 --2- 172.21.15.46:0/1332356318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa85014ffb0 0x7fa850169910 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00a5a0 tx=0x7fa84c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.146+0000 7fa8417fa700 1 -- 172.21.15.46:0/1332356318 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c018da0 con 0x7fa85014ffb0 2024-01-26T23:30:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.146+0000 7fa8417fa700 1 -- 172.21.15.46:0/1332356318 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa84c021420 con 0x7fa85014ffb0 2024-01-26T23:30:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.146+0000 7fa857325700 1 -- 172.21.15.46:0/1332356318 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa85014ee60 con 0x7fa85014ffb0 2024-01-26T23:30:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.146+0000 7fa8417fa700 1 -- 172.21.15.46:0/1332356318 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c0217e0 con 0x7fa85014ffb0 2024-01-26T23:30:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.146+0000 7fa857325700 1 -- 172.21.15.46:0/1332356318 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8500ac240 con 0x7fa85014ffb0 2024-01-26T23:30:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.148+0000 7fa857325700 1 -- 172.21.15.46:0/1332356318 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa85016b000 con 0x7fa85014ffb0 2024-01-26T23:30:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.148+0000 7fa8417fa700 1 -- 172.21.15.46:0/1332356318 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa84c021bd0 con 0x7fa85014ffb0 2024-01-26T23:30:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.149+0000 7fa8417fa700 1 --2- 172.21.15.46:0/1332356318 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa834066f30 0x7fa8340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.149+0000 7fa8417fa700 1 -- 172.21.15.46:0/1332356318 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7fa84c0858b0 con 0x7fa85014ffb0 2024-01-26T23:30:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.149+0000 7fa8550c1700 1 --2- 172.21.15.46:0/1332356318 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa834066f30 0x7fa8340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.149+0000 7fa8550c1700 1 --2- 172.21.15.46:0/1332356318 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa834066f30 0x7fa8340693e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa844007fd0 tx=0x7fa844009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:47.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.153+0000 7fa8417fa700 1 -- 172.21.15.46:0/1332356318 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa85016b000 con 0x7fa85014ffb0 2024-01-26T23:30:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.441+0000 7fa857325700 1 -- 172.21.15.46:0/1332356318 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fa85004c710 con 0x7fa85014ffb0 2024-01-26T23:30:47.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.445+0000 7fa8417fa700 1 -- 172.21.15.46:0/1332356318 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7fa84c052de0 con 0x7fa85014ffb0 2024-01-26T23:30:47.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.447+0000 7fa857325700 1 -- 172.21.15.46:0/1332356318 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa834066f30 msgr2=0x7fa8340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:47.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.447+0000 7fa857325700 1 --2- 172.21.15.46:0/1332356318 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa834066f30 0x7fa8340693e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa844007fd0 tx=0x7fa844009040 comp rx=0 tx=0).stop 2024-01-26T23:30:47.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.447+0000 7fa857325700 1 -- 172.21.15.46:0/1332356318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa85014ffb0 msgr2=0x7fa850169910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:47.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.447+0000 7fa857325700 1 --2- 172.21.15.46:0/1332356318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa85014ffb0 0x7fa850169910 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00a5a0 tx=0x7fa84c01a000 comp rx=0 tx=0).stop 2024-01-26T23:30:47.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.447+0000 7fa857325700 1 -- 172.21.15.46:0/1332356318 shutdown_connections 2024-01-26T23:30:47.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.447+0000 7fa857325700 1 --2- 172.21.15.46:0/1332356318 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa834066f30 0x7fa8340693e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:47.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.447+0000 7fa857325700 1 --2- 172.21.15.46:0/1332356318 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa850150380 0x7fa850169e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:47.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.447+0000 7fa857325700 1 --2- 172.21.15.46:0/1332356318 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa85006ab20 0x7fa85014df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:47.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.447+0000 7fa857325700 1 --2- 172.21.15.46:0/1332356318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa85014ffb0 0x7fa850169910 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:47.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.447+0000 7fa857325700 1 -- 172.21.15.46:0/1332356318 >> 172.21.15.46:0/1332356318 conn(0x7fa8500ccbc0 msgr2=0x7fa8500afd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:47.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.447+0000 7fa857325700 1 -- 172.21.15.46:0/1332356318 shutdown_connections 2024-01-26T23:30:47.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.447+0000 7fa857325700 1 -- 172.21.15.46:0/1332356318 wait complete. 2024-01-26T23:30:47.462 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-01-26T23:30:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.593+0000 7fd1024ce700 1 Processor -- start 2024-01-26T23:30:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.594+0000 7fd1024ce700 1 -- start start 2024-01-26T23:30:47.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.594+0000 7fd1024ce700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0fc0ae520 0x7fd0fc0b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:47.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.594+0000 7fd1024ce700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd0fc0b3fb0 0x7fd0fc157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:47.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.594+0000 7fd1024ce700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0fc0b4410 con 0x7fd0fc157340 2024-01-26T23:30:47.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.594+0000 7fd1024ce700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0fc158ae0 con 0x7fd0fc0ae520 2024-01-26T23:30:47.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.594+0000 7fd1024ce700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc15b650 con 0x7fd0fc0b3fb0 2024-01-26T23:30:47.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.595+0000 7fd0fb7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0fc0ae520 0x7fd0fc0b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:47.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.595+0000 7fd0fbfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd0fc157340 0x7fd0fc15bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54296/0 (socket says 172.21.15.46:54296) 2024-01-26T23:30:47.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.595+0000 7fd0fb7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0fc0ae520 0x7fd0fc0b3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47854/0 (socket says 172.21.15.46:47854) 2024-01-26T23:30:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.595+0000 7fd0fbfff700 1 -- 172.21.15.46:0/3202592501 learned_addr learned my addr 172.21.15.46:0/3202592501 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.595+0000 7fd100a6b700 1 --2- 172.21.15.46:0/3202592501 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd0fc0b3fb0 0x7fd0fc157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.595+0000 7fd0fb7fe700 1 -- 172.21.15.46:0/3202592501 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd0fc0b3fb0 msgr2=0x7fd0fc157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.595+0000 7fd0fb7fe700 1 --2- 172.21.15.46:0/3202592501 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd0fc0b3fb0 0x7fd0fc157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.595+0000 7fd0fb7fe700 1 -- 172.21.15.46:0/3202592501 >> v1:172.21.15.46:6789/0 conn(0x7fd0fc157340 legacy=0x7fd0fc15bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.595+0000 7fd0fb7fe700 1 -- 172.21.15.46:0/3202592501 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc158520 con 0x7fd0fc0ae520 2024-01-26T23:30:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.595+0000 7fd0faffd700 1 -- 172.21.15.46:0/3202592501 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2913655382 0 0) 0x7fd0ec001110 con 0x7fd0fc157340 2024-01-26T23:30:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.595+0000 7fd100a6b700 1 --2- 172.21.15.46:0/3202592501 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd0fc0b3fb0 0x7fd0fc157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.595+0000 7fd0fb7fe700 1 --2- 172.21.15.46:0/3202592501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0fc0ae520 0x7fd0fc0b3a70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40098e0 tx=0x7fd0e40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=499f2f68bd477fd2 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.595+0000 7fd0faffd700 1 -- 172.21.15.46:0/3202592501 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4009e80 con 0x7fd0fc0ae520 2024-01-26T23:30:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.595+0000 7fd0faffd700 1 -- 172.21.15.46:0/3202592501 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e400b860 con 0x7fd0fc0ae520 2024-01-26T23:30:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.595+0000 7fd0faffd700 1 -- 172.21.15.46:0/3202592501 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e400bbe0 con 0x7fd0fc0ae520 2024-01-26T23:30:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.596+0000 7fd1024ce700 1 -- 172.21.15.46:0/3202592501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0fc0ae520 msgr2=0x7fd0fc0b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:47.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.596+0000 7fd1024ce700 1 --2- 172.21.15.46:0/3202592501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0fc0ae520 0x7fd0fc0b3a70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40098e0 tx=0x7fd0e40091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:47.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.596+0000 7fd1024ce700 1 -- 172.21.15.46:0/3202592501 shutdown_connections 2024-01-26T23:30:47.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.596+0000 7fd1024ce700 1 --2- 172.21.15.46:0/3202592501 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd0fc0b3fb0 0x7fd0fc157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:47.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.596+0000 7fd1024ce700 1 --2- 172.21.15.46:0/3202592501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0fc0ae520 0x7fd0fc0b3a70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:47.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.596+0000 7fd1024ce700 1 -- 172.21.15.46:0/3202592501 >> 172.21.15.46:0/3202592501 conn(0x7fd0fc0ce530 msgr2=0x7fd0fc0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:47.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.596+0000 7fd1024ce700 1 -- 172.21.15.46:0/3202592501 shutdown_connections 2024-01-26T23:30:47.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.596+0000 7fd1024ce700 1 -- 172.21.15.46:0/3202592501 wait complete. 2024-01-26T23:30:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.597+0000 7fd1024ce700 1 Processor -- start 2024-01-26T23:30:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.597+0000 7fd1024ce700 1 -- start start 2024-01-26T23:30:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.598+0000 7fd1024ce700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd0fc0ae520 0x7fd0fc145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.598+0000 7fd1024ce700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0fc0b3fb0 0x7fd0fc0c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.598+0000 7fd1024ce700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd0fc157340 0x7fd0fc0c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.598+0000 7fd1024ce700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0ca580 con 0x7fd0fc0b3fb0 2024-01-26T23:30:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.598+0000 7fd1024ce700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0c5800 con 0x7fd0fc157340 2024-01-26T23:30:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.598+0000 7fd1024ce700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0c9970 con 0x7fd0fc0ae520 2024-01-26T23:30:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.598+0000 7fd0fb7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0fc0b3fb0 0x7fd0fc0c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.598+0000 7fd0fb7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0fc0b3fb0 0x7fd0fc0c8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47864/0 (socket says 172.21.15.46:47864) 2024-01-26T23:30:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.598+0000 7fd0fb7fe700 1 -- 172.21.15.46:0/949857037 learned_addr learned my addr 172.21.15.46:0/949857037 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.598+0000 7fd100a6b700 1 --2- 172.21.15.46:0/949857037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd0fc157340 0x7fd0fc0c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.598+0000 7fd0fbfff700 1 --2- 172.21.15.46:0/949857037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd0fc0ae520 0x7fd0fc145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.599+0000 7fd0fb7fe700 1 -- 172.21.15.46:0/949857037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd0fc0ae520 msgr2=0x7fd0fc145710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.599+0000 7fd0fb7fe700 1 --2- 172.21.15.46:0/949857037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd0fc0ae520 0x7fd0fc145710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.599+0000 7fd0fb7fe700 1 -- 172.21.15.46:0/949857037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd0fc157340 msgr2=0x7fd0fc0c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.599+0000 7fd0fb7fe700 1 --2- 172.21.15.46:0/949857037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd0fc157340 0x7fd0fc0c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.599+0000 7fd0fb7fe700 1 -- 172.21.15.46:0/949857037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e4009420 con 0x7fd0fc0b3fb0 2024-01-26T23:30:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.599+0000 7fd0fb7fe700 1 --2- 172.21.15.46:0/949857037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0fc0b3fb0 0x7fd0fc0c8e00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd0e401bf90 tx=0x7fd0e4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.599+0000 7fd0f3fff700 1 -- 172.21.15.46:0/949857037 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e400b9d0 con 0x7fd0fc0b3fb0 2024-01-26T23:30:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.599+0000 7fd0f3fff700 1 -- 172.21.15.46:0/949857037 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e400bb70 con 0x7fd0fc0b3fb0 2024-01-26T23:30:47.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.599+0000 7fd1024ce700 1 -- 172.21.15.46:0/949857037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0fc0c9c50 con 0x7fd0fc0b3fb0 2024-01-26T23:30:47.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.599+0000 7fd0f3fff700 1 -- 172.21.15.46:0/949857037 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4011670 con 0x7fd0fc0b3fb0 2024-01-26T23:30:47.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.599+0000 7fd1024ce700 1 -- 172.21.15.46:0/949857037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0fc0ca210 con 0x7fd0fc0b3fb0 2024-01-26T23:30:47.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.600+0000 7fd1024ce700 1 -- 172.21.15.46:0/949857037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0fc0ca730 con 0x7fd0fc0b3fb0 2024-01-26T23:30:47.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.601+0000 7fd0f3fff700 1 -- 172.21.15.46:0/949857037 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd0e4011a20 con 0x7fd0fc0b3fb0 2024-01-26T23:30:47.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.601+0000 7fd0f3fff700 1 --2- 172.21.15.46:0/949857037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd0dc066f30 0x7fd0dc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:47.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.601+0000 7fd0f3fff700 1 -- 172.21.15.46:0/949857037 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7fd0e40859d0 con 0x7fd0fc0b3fb0 2024-01-26T23:30:47.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.601+0000 7fd0fbfff700 1 --2- 172.21.15.46:0/949857037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd0dc066f30 0x7fd0dc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:47.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.602+0000 7fd0fbfff700 1 --2- 172.21.15.46:0/949857037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd0dc066f30 0x7fd0dc0693e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec006fc0 tx=0x7fd0ec007b90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.606+0000 7fd0f3fff700 1 -- 172.21.15.46:0/949857037 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd0fc0ca730 con 0x7fd0fc0b3fb0 2024-01-26T23:30:47.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.895+0000 7fd1024ce700 1 -- 172.21.15.46:0/949857037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fd0fc0ca730 con 0x7fd0fc0b3fb0 2024-01-26T23:30:47.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.899+0000 7fd0f3fff700 1 -- 172.21.15.46:0/949857037 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7fd0fc0ca730 con 0x7fd0fc0b3fb0 2024-01-26T23:30:47.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.901+0000 7fd1024ce700 1 -- 172.21.15.46:0/949857037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd0dc066f30 msgr2=0x7fd0dc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:47.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.901+0000 7fd1024ce700 1 --2- 172.21.15.46:0/949857037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd0dc066f30 0x7fd0dc0693e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec006fc0 tx=0x7fd0ec007b90 comp rx=0 tx=0).stop 2024-01-26T23:30:47.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.901+0000 7fd1024ce700 1 -- 172.21.15.46:0/949857037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0fc0b3fb0 msgr2=0x7fd0fc0c8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:47.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.901+0000 7fd1024ce700 1 --2- 172.21.15.46:0/949857037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0fc0b3fb0 0x7fd0fc0c8e00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd0e401bf90 tx=0x7fd0e4017000 comp rx=0 tx=0).stop 2024-01-26T23:30:47.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.902+0000 7fd1024ce700 1 -- 172.21.15.46:0/949857037 shutdown_connections 2024-01-26T23:30:47.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.902+0000 7fd1024ce700 1 --2- 172.21.15.46:0/949857037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd0dc066f30 0x7fd0dc0693e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:47.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.902+0000 7fd1024ce700 1 --2- 172.21.15.46:0/949857037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd0fc157340 0x7fd0fc0c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:47.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.902+0000 7fd1024ce700 1 --2- 172.21.15.46:0/949857037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd0fc0ae520 0x7fd0fc145710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:47.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.902+0000 7fd1024ce700 1 --2- 172.21.15.46:0/949857037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0fc0b3fb0 0x7fd0fc0c8e00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:47.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.902+0000 7fd1024ce700 1 -- 172.21.15.46:0/949857037 >> 172.21.15.46:0/949857037 conn(0x7fd0fc0ce530 msgr2=0x7fd0fc15bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:47.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.902+0000 7fd1024ce700 1 -- 172.21.15.46:0/949857037 shutdown_connections 2024-01-26T23:30:47.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:47.902+0000 7fd1024ce700 1 -- 172.21.15.46:0/949857037 wait complete. 2024-01-26T23:30:47.917 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-01-26T23:30:48.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.048+0000 7f1202249700 1 Processor -- start 2024-01-26T23:30:48.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.048+0000 7f1202249700 1 -- start start 2024-01-26T23:30:48.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.049+0000 7f1202249700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f11fc1561d0 0x7f11fc0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.049+0000 7f1202249700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11fc0b0280 0x7f11fc1565a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.049+0000 7f1202249700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11fc0b0760 con 0x7f11fc06eb70 2024-01-26T23:30:48.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.049+0000 7f1202249700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11fc0b0a40 con 0x7f11fc1561d0 2024-01-26T23:30:48.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.049+0000 7f1202249700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc0b0d20 con 0x7f11fc0b0280 2024-01-26T23:30:48.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.049+0000 7f11fbfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f11fc06eb70 0x7f11fc06c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54300/0 (socket says 172.21.15.46:54300) 2024-01-26T23:30:48.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.049+0000 7f11fbfff700 1 -- 172.21.15.46:0/4091086652 learned_addr learned my addr 172.21.15.46:0/4091086652 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:48.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.050+0000 7f11faffd700 1 --2- 172.21.15.46:0/4091086652 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11fc0b0280 0x7f11fc1565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.050+0000 7f11fb7fe700 1 --2- 172.21.15.46:0/4091086652 >> v2:172.21.15.106:3300/0 conn(0x7f11fc1561d0 0x7f11fc0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.050+0000 7f11fa7fc700 1 -- 172.21.15.46:0/4091086652 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865189139 0 0) 0x7f11fc0b0760 con 0x7f11fc06eb70 2024-01-26T23:30:48.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.050+0000 7f11fa7fc700 1 -- 172.21.15.46:0/4091086652 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11e4003570 con 0x7f11fc06eb70 2024-01-26T23:30:48.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.050+0000 7f11fa7fc700 1 -- 172.21.15.46:0/4091086652 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3093608114 0 0) 0x7f11e4003570 con 0x7f11fc06eb70 2024-01-26T23:30:48.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.050+0000 7f11faffd700 1 -- 172.21.15.46:0/4091086652 >> v2:172.21.15.106:3300/0 conn(0x7f11fc1561d0 msgr2=0x7f11fc0ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:48.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.050+0000 7f11faffd700 1 --2- 172.21.15.46:0/4091086652 >> v2:172.21.15.106:3300/0 conn(0x7f11fc1561d0 0x7f11fc0ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.050+0000 7f11faffd700 1 -- 172.21.15.46:0/4091086652 >> v1:172.21.15.46:6789/0 conn(0x7f11fc06eb70 legacy=0x7f11fc06c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:48.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.050+0000 7f11faffd700 1 -- 172.21.15.46:0/4091086652 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc0b1000 con 0x7f11fc0b0280 2024-01-26T23:30:48.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.050+0000 7f11fb7fe700 1 --2- 172.21.15.46:0/4091086652 >> v2:172.21.15.106:3300/0 conn(0x7f11fc1561d0 0x7f11fc0ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:48.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.050+0000 7f11faffd700 1 --2- 172.21.15.46:0/4091086652 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11fc0b0280 0x7f11fc1565a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f11f00098e0 tx=0x7f11f00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e909b9bf56f88c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:48.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.050+0000 7f11fa7fc700 1 -- 172.21.15.46:0/4091086652 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f40017f0 con 0x7f11fc0b0280 2024-01-26T23:30:48.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.051+0000 7f11fa7fc700 1 -- 172.21.15.46:0/4091086652 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f0011420 con 0x7f11fc0b0280 2024-01-26T23:30:48.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.051+0000 7f11fa7fc700 1 -- 172.21.15.46:0/4091086652 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f00117c0 con 0x7f11fc0b0280 2024-01-26T23:30:48.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.051+0000 7f1202249700 1 -- 172.21.15.46:0/4091086652 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11fc0b0280 msgr2=0x7f11fc1565a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:48.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.051+0000 7f1202249700 1 --2- 172.21.15.46:0/4091086652 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11fc0b0280 0x7f11fc1565a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f11f00098e0 tx=0x7f11f00091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.051+0000 7f1202249700 1 -- 172.21.15.46:0/4091086652 shutdown_connections 2024-01-26T23:30:48.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.051+0000 7f1202249700 1 --2- 172.21.15.46:0/4091086652 >> v2:172.21.15.106:3300/0 conn(0x7f11fc1561d0 0x7f11fc0ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.051+0000 7f1202249700 1 --2- 172.21.15.46:0/4091086652 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11fc0b0280 0x7f11fc1565a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.051+0000 7f1202249700 1 -- 172.21.15.46:0/4091086652 >> 172.21.15.46:0/4091086652 conn(0x7f11fc06b630 msgr2=0x7f11fc0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:48.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.051+0000 7f1202249700 1 -- 172.21.15.46:0/4091086652 shutdown_connections 2024-01-26T23:30:48.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.051+0000 7f1202249700 1 -- 172.21.15.46:0/4091086652 wait complete. 2024-01-26T23:30:48.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.053+0000 7f1202249700 1 Processor -- start 2024-01-26T23:30:48.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.053+0000 7f1202249700 1 -- start start 2024-01-26T23:30:48.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.053+0000 7f1202249700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11fc06eb70 0x7f11fc158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.053+0000 7f1202249700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11fc0b0280 0x7f11fc14f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.053+0000 7f1202249700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11fc1561d0 0x7f11fc067cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.053+0000 7f1202249700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc159100 con 0x7f11fc0b0280 2024-01-26T23:30:48.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.053+0000 7f1202249700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc159340 con 0x7f11fc1561d0 2024-01-26T23:30:48.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.053+0000 7f1202249700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc159620 con 0x7f11fc06eb70 2024-01-26T23:30:48.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.054+0000 7f11faffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11fc0b0280 0x7f11fc14f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.054+0000 7f11faffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11fc0b0280 0x7f11fc14f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47878/0 (socket says 172.21.15.46:47878) 2024-01-26T23:30:48.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.054+0000 7f11faffd700 1 -- 172.21.15.46:0/3174367271 learned_addr learned my addr 172.21.15.46:0/3174367271 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:48.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.054+0000 7f11fbfff700 1 --2- 172.21.15.46:0/3174367271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11fc1561d0 0x7f11fc067cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.054+0000 7f11fb7fe700 1 --2- 172.21.15.46:0/3174367271 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11fc06eb70 0x7f11fc158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.054+0000 7f11faffd700 1 -- 172.21.15.46:0/3174367271 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11fc06eb70 msgr2=0x7f11fc158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:48.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.054+0000 7f11faffd700 1 --2- 172.21.15.46:0/3174367271 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11fc06eb70 0x7f11fc158bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.054+0000 7f11faffd700 1 -- 172.21.15.46:0/3174367271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11fc1561d0 msgr2=0x7f11fc067cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:48.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.054+0000 7f11faffd700 1 --2- 172.21.15.46:0/3174367271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11fc1561d0 0x7f11fc067cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.054+0000 7f11faffd700 1 -- 172.21.15.46:0/3174367271 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ec009420 con 0x7f11fc0b0280 2024-01-26T23:30:48.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.054+0000 7f11faffd700 1 --2- 172.21.15.46:0/3174367271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11fc0b0280 0x7f11fc14f9f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f11f00030c0 tx=0x7f11f0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:48.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.054+0000 7f11e37fe700 1 -- 172.21.15.46:0/3174367271 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f0000d10 con 0x7f11fc0b0280 2024-01-26T23:30:48.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.054+0000 7f11e37fe700 1 -- 172.21.15.46:0/3174367271 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f001cb30 con 0x7f11fc0b0280 2024-01-26T23:30:48.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.054+0000 7f1202249700 1 -- 172.21.15.46:0/3174367271 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f0009420 con 0x7f11fc0b0280 2024-01-26T23:30:48.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.054+0000 7f11e37fe700 1 -- 172.21.15.46:0/3174367271 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f001a450 con 0x7f11fc0b0280 2024-01-26T23:30:48.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.055+0000 7f1202249700 1 -- 172.21.15.46:0/3174367271 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11fc159be0 con 0x7f11fc0b0280 2024-01-26T23:30:48.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.055+0000 7f1202249700 1 -- 172.21.15.46:0/3174367271 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11fc0c7dc0 con 0x7f11fc0b0280 2024-01-26T23:30:48.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.056+0000 7f11e37fe700 1 -- 172.21.15.46:0/3174367271 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f11f001a840 con 0x7f11fc0b0280 2024-01-26T23:30:48.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.056+0000 7f11e37fe700 1 --2- 172.21.15.46:0/3174367271 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f11e8066f30 0x7f11e80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.056+0000 7f11e37fe700 1 -- 172.21.15.46:0/3174367271 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f11f0085b30 con 0x7f11fc0b0280 2024-01-26T23:30:48.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.057+0000 7f11fb7fe700 1 --2- 172.21.15.46:0/3174367271 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f11e8066f30 0x7f11e80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.057+0000 7f11fb7fe700 1 --2- 172.21.15.46:0/3174367271 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f11e8066f30 0x7f11e80693e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f11ec009ea0 tx=0x7f11ec0091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:48.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.061+0000 7f11e37fe700 1 -- 172.21.15.46:0/3174367271 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f11fc0c7dc0 con 0x7f11fc0b0280 2024-01-26T23:30:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.356+0000 7f1202249700 1 -- 172.21.15.46:0/3174367271 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f11fc0c7dc0 con 0x7f11fc0b0280 2024-01-26T23:30:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.360+0000 7f11e37fe700 1 -- 172.21.15.46:0/3174367271 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f11fc0c7dc0 con 0x7f11fc0b0280 2024-01-26T23:30:48.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.362+0000 7f1202249700 1 -- 172.21.15.46:0/3174367271 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f11e8066f30 msgr2=0x7f11e80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:48.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.362+0000 7f1202249700 1 --2- 172.21.15.46:0/3174367271 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f11e8066f30 0x7f11e80693e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f11ec009ea0 tx=0x7f11ec0091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.362+0000 7f1202249700 1 -- 172.21.15.46:0/3174367271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11fc0b0280 msgr2=0x7f11fc14f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:48.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.362+0000 7f1202249700 1 --2- 172.21.15.46:0/3174367271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11fc0b0280 0x7f11fc14f9f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f11f00030c0 tx=0x7f11f0016000 comp rx=0 tx=0).stop 2024-01-26T23:30:48.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.362+0000 7f1202249700 1 -- 172.21.15.46:0/3174367271 shutdown_connections 2024-01-26T23:30:48.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.362+0000 7f1202249700 1 --2- 172.21.15.46:0/3174367271 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f11e8066f30 0x7f11e80693e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.362+0000 7f1202249700 1 --2- 172.21.15.46:0/3174367271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11fc1561d0 0x7f11fc067cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.362+0000 7f1202249700 1 --2- 172.21.15.46:0/3174367271 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11fc06eb70 0x7f11fc158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.362+0000 7f1202249700 1 --2- 172.21.15.46:0/3174367271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11fc0b0280 0x7f11fc14f9f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.362+0000 7f1202249700 1 -- 172.21.15.46:0/3174367271 >> 172.21.15.46:0/3174367271 conn(0x7f11fc06b630 msgr2=0x7f11fc1519c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.362+0000 7f1202249700 1 -- 172.21.15.46:0/3174367271 shutdown_connections 2024-01-26T23:30:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.362+0000 7f1202249700 1 -- 172.21.15.46:0/3174367271 wait complete. 2024-01-26T23:30:48.377 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-01-26T23:30:48.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.508+0000 7f13e959f700 1 Processor -- start 2024-01-26T23:30:48.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.508+0000 7f13e959f700 1 -- start start 2024-01-26T23:30:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.509+0000 7f13e959f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13e40af850 0x7f13e4159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.509+0000 7f13e959f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e41509d0 con 0x7f13e40b0330 2024-01-26T23:30:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.509+0000 7f13e959f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e4157820 con 0x7f13e415a000 2024-01-26T23:30:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.509+0000 7f13e959f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13e415a450 con 0x7f13e40af850 2024-01-26T23:30:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.509+0000 7f13e2ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13e40af850 0x7f13e4159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.509+0000 7f13e37fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f13e40b0330 0x7f13e4157cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54308/0 (socket says 172.21.15.46:54308) 2024-01-26T23:30:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.509+0000 7f13e37fe700 1 -- 172.21.15.46:0/2680476927 learned_addr learned my addr 172.21.15.46:0/2680476927 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.510+0000 7f13e2ffd700 1 -- 172.21.15.46:0/2680476927 >> v1:172.21.15.106:6789/0 conn(0x7f13e415a000 legacy=0x7f13e40afc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.510+0000 7f13e2ffd700 1 -- 172.21.15.46:0/2680476927 >> v1:172.21.15.46:6789/0 conn(0x7f13e40b0330 legacy=0x7f13e4157cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.510+0000 7f13e1ffb700 1 -- 172.21.15.46:0/2680476927 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791879420 0 0) 0x7f13dc001490 con 0x7f13e40b0330 2024-01-26T23:30:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.510+0000 7f13e2ffd700 1 -- 172.21.15.46:0/2680476927 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e40b10d0 con 0x7f13e40af850 2024-01-26T23:30:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.510+0000 7f13e2ffd700 1 --2- 172.21.15.46:0/2680476927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e40af850 0x7f13e4159ac0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f13d40098e0 tx=0x7f13d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2416ceee9a7c84c2 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.510+0000 7f13e1ffb700 1 -- 172.21.15.46:0/2680476927 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d4009e80 con 0x7f13e40af850 2024-01-26T23:30:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.510+0000 7f13e1ffb700 1 -- 172.21.15.46:0/2680476927 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d400b860 con 0x7f13e40af850 2024-01-26T23:30:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.510+0000 7f13e1ffb700 1 -- 172.21.15.46:0/2680476927 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d400bbe0 con 0x7f13e40af850 2024-01-26T23:30:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.511+0000 7f13e959f700 1 -- 172.21.15.46:0/2680476927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e40af850 msgr2=0x7f13e4159ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.511+0000 7f13e959f700 1 --2- 172.21.15.46:0/2680476927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e40af850 0x7f13e4159ac0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f13d40098e0 tx=0x7f13d40091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.511+0000 7f13e959f700 1 -- 172.21.15.46:0/2680476927 shutdown_connections 2024-01-26T23:30:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.511+0000 7f13e959f700 1 --2- 172.21.15.46:0/2680476927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e40af850 0x7f13e4159ac0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.511+0000 7f13e959f700 1 -- 172.21.15.46:0/2680476927 >> 172.21.15.46:0/2680476927 conn(0x7f13e4068400 msgr2=0x7f13e40cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.511+0000 7f13e959f700 1 -- 172.21.15.46:0/2680476927 shutdown_connections 2024-01-26T23:30:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.511+0000 7f13e959f700 1 -- 172.21.15.46:0/2680476927 wait complete. 2024-01-26T23:30:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.512+0000 7f13e959f700 1 Processor -- start 2024-01-26T23:30:48.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.512+0000 7f13e959f700 1 -- start start 2024-01-26T23:30:48.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.513+0000 7f13e959f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e40af850 0x7f13e41687a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.513+0000 7f13e959f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13e40b0330 0x7f13e40c2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.513+0000 7f13e959f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13e415a000 0x7f13e40cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.513+0000 7f13e959f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e40c2f70 con 0x7f13e40af850 2024-01-26T23:30:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.513+0000 7f13e959f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e4168310 con 0x7f13e40b0330 2024-01-26T23:30:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.513+0000 7f13e959f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e4167cf0 con 0x7f13e415a000 2024-01-26T23:30:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.513+0000 7f13e2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e40af850 0x7f13e41687a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.513+0000 7f13e2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e40af850 0x7f13e41687a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47894/0 (socket says 172.21.15.46:47894) 2024-01-26T23:30:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.513+0000 7f13e2ffd700 1 -- 172.21.15.46:0/1086915219 learned_addr learned my addr 172.21.15.46:0/1086915219 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.513+0000 7f13e37fe700 1 --2- 172.21.15.46:0/1086915219 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13e415a000 0x7f13e40cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.513+0000 7f13e2ffd700 1 -- 172.21.15.46:0/1086915219 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13e415a000 msgr2=0x7f13e40cb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.513+0000 7f13e27fc700 1 --2- 172.21.15.46:0/1086915219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13e40b0330 0x7f13e40c2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.513+0000 7f13e2ffd700 1 --2- 172.21.15.46:0/1086915219 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13e415a000 0x7f13e40cb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.513+0000 7f13e2ffd700 1 -- 172.21.15.46:0/1086915219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13e40b0330 msgr2=0x7f13e40c2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.513+0000 7f13e2ffd700 1 --2- 172.21.15.46:0/1086915219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13e40b0330 0x7f13e40c2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.513+0000 7f13e2ffd700 1 -- 172.21.15.46:0/1086915219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d4009420 con 0x7f13e40af850 2024-01-26T23:30:48.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.514+0000 7f13e2ffd700 1 --2- 172.21.15.46:0/1086915219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e40af850 0x7f13e41687a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f13e4070360 tx=0x7f13d4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:48.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.514+0000 7f13daffd700 1 -- 172.21.15.46:0/1086915219 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d401f070 con 0x7f13e40af850 2024-01-26T23:30:48.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.514+0000 7f13daffd700 1 -- 172.21.15.46:0/1086915219 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d4011420 con 0x7f13e40af850 2024-01-26T23:30:48.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.514+0000 7f13e959f700 1 -- 172.21.15.46:0/1086915219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e40cbed0 con 0x7f13e40af850 2024-01-26T23:30:48.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.514+0000 7f13daffd700 1 -- 172.21.15.46:0/1086915219 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d400bbe0 con 0x7f13e40af850 2024-01-26T23:30:48.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.514+0000 7f13e959f700 1 -- 172.21.15.46:0/1086915219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e4167fa0 con 0x7f13e40af850 2024-01-26T23:30:48.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.515+0000 7f13e959f700 1 -- 172.21.15.46:0/1086915219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e404c710 con 0x7f13e40af850 2024-01-26T23:30:48.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.515+0000 7f13daffd700 1 -- 172.21.15.46:0/1086915219 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f13d4011a10 con 0x7f13e40af850 2024-01-26T23:30:48.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.516+0000 7f13daffd700 1 --2- 172.21.15.46:0/1086915219 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f13c4066f30 0x7f13c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.516+0000 7f13daffd700 1 -- 172.21.15.46:0/1086915219 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f13d4085900 con 0x7f13e40af850 2024-01-26T23:30:48.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.516+0000 7f13e27fc700 1 --2- 172.21.15.46:0/1086915219 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f13c4066f30 0x7f13c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.517+0000 7f13e27fc700 1 --2- 172.21.15.46:0/1086915219 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f13c4066f30 0x7f13c40693e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f13e414f960 tx=0x7f13cc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.520+0000 7f13daffd700 1 -- 172.21.15.46:0/1086915219 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f13d4052ce0 con 0x7f13e40af850 2024-01-26T23:30:48.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.809+0000 7f13e959f700 1 -- 172.21.15.46:0/1086915219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f13e4068800 con 0x7f13e40af850 2024-01-26T23:30:48.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.813+0000 7f13daffd700 1 -- 172.21.15.46:0/1086915219 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7f13d40150a0 con 0x7f13e40af850 2024-01-26T23:30:48.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.815+0000 7f13e959f700 1 -- 172.21.15.46:0/1086915219 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f13c4066f30 msgr2=0x7f13c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:48.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.815+0000 7f13e959f700 1 --2- 172.21.15.46:0/1086915219 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f13c4066f30 0x7f13c40693e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f13e414f960 tx=0x7f13cc009040 comp rx=0 tx=0).stop 2024-01-26T23:30:48.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.815+0000 7f13e959f700 1 -- 172.21.15.46:0/1086915219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e40af850 msgr2=0x7f13e41687a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:48.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.815+0000 7f13e959f700 1 --2- 172.21.15.46:0/1086915219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e40af850 0x7f13e41687a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f13e4070360 tx=0x7f13d4017000 comp rx=0 tx=0).stop 2024-01-26T23:30:48.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.815+0000 7f13e959f700 1 -- 172.21.15.46:0/1086915219 shutdown_connections 2024-01-26T23:30:48.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.815+0000 7f13e959f700 1 --2- 172.21.15.46:0/1086915219 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f13c4066f30 0x7f13c40693e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.815+0000 7f13e959f700 1 --2- 172.21.15.46:0/1086915219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e40af850 0x7f13e41687a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.815+0000 7f13e959f700 1 --2- 172.21.15.46:0/1086915219 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13e415a000 0x7f13e40cb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.815+0000 7f13e959f700 1 --2- 172.21.15.46:0/1086915219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13e40b0330 0x7f13e40c2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.815+0000 7f13e959f700 1 -- 172.21.15.46:0/1086915219 >> 172.21.15.46:0/1086915219 conn(0x7f13e4068400 msgr2=0x7f13e4150020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:48.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.816+0000 7f13e959f700 1 -- 172.21.15.46:0/1086915219 shutdown_connections 2024-01-26T23:30:48.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.816+0000 7f13e959f700 1 -- 172.21.15.46:0/1086915219 wait complete. 2024-01-26T23:30:48.831 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-01-26T23:30:48.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.961+0000 7f546bda5700 1 Processor -- start 2024-01-26T23:30:48.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.961+0000 7f546bda5700 1 -- start start 2024-01-26T23:30:48.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.962+0000 7f546bda5700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f546415cd60 0x7f54640c3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.962+0000 7f546bda5700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f54640c2240 0x7f54640cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.962+0000 7f546bda5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f546415d130 0x7f54640ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.962+0000 7f546bda5700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5464068220 con 0x7f54640c2240 2024-01-26T23:30:48.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.962+0000 7f546bda5700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5464067930 con 0x7f546415cd60 2024-01-26T23:30:48.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.962+0000 7f546bda5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54640c4210 con 0x7f546415d130 2024-01-26T23:30:48.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.962+0000 7f5469340700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f54640c2240 0x7f54640cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.963+0000 7f5469340700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f54640c2240 0x7f54640cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47904/0 (socket says 172.21.15.46:47904) 2024-01-26T23:30:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.963+0000 7f5469340700 1 -- 172.21.15.46:0/493300074 learned_addr learned my addr 172.21.15.46:0/493300074 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.963+0000 7f546a342700 1 --2- 172.21.15.46:0/493300074 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f546415d130 0x7f54640ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.963+0000 7f5469b41700 1 --2- 172.21.15.46:0/493300074 >> v2:172.21.15.106:3300/0 conn(0x7f546415cd60 0x7f54640c3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.963+0000 7f5469340700 1 -- 172.21.15.46:0/493300074 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f546415d130 msgr2=0x7f54640ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.963+0000 7f5469340700 1 --2- 172.21.15.46:0/493300074 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f546415d130 0x7f54640ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.963+0000 7f5469340700 1 -- 172.21.15.46:0/493300074 >> v2:172.21.15.106:3300/0 conn(0x7f546415cd60 msgr2=0x7f54640c3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.963+0000 7f5469340700 1 --2- 172.21.15.46:0/493300074 >> v2:172.21.15.106:3300/0 conn(0x7f546415cd60 0x7f54640c3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.963+0000 7f5469340700 1 -- 172.21.15.46:0/493300074 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5464067bd0 con 0x7f54640c2240 2024-01-26T23:30:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.963+0000 7f5469340700 1 --2- 172.21.15.46:0/493300074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54640c2240 0x7f54640cc5c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f546406d8e0 tx=0x7f5458009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa84e0d149a0369 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.963+0000 7f5468b3f700 1 -- 172.21.15.46:0/493300074 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545800b640 con 0x7f54640c2240 2024-01-26T23:30:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.963+0000 7f5468b3f700 1 -- 172.21.15.46:0/493300074 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545800b7a0 con 0x7f54640c2240 2024-01-26T23:30:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.963+0000 7f5468b3f700 1 -- 172.21.15.46:0/493300074 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545800bb60 con 0x7f54640c2240 2024-01-26T23:30:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.964+0000 7f546bda5700 1 -- 172.21.15.46:0/493300074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54640c2240 msgr2=0x7f54640cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.964+0000 7f546bda5700 1 --2- 172.21.15.46:0/493300074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54640c2240 0x7f54640cc5c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f546406d8e0 tx=0x7f5458009250 comp rx=0 tx=0).stop 2024-01-26T23:30:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.964+0000 7f546bda5700 1 -- 172.21.15.46:0/493300074 shutdown_connections 2024-01-26T23:30:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.964+0000 7f546bda5700 1 --2- 172.21.15.46:0/493300074 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f546415d130 0x7f54640ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.964+0000 7f546bda5700 1 --2- 172.21.15.46:0/493300074 >> v2:172.21.15.106:3300/0 conn(0x7f546415cd60 0x7f54640c3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.964+0000 7f546bda5700 1 --2- 172.21.15.46:0/493300074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54640c2240 0x7f54640cc5c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.964+0000 7f546bda5700 1 -- 172.21.15.46:0/493300074 >> 172.21.15.46:0/493300074 conn(0x7f5464069220 msgr2=0x7f54640cd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.964+0000 7f546bda5700 1 -- 172.21.15.46:0/493300074 shutdown_connections 2024-01-26T23:30:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.964+0000 7f546bda5700 1 -- 172.21.15.46:0/493300074 wait complete. 2024-01-26T23:30:48.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.965+0000 7f546bda5700 1 Processor -- start 2024-01-26T23:30:48.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.965+0000 7f546bda5700 1 -- start start 2024-01-26T23:30:48.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.966+0000 7f546bda5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54640c2240 0x7f546415f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.966+0000 7f546bda5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f546415cd60 0x7f54640aa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.966+0000 7f546bda5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546415d130 0x7f546415a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.966+0000 7f546bda5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54640973c0 con 0x7f546415d130 2024-01-26T23:30:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.966+0000 7f546bda5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54640976a0 con 0x7f54640c2240 2024-01-26T23:30:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.966+0000 7f546bda5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5464097980 con 0x7f546415cd60 2024-01-26T23:30:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.966+0000 7f546a342700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546415d130 0x7f546415a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.966+0000 7f546a342700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546415d130 0x7f546415a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47918/0 (socket says 172.21.15.46:47918) 2024-01-26T23:30:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.966+0000 7f546a342700 1 -- 172.21.15.46:0/3335376765 learned_addr learned my addr 172.21.15.46:0/3335376765 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.966+0000 7f5469b41700 1 --2- 172.21.15.46:0/3335376765 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54640c2240 0x7f546415f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.966+0000 7f5469340700 1 --2- 172.21.15.46:0/3335376765 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f546415cd60 0x7f54640aa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.967+0000 7f546a342700 1 -- 172.21.15.46:0/3335376765 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f546415cd60 msgr2=0x7f54640aa2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.967+0000 7f546a342700 1 --2- 172.21.15.46:0/3335376765 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f546415cd60 0x7f54640aa2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.967+0000 7f546a342700 1 -- 172.21.15.46:0/3335376765 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54640c2240 msgr2=0x7f546415f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.967+0000 7f546a342700 1 --2- 172.21.15.46:0/3335376765 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54640c2240 0x7f546415f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.967+0000 7f546a342700 1 -- 172.21.15.46:0/3335376765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54580094f0 con 0x7f546415d130 2024-01-26T23:30:48.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.967+0000 7f546a342700 1 --2- 172.21.15.46:0/3335376765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546415d130 0x7f546415a630 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f545c009df0 tx=0x7f545c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:48.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.967+0000 7f5455ffb700 1 -- 172.21.15.46:0/3335376765 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c00caa0 con 0x7f546415d130 2024-01-26T23:30:48.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.967+0000 7f5455ffb700 1 -- 172.21.15.46:0/3335376765 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545c00f040 con 0x7f546415d130 2024-01-26T23:30:48.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.967+0000 7f546bda5700 1 -- 172.21.15.46:0/3335376765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5464097c60 con 0x7f546415d130 2024-01-26T23:30:48.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.967+0000 7f5455ffb700 1 -- 172.21.15.46:0/3335376765 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c00ce50 con 0x7f546415d130 2024-01-26T23:30:48.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.967+0000 7f546bda5700 1 -- 172.21.15.46:0/3335376765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5464097050 con 0x7f546415d130 2024-01-26T23:30:48.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.968+0000 7f546bda5700 1 -- 172.21.15.46:0/3335376765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54640ac020 con 0x7f546415d130 2024-01-26T23:30:48.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.969+0000 7f5455ffb700 1 -- 172.21.15.46:0/3335376765 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f545c014690 con 0x7f546415d130 2024-01-26T23:30:48.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.969+0000 7f5455ffb700 1 --2- 172.21.15.46:0/3335376765 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5448066f30 0x7f54480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:48.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.969+0000 7f5455ffb700 1 -- 172.21.15.46:0/3335376765 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f545c084b30 con 0x7f546415d130 2024-01-26T23:30:48.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.969+0000 7f5469b41700 1 --2- 172.21.15.46:0/3335376765 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5448066f30 0x7f54480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:48.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.970+0000 7f5469b41700 1 --2- 172.21.15.46:0/3335376765 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5448066f30 0x7f54480693e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5460009da0 tx=0x7f5460009120 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:48.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:48.974+0000 7f5455ffb700 1 -- 172.21.15.46:0/3335376765 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f545c055a40 con 0x7f546415d130 2024-01-26T23:30:49.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.263+0000 7f546bda5700 1 -- 172.21.15.46:0/3335376765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f546404c710 con 0x7f546415d130 2024-01-26T23:30:49.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.267+0000 7f5455ffb700 1 -- 172.21.15.46:0/3335376765 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7f545c0555d0 con 0x7f546415d130 2024-01-26T23:30:49.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.269+0000 7f546bda5700 1 -- 172.21.15.46:0/3335376765 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5448066f30 msgr2=0x7f54480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:49.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.269+0000 7f546bda5700 1 --2- 172.21.15.46:0/3335376765 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5448066f30 0x7f54480693e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5460009da0 tx=0x7f5460009120 comp rx=0 tx=0).stop 2024-01-26T23:30:49.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.269+0000 7f546bda5700 1 -- 172.21.15.46:0/3335376765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546415d130 msgr2=0x7f546415a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:49.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.269+0000 7f546bda5700 1 --2- 172.21.15.46:0/3335376765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546415d130 0x7f546415a630 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f545c009df0 tx=0x7f545c00b040 comp rx=0 tx=0).stop 2024-01-26T23:30:49.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.269+0000 7f546bda5700 1 -- 172.21.15.46:0/3335376765 shutdown_connections 2024-01-26T23:30:49.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.269+0000 7f546bda5700 1 --2- 172.21.15.46:0/3335376765 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5448066f30 0x7f54480693e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.269+0000 7f546bda5700 1 --2- 172.21.15.46:0/3335376765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f546415d130 0x7f546415a630 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.269+0000 7f546bda5700 1 --2- 172.21.15.46:0/3335376765 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54640c2240 0x7f546415f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.269+0000 7f546bda5700 1 --2- 172.21.15.46:0/3335376765 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f546415cd60 0x7f54640aa2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.269+0000 7f546bda5700 1 -- 172.21.15.46:0/3335376765 >> 172.21.15.46:0/3335376765 conn(0x7f5464069220 msgr2=0x7f54640cb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:49.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.269+0000 7f546bda5700 1 -- 172.21.15.46:0/3335376765 shutdown_connections 2024-01-26T23:30:49.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.269+0000 7f546bda5700 1 -- 172.21.15.46:0/3335376765 wait complete. 2024-01-26T23:30:49.285 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:49.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.473+0000 7f026c1a7700 1 Processor -- start 2024-01-26T23:30:49.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.473+0000 7f026c1a7700 1 -- start start 2024-01-26T23:30:49.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.474+0000 7f026c1a7700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f02640ae520 0x7f02640b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:49.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.474+0000 7f026c1a7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02640b3fb0 0x7f0264157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:49.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.474+0000 7f026c1a7700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02640b4410 con 0x7f0264157340 2024-01-26T23:30:49.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.474+0000 7f026c1a7700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0264158ae0 con 0x7f02640ae520 2024-01-26T23:30:49.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.474+0000 7f026c1a7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026415b650 con 0x7f02640b3fb0 2024-01-26T23:30:49.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0269f43700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0264157340 0x7f026415bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54314/0 (socket says 172.21.15.46:54314) 2024-01-26T23:30:49.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0269f43700 1 -- 172.21.15.46:0/2631660447 learned_addr learned my addr 172.21.15.46:0/2631660447 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:49.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0269742700 1 --2- 172.21.15.46:0/2631660447 >> v2:172.21.15.106:3300/0 conn(0x7f02640ae520 0x7f02640b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:49.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f026a744700 1 --2- 172.21.15.46:0/2631660447 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02640b3fb0 0x7f0264157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:49.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0268f41700 1 -- 172.21.15.46:0/2631660447 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053082219 0 0) 0x7f0260001110 con 0x7f0264157340 2024-01-26T23:30:49.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0268f41700 1 -- 172.21.15.46:0/2631660447 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0250003570 con 0x7f0264157340 2024-01-26T23:30:49.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0268f41700 1 -- 172.21.15.46:0/2631660447 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 474775088 0 0) 0x7f0250003570 con 0x7f0264157340 2024-01-26T23:30:49.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0268f41700 1 -- 172.21.15.46:0/2631660447 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02640b3fb0 msgr2=0x7f0264157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:49.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0268f41700 1 --2- 172.21.15.46:0/2631660447 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02640b3fb0 0x7f0264157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0268f41700 1 -- 172.21.15.46:0/2631660447 >> v2:172.21.15.106:3300/0 conn(0x7f02640ae520 msgr2=0x7f02640b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:49.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0268f41700 1 --2- 172.21.15.46:0/2631660447 >> v2:172.21.15.106:3300/0 conn(0x7f02640ae520 0x7f02640b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0268f41700 1 -- 172.21.15.46:0/2631660447 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0264158520 con 0x7f0264157340 2024-01-26T23:30:49.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0268f41700 1 -- 172.21.15.46:0/2631660447 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f0260001600 con 0x7f0264157340 2024-01-26T23:30:49.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0269742700 1 --2- 172.21.15.46:0/2631660447 >> v2:172.21.15.106:3300/0 conn(0x7f02640ae520 0x7f02640b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:49.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f026a744700 1 --2- 172.21.15.46:0/2631660447 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02640b3fb0 0x7f0264157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:49.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0268f41700 1 -- 172.21.15.46:0/2631660447 >> v1:172.21.15.46:6789/0 conn(0x7f0264157340 legacy=0x7f026415bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:49.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0268f41700 1 --2- 172.21.15.46:0/2631660447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0250003ae0 0x7f0250003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:49.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0268f41700 1 --2- 172.21.15.46:0/2631660447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0250004810 0x7f0250004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:49.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0268f41700 1 --2- 172.21.15.46:0/2631660447 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02640b3fb0 0x7f02500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:49.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0268f41700 1 -- 172.21.15.46:0/2631660447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0264158ae0 con 0x7f0250003ae0 2024-01-26T23:30:49.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0268f41700 1 -- 172.21.15.46:0/2631660447 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026415b650 con 0x7f0250004810 2024-01-26T23:30:49.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.475+0000 7f0268f41700 1 -- 172.21.15.46:0/2631660447 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f025000bb90 con 0x7f02640b3fb0 2024-01-26T23:30:49.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.476+0000 7f0269f43700 1 --2- 172.21.15.46:0/2631660447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0250003ae0 0x7f0250003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:49.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.476+0000 7f0269742700 1 --2- 172.21.15.46:0/2631660447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0250004810 0x7f0250004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:49.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.476+0000 7f026a744700 1 --2- 172.21.15.46:0/2631660447 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02640b3fb0 0x7f02500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:49.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.476+0000 7f0269f43700 1 -- 172.21.15.46:0/2631660447 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02640b3fb0 msgr2=0x7f02500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:49.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.476+0000 7f0269f43700 1 --2- 172.21.15.46:0/2631660447 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02640b3fb0 0x7f02500051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.476+0000 7f0269f43700 1 -- 172.21.15.46:0/2631660447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0250004810 msgr2=0x7f0250004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:49.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.476+0000 7f0269f43700 1 --2- 172.21.15.46:0/2631660447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0250004810 0x7f0250004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.476+0000 7f0269f43700 1 -- 172.21.15.46:0/2631660447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025c009a20 con 0x7f0250003ae0 2024-01-26T23:30:49.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.476+0000 7f0269f43700 1 --2- 172.21.15.46:0/2631660447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0250003ae0 0x7f0250003ef0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f026000c440 tx=0x7f02600079d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e545db4d05912dc server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:49.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.476+0000 7f0268f41700 1 -- 172.21.15.46:0/2631660447 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026000cd70 con 0x7f0250003ae0 2024-01-26T23:30:49.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.476+0000 7f0268f41700 1 -- 172.21.15.46:0/2631660447 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0260005920 con 0x7f0250003ae0 2024-01-26T23:30:49.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.476+0000 7f0268f41700 1 -- 172.21.15.46:0/2631660447 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0260005ce0 con 0x7f0250003ae0 2024-01-26T23:30:49.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.477+0000 7f026c1a7700 1 -- 172.21.15.46:0/2631660447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0250003ae0 msgr2=0x7f0250003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:49.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.477+0000 7f026c1a7700 1 --2- 172.21.15.46:0/2631660447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0250003ae0 0x7f0250003ef0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f026000c440 tx=0x7f02600079d0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.477+0000 7f026a744700 1 -- 172.21.15.46:0/2631660447 reap_dead start 2024-01-26T23:30:49.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.477+0000 7f026c1a7700 1 -- 172.21.15.46:0/2631660447 shutdown_connections 2024-01-26T23:30:49.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.477+0000 7f026c1a7700 1 -- 172.21.15.46:0/2631660447 >> 172.21.15.46:0/2631660447 conn(0x7f02640ce530 msgr2=0x7f02640ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:49.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.477+0000 7f026c1a7700 1 -- 172.21.15.46:0/2631660447 shutdown_connections 2024-01-26T23:30:49.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.477+0000 7f026c1a7700 1 -- 172.21.15.46:0/2631660447 wait complete. 2024-01-26T23:30:49.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.479+0000 7f026c1a7700 1 Processor -- start 2024-01-26T23:30:49.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.479+0000 7f026c1a7700 1 -- start start 2024-01-26T23:30:49.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f026c1a7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0250003ae0 0x7f026416abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f026c1a7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02641651b0 0x7f02640cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f026c1a7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02640cd930 0x7f0264168f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f026c1a7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026416b0f0 con 0x7f0250003ae0 2024-01-26T23:30:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f026c1a7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02640cddb0 con 0x7f02640cd930 2024-01-26T23:30:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f026c1a7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0264168120 con 0x7f02641651b0 2024-01-26T23:30:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f0269f43700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0250003ae0 0x7f026416abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f0269f43700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0250003ae0 0x7f026416abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47934/0 (socket says 172.21.15.46:47934) 2024-01-26T23:30:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f0269f43700 1 -- 172.21.15.46:0/3022168659 learned_addr learned my addr 172.21.15.46:0/3022168659 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f026a744700 1 --2- 172.21.15.46:0/3022168659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02640cd930 0x7f0264168f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f0269742700 1 --2- 172.21.15.46:0/3022168659 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02641651b0 0x7f02640cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f0269f43700 1 -- 172.21.15.46:0/3022168659 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02641651b0 msgr2=0x7f02640cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f0269f43700 1 --2- 172.21.15.46:0/3022168659 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02641651b0 0x7f02640cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f0269f43700 1 -- 172.21.15.46:0/3022168659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02640cd930 msgr2=0x7f0264168f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:49.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f0269f43700 1 --2- 172.21.15.46:0/3022168659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02640cd930 0x7f0264168f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f0269f43700 1 -- 172.21.15.46:0/3022168659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026416a720 con 0x7f0250003ae0 2024-01-26T23:30:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f0269742700 1 --2- 172.21.15.46:0/3022168659 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02641651b0 0x7f02640cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.480+0000 7f0269f43700 1 --2- 172.21.15.46:0/3022168659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0250003ae0 0x7f026416abb0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f026000ca30 tx=0x7f0260015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.481+0000 7f0259ffb700 1 -- 172.21.15.46:0/3022168659 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0260005ca0 con 0x7f0250003ae0 2024-01-26T23:30:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.481+0000 7f0259ffb700 1 -- 172.21.15.46:0/3022168659 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0260005e00 con 0x7f0250003ae0 2024-01-26T23:30:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.481+0000 7f0259ffb700 1 -- 172.21.15.46:0/3022168659 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0260007510 con 0x7f0250003ae0 2024-01-26T23:30:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.481+0000 7f026c1a7700 1 -- 172.21.15.46:0/3022168659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0264169b10 con 0x7f0250003ae0 2024-01-26T23:30:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.481+0000 7f026c1a7700 1 -- 172.21.15.46:0/3022168659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f026416a0d0 con 0x7f0250003ae0 2024-01-26T23:30:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.482+0000 7f026c1a7700 1 -- 172.21.15.46:0/3022168659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f026404c710 con 0x7f0250003ae0 2024-01-26T23:30:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.482+0000 7f0259ffb700 1 -- 172.21.15.46:0/3022168659 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0260019030 con 0x7f0250003ae0 2024-01-26T23:30:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.483+0000 7f0259ffb700 1 --2- 172.21.15.46:0/3022168659 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0248066f30 0x7f02480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.483+0000 7f0259ffb700 1 -- 172.21.15.46:0/3022168659 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f02600851f0 con 0x7f0250003ae0 2024-01-26T23:30:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.483+0000 7f0269742700 1 --2- 172.21.15.46:0/3022168659 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0248066f30 0x7f02480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.484+0000 7f0269742700 1 --2- 172.21.15.46:0/3022168659 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0248066f30 0x7f02480693e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0254006dc0 tx=0x7f0254008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:49.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.487+0000 7f0259ffb700 1 -- 172.21.15.46:0/3022168659 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0260052670 con 0x7f0250003ae0 2024-01-26T23:30:49.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.694+0000 7f026c1a7700 1 -- 172.21.15.46:0/3022168659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02640b4410 con 0x7f0250003ae0 2024-01-26T23:30:49.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.695+0000 7f0259ffb700 1 -- 172.21.15.46:0/3022168659 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+12339 (secure 0 0 0) 0x7f02640b4410 con 0x7f0250003ae0 2024-01-26T23:30:49.696 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:49.697 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":34,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:30:46.087230+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:49.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.697+0000 7f026c1a7700 1 -- 172.21.15.46:0/3022168659 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0248066f30 msgr2=0x7f02480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:49.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.697+0000 7f026c1a7700 1 --2- 172.21.15.46:0/3022168659 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0248066f30 0x7f02480693e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0254006dc0 tx=0x7f0254008040 comp rx=0 tx=0).stop 2024-01-26T23:30:49.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.697+0000 7f026c1a7700 1 -- 172.21.15.46:0/3022168659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0250003ae0 msgr2=0x7f026416abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:49.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.697+0000 7f026c1a7700 1 --2- 172.21.15.46:0/3022168659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0250003ae0 0x7f026416abb0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f026000ca30 tx=0x7f0260015000 comp rx=0 tx=0).stop 2024-01-26T23:30:49.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.697+0000 7f026c1a7700 1 -- 172.21.15.46:0/3022168659 shutdown_connections 2024-01-26T23:30:49.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.697+0000 7f026c1a7700 1 --2- 172.21.15.46:0/3022168659 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0248066f30 0x7f02480693e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.697+0000 7f026c1a7700 1 --2- 172.21.15.46:0/3022168659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0250003ae0 0x7f026416abb0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.697+0000 7f026c1a7700 1 --2- 172.21.15.46:0/3022168659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02640cd930 0x7f0264168f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.697+0000 7f026c1a7700 1 --2- 172.21.15.46:0/3022168659 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02641651b0 0x7f02640cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.697+0000 7f026c1a7700 1 -- 172.21.15.46:0/3022168659 >> 172.21.15.46:0/3022168659 conn(0x7f02640ce530 msgr2=0x7f0264158ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:49.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.697+0000 7f026c1a7700 1 -- 172.21.15.46:0/3022168659 shutdown_connections 2024-01-26T23:30:49.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.697+0000 7f026c1a7700 1 -- 172.21.15.46:0/3022168659 wait complete. 2024-01-26T23:30:49.713 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-26T23:30:49.713 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:30:49.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.847+0000 7efe1a0bc700 1 Processor -- start 2024-01-26T23:30:49.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.847+0000 7efe1a0bc700 1 -- start start 2024-01-26T23:30:49.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.848+0000 7efe1a0bc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efe14068ed0 0x7efe1406a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:49.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.848+0000 7efe1a0bc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe1406add0 0x7efe1406b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:49.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.848+0000 7efe1a0bc700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe140a9a80 con 0x7efe14068b00 2024-01-26T23:30:49.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.848+0000 7efe1a0bc700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe14069320 con 0x7efe14068ed0 2024-01-26T23:30:49.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.848+0000 7efe1a0bc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1406b790 con 0x7efe1406add0 2024-01-26T23:30:49.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.848+0000 7efe12ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efe14068ed0 0x7efe1406a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:49.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.848+0000 7efe137fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efe14068b00 0x7efe14159ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54324/0 (socket says 172.21.15.46:54324) 2024-01-26T23:30:49.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.848+0000 7efe137fe700 1 -- 172.21.15.46:0/1133997553 learned_addr learned my addr 172.21.15.46:0/1133997553 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:49.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.848+0000 7efe13fff700 1 --2- 172.21.15.46:0/1133997553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe1406add0 0x7efe1406b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:49.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.848+0000 7efe12ffd700 1 -- 172.21.15.46:0/1133997553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe1406add0 msgr2=0x7efe1406b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:49.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.848+0000 7efe12ffd700 1 --2- 172.21.15.46:0/1133997553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe1406add0 0x7efe1406b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.848+0000 7efe12ffd700 1 -- 172.21.15.46:0/1133997553 >> v1:172.21.15.46:6789/0 conn(0x7efe14068b00 legacy=0x7efe14159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:49.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.848+0000 7efe12ffd700 1 -- 172.21.15.46:0/1133997553 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1415a680 con 0x7efe14068ed0 2024-01-26T23:30:49.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.848+0000 7efe127fc700 1 -- 172.21.15.46:0/1133997553 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341187739 0 0) 0x7efe140a9a80 con 0x7efe14068b00 2024-01-26T23:30:49.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.848+0000 7efe13fff700 1 --2- 172.21.15.46:0/1133997553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe1406add0 0x7efe1406b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:49.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.848+0000 7efe12ffd700 1 --2- 172.21.15.46:0/1133997553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe14068ed0 0x7efe1406a660 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efe080098e0 tx=0x7efe080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6de16390991b7239 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:49.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.849+0000 7efe127fc700 1 -- 172.21.15.46:0/1133997553 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe08009e80 con 0x7efe14068ed0 2024-01-26T23:30:49.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.849+0000 7efe127fc700 1 -- 172.21.15.46:0/1133997553 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0800b860 con 0x7efe14068ed0 2024-01-26T23:30:49.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.849+0000 7efe127fc700 1 -- 172.21.15.46:0/1133997553 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0800bbe0 con 0x7efe14068ed0 2024-01-26T23:30:49.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.849+0000 7efe1a0bc700 1 -- 172.21.15.46:0/1133997553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe14068ed0 msgr2=0x7efe1406a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:49.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.849+0000 7efe1a0bc700 1 --2- 172.21.15.46:0/1133997553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe14068ed0 0x7efe1406a660 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efe080098e0 tx=0x7efe080091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.849+0000 7efe1a0bc700 1 -- 172.21.15.46:0/1133997553 shutdown_connections 2024-01-26T23:30:49.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.849+0000 7efe1a0bc700 1 --2- 172.21.15.46:0/1133997553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe1406add0 0x7efe1406b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.849+0000 7efe1a0bc700 1 --2- 172.21.15.46:0/1133997553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe14068ed0 0x7efe1406a660 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.849+0000 7efe1a0bc700 1 -- 172.21.15.46:0/1133997553 >> 172.21.15.46:0/1133997553 conn(0x7efe140ca100 msgr2=0x7efe1406e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:49.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.849+0000 7efe1a0bc700 1 -- 172.21.15.46:0/1133997553 shutdown_connections 2024-01-26T23:30:49.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.849+0000 7efe1a0bc700 1 -- 172.21.15.46:0/1133997553 wait complete. 2024-01-26T23:30:49.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.851+0000 7efe1a0bc700 1 Processor -- start 2024-01-26T23:30:49.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.851+0000 7efe1a0bc700 1 -- start start 2024-01-26T23:30:49.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.851+0000 7efe1a0bc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe14068b00 0x7efe14154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:49.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.851+0000 7efe1a0bc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe14068ed0 0x7efe140b0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:49.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.851+0000 7efe1a0bc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe1406add0 0x7efe14154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:49.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.851+0000 7efe1a0bc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe140ae740 con 0x7efe14068b00 2024-01-26T23:30:49.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.851+0000 7efe1a0bc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe140afa10 con 0x7efe1406add0 2024-01-26T23:30:49.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.851+0000 7efe1a0bc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe140afc80 con 0x7efe14068ed0 2024-01-26T23:30:49.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.852+0000 7efe137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe14068b00 0x7efe14154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:49.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.852+0000 7efe137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe14068b00 0x7efe14154140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47950/0 (socket says 172.21.15.46:47950) 2024-01-26T23:30:49.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.852+0000 7efe137fe700 1 -- 172.21.15.46:0/196034605 learned_addr learned my addr 172.21.15.46:0/196034605 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:49.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.852+0000 7efe12ffd700 1 --2- 172.21.15.46:0/196034605 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe14068ed0 0x7efe140b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:49.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.852+0000 7efe13fff700 1 --2- 172.21.15.46:0/196034605 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe1406add0 0x7efe14154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:49.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.852+0000 7efe137fe700 1 -- 172.21.15.46:0/196034605 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe14068ed0 msgr2=0x7efe140b0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:49.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.852+0000 7efe137fe700 1 --2- 172.21.15.46:0/196034605 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe14068ed0 0x7efe140b0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.852+0000 7efe137fe700 1 -- 172.21.15.46:0/196034605 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe1406add0 msgr2=0x7efe14154a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:49.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.852+0000 7efe137fe700 1 --2- 172.21.15.46:0/196034605 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe1406add0 0x7efe14154a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:49.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.852+0000 7efe137fe700 1 -- 172.21.15.46:0/196034605 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe08009420 con 0x7efe14068b00 2024-01-26T23:30:49.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.852+0000 7efe12ffd700 1 --2- 172.21.15.46:0/196034605 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe14068ed0 0x7efe140b0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:49.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.852+0000 7efe137fe700 1 --2- 172.21.15.46:0/196034605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe14068b00 0x7efe14154140 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efe0400c5e0 tx=0x7efe04007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:49.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.852+0000 7efdfb7fe700 1 -- 172.21.15.46:0/196034605 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe040072e0 con 0x7efe14068b00 2024-01-26T23:30:49.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.852+0000 7efdfb7fe700 1 -- 172.21.15.46:0/196034605 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe04007460 con 0x7efe14068b00 2024-01-26T23:30:49.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.852+0000 7efe1a0bc700 1 -- 172.21.15.46:0/196034605 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe140aff60 con 0x7efe14068b00 2024-01-26T23:30:49.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.852+0000 7efdfb7fe700 1 -- 172.21.15.46:0/196034605 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe040057f0 con 0x7efe14068b00 2024-01-26T23:30:49.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.852+0000 7efe1a0bc700 1 -- 172.21.15.46:0/196034605 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe140b1e10 con 0x7efe14068b00 2024-01-26T23:30:49.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.853+0000 7efe1a0bc700 1 -- 172.21.15.46:0/196034605 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe140b0480 con 0x7efe14068b00 2024-01-26T23:30:49.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.854+0000 7efdfb7fe700 1 -- 172.21.15.46:0/196034605 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7efe04005bc0 con 0x7efe14068b00 2024-01-26T23:30:49.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.854+0000 7efdfb7fe700 1 --2- 172.21.15.46:0/196034605 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efe00066f30 0x7efe000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:49.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.854+0000 7efdfb7fe700 1 -- 172.21.15.46:0/196034605 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7efe040837f0 con 0x7efe14068b00 2024-01-26T23:30:49.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.854+0000 7efe12ffd700 1 --2- 172.21.15.46:0/196034605 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efe00066f30 0x7efe000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:49.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.855+0000 7efe12ffd700 1 --2- 172.21.15.46:0/196034605 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efe00066f30 0x7efe000693e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efe080091e0 tx=0x7efe08017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:49.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:49.859+0000 7efdfb7fe700 1 -- 172.21.15.46:0/196034605 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efe04050c70 con 0x7efe14068b00 2024-01-26T23:30:50.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.060+0000 7efe1a0bc700 1 -- 172.21.15.46:0/196034605 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efe1404c710 con 0x7efe14068b00 2024-01-26T23:30:50.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.060+0000 7efdfb7fe700 1 -- 172.21.15.46:0/196034605 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7efe04017070 con 0x7efe14068b00 2024-01-26T23:30:50.062 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:30:50.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.063+0000 7efe1a0bc700 1 -- 172.21.15.46:0/196034605 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efe00066f30 msgr2=0x7efe000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:50.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.063+0000 7efe1a0bc700 1 --2- 172.21.15.46:0/196034605 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efe00066f30 0x7efe000693e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efe080091e0 tx=0x7efe08017000 comp rx=0 tx=0).stop 2024-01-26T23:30:50.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.063+0000 7efe1a0bc700 1 -- 172.21.15.46:0/196034605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe14068b00 msgr2=0x7efe14154140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:50.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.063+0000 7efe1a0bc700 1 --2- 172.21.15.46:0/196034605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe14068b00 0x7efe14154140 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efe0400c5e0 tx=0x7efe04007e80 comp rx=0 tx=0).stop 2024-01-26T23:30:50.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.063+0000 7efe1a0bc700 1 -- 172.21.15.46:0/196034605 shutdown_connections 2024-01-26T23:30:50.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.063+0000 7efe1a0bc700 1 --2- 172.21.15.46:0/196034605 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efe00066f30 0x7efe000693e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:50.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.063+0000 7efe1a0bc700 1 --2- 172.21.15.46:0/196034605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe14068b00 0x7efe14154140 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:50.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.063+0000 7efe1a0bc700 1 --2- 172.21.15.46:0/196034605 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe1406add0 0x7efe14154a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:50.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.063+0000 7efe1a0bc700 1 --2- 172.21.15.46:0/196034605 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe14068ed0 0x7efe140b0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:50.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.063+0000 7efe1a0bc700 1 -- 172.21.15.46:0/196034605 >> 172.21.15.46:0/196034605 conn(0x7efe140ca100 msgr2=0x7efe14070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:50.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.063+0000 7efe1a0bc700 1 -- 172.21.15.46:0/196034605 shutdown_connections 2024-01-26T23:30:50.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.063+0000 7efe1a0bc700 1 -- 172.21.15.46:0/196034605 wait complete. 2024-01-26T23:30:50.078 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-26T23:30:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.208+0000 7f5cc5aa5700 1 Processor -- start 2024-01-26T23:30:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.209+0000 7f5cc5aa5700 1 -- start start 2024-01-26T23:30:50.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.209+0000 7f5cc5aa5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cc0159590 0x7f5cc015d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:50.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.209+0000 7f5cc5aa5700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5cc006c3c0 0x7f5cc0069370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:50.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.209+0000 7f5cc5aa5700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cc015dd90 con 0x7f5cc015d8f0 2024-01-26T23:30:50.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.209+0000 7f5cc5aa5700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc015e350 con 0x7f5cc006c3c0 2024-01-26T23:30:50.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.209+0000 7f5cc5aa5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc00c97c0 con 0x7f5cc0159590 2024-01-26T23:30:50.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.210+0000 7f5cbffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5cc006c3c0 0x7f5cc0069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:50.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.210+0000 7f5cb6ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5cc015d8f0 0x7f5cc006bcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54338/0 (socket says 172.21.15.46:54338) 2024-01-26T23:30:50.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.210+0000 7f5cb6ffd700 1 -- 172.21.15.46:0/3762572257 learned_addr learned my addr 172.21.15.46:0/3762572257 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:50.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.210+0000 7f5cbf7fe700 1 --2- 172.21.15.46:0/3762572257 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cc0159590 0x7f5cc015d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:50.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.210+0000 7f5cbffff700 1 -- 172.21.15.46:0/3762572257 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cc0159590 msgr2=0x7f5cc015d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:50.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.210+0000 7f5cbffff700 1 --2- 172.21.15.46:0/3762572257 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cc0159590 0x7f5cc015d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:50.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.210+0000 7f5cbffff700 1 -- 172.21.15.46:0/3762572257 >> v1:172.21.15.46:6789/0 conn(0x7f5cc015d8f0 legacy=0x7f5cc006bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:50.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.210+0000 7f5cbeffd700 1 -- 172.21.15.46:0/3762572257 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047720957 0 0) 0x7f5cc015dd90 con 0x7f5cc015d8f0 2024-01-26T23:30:50.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.210+0000 7f5cbffff700 1 -- 172.21.15.46:0/3762572257 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc015e070 con 0x7f5cc006c3c0 2024-01-26T23:30:50.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.210+0000 7f5cbf7fe700 1 --2- 172.21.15.46:0/3762572257 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cc0159590 0x7f5cc015d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:50.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.210+0000 7f5cbffff700 1 --2- 172.21.15.46:0/3762572257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc006c3c0 0x7f5cc0069370 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8009e90 tx=0x7f5cb8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acc0c9a996eec5e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:50.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.210+0000 7f5cbeffd700 1 -- 172.21.15.46:0/3762572257 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb800e070 con 0x7f5cc006c3c0 2024-01-26T23:30:50.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.210+0000 7f5cbeffd700 1 -- 172.21.15.46:0/3762572257 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb800bbf0 con 0x7f5cc006c3c0 2024-01-26T23:30:50.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.210+0000 7f5cbeffd700 1 -- 172.21.15.46:0/3762572257 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8012430 con 0x7f5cc006c3c0 2024-01-26T23:30:50.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.211+0000 7f5cc5aa5700 1 -- 172.21.15.46:0/3762572257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc006c3c0 msgr2=0x7f5cc0069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:50.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.211+0000 7f5cc5aa5700 1 --2- 172.21.15.46:0/3762572257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc006c3c0 0x7f5cc0069370 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8009e90 tx=0x7f5cb8009780 comp rx=0 tx=0).stop 2024-01-26T23:30:50.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.211+0000 7f5cc5aa5700 1 -- 172.21.15.46:0/3762572257 shutdown_connections 2024-01-26T23:30:50.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.211+0000 7f5cc5aa5700 1 --2- 172.21.15.46:0/3762572257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc006c3c0 0x7f5cc0069370 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:50.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.211+0000 7f5cc5aa5700 1 --2- 172.21.15.46:0/3762572257 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cc0159590 0x7f5cc015d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:50.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.211+0000 7f5cc5aa5700 1 -- 172.21.15.46:0/3762572257 >> 172.21.15.46:0/3762572257 conn(0x7f5cc00ce530 msgr2=0x7f5cc00ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:50.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.211+0000 7f5cc5aa5700 1 -- 172.21.15.46:0/3762572257 shutdown_connections 2024-01-26T23:30:50.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.211+0000 7f5cc5aa5700 1 -- 172.21.15.46:0/3762572257 wait complete. 2024-01-26T23:30:50.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.213+0000 7f5cc5aa5700 1 Processor -- start 2024-01-26T23:30:50.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.213+0000 7f5cc5aa5700 1 -- start start 2024-01-26T23:30:50.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.214+0000 7f5cc5aa5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc006c3c0 0x7f5cc00c8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:50.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.214+0000 7f5cc5aa5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cc0159590 0x7f5cc0155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:50.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.214+0000 7f5cc5aa5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cc015d8f0 0x7f5cc00c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:50.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.214+0000 7f5cc5aa5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc015b5c0 con 0x7f5cc006c3c0 2024-01-26T23:30:50.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.214+0000 7f5cc5aa5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc015a9b0 con 0x7f5cc0159590 2024-01-26T23:30:50.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.214+0000 7f5cc5aa5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc015ac90 con 0x7f5cc015d8f0 2024-01-26T23:30:50.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.214+0000 7f5cbf7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc006c3c0 0x7f5cc00c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:50.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.214+0000 7f5cbf7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc006c3c0 0x7f5cc00c8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47978/0 (socket says 172.21.15.46:47978) 2024-01-26T23:30:50.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.214+0000 7f5cbf7fe700 1 -- 172.21.15.46:0/522571061 learned_addr learned my addr 172.21.15.46:0/522571061 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:50.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.214+0000 7f5cb6ffd700 1 --2- 172.21.15.46:0/522571061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cc0159590 0x7f5cc0155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:50.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.214+0000 7f5cbffff700 1 --2- 172.21.15.46:0/522571061 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cc015d8f0 0x7f5cc00c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:50.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.214+0000 7f5cbf7fe700 1 -- 172.21.15.46:0/522571061 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cc015d8f0 msgr2=0x7f5cc00c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:50.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.214+0000 7f5cbf7fe700 1 --2- 172.21.15.46:0/522571061 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cc015d8f0 0x7f5cc00c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:50.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.214+0000 7f5cbf7fe700 1 -- 172.21.15.46:0/522571061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cc0159590 msgr2=0x7f5cc0155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:50.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.214+0000 7f5cbf7fe700 1 --2- 172.21.15.46:0/522571061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cc0159590 0x7f5cc0155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:50.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.214+0000 7f5cbf7fe700 1 -- 172.21.15.46:0/522571061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb8009a20 con 0x7f5cc006c3c0 2024-01-26T23:30:50.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.214+0000 7f5cbffff700 1 --2- 172.21.15.46:0/522571061 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cc015d8f0 0x7f5cc00c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:30:50.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.215+0000 7f5cbf7fe700 1 --2- 172.21.15.46:0/522571061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc006c3c0 0x7f5cc00c8450 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5cb000e8d0 tx=0x7f5cb0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:50.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.215+0000 7f5cb7fff700 1 -- 172.21.15.46:0/522571061 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb000ee70 con 0x7f5cc006c3c0 2024-01-26T23:30:50.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.215+0000 7f5cb7fff700 1 -- 172.21.15.46:0/522571061 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb000c5e0 con 0x7f5cc006c3c0 2024-01-26T23:30:50.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.215+0000 7f5cb7fff700 1 -- 172.21.15.46:0/522571061 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb000c980 con 0x7f5cc006c3c0 2024-01-26T23:30:50.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.215+0000 7f5cc5aa5700 1 -- 172.21.15.46:0/522571061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc015af70 con 0x7f5cc006c3c0 2024-01-26T23:30:50.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.215+0000 7f5cc5aa5700 1 -- 172.21.15.46:0/522571061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc0156080 con 0x7f5cc006c3c0 2024-01-26T23:30:50.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.216+0000 7f5cc5aa5700 1 -- 172.21.15.46:0/522571061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc004c710 con 0x7f5cc006c3c0 2024-01-26T23:30:50.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.216+0000 7f5cb7fff700 1 -- 172.21.15.46:0/522571061 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5cb0013410 con 0x7f5cc006c3c0 2024-01-26T23:30:50.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.217+0000 7f5cb7fff700 1 --2- 172.21.15.46:0/522571061 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5ca0066f30 0x7f5ca00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:50.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.217+0000 7f5cb7fff700 1 -- 172.21.15.46:0/522571061 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f5cb0084100 con 0x7f5cc006c3c0 2024-01-26T23:30:50.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.217+0000 7f5cb6ffd700 1 --2- 172.21.15.46:0/522571061 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5ca0066f30 0x7f5ca00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:50.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.218+0000 7f5cb6ffd700 1 --2- 172.21.15.46:0/522571061 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5ca0066f30 0x7f5ca00693e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5cc015dcc0 tx=0x7f5ca8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:50.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.222+0000 7f5cb7fff700 1 -- 172.21.15.46:0/522571061 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5cb0055010 con 0x7f5cc006c3c0 2024-01-26T23:30:50.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.428+0000 7f5cc5aa5700 1 -- 172.21.15.46:0/522571061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5cc00c1ef0 con 0x7f5cc006c3c0 2024-01-26T23:30:50.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.429+0000 7f5cb7fff700 1 -- 172.21.15.46:0/522571061 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v34) v1 ==== 84+0+65 (secure 0 0 0) 0x7f5cb0054ba0 con 0x7f5cc006c3c0 2024-01-26T23:30:50.430 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:50.431 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-26T23:30:50.431 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:30:50.431 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-01-26T23:30:50.431 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-01-26T23:30:50.431 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:30:50.431 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-26T23:30:50.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.431+0000 7f5cc5aa5700 1 -- 172.21.15.46:0/522571061 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5ca0066f30 msgr2=0x7f5ca00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:50.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.431+0000 7f5cc5aa5700 1 --2- 172.21.15.46:0/522571061 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5ca0066f30 0x7f5ca00693e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5cc015dcc0 tx=0x7f5ca8009040 comp rx=0 tx=0).stop 2024-01-26T23:30:50.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.431+0000 7f5cc5aa5700 1 -- 172.21.15.46:0/522571061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc006c3c0 msgr2=0x7f5cc00c8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:50.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.431+0000 7f5cc5aa5700 1 --2- 172.21.15.46:0/522571061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc006c3c0 0x7f5cc00c8450 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5cb000e8d0 tx=0x7f5cb0008040 comp rx=0 tx=0).stop 2024-01-26T23:30:50.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.431+0000 7f5cc5aa5700 1 -- 172.21.15.46:0/522571061 shutdown_connections 2024-01-26T23:30:50.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.431+0000 7f5cc5aa5700 1 --2- 172.21.15.46:0/522571061 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5ca0066f30 0x7f5ca00693e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:50.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.431+0000 7f5cc5aa5700 1 --2- 172.21.15.46:0/522571061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc006c3c0 0x7f5cc00c8450 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:50.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.431+0000 7f5cc5aa5700 1 --2- 172.21.15.46:0/522571061 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cc015d8f0 0x7f5cc00c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:50.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.431+0000 7f5cc5aa5700 1 --2- 172.21.15.46:0/522571061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cc0159590 0x7f5cc0155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:50.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.431+0000 7f5cc5aa5700 1 -- 172.21.15.46:0/522571061 >> 172.21.15.46:0/522571061 conn(0x7f5cc00ce530 msgr2=0x7f5cc0069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:50.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.431+0000 7f5cc5aa5700 1 -- 172.21.15.46:0/522571061 shutdown_connections 2024-01-26T23:30:50.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.431+0000 7f5cc5aa5700 1 -- 172.21.15.46:0/522571061 wait complete. 2024-01-26T23:30:50.446 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-26T23:30:50.446 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-26T23:30:50.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.580+0000 7f926d95c700 1 Processor -- start 2024-01-26T23:30:50.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.580+0000 7f926d95c700 1 -- start start 2024-01-26T23:30:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.581+0000 7f926d95c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f92680cc1c0 0x7f9268164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.581+0000 7f926d95c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92680cff70 con 0x7f926815ce40 2024-01-26T23:30:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.581+0000 7f926d95c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92680cf400 con 0x7f926815d210 2024-01-26T23:30:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.581+0000 7f926d95c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92680cf660 con 0x7f92680cc1c0 2024-01-26T23:30:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.581+0000 7f92677fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f92680cc1c0 0x7f9268164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.581+0000 7f92677fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f92680cc1c0 0x7f9268164700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59938/0 (socket says 172.21.15.46:59938) 2024-01-26T23:30:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.581+0000 7f92677fe700 1 -- 172.21.15.46:0/3511456813 learned_addr learned my addr 172.21.15.46:0/3511456813 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.581+0000 7f9266ffd700 1 --1- 172.21.15.46:0/3511456813 >> v1:172.21.15.46:6789/0 conn(0x7f926815ce40 0x7f92680d0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46948/0 (socket says 172.21.15.46:46948) 2024-01-26T23:30:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.581+0000 7f92677fe700 1 -- 172.21.15.46:0/3511456813 >> v1:172.21.15.106:6789/0 conn(0x7f926815d210 legacy=0x7f92680cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.581+0000 7f92677fe700 1 -- 172.21.15.46:0/3511456813 >> v1:172.21.15.46:6789/0 conn(0x7f926815ce40 legacy=0x7f92680d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.581+0000 7f92677fe700 1 -- 172.21.15.46:0/3511456813 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92680d0500 con 0x7f92680cc1c0 2024-01-26T23:30:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.581+0000 7f9265ffb700 1 -- 172.21.15.46:0/3511456813 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077584979 0 0) 0x7f92680cff70 con 0x7f926815ce40 2024-01-26T23:30:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.582+0000 7f92677fe700 1 --2- 172.21.15.46:0/3511456813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92680cc1c0 0x7f9268164700 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9260009f20 tx=0x7f9260009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82824d19672065a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.582+0000 7f9265ffb700 1 -- 172.21.15.46:0/3511456813 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926000e070 con 0x7f92680cc1c0 2024-01-26T23:30:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.582+0000 7f9265ffb700 1 -- 172.21.15.46:0/3511456813 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926000ba40 con 0x7f92680cc1c0 2024-01-26T23:30:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.582+0000 7f9265ffb700 1 -- 172.21.15.46:0/3511456813 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92600092e0 con 0x7f92680cc1c0 2024-01-26T23:30:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.582+0000 7f926d95c700 1 -- 172.21.15.46:0/3511456813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92680cc1c0 msgr2=0x7f9268164700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.582+0000 7f926d95c700 1 --2- 172.21.15.46:0/3511456813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92680cc1c0 0x7f9268164700 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9260009f20 tx=0x7f9260009840 comp rx=0 tx=0).stop 2024-01-26T23:30:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.582+0000 7f926d95c700 1 -- 172.21.15.46:0/3511456813 shutdown_connections 2024-01-26T23:30:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.582+0000 7f926d95c700 1 --2- 172.21.15.46:0/3511456813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92680cc1c0 0x7f9268164700 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.582+0000 7f926d95c700 1 -- 172.21.15.46:0/3511456813 >> 172.21.15.46:0/3511456813 conn(0x7f926806f840 msgr2=0x7f92680d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.582+0000 7f926d95c700 1 -- 172.21.15.46:0/3511456813 shutdown_connections 2024-01-26T23:30:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.582+0000 7f926d95c700 1 -- 172.21.15.46:0/3511456813 wait complete. 2024-01-26T23:30:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.584+0000 7f926d95c700 1 Processor -- start 2024-01-26T23:30:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.584+0000 7f926d95c700 1 -- start start 2024-01-26T23:30:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.585+0000 7f926d95c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92680cc1c0 0x7f9268154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.585+0000 7f926d95c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f926815ce40 0x7f926815ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.585+0000 7f926d95c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f926815d210 0x7f926815f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.585+0000 7f926d95c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92680ca710 con 0x7f92680cc1c0 2024-01-26T23:30:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.585+0000 7f926d95c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92680c66d0 con 0x7f926815d210 2024-01-26T23:30:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.585+0000 7f926d95c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926815f8f0 con 0x7f926815ce40 2024-01-26T23:30:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.586+0000 7f9266ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92680cc1c0 0x7f9268154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.586+0000 7f9266ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92680cc1c0 0x7f9268154c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59944/0 (socket says 172.21.15.46:59944) 2024-01-26T23:30:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.586+0000 7f9266ffd700 1 -- 172.21.15.46:0/1108030617 learned_addr learned my addr 172.21.15.46:0/1108030617 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.586+0000 7f92677fe700 1 --2- 172.21.15.46:0/1108030617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f926815d210 0x7f926815f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.586+0000 7f92667fc700 1 --2- 172.21.15.46:0/1108030617 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f926815ce40 0x7f926815ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.586+0000 7f9266ffd700 1 -- 172.21.15.46:0/1108030617 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f926815ce40 msgr2=0x7f926815ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.586+0000 7f9266ffd700 1 --2- 172.21.15.46:0/1108030617 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f926815ce40 0x7f926815ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.586+0000 7f9266ffd700 1 -- 172.21.15.46:0/1108030617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f926815d210 msgr2=0x7f926815f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.586+0000 7f9266ffd700 1 --2- 172.21.15.46:0/1108030617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f926815d210 0x7f926815f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.586+0000 7f9266ffd700 1 -- 172.21.15.46:0/1108030617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9260009ae0 con 0x7f92680cc1c0 2024-01-26T23:30:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.586+0000 7f92667fc700 1 --2- 172.21.15.46:0/1108030617 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f926815ce40 0x7f926815ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.586+0000 7f9266ffd700 1 --2- 172.21.15.46:0/1108030617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92680cc1c0 0x7f9268154c00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f92580011e0 tx=0x7f9258007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:50.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.586+0000 7f924effd700 1 -- 172.21.15.46:0/1108030617 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9258005a70 con 0x7f92680cc1c0 2024-01-26T23:30:50.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.586+0000 7f924effd700 1 -- 172.21.15.46:0/1108030617 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9258005bd0 con 0x7f92680cc1c0 2024-01-26T23:30:50.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.586+0000 7f926d95c700 1 -- 172.21.15.46:0/1108030617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92680c5da0 con 0x7f92680cc1c0 2024-01-26T23:30:50.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.586+0000 7f924effd700 1 -- 172.21.15.46:0/1108030617 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92580070d0 con 0x7f92680cc1c0 2024-01-26T23:30:50.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.586+0000 7f926d95c700 1 -- 172.21.15.46:0/1108030617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92680c6360 con 0x7f92680cc1c0 2024-01-26T23:30:50.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.588+0000 7f926d95c700 1 -- 172.21.15.46:0/1108030617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f926804c710 con 0x7f92680cc1c0 2024-01-26T23:30:50.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.588+0000 7f924effd700 1 -- 172.21.15.46:0/1108030617 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9258007480 con 0x7f92680cc1c0 2024-01-26T23:30:50.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.588+0000 7f924effd700 1 --2- 172.21.15.46:0/1108030617 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9254066f30 0x7f92540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:50.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.588+0000 7f924effd700 1 -- 172.21.15.46:0/1108030617 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f92580837f0 con 0x7f92680cc1c0 2024-01-26T23:30:50.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.589+0000 7f92667fc700 1 --2- 172.21.15.46:0/1108030617 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9254066f30 0x7f92540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:50.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.589+0000 7f92667fc700 1 --2- 172.21.15.46:0/1108030617 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9254066f30 0x7f92540693e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f925c006fd0 tx=0x7f925c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:50.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.593+0000 7f924effd700 1 -- 172.21.15.46:0/1108030617 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9258050d20 con 0x7f92680cc1c0 2024-01-26T23:30:50.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:50.798+0000 7f926d95c700 1 -- 172.21.15.46:0/1108030617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f92680c6520 con 0x7f92680cc1c0 2024-01-26T23:30:51.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.110+0000 7f924effd700 1 -- 172.21.15.46:0/1108030617 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9258017070 con 0x7f92680cc1c0 2024-01-26T23:30:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.114+0000 7f926d95c700 1 -- 172.21.15.46:0/1108030617 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9254066f30 msgr2=0x7f92540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.114+0000 7f926d95c700 1 --2- 172.21.15.46:0/1108030617 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9254066f30 0x7f92540693e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f925c006fd0 tx=0x7f925c008040 comp rx=0 tx=0).stop 2024-01-26T23:30:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.114+0000 7f926d95c700 1 -- 172.21.15.46:0/1108030617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92680cc1c0 msgr2=0x7f9268154c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.114+0000 7f926d95c700 1 --2- 172.21.15.46:0/1108030617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92680cc1c0 0x7f9268154c00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f92580011e0 tx=0x7f9258007e10 comp rx=0 tx=0).stop 2024-01-26T23:30:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.114+0000 7f926d95c700 1 -- 172.21.15.46:0/1108030617 shutdown_connections 2024-01-26T23:30:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.114+0000 7f926d95c700 1 --2- 172.21.15.46:0/1108030617 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9254066f30 0x7f92540693e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.114+0000 7f926d95c700 1 --2- 172.21.15.46:0/1108030617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92680cc1c0 0x7f9268154c00 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.114+0000 7f926d95c700 1 --2- 172.21.15.46:0/1108030617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f926815d210 0x7f926815f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.114+0000 7f926d95c700 1 --2- 172.21.15.46:0/1108030617 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f926815ce40 0x7f926815ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.114+0000 7f926d95c700 1 -- 172.21.15.46:0/1108030617 >> 172.21.15.46:0/1108030617 conn(0x7f926806f840 msgr2=0x7f92680b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.114+0000 7f926d95c700 1 -- 172.21.15.46:0/1108030617 shutdown_connections 2024-01-26T23:30:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.114+0000 7f926d95c700 1 -- 172.21.15.46:0/1108030617 wait complete. 2024-01-26T23:30:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-01-26T23:30:51.135 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-26T23:30:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.275+0000 7fd8d0b87700 1 Processor -- start 2024-01-26T23:30:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.275+0000 7fd8d0b87700 1 -- start start 2024-01-26T23:30:51.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.276+0000 7fd8d0b87700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8cc0c9770 0x7fd8cc159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:51.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.276+0000 7fd8d0b87700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd8cc159760 0x7fd8cc0c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:51.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.276+0000 7fd8d0b87700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8cc15e390 con 0x7fd8cc0c93a0 2024-01-26T23:30:51.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.276+0000 7fd8d0b87700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8cc068460 con 0x7fd8cc0c9770 2024-01-26T23:30:51.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.276+0000 7fd8d0b87700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8cc15d810 con 0x7fd8cc159760 2024-01-26T23:30:51.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.276+0000 7fd8c3fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8cc0c9770 0x7fd8cc159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:51.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.276+0000 7fd8ca59c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd8cc0c93a0 0x7fd8cc067d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46952/0 (socket says 172.21.15.46:46952) 2024-01-26T23:30:51.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.276+0000 7fd8c3fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8cc0c9770 0x7fd8cc159220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59948/0 (socket says 172.21.15.46:59948) 2024-01-26T23:30:51.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.276+0000 7fd8ca59c700 1 -- 172.21.15.46:0/2394090029 learned_addr learned my addr 172.21.15.46:0/2394090029 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:51.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.277+0000 7fd8cad9d700 1 --2- 172.21.15.46:0/2394090029 >> v2:172.21.15.106:3300/0 conn(0x7fd8cc159760 0x7fd8cc0c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:51.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.277+0000 7fd8c3fff700 1 -- 172.21.15.46:0/2394090029 >> v2:172.21.15.106:3300/0 conn(0x7fd8cc159760 msgr2=0x7fd8cc0c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:51.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.277+0000 7fd8c3fff700 1 --2- 172.21.15.46:0/2394090029 >> v2:172.21.15.106:3300/0 conn(0x7fd8cc159760 0x7fd8cc0c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:51.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.277+0000 7fd8c3fff700 1 -- 172.21.15.46:0/2394090029 >> v1:172.21.15.46:6789/0 conn(0x7fd8cc0c93a0 legacy=0x7fd8cc067d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:51.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.277+0000 7fd8c3fff700 1 -- 172.21.15.46:0/2394090029 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc15daf0 con 0x7fd8cc0c9770 2024-01-26T23:30:51.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.277+0000 7fd8c3fff700 1 --2- 172.21.15.46:0/2394090029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8cc0c9770 0x7fd8cc159220 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc0cd300 tx=0x7fd8b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=232a6250da9399a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:51.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.277+0000 7fd8c9d9b700 1 -- 172.21.15.46:0/2394090029 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4011630 con 0x7fd8cc0c9770 2024-01-26T23:30:51.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.277+0000 7fd8c9d9b700 1 -- 172.21.15.46:0/2394090029 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b4011790 con 0x7fd8cc0c9770 2024-01-26T23:30:51.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.277+0000 7fd8c9d9b700 1 -- 172.21.15.46:0/2394090029 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4011b30 con 0x7fd8cc0c9770 2024-01-26T23:30:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.278+0000 7fd8d0b87700 1 -- 172.21.15.46:0/2394090029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8cc0c9770 msgr2=0x7fd8cc159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.278+0000 7fd8d0b87700 1 --2- 172.21.15.46:0/2394090029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8cc0c9770 0x7fd8cc159220 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc0cd300 tx=0x7fd8b40091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.278+0000 7fd8d0b87700 1 -- 172.21.15.46:0/2394090029 shutdown_connections 2024-01-26T23:30:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.278+0000 7fd8d0b87700 1 --2- 172.21.15.46:0/2394090029 >> v2:172.21.15.106:3300/0 conn(0x7fd8cc159760 0x7fd8cc0c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.278+0000 7fd8d0b87700 1 --2- 172.21.15.46:0/2394090029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8cc0c9770 0x7fd8cc159220 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.278+0000 7fd8d0b87700 1 -- 172.21.15.46:0/2394090029 >> 172.21.15.46:0/2394090029 conn(0x7fd8cc0ca9a0 msgr2=0x7fd8cc068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.278+0000 7fd8d0b87700 1 -- 172.21.15.46:0/2394090029 shutdown_connections 2024-01-26T23:30:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.278+0000 7fd8d0b87700 1 -- 172.21.15.46:0/2394090029 wait complete. 2024-01-26T23:30:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.279+0000 7fd8d0b87700 1 Processor -- start 2024-01-26T23:30:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.279+0000 7fd8d0b87700 1 -- start start 2024-01-26T23:30:51.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.280+0000 7fd8d0b87700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8cc0c93a0 0x7fd8cc143860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:51.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.280+0000 7fd8d0b87700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8cc0c9770 0x7fd8cc15c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:51.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.280+0000 7fd8d0b87700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8cc159760 0x7fd8cc0b2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:51.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.280+0000 7fd8d0b87700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc144190 con 0x7fd8cc0c9770 2024-01-26T23:30:51.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.280+0000 7fd8d0b87700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc144470 con 0x7fd8cc0c93a0 2024-01-26T23:30:51.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.280+0000 7fd8d0b87700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc159f40 con 0x7fd8cc159760 2024-01-26T23:30:51.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.280+0000 7fd8c3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8cc0c9770 0x7fd8cc15c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:51.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.280+0000 7fd8c3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8cc0c9770 0x7fd8cc15c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59952/0 (socket says 172.21.15.46:59952) 2024-01-26T23:30:51.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.280+0000 7fd8c3fff700 1 -- 172.21.15.46:0/2990116744 learned_addr learned my addr 172.21.15.46:0/2990116744 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:51.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.280+0000 7fd8cad9d700 1 --2- 172.21.15.46:0/2990116744 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8cc159760 0x7fd8cc0b2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:51.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.280+0000 7fd8ca59c700 1 --2- 172.21.15.46:0/2990116744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8cc0c93a0 0x7fd8cc143860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:51.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.280+0000 7fd8c3fff700 1 -- 172.21.15.46:0/2990116744 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8cc159760 msgr2=0x7fd8cc0b2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:51.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.280+0000 7fd8c3fff700 1 --2- 172.21.15.46:0/2990116744 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8cc159760 0x7fd8cc0b2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:51.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.280+0000 7fd8c3fff700 1 -- 172.21.15.46:0/2990116744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8cc0c93a0 msgr2=0x7fd8cc143860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:51.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.280+0000 7fd8c3fff700 1 --2- 172.21.15.46:0/2990116744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8cc0c93a0 0x7fd8cc143860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:51.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.281+0000 7fd8c3fff700 1 -- 172.21.15.46:0/2990116744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b4009420 con 0x7fd8cc0c9770 2024-01-26T23:30:51.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.281+0000 7fd8c3fff700 1 --2- 172.21.15.46:0/2990116744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8cc0c9770 0x7fd8cc15c960 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009f50 tx=0x7fd8b4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:51.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.281+0000 7fd8c27fc700 1 -- 172.21.15.46:0/2990116744 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4011b10 con 0x7fd8cc0c9770 2024-01-26T23:30:51.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.281+0000 7fd8c27fc700 1 -- 172.21.15.46:0/2990116744 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b40253f0 con 0x7fd8cc0c9770 2024-01-26T23:30:51.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.281+0000 7fd8d0b87700 1 -- 172.21.15.46:0/2990116744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8cc15a220 con 0x7fd8cc0c9770 2024-01-26T23:30:51.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.281+0000 7fd8c27fc700 1 -- 172.21.15.46:0/2990116744 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4025790 con 0x7fd8cc0c9770 2024-01-26T23:30:51.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.281+0000 7fd8d0b87700 1 -- 172.21.15.46:0/2990116744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8cc143da0 con 0x7fd8cc0c9770 2024-01-26T23:30:51.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.282+0000 7fd8d0b87700 1 -- 172.21.15.46:0/2990116744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8cc144600 con 0x7fd8cc0c9770 2024-01-26T23:30:51.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.282+0000 7fd8c27fc700 1 -- 172.21.15.46:0/2990116744 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd8b4025bc0 con 0x7fd8cc0c9770 2024-01-26T23:30:51.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.283+0000 7fd8c27fc700 1 --2- 172.21.15.46:0/2990116744 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd8ac066f30 0x7fd8ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:51.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.283+0000 7fd8c27fc700 1 -- 172.21.15.46:0/2990116744 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5543+0+0 (secure 0 0 0) 0x7fd8b4085c50 con 0x7fd8cc0c9770 2024-01-26T23:30:51.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.283+0000 7fd8ca59c700 1 --2- 172.21.15.46:0/2990116744 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd8ac066f30 0x7fd8ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:51.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.284+0000 7fd8ca59c700 1 --2- 172.21.15.46:0/2990116744 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd8ac066f30 0x7fd8ac0693e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc001780 tx=0x7fd8bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:51.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.288+0000 7fd8c27fc700 1 -- 172.21.15.46:0/2990116744 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd8b408b020 con 0x7fd8cc0c9770 2024-01-26T23:30:51.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:51.493+0000 7fd8d0b87700 1 -- 172.21.15.46:0/2990116744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd8cc15a740 con 0x7fd8cc0c9770 2024-01-26T23:30:52.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.113+0000 7fd8c27fc700 1 -- 172.21.15.46:0/2990116744 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v36) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd8b40150d0 con 0x7fd8cc0c9770 2024-01-26T23:30:52.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.120+0000 7fd8d0b87700 1 -- 172.21.15.46:0/2990116744 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd8ac066f30 msgr2=0x7fd8ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:52.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.120+0000 7fd8d0b87700 1 --2- 172.21.15.46:0/2990116744 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd8ac066f30 0x7fd8ac0693e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc001780 tx=0x7fd8bc009040 comp rx=0 tx=0).stop 2024-01-26T23:30:52.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.120+0000 7fd8d0b87700 1 -- 172.21.15.46:0/2990116744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8cc0c9770 msgr2=0x7fd8cc15c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:52.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.120+0000 7fd8d0b87700 1 --2- 172.21.15.46:0/2990116744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8cc0c9770 0x7fd8cc15c960 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009f50 tx=0x7fd8b4017000 comp rx=0 tx=0).stop 2024-01-26T23:30:52.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.122+0000 7fd8d0b87700 1 -- 172.21.15.46:0/2990116744 shutdown_connections 2024-01-26T23:30:52.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.122+0000 7fd8d0b87700 1 --2- 172.21.15.46:0/2990116744 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd8ac066f30 0x7fd8ac0693e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:52.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.122+0000 7fd8d0b87700 1 --2- 172.21.15.46:0/2990116744 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8cc159760 0x7fd8cc0b2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:52.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.122+0000 7fd8d0b87700 1 --2- 172.21.15.46:0/2990116744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8cc0c93a0 0x7fd8cc143860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:52.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.122+0000 7fd8d0b87700 1 --2- 172.21.15.46:0/2990116744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8cc0c9770 0x7fd8cc15c960 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:52.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.122+0000 7fd8d0b87700 1 -- 172.21.15.46:0/2990116744 >> 172.21.15.46:0/2990116744 conn(0x7fd8cc0ca9a0 msgr2=0x7fd8cc156290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:52.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.122+0000 7fd8d0b87700 1 -- 172.21.15.46:0/2990116744 shutdown_connections 2024-01-26T23:30:52.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.122+0000 7fd8d0b87700 1 -- 172.21.15.46:0/2990116744 wait complete. 2024-01-26T23:30:52.128 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-01-26T23:30:52.148 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-26T23:30:52.153 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:30:52.194 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:30:52.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.287+0000 7f73d5a13700 1 Processor -- start 2024-01-26T23:30:52.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.287+0000 7f73d5a13700 1 -- start start 2024-01-26T23:30:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.287+0000 7f73d5a13700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73d015dfd0 0x7f73d00ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.288+0000 7f73d5a13700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d015a200 con 0x7f73d0159da0 2024-01-26T23:30:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.288+0000 7f73d5a13700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d00b1a60 con 0x7f73d01599d0 2024-01-26T23:30:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.288+0000 7f73d5a13700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73d00b11c0 con 0x7f73d015dfd0 2024-01-26T23:30:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.288+0000 7f73cf7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73d015dfd0 0x7f73d00ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.288+0000 7f73cf7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73d015dfd0 0x7f73d00ab1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59960/0 (socket says 172.21.15.46:59960) 2024-01-26T23:30:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.288+0000 7f73ce7fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f73d0159da0 0x7f73d015d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46958/0 (socket says 172.21.15.46:46958) 2024-01-26T23:30:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.288+0000 7f73cf7fe700 1 -- 172.21.15.46:0/2218216674 learned_addr learned my addr 172.21.15.46:0/2218216674 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.288+0000 7f73cf7fe700 1 -- 172.21.15.46:0/2218216674 >> v1:172.21.15.106:6789/0 conn(0x7f73d01599d0 legacy=0x7f73d00b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.288+0000 7f73cf7fe700 1 -- 172.21.15.46:0/2218216674 >> v1:172.21.15.46:6789/0 conn(0x7f73d0159da0 legacy=0x7f73d015d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.288+0000 7f73cf7fe700 1 -- 172.21.15.46:0/2218216674 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d00b14a0 con 0x7f73d015dfd0 2024-01-26T23:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.288+0000 7f73cdffb700 1 -- 172.21.15.46:0/2218216674 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799883816 0 0) 0x7f73d015a200 con 0x7f73d0159da0 2024-01-26T23:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.288+0000 7f73cf7fe700 1 --2- 172.21.15.46:0/2218216674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73d015dfd0 0x7f73d00ab1c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f73c8009f20 tx=0x7f73c8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f826de611f0bbbb7 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.288+0000 7f73cdffb700 1 -- 172.21.15.46:0/2218216674 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c800e070 con 0x7f73d015dfd0 2024-01-26T23:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.288+0000 7f73cdffb700 1 -- 172.21.15.46:0/2218216674 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c800ba80 con 0x7f73d015dfd0 2024-01-26T23:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.288+0000 7f73cdffb700 1 -- 172.21.15.46:0/2218216674 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c80092e0 con 0x7f73d015dfd0 2024-01-26T23:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.289+0000 7f73d5a13700 1 -- 172.21.15.46:0/2218216674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73d015dfd0 msgr2=0x7f73d00ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.289+0000 7f73d5a13700 1 --2- 172.21.15.46:0/2218216674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73d015dfd0 0x7f73d00ab1c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f73c8009f20 tx=0x7f73c8009840 comp rx=0 tx=0).stop 2024-01-26T23:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.289+0000 7f73d5a13700 1 -- 172.21.15.46:0/2218216674 shutdown_connections 2024-01-26T23:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.289+0000 7f73d5a13700 1 --2- 172.21.15.46:0/2218216674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73d015dfd0 0x7f73d00ab1c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.289+0000 7f73d5a13700 1 -- 172.21.15.46:0/2218216674 >> 172.21.15.46:0/2218216674 conn(0x7f73d006f840 msgr2=0x7f73d0087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.289+0000 7f73d5a13700 1 -- 172.21.15.46:0/2218216674 shutdown_connections 2024-01-26T23:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.289+0000 7f73d5a13700 1 -- 172.21.15.46:0/2218216674 wait complete. 2024-01-26T23:30:52.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.291+0000 7f73d5a13700 1 Processor -- start 2024-01-26T23:30:52.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.291+0000 7f73d5a13700 1 -- start start 2024-01-26T23:30:52.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.291+0000 7f73d5a13700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73d01599d0 0x7f73d00ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:52.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.291+0000 7f73d5a13700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73d0159da0 0x7f73d016de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:52.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.291+0000 7f73d5a13700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73d015dfd0 0x7f73d01585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:52.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.291+0000 7f73d5a13700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d00addd0 con 0x7f73d01599d0 2024-01-26T23:30:52.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.291+0000 7f73d5a13700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d016e360 con 0x7f73d0159da0 2024-01-26T23:30:52.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.291+0000 7f73d5a13700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d00abda0 con 0x7f73d015dfd0 2024-01-26T23:30:52.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.292+0000 7f73ceffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73d01599d0 0x7f73d00ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:52.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.292+0000 7f73ceffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73d01599d0 0x7f73d00ab7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59966/0 (socket says 172.21.15.46:59966) 2024-01-26T23:30:52.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.292+0000 7f73ceffd700 1 -- 172.21.15.46:0/2624396793 learned_addr learned my addr 172.21.15.46:0/2624396793 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:52.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.292+0000 7f73ce7fc700 1 --2- 172.21.15.46:0/2624396793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73d0159da0 0x7f73d016de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:52.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.292+0000 7f73cf7fe700 1 --2- 172.21.15.46:0/2624396793 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73d015dfd0 0x7f73d01585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:52.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.292+0000 7f73ceffd700 1 -- 172.21.15.46:0/2624396793 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73d015dfd0 msgr2=0x7f73d01585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:52.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.292+0000 7f73ceffd700 1 --2- 172.21.15.46:0/2624396793 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73d015dfd0 0x7f73d01585d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:52.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.292+0000 7f73ceffd700 1 -- 172.21.15.46:0/2624396793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73d0159da0 msgr2=0x7f73d016de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:52.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.292+0000 7f73ceffd700 1 --2- 172.21.15.46:0/2624396793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73d0159da0 0x7f73d016de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:52.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.292+0000 7f73ceffd700 1 -- 172.21.15.46:0/2624396793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c8009ae0 con 0x7f73d01599d0 2024-01-26T23:30:52.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.292+0000 7f73cf7fe700 1 --2- 172.21.15.46:0/2624396793 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73d015dfd0 0x7f73d01585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:52.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.292+0000 7f73ceffd700 1 --2- 172.21.15.46:0/2624396793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73d01599d0 0x7f73d00ab7c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f73c000c7c0 tx=0x7f73c0007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:52.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.292+0000 7f73b6ffd700 1 -- 172.21.15.46:0/2624396793 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c0011590 con 0x7f73d01599d0 2024-01-26T23:30:52.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.292+0000 7f73b6ffd700 1 -- 172.21.15.46:0/2624396793 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c0011710 con 0x7f73d01599d0 2024-01-26T23:30:52.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.292+0000 7f73d5a13700 1 -- 172.21.15.46:0/2624396793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d00ad780 con 0x7f73d01599d0 2024-01-26T23:30:52.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.292+0000 7f73b6ffd700 1 -- 172.21.15.46:0/2624396793 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c0011ad0 con 0x7f73d01599d0 2024-01-26T23:30:52.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.292+0000 7f73d5a13700 1 -- 172.21.15.46:0/2624396793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d0158b10 con 0x7f73d01599d0 2024-01-26T23:30:52.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.293+0000 7f73d5a13700 1 -- 172.21.15.46:0/2624396793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d004c710 con 0x7f73d01599d0 2024-01-26T23:30:52.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.294+0000 7f73b6ffd700 1 -- 172.21.15.46:0/2624396793 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f73c001b3b0 con 0x7f73d01599d0 2024-01-26T23:30:52.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.294+0000 7f73b6ffd700 1 --2- 172.21.15.46:0/2624396793 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73bc066f30 0x7f73bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:52.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.295+0000 7f73b6ffd700 1 -- 172.21.15.46:0/2624396793 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5925+0+0 (secure 0 0 0) 0x7f73c0083a40 con 0x7f73d01599d0 2024-01-26T23:30:52.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.295+0000 7f73ce7fc700 1 --2- 172.21.15.46:0/2624396793 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73bc066f30 0x7f73bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:52.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.295+0000 7f73ce7fc700 1 --2- 172.21.15.46:0/2624396793 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73bc066f30 0x7f73bc0693e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f73d0159240 tx=0x7f73c4007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:52.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.299+0000 7f73b6ffd700 1 -- 172.21.15.46:0/2624396793 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f73c0050be0 con 0x7f73d01599d0 2024-01-26T23:30:52.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:52.551+0000 7f73d5a13700 1 -- 172.21.15.46:0/2624396793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f73d00b17f0 con 0x7f73d01599d0 2024-01-26T23:30:53.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.144+0000 7f73b6ffd700 1 -- 172.21.15.46:0/2624396793 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f73c0017070 con 0x7f73d01599d0 2024-01-26T23:30:53.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.150+0000 7f73d5a13700 1 -- 172.21.15.46:0/2624396793 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73bc066f30 msgr2=0x7f73bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:53.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.150+0000 7f73d5a13700 1 --2- 172.21.15.46:0/2624396793 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73bc066f30 0x7f73bc0693e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f73d0159240 tx=0x7f73c4007d30 comp rx=0 tx=0).stop 2024-01-26T23:30:53.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.150+0000 7f73d5a13700 1 -- 172.21.15.46:0/2624396793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73d01599d0 msgr2=0x7f73d00ab7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:53.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.150+0000 7f73d5a13700 1 --2- 172.21.15.46:0/2624396793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73d01599d0 0x7f73d00ab7c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f73c000c7c0 tx=0x7f73c0007f70 comp rx=0 tx=0).stop 2024-01-26T23:30:53.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.155+0000 7f73d5a13700 1 -- 172.21.15.46:0/2624396793 shutdown_connections 2024-01-26T23:30:53.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.155+0000 7f73d5a13700 1 --2- 172.21.15.46:0/2624396793 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73bc066f30 0x7f73bc0693e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.155+0000 7f73d5a13700 1 --2- 172.21.15.46:0/2624396793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73d01599d0 0x7f73d00ab7c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.155+0000 7f73d5a13700 1 --2- 172.21.15.46:0/2624396793 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73d015dfd0 0x7f73d01585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.155+0000 7f73d5a13700 1 --2- 172.21.15.46:0/2624396793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73d0159da0 0x7f73d016de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.155+0000 7f73d5a13700 1 -- 172.21.15.46:0/2624396793 >> 172.21.15.46:0/2624396793 conn(0x7f73d006f840 msgr2=0x7f73d015d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:53.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.155+0000 7f73d5a13700 1 -- 172.21.15.46:0/2624396793 shutdown_connections 2024-01-26T23:30:53.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.155+0000 7f73d5a13700 1 -- 172.21.15.46:0/2624396793 wait complete. 2024-01-26T23:30:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 5 and data pool 6 2024-01-26T23:30:53.176 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:53.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.351+0000 7f15bd8a1700 1 Processor -- start 2024-01-26T23:30:53.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.351+0000 7f15bd8a1700 1 -- start start 2024-01-26T23:30:53.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.352+0000 7f15bd8a1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f15b80b2b40 0x7f15b80ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:53.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.352+0000 7f15bd8a1700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f15b80aedd0 0x7f15b80cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:53.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.352+0000 7f15bd8a1700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b815a0b0 con 0x7f15b80ae960 2024-01-26T23:30:53.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.352+0000 7f15bd8a1700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b8099fc0 con 0x7f15b80b2b40 2024-01-26T23:30:53.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.352+0000 7f15bd8a1700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b80ae4d0 con 0x7f15b80aedd0 2024-01-26T23:30:53.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.352+0000 7f15b6ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f15b80b2b40 0x7f15b80ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:53.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.352+0000 7f15b67fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f15b80ae960 0x7f15b80ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46962/0 (socket says 172.21.15.46:46962) 2024-01-26T23:30:53.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.352+0000 7f15b67fc700 1 -- 172.21.15.46:0/627397899 learned_addr learned my addr 172.21.15.46:0/627397899 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:53.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.352+0000 7f15b77fe700 1 --2- 172.21.15.46:0/627397899 >> v2:172.21.15.106:3300/0 conn(0x7f15b80aedd0 0x7f15b80cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:53.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.352+0000 7f15b5ffb700 1 -- 172.21.15.46:0/627397899 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922696412 0 0) 0x7f15b815a0b0 con 0x7f15b80ae960 2024-01-26T23:30:53.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.352+0000 7f15b5ffb700 1 -- 172.21.15.46:0/627397899 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a4003570 con 0x7f15b80ae960 2024-01-26T23:30:53.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.352+0000 7f15b6ffd700 1 -- 172.21.15.46:0/627397899 >> v2:172.21.15.106:3300/0 conn(0x7f15b80aedd0 msgr2=0x7f15b80cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:53.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.352+0000 7f15b6ffd700 1 --2- 172.21.15.46:0/627397899 >> v2:172.21.15.106:3300/0 conn(0x7f15b80aedd0 0x7f15b80cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.352+0000 7f15b6ffd700 1 -- 172.21.15.46:0/627397899 >> v1:172.21.15.46:6789/0 conn(0x7f15b80ae960 legacy=0x7f15b80ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:53.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.352+0000 7f15b6ffd700 1 -- 172.21.15.46:0/627397899 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b80ad8c0 con 0x7f15b80b2b40 2024-01-26T23:30:53.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.353+0000 7f15b6ffd700 1 --2- 172.21.15.46:0/627397899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b80b2b40 0x7f15b80ad1c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f15b8089b60 tx=0x7f15a8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26fd26b24ffaf0be server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:53.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.353+0000 7f15b5ffb700 1 -- 172.21.15.46:0/627397899 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a8005c10 con 0x7f15b80b2b40 2024-01-26T23:30:53.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.353+0000 7f15b5ffb700 1 -- 172.21.15.46:0/627397899 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a8005d70 con 0x7f15b80b2b40 2024-01-26T23:30:53.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.353+0000 7f15b5ffb700 1 -- 172.21.15.46:0/627397899 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a8006260 con 0x7f15b80b2b40 2024-01-26T23:30:53.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.353+0000 7f15bd8a1700 1 -- 172.21.15.46:0/627397899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b80b2b40 msgr2=0x7f15b80ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:53.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.353+0000 7f15bd8a1700 1 --2- 172.21.15.46:0/627397899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b80b2b40 0x7f15b80ad1c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f15b8089b60 tx=0x7f15a8009120 comp rx=0 tx=0).stop 2024-01-26T23:30:53.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.353+0000 7f15bd8a1700 1 -- 172.21.15.46:0/627397899 shutdown_connections 2024-01-26T23:30:53.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.353+0000 7f15bd8a1700 1 --2- 172.21.15.46:0/627397899 >> v2:172.21.15.106:3300/0 conn(0x7f15b80aedd0 0x7f15b80cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.353+0000 7f15bd8a1700 1 --2- 172.21.15.46:0/627397899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b80b2b40 0x7f15b80ad1c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.353+0000 7f15bd8a1700 1 -- 172.21.15.46:0/627397899 >> 172.21.15.46:0/627397899 conn(0x7f15b8099a30 msgr2=0x7f15b806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:53.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.353+0000 7f15bd8a1700 1 -- 172.21.15.46:0/627397899 shutdown_connections 2024-01-26T23:30:53.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.353+0000 7f15bd8a1700 1 -- 172.21.15.46:0/627397899 wait complete. 2024-01-26T23:30:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.355+0000 7f15bd8a1700 1 Processor -- start 2024-01-26T23:30:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.355+0000 7f15bd8a1700 1 -- start start 2024-01-26T23:30:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.356+0000 7f15bd8a1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15b80ae960 0x7f15b8157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.356+0000 7f15bd8a1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b80b2b40 0x7f15b80b3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.356+0000 7f15bd8a1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15b80b3a50 0x7f15b80ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.356+0000 7f15bd8a1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b80afc80 con 0x7f15b80b2b40 2024-01-26T23:30:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.356+0000 7f15bd8a1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b80af950 con 0x7f15b80ae960 2024-01-26T23:30:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.356+0000 7f15bd8a1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b80b3ed0 con 0x7f15b80b3a50 2024-01-26T23:30:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.356+0000 7f15b67fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b80b2b40 0x7f15b80b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.357+0000 7f15b67fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b80b2b40 0x7f15b80b3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59988/0 (socket says 172.21.15.46:59988) 2024-01-26T23:30:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.357+0000 7f15b67fc700 1 -- 172.21.15.46:0/133849859 learned_addr learned my addr 172.21.15.46:0/133849859 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.357+0000 7f15b6ffd700 1 --2- 172.21.15.46:0/133849859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15b80ae960 0x7f15b8157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.357+0000 7f15b77fe700 1 --2- 172.21.15.46:0/133849859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15b80b3a50 0x7f15b80ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.357+0000 7f15b67fc700 1 -- 172.21.15.46:0/133849859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15b80b3a50 msgr2=0x7f15b80ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.357+0000 7f15b67fc700 1 --2- 172.21.15.46:0/133849859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15b80b3a50 0x7f15b80ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.357+0000 7f15b67fc700 1 -- 172.21.15.46:0/133849859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15b80ae960 msgr2=0x7f15b8157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:53.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.357+0000 7f15b67fc700 1 --2- 172.21.15.46:0/133849859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15b80ae960 0x7f15b8157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.357+0000 7f15b67fc700 1 -- 172.21.15.46:0/133849859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a80093c0 con 0x7f15b80b2b40 2024-01-26T23:30:53.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.357+0000 7f15b67fc700 1 --2- 172.21.15.46:0/133849859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b80b2b40 0x7f15b80b3510 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f15a000c590 tx=0x7f15a0007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:53.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.357+0000 7f15aeffd700 1 -- 172.21.15.46:0/133849859 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a0005b30 con 0x7f15b80b2b40 2024-01-26T23:30:53.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.357+0000 7f15aeffd700 1 -- 172.21.15.46:0/133849859 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a0005cb0 con 0x7f15b80b2b40 2024-01-26T23:30:53.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.357+0000 7f15bd8a1700 1 -- 172.21.15.46:0/133849859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b80b41b0 con 0x7f15b80b2b40 2024-01-26T23:30:53.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.357+0000 7f15aeffd700 1 -- 172.21.15.46:0/133849859 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a0007320 con 0x7f15b80b2b40 2024-01-26T23:30:53.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.357+0000 7f15bd8a1700 1 -- 172.21.15.46:0/133849859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b81578a0 con 0x7f15b80b2b40 2024-01-26T23:30:53.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.358+0000 7f15bd8a1700 1 -- 172.21.15.46:0/133849859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b80afdf0 con 0x7f15b80b2b40 2024-01-26T23:30:53.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.359+0000 7f15aeffd700 1 -- 172.21.15.46:0/133849859 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f15a001b3b0 con 0x7f15b80b2b40 2024-01-26T23:30:53.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.359+0000 7f15aeffd700 1 --2- 172.21.15.46:0/133849859 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1598066f30 0x7f15980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:53.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.359+0000 7f15aeffd700 1 -- 172.21.15.46:0/133849859 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f15a0083aa0 con 0x7f15b80b2b40 2024-01-26T23:30:53.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.359+0000 7f15b6ffd700 1 --2- 172.21.15.46:0/133849859 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1598066f30 0x7f15980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:53.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.360+0000 7f15b6ffd700 1 --2- 172.21.15.46:0/133849859 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1598066f30 0x7f15980693e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f15a8009160 tx=0x7f15a8016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:53.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.364+0000 7f15aeffd700 1 -- 172.21.15.46:0/133849859 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f15a0050bd0 con 0x7f15b80b2b40 2024-01-26T23:30:53.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.569+0000 7f15bd8a1700 1 -- 172.21.15.46:0/133849859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15b804c710 con 0x7f15b80b2b40 2024-01-26T23:30:53.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.570+0000 7f15aeffd700 1 -- 172.21.15.46:0/133849859 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+15568 (secure 0 0 0) 0x7f15a0017020 con 0x7f15b80b2b40 2024-01-26T23:30:53.572 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:53.572 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:30:52.553223+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:30:50.831232+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-01-26T23:30:51.525734+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:53.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.573+0000 7f15bd8a1700 1 -- 172.21.15.46:0/133849859 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1598066f30 msgr2=0x7f15980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:53.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.573+0000 7f15bd8a1700 1 --2- 172.21.15.46:0/133849859 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1598066f30 0x7f15980693e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f15a8009160 tx=0x7f15a8016000 comp rx=0 tx=0).stop 2024-01-26T23:30:53.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.573+0000 7f15bd8a1700 1 -- 172.21.15.46:0/133849859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b80b2b40 msgr2=0x7f15b80b3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:53.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.573+0000 7f15bd8a1700 1 --2- 172.21.15.46:0/133849859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b80b2b40 0x7f15b80b3510 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f15a000c590 tx=0x7f15a0007e00 comp rx=0 tx=0).stop 2024-01-26T23:30:53.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.573+0000 7f15bd8a1700 1 -- 172.21.15.46:0/133849859 shutdown_connections 2024-01-26T23:30:53.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.573+0000 7f15bd8a1700 1 --2- 172.21.15.46:0/133849859 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1598066f30 0x7f15980693e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.573+0000 7f15bd8a1700 1 --2- 172.21.15.46:0/133849859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15b80b3a50 0x7f15b80ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.573+0000 7f15bd8a1700 1 --2- 172.21.15.46:0/133849859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15b80ae960 0x7f15b8157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.573+0000 7f15bd8a1700 1 --2- 172.21.15.46:0/133849859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b80b2b40 0x7f15b80b3510 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.573+0000 7f15bd8a1700 1 -- 172.21.15.46:0/133849859 >> 172.21.15.46:0/133849859 conn(0x7f15b8099a30 msgr2=0x7f15b80c6a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:53.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.573+0000 7f15bd8a1700 1 -- 172.21.15.46:0/133849859 shutdown_connections 2024-01-26T23:30:53.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.573+0000 7f15bd8a1700 1 -- 172.21.15.46:0/133849859 wait complete. 2024-01-26T23:30:53.591 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:53.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.721+0000 7ffa37287700 1 Processor -- start 2024-01-26T23:30:53.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.721+0000 7ffa37287700 1 -- start start 2024-01-26T23:30:53.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.722+0000 7ffa37287700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ffa300acf30 0x7ffa300c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:53.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.722+0000 7ffa37287700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa300ad300 0x7ffa300ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:53.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.722+0000 7ffa37287700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa300c4230 con 0x7ffa300ad750 2024-01-26T23:30:53.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.722+0000 7ffa37287700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa30157be0 con 0x7ffa300acf30 2024-01-26T23:30:53.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.722+0000 7ffa37287700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa300cbf10 con 0x7ffa300ad300 2024-01-26T23:30:53.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.722+0000 7ffa35023700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ffa300acf30 0x7ffa300c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:53.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.722+0000 7ffa35824700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7ffa300ad750 0x7ffa3015b270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:47278/0 (socket says 172.21.15.46:47278) 2024-01-26T23:30:53.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.722+0000 7ffa34822700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa300ad300 0x7ffa300ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:53.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.722+0000 7ffa35824700 1 -- 172.21.15.46:0/3898207119 learned_addr learned my addr 172.21.15.46:0/3898207119 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:53.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.723+0000 7ffa23fff700 1 -- 172.21.15.46:0/3898207119 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499736356 0 0) 0x7ffa300c4230 con 0x7ffa300ad750 2024-01-26T23:30:53.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.723+0000 7ffa23fff700 1 -- 172.21.15.46:0/3898207119 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa1c003570 con 0x7ffa300ad750 2024-01-26T23:30:53.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.723+0000 7ffa35023700 1 -- 172.21.15.46:0/3898207119 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa300ad300 msgr2=0x7ffa300ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:53.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.723+0000 7ffa35023700 1 --2- 172.21.15.46:0/3898207119 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa300ad300 0x7ffa300ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.723+0000 7ffa35023700 1 -- 172.21.15.46:0/3898207119 >> v1:172.21.15.106:6789/0 conn(0x7ffa300ad750 legacy=0x7ffa3015b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:53.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.723+0000 7ffa35023700 1 -- 172.21.15.46:0/3898207119 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa300c4460 con 0x7ffa300acf30 2024-01-26T23:30:53.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.723+0000 7ffa35023700 1 --2- 172.21.15.46:0/3898207119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa300acf30 0x7ffa300c3cf0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ffa240098e0 tx=0x7ffa240091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81373ba89d80c84b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:53.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.723+0000 7ffa23fff700 1 -- 172.21.15.46:0/3898207119 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24011630 con 0x7ffa300acf30 2024-01-26T23:30:53.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.723+0000 7ffa23fff700 1 -- 172.21.15.46:0/3898207119 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa24011790 con 0x7ffa300acf30 2024-01-26T23:30:53.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.723+0000 7ffa23fff700 1 -- 172.21.15.46:0/3898207119 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24011b30 con 0x7ffa300acf30 2024-01-26T23:30:53.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.724+0000 7ffa37287700 1 -- 172.21.15.46:0/3898207119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa300acf30 msgr2=0x7ffa300c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:53.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.724+0000 7ffa37287700 1 --2- 172.21.15.46:0/3898207119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa300acf30 0x7ffa300c3cf0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ffa240098e0 tx=0x7ffa240091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.724+0000 7ffa37287700 1 -- 172.21.15.46:0/3898207119 shutdown_connections 2024-01-26T23:30:53.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.724+0000 7ffa37287700 1 --2- 172.21.15.46:0/3898207119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa300acf30 0x7ffa300c3cf0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.724+0000 7ffa37287700 1 --2- 172.21.15.46:0/3898207119 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa300ad300 0x7ffa300ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.724+0000 7ffa37287700 1 -- 172.21.15.46:0/3898207119 >> 172.21.15.46:0/3898207119 conn(0x7ffa30097820 msgr2=0x7ffa30068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:53.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.724+0000 7ffa37287700 1 -- 172.21.15.46:0/3898207119 shutdown_connections 2024-01-26T23:30:53.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.724+0000 7ffa37287700 1 -- 172.21.15.46:0/3898207119 wait complete. 2024-01-26T23:30:53.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.725+0000 7ffa37287700 1 Processor -- start 2024-01-26T23:30:53.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.725+0000 7ffa37287700 1 -- start start 2024-01-26T23:30:53.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.726+0000 7ffa37287700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa300acf30 0x7ffa300c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:53.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.726+0000 7ffa37287700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa300ad300 0x7ffa301533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:53.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.726+0000 7ffa37287700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa300ad750 0x7ffa300ae730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:53.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.726+0000 7ffa37287700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa300cb7c0 con 0x7ffa300ad750 2024-01-26T23:30:53.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.726+0000 7ffa37287700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa300ca720 con 0x7ffa300ad300 2024-01-26T23:30:53.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.726+0000 7ffa37287700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa300caa00 con 0x7ffa300acf30 2024-01-26T23:30:53.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.726+0000 7ffa35824700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa300ad750 0x7ffa300ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:53.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.726+0000 7ffa35824700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa300ad750 0x7ffa300ae730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59998/0 (socket says 172.21.15.46:59998) 2024-01-26T23:30:53.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.726+0000 7ffa35824700 1 -- 172.21.15.46:0/97242522 learned_addr learned my addr 172.21.15.46:0/97242522 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:53.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.726+0000 7ffa35023700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa300acf30 0x7ffa300c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:53.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.726+0000 7ffa34822700 1 --2- 172.21.15.46:0/97242522 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa300ad300 0x7ffa301533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:53.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.727+0000 7ffa35824700 1 -- 172.21.15.46:0/97242522 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa300acf30 msgr2=0x7ffa300c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:53.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.727+0000 7ffa35824700 1 --2- 172.21.15.46:0/97242522 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa300acf30 0x7ffa300c9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.727+0000 7ffa35824700 1 -- 172.21.15.46:0/97242522 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa300ad300 msgr2=0x7ffa301533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:53.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.727+0000 7ffa35824700 1 --2- 172.21.15.46:0/97242522 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa300ad300 0x7ffa301533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.727+0000 7ffa35824700 1 -- 172.21.15.46:0/97242522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa24009420 con 0x7ffa300ad750 2024-01-26T23:30:53.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.727+0000 7ffa35824700 1 --2- 172.21.15.46:0/97242522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa300ad750 0x7ffa300ae730 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ffa2800cc40 tx=0x7ffa28001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:53.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.727+0000 7ffa217fa700 1 -- 172.21.15.46:0/97242522 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2800a8b0 con 0x7ffa300ad750 2024-01-26T23:30:53.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.727+0000 7ffa217fa700 1 -- 172.21.15.46:0/97242522 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2800aa10 con 0x7ffa300ad750 2024-01-26T23:30:53.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.727+0000 7ffa37287700 1 -- 172.21.15.46:0/97242522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa300cace0 con 0x7ffa300ad750 2024-01-26T23:30:53.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.727+0000 7ffa217fa700 1 -- 172.21.15.46:0/97242522 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2800ad90 con 0x7ffa300ad750 2024-01-26T23:30:53.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.727+0000 7ffa37287700 1 -- 172.21.15.46:0/97242522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa300cb2a0 con 0x7ffa300ad750 2024-01-26T23:30:53.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.728+0000 7ffa37287700 1 -- 172.21.15.46:0/97242522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa3004c710 con 0x7ffa300ad750 2024-01-26T23:30:53.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.729+0000 7ffa217fa700 1 -- 172.21.15.46:0/97242522 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ffa28007300 con 0x7ffa300ad750 2024-01-26T23:30:53.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.729+0000 7ffa217fa700 1 --2- 172.21.15.46:0/97242522 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffa14066f30 0x7ffa140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:53.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.729+0000 7ffa217fa700 1 -- 172.21.15.46:0/97242522 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6025+0+0 (secure 0 0 0) 0x7ffa28084960 con 0x7ffa300ad750 2024-01-26T23:30:53.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.729+0000 7ffa35023700 1 --2- 172.21.15.46:0/97242522 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffa14066f30 0x7ffa140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:53.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.730+0000 7ffa35023700 1 --2- 172.21.15.46:0/97242522 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffa14066f30 0x7ffa140693e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffa240091e0 tx=0x7ffa24017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:53.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.734+0000 7ffa217fa700 1 -- 172.21.15.46:0/97242522 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffa28051a90 con 0x7ffa300ad750 2024-01-26T23:30:53.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.940+0000 7ffa37287700 1 -- 172.21.15.46:0/97242522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa300c4740 con 0x7ffa300ad750 2024-01-26T23:30:53.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.941+0000 7ffa217fa700 1 -- 172.21.15.46:0/97242522 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+15568 (secure 0 0 0) 0x7ffa280550b0 con 0x7ffa300ad750 2024-01-26T23:30:53.943 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:53.943 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:30:52.553223+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:30:50.831232+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-01-26T23:30:51.525734+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:53.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.943+0000 7ffa37287700 1 -- 172.21.15.46:0/97242522 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffa14066f30 msgr2=0x7ffa140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.943+0000 7ffa37287700 1 --2- 172.21.15.46:0/97242522 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffa14066f30 0x7ffa140693e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffa240091e0 tx=0x7ffa24017000 comp rx=0 tx=0).stop 2024-01-26T23:30:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.943+0000 7ffa37287700 1 -- 172.21.15.46:0/97242522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa300ad750 msgr2=0x7ffa300ae730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.943+0000 7ffa37287700 1 --2- 172.21.15.46:0/97242522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa300ad750 0x7ffa300ae730 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ffa2800cc40 tx=0x7ffa28001a00 comp rx=0 tx=0).stop 2024-01-26T23:30:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.943+0000 7ffa37287700 1 -- 172.21.15.46:0/97242522 shutdown_connections 2024-01-26T23:30:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.943+0000 7ffa37287700 1 --2- 172.21.15.46:0/97242522 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffa14066f30 0x7ffa140693e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.943+0000 7ffa37287700 1 --2- 172.21.15.46:0/97242522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa300ad750 0x7ffa300ae730 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.943+0000 7ffa37287700 1 --2- 172.21.15.46:0/97242522 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa300acf30 0x7ffa300c9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.943+0000 7ffa37287700 1 --2- 172.21.15.46:0/97242522 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa300ad300 0x7ffa301533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.943+0000 7ffa37287700 1 -- 172.21.15.46:0/97242522 >> 172.21.15.46:0/97242522 conn(0x7ffa30097820 msgr2=0x7ffa300aba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.943+0000 7ffa37287700 1 -- 172.21.15.46:0/97242522 shutdown_connections 2024-01-26T23:30:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:53.944+0000 7ffa37287700 1 -- 172.21.15.46:0/97242522 wait complete. 2024-01-26T23:30:53.959 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-26T23:30:54.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.093+0000 7f8333a20700 1 Processor -- start 2024-01-26T23:30:54.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.093+0000 7f8333a20700 1 -- start start 2024-01-26T23:30:54.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.094+0000 7f8333a20700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f832c0c8da0 0x7f832c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:54.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.094+0000 7f8333a20700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f832c0ccc00 con 0x7f832c140af0 2024-01-26T23:30:54.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.094+0000 7f8333a20700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f832c0cc310 con 0x7f832c0caf30 2024-01-26T23:30:54.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.094+0000 7f8333a20700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832c0cbbf0 con 0x7f832c0c8da0 2024-01-26T23:30:54.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.094+0000 7f8330fbb700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f832c140af0 0x7f832c140f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46966/0 (socket says 172.21.15.46:46966) 2024-01-26T23:30:54.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.094+0000 7f8330fbb700 1 -- 172.21.15.46:0/3143211135 learned_addr learned my addr 172.21.15.46:0/3143211135 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:54.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.094+0000 7f83317bc700 1 --2- 172.21.15.46:0/3143211135 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f832c0c8da0 0x7f832c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:54.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.094+0000 7f8323fff700 1 -- 172.21.15.46:0/3143211135 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675530098 0 0) 0x7f832c0ccc00 con 0x7f832c140af0 2024-01-26T23:30:54.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.094+0000 7f8323fff700 1 -- 172.21.15.46:0/3143211135 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8318003570 con 0x7f832c140af0 2024-01-26T23:30:54.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.094+0000 7f8323fff700 1 -- 172.21.15.46:0/3143211135 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868497371 0 0) 0x7f832c0cc310 con 0x7f832c0caf30 2024-01-26T23:30:54.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.095+0000 7f8323fff700 1 -- 172.21.15.46:0/3143211135 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f832c0ccc00 con 0x7f832c0caf30 2024-01-26T23:30:54.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.095+0000 7f83317bc700 1 -- 172.21.15.46:0/3143211135 >> v1:172.21.15.106:6789/0 conn(0x7f832c0caf30 legacy=0x7f832c0cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:54.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.095+0000 7f83317bc700 1 -- 172.21.15.46:0/3143211135 >> v1:172.21.15.46:6789/0 conn(0x7f832c140af0 legacy=0x7f832c140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:54.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.095+0000 7f83317bc700 1 -- 172.21.15.46:0/3143211135 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c0cc5b0 con 0x7f832c0c8da0 2024-01-26T23:30:54.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.095+0000 7f83317bc700 1 --2- 172.21.15.46:0/3143211135 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f832c0c8da0 0x7f832c0c9170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8328009910 tx=0x7f83280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9036d4283e6c2b08 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:54.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.095+0000 7f8323fff700 1 -- 172.21.15.46:0/3143211135 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328011630 con 0x7f832c0c8da0 2024-01-26T23:30:54.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.095+0000 7f8323fff700 1 -- 172.21.15.46:0/3143211135 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8328011790 con 0x7f832c0c8da0 2024-01-26T23:30:54.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.095+0000 7f8323fff700 1 -- 172.21.15.46:0/3143211135 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328011b30 con 0x7f832c0c8da0 2024-01-26T23:30:54.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.096+0000 7f8333a20700 1 -- 172.21.15.46:0/3143211135 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f832c0c8da0 msgr2=0x7f832c0c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:54.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.096+0000 7f8333a20700 1 --2- 172.21.15.46:0/3143211135 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f832c0c8da0 0x7f832c0c9170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8328009910 tx=0x7f83280091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:54.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.096+0000 7f8333a20700 1 -- 172.21.15.46:0/3143211135 shutdown_connections 2024-01-26T23:30:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.096+0000 7f8333a20700 1 --2- 172.21.15.46:0/3143211135 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f832c0c8da0 0x7f832c0c9170 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.096+0000 7f8333a20700 1 -- 172.21.15.46:0/3143211135 >> 172.21.15.46:0/3143211135 conn(0x7f832c06bd90 msgr2=0x7f832c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.096+0000 7f8333a20700 1 -- 172.21.15.46:0/3143211135 shutdown_connections 2024-01-26T23:30:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.096+0000 7f8333a20700 1 -- 172.21.15.46:0/3143211135 wait complete. 2024-01-26T23:30:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.097+0000 7f8333a20700 1 Processor -- start 2024-01-26T23:30:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.097+0000 7f8333a20700 1 -- start start 2024-01-26T23:30:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.099+0000 7f8333a20700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f832c0c8da0 0x7f832c157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.099+0000 7f8333a20700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f832c0caf30 0x7f832c0a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.099+0000 7f8333a20700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f832c140af0 0x7f832c0b3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.099+0000 7f8333a20700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c169740 con 0x7f832c0caf30 2024-01-26T23:30:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.099+0000 7f8333a20700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c0a6970 con 0x7f832c0c8da0 2024-01-26T23:30:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.099+0000 7f8333a20700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832c0a6690 con 0x7f832c140af0 2024-01-26T23:30:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.099+0000 7f8330fbb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f832c0caf30 0x7f832c0a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.099+0000 7f8330fbb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f832c0caf30 0x7f832c0a5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60008/0 (socket says 172.21.15.46:60008) 2024-01-26T23:30:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.099+0000 7f8330fbb700 1 -- 172.21.15.46:0/738409048 learned_addr learned my addr 172.21.15.46:0/738409048 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.099+0000 7f83317bc700 1 --2- 172.21.15.46:0/738409048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f832c0c8da0 0x7f832c157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.099+0000 7f8331fbd700 1 --2- 172.21.15.46:0/738409048 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f832c140af0 0x7f832c0b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:54.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.099+0000 7f8330fbb700 1 -- 172.21.15.46:0/738409048 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f832c140af0 msgr2=0x7f832c0b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:54.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.099+0000 7f8330fbb700 1 --2- 172.21.15.46:0/738409048 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f832c140af0 0x7f832c0b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:54.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.099+0000 7f8330fbb700 1 -- 172.21.15.46:0/738409048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f832c0c8da0 msgr2=0x7f832c157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:54.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.099+0000 7f8330fbb700 1 --2- 172.21.15.46:0/738409048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f832c0c8da0 0x7f832c157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:54.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.099+0000 7f8330fbb700 1 -- 172.21.15.46:0/738409048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8328009420 con 0x7f832c0caf30 2024-01-26T23:30:54.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.099+0000 7f8330fbb700 1 --2- 172.21.15.46:0/738409048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f832c0caf30 0x7f832c0a5cd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f831c00c790 tx=0x7f831c007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:54.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.100+0000 7f83217fa700 1 -- 172.21.15.46:0/738409048 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c005b30 con 0x7f832c0caf30 2024-01-26T23:30:54.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.100+0000 7f83217fa700 1 -- 172.21.15.46:0/738409048 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831c005cb0 con 0x7f832c0caf30 2024-01-26T23:30:54.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.100+0000 7f8333a20700 1 -- 172.21.15.46:0/738409048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f832c0c6cf0 con 0x7f832c0caf30 2024-01-26T23:30:54.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.100+0000 7f83217fa700 1 -- 172.21.15.46:0/738409048 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c007320 con 0x7f832c0caf30 2024-01-26T23:30:54.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.100+0000 7f8333a20700 1 -- 172.21.15.46:0/738409048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832c0c72b0 con 0x7f832c0caf30 2024-01-26T23:30:54.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.101+0000 7f8333a20700 1 -- 172.21.15.46:0/738409048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f832c04c710 con 0x7f832c0caf30 2024-01-26T23:30:54.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.101+0000 7f83217fa700 1 -- 172.21.15.46:0/738409048 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f831c01b3b0 con 0x7f832c0caf30 2024-01-26T23:30:54.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.102+0000 7f83217fa700 1 --2- 172.21.15.46:0/738409048 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8310066f30 0x7f83100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:54.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.102+0000 7f83217fa700 1 -- 172.21.15.46:0/738409048 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f831c083ae0 con 0x7f832c0caf30 2024-01-26T23:30:54.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.102+0000 7f83317bc700 1 --2- 172.21.15.46:0/738409048 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8310066f30 0x7f83100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:54.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.103+0000 7f83317bc700 1 --2- 172.21.15.46:0/738409048 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8310066f30 0x7f83100693e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8328009860 tx=0x7f8328017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:54.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.106+0000 7f83217fa700 1 -- 172.21.15.46:0/738409048 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f831c050b70 con 0x7f832c0caf30 2024-01-26T23:30:54.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:54.363+0000 7f8333a20700 1 -- 172.21.15.46:0/738409048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f832c0cc890 con 0x7f832c0caf30 2024-01-26T23:30:55.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.149+0000 7f83217fa700 1 -- 172.21.15.46:0/738409048 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f831c017020 con 0x7f832c0caf30 2024-01-26T23:30:55.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.152+0000 7f8333a20700 1 -- 172.21.15.46:0/738409048 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8310066f30 msgr2=0x7f83100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:55.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.152+0000 7f8333a20700 1 --2- 172.21.15.46:0/738409048 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8310066f30 0x7f83100693e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8328009860 tx=0x7f8328017000 comp rx=0 tx=0).stop 2024-01-26T23:30:55.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.152+0000 7f8333a20700 1 -- 172.21.15.46:0/738409048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f832c0caf30 msgr2=0x7f832c0a5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:55.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.152+0000 7f8333a20700 1 --2- 172.21.15.46:0/738409048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f832c0caf30 0x7f832c0a5cd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f831c00c790 tx=0x7f831c007ea0 comp rx=0 tx=0).stop 2024-01-26T23:30:55.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.153+0000 7f8333a20700 1 -- 172.21.15.46:0/738409048 shutdown_connections 2024-01-26T23:30:55.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.153+0000 7f8333a20700 1 --2- 172.21.15.46:0/738409048 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8310066f30 0x7f83100693e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:55.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.153+0000 7f8333a20700 1 --2- 172.21.15.46:0/738409048 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f832c140af0 0x7f832c0b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:55.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.153+0000 7f8333a20700 1 --2- 172.21.15.46:0/738409048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f832c0c8da0 0x7f832c157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:55.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.153+0000 7f8333a20700 1 --2- 172.21.15.46:0/738409048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f832c0caf30 0x7f832c0a5cd0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:55.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.153+0000 7f8333a20700 1 -- 172.21.15.46:0/738409048 >> 172.21.15.46:0/738409048 conn(0x7f832c06bd90 msgr2=0x7f832c13dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:55.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.153+0000 7f8333a20700 1 -- 172.21.15.46:0/738409048 shutdown_connections 2024-01-26T23:30:55.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.153+0000 7f8333a20700 1 -- 172.21.15.46:0/738409048 wait complete. 2024-01-26T23:30:55.170 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-26T23:30:55.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.308+0000 7fe34b798700 1 Processor -- start 2024-01-26T23:30:55.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.308+0000 7fe34b798700 1 -- start start 2024-01-26T23:30:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.309+0000 7fe34b798700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3441611e0 0x7fe34406c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.309+0000 7fe34b798700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe3441615b0 0x7fe344067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.309+0000 7fe34b798700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3440caad0 0x7fe344067e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.309+0000 7fe34b798700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3440ca950 con 0x7fe3441611e0 2024-01-26T23:30:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.309+0000 7fe34b798700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe34406c810 con 0x7fe3441615b0 2024-01-26T23:30:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.309+0000 7fe34b798700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34406caf0 con 0x7fe3440caad0 2024-01-26T23:30:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.309+0000 7fe349534700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3441611e0 0x7fe34406c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.309+0000 7fe349534700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3441611e0 0x7fe34406c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60012/0 (socket says 172.21.15.46:60012) 2024-01-26T23:30:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.309+0000 7fe349534700 1 -- 172.21.15.46:0/288349054 learned_addr learned my addr 172.21.15.46:0/288349054 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.309+0000 7fe349d35700 1 --2- 172.21.15.46:0/288349054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3440caad0 0x7fe344067e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.309+0000 7fe348d33700 1 --2- 172.21.15.46:0/288349054 >> v2:172.21.15.106:3300/0 conn(0x7fe3441615b0 0x7fe344067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.309+0000 7fe349534700 1 -- 172.21.15.46:0/288349054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3440caad0 msgr2=0x7fe344067e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.309+0000 7fe349534700 1 --2- 172.21.15.46:0/288349054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3440caad0 0x7fe344067e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.309+0000 7fe349534700 1 -- 172.21.15.46:0/288349054 >> v2:172.21.15.106:3300/0 conn(0x7fe3441615b0 msgr2=0x7fe344067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.309+0000 7fe349534700 1 --2- 172.21.15.46:0/288349054 >> v2:172.21.15.106:3300/0 conn(0x7fe3441615b0 0x7fe344067930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.309+0000 7fe349534700 1 -- 172.21.15.46:0/288349054 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34406cdd0 con 0x7fe3441611e0 2024-01-26T23:30:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.310+0000 7fe349534700 1 --2- 172.21.15.46:0/288349054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3441611e0 0x7fe34406c110 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe34406e3f0 tx=0x7fe338009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c340d5189dd4f1c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.310+0000 7fe337fff700 1 -- 172.21.15.46:0/288349054 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33800b640 con 0x7fe3441611e0 2024-01-26T23:30:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.310+0000 7fe337fff700 1 -- 172.21.15.46:0/288349054 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33800b7a0 con 0x7fe3441611e0 2024-01-26T23:30:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.310+0000 7fe337fff700 1 -- 172.21.15.46:0/288349054 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33800bb40 con 0x7fe3441611e0 2024-01-26T23:30:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.310+0000 7fe34b798700 1 -- 172.21.15.46:0/288349054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3441611e0 msgr2=0x7fe34406c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.310+0000 7fe34b798700 1 --2- 172.21.15.46:0/288349054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3441611e0 0x7fe34406c110 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe34406e3f0 tx=0x7fe338009250 comp rx=0 tx=0).stop 2024-01-26T23:30:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.310+0000 7fe34b798700 1 -- 172.21.15.46:0/288349054 shutdown_connections 2024-01-26T23:30:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.310+0000 7fe34b798700 1 --2- 172.21.15.46:0/288349054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3441611e0 0x7fe34406c110 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.310+0000 7fe34b798700 1 --2- 172.21.15.46:0/288349054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3440caad0 0x7fe344067e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.310+0000 7fe34b798700 1 --2- 172.21.15.46:0/288349054 >> v2:172.21.15.106:3300/0 conn(0x7fe3441615b0 0x7fe344067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.310+0000 7fe34b798700 1 -- 172.21.15.46:0/288349054 >> 172.21.15.46:0/288349054 conn(0x7fe3440cdf50 msgr2=0x7fe3440c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.311+0000 7fe34b798700 1 -- 172.21.15.46:0/288349054 shutdown_connections 2024-01-26T23:30:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.311+0000 7fe34b798700 1 -- 172.21.15.46:0/288349054 wait complete. 2024-01-26T23:30:55.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.312+0000 7fe34b798700 1 Processor -- start 2024-01-26T23:30:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.312+0000 7fe34b798700 1 -- start start 2024-01-26T23:30:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.313+0000 7fe34b798700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3440caad0 0x7fe34415dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.313+0000 7fe34b798700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3441611e0 0x7fe344162420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.313+0000 7fe34b798700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3441615b0 0x7fe3440b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.313+0000 7fe34b798700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3440b1110 con 0x7fe3440caad0 2024-01-26T23:30:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.313+0000 7fe34b798700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3440b3050 con 0x7fe3441611e0 2024-01-26T23:30:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.313+0000 7fe34b798700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3440b0220 con 0x7fe3441615b0 2024-01-26T23:30:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.313+0000 7fe349534700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3440caad0 0x7fe34415dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.313+0000 7fe349534700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3440caad0 0x7fe34415dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60020/0 (socket says 172.21.15.46:60020) 2024-01-26T23:30:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.313+0000 7fe349534700 1 -- 172.21.15.46:0/63208709 learned_addr learned my addr 172.21.15.46:0/63208709 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.313+0000 7fe348d33700 1 --2- 172.21.15.46:0/63208709 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3441611e0 0x7fe344162420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.313+0000 7fe349d35700 1 --2- 172.21.15.46:0/63208709 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3441615b0 0x7fe3440b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.313+0000 7fe349534700 1 -- 172.21.15.46:0/63208709 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3441615b0 msgr2=0x7fe3440b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.313+0000 7fe349534700 1 --2- 172.21.15.46:0/63208709 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3441615b0 0x7fe3440b15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.313+0000 7fe349534700 1 -- 172.21.15.46:0/63208709 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3441611e0 msgr2=0x7fe344162420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.313+0000 7fe349534700 1 --2- 172.21.15.46:0/63208709 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3441611e0 0x7fe344162420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.313+0000 7fe349534700 1 -- 172.21.15.46:0/63208709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3380094f0 con 0x7fe3440caad0 2024-01-26T23:30:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.313+0000 7fe348d33700 1 --2- 172.21.15.46:0/63208709 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3441611e0 0x7fe344162420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.314+0000 7fe349534700 1 --2- 172.21.15.46:0/63208709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3440caad0 0x7fe34415dce0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe338006fd0 tx=0x7fe338017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.314+0000 7fe3357fa700 1 -- 172.21.15.46:0/63208709 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33800b910 con 0x7fe3440caad0 2024-01-26T23:30:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.314+0000 7fe3357fa700 1 -- 172.21.15.46:0/63208709 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33800ba90 con 0x7fe3440caad0 2024-01-26T23:30:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.314+0000 7fe3357fa700 1 -- 172.21.15.46:0/63208709 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338011670 con 0x7fe3440caad0 2024-01-26T23:30:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.314+0000 7fe34b798700 1 -- 172.21.15.46:0/63208709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3440b0500 con 0x7fe3440caad0 2024-01-26T23:30:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.314+0000 7fe34b798700 1 -- 172.21.15.46:0/63208709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3440b0ac0 con 0x7fe3440caad0 2024-01-26T23:30:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.315+0000 7fe34b798700 1 -- 172.21.15.46:0/63208709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34404c710 con 0x7fe3440caad0 2024-01-26T23:30:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.315+0000 7fe3357fa700 1 -- 172.21.15.46:0/63208709 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe338011a60 con 0x7fe3440caad0 2024-01-26T23:30:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.316+0000 7fe3357fa700 1 --2- 172.21.15.46:0/63208709 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe328066f30 0x7fe3280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.316+0000 7fe3357fa700 1 -- 172.21.15.46:0/63208709 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fe338085e70 con 0x7fe3440caad0 2024-01-26T23:30:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.316+0000 7fe348d33700 1 --2- 172.21.15.46:0/63208709 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe328066f30 0x7fe3280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.316+0000 7fe348d33700 1 --2- 172.21.15.46:0/63208709 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe328066f30 0x7fe3280693e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe344067e30 tx=0x7fe340008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:55.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.320+0000 7fe3357fa700 1 -- 172.21.15.46:0/63208709 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe338052f00 con 0x7fe3440caad0 2024-01-26T23:30:55.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:55.574+0000 7fe34b798700 1 -- 172.21.15.46:0/63208709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe3440ca730 con 0x7fe3440caad0 2024-01-26T23:30:56.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.153+0000 7fe3357fa700 1 -- 172.21.15.46:0/63208709 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe338056520 con 0x7fe3440caad0 2024-01-26T23:30:56.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.156+0000 7fe34b798700 1 -- 172.21.15.46:0/63208709 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe328066f30 msgr2=0x7fe3280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:56.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.156+0000 7fe34b798700 1 --2- 172.21.15.46:0/63208709 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe328066f30 0x7fe3280693e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe344067e30 tx=0x7fe340008040 comp rx=0 tx=0).stop 2024-01-26T23:30:56.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.156+0000 7fe34b798700 1 -- 172.21.15.46:0/63208709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3440caad0 msgr2=0x7fe34415dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:56.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.156+0000 7fe34b798700 1 --2- 172.21.15.46:0/63208709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3440caad0 0x7fe34415dce0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe338006fd0 tx=0x7fe338017000 comp rx=0 tx=0).stop 2024-01-26T23:30:56.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.156+0000 7fe34b798700 1 -- 172.21.15.46:0/63208709 shutdown_connections 2024-01-26T23:30:56.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.156+0000 7fe34b798700 1 --2- 172.21.15.46:0/63208709 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe328066f30 0x7fe3280693e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.156+0000 7fe34b798700 1 --2- 172.21.15.46:0/63208709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3440caad0 0x7fe34415dce0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.156+0000 7fe34b798700 1 --2- 172.21.15.46:0/63208709 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3441615b0 0x7fe3440b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.156+0000 7fe34b798700 1 --2- 172.21.15.46:0/63208709 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3441611e0 0x7fe344162420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.156+0000 7fe34b798700 1 -- 172.21.15.46:0/63208709 >> 172.21.15.46:0/63208709 conn(0x7fe3440cdf50 msgr2=0x7fe3440c2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.156+0000 7fe34b798700 1 -- 172.21.15.46:0/63208709 shutdown_connections 2024-01-26T23:30:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.156+0000 7fe34b798700 1 -- 172.21.15.46:0/63208709 wait complete. 2024-01-26T23:30:56.174 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:30:56.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.306+0000 7fb9f4705700 1 Processor -- start 2024-01-26T23:30:56.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.306+0000 7fb9f4705700 1 -- start start 2024-01-26T23:30:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.307+0000 7fb9f4705700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9ec15ce40 0x7fb9ec0d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.307+0000 7fb9f4705700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb9ec15d210 0x7fb9ec0cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.307+0000 7fb9f4705700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb9ec0d1390 0x7fb9ec0cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.307+0000 7fb9f4705700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9ec0cff70 con 0x7fb9ec15d210 2024-01-26T23:30:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.307+0000 7fb9f4705700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9ec0cf400 con 0x7fb9ec0d1390 2024-01-26T23:30:56.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.307+0000 7fb9f4705700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec0cf660 con 0x7fb9ec15ce40 2024-01-26T23:30:56.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.307+0000 7fb9f1ca0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb9ec15d210 0x7fb9ec0cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:56.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.307+0000 7fb9f1ca0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb9ec15d210 0x7fb9ec0cbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60022/0 (socket says 172.21.15.46:60022) 2024-01-26T23:30:56.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.307+0000 7fb9f1ca0700 1 -- 172.21.15.46:0/2469639515 learned_addr learned my addr 172.21.15.46:0/2469639515 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:56.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.308+0000 7fb9f24a1700 1 --2- 172.21.15.46:0/2469639515 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9ec15ce40 0x7fb9ec0d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:56.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.308+0000 7fb9f2ca2700 1 --2- 172.21.15.46:0/2469639515 >> v2:172.21.15.106:3300/0 conn(0x7fb9ec0d1390 0x7fb9ec0cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:56.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.308+0000 7fb9f1ca0700 1 -- 172.21.15.46:0/2469639515 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9ec15ce40 msgr2=0x7fb9ec0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:56.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.308+0000 7fb9f1ca0700 1 --2- 172.21.15.46:0/2469639515 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9ec15ce40 0x7fb9ec0d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.308+0000 7fb9f1ca0700 1 -- 172.21.15.46:0/2469639515 >> v2:172.21.15.106:3300/0 conn(0x7fb9ec0d1390 msgr2=0x7fb9ec0cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:56.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.308+0000 7fb9f1ca0700 1 --2- 172.21.15.46:0/2469639515 >> v2:172.21.15.106:3300/0 conn(0x7fb9ec0d1390 0x7fb9ec0cbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.308+0000 7fb9f1ca0700 1 -- 172.21.15.46:0/2469639515 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ec0d0500 con 0x7fb9ec15d210 2024-01-26T23:30:56.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.308+0000 7fb9f1ca0700 1 --2- 172.21.15.46:0/2469639515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec15d210 0x7fb9ec0cbab0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec089b70 tx=0x7fb9dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30c65215958c928 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:56.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.308+0000 7fb9f149f700 1 -- 172.21.15.46:0/2469639515 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc00b860 con 0x7fb9ec15d210 2024-01-26T23:30:56.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.308+0000 7fb9f149f700 1 -- 172.21.15.46:0/2469639515 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9dc00b9c0 con 0x7fb9ec15d210 2024-01-26T23:30:56.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.308+0000 7fb9f149f700 1 -- 172.21.15.46:0/2469639515 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc00bd60 con 0x7fb9ec15d210 2024-01-26T23:30:56.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.309+0000 7fb9f4705700 1 -- 172.21.15.46:0/2469639515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec15d210 msgr2=0x7fb9ec0cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:56.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.309+0000 7fb9f4705700 1 --2- 172.21.15.46:0/2469639515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec15d210 0x7fb9ec0cbab0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec089b70 tx=0x7fb9dc009250 comp rx=0 tx=0).stop 2024-01-26T23:30:56.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.309+0000 7fb9f4705700 1 -- 172.21.15.46:0/2469639515 shutdown_connections 2024-01-26T23:30:56.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.309+0000 7fb9f4705700 1 --2- 172.21.15.46:0/2469639515 >> v2:172.21.15.106:3300/0 conn(0x7fb9ec0d1390 0x7fb9ec0cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.309+0000 7fb9f4705700 1 --2- 172.21.15.46:0/2469639515 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9ec15ce40 0x7fb9ec0d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.309+0000 7fb9f4705700 1 --2- 172.21.15.46:0/2469639515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec15d210 0x7fb9ec0cbab0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.309+0000 7fb9f4705700 1 -- 172.21.15.46:0/2469639515 >> 172.21.15.46:0/2469639515 conn(0x7fb9ec06f840 msgr2=0x7fb9ec0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.309+0000 7fb9f4705700 1 -- 172.21.15.46:0/2469639515 shutdown_connections 2024-01-26T23:30:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.309+0000 7fb9f4705700 1 -- 172.21.15.46:0/2469639515 wait complete. 2024-01-26T23:30:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.310+0000 7fb9f4705700 1 Processor -- start 2024-01-26T23:30:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.310+0000 7fb9f4705700 1 -- start start 2024-01-26T23:30:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f4705700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9ec0d1390 0x7fb9ec159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f4705700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec15ce40 0x7fb9ec0c7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f4705700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9ec15d210 0x7fb9ec09ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f4705700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec15a3e0 con 0x7fb9ec15ce40 2024-01-26T23:30:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f4705700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec0adf40 con 0x7fb9ec15d210 2024-01-26T23:30:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f4705700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec09c600 con 0x7fb9ec0d1390 2024-01-26T23:30:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f1ca0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec15ce40 0x7fb9ec0c7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f1ca0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec15ce40 0x7fb9ec0c7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60032/0 (socket says 172.21.15.46:60032) 2024-01-26T23:30:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f1ca0700 1 -- 172.21.15.46:0/3652565800 learned_addr learned my addr 172.21.15.46:0/3652565800 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f2ca2700 1 --2- 172.21.15.46:0/3652565800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9ec15d210 0x7fb9ec09ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f24a1700 1 --2- 172.21.15.46:0/3652565800 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9ec0d1390 0x7fb9ec159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f1ca0700 1 -- 172.21.15.46:0/3652565800 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9ec0d1390 msgr2=0x7fb9ec159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f1ca0700 1 --2- 172.21.15.46:0/3652565800 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9ec0d1390 0x7fb9ec159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f1ca0700 1 -- 172.21.15.46:0/3652565800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9ec15d210 msgr2=0x7fb9ec09ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f1ca0700 1 --2- 172.21.15.46:0/3652565800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9ec15d210 0x7fb9ec09ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f1ca0700 1 -- 172.21.15.46:0/3652565800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc0094f0 con 0x7fb9ec15ce40 2024-01-26T23:30:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f24a1700 1 --2- 172.21.15.46:0/3652565800 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9ec0d1390 0x7fb9ec159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:30:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.311+0000 7fb9f1ca0700 1 --2- 172.21.15.46:0/3652565800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec15ce40 0x7fb9ec0c7c20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec088ec0 tx=0x7fb9dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.312+0000 7fb9e27fc700 1 -- 172.21.15.46:0/3652565800 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc024440 con 0x7fb9ec15ce40 2024-01-26T23:30:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.312+0000 7fb9e27fc700 1 -- 172.21.15.46:0/3652565800 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9dc0245a0 con 0x7fb9ec15ce40 2024-01-26T23:30:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.312+0000 7fb9f4705700 1 -- 172.21.15.46:0/3652565800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9ec09c2b0 con 0x7fb9ec15ce40 2024-01-26T23:30:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.312+0000 7fb9e27fc700 1 -- 172.21.15.46:0/3652565800 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc024960 con 0x7fb9ec15ce40 2024-01-26T23:30:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.312+0000 7fb9f4705700 1 -- 172.21.15.46:0/3652565800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9ec0c8800 con 0x7fb9ec15ce40 2024-01-26T23:30:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.313+0000 7fb9f4705700 1 -- 172.21.15.46:0/3652565800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9ec09c790 con 0x7fb9ec15ce40 2024-01-26T23:30:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.313+0000 7fb9e27fc700 1 -- 172.21.15.46:0/3652565800 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb9dc01a420 con 0x7fb9ec15ce40 2024-01-26T23:30:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.314+0000 7fb9e27fc700 1 --2- 172.21.15.46:0/3652565800 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb9d0066f30 0x7fb9d00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.314+0000 7fb9e27fc700 1 -- 172.21.15.46:0/3652565800 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fb9dc0863c0 con 0x7fb9ec15ce40 2024-01-26T23:30:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.314+0000 7fb9f24a1700 1 --2- 172.21.15.46:0/3652565800 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb9d0066f30 0x7fb9d00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.314+0000 7fb9f24a1700 1 --2- 172.21.15.46:0/3652565800 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb9d0066f30 0x7fb9d00693e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8006aa0 tx=0x7fb9e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:56.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.318+0000 7fb9e27fc700 1 -- 172.21.15.46:0/3652565800 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb9dc0534f0 con 0x7fb9ec15ce40 2024-01-26T23:30:56.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.571+0000 7fb9f4705700 1 -- 172.21.15.46:0/3652565800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9ec0cf920 con 0x7fb9ec15ce40 2024-01-26T23:30:56.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.572+0000 7fb9e27fc700 1 -- 172.21.15.46:0/3652565800 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4597 (secure 0 0 0) 0x7fb9dc056b10 con 0x7fb9ec15ce40 2024-01-26T23:30:56.573 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:56.573 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1737713978","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1737713978},{"type":"v1","addr":"172.21.15.46:6835","nonce":1737713978}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":4969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3857864387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3857864387},{"type":"v1","addr":"172.21.15.106:6836","nonce":3857864387}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/639000251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":639000251},{"type":"v1","addr":"172.21.15.106:6837","nonce":639000251}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:30:52.553580+0000","modified":"2024-01-26T23:30:56.149313+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4984":{"gid":4984,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/453078862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":453078862},{"type":"v1","addr":"172.21.15.46:6834","nonce":453078862}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2024-01-26T23:30:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.574+0000 7fb9f4705700 1 -- 172.21.15.46:0/3652565800 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb9d0066f30 msgr2=0x7fb9d00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.574+0000 7fb9f4705700 1 --2- 172.21.15.46:0/3652565800 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb9d0066f30 0x7fb9d00693e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8006aa0 tx=0x7fb9e8008040 comp rx=0 tx=0).stop 2024-01-26T23:30:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.574+0000 7fb9f4705700 1 -- 172.21.15.46:0/3652565800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec15ce40 msgr2=0x7fb9ec0c7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.574+0000 7fb9f4705700 1 --2- 172.21.15.46:0/3652565800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec15ce40 0x7fb9ec0c7c20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec088ec0 tx=0x7fb9dc017000 comp rx=0 tx=0).stop 2024-01-26T23:30:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.574+0000 7fb9f4705700 1 -- 172.21.15.46:0/3652565800 shutdown_connections 2024-01-26T23:30:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.574+0000 7fb9f4705700 1 --2- 172.21.15.46:0/3652565800 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb9d0066f30 0x7fb9d00693e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.574+0000 7fb9f4705700 1 --2- 172.21.15.46:0/3652565800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9ec15d210 0x7fb9ec09ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.574+0000 7fb9f4705700 1 --2- 172.21.15.46:0/3652565800 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9ec0d1390 0x7fb9ec159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.574+0000 7fb9f4705700 1 --2- 172.21.15.46:0/3652565800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec15ce40 0x7fb9ec0c7c20 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.574+0000 7fb9f4705700 1 -- 172.21.15.46:0/3652565800 >> 172.21.15.46:0/3652565800 conn(0x7fb9ec06f840 msgr2=0x7fb9ec15a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.574+0000 7fb9f4705700 1 -- 172.21.15.46:0/3652565800 shutdown_connections 2024-01-26T23:30:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.574+0000 7fb9f4705700 1 -- 172.21.15.46:0/3652565800 wait complete. 2024-01-26T23:30:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 18 2024-01-26T23:30:56.590 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:30:56.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.721+0000 7f64efd3a700 1 Processor -- start 2024-01-26T23:30:56.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.721+0000 7f64efd3a700 1 -- start start 2024-01-26T23:30:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.722+0000 7f64efd3a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f64e8153ec0 0x7f64e8154290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.722+0000 7f64efd3a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e8157630 0x7f64e80b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.722+0000 7f64efd3a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e80b1500 con 0x7f64e80b2100 2024-01-26T23:30:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.722+0000 7f64efd3a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e80b2580 con 0x7f64e8153ec0 2024-01-26T23:30:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.722+0000 7f64efd3a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e815b140 con 0x7f64e8157630 2024-01-26T23:30:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.722+0000 7f64ed2d5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e8157630 0x7f64e80b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.722+0000 7f64ee2d7700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f64e80b2100 0x7f64e806eb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:47296/0 (socket says 172.21.15.46:47296) 2024-01-26T23:30:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.722+0000 7f64edad6700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f64e8153ec0 0x7f64e8154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.722+0000 7f64ee2d7700 1 -- 172.21.15.46:0/1212247601 learned_addr learned my addr 172.21.15.46:0/1212247601 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.723+0000 7f64ed2d5700 1 -- 172.21.15.46:0/1212247601 >> v2:172.21.15.106:3300/0 conn(0x7f64e8153ec0 msgr2=0x7f64e8154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.723+0000 7f64ed2d5700 1 --2- 172.21.15.46:0/1212247601 >> v2:172.21.15.106:3300/0 conn(0x7f64e8153ec0 0x7f64e8154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.723+0000 7f64ecad4700 1 -- 172.21.15.46:0/1212247601 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4037174314 0 0) 0x7f64e80b1500 con 0x7f64e80b2100 2024-01-26T23:30:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.723+0000 7f64ed2d5700 1 -- 172.21.15.46:0/1212247601 >> v1:172.21.15.106:6789/0 conn(0x7f64e80b2100 legacy=0x7f64e806eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.723+0000 7f64ed2d5700 1 -- 172.21.15.46:0/1212247601 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e815a5c0 con 0x7f64e8157630 2024-01-26T23:30:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.723+0000 7f64ed2d5700 1 --2- 172.21.15.46:0/1212247601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e8157630 0x7f64e80b1bc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f64d80098e0 tx=0x7f64d80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73eebcd02da2c349 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.723+0000 7f64ecad4700 1 -- 172.21.15.46:0/1212247601 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d800b640 con 0x7f64e8157630 2024-01-26T23:30:56.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.723+0000 7f64ecad4700 1 -- 172.21.15.46:0/1212247601 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d800b7a0 con 0x7f64e8157630 2024-01-26T23:30:56.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.723+0000 7f64ecad4700 1 -- 172.21.15.46:0/1212247601 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d800bb40 con 0x7f64e8157630 2024-01-26T23:30:56.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.724+0000 7f64efd3a700 1 -- 172.21.15.46:0/1212247601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e8157630 msgr2=0x7f64e80b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:56.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.724+0000 7f64efd3a700 1 --2- 172.21.15.46:0/1212247601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e8157630 0x7f64e80b1bc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f64d80098e0 tx=0x7f64d80091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.724+0000 7f64efd3a700 1 -- 172.21.15.46:0/1212247601 shutdown_connections 2024-01-26T23:30:56.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.724+0000 7f64efd3a700 1 --2- 172.21.15.46:0/1212247601 >> v2:172.21.15.106:3300/0 conn(0x7f64e8153ec0 0x7f64e8154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.724+0000 7f64efd3a700 1 --2- 172.21.15.46:0/1212247601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e8157630 0x7f64e80b1bc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.724+0000 7f64efd3a700 1 -- 172.21.15.46:0/1212247601 >> 172.21.15.46:0/1212247601 conn(0x7f64e8069510 msgr2=0x7f64e80aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:56.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.724+0000 7f64efd3a700 1 -- 172.21.15.46:0/1212247601 shutdown_connections 2024-01-26T23:30:56.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.724+0000 7f64efd3a700 1 -- 172.21.15.46:0/1212247601 wait complete. 2024-01-26T23:30:56.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.725+0000 7f64efd3a700 1 Processor -- start 2024-01-26T23:30:56.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.725+0000 7f64efd3a700 1 -- start start 2024-01-26T23:30:56.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64efd3a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64e80b2100 0x7f64e8152fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:56.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64efd3a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e8153ec0 0x7f64e80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:56.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64efd3a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e8157630 0x7f64e80a65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:56.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64efd3a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80cc770 con 0x7f64e80b2100 2024-01-26T23:30:56.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64efd3a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e81658a0 con 0x7f64e8157630 2024-01-26T23:30:56.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64efd3a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e8153510 con 0x7f64e8153ec0 2024-01-26T23:30:56.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64edad6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64e80b2100 0x7f64e8152fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:56.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64edad6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64e80b2100 0x7f64e8152fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60042/0 (socket says 172.21.15.46:60042) 2024-01-26T23:30:56.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64edad6700 1 -- 172.21.15.46:0/1288767542 learned_addr learned my addr 172.21.15.46:0/1288767542 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:56.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64ed2d5700 1 --2- 172.21.15.46:0/1288767542 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e8153ec0 0x7f64e80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:56.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64ee2d7700 1 --2- 172.21.15.46:0/1288767542 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e8157630 0x7f64e80a65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:56.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64edad6700 1 -- 172.21.15.46:0/1288767542 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e8153ec0 msgr2=0x7f64e80a6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:56.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64edad6700 1 --2- 172.21.15.46:0/1288767542 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e8153ec0 0x7f64e80a6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64edad6700 1 -- 172.21.15.46:0/1288767542 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e8157630 msgr2=0x7f64e80a65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:56.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64edad6700 1 --2- 172.21.15.46:0/1288767542 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e8157630 0x7f64e80a65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64edad6700 1 -- 172.21.15.46:0/1288767542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d8009420 con 0x7f64e80b2100 2024-01-26T23:30:56.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64edad6700 1 --2- 172.21.15.46:0/1288767542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64e80b2100 0x7f64e8152fd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f64e400e960 tx=0x7f64e40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:56.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.726+0000 7f64ddffb700 1 -- 172.21.15.46:0/1288767542 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e400c850 con 0x7f64e80b2100 2024-01-26T23:30:56.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.727+0000 7f64ddffb700 1 -- 172.21.15.46:0/1288767542 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e400c9b0 con 0x7f64e80b2100 2024-01-26T23:30:56.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.727+0000 7f64efd3a700 1 -- 172.21.15.46:0/1288767542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e80cc9c0 con 0x7f64e80b2100 2024-01-26T23:30:56.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.727+0000 7f64ddffb700 1 -- 172.21.15.46:0/1288767542 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e400cd50 con 0x7f64e80b2100 2024-01-26T23:30:56.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.727+0000 7f64efd3a700 1 -- 172.21.15.46:0/1288767542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e80ccf80 con 0x7f64e80b2100 2024-01-26T23:30:56.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.728+0000 7f64efd3a700 1 -- 172.21.15.46:0/1288767542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e8165b80 con 0x7f64e80b2100 2024-01-26T23:30:56.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.728+0000 7f64ddffb700 1 -- 172.21.15.46:0/1288767542 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f64e4013680 con 0x7f64e80b2100 2024-01-26T23:30:56.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.729+0000 7f64ddffb700 1 --2- 172.21.15.46:0/1288767542 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f64cc066f30 0x7f64cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:56.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.729+0000 7f64ddffb700 1 -- 172.21.15.46:0/1288767542 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f64e4084910 con 0x7f64e80b2100 2024-01-26T23:30:56.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.729+0000 7f64ed2d5700 1 --2- 172.21.15.46:0/1288767542 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f64cc066f30 0x7f64cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:56.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.730+0000 7f64ed2d5700 1 --2- 172.21.15.46:0/1288767542 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f64cc066f30 0x7f64cc0693e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f64d80091e0 tx=0x7f64d8017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:56.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.734+0000 7f64ddffb700 1 -- 172.21.15.46:0/1288767542 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f64e4051a40 con 0x7f64e80b2100 2024-01-26T23:30:56.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.939+0000 7f64efd3a700 1 -- 172.21.15.46:0/1288767542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64e804c710 con 0x7f64e80b2100 2024-01-26T23:30:56.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.940+0000 7f64ddffb700 1 -- 172.21.15.46:0/1288767542 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+15555 (secure 0 0 0) 0x7f64e4017090 con 0x7f64e80b2100 2024-01-26T23:30:56.943 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:56.943 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":38,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:30:54.125375+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:30:50.831232+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-01-26T23:30:51.525734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:30:56.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.942+0000 7f64efd3a700 1 -- 172.21.15.46:0/1288767542 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f64cc066f30 msgr2=0x7f64cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:56.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.942+0000 7f64efd3a700 1 --2- 172.21.15.46:0/1288767542 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f64cc066f30 0x7f64cc0693e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f64d80091e0 tx=0x7f64d8017000 comp rx=0 tx=0).stop 2024-01-26T23:30:56.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.942+0000 7f64efd3a700 1 -- 172.21.15.46:0/1288767542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64e80b2100 msgr2=0x7f64e8152fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:56.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.942+0000 7f64efd3a700 1 --2- 172.21.15.46:0/1288767542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64e80b2100 0x7f64e8152fd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f64e400e960 tx=0x7f64e40091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.942+0000 7f64efd3a700 1 -- 172.21.15.46:0/1288767542 shutdown_connections 2024-01-26T23:30:56.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.942+0000 7f64efd3a700 1 --2- 172.21.15.46:0/1288767542 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f64cc066f30 0x7f64cc0693e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.942+0000 7f64efd3a700 1 --2- 172.21.15.46:0/1288767542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64e80b2100 0x7f64e8152fd0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.942+0000 7f64efd3a700 1 --2- 172.21.15.46:0/1288767542 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e8157630 0x7f64e80a65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.942+0000 7f64efd3a700 1 --2- 172.21.15.46:0/1288767542 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e8153ec0 0x7f64e80a6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:56.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.942+0000 7f64efd3a700 1 -- 172.21.15.46:0/1288767542 >> 172.21.15.46:0/1288767542 conn(0x7f64e8069510 msgr2=0x7f64e806eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.943+0000 7f64efd3a700 1 -- 172.21.15.46:0/1288767542 shutdown_connections 2024-01-26T23:30:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:56.943+0000 7f64efd3a700 1 -- 172.21.15.46:0/1288767542 wait complete. 2024-01-26T23:30:56.959 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-26T23:30:56.959 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:30:57.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.089+0000 7f99b4f2b700 1 Processor -- start 2024-01-26T23:30:57.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.089+0000 7f99b4f2b700 1 -- start start 2024-01-26T23:30:57.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.090+0000 7f99b4f2b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f99b0154290 0x7f99b00b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.090+0000 7f99b4f2b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99b00b2100 0x7f99b00b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.090+0000 7f99b4f2b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b015b140 con 0x7f99b0153ec0 2024-01-26T23:30:57.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.090+0000 7f99b4f2b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b00b2580 con 0x7f99b0154290 2024-01-26T23:30:57.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.090+0000 7f99b4f2b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b015a5c0 con 0x7f99b00b2100 2024-01-26T23:30:57.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.090+0000 7f99ae59c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f99b0153ec0 0x7f99b0157630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46980/0 (socket says 172.21.15.46:46980) 2024-01-26T23:30:57.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.090+0000 7f99ae59c700 1 -- 172.21.15.46:0/2570012846 learned_addr learned my addr 172.21.15.46:0/2570012846 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:57.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.090+0000 7f99aed9d700 1 --2- 172.21.15.46:0/2570012846 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99b00b2100 0x7f99b00b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.090+0000 7f99add9b700 1 --2- 172.21.15.46:0/2570012846 >> v2:172.21.15.106:3300/0 conn(0x7f99b0154290 0x7f99b00b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.090+0000 7f99ad59a700 1 -- 172.21.15.46:0/2570012846 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053888785 0 0) 0x7f99b015b140 con 0x7f99b0153ec0 2024-01-26T23:30:57.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.090+0000 7f99ad59a700 1 -- 172.21.15.46:0/2570012846 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9998003570 con 0x7f99b0153ec0 2024-01-26T23:30:57.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.090+0000 7f99aed9d700 1 -- 172.21.15.46:0/2570012846 >> v2:172.21.15.106:3300/0 conn(0x7f99b0154290 msgr2=0x7f99b00b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:57.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.090+0000 7f99aed9d700 1 --2- 172.21.15.46:0/2570012846 >> v2:172.21.15.106:3300/0 conn(0x7f99b0154290 0x7f99b00b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.090+0000 7f99aed9d700 1 -- 172.21.15.46:0/2570012846 >> v1:172.21.15.46:6789/0 conn(0x7f99b0153ec0 legacy=0x7f99b0157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:57.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.090+0000 7f99aed9d700 1 -- 172.21.15.46:0/2570012846 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b015a8a0 con 0x7f99b00b2100 2024-01-26T23:30:57.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.091+0000 7f99aed9d700 1 --2- 172.21.15.46:0/2570012846 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99b00b2100 0x7f99b00b04f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f99a8009e60 tx=0x7f99a8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cae233cce5e840db server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:57.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.091+0000 7f99ad59a700 1 -- 172.21.15.46:0/2570012846 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a800e070 con 0x7f99b00b2100 2024-01-26T23:30:57.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.091+0000 7f99ad59a700 1 -- 172.21.15.46:0/2570012846 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a800bab0 con 0x7f99b00b2100 2024-01-26T23:30:57.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.091+0000 7f99ad59a700 1 -- 172.21.15.46:0/2570012846 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a800be70 con 0x7f99b00b2100 2024-01-26T23:30:57.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.091+0000 7f99b4f2b700 1 -- 172.21.15.46:0/2570012846 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99b00b2100 msgr2=0x7f99b00b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:57.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.091+0000 7f99b4f2b700 1 --2- 172.21.15.46:0/2570012846 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99b00b2100 0x7f99b00b04f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f99a8009e60 tx=0x7f99a8009780 comp rx=0 tx=0).stop 2024-01-26T23:30:57.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.091+0000 7f99b4f2b700 1 -- 172.21.15.46:0/2570012846 shutdown_connections 2024-01-26T23:30:57.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.091+0000 7f99b4f2b700 1 --2- 172.21.15.46:0/2570012846 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99b00b2100 0x7f99b00b04f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.091+0000 7f99b4f2b700 1 --2- 172.21.15.46:0/2570012846 >> v2:172.21.15.106:3300/0 conn(0x7f99b0154290 0x7f99b00b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.091+0000 7f99b4f2b700 1 -- 172.21.15.46:0/2570012846 >> 172.21.15.46:0/2570012846 conn(0x7f99b0069510 msgr2=0x7f99b00aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.092+0000 7f99b4f2b700 1 -- 172.21.15.46:0/2570012846 shutdown_connections 2024-01-26T23:30:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.092+0000 7f99b4f2b700 1 -- 172.21.15.46:0/2570012846 wait complete. 2024-01-26T23:30:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.093+0000 7f99b4f2b700 1 Processor -- start 2024-01-26T23:30:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.093+0000 7f99b4f2b700 1 -- start start 2024-01-26T23:30:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99b4f2b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b00b2100 0x7f99b0144e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99b4f2b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99b0153ec0 0x7f99b00a8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99b4f2b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99b0154290 0x7f99b00a86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99b4f2b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b00ab300 con 0x7f99b00b2100 2024-01-26T23:30:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99b4f2b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b00ab5e0 con 0x7f99b0154290 2024-01-26T23:30:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99b4f2b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b00ab8c0 con 0x7f99b0153ec0 2024-01-26T23:30:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99ae59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b00b2100 0x7f99b0144e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99ae59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b00b2100 0x7f99b0144e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60044/0 (socket says 172.21.15.46:60044) 2024-01-26T23:30:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99ae59c700 1 -- 172.21.15.46:0/1028440626 learned_addr learned my addr 172.21.15.46:0/1028440626 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99aed9d700 1 --2- 172.21.15.46:0/1028440626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99b0154290 0x7f99b00a86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99add9b700 1 --2- 172.21.15.46:0/1028440626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99b0153ec0 0x7f99b00a8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99ae59c700 1 -- 172.21.15.46:0/1028440626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99b0153ec0 msgr2=0x7f99b00a8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99ae59c700 1 --2- 172.21.15.46:0/1028440626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99b0153ec0 0x7f99b00a8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99ae59c700 1 -- 172.21.15.46:0/1028440626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99b0154290 msgr2=0x7f99b00a86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99ae59c700 1 --2- 172.21.15.46:0/1028440626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99b0154290 0x7f99b00a86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99ae59c700 1 -- 172.21.15.46:0/1028440626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a8009a20 con 0x7f99b00b2100 2024-01-26T23:30:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99ae59c700 1 --2- 172.21.15.46:0/1028440626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b00b2100 0x7f99b0144e70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f99a000c590 tx=0x7f99a0007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f999e7fc700 1 -- 172.21.15.46:0/1028440626 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a0005b30 con 0x7f99b00b2100 2024-01-26T23:30:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f999e7fc700 1 -- 172.21.15.46:0/1028440626 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a0005cb0 con 0x7f99b00b2100 2024-01-26T23:30:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f99b4f2b700 1 -- 172.21.15.46:0/1028440626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b00abba0 con 0x7f99b00b2100 2024-01-26T23:30:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.094+0000 7f999e7fc700 1 -- 172.21.15.46:0/1028440626 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a0007320 con 0x7f99b00b2100 2024-01-26T23:30:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.095+0000 7f99b4f2b700 1 -- 172.21.15.46:0/1028440626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b01453b0 con 0x7f99b00b2100 2024-01-26T23:30:57.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.096+0000 7f99b4f2b700 1 -- 172.21.15.46:0/1028440626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b00ac0c0 con 0x7f99b00b2100 2024-01-26T23:30:57.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.096+0000 7f999e7fc700 1 -- 172.21.15.46:0/1028440626 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f99a001b3b0 con 0x7f99b00b2100 2024-01-26T23:30:57.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.097+0000 7f999e7fc700 1 --2- 172.21.15.46:0/1028440626 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9990066f30 0x7f99900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.097+0000 7f999e7fc700 1 -- 172.21.15.46:0/1028440626 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f99a0083fe0 con 0x7f99b00b2100 2024-01-26T23:30:57.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.097+0000 7f99add9b700 1 --2- 172.21.15.46:0/1028440626 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9990066f30 0x7f99900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.098+0000 7f99add9b700 1 --2- 172.21.15.46:0/1028440626 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9990066f30 0x7f99900693e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f99b015a110 tx=0x7f99a4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:57.101 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-01-26T23:30:57.099+0000 7fcb503f6700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-26T23:30:57.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.101+0000 7f999e7fc700 1 -- 172.21.15.46:0/1028440626 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f99a0051070 con 0x7f99b00b2100 2024-01-26T23:30:57.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.301+0000 7f99b4f2b700 1 -- 172.21.15.46:0/1028440626 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f99b00ab750 con 0x7f9990066f30 2024-01-26T23:30:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.310+0000 7f999e7fc700 1 -- 172.21.15.46:0/1028440626 <== mgr.4102 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289349 (secure 0 0 0) 0x7f99b00ab750 con 0x7f9990066f30 2024-01-26T23:30:57.316 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:57.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.314+0000 7f99b4f2b700 1 -- 172.21.15.46:0/1028440626 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9990066f30 msgr2=0x7f99900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:57.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.314+0000 7f99b4f2b700 1 --2- 172.21.15.46:0/1028440626 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9990066f30 0x7f99900693e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f99b015a110 tx=0x7f99a4008040 comp rx=0 tx=0).stop 2024-01-26T23:30:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.314+0000 7f99b4f2b700 1 -- 172.21.15.46:0/1028440626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b00b2100 msgr2=0x7f99b0144e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.314+0000 7f99b4f2b700 1 --2- 172.21.15.46:0/1028440626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b00b2100 0x7f99b0144e70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f99a000c590 tx=0x7f99a0007e00 comp rx=0 tx=0).stop 2024-01-26T23:30:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.314+0000 7f99b4f2b700 1 -- 172.21.15.46:0/1028440626 shutdown_connections 2024-01-26T23:30:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.314+0000 7f99b4f2b700 1 --2- 172.21.15.46:0/1028440626 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9990066f30 0x7f99900693e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.314+0000 7f99b4f2b700 1 --2- 172.21.15.46:0/1028440626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b00b2100 0x7f99b0144e70 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.314+0000 7f99b4f2b700 1 --2- 172.21.15.46:0/1028440626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99b0154290 0x7f99b00a86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.314+0000 7f99b4f2b700 1 --2- 172.21.15.46:0/1028440626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99b0153ec0 0x7f99b00a8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.314+0000 7f99b4f2b700 1 -- 172.21.15.46:0/1028440626 >> 172.21.15.46:0/1028440626 conn(0x7f99b0069510 msgr2=0x7f99b0154820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.314+0000 7f99b4f2b700 1 -- 172.21.15.46:0/1028440626 shutdown_connections 2024-01-26T23:30:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.314+0000 7f99b4f2b700 1 -- 172.21.15.46:0/1028440626 wait complete. 2024-01-26T23:30:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:30:57.330 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2024-01-26T23:30:56.242553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2382724,"kb_used_data":4420,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752591996,"statfs":{"total":773094113280,"available":770654203904,"internally_reserved":0,"allocated":4526080,"data_stored":2589624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.129728"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.148930+0000","last_change":"2024-01-26T23:30:53.148930+0000","last_active":"2024-01-26T23:30:53.148930+0000","last_peered":"2024-01-26T23:30:53.148930+0000","last_clean":"2024-01-26T23:30:53.148930+0000","last_became_active":"2024-01-26T23:30:53.147918+0000","last_became_peered":"2024-01-26T23:30:53.147918+0000","last_unstale":"2024-01-26T23:30:53.148930+0000","last_undegraded":"2024-01-26T23:30:53.148930+0000","last_fullsized":"2024-01-26T23:30:53.148930+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:50.289763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.135001+0000","last_change":"2024-01-26T23:30:53.135001+0000","last_active":"2024-01-26T23:30:53.135001+0000","last_peered":"2024-01-26T23:30:53.135001+0000","last_clean":"2024-01-26T23:30:53.135001+0000","last_became_active":"2024-01-26T23:30:53.134680+0000","last_became_peered":"2024-01-26T23:30:53.134680+0000","last_unstale":"2024-01-26T23:30:53.135001+0000","last_undegraded":"2024-01-26T23:30:53.135001+0000","last_fullsized":"2024-01-26T23:30:53.135001+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:15:26.285891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.153204+0000","last_change":"2024-01-26T23:30:53.153204+0000","last_active":"2024-01-26T23:30:53.153204+0000","last_peered":"2024-01-26T23:30:53.153204+0000","last_clean":"2024-01-26T23:30:53.153204+0000","last_became_active":"2024-01-26T23:30:53.152966+0000","last_became_peered":"2024-01-26T23:30:53.152966+0000","last_unstale":"2024-01-26T23:30:53.153204+0000","last_undegraded":"2024-01-26T23:30:53.153204+0000","last_fullsized":"2024-01-26T23:30:53.153204+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:15:26.285891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150803+0000","last_change":"2024-01-26T23:30:53.150803+0000","last_active":"2024-01-26T23:30:53.150803+0000","last_peered":"2024-01-26T23:30:53.150803+0000","last_clean":"2024-01-26T23:30:53.150803+0000","last_became_active":"2024-01-26T23:30:53.150342+0000","last_became_peered":"2024-01-26T23:30:53.150342+0000","last_unstale":"2024-01-26T23:30:53.150803+0000","last_undegraded":"2024-01-26T23:30:53.150803+0000","last_fullsized":"2024-01-26T23:30:53.150803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:52:04.566977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150402+0000","last_change":"2024-01-26T23:30:53.150402+0000","last_active":"2024-01-26T23:30:53.150402+0000","last_peered":"2024-01-26T23:30:53.150402+0000","last_clean":"2024-01-26T23:30:53.150402+0000","last_became_active":"2024-01-26T23:30:53.149876+0000","last_became_peered":"2024-01-26T23:30:53.149876+0000","last_unstale":"2024-01-26T23:30:53.150402+0000","last_undegraded":"2024-01-26T23:30:53.150402+0000","last_fullsized":"2024-01-26T23:30:53.150402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:48:21.769810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.146114+0000","last_change":"2024-01-26T23:30:53.146114+0000","last_active":"2024-01-26T23:30:53.146114+0000","last_peered":"2024-01-26T23:30:53.146114+0000","last_clean":"2024-01-26T23:30:53.146114+0000","last_became_active":"2024-01-26T23:30:53.145832+0000","last_became_peered":"2024-01-26T23:30:53.145832+0000","last_unstale":"2024-01-26T23:30:53.146114+0000","last_undegraded":"2024-01-26T23:30:53.146114+0000","last_fullsized":"2024-01-26T23:30:53.146114+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:05:11.628582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.149995+0000","last_change":"2024-01-26T23:30:53.149995+0000","last_active":"2024-01-26T23:30:53.149995+0000","last_peered":"2024-01-26T23:30:53.149995+0000","last_clean":"2024-01-26T23:30:53.149995+0000","last_became_active":"2024-01-26T23:30:53.149722+0000","last_became_peered":"2024-01-26T23:30:53.149722+0000","last_unstale":"2024-01-26T23:30:53.149995+0000","last_undegraded":"2024-01-26T23:30:53.149995+0000","last_fullsized":"2024-01-26T23:30:53.149995+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:53:34.849263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.155843+0000","last_change":"2024-01-26T23:30:53.155843+0000","last_active":"2024-01-26T23:30:53.155843+0000","last_peered":"2024-01-26T23:30:53.155843+0000","last_clean":"2024-01-26T23:30:53.155843+0000","last_became_active":"2024-01-26T23:30:53.155361+0000","last_became_peered":"2024-01-26T23:30:53.155361+0000","last_unstale":"2024-01-26T23:30:53.155843+0000","last_undegraded":"2024-01-26T23:30:53.155843+0000","last_fullsized":"2024-01-26T23:30:53.155843+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:00:27.950751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.152696+0000","last_change":"2024-01-26T23:30:53.152696+0000","last_active":"2024-01-26T23:30:53.152696+0000","last_peered":"2024-01-26T23:30:53.152696+0000","last_clean":"2024-01-26T23:30:53.152696+0000","last_became_active":"2024-01-26T23:30:53.152375+0000","last_became_peered":"2024-01-26T23:30:53.152375+0000","last_unstale":"2024-01-26T23:30:53.152696+0000","last_undegraded":"2024-01-26T23:30:53.152696+0000","last_fullsized":"2024-01-26T23:30:53.152696+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:50.289763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.153233+0000","last_change":"2024-01-26T23:30:53.153233+0000","last_active":"2024-01-26T23:30:53.153233+0000","last_peered":"2024-01-26T23:30:53.153233+0000","last_clean":"2024-01-26T23:30:53.153233+0000","last_became_active":"2024-01-26T23:30:53.152926+0000","last_became_peered":"2024-01-26T23:30:53.152926+0000","last_unstale":"2024-01-26T23:30:53.153233+0000","last_undegraded":"2024-01-26T23:30:53.153233+0000","last_fullsized":"2024-01-26T23:30:53.153233+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:35:06.371452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150792+0000","last_change":"2024-01-26T23:30:53.150792+0000","last_active":"2024-01-26T23:30:53.150792+0000","last_peered":"2024-01-26T23:30:53.150792+0000","last_clean":"2024-01-26T23:30:53.150792+0000","last_became_active":"2024-01-26T23:30:53.150192+0000","last_became_peered":"2024-01-26T23:30:53.150192+0000","last_unstale":"2024-01-26T23:30:53.150792+0000","last_undegraded":"2024-01-26T23:30:53.150792+0000","last_fullsized":"2024-01-26T23:30:53.150792+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:21:28.076604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.136695+0000","last_change":"2024-01-26T23:30:53.136695+0000","last_active":"2024-01-26T23:30:53.136695+0000","last_peered":"2024-01-26T23:30:53.136695+0000","last_clean":"2024-01-26T23:30:53.136695+0000","last_became_active":"2024-01-26T23:30:53.136370+0000","last_became_peered":"2024-01-26T23:30:53.136370+0000","last_unstale":"2024-01-26T23:30:53.136695+0000","last_undegraded":"2024-01-26T23:30:53.136695+0000","last_fullsized":"2024-01-26T23:30:53.136695+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:42:40.534420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150290+0000","last_change":"2024-01-26T23:30:53.150290+0000","last_active":"2024-01-26T23:30:53.150290+0000","last_peered":"2024-01-26T23:30:53.150290+0000","last_clean":"2024-01-26T23:30:53.150290+0000","last_became_active":"2024-01-26T23:30:53.149997+0000","last_became_peered":"2024-01-26T23:30:53.149997+0000","last_unstale":"2024-01-26T23:30:53.150290+0000","last_undegraded":"2024-01-26T23:30:53.150290+0000","last_fullsized":"2024-01-26T23:30:53.150290+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:00:27.950751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.149648+0000","last_change":"2024-01-26T23:30:53.149648+0000","last_active":"2024-01-26T23:30:53.149648+0000","last_peered":"2024-01-26T23:30:53.149648+0000","last_clean":"2024-01-26T23:30:53.149648+0000","last_became_active":"2024-01-26T23:30:53.149300+0000","last_became_peered":"2024-01-26T23:30:53.149300+0000","last_unstale":"2024-01-26T23:30:53.149648+0000","last_undegraded":"2024-01-26T23:30:53.149648+0000","last_fullsized":"2024-01-26T23:30:53.149648+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:54:59.331467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150897+0000","last_change":"2024-01-26T23:30:53.150897+0000","last_active":"2024-01-26T23:30:53.150897+0000","last_peered":"2024-01-26T23:30:53.150897+0000","last_clean":"2024-01-26T23:30:53.150897+0000","last_became_active":"2024-01-26T23:30:53.150634+0000","last_became_peered":"2024-01-26T23:30:53.150634+0000","last_unstale":"2024-01-26T23:30:53.150897+0000","last_undegraded":"2024-01-26T23:30:53.150897+0000","last_fullsized":"2024-01-26T23:30:53.150897+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:54:59.331467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.154888+0000","last_change":"2024-01-26T23:30:53.154888+0000","last_active":"2024-01-26T23:30:53.154888+0000","last_peered":"2024-01-26T23:30:53.154888+0000","last_clean":"2024-01-26T23:30:53.154888+0000","last_became_active":"2024-01-26T23:30:53.154417+0000","last_became_peered":"2024-01-26T23:30:53.154417+0000","last_unstale":"2024-01-26T23:30:53.154888+0000","last_undegraded":"2024-01-26T23:30:53.154888+0000","last_fullsized":"2024-01-26T23:30:53.154888+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:46:41.384689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.154208+0000","last_change":"2024-01-26T23:30:53.154208+0000","last_active":"2024-01-26T23:30:53.154208+0000","last_peered":"2024-01-26T23:30:53.154208+0000","last_clean":"2024-01-26T23:30:53.154208+0000","last_became_active":"2024-01-26T23:30:53.153879+0000","last_became_peered":"2024-01-26T23:30:53.153879+0000","last_unstale":"2024-01-26T23:30:53.154208+0000","last_undegraded":"2024-01-26T23:30:53.154208+0000","last_fullsized":"2024-01-26T23:30:53.154208+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:13:09.311848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.149050+0000","last_change":"2024-01-26T23:30:53.149050+0000","last_active":"2024-01-26T23:30:53.149050+0000","last_peered":"2024-01-26T23:30:53.149050+0000","last_clean":"2024-01-26T23:30:53.149050+0000","last_became_active":"2024-01-26T23:30:53.148793+0000","last_became_peered":"2024-01-26T23:30:53.148793+0000","last_unstale":"2024-01-26T23:30:53.149050+0000","last_undegraded":"2024-01-26T23:30:53.149050+0000","last_fullsized":"2024-01-26T23:30:53.149050+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:40:59.362637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.142059+0000","last_change":"2024-01-26T23:30:53.142059+0000","last_active":"2024-01-26T23:30:53.142059+0000","last_peered":"2024-01-26T23:30:53.142059+0000","last_clean":"2024-01-26T23:30:53.142059+0000","last_became_active":"2024-01-26T23:30:53.141752+0000","last_became_peered":"2024-01-26T23:30:53.141752+0000","last_unstale":"2024-01-26T23:30:53.142059+0000","last_undegraded":"2024-01-26T23:30:53.142059+0000","last_fullsized":"2024-01-26T23:30:53.142059+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:31:05.339820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.149345+0000","last_change":"2024-01-26T23:30:53.149345+0000","last_active":"2024-01-26T23:30:53.149345+0000","last_peered":"2024-01-26T23:30:53.149345+0000","last_clean":"2024-01-26T23:30:53.149345+0000","last_became_active":"2024-01-26T23:30:53.149079+0000","last_became_peered":"2024-01-26T23:30:53.149079+0000","last_unstale":"2024-01-26T23:30:53.149345+0000","last_undegraded":"2024-01-26T23:30:53.149345+0000","last_fullsized":"2024-01-26T23:30:53.149345+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:28.464237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.149770+0000","last_change":"2024-01-26T23:30:53.149770+0000","last_active":"2024-01-26T23:30:53.149770+0000","last_peered":"2024-01-26T23:30:53.149770+0000","last_clean":"2024-01-26T23:30:53.149770+0000","last_became_active":"2024-01-26T23:30:53.149458+0000","last_became_peered":"2024-01-26T23:30:53.149458+0000","last_unstale":"2024-01-26T23:30:53.149770+0000","last_undegraded":"2024-01-26T23:30:53.149770+0000","last_fullsized":"2024-01-26T23:30:53.149770+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:19:47.941469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.138848+0000","last_change":"2024-01-26T23:30:53.138848+0000","last_active":"2024-01-26T23:30:53.138848+0000","last_peered":"2024-01-26T23:30:53.138848+0000","last_clean":"2024-01-26T23:30:53.138848+0000","last_became_active":"2024-01-26T23:30:53.138555+0000","last_became_peered":"2024-01-26T23:30:53.138555+0000","last_unstale":"2024-01-26T23:30:53.138848+0000","last_undegraded":"2024-01-26T23:30:53.138848+0000","last_fullsized":"2024-01-26T23:30:53.138848+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:00:27.950751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.151596+0000","last_change":"2024-01-26T23:30:53.151596+0000","last_active":"2024-01-26T23:30:53.151596+0000","last_peered":"2024-01-26T23:30:53.151596+0000","last_clean":"2024-01-26T23:30:53.151596+0000","last_became_active":"2024-01-26T23:30:53.151183+0000","last_became_peered":"2024-01-26T23:30:53.151183+0000","last_unstale":"2024-01-26T23:30:53.151596+0000","last_undegraded":"2024-01-26T23:30:53.151596+0000","last_fullsized":"2024-01-26T23:30:53.151596+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:14.479539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.137323+0000","last_change":"2024-01-26T23:30:53.137323+0000","last_active":"2024-01-26T23:30:53.137323+0000","last_peered":"2024-01-26T23:30:53.137323+0000","last_clean":"2024-01-26T23:30:53.137323+0000","last_became_active":"2024-01-26T23:30:53.137046+0000","last_became_peered":"2024-01-26T23:30:53.137046+0000","last_unstale":"2024-01-26T23:30:53.137323+0000","last_undegraded":"2024-01-26T23:30:53.137323+0000","last_fullsized":"2024-01-26T23:30:53.137323+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:34:35.003101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.144330+0000","last_change":"2024-01-26T23:30:53.144330+0000","last_active":"2024-01-26T23:30:53.144330+0000","last_peered":"2024-01-26T23:30:53.144330+0000","last_clean":"2024-01-26T23:30:53.144330+0000","last_became_active":"2024-01-26T23:30:53.143985+0000","last_became_peered":"2024-01-26T23:30:53.143985+0000","last_unstale":"2024-01-26T23:30:53.144330+0000","last_undegraded":"2024-01-26T23:30:53.144330+0000","last_fullsized":"2024-01-26T23:30:53.144330+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:01:39.008256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.155012+0000","last_change":"2024-01-26T23:30:53.155012+0000","last_active":"2024-01-26T23:30:53.155012+0000","last_peered":"2024-01-26T23:30:53.155012+0000","last_clean":"2024-01-26T23:30:53.155012+0000","last_became_active":"2024-01-26T23:30:53.154728+0000","last_became_peered":"2024-01-26T23:30:53.154728+0000","last_unstale":"2024-01-26T23:30:53.155012+0000","last_undegraded":"2024-01-26T23:30:53.155012+0000","last_fullsized":"2024-01-26T23:30:53.155012+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:06:00.135263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.145200+0000","last_change":"2024-01-26T23:30:53.145200+0000","last_active":"2024-01-26T23:30:53.145200+0000","last_peered":"2024-01-26T23:30:53.145200+0000","last_clean":"2024-01-26T23:30:53.145200+0000","last_became_active":"2024-01-26T23:30:53.144929+0000","last_became_peered":"2024-01-26T23:30:53.144929+0000","last_unstale":"2024-01-26T23:30:53.145200+0000","last_undegraded":"2024-01-26T23:30:53.145200+0000","last_fullsized":"2024-01-26T23:30:53.145200+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:41:52.600539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.148496+0000","last_change":"2024-01-26T23:30:53.148496+0000","last_active":"2024-01-26T23:30:53.148496+0000","last_peered":"2024-01-26T23:30:53.148496+0000","last_clean":"2024-01-26T23:30:53.148496+0000","last_became_active":"2024-01-26T23:30:53.148178+0000","last_became_peered":"2024-01-26T23:30:53.148178+0000","last_unstale":"2024-01-26T23:30:53.148496+0000","last_undegraded":"2024-01-26T23:30:53.148496+0000","last_fullsized":"2024-01-26T23:30:53.148496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:06:09.705120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.144635+0000","last_change":"2024-01-26T23:30:53.144635+0000","last_active":"2024-01-26T23:30:53.144635+0000","last_peered":"2024-01-26T23:30:53.144635+0000","last_clean":"2024-01-26T23:30:53.144635+0000","last_became_active":"2024-01-26T23:30:53.144369+0000","last_became_peered":"2024-01-26T23:30:53.144369+0000","last_unstale":"2024-01-26T23:30:53.144635+0000","last_undegraded":"2024-01-26T23:30:53.144635+0000","last_fullsized":"2024-01-26T23:30:53.144635+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:59.531509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.144551+0000","last_change":"2024-01-26T23:30:53.144551+0000","last_active":"2024-01-26T23:30:53.144551+0000","last_peered":"2024-01-26T23:30:53.144551+0000","last_clean":"2024-01-26T23:30:53.144551+0000","last_became_active":"2024-01-26T23:30:53.144137+0000","last_became_peered":"2024-01-26T23:30:53.144137+0000","last_unstale":"2024-01-26T23:30:53.144551+0000","last_undegraded":"2024-01-26T23:30:53.144551+0000","last_fullsized":"2024-01-26T23:30:53.144551+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:42:21.954480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.151420+0000","last_change":"2024-01-26T23:30:53.151420+0000","last_active":"2024-01-26T23:30:53.151420+0000","last_peered":"2024-01-26T23:30:53.151420+0000","last_clean":"2024-01-26T23:30:53.151420+0000","last_became_active":"2024-01-26T23:30:53.150809+0000","last_became_peered":"2024-01-26T23:30:53.150809+0000","last_unstale":"2024-01-26T23:30:53.151420+0000","last_undegraded":"2024-01-26T23:30:53.151420+0000","last_fullsized":"2024-01-26T23:30:53.151420+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:06:00.135263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150644+0000","last_change":"2024-01-26T23:30:53.150644+0000","last_active":"2024-01-26T23:30:53.150644+0000","last_peered":"2024-01-26T23:30:53.150644+0000","last_clean":"2024-01-26T23:30:53.150644+0000","last_became_active":"2024-01-26T23:30:53.150264+0000","last_became_peered":"2024-01-26T23:30:53.150264+0000","last_unstale":"2024-01-26T23:30:53.150644+0000","last_undegraded":"2024-01-26T23:30:53.150644+0000","last_fullsized":"2024-01-26T23:30:53.150644+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:35:06.371452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.155634+0000","last_change":"2024-01-26T23:30:53.155634+0000","last_active":"2024-01-26T23:30:53.155634+0000","last_peered":"2024-01-26T23:30:53.155634+0000","last_clean":"2024-01-26T23:30:53.155634+0000","last_became_active":"2024-01-26T23:30:53.155141+0000","last_became_peered":"2024-01-26T23:30:53.155141+0000","last_unstale":"2024-01-26T23:30:53.155634+0000","last_undegraded":"2024-01-26T23:30:53.155634+0000","last_fullsized":"2024-01-26T23:30:53.155634+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:41:37.013476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.163776+0000","last_change":"2024-01-26T23:30:52.135994+0000","last_active":"2024-01-26T23:30:53.163776+0000","last_peered":"2024-01-26T23:30:53.163776+0000","last_clean":"2024-01-26T23:30:53.163776+0000","last_became_active":"2024-01-26T23:30:52.135709+0000","last_became_peered":"2024-01-26T23:30:52.135709+0000","last_unstale":"2024-01-26T23:30:53.163776+0000","last_undegraded":"2024-01-26T23:30:53.163776+0000","last_fullsized":"2024-01-26T23:30:53.163776+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:31:04.332740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.158297+0000","last_change":"2024-01-26T23:30:53.158297+0000","last_active":"2024-01-26T23:30:53.158297+0000","last_peered":"2024-01-26T23:30:53.158297+0000","last_clean":"2024-01-26T23:30:53.158297+0000","last_became_active":"2024-01-26T23:30:53.158014+0000","last_became_peered":"2024-01-26T23:30:53.158014+0000","last_unstale":"2024-01-26T23:30:53.158297+0000","last_undegraded":"2024-01-26T23:30:53.158297+0000","last_fullsized":"2024-01-26T23:30:53.158297+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:35:46.571656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.129214+0000","last_change":"2024-01-26T23:30:52.133411+0000","last_active":"2024-01-26T23:30:53.129214+0000","last_peered":"2024-01-26T23:30:53.129214+0000","last_clean":"2024-01-26T23:30:53.129214+0000","last_became_active":"2024-01-26T23:30:52.132169+0000","last_became_peered":"2024-01-26T23:30:52.132169+0000","last_unstale":"2024-01-26T23:30:53.129214+0000","last_undegraded":"2024-01-26T23:30:53.129214+0000","last_fullsized":"2024-01-26T23:30:53.129214+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:05:59.128183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.149529+0000","last_change":"2024-01-26T23:30:53.149529+0000","last_active":"2024-01-26T23:30:53.149529+0000","last_peered":"2024-01-26T23:30:53.149529+0000","last_clean":"2024-01-26T23:30:53.149529+0000","last_became_active":"2024-01-26T23:30:53.149235+0000","last_became_peered":"2024-01-26T23:30:53.149235+0000","last_unstale":"2024-01-26T23:30:53.149529+0000","last_undegraded":"2024-01-26T23:30:53.149529+0000","last_fullsized":"2024-01-26T23:30:53.149529+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:50.289763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.163961+0000","last_change":"2024-01-26T23:30:52.139695+0000","last_active":"2024-01-26T23:30:53.163961+0000","last_peered":"2024-01-26T23:30:53.163961+0000","last_clean":"2024-01-26T23:30:53.163961+0000","last_became_active":"2024-01-26T23:30:52.139478+0000","last_became_peered":"2024-01-26T23:30:52.139478+0000","last_unstale":"2024-01-26T23:30:53.163961+0000","last_undegraded":"2024-01-26T23:30:53.163961+0000","last_fullsized":"2024-01-26T23:30:53.163961+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:01:38.001177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.148939+0000","last_change":"2024-01-26T23:30:53.148939+0000","last_active":"2024-01-26T23:30:53.148939+0000","last_peered":"2024-01-26T23:30:53.148939+0000","last_clean":"2024-01-26T23:30:53.148939+0000","last_became_active":"2024-01-26T23:30:53.147452+0000","last_became_peered":"2024-01-26T23:30:53.147452+0000","last_unstale":"2024-01-26T23:30:53.148939+0000","last_undegraded":"2024-01-26T23:30:53.148939+0000","last_fullsized":"2024-01-26T23:30:53.148939+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:19:47.941469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.163553+0000","last_change":"2024-01-26T23:30:52.140374+0000","last_active":"2024-01-26T23:30:53.163553+0000","last_peered":"2024-01-26T23:30:53.163553+0000","last_clean":"2024-01-26T23:30:53.163553+0000","last_became_active":"2024-01-26T23:30:52.139872+0000","last_became_peered":"2024-01-26T23:30:52.139872+0000","last_unstale":"2024-01-26T23:30:53.163553+0000","last_undegraded":"2024-01-26T23:30:53.163553+0000","last_fullsized":"2024-01-26T23:30:53.163553+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:49.828311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.140574+0000","last_change":"2024-01-26T23:30:53.140574+0000","last_active":"2024-01-26T23:30:53.140574+0000","last_peered":"2024-01-26T23:30:53.140574+0000","last_clean":"2024-01-26T23:30:53.140574+0000","last_became_active":"2024-01-26T23:30:53.140236+0000","last_became_peered":"2024-01-26T23:30:53.140236+0000","last_unstale":"2024-01-26T23:30:53.140574+0000","last_undegraded":"2024-01-26T23:30:53.140574+0000","last_fullsized":"2024-01-26T23:30:53.140574+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:50.781140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.143641+0000","last_change":"2024-01-26T23:30:52.133011+0000","last_active":"2024-01-26T23:30:53.143641+0000","last_peered":"2024-01-26T23:30:53.143641+0000","last_clean":"2024-01-26T23:30:53.143641+0000","last_became_active":"2024-01-26T23:30:52.132774+0000","last_became_peered":"2024-01-26T23:30:52.132774+0000","last_unstale":"2024-01-26T23:30:53.143641+0000","last_undegraded":"2024-01-26T23:30:53.143641+0000","last_fullsized":"2024-01-26T23:30:53.143641+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:52:45.442327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.153343+0000","last_change":"2024-01-26T23:30:53.153343+0000","last_active":"2024-01-26T23:30:53.153343+0000","last_peered":"2024-01-26T23:30:53.153343+0000","last_clean":"2024-01-26T23:30:53.153343+0000","last_became_active":"2024-01-26T23:30:53.153079+0000","last_became_peered":"2024-01-26T23:30:53.153079+0000","last_unstale":"2024-01-26T23:30:53.153343+0000","last_undegraded":"2024-01-26T23:30:53.153343+0000","last_fullsized":"2024-01-26T23:30:53.153343+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:47:22.375886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.131359+0000","last_change":"2024-01-26T23:30:52.136591+0000","last_active":"2024-01-26T23:30:53.131359+0000","last_peered":"2024-01-26T23:30:53.131359+0000","last_clean":"2024-01-26T23:30:53.131359+0000","last_became_active":"2024-01-26T23:30:52.136290+0000","last_became_peered":"2024-01-26T23:30:52.136290+0000","last_unstale":"2024-01-26T23:30:53.131359+0000","last_undegraded":"2024-01-26T23:30:53.131359+0000","last_fullsized":"2024-01-26T23:30:53.131359+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:48:20.762731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.154657+0000","last_change":"2024-01-26T23:30:53.154657+0000","last_active":"2024-01-26T23:30:53.154657+0000","last_peered":"2024-01-26T23:30:53.154657+0000","last_clean":"2024-01-26T23:30:53.154657+0000","last_became_active":"2024-01-26T23:30:53.154321+0000","last_became_peered":"2024-01-26T23:30:53.154321+0000","last_unstale":"2024-01-26T23:30:53.154657+0000","last_undegraded":"2024-01-26T23:30:53.154657+0000","last_fullsized":"2024-01-26T23:30:53.154657+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:47:22.375886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.138623+0000","last_change":"2024-01-26T23:30:52.137622+0000","last_active":"2024-01-26T23:30:52.138623+0000","last_peered":"2024-01-26T23:30:52.138623+0000","last_clean":"2024-01-26T23:30:52.138623+0000","last_became_active":"2024-01-26T23:30:52.137276+0000","last_became_peered":"2024-01-26T23:30:52.137276+0000","last_unstale":"2024-01-26T23:30:52.138623+0000","last_undegraded":"2024-01-26T23:30:52.138623+0000","last_fullsized":"2024-01-26T23:30:52.138623+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:42:20.947400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.139415+0000","last_change":"2024-01-26T23:30:53.139415+0000","last_active":"2024-01-26T23:30:53.139415+0000","last_peered":"2024-01-26T23:30:53.139415+0000","last_clean":"2024-01-26T23:30:53.139415+0000","last_became_active":"2024-01-26T23:30:53.138522+0000","last_became_peered":"2024-01-26T23:30:53.138522+0000","last_unstale":"2024-01-26T23:30:53.139415+0000","last_undegraded":"2024-01-26T23:30:53.139415+0000","last_fullsized":"2024-01-26T23:30:53.139415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:17:39.550438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.133686+0000","last_change":"2024-01-26T23:30:52.133105+0000","last_active":"2024-01-26T23:30:52.133686+0000","last_peered":"2024-01-26T23:30:52.133686+0000","last_clean":"2024-01-26T23:30:52.133686+0000","last_became_active":"2024-01-26T23:30:52.132828+0000","last_became_peered":"2024-01-26T23:30:52.132828+0000","last_unstale":"2024-01-26T23:30:52.133686+0000","last_undegraded":"2024-01-26T23:30:52.133686+0000","last_fullsized":"2024-01-26T23:30:52.133686+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:41:36.006396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150321+0000","last_change":"2024-01-26T23:30:53.150321+0000","last_active":"2024-01-26T23:30:53.150321+0000","last_peered":"2024-01-26T23:30:53.150321+0000","last_clean":"2024-01-26T23:30:53.150321+0000","last_became_active":"2024-01-26T23:30:53.149935+0000","last_became_peered":"2024-01-26T23:30:53.149935+0000","last_unstale":"2024-01-26T23:30:53.150321+0000","last_undegraded":"2024-01-26T23:30:53.150321+0000","last_fullsized":"2024-01-26T23:30:53.150321+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:55:40.719163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.139027+0000","last_change":"2024-01-26T23:30:52.130243+0000","last_active":"2024-01-26T23:30:52.139027+0000","last_peered":"2024-01-26T23:30:52.139027+0000","last_clean":"2024-01-26T23:30:52.139027+0000","last_became_active":"2024-01-26T23:30:52.129945+0000","last_became_peered":"2024-01-26T23:30:52.129945+0000","last_unstale":"2024-01-26T23:30:52.139027+0000","last_undegraded":"2024-01-26T23:30:52.139027+0000","last_fullsized":"2024-01-26T23:30:52.139027+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:13:08.304768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.146393+0000","last_change":"2024-01-26T23:30:53.146393+0000","last_active":"2024-01-26T23:30:53.146393+0000","last_peered":"2024-01-26T23:30:53.146393+0000","last_clean":"2024-01-26T23:30:53.146393+0000","last_became_active":"2024-01-26T23:30:53.146095+0000","last_became_peered":"2024-01-26T23:30:53.146095+0000","last_unstale":"2024-01-26T23:30:53.146393+0000","last_undegraded":"2024-01-26T23:30:53.146393+0000","last_fullsized":"2024-01-26T23:30:53.146393+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:28.464237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.140804+0000","last_change":"2024-01-26T23:30:52.139866+0000","last_active":"2024-01-26T23:30:52.140804+0000","last_peered":"2024-01-26T23:30:52.140804+0000","last_clean":"2024-01-26T23:30:52.140804+0000","last_became_active":"2024-01-26T23:30:52.139621+0000","last_became_peered":"2024-01-26T23:30:52.139621+0000","last_unstale":"2024-01-26T23:30:52.140804+0000","last_undegraded":"2024-01-26T23:30:52.140804+0000","last_fullsized":"2024-01-26T23:30:52.140804+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:40:58.355558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.140424+0000","last_change":"2024-01-26T23:30:53.140424+0000","last_active":"2024-01-26T23:30:53.140424+0000","last_peered":"2024-01-26T23:30:53.140424+0000","last_clean":"2024-01-26T23:30:53.140424+0000","last_became_active":"2024-01-26T23:30:53.140116+0000","last_became_peered":"2024-01-26T23:30:53.140116+0000","last_unstale":"2024-01-26T23:30:53.140424+0000","last_undegraded":"2024-01-26T23:30:53.140424+0000","last_fullsized":"2024-01-26T23:30:53.140424+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:19:47.941469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.163238+0000","last_change":"2024-01-26T23:30:52.130204+0000","last_active":"2024-01-26T23:30:53.163238+0000","last_peered":"2024-01-26T23:30:53.163238+0000","last_clean":"2024-01-26T23:30:53.163238+0000","last_became_active":"2024-01-26T23:30:52.129898+0000","last_became_peered":"2024-01-26T23:30:52.129898+0000","last_unstale":"2024-01-26T23:30:53.163238+0000","last_undegraded":"2024-01-26T23:30:53.163238+0000","last_fullsized":"2024-01-26T23:30:53.163238+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:58.524429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.137432+0000","last_change":"2024-01-26T23:30:53.137432+0000","last_active":"2024-01-26T23:30:53.137432+0000","last_peered":"2024-01-26T23:30:53.137432+0000","last_clean":"2024-01-26T23:30:53.137432+0000","last_became_active":"2024-01-26T23:30:53.137111+0000","last_became_peered":"2024-01-26T23:30:53.137111+0000","last_unstale":"2024-01-26T23:30:53.137432+0000","last_undegraded":"2024-01-26T23:30:53.137432+0000","last_fullsized":"2024-01-26T23:30:53.137432+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:19:47.941469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.165407+0000","last_change":"2024-01-26T23:30:52.135583+0000","last_active":"2024-01-26T23:30:53.165407+0000","last_peered":"2024-01-26T23:30:53.165407+0000","last_clean":"2024-01-26T23:30:53.165407+0000","last_became_active":"2024-01-26T23:30:52.135252+0000","last_became_peered":"2024-01-26T23:30:52.135252+0000","last_unstale":"2024-01-26T23:30:53.165407+0000","last_undegraded":"2024-01-26T23:30:53.165407+0000","last_fullsized":"2024-01-26T23:30:53.165407+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:46:40.377610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.152729+0000","last_change":"2024-01-26T23:30:53.152729+0000","last_active":"2024-01-26T23:30:53.152729+0000","last_peered":"2024-01-26T23:30:53.152729+0000","last_clean":"2024-01-26T23:30:53.152729+0000","last_became_active":"2024-01-26T23:30:53.152486+0000","last_became_peered":"2024-01-26T23:30:53.152486+0000","last_unstale":"2024-01-26T23:30:53.152729+0000","last_undegraded":"2024-01-26T23:30:53.152729+0000","last_fullsized":"2024-01-26T23:30:53.152729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:53:34.849263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.164536+0000","last_change":"2024-01-26T23:30:52.134887+0000","last_active":"2024-01-26T23:30:53.164536+0000","last_peered":"2024-01-26T23:30:53.164536+0000","last_clean":"2024-01-26T23:30:53.164536+0000","last_became_active":"2024-01-26T23:30:52.134395+0000","last_became_peered":"2024-01-26T23:30:52.134395+0000","last_unstale":"2024-01-26T23:30:53.164536+0000","last_undegraded":"2024-01-26T23:30:53.164536+0000","last_fullsized":"2024-01-26T23:30:53.164536+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:41:36.006396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.147976+0000","last_change":"2024-01-26T23:30:53.147976+0000","last_active":"2024-01-26T23:30:53.147976+0000","last_peered":"2024-01-26T23:30:53.147976+0000","last_clean":"2024-01-26T23:30:53.147976+0000","last_became_active":"2024-01-26T23:30:53.147639+0000","last_became_peered":"2024-01-26T23:30:53.147639+0000","last_unstale":"2024-01-26T23:30:53.147976+0000","last_undegraded":"2024-01-26T23:30:53.147976+0000","last_fullsized":"2024-01-26T23:30:53.147976+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:53:34.849263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.138584+0000","last_change":"2024-01-26T23:30:52.137142+0000","last_active":"2024-01-26T23:30:52.138584+0000","last_peered":"2024-01-26T23:30:52.138584+0000","last_clean":"2024-01-26T23:30:52.138584+0000","last_became_active":"2024-01-26T23:30:52.136803+0000","last_became_peered":"2024-01-26T23:30:52.136803+0000","last_unstale":"2024-01-26T23:30:52.138584+0000","last_undegraded":"2024-01-26T23:30:52.138584+0000","last_fullsized":"2024-01-26T23:30:52.138584+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:58.524429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.149287+0000","last_change":"2024-01-26T23:30:53.149287+0000","last_active":"2024-01-26T23:30:53.149287+0000","last_peered":"2024-01-26T23:30:53.149287+0000","last_clean":"2024-01-26T23:30:53.149287+0000","last_became_active":"2024-01-26T23:30:53.149018+0000","last_became_peered":"2024-01-26T23:30:53.149018+0000","last_unstale":"2024-01-26T23:30:53.149287+0000","last_undegraded":"2024-01-26T23:30:53.149287+0000","last_fullsized":"2024-01-26T23:30:53.149287+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:35:46.571656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.139804+0000","last_change":"2024-01-26T23:30:52.136062+0000","last_active":"2024-01-26T23:30:52.139804+0000","last_peered":"2024-01-26T23:30:52.139804+0000","last_clean":"2024-01-26T23:30:52.139804+0000","last_became_active":"2024-01-26T23:30:52.135799+0000","last_became_peered":"2024-01-26T23:30:52.135799+0000","last_unstale":"2024-01-26T23:30:52.139804+0000","last_undegraded":"2024-01-26T23:30:52.139804+0000","last_fullsized":"2024-01-26T23:30:52.139804+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:16:19.376392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.148268+0000","last_change":"2024-01-26T23:30:53.148268+0000","last_active":"2024-01-26T23:30:53.148268+0000","last_peered":"2024-01-26T23:30:53.148268+0000","last_clean":"2024-01-26T23:30:53.148268+0000","last_became_active":"2024-01-26T23:30:53.147961+0000","last_became_peered":"2024-01-26T23:30:53.147961+0000","last_unstale":"2024-01-26T23:30:53.148268+0000","last_undegraded":"2024-01-26T23:30:53.148268+0000","last_fullsized":"2024-01-26T23:30:53.148268+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:06:09.705120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":17,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.163523+0000","last_change":"2024-01-26T23:30:52.134957+0000","last_active":"2024-01-26T23:30:53.163523+0000","last_peered":"2024-01-26T23:30:53.163523+0000","last_clean":"2024-01-26T23:30:53.163523+0000","last_became_active":"2024-01-26T23:30:52.134683+0000","last_became_peered":"2024-01-26T23:30:52.134683+0000","last_unstale":"2024-01-26T23:30:53.163523+0000","last_undegraded":"2024-01-26T23:30:53.163523+0000","last_fullsized":"2024-01-26T23:30:53.163523+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:34:33.996022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.147505+0000","last_change":"2024-01-26T23:30:53.147505+0000","last_active":"2024-01-26T23:30:53.147505+0000","last_peered":"2024-01-26T23:30:53.147505+0000","last_clean":"2024-01-26T23:30:53.147505+0000","last_became_active":"2024-01-26T23:30:53.146950+0000","last_became_peered":"2024-01-26T23:30:53.146950+0000","last_unstale":"2024-01-26T23:30:53.147505+0000","last_undegraded":"2024-01-26T23:30:53.147505+0000","last_fullsized":"2024-01-26T23:30:53.147505+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:05:11.628582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.145542+0000","last_change":"2024-01-26T23:30:52.134426+0000","last_active":"2024-01-26T23:30:53.145542+0000","last_peered":"2024-01-26T23:30:53.145542+0000","last_clean":"2024-01-26T23:30:53.145542+0000","last_became_active":"2024-01-26T23:30:52.134181+0000","last_became_peered":"2024-01-26T23:30:52.134181+0000","last_unstale":"2024-01-26T23:30:53.145542+0000","last_undegraded":"2024-01-26T23:30:53.145542+0000","last_fullsized":"2024-01-26T23:30:53.145542+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:01:38.001177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150982+0000","last_change":"2024-01-26T23:30:53.150982+0000","last_active":"2024-01-26T23:30:53.150982+0000","last_peered":"2024-01-26T23:30:53.150982+0000","last_clean":"2024-01-26T23:30:53.150982+0000","last_became_active":"2024-01-26T23:30:53.150541+0000","last_became_peered":"2024-01-26T23:30:53.150541+0000","last_unstale":"2024-01-26T23:30:53.150982+0000","last_undegraded":"2024-01-26T23:30:53.150982+0000","last_fullsized":"2024-01-26T23:30:53.150982+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:42:40.534420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.163598+0000","last_change":"2024-01-26T23:30:52.137001+0000","last_active":"2024-01-26T23:30:53.163598+0000","last_peered":"2024-01-26T23:30:53.163598+0000","last_clean":"2024-01-26T23:30:53.163598+0000","last_became_active":"2024-01-26T23:30:52.136689+0000","last_became_peered":"2024-01-26T23:30:52.136689+0000","last_unstale":"2024-01-26T23:30:53.163598+0000","last_undegraded":"2024-01-26T23:30:53.163598+0000","last_fullsized":"2024-01-26T23:30:53.163598+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:13.472460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.143475+0000","last_change":"2024-01-26T23:30:53.143475+0000","last_active":"2024-01-26T23:30:53.143475+0000","last_peered":"2024-01-26T23:30:53.143475+0000","last_clean":"2024-01-26T23:30:53.143475+0000","last_became_active":"2024-01-26T23:30:53.143203+0000","last_became_peered":"2024-01-26T23:30:53.143203+0000","last_unstale":"2024-01-26T23:30:53.143475+0000","last_undegraded":"2024-01-26T23:30:53.143475+0000","last_fullsized":"2024-01-26T23:30:53.143475+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:42:40.534420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.131127+0000","last_change":"2024-01-26T23:30:52.135188+0000","last_active":"2024-01-26T23:30:53.131127+0000","last_peered":"2024-01-26T23:30:53.131127+0000","last_clean":"2024-01-26T23:30:53.131127+0000","last_became_active":"2024-01-26T23:30:52.134868+0000","last_became_peered":"2024-01-26T23:30:52.134868+0000","last_unstale":"2024-01-26T23:30:53.131127+0000","last_undegraded":"2024-01-26T23:30:53.131127+0000","last_fullsized":"2024-01-26T23:30:53.131127+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:51:32.360927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.134504+0000","last_change":"2024-01-26T23:30:53.134504+0000","last_active":"2024-01-26T23:30:53.134504+0000","last_peered":"2024-01-26T23:30:53.134504+0000","last_clean":"2024-01-26T23:30:53.134504+0000","last_became_active":"2024-01-26T23:30:53.134064+0000","last_became_peered":"2024-01-26T23:30:53.134064+0000","last_unstale":"2024-01-26T23:30:53.134504+0000","last_undegraded":"2024-01-26T23:30:53.134504+0000","last_fullsized":"2024-01-26T23:30:53.134504+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:25:38.452947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.141259+0000","last_change":"2024-01-26T23:30:52.139344+0000","last_active":"2024-01-26T23:30:52.141259+0000","last_peered":"2024-01-26T23:30:52.141259+0000","last_clean":"2024-01-26T23:30:52.141259+0000","last_became_active":"2024-01-26T23:30:52.139058+0000","last_became_peered":"2024-01-26T23:30:52.139058+0000","last_unstale":"2024-01-26T23:30:52.141259+0000","last_undegraded":"2024-01-26T23:30:52.141259+0000","last_fullsized":"2024-01-26T23:30:52.141259+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:41:51.593459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150669+0000","last_change":"2024-01-26T23:30:53.150669+0000","last_active":"2024-01-26T23:30:53.150669+0000","last_peered":"2024-01-26T23:30:53.150669+0000","last_clean":"2024-01-26T23:30:53.150669+0000","last_became_active":"2024-01-26T23:30:53.150389+0000","last_became_peered":"2024-01-26T23:30:53.150389+0000","last_unstale":"2024-01-26T23:30:53.150669+0000","last_undegraded":"2024-01-26T23:30:53.150669+0000","last_fullsized":"2024-01-26T23:30:53.150669+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:50.835391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.162023+0000","last_change":"2024-01-26T23:30:52.127727+0000","last_active":"2024-01-26T23:30:53.162023+0000","last_peered":"2024-01-26T23:30:53.162023+0000","last_clean":"2024-01-26T23:30:53.162023+0000","last_became_active":"2024-01-26T23:30:52.127444+0000","last_became_peered":"2024-01-26T23:30:52.127444+0000","last_unstale":"2024-01-26T23:30:53.162023+0000","last_undegraded":"2024-01-26T23:30:53.162023+0000","last_fullsized":"2024-01-26T23:30:53.162023+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:15:51.865289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.137131+0000","last_change":"2024-01-26T23:30:53.137131+0000","last_active":"2024-01-26T23:30:53.137131+0000","last_peered":"2024-01-26T23:30:53.137131+0000","last_clean":"2024-01-26T23:30:53.137131+0000","last_became_active":"2024-01-26T23:30:53.136869+0000","last_became_peered":"2024-01-26T23:30:53.136869+0000","last_unstale":"2024-01-26T23:30:53.137131+0000","last_undegraded":"2024-01-26T23:30:53.137131+0000","last_fullsized":"2024-01-26T23:30:53.137131+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:00:27.950751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.136366+0000","last_change":"2024-01-26T23:30:52.134140+0000","last_active":"2024-01-26T23:30:52.136366+0000","last_peered":"2024-01-26T23:30:52.136366+0000","last_clean":"2024-01-26T23:30:52.136366+0000","last_became_active":"2024-01-26T23:30:52.133838+0000","last_became_peered":"2024-01-26T23:30:52.133838+0000","last_unstale":"2024-01-26T23:30:52.136366+0000","last_undegraded":"2024-01-26T23:30:52.136366+0000","last_fullsized":"2024-01-26T23:30:52.136366+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:53:04.923997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.149915+0000","last_change":"2024-01-26T23:30:53.149915+0000","last_active":"2024-01-26T23:30:53.149915+0000","last_peered":"2024-01-26T23:30:53.149915+0000","last_clean":"2024-01-26T23:30:53.149915+0000","last_became_active":"2024-01-26T23:30:53.149567+0000","last_became_peered":"2024-01-26T23:30:53.149567+0000","last_unstale":"2024-01-26T23:30:53.149915+0000","last_undegraded":"2024-01-26T23:30:53.149915+0000","last_fullsized":"2024-01-26T23:30:53.149915+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:35:46.571656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.140119+0000","last_change":"2024-01-26T23:30:52.134836+0000","last_active":"2024-01-26T23:30:52.140119+0000","last_peered":"2024-01-26T23:30:52.140119+0000","last_clean":"2024-01-26T23:30:52.140119+0000","last_became_active":"2024-01-26T23:30:52.134563+0000","last_became_peered":"2024-01-26T23:30:52.134563+0000","last_unstale":"2024-01-26T23:30:52.140119+0000","last_undegraded":"2024-01-26T23:30:52.140119+0000","last_fullsized":"2024-01-26T23:30:52.140119+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:31:04.332740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.138819+0000","last_change":"2024-01-26T23:30:53.138819+0000","last_active":"2024-01-26T23:30:53.138819+0000","last_peered":"2024-01-26T23:30:53.138819+0000","last_clean":"2024-01-26T23:30:53.138819+0000","last_became_active":"2024-01-26T23:30:53.138106+0000","last_became_peered":"2024-01-26T23:30:53.138106+0000","last_unstale":"2024-01-26T23:30:53.138819+0000","last_undegraded":"2024-01-26T23:30:53.138819+0000","last_fullsized":"2024-01-26T23:30:53.138819+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:35:06.371452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.131812+0000","last_change":"2024-01-26T23:30:52.127739+0000","last_active":"2024-01-26T23:30:52.131812+0000","last_peered":"2024-01-26T23:30:52.131812+0000","last_clean":"2024-01-26T23:30:52.131812+0000","last_became_active":"2024-01-26T23:30:52.127132+0000","last_became_peered":"2024-01-26T23:30:52.127132+0000","last_unstale":"2024-01-26T23:30:52.131812+0000","last_undegraded":"2024-01-26T23:30:52.131812+0000","last_fullsized":"2024-01-26T23:30:52.131812+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:19:41.666152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.153869+0000","last_change":"2024-01-26T23:30:53.153869+0000","last_active":"2024-01-26T23:30:53.153869+0000","last_peered":"2024-01-26T23:30:53.153869+0000","last_clean":"2024-01-26T23:30:53.153869+0000","last_became_active":"2024-01-26T23:30:53.153556+0000","last_became_peered":"2024-01-26T23:30:53.153556+0000","last_unstale":"2024-01-26T23:30:53.153869+0000","last_undegraded":"2024-01-26T23:30:53.153869+0000","last_fullsized":"2024-01-26T23:30:53.153869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:14.479539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.163263+0000","last_change":"2024-01-26T23:30:52.133918+0000","last_active":"2024-01-26T23:30:53.163263+0000","last_peered":"2024-01-26T23:30:53.163263+0000","last_clean":"2024-01-26T23:30:53.163263+0000","last_became_active":"2024-01-26T23:30:52.133679+0000","last_became_peered":"2024-01-26T23:30:52.133679+0000","last_unstale":"2024-01-26T23:30:53.163263+0000","last_undegraded":"2024-01-26T23:30:53.163263+0000","last_fullsized":"2024-01-26T23:30:53.163263+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:27.457158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.146855+0000","last_change":"2024-01-26T23:30:53.146855+0000","last_active":"2024-01-26T23:30:53.146855+0000","last_peered":"2024-01-26T23:30:53.146855+0000","last_clean":"2024-01-26T23:30:53.146855+0000","last_became_active":"2024-01-26T23:30:53.146583+0000","last_became_peered":"2024-01-26T23:30:53.146583+0000","last_unstale":"2024-01-26T23:30:53.146855+0000","last_undegraded":"2024-01-26T23:30:53.146855+0000","last_fullsized":"2024-01-26T23:30:53.146855+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:01:39.008256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.136093+0000","last_change":"2024-01-26T23:30:52.132982+0000","last_active":"2024-01-26T23:30:52.136093+0000","last_peered":"2024-01-26T23:30:52.136093+0000","last_clean":"2024-01-26T23:30:52.136093+0000","last_became_active":"2024-01-26T23:30:52.132735+0000","last_became_peered":"2024-01-26T23:30:52.132735+0000","last_unstale":"2024-01-26T23:30:52.136093+0000","last_undegraded":"2024-01-26T23:30:52.136093+0000","last_fullsized":"2024-01-26T23:30:52.136093+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:10:08.015052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.145179+0000","last_change":"2024-01-26T23:30:53.145179+0000","last_active":"2024-01-26T23:30:53.145179+0000","last_peered":"2024-01-26T23:30:53.145179+0000","last_clean":"2024-01-26T23:30:53.145179+0000","last_became_active":"2024-01-26T23:30:53.144878+0000","last_became_peered":"2024-01-26T23:30:53.144878+0000","last_unstale":"2024-01-26T23:30:53.145179+0000","last_undegraded":"2024-01-26T23:30:53.145179+0000","last_fullsized":"2024-01-26T23:30:53.145179+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:50.781140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.132445+0000","last_change":"2024-01-26T23:30:52.130274+0000","last_active":"2024-01-26T23:30:52.132445+0000","last_peered":"2024-01-26T23:30:52.132445+0000","last_clean":"2024-01-26T23:30:52.132445+0000","last_became_active":"2024-01-26T23:30:52.130023+0000","last_became_peered":"2024-01-26T23:30:52.130023+0000","last_unstale":"2024-01-26T23:30:52.132445+0000","last_undegraded":"2024-01-26T23:30:52.132445+0000","last_fullsized":"2024-01-26T23:30:52.132445+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:49.828311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.148491+0000","last_change":"2024-01-26T23:30:53.148491+0000","last_active":"2024-01-26T23:30:53.148491+0000","last_peered":"2024-01-26T23:30:53.148491+0000","last_clean":"2024-01-26T23:30:53.148491+0000","last_became_active":"2024-01-26T23:30:53.148059+0000","last_became_peered":"2024-01-26T23:30:53.148059+0000","last_unstale":"2024-01-26T23:30:53.148491+0000","last_undegraded":"2024-01-26T23:30:53.148491+0000","last_fullsized":"2024-01-26T23:30:53.148491+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:06:09.705120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.136621+0000","last_change":"2024-01-26T23:30:52.132426+0000","last_active":"2024-01-26T23:30:52.136621+0000","last_peered":"2024-01-26T23:30:52.136621+0000","last_clean":"2024-01-26T23:30:52.136621+0000","last_became_active":"2024-01-26T23:30:52.132083+0000","last_became_peered":"2024-01-26T23:30:52.132083+0000","last_unstale":"2024-01-26T23:30:52.136621+0000","last_undegraded":"2024-01-26T23:30:52.136621+0000","last_fullsized":"2024-01-26T23:30:52.136621+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:13.472460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.147932+0000","last_change":"2024-01-26T23:30:53.147932+0000","last_active":"2024-01-26T23:30:53.147932+0000","last_peered":"2024-01-26T23:30:53.147932+0000","last_clean":"2024-01-26T23:30:53.147932+0000","last_became_active":"2024-01-26T23:30:53.147653+0000","last_became_peered":"2024-01-26T23:30:53.147653+0000","last_unstale":"2024-01-26T23:30:53.147932+0000","last_undegraded":"2024-01-26T23:30:53.147932+0000","last_fullsized":"2024-01-26T23:30:53.147932+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:50.835391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.131859+0000","last_change":"2024-01-26T23:30:52.129625+0000","last_active":"2024-01-26T23:30:52.131859+0000","last_peered":"2024-01-26T23:30:52.131859+0000","last_clean":"2024-01-26T23:30:52.131859+0000","last_became_active":"2024-01-26T23:30:52.129329+0000","last_became_peered":"2024-01-26T23:30:52.129329+0000","last_unstale":"2024-01-26T23:30:52.131859+0000","last_undegraded":"2024-01-26T23:30:52.131859+0000","last_fullsized":"2024-01-26T23:30:52.131859+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:34:33.996022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.140434+0000","last_change":"2024-01-26T23:30:53.140434+0000","last_active":"2024-01-26T23:30:53.140434+0000","last_peered":"2024-01-26T23:30:53.140434+0000","last_clean":"2024-01-26T23:30:53.140434+0000","last_became_active":"2024-01-26T23:30:53.140202+0000","last_became_peered":"2024-01-26T23:30:53.140202+0000","last_unstale":"2024-01-26T23:30:53.140434+0000","last_undegraded":"2024-01-26T23:30:53.140434+0000","last_fullsized":"2024-01-26T23:30:53.140434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:53:34.849263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.140902+0000","last_change":"2024-01-26T23:30:52.139811+0000","last_active":"2024-01-26T23:30:52.140902+0000","last_peered":"2024-01-26T23:30:52.140902+0000","last_clean":"2024-01-26T23:30:52.140902+0000","last_became_active":"2024-01-26T23:30:52.139520+0000","last_became_peered":"2024-01-26T23:30:52.139520+0000","last_unstale":"2024-01-26T23:30:52.140902+0000","last_undegraded":"2024-01-26T23:30:52.140902+0000","last_fullsized":"2024-01-26T23:30:52.140902+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:46:40.377610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.147895+0000","last_change":"2024-01-26T23:30:53.147895+0000","last_active":"2024-01-26T23:30:53.147895+0000","last_peered":"2024-01-26T23:30:53.147895+0000","last_clean":"2024-01-26T23:30:53.147895+0000","last_became_active":"2024-01-26T23:30:53.147559+0000","last_became_peered":"2024-01-26T23:30:53.147559+0000","last_unstale":"2024-01-26T23:30:53.147895+0000","last_undegraded":"2024-01-26T23:30:53.147895+0000","last_fullsized":"2024-01-26T23:30:53.147895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:01:40.621482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.139076+0000","last_change":"2024-01-26T23:30:52.131891+0000","last_active":"2024-01-26T23:30:52.139076+0000","last_peered":"2024-01-26T23:30:52.139076+0000","last_clean":"2024-01-26T23:30:52.139076+0000","last_became_active":"2024-01-26T23:30:52.131618+0000","last_became_peered":"2024-01-26T23:30:52.131618+0000","last_unstale":"2024-01-26T23:30:52.139076+0000","last_undegraded":"2024-01-26T23:30:52.139076+0000","last_fullsized":"2024-01-26T23:30:52.139076+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:01:39.614403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.159160+0000","last_change":"2024-01-26T23:30:53.159160+0000","last_active":"2024-01-26T23:30:53.159160+0000","last_peered":"2024-01-26T23:30:53.159160+0000","last_clean":"2024-01-26T23:30:53.159160+0000","last_became_active":"2024-01-26T23:30:53.157580+0000","last_became_peered":"2024-01-26T23:30:53.157580+0000","last_unstale":"2024-01-26T23:30:53.159160+0000","last_undegraded":"2024-01-26T23:30:53.159160+0000","last_fullsized":"2024-01-26T23:30:53.159160+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:50.289763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.140436+0000","last_change":"2024-01-26T23:30:52.135416+0000","last_active":"2024-01-26T23:30:52.140436+0000","last_peered":"2024-01-26T23:30:52.140436+0000","last_clean":"2024-01-26T23:30:52.140436+0000","last_became_active":"2024-01-26T23:30:52.135144+0000","last_became_peered":"2024-01-26T23:30:52.135144+0000","last_unstale":"2024-01-26T23:30:52.140436+0000","last_undegraded":"2024-01-26T23:30:52.140436+0000","last_fullsized":"2024-01-26T23:30:52.140436+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:52:03.559898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":177,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.125470+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:30:52.125470+0000","last_peered":"2024-01-26T23:30:52.125470+0000","last_clean":"2024-01-26T23:30:52.125470+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:30:52.125470+0000","last_undegraded":"2024-01-26T23:30:52.125470+0000","last_fullsized":"2024-01-26T23:30:52.125470+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298272,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073568,"statfs":{"total":96636764160,"available":96331333632,"internally_reserved":0,"allocated":1007616,"data_stored":764697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.65000000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.495},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.63700000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.625},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.60799999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.56399999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.52400000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.66800000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297692,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074148,"statfs":{"total":96636764160,"available":96331927552,"internally_reserved":0,"allocated":413696,"data_stored":174329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55600000000000005}]},{"osd":1,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.56899999999999995}]},{"osd":2,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.58099999999999996}]},{"osd":3,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.626}]},{"osd":4,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.45800000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.495}]},{"osd":7,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.48399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297708,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074132,"statfs":{"total":96636764160,"available":96331911168,"internally_reserved":0,"allocated":430080,"data_stored":181502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.432}]},{"osd":2,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.39400000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.40899999999999997}]},{"osd":4,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.69499999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.51100000000000001}]},{"osd":6,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.746}]},{"osd":7,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.71399999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298284,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073556,"statfs":{"total":96636764160,"available":96331321344,"internally_reserved":0,"allocated":1019904,"data_stored":771780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.33600000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.41099999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.46999999999999997}]},{"osd":4,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.60199999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.61899999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.58899999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.70999999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297692,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074148,"statfs":{"total":96636764160,"available":96331927552,"internally_reserved":0,"allocated":413696,"data_stored":174329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":9.0540000000000003,"5min":9.0540000000000003,"15min":9.0540000000000003},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":4.1970000000000001,"5min":4.1970000000000001,"15min":4.1970000000000001},"last":0.32700000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.311},{"interface":"front","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":12.157,"5min":12.157,"15min":12.157},"last":0.41099999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":9.0879999999999992,"5min":9.0879999999999992,"15min":9.0879999999999992},"last":0.436}]},{"osd":4,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":9.0150000000000006,"5min":9.0150000000000006,"15min":9.0150000000000006},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":4.2480000000000002,"5min":4.2480000000000002,"15min":4.2480000000000002},"last":0.59499999999999997}]},{"osd":5,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":4.3200000000000003,"5min":4.3200000000000003,"15min":4.3200000000000003},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.60899999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.57999999999999996},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":8.9700000000000006,"5min":8.9700000000000006,"15min":8.9700000000000006},"last":0.70499999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":4.282,"5min":4.282,"15min":4.282},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.79600000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297692,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074148,"statfs":{"total":96636764160,"available":96331927552,"internally_reserved":0,"allocated":413696,"data_stored":174329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.26600000000000001},{"interface":"front","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":11.968,"5min":11.968,"15min":11.968},"last":0.31}]},{"osd":1,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":12.050000000000001,"5min":12.050000000000001,"15min":12.050000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.40699999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":12.234,"5min":12.234,"15min":12.234},"last":0.32900000000000001},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":11.925000000000001,"5min":11.925000000000001,"15min":11.925000000000001},"last":0.38900000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.222,"15min":1.222},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":12.099,"5min":12.099,"15min":12.099},"last":0.629},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.50700000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":12.199999999999999,"5min":12.199999999999999,"15min":12.199999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.54100000000000004}]},{"osd":6,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.25,"15min":1.25},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":12.132999999999999,"5min":12.132999999999999,"15min":12.132999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.61399999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":12.167,"5min":12.167,"15min":12.167},"last":0.48699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297692,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074148,"statfs":{"total":96636764160,"available":96331927552,"internally_reserved":0,"allocated":413696,"data_stored":174329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.60199999999999998}]},{"osd":1,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.66900000000000004}]},{"osd":2,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.5},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.65800000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.53200000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.622}]},{"osd":6,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.443},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.51400000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.63600000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297692,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074148,"statfs":{"total":96636764160,"available":96331927552,"internally_reserved":0,"allocated":413696,"data_stored":174329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.72899999999999998}]},{"osd":1,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.63300000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":6.4589999999999996,"5min":6.4589999999999996,"15min":6.4589999999999996},"last":0.59399999999999997}]},{"osd":3,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.497},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"last":0.54500000000000004}]},{"osd":4,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.58099999999999996}]},{"osd":6,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.65300000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.71599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:30:57.337 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:30:57.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.502+0000 7f2dd48d0700 1 Processor -- start 2024-01-26T23:30:57.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.502+0000 7f2dd48d0700 1 -- start start 2024-01-26T23:30:57.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.503+0000 7f2dd48d0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2dd01596e0 0x7f2dd0159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.503+0000 7f2dd48d0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2dd00cb4c0 0x7f2dd0068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.503+0000 7f2dd48d0700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd00c97f0 con 0x7f2dd00c4720 2024-01-26T23:30:57.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.503+0000 7f2dd48d0700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd00cbf90 con 0x7f2dd00cb4c0 2024-01-26T23:30:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.503+0000 7f2dd48d0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd00c8c20 con 0x7f2dd01596e0 2024-01-26T23:30:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.503+0000 7f2dced9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2dd00cb4c0 0x7f2dd0068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.503+0000 7f2dced9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2dd00cb4c0 0x7f2dd0068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60046/0 (socket says 172.21.15.46:60046) 2024-01-26T23:30:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.503+0000 7f2dced9d700 1 -- 172.21.15.46:0/4042204909 learned_addr learned my addr 172.21.15.46:0/4042204909 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.503+0000 7f2dce59c700 1 --2- 172.21.15.46:0/4042204909 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2dd01596e0 0x7f2dd0159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.503+0000 7f2dced9d700 1 -- 172.21.15.46:0/4042204909 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2dd01596e0 msgr2=0x7f2dd0159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.503+0000 7f2dced9d700 1 --2- 172.21.15.46:0/4042204909 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2dd01596e0 0x7f2dd0159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.503+0000 7f2dced9d700 1 -- 172.21.15.46:0/4042204909 >> v1:172.21.15.106:6789/0 conn(0x7f2dd00c4720 legacy=0x7f2dd00cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.503+0000 7f2dced9d700 1 -- 172.21.15.46:0/4042204909 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd00c8ec0 con 0x7f2dd00cb4c0 2024-01-26T23:30:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.503+0000 7f2dced9d700 1 --2- 172.21.15.46:0/4042204909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd00cb4c0 0x7f2dd0068a40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2dc80099b0 tx=0x7f2dc8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0563b4af946f4b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.504+0000 7f2dcd59a700 1 -- 172.21.15.46:0/4042204909 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc800b860 con 0x7f2dd00cb4c0 2024-01-26T23:30:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.504+0000 7f2dcd59a700 1 -- 172.21.15.46:0/4042204909 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc800b9c0 con 0x7f2dd00cb4c0 2024-01-26T23:30:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.504+0000 7f2dcd59a700 1 -- 172.21.15.46:0/4042204909 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc800bd60 con 0x7f2dd00cb4c0 2024-01-26T23:30:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.504+0000 7f2dd48d0700 1 -- 172.21.15.46:0/4042204909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd00cb4c0 msgr2=0x7f2dd0068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.504+0000 7f2dd48d0700 1 --2- 172.21.15.46:0/4042204909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd00cb4c0 0x7f2dd0068a40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2dc80099b0 tx=0x7f2dc8009250 comp rx=0 tx=0).stop 2024-01-26T23:30:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.504+0000 7f2dd48d0700 1 -- 172.21.15.46:0/4042204909 shutdown_connections 2024-01-26T23:30:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.504+0000 7f2dd48d0700 1 --2- 172.21.15.46:0/4042204909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd00cb4c0 0x7f2dd0068a40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.504+0000 7f2dd48d0700 1 --2- 172.21.15.46:0/4042204909 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2dd01596e0 0x7f2dd0159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.504+0000 7f2dd48d0700 1 -- 172.21.15.46:0/4042204909 >> 172.21.15.46:0/4042204909 conn(0x7f2dd0097820 msgr2=0x7f2dd0068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.504+0000 7f2dd48d0700 1 -- 172.21.15.46:0/4042204909 shutdown_connections 2024-01-26T23:30:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.504+0000 7f2dd48d0700 1 -- 172.21.15.46:0/4042204909 wait complete. 2024-01-26T23:30:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.505+0000 7f2dd48d0700 1 Processor -- start 2024-01-26T23:30:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.506+0000 7f2dd48d0700 1 -- start start 2024-01-26T23:30:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.506+0000 7f2dd48d0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2dd00c4720 0x7f2dd015ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.506+0000 7f2dd48d0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd00cb4c0 0x7f2dd0158700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.506+0000 7f2dd48d0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2dd01596e0 0x7f2dd015aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.506+0000 7f2dd48d0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd015eaa0 con 0x7f2dd00cb4c0 2024-01-26T23:30:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.506+0000 7f2dd48d0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd0158c40 con 0x7f2dd01596e0 2024-01-26T23:30:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.506+0000 7f2dd48d0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd015e770 con 0x7f2dd00c4720 2024-01-26T23:30:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.506+0000 7f2dcdd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd00cb4c0 0x7f2dd0158700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.506+0000 7f2dcdd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd00cb4c0 0x7f2dd0158700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60058/0 (socket says 172.21.15.46:60058) 2024-01-26T23:30:57.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.506+0000 7f2dcdd9b700 1 -- 172.21.15.46:0/3336340941 learned_addr learned my addr 172.21.15.46:0/3336340941 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:57.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.507+0000 7f2dced9d700 1 --2- 172.21.15.46:0/3336340941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2dd01596e0 0x7f2dd015aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.507+0000 7f2dce59c700 1 --2- 172.21.15.46:0/3336340941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2dd00c4720 0x7f2dd015ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.507+0000 7f2dcdd9b700 1 -- 172.21.15.46:0/3336340941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2dd00c4720 msgr2=0x7f2dd015ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:57.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.507+0000 7f2dcdd9b700 1 --2- 172.21.15.46:0/3336340941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2dd00c4720 0x7f2dd015ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.507+0000 7f2dcdd9b700 1 -- 172.21.15.46:0/3336340941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2dd01596e0 msgr2=0x7f2dd015aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:57.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.507+0000 7f2dcdd9b700 1 --2- 172.21.15.46:0/3336340941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2dd01596e0 0x7f2dd015aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.507+0000 7f2dcdd9b700 1 -- 172.21.15.46:0/3336340941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc80094f0 con 0x7f2dd00cb4c0 2024-01-26T23:30:57.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.507+0000 7f2dcdd9b700 1 --2- 172.21.15.46:0/3336340941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd00cb4c0 0x7f2dd0158700 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2db800c590 tx=0x7f2db800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:57.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.507+0000 7f2dc67fc700 1 -- 172.21.15.46:0/3336340941 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db8011590 con 0x7f2dd00cb4c0 2024-01-26T23:30:57.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.507+0000 7f2dc67fc700 1 -- 172.21.15.46:0/3336340941 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db8011710 con 0x7f2dd00cb4c0 2024-01-26T23:30:57.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.507+0000 7f2dd48d0700 1 -- 172.21.15.46:0/3336340941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd015b240 con 0x7f2dd00cb4c0 2024-01-26T23:30:57.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.507+0000 7f2dc67fc700 1 -- 172.21.15.46:0/3336340941 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db8011ad0 con 0x7f2dd00cb4c0 2024-01-26T23:30:57.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.507+0000 7f2dd48d0700 1 -- 172.21.15.46:0/3336340941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd015b800 con 0x7f2dd00cb4c0 2024-01-26T23:30:57.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.508+0000 7f2dd48d0700 1 -- 172.21.15.46:0/3336340941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd004c710 con 0x7f2dd00cb4c0 2024-01-26T23:30:57.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.509+0000 7f2dc67fc700 1 -- 172.21.15.46:0/3336340941 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2db801b3b0 con 0x7f2dd00cb4c0 2024-01-26T23:30:57.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.509+0000 7f2dc67fc700 1 --2- 172.21.15.46:0/3336340941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2db0066f30 0x7f2db00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.509+0000 7f2dc67fc700 1 -- 172.21.15.46:0/3336340941 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f2db8083b00 con 0x7f2dd00cb4c0 2024-01-26T23:30:57.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.509+0000 7f2dce59c700 1 --2- 172.21.15.46:0/3336340941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2db0066f30 0x7f2db00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.510+0000 7f2dce59c700 1 --2- 172.21.15.46:0/3336340941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2db0066f30 0x7f2db00693e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2dc00075a0 tx=0x7f2dc00097b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:57.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.514+0000 7f2dc67fc700 1 -- 172.21.15.46:0/3336340941 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2db8050c30 con 0x7f2dd00cb4c0 2024-01-26T23:30:57.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.714+0000 7f2dd48d0700 1 -- 172.21.15.46:0/3336340941 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2dd015ebe0 con 0x7f2db0066f30 2024-01-26T23:30:57.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.722+0000 7f2dc67fc700 1 -- 172.21.15.46:0/3336340941 <== mgr.4102 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289349 (secure 0 0 0) 0x7f2dd015ebe0 con 0x7f2db0066f30 2024-01-26T23:30:57.727 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:57.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.725+0000 7f2dd48d0700 1 -- 172.21.15.46:0/3336340941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2db0066f30 msgr2=0x7f2db00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:57.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.725+0000 7f2dd48d0700 1 --2- 172.21.15.46:0/3336340941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2db0066f30 0x7f2db00693e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2dc00075a0 tx=0x7f2dc00097b0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.725+0000 7f2dd48d0700 1 -- 172.21.15.46:0/3336340941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd00cb4c0 msgr2=0x7f2dd0158700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:57.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.726+0000 7f2dd48d0700 1 --2- 172.21.15.46:0/3336340941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd00cb4c0 0x7f2dd0158700 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2db800c590 tx=0x7f2db800c8a0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.726+0000 7f2dd48d0700 1 -- 172.21.15.46:0/3336340941 shutdown_connections 2024-01-26T23:30:57.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.726+0000 7f2dd48d0700 1 --2- 172.21.15.46:0/3336340941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2db0066f30 0x7f2db00693e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.726+0000 7f2dd48d0700 1 --2- 172.21.15.46:0/3336340941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2dd01596e0 0x7f2dd015aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.726+0000 7f2dd48d0700 1 --2- 172.21.15.46:0/3336340941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2dd00c4720 0x7f2dd015ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.726+0000 7f2dd48d0700 1 --2- 172.21.15.46:0/3336340941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd00cb4c0 0x7f2dd0158700 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.726+0000 7f2dd48d0700 1 -- 172.21.15.46:0/3336340941 >> 172.21.15.46:0/3336340941 conn(0x7f2dd0097820 msgr2=0x7f2dd006ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:57.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.726+0000 7f2dd48d0700 1 -- 172.21.15.46:0/3336340941 shutdown_connections 2024-01-26T23:30:57.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.726+0000 7f2dd48d0700 1 -- 172.21.15.46:0/3336340941 wait complete. 2024-01-26T23:30:57.730 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:30:57.742 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2024-01-26T23:30:56.242553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2382724,"kb_used_data":4420,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752591996,"statfs":{"total":773094113280,"available":770654203904,"internally_reserved":0,"allocated":4526080,"data_stored":2589624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.129728"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.148930+0000","last_change":"2024-01-26T23:30:53.148930+0000","last_active":"2024-01-26T23:30:53.148930+0000","last_peered":"2024-01-26T23:30:53.148930+0000","last_clean":"2024-01-26T23:30:53.148930+0000","last_became_active":"2024-01-26T23:30:53.147918+0000","last_became_peered":"2024-01-26T23:30:53.147918+0000","last_unstale":"2024-01-26T23:30:53.148930+0000","last_undegraded":"2024-01-26T23:30:53.148930+0000","last_fullsized":"2024-01-26T23:30:53.148930+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:50.289763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.135001+0000","last_change":"2024-01-26T23:30:53.135001+0000","last_active":"2024-01-26T23:30:53.135001+0000","last_peered":"2024-01-26T23:30:53.135001+0000","last_clean":"2024-01-26T23:30:53.135001+0000","last_became_active":"2024-01-26T23:30:53.134680+0000","last_became_peered":"2024-01-26T23:30:53.134680+0000","last_unstale":"2024-01-26T23:30:53.135001+0000","last_undegraded":"2024-01-26T23:30:53.135001+0000","last_fullsized":"2024-01-26T23:30:53.135001+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:15:26.285891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.153204+0000","last_change":"2024-01-26T23:30:53.153204+0000","last_active":"2024-01-26T23:30:53.153204+0000","last_peered":"2024-01-26T23:30:53.153204+0000","last_clean":"2024-01-26T23:30:53.153204+0000","last_became_active":"2024-01-26T23:30:53.152966+0000","last_became_peered":"2024-01-26T23:30:53.152966+0000","last_unstale":"2024-01-26T23:30:53.153204+0000","last_undegraded":"2024-01-26T23:30:53.153204+0000","last_fullsized":"2024-01-26T23:30:53.153204+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:15:26.285891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150803+0000","last_change":"2024-01-26T23:30:53.150803+0000","last_active":"2024-01-26T23:30:53.150803+0000","last_peered":"2024-01-26T23:30:53.150803+0000","last_clean":"2024-01-26T23:30:53.150803+0000","last_became_active":"2024-01-26T23:30:53.150342+0000","last_became_peered":"2024-01-26T23:30:53.150342+0000","last_unstale":"2024-01-26T23:30:53.150803+0000","last_undegraded":"2024-01-26T23:30:53.150803+0000","last_fullsized":"2024-01-26T23:30:53.150803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:52:04.566977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150402+0000","last_change":"2024-01-26T23:30:53.150402+0000","last_active":"2024-01-26T23:30:53.150402+0000","last_peered":"2024-01-26T23:30:53.150402+0000","last_clean":"2024-01-26T23:30:53.150402+0000","last_became_active":"2024-01-26T23:30:53.149876+0000","last_became_peered":"2024-01-26T23:30:53.149876+0000","last_unstale":"2024-01-26T23:30:53.150402+0000","last_undegraded":"2024-01-26T23:30:53.150402+0000","last_fullsized":"2024-01-26T23:30:53.150402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:48:21.769810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.146114+0000","last_change":"2024-01-26T23:30:53.146114+0000","last_active":"2024-01-26T23:30:53.146114+0000","last_peered":"2024-01-26T23:30:53.146114+0000","last_clean":"2024-01-26T23:30:53.146114+0000","last_became_active":"2024-01-26T23:30:53.145832+0000","last_became_peered":"2024-01-26T23:30:53.145832+0000","last_unstale":"2024-01-26T23:30:53.146114+0000","last_undegraded":"2024-01-26T23:30:53.146114+0000","last_fullsized":"2024-01-26T23:30:53.146114+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:05:11.628582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.149995+0000","last_change":"2024-01-26T23:30:53.149995+0000","last_active":"2024-01-26T23:30:53.149995+0000","last_peered":"2024-01-26T23:30:53.149995+0000","last_clean":"2024-01-26T23:30:53.149995+0000","last_became_active":"2024-01-26T23:30:53.149722+0000","last_became_peered":"2024-01-26T23:30:53.149722+0000","last_unstale":"2024-01-26T23:30:53.149995+0000","last_undegraded":"2024-01-26T23:30:53.149995+0000","last_fullsized":"2024-01-26T23:30:53.149995+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:53:34.849263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.155843+0000","last_change":"2024-01-26T23:30:53.155843+0000","last_active":"2024-01-26T23:30:53.155843+0000","last_peered":"2024-01-26T23:30:53.155843+0000","last_clean":"2024-01-26T23:30:53.155843+0000","last_became_active":"2024-01-26T23:30:53.155361+0000","last_became_peered":"2024-01-26T23:30:53.155361+0000","last_unstale":"2024-01-26T23:30:53.155843+0000","last_undegraded":"2024-01-26T23:30:53.155843+0000","last_fullsized":"2024-01-26T23:30:53.155843+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:00:27.950751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.152696+0000","last_change":"2024-01-26T23:30:53.152696+0000","last_active":"2024-01-26T23:30:53.152696+0000","last_peered":"2024-01-26T23:30:53.152696+0000","last_clean":"2024-01-26T23:30:53.152696+0000","last_became_active":"2024-01-26T23:30:53.152375+0000","last_became_peered":"2024-01-26T23:30:53.152375+0000","last_unstale":"2024-01-26T23:30:53.152696+0000","last_undegraded":"2024-01-26T23:30:53.152696+0000","last_fullsized":"2024-01-26T23:30:53.152696+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:50.289763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.153233+0000","last_change":"2024-01-26T23:30:53.153233+0000","last_active":"2024-01-26T23:30:53.153233+0000","last_peered":"2024-01-26T23:30:53.153233+0000","last_clean":"2024-01-26T23:30:53.153233+0000","last_became_active":"2024-01-26T23:30:53.152926+0000","last_became_peered":"2024-01-26T23:30:53.152926+0000","last_unstale":"2024-01-26T23:30:53.153233+0000","last_undegraded":"2024-01-26T23:30:53.153233+0000","last_fullsized":"2024-01-26T23:30:53.153233+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:35:06.371452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150792+0000","last_change":"2024-01-26T23:30:53.150792+0000","last_active":"2024-01-26T23:30:53.150792+0000","last_peered":"2024-01-26T23:30:53.150792+0000","last_clean":"2024-01-26T23:30:53.150792+0000","last_became_active":"2024-01-26T23:30:53.150192+0000","last_became_peered":"2024-01-26T23:30:53.150192+0000","last_unstale":"2024-01-26T23:30:53.150792+0000","last_undegraded":"2024-01-26T23:30:53.150792+0000","last_fullsized":"2024-01-26T23:30:53.150792+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:21:28.076604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.136695+0000","last_change":"2024-01-26T23:30:53.136695+0000","last_active":"2024-01-26T23:30:53.136695+0000","last_peered":"2024-01-26T23:30:53.136695+0000","last_clean":"2024-01-26T23:30:53.136695+0000","last_became_active":"2024-01-26T23:30:53.136370+0000","last_became_peered":"2024-01-26T23:30:53.136370+0000","last_unstale":"2024-01-26T23:30:53.136695+0000","last_undegraded":"2024-01-26T23:30:53.136695+0000","last_fullsized":"2024-01-26T23:30:53.136695+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:42:40.534420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150290+0000","last_change":"2024-01-26T23:30:53.150290+0000","last_active":"2024-01-26T23:30:53.150290+0000","last_peered":"2024-01-26T23:30:53.150290+0000","last_clean":"2024-01-26T23:30:53.150290+0000","last_became_active":"2024-01-26T23:30:53.149997+0000","last_became_peered":"2024-01-26T23:30:53.149997+0000","last_unstale":"2024-01-26T23:30:53.150290+0000","last_undegraded":"2024-01-26T23:30:53.150290+0000","last_fullsized":"2024-01-26T23:30:53.150290+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:00:27.950751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.149648+0000","last_change":"2024-01-26T23:30:53.149648+0000","last_active":"2024-01-26T23:30:53.149648+0000","last_peered":"2024-01-26T23:30:53.149648+0000","last_clean":"2024-01-26T23:30:53.149648+0000","last_became_active":"2024-01-26T23:30:53.149300+0000","last_became_peered":"2024-01-26T23:30:53.149300+0000","last_unstale":"2024-01-26T23:30:53.149648+0000","last_undegraded":"2024-01-26T23:30:53.149648+0000","last_fullsized":"2024-01-26T23:30:53.149648+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:54:59.331467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150897+0000","last_change":"2024-01-26T23:30:53.150897+0000","last_active":"2024-01-26T23:30:53.150897+0000","last_peered":"2024-01-26T23:30:53.150897+0000","last_clean":"2024-01-26T23:30:53.150897+0000","last_became_active":"2024-01-26T23:30:53.150634+0000","last_became_peered":"2024-01-26T23:30:53.150634+0000","last_unstale":"2024-01-26T23:30:53.150897+0000","last_undegraded":"2024-01-26T23:30:53.150897+0000","last_fullsized":"2024-01-26T23:30:53.150897+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:54:59.331467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.154888+0000","last_change":"2024-01-26T23:30:53.154888+0000","last_active":"2024-01-26T23:30:53.154888+0000","last_peered":"2024-01-26T23:30:53.154888+0000","last_clean":"2024-01-26T23:30:53.154888+0000","last_became_active":"2024-01-26T23:30:53.154417+0000","last_became_peered":"2024-01-26T23:30:53.154417+0000","last_unstale":"2024-01-26T23:30:53.154888+0000","last_undegraded":"2024-01-26T23:30:53.154888+0000","last_fullsized":"2024-01-26T23:30:53.154888+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:46:41.384689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.154208+0000","last_change":"2024-01-26T23:30:53.154208+0000","last_active":"2024-01-26T23:30:53.154208+0000","last_peered":"2024-01-26T23:30:53.154208+0000","last_clean":"2024-01-26T23:30:53.154208+0000","last_became_active":"2024-01-26T23:30:53.153879+0000","last_became_peered":"2024-01-26T23:30:53.153879+0000","last_unstale":"2024-01-26T23:30:53.154208+0000","last_undegraded":"2024-01-26T23:30:53.154208+0000","last_fullsized":"2024-01-26T23:30:53.154208+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:13:09.311848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.149050+0000","last_change":"2024-01-26T23:30:53.149050+0000","last_active":"2024-01-26T23:30:53.149050+0000","last_peered":"2024-01-26T23:30:53.149050+0000","last_clean":"2024-01-26T23:30:53.149050+0000","last_became_active":"2024-01-26T23:30:53.148793+0000","last_became_peered":"2024-01-26T23:30:53.148793+0000","last_unstale":"2024-01-26T23:30:53.149050+0000","last_undegraded":"2024-01-26T23:30:53.149050+0000","last_fullsized":"2024-01-26T23:30:53.149050+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:40:59.362637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.142059+0000","last_change":"2024-01-26T23:30:53.142059+0000","last_active":"2024-01-26T23:30:53.142059+0000","last_peered":"2024-01-26T23:30:53.142059+0000","last_clean":"2024-01-26T23:30:53.142059+0000","last_became_active":"2024-01-26T23:30:53.141752+0000","last_became_peered":"2024-01-26T23:30:53.141752+0000","last_unstale":"2024-01-26T23:30:53.142059+0000","last_undegraded":"2024-01-26T23:30:53.142059+0000","last_fullsized":"2024-01-26T23:30:53.142059+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:31:05.339820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.149345+0000","last_change":"2024-01-26T23:30:53.149345+0000","last_active":"2024-01-26T23:30:53.149345+0000","last_peered":"2024-01-26T23:30:53.149345+0000","last_clean":"2024-01-26T23:30:53.149345+0000","last_became_active":"2024-01-26T23:30:53.149079+0000","last_became_peered":"2024-01-26T23:30:53.149079+0000","last_unstale":"2024-01-26T23:30:53.149345+0000","last_undegraded":"2024-01-26T23:30:53.149345+0000","last_fullsized":"2024-01-26T23:30:53.149345+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:28.464237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.149770+0000","last_change":"2024-01-26T23:30:53.149770+0000","last_active":"2024-01-26T23:30:53.149770+0000","last_peered":"2024-01-26T23:30:53.149770+0000","last_clean":"2024-01-26T23:30:53.149770+0000","last_became_active":"2024-01-26T23:30:53.149458+0000","last_became_peered":"2024-01-26T23:30:53.149458+0000","last_unstale":"2024-01-26T23:30:53.149770+0000","last_undegraded":"2024-01-26T23:30:53.149770+0000","last_fullsized":"2024-01-26T23:30:53.149770+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:19:47.941469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.138848+0000","last_change":"2024-01-26T23:30:53.138848+0000","last_active":"2024-01-26T23:30:53.138848+0000","last_peered":"2024-01-26T23:30:53.138848+0000","last_clean":"2024-01-26T23:30:53.138848+0000","last_became_active":"2024-01-26T23:30:53.138555+0000","last_became_peered":"2024-01-26T23:30:53.138555+0000","last_unstale":"2024-01-26T23:30:53.138848+0000","last_undegraded":"2024-01-26T23:30:53.138848+0000","last_fullsized":"2024-01-26T23:30:53.138848+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:00:27.950751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.151596+0000","last_change":"2024-01-26T23:30:53.151596+0000","last_active":"2024-01-26T23:30:53.151596+0000","last_peered":"2024-01-26T23:30:53.151596+0000","last_clean":"2024-01-26T23:30:53.151596+0000","last_became_active":"2024-01-26T23:30:53.151183+0000","last_became_peered":"2024-01-26T23:30:53.151183+0000","last_unstale":"2024-01-26T23:30:53.151596+0000","last_undegraded":"2024-01-26T23:30:53.151596+0000","last_fullsized":"2024-01-26T23:30:53.151596+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:14.479539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.137323+0000","last_change":"2024-01-26T23:30:53.137323+0000","last_active":"2024-01-26T23:30:53.137323+0000","last_peered":"2024-01-26T23:30:53.137323+0000","last_clean":"2024-01-26T23:30:53.137323+0000","last_became_active":"2024-01-26T23:30:53.137046+0000","last_became_peered":"2024-01-26T23:30:53.137046+0000","last_unstale":"2024-01-26T23:30:53.137323+0000","last_undegraded":"2024-01-26T23:30:53.137323+0000","last_fullsized":"2024-01-26T23:30:53.137323+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:34:35.003101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.144330+0000","last_change":"2024-01-26T23:30:53.144330+0000","last_active":"2024-01-26T23:30:53.144330+0000","last_peered":"2024-01-26T23:30:53.144330+0000","last_clean":"2024-01-26T23:30:53.144330+0000","last_became_active":"2024-01-26T23:30:53.143985+0000","last_became_peered":"2024-01-26T23:30:53.143985+0000","last_unstale":"2024-01-26T23:30:53.144330+0000","last_undegraded":"2024-01-26T23:30:53.144330+0000","last_fullsized":"2024-01-26T23:30:53.144330+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:01:39.008256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.155012+0000","last_change":"2024-01-26T23:30:53.155012+0000","last_active":"2024-01-26T23:30:53.155012+0000","last_peered":"2024-01-26T23:30:53.155012+0000","last_clean":"2024-01-26T23:30:53.155012+0000","last_became_active":"2024-01-26T23:30:53.154728+0000","last_became_peered":"2024-01-26T23:30:53.154728+0000","last_unstale":"2024-01-26T23:30:53.155012+0000","last_undegraded":"2024-01-26T23:30:53.155012+0000","last_fullsized":"2024-01-26T23:30:53.155012+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:06:00.135263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.145200+0000","last_change":"2024-01-26T23:30:53.145200+0000","last_active":"2024-01-26T23:30:53.145200+0000","last_peered":"2024-01-26T23:30:53.145200+0000","last_clean":"2024-01-26T23:30:53.145200+0000","last_became_active":"2024-01-26T23:30:53.144929+0000","last_became_peered":"2024-01-26T23:30:53.144929+0000","last_unstale":"2024-01-26T23:30:53.145200+0000","last_undegraded":"2024-01-26T23:30:53.145200+0000","last_fullsized":"2024-01-26T23:30:53.145200+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:41:52.600539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.148496+0000","last_change":"2024-01-26T23:30:53.148496+0000","last_active":"2024-01-26T23:30:53.148496+0000","last_peered":"2024-01-26T23:30:53.148496+0000","last_clean":"2024-01-26T23:30:53.148496+0000","last_became_active":"2024-01-26T23:30:53.148178+0000","last_became_peered":"2024-01-26T23:30:53.148178+0000","last_unstale":"2024-01-26T23:30:53.148496+0000","last_undegraded":"2024-01-26T23:30:53.148496+0000","last_fullsized":"2024-01-26T23:30:53.148496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:06:09.705120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.144635+0000","last_change":"2024-01-26T23:30:53.144635+0000","last_active":"2024-01-26T23:30:53.144635+0000","last_peered":"2024-01-26T23:30:53.144635+0000","last_clean":"2024-01-26T23:30:53.144635+0000","last_became_active":"2024-01-26T23:30:53.144369+0000","last_became_peered":"2024-01-26T23:30:53.144369+0000","last_unstale":"2024-01-26T23:30:53.144635+0000","last_undegraded":"2024-01-26T23:30:53.144635+0000","last_fullsized":"2024-01-26T23:30:53.144635+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:59.531509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.144551+0000","last_change":"2024-01-26T23:30:53.144551+0000","last_active":"2024-01-26T23:30:53.144551+0000","last_peered":"2024-01-26T23:30:53.144551+0000","last_clean":"2024-01-26T23:30:53.144551+0000","last_became_active":"2024-01-26T23:30:53.144137+0000","last_became_peered":"2024-01-26T23:30:53.144137+0000","last_unstale":"2024-01-26T23:30:53.144551+0000","last_undegraded":"2024-01-26T23:30:53.144551+0000","last_fullsized":"2024-01-26T23:30:53.144551+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:42:21.954480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.151420+0000","last_change":"2024-01-26T23:30:53.151420+0000","last_active":"2024-01-26T23:30:53.151420+0000","last_peered":"2024-01-26T23:30:53.151420+0000","last_clean":"2024-01-26T23:30:53.151420+0000","last_became_active":"2024-01-26T23:30:53.150809+0000","last_became_peered":"2024-01-26T23:30:53.150809+0000","last_unstale":"2024-01-26T23:30:53.151420+0000","last_undegraded":"2024-01-26T23:30:53.151420+0000","last_fullsized":"2024-01-26T23:30:53.151420+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:06:00.135263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150644+0000","last_change":"2024-01-26T23:30:53.150644+0000","last_active":"2024-01-26T23:30:53.150644+0000","last_peered":"2024-01-26T23:30:53.150644+0000","last_clean":"2024-01-26T23:30:53.150644+0000","last_became_active":"2024-01-26T23:30:53.150264+0000","last_became_peered":"2024-01-26T23:30:53.150264+0000","last_unstale":"2024-01-26T23:30:53.150644+0000","last_undegraded":"2024-01-26T23:30:53.150644+0000","last_fullsized":"2024-01-26T23:30:53.150644+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:35:06.371452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.155634+0000","last_change":"2024-01-26T23:30:53.155634+0000","last_active":"2024-01-26T23:30:53.155634+0000","last_peered":"2024-01-26T23:30:53.155634+0000","last_clean":"2024-01-26T23:30:53.155634+0000","last_became_active":"2024-01-26T23:30:53.155141+0000","last_became_peered":"2024-01-26T23:30:53.155141+0000","last_unstale":"2024-01-26T23:30:53.155634+0000","last_undegraded":"2024-01-26T23:30:53.155634+0000","last_fullsized":"2024-01-26T23:30:53.155634+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:41:37.013476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.163776+0000","last_change":"2024-01-26T23:30:52.135994+0000","last_active":"2024-01-26T23:30:53.163776+0000","last_peered":"2024-01-26T23:30:53.163776+0000","last_clean":"2024-01-26T23:30:53.163776+0000","last_became_active":"2024-01-26T23:30:52.135709+0000","last_became_peered":"2024-01-26T23:30:52.135709+0000","last_unstale":"2024-01-26T23:30:53.163776+0000","last_undegraded":"2024-01-26T23:30:53.163776+0000","last_fullsized":"2024-01-26T23:30:53.163776+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:31:04.332740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.158297+0000","last_change":"2024-01-26T23:30:53.158297+0000","last_active":"2024-01-26T23:30:53.158297+0000","last_peered":"2024-01-26T23:30:53.158297+0000","last_clean":"2024-01-26T23:30:53.158297+0000","last_became_active":"2024-01-26T23:30:53.158014+0000","last_became_peered":"2024-01-26T23:30:53.158014+0000","last_unstale":"2024-01-26T23:30:53.158297+0000","last_undegraded":"2024-01-26T23:30:53.158297+0000","last_fullsized":"2024-01-26T23:30:53.158297+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:35:46.571656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.129214+0000","last_change":"2024-01-26T23:30:52.133411+0000","last_active":"2024-01-26T23:30:53.129214+0000","last_peered":"2024-01-26T23:30:53.129214+0000","last_clean":"2024-01-26T23:30:53.129214+0000","last_became_active":"2024-01-26T23:30:52.132169+0000","last_became_peered":"2024-01-26T23:30:52.132169+0000","last_unstale":"2024-01-26T23:30:53.129214+0000","last_undegraded":"2024-01-26T23:30:53.129214+0000","last_fullsized":"2024-01-26T23:30:53.129214+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:05:59.128183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.149529+0000","last_change":"2024-01-26T23:30:53.149529+0000","last_active":"2024-01-26T23:30:53.149529+0000","last_peered":"2024-01-26T23:30:53.149529+0000","last_clean":"2024-01-26T23:30:53.149529+0000","last_became_active":"2024-01-26T23:30:53.149235+0000","last_became_peered":"2024-01-26T23:30:53.149235+0000","last_unstale":"2024-01-26T23:30:53.149529+0000","last_undegraded":"2024-01-26T23:30:53.149529+0000","last_fullsized":"2024-01-26T23:30:53.149529+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:50.289763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.163961+0000","last_change":"2024-01-26T23:30:52.139695+0000","last_active":"2024-01-26T23:30:53.163961+0000","last_peered":"2024-01-26T23:30:53.163961+0000","last_clean":"2024-01-26T23:30:53.163961+0000","last_became_active":"2024-01-26T23:30:52.139478+0000","last_became_peered":"2024-01-26T23:30:52.139478+0000","last_unstale":"2024-01-26T23:30:53.163961+0000","last_undegraded":"2024-01-26T23:30:53.163961+0000","last_fullsized":"2024-01-26T23:30:53.163961+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:01:38.001177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.148939+0000","last_change":"2024-01-26T23:30:53.148939+0000","last_active":"2024-01-26T23:30:53.148939+0000","last_peered":"2024-01-26T23:30:53.148939+0000","last_clean":"2024-01-26T23:30:53.148939+0000","last_became_active":"2024-01-26T23:30:53.147452+0000","last_became_peered":"2024-01-26T23:30:53.147452+0000","last_unstale":"2024-01-26T23:30:53.148939+0000","last_undegraded":"2024-01-26T23:30:53.148939+0000","last_fullsized":"2024-01-26T23:30:53.148939+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:19:47.941469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.163553+0000","last_change":"2024-01-26T23:30:52.140374+0000","last_active":"2024-01-26T23:30:53.163553+0000","last_peered":"2024-01-26T23:30:53.163553+0000","last_clean":"2024-01-26T23:30:53.163553+0000","last_became_active":"2024-01-26T23:30:52.139872+0000","last_became_peered":"2024-01-26T23:30:52.139872+0000","last_unstale":"2024-01-26T23:30:53.163553+0000","last_undegraded":"2024-01-26T23:30:53.163553+0000","last_fullsized":"2024-01-26T23:30:53.163553+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:49.828311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.140574+0000","last_change":"2024-01-26T23:30:53.140574+0000","last_active":"2024-01-26T23:30:53.140574+0000","last_peered":"2024-01-26T23:30:53.140574+0000","last_clean":"2024-01-26T23:30:53.140574+0000","last_became_active":"2024-01-26T23:30:53.140236+0000","last_became_peered":"2024-01-26T23:30:53.140236+0000","last_unstale":"2024-01-26T23:30:53.140574+0000","last_undegraded":"2024-01-26T23:30:53.140574+0000","last_fullsized":"2024-01-26T23:30:53.140574+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:50.781140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.143641+0000","last_change":"2024-01-26T23:30:52.133011+0000","last_active":"2024-01-26T23:30:53.143641+0000","last_peered":"2024-01-26T23:30:53.143641+0000","last_clean":"2024-01-26T23:30:53.143641+0000","last_became_active":"2024-01-26T23:30:52.132774+0000","last_became_peered":"2024-01-26T23:30:52.132774+0000","last_unstale":"2024-01-26T23:30:53.143641+0000","last_undegraded":"2024-01-26T23:30:53.143641+0000","last_fullsized":"2024-01-26T23:30:53.143641+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:52:45.442327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.153343+0000","last_change":"2024-01-26T23:30:53.153343+0000","last_active":"2024-01-26T23:30:53.153343+0000","last_peered":"2024-01-26T23:30:53.153343+0000","last_clean":"2024-01-26T23:30:53.153343+0000","last_became_active":"2024-01-26T23:30:53.153079+0000","last_became_peered":"2024-01-26T23:30:53.153079+0000","last_unstale":"2024-01-26T23:30:53.153343+0000","last_undegraded":"2024-01-26T23:30:53.153343+0000","last_fullsized":"2024-01-26T23:30:53.153343+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:47:22.375886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.131359+0000","last_change":"2024-01-26T23:30:52.136591+0000","last_active":"2024-01-26T23:30:53.131359+0000","last_peered":"2024-01-26T23:30:53.131359+0000","last_clean":"2024-01-26T23:30:53.131359+0000","last_became_active":"2024-01-26T23:30:52.136290+0000","last_became_peered":"2024-01-26T23:30:52.136290+0000","last_unstale":"2024-01-26T23:30:53.131359+0000","last_undegraded":"2024-01-26T23:30:53.131359+0000","last_fullsized":"2024-01-26T23:30:53.131359+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:48:20.762731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.154657+0000","last_change":"2024-01-26T23:30:53.154657+0000","last_active":"2024-01-26T23:30:53.154657+0000","last_peered":"2024-01-26T23:30:53.154657+0000","last_clean":"2024-01-26T23:30:53.154657+0000","last_became_active":"2024-01-26T23:30:53.154321+0000","last_became_peered":"2024-01-26T23:30:53.154321+0000","last_unstale":"2024-01-26T23:30:53.154657+0000","last_undegraded":"2024-01-26T23:30:53.154657+0000","last_fullsized":"2024-01-26T23:30:53.154657+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:47:22.375886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.138623+0000","last_change":"2024-01-26T23:30:52.137622+0000","last_active":"2024-01-26T23:30:52.138623+0000","last_peered":"2024-01-26T23:30:52.138623+0000","last_clean":"2024-01-26T23:30:52.138623+0000","last_became_active":"2024-01-26T23:30:52.137276+0000","last_became_peered":"2024-01-26T23:30:52.137276+0000","last_unstale":"2024-01-26T23:30:52.138623+0000","last_undegraded":"2024-01-26T23:30:52.138623+0000","last_fullsized":"2024-01-26T23:30:52.138623+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:42:20.947400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.139415+0000","last_change":"2024-01-26T23:30:53.139415+0000","last_active":"2024-01-26T23:30:53.139415+0000","last_peered":"2024-01-26T23:30:53.139415+0000","last_clean":"2024-01-26T23:30:53.139415+0000","last_became_active":"2024-01-26T23:30:53.138522+0000","last_became_peered":"2024-01-26T23:30:53.138522+0000","last_unstale":"2024-01-26T23:30:53.139415+0000","last_undegraded":"2024-01-26T23:30:53.139415+0000","last_fullsized":"2024-01-26T23:30:53.139415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:17:39.550438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.133686+0000","last_change":"2024-01-26T23:30:52.133105+0000","last_active":"2024-01-26T23:30:52.133686+0000","last_peered":"2024-01-26T23:30:52.133686+0000","last_clean":"2024-01-26T23:30:52.133686+0000","last_became_active":"2024-01-26T23:30:52.132828+0000","last_became_peered":"2024-01-26T23:30:52.132828+0000","last_unstale":"2024-01-26T23:30:52.133686+0000","last_undegraded":"2024-01-26T23:30:52.133686+0000","last_fullsized":"2024-01-26T23:30:52.133686+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:41:36.006396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150321+0000","last_change":"2024-01-26T23:30:53.150321+0000","last_active":"2024-01-26T23:30:53.150321+0000","last_peered":"2024-01-26T23:30:53.150321+0000","last_clean":"2024-01-26T23:30:53.150321+0000","last_became_active":"2024-01-26T23:30:53.149935+0000","last_became_peered":"2024-01-26T23:30:53.149935+0000","last_unstale":"2024-01-26T23:30:53.150321+0000","last_undegraded":"2024-01-26T23:30:53.150321+0000","last_fullsized":"2024-01-26T23:30:53.150321+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:55:40.719163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.139027+0000","last_change":"2024-01-26T23:30:52.130243+0000","last_active":"2024-01-26T23:30:52.139027+0000","last_peered":"2024-01-26T23:30:52.139027+0000","last_clean":"2024-01-26T23:30:52.139027+0000","last_became_active":"2024-01-26T23:30:52.129945+0000","last_became_peered":"2024-01-26T23:30:52.129945+0000","last_unstale":"2024-01-26T23:30:52.139027+0000","last_undegraded":"2024-01-26T23:30:52.139027+0000","last_fullsized":"2024-01-26T23:30:52.139027+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:13:08.304768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.146393+0000","last_change":"2024-01-26T23:30:53.146393+0000","last_active":"2024-01-26T23:30:53.146393+0000","last_peered":"2024-01-26T23:30:53.146393+0000","last_clean":"2024-01-26T23:30:53.146393+0000","last_became_active":"2024-01-26T23:30:53.146095+0000","last_became_peered":"2024-01-26T23:30:53.146095+0000","last_unstale":"2024-01-26T23:30:53.146393+0000","last_undegraded":"2024-01-26T23:30:53.146393+0000","last_fullsized":"2024-01-26T23:30:53.146393+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:28.464237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.140804+0000","last_change":"2024-01-26T23:30:52.139866+0000","last_active":"2024-01-26T23:30:52.140804+0000","last_peered":"2024-01-26T23:30:52.140804+0000","last_clean":"2024-01-26T23:30:52.140804+0000","last_became_active":"2024-01-26T23:30:52.139621+0000","last_became_peered":"2024-01-26T23:30:52.139621+0000","last_unstale":"2024-01-26T23:30:52.140804+0000","last_undegraded":"2024-01-26T23:30:52.140804+0000","last_fullsized":"2024-01-26T23:30:52.140804+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:40:58.355558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.140424+0000","last_change":"2024-01-26T23:30:53.140424+0000","last_active":"2024-01-26T23:30:53.140424+0000","last_peered":"2024-01-26T23:30:53.140424+0000","last_clean":"2024-01-26T23:30:53.140424+0000","last_became_active":"2024-01-26T23:30:53.140116+0000","last_became_peered":"2024-01-26T23:30:53.140116+0000","last_unstale":"2024-01-26T23:30:53.140424+0000","last_undegraded":"2024-01-26T23:30:53.140424+0000","last_fullsized":"2024-01-26T23:30:53.140424+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:19:47.941469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.163238+0000","last_change":"2024-01-26T23:30:52.130204+0000","last_active":"2024-01-26T23:30:53.163238+0000","last_peered":"2024-01-26T23:30:53.163238+0000","last_clean":"2024-01-26T23:30:53.163238+0000","last_became_active":"2024-01-26T23:30:52.129898+0000","last_became_peered":"2024-01-26T23:30:52.129898+0000","last_unstale":"2024-01-26T23:30:53.163238+0000","last_undegraded":"2024-01-26T23:30:53.163238+0000","last_fullsized":"2024-01-26T23:30:53.163238+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:58.524429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.137432+0000","last_change":"2024-01-26T23:30:53.137432+0000","last_active":"2024-01-26T23:30:53.137432+0000","last_peered":"2024-01-26T23:30:53.137432+0000","last_clean":"2024-01-26T23:30:53.137432+0000","last_became_active":"2024-01-26T23:30:53.137111+0000","last_became_peered":"2024-01-26T23:30:53.137111+0000","last_unstale":"2024-01-26T23:30:53.137432+0000","last_undegraded":"2024-01-26T23:30:53.137432+0000","last_fullsized":"2024-01-26T23:30:53.137432+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:19:47.941469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.165407+0000","last_change":"2024-01-26T23:30:52.135583+0000","last_active":"2024-01-26T23:30:53.165407+0000","last_peered":"2024-01-26T23:30:53.165407+0000","last_clean":"2024-01-26T23:30:53.165407+0000","last_became_active":"2024-01-26T23:30:52.135252+0000","last_became_peered":"2024-01-26T23:30:52.135252+0000","last_unstale":"2024-01-26T23:30:53.165407+0000","last_undegraded":"2024-01-26T23:30:53.165407+0000","last_fullsized":"2024-01-26T23:30:53.165407+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:46:40.377610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.152729+0000","last_change":"2024-01-26T23:30:53.152729+0000","last_active":"2024-01-26T23:30:53.152729+0000","last_peered":"2024-01-26T23:30:53.152729+0000","last_clean":"2024-01-26T23:30:53.152729+0000","last_became_active":"2024-01-26T23:30:53.152486+0000","last_became_peered":"2024-01-26T23:30:53.152486+0000","last_unstale":"2024-01-26T23:30:53.152729+0000","last_undegraded":"2024-01-26T23:30:53.152729+0000","last_fullsized":"2024-01-26T23:30:53.152729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:53:34.849263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.164536+0000","last_change":"2024-01-26T23:30:52.134887+0000","last_active":"2024-01-26T23:30:53.164536+0000","last_peered":"2024-01-26T23:30:53.164536+0000","last_clean":"2024-01-26T23:30:53.164536+0000","last_became_active":"2024-01-26T23:30:52.134395+0000","last_became_peered":"2024-01-26T23:30:52.134395+0000","last_unstale":"2024-01-26T23:30:53.164536+0000","last_undegraded":"2024-01-26T23:30:53.164536+0000","last_fullsized":"2024-01-26T23:30:53.164536+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:41:36.006396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.147976+0000","last_change":"2024-01-26T23:30:53.147976+0000","last_active":"2024-01-26T23:30:53.147976+0000","last_peered":"2024-01-26T23:30:53.147976+0000","last_clean":"2024-01-26T23:30:53.147976+0000","last_became_active":"2024-01-26T23:30:53.147639+0000","last_became_peered":"2024-01-26T23:30:53.147639+0000","last_unstale":"2024-01-26T23:30:53.147976+0000","last_undegraded":"2024-01-26T23:30:53.147976+0000","last_fullsized":"2024-01-26T23:30:53.147976+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:53:34.849263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.138584+0000","last_change":"2024-01-26T23:30:52.137142+0000","last_active":"2024-01-26T23:30:52.138584+0000","last_peered":"2024-01-26T23:30:52.138584+0000","last_clean":"2024-01-26T23:30:52.138584+0000","last_became_active":"2024-01-26T23:30:52.136803+0000","last_became_peered":"2024-01-26T23:30:52.136803+0000","last_unstale":"2024-01-26T23:30:52.138584+0000","last_undegraded":"2024-01-26T23:30:52.138584+0000","last_fullsized":"2024-01-26T23:30:52.138584+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:58.524429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.149287+0000","last_change":"2024-01-26T23:30:53.149287+0000","last_active":"2024-01-26T23:30:53.149287+0000","last_peered":"2024-01-26T23:30:53.149287+0000","last_clean":"2024-01-26T23:30:53.149287+0000","last_became_active":"2024-01-26T23:30:53.149018+0000","last_became_peered":"2024-01-26T23:30:53.149018+0000","last_unstale":"2024-01-26T23:30:53.149287+0000","last_undegraded":"2024-01-26T23:30:53.149287+0000","last_fullsized":"2024-01-26T23:30:53.149287+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:35:46.571656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.139804+0000","last_change":"2024-01-26T23:30:52.136062+0000","last_active":"2024-01-26T23:30:52.139804+0000","last_peered":"2024-01-26T23:30:52.139804+0000","last_clean":"2024-01-26T23:30:52.139804+0000","last_became_active":"2024-01-26T23:30:52.135799+0000","last_became_peered":"2024-01-26T23:30:52.135799+0000","last_unstale":"2024-01-26T23:30:52.139804+0000","last_undegraded":"2024-01-26T23:30:52.139804+0000","last_fullsized":"2024-01-26T23:30:52.139804+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:16:19.376392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.148268+0000","last_change":"2024-01-26T23:30:53.148268+0000","last_active":"2024-01-26T23:30:53.148268+0000","last_peered":"2024-01-26T23:30:53.148268+0000","last_clean":"2024-01-26T23:30:53.148268+0000","last_became_active":"2024-01-26T23:30:53.147961+0000","last_became_peered":"2024-01-26T23:30:53.147961+0000","last_unstale":"2024-01-26T23:30:53.148268+0000","last_undegraded":"2024-01-26T23:30:53.148268+0000","last_fullsized":"2024-01-26T23:30:53.148268+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:06:09.705120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":17,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.163523+0000","last_change":"2024-01-26T23:30:52.134957+0000","last_active":"2024-01-26T23:30:53.163523+0000","last_peered":"2024-01-26T23:30:53.163523+0000","last_clean":"2024-01-26T23:30:53.163523+0000","last_became_active":"2024-01-26T23:30:52.134683+0000","last_became_peered":"2024-01-26T23:30:52.134683+0000","last_unstale":"2024-01-26T23:30:53.163523+0000","last_undegraded":"2024-01-26T23:30:53.163523+0000","last_fullsized":"2024-01-26T23:30:53.163523+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:34:33.996022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.147505+0000","last_change":"2024-01-26T23:30:53.147505+0000","last_active":"2024-01-26T23:30:53.147505+0000","last_peered":"2024-01-26T23:30:53.147505+0000","last_clean":"2024-01-26T23:30:53.147505+0000","last_became_active":"2024-01-26T23:30:53.146950+0000","last_became_peered":"2024-01-26T23:30:53.146950+0000","last_unstale":"2024-01-26T23:30:53.147505+0000","last_undegraded":"2024-01-26T23:30:53.147505+0000","last_fullsized":"2024-01-26T23:30:53.147505+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:05:11.628582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.145542+0000","last_change":"2024-01-26T23:30:52.134426+0000","last_active":"2024-01-26T23:30:53.145542+0000","last_peered":"2024-01-26T23:30:53.145542+0000","last_clean":"2024-01-26T23:30:53.145542+0000","last_became_active":"2024-01-26T23:30:52.134181+0000","last_became_peered":"2024-01-26T23:30:52.134181+0000","last_unstale":"2024-01-26T23:30:53.145542+0000","last_undegraded":"2024-01-26T23:30:53.145542+0000","last_fullsized":"2024-01-26T23:30:53.145542+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:01:38.001177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150982+0000","last_change":"2024-01-26T23:30:53.150982+0000","last_active":"2024-01-26T23:30:53.150982+0000","last_peered":"2024-01-26T23:30:53.150982+0000","last_clean":"2024-01-26T23:30:53.150982+0000","last_became_active":"2024-01-26T23:30:53.150541+0000","last_became_peered":"2024-01-26T23:30:53.150541+0000","last_unstale":"2024-01-26T23:30:53.150982+0000","last_undegraded":"2024-01-26T23:30:53.150982+0000","last_fullsized":"2024-01-26T23:30:53.150982+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:42:40.534420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.163598+0000","last_change":"2024-01-26T23:30:52.137001+0000","last_active":"2024-01-26T23:30:53.163598+0000","last_peered":"2024-01-26T23:30:53.163598+0000","last_clean":"2024-01-26T23:30:53.163598+0000","last_became_active":"2024-01-26T23:30:52.136689+0000","last_became_peered":"2024-01-26T23:30:52.136689+0000","last_unstale":"2024-01-26T23:30:53.163598+0000","last_undegraded":"2024-01-26T23:30:53.163598+0000","last_fullsized":"2024-01-26T23:30:53.163598+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:13.472460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.143475+0000","last_change":"2024-01-26T23:30:53.143475+0000","last_active":"2024-01-26T23:30:53.143475+0000","last_peered":"2024-01-26T23:30:53.143475+0000","last_clean":"2024-01-26T23:30:53.143475+0000","last_became_active":"2024-01-26T23:30:53.143203+0000","last_became_peered":"2024-01-26T23:30:53.143203+0000","last_unstale":"2024-01-26T23:30:53.143475+0000","last_undegraded":"2024-01-26T23:30:53.143475+0000","last_fullsized":"2024-01-26T23:30:53.143475+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:42:40.534420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.131127+0000","last_change":"2024-01-26T23:30:52.135188+0000","last_active":"2024-01-26T23:30:53.131127+0000","last_peered":"2024-01-26T23:30:53.131127+0000","last_clean":"2024-01-26T23:30:53.131127+0000","last_became_active":"2024-01-26T23:30:52.134868+0000","last_became_peered":"2024-01-26T23:30:52.134868+0000","last_unstale":"2024-01-26T23:30:53.131127+0000","last_undegraded":"2024-01-26T23:30:53.131127+0000","last_fullsized":"2024-01-26T23:30:53.131127+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:51:32.360927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.134504+0000","last_change":"2024-01-26T23:30:53.134504+0000","last_active":"2024-01-26T23:30:53.134504+0000","last_peered":"2024-01-26T23:30:53.134504+0000","last_clean":"2024-01-26T23:30:53.134504+0000","last_became_active":"2024-01-26T23:30:53.134064+0000","last_became_peered":"2024-01-26T23:30:53.134064+0000","last_unstale":"2024-01-26T23:30:53.134504+0000","last_undegraded":"2024-01-26T23:30:53.134504+0000","last_fullsized":"2024-01-26T23:30:53.134504+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:25:38.452947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.141259+0000","last_change":"2024-01-26T23:30:52.139344+0000","last_active":"2024-01-26T23:30:52.141259+0000","last_peered":"2024-01-26T23:30:52.141259+0000","last_clean":"2024-01-26T23:30:52.141259+0000","last_became_active":"2024-01-26T23:30:52.139058+0000","last_became_peered":"2024-01-26T23:30:52.139058+0000","last_unstale":"2024-01-26T23:30:52.141259+0000","last_undegraded":"2024-01-26T23:30:52.141259+0000","last_fullsized":"2024-01-26T23:30:52.141259+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:41:51.593459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.150669+0000","last_change":"2024-01-26T23:30:53.150669+0000","last_active":"2024-01-26T23:30:53.150669+0000","last_peered":"2024-01-26T23:30:53.150669+0000","last_clean":"2024-01-26T23:30:53.150669+0000","last_became_active":"2024-01-26T23:30:53.150389+0000","last_became_peered":"2024-01-26T23:30:53.150389+0000","last_unstale":"2024-01-26T23:30:53.150669+0000","last_undegraded":"2024-01-26T23:30:53.150669+0000","last_fullsized":"2024-01-26T23:30:53.150669+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:50.835391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.162023+0000","last_change":"2024-01-26T23:30:52.127727+0000","last_active":"2024-01-26T23:30:53.162023+0000","last_peered":"2024-01-26T23:30:53.162023+0000","last_clean":"2024-01-26T23:30:53.162023+0000","last_became_active":"2024-01-26T23:30:52.127444+0000","last_became_peered":"2024-01-26T23:30:52.127444+0000","last_unstale":"2024-01-26T23:30:53.162023+0000","last_undegraded":"2024-01-26T23:30:53.162023+0000","last_fullsized":"2024-01-26T23:30:53.162023+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:15:51.865289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.137131+0000","last_change":"2024-01-26T23:30:53.137131+0000","last_active":"2024-01-26T23:30:53.137131+0000","last_peered":"2024-01-26T23:30:53.137131+0000","last_clean":"2024-01-26T23:30:53.137131+0000","last_became_active":"2024-01-26T23:30:53.136869+0000","last_became_peered":"2024-01-26T23:30:53.136869+0000","last_unstale":"2024-01-26T23:30:53.137131+0000","last_undegraded":"2024-01-26T23:30:53.137131+0000","last_fullsized":"2024-01-26T23:30:53.137131+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:00:27.950751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.136366+0000","last_change":"2024-01-26T23:30:52.134140+0000","last_active":"2024-01-26T23:30:52.136366+0000","last_peered":"2024-01-26T23:30:52.136366+0000","last_clean":"2024-01-26T23:30:52.136366+0000","last_became_active":"2024-01-26T23:30:52.133838+0000","last_became_peered":"2024-01-26T23:30:52.133838+0000","last_unstale":"2024-01-26T23:30:52.136366+0000","last_undegraded":"2024-01-26T23:30:52.136366+0000","last_fullsized":"2024-01-26T23:30:52.136366+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:53:04.923997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.149915+0000","last_change":"2024-01-26T23:30:53.149915+0000","last_active":"2024-01-26T23:30:53.149915+0000","last_peered":"2024-01-26T23:30:53.149915+0000","last_clean":"2024-01-26T23:30:53.149915+0000","last_became_active":"2024-01-26T23:30:53.149567+0000","last_became_peered":"2024-01-26T23:30:53.149567+0000","last_unstale":"2024-01-26T23:30:53.149915+0000","last_undegraded":"2024-01-26T23:30:53.149915+0000","last_fullsized":"2024-01-26T23:30:53.149915+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:35:46.571656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.140119+0000","last_change":"2024-01-26T23:30:52.134836+0000","last_active":"2024-01-26T23:30:52.140119+0000","last_peered":"2024-01-26T23:30:52.140119+0000","last_clean":"2024-01-26T23:30:52.140119+0000","last_became_active":"2024-01-26T23:30:52.134563+0000","last_became_peered":"2024-01-26T23:30:52.134563+0000","last_unstale":"2024-01-26T23:30:52.140119+0000","last_undegraded":"2024-01-26T23:30:52.140119+0000","last_fullsized":"2024-01-26T23:30:52.140119+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:31:04.332740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.138819+0000","last_change":"2024-01-26T23:30:53.138819+0000","last_active":"2024-01-26T23:30:53.138819+0000","last_peered":"2024-01-26T23:30:53.138819+0000","last_clean":"2024-01-26T23:30:53.138819+0000","last_became_active":"2024-01-26T23:30:53.138106+0000","last_became_peered":"2024-01-26T23:30:53.138106+0000","last_unstale":"2024-01-26T23:30:53.138819+0000","last_undegraded":"2024-01-26T23:30:53.138819+0000","last_fullsized":"2024-01-26T23:30:53.138819+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:35:06.371452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.131812+0000","last_change":"2024-01-26T23:30:52.127739+0000","last_active":"2024-01-26T23:30:52.131812+0000","last_peered":"2024-01-26T23:30:52.131812+0000","last_clean":"2024-01-26T23:30:52.131812+0000","last_became_active":"2024-01-26T23:30:52.127132+0000","last_became_peered":"2024-01-26T23:30:52.127132+0000","last_unstale":"2024-01-26T23:30:52.131812+0000","last_undegraded":"2024-01-26T23:30:52.131812+0000","last_fullsized":"2024-01-26T23:30:52.131812+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:19:41.666152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.153869+0000","last_change":"2024-01-26T23:30:53.153869+0000","last_active":"2024-01-26T23:30:53.153869+0000","last_peered":"2024-01-26T23:30:53.153869+0000","last_clean":"2024-01-26T23:30:53.153869+0000","last_became_active":"2024-01-26T23:30:53.153556+0000","last_became_peered":"2024-01-26T23:30:53.153556+0000","last_unstale":"2024-01-26T23:30:53.153869+0000","last_undegraded":"2024-01-26T23:30:53.153869+0000","last_fullsized":"2024-01-26T23:30:53.153869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:14.479539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.163263+0000","last_change":"2024-01-26T23:30:52.133918+0000","last_active":"2024-01-26T23:30:53.163263+0000","last_peered":"2024-01-26T23:30:53.163263+0000","last_clean":"2024-01-26T23:30:53.163263+0000","last_became_active":"2024-01-26T23:30:52.133679+0000","last_became_peered":"2024-01-26T23:30:52.133679+0000","last_unstale":"2024-01-26T23:30:53.163263+0000","last_undegraded":"2024-01-26T23:30:53.163263+0000","last_fullsized":"2024-01-26T23:30:53.163263+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:27.457158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.146855+0000","last_change":"2024-01-26T23:30:53.146855+0000","last_active":"2024-01-26T23:30:53.146855+0000","last_peered":"2024-01-26T23:30:53.146855+0000","last_clean":"2024-01-26T23:30:53.146855+0000","last_became_active":"2024-01-26T23:30:53.146583+0000","last_became_peered":"2024-01-26T23:30:53.146583+0000","last_unstale":"2024-01-26T23:30:53.146855+0000","last_undegraded":"2024-01-26T23:30:53.146855+0000","last_fullsized":"2024-01-26T23:30:53.146855+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:01:39.008256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.136093+0000","last_change":"2024-01-26T23:30:52.132982+0000","last_active":"2024-01-26T23:30:52.136093+0000","last_peered":"2024-01-26T23:30:52.136093+0000","last_clean":"2024-01-26T23:30:52.136093+0000","last_became_active":"2024-01-26T23:30:52.132735+0000","last_became_peered":"2024-01-26T23:30:52.132735+0000","last_unstale":"2024-01-26T23:30:52.136093+0000","last_undegraded":"2024-01-26T23:30:52.136093+0000","last_fullsized":"2024-01-26T23:30:52.136093+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:10:08.015052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.145179+0000","last_change":"2024-01-26T23:30:53.145179+0000","last_active":"2024-01-26T23:30:53.145179+0000","last_peered":"2024-01-26T23:30:53.145179+0000","last_clean":"2024-01-26T23:30:53.145179+0000","last_became_active":"2024-01-26T23:30:53.144878+0000","last_became_peered":"2024-01-26T23:30:53.144878+0000","last_unstale":"2024-01-26T23:30:53.145179+0000","last_undegraded":"2024-01-26T23:30:53.145179+0000","last_fullsized":"2024-01-26T23:30:53.145179+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:50.781140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.132445+0000","last_change":"2024-01-26T23:30:52.130274+0000","last_active":"2024-01-26T23:30:52.132445+0000","last_peered":"2024-01-26T23:30:52.132445+0000","last_clean":"2024-01-26T23:30:52.132445+0000","last_became_active":"2024-01-26T23:30:52.130023+0000","last_became_peered":"2024-01-26T23:30:52.130023+0000","last_unstale":"2024-01-26T23:30:52.132445+0000","last_undegraded":"2024-01-26T23:30:52.132445+0000","last_fullsized":"2024-01-26T23:30:52.132445+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:49.828311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.148491+0000","last_change":"2024-01-26T23:30:53.148491+0000","last_active":"2024-01-26T23:30:53.148491+0000","last_peered":"2024-01-26T23:30:53.148491+0000","last_clean":"2024-01-26T23:30:53.148491+0000","last_became_active":"2024-01-26T23:30:53.148059+0000","last_became_peered":"2024-01-26T23:30:53.148059+0000","last_unstale":"2024-01-26T23:30:53.148491+0000","last_undegraded":"2024-01-26T23:30:53.148491+0000","last_fullsized":"2024-01-26T23:30:53.148491+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:06:09.705120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.136621+0000","last_change":"2024-01-26T23:30:52.132426+0000","last_active":"2024-01-26T23:30:52.136621+0000","last_peered":"2024-01-26T23:30:52.136621+0000","last_clean":"2024-01-26T23:30:52.136621+0000","last_became_active":"2024-01-26T23:30:52.132083+0000","last_became_peered":"2024-01-26T23:30:52.132083+0000","last_unstale":"2024-01-26T23:30:52.136621+0000","last_undegraded":"2024-01-26T23:30:52.136621+0000","last_fullsized":"2024-01-26T23:30:52.136621+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:13.472460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.147932+0000","last_change":"2024-01-26T23:30:53.147932+0000","last_active":"2024-01-26T23:30:53.147932+0000","last_peered":"2024-01-26T23:30:53.147932+0000","last_clean":"2024-01-26T23:30:53.147932+0000","last_became_active":"2024-01-26T23:30:53.147653+0000","last_became_peered":"2024-01-26T23:30:53.147653+0000","last_unstale":"2024-01-26T23:30:53.147932+0000","last_undegraded":"2024-01-26T23:30:53.147932+0000","last_fullsized":"2024-01-26T23:30:53.147932+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:50.835391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.131859+0000","last_change":"2024-01-26T23:30:52.129625+0000","last_active":"2024-01-26T23:30:52.131859+0000","last_peered":"2024-01-26T23:30:52.131859+0000","last_clean":"2024-01-26T23:30:52.131859+0000","last_became_active":"2024-01-26T23:30:52.129329+0000","last_became_peered":"2024-01-26T23:30:52.129329+0000","last_unstale":"2024-01-26T23:30:52.131859+0000","last_undegraded":"2024-01-26T23:30:52.131859+0000","last_fullsized":"2024-01-26T23:30:52.131859+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:34:33.996022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.140434+0000","last_change":"2024-01-26T23:30:53.140434+0000","last_active":"2024-01-26T23:30:53.140434+0000","last_peered":"2024-01-26T23:30:53.140434+0000","last_clean":"2024-01-26T23:30:53.140434+0000","last_became_active":"2024-01-26T23:30:53.140202+0000","last_became_peered":"2024-01-26T23:30:53.140202+0000","last_unstale":"2024-01-26T23:30:53.140434+0000","last_undegraded":"2024-01-26T23:30:53.140434+0000","last_fullsized":"2024-01-26T23:30:53.140434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:53:34.849263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.140902+0000","last_change":"2024-01-26T23:30:52.139811+0000","last_active":"2024-01-26T23:30:52.140902+0000","last_peered":"2024-01-26T23:30:52.140902+0000","last_clean":"2024-01-26T23:30:52.140902+0000","last_became_active":"2024-01-26T23:30:52.139520+0000","last_became_peered":"2024-01-26T23:30:52.139520+0000","last_unstale":"2024-01-26T23:30:52.140902+0000","last_undegraded":"2024-01-26T23:30:52.140902+0000","last_fullsized":"2024-01-26T23:30:52.140902+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:46:40.377610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.147895+0000","last_change":"2024-01-26T23:30:53.147895+0000","last_active":"2024-01-26T23:30:53.147895+0000","last_peered":"2024-01-26T23:30:53.147895+0000","last_clean":"2024-01-26T23:30:53.147895+0000","last_became_active":"2024-01-26T23:30:53.147559+0000","last_became_peered":"2024-01-26T23:30:53.147559+0000","last_unstale":"2024-01-26T23:30:53.147895+0000","last_undegraded":"2024-01-26T23:30:53.147895+0000","last_fullsized":"2024-01-26T23:30:53.147895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:01:40.621482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.139076+0000","last_change":"2024-01-26T23:30:52.131891+0000","last_active":"2024-01-26T23:30:52.139076+0000","last_peered":"2024-01-26T23:30:52.139076+0000","last_clean":"2024-01-26T23:30:52.139076+0000","last_became_active":"2024-01-26T23:30:52.131618+0000","last_became_peered":"2024-01-26T23:30:52.131618+0000","last_unstale":"2024-01-26T23:30:52.139076+0000","last_undegraded":"2024-01-26T23:30:52.139076+0000","last_fullsized":"2024-01-26T23:30:52.139076+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:01:39.614403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":8,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-01-26T23:30:53.159160+0000","last_change":"2024-01-26T23:30:53.159160+0000","last_active":"2024-01-26T23:30:53.159160+0000","last_peered":"2024-01-26T23:30:53.159160+0000","last_clean":"2024-01-26T23:30:53.159160+0000","last_became_active":"2024-01-26T23:30:53.157580+0000","last_became_peered":"2024-01-26T23:30:53.157580+0000","last_unstale":"2024-01-26T23:30:53.159160+0000","last_undegraded":"2024-01-26T23:30:53.159160+0000","last_fullsized":"2024-01-26T23:30:53.159160+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:52.109162+0000","last_clean_scrub_stamp":"2024-01-26T23:30:52.109162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:50.289763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.140436+0000","last_change":"2024-01-26T23:30:52.135416+0000","last_active":"2024-01-26T23:30:52.140436+0000","last_peered":"2024-01-26T23:30:52.140436+0000","last_clean":"2024-01-26T23:30:52.140436+0000","last_became_active":"2024-01-26T23:30:52.135144+0000","last_became_peered":"2024-01-26T23:30:52.135144+0000","last_unstale":"2024-01-26T23:30:52.140436+0000","last_undegraded":"2024-01-26T23:30:52.140436+0000","last_fullsized":"2024-01-26T23:30:52.140436+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:30:51.102082+0000","last_clean_scrub_stamp":"2024-01-26T23:30:51.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:52:03.559898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":177,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-01-26T23:30:52.125470+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:30:52.125470+0000","last_peered":"2024-01-26T23:30:52.125470+0000","last_clean":"2024-01-26T23:30:52.125470+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:30:52.125470+0000","last_undegraded":"2024-01-26T23:30:52.125470+0000","last_fullsized":"2024-01-26T23:30:52.125470+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298272,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073568,"statfs":{"total":96636764160,"available":96331333632,"internally_reserved":0,"allocated":1007616,"data_stored":764697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.65000000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.495},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.63700000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.625},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.60799999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.56399999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.52400000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.66800000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297692,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074148,"statfs":{"total":96636764160,"available":96331927552,"internally_reserved":0,"allocated":413696,"data_stored":174329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55600000000000005}]},{"osd":1,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.56899999999999995}]},{"osd":2,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.58099999999999996}]},{"osd":3,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.626}]},{"osd":4,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.45800000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.495}]},{"osd":7,"last update":"Fri Jan 26 23:30:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.48399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297708,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074132,"statfs":{"total":96636764160,"available":96331911168,"internally_reserved":0,"allocated":430080,"data_stored":181502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.432}]},{"osd":2,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.39400000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.40899999999999997}]},{"osd":4,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.69499999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.51100000000000001}]},{"osd":6,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.746}]},{"osd":7,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.71399999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298284,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073556,"statfs":{"total":96636764160,"available":96331321344,"internally_reserved":0,"allocated":1019904,"data_stored":771780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.33600000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.41099999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.46999999999999997}]},{"osd":4,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.60199999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.61899999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.58899999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.70999999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297692,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074148,"statfs":{"total":96636764160,"available":96331927552,"internally_reserved":0,"allocated":413696,"data_stored":174329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":9.0540000000000003,"5min":9.0540000000000003,"15min":9.0540000000000003},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":4.1970000000000001,"5min":4.1970000000000001,"15min":4.1970000000000001},"last":0.32700000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.311},{"interface":"front","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":12.157,"5min":12.157,"15min":12.157},"last":0.41099999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":9.0879999999999992,"5min":9.0879999999999992,"15min":9.0879999999999992},"last":0.436}]},{"osd":4,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":9.0150000000000006,"5min":9.0150000000000006,"15min":9.0150000000000006},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":4.2480000000000002,"5min":4.2480000000000002,"15min":4.2480000000000002},"last":0.59499999999999997}]},{"osd":5,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":4.3200000000000003,"5min":4.3200000000000003,"15min":4.3200000000000003},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.60899999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.57999999999999996},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":8.9700000000000006,"5min":8.9700000000000006,"15min":8.9700000000000006},"last":0.70499999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":4.282,"5min":4.282,"15min":4.282},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.79600000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297692,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074148,"statfs":{"total":96636764160,"available":96331927552,"internally_reserved":0,"allocated":413696,"data_stored":174329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.26600000000000001},{"interface":"front","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":11.968,"5min":11.968,"15min":11.968},"last":0.31}]},{"osd":1,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":12.050000000000001,"5min":12.050000000000001,"15min":12.050000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.40699999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":12.234,"5min":12.234,"15min":12.234},"last":0.32900000000000001},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":11.925000000000001,"5min":11.925000000000001,"15min":11.925000000000001},"last":0.38900000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.222,"15min":1.222},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":12.099,"5min":12.099,"15min":12.099},"last":0.629},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.50700000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":12.199999999999999,"5min":12.199999999999999,"15min":12.199999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.54100000000000004}]},{"osd":6,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.25,"15min":1.25},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":12.132999999999999,"5min":12.132999999999999,"15min":12.132999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.61399999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":12.167,"5min":12.167,"15min":12.167},"last":0.48699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297692,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074148,"statfs":{"total":96636764160,"available":96331927552,"internally_reserved":0,"allocated":413696,"data_stored":174329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.60199999999999998}]},{"osd":1,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.66900000000000004}]},{"osd":2,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.5},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.65800000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.53200000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.622}]},{"osd":6,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.443},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.51400000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.63600000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297692,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074148,"statfs":{"total":96636764160,"available":96331927552,"internally_reserved":0,"allocated":413696,"data_stored":174329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.72899999999999998}]},{"osd":1,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.63300000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":6.4589999999999996,"5min":6.4589999999999996,"15min":6.4589999999999996},"last":0.59399999999999997}]},{"osd":3,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.497},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"last":0.54500000000000004}]},{"osd":4,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.58099999999999996}]},{"osd":6,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.65300000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.71599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:30:57.748 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-26T23:30:57.748 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-26T23:30:57.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.922+0000 7f61604e5700 1 Processor -- start 2024-01-26T23:30:57.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.922+0000 7f61604e5700 1 -- start start 2024-01-26T23:30:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.923+0000 7f61604e5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6158159510 0x7f61581598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.923+0000 7f61604e5700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f61580ca690 0x7f61580cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.923+0000 7f61604e5700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6158069260 con 0x7f61580ca220 2024-01-26T23:30:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.923+0000 7f61604e5700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f615815d670 con 0x7f61580ca690 2024-01-26T23:30:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.923+0000 7f61604e5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61580c9750 con 0x7f6158159510 2024-01-26T23:30:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.923+0000 7f615ea82700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f61580ca690 0x7f61580cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.923+0000 7f615e281700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6158159510 0x7f61581598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.923+0000 7f615da80700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f61580ca220 0x7f61580c8be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:47320/0 (socket says 172.21.15.46:47320) 2024-01-26T23:30:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.923+0000 7f615da80700 1 -- 172.21.15.46:0/3066828975 learned_addr learned my addr 172.21.15.46:0/3066828975 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.923+0000 7f615ea82700 1 --2- 172.21.15.46:0/3066828975 >> v2:172.21.15.106:3300/0 conn(0x7f61580ca690 0x7f61580cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:49872/0 (socket says 172.21.15.46:49872) 2024-01-26T23:30:57.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.923+0000 7f615e281700 1 -- 172.21.15.46:0/3066828975 >> v2:172.21.15.106:3300/0 conn(0x7f61580ca690 msgr2=0x7f61580cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:57.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.923+0000 7f615e281700 1 --2- 172.21.15.46:0/3066828975 >> v2:172.21.15.106:3300/0 conn(0x7f61580ca690 0x7f61580cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.923+0000 7f615e281700 1 -- 172.21.15.46:0/3066828975 >> v1:172.21.15.106:6789/0 conn(0x7f61580ca220 legacy=0x7f61580c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:57.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.923+0000 7f615d27f700 1 -- 172.21.15.46:0/3066828975 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290007876 0 0) 0x7f6158069260 con 0x7f61580ca220 2024-01-26T23:30:57.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.923+0000 7f615e281700 1 -- 172.21.15.46:0/3066828975 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61580c9a30 con 0x7f6158159510 2024-01-26T23:30:57.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.923+0000 7f615ea82700 1 --2- 172.21.15.46:0/3066828975 >> v2:172.21.15.106:3300/0 conn(0x7f61580ca690 0x7f61580cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:57.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.924+0000 7f615e281700 1 --2- 172.21.15.46:0/3066828975 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6158159510 0x7f61581598e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6154009910 tx=0x7f61540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9b3b3d6a2fcd36b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:57.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.924+0000 7f615d27f700 1 -- 172.21.15.46:0/3066828975 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615400b640 con 0x7f6158159510 2024-01-26T23:30:57.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.924+0000 7f615d27f700 1 -- 172.21.15.46:0/3066828975 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615400b7a0 con 0x7f6158159510 2024-01-26T23:30:57.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.924+0000 7f615d27f700 1 -- 172.21.15.46:0/3066828975 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615400bb60 con 0x7f6158159510 2024-01-26T23:30:57.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.924+0000 7f61604e5700 1 -- 172.21.15.46:0/3066828975 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6158159510 msgr2=0x7f61581598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:57.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.924+0000 7f61604e5700 1 --2- 172.21.15.46:0/3066828975 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6158159510 0x7f61581598e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6154009910 tx=0x7f61540091a0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.925+0000 7f61604e5700 1 -- 172.21.15.46:0/3066828975 shutdown_connections 2024-01-26T23:30:57.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.925+0000 7f61604e5700 1 --2- 172.21.15.46:0/3066828975 >> v2:172.21.15.106:3300/0 conn(0x7f61580ca690 0x7f61580cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.925+0000 7f61604e5700 1 --2- 172.21.15.46:0/3066828975 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6158159510 0x7f61581598e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.925+0000 7f61604e5700 1 -- 172.21.15.46:0/3066828975 >> 172.21.15.46:0/3066828975 conn(0x7f615806bd90 msgr2=0x7f615806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:57.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.925+0000 7f61604e5700 1 -- 172.21.15.46:0/3066828975 shutdown_connections 2024-01-26T23:30:57.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.925+0000 7f61604e5700 1 -- 172.21.15.46:0/3066828975 wait complete. 2024-01-26T23:30:57.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.926+0000 7f61604e5700 1 Processor -- start 2024-01-26T23:30:57.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.926+0000 7f61604e5700 1 -- start start 2024-01-26T23:30:57.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.927+0000 7f61604e5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61580ca220 0x7f61580c36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.927+0000 7f61604e5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6158159510 0x7f615815c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.927+0000 7f61604e5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f615815cb40 0x7f6158145bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.927+0000 7f61604e5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61580c5a30 con 0x7f6158159510 2024-01-26T23:30:57.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.927+0000 7f61604e5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61580c5d10 con 0x7f615815cb40 2024-01-26T23:30:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.927+0000 7f61604e5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61580c5ff0 con 0x7f61580ca220 2024-01-26T23:30:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.927+0000 7f615da80700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6158159510 0x7f615815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.927+0000 7f615da80700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6158159510 0x7f615815c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60060/0 (socket says 172.21.15.46:60060) 2024-01-26T23:30:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.927+0000 7f615da80700 1 -- 172.21.15.46:0/511759469 learned_addr learned my addr 172.21.15.46:0/511759469 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.927+0000 7f615e281700 1 --2- 172.21.15.46:0/511759469 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61580ca220 0x7f61580c36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.927+0000 7f615ea82700 1 --2- 172.21.15.46:0/511759469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f615815cb40 0x7f6158145bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.927+0000 7f615da80700 1 -- 172.21.15.46:0/511759469 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61580ca220 msgr2=0x7f61580c36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.927+0000 7f615da80700 1 --2- 172.21.15.46:0/511759469 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61580ca220 0x7f61580c36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.927+0000 7f615da80700 1 -- 172.21.15.46:0/511759469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f615815cb40 msgr2=0x7f6158145bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.927+0000 7f615da80700 1 --2- 172.21.15.46:0/511759469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f615815cb40 0x7f6158145bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.927+0000 7f615da80700 1 -- 172.21.15.46:0/511759469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6150009a20 con 0x7f6158159510 2024-01-26T23:30:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.927+0000 7f615da80700 1 --2- 172.21.15.46:0/511759469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6158159510 0x7f615815c600 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f614800c7e0 tx=0x7f6148007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.928+0000 7f614e7fc700 1 -- 172.21.15.46:0/511759469 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6148005b30 con 0x7f6158159510 2024-01-26T23:30:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.928+0000 7f614e7fc700 1 -- 172.21.15.46:0/511759469 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6148005c90 con 0x7f6158159510 2024-01-26T23:30:57.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.928+0000 7f61604e5700 1 -- 172.21.15.46:0/511759469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6154009420 con 0x7f6158159510 2024-01-26T23:30:57.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.928+0000 7f614e7fc700 1 -- 172.21.15.46:0/511759469 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6148007300 con 0x7f6158159510 2024-01-26T23:30:57.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.928+0000 7f61604e5700 1 -- 172.21.15.46:0/511759469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61580c65b0 con 0x7f6158159510 2024-01-26T23:30:57.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.929+0000 7f61604e5700 1 -- 172.21.15.46:0/511759469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6158169a90 con 0x7f6158159510 2024-01-26T23:30:57.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.929+0000 7f614e7fc700 1 -- 172.21.15.46:0/511759469 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f61480076b0 con 0x7f6158159510 2024-01-26T23:30:57.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.930+0000 7f614e7fc700 1 --2- 172.21.15.46:0/511759469 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f613c066f30 0x7f613c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:57.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.930+0000 7f614e7fc700 1 -- 172.21.15.46:0/511759469 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f6148083b20 con 0x7f6158159510 2024-01-26T23:30:57.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.930+0000 7f615e281700 1 --2- 172.21.15.46:0/511759469 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f613c066f30 0x7f613c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:57.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.930+0000 7f615e281700 1 --2- 172.21.15.46:0/511759469 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f613c066f30 0x7f613c0693e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f61540098e0 tx=0x7f6154017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:57.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:57.934+0000 7f614e7fc700 1 -- 172.21.15.46:0/511759469 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6148050d10 con 0x7f6158159510 2024-01-26T23:30:58.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.224+0000 7f61604e5700 1 -- 172.21.15.46:0/511759469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f615804c710 con 0x7f6158159510 2024-01-26T23:30:58.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.227+0000 7f614e7fc700 1 -- 172.21.15.46:0/511759469 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+0 (secure 0 0 0) 0x7f61480170e0 con 0x7f6158159510 2024-01-26T23:30:58.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.230+0000 7f61604e5700 1 -- 172.21.15.46:0/511759469 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f613c066f30 msgr2=0x7f613c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:58.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.230+0000 7f61604e5700 1 --2- 172.21.15.46:0/511759469 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f613c066f30 0x7f613c0693e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f61540098e0 tx=0x7f6154017000 comp rx=0 tx=0).stop 2024-01-26T23:30:58.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.230+0000 7f61604e5700 1 -- 172.21.15.46:0/511759469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6158159510 msgr2=0x7f615815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:58.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.230+0000 7f61604e5700 1 --2- 172.21.15.46:0/511759469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6158159510 0x7f615815c600 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f614800c7e0 tx=0x7f6148007df0 comp rx=0 tx=0).stop 2024-01-26T23:30:58.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.230+0000 7f61604e5700 1 -- 172.21.15.46:0/511759469 shutdown_connections 2024-01-26T23:30:58.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.230+0000 7f61604e5700 1 --2- 172.21.15.46:0/511759469 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f613c066f30 0x7f613c0693e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:58.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.230+0000 7f61604e5700 1 --2- 172.21.15.46:0/511759469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f615815cb40 0x7f6158145bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:58.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.230+0000 7f61604e5700 1 --2- 172.21.15.46:0/511759469 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61580ca220 0x7f61580c36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:58.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.230+0000 7f61604e5700 1 --2- 172.21.15.46:0/511759469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6158159510 0x7f615815c600 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:58.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.230+0000 7f61604e5700 1 -- 172.21.15.46:0/511759469 >> 172.21.15.46:0/511759469 conn(0x7f615806bd90 msgr2=0x7f61580b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:58.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.230+0000 7f61604e5700 1 -- 172.21.15.46:0/511759469 shutdown_connections 2024-01-26T23:30:58.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.230+0000 7f61604e5700 1 -- 172.21.15.46:0/511759469 wait complete. 2024-01-26T23:30:58.233 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-01-26T23:30:58.245 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:30:58.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.375+0000 7f9f1c4fb700 1 Processor -- start 2024-01-26T23:30:58.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.375+0000 7f9f1c4fb700 1 -- start start 2024-01-26T23:30:58.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.376+0000 7f9f1c4fb700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9f14157920 0x7f9f140b2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:58.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.377+0000 7f9f1c4fb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f140b2a60 0x7f9f140cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:58.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.377+0000 7f9f1c4fb700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f140cc550 con 0x7f9f14157550 2024-01-26T23:30:58.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.377+0000 7f9f1c4fb700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f140ca020 con 0x7f9f14157920 2024-01-26T23:30:58.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.377+0000 7f9f1c4fb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f140b2ee0 con 0x7f9f140b2a60 2024-01-26T23:30:58.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.377+0000 7f9f1aa98700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f140b2a60 0x7f9f140cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:58.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.377+0000 7f9f1a297700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f9f14157550 0x7f9f1406f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:47336/0 (socket says 172.21.15.46:47336) 2024-01-26T23:30:58.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.377+0000 7f9f19a96700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9f14157920 0x7f9f140b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:58.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.377+0000 7f9f1a297700 1 -- 172.21.15.46:0/2293005609 learned_addr learned my addr 172.21.15.46:0/2293005609 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:58.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.377+0000 7f9f19295700 1 -- 172.21.15.46:0/2293005609 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813886773 0 0) 0x7f9f140cc550 con 0x7f9f14157550 2024-01-26T23:30:58.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.377+0000 7f9f19295700 1 -- 172.21.15.46:0/2293005609 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f00003570 con 0x7f9f14157550 2024-01-26T23:30:58.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.377+0000 7f9f1aa98700 1 -- 172.21.15.46:0/2293005609 >> v2:172.21.15.106:3300/0 conn(0x7f9f14157920 msgr2=0x7f9f140b2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:58.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.377+0000 7f9f1aa98700 1 --2- 172.21.15.46:0/2293005609 >> v2:172.21.15.106:3300/0 conn(0x7f9f14157920 0x7f9f140b2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:58.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.377+0000 7f9f1aa98700 1 -- 172.21.15.46:0/2293005609 >> v1:172.21.15.106:6789/0 conn(0x7f9f14157550 legacy=0x7f9f1406f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:58.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.377+0000 7f9f1aa98700 1 -- 172.21.15.46:0/2293005609 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f140cb6a0 con 0x7f9f140b2a60 2024-01-26T23:30:58.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.377+0000 7f9f19a96700 1 --2- 172.21.15.46:0/2293005609 >> v2:172.21.15.106:3300/0 conn(0x7f9f14157920 0x7f9f140b2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:58.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.378+0000 7f9f1aa98700 1 --2- 172.21.15.46:0/2293005609 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f140b2a60 0x7f9f140cc9e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c009e60 tx=0x7f9f0c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=310b08f152c4dfea server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:58.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.378+0000 7f9f19295700 1 -- 172.21.15.46:0/2293005609 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c00e070 con 0x7f9f140b2a60 2024-01-26T23:30:58.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.378+0000 7f9f19295700 1 -- 172.21.15.46:0/2293005609 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0c00bab0 con 0x7f9f140b2a60 2024-01-26T23:30:58.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.378+0000 7f9f19295700 1 -- 172.21.15.46:0/2293005609 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c00be50 con 0x7f9f140b2a60 2024-01-26T23:30:58.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.378+0000 7f9f1c4fb700 1 -- 172.21.15.46:0/2293005609 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f140b2a60 msgr2=0x7f9f140cc9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:58.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.378+0000 7f9f1c4fb700 1 --2- 172.21.15.46:0/2293005609 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f140b2a60 0x7f9f140cc9e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c009e60 tx=0x7f9f0c009780 comp rx=0 tx=0).stop 2024-01-26T23:30:58.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.378+0000 7f9f1c4fb700 1 -- 172.21.15.46:0/2293005609 shutdown_connections 2024-01-26T23:30:58.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.378+0000 7f9f1c4fb700 1 --2- 172.21.15.46:0/2293005609 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f140b2a60 0x7f9f140cc9e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:58.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.378+0000 7f9f1c4fb700 1 --2- 172.21.15.46:0/2293005609 >> v2:172.21.15.106:3300/0 conn(0x7f9f14157920 0x7f9f140b2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:58.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.378+0000 7f9f1c4fb700 1 -- 172.21.15.46:0/2293005609 >> 172.21.15.46:0/2293005609 conn(0x7f9f14069510 msgr2=0x7f9f140aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:58.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.378+0000 7f9f1c4fb700 1 -- 172.21.15.46:0/2293005609 shutdown_connections 2024-01-26T23:30:58.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.378+0000 7f9f1c4fb700 1 -- 172.21.15.46:0/2293005609 wait complete. 2024-01-26T23:30:58.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.380+0000 7f9f1c4fb700 1 Processor -- start 2024-01-26T23:30:58.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.380+0000 7f9f1c4fb700 1 -- start start 2024-01-26T23:30:58.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.380+0000 7f9f1c4fb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f140b2a60 0x7f9f14144f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:58.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.380+0000 7f9f1c4fb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f14157550 0x7f9f140b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:58.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.380+0000 7f9f1c4fb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f14157920 0x7f9f141509c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:58.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f1c4fb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f141454a0 con 0x7f9f140b2a60 2024-01-26T23:30:58.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f1c4fb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f14150530 con 0x7f9f14157920 2024-01-26T23:30:58.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f1c4fb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f140b0850 con 0x7f9f14157550 2024-01-26T23:30:58.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f1a297700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f140b2a60 0x7f9f14144f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:58.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f1a297700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f140b2a60 0x7f9f14144f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60072/0 (socket says 172.21.15.46:60072) 2024-01-26T23:30:58.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f1a297700 1 -- 172.21.15.46:0/610597944 learned_addr learned my addr 172.21.15.46:0/610597944 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:58.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f1aa98700 1 --2- 172.21.15.46:0/610597944 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f14157920 0x7f9f141509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:58.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f19a96700 1 --2- 172.21.15.46:0/610597944 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f14157550 0x7f9f140b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:58.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f1a297700 1 -- 172.21.15.46:0/610597944 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f14157550 msgr2=0x7f9f140b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:58.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f1a297700 1 --2- 172.21.15.46:0/610597944 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f14157550 0x7f9f140b0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:58.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f1a297700 1 -- 172.21.15.46:0/610597944 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f14157920 msgr2=0x7f9f141509c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:58.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f1a297700 1 --2- 172.21.15.46:0/610597944 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f14157920 0x7f9f141509c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:58.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f1a297700 1 -- 172.21.15.46:0/610597944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f08009420 con 0x7f9f140b2a60 2024-01-26T23:30:58.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f1a297700 1 --2- 172.21.15.46:0/610597944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f140b2a60 0x7f9f14144f60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000c7f0 tx=0x7f9f10007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:58.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f067fc700 1 -- 172.21.15.46:0/610597944 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f10005b30 con 0x7f9f140b2a60 2024-01-26T23:30:58.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f067fc700 1 -- 172.21.15.46:0/610597944 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f10005c90 con 0x7f9f140b2a60 2024-01-26T23:30:58.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f067fc700 1 -- 172.21.15.46:0/610597944 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f10007340 con 0x7f9f140b2a60 2024-01-26T23:30:58.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.381+0000 7f9f1c4fb700 1 -- 172.21.15.46:0/610597944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f0c009a20 con 0x7f9f140b2a60 2024-01-26T23:30:58.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.382+0000 7f9f1c4fb700 1 -- 172.21.15.46:0/610597944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f1414fc00 con 0x7f9f140b2a60 2024-01-26T23:30:58.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.383+0000 7f9f1c4fb700 1 -- 172.21.15.46:0/610597944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f1404c710 con 0x7f9f140b2a60 2024-01-26T23:30:58.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.383+0000 7f9f067fc700 1 -- 172.21.15.46:0/610597944 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9f1001b400 con 0x7f9f140b2a60 2024-01-26T23:30:58.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.383+0000 7f9f067fc700 1 --2- 172.21.15.46:0/610597944 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9ef8066f30 0x7f9ef80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:58.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.384+0000 7f9f067fc700 1 -- 172.21.15.46:0/610597944 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f9f10084170 con 0x7f9f140b2a60 2024-01-26T23:30:58.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.384+0000 7f9f19a96700 1 --2- 172.21.15.46:0/610597944 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9ef8066f30 0x7f9ef80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:58.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.384+0000 7f9f19a96700 1 --2- 172.21.15.46:0/610597944 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9ef8066f30 0x7f9ef80693e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9f080093f0 tx=0x7f9f08008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:58.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.388+0000 7f9f067fc700 1 -- 172.21.15.46:0/610597944 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9f100513d0 con 0x7f9f140b2a60 2024-01-26T23:30:58.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.643+0000 7f9f1c4fb700 1 -- 172.21.15.46:0/610597944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f140cc980 con 0x7f9f140b2a60 2024-01-26T23:30:58.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.644+0000 7f9f067fc700 1 -- 172.21.15.46:0/610597944 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4597 (secure 0 0 0) 0x7f9f100076f0 con 0x7f9f140b2a60 2024-01-26T23:30:58.646 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:30:58.646 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1737713978","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1737713978},{"type":"v1","addr":"172.21.15.46:6835","nonce":1737713978}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":4969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3857864387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3857864387},{"type":"v1","addr":"172.21.15.106:6836","nonce":3857864387}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/639000251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":639000251},{"type":"v1","addr":"172.21.15.106:6837","nonce":639000251}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:30:52.553580+0000","modified":"2024-01-26T23:30:56.149313+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4984":{"gid":4984,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/453078862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":453078862},{"type":"v1","addr":"172.21.15.46:6834","nonce":453078862}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2024-01-26T23:30:58.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.646+0000 7f9f1c4fb700 1 -- 172.21.15.46:0/610597944 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9ef8066f30 msgr2=0x7f9ef80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:58.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.646+0000 7f9f1c4fb700 1 --2- 172.21.15.46:0/610597944 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9ef8066f30 0x7f9ef80693e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9f080093f0 tx=0x7f9f08008040 comp rx=0 tx=0).stop 2024-01-26T23:30:58.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.646+0000 7f9f1c4fb700 1 -- 172.21.15.46:0/610597944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f140b2a60 msgr2=0x7f9f14144f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:58.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.646+0000 7f9f1c4fb700 1 --2- 172.21.15.46:0/610597944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f140b2a60 0x7f9f14144f60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000c7f0 tx=0x7f9f10007e00 comp rx=0 tx=0).stop 2024-01-26T23:30:58.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.646+0000 7f9f1c4fb700 1 -- 172.21.15.46:0/610597944 shutdown_connections 2024-01-26T23:30:58.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.646+0000 7f9f1c4fb700 1 --2- 172.21.15.46:0/610597944 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9ef8066f30 0x7f9ef80693e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:58.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.646+0000 7f9f1c4fb700 1 --2- 172.21.15.46:0/610597944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f140b2a60 0x7f9f14144f60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:58.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.646+0000 7f9f1c4fb700 1 --2- 172.21.15.46:0/610597944 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f14157920 0x7f9f141509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:58.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.646+0000 7f9f1c4fb700 1 --2- 172.21.15.46:0/610597944 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f14157550 0x7f9f140b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:58.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.646+0000 7f9f1c4fb700 1 -- 172.21.15.46:0/610597944 >> 172.21.15.46:0/610597944 conn(0x7f9f14069510 msgr2=0x7f9f1406f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:58.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.646+0000 7f9f1c4fb700 1 -- 172.21.15.46:0/610597944 shutdown_connections 2024-01-26T23:30:58.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:58.646+0000 7f9f1c4fb700 1 -- 172.21.15.46:0/610597944 wait complete. 2024-01-26T23:30:58.649 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 18 2024-01-26T23:30:59.064 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-26T23:30:52.553580+0000', 'modified': '2024-01-26T23:30:56.149313+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4984}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4984': {'gid': 4984, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6834/453078862', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 453078862}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 453078862}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-26T23:30:59.064 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-26T23:30:59.065 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-26T23:30:59.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.205+0000 7f6ff9cd4700 1 Processor -- start 2024-01-26T23:30:59.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.205+0000 7f6ff9cd4700 1 -- start start 2024-01-26T23:30:59.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.206+0000 7f6ff9cd4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ff40cd250 0x7f6ff4162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.206+0000 7f6ff9cd4700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff40cbf10 con 0x7f6ff40acf30 2024-01-26T23:30:59.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.206+0000 7f6ff9cd4700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff40c4460 con 0x7f6ff40c3cf0 2024-01-26T23:30:59.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.206+0000 7f6ff9cd4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40ada10 con 0x7f6ff40cd250 2024-01-26T23:30:59.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.206+0000 7f6ff37fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6ff40acf30 0x7f6ff40ad300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46988/0 (socket says 172.21.15.46:46988) 2024-01-26T23:30:59.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.206+0000 7f6ff37fe700 1 -- 172.21.15.46:0/1653182121 learned_addr learned my addr 172.21.15.46:0/1653182121 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:30:59.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff3fff700 1 --2- 172.21.15.46:0/1653182121 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ff40cd250 0x7f6ff4162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff27fc700 1 -- 172.21.15.46:0/1653182121 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470039163 0 0) 0x7f6ff40cbf10 con 0x7f6ff40acf30 2024-01-26T23:30:59.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff27fc700 1 -- 172.21.15.46:0/1653182121 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fdc003570 con 0x7f6ff40acf30 2024-01-26T23:30:59.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff27fc700 1 -- 172.21.15.46:0/1653182121 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526213036 0 0) 0x7f6ff40c4460 con 0x7f6ff40c3cf0 2024-01-26T23:30:59.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff27fc700 1 -- 172.21.15.46:0/1653182121 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff40cbf10 con 0x7f6ff40c3cf0 2024-01-26T23:30:59.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff27fc700 1 -- 172.21.15.46:0/1653182121 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 731613337 0 0) 0x7f6fdc003570 con 0x7f6ff40acf30 2024-01-26T23:30:59.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff27fc700 1 -- 172.21.15.46:0/1653182121 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ff40cd250 msgr2=0x7f6ff4162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:59.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff27fc700 1 --2- 172.21.15.46:0/1653182121 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ff40cd250 0x7f6ff4162360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff27fc700 1 -- 172.21.15.46:0/1653182121 >> v1:172.21.15.106:6789/0 conn(0x7f6ff40c3cf0 legacy=0x7f6ff40ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:59.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff27fc700 1 -- 172.21.15.46:0/1653182121 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff40c4740 con 0x7f6ff40acf30 2024-01-26T23:30:59.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff27fc700 1 -- 172.21.15.46:0/1653182121 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f6fe4001470 con 0x7f6ff40acf30 2024-01-26T23:30:59.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff3fff700 1 --2- 172.21.15.46:0/1653182121 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ff40cd250 0x7f6ff4162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:59.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff27fc700 1 -- 172.21.15.46:0/1653182121 >> v1:172.21.15.46:6789/0 conn(0x7f6ff40acf30 legacy=0x7f6ff40ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:59.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff27fc700 1 --2- 172.21.15.46:0/1653182121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fdc003c20 0x7f6fdc004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff27fc700 1 --2- 172.21.15.46:0/1653182121 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ff40cd250 0x7f6fdc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff27fc700 1 --2- 172.21.15.46:0/1653182121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6fdc004d50 0x7f6fdc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff27fc700 1 -- 172.21.15.46:0/1653182121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40ada10 con 0x7f6fdc003c20 2024-01-26T23:30:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff27fc700 1 -- 172.21.15.46:0/1653182121 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc00bb10 con 0x7f6fdc004d50 2024-01-26T23:30:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff27fc700 1 -- 172.21.15.46:0/1653182121 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc00bcd0 con 0x7f6ff40cd250 2024-01-26T23:30:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff37fe700 1 --2- 172.21.15.46:0/1653182121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fdc003c20 0x7f6fdc004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff2ffd700 1 --2- 172.21.15.46:0/1653182121 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ff40cd250 0x7f6fdc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff3fff700 1 --2- 172.21.15.46:0/1653182121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6fdc004d50 0x7f6fdc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff37fe700 1 -- 172.21.15.46:0/1653182121 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ff40cd250 msgr2=0x7f6fdc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff37fe700 1 --2- 172.21.15.46:0/1653182121 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ff40cd250 0x7f6fdc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff37fe700 1 -- 172.21.15.46:0/1653182121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6fdc004d50 msgr2=0x7f6fdc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.207+0000 7f6ff37fe700 1 --2- 172.21.15.46:0/1653182121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6fdc004d50 0x7f6fdc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.208+0000 7f6ff37fe700 1 -- 172.21.15.46:0/1653182121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec009a20 con 0x7f6fdc003c20 2024-01-26T23:30:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.208+0000 7f6ff37fe700 1 --2- 172.21.15.46:0/1653182121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fdc003c20 0x7f6fdc004010 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400c2d0 tx=0x7f6fe4001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149555965df717c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.208+0000 7f6ff27fc700 1 -- 172.21.15.46:0/1653182121 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe400cba0 con 0x7f6fdc003c20 2024-01-26T23:30:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.208+0000 7f6ff27fc700 1 -- 172.21.15.46:0/1653182121 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe400cd00 con 0x7f6fdc003c20 2024-01-26T23:30:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.208+0000 7f6ff27fc700 1 -- 172.21.15.46:0/1653182121 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4005b70 con 0x7f6fdc003c20 2024-01-26T23:30:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.208+0000 7f6ff9cd4700 1 -- 172.21.15.46:0/1653182121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fdc003c20 msgr2=0x7f6fdc004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.208+0000 7f6ff9cd4700 1 --2- 172.21.15.46:0/1653182121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fdc003c20 0x7f6fdc004010 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400c2d0 tx=0x7f6fe4001e10 comp rx=0 tx=0).stop 2024-01-26T23:30:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.208+0000 7f6ff3fff700 1 -- 172.21.15.46:0/1653182121 reap_dead start 2024-01-26T23:30:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.208+0000 7f6ff9cd4700 1 -- 172.21.15.46:0/1653182121 shutdown_connections 2024-01-26T23:30:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.208+0000 7f6ff9cd4700 1 -- 172.21.15.46:0/1653182121 >> 172.21.15.46:0/1653182121 conn(0x7f6ff4097820 msgr2=0x7f6ff4068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.209+0000 7f6ff9cd4700 1 -- 172.21.15.46:0/1653182121 shutdown_connections 2024-01-26T23:30:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.209+0000 7f6ff9cd4700 1 -- 172.21.15.46:0/1653182121 wait complete. 2024-01-26T23:30:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.210+0000 7f6ff9cd4700 1 Processor -- start 2024-01-26T23:30:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.210+0000 7f6ff9cd4700 1 -- start start 2024-01-26T23:30:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.211+0000 7f6ff9cd4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6fdc003c20 0x7f6ff4153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.211+0000 7f6ff9cd4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff40c72e0 0x7f6ff40c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.211+0000 7f6ff9cd4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ff40b36e0 0x7f6ff40b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.211+0000 7f6ff9cd4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40c7c90 con 0x7f6ff40c72e0 2024-01-26T23:30:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.211+0000 7f6ff9cd4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40b40c0 con 0x7f6fdc003c20 2024-01-26T23:30:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.211+0000 7f6ff9cd4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40c5720 con 0x7f6ff40b36e0 2024-01-26T23:30:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.211+0000 7f6ff2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff40c72e0 0x7f6ff40c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.211+0000 7f6ff2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff40c72e0 0x7f6ff40c7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60100/0 (socket says 172.21.15.46:60100) 2024-01-26T23:30:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.211+0000 7f6ff2ffd700 1 -- 172.21.15.46:0/1972372346 learned_addr learned my addr 172.21.15.46:0/1972372346 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.211+0000 7f6ff37fe700 1 --2- 172.21.15.46:0/1972372346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6fdc003c20 0x7f6ff4153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.211+0000 7f6ff3fff700 1 --2- 172.21.15.46:0/1972372346 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ff40b36e0 0x7f6ff40b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.211+0000 7f6ff2ffd700 1 -- 172.21.15.46:0/1972372346 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ff40b36e0 msgr2=0x7f6ff40b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.211+0000 7f6ff2ffd700 1 --2- 172.21.15.46:0/1972372346 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ff40b36e0 0x7f6ff40b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.211+0000 7f6ff2ffd700 1 -- 172.21.15.46:0/1972372346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6fdc003c20 msgr2=0x7f6ff4153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.211+0000 7f6ff2ffd700 1 --2- 172.21.15.46:0/1972372346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6fdc003c20 0x7f6ff4153c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.211+0000 7f6ff2ffd700 1 -- 172.21.15.46:0/1972372346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff40c5a00 con 0x7f6ff40c72e0 2024-01-26T23:30:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.212+0000 7f6ff2ffd700 1 --2- 172.21.15.46:0/1972372346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff40c72e0 0x7f6ff40c7750 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8002bd0 tx=0x7f6fe8007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.212+0000 7f6fdb7fe700 1 -- 172.21.15.46:0/1972372346 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8006a30 con 0x7f6ff40c72e0 2024-01-26T23:30:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.212+0000 7f6fdb7fe700 1 -- 172.21.15.46:0/1972372346 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe8006b90 con 0x7f6ff40c72e0 2024-01-26T23:30:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.212+0000 7f6ff9cd4700 1 -- 172.21.15.46:0/1972372346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff40c5ce0 con 0x7f6ff40c72e0 2024-01-26T23:30:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.212+0000 7f6fdb7fe700 1 -- 172.21.15.46:0/1972372346 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8007030 con 0x7f6ff40c72e0 2024-01-26T23:30:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.212+0000 7f6ff9cd4700 1 -- 172.21.15.46:0/1972372346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff40c62a0 con 0x7f6ff40c72e0 2024-01-26T23:30:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.213+0000 7f6fdb7fe700 1 -- 172.21.15.46:0/1972372346 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6fe80073e0 con 0x7f6ff40c72e0 2024-01-26T23:30:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.214+0000 7f6fdb7fe700 1 --2- 172.21.15.46:0/1972372346 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6fe0066f30 0x7f6fe00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.214+0000 7f6fdb7fe700 1 -- 172.21.15.46:0/1972372346 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f6fe8084ad0 con 0x7f6ff40c72e0 2024-01-26T23:30:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.214+0000 7f6ff37fe700 1 --2- 172.21.15.46:0/1972372346 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6fe0066f30 0x7f6fe00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.215+0000 7f6ff37fe700 1 --2- 172.21.15.46:0/1972372346 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6fe0066f30 0x7f6fe00693e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4007c90 tx=0x7f6fe4015000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.216+0000 7f7002fd7b80 1 Processor -- start 2024-01-26T23:30:59.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.216+0000 7f7002fd7b80 1 -- start start 2024-01-26T23:30:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.217+0000 7f7002fd7b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e9a010 0x559903e9a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.217+0000 7f7002fd7b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 0x559903ec4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.217+0000 7f7002fd7b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e995d0 0x559903ec4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.217+0000 7f7002fd7b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x559903d9d9f0 con 0x559903e9a010 2024-01-26T23:30:59.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.217+0000 7f7002fd7b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x559903def130 con 0x559903ee2360 2024-01-26T23:30:59.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.217+0000 7f7002fd7b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x559903dec930 con 0x559903e995d0 2024-01-26T23:30:59.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.217+0000 7f6ff37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e9a010 0x559903e9a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.217+0000 7f6ff37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e9a010 0x559903e9a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60110/0 (socket says 172.21.15.46:60110) 2024-01-26T23:30:59.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.217+0000 7f6ff37fe700 1 -- 172.21.15.46:0/3994951338 learned_addr learned my addr 172.21.15.46:0/3994951338 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:59.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.217+0000 7f6ff2ffd700 1 --2- 172.21.15.46:0/3994951338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 0x559903ec4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.217+0000 7f6ff3fff700 1 --2- 172.21.15.46:0/3994951338 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e995d0 0x559903ec4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.217+0000 7f6ff37fe700 1 -- 172.21.15.46:0/3994951338 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e995d0 msgr2=0x559903ec4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.217+0000 7f6ff37fe700 1 --2- 172.21.15.46:0/3994951338 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e995d0 0x559903ec4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.217+0000 7f6ff37fe700 1 -- 172.21.15.46:0/3994951338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 msgr2=0x559903ec4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.217+0000 7f6ff37fe700 1 --2- 172.21.15.46:0/3994951338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 0x559903ec4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.217+0000 7f6ff37fe700 1 -- 172.21.15.46:0/3994951338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559903c27b10 con 0x559903e9a010 2024-01-26T23:30:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.217+0000 7f6ff37fe700 1 --2- 172.21.15.46:0/3994951338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e9a010 0x559903e9a3e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4000cb0 tx=0x7f6fe4021040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad5b469ad18678d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.218+0000 7f6fd8ff9700 1 -- 172.21.15.46:0/3994951338 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4001710 con 0x559903e9a010 2024-01-26T23:30:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.218+0000 7f6fd8ff9700 1 -- 172.21.15.46:0/3994951338 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe4000fe0 con 0x559903e9a010 2024-01-26T23:30:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.218+0000 7f6fd8ff9700 1 -- 172.21.15.46:0/3994951338 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4004e50 con 0x559903e9a010 2024-01-26T23:30:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.218+0000 7f7002fd7b80 1 -- 172.21.15.46:0/3994951338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e9a010 msgr2=0x559903e9a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.218+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/3994951338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e9a010 0x559903e9a3e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4000cb0 tx=0x7f6fe4021040 comp rx=0 tx=0).stop 2024-01-26T23:30:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.218+0000 7f7002fd7b80 1 -- 172.21.15.46:0/3994951338 shutdown_connections 2024-01-26T23:30:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.218+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/3994951338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e9a010 0x559903e9a3e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.218+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/3994951338 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e995d0 0x559903ec4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.218+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/3994951338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 0x559903ec4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.218+0000 7f7002fd7b80 1 -- 172.21.15.46:0/3994951338 >> 172.21.15.46:0/3994951338 conn(0x559903e99a70 msgr2=0x559903ee2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.218+0000 7f7002fd7b80 1 -- 172.21.15.46:0/3994951338 shutdown_connections 2024-01-26T23:30:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.219+0000 7f7002fd7b80 1 -- 172.21.15.46:0/3994951338 wait complete. 2024-01-26T23:30:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.219+0000 7f7002fd7b80 1 -- start start 2024-01-26T23:30:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.219+0000 7f7002fd7b80 1 Processor -- start 2024-01-26T23:30:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.220+0000 7f7002fd7b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e995d0 0x559903ed99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.220+0000 7f7002fd7b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e9a010 0x559903ebf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.220+0000 7f7002fd7b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 0x559903ee0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.220+0000 7f7002fd7b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x559903de8720 con 0x559903e995d0 2024-01-26T23:30:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.220+0000 7f7002fd7b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x559903d941b0 con 0x559903ee2360 2024-01-26T23:30:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.220+0000 7f7002fd7b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x559903d93e50 con 0x559903e9a010 2024-01-26T23:30:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.220+0000 7f7002fd7b80 10 client.0 _ll_register_callbacks cb 0x559903bf67b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:30:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.220+0000 7f6ff37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e995d0 0x559903ed99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.220+0000 7f6ff37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e995d0 0x559903ed99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60120/0 (socket says 172.21.15.46:60120) 2024-01-26T23:30:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.220+0000 7f6ff37fe700 1 -- 172.21.15.46:0/3893997600 learned_addr learned my addr 172.21.15.46:0/3893997600 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.220+0000 7f6ff3fff700 1 --2- 172.21.15.46:0/3893997600 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 0x559903ee0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.220+0000 7f6ff2ffd700 1 --2- 172.21.15.46:0/3893997600 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e9a010 0x559903ebf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f6ff37fe700 1 -- 172.21.15.46:0/3893997600 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e9a010 msgr2=0x559903ebf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f6ff37fe700 1 --2- 172.21.15.46:0/3893997600 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e9a010 0x559903ebf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f6ff37fe700 1 -- 172.21.15.46:0/3893997600 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 msgr2=0x559903ee0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f6ff37fe700 1 --2- 172.21.15.46:0/3893997600 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 0x559903ee0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f6ff37fe700 1 -- 172.21.15.46:0/3893997600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe4024040 con 0x559903e995d0 2024-01-26T23:30:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f6ff37fe700 1 --2- 172.21.15.46:0/3893997600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e995d0 0x559903ed99c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4001ca0 tx=0x7f6fe402b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b21d23ba65de404 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f6fd1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:30:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f6fd1ffb700 1 -- 172.21.15.46:0/3893997600 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4001710 con 0x559903e995d0 2024-01-26T23:30:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f6fd1ffb700 1 -- 172.21.15.46:0/3893997600 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f6fe40074f0 con 0x559903e995d0 2024-01-26T23:30:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f7002fd7b80 1 -- 172.21.15.46:0/3893997600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559903e8d190 con 0x559903e995d0 2024-01-26T23:30:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f6fd1ffb700 1 -- 172.21.15.46:0/3893997600 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fe401f410 con 0x559903e995d0 2024-01-26T23:30:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f7002fd7b80 10 client.5214 fetch_fsmap learned FSMap version 18 2024-01-26T23:30:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f7002fd7b80 1 -- 172.21.15.46:0/3893997600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x559903db6290 con 0x559903e995d0 2024-01-26T23:30:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f6fd1ffb700 1 -- 172.21.15.46:0/3893997600 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559903e8d190 con 0x559903e995d0 2024-01-26T23:30:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f6fd1ffb700 1 -- 172.21.15.46:0/3893997600 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 18) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f6fe401f570 con 0x559903e995d0 2024-01-26T23:30:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f7002fd7b80 10 client.5214 fetch_fsmap finished waiting for FSMap version 18 2024-01-26T23:30:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f7002fd7b80 10 client.5214 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-26T23:30:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f7002fd7b80 20 client.5214 populate_metadata read hostname 'smithi046' 2024-01-26T23:30:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/3893997600 >> [v2:172.21.15.46:6832/453078862,v1:172.21.15.46:6834/453078862] conn(0x559903eda330 0x559903eda700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f7002fd7b80 4 client.5214 mds_command: new command op to 4984 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-26T23:30:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.221+0000 7f7002fd7b80 1 -- 172.21.15.46:0/3893997600 --> [v2:172.21.15.46:6832/453078862,v1:172.21.15.46:6834/453078862] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559903d8aee0 con 0x559903eda330 2024-01-26T23:30:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.222+0000 7f6ff2ffd700 1 --2- 172.21.15.46:0/3893997600 >> [v2:172.21.15.46:6832/453078862,v1:172.21.15.46:6834/453078862] conn(0x559903eda330 0x559903eda700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.222+0000 7f6ff2ffd700 1 --2- 172.21.15.46:0/3893997600 >> [v2:172.21.15.46:6832/453078862,v1:172.21.15.46:6834/453078862] conn(0x559903eda330 0x559903eda700 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdaee864757dfe05 server_cookie=9aaeaa9717cb4081 in_seq=0 out_seq=0 2024-01-26T23:30:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.222+0000 7f6fd1ffb700 10 client.5214 ms_handle_connect on v2:172.21.15.46:6832/453078862 2024-01-26T23:30:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.223+0000 7f6fd1ffb700 1 -- 172.21.15.46:0/3893997600 <== mds.0 v2:172.21.15.46:6832/453078862 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x559903d8aee0 con 0x559903eda330 2024-01-26T23:30:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.223+0000 7f6fd1ffb700 10 client.5214 handle_command_reply: tid=0 2024-01-26T23:30:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.223+0000 7f7002fd7b80 1 client.5214 shutdown 2024-01-26T23:30:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.223+0000 7f7002fd7b80 1 -- 172.21.15.46:0/3893997600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e995d0 msgr2=0x559903ed99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.223+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/3893997600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e995d0 0x559903ed99c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4001ca0 tx=0x7f6fe402b000 comp rx=0 tx=0).stop 2024-01-26T23:30:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.223+0000 7f7002fd7b80 1 -- 172.21.15.46:0/3893997600 shutdown_connections 2024-01-26T23:30:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.223+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/3893997600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e995d0 0x559903ed99c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.223+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/3893997600 >> [v2:172.21.15.46:6832/453078862,v1:172.21.15.46:6834/453078862] conn(0x559903eda330 0x559903eda700 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.223+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/3893997600 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 0x559903ee0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.223+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/3893997600 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e9a010 0x559903ebf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.223+0000 7f6fd1ffb700 0 client.5214 ms_handle_reset on v2:172.21.15.46:6832/453078862 2024-01-26T23:30:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.223+0000 7f7002fd7b80 1 -- 172.21.15.46:0/3893997600 >> 172.21.15.46:0/3893997600 conn(0x559903e99a70 msgr2=0x559903e94d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.223+0000 7f7002fd7b80 1 -- 172.21.15.46:0/3893997600 shutdown_connections 2024-01-26T23:30:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.223+0000 7f7002fd7b80 1 -- 172.21.15.46:0/3893997600 wait complete. 2024-01-26T23:30:59.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.223+0000 7f7002fd7b80 20 client.5214 trim_cache size 0 max 16384 2024-01-26T23:30:59.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.241+0000 7f7002fd7b80 1 Processor -- start 2024-01-26T23:30:59.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.241+0000 7f7002fd7b80 1 -- start start 2024-01-26T23:30:59.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.242+0000 7f7002fd7b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e995d0 0x559903ee1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.242+0000 7f7002fd7b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e9a010 0x559903ed8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.242+0000 7f7002fd7b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 0x559903ef6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.242+0000 7f7002fd7b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x559903ce0e40 con 0x559903e9a010 2024-01-26T23:30:59.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.242+0000 7f7002fd7b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x559903ef74b0 con 0x559903ee2360 2024-01-26T23:30:59.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.242+0000 7f7002fd7b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x559903edc050 con 0x559903e995d0 2024-01-26T23:30:59.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.242+0000 7f6ff2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e9a010 0x559903ed8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.242+0000 7f6ff2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e9a010 0x559903ed8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60126/0 (socket says 172.21.15.46:60126) 2024-01-26T23:30:59.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.242+0000 7f6ff2ffd700 1 -- 172.21.15.46:0/241047523 learned_addr learned my addr 172.21.15.46:0/241047523 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:59.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.242+0000 7f6ff3fff700 1 --2- 172.21.15.46:0/241047523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 0x559903ef6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.242+0000 7f6ff37fe700 1 --2- 172.21.15.46:0/241047523 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e995d0 0x559903ee1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.243+0000 7f6ff2ffd700 1 -- 172.21.15.46:0/241047523 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e995d0 msgr2=0x559903ee1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:59.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.243+0000 7f6ff2ffd700 1 --2- 172.21.15.46:0/241047523 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e995d0 0x559903ee1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.243+0000 7f6ff2ffd700 1 -- 172.21.15.46:0/241047523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 msgr2=0x559903ef6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:59.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.243+0000 7f6ff2ffd700 1 --2- 172.21.15.46:0/241047523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 0x559903ef6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.243+0000 7f6ff2ffd700 1 -- 172.21.15.46:0/241047523 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559903c7fc80 con 0x559903e9a010 2024-01-26T23:30:59.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.243+0000 7f6ff3fff700 1 --2- 172.21.15.46:0/241047523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 0x559903ef6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:59.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.243+0000 7f6ff2ffd700 1 --2- 172.21.15.46:0/241047523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e9a010 0x559903ed8be0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8051d20 tx=0x7f6fe801f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb75d4a21c08355 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:59.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.243+0000 7f6fd97fa700 1 -- 172.21.15.46:0/241047523 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8007690 con 0x559903e9a010 2024-01-26T23:30:59.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.243+0000 7f6fd97fa700 1 -- 172.21.15.46:0/241047523 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe8053960 con 0x559903e9a010 2024-01-26T23:30:59.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.243+0000 7f6fd97fa700 1 -- 172.21.15.46:0/241047523 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8071380 con 0x559903e9a010 2024-01-26T23:30:59.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.244+0000 7f7002fd7b80 1 -- 172.21.15.46:0/241047523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e9a010 msgr2=0x559903ed8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:59.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.244+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/241047523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e9a010 0x559903ed8be0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8051d20 tx=0x7f6fe801f040 comp rx=0 tx=0).stop 2024-01-26T23:30:59.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.244+0000 7f7002fd7b80 1 -- 172.21.15.46:0/241047523 shutdown_connections 2024-01-26T23:30:59.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.244+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/241047523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 0x559903ef6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.244+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/241047523 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e995d0 0x559903ee1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.244+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/241047523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e9a010 0x559903ed8be0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.244+0000 7f7002fd7b80 1 -- 172.21.15.46:0/241047523 >> 172.21.15.46:0/241047523 conn(0x559903e99a70 msgr2=0x559903e96010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.244+0000 7f7002fd7b80 1 -- 172.21.15.46:0/241047523 shutdown_connections 2024-01-26T23:30:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.244+0000 7f7002fd7b80 1 -- 172.21.15.46:0/241047523 wait complete. 2024-01-26T23:30:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.244+0000 7f7002fd7b80 1 -- start start 2024-01-26T23:30:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.244+0000 7f7002fd7b80 1 Processor -- start 2024-01-26T23:30:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.245+0000 7f7002fd7b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e995d0 0x559903f46d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.245+0000 7f7002fd7b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e9a010 0x559903edd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.245+0000 7f7002fd7b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 0x559903eddf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.245+0000 7f7002fd7b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x559903efe060 con 0x559903e995d0 2024-01-26T23:30:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.245+0000 7f7002fd7b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x559903ed91a0 con 0x559903ee2360 2024-01-26T23:30:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.245+0000 7f7002fd7b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x559903f47270 con 0x559903e9a010 2024-01-26T23:30:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.245+0000 7f7002fd7b80 10 client.0 _ll_register_callbacks cb 0x559903bf67b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:30:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.245+0000 7f6ff37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e995d0 0x559903f46d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.245+0000 7f6ff37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e995d0 0x559903f46d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60138/0 (socket says 172.21.15.46:60138) 2024-01-26T23:30:59.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.245+0000 7f6ff37fe700 1 -- 172.21.15.46:0/983247929 learned_addr learned my addr 172.21.15.46:0/983247929 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:59.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.245+0000 7f6ff2ffd700 1 --2- 172.21.15.46:0/983247929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e9a010 0x559903edd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.245+0000 7f6ff3fff700 1 --2- 172.21.15.46:0/983247929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 0x559903eddf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.246+0000 7f6ff37fe700 1 -- 172.21.15.46:0/983247929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e9a010 msgr2=0x559903edd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:59.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.246+0000 7f6ff37fe700 1 --2- 172.21.15.46:0/983247929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e9a010 0x559903edd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.246+0000 7f6ff37fe700 1 -- 172.21.15.46:0/983247929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 msgr2=0x559903eddf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:59.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.246+0000 7f6ff37fe700 1 --2- 172.21.15.46:0/983247929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 0x559903eddf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.246+0000 7f6ff37fe700 1 -- 172.21.15.46:0/983247929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff40c62a0 con 0x559903e995d0 2024-01-26T23:30:59.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.246+0000 7f6ff2ffd700 1 --2- 172.21.15.46:0/983247929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e9a010 0x559903edd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:30:59.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.246+0000 7f6ff37fe700 1 --2- 172.21.15.46:0/983247929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e995d0 0x559903f46d30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4001bc0 tx=0x7f6fe4014070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e01ba3d7ecd85ace server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:59.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.246+0000 7f6fd1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:30:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.246+0000 7f6fd1ffb700 1 -- 172.21.15.46:0/983247929 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4039030 con 0x559903e995d0 2024-01-26T23:30:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.246+0000 7f6fd1ffb700 1 -- 172.21.15.46:0/983247929 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f6fe40074f0 con 0x559903e995d0 2024-01-26T23:30:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.246+0000 7f7002fd7b80 1 -- 172.21.15.46:0/983247929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6fe4000fe0 con 0x559903e995d0 2024-01-26T23:30:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.246+0000 7f6fd1ffb700 1 -- 172.21.15.46:0/983247929 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fe4000cb0 con 0x559903e995d0 2024-01-26T23:30:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.246+0000 7f7002fd7b80 10 client.5220 fetch_fsmap learned FSMap version 18 2024-01-26T23:30:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.246+0000 7f7002fd7b80 1 -- 172.21.15.46:0/983247929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x559903c25fe0 con 0x559903e995d0 2024-01-26T23:30:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.246+0000 7f6fd1ffb700 1 -- 172.21.15.46:0/983247929 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe4000fe0 con 0x559903e995d0 2024-01-26T23:30:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.246+0000 7f6fd1ffb700 1 -- 172.21.15.46:0/983247929 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 18) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f6fe4012610 con 0x559903e995d0 2024-01-26T23:30:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.247+0000 7f7002fd7b80 10 client.5220 fetch_fsmap finished waiting for FSMap version 18 2024-01-26T23:30:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.247+0000 7f7002fd7b80 10 client.5220 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-26T23:30:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.247+0000 7f7002fd7b80 20 client.5220 populate_metadata read hostname 'smithi046' 2024-01-26T23:30:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.247+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/983247929 >> [v2:172.21.15.46:6832/453078862,v1:172.21.15.46:6834/453078862] conn(0x559903ebd6f0 0x559903ebdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.247+0000 7f7002fd7b80 4 client.5220 mds_command: new command op to 4984 tid=0[{"prefix": "status"}] 2024-01-26T23:30:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.247+0000 7f7002fd7b80 1 -- 172.21.15.46:0/983247929 --> [v2:172.21.15.46:6832/453078862,v1:172.21.15.46:6834/453078862] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559903d8ac00 con 0x559903ebd6f0 2024-01-26T23:30:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.247+0000 7f6ff2ffd700 1 --2- 172.21.15.46:0/983247929 >> [v2:172.21.15.46:6832/453078862,v1:172.21.15.46:6834/453078862] conn(0x559903ebd6f0 0x559903ebdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.247+0000 7f6fd1ffb700 10 client.5220 ms_handle_connect on v2:172.21.15.46:6832/453078862 2024-01-26T23:30:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.247+0000 7f6ff2ffd700 1 --2- 172.21.15.46:0/983247929 >> [v2:172.21.15.46:6832/453078862,v1:172.21.15.46:6834/453078862] conn(0x559903ebd6f0 0x559903ebdac0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ed9d1cb81538146 server_cookie=425cb7eea0ed811c in_seq=0 out_seq=0 2024-01-26T23:30:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.247+0000 7f6fd1ffb700 1 -- 172.21.15.46:0/983247929 <== mds.0 v2:172.21.15.46:6832/453078862 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x559903d8ac00 con 0x559903ebd6f0 2024-01-26T23:30:59.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.247+0000 7f6fd1ffb700 10 client.5220 handle_command_reply: tid=0 2024-01-26T23:30:59.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.247+0000 7f7002fd7b80 1 client.5220 shutdown 2024-01-26T23:30:59.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.247+0000 7f7002fd7b80 1 -- 172.21.15.46:0/983247929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e995d0 msgr2=0x559903f46d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:59.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.247+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/983247929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e995d0 0x559903f46d30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4001bc0 tx=0x7f6fe4014070 comp rx=0 tx=0).stop 2024-01-26T23:30:59.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f7002fd7b80 1 -- 172.21.15.46:0/983247929 shutdown_connections 2024-01-26T23:30:59.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/983247929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559903e995d0 0x559903f46d30 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/983247929 >> [v2:172.21.15.46:6832/453078862,v1:172.21.15.46:6834/453078862] conn(0x559903ebd6f0 0x559903ebdac0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/983247929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559903ee2360 0x559903eddf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f7002fd7b80 1 --2- 172.21.15.46:0/983247929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x559903e9a010 0x559903edd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f6fd1ffb700 0 client.5220 ms_handle_reset on v2:172.21.15.46:6832/453078862 2024-01-26T23:30:59.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f7002fd7b80 1 -- 172.21.15.46:0/983247929 >> 172.21.15.46:0/983247929 conn(0x559903e99a70 msgr2=0x559903eed9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:59.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f7002fd7b80 1 -- 172.21.15.46:0/983247929 shutdown_connections 2024-01-26T23:30:59.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f7002fd7b80 1 -- 172.21.15.46:0/983247929 wait complete. 2024-01-26T23:30:59.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f7002fd7b80 20 client.5220 trim_cache size 0 max 16384 2024-01-26T23:30:59.268 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:30:59.268 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-26T23:30:59.268 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-26T23:30:59.268 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4984, 2024-01-26T23:30:59.268 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-26T23:30:59.268 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-26T23:30:59.268 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-01-26T23:30:59.268 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.1073757320000004, 2024-01-26T23:30:59.269 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 18, 2024-01-26T23:30:59.269 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 38, 2024-01-26T23:30:59.269 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 38, 2024-01-26T23:30:59.269 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 14.001764461 2024-01-26T23:30:59.269 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:30:59.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f6ff9cd4700 1 -- 172.21.15.46:0/1972372346 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6fe0066f30 msgr2=0x7f6fe00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:59.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f6ff9cd4700 1 --2- 172.21.15.46:0/1972372346 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6fe0066f30 0x7f6fe00693e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4007c90 tx=0x7f6fe4015000 comp rx=0 tx=0).stop 2024-01-26T23:30:59.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f6ff9cd4700 1 -- 172.21.15.46:0/1972372346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff40c72e0 msgr2=0x7f6ff40c7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:59.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f6ff9cd4700 1 --2- 172.21.15.46:0/1972372346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff40c72e0 0x7f6ff40c7750 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8002bd0 tx=0x7f6fe8007d30 comp rx=0 tx=0).stop 2024-01-26T23:30:59.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f6ff9cd4700 1 -- 172.21.15.46:0/1972372346 shutdown_connections 2024-01-26T23:30:59.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f6ff9cd4700 1 --2- 172.21.15.46:0/1972372346 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6fe0066f30 0x7f6fe00693e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f6ff9cd4700 1 --2- 172.21.15.46:0/1972372346 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ff40b36e0 0x7f6ff40b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f6ff9cd4700 1 --2- 172.21.15.46:0/1972372346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6fdc003c20 0x7f6ff4153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f6ff9cd4700 1 --2- 172.21.15.46:0/1972372346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff40c72e0 0x7f6ff40c7750 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f6ff9cd4700 1 -- 172.21.15.46:0/1972372346 >> 172.21.15.46:0/1972372346 conn(0x7f6ff4097820 msgr2=0x7f6ff40cb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:59.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f6ff9cd4700 1 -- 172.21.15.46:0/1972372346 shutdown_connections 2024-01-26T23:30:59.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.248+0000 7f6ff9cd4700 1 -- 172.21.15.46:0/1972372346 wait complete. 2024-01-26T23:30:59.273 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-26T23:30:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.403+0000 7f6a50ffd700 1 Processor -- start 2024-01-26T23:30:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.403+0000 7f6a50ffd700 1 -- start start 2024-01-26T23:30:59.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.404+0000 7f6a50ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a4c159790 0x7f6a4c068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.404+0000 7f6a50ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6a4c0c3b10 0x7f6a4c0c3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.404+0000 7f6a50ffd700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4c0c4460 con 0x7f6a4c0cd960 2024-01-26T23:30:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.404+0000 7f6a50ffd700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a4c06a880 con 0x7f6a4c0c3b10 2024-01-26T23:30:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.404+0000 7f6a50ffd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c0690d0 con 0x7f6a4c159790 2024-01-26T23:30:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.404+0000 7f6a4ad9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6a4c0c3b10 0x7f6a4c0c3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.404+0000 7f6a4ad9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6a4c0c3b10 0x7f6a4c0c3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60142/0 (socket says 172.21.15.46:60142) 2024-01-26T23:30:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.404+0000 7f6a4ad9d700 1 -- 172.21.15.46:0/4018146841 learned_addr learned my addr 172.21.15.46:0/4018146841 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.404+0000 7f6a41d9b700 1 --1- 172.21.15.46:0/4018146841 >> v1:172.21.15.46:6789/0 conn(0x7f6a4c0cd960 0x7f6a4c06a170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47000/0 (socket says 172.21.15.46:47000) 2024-01-26T23:30:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.404+0000 7f6a4a59c700 1 --2- 172.21.15.46:0/4018146841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a4c159790 0x7f6a4c068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.405+0000 7f6a4ad9d700 1 -- 172.21.15.46:0/4018146841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a4c159790 msgr2=0x7f6a4c068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:30:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.405+0000 7f6a4ad9d700 1 --2- 172.21.15.46:0/4018146841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a4c159790 0x7f6a4c068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.405+0000 7f6a4ad9d700 1 -- 172.21.15.46:0/4018146841 >> v1:172.21.15.46:6789/0 conn(0x7f6a4c0cd960 legacy=0x7f6a4c06a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:59.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.405+0000 7f6a4ad9d700 1 -- 172.21.15.46:0/4018146841 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4c0693b0 con 0x7f6a4c0c3b10 2024-01-26T23:30:59.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.405+0000 7f6a49d9b700 1 -- 172.21.15.46:0/4018146841 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260489889 0 0) 0x7f6a4c0c4460 con 0x7f6a4c0cd960 2024-01-26T23:30:59.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.405+0000 7f6a4a59c700 1 --2- 172.21.15.46:0/4018146841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a4c159790 0x7f6a4c068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:59.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.405+0000 7f6a4ad9d700 1 --2- 172.21.15.46:0/4018146841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a4c0c3b10 0x7f6a4c0c3f20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6a44009f20 tx=0x7f6a44009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d29be700a9a25a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:59.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.405+0000 7f6a49d9b700 1 -- 172.21.15.46:0/4018146841 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4400e070 con 0x7f6a4c0c3b10 2024-01-26T23:30:59.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.405+0000 7f6a49d9b700 1 -- 172.21.15.46:0/4018146841 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4400ba80 con 0x7f6a4c0c3b10 2024-01-26T23:30:59.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.405+0000 7f6a49d9b700 1 -- 172.21.15.46:0/4018146841 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a440092e0 con 0x7f6a4c0c3b10 2024-01-26T23:30:59.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.406+0000 7f6a50ffd700 1 -- 172.21.15.46:0/4018146841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a4c0c3b10 msgr2=0x7f6a4c0c3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:59.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.406+0000 7f6a50ffd700 1 --2- 172.21.15.46:0/4018146841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a4c0c3b10 0x7f6a4c0c3f20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6a44009f20 tx=0x7f6a44009840 comp rx=0 tx=0).stop 2024-01-26T23:30:59.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.406+0000 7f6a50ffd700 1 -- 172.21.15.46:0/4018146841 shutdown_connections 2024-01-26T23:30:59.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.406+0000 7f6a50ffd700 1 --2- 172.21.15.46:0/4018146841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a4c0c3b10 0x7f6a4c0c3f20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.406+0000 7f6a50ffd700 1 --2- 172.21.15.46:0/4018146841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a4c159790 0x7f6a4c068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.406+0000 7f6a50ffd700 1 -- 172.21.15.46:0/4018146841 >> 172.21.15.46:0/4018146841 conn(0x7f6a4c097820 msgr2=0x7f6a4c068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:59.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.406+0000 7f6a50ffd700 1 -- 172.21.15.46:0/4018146841 shutdown_connections 2024-01-26T23:30:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.406+0000 7f6a50ffd700 1 -- 172.21.15.46:0/4018146841 wait complete. 2024-01-26T23:30:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.407+0000 7f6a50ffd700 1 Processor -- start 2024-01-26T23:30:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.407+0000 7f6a50ffd700 1 -- start start 2024-01-26T23:30:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a50ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a4c0c3b10 0x7f6a4c148460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a50ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a4c0cd960 0x7f6a4c15bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a50ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a4c159790 0x7f6a4c0b3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a50ffd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c099280 con 0x7f6a4c0cd960 2024-01-26T23:30:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a50ffd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c1489a0 con 0x7f6a4c0c3b10 2024-01-26T23:30:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a50ffd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c09b170 con 0x7f6a4c159790 2024-01-26T23:30:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a41d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a4c0cd960 0x7f6a4c15bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a41d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a4c0cd960 0x7f6a4c15bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60148/0 (socket says 172.21.15.46:60148) 2024-01-26T23:30:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a41d9b700 1 -- 172.21.15.46:0/2858461357 learned_addr learned my addr 172.21.15.46:0/2858461357 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:30:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a4ad9d700 1 --2- 172.21.15.46:0/2858461357 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a4c159790 0x7f6a4c0b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a4a59c700 1 --2- 172.21.15.46:0/2858461357 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a4c0c3b10 0x7f6a4c148460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a41d9b700 1 -- 172.21.15.46:0/2858461357 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a4c159790 msgr2=0x7f6a4c0b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a41d9b700 1 --2- 172.21.15.46:0/2858461357 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a4c159790 0x7f6a4c0b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a41d9b700 1 -- 172.21.15.46:0/2858461357 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a4c0c3b10 msgr2=0x7f6a4c148460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a41d9b700 1 --2- 172.21.15.46:0/2858461357 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a4c0c3b10 0x7f6a4c148460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a41d9b700 1 -- 172.21.15.46:0/2858461357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a44009ae0 con 0x7f6a4c0cd960 2024-01-26T23:30:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a4a59c700 1 --2- 172.21.15.46:0/2858461357 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a4c0c3b10 0x7f6a4c148460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:30:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.408+0000 7f6a41d9b700 1 --2- 172.21.15.46:0/2858461357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a4c0cd960 0x7f6a4c15bf40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6a340011e0 tx=0x7f6a34007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.409+0000 7f6a42ffd700 1 -- 172.21.15.46:0/2858461357 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a34005a70 con 0x7f6a4c0cd960 2024-01-26T23:30:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.409+0000 7f6a42ffd700 1 -- 172.21.15.46:0/2858461357 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a34005bd0 con 0x7f6a4c0cd960 2024-01-26T23:30:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.409+0000 7f6a42ffd700 1 -- 172.21.15.46:0/2858461357 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a340070d0 con 0x7f6a4c0cd960 2024-01-26T23:30:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.409+0000 7f6a50ffd700 1 -- 172.21.15.46:0/2858461357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a4c09a840 con 0x7f6a4c0cd960 2024-01-26T23:30:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.409+0000 7f6a50ffd700 1 -- 172.21.15.46:0/2858461357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a4c09ae00 con 0x7f6a4c0cd960 2024-01-26T23:30:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.410+0000 7f6a50ffd700 1 -- 172.21.15.46:0/2858461357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a4c09b2e0 con 0x7f6a4c0cd960 2024-01-26T23:30:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.410+0000 7f6a42ffd700 1 -- 172.21.15.46:0/2858461357 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6a34007480 con 0x7f6a4c0cd960 2024-01-26T23:30:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.411+0000 7f6a42ffd700 1 --2- 172.21.15.46:0/2858461357 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6a2c066f30 0x7f6a2c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:30:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.411+0000 7f6a42ffd700 1 -- 172.21.15.46:0/2858461357 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f6a34083ac0 con 0x7f6a4c0cd960 2024-01-26T23:30:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.411+0000 7f6a4a59c700 1 --2- 172.21.15.46:0/2858461357 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6a2c066f30 0x7f6a2c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:30:59.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.411+0000 7f6a4a59c700 1 --2- 172.21.15.46:0/2858461357 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6a2c066f30 0x7f6a2c0693e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c006ea0 tx=0x7f6a3c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:30:59.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.415+0000 7f6a42ffd700 1 -- 172.21.15.46:0/2858461357 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6a3404b5a0 con 0x7f6a4c0cd960 2024-01-26T23:30:59.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.667+0000 7f6a50ffd700 1 -- 172.21.15.46:0/2858461357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6a4c04c710 con 0x7f6a4c0cd960 2024-01-26T23:30:59.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.667+0000 7f6a42ffd700 1 -- 172.21.15.46:0/2858461357 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6a34083e60 con 0x7f6a4c0cd960 2024-01-26T23:30:59.669 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-26T23:30:59.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.669+0000 7f6a50ffd700 1 -- 172.21.15.46:0/2858461357 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6a2c066f30 msgr2=0x7f6a2c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:59.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.669+0000 7f6a50ffd700 1 --2- 172.21.15.46:0/2858461357 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6a2c066f30 0x7f6a2c0693e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c006ea0 tx=0x7f6a3c008040 comp rx=0 tx=0).stop 2024-01-26T23:30:59.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.669+0000 7f6a50ffd700 1 -- 172.21.15.46:0/2858461357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a4c0cd960 msgr2=0x7f6a4c15bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:30:59.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.669+0000 7f6a50ffd700 1 --2- 172.21.15.46:0/2858461357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a4c0cd960 0x7f6a4c15bf40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6a340011e0 tx=0x7f6a34007e10 comp rx=0 tx=0).stop 2024-01-26T23:30:59.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.670+0000 7f6a50ffd700 1 -- 172.21.15.46:0/2858461357 shutdown_connections 2024-01-26T23:30:59.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.670+0000 7f6a50ffd700 1 --2- 172.21.15.46:0/2858461357 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6a2c066f30 0x7f6a2c0693e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.670+0000 7f6a50ffd700 1 --2- 172.21.15.46:0/2858461357 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a4c159790 0x7f6a4c0b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.670+0000 7f6a50ffd700 1 --2- 172.21.15.46:0/2858461357 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a4c0c3b10 0x7f6a4c148460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.670+0000 7f6a50ffd700 1 --2- 172.21.15.46:0/2858461357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a4c0cd960 0x7f6a4c15bf40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:30:59.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.670+0000 7f6a50ffd700 1 -- 172.21.15.46:0/2858461357 >> 172.21.15.46:0/2858461357 conn(0x7f6a4c097820 msgr2=0x7f6a4c06a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:30:59.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.670+0000 7f6a50ffd700 1 -- 172.21.15.46:0/2858461357 shutdown_connections 2024-01-26T23:30:59.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:30:59.670+0000 7f6a50ffd700 1 -- 172.21.15.46:0/2858461357 wait complete. 2024-01-26T23:30:59.685 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-26T23:30:59.685 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-26T23:30:59.685 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-26T23:30:59.685 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-26T23:30:59.686 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-26T23:30:59.686 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-26T23:30:59.686 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-26T23:30:59.686 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-26T23:30:59.686 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:30:59.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:59.686 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-26T23:30:59.705 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-26T23:30:59.706 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-26T23:30:59.706 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-26T23:30:59.706 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:59.706 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-26T23:30:59.706 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:59.706 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:30:59.706 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:59.706 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-26T23:30:59.706 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:30:59.706 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:59.706 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-26T23:30:59.707 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-26T23:30:59.707 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:59.707 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-26T23:30:59.707 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-26T23:30:59.707 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42271sec preferred_lft 42271sec 2024-01-26T23:30:59.707 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-26T23:30:59.707 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:59.707 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:30:59.707 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:59.707 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-26T23:30:59.707 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-26T23:30:59.707 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:30:59.708 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-26T23:30:59.708 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:59.708 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::cce6:a7ff:fe4f:38d/64 scope link 2024-01-26T23:30:59.708 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:59.708 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:30:59.708 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 76:ef:90:4c:9d:6f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-26T23:30:59.708 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::74ef:90ff:fe4c:9d6f/64 scope link 2024-01-26T23:30:59.708 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:59.708 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:30:59.708 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-26T23:30:59.708 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::2c73:afff:fef0:6f39/64 scope link 2024-01-26T23:30:59.709 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:30:59.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:59.709 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-26T23:30:59.777 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-26T23:30:59.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:59.779 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-26T23:30:59.844 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-26T23:30:59.845 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-26T23:30:59.845 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-26T23:30:59.845 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-26T23:30:59.845 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-26T23:30:59.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:59.845 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:30:59.845 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:30:59.845 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-26T23:30:59.845 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-26T23:30:59.845 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-26T23:30:59.846 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:30:59.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:59.970 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-26T23:30:59.989 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-26T23:30:59.990 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-26T23:30:59.990 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:30:59.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:30:59.990 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-01-26T23:31:00.064 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-26T23:31:00.064 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:31:00.064 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-26T23:31:00.127 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:31:00.127 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:31:00.127 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:31:00.189 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:31:00.266 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:31:00.266 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:31:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:31:00.341 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:31:00.341 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:31:00.342 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:31:00.403 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-26T23:31:00.403 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-26T23:31:00.446 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:31:00.481 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:31:00.481 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:31:00.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-01-26T23:31:00.546+0000 7fd83f233540 -1 init, newargv = 0x5601f2519f90 newargc=15 2024-01-26T23:31:00.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[58537]: starting ceph client 2024-01-26T23:31:00.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[58537]: starting fuse 2024-01-26T23:31:00.553 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:31:00.555 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:31:00.555 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:31:00.555 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:31:00.615 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-26T23:31:00.615 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-26T23:31:00.615 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-26T23:31:00.615 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-26T23:31:00.615 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-26T23:31:00.615 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-26T23:31:00.615 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-26T23:31:00.615 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:31:00.615 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-26T23:31:00.616 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-26T23:31:00.616 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-26T23:31:00.616 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-26T23:31:00.616 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:31:00.616 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-26T23:31:00.616 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-26T23:31:00.616 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-26T23:31:00.616 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-26T23:31:00.616 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:31:00.616 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-26T23:31:00.616 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-26T23:31:00.616 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-26T23:31:00.616 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-26T23:31:00.616 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-26T23:31:00.616 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-26T23:31:00.617 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-26T23:31:00.617 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-26T23:31:00.617 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:31:00.617 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-01-26T23:31:00.617 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-26T23:31:00.753 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.58537.asok 2024-01-26T23:31:00.934 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.58537.asok 2024-01-26T23:31:00.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:31:00.935 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.58537.asok status 2024-01-26T23:31:01.109 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:31:01.109 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-26T23:31:01.109 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-26T23:31:01.109 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-26T23:31:01.109 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-01-26T23:31:01.109 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-26T23:31:01.109 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-26T23:31:01.109 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "58537", 2024-01-26T23:31:01.109 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-26T23:31:01.109 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:31:01.110 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-26T23:31:01.110 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-26T23:31:01.110 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5238, 2024-01-26T23:31:01.110 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-26T23:31:01.110 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-26T23:31:01.110 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-26T23:31:01.110 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5238 2024-01-26T23:31:01.110 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:31:01.110 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:31:01.110 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:31:01.110 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:31:01.110 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3037143548 2024-01-26T23:31:01.110 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:31:01.111 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:31:01.111 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:31:01.111 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:31:01.111 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:31:01.111 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3037143548 2024-01-26T23:31:01.111 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:31:01.111 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.5238 192.168.0.1:0/3037143548", 2024-01-26T23:31:01.111 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/3037143548", 2024-01-26T23:31:01.111 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-26T23:31:01.111 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 18, 2024-01-26T23:31:01.111 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 38, 2024-01-26T23:31:01.111 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-26T23:31:01.111 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-26T23:31:01.112 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-26T23:31:01.112 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:31:01.120 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:31:01.120 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-26T23:31:01.120 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-26T23:31:01.139 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-26T23:31:01.139 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-26T23:31:01.140 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:31:01.140 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:31:01.214 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-26T23:31:01.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.390+0000 7f8755616700 1 Processor -- start 2024-01-26T23:31:01.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.390+0000 7f8755616700 1 -- start start 2024-01-26T23:31:01.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.392+0000 7f8755616700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f87500b4180 0x7f8750157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:01.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.392+0000 7f8755616700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8750158ae0 con 0x7f8750157340 2024-01-26T23:31:01.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.392+0000 7f8755616700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f875015b650 con 0x7f87500ae520 2024-01-26T23:31:01.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.392+0000 7f8755616700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8750158520 con 0x7f87500b4180 2024-01-26T23:31:01.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.392+0000 7f874effd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8750157340 0x7f875015bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42024/0 (socket says 172.21.15.46:42024) 2024-01-26T23:31:01.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.392+0000 7f874effd700 1 -- 172.21.15.46:0/3967426659 learned_addr learned my addr 172.21.15.46:0/3967426659 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:31:01.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.392+0000 7f874f7fe700 1 --2- 172.21.15.46:0/3967426659 >> v2:172.21.15.46:3300/0 conn(0x7f87500b4180 0x7f8750157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:01.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.392+0000 7f874f7fe700 1 -- 172.21.15.46:0/3967426659 >> v1:172.21.15.106:6789/0 conn(0x7f87500ae520 legacy=0x7f87500b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:01.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.392+0000 7f874f7fe700 1 -- 172.21.15.46:0/3967426659 >> v1:172.21.15.46:6789/0 conn(0x7f8750157340 legacy=0x7f875015bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:01.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.392+0000 7f874f7fe700 1 -- 172.21.15.46:0/3967426659 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8750158800 con 0x7f87500b4180 2024-01-26T23:31:01.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.392+0000 7f874f7fe700 1 --2- 172.21.15.46:0/3967426659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87500b4180 0x7f8750157710 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8748009e60 tx=0x7f8748009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d22607ac34eea3c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:01.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.393+0000 7f874dffb700 1 -- 172.21.15.46:0/3967426659 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874800e070 con 0x7f87500b4180 2024-01-26T23:31:01.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.393+0000 7f874dffb700 1 -- 172.21.15.46:0/3967426659 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874800bbf0 con 0x7f87500b4180 2024-01-26T23:31:01.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.393+0000 7f874dffb700 1 -- 172.21.15.46:0/3967426659 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8748012430 con 0x7f87500b4180 2024-01-26T23:31:01.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.393+0000 7f8755616700 1 -- 172.21.15.46:0/3967426659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87500b4180 msgr2=0x7f8750157710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:01.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.393+0000 7f8755616700 1 --2- 172.21.15.46:0/3967426659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87500b4180 0x7f8750157710 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8748009e60 tx=0x7f8748009780 comp rx=0 tx=0).stop 2024-01-26T23:31:01.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.393+0000 7f8755616700 1 -- 172.21.15.46:0/3967426659 shutdown_connections 2024-01-26T23:31:01.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.393+0000 7f8755616700 1 --2- 172.21.15.46:0/3967426659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87500b4180 0x7f8750157710 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:01.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.393+0000 7f8755616700 1 -- 172.21.15.46:0/3967426659 >> 172.21.15.46:0/3967426659 conn(0x7f87500ce530 msgr2=0x7f87500ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:01.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.393+0000 7f8755616700 1 -- 172.21.15.46:0/3967426659 shutdown_connections 2024-01-26T23:31:01.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.393+0000 7f8755616700 1 -- 172.21.15.46:0/3967426659 wait complete. 2024-01-26T23:31:01.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.395+0000 7f8755616700 1 Processor -- start 2024-01-26T23:31:01.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.395+0000 7f8755616700 1 -- start start 2024-01-26T23:31:01.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.395+0000 7f8755616700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87500ae520 0x7f875014fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:01.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.395+0000 7f8755616700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87500b4180 0x7f8750155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:01.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.395+0000 7f8755616700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8750157340 0x7f8750156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:01.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.395+0000 7f8755616700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87500cbcc0 con 0x7f8750157340 2024-01-26T23:31:01.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.395+0000 7f8755616700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8750150010 con 0x7f87500b4180 2024-01-26T23:31:01.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.395+0000 7f8755616700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87500cb0b0 con 0x7f87500ae520 2024-01-26T23:31:01.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.396+0000 7f874f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8750157340 0x7f8750156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:01.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.396+0000 7f874f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8750157340 0x7f8750156110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53364/0 (socket says 172.21.15.46:53364) 2024-01-26T23:31:01.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.396+0000 7f874f7fe700 1 -- 172.21.15.46:0/3012502954 learned_addr learned my addr 172.21.15.46:0/3012502954 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:01.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.396+0000 7f874effd700 1 --2- 172.21.15.46:0/3012502954 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87500ae520 0x7f875014fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:01.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.396+0000 7f874e7fc700 1 --2- 172.21.15.46:0/3012502954 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87500b4180 0x7f8750155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:01.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.396+0000 7f874f7fe700 1 -- 172.21.15.46:0/3012502954 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87500ae520 msgr2=0x7f875014fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:01.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.396+0000 7f874f7fe700 1 --2- 172.21.15.46:0/3012502954 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87500ae520 0x7f875014fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:01.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.396+0000 7f874f7fe700 1 -- 172.21.15.46:0/3012502954 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87500b4180 msgr2=0x7f8750155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:01.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.396+0000 7f874f7fe700 1 --2- 172.21.15.46:0/3012502954 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87500b4180 0x7f8750155bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:01.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.396+0000 7f874f7fe700 1 -- 172.21.15.46:0/3012502954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8748009a20 con 0x7f8750157340 2024-01-26T23:31:01.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.396+0000 7f874f7fe700 1 --2- 172.21.15.46:0/3012502954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8750157340 0x7f8750156110 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f875006f390 tx=0x7f8748018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:01.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.396+0000 7f8736ffd700 1 -- 172.21.15.46:0/3012502954 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8748007000 con 0x7f8750157340 2024-01-26T23:31:01.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.396+0000 7f8736ffd700 1 -- 172.21.15.46:0/3012502954 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8748012e30 con 0x7f8750157340 2024-01-26T23:31:01.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.396+0000 7f8736ffd700 1 -- 172.21.15.46:0/3012502954 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874801d880 con 0x7f8750157340 2024-01-26T23:31:01.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.396+0000 7f8755616700 1 -- 172.21.15.46:0/3012502954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87500cb390 con 0x7f8750157340 2024-01-26T23:31:01.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.396+0000 7f8755616700 1 -- 172.21.15.46:0/3012502954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87500cb950 con 0x7f8750157340 2024-01-26T23:31:01.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.398+0000 7f8755616700 1 -- 172.21.15.46:0/3012502954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f875004c710 con 0x7f8750157340 2024-01-26T23:31:01.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.398+0000 7f8736ffd700 1 -- 172.21.15.46:0/3012502954 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f874801dc30 con 0x7f8750157340 2024-01-26T23:31:01.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.398+0000 7f8736ffd700 1 --2- 172.21.15.46:0/3012502954 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f873c066f30 0x7f873c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.398+0000 7f8736ffd700 1 -- 172.21.15.46:0/3012502954 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f8748086d30 con 0x7f8750157340 2024-01-26T23:31:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.399+0000 7f874effd700 1 --2- 172.21.15.46:0/3012502954 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f873c066f30 0x7f873c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.399+0000 7f874effd700 1 --2- 172.21.15.46:0/3012502954 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f873c066f30 0x7f873c0693e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8740007130 tx=0x7f8740007eb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:01.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.403+0000 7f8736ffd700 1 -- 172.21.15.46:0/3012502954 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8748053e60 con 0x7f8750157340 2024-01-26T23:31:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.641+0000 7f8755616700 1 -- 172.21.15.46:0/3012502954 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f875015b2e0 con 0x7f873c066f30 2024-01-26T23:31:01.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.644+0000 7f8736ffd700 1 -- 172.21.15.46:0/3012502954 <== mgr.4102 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f875015b2e0 con 0x7f873c066f30 2024-01-26T23:31:01.645 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-26T23:31:01.645 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:31:01.645 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephfs" 2024-01-26T23:31:01.645 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:31:01.645 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-26T23:31:01.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.646+0000 7f8755616700 1 -- 172.21.15.46:0/3012502954 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f873c066f30 msgr2=0x7f873c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:01.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.646+0000 7f8755616700 1 --2- 172.21.15.46:0/3012502954 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f873c066f30 0x7f873c0693e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8740007130 tx=0x7f8740007eb0 comp rx=0 tx=0).stop 2024-01-26T23:31:01.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.646+0000 7f8755616700 1 -- 172.21.15.46:0/3012502954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8750157340 msgr2=0x7f8750156110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:01.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.646+0000 7f8755616700 1 --2- 172.21.15.46:0/3012502954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8750157340 0x7f8750156110 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f875006f390 tx=0x7f8748018000 comp rx=0 tx=0).stop 2024-01-26T23:31:01.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.646+0000 7f8755616700 1 -- 172.21.15.46:0/3012502954 shutdown_connections 2024-01-26T23:31:01.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.646+0000 7f8755616700 1 --2- 172.21.15.46:0/3012502954 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f873c066f30 0x7f873c0693e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:01.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.646+0000 7f8755616700 1 --2- 172.21.15.46:0/3012502954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8750157340 0x7f8750156110 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:01.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.646+0000 7f8755616700 1 --2- 172.21.15.46:0/3012502954 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87500ae520 0x7f875014fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:01.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.646+0000 7f8755616700 1 --2- 172.21.15.46:0/3012502954 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87500b4180 0x7f8750155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:01.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.646+0000 7f8755616700 1 -- 172.21.15.46:0/3012502954 >> 172.21.15.46:0/3012502954 conn(0x7f87500ce530 msgr2=0x7f87500cc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:01.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.646+0000 7f8755616700 1 -- 172.21.15.46:0/3012502954 shutdown_connections 2024-01-26T23:31:01.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.646+0000 7f8755616700 1 -- 172.21.15.46:0/3012502954 wait complete. 2024-01-26T23:31:01.661 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-26T23:31:01.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.794+0000 7f6f17484700 1 Processor -- start 2024-01-26T23:31:01.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.794+0000 7f6f17484700 1 -- start start 2024-01-26T23:31:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.795+0000 7f6f17484700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6f10153c40 0x7f6f1006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.795+0000 7f6f17484700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f10154010 0x7f6f1006dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.795+0000 7f6f17484700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f1015ad60 con 0x7f6f100b1e80 2024-01-26T23:31:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.795+0000 7f6f17484700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f100b1d20 con 0x7f6f10153c40 2024-01-26T23:31:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.795+0000 7f6f17484700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1015a150 con 0x7f6f10154010 2024-01-26T23:31:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.795+0000 7f6f15a21700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6f100b1e80 0x7f6f1015d400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42026/0 (socket says 172.21.15.46:42026) 2024-01-26T23:31:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.795+0000 7f6f15a21700 1 -- 172.21.15.46:0/3970195511 learned_addr learned my addr 172.21.15.46:0/3970195511 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:31:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.795+0000 7f6f14a1f700 1 --2- 172.21.15.46:0/3970195511 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f10154010 0x7f6f1006dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.795+0000 7f6f15220700 1 --2- 172.21.15.46:0/3970195511 >> v2:172.21.15.106:3300/0 conn(0x7f6f10153c40 0x7f6f1006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.796+0000 7f6f07fff700 1 -- 172.21.15.46:0/3970195511 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943252212 0 0) 0x7f6f1015ad60 con 0x7f6f100b1e80 2024-01-26T23:31:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.796+0000 7f6f07fff700 1 -- 172.21.15.46:0/3970195511 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6efc003570 con 0x7f6f100b1e80 2024-01-26T23:31:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.796+0000 7f6f14a1f700 1 -- 172.21.15.46:0/3970195511 >> v2:172.21.15.106:3300/0 conn(0x7f6f10153c40 msgr2=0x7f6f1006d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.796+0000 7f6f14a1f700 1 --2- 172.21.15.46:0/3970195511 >> v2:172.21.15.106:3300/0 conn(0x7f6f10153c40 0x7f6f1006d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.796+0000 7f6f14a1f700 1 -- 172.21.15.46:0/3970195511 >> v1:172.21.15.46:6789/0 conn(0x7f6f100b1e80 legacy=0x7f6f1015d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.796+0000 7f6f14a1f700 1 -- 172.21.15.46:0/3970195511 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f1015a430 con 0x7f6f10154010 2024-01-26T23:31:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.796+0000 7f6f15220700 1 --2- 172.21.15.46:0/3970195511 >> v2:172.21.15.106:3300/0 conn(0x7f6f10153c40 0x7f6f1006d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:31:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.796+0000 7f6f14a1f700 1 --2- 172.21.15.46:0/3970195511 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f10154010 0x7f6f1006dbe0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0098e0 tx=0x7f6f0c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=534cfb0878ba183 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.796+0000 7f6f07fff700 1 -- 172.21.15.46:0/3970195511 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c011630 con 0x7f6f10154010 2024-01-26T23:31:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.796+0000 7f6f07fff700 1 -- 172.21.15.46:0/3970195511 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f0c011790 con 0x7f6f10154010 2024-01-26T23:31:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.796+0000 7f6f07fff700 1 -- 172.21.15.46:0/3970195511 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c011b30 con 0x7f6f10154010 2024-01-26T23:31:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.797+0000 7f6f17484700 1 -- 172.21.15.46:0/3970195511 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f10154010 msgr2=0x7f6f1006dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.797+0000 7f6f17484700 1 --2- 172.21.15.46:0/3970195511 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f10154010 0x7f6f1006dbe0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0098e0 tx=0x7f6f0c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:31:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.797+0000 7f6f17484700 1 -- 172.21.15.46:0/3970195511 shutdown_connections 2024-01-26T23:31:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.797+0000 7f6f17484700 1 --2- 172.21.15.46:0/3970195511 >> v2:172.21.15.106:3300/0 conn(0x7f6f10153c40 0x7f6f1006d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.797+0000 7f6f17484700 1 --2- 172.21.15.46:0/3970195511 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f10154010 0x7f6f1006dbe0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.797+0000 7f6f17484700 1 -- 172.21.15.46:0/3970195511 >> 172.21.15.46:0/3970195511 conn(0x7f6f10067930 msgr2=0x7f6f10067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.797+0000 7f6f17484700 1 -- 172.21.15.46:0/3970195511 shutdown_connections 2024-01-26T23:31:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.797+0000 7f6f17484700 1 -- 172.21.15.46:0/3970195511 wait complete. 2024-01-26T23:31:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.798+0000 7f6f17484700 1 Processor -- start 2024-01-26T23:31:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.798+0000 7f6f17484700 1 -- start start 2024-01-26T23:31:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.799+0000 7f6f17484700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f100b1e80 0x7f6f100c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.799+0000 7f6f17484700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6f10153c40 0x7f6f1016d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.799+0000 7f6f17484700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10154010 0x7f6f1014ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.799+0000 7f6f17484700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f100c7760 con 0x7f6f10154010 2024-01-26T23:31:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.799+0000 7f6f17484700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f100cefc0 con 0x7f6f10153c40 2024-01-26T23:31:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.799+0000 7f6f17484700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f100c82a0 con 0x7f6f100b1e80 2024-01-26T23:31:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.799+0000 7f6f15a21700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10154010 0x7f6f1014ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.799+0000 7f6f15a21700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10154010 0x7f6f1014ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53366/0 (socket says 172.21.15.46:53366) 2024-01-26T23:31:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.799+0000 7f6f15a21700 1 -- 172.21.15.46:0/3624855922 learned_addr learned my addr 172.21.15.46:0/3624855922 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.799+0000 7f6f14a1f700 1 --2- 172.21.15.46:0/3624855922 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6f10153c40 0x7f6f1016d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.799+0000 7f6f15220700 1 --2- 172.21.15.46:0/3624855922 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f100b1e80 0x7f6f100c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.800+0000 7f6f15a21700 1 -- 172.21.15.46:0/3624855922 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f100b1e80 msgr2=0x7f6f100c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.800+0000 7f6f15a21700 1 --2- 172.21.15.46:0/3624855922 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f100b1e80 0x7f6f100c8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.800+0000 7f6f15a21700 1 -- 172.21.15.46:0/3624855922 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6f10153c40 msgr2=0x7f6f1016d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.800+0000 7f6f15a21700 1 --2- 172.21.15.46:0/3624855922 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6f10153c40 0x7f6f1016d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.800+0000 7f6f15a21700 1 -- 172.21.15.46:0/3624855922 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f00009420 con 0x7f6f10154010 2024-01-26T23:31:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.800+0000 7f6f15a21700 1 --2- 172.21.15.46:0/3624855922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10154010 0x7f6f1014ec80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800cca0 tx=0x7f6f08001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.800+0000 7f6f057fa700 1 -- 172.21.15.46:0/3624855922 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0800ab30 con 0x7f6f10154010 2024-01-26T23:31:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.800+0000 7f6f057fa700 1 -- 172.21.15.46:0/3624855922 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f0800ac90 con 0x7f6f10154010 2024-01-26T23:31:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.800+0000 7f6f057fa700 1 -- 172.21.15.46:0/3624855922 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f08007300 con 0x7f6f10154010 2024-01-26T23:31:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.800+0000 7f6f17484700 1 -- 172.21.15.46:0/3624855922 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f0c009420 con 0x7f6f10154010 2024-01-26T23:31:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.800+0000 7f6f17484700 1 -- 172.21.15.46:0/3624855922 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f100c7c50 con 0x7f6f10154010 2024-01-26T23:31:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.801+0000 7f6f17484700 1 -- 172.21.15.46:0/3624855922 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f1004c710 con 0x7f6f10154010 2024-01-26T23:31:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.802+0000 7f6f057fa700 1 -- 172.21.15.46:0/3624855922 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6f08007730 con 0x7f6f10154010 2024-01-26T23:31:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.802+0000 7f6f057fa700 1 --2- 172.21.15.46:0/3624855922 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6ef4066f30 0x7f6ef40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.802+0000 7f6f057fa700 1 -- 172.21.15.46:0/3624855922 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f6f08084b90 con 0x7f6f10154010 2024-01-26T23:31:01.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.802+0000 7f6f15220700 1 --2- 172.21.15.46:0/3624855922 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6ef4066f30 0x7f6ef40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:01.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.803+0000 7f6f15220700 1 --2- 172.21.15.46:0/3624855922 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6ef4066f30 0x7f6ef40693e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6f000093f0 tx=0x7f6f00008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:01.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:01.807+0000 7f6f057fa700 1 -- 172.21.15.46:0/3624855922 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6f08051cb0 con 0x7f6f10154010 2024-01-26T23:31:02.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.072+0000 7f6f17484700 1 -- 172.21.15.46:0/3624855922 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f6f100c83e0 con 0x7f6f10154010 2024-01-26T23:31:02.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.458+0000 7f6f057fa700 1 -- 172.21.15.46:0/3624855922 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 6) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f6f080079e0 con 0x7f6f10154010 2024-01-26T23:31:02.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.460+0000 7f6f057fa700 1 -- 172.21.15.46:0/3624855922 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7f6f08053ab0 con 0x7f6f10154010 2024-01-26T23:31:02.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.463+0000 7f6f17484700 1 -- 172.21.15.46:0/3624855922 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6ef4066f30 msgr2=0x7f6ef40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:02.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.463+0000 7f6f17484700 1 --2- 172.21.15.46:0/3624855922 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6ef4066f30 0x7f6ef40693e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6f000093f0 tx=0x7f6f00008040 comp rx=0 tx=0).stop 2024-01-26T23:31:02.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.463+0000 7f6f17484700 1 -- 172.21.15.46:0/3624855922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10154010 msgr2=0x7f6f1014ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:02.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.463+0000 7f6f17484700 1 --2- 172.21.15.46:0/3624855922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10154010 0x7f6f1014ec80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800cca0 tx=0x7f6f08001ae0 comp rx=0 tx=0).stop 2024-01-26T23:31:02.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.463+0000 7f6f17484700 1 -- 172.21.15.46:0/3624855922 shutdown_connections 2024-01-26T23:31:02.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.463+0000 7f6f17484700 1 --2- 172.21.15.46:0/3624855922 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6ef4066f30 0x7f6ef40693e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:02.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.463+0000 7f6f17484700 1 --2- 172.21.15.46:0/3624855922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10154010 0x7f6f1014ec80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:02.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.463+0000 7f6f17484700 1 --2- 172.21.15.46:0/3624855922 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f100b1e80 0x7f6f100c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:02.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.463+0000 7f6f17484700 1 --2- 172.21.15.46:0/3624855922 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6f10153c40 0x7f6f1016d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:02.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.463+0000 7f6f17484700 1 -- 172.21.15.46:0/3624855922 >> 172.21.15.46:0/3624855922 conn(0x7f6f10067930 msgr2=0x7f6f1015dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:02.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.463+0000 7f6f17484700 1 -- 172.21.15.46:0/3624855922 shutdown_connections 2024-01-26T23:31:02.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.463+0000 7f6f17484700 1 -- 172.21.15.46:0/3624855922 wait complete. 2024-01-26T23:31:02.481 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-26T23:31:02.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.619+0000 7f705df4b700 1 Processor -- start 2024-01-26T23:31:02.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.619+0000 7f705df4b700 1 -- start start 2024-01-26T23:31:02.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.620+0000 7f705df4b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7058159cb0 0x7f70580cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:02.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.620+0000 7f705df4b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70580cc100 con 0x7f70580afb40 2024-01-26T23:31:02.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.620+0000 7f705df4b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70580cb850 con 0x7f70580cf050 2024-01-26T23:31:02.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.620+0000 7f705df4b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70580cbaf0 con 0x7f7058159cb0 2024-01-26T23:31:02.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.620+0000 7f7057fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f70580afb40 0x7f70580c5530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42030/0 (socket says 172.21.15.46:42030) 2024-01-26T23:31:02.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.620+0000 7f7057fff700 1 -- 172.21.15.46:0/429648424 learned_addr learned my addr 172.21.15.46:0/429648424 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:31:02.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.620+0000 7f70577fe700 1 --2- 172.21.15.46:0/429648424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7058159cb0 0x7f70580cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:02.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.620+0000 7f70567fc700 1 -- 172.21.15.46:0/429648424 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721942265 0 0) 0x7f70580cc100 con 0x7f70580afb40 2024-01-26T23:31:02.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.621+0000 7f70567fc700 1 -- 172.21.15.46:0/429648424 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7040003570 con 0x7f70580afb40 2024-01-26T23:31:02.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.621+0000 7f70567fc700 1 -- 172.21.15.46:0/429648424 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377713827 0 0) 0x7f70580cb850 con 0x7f70580cf050 2024-01-26T23:31:02.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.621+0000 7f70567fc700 1 -- 172.21.15.46:0/429648424 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70580cc100 con 0x7f70580cf050 2024-01-26T23:31:02.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.621+0000 7f70577fe700 1 -- 172.21.15.46:0/429648424 >> v1:172.21.15.106:6789/0 conn(0x7f70580cf050 legacy=0x7f70580af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:02.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.621+0000 7f70577fe700 1 -- 172.21.15.46:0/429648424 >> v1:172.21.15.46:6789/0 conn(0x7f70580afb40 legacy=0x7f70580c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:02.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.621+0000 7f70577fe700 1 -- 172.21.15.46:0/429648424 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70580cfc80 con 0x7f7058159cb0 2024-01-26T23:31:02.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.621+0000 7f70577fe700 1 --2- 172.21.15.46:0/429648424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7058159cb0 0x7f70580cc590 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7048009ee0 tx=0x7f7048009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c23b37fef2c8c2c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:02.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.621+0000 7f70567fc700 1 -- 172.21.15.46:0/429648424 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704800e070 con 0x7f7058159cb0 2024-01-26T23:31:02.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.621+0000 7f70567fc700 1 -- 172.21.15.46:0/429648424 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704800bb80 con 0x7f7058159cb0 2024-01-26T23:31:02.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.621+0000 7f70567fc700 1 -- 172.21.15.46:0/429648424 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7048012430 con 0x7f7058159cb0 2024-01-26T23:31:02.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.622+0000 7f705df4b700 1 -- 172.21.15.46:0/429648424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7058159cb0 msgr2=0x7f70580cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:02.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.622+0000 7f705df4b700 1 --2- 172.21.15.46:0/429648424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7058159cb0 0x7f70580cc590 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7048009ee0 tx=0x7f7048009780 comp rx=0 tx=0).stop 2024-01-26T23:31:02.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.622+0000 7f705df4b700 1 -- 172.21.15.46:0/429648424 shutdown_connections 2024-01-26T23:31:02.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.622+0000 7f705df4b700 1 --2- 172.21.15.46:0/429648424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7058159cb0 0x7f70580cc590 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:02.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.622+0000 7f705df4b700 1 -- 172.21.15.46:0/429648424 >> 172.21.15.46:0/429648424 conn(0x7f7058087f60 msgr2=0x7f705806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:02.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.622+0000 7f705df4b700 1 -- 172.21.15.46:0/429648424 shutdown_connections 2024-01-26T23:31:02.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.622+0000 7f705df4b700 1 -- 172.21.15.46:0/429648424 wait complete. 2024-01-26T23:31:02.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.623+0000 7f705df4b700 1 Processor -- start 2024-01-26T23:31:02.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.623+0000 7f705df4b700 1 -- start start 2024-01-26T23:31:02.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.624+0000 7f705df4b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70580afb40 0x7f70580b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:02.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.624+0000 7f705df4b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70580cf050 0x7f70581475c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:02.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.624+0000 7f705df4b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7058159cb0 0x7f70580b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.624+0000 7f705df4b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70580b2e50 con 0x7f70580afb40 2024-01-26T23:31:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.624+0000 7f705df4b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70580b02e0 con 0x7f7058159cb0 2024-01-26T23:31:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.624+0000 7f705df4b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70580b2800 con 0x7f70580cf050 2024-01-26T23:31:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.624+0000 7f70577fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70580afb40 0x7f70580b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.624+0000 7f70577fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70580afb40 0x7f70580b3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53394/0 (socket says 172.21.15.46:53394) 2024-01-26T23:31:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.624+0000 7f70577fe700 1 -- 172.21.15.46:0/2868818265 learned_addr learned my addr 172.21.15.46:0/2868818265 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.624+0000 7f7057fff700 1 --2- 172.21.15.46:0/2868818265 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7058159cb0 0x7f70580b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.624+0000 7f7056ffd700 1 --2- 172.21.15.46:0/2868818265 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70580cf050 0x7f70581475c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.625+0000 7f70577fe700 1 -- 172.21.15.46:0/2868818265 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70580cf050 msgr2=0x7f70581475c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.625+0000 7f70577fe700 1 --2- 172.21.15.46:0/2868818265 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70580cf050 0x7f70581475c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.625+0000 7f70577fe700 1 -- 172.21.15.46:0/2868818265 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7058159cb0 msgr2=0x7f70580b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.625+0000 7f70577fe700 1 --2- 172.21.15.46:0/2868818265 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7058159cb0 0x7f70580b1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.625+0000 7f70577fe700 1 -- 172.21.15.46:0/2868818265 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7048009a20 con 0x7f70580afb40 2024-01-26T23:31:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.625+0000 7f70577fe700 1 --2- 172.21.15.46:0/2868818265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70580afb40 0x7f70580b3550 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f70480097c0 tx=0x7f7048019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.625+0000 7f703f7fe700 1 -- 172.21.15.46:0/2868818265 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7048012cd0 con 0x7f70580afb40 2024-01-26T23:31:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.625+0000 7f703f7fe700 1 -- 172.21.15.46:0/2868818265 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7048012e30 con 0x7f70580afb40 2024-01-26T23:31:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.625+0000 7f705df4b700 1 -- 172.21.15.46:0/2868818265 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70580b2ae0 con 0x7f70580afb40 2024-01-26T23:31:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.625+0000 7f703f7fe700 1 -- 172.21.15.46:0/2868818265 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704801b650 con 0x7f70580afb40 2024-01-26T23:31:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.625+0000 7f705df4b700 1 -- 172.21.15.46:0/2868818265 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70580aff70 con 0x7f70580afb40 2024-01-26T23:31:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.626+0000 7f705df4b700 1 -- 172.21.15.46:0/2868818265 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f705804c710 con 0x7f70580afb40 2024-01-26T23:31:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.627+0000 7f703f7fe700 1 -- 172.21.15.46:0/2868818265 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f704801ba40 con 0x7f70580afb40 2024-01-26T23:31:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.627+0000 7f703f7fe700 1 --2- 172.21.15.46:0/2868818265 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7044066f80 0x7f7044069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.627+0000 7f703f7fe700 1 -- 172.21.15.46:0/2868818265 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f7048085eb0 con 0x7f70580afb40 2024-01-26T23:31:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.627+0000 7f7056ffd700 1 -- 172.21.15.46:0/2868818265 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7044066f80 msgr2=0x7f7044069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:31:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.627+0000 7f7056ffd700 1 --2- 172.21.15.46:0/2868818265 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7044066f80 0x7f7044069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:31:02.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.632+0000 7f703f7fe700 1 -- 172.21.15.46:0/2868818265 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7048052fe0 con 0x7f70580afb40 2024-01-26T23:31:02.790 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:02.789+0000 7f5af1d461c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:31:02.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.828+0000 7f7056ffd700 1 -- 172.21.15.46:0/2868818265 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7044066f80 msgr2=0x7f7044069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:31:02.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.828+0000 7f7056ffd700 1 --2- 172.21.15.46:0/2868818265 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7044066f80 0x7f7044069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:31:02.839 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:02.837+0000 7fe9aa1581c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:31:02.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.838+0000 7f705df4b700 1 -- 172.21.15.46:0/2868818265 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f70580cf9a0 con 0x7f70580afb40 2024-01-26T23:31:02.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.842+0000 7f703f7fe700 1 -- 172.21.15.46:0/2868818265 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) v1 ==== 138+0+0 (secure 0 0 0) 0x7f704802a0d0 con 0x7f70580afb40 2024-01-26T23:31:02.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.844+0000 7f705df4b700 1 -- 172.21.15.46:0/2868818265 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7044066f80 msgr2=0x7f7044069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:31:02.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.844+0000 7f705df4b700 1 --2- 172.21.15.46:0/2868818265 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7044066f80 0x7f7044069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:02.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.844+0000 7f705df4b700 1 -- 172.21.15.46:0/2868818265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70580afb40 msgr2=0x7f70580b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:02.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.844+0000 7f705df4b700 1 --2- 172.21.15.46:0/2868818265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70580afb40 0x7f70580b3550 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f70480097c0 tx=0x7f7048019000 comp rx=0 tx=0).stop 2024-01-26T23:31:02.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.845+0000 7f705df4b700 1 -- 172.21.15.46:0/2868818265 shutdown_connections 2024-01-26T23:31:02.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.845+0000 7f705df4b700 1 --2- 172.21.15.46:0/2868818265 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7044066f80 0x7f7044069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:02.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.845+0000 7f705df4b700 1 --2- 172.21.15.46:0/2868818265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70580afb40 0x7f70580b3550 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:02.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.845+0000 7f705df4b700 1 --2- 172.21.15.46:0/2868818265 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7058159cb0 0x7f70580b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:02.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.845+0000 7f705df4b700 1 --2- 172.21.15.46:0/2868818265 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70580cf050 0x7f70581475c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:02.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.845+0000 7f705df4b700 1 -- 172.21.15.46:0/2868818265 >> 172.21.15.46:0/2868818265 conn(0x7f7058087f60 msgr2=0x7f70581554c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:02.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.845+0000 7f705df4b700 1 -- 172.21.15.46:0/2868818265 shutdown_connections 2024-01-26T23:31:02.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.845+0000 7f705df4b700 1 -- 172.21.15.46:0/2868818265 wait complete. 2024-01-26T23:31:02.861 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-26T23:31:02.866 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:02.865+0000 7f5af1d461c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:31:02.930 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:02.928+0000 7fe9aa1581c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:31:02.936 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:02.935+0000 7f5af1d461c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:31:02.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.994+0000 7f7cc3f5f700 1 Processor -- start 2024-01-26T23:31:02.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.994+0000 7f7cc3f5f700 1 -- start start 2024-01-26T23:31:02.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.995+0000 7f7cc3f5f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7cbc068940 0x7f7cbc068d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:02.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.995+0000 7f7cc3f5f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cbc0c8c50 0x7f7cbc0c7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:02.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.995+0000 7f7cc3f5f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cbc0c87c0 con 0x7f7cbc0c76a0 2024-01-26T23:31:02.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.995+0000 7f7cc3f5f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cbc069250 con 0x7f7cbc068940 2024-01-26T23:31:02.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.995+0000 7f7cc3f5f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0c7bb0 con 0x7f7cbc0c8c50 2024-01-26T23:31:02.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.996+0000 7f7cc1cfb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7cbc068940 0x7f7cbc068d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:02.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.996+0000 7f7cc1cfb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7cbc068940 0x7f7cbc068d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53408/0 (socket says 172.21.15.46:53408) 2024-01-26T23:31:02.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.996+0000 7f7cc14fa700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cbc0c8c50 0x7f7cbc0c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:02.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.996+0000 7f7cc1cfb700 1 -- 172.21.15.46:0/3246689798 learned_addr learned my addr 172.21.15.46:0/3246689798 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:02.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.996+0000 7f7cc1cfb700 1 -- 172.21.15.46:0/3246689798 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cbc0c8c50 msgr2=0x7f7cbc0c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:02.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.996+0000 7f7cc1cfb700 1 --2- 172.21.15.46:0/3246689798 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cbc0c8c50 0x7f7cbc0c7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:02.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.996+0000 7f7cc1cfb700 1 -- 172.21.15.46:0/3246689798 >> v1:172.21.15.106:6789/0 conn(0x7f7cbc0c76a0 legacy=0x7f7cbc0af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:02.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.996+0000 7f7cc1cfb700 1 -- 172.21.15.46:0/3246689798 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cbc0c7e90 con 0x7f7cbc068940 2024-01-26T23:31:02.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.996+0000 7f7cc0cf9700 1 -- 172.21.15.46:0/3246689798 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925143032 0 0) 0x7f7cbc0c87c0 con 0x7f7cbc0c76a0 2024-01-26T23:31:02.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.996+0000 7f7cc1cfb700 1 --2- 172.21.15.46:0/3246689798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cbc068940 0x7f7cbc068d10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009990 tx=0x7f7cb8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a3399669a8045e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:03.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.996+0000 7f7cc0cf9700 1 -- 172.21.15.46:0/3246689798 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8009e80 con 0x7f7cbc068940 2024-01-26T23:31:03.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.996+0000 7f7cc0cf9700 1 -- 172.21.15.46:0/3246689798 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb8005c10 con 0x7f7cbc068940 2024-01-26T23:31:03.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.996+0000 7f7cc0cf9700 1 -- 172.21.15.46:0/3246689798 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb800b650 con 0x7f7cbc068940 2024-01-26T23:31:03.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.997+0000 7f7cc3f5f700 1 -- 172.21.15.46:0/3246689798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cbc068940 msgr2=0x7f7cbc068d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:03.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.997+0000 7f7cc3f5f700 1 --2- 172.21.15.46:0/3246689798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cbc068940 0x7f7cbc068d10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009990 tx=0x7f7cb8009220 comp rx=0 tx=0).stop 2024-01-26T23:31:03.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.997+0000 7f7cc3f5f700 1 -- 172.21.15.46:0/3246689798 shutdown_connections 2024-01-26T23:31:03.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.997+0000 7f7cc3f5f700 1 --2- 172.21.15.46:0/3246689798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cbc068940 0x7f7cbc068d10 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:03.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.997+0000 7f7cc3f5f700 1 --2- 172.21.15.46:0/3246689798 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cbc0c8c50 0x7f7cbc0c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:03.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.997+0000 7f7cc3f5f700 1 -- 172.21.15.46:0/3246689798 >> 172.21.15.46:0/3246689798 conn(0x7f7cbc0ca100 msgr2=0x7f7cbc06e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:03.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.997+0000 7f7cc3f5f700 1 -- 172.21.15.46:0/3246689798 shutdown_connections 2024-01-26T23:31:03.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.997+0000 7f7cc3f5f700 1 -- 172.21.15.46:0/3246689798 wait complete. 2024-01-26T23:31:03.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.999+0000 7f7cc3f5f700 1 Processor -- start 2024-01-26T23:31:03.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:02.999+0000 7f7cc3f5f700 1 -- start start 2024-01-26T23:31:03.002 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:03.000+0000 7f5af1d461c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:31:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.000+0000 7f7cc3f5f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cbc068940 0x7f7cbc1553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.000+0000 7f7cc3f5f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cbc0c76a0 0x7f7cbc0b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.000+0000 7f7cc3f5f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cbc0c8c50 0x7f7cbc0b2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.000+0000 7f7cc3f5f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc1576e0 con 0x7f7cbc0c8c50 2024-01-26T23:31:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.000+0000 7f7cc3f5f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc156db0 con 0x7f7cbc068940 2024-01-26T23:31:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.000+0000 7f7cc3f5f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc157090 con 0x7f7cbc0c76a0 2024-01-26T23:31:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.000+0000 7f7cc24fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cbc0c8c50 0x7f7cbc0b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.000+0000 7f7cc1cfb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cbc068940 0x7f7cbc1553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.000+0000 7f7cc14fa700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cbc0c76a0 0x7f7cbc0b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.000+0000 7f7cc24fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cbc0c8c50 0x7f7cbc0b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53416/0 (socket says 172.21.15.46:53416) 2024-01-26T23:31:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.000+0000 7f7cc14fa700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cbc0c76a0 0x7f7cbc0b1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:44112/0 (socket says 172.21.15.46:44112) 2024-01-26T23:31:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.000+0000 7f7cc1cfb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cbc068940 0x7f7cbc1553f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:58434/0 (socket says 172.21.15.46:58434) 2024-01-26T23:31:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.000+0000 7f7cc24fc700 1 -- 172.21.15.46:0/3330827695 learned_addr learned my addr 172.21.15.46:0/3330827695 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.001+0000 7f7cc24fc700 1 -- 172.21.15.46:0/3330827695 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cbc0c76a0 msgr2=0x7f7cbc0b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.001+0000 7f7cc24fc700 1 --2- 172.21.15.46:0/3330827695 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cbc0c76a0 0x7f7cbc0b1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.001+0000 7f7cc24fc700 1 -- 172.21.15.46:0/3330827695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cbc068940 msgr2=0x7f7cbc1553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.001+0000 7f7cc24fc700 1 --2- 172.21.15.46:0/3330827695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cbc068940 0x7f7cbc1553f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.001+0000 7f7cc24fc700 1 -- 172.21.15.46:0/3330827695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb8009480 con 0x7f7cbc0c8c50 2024-01-26T23:31:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.001+0000 7f7cc14fa700 1 --2- 172.21.15.46:0/3330827695 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cbc0c76a0 0x7f7cbc0b1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:31:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.001+0000 7f7cc24fc700 1 --2- 172.21.15.46:0/3330827695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cbc0c8c50 0x7f7cbc0b2500 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7cb400c8b0 tx=0x7f7cb40016c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.001+0000 7f7cb1ffb700 1 -- 172.21.15.46:0/3330827695 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb4012630 con 0x7f7cbc0c8c50 2024-01-26T23:31:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.001+0000 7f7cb1ffb700 1 -- 172.21.15.46:0/3330827695 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb4012790 con 0x7f7cbc0c8c50 2024-01-26T23:31:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.001+0000 7f7cc3f5f700 1 -- 172.21.15.46:0/3330827695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cbc157370 con 0x7f7cbc0c8c50 2024-01-26T23:31:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.001+0000 7f7cb1ffb700 1 -- 172.21.15.46:0/3330827695 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb4012b50 con 0x7f7cbc0c8c50 2024-01-26T23:31:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.001+0000 7f7cc3f5f700 1 -- 172.21.15.46:0/3330827695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cbc155930 con 0x7f7cbc0c8c50 2024-01-26T23:31:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.003+0000 7f7cc3f5f700 1 -- 172.21.15.46:0/3330827695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cbc04c710 con 0x7f7cbc0c8c50 2024-01-26T23:31:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.003+0000 7f7cb1ffb700 1 -- 172.21.15.46:0/3330827695 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f7cb4003030 con 0x7f7cbc0c8c50 2024-01-26T23:31:03.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.003+0000 7f7cb1ffb700 1 --2- 172.21.15.46:0/3330827695 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7ca0066f80 0x7f7ca0069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:03.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.004+0000 7f7cc1cfb700 1 -- 172.21.15.46:0/3330827695 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7ca0066f80 msgr2=0x7f7ca0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:31:03.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.004+0000 7f7cb1ffb700 1 -- 172.21.15.46:0/3330827695 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f7cb40846a0 con 0x7f7cbc0c8c50 2024-01-26T23:31:03.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.004+0000 7f7cc1cfb700 1 --2- 172.21.15.46:0/3330827695 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7ca0066f80 0x7f7ca0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:31:03.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.008+0000 7f7cb1ffb700 1 -- 172.21.15.46:0/3330827695 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7cb4051730 con 0x7f7cbc0c8c50 2024-01-26T23:31:03.014 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:03.013+0000 7fe9aa1581c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:31:03.090 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:03.089+0000 7fe9aa1581c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:31:03.155 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:03.154+0000 7f5af1d461c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:31:03.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.205+0000 7f7cc1cfb700 1 -- 172.21.15.46:0/3330827695 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7ca0066f80 msgr2=0x7f7ca0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:31:03.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.205+0000 7f7cc1cfb700 1 --2- 172.21.15.46:0/3330827695 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7ca0066f80 0x7f7ca0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:31:03.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.214+0000 7f7cc3f5f700 1 -- 172.21.15.46:0/3330827695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f7cbc0b0980 con 0x7f7cbc0c8c50 2024-01-26T23:31:03.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.217+0000 7f7cb1ffb700 1 -- 172.21.15.46:0/3330827695 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7f7cb4054d50 con 0x7f7cbc0c8c50 2024-01-26T23:31:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.219+0000 7f7cc3f5f700 1 -- 172.21.15.46:0/3330827695 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7ca0066f80 msgr2=0x7f7ca0069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:31:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.219+0000 7f7cc3f5f700 1 --2- 172.21.15.46:0/3330827695 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7ca0066f80 0x7f7ca0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.219+0000 7f7cc3f5f700 1 -- 172.21.15.46:0/3330827695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cbc0c8c50 msgr2=0x7f7cbc0b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.219+0000 7f7cc3f5f700 1 --2- 172.21.15.46:0/3330827695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cbc0c8c50 0x7f7cbc0b2500 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7cb400c8b0 tx=0x7f7cb40016c0 comp rx=0 tx=0).stop 2024-01-26T23:31:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.219+0000 7f7cc3f5f700 1 -- 172.21.15.46:0/3330827695 shutdown_connections 2024-01-26T23:31:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.219+0000 7f7cc3f5f700 1 --2- 172.21.15.46:0/3330827695 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7ca0066f80 0x7f7ca0069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.219+0000 7f7cc3f5f700 1 --2- 172.21.15.46:0/3330827695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cbc0c8c50 0x7f7cbc0b2500 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.219+0000 7f7cc3f5f700 1 --2- 172.21.15.46:0/3330827695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cbc068940 0x7f7cbc1553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.220+0000 7f7cc3f5f700 1 --2- 172.21.15.46:0/3330827695 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cbc0c76a0 0x7f7cbc0b1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.220+0000 7f7cc3f5f700 1 -- 172.21.15.46:0/3330827695 >> 172.21.15.46:0/3330827695 conn(0x7f7cbc0ca100 msgr2=0x7f7cbc0cb580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.220+0000 7f7cc3f5f700 1 -- 172.21.15.46:0/3330827695 shutdown_connections 2024-01-26T23:31:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.220+0000 7f7cc3f5f700 1 -- 172.21.15.46:0/3330827695 wait complete. 2024-01-26T23:31:03.235 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:31:03.236 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-26T23:31:03.262 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:31:03.262 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2024-01-26T23:31:03.269 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:03.267+0000 7fe9aa1581c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:31:03.291 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir0 1m 2024-01-26T23:31:03.292 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:03.291+0000 7f5af1d461c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:31:03.358 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:03.357+0000 7f5af1d461c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:31:03.427 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:03.425+0000 7fe9aa1581c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:31:03.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.466+0000 7fa9e2d39700 1 Processor -- start 2024-01-26T23:31:03.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.466+0000 7fa9e2d39700 1 -- start start 2024-01-26T23:31:03.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.467+0000 7fa9e2d39700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa9dc067af0 0x7fa9dc067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:03.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.467+0000 7fa9e2d39700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9dc06ce90 0x7fa9dc0c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:03.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.467+0000 7fa9e2d39700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9dc06c160 con 0x7fa9dc0c7fe0 2024-01-26T23:31:03.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.467+0000 7fa9e2d39700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9dc06ca00 con 0x7fa9dc067af0 2024-01-26T23:31:03.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.467+0000 7fa9e2d39700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc0c84f0 con 0x7fa9dc06ce90 2024-01-26T23:31:03.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.467+0000 7fa9dbfff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9dc06ce90 0x7fa9dc0c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:03.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.467+0000 7fa9e0ad5700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa9dc067af0 0x7fa9dc067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:03.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.467+0000 7fa9e0ad5700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa9dc067af0 0x7fa9dc067ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:58450/0 (socket says 172.21.15.46:58450) 2024-01-26T23:31:03.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.467+0000 7fa9dbfff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9dc06ce90 0x7fa9dc0c7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:44128/0 (socket says 172.21.15.46:44128) 2024-01-26T23:31:03.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.467+0000 7fa9e0ad5700 1 -- 172.21.15.46:0/1143679678 learned_addr learned my addr 172.21.15.46:0/1143679678 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:03.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.467+0000 7fa9e12d6700 1 --1- 172.21.15.46:0/1143679678 >> v1:172.21.15.106:6789/0 conn(0x7fa9dc0c7fe0 0x7fa9dc157200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:44732/0 (socket says 172.21.15.46:44732) 2024-01-26T23:31:03.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.467+0000 7fa9e0ad5700 1 -- 172.21.15.46:0/1143679678 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9dc06ce90 msgr2=0x7fa9dc0c7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:03.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.467+0000 7fa9e0ad5700 1 --2- 172.21.15.46:0/1143679678 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9dc06ce90 0x7fa9dc0c7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:03.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.467+0000 7fa9e0ad5700 1 -- 172.21.15.46:0/1143679678 >> v1:172.21.15.106:6789/0 conn(0x7fa9dc0c7fe0 legacy=0x7fa9dc157200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.467+0000 7fa9e0ad5700 1 -- 172.21.15.46:0/1143679678 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc06c3b0 con 0x7fa9dc067af0 2024-01-26T23:31:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.467+0000 7fa9db7fe700 1 -- 172.21.15.46:0/1143679678 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936232561 0 0) 0x7fa9dc06c160 con 0x7fa9dc0c7fe0 2024-01-26T23:31:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.467+0000 7fa9dbfff700 1 --2- 172.21.15.46:0/1143679678 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9dc06ce90 0x7fa9dc0c7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:31:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.468+0000 7fa9e0ad5700 1 --2- 172.21.15.46:0/1143679678 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9dc067af0 0x7fa9dc067ec0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa9d00099b0 tx=0x7fa9d0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e93fc97c108e733c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.468+0000 7fa9db7fe700 1 -- 172.21.15.46:0/1143679678 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0009ea0 con 0x7fa9dc067af0 2024-01-26T23:31:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.468+0000 7fa9db7fe700 1 -- 172.21.15.46:0/1143679678 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d000b860 con 0x7fa9dc067af0 2024-01-26T23:31:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.468+0000 7fa9db7fe700 1 -- 172.21.15.46:0/1143679678 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d000bc00 con 0x7fa9dc067af0 2024-01-26T23:31:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.468+0000 7fa9e2d39700 1 -- 172.21.15.46:0/1143679678 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9dc067af0 msgr2=0x7fa9dc067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.468+0000 7fa9e2d39700 1 --2- 172.21.15.46:0/1143679678 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9dc067af0 0x7fa9dc067ec0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa9d00099b0 tx=0x7fa9d0009250 comp rx=0 tx=0).stop 2024-01-26T23:31:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.468+0000 7fa9e2d39700 1 -- 172.21.15.46:0/1143679678 shutdown_connections 2024-01-26T23:31:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.468+0000 7fa9e2d39700 1 --2- 172.21.15.46:0/1143679678 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9dc067af0 0x7fa9dc067ec0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.468+0000 7fa9e2d39700 1 --2- 172.21.15.46:0/1143679678 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9dc06ce90 0x7fa9dc0c7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.468+0000 7fa9e2d39700 1 -- 172.21.15.46:0/1143679678 >> 172.21.15.46:0/1143679678 conn(0x7fa9dc0ca9a0 msgr2=0x7fa9dc068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.469+0000 7fa9e2d39700 1 -- 172.21.15.46:0/1143679678 shutdown_connections 2024-01-26T23:31:03.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.469+0000 7fa9e2d39700 1 -- 172.21.15.46:0/1143679678 wait complete. 2024-01-26T23:31:03.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.470+0000 7fa9e2d39700 1 Processor -- start 2024-01-26T23:31:03.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.470+0000 7fa9e2d39700 1 -- start start 2024-01-26T23:31:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.471+0000 7fa9e2d39700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9dc067af0 0x7fa9dc154080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.471+0000 7fa9e2d39700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9dc06ce90 0x7fa9dc158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.471+0000 7fa9e2d39700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9dc0c7fe0 0x7fa9dc159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.471+0000 7fa9e2d39700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc0cb030 con 0x7fa9dc067af0 2024-01-26T23:31:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.471+0000 7fa9e2d39700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc0cbe90 con 0x7fa9dc06ce90 2024-01-26T23:31:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.471+0000 7fa9e2d39700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc1545c0 con 0x7fa9dc0c7fe0 2024-01-26T23:31:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.471+0000 7fa9e0ad5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9dc067af0 0x7fa9dc154080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.471+0000 7fa9e0ad5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9dc067af0 0x7fa9dc154080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53430/0 (socket says 172.21.15.46:53430) 2024-01-26T23:31:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.471+0000 7fa9e0ad5700 1 -- 172.21.15.46:0/2301868855 learned_addr learned my addr 172.21.15.46:0/2301868855 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.471+0000 7fa9dbfff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9dc06ce90 0x7fa9dc158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.471+0000 7fa9e12d6700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9dc0c7fe0 0x7fa9dc159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.471+0000 7fa9dbfff700 1 -- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9dc0c7fe0 msgr2=0x7fa9dc159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.471+0000 7fa9dbfff700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9dc0c7fe0 0x7fa9dc159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:03.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.471+0000 7fa9dbfff700 1 -- 172.21.15.46:0/2301868855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9dc067af0 msgr2=0x7fa9dc154080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.471+0000 7fa9dbfff700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9dc067af0 0x7fa9dc154080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.471+0000 7fa9dbfff700 1 -- 172.21.15.46:0/2301868855 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c8009420 con 0x7fa9dc06ce90 2024-01-26T23:31:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.472+0000 7fa9dbfff700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9dc06ce90 0x7fa9dc158ce0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800ecb0 tx=0x7fa9c80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.472+0000 7fa9d8ff9700 1 -- 172.21.15.46:0/2301868855 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c800c850 con 0x7fa9dc06ce90 2024-01-26T23:31:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.472+0000 7fa9d8ff9700 1 -- 172.21.15.46:0/2301868855 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c800c9b0 con 0x7fa9dc06ce90 2024-01-26T23:31:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.472+0000 7fa9e2d39700 1 -- 172.21.15.46:0/2301868855 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d00094f0 con 0x7fa9dc06ce90 2024-01-26T23:31:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.472+0000 7fa9d8ff9700 1 -- 172.21.15.46:0/2301868855 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c800cd50 con 0x7fa9dc06ce90 2024-01-26T23:31:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.472+0000 7fa9e2d39700 1 -- 172.21.15.46:0/2301868855 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9dc0cb560 con 0x7fa9dc06ce90 2024-01-26T23:31:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.473+0000 7fa9e2d39700 1 -- 172.21.15.46:0/2301868855 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9dc0cc040 con 0x7fa9dc06ce90 2024-01-26T23:31:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.473+0000 7fa9d8ff9700 1 -- 172.21.15.46:0/2301868855 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 6) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fa9c8013680 con 0x7fa9dc06ce90 2024-01-26T23:31:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.474+0000 7fa9d8ff9700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 0x7fa9c0069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.474+0000 7fa9d8ff9700 1 -- 172.21.15.46:0/2301868855 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fa9c8084950 con 0x7fa9dc06ce90 2024-01-26T23:31:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.474+0000 7fa9e0ad5700 1 -- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 msgr2=0x7fa9c0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:31:03.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.474+0000 7fa9e0ad5700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 0x7fa9c0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:31:03.479 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:03.477+0000 7f5af1d461c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:31:03.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.478+0000 7fa9d8ff9700 1 -- 172.21.15.46:0/2301868855 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa9dc0cc040 con 0x7fa9dc06ce90 2024-01-26T23:31:03.506 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:03.504+0000 7fe9aa1581c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:31:03.551 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:03.550+0000 7f5af1d461c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:31:03.645 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:03.644+0000 7fe9aa1581c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:31:03.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.675+0000 7fa9e0ad5700 1 -- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 msgr2=0x7fa9c0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:31:03.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.675+0000 7fa9e0ad5700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 0x7fa9c0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:31:03.692 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:03.690+0000 7f5af1d461c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:31:03.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:03.715+0000 7fa9e2d39700 1 -- 172.21.15.46:0/2301868855 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9dc04c710 con 0x7fa9c0066f80 2024-01-26T23:31:03.728 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:03.727+0000 7fe9aa1581c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:31:03.816 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:03.815+0000 7f5af1d461c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:31:03.904 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:03.903+0000 7fe9aa1581c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:31:04.045 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:04.043+0000 7fe9aa1581c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:31:04.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:04.076+0000 7fa9e0ad5700 1 -- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 msgr2=0x7fa9c0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:31:04.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:04.076+0000 7fa9e0ad5700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 0x7fa9c0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-26T23:31:04.127 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:04.126+0000 7f5af1d461c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:31:04.335 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:04.334+0000 7f5af1d461c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:31:04.405 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:04.404+0000 7fe9aa1581c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:31:04.557 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:04.556+0000 7f5af1d461c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:31:04.627 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:04.626+0000 7f5af1d461c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:31:04.643 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:04.641+0000 7fe9aa1581c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:31:04.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:04.877+0000 7fa9e0ad5700 1 -- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 msgr2=0x7fa9c0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:31:04.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:04.877+0000 7fa9e0ad5700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 0x7fa9c0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-26T23:31:04.901 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:04.899+0000 7fe9aa1581c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:31:04.980 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:04.978+0000 7fe9aa1581c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:31:05.221 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:05.220+0000 7f5af1d461c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:31:05.295 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:05.294+0000 7f5af1d461c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:31:05.416 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:05.415+0000 7f5af1d461c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:31:05.671 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:05.669+0000 7fe9aa1581c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:31:05.755 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:05.753+0000 7fe9aa1581c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:31:05.894 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:05.893+0000 7fe9aa1581c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:31:05.913 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:05.912+0000 7f5af1d461c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:31:06.058 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:06.057+0000 7f5af1d461c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:31:06.205 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:06.204+0000 7f5af1d461c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:31:06.474 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:06.472+0000 7fe9aa1581c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:31:06.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:06.478+0000 7fa9e0ad5700 1 -- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 msgr2=0x7fa9c0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:31:06.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:06.478+0000 7fa9e0ad5700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 0x7fa9c0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-26T23:31:06.640 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:06.639+0000 7fe9aa1581c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:31:06.809 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:06.807+0000 7fe9aa1581c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:31:06.898 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:06.897+0000 7f5af1d461c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:31:07.111 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:07.109+0000 7f5af1d461c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:31:07.182 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:07.180+0000 7f5af1d461c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:31:07.333 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:07.331+0000 7f5af1d461c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:31:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:07.354+0000 7fa9d8ff9700 1 -- 172.21.15.46:0/2301868855 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mgrmap(e 7) v1 ==== 84470+0+0 (secure 0 0 0) 0x7fa9c8051e50 con 0x7fa9dc06ce90 2024-01-26T23:31:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:07.354+0000 7fa9d8ff9700 1 -- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 msgr2=0x7fa9c0069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:31:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:07.354+0000 7fa9d8ff9700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 0x7fa9c0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:07.452 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:07.451+0000 7f5a67622700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:31:07.452 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:07.451+0000 7f5a67622700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:31:07.452 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:07.451+0000 7f5a67622700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:31:07.452 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:07.451+0000 7f5a67622700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:31:07.453 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:07.451+0000 7f5a67622700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:31:07.453 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:07.452+0000 7f5a6b62a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:31:07.453 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:07.452+0000 7f5a6b62a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:31:07.453 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:07.452+0000 7f5a6b62a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:31:07.453 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:07.452+0000 7f5a6b62a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:31:07.453 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:31:07.452+0000 7f5a6b62a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:31:07.620 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:07.618+0000 7fe9aa1581c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:31:07.861 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:07.860+0000 7fe9aa1581c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:31:07.942 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:07.940+0000 7fe9aa1581c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:31:08.116 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:31:08.114+0000 7fe9aa1581c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:31:08.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.356+0000 7fa9d8ff9700 1 -- 172.21.15.46:0/2301868855 <== mon.1 v2:172.21.15.106:3300/0 8 ==== mgrmap(e 8) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fa9c80534f0 con 0x7fa9dc06ce90 2024-01-26T23:31:08.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.356+0000 7fa9d8ff9700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 0x7fa9c00693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:08.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.356+0000 7fa9d8ff9700 1 -- 172.21.15.46:0/2301868855 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9dc04c710 con 0x7fa9c0066f80 2024-01-26T23:31:08.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.359+0000 7fa9e0ad5700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 0x7fa9c00693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:08.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.361+0000 7fa9e0ad5700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 0x7fa9c00693b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0009470 tx=0x7fa9d0017000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:08.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.439+0000 7fa9d8ff9700 1 -- 172.21.15.46:0/2301868855 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fa9dc04c710 con 0x7fa9c0066f80 2024-01-26T23:31:08.441 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path /snap_test_dir1/dir0 2024-01-26T23:31:08.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.442+0000 7fa9e2d39700 1 -- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 msgr2=0x7fa9c00693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:08.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.442+0000 7fa9e2d39700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 0x7fa9c00693b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0009470 tx=0x7fa9d0017000 comp rx=0 tx=0).stop 2024-01-26T23:31:08.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.442+0000 7fa9e2d39700 1 -- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9dc06ce90 msgr2=0x7fa9dc158ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:08.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.442+0000 7fa9e2d39700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9dc06ce90 0x7fa9dc158ce0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800ecb0 tx=0x7fa9c80091a0 comp rx=0 tx=0).stop 2024-01-26T23:31:08.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.442+0000 7fa9e2d39700 1 -- 172.21.15.46:0/2301868855 shutdown_connections 2024-01-26T23:31:08.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.442+0000 7fa9e2d39700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9c0066f80 0x7fa9c00693b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:08.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.442+0000 7fa9e2d39700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9dc067af0 0x7fa9dc154080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:08.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.442+0000 7fa9e2d39700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9dc0c7fe0 0x7fa9dc159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:08.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.442+0000 7fa9e2d39700 1 --2- 172.21.15.46:0/2301868855 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9dc06ce90 0x7fa9dc158ce0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:08.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.442+0000 7fa9e2d39700 1 -- 172.21.15.46:0/2301868855 >> 172.21.15.46:0/2301868855 conn(0x7fa9dc0ca9a0 msgr2=0x7fa9dc0b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:08.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.443+0000 7fa9e2d39700 1 -- 172.21.15.46:0/2301868855 shutdown_connections 2024-01-26T23:31:08.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.443+0000 7fa9e2d39700 1 -- 172.21.15.46:0/2301868855 wait complete. 2024-01-26T23:31:08.460 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2024-01-26T23:31:08.460 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:31:08.461 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2024-01-26T23:31:08.483 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir1 1m 2024-01-26T23:31:08.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.665+0000 7f1183a85700 1 Processor -- start 2024-01-26T23:31:08.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.665+0000 7f1183a85700 1 -- start start 2024-01-26T23:31:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.666+0000 7f1183a85700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f117c159510 0x7f117c1598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.666+0000 7f1183a85700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f117c0ca220 0x7f117c0c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.666+0000 7f1183a85700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117c069260 con 0x7f117c0ca690 2024-01-26T23:31:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.666+0000 7f1183a85700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f117c15d670 con 0x7f117c159510 2024-01-26T23:31:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.666+0000 7f1183a85700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0c9750 con 0x7f117c0ca220 2024-01-26T23:31:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.666+0000 7f1182022700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f117c0ca690 0x7f117c0cc380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42042/0 (socket says 172.21.15.46:42042) 2024-01-26T23:31:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.666+0000 7f1182022700 1 -- 172.21.15.46:0/1304560505 learned_addr learned my addr 172.21.15.46:0/1304560505 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:31:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.666+0000 7f1181020700 1 --2- 172.21.15.46:0/1304560505 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f117c0ca220 0x7f117c0c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.666+0000 7f1181821700 1 --2- 172.21.15.46:0/1304560505 >> v2:172.21.15.106:3300/0 conn(0x7f117c159510 0x7f117c1598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.667+0000 7f118081f700 1 -- 172.21.15.46:0/1304560505 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689251633 0 0) 0x7f117c069260 con 0x7f117c0ca690 2024-01-26T23:31:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.667+0000 7f118081f700 1 -- 172.21.15.46:0/1304560505 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1168003570 con 0x7f117c0ca690 2024-01-26T23:31:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.667+0000 7f1181020700 1 -- 172.21.15.46:0/1304560505 >> v2:172.21.15.106:3300/0 conn(0x7f117c159510 msgr2=0x7f117c1598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.667+0000 7f1181020700 1 --2- 172.21.15.46:0/1304560505 >> v2:172.21.15.106:3300/0 conn(0x7f117c159510 0x7f117c1598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.667+0000 7f1181020700 1 -- 172.21.15.46:0/1304560505 >> v1:172.21.15.46:6789/0 conn(0x7f117c0ca690 legacy=0x7f117c0cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.667+0000 7f1181020700 1 -- 172.21.15.46:0/1304560505 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c0c9a30 con 0x7f117c0ca220 2024-01-26T23:31:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.667+0000 7f1181821700 1 --2- 172.21.15.46:0/1304560505 >> v2:172.21.15.106:3300/0 conn(0x7f117c159510 0x7f117c1598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:31:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.667+0000 7f1181020700 1 --2- 172.21.15.46:0/1304560505 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f117c0ca220 0x7f117c0c8be0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f116c009910 tx=0x7f116c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=826fd88a7025873a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.667+0000 7f118081f700 1 -- 172.21.15.46:0/1304560505 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c011630 con 0x7f117c0ca220 2024-01-26T23:31:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.667+0000 7f118081f700 1 -- 172.21.15.46:0/1304560505 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116c011790 con 0x7f117c0ca220 2024-01-26T23:31:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.667+0000 7f118081f700 1 -- 172.21.15.46:0/1304560505 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c011b30 con 0x7f117c0ca220 2024-01-26T23:31:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.668+0000 7f1183a85700 1 -- 172.21.15.46:0/1304560505 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f117c0ca220 msgr2=0x7f117c0c8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.668+0000 7f1183a85700 1 --2- 172.21.15.46:0/1304560505 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f117c0ca220 0x7f117c0c8be0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f116c009910 tx=0x7f116c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:31:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.668+0000 7f1183a85700 1 -- 172.21.15.46:0/1304560505 shutdown_connections 2024-01-26T23:31:08.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.668+0000 7f1183a85700 1 --2- 172.21.15.46:0/1304560505 >> v2:172.21.15.106:3300/0 conn(0x7f117c159510 0x7f117c1598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:08.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.668+0000 7f1183a85700 1 --2- 172.21.15.46:0/1304560505 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f117c0ca220 0x7f117c0c8be0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:08.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.668+0000 7f1183a85700 1 -- 172.21.15.46:0/1304560505 >> 172.21.15.46:0/1304560505 conn(0x7f117c06bd90 msgr2=0x7f117c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:08.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.668+0000 7f1183a85700 1 -- 172.21.15.46:0/1304560505 shutdown_connections 2024-01-26T23:31:08.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.668+0000 7f1183a85700 1 -- 172.21.15.46:0/1304560505 wait complete. 2024-01-26T23:31:08.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.669+0000 7f1183a85700 1 Processor -- start 2024-01-26T23:31:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.669+0000 7f1183a85700 1 -- start start 2024-01-26T23:31:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.670+0000 7f1183a85700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f117c0ca220 0x7f117c0b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.670+0000 7f1183a85700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f117c159510 0x7f117c0c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.670+0000 7f1183a85700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f117c0b1340 0x7f117c0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.670+0000 7f1183a85700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c14d710 con 0x7f117c159510 2024-01-26T23:31:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.670+0000 7f1183a85700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0b2df0 con 0x7f117c0b1340 2024-01-26T23:31:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.670+0000 7f1183a85700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0c75a0 con 0x7f117c0ca220 2024-01-26T23:31:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.670+0000 7f1181020700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f117c159510 0x7f117c0c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.670+0000 7f1181020700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f117c159510 0x7f117c0c61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53432/0 (socket says 172.21.15.46:53432) 2024-01-26T23:31:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.670+0000 7f1181020700 1 -- 172.21.15.46:0/2035460026 learned_addr learned my addr 172.21.15.46:0/2035460026 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.670+0000 7f1182022700 1 --2- 172.21.15.46:0/2035460026 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f117c0b1340 0x7f117c0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.670+0000 7f1181821700 1 --2- 172.21.15.46:0/2035460026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f117c0ca220 0x7f117c0b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.670+0000 7f1181020700 1 -- 172.21.15.46:0/2035460026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f117c0ca220 msgr2=0x7f117c0b28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.670+0000 7f1181020700 1 --2- 172.21.15.46:0/2035460026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f117c0ca220 0x7f117c0b28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.670+0000 7f1181020700 1 -- 172.21.15.46:0/2035460026 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f117c0b1340 msgr2=0x7f117c0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.670+0000 7f1181020700 1 --2- 172.21.15.46:0/2035460026 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f117c0b1340 0x7f117c0b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.670+0000 7f1181020700 1 -- 172.21.15.46:0/2035460026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1178009420 con 0x7f117c159510 2024-01-26T23:31:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.671+0000 7f1181020700 1 --2- 172.21.15.46:0/2035460026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f117c159510 0x7f117c0c61d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f116c011fa0 tx=0x7f116c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.671+0000 7f1171ffb700 1 -- 172.21.15.46:0/2035460026 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c000d10 con 0x7f117c159510 2024-01-26T23:31:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.671+0000 7f1171ffb700 1 -- 172.21.15.46:0/2035460026 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116c011b10 con 0x7f117c159510 2024-01-26T23:31:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.671+0000 7f1183a85700 1 -- 172.21.15.46:0/2035460026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f116c009420 con 0x7f117c159510 2024-01-26T23:31:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.671+0000 7f1171ffb700 1 -- 172.21.15.46:0/2035460026 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c025620 con 0x7f117c159510 2024-01-26T23:31:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.671+0000 7f1183a85700 1 -- 172.21.15.46:0/2035460026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117c0c6990 con 0x7f117c159510 2024-01-26T23:31:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.672+0000 7f1183a85700 1 -- 172.21.15.46:0/2035460026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117c0c7750 con 0x7f117c159510 2024-01-26T23:31:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.673+0000 7f1171ffb700 1 -- 172.21.15.46:0/2035460026 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f116c025a10 con 0x7f117c159510 2024-01-26T23:31:08.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.673+0000 7f1171ffb700 1 --2- 172.21.15.46:0/2035460026 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1160066dd0 0x7f1160069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:08.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.673+0000 7f1171ffb700 1 -- 172.21.15.46:0/2035460026 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f116c085c40 con 0x7f117c159510 2024-01-26T23:31:08.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.673+0000 7f1181821700 1 --2- 172.21.15.46:0/2035460026 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1160066dd0 0x7f1160069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:08.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.674+0000 7f1181821700 1 --2- 172.21.15.46:0/2035460026 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1160066dd0 0x7f1160069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1178009ea0 tx=0x7f11780091a0 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:08.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.678+0000 7f1171ffb700 1 -- 172.21.15.46:0/2035460026 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f117c0c7750 con 0x7f117c159510 2024-01-26T23:31:08.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.908+0000 7f1183a85700 1 -- 172.21.15.46:0/2035460026 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f117c0c7710 con 0x7f1160066dd0 2024-01-26T23:31:08.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.966+0000 7f1171ffb700 1 -- 172.21.15.46:0/2035460026 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f117c0c7710 con 0x7f1160066dd0 2024-01-26T23:31:08.968 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path /snap_test_dir1/dir1 2024-01-26T23:31:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.969+0000 7f1183a85700 1 -- 172.21.15.46:0/2035460026 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1160066dd0 msgr2=0x7f1160069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.969+0000 7f1183a85700 1 --2- 172.21.15.46:0/2035460026 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1160066dd0 0x7f1160069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1178009ea0 tx=0x7f11780091a0 comp rx=0 tx=0).stop 2024-01-26T23:31:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.969+0000 7f1183a85700 1 -- 172.21.15.46:0/2035460026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f117c159510 msgr2=0x7f117c0c61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.969+0000 7f1183a85700 1 --2- 172.21.15.46:0/2035460026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f117c159510 0x7f117c0c61d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f116c011fa0 tx=0x7f116c017000 comp rx=0 tx=0).stop 2024-01-26T23:31:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.969+0000 7f1183a85700 1 -- 172.21.15.46:0/2035460026 shutdown_connections 2024-01-26T23:31:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.969+0000 7f1183a85700 1 --2- 172.21.15.46:0/2035460026 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1160066dd0 0x7f1160069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.969+0000 7f1183a85700 1 --2- 172.21.15.46:0/2035460026 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f117c0b1340 0x7f117c0b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.969+0000 7f1183a85700 1 --2- 172.21.15.46:0/2035460026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f117c0ca220 0x7f117c0b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.969+0000 7f1183a85700 1 --2- 172.21.15.46:0/2035460026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f117c159510 0x7f117c0c61d0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:08.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.969+0000 7f1183a85700 1 -- 172.21.15.46:0/2035460026 >> 172.21.15.46:0/2035460026 conn(0x7f117c06bd90 msgr2=0x7f117c06df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:08.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.969+0000 7f1183a85700 1 -- 172.21.15.46:0/2035460026 shutdown_connections 2024-01-26T23:31:08.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:08.969+0000 7f1183a85700 1 -- 172.21.15.46:0/2035460026 wait complete. 2024-01-26T23:31:08.985 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2024-01-26T23:31:08.985 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:31:08.985 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2024-01-26T23:31:09.007 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir2 1m 2024-01-26T23:31:09.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.188+0000 7f5ab1f37700 1 Processor -- start 2024-01-26T23:31:09.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.188+0000 7f5ab1f37700 1 -- start start 2024-01-26T23:31:09.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.189+0000 7f5ab1f37700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5aac068940 0x7f5aac068d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:09.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.189+0000 7f5ab1f37700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5aac0c8c50 0x7f5aac0c7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:09.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.189+0000 7f5ab1f37700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aac0c87c0 con 0x7f5aac0c76a0 2024-01-26T23:31:09.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.189+0000 7f5ab1f37700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aac069250 con 0x7f5aac068940 2024-01-26T23:31:09.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.189+0000 7f5ab1f37700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac0c7bb0 con 0x7f5aac0c8c50 2024-01-26T23:31:09.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.189+0000 7f5aab7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5aac068940 0x7f5aac068d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:09.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.189+0000 7f5aabfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5aac0c76a0 0x7f5aac0af990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42044/0 (socket says 172.21.15.46:42044) 2024-01-26T23:31:09.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.189+0000 7f5aabfff700 1 -- 172.21.15.46:0/2862562898 learned_addr learned my addr 172.21.15.46:0/2862562898 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:31:09.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.190+0000 7f5aaaffd700 1 --2- 172.21.15.46:0/2862562898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5aac0c8c50 0x7f5aac0c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:09.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.190+0000 7f5aab7fe700 1 -- 172.21.15.46:0/2862562898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5aac0c8c50 msgr2=0x7f5aac0c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:09.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.190+0000 7f5aab7fe700 1 --2- 172.21.15.46:0/2862562898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5aac0c8c50 0x7f5aac0c7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:09.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.190+0000 7f5aab7fe700 1 -- 172.21.15.46:0/2862562898 >> v1:172.21.15.46:6789/0 conn(0x7f5aac0c76a0 legacy=0x7f5aac0af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:09.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.190+0000 7f5aaa7fc700 1 -- 172.21.15.46:0/2862562898 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657701588 0 0) 0x7f5aac0c87c0 con 0x7f5aac0c76a0 2024-01-26T23:31:09.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.190+0000 7f5aab7fe700 1 -- 172.21.15.46:0/2862562898 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aac0c7e90 con 0x7f5aac068940 2024-01-26T23:31:09.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.190+0000 7f5aab7fe700 1 --2- 172.21.15.46:0/2862562898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aac068940 0x7f5aac068d10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5a940098d0 tx=0x7f5a94009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a1585a70db8a27 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:09.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.190+0000 7f5aaa7fc700 1 -- 172.21.15.46:0/2862562898 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a94005c10 con 0x7f5aac068940 2024-01-26T23:31:09.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.190+0000 7f5aaa7fc700 1 -- 172.21.15.46:0/2862562898 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a94005d70 con 0x7f5aac068940 2024-01-26T23:31:09.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.190+0000 7f5aaa7fc700 1 -- 172.21.15.46:0/2862562898 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a94006280 con 0x7f5aac068940 2024-01-26T23:31:09.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.191+0000 7f5ab1f37700 1 -- 172.21.15.46:0/2862562898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aac068940 msgr2=0x7f5aac068d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:09.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.191+0000 7f5ab1f37700 1 --2- 172.21.15.46:0/2862562898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aac068940 0x7f5aac068d10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5a940098d0 tx=0x7f5a94009120 comp rx=0 tx=0).stop 2024-01-26T23:31:09.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.191+0000 7f5ab1f37700 1 -- 172.21.15.46:0/2862562898 shutdown_connections 2024-01-26T23:31:09.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.191+0000 7f5ab1f37700 1 --2- 172.21.15.46:0/2862562898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aac068940 0x7f5aac068d10 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:09.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.191+0000 7f5ab1f37700 1 --2- 172.21.15.46:0/2862562898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5aac0c8c50 0x7f5aac0c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:09.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.191+0000 7f5ab1f37700 1 -- 172.21.15.46:0/2862562898 >> 172.21.15.46:0/2862562898 conn(0x7f5aac0ca100 msgr2=0x7f5aac06e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:09.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.191+0000 7f5ab1f37700 1 -- 172.21.15.46:0/2862562898 shutdown_connections 2024-01-26T23:31:09.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.191+0000 7f5ab1f37700 1 -- 172.21.15.46:0/2862562898 wait complete. 2024-01-26T23:31:09.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.193+0000 7f5ab1f37700 1 Processor -- start 2024-01-26T23:31:09.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.193+0000 7f5ab1f37700 1 -- start start 2024-01-26T23:31:09.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5ab1f37700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5aac068940 0x7f5aac14ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5ab1f37700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aac0c76a0 0x7f5aac0c0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5ab1f37700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5aac0c8c50 0x7f5aac169e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5ab1f37700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac0c19a0 con 0x7f5aac0c76a0 2024-01-26T23:31:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5ab1f37700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac0c1c80 con 0x7f5aac068940 2024-01-26T23:31:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5ab1f37700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac0c1f60 con 0x7f5aac0c8c50 2024-01-26T23:31:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5aaaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aac0c76a0 0x7f5aac0c0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5aaaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aac0c76a0 0x7f5aac0c0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53450/0 (socket says 172.21.15.46:53450) 2024-01-26T23:31:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5aaaffd700 1 -- 172.21.15.46:0/2350473650 learned_addr learned my addr 172.21.15.46:0/2350473650 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5aabfff700 1 --2- 172.21.15.46:0/2350473650 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5aac0c8c50 0x7f5aac169e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5aab7fe700 1 --2- 172.21.15.46:0/2350473650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5aac068940 0x7f5aac14ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5aaaffd700 1 -- 172.21.15.46:0/2350473650 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5aac0c8c50 msgr2=0x7f5aac169e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5aaaffd700 1 --2- 172.21.15.46:0/2350473650 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5aac0c8c50 0x7f5aac169e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5aaaffd700 1 -- 172.21.15.46:0/2350473650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5aac068940 msgr2=0x7f5aac14ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5aaaffd700 1 --2- 172.21.15.46:0/2350473650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5aac068940 0x7f5aac14ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:09.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5aaaffd700 1 -- 172.21.15.46:0/2350473650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a940093c0 con 0x7f5aac0c76a0 2024-01-26T23:31:09.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5aabfff700 1 --2- 172.21.15.46:0/2350473650 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5aac0c8c50 0x7f5aac169e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:31:09.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.194+0000 7f5aaaffd700 1 --2- 172.21.15.46:0/2350473650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aac0c76a0 0x7f5aac0c0fe0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00e930 tx=0x7f5a9c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:09.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.195+0000 7f5aa37fe700 1 -- 172.21.15.46:0/2350473650 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c00eed0 con 0x7f5aac0c76a0 2024-01-26T23:31:09.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.195+0000 7f5aa37fe700 1 -- 172.21.15.46:0/2350473650 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9c00c850 con 0x7f5aac0c76a0 2024-01-26T23:31:09.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.195+0000 7f5ab1f37700 1 -- 172.21.15.46:0/2350473650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aac0c2240 con 0x7f5aac0c76a0 2024-01-26T23:31:09.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.195+0000 7f5aa37fe700 1 -- 172.21.15.46:0/2350473650 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c00cbf0 con 0x7f5aac0c76a0 2024-01-26T23:31:09.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.195+0000 7f5ab1f37700 1 -- 172.21.15.46:0/2350473650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aac14f4a0 con 0x7f5aac0c76a0 2024-01-26T23:31:09.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.196+0000 7f5ab1f37700 1 -- 172.21.15.46:0/2350473650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aac04c710 con 0x7f5aac0c76a0 2024-01-26T23:31:09.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.196+0000 7f5aa37fe700 1 -- 172.21.15.46:0/2350473650 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f5a9c013470 con 0x7f5aac0c76a0 2024-01-26T23:31:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.196+0000 7f5aa37fe700 1 --2- 172.21.15.46:0/2350473650 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5a8c066dd0 0x7f5a8c069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.197+0000 7f5aa37fe700 1 -- 172.21.15.46:0/2350473650 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f5a9c084670 con 0x7f5aac0c76a0 2024-01-26T23:31:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.197+0000 7f5aab7fe700 1 --2- 172.21.15.46:0/2350473650 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5a8c066dd0 0x7f5a8c069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.197+0000 7f5aab7fe700 1 --2- 172.21.15.46:0/2350473650 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5a8c066dd0 0x7f5a8c069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5a94009340 tx=0x7f5a94016000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:09.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.201+0000 7f5aa37fe700 1 -- 172.21.15.46:0/2350473650 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a9c051950 con 0x7f5aac0c76a0 2024-01-26T23:31:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.436+0000 7f5ab1f37700 1 -- 172.21.15.46:0/2350473650 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir2", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5aac0c1df0 con 0x7f5a8c066dd0 2024-01-26T23:31:09.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.489+0000 7f5aa37fe700 1 -- 172.21.15.46:0/2350473650 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f5aac0c1df0 con 0x7f5a8c066dd0 2024-01-26T23:31:09.491 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path /snap_test_dir1/dir2 2024-01-26T23:31:09.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.492+0000 7f5ab1f37700 1 -- 172.21.15.46:0/2350473650 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5a8c066dd0 msgr2=0x7f5a8c069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:09.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.492+0000 7f5ab1f37700 1 --2- 172.21.15.46:0/2350473650 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5a8c066dd0 0x7f5a8c069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5a94009340 tx=0x7f5a94016000 comp rx=0 tx=0).stop 2024-01-26T23:31:09.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.492+0000 7f5ab1f37700 1 -- 172.21.15.46:0/2350473650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aac0c76a0 msgr2=0x7f5aac0c0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:09.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.492+0000 7f5ab1f37700 1 --2- 172.21.15.46:0/2350473650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aac0c76a0 0x7f5aac0c0fe0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00e930 tx=0x7f5a9c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:31:09.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.492+0000 7f5ab1f37700 1 -- 172.21.15.46:0/2350473650 shutdown_connections 2024-01-26T23:31:09.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.492+0000 7f5ab1f37700 1 --2- 172.21.15.46:0/2350473650 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5a8c066dd0 0x7f5a8c069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:09.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.492+0000 7f5ab1f37700 1 --2- 172.21.15.46:0/2350473650 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5aac0c8c50 0x7f5aac169e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:09.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.492+0000 7f5ab1f37700 1 --2- 172.21.15.46:0/2350473650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5aac068940 0x7f5aac14ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:09.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.492+0000 7f5ab1f37700 1 --2- 172.21.15.46:0/2350473650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aac0c76a0 0x7f5aac0c0fe0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:09.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.492+0000 7f5ab1f37700 1 -- 172.21.15.46:0/2350473650 >> 172.21.15.46:0/2350473650 conn(0x7f5aac0ca100 msgr2=0x7f5aac0c63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:09.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.492+0000 7f5ab1f37700 1 -- 172.21.15.46:0/2350473650 shutdown_connections 2024-01-26T23:31:09.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.492+0000 7f5ab1f37700 1 -- 172.21.15.46:0/2350473650 wait complete. 2024-01-26T23:31:09.508 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2024-01-26T23:31:09.508 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:31:09.508 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2024-01-26T23:31:09.531 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir3 1m 2024-01-26T23:31:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.711+0000 7f71cd3d9700 1 Processor -- start 2024-01-26T23:31:09.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.711+0000 7f71cd3d9700 1 -- start start 2024-01-26T23:31:09.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.712+0000 7f71cd3d9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71c81565e0 0x7f71c80a98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:09.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.712+0000 7f71cd3d9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71c81569b0 0x7f71c80ac4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:09.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.712+0000 7f71cd3d9700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c806ba40 con 0x7f71c8069b20 2024-01-26T23:31:09.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.712+0000 7f71cd3d9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c806e470 con 0x7f71c81569b0 2024-01-26T23:31:09.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.712+0000 7f71cd3d9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c806db60 con 0x7f71c81565e0 2024-01-26T23:31:09.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.712+0000 7f71c67fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71c81569b0 0x7f71c80ac4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:09.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.712+0000 7f71c67fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71c81569b0 0x7f71c80ac4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53464/0 (socket says 172.21.15.46:53464) 2024-01-26T23:31:09.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.712+0000 7f71c67fc700 1 -- 172.21.15.46:0/2280128219 learned_addr learned my addr 172.21.15.46:0/2280128219 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:09.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.712+0000 7f71c6ffd700 1 --2- 172.21.15.46:0/2280128219 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71c81565e0 0x7f71c80a98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:09.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.712+0000 7f71c67fc700 1 -- 172.21.15.46:0/2280128219 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71c81565e0 msgr2=0x7f71c80a98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:09.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.712+0000 7f71c67fc700 1 --2- 172.21.15.46:0/2280128219 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71c81565e0 0x7f71c80a98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:09.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.712+0000 7f71c67fc700 1 -- 172.21.15.46:0/2280128219 >> v1:172.21.15.106:6789/0 conn(0x7f71c8069b20 legacy=0x7f71c80aa2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:09.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.712+0000 7f71c67fc700 1 -- 172.21.15.46:0/2280128219 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c80a9fc0 con 0x7f71c81569b0 2024-01-26T23:31:09.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.712+0000 7f71c6ffd700 1 --2- 172.21.15.46:0/2280128219 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71c81565e0 0x7f71c80a98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:31:09.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.712+0000 7f71c67fc700 1 --2- 172.21.15.46:0/2280128219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c81569b0 0x7f71c80ac4a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f71bc0099b0 tx=0x7f71bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f4cf57bedddcb9a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:09.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.713+0000 7f71c5ffb700 1 -- 172.21.15.46:0/2280128219 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc00b860 con 0x7f71c81569b0 2024-01-26T23:31:09.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.713+0000 7f71c5ffb700 1 -- 172.21.15.46:0/2280128219 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71bc00b9c0 con 0x7f71c81569b0 2024-01-26T23:31:09.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.713+0000 7f71c5ffb700 1 -- 172.21.15.46:0/2280128219 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc00bd60 con 0x7f71c81569b0 2024-01-26T23:31:09.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.713+0000 7f71cd3d9700 1 -- 172.21.15.46:0/2280128219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c81569b0 msgr2=0x7f71c80ac4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:09.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.713+0000 7f71cd3d9700 1 --2- 172.21.15.46:0/2280128219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c81569b0 0x7f71c80ac4a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f71bc0099b0 tx=0x7f71bc009250 comp rx=0 tx=0).stop 2024-01-26T23:31:09.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.713+0000 7f71cd3d9700 1 -- 172.21.15.46:0/2280128219 shutdown_connections 2024-01-26T23:31:09.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.713+0000 7f71cd3d9700 1 --2- 172.21.15.46:0/2280128219 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71c81565e0 0x7f71c80a98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:09.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.713+0000 7f71cd3d9700 1 --2- 172.21.15.46:0/2280128219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c81569b0 0x7f71c80ac4a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:09.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.713+0000 7f71cd3d9700 1 -- 172.21.15.46:0/2280128219 >> 172.21.15.46:0/2280128219 conn(0x7f71c8067930 msgr2=0x7f71c8067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:09.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.713+0000 7f71cd3d9700 1 -- 172.21.15.46:0/2280128219 shutdown_connections 2024-01-26T23:31:09.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.713+0000 7f71cd3d9700 1 -- 172.21.15.46:0/2280128219 wait complete. 2024-01-26T23:31:09.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.715+0000 7f71cd3d9700 1 Processor -- start 2024-01-26T23:31:09.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.715+0000 7f71cd3d9700 1 -- start start 2024-01-26T23:31:09.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.715+0000 7f71cd3d9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71c8069b20 0x7f71c816b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:09.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.715+0000 7f71cd3d9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71c81565e0 0x7f71c814a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.715+0000 7f71cd3d9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c81569b0 0x7f71c80ad660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.715+0000 7f71cd3d9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c8168670 con 0x7f71c81569b0 2024-01-26T23:31:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.715+0000 7f71cd3d9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c814a8c0 con 0x7f71c81565e0 2024-01-26T23:31:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.715+0000 7f71cd3d9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c814b300 con 0x7f71c8069b20 2024-01-26T23:31:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71c77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c81569b0 0x7f71c80ad660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71c77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c81569b0 0x7f71c80ad660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53476/0 (socket says 172.21.15.46:53476) 2024-01-26T23:31:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71c77fe700 1 -- 172.21.15.46:0/2784569725 learned_addr learned my addr 172.21.15.46:0/2784569725 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71c6ffd700 1 --2- 172.21.15.46:0/2784569725 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71c8069b20 0x7f71c816b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71c67fc700 1 --2- 172.21.15.46:0/2784569725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71c81565e0 0x7f71c814a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71c77fe700 1 -- 172.21.15.46:0/2784569725 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71c8069b20 msgr2=0x7f71c816b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71c77fe700 1 --2- 172.21.15.46:0/2784569725 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71c8069b20 0x7f71c816b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71c77fe700 1 -- 172.21.15.46:0/2784569725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71c81565e0 msgr2=0x7f71c814a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71c77fe700 1 --2- 172.21.15.46:0/2784569725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71c81565e0 0x7f71c814a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71c77fe700 1 -- 172.21.15.46:0/2784569725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc0094f0 con 0x7f71c81569b0 2024-01-26T23:31:09.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71c6ffd700 1 --2- 172.21.15.46:0/2784569725 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71c8069b20 0x7f71c816b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:31:09.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71c67fc700 1 --2- 172.21.15.46:0/2784569725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71c81565e0 0x7f71c814a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:31:09.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71c77fe700 1 --2- 172.21.15.46:0/2784569725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c81569b0 0x7f71c80ad660 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f71c000c8e0 tx=0x7f71c0001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:09.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71aeffd700 1 -- 172.21.15.46:0/2784569725 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c000ce80 con 0x7f71c81569b0 2024-01-26T23:31:09.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71aeffd700 1 -- 172.21.15.46:0/2784569725 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c000a7e0 con 0x7f71c81569b0 2024-01-26T23:31:09.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71cd3d9700 1 -- 172.21.15.46:0/2784569725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c8167d40 con 0x7f71c81569b0 2024-01-26T23:31:09.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71aeffd700 1 -- 172.21.15.46:0/2784569725 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c000ab80 con 0x7f71c81569b0 2024-01-26T23:31:09.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.716+0000 7f71cd3d9700 1 -- 172.21.15.46:0/2784569725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c8168300 con 0x7f71c81569b0 2024-01-26T23:31:09.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.717+0000 7f71cd3d9700 1 -- 172.21.15.46:0/2784569725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c804c710 con 0x7f71c81569b0 2024-01-26T23:31:09.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.718+0000 7f71aeffd700 1 -- 172.21.15.46:0/2784569725 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f71c0012420 con 0x7f71c81569b0 2024-01-26T23:31:09.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.718+0000 7f71aeffd700 1 --2- 172.21.15.46:0/2784569725 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f71b4066dd0 0x7f71b4069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:09.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.718+0000 7f71aeffd700 1 -- 172.21.15.46:0/2784569725 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f71c00847d0 con 0x7f71c81569b0 2024-01-26T23:31:09.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.718+0000 7f71c6ffd700 1 --2- 172.21.15.46:0/2784569725 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f71b4066dd0 0x7f71b4069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:09.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.719+0000 7f71c6ffd700 1 --2- 172.21.15.46:0/2784569725 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f71b4066dd0 0x7f71b4069280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f71b8006ea0 tx=0x7f71b8008040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:09.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.723+0000 7f71aeffd700 1 -- 172.21.15.46:0/2784569725 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f71c0051ab0 con 0x7f71c81569b0 2024-01-26T23:31:09.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:09.953+0000 7f71cd3d9700 1 -- 172.21.15.46:0/2784569725 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir3", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f71c81687b0 con 0x7f71b4066dd0 2024-01-26T23:31:10.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.008+0000 7f71aeffd700 1 -- 172.21.15.46:0/2784569725 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f71c81687b0 con 0x7f71b4066dd0 2024-01-26T23:31:10.011 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path /snap_test_dir1/dir3 2024-01-26T23:31:10.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.011+0000 7f71cd3d9700 1 -- 172.21.15.46:0/2784569725 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f71b4066dd0 msgr2=0x7f71b4069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:10.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.011+0000 7f71cd3d9700 1 --2- 172.21.15.46:0/2784569725 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f71b4066dd0 0x7f71b4069280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f71b8006ea0 tx=0x7f71b8008040 comp rx=0 tx=0).stop 2024-01-26T23:31:10.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.011+0000 7f71cd3d9700 1 -- 172.21.15.46:0/2784569725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c81569b0 msgr2=0x7f71c80ad660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:10.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.011+0000 7f71cd3d9700 1 --2- 172.21.15.46:0/2784569725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c81569b0 0x7f71c80ad660 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f71c000c8e0 tx=0x7f71c0001710 comp rx=0 tx=0).stop 2024-01-26T23:31:10.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.011+0000 7f71cd3d9700 1 -- 172.21.15.46:0/2784569725 shutdown_connections 2024-01-26T23:31:10.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.011+0000 7f71cd3d9700 1 --2- 172.21.15.46:0/2784569725 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f71b4066dd0 0x7f71b4069280 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.011+0000 7f71cd3d9700 1 --2- 172.21.15.46:0/2784569725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c81569b0 0x7f71c80ad660 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.011+0000 7f71cd3d9700 1 --2- 172.21.15.46:0/2784569725 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71c8069b20 0x7f71c816b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.011+0000 7f71cd3d9700 1 --2- 172.21.15.46:0/2784569725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71c81565e0 0x7f71c814a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.011+0000 7f71cd3d9700 1 -- 172.21.15.46:0/2784569725 >> 172.21.15.46:0/2784569725 conn(0x7f71c8067930 msgr2=0x7f71c80ae250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:10.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.011+0000 7f71cd3d9700 1 -- 172.21.15.46:0/2784569725 shutdown_connections 2024-01-26T23:31:10.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.011+0000 7f71cd3d9700 1 -- 172.21.15.46:0/2784569725 wait complete. 2024-01-26T23:31:10.027 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2024-01-26T23:31:10.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:31:10.027 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2024-01-26T23:31:10.049 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir4 1m 2024-01-26T23:31:10.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.227+0000 7f8660f81700 1 Processor -- start 2024-01-26T23:31:10.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.227+0000 7f8660f81700 1 -- start start 2024-01-26T23:31:10.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.228+0000 7f8660f81700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f865c156570 0x7f865c06c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:10.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.228+0000 7f8660f81700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f865c06f9a0 0x7f865c06eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:10.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.228+0000 7f8660f81700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f865c0c8b90 con 0x7f865c156940 2024-01-26T23:31:10.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.228+0000 7f8660f81700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f865c06d0b0 con 0x7f865c06f9a0 2024-01-26T23:31:10.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.228+0000 7f8660f81700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f865c0c8e50 con 0x7f865c156570 2024-01-26T23:31:10.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.228+0000 7f865ad9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f865c06f9a0 0x7f865c06eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:10.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.228+0000 7f865ad9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f865c06f9a0 0x7f865c06eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53482/0 (socket says 172.21.15.46:53482) 2024-01-26T23:31:10.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.228+0000 7f865ad9d700 1 -- 172.21.15.46:0/1825590765 learned_addr learned my addr 172.21.15.46:0/1825590765 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:10.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.228+0000 7f865a59c700 1 --2- 172.21.15.46:0/1825590765 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f865c156570 0x7f865c06c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:10.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.229+0000 7f865ad9d700 1 -- 172.21.15.46:0/1825590765 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f865c156570 msgr2=0x7f865c06c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:10.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.229+0000 7f865ad9d700 1 --2- 172.21.15.46:0/1825590765 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f865c156570 0x7f865c06c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.229+0000 7f865ad9d700 1 -- 172.21.15.46:0/1825590765 >> v1:172.21.15.106:6789/0 conn(0x7f865c156940 legacy=0x7f865c0c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:10.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.229+0000 7f865ad9d700 1 -- 172.21.15.46:0/1825590765 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865c0c4c40 con 0x7f865c06f9a0 2024-01-26T23:31:10.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.229+0000 7f865a59c700 1 --2- 172.21.15.46:0/1825590765 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f865c156570 0x7f865c06c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:31:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.229+0000 7f865ad9d700 1 --2- 172.21.15.46:0/1825590765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c06f9a0 0x7f865c06eb90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f865400d1d0 tx=0x7f865400ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc6059f3c2057e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.229+0000 7f865959a700 1 -- 172.21.15.46:0/1825590765 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8654010070 con 0x7f865c06f9a0 2024-01-26T23:31:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.229+0000 7f865959a700 1 -- 172.21.15.46:0/1825590765 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865400b0d0 con 0x7f865c06f9a0 2024-01-26T23:31:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.229+0000 7f865959a700 1 -- 172.21.15.46:0/1825590765 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865400b490 con 0x7f865c06f9a0 2024-01-26T23:31:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.229+0000 7f8660f81700 1 -- 172.21.15.46:0/1825590765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c06f9a0 msgr2=0x7f865c06eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.229+0000 7f8660f81700 1 --2- 172.21.15.46:0/1825590765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c06f9a0 0x7f865c06eb90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f865400d1d0 tx=0x7f865400ba50 comp rx=0 tx=0).stop 2024-01-26T23:31:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.230+0000 7f8660f81700 1 -- 172.21.15.46:0/1825590765 shutdown_connections 2024-01-26T23:31:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.230+0000 7f8660f81700 1 --2- 172.21.15.46:0/1825590765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c06f9a0 0x7f865c06eb90 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.230+0000 7f8660f81700 1 --2- 172.21.15.46:0/1825590765 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f865c156570 0x7f865c06c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.230+0000 7f8660f81700 1 -- 172.21.15.46:0/1825590765 >> 172.21.15.46:0/1825590765 conn(0x7f865c0cdf50 msgr2=0x7f865c0c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.230+0000 7f8660f81700 1 -- 172.21.15.46:0/1825590765 shutdown_connections 2024-01-26T23:31:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.230+0000 7f8660f81700 1 -- 172.21.15.46:0/1825590765 wait complete. 2024-01-26T23:31:10.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.231+0000 7f8660f81700 1 Processor -- start 2024-01-26T23:31:10.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.231+0000 7f8660f81700 1 -- start start 2024-01-26T23:31:10.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f8660f81700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f865c06f9a0 0x7f865c0accc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:10.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f8660f81700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c156570 0x7f865c141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:10.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f8660f81700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f865c156940 0x7f865c0ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:10.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f8660f81700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865c153e00 con 0x7f865c156570 2024-01-26T23:31:10.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f8660f81700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865c0ad630 con 0x7f865c06f9a0 2024-01-26T23:31:10.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f8660f81700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f865c0ad910 con 0x7f865c156940 2024-01-26T23:31:10.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f8659d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c156570 0x7f865c141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:10.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f8659d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c156570 0x7f865c141ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53490/0 (socket says 172.21.15.46:53490) 2024-01-26T23:31:10.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f8659d9b700 1 -- 172.21.15.46:0/1921707825 learned_addr learned my addr 172.21.15.46:0/1921707825 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:10.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f865a59c700 1 --2- 172.21.15.46:0/1921707825 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f865c06f9a0 0x7f865c0accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:10.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f865ad9d700 1 --2- 172.21.15.46:0/1921707825 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f865c156940 0x7f865c0ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:10.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f8659d9b700 1 -- 172.21.15.46:0/1921707825 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f865c156940 msgr2=0x7f865c0ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:10.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f8659d9b700 1 --2- 172.21.15.46:0/1921707825 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f865c156940 0x7f865c0ae290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f8659d9b700 1 -- 172.21.15.46:0/1921707825 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f865c06f9a0 msgr2=0x7f865c0accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:10.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f8659d9b700 1 --2- 172.21.15.46:0/1921707825 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f865c06f9a0 0x7f865c0accc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f8659d9b700 1 -- 172.21.15.46:0/1921707825 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865400bcf0 con 0x7f865c156570 2024-01-26T23:31:10.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f865a59c700 1 --2- 172.21.15.46:0/1921707825 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f865c06f9a0 0x7f865c0accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:31:10.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f865ad9d700 1 --2- 172.21.15.46:0/1921707825 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f865c156940 0x7f865c0ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:31:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.232+0000 7f8659d9b700 1 --2- 172.21.15.46:0/1921707825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c156570 0x7f865c141ed0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f864c00c4e0 tx=0x7f864c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.233+0000 7f86527fc700 1 -- 172.21.15.46:0/1921707825 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c011590 con 0x7f865c156570 2024-01-26T23:31:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.233+0000 7f86527fc700 1 -- 172.21.15.46:0/1921707825 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864c011710 con 0x7f865c156570 2024-01-26T23:31:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.233+0000 7f8660f81700 1 -- 172.21.15.46:0/1921707825 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f865c0adbf0 con 0x7f865c156570 2024-01-26T23:31:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.233+0000 7f86527fc700 1 -- 172.21.15.46:0/1921707825 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c011ad0 con 0x7f865c156570 2024-01-26T23:31:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.233+0000 7f8660f81700 1 -- 172.21.15.46:0/1921707825 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f865c0ad200 con 0x7f865c156570 2024-01-26T23:31:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.234+0000 7f8660f81700 1 -- 172.21.15.46:0/1921707825 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f865c153fb0 con 0x7f865c156570 2024-01-26T23:31:10.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.235+0000 7f86527fc700 1 -- 172.21.15.46:0/1921707825 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f864c01b3b0 con 0x7f865c156570 2024-01-26T23:31:10.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.235+0000 7f86527fc700 1 --2- 172.21.15.46:0/1921707825 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f863c066dd0 0x7f863c069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:10.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.235+0000 7f86527fc700 1 -- 172.21.15.46:0/1921707825 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f864c083c90 con 0x7f865c156570 2024-01-26T23:31:10.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.236+0000 7f865a59c700 1 --2- 172.21.15.46:0/1921707825 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f863c066dd0 0x7f863c069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:10.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.236+0000 7f865a59c700 1 --2- 172.21.15.46:0/1921707825 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f863c066dd0 0x7f863c069280 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8644006f00 tx=0x7f8644008040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:10.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.239+0000 7f86527fc700 1 -- 172.21.15.46:0/1921707825 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f864c050f20 con 0x7f865c156570 2024-01-26T23:31:10.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.361+0000 7f86527fc700 1 -- 172.21.15.46:0/1921707825 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 9) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f864c0512f0 con 0x7f865c156570 2024-01-26T23:31:10.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.472+0000 7f8660f81700 1 -- 172.21.15.46:0/1921707825 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir4", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f865c0ad420 con 0x7f863c066dd0 2024-01-26T23:31:10.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.527+0000 7f86527fc700 1 -- 172.21.15.46:0/1921707825 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f865c0ad420 con 0x7f863c066dd0 2024-01-26T23:31:10.529 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path /snap_test_dir1/dir4 2024-01-26T23:31:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.530+0000 7f8660f81700 1 -- 172.21.15.46:0/1921707825 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f863c066dd0 msgr2=0x7f863c069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.530+0000 7f8660f81700 1 --2- 172.21.15.46:0/1921707825 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f863c066dd0 0x7f863c069280 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8644006f00 tx=0x7f8644008040 comp rx=0 tx=0).stop 2024-01-26T23:31:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.530+0000 7f8660f81700 1 -- 172.21.15.46:0/1921707825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c156570 msgr2=0x7f865c141ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.530+0000 7f8660f81700 1 --2- 172.21.15.46:0/1921707825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c156570 0x7f865c141ed0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f864c00c4e0 tx=0x7f864c00c8a0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.530+0000 7f8660f81700 1 -- 172.21.15.46:0/1921707825 shutdown_connections 2024-01-26T23:31:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.530+0000 7f8660f81700 1 --2- 172.21.15.46:0/1921707825 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f863c066dd0 0x7f863c069280 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.530+0000 7f8660f81700 1 --2- 172.21.15.46:0/1921707825 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f865c156940 0x7f865c0ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.530+0000 7f8660f81700 1 --2- 172.21.15.46:0/1921707825 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f865c06f9a0 0x7f865c0accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.530+0000 7f8660f81700 1 --2- 172.21.15.46:0/1921707825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c156570 0x7f865c141ed0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.530+0000 7f8660f81700 1 -- 172.21.15.46:0/1921707825 >> 172.21.15.46:0/1921707825 conn(0x7f865c0cdf50 msgr2=0x7f865c067930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.530+0000 7f8660f81700 1 -- 172.21.15.46:0/1921707825 shutdown_connections 2024-01-26T23:31:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.530+0000 7f8660f81700 1 -- 172.21.15.46:0/1921707825 wait complete. 2024-01-26T23:31:10.546 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2024-01-26T23:31:10.546 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:31:10.546 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2024-01-26T23:31:10.568 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir5 1m 2024-01-26T23:31:10.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.748+0000 7f90d9091700 1 Processor -- start 2024-01-26T23:31:10.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.748+0000 7f90d9091700 1 -- start start 2024-01-26T23:31:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.749+0000 7f90d9091700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90d4067af0 0x7f90d4067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.749+0000 7f90d9091700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f90d41597e0 0x7f90d4159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.749+0000 7f90d9091700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d406fdb0 con 0x7f90d406ebe0 2024-01-26T23:31:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.749+0000 7f90d9091700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d406f1a0 con 0x7f90d4067af0 2024-01-26T23:31:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.749+0000 7f90d9091700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d406f480 con 0x7f90d41597e0 2024-01-26T23:31:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.749+0000 7f90d2d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90d4067af0 0x7f90d4067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.749+0000 7f90d259c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f90d406ebe0 0x7f90d40ad8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55854/0 (socket says 172.21.15.46:55854) 2024-01-26T23:31:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.749+0000 7f90d259c700 1 -- 172.21.15.46:0/1321641025 learned_addr learned my addr 172.21.15.46:0/1321641025 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:31:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.749+0000 7f90d359e700 1 --2- 172.21.15.46:0/1321641025 >> v2:172.21.15.106:3300/0 conn(0x7f90d41597e0 0x7f90d4159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.749+0000 7f90d2d9d700 1 -- 172.21.15.46:0/1321641025 >> v2:172.21.15.106:3300/0 conn(0x7f90d41597e0 msgr2=0x7f90d4159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.749+0000 7f90d2d9d700 1 --2- 172.21.15.46:0/1321641025 >> v2:172.21.15.106:3300/0 conn(0x7f90d41597e0 0x7f90d4159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.749+0000 7f90d2d9d700 1 -- 172.21.15.46:0/1321641025 >> v1:172.21.15.46:6789/0 conn(0x7f90d406ebe0 legacy=0x7f90d40ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.749+0000 7f90d2d9d700 1 -- 172.21.15.46:0/1321641025 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d406f760 con 0x7f90d4067af0 2024-01-26T23:31:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.749+0000 7f90d359e700 1 --2- 172.21.15.46:0/1321641025 >> v2:172.21.15.106:3300/0 conn(0x7f90d41597e0 0x7f90d4159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:31:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.749+0000 7f90d2d9d700 1 --2- 172.21.15.46:0/1321641025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d4067af0 0x7f90d4067ec0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f90d415eb20 tx=0x7f90c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df11bdd28f4d74e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.750+0000 7f90d1d9b700 1 -- 172.21.15.46:0/1321641025 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c800b6b0 con 0x7f90d4067af0 2024-01-26T23:31:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.750+0000 7f90d1d9b700 1 -- 172.21.15.46:0/1321641025 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c800b810 con 0x7f90d4067af0 2024-01-26T23:31:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.750+0000 7f90d1d9b700 1 -- 172.21.15.46:0/1321641025 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c800bbb0 con 0x7f90d4067af0 2024-01-26T23:31:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.750+0000 7f90d9091700 1 -- 172.21.15.46:0/1321641025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d4067af0 msgr2=0x7f90d4067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.750+0000 7f90d9091700 1 --2- 172.21.15.46:0/1321641025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d4067af0 0x7f90d4067ec0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f90d415eb20 tx=0x7f90c80091a0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.750+0000 7f90d9091700 1 -- 172.21.15.46:0/1321641025 shutdown_connections 2024-01-26T23:31:10.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.750+0000 7f90d9091700 1 --2- 172.21.15.46:0/1321641025 >> v2:172.21.15.106:3300/0 conn(0x7f90d41597e0 0x7f90d4159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.750+0000 7f90d9091700 1 --2- 172.21.15.46:0/1321641025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d4067af0 0x7f90d4067ec0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.750+0000 7f90d9091700 1 -- 172.21.15.46:0/1321641025 >> 172.21.15.46:0/1321641025 conn(0x7f90d40cdf80 msgr2=0x7f90d406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:10.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.750+0000 7f90d9091700 1 -- 172.21.15.46:0/1321641025 shutdown_connections 2024-01-26T23:31:10.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.750+0000 7f90d9091700 1 -- 172.21.15.46:0/1321641025 wait complete. 2024-01-26T23:31:10.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.752+0000 7f90d9091700 1 Processor -- start 2024-01-26T23:31:10.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.752+0000 7f90d9091700 1 -- start start 2024-01-26T23:31:10.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.752+0000 7f90d9091700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90d4067af0 0x7f90d4152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:10.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.752+0000 7f90d9091700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d406ebe0 0x7f90d416b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:10.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90d9091700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90d41597e0 0x7f90d40cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:10.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90d9091700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d41534c0 con 0x7f90d406ebe0 2024-01-26T23:31:10.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90d9091700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d4155ea0 con 0x7f90d4067af0 2024-01-26T23:31:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90d9091700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d4156180 con 0x7f90d41597e0 2024-01-26T23:31:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90d259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d406ebe0 0x7f90d416b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90d259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d406ebe0 0x7f90d416b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42710/0 (socket says 172.21.15.46:42710) 2024-01-26T23:31:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90d259c700 1 -- 172.21.15.46:0/2751133479 learned_addr learned my addr 172.21.15.46:0/2751133479 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90d2d9d700 1 --2- 172.21.15.46:0/2751133479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90d4067af0 0x7f90d4152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90d359e700 1 --2- 172.21.15.46:0/2751133479 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90d41597e0 0x7f90d40cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90d259c700 1 -- 172.21.15.46:0/2751133479 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90d41597e0 msgr2=0x7f90d40cbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90d259c700 1 --2- 172.21.15.46:0/2751133479 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90d41597e0 0x7f90d40cbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90d259c700 1 -- 172.21.15.46:0/2751133479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90d4067af0 msgr2=0x7f90d4152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90d259c700 1 --2- 172.21.15.46:0/2751133479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90d4067af0 0x7f90d4152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90d259c700 1 -- 172.21.15.46:0/2751133479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c8009420 con 0x7f90d406ebe0 2024-01-26T23:31:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90d259c700 1 --2- 172.21.15.46:0/2751133479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d406ebe0 0x7f90d416b250 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f90c400c560 tx=0x7f90c4001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90baffd700 1 -- 172.21.15.46:0/2751133479 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4005b30 con 0x7f90d406ebe0 2024-01-26T23:31:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90baffd700 1 -- 172.21.15.46:0/2751133479 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c4005c90 con 0x7f90d406ebe0 2024-01-26T23:31:10.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.753+0000 7f90d9091700 1 -- 172.21.15.46:0/2751133479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d4156460 con 0x7f90d406ebe0 2024-01-26T23:31:10.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.754+0000 7f90baffd700 1 -- 172.21.15.46:0/2751133479 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4007320 con 0x7f90d406ebe0 2024-01-26T23:31:10.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.754+0000 7f90d9091700 1 -- 172.21.15.46:0/2751133479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d416af90 con 0x7f90d406ebe0 2024-01-26T23:31:10.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.754+0000 7f90d9091700 1 -- 172.21.15.46:0/2751133479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d4156980 con 0x7f90d406ebe0 2024-01-26T23:31:10.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.755+0000 7f90baffd700 1 -- 172.21.15.46:0/2751133479 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 9) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f90c401b3b0 con 0x7f90d406ebe0 2024-01-26T23:31:10.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.755+0000 7f90baffd700 1 --2- 172.21.15.46:0/2751133479 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f90c00670a0 0x7f90c0069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:10.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.755+0000 7f90baffd700 1 -- 172.21.15.46:0/2751133479 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f90c4083d80 con 0x7f90d406ebe0 2024-01-26T23:31:10.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.756+0000 7f90d2d9d700 1 --2- 172.21.15.46:0/2751133479 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f90c00670a0 0x7f90c0069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:10.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.756+0000 7f90d2d9d700 1 --2- 172.21.15.46:0/2751133479 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f90c00670a0 0x7f90c0069550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f90c80093f0 tx=0x7f90c8017000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:10.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.760+0000 7f90baffd700 1 -- 172.21.15.46:0/2751133479 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f90d4156980 con 0x7f90d406ebe0 2024-01-26T23:31:10.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:10.992+0000 7f90d9091700 1 -- 172.21.15.46:0/2751133479 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir5", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f90d4156010 con 0x7f90c00670a0 2024-01-26T23:31:11.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.054+0000 7f90baffd700 1 -- 172.21.15.46:0/2751133479 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f90d4156010 con 0x7f90c00670a0 2024-01-26T23:31:11.056 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path /snap_test_dir1/dir5 2024-01-26T23:31:11.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.057+0000 7f90d9091700 1 -- 172.21.15.46:0/2751133479 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f90c00670a0 msgr2=0x7f90c0069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:11.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.057+0000 7f90d9091700 1 --2- 172.21.15.46:0/2751133479 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f90c00670a0 0x7f90c0069550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f90c80093f0 tx=0x7f90c8017000 comp rx=0 tx=0).stop 2024-01-26T23:31:11.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.057+0000 7f90d9091700 1 -- 172.21.15.46:0/2751133479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d406ebe0 msgr2=0x7f90d416b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:11.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.057+0000 7f90d9091700 1 --2- 172.21.15.46:0/2751133479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d406ebe0 0x7f90d416b250 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f90c400c560 tx=0x7f90c4001020 comp rx=0 tx=0).stop 2024-01-26T23:31:11.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.057+0000 7f90d9091700 1 -- 172.21.15.46:0/2751133479 shutdown_connections 2024-01-26T23:31:11.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.057+0000 7f90d9091700 1 --2- 172.21.15.46:0/2751133479 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f90c00670a0 0x7f90c0069550 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.057+0000 7f90d9091700 1 --2- 172.21.15.46:0/2751133479 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90d41597e0 0x7f90d40cbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.057+0000 7f90d9091700 1 --2- 172.21.15.46:0/2751133479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90d4067af0 0x7f90d4152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.057+0000 7f90d9091700 1 --2- 172.21.15.46:0/2751133479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d406ebe0 0x7f90d416b250 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.057+0000 7f90d9091700 1 -- 172.21.15.46:0/2751133479 >> 172.21.15.46:0/2751133479 conn(0x7f90d40cdf80 msgr2=0x7f90d40cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:11.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.057+0000 7f90d9091700 1 -- 172.21.15.46:0/2751133479 shutdown_connections 2024-01-26T23:31:11.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.057+0000 7f90d9091700 1 -- 172.21.15.46:0/2751133479 wait complete. 2024-01-26T23:31:11.073 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2024-01-26T23:31:11.073 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:31:11.073 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2024-01-26T23:31:11.095 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir6 1m 2024-01-26T23:31:11.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.274+0000 7f46488a2700 1 Processor -- start 2024-01-26T23:31:11.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.274+0000 7f46488a2700 1 -- start start 2024-01-26T23:31:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.275+0000 7f46488a2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4644068b00 0x7f4644068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.275+0000 7f46488a2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f464406add0 0x7f464406b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.275+0000 7f46488a2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46440a9a80 con 0x7f464406a660 2024-01-26T23:31:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.275+0000 7f46488a2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464406b790 con 0x7f4644068b00 2024-01-26T23:31:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.275+0000 7f46488a2700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464406aab0 con 0x7f464406add0 2024-01-26T23:31:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.275+0000 7f464259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4644068b00 0x7f4644068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.276+0000 7f464259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4644068b00 0x7f4644068ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42714/0 (socket says 172.21.15.46:42714) 2024-01-26T23:31:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.276+0000 7f464259c700 1 -- 172.21.15.46:0/760914851 learned_addr learned my addr 172.21.15.46:0/760914851 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.276+0000 7f4642d9d700 1 --2- 172.21.15.46:0/760914851 >> v2:172.21.15.106:3300/0 conn(0x7f464406add0 0x7f464406b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.276+0000 7f464259c700 1 -- 172.21.15.46:0/760914851 >> v2:172.21.15.106:3300/0 conn(0x7f464406add0 msgr2=0x7f464406b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:11.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.276+0000 7f464259c700 1 --2- 172.21.15.46:0/760914851 >> v2:172.21.15.106:3300/0 conn(0x7f464406add0 0x7f464406b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.276+0000 7f464259c700 1 -- 172.21.15.46:0/760914851 >> v1:172.21.15.106:6789/0 conn(0x7f464406a660 legacy=0x7f4644159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:11.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.276+0000 7f464259c700 1 -- 172.21.15.46:0/760914851 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464415a680 con 0x7f4644068b00 2024-01-26T23:31:11.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.276+0000 7f464259c700 1 --2- 172.21.15.46:0/760914851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4644068b00 0x7f4644068ed0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4644069fa0 tx=0x7f4634009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c4d0edd7cf9cc2 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:11.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.276+0000 7f464159a700 1 -- 172.21.15.46:0/760914851 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4634005c10 con 0x7f4644068b00 2024-01-26T23:31:11.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.276+0000 7f464159a700 1 -- 172.21.15.46:0/760914851 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4634005d70 con 0x7f4644068b00 2024-01-26T23:31:11.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.276+0000 7f464159a700 1 -- 172.21.15.46:0/760914851 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463400b840 con 0x7f4644068b00 2024-01-26T23:31:11.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.277+0000 7f46488a2700 1 -- 172.21.15.46:0/760914851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4644068b00 msgr2=0x7f4644068ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:11.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.277+0000 7f46488a2700 1 --2- 172.21.15.46:0/760914851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4644068b00 0x7f4644068ed0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4644069fa0 tx=0x7f4634009220 comp rx=0 tx=0).stop 2024-01-26T23:31:11.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.277+0000 7f46488a2700 1 -- 172.21.15.46:0/760914851 shutdown_connections 2024-01-26T23:31:11.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.277+0000 7f46488a2700 1 --2- 172.21.15.46:0/760914851 >> v2:172.21.15.106:3300/0 conn(0x7f464406add0 0x7f464406b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.277+0000 7f46488a2700 1 --2- 172.21.15.46:0/760914851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4644068b00 0x7f4644068ed0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.277+0000 7f46488a2700 1 -- 172.21.15.46:0/760914851 >> 172.21.15.46:0/760914851 conn(0x7f46440ca100 msgr2=0x7f464406e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:11.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.277+0000 7f46488a2700 1 -- 172.21.15.46:0/760914851 shutdown_connections 2024-01-26T23:31:11.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.277+0000 7f46488a2700 1 -- 172.21.15.46:0/760914851 wait complete. 2024-01-26T23:31:11.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.278+0000 7f46488a2700 1 Processor -- start 2024-01-26T23:31:11.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.278+0000 7f46488a2700 1 -- start start 2024-01-26T23:31:11.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.279+0000 7f46488a2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4644068b00 0x7f46441571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:11.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.279+0000 7f46488a2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f464406a660 0x7f46440c8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:11.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.279+0000 7f46488a2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f464406add0 0x7f46440ad550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:11.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.279+0000 7f46488a2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46441576e0 con 0x7f464406add0 2024-01-26T23:31:11.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.279+0000 7f46488a2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46440c7fc0 con 0x7f4644068b00 2024-01-26T23:31:11.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.279+0000 7f46488a2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46440c7150 con 0x7f464406a660 2024-01-26T23:31:11.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.279+0000 7f4642d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f464406add0 0x7f46440ad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:11.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.279+0000 7f4642d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f464406add0 0x7f46440ad550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42728/0 (socket says 172.21.15.46:42728) 2024-01-26T23:31:11.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.279+0000 7f4642d9d700 1 -- 172.21.15.46:0/2341086353 learned_addr learned my addr 172.21.15.46:0/2341086353 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:11.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.279+0000 7f464259c700 1 --2- 172.21.15.46:0/2341086353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4644068b00 0x7f46441571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:11.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.279+0000 7f4641d9b700 1 --2- 172.21.15.46:0/2341086353 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f464406a660 0x7f46440c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:11.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.280+0000 7f4642d9d700 1 -- 172.21.15.46:0/2341086353 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f464406a660 msgr2=0x7f46440c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:11.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.280+0000 7f4642d9d700 1 --2- 172.21.15.46:0/2341086353 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f464406a660 0x7f46440c8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.280+0000 7f4642d9d700 1 -- 172.21.15.46:0/2341086353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4644068b00 msgr2=0x7f46441571a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:11.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.280+0000 7f4642d9d700 1 --2- 172.21.15.46:0/2341086353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4644068b00 0x7f46441571a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.280+0000 7f4642d9d700 1 -- 172.21.15.46:0/2341086353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4634009480 con 0x7f464406add0 2024-01-26T23:31:11.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.280+0000 7f4642d9d700 1 --2- 172.21.15.46:0/2341086353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f464406add0 0x7f46440ad550 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f463c009fd0 tx=0x7f463c009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:11.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.280+0000 7f46327fc700 1 -- 172.21.15.46:0/2341086353 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c00eca0 con 0x7f464406add0 2024-01-26T23:31:11.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.280+0000 7f46327fc700 1 -- 172.21.15.46:0/2341086353 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463c00ee00 con 0x7f464406add0 2024-01-26T23:31:11.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.280+0000 7f46488a2700 1 -- 172.21.15.46:0/2341086353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46440c73f0 con 0x7f464406add0 2024-01-26T23:31:11.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.280+0000 7f46327fc700 1 -- 172.21.15.46:0/2341086353 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c014670 con 0x7f464406add0 2024-01-26T23:31:11.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.280+0000 7f46488a2700 1 -- 172.21.15.46:0/2341086353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46440c7970 con 0x7f464406add0 2024-01-26T23:31:11.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.281+0000 7f46488a2700 1 -- 172.21.15.46:0/2341086353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464404c710 con 0x7f464406add0 2024-01-26T23:31:11.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.281+0000 7f46327fc700 1 -- 172.21.15.46:0/2341086353 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 9) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f463c0149e0 con 0x7f464406add0 2024-01-26T23:31:11.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.282+0000 7f46327fc700 1 --2- 172.21.15.46:0/2341086353 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f46240670a0 0x7f4624069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:11.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.282+0000 7f46327fc700 1 -- 172.21.15.46:0/2341086353 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f463c085020 con 0x7f464406add0 2024-01-26T23:31:11.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.282+0000 7f464259c700 1 --2- 172.21.15.46:0/2341086353 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f46240670a0 0x7f4624069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:11.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.283+0000 7f464259c700 1 --2- 172.21.15.46:0/2341086353 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f46240670a0 0x7f4624069550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4634003080 tx=0x7f4634015000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:11.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.286+0000 7f46327fc700 1 -- 172.21.15.46:0/2341086353 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f463c051f40 con 0x7f464406add0 2024-01-26T23:31:11.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.460+0000 7f46327fc700 1 -- 172.21.15.46:0/2341086353 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f463c012dc0 con 0x7f464406add0 2024-01-26T23:31:11.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.521+0000 7f46488a2700 1 -- 172.21.15.46:0/2341086353 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir6", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f46440c7d10 con 0x7f46240670a0 2024-01-26T23:31:11.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.575+0000 7f46327fc700 1 -- 172.21.15.46:0/2341086353 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f46440c7d10 con 0x7f46240670a0 2024-01-26T23:31:11.577 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path /snap_test_dir1/dir6 2024-01-26T23:31:11.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.578+0000 7f46488a2700 1 -- 172.21.15.46:0/2341086353 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f46240670a0 msgr2=0x7f4624069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:11.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.578+0000 7f46488a2700 1 --2- 172.21.15.46:0/2341086353 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f46240670a0 0x7f4624069550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4634003080 tx=0x7f4634015000 comp rx=0 tx=0).stop 2024-01-26T23:31:11.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.578+0000 7f46488a2700 1 -- 172.21.15.46:0/2341086353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f464406add0 msgr2=0x7f46440ad550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:11.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.578+0000 7f46488a2700 1 --2- 172.21.15.46:0/2341086353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f464406add0 0x7f46440ad550 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f463c009fd0 tx=0x7f463c009900 comp rx=0 tx=0).stop 2024-01-26T23:31:11.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.578+0000 7f46488a2700 1 -- 172.21.15.46:0/2341086353 shutdown_connections 2024-01-26T23:31:11.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.578+0000 7f46488a2700 1 --2- 172.21.15.46:0/2341086353 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f46240670a0 0x7f4624069550 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.578+0000 7f46488a2700 1 --2- 172.21.15.46:0/2341086353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f464406add0 0x7f46440ad550 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.578+0000 7f46488a2700 1 --2- 172.21.15.46:0/2341086353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4644068b00 0x7f46441571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.578+0000 7f46488a2700 1 --2- 172.21.15.46:0/2341086353 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f464406a660 0x7f46440c8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.578+0000 7f46488a2700 1 -- 172.21.15.46:0/2341086353 >> 172.21.15.46:0/2341086353 conn(0x7f46440ca100 msgr2=0x7f46440cba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:11.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.578+0000 7f46488a2700 1 -- 172.21.15.46:0/2341086353 shutdown_connections 2024-01-26T23:31:11.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.578+0000 7f46488a2700 1 -- 172.21.15.46:0/2341086353 wait complete. 2024-01-26T23:31:11.594 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2024-01-26T23:31:11.594 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:31:11.594 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2024-01-26T23:31:11.616 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir7 1m 2024-01-26T23:31:11.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.796+0000 7f914253d700 1 Processor -- start 2024-01-26T23:31:11.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.796+0000 7f914253d700 1 -- start start 2024-01-26T23:31:11.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.797+0000 7f914253d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f913c157490 0x7f913c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:11.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.797+0000 7f914253d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f913c157860 0x7f913c0cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:11.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.797+0000 7f914253d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c0cea80 con 0x7f913c06d6a0 2024-01-26T23:31:11.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.797+0000 7f914253d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f913c0cdc60 con 0x7f913c157860 2024-01-26T23:31:11.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.797+0000 7f914253d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0b2ae0 con 0x7f913c157490 2024-01-26T23:31:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.797+0000 7f9140ada700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f913c06d6a0 0x7f913c06dab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55858/0 (socket says 172.21.15.46:55858) 2024-01-26T23:31:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.797+0000 7f9140ada700 1 -- 172.21.15.46:0/4292110054 learned_addr learned my addr 172.21.15.46:0/4292110054 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:31:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.798+0000 7f913b7fe700 1 --2- 172.21.15.46:0/4292110054 >> v2:172.21.15.106:3300/0 conn(0x7f913c157860 0x7f913c0cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.798+0000 7f913bfff700 1 --2- 172.21.15.46:0/4292110054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f913c157490 0x7f913c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.798+0000 7f913affd700 1 -- 172.21.15.46:0/4292110054 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629081693 0 0) 0x7f913c0cea80 con 0x7f913c06d6a0 2024-01-26T23:31:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.798+0000 7f913affd700 1 -- 172.21.15.46:0/4292110054 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9124003570 con 0x7f913c06d6a0 2024-01-26T23:31:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.798+0000 7f913bfff700 1 -- 172.21.15.46:0/4292110054 >> v2:172.21.15.106:3300/0 conn(0x7f913c157860 msgr2=0x7f913c0cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.798+0000 7f913bfff700 1 --2- 172.21.15.46:0/4292110054 >> v2:172.21.15.106:3300/0 conn(0x7f913c157860 0x7f913c0cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.798+0000 7f913affd700 1 -- 172.21.15.46:0/4292110054 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1296536674 0 0) 0x7f9124003570 con 0x7f913c06d6a0 2024-01-26T23:31:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.798+0000 7f913bfff700 1 -- 172.21.15.46:0/4292110054 >> v1:172.21.15.46:6789/0 conn(0x7f913c06d6a0 legacy=0x7f913c06dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.798+0000 7f913bfff700 1 -- 172.21.15.46:0/4292110054 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c0cde70 con 0x7f913c157490 2024-01-26T23:31:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.798+0000 7f913b7fe700 1 --2- 172.21.15.46:0/4292110054 >> v2:172.21.15.106:3300/0 conn(0x7f913c157860 0x7f913c0cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:31:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.798+0000 7f913bfff700 1 --2- 172.21.15.46:0/4292110054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f913c157490 0x7f913c0cc600 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f912c0098e0 tx=0x7f912c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26e8cd9760805ddb server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.798+0000 7f913affd700 1 -- 172.21.15.46:0/4292110054 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91340017f0 con 0x7f913c157490 2024-01-26T23:31:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.798+0000 7f913affd700 1 -- 172.21.15.46:0/4292110054 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912c011420 con 0x7f913c157490 2024-01-26T23:31:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.799+0000 7f913affd700 1 -- 172.21.15.46:0/4292110054 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c0117c0 con 0x7f913c157490 2024-01-26T23:31:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.799+0000 7f914253d700 1 -- 172.21.15.46:0/4292110054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f913c157490 msgr2=0x7f913c0cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.799+0000 7f914253d700 1 --2- 172.21.15.46:0/4292110054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f913c157490 0x7f913c0cc600 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f912c0098e0 tx=0x7f912c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.799+0000 7f914253d700 1 -- 172.21.15.46:0/4292110054 shutdown_connections 2024-01-26T23:31:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.799+0000 7f914253d700 1 --2- 172.21.15.46:0/4292110054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f913c157490 0x7f913c0cc600 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.799+0000 7f914253d700 1 --2- 172.21.15.46:0/4292110054 >> v2:172.21.15.106:3300/0 conn(0x7f913c157860 0x7f913c0cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.799+0000 7f914253d700 1 -- 172.21.15.46:0/4292110054 >> 172.21.15.46:0/4292110054 conn(0x7f913c067930 msgr2=0x7f913c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.799+0000 7f914253d700 1 -- 172.21.15.46:0/4292110054 shutdown_connections 2024-01-26T23:31:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.799+0000 7f914253d700 1 -- 172.21.15.46:0/4292110054 wait complete. 2024-01-26T23:31:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.800+0000 7f914253d700 1 Processor -- start 2024-01-26T23:31:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.801+0000 7f914253d700 1 -- start start 2024-01-26T23:31:11.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.801+0000 7f914253d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f913c06d6a0 0x7f913c14bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:11.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.801+0000 7f914253d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f913c157490 0x7f913c1553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:11.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.801+0000 7f914253d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f913c157860 0x7f913c1558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:11.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.801+0000 7f914253d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c1583e0 con 0x7f913c06d6a0 2024-01-26T23:31:11.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.801+0000 7f914253d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c158f60 con 0x7f913c157860 2024-01-26T23:31:11.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.801+0000 7f914253d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c158630 con 0x7f913c157490 2024-01-26T23:31:11.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.802+0000 7f913bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f913c06d6a0 0x7f913c14bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:11.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.802+0000 7f913bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f913c06d6a0 0x7f913c14bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42734/0 (socket says 172.21.15.46:42734) 2024-01-26T23:31:11.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.802+0000 7f913bfff700 1 -- 172.21.15.46:0/438878849 learned_addr learned my addr 172.21.15.46:0/438878849 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:11.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.802+0000 7f9140ada700 1 --2- 172.21.15.46:0/438878849 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f913c157860 0x7f913c1558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:11.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.802+0000 7f913b7fe700 1 --2- 172.21.15.46:0/438878849 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f913c157490 0x7f913c1553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:11.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.802+0000 7f913bfff700 1 -- 172.21.15.46:0/438878849 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f913c157490 msgr2=0x7f913c1553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:11.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.802+0000 7f913bfff700 1 --2- 172.21.15.46:0/438878849 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f913c157490 0x7f913c1553b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.802+0000 7f913bfff700 1 -- 172.21.15.46:0/438878849 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f913c157860 msgr2=0x7f913c1558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:11.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.802+0000 7f913bfff700 1 --2- 172.21.15.46:0/438878849 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f913c157860 0x7f913c1558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:11.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.802+0000 7f913bfff700 1 -- 172.21.15.46:0/438878849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9130009420 con 0x7f913c06d6a0 2024-01-26T23:31:11.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.802+0000 7f913b7fe700 1 --2- 172.21.15.46:0/438878849 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f913c157490 0x7f913c1553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:31:11.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.802+0000 7f913bfff700 1 --2- 172.21.15.46:0/438878849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f913c06d6a0 0x7f913c14bd00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f912c0030c0 tx=0x7f912c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:11.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.802+0000 7f9123fff700 1 -- 172.21.15.46:0/438878849 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c01cc90 con 0x7f913c06d6a0 2024-01-26T23:31:11.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.802+0000 7f9123fff700 1 -- 172.21.15.46:0/438878849 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912c01cdf0 con 0x7f913c06d6a0 2024-01-26T23:31:11.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.802+0000 7f914253d700 1 -- 172.21.15.46:0/438878849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f912c009420 con 0x7f913c06d6a0 2024-01-26T23:31:11.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.802+0000 7f9123fff700 1 -- 172.21.15.46:0/438878849 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c01a6a0 con 0x7f913c06d6a0 2024-01-26T23:31:11.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.802+0000 7f914253d700 1 -- 172.21.15.46:0/438878849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913c158bf0 con 0x7f913c06d6a0 2024-01-26T23:31:11.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.803+0000 7f914253d700 1 -- 172.21.15.46:0/438878849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913c04c710 con 0x7f913c06d6a0 2024-01-26T23:31:11.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.804+0000 7f9123fff700 1 -- 172.21.15.46:0/438878849 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f912c01aad0 con 0x7f913c06d6a0 2024-01-26T23:31:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.804+0000 7f9123fff700 1 --2- 172.21.15.46:0/438878849 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9128067130 0x7f91280695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.805+0000 7f9123fff700 1 -- 172.21.15.46:0/438878849 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f912c0864c0 con 0x7f913c06d6a0 2024-01-26T23:31:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.805+0000 7f913b7fe700 1 --2- 172.21.15.46:0/438878849 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9128067130 0x7f91280695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.805+0000 7f913b7fe700 1 --2- 172.21.15.46:0/438878849 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9128067130 0x7f91280695e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f91300093f0 tx=0x7f9130008040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:11.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:11.809+0000 7f9123fff700 1 -- 172.21.15.46:0/438878849 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f912c053270 con 0x7f913c06d6a0 2024-01-26T23:31:12.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.040+0000 7f914253d700 1 -- 172.21.15.46:0/438878849 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir7", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f913c0b14d0 con 0x7f9128067130 2024-01-26T23:31:12.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.098+0000 7f9123fff700 1 -- 172.21.15.46:0/438878849 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f913c0b14d0 con 0x7f9128067130 2024-01-26T23:31:12.100 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path /snap_test_dir1/dir7 2024-01-26T23:31:12.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.101+0000 7f914253d700 1 -- 172.21.15.46:0/438878849 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9128067130 msgr2=0x7f91280695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:12.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.101+0000 7f914253d700 1 --2- 172.21.15.46:0/438878849 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9128067130 0x7f91280695e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f91300093f0 tx=0x7f9130008040 comp rx=0 tx=0).stop 2024-01-26T23:31:12.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.101+0000 7f914253d700 1 -- 172.21.15.46:0/438878849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f913c06d6a0 msgr2=0x7f913c14bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:12.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.101+0000 7f914253d700 1 --2- 172.21.15.46:0/438878849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f913c06d6a0 0x7f913c14bd00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f912c0030c0 tx=0x7f912c016000 comp rx=0 tx=0).stop 2024-01-26T23:31:12.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.101+0000 7f914253d700 1 -- 172.21.15.46:0/438878849 shutdown_connections 2024-01-26T23:31:12.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.101+0000 7f914253d700 1 --2- 172.21.15.46:0/438878849 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9128067130 0x7f91280695e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.101+0000 7f914253d700 1 --2- 172.21.15.46:0/438878849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f913c06d6a0 0x7f913c14bd00 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.101+0000 7f914253d700 1 --2- 172.21.15.46:0/438878849 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f913c157860 0x7f913c1558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.101+0000 7f914253d700 1 --2- 172.21.15.46:0/438878849 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f913c157490 0x7f913c1553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.101+0000 7f914253d700 1 -- 172.21.15.46:0/438878849 >> 172.21.15.46:0/438878849 conn(0x7f913c067930 msgr2=0x7f913c153f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:12.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.101+0000 7f914253d700 1 -- 172.21.15.46:0/438878849 shutdown_connections 2024-01-26T23:31:12.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.101+0000 7f914253d700 1 -- 172.21.15.46:0/438878849 wait complete. 2024-01-26T23:31:12.117 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2024-01-26T23:31:12.117 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:31:12.117 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2024-01-26T23:31:12.140 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir8 1m 2024-01-26T23:31:12.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.321+0000 7f1c92fc4700 1 Processor -- start 2024-01-26T23:31:12.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.322+0000 7f1c92fc4700 1 -- start start 2024-01-26T23:31:12.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.322+0000 7f1c92fc4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c8c0c93a0 0x7f1c8c0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:12.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.322+0000 7f1c92fc4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c8c067d50 0x7f1c8c159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:12.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.322+0000 7f1c92fc4700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1c8c0681c0 0x7f1c8c159760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:12.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.322+0000 7f1c92fc4700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c8c15e390 con 0x7f1c8c0c93a0 2024-01-26T23:31:12.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.322+0000 7f1c92fc4700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c8c159ca0 con 0x7f1c8c0681c0 2024-01-26T23:31:12.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.322+0000 7f1c92fc4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c15d810 con 0x7f1c8c067d50 2024-01-26T23:31:12.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.323+0000 7f1c90d60700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c8c0c93a0 0x7f1c8c0c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:12.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.323+0000 7f1c90d60700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c8c0c93a0 0x7f1c8c0c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42750/0 (socket says 172.21.15.46:42750) 2024-01-26T23:31:12.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.323+0000 7f1c90d60700 1 -- 172.21.15.46:0/1992256306 learned_addr learned my addr 172.21.15.46:0/1992256306 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:12.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.323+0000 7f1c91561700 1 --2- 172.21.15.46:0/1992256306 >> v2:172.21.15.106:3300/0 conn(0x7f1c8c0681c0 0x7f1c8c159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.323+0000 7f1c8bfff700 1 --2- 172.21.15.46:0/1992256306 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c8c067d50 0x7f1c8c159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.323+0000 7f1c90d60700 1 -- 172.21.15.46:0/1992256306 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c8c067d50 msgr2=0x7f1c8c159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.323+0000 7f1c90d60700 1 --2- 172.21.15.46:0/1992256306 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c8c067d50 0x7f1c8c159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.323+0000 7f1c90d60700 1 -- 172.21.15.46:0/1992256306 >> v2:172.21.15.106:3300/0 conn(0x7f1c8c0681c0 msgr2=0x7f1c8c159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.323+0000 7f1c90d60700 1 --2- 172.21.15.46:0/1992256306 >> v2:172.21.15.106:3300/0 conn(0x7f1c8c0681c0 0x7f1c8c159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.323+0000 7f1c90d60700 1 -- 172.21.15.46:0/1992256306 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c15daf0 con 0x7f1c8c0c93a0 2024-01-26T23:31:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.323+0000 7f1c8bfff700 1 --2- 172.21.15.46:0/1992256306 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c8c067d50 0x7f1c8c159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:31:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.323+0000 7f1c90d60700 1 --2- 172.21.15.46:0/1992256306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c8c0c93a0 0x7f1c8c0c9770 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0c7a30 tx=0x7f1c80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5138abfd7cc445 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.323+0000 7f1c8b7fe700 1 -- 172.21.15.46:0/1992256306 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80009e70 con 0x7f1c8c0c93a0 2024-01-26T23:31:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.323+0000 7f1c8b7fe700 1 -- 172.21.15.46:0/1992256306 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c8000b860 con 0x7f1c8c0c93a0 2024-01-26T23:31:12.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.323+0000 7f1c8b7fe700 1 -- 172.21.15.46:0/1992256306 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8000bc20 con 0x7f1c8c0c93a0 2024-01-26T23:31:12.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.324+0000 7f1c92fc4700 1 -- 172.21.15.46:0/1992256306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c8c0c93a0 msgr2=0x7f1c8c0c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:12.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.324+0000 7f1c92fc4700 1 --2- 172.21.15.46:0/1992256306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c8c0c93a0 0x7f1c8c0c9770 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0c7a30 tx=0x7f1c80009250 comp rx=0 tx=0).stop 2024-01-26T23:31:12.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.324+0000 7f1c92fc4700 1 -- 172.21.15.46:0/1992256306 shutdown_connections 2024-01-26T23:31:12.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.324+0000 7f1c92fc4700 1 --2- 172.21.15.46:0/1992256306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c8c0c93a0 0x7f1c8c0c9770 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.324+0000 7f1c92fc4700 1 --2- 172.21.15.46:0/1992256306 >> v2:172.21.15.106:3300/0 conn(0x7f1c8c0681c0 0x7f1c8c159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.324+0000 7f1c92fc4700 1 --2- 172.21.15.46:0/1992256306 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c8c067d50 0x7f1c8c159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.324+0000 7f1c92fc4700 1 -- 172.21.15.46:0/1992256306 >> 172.21.15.46:0/1992256306 conn(0x7f1c8c0ca9a0 msgr2=0x7f1c8c068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:12.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.324+0000 7f1c92fc4700 1 -- 172.21.15.46:0/1992256306 shutdown_connections 2024-01-26T23:31:12.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.324+0000 7f1c92fc4700 1 -- 172.21.15.46:0/1992256306 wait complete. 2024-01-26T23:31:12.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.325+0000 7f1c92fc4700 1 Processor -- start 2024-01-26T23:31:12.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.325+0000 7f1c92fc4700 1 -- start start 2024-01-26T23:31:12.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.326+0000 7f1c92fc4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c8c067d50 0x7f1c8c0c5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:12.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.326+0000 7f1c92fc4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c8c0681c0 0x7f1c8c14cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:12.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.326+0000 7f1c92fc4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c8c0c93a0 0x7f1c8c0c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:12.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.326+0000 7f1c92fc4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c0c54c0 con 0x7f1c8c067d50 2024-01-26T23:31:12.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.326+0000 7f1c92fc4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c0c5e90 con 0x7f1c8c0c93a0 2024-01-26T23:31:12.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.326+0000 7f1c92fc4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c0c48b0 con 0x7f1c8c0681c0 2024-01-26T23:31:12.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.326+0000 7f1c90d60700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c8c067d50 0x7f1c8c0c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:12.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.326+0000 7f1c90d60700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c8c067d50 0x7f1c8c0c5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42766/0 (socket says 172.21.15.46:42766) 2024-01-26T23:31:12.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.326+0000 7f1c90d60700 1 -- 172.21.15.46:0/2555009064 learned_addr learned my addr 172.21.15.46:0/2555009064 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:12.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.326+0000 7f1c91561700 1 --2- 172.21.15.46:0/2555009064 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c8c0c93a0 0x7f1c8c0c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:12.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.326+0000 7f1c8bfff700 1 --2- 172.21.15.46:0/2555009064 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c8c0681c0 0x7f1c8c14cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:12.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.327+0000 7f1c90d60700 1 -- 172.21.15.46:0/2555009064 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c8c0681c0 msgr2=0x7f1c8c14cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:12.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.327+0000 7f1c90d60700 1 --2- 172.21.15.46:0/2555009064 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c8c0681c0 0x7f1c8c14cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.327+0000 7f1c90d60700 1 -- 172.21.15.46:0/2555009064 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c8c0c93a0 msgr2=0x7f1c8c0c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:12.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.327+0000 7f1c90d60700 1 --2- 172.21.15.46:0/2555009064 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c8c0c93a0 0x7f1c8c0c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.327+0000 7f1c90d60700 1 -- 172.21.15.46:0/2555009064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c800094f0 con 0x7f1c8c067d50 2024-01-26T23:31:12.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.327+0000 7f1c8bfff700 1 --2- 172.21.15.46:0/2555009064 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c8c0681c0 0x7f1c8c14cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:31:12.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.327+0000 7f1c90d60700 1 --2- 172.21.15.46:0/2555009064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c8c067d50 0x7f1c8c0c5950 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1c800094c0 tx=0x7f1c80017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:12.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.327+0000 7f1c88ff9700 1 -- 172.21.15.46:0/2555009064 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80006af0 con 0x7f1c8c067d50 2024-01-26T23:31:12.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.327+0000 7f1c88ff9700 1 -- 172.21.15.46:0/2555009064 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c80000bc0 con 0x7f1c8c067d50 2024-01-26T23:31:12.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.327+0000 7f1c88ff9700 1 -- 172.21.15.46:0/2555009064 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8000bc00 con 0x7f1c8c067d50 2024-01-26T23:31:12.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.327+0000 7f1c92fc4700 1 -- 172.21.15.46:0/2555009064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c8c0c4b90 con 0x7f1c8c067d50 2024-01-26T23:31:12.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.327+0000 7f1c92fc4700 1 -- 172.21.15.46:0/2555009064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c8c0c5150 con 0x7f1c8c067d50 2024-01-26T23:31:12.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.328+0000 7f1c92fc4700 1 -- 172.21.15.46:0/2555009064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c8c04c710 con 0x7f1c8c067d50 2024-01-26T23:31:12.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.329+0000 7f1c88ff9700 1 -- 172.21.15.46:0/2555009064 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f1c80023070 con 0x7f1c8c067d50 2024-01-26T23:31:12.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.329+0000 7f1c88ff9700 1 --2- 172.21.15.46:0/2555009064 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1c70067130 0x7f1c700695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:12.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.329+0000 7f1c88ff9700 1 -- 172.21.15.46:0/2555009064 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f1c80085870 con 0x7f1c8c067d50 2024-01-26T23:31:12.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.329+0000 7f1c8bfff700 1 --2- 172.21.15.46:0/2555009064 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1c70067130 0x7f1c700695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:12.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.330+0000 7f1c8bfff700 1 --2- 172.21.15.46:0/2555009064 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1c70067130 0x7f1c700695e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1c78006f00 tx=0x7f1c78008040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:12.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.333+0000 7f1c88ff9700 1 -- 172.21.15.46:0/2555009064 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1c800526d0 con 0x7f1c8c067d50 2024-01-26T23:31:12.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.567+0000 7f1c92fc4700 1 -- 172.21.15.46:0/2555009064 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir8", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c8c0c5600 con 0x7f1c70067130 2024-01-26T23:31:12.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.619+0000 7f1c88ff9700 1 -- 172.21.15.46:0/2555009064 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f1c7800f740 con 0x7f1c70067130 2024-01-26T23:31:12.621 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path /snap_test_dir1/dir8 2024-01-26T23:31:12.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.622+0000 7f1c92fc4700 1 -- 172.21.15.46:0/2555009064 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1c70067130 msgr2=0x7f1c700695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:12.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.622+0000 7f1c92fc4700 1 --2- 172.21.15.46:0/2555009064 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1c70067130 0x7f1c700695e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1c78006f00 tx=0x7f1c78008040 comp rx=0 tx=0).stop 2024-01-26T23:31:12.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.622+0000 7f1c92fc4700 1 -- 172.21.15.46:0/2555009064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c8c067d50 msgr2=0x7f1c8c0c5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:12.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.622+0000 7f1c92fc4700 1 --2- 172.21.15.46:0/2555009064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c8c067d50 0x7f1c8c0c5950 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1c800094c0 tx=0x7f1c80017000 comp rx=0 tx=0).stop 2024-01-26T23:31:12.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.622+0000 7f1c92fc4700 1 -- 172.21.15.46:0/2555009064 shutdown_connections 2024-01-26T23:31:12.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.622+0000 7f1c92fc4700 1 --2- 172.21.15.46:0/2555009064 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1c70067130 0x7f1c700695e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.622+0000 7f1c92fc4700 1 --2- 172.21.15.46:0/2555009064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c8c067d50 0x7f1c8c0c5950 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.622+0000 7f1c92fc4700 1 --2- 172.21.15.46:0/2555009064 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c8c0c93a0 0x7f1c8c0c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.622+0000 7f1c92fc4700 1 --2- 172.21.15.46:0/2555009064 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c8c0681c0 0x7f1c8c14cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.622+0000 7f1c92fc4700 1 -- 172.21.15.46:0/2555009064 >> 172.21.15.46:0/2555009064 conn(0x7f1c8c0ca9a0 msgr2=0x7f1c8c1532c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:12.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.622+0000 7f1c92fc4700 1 -- 172.21.15.46:0/2555009064 shutdown_connections 2024-01-26T23:31:12.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.622+0000 7f1c92fc4700 1 -- 172.21.15.46:0/2555009064 wait complete. 2024-01-26T23:31:12.638 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2024-01-26T23:31:12.638 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:31:12.638 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2024-01-26T23:31:12.660 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir9 1m 2024-01-26T23:31:12.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.841+0000 7f538e712700 1 Processor -- start 2024-01-26T23:31:12.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.841+0000 7f538e712700 1 -- start start 2024-01-26T23:31:12.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.842+0000 7f538e712700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f53881565a0 0x7f5388156970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:12.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.842+0000 7f538e712700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53880b0830 con 0x7f538815a520 2024-01-26T23:31:12.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.842+0000 7f538e712700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53880ad350 con 0x7f53880ae2f0 2024-01-26T23:31:12.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.842+0000 7f538e712700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53880ae130 con 0x7f53881565a0 2024-01-26T23:31:12.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f53877fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f538815a520 0x7f53880acc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55866/0 (socket says 172.21.15.46:55866) 2024-01-26T23:31:12.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f53877fe700 1 -- 172.21.15.46:0/4292297324 learned_addr learned my addr 172.21.15.46:0/4292297324 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:31:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f5387fff700 1 --2- 172.21.15.46:0/4292297324 >> v2:172.21.15.106:3300/0 conn(0x7f53881565a0 0x7f5388156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f5386ffd700 1 -- 172.21.15.46:0/4292297324 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316358790 0 0) 0x7f53880b0830 con 0x7f538815a520 2024-01-26T23:31:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f5386ffd700 1 -- 172.21.15.46:0/4292297324 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5370003570 con 0x7f538815a520 2024-01-26T23:31:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f5386ffd700 1 -- 172.21.15.46:0/4292297324 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 72225786 0 0) 0x7f5370003570 con 0x7f538815a520 2024-01-26T23:31:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f5386ffd700 1 -- 172.21.15.46:0/4292297324 >> v2:172.21.15.106:3300/0 conn(0x7f53881565a0 msgr2=0x7f5388156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f5386ffd700 1 --2- 172.21.15.46:0/4292297324 >> v2:172.21.15.106:3300/0 conn(0x7f53881565a0 0x7f5388156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f5386ffd700 1 -- 172.21.15.46:0/4292297324 >> v1:172.21.15.106:6789/0 conn(0x7f53880ae2f0 legacy=0x7f53880ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f5386ffd700 1 -- 172.21.15.46:0/4292297324 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53880ad5b0 con 0x7f538815a520 2024-01-26T23:31:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f5386ffd700 1 -- 172.21.15.46:0/4292297324 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f537c001470 con 0x7f538815a520 2024-01-26T23:31:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f5386ffd700 1 -- 172.21.15.46:0/4292297324 >> v1:172.21.15.46:6789/0 conn(0x7f538815a520 legacy=0x7f53880acc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f5386ffd700 1 --2- 172.21.15.46:0/4292297324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5370003a60 0x7f5370003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f5386ffd700 1 --2- 172.21.15.46:0/4292297324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5370004810 0x7f5370004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f5386ffd700 1 --2- 172.21.15.46:0/4292297324 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53700051c0 0x7f537000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f5386ffd700 1 -- 172.21.15.46:0/4292297324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53880ae130 con 0x7f5370003a60 2024-01-26T23:31:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f5386ffd700 1 -- 172.21.15.46:0/4292297324 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537000c160 con 0x7f5370004810 2024-01-26T23:31:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f5386ffd700 1 -- 172.21.15.46:0/4292297324 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537000c320 con 0x7f53700051c0 2024-01-26T23:31:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f5387fff700 1 --2- 172.21.15.46:0/4292297324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5370003a60 0x7f5370003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.843+0000 7f53877fe700 1 --2- 172.21.15.46:0/4292297324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5370004810 0x7f5370004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.844+0000 7f538ccaf700 1 --2- 172.21.15.46:0/4292297324 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53700051c0 0x7f537000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.844+0000 7f5387fff700 1 -- 172.21.15.46:0/4292297324 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53700051c0 msgr2=0x7f537000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.844+0000 7f5387fff700 1 --2- 172.21.15.46:0/4292297324 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53700051c0 0x7f537000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.844+0000 7f5387fff700 1 -- 172.21.15.46:0/4292297324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5370004810 msgr2=0x7f5370004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.844+0000 7f5387fff700 1 --2- 172.21.15.46:0/4292297324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5370004810 0x7f5370004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.844+0000 7f5387fff700 1 -- 172.21.15.46:0/4292297324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5370003570 con 0x7f5370003a60 2024-01-26T23:31:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.844+0000 7f538ccaf700 1 --2- 172.21.15.46:0/4292297324 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53700051c0 0x7f537000ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:31:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.844+0000 7f53877fe700 1 --2- 172.21.15.46:0/4292297324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5370004810 0x7f5370004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:31:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.844+0000 7f538ccaf700 1 -- 172.21.15.46:0/4292297324 reap_dead start 2024-01-26T23:31:12.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.844+0000 7f5387fff700 1 --2- 172.21.15.46:0/4292297324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5370003a60 0x7f5370003e70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5378009c00 tx=0x7f53780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fb787ba15841ac9 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:12.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.844+0000 7f5386ffd700 1 -- 172.21.15.46:0/4292297324 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537800e710 con 0x7f5370003a60 2024-01-26T23:31:12.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.844+0000 7f5386ffd700 1 -- 172.21.15.46:0/4292297324 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f537800e870 con 0x7f5370003a60 2024-01-26T23:31:12.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.844+0000 7f5386ffd700 1 -- 172.21.15.46:0/4292297324 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537800ec30 con 0x7f5370003a60 2024-01-26T23:31:12.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.844+0000 7f538e712700 1 -- 172.21.15.46:0/4292297324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5370003a60 msgr2=0x7f5370003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:12.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.844+0000 7f538e712700 1 --2- 172.21.15.46:0/4292297324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5370003a60 0x7f5370003e70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5378009c00 tx=0x7f53780091a0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.845+0000 7f538e712700 1 -- 172.21.15.46:0/4292297324 shutdown_connections 2024-01-26T23:31:12.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.845+0000 7f538e712700 1 --2- 172.21.15.46:0/4292297324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5370003a60 0x7f5370003e70 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.845+0000 7f538e712700 1 -- 172.21.15.46:0/4292297324 >> 172.21.15.46:0/4292297324 conn(0x7f53880cdf80 msgr2=0x7f538806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:12.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.845+0000 7f538e712700 1 -- 172.21.15.46:0/4292297324 shutdown_connections 2024-01-26T23:31:12.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.845+0000 7f538e712700 1 -- 172.21.15.46:0/4292297324 wait complete. 2024-01-26T23:31:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.846+0000 7f538e712700 1 Processor -- start 2024-01-26T23:31:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.847+0000 7f538e712700 1 -- start start 2024-01-26T23:31:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.847+0000 7f538e712700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5370003a60 0x7f5388151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f538e712700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5388156a90 0x7f538814a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f538e712700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53880b1fa0 0x7f538814fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f538e712700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5388156f00 con 0x7f53880b1fa0 2024-01-26T23:31:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f538e712700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53881525f0 con 0x7f5388156a90 2024-01-26T23:31:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f538e712700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5388153e20 con 0x7f5370003a60 2024-01-26T23:31:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f538ccaf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53880b1fa0 0x7f538814fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f538ccaf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53880b1fa0 0x7f538814fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42794/0 (socket says 172.21.15.46:42794) 2024-01-26T23:31:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f538ccaf700 1 -- 172.21.15.46:0/410347534 learned_addr learned my addr 172.21.15.46:0/410347534 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f53877fe700 1 --2- 172.21.15.46:0/410347534 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5388156a90 0x7f538814a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f5387fff700 1 --2- 172.21.15.46:0/410347534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5370003a60 0x7f5388151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f538ccaf700 1 -- 172.21.15.46:0/410347534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5370003a60 msgr2=0x7f5388151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f538ccaf700 1 --2- 172.21.15.46:0/410347534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5370003a60 0x7f5388151a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f538ccaf700 1 -- 172.21.15.46:0/410347534 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5388156a90 msgr2=0x7f538814a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f538ccaf700 1 --2- 172.21.15.46:0/410347534 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5388156a90 0x7f538814a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f538ccaf700 1 -- 172.21.15.46:0/410347534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538814a7f0 con 0x7f53880b1fa0 2024-01-26T23:31:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f538ccaf700 1 --2- 172.21.15.46:0/410347534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53880b1fa0 0x7f538814fbc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f53880b0370 tx=0x7f5380009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f536ffff700 1 -- 172.21.15.46:0/410347534 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5380017070 con 0x7f53880b1fa0 2024-01-26T23:31:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f536ffff700 1 -- 172.21.15.46:0/410347534 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5380006ba0 con 0x7f53880b1fa0 2024-01-26T23:31:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f536ffff700 1 -- 172.21.15.46:0/410347534 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538000b070 con 0x7f53880b1fa0 2024-01-26T23:31:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.848+0000 7f538e712700 1 -- 172.21.15.46:0/410347534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5388152030 con 0x7f53880b1fa0 2024-01-26T23:31:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.849+0000 7f538e712700 1 -- 172.21.15.46:0/410347534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5388153ab0 con 0x7f53880b1fa0 2024-01-26T23:31:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.849+0000 7f538e712700 1 -- 172.21.15.46:0/410347534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f538804c710 con 0x7f53880b1fa0 2024-01-26T23:31:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.850+0000 7f536ffff700 1 -- 172.21.15.46:0/410347534 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f5380003030 con 0x7f53880b1fa0 2024-01-26T23:31:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.850+0000 7f536ffff700 1 --2- 172.21.15.46:0/410347534 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5374067130 0x7f53740695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.851+0000 7f536ffff700 1 -- 172.21.15.46:0/410347534 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f5380084240 con 0x7f53880b1fa0 2024-01-26T23:31:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.851+0000 7f5387fff700 1 --2- 172.21.15.46:0/410347534 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5374067130 0x7f53740695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:12.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.851+0000 7f5387fff700 1 --2- 172.21.15.46:0/410347534 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5374067130 0x7f53740695e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f53780091e0 tx=0x7f537801a000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:12.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:12.855+0000 7f536ffff700 1 -- 172.21.15.46:0/410347534 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5380050e40 con 0x7f53880b1fa0 2024-01-26T23:31:13.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.086+0000 7f538e712700 1 -- 172.21.15.46:0/410347534 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir9", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5388153f60 con 0x7f5374067130 2024-01-26T23:31:13.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.146+0000 7f536ffff700 1 -- 172.21.15.46:0/410347534 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f5388153f60 con 0x7f5374067130 2024-01-26T23:31:13.148 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path /snap_test_dir1/dir9 2024-01-26T23:31:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.149+0000 7f538e712700 1 -- 172.21.15.46:0/410347534 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5374067130 msgr2=0x7f53740695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.149+0000 7f538e712700 1 --2- 172.21.15.46:0/410347534 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5374067130 0x7f53740695e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f53780091e0 tx=0x7f537801a000 comp rx=0 tx=0).stop 2024-01-26T23:31:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.149+0000 7f538e712700 1 -- 172.21.15.46:0/410347534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53880b1fa0 msgr2=0x7f538814fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.149+0000 7f538e712700 1 --2- 172.21.15.46:0/410347534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53880b1fa0 0x7f538814fbc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f53880b0370 tx=0x7f5380009040 comp rx=0 tx=0).stop 2024-01-26T23:31:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.149+0000 7f538e712700 1 -- 172.21.15.46:0/410347534 shutdown_connections 2024-01-26T23:31:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.149+0000 7f538e712700 1 --2- 172.21.15.46:0/410347534 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5374067130 0x7f53740695e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.149+0000 7f538e712700 1 --2- 172.21.15.46:0/410347534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53880b1fa0 0x7f538814fbc0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.149+0000 7f538e712700 1 --2- 172.21.15.46:0/410347534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5370003a60 0x7f5388151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.149+0000 7f538e712700 1 --2- 172.21.15.46:0/410347534 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5388156a90 0x7f538814a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.149+0000 7f538e712700 1 -- 172.21.15.46:0/410347534 >> 172.21.15.46:0/410347534 conn(0x7f53880cdf80 msgr2=0x7f538815a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:13.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.149+0000 7f538e712700 1 -- 172.21.15.46:0/410347534 shutdown_connections 2024-01-26T23:31:13.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.149+0000 7f538e712700 1 -- 172.21.15.46:0/410347534 wait complete. 2024-01-26T23:31:13.165 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2024-01-26T23:31:13.165 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir0 1m 2024-01-26T23:31:13.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.302+0000 7f409b157700 1 Processor -- start 2024-01-26T23:31:13.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.302+0000 7f409b157700 1 -- start start 2024-01-26T23:31:13.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.302+0000 7f409b157700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4094141200 0x7f40940c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:13.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.303+0000 7f409b157700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40940ccc00 con 0x7f40940caf30 2024-01-26T23:31:13.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.303+0000 7f409b157700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40940cc310 con 0x7f40940c8da0 2024-01-26T23:31:13.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.303+0000 7f409b157700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40940cbbf0 con 0x7f4094141200 2024-01-26T23:31:13.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.303+0000 7f40996f4700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f40940caf30 0x7f40940cb370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55874/0 (socket says 172.21.15.46:55874) 2024-01-26T23:31:13.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.303+0000 7f40996f4700 1 -- 172.21.15.46:0/923586771 learned_addr learned my addr 172.21.15.46:0/923586771 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:31:13.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.303+0000 7f4093fff700 1 --2- 172.21.15.46:0/923586771 >> v2:172.21.15.106:3300/0 conn(0x7f4094141200 0x7f40940c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:13.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.303+0000 7f40937fe700 1 -- 172.21.15.46:0/923586771 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052520289 0 0) 0x7f40940ccc00 con 0x7f40940caf30 2024-01-26T23:31:13.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.303+0000 7f40937fe700 1 -- 172.21.15.46:0/923586771 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4080003570 con 0x7f40940caf30 2024-01-26T23:31:13.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.303+0000 7f40937fe700 1 -- 172.21.15.46:0/923586771 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 835972470 0 0) 0x7f4080003570 con 0x7f40940caf30 2024-01-26T23:31:13.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.303+0000 7f40937fe700 1 -- 172.21.15.46:0/923586771 >> v2:172.21.15.106:3300/0 conn(0x7f4094141200 msgr2=0x7f40940c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:13.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.303+0000 7f40937fe700 1 --2- 172.21.15.46:0/923586771 >> v2:172.21.15.106:3300/0 conn(0x7f4094141200 0x7f40940c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.303+0000 7f40937fe700 1 -- 172.21.15.46:0/923586771 >> v1:172.21.15.106:6789/0 conn(0x7f40940c8da0 legacy=0x7f4094140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.303+0000 7f40937fe700 1 -- 172.21.15.46:0/923586771 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40940cc5b0 con 0x7f40940caf30 2024-01-26T23:31:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.303+0000 7f40937fe700 1 -- 172.21.15.46:0/923586771 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f408c0017f0 con 0x7f40940caf30 2024-01-26T23:31:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.303+0000 7f40937fe700 1 -- 172.21.15.46:0/923586771 >> v1:172.21.15.46:6789/0 conn(0x7f40940caf30 legacy=0x7f40940cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.303+0000 7f40937fe700 1 --2- 172.21.15.46:0/923586771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4080003a60 0x7f4080003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.303+0000 7f40937fe700 1 --2- 172.21.15.46:0/923586771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4080004810 0x7f4080004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.304+0000 7f40937fe700 1 --2- 172.21.15.46:0/923586771 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40800051c0 0x7f408000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.304+0000 7f40937fe700 1 -- 172.21.15.46:0/923586771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40940cbbf0 con 0x7f4080003a60 2024-01-26T23:31:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.304+0000 7f40937fe700 1 -- 172.21.15.46:0/923586771 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408000c160 con 0x7f4080004810 2024-01-26T23:31:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.304+0000 7f40937fe700 1 -- 172.21.15.46:0/923586771 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f408000c320 con 0x7f40800051c0 2024-01-26T23:31:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.304+0000 7f4098ef3700 1 --2- 172.21.15.46:0/923586771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4080003a60 0x7f4080003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.304+0000 7f4093fff700 1 --2- 172.21.15.46:0/923586771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4080004810 0x7f4080004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.304+0000 7f40996f4700 1 --2- 172.21.15.46:0/923586771 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40800051c0 0x7f408000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.304+0000 7f4098ef3700 1 -- 172.21.15.46:0/923586771 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40800051c0 msgr2=0x7f408000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.304+0000 7f4098ef3700 1 --2- 172.21.15.46:0/923586771 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40800051c0 0x7f408000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.304+0000 7f4098ef3700 1 -- 172.21.15.46:0/923586771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4080004810 msgr2=0x7f4080004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.304+0000 7f4098ef3700 1 --2- 172.21.15.46:0/923586771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4080004810 0x7f4080004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.304+0000 7f4098ef3700 1 -- 172.21.15.46:0/923586771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4080003570 con 0x7f4080003a60 2024-01-26T23:31:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.304+0000 7f40996f4700 1 -- 172.21.15.46:0/923586771 reap_dead start 2024-01-26T23:31:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.304+0000 7f4098ef3700 1 --2- 172.21.15.46:0/923586771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4080003a60 0x7f4080003e70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f40840018b0 tx=0x7f4084007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91bb72170b9320ad server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.304+0000 7f40937fe700 1 -- 172.21.15.46:0/923586771 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4084012420 con 0x7f4080003a60 2024-01-26T23:31:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.304+0000 7f40937fe700 1 -- 172.21.15.46:0/923586771 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40840125a0 con 0x7f4080003a60 2024-01-26T23:31:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.304+0000 7f40937fe700 1 -- 172.21.15.46:0/923586771 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4084012960 con 0x7f4080003a60 2024-01-26T23:31:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.305+0000 7f409b157700 1 -- 172.21.15.46:0/923586771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4080003a60 msgr2=0x7f4080003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.305+0000 7f409b157700 1 --2- 172.21.15.46:0/923586771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4080003a60 0x7f4080003e70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f40840018b0 tx=0x7f4084007d30 comp rx=0 tx=0).stop 2024-01-26T23:31:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.305+0000 7f409b157700 1 -- 172.21.15.46:0/923586771 shutdown_connections 2024-01-26T23:31:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.305+0000 7f409b157700 1 --2- 172.21.15.46:0/923586771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4080003a60 0x7f4080003e70 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.305+0000 7f409b157700 1 -- 172.21.15.46:0/923586771 >> 172.21.15.46:0/923586771 conn(0x7f409406bd90 msgr2=0x7f409406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.305+0000 7f409b157700 1 -- 172.21.15.46:0/923586771 shutdown_connections 2024-01-26T23:31:13.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.305+0000 7f409b157700 1 -- 172.21.15.46:0/923586771 wait complete. 2024-01-26T23:31:13.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.306+0000 7f409b157700 1 Processor -- start 2024-01-26T23:31:13.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.306+0000 7f409b157700 1 -- start start 2024-01-26T23:31:13.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.307+0000 7f409b157700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4080003a60 0x7f40940c3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:13.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.307+0000 7f409b157700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40940b1340 0x7f40940b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:13.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.307+0000 7f409b157700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4094130a00 0x7f4094130e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:13.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.307+0000 7f409b157700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40940c5650 con 0x7f4080003a60 2024-01-26T23:31:13.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.307+0000 7f409b157700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40940c5c10 con 0x7f40940b1340 2024-01-26T23:31:13.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.307+0000 7f409b157700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f409413d270 con 0x7f4094130a00 2024-01-26T23:31:13.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.307+0000 7f4098ef3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4080003a60 0x7f40940c3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:13.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.307+0000 7f4098ef3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4080003a60 0x7f40940c3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42818/0 (socket says 172.21.15.46:42818) 2024-01-26T23:31:13.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.307+0000 7f4098ef3700 1 -- 172.21.15.46:0/2358046267 learned_addr learned my addr 172.21.15.46:0/2358046267 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:13.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.307+0000 7f4093fff700 1 --2- 172.21.15.46:0/2358046267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40940b1340 0x7f40940b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:13.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.307+0000 7f40996f4700 1 --2- 172.21.15.46:0/2358046267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4094130a00 0x7f4094130e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:13.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.308+0000 7f4098ef3700 1 -- 172.21.15.46:0/2358046267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4094130a00 msgr2=0x7f4094130e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:13.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.308+0000 7f4098ef3700 1 --2- 172.21.15.46:0/2358046267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4094130a00 0x7f4094130e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.308+0000 7f4098ef3700 1 -- 172.21.15.46:0/2358046267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40940b1340 msgr2=0x7f40940b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:13.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.308+0000 7f4098ef3700 1 --2- 172.21.15.46:0/2358046267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40940b1340 0x7f40940b17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.308+0000 7f4098ef3700 1 -- 172.21.15.46:0/2358046267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40940b1e70 con 0x7f4080003a60 2024-01-26T23:31:13.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.308+0000 7f4093fff700 1 --2- 172.21.15.46:0/2358046267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40940b1340 0x7f40940b17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:31:13.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.308+0000 7f4098ef3700 1 --2- 172.21.15.46:0/2358046267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4080003a60 0x7f40940c3590 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f408400c3f0 tx=0x7f4084015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:13.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.308+0000 7f4090ff9700 1 -- 172.21.15.46:0/2358046267 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4084001200 con 0x7f4080003a60 2024-01-26T23:31:13.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.308+0000 7f4090ff9700 1 -- 172.21.15.46:0/2358046267 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4084012cd0 con 0x7f4080003a60 2024-01-26T23:31:13.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.308+0000 7f4090ff9700 1 -- 172.21.15.46:0/2358046267 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4084012940 con 0x7f4080003a60 2024-01-26T23:31:13.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.308+0000 7f409b157700 1 -- 172.21.15.46:0/2358046267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40940b2150 con 0x7f4080003a60 2024-01-26T23:31:13.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.308+0000 7f409b157700 1 -- 172.21.15.46:0/2358046267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40941313a0 con 0x7f4080003a60 2024-01-26T23:31:13.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.309+0000 7f409b157700 1 -- 172.21.15.46:0/2358046267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f409404c710 con 0x7f4080003a60 2024-01-26T23:31:13.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.310+0000 7f4090ff9700 1 -- 172.21.15.46:0/2358046267 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f408401f620 con 0x7f4080003a60 2024-01-26T23:31:13.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.310+0000 7f4090ff9700 1 --2- 172.21.15.46:0/2358046267 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4074067130 0x7f40740695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:13.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.310+0000 7f4090ff9700 1 -- 172.21.15.46:0/2358046267 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f4084086410 con 0x7f4080003a60 2024-01-26T23:31:13.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.310+0000 7f4093fff700 1 --2- 172.21.15.46:0/2358046267 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4074067130 0x7f40740695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:13.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.311+0000 7f4093fff700 1 --2- 172.21.15.46:0/2358046267 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4074067130 0x7f40740695e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f409416cf40 tx=0x7f407c0091a0 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:13.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.315+0000 7f4090ff9700 1 -- 172.21.15.46:0/2358046267 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f40840531c0 con 0x7f4080003a60 2024-01-26T23:31:13.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.552+0000 7f409b157700 1 -- 172.21.15.46:0/2358046267 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f40940cc890 con 0x7f4074067130 2024-01-26T23:31:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.598+0000 7f4090ff9700 1 -- 172.21.15.46:0/2358046267 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f40940cc890 con 0x7f4074067130 2024-01-26T23:31:13.600 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule activated for path /snap_test_dir1/dir0 2024-01-26T23:31:13.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.601+0000 7f409b157700 1 -- 172.21.15.46:0/2358046267 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4074067130 msgr2=0x7f40740695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:13.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.601+0000 7f409b157700 1 --2- 172.21.15.46:0/2358046267 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4074067130 0x7f40740695e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f409416cf40 tx=0x7f407c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.601+0000 7f409b157700 1 -- 172.21.15.46:0/2358046267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4080003a60 msgr2=0x7f40940c3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:13.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.601+0000 7f409b157700 1 --2- 172.21.15.46:0/2358046267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4080003a60 0x7f40940c3590 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f408400c3f0 tx=0x7f4084015000 comp rx=0 tx=0).stop 2024-01-26T23:31:13.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.601+0000 7f409b157700 1 -- 172.21.15.46:0/2358046267 shutdown_connections 2024-01-26T23:31:13.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.601+0000 7f409b157700 1 --2- 172.21.15.46:0/2358046267 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4074067130 0x7f40740695e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.601+0000 7f409b157700 1 --2- 172.21.15.46:0/2358046267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4080003a60 0x7f40940c3590 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.601+0000 7f409b157700 1 --2- 172.21.15.46:0/2358046267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4094130a00 0x7f4094130e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.601+0000 7f409b157700 1 --2- 172.21.15.46:0/2358046267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40940b1340 0x7f40940b17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.601+0000 7f409b157700 1 -- 172.21.15.46:0/2358046267 >> 172.21.15.46:0/2358046267 conn(0x7f409406bd90 msgr2=0x7f409413e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:13.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.601+0000 7f409b157700 1 -- 172.21.15.46:0/2358046267 shutdown_connections 2024-01-26T23:31:13.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.601+0000 7f409b157700 1 -- 172.21.15.46:0/2358046267 wait complete. 2024-01-26T23:31:13.618 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2024-01-26T23:31:13.618 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir1 1m 2024-01-26T23:31:13.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.753+0000 7f79307a3700 1 Processor -- start 2024-01-26T23:31:13.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.753+0000 7f79307a3700 1 -- start start 2024-01-26T23:31:13.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.754+0000 7f79307a3700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f79280c9770 0x7f7928159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:13.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.754+0000 7f79307a3700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792815e390 con 0x7f7928159760 2024-01-26T23:31:13.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.754+0000 7f79307a3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7928068460 con 0x7f79280c93a0 2024-01-26T23:31:13.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.754+0000 7f79307a3700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f792815d810 con 0x7f79280c9770 2024-01-26T23:31:13.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.754+0000 7f792ed40700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7928159760 0x7f79280c7e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55878/0 (socket says 172.21.15.46:55878) 2024-01-26T23:31:13.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.754+0000 7f792ed40700 1 -- 172.21.15.46:0/2347224100 learned_addr learned my addr 172.21.15.46:0/2347224100 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:31:13.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.754+0000 7f792dd3e700 1 --2- 172.21.15.46:0/2347224100 >> v2:172.21.15.106:3300/0 conn(0x7f79280c9770 0x7f7928159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:13.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.754+0000 7f792d53d700 1 -- 172.21.15.46:0/2347224100 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042295994 0 0) 0x7f792815e390 con 0x7f7928159760 2024-01-26T23:31:13.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.754+0000 7f792d53d700 1 -- 172.21.15.46:0/2347224100 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7914003570 con 0x7f7928159760 2024-01-26T23:31:13.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792d53d700 1 -- 172.21.15.46:0/2347224100 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1278755429 0 0) 0x7f7914003570 con 0x7f7928159760 2024-01-26T23:31:13.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792d53d700 1 -- 172.21.15.46:0/2347224100 >> v2:172.21.15.106:3300/0 conn(0x7f79280c9770 msgr2=0x7f7928159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:13.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792d53d700 1 --2- 172.21.15.46:0/2347224100 >> v2:172.21.15.106:3300/0 conn(0x7f79280c9770 0x7f7928159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792d53d700 1 -- 172.21.15.46:0/2347224100 >> v1:172.21.15.106:6789/0 conn(0x7f79280c93a0 legacy=0x7f7928067d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:13.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792d53d700 1 -- 172.21.15.46:0/2347224100 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792815daf0 con 0x7f7928159760 2024-01-26T23:31:13.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792d53d700 1 -- 172.21.15.46:0/2347224100 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f79200017f0 con 0x7f7928159760 2024-01-26T23:31:13.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792d53d700 1 -- 172.21.15.46:0/2347224100 >> v1:172.21.15.46:6789/0 conn(0x7f7928159760 legacy=0x7f79280c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:13.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792d53d700 1 --2- 172.21.15.46:0/2347224100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7914003a60 0x7f7914003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:13.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792d53d700 1 --2- 172.21.15.46:0/2347224100 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7914004810 0x7f7914004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:13.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792d53d700 1 --2- 172.21.15.46:0/2347224100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79140052e0 0x7f791400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:13.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792d53d700 1 -- 172.21.15.46:0/2347224100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792815d810 con 0x7f79140052e0 2024-01-26T23:31:13.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792d53d700 1 -- 172.21.15.46:0/2347224100 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791400c2a0 con 0x7f7914003a60 2024-01-26T23:31:13.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792d53d700 1 -- 172.21.15.46:0/2347224100 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791400c460 con 0x7f7914004810 2024-01-26T23:31:13.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792e53f700 1 --2- 172.21.15.46:0/2347224100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7914003a60 0x7f7914003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:13.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792dd3e700 1 --2- 172.21.15.46:0/2347224100 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7914004810 0x7f7914004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:13.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792ed40700 1 --2- 172.21.15.46:0/2347224100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79140052e0 0x7f791400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:13.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792e53f700 1 -- 172.21.15.46:0/2347224100 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7914004810 msgr2=0x7f7914004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:13.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792e53f700 1 --2- 172.21.15.46:0/2347224100 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7914004810 0x7f7914004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792e53f700 1 -- 172.21.15.46:0/2347224100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79140052e0 msgr2=0x7f791400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:13.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792e53f700 1 --2- 172.21.15.46:0/2347224100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79140052e0 0x7f791400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792e53f700 1 -- 172.21.15.46:0/2347224100 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7914003570 con 0x7f7914003a60 2024-01-26T23:31:13.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792ed40700 1 --2- 172.21.15.46:0/2347224100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79140052e0 0x7f791400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:31:13.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792ed40700 1 -- 172.21.15.46:0/2347224100 reap_dead start 2024-01-26T23:31:13.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.755+0000 7f792e53f700 1 --2- 172.21.15.46:0/2347224100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7914003a60 0x7f7914003e70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f79180018b0 tx=0x7f7918007d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc866e21093a2b3b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:13.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.756+0000 7f792d53d700 1 -- 172.21.15.46:0/2347224100 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7918012420 con 0x7f7914003a60 2024-01-26T23:31:13.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.756+0000 7f792d53d700 1 -- 172.21.15.46:0/2347224100 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79180125a0 con 0x7f7914003a60 2024-01-26T23:31:13.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.756+0000 7f792d53d700 1 -- 172.21.15.46:0/2347224100 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7918012960 con 0x7f7914003a60 2024-01-26T23:31:13.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.756+0000 7f79307a3700 1 -- 172.21.15.46:0/2347224100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7914003a60 msgr2=0x7f7914003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:13.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.756+0000 7f79307a3700 1 --2- 172.21.15.46:0/2347224100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7914003a60 0x7f7914003e70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f79180018b0 tx=0x7f7918007d30 comp rx=0 tx=0).stop 2024-01-26T23:31:13.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.756+0000 7f79307a3700 1 -- 172.21.15.46:0/2347224100 shutdown_connections 2024-01-26T23:31:13.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.756+0000 7f79307a3700 1 --2- 172.21.15.46:0/2347224100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7914003a60 0x7f7914003e70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.756+0000 7f79307a3700 1 -- 172.21.15.46:0/2347224100 >> 172.21.15.46:0/2347224100 conn(0x7f79280ca9a0 msgr2=0x7f7928068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:13.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.756+0000 7f79307a3700 1 -- 172.21.15.46:0/2347224100 shutdown_connections 2024-01-26T23:31:13.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.756+0000 7f79307a3700 1 -- 172.21.15.46:0/2347224100 wait complete. 2024-01-26T23:31:13.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.758+0000 7f79307a3700 1 Processor -- start 2024-01-26T23:31:13.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.758+0000 7f79307a3700 1 -- start start 2024-01-26T23:31:13.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.759+0000 7f79307a3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7914003a60 0x7f79280b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:13.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.759+0000 7f79307a3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f792815cb00 0x7f792814ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:13.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.759+0000 7f79307a3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79280b1e70 0x7f79280b22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:13.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.759+0000 7f79307a3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79280b10f0 con 0x7f792815cb00 2024-01-26T23:31:13.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.759+0000 7f79307a3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79280c4640 con 0x7f79280b1e70 2024-01-26T23:31:13.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.759+0000 7f79307a3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79280b2800 con 0x7f7914003a60 2024-01-26T23:31:13.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.759+0000 7f792dd3e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f792815cb00 0x7f792814ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:13.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.759+0000 7f792dd3e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f792815cb00 0x7f792814ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42846/0 (socket says 172.21.15.46:42846) 2024-01-26T23:31:13.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.759+0000 7f792dd3e700 1 -- 172.21.15.46:0/691544725 learned_addr learned my addr 172.21.15.46:0/691544725 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:13.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.759+0000 7f792e53f700 1 --2- 172.21.15.46:0/691544725 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7914003a60 0x7f79280b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:13.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.759+0000 7f792ed40700 1 --2- 172.21.15.46:0/691544725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79280b1e70 0x7f79280b22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:13.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.760+0000 7f792dd3e700 1 -- 172.21.15.46:0/691544725 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7914003a60 msgr2=0x7f79280b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:13.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.760+0000 7f792dd3e700 1 --2- 172.21.15.46:0/691544725 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7914003a60 0x7f79280b0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.760+0000 7f792dd3e700 1 -- 172.21.15.46:0/691544725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79280b1e70 msgr2=0x7f79280b22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:13.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.760+0000 7f792dd3e700 1 --2- 172.21.15.46:0/691544725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79280b1e70 0x7f79280b22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:13.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.760+0000 7f792dd3e700 1 -- 172.21.15.46:0/691544725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79280b12d0 con 0x7f792815cb00 2024-01-26T23:31:13.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.760+0000 7f792dd3e700 1 --2- 172.21.15.46:0/691544725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f792815cb00 0x7f792814ff20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7924011da0 tx=0x7f79240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:13.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.760+0000 7f791e7fc700 1 -- 172.21.15.46:0/691544725 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792400faa0 con 0x7f792815cb00 2024-01-26T23:31:13.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.760+0000 7f791e7fc700 1 -- 172.21.15.46:0/691544725 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f792400fc00 con 0x7f792815cb00 2024-01-26T23:31:13.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.760+0000 7f791e7fc700 1 -- 172.21.15.46:0/691544725 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7924014440 con 0x7f792815cb00 2024-01-26T23:31:13.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.760+0000 7f79307a3700 1 -- 172.21.15.46:0/691544725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79280b15b0 con 0x7f792815cb00 2024-01-26T23:31:13.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.760+0000 7f79307a3700 1 -- 172.21.15.46:0/691544725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79280b1b70 con 0x7f792815cb00 2024-01-26T23:31:13.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.761+0000 7f79307a3700 1 -- 172.21.15.46:0/691544725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79280c47d0 con 0x7f792815cb00 2024-01-26T23:31:13.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.762+0000 7f791e7fc700 1 -- 172.21.15.46:0/691544725 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f7924014870 con 0x7f792815cb00 2024-01-26T23:31:13.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.762+0000 7f791e7fc700 1 --2- 172.21.15.46:0/691544725 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f790c067130 0x7f790c0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:13.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.762+0000 7f791e7fc700 1 -- 172.21.15.46:0/691544725 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f7924018030 con 0x7f792815cb00 2024-01-26T23:31:13.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.762+0000 7f792e53f700 1 --2- 172.21.15.46:0/691544725 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f790c067130 0x7f790c0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:13.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.763+0000 7f792e53f700 1 --2- 172.21.15.46:0/691544725 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f790c067130 0x7f790c0695e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7918001010 tx=0x7f7918015000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:13.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.767+0000 7f791e7fc700 1 -- 172.21.15.46:0/691544725 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7924052220 con 0x7f792815cb00 2024-01-26T23:31:14.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:13.998+0000 7f79307a3700 1 -- 172.21.15.46:0/691544725 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f792804c710 con 0x7f790c067130 2024-01-26T23:31:14.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.046+0000 7f791e7fc700 1 -- 172.21.15.46:0/691544725 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f792804c710 con 0x7f790c067130 2024-01-26T23:31:14.048 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule activated for path /snap_test_dir1/dir1 2024-01-26T23:31:14.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.048+0000 7f79307a3700 1 -- 172.21.15.46:0/691544725 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f790c067130 msgr2=0x7f790c0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:14.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.048+0000 7f79307a3700 1 --2- 172.21.15.46:0/691544725 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f790c067130 0x7f790c0695e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7918001010 tx=0x7f7918015000 comp rx=0 tx=0).stop 2024-01-26T23:31:14.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.048+0000 7f79307a3700 1 -- 172.21.15.46:0/691544725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f792815cb00 msgr2=0x7f792814ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.048+0000 7f79307a3700 1 --2- 172.21.15.46:0/691544725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f792815cb00 0x7f792814ff20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7924011da0 tx=0x7f79240091a0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.048+0000 7f79307a3700 1 -- 172.21.15.46:0/691544725 shutdown_connections 2024-01-26T23:31:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.048+0000 7f79307a3700 1 --2- 172.21.15.46:0/691544725 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f790c067130 0x7f790c0695e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.048+0000 7f79307a3700 1 --2- 172.21.15.46:0/691544725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79280b1e70 0x7f79280b22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.048+0000 7f79307a3700 1 --2- 172.21.15.46:0/691544725 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7914003a60 0x7f79280b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.048+0000 7f79307a3700 1 --2- 172.21.15.46:0/691544725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f792815cb00 0x7f792814ff20 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.048+0000 7f79307a3700 1 -- 172.21.15.46:0/691544725 >> 172.21.15.46:0/691544725 conn(0x7f79280ca9a0 msgr2=0x7f79280c2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.048+0000 7f79307a3700 1 -- 172.21.15.46:0/691544725 shutdown_connections 2024-01-26T23:31:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.048+0000 7f79307a3700 1 -- 172.21.15.46:0/691544725 wait complete. 2024-01-26T23:31:14.064 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2024-01-26T23:31:14.065 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir2 1m 2024-01-26T23:31:14.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.200+0000 7f0ecd141700 1 Processor -- start 2024-01-26T23:31:14.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.200+0000 7f0ecd141700 1 -- start start 2024-01-26T23:31:14.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.201+0000 7f0ecd141700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0ec8157860 0x7f0ec80cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.201+0000 7f0ecd141700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ec806ddb0 0x7f0ec80cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.201+0000 7f0ecd141700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec80cea80 con 0x7f0ec8157490 2024-01-26T23:31:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.201+0000 7f0ecd141700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec80cdc60 con 0x7f0ec8157860 2024-01-26T23:31:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.201+0000 7f0ecd141700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec80b2ae0 con 0x7f0ec806ddb0 2024-01-26T23:31:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.201+0000 7f0ec659c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0ec8157860 0x7f0ec80cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.201+0000 7f0ec759e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ec806ddb0 0x7f0ec80cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.201+0000 7f0ec759e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ec806ddb0 0x7f0ec80cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:51618/0 (socket says 172.21.15.46:51618) 2024-01-26T23:31:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.201+0000 7f0ec659c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0ec8157860 0x7f0ec80cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:47906/0 (socket says 172.21.15.46:47906) 2024-01-26T23:31:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.201+0000 7f0ec759e700 1 -- 172.21.15.46:0/1238395248 learned_addr learned my addr 172.21.15.46:0/1238395248 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.202+0000 7f0ec759e700 1 -- 172.21.15.46:0/1238395248 >> v2:172.21.15.106:3300/0 conn(0x7f0ec8157860 msgr2=0x7f0ec80cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.202+0000 7f0ec759e700 1 --2- 172.21.15.46:0/1238395248 >> v2:172.21.15.106:3300/0 conn(0x7f0ec8157860 0x7f0ec80cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.202+0000 7f0ec759e700 1 -- 172.21.15.46:0/1238395248 >> v1:172.21.15.106:6789/0 conn(0x7f0ec8157490 legacy=0x7f0ec806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:14.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.202+0000 7f0ec759e700 1 -- 172.21.15.46:0/1238395248 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec80cde70 con 0x7f0ec806ddb0 2024-01-26T23:31:14.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.202+0000 7f0ec659c700 1 --2- 172.21.15.46:0/1238395248 >> v2:172.21.15.106:3300/0 conn(0x7f0ec8157860 0x7f0ec80cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:31:14.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.202+0000 7f0ec759e700 1 --2- 172.21.15.46:0/1238395248 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ec806ddb0 0x7f0ec80cd620 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0009f20 tx=0x7f0ec0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d4006b617f2688 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:14.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.202+0000 7f0ec5d9b700 1 -- 172.21.15.46:0/1238395248 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec000e070 con 0x7f0ec806ddb0 2024-01-26T23:31:14.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.202+0000 7f0ec5d9b700 1 -- 172.21.15.46:0/1238395248 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec000b9d0 con 0x7f0ec806ddb0 2024-01-26T23:31:14.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.202+0000 7f0ec5d9b700 1 -- 172.21.15.46:0/1238395248 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec00092e0 con 0x7f0ec806ddb0 2024-01-26T23:31:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.203+0000 7f0ecd141700 1 -- 172.21.15.46:0/1238395248 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ec806ddb0 msgr2=0x7f0ec80cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.203+0000 7f0ecd141700 1 --2- 172.21.15.46:0/1238395248 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ec806ddb0 0x7f0ec80cd620 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0009f20 tx=0x7f0ec0009840 comp rx=0 tx=0).stop 2024-01-26T23:31:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.203+0000 7f0ecd141700 1 -- 172.21.15.46:0/1238395248 shutdown_connections 2024-01-26T23:31:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.203+0000 7f0ecd141700 1 --2- 172.21.15.46:0/1238395248 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ec806ddb0 0x7f0ec80cd620 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.203+0000 7f0ecd141700 1 --2- 172.21.15.46:0/1238395248 >> v2:172.21.15.106:3300/0 conn(0x7f0ec8157860 0x7f0ec80cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.203+0000 7f0ecd141700 1 -- 172.21.15.46:0/1238395248 >> 172.21.15.46:0/1238395248 conn(0x7f0ec8067930 msgr2=0x7f0ec8067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.203+0000 7f0ecd141700 1 -- 172.21.15.46:0/1238395248 shutdown_connections 2024-01-26T23:31:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.203+0000 7f0ecd141700 1 -- 172.21.15.46:0/1238395248 wait complete. 2024-01-26T23:31:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.204+0000 7f0ecd141700 1 Processor -- start 2024-01-26T23:31:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.204+0000 7f0ecd141700 1 -- start start 2024-01-26T23:31:14.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.205+0000 7f0ecd141700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec806ddb0 0x7f0ec80c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:14.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.205+0000 7f0ecd141700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ec8157490 0x7f0ec816a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.205+0000 7f0ecd141700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ec8157860 0x7f0ec815c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.205+0000 7f0ecd141700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec80c9ae0 con 0x7f0ec806ddb0 2024-01-26T23:31:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.205+0000 7f0ecd141700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec80ca4b0 con 0x7f0ec8157490 2024-01-26T23:31:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.205+0000 7f0ecd141700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec80c8c30 con 0x7f0ec8157860 2024-01-26T23:31:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.205+0000 7f0ec6d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec806ddb0 0x7f0ec80c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.205+0000 7f0ec6d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec806ddb0 0x7f0ec80c9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42848/0 (socket says 172.21.15.46:42848) 2024-01-26T23:31:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.205+0000 7f0ec6d9d700 1 -- 172.21.15.46:0/1699440328 learned_addr learned my addr 172.21.15.46:0/1699440328 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.205+0000 7f0ec759e700 1 --2- 172.21.15.46:0/1699440328 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ec8157860 0x7f0ec815c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.205+0000 7f0ec659c700 1 --2- 172.21.15.46:0/1699440328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ec8157490 0x7f0ec816a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.205+0000 7f0ec6d9d700 1 -- 172.21.15.46:0/1699440328 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ec8157860 msgr2=0x7f0ec815c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.205+0000 7f0ec6d9d700 1 --2- 172.21.15.46:0/1699440328 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ec8157860 0x7f0ec815c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.205+0000 7f0ec6d9d700 1 -- 172.21.15.46:0/1699440328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ec8157490 msgr2=0x7f0ec816a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.205+0000 7f0ec6d9d700 1 --2- 172.21.15.46:0/1699440328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ec8157490 0x7f0ec816a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.205+0000 7f0ec6d9d700 1 -- 172.21.15.46:0/1699440328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ebc009420 con 0x7f0ec806ddb0 2024-01-26T23:31:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.206+0000 7f0ec6d9d700 1 --2- 172.21.15.46:0/1699440328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec806ddb0 0x7f0ec80c9f70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800c590 tx=0x7f0eb800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.206+0000 7f0eaeffd700 1 -- 172.21.15.46:0/1699440328 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8011590 con 0x7f0ec806ddb0 2024-01-26T23:31:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.206+0000 7f0eaeffd700 1 -- 172.21.15.46:0/1699440328 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb8011710 con 0x7f0ec806ddb0 2024-01-26T23:31:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.206+0000 7f0ecd141700 1 -- 172.21.15.46:0/1699440328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec0009ae0 con 0x7f0ec806ddb0 2024-01-26T23:31:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.206+0000 7f0eaeffd700 1 -- 172.21.15.46:0/1699440328 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8011ad0 con 0x7f0ec806ddb0 2024-01-26T23:31:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.206+0000 7f0ecd141700 1 -- 172.21.15.46:0/1699440328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec80c9490 con 0x7f0ec806ddb0 2024-01-26T23:31:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.207+0000 7f0ecd141700 1 -- 172.21.15.46:0/1699440328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec804c710 con 0x7f0ec806ddb0 2024-01-26T23:31:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.207+0000 7f0eaeffd700 1 -- 172.21.15.46:0/1699440328 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f0eb801b3b0 con 0x7f0ec806ddb0 2024-01-26T23:31:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.208+0000 7f0eaeffd700 1 --2- 172.21.15.46:0/1699440328 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0eb4067130 0x7f0eb40695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.208+0000 7f0eaeffd700 1 -- 172.21.15.46:0/1699440328 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f0eb8083f20 con 0x7f0ec806ddb0 2024-01-26T23:31:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.208+0000 7f0ec659c700 1 --2- 172.21.15.46:0/1699440328 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0eb4067130 0x7f0eb40695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:14.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.208+0000 7f0ec659c700 1 --2- 172.21.15.46:0/1699440328 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0eb4067130 0x7f0eb40695e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0ec80adb90 tx=0x7f0ebc0091a0 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:14.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.212+0000 7f0eaeffd700 1 -- 172.21.15.46:0/1699440328 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0eb8050cd0 con 0x7f0ec806ddb0 2024-01-26T23:31:14.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.447+0000 7f0ecd141700 1 -- 172.21.15.46:0/1699440328 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ec80c9c20 con 0x7f0eb4067130 2024-01-26T23:31:14.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.491+0000 7f0eaeffd700 1 -- 172.21.15.46:0/1699440328 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f0ebc01a030 con 0x7f0eb4067130 2024-01-26T23:31:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule activated for path /snap_test_dir1/dir2 2024-01-26T23:31:14.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.494+0000 7f0ecd141700 1 -- 172.21.15.46:0/1699440328 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0eb4067130 msgr2=0x7f0eb40695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:14.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.494+0000 7f0ecd141700 1 --2- 172.21.15.46:0/1699440328 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0eb4067130 0x7f0eb40695e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0ec80adb90 tx=0x7f0ebc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.494+0000 7f0ecd141700 1 -- 172.21.15.46:0/1699440328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec806ddb0 msgr2=0x7f0ec80c9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:14.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.494+0000 7f0ecd141700 1 --2- 172.21.15.46:0/1699440328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec806ddb0 0x7f0ec80c9f70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800c590 tx=0x7f0eb800c8a0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.494+0000 7f0ecd141700 1 -- 172.21.15.46:0/1699440328 shutdown_connections 2024-01-26T23:31:14.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.494+0000 7f0ecd141700 1 --2- 172.21.15.46:0/1699440328 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0eb4067130 0x7f0eb40695e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.494+0000 7f0ecd141700 1 --2- 172.21.15.46:0/1699440328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec806ddb0 0x7f0ec80c9f70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.494+0000 7f0ecd141700 1 --2- 172.21.15.46:0/1699440328 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ec8157860 0x7f0ec815c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.494+0000 7f0ecd141700 1 --2- 172.21.15.46:0/1699440328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ec8157490 0x7f0ec816a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.494+0000 7f0ecd141700 1 -- 172.21.15.46:0/1699440328 >> 172.21.15.46:0/1699440328 conn(0x7f0ec8067930 msgr2=0x7f0ec8156730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:14.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.494+0000 7f0ecd141700 1 -- 172.21.15.46:0/1699440328 shutdown_connections 2024-01-26T23:31:14.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.495+0000 7f0ecd141700 1 -- 172.21.15.46:0/1699440328 wait complete. 2024-01-26T23:31:14.510 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2024-01-26T23:31:14.510 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir3 1m 2024-01-26T23:31:14.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.645+0000 7efd67c4c700 1 Processor -- start 2024-01-26T23:31:14.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.645+0000 7efd67c4c700 1 -- start start 2024-01-26T23:31:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.646+0000 7efd67c4c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efd60159510 0x7efd601598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.646+0000 7efd67c4c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd600ca690 0x7efd600cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.646+0000 7efd67c4c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd60069260 con 0x7efd600ca220 2024-01-26T23:31:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.646+0000 7efd67c4c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd6015d670 con 0x7efd60159510 2024-01-26T23:31:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.646+0000 7efd67c4c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd600c9750 con 0x7efd600ca690 2024-01-26T23:31:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.646+0000 7efd659e8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efd60159510 0x7efd601598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.646+0000 7efd659e8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efd60159510 0x7efd601598e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42850/0 (socket says 172.21.15.46:42850) 2024-01-26T23:31:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.646+0000 7efd659e8700 1 -- 172.21.15.46:0/748055970 learned_addr learned my addr 172.21.15.46:0/748055970 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.647+0000 7efd661e9700 1 --2- 172.21.15.46:0/748055970 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd600ca690 0x7efd600cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.647+0000 7efd659e8700 1 -- 172.21.15.46:0/748055970 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd600ca690 msgr2=0x7efd600cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.647+0000 7efd659e8700 1 --2- 172.21.15.46:0/748055970 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd600ca690 0x7efd600cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.647+0000 7efd659e8700 1 -- 172.21.15.46:0/748055970 >> v1:172.21.15.106:6789/0 conn(0x7efd600ca220 legacy=0x7efd600c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:14.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.647+0000 7efd659e8700 1 -- 172.21.15.46:0/748055970 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd600c9a30 con 0x7efd60159510 2024-01-26T23:31:14.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.647+0000 7efd659e8700 1 --2- 172.21.15.46:0/748055970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd60159510 0x7efd601598e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efd540099b0 tx=0x7efd54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c7da8f929ff444 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:14.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.647+0000 7efd649e6700 1 -- 172.21.15.46:0/748055970 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54009ea0 con 0x7efd60159510 2024-01-26T23:31:14.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.647+0000 7efd649e6700 1 -- 172.21.15.46:0/748055970 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5400b860 con 0x7efd60159510 2024-01-26T23:31:14.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.647+0000 7efd649e6700 1 -- 172.21.15.46:0/748055970 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5400bc00 con 0x7efd60159510 2024-01-26T23:31:14.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.648+0000 7efd67c4c700 1 -- 172.21.15.46:0/748055970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd60159510 msgr2=0x7efd601598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:14.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.648+0000 7efd67c4c700 1 --2- 172.21.15.46:0/748055970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd60159510 0x7efd601598e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efd540099b0 tx=0x7efd54009250 comp rx=0 tx=0).stop 2024-01-26T23:31:14.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.648+0000 7efd67c4c700 1 -- 172.21.15.46:0/748055970 shutdown_connections 2024-01-26T23:31:14.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.648+0000 7efd67c4c700 1 --2- 172.21.15.46:0/748055970 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd600ca690 0x7efd600cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.648+0000 7efd67c4c700 1 --2- 172.21.15.46:0/748055970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd60159510 0x7efd601598e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.648+0000 7efd67c4c700 1 -- 172.21.15.46:0/748055970 >> 172.21.15.46:0/748055970 conn(0x7efd6006bd90 msgr2=0x7efd6006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:14.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.648+0000 7efd67c4c700 1 -- 172.21.15.46:0/748055970 shutdown_connections 2024-01-26T23:31:14.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.648+0000 7efd67c4c700 1 -- 172.21.15.46:0/748055970 wait complete. 2024-01-26T23:31:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.649+0000 7efd67c4c700 1 Processor -- start 2024-01-26T23:31:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.649+0000 7efd67c4c700 1 -- start start 2024-01-26T23:31:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd67c4c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600ca220 0x7efd600c6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd67c4c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd60159510 0x7efd60151a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd67c4c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd600b3900 0x7efd600b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd67c4c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd600b3470 con 0x7efd600ca220 2024-01-26T23:31:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd67c4c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd600b2860 con 0x7efd60159510 2024-01-26T23:31:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd67c4c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd600b2b40 con 0x7efd600b3900 2024-01-26T23:31:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd659e8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600ca220 0x7efd600c6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd659e8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600ca220 0x7efd600c6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42860/0 (socket says 172.21.15.46:42860) 2024-01-26T23:31:14.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd659e8700 1 -- 172.21.15.46:0/2739052915 learned_addr learned my addr 172.21.15.46:0/2739052915 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:14.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd661e9700 1 --2- 172.21.15.46:0/2739052915 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd600b3900 0x7efd600b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:14.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd651e7700 1 --2- 172.21.15.46:0/2739052915 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd60159510 0x7efd60151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:14.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd659e8700 1 -- 172.21.15.46:0/2739052915 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd600b3900 msgr2=0x7efd600b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:14.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd659e8700 1 --2- 172.21.15.46:0/2739052915 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd600b3900 0x7efd600b3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd659e8700 1 -- 172.21.15.46:0/2739052915 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd60159510 msgr2=0x7efd60151a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:14.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd659e8700 1 --2- 172.21.15.46:0/2739052915 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd60159510 0x7efd60151a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd659e8700 1 -- 172.21.15.46:0/2739052915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd540094f0 con 0x7efd600ca220 2024-01-26T23:31:14.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd661e9700 1 --2- 172.21.15.46:0/2739052915 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd600b3900 0x7efd600b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:31:14.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.650+0000 7efd651e7700 1 --2- 172.21.15.46:0/2739052915 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd60159510 0x7efd60151a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:31:14.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.651+0000 7efd659e8700 1 --2- 172.21.15.46:0/2739052915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600ca220 0x7efd600c6190 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7efd6006f5a0 tx=0x7efd54017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:14.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.651+0000 7efd51ffb700 1 -- 172.21.15.46:0/2739052915 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5400bbe0 con 0x7efd600ca220 2024-01-26T23:31:14.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.651+0000 7efd51ffb700 1 -- 172.21.15.46:0/2739052915 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd54011420 con 0x7efd600ca220 2024-01-26T23:31:14.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.651+0000 7efd67c4c700 1 -- 172.21.15.46:0/2739052915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd600b2e20 con 0x7efd600ca220 2024-01-26T23:31:14.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.651+0000 7efd51ffb700 1 -- 172.21.15.46:0/2739052915 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd540117c0 con 0x7efd600ca220 2024-01-26T23:31:14.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.651+0000 7efd67c4c700 1 -- 172.21.15.46:0/2739052915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd600c66d0 con 0x7efd600ca220 2024-01-26T23:31:14.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.652+0000 7efd67c4c700 1 -- 172.21.15.46:0/2739052915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd600b3750 con 0x7efd600ca220 2024-01-26T23:31:14.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.653+0000 7efd51ffb700 1 -- 172.21.15.46:0/2739052915 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7efd54011bf0 con 0x7efd600ca220 2024-01-26T23:31:14.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.653+0000 7efd51ffb700 1 --2- 172.21.15.46:0/2739052915 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd44067130 0x7efd440695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:14.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.653+0000 7efd51ffb700 1 -- 172.21.15.46:0/2739052915 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7efd540864f0 con 0x7efd600ca220 2024-01-26T23:31:14.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.654+0000 7efd651e7700 1 --2- 172.21.15.46:0/2739052915 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd44067130 0x7efd440695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:14.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.654+0000 7efd651e7700 1 --2- 172.21.15.46:0/2739052915 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd44067130 0x7efd440695e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efd600c4df0 tx=0x7efd5c009040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:14.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.658+0000 7efd51ffb700 1 -- 172.21.15.46:0/2739052915 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efd54053300 con 0x7efd600ca220 2024-01-26T23:31:14.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.887+0000 7efd67c4c700 1 -- 172.21.15.46:0/2739052915 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7efd600c6860 con 0x7efd44067130 2024-01-26T23:31:14.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.933+0000 7efd51ffb700 1 -- 172.21.15.46:0/2739052915 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7efd600c6860 con 0x7efd44067130 2024-01-26T23:31:14.936 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule activated for path /snap_test_dir1/dir3 2024-01-26T23:31:14.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.936+0000 7efd67c4c700 1 -- 172.21.15.46:0/2739052915 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd44067130 msgr2=0x7efd440695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:14.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.936+0000 7efd67c4c700 1 --2- 172.21.15.46:0/2739052915 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd44067130 0x7efd440695e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efd600c4df0 tx=0x7efd5c009040 comp rx=0 tx=0).stop 2024-01-26T23:31:14.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.936+0000 7efd67c4c700 1 -- 172.21.15.46:0/2739052915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600ca220 msgr2=0x7efd600c6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:14.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.936+0000 7efd67c4c700 1 --2- 172.21.15.46:0/2739052915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600ca220 0x7efd600c6190 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7efd6006f5a0 tx=0x7efd54017000 comp rx=0 tx=0).stop 2024-01-26T23:31:14.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.936+0000 7efd67c4c700 1 -- 172.21.15.46:0/2739052915 shutdown_connections 2024-01-26T23:31:14.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.936+0000 7efd67c4c700 1 --2- 172.21.15.46:0/2739052915 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd44067130 0x7efd440695e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.936+0000 7efd67c4c700 1 --2- 172.21.15.46:0/2739052915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600ca220 0x7efd600c6190 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.936+0000 7efd67c4c700 1 --2- 172.21.15.46:0/2739052915 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd600b3900 0x7efd600b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.936+0000 7efd67c4c700 1 --2- 172.21.15.46:0/2739052915 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd60159510 0x7efd60151a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:14.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.936+0000 7efd67c4c700 1 -- 172.21.15.46:0/2739052915 >> 172.21.15.46:0/2739052915 conn(0x7efd6006bd90 msgr2=0x7efd60152c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:14.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.936+0000 7efd67c4c700 1 -- 172.21.15.46:0/2739052915 shutdown_connections 2024-01-26T23:31:14.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:14.936+0000 7efd67c4c700 1 -- 172.21.15.46:0/2739052915 wait complete. 2024-01-26T23:31:14.952 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2024-01-26T23:31:14.952 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir4 1m 2024-01-26T23:31:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.087+0000 7f7254f8a700 1 Processor -- start 2024-01-26T23:31:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.087+0000 7f7254f8a700 1 -- start start 2024-01-26T23:31:15.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.088+0000 7f7254f8a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7250157860 0x7f72500cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.088+0000 7f7254f8a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72500cea80 con 0x7f725006ddb0 2024-01-26T23:31:15.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.088+0000 7f7254f8a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72500cde70 con 0x7f7250157490 2024-01-26T23:31:15.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.088+0000 7f7254f8a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72500cdc60 con 0x7f7250157860 2024-01-26T23:31:15.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.089+0000 7f724ed9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f725006ddb0 0x7f72500b1ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55894/0 (socket says 172.21.15.46:55894) 2024-01-26T23:31:15.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.089+0000 7f724ed9d700 1 -- 172.21.15.46:0/2526358998 learned_addr learned my addr 172.21.15.46:0/2526358998 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:31:15.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.089+0000 7f724dd9b700 1 --2- 172.21.15.46:0/2526358998 >> v2:172.21.15.106:3300/0 conn(0x7f7250157860 0x7f72500cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.089+0000 7f724d59a700 1 -- 172.21.15.46:0/2526358998 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484231868 0 0) 0x7f72500cea80 con 0x7f725006ddb0 2024-01-26T23:31:15.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.089+0000 7f724d59a700 1 -- 172.21.15.46:0/2526358998 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f723c003570 con 0x7f725006ddb0 2024-01-26T23:31:15.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.089+0000 7f724dd9b700 1 -- 172.21.15.46:0/2526358998 >> v1:172.21.15.106:6789/0 conn(0x7f7250157490 legacy=0x7f725006d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:15.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.089+0000 7f724dd9b700 1 -- 172.21.15.46:0/2526358998 >> v1:172.21.15.46:6789/0 conn(0x7f725006ddb0 legacy=0x7f72500b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:15.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.089+0000 7f724dd9b700 1 -- 172.21.15.46:0/2526358998 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72500ce150 con 0x7f7250157860 2024-01-26T23:31:15.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.089+0000 7f724dd9b700 1 --2- 172.21.15.46:0/2526358998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7250157860 0x7f72500cc600 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f72400098e0 tx=0x7f72400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd97e5392837d0da server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:15.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.090+0000 7f724d59a700 1 -- 172.21.15.46:0/2526358998 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724000b640 con 0x7f7250157860 2024-01-26T23:31:15.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.090+0000 7f724d59a700 1 -- 172.21.15.46:0/2526358998 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724000b7a0 con 0x7f7250157860 2024-01-26T23:31:15.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.090+0000 7f724d59a700 1 -- 172.21.15.46:0/2526358998 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724000bb20 con 0x7f7250157860 2024-01-26T23:31:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.090+0000 7f7254f8a700 1 -- 172.21.15.46:0/2526358998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7250157860 msgr2=0x7f72500cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.090+0000 7f7254f8a700 1 --2- 172.21.15.46:0/2526358998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7250157860 0x7f72500cc600 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f72400098e0 tx=0x7f72400091a0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.090+0000 7f7254f8a700 1 -- 172.21.15.46:0/2526358998 shutdown_connections 2024-01-26T23:31:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.090+0000 7f7254f8a700 1 --2- 172.21.15.46:0/2526358998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7250157860 0x7f72500cc600 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.090+0000 7f7254f8a700 1 -- 172.21.15.46:0/2526358998 >> 172.21.15.46:0/2526358998 conn(0x7f7250067930 msgr2=0x7f7250067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.090+0000 7f7254f8a700 1 -- 172.21.15.46:0/2526358998 shutdown_connections 2024-01-26T23:31:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.090+0000 7f7254f8a700 1 -- 172.21.15.46:0/2526358998 wait complete. 2024-01-26T23:31:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.091+0000 7f7254f8a700 1 Processor -- start 2024-01-26T23:31:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.092+0000 7f7254f8a700 1 -- start start 2024-01-26T23:31:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.092+0000 7f7254f8a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f725006ddb0 0x7f725015a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.092+0000 7f7254f8a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7250157490 0x7f72501592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.092+0000 7f7254f8a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7250157860 0x7f7250149cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.092+0000 7f7254f8a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72500c84a0 con 0x7f7250157490 2024-01-26T23:31:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.092+0000 7f7254f8a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725014a700 con 0x7f7250157860 2024-01-26T23:31:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.092+0000 7f7254f8a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725014a210 con 0x7f725006ddb0 2024-01-26T23:31:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.092+0000 7f724dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7250157490 0x7f72501592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.092+0000 7f724dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7250157490 0x7f72501592b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42876/0 (socket says 172.21.15.46:42876) 2024-01-26T23:31:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.092+0000 7f724dd9b700 1 -- 172.21.15.46:0/1325200618 learned_addr learned my addr 172.21.15.46:0/1325200618 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:15.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.092+0000 7f724e59c700 1 --2- 172.21.15.46:0/1325200618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f725006ddb0 0x7f725015a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.092+0000 7f724ed9d700 1 --2- 172.21.15.46:0/1325200618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7250157860 0x7f7250149cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.093+0000 7f724dd9b700 1 -- 172.21.15.46:0/1325200618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f725006ddb0 msgr2=0x7f725015a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:15.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.093+0000 7f724dd9b700 1 --2- 172.21.15.46:0/1325200618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f725006ddb0 0x7f725015a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.093+0000 7f724dd9b700 1 -- 172.21.15.46:0/1325200618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7250157860 msgr2=0x7f7250149cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:15.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.093+0000 7f724dd9b700 1 --2- 172.21.15.46:0/1325200618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7250157860 0x7f7250149cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.093+0000 7f724dd9b700 1 -- 172.21.15.46:0/1325200618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7240009420 con 0x7f7250157490 2024-01-26T23:31:15.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.093+0000 7f724dd9b700 1 --2- 172.21.15.46:0/1325200618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7250157490 0x7f72501592b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f72400097b0 tx=0x7f7240017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:15.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.093+0000 7f72467fc700 1 -- 172.21.15.46:0/1325200618 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724000b910 con 0x7f7250157490 2024-01-26T23:31:15.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.093+0000 7f72467fc700 1 -- 172.21.15.46:0/1325200618 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724000ba70 con 0x7f7250157490 2024-01-26T23:31:15.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.093+0000 7f72467fc700 1 -- 172.21.15.46:0/1325200618 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240011670 con 0x7f7250157490 2024-01-26T23:31:15.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.093+0000 7f7254f8a700 1 -- 172.21.15.46:0/1325200618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72500c7890 con 0x7f7250157490 2024-01-26T23:31:15.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.093+0000 7f7254f8a700 1 -- 172.21.15.46:0/1325200618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72500c7e50 con 0x7f7250157490 2024-01-26T23:31:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.094+0000 7f7254f8a700 1 -- 172.21.15.46:0/1325200618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72500c8650 con 0x7f7250157490 2024-01-26T23:31:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.095+0000 7f72467fc700 1 -- 172.21.15.46:0/1325200618 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f7240011a60 con 0x7f7250157490 2024-01-26T23:31:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.095+0000 7f72467fc700 1 --2- 172.21.15.46:0/1325200618 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7230067130 0x7f72300695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.095+0000 7f72467fc700 1 -- 172.21.15.46:0/1325200618 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f72400861b0 con 0x7f7250157490 2024-01-26T23:31:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.095+0000 7f724e59c700 1 --2- 172.21.15.46:0/1325200618 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7230067130 0x7f72300695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.096+0000 7f724e59c700 1 --2- 172.21.15.46:0/1325200618 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7230067130 0x7f72300695e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7238003aa0 tx=0x7f7238000bc0 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:15.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.100+0000 7f72467fc700 1 -- 172.21.15.46:0/1325200618 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f72500c8650 con 0x7f7250157490 2024-01-26T23:31:15.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.334+0000 7f7254f8a700 1 -- 172.21.15.46:0/1325200618 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f72500c8650 con 0x7f7230067130 2024-01-26T23:31:15.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.380+0000 7f72467fc700 1 -- 172.21.15.46:0/1325200618 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f72500c8650 con 0x7f7230067130 2024-01-26T23:31:15.382 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule activated for path /snap_test_dir1/dir4 2024-01-26T23:31:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.383+0000 7f7254f8a700 1 -- 172.21.15.46:0/1325200618 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7230067130 msgr2=0x7f72300695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.383+0000 7f7254f8a700 1 --2- 172.21.15.46:0/1325200618 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7230067130 0x7f72300695e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7238003aa0 tx=0x7f7238000bc0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.383+0000 7f7254f8a700 1 -- 172.21.15.46:0/1325200618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7250157490 msgr2=0x7f72501592b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.383+0000 7f7254f8a700 1 --2- 172.21.15.46:0/1325200618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7250157490 0x7f72501592b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f72400097b0 tx=0x7f7240017000 comp rx=0 tx=0).stop 2024-01-26T23:31:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.383+0000 7f7254f8a700 1 -- 172.21.15.46:0/1325200618 shutdown_connections 2024-01-26T23:31:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.383+0000 7f7254f8a700 1 --2- 172.21.15.46:0/1325200618 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7230067130 0x7f72300695e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.383+0000 7f7254f8a700 1 --2- 172.21.15.46:0/1325200618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7250157860 0x7f7250149cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.383+0000 7f7254f8a700 1 --2- 172.21.15.46:0/1325200618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f725006ddb0 0x7f725015a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.383+0000 7f7254f8a700 1 --2- 172.21.15.46:0/1325200618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7250157490 0x7f72501592b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.383+0000 7f7254f8a700 1 -- 172.21.15.46:0/1325200618 >> 172.21.15.46:0/1325200618 conn(0x7f7250067930 msgr2=0x7f72500b3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:15.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.383+0000 7f7254f8a700 1 -- 172.21.15.46:0/1325200618 shutdown_connections 2024-01-26T23:31:15.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.383+0000 7f7254f8a700 1 -- 172.21.15.46:0/1325200618 wait complete. 2024-01-26T23:31:15.399 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2024-01-26T23:31:15.399 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir5 1m 2024-01-26T23:31:15.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.535+0000 7f348a22c700 1 Processor -- start 2024-01-26T23:31:15.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.536+0000 7f348a22c700 1 -- start start 2024-01-26T23:31:15.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.536+0000 7f348a22c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f348406ebe0 0x7f3484067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.536+0000 7f348a22c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f34841597e0 0x7f3484159c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.536+0000 7f348a22c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348406fdb0 con 0x7f3484067af0 2024-01-26T23:31:15.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.536+0000 7f348a22c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f348406f1a0 con 0x7f34841597e0 2024-01-26T23:31:15.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.536+0000 7f348a22c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348406f480 con 0x7f348406ebe0 2024-01-26T23:31:15.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34837fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3484067af0 0x7f34840ad8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55906/0 (socket says 172.21.15.46:55906) 2024-01-26T23:31:15.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34837fe700 1 -- 172.21.15.46:0/2117323482 learned_addr learned my addr 172.21.15.46:0/2117323482 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:31:15.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f3483fff700 1 --2- 172.21.15.46:0/2117323482 >> v2:172.21.15.106:3300/0 conn(0x7f34841597e0 0x7f3484159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f3482ffd700 1 --2- 172.21.15.46:0/2117323482 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f348406ebe0 0x7f3484067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34827fc700 1 -- 172.21.15.46:0/2117323482 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19014364 0 0) 0x7f348406fdb0 con 0x7f3484067af0 2024-01-26T23:31:15.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34827fc700 1 -- 172.21.15.46:0/2117323482 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3470003570 con 0x7f3484067af0 2024-01-26T23:31:15.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34827fc700 1 -- 172.21.15.46:0/2117323482 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1191488066 0 0) 0x7f3470003570 con 0x7f3484067af0 2024-01-26T23:31:15.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34827fc700 1 -- 172.21.15.46:0/2117323482 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f348406ebe0 msgr2=0x7f3484067ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:15.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34827fc700 1 --2- 172.21.15.46:0/2117323482 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f348406ebe0 0x7f3484067ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34827fc700 1 -- 172.21.15.46:0/2117323482 >> v2:172.21.15.106:3300/0 conn(0x7f34841597e0 msgr2=0x7f3484159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:15.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34827fc700 1 --2- 172.21.15.46:0/2117323482 >> v2:172.21.15.106:3300/0 conn(0x7f34841597e0 0x7f3484159c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34827fc700 1 -- 172.21.15.46:0/2117323482 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348406f760 con 0x7f3484067af0 2024-01-26T23:31:15.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34827fc700 1 -- 172.21.15.46:0/2117323482 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f346c001470 con 0x7f3484067af0 2024-01-26T23:31:15.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f3483fff700 1 --2- 172.21.15.46:0/2117323482 >> v2:172.21.15.106:3300/0 conn(0x7f34841597e0 0x7f3484159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:31:15.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34827fc700 1 -- 172.21.15.46:0/2117323482 >> v1:172.21.15.46:6789/0 conn(0x7f3484067af0 legacy=0x7f34840ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:15.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34827fc700 1 --2- 172.21.15.46:0/2117323482 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3470003ae0 0x7f3470003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34827fc700 1 --2- 172.21.15.46:0/2117323482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3470004810 0x7f3470004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34827fc700 1 --2- 172.21.15.46:0/2117323482 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f348406ebe0 0x7f34700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34827fc700 1 -- 172.21.15.46:0/2117323482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348406f1a0 con 0x7f3470004810 2024-01-26T23:31:15.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34827fc700 1 -- 172.21.15.46:0/2117323482 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348406f480 con 0x7f3470003ae0 2024-01-26T23:31:15.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f34827fc700 1 -- 172.21.15.46:0/2117323482 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347000bbf0 con 0x7f348406ebe0 2024-01-26T23:31:15.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.537+0000 7f3482ffd700 1 --2- 172.21.15.46:0/2117323482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3470004810 0x7f3470004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.538+0000 7f34837fe700 1 --2- 172.21.15.46:0/2117323482 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3470003ae0 0x7f3470003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.538+0000 7f3483fff700 1 --2- 172.21.15.46:0/2117323482 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f348406ebe0 0x7f34700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.538+0000 7f3482ffd700 1 -- 172.21.15.46:0/2117323482 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f348406ebe0 msgr2=0x7f34700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:15.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.538+0000 7f3482ffd700 1 --2- 172.21.15.46:0/2117323482 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f348406ebe0 0x7f34700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.538+0000 7f3482ffd700 1 -- 172.21.15.46:0/2117323482 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3470003ae0 msgr2=0x7f3470003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:15.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.538+0000 7f3482ffd700 1 --2- 172.21.15.46:0/2117323482 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3470003ae0 0x7f3470003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.538+0000 7f3482ffd700 1 -- 172.21.15.46:0/2117323482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3470003570 con 0x7f3470004810 2024-01-26T23:31:15.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.538+0000 7f34837fe700 1 --2- 172.21.15.46:0/2117323482 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3470003ae0 0x7f3470003ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:31:15.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.538+0000 7f3482ffd700 1 --2- 172.21.15.46:0/2117323482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3470004810 0x7f3470004c80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3474009c00 tx=0x7f34740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29139276639ba730 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:15.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.538+0000 7f34827fc700 1 -- 172.21.15.46:0/2117323482 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347400e710 con 0x7f3470004810 2024-01-26T23:31:15.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.538+0000 7f34827fc700 1 -- 172.21.15.46:0/2117323482 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347400e870 con 0x7f3470004810 2024-01-26T23:31:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.538+0000 7f34827fc700 1 -- 172.21.15.46:0/2117323482 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347400ec30 con 0x7f3470004810 2024-01-26T23:31:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.539+0000 7f348a22c700 1 -- 172.21.15.46:0/2117323482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3470004810 msgr2=0x7f3470004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.539+0000 7f348a22c700 1 --2- 172.21.15.46:0/2117323482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3470004810 0x7f3470004c80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3474009c00 tx=0x7f34740091a0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.539+0000 7f3483fff700 1 -- 172.21.15.46:0/2117323482 reap_dead start 2024-01-26T23:31:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.539+0000 7f348a22c700 1 -- 172.21.15.46:0/2117323482 shutdown_connections 2024-01-26T23:31:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.539+0000 7f348a22c700 1 -- 172.21.15.46:0/2117323482 >> 172.21.15.46:0/2117323482 conn(0x7f34840cdf80 msgr2=0x7f348406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.539+0000 7f348a22c700 1 -- 172.21.15.46:0/2117323482 shutdown_connections 2024-01-26T23:31:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.539+0000 7f348a22c700 1 -- 172.21.15.46:0/2117323482 wait complete. 2024-01-26T23:31:15.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.540+0000 7f348a22c700 1 Processor -- start 2024-01-26T23:31:15.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.540+0000 7f348a22c700 1 -- start start 2024-01-26T23:31:15.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.541+0000 7f348a22c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3470004810 0x7f348414d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.541+0000 7f348a22c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f348414fba0 0x7f3484150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.541+0000 7f348a22c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f348414eef0 0x7f348414f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.541+0000 7f348a22c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34840b1190 con 0x7f348414fba0 2024-01-26T23:31:15.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.541+0000 7f348a22c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34840b0580 con 0x7f3470004810 2024-01-26T23:31:15.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.541+0000 7f348a22c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34840b0860 con 0x7f348414eef0 2024-01-26T23:31:15.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.541+0000 7f3482ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f348414fba0 0x7f3484150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.541+0000 7f3482ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f348414fba0 0x7f3484150010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42900/0 (socket says 172.21.15.46:42900) 2024-01-26T23:31:15.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.541+0000 7f3482ffd700 1 -- 172.21.15.46:0/2784231863 learned_addr learned my addr 172.21.15.46:0/2784231863 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:15.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.541+0000 7f34837fe700 1 --2- 172.21.15.46:0/2784231863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3470004810 0x7f348414d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.541+0000 7f3483fff700 1 --2- 172.21.15.46:0/2784231863 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f348414eef0 0x7f348414f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.541+0000 7f3482ffd700 1 -- 172.21.15.46:0/2784231863 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f348414eef0 msgr2=0x7f348414f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:15.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.541+0000 7f3482ffd700 1 --2- 172.21.15.46:0/2784231863 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f348414eef0 0x7f348414f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.541+0000 7f3482ffd700 1 -- 172.21.15.46:0/2784231863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3470004810 msgr2=0x7f348414d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:15.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.541+0000 7f3482ffd700 1 --2- 172.21.15.46:0/2784231863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3470004810 0x7f348414d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.541+0000 7f3482ffd700 1 -- 172.21.15.46:0/2784231863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34840b0b40 con 0x7f348414fba0 2024-01-26T23:31:15.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.541+0000 7f34837fe700 1 --2- 172.21.15.46:0/2784231863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3470004810 0x7f348414d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:31:15.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.542+0000 7f3482ffd700 1 --2- 172.21.15.46:0/2784231863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f348414fba0 0x7f3484150010 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f347000bb40 tx=0x7f347401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:15.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.542+0000 7f347b7fe700 1 -- 172.21.15.46:0/2784231863 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474022070 con 0x7f348414fba0 2024-01-26T23:31:15.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.542+0000 7f347b7fe700 1 -- 172.21.15.46:0/2784231863 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347400e9e0 con 0x7f348414fba0 2024-01-26T23:31:15.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.542+0000 7f348a22c700 1 -- 172.21.15.46:0/2784231863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34840b1190 con 0x7f348414fba0 2024-01-26T23:31:15.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.542+0000 7f347b7fe700 1 -- 172.21.15.46:0/2784231863 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347401c450 con 0x7f348414fba0 2024-01-26T23:31:15.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.542+0000 7f348a22c700 1 -- 172.21.15.46:0/2784231863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34840b0250 con 0x7f348414fba0 2024-01-26T23:31:15.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.543+0000 7f348a22c700 1 -- 172.21.15.46:0/2784231863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348404c710 con 0x7f348414fba0 2024-01-26T23:31:15.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.543+0000 7f347b7fe700 1 -- 172.21.15.46:0/2784231863 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f347401c880 con 0x7f348414fba0 2024-01-26T23:31:15.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.544+0000 7f347b7fe700 1 --2- 172.21.15.46:0/2784231863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3464067130 0x7f34640695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.544+0000 7f347b7fe700 1 -- 172.21.15.46:0/2784231863 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f3474086090 con 0x7f348414fba0 2024-01-26T23:31:15.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.544+0000 7f34837fe700 1 --2- 172.21.15.46:0/2784231863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3464067130 0x7f34640695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.544+0000 7f34837fe700 1 --2- 172.21.15.46:0/2784231863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3464067130 0x7f34640695e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f346c000e10 tx=0x7f346c009040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:15.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.548+0000 7f347b7fe700 1 -- 172.21.15.46:0/2784231863 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3474052ef0 con 0x7f348414fba0 2024-01-26T23:31:15.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.781+0000 7f348a22c700 1 -- 172.21.15.46:0/2784231863 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3484068690 con 0x7f3464067130 2024-01-26T23:31:15.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.825+0000 7f347b7fe700 1 -- 172.21.15.46:0/2784231863 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f3484068690 con 0x7f3464067130 2024-01-26T23:31:15.827 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule activated for path /snap_test_dir1/dir5 2024-01-26T23:31:15.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.827+0000 7f348a22c700 1 -- 172.21.15.46:0/2784231863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3464067130 msgr2=0x7f34640695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:15.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.827+0000 7f348a22c700 1 --2- 172.21.15.46:0/2784231863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3464067130 0x7f34640695e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f346c000e10 tx=0x7f346c009040 comp rx=0 tx=0).stop 2024-01-26T23:31:15.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.827+0000 7f348a22c700 1 -- 172.21.15.46:0/2784231863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f348414fba0 msgr2=0x7f3484150010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:15.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.827+0000 7f348a22c700 1 --2- 172.21.15.46:0/2784231863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f348414fba0 0x7f3484150010 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f347000bb40 tx=0x7f347401a000 comp rx=0 tx=0).stop 2024-01-26T23:31:15.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.827+0000 7f348a22c700 1 -- 172.21.15.46:0/2784231863 shutdown_connections 2024-01-26T23:31:15.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.827+0000 7f348a22c700 1 --2- 172.21.15.46:0/2784231863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3464067130 0x7f34640695e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.827+0000 7f348a22c700 1 --2- 172.21.15.46:0/2784231863 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f348414eef0 0x7f348414f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.827+0000 7f348a22c700 1 --2- 172.21.15.46:0/2784231863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3470004810 0x7f348414d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.827+0000 7f348a22c700 1 --2- 172.21.15.46:0/2784231863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f348414fba0 0x7f3484150010 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.827+0000 7f348a22c700 1 -- 172.21.15.46:0/2784231863 >> 172.21.15.46:0/2784231863 conn(0x7f34840cdf80 msgr2=0x7f34840ad1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:15.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.827+0000 7f348a22c700 1 -- 172.21.15.46:0/2784231863 shutdown_connections 2024-01-26T23:31:15.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.827+0000 7f348a22c700 1 -- 172.21.15.46:0/2784231863 wait complete. 2024-01-26T23:31:15.843 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2024-01-26T23:31:15.844 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir6 1m 2024-01-26T23:31:15.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.978+0000 7efff398c700 1 Processor -- start 2024-01-26T23:31:15.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.978+0000 7efff398c700 1 -- start start 2024-01-26T23:31:15.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.979+0000 7efff398c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7effec15ccb0 0x7effec15d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.979+0000 7efff398c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec06adf0 0x7effec068a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.979+0000 7efff398c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effec0a9840 con 0x7effec0caa30 2024-01-26T23:31:15.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.979+0000 7efff398c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effec068f60 con 0x7effec15ccb0 2024-01-26T23:31:15.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.979+0000 7efff398c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effec06b240 con 0x7effec06adf0 2024-01-26T23:31:15.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.979+0000 7efff1728700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7effec15ccb0 0x7effec15d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.979+0000 7efff1f29700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7effec0caa30 0x7effec067930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55910/0 (socket says 172.21.15.46:55910) 2024-01-26T23:31:15.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.979+0000 7efff1f29700 1 -- 172.21.15.46:0/2676980445 learned_addr learned my addr 172.21.15.46:0/2676980445 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:31:15.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.979+0000 7efff0f27700 1 --2- 172.21.15.46:0/2676980445 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec06adf0 0x7effec068a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.980+0000 7efff1728700 1 -- 172.21.15.46:0/2676980445 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec06adf0 msgr2=0x7effec068a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.980+0000 7efff1728700 1 --2- 172.21.15.46:0/2676980445 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec06adf0 0x7effec068a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.980+0000 7efff1728700 1 -- 172.21.15.46:0/2676980445 >> v1:172.21.15.46:6789/0 conn(0x7effec0caa30 legacy=0x7effec067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.980+0000 7efff1728700 1 -- 172.21.15.46:0/2676980445 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effec06b4e0 con 0x7effec15ccb0 2024-01-26T23:31:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.980+0000 7effe3fff700 1 -- 172.21.15.46:0/2676980445 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617413761 0 0) 0x7effec0a9840 con 0x7effec0caa30 2024-01-26T23:31:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.980+0000 7efff1728700 1 --2- 172.21.15.46:0/2676980445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec15ccb0 0x7effec15d080 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7effe80098e0 tx=0x7effe80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=231fba08e35ec748 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.980+0000 7effe3fff700 1 -- 172.21.15.46:0/2676980445 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe800b640 con 0x7effec15ccb0 2024-01-26T23:31:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.980+0000 7effe3fff700 1 -- 172.21.15.46:0/2676980445 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe800b7a0 con 0x7effec15ccb0 2024-01-26T23:31:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.980+0000 7effe3fff700 1 -- 172.21.15.46:0/2676980445 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe800bb40 con 0x7effec15ccb0 2024-01-26T23:31:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.980+0000 7efff398c700 1 -- 172.21.15.46:0/2676980445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec15ccb0 msgr2=0x7effec15d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.980+0000 7efff398c700 1 --2- 172.21.15.46:0/2676980445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec15ccb0 0x7effec15d080 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7effe80098e0 tx=0x7effe80091a0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.981+0000 7efff398c700 1 -- 172.21.15.46:0/2676980445 shutdown_connections 2024-01-26T23:31:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.981+0000 7efff398c700 1 --2- 172.21.15.46:0/2676980445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec15ccb0 0x7effec15d080 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.981+0000 7efff398c700 1 --2- 172.21.15.46:0/2676980445 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec06adf0 0x7effec068a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.981+0000 7efff398c700 1 -- 172.21.15.46:0/2676980445 >> 172.21.15.46:0/2676980445 conn(0x7effec0cd720 msgr2=0x7effec0c96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:15.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.981+0000 7efff398c700 1 -- 172.21.15.46:0/2676980445 shutdown_connections 2024-01-26T23:31:15.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.981+0000 7efff398c700 1 -- 172.21.15.46:0/2676980445 wait complete. 2024-01-26T23:31:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.982+0000 7efff398c700 1 Processor -- start 2024-01-26T23:31:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.982+0000 7efff398c700 1 -- start start 2024-01-26T23:31:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.983+0000 7efff398c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effec06adf0 0x7effec15fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.983+0000 7efff398c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec0caa30 0x7effec16a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.983+0000 7efff398c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec15ccb0 0x7effec145e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.983+0000 7efff398c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effec15ddb0 con 0x7effec0caa30 2024-01-26T23:31:15.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.983+0000 7efff398c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effec0b0650 con 0x7effec06adf0 2024-01-26T23:31:15.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.983+0000 7efff398c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effec16a270 con 0x7effec15ccb0 2024-01-26T23:31:15.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.983+0000 7efff0f27700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec0caa30 0x7effec16a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.983+0000 7efff0f27700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec0caa30 0x7effec16a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42924/0 (socket says 172.21.15.46:42924) 2024-01-26T23:31:15.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.983+0000 7efff0f27700 1 -- 172.21.15.46:0/3503849651 learned_addr learned my addr 172.21.15.46:0/3503849651 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:15.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.983+0000 7efff1f29700 1 --2- 172.21.15.46:0/3503849651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec15ccb0 0x7effec145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.983+0000 7efff0f27700 1 -- 172.21.15.46:0/3503849651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec15ccb0 msgr2=0x7effec145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:15.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.983+0000 7efff1728700 1 --2- 172.21.15.46:0/3503849651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effec06adf0 0x7effec15fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.983+0000 7efff0f27700 1 --2- 172.21.15.46:0/3503849651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec15ccb0 0x7effec145e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.983+0000 7efff0f27700 1 -- 172.21.15.46:0/3503849651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effec06adf0 msgr2=0x7effec15fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:15.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.983+0000 7efff0f27700 1 --2- 172.21.15.46:0/3503849651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effec06adf0 0x7effec15fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:15.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.983+0000 7efff0f27700 1 -- 172.21.15.46:0/3503849651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe8009420 con 0x7effec0caa30 2024-01-26T23:31:15.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.984+0000 7efff0f27700 1 --2- 172.21.15.46:0/3503849651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec0caa30 0x7effec16a6b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7effdc00e930 tx=0x7effdc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:15.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.984+0000 7effe17fa700 1 -- 172.21.15.46:0/3503849651 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effdc00eed0 con 0x7effec0caa30 2024-01-26T23:31:15.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.984+0000 7effe17fa700 1 -- 172.21.15.46:0/3503849651 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effdc00c850 con 0x7effec0caa30 2024-01-26T23:31:15.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.984+0000 7efff398c700 1 -- 172.21.15.46:0/3503849651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effec15dad0 con 0x7effec0caa30 2024-01-26T23:31:15.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.984+0000 7effe17fa700 1 -- 172.21.15.46:0/3503849651 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effdc00cbf0 con 0x7effec0caa30 2024-01-26T23:31:15.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.984+0000 7efff398c700 1 -- 172.21.15.46:0/3503849651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effec0b02e0 con 0x7effec0caa30 2024-01-26T23:31:15.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.985+0000 7efff398c700 1 -- 172.21.15.46:0/3503849651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effec0afe60 con 0x7effec0caa30 2024-01-26T23:31:15.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.985+0000 7effe17fa700 1 -- 172.21.15.46:0/3503849651 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7effdc013470 con 0x7effec0caa30 2024-01-26T23:31:15.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.986+0000 7effe17fa700 1 --2- 172.21.15.46:0/3503849651 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7effd0067130 0x7effd00695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:15.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.986+0000 7effe17fa700 1 -- 172.21.15.46:0/3503849651 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7effdc084d20 con 0x7effec0caa30 2024-01-26T23:31:15.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.986+0000 7efff1728700 1 --2- 172.21.15.46:0/3503849651 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7effd0067130 0x7effd00695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:15.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.986+0000 7efff1728700 1 --2- 172.21.15.46:0/3503849651 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7effd0067130 0x7effd00695e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7effe8003080 tx=0x7effe8017000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:15.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:15.990+0000 7effe17fa700 1 -- 172.21.15.46:0/3503849651 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7effec0afe60 con 0x7effec0caa30 2024-01-26T23:31:16.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.229+0000 7efff398c700 1 -- 172.21.15.46:0/3503849651 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7effec0afe60 con 0x7effd0067130 2024-01-26T23:31:16.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.279+0000 7effe17fa700 1 -- 172.21.15.46:0/3503849651 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7effec0afe60 con 0x7effd0067130 2024-01-26T23:31:16.281 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule activated for path /snap_test_dir1/dir6 2024-01-26T23:31:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.282+0000 7efff398c700 1 -- 172.21.15.46:0/3503849651 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7effd0067130 msgr2=0x7effd00695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.282+0000 7efff398c700 1 --2- 172.21.15.46:0/3503849651 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7effd0067130 0x7effd00695e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7effe8003080 tx=0x7effe8017000 comp rx=0 tx=0).stop 2024-01-26T23:31:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.282+0000 7efff398c700 1 -- 172.21.15.46:0/3503849651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec0caa30 msgr2=0x7effec16a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.282+0000 7efff398c700 1 --2- 172.21.15.46:0/3503849651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec0caa30 0x7effec16a6b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7effdc00e930 tx=0x7effdc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.282+0000 7efff398c700 1 -- 172.21.15.46:0/3503849651 shutdown_connections 2024-01-26T23:31:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.282+0000 7efff398c700 1 --2- 172.21.15.46:0/3503849651 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7effd0067130 0x7effd00695e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.282+0000 7efff398c700 1 --2- 172.21.15.46:0/3503849651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec15ccb0 0x7effec145e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.282+0000 7efff398c700 1 --2- 172.21.15.46:0/3503849651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effec06adf0 0x7effec15fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.282+0000 7efff398c700 1 --2- 172.21.15.46:0/3503849651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec0caa30 0x7effec16a6b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.282+0000 7efff398c700 1 -- 172.21.15.46:0/3503849651 >> 172.21.15.46:0/3503849651 conn(0x7effec0cd720 msgr2=0x7effec0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.282+0000 7efff398c700 1 -- 172.21.15.46:0/3503849651 shutdown_connections 2024-01-26T23:31:16.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.282+0000 7efff398c700 1 -- 172.21.15.46:0/3503849651 wait complete. 2024-01-26T23:31:16.298 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2024-01-26T23:31:16.298 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir7 1m 2024-01-26T23:31:16.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.433+0000 7f6d5362f700 1 Processor -- start 2024-01-26T23:31:16.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.433+0000 7f6d5362f700 1 -- start start 2024-01-26T23:31:16.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.434+0000 7f6d5362f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6d4c068040 0x7f6d4c06c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:16.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.434+0000 7f6d5362f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6d4c1615b0 0x7f6d4c0c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:16.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.434+0000 7f6d5362f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4c06c810 con 0x7f6d4c1611e0 2024-01-26T23:31:16.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.434+0000 7f6d5362f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d4c0ca950 con 0x7f6d4c068040 2024-01-26T23:31:16.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.434+0000 7f6d5362f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d4c06caf0 con 0x7f6d4c1615b0 2024-01-26T23:31:16.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.435+0000 7f6d50bca700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6d4c068040 0x7f6d4c06c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:16.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.435+0000 7f6d513cb700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6d4c1611e0 0x7f6d4c067930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55914/0 (socket says 172.21.15.46:55914) 2024-01-26T23:31:16.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.435+0000 7f6d513cb700 1 -- 172.21.15.46:0/2718783285 learned_addr learned my addr 172.21.15.46:0/2718783285 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:31:16.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.435+0000 7f6d51bcc700 1 --2- 172.21.15.46:0/2718783285 >> v2:172.21.15.106:3300/0 conn(0x7f6d4c1615b0 0x7f6d4c0c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:16.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.435+0000 7f6d50bca700 1 -- 172.21.15.46:0/2718783285 >> v2:172.21.15.106:3300/0 conn(0x7f6d4c1615b0 msgr2=0x7f6d4c0c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:16.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.435+0000 7f6d50bca700 1 --2- 172.21.15.46:0/2718783285 >> v2:172.21.15.106:3300/0 conn(0x7f6d4c1615b0 0x7f6d4c0c7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.435+0000 7f6d50bca700 1 -- 172.21.15.46:0/2718783285 >> v1:172.21.15.46:6789/0 conn(0x7f6d4c1611e0 legacy=0x7f6d4c067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:16.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.435+0000 7f6d50bca700 1 -- 172.21.15.46:0/2718783285 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c06cdd0 con 0x7f6d4c068040 2024-01-26T23:31:16.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.435+0000 7f6d3ffff700 1 -- 172.21.15.46:0/2718783285 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256012046 0 0) 0x7f6d4c06c810 con 0x7f6d4c1611e0 2024-01-26T23:31:16.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.435+0000 7f6d51bcc700 1 --2- 172.21.15.46:0/2718783285 >> v2:172.21.15.106:3300/0 conn(0x7f6d4c1615b0 0x7f6d4c0c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:31:16.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.435+0000 7f6d50bca700 1 --2- 172.21.15.46:0/2718783285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c068040 0x7f6d4c06c110 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0c9e50 tx=0x7f6d480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e891c25f509c8d27 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:16.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.435+0000 7f6d3ffff700 1 -- 172.21.15.46:0/2718783285 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4800b6b0 con 0x7f6d4c068040 2024-01-26T23:31:16.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.435+0000 7f6d3ffff700 1 -- 172.21.15.46:0/2718783285 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4800b810 con 0x7f6d4c068040 2024-01-26T23:31:16.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.435+0000 7f6d3ffff700 1 -- 172.21.15.46:0/2718783285 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4800bbb0 con 0x7f6d4c068040 2024-01-26T23:31:16.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.436+0000 7f6d5362f700 1 -- 172.21.15.46:0/2718783285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c068040 msgr2=0x7f6d4c06c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:16.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.436+0000 7f6d5362f700 1 --2- 172.21.15.46:0/2718783285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c068040 0x7f6d4c06c110 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0c9e50 tx=0x7f6d480091a0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.436+0000 7f6d5362f700 1 -- 172.21.15.46:0/2718783285 shutdown_connections 2024-01-26T23:31:16.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.436+0000 7f6d5362f700 1 --2- 172.21.15.46:0/2718783285 >> v2:172.21.15.106:3300/0 conn(0x7f6d4c1615b0 0x7f6d4c0c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.436+0000 7f6d5362f700 1 --2- 172.21.15.46:0/2718783285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c068040 0x7f6d4c06c110 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.436+0000 7f6d5362f700 1 -- 172.21.15.46:0/2718783285 >> 172.21.15.46:0/2718783285 conn(0x7f6d4c0cdf50 msgr2=0x7f6d4c0c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:16.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.436+0000 7f6d5362f700 1 -- 172.21.15.46:0/2718783285 shutdown_connections 2024-01-26T23:31:16.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.436+0000 7f6d5362f700 1 -- 172.21.15.46:0/2718783285 wait complete. 2024-01-26T23:31:16.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.437+0000 7f6d5362f700 1 Processor -- start 2024-01-26T23:31:16.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.437+0000 7f6d5362f700 1 -- start start 2024-01-26T23:31:16.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.438+0000 7f6d5362f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d4c068040 0x7f6d4c0c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:16.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.438+0000 7f6d5362f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d4c1611e0 0x7f6d4c0c33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:16.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.438+0000 7f6d5362f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c1615b0 0x7f6d4c0c38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:16.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.438+0000 7f6d5362f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c0c6e50 con 0x7f6d4c1615b0 2024-01-26T23:31:16.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.438+0000 7f6d5362f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c0c5fa0 con 0x7f6d4c1611e0 2024-01-26T23:31:16.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.438+0000 7f6d5362f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c0c72e0 con 0x7f6d4c068040 2024-01-26T23:31:16.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.439+0000 7f6d51bcc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c1615b0 0x7f6d4c0c38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:16.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.439+0000 7f6d51bcc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c1615b0 0x7f6d4c0c38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42948/0 (socket says 172.21.15.46:42948) 2024-01-26T23:31:16.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.439+0000 7f6d51bcc700 1 -- 172.21.15.46:0/4286491769 learned_addr learned my addr 172.21.15.46:0/4286491769 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:16.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.439+0000 7f6d50bca700 1 --2- 172.21.15.46:0/4286491769 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d4c1611e0 0x7f6d4c0c33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:16.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.439+0000 7f6d513cb700 1 --2- 172.21.15.46:0/4286491769 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d4c068040 0x7f6d4c0c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:16.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.439+0000 7f6d51bcc700 1 -- 172.21.15.46:0/4286491769 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d4c068040 msgr2=0x7f6d4c0c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:16.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.439+0000 7f6d51bcc700 1 --2- 172.21.15.46:0/4286491769 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d4c068040 0x7f6d4c0c5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.439+0000 7f6d51bcc700 1 -- 172.21.15.46:0/4286491769 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d4c1611e0 msgr2=0x7f6d4c0c33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:16.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.439+0000 7f6d51bcc700 1 --2- 172.21.15.46:0/4286491769 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d4c1611e0 0x7f6d4c0c33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.439+0000 7f6d51bcc700 1 -- 172.21.15.46:0/4286491769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d48009420 con 0x7f6d4c1615b0 2024-01-26T23:31:16.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.439+0000 7f6d513cb700 1 --2- 172.21.15.46:0/4286491769 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d4c068040 0x7f6d4c0c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:31:16.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.439+0000 7f6d51bcc700 1 --2- 172.21.15.46:0/4286491769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c1615b0 0x7f6d4c0c38e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400a380 tx=0x7f6d440078f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:16.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.439+0000 7f6d3d7fa700 1 -- 172.21.15.46:0/4286491769 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4400ae10 con 0x7f6d4c1615b0 2024-01-26T23:31:16.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.439+0000 7f6d3d7fa700 1 -- 172.21.15.46:0/4286491769 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4400d650 con 0x7f6d4c1615b0 2024-01-26T23:31:16.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.439+0000 7f6d5362f700 1 -- 172.21.15.46:0/4286491769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d4c0c6240 con 0x7f6d4c1615b0 2024-01-26T23:31:16.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.439+0000 7f6d3d7fa700 1 -- 172.21.15.46:0/4286491769 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4400da10 con 0x7f6d4c1615b0 2024-01-26T23:31:16.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.439+0000 7f6d5362f700 1 -- 172.21.15.46:0/4286491769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d4c0c6800 con 0x7f6d4c1615b0 2024-01-26T23:31:16.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.441+0000 7f6d5362f700 1 -- 172.21.15.46:0/4286491769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d4c0c6fc0 con 0x7f6d4c1615b0 2024-01-26T23:31:16.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.441+0000 7f6d3d7fa700 1 -- 172.21.15.46:0/4286491769 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f6d44003020 con 0x7f6d4c1615b0 2024-01-26T23:31:16.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.441+0000 7f6d3d7fa700 1 --2- 172.21.15.46:0/4286491769 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6d30067130 0x7f6d300695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:16.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.442+0000 7f6d3d7fa700 1 -- 172.21.15.46:0/4286491769 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f6d44085310 con 0x7f6d4c1615b0 2024-01-26T23:31:16.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.442+0000 7f6d513cb700 1 --2- 172.21.15.46:0/4286491769 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6d30067130 0x7f6d300695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:16.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.442+0000 7f6d513cb700 1 --2- 172.21.15.46:0/4286491769 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6d30067130 0x7f6d300695e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6d40007fd0 tx=0x7f6d40009040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:16.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.446+0000 7f6d3d7fa700 1 -- 172.21.15.46:0/4286491769 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6d4404d5a0 con 0x7f6d4c1615b0 2024-01-26T23:31:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.681+0000 7f6d5362f700 1 -- 172.21.15.46:0/4286491769 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d4c04c710 con 0x7f6d30067130 2024-01-26T23:31:16.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.730+0000 7f6d3d7fa700 1 -- 172.21.15.46:0/4286491769 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f6d4c04c710 con 0x7f6d30067130 2024-01-26T23:31:16.732 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule activated for path /snap_test_dir1/dir7 2024-01-26T23:31:16.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.732+0000 7f6d5362f700 1 -- 172.21.15.46:0/4286491769 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6d30067130 msgr2=0x7f6d300695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:16.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.732+0000 7f6d5362f700 1 --2- 172.21.15.46:0/4286491769 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6d30067130 0x7f6d300695e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6d40007fd0 tx=0x7f6d40009040 comp rx=0 tx=0).stop 2024-01-26T23:31:16.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.732+0000 7f6d5362f700 1 -- 172.21.15.46:0/4286491769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c1615b0 msgr2=0x7f6d4c0c38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:16.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.732+0000 7f6d5362f700 1 --2- 172.21.15.46:0/4286491769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c1615b0 0x7f6d4c0c38e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400a380 tx=0x7f6d440078f0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.732+0000 7f6d5362f700 1 -- 172.21.15.46:0/4286491769 shutdown_connections 2024-01-26T23:31:16.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.732+0000 7f6d5362f700 1 --2- 172.21.15.46:0/4286491769 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6d30067130 0x7f6d300695e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.732+0000 7f6d5362f700 1 --2- 172.21.15.46:0/4286491769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c1615b0 0x7f6d4c0c38e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.732+0000 7f6d5362f700 1 --2- 172.21.15.46:0/4286491769 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d4c068040 0x7f6d4c0c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.732+0000 7f6d5362f700 1 --2- 172.21.15.46:0/4286491769 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d4c1611e0 0x7f6d4c0c33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.732+0000 7f6d5362f700 1 -- 172.21.15.46:0/4286491769 >> 172.21.15.46:0/4286491769 conn(0x7f6d4c0cdf50 msgr2=0x7f6d4c15e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:16.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.732+0000 7f6d5362f700 1 -- 172.21.15.46:0/4286491769 shutdown_connections 2024-01-26T23:31:16.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.732+0000 7f6d5362f700 1 -- 172.21.15.46:0/4286491769 wait complete. 2024-01-26T23:31:16.748 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2024-01-26T23:31:16.748 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir8 1m 2024-01-26T23:31:16.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.884+0000 7f70781ce700 1 Processor -- start 2024-01-26T23:31:16.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.884+0000 7f70781ce700 1 -- start start 2024-01-26T23:31:16.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.885+0000 7f70781ce700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f70701565a0 0x7f7070156970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:16.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.885+0000 7f70781ce700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f70700ae2f0 0x7f70700ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:16.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.885+0000 7f70781ce700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70700b0830 con 0x7f707015a520 2024-01-26T23:31:16.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.885+0000 7f70781ce700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70700ae130 con 0x7f70700ae2f0 2024-01-26T23:31:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.885+0000 7f70781ce700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70700ad350 con 0x7f70701565a0 2024-01-26T23:31:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.885+0000 7f707676b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f70700ae2f0 0x7f70700ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.885+0000 7f7075769700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f707015a520 0x7f70700acc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55918/0 (socket says 172.21.15.46:55918) 2024-01-26T23:31:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.885+0000 7f7075769700 1 -- 172.21.15.46:0/2089528327 learned_addr learned my addr 172.21.15.46:0/2089528327 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:31:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.885+0000 7f7075f6a700 1 --2- 172.21.15.46:0/2089528327 >> v2:172.21.15.106:3300/0 conn(0x7f70701565a0 0x7f7070156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.885+0000 7f707676b700 1 -- 172.21.15.46:0/2089528327 >> v2:172.21.15.106:3300/0 conn(0x7f70701565a0 msgr2=0x7f7070156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.885+0000 7f707676b700 1 --2- 172.21.15.46:0/2089528327 >> v2:172.21.15.106:3300/0 conn(0x7f70701565a0 0x7f7070156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.885+0000 7f707676b700 1 -- 172.21.15.46:0/2089528327 >> v1:172.21.15.46:6789/0 conn(0x7f707015a520 legacy=0x7f70700acc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.885+0000 7f707676b700 1 -- 172.21.15.46:0/2089528327 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70700aec40 con 0x7f70700ae2f0 2024-01-26T23:31:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.885+0000 7f7074f68700 1 -- 172.21.15.46:0/2089528327 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337970283 0 0) 0x7f70700b0830 con 0x7f707015a520 2024-01-26T23:31:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.886+0000 7f707676b700 1 --2- 172.21.15.46:0/2089528327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70700ae2f0 0x7f70700ae700 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7070064a50 tx=0x7f7068009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32dd8a4e11d09c05 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.886+0000 7f7074f68700 1 -- 172.21.15.46:0/2089528327 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706800e070 con 0x7f70700ae2f0 2024-01-26T23:31:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.886+0000 7f7074f68700 1 -- 172.21.15.46:0/2089528327 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706800bc30 con 0x7f70700ae2f0 2024-01-26T23:31:16.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.886+0000 7f7074f68700 1 -- 172.21.15.46:0/2089528327 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7068012430 con 0x7f70700ae2f0 2024-01-26T23:31:16.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.886+0000 7f70781ce700 1 -- 172.21.15.46:0/2089528327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70700ae2f0 msgr2=0x7f70700ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:16.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.886+0000 7f70781ce700 1 --2- 172.21.15.46:0/2089528327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70700ae2f0 0x7f70700ae700 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7070064a50 tx=0x7f7068009780 comp rx=0 tx=0).stop 2024-01-26T23:31:16.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.886+0000 7f70781ce700 1 -- 172.21.15.46:0/2089528327 shutdown_connections 2024-01-26T23:31:16.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.886+0000 7f70781ce700 1 --2- 172.21.15.46:0/2089528327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70700ae2f0 0x7f70700ae700 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.886+0000 7f70781ce700 1 --2- 172.21.15.46:0/2089528327 >> v2:172.21.15.106:3300/0 conn(0x7f70701565a0 0x7f7070156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.886+0000 7f70781ce700 1 -- 172.21.15.46:0/2089528327 >> 172.21.15.46:0/2089528327 conn(0x7f70700cdf80 msgr2=0x7f707006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:16.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.887+0000 7f70781ce700 1 -- 172.21.15.46:0/2089528327 shutdown_connections 2024-01-26T23:31:16.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.887+0000 7f70781ce700 1 -- 172.21.15.46:0/2089528327 wait complete. 2024-01-26T23:31:16.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.888+0000 7f70781ce700 1 Processor -- start 2024-01-26T23:31:16.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.888+0000 7f70781ce700 1 -- start start 2024-01-26T23:31:16.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.889+0000 7f70781ce700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70700ae2f0 0x7f7070145030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:16.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.889+0000 7f70781ce700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70701565a0 0x7f70700b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:16.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.889+0000 7f70781ce700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f707015a520 0x7f7070067930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:16.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.889+0000 7f70781ce700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70700b3240 con 0x7f70701565a0 2024-01-26T23:31:16.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.889+0000 7f70781ce700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70700b2630 con 0x7f707015a520 2024-01-26T23:31:16.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.889+0000 7f70781ce700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70700b2910 con 0x7f70700ae2f0 2024-01-26T23:31:16.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.889+0000 7f7075769700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70701565a0 0x7f70700b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:16.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.889+0000 7f7075769700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70701565a0 0x7f70700b1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42966/0 (socket says 172.21.15.46:42966) 2024-01-26T23:31:16.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.889+0000 7f7075769700 1 -- 172.21.15.46:0/959549827 learned_addr learned my addr 172.21.15.46:0/959549827 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:16.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.890+0000 7f7075f6a700 1 --2- 172.21.15.46:0/959549827 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70700ae2f0 0x7f7070145030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:16.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.890+0000 7f707676b700 1 --2- 172.21.15.46:0/959549827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f707015a520 0x7f7070067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:16.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.890+0000 7f7075769700 1 -- 172.21.15.46:0/959549827 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70700ae2f0 msgr2=0x7f7070145030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:16.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.890+0000 7f7075769700 1 --2- 172.21.15.46:0/959549827 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70700ae2f0 0x7f7070145030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.890+0000 7f7075769700 1 -- 172.21.15.46:0/959549827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f707015a520 msgr2=0x7f7070067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:16.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.890+0000 7f7075769700 1 --2- 172.21.15.46:0/959549827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f707015a520 0x7f7070067930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:16.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.890+0000 7f7075769700 1 -- 172.21.15.46:0/959549827 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7068009a20 con 0x7f70701565a0 2024-01-26T23:31:16.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.890+0000 7f7075769700 1 --2- 172.21.15.46:0/959549827 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70701565a0 0x7f70700b1b70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f706000c7e0 tx=0x7f7060007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:16.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.890+0000 7f7065ffb700 1 -- 172.21.15.46:0/959549827 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7060005b30 con 0x7f70701565a0 2024-01-26T23:31:16.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.890+0000 7f7065ffb700 1 -- 172.21.15.46:0/959549827 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7060005c90 con 0x7f70701565a0 2024-01-26T23:31:16.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.890+0000 7f70781ce700 1 -- 172.21.15.46:0/959549827 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70700b2bf0 con 0x7f70701565a0 2024-01-26T23:31:16.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.890+0000 7f7065ffb700 1 -- 172.21.15.46:0/959549827 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7060007300 con 0x7f70701565a0 2024-01-26T23:31:16.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.890+0000 7f70781ce700 1 -- 172.21.15.46:0/959549827 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7070152370 con 0x7f70701565a0 2024-01-26T23:31:16.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.892+0000 7f70781ce700 1 -- 172.21.15.46:0/959549827 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70700b33f0 con 0x7f70701565a0 2024-01-26T23:31:16.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.892+0000 7f7065ffb700 1 -- 172.21.15.46:0/959549827 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f706001b3b0 con 0x7f70701565a0 2024-01-26T23:31:16.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.892+0000 7f7065ffb700 1 --2- 172.21.15.46:0/959549827 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7054067130 0x7f70540695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:16.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.892+0000 7f7065ffb700 1 -- 172.21.15.46:0/959549827 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f7060084080 con 0x7f70701565a0 2024-01-26T23:31:16.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.893+0000 7f7075f6a700 1 --2- 172.21.15.46:0/959549827 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7054067130 0x7f70540695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:16.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.893+0000 7f7075f6a700 1 --2- 172.21.15.46:0/959549827 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7054067130 0x7f70540695e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f706c006fd0 tx=0x7f706c008040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:16.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:16.897+0000 7f7065ffb700 1 -- 172.21.15.46:0/959549827 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f70700b33f0 con 0x7f70701565a0 2024-01-26T23:31:17.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.130+0000 7f70781ce700 1 -- 172.21.15.46:0/959549827 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f707004c710 con 0x7f7054067130 2024-01-26T23:31:17.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.179+0000 7f7065ffb700 1 -- 172.21.15.46:0/959549827 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f707004c710 con 0x7f7054067130 2024-01-26T23:31:17.181 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule activated for path /snap_test_dir1/dir8 2024-01-26T23:31:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.182+0000 7f70781ce700 1 -- 172.21.15.46:0/959549827 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7054067130 msgr2=0x7f70540695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.182+0000 7f70781ce700 1 --2- 172.21.15.46:0/959549827 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7054067130 0x7f70540695e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f706c006fd0 tx=0x7f706c008040 comp rx=0 tx=0).stop 2024-01-26T23:31:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.182+0000 7f70781ce700 1 -- 172.21.15.46:0/959549827 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70701565a0 msgr2=0x7f70700b1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.182+0000 7f70781ce700 1 --2- 172.21.15.46:0/959549827 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70701565a0 0x7f70700b1b70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f706000c7e0 tx=0x7f7060007df0 comp rx=0 tx=0).stop 2024-01-26T23:31:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.182+0000 7f70781ce700 1 -- 172.21.15.46:0/959549827 shutdown_connections 2024-01-26T23:31:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.182+0000 7f70781ce700 1 --2- 172.21.15.46:0/959549827 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7054067130 0x7f70540695e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.182+0000 7f70781ce700 1 --2- 172.21.15.46:0/959549827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f707015a520 0x7f7070067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.182+0000 7f70781ce700 1 --2- 172.21.15.46:0/959549827 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70700ae2f0 0x7f7070145030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.182+0000 7f70781ce700 1 --2- 172.21.15.46:0/959549827 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70701565a0 0x7f70700b1b70 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.182+0000 7f70781ce700 1 -- 172.21.15.46:0/959549827 >> 172.21.15.46:0/959549827 conn(0x7f70700cdf80 msgr2=0x7f70700cc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.182+0000 7f70781ce700 1 -- 172.21.15.46:0/959549827 shutdown_connections 2024-01-26T23:31:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.182+0000 7f70781ce700 1 -- 172.21.15.46:0/959549827 wait complete. 2024-01-26T23:31:17.198 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2024-01-26T23:31:17.198 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir9 1m 2024-01-26T23:31:17.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.334+0000 7f3035079700 1 Processor -- start 2024-01-26T23:31:17.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.334+0000 7f3035079700 1 -- start start 2024-01-26T23:31:17.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.335+0000 7f3035079700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3030068b00 0x7f3030068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:17.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.335+0000 7f3035079700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f303006a660 0x7f3030159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:17.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.335+0000 7f3035079700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f303006add0 0x7f303006b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:17.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.335+0000 7f3035079700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30300a9a80 con 0x7f303006a660 2024-01-26T23:31:17.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.335+0000 7f3035079700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f303006b790 con 0x7f3030068b00 2024-01-26T23:31:17.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.335+0000 7f3035079700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f303006aab0 con 0x7f303006add0 2024-01-26T23:31:17.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.335+0000 7f302e59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f303006a660 0x7f3030159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:17.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.335+0000 7f302e59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f303006a660 0x7f3030159ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42968/0 (socket says 172.21.15.46:42968) 2024-01-26T23:31:17.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.335+0000 7f302e59c700 1 -- 172.21.15.46:0/686674929 learned_addr learned my addr 172.21.15.46:0/686674929 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:17.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.335+0000 7f302ed9d700 1 --2- 172.21.15.46:0/686674929 >> v2:172.21.15.106:3300/0 conn(0x7f3030068b00 0x7f3030068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:17.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.335+0000 7f302f59e700 1 --2- 172.21.15.46:0/686674929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f303006add0 0x7f303006b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:17.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.335+0000 7f302e59c700 1 -- 172.21.15.46:0/686674929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f303006add0 msgr2=0x7f303006b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:17.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.335+0000 7f302e59c700 1 --2- 172.21.15.46:0/686674929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f303006add0 0x7f303006b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:17.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.335+0000 7f302e59c700 1 -- 172.21.15.46:0/686674929 >> v2:172.21.15.106:3300/0 conn(0x7f3030068b00 msgr2=0x7f3030068ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:31:17.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.335+0000 7f302e59c700 1 --2- 172.21.15.46:0/686674929 >> v2:172.21.15.106:3300/0 conn(0x7f3030068b00 0x7f3030068ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:17.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.335+0000 7f302e59c700 1 -- 172.21.15.46:0/686674929 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f303015a3e0 con 0x7f303006a660 2024-01-26T23:31:17.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.336+0000 7f302e59c700 1 --2- 172.21.15.46:0/686674929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f303006a660 0x7f3030159ea0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f303006dc50 tx=0x7f3024009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=231631a5ffd2f346 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:17.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.336+0000 7f302dd9b700 1 -- 172.21.15.46:0/686674929 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302400b860 con 0x7f303006a660 2024-01-26T23:31:17.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.336+0000 7f302dd9b700 1 -- 172.21.15.46:0/686674929 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302400b9c0 con 0x7f303006a660 2024-01-26T23:31:17.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.336+0000 7f302dd9b700 1 -- 172.21.15.46:0/686674929 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302400bd60 con 0x7f303006a660 2024-01-26T23:31:17.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.336+0000 7f3035079700 1 -- 172.21.15.46:0/686674929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f303006a660 msgr2=0x7f3030159ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:17.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.336+0000 7f3035079700 1 --2- 172.21.15.46:0/686674929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f303006a660 0x7f3030159ea0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f303006dc50 tx=0x7f3024009250 comp rx=0 tx=0).stop 2024-01-26T23:31:17.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.337+0000 7f3035079700 1 -- 172.21.15.46:0/686674929 shutdown_connections 2024-01-26T23:31:17.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.337+0000 7f3035079700 1 --2- 172.21.15.46:0/686674929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f303006add0 0x7f303006b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:17.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.337+0000 7f3035079700 1 --2- 172.21.15.46:0/686674929 >> v2:172.21.15.106:3300/0 conn(0x7f3030068b00 0x7f3030068ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:17.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.337+0000 7f3035079700 1 --2- 172.21.15.46:0/686674929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f303006a660 0x7f3030159ea0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:17.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.337+0000 7f3035079700 1 -- 172.21.15.46:0/686674929 >> 172.21.15.46:0/686674929 conn(0x7f30300ca100 msgr2=0x7f303006e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:17.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.337+0000 7f3035079700 1 -- 172.21.15.46:0/686674929 shutdown_connections 2024-01-26T23:31:17.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.337+0000 7f3035079700 1 -- 172.21.15.46:0/686674929 wait complete. 2024-01-26T23:31:17.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.338+0000 7f3035079700 1 Processor -- start 2024-01-26T23:31:17.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.338+0000 7f3035079700 1 -- start start 2024-01-26T23:31:17.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.339+0000 7f3035079700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3030068b00 0x7f303016dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:17.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.339+0000 7f3035079700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f303006a660 0x7f30300c3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:17.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.339+0000 7f3035079700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f303006add0 0x7f30300aa060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:17.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.339+0000 7f3035079700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30300b10b0 con 0x7f303006a660 2024-01-26T23:31:17.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.339+0000 7f3035079700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30300c3880 con 0x7f3030068b00 2024-01-26T23:31:17.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.339+0000 7f3035079700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30300c4600 con 0x7f303006add0 2024-01-26T23:31:17.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.339+0000 7f302e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f303006a660 0x7f30300c3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:17.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.339+0000 7f302e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f303006a660 0x7f30300c3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42978/0 (socket says 172.21.15.46:42978) 2024-01-26T23:31:17.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.339+0000 7f302e59c700 1 -- 172.21.15.46:0/162162867 learned_addr learned my addr 172.21.15.46:0/162162867 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:31:17.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.339+0000 7f302ed9d700 1 --2- 172.21.15.46:0/162162867 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3030068b00 0x7f303016dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:17.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.339+0000 7f302f59e700 1 --2- 172.21.15.46:0/162162867 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f303006add0 0x7f30300aa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:17.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.339+0000 7f302e59c700 1 -- 172.21.15.46:0/162162867 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f303006add0 msgr2=0x7f30300aa060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:17.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.339+0000 7f302e59c700 1 --2- 172.21.15.46:0/162162867 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f303006add0 0x7f30300aa060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:17.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.339+0000 7f302e59c700 1 -- 172.21.15.46:0/162162867 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3030068b00 msgr2=0x7f303016dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:17.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.339+0000 7f302e59c700 1 --2- 172.21.15.46:0/162162867 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3030068b00 0x7f303016dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:17.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.339+0000 7f302e59c700 1 -- 172.21.15.46:0/162162867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30240094f0 con 0x7f303006a660 2024-01-26T23:31:17.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.339+0000 7f302e59c700 1 --2- 172.21.15.46:0/162162867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f303006a660 0x7f30300c3340 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3030069d70 tx=0x7f3024017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:17.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.340+0000 7f3016ffd700 1 -- 172.21.15.46:0/162162867 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3024011dc0 con 0x7f303006a660 2024-01-26T23:31:17.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.340+0000 7f3016ffd700 1 -- 172.21.15.46:0/162162867 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3024024440 con 0x7f303006a660 2024-01-26T23:31:17.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.340+0000 7f3035079700 1 -- 172.21.15.46:0/162162867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30300c48e0 con 0x7f303006a660 2024-01-26T23:31:17.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.340+0000 7f3016ffd700 1 -- 172.21.15.46:0/162162867 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3024024800 con 0x7f303006a660 2024-01-26T23:31:17.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.340+0000 7f3035079700 1 -- 172.21.15.46:0/162162867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30300aa5a0 con 0x7f303006a660 2024-01-26T23:31:17.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.341+0000 7f3035079700 1 -- 172.21.15.46:0/162162867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30300aa790 con 0x7f303006a660 2024-01-26T23:31:17.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.341+0000 7f3016ffd700 1 -- 172.21.15.46:0/162162867 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f302401a420 con 0x7f303006a660 2024-01-26T23:31:17.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.342+0000 7f3016ffd700 1 --2- 172.21.15.46:0/162162867 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f301c067130 0x7f301c0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:31:17.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.342+0000 7f3016ffd700 1 -- 172.21.15.46:0/162162867 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f3024086960 con 0x7f303006a660 2024-01-26T23:31:17.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.342+0000 7f302ed9d700 1 --2- 172.21.15.46:0/162162867 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f301c067130 0x7f301c0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:31:17.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.342+0000 7f302ed9d700 1 --2- 172.21.15.46:0/162162867 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f301c067130 0x7f301c0695e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f30200061f0 tx=0x7f3020008040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:31:17.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.346+0000 7f3016ffd700 1 -- 172.21.15.46:0/162162867 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3024053710 con 0x7f303006a660 2024-01-26T23:31:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.581+0000 7f3035079700 1 -- 172.21.15.46:0/162162867 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f303004c710 con 0x7f301c067130 2024-01-26T23:31:17.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.629+0000 7f3016ffd700 1 -- 172.21.15.46:0/162162867 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f303004c710 con 0x7f301c067130 2024-01-26T23:31:17.631 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule activated for path /snap_test_dir1/dir9 2024-01-26T23:31:17.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.632+0000 7f3035079700 1 -- 172.21.15.46:0/162162867 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f301c067130 msgr2=0x7f301c0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:17.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.632+0000 7f3035079700 1 --2- 172.21.15.46:0/162162867 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f301c067130 0x7f301c0695e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f30200061f0 tx=0x7f3020008040 comp rx=0 tx=0).stop 2024-01-26T23:31:17.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.632+0000 7f3035079700 1 -- 172.21.15.46:0/162162867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f303006a660 msgr2=0x7f30300c3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:31:17.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.632+0000 7f3035079700 1 --2- 172.21.15.46:0/162162867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f303006a660 0x7f30300c3340 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3030069d70 tx=0x7f3024017000 comp rx=0 tx=0).stop 2024-01-26T23:31:17.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.632+0000 7f3035079700 1 -- 172.21.15.46:0/162162867 shutdown_connections 2024-01-26T23:31:17.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.632+0000 7f3035079700 1 --2- 172.21.15.46:0/162162867 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f301c067130 0x7f301c0695e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:17.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.632+0000 7f3035079700 1 --2- 172.21.15.46:0/162162867 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f303006add0 0x7f30300aa060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:17.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.632+0000 7f3035079700 1 --2- 172.21.15.46:0/162162867 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3030068b00 0x7f303016dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:17.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.632+0000 7f3035079700 1 --2- 172.21.15.46:0/162162867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f303006a660 0x7f30300c3340 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:31:17.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.632+0000 7f3035079700 1 -- 172.21.15.46:0/162162867 >> 172.21.15.46:0/162162867 conn(0x7f30300ca100 msgr2=0x7f30300cb800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:31:17.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.632+0000 7f3035079700 1 -- 172.21.15.46:0/162162867 shutdown_connections 2024-01-26T23:31:17.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:31:17.632+0000 7f3035079700 1 -- 172.21.15.46:0/162162867 wait complete. 2024-01-26T23:31:17.648 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2024-01-26T23:31:22.286 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:31:22.288 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:31:52.352 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:31:52.356 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:32:00.059 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:32:00.058+0000 7f5a5ae09700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:32:00.059 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:32:00.058+0000 7f5a5ae09700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:32:00.059 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:32:00.058+0000 7f5a5ae09700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:32:00.059 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:32:00.058+0000 7f5a5ae09700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:32:00.060 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:32:00.058+0000 7f5a5ae09700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:32:22.430 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:32:22.434 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:32:52.523 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:32:52.542 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:33:22.595 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:33:22.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:33:52.662 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:33:52.666 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:34:22.759 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:34:22.763 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:34:52.831 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:34:52.834 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:35:22.910 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:35:22.916 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:35:52.994 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:35:52.997 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:36:23.083 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:36:23.086 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:36:53.183 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:36:53.187 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:37:23.269 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:37:23.273 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:37:53.342 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:37:53.346 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:38:23.439 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:38:23.443 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:38:53.530 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:38:53.533 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:39:00.108 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:39:00.106+0000 7f5a5357a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:39:00.108 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:39:00.106+0000 7f5a5357a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:39:00.108 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:39:00.106+0000 7f5a5357a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:39:00.108 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:39:00.106+0000 7f5a5357a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:39:00.108 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:39:00.106+0000 7f5a5357a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:39:23.651 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:39:23.655 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:39:53.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:39:53.728 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:40:23.797 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:40:23.801 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:40:53.872 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:40:53.876 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:41:23.975 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:41:23.979 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:41:54.065 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:41:54.068 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:42:19.650 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir0 1m 2024-01-26T23:42:19.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.795+0000 7fd3d55be700 1 Processor -- start 2024-01-26T23:42:19.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.795+0000 7fd3d55be700 1 -- start start 2024-01-26T23:42:19.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.796+0000 7fd3d55be700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d0159da0 0x7fd3d00ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.796+0000 7fd3d55be700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d00b1a60 con 0x7fd3d015d8c0 2024-01-26T23:42:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.796+0000 7fd3d55be700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d015a1d0 con 0x7fd3d01599d0 2024-01-26T23:42:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.796+0000 7fd3d55be700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d00b11c0 con 0x7fd3d0159da0 2024-01-26T23:42:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.796+0000 7fd3cf7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd3d015d8c0 0x7fd3d00b3b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55804/0 (socket says 172.21.15.46:55804) 2024-01-26T23:42:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.796+0000 7fd3cf7fe700 1 -- 172.21.15.46:0/1044257345 learned_addr learned my addr 172.21.15.46:0/1044257345 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.796+0000 7fd3ce7fc700 1 --2- 172.21.15.46:0/1044257345 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d0159da0 0x7fd3d00ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.796+0000 7fd3cdffb700 1 -- 172.21.15.46:0/1044257345 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407354207 0 0) 0x7fd3d00b1a60 con 0x7fd3d015d8c0 2024-01-26T23:42:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.796+0000 7fd3cdffb700 1 -- 172.21.15.46:0/1044257345 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3bc003570 con 0x7fd3d015d8c0 2024-01-26T23:42:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.796+0000 7fd3cdffb700 1 -- 172.21.15.46:0/1044257345 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905677868 0 0) 0x7fd3d015a1d0 con 0x7fd3d01599d0 2024-01-26T23:42:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.797+0000 7fd3cdffb700 1 -- 172.21.15.46:0/1044257345 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d00b1a60 con 0x7fd3d01599d0 2024-01-26T23:42:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.797+0000 7fd3ce7fc700 1 -- 172.21.15.46:0/1044257345 >> v1:172.21.15.106:6789/0 conn(0x7fd3d01599d0 legacy=0x7fd3d00b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.797+0000 7fd3ce7fc700 1 -- 172.21.15.46:0/1044257345 >> v1:172.21.15.46:6789/0 conn(0x7fd3d015d8c0 legacy=0x7fd3d00b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.797+0000 7fd3ce7fc700 1 -- 172.21.15.46:0/1044257345 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d00b14a0 con 0x7fd3d0159da0 2024-01-26T23:42:19.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.797+0000 7fd3ce7fc700 1 --2- 172.21.15.46:0/1044257345 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d0159da0 0x7fd3d00ab1c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8009910 tx=0x7fd3b80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=791599490db2bd7 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:19.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.797+0000 7fd3cdffb700 1 -- 172.21.15.46:0/1044257345 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8011630 con 0x7fd3d0159da0 2024-01-26T23:42:19.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.797+0000 7fd3cdffb700 1 -- 172.21.15.46:0/1044257345 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b8011790 con 0x7fd3d0159da0 2024-01-26T23:42:19.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.797+0000 7fd3cdffb700 1 -- 172.21.15.46:0/1044257345 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8011b30 con 0x7fd3d0159da0 2024-01-26T23:42:19.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.798+0000 7fd3d55be700 1 -- 172.21.15.46:0/1044257345 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d0159da0 msgr2=0x7fd3d00ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:19.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.798+0000 7fd3d55be700 1 --2- 172.21.15.46:0/1044257345 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d0159da0 0x7fd3d00ab1c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8009910 tx=0x7fd3b80091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:19.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.798+0000 7fd3d55be700 1 -- 172.21.15.46:0/1044257345 shutdown_connections 2024-01-26T23:42:19.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.798+0000 7fd3d55be700 1 --2- 172.21.15.46:0/1044257345 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d0159da0 0x7fd3d00ab1c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:19.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.798+0000 7fd3d55be700 1 -- 172.21.15.46:0/1044257345 >> 172.21.15.46:0/1044257345 conn(0x7fd3d006f840 msgr2=0x7fd3d0087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:19.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.798+0000 7fd3d55be700 1 -- 172.21.15.46:0/1044257345 shutdown_connections 2024-01-26T23:42:19.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.798+0000 7fd3d55be700 1 -- 172.21.15.46:0/1044257345 wait complete. 2024-01-26T23:42:19.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.799+0000 7fd3d55be700 1 Processor -- start 2024-01-26T23:42:19.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.799+0000 7fd3d55be700 1 -- start start 2024-01-26T23:42:19.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.800+0000 7fd3d55be700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d01599d0 0x7fd3d009de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:19.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.800+0000 7fd3d55be700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3d0159da0 0x7fd3d00ab910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:19.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.800+0000 7fd3d55be700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d015d8c0 0x7fd3d01579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:19.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.800+0000 7fd3d55be700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d009e3b0 con 0x7fd3d01599d0 2024-01-26T23:42:19.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.800+0000 7fd3d55be700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d009d9e0 con 0x7fd3d0159da0 2024-01-26T23:42:19.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.800+0000 7fd3d55be700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d009cdd0 con 0x7fd3d015d8c0 2024-01-26T23:42:19.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.800+0000 7fd3ceffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d01599d0 0x7fd3d009de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.800+0000 7fd3ceffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d01599d0 0x7fd3d009de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35090/0 (socket says 172.21.15.46:35090) 2024-01-26T23:42:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.800+0000 7fd3ceffd700 1 -- 172.21.15.46:0/2816631878 learned_addr learned my addr 172.21.15.46:0/2816631878 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.800+0000 7fd3ce7fc700 1 --2- 172.21.15.46:0/2816631878 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3d0159da0 0x7fd3d00ab910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.800+0000 7fd3cf7fe700 1 --2- 172.21.15.46:0/2816631878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d015d8c0 0x7fd3d01579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.801+0000 7fd3ceffd700 1 -- 172.21.15.46:0/2816631878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d015d8c0 msgr2=0x7fd3d01579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.801+0000 7fd3ceffd700 1 --2- 172.21.15.46:0/2816631878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d015d8c0 0x7fd3d01579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.801+0000 7fd3ceffd700 1 -- 172.21.15.46:0/2816631878 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3d0159da0 msgr2=0x7fd3d00ab910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.801+0000 7fd3ceffd700 1 --2- 172.21.15.46:0/2816631878 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3d0159da0 0x7fd3d00ab910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.801+0000 7fd3ceffd700 1 -- 172.21.15.46:0/2816631878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b8009420 con 0x7fd3d01599d0 2024-01-26T23:42:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.801+0000 7fd3ceffd700 1 --2- 172.21.15.46:0/2816631878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d01599d0 0x7fd3d009de70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd3c000c7c0 tx=0x7fd3c0007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.801+0000 7fd3c6ffd700 1 -- 172.21.15.46:0/2816631878 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c0001530 con 0x7fd3d01599d0 2024-01-26T23:42:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.801+0000 7fd3c6ffd700 1 -- 172.21.15.46:0/2816631878 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c00016b0 con 0x7fd3d01599d0 2024-01-26T23:42:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.801+0000 7fd3d55be700 1 -- 172.21.15.46:0/2816631878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d009d0b0 con 0x7fd3d01599d0 2024-01-26T23:42:19.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.801+0000 7fd3c6ffd700 1 -- 172.21.15.46:0/2816631878 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c0001a70 con 0x7fd3d01599d0 2024-01-26T23:42:19.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.801+0000 7fd3d55be700 1 -- 172.21.15.46:0/2816631878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d009d6f0 con 0x7fd3d01599d0 2024-01-26T23:42:19.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.802+0000 7fd3d55be700 1 -- 172.21.15.46:0/2816631878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d009dcc0 con 0x7fd3d01599d0 2024-01-26T23:42:19.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.803+0000 7fd3c6ffd700 1 -- 172.21.15.46:0/2816631878 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fd3c001b3b0 con 0x7fd3d01599d0 2024-01-26T23:42:19.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.803+0000 7fd3c6ffd700 1 --2- 172.21.15.46:0/2816631878 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd3b0067130 0x7fd3b00695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:19.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.803+0000 7fd3c6ffd700 1 -- 172.21.15.46:0/2816631878 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fd3c0084270 con 0x7fd3d01599d0 2024-01-26T23:42:19.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.804+0000 7fd3ce7fc700 1 --2- 172.21.15.46:0/2816631878 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd3b0067130 0x7fd3b00695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:19.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.804+0000 7fd3ce7fc700 1 --2- 172.21.15.46:0/2816631878 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd3b0067130 0x7fd3b00695e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8009860 tx=0x7fd3b8017000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:19.808+0000 7fd3c6ffd700 1 -- 172.21.15.46:0/2816631878 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd3c0051000 con 0x7fd3d01599d0 2024-01-26T23:42:20.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.040+0000 7fd3d55be700 1 -- 172.21.15.46:0/2816631878 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3d004c710 con 0x7fd3b0067130 2024-01-26T23:42:20.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.106+0000 7fd3c6ffd700 1 -- 172.21.15.46:0/2816631878 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd3d004c710 con 0x7fd3b0067130 2024-01-26T23:42:20.107 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2024-01-26T23:42:20.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.108+0000 7fd3d55be700 1 -- 172.21.15.46:0/2816631878 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd3b0067130 msgr2=0x7fd3b00695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:20.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.109+0000 7fd3d55be700 1 --2- 172.21.15.46:0/2816631878 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd3b0067130 0x7fd3b00695e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8009860 tx=0x7fd3b8017000 comp rx=0 tx=0).stop 2024-01-26T23:42:20.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.109+0000 7fd3d55be700 1 -- 172.21.15.46:0/2816631878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d01599d0 msgr2=0x7fd3d009de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:20.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.109+0000 7fd3d55be700 1 --2- 172.21.15.46:0/2816631878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d01599d0 0x7fd3d009de70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd3c000c7c0 tx=0x7fd3c0007e00 comp rx=0 tx=0).stop 2024-01-26T23:42:20.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.109+0000 7fd3d55be700 1 -- 172.21.15.46:0/2816631878 shutdown_connections 2024-01-26T23:42:20.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.109+0000 7fd3d55be700 1 --2- 172.21.15.46:0/2816631878 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd3b0067130 0x7fd3b00695e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.109+0000 7fd3d55be700 1 --2- 172.21.15.46:0/2816631878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d01599d0 0x7fd3d009de70 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.109+0000 7fd3d55be700 1 --2- 172.21.15.46:0/2816631878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d015d8c0 0x7fd3d01579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.109+0000 7fd3d55be700 1 --2- 172.21.15.46:0/2816631878 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3d0159da0 0x7fd3d00ab910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.109+0000 7fd3d55be700 1 -- 172.21.15.46:0/2816631878 >> 172.21.15.46:0/2816631878 conn(0x7fd3d006f840 msgr2=0x7fd3d00b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:20.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.109+0000 7fd3d55be700 1 -- 172.21.15.46:0/2816631878 shutdown_connections 2024-01-26T23:42:20.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.109+0000 7fd3d55be700 1 -- 172.21.15.46:0/2816631878 wait complete. 2024-01-26T23:42:20.124 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2024-01-26T23:42:20.124 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir1 1m 2024-01-26T23:42:20.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.262+0000 7fa8f3d66700 1 Processor -- start 2024-01-26T23:42:20.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.262+0000 7fa8f3d66700 1 -- start start 2024-01-26T23:42:20.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.263+0000 7fa8f3d66700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa8ec0cf840 0x7fa8ec0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:20.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.263+0000 7fa8f3d66700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8ec0cb850 0x7fa8ec0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:20.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.263+0000 7fa8f3d66700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8ec0c62e0 con 0x7fa8ec15e1e0 2024-01-26T23:42:20.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.263+0000 7fa8f3d66700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8ec0cbcd0 con 0x7fa8ec0cf840 2024-01-26T23:42:20.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.263+0000 7fa8f3d66700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0b3a90 con 0x7fa8ec0cb850 2024-01-26T23:42:20.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.263+0000 7fa8f1301700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa8ec0cf840 0x7fa8ec0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:20.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.263+0000 7fa8f2303700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8ec0cb850 0x7fa8ec0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:20.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.263+0000 7fa8f2303700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8ec0cb850 0x7fa8ec0b3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:60600/0 (socket says 172.21.15.46:60600) 2024-01-26T23:42:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.263+0000 7fa8f1301700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa8ec0cf840 0x7fa8ec0cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:44786/0 (socket says 172.21.15.46:44786) 2024-01-26T23:42:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.263+0000 7fa8f2303700 1 -- 172.21.15.46:0/1973749928 learned_addr learned my addr 172.21.15.46:0/1973749928 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.264+0000 7fa8f1301700 1 -- 172.21.15.46:0/1973749928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8ec0cb850 msgr2=0x7fa8ec0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.264+0000 7fa8f1301700 1 --2- 172.21.15.46:0/1973749928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8ec0cb850 0x7fa8ec0b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.264+0000 7fa8f1301700 1 -- 172.21.15.46:0/1973749928 >> v1:172.21.15.106:6789/0 conn(0x7fa8ec15e1e0 legacy=0x7fa8ec0cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.264+0000 7fa8f1301700 1 -- 172.21.15.46:0/1973749928 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec0ce4b0 con 0x7fa8ec0cf840 2024-01-26T23:42:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.264+0000 7fa8f2303700 1 --2- 172.21.15.46:0/1973749928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8ec0cb850 0x7fa8ec0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.264+0000 7fa8f0b00700 1 -- 172.21.15.46:0/1973749928 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837970499 0 0) 0x7fa8ec0c62e0 con 0x7fa8ec15e1e0 2024-01-26T23:42:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.264+0000 7fa8f1301700 1 --2- 172.21.15.46:0/1973749928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8ec0cf840 0x7fa8ec0cc870 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc0098e0 tx=0x7fa8dc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91289f9e3f6e9fc2 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.264+0000 7fa8f0b00700 1 -- 172.21.15.46:0/1973749928 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc00b640 con 0x7fa8ec0cf840 2024-01-26T23:42:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.264+0000 7fa8f0b00700 1 -- 172.21.15.46:0/1973749928 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8dc00b7a0 con 0x7fa8ec0cf840 2024-01-26T23:42:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.264+0000 7fa8f0b00700 1 -- 172.21.15.46:0/1973749928 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc00bb40 con 0x7fa8ec0cf840 2024-01-26T23:42:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.265+0000 7fa8f3d66700 1 -- 172.21.15.46:0/1973749928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8ec0cf840 msgr2=0x7fa8ec0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.265+0000 7fa8f3d66700 1 --2- 172.21.15.46:0/1973749928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8ec0cf840 0x7fa8ec0cc870 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc0098e0 tx=0x7fa8dc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.265+0000 7fa8f3d66700 1 -- 172.21.15.46:0/1973749928 shutdown_connections 2024-01-26T23:42:20.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.265+0000 7fa8f3d66700 1 --2- 172.21.15.46:0/1973749928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8ec0cb850 0x7fa8ec0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.265+0000 7fa8f3d66700 1 --2- 172.21.15.46:0/1973749928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8ec0cf840 0x7fa8ec0cc870 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.265+0000 7fa8f3d66700 1 -- 172.21.15.46:0/1973749928 >> 172.21.15.46:0/1973749928 conn(0x7fa8ec087f60 msgr2=0x7fa8ec06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:20.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.265+0000 7fa8f3d66700 1 -- 172.21.15.46:0/1973749928 shutdown_connections 2024-01-26T23:42:20.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.265+0000 7fa8f3d66700 1 -- 172.21.15.46:0/1973749928 wait complete. 2024-01-26T23:42:20.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.266+0000 7fa8f3d66700 1 Processor -- start 2024-01-26T23:42:20.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.266+0000 7fa8f3d66700 1 -- start start 2024-01-26T23:42:20.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.267+0000 7fa8f3d66700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8ec0cb850 0x7fa8ec171560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:20.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.267+0000 7fa8f3d66700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8ec0cf840 0x7fa8ec14cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:20.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.267+0000 7fa8f3d66700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8ec15e1e0 0x7fa8ec1612e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:20.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.267+0000 7fa8f3d66700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec172500 con 0x7fa8ec0cb850 2024-01-26T23:42:20.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.267+0000 7fa8f3d66700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0b5530 con 0x7fa8ec0cf840 2024-01-26T23:42:20.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.267+0000 7fa8f3d66700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0c79e0 con 0x7fa8ec15e1e0 2024-01-26T23:42:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.267+0000 7fa8f1b02700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8ec0cb850 0x7fa8ec171560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.267+0000 7fa8f1b02700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8ec0cb850 0x7fa8ec171560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35098/0 (socket says 172.21.15.46:35098) 2024-01-26T23:42:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.267+0000 7fa8f1b02700 1 -- 172.21.15.46:0/2028907774 learned_addr learned my addr 172.21.15.46:0/2028907774 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.267+0000 7fa8f1301700 1 --2- 172.21.15.46:0/2028907774 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8ec0cf840 0x7fa8ec14cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.267+0000 7fa8f2303700 1 --2- 172.21.15.46:0/2028907774 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8ec15e1e0 0x7fa8ec1612e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.268+0000 7fa8f1b02700 1 -- 172.21.15.46:0/2028907774 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8ec15e1e0 msgr2=0x7fa8ec1612e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.268+0000 7fa8f1b02700 1 --2- 172.21.15.46:0/2028907774 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8ec15e1e0 0x7fa8ec1612e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.268+0000 7fa8f1b02700 1 -- 172.21.15.46:0/2028907774 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8ec0cf840 msgr2=0x7fa8ec14cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.268+0000 7fa8f1b02700 1 --2- 172.21.15.46:0/2028907774 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8ec0cf840 0x7fa8ec14cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.268+0000 7fa8f1b02700 1 -- 172.21.15.46:0/2028907774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e4009ae0 con 0x7fa8ec0cb850 2024-01-26T23:42:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.268+0000 7fa8f1b02700 1 --2- 172.21.15.46:0/2028907774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8ec0cb850 0x7fa8ec171560 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800c840 tx=0x7fa8e8007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.268+0000 7fa8e1ffb700 1 -- 172.21.15.46:0/2028907774 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8011590 con 0x7fa8ec0cb850 2024-01-26T23:42:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.268+0000 7fa8e1ffb700 1 -- 172.21.15.46:0/2028907774 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e8011710 con 0x7fa8ec0cb850 2024-01-26T23:42:20.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.268+0000 7fa8f3d66700 1 -- 172.21.15.46:0/2028907774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8dc009420 con 0x7fa8ec0cb850 2024-01-26T23:42:20.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.268+0000 7fa8e1ffb700 1 -- 172.21.15.46:0/2028907774 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8011ad0 con 0x7fa8ec0cb850 2024-01-26T23:42:20.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.268+0000 7fa8f3d66700 1 -- 172.21.15.46:0/2028907774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8ec171f40 con 0x7fa8ec0cb850 2024-01-26T23:42:20.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.269+0000 7fa8f3d66700 1 -- 172.21.15.46:0/2028907774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8ec04c710 con 0x7fa8ec0cb850 2024-01-26T23:42:20.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.269+0000 7fa8e1ffb700 1 -- 172.21.15.46:0/2028907774 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fa8e801b3b0 con 0x7fa8ec0cb850 2024-01-26T23:42:20.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.270+0000 7fa8e1ffb700 1 --2- 172.21.15.46:0/2028907774 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa8d0067130 0x7fa8d00695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:20.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.270+0000 7fa8e1ffb700 1 -- 172.21.15.46:0/2028907774 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fa8e8083ee0 con 0x7fa8ec0cb850 2024-01-26T23:42:20.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.270+0000 7fa8f1301700 1 --2- 172.21.15.46:0/2028907774 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa8d0067130 0x7fa8d00695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:20.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.271+0000 7fa8f1301700 1 --2- 172.21.15.46:0/2028907774 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa8d0067130 0x7fa8d00695e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc0093f0 tx=0x7fa8dc017000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:20.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.275+0000 7fa8e1ffb700 1 -- 172.21.15.46:0/2028907774 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa8e8050d20 con 0x7fa8ec0cb850 2024-01-26T23:42:20.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.508+0000 7fa8f3d66700 1 -- 172.21.15.46:0/2028907774 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8ec0c56d0 con 0x7fa8d0067130 2024-01-26T23:42:20.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.572+0000 7fa8e1ffb700 1 -- 172.21.15.46:0/2028907774 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa8ec0c56d0 con 0x7fa8d0067130 2024-01-26T23:42:20.573 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2024-01-26T23:42:20.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.574+0000 7fa8f3d66700 1 -- 172.21.15.46:0/2028907774 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa8d0067130 msgr2=0x7fa8d00695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:20.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.574+0000 7fa8f3d66700 1 --2- 172.21.15.46:0/2028907774 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa8d0067130 0x7fa8d00695e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc0093f0 tx=0x7fa8dc017000 comp rx=0 tx=0).stop 2024-01-26T23:42:20.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.574+0000 7fa8f3d66700 1 -- 172.21.15.46:0/2028907774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8ec0cb850 msgr2=0x7fa8ec171560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:20.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.574+0000 7fa8f3d66700 1 --2- 172.21.15.46:0/2028907774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8ec0cb850 0x7fa8ec171560 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800c840 tx=0x7fa8e8007dd0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.574+0000 7fa8f3d66700 1 -- 172.21.15.46:0/2028907774 shutdown_connections 2024-01-26T23:42:20.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.574+0000 7fa8f3d66700 1 --2- 172.21.15.46:0/2028907774 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa8d0067130 0x7fa8d00695e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.575+0000 7fa8f3d66700 1 --2- 172.21.15.46:0/2028907774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8ec0cb850 0x7fa8ec171560 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.575+0000 7fa8f3d66700 1 --2- 172.21.15.46:0/2028907774 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8ec15e1e0 0x7fa8ec1612e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.575+0000 7fa8f3d66700 1 --2- 172.21.15.46:0/2028907774 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8ec0cf840 0x7fa8ec14cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.575+0000 7fa8f3d66700 1 -- 172.21.15.46:0/2028907774 >> 172.21.15.46:0/2028907774 conn(0x7fa8ec087f60 msgr2=0x7fa8ec15e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:20.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.575+0000 7fa8f3d66700 1 -- 172.21.15.46:0/2028907774 shutdown_connections 2024-01-26T23:42:20.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.575+0000 7fa8f3d66700 1 -- 172.21.15.46:0/2028907774 wait complete. 2024-01-26T23:42:20.589 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2024-01-26T23:42:20.589 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir2 1m 2024-01-26T23:42:20.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.727+0000 7fb36ebee700 1 Processor -- start 2024-01-26T23:42:20.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.727+0000 7fb36ebee700 1 -- start start 2024-01-26T23:42:20.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.728+0000 7fb36ebee700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb368068b00 0x7fb368068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:20.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.728+0000 7fb36ebee700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3680a9a80 con 0x7fb36806add0 2024-01-26T23:42:20.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.728+0000 7fb36ebee700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36806aad0 con 0x7fb36806a660 2024-01-26T23:42:20.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.728+0000 7fb36ebee700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36815a680 con 0x7fb368068b00 2024-01-26T23:42:20.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.728+0000 7fb36d18b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb36806add0 0x7fb3680a9f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34796/0 (socket says 172.21.15.46:34796) 2024-01-26T23:42:20.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.728+0000 7fb36d18b700 1 -- 172.21.15.46:0/2179516440 learned_addr learned my addr 172.21.15.46:0/2179516440 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:20.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.728+0000 7fb36c98a700 1 --2- 172.21.15.46:0/2179516440 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb368068b00 0x7fb368068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:20.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.729+0000 7fb3677fe700 1 -- 172.21.15.46:0/2179516440 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184719344 0 0) 0x7fb3680a9a80 con 0x7fb36806add0 2024-01-26T23:42:20.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.729+0000 7fb3677fe700 1 -- 172.21.15.46:0/2179516440 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb354003570 con 0x7fb36806add0 2024-01-26T23:42:20.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.729+0000 7fb3677fe700 1 -- 172.21.15.46:0/2179516440 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876556875 0 0) 0x7fb36806aad0 con 0x7fb36806a660 2024-01-26T23:42:20.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.729+0000 7fb3677fe700 1 -- 172.21.15.46:0/2179516440 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3680a9a80 con 0x7fb36806a660 2024-01-26T23:42:20.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.729+0000 7fb36c98a700 1 -- 172.21.15.46:0/2179516440 >> v1:172.21.15.106:6789/0 conn(0x7fb36806a660 legacy=0x7fb368159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:20.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.729+0000 7fb36c98a700 1 -- 172.21.15.46:0/2179516440 >> v1:172.21.15.46:6789/0 conn(0x7fb36806add0 legacy=0x7fb3680a9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:20.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.729+0000 7fb36c98a700 1 -- 172.21.15.46:0/2179516440 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36815a960 con 0x7fb368068b00 2024-01-26T23:42:20.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.729+0000 7fb36c98a700 1 --2- 172.21.15.46:0/2179516440 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb368068b00 0x7fb368068ed0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb358009900 tx=0x7fb358009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f84340db2da4e2be server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:20.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.729+0000 7fb3677fe700 1 -- 172.21.15.46:0/2179516440 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358005c10 con 0x7fb368068b00 2024-01-26T23:42:20.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.729+0000 7fb3677fe700 1 -- 172.21.15.46:0/2179516440 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb358005d70 con 0x7fb368068b00 2024-01-26T23:42:20.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.729+0000 7fb3677fe700 1 -- 172.21.15.46:0/2179516440 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358006240 con 0x7fb368068b00 2024-01-26T23:42:20.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.730+0000 7fb36ebee700 1 -- 172.21.15.46:0/2179516440 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb368068b00 msgr2=0x7fb368068ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:20.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.730+0000 7fb36ebee700 1 --2- 172.21.15.46:0/2179516440 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb368068b00 0x7fb368068ed0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb358009900 tx=0x7fb358009120 comp rx=0 tx=0).stop 2024-01-26T23:42:20.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.730+0000 7fb36ebee700 1 -- 172.21.15.46:0/2179516440 shutdown_connections 2024-01-26T23:42:20.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.730+0000 7fb36ebee700 1 --2- 172.21.15.46:0/2179516440 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb368068b00 0x7fb368068ed0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.730+0000 7fb36ebee700 1 -- 172.21.15.46:0/2179516440 >> 172.21.15.46:0/2179516440 conn(0x7fb3680ca100 msgr2=0x7fb36806e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:20.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.730+0000 7fb36ebee700 1 -- 172.21.15.46:0/2179516440 shutdown_connections 2024-01-26T23:42:20.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.730+0000 7fb36ebee700 1 -- 172.21.15.46:0/2179516440 wait complete. 2024-01-26T23:42:20.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.731+0000 7fb36ebee700 1 Processor -- start 2024-01-26T23:42:20.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.731+0000 7fb36ebee700 1 -- start start 2024-01-26T23:42:20.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.732+0000 7fb36ebee700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb368068b00 0x7fb368169890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:20.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.732+0000 7fb36ebee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36806a660 0x7fb36814d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:20.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.732+0000 7fb36ebee700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb36806add0 0x7fb3680af970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:20.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.732+0000 7fb36ebee700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3680aca20 con 0x7fb36806a660 2024-01-26T23:42:20.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.732+0000 7fb36ebee700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb368169ea0 con 0x7fb36806add0 2024-01-26T23:42:20.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.732+0000 7fb36ebee700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36816a180 con 0x7fb368068b00 2024-01-26T23:42:20.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.732+0000 7fb367fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36806a660 0x7fb36814d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:20.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.732+0000 7fb367fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36806a660 0x7fb36814d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51290/0 (socket says 172.21.15.46:51290) 2024-01-26T23:42:20.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.732+0000 7fb367fff700 1 -- 172.21.15.46:0/1259143870 learned_addr learned my addr 172.21.15.46:0/1259143870 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:20.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.732+0000 7fb36c98a700 1 --2- 172.21.15.46:0/1259143870 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb368068b00 0x7fb368169890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:20.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.732+0000 7fb36d18b700 1 --2- 172.21.15.46:0/1259143870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb36806add0 0x7fb3680af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:20.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.733+0000 7fb367fff700 1 -- 172.21.15.46:0/1259143870 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb368068b00 msgr2=0x7fb368169890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:20.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.733+0000 7fb367fff700 1 --2- 172.21.15.46:0/1259143870 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb368068b00 0x7fb368169890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.733+0000 7fb367fff700 1 -- 172.21.15.46:0/1259143870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb36806add0 msgr2=0x7fb3680af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:20.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.733+0000 7fb367fff700 1 --2- 172.21.15.46:0/1259143870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb36806add0 0x7fb3680af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:20.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.733+0000 7fb367fff700 1 -- 172.21.15.46:0/1259143870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3580093c0 con 0x7fb36806a660 2024-01-26T23:42:20.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.733+0000 7fb36d18b700 1 --2- 172.21.15.46:0/1259143870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb36806add0 0x7fb3680af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:20.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.733+0000 7fb367fff700 1 --2- 172.21.15.46:0/1259143870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36806a660 0x7fb36814d790 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb35000c590 tx=0x7fb350007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:20.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.733+0000 7fb364ff9700 1 -- 172.21.15.46:0/1259143870 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3500105c0 con 0x7fb36806a660 2024-01-26T23:42:20.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.733+0000 7fb364ff9700 1 -- 172.21.15.46:0/1259143870 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb350010720 con 0x7fb36806a660 2024-01-26T23:42:20.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.733+0000 7fb36ebee700 1 -- 172.21.15.46:0/1259143870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36816a460 con 0x7fb36806a660 2024-01-26T23:42:20.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.733+0000 7fb364ff9700 1 -- 172.21.15.46:0/1259143870 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb350010ae0 con 0x7fb36806a660 2024-01-26T23:42:20.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.733+0000 7fb36ebee700 1 -- 172.21.15.46:0/1259143870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36814dcd0 con 0x7fb36806a660 2024-01-26T23:42:20.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.734+0000 7fb36ebee700 1 -- 172.21.15.46:0/1259143870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36804c710 con 0x7fb36806a660 2024-01-26T23:42:20.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.735+0000 7fb364ff9700 1 -- 172.21.15.46:0/1259143870 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fb35001b3b0 con 0x7fb36806a660 2024-01-26T23:42:20.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.735+0000 7fb364ff9700 1 --2- 172.21.15.46:0/1259143870 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb348067130 0x7fb3480695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:20.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.735+0000 7fb364ff9700 1 -- 172.21.15.46:0/1259143870 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fb3500842b0 con 0x7fb36806a660 2024-01-26T23:42:20.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.735+0000 7fb36c98a700 1 --2- 172.21.15.46:0/1259143870 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb348067130 0x7fb3480695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:20.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.736+0000 7fb36c98a700 1 --2- 172.21.15.46:0/1259143870 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb348067130 0x7fb3480695e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb358009850 tx=0x7fb358016000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:20.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.739+0000 7fb364ff9700 1 -- 172.21.15.46:0/1259143870 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb350051040 con 0x7fb36806a660 2024-01-26T23:42:20.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:20.973+0000 7fb36ebee700 1 -- 172.21.15.46:0/1259143870 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb36806edf0 con 0x7fb348067130 2024-01-26T23:42:21.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.033+0000 7fb364ff9700 1 -- 172.21.15.46:0/1259143870 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fb36806edf0 con 0x7fb348067130 2024-01-26T23:42:21.034 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2024-01-26T23:42:21.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.035+0000 7fb36ebee700 1 -- 172.21.15.46:0/1259143870 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb348067130 msgr2=0x7fb3480695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:21.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.035+0000 7fb36ebee700 1 --2- 172.21.15.46:0/1259143870 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb348067130 0x7fb3480695e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb358009850 tx=0x7fb358016000 comp rx=0 tx=0).stop 2024-01-26T23:42:21.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.036+0000 7fb36ebee700 1 -- 172.21.15.46:0/1259143870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36806a660 msgr2=0x7fb36814d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:21.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.036+0000 7fb36ebee700 1 --2- 172.21.15.46:0/1259143870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36806a660 0x7fb36814d790 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb35000c590 tx=0x7fb350007d90 comp rx=0 tx=0).stop 2024-01-26T23:42:21.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.036+0000 7fb36ebee700 1 -- 172.21.15.46:0/1259143870 shutdown_connections 2024-01-26T23:42:21.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.036+0000 7fb36ebee700 1 --2- 172.21.15.46:0/1259143870 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb348067130 0x7fb3480695e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.036+0000 7fb36ebee700 1 --2- 172.21.15.46:0/1259143870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb36806add0 0x7fb3680af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.036+0000 7fb36ebee700 1 --2- 172.21.15.46:0/1259143870 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb368068b00 0x7fb368169890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.036+0000 7fb36ebee700 1 --2- 172.21.15.46:0/1259143870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36806a660 0x7fb36814d790 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.036+0000 7fb36ebee700 1 -- 172.21.15.46:0/1259143870 >> 172.21.15.46:0/1259143870 conn(0x7fb3680ca100 msgr2=0x7fb3680c3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:21.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.036+0000 7fb36ebee700 1 -- 172.21.15.46:0/1259143870 shutdown_connections 2024-01-26T23:42:21.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.036+0000 7fb36ebee700 1 -- 172.21.15.46:0/1259143870 wait complete. 2024-01-26T23:42:21.050 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2024-01-26T23:42:21.051 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir3 1m 2024-01-26T23:42:21.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.189+0000 7fbbd20a2700 1 Processor -- start 2024-01-26T23:42:21.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.189+0000 7fbbd20a2700 1 -- start start 2024-01-26T23:42:21.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.190+0000 7fbbd20a2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbbcc159510 0x7fbbcc1598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:21.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.190+0000 7fbbd20a2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbbcc0ca690 0x7fbbcc0cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:21.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.190+0000 7fbbd20a2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbcc069260 con 0x7fbbcc0ca220 2024-01-26T23:42:21.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.190+0000 7fbbd20a2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbcc15d670 con 0x7fbbcc0ca690 2024-01-26T23:42:21.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.190+0000 7fbbd20a2700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbcc0c9750 con 0x7fbbcc159510 2024-01-26T23:42:21.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.190+0000 7fbbcbfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbbcc0ca690 0x7fbbcc0cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:21.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.190+0000 7fbbcbfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbbcc0ca690 0x7fbbcc0cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51294/0 (socket says 172.21.15.46:51294) 2024-01-26T23:42:21.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.190+0000 7fbbcbfff700 1 -- 172.21.15.46:0/3872909491 learned_addr learned my addr 172.21.15.46:0/3872909491 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:21.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.190+0000 7fbbcaffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbbcc0ca220 0x7fbbcc0c8be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34800/0 (socket says 172.21.15.46:34800) 2024-01-26T23:42:21.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.190+0000 7fbbcb7fe700 1 --2- 172.21.15.46:0/3872909491 >> v2:172.21.15.106:3300/0 conn(0x7fbbcc159510 0x7fbbcc1598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:21.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.191+0000 7fbbcbfff700 1 -- 172.21.15.46:0/3872909491 >> v2:172.21.15.106:3300/0 conn(0x7fbbcc159510 msgr2=0x7fbbcc1598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:21.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.191+0000 7fbbcbfff700 1 --2- 172.21.15.46:0/3872909491 >> v2:172.21.15.106:3300/0 conn(0x7fbbcc159510 0x7fbbcc1598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.191+0000 7fbbcbfff700 1 -- 172.21.15.46:0/3872909491 >> v1:172.21.15.46:6789/0 conn(0x7fbbcc0ca220 legacy=0x7fbbcc0c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:21.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.191+0000 7fbbcbfff700 1 -- 172.21.15.46:0/3872909491 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbcc0c9a30 con 0x7fbbcc0ca690 2024-01-26T23:42:21.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.191+0000 7fbbca7fc700 1 -- 172.21.15.46:0/3872909491 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539989933 0 0) 0x7fbbcc069260 con 0x7fbbcc0ca220 2024-01-26T23:42:21.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.191+0000 7fbbcbfff700 1 --2- 172.21.15.46:0/3872909491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc0ca690 0x7fbbcc0cc380 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc06f760 tx=0x7fbbc4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc336b13f6e7a3e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:21.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.191+0000 7fbbca7fc700 1 -- 172.21.15.46:0/3872909491 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc400e070 con 0x7fbbcc0ca690 2024-01-26T23:42:21.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.191+0000 7fbbca7fc700 1 -- 172.21.15.46:0/3872909491 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc400ba20 con 0x7fbbcc0ca690 2024-01-26T23:42:21.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.191+0000 7fbbca7fc700 1 -- 172.21.15.46:0/3872909491 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc40092e0 con 0x7fbbcc0ca690 2024-01-26T23:42:21.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.192+0000 7fbbd20a2700 1 -- 172.21.15.46:0/3872909491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc0ca690 msgr2=0x7fbbcc0cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:21.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.192+0000 7fbbd20a2700 1 --2- 172.21.15.46:0/3872909491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc0ca690 0x7fbbcc0cc380 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc06f760 tx=0x7fbbc4009840 comp rx=0 tx=0).stop 2024-01-26T23:42:21.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.192+0000 7fbbd20a2700 1 -- 172.21.15.46:0/3872909491 shutdown_connections 2024-01-26T23:42:21.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.192+0000 7fbbd20a2700 1 --2- 172.21.15.46:0/3872909491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc0ca690 0x7fbbcc0cc380 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.192+0000 7fbbd20a2700 1 --2- 172.21.15.46:0/3872909491 >> v2:172.21.15.106:3300/0 conn(0x7fbbcc159510 0x7fbbcc1598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.192+0000 7fbbd20a2700 1 -- 172.21.15.46:0/3872909491 >> 172.21.15.46:0/3872909491 conn(0x7fbbcc06bd90 msgr2=0x7fbbcc06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:21.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.192+0000 7fbbd20a2700 1 -- 172.21.15.46:0/3872909491 shutdown_connections 2024-01-26T23:42:21.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.192+0000 7fbbd20a2700 1 -- 172.21.15.46:0/3872909491 wait complete. 2024-01-26T23:42:21.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.193+0000 7fbbd20a2700 1 Processor -- start 2024-01-26T23:42:21.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.193+0000 7fbbd20a2700 1 -- start start 2024-01-26T23:42:21.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.194+0000 7fbbd20a2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbbcc0ca220 0x7fbbcc158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:21.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.194+0000 7fbbd20a2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc159510 0x7fbbcc0c6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:21.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.194+0000 7fbbd20a2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbcc0c6b60 0x7fbbcc0b3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:21.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.194+0000 7fbbd20a2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc169370 con 0x7fbbcc159510 2024-01-26T23:42:21.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.194+0000 7fbbd20a2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc16a3a0 con 0x7fbbcc0c6b60 2024-01-26T23:42:21.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.194+0000 7fbbd20a2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc1695e0 con 0x7fbbcc0ca220 2024-01-26T23:42:21.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.194+0000 7fbbcaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc159510 0x7fbbcc0c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:21.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.194+0000 7fbbcaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc159510 0x7fbbcc0c6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51310/0 (socket says 172.21.15.46:51310) 2024-01-26T23:42:21.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.194+0000 7fbbcaffd700 1 -- 172.21.15.46:0/2158870801 learned_addr learned my addr 172.21.15.46:0/2158870801 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:21.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.194+0000 7fbbcb7fe700 1 --2- 172.21.15.46:0/2158870801 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbbcc0ca220 0x7fbbcc158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.194+0000 7fbbcbfff700 1 --2- 172.21.15.46:0/2158870801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbcc0c6b60 0x7fbbcc0b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.194+0000 7fbbcaffd700 1 -- 172.21.15.46:0/2158870801 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbbcc0ca220 msgr2=0x7fbbcc158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.194+0000 7fbbcaffd700 1 --2- 172.21.15.46:0/2158870801 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbbcc0ca220 0x7fbbcc158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.194+0000 7fbbcaffd700 1 -- 172.21.15.46:0/2158870801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbcc0c6b60 msgr2=0x7fbbcc0b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.194+0000 7fbbcaffd700 1 --2- 172.21.15.46:0/2158870801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbcc0c6b60 0x7fbbcc0b3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.195+0000 7fbbcaffd700 1 -- 172.21.15.46:0/2158870801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc4009ae0 con 0x7fbbcc159510 2024-01-26T23:42:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.195+0000 7fbbcbfff700 1 --2- 172.21.15.46:0/2158870801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbcc0c6b60 0x7fbbcc0b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:42:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.195+0000 7fbbcaffd700 1 --2- 172.21.15.46:0/2158870801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc159510 0x7fbbcc0c6010 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbbc00011e0 tx=0x7fbbc0007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.195+0000 7fbbb37fe700 1 -- 172.21.15.46:0/2158870801 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0007300 con 0x7fbbcc159510 2024-01-26T23:42:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.195+0000 7fbbb37fe700 1 -- 172.21.15.46:0/2158870801 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc0007460 con 0x7fbbcc159510 2024-01-26T23:42:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.195+0000 7fbbd20a2700 1 -- 172.21.15.46:0/2158870801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbcc1698c0 con 0x7fbbcc159510 2024-01-26T23:42:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.195+0000 7fbbb37fe700 1 -- 172.21.15.46:0/2158870801 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0004830 con 0x7fbbcc159510 2024-01-26T23:42:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.195+0000 7fbbd20a2700 1 -- 172.21.15.46:0/2158870801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbcc169e80 con 0x7fbbcc159510 2024-01-26T23:42:21.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.196+0000 7fbbd20a2700 1 -- 172.21.15.46:0/2158870801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbcc16a0a0 con 0x7fbbcc159510 2024-01-26T23:42:21.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.197+0000 7fbbb37fe700 1 -- 172.21.15.46:0/2158870801 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fbbc0004be0 con 0x7fbbcc159510 2024-01-26T23:42:21.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.197+0000 7fbbb37fe700 1 --2- 172.21.15.46:0/2158870801 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbbb8067130 0x7fbbb80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:21.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.197+0000 7fbbb37fe700 1 -- 172.21.15.46:0/2158870801 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fbbc0084060 con 0x7fbbcc159510 2024-01-26T23:42:21.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.197+0000 7fbbcb7fe700 1 --2- 172.21.15.46:0/2158870801 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbbb8067130 0x7fbbb80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:21.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.198+0000 7fbbcb7fe700 1 --2- 172.21.15.46:0/2158870801 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbbb8067130 0x7fbbb80695e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009ca0 tx=0x7fbbbc008040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:21.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.201+0000 7fbbb37fe700 1 -- 172.21.15.46:0/2158870801 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbbc0050e70 con 0x7fbbcc159510 2024-01-26T23:42:21.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.435+0000 7fbbd20a2700 1 -- 172.21.15.46:0/2158870801 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbcc04c710 con 0x7fbbb8067130 2024-01-26T23:42:21.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.499+0000 7fbbb37fe700 1 -- 172.21.15.46:0/2158870801 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fbbcc04c710 con 0x7fbbb8067130 2024-01-26T23:42:21.500 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2024-01-26T23:42:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.501+0000 7fbbd20a2700 1 -- 172.21.15.46:0/2158870801 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbbb8067130 msgr2=0x7fbbb80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.501+0000 7fbbd20a2700 1 --2- 172.21.15.46:0/2158870801 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbbb8067130 0x7fbbb80695e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009ca0 tx=0x7fbbbc008040 comp rx=0 tx=0).stop 2024-01-26T23:42:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.501+0000 7fbbd20a2700 1 -- 172.21.15.46:0/2158870801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc159510 msgr2=0x7fbbcc0c6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.501+0000 7fbbd20a2700 1 --2- 172.21.15.46:0/2158870801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc159510 0x7fbbcc0c6010 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbbc00011e0 tx=0x7fbbc0007e10 comp rx=0 tx=0).stop 2024-01-26T23:42:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.501+0000 7fbbd20a2700 1 -- 172.21.15.46:0/2158870801 shutdown_connections 2024-01-26T23:42:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.501+0000 7fbbd20a2700 1 --2- 172.21.15.46:0/2158870801 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbbb8067130 0x7fbbb80695e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.501+0000 7fbbd20a2700 1 --2- 172.21.15.46:0/2158870801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbcc0c6b60 0x7fbbcc0b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.501+0000 7fbbd20a2700 1 --2- 172.21.15.46:0/2158870801 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbbcc0ca220 0x7fbbcc158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.501+0000 7fbbd20a2700 1 --2- 172.21.15.46:0/2158870801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc159510 0x7fbbcc0c6010 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.501+0000 7fbbd20a2700 1 -- 172.21.15.46:0/2158870801 >> 172.21.15.46:0/2158870801 conn(0x7fbbcc06bd90 msgr2=0x7fbbcc0c3900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.501+0000 7fbbd20a2700 1 -- 172.21.15.46:0/2158870801 shutdown_connections 2024-01-26T23:42:21.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.501+0000 7fbbd20a2700 1 -- 172.21.15.46:0/2158870801 wait complete. 2024-01-26T23:42:21.516 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2024-01-26T23:42:21.516 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir4 1m 2024-01-26T23:42:21.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.653+0000 7fafbb59d700 1 Processor -- start 2024-01-26T23:42:21.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.653+0000 7fafbb59d700 1 -- start start 2024-01-26T23:42:21.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.654+0000 7fafbb59d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fafb4159510 0x7fafb41598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:21.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.654+0000 7fafbb59d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fafb40ca220 0x7fafb40c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:21.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.654+0000 7fafbb59d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fafb40ca690 0x7fafb40c9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:21.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.654+0000 7fafbb59d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb4069260 con 0x7fafb40ca220 2024-01-26T23:42:21.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.654+0000 7fafbb59d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb415d670 con 0x7fafb40ca690 2024-01-26T23:42:21.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.654+0000 7fafbb59d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb40c9750 con 0x7fafb4159510 2024-01-26T23:42:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.654+0000 7fafb8b38700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fafb40ca220 0x7fafb40c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.654+0000 7fafb8b38700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fafb40ca220 0x7fafb40c8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51312/0 (socket says 172.21.15.46:51312) 2024-01-26T23:42:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.654+0000 7fafb8b38700 1 -- 172.21.15.46:0/4134884528 learned_addr learned my addr 172.21.15.46:0/4134884528 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.655+0000 7fafb9b3a700 1 --2- 172.21.15.46:0/4134884528 >> v2:172.21.15.106:3300/0 conn(0x7fafb40ca690 0x7fafb40c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.655+0000 7fafb9339700 1 --2- 172.21.15.46:0/4134884528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fafb4159510 0x7fafb41598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.655+0000 7fafb8b38700 1 -- 172.21.15.46:0/4134884528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fafb4159510 msgr2=0x7fafb41598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.655+0000 7fafb8b38700 1 --2- 172.21.15.46:0/4134884528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fafb4159510 0x7fafb41598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.655+0000 7fafb8b38700 1 -- 172.21.15.46:0/4134884528 >> v2:172.21.15.106:3300/0 conn(0x7fafb40ca690 msgr2=0x7fafb40c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.655+0000 7fafb8b38700 1 --2- 172.21.15.46:0/4134884528 >> v2:172.21.15.106:3300/0 conn(0x7fafb40ca690 0x7fafb40c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.655+0000 7fafb8b38700 1 -- 172.21.15.46:0/4134884528 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb40c9a30 con 0x7fafb40ca220 2024-01-26T23:42:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.655+0000 7fafb8b38700 1 --2- 172.21.15.46:0/4134884528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafb40ca220 0x7fafb40c8be0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fafb4159e20 tx=0x7fafa8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3d4f121d30ba1d server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.655+0000 7fafa7fff700 1 -- 172.21.15.46:0/4134884528 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa800b860 con 0x7fafb40ca220 2024-01-26T23:42:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.655+0000 7fafa7fff700 1 -- 172.21.15.46:0/4134884528 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa800b9c0 con 0x7fafb40ca220 2024-01-26T23:42:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.655+0000 7fafa7fff700 1 -- 172.21.15.46:0/4134884528 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa800bd60 con 0x7fafb40ca220 2024-01-26T23:42:21.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.656+0000 7fafbb59d700 1 -- 172.21.15.46:0/4134884528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafb40ca220 msgr2=0x7fafb40c8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:21.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.656+0000 7fafbb59d700 1 --2- 172.21.15.46:0/4134884528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafb40ca220 0x7fafb40c8be0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fafb4159e20 tx=0x7fafa8009250 comp rx=0 tx=0).stop 2024-01-26T23:42:21.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.656+0000 7fafbb59d700 1 -- 172.21.15.46:0/4134884528 shutdown_connections 2024-01-26T23:42:21.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.656+0000 7fafbb59d700 1 --2- 172.21.15.46:0/4134884528 >> v2:172.21.15.106:3300/0 conn(0x7fafb40ca690 0x7fafb40c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.656+0000 7fafbb59d700 1 --2- 172.21.15.46:0/4134884528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fafb4159510 0x7fafb41598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.656+0000 7fafbb59d700 1 --2- 172.21.15.46:0/4134884528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafb40ca220 0x7fafb40c8be0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.656+0000 7fafbb59d700 1 -- 172.21.15.46:0/4134884528 >> 172.21.15.46:0/4134884528 conn(0x7fafb406bd90 msgr2=0x7fafb406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:21.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.656+0000 7fafbb59d700 1 -- 172.21.15.46:0/4134884528 shutdown_connections 2024-01-26T23:42:21.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.656+0000 7fafbb59d700 1 -- 172.21.15.46:0/4134884528 wait complete. 2024-01-26T23:42:21.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.657+0000 7fafbb59d700 1 Processor -- start 2024-01-26T23:42:21.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.657+0000 7fafbb59d700 1 -- start start 2024-01-26T23:42:21.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.658+0000 7fafbb59d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafb40ca220 0x7fafb415be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:21.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.658+0000 7fafbb59d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafb4159510 0x7fafb4144f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:21.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.658+0000 7fafbb59d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fafb4159e50 0x7fafb4153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:21.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.658+0000 7fafbb59d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb415bc00 con 0x7fafb40ca220 2024-01-26T23:42:21.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.658+0000 7fafbb59d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb415c3b0 con 0x7fafb4159510 2024-01-26T23:42:21.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.658+0000 7fafbb59d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb415af40 con 0x7fafb4159e50 2024-01-26T23:42:21.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.658+0000 7fafb9339700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafb40ca220 0x7fafb415be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:21.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.658+0000 7fafb9339700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafb40ca220 0x7fafb415be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51320/0 (socket says 172.21.15.46:51320) 2024-01-26T23:42:21.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.658+0000 7fafb9339700 1 -- 172.21.15.46:0/3665839083 learned_addr learned my addr 172.21.15.46:0/3665839083 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:21.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.658+0000 7fafb9b3a700 1 --2- 172.21.15.46:0/3665839083 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fafb4159e50 0x7fafb4153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:21.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.658+0000 7fafb8b38700 1 --2- 172.21.15.46:0/3665839083 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafb4159510 0x7fafb4144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:21.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.658+0000 7fafb9339700 1 -- 172.21.15.46:0/3665839083 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fafb4159e50 msgr2=0x7fafb4153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:21.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.659+0000 7fafb9339700 1 --2- 172.21.15.46:0/3665839083 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fafb4159e50 0x7fafb4153ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.659+0000 7fafb9339700 1 -- 172.21.15.46:0/3665839083 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafb4159510 msgr2=0x7fafb4144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:21.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.659+0000 7fafb9339700 1 --2- 172.21.15.46:0/3665839083 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafb4159510 0x7fafb4144f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.659+0000 7fafb9339700 1 -- 172.21.15.46:0/3665839083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa80094f0 con 0x7fafb40ca220 2024-01-26T23:42:21.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.659+0000 7fafb8b38700 1 --2- 172.21.15.46:0/3665839083 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafb4159510 0x7fafb4144f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:21.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.659+0000 7fafb9339700 1 --2- 172.21.15.46:0/3665839083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafb40ca220 0x7fafb415be70 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fafb000d860 tx=0x7fafb0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:21.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.659+0000 7fafa57fa700 1 -- 172.21.15.46:0/3665839083 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb00125f0 con 0x7fafb40ca220 2024-01-26T23:42:21.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.659+0000 7fafa57fa700 1 -- 172.21.15.46:0/3665839083 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb0012750 con 0x7fafb40ca220 2024-01-26T23:42:21.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.659+0000 7fafbb59d700 1 -- 172.21.15.46:0/3665839083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb415a330 con 0x7fafb40ca220 2024-01-26T23:42:21.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.659+0000 7fafa57fa700 1 -- 172.21.15.46:0/3665839083 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb0012b10 con 0x7fafb40ca220 2024-01-26T23:42:21.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.659+0000 7fafbb59d700 1 -- 172.21.15.46:0/3665839083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb415a8f0 con 0x7fafb40ca220 2024-01-26T23:42:21.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.660+0000 7fafbb59d700 1 -- 172.21.15.46:0/3665839083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb404c710 con 0x7fafb40ca220 2024-01-26T23:42:21.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.661+0000 7fafa57fa700 1 -- 172.21.15.46:0/3665839083 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fafb001e460 con 0x7fafb40ca220 2024-01-26T23:42:21.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.661+0000 7fafa57fa700 1 --2- 172.21.15.46:0/3665839083 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faf98067130 0x7faf980695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:21.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.661+0000 7fafa57fa700 1 -- 172.21.15.46:0/3665839083 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fafb0085060 con 0x7fafb40ca220 2024-01-26T23:42:21.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.661+0000 7fafb8b38700 1 --2- 172.21.15.46:0/3665839083 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faf98067130 0x7faf980695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:21.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.662+0000 7fafb8b38700 1 --2- 172.21.15.46:0/3665839083 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faf98067130 0x7faf980695e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fafb4159e20 tx=0x7fafa8017000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:21.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.665+0000 7fafa57fa700 1 -- 172.21.15.46:0/3665839083 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fafb0051df0 con 0x7fafb40ca220 2024-01-26T23:42:21.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.897+0000 7fafbb59d700 1 -- 172.21.15.46:0/3665839083 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fafb40c9d10 con 0x7faf98067130 2024-01-26T23:42:21.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.961+0000 7fafa57fa700 1 -- 172.21.15.46:0/3665839083 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fafb40c9d10 con 0x7faf98067130 2024-01-26T23:42:21.961 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2024-01-26T23:42:21.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.964+0000 7fafbb59d700 1 -- 172.21.15.46:0/3665839083 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faf98067130 msgr2=0x7faf980695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:21.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.964+0000 7fafbb59d700 1 --2- 172.21.15.46:0/3665839083 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faf98067130 0x7faf980695e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fafb4159e20 tx=0x7fafa8017000 comp rx=0 tx=0).stop 2024-01-26T23:42:21.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.964+0000 7fafbb59d700 1 -- 172.21.15.46:0/3665839083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafb40ca220 msgr2=0x7fafb415be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.964+0000 7fafbb59d700 1 --2- 172.21.15.46:0/3665839083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafb40ca220 0x7fafb415be70 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fafb000d860 tx=0x7fafb0008040 comp rx=0 tx=0).stop 2024-01-26T23:42:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.964+0000 7fafbb59d700 1 -- 172.21.15.46:0/3665839083 shutdown_connections 2024-01-26T23:42:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.964+0000 7fafbb59d700 1 --2- 172.21.15.46:0/3665839083 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faf98067130 0x7faf980695e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.964+0000 7fafbb59d700 1 --2- 172.21.15.46:0/3665839083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafb40ca220 0x7fafb415be70 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.964+0000 7fafbb59d700 1 --2- 172.21.15.46:0/3665839083 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fafb4159e50 0x7fafb4153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.964+0000 7fafbb59d700 1 --2- 172.21.15.46:0/3665839083 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafb4159510 0x7fafb4144f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.964+0000 7fafbb59d700 1 -- 172.21.15.46:0/3665839083 >> 172.21.15.46:0/3665839083 conn(0x7fafb406bd90 msgr2=0x7fafb4158170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.964+0000 7fafbb59d700 1 -- 172.21.15.46:0/3665839083 shutdown_connections 2024-01-26T23:42:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:21.964+0000 7fafbb59d700 1 -- 172.21.15.46:0/3665839083 wait complete. 2024-01-26T23:42:21.978 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2024-01-26T23:42:21.978 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir5 1m 2024-01-26T23:42:22.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.115+0000 7f2785784700 1 Processor -- start 2024-01-26T23:42:22.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.115+0000 7f2785784700 1 -- start start 2024-01-26T23:42:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.116+0000 7f2785784700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f27801569b0 0x7f27800a98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.116+0000 7f2785784700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278006ba40 con 0x7f2780069b20 2024-01-26T23:42:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.116+0000 7f2785784700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278006e470 con 0x7f27801565e0 2024-01-26T23:42:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.116+0000 7f2785784700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27800acbb0 con 0x7f27801569b0 2024-01-26T23:42:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.116+0000 7f277f7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2780069b20 0x7f27800aa2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34802/0 (socket says 172.21.15.46:34802) 2024-01-26T23:42:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.116+0000 7f277f7fe700 1 -- 172.21.15.46:0/3284524802 learned_addr learned my addr 172.21.15.46:0/3284524802 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.116+0000 7f277e7fc700 1 --2- 172.21.15.46:0/3284524802 >> v2:172.21.15.106:3300/0 conn(0x7f27801569b0 0x7f27800a98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.116+0000 7f277dffb700 1 -- 172.21.15.46:0/3284524802 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778871459 0 0) 0x7f278006ba40 con 0x7f2780069b20 2024-01-26T23:42:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.116+0000 7f277dffb700 1 -- 172.21.15.46:0/3284524802 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2768003570 con 0x7f2780069b20 2024-01-26T23:42:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.116+0000 7f277dffb700 1 -- 172.21.15.46:0/3284524802 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2498977685 0 0) 0x7f2768003570 con 0x7f2780069b20 2024-01-26T23:42:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.116+0000 7f277dffb700 1 -- 172.21.15.46:0/3284524802 >> v2:172.21.15.106:3300/0 conn(0x7f27801569b0 msgr2=0x7f27800a98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.116+0000 7f277dffb700 1 --2- 172.21.15.46:0/3284524802 >> v2:172.21.15.106:3300/0 conn(0x7f27801569b0 0x7f27800a98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.116+0000 7f277dffb700 1 -- 172.21.15.46:0/3284524802 >> v1:172.21.15.106:6789/0 conn(0x7f27801565e0 legacy=0x7f27800ac4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.116+0000 7f277dffb700 1 -- 172.21.15.46:0/3284524802 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278006db60 con 0x7f2780069b20 2024-01-26T23:42:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.116+0000 7f277dffb700 1 -- 172.21.15.46:0/3284524802 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f27780017f0 con 0x7f2780069b20 2024-01-26T23:42:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.116+0000 7f277dffb700 1 -- 172.21.15.46:0/3284524802 >> v1:172.21.15.46:6789/0 conn(0x7f2780069b20 legacy=0x7f27800aa2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.116+0000 7f277dffb700 1 --2- 172.21.15.46:0/3284524802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2768003a60 0x7f2768003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277dffb700 1 --2- 172.21.15.46:0/3284524802 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2768004810 0x7f2768004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277dffb700 1 --2- 172.21.15.46:0/3284524802 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27680051c0 0x7f276800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277dffb700 1 -- 172.21.15.46:0/3284524802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27800acbb0 con 0x7f2768003a60 2024-01-26T23:42:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277dffb700 1 -- 172.21.15.46:0/3284524802 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276800c0e0 con 0x7f27680051c0 2024-01-26T23:42:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277dffb700 1 -- 172.21.15.46:0/3284524802 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f276800c2a0 con 0x7f2768004810 2024-01-26T23:42:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277effd700 1 --2- 172.21.15.46:0/3284524802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2768003a60 0x7f2768003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277f7fe700 1 --2- 172.21.15.46:0/3284524802 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27680051c0 0x7f276800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277e7fc700 1 --2- 172.21.15.46:0/3284524802 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2768004810 0x7f2768004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277effd700 1 -- 172.21.15.46:0/3284524802 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2768004810 msgr2=0x7f2768004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277effd700 1 --2- 172.21.15.46:0/3284524802 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2768004810 0x7f2768004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277effd700 1 -- 172.21.15.46:0/3284524802 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27680051c0 msgr2=0x7f276800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277effd700 1 --2- 172.21.15.46:0/3284524802 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27680051c0 0x7f276800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277f7fe700 1 -- 172.21.15.46:0/3284524802 reap_dead start 2024-01-26T23:42:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277effd700 1 -- 172.21.15.46:0/3284524802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2768003570 con 0x7f2768003a60 2024-01-26T23:42:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277e7fc700 1 --2- 172.21.15.46:0/3284524802 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2768004810 0x7f2768004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277effd700 1 --2- 172.21.15.46:0/3284524802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2768003a60 0x7f2768003e30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f27700019c0 tx=0x7f2770007d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96860c581b4db8e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277dffb700 1 -- 172.21.15.46:0/3284524802 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770001330 con 0x7f2768003a60 2024-01-26T23:42:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277dffb700 1 -- 172.21.15.46:0/3284524802 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277000c5d0 con 0x7f2768003a60 2024-01-26T23:42:22.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.117+0000 7f277dffb700 1 -- 172.21.15.46:0/3284524802 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277000c970 con 0x7f2768003a60 2024-01-26T23:42:22.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.118+0000 7f2785784700 1 -- 172.21.15.46:0/3284524802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2768003a60 msgr2=0x7f2768003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:22.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.118+0000 7f2785784700 1 --2- 172.21.15.46:0/3284524802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2768003a60 0x7f2768003e30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f27700019c0 tx=0x7f2770007d60 comp rx=0 tx=0).stop 2024-01-26T23:42:22.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.118+0000 7f2785784700 1 -- 172.21.15.46:0/3284524802 shutdown_connections 2024-01-26T23:42:22.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.118+0000 7f2785784700 1 --2- 172.21.15.46:0/3284524802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2768003a60 0x7f2768003e30 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.118+0000 7f2785784700 1 -- 172.21.15.46:0/3284524802 >> 172.21.15.46:0/3284524802 conn(0x7f2780067930 msgr2=0x7f2780067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:22.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.118+0000 7f2785784700 1 -- 172.21.15.46:0/3284524802 shutdown_connections 2024-01-26T23:42:22.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.118+0000 7f2785784700 1 -- 172.21.15.46:0/3284524802 wait complete. 2024-01-26T23:42:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.119+0000 7f2785784700 1 Processor -- start 2024-01-26T23:42:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.119+0000 7f2785784700 1 -- start start 2024-01-26T23:42:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.120+0000 7f2785784700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2768003a60 0x7f2780168b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.120+0000 7f2785784700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2780152bf0 0x7f27800969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.120+0000 7f2785784700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2780096f10 0x7f2780097380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.120+0000 7f2785784700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2780157a10 con 0x7f2780152bf0 2024-01-26T23:42:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.120+0000 7f2785784700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27801558b0 con 0x7f2780096f10 2024-01-26T23:42:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.120+0000 7f2785784700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2780157c20 con 0x7f2768003a60 2024-01-26T23:42:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.120+0000 7f277e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2780152bf0 0x7f27800969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.120+0000 7f277e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2780152bf0 0x7f27800969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51332/0 (socket says 172.21.15.46:51332) 2024-01-26T23:42:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.120+0000 7f277e7fc700 1 -- 172.21.15.46:0/2160307896 learned_addr learned my addr 172.21.15.46:0/2160307896 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.120+0000 7f277effd700 1 --2- 172.21.15.46:0/2160307896 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2768003a60 0x7f2780168b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.120+0000 7f277f7fe700 1 --2- 172.21.15.46:0/2160307896 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2780096f10 0x7f2780097380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.121+0000 7f277e7fc700 1 -- 172.21.15.46:0/2160307896 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2768003a60 msgr2=0x7f2780168b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.121+0000 7f277e7fc700 1 --2- 172.21.15.46:0/2160307896 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2768003a60 0x7f2780168b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.121+0000 7f277e7fc700 1 -- 172.21.15.46:0/2160307896 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2780096f10 msgr2=0x7f2780097380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.121+0000 7f277e7fc700 1 --2- 172.21.15.46:0/2160307896 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2780096f10 0x7f2780097380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.121+0000 7f277e7fc700 1 -- 172.21.15.46:0/2160307896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2780157f00 con 0x7f2780152bf0 2024-01-26T23:42:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.121+0000 7f277e7fc700 1 --2- 172.21.15.46:0/2160307896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2780152bf0 0x7f27800969d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2774011710 tx=0x7f2774008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.121+0000 7f2766ffd700 1 -- 172.21.15.46:0/2160307896 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2774015670 con 0x7f2780152bf0 2024-01-26T23:42:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.121+0000 7f2766ffd700 1 -- 172.21.15.46:0/2160307896 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27740157d0 con 0x7f2780152bf0 2024-01-26T23:42:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.121+0000 7f2785784700 1 -- 172.21.15.46:0/2160307896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27801581e0 con 0x7f2780152bf0 2024-01-26T23:42:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.121+0000 7f2766ffd700 1 -- 172.21.15.46:0/2160307896 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2774015b70 con 0x7f2780152bf0 2024-01-26T23:42:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.121+0000 7f2785784700 1 -- 172.21.15.46:0/2160307896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2780154420 con 0x7f2780152bf0 2024-01-26T23:42:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.122+0000 7f2785784700 1 -- 172.21.15.46:0/2160307896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2780158400 con 0x7f2780152bf0 2024-01-26T23:42:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.123+0000 7f2766ffd700 1 -- 172.21.15.46:0/2160307896 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f2774013410 con 0x7f2780152bf0 2024-01-26T23:42:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.123+0000 7f2766ffd700 1 --2- 172.21.15.46:0/2160307896 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f276c067160 0x7f276c069610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.123+0000 7f2766ffd700 1 -- 172.21.15.46:0/2160307896 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f2774084df0 con 0x7f2780152bf0 2024-01-26T23:42:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.123+0000 7f277effd700 1 --2- 172.21.15.46:0/2160307896 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f276c067160 0x7f276c069610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:22.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.124+0000 7f277effd700 1 --2- 172.21.15.46:0/2160307896 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f276c067160 0x7f276c069610 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2770001020 tx=0x7f2770015000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:22.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.128+0000 7f2766ffd700 1 -- 172.21.15.46:0/2160307896 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2774051ae0 con 0x7f2780152bf0 2024-01-26T23:42:22.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.360+0000 7f2785784700 1 -- 172.21.15.46:0/2160307896 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f278004c710 con 0x7f276c067160 2024-01-26T23:42:22.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.422+0000 7f2766ffd700 1 -- 172.21.15.46:0/2160307896 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f278004c710 con 0x7f276c067160 2024-01-26T23:42:22.422 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2024-01-26T23:42:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.424+0000 7f2785784700 1 -- 172.21.15.46:0/2160307896 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f276c067160 msgr2=0x7f276c069610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.424+0000 7f2785784700 1 --2- 172.21.15.46:0/2160307896 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f276c067160 0x7f276c069610 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2770001020 tx=0x7f2770015000 comp rx=0 tx=0).stop 2024-01-26T23:42:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.424+0000 7f2785784700 1 -- 172.21.15.46:0/2160307896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2780152bf0 msgr2=0x7f27800969d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.424+0000 7f2785784700 1 --2- 172.21.15.46:0/2160307896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2780152bf0 0x7f27800969d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2774011710 tx=0x7f2774008040 comp rx=0 tx=0).stop 2024-01-26T23:42:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.424+0000 7f2785784700 1 -- 172.21.15.46:0/2160307896 shutdown_connections 2024-01-26T23:42:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.424+0000 7f2785784700 1 --2- 172.21.15.46:0/2160307896 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f276c067160 0x7f276c069610 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.424+0000 7f2785784700 1 --2- 172.21.15.46:0/2160307896 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2780096f10 0x7f2780097380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.424+0000 7f2785784700 1 --2- 172.21.15.46:0/2160307896 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2768003a60 0x7f2780168b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.424+0000 7f2785784700 1 --2- 172.21.15.46:0/2160307896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2780152bf0 0x7f27800969d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.424+0000 7f2785784700 1 -- 172.21.15.46:0/2160307896 >> 172.21.15.46:0/2160307896 conn(0x7f2780067930 msgr2=0x7f2780069fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:22.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.424+0000 7f2785784700 1 -- 172.21.15.46:0/2160307896 shutdown_connections 2024-01-26T23:42:22.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.424+0000 7f2785784700 1 -- 172.21.15.46:0/2160307896 wait complete. 2024-01-26T23:42:22.439 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2024-01-26T23:42:22.439 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir6 1m 2024-01-26T23:42:22.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.575+0000 7fc3eaddb700 1 Processor -- start 2024-01-26T23:42:22.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.575+0000 7fc3eaddb700 1 -- start start 2024-01-26T23:42:22.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.576+0000 7fc3eaddb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3e40cb640 0x7fc3e40c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:22.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.576+0000 7fc3eaddb700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e40ccc00 con 0x7fc3e4141200 2024-01-26T23:42:22.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.576+0000 7fc3eaddb700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e40cc310 con 0x7fc3e40c8da0 2024-01-26T23:42:22.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.576+0000 7fc3eaddb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e40cbbf0 con 0x7fc3e40cb640 2024-01-26T23:42:22.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.576+0000 7fc3e3fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc3e4141200 0x7fc3e40caf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34818/0 (socket says 172.21.15.46:34818) 2024-01-26T23:42:22.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.576+0000 7fc3e3fff700 1 -- 172.21.15.46:0/2981658229 learned_addr learned my addr 172.21.15.46:0/2981658229 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:22.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.576+0000 7fc3e9378700 1 --2- 172.21.15.46:0/2981658229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3e40cb640 0x7fc3e40c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:22.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.576+0000 7fc3e37fe700 1 -- 172.21.15.46:0/2981658229 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617494837 0 0) 0x7fc3e40ccc00 con 0x7fc3e4141200 2024-01-26T23:42:22.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.576+0000 7fc3e37fe700 1 -- 172.21.15.46:0/2981658229 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d4003570 con 0x7fc3e4141200 2024-01-26T23:42:22.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.576+0000 7fc3e37fe700 1 -- 172.21.15.46:0/2981658229 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4198051804 0 0) 0x7fc3d4003570 con 0x7fc3e4141200 2024-01-26T23:42:22.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.576+0000 7fc3e37fe700 1 -- 172.21.15.46:0/2981658229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3e40cb640 msgr2=0x7fc3e40c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:22.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.576+0000 7fc3e37fe700 1 --2- 172.21.15.46:0/2981658229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3e40cb640 0x7fc3e40c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.576+0000 7fc3e37fe700 1 -- 172.21.15.46:0/2981658229 >> v1:172.21.15.106:6789/0 conn(0x7fc3e40c8da0 legacy=0x7fc3e4140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:22.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.576+0000 7fc3e37fe700 1 -- 172.21.15.46:0/2981658229 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e40cc5b0 con 0x7fc3e4141200 2024-01-26T23:42:22.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.576+0000 7fc3e37fe700 1 -- 172.21.15.46:0/2981658229 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fc3d0001470 con 0x7fc3e4141200 2024-01-26T23:42:22.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.576+0000 7fc3e37fe700 1 -- 172.21.15.46:0/2981658229 >> v1:172.21.15.46:6789/0 conn(0x7fc3e4141200 legacy=0x7fc3e40caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:22.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e37fe700 1 --2- 172.21.15.46:0/2981658229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d4003a90 0x7fc3d4003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:22.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e37fe700 1 --2- 172.21.15.46:0/2981658229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3d4004810 0x7fc3d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:22.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e37fe700 1 --2- 172.21.15.46:0/2981658229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3d40051c0 0x7fc3d400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:22.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e37fe700 1 -- 172.21.15.46:0/2981658229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e40cbbf0 con 0x7fc3d4003a90 2024-01-26T23:42:22.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e37fe700 1 -- 172.21.15.46:0/2981658229 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d400c050 con 0x7fc3d4004810 2024-01-26T23:42:22.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e37fe700 1 -- 172.21.15.46:0/2981658229 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d400c210 con 0x7fc3d40051c0 2024-01-26T23:42:22.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e9378700 1 --2- 172.21.15.46:0/2981658229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3e40cb640 0x7fc3e40c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:22.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e8b77700 1 --2- 172.21.15.46:0/2981658229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d4003a90 0x7fc3d4003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:22.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e3fff700 1 --2- 172.21.15.46:0/2981658229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3d4004810 0x7fc3d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:22.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e9378700 1 --2- 172.21.15.46:0/2981658229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3d40051c0 0x7fc3d400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:22.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e8b77700 1 -- 172.21.15.46:0/2981658229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3d40051c0 msgr2=0x7fc3d400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:22.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e8b77700 1 --2- 172.21.15.46:0/2981658229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3d40051c0 0x7fc3d400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e8b77700 1 -- 172.21.15.46:0/2981658229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3d4004810 msgr2=0x7fc3d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:22.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e8b77700 1 --2- 172.21.15.46:0/2981658229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3d4004810 0x7fc3d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e8b77700 1 -- 172.21.15.46:0/2981658229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d4003570 con 0x7fc3d4003a90 2024-01-26T23:42:22.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e8b77700 1 --2- 172.21.15.46:0/2981658229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d4003a90 0x7fc3d4003e60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc3d80019a0 tx=0x7fc3d8007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84a6aa9da1a9d5e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:22.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e37fe700 1 -- 172.21.15.46:0/2981658229 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d800c5d0 con 0x7fc3d4003a90 2024-01-26T23:42:22.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e37fe700 1 -- 172.21.15.46:0/2981658229 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d800c730 con 0x7fc3d4003a90 2024-01-26T23:42:22.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.577+0000 7fc3e37fe700 1 -- 172.21.15.46:0/2981658229 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d800cad0 con 0x7fc3d4003a90 2024-01-26T23:42:22.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.578+0000 7fc3eaddb700 1 -- 172.21.15.46:0/2981658229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d4003a90 msgr2=0x7fc3d4003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:22.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.578+0000 7fc3eaddb700 1 --2- 172.21.15.46:0/2981658229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d4003a90 0x7fc3d4003e60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc3d80019a0 tx=0x7fc3d8007d10 comp rx=0 tx=0).stop 2024-01-26T23:42:22.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.578+0000 7fc3e9378700 1 -- 172.21.15.46:0/2981658229 reap_dead start 2024-01-26T23:42:22.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.578+0000 7fc3eaddb700 1 -- 172.21.15.46:0/2981658229 shutdown_connections 2024-01-26T23:42:22.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.578+0000 7fc3eaddb700 1 -- 172.21.15.46:0/2981658229 >> 172.21.15.46:0/2981658229 conn(0x7fc3e406bd90 msgr2=0x7fc3e406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:22.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.578+0000 7fc3eaddb700 1 -- 172.21.15.46:0/2981658229 shutdown_connections 2024-01-26T23:42:22.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.578+0000 7fc3eaddb700 1 -- 172.21.15.46:0/2981658229 wait complete. 2024-01-26T23:42:22.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.579+0000 7fc3eaddb700 1 Processor -- start 2024-01-26T23:42:22.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.579+0000 7fc3eaddb700 1 -- start start 2024-01-26T23:42:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.580+0000 7fc3eaddb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3d4003a90 0x7fc3e413c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.580+0000 7fc3eaddb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3e40c3cf0 0x7fc3e40c4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.580+0000 7fc3eaddb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e40c5650 0x7fc3e40c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.580+0000 7fc3eaddb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e40b0ea0 con 0x7fc3e40c5650 2024-01-26T23:42:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.580+0000 7fc3eaddb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e40c46a0 con 0x7fc3e40c3cf0 2024-01-26T23:42:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.580+0000 7fc3eaddb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e40c6000 con 0x7fc3d4003a90 2024-01-26T23:42:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.580+0000 7fc3e9378700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e40c5650 0x7fc3e40c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.580+0000 7fc3e8b77700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3d4003a90 0x7fc3e413c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.580+0000 7fc3e9378700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e40c5650 0x7fc3e40c5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51352/0 (socket says 172.21.15.46:51352) 2024-01-26T23:42:22.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.580+0000 7fc3e9378700 1 -- 172.21.15.46:0/1142893379 learned_addr learned my addr 172.21.15.46:0/1142893379 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:22.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.580+0000 7fc3e3fff700 1 --2- 172.21.15.46:0/1142893379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3e40c3cf0 0x7fc3e40c4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:22.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.581+0000 7fc3e9378700 1 -- 172.21.15.46:0/1142893379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3d4003a90 msgr2=0x7fc3e413c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:22.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.581+0000 7fc3e9378700 1 --2- 172.21.15.46:0/1142893379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3d4003a90 0x7fc3e413c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.581+0000 7fc3e9378700 1 -- 172.21.15.46:0/1142893379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3e40c3cf0 msgr2=0x7fc3e40c4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:22.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.581+0000 7fc3e9378700 1 --2- 172.21.15.46:0/1142893379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3e40c3cf0 0x7fc3e40c4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.581+0000 7fc3e9378700 1 -- 172.21.15.46:0/1142893379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e413d380 con 0x7fc3e40c5650 2024-01-26T23:42:22.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.581+0000 7fc3e9378700 1 --2- 172.21.15.46:0/1142893379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e40c5650 0x7fc3e40c5ac0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4004790 tx=0x7fc3dc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:22.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.581+0000 7fc3e0ff9700 1 -- 172.21.15.46:0/1142893379 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc00e070 con 0x7fc3e40c5650 2024-01-26T23:42:22.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.581+0000 7fc3e0ff9700 1 -- 172.21.15.46:0/1142893379 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3dc011c50 con 0x7fc3e40c5650 2024-01-26T23:42:22.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.581+0000 7fc3e0ff9700 1 -- 172.21.15.46:0/1142893379 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc016470 con 0x7fc3e40c5650 2024-01-26T23:42:22.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.581+0000 7fc3eaddb700 1 -- 172.21.15.46:0/1142893379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e40c48c0 con 0x7fc3e40c5650 2024-01-26T23:42:22.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.581+0000 7fc3eaddb700 1 -- 172.21.15.46:0/1142893379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e4127580 con 0x7fc3e40c5650 2024-01-26T23:42:22.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.582+0000 7fc3eaddb700 1 -- 172.21.15.46:0/1142893379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e41408c0 con 0x7fc3e40c5650 2024-01-26T23:42:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.583+0000 7fc3e0ff9700 1 -- 172.21.15.46:0/1142893379 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fc3dc016860 con 0x7fc3e40c5650 2024-01-26T23:42:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.583+0000 7fc3e0ff9700 1 --2- 172.21.15.46:0/1142893379 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc3c8067130 0x7fc3c80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.583+0000 7fc3e0ff9700 1 -- 172.21.15.46:0/1142893379 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fc3dc085910 con 0x7fc3e40c5650 2024-01-26T23:42:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.583+0000 7fc3e8b77700 1 --2- 172.21.15.46:0/1142893379 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc3c8067130 0x7fc3c80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.584+0000 7fc3e8b77700 1 --2- 172.21.15.46:0/1142893379 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc3c8067130 0x7fc3c80695e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8000fd0 tx=0x7fc3d8015000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:22.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.588+0000 7fc3e0ff9700 1 -- 172.21.15.46:0/1142893379 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc3dc052590 con 0x7fc3e40c5650 2024-01-26T23:42:22.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.824+0000 7fc3eaddb700 1 -- 172.21.15.46:0/1142893379 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3e404c710 con 0x7fc3c8067130 2024-01-26T23:42:22.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.885+0000 7fc3e0ff9700 1 -- 172.21.15.46:0/1142893379 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fc3e404c710 con 0x7fc3c8067130 2024-01-26T23:42:22.886 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2024-01-26T23:42:22.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.888+0000 7fc3eaddb700 1 -- 172.21.15.46:0/1142893379 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc3c8067130 msgr2=0x7fc3c80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:22.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.888+0000 7fc3eaddb700 1 --2- 172.21.15.46:0/1142893379 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc3c8067130 0x7fc3c80695e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8000fd0 tx=0x7fc3d8015000 comp rx=0 tx=0).stop 2024-01-26T23:42:22.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.888+0000 7fc3eaddb700 1 -- 172.21.15.46:0/1142893379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e40c5650 msgr2=0x7fc3e40c5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:22.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.888+0000 7fc3eaddb700 1 --2- 172.21.15.46:0/1142893379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e40c5650 0x7fc3e40c5ac0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4004790 tx=0x7fc3dc00b040 comp rx=0 tx=0).stop 2024-01-26T23:42:22.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.888+0000 7fc3eaddb700 1 -- 172.21.15.46:0/1142893379 shutdown_connections 2024-01-26T23:42:22.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.888+0000 7fc3eaddb700 1 --2- 172.21.15.46:0/1142893379 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc3c8067130 0x7fc3c80695e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.888+0000 7fc3eaddb700 1 --2- 172.21.15.46:0/1142893379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e40c5650 0x7fc3e40c5ac0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.888+0000 7fc3eaddb700 1 --2- 172.21.15.46:0/1142893379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3d4003a90 0x7fc3e413c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.888+0000 7fc3eaddb700 1 --2- 172.21.15.46:0/1142893379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3e40c3cf0 0x7fc3e40c4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:22.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.888+0000 7fc3eaddb700 1 -- 172.21.15.46:0/1142893379 >> 172.21.15.46:0/1142893379 conn(0x7fc3e406bd90 msgr2=0x7fc3e413f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:22.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.888+0000 7fc3eaddb700 1 -- 172.21.15.46:0/1142893379 shutdown_connections 2024-01-26T23:42:22.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:22.888+0000 7fc3eaddb700 1 -- 172.21.15.46:0/1142893379 wait complete. 2024-01-26T23:42:22.902 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2024-01-26T23:42:22.903 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir7 1m 2024-01-26T23:42:23.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.040+0000 7fee61c6a700 1 Processor -- start 2024-01-26T23:42:23.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.040+0000 7fee61c6a700 1 -- start start 2024-01-26T23:42:23.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.041+0000 7fee61c6a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fee5c141200 0x7fee5c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.041+0000 7fee61c6a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5c0ccc00 con 0x7fee5c0c8da0 2024-01-26T23:42:23.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.041+0000 7fee61c6a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5c0cc310 con 0x7fee5c0caf30 2024-01-26T23:42:23.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.041+0000 7fee61c6a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee5c0cbbf0 con 0x7fee5c141200 2024-01-26T23:42:23.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.041+0000 7fee5b7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fee5c0c8da0 0x7fee5c140af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34826/0 (socket says 172.21.15.46:34826) 2024-01-26T23:42:23.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.041+0000 7fee5b7fe700 1 -- 172.21.15.46:0/521143020 learned_addr learned my addr 172.21.15.46:0/521143020 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:23.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.041+0000 7fee5affd700 1 --2- 172.21.15.46:0/521143020 >> v2:172.21.15.106:3300/0 conn(0x7fee5c141200 0x7fee5c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5a7fc700 1 -- 172.21.15.46:0/521143020 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854216236 0 0) 0x7fee5c0ccc00 con 0x7fee5c0c8da0 2024-01-26T23:42:23.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5a7fc700 1 -- 172.21.15.46:0/521143020 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee44003570 con 0x7fee5c0c8da0 2024-01-26T23:42:23.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5a7fc700 1 -- 172.21.15.46:0/521143020 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271081636 0 0) 0x7fee5c0cc310 con 0x7fee5c0caf30 2024-01-26T23:42:23.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5a7fc700 1 -- 172.21.15.46:0/521143020 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee5c0ccc00 con 0x7fee5c0caf30 2024-01-26T23:42:23.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5a7fc700 1 -- 172.21.15.46:0/521143020 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1384444355 0 0) 0x7fee44003570 con 0x7fee5c0c8da0 2024-01-26T23:42:23.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5a7fc700 1 -- 172.21.15.46:0/521143020 >> v2:172.21.15.106:3300/0 conn(0x7fee5c141200 msgr2=0x7fee5c0c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:23.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5a7fc700 1 --2- 172.21.15.46:0/521143020 >> v2:172.21.15.106:3300/0 conn(0x7fee5c141200 0x7fee5c0c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5a7fc700 1 -- 172.21.15.46:0/521143020 >> v1:172.21.15.106:6789/0 conn(0x7fee5c0caf30 legacy=0x7fee5c0cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:23.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5a7fc700 1 -- 172.21.15.46:0/521143020 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5c0cc5b0 con 0x7fee5c0c8da0 2024-01-26T23:42:23.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5a7fc700 1 -- 172.21.15.46:0/521143020 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fee4c001470 con 0x7fee5c0c8da0 2024-01-26T23:42:23.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5a7fc700 1 -- 172.21.15.46:0/521143020 >> v1:172.21.15.46:6789/0 conn(0x7fee5c0c8da0 legacy=0x7fee5c140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:23.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5a7fc700 1 --2- 172.21.15.46:0/521143020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee44003bf0 0x7fee44003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5a7fc700 1 --2- 172.21.15.46:0/521143020 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee44004810 0x7fee44004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5a7fc700 1 --2- 172.21.15.46:0/521143020 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee440051c0 0x7fee4400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5a7fc700 1 -- 172.21.15.46:0/521143020 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c0cbbf0 con 0x7fee44003bf0 2024-01-26T23:42:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5a7fc700 1 -- 172.21.15.46:0/521143020 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4400c160 con 0x7fee440051c0 2024-01-26T23:42:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5a7fc700 1 -- 172.21.15.46:0/521143020 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4400c320 con 0x7fee44004810 2024-01-26T23:42:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5b7fe700 1 --2- 172.21.15.46:0/521143020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee44003bf0 0x7fee44003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5bfff700 1 --2- 172.21.15.46:0/521143020 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee440051c0 0x7fee4400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5affd700 1 --2- 172.21.15.46:0/521143020 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee44004810 0x7fee44004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5b7fe700 1 -- 172.21.15.46:0/521143020 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee44004810 msgr2=0x7fee44004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5b7fe700 1 --2- 172.21.15.46:0/521143020 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee44004810 0x7fee44004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5b7fe700 1 -- 172.21.15.46:0/521143020 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee440051c0 msgr2=0x7fee4400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5b7fe700 1 --2- 172.21.15.46:0/521143020 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee440051c0 0x7fee4400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5b7fe700 1 -- 172.21.15.46:0/521143020 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee44003570 con 0x7fee44003bf0 2024-01-26T23:42:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5bfff700 1 --2- 172.21.15.46:0/521143020 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee440051c0 0x7fee4400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.042+0000 7fee5bfff700 1 -- 172.21.15.46:0/521143020 reap_dead start 2024-01-26T23:42:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.043+0000 7fee5b7fe700 1 --2- 172.21.15.46:0/521143020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee44003bf0 0x7fee44003fe0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fee4c00c2a0 tx=0x7fee5c0ce0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8b934e0cbbda41 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.043+0000 7fee5a7fc700 1 -- 172.21.15.46:0/521143020 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c00cbd0 con 0x7fee44003bf0 2024-01-26T23:42:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.043+0000 7fee5a7fc700 1 -- 172.21.15.46:0/521143020 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4c00cd70 con 0x7fee44003bf0 2024-01-26T23:42:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.043+0000 7fee5a7fc700 1 -- 172.21.15.46:0/521143020 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c005b70 con 0x7fee44003bf0 2024-01-26T23:42:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.043+0000 7fee61c6a700 1 -- 172.21.15.46:0/521143020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee44003bf0 msgr2=0x7fee44003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.043+0000 7fee61c6a700 1 --2- 172.21.15.46:0/521143020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee44003bf0 0x7fee44003fe0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fee4c00c2a0 tx=0x7fee5c0ce0a0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.043+0000 7fee61c6a700 1 -- 172.21.15.46:0/521143020 shutdown_connections 2024-01-26T23:42:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.043+0000 7fee61c6a700 1 --2- 172.21.15.46:0/521143020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee44003bf0 0x7fee44003fe0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.043+0000 7fee61c6a700 1 -- 172.21.15.46:0/521143020 >> 172.21.15.46:0/521143020 conn(0x7fee5c06bd90 msgr2=0x7fee5c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.043+0000 7fee61c6a700 1 -- 172.21.15.46:0/521143020 shutdown_connections 2024-01-26T23:42:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.044+0000 7fee61c6a700 1 -- 172.21.15.46:0/521143020 wait complete. 2024-01-26T23:42:23.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.045+0000 7fee61c6a700 1 Processor -- start 2024-01-26T23:42:23.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.045+0000 7fee61c6a700 1 -- start start 2024-01-26T23:42:23.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee61c6a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee44003bf0 0x7fee5c133430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee61c6a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee5c0a56d0 0x7fee5c0a5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee61c6a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee5c0b3350 0x7fee5c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee61c6a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c0b3d10 con 0x7fee44003bf0 2024-01-26T23:42:23.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee61c6a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c133970 con 0x7fee5c0a56d0 2024-01-26T23:42:23.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee61c6a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c069c00 con 0x7fee5c0b3350 2024-01-26T23:42:23.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee5b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee44003bf0 0x7fee5c133430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee5b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee44003bf0 0x7fee5c133430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51362/0 (socket says 172.21.15.46:51362) 2024-01-26T23:42:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee5b7fe700 1 -- 172.21.15.46:0/1533138961 learned_addr learned my addr 172.21.15.46:0/1533138961 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee5affd700 1 --2- 172.21.15.46:0/1533138961 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee5c0a56d0 0x7fee5c0a5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee5bfff700 1 --2- 172.21.15.46:0/1533138961 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee5c0b3350 0x7fee5c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee5b7fe700 1 -- 172.21.15.46:0/1533138961 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee5c0b3350 msgr2=0x7fee5c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee5b7fe700 1 --2- 172.21.15.46:0/1533138961 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee5c0b3350 0x7fee5c0b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee5b7fe700 1 -- 172.21.15.46:0/1533138961 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee5c0a56d0 msgr2=0x7fee5c0a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee5b7fe700 1 --2- 172.21.15.46:0/1533138961 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee5c0a56d0 0x7fee5c0a5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee5b7fe700 1 -- 172.21.15.46:0/1533138961 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5c069ee0 con 0x7fee44003bf0 2024-01-26T23:42:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee5affd700 1 --2- 172.21.15.46:0/1533138961 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee5c0a56d0 0x7fee5c0a5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:42:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee5b7fe700 1 --2- 172.21.15.46:0/1533138961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee44003bf0 0x7fee5c133430 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fee4c001ed0 tx=0x7fee4c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.046+0000 7fee437fe700 1 -- 172.21.15.46:0/1533138961 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c0010a0 con 0x7fee44003bf0 2024-01-26T23:42:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.047+0000 7fee437fe700 1 -- 172.21.15.46:0/1533138961 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4c000bc0 con 0x7fee44003bf0 2024-01-26T23:42:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.047+0000 7fee61c6a700 1 -- 172.21.15.46:0/1533138961 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee5c06a1c0 con 0x7fee44003bf0 2024-01-26T23:42:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.047+0000 7fee437fe700 1 -- 172.21.15.46:0/1533138961 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c007340 con 0x7fee44003bf0 2024-01-26T23:42:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.047+0000 7fee61c6a700 1 -- 172.21.15.46:0/1533138961 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee5c06a780 con 0x7fee44003bf0 2024-01-26T23:42:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.047+0000 7fee61c6a700 1 -- 172.21.15.46:0/1533138961 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5c06a970 con 0x7fee44003bf0 2024-01-26T23:42:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.048+0000 7fee437fe700 1 -- 172.21.15.46:0/1533138961 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fee4c0076f0 con 0x7fee44003bf0 2024-01-26T23:42:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.048+0000 7fee437fe700 1 --2- 172.21.15.46:0/1533138961 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fee48067130 0x7fee480695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.048+0000 7fee437fe700 1 -- 172.21.15.46:0/1533138961 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fee4c0858d0 con 0x7fee44003bf0 2024-01-26T23:42:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.049+0000 7fee5affd700 1 --2- 172.21.15.46:0/1533138961 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fee48067130 0x7fee480695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.049+0000 7fee5affd700 1 --2- 172.21.15.46:0/1533138961 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fee48067130 0x7fee480695e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fee50006f00 tx=0x7fee50008040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:23.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.053+0000 7fee437fe700 1 -- 172.21.15.46:0/1533138961 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fee4c052660 con 0x7fee44003bf0 2024-01-26T23:42:23.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.288+0000 7fee61c6a700 1 -- 172.21.15.46:0/1533138961 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fee5c04c710 con 0x7fee48067130 2024-01-26T23:42:23.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.350+0000 7fee437fe700 1 -- 172.21.15.46:0/1533138961 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fee5c04c710 con 0x7fee48067130 2024-01-26T23:42:23.351 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2024-01-26T23:42:23.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.352+0000 7fee61c6a700 1 -- 172.21.15.46:0/1533138961 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fee48067130 msgr2=0x7fee480695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:23.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.352+0000 7fee61c6a700 1 --2- 172.21.15.46:0/1533138961 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fee48067130 0x7fee480695e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fee50006f00 tx=0x7fee50008040 comp rx=0 tx=0).stop 2024-01-26T23:42:23.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.352+0000 7fee61c6a700 1 -- 172.21.15.46:0/1533138961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee44003bf0 msgr2=0x7fee5c133430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:23.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.352+0000 7fee61c6a700 1 --2- 172.21.15.46:0/1533138961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee44003bf0 0x7fee5c133430 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fee4c001ed0 tx=0x7fee4c015000 comp rx=0 tx=0).stop 2024-01-26T23:42:23.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.352+0000 7fee61c6a700 1 -- 172.21.15.46:0/1533138961 shutdown_connections 2024-01-26T23:42:23.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.352+0000 7fee61c6a700 1 --2- 172.21.15.46:0/1533138961 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fee48067130 0x7fee480695e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.352+0000 7fee61c6a700 1 --2- 172.21.15.46:0/1533138961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee44003bf0 0x7fee5c133430 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.352+0000 7fee61c6a700 1 --2- 172.21.15.46:0/1533138961 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee5c0b3350 0x7fee5c0b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.352+0000 7fee61c6a700 1 --2- 172.21.15.46:0/1533138961 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee5c0a56d0 0x7fee5c0a5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.352+0000 7fee61c6a700 1 -- 172.21.15.46:0/1533138961 >> 172.21.15.46:0/1533138961 conn(0x7fee5c06bd90 msgr2=0x7fee5c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:23.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.352+0000 7fee61c6a700 1 -- 172.21.15.46:0/1533138961 shutdown_connections 2024-01-26T23:42:23.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.353+0000 7fee61c6a700 1 -- 172.21.15.46:0/1533138961 wait complete. 2024-01-26T23:42:23.367 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2024-01-26T23:42:23.367 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir8 1m 2024-01-26T23:42:23.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.504+0000 7f79d8568700 1 Processor -- start 2024-01-26T23:42:23.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.504+0000 7f79d8568700 1 -- start start 2024-01-26T23:42:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.505+0000 7f79d8568700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f79d01598e0 0x7f79d00c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.505+0000 7f79d8568700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f79d00c9120 0x7f79d00cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.505+0000 7f79d8568700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d0069260 con 0x7f79d0159510 2024-01-26T23:42:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.505+0000 7f79d8568700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d00ca930 con 0x7f79d00c9120 2024-01-26T23:42:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.505+0000 7f79d8568700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d0159d50 con 0x7f79d01598e0 2024-01-26T23:42:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.505+0000 7f79d6b05700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f79d00c9120 0x7f79d00cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.505+0000 7f79d6b05700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f79d00c9120 0x7f79d00cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51376/0 (socket says 172.21.15.46:51376) 2024-01-26T23:42:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.505+0000 7f79d6b05700 1 -- 172.21.15.46:0/1041875156 learned_addr learned my addr 172.21.15.46:0/1041875156 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.505+0000 7f79d5b03700 1 --2- 172.21.15.46:0/1041875156 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f79d01598e0 0x7f79d00c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.505+0000 7f79d6b05700 1 -- 172.21.15.46:0/1041875156 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f79d01598e0 msgr2=0x7f79d00c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.505+0000 7f79d6b05700 1 --2- 172.21.15.46:0/1041875156 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f79d01598e0 0x7f79d00c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.505+0000 7f79d6b05700 1 -- 172.21.15.46:0/1041875156 >> v1:172.21.15.106:6789/0 conn(0x7f79d0159510 legacy=0x7f79d00ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.505+0000 7f79d6b05700 1 -- 172.21.15.46:0/1041875156 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d015d670 con 0x7f79d00c9120 2024-01-26T23:42:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.505+0000 7f79d5b03700 1 --2- 172.21.15.46:0/1041875156 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f79d01598e0 0x7f79d00c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:42:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.505+0000 7f79d6b05700 1 --2- 172.21.15.46:0/1041875156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79d00c9120 0x7f79d00cc380 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f79c80099b0 tx=0x7f79c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb77ed7d2f6f763 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.506+0000 7f79d5302700 1 -- 172.21.15.46:0/1041875156 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c800b820 con 0x7f79d00c9120 2024-01-26T23:42:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.506+0000 7f79d5302700 1 -- 172.21.15.46:0/1041875156 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c800b980 con 0x7f79d00c9120 2024-01-26T23:42:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.506+0000 7f79d5302700 1 -- 172.21.15.46:0/1041875156 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c800bd20 con 0x7f79d00c9120 2024-01-26T23:42:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.506+0000 7f79d8568700 1 -- 172.21.15.46:0/1041875156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79d00c9120 msgr2=0x7f79d00cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.506+0000 7f79d8568700 1 --2- 172.21.15.46:0/1041875156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79d00c9120 0x7f79d00cc380 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f79c80099b0 tx=0x7f79c8009250 comp rx=0 tx=0).stop 2024-01-26T23:42:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.506+0000 7f79d8568700 1 -- 172.21.15.46:0/1041875156 shutdown_connections 2024-01-26T23:42:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.506+0000 7f79d8568700 1 --2- 172.21.15.46:0/1041875156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79d00c9120 0x7f79d00cc380 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.506+0000 7f79d8568700 1 --2- 172.21.15.46:0/1041875156 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f79d01598e0 0x7f79d00c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.506+0000 7f79d8568700 1 -- 172.21.15.46:0/1041875156 >> 172.21.15.46:0/1041875156 conn(0x7f79d006bd90 msgr2=0x7f79d006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.506+0000 7f79d8568700 1 -- 172.21.15.46:0/1041875156 shutdown_connections 2024-01-26T23:42:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.506+0000 7f79d8568700 1 -- 172.21.15.46:0/1041875156 wait complete. 2024-01-26T23:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.507+0000 7f79d8568700 1 Processor -- start 2024-01-26T23:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.508+0000 7f79d8568700 1 -- start start 2024-01-26T23:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.508+0000 7f79d8568700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f79d00c9120 0x7f79d0152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.508+0000 7f79d8568700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79d0159510 0x7f79d00c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.508+0000 7f79d8568700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79d01598e0 0x7f79d00c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.508+0000 7f79d8568700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d015c710 con 0x7f79d01598e0 2024-01-26T23:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.508+0000 7f79d8568700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d01531f0 con 0x7f79d0159510 2024-01-26T23:42:23.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.508+0000 7f79d8568700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d015bb00 con 0x7f79d00c9120 2024-01-26T23:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.508+0000 7f79d6b05700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79d01598e0 0x7f79d00c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.508+0000 7f79d6b05700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79d01598e0 0x7f79d00c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51384/0 (socket says 172.21.15.46:51384) 2024-01-26T23:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.508+0000 7f79d6b05700 1 -- 172.21.15.46:0/3012379635 learned_addr learned my addr 172.21.15.46:0/3012379635 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.508+0000 7f79d5b03700 1 --2- 172.21.15.46:0/3012379635 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79d0159510 0x7f79d00c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.508+0000 7f79d6304700 1 --2- 172.21.15.46:0/3012379635 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f79d00c9120 0x7f79d0152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.509+0000 7f79d6b05700 1 -- 172.21.15.46:0/3012379635 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f79d00c9120 msgr2=0x7f79d0152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.509+0000 7f79d6b05700 1 --2- 172.21.15.46:0/3012379635 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f79d00c9120 0x7f79d0152cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.509+0000 7f79d6b05700 1 -- 172.21.15.46:0/3012379635 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79d0159510 msgr2=0x7f79d00c6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.509+0000 7f79d6b05700 1 --2- 172.21.15.46:0/3012379635 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79d0159510 0x7f79d00c6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.509+0000 7f79d6b05700 1 -- 172.21.15.46:0/3012379635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c80094f0 con 0x7f79d01598e0 2024-01-26T23:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.509+0000 7f79d6b05700 1 --2- 172.21.15.46:0/3012379635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79d01598e0 0x7f79d00c73f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f79c80030c0 tx=0x7f79c8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.509+0000 7f79c67fc700 1 -- 172.21.15.46:0/3012379635 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c8011ea0 con 0x7f79d01598e0 2024-01-26T23:42:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.509+0000 7f79c67fc700 1 -- 172.21.15.46:0/3012379635 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c8020070 con 0x7f79d01598e0 2024-01-26T23:42:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.509+0000 7f79d8568700 1 -- 172.21.15.46:0/3012379635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d015bde0 con 0x7f79d01598e0 2024-01-26T23:42:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.509+0000 7f79c67fc700 1 -- 172.21.15.46:0/3012379635 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c801b6a0 con 0x7f79d01598e0 2024-01-26T23:42:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.509+0000 7f79d8568700 1 -- 172.21.15.46:0/3012379635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d015c3a0 con 0x7f79d01598e0 2024-01-26T23:42:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.510+0000 7f79d8568700 1 -- 172.21.15.46:0/3012379635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d015c8c0 con 0x7f79d01598e0 2024-01-26T23:42:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.511+0000 7f79c67fc700 1 -- 172.21.15.46:0/3012379635 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f79c8022030 con 0x7f79d01598e0 2024-01-26T23:42:23.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.511+0000 7f79c67fc700 1 --2- 172.21.15.46:0/3012379635 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f79b4067130 0x7f79b40695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.511+0000 7f79c67fc700 1 -- 172.21.15.46:0/3012379635 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f79c8086b00 con 0x7f79d01598e0 2024-01-26T23:42:23.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.512+0000 7f79d6304700 1 --2- 172.21.15.46:0/3012379635 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f79b4067130 0x7f79b40695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.512+0000 7f79d6304700 1 --2- 172.21.15.46:0/3012379635 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f79b4067130 0x7f79b40695e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f79cc007fd0 tx=0x7f79cc0026a0 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:23.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.516+0000 7f79c67fc700 1 -- 172.21.15.46:0/3012379635 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f79d015c8c0 con 0x7f79d01598e0 2024-01-26T23:42:23.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.754+0000 7f79d8568700 1 -- 172.21.15.46:0/3012379635 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f79d015c8c0 con 0x7f79b4067130 2024-01-26T23:42:23.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.816+0000 7f79c67fc700 1 -- 172.21.15.46:0/3012379635 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f79d015c8c0 con 0x7f79b4067130 2024-01-26T23:42:23.817 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2024-01-26T23:42:23.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.819+0000 7f79d8568700 1 -- 172.21.15.46:0/3012379635 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f79b4067130 msgr2=0x7f79b40695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:23.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.819+0000 7f79d8568700 1 --2- 172.21.15.46:0/3012379635 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f79b4067130 0x7f79b40695e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f79cc007fd0 tx=0x7f79cc0026a0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.819+0000 7f79d8568700 1 -- 172.21.15.46:0/3012379635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79d01598e0 msgr2=0x7f79d00c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:23.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.819+0000 7f79d8568700 1 --2- 172.21.15.46:0/3012379635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79d01598e0 0x7f79d00c73f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f79c80030c0 tx=0x7f79c8016000 comp rx=0 tx=0).stop 2024-01-26T23:42:23.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.819+0000 7f79d8568700 1 -- 172.21.15.46:0/3012379635 shutdown_connections 2024-01-26T23:42:23.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.819+0000 7f79d8568700 1 --2- 172.21.15.46:0/3012379635 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f79b4067130 0x7f79b40695e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.819+0000 7f79d8568700 1 --2- 172.21.15.46:0/3012379635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79d01598e0 0x7f79d00c73f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.819+0000 7f79d8568700 1 --2- 172.21.15.46:0/3012379635 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f79d00c9120 0x7f79d0152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.819+0000 7f79d8568700 1 --2- 172.21.15.46:0/3012379635 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79d0159510 0x7f79d00c6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.819+0000 7f79d8568700 1 -- 172.21.15.46:0/3012379635 >> 172.21.15.46:0/3012379635 conn(0x7f79d006bd90 msgr2=0x7f79d00caa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:23.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.819+0000 7f79d8568700 1 -- 172.21.15.46:0/3012379635 shutdown_connections 2024-01-26T23:42:23.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.819+0000 7f79d8568700 1 -- 172.21.15.46:0/3012379635 wait complete. 2024-01-26T23:42:23.834 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2024-01-26T23:42:23.834 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir9 1m 2024-01-26T23:42:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.972+0000 7f2294093700 1 Processor -- start 2024-01-26T23:42:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.972+0000 7f2294093700 1 -- start start 2024-01-26T23:42:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.973+0000 7f2294093700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f228c06f2a0 0x7f228c06c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.973+0000 7f2294093700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c0c8b90 con 0x7f228c156570 2024-01-26T23:42:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.973+0000 7f2294093700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c0c8e50 con 0x7f228c156940 2024-01-26T23:42:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.973+0000 7f2294093700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c06d0b0 con 0x7f228c06f2a0 2024-01-26T23:42:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.973+0000 7f2291e2f700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f228c156570 0x7f228c0c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34838/0 (socket says 172.21.15.46:34838) 2024-01-26T23:42:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.973+0000 7f2291e2f700 1 -- 172.21.15.46:0/2247436842 learned_addr learned my addr 172.21.15.46:0/2247436842 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.973+0000 7f2292630700 1 --2- 172.21.15.46:0/2247436842 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f228c06f2a0 0x7f228c06c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.973+0000 7f2290e2d700 1 -- 172.21.15.46:0/2247436842 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547466092 0 0) 0x7f228c0c8b90 con 0x7f228c156570 2024-01-26T23:42:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.973+0000 7f2290e2d700 1 -- 172.21.15.46:0/2247436842 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2278003570 con 0x7f228c156570 2024-01-26T23:42:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.973+0000 7f2290e2d700 1 -- 172.21.15.46:0/2247436842 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285377385 0 0) 0x7f228c0c8e50 con 0x7f228c156940 2024-01-26T23:42:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.973+0000 7f2290e2d700 1 -- 172.21.15.46:0/2247436842 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f228c0c8b90 con 0x7f228c156940 2024-01-26T23:42:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.974+0000 7f2292630700 1 -- 172.21.15.46:0/2247436842 >> v1:172.21.15.106:6789/0 conn(0x7f228c156940 legacy=0x7f228c06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.974+0000 7f2292630700 1 -- 172.21.15.46:0/2247436842 >> v1:172.21.15.46:6789/0 conn(0x7f228c156570 legacy=0x7f228c0c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.974+0000 7f2292630700 1 -- 172.21.15.46:0/2247436842 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228c0c4c40 con 0x7f228c06f2a0 2024-01-26T23:42:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.974+0000 7f2292630700 1 --2- 172.21.15.46:0/2247436842 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f228c06f2a0 0x7f228c06c110 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2284009e90 tx=0x7f2284009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4f95f70cbefe506 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.974+0000 7f2290e2d700 1 -- 172.21.15.46:0/2247436842 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228400e070 con 0x7f228c06f2a0 2024-01-26T23:42:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.974+0000 7f2290e2d700 1 -- 172.21.15.46:0/2247436842 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228400bb80 con 0x7f228c06f2a0 2024-01-26T23:42:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.974+0000 7f2290e2d700 1 -- 172.21.15.46:0/2247436842 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2284012430 con 0x7f228c06f2a0 2024-01-26T23:42:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.975+0000 7f2294093700 1 -- 172.21.15.46:0/2247436842 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f228c06f2a0 msgr2=0x7f228c06c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.975+0000 7f2294093700 1 --2- 172.21.15.46:0/2247436842 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f228c06f2a0 0x7f228c06c110 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2284009e90 tx=0x7f2284009780 comp rx=0 tx=0).stop 2024-01-26T23:42:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.975+0000 7f2294093700 1 -- 172.21.15.46:0/2247436842 shutdown_connections 2024-01-26T23:42:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.975+0000 7f2294093700 1 --2- 172.21.15.46:0/2247436842 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f228c06f2a0 0x7f228c06c110 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.975+0000 7f2294093700 1 -- 172.21.15.46:0/2247436842 >> 172.21.15.46:0/2247436842 conn(0x7f228c0cdf50 msgr2=0x7f228c0c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.975+0000 7f2294093700 1 -- 172.21.15.46:0/2247436842 shutdown_connections 2024-01-26T23:42:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.975+0000 7f2294093700 1 -- 172.21.15.46:0/2247436842 wait complete. 2024-01-26T23:42:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.976+0000 7f2294093700 1 Processor -- start 2024-01-26T23:42:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.976+0000 7f2294093700 1 -- start start 2024-01-26T23:42:23.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.977+0000 7f2294093700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f228c06f2a0 0x7f228c14a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.977+0000 7f2294093700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f228c156570 0x7f228c0adc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.977+0000 7f2294093700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f228c156940 0x7f228c0c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.977+0000 7f2294093700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c14a750 con 0x7f228c156940 2024-01-26T23:42:23.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.977+0000 7f2294093700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c0af2a0 con 0x7f228c06f2a0 2024-01-26T23:42:23.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.977+0000 7f2294093700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c0ae690 con 0x7f228c156570 2024-01-26T23:42:23.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.978+0000 7f2292630700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f228c156940 0x7f228c0c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.978+0000 7f2292630700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f228c156940 0x7f228c0c9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51392/0 (socket says 172.21.15.46:51392) 2024-01-26T23:42:23.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.978+0000 7f2292630700 1 -- 172.21.15.46:0/3342224941 learned_addr learned my addr 172.21.15.46:0/3342224941 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:23.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.978+0000 7f2291e2f700 1 --2- 172.21.15.46:0/3342224941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f228c06f2a0 0x7f228c14a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.978+0000 7f229162e700 1 --2- 172.21.15.46:0/3342224941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f228c156570 0x7f228c0adc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.978+0000 7f2292630700 1 -- 172.21.15.46:0/3342224941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f228c156570 msgr2=0x7f228c0adc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:23.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.978+0000 7f2292630700 1 --2- 172.21.15.46:0/3342224941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f228c156570 0x7f228c0adc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.978+0000 7f2292630700 1 -- 172.21.15.46:0/3342224941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f228c06f2a0 msgr2=0x7f228c14a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:23.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.978+0000 7f2292630700 1 --2- 172.21.15.46:0/3342224941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f228c06f2a0 0x7f228c14a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:23.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.978+0000 7f2292630700 1 -- 172.21.15.46:0/3342224941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2284009a20 con 0x7f228c156940 2024-01-26T23:42:23.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.978+0000 7f2292630700 1 --2- 172.21.15.46:0/3342224941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f228c156940 0x7f228c0c9490 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2284013000 tx=0x7f2284019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:23.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.978+0000 7f2281ffb700 1 -- 172.21.15.46:0/3342224941 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2284009420 con 0x7f228c156940 2024-01-26T23:42:23.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.978+0000 7f2281ffb700 1 -- 172.21.15.46:0/3342224941 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2284027470 con 0x7f228c156940 2024-01-26T23:42:23.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.978+0000 7f2281ffb700 1 -- 172.21.15.46:0/3342224941 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2284027810 con 0x7f228c156940 2024-01-26T23:42:23.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.978+0000 7f2294093700 1 -- 172.21.15.46:0/3342224941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228c0ae970 con 0x7f228c156940 2024-01-26T23:42:23.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.979+0000 7f2294093700 1 -- 172.21.15.46:0/3342224941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f228c0aef30 con 0x7f228c156940 2024-01-26T23:42:23.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.980+0000 7f2294093700 1 -- 172.21.15.46:0/3342224941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f228c04c710 con 0x7f228c156940 2024-01-26T23:42:23.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.980+0000 7f2281ffb700 1 -- 172.21.15.46:0/3342224941 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f2284024070 con 0x7f228c156940 2024-01-26T23:42:23.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.980+0000 7f2281ffb700 1 --2- 172.21.15.46:0/3342224941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2270067130 0x7f22700695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:23.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.981+0000 7f2281ffb700 1 -- 172.21.15.46:0/3342224941 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f2284087210 con 0x7f228c156940 2024-01-26T23:42:23.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.981+0000 7f2291e2f700 1 --2- 172.21.15.46:0/3342224941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2270067130 0x7f22700695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:23.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.981+0000 7f2291e2f700 1 --2- 172.21.15.46:0/3342224941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2270067130 0x7f22700695e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2288007fd0 tx=0x7f2288009040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:23.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:23.985+0000 7f2281ffb700 1 -- 172.21.15.46:0/3342224941 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2284053f00 con 0x7f228c156940 2024-01-26T23:42:24.156 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:42:24.158 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:42:24.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.218+0000 7f2294093700 1 -- 172.21.15.46:0/3342224941 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f228c0af550 con 0x7f2270067130 2024-01-26T23:42:24.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.282+0000 7f2281ffb700 1 -- 172.21.15.46:0/3342224941 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f228c0af550 con 0x7f2270067130 2024-01-26T23:42:24.283 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2024-01-26T23:42:24.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.285+0000 7f2294093700 1 -- 172.21.15.46:0/3342224941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2270067130 msgr2=0x7f22700695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:24.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.285+0000 7f2294093700 1 --2- 172.21.15.46:0/3342224941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2270067130 0x7f22700695e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2288007fd0 tx=0x7f2288009040 comp rx=0 tx=0).stop 2024-01-26T23:42:24.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.285+0000 7f2294093700 1 -- 172.21.15.46:0/3342224941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f228c156940 msgr2=0x7f228c0c9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:24.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.285+0000 7f2294093700 1 --2- 172.21.15.46:0/3342224941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f228c156940 0x7f228c0c9490 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2284013000 tx=0x7f2284019000 comp rx=0 tx=0).stop 2024-01-26T23:42:24.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.285+0000 7f2294093700 1 -- 172.21.15.46:0/3342224941 shutdown_connections 2024-01-26T23:42:24.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.285+0000 7f2294093700 1 --2- 172.21.15.46:0/3342224941 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2270067130 0x7f22700695e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.285+0000 7f2294093700 1 --2- 172.21.15.46:0/3342224941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f228c156940 0x7f228c0c9490 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.285+0000 7f2294093700 1 --2- 172.21.15.46:0/3342224941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f228c06f2a0 0x7f228c14a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.285+0000 7f2294093700 1 --2- 172.21.15.46:0/3342224941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f228c156570 0x7f228c0adc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.285+0000 7f2294093700 1 -- 172.21.15.46:0/3342224941 >> 172.21.15.46:0/3342224941 conn(0x7f228c0cdf50 msgr2=0x7f228c0cc3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:24.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.285+0000 7f2294093700 1 -- 172.21.15.46:0/3342224941 shutdown_connections 2024-01-26T23:42:24.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.285+0000 7f2294093700 1 -- 172.21.15.46:0/3342224941 wait complete. 2024-01-26T23:42:24.299 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2024-01-26T23:42:24.300 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:24.303 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-01-26T23:42:24.328 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:24.328 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:24.328 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:24.328 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:24.329 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:24.329 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:24.329 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:24.329 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:24.329 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:24.329 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:24.329 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:24.330 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-26-23_32_00_UTC', 'scheduled-2024-01-26-23_33_00_UTC', 'scheduled-2024-01-26-23_34_00_UTC', 'scheduled-2024-01-26-23_35_00_UTC', 'scheduled-2024-01-26-23_36_00_UTC', 'scheduled-2024-01-26-23_37_00_UTC', 'scheduled-2024-01-26-23_38_00_UTC', 'scheduled-2024-01-26-23_39_00_UTC', 'scheduled-2024-01-26-23_40_00_UTC', 'scheduled-2024-01-26-23_41_00_UTC', 'scheduled-2024-01-26-23_42_00_UTC'] 2024-01-26T23:42:24.330 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir0 2024-01-26T23:42:24.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.508+0000 7f9cca525700 1 Processor -- start 2024-01-26T23:42:24.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.508+0000 7f9cca525700 1 -- start start 2024-01-26T23:42:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cca525700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9cc40c8da0 0x7f9cc40c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cca525700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9cc4140af0 0x7f9cc4140f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cca525700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9cc40caf30 0x7f9cc40cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cca525700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc40cbbf0 con 0x7f9cc40caf30 2024-01-26T23:42:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cca525700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc40ccc00 con 0x7f9cc4140af0 2024-01-26T23:42:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cca525700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc40cc310 con 0x7f9cc40c8da0 2024-01-26T23:42:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cc8ac2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9cc40caf30 0x7f9cc40cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cc8ac2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9cc40caf30 0x7f9cc40cb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51396/0 (socket says 172.21.15.46:51396) 2024-01-26T23:42:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cc8ac2700 1 -- 172.21.15.46:0/3207482561 learned_addr learned my addr 172.21.15.46:0/3207482561 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cc3fff700 1 --2- 172.21.15.46:0/3207482561 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9cc40c8da0 0x7f9cc40c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cc37fe700 1 --2- 172.21.15.46:0/3207482561 >> v2:172.21.15.106:3300/0 conn(0x7f9cc4140af0 0x7f9cc4140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cc8ac2700 1 -- 172.21.15.46:0/3207482561 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9cc40c8da0 msgr2=0x7f9cc40c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cc8ac2700 1 --2- 172.21.15.46:0/3207482561 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9cc40c8da0 0x7f9cc40c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cc8ac2700 1 -- 172.21.15.46:0/3207482561 >> v2:172.21.15.106:3300/0 conn(0x7f9cc4140af0 msgr2=0x7f9cc4140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cc8ac2700 1 --2- 172.21.15.46:0/3207482561 >> v2:172.21.15.106:3300/0 conn(0x7f9cc4140af0 0x7f9cc4140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cc8ac2700 1 -- 172.21.15.46:0/3207482561 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc40cc5b0 con 0x7f9cc40caf30 2024-01-26T23:42:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cc3fff700 1 --2- 172.21.15.46:0/3207482561 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9cc40c8da0 0x7f9cc40c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.509+0000 7f9cc37fe700 1 --2- 172.21.15.46:0/3207482561 >> v2:172.21.15.106:3300/0 conn(0x7f9cc4140af0 0x7f9cc4140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.510+0000 7f9cc8ac2700 1 --2- 172.21.15.46:0/3207482561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cc40caf30 0x7f9cc40cb370 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9cc406f9a0 tx=0x7f9cbc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79221397f3e2366f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.510+0000 7f9cc2ffd700 1 -- 172.21.15.46:0/3207482561 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc00e070 con 0x7f9cc40caf30 2024-01-26T23:42:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.510+0000 7f9cc2ffd700 1 -- 172.21.15.46:0/3207482561 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cbc00ba50 con 0x7f9cc40caf30 2024-01-26T23:42:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.510+0000 7f9cc2ffd700 1 -- 172.21.15.46:0/3207482561 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc0092e0 con 0x7f9cc40caf30 2024-01-26T23:42:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.510+0000 7f9cca525700 1 -- 172.21.15.46:0/3207482561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cc40caf30 msgr2=0x7f9cc40cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.510+0000 7f9cca525700 1 --2- 172.21.15.46:0/3207482561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cc40caf30 0x7f9cc40cb370 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9cc406f9a0 tx=0x7f9cbc009840 comp rx=0 tx=0).stop 2024-01-26T23:42:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.510+0000 7f9cca525700 1 -- 172.21.15.46:0/3207482561 shutdown_connections 2024-01-26T23:42:24.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.510+0000 7f9cca525700 1 --2- 172.21.15.46:0/3207482561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cc40caf30 0x7f9cc40cb370 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.510+0000 7f9cca525700 1 --2- 172.21.15.46:0/3207482561 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9cc40c8da0 0x7f9cc40c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.510+0000 7f9cca525700 1 --2- 172.21.15.46:0/3207482561 >> v2:172.21.15.106:3300/0 conn(0x7f9cc4140af0 0x7f9cc4140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.510+0000 7f9cca525700 1 -- 172.21.15.46:0/3207482561 >> 172.21.15.46:0/3207482561 conn(0x7f9cc406bd90 msgr2=0x7f9cc406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:24.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.510+0000 7f9cca525700 1 -- 172.21.15.46:0/3207482561 shutdown_connections 2024-01-26T23:42:24.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.510+0000 7f9cca525700 1 -- 172.21.15.46:0/3207482561 wait complete. 2024-01-26T23:42:24.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.512+0000 7f9cca525700 1 Processor -- start 2024-01-26T23:42:24.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.512+0000 7f9cca525700 1 -- start start 2024-01-26T23:42:24.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.513+0000 7f9cca525700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9cc40c8da0 0x7f9cc40c7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:24.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.513+0000 7f9cca525700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc40caf30 0x7f9cc40c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:24.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.513+0000 7f9cca525700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cc4140af0 0x7f9cc40b3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:24.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.513+0000 7f9cca525700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc40b34f0 con 0x7f9cc4140af0 2024-01-26T23:42:24.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.513+0000 7f9cca525700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc40b1700 con 0x7f9cc40caf30 2024-01-26T23:42:24.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.513+0000 7f9cca525700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc40c8130 con 0x7f9cc40c8da0 2024-01-26T23:42:24.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.513+0000 7f9cc8ac2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cc4140af0 0x7f9cc40b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:24.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.514+0000 7f9cc8ac2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cc4140af0 0x7f9cc40b3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51404/0 (socket says 172.21.15.46:51404) 2024-01-26T23:42:24.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.514+0000 7f9cc8ac2700 1 -- 172.21.15.46:0/3495452210 learned_addr learned my addr 172.21.15.46:0/3495452210 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:24.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.514+0000 7f9cc37fe700 1 --2- 172.21.15.46:0/3495452210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc40caf30 0x7f9cc40c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:24.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.514+0000 7f9cc3fff700 1 --2- 172.21.15.46:0/3495452210 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9cc40c8da0 0x7f9cc40c7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:24.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.514+0000 7f9cc8ac2700 1 -- 172.21.15.46:0/3495452210 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9cc40c8da0 msgr2=0x7f9cc40c7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:24.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.514+0000 7f9cc8ac2700 1 --2- 172.21.15.46:0/3495452210 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9cc40c8da0 0x7f9cc40c7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.514+0000 7f9cc8ac2700 1 -- 172.21.15.46:0/3495452210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc40caf30 msgr2=0x7f9cc40c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:24.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.514+0000 7f9cc8ac2700 1 --2- 172.21.15.46:0/3495452210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc40caf30 0x7f9cc40c5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.514+0000 7f9cc8ac2700 1 -- 172.21.15.46:0/3495452210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cbc009ae0 con 0x7f9cc4140af0 2024-01-26T23:42:24.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.514+0000 7f9cc8ac2700 1 --2- 172.21.15.46:0/3495452210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cc4140af0 0x7f9cc40b3980 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9cc406f5a0 tx=0x7f9cbc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:24.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.514+0000 7f9cabfff700 1 -- 172.21.15.46:0/3495452210 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc0032f0 con 0x7f9cc4140af0 2024-01-26T23:42:24.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.514+0000 7f9cabfff700 1 -- 172.21.15.46:0/3495452210 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cbc0094e0 con 0x7f9cc4140af0 2024-01-26T23:42:24.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.514+0000 7f9cca525700 1 -- 172.21.15.46:0/3495452210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc40c83b0 con 0x7f9cc4140af0 2024-01-26T23:42:24.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.514+0000 7f9cabfff700 1 -- 172.21.15.46:0/3495452210 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc026460 con 0x7f9cc4140af0 2024-01-26T23:42:24.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.514+0000 7f9cca525700 1 -- 172.21.15.46:0/3495452210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc40b3180 con 0x7f9cc4140af0 2024-01-26T23:42:24.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.515+0000 7f9cca525700 1 -- 172.21.15.46:0/3495452210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc40b36a0 con 0x7f9cc4140af0 2024-01-26T23:42:24.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.516+0000 7f9cabfff700 1 -- 172.21.15.46:0/3495452210 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f9cbc003040 con 0x7f9cc4140af0 2024-01-26T23:42:24.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.516+0000 7f9cabfff700 1 --2- 172.21.15.46:0/3495452210 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9cb0067130 0x7f9cb00695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:24.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.516+0000 7f9cabfff700 1 -- 172.21.15.46:0/3495452210 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f9cbc086a50 con 0x7f9cc4140af0 2024-01-26T23:42:24.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.517+0000 7f9cc3fff700 1 --2- 172.21.15.46:0/3495452210 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9cb0067130 0x7f9cb00695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:24.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.517+0000 7f9cc3fff700 1 --2- 172.21.15.46:0/3495452210 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9cb0067130 0x7f9cb00695e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4006f00 tx=0x7f9cb4008040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:24.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.521+0000 7f9cabfff700 1 -- 172.21.15.46:0/3495452210 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9cc40b36a0 con 0x7f9cc4140af0 2024-01-26T23:42:24.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.758+0000 7f9cca525700 1 -- 172.21.15.46:0/3495452210 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9cc40b36a0 con 0x7f9cb0067130 2024-01-26T23:42:24.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.767+0000 7f9cabfff700 1 -- 172.21.15.46:0/3495452210 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f9cc40b36a0 con 0x7f9cb0067130 2024-01-26T23:42:24.768 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:24.768 INFO:teuthology.orchestra.run.smithi046.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:08", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:24.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.769+0000 7f9cca525700 1 -- 172.21.15.46:0/3495452210 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9cb0067130 msgr2=0x7f9cb00695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:24.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.769+0000 7f9cca525700 1 --2- 172.21.15.46:0/3495452210 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9cb0067130 0x7f9cb00695e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4006f00 tx=0x7f9cb4008040 comp rx=0 tx=0).stop 2024-01-26T23:42:24.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.769+0000 7f9cca525700 1 -- 172.21.15.46:0/3495452210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cc4140af0 msgr2=0x7f9cc40b3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:24.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.769+0000 7f9cca525700 1 --2- 172.21.15.46:0/3495452210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cc4140af0 0x7f9cc40b3980 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9cc406f5a0 tx=0x7f9cbc018000 comp rx=0 tx=0).stop 2024-01-26T23:42:24.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.770+0000 7f9cca525700 1 -- 172.21.15.46:0/3495452210 shutdown_connections 2024-01-26T23:42:24.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.770+0000 7f9cca525700 1 --2- 172.21.15.46:0/3495452210 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9cb0067130 0x7f9cb00695e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.770+0000 7f9cca525700 1 --2- 172.21.15.46:0/3495452210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cc4140af0 0x7f9cc40b3980 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.770+0000 7f9cca525700 1 --2- 172.21.15.46:0/3495452210 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9cc40c8da0 0x7f9cc40c7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.770+0000 7f9cca525700 1 --2- 172.21.15.46:0/3495452210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cc40caf30 0x7f9cc40c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.770+0000 7f9cca525700 1 -- 172.21.15.46:0/3495452210 >> 172.21.15.46:0/3495452210 conn(0x7f9cc406bd90 msgr2=0x7f9cc406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:24.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.770+0000 7f9cca525700 1 -- 172.21.15.46:0/3495452210 shutdown_connections 2024-01-26T23:42:24.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.770+0000 7f9cca525700 1 -- 172.21.15.46:0/3495452210 wait complete. 2024-01-26T23:42:24.783 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:08", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:24.783 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1m', 'retention': {}, 'start': '2024-01-26T00:00:00', 'created': '2024-01-26T23:31:08', 'first': '2024-01-26T23:32:00', 'last': '2024-01-26T23:42:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-26T23:42:24.784 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-26T23:42:24.784 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-26T23:42:24.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:24.784 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-01-26T23:42:24.808 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:24.808 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:24.808 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:24.808 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:24.808 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:24.808 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:24.808 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:24.808 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:24.808 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:24.808 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:24.809 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:24.809 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-26-23_32_00_UTC', 'scheduled-2024-01-26-23_33_00_UTC', 'scheduled-2024-01-26-23_34_00_UTC', 'scheduled-2024-01-26-23_35_00_UTC', 'scheduled-2024-01-26-23_36_00_UTC', 'scheduled-2024-01-26-23_37_00_UTC', 'scheduled-2024-01-26-23_38_00_UTC', 'scheduled-2024-01-26-23_39_00_UTC', 'scheduled-2024-01-26-23_40_00_UTC', 'scheduled-2024-01-26-23_41_00_UTC', 'scheduled-2024-01-26-23_42_00_UTC'] 2024-01-26T23:42:24.809 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir1 2024-01-26T23:42:24.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.988+0000 7fefb8b68700 1 Processor -- start 2024-01-26T23:42:24.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.988+0000 7fefb8b68700 1 -- start start 2024-01-26T23:42:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.989+0000 7fefb8b68700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fefb4159510 0x7fefb41598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.989+0000 7fefb8b68700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fefb40ca690 0x7fefb40cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.989+0000 7fefb8b68700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb4069260 con 0x7fefb40ca220 2024-01-26T23:42:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.989+0000 7fefb8b68700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefb415d670 con 0x7fefb4159510 2024-01-26T23:42:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.989+0000 7fefb8b68700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb40c9750 con 0x7fefb40ca690 2024-01-26T23:42:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.989+0000 7fefb259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fefb4159510 0x7fefb41598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.989+0000 7fefb1d9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fefb40ca220 0x7fefb40c8be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34852/0 (socket says 172.21.15.46:34852) 2024-01-26T23:42:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.989+0000 7fefb1d9b700 1 -- 172.21.15.46:0/347427879 learned_addr learned my addr 172.21.15.46:0/347427879 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.989+0000 7fefb2d9d700 1 --2- 172.21.15.46:0/347427879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fefb40ca690 0x7fefb40cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.989+0000 7fefb259c700 1 -- 172.21.15.46:0/347427879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fefb40ca690 msgr2=0x7fefb40cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.989+0000 7fefb259c700 1 --2- 172.21.15.46:0/347427879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fefb40ca690 0x7fefb40cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.989+0000 7fefb259c700 1 -- 172.21.15.46:0/347427879 >> v1:172.21.15.46:6789/0 conn(0x7fefb40ca220 legacy=0x7fefb40c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:24.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.989+0000 7fefb159a700 1 -- 172.21.15.46:0/347427879 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348692194 0 0) 0x7fefb4069260 con 0x7fefb40ca220 2024-01-26T23:42:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.989+0000 7fefb259c700 1 -- 172.21.15.46:0/347427879 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb40c9a30 con 0x7fefb4159510 2024-01-26T23:42:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.989+0000 7fefb2d9d700 1 --2- 172.21.15.46:0/347427879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fefb40ca690 0x7fefb40cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.990+0000 7fefb259c700 1 --2- 172.21.15.46:0/347427879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb4159510 0x7fefb41598e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fefa40098e0 tx=0x7fefa40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c307952542c982 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.990+0000 7fefb159a700 1 -- 172.21.15.46:0/347427879 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa4009e80 con 0x7fefb4159510 2024-01-26T23:42:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.990+0000 7fefb159a700 1 -- 172.21.15.46:0/347427879 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa400b860 con 0x7fefb4159510 2024-01-26T23:42:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.990+0000 7fefb159a700 1 -- 172.21.15.46:0/347427879 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa400bbe0 con 0x7fefb4159510 2024-01-26T23:42:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.990+0000 7fefb8b68700 1 -- 172.21.15.46:0/347427879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb4159510 msgr2=0x7fefb41598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.990+0000 7fefb8b68700 1 --2- 172.21.15.46:0/347427879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb4159510 0x7fefb41598e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fefa40098e0 tx=0x7fefa40091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.990+0000 7fefb8b68700 1 -- 172.21.15.46:0/347427879 shutdown_connections 2024-01-26T23:42:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.990+0000 7fefb8b68700 1 --2- 172.21.15.46:0/347427879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fefb40ca690 0x7fefb40cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.990+0000 7fefb8b68700 1 --2- 172.21.15.46:0/347427879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb4159510 0x7fefb41598e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.990+0000 7fefb8b68700 1 -- 172.21.15.46:0/347427879 >> 172.21.15.46:0/347427879 conn(0x7fefb406bd90 msgr2=0x7fefb406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:24.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.990+0000 7fefb8b68700 1 -- 172.21.15.46:0/347427879 shutdown_connections 2024-01-26T23:42:24.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.990+0000 7fefb8b68700 1 -- 172.21.15.46:0/347427879 wait complete. 2024-01-26T23:42:24.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.992+0000 7fefb8b68700 1 Processor -- start 2024-01-26T23:42:24.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.992+0000 7fefb8b68700 1 -- start start 2024-01-26T23:42:24.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.992+0000 7fefb8b68700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fefb40ca220 0x7fefb416a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:24.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.992+0000 7fefb8b68700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb4159510 0x7fefb416aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:24.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.992+0000 7fefb8b68700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fefb40b3810 0x7fefb40b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:24.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.992+0000 7fefb8b68700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb40b41c0 con 0x7fefb4159510 2024-01-26T23:42:24.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.992+0000 7fefb8b68700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb416a080 con 0x7fefb40ca220 2024-01-26T23:42:24.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.992+0000 7fefb8b68700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb415ba10 con 0x7fefb40b3810 2024-01-26T23:42:24.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.992+0000 7fefb1d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb4159510 0x7fefb416aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:24.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.993+0000 7fefb1d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb4159510 0x7fefb416aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51426/0 (socket says 172.21.15.46:51426) 2024-01-26T23:42:24.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.993+0000 7fefb1d9b700 1 -- 172.21.15.46:0/1874447230 learned_addr learned my addr 172.21.15.46:0/1874447230 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:24.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.993+0000 7fefb259c700 1 --2- 172.21.15.46:0/1874447230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fefb40ca220 0x7fefb416a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:24.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.993+0000 7fefb2d9d700 1 --2- 172.21.15.46:0/1874447230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fefb40b3810 0x7fefb40b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:24.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.993+0000 7fefb1d9b700 1 -- 172.21.15.46:0/1874447230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fefb40b3810 msgr2=0x7fefb40b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:24.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.993+0000 7fefb1d9b700 1 --2- 172.21.15.46:0/1874447230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fefb40b3810 0x7fefb40b3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.993+0000 7fefb1d9b700 1 -- 172.21.15.46:0/1874447230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fefb40ca220 msgr2=0x7fefb416a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:24.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.993+0000 7fefb1d9b700 1 --2- 172.21.15.46:0/1874447230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fefb40ca220 0x7fefb416a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:24.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.993+0000 7fefb1d9b700 1 -- 172.21.15.46:0/1874447230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa4009420 con 0x7fefb4159510 2024-01-26T23:42:24.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.993+0000 7fefb2d9d700 1 --2- 172.21.15.46:0/1874447230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fefb40b3810 0x7fefb40b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:24.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.993+0000 7fefb1d9b700 1 --2- 172.21.15.46:0/1874447230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb4159510 0x7fefb416aa50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fefa800c7e0 tx=0x7fefa8007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:24.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.993+0000 7fefa27fc700 1 -- 172.21.15.46:0/1874447230 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa8005b30 con 0x7fefb4159510 2024-01-26T23:42:24.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.993+0000 7fefa27fc700 1 -- 172.21.15.46:0/1874447230 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa8005c90 con 0x7fefb4159510 2024-01-26T23:42:24.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.993+0000 7fefb8b68700 1 -- 172.21.15.46:0/1874447230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb415bcb0 con 0x7fefb4159510 2024-01-26T23:42:24.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.993+0000 7fefa27fc700 1 -- 172.21.15.46:0/1874447230 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa8007300 con 0x7fefb4159510 2024-01-26T23:42:24.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.993+0000 7fefb8b68700 1 -- 172.21.15.46:0/1874447230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb415c270 con 0x7fefb4159510 2024-01-26T23:42:24.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.994+0000 7fefb8b68700 1 -- 172.21.15.46:0/1874447230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb416a230 con 0x7fefb4159510 2024-01-26T23:42:24.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.995+0000 7fefa27fc700 1 -- 172.21.15.46:0/1874447230 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fefa80076b0 con 0x7fefb4159510 2024-01-26T23:42:24.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.995+0000 7fefa27fc700 1 --2- 172.21.15.46:0/1874447230 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fef94067130 0x7fef940695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:24.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.995+0000 7fefa27fc700 1 -- 172.21.15.46:0/1874447230 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fefa8084080 con 0x7fefb4159510 2024-01-26T23:42:24.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.995+0000 7fefb259c700 1 --2- 172.21.15.46:0/1874447230 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fef94067130 0x7fef940695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:24.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:24.996+0000 7fefb259c700 1 --2- 172.21.15.46:0/1874447230 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fef94067130 0x7fef940695e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fefa40091e0 tx=0x7fefa4017000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:25.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.000+0000 7fefa27fc700 1 -- 172.21.15.46:0/1874447230 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fefb416a230 con 0x7fefb4159510 2024-01-26T23:42:25.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.229+0000 7fefb8b68700 1 -- 172.21.15.46:0/1874447230 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fefb415c490 con 0x7fef94067130 2024-01-26T23:42:25.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.238+0000 7fefa27fc700 1 -- 172.21.15.46:0/1874447230 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fefb415c490 con 0x7fef94067130 2024-01-26T23:42:25.239 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:25.239 INFO:teuthology.orchestra.run.smithi046.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:08", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:25.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.240+0000 7fefb8b68700 1 -- 172.21.15.46:0/1874447230 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fef94067130 msgr2=0x7fef940695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:25.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.240+0000 7fefb8b68700 1 --2- 172.21.15.46:0/1874447230 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fef94067130 0x7fef940695e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fefa40091e0 tx=0x7fefa4017000 comp rx=0 tx=0).stop 2024-01-26T23:42:25.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.240+0000 7fefb8b68700 1 -- 172.21.15.46:0/1874447230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb4159510 msgr2=0x7fefb416aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.240+0000 7fefb8b68700 1 --2- 172.21.15.46:0/1874447230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb4159510 0x7fefb416aa50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fefa800c7e0 tx=0x7fefa8007df0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.241+0000 7fefb8b68700 1 -- 172.21.15.46:0/1874447230 shutdown_connections 2024-01-26T23:42:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.241+0000 7fefb8b68700 1 --2- 172.21.15.46:0/1874447230 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fef94067130 0x7fef940695e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.241+0000 7fefb8b68700 1 --2- 172.21.15.46:0/1874447230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fefb40b3810 0x7fefb40b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.241+0000 7fefb8b68700 1 --2- 172.21.15.46:0/1874447230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fefb40ca220 0x7fefb416a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.241+0000 7fefb8b68700 1 --2- 172.21.15.46:0/1874447230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb4159510 0x7fefb416aa50 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.241+0000 7fefb8b68700 1 -- 172.21.15.46:0/1874447230 >> 172.21.15.46:0/1874447230 conn(0x7fefb406bd90 msgr2=0x7fefb40c8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.241+0000 7fefb8b68700 1 -- 172.21.15.46:0/1874447230 shutdown_connections 2024-01-26T23:42:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.241+0000 7fefb8b68700 1 -- 172.21.15.46:0/1874447230 wait complete. 2024-01-26T23:42:25.254 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:08", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:25.254 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1m', 'retention': {}, 'start': '2024-01-26T00:00:00', 'created': '2024-01-26T23:31:08', 'first': '2024-01-26T23:32:00', 'last': '2024-01-26T23:42:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-26T23:42:25.254 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-26T23:42:25.254 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-26T23:42:25.255 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:25.255 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-01-26T23:42:25.279 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:25.279 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:25.279 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:25.279 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:25.279 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:25.279 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:25.279 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:25.279 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:25.279 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:25.279 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:25.279 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:25.280 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-26-23_32_00_UTC', 'scheduled-2024-01-26-23_33_00_UTC', 'scheduled-2024-01-26-23_34_00_UTC', 'scheduled-2024-01-26-23_35_00_UTC', 'scheduled-2024-01-26-23_36_00_UTC', 'scheduled-2024-01-26-23_37_00_UTC', 'scheduled-2024-01-26-23_38_00_UTC', 'scheduled-2024-01-26-23_39_00_UTC', 'scheduled-2024-01-26-23_40_00_UTC', 'scheduled-2024-01-26-23_41_00_UTC', 'scheduled-2024-01-26-23_42_00_UTC'] 2024-01-26T23:42:25.280 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir2 2024-01-26T23:42:25.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.459+0000 7f7dce6f6700 1 Processor -- start 2024-01-26T23:42:25.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.459+0000 7f7dce6f6700 1 -- start start 2024-01-26T23:42:25.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.460+0000 7f7dce6f6700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7dc80c8da0 0x7f7dc80c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:25.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.460+0000 7f7dce6f6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7dc80caf30 0x7f7dc80cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:25.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.460+0000 7f7dce6f6700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc80ccc00 con 0x7f7dc8140af0 2024-01-26T23:42:25.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.460+0000 7f7dce6f6700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc80cbbf0 con 0x7f7dc80c8da0 2024-01-26T23:42:25.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.460+0000 7f7dce6f6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc80cc310 con 0x7f7dc80caf30 2024-01-26T23:42:25.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.460+0000 7f7dc77fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7dc8140af0 0x7f7dc8140f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34856/0 (socket says 172.21.15.46:34856) 2024-01-26T23:42:25.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.460+0000 7f7dc77fe700 1 -- 172.21.15.46:0/2067283184 learned_addr learned my addr 172.21.15.46:0/2067283184 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:25.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.460+0000 7f7dccc93700 1 --2- 172.21.15.46:0/2067283184 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7dc80caf30 0x7f7dc80cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:25.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.460+0000 7f7dc7fff700 1 --2- 172.21.15.46:0/2067283184 >> v2:172.21.15.106:3300/0 conn(0x7f7dc80c8da0 0x7f7dc80c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:25.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.460+0000 7f7dc6ffd700 1 -- 172.21.15.46:0/2067283184 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676751857 0 0) 0x7f7dc80ccc00 con 0x7f7dc8140af0 2024-01-26T23:42:25.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.460+0000 7f7dc6ffd700 1 -- 172.21.15.46:0/2067283184 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7db4003570 con 0x7f7dc8140af0 2024-01-26T23:42:25.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.461+0000 7f7dccc93700 1 -- 172.21.15.46:0/2067283184 >> v2:172.21.15.106:3300/0 conn(0x7f7dc80c8da0 msgr2=0x7f7dc80c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:25.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.461+0000 7f7dccc93700 1 --2- 172.21.15.46:0/2067283184 >> v2:172.21.15.106:3300/0 conn(0x7f7dc80c8da0 0x7f7dc80c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.461+0000 7f7dccc93700 1 -- 172.21.15.46:0/2067283184 >> v1:172.21.15.46:6789/0 conn(0x7f7dc8140af0 legacy=0x7f7dc8140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:25.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.461+0000 7f7dc6ffd700 1 -- 172.21.15.46:0/2067283184 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3490302851 0 0) 0x7f7db4003570 con 0x7f7dc8140af0 2024-01-26T23:42:25.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.461+0000 7f7dccc93700 1 -- 172.21.15.46:0/2067283184 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc80cc5b0 con 0x7f7dc80caf30 2024-01-26T23:42:25.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.461+0000 7f7dc7fff700 1 --2- 172.21.15.46:0/2067283184 >> v2:172.21.15.106:3300/0 conn(0x7f7dc80c8da0 0x7f7dc80c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:25.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.461+0000 7f7dccc93700 1 --2- 172.21.15.46:0/2067283184 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7dc80caf30 0x7f7dc80cb370 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0009e60 tx=0x7f7dc0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5fa966ed56e1789 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:25.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.461+0000 7f7dc6ffd700 1 -- 172.21.15.46:0/2067283184 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc000e070 con 0x7f7dc80caf30 2024-01-26T23:42:25.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.461+0000 7f7dc6ffd700 1 -- 172.21.15.46:0/2067283184 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc000b7a0 con 0x7f7dc80caf30 2024-01-26T23:42:25.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.461+0000 7f7dc6ffd700 1 -- 172.21.15.46:0/2067283184 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db0001470 con 0x7f7dc80caf30 2024-01-26T23:42:25.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.462+0000 7f7dce6f6700 1 -- 172.21.15.46:0/2067283184 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7dc80caf30 msgr2=0x7f7dc80cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:25.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.462+0000 7f7dce6f6700 1 --2- 172.21.15.46:0/2067283184 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7dc80caf30 0x7f7dc80cb370 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0009e60 tx=0x7f7dc0009780 comp rx=0 tx=0).stop 2024-01-26T23:42:25.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.462+0000 7f7dce6f6700 1 -- 172.21.15.46:0/2067283184 shutdown_connections 2024-01-26T23:42:25.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.462+0000 7f7dce6f6700 1 --2- 172.21.15.46:0/2067283184 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7dc80caf30 0x7f7dc80cb370 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.462+0000 7f7dce6f6700 1 --2- 172.21.15.46:0/2067283184 >> v2:172.21.15.106:3300/0 conn(0x7f7dc80c8da0 0x7f7dc80c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.462+0000 7f7dce6f6700 1 -- 172.21.15.46:0/2067283184 >> 172.21.15.46:0/2067283184 conn(0x7f7dc806bd90 msgr2=0x7f7dc806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:25.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.462+0000 7f7dce6f6700 1 -- 172.21.15.46:0/2067283184 shutdown_connections 2024-01-26T23:42:25.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.462+0000 7f7dce6f6700 1 -- 172.21.15.46:0/2067283184 wait complete. 2024-01-26T23:42:25.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.463+0000 7f7dce6f6700 1 Processor -- start 2024-01-26T23:42:25.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.463+0000 7f7dce6f6700 1 -- start start 2024-01-26T23:42:25.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.464+0000 7f7dce6f6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dc80c8da0 0x7f7dc80c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:25.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.464+0000 7f7dce6f6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc80caf30 0x7f7dc81570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:25.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.464+0000 7f7dce6f6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7dc8140af0 0x7f7dc80b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:25.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.464+0000 7f7dce6f6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc813d150 con 0x7f7dc80caf30 2024-01-26T23:42:25.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.464+0000 7f7dce6f6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc80c5990 con 0x7f7dc80c8da0 2024-01-26T23:42:25.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.464+0000 7f7dce6f6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc80b3240 con 0x7f7dc8140af0 2024-01-26T23:42:25.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.464+0000 7f7dc77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc80caf30 0x7f7dc81570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:25.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.464+0000 7f7dc77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc80caf30 0x7f7dc81570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51430/0 (socket says 172.21.15.46:51430) 2024-01-26T23:42:25.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.464+0000 7f7dc77fe700 1 -- 172.21.15.46:0/3872951358 learned_addr learned my addr 172.21.15.46:0/3872951358 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:25.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.464+0000 7f7dc7fff700 1 --2- 172.21.15.46:0/3872951358 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dc80c8da0 0x7f7dc80c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:25.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.464+0000 7f7dccc93700 1 --2- 172.21.15.46:0/3872951358 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7dc8140af0 0x7f7dc80b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:25.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.464+0000 7f7dc77fe700 1 -- 172.21.15.46:0/3872951358 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7dc8140af0 msgr2=0x7f7dc80b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:25.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.464+0000 7f7dc77fe700 1 --2- 172.21.15.46:0/3872951358 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7dc8140af0 0x7f7dc80b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.464+0000 7f7dc77fe700 1 -- 172.21.15.46:0/3872951358 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dc80c8da0 msgr2=0x7f7dc80c84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:25.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.464+0000 7f7dc77fe700 1 --2- 172.21.15.46:0/3872951358 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dc80c8da0 0x7f7dc80c84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.464+0000 7f7dc77fe700 1 -- 172.21.15.46:0/3872951358 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db8009420 con 0x7f7dc80caf30 2024-01-26T23:42:25.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.465+0000 7f7dc77fe700 1 --2- 172.21.15.46:0/3872951358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc80caf30 0x7f7dc81570f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7db000cb90 tx=0x7f7db0001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:25.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.465+0000 7f7dbffff700 1 -- 172.21.15.46:0/3872951358 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db0001d40 con 0x7f7dc80caf30 2024-01-26T23:42:25.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.465+0000 7f7dbffff700 1 -- 172.21.15.46:0/3872951358 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db0007360 con 0x7f7dc80caf30 2024-01-26T23:42:25.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.465+0000 7f7dce6f6700 1 -- 172.21.15.46:0/3872951358 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc0009a20 con 0x7f7dc80caf30 2024-01-26T23:42:25.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.465+0000 7f7dbffff700 1 -- 172.21.15.46:0/3872951358 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db0004830 con 0x7f7dc80caf30 2024-01-26T23:42:25.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.465+0000 7f7dce6f6700 1 -- 172.21.15.46:0/3872951358 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc80c5e80 con 0x7f7dc80caf30 2024-01-26T23:42:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.466+0000 7f7dce6f6700 1 -- 172.21.15.46:0/3872951358 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc8157690 con 0x7f7dc80caf30 2024-01-26T23:42:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.466+0000 7f7dbffff700 1 -- 172.21.15.46:0/3872951358 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f7db0004ba0 con 0x7f7dc80caf30 2024-01-26T23:42:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.467+0000 7f7dbffff700 1 --2- 172.21.15.46:0/3872951358 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7da8067130 0x7f7da80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.467+0000 7f7dbffff700 1 -- 172.21.15.46:0/3872951358 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f7db0084400 con 0x7f7dc80caf30 2024-01-26T23:42:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.467+0000 7f7dc7fff700 1 --2- 172.21.15.46:0/3872951358 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7da8067130 0x7f7da80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.468+0000 7f7dc7fff700 1 --2- 172.21.15.46:0/3872951358 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7da8067130 0x7f7da80695e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7db80093f0 tx=0x7f7db8008040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:25.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.471+0000 7f7dbffff700 1 -- 172.21.15.46:0/3872951358 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7db0051210 con 0x7f7dc80caf30 2024-01-26T23:42:25.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.706+0000 7f7dce6f6700 1 -- 172.21.15.46:0/3872951358 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7dc804c710 con 0x7f7da8067130 2024-01-26T23:42:25.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.715+0000 7f7dbffff700 1 -- 172.21.15.46:0/3872951358 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f7dc804c710 con 0x7f7da8067130 2024-01-26T23:42:25.715 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:25.715 INFO:teuthology.orchestra.run.smithi046.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:09", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:25.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.717+0000 7f7dce6f6700 1 -- 172.21.15.46:0/3872951358 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7da8067130 msgr2=0x7f7da80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:25.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.717+0000 7f7dce6f6700 1 --2- 172.21.15.46:0/3872951358 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7da8067130 0x7f7da80695e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7db80093f0 tx=0x7f7db8008040 comp rx=0 tx=0).stop 2024-01-26T23:42:25.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.717+0000 7f7dce6f6700 1 -- 172.21.15.46:0/3872951358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc80caf30 msgr2=0x7f7dc81570f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.717+0000 7f7dce6f6700 1 --2- 172.21.15.46:0/3872951358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc80caf30 0x7f7dc81570f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7db000cb90 tx=0x7f7db0001ab0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.717+0000 7f7dce6f6700 1 -- 172.21.15.46:0/3872951358 shutdown_connections 2024-01-26T23:42:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.717+0000 7f7dce6f6700 1 --2- 172.21.15.46:0/3872951358 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7da8067130 0x7f7da80695e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.717+0000 7f7dce6f6700 1 --2- 172.21.15.46:0/3872951358 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7dc8140af0 0x7f7dc80b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.717+0000 7f7dce6f6700 1 --2- 172.21.15.46:0/3872951358 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dc80c8da0 0x7f7dc80c84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.717+0000 7f7dce6f6700 1 --2- 172.21.15.46:0/3872951358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc80caf30 0x7f7dc81570f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.717+0000 7f7dce6f6700 1 -- 172.21.15.46:0/3872951358 >> 172.21.15.46:0/3872951358 conn(0x7f7dc806bd90 msgr2=0x7f7dc80c3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.717+0000 7f7dce6f6700 1 -- 172.21.15.46:0/3872951358 shutdown_connections 2024-01-26T23:42:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.717+0000 7f7dce6f6700 1 -- 172.21.15.46:0/3872951358 wait complete. 2024-01-26T23:42:25.731 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:09", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:25.731 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1m', 'retention': {}, 'start': '2024-01-26T00:00:00', 'created': '2024-01-26T23:31:09', 'first': '2024-01-26T23:32:00', 'last': '2024-01-26T23:42:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-26T23:42:25.731 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-26T23:42:25.731 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-26T23:42:25.731 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:25.731 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-01-26T23:42:25.757 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:25.757 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:25.757 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:25.757 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:25.757 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:25.757 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:25.757 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:25.757 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:25.757 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:25.757 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:25.757 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:25.758 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-26-23_32_00_UTC', 'scheduled-2024-01-26-23_33_00_UTC', 'scheduled-2024-01-26-23_34_00_UTC', 'scheduled-2024-01-26-23_35_00_UTC', 'scheduled-2024-01-26-23_36_00_UTC', 'scheduled-2024-01-26-23_37_00_UTC', 'scheduled-2024-01-26-23_38_00_UTC', 'scheduled-2024-01-26-23_39_00_UTC', 'scheduled-2024-01-26-23_40_00_UTC', 'scheduled-2024-01-26-23_41_00_UTC', 'scheduled-2024-01-26-23_42_00_UTC'] 2024-01-26T23:42:25.758 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir3 2024-01-26T23:42:25.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.935+0000 7fab85e67700 1 Processor -- start 2024-01-26T23:42:25.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.935+0000 7fab85e67700 1 -- start start 2024-01-26T23:42:25.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.936+0000 7fab85e67700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab80068ed0 0x7fab8006a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.936+0000 7fab85e67700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab800a9a80 con 0x7fab8006add0 2024-01-26T23:42:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.936+0000 7fab85e67700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8015a680 con 0x7fab80068b00 2024-01-26T23:42:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.936+0000 7fab85e67700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab80069320 con 0x7fab80068ed0 2024-01-26T23:42:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.936+0000 7fab7ffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fab8006add0 0x7fab800a9f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34858/0 (socket says 172.21.15.46:34858) 2024-01-26T23:42:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.936+0000 7fab7ffff700 1 -- 172.21.15.46:0/2056153069 learned_addr learned my addr 172.21.15.46:0/2056153069 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7effd700 1 --2- 172.21.15.46:0/2056153069 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab80068ed0 0x7fab8006a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7e7fc700 1 -- 172.21.15.46:0/2056153069 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317885437 0 0) 0x7fab800a9a80 con 0x7fab8006add0 2024-01-26T23:42:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7e7fc700 1 -- 172.21.15.46:0/2056153069 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab6c003570 con 0x7fab8006add0 2024-01-26T23:42:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7e7fc700 1 -- 172.21.15.46:0/2056153069 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834003098 0 0) 0x7fab8015a680 con 0x7fab80068b00 2024-01-26T23:42:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7e7fc700 1 -- 172.21.15.46:0/2056153069 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab800a9a80 con 0x7fab80068b00 2024-01-26T23:42:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7e7fc700 1 -- 172.21.15.46:0/2056153069 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 173763231 0 0) 0x7fab6c003570 con 0x7fab8006add0 2024-01-26T23:42:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7e7fc700 1 -- 172.21.15.46:0/2056153069 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab80068ed0 msgr2=0x7fab8006a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7e7fc700 1 --2- 172.21.15.46:0/2056153069 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab80068ed0 0x7fab8006a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7e7fc700 1 -- 172.21.15.46:0/2056153069 >> v1:172.21.15.106:6789/0 conn(0x7fab80068b00 legacy=0x7fab80159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7e7fc700 1 -- 172.21.15.46:0/2056153069 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8015a960 con 0x7fab8006add0 2024-01-26T23:42:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7e7fc700 1 -- 172.21.15.46:0/2056153069 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fab780017f0 con 0x7fab8006add0 2024-01-26T23:42:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7effd700 1 --2- 172.21.15.46:0/2056153069 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab80068ed0 0x7fab8006a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7e7fc700 1 -- 172.21.15.46:0/2056153069 >> v1:172.21.15.46:6789/0 conn(0x7fab8006add0 legacy=0x7fab800a9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7e7fc700 1 --2- 172.21.15.46:0/2056153069 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c003c20 0x7fab6c004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7e7fc700 1 --2- 172.21.15.46:0/2056153069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab6c004810 0x7fab6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7e7fc700 1 --2- 172.21.15.46:0/2056153069 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab80068ed0 0x7fab6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7e7fc700 1 -- 172.21.15.46:0/2056153069 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab80069320 con 0x7fab6c004810 2024-01-26T23:42:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7e7fc700 1 -- 172.21.15.46:0/2056153069 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c00bb10 con 0x7fab6c003c20 2024-01-26T23:42:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7e7fc700 1 -- 172.21.15.46:0/2056153069 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c00bcd0 con 0x7fab80068ed0 2024-01-26T23:42:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7effd700 1 --2- 172.21.15.46:0/2056153069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab6c004810 0x7fab6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7f7fe700 1 --2- 172.21.15.46:0/2056153069 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c003c20 0x7fab6c004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7ffff700 1 --2- 172.21.15.46:0/2056153069 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab80068ed0 0x7fab6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7effd700 1 -- 172.21.15.46:0/2056153069 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab80068ed0 msgr2=0x7fab6c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7effd700 1 --2- 172.21.15.46:0/2056153069 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab80068ed0 0x7fab6c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7effd700 1 -- 172.21.15.46:0/2056153069 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c003c20 msgr2=0x7fab6c004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7effd700 1 --2- 172.21.15.46:0/2056153069 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab6c003c20 0x7fab6c004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.937+0000 7fab7effd700 1 -- 172.21.15.46:0/2056153069 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab68009420 con 0x7fab6c004810 2024-01-26T23:42:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.938+0000 7fab7ffff700 1 --2- 172.21.15.46:0/2056153069 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab80068ed0 0x7fab6c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.938+0000 7fab7effd700 1 --2- 172.21.15.46:0/2056153069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab6c004810 0x7fab6c004c80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fab68009df0 tx=0x7fab680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e8c539b635a8780 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.938+0000 7fab7e7fc700 1 -- 172.21.15.46:0/2056153069 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68009740 con 0x7fab6c004810 2024-01-26T23:42:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.938+0000 7fab7e7fc700 1 -- 172.21.15.46:0/2056153069 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6800e920 con 0x7fab6c004810 2024-01-26T23:42:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.938+0000 7fab7e7fc700 1 -- 172.21.15.46:0/2056153069 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6800ece0 con 0x7fab6c004810 2024-01-26T23:42:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.938+0000 7fab85e67700 1 -- 172.21.15.46:0/2056153069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab6c004810 msgr2=0x7fab6c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.938+0000 7fab85e67700 1 --2- 172.21.15.46:0/2056153069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab6c004810 0x7fab6c004c80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fab68009df0 tx=0x7fab680091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.938+0000 7fab7ffff700 1 -- 172.21.15.46:0/2056153069 reap_dead start 2024-01-26T23:42:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.938+0000 7fab85e67700 1 -- 172.21.15.46:0/2056153069 shutdown_connections 2024-01-26T23:42:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.938+0000 7fab85e67700 1 -- 172.21.15.46:0/2056153069 >> 172.21.15.46:0/2056153069 conn(0x7fab800ca100 msgr2=0x7fab8006e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.938+0000 7fab85e67700 1 -- 172.21.15.46:0/2056153069 shutdown_connections 2024-01-26T23:42:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.938+0000 7fab85e67700 1 -- 172.21.15.46:0/2056153069 wait complete. 2024-01-26T23:42:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.940+0000 7fab85e67700 1 Processor -- start 2024-01-26T23:42:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.940+0000 7fab85e67700 1 -- start start 2024-01-26T23:42:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.941+0000 7fab85e67700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab6c004810 0x7fab80168ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.941+0000 7fab85e67700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab800b0ae0 0x7fab800ae190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.941+0000 7fab85e67700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab800afbc0 0x7fab800ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.941+0000 7fab85e67700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab800aec10 con 0x7fab800afbc0 2024-01-26T23:42:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.941+0000 7fab85e67700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab800ac560 con 0x7fab800b0ae0 2024-01-26T23:42:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.941+0000 7fab85e67700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab800ac230 con 0x7fab6c004810 2024-01-26T23:42:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.942+0000 7fab7ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab800afbc0 0x7fab800ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.942+0000 7fab7ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab800afbc0 0x7fab800ae6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51448/0 (socket says 172.21.15.46:51448) 2024-01-26T23:42:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.942+0000 7fab7ffff700 1 -- 172.21.15.46:0/1396765154 learned_addr learned my addr 172.21.15.46:0/1396765154 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.942+0000 7fab7effd700 1 --2- 172.21.15.46:0/1396765154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab800b0ae0 0x7fab800ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.942+0000 7fab7f7fe700 1 --2- 172.21.15.46:0/1396765154 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab6c004810 0x7fab80168ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.942+0000 7fab7ffff700 1 -- 172.21.15.46:0/1396765154 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab6c004810 msgr2=0x7fab80168ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.942+0000 7fab7ffff700 1 --2- 172.21.15.46:0/1396765154 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab6c004810 0x7fab80168ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.942+0000 7fab7ffff700 1 -- 172.21.15.46:0/1396765154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab800b0ae0 msgr2=0x7fab800ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.942+0000 7fab7ffff700 1 --2- 172.21.15.46:0/1396765154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab800b0ae0 0x7fab800ae190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.942+0000 7fab7ffff700 1 -- 172.21.15.46:0/1396765154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab800b0fe0 con 0x7fab800afbc0 2024-01-26T23:42:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.942+0000 7fab7effd700 1 --2- 172.21.15.46:0/1396765154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab800b0ae0 0x7fab800ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.942+0000 7fab7ffff700 1 --2- 172.21.15.46:0/1396765154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab800afbc0 0x7fab800ae6d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fab8006b1a0 tx=0x7fab78001540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.942+0000 7fab777fe700 1 -- 172.21.15.46:0/1396765154 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7800ebc0 con 0x7fab800afbc0 2024-01-26T23:42:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.942+0000 7fab777fe700 1 -- 172.21.15.46:0/1396765154 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7800ed40 con 0x7fab800afbc0 2024-01-26T23:42:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.942+0000 7fab85e67700 1 -- 172.21.15.46:0/1396765154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab800b12c0 con 0x7fab800afbc0 2024-01-26T23:42:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.942+0000 7fab777fe700 1 -- 172.21.15.46:0/1396765154 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78013670 con 0x7fab800afbc0 2024-01-26T23:42:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.942+0000 7fab85e67700 1 -- 172.21.15.46:0/1396765154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab80169020 con 0x7fab800afbc0 2024-01-26T23:42:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.943+0000 7fab85e67700 1 -- 172.21.15.46:0/1396765154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab800ac050 con 0x7fab800afbc0 2024-01-26T23:42:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.944+0000 7fab777fe700 1 -- 172.21.15.46:0/1396765154 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fab78013aa0 con 0x7fab800afbc0 2024-01-26T23:42:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.944+0000 7fab777fe700 1 --2- 172.21.15.46:0/1396765154 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fab60067130 0x7fab600695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.944+0000 7fab777fe700 1 -- 172.21.15.46:0/1396765154 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fab78085220 con 0x7fab800afbc0 2024-01-26T23:42:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.944+0000 7fab7f7fe700 1 --2- 172.21.15.46:0/1396765154 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fab60067130 0x7fab600695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.945+0000 7fab7f7fe700 1 --2- 172.21.15.46:0/1396765154 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fab60067130 0x7fab600695e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fab70001130 tx=0x7fab70009040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:25.949+0000 7fab777fe700 1 -- 172.21.15.46:0/1396765154 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fab78051f10 con 0x7fab800afbc0 2024-01-26T23:42:26.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.180+0000 7fab85e67700 1 -- 172.21.15.46:0/1396765154 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fab800ac6d0 con 0x7fab60067130 2024-01-26T23:42:26.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.189+0000 7fab777fe700 1 -- 172.21.15.46:0/1396765154 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fab800ac6d0 con 0x7fab60067130 2024-01-26T23:42:26.189 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:26.190 INFO:teuthology.orchestra.run.smithi046.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:09", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:26.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.191+0000 7fab85e67700 1 -- 172.21.15.46:0/1396765154 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fab60067130 msgr2=0x7fab600695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:26.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.191+0000 7fab85e67700 1 --2- 172.21.15.46:0/1396765154 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fab60067130 0x7fab600695e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fab70001130 tx=0x7fab70009040 comp rx=0 tx=0).stop 2024-01-26T23:42:26.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.191+0000 7fab85e67700 1 -- 172.21.15.46:0/1396765154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab800afbc0 msgr2=0x7fab800ae6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:26.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.191+0000 7fab85e67700 1 --2- 172.21.15.46:0/1396765154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab800afbc0 0x7fab800ae6d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fab8006b1a0 tx=0x7fab78001540 comp rx=0 tx=0).stop 2024-01-26T23:42:26.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.192+0000 7fab85e67700 1 -- 172.21.15.46:0/1396765154 shutdown_connections 2024-01-26T23:42:26.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.192+0000 7fab85e67700 1 --2- 172.21.15.46:0/1396765154 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fab60067130 0x7fab600695e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.192+0000 7fab85e67700 1 --2- 172.21.15.46:0/1396765154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab800afbc0 0x7fab800ae6d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.192+0000 7fab85e67700 1 --2- 172.21.15.46:0/1396765154 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab6c004810 0x7fab80168ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.192+0000 7fab85e67700 1 --2- 172.21.15.46:0/1396765154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab800b0ae0 0x7fab800ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.192+0000 7fab85e67700 1 -- 172.21.15.46:0/1396765154 >> 172.21.15.46:0/1396765154 conn(0x7fab800ca100 msgr2=0x7fab8006b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:26.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.192+0000 7fab85e67700 1 -- 172.21.15.46:0/1396765154 shutdown_connections 2024-01-26T23:42:26.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.192+0000 7fab85e67700 1 -- 172.21.15.46:0/1396765154 wait complete. 2024-01-26T23:42:26.205 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:09", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:26.205 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1m', 'retention': {}, 'start': '2024-01-26T00:00:00', 'created': '2024-01-26T23:31:09', 'first': '2024-01-26T23:32:00', 'last': '2024-01-26T23:42:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-26T23:42:26.206 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-26T23:42:26.206 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-26T23:42:26.206 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:26.206 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-01-26T23:42:26.229 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:26.229 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:26.229 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:26.229 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:26.229 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:26.230 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:26.230 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:26.230 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:26.230 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:26.230 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:26.230 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_01_UTC 2024-01-26T23:42:26.231 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-26-23_32_00_UTC', 'scheduled-2024-01-26-23_33_00_UTC', 'scheduled-2024-01-26-23_34_00_UTC', 'scheduled-2024-01-26-23_35_00_UTC', 'scheduled-2024-01-26-23_36_00_UTC', 'scheduled-2024-01-26-23_37_00_UTC', 'scheduled-2024-01-26-23_38_00_UTC', 'scheduled-2024-01-26-23_39_00_UTC', 'scheduled-2024-01-26-23_40_00_UTC', 'scheduled-2024-01-26-23_41_00_UTC', 'scheduled-2024-01-26-23_42_01_UTC'] 2024-01-26T23:42:26.231 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir4 2024-01-26T23:42:26.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.409+0000 7f314764c700 1 Processor -- start 2024-01-26T23:42:26.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.410+0000 7f314764c700 1 -- start start 2024-01-26T23:42:26.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f314764c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3140159510 0x7f31401598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:26.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f314764c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31400ca220 0x7f31400c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:26.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f314764c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f31400ca690 0x7f31400c9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:26.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f314764c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3140069260 con 0x7f3140159510 2024-01-26T23:42:26.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f314764c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f314015d670 con 0x7f31400ca690 2024-01-26T23:42:26.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f314764c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31400c9750 con 0x7f31400ca220 2024-01-26T23:42:26.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f31453e8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3140159510 0x7f31401598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:26.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f31453e8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3140159510 0x7f31401598e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51454/0 (socket says 172.21.15.46:51454) 2024-01-26T23:42:26.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f31453e8700 1 -- 172.21.15.46:0/2988441438 learned_addr learned my addr 172.21.15.46:0/2988441438 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:26.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f3144be7700 1 --2- 172.21.15.46:0/2988441438 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31400ca220 0x7f31400c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:26.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f3145be9700 1 --2- 172.21.15.46:0/2988441438 >> v2:172.21.15.106:3300/0 conn(0x7f31400ca690 0x7f31400c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:26.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f31453e8700 1 -- 172.21.15.46:0/2988441438 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31400ca220 msgr2=0x7f31400c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:26.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f31453e8700 1 --2- 172.21.15.46:0/2988441438 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31400ca220 0x7f31400c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f31453e8700 1 -- 172.21.15.46:0/2988441438 >> v2:172.21.15.106:3300/0 conn(0x7f31400ca690 msgr2=0x7f31400c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:26.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f31453e8700 1 --2- 172.21.15.46:0/2988441438 >> v2:172.21.15.106:3300/0 conn(0x7f31400ca690 0x7f31400c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f31453e8700 1 -- 172.21.15.46:0/2988441438 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31400c9a30 con 0x7f3140159510 2024-01-26T23:42:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f3144be7700 1 --2- 172.21.15.46:0/2988441438 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31400ca220 0x7f31400c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f3145be9700 1 --2- 172.21.15.46:0/2988441438 >> v2:172.21.15.106:3300/0 conn(0x7f31400ca690 0x7f31400c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.411+0000 7f31453e8700 1 --2- 172.21.15.46:0/2988441438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3140159510 0x7f31401598e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f3140159e20 tx=0x7f3130009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f83d832abbeabc33 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.412+0000 7f3137fff700 1 -- 172.21.15.46:0/2988441438 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313000b860 con 0x7f3140159510 2024-01-26T23:42:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.412+0000 7f3137fff700 1 -- 172.21.15.46:0/2988441438 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313000b9c0 con 0x7f3140159510 2024-01-26T23:42:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.412+0000 7f3137fff700 1 -- 172.21.15.46:0/2988441438 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313000bd60 con 0x7f3140159510 2024-01-26T23:42:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.412+0000 7f314764c700 1 -- 172.21.15.46:0/2988441438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3140159510 msgr2=0x7f31401598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.412+0000 7f314764c700 1 --2- 172.21.15.46:0/2988441438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3140159510 0x7f31401598e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f3140159e20 tx=0x7f3130009250 comp rx=0 tx=0).stop 2024-01-26T23:42:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.412+0000 7f314764c700 1 -- 172.21.15.46:0/2988441438 shutdown_connections 2024-01-26T23:42:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.412+0000 7f314764c700 1 --2- 172.21.15.46:0/2988441438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3140159510 0x7f31401598e0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.412+0000 7f314764c700 1 --2- 172.21.15.46:0/2988441438 >> v2:172.21.15.106:3300/0 conn(0x7f31400ca690 0x7f31400c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.412+0000 7f314764c700 1 --2- 172.21.15.46:0/2988441438 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31400ca220 0x7f31400c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.412+0000 7f314764c700 1 -- 172.21.15.46:0/2988441438 >> 172.21.15.46:0/2988441438 conn(0x7f314006bd90 msgr2=0x7f314006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:26.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.412+0000 7f314764c700 1 -- 172.21.15.46:0/2988441438 shutdown_connections 2024-01-26T23:42:26.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.412+0000 7f314764c700 1 -- 172.21.15.46:0/2988441438 wait complete. 2024-01-26T23:42:26.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.414+0000 7f314764c700 1 Processor -- start 2024-01-26T23:42:26.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.414+0000 7f314764c700 1 -- start start 2024-01-26T23:42:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.414+0000 7f314764c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31400ca220 0x7f3140152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f314764c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3140159510 0x7f314015c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f314764c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31400c5650 0x7f3140158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f314764c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31401526d0 con 0x7f31400c5650 2024-01-26T23:42:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f314764c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31400c5b50 con 0x7f3140159510 2024-01-26T23:42:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f314764c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31400c5e30 con 0x7f31400ca220 2024-01-26T23:42:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f3145be9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31400c5650 0x7f3140158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f3145be9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31400c5650 0x7f3140158270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51462/0 (socket says 172.21.15.46:51462) 2024-01-26T23:42:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f3145be9700 1 -- 172.21.15.46:0/2908041727 learned_addr learned my addr 172.21.15.46:0/2908041727 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f31453e8700 1 --2- 172.21.15.46:0/2908041727 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31400ca220 0x7f3140152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f3144be7700 1 --2- 172.21.15.46:0/2908041727 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3140159510 0x7f314015c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f3145be9700 1 -- 172.21.15.46:0/2908041727 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31400ca220 msgr2=0x7f3140152190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f3145be9700 1 --2- 172.21.15.46:0/2908041727 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31400ca220 0x7f3140152190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f3145be9700 1 -- 172.21.15.46:0/2908041727 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3140159510 msgr2=0x7f314015c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f3145be9700 1 --2- 172.21.15.46:0/2908041727 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3140159510 0x7f314015c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f3145be9700 1 -- 172.21.15.46:0/2908041727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31300094f0 con 0x7f31400c5650 2024-01-26T23:42:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f3145be9700 1 --2- 172.21.15.46:0/2908041727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31400c5650 0x7f3140158270 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f313800d0d0 tx=0x7f31380099d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f31357fa700 1 -- 172.21.15.46:0/2908041727 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313800dd70 con 0x7f31400c5650 2024-01-26T23:42:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f31357fa700 1 -- 172.21.15.46:0/2908041727 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313800f040 con 0x7f31400c5650 2024-01-26T23:42:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f314764c700 1 -- 172.21.15.46:0/2908041727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31400c6110 con 0x7f31400c5650 2024-01-26T23:42:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f31357fa700 1 -- 172.21.15.46:0/2908041727 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3138015670 con 0x7f31400c5650 2024-01-26T23:42:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.415+0000 7f314764c700 1 -- 172.21.15.46:0/2908041727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f314015c7b0 con 0x7f31400c5650 2024-01-26T23:42:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.417+0000 7f314764c700 1 -- 172.21.15.46:0/2908041727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3140159ee0 con 0x7f31400c5650 2024-01-26T23:42:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.417+0000 7f31357fa700 1 -- 172.21.15.46:0/2908041727 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f313801e3b0 con 0x7f31400c5650 2024-01-26T23:42:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.417+0000 7f31357fa700 1 --2- 172.21.15.46:0/2908041727 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3124067130 0x7f31240695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.417+0000 7f31357fa700 1 -- 172.21.15.46:0/2908041727 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f31380851e0 con 0x7f31400c5650 2024-01-26T23:42:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.417+0000 7f31453e8700 1 --2- 172.21.15.46:0/2908041727 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3124067130 0x7f31240695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:26.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.418+0000 7f31453e8700 1 --2- 172.21.15.46:0/2908041727 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3124067130 0x7f31240695e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3140159e20 tx=0x7f3130017000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:26.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.422+0000 7f31357fa700 1 -- 172.21.15.46:0/2908041727 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3138051e60 con 0x7f31400c5650 2024-01-26T23:42:26.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.652+0000 7f314764c700 1 -- 172.21.15.46:0/2908041727 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f314004c710 con 0x7f3124067130 2024-01-26T23:42:26.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.662+0000 7f31357fa700 1 -- 172.21.15.46:0/2908041727 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f314004c710 con 0x7f3124067130 2024-01-26T23:42:26.662 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:26.663 INFO:teuthology.orchestra.run.smithi046.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:10", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:26.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.665+0000 7f314764c700 1 -- 172.21.15.46:0/2908041727 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3124067130 msgr2=0x7f31240695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:26.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.665+0000 7f314764c700 1 --2- 172.21.15.46:0/2908041727 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3124067130 0x7f31240695e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3140159e20 tx=0x7f3130017000 comp rx=0 tx=0).stop 2024-01-26T23:42:26.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.665+0000 7f314764c700 1 -- 172.21.15.46:0/2908041727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31400c5650 msgr2=0x7f3140158270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:26.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.665+0000 7f314764c700 1 --2- 172.21.15.46:0/2908041727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31400c5650 0x7f3140158270 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f313800d0d0 tx=0x7f31380099d0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.665+0000 7f314764c700 1 -- 172.21.15.46:0/2908041727 shutdown_connections 2024-01-26T23:42:26.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.665+0000 7f314764c700 1 --2- 172.21.15.46:0/2908041727 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3124067130 0x7f31240695e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.665+0000 7f314764c700 1 --2- 172.21.15.46:0/2908041727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31400c5650 0x7f3140158270 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.665+0000 7f314764c700 1 --2- 172.21.15.46:0/2908041727 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31400ca220 0x7f3140152190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.665+0000 7f314764c700 1 --2- 172.21.15.46:0/2908041727 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3140159510 0x7f314015c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.665+0000 7f314764c700 1 -- 172.21.15.46:0/2908041727 >> 172.21.15.46:0/2908041727 conn(0x7f314006bd90 msgr2=0x7f31401529d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:26.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.665+0000 7f314764c700 1 -- 172.21.15.46:0/2908041727 shutdown_connections 2024-01-26T23:42:26.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.665+0000 7f314764c700 1 -- 172.21.15.46:0/2908041727 wait complete. 2024-01-26T23:42:26.679 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:10", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:26.679 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1m', 'retention': {}, 'start': '2024-01-26T00:00:00', 'created': '2024-01-26T23:31:10', 'first': '2024-01-26T23:32:00', 'last': '2024-01-26T23:42:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-26T23:42:26.679 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-26T23:42:26.679 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-26T23:42:26.679 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:26.679 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-01-26T23:42:26.703 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:26.703 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:26.703 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:26.703 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:26.703 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:26.703 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:26.704 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:26.704 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:26.704 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:26.704 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:26.704 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:26.704 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-26-23_32_00_UTC', 'scheduled-2024-01-26-23_33_00_UTC', 'scheduled-2024-01-26-23_34_00_UTC', 'scheduled-2024-01-26-23_35_00_UTC', 'scheduled-2024-01-26-23_36_00_UTC', 'scheduled-2024-01-26-23_37_00_UTC', 'scheduled-2024-01-26-23_38_00_UTC', 'scheduled-2024-01-26-23_39_00_UTC', 'scheduled-2024-01-26-23_40_00_UTC', 'scheduled-2024-01-26-23_41_00_UTC', 'scheduled-2024-01-26-23_42_00_UTC'] 2024-01-26T23:42:26.704 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir5 2024-01-26T23:42:26.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.880+0000 7f4cf4aa8700 1 Processor -- start 2024-01-26T23:42:26.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.880+0000 7f4cf4aa8700 1 -- start start 2024-01-26T23:42:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.881+0000 7f4cf4aa8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4cf015cd40 0x7f4cf00cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.881+0000 7f4cf4aa8700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4cf015d110 0x7f4cf00cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.881+0000 7f4cf4aa8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cf006caa0 con 0x7f4cf0067930 2024-01-26T23:42:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.881+0000 7f4cf4aa8700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cf00c7ac0 con 0x7f4cf015cd40 2024-01-26T23:42:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.881+0000 7f4cf4aa8700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cf00ca8b0 con 0x7f4cf015d110 2024-01-26T23:42:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.881+0000 7f4cee59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4cf015cd40 0x7f4cf00cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.881+0000 7f4ceed9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4cf0067930 0x7f4cf015d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34864/0 (socket says 172.21.15.46:34864) 2024-01-26T23:42:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.881+0000 7f4cee59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4cf015cd40 0x7f4cf00cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51474/0 (socket says 172.21.15.46:51474) 2024-01-26T23:42:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.882+0000 7f4ceed9d700 1 -- 172.21.15.46:0/2141618814 learned_addr learned my addr 172.21.15.46:0/2141618814 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.882+0000 7f4cedd9b700 1 --2- 172.21.15.46:0/2141618814 >> v2:172.21.15.106:3300/0 conn(0x7f4cf015d110 0x7f4cf00cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.882+0000 7f4cee59c700 1 -- 172.21.15.46:0/2141618814 >> v2:172.21.15.106:3300/0 conn(0x7f4cf015d110 msgr2=0x7f4cf00cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.882+0000 7f4cee59c700 1 --2- 172.21.15.46:0/2141618814 >> v2:172.21.15.106:3300/0 conn(0x7f4cf015d110 0x7f4cf00cb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.882+0000 7f4cee59c700 1 -- 172.21.15.46:0/2141618814 >> v1:172.21.15.46:6789/0 conn(0x7f4cf0067930 legacy=0x7f4cf015d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:26.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.882+0000 7f4cee59c700 1 -- 172.21.15.46:0/2141618814 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf006c1b0 con 0x7f4cf015cd40 2024-01-26T23:42:26.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.882+0000 7f4ced59a700 1 -- 172.21.15.46:0/2141618814 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850078429 0 0) 0x7f4cf006caa0 con 0x7f4cf0067930 2024-01-26T23:42:26.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.882+0000 7f4cee59c700 1 --2- 172.21.15.46:0/2141618814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf015cd40 0x7f4cf00cb260 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4cf00c7a70 tx=0x7f4ce40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a73edf778848cf server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:26.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.882+0000 7f4ced59a700 1 -- 172.21.15.46:0/2141618814 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4011630 con 0x7f4cf015cd40 2024-01-26T23:42:26.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.882+0000 7f4ced59a700 1 -- 172.21.15.46:0/2141618814 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce4011790 con 0x7f4cf015cd40 2024-01-26T23:42:26.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.882+0000 7f4ced59a700 1 -- 172.21.15.46:0/2141618814 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4011b30 con 0x7f4cf015cd40 2024-01-26T23:42:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.883+0000 7f4cf4aa8700 1 -- 172.21.15.46:0/2141618814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf015cd40 msgr2=0x7f4cf00cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.883+0000 7f4cf4aa8700 1 --2- 172.21.15.46:0/2141618814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf015cd40 0x7f4cf00cb260 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4cf00c7a70 tx=0x7f4ce40091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.883+0000 7f4cf4aa8700 1 -- 172.21.15.46:0/2141618814 shutdown_connections 2024-01-26T23:42:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.883+0000 7f4cf4aa8700 1 --2- 172.21.15.46:0/2141618814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf015cd40 0x7f4cf00cb260 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.883+0000 7f4cf4aa8700 1 --2- 172.21.15.46:0/2141618814 >> v2:172.21.15.106:3300/0 conn(0x7f4cf015d110 0x7f4cf00cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.883+0000 7f4cf4aa8700 1 -- 172.21.15.46:0/2141618814 >> 172.21.15.46:0/2141618814 conn(0x7f4cf00cdf50 msgr2=0x7f4cf00c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.883+0000 7f4cf4aa8700 1 -- 172.21.15.46:0/2141618814 shutdown_connections 2024-01-26T23:42:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.883+0000 7f4cf4aa8700 1 -- 172.21.15.46:0/2141618814 wait complete. 2024-01-26T23:42:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.884+0000 7f4cf4aa8700 1 Processor -- start 2024-01-26T23:42:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.884+0000 7f4cf4aa8700 1 -- start start 2024-01-26T23:42:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4cf4aa8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf0067930 0x7f4cf00c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4cf4aa8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf015cd40 0x7f4cf00b2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4cf4aa8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf015d110 0x7f4cf0145d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4cf4aa8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf015b020 con 0x7f4cf0067930 2024-01-26T23:42:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4cf4aa8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf016dba0 con 0x7f4cf015cd40 2024-01-26T23:42:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4cf4aa8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf015a6f0 con 0x7f4cf015d110 2024-01-26T23:42:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4cee59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf0067930 0x7f4cf00c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4cee59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf0067930 0x7f4cf00c7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51478/0 (socket says 172.21.15.46:51478) 2024-01-26T23:42:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4cee59c700 1 -- 172.21.15.46:0/3612449077 learned_addr learned my addr 172.21.15.46:0/3612449077 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4ceed9d700 1 --2- 172.21.15.46:0/3612449077 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf015d110 0x7f4cf0145d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4cedd9b700 1 --2- 172.21.15.46:0/3612449077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf015cd40 0x7f4cf00b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4cee59c700 1 -- 172.21.15.46:0/3612449077 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf015d110 msgr2=0x7f4cf0145d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4cee59c700 1 --2- 172.21.15.46:0/3612449077 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf015d110 0x7f4cf0145d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4cee59c700 1 -- 172.21.15.46:0/3612449077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf015cd40 msgr2=0x7f4cf00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4cee59c700 1 --2- 172.21.15.46:0/3612449077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf015cd40 0x7f4cf00b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4cee59c700 1 -- 172.21.15.46:0/3612449077 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce4009420 con 0x7f4cf0067930 2024-01-26T23:42:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4ceed9d700 1 --2- 172.21.15.46:0/3612449077 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf015d110 0x7f4cf0145d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:42:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4cedd9b700 1 --2- 172.21.15.46:0/3612449077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf015cd40 0x7f4cf00b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:42:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.885+0000 7f4cee59c700 1 --2- 172.21.15.46:0/3612449077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf0067930 0x7f4cf00c7e50 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4cf006e800 tx=0x7f4ce4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.886+0000 7f4cde7fc700 1 -- 172.21.15.46:0/3612449077 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4011900 con 0x7f4cf0067930 2024-01-26T23:42:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.886+0000 7f4cde7fc700 1 -- 172.21.15.46:0/3612449077 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce4011a60 con 0x7f4cf0067930 2024-01-26T23:42:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.886+0000 7f4cf4aa8700 1 -- 172.21.15.46:0/3612449077 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf015a9d0 con 0x7f4cf0067930 2024-01-26T23:42:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.886+0000 7f4cde7fc700 1 -- 172.21.15.46:0/3612449077 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4025670 con 0x7f4cf0067930 2024-01-26T23:42:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.886+0000 7f4cf4aa8700 1 -- 172.21.15.46:0/3612449077 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf015a3a0 con 0x7f4cf0067930 2024-01-26T23:42:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.887+0000 7f4cf4aa8700 1 -- 172.21.15.46:0/3612449077 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf004c710 con 0x7f4cf0067930 2024-01-26T23:42:26.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.887+0000 7f4cde7fc700 1 -- 172.21.15.46:0/3612449077 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f4ce4023070 con 0x7f4cf0067930 2024-01-26T23:42:26.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.888+0000 7f4cde7fc700 1 --2- 172.21.15.46:0/3612449077 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4cd0067130 0x7f4cd00695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:26.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.888+0000 7f4cde7fc700 1 -- 172.21.15.46:0/3612449077 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f4ce40860b0 con 0x7f4cf0067930 2024-01-26T23:42:26.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.888+0000 7f4cedd9b700 1 --2- 172.21.15.46:0/3612449077 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4cd0067130 0x7f4cd00695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:26.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.888+0000 7f4cedd9b700 1 --2- 172.21.15.46:0/3612449077 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4cd0067130 0x7f4cd00695e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0006f00 tx=0x7f4ce0008040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:26.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:26.892+0000 7f4cde7fc700 1 -- 172.21.15.46:0/3612449077 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4ce4052e40 con 0x7f4cf0067930 2024-01-26T23:42:27.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.124+0000 7f4cf4aa8700 1 -- 172.21.15.46:0/3612449077 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4cf015b160 con 0x7f4cd0067130 2024-01-26T23:42:27.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.132+0000 7f4cde7fc700 1 -- 172.21.15.46:0/3612449077 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f4ce0009dd0 con 0x7f4cd0067130 2024-01-26T23:42:27.133 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:27.133 INFO:teuthology.orchestra.run.smithi046.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:10", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:27.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.135+0000 7f4cf4aa8700 1 -- 172.21.15.46:0/3612449077 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4cd0067130 msgr2=0x7f4cd00695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:27.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.135+0000 7f4cf4aa8700 1 --2- 172.21.15.46:0/3612449077 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4cd0067130 0x7f4cd00695e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0006f00 tx=0x7f4ce0008040 comp rx=0 tx=0).stop 2024-01-26T23:42:27.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.135+0000 7f4cf4aa8700 1 -- 172.21.15.46:0/3612449077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf0067930 msgr2=0x7f4cf00c7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:27.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.135+0000 7f4cf4aa8700 1 --2- 172.21.15.46:0/3612449077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf0067930 0x7f4cf00c7e50 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4cf006e800 tx=0x7f4ce4017000 comp rx=0 tx=0).stop 2024-01-26T23:42:27.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.135+0000 7f4cf4aa8700 1 -- 172.21.15.46:0/3612449077 shutdown_connections 2024-01-26T23:42:27.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.135+0000 7f4cf4aa8700 1 --2- 172.21.15.46:0/3612449077 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4cd0067130 0x7f4cd00695e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.135+0000 7f4cf4aa8700 1 --2- 172.21.15.46:0/3612449077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cf0067930 0x7f4cf00c7e50 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.135+0000 7f4cf4aa8700 1 --2- 172.21.15.46:0/3612449077 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf015d110 0x7f4cf0145d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.135+0000 7f4cf4aa8700 1 --2- 172.21.15.46:0/3612449077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf015cd40 0x7f4cf00b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.135+0000 7f4cf4aa8700 1 -- 172.21.15.46:0/3612449077 >> 172.21.15.46:0/3612449077 conn(0x7f4cf00cdf50 msgr2=0x7f4cf006cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:27.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.135+0000 7f4cf4aa8700 1 -- 172.21.15.46:0/3612449077 shutdown_connections 2024-01-26T23:42:27.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.135+0000 7f4cf4aa8700 1 -- 172.21.15.46:0/3612449077 wait complete. 2024-01-26T23:42:27.148 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:10", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:27.149 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1m', 'retention': {}, 'start': '2024-01-26T00:00:00', 'created': '2024-01-26T23:31:10', 'first': '2024-01-26T23:32:00', 'last': '2024-01-26T23:42:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-26T23:42:27.149 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-26T23:42:27.149 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-26T23:42:27.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:27.149 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-01-26T23:42:27.172 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:27.175 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:27.175 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:27.175 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:27.175 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:27.175 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:27.175 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:27.175 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:27.175 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:27.175 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:27.175 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:27.176 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-26-23_32_00_UTC', 'scheduled-2024-01-26-23_33_00_UTC', 'scheduled-2024-01-26-23_34_00_UTC', 'scheduled-2024-01-26-23_35_00_UTC', 'scheduled-2024-01-26-23_36_00_UTC', 'scheduled-2024-01-26-23_37_00_UTC', 'scheduled-2024-01-26-23_38_00_UTC', 'scheduled-2024-01-26-23_39_00_UTC', 'scheduled-2024-01-26-23_40_00_UTC', 'scheduled-2024-01-26-23_41_00_UTC', 'scheduled-2024-01-26-23_42_00_UTC'] 2024-01-26T23:42:27.177 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir6 2024-01-26T23:42:27.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.354+0000 7fb9147c5700 1 Processor -- start 2024-01-26T23:42:27.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.354+0000 7fb9147c5700 1 -- start start 2024-01-26T23:42:27.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.355+0000 7fb9147c5700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb90c15cd40 0x7fb90c0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:27.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.356+0000 7fb9147c5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb90c15d110 0x7fb90c0cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:27.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.356+0000 7fb9147c5700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90c06caa0 con 0x7fb90c067930 2024-01-26T23:42:27.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.356+0000 7fb9147c5700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb90c0c7ac0 con 0x7fb90c15cd40 2024-01-26T23:42:27.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.356+0000 7fb9147c5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c0ca8b0 con 0x7fb90c15d110 2024-01-26T23:42:27.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.356+0000 7fb912561700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb90c15cd40 0x7fb90c0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:27.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.356+0000 7fb912561700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb90c15cd40 0x7fb90c0cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51486/0 (socket says 172.21.15.46:51486) 2024-01-26T23:42:27.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.356+0000 7fb912561700 1 -- 172.21.15.46:0/2108338482 learned_addr learned my addr 172.21.15.46:0/2108338482 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:27.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.356+0000 7fb911d60700 1 --2- 172.21.15.46:0/2108338482 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb90c15d110 0x7fb90c0cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:27.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.356+0000 7fb912561700 1 -- 172.21.15.46:0/2108338482 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb90c15d110 msgr2=0x7fb90c0cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:27.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.356+0000 7fb912561700 1 --2- 172.21.15.46:0/2108338482 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb90c15d110 0x7fb90c0cb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.356+0000 7fb912561700 1 -- 172.21.15.46:0/2108338482 >> v1:172.21.15.106:6789/0 conn(0x7fb90c067930 legacy=0x7fb90c15d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:27.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.356+0000 7fb912561700 1 -- 172.21.15.46:0/2108338482 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90c06c1b0 con 0x7fb90c15cd40 2024-01-26T23:42:27.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.356+0000 7fb911d60700 1 --2- 172.21.15.46:0/2108338482 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb90c15d110 0x7fb90c0cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:27.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.356+0000 7fb912561700 1 --2- 172.21.15.46:0/2108338482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c15cd40 0x7fb90c0cb260 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb9000099b0 tx=0x7fb900009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef08f6e6aef526b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:27.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.356+0000 7fb91155f700 1 -- 172.21.15.46:0/2108338482 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90000b860 con 0x7fb90c15cd40 2024-01-26T23:42:27.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.356+0000 7fb91155f700 1 -- 172.21.15.46:0/2108338482 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90000b9c0 con 0x7fb90c15cd40 2024-01-26T23:42:27.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.356+0000 7fb91155f700 1 -- 172.21.15.46:0/2108338482 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90000bd60 con 0x7fb90c15cd40 2024-01-26T23:42:27.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.357+0000 7fb9147c5700 1 -- 172.21.15.46:0/2108338482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c15cd40 msgr2=0x7fb90c0cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:27.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.357+0000 7fb9147c5700 1 --2- 172.21.15.46:0/2108338482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c15cd40 0x7fb90c0cb260 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb9000099b0 tx=0x7fb900009250 comp rx=0 tx=0).stop 2024-01-26T23:42:27.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.357+0000 7fb9147c5700 1 -- 172.21.15.46:0/2108338482 shutdown_connections 2024-01-26T23:42:27.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.357+0000 7fb9147c5700 1 --2- 172.21.15.46:0/2108338482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c15cd40 0x7fb90c0cb260 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.357+0000 7fb9147c5700 1 --2- 172.21.15.46:0/2108338482 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb90c15d110 0x7fb90c0cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.357+0000 7fb9147c5700 1 -- 172.21.15.46:0/2108338482 >> 172.21.15.46:0/2108338482 conn(0x7fb90c0cdf50 msgr2=0x7fb90c0c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:27.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.357+0000 7fb9147c5700 1 -- 172.21.15.46:0/2108338482 shutdown_connections 2024-01-26T23:42:27.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.357+0000 7fb9147c5700 1 -- 172.21.15.46:0/2108338482 wait complete. 2024-01-26T23:42:27.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.358+0000 7fb9147c5700 1 Processor -- start 2024-01-26T23:42:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.358+0000 7fb9147c5700 1 -- start start 2024-01-26T23:42:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.359+0000 7fb9147c5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb90c067930 0x7fb90c168fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.359+0000 7fb9147c5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb90c15cd40 0x7fb90c0c5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.359+0000 7fb9147c5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c15d110 0x7fb90c157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.359+0000 7fb9147c5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c0c3660 con 0x7fb90c15d110 2024-01-26T23:42:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.359+0000 7fb9147c5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c0c3380 con 0x7fb90c067930 2024-01-26T23:42:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.359+0000 7fb9147c5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c0c4660 con 0x7fb90c15cd40 2024-01-26T23:42:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.359+0000 7fb912d62700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c15d110 0x7fb90c157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.359+0000 7fb912d62700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c15d110 0x7fb90c157cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51498/0 (socket says 172.21.15.46:51498) 2024-01-26T23:42:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.359+0000 7fb912d62700 1 -- 172.21.15.46:0/3599480502 learned_addr learned my addr 172.21.15.46:0/3599480502 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.359+0000 7fb912561700 1 --2- 172.21.15.46:0/3599480502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb90c067930 0x7fb90c168fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.359+0000 7fb911d60700 1 --2- 172.21.15.46:0/3599480502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb90c15cd40 0x7fb90c0c5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.360+0000 7fb912d62700 1 -- 172.21.15.46:0/3599480502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb90c15cd40 msgr2=0x7fb90c0c5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:27.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.360+0000 7fb912d62700 1 --2- 172.21.15.46:0/3599480502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb90c15cd40 0x7fb90c0c5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.360+0000 7fb912d62700 1 -- 172.21.15.46:0/3599480502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb90c067930 msgr2=0x7fb90c168fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:27.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.360+0000 7fb912d62700 1 --2- 172.21.15.46:0/3599480502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb90c067930 0x7fb90c168fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.360+0000 7fb912d62700 1 -- 172.21.15.46:0/3599480502 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9000094f0 con 0x7fb90c15d110 2024-01-26T23:42:27.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.360+0000 7fb912d62700 1 --2- 172.21.15.46:0/3599480502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c15d110 0x7fb90c157cc0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb90400c8e0 tx=0x7fb904001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:27.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.360+0000 7fb8fe7fc700 1 -- 172.21.15.46:0/3599480502 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb904012630 con 0x7fb90c15d110 2024-01-26T23:42:27.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.360+0000 7fb8fe7fc700 1 -- 172.21.15.46:0/3599480502 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb904012790 con 0x7fb90c15d110 2024-01-26T23:42:27.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.360+0000 7fb9147c5700 1 -- 172.21.15.46:0/3599480502 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90c0c4940 con 0x7fb90c15d110 2024-01-26T23:42:27.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.360+0000 7fb8fe7fc700 1 -- 172.21.15.46:0/3599480502 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb904012b50 con 0x7fb90c15d110 2024-01-26T23:42:27.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.360+0000 7fb9147c5700 1 -- 172.21.15.46:0/3599480502 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90c0c5680 con 0x7fb90c15d110 2024-01-26T23:42:27.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.361+0000 7fb9147c5700 1 -- 172.21.15.46:0/3599480502 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb90c04c710 con 0x7fb90c15d110 2024-01-26T23:42:27.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.362+0000 7fb8fe7fc700 1 -- 172.21.15.46:0/3599480502 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fb90401a3a0 con 0x7fb90c15d110 2024-01-26T23:42:27.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.362+0000 7fb8fe7fc700 1 --2- 172.21.15.46:0/3599480502 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8f0067130 0x7fb8f00695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:27.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.362+0000 7fb8fe7fc700 1 -- 172.21.15.46:0/3599480502 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fb904084d30 con 0x7fb90c15d110 2024-01-26T23:42:27.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.362+0000 7fb912561700 1 --2- 172.21.15.46:0/3599480502 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8f0067130 0x7fb8f00695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:27.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.363+0000 7fb912561700 1 --2- 172.21.15.46:0/3599480502 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8f0067130 0x7fb8f00695e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb9000099b0 tx=0x7fb900017000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:27.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.367+0000 7fb8fe7fc700 1 -- 172.21.15.46:0/3599480502 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb904051ac0 con 0x7fb90c15d110 2024-01-26T23:42:27.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.597+0000 7fb9147c5700 1 -- 172.21.15.46:0/3599480502 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb90c0c47d0 con 0x7fb8f0067130 2024-01-26T23:42:27.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.606+0000 7fb8fe7fc700 1 -- 172.21.15.46:0/3599480502 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fb90c0c47d0 con 0x7fb8f0067130 2024-01-26T23:42:27.607 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:27.607 INFO:teuthology.orchestra.run.smithi046.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:11", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:27.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.608+0000 7fb9147c5700 1 -- 172.21.15.46:0/3599480502 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8f0067130 msgr2=0x7fb8f00695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:27.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.608+0000 7fb9147c5700 1 --2- 172.21.15.46:0/3599480502 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8f0067130 0x7fb8f00695e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb9000099b0 tx=0x7fb900017000 comp rx=0 tx=0).stop 2024-01-26T23:42:27.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.608+0000 7fb9147c5700 1 -- 172.21.15.46:0/3599480502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c15d110 msgr2=0x7fb90c157cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:27.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.608+0000 7fb9147c5700 1 --2- 172.21.15.46:0/3599480502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c15d110 0x7fb90c157cc0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb90400c8e0 tx=0x7fb904001710 comp rx=0 tx=0).stop 2024-01-26T23:42:27.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.608+0000 7fb9147c5700 1 -- 172.21.15.46:0/3599480502 shutdown_connections 2024-01-26T23:42:27.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.608+0000 7fb9147c5700 1 --2- 172.21.15.46:0/3599480502 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8f0067130 0x7fb8f00695e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.608+0000 7fb9147c5700 1 --2- 172.21.15.46:0/3599480502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c15d110 0x7fb90c157cc0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.608+0000 7fb9147c5700 1 --2- 172.21.15.46:0/3599480502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb90c067930 0x7fb90c168fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.608+0000 7fb9147c5700 1 --2- 172.21.15.46:0/3599480502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb90c15cd40 0x7fb90c0c5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.608+0000 7fb9147c5700 1 -- 172.21.15.46:0/3599480502 >> 172.21.15.46:0/3599480502 conn(0x7fb90c0cdf50 msgr2=0x7fb90c06f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:27.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.608+0000 7fb9147c5700 1 -- 172.21.15.46:0/3599480502 shutdown_connections 2024-01-26T23:42:27.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.608+0000 7fb9147c5700 1 -- 172.21.15.46:0/3599480502 wait complete. 2024-01-26T23:42:27.622 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:11", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:27.622 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1m', 'retention': {}, 'start': '2024-01-26T00:00:00', 'created': '2024-01-26T23:31:11', 'first': '2024-01-26T23:32:00', 'last': '2024-01-26T23:42:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-26T23:42:27.622 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-26T23:42:27.622 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-26T23:42:27.623 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:27.623 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-01-26T23:42:27.647 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:27.647 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:27.647 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:27.647 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:27.647 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:27.647 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:27.647 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:27.647 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:27.647 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:27.647 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:27.647 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:27.648 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-26-23_32_00_UTC', 'scheduled-2024-01-26-23_33_00_UTC', 'scheduled-2024-01-26-23_34_00_UTC', 'scheduled-2024-01-26-23_35_00_UTC', 'scheduled-2024-01-26-23_36_00_UTC', 'scheduled-2024-01-26-23_37_00_UTC', 'scheduled-2024-01-26-23_38_00_UTC', 'scheduled-2024-01-26-23_39_00_UTC', 'scheduled-2024-01-26-23_40_00_UTC', 'scheduled-2024-01-26-23_41_00_UTC', 'scheduled-2024-01-26-23_42_00_UTC'] 2024-01-26T23:42:27.648 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir7 2024-01-26T23:42:27.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.825+0000 7f8a87234700 1 Processor -- start 2024-01-26T23:42:27.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.825+0000 7f8a87234700 1 -- start start 2024-01-26T23:42:27.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.826+0000 7f8a87234700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8a800c8da0 0x7f8a800c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:27.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.826+0000 7f8a87234700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8a80140af0 0x7f8a80140f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:27.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.826+0000 7f8a87234700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a800ccc00 con 0x7f8a800caf30 2024-01-26T23:42:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.826+0000 7f8a87234700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a800cbbf0 con 0x7f8a800c8da0 2024-01-26T23:42:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.826+0000 7f8a87234700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a800cc310 con 0x7f8a80140af0 2024-01-26T23:42:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.826+0000 7f8a84fd0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8a800c8da0 0x7f8a800c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.827+0000 7f8a84fd0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8a800c8da0 0x7f8a800c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51500/0 (socket says 172.21.15.46:51500) 2024-01-26T23:42:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.827+0000 7f8a84fd0700 1 -- 172.21.15.46:0/513756831 learned_addr learned my addr 172.21.15.46:0/513756831 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.827+0000 7f8a7ffff700 1 --2- 172.21.15.46:0/513756831 >> v2:172.21.15.106:3300/0 conn(0x7f8a80140af0 0x7f8a80140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.827+0000 7f8a84fd0700 1 -- 172.21.15.46:0/513756831 >> v2:172.21.15.106:3300/0 conn(0x7f8a80140af0 msgr2=0x7f8a80140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.827+0000 7f8a84fd0700 1 --2- 172.21.15.46:0/513756831 >> v2:172.21.15.106:3300/0 conn(0x7f8a80140af0 0x7f8a80140f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.827+0000 7f8a84fd0700 1 -- 172.21.15.46:0/513756831 >> v1:172.21.15.106:6789/0 conn(0x7f8a800caf30 legacy=0x7f8a800cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.827+0000 7f8a84fd0700 1 -- 172.21.15.46:0/513756831 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a801414d0 con 0x7f8a800c8da0 2024-01-26T23:42:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.827+0000 7f8a7ffff700 1 --2- 172.21.15.46:0/513756831 >> v2:172.21.15.106:3300/0 conn(0x7f8a80140af0 0x7f8a80140f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:42:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.827+0000 7f8a84fd0700 1 --2- 172.21.15.46:0/513756831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a800c8da0 0x7f8a800c9170 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8a800cc120 tx=0x7f8a74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d29fd1f4aac644d server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.827+0000 7f8a7f7fe700 1 -- 172.21.15.46:0/513756831 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7400b820 con 0x7f8a800c8da0 2024-01-26T23:42:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.827+0000 7f8a7f7fe700 1 -- 172.21.15.46:0/513756831 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7400b980 con 0x7f8a800c8da0 2024-01-26T23:42:27.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.827+0000 7f8a7f7fe700 1 -- 172.21.15.46:0/513756831 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7400bd20 con 0x7f8a800c8da0 2024-01-26T23:42:27.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.828+0000 7f8a87234700 1 -- 172.21.15.46:0/513756831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a800c8da0 msgr2=0x7f8a800c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:27.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.828+0000 7f8a87234700 1 --2- 172.21.15.46:0/513756831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a800c8da0 0x7f8a800c9170 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8a800cc120 tx=0x7f8a74009250 comp rx=0 tx=0).stop 2024-01-26T23:42:27.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.828+0000 7f8a87234700 1 -- 172.21.15.46:0/513756831 shutdown_connections 2024-01-26T23:42:27.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.828+0000 7f8a87234700 1 --2- 172.21.15.46:0/513756831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a800c8da0 0x7f8a800c9170 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.828+0000 7f8a87234700 1 --2- 172.21.15.46:0/513756831 >> v2:172.21.15.106:3300/0 conn(0x7f8a80140af0 0x7f8a80140f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.828+0000 7f8a87234700 1 -- 172.21.15.46:0/513756831 >> 172.21.15.46:0/513756831 conn(0x7f8a8006bd90 msgr2=0x7f8a8006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:27.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.828+0000 7f8a87234700 1 -- 172.21.15.46:0/513756831 shutdown_connections 2024-01-26T23:42:27.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.828+0000 7f8a87234700 1 -- 172.21.15.46:0/513756831 wait complete. 2024-01-26T23:42:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.829+0000 7f8a87234700 1 Processor -- start 2024-01-26T23:42:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.829+0000 7f8a87234700 1 -- start start 2024-01-26T23:42:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.830+0000 7f8a87234700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a800c8da0 0x7f8a80168100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.830+0000 7f8a87234700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a800caf30 0x7f8a800c5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.830+0000 7f8a87234700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a80140af0 0x7f8a800c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.830+0000 7f8a87234700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800c6140 con 0x7f8a800caf30 2024-01-26T23:42:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.830+0000 7f8a87234700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800c6420 con 0x7f8a80140af0 2024-01-26T23:42:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.830+0000 7f8a87234700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800c6700 con 0x7f8a800c8da0 2024-01-26T23:42:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.830+0000 7f8a7ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a800caf30 0x7f8a800c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.830+0000 7f8a7ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a800caf30 0x7f8a800c5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51504/0 (socket says 172.21.15.46:51504) 2024-01-26T23:42:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.830+0000 7f8a7ffff700 1 -- 172.21.15.46:0/561783902 learned_addr learned my addr 172.21.15.46:0/561783902 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.830+0000 7f8a84fd0700 1 --2- 172.21.15.46:0/561783902 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a800c8da0 0x7f8a80168100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.830+0000 7f8a857d1700 1 --2- 172.21.15.46:0/561783902 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a80140af0 0x7f8a800c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.831+0000 7f8a7ffff700 1 -- 172.21.15.46:0/561783902 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a800c8da0 msgr2=0x7f8a80168100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.831+0000 7f8a7ffff700 1 --2- 172.21.15.46:0/561783902 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a800c8da0 0x7f8a80168100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.831+0000 7f8a7ffff700 1 -- 172.21.15.46:0/561783902 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a80140af0 msgr2=0x7f8a800c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.831+0000 7f8a7ffff700 1 --2- 172.21.15.46:0/561783902 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a80140af0 0x7f8a800c7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.831+0000 7f8a7ffff700 1 -- 172.21.15.46:0/561783902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a740094f0 con 0x7f8a800caf30 2024-01-26T23:42:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.831+0000 7f8a7ffff700 1 --2- 172.21.15.46:0/561783902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a800caf30 0x7f8a800c5840 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c00c850 tx=0x7f8a6c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:27.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.831+0000 7f8a7cff9700 1 -- 172.21.15.46:0/561783902 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c011600 con 0x7f8a800caf30 2024-01-26T23:42:27.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.831+0000 7f8a7cff9700 1 -- 172.21.15.46:0/561783902 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6c011760 con 0x7f8a800caf30 2024-01-26T23:42:27.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.831+0000 7f8a87234700 1 -- 172.21.15.46:0/561783902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a800c69e0 con 0x7f8a800caf30 2024-01-26T23:42:27.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.831+0000 7f8a7cff9700 1 -- 172.21.15.46:0/561783902 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c011b00 con 0x7f8a800caf30 2024-01-26T23:42:27.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.831+0000 7f8a87234700 1 -- 172.21.15.46:0/561783902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a80168640 con 0x7f8a800caf30 2024-01-26T23:42:27.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.832+0000 7f8a87234700 1 -- 172.21.15.46:0/561783902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a8004c710 con 0x7f8a800caf30 2024-01-26T23:42:27.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.833+0000 7f8a7cff9700 1 -- 172.21.15.46:0/561783902 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f8a6c018350 con 0x7f8a800caf30 2024-01-26T23:42:27.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.833+0000 7f8a7cff9700 1 --2- 172.21.15.46:0/561783902 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8a64067130 0x7f8a640695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:27.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.833+0000 7f8a7cff9700 1 -- 172.21.15.46:0/561783902 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f8a6c084ad0 con 0x7f8a800caf30 2024-01-26T23:42:27.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.833+0000 7f8a84fd0700 1 --2- 172.21.15.46:0/561783902 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8a64067130 0x7f8a640695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:27.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.834+0000 7f8a84fd0700 1 --2- 172.21.15.46:0/561783902 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8a64067130 0x7f8a640695e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8a800cc120 tx=0x7f8a74016000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:27.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:27.837+0000 7f8a7cff9700 1 -- 172.21.15.46:0/561783902 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8a6c051750 con 0x7f8a800caf30 2024-01-26T23:42:28.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.070+0000 7f8a87234700 1 -- 172.21.15.46:0/561783902 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a800cc5b0 con 0x7f8a64067130 2024-01-26T23:42:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.078+0000 7f8a7cff9700 1 -- 172.21.15.46:0/561783902 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f8a800cc5b0 con 0x7f8a64067130 2024-01-26T23:42:28.078 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:28.079 INFO:teuthology.orchestra.run.smithi046.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:12", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:28.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.080+0000 7f8a87234700 1 -- 172.21.15.46:0/561783902 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8a64067130 msgr2=0x7f8a640695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:28.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.080+0000 7f8a87234700 1 --2- 172.21.15.46:0/561783902 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8a64067130 0x7f8a640695e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8a800cc120 tx=0x7f8a74016000 comp rx=0 tx=0).stop 2024-01-26T23:42:28.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.080+0000 7f8a87234700 1 -- 172.21.15.46:0/561783902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a800caf30 msgr2=0x7f8a800c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:28.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.080+0000 7f8a87234700 1 --2- 172.21.15.46:0/561783902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a800caf30 0x7f8a800c5840 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c00c850 tx=0x7f8a6c008040 comp rx=0 tx=0).stop 2024-01-26T23:42:28.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.080+0000 7f8a87234700 1 -- 172.21.15.46:0/561783902 shutdown_connections 2024-01-26T23:42:28.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.080+0000 7f8a87234700 1 --2- 172.21.15.46:0/561783902 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8a64067130 0x7f8a640695e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.080+0000 7f8a87234700 1 --2- 172.21.15.46:0/561783902 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a80140af0 0x7f8a800c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.080+0000 7f8a87234700 1 --2- 172.21.15.46:0/561783902 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a800c8da0 0x7f8a80168100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.080+0000 7f8a87234700 1 --2- 172.21.15.46:0/561783902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a800caf30 0x7f8a800c5840 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.080+0000 7f8a87234700 1 -- 172.21.15.46:0/561783902 >> 172.21.15.46:0/561783902 conn(0x7f8a8006bd90 msgr2=0x7f8a8013d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:28.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.081+0000 7f8a87234700 1 -- 172.21.15.46:0/561783902 shutdown_connections 2024-01-26T23:42:28.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.081+0000 7f8a87234700 1 -- 172.21.15.46:0/561783902 wait complete. 2024-01-26T23:42:28.094 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:12", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:28.094 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1m', 'retention': {}, 'start': '2024-01-26T00:00:00', 'created': '2024-01-26T23:31:12', 'first': '2024-01-26T23:32:00', 'last': '2024-01-26T23:42:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-26T23:42:28.094 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-26T23:42:28.094 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-26T23:42:28.094 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:28.095 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-01-26T23:42:28.118 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:28.119 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:28.119 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:28.119 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:28.119 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:28.119 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:28.119 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:28.119 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:28.119 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:28.119 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:28.119 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:28.119 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-26-23_32_00_UTC', 'scheduled-2024-01-26-23_33_00_UTC', 'scheduled-2024-01-26-23_34_00_UTC', 'scheduled-2024-01-26-23_35_00_UTC', 'scheduled-2024-01-26-23_36_00_UTC', 'scheduled-2024-01-26-23_37_00_UTC', 'scheduled-2024-01-26-23_38_00_UTC', 'scheduled-2024-01-26-23_39_00_UTC', 'scheduled-2024-01-26-23_40_00_UTC', 'scheduled-2024-01-26-23_41_00_UTC', 'scheduled-2024-01-26-23_42_00_UTC'] 2024-01-26T23:42:28.119 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir8 2024-01-26T23:42:28.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.297+0000 7fb03416c700 1 Processor -- start 2024-01-26T23:42:28.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.297+0000 7fb03416c700 1 -- start start 2024-01-26T23:42:28.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.298+0000 7fb03416c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c157860 0x7fb02c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:28.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.298+0000 7fb03416c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb02c06ddb0 0x7fb02c0cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:28.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.298+0000 7fb03416c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb02c0cea80 con 0x7fb02c157490 2024-01-26T23:42:28.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.298+0000 7fb03416c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02c0cdc60 con 0x7fb02c06ddb0 2024-01-26T23:42:28.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.298+0000 7fb03416c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c0b2ae0 con 0x7fb02c157860 2024-01-26T23:42:28.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.298+0000 7fb031f08700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb02c157490 0x7fb02c06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34866/0 (socket says 172.21.15.46:34866) 2024-01-26T23:42:28.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.298+0000 7fb031f08700 1 -- 172.21.15.46:0/3558954753 learned_addr learned my addr 172.21.15.46:0/3558954753 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:28.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.298+0000 7fb031707700 1 --2- 172.21.15.46:0/3558954753 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c157860 0x7fb02c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:28.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.298+0000 7fb032709700 1 --2- 172.21.15.46:0/3558954753 >> v2:172.21.15.106:3300/0 conn(0x7fb02c06ddb0 0x7fb02c0cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:28.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.298+0000 7fb030f06700 1 -- 172.21.15.46:0/3558954753 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439043687 0 0) 0x7fb02c0cea80 con 0x7fb02c157490 2024-01-26T23:42:28.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.298+0000 7fb030f06700 1 -- 172.21.15.46:0/3558954753 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb018003570 con 0x7fb02c157490 2024-01-26T23:42:28.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.299+0000 7fb031707700 1 -- 172.21.15.46:0/3558954753 >> v2:172.21.15.106:3300/0 conn(0x7fb02c06ddb0 msgr2=0x7fb02c0cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:28.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.299+0000 7fb031707700 1 --2- 172.21.15.46:0/3558954753 >> v2:172.21.15.106:3300/0 conn(0x7fb02c06ddb0 0x7fb02c0cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.299+0000 7fb031707700 1 -- 172.21.15.46:0/3558954753 >> v1:172.21.15.46:6789/0 conn(0x7fb02c157490 legacy=0x7fb02c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:28.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.299+0000 7fb031707700 1 -- 172.21.15.46:0/3558954753 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02c0cde70 con 0x7fb02c157860 2024-01-26T23:42:28.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.299+0000 7fb032709700 1 --2- 172.21.15.46:0/3558954753 >> v2:172.21.15.106:3300/0 conn(0x7fb02c06ddb0 0x7fb02c0cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:28.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.299+0000 7fb031707700 1 --2- 172.21.15.46:0/3558954753 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c157860 0x7fb02c0cc600 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0098e0 tx=0x7fb01c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbb549a4c71a2a8d server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:28.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.299+0000 7fb030f06700 1 -- 172.21.15.46:0/3558954753 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c011630 con 0x7fb02c157860 2024-01-26T23:42:28.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.299+0000 7fb030f06700 1 -- 172.21.15.46:0/3558954753 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01c011790 con 0x7fb02c157860 2024-01-26T23:42:28.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.299+0000 7fb030f06700 1 -- 172.21.15.46:0/3558954753 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c011b30 con 0x7fb02c157860 2024-01-26T23:42:28.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.300+0000 7fb03416c700 1 -- 172.21.15.46:0/3558954753 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c157860 msgr2=0x7fb02c0cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:28.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.300+0000 7fb03416c700 1 --2- 172.21.15.46:0/3558954753 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c157860 0x7fb02c0cc600 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0098e0 tx=0x7fb01c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.300+0000 7fb03416c700 1 -- 172.21.15.46:0/3558954753 shutdown_connections 2024-01-26T23:42:28.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.300+0000 7fb03416c700 1 --2- 172.21.15.46:0/3558954753 >> v2:172.21.15.106:3300/0 conn(0x7fb02c06ddb0 0x7fb02c0cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.300+0000 7fb03416c700 1 --2- 172.21.15.46:0/3558954753 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c157860 0x7fb02c0cc600 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.300+0000 7fb03416c700 1 -- 172.21.15.46:0/3558954753 >> 172.21.15.46:0/3558954753 conn(0x7fb02c067930 msgr2=0x7fb02c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.300+0000 7fb03416c700 1 -- 172.21.15.46:0/3558954753 shutdown_connections 2024-01-26T23:42:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.300+0000 7fb03416c700 1 -- 172.21.15.46:0/3558954753 wait complete. 2024-01-26T23:42:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.301+0000 7fb03416c700 1 Processor -- start 2024-01-26T23:42:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.301+0000 7fb03416c700 1 -- start start 2024-01-26T23:42:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.302+0000 7fb03416c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c06ddb0 0x7fb02c0c3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.302+0000 7fb03416c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c157490 0x7fb02c153230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.302+0000 7fb03416c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c157860 0x7fb02c0c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.302+0000 7fb03416c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c153770 con 0x7fb02c157490 2024-01-26T23:42:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.302+0000 7fb03416c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c0c8960 con 0x7fb02c157860 2024-01-26T23:42:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.302+0000 7fb03416c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c0c7d50 con 0x7fb02c06ddb0 2024-01-26T23:42:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.302+0000 7fb031707700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c157490 0x7fb02c153230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.302+0000 7fb031707700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c157490 0x7fb02c153230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51510/0 (socket says 172.21.15.46:51510) 2024-01-26T23:42:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.302+0000 7fb031707700 1 -- 172.21.15.46:0/3186930309 learned_addr learned my addr 172.21.15.46:0/3186930309 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:28.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.302+0000 7fb032709700 1 --2- 172.21.15.46:0/3186930309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c157860 0x7fb02c0c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:28.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.302+0000 7fb031f08700 1 --2- 172.21.15.46:0/3186930309 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c06ddb0 0x7fb02c0c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:28.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.303+0000 7fb031707700 1 -- 172.21.15.46:0/3186930309 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c06ddb0 msgr2=0x7fb02c0c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.303+0000 7fb031707700 1 --2- 172.21.15.46:0/3186930309 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c06ddb0 0x7fb02c0c3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.303+0000 7fb031707700 1 -- 172.21.15.46:0/3186930309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c157860 msgr2=0x7fb02c0c72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.303+0000 7fb031707700 1 --2- 172.21.15.46:0/3186930309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c157860 0x7fb02c0c72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.303+0000 7fb031707700 1 -- 172.21.15.46:0/3186930309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb024009a20 con 0x7fb02c157490 2024-01-26T23:42:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.303+0000 7fb032709700 1 --2- 172.21.15.46:0/3186930309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c157860 0x7fb02c0c72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.303+0000 7fb031707700 1 --2- 172.21.15.46:0/3186930309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c157490 0x7fb02c153230 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb01c013000 tx=0x7fb01c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.303+0000 7fb021ffb700 1 -- 172.21.15.46:0/3186930309 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c011b10 con 0x7fb02c157490 2024-01-26T23:42:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.303+0000 7fb021ffb700 1 -- 172.21.15.46:0/3186930309 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01c0253f0 con 0x7fb02c157490 2024-01-26T23:42:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.303+0000 7fb03416c700 1 -- 172.21.15.46:0/3186930309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01c009420 con 0x7fb02c157490 2024-01-26T23:42:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.303+0000 7fb021ffb700 1 -- 172.21.15.46:0/3186930309 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c0257b0 con 0x7fb02c157490 2024-01-26T23:42:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.303+0000 7fb03416c700 1 -- 172.21.15.46:0/3186930309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb02c0c8310 con 0x7fb02c157490 2024-01-26T23:42:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.304+0000 7fb03416c700 1 -- 172.21.15.46:0/3186930309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02c0c8ad0 con 0x7fb02c157490 2024-01-26T23:42:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.305+0000 7fb021ffb700 1 -- 172.21.15.46:0/3186930309 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fb01c025b20 con 0x7fb02c157490 2024-01-26T23:42:28.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.305+0000 7fb021ffb700 1 --2- 172.21.15.46:0/3186930309 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb010067130 0x7fb0100695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:28.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.305+0000 7fb021ffb700 1 -- 172.21.15.46:0/3186930309 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fb01c0862e0 con 0x7fb02c157490 2024-01-26T23:42:28.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.305+0000 7fb031f08700 1 --2- 172.21.15.46:0/3186930309 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb010067130 0x7fb0100695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:28.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.306+0000 7fb031f08700 1 --2- 172.21.15.46:0/3186930309 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb010067130 0x7fb0100695e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb0280058d0 tx=0x7fb028009040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:28.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.310+0000 7fb021ffb700 1 -- 172.21.15.46:0/3186930309 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb01c052fd0 con 0x7fb02c157490 2024-01-26T23:42:28.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.541+0000 7fb03416c700 1 -- 172.21.15.46:0/3186930309 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb02c04c710 con 0x7fb010067130 2024-01-26T23:42:28.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.551+0000 7fb021ffb700 1 -- 172.21.15.46:0/3186930309 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fb02c04c710 con 0x7fb010067130 2024-01-26T23:42:28.551 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:28.551 INFO:teuthology.orchestra.run.smithi046.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:12", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:28.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.553+0000 7fb03416c700 1 -- 172.21.15.46:0/3186930309 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb010067130 msgr2=0x7fb0100695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:28.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.553+0000 7fb03416c700 1 --2- 172.21.15.46:0/3186930309 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb010067130 0x7fb0100695e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb0280058d0 tx=0x7fb028009040 comp rx=0 tx=0).stop 2024-01-26T23:42:28.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.553+0000 7fb03416c700 1 -- 172.21.15.46:0/3186930309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c157490 msgr2=0x7fb02c153230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:28.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.553+0000 7fb03416c700 1 --2- 172.21.15.46:0/3186930309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c157490 0x7fb02c153230 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb01c013000 tx=0x7fb01c017000 comp rx=0 tx=0).stop 2024-01-26T23:42:28.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.553+0000 7fb03416c700 1 -- 172.21.15.46:0/3186930309 shutdown_connections 2024-01-26T23:42:28.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.553+0000 7fb03416c700 1 --2- 172.21.15.46:0/3186930309 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb010067130 0x7fb0100695e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.553+0000 7fb03416c700 1 --2- 172.21.15.46:0/3186930309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c157860 0x7fb02c0c72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.553+0000 7fb03416c700 1 --2- 172.21.15.46:0/3186930309 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c06ddb0 0x7fb02c0c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.553+0000 7fb03416c700 1 --2- 172.21.15.46:0/3186930309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c157490 0x7fb02c153230 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.553+0000 7fb03416c700 1 -- 172.21.15.46:0/3186930309 >> 172.21.15.46:0/3186930309 conn(0x7fb02c067930 msgr2=0x7fb02c154f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:28.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.553+0000 7fb03416c700 1 -- 172.21.15.46:0/3186930309 shutdown_connections 2024-01-26T23:42:28.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.553+0000 7fb03416c700 1 -- 172.21.15.46:0/3186930309 wait complete. 2024-01-26T23:42:28.567 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:12", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:28.567 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1m', 'retention': {}, 'start': '2024-01-26T00:00:00', 'created': '2024-01-26T23:31:12', 'first': '2024-01-26T23:32:00', 'last': '2024-01-26T23:42:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-26T23:42:28.567 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-26T23:42:28.567 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-26T23:42:28.567 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:28.567 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-01-26T23:42:28.591 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:28.591 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:28.591 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:28.591 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:28.591 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:28.591 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:28.591 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:28.591 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:28.591 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:28.591 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:28.591 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:28.592 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-26-23_32_00_UTC', 'scheduled-2024-01-26-23_33_00_UTC', 'scheduled-2024-01-26-23_34_00_UTC', 'scheduled-2024-01-26-23_35_00_UTC', 'scheduled-2024-01-26-23_36_00_UTC', 'scheduled-2024-01-26-23_37_00_UTC', 'scheduled-2024-01-26-23_38_00_UTC', 'scheduled-2024-01-26-23_39_00_UTC', 'scheduled-2024-01-26-23_40_00_UTC', 'scheduled-2024-01-26-23_41_00_UTC', 'scheduled-2024-01-26-23_42_00_UTC'] 2024-01-26T23:42:28.592 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir9 2024-01-26T23:42:28.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.769+0000 7fae0a8f4700 1 Processor -- start 2024-01-26T23:42:28.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.769+0000 7fae0a8f4700 1 -- start start 2024-01-26T23:42:28.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.770+0000 7fae0a8f4700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fae040c8da0 0x7fae040c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:28.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.770+0000 7fae0a8f4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fae04140af0 0x7fae04140f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:28.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.770+0000 7fae0a8f4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae040caf30 0x7fae040cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:28.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.770+0000 7fae0a8f4700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae040cbbf0 con 0x7fae04140af0 2024-01-26T23:42:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.770+0000 7fae0a8f4700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae040ccc00 con 0x7fae040c8da0 2024-01-26T23:42:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.770+0000 7fae0a8f4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae040cc310 con 0x7fae040caf30 2024-01-26T23:42:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.770+0000 7fae037fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fae04140af0 0x7fae04140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.770+0000 7fae037fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fae04140af0 0x7fae04140f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51524/0 (socket says 172.21.15.46:51524) 2024-01-26T23:42:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.770+0000 7fae037fe700 1 -- 172.21.15.46:0/3271206626 learned_addr learned my addr 172.21.15.46:0/3271206626 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.771+0000 7fae03fff700 1 --2- 172.21.15.46:0/3271206626 >> v2:172.21.15.106:3300/0 conn(0x7fae040c8da0 0x7fae040c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.771+0000 7fae08e91700 1 --2- 172.21.15.46:0/3271206626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae040caf30 0x7fae040cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.771+0000 7fae037fe700 1 -- 172.21.15.46:0/3271206626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae040caf30 msgr2=0x7fae040cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.771+0000 7fae037fe700 1 --2- 172.21.15.46:0/3271206626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae040caf30 0x7fae040cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.771+0000 7fae037fe700 1 -- 172.21.15.46:0/3271206626 >> v2:172.21.15.106:3300/0 conn(0x7fae040c8da0 msgr2=0x7fae040c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.771+0000 7fae037fe700 1 --2- 172.21.15.46:0/3271206626 >> v2:172.21.15.106:3300/0 conn(0x7fae040c8da0 0x7fae040c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.771+0000 7fae037fe700 1 -- 172.21.15.46:0/3271206626 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae041414d0 con 0x7fae04140af0 2024-01-26T23:42:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.771+0000 7fae037fe700 1 --2- 172.21.15.46:0/3271206626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae04140af0 0x7fae04140f90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fae0406f100 tx=0x7fadf8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2713020008f9f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.771+0000 7fae02ffd700 1 -- 172.21.15.46:0/3271206626 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf800b860 con 0x7fae04140af0 2024-01-26T23:42:28.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.771+0000 7fae02ffd700 1 -- 172.21.15.46:0/3271206626 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf800b9c0 con 0x7fae04140af0 2024-01-26T23:42:28.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.771+0000 7fae02ffd700 1 -- 172.21.15.46:0/3271206626 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf800bd60 con 0x7fae04140af0 2024-01-26T23:42:28.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.772+0000 7fae0a8f4700 1 -- 172.21.15.46:0/3271206626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae04140af0 msgr2=0x7fae04140f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:28.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.772+0000 7fae0a8f4700 1 --2- 172.21.15.46:0/3271206626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae04140af0 0x7fae04140f90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fae0406f100 tx=0x7fadf8009250 comp rx=0 tx=0).stop 2024-01-26T23:42:28.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.772+0000 7fae0a8f4700 1 -- 172.21.15.46:0/3271206626 shutdown_connections 2024-01-26T23:42:28.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.772+0000 7fae0a8f4700 1 --2- 172.21.15.46:0/3271206626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae040caf30 0x7fae040cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.772+0000 7fae0a8f4700 1 --2- 172.21.15.46:0/3271206626 >> v2:172.21.15.106:3300/0 conn(0x7fae040c8da0 0x7fae040c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.772+0000 7fae0a8f4700 1 --2- 172.21.15.46:0/3271206626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae04140af0 0x7fae04140f90 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.772+0000 7fae0a8f4700 1 -- 172.21.15.46:0/3271206626 >> 172.21.15.46:0/3271206626 conn(0x7fae0406bd90 msgr2=0x7fae0406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:28.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.772+0000 7fae0a8f4700 1 -- 172.21.15.46:0/3271206626 shutdown_connections 2024-01-26T23:42:28.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.772+0000 7fae0a8f4700 1 -- 172.21.15.46:0/3271206626 wait complete. 2024-01-26T23:42:28.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.773+0000 7fae0a8f4700 1 Processor -- start 2024-01-26T23:42:28.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.773+0000 7fae0a8f4700 1 -- start start 2024-01-26T23:42:28.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.774+0000 7fae0a8f4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae040c8da0 0x7fae04156160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:28.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.774+0000 7fae0a8f4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae040caf30 0x7fae0415afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:28.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.774+0000 7fae0a8f4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae04140af0 0x7fae040c3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:28.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.774+0000 7fae0a8f4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae041566a0 con 0x7fae040c8da0 2024-01-26T23:42:28.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.774+0000 7fae0a8f4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0415b860 con 0x7fae04140af0 2024-01-26T23:42:28.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.774+0000 7fae0a8f4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae040c6cd0 con 0x7fae040caf30 2024-01-26T23:42:28.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.774+0000 7fae03fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae040c8da0 0x7fae04156160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:28.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.774+0000 7fae03fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae040c8da0 0x7fae04156160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51534/0 (socket says 172.21.15.46:51534) 2024-01-26T23:42:28.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.774+0000 7fae03fff700 1 -- 172.21.15.46:0/2204999816 learned_addr learned my addr 172.21.15.46:0/2204999816 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:28.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.774+0000 7fae037fe700 1 --2- 172.21.15.46:0/2204999816 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae040caf30 0x7fae0415afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:28.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.774+0000 7fae08e91700 1 --2- 172.21.15.46:0/2204999816 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae04140af0 0x7fae040c3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:28.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.774+0000 7fae03fff700 1 -- 172.21.15.46:0/2204999816 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae040caf30 msgr2=0x7fae0415afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:28.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.774+0000 7fae03fff700 1 --2- 172.21.15.46:0/2204999816 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae040caf30 0x7fae0415afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.774+0000 7fae03fff700 1 -- 172.21.15.46:0/2204999816 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae04140af0 msgr2=0x7fae040c3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:28.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.775+0000 7fae03fff700 1 --2- 172.21.15.46:0/2204999816 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae04140af0 0x7fae040c3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:28.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.775+0000 7fae03fff700 1 -- 172.21.15.46:0/2204999816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf80094f0 con 0x7fae040c8da0 2024-01-26T23:42:28.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.775+0000 7fae08e91700 1 --2- 172.21.15.46:0/2204999816 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae04140af0 0x7fae040c3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:42:28.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.775+0000 7fae03fff700 1 --2- 172.21.15.46:0/2204999816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae040c8da0 0x7fae04156160 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fadf400d890 tx=0x7fadf4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:28.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.775+0000 7fadebfff700 1 -- 172.21.15.46:0/2204999816 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf400b690 con 0x7fae040c8da0 2024-01-26T23:42:28.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.775+0000 7fadebfff700 1 -- 172.21.15.46:0/2204999816 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf400b7f0 con 0x7fae040c8da0 2024-01-26T23:42:28.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.775+0000 7fae0a8f4700 1 -- 172.21.15.46:0/2204999816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae040c6fb0 con 0x7fae040c8da0 2024-01-26T23:42:28.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.775+0000 7fadebfff700 1 -- 172.21.15.46:0/2204999816 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf400bb70 con 0x7fae040c8da0 2024-01-26T23:42:28.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.775+0000 7fae0a8f4700 1 -- 172.21.15.46:0/2204999816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae040c7570 con 0x7fae040c8da0 2024-01-26T23:42:28.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.776+0000 7fae0a8f4700 1 -- 172.21.15.46:0/2204999816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae040c7760 con 0x7fae040c8da0 2024-01-26T23:42:28.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.776+0000 7fadebfff700 1 -- 172.21.15.46:0/2204999816 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fadf401e460 con 0x7fae040c8da0 2024-01-26T23:42:28.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.777+0000 7fadebfff700 1 --2- 172.21.15.46:0/2204999816 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadf0067130 0x7fadf00695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:28.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.777+0000 7fadebfff700 1 -- 172.21.15.46:0/2204999816 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fadf4085340 con 0x7fae040c8da0 2024-01-26T23:42:28.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.777+0000 7fae037fe700 1 --2- 172.21.15.46:0/2204999816 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadf0067130 0x7fadf00695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:28.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.778+0000 7fae037fe700 1 --2- 172.21.15.46:0/2204999816 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadf0067130 0x7fadf00695e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fae0406f100 tx=0x7fadf8017000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:28.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:28.781+0000 7fadebfff700 1 -- 172.21.15.46:0/2204999816 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fadf4052030 con 0x7fae040c8da0 2024-01-26T23:42:29.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.010+0000 7fae0a8f4700 1 -- 172.21.15.46:0/2204999816 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fae0404c710 con 0x7fadf0067130 2024-01-26T23:42:29.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.019+0000 7fadebfff700 1 -- 172.21.15.46:0/2204999816 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fae0404c710 con 0x7fadf0067130 2024-01-26T23:42:29.019 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:29.019 INFO:teuthology.orchestra.run.smithi046.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:13", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:29.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.021+0000 7fae0a8f4700 1 -- 172.21.15.46:0/2204999816 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadf0067130 msgr2=0x7fadf00695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:29.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.021+0000 7fae0a8f4700 1 --2- 172.21.15.46:0/2204999816 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadf0067130 0x7fadf00695e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fae0406f100 tx=0x7fadf8017000 comp rx=0 tx=0).stop 2024-01-26T23:42:29.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.021+0000 7fae0a8f4700 1 -- 172.21.15.46:0/2204999816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae040c8da0 msgr2=0x7fae04156160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:29.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.021+0000 7fae0a8f4700 1 --2- 172.21.15.46:0/2204999816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae040c8da0 0x7fae04156160 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fadf400d890 tx=0x7fadf4008040 comp rx=0 tx=0).stop 2024-01-26T23:42:29.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.021+0000 7fae0a8f4700 1 -- 172.21.15.46:0/2204999816 shutdown_connections 2024-01-26T23:42:29.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.021+0000 7fae0a8f4700 1 --2- 172.21.15.46:0/2204999816 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadf0067130 0x7fadf00695e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:29.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.021+0000 7fae0a8f4700 1 --2- 172.21.15.46:0/2204999816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae040c8da0 0x7fae04156160 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:29.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.021+0000 7fae0a8f4700 1 --2- 172.21.15.46:0/2204999816 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae04140af0 0x7fae040c3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:29.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.021+0000 7fae0a8f4700 1 --2- 172.21.15.46:0/2204999816 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae040caf30 0x7fae0415afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:29.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.021+0000 7fae0a8f4700 1 -- 172.21.15.46:0/2204999816 >> 172.21.15.46:0/2204999816 conn(0x7fae0406bd90 msgr2=0x7fae04156f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:29.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.021+0000 7fae0a8f4700 1 -- 172.21.15.46:0/2204999816 shutdown_connections 2024-01-26T23:42:29.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.021+0000 7fae0a8f4700 1 -- 172.21.15.46:0/2204999816 wait complete. 2024-01-26T23:42:29.035 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:31:13", "first": "2024-01-26T23:32:00", "last": "2024-01-26T23:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-26T23:42:29.035 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1m', 'retention': {}, 'start': '2024-01-26T00:00:00', 'created': '2024-01-26T23:31:13', 'first': '2024-01-26T23:32:00', 'last': '2024-01-26T23:42:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-26T23:42:29.035 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-26T23:42:29.035 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-26T23:42:29.035 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir0 1m 2024-01-26T23:42:29.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.171+0000 7f3da3f60700 1 Processor -- start 2024-01-26T23:42:29.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.171+0000 7f3da3f60700 1 -- start start 2024-01-26T23:42:29.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.172+0000 7f3da3f60700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3d9c1611e0 0x7f3d9c06c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:29.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.172+0000 7f3da3f60700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d9c1615b0 0x7f3d9c067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:29.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.172+0000 7f3da3f60700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9c06c810 con 0x7f3d9c0caad0 2024-01-26T23:42:29.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.172+0000 7f3da3f60700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9c0ca950 con 0x7f3d9c1611e0 2024-01-26T23:42:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.172+0000 7f3da3f60700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c06caf0 con 0x7f3d9c1615b0 2024-01-26T23:42:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.172+0000 7f3da1cfc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3d9c1611e0 0x7f3d9c06c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.172+0000 7f3da24fd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3d9c0caad0 0x7f3d9c06eb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34870/0 (socket says 172.21.15.46:34870) 2024-01-26T23:42:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.173+0000 7f3da24fd700 1 -- 172.21.15.46:0/65201015 learned_addr learned my addr 172.21.15.46:0/65201015 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.173+0000 7f3da14fb700 1 --2- 172.21.15.46:0/65201015 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d9c1615b0 0x7f3d9c067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.173+0000 7f3da1cfc700 1 -- 172.21.15.46:0/65201015 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d9c1615b0 msgr2=0x7f3d9c067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.173+0000 7f3da1cfc700 1 --2- 172.21.15.46:0/65201015 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d9c1615b0 0x7f3d9c067930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.173+0000 7f3da1cfc700 1 -- 172.21.15.46:0/65201015 >> v1:172.21.15.46:6789/0 conn(0x7f3d9c0caad0 legacy=0x7f3d9c06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.173+0000 7f3da1cfc700 1 -- 172.21.15.46:0/65201015 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9c06cdd0 con 0x7f3d9c1611e0 2024-01-26T23:42:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.173+0000 7f3da0cfa700 1 -- 172.21.15.46:0/65201015 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618972722 0 0) 0x7f3d9c06c810 con 0x7f3d9c0caad0 2024-01-26T23:42:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.173+0000 7f3da1cfc700 1 --2- 172.21.15.46:0/65201015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c1611e0 0x7f3d9c06c110 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3d980098e0 tx=0x7f3d980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab5ec22506f9c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.173+0000 7f3da0cfa700 1 -- 172.21.15.46:0/65201015 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9800b640 con 0x7f3d9c1611e0 2024-01-26T23:42:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.173+0000 7f3da0cfa700 1 -- 172.21.15.46:0/65201015 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9800b7a0 con 0x7f3d9c1611e0 2024-01-26T23:42:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.173+0000 7f3da0cfa700 1 -- 172.21.15.46:0/65201015 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9800bb40 con 0x7f3d9c1611e0 2024-01-26T23:42:29.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.174+0000 7f3da3f60700 1 -- 172.21.15.46:0/65201015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c1611e0 msgr2=0x7f3d9c06c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:29.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.174+0000 7f3da3f60700 1 --2- 172.21.15.46:0/65201015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c1611e0 0x7f3d9c06c110 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3d980098e0 tx=0x7f3d980091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:29.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.174+0000 7f3da3f60700 1 -- 172.21.15.46:0/65201015 shutdown_connections 2024-01-26T23:42:29.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.174+0000 7f3da3f60700 1 --2- 172.21.15.46:0/65201015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c1611e0 0x7f3d9c06c110 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:29.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.174+0000 7f3da3f60700 1 --2- 172.21.15.46:0/65201015 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d9c1615b0 0x7f3d9c067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:29.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.174+0000 7f3da3f60700 1 -- 172.21.15.46:0/65201015 >> 172.21.15.46:0/65201015 conn(0x7f3d9c0cdf50 msgr2=0x7f3d9c0c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:29.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.174+0000 7f3da3f60700 1 -- 172.21.15.46:0/65201015 shutdown_connections 2024-01-26T23:42:29.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.174+0000 7f3da3f60700 1 -- 172.21.15.46:0/65201015 wait complete. 2024-01-26T23:42:29.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.175+0000 7f3da3f60700 1 Processor -- start 2024-01-26T23:42:29.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.175+0000 7f3da3f60700 1 -- start start 2024-01-26T23:42:29.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.176+0000 7f3da3f60700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c0caad0 0x7f3d9c0b2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:29.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.176+0000 7f3da3f60700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d9c1611e0 0x7f3d9c15f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:29.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.176+0000 7f3da3f60700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d9c1615b0 0x7f3d9c15fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:29.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.176+0000 7f3da3f60700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c0b0fd0 con 0x7f3d9c0caad0 2024-01-26T23:42:29.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.176+0000 7f3da3f60700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c0b03c0 con 0x7f3d9c1615b0 2024-01-26T23:42:29.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.176+0000 7f3da3f60700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c0b06a0 con 0x7f3d9c1611e0 2024-01-26T23:42:29.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.176+0000 7f3da1cfc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c0caad0 0x7f3d9c0b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:29.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.176+0000 7f3da1cfc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c0caad0 0x7f3d9c0b2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51554/0 (socket says 172.21.15.46:51554) 2024-01-26T23:42:29.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.176+0000 7f3da1cfc700 1 -- 172.21.15.46:0/3034150147 learned_addr learned my addr 172.21.15.46:0/3034150147 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:29.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.176+0000 7f3da14fb700 1 --2- 172.21.15.46:0/3034150147 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d9c1611e0 0x7f3d9c15f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:29.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.176+0000 7f3da24fd700 1 --2- 172.21.15.46:0/3034150147 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d9c1615b0 0x7f3d9c15fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:29.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.177+0000 7f3da1cfc700 1 -- 172.21.15.46:0/3034150147 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d9c1611e0 msgr2=0x7f3d9c15f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:29.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.177+0000 7f3da1cfc700 1 --2- 172.21.15.46:0/3034150147 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d9c1611e0 0x7f3d9c15f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:29.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.177+0000 7f3da1cfc700 1 -- 172.21.15.46:0/3034150147 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d9c1615b0 msgr2=0x7f3d9c15fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:29.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.177+0000 7f3da1cfc700 1 --2- 172.21.15.46:0/3034150147 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d9c1615b0 0x7f3d9c15fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:29.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.177+0000 7f3da1cfc700 1 -- 172.21.15.46:0/3034150147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d98009420 con 0x7f3d9c0caad0 2024-01-26T23:42:29.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.177+0000 7f3da24fd700 1 --2- 172.21.15.46:0/3034150147 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d9c1615b0 0x7f3d9c15fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:29.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.177+0000 7f3da1cfc700 1 --2- 172.21.15.46:0/3034150147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c0caad0 0x7f3d9c0b2240 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c06e800 tx=0x7f3d98017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:29.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.177+0000 7f3d91ffb700 1 -- 172.21.15.46:0/3034150147 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d98000d10 con 0x7f3d9c0caad0 2024-01-26T23:42:29.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.177+0000 7f3d91ffb700 1 -- 172.21.15.46:0/3034150147 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9800bb20 con 0x7f3d9c0caad0 2024-01-26T23:42:29.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.177+0000 7f3da3f60700 1 -- 172.21.15.46:0/3034150147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d9c0b0980 con 0x7f3d9c0caad0 2024-01-26T23:42:29.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.177+0000 7f3d91ffb700 1 -- 172.21.15.46:0/3034150147 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d98011670 con 0x7f3d9c0caad0 2024-01-26T23:42:29.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.177+0000 7f3da3f60700 1 -- 172.21.15.46:0/3034150147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9c156c30 con 0x7f3d9c0caad0 2024-01-26T23:42:29.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.178+0000 7f3da3f60700 1 -- 172.21.15.46:0/3034150147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9c0b1340 con 0x7f3d9c0caad0 2024-01-26T23:42:29.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.179+0000 7f3d91ffb700 1 -- 172.21.15.46:0/3034150147 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f3d98011a20 con 0x7f3d9c0caad0 2024-01-26T23:42:29.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.179+0000 7f3d91ffb700 1 --2- 172.21.15.46:0/3034150147 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3d80067130 0x7f3d800695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:29.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.179+0000 7f3d91ffb700 1 -- 172.21.15.46:0/3034150147 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f3d980860b0 con 0x7f3d9c0caad0 2024-01-26T23:42:29.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.179+0000 7f3da14fb700 1 --2- 172.21.15.46:0/3034150147 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3d80067130 0x7f3d800695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:29.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.180+0000 7f3da14fb700 1 --2- 172.21.15.46:0/3034150147 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3d80067130 0x7f3d800695e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0c7ca0 tx=0x7f3d8c0091a0 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:29.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.184+0000 7f3d91ffb700 1 -- 172.21.15.46:0/3034150147 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3d98052e40 con 0x7f3d9c0caad0 2024-01-26T23:42:29.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.416+0000 7f3da3f60700 1 -- 172.21.15.46:0/3034150147 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d9c0b1110 con 0x7f3d80067130 2024-01-26T23:42:29.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.492+0000 7f3d91ffb700 1 -- 172.21.15.46:0/3034150147 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f3d8c009ad0 con 0x7f3d80067130 2024-01-26T23:42:29.493 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path /snap_test_dir1/dir0 2024-01-26T23:42:29.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.495+0000 7f3da3f60700 1 -- 172.21.15.46:0/3034150147 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3d80067130 msgr2=0x7f3d800695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:29.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.495+0000 7f3da3f60700 1 --2- 172.21.15.46:0/3034150147 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3d80067130 0x7f3d800695e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0c7ca0 tx=0x7f3d8c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:29.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.495+0000 7f3da3f60700 1 -- 172.21.15.46:0/3034150147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c0caad0 msgr2=0x7f3d9c0b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:29.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.495+0000 7f3da3f60700 1 --2- 172.21.15.46:0/3034150147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c0caad0 0x7f3d9c0b2240 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c06e800 tx=0x7f3d98017000 comp rx=0 tx=0).stop 2024-01-26T23:42:29.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.495+0000 7f3da3f60700 1 -- 172.21.15.46:0/3034150147 shutdown_connections 2024-01-26T23:42:29.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.495+0000 7f3da3f60700 1 --2- 172.21.15.46:0/3034150147 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3d80067130 0x7f3d800695e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:29.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.495+0000 7f3da3f60700 1 --2- 172.21.15.46:0/3034150147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c0caad0 0x7f3d9c0b2240 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:29.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.495+0000 7f3da3f60700 1 --2- 172.21.15.46:0/3034150147 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d9c1615b0 0x7f3d9c15fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:29.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.495+0000 7f3da3f60700 1 --2- 172.21.15.46:0/3034150147 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d9c1611e0 0x7f3d9c15f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:29.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.495+0000 7f3da3f60700 1 -- 172.21.15.46:0/3034150147 >> 172.21.15.46:0/3034150147 conn(0x7f3d9c0cdf50 msgr2=0x7f3d9c0cb260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:29.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.495+0000 7f3da3f60700 1 -- 172.21.15.46:0/3034150147 shutdown_connections 2024-01-26T23:42:29.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:29.495+0000 7f3da3f60700 1 -- 172.21.15.46:0/3034150147 wait complete. 2024-01-26T23:42:29.510 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2024-01-26T23:42:29.510 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:29.510 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-01-26T23:42:29.534 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:29.534 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:29.535 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:29.535 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:29.535 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:29.535 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:29.535 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:29.535 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:29.535 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:29.535 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:29.535 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:29.535 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:29.535 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:29.536 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_32_00_UTC) 2024-01-26T23:42:29.608 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:29.608 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:29.608 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_33_00_UTC) 2024-01-26T23:42:29.680 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:29.680 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:29.680 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_34_00_UTC) 2024-01-26T23:42:29.751 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:29.751 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:29.751 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_35_00_UTC) 2024-01-26T23:42:29.822 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:29.822 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:29.822 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_36_00_UTC) 2024-01-26T23:42:29.894 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:29.894 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:29.894 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_37_00_UTC) 2024-01-26T23:42:29.965 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:29.965 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:29.965 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_38_00_UTC) 2024-01-26T23:42:30.038 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:30.038 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:30.038 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_39_00_UTC) 2024-01-26T23:42:30.109 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:30.110 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:30.110 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_40_00_UTC) 2024-01-26T23:42:30.181 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:30.181 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:30.181 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_41_00_UTC) 2024-01-26T23:42:30.252 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:30.252 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:30.253 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-26-23_42_00_UTC) 2024-01-26T23:42:30.323 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:30.324 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2024-01-26T23:42:30.391 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir1 1m 2024-01-26T23:42:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.576+0000 7fe3ce839700 1 Processor -- start 2024-01-26T23:42:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.576+0000 7fe3ce839700 1 -- start start 2024-01-26T23:42:30.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.577+0000 7fe3ce839700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3c80c9770 0x7fe3c8159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.577+0000 7fe3ce839700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe3c8159760 0x7fe3c80c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.577+0000 7fe3ce839700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c815e390 con 0x7fe3c80c93a0 2024-01-26T23:42:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.577+0000 7fe3ce839700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c8068460 con 0x7fe3c80c9770 2024-01-26T23:42:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.577+0000 7fe3ce839700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c815d810 con 0x7fe3c8159760 2024-01-26T23:42:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.577+0000 7fe3c77fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3c80c9770 0x7fe3c8159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.578+0000 7fe3c77fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3c80c9770 0x7fe3c8159220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59958/0 (socket says 172.21.15.46:59958) 2024-01-26T23:42:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.578+0000 7fe3c77fe700 1 -- 172.21.15.46:0/2728169820 learned_addr learned my addr 172.21.15.46:0/2728169820 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.578+0000 7fe3ccdd6700 1 --2- 172.21.15.46:0/2728169820 >> v2:172.21.15.106:3300/0 conn(0x7fe3c8159760 0x7fe3c80c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.578+0000 7fe3c77fe700 1 -- 172.21.15.46:0/2728169820 >> v2:172.21.15.106:3300/0 conn(0x7fe3c8159760 msgr2=0x7fe3c80c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.578+0000 7fe3c77fe700 1 --2- 172.21.15.46:0/2728169820 >> v2:172.21.15.106:3300/0 conn(0x7fe3c8159760 0x7fe3c80c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.578+0000 7fe3c77fe700 1 -- 172.21.15.46:0/2728169820 >> v1:172.21.15.106:6789/0 conn(0x7fe3c80c93a0 legacy=0x7fe3c8067d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.578+0000 7fe3c77fe700 1 -- 172.21.15.46:0/2728169820 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c815daf0 con 0x7fe3c80c9770 2024-01-26T23:42:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.578+0000 7fe3c77fe700 1 --2- 172.21.15.46:0/2728169820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80c9770 0x7fe3c8159220 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe3c80cd300 tx=0x7fe3b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b6b7211a8aa2c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:30.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.578+0000 7fe3c6ffd700 1 -- 172.21.15.46:0/2728169820 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8009e70 con 0x7fe3c80c9770 2024-01-26T23:42:30.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.578+0000 7fe3c6ffd700 1 -- 172.21.15.46:0/2728169820 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b800b860 con 0x7fe3c80c9770 2024-01-26T23:42:30.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.578+0000 7fe3c6ffd700 1 -- 172.21.15.46:0/2728169820 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b800bc20 con 0x7fe3c80c9770 2024-01-26T23:42:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.579+0000 7fe3ce839700 1 -- 172.21.15.46:0/2728169820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80c9770 msgr2=0x7fe3c8159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.579+0000 7fe3ce839700 1 --2- 172.21.15.46:0/2728169820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80c9770 0x7fe3c8159220 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe3c80cd300 tx=0x7fe3b8009250 comp rx=0 tx=0).stop 2024-01-26T23:42:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.579+0000 7fe3ce839700 1 -- 172.21.15.46:0/2728169820 shutdown_connections 2024-01-26T23:42:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.579+0000 7fe3ce839700 1 --2- 172.21.15.46:0/2728169820 >> v2:172.21.15.106:3300/0 conn(0x7fe3c8159760 0x7fe3c80c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.579+0000 7fe3ce839700 1 --2- 172.21.15.46:0/2728169820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80c9770 0x7fe3c8159220 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.579+0000 7fe3ce839700 1 -- 172.21.15.46:0/2728169820 >> 172.21.15.46:0/2728169820 conn(0x7fe3c80ca9a0 msgr2=0x7fe3c8068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.579+0000 7fe3ce839700 1 -- 172.21.15.46:0/2728169820 shutdown_connections 2024-01-26T23:42:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.579+0000 7fe3ce839700 1 -- 172.21.15.46:0/2728169820 wait complete. 2024-01-26T23:42:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.580+0000 7fe3ce839700 1 Processor -- start 2024-01-26T23:42:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.580+0000 7fe3ce839700 1 -- start start 2024-01-26T23:42:30.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.581+0000 7fe3ce839700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3c80c93a0 0x7fe3c81506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:30.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.581+0000 7fe3ce839700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80c9770 0x7fe3c80c4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.581+0000 7fe3ce839700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3c8159760 0x7fe3c815be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.581+0000 7fe3ce839700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c815a050 con 0x7fe3c80c9770 2024-01-26T23:42:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.581+0000 7fe3ce839700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c816e030 con 0x7fe3c8159760 2024-01-26T23:42:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.581+0000 7fe3ce839700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c80c5970 con 0x7fe3c80c93a0 2024-01-26T23:42:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.581+0000 7fe3c77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80c9770 0x7fe3c80c4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.581+0000 7fe3c77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80c9770 0x7fe3c80c4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59970/0 (socket says 172.21.15.46:59970) 2024-01-26T23:42:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.581+0000 7fe3c77fe700 1 -- 172.21.15.46:0/3404423655 learned_addr learned my addr 172.21.15.46:0/3404423655 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.582+0000 7fe3ccdd6700 1 --2- 172.21.15.46:0/3404423655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3c8159760 0x7fe3c815be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.582+0000 7fe3c7fff700 1 --2- 172.21.15.46:0/3404423655 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3c80c93a0 0x7fe3c81506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.582+0000 7fe3c77fe700 1 -- 172.21.15.46:0/3404423655 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3c80c93a0 msgr2=0x7fe3c81506d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.582+0000 7fe3c77fe700 1 --2- 172.21.15.46:0/3404423655 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3c80c93a0 0x7fe3c81506d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.582+0000 7fe3c77fe700 1 -- 172.21.15.46:0/3404423655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3c8159760 msgr2=0x7fe3c815be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.582+0000 7fe3c77fe700 1 --2- 172.21.15.46:0/3404423655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3c8159760 0x7fe3c815be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.582+0000 7fe3c77fe700 1 -- 172.21.15.46:0/3404423655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b80094f0 con 0x7fe3c80c9770 2024-01-26T23:42:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.582+0000 7fe3ccdd6700 1 --2- 172.21.15.46:0/3404423655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3c8159760 0x7fe3c815be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.582+0000 7fe3c77fe700 1 --2- 172.21.15.46:0/3404423655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80c9770 0x7fe3c80c4ed0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe3c806e820 tx=0x7fe3b8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.582+0000 7fe3bffff700 1 -- 172.21.15.46:0/3404423655 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b801f070 con 0x7fe3c80c9770 2024-01-26T23:42:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.582+0000 7fe3bffff700 1 -- 172.21.15.46:0/3404423655 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b8006b60 con 0x7fe3c80c9770 2024-01-26T23:42:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.582+0000 7fe3ce839700 1 -- 172.21.15.46:0/3404423655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c80c5c50 con 0x7fe3c80c9770 2024-01-26T23:42:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.582+0000 7fe3bffff700 1 -- 172.21.15.46:0/3404423655 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b800b9d0 con 0x7fe3c80c9770 2024-01-26T23:42:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.582+0000 7fe3ce839700 1 -- 172.21.15.46:0/3404423655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c80c71d0 con 0x7fe3c80c9770 2024-01-26T23:42:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.583+0000 7fe3ce839700 1 -- 172.21.15.46:0/3404423655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c80c5e60 con 0x7fe3c80c9770 2024-01-26T23:42:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.584+0000 7fe3bffff700 1 -- 172.21.15.46:0/3404423655 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fe3b8023070 con 0x7fe3c80c9770 2024-01-26T23:42:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.584+0000 7fe3bffff700 1 --2- 172.21.15.46:0/3404423655 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe3a8067130 0x7fe3a80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.584+0000 7fe3bffff700 1 -- 172.21.15.46:0/3404423655 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fe3b8085d60 con 0x7fe3c80c9770 2024-01-26T23:42:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.584+0000 7fe3c7fff700 1 --2- 172.21.15.46:0/3404423655 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe3a8067130 0x7fe3a80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.585+0000 7fe3c7fff700 1 --2- 172.21.15.46:0/3404423655 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe3a8067130 0x7fe3a80695e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00037c0 tx=0x7fe3b0009040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:30.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.589+0000 7fe3bffff700 1 -- 172.21.15.46:0/3404423655 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe3b8052af0 con 0x7fe3c80c9770 2024-01-26T23:42:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.858+0000 7fe3ce839700 1 -- 172.21.15.46:0/3404423655 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3c8159b60 con 0x7fe3a8067130 2024-01-26T23:42:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.959+0000 7fe3bffff700 1 -- 172.21.15.46:0/3404423655 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fe3b0013420 con 0x7fe3a8067130 2024-01-26T23:42:30.960 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path /snap_test_dir1/dir1 2024-01-26T23:42:30.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.962+0000 7fe3ce839700 1 -- 172.21.15.46:0/3404423655 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe3a8067130 msgr2=0x7fe3a80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:30.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.962+0000 7fe3ce839700 1 --2- 172.21.15.46:0/3404423655 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe3a8067130 0x7fe3a80695e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00037c0 tx=0x7fe3b0009040 comp rx=0 tx=0).stop 2024-01-26T23:42:30.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.962+0000 7fe3ce839700 1 -- 172.21.15.46:0/3404423655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80c9770 msgr2=0x7fe3c80c4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:30.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.962+0000 7fe3ce839700 1 --2- 172.21.15.46:0/3404423655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80c9770 0x7fe3c80c4ed0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe3c806e820 tx=0x7fe3b8017000 comp rx=0 tx=0).stop 2024-01-26T23:42:30.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.962+0000 7fe3ce839700 1 -- 172.21.15.46:0/3404423655 shutdown_connections 2024-01-26T23:42:30.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.962+0000 7fe3ce839700 1 --2- 172.21.15.46:0/3404423655 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe3a8067130 0x7fe3a80695e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:30.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.962+0000 7fe3ce839700 1 --2- 172.21.15.46:0/3404423655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3c8159760 0x7fe3c815be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:30.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.962+0000 7fe3ce839700 1 --2- 172.21.15.46:0/3404423655 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe3c80c93a0 0x7fe3c81506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:30.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.962+0000 7fe3ce839700 1 --2- 172.21.15.46:0/3404423655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80c9770 0x7fe3c80c4ed0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:30.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.962+0000 7fe3ce839700 1 -- 172.21.15.46:0/3404423655 >> 172.21.15.46:0/3404423655 conn(0x7fe3c80ca9a0 msgr2=0x7fe3c80c2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:30.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.962+0000 7fe3ce839700 1 -- 172.21.15.46:0/3404423655 shutdown_connections 2024-01-26T23:42:30.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:30.962+0000 7fe3ce839700 1 -- 172.21.15.46:0/3404423655 wait complete. 2024-01-26T23:42:30.977 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2024-01-26T23:42:30.977 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:30.977 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-01-26T23:42:31.006 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:31.006 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:31.006 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:31.006 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:31.006 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:31.006 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:31.006 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:31.006 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:31.006 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:31.006 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:31.006 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:31.007 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:31.007 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:31.008 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_32_00_UTC) 2024-01-26T23:42:31.080 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:31.081 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:31.081 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_33_00_UTC) 2024-01-26T23:42:31.152 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:31.152 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:31.152 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_34_00_UTC) 2024-01-26T23:42:31.224 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:31.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:31.225 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_35_00_UTC) 2024-01-26T23:42:31.296 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:31.297 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:31.297 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_36_00_UTC) 2024-01-26T23:42:31.368 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:31.369 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:31.369 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_37_00_UTC) 2024-01-26T23:42:31.440 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:31.441 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:31.441 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_38_00_UTC) 2024-01-26T23:42:31.512 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:31.512 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:31.513 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_39_00_UTC) 2024-01-26T23:42:31.583 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:31.583 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:31.583 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_40_00_UTC) 2024-01-26T23:42:31.662 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:31.662 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:31.663 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_41_00_UTC) 2024-01-26T23:42:31.736 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:31.736 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:31.736 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-26-23_42_00_UTC) 2024-01-26T23:42:31.809 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:31.809 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2024-01-26T23:42:31.876 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir2 1m 2024-01-26T23:42:32.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.061+0000 7f364759e700 1 Processor -- start 2024-01-26T23:42:32.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.061+0000 7f364759e700 1 -- start start 2024-01-26T23:42:32.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.062+0000 7f364759e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f364806b050 0x7f3648157500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:32.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.062+0000 7f364759e700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3648157070 con 0x7f36480c8e30 2024-01-26T23:42:32.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.062+0000 7f364759e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364806cb80 con 0x7f36480c9200 2024-01-26T23:42:32.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.062+0000 7f364759e700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f364806ce60 con 0x7f364806b050 2024-01-26T23:42:32.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.063+0000 7f3646d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f364806b050 0x7f3648157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:32.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.063+0000 7f364659c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f36480c8e30 0x7f364806c0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49986/0 (socket says 172.21.15.46:49986) 2024-01-26T23:42:32.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.063+0000 7f364659c700 1 -- 172.21.15.46:0/1077318158 learned_addr learned my addr 172.21.15.46:0/1077318158 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:32.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.063+0000 7f3646d9d700 1 -- 172.21.15.46:0/1077318158 >> v1:172.21.15.106:6789/0 conn(0x7f36480c9200 legacy=0x7f364806a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:32.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.063+0000 7f3646d9d700 1 -- 172.21.15.46:0/1077318158 >> v1:172.21.15.46:6789/0 conn(0x7f36480c8e30 legacy=0x7f364806c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:32.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.063+0000 7f364559a700 1 -- 172.21.15.46:0/1077318158 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771396139 0 0) 0x7f3648157070 con 0x7f36480c8e30 2024-01-26T23:42:32.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.063+0000 7f3646d9d700 1 -- 172.21.15.46:0/1077318158 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3648156a20 con 0x7f364806b050 2024-01-26T23:42:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.063+0000 7f3646d9d700 1 --2- 172.21.15.46:0/1077318158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f364806b050 0x7f3648157500 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f364000d0d0 tx=0x7f364000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c694ee9f9d3e29e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.063+0000 7f364559a700 1 -- 172.21.15.46:0/1077318158 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3640010070 con 0x7f364806b050 2024-01-26T23:42:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.063+0000 7f364559a700 1 -- 172.21.15.46:0/1077318158 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f364000ddb0 con 0x7f364806b050 2024-01-26T23:42:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.063+0000 7f364559a700 1 -- 172.21.15.46:0/1077318158 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364000b320 con 0x7f364806b050 2024-01-26T23:42:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.064+0000 7f364759e700 1 -- 172.21.15.46:0/1077318158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f364806b050 msgr2=0x7f3648157500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.064+0000 7f364759e700 1 --2- 172.21.15.46:0/1077318158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f364806b050 0x7f3648157500 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f364000d0d0 tx=0x7f364000b990 comp rx=0 tx=0).stop 2024-01-26T23:42:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.064+0000 7f364759e700 1 -- 172.21.15.46:0/1077318158 shutdown_connections 2024-01-26T23:42:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.064+0000 7f364759e700 1 --2- 172.21.15.46:0/1077318158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f364806b050 0x7f3648157500 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.064+0000 7f364759e700 1 -- 172.21.15.46:0/1077318158 >> 172.21.15.46:0/1077318158 conn(0x7f36480cd260 msgr2=0x7f36480cd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.064+0000 7f364759e700 1 -- 172.21.15.46:0/1077318158 shutdown_connections 2024-01-26T23:42:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.064+0000 7f364759e700 1 -- 172.21.15.46:0/1077318158 wait complete. 2024-01-26T23:42:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.065+0000 7f364759e700 1 Processor -- start 2024-01-26T23:42:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.065+0000 7f364759e700 1 -- start start 2024-01-26T23:42:32.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.066+0000 7f364759e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f364806b050 0x7f364815a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:32.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.066+0000 7f364759e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36480c8e30 0x7f3648146670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:32.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.066+0000 7f364759e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36480c9200 0x7f3648157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:32.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.066+0000 7f364759e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36481544b0 con 0x7f364806b050 2024-01-26T23:42:32.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.066+0000 7f364759e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36480c1dc0 con 0x7f36480c9200 2024-01-26T23:42:32.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.066+0000 7f364759e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36480c2060 con 0x7f36480c8e30 2024-01-26T23:42:32.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.066+0000 7f364659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f364806b050 0x7f364815a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:32.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.066+0000 7f364659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f364806b050 0x7f364815a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59986/0 (socket says 172.21.15.46:59986) 2024-01-26T23:42:32.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.066+0000 7f364659c700 1 -- 172.21.15.46:0/2847770544 learned_addr learned my addr 172.21.15.46:0/2847770544 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:32.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.066+0000 7f3646d9d700 1 --2- 172.21.15.46:0/2847770544 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36480c9200 0x7f3648157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:32.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.066+0000 7f3645d9b700 1 --2- 172.21.15.46:0/2847770544 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36480c8e30 0x7f3648146670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:32.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.067+0000 7f364659c700 1 -- 172.21.15.46:0/2847770544 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36480c8e30 msgr2=0x7f3648146670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:32.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.067+0000 7f364659c700 1 --2- 172.21.15.46:0/2847770544 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36480c8e30 0x7f3648146670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:32.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.067+0000 7f364659c700 1 -- 172.21.15.46:0/2847770544 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36480c9200 msgr2=0x7f3648157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:32.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.067+0000 7f364659c700 1 --2- 172.21.15.46:0/2847770544 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36480c9200 0x7f3648157ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:32.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.067+0000 7f364659c700 1 -- 172.21.15.46:0/2847770544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364000bc30 con 0x7f364806b050 2024-01-26T23:42:32.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.067+0000 7f364659c700 1 --2- 172.21.15.46:0/2847770544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f364806b050 0x7f364815a980 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f363800c7e0 tx=0x7f3638007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:32.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.067+0000 7f36367fc700 1 -- 172.21.15.46:0/2847770544 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3638005b30 con 0x7f364806b050 2024-01-26T23:42:32.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.067+0000 7f36367fc700 1 -- 172.21.15.46:0/2847770544 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3638005c90 con 0x7f364806b050 2024-01-26T23:42:32.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.067+0000 7f364759e700 1 -- 172.21.15.46:0/2847770544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36480c2340 con 0x7f364806b050 2024-01-26T23:42:32.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.067+0000 7f36367fc700 1 -- 172.21.15.46:0/2847770544 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3638007300 con 0x7f364806b050 2024-01-26T23:42:32.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.067+0000 7f364759e700 1 -- 172.21.15.46:0/2847770544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3648152b80 con 0x7f364806b050 2024-01-26T23:42:32.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.068+0000 7f364759e700 1 -- 172.21.15.46:0/2847770544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364806dee0 con 0x7f364806b050 2024-01-26T23:42:32.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.069+0000 7f36367fc700 1 -- 172.21.15.46:0/2847770544 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f363801b3b0 con 0x7f364806b050 2024-01-26T23:42:32.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.069+0000 7f36367fc700 1 --2- 172.21.15.46:0/2847770544 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3628067130 0x7f36280695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:32.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.069+0000 7f36367fc700 1 -- 172.21.15.46:0/2847770544 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6528+0+0 (secure 0 0 0) 0x7f3638084020 con 0x7f364806b050 2024-01-26T23:42:32.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.069+0000 7f3645d9b700 1 --2- 172.21.15.46:0/2847770544 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3628067130 0x7f36280695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:32.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.070+0000 7f3645d9b700 1 --2- 172.21.15.46:0/2847770544 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3628067130 0x7f36280695e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3648168b20 tx=0x7f363c009060 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:32.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.074+0000 7f36367fc700 1 -- 172.21.15.46:0/2847770544 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3638050ce0 con 0x7f364806b050 2024-01-26T23:42:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.305+0000 7f364759e700 1 -- 172.21.15.46:0/2847770544 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f36481545f0 con 0x7f3628067130 2024-01-26T23:42:32.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.377+0000 7f36367fc700 1 -- 172.21.15.46:0/2847770544 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f363c009780 con 0x7f3628067130 2024-01-26T23:42:32.378 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path /snap_test_dir1/dir2 2024-01-26T23:42:32.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.379+0000 7f364759e700 1 -- 172.21.15.46:0/2847770544 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3628067130 msgr2=0x7f36280695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:32.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.379+0000 7f364759e700 1 --2- 172.21.15.46:0/2847770544 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3628067130 0x7f36280695e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3648168b20 tx=0x7f363c009060 comp rx=0 tx=0).stop 2024-01-26T23:42:32.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.379+0000 7f364759e700 1 -- 172.21.15.46:0/2847770544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f364806b050 msgr2=0x7f364815a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:32.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.379+0000 7f364759e700 1 --2- 172.21.15.46:0/2847770544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f364806b050 0x7f364815a980 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f363800c7e0 tx=0x7f3638007df0 comp rx=0 tx=0).stop 2024-01-26T23:42:32.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.379+0000 7f364759e700 1 -- 172.21.15.46:0/2847770544 shutdown_connections 2024-01-26T23:42:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.379+0000 7f364759e700 1 --2- 172.21.15.46:0/2847770544 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3628067130 0x7f36280695e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.379+0000 7f364759e700 1 --2- 172.21.15.46:0/2847770544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f364806b050 0x7f364815a980 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.380+0000 7f364759e700 1 --2- 172.21.15.46:0/2847770544 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36480c9200 0x7f3648157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.380+0000 7f364759e700 1 --2- 172.21.15.46:0/2847770544 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36480c8e30 0x7f3648146670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.380+0000 7f364759e700 1 -- 172.21.15.46:0/2847770544 >> 172.21.15.46:0/2847770544 conn(0x7f36480cd260 msgr2=0x7f3648067930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.380+0000 7f364759e700 1 -- 172.21.15.46:0/2847770544 shutdown_connections 2024-01-26T23:42:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:32.380+0000 7f364759e700 1 -- 172.21.15.46:0/2847770544 wait complete. 2024-01-26T23:42:32.394 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2024-01-26T23:42:32.394 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:32.394 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-01-26T23:42:32.426 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:32.426 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:32.426 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:32.426 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:32.426 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:32.426 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:32.426 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:32.426 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:32.426 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:32.426 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:32.426 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:32.427 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:32.427 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:32.427 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_32_00_UTC) 2024-01-26T23:42:32.500 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:32.500 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:32.500 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_33_00_UTC) 2024-01-26T23:42:32.573 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:32.573 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:32.573 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_34_00_UTC) 2024-01-26T23:42:32.668 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:32.668 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:32.669 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_35_00_UTC) 2024-01-26T23:42:32.697 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:32.697 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:32.697 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_36_00_UTC) 2024-01-26T23:42:32.768 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:32.768 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:32.768 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_37_00_UTC) 2024-01-26T23:42:32.841 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:32.841 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:32.841 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_38_00_UTC) 2024-01-26T23:42:32.922 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:32.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:32.922 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_39_00_UTC) 2024-01-26T23:42:32.993 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:32.993 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:32.993 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_40_00_UTC) 2024-01-26T23:42:33.062 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:33.063 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:33.063 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_41_00_UTC) 2024-01-26T23:42:33.134 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:33.134 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:33.134 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-26-23_42_00_UTC) 2024-01-26T23:42:33.205 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:33.206 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2024-01-26T23:42:33.272 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir3 1m 2024-01-26T23:42:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.456+0000 7fb2547aa700 1 Processor -- start 2024-01-26T23:42:33.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.456+0000 7fb2547aa700 1 -- start start 2024-01-26T23:42:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.457+0000 7fb2547aa700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c157860 0x7fb24c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.457+0000 7fb2547aa700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb24c06ddb0 0x7fb24c0cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.457+0000 7fb2547aa700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c0cea80 con 0x7fb24c157490 2024-01-26T23:42:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.457+0000 7fb2547aa700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb24c0cdc60 con 0x7fb24c06ddb0 2024-01-26T23:42:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.457+0000 7fb2547aa700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c0b2ae0 con 0x7fb24c157860 2024-01-26T23:42:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.457+0000 7fb252546700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb24c157490 0x7fb24c06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49998/0 (socket says 172.21.15.46:49998) 2024-01-26T23:42:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.457+0000 7fb252546700 1 -- 172.21.15.46:0/3238689498 learned_addr learned my addr 172.21.15.46:0/3238689498 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251d45700 1 --2- 172.21.15.46:0/3238689498 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c157860 0x7fb24c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb252d47700 1 --2- 172.21.15.46:0/3238689498 >> v2:172.21.15.106:3300/0 conn(0x7fb24c06ddb0 0x7fb24c0cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251544700 1 -- 172.21.15.46:0/3238689498 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4132551421 0 0) 0x7fb24c0cea80 con 0x7fb24c157490 2024-01-26T23:42:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251544700 1 -- 172.21.15.46:0/3238689498 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb238003570 con 0x7fb24c157490 2024-01-26T23:42:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251544700 1 -- 172.21.15.46:0/3238689498 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1652629197 0 0) 0x7fb238003570 con 0x7fb24c157490 2024-01-26T23:42:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251544700 1 -- 172.21.15.46:0/3238689498 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c157860 msgr2=0x7fb24c0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251544700 1 --2- 172.21.15.46:0/3238689498 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c157860 0x7fb24c0cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251544700 1 -- 172.21.15.46:0/3238689498 >> v2:172.21.15.106:3300/0 conn(0x7fb24c06ddb0 msgr2=0x7fb24c0cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251544700 1 --2- 172.21.15.46:0/3238689498 >> v2:172.21.15.106:3300/0 conn(0x7fb24c06ddb0 0x7fb24c0cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251544700 1 -- 172.21.15.46:0/3238689498 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c0cde70 con 0x7fb24c157490 2024-01-26T23:42:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251544700 1 -- 172.21.15.46:0/3238689498 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fb23c001470 con 0x7fb24c157490 2024-01-26T23:42:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251544700 1 -- 172.21.15.46:0/3238689498 >> v1:172.21.15.46:6789/0 conn(0x7fb24c157490 legacy=0x7fb24c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251544700 1 --2- 172.21.15.46:0/3238689498 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb238003a60 0x7fb238003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251544700 1 --2- 172.21.15.46:0/3238689498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb238004810 0x7fb238004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251544700 1 --2- 172.21.15.46:0/3238689498 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c157860 0x7fb2380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251544700 1 -- 172.21.15.46:0/3238689498 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c0cdc60 con 0x7fb238004810 2024-01-26T23:42:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251544700 1 -- 172.21.15.46:0/3238689498 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c0b2ae0 con 0x7fb238003a60 2024-01-26T23:42:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251544700 1 -- 172.21.15.46:0/3238689498 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23800ba60 con 0x7fb24c157860 2024-01-26T23:42:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb251d45700 1 --2- 172.21.15.46:0/3238689498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb238004810 0x7fb238004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb252546700 1 --2- 172.21.15.46:0/3238689498 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb238003a60 0x7fb238003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.458+0000 7fb252d47700 1 --2- 172.21.15.46:0/3238689498 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c157860 0x7fb2380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.459+0000 7fb251d45700 1 -- 172.21.15.46:0/3238689498 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c157860 msgr2=0x7fb2380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.459+0000 7fb251d45700 1 --2- 172.21.15.46:0/3238689498 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c157860 0x7fb2380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.459+0000 7fb251d45700 1 -- 172.21.15.46:0/3238689498 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb238003a60 msgr2=0x7fb238003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.459+0000 7fb251d45700 1 --2- 172.21.15.46:0/3238689498 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb238003a60 0x7fb238003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.459+0000 7fb251d45700 1 -- 172.21.15.46:0/3238689498 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb238003570 con 0x7fb238004810 2024-01-26T23:42:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.459+0000 7fb252d47700 1 --2- 172.21.15.46:0/3238689498 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c157860 0x7fb2380051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.459+0000 7fb251d45700 1 --2- 172.21.15.46:0/3238689498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb238004810 0x7fb238004c80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb248009c00 tx=0x7fb2480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db977829fc5056c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.459+0000 7fb251544700 1 -- 172.21.15.46:0/3238689498 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24800e710 con 0x7fb238004810 2024-01-26T23:42:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.459+0000 7fb251544700 1 -- 172.21.15.46:0/3238689498 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb24800e870 con 0x7fb238004810 2024-01-26T23:42:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.459+0000 7fb251544700 1 -- 172.21.15.46:0/3238689498 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24800ec30 con 0x7fb238004810 2024-01-26T23:42:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.459+0000 7fb2547aa700 1 -- 172.21.15.46:0/3238689498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb238004810 msgr2=0x7fb238004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.459+0000 7fb2547aa700 1 --2- 172.21.15.46:0/3238689498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb238004810 0x7fb238004c80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb248009c00 tx=0x7fb2480091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.459+0000 7fb252d47700 1 -- 172.21.15.46:0/3238689498 reap_dead start 2024-01-26T23:42:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.460+0000 7fb2547aa700 1 -- 172.21.15.46:0/3238689498 shutdown_connections 2024-01-26T23:42:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.460+0000 7fb2547aa700 1 -- 172.21.15.46:0/3238689498 >> 172.21.15.46:0/3238689498 conn(0x7fb24c067930 msgr2=0x7fb24c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.460+0000 7fb2547aa700 1 -- 172.21.15.46:0/3238689498 shutdown_connections 2024-01-26T23:42:33.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.460+0000 7fb2547aa700 1 -- 172.21.15.46:0/3238689498 wait complete. 2024-01-26T23:42:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.461+0000 7fb2547aa700 1 Processor -- start 2024-01-26T23:42:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.461+0000 7fb2547aa700 1 -- start start 2024-01-26T23:42:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb2547aa700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb238004810 0x7fb24c148ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb2547aa700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c14ca80 0x7fb24c14b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb2547aa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb24c14cef0 0x7fb24c14b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb2547aa700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c14a160 con 0x7fb24c14cef0 2024-01-26T23:42:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb2547aa700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c14ace0 con 0x7fb238004810 2024-01-26T23:42:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb2547aa700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c14bbf0 con 0x7fb24c14ca80 2024-01-26T23:42:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb252d47700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb24c14cef0 0x7fb24c14b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb252d47700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb24c14cef0 0x7fb24c14b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59998/0 (socket says 172.21.15.46:59998) 2024-01-26T23:42:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb252d47700 1 -- 172.21.15.46:0/3347275364 learned_addr learned my addr 172.21.15.46:0/3347275364 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb251d45700 1 --2- 172.21.15.46:0/3347275364 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c14ca80 0x7fb24c14b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb252546700 1 --2- 172.21.15.46:0/3347275364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb238004810 0x7fb24c148ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb252d47700 1 -- 172.21.15.46:0/3347275364 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c14ca80 msgr2=0x7fb24c14b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb252d47700 1 --2- 172.21.15.46:0/3347275364 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c14ca80 0x7fb24c14b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb252d47700 1 -- 172.21.15.46:0/3347275364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb238004810 msgr2=0x7fb24c148ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb252d47700 1 --2- 172.21.15.46:0/3347275364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb238004810 0x7fb24c148ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb252d47700 1 -- 172.21.15.46:0/3347275364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c14a3b0 con 0x7fb24c14cef0 2024-01-26T23:42:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb252546700 1 --2- 172.21.15.46:0/3347275364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb238004810 0x7fb24c148ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.462+0000 7fb252d47700 1 --2- 172.21.15.46:0/3347275364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb24c14cef0 0x7fb24c14b6b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb24c0adc10 tx=0x7fb24400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.463+0000 7fb2427fc700 1 -- 172.21.15.46:0/3347275364 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24400e050 con 0x7fb24c14cef0 2024-01-26T23:42:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.463+0000 7fb2427fc700 1 -- 172.21.15.46:0/3347275364 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb24400f040 con 0x7fb24c14cef0 2024-01-26T23:42:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.463+0000 7fb2547aa700 1 -- 172.21.15.46:0/3347275364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb24c14a690 con 0x7fb24c14cef0 2024-01-26T23:42:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.463+0000 7fb2427fc700 1 -- 172.21.15.46:0/3347275364 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244010e20 con 0x7fb24c14cef0 2024-01-26T23:42:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.463+0000 7fb2547aa700 1 -- 172.21.15.46:0/3347275364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24c149530 con 0x7fb24c14cef0 2024-01-26T23:42:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.464+0000 7fb2547aa700 1 -- 172.21.15.46:0/3347275364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24c155800 con 0x7fb24c14cef0 2024-01-26T23:42:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.464+0000 7fb2427fc700 1 -- 172.21.15.46:0/3347275364 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fb244003020 con 0x7fb24c14cef0 2024-01-26T23:42:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.465+0000 7fb2427fc700 1 --2- 172.21.15.46:0/3347275364 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb230067130 0x7fb2300695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.465+0000 7fb2427fc700 1 -- 172.21.15.46:0/3347275364 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6768+0+0 (secure 0 0 0) 0x7fb244085140 con 0x7fb24c14cef0 2024-01-26T23:42:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.465+0000 7fb252546700 1 --2- 172.21.15.46:0/3347275364 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb230067130 0x7fb2300695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.465+0000 7fb252546700 1 --2- 172.21.15.46:0/3347275364 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb230067130 0x7fb2300695e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb23c000e10 tx=0x7fb23c009040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:33.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.469+0000 7fb2427fc700 1 -- 172.21.15.46:0/3347275364 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb244051cb0 con 0x7fb24c14cef0 2024-01-26T23:42:33.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.722+0000 7fb2547aa700 1 -- 172.21.15.46:0/3347275364 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb24c04c710 con 0x7fb230067130 2024-01-26T23:42:33.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.809+0000 7fb2427fc700 1 -- 172.21.15.46:0/3347275364 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fb24c04c710 con 0x7fb230067130 2024-01-26T23:42:33.809 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path /snap_test_dir1/dir3 2024-01-26T23:42:33.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.811+0000 7fb2547aa700 1 -- 172.21.15.46:0/3347275364 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb230067130 msgr2=0x7fb2300695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:33.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.811+0000 7fb2547aa700 1 --2- 172.21.15.46:0/3347275364 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb230067130 0x7fb2300695e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb23c000e10 tx=0x7fb23c009040 comp rx=0 tx=0).stop 2024-01-26T23:42:33.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.811+0000 7fb2547aa700 1 -- 172.21.15.46:0/3347275364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb24c14cef0 msgr2=0x7fb24c14b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:33.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.811+0000 7fb2547aa700 1 --2- 172.21.15.46:0/3347275364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb24c14cef0 0x7fb24c14b6b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb24c0adc10 tx=0x7fb24400b040 comp rx=0 tx=0).stop 2024-01-26T23:42:33.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.811+0000 7fb2547aa700 1 -- 172.21.15.46:0/3347275364 shutdown_connections 2024-01-26T23:42:33.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.811+0000 7fb2547aa700 1 --2- 172.21.15.46:0/3347275364 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb230067130 0x7fb2300695e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:33.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.811+0000 7fb2547aa700 1 --2- 172.21.15.46:0/3347275364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb24c14cef0 0x7fb24c14b6b0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:33.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.811+0000 7fb2547aa700 1 --2- 172.21.15.46:0/3347275364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb238004810 0x7fb24c148ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:33.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.811+0000 7fb2547aa700 1 --2- 172.21.15.46:0/3347275364 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c14ca80 0x7fb24c14b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:33.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.811+0000 7fb2547aa700 1 -- 172.21.15.46:0/3347275364 >> 172.21.15.46:0/3347275364 conn(0x7fb24c067930 msgr2=0x7fb24c155e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:33.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.811+0000 7fb2547aa700 1 -- 172.21.15.46:0/3347275364 shutdown_connections 2024-01-26T23:42:33.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:33.811+0000 7fb2547aa700 1 -- 172.21.15.46:0/3347275364 wait complete. 2024-01-26T23:42:33.826 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2024-01-26T23:42:33.826 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:33.826 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-01-26T23:42:33.855 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:33.855 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:33.855 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:33.855 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:33.855 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:33.855 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:33.856 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:33.856 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:33.856 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:33.856 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:33.856 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:33.856 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:33.856 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:33.856 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_32_00_UTC) 2024-01-26T23:42:33.928 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:33.928 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:33.928 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_33_00_UTC) 2024-01-26T23:42:33.999 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:34.000 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:34.000 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_34_00_UTC) 2024-01-26T23:42:34.071 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:34.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:34.072 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_35_00_UTC) 2024-01-26T23:42:34.143 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:34.144 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:34.144 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_36_00_UTC) 2024-01-26T23:42:34.220 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:34.221 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:34.221 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_37_00_UTC) 2024-01-26T23:42:34.292 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:34.293 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:34.293 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_38_00_UTC) 2024-01-26T23:42:34.364 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:34.364 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:34.365 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_39_00_UTC) 2024-01-26T23:42:34.436 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:34.437 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:34.437 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_40_00_UTC) 2024-01-26T23:42:34.509 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:34.509 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:34.509 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_41_00_UTC) 2024-01-26T23:42:34.580 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:34.581 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:34.581 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-26-23_42_00_UTC) 2024-01-26T23:42:34.667 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:34.667 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2024-01-26T23:42:34.697 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir4 1m 2024-01-26T23:42:34.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.883+0000 7f2412274700 1 Processor -- start 2024-01-26T23:42:34.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.883+0000 7f2412274700 1 -- start start 2024-01-26T23:42:34.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.884+0000 7f2412274700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f240c1611e0 0x7f240c06c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:34.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.884+0000 7f2412274700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f240c1615b0 0x7f240c067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:34.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.884+0000 7f2412274700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240c06c810 con 0x7f240c0caad0 2024-01-26T23:42:34.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.884+0000 7f2412274700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f240c0ca950 con 0x7f240c1611e0 2024-01-26T23:42:34.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.884+0000 7f2412274700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240c06caf0 con 0x7f240c1615b0 2024-01-26T23:42:34.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.885+0000 7f240bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f240c1611e0 0x7f240c06c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:34.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.885+0000 7f240bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f240c1611e0 0x7f240c06c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60002/0 (socket says 172.21.15.46:60002) 2024-01-26T23:42:34.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.885+0000 7f240bfff700 1 -- 172.21.15.46:0/2680689076 learned_addr learned my addr 172.21.15.46:0/2680689076 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:34.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.885+0000 7f240b7fe700 1 --2- 172.21.15.46:0/2680689076 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f240c1615b0 0x7f240c067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:34.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.885+0000 7f240bfff700 1 -- 172.21.15.46:0/2680689076 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f240c1615b0 msgr2=0x7f240c067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:34.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.885+0000 7f240bfff700 1 --2- 172.21.15.46:0/2680689076 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f240c1615b0 0x7f240c067930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:34.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.885+0000 7f240bfff700 1 -- 172.21.15.46:0/2680689076 >> v1:172.21.15.106:6789/0 conn(0x7f240c0caad0 legacy=0x7f240c06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:34.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.885+0000 7f240bfff700 1 -- 172.21.15.46:0/2680689076 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c06cdd0 con 0x7f240c1611e0 2024-01-26T23:42:34.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.886+0000 7f240bfff700 1 --2- 172.21.15.46:0/2680689076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f240c1611e0 0x7f240c06c110 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0099b0 tx=0x7f23fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ae4d6eb80225877 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:34.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.886+0000 7f240affd700 1 -- 172.21.15.46:0/2680689076 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc009ea0 con 0x7f240c1611e0 2024-01-26T23:42:34.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.886+0000 7f240affd700 1 -- 172.21.15.46:0/2680689076 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23fc00b860 con 0x7f240c1611e0 2024-01-26T23:42:34.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.886+0000 7f240affd700 1 -- 172.21.15.46:0/2680689076 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc00bc00 con 0x7f240c1611e0 2024-01-26T23:42:34.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.887+0000 7f2412274700 1 -- 172.21.15.46:0/2680689076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f240c1611e0 msgr2=0x7f240c06c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:34.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.887+0000 7f2412274700 1 --2- 172.21.15.46:0/2680689076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f240c1611e0 0x7f240c06c110 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0099b0 tx=0x7f23fc009250 comp rx=0 tx=0).stop 2024-01-26T23:42:34.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.888+0000 7f2412274700 1 -- 172.21.15.46:0/2680689076 shutdown_connections 2024-01-26T23:42:34.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.888+0000 7f2412274700 1 --2- 172.21.15.46:0/2680689076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f240c1611e0 0x7f240c06c110 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:34.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.888+0000 7f2412274700 1 --2- 172.21.15.46:0/2680689076 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f240c1615b0 0x7f240c067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:34.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.888+0000 7f2412274700 1 -- 172.21.15.46:0/2680689076 >> 172.21.15.46:0/2680689076 conn(0x7f240c0cdf50 msgr2=0x7f240c0c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:34.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.888+0000 7f2412274700 1 -- 172.21.15.46:0/2680689076 shutdown_connections 2024-01-26T23:42:34.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.889+0000 7f2412274700 1 -- 172.21.15.46:0/2680689076 wait complete. 2024-01-26T23:42:34.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.891+0000 7f2412274700 1 Processor -- start 2024-01-26T23:42:34.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.892+0000 7f2412274700 1 -- start start 2024-01-26T23:42:34.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.893+0000 7f2412274700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f240c0caad0 0x7f240c14f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:34.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.893+0000 7f2412274700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f240c1615b0 0x7f240c0c4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:34.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.893+0000 7f2412274700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f240c0c4a70 0x7f240c15ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:34.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.893+0000 7f2412274700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0c4f10 con 0x7f240c0caad0 2024-01-26T23:42:34.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.893+0000 7f2412274700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0c5d20 con 0x7f240c1615b0 2024-01-26T23:42:34.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.893+0000 7f2412274700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240c15d4f0 con 0x7f240c0c4a70 2024-01-26T23:42:34.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.893+0000 7f240bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f240c0caad0 0x7f240c14f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:34.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.893+0000 7f240bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f240c0caad0 0x7f240c14f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60004/0 (socket says 172.21.15.46:60004) 2024-01-26T23:42:34.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.893+0000 7f240bfff700 1 -- 172.21.15.46:0/1425987927 learned_addr learned my addr 172.21.15.46:0/1425987927 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:34.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.893+0000 7f2410811700 1 --2- 172.21.15.46:0/1425987927 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f240c0c4a70 0x7f240c15ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:34.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.894+0000 7f240b7fe700 1 --2- 172.21.15.46:0/1425987927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f240c1615b0 0x7f240c0c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:34.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.894+0000 7f240bfff700 1 -- 172.21.15.46:0/1425987927 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f240c0c4a70 msgr2=0x7f240c15ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:34.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.894+0000 7f240bfff700 1 --2- 172.21.15.46:0/1425987927 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f240c0c4a70 0x7f240c15ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:34.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.894+0000 7f240bfff700 1 -- 172.21.15.46:0/1425987927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f240c1615b0 msgr2=0x7f240c0c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:34.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.894+0000 7f240bfff700 1 --2- 172.21.15.46:0/1425987927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f240c1615b0 0x7f240c0c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:34.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.894+0000 7f240bfff700 1 -- 172.21.15.46:0/1425987927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc0094f0 con 0x7f240c0caad0 2024-01-26T23:42:34.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.895+0000 7f240bfff700 1 --2- 172.21.15.46:0/1425987927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f240c0caad0 0x7f240c14f9f0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f23fc006fd0 tx=0x7f23fc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:34.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.895+0000 7f23f3fff700 1 -- 172.21.15.46:0/1425987927 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc00bbe0 con 0x7f240c0caad0 2024-01-26T23:42:34.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.895+0000 7f2412274700 1 -- 172.21.15.46:0/1425987927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240c0c5110 con 0x7f240c0caad0 2024-01-26T23:42:34.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.895+0000 7f2412274700 1 -- 172.21.15.46:0/1425987927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f240c0c56d0 con 0x7f240c0caad0 2024-01-26T23:42:34.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.896+0000 7f23f3fff700 1 -- 172.21.15.46:0/1425987927 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23fc011630 con 0x7f240c0caad0 2024-01-26T23:42:34.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.896+0000 7f23f3fff700 1 -- 172.21.15.46:0/1425987927 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc0117a0 con 0x7f240c0caad0 2024-01-26T23:42:34.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.897+0000 7f2412274700 1 -- 172.21.15.46:0/1425987927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23ec0052b0 con 0x7f240c0caad0 2024-01-26T23:42:34.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.898+0000 7f23f3fff700 1 -- 172.21.15.46:0/1425987927 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f23fc011bd0 con 0x7f240c0caad0 2024-01-26T23:42:34.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.898+0000 7f23f3fff700 1 --2- 172.21.15.46:0/1425987927 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f23f8067130 0x7f23f80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:34.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.898+0000 7f23f3fff700 1 -- 172.21.15.46:0/1425987927 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6832+0+0 (secure 0 0 0) 0x7f23fc0865e0 con 0x7f240c0caad0 2024-01-26T23:42:34.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.899+0000 7f240b7fe700 1 --2- 172.21.15.46:0/1425987927 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f23f8067130 0x7f23f80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:34.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.900+0000 7f240b7fe700 1 --2- 172.21.15.46:0/1425987927 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f23f8067130 0x7f23f80695e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f240c0c6610 tx=0x7f2400008040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:34.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:34.906+0000 7f23f3fff700 1 -- 172.21.15.46:0/1425987927 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f23fc053170 con 0x7f240c0caad0 2024-01-26T23:42:35.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:35.142+0000 7f2412274700 1 -- 172.21.15.46:0/1425987927 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f23ec000bc0 con 0x7f23f8067130 2024-01-26T23:42:35.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:35.225+0000 7f23f3fff700 1 -- 172.21.15.46:0/1425987927 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f23ec000bc0 con 0x7f23f8067130 2024-01-26T23:42:35.225 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path /snap_test_dir1/dir4 2024-01-26T23:42:35.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:35.227+0000 7f2412274700 1 -- 172.21.15.46:0/1425987927 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f23f8067130 msgr2=0x7f23f80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:35.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:35.227+0000 7f2412274700 1 --2- 172.21.15.46:0/1425987927 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f23f8067130 0x7f23f80695e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f240c0c6610 tx=0x7f2400008040 comp rx=0 tx=0).stop 2024-01-26T23:42:35.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:35.227+0000 7f2412274700 1 -- 172.21.15.46:0/1425987927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f240c0caad0 msgr2=0x7f240c14f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:35.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:35.227+0000 7f2412274700 1 --2- 172.21.15.46:0/1425987927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f240c0caad0 0x7f240c14f9f0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f23fc006fd0 tx=0x7f23fc017000 comp rx=0 tx=0).stop 2024-01-26T23:42:35.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:35.227+0000 7f2412274700 1 -- 172.21.15.46:0/1425987927 shutdown_connections 2024-01-26T23:42:35.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:35.227+0000 7f2412274700 1 --2- 172.21.15.46:0/1425987927 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f23f8067130 0x7f23f80695e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:35.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:35.227+0000 7f2412274700 1 --2- 172.21.15.46:0/1425987927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f240c0caad0 0x7f240c14f9f0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:35.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:35.227+0000 7f2412274700 1 --2- 172.21.15.46:0/1425987927 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f240c0c4a70 0x7f240c15ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:35.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:35.227+0000 7f2412274700 1 --2- 172.21.15.46:0/1425987927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f240c1615b0 0x7f240c0c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:35.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:35.227+0000 7f2412274700 1 -- 172.21.15.46:0/1425987927 >> 172.21.15.46:0/1425987927 conn(0x7f240c0cdf50 msgr2=0x7f240c0c7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:35.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:35.227+0000 7f2412274700 1 -- 172.21.15.46:0/1425987927 shutdown_connections 2024-01-26T23:42:35.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:35.227+0000 7f2412274700 1 -- 172.21.15.46:0/1425987927 wait complete. 2024-01-26T23:42:35.242 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2024-01-26T23:42:35.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:35.242 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-01-26T23:42:35.273 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:35.273 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:35.274 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:35.274 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:35.274 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:35.274 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:35.274 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:35.274 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:35.274 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:35.274 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:35.274 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_01_UTC 2024-01-26T23:42:35.274 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:35.275 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:35.275 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_32_00_UTC) 2024-01-26T23:42:35.344 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:35.345 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:35.345 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_33_00_UTC) 2024-01-26T23:42:35.416 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:35.417 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:35.417 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_34_00_UTC) 2024-01-26T23:42:35.488 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:35.489 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:35.489 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_35_00_UTC) 2024-01-26T23:42:35.561 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:35.561 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:35.561 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_36_00_UTC) 2024-01-26T23:42:35.638 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:35.639 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:35.639 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_37_00_UTC) 2024-01-26T23:42:35.755 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:35.755 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:35.756 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_38_00_UTC) 2024-01-26T23:42:35.784 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:35.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:35.784 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_39_00_UTC) 2024-01-26T23:42:35.855 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:35.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:35.856 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_40_00_UTC) 2024-01-26T23:42:35.929 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:35.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:35.930 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_41_00_UTC) 2024-01-26T23:42:36.001 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_42_01_UTC 2024-01-26T23:42:36.001 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:36.001 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-26-23_42_01_UTC) 2024-01-26T23:42:36.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:36.072 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2024-01-26T23:42:36.140 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir5 1m 2024-01-26T23:42:36.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.323+0000 7ff5f0101700 1 Processor -- start 2024-01-26T23:42:36.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.324+0000 7ff5f0101700 1 -- start start 2024-01-26T23:42:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.324+0000 7ff5f0101700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5e8157490 0x7ff5e80cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.324+0000 7ff5f0101700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff5e806ddb0 0x7ff5e80cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.324+0000 7ff5f0101700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e80cea80 con 0x7ff5e8157860 2024-01-26T23:42:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.324+0000 7ff5f0101700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e80cdc60 con 0x7ff5e806ddb0 2024-01-26T23:42:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.324+0000 7ff5f0101700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e80b2ae0 con 0x7ff5e8157490 2024-01-26T23:42:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.325+0000 7ff5ee69e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff5e806ddb0 0x7ff5e80cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.325+0000 7ff5ed69c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff5e8157860 0x7ff5e806d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50006/0 (socket says 172.21.15.46:50006) 2024-01-26T23:42:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.325+0000 7ff5ed69c700 1 -- 172.21.15.46:0/89584187 learned_addr learned my addr 172.21.15.46:0/89584187 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.325+0000 7ff5ede9d700 1 --2- 172.21.15.46:0/89584187 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5e8157490 0x7ff5e80cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.325+0000 7ff5ee69e700 1 -- 172.21.15.46:0/89584187 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5e8157490 msgr2=0x7ff5e80cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.325+0000 7ff5ee69e700 1 --2- 172.21.15.46:0/89584187 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5e8157490 0x7ff5e80cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.325+0000 7ff5ee69e700 1 -- 172.21.15.46:0/89584187 >> v1:172.21.15.46:6789/0 conn(0x7ff5e8157860 legacy=0x7ff5e806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.325+0000 7ff5ee69e700 1 -- 172.21.15.46:0/89584187 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e80cde70 con 0x7ff5e806ddb0 2024-01-26T23:42:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.325+0000 7ff5ee69e700 1 --2- 172.21.15.46:0/89584187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5e806ddb0 0x7ff5e80cd620 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0009e60 tx=0x7ff5e0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd1669fdc0bd90d server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:36.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.325+0000 7ff5ece9b700 1 -- 172.21.15.46:0/89584187 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e000e070 con 0x7ff5e806ddb0 2024-01-26T23:42:36.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.325+0000 7ff5ece9b700 1 -- 172.21.15.46:0/89584187 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e000bc30 con 0x7ff5e806ddb0 2024-01-26T23:42:36.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.325+0000 7ff5ece9b700 1 -- 172.21.15.46:0/89584187 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0012430 con 0x7ff5e806ddb0 2024-01-26T23:42:36.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.326+0000 7ff5f0101700 1 -- 172.21.15.46:0/89584187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5e806ddb0 msgr2=0x7ff5e80cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:36.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.326+0000 7ff5f0101700 1 --2- 172.21.15.46:0/89584187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5e806ddb0 0x7ff5e80cd620 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0009e60 tx=0x7ff5e0009780 comp rx=0 tx=0).stop 2024-01-26T23:42:36.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.326+0000 7ff5f0101700 1 -- 172.21.15.46:0/89584187 shutdown_connections 2024-01-26T23:42:36.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.326+0000 7ff5f0101700 1 --2- 172.21.15.46:0/89584187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5e806ddb0 0x7ff5e80cd620 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:36.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.326+0000 7ff5f0101700 1 --2- 172.21.15.46:0/89584187 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5e8157490 0x7ff5e80cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:36.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.326+0000 7ff5f0101700 1 -- 172.21.15.46:0/89584187 >> 172.21.15.46:0/89584187 conn(0x7ff5e8067930 msgr2=0x7ff5e8067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:36.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.326+0000 7ff5f0101700 1 -- 172.21.15.46:0/89584187 shutdown_connections 2024-01-26T23:42:36.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.326+0000 7ff5f0101700 1 -- 172.21.15.46:0/89584187 wait complete. 2024-01-26T23:42:36.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.327+0000 7ff5f0101700 1 Processor -- start 2024-01-26T23:42:36.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.327+0000 7ff5f0101700 1 -- start start 2024-01-26T23:42:36.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.328+0000 7ff5f0101700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5e806ddb0 0x7ff5e8159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.328+0000 7ff5f0101700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5e8157490 0x7ff5e8167c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.328+0000 7ff5f0101700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5e8157860 0x7ff5e813d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.328+0000 7ff5f0101700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e81681d0 con 0x7ff5e8157860 2024-01-26T23:42:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.328+0000 7ff5f0101700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e8169010 con 0x7ff5e8157490 2024-01-26T23:42:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.328+0000 7ff5f0101700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e81686e0 con 0x7ff5e806ddb0 2024-01-26T23:42:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.328+0000 7ff5ee69e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5e8157860 0x7ff5e813d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.328+0000 7ff5ee69e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5e8157860 0x7ff5e813d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60024/0 (socket says 172.21.15.46:60024) 2024-01-26T23:42:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.328+0000 7ff5ee69e700 1 -- 172.21.15.46:0/2084142477 learned_addr learned my addr 172.21.15.46:0/2084142477 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.328+0000 7ff5ede9d700 1 --2- 172.21.15.46:0/2084142477 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5e806ddb0 0x7ff5e8159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.328+0000 7ff5ed69c700 1 --2- 172.21.15.46:0/2084142477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5e8157490 0x7ff5e8167c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.328+0000 7ff5ee69e700 1 -- 172.21.15.46:0/2084142477 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5e806ddb0 msgr2=0x7ff5e8159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.328+0000 7ff5ee69e700 1 --2- 172.21.15.46:0/2084142477 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5e806ddb0 0x7ff5e8159970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.328+0000 7ff5ee69e700 1 -- 172.21.15.46:0/2084142477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5e8157490 msgr2=0x7ff5e8167c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:36.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.328+0000 7ff5ee69e700 1 --2- 172.21.15.46:0/2084142477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5e8157490 0x7ff5e8167c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:36.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.328+0000 7ff5ee69e700 1 -- 172.21.15.46:0/2084142477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e0009a20 con 0x7ff5e8157860 2024-01-26T23:42:36.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.328+0000 7ff5ede9d700 1 --2- 172.21.15.46:0/2084142477 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5e806ddb0 0x7ff5e8159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:42:36.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.329+0000 7ff5ee69e700 1 --2- 172.21.15.46:0/2084142477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5e8157860 0x7ff5e813d6f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0013000 tx=0x7ff5e0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:36.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.329+0000 7ff5ddffb700 1 -- 172.21.15.46:0/2084142477 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0012e50 con 0x7ff5e8157860 2024-01-26T23:42:36.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.329+0000 7ff5ddffb700 1 -- 172.21.15.46:0/2084142477 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e001e630 con 0x7ff5e8157860 2024-01-26T23:42:36.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.329+0000 7ff5f0101700 1 -- 172.21.15.46:0/2084142477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e81689c0 con 0x7ff5e8157860 2024-01-26T23:42:36.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.329+0000 7ff5ddffb700 1 -- 172.21.15.46:0/2084142477 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e001ea10 con 0x7ff5e8157860 2024-01-26T23:42:36.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.329+0000 7ff5f0101700 1 -- 172.21.15.46:0/2084142477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e81683e0 con 0x7ff5e8157860 2024-01-26T23:42:36.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.330+0000 7ff5f0101700 1 -- 172.21.15.46:0/2084142477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e804c710 con 0x7ff5e8157860 2024-01-26T23:42:36.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.330+0000 7ff5ddffb700 1 -- 172.21.15.46:0/2084142477 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7ff5e0024070 con 0x7ff5e8157860 2024-01-26T23:42:36.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.331+0000 7ff5ddffb700 1 --2- 172.21.15.46:0/2084142477 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5cc067130 0x7ff5cc0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:36.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.331+0000 7ff5ddffb700 1 -- 172.21.15.46:0/2084142477 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6880+0+0 (secure 0 0 0) 0x7ff5e00875a0 con 0x7ff5e8157860 2024-01-26T23:42:36.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.331+0000 7ff5ede9d700 1 --2- 172.21.15.46:0/2084142477 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5cc067130 0x7ff5cc0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:36.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.332+0000 7ff5ede9d700 1 --2- 172.21.15.46:0/2084142477 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5cc067130 0x7ff5cc0695e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8006fd0 tx=0x7ff5d8008040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:36.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.335+0000 7ff5ddffb700 1 -- 172.21.15.46:0/2084142477 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff5e0054080 con 0x7ff5e8157860 2024-01-26T23:42:36.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.567+0000 7ff5f0101700 1 -- 172.21.15.46:0/2084142477 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5e80b28c0 con 0x7ff5cc067130 2024-01-26T23:42:36.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.661+0000 7ff5ddffb700 1 -- 172.21.15.46:0/2084142477 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7ff5e80b28c0 con 0x7ff5cc067130 2024-01-26T23:42:36.662 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path /snap_test_dir1/dir5 2024-01-26T23:42:36.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.664+0000 7ff5f0101700 1 -- 172.21.15.46:0/2084142477 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5cc067130 msgr2=0x7ff5cc0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:36.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.664+0000 7ff5f0101700 1 --2- 172.21.15.46:0/2084142477 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5cc067130 0x7ff5cc0695e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8006fd0 tx=0x7ff5d8008040 comp rx=0 tx=0).stop 2024-01-26T23:42:36.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.664+0000 7ff5f0101700 1 -- 172.21.15.46:0/2084142477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5e8157860 msgr2=0x7ff5e813d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:36.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.664+0000 7ff5f0101700 1 --2- 172.21.15.46:0/2084142477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5e8157860 0x7ff5e813d6f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0013000 tx=0x7ff5e0019000 comp rx=0 tx=0).stop 2024-01-26T23:42:36.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.664+0000 7ff5f0101700 1 -- 172.21.15.46:0/2084142477 shutdown_connections 2024-01-26T23:42:36.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.664+0000 7ff5f0101700 1 --2- 172.21.15.46:0/2084142477 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5cc067130 0x7ff5cc0695e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:36.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.664+0000 7ff5f0101700 1 --2- 172.21.15.46:0/2084142477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5e8157860 0x7ff5e813d6f0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:36.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.664+0000 7ff5f0101700 1 --2- 172.21.15.46:0/2084142477 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5e806ddb0 0x7ff5e8159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:36.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.664+0000 7ff5f0101700 1 --2- 172.21.15.46:0/2084142477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5e8157490 0x7ff5e8167c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:36.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.664+0000 7ff5f0101700 1 -- 172.21.15.46:0/2084142477 >> 172.21.15.46:0/2084142477 conn(0x7ff5e8067930 msgr2=0x7ff5e80b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:36.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.664+0000 7ff5f0101700 1 -- 172.21.15.46:0/2084142477 shutdown_connections 2024-01-26T23:42:36.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:36.664+0000 7ff5f0101700 1 -- 172.21.15.46:0/2084142477 wait complete. 2024-01-26T23:42:36.678 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2024-01-26T23:42:36.678 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:36.678 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-01-26T23:42:36.712 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:36.712 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:36.712 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:36.712 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:36.712 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:36.712 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:36.713 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:36.713 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:36.713 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:36.713 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:36.713 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:36.714 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:36.714 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:36.714 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_32_00_UTC) 2024-01-26T23:42:36.787 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:36.787 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:36.787 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_33_00_UTC) 2024-01-26T23:42:36.864 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:36.864 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:36.865 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_34_00_UTC) 2024-01-26T23:42:36.937 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:36.937 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:36.937 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_35_00_UTC) 2024-01-26T23:42:37.009 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:37.009 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:37.009 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_36_00_UTC) 2024-01-26T23:42:37.080 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:37.080 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:37.081 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_37_00_UTC) 2024-01-26T23:42:37.151 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:37.152 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:37.152 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_38_00_UTC) 2024-01-26T23:42:37.224 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:37.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:37.224 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_39_00_UTC) 2024-01-26T23:42:37.296 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:37.297 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:37.297 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_40_00_UTC) 2024-01-26T23:42:37.369 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:37.369 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:37.369 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_41_00_UTC) 2024-01-26T23:42:37.442 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:37.442 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:37.442 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-26-23_42_00_UTC) 2024-01-26T23:42:37.513 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:37.514 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2024-01-26T23:42:37.582 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir6 1m 2024-01-26T23:42:37.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.790+0000 7f101af95700 1 Processor -- start 2024-01-26T23:42:37.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.790+0000 7f101af95700 1 -- start start 2024-01-26T23:42:37.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.791+0000 7f101af95700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1014159cb0 0x7f10140cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:37.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.791+0000 7f101af95700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10140cf050 0x7f10140af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:37.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.791+0000 7f101af95700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f10140cf4a0 0x7f10140af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:37.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.791+0000 7f101af95700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10140cc100 con 0x7f1014159cb0 2024-01-26T23:42:37.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.791+0000 7f101af95700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10140cb850 con 0x7f10140cf4a0 2024-01-26T23:42:37.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.791+0000 7f101af95700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10140cbaf0 con 0x7f10140cf050 2024-01-26T23:42:37.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.791+0000 7f1018d31700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1014159cb0 0x7f10140cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:37.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.792+0000 7f1018d31700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1014159cb0 0x7f10140cc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60032/0 (socket says 172.21.15.46:60032) 2024-01-26T23:42:37.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.792+0000 7f1018d31700 1 -- 172.21.15.46:0/3886235430 learned_addr learned my addr 172.21.15.46:0/3886235430 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:37.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.792+0000 7f1013fff700 1 --2- 172.21.15.46:0/3886235430 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10140cf050 0x7f10140af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:37.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.792+0000 7f1019532700 1 --2- 172.21.15.46:0/3886235430 >> v2:172.21.15.106:3300/0 conn(0x7f10140cf4a0 0x7f10140af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:37.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.792+0000 7f1018d31700 1 -- 172.21.15.46:0/3886235430 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10140cf050 msgr2=0x7f10140af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:37.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.792+0000 7f1018d31700 1 --2- 172.21.15.46:0/3886235430 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10140cf050 0x7f10140af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:37.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.792+0000 7f1018d31700 1 -- 172.21.15.46:0/3886235430 >> v2:172.21.15.106:3300/0 conn(0x7f10140cf4a0 msgr2=0x7f10140af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:37.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.792+0000 7f1018d31700 1 --2- 172.21.15.46:0/3886235430 >> v2:172.21.15.106:3300/0 conn(0x7f10140cf4a0 0x7f10140af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:37.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.792+0000 7f1018d31700 1 -- 172.21.15.46:0/3886235430 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10140cfc80 con 0x7f1014159cb0 2024-01-26T23:42:37.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.792+0000 7f1018d31700 1 --2- 172.21.15.46:0/3886235430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1014159cb0 0x7f10140cc590 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f10140ce9e0 tx=0x7f1004009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44f4d513b7cfdf77 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:37.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.792+0000 7f10137fe700 1 -- 172.21.15.46:0/3886235430 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1004005c10 con 0x7f1014159cb0 2024-01-26T23:42:37.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.792+0000 7f10137fe700 1 -- 172.21.15.46:0/3886235430 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1004005d70 con 0x7f1014159cb0 2024-01-26T23:42:37.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.792+0000 7f10137fe700 1 -- 172.21.15.46:0/3886235430 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100400b880 con 0x7f1014159cb0 2024-01-26T23:42:37.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.793+0000 7f101af95700 1 -- 172.21.15.46:0/3886235430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1014159cb0 msgr2=0x7f10140cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:37.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.793+0000 7f101af95700 1 --2- 172.21.15.46:0/3886235430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1014159cb0 0x7f10140cc590 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f10140ce9e0 tx=0x7f1004009220 comp rx=0 tx=0).stop 2024-01-26T23:42:37.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.793+0000 7f101af95700 1 -- 172.21.15.46:0/3886235430 shutdown_connections 2024-01-26T23:42:37.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.793+0000 7f101af95700 1 --2- 172.21.15.46:0/3886235430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1014159cb0 0x7f10140cc590 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:37.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.793+0000 7f101af95700 1 --2- 172.21.15.46:0/3886235430 >> v2:172.21.15.106:3300/0 conn(0x7f10140cf4a0 0x7f10140af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:37.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.793+0000 7f101af95700 1 --2- 172.21.15.46:0/3886235430 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10140cf050 0x7f10140af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:37.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.793+0000 7f101af95700 1 -- 172.21.15.46:0/3886235430 >> 172.21.15.46:0/3886235430 conn(0x7f1014087f60 msgr2=0x7f101406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:37.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.793+0000 7f101af95700 1 -- 172.21.15.46:0/3886235430 shutdown_connections 2024-01-26T23:42:37.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.793+0000 7f101af95700 1 -- 172.21.15.46:0/3886235430 wait complete. 2024-01-26T23:42:37.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.794+0000 7f101af95700 1 Processor -- start 2024-01-26T23:42:37.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.794+0000 7f101af95700 1 -- start start 2024-01-26T23:42:37.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.795+0000 7f101af95700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10140cf050 0x7f1014156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:37.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.795+0000 7f101af95700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10140cf4a0 0x7f10140aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:37.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.795+0000 7f101af95700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1014159cb0 0x7f101415c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:37.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.795+0000 7f101af95700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101415e010 con 0x7f10140cf4a0 2024-01-26T23:42:37.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.795+0000 7f101af95700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10140b4070 con 0x7f10140cf050 2024-01-26T23:42:37.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.795+0000 7f101af95700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f101415cc80 con 0x7f1014159cb0 2024-01-26T23:42:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.795+0000 7f1013fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10140cf4a0 0x7f10140aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.795+0000 7f1013fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10140cf4a0 0x7f10140aff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60044/0 (socket says 172.21.15.46:60044) 2024-01-26T23:42:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.795+0000 7f1013fff700 1 -- 172.21.15.46:0/2504096763 learned_addr learned my addr 172.21.15.46:0/2504096763 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.795+0000 7f1018d31700 1 --2- 172.21.15.46:0/2504096763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10140cf050 0x7f1014156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.795+0000 7f1019532700 1 --2- 172.21.15.46:0/2504096763 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1014159cb0 0x7f101415c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.796+0000 7f1013fff700 1 -- 172.21.15.46:0/2504096763 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1014159cb0 msgr2=0x7f101415c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.796+0000 7f1013fff700 1 --2- 172.21.15.46:0/2504096763 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1014159cb0 0x7f101415c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.796+0000 7f1013fff700 1 -- 172.21.15.46:0/2504096763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10140cf050 msgr2=0x7f1014156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.796+0000 7f1013fff700 1 --2- 172.21.15.46:0/2504096763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10140cf050 0x7f1014156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.796+0000 7f1013fff700 1 -- 172.21.15.46:0/2504096763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1004009480 con 0x7f10140cf4a0 2024-01-26T23:42:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.796+0000 7f1013fff700 1 --2- 172.21.15.46:0/2504096763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10140cf4a0 0x7f10140aff40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f100800d890 tx=0x7f1008008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.796+0000 7f1010ff9700 1 -- 172.21.15.46:0/2504096763 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10080125f0 con 0x7f10140cf4a0 2024-01-26T23:42:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.796+0000 7f1010ff9700 1 -- 172.21.15.46:0/2504096763 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1008012750 con 0x7f10140cf4a0 2024-01-26T23:42:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.796+0000 7f101af95700 1 -- 172.21.15.46:0/2504096763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f101415cf60 con 0x7f10140cf4a0 2024-01-26T23:42:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.796+0000 7f1010ff9700 1 -- 172.21.15.46:0/2504096763 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008012af0 con 0x7f10140cf4a0 2024-01-26T23:42:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.796+0000 7f101af95700 1 -- 172.21.15.46:0/2504096763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f101415dca0 con 0x7f10140cf4a0 2024-01-26T23:42:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.797+0000 7f101af95700 1 -- 172.21.15.46:0/2504096763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f101415e1c0 con 0x7f10140cf4a0 2024-01-26T23:42:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.798+0000 7f1010ff9700 1 -- 172.21.15.46:0/2504096763 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f100801e460 con 0x7f10140cf4a0 2024-01-26T23:42:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.798+0000 7f1010ff9700 1 --2- 172.21.15.46:0/2504096763 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1000067130 0x7f10000695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.798+0000 7f1010ff9700 1 -- 172.21.15.46:0/2504096763 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f1008085230 con 0x7f10140cf4a0 2024-01-26T23:42:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.798+0000 7f1018d31700 1 --2- 172.21.15.46:0/2504096763 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1000067130 0x7f10000695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.799+0000 7f1018d31700 1 --2- 172.21.15.46:0/2504096763 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1000067130 0x7f10000695e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1004009260 tx=0x7f1004016000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:37.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:37.803+0000 7f1010ff9700 1 -- 172.21.15.46:0/2504096763 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f101415e1c0 con 0x7f10140cf4a0 2024-01-26T23:42:38.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:38.033+0000 7f101af95700 1 -- 172.21.15.46:0/2504096763 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f101404c7b0 con 0x7f1000067130 2024-01-26T23:42:38.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:38.107+0000 7f1010ff9700 1 -- 172.21.15.46:0/2504096763 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f101404c7b0 con 0x7f1000067130 2024-01-26T23:42:38.108 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path /snap_test_dir1/dir6 2024-01-26T23:42:38.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:38.110+0000 7f101af95700 1 -- 172.21.15.46:0/2504096763 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1000067130 msgr2=0x7f10000695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:38.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:38.110+0000 7f101af95700 1 --2- 172.21.15.46:0/2504096763 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1000067130 0x7f10000695e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1004009260 tx=0x7f1004016000 comp rx=0 tx=0).stop 2024-01-26T23:42:38.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:38.110+0000 7f101af95700 1 -- 172.21.15.46:0/2504096763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10140cf4a0 msgr2=0x7f10140aff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:38.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:38.110+0000 7f101af95700 1 --2- 172.21.15.46:0/2504096763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10140cf4a0 0x7f10140aff40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f100800d890 tx=0x7f1008008040 comp rx=0 tx=0).stop 2024-01-26T23:42:38.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:38.110+0000 7f101af95700 1 -- 172.21.15.46:0/2504096763 shutdown_connections 2024-01-26T23:42:38.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:38.110+0000 7f101af95700 1 --2- 172.21.15.46:0/2504096763 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1000067130 0x7f10000695e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:38.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:38.110+0000 7f101af95700 1 --2- 172.21.15.46:0/2504096763 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1014159cb0 0x7f101415c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:38.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:38.110+0000 7f101af95700 1 --2- 172.21.15.46:0/2504096763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10140cf050 0x7f1014156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:38.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:38.110+0000 7f101af95700 1 --2- 172.21.15.46:0/2504096763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10140cf4a0 0x7f10140aff40 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:38.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:38.110+0000 7f101af95700 1 -- 172.21.15.46:0/2504096763 >> 172.21.15.46:0/2504096763 conn(0x7f1014087f60 msgr2=0x7f1014158d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:38.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:38.110+0000 7f101af95700 1 -- 172.21.15.46:0/2504096763 shutdown_connections 2024-01-26T23:42:38.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:38.110+0000 7f101af95700 1 -- 172.21.15.46:0/2504096763 wait complete. 2024-01-26T23:42:38.124 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2024-01-26T23:42:38.124 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:38.124 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-01-26T23:42:38.152 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:38.152 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:38.152 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:38.152 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:38.152 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:38.152 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:38.153 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:38.153 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:38.153 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:38.153 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:38.153 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:38.154 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:38.154 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:38.154 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_32_00_UTC) 2024-01-26T23:42:38.224 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:38.225 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:38.225 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_33_00_UTC) 2024-01-26T23:42:38.297 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:38.297 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:38.297 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_34_00_UTC) 2024-01-26T23:42:38.369 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:38.369 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:38.370 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_35_00_UTC) 2024-01-26T23:42:38.441 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:38.442 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:38.442 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_36_00_UTC) 2024-01-26T23:42:38.513 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:38.514 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:38.514 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_37_00_UTC) 2024-01-26T23:42:38.585 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:38.585 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:38.585 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_38_00_UTC) 2024-01-26T23:42:38.699 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:38.699 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:38.699 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_39_00_UTC) 2024-01-26T23:42:38.742 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:38.742 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:38.742 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_40_00_UTC) 2024-01-26T23:42:38.810 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:38.810 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:38.811 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_41_00_UTC) 2024-01-26T23:42:38.882 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:38.882 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:38.882 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-26-23_42_00_UTC) 2024-01-26T23:42:38.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:38.954 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2024-01-26T23:42:39.021 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir7 1m 2024-01-26T23:42:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.205+0000 7f0e77658700 1 Processor -- start 2024-01-26T23:42:39.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.205+0000 7f0e77658700 1 -- start start 2024-01-26T23:42:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.206+0000 7f0e77658700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e70159cb0 0x7f0e700cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.206+0000 7f0e77658700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e700cf050 0x7f0e700af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.206+0000 7f0e77658700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0e700cf4a0 0x7f0e700af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.206+0000 7f0e77658700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e700cc100 con 0x7f0e70159cb0 2024-01-26T23:42:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.206+0000 7f0e77658700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e700cb850 con 0x7f0e700cf4a0 2024-01-26T23:42:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.206+0000 7f0e77658700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e700cbaf0 con 0x7f0e700cf050 2024-01-26T23:42:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.206+0000 7f0e753f4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e70159cb0 0x7f0e700cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.206+0000 7f0e753f4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e70159cb0 0x7f0e700cc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60048/0 (socket says 172.21.15.46:60048) 2024-01-26T23:42:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.206+0000 7f0e753f4700 1 -- 172.21.15.46:0/3488406777 learned_addr learned my addr 172.21.15.46:0/3488406777 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.206+0000 7f0e74bf3700 1 --2- 172.21.15.46:0/3488406777 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e700cf050 0x7f0e700af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.206+0000 7f0e75bf5700 1 --2- 172.21.15.46:0/3488406777 >> v2:172.21.15.106:3300/0 conn(0x7f0e700cf4a0 0x7f0e700af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.207+0000 7f0e753f4700 1 -- 172.21.15.46:0/3488406777 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e700cf050 msgr2=0x7f0e700af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.207+0000 7f0e753f4700 1 --2- 172.21.15.46:0/3488406777 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e700cf050 0x7f0e700af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.207+0000 7f0e753f4700 1 -- 172.21.15.46:0/3488406777 >> v2:172.21.15.106:3300/0 conn(0x7f0e700cf4a0 msgr2=0x7f0e700af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.207+0000 7f0e753f4700 1 --2- 172.21.15.46:0/3488406777 >> v2:172.21.15.106:3300/0 conn(0x7f0e700cf4a0 0x7f0e700af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.207+0000 7f0e753f4700 1 -- 172.21.15.46:0/3488406777 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e700cfc80 con 0x7f0e70159cb0 2024-01-26T23:42:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.207+0000 7f0e753f4700 1 --2- 172.21.15.46:0/3488406777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e70159cb0 0x7f0e700cc590 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0e700ce9e0 tx=0x7f0e6c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0f8dac6fffa178 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.207+0000 7f0e67fff700 1 -- 172.21.15.46:0/3488406777 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c005c10 con 0x7f0e70159cb0 2024-01-26T23:42:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.207+0000 7f0e67fff700 1 -- 172.21.15.46:0/3488406777 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e6c005d70 con 0x7f0e70159cb0 2024-01-26T23:42:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.207+0000 7f0e67fff700 1 -- 172.21.15.46:0/3488406777 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c006260 con 0x7f0e70159cb0 2024-01-26T23:42:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.208+0000 7f0e77658700 1 -- 172.21.15.46:0/3488406777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e70159cb0 msgr2=0x7f0e700cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.208+0000 7f0e77658700 1 --2- 172.21.15.46:0/3488406777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e70159cb0 0x7f0e700cc590 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0e700ce9e0 tx=0x7f0e6c009220 comp rx=0 tx=0).stop 2024-01-26T23:42:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.208+0000 7f0e77658700 1 -- 172.21.15.46:0/3488406777 shutdown_connections 2024-01-26T23:42:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.208+0000 7f0e77658700 1 --2- 172.21.15.46:0/3488406777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e70159cb0 0x7f0e700cc590 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.208+0000 7f0e77658700 1 --2- 172.21.15.46:0/3488406777 >> v2:172.21.15.106:3300/0 conn(0x7f0e700cf4a0 0x7f0e700af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.208+0000 7f0e77658700 1 --2- 172.21.15.46:0/3488406777 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e700cf050 0x7f0e700af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.208+0000 7f0e77658700 1 -- 172.21.15.46:0/3488406777 >> 172.21.15.46:0/3488406777 conn(0x7f0e70087f60 msgr2=0x7f0e7006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.208+0000 7f0e77658700 1 -- 172.21.15.46:0/3488406777 shutdown_connections 2024-01-26T23:42:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.208+0000 7f0e77658700 1 -- 172.21.15.46:0/3488406777 wait complete. 2024-01-26T23:42:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.209+0000 7f0e77658700 1 Processor -- start 2024-01-26T23:42:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.209+0000 7f0e77658700 1 -- start start 2024-01-26T23:42:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e77658700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e700cf050 0x7f0e70156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e77658700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e700cf4a0 0x7f0e700aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e77658700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e70159cb0 0x7f0e7015c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e77658700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7015e010 con 0x7f0e700cf050 2024-01-26T23:42:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e77658700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e700b4070 con 0x7f0e700cf4a0 2024-01-26T23:42:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e77658700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7015cc80 con 0x7f0e70159cb0 2024-01-26T23:42:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e753f4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e700cf050 0x7f0e70156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e753f4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e700cf050 0x7f0e70156ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60064/0 (socket says 172.21.15.46:60064) 2024-01-26T23:42:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e753f4700 1 -- 172.21.15.46:0/3039991731 learned_addr learned my addr 172.21.15.46:0/3039991731 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e74bf3700 1 --2- 172.21.15.46:0/3039991731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e700cf4a0 0x7f0e700aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e75bf5700 1 --2- 172.21.15.46:0/3039991731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e70159cb0 0x7f0e7015c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e753f4700 1 -- 172.21.15.46:0/3039991731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e70159cb0 msgr2=0x7f0e7015c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e753f4700 1 --2- 172.21.15.46:0/3039991731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e70159cb0 0x7f0e7015c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e753f4700 1 -- 172.21.15.46:0/3039991731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e700cf4a0 msgr2=0x7f0e700aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e753f4700 1 --2- 172.21.15.46:0/3039991731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e700cf4a0 0x7f0e700aff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e753f4700 1 -- 172.21.15.46:0/3039991731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6c009480 con 0x7f0e700cf050 2024-01-26T23:42:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e74bf3700 1 --2- 172.21.15.46:0/3039991731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e700cf4a0 0x7f0e700aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:42:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.210+0000 7f0e75bf5700 1 --2- 172.21.15.46:0/3039991731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e70159cb0 0x7f0e7015c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:42:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.211+0000 7f0e753f4700 1 --2- 172.21.15.46:0/3039991731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e700cf050 0x7f0e70156ea0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c009f50 tx=0x7f0e6c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.211+0000 7f0e657fa700 1 -- 172.21.15.46:0/3039991731 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c011800 con 0x7f0e700cf050 2024-01-26T23:42:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.211+0000 7f0e657fa700 1 -- 172.21.15.46:0/3039991731 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e6c020070 con 0x7f0e700cf050 2024-01-26T23:42:39.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.211+0000 7f0e77658700 1 -- 172.21.15.46:0/3039991731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7015cf60 con 0x7f0e700cf050 2024-01-26T23:42:39.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.211+0000 7f0e657fa700 1 -- 172.21.15.46:0/3039991731 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c011b90 con 0x7f0e700cf050 2024-01-26T23:42:39.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.211+0000 7f0e77658700 1 -- 172.21.15.46:0/3039991731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7015dca0 con 0x7f0e700cf050 2024-01-26T23:42:39.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.212+0000 7f0e77658700 1 -- 172.21.15.46:0/3039991731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e7004c710 con 0x7f0e700cf050 2024-01-26T23:42:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.212+0000 7f0e657fa700 1 -- 172.21.15.46:0/3039991731 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f0e6c022030 con 0x7f0e700cf050 2024-01-26T23:42:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.213+0000 7f0e657fa700 1 --2- 172.21.15.46:0/3039991731 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0e54067130 0x7f0e540695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.213+0000 7f0e657fa700 1 -- 172.21.15.46:0/3039991731 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6864+0+0 (secure 0 0 0) 0x7f0e6c086890 con 0x7f0e700cf050 2024-01-26T23:42:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.213+0000 7f0e74bf3700 1 --2- 172.21.15.46:0/3039991731 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0e54067130 0x7f0e540695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.214+0000 7f0e74bf3700 1 --2- 172.21.15.46:0/3039991731 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0e54067130 0x7f0e540695e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0e60006f00 tx=0x7f0e60008040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:39.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.217+0000 7f0e657fa700 1 -- 172.21.15.46:0/3039991731 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0e6c053400 con 0x7f0e700cf050 2024-01-26T23:42:39.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.449+0000 7f0e77658700 1 -- 172.21.15.46:0/3039991731 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e7015de30 con 0x7f0e54067130 2024-01-26T23:42:39.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.522+0000 7f0e657fa700 1 -- 172.21.15.46:0/3039991731 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f0e7015de30 con 0x7f0e54067130 2024-01-26T23:42:39.522 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path /snap_test_dir1/dir7 2024-01-26T23:42:39.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.525+0000 7f0e77658700 1 -- 172.21.15.46:0/3039991731 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0e54067130 msgr2=0x7f0e540695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:39.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.525+0000 7f0e77658700 1 --2- 172.21.15.46:0/3039991731 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0e54067130 0x7f0e540695e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0e60006f00 tx=0x7f0e60008040 comp rx=0 tx=0).stop 2024-01-26T23:42:39.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.525+0000 7f0e77658700 1 -- 172.21.15.46:0/3039991731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e700cf050 msgr2=0x7f0e70156ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:39.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.525+0000 7f0e77658700 1 --2- 172.21.15.46:0/3039991731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e700cf050 0x7f0e70156ea0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c009f50 tx=0x7f0e6c016000 comp rx=0 tx=0).stop 2024-01-26T23:42:39.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.525+0000 7f0e77658700 1 -- 172.21.15.46:0/3039991731 shutdown_connections 2024-01-26T23:42:39.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.525+0000 7f0e77658700 1 --2- 172.21.15.46:0/3039991731 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0e54067130 0x7f0e540695e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:39.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.525+0000 7f0e77658700 1 --2- 172.21.15.46:0/3039991731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e700cf050 0x7f0e70156ea0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:39.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.525+0000 7f0e77658700 1 --2- 172.21.15.46:0/3039991731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e70159cb0 0x7f0e7015c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:39.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.525+0000 7f0e77658700 1 --2- 172.21.15.46:0/3039991731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e700cf4a0 0x7f0e700aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:39.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.525+0000 7f0e77658700 1 -- 172.21.15.46:0/3039991731 >> 172.21.15.46:0/3039991731 conn(0x7f0e70087f60 msgr2=0x7f0e70158d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:39.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.525+0000 7f0e77658700 1 -- 172.21.15.46:0/3039991731 shutdown_connections 2024-01-26T23:42:39.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:39.525+0000 7f0e77658700 1 -- 172.21.15.46:0/3039991731 wait complete. 2024-01-26T23:42:39.539 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2024-01-26T23:42:39.539 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:39.540 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-01-26T23:42:39.568 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:39.569 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:39.569 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:39.569 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:39.569 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:39.569 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:39.569 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:39.569 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:39.569 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:39.569 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:39.569 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:39.570 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:39.570 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:39.570 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_32_00_UTC) 2024-01-26T23:42:39.684 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:39.685 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:39.685 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_33_00_UTC) 2024-01-26T23:42:39.712 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:39.712 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:39.712 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_34_00_UTC) 2024-01-26T23:42:39.781 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:39.782 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:39.782 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_35_00_UTC) 2024-01-26T23:42:39.853 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:39.854 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:39.854 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_36_00_UTC) 2024-01-26T23:42:39.926 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:39.926 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:39.926 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_37_00_UTC) 2024-01-26T23:42:39.998 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:39.998 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:39.998 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_38_00_UTC) 2024-01-26T23:42:40.070 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:40.070 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:40.070 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_39_00_UTC) 2024-01-26T23:42:40.141 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:40.142 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:40.142 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_40_00_UTC) 2024-01-26T23:42:40.213 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:40.214 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:40.214 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_41_00_UTC) 2024-01-26T23:42:40.286 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:40.286 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:40.287 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-26-23_42_00_UTC) 2024-01-26T23:42:40.358 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:40.358 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2024-01-26T23:42:40.425 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir8 1m 2024-01-26T23:42:40.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.608+0000 7ff6d346f700 1 Processor -- start 2024-01-26T23:42:40.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.608+0000 7ff6d346f700 1 -- start start 2024-01-26T23:42:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.609+0000 7ff6d346f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff6cc067af0 0x7ff6cc067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.609+0000 7ff6d346f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6cc06ebe0 0x7ff6cc0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.609+0000 7ff6d346f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6cc06fdb0 con 0x7ff6cc1597e0 2024-01-26T23:42:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.609+0000 7ff6d346f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6cc0ade10 con 0x7ff6cc067af0 2024-01-26T23:42:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.609+0000 7ff6d346f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc06f1a0 con 0x7ff6cc06ebe0 2024-01-26T23:42:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.609+0000 7ff6d120b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff6cc067af0 0x7ff6cc067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.609+0000 7ff6d1a0c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff6cc1597e0 0x7ff6cc159bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42070/0 (socket says 172.21.15.46:42070) 2024-01-26T23:42:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.609+0000 7ff6d1a0c700 1 -- 172.21.15.46:0/3093270878 learned_addr learned my addr 172.21.15.46:0/3093270878 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.609+0000 7ff6d0a0a700 1 --2- 172.21.15.46:0/3093270878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6cc06ebe0 0x7ff6cc0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.609+0000 7ff6d120b700 1 -- 172.21.15.46:0/3093270878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6cc06ebe0 msgr2=0x7ff6cc0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:40.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.609+0000 7ff6d120b700 1 --2- 172.21.15.46:0/3093270878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6cc06ebe0 0x7ff6cc0ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:40.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.609+0000 7ff6d120b700 1 -- 172.21.15.46:0/3093270878 >> v1:172.21.15.46:6789/0 conn(0x7ff6cc1597e0 legacy=0x7ff6cc159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:40.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.609+0000 7ff6d120b700 1 -- 172.21.15.46:0/3093270878 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6cc06f480 con 0x7ff6cc067af0 2024-01-26T23:42:40.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.609+0000 7ff6bffff700 1 -- 172.21.15.46:0/3093270878 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293600353 0 0) 0x7ff6cc06fdb0 con 0x7ff6cc1597e0 2024-01-26T23:42:40.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.609+0000 7ff6d0a0a700 1 --2- 172.21.15.46:0/3093270878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6cc06ebe0 0x7ff6cc0ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:42:40.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.610+0000 7ff6d120b700 1 --2- 172.21.15.46:0/3093270878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6cc067af0 0x7ff6cc067ec0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00098e0 tx=0x7ff6c00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=752d4f270e9b96b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:40.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.610+0000 7ff6bffff700 1 -- 172.21.15.46:0/3093270878 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0009e80 con 0x7ff6cc067af0 2024-01-26T23:42:40.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.610+0000 7ff6bffff700 1 -- 172.21.15.46:0/3093270878 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c000b860 con 0x7ff6cc067af0 2024-01-26T23:42:40.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.610+0000 7ff6bffff700 1 -- 172.21.15.46:0/3093270878 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c000bbe0 con 0x7ff6cc067af0 2024-01-26T23:42:40.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.610+0000 7ff6d346f700 1 -- 172.21.15.46:0/3093270878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6cc067af0 msgr2=0x7ff6cc067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:40.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.610+0000 7ff6d346f700 1 --2- 172.21.15.46:0/3093270878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6cc067af0 0x7ff6cc067ec0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00098e0 tx=0x7ff6c00091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:40.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.610+0000 7ff6d346f700 1 -- 172.21.15.46:0/3093270878 shutdown_connections 2024-01-26T23:42:40.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.610+0000 7ff6d346f700 1 --2- 172.21.15.46:0/3093270878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6cc067af0 0x7ff6cc067ec0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:40.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.610+0000 7ff6d346f700 1 --2- 172.21.15.46:0/3093270878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6cc06ebe0 0x7ff6cc0ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:40.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.610+0000 7ff6d346f700 1 -- 172.21.15.46:0/3093270878 >> 172.21.15.46:0/3093270878 conn(0x7ff6cc0cdf80 msgr2=0x7ff6cc06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:40.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.611+0000 7ff6d346f700 1 -- 172.21.15.46:0/3093270878 shutdown_connections 2024-01-26T23:42:40.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.611+0000 7ff6d346f700 1 -- 172.21.15.46:0/3093270878 wait complete. 2024-01-26T23:42:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.612+0000 7ff6d346f700 1 Processor -- start 2024-01-26T23:42:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.612+0000 7ff6d346f700 1 -- start start 2024-01-26T23:42:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.613+0000 7ff6d346f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6cc067af0 0x7ff6cc152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.613+0000 7ff6d346f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6cc06ebe0 0x7ff6cc159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.613+0000 7ff6d346f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6cc1597e0 0x7ff6cc158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.613+0000 7ff6d346f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc152770 con 0x7ff6cc067af0 2024-01-26T23:42:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.613+0000 7ff6d346f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc15a4f0 con 0x7ff6cc06ebe0 2024-01-26T23:42:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.613+0000 7ff6d346f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc15a7d0 con 0x7ff6cc1597e0 2024-01-26T23:42:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.613+0000 7ff6d120b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6cc067af0 0x7ff6cc152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.613+0000 7ff6d120b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6cc067af0 0x7ff6cc152230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38994/0 (socket says 172.21.15.46:38994) 2024-01-26T23:42:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.613+0000 7ff6d120b700 1 -- 172.21.15.46:0/2521233117 learned_addr learned my addr 172.21.15.46:0/2521233117 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:40.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.613+0000 7ff6d0a0a700 1 --2- 172.21.15.46:0/2521233117 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6cc06ebe0 0x7ff6cc159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:40.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.613+0000 7ff6d1a0c700 1 --2- 172.21.15.46:0/2521233117 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6cc1597e0 0x7ff6cc158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:40.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.613+0000 7ff6d120b700 1 -- 172.21.15.46:0/2521233117 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6cc1597e0 msgr2=0x7ff6cc158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:40.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.613+0000 7ff6d120b700 1 --2- 172.21.15.46:0/2521233117 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6cc1597e0 0x7ff6cc158910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:40.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.613+0000 7ff6d120b700 1 -- 172.21.15.46:0/2521233117 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6cc06ebe0 msgr2=0x7ff6cc159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:40.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.613+0000 7ff6d120b700 1 --2- 172.21.15.46:0/2521233117 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6cc06ebe0 0x7ff6cc159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:40.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.613+0000 7ff6d120b700 1 -- 172.21.15.46:0/2521233117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c0009420 con 0x7ff6cc067af0 2024-01-26T23:42:40.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.613+0000 7ff6d120b700 1 --2- 172.21.15.46:0/2521233117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6cc067af0 0x7ff6cc152230 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff6c000b3c0 tx=0x7ff6c0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:40.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.614+0000 7ff6bd7fa700 1 -- 172.21.15.46:0/2521233117 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0011420 con 0x7ff6cc067af0 2024-01-26T23:42:40.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.614+0000 7ff6bd7fa700 1 -- 172.21.15.46:0/2521233117 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c00115c0 con 0x7ff6cc067af0 2024-01-26T23:42:40.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.614+0000 7ff6bd7fa700 1 -- 172.21.15.46:0/2521233117 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0011980 con 0x7ff6cc067af0 2024-01-26T23:42:40.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.614+0000 7ff6d346f700 1 -- 172.21.15.46:0/2521233117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6cc15aab0 con 0x7ff6cc067af0 2024-01-26T23:42:40.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.614+0000 7ff6d346f700 1 -- 172.21.15.46:0/2521233117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6cc15ad90 con 0x7ff6cc067af0 2024-01-26T23:42:40.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.615+0000 7ff6d346f700 1 -- 172.21.15.46:0/2521233117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6cc04c710 con 0x7ff6cc067af0 2024-01-26T23:42:40.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.615+0000 7ff6bd7fa700 1 -- 172.21.15.46:0/2521233117 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7ff6c00253f0 con 0x7ff6cc067af0 2024-01-26T23:42:40.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.616+0000 7ff6bd7fa700 1 --2- 172.21.15.46:0/2521233117 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff6b00671b0 0x7ff6b0069660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:40.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.616+0000 7ff6bd7fa700 1 -- 172.21.15.46:0/2521233117 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6800+0+0 (secure 0 0 0) 0x7ff6c0086680 con 0x7ff6cc067af0 2024-01-26T23:42:40.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.616+0000 7ff6d0a0a700 1 --2- 172.21.15.46:0/2521233117 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff6b00671b0 0x7ff6b0069660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:40.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.616+0000 7ff6d0a0a700 1 --2- 172.21.15.46:0/2521233117 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff6b00671b0 0x7ff6b0069660 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc1588c0 tx=0x7ff6c8008040 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:40.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.620+0000 7ff6bd7fa700 1 -- 172.21.15.46:0/2521233117 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff6c00531b0 con 0x7ff6cc067af0 2024-01-26T23:42:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:40.885+0000 7ff6d346f700 1 -- 172.21.15.46:0/2521233117 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6cc068690 con 0x7ff6b00671b0 2024-01-26T23:42:41.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:41.008+0000 7ff6bd7fa700 1 -- 172.21.15.46:0/2521233117 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7ff6cc068690 con 0x7ff6b00671b0 2024-01-26T23:42:41.009 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path /snap_test_dir1/dir8 2024-01-26T23:42:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:41.011+0000 7ff6d346f700 1 -- 172.21.15.46:0/2521233117 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff6b00671b0 msgr2=0x7ff6b0069660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:41.011+0000 7ff6d346f700 1 --2- 172.21.15.46:0/2521233117 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff6b00671b0 0x7ff6b0069660 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc1588c0 tx=0x7ff6c8008040 comp rx=0 tx=0).stop 2024-01-26T23:42:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:41.011+0000 7ff6d346f700 1 -- 172.21.15.46:0/2521233117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6cc067af0 msgr2=0x7ff6cc152230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:41.011+0000 7ff6d346f700 1 --2- 172.21.15.46:0/2521233117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6cc067af0 0x7ff6cc152230 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff6c000b3c0 tx=0x7ff6c0017000 comp rx=0 tx=0).stop 2024-01-26T23:42:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:41.011+0000 7ff6d346f700 1 -- 172.21.15.46:0/2521233117 shutdown_connections 2024-01-26T23:42:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:41.011+0000 7ff6d346f700 1 --2- 172.21.15.46:0/2521233117 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff6b00671b0 0x7ff6b0069660 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:41.011+0000 7ff6d346f700 1 --2- 172.21.15.46:0/2521233117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6cc067af0 0x7ff6cc152230 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:41.011+0000 7ff6d346f700 1 --2- 172.21.15.46:0/2521233117 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6cc1597e0 0x7ff6cc158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:41.011+0000 7ff6d346f700 1 --2- 172.21.15.46:0/2521233117 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6cc06ebe0 0x7ff6cc159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:41.011+0000 7ff6d346f700 1 -- 172.21.15.46:0/2521233117 >> 172.21.15.46:0/2521233117 conn(0x7ff6cc0cdf80 msgr2=0x7ff6cc0c9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:41.011+0000 7ff6d346f700 1 -- 172.21.15.46:0/2521233117 shutdown_connections 2024-01-26T23:42:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:41.011+0000 7ff6d346f700 1 -- 172.21.15.46:0/2521233117 wait complete. 2024-01-26T23:42:41.025 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2024-01-26T23:42:41.025 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:41.025 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-01-26T23:42:41.059 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:41.059 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:41.059 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:41.059 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:41.059 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:41.059 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:41.059 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:41.059 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:41.060 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:41.060 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:41.060 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:41.061 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:41.061 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:41.061 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_32_00_UTC) 2024-01-26T23:42:41.129 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:41.129 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:41.130 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_33_00_UTC) 2024-01-26T23:42:41.199 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:41.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:41.199 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_34_00_UTC) 2024-01-26T23:42:41.270 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:41.271 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:41.271 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_35_00_UTC) 2024-01-26T23:42:41.343 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:41.343 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:41.344 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_36_00_UTC) 2024-01-26T23:42:41.415 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:41.415 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:41.415 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_37_00_UTC) 2024-01-26T23:42:41.486 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:41.486 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:41.486 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_38_00_UTC) 2024-01-26T23:42:41.558 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:41.558 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:41.558 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_39_00_UTC) 2024-01-26T23:42:41.641 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:41.641 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:41.642 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_40_00_UTC) 2024-01-26T23:42:41.713 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:41.713 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:41.714 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_41_00_UTC) 2024-01-26T23:42:41.785 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:41.786 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:41.786 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-26-23_42_00_UTC) 2024-01-26T23:42:41.859 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:41.859 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2024-01-26T23:42:41.926 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir9 1m 2024-01-26T23:42:42.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.109+0000 7f9622bea700 1 Processor -- start 2024-01-26T23:42:42.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.109+0000 7f9622bea700 1 -- start start 2024-01-26T23:42:42.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.110+0000 7f9622bea700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f961c0c8da0 0x7f961c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:42.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.110+0000 7f9622bea700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f961c0caf30 0x7f961c0cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:42.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.110+0000 7f9622bea700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f961c0ccc00 con 0x7f961c140af0 2024-01-26T23:42:42.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.110+0000 7f9622bea700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f961c0cbbf0 con 0x7f961c0caf30 2024-01-26T23:42:42.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.110+0000 7f9622bea700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961c0cc310 con 0x7f961c0c8da0 2024-01-26T23:42:42.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.110+0000 7f9621187700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f961c0caf30 0x7f961c0cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:42.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.110+0000 7f961bfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f961c140af0 0x7f961c140f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42082/0 (socket says 172.21.15.46:42082) 2024-01-26T23:42:42.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.110+0000 7f961bfff700 1 -- 172.21.15.46:0/738824421 learned_addr learned my addr 172.21.15.46:0/738824421 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:42.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.110+0000 7f9620986700 1 --2- 172.21.15.46:0/738824421 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f961c0c8da0 0x7f961c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:42.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.111+0000 7f9621187700 1 -- 172.21.15.46:0/738824421 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f961c0c8da0 msgr2=0x7f961c0c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:42.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.111+0000 7f9621187700 1 --2- 172.21.15.46:0/738824421 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f961c0c8da0 0x7f961c0c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:42.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.111+0000 7f9621187700 1 -- 172.21.15.46:0/738824421 >> v1:172.21.15.46:6789/0 conn(0x7f961c140af0 legacy=0x7f961c140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:42.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.111+0000 7f9621187700 1 -- 172.21.15.46:0/738824421 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f961c0cc5b0 con 0x7f961c0caf30 2024-01-26T23:42:42.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.111+0000 7f961b7fe700 1 -- 172.21.15.46:0/738824421 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225512720 0 0) 0x7f961c0ccc00 con 0x7f961c140af0 2024-01-26T23:42:42.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.111+0000 7f9621187700 1 --2- 172.21.15.46:0/738824421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f961c0caf30 0x7f961c0cb370 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f9614009e60 tx=0x7f9614009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=712408c924eb2166 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:42.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.111+0000 7f961b7fe700 1 -- 172.21.15.46:0/738824421 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961400e070 con 0x7f961c0caf30 2024-01-26T23:42:42.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.111+0000 7f961b7fe700 1 -- 172.21.15.46:0/738824421 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961400bb60 con 0x7f961c0caf30 2024-01-26T23:42:42.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.111+0000 7f961b7fe700 1 -- 172.21.15.46:0/738824421 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9614012430 con 0x7f961c0caf30 2024-01-26T23:42:42.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.112+0000 7f9622bea700 1 -- 172.21.15.46:0/738824421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f961c0caf30 msgr2=0x7f961c0cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:42.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.112+0000 7f9622bea700 1 --2- 172.21.15.46:0/738824421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f961c0caf30 0x7f961c0cb370 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f9614009e60 tx=0x7f9614009780 comp rx=0 tx=0).stop 2024-01-26T23:42:42.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.112+0000 7f9622bea700 1 -- 172.21.15.46:0/738824421 shutdown_connections 2024-01-26T23:42:42.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.112+0000 7f9622bea700 1 --2- 172.21.15.46:0/738824421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f961c0caf30 0x7f961c0cb370 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:42.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.112+0000 7f9622bea700 1 --2- 172.21.15.46:0/738824421 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f961c0c8da0 0x7f961c0c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:42.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.112+0000 7f9622bea700 1 -- 172.21.15.46:0/738824421 >> 172.21.15.46:0/738824421 conn(0x7f961c06bd90 msgr2=0x7f961c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:42.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.112+0000 7f9622bea700 1 -- 172.21.15.46:0/738824421 shutdown_connections 2024-01-26T23:42:42.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.112+0000 7f9622bea700 1 -- 172.21.15.46:0/738824421 wait complete. 2024-01-26T23:42:42.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.113+0000 7f9622bea700 1 Processor -- start 2024-01-26T23:42:42.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.113+0000 7f9622bea700 1 -- start start 2024-01-26T23:42:42.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.114+0000 7f9622bea700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f961c0c8da0 0x7f961c157000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:42.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.114+0000 7f9622bea700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f961c0caf30 0x7f961c0c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:42.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.114+0000 7f9622bea700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f961c140af0 0x7f961c16c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:42.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.114+0000 7f9622bea700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c0698c0 con 0x7f961c0caf30 2024-01-26T23:42:42.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.114+0000 7f9622bea700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c069ba0 con 0x7f961c0c8da0 2024-01-26T23:42:42.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.114+0000 7f9622bea700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961c069e80 con 0x7f961c140af0 2024-01-26T23:42:42.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.114+0000 7f961bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f961c0caf30 0x7f961c0c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:42.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.114+0000 7f961bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f961c0caf30 0x7f961c0c8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39010/0 (socket says 172.21.15.46:39010) 2024-01-26T23:42:42.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.114+0000 7f961bfff700 1 -- 172.21.15.46:0/352301628 learned_addr learned my addr 172.21.15.46:0/352301628 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:42.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.114+0000 7f9621187700 1 --2- 172.21.15.46:0/352301628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f961c140af0 0x7f961c16c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:42.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.114+0000 7f9620986700 1 --2- 172.21.15.46:0/352301628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f961c0c8da0 0x7f961c157000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:42.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.114+0000 7f961bfff700 1 -- 172.21.15.46:0/352301628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f961c140af0 msgr2=0x7f961c16c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:42.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.114+0000 7f961bfff700 1 --2- 172.21.15.46:0/352301628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f961c140af0 0x7f961c16c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:42.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.114+0000 7f961bfff700 1 -- 172.21.15.46:0/352301628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f961c0c8da0 msgr2=0x7f961c157000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:42.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.114+0000 7f961bfff700 1 --2- 172.21.15.46:0/352301628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f961c0c8da0 0x7f961c157000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:42.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.114+0000 7f961bfff700 1 -- 172.21.15.46:0/352301628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9614009a20 con 0x7f961c0caf30 2024-01-26T23:42:42.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.115+0000 7f961bfff700 1 --2- 172.21.15.46:0/352301628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f961c0caf30 0x7f961c0c8210 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f960400c5b0 tx=0x7f9604001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:42.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.115+0000 7f9618ff9700 1 -- 172.21.15.46:0/352301628 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9604005b30 con 0x7f961c0caf30 2024-01-26T23:42:42.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.115+0000 7f9618ff9700 1 -- 172.21.15.46:0/352301628 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9604005c90 con 0x7f961c0caf30 2024-01-26T23:42:42.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.115+0000 7f9622bea700 1 -- 172.21.15.46:0/352301628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f961c06a160 con 0x7f961c0caf30 2024-01-26T23:42:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.115+0000 7f9618ff9700 1 -- 172.21.15.46:0/352301628 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9604007300 con 0x7f961c0caf30 2024-01-26T23:42:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.115+0000 7f9622bea700 1 -- 172.21.15.46:0/352301628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961c0c8750 con 0x7f961c0caf30 2024-01-26T23:42:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.116+0000 7f9622bea700 1 -- 172.21.15.46:0/352301628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961c0c8970 con 0x7f961c0caf30 2024-01-26T23:42:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.116+0000 7f9618ff9700 1 -- 172.21.15.46:0/352301628 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f960401b3b0 con 0x7f961c0caf30 2024-01-26T23:42:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.117+0000 7f9618ff9700 1 --2- 172.21.15.46:0/352301628 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f95fc067130 0x7f95fc0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.117+0000 7f9618ff9700 1 -- 172.21.15.46:0/352301628 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7084+0+0 (secure 0 0 0) 0x7f9604084250 con 0x7f961c0caf30 2024-01-26T23:42:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.117+0000 7f9620986700 1 --2- 172.21.15.46:0/352301628 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f95fc067130 0x7f95fc0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:42.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.117+0000 7f9620986700 1 --2- 172.21.15.46:0/352301628 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f95fc067130 0x7f95fc0695e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f960c009db0 tx=0x7f960c0091a0 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:42.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.121+0000 7f9618ff9700 1 -- 172.21.15.46:0/352301628 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9604050ce0 con 0x7f961c0caf30 2024-01-26T23:42:42.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.355+0000 7f9622bea700 1 -- 172.21.15.46:0/352301628 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f961c04c710 con 0x7f95fc067130 2024-01-26T23:42:42.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.427+0000 7f9618ff9700 1 -- 172.21.15.46:0/352301628 <== mgr.4970 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f961c04c710 con 0x7f95fc067130 2024-01-26T23:42:42.427 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path /snap_test_dir1/dir9 2024-01-26T23:42:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.429+0000 7f9622bea700 1 -- 172.21.15.46:0/352301628 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f95fc067130 msgr2=0x7f95fc0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.429+0000 7f9622bea700 1 --2- 172.21.15.46:0/352301628 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f95fc067130 0x7f95fc0695e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f960c009db0 tx=0x7f960c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.429+0000 7f9622bea700 1 -- 172.21.15.46:0/352301628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f961c0caf30 msgr2=0x7f961c0c8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.429+0000 7f9622bea700 1 --2- 172.21.15.46:0/352301628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f961c0caf30 0x7f961c0c8210 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f960400c5b0 tx=0x7f9604001070 comp rx=0 tx=0).stop 2024-01-26T23:42:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.429+0000 7f9622bea700 1 -- 172.21.15.46:0/352301628 shutdown_connections 2024-01-26T23:42:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.429+0000 7f9622bea700 1 --2- 172.21.15.46:0/352301628 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f95fc067130 0x7f95fc0695e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.429+0000 7f9622bea700 1 --2- 172.21.15.46:0/352301628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f961c140af0 0x7f961c16c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.429+0000 7f9622bea700 1 --2- 172.21.15.46:0/352301628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f961c0c8da0 0x7f961c157000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.429+0000 7f9622bea700 1 --2- 172.21.15.46:0/352301628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f961c0caf30 0x7f961c0c8210 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.429+0000 7f9622bea700 1 -- 172.21.15.46:0/352301628 >> 172.21.15.46:0/352301628 conn(0x7f961c06bd90 msgr2=0x7f961c0cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.429+0000 7f9622bea700 1 -- 172.21.15.46:0/352301628 shutdown_connections 2024-01-26T23:42:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:42.429+0000 7f9622bea700 1 -- 172.21.15.46:0/352301628 wait complete. 2024-01-26T23:42:42.444 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2024-01-26T23:42:42.444 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:42.444 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-01-26T23:42:42.473 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:42.474 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:42.474 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:42.474 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:42.474 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:42.474 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:42.474 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:42.474 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:42.474 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:42.474 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:42.474 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:42.475 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_32_00_UTC 2024-01-26T23:42:42.475 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:42.475 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_32_00_UTC) 2024-01-26T23:42:42.545 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_33_00_UTC 2024-01-26T23:42:42.545 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:42.545 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_33_00_UTC) 2024-01-26T23:42:42.642 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_34_00_UTC 2024-01-26T23:42:42.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:42.643 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_34_00_UTC) 2024-01-26T23:42:42.676 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_35_00_UTC 2024-01-26T23:42:42.676 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:42.677 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_35_00_UTC) 2024-01-26T23:42:42.750 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_36_00_UTC 2024-01-26T23:42:42.750 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:42.751 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_36_00_UTC) 2024-01-26T23:42:42.822 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_37_00_UTC 2024-01-26T23:42:42.822 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:42.822 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_37_00_UTC) 2024-01-26T23:42:42.906 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_38_00_UTC 2024-01-26T23:42:42.907 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:42.907 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_38_00_UTC) 2024-01-26T23:42:42.937 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_39_00_UTC 2024-01-26T23:42:42.937 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:42.937 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_39_00_UTC) 2024-01-26T23:42:43.011 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_40_00_UTC 2024-01-26T23:42:43.011 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:43.011 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_40_00_UTC) 2024-01-26T23:42:43.083 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_41_00_UTC 2024-01-26T23:42:43.083 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:43.083 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_41_00_UTC) 2024-01-26T23:42:43.157 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_42_00_UTC 2024-01-26T23:42:43.158 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:43.158 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-26-23_42_00_UTC) 2024-01-26T23:42:43.229 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:42:43.230 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2024-01-26T23:42:43.297 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-26T23:42:43.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.483+0000 7f6dcbb08700 1 Processor -- start 2024-01-26T23:42:43.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.483+0000 7f6dcbb08700 1 -- start start 2024-01-26T23:42:43.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.484+0000 7f6dcbb08700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6dc40cc910 0x7f6dc40692c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.484+0000 7f6dcbb08700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6dc415d560 0x7f6dc415d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.484+0000 7f6dcbb08700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc40c8c80 con 0x7f6dc40cc540 2024-01-26T23:42:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.484+0000 7f6dcbb08700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dc40ccd80 con 0x7f6dc415d560 2024-01-26T23:42:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.484+0000 7f6dcbb08700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc4069920 con 0x7f6dc40cc910 2024-01-26T23:42:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.484+0000 7f6dc98a4700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6dc40cc540 0x7f6dc4068bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42090/0 (socket says 172.21.15.46:42090) 2024-01-26T23:42:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.484+0000 7f6dc98a4700 1 -- 172.21.15.46:0/2840558657 learned_addr learned my addr 172.21.15.46:0/2840558657 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.485+0000 7f6dc90a3700 1 --2- 172.21.15.46:0/2840558657 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6dc40cc910 0x7f6dc40692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.485+0000 7f6dca0a5700 1 --2- 172.21.15.46:0/2840558657 >> v2:172.21.15.106:3300/0 conn(0x7f6dc415d560 0x7f6dc415d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.485+0000 7f6dc88a2700 1 -- 172.21.15.46:0/2840558657 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537868062 0 0) 0x7f6dc40c8c80 con 0x7f6dc40cc540 2024-01-26T23:42:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.485+0000 7f6dc88a2700 1 -- 172.21.15.46:0/2840558657 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db0003570 con 0x7f6dc40cc540 2024-01-26T23:42:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.485+0000 7f6dc90a3700 1 -- 172.21.15.46:0/2840558657 >> v2:172.21.15.106:3300/0 conn(0x7f6dc415d560 msgr2=0x7f6dc415d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.485+0000 7f6dc90a3700 1 --2- 172.21.15.46:0/2840558657 >> v2:172.21.15.106:3300/0 conn(0x7f6dc415d560 0x7f6dc415d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.485+0000 7f6dc90a3700 1 -- 172.21.15.46:0/2840558657 >> v1:172.21.15.46:6789/0 conn(0x7f6dc40cc540 legacy=0x7f6dc4068bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:43.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.485+0000 7f6dc90a3700 1 -- 172.21.15.46:0/2840558657 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc4069c00 con 0x7f6dc40cc910 2024-01-26T23:42:43.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.485+0000 7f6dca0a5700 1 --2- 172.21.15.46:0/2840558657 >> v2:172.21.15.106:3300/0 conn(0x7f6dc415d560 0x7f6dc415d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:43.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.485+0000 7f6dc90a3700 1 --2- 172.21.15.46:0/2840558657 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6dc40cc910 0x7f6dc40692c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6dc00098e0 tx=0x7f6dc00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40c8c83c7881de64 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:43.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.485+0000 7f6dc88a2700 1 -- 172.21.15.46:0/2840558657 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0011630 con 0x7f6dc40cc910 2024-01-26T23:42:43.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.486+0000 7f6dc88a2700 1 -- 172.21.15.46:0/2840558657 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc0011790 con 0x7f6dc40cc910 2024-01-26T23:42:43.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.486+0000 7f6dc88a2700 1 -- 172.21.15.46:0/2840558657 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0011b30 con 0x7f6dc40cc910 2024-01-26T23:42:43.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.486+0000 7f6dcbb08700 1 -- 172.21.15.46:0/2840558657 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6dc40cc910 msgr2=0x7f6dc40692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:43.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.486+0000 7f6dcbb08700 1 --2- 172.21.15.46:0/2840558657 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6dc40cc910 0x7f6dc40692c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6dc00098e0 tx=0x7f6dc00091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:43.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.486+0000 7f6dcbb08700 1 -- 172.21.15.46:0/2840558657 shutdown_connections 2024-01-26T23:42:43.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.486+0000 7f6dcbb08700 1 --2- 172.21.15.46:0/2840558657 >> v2:172.21.15.106:3300/0 conn(0x7f6dc415d560 0x7f6dc415d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:43.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.486+0000 7f6dcbb08700 1 --2- 172.21.15.46:0/2840558657 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6dc40cc910 0x7f6dc40692c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:43.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.486+0000 7f6dcbb08700 1 -- 172.21.15.46:0/2840558657 >> 172.21.15.46:0/2840558657 conn(0x7f6dc406bd90 msgr2=0x7f6dc406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:43.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.486+0000 7f6dcbb08700 1 -- 172.21.15.46:0/2840558657 shutdown_connections 2024-01-26T23:42:43.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.486+0000 7f6dcbb08700 1 -- 172.21.15.46:0/2840558657 wait complete. 2024-01-26T23:42:43.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.487+0000 7f6dcbb08700 1 Processor -- start 2024-01-26T23:42:43.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.487+0000 7f6dcbb08700 1 -- start start 2024-01-26T23:42:43.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.488+0000 7f6dcbb08700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dc40cc540 0x7f6dc415c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:43.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.488+0000 7f6dcbb08700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6dc40cc910 0x7f6dc416da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:43.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.488+0000 7f6dcbb08700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6dc415d560 0x7f6dc416dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:43.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.488+0000 7f6dcbb08700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc415cce0 con 0x7f6dc40cc540 2024-01-26T23:42:43.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.488+0000 7f6dcbb08700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc40c7d70 con 0x7f6dc415d560 2024-01-26T23:42:43.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.488+0000 7f6dcbb08700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc416e650 con 0x7f6dc40cc910 2024-01-26T23:42:43.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.488+0000 7f6dc98a4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dc40cc540 0x7f6dc415c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:43.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.488+0000 7f6dc98a4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dc40cc540 0x7f6dc415c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39020/0 (socket says 172.21.15.46:39020) 2024-01-26T23:42:43.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.488+0000 7f6dc98a4700 1 -- 172.21.15.46:0/4229597708 learned_addr learned my addr 172.21.15.46:0/4229597708 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:43.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.488+0000 7f6dc90a3700 1 --2- 172.21.15.46:0/4229597708 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6dc40cc910 0x7f6dc416da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:43.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.489+0000 7f6dca0a5700 1 --2- 172.21.15.46:0/4229597708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6dc415d560 0x7f6dc416dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:43.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.489+0000 7f6dc98a4700 1 -- 172.21.15.46:0/4229597708 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6dc40cc910 msgr2=0x7f6dc416da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.489+0000 7f6dc98a4700 1 --2- 172.21.15.46:0/4229597708 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6dc40cc910 0x7f6dc416da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.489+0000 7f6dc98a4700 1 -- 172.21.15.46:0/4229597708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6dc415d560 msgr2=0x7f6dc416dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.489+0000 7f6dc98a4700 1 --2- 172.21.15.46:0/4229597708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6dc415d560 0x7f6dc416dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.489+0000 7f6dc98a4700 1 -- 172.21.15.46:0/4229597708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dbc009a20 con 0x7f6dc40cc540 2024-01-26T23:42:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.489+0000 7f6dca0a5700 1 --2- 172.21.15.46:0/4229597708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6dc415d560 0x7f6dc416dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.489+0000 7f6dc98a4700 1 --2- 172.21.15.46:0/4229597708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dc40cc540 0x7f6dc415c7a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6db800c7c0 tx=0x7f6db8007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.489+0000 7f6db5ffb700 1 -- 172.21.15.46:0/4229597708 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db8005b30 con 0x7f6dc40cc540 2024-01-26T23:42:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.489+0000 7f6db5ffb700 1 -- 172.21.15.46:0/4229597708 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db8005c90 con 0x7f6dc40cc540 2024-01-26T23:42:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.489+0000 7f6dcbb08700 1 -- 172.21.15.46:0/4229597708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc0009420 con 0x7f6dc40cc540 2024-01-26T23:42:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.489+0000 7f6db5ffb700 1 -- 172.21.15.46:0/4229597708 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db8007320 con 0x7f6dc40cc540 2024-01-26T23:42:43.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.489+0000 7f6dcbb08700 1 -- 172.21.15.46:0/4229597708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc416ef70 con 0x7f6dc40cc540 2024-01-26T23:42:43.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.490+0000 7f6dcbb08700 1 -- 172.21.15.46:0/4229597708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc404c710 con 0x7f6dc40cc540 2024-01-26T23:42:43.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.491+0000 7f6db5ffb700 1 -- 172.21.15.46:0/4229597708 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f6db801b3b0 con 0x7f6dc40cc540 2024-01-26T23:42:43.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.491+0000 7f6db5ffb700 1 --2- 172.21.15.46:0/4229597708 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6da8067130 0x7f6da80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:43.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.491+0000 7f6db5ffb700 1 -- 172.21.15.46:0/4229597708 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6816+0+0 (secure 0 0 0) 0x7f6db8084240 con 0x7f6dc40cc540 2024-01-26T23:42:43.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.491+0000 7f6dc90a3700 1 --2- 172.21.15.46:0/4229597708 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6da8067130 0x7f6da80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:43.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.492+0000 7f6dc90a3700 1 --2- 172.21.15.46:0/4229597708 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6da8067130 0x7f6da80695e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6dc00091e0 tx=0x7f6dc0017000 comp rx=0 tx=0).ready entity=mgr.4970 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:43.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.495+0000 7f6db5ffb700 1 -- 172.21.15.46:0/4229597708 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6db8050de0 con 0x7f6dc40cc540 2024-01-26T23:42:43.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:43.775+0000 7f6dcbb08700 1 -- 172.21.15.46:0/4229597708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f6dc415ce20 con 0x7f6dc40cc540 2024-01-26T23:42:44.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:44.622+0000 7f6db5ffb700 1 -- 172.21.15.46:0/4229597708 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f6db80511b0 con 0x7f6dc40cc540 2024-01-26T23:42:44.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:44.624+0000 7f6db5ffb700 1 -- 172.21.15.46:0/4229597708 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v14) v1 ==== 93+0+0 (secure 0 0 0) 0x7f6db80840a0 con 0x7f6dc40cc540 2024-01-26T23:42:44.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:44.631+0000 7f6dcbb08700 1 -- 172.21.15.46:0/4229597708 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6da8067130 msgr2=0x7f6da80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:44.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:44.631+0000 7f6dcbb08700 1 --2- 172.21.15.46:0/4229597708 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6da8067130 0x7f6da80695e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6dc00091e0 tx=0x7f6dc0017000 comp rx=0 tx=0).stop 2024-01-26T23:42:44.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:44.631+0000 7f6dcbb08700 1 -- 172.21.15.46:0/4229597708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dc40cc540 msgr2=0x7f6dc415c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:44.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:44.631+0000 7f6dcbb08700 1 --2- 172.21.15.46:0/4229597708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dc40cc540 0x7f6dc415c7a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6db800c7c0 tx=0x7f6db8007e00 comp rx=0 tx=0).stop 2024-01-26T23:42:44.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:44.632+0000 7f6dcbb08700 1 -- 172.21.15.46:0/4229597708 shutdown_connections 2024-01-26T23:42:44.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:44.632+0000 7f6dcbb08700 1 --2- 172.21.15.46:0/4229597708 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6da8067130 0x7f6da80695e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:44.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:44.632+0000 7f6dcbb08700 1 --2- 172.21.15.46:0/4229597708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dc40cc540 0x7f6dc415c7a0 secure :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6db800c7c0 tx=0x7f6db8007e00 comp rx=0 tx=0).stop 2024-01-26T23:42:44.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:44.632+0000 7f6dcbb08700 1 --2- 172.21.15.46:0/4229597708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6dc415d560 0x7f6dc416dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:44.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:44.632+0000 7f6dcbb08700 1 --2- 172.21.15.46:0/4229597708 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6dc40cc910 0x7f6dc416da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:44.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:44.632+0000 7f6dcbb08700 1 -- 172.21.15.46:0/4229597708 >> 172.21.15.46:0/4229597708 conn(0x7f6dc406bd90 msgr2=0x7f6dc40ca260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:44.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:44.637+0000 7f6dcbb08700 1 -- 172.21.15.46:0/4229597708 shutdown_connections 2024-01-26T23:42:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:44.648+0000 7f6dcbb08700 1 -- 172.21.15.46:0/4229597708 wait complete. 2024-01-26T23:42:44.682 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-26T23:42:44.683 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:42:44.751 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:42:44.793 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-26T23:42:44.794 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-26T23:42:44.794 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-26T23:42:44.794 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-26T23:42:44.834 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[58537]: fuse finished with error 0 and tester_r 0 2024-01-26T23:42:44.836 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:42:44.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:42:44.836 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-26T23:42:45.001 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:45.001+0000 7fce78a281c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:42:45.008 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:42:45.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:42:45.008 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-26T23:42:45.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:42:45.025 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-26T23:42:45.025 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:42:45.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:42:45.026 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-26T23:42:45.062 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:45.061+0000 7fc3d11d71c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:42:45.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:42:45.067 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-26T23:42:45.067 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:42:45.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:42:45.067 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-26T23:42:45.080 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:45.079+0000 7fce78a281c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:42:45.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:42:45.084 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-26T23:42:45.084 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:42:45.151 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:45.150+0000 7fc3d11d71c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:42:45.152 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:45.151+0000 7fce78a281c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:42:45.226 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:45.224+0000 7fce78a281c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:42:45.243 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:45.242+0000 7fc3d11d71c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:42:45.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.251+0000 7fcf08747700 1 Processor -- start 2024-01-26T23:42:45.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.251+0000 7fcf08747700 1 -- start start 2024-01-26T23:42:45.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.252+0000 7fcf08747700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcf00159590 0x7fcf0015d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:45.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.252+0000 7fcf08747700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf0015dd90 con 0x7fcf0006c3c0 2024-01-26T23:42:45.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.252+0000 7fcf08747700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf0015e070 con 0x7fcf0015d8f0 2024-01-26T23:42:45.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.252+0000 7fcf08747700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf0015e350 con 0x7fcf00159590 2024-01-26T23:42:45.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.252+0000 7fcf064e3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcf00159590 0x7fcf0015d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:45.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.252+0000 7fcf06ce4700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcf0006c3c0 0x7fcf00069370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42098/0 (socket says 172.21.15.46:42098) 2024-01-26T23:42:45.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.252+0000 7fcf06ce4700 1 -- 172.21.15.46:0/257639961 learned_addr learned my addr 172.21.15.46:0/257639961 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:45.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.253+0000 7fcf064e3700 1 -- 172.21.15.46:0/257639961 >> v1:172.21.15.106:6789/0 conn(0x7fcf0015d8f0 legacy=0x7fcf0006bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:45.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.253+0000 7fcf064e3700 1 -- 172.21.15.46:0/257639961 >> v1:172.21.15.46:6789/0 conn(0x7fcf0006c3c0 legacy=0x7fcf00069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:45.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.253+0000 7fcf054e1700 1 -- 172.21.15.46:0/257639961 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282137083 0 0) 0x7fcf0015e070 con 0x7fcf0015d8f0 2024-01-26T23:42:45.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.253+0000 7fcf064e3700 1 -- 172.21.15.46:0/257639961 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf00068b70 con 0x7fcf00159590 2024-01-26T23:42:45.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.253+0000 7fcf064e3700 1 --2- 172.21.15.46:0/257639961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf00159590 0x7fcf0015d3b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0098e0 tx=0x7fcefc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84db3ae696af5c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:45.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.253+0000 7fcf054e1700 1 -- 172.21.15.46:0/257639961 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc009e80 con 0x7fcf00159590 2024-01-26T23:42:45.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.253+0000 7fcf054e1700 1 -- 172.21.15.46:0/257639961 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcefc00b760 con 0x7fcf00159590 2024-01-26T23:42:45.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.253+0000 7fcf054e1700 1 -- 172.21.15.46:0/257639961 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc00bae0 con 0x7fcf00159590 2024-01-26T23:42:45.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.254+0000 7fcf08747700 1 -- 172.21.15.46:0/257639961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf00159590 msgr2=0x7fcf0015d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:45.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.254+0000 7fcf08747700 1 --2- 172.21.15.46:0/257639961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf00159590 0x7fcf0015d3b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0098e0 tx=0x7fcefc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.254+0000 7fcf08747700 1 -- 172.21.15.46:0/257639961 shutdown_connections 2024-01-26T23:42:45.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.254+0000 7fcf08747700 1 --2- 172.21.15.46:0/257639961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf00159590 0x7fcf0015d3b0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.254+0000 7fcf08747700 1 -- 172.21.15.46:0/257639961 >> 172.21.15.46:0/257639961 conn(0x7fcf000ce530 msgr2=0x7fcf000ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:45.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.254+0000 7fcf08747700 1 -- 172.21.15.46:0/257639961 shutdown_connections 2024-01-26T23:42:45.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.254+0000 7fcf08747700 1 -- 172.21.15.46:0/257639961 wait complete. 2024-01-26T23:42:45.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.255+0000 7fcf08747700 1 Processor -- start 2024-01-26T23:42:45.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.255+0000 7fcf08747700 1 -- start start 2024-01-26T23:42:45.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.256+0000 7fcf08747700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcf0006c3c0 0x7fcf0015cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.256+0000 7fcf08747700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf00159590 0x7fcf00157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.256+0000 7fcf08747700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0015d8f0 0x7fcf0015a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.256+0000 7fcf08747700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0015cb10 con 0x7fcf0015d8f0 2024-01-26T23:42:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.256+0000 7fcf08747700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0015d4e0 con 0x7fcf00159590 2024-01-26T23:42:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.256+0000 7fcf08747700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf000c6370 con 0x7fcf0006c3c0 2024-01-26T23:42:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.256+0000 7fcf06ce4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0015d8f0 0x7fcf0015a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.256+0000 7fcf064e3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcf0006c3c0 0x7fcf0015cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.256+0000 7fcf06ce4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0015d8f0 0x7fcf0015a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39050/0 (socket says 172.21.15.46:39050) 2024-01-26T23:42:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.256+0000 7fcf06ce4700 1 -- 172.21.15.46:0/2498052379 learned_addr learned my addr 172.21.15.46:0/2498052379 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.256+0000 7fcf05ce2700 1 --2- 172.21.15.46:0/2498052379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf00159590 0x7fcf00157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.257+0000 7fcf06ce4700 1 -- 172.21.15.46:0/2498052379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcf0006c3c0 msgr2=0x7fcf0015cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.257+0000 7fcf06ce4700 1 --2- 172.21.15.46:0/2498052379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcf0006c3c0 0x7fcf0015cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.257+0000 7fcf06ce4700 1 -- 172.21.15.46:0/2498052379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf00159590 msgr2=0x7fcf00157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:45.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.257+0000 7fcf06ce4700 1 --2- 172.21.15.46:0/2498052379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf00159590 0x7fcf00157b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.257+0000 7fcf06ce4700 1 -- 172.21.15.46:0/2498052379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc009420 con 0x7fcf0015d8f0 2024-01-26T23:42:45.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.257+0000 7fcf06ce4700 1 --2- 172.21.15.46:0/2498052379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0015d8f0 0x7fcf0015a250 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fcef800cc90 tx=0x7fcef8001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:45.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.257+0000 7fcef67fc700 1 -- 172.21.15.46:0/2498052379 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef800a8b0 con 0x7fcf0015d8f0 2024-01-26T23:42:45.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.257+0000 7fcef67fc700 1 -- 172.21.15.46:0/2498052379 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef800aa10 con 0x7fcf0015d8f0 2024-01-26T23:42:45.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.257+0000 7fcf08747700 1 -- 172.21.15.46:0/2498052379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0015c4c0 con 0x7fcf0015d8f0 2024-01-26T23:42:45.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.257+0000 7fcef67fc700 1 -- 172.21.15.46:0/2498052379 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef800add0 con 0x7fcf0015d8f0 2024-01-26T23:42:45.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.257+0000 7fcf08747700 1 -- 172.21.15.46:0/2498052379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf00158080 con 0x7fcf0015d8f0 2024-01-26T23:42:45.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.258+0000 7fcf08747700 1 -- 172.21.15.46:0/2498052379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf0015ccc0 con 0x7fcf0015d8f0 2024-01-26T23:42:45.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.259+0000 7fcef67fc700 1 -- 172.21.15.46:0/2498052379 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7fcef8007300 con 0x7fcf0015d8f0 2024-01-26T23:42:45.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.260+0000 7fcef67fc700 1 --2- 172.21.15.46:0/2498052379 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcee40670e0 0x7fcee4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:45.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.260+0000 7fcf064e3700 1 -- 172.21.15.46:0/2498052379 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcee40670e0 msgr2=0x7fcee4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:45.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.260+0000 7fcef67fc700 1 -- 172.21.15.46:0/2498052379 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6848+0+0 (secure 0 0 0) 0x7fcef8085120 con 0x7fcf0015d8f0 2024-01-26T23:42:45.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.260+0000 7fcf064e3700 1 --2- 172.21.15.46:0/2498052379 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcee40670e0 0x7fcee4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:42:45.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.264+0000 7fcef67fc700 1 -- 172.21.15.46:0/2498052379 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcef8051c20 con 0x7fcf0015d8f0 2024-01-26T23:42:45.323 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:45.322+0000 7fc3d11d71c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:42:45.391 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:45.390+0000 7fce78a281c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:42:45.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.461+0000 7fcf064e3700 1 -- 172.21.15.46:0/2498052379 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcee40670e0 msgr2=0x7fcee4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:45.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.461+0000 7fcf064e3700 1 --2- 172.21.15.46:0/2498052379 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcee40670e0 0x7fcee4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:42:45.500 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:45.499+0000 7fc3d11d71c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:42:45.526 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:45.525+0000 7fce78a281c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:42:45.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.525+0000 7fcf08747700 1 -- 172.21.15.46:0/2498052379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf0004c710 con 0x7fcf0015d8f0 2024-01-26T23:42:45.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.526+0000 7fcef67fc700 1 -- 172.21.15.46:0/2498052379 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4597 (secure 0 0 0) 0x7fcef8055240 con 0x7fcf0015d8f0 2024-01-26T23:42:45.527 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:45.527 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1737713978","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1737713978},{"type":"v1","addr":"172.21.15.46:6835","nonce":1737713978}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":4969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3857864387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3857864387},{"type":"v1","addr":"172.21.15.106:6836","nonce":3857864387}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/639000251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":639000251},{"type":"v1","addr":"172.21.15.106:6837","nonce":639000251}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:30:52.553580+0000","modified":"2024-01-26T23:30:56.149313+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4984":{"gid":4984,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/453078862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":453078862},{"type":"v1","addr":"172.21.15.46:6834","nonce":453078862}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2024-01-26T23:42:45.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.528+0000 7fcf08747700 1 -- 172.21.15.46:0/2498052379 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcee40670e0 msgr2=0x7fcee4069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:42:45.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.528+0000 7fcf08747700 1 --2- 172.21.15.46:0/2498052379 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcee40670e0 0x7fcee4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.528+0000 7fcf08747700 1 -- 172.21.15.46:0/2498052379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0015d8f0 msgr2=0x7fcf0015a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:45.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.528+0000 7fcf08747700 1 --2- 172.21.15.46:0/2498052379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0015d8f0 0x7fcf0015a250 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fcef800cc90 tx=0x7fcef8001a80 comp rx=0 tx=0).stop 2024-01-26T23:42:45.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.528+0000 7fcf08747700 1 -- 172.21.15.46:0/2498052379 shutdown_connections 2024-01-26T23:42:45.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.528+0000 7fcf08747700 1 --2- 172.21.15.46:0/2498052379 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcee40670e0 0x7fcee4069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.528+0000 7fcf08747700 1 --2- 172.21.15.46:0/2498052379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0015d8f0 0x7fcf0015a250 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.528+0000 7fcf08747700 1 --2- 172.21.15.46:0/2498052379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcf0006c3c0 0x7fcf0015cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.528+0000 7fcf08747700 1 --2- 172.21.15.46:0/2498052379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf00159590 0x7fcf00157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.528+0000 7fcf08747700 1 -- 172.21.15.46:0/2498052379 >> 172.21.15.46:0/2498052379 conn(0x7fcf000ce530 msgr2=0x7fcf000cd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:45.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.528+0000 7fcf08747700 1 -- 172.21.15.46:0/2498052379 shutdown_connections 2024-01-26T23:42:45.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.528+0000 7fcf08747700 1 -- 172.21.15.46:0/2498052379 wait complete. 2024-01-26T23:42:45.531 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 18 2024-01-26T23:42:45.544 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:42:45.591 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:45.590+0000 7fce78a281c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:42:45.670 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:45.669+0000 7fc3d11d71c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:42:45.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.685+0000 7fda88417700 1 Processor -- start 2024-01-26T23:42:45.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.686+0000 7fda88417700 1 -- start start 2024-01-26T23:42:45.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.687+0000 7fda88417700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda8015d0e0 0x7fda800d0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:45.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.687+0000 7fda88417700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fda800cace0 0x7fda800cb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:45.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.687+0000 7fda88417700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda800adb60 con 0x7fda800cb670 2024-01-26T23:42:45.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.687+0000 7fda88417700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda800cc5f0 con 0x7fda800cace0 2024-01-26T23:42:45.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.687+0000 7fda88417700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda800ade20 con 0x7fda8015d0e0 2024-01-26T23:42:45.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.687+0000 7fda861b3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda8015d0e0 0x7fda800d0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:45.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.687+0000 7fda859b2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fda800cace0 0x7fda800cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.687+0000 7fda869b4700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fda800cb670 0x7fda800ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:57206/0 (socket says 172.21.15.46:57206) 2024-01-26T23:42:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.687+0000 7fda861b3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda8015d0e0 0x7fda800d0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:34366/0 (socket says 172.21.15.46:34366) 2024-01-26T23:42:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.687+0000 7fda869b4700 1 -- 172.21.15.46:0/289200259 learned_addr learned my addr 172.21.15.46:0/289200259 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.687+0000 7fda859b2700 1 -- 172.21.15.46:0/289200259 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda8015d0e0 msgr2=0x7fda800d0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.687+0000 7fda859b2700 1 --2- 172.21.15.46:0/289200259 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda8015d0e0 0x7fda800d0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.687+0000 7fda859b2700 1 -- 172.21.15.46:0/289200259 >> v1:172.21.15.106:6789/0 conn(0x7fda800cb670 legacy=0x7fda800ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.687+0000 7fda851b1700 1 -- 172.21.15.46:0/289200259 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221408756 0 0) 0x7fda800adb60 con 0x7fda800cb670 2024-01-26T23:42:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.687+0000 7fda859b2700 1 -- 172.21.15.46:0/289200259 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda800cbd50 con 0x7fda800cace0 2024-01-26T23:42:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.687+0000 7fda861b3700 1 --2- 172.21.15.46:0/289200259 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda8015d0e0 0x7fda800d0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.688+0000 7fda859b2700 1 --2- 172.21.15.46:0/289200259 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda800cace0 0x7fda800cb130 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fda700098e0 tx=0x7fda700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d20347a762a0d4d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.688+0000 7fda851b1700 1 -- 172.21.15.46:0/289200259 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7000b640 con 0x7fda800cace0 2024-01-26T23:42:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.688+0000 7fda851b1700 1 -- 172.21.15.46:0/289200259 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7000b7a0 con 0x7fda800cace0 2024-01-26T23:42:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.688+0000 7fda851b1700 1 -- 172.21.15.46:0/289200259 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7000bb40 con 0x7fda800cace0 2024-01-26T23:42:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.688+0000 7fda88417700 1 -- 172.21.15.46:0/289200259 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda800cace0 msgr2=0x7fda800cb130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.688+0000 7fda88417700 1 --2- 172.21.15.46:0/289200259 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda800cace0 0x7fda800cb130 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fda700098e0 tx=0x7fda700091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.689+0000 7fda88417700 1 -- 172.21.15.46:0/289200259 shutdown_connections 2024-01-26T23:42:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.689+0000 7fda88417700 1 --2- 172.21.15.46:0/289200259 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda8015d0e0 0x7fda800d0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.689+0000 7fda88417700 1 --2- 172.21.15.46:0/289200259 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda800cace0 0x7fda800cb130 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.689+0000 7fda88417700 1 -- 172.21.15.46:0/289200259 >> 172.21.15.46:0/289200259 conn(0x7fda8006f840 msgr2=0x7fda800d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.689+0000 7fda88417700 1 -- 172.21.15.46:0/289200259 shutdown_connections 2024-01-26T23:42:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.689+0000 7fda88417700 1 -- 172.21.15.46:0/289200259 wait complete. 2024-01-26T23:42:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.690+0000 7fda88417700 1 Processor -- start 2024-01-26T23:42:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.690+0000 7fda88417700 1 -- start start 2024-01-26T23:42:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda88417700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda800cace0 0x7fda800c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda88417700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda800cb670 0x7fda800b3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda88417700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda8015d0e0 0x7fda800b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda88417700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda800b4c90 con 0x7fda8015d0e0 2024-01-26T23:42:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda88417700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda80159100 con 0x7fda800cb670 2024-01-26T23:42:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda88417700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda800b4080 con 0x7fda800cace0 2024-01-26T23:42:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda869b4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda8015d0e0 0x7fda800b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda861b3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda800cace0 0x7fda800c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda859b2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda800cb670 0x7fda800b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda869b4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda8015d0e0 0x7fda800b3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39064/0 (socket says 172.21.15.46:39064) 2024-01-26T23:42:45.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda861b3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda800cace0 0x7fda800c9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:34368/0 (socket says 172.21.15.46:34368) 2024-01-26T23:42:45.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda869b4700 1 -- 172.21.15.46:0/2940702720 learned_addr learned my addr 172.21.15.46:0/2940702720 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:45.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda861b3700 1 -- 172.21.15.46:0/2940702720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda800cb670 msgr2=0x7fda800b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:45.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda861b3700 1 --2- 172.21.15.46:0/2940702720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda800cb670 0x7fda800b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda861b3700 1 -- 172.21.15.46:0/2940702720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda8015d0e0 msgr2=0x7fda800b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:45.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda861b3700 1 --2- 172.21.15.46:0/2940702720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda8015d0e0 0x7fda800b3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda861b3700 1 -- 172.21.15.46:0/2940702720 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7c0093c0 con 0x7fda800cace0 2024-01-26T23:42:45.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda869b4700 1 --2- 172.21.15.46:0/2940702720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda8015d0e0 0x7fda800b3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:45.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda861b3700 1 --2- 172.21.15.46:0/2940702720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda800cace0 0x7fda800c9250 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00ea80 tx=0x7fda7c009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:45.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.691+0000 7fda767fc700 1 -- 172.21.15.46:0/2940702720 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c00eec0 con 0x7fda800cace0 2024-01-26T23:42:45.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.692+0000 7fda767fc700 1 -- 172.21.15.46:0/2940702720 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7c005a00 con 0x7fda800cace0 2024-01-26T23:42:45.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.692+0000 7fda767fc700 1 -- 172.21.15.46:0/2940702720 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c005da0 con 0x7fda800cace0 2024-01-26T23:42:45.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.692+0000 7fda88417700 1 -- 172.21.15.46:0/2940702720 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda70009420 con 0x7fda800cace0 2024-01-26T23:42:45.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.692+0000 7fda88417700 1 -- 172.21.15.46:0/2940702720 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda800b4640 con 0x7fda800cace0 2024-01-26T23:42:45.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.693+0000 7fda88417700 1 -- 172.21.15.46:0/2940702720 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8004c710 con 0x7fda800cace0 2024-01-26T23:42:45.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.693+0000 7fda767fc700 1 -- 172.21.15.46:0/2940702720 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7fda7c00ca00 con 0x7fda800cace0 2024-01-26T23:42:45.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.694+0000 7fda767fc700 1 --2- 172.21.15.46:0/2940702720 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fda640670e0 0x7fda64069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:45.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.694+0000 7fda767fc700 1 -- 172.21.15.46:0/2940702720 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6720+0+0 (secure 0 0 0) 0x7fda7c084fd0 con 0x7fda800cace0 2024-01-26T23:42:45.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.694+0000 7fda859b2700 1 -- 172.21.15.46:0/2940702720 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fda640670e0 msgr2=0x7fda64069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:45.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.694+0000 7fda859b2700 1 --2- 172.21.15.46:0/2940702720 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fda640670e0 0x7fda64069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:42:45.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.698+0000 7fda767fc700 1 -- 172.21.15.46:0/2940702720 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fda7c051bd0 con 0x7fda800cace0 2024-01-26T23:42:45.718 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:45.717+0000 7fce78a281c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:42:45.749 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:45.748+0000 7fc3d11d71c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:42:45.790 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:45.789+0000 7fce78a281c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:42:45.891 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:45.889+0000 7fc3d11d71c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:42:45.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.895+0000 7fda859b2700 1 -- 172.21.15.46:0/2940702720 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fda640670e0 msgr2=0x7fda64069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:45.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.895+0000 7fda859b2700 1 --2- 172.21.15.46:0/2940702720 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fda640670e0 0x7fda64069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:42:45.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.904+0000 7fda88417700 1 -- 172.21.15.46:0/2940702720 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda800cc030 con 0x7fda800cace0 2024-01-26T23:42:45.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.905+0000 7fda767fc700 1 -- 172.21.15.46:0/2940702720 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16620 (secure 0 0 0) 0x7fda7c0551f0 con 0x7fda800cace0 2024-01-26T23:42:45.907 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:45.907 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":57,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:42:45.619220+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:30:50.831232+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-01-26T23:30:51.525734+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.36","mappings":[{"from":5,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/3967117552":"2024-01-27T23:31:07.345815+0000","172.21.15.106:0/2294180308":"2024-01-27T23:31:07.345815+0000","172.21.15.106:0/3492442819":"2024-01-27T23:31:07.345815+0000","172.21.15.106:6833/43388":"2024-01-27T23:31:07.345815+0000","172.21.15.106:0/927283051":"2024-01-27T23:31:07.345815+0000","172.21.15.106:6832/43388":"2024-01-27T23:31:07.345815+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":2},{"begin":8,"length":3},{"begin":14,"length":2},{"begin":19,"length":3},{"begin":24,"length":2},{"begin":29,"length":4},{"begin":35,"length":2},{"begin":40,"length":3},{"begin":45,"length":2},{"begin":50,"length":4},{"begin":56,"length":2},{"begin":61,"length":4},{"begin":67,"length":2},{"begin":73,"length":3},{"begin":78,"length":2},{"begin":83,"length":3},{"begin":88,"length":2},{"begin":94,"length":3},{"begin":99,"length":2},{"begin":104,"length":3},{"begin":109,"length":3},{"begin":166,"length":56}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":104,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:42:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.907+0000 7fda88417700 1 -- 172.21.15.46:0/2940702720 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fda640670e0 msgr2=0x7fda64069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:42:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.907+0000 7fda88417700 1 --2- 172.21.15.46:0/2940702720 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fda640670e0 0x7fda64069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.907+0000 7fda88417700 1 -- 172.21.15.46:0/2940702720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda800cace0 msgr2=0x7fda800c9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.907+0000 7fda88417700 1 --2- 172.21.15.46:0/2940702720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda800cace0 0x7fda800c9250 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00ea80 tx=0x7fda7c009120 comp rx=0 tx=0).stop 2024-01-26T23:42:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.908+0000 7fda88417700 1 -- 172.21.15.46:0/2940702720 shutdown_connections 2024-01-26T23:42:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.908+0000 7fda88417700 1 --2- 172.21.15.46:0/2940702720 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fda640670e0 0x7fda64069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.908+0000 7fda88417700 1 --2- 172.21.15.46:0/2940702720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda8015d0e0 0x7fda800b3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.908+0000 7fda88417700 1 --2- 172.21.15.46:0/2940702720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda800cace0 0x7fda800c9250 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.908+0000 7fda88417700 1 --2- 172.21.15.46:0/2940702720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda800cb670 0x7fda800b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.908+0000 7fda88417700 1 -- 172.21.15.46:0/2940702720 >> 172.21.15.46:0/2940702720 conn(0x7fda8006f840 msgr2=0x7fda80159ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.908+0000 7fda88417700 1 -- 172.21.15.46:0/2940702720 shutdown_connections 2024-01-26T23:42:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:45.908+0000 7fda88417700 1 -- 172.21.15.46:0/2940702720 wait complete. 2024-01-26T23:42:45.923 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-26T23:30:50.831232+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-01-26T23:30:51.525734+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 221, 'snap_epoch': 57, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-26T23:42:45.923 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:42:45.930 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:45.929+0000 7fce78a281c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:42:45.977 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:45.976+0000 7fc3d11d71c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:42:46.054 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:46.053+0000 7fce78a281c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:42:46.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.058+0000 7f5ef9211700 1 Processor -- start 2024-01-26T23:42:46.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.058+0000 7f5ef9211700 1 -- start start 2024-01-26T23:42:46.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.059+0000 7f5ef9211700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5ef406ebe0 0x7f5ef4067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:46.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.059+0000 7f5ef9211700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ef41597e0 0x7f5ef4159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:46.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.059+0000 7f5ef9211700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef406fdb0 con 0x7f5ef4067af0 2024-01-26T23:42:46.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.059+0000 7f5ef9211700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef406f1a0 con 0x7f5ef406ebe0 2024-01-26T23:42:46.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.059+0000 7f5ef9211700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef406f480 con 0x7f5ef41597e0 2024-01-26T23:42:46.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.059+0000 7f5ef259c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5ef406ebe0 0x7f5ef4067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:46.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.059+0000 7f5ef2d9d700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f5ef4067af0 0x7f5ef40ad8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:57212/0 (socket says 172.21.15.46:57212) 2024-01-26T23:42:46.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.059+0000 7f5ef359e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ef41597e0 0x7f5ef4159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:46.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.059+0000 7f5ef2d9d700 1 -- 172.21.15.46:0/3682487752 learned_addr learned my addr 172.21.15.46:0/3682487752 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:46.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.059+0000 7f5ef259c700 1 -- 172.21.15.46:0/3682487752 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ef41597e0 msgr2=0x7f5ef4159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:46.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.059+0000 7f5ef259c700 1 --2- 172.21.15.46:0/3682487752 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ef41597e0 0x7f5ef4159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.059+0000 7f5ef259c700 1 -- 172.21.15.46:0/3682487752 >> v1:172.21.15.106:6789/0 conn(0x7f5ef4067af0 legacy=0x7f5ef40ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:46.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.059+0000 7f5ef259c700 1 -- 172.21.15.46:0/3682487752 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef406f760 con 0x7f5ef406ebe0 2024-01-26T23:42:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.059+0000 7f5ef1d9b700 1 -- 172.21.15.46:0/3682487752 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253613829 0 0) 0x7f5ef406fdb0 con 0x7f5ef4067af0 2024-01-26T23:42:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.060+0000 7f5ef259c700 1 --2- 172.21.15.46:0/3682487752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ef406ebe0 0x7f5ef4067ec0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5ee80098e0 tx=0x7f5ee80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee12dc9a785a9a80 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.060+0000 7f5ef1d9b700 1 -- 172.21.15.46:0/3682487752 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee800b640 con 0x7f5ef406ebe0 2024-01-26T23:42:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.060+0000 7f5ef1d9b700 1 -- 172.21.15.46:0/3682487752 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee800b7a0 con 0x7f5ef406ebe0 2024-01-26T23:42:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.060+0000 7f5ef1d9b700 1 -- 172.21.15.46:0/3682487752 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee800bb40 con 0x7f5ef406ebe0 2024-01-26T23:42:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.060+0000 7f5ef9211700 1 -- 172.21.15.46:0/3682487752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ef406ebe0 msgr2=0x7f5ef4067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.060+0000 7f5ef9211700 1 --2- 172.21.15.46:0/3682487752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ef406ebe0 0x7f5ef4067ec0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5ee80098e0 tx=0x7f5ee80091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.060+0000 7f5ef9211700 1 -- 172.21.15.46:0/3682487752 shutdown_connections 2024-01-26T23:42:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.060+0000 7f5ef9211700 1 --2- 172.21.15.46:0/3682487752 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ef41597e0 0x7f5ef4159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.060+0000 7f5ef9211700 1 --2- 172.21.15.46:0/3682487752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ef406ebe0 0x7f5ef4067ec0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.060+0000 7f5ef9211700 1 -- 172.21.15.46:0/3682487752 >> 172.21.15.46:0/3682487752 conn(0x7f5ef40cdf80 msgr2=0x7f5ef406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.060+0000 7f5ef9211700 1 -- 172.21.15.46:0/3682487752 shutdown_connections 2024-01-26T23:42:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.060+0000 7f5ef9211700 1 -- 172.21.15.46:0/3682487752 wait complete. 2024-01-26T23:42:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.062+0000 7f5ef9211700 1 Processor -- start 2024-01-26T23:42:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.062+0000 7f5ef9211700 1 -- start start 2024-01-26T23:42:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef9211700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ef4067af0 0x7f5ef416a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef9211700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ef406ebe0 0x7f5ef40a9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef9211700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ef41597e0 0x7f5ef40a9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef9211700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef416ab30 con 0x7f5ef406ebe0 2024-01-26T23:42:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef9211700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef40abe70 con 0x7f5ef41597e0 2024-01-26T23:42:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef9211700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef40aa620 con 0x7f5ef4067af0 2024-01-26T23:42:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ef406ebe0 0x7f5ef40a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef2d9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ef4067af0 0x7f5ef416a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ef406ebe0 0x7f5ef40a9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39070/0 (socket says 172.21.15.46:39070) 2024-01-26T23:42:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef259c700 1 -- 172.21.15.46:0/658852991 learned_addr learned my addr 172.21.15.46:0/658852991 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef2d9d700 1 --2- 172.21.15.46:0/658852991 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ef4067af0 0x7f5ef416a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:34378/0 (socket says 172.21.15.46:34378) 2024-01-26T23:42:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef359e700 1 --2- 172.21.15.46:0/658852991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ef41597e0 0x7f5ef40a9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef2d9d700 1 -- 172.21.15.46:0/658852991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ef41597e0 msgr2=0x7f5ef40a9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef2d9d700 1 --2- 172.21.15.46:0/658852991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ef41597e0 0x7f5ef40a9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef2d9d700 1 -- 172.21.15.46:0/658852991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ef406ebe0 msgr2=0x7f5ef40a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef2d9d700 1 --2- 172.21.15.46:0/658852991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ef406ebe0 0x7f5ef40a9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef2d9d700 1 -- 172.21.15.46:0/658852991 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee8009420 con 0x7f5ef4067af0 2024-01-26T23:42:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef259c700 1 --2- 172.21.15.46:0/658852991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ef406ebe0 0x7f5ef40a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.063+0000 7f5ef2d9d700 1 --2- 172.21.15.46:0/658852991 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ef4067af0 0x7f5ef416a5f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5ee400c4f0 tx=0x7f5ee4007f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.064+0000 7f5edaffd700 1 -- 172.21.15.46:0/658852991 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee400cea0 con 0x7f5ef4067af0 2024-01-26T23:42:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.064+0000 7f5edaffd700 1 -- 172.21.15.46:0/658852991 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee4005b30 con 0x7f5ef4067af0 2024-01-26T23:42:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.064+0000 7f5ef9211700 1 -- 172.21.15.46:0/658852991 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef40ac300 con 0x7f5ef4067af0 2024-01-26T23:42:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.064+0000 7f5edaffd700 1 -- 172.21.15.46:0/658852991 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee40070d0 con 0x7f5ef4067af0 2024-01-26T23:42:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.064+0000 7f5ef9211700 1 -- 172.21.15.46:0/658852991 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef40ab540 con 0x7f5ef4067af0 2024-01-26T23:42:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.065+0000 7f5ef9211700 1 -- 172.21.15.46:0/658852991 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef404c710 con 0x7f5ef4067af0 2024-01-26T23:42:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.065+0000 7f5edaffd700 1 -- 172.21.15.46:0/658852991 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f5ee4007480 con 0x7f5ef4067af0 2024-01-26T23:42:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.065+0000 7f5edaffd700 1 --2- 172.21.15.46:0/658852991 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5ee00670e0 0x7f5ee0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.066+0000 7f5ef259c700 1 -- 172.21.15.46:0/658852991 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5ee00670e0 msgr2=0x7f5ee0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.066+0000 7f5edaffd700 1 -- 172.21.15.46:0/658852991 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6720+0+0 (secure 0 0 0) 0x7f5ee4083f50 con 0x7f5ef4067af0 2024-01-26T23:42:46.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.066+0000 7f5ef259c700 1 --2- 172.21.15.46:0/658852991 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5ee00670e0 0x7f5ee0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:42:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.070+0000 7f5edaffd700 1 -- 172.21.15.46:0/658852991 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5ee4050b50 con 0x7f5ef4067af0 2024-01-26T23:42:46.155 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:46.154+0000 7fc3d11d71c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:42:46.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.267+0000 7f5ef259c700 1 -- 172.21.15.46:0/658852991 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5ee00670e0 msgr2=0x7f5ee0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:46.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.267+0000 7f5ef259c700 1 --2- 172.21.15.46:0/658852991 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5ee00670e0 0x7f5ee0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:42:46.297 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:46.296+0000 7fc3d11d71c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:42:46.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.323+0000 7f5ef9211700 1 -- 172.21.15.46:0/658852991 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5ef4068690 con 0x7f5ef4067af0 2024-01-26T23:42:46.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.324+0000 7f5edaffd700 1 -- 172.21.15.46:0/658852991 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5ee40170a0 con 0x7f5ef4067af0 2024-01-26T23:42:46.325 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:42:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.326+0000 7f5ef9211700 1 -- 172.21.15.46:0/658852991 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5ee00670e0 msgr2=0x7f5ee0069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:42:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.326+0000 7f5ef9211700 1 --2- 172.21.15.46:0/658852991 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5ee00670e0 0x7f5ee0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.326+0000 7f5ef9211700 1 -- 172.21.15.46:0/658852991 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ef4067af0 msgr2=0x7f5ef416a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.326+0000 7f5ef9211700 1 --2- 172.21.15.46:0/658852991 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ef4067af0 0x7f5ef416a5f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5ee400c4f0 tx=0x7f5ee4007f50 comp rx=0 tx=0).stop 2024-01-26T23:42:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.326+0000 7f5ef9211700 1 -- 172.21.15.46:0/658852991 shutdown_connections 2024-01-26T23:42:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.326+0000 7f5ef9211700 1 --2- 172.21.15.46:0/658852991 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5ee00670e0 0x7f5ee0069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.326+0000 7f5ef9211700 1 --2- 172.21.15.46:0/658852991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ef41597e0 0x7f5ef40a9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.326+0000 7f5ef9211700 1 --2- 172.21.15.46:0/658852991 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ef4067af0 0x7f5ef416a5f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.326+0000 7f5ef9211700 1 --2- 172.21.15.46:0/658852991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ef406ebe0 0x7f5ef40a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.326+0000 7f5ef9211700 1 -- 172.21.15.46:0/658852991 >> 172.21.15.46:0/658852991 conn(0x7f5ef40cdf80 msgr2=0x7f5ef40b0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.326+0000 7f5ef9211700 1 -- 172.21.15.46:0/658852991 shutdown_connections 2024-01-26T23:42:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.326+0000 7f5ef9211700 1 -- 172.21.15.46:0/658852991 wait complete. 2024-01-26T23:42:46.342 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-26T23:42:46.372 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:46.371+0000 7fce78a281c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:42:46.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.477+0000 7f89e5a88700 1 Processor -- start 2024-01-26T23:42:46.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.477+0000 7f89e5a88700 1 -- start start 2024-01-26T23:42:46.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.479+0000 7f89e5a88700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f89e006b7f0 0x7f89e006a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:46.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.479+0000 7f89e5a88700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e006db60 con 0x7f89e006a570 2024-01-26T23:42:46.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.479+0000 7f89e5a88700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e00a9e60 con 0x7f89e00a9980 2024-01-26T23:42:46.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.479+0000 7f89e5a88700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e00aa140 con 0x7f89e006b7f0 2024-01-26T23:42:46.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.479+0000 7f89df7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f89e006a570 0x7f89e006e2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42108/0 (socket says 172.21.15.46:42108) 2024-01-26T23:42:46.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.479+0000 7f89df7fe700 1 -- 172.21.15.46:0/55953452 learned_addr learned my addr 172.21.15.46:0/55953452 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:46.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.479+0000 7f89deffd700 1 --2- 172.21.15.46:0/55953452 >> v2:172.21.15.106:3300/0 conn(0x7f89e006b7f0 0x7f89e006a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:46.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.479+0000 7f89de7fc700 1 -- 172.21.15.46:0/55953452 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599489487 0 0) 0x7f89e006db60 con 0x7f89e006a570 2024-01-26T23:42:46.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.479+0000 7f89de7fc700 1 -- 172.21.15.46:0/55953452 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89c8003570 con 0x7f89e006a570 2024-01-26T23:42:46.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.479+0000 7f89deffd700 1 -- 172.21.15.46:0/55953452 >> v1:172.21.15.106:6789/0 conn(0x7f89e00a9980 legacy=0x7f89e00aba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:46.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.479+0000 7f89deffd700 1 -- 172.21.15.46:0/55953452 >> v1:172.21.15.46:6789/0 conn(0x7f89e006a570 legacy=0x7f89e006e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:46.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.479+0000 7f89de7fc700 1 -- 172.21.15.46:0/55953452 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697846069 0 0) 0x7f89e00a9e60 con 0x7f89e00a9980 2024-01-26T23:42:46.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.479+0000 7f89deffd700 1 -- 172.21.15.46:0/55953452 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e00aa420 con 0x7f89e006b7f0 2024-01-26T23:42:46.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.480+0000 7f89deffd700 1 --2- 172.21.15.46:0/55953452 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89e006b7f0 0x7f89e006a940 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f89d400aa90 tx=0x7f89d400a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=544213e02b46bbc6 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:46.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.480+0000 7f89de7fc700 1 -- 172.21.15.46:0/55953452 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d400c970 con 0x7f89e006b7f0 2024-01-26T23:42:46.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.480+0000 7f89de7fc700 1 -- 172.21.15.46:0/55953452 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d400cad0 con 0x7f89e006b7f0 2024-01-26T23:42:46.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.480+0000 7f89de7fc700 1 -- 172.21.15.46:0/55953452 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d400ce70 con 0x7f89e006b7f0 2024-01-26T23:42:46.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.481+0000 7f89e5a88700 1 -- 172.21.15.46:0/55953452 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89e006b7f0 msgr2=0x7f89e006a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:46.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.481+0000 7f89e5a88700 1 --2- 172.21.15.46:0/55953452 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89e006b7f0 0x7f89e006a940 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f89d400aa90 tx=0x7f89d400a330 comp rx=0 tx=0).stop 2024-01-26T23:42:46.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.481+0000 7f89e5a88700 1 -- 172.21.15.46:0/55953452 shutdown_connections 2024-01-26T23:42:46.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.481+0000 7f89e5a88700 1 --2- 172.21.15.46:0/55953452 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89e006b7f0 0x7f89e006a940 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.481+0000 7f89e5a88700 1 -- 172.21.15.46:0/55953452 >> 172.21.15.46:0/55953452 conn(0x7f89e0067930 msgr2=0x7f89e0067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:46.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.481+0000 7f89e5a88700 1 -- 172.21.15.46:0/55953452 shutdown_connections 2024-01-26T23:42:46.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.481+0000 7f89e5a88700 1 -- 172.21.15.46:0/55953452 wait complete. 2024-01-26T23:42:46.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.482+0000 7f89e5a88700 1 Processor -- start 2024-01-26T23:42:46.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.482+0000 7f89e5a88700 1 -- start start 2024-01-26T23:42:46.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.483+0000 7f89e5a88700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e006a570 0x7f89e00c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:46.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.483+0000 7f89e5a88700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89e006b7f0 0x7f89e00cb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:46.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.483+0000 7f89e5a88700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e00a9980 0x7f89e00ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:46.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.483+0000 7f89e5a88700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e00ca2b0 con 0x7f89e006a570 2024-01-26T23:42:46.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.483+0000 7f89e5a88700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e00cb0d0 con 0x7f89e006b7f0 2024-01-26T23:42:46.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.483+0000 7f89e5a88700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e00cbaa0 con 0x7f89e00a9980 2024-01-26T23:42:46.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.483+0000 7f89df7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e006a570 0x7f89e00c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:46.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.483+0000 7f89df7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e006a570 0x7f89e00c9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39086/0 (socket says 172.21.15.46:39086) 2024-01-26T23:42:46.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.483+0000 7f89df7fe700 1 -- 172.21.15.46:0/3420409726 learned_addr learned my addr 172.21.15.46:0/3420409726 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.483+0000 7f89deffd700 1 --2- 172.21.15.46:0/3420409726 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89e006b7f0 0x7f89e00cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.483+0000 7f89dffff700 1 --2- 172.21.15.46:0/3420409726 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e00a9980 0x7f89e00ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.484+0000 7f89df7fe700 1 -- 172.21.15.46:0/3420409726 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e00a9980 msgr2=0x7f89e00ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.484+0000 7f89df7fe700 1 --2- 172.21.15.46:0/3420409726 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e00a9980 0x7f89e00ace80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.484+0000 7f89df7fe700 1 -- 172.21.15.46:0/3420409726 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89e006b7f0 msgr2=0x7f89e00cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.484+0000 7f89df7fe700 1 --2- 172.21.15.46:0/3420409726 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89e006b7f0 0x7f89e00cb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.484+0000 7f89df7fe700 1 -- 172.21.15.46:0/3420409726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d400a5d0 con 0x7f89e006a570 2024-01-26T23:42:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.484+0000 7f89df7fe700 1 --2- 172.21.15.46:0/3420409726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e006a570 0x7f89e00c9d70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f89d000cb10 tx=0x7f89d00027e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.484+0000 7f89c77fe700 1 -- 172.21.15.46:0/3420409726 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d000ce20 con 0x7f89e006a570 2024-01-26T23:42:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.484+0000 7f89c77fe700 1 -- 172.21.15.46:0/3420409726 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d0005d80 con 0x7f89e006a570 2024-01-26T23:42:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.484+0000 7f89c77fe700 1 -- 172.21.15.46:0/3420409726 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d000a820 con 0x7f89e006a570 2024-01-26T23:42:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.484+0000 7f89e5a88700 1 -- 172.21.15.46:0/3420409726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e00ad3c0 con 0x7f89e006a570 2024-01-26T23:42:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.484+0000 7f89e5a88700 1 -- 172.21.15.46:0/3420409726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e00caa80 con 0x7f89e006a570 2024-01-26T23:42:46.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.485+0000 7f89e5a88700 1 -- 172.21.15.46:0/3420409726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e006bee0 con 0x7f89e006a570 2024-01-26T23:42:46.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.486+0000 7f89c77fe700 1 -- 172.21.15.46:0/3420409726 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f89d000ac10 con 0x7f89e006a570 2024-01-26T23:42:46.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.486+0000 7f89c77fe700 1 --2- 172.21.15.46:0/3420409726 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f89cc0670e0 0x7f89cc069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:46.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.486+0000 7f89deffd700 1 -- 172.21.15.46:0/3420409726 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f89cc0670e0 msgr2=0x7f89cc069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:46.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.486+0000 7f89c77fe700 1 -- 172.21.15.46:0/3420409726 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6720+0+0 (secure 0 0 0) 0x7f89d0084940 con 0x7f89e006a570 2024-01-26T23:42:46.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.486+0000 7f89deffd700 1 --2- 172.21.15.46:0/3420409726 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f89cc0670e0 0x7f89cc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:42:46.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.491+0000 7f89c77fe700 1 -- 172.21.15.46:0/3420409726 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f89d00515c0 con 0x7f89e006a570 2024-01-26T23:42:46.582 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:46.581+0000 7fce78a281c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:42:46.675 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:46.674+0000 7fc3d11d71c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:42:46.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.687+0000 7f89deffd700 1 -- 172.21.15.46:0/3420409726 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f89cc0670e0 msgr2=0x7f89cc069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:46.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.687+0000 7f89deffd700 1 --2- 172.21.15.46:0/3420409726 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f89cc0670e0 0x7f89cc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:42:46.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.700+0000 7f89e5a88700 1 -- 172.21.15.46:0/3420409726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f89e004c710 con 0x7f89e006a570 2024-01-26T23:42:46.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.700+0000 7f89c77fe700 1 -- 172.21.15.46:0/3420409726 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7f89d0054be0 con 0x7f89e006a570 2024-01-26T23:42:46.702 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:42:46.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.702+0000 7f89e5a88700 1 -- 172.21.15.46:0/3420409726 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f89cc0670e0 msgr2=0x7f89cc069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:42:46.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.702+0000 7f89e5a88700 1 --2- 172.21.15.46:0/3420409726 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f89cc0670e0 0x7f89cc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.702+0000 7f89e5a88700 1 -- 172.21.15.46:0/3420409726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e006a570 msgr2=0x7f89e00c9d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:46.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.702+0000 7f89e5a88700 1 --2- 172.21.15.46:0/3420409726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e006a570 0x7f89e00c9d70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f89d000cb10 tx=0x7f89d00027e0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.703+0000 7f89e5a88700 1 -- 172.21.15.46:0/3420409726 shutdown_connections 2024-01-26T23:42:46.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.703+0000 7f89e5a88700 1 --2- 172.21.15.46:0/3420409726 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f89cc0670e0 0x7f89cc069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.703+0000 7f89e5a88700 1 --2- 172.21.15.46:0/3420409726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e006a570 0x7f89e00c9d70 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.703+0000 7f89e5a88700 1 --2- 172.21.15.46:0/3420409726 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e00a9980 0x7f89e00ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.703+0000 7f89e5a88700 1 --2- 172.21.15.46:0/3420409726 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89e006b7f0 0x7f89e00cb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.703+0000 7f89e5a88700 1 -- 172.21.15.46:0/3420409726 >> 172.21.15.46:0/3420409726 conn(0x7f89e0067930 msgr2=0x7f89e006ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:46.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.703+0000 7f89e5a88700 1 -- 172.21.15.46:0/3420409726 shutdown_connections 2024-01-26T23:42:46.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.703+0000 7f89e5a88700 1 -- 172.21.15.46:0/3420409726 wait complete. 2024-01-26T23:42:46.718 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-26T23:42:46.811 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:46.810+0000 7fce78a281c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:42:46.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.852+0000 7f1d4dc43700 1 Processor -- start 2024-01-26T23:42:46.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.852+0000 7f1d4dc43700 1 -- start start 2024-01-26T23:42:46.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.853+0000 7f1d4dc43700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d4815dfd0 0x7f1d480ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:46.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.853+0000 7f1d4dc43700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d4815a200 con 0x7f1d481599d0 2024-01-26T23:42:46.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.853+0000 7f1d4dc43700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d480b1a60 con 0x7f1d48159da0 2024-01-26T23:42:46.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.853+0000 7f1d4dc43700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d480b11c0 con 0x7f1d4815dfd0 2024-01-26T23:42:46.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.853+0000 7f1d47fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d4815dfd0 0x7f1d480ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:46.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.853+0000 7f1d477fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1d481599d0 0x7f1d480b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42114/0 (socket says 172.21.15.46:42114) 2024-01-26T23:42:46.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.853+0000 7f1d47fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d4815dfd0 0x7f1d480ab1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39090/0 (socket says 172.21.15.46:39090) 2024-01-26T23:42:46.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.853+0000 7f1d477fe700 1 -- 172.21.15.46:0/1992001359 learned_addr learned my addr 172.21.15.46:0/1992001359 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:46.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.854+0000 7f1d47fff700 1 -- 172.21.15.46:0/1992001359 >> v1:172.21.15.106:6789/0 conn(0x7f1d48159da0 legacy=0x7f1d4815d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:46.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.854+0000 7f1d47fff700 1 -- 172.21.15.46:0/1992001359 >> v1:172.21.15.46:6789/0 conn(0x7f1d481599d0 legacy=0x7f1d480b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:46.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.854+0000 7f1d467fc700 1 -- 172.21.15.46:0/1992001359 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630061254 0 0) 0x7f1d4815a200 con 0x7f1d481599d0 2024-01-26T23:42:46.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.854+0000 7f1d47fff700 1 -- 172.21.15.46:0/1992001359 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d480b14a0 con 0x7f1d4815dfd0 2024-01-26T23:42:46.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.854+0000 7f1d47fff700 1 --2- 172.21.15.46:0/1992001359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d4815dfd0 0x7f1d480ab1c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009e60 tx=0x7f1d40009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16105ed3c1423d91 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:46.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.854+0000 7f1d467fc700 1 -- 172.21.15.46:0/1992001359 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4000e070 con 0x7f1d4815dfd0 2024-01-26T23:42:46.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.854+0000 7f1d467fc700 1 -- 172.21.15.46:0/1992001359 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d4000bc30 con 0x7f1d4815dfd0 2024-01-26T23:42:46.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.854+0000 7f1d467fc700 1 -- 172.21.15.46:0/1992001359 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d40012430 con 0x7f1d4815dfd0 2024-01-26T23:42:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.855+0000 7f1d4dc43700 1 -- 172.21.15.46:0/1992001359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d4815dfd0 msgr2=0x7f1d480ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.855+0000 7f1d4dc43700 1 --2- 172.21.15.46:0/1992001359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d4815dfd0 0x7f1d480ab1c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009e60 tx=0x7f1d40009780 comp rx=0 tx=0).stop 2024-01-26T23:42:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.855+0000 7f1d4dc43700 1 -- 172.21.15.46:0/1992001359 shutdown_connections 2024-01-26T23:42:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.855+0000 7f1d4dc43700 1 --2- 172.21.15.46:0/1992001359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d4815dfd0 0x7f1d480ab1c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.855+0000 7f1d4dc43700 1 -- 172.21.15.46:0/1992001359 >> 172.21.15.46:0/1992001359 conn(0x7f1d4806f840 msgr2=0x7f1d48087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.855+0000 7f1d4dc43700 1 -- 172.21.15.46:0/1992001359 shutdown_connections 2024-01-26T23:42:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.855+0000 7f1d4dc43700 1 -- 172.21.15.46:0/1992001359 wait complete. 2024-01-26T23:42:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.856+0000 7f1d4dc43700 1 Processor -- start 2024-01-26T23:42:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.857+0000 7f1d4dc43700 1 -- start start 2024-01-26T23:42:46.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.857+0000 7f1d4dc43700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d481599d0 0x7f1d48158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.857+0000 7f1d4dc43700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d48159da0 0x7f1d480ae110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.857+0000 7f1d4dc43700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d4815dfd0 0x7f1d480ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.857+0000 7f1d4dc43700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4816a6c0 con 0x7f1d481599d0 2024-01-26T23:42:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.857+0000 7f1d4dc43700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d48169d90 con 0x7f1d48159da0 2024-01-26T23:42:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.857+0000 7f1d4dc43700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4816a070 con 0x7f1d4815dfd0 2024-01-26T23:42:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.857+0000 7f1d477fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d481599d0 0x7f1d48158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.857+0000 7f1d477fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d481599d0 0x7f1d48158820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39104/0 (socket says 172.21.15.46:39104) 2024-01-26T23:42:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.857+0000 7f1d477fe700 1 -- 172.21.15.46:0/2938789518 learned_addr learned my addr 172.21.15.46:0/2938789518 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.857+0000 7f1d46ffd700 1 --2- 172.21.15.46:0/2938789518 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d48159da0 0x7f1d480ae110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.858+0000 7f1d47fff700 1 --2- 172.21.15.46:0/2938789518 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d4815dfd0 0x7f1d480ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.858+0000 7f1d477fe700 1 -- 172.21.15.46:0/2938789518 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d4815dfd0 msgr2=0x7f1d480ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.858+0000 7f1d477fe700 1 --2- 172.21.15.46:0/2938789518 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d4815dfd0 0x7f1d480ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.858+0000 7f1d477fe700 1 -- 172.21.15.46:0/2938789518 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d48159da0 msgr2=0x7f1d480ae110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.858+0000 7f1d477fe700 1 --2- 172.21.15.46:0/2938789518 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d48159da0 0x7f1d480ae110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.858+0000 7f1d477fe700 1 -- 172.21.15.46:0/2938789518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d40009a20 con 0x7f1d481599d0 2024-01-26T23:42:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.858+0000 7f1d477fe700 1 --2- 172.21.15.46:0/2938789518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d481599d0 0x7f1d48158820 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800c790 tx=0x7f1d38007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.858+0000 7f1d2f7fe700 1 -- 172.21.15.46:0/2938789518 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d380072e0 con 0x7f1d481599d0 2024-01-26T23:42:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.858+0000 7f1d2f7fe700 1 -- 172.21.15.46:0/2938789518 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d38007440 con 0x7f1d481599d0 2024-01-26T23:42:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.858+0000 7f1d4dc43700 1 -- 172.21.15.46:0/2938789518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d4816a350 con 0x7f1d481599d0 2024-01-26T23:42:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.858+0000 7f1d2f7fe700 1 -- 172.21.15.46:0/2938789518 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d380057f0 con 0x7f1d481599d0 2024-01-26T23:42:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.858+0000 7f1d4dc43700 1 -- 172.21.15.46:0/2938789518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d48158d60 con 0x7f1d481599d0 2024-01-26T23:42:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.859+0000 7f1d4dc43700 1 -- 172.21.15.46:0/2938789518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d4804c710 con 0x7f1d481599d0 2024-01-26T23:42:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.860+0000 7f1d2f7fe700 1 -- 172.21.15.46:0/2938789518 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f1d38005c00 con 0x7f1d481599d0 2024-01-26T23:42:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.860+0000 7f1d2f7fe700 1 --2- 172.21.15.46:0/2938789518 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1d340670e0 0x7f1d34069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.861+0000 7f1d46ffd700 1 -- 172.21.15.46:0/2938789518 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1d340670e0 msgr2=0x7f1d34069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.861+0000 7f1d2f7fe700 1 -- 172.21.15.46:0/2938789518 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7040+0+0 (secure 0 0 0) 0x7f1d38001e40 con 0x7f1d481599d0 2024-01-26T23:42:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.861+0000 7f1d46ffd700 1 --2- 172.21.15.46:0/2938789518 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1d340670e0 0x7f1d34069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:42:46.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:46.865+0000 7f1d2f7fe700 1 -- 172.21.15.46:0/2938789518 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1d3804b5a0 con 0x7f1d481599d0 2024-01-26T23:42:46.881 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:46.880+0000 7fce78a281c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:42:46.922 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:46.921+0000 7fc3d11d71c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:42:47.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.062+0000 7f1d46ffd700 1 -- 172.21.15.46:0/2938789518 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1d340670e0 msgr2=0x7f1d34069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:47.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.062+0000 7f1d46ffd700 1 --2- 172.21.15.46:0/2938789518 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1d340670e0 0x7f1d34069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:42:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.073+0000 7f1d4dc43700 1 -- 172.21.15.46:0/2938789518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1d480b1780 con 0x7f1d481599d0 2024-01-26T23:42:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.073+0000 7f1d2f7fe700 1 -- 172.21.15.46:0/2938789518 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1d38050d20 con 0x7f1d481599d0 2024-01-26T23:42:47.074 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-26T23:42:47.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.076+0000 7f1d4dc43700 1 -- 172.21.15.46:0/2938789518 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1d340670e0 msgr2=0x7f1d34069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:42:47.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.076+0000 7f1d4dc43700 1 --2- 172.21.15.46:0/2938789518 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1d340670e0 0x7f1d34069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.076+0000 7f1d4dc43700 1 -- 172.21.15.46:0/2938789518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d481599d0 msgr2=0x7f1d48158820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.076+0000 7f1d4dc43700 1 --2- 172.21.15.46:0/2938789518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d481599d0 0x7f1d48158820 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800c790 tx=0x7f1d38007c80 comp rx=0 tx=0).stop 2024-01-26T23:42:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.076+0000 7f1d4dc43700 1 -- 172.21.15.46:0/2938789518 shutdown_connections 2024-01-26T23:42:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.076+0000 7f1d4dc43700 1 --2- 172.21.15.46:0/2938789518 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1d340670e0 0x7f1d34069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.076+0000 7f1d4dc43700 1 --2- 172.21.15.46:0/2938789518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d481599d0 0x7f1d48158820 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.076+0000 7f1d4dc43700 1 --2- 172.21.15.46:0/2938789518 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d4815dfd0 0x7f1d480ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.076+0000 7f1d4dc43700 1 --2- 172.21.15.46:0/2938789518 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d48159da0 0x7f1d480ae110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.076+0000 7f1d4dc43700 1 -- 172.21.15.46:0/2938789518 >> 172.21.15.46:0/2938789518 conn(0x7f1d4806f840 msgr2=0x7f1d480cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.076+0000 7f1d4dc43700 1 -- 172.21.15.46:0/2938789518 shutdown_connections 2024-01-26T23:42:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.076+0000 7f1d4dc43700 1 -- 172.21.15.46:0/2938789518 wait complete. 2024-01-26T23:42:47.092 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:42:47.194 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:47.193+0000 7fc3d11d71c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:42:47.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.232+0000 7fd3d6952700 1 Processor -- start 2024-01-26T23:42:47.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.232+0000 7fd3d6952700 1 -- start start 2024-01-26T23:42:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.233+0000 7fd3d6952700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3d0157340 0x7fd3d015bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.233+0000 7fd3d6952700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d00ae520 0x7fd3d00b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.233+0000 7fd3d6952700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d0158ae0 con 0x7fd3d00b3fb0 2024-01-26T23:42:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.233+0000 7fd3d6952700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3d00b4430 con 0x7fd3d0157340 2024-01-26T23:42:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.233+0000 7fd3d6952700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d015b650 con 0x7fd3d00ae520 2024-01-26T23:42:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.234+0000 7fd3cffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3d0157340 0x7fd3d015bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.234+0000 7fd3cffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3d0157340 0x7fd3d015bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39120/0 (socket says 172.21.15.46:39120) 2024-01-26T23:42:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.234+0000 7fd3d4eef700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd3d00b3fb0 0x7fd3d0157710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42126/0 (socket says 172.21.15.46:42126) 2024-01-26T23:42:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.234+0000 7fd3cffff700 1 -- 172.21.15.46:0/3919334980 learned_addr learned my addr 172.21.15.46:0/3919334980 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.234+0000 7fd3cf7fe700 1 --2- 172.21.15.46:0/3919334980 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d00ae520 0x7fd3d00b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.234+0000 7fd3cffff700 1 -- 172.21.15.46:0/3919334980 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d00ae520 msgr2=0x7fd3d00b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.234+0000 7fd3cffff700 1 --2- 172.21.15.46:0/3919334980 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d00ae520 0x7fd3d00b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.234+0000 7fd3cffff700 1 -- 172.21.15.46:0/3919334980 >> v1:172.21.15.46:6789/0 conn(0x7fd3d00b3fb0 legacy=0x7fd3d0157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.234+0000 7fd3ceffd700 1 -- 172.21.15.46:0/3919334980 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300494055 0 0) 0x7fd3d0158ae0 con 0x7fd3d00b3fb0 2024-01-26T23:42:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.234+0000 7fd3cffff700 1 -- 172.21.15.46:0/3919334980 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d015c020 con 0x7fd3d0157340 2024-01-26T23:42:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.234+0000 7fd3cffff700 1 --2- 172.21.15.46:0/3919334980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0157340 0x7fd3d015bae0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd3c00099b0 tx=0x7fd3c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b53d892ee10833c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.234+0000 7fd3ceffd700 1 -- 172.21.15.46:0/3919334980 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c0009ea0 con 0x7fd3d0157340 2024-01-26T23:42:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.235+0000 7fd3ceffd700 1 -- 172.21.15.46:0/3919334980 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c000b860 con 0x7fd3d0157340 2024-01-26T23:42:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.235+0000 7fd3ceffd700 1 -- 172.21.15.46:0/3919334980 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c000bc20 con 0x7fd3d0157340 2024-01-26T23:42:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.235+0000 7fd3d6952700 1 -- 172.21.15.46:0/3919334980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0157340 msgr2=0x7fd3d015bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.235+0000 7fd3d6952700 1 --2- 172.21.15.46:0/3919334980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0157340 0x7fd3d015bae0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd3c00099b0 tx=0x7fd3c0009250 comp rx=0 tx=0).stop 2024-01-26T23:42:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.235+0000 7fd3d6952700 1 -- 172.21.15.46:0/3919334980 shutdown_connections 2024-01-26T23:42:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.235+0000 7fd3d6952700 1 --2- 172.21.15.46:0/3919334980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0157340 0x7fd3d015bae0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.235+0000 7fd3d6952700 1 --2- 172.21.15.46:0/3919334980 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d00ae520 0x7fd3d00b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.235+0000 7fd3d6952700 1 -- 172.21.15.46:0/3919334980 >> 172.21.15.46:0/3919334980 conn(0x7fd3d00ce530 msgr2=0x7fd3d00ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.235+0000 7fd3d6952700 1 -- 172.21.15.46:0/3919334980 shutdown_connections 2024-01-26T23:42:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.235+0000 7fd3d6952700 1 -- 172.21.15.46:0/3919334980 wait complete. 2024-01-26T23:42:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.236+0000 7fd3d6952700 1 Processor -- start 2024-01-26T23:42:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.237+0000 7fd3d6952700 1 -- start start 2024-01-26T23:42:47.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.237+0000 7fd3d6952700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d00ae520 0x7fd3d0156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:47.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.237+0000 7fd3d6952700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3d00b3fb0 0x7fd3d015d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:47.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3d6952700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0157340 0x7fd3d015d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:47.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3d6952700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d00c8920 con 0x7fd3d0157340 2024-01-26T23:42:47.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3d6952700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d006a5f0 con 0x7fd3d00b3fb0 2024-01-26T23:42:47.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3d6952700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d00b1510 con 0x7fd3d00ae520 2024-01-26T23:42:47.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3d4eef700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0157340 0x7fd3d015d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:47.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3d4eef700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0157340 0x7fd3d015d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39134/0 (socket says 172.21.15.46:39134) 2024-01-26T23:42:47.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3d4eef700 1 -- 172.21.15.46:0/137281879 learned_addr learned my addr 172.21.15.46:0/137281879 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:47.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3cffff700 1 --2- 172.21.15.46:0/137281879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d00ae520 0x7fd3d0156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:47.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3cf7fe700 1 --2- 172.21.15.46:0/137281879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3d00b3fb0 0x7fd3d015d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:47.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3d4eef700 1 -- 172.21.15.46:0/137281879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d00ae520 msgr2=0x7fd3d0156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:47.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3d4eef700 1 --2- 172.21.15.46:0/137281879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d00ae520 0x7fd3d0156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3d4eef700 1 -- 172.21.15.46:0/137281879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3d00b3fb0 msgr2=0x7fd3d015d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:47.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3d4eef700 1 --2- 172.21.15.46:0/137281879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3d00b3fb0 0x7fd3d015d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3d4eef700 1 -- 172.21.15.46:0/137281879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c00094f0 con 0x7fd3d0157340 2024-01-26T23:42:47.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3cffff700 1 --2- 172.21.15.46:0/137281879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d00ae520 0x7fd3d0156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:47.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3d4eef700 1 --2- 172.21.15.46:0/137281879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0157340 0x7fd3d015d720 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd3c800cd50 tx=0x7fd3c8001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:47.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3b7fff700 1 -- 172.21.15.46:0/137281879 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8005df0 con 0x7fd3d0157340 2024-01-26T23:42:47.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3b7fff700 1 -- 172.21.15.46:0/137281879 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c800a850 con 0x7fd3d0157340 2024-01-26T23:42:47.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3d6952700 1 -- 172.21.15.46:0/137281879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d00c8080 con 0x7fd3d0157340 2024-01-26T23:42:47.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.238+0000 7fd3b7fff700 1 -- 172.21.15.46:0/137281879 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c800ac10 con 0x7fd3d0157340 2024-01-26T23:42:47.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.239+0000 7fd3d6952700 1 -- 172.21.15.46:0/137281879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d00c8640 con 0x7fd3d0157340 2024-01-26T23:42:47.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.240+0000 7fd3d6952700 1 -- 172.21.15.46:0/137281879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d00b12e0 con 0x7fd3d0157340 2024-01-26T23:42:47.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.240+0000 7fd3b7fff700 1 -- 172.21.15.46:0/137281879 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7fd3c80070f0 con 0x7fd3d0157340 2024-01-26T23:42:47.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.241+0000 7fd3b7fff700 1 --2- 172.21.15.46:0/137281879 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd3bc0670e0 0x7fd3bc069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:47.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.241+0000 7fd3cffff700 1 -- 172.21.15.46:0/137281879 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd3bc0670e0 msgr2=0x7fd3bc069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:47.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.241+0000 7fd3b7fff700 1 -- 172.21.15.46:0/137281879 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7040+0+0 (secure 0 0 0) 0x7fd3c8084d70 con 0x7fd3d0157340 2024-01-26T23:42:47.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.241+0000 7fd3cffff700 1 --2- 172.21.15.46:0/137281879 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd3bc0670e0 0x7fd3bc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:42:47.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.246+0000 7fd3b7fff700 1 -- 172.21.15.46:0/137281879 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd3d00b12e0 con 0x7fd3d0157340 2024-01-26T23:42:47.278 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:47.277+0000 7fc3d11d71c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:42:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.442+0000 7fd3cffff700 1 -- 172.21.15.46:0/137281879 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd3bc0670e0 msgr2=0x7fd3bc069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.442+0000 7fd3cffff700 1 --2- 172.21.15.46:0/137281879 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd3bc0670e0 0x7fd3bc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:42:47.479 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:47.478+0000 7fce78a281c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:42:47.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.498+0000 7fd3d6952700 1 -- 172.21.15.46:0/137281879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3d00b12e0 con 0x7fd3d0157340 2024-01-26T23:42:47.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.498+0000 7fd3b7fff700 1 -- 172.21.15.46:0/137281879 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4597 (secure 0 0 0) 0x7fd3d00b12e0 con 0x7fd3d0157340 2024-01-26T23:42:47.499 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:47.500 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1737713978","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1737713978},{"type":"v1","addr":"172.21.15.46:6835","nonce":1737713978}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":4969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3857864387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3857864387},{"type":"v1","addr":"172.21.15.106:6836","nonce":3857864387}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/639000251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":639000251},{"type":"v1","addr":"172.21.15.106:6837","nonce":639000251}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:30:52.553580+0000","modified":"2024-01-26T23:30:56.149313+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4984":{"gid":4984,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/453078862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":453078862},{"type":"v1","addr":"172.21.15.46:6834","nonce":453078862}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2024-01-26T23:42:47.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.500+0000 7fd3d6952700 1 -- 172.21.15.46:0/137281879 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd3bc0670e0 msgr2=0x7fd3bc069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:42:47.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.501+0000 7fd3d6952700 1 --2- 172.21.15.46:0/137281879 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd3bc0670e0 0x7fd3bc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.501+0000 7fd3d6952700 1 -- 172.21.15.46:0/137281879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0157340 msgr2=0x7fd3d015d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:47.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.501+0000 7fd3d6952700 1 --2- 172.21.15.46:0/137281879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0157340 0x7fd3d015d720 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd3c800cd50 tx=0x7fd3c8001b10 comp rx=0 tx=0).stop 2024-01-26T23:42:47.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.501+0000 7fd3d6952700 1 -- 172.21.15.46:0/137281879 shutdown_connections 2024-01-26T23:42:47.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.501+0000 7fd3d6952700 1 --2- 172.21.15.46:0/137281879 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd3bc0670e0 0x7fd3bc069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.501+0000 7fd3d6952700 1 --2- 172.21.15.46:0/137281879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d0157340 0x7fd3d015d720 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.501+0000 7fd3d6952700 1 --2- 172.21.15.46:0/137281879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3d00ae520 0x7fd3d0156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.501+0000 7fd3d6952700 1 --2- 172.21.15.46:0/137281879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3d00b3fb0 0x7fd3d015d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.501+0000 7fd3d6952700 1 -- 172.21.15.46:0/137281879 >> 172.21.15.46:0/137281879 conn(0x7fd3d00ce530 msgr2=0x7fd3d0157710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:47.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.501+0000 7fd3d6952700 1 -- 172.21.15.46:0/137281879 shutdown_connections 2024-01-26T23:42:47.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.501+0000 7fd3d6952700 1 -- 172.21.15.46:0/137281879 wait complete. 2024-01-26T23:42:47.503 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 18 2024-01-26T23:42:47.517 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:42:47.554 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:47.553+0000 7fce78a281c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:42:47.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.651+0000 7f59685c4700 1 Processor -- start 2024-01-26T23:42:47.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.651+0000 7f59685c4700 1 -- start start 2024-01-26T23:42:47.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.653+0000 7f59685c4700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f596015ce40 0x7f59600d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:47.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.653+0000 7f59685c4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f596015d210 0x7f59600cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:47.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.653+0000 7f59685c4700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59600cff70 con 0x7f59600d1390 2024-01-26T23:42:47.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.653+0000 7f59685c4700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59600cf400 con 0x7f596015d210 2024-01-26T23:42:47.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.653+0000 7f59685c4700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59600cf660 con 0x7f596015ce40 2024-01-26T23:42:47.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.653+0000 7f5966360700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f596015ce40 0x7f59600d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:47.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.653+0000 7f5965b5f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f596015d210 0x7f59600cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:47.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.653+0000 7f5966b61700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f59600d1390 0x7f59600ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:57224/0 (socket says 172.21.15.46:57224) 2024-01-26T23:42:47.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.653+0000 7f5965b5f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f596015d210 0x7f59600cbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39148/0 (socket says 172.21.15.46:39148) 2024-01-26T23:42:47.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.653+0000 7f5966360700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f596015ce40 0x7f59600d0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:52694/0 (socket says 172.21.15.46:52694) 2024-01-26T23:42:47.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.653+0000 7f5966b61700 1 -- 172.21.15.46:0/3897082012 learned_addr learned my addr 172.21.15.46:0/3897082012 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:47.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.653+0000 7f5966360700 1 -- 172.21.15.46:0/3897082012 >> v2:172.21.15.46:3300/0 conn(0x7f596015d210 msgr2=0x7f59600cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:47.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.653+0000 7f5966360700 1 --2- 172.21.15.46:0/3897082012 >> v2:172.21.15.46:3300/0 conn(0x7f596015d210 0x7f59600cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.653+0000 7f5966360700 1 -- 172.21.15.46:0/3897082012 >> v1:172.21.15.106:6789/0 conn(0x7f59600d1390 legacy=0x7f59600ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:47.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.653+0000 7f5966360700 1 -- 172.21.15.46:0/3897082012 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59600d0500 con 0x7f596015ce40 2024-01-26T23:42:47.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.653+0000 7f596535e700 1 -- 172.21.15.46:0/3897082012 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821253188 0 0) 0x7f59600cff70 con 0x7f59600d1390 2024-01-26T23:42:47.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.654+0000 7f5965b5f700 1 --2- 172.21.15.46:0/3897082012 >> v2:172.21.15.46:3300/0 conn(0x7f596015d210 0x7f59600cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:47.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.654+0000 7f5966360700 1 --2- 172.21.15.46:0/3897082012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f596015ce40 0x7f59600d0e50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5960088e90 tx=0x7f595c009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2219871feb86d3e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:47.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.654+0000 7f596535e700 1 -- 172.21.15.46:0/3897082012 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595c005c10 con 0x7f596015ce40 2024-01-26T23:42:47.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.654+0000 7f596535e700 1 -- 172.21.15.46:0/3897082012 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595c005d70 con 0x7f596015ce40 2024-01-26T23:42:47.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.654+0000 7f596535e700 1 -- 172.21.15.46:0/3897082012 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595c006260 con 0x7f596015ce40 2024-01-26T23:42:47.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.655+0000 7f59685c4700 1 -- 172.21.15.46:0/3897082012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f596015ce40 msgr2=0x7f59600d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:47.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.655+0000 7f59685c4700 1 --2- 172.21.15.46:0/3897082012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f596015ce40 0x7f59600d0e50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5960088e90 tx=0x7f595c009120 comp rx=0 tx=0).stop 2024-01-26T23:42:47.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.655+0000 7f59685c4700 1 -- 172.21.15.46:0/3897082012 shutdown_connections 2024-01-26T23:42:47.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.655+0000 7f59685c4700 1 --2- 172.21.15.46:0/3897082012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f596015ce40 0x7f59600d0e50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.655+0000 7f59685c4700 1 --2- 172.21.15.46:0/3897082012 >> v2:172.21.15.46:3300/0 conn(0x7f596015d210 0x7f59600cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.655+0000 7f59685c4700 1 -- 172.21.15.46:0/3897082012 >> 172.21.15.46:0/3897082012 conn(0x7f596006f840 msgr2=0x7f59600d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:47.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.655+0000 7f59685c4700 1 -- 172.21.15.46:0/3897082012 shutdown_connections 2024-01-26T23:42:47.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.655+0000 7f59685c4700 1 -- 172.21.15.46:0/3897082012 wait complete. 2024-01-26T23:42:47.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.656+0000 7f59685c4700 1 Processor -- start 2024-01-26T23:42:47.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.656+0000 7f59685c4700 1 -- start start 2024-01-26T23:42:47.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f59685c4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59600d1390 0x7f596015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:47.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f59685c4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f596015ce40 0x7f59600c5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:47.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f59685c4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f596015d210 0x7f596009d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:47.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f59685c4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596015c240 con 0x7f59600d1390 2024-01-26T23:42:47.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f59685c4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59600c6b60 con 0x7f596015d210 2024-01-26T23:42:47.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f59685c4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59600c65a0 con 0x7f596015ce40 2024-01-26T23:42:47.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f5966360700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59600d1390 0x7f596015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:47.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f5965b5f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f596015ce40 0x7f59600c5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:47.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f5966360700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59600d1390 0x7f596015bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39158/0 (socket says 172.21.15.46:39158) 2024-01-26T23:42:47.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f5966360700 1 -- 172.21.15.46:0/591217409 learned_addr learned my addr 172.21.15.46:0/591217409 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:47.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f5966b61700 1 --2- 172.21.15.46:0/591217409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f596015d210 0x7f596009d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:47.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f5965b5f700 1 -- 172.21.15.46:0/591217409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f596015d210 msgr2=0x7f596009d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:47.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f5965b5f700 1 --2- 172.21.15.46:0/591217409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f596015d210 0x7f596009d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f5965b5f700 1 -- 172.21.15.46:0/591217409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59600d1390 msgr2=0x7f596015bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:47.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f5965b5f700 1 --2- 172.21.15.46:0/591217409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59600d1390 0x7f596015bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f5965b5f700 1 -- 172.21.15.46:0/591217409 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5950009420 con 0x7f596015ce40 2024-01-26T23:42:47.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f5966360700 1 --2- 172.21.15.46:0/591217409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59600d1390 0x7f596015bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:47.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.657+0000 7f5966b61700 1 --2- 172.21.15.46:0/591217409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f596015d210 0x7f596009d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:47.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.658+0000 7f5965b5f700 1 --2- 172.21.15.46:0/591217409 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f596015ce40 0x7f59600c5b50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f595000e9d0 tx=0x7f59500091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:47.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.658+0000 7f59567fc700 1 -- 172.21.15.46:0/591217409 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950013680 con 0x7f596015ce40 2024-01-26T23:42:47.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.658+0000 7f59567fc700 1 -- 172.21.15.46:0/591217409 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59500137e0 con 0x7f596015ce40 2024-01-26T23:42:47.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.658+0000 7f59685c4700 1 -- 172.21.15.46:0/591217409 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f595c0093c0 con 0x7f596015ce40 2024-01-26T23:42:47.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.658+0000 7f59567fc700 1 -- 172.21.15.46:0/591217409 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950013ba0 con 0x7f596015ce40 2024-01-26T23:42:47.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.658+0000 7f59685c4700 1 -- 172.21.15.46:0/591217409 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59600c8d50 con 0x7f596015ce40 2024-01-26T23:42:47.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.659+0000 7f59685c4700 1 -- 172.21.15.46:0/591217409 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59600c8bb0 con 0x7f596015ce40 2024-01-26T23:42:47.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.659+0000 7f59567fc700 1 -- 172.21.15.46:0/591217409 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f595000c490 con 0x7f596015ce40 2024-01-26T23:42:47.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.660+0000 7f59567fc700 1 --2- 172.21.15.46:0/591217409 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f59440670e0 0x7f5944069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:47.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.660+0000 7f5966360700 1 -- 172.21.15.46:0/591217409 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f59440670e0 msgr2=0x7f5944069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:47.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.660+0000 7f59567fc700 1 -- 172.21.15.46:0/591217409 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7040+0+0 (secure 0 0 0) 0x7f5950084f60 con 0x7f596015ce40 2024-01-26T23:42:47.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.660+0000 7f5966360700 1 --2- 172.21.15.46:0/591217409 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f59440670e0 0x7f5944069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:42:47.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.664+0000 7f59567fc700 1 -- 172.21.15.46:0/591217409 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f59600c8bb0 con 0x7f596015ce40 2024-01-26T23:42:47.679 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:47.678+0000 7fce78a281c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:42:47.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.861+0000 7f5966360700 1 -- 172.21.15.46:0/591217409 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f59440670e0 msgr2=0x7f5944069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:47.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.861+0000 7f5966360700 1 --2- 172.21.15.46:0/591217409 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f59440670e0 0x7f5944069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:42:47.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.871+0000 7f59685c4700 1 -- 172.21.15.46:0/591217409 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59600c8bb0 con 0x7f596015ce40 2024-01-26T23:42:47.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.872+0000 7f59567fc700 1 -- 172.21.15.46:0/591217409 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+17101 (secure 0 0 0) 0x7f59600c8bb0 con 0x7f596015ce40 2024-01-26T23:42:47.874 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:47.874 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":58,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:42:46.593860+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:30:50.831232+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-01-26T23:30:51.525734+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.36","mappings":[{"from":5,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/3967117552":"2024-01-27T23:31:07.345815+0000","172.21.15.106:0/2294180308":"2024-01-27T23:31:07.345815+0000","172.21.15.106:0/3492442819":"2024-01-27T23:31:07.345815+0000","172.21.15.106:6833/43388":"2024-01-27T23:31:07.345815+0000","172.21.15.106:0/927283051":"2024-01-27T23:31:07.345815+0000","172.21.15.106:6832/43388":"2024-01-27T23:31:07.345815+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":9,"length":2},{"begin":15,"length":1},{"begin":20,"length":2},{"begin":25,"length":1},{"begin":30,"length":3},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":46,"length":1},{"begin":51,"length":3},{"begin":57,"length":1},{"begin":63,"length":2},{"begin":68,"length":1},{"begin":74,"length":2},{"begin":79,"length":1},{"begin":84,"length":2},{"begin":89,"length":1},{"begin":95,"length":2},{"begin":100,"length":1},{"begin":106,"length":1},{"begin":110,"length":2},{"begin":166,"length":33}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:42:47.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.876+0000 7f59685c4700 1 -- 172.21.15.46:0/591217409 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f59440670e0 msgr2=0x7f5944069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:42:47.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.876+0000 7f59685c4700 1 --2- 172.21.15.46:0/591217409 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f59440670e0 0x7f5944069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.876+0000 7f59685c4700 1 -- 172.21.15.46:0/591217409 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f596015ce40 msgr2=0x7f59600c5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.876+0000 7f59685c4700 1 --2- 172.21.15.46:0/591217409 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f596015ce40 0x7f59600c5b50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f595000e9d0 tx=0x7f59500091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.876+0000 7f59685c4700 1 -- 172.21.15.46:0/591217409 shutdown_connections 2024-01-26T23:42:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.876+0000 7f59685c4700 1 --2- 172.21.15.46:0/591217409 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f59440670e0 0x7f5944069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.876+0000 7f59685c4700 1 --2- 172.21.15.46:0/591217409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59600d1390 0x7f596015bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.876+0000 7f59685c4700 1 --2- 172.21.15.46:0/591217409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f596015d210 0x7f596009d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.876+0000 7f59685c4700 1 --2- 172.21.15.46:0/591217409 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f596015ce40 0x7f59600c5b50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.876+0000 7f59685c4700 1 -- 172.21.15.46:0/591217409 >> 172.21.15.46:0/591217409 conn(0x7f596006f840 msgr2=0x7f59600d0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.876+0000 7f59685c4700 1 -- 172.21.15.46:0/591217409 shutdown_connections 2024-01-26T23:42:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:47.876+0000 7f59685c4700 1 -- 172.21.15.46:0/591217409 wait complete. 2024-01-26T23:42:47.896 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-26T23:42:47.896 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:42:47.999 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:47.998+0000 7fc3d11d71c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:42:48.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.030+0000 7f62efd6e700 1 Processor -- start 2024-01-26T23:42:48.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.030+0000 7f62efd6e700 1 -- start start 2024-01-26T23:42:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.031+0000 7f62efd6e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e8159590 0x7f62e815d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.031+0000 7f62efd6e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f62e815d8f0 0x7f62e806bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.031+0000 7f62efd6e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e815dd90 con 0x7f62e8159960 2024-01-26T23:42:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.031+0000 7f62efd6e700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62e815e350 con 0x7f62e815d8f0 2024-01-26T23:42:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.031+0000 7f62efd6e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e80c97c0 con 0x7f62e8159590 2024-01-26T23:42:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.031+0000 7f62ed309700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f62e815d8f0 0x7f62e806bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.031+0000 7f62ed309700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f62e815d8f0 0x7f62e806bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39174/0 (socket says 172.21.15.46:39174) 2024-01-26T23:42:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.031+0000 7f62ed309700 1 -- 172.21.15.46:0/2956039940 learned_addr learned my addr 172.21.15.46:0/2956039940 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.031+0000 7f62edb0a700 1 --2- 172.21.15.46:0/2956039940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e8159590 0x7f62e815d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.031+0000 7f62ed309700 1 -- 172.21.15.46:0/2956039940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e8159590 msgr2=0x7f62e815d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.031+0000 7f62ed309700 1 --2- 172.21.15.46:0/2956039940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e8159590 0x7f62e815d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.031+0000 7f62ed309700 1 -- 172.21.15.46:0/2956039940 >> v1:172.21.15.106:6789/0 conn(0x7f62e8159960 legacy=0x7f62e806d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:48.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.031+0000 7f62ed309700 1 -- 172.21.15.46:0/2956039940 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e815e070 con 0x7f62e815d8f0 2024-01-26T23:42:48.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.031+0000 7f62ed309700 1 --2- 172.21.15.46:0/2956039940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62e815d8f0 0x7f62e806bcb0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0099e0 tx=0x7f62dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8c41538ffaa74f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:48.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.032+0000 7f62ecb08700 1 -- 172.21.15.46:0/2956039940 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc00b820 con 0x7f62e815d8f0 2024-01-26T23:42:48.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.032+0000 7f62ecb08700 1 -- 172.21.15.46:0/2956039940 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62dc00b980 con 0x7f62e815d8f0 2024-01-26T23:42:48.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.032+0000 7f62ecb08700 1 -- 172.21.15.46:0/2956039940 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc00bd40 con 0x7f62e815d8f0 2024-01-26T23:42:48.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.032+0000 7f62efd6e700 1 -- 172.21.15.46:0/2956039940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62e815d8f0 msgr2=0x7f62e806bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:48.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.032+0000 7f62efd6e700 1 --2- 172.21.15.46:0/2956039940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62e815d8f0 0x7f62e806bcb0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0099e0 tx=0x7f62dc009250 comp rx=0 tx=0).stop 2024-01-26T23:42:48.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.032+0000 7f62efd6e700 1 -- 172.21.15.46:0/2956039940 shutdown_connections 2024-01-26T23:42:48.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.032+0000 7f62efd6e700 1 --2- 172.21.15.46:0/2956039940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e8159590 0x7f62e815d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.032+0000 7f62efd6e700 1 --2- 172.21.15.46:0/2956039940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62e815d8f0 0x7f62e806bcb0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.032+0000 7f62efd6e700 1 -- 172.21.15.46:0/2956039940 >> 172.21.15.46:0/2956039940 conn(0x7f62e80ce530 msgr2=0x7f62e80ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:48.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.032+0000 7f62efd6e700 1 -- 172.21.15.46:0/2956039940 shutdown_connections 2024-01-26T23:42:48.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.032+0000 7f62efd6e700 1 -- 172.21.15.46:0/2956039940 wait complete. 2024-01-26T23:42:48.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.034+0000 7f62efd6e700 1 Processor -- start 2024-01-26T23:42:48.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.034+0000 7f62efd6e700 1 -- start start 2024-01-26T23:42:48.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.034+0000 7f62efd6e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62e8159590 0x7f62e815e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:48.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.034+0000 7f62efd6e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62e8159960 0x7f62e80b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:48.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.035+0000 7f62efd6e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e815d8f0 0x7f62e81555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:48.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.035+0000 7f62efd6e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e80c3420 con 0x7f62e8159960 2024-01-26T23:42:48.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.035+0000 7f62efd6e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e80c4850 con 0x7f62e8159590 2024-01-26T23:42:48.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.035+0000 7f62efd6e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e80c3c40 con 0x7f62e815d8f0 2024-01-26T23:42:48.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.035+0000 7f62ed309700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62e8159960 0x7f62e80b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:48.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.035+0000 7f62ed309700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62e8159960 0x7f62e80b3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39188/0 (socket says 172.21.15.46:39188) 2024-01-26T23:42:48.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.035+0000 7f62ed309700 1 -- 172.21.15.46:0/4286054037 learned_addr learned my addr 172.21.15.46:0/4286054037 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:48.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.035+0000 7f62edb0a700 1 --2- 172.21.15.46:0/4286054037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62e8159590 0x7f62e815e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:48.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.035+0000 7f62ee30b700 1 --2- 172.21.15.46:0/4286054037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e815d8f0 0x7f62e81555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:48.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.035+0000 7f62ed309700 1 -- 172.21.15.46:0/4286054037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e815d8f0 msgr2=0x7f62e81555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:48.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.035+0000 7f62ed309700 1 --2- 172.21.15.46:0/4286054037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e815d8f0 0x7f62e81555c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.035+0000 7f62ed309700 1 -- 172.21.15.46:0/4286054037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62e8159590 msgr2=0x7f62e815e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:48.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.035+0000 7f62ed309700 1 --2- 172.21.15.46:0/4286054037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62e8159590 0x7f62e815e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.035+0000 7f62ed309700 1 -- 172.21.15.46:0/4286054037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62dc0094f0 con 0x7f62e8159960 2024-01-26T23:42:48.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.035+0000 7f62ee30b700 1 --2- 172.21.15.46:0/4286054037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e815d8f0 0x7f62e81555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:48.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.035+0000 7f62ed309700 1 --2- 172.21.15.46:0/4286054037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62e8159960 0x7f62e80b3ca0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f62e806f390 tx=0x7f62dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:48.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.035+0000 7f62e5ffb700 1 -- 172.21.15.46:0/4286054037 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc01f070 con 0x7f62e8159960 2024-01-26T23:42:48.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.036+0000 7f62e5ffb700 1 -- 172.21.15.46:0/4286054037 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62dc002f30 con 0x7f62e8159960 2024-01-26T23:42:48.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.036+0000 7f62efd6e700 1 -- 172.21.15.46:0/4286054037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e80c3f20 con 0x7f62e8159960 2024-01-26T23:42:48.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.036+0000 7f62e5ffb700 1 -- 172.21.15.46:0/4286054037 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc00b820 con 0x7f62e8159960 2024-01-26T23:42:48.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.036+0000 7f62efd6e700 1 -- 172.21.15.46:0/4286054037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e80c42b0 con 0x7f62e8159960 2024-01-26T23:42:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.037+0000 7f62efd6e700 1 -- 172.21.15.46:0/4286054037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e80c49c0 con 0x7f62e8159960 2024-01-26T23:42:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.037+0000 7f62e5ffb700 1 -- 172.21.15.46:0/4286054037 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f62dc023070 con 0x7f62e8159960 2024-01-26T23:42:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.038+0000 7f62e5ffb700 1 --2- 172.21.15.46:0/4286054037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f62d40670e0 0x7f62d4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.038+0000 7f62e5ffb700 1 -- 172.21.15.46:0/4286054037 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7040+0+0 (secure 0 0 0) 0x7f62dc085f70 con 0x7f62e8159960 2024-01-26T23:42:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.038+0000 7f62edb0a700 1 -- 172.21.15.46:0/4286054037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f62d40670e0 msgr2=0x7f62d4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.038+0000 7f62edb0a700 1 --2- 172.21.15.46:0/4286054037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f62d40670e0 0x7f62d4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:42:48.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.042+0000 7f62e5ffb700 1 -- 172.21.15.46:0/4286054037 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f62dc0b75a0 con 0x7f62e8159960 2024-01-26T23:42:48.112 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:48.110+0000 7fc3d11d71c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:42:48.179 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:48.178+0000 7fce78a281c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:42:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.239+0000 7f62edb0a700 1 -- 172.21.15.46:0/4286054037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f62d40670e0 msgr2=0x7f62d4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.239+0000 7f62edb0a700 1 --2- 172.21.15.46:0/4286054037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f62d40670e0 0x7f62d4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:42:48.253 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:48.252+0000 7fc3d11d71c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:42:48.324 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:48.323+0000 7fce78a281c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:42:48.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.331+0000 7f62efd6e700 1 -- 172.21.15.46:0/4286054037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62e804c710 con 0x7f62e8159960 2024-01-26T23:42:48.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.332+0000 7f62e5ffb700 1 -- 172.21.15.46:0/4286054037 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4597 (secure 0 0 0) 0x7f62dc02a090 con 0x7f62e8159960 2024-01-26T23:42:48.333 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:48.333 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1737713978","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1737713978},{"type":"v1","addr":"172.21.15.46:6835","nonce":1737713978}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":4969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3857864387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3857864387},{"type":"v1","addr":"172.21.15.106:6836","nonce":3857864387}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/639000251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":639000251},{"type":"v1","addr":"172.21.15.106:6837","nonce":639000251}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:30:52.553580+0000","modified":"2024-01-26T23:30:56.149313+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4984":{"gid":4984,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/453078862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":453078862},{"type":"v1","addr":"172.21.15.46:6834","nonce":453078862}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2024-01-26T23:42:48.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.334+0000 7f62efd6e700 1 -- 172.21.15.46:0/4286054037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f62d40670e0 msgr2=0x7f62d4069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:42:48.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.334+0000 7f62efd6e700 1 --2- 172.21.15.46:0/4286054037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f62d40670e0 0x7f62d4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.335+0000 7f62efd6e700 1 -- 172.21.15.46:0/4286054037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62e8159960 msgr2=0x7f62e80b3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.335+0000 7f62efd6e700 1 --2- 172.21.15.46:0/4286054037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62e8159960 0x7f62e80b3ca0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f62e806f390 tx=0x7f62dc017000 comp rx=0 tx=0).stop 2024-01-26T23:42:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.335+0000 7f62efd6e700 1 -- 172.21.15.46:0/4286054037 shutdown_connections 2024-01-26T23:42:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.335+0000 7f62efd6e700 1 --2- 172.21.15.46:0/4286054037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f62d40670e0 0x7f62d4069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.335+0000 7f62efd6e700 1 --2- 172.21.15.46:0/4286054037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62e815d8f0 0x7f62e81555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.335+0000 7f62efd6e700 1 --2- 172.21.15.46:0/4286054037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62e8159590 0x7f62e815e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.335+0000 7f62efd6e700 1 --2- 172.21.15.46:0/4286054037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62e8159960 0x7f62e80b3ca0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.335+0000 7f62efd6e700 1 -- 172.21.15.46:0/4286054037 >> 172.21.15.46:0/4286054037 conn(0x7f62e80ce530 msgr2=0x7f62e80ccd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.335+0000 7f62efd6e700 1 -- 172.21.15.46:0/4286054037 shutdown_connections 2024-01-26T23:42:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.335+0000 7f62efd6e700 1 -- 172.21.15.46:0/4286054037 wait complete. 2024-01-26T23:42:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 18 2024-01-26T23:42:48.350 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:42:48.471 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:48.470+0000 7fce78a281c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:42:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.481+0000 7f6a893f0700 1 Processor -- start 2024-01-26T23:42:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.481+0000 7f6a893f0700 1 -- start start 2024-01-26T23:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.482+0000 7f6a893f0700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6a84159760 0x7f6a840b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.482+0000 7f6a893f0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6a840b3060 0x7f6a840b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.482+0000 7f6a893f0700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a8406f940 con 0x7f6a84159b30 2024-01-26T23:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.482+0000 7f6a893f0700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a8406ed30 con 0x7f6a840b3060 2024-01-26T23:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.482+0000 7f6a893f0700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a8406f010 con 0x7f6a84159760 2024-01-26T23:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.482+0000 7f6a827fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6a840b3060 0x7f6a840b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.482+0000 7f6a827fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6a840b3060 0x7f6a840b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39190/0 (socket says 172.21.15.46:39190) 2024-01-26T23:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.483+0000 7f6a827fc700 1 -- 172.21.15.46:0/1868487255 learned_addr learned my addr 172.21.15.46:0/1868487255 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.483+0000 7f6a82ffd700 1 --2- 172.21.15.46:0/1868487255 >> v2:172.21.15.106:3300/0 conn(0x7f6a84159760 0x7f6a840b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.483+0000 7f6a827fc700 1 -- 172.21.15.46:0/1868487255 >> v2:172.21.15.106:3300/0 conn(0x7f6a84159760 msgr2=0x7f6a840b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.483+0000 7f6a827fc700 1 --2- 172.21.15.46:0/1868487255 >> v2:172.21.15.106:3300/0 conn(0x7f6a84159760 0x7f6a840b2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.483+0000 7f6a827fc700 1 -- 172.21.15.46:0/1868487255 >> v1:172.21.15.106:6789/0 conn(0x7f6a84159b30 legacy=0x7f6a8415d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.483+0000 7f6a827fc700 1 -- 172.21.15.46:0/1868487255 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8406f2f0 con 0x7f6a840b3060 2024-01-26T23:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.483+0000 7f6a81ffb700 1 -- 172.21.15.46:0/1868487255 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2857804824 0 0) 0x7f6a8406f940 con 0x7f6a84159b30 2024-01-26T23:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.483+0000 7f6a827fc700 1 --2- 172.21.15.46:0/1868487255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a840b3060 0x7f6a840b0bb0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6a84076af0 tx=0x7f6a6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610c7ebef279ca5b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.483+0000 7f6a81ffb700 1 -- 172.21.15.46:0/1868487255 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c009e70 con 0x7f6a840b3060 2024-01-26T23:42:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.483+0000 7f6a81ffb700 1 -- 172.21.15.46:0/1868487255 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6c00b860 con 0x7f6a840b3060 2024-01-26T23:42:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.483+0000 7f6a81ffb700 1 -- 172.21.15.46:0/1868487255 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c00bc00 con 0x7f6a840b3060 2024-01-26T23:42:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.484+0000 7f6a893f0700 1 -- 172.21.15.46:0/1868487255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a840b3060 msgr2=0x7f6a840b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.484+0000 7f6a893f0700 1 --2- 172.21.15.46:0/1868487255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a840b3060 0x7f6a840b0bb0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6a84076af0 tx=0x7f6a6c009250 comp rx=0 tx=0).stop 2024-01-26T23:42:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.484+0000 7f6a893f0700 1 -- 172.21.15.46:0/1868487255 shutdown_connections 2024-01-26T23:42:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.484+0000 7f6a893f0700 1 --2- 172.21.15.46:0/1868487255 >> v2:172.21.15.106:3300/0 conn(0x7f6a84159760 0x7f6a840b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.484+0000 7f6a893f0700 1 --2- 172.21.15.46:0/1868487255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a840b3060 0x7f6a840b0bb0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.484+0000 7f6a893f0700 1 -- 172.21.15.46:0/1868487255 >> 172.21.15.46:0/1868487255 conn(0x7f6a84069510 msgr2=0x7f6a840aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.484+0000 7f6a893f0700 1 -- 172.21.15.46:0/1868487255 shutdown_connections 2024-01-26T23:42:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.484+0000 7f6a893f0700 1 -- 172.21.15.46:0/1868487255 wait complete. 2024-01-26T23:42:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.485+0000 7f6a893f0700 1 Processor -- start 2024-01-26T23:42:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.485+0000 7f6a893f0700 1 -- start start 2024-01-26T23:42:48.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.486+0000 7f6a893f0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a840b3060 0x7f6a8415df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:48.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.486+0000 7f6a893f0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a84159760 0x7f6a841527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:48.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.486+0000 7f6a893f0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a84159b30 0x7f6a8416a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:48.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.486+0000 7f6a893f0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a84154220 con 0x7f6a840b3060 2024-01-26T23:42:48.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.486+0000 7f6a893f0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8414e650 con 0x7f6a84159760 2024-01-26T23:42:48.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.486+0000 7f6a893f0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8413a940 con 0x7f6a84159b30 2024-01-26T23:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.486+0000 7f6a82ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a840b3060 0x7f6a8415df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.486+0000 7f6a82ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a840b3060 0x7f6a8415df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39204/0 (socket says 172.21.15.46:39204) 2024-01-26T23:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.486+0000 7f6a82ffd700 1 -- 172.21.15.46:0/2142132940 learned_addr learned my addr 172.21.15.46:0/2142132940 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.486+0000 7f6a827fc700 1 --2- 172.21.15.46:0/2142132940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a84159760 0x7f6a841527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.486+0000 7f6a837fe700 1 --2- 172.21.15.46:0/2142132940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a84159b30 0x7f6a8416a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.487+0000 7f6a82ffd700 1 -- 172.21.15.46:0/2142132940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a84159b30 msgr2=0x7f6a8416a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.487+0000 7f6a82ffd700 1 --2- 172.21.15.46:0/2142132940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a84159b30 0x7f6a8416a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.487+0000 7f6a82ffd700 1 -- 172.21.15.46:0/2142132940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a84159760 msgr2=0x7f6a841527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.487+0000 7f6a82ffd700 1 --2- 172.21.15.46:0/2142132940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a84159760 0x7f6a841527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.487+0000 7f6a82ffd700 1 -- 172.21.15.46:0/2142132940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c0094f0 con 0x7f6a840b3060 2024-01-26T23:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.487+0000 7f6a82ffd700 1 --2- 172.21.15.46:0/2142132940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a840b3060 0x7f6a8415df60 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c009f20 tx=0x7f6a7c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.487+0000 7f6a7affd700 1 -- 172.21.15.46:0/2142132940 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c00cb10 con 0x7f6a840b3060 2024-01-26T23:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.487+0000 7f6a7affd700 1 -- 172.21.15.46:0/2142132940 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7c00f040 con 0x7f6a840b3060 2024-01-26T23:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.487+0000 7f6a893f0700 1 -- 172.21.15.46:0/2142132940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a8416b640 con 0x7f6a840b3060 2024-01-26T23:42:48.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.487+0000 7f6a7affd700 1 -- 172.21.15.46:0/2142132940 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c00cea0 con 0x7f6a840b3060 2024-01-26T23:42:48.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.487+0000 7f6a893f0700 1 -- 172.21.15.46:0/2142132940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a84154410 con 0x7f6a840b3060 2024-01-26T23:42:48.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.488+0000 7f6a893f0700 1 -- 172.21.15.46:0/2142132940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a8404c710 con 0x7f6a840b3060 2024-01-26T23:42:48.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.489+0000 7f6a7affd700 1 -- 172.21.15.46:0/2142132940 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f6a7c014690 con 0x7f6a840b3060 2024-01-26T23:42:48.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.490+0000 7f6a7affd700 1 --2- 172.21.15.46:0/2142132940 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6a640670e0 0x7f6a64069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.490+0000 7f6a827fc700 1 -- 172.21.15.46:0/2142132940 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6a640670e0 msgr2=0x7f6a64069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.490+0000 7f6a7affd700 1 -- 172.21.15.46:0/2142132940 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7040+0+0 (secure 0 0 0) 0x7f6a7c085830 con 0x7f6a840b3060 2024-01-26T23:42:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.490+0000 7f6a827fc700 1 --2- 172.21.15.46:0/2142132940 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6a640670e0 0x7f6a64069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:42:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.494+0000 7f6a7affd700 1 -- 172.21.15.46:0/2142132940 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6a7c0521e0 con 0x7f6a840b3060 2024-01-26T23:42:48.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.691+0000 7f6a827fc700 1 -- 172.21.15.46:0/2142132940 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6a640670e0 msgr2=0x7f6a64069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:48.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.691+0000 7f6a827fc700 1 --2- 172.21.15.46:0/2142132940 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6a640670e0 0x7f6a64069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:42:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.749+0000 7f6a893f0700 1 -- 172.21.15.46:0/2142132940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a8406e0b0 con 0x7f6a840b3060 2024-01-26T23:42:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.750+0000 7f6a7affd700 1 -- 172.21.15.46:0/2142132940 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4597 (secure 0 0 0) 0x7f6a7c055800 con 0x7f6a840b3060 2024-01-26T23:42:48.751 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:48.751 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1737713978","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1737713978},{"type":"v1","addr":"172.21.15.46:6835","nonce":1737713978}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":4969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3857864387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3857864387},{"type":"v1","addr":"172.21.15.106:6836","nonce":3857864387}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/639000251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":639000251},{"type":"v1","addr":"172.21.15.106:6837","nonce":639000251}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:30:52.553580+0000","modified":"2024-01-26T23:30:56.149313+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4984":{"gid":4984,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/453078862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":453078862},{"type":"v1","addr":"172.21.15.46:6834","nonce":453078862}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2024-01-26T23:42:48.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.752+0000 7f6a893f0700 1 -- 172.21.15.46:0/2142132940 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6a640670e0 msgr2=0x7f6a64069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:42:48.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.752+0000 7f6a893f0700 1 --2- 172.21.15.46:0/2142132940 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6a640670e0 0x7f6a64069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.752+0000 7f6a893f0700 1 -- 172.21.15.46:0/2142132940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a840b3060 msgr2=0x7f6a8415df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:48.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.752+0000 7f6a893f0700 1 --2- 172.21.15.46:0/2142132940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a840b3060 0x7f6a8415df60 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c009f20 tx=0x7f6a7c00b040 comp rx=0 tx=0).stop 2024-01-26T23:42:48.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.752+0000 7f6a893f0700 1 -- 172.21.15.46:0/2142132940 shutdown_connections 2024-01-26T23:42:48.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.752+0000 7f6a893f0700 1 --2- 172.21.15.46:0/2142132940 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6a640670e0 0x7f6a64069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.752+0000 7f6a893f0700 1 --2- 172.21.15.46:0/2142132940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a840b3060 0x7f6a8415df60 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.752+0000 7f6a893f0700 1 --2- 172.21.15.46:0/2142132940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a84159b30 0x7f6a8416a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.752+0000 7f6a893f0700 1 --2- 172.21.15.46:0/2142132940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a84159760 0x7f6a841527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.752+0000 7f6a893f0700 1 -- 172.21.15.46:0/2142132940 >> 172.21.15.46:0/2142132940 conn(0x7f6a84069510 msgr2=0x7f6a840ad620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:48.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.752+0000 7f6a893f0700 1 -- 172.21.15.46:0/2142132940 shutdown_connections 2024-01-26T23:42:48.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.752+0000 7f6a893f0700 1 -- 172.21.15.46:0/2142132940 wait complete. 2024-01-26T23:42:48.755 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 18 2024-01-26T23:42:48.768 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:42:48.850 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:48.849+0000 7fc3d11d71c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:42:48.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.900+0000 7fac1fcba700 1 Processor -- start 2024-01-26T23:42:48.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.900+0000 7fac1fcba700 1 -- start start 2024-01-26T23:42:48.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.901+0000 7fac1fcba700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac1806e9b0 0x7fac18150af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:48.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.901+0000 7fac1fcba700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac18068d20 con 0x7fac18150720 2024-01-26T23:42:48.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.901+0000 7fac1fcba700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac18068110 con 0x7fac181575a0 2024-01-26T23:42:48.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.901+0000 7fac1fcba700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac180683f0 con 0x7fac1806e9b0 2024-01-26T23:42:48.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.901+0000 7fac1da56700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fac18150720 0x7fac18067930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42140/0 (socket says 172.21.15.46:42140) 2024-01-26T23:42:48.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.901+0000 7fac1da56700 1 -- 172.21.15.46:0/3663591446 learned_addr learned my addr 172.21.15.46:0/3663591446 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:48.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.901+0000 7fac1e257700 1 --2- 172.21.15.46:0/3663591446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac1806e9b0 0x7fac18150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:48.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.902+0000 7fac1ca54700 1 -- 172.21.15.46:0/3663591446 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731282269 0 0) 0x7fac18068d20 con 0x7fac18150720 2024-01-26T23:42:48.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.902+0000 7fac1ca54700 1 -- 172.21.15.46:0/3663591446 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac04003570 con 0x7fac18150720 2024-01-26T23:42:48.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.902+0000 7fac1ca54700 1 -- 172.21.15.46:0/3663591446 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230705683 0 0) 0x7fac18068110 con 0x7fac181575a0 2024-01-26T23:42:48.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.902+0000 7fac1ca54700 1 -- 172.21.15.46:0/3663591446 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac18068d20 con 0x7fac181575a0 2024-01-26T23:42:48.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.902+0000 7fac1e257700 1 -- 172.21.15.46:0/3663591446 >> v1:172.21.15.106:6789/0 conn(0x7fac181575a0 legacy=0x7fac1806e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:48.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.902+0000 7fac1e257700 1 -- 172.21.15.46:0/3663591446 >> v1:172.21.15.46:6789/0 conn(0x7fac18150720 legacy=0x7fac18067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:48.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.902+0000 7fac1e257700 1 -- 172.21.15.46:0/3663591446 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac180686d0 con 0x7fac1806e9b0 2024-01-26T23:42:48.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.902+0000 7fac1e257700 1 --2- 172.21.15.46:0/3663591446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac1806e9b0 0x7fac18150af0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fac10009e90 tx=0x7fac10009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e152f4ee9567087 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:48.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.902+0000 7fac1ca54700 1 -- 172.21.15.46:0/3663591446 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1000e070 con 0x7fac1806e9b0 2024-01-26T23:42:48.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.902+0000 7fac1ca54700 1 -- 172.21.15.46:0/3663591446 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1000bb80 con 0x7fac1806e9b0 2024-01-26T23:42:48.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.902+0000 7fac1ca54700 1 -- 172.21.15.46:0/3663591446 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac10012430 con 0x7fac1806e9b0 2024-01-26T23:42:48.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.903+0000 7fac1fcba700 1 -- 172.21.15.46:0/3663591446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac1806e9b0 msgr2=0x7fac18150af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:48.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.903+0000 7fac1fcba700 1 --2- 172.21.15.46:0/3663591446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac1806e9b0 0x7fac18150af0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fac10009e90 tx=0x7fac10009780 comp rx=0 tx=0).stop 2024-01-26T23:42:48.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.903+0000 7fac1fcba700 1 -- 172.21.15.46:0/3663591446 shutdown_connections 2024-01-26T23:42:48.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.903+0000 7fac1fcba700 1 --2- 172.21.15.46:0/3663591446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac1806e9b0 0x7fac18150af0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.903+0000 7fac1fcba700 1 -- 172.21.15.46:0/3663591446 >> 172.21.15.46:0/3663591446 conn(0x7fac180ccbc0 msgr2=0x7fac180ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:48.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.903+0000 7fac1fcba700 1 -- 172.21.15.46:0/3663591446 shutdown_connections 2024-01-26T23:42:48.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.903+0000 7fac1fcba700 1 -- 172.21.15.46:0/3663591446 wait complete. 2024-01-26T23:42:48.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.904+0000 7fac1fcba700 1 Processor -- start 2024-01-26T23:42:48.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.904+0000 7fac1fcba700 1 -- start start 2024-01-26T23:42:48.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.905+0000 7fac1fcba700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac1806e9b0 0x7fac18142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:48.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.905+0000 7fac1fcba700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac18150720 0x7fac1814ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:48.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.905+0000 7fac1fcba700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac181575a0 0x7fac181539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:48.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.905+0000 7fac1fcba700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac18152940 con 0x7fac181575a0 2024-01-26T23:42:48.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.905+0000 7fac1fcba700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac18152c20 con 0x7fac18150720 2024-01-26T23:42:48.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.905+0000 7fac1fcba700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac18152f00 con 0x7fac1806e9b0 2024-01-26T23:42:48.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.905+0000 7fac1e257700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac181575a0 0x7fac181539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:48.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.905+0000 7fac1da56700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac1806e9b0 0x7fac18142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:48.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.905+0000 7fac1d255700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac18150720 0x7fac1814ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.905+0000 7fac1e257700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac181575a0 0x7fac181539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39220/0 (socket says 172.21.15.46:39220) 2024-01-26T23:42:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.905+0000 7fac1e257700 1 -- 172.21.15.46:0/3509287070 learned_addr learned my addr 172.21.15.46:0/3509287070 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.906+0000 7fac1da56700 1 -- 172.21.15.46:0/3509287070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac18150720 msgr2=0x7fac1814ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.906+0000 7fac1da56700 1 --2- 172.21.15.46:0/3509287070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac18150720 0x7fac1814ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.906+0000 7fac1da56700 1 -- 172.21.15.46:0/3509287070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac181575a0 msgr2=0x7fac181539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.906+0000 7fac1da56700 1 --2- 172.21.15.46:0/3509287070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac181575a0 0x7fac181539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.906+0000 7fac1da56700 1 -- 172.21.15.46:0/3509287070 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac10009a20 con 0x7fac1806e9b0 2024-01-26T23:42:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.906+0000 7fac1da56700 1 --2- 172.21.15.46:0/3509287070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac1806e9b0 0x7fac18142a00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fac1400c580 tx=0x7fac14007d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.906+0000 7fac0dffb700 1 -- 172.21.15.46:0/3509287070 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac14005b30 con 0x7fac1806e9b0 2024-01-26T23:42:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.906+0000 7fac0dffb700 1 -- 172.21.15.46:0/3509287070 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac14005c90 con 0x7fac1806e9b0 2024-01-26T23:42:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.906+0000 7fac0dffb700 1 -- 172.21.15.46:0/3509287070 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac14007320 con 0x7fac1806e9b0 2024-01-26T23:42:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.906+0000 7fac1fcba700 1 -- 172.21.15.46:0/3509287070 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac181531e0 con 0x7fac1806e9b0 2024-01-26T23:42:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.906+0000 7fac1fcba700 1 -- 172.21.15.46:0/3509287070 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac18152560 con 0x7fac1806e9b0 2024-01-26T23:42:48.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.907+0000 7fac1fcba700 1 -- 172.21.15.46:0/3509287070 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac18153f20 con 0x7fac1806e9b0 2024-01-26T23:42:48.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.908+0000 7fac0dffb700 1 -- 172.21.15.46:0/3509287070 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7fac1401b3b0 con 0x7fac1806e9b0 2024-01-26T23:42:48.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.908+0000 7fac0dffb700 1 --2- 172.21.15.46:0/3509287070 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fabfc0670e0 0x7fabfc069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:48.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.908+0000 7fac1d255700 1 -- 172.21.15.46:0/3509287070 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fabfc0670e0 msgr2=0x7fabfc069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:48.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.908+0000 7fac0dffb700 1 -- 172.21.15.46:0/3509287070 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7040+0+0 (secure 0 0 0) 0x7fac140841f0 con 0x7fac1806e9b0 2024-01-26T23:42:48.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.908+0000 7fac1d255700 1 --2- 172.21.15.46:0/3509287070 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fabfc0670e0 0x7fabfc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:42:48.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:48.912+0000 7fac0dffb700 1 -- 172.21.15.46:0/3509287070 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fac14050cb0 con 0x7fac1806e9b0 2024-01-26T23:42:49.025 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:49.024+0000 7fc3d11d71c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:42:49.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.109+0000 7fac1d255700 1 -- 172.21.15.46:0/3509287070 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fabfc0670e0 msgr2=0x7fabfc069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:49.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.109+0000 7fac1d255700 1 --2- 172.21.15.46:0/3509287070 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fabfc0670e0 0x7fabfc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:42:49.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.120+0000 7fac1fcba700 1 -- 172.21.15.46:0/3509287070 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac18153700 con 0x7fac1806e9b0 2024-01-26T23:42:49.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.122+0000 7fac0dffb700 1 -- 172.21.15.46:0/3509287070 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+17101 (secure 0 0 0) 0x7fac140542d0 con 0x7fac1806e9b0 2024-01-26T23:42:49.123 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:49.124 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":58,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:42:46.593860+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:30:50.831232+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-01-26T23:30:51.525734+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.36","mappings":[{"from":5,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/3967117552":"2024-01-27T23:31:07.345815+0000","172.21.15.106:0/2294180308":"2024-01-27T23:31:07.345815+0000","172.21.15.106:0/3492442819":"2024-01-27T23:31:07.345815+0000","172.21.15.106:6833/43388":"2024-01-27T23:31:07.345815+0000","172.21.15.106:0/927283051":"2024-01-27T23:31:07.345815+0000","172.21.15.106:6832/43388":"2024-01-27T23:31:07.345815+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":9,"length":2},{"begin":15,"length":1},{"begin":20,"length":2},{"begin":25,"length":1},{"begin":30,"length":3},{"begin":36,"length":1},{"begin":41,"length":2},{"begin":46,"length":1},{"begin":51,"length":3},{"begin":57,"length":1},{"begin":63,"length":2},{"begin":68,"length":1},{"begin":74,"length":2},{"begin":79,"length":1},{"begin":84,"length":2},{"begin":89,"length":1},{"begin":95,"length":2},{"begin":100,"length":1},{"begin":106,"length":1},{"begin":110,"length":2},{"begin":166,"length":33}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:42:49.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.124+0000 7fac1fcba700 1 -- 172.21.15.46:0/3509287070 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fabfc0670e0 msgr2=0x7fabfc069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:42:49.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.124+0000 7fac1fcba700 1 --2- 172.21.15.46:0/3509287070 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fabfc0670e0 0x7fabfc069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.124+0000 7fac1fcba700 1 -- 172.21.15.46:0/3509287070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac1806e9b0 msgr2=0x7fac18142a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:49.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.124+0000 7fac1fcba700 1 --2- 172.21.15.46:0/3509287070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac1806e9b0 0x7fac18142a00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fac1400c580 tx=0x7fac14007d70 comp rx=0 tx=0).stop 2024-01-26T23:42:49.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.124+0000 7fac1fcba700 1 -- 172.21.15.46:0/3509287070 shutdown_connections 2024-01-26T23:42:49.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.124+0000 7fac1fcba700 1 --2- 172.21.15.46:0/3509287070 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fabfc0670e0 0x7fabfc069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.124+0000 7fac1fcba700 1 --2- 172.21.15.46:0/3509287070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac181575a0 0x7fac181539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.124+0000 7fac1fcba700 1 --2- 172.21.15.46:0/3509287070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac1806e9b0 0x7fac18142a00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.124+0000 7fac1fcba700 1 --2- 172.21.15.46:0/3509287070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac18150720 0x7fac1814ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.124+0000 7fac1fcba700 1 -- 172.21.15.46:0/3509287070 >> 172.21.15.46:0/3509287070 conn(0x7fac180ccbc0 msgr2=0x7fac180b0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:49.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.124+0000 7fac1fcba700 1 -- 172.21.15.46:0/3509287070 shutdown_connections 2024-01-26T23:42:49.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.124+0000 7fac1fcba700 1 -- 172.21.15.46:0/3509287070 wait complete. 2024-01-26T23:42:49.140 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-26T23:42:49.168 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:49.167+0000 7fce78a281c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:42:49.201 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:49.200+0000 7fc3d11d71c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:42:49.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.276+0000 7f86526e2700 1 Processor -- start 2024-01-26T23:42:49.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.276+0000 7f86526e2700 1 -- start start 2024-01-26T23:42:49.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.277+0000 7f86526e2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f864c159760 0x7f864c0b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:49.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.277+0000 7f86526e2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864c06f940 con 0x7f864c0b3060 2024-01-26T23:42:49.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.277+0000 7f86526e2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864c06ed30 con 0x7f864c0b12c0 2024-01-26T23:42:49.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.277+0000 7f86526e2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f864c06f010 con 0x7f864c159760 2024-01-26T23:42:49.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.278+0000 7f864bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f864c159760 0x7f864c0b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:49.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.278+0000 7f864bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f864c159760 0x7f864c0b2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39230/0 (socket says 172.21.15.46:39230) 2024-01-26T23:42:49.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.278+0000 7f864b7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f864c0b3060 0x7f864c0b0bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42146/0 (socket says 172.21.15.46:42146) 2024-01-26T23:42:49.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.278+0000 7f864bfff700 1 -- 172.21.15.46:0/208741842 learned_addr learned my addr 172.21.15.46:0/208741842 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:49.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.278+0000 7f864bfff700 1 -- 172.21.15.46:0/208741842 >> v1:172.21.15.106:6789/0 conn(0x7f864c0b12c0 legacy=0x7f864c0a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:49.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.278+0000 7f864bfff700 1 -- 172.21.15.46:0/208741842 >> v1:172.21.15.46:6789/0 conn(0x7f864c0b3060 legacy=0x7f864c0b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:49.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.278+0000 7f864affd700 1 -- 172.21.15.46:0/208741842 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745775673 0 0) 0x7f864c06ed30 con 0x7f864c0b12c0 2024-01-26T23:42:49.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.278+0000 7f864bfff700 1 -- 172.21.15.46:0/208741842 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c06f2f0 con 0x7f864c159760 2024-01-26T23:42:49.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.278+0000 7f864bfff700 1 --2- 172.21.15.46:0/208741842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c159760 0x7f864c0b2b20 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f863c0099b0 tx=0x7f863c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8edb3579f7bd4158 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:49.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.278+0000 7f864affd700 1 -- 172.21.15.46:0/208741842 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c00b760 con 0x7f864c159760 2024-01-26T23:42:49.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.278+0000 7f864affd700 1 -- 172.21.15.46:0/208741842 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863c00b8c0 con 0x7f864c159760 2024-01-26T23:42:49.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.278+0000 7f864affd700 1 -- 172.21.15.46:0/208741842 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c00bc80 con 0x7f864c159760 2024-01-26T23:42:49.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.279+0000 7f86526e2700 1 -- 172.21.15.46:0/208741842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c159760 msgr2=0x7f864c0b2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:49.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.279+0000 7f86526e2700 1 --2- 172.21.15.46:0/208741842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c159760 0x7f864c0b2b20 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f863c0099b0 tx=0x7f863c009250 comp rx=0 tx=0).stop 2024-01-26T23:42:49.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.279+0000 7f86526e2700 1 -- 172.21.15.46:0/208741842 shutdown_connections 2024-01-26T23:42:49.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.279+0000 7f86526e2700 1 --2- 172.21.15.46:0/208741842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c159760 0x7f864c0b2b20 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.279+0000 7f86526e2700 1 -- 172.21.15.46:0/208741842 >> 172.21.15.46:0/208741842 conn(0x7f864c069510 msgr2=0x7f864c0aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:49.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.279+0000 7f86526e2700 1 -- 172.21.15.46:0/208741842 shutdown_connections 2024-01-26T23:42:49.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.279+0000 7f86526e2700 1 -- 172.21.15.46:0/208741842 wait complete. 2024-01-26T23:42:49.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.281+0000 7f86526e2700 1 Processor -- start 2024-01-26T23:42:49.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.281+0000 7f86526e2700 1 -- start start 2024-01-26T23:42:49.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.282+0000 7f86526e2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f864c0b12c0 0x7f864c0cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:49.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.282+0000 7f86526e2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c0b3060 0x7f864c0c21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:49.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.282+0000 7f86526e2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f864c159760 0x7f864c16bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:49.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.282+0000 7f86526e2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c1439d0 con 0x7f864c0b3060 2024-01-26T23:42:49.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.282+0000 7f86526e2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c16bb20 con 0x7f864c159760 2024-01-26T23:42:49.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.282+0000 7f86526e2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864c16af10 con 0x7f864c0b12c0 2024-01-26T23:42:49.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.282+0000 7f864b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c0b3060 0x7f864c0c21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:49.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.282+0000 7f864b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c0b3060 0x7f864c0c21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39238/0 (socket says 172.21.15.46:39238) 2024-01-26T23:42:49.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.282+0000 7f864b7fe700 1 -- 172.21.15.46:0/432761354 learned_addr learned my addr 172.21.15.46:0/432761354 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:49.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.282+0000 7f8650c7f700 1 --2- 172.21.15.46:0/432761354 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f864c159760 0x7f864c16bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:49.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.282+0000 7f864bfff700 1 --2- 172.21.15.46:0/432761354 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f864c0b12c0 0x7f864c0cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:49.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.283+0000 7f864bfff700 1 -- 172.21.15.46:0/432761354 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f864c159760 msgr2=0x7f864c16bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:49.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.283+0000 7f864bfff700 1 --2- 172.21.15.46:0/432761354 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f864c159760 0x7f864c16bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.283+0000 7f864bfff700 1 -- 172.21.15.46:0/432761354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c0b3060 msgr2=0x7f864c0c21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:49.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.283+0000 7f864bfff700 1 --2- 172.21.15.46:0/432761354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c0b3060 0x7f864c0c21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.283+0000 7f864bfff700 1 -- 172.21.15.46:0/432761354 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f863c0094f0 con 0x7f864c0b12c0 2024-01-26T23:42:49.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.283+0000 7f864bfff700 1 --2- 172.21.15.46:0/432761354 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f864c0b12c0 0x7f864c0cdee0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f864c0660b0 tx=0x7f863c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:49.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.283+0000 7f8633fff700 1 -- 172.21.15.46:0/432761354 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c00ba30 con 0x7f864c0b12c0 2024-01-26T23:42:49.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.283+0000 7f8633fff700 1 -- 172.21.15.46:0/432761354 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863c00bbb0 con 0x7f864c0b12c0 2024-01-26T23:42:49.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.283+0000 7f86526e2700 1 -- 172.21.15.46:0/432761354 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864c16b1f0 con 0x7f864c0b12c0 2024-01-26T23:42:49.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.283+0000 7f8633fff700 1 -- 172.21.15.46:0/432761354 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c01c6b0 con 0x7f864c0b12c0 2024-01-26T23:42:49.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.283+0000 7f86526e2700 1 -- 172.21.15.46:0/432761354 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f864c16b7b0 con 0x7f864c0b12c0 2024-01-26T23:42:49.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.284+0000 7f86526e2700 1 -- 172.21.15.46:0/432761354 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864c16be00 con 0x7f864c0b12c0 2024-01-26T23:42:49.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.285+0000 7f8633fff700 1 -- 172.21.15.46:0/432761354 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f863c01caa0 con 0x7f864c0b12c0 2024-01-26T23:42:49.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.285+0000 7f8633fff700 1 --2- 172.21.15.46:0/432761354 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f86380670e0 0x7f8638069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:49.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.285+0000 7f864b7fe700 1 -- 172.21.15.46:0/432761354 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f86380670e0 msgr2=0x7f8638069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:49.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.285+0000 7f8633fff700 1 -- 172.21.15.46:0/432761354 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7040+0+0 (secure 0 0 0) 0x7f863c087220 con 0x7f864c0b12c0 2024-01-26T23:42:49.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.285+0000 7f864b7fe700 1 --2- 172.21.15.46:0/432761354 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f86380670e0 0x7f8638069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:42:49.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.289+0000 7f8633fff700 1 -- 172.21.15.46:0/432761354 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f863c053c60 con 0x7f864c0b12c0 2024-01-26T23:42:49.384 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:49.383+0000 7fce78a281c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:42:49.456 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:49.454+0000 7fce78a281c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:42:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.486+0000 7f864b7fe700 1 -- 172.21.15.46:0/432761354 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f86380670e0 msgr2=0x7f8638069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:42:49.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.486+0000 7f864b7fe700 1 --2- 172.21.15.46:0/432761354 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f86380670e0 0x7f8638069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:42:49.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.545+0000 7f86526e2700 1 -- 172.21.15.46:0/432761354 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f864c16be00 con 0x7f864c0b12c0 2024-01-26T23:42:49.548 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-26T23:42:49.546+0000 7f33ffe86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-26T23:42:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.551+0000 7f8633fff700 1 -- 172.21.15.46:0/432761354 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v19) v1 ==== 161+0+0 (secure 0 0 0) 0x7f863c018350 con 0x7f864c0b12c0 2024-01-26T23:42:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.554+0000 7f86526e2700 1 -- 172.21.15.46:0/432761354 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f86380670e0 msgr2=0x7f8638069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:42:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.554+0000 7f86526e2700 1 --2- 172.21.15.46:0/432761354 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f86380670e0 0x7f8638069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.554+0000 7f86526e2700 1 -- 172.21.15.46:0/432761354 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f864c0b12c0 msgr2=0x7f864c0cdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.554+0000 7f86526e2700 1 --2- 172.21.15.46:0/432761354 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f864c0b12c0 0x7f864c0cdee0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f864c0660b0 tx=0x7f863c016000 comp rx=0 tx=0).stop 2024-01-26T23:42:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.554+0000 7f86526e2700 1 -- 172.21.15.46:0/432761354 shutdown_connections 2024-01-26T23:42:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.554+0000 7f86526e2700 1 --2- 172.21.15.46:0/432761354 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f86380670e0 0x7f8638069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.554+0000 7f86526e2700 1 --2- 172.21.15.46:0/432761354 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f864c159760 0x7f864c16bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.554+0000 7f86526e2700 1 --2- 172.21.15.46:0/432761354 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f864c0b12c0 0x7f864c0cdee0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.554+0000 7f86526e2700 1 --2- 172.21.15.46:0/432761354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c0b3060 0x7f864c0c21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.554+0000 7f86526e2700 1 -- 172.21.15.46:0/432761354 >> 172.21.15.46:0/432761354 conn(0x7f864c069510 msgr2=0x7f864c0ad790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.554+0000 7f86526e2700 1 -- 172.21.15.46:0/432761354 shutdown_connections 2024-01-26T23:42:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.554+0000 7f86526e2700 1 -- 172.21.15.46:0/432761354 wait complete. 2024-01-26T23:42:49.557 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-26T23:42:49.571 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-26T23:42:49.592 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-26T23:42:49.609 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:49.609+0000 7fce78a281c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:42:49.693 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:49.692+0000 7fcdf3b0f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:42:49.694 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:49.692+0000 7fcdf3b0f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:42:49.694 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:49.692+0000 7fcdf3b0f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:42:49.694 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:49.692+0000 7fcdf3b0f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:42:49.694 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:49.692+0000 7fcdf3b0f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:42:49.694 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:49.692+0000 7fcdf6314700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:42:49.694 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:49.692+0000 7fcdf6314700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:42:49.694 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:49.692+0000 7fcdf6314700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:42:49.694 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:49.692+0000 7fcdf6314700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:42:49.694 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:42:49.692+0000 7fcdf6314700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:42:49.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.720+0000 7f53f10cb700 1 Processor -- start 2024-01-26T23:42:49.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.720+0000 7f53f10cb700 1 -- start start 2024-01-26T23:42:49.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.721+0000 7f53f10cb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53ec159510 0x7f53ec1598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:49.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.721+0000 7f53f10cb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53ec0ca220 0x7f53ec0c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:49.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.721+0000 7f53f10cb700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53ec069260 con 0x7f53ec0ca690 2024-01-26T23:42:49.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.721+0000 7f53f10cb700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53ec15d670 con 0x7f53ec159510 2024-01-26T23:42:49.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.721+0000 7f53f10cb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec0c9750 con 0x7f53ec0ca220 2024-01-26T23:42:49.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.721+0000 7f53ead9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53ec159510 0x7f53ec1598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:49.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.721+0000 7f53ead9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53ec159510 0x7f53ec1598e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39276/0 (socket says 172.21.15.46:39276) 2024-01-26T23:42:49.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.721+0000 7f53ead9d700 1 -- 172.21.15.46:0/3383895708 learned_addr learned my addr 172.21.15.46:0/3383895708 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:49.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.721+0000 7f53eb59e700 1 --1- 172.21.15.46:0/3383895708 >> v1:172.21.15.46:6789/0 conn(0x7f53ec0ca690 0x7f53ec0cc380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42178/0 (socket says 172.21.15.46:42178) 2024-01-26T23:42:49.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.721+0000 7f53ea59c700 1 --2- 172.21.15.46:0/3383895708 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53ec0ca220 0x7f53ec0c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:49.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.721+0000 7f53ead9d700 1 -- 172.21.15.46:0/3383895708 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53ec0ca220 msgr2=0x7f53ec0c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:49.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.721+0000 7f53ead9d700 1 --2- 172.21.15.46:0/3383895708 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53ec0ca220 0x7f53ec0c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.721+0000 7f53ead9d700 1 -- 172.21.15.46:0/3383895708 >> v1:172.21.15.46:6789/0 conn(0x7f53ec0ca690 legacy=0x7f53ec0cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:49.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.721+0000 7f53e9d9b700 1 -- 172.21.15.46:0/3383895708 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662331799 0 0) 0x7f53ec069260 con 0x7f53ec0ca690 2024-01-26T23:42:49.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.721+0000 7f53ead9d700 1 -- 172.21.15.46:0/3383895708 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ec0c9a30 con 0x7f53ec159510 2024-01-26T23:42:49.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.722+0000 7f53ead9d700 1 --2- 172.21.15.46:0/3383895708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec159510 0x7f53ec1598e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f53dc0099e0 tx=0x7f53dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92627d1b367f9e26 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:49.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.722+0000 7f53e9d9b700 1 -- 172.21.15.46:0/3383895708 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53dc00b820 con 0x7f53ec159510 2024-01-26T23:42:49.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.722+0000 7f53e9d9b700 1 -- 172.21.15.46:0/3383895708 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53dc00b980 con 0x7f53ec159510 2024-01-26T23:42:49.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.722+0000 7f53e9d9b700 1 -- 172.21.15.46:0/3383895708 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53dc00bd20 con 0x7f53ec159510 2024-01-26T23:42:49.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.722+0000 7f53f10cb700 1 -- 172.21.15.46:0/3383895708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec159510 msgr2=0x7f53ec1598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:49.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.722+0000 7f53f10cb700 1 --2- 172.21.15.46:0/3383895708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec159510 0x7f53ec1598e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f53dc0099e0 tx=0x7f53dc009250 comp rx=0 tx=0).stop 2024-01-26T23:42:49.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.722+0000 7f53f10cb700 1 -- 172.21.15.46:0/3383895708 shutdown_connections 2024-01-26T23:42:49.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.722+0000 7f53f10cb700 1 --2- 172.21.15.46:0/3383895708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec159510 0x7f53ec1598e0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.722+0000 7f53f10cb700 1 --2- 172.21.15.46:0/3383895708 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53ec0ca220 0x7f53ec0c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.722+0000 7f53f10cb700 1 -- 172.21.15.46:0/3383895708 >> 172.21.15.46:0/3383895708 conn(0x7f53ec06bd90 msgr2=0x7f53ec06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:49.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.722+0000 7f53f10cb700 1 -- 172.21.15.46:0/3383895708 shutdown_connections 2024-01-26T23:42:49.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.722+0000 7f53f10cb700 1 -- 172.21.15.46:0/3383895708 wait complete. 2024-01-26T23:42:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.724+0000 7f53f10cb700 1 Processor -- start 2024-01-26T23:42:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.724+0000 7f53f10cb700 1 -- start start 2024-01-26T23:42:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53f10cb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ec0ca220 0x7f53ec15bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53f10cb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec159510 0x7f53ec0cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53f10cb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53ec0c7ee0 0x7f53ec0c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53f10cb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec16aef0 con 0x7f53ec159510 2024-01-26T23:42:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53f10cb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec15c170 con 0x7f53ec0ca220 2024-01-26T23:42:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53f10cb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec15b7a0 con 0x7f53ec0c7ee0 2024-01-26T23:42:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53ea59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec159510 0x7f53ec0cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53ea59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec159510 0x7f53ec0cacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39284/0 (socket says 172.21.15.46:39284) 2024-01-26T23:42:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53ea59c700 1 -- 172.21.15.46:0/1348716209 learned_addr learned my addr 172.21.15.46:0/1348716209 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53ead9d700 1 --2- 172.21.15.46:0/1348716209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ec0ca220 0x7f53ec15bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53eb59e700 1 --2- 172.21.15.46:0/1348716209 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53ec0c7ee0 0x7f53ec0c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53ea59c700 1 -- 172.21.15.46:0/1348716209 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53ec0c7ee0 msgr2=0x7f53ec0c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53ea59c700 1 --2- 172.21.15.46:0/1348716209 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53ec0c7ee0 0x7f53ec0c8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53ea59c700 1 -- 172.21.15.46:0/1348716209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ec0ca220 msgr2=0x7f53ec15bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53ea59c700 1 --2- 172.21.15.46:0/1348716209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ec0ca220 0x7f53ec15bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53ea59c700 1 -- 172.21.15.46:0/1348716209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53dc0094f0 con 0x7f53ec159510 2024-01-26T23:42:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53ead9d700 1 --2- 172.21.15.46:0/1348716209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ec0ca220 0x7f53ec15bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53ea59c700 1 --2- 172.21.15.46:0/1348716209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec159510 0x7f53ec0cacc0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f53e000ecb0 tx=0x7f53e00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.725+0000 7f53d2ffd700 1 -- 172.21.15.46:0/1348716209 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e000c9d0 con 0x7f53ec159510 2024-01-26T23:42:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.726+0000 7f53d2ffd700 1 -- 172.21.15.46:0/1348716209 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e000cb30 con 0x7f53ec159510 2024-01-26T23:42:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.726+0000 7f53f10cb700 1 -- 172.21.15.46:0/1348716209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53ec0c88a0 con 0x7f53ec159510 2024-01-26T23:42:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.726+0000 7f53d2ffd700 1 -- 172.21.15.46:0/1348716209 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e000ced0 con 0x7f53ec159510 2024-01-26T23:42:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.726+0000 7f53f10cb700 1 -- 172.21.15.46:0/1348716209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53ec15b150 con 0x7f53ec159510 2024-01-26T23:42:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.727+0000 7f53f10cb700 1 -- 172.21.15.46:0/1348716209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53ec15b930 con 0x7f53ec159510 2024-01-26T23:42:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.727+0000 7f53d2ffd700 1 -- 172.21.15.46:0/1348716209 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 15) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f53e0013680 con 0x7f53ec159510 2024-01-26T23:42:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.728+0000 7f53d2ffd700 1 -- 172.21.15.46:0/1348716209 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f53e0084910 con 0x7f53ec159510 2024-01-26T23:42:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.739+0000 7f53d2ffd700 1 -- 172.21.15.46:0/1348716209 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f53e00519f0 con 0x7f53ec159510 2024-01-26T23:42:50.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:49.998+0000 7f53f10cb700 1 -- 172.21.15.46:0/1348716209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f53ec04c710 con 0x7f53ec159510 2024-01-26T23:42:50.038 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:50.037+0000 7fc3d11d71c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:42:50.285 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:50.284+0000 7fc3d11d71c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:42:50.368 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:50.367+0000 7fc3d11d71c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:42:50.544 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:42:50.543+0000 7fc3d11d71c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:42:50.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.551+0000 7f53d2ffd700 1 -- 172.21.15.46:0/1348716209 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v20) v1 ==== 104+0+0 (secure 0 0 0) 0x7f53e0055010 con 0x7f53ec159510 2024-01-26T23:42:50.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.553+0000 7f53f10cb700 1 -- 172.21.15.46:0/1348716209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec159510 msgr2=0x7f53ec0cacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:50.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.553+0000 7f53f10cb700 1 --2- 172.21.15.46:0/1348716209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec159510 0x7f53ec0cacc0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f53e000ecb0 tx=0x7f53e00091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:50.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.553+0000 7f53f10cb700 1 -- 172.21.15.46:0/1348716209 shutdown_connections 2024-01-26T23:42:50.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.553+0000 7f53f10cb700 1 --2- 172.21.15.46:0/1348716209 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53ec0c7ee0 0x7f53ec0c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:50.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.553+0000 7f53f10cb700 1 --2- 172.21.15.46:0/1348716209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ec0ca220 0x7f53ec15bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:50.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.553+0000 7f53f10cb700 1 --2- 172.21.15.46:0/1348716209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ec159510 0x7f53ec0cacc0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:50.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.553+0000 7f53f10cb700 1 -- 172.21.15.46:0/1348716209 >> 172.21.15.46:0/1348716209 conn(0x7f53ec06bd90 msgr2=0x7f53ec069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:50.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.553+0000 7f53f10cb700 1 -- 172.21.15.46:0/1348716209 shutdown_connections 2024-01-26T23:42:50.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.553+0000 7f53f10cb700 1 -- 172.21.15.46:0/1348716209 wait complete. 2024-01-26T23:42:50.568 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-26T23:42:50.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.748+0000 7ffad58b6700 1 Processor -- start 2024-01-26T23:42:50.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.748+0000 7ffad58b6700 1 -- start start 2024-01-26T23:42:50.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.749+0000 7ffad58b6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffad0159750 0x7ffad0069650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:50.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.749+0000 7ffad58b6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffad0159b20 0x7ffad00ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:50.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.749+0000 7ffad58b6700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad00691c0 con 0x7ffad0069b90 2024-01-26T23:42:50.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.749+0000 7ffad58b6700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad006a5e0 con 0x7ffad0159b20 2024-01-26T23:42:50.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.749+0000 7ffad58b6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad015dfd0 con 0x7ffad0159750 2024-01-26T23:42:50.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.749+0000 7fface7fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffad0159b20 0x7ffad00ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:50.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.750+0000 7ffacf7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ffad0069b90 0x7ffad015d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39828/0 (socket says 172.21.15.46:39828) 2024-01-26T23:42:50.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.750+0000 7ffacf7fe700 1 -- 172.21.15.46:0/2805891456 learned_addr learned my addr 172.21.15.46:0/2805891456 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:50.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.750+0000 7ffaceffd700 1 --2- 172.21.15.46:0/2805891456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffad0159750 0x7ffad0069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:50.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.750+0000 7fface7fc700 1 -- 172.21.15.46:0/2805891456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffad0159750 msgr2=0x7ffad0069650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:50.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.750+0000 7fface7fc700 1 --2- 172.21.15.46:0/2805891456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffad0159750 0x7ffad0069650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:50.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.750+0000 7fface7fc700 1 -- 172.21.15.46:0/2805891456 >> v1:172.21.15.46:6789/0 conn(0x7ffad0069b90 legacy=0x7ffad015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:50.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.750+0000 7fface7fc700 1 -- 172.21.15.46:0/2805891456 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad0068b70 con 0x7ffad0159b20 2024-01-26T23:42:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.750+0000 7ffacdffb700 1 -- 172.21.15.46:0/2805891456 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229295861 0 0) 0x7ffad00691c0 con 0x7ffad0069b90 2024-01-26T23:42:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.750+0000 7ffaceffd700 1 --2- 172.21.15.46:0/2805891456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffad0159750 0x7ffad0069650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.750+0000 7fface7fc700 1 --2- 172.21.15.46:0/2805891456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffad0159b20 0x7ffad00ade00 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ffab80098e0 tx=0x7ffab80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee4cc7cfac74b70d server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.750+0000 7ffacdffb700 1 -- 172.21.15.46:0/2805891456 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8009e80 con 0x7ffad0159b20 2024-01-26T23:42:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.750+0000 7ffacdffb700 1 -- 172.21.15.46:0/2805891456 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab800b860 con 0x7ffad0159b20 2024-01-26T23:42:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.750+0000 7ffacdffb700 1 -- 172.21.15.46:0/2805891456 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab800bbe0 con 0x7ffad0159b20 2024-01-26T23:42:50.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.751+0000 7ffad58b6700 1 -- 172.21.15.46:0/2805891456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffad0159b20 msgr2=0x7ffad00ade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:50.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.751+0000 7ffad58b6700 1 --2- 172.21.15.46:0/2805891456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffad0159b20 0x7ffad00ade00 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ffab80098e0 tx=0x7ffab80091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:50.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.751+0000 7ffad58b6700 1 -- 172.21.15.46:0/2805891456 shutdown_connections 2024-01-26T23:42:50.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.751+0000 7ffad58b6700 1 --2- 172.21.15.46:0/2805891456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffad0159750 0x7ffad0069650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:50.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.751+0000 7ffad58b6700 1 --2- 172.21.15.46:0/2805891456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffad0159b20 0x7ffad00ade00 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:50.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.751+0000 7ffad58b6700 1 -- 172.21.15.46:0/2805891456 >> 172.21.15.46:0/2805891456 conn(0x7ffad00ce530 msgr2=0x7ffad00ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:50.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.751+0000 7ffad58b6700 1 -- 172.21.15.46:0/2805891456 shutdown_connections 2024-01-26T23:42:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.751+0000 7ffad58b6700 1 -- 172.21.15.46:0/2805891456 wait complete. 2024-01-26T23:42:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.752+0000 7ffad58b6700 1 Processor -- start 2024-01-26T23:42:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.752+0000 7ffad58b6700 1 -- start start 2024-01-26T23:42:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.753+0000 7ffad58b6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffad0069b90 0x7ffad016a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.753+0000 7ffad58b6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffad0159750 0x7ffad00afd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.753+0000 7ffad58b6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffad0159b20 0x7ffad00ade00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.753+0000 7ffad58b6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad016ad80 con 0x7ffad0159750 2024-01-26T23:42:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.753+0000 7ffad58b6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad00b2630 con 0x7ffad0159b20 2024-01-26T23:42:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.753+0000 7ffad58b6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad00b2910 con 0x7ffad0069b90 2024-01-26T23:42:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.753+0000 7fface7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffad0159750 0x7ffad00afd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.753+0000 7fface7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffad0159750 0x7ffad00afd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55982/0 (socket says 172.21.15.46:55982) 2024-01-26T23:42:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.753+0000 7fface7fc700 1 -- 172.21.15.46:0/785836519 learned_addr learned my addr 172.21.15.46:0/785836519 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.753+0000 7ffaceffd700 1 --2- 172.21.15.46:0/785836519 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffad0069b90 0x7ffad016a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.753+0000 7ffacf7fe700 1 --2- 172.21.15.46:0/785836519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffad0159b20 0x7ffad00ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.754+0000 7fface7fc700 1 -- 172.21.15.46:0/785836519 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffad0069b90 msgr2=0x7ffad016a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.754+0000 7fface7fc700 1 --2- 172.21.15.46:0/785836519 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffad0069b90 0x7ffad016a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.754+0000 7fface7fc700 1 -- 172.21.15.46:0/785836519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffad0159b20 msgr2=0x7ffad00ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.754+0000 7fface7fc700 1 --2- 172.21.15.46:0/785836519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffad0159b20 0x7ffad00ade00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.754+0000 7fface7fc700 1 -- 172.21.15.46:0/785836519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab8009420 con 0x7ffad0159750 2024-01-26T23:42:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.754+0000 7fface7fc700 1 --2- 172.21.15.46:0/785836519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffad0159750 0x7ffad00afd40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ffad006f390 tx=0x7ffab8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.754+0000 7ffac6ffd700 1 -- 172.21.15.46:0/785836519 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab800bbe0 con 0x7ffad0159750 2024-01-26T23:42:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.754+0000 7ffac6ffd700 1 -- 172.21.15.46:0/785836519 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab8011420 con 0x7ffad0159750 2024-01-26T23:42:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.754+0000 7ffad58b6700 1 -- 172.21.15.46:0/785836519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad00b0720 con 0x7ffad0159750 2024-01-26T23:42:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.754+0000 7ffac6ffd700 1 -- 172.21.15.46:0/785836519 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab80117a0 con 0x7ffad0159750 2024-01-26T23:42:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.754+0000 7ffad58b6700 1 -- 172.21.15.46:0/785836519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad00b0ce0 con 0x7ffad0159750 2024-01-26T23:42:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.755+0000 7ffad58b6700 1 -- 172.21.15.46:0/785836519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad004c710 con 0x7ffad0159750 2024-01-26T23:42:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.755+0000 7ffac6ffd700 1 -- 172.21.15.46:0/785836519 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 84580+0+0 (secure 0 0 0) 0x7ffab8011b50 con 0x7ffad0159750 2024-01-26T23:42:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.756+0000 7ffac6ffd700 1 --2- 172.21.15.46:0/785836519 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffab0066d80 0x7ffab0069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.756+0000 7ffac6ffd700 1 -- 172.21.15.46:0/785836519 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7063+0+0 (secure 0 0 0) 0x7ffab8086000 con 0x7ffad0159750 2024-01-26T23:42:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.756+0000 7ffaceffd700 1 --2- 172.21.15.46:0/785836519 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffab0066d80 0x7ffab0069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.757+0000 7ffaceffd700 1 --2- 172.21.15.46:0/785836519 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffab0066d80 0x7ffab0069230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffac0006f00 tx=0x7ffac0008040 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:50.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.761+0000 7ffac6ffd700 1 -- 172.21.15.46:0/785836519 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffab8052fd0 con 0x7ffad0159750 2024-01-26T23:42:50.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:50.967+0000 7ffad58b6700 1 -- 172.21.15.46:0/785836519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ffad00b2a50 con 0x7ffad0159750 2024-01-26T23:42:51.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.649+0000 7ffac6ffd700 1 -- 172.21.15.46:0/785836519 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v61) v1 ==== 181+0+0 (secure 0 0 0) 0x7ffab80565f0 con 0x7ffad0159750 2024-01-26T23:42:51.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.653+0000 7ffad58b6700 1 -- 172.21.15.46:0/785836519 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffab0066d80 msgr2=0x7ffab0069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:51.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.653+0000 7ffad58b6700 1 --2- 172.21.15.46:0/785836519 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffab0066d80 0x7ffab0069230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffac0006f00 tx=0x7ffac0008040 comp rx=0 tx=0).stop 2024-01-26T23:42:51.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.653+0000 7ffad58b6700 1 -- 172.21.15.46:0/785836519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffad0159750 msgr2=0x7ffad00afd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:51.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.653+0000 7ffad58b6700 1 --2- 172.21.15.46:0/785836519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffad0159750 0x7ffad00afd40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ffad006f390 tx=0x7ffab8017000 comp rx=0 tx=0).stop 2024-01-26T23:42:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.657+0000 7ffad58b6700 1 -- 172.21.15.46:0/785836519 shutdown_connections 2024-01-26T23:42:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.657+0000 7ffad58b6700 1 --2- 172.21.15.46:0/785836519 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffab0066d80 0x7ffab0069230 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.657+0000 7ffad58b6700 1 --2- 172.21.15.46:0/785836519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffad0159b20 0x7ffad00ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.657+0000 7ffad58b6700 1 --2- 172.21.15.46:0/785836519 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffad0069b90 0x7ffad016a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.657+0000 7ffad58b6700 1 --2- 172.21.15.46:0/785836519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffad0159750 0x7ffad00afd40 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.657+0000 7ffad58b6700 1 -- 172.21.15.46:0/785836519 >> 172.21.15.46:0/785836519 conn(0x7ffad00ce530 msgr2=0x7ffad0158890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.659+0000 7ffad58b6700 1 -- 172.21.15.46:0/785836519 shutdown_connections 2024-01-26T23:42:51.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.660+0000 7ffad58b6700 1 -- 172.21.15.46:0/785836519 wait complete. 2024-01-26T23:42:51.662 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-01-26T23:42:51.680 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-26T23:42:51.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.817+0000 7f95978b7700 1 Processor -- start 2024-01-26T23:42:51.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.817+0000 7f95978b7700 1 -- start start 2024-01-26T23:42:51.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.818+0000 7f95978b7700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f95900c1ab0 0x7f9590068f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:51.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.818+0000 7f95978b7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f959006bcb0 0x7f959006c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:51.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.818+0000 7f95978b7700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9590161f70 con 0x7f95900c16e0 2024-01-26T23:42:51.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.818+0000 7f95978b7700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95900c98c0 con 0x7f95900c1ab0 2024-01-26T23:42:51.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.818+0000 7f95978b7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f959006c670 con 0x7f959006bcb0 2024-01-26T23:42:51.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.818+0000 7f9595653700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f95900c16e0 0x7f9590162400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39840/0 (socket says 172.21.15.46:39840) 2024-01-26T23:42:51.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.818+0000 7f9595653700 1 -- 172.21.15.46:0/3292610787 learned_addr learned my addr 172.21.15.46:0/3292610787 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:51.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.818+0000 7f9594e52700 1 --2- 172.21.15.46:0/3292610787 >> v2:172.21.15.106:3300/0 conn(0x7f95900c1ab0 0x7f9590068f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:51.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.818+0000 7f9595e54700 1 --2- 172.21.15.46:0/3292610787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f959006bcb0 0x7f959006c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:51.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.819+0000 7f9587fff700 1 -- 172.21.15.46:0/3292610787 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945217200 0 0) 0x7f9590161f70 con 0x7f95900c16e0 2024-01-26T23:42:51.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.819+0000 7f9587fff700 1 -- 172.21.15.46:0/3292610787 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f957c003570 con 0x7f95900c16e0 2024-01-26T23:42:51.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.819+0000 7f9594e52700 1 -- 172.21.15.46:0/3292610787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f959006bcb0 msgr2=0x7f959006c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:51.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.819+0000 7f9594e52700 1 --2- 172.21.15.46:0/3292610787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f959006bcb0 0x7f959006c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:51.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.819+0000 7f9594e52700 1 -- 172.21.15.46:0/3292610787 >> v1:172.21.15.46:6789/0 conn(0x7f95900c16e0 legacy=0x7f9590162400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:51.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.819+0000 7f9594e52700 1 -- 172.21.15.46:0/3292610787 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f959015e390 con 0x7f95900c1ab0 2024-01-26T23:42:51.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.819+0000 7f9595e54700 1 --2- 172.21.15.46:0/3292610787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f959006bcb0 0x7f959006c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:51.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.819+0000 7f9594e52700 1 --2- 172.21.15.46:0/3292610787 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95900c1ab0 0x7f9590068f10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9580009910 tx=0x7f95800091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fba6312ec94b4df server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:51.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.819+0000 7f9587fff700 1 -- 172.21.15.46:0/3292610787 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9580011630 con 0x7f95900c1ab0 2024-01-26T23:42:51.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.819+0000 7f9587fff700 1 -- 172.21.15.46:0/3292610787 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9580011790 con 0x7f95900c1ab0 2024-01-26T23:42:51.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.819+0000 7f9587fff700 1 -- 172.21.15.46:0/3292610787 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9580011b30 con 0x7f95900c1ab0 2024-01-26T23:42:51.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.820+0000 7f95978b7700 1 -- 172.21.15.46:0/3292610787 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95900c1ab0 msgr2=0x7f9590068f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:51.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.820+0000 7f95978b7700 1 --2- 172.21.15.46:0/3292610787 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95900c1ab0 0x7f9590068f10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9580009910 tx=0x7f95800091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:51.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.820+0000 7f95978b7700 1 -- 172.21.15.46:0/3292610787 shutdown_connections 2024-01-26T23:42:51.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.820+0000 7f95978b7700 1 --2- 172.21.15.46:0/3292610787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f959006bcb0 0x7f959006c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:51.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.820+0000 7f95978b7700 1 --2- 172.21.15.46:0/3292610787 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95900c1ab0 0x7f9590068f10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:51.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.820+0000 7f95978b7700 1 -- 172.21.15.46:0/3292610787 >> 172.21.15.46:0/3292610787 conn(0x7f95900ce530 msgr2=0x7f95900ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:51.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.820+0000 7f95978b7700 1 -- 172.21.15.46:0/3292610787 shutdown_connections 2024-01-26T23:42:51.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.820+0000 7f95978b7700 1 -- 172.21.15.46:0/3292610787 wait complete. 2024-01-26T23:42:51.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.821+0000 7f95978b7700 1 Processor -- start 2024-01-26T23:42:51.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.821+0000 7f95978b7700 1 -- start start 2024-01-26T23:42:51.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.822+0000 7f95978b7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f959006bcb0 0x7f9590172190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.822+0000 7f95978b7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95900c16e0 0x7f959015db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.822+0000 7f95978b7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95900c1ab0 0x7f959015e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.822+0000 7f95978b7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95900aff30 con 0x7f95900c1ab0 2024-01-26T23:42:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.822+0000 7f95978b7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95900ae930 con 0x7f95900c16e0 2024-01-26T23:42:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.822+0000 7f95978b7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95901726d0 con 0x7f959006bcb0 2024-01-26T23:42:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.822+0000 7f9595e54700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95900c1ab0 0x7f959015e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.823+0000 7f9595e54700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95900c1ab0 0x7f959015e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55998/0 (socket says 172.21.15.46:55998) 2024-01-26T23:42:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.823+0000 7f9595e54700 1 -- 172.21.15.46:0/2435520655 learned_addr learned my addr 172.21.15.46:0/2435520655 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.823+0000 7f9595653700 1 --2- 172.21.15.46:0/2435520655 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f959006bcb0 0x7f9590172190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.823+0000 7f9594e52700 1 --2- 172.21.15.46:0/2435520655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95900c16e0 0x7f959015db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.823+0000 7f9595e54700 1 -- 172.21.15.46:0/2435520655 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f959006bcb0 msgr2=0x7f9590172190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.823+0000 7f9595e54700 1 --2- 172.21.15.46:0/2435520655 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f959006bcb0 0x7f9590172190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.823+0000 7f9595e54700 1 -- 172.21.15.46:0/2435520655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95900c16e0 msgr2=0x7f959015db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.823+0000 7f9595e54700 1 --2- 172.21.15.46:0/2435520655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95900c16e0 0x7f959015db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.823+0000 7f9595e54700 1 -- 172.21.15.46:0/2435520655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9588009a20 con 0x7f95900c1ab0 2024-01-26T23:42:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.823+0000 7f9595e54700 1 --2- 172.21.15.46:0/2435520655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95900c1ab0 0x7f959015e570 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f958800c0f0 tx=0x7f958800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.823+0000 7f95857fa700 1 -- 172.21.15.46:0/2435520655 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958800cc30 con 0x7f95900c1ab0 2024-01-26T23:42:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.823+0000 7f95857fa700 1 -- 172.21.15.46:0/2435520655 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958800f040 con 0x7f95900c1ab0 2024-01-26T23:42:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.823+0000 7f95978b7700 1 -- 172.21.15.46:0/2435520655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9580009420 con 0x7f95900c1ab0 2024-01-26T23:42:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.823+0000 7f95857fa700 1 -- 172.21.15.46:0/2435520655 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9588014480 con 0x7f95900c1ab0 2024-01-26T23:42:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.823+0000 7f95978b7700 1 -- 172.21.15.46:0/2435520655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95900af320 con 0x7f95900c1ab0 2024-01-26T23:42:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.824+0000 7f95978b7700 1 -- 172.21.15.46:0/2435520655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95900b00a0 con 0x7f95900c1ab0 2024-01-26T23:42:51.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.825+0000 7f95857fa700 1 -- 172.21.15.46:0/2435520655 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f9588003020 con 0x7f95900c1ab0 2024-01-26T23:42:51.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.825+0000 7f95857fa700 1 --2- 172.21.15.46:0/2435520655 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9574066d80 0x7f9574069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:51.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.825+0000 7f95857fa700 1 -- 172.21.15.46:0/2435520655 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6625+0+0 (secure 0 0 0) 0x7f9588084e90 con 0x7f95900c1ab0 2024-01-26T23:42:51.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.825+0000 7f9595653700 1 --2- 172.21.15.46:0/2435520655 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9574066d80 0x7f9574069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:51.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.826+0000 7f9595653700 1 --2- 172.21.15.46:0/2435520655 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9574066d80 0x7f9574069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f958c007fd0 tx=0x7f958c009040 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:51.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:51.830+0000 7f95857fa700 1 -- 172.21.15.46:0/2435520655 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9588052010 con 0x7f95900c1ab0 2024-01-26T23:42:52.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.033+0000 7f95978b7700 1 -- 172.21.15.46:0/2435520655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f959004c710 con 0x7f95900c1ab0 2024-01-26T23:42:52.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.602+0000 7f95857fa700 1 -- 172.21.15.46:0/2435520655 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 17) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f95880523e0 con 0x7f95900c1ab0 2024-01-26T23:42:52.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.664+0000 7f95857fa700 1 -- 172.21.15.46:0/2435520655 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v62) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9588055810 con 0x7f95900c1ab0 2024-01-26T23:42:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.668+0000 7f95978b7700 1 -- 172.21.15.46:0/2435520655 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9574066d80 msgr2=0x7f9574069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.668+0000 7f95978b7700 1 --2- 172.21.15.46:0/2435520655 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9574066d80 0x7f9574069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f958c007fd0 tx=0x7f958c009040 comp rx=0 tx=0).stop 2024-01-26T23:42:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.668+0000 7f95978b7700 1 -- 172.21.15.46:0/2435520655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95900c1ab0 msgr2=0x7f959015e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.668+0000 7f95978b7700 1 --2- 172.21.15.46:0/2435520655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95900c1ab0 0x7f959015e570 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f958800c0f0 tx=0x7f958800b040 comp rx=0 tx=0).stop 2024-01-26T23:42:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.668+0000 7f95978b7700 1 -- 172.21.15.46:0/2435520655 shutdown_connections 2024-01-26T23:42:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.668+0000 7f95978b7700 1 --2- 172.21.15.46:0/2435520655 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9574066d80 0x7f9574069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:52.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.668+0000 7f95978b7700 1 --2- 172.21.15.46:0/2435520655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95900c1ab0 0x7f959015e570 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:52.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.669+0000 7f95978b7700 1 --2- 172.21.15.46:0/2435520655 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f959006bcb0 0x7f9590172190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:52.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.669+0000 7f95978b7700 1 --2- 172.21.15.46:0/2435520655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95900c16e0 0x7f959015db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:52.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.669+0000 7f95978b7700 1 -- 172.21.15.46:0/2435520655 >> 172.21.15.46:0/2435520655 conn(0x7f95900ce530 msgr2=0x7f95900cd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:52.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.669+0000 7f95978b7700 1 -- 172.21.15.46:0/2435520655 shutdown_connections 2024-01-26T23:42:52.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.669+0000 7f95978b7700 1 -- 172.21.15.46:0/2435520655 wait complete. 2024-01-26T23:42:52.671 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-01-26T23:42:52.687 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-26T23:42:52.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.824+0000 7faa7d5dd700 1 Processor -- start 2024-01-26T23:42:52.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.824+0000 7faa7d5dd700 1 -- start start 2024-01-26T23:42:52.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.825+0000 7faa7d5dd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faa781599d0 0x7faa78159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:52.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.825+0000 7faa7d5dd700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa780ab620 con 0x7faa7815d8c0 2024-01-26T23:42:52.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.825+0000 7faa7d5dd700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa780b1a60 con 0x7faa780ab1c0 2024-01-26T23:42:52.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.825+0000 7faa7d5dd700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa780b11c0 con 0x7faa781599d0 2024-01-26T23:42:52.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.825+0000 7faa76ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faa781599d0 0x7faa78159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:52.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.825+0000 7faa777fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faa7815d8c0 0x7faa780b3b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39846/0 (socket says 172.21.15.46:39846) 2024-01-26T23:42:52.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.825+0000 7faa777fe700 1 -- 172.21.15.46:0/3564123605 learned_addr learned my addr 172.21.15.46:0/3564123605 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:52.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.825+0000 7faa76ffd700 1 -- 172.21.15.46:0/3564123605 >> v1:172.21.15.106:6789/0 conn(0x7faa780ab1c0 legacy=0x7faa780b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:52.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.825+0000 7faa76ffd700 1 -- 172.21.15.46:0/3564123605 >> v1:172.21.15.46:6789/0 conn(0x7faa7815d8c0 legacy=0x7faa780b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.825+0000 7faa76ffd700 1 -- 172.21.15.46:0/3564123605 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa780b14a0 con 0x7faa781599d0 2024-01-26T23:42:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.825+0000 7faa76ffd700 1 --2- 172.21.15.46:0/3564123605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa781599d0 0x7faa78159da0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7faa680098d0 tx=0x7faa68009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39e6e6b5e4f60f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.826+0000 7faa75ffb700 1 -- 172.21.15.46:0/3564123605 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa68009e70 con 0x7faa781599d0 2024-01-26T23:42:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.826+0000 7faa75ffb700 1 -- 172.21.15.46:0/3564123605 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa68005c10 con 0x7faa781599d0 2024-01-26T23:42:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.826+0000 7faa75ffb700 1 -- 172.21.15.46:0/3564123605 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6800b650 con 0x7faa781599d0 2024-01-26T23:42:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.826+0000 7faa7d5dd700 1 -- 172.21.15.46:0/3564123605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa781599d0 msgr2=0x7faa78159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.826+0000 7faa7d5dd700 1 --2- 172.21.15.46:0/3564123605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa781599d0 0x7faa78159da0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7faa680098d0 tx=0x7faa68009120 comp rx=0 tx=0).stop 2024-01-26T23:42:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.826+0000 7faa7d5dd700 1 -- 172.21.15.46:0/3564123605 shutdown_connections 2024-01-26T23:42:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.826+0000 7faa7d5dd700 1 --2- 172.21.15.46:0/3564123605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa781599d0 0x7faa78159da0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.826+0000 7faa7d5dd700 1 -- 172.21.15.46:0/3564123605 >> 172.21.15.46:0/3564123605 conn(0x7faa7806f840 msgr2=0x7faa78087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.826+0000 7faa7d5dd700 1 -- 172.21.15.46:0/3564123605 shutdown_connections 2024-01-26T23:42:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.826+0000 7faa7d5dd700 1 -- 172.21.15.46:0/3564123605 wait complete. 2024-01-26T23:42:52.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.828+0000 7faa7d5dd700 1 Processor -- start 2024-01-26T23:42:52.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.828+0000 7faa7d5dd700 1 -- start start 2024-01-26T23:42:52.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.828+0000 7faa7d5dd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa780ab1c0 0x7faa78145e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:52.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.828+0000 7faa7d5dd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa781599d0 0x7faa7809b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:52.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa7d5dd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa7815d8c0 0x7faa7809bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:52.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa7d5dd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7816a400 con 0x7faa781599d0 2024-01-26T23:42:52.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa7d5dd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7809ce90 con 0x7faa780ab1c0 2024-01-26T23:42:52.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa7d5dd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7809c280 con 0x7faa7815d8c0 2024-01-26T23:42:52.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa767fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa781599d0 0x7faa7809b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:52.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa767fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa781599d0 0x7faa7809b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56024/0 (socket says 172.21.15.46:56024) 2024-01-26T23:42:52.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa767fc700 1 -- 172.21.15.46:0/3408678892 learned_addr learned my addr 172.21.15.46:0/3408678892 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:52.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa777fe700 1 --2- 172.21.15.46:0/3408678892 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa7815d8c0 0x7faa7809bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:52.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa76ffd700 1 --2- 172.21.15.46:0/3408678892 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa780ab1c0 0x7faa78145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:52.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa767fc700 1 -- 172.21.15.46:0/3408678892 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa7815d8c0 msgr2=0x7faa7809bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:52.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa767fc700 1 --2- 172.21.15.46:0/3408678892 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa7815d8c0 0x7faa7809bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:52.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa767fc700 1 -- 172.21.15.46:0/3408678892 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa780ab1c0 msgr2=0x7faa78145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:52.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa767fc700 1 --2- 172.21.15.46:0/3408678892 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa780ab1c0 0x7faa78145e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:52.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa767fc700 1 -- 172.21.15.46:0/3408678892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa680093c0 con 0x7faa781599d0 2024-01-26T23:42:52.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa767fc700 1 --2- 172.21.15.46:0/3408678892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa781599d0 0x7faa7809b730 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7faa6c00c590 tx=0x7faa6c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:52.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa5effd700 1 -- 172.21.15.46:0/3408678892 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c011590 con 0x7faa781599d0 2024-01-26T23:42:52.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa5effd700 1 -- 172.21.15.46:0/3408678892 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6c011710 con 0x7faa781599d0 2024-01-26T23:42:52.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa7d5dd700 1 -- 172.21.15.46:0/3408678892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa7809c560 con 0x7faa781599d0 2024-01-26T23:42:52.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.829+0000 7faa5effd700 1 -- 172.21.15.46:0/3408678892 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c011ad0 con 0x7faa781599d0 2024-01-26T23:42:52.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.830+0000 7faa7d5dd700 1 -- 172.21.15.46:0/3408678892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7809cba0 con 0x7faa781599d0 2024-01-26T23:42:52.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.831+0000 7faa7d5dd700 1 -- 172.21.15.46:0/3408678892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7804c710 con 0x7faa781599d0 2024-01-26T23:42:52.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.831+0000 7faa5effd700 1 -- 172.21.15.46:0/3408678892 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 84686+0+0 (secure 0 0 0) 0x7faa6c01b3b0 con 0x7faa781599d0 2024-01-26T23:42:52.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.831+0000 7faa5effd700 1 --2- 172.21.15.46:0/3408678892 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa64066ea0 0x7faa64069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:52.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.831+0000 7faa5effd700 1 -- 172.21.15.46:0/3408678892 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6182+0+0 (secure 0 0 0) 0x7faa6c083a80 con 0x7faa781599d0 2024-01-26T23:42:52.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.832+0000 7faa76ffd700 1 --2- 172.21.15.46:0/3408678892 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa64066ea0 0x7faa64069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:52.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.832+0000 7faa76ffd700 1 --2- 172.21.15.46:0/3408678892 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa64066ea0 0x7faa64069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faa68009160 tx=0x7faa68016000 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:52.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:52.836+0000 7faa5effd700 1 -- 172.21.15.46:0/3408678892 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faa6c085e60 con 0x7faa781599d0 2024-01-26T23:42:53.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.047+0000 7faa7d5dd700 1 -- 172.21.15.46:0/3408678892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7faa7809cfd0 con 0x7faa781599d0 2024-01-26T23:42:53.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.051+0000 7faa5effd700 1 -- 172.21.15.46:0/3408678892 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7faa6c017020 con 0x7faa781599d0 2024-01-26T23:42:53.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.054+0000 7faa7d5dd700 1 -- 172.21.15.46:0/3408678892 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa64066ea0 msgr2=0x7faa64069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.054+0000 7faa7d5dd700 1 --2- 172.21.15.46:0/3408678892 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa64066ea0 0x7faa64069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faa68009160 tx=0x7faa68016000 comp rx=0 tx=0).stop 2024-01-26T23:42:53.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.054+0000 7faa7d5dd700 1 -- 172.21.15.46:0/3408678892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa781599d0 msgr2=0x7faa7809b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.054+0000 7faa7d5dd700 1 --2- 172.21.15.46:0/3408678892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa781599d0 0x7faa7809b730 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7faa6c00c590 tx=0x7faa6c00c8a0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.054+0000 7faa7d5dd700 1 -- 172.21.15.46:0/3408678892 shutdown_connections 2024-01-26T23:42:53.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.054+0000 7faa7d5dd700 1 --2- 172.21.15.46:0/3408678892 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa64066ea0 0x7faa64069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.054+0000 7faa7d5dd700 1 --2- 172.21.15.46:0/3408678892 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa7815d8c0 0x7faa7809bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.054+0000 7faa7d5dd700 1 --2- 172.21.15.46:0/3408678892 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa780ab1c0 0x7faa78145e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.054+0000 7faa7d5dd700 1 --2- 172.21.15.46:0/3408678892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa781599d0 0x7faa7809b730 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.054+0000 7faa7d5dd700 1 -- 172.21.15.46:0/3408678892 >> 172.21.15.46:0/3408678892 conn(0x7faa7806f840 msgr2=0x7faa78158ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:53.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.054+0000 7faa7d5dd700 1 -- 172.21.15.46:0/3408678892 shutdown_connections 2024-01-26T23:42:53.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.054+0000 7faa7d5dd700 1 -- 172.21.15.46:0/3408678892 wait complete. 2024-01-26T23:42:53.069 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-26T23:42:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.201+0000 7fc70a72c700 1 Processor -- start 2024-01-26T23:42:53.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.201+0000 7fc70a72c700 1 -- start start 2024-01-26T23:42:53.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.202+0000 7fc70a72c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc704159510 0x7fc7041598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:53.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.202+0000 7fc70a72c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7040ca690 0x7fc7040cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:53.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.202+0000 7fc70a72c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc704069260 con 0x7fc7040ca220 2024-01-26T23:42:53.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.202+0000 7fc70a72c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc70415d670 con 0x7fc704159510 2024-01-26T23:42:53.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.202+0000 7fc70a72c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7040c9750 con 0x7fc7040ca690 2024-01-26T23:42:53.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.202+0000 7fc7037fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc7040ca220 0x7fc7040c8be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39850/0 (socket says 172.21.15.46:39850) 2024-01-26T23:42:53.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.202+0000 7fc7037fe700 1 -- 172.21.15.46:0/629626900 learned_addr learned my addr 172.21.15.46:0/629626900 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:53.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.202+0000 7fc708cc9700 1 --2- 172.21.15.46:0/629626900 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7040ca690 0x7fc7040cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:53.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.202+0000 7fc703fff700 1 --2- 172.21.15.46:0/629626900 >> v2:172.21.15.106:3300/0 conn(0x7fc704159510 0x7fc7041598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:53.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.202+0000 7fc702ffd700 1 -- 172.21.15.46:0/629626900 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332371108 0 0) 0x7fc704069260 con 0x7fc7040ca220 2024-01-26T23:42:53.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.202+0000 7fc702ffd700 1 -- 172.21.15.46:0/629626900 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6ec003570 con 0x7fc7040ca220 2024-01-26T23:42:53.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.203+0000 7fc708cc9700 1 -- 172.21.15.46:0/629626900 >> v2:172.21.15.106:3300/0 conn(0x7fc704159510 msgr2=0x7fc7041598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:53.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.203+0000 7fc708cc9700 1 --2- 172.21.15.46:0/629626900 >> v2:172.21.15.106:3300/0 conn(0x7fc704159510 0x7fc7041598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.203+0000 7fc708cc9700 1 -- 172.21.15.46:0/629626900 >> v1:172.21.15.46:6789/0 conn(0x7fc7040ca220 legacy=0x7fc7040c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.203+0000 7fc708cc9700 1 -- 172.21.15.46:0/629626900 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7040c9a30 con 0x7fc7040ca690 2024-01-26T23:42:53.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.203+0000 7fc708cc9700 1 --2- 172.21.15.46:0/629626900 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7040ca690 0x7fc7040cc380 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc009e60 tx=0x7fc6fc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e045a75a78d4b06 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:53.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.203+0000 7fc702ffd700 1 -- 172.21.15.46:0/629626900 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc00e070 con 0x7fc7040ca690 2024-01-26T23:42:53.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.203+0000 7fc702ffd700 1 -- 172.21.15.46:0/629626900 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6fc00bab0 con 0x7fc7040ca690 2024-01-26T23:42:53.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.203+0000 7fc702ffd700 1 -- 172.21.15.46:0/629626900 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc00be70 con 0x7fc7040ca690 2024-01-26T23:42:53.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.204+0000 7fc70a72c700 1 -- 172.21.15.46:0/629626900 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7040ca690 msgr2=0x7fc7040cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.204+0000 7fc70a72c700 1 --2- 172.21.15.46:0/629626900 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7040ca690 0x7fc7040cc380 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc009e60 tx=0x7fc6fc009780 comp rx=0 tx=0).stop 2024-01-26T23:42:53.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.204+0000 7fc70a72c700 1 -- 172.21.15.46:0/629626900 shutdown_connections 2024-01-26T23:42:53.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.204+0000 7fc70a72c700 1 --2- 172.21.15.46:0/629626900 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7040ca690 0x7fc7040cc380 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.204+0000 7fc70a72c700 1 --2- 172.21.15.46:0/629626900 >> v2:172.21.15.106:3300/0 conn(0x7fc704159510 0x7fc7041598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.204+0000 7fc70a72c700 1 -- 172.21.15.46:0/629626900 >> 172.21.15.46:0/629626900 conn(0x7fc70406bd90 msgr2=0x7fc70406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:53.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.204+0000 7fc70a72c700 1 -- 172.21.15.46:0/629626900 shutdown_connections 2024-01-26T23:42:53.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.204+0000 7fc70a72c700 1 -- 172.21.15.46:0/629626900 wait complete. 2024-01-26T23:42:53.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.205+0000 7fc70a72c700 1 Processor -- start 2024-01-26T23:42:53.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.205+0000 7fc70a72c700 1 -- start start 2024-01-26T23:42:53.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.206+0000 7fc70a72c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7040ca220 0x7fc7040b1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:53.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.206+0000 7fc70a72c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc704159510 0x7fc7041543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:53.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.206+0000 7fc70a72c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc704154ed0 0x7fc704155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:53.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.206+0000 7fc70a72c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70415b510 con 0x7fc704154ed0 2024-01-26T23:42:53.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.206+0000 7fc70a72c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70415af00 con 0x7fc7040ca220 2024-01-26T23:42:53.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.206+0000 7fc70a72c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7040b1c20 con 0x7fc704159510 2024-01-26T23:42:53.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.206+0000 7fc703fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7040ca220 0x7fc7040b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:53.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.206+0000 7fc703fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7040ca220 0x7fc7040b1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:43148/0 (socket says 172.21.15.46:43148) 2024-01-26T23:42:53.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.206+0000 7fc703fff700 1 -- 172.21.15.46:0/2417957759 learned_addr learned my addr 172.21.15.46:0/2417957759 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:53.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.206+0000 7fc708cc9700 1 --2- 172.21.15.46:0/2417957759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc704154ed0 0x7fc704155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:53.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.206+0000 7fc7037fe700 1 --2- 172.21.15.46:0/2417957759 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc704159510 0x7fc7041543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:53.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.207+0000 7fc703fff700 1 -- 172.21.15.46:0/2417957759 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc704159510 msgr2=0x7fc7041543e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.207+0000 7fc703fff700 1 --2- 172.21.15.46:0/2417957759 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc704159510 0x7fc7041543e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.207+0000 7fc703fff700 1 -- 172.21.15.46:0/2417957759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc704154ed0 msgr2=0x7fc704155870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.207+0000 7fc703fff700 1 --2- 172.21.15.46:0/2417957759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc704154ed0 0x7fc704155870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.207+0000 7fc703fff700 1 -- 172.21.15.46:0/2417957759 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6fc009a20 con 0x7fc7040ca220 2024-01-26T23:42:53.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.207+0000 7fc703fff700 1 --2- 172.21.15.46:0/2417957759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7040ca220 0x7fc7040b1340 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc6f400e930 tx=0x7fc6f40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:53.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.207+0000 7fc6ebfff700 1 -- 172.21.15.46:0/2417957759 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4013680 con 0x7fc7040ca220 2024-01-26T23:42:53.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.207+0000 7fc6ebfff700 1 -- 172.21.15.46:0/2417957759 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f40137e0 con 0x7fc7040ca220 2024-01-26T23:42:53.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.207+0000 7fc70a72c700 1 -- 172.21.15.46:0/2417957759 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70415b1a0 con 0x7fc7040ca220 2024-01-26T23:42:53.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.207+0000 7fc6ebfff700 1 -- 172.21.15.46:0/2417957759 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4013b80 con 0x7fc7040ca220 2024-01-26T23:42:53.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.207+0000 7fc70a72c700 1 -- 172.21.15.46:0/2417957759 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7040b1880 con 0x7fc7040ca220 2024-01-26T23:42:53.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.208+0000 7fc70a72c700 1 -- 172.21.15.46:0/2417957759 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc70415b330 con 0x7fc7040ca220 2024-01-26T23:42:53.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.209+0000 7fc6ebfff700 1 -- 172.21.15.46:0/2417957759 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fc6f400c470 con 0x7fc7040ca220 2024-01-26T23:42:53.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.209+0000 7fc6ebfff700 1 --2- 172.21.15.46:0/2417957759 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc6f0066ea0 0x7fc6f0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:53.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.209+0000 7fc6ebfff700 1 -- 172.21.15.46:0/2417957759 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6182+0+0 (secure 0 0 0) 0x7fc6f40847a0 con 0x7fc7040ca220 2024-01-26T23:42:53.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.209+0000 7fc7037fe700 1 --2- 172.21.15.46:0/2417957759 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc6f0066ea0 0x7fc6f0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:53.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.210+0000 7fc7037fe700 1 --2- 172.21.15.46:0/2417957759 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc6f0066ea0 0x7fc6f0069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc70416cf10 tx=0x7fc6f8009040 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:53.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.214+0000 7fc6ebfff700 1 -- 172.21.15.46:0/2417957759 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc6f4051910 con 0x7fc7040ca220 2024-01-26T23:42:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.418+0000 7fc70a72c700 1 -- 172.21.15.46:0/2417957759 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fc70404c710 con 0x7fc7040ca220 2024-01-26T23:42:53.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.424+0000 7fc6ebfff700 1 -- 172.21.15.46:0/2417957759 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v14) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc6f4054f30 con 0x7fc7040ca220 2024-01-26T23:42:53.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.426+0000 7fc70a72c700 1 -- 172.21.15.46:0/2417957759 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc6f0066ea0 msgr2=0x7fc6f0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.426+0000 7fc70a72c700 1 --2- 172.21.15.46:0/2417957759 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc6f0066ea0 0x7fc6f0069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc70416cf10 tx=0x7fc6f8009040 comp rx=0 tx=0).stop 2024-01-26T23:42:53.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.426+0000 7fc70a72c700 1 -- 172.21.15.46:0/2417957759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7040ca220 msgr2=0x7fc7040b1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.426+0000 7fc70a72c700 1 --2- 172.21.15.46:0/2417957759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7040ca220 0x7fc7040b1340 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc6f400e930 tx=0x7fc6f40091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.426+0000 7fc70a72c700 1 -- 172.21.15.46:0/2417957759 shutdown_connections 2024-01-26T23:42:53.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.426+0000 7fc70a72c700 1 --2- 172.21.15.46:0/2417957759 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc6f0066ea0 0x7fc6f0069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.426+0000 7fc70a72c700 1 --2- 172.21.15.46:0/2417957759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc704154ed0 0x7fc704155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.426+0000 7fc70a72c700 1 --2- 172.21.15.46:0/2417957759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7040ca220 0x7fc7040b1340 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.426+0000 7fc70a72c700 1 --2- 172.21.15.46:0/2417957759 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc704159510 0x7fc7041543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.426+0000 7fc70a72c700 1 -- 172.21.15.46:0/2417957759 >> 172.21.15.46:0/2417957759 conn(0x7fc70406bd90 msgr2=0x7fc70406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:53.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.427+0000 7fc70a72c700 1 -- 172.21.15.46:0/2417957759 shutdown_connections 2024-01-26T23:42:53.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.427+0000 7fc70a72c700 1 -- 172.21.15.46:0/2417957759 wait complete. 2024-01-26T23:42:53.441 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-26T23:42:53.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.574+0000 7f187ed80700 1 Processor -- start 2024-01-26T23:42:53.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.574+0000 7f187ed80700 1 -- start start 2024-01-26T23:42:53.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.575+0000 7f187ed80700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f18780cc380 0x7f18780cc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:53.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.575+0000 7f187ed80700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1878069260 con 0x7f1878159510 2024-01-26T23:42:53.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.575+0000 7f187ed80700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f187815d670 con 0x7f18781598e0 2024-01-26T23:42:53.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.575+0000 7f187ed80700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18780ca930 con 0x7f18780cc380 2024-01-26T23:42:53.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.575+0000 7f187cb1c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1878159510 0x7f18780ca220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39854/0 (socket says 172.21.15.46:39854) 2024-01-26T23:42:53.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.575+0000 7f187cb1c700 1 -- 172.21.15.46:0/4258348789 learned_addr learned my addr 172.21.15.46:0/4258348789 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:53.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.575+0000 7f187d31d700 1 --2- 172.21.15.46:0/4258348789 >> v2:172.21.15.106:3300/0 conn(0x7f18780cc380 0x7f18780cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:53.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.575+0000 7f18777fe700 1 -- 172.21.15.46:0/4258348789 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616813908 0 0) 0x7f1878069260 con 0x7f1878159510 2024-01-26T23:42:53.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.575+0000 7f18777fe700 1 -- 172.21.15.46:0/4258348789 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1868003570 con 0x7f1878159510 2024-01-26T23:42:53.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.576+0000 7f187d31d700 1 -- 172.21.15.46:0/4258348789 >> v1:172.21.15.106:6789/0 conn(0x7f18781598e0 legacy=0x7f18780c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.576+0000 7f18777fe700 1 -- 172.21.15.46:0/4258348789 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692821207 0 0) 0x7f187815d670 con 0x7f18781598e0 2024-01-26T23:42:53.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.576+0000 7f187d31d700 1 -- 172.21.15.46:0/4258348789 >> v1:172.21.15.46:6789/0 conn(0x7f1878159510 legacy=0x7f18780ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.576+0000 7f187d31d700 1 -- 172.21.15.46:0/4258348789 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18780c9750 con 0x7f18780cc380 2024-01-26T23:42:53.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.576+0000 7f187d31d700 1 --2- 172.21.15.46:0/4258348789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18780cc380 0x7f18780cc800 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1870009e60 tx=0x7f1870009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c36d14d8cf362907 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:53.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.576+0000 7f18777fe700 1 -- 172.21.15.46:0/4258348789 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187000e070 con 0x7f18780cc380 2024-01-26T23:42:53.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.576+0000 7f18777fe700 1 -- 172.21.15.46:0/4258348789 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f187000ba60 con 0x7f18780cc380 2024-01-26T23:42:53.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.576+0000 7f18777fe700 1 -- 172.21.15.46:0/4258348789 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187000be00 con 0x7f18780cc380 2024-01-26T23:42:53.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.577+0000 7f187ed80700 1 -- 172.21.15.46:0/4258348789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18780cc380 msgr2=0x7f18780cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.577+0000 7f187ed80700 1 --2- 172.21.15.46:0/4258348789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18780cc380 0x7f18780cc800 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1870009e60 tx=0x7f1870009780 comp rx=0 tx=0).stop 2024-01-26T23:42:53.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.577+0000 7f187ed80700 1 -- 172.21.15.46:0/4258348789 shutdown_connections 2024-01-26T23:42:53.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.577+0000 7f187ed80700 1 --2- 172.21.15.46:0/4258348789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18780cc380 0x7f18780cc800 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.577+0000 7f187ed80700 1 -- 172.21.15.46:0/4258348789 >> 172.21.15.46:0/4258348789 conn(0x7f187806bd90 msgr2=0x7f187806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:53.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.577+0000 7f187ed80700 1 -- 172.21.15.46:0/4258348789 shutdown_connections 2024-01-26T23:42:53.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.577+0000 7f187ed80700 1 -- 172.21.15.46:0/4258348789 wait complete. 2024-01-26T23:42:53.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.578+0000 7f187ed80700 1 Processor -- start 2024-01-26T23:42:53.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.578+0000 7f187ed80700 1 -- start start 2024-01-26T23:42:53.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.579+0000 7f187ed80700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18780cc380 0x7f1878168dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.579+0000 7f187ed80700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1878159510 0x7f18780c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.579+0000 7f187ed80700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18781598e0 0x7f187816cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.579+0000 7f187ed80700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1878159d50 con 0x7f1878159510 2024-01-26T23:42:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.579+0000 7f187ed80700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187815b6f0 con 0x7f18780cc380 2024-01-26T23:42:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.579+0000 7f187ed80700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f187815c520 con 0x7f18781598e0 2024-01-26T23:42:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.579+0000 7f1877fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1878159510 0x7f18780c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.579+0000 7f1877fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1878159510 0x7f18780c5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56052/0 (socket says 172.21.15.46:56052) 2024-01-26T23:42:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.579+0000 7f1877fff700 1 -- 172.21.15.46:0/997202511 learned_addr learned my addr 172.21.15.46:0/997202511 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.579+0000 7f187cb1c700 1 --2- 172.21.15.46:0/997202511 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18780cc380 0x7f1878168dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.579+0000 7f187d31d700 1 --2- 172.21.15.46:0/997202511 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18781598e0 0x7f187816cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.579+0000 7f1877fff700 1 -- 172.21.15.46:0/997202511 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18781598e0 msgr2=0x7f187816cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.579+0000 7f1877fff700 1 --2- 172.21.15.46:0/997202511 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18781598e0 0x7f187816cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.579+0000 7f1877fff700 1 -- 172.21.15.46:0/997202511 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18780cc380 msgr2=0x7f1878168dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.579+0000 7f1877fff700 1 --2- 172.21.15.46:0/997202511 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18780cc380 0x7f1878168dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.580+0000 7f1877fff700 1 -- 172.21.15.46:0/997202511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1870009a20 con 0x7f1878159510 2024-01-26T23:42:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.580+0000 7f1877fff700 1 --2- 172.21.15.46:0/997202511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1878159510 0x7f18780c5650 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f186400c560 tx=0x7f186400c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.580+0000 7f1874ff9700 1 -- 172.21.15.46:0/997202511 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864011590 con 0x7f1878159510 2024-01-26T23:42:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.580+0000 7f1874ff9700 1 -- 172.21.15.46:0/997202511 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1864011710 con 0x7f1878159510 2024-01-26T23:42:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.580+0000 7f187ed80700 1 -- 172.21.15.46:0/997202511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f187815b910 con 0x7f1878159510 2024-01-26T23:42:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.580+0000 7f1874ff9700 1 -- 172.21.15.46:0/997202511 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864011ad0 con 0x7f1878159510 2024-01-26T23:42:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.580+0000 7f187ed80700 1 -- 172.21.15.46:0/997202511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f187815c1b0 con 0x7f1878159510 2024-01-26T23:42:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.581+0000 7f187ed80700 1 -- 172.21.15.46:0/997202511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f187815c6d0 con 0x7f1878159510 2024-01-26T23:42:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.581+0000 7f1874ff9700 1 -- 172.21.15.46:0/997202511 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f186401b3b0 con 0x7f1878159510 2024-01-26T23:42:53.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.582+0000 7f1874ff9700 1 --2- 172.21.15.46:0/997202511 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f185c066ea0 0x7f185c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:53.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.582+0000 7f1874ff9700 1 -- 172.21.15.46:0/997202511 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6182+0+0 (secure 0 0 0) 0x7f1864083a00 con 0x7f1878159510 2024-01-26T23:42:53.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.582+0000 7f187cb1c700 1 --2- 172.21.15.46:0/997202511 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f185c066ea0 0x7f185c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:53.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.583+0000 7f187cb1c700 1 --2- 172.21.15.46:0/997202511 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f185c066ea0 0x7f185c069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f186c007fd0 tx=0x7f186c000bc0 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:53.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.586+0000 7f1874ff9700 1 -- 172.21.15.46:0/997202511 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f187815c6d0 con 0x7f1878159510 2024-01-26T23:42:53.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.791+0000 7f187ed80700 1 -- 172.21.15.46:0/997202511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f187815bd00 con 0x7f1878159510 2024-01-26T23:42:53.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.794+0000 7f1874ff9700 1 -- 172.21.15.46:0/997202511 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v15) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1864017070 con 0x7f1878159510 2024-01-26T23:42:53.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.797+0000 7f187ed80700 1 -- 172.21.15.46:0/997202511 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f185c066ea0 msgr2=0x7f185c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.797+0000 7f187ed80700 1 --2- 172.21.15.46:0/997202511 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f185c066ea0 0x7f185c069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f186c007fd0 tx=0x7f186c000bc0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.797+0000 7f187ed80700 1 -- 172.21.15.46:0/997202511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1878159510 msgr2=0x7f18780c5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.797+0000 7f187ed80700 1 --2- 172.21.15.46:0/997202511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1878159510 0x7f18780c5650 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f186400c560 tx=0x7f186400c870 comp rx=0 tx=0).stop 2024-01-26T23:42:53.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.797+0000 7f187ed80700 1 -- 172.21.15.46:0/997202511 shutdown_connections 2024-01-26T23:42:53.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.797+0000 7f187ed80700 1 --2- 172.21.15.46:0/997202511 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f185c066ea0 0x7f185c069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.797+0000 7f187ed80700 1 --2- 172.21.15.46:0/997202511 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18781598e0 0x7f187816cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.797+0000 7f187ed80700 1 --2- 172.21.15.46:0/997202511 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18780cc380 0x7f1878168dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.797+0000 7f187ed80700 1 --2- 172.21.15.46:0/997202511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1878159510 0x7f18780c5650 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.797+0000 7f187ed80700 1 -- 172.21.15.46:0/997202511 >> 172.21.15.46:0/997202511 conn(0x7f187806bd90 msgr2=0x7f1878068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:53.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.797+0000 7f187ed80700 1 -- 172.21.15.46:0/997202511 shutdown_connections 2024-01-26T23:42:53.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.797+0000 7f187ed80700 1 -- 172.21.15.46:0/997202511 wait complete. 2024-01-26T23:42:53.812 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-01-26T23:42:53.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.942+0000 7eff398bf700 1 Processor -- start 2024-01-26T23:42:53.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.942+0000 7eff398bf700 1 -- start start 2024-01-26T23:42:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.943+0000 7eff398bf700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7eff340b2b40 0x7eff340ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.943+0000 7eff398bf700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff340ae960 0x7eff340ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.943+0000 7eff398bf700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff3415a0b0 con 0x7eff340aedd0 2024-01-26T23:42:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.943+0000 7eff398bf700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff34099fc0 con 0x7eff340ae960 2024-01-26T23:42:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.943+0000 7eff398bf700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff340ae4d0 con 0x7eff340b2b40 2024-01-26T23:42:53.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.943+0000 7eff327fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff340ae960 0x7eff340ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:53.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.943+0000 7eff337fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7eff340aedd0 0x7eff340cb5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39862/0 (socket says 172.21.15.46:39862) 2024-01-26T23:42:53.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.943+0000 7eff337fe700 1 -- 172.21.15.46:0/1770365813 learned_addr learned my addr 172.21.15.46:0/1770365813 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:53.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.943+0000 7eff32ffd700 1 --2- 172.21.15.46:0/1770365813 >> v2:172.21.15.106:3300/0 conn(0x7eff340b2b40 0x7eff340ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:53.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.943+0000 7eff327fc700 1 -- 172.21.15.46:0/1770365813 >> v2:172.21.15.106:3300/0 conn(0x7eff340b2b40 msgr2=0x7eff340ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:53.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.943+0000 7eff327fc700 1 --2- 172.21.15.46:0/1770365813 >> v2:172.21.15.106:3300/0 conn(0x7eff340b2b40 0x7eff340ad1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.944+0000 7eff327fc700 1 -- 172.21.15.46:0/1770365813 >> v1:172.21.15.46:6789/0 conn(0x7eff340aedd0 legacy=0x7eff340cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.944+0000 7eff327fc700 1 -- 172.21.15.46:0/1770365813 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff340ad8c0 con 0x7eff340ae960 2024-01-26T23:42:53.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.944+0000 7eff31ffb700 1 -- 172.21.15.46:0/1770365813 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618888569 0 0) 0x7eff3415a0b0 con 0x7eff340aedd0 2024-01-26T23:42:53.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.944+0000 7eff327fc700 1 --2- 172.21.15.46:0/1770365813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff340ae960 0x7eff340ced50 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7eff34088850 tx=0x7eff280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c26e21610eeda0f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:53.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.944+0000 7eff31ffb700 1 -- 172.21.15.46:0/1770365813 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2800b6b0 con 0x7eff340ae960 2024-01-26T23:42:53.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.944+0000 7eff31ffb700 1 -- 172.21.15.46:0/1770365813 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2800b810 con 0x7eff340ae960 2024-01-26T23:42:53.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.944+0000 7eff31ffb700 1 -- 172.21.15.46:0/1770365813 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2800bbb0 con 0x7eff340ae960 2024-01-26T23:42:53.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.944+0000 7eff398bf700 1 -- 172.21.15.46:0/1770365813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff340ae960 msgr2=0x7eff340ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.944+0000 7eff398bf700 1 --2- 172.21.15.46:0/1770365813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff340ae960 0x7eff340ced50 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7eff34088850 tx=0x7eff280091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.944+0000 7eff398bf700 1 -- 172.21.15.46:0/1770365813 shutdown_connections 2024-01-26T23:42:53.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.944+0000 7eff398bf700 1 --2- 172.21.15.46:0/1770365813 >> v2:172.21.15.106:3300/0 conn(0x7eff340b2b40 0x7eff340ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.944+0000 7eff398bf700 1 --2- 172.21.15.46:0/1770365813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff340ae960 0x7eff340ced50 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.944+0000 7eff398bf700 1 -- 172.21.15.46:0/1770365813 >> 172.21.15.46:0/1770365813 conn(0x7eff34099a30 msgr2=0x7eff3406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:53.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.945+0000 7eff398bf700 1 -- 172.21.15.46:0/1770365813 shutdown_connections 2024-01-26T23:42:53.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.945+0000 7eff398bf700 1 -- 172.21.15.46:0/1770365813 wait complete. 2024-01-26T23:42:53.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.946+0000 7eff398bf700 1 Processor -- start 2024-01-26T23:42:53.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.946+0000 7eff398bf700 1 -- start start 2024-01-26T23:42:53.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff398bf700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff340ae960 0x7eff340af290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:53.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff398bf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff340b2b40 0x7eff340afb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:53.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff398bf700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff34156250 0x7eff3416cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:53.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff398bf700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff340b0040 con 0x7eff340b2b40 2024-01-26T23:42:53.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff398bf700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff340b4180 con 0x7eff340ae960 2024-01-26T23:42:53.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff398bf700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff340b1960 con 0x7eff34156250 2024-01-26T23:42:53.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff327fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff340b2b40 0x7eff340afb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:53.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff327fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff340b2b40 0x7eff340afb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56066/0 (socket says 172.21.15.46:56066) 2024-01-26T23:42:53.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff327fc700 1 -- 172.21.15.46:0/2039173206 learned_addr learned my addr 172.21.15.46:0/2039173206 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:53.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff32ffd700 1 --2- 172.21.15.46:0/2039173206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff340ae960 0x7eff340af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:53.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff337fe700 1 --2- 172.21.15.46:0/2039173206 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff34156250 0x7eff3416cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:53.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff327fc700 1 -- 172.21.15.46:0/2039173206 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff34156250 msgr2=0x7eff3416cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff327fc700 1 --2- 172.21.15.46:0/2039173206 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff34156250 0x7eff3416cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff327fc700 1 -- 172.21.15.46:0/2039173206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff340ae960 msgr2=0x7eff340af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:53.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff327fc700 1 --2- 172.21.15.46:0/2039173206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff340ae960 0x7eff340af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:53.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff327fc700 1 -- 172.21.15.46:0/2039173206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff28009420 con 0x7eff340b2b40 2024-01-26T23:42:53.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff32ffd700 1 --2- 172.21.15.46:0/2039173206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff340ae960 0x7eff340af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:53.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff337fe700 1 --2- 172.21.15.46:0/2039173206 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff34156250 0x7eff3416cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:53.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.947+0000 7eff327fc700 1 --2- 172.21.15.46:0/2039173206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff340b2b40 0x7eff340afb00 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7eff34088e10 tx=0x7eff28017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:53.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.948+0000 7eff1affd700 1 -- 172.21.15.46:0/2039173206 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff28011420 con 0x7eff340b2b40 2024-01-26T23:42:53.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.948+0000 7eff1affd700 1 -- 172.21.15.46:0/2039173206 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff28011580 con 0x7eff340b2b40 2024-01-26T23:42:53.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.948+0000 7eff398bf700 1 -- 172.21.15.46:0/2039173206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff340b1c40 con 0x7eff340b2b40 2024-01-26T23:42:53.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.948+0000 7eff1affd700 1 -- 172.21.15.46:0/2039173206 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff28011920 con 0x7eff340b2b40 2024-01-26T23:42:53.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.948+0000 7eff398bf700 1 -- 172.21.15.46:0/2039173206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff340b0280 con 0x7eff340b2b40 2024-01-26T23:42:53.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.949+0000 7eff398bf700 1 -- 172.21.15.46:0/2039173206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff340639f0 con 0x7eff340b2b40 2024-01-26T23:42:53.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.949+0000 7eff1affd700 1 -- 172.21.15.46:0/2039173206 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 84686+0+0 (secure 0 0 0) 0x7eff28011d10 con 0x7eff340b2b40 2024-01-26T23:42:53.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.949+0000 7eff1affd700 1 --2- 172.21.15.46:0/2039173206 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff20066ea0 0x7eff20069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:53.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.950+0000 7eff1affd700 1 -- 172.21.15.46:0/2039173206 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6182+0+0 (secure 0 0 0) 0x7eff28085cd0 con 0x7eff340b2b40 2024-01-26T23:42:53.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.950+0000 7eff32ffd700 1 --2- 172.21.15.46:0/2039173206 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff20066ea0 0x7eff20069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:53.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.950+0000 7eff32ffd700 1 --2- 172.21.15.46:0/2039173206 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff20066ea0 0x7eff20069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7eff24006fd0 tx=0x7eff24008040 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:53.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:53.954+0000 7eff1affd700 1 -- 172.21.15.46:0/2039173206 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7eff28052ec0 con 0x7eff340b2b40 2024-01-26T23:42:54.238 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:42:54.240 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:42:54.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.248+0000 7eff398bf700 1 -- 172.21.15.46:0/2039173206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7eff3404c710 con 0x7eff340b2b40 2024-01-26T23:42:54.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.429+0000 7eff1affd700 1 -- 172.21.15.46:0/2039173206 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7eff28053290 con 0x7eff340b2b40 2024-01-26T23:42:54.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.668+0000 7eff1affd700 1 -- 172.21.15.46:0/2039173206 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v470) v1 ==== 158+0+0 (secure 0 0 0) 0x7eff28056950 con 0x7eff340b2b40 2024-01-26T23:42:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.671+0000 7eff398bf700 1 -- 172.21.15.46:0/2039173206 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff20066ea0 msgr2=0x7eff20069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.671+0000 7eff398bf700 1 --2- 172.21.15.46:0/2039173206 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff20066ea0 0x7eff20069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7eff24006fd0 tx=0x7eff24008040 comp rx=0 tx=0).stop 2024-01-26T23:42:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.671+0000 7eff398bf700 1 -- 172.21.15.46:0/2039173206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff340b2b40 msgr2=0x7eff340afb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.671+0000 7eff398bf700 1 --2- 172.21.15.46:0/2039173206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff340b2b40 0x7eff340afb00 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7eff34088e10 tx=0x7eff28017000 comp rx=0 tx=0).stop 2024-01-26T23:42:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.671+0000 7eff398bf700 1 -- 172.21.15.46:0/2039173206 shutdown_connections 2024-01-26T23:42:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.671+0000 7eff398bf700 1 --2- 172.21.15.46:0/2039173206 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff20066ea0 0x7eff20069350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.671+0000 7eff398bf700 1 --2- 172.21.15.46:0/2039173206 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff34156250 0x7eff3416cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.671+0000 7eff398bf700 1 --2- 172.21.15.46:0/2039173206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff340ae960 0x7eff340af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.671+0000 7eff398bf700 1 --2- 172.21.15.46:0/2039173206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff340b2b40 0x7eff340afb00 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.671+0000 7eff398bf700 1 -- 172.21.15.46:0/2039173206 >> 172.21.15.46:0/2039173206 conn(0x7eff34099a30 msgr2=0x7eff34068910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.671+0000 7eff398bf700 1 -- 172.21.15.46:0/2039173206 shutdown_connections 2024-01-26T23:42:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.671+0000 7eff398bf700 1 -- 172.21.15.46:0/2039173206 wait complete. 2024-01-26T23:42:54.688 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2024-01-26T23:42:54.688 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2024-01-26T23:42:54.688 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-26T23:42:54.688 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-01-26T23:42:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.824+0000 7f393e944700 1 Processor -- start 2024-01-26T23:42:54.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.824+0000 7f393e944700 1 -- start start 2024-01-26T23:42:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.825+0000 7f393e944700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3938159b30 0x7f39380b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.825+0000 7f393e944700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f39380b10f0 0x7f39380a9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.825+0000 7f393e944700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f393806f940 con 0x7f3938159760 2024-01-26T23:42:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.825+0000 7f393e944700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39380b1530 con 0x7f3938159b30 2024-01-26T23:42:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.825+0000 7f393e944700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39380b3230 con 0x7f39380b10f0 2024-01-26T23:42:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.825+0000 7f39377fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3938159b30 0x7f39380b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:54.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.825+0000 7f39377fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3938159b30 0x7f39380b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56070/0 (socket says 172.21.15.46:56070) 2024-01-26T23:42:54.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.825+0000 7f39377fe700 1 -- 172.21.15.46:0/569068331 learned_addr learned my addr 172.21.15.46:0/569068331 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:54.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.825+0000 7f393cee1700 1 --2- 172.21.15.46:0/569068331 >> v2:172.21.15.106:3300/0 conn(0x7f39380b10f0 0x7f39380a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:54.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.826+0000 7f39377fe700 1 -- 172.21.15.46:0/569068331 >> v2:172.21.15.106:3300/0 conn(0x7f39380b10f0 msgr2=0x7f39380a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:54.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.826+0000 7f39377fe700 1 --2- 172.21.15.46:0/569068331 >> v2:172.21.15.106:3300/0 conn(0x7f39380b10f0 0x7f39380a9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:54.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.826+0000 7f39377fe700 1 -- 172.21.15.46:0/569068331 >> v1:172.21.15.106:6789/0 conn(0x7f3938159760 legacy=0x7f39380b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:54.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.826+0000 7f39377fe700 1 -- 172.21.15.46:0/569068331 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393806ed30 con 0x7f3938159b30 2024-01-26T23:42:54.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.826+0000 7f39377fe700 1 --2- 172.21.15.46:0/569068331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3938159b30 0x7f39380b0bb0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f392c0099b0 tx=0x7f392c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c9cd1f724e2798 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:54.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.826+0000 7f3936ffd700 1 -- 172.21.15.46:0/569068331 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c009ea0 con 0x7f3938159b30 2024-01-26T23:42:54.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.826+0000 7f3936ffd700 1 -- 172.21.15.46:0/569068331 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392c00b820 con 0x7f3938159b30 2024-01-26T23:42:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.826+0000 7f3936ffd700 1 -- 172.21.15.46:0/569068331 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c00bbc0 con 0x7f3938159b30 2024-01-26T23:42:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.827+0000 7f393e944700 1 -- 172.21.15.46:0/569068331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3938159b30 msgr2=0x7f39380b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.827+0000 7f393e944700 1 --2- 172.21.15.46:0/569068331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3938159b30 0x7f39380b0bb0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f392c0099b0 tx=0x7f392c009250 comp rx=0 tx=0).stop 2024-01-26T23:42:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.827+0000 7f393e944700 1 -- 172.21.15.46:0/569068331 shutdown_connections 2024-01-26T23:42:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.827+0000 7f393e944700 1 --2- 172.21.15.46:0/569068331 >> v2:172.21.15.106:3300/0 conn(0x7f39380b10f0 0x7f39380a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.827+0000 7f393e944700 1 --2- 172.21.15.46:0/569068331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3938159b30 0x7f39380b0bb0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.827+0000 7f393e944700 1 -- 172.21.15.46:0/569068331 >> 172.21.15.46:0/569068331 conn(0x7f3938069510 msgr2=0x7f39380aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.827+0000 7f393e944700 1 -- 172.21.15.46:0/569068331 shutdown_connections 2024-01-26T23:42:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.827+0000 7f393e944700 1 -- 172.21.15.46:0/569068331 wait complete. 2024-01-26T23:42:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.828+0000 7f393e944700 1 Processor -- start 2024-01-26T23:42:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.828+0000 7f393e944700 1 -- start start 2024-01-26T23:42:54.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.829+0000 7f393e944700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39380b10f0 0x7f3938152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.829+0000 7f393e944700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3938159760 0x7f3938157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.829+0000 7f393e944700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3938159b30 0x7f393815c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.829+0000 7f393e944700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3938153920 con 0x7f39380b10f0 2024-01-26T23:42:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.829+0000 7f393e944700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3938154780 con 0x7f3938159760 2024-01-26T23:42:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.829+0000 7f393e944700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3938153b70 con 0x7f3938159b30 2024-01-26T23:42:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.829+0000 7f3937fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39380b10f0 0x7f3938152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.829+0000 7f3937fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39380b10f0 0x7f3938152ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56076/0 (socket says 172.21.15.46:56076) 2024-01-26T23:42:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.829+0000 7f3937fff700 1 -- 172.21.15.46:0/1660811517 learned_addr learned my addr 172.21.15.46:0/1660811517 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.829+0000 7f39377fe700 1 --2- 172.21.15.46:0/1660811517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3938159760 0x7f3938157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.829+0000 7f393cee1700 1 --2- 172.21.15.46:0/1660811517 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3938159b30 0x7f393815c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.829+0000 7f3937fff700 1 -- 172.21.15.46:0/1660811517 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3938159b30 msgr2=0x7f393815c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.829+0000 7f3937fff700 1 --2- 172.21.15.46:0/1660811517 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3938159b30 0x7f393815c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.829+0000 7f3937fff700 1 -- 172.21.15.46:0/1660811517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3938159760 msgr2=0x7f3938157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.829+0000 7f3937fff700 1 --2- 172.21.15.46:0/1660811517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3938159760 0x7f3938157e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:54.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.829+0000 7f3937fff700 1 -- 172.21.15.46:0/1660811517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c0094f0 con 0x7f39380b10f0 2024-01-26T23:42:54.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.829+0000 7f39377fe700 1 --2- 172.21.15.46:0/1660811517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3938159760 0x7f3938157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:42:54.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.830+0000 7f3937fff700 1 --2- 172.21.15.46:0/1660811517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39380b10f0 0x7f3938152ee0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f392800c5c0 tx=0x7f392800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:54.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.830+0000 7f391ffff700 1 -- 172.21.15.46:0/1660811517 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3928011590 con 0x7f39380b10f0 2024-01-26T23:42:54.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.830+0000 7f391ffff700 1 -- 172.21.15.46:0/1660811517 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3928011710 con 0x7f39380b10f0 2024-01-26T23:42:54.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.830+0000 7f391ffff700 1 -- 172.21.15.46:0/1660811517 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3928011ad0 con 0x7f39380b10f0 2024-01-26T23:42:54.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.830+0000 7f393e944700 1 -- 172.21.15.46:0/1660811517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3938153e50 con 0x7f39380b10f0 2024-01-26T23:42:54.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.830+0000 7f393e944700 1 -- 172.21.15.46:0/1660811517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3938154410 con 0x7f39380b10f0 2024-01-26T23:42:54.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.831+0000 7f393e944700 1 -- 172.21.15.46:0/1660811517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39381545a0 con 0x7f39380b10f0 2024-01-26T23:42:54.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.831+0000 7f391ffff700 1 -- 172.21.15.46:0/1660811517 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f392801b3b0 con 0x7f39380b10f0 2024-01-26T23:42:54.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.832+0000 7f391ffff700 1 --2- 172.21.15.46:0/1660811517 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3924066f30 0x7f39240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:54.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.832+0000 7f391ffff700 1 -- 172.21.15.46:0/1660811517 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6182+0+0 (secure 0 0 0) 0x7f3928083d90 con 0x7f39380b10f0 2024-01-26T23:42:54.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.832+0000 7f39377fe700 1 --2- 172.21.15.46:0/1660811517 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3924066f30 0x7f39240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:54.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.832+0000 7f39377fe700 1 --2- 172.21.15.46:0/1660811517 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3924066f30 0x7f39240693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f392c0099b0 tx=0x7f392c017000 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:54.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:54.836+0000 7f391ffff700 1 -- 172.21.15.46:0/1660811517 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3928050d80 con 0x7f39380b10f0 2024-01-26T23:42:55.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.124+0000 7f393e944700 1 -- 172.21.15.46:0/1660811517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7f393804c710 con 0x7f39380b10f0 2024-01-26T23:42:55.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.672+0000 7f391ffff700 1 -- 172.21.15.46:0/1660811517 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v471) v1 ==== 157+0+0 (secure 0 0 0) 0x7f3928017020 con 0x7f39380b10f0 2024-01-26T23:42:55.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.675+0000 7f393e944700 1 -- 172.21.15.46:0/1660811517 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3924066f30 msgr2=0x7f39240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:55.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.675+0000 7f393e944700 1 --2- 172.21.15.46:0/1660811517 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3924066f30 0x7f39240693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f392c0099b0 tx=0x7f392c017000 comp rx=0 tx=0).stop 2024-01-26T23:42:55.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.675+0000 7f393e944700 1 -- 172.21.15.46:0/1660811517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39380b10f0 msgr2=0x7f3938152ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:55.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.675+0000 7f393e944700 1 --2- 172.21.15.46:0/1660811517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39380b10f0 0x7f3938152ee0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f392800c5c0 tx=0x7f392800c980 comp rx=0 tx=0).stop 2024-01-26T23:42:55.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.675+0000 7f393e944700 1 -- 172.21.15.46:0/1660811517 shutdown_connections 2024-01-26T23:42:55.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.675+0000 7f393e944700 1 --2- 172.21.15.46:0/1660811517 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3924066f30 0x7f39240693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:55.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.675+0000 7f393e944700 1 --2- 172.21.15.46:0/1660811517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39380b10f0 0x7f3938152ee0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:55.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.675+0000 7f393e944700 1 --2- 172.21.15.46:0/1660811517 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3938159b30 0x7f393815c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:55.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.675+0000 7f393e944700 1 --2- 172.21.15.46:0/1660811517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3938159760 0x7f3938157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:55.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.675+0000 7f393e944700 1 -- 172.21.15.46:0/1660811517 >> 172.21.15.46:0/1660811517 conn(0x7f3938069510 msgr2=0x7f3938067930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:55.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.675+0000 7f393e944700 1 -- 172.21.15.46:0/1660811517 shutdown_connections 2024-01-26T23:42:55.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.675+0000 7f393e944700 1 -- 172.21.15.46:0/1660811517 wait complete. 2024-01-26T23:42:55.692 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-26T23:42:55.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.830+0000 7f3f0f297700 1 Processor -- start 2024-01-26T23:42:55.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.830+0000 7f3f0f297700 1 -- start start 2024-01-26T23:42:55.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.831+0000 7f3f0f297700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3f08157490 0x7f3f080cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:55.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.831+0000 7f3f0f297700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f0806ddb0 0x7f3f080cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:55.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.831+0000 7f3f0f297700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f080cea80 con 0x7f3f08157860 2024-01-26T23:42:55.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.831+0000 7f3f0f297700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f080cdc60 con 0x7f3f08157490 2024-01-26T23:42:55.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.831+0000 7f3f0f297700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f080b2ae0 con 0x7f3f0806ddb0 2024-01-26T23:42:55.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.831+0000 7f3f0c832700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3f08157860 0x7f3f0806d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39872/0 (socket says 172.21.15.46:39872) 2024-01-26T23:42:55.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.831+0000 7f3f0c832700 1 -- 172.21.15.46:0/307916402 learned_addr learned my addr 172.21.15.46:0/307916402 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:55.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.831+0000 7f3f0d033700 1 --2- 172.21.15.46:0/307916402 >> v2:172.21.15.106:3300/0 conn(0x7f3f08157490 0x7f3f080cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:55.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.831+0000 7f3f0d834700 1 --2- 172.21.15.46:0/307916402 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f0806ddb0 0x7f3f080cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:55.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.831+0000 7f3efffff700 1 -- 172.21.15.46:0/307916402 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454236188 0 0) 0x7f3f080cea80 con 0x7f3f08157860 2024-01-26T23:42:55.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.831+0000 7f3efffff700 1 -- 172.21.15.46:0/307916402 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ef4003570 con 0x7f3f08157860 2024-01-26T23:42:55.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3efffff700 1 -- 172.21.15.46:0/307916402 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3707557449 0 0) 0x7f3ef4003570 con 0x7f3f08157860 2024-01-26T23:42:55.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3efffff700 1 -- 172.21.15.46:0/307916402 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f0806ddb0 msgr2=0x7f3f080cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:55.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3efffff700 1 --2- 172.21.15.46:0/307916402 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f0806ddb0 0x7f3f080cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:55.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3efffff700 1 -- 172.21.15.46:0/307916402 >> v2:172.21.15.106:3300/0 conn(0x7f3f08157490 msgr2=0x7f3f080cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:55.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3efffff700 1 --2- 172.21.15.46:0/307916402 >> v2:172.21.15.106:3300/0 conn(0x7f3f08157490 0x7f3f080cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:55.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3efffff700 1 -- 172.21.15.46:0/307916402 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f080cde70 con 0x7f3f08157860 2024-01-26T23:42:55.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3efffff700 1 -- 172.21.15.46:0/307916402 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f3f04001470 con 0x7f3f08157860 2024-01-26T23:42:55.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3f0d033700 1 --2- 172.21.15.46:0/307916402 >> v2:172.21.15.106:3300/0 conn(0x7f3f08157490 0x7f3f080cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:55.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3efffff700 1 -- 172.21.15.46:0/307916402 >> v1:172.21.15.46:6789/0 conn(0x7f3f08157860 legacy=0x7f3f0806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:55.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3efffff700 1 --2- 172.21.15.46:0/307916402 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ef4003ae0 0x7f3ef4003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:55.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3efffff700 1 --2- 172.21.15.46:0/307916402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ef4004810 0x7f3ef4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:55.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3efffff700 1 --2- 172.21.15.46:0/307916402 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f0806ddb0 0x7f3ef40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:55.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3efffff700 1 -- 172.21.15.46:0/307916402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f080cdc60 con 0x7f3ef4004810 2024-01-26T23:42:55.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3efffff700 1 -- 172.21.15.46:0/307916402 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f080b2ae0 con 0x7f3ef4003ae0 2024-01-26T23:42:55.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3efffff700 1 -- 172.21.15.46:0/307916402 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef400bb10 con 0x7f3f0806ddb0 2024-01-26T23:42:55.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3f0c832700 1 --2- 172.21.15.46:0/307916402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ef4004810 0x7f3ef4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:55.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3f0d834700 1 --2- 172.21.15.46:0/307916402 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f0806ddb0 0x7f3ef40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3f0d033700 1 --2- 172.21.15.46:0/307916402 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ef4003ae0 0x7f3ef4003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3f0c832700 1 -- 172.21.15.46:0/307916402 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f0806ddb0 msgr2=0x7f3ef40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3f0c832700 1 --2- 172.21.15.46:0/307916402 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f0806ddb0 0x7f3ef40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3f0c832700 1 -- 172.21.15.46:0/307916402 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ef4003ae0 msgr2=0x7f3ef4003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3f0c832700 1 --2- 172.21.15.46:0/307916402 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ef4003ae0 0x7f3ef4003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3f0c832700 1 -- 172.21.15.46:0/307916402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef4003570 con 0x7f3ef4004810 2024-01-26T23:42:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.832+0000 7f3f0c832700 1 --2- 172.21.15.46:0/307916402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ef4004810 0x7f3ef4004c80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3f0400c2a0 tx=0x7f3f080ce9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3057a77554083d5a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.833+0000 7f3efffff700 1 -- 172.21.15.46:0/307916402 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0400cbd0 con 0x7f3ef4004810 2024-01-26T23:42:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.833+0000 7f3efffff700 1 -- 172.21.15.46:0/307916402 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0400cd30 con 0x7f3ef4004810 2024-01-26T23:42:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.833+0000 7f3efffff700 1 -- 172.21.15.46:0/307916402 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0400cea0 con 0x7f3ef4004810 2024-01-26T23:42:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.833+0000 7f3f0f297700 1 -- 172.21.15.46:0/307916402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ef4004810 msgr2=0x7f3ef4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.833+0000 7f3f0f297700 1 --2- 172.21.15.46:0/307916402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ef4004810 0x7f3ef4004c80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3f0400c2a0 tx=0x7f3f080ce9f0 comp rx=0 tx=0).stop 2024-01-26T23:42:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.833+0000 7f3f0d834700 1 -- 172.21.15.46:0/307916402 reap_dead start 2024-01-26T23:42:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.833+0000 7f3f0f297700 1 -- 172.21.15.46:0/307916402 shutdown_connections 2024-01-26T23:42:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.833+0000 7f3f0f297700 1 -- 172.21.15.46:0/307916402 >> 172.21.15.46:0/307916402 conn(0x7f3f08067930 msgr2=0x7f3f08067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.833+0000 7f3f0f297700 1 -- 172.21.15.46:0/307916402 shutdown_connections 2024-01-26T23:42:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.833+0000 7f3f0f297700 1 -- 172.21.15.46:0/307916402 wait complete. 2024-01-26T23:42:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.834+0000 7f3f0f297700 1 Processor -- start 2024-01-26T23:42:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.835+0000 7f3f0f297700 1 -- start start 2024-01-26T23:42:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.835+0000 7f3f0f297700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ef4004810 0x7f3f080c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.835+0000 7f3f0f297700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f08159fb0 0x7f3f0806a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.835+0000 7f3f0f297700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f0815a420 0x7f3f080adf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.835+0000 7f3f0f297700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f08147790 con 0x7f3f0815a420 2024-01-26T23:42:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.835+0000 7f3f0f297700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0816a980 con 0x7f3f08159fb0 2024-01-26T23:42:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.835+0000 7f3f0f297700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f08159b20 con 0x7f3ef4004810 2024-01-26T23:42:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.836+0000 7f3f0d834700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f0815a420 0x7f3f080adf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.836+0000 7f3f0d834700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f0815a420 0x7f3f080adf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56086/0 (socket says 172.21.15.46:56086) 2024-01-26T23:42:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.836+0000 7f3f0d834700 1 -- 172.21.15.46:0/1783886384 learned_addr learned my addr 172.21.15.46:0/1783886384 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.836+0000 7f3f0d033700 1 --2- 172.21.15.46:0/1783886384 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ef4004810 0x7f3f080c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.836+0000 7f3f0c832700 1 --2- 172.21.15.46:0/1783886384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f08159fb0 0x7f3f0806a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.836+0000 7f3f0d834700 1 -- 172.21.15.46:0/1783886384 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ef4004810 msgr2=0x7f3f080c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.836+0000 7f3f0d834700 1 --2- 172.21.15.46:0/1783886384 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ef4004810 0x7f3f080c9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.836+0000 7f3f0d834700 1 -- 172.21.15.46:0/1783886384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f08159fb0 msgr2=0x7f3f0806a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.836+0000 7f3f0d834700 1 --2- 172.21.15.46:0/1783886384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f08159fb0 0x7f3f0806a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.836+0000 7f3f0d834700 1 -- 172.21.15.46:0/1783886384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f080ca490 con 0x7f3f0815a420 2024-01-26T23:42:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.836+0000 7f3f0d834700 1 --2- 172.21.15.46:0/1783886384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f0815a420 0x7f3f080adf50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3f000094d0 tx=0x7f3f0000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.836+0000 7f3efd7fa700 1 -- 172.21.15.46:0/1783886384 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0000e040 con 0x7f3f0815a420 2024-01-26T23:42:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.836+0000 7f3efd7fa700 1 -- 172.21.15.46:0/1783886384 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0000f040 con 0x7f3f0815a420 2024-01-26T23:42:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.836+0000 7f3f0f297700 1 -- 172.21.15.46:0/1783886384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f08158f10 con 0x7f3f0815a420 2024-01-26T23:42:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.836+0000 7f3efd7fa700 1 -- 172.21.15.46:0/1783886384 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0000fe20 con 0x7f3f0815a420 2024-01-26T23:42:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.836+0000 7f3f0f297700 1 -- 172.21.15.46:0/1783886384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f081594d0 con 0x7f3f0815a420 2024-01-26T23:42:55.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.837+0000 7f3f0f297700 1 -- 172.21.15.46:0/1783886384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f08159cb0 con 0x7f3f0815a420 2024-01-26T23:42:55.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.838+0000 7f3efd7fa700 1 -- 172.21.15.46:0/1783886384 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3f00015640 con 0x7f3f0815a420 2024-01-26T23:42:55.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.838+0000 7f3efd7fa700 1 --2- 172.21.15.46:0/1783886384 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3eec066f60 0x7f3eec069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:55.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.838+0000 7f3efd7fa700 1 -- 172.21.15.46:0/1783886384 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6182+0+0 (secure 0 0 0) 0x7f3f00084ff0 con 0x7f3f0815a420 2024-01-26T23:42:55.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.838+0000 7f3f0d033700 1 --2- 172.21.15.46:0/1783886384 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3eec066f60 0x7f3eec069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:55.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.839+0000 7f3f0d033700 1 --2- 172.21.15.46:0/1783886384 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3eec066f60 0x7f3eec069410 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8006f00 tx=0x7f3ef8008040 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:55.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:55.843+0000 7f3efd7fa700 1 -- 172.21.15.46:0/1783886384 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3f00052080 con 0x7f3f0815a420 2024-01-26T23:42:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.052+0000 7f3f0f297700 1 -- 172.21.15.46:0/1783886384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3f0804c710 con 0x7f3f0815a420 2024-01-26T23:42:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.056+0000 7f3efd7fa700 1 -- 172.21.15.46:0/1783886384 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3f000556a0 con 0x7f3f0815a420 2024-01-26T23:42:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.058+0000 7f3f0f297700 1 -- 172.21.15.46:0/1783886384 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3eec066f60 msgr2=0x7f3eec069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.058+0000 7f3f0f297700 1 --2- 172.21.15.46:0/1783886384 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3eec066f60 0x7f3eec069410 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8006f00 tx=0x7f3ef8008040 comp rx=0 tx=0).stop 2024-01-26T23:42:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.058+0000 7f3f0f297700 1 -- 172.21.15.46:0/1783886384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f0815a420 msgr2=0x7f3f080adf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.058+0000 7f3f0f297700 1 --2- 172.21.15.46:0/1783886384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f0815a420 0x7f3f080adf50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3f000094d0 tx=0x7f3f0000b040 comp rx=0 tx=0).stop 2024-01-26T23:42:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.058+0000 7f3f0f297700 1 -- 172.21.15.46:0/1783886384 shutdown_connections 2024-01-26T23:42:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.058+0000 7f3f0f297700 1 --2- 172.21.15.46:0/1783886384 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3eec066f60 0x7f3eec069410 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.058+0000 7f3f0f297700 1 --2- 172.21.15.46:0/1783886384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f0815a420 0x7f3f080adf50 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.058+0000 7f3f0f297700 1 --2- 172.21.15.46:0/1783886384 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ef4004810 0x7f3f080c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.058+0000 7f3f0f297700 1 --2- 172.21.15.46:0/1783886384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f08159fb0 0x7f3f0806a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.058+0000 7f3f0f297700 1 -- 172.21.15.46:0/1783886384 >> 172.21.15.46:0/1783886384 conn(0x7f3f08067930 msgr2=0x7f3f080cd100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.058+0000 7f3f0f297700 1 -- 172.21.15.46:0/1783886384 shutdown_connections 2024-01-26T23:42:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.058+0000 7f3f0f297700 1 -- 172.21.15.46:0/1783886384 wait complete. 2024-01-26T23:42:56.073 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-26T23:42:56.073 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:42:56.121 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:42:56.175 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-26T23:42:56.176 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-26T23:42:56.176 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:42:56.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.311+0000 7f6100dbb700 1 Processor -- start 2024-01-26T23:42:56.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.311+0000 7f6100dbb700 1 -- start start 2024-01-26T23:42:56.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.312+0000 7f6100dbb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f60fc15ce40 0x7f60fc0d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:56.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.312+0000 7f6100dbb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f60fc0cc1c0 0x7f60fc0ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:56.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.312+0000 7f6100dbb700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60fc0cff70 con 0x7f60fc15d210 2024-01-26T23:42:56.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.312+0000 7f6100dbb700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60fc0cf400 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.312+0000 7f6100dbb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc0cf660 con 0x7f60fc15ce40 2024-01-26T23:42:56.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.312+0000 7f60fad9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f60fc0cc1c0 0x7f60fc0ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:56.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.312+0000 7f60fad9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f60fc0cc1c0 0x7f60fc0ade30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56096/0 (socket says 172.21.15.46:56096) 2024-01-26T23:42:56.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.312+0000 7f60fad9d700 1 -- 172.21.15.46:0/1040028955 learned_addr learned my addr 172.21.15.46:0/1040028955 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:56.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.312+0000 7f60fa59c700 1 --2- 172.21.15.46:0/1040028955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f60fc15ce40 0x7f60fc0d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:56.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.312+0000 7f60fad9d700 1 -- 172.21.15.46:0/1040028955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f60fc15ce40 msgr2=0x7f60fc0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:56.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.312+0000 7f60fad9d700 1 --2- 172.21.15.46:0/1040028955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f60fc15ce40 0x7f60fc0d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.312+0000 7f60fad9d700 1 -- 172.21.15.46:0/1040028955 >> v1:172.21.15.106:6789/0 conn(0x7f60fc15d210 legacy=0x7f60fc0cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:56.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.312+0000 7f60fad9d700 1 -- 172.21.15.46:0/1040028955 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60fc0d0500 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.312+0000 7f60fad9d700 1 --2- 172.21.15.46:0/1040028955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60fc0cc1c0 0x7f60fc0ade30 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f60f4009f20 tx=0x7f60f4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fc04646f18bfded server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:56.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.313+0000 7f60f959a700 1 -- 172.21.15.46:0/1040028955 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f400e070 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.313+0000 7f60f959a700 1 -- 172.21.15.46:0/1040028955 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f400ba80 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.313+0000 7f60f959a700 1 -- 172.21.15.46:0/1040028955 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f40092e0 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.313+0000 7f6100dbb700 1 -- 172.21.15.46:0/1040028955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60fc0cc1c0 msgr2=0x7f60fc0ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.313+0000 7f6100dbb700 1 --2- 172.21.15.46:0/1040028955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60fc0cc1c0 0x7f60fc0ade30 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f60f4009f20 tx=0x7f60f4009840 comp rx=0 tx=0).stop 2024-01-26T23:42:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.313+0000 7f6100dbb700 1 -- 172.21.15.46:0/1040028955 shutdown_connections 2024-01-26T23:42:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.313+0000 7f6100dbb700 1 --2- 172.21.15.46:0/1040028955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60fc0cc1c0 0x7f60fc0ade30 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.313+0000 7f6100dbb700 1 --2- 172.21.15.46:0/1040028955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f60fc15ce40 0x7f60fc0d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.313+0000 7f6100dbb700 1 -- 172.21.15.46:0/1040028955 >> 172.21.15.46:0/1040028955 conn(0x7f60fc06f840 msgr2=0x7f60fc0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.313+0000 7f6100dbb700 1 -- 172.21.15.46:0/1040028955 shutdown_connections 2024-01-26T23:42:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.313+0000 7f6100dbb700 1 -- 172.21.15.46:0/1040028955 wait complete. 2024-01-26T23:42:56.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.315+0000 7f6100dbb700 1 Processor -- start 2024-01-26T23:42:56.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.315+0000 7f6100dbb700 1 -- start start 2024-01-26T23:42:56.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.316+0000 7f6100dbb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60fc0cc1c0 0x7f60fc154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:56.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.316+0000 7f6100dbb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f60fc15ce40 0x7f60fc15f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:56.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.316+0000 7f6100dbb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f60fc15d210 0x7f60fc15f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:56.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.316+0000 7f6100dbb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc0c4a80 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.316+0000 7f6100dbb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc0c7500 con 0x7f60fc15ce40 2024-01-26T23:42:56.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.316+0000 7f6100dbb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc0c6c60 con 0x7f60fc15d210 2024-01-26T23:42:56.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.316+0000 7f60fa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60fc0cc1c0 0x7f60fc154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:56.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.316+0000 7f60fa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60fc0cc1c0 0x7f60fc154740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56108/0 (socket says 172.21.15.46:56108) 2024-01-26T23:42:56.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.316+0000 7f60fa59c700 1 -- 172.21.15.46:0/2938021241 learned_addr learned my addr 172.21.15.46:0/2938021241 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:56.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.316+0000 7f60fad9d700 1 --2- 172.21.15.46:0/2938021241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f60fc15d210 0x7f60fc15f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.316+0000 7f60f9d9b700 1 --2- 172.21.15.46:0/2938021241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f60fc15ce40 0x7f60fc15f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.317+0000 7f60fa59c700 1 -- 172.21.15.46:0/2938021241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f60fc15d210 msgr2=0x7f60fc15f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.317+0000 7f60fa59c700 1 --2- 172.21.15.46:0/2938021241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f60fc15d210 0x7f60fc15f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.317+0000 7f60fa59c700 1 -- 172.21.15.46:0/2938021241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f60fc15ce40 msgr2=0x7f60fc15f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.317+0000 7f60fa59c700 1 --2- 172.21.15.46:0/2938021241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f60fc15ce40 0x7f60fc15f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.317+0000 7f60fa59c700 1 -- 172.21.15.46:0/2938021241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f4009ae0 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.317+0000 7f60f9d9b700 1 --2- 172.21.15.46:0/2938021241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f60fc15ce40 0x7f60fc15f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.317+0000 7f60fad9d700 1 --2- 172.21.15.46:0/2938021241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f60fc15d210 0x7f60fc15f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.317+0000 7f60fa59c700 1 --2- 172.21.15.46:0/2938021241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60fc0cc1c0 0x7f60fc154740 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f60ec00d810 tx=0x7f60ec008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.317+0000 7f60ea7fc700 1 -- 172.21.15.46:0/2938021241 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec0125f0 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.317+0000 7f60ea7fc700 1 -- 172.21.15.46:0/2938021241 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ec012750 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.317+0000 7f6100dbb700 1 -- 172.21.15.46:0/2938021241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60fc0c6f40 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.317+0000 7f60ea7fc700 1 -- 172.21.15.46:0/2938021241 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec012b10 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.317+0000 7f6100dbb700 1 -- 172.21.15.46:0/2938021241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60fc1566c0 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.318+0000 7f6100dbb700 1 -- 172.21.15.46:0/2938021241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60fc156a30 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.318+0000 7f60ea7fc700 1 -- 172.21.15.46:0/2938021241 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f60ec01e460 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.319+0000 7f60ea7fc700 1 --2- 172.21.15.46:0/2938021241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f60dc066f30 0x7f60dc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.319+0000 7f60ea7fc700 1 -- 172.21.15.46:0/2938021241 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6182+0+0 (secure 0 0 0) 0x7f60ec084d60 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.319+0000 7f60f9d9b700 1 --2- 172.21.15.46:0/2938021241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f60dc066f30 0x7f60dc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.320+0000 7f60f9d9b700 1 --2- 172.21.15.46:0/2938021241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f60dc066f30 0x7f60dc0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f60f0007fd0 tx=0x7f60f0009040 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:56.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.323+0000 7f60ea7fc700 1 -- 172.21.15.46:0/2938021241 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f60ec051df0 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.575+0000 7f6100dbb700 1 -- 172.21.15.46:0/2938021241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60fc0cf920 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.575+0000 7f60ea7fc700 1 -- 172.21.15.46:0/2938021241 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+3532 (secure 0 0 0) 0x7f60ec055410 con 0x7f60fc0cc1c0 2024-01-26T23:42:56.577 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:56.577 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1737713978","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1737713978},{"type":"v1","addr":"172.21.15.46:6835","nonce":1737713978}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":4969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3857864387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3857864387},{"type":"v1","addr":"172.21.15.106:6836","nonce":3857864387}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":4972,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/639000251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":639000251},{"type":"v1","addr":"172.21.15.106:6837","nonce":639000251}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5748,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4243883126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4243883126},{"type":"v1","addr":"172.21.15.46:6834","nonce":4243883126}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[]} 2024-01-26T23:42:56.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.577+0000 7f6100dbb700 1 -- 172.21.15.46:0/2938021241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f60dc066f30 msgr2=0x7f60dc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:56.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.577+0000 7f6100dbb700 1 --2- 172.21.15.46:0/2938021241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f60dc066f30 0x7f60dc0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f60f0007fd0 tx=0x7f60f0009040 comp rx=0 tx=0).stop 2024-01-26T23:42:56.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.577+0000 7f6100dbb700 1 -- 172.21.15.46:0/2938021241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60fc0cc1c0 msgr2=0x7f60fc154740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:56.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.577+0000 7f6100dbb700 1 --2- 172.21.15.46:0/2938021241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60fc0cc1c0 0x7f60fc154740 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f60ec00d810 tx=0x7f60ec008040 comp rx=0 tx=0).stop 2024-01-26T23:42:56.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.577+0000 7f6100dbb700 1 -- 172.21.15.46:0/2938021241 shutdown_connections 2024-01-26T23:42:56.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.577+0000 7f6100dbb700 1 --2- 172.21.15.46:0/2938021241 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f60dc066f30 0x7f60dc0693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.577+0000 7f6100dbb700 1 --2- 172.21.15.46:0/2938021241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60fc0cc1c0 0x7f60fc154740 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.577+0000 7f6100dbb700 1 --2- 172.21.15.46:0/2938021241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f60fc15d210 0x7f60fc15f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.577+0000 7f6100dbb700 1 --2- 172.21.15.46:0/2938021241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f60fc15ce40 0x7f60fc15f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.577+0000 7f6100dbb700 1 -- 172.21.15.46:0/2938021241 >> 172.21.15.46:0/2938021241 conn(0x7f60fc06f840 msgr2=0x7f60fc15b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:56.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.578+0000 7f6100dbb700 1 -- 172.21.15.46:0/2938021241 shutdown_connections 2024-01-26T23:42:56.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.578+0000 7f6100dbb700 1 -- 172.21.15.46:0/2938021241 wait complete. 2024-01-26T23:42:56.580 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 20 2024-01-26T23:42:56.593 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-26T23:42:56.593 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-26T23:42:56.593 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-26T23:42:56.593 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:42:56.593 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-26T23:42:56.594 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-26T23:42:56.594 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-26T23:42:56.594 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:42:56.594 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-26T23:42:56.594 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-26T23:42:56.594 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-26T23:42:56.594 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:42:56.595 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-26T23:42:56.595 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-26T23:42:56.595 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-26T23:42:56.595 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:42:56.603 INFO:tasks.ceph.mds.a:Stopped 2024-01-26T23:42:56.604 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-26T23:42:56.604 INFO:tasks.ceph.mds.c:Stopped 2024-01-26T23:42:56.605 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-26T23:42:56.606 INFO:tasks.ceph.mds.d:Stopped 2024-01-26T23:42:56.606 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-26T23:42:56.607 INFO:tasks.ceph.mds.b:Stopped 2024-01-26T23:42:56.607 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-26T23:42:56.646 INFO:tasks.ceph.mds.a:Started 2024-01-26T23:42:56.647 DEBUG:teuthology.parallel:result is None 2024-01-26T23:42:56.648 INFO:tasks.ceph.mds.c:Started 2024-01-26T23:42:56.648 DEBUG:teuthology.parallel:result is None 2024-01-26T23:42:56.649 INFO:tasks.ceph.mds.d:Started 2024-01-26T23:42:56.649 DEBUG:teuthology.parallel:result is None 2024-01-26T23:42:56.650 INFO:tasks.ceph.mds.b:Started 2024-01-26T23:42:56.650 DEBUG:teuthology.parallel:result is None 2024-01-26T23:42:56.650 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-26T23:42:56.734 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-26T23:42:56.735 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-01-26T23:42:56.735 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-26T23:42:56.736 INFO:tasks.ceph.mds.b.smithi106.stdout:starting mds.b at 2024-01-26T23:42:56.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.791+0000 7f04dd7ea700 1 Processor -- start 2024-01-26T23:42:56.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.791+0000 7f04dd7ea700 1 -- start start 2024-01-26T23:42:56.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.792+0000 7f04dd7ea700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f04d80b2b40 0x7f04d80ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:56.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.792+0000 7f04dd7ea700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f04d80ae960 0x7f04d80ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:56.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.792+0000 7f04dd7ea700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04d80aedd0 0x7f04d80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:56.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.792+0000 7f04dd7ea700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04d8099fc0 con 0x7f04d80ae960 2024-01-26T23:42:56.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.792+0000 7f04dd7ea700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04d815a0b0 con 0x7f04d80b2b40 2024-01-26T23:42:56.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.792+0000 7f04dd7ea700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d80ae4d0 con 0x7f04d80aedd0 2024-01-26T23:42:56.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.792+0000 7f04d67fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f04d80ae960 0x7f04d80ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:56.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.792+0000 7f04d67fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f04d80ae960 0x7f04d80ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56156/0 (socket says 172.21.15.46:56156) 2024-01-26T23:42:56.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.792+0000 7f04d6ffd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f04d80b2b40 0x7f04d80ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:56.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.792+0000 7f04d77fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04d80aedd0 0x7f04d80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.792+0000 7f04d67fc700 1 -- 172.21.15.46:0/1624028589 learned_addr learned my addr 172.21.15.46:0/1624028589 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.792+0000 7f04d77fe700 1 -- 172.21.15.46:0/1624028589 >> v2:172.21.15.106:3300/0 conn(0x7f04d80b2b40 msgr2=0x7f04d80ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.792+0000 7f04d77fe700 1 --2- 172.21.15.46:0/1624028589 >> v2:172.21.15.106:3300/0 conn(0x7f04d80b2b40 0x7f04d80ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.792+0000 7f04d77fe700 1 -- 172.21.15.46:0/1624028589 >> v2:172.21.15.46:3300/0 conn(0x7f04d80ae960 msgr2=0x7f04d80ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.792+0000 7f04d77fe700 1 --2- 172.21.15.46:0/1624028589 >> v2:172.21.15.46:3300/0 conn(0x7f04d80ae960 0x7f04d80ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.792+0000 7f04d77fe700 1 -- 172.21.15.46:0/1624028589 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d80ad8c0 con 0x7f04d80aedd0 2024-01-26T23:42:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.792+0000 7f04d67fc700 1 --2- 172.21.15.46:0/1624028589 >> v2:172.21.15.46:3300/0 conn(0x7f04d80ae960 0x7f04d80ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.793+0000 7f04d77fe700 1 --2- 172.21.15.46:0/1624028589 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04d80aedd0 0x7f04d80cf290 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f04d8088850 tx=0x7f04d0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9aa25415c3f9500 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.793+0000 7f04d5ffb700 1 -- 172.21.15.46:0/1624028589 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d000e070 con 0x7f04d80aedd0 2024-01-26T23:42:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.793+0000 7f04d5ffb700 1 -- 172.21.15.46:0/1624028589 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d000bab0 con 0x7f04d80aedd0 2024-01-26T23:42:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.793+0000 7f04d5ffb700 1 -- 172.21.15.46:0/1624028589 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d000be70 con 0x7f04d80aedd0 2024-01-26T23:42:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.793+0000 7f04dd7ea700 1 -- 172.21.15.46:0/1624028589 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04d80aedd0 msgr2=0x7f04d80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.793+0000 7f04dd7ea700 1 --2- 172.21.15.46:0/1624028589 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04d80aedd0 0x7f04d80cf290 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f04d8088850 tx=0x7f04d0009780 comp rx=0 tx=0).stop 2024-01-26T23:42:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.793+0000 7f04dd7ea700 1 -- 172.21.15.46:0/1624028589 shutdown_connections 2024-01-26T23:42:56.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.793+0000 7f04dd7ea700 1 --2- 172.21.15.46:0/1624028589 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04d80aedd0 0x7f04d80cf290 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.793+0000 7f04dd7ea700 1 --2- 172.21.15.46:0/1624028589 >> v2:172.21.15.106:3300/0 conn(0x7f04d80b2b40 0x7f04d80ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.793+0000 7f04dd7ea700 1 --2- 172.21.15.46:0/1624028589 >> v2:172.21.15.46:3300/0 conn(0x7f04d80ae960 0x7f04d80ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.793+0000 7f04dd7ea700 1 -- 172.21.15.46:0/1624028589 >> 172.21.15.46:0/1624028589 conn(0x7f04d8099a30 msgr2=0x7f04d806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:56.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.793+0000 7f04dd7ea700 1 -- 172.21.15.46:0/1624028589 shutdown_connections 2024-01-26T23:42:56.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.793+0000 7f04dd7ea700 1 -- 172.21.15.46:0/1624028589 wait complete. 2024-01-26T23:42:56.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.795+0000 7f04dd7ea700 1 Processor -- start 2024-01-26T23:42:56.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.795+0000 7f04dd7ea700 1 -- start start 2024-01-26T23:42:56.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04dd7ea700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04d80ae960 0x7f04d814b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04dd7ea700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04d80b2b40 0x7f04d80c8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04dd7ea700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04d80afd40 0x7f04d80b10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04dd7ea700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d80b6370 con 0x7f04d80afd40 2024-01-26T23:42:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04dd7ea700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d80b4030 con 0x7f04d80ae960 2024-01-26T23:42:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04dd7ea700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d80c8ad0 con 0x7f04d80b2b40 2024-01-26T23:42:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04d77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04d80afd40 0x7f04d80b10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04d77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04d80afd40 0x7f04d80b10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56172/0 (socket says 172.21.15.46:56172) 2024-01-26T23:42:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04d77fe700 1 -- 172.21.15.46:0/3723907480 learned_addr learned my addr 172.21.15.46:0/3723907480 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04d67fc700 1 --2- 172.21.15.46:0/3723907480 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04d80b2b40 0x7f04d80c8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04d6ffd700 1 --2- 172.21.15.46:0/3723907480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04d80ae960 0x7f04d814b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04d77fe700 1 -- 172.21.15.46:0/3723907480 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04d80b2b40 msgr2=0x7f04d80c8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04d77fe700 1 --2- 172.21.15.46:0/3723907480 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04d80b2b40 0x7f04d80c8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04d77fe700 1 -- 172.21.15.46:0/3723907480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04d80ae960 msgr2=0x7f04d814b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04d77fe700 1 --2- 172.21.15.46:0/3723907480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04d80ae960 0x7f04d814b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:56.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04d77fe700 1 -- 172.21.15.46:0/3723907480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c00094f0 con 0x7f04d80afd40 2024-01-26T23:42:56.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04d6ffd700 1 --2- 172.21.15.46:0/3723907480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04d80ae960 0x7f04d814b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:42:56.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.796+0000 7f04d77fe700 1 --2- 172.21.15.46:0/3723907480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04d80afd40 0x7f04d80b10e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f04d0013000 tx=0x7f04d0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:56.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.797+0000 7f04ceffd700 1 -- 172.21.15.46:0/3723907480 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0027470 con 0x7f04d80afd40 2024-01-26T23:42:56.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.797+0000 7f04ceffd700 1 -- 172.21.15.46:0/3723907480 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d00275f0 con 0x7f04d80afd40 2024-01-26T23:42:56.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.797+0000 7f04ceffd700 1 -- 172.21.15.46:0/3723907480 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d00279b0 con 0x7f04d80afd40 2024-01-26T23:42:56.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.797+0000 7f04dd7ea700 1 -- 172.21.15.46:0/3723907480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d0009a20 con 0x7f04d80afd40 2024-01-26T23:42:56.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.797+0000 7f04dd7ea700 1 -- 172.21.15.46:0/3723907480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d80b6090 con 0x7f04d80afd40 2024-01-26T23:42:56.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.798+0000 7f04dd7ea700 1 -- 172.21.15.46:0/3723907480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d804c710 con 0x7f04d80afd40 2024-01-26T23:42:56.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.798+0000 7f04ceffd700 1 -- 172.21.15.46:0/3723907480 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f04d0024070 con 0x7f04d80afd40 2024-01-26T23:42:56.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.799+0000 7f04ceffd700 1 --2- 172.21.15.46:0/3723907480 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f04b8066f30 0x7f04b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:56.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.799+0000 7f04ceffd700 1 -- 172.21.15.46:0/3723907480 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6182+0+0 (secure 0 0 0) 0x7f04d0087090 con 0x7f04d80afd40 2024-01-26T23:42:56.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.799+0000 7f04d6ffd700 1 --2- 172.21.15.46:0/3723907480 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f04b8066f30 0x7f04b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:56.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.799+0000 7f04d6ffd700 1 --2- 172.21.15.46:0/3723907480 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f04b8066f30 0x7f04b80693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f04c8006ea0 tx=0x7f04c8008040 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:56.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:56.803+0000 7f04ceffd700 1 -- 172.21.15.46:0/3723907480 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f04d0054120 con 0x7f04d80afd40 2024-01-26T23:42:57.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.027+0000 7f04dd7ea700 1 -- 172.21.15.46:0/3723907480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f04d80b4170 con 0x7f04d80afd40 2024-01-26T23:42:57.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.690+0000 7f04ceffd700 1 -- 172.21.15.46:0/3723907480 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v64) v1 ==== 97+0+0 (secure 0 0 0) 0x7f04d0057740 con 0x7f04d80afd40 2024-01-26T23:42:57.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.693+0000 7f04dd7ea700 1 -- 172.21.15.46:0/3723907480 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f04b8066f30 msgr2=0x7f04b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:57.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.693+0000 7f04dd7ea700 1 --2- 172.21.15.46:0/3723907480 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f04b8066f30 0x7f04b80693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f04c8006ea0 tx=0x7f04c8008040 comp rx=0 tx=0).stop 2024-01-26T23:42:57.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.693+0000 7f04dd7ea700 1 -- 172.21.15.46:0/3723907480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04d80afd40 msgr2=0x7f04d80b10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:57.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.693+0000 7f04dd7ea700 1 --2- 172.21.15.46:0/3723907480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04d80afd40 0x7f04d80b10e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f04d0013000 tx=0x7f04d0019000 comp rx=0 tx=0).stop 2024-01-26T23:42:57.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.693+0000 7f04dd7ea700 1 -- 172.21.15.46:0/3723907480 shutdown_connections 2024-01-26T23:42:57.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.693+0000 7f04dd7ea700 1 --2- 172.21.15.46:0/3723907480 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f04b8066f30 0x7f04b80693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:57.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.693+0000 7f04dd7ea700 1 --2- 172.21.15.46:0/3723907480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04d80afd40 0x7f04d80b10e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:57.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.693+0000 7f04dd7ea700 1 --2- 172.21.15.46:0/3723907480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04d80ae960 0x7f04d814b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:57.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.693+0000 7f04dd7ea700 1 --2- 172.21.15.46:0/3723907480 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04d80b2b40 0x7f04d80c8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:57.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.693+0000 7f04dd7ea700 1 -- 172.21.15.46:0/3723907480 >> 172.21.15.46:0/3723907480 conn(0x7f04d8099a30 msgr2=0x7f04d8158e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:57.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.694+0000 7f04dd7ea700 1 -- 172.21.15.46:0/3723907480 shutdown_connections 2024-01-26T23:42:57.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.694+0000 7f04dd7ea700 1 -- 172.21.15.46:0/3723907480 wait complete. 2024-01-26T23:42:57.696 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-01-26T23:42:57.712 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-26T23:42:57.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.847+0000 7f06516e6700 1 Processor -- start 2024-01-26T23:42:57.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.847+0000 7f06516e6700 1 -- start start 2024-01-26T23:42:57.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f06516e6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f064c0b2b40 0x7f064c0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:57.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f06516e6700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f064c0ae960 0x7f064c0ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:57.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f06516e6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f064c0aedd0 0x7f064c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:57.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f06516e6700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f064c099fc0 con 0x7f064c0b2b40 2024-01-26T23:42:57.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f06516e6700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f064c15a0b0 con 0x7f064c0ae960 2024-01-26T23:42:57.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f06516e6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f064c0ae4d0 con 0x7f064c0aedd0 2024-01-26T23:42:57.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f064affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f064c0b2b40 0x7f064c0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:57.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f064affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f064c0b2b40 0x7f064c0ad1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56176/0 (socket says 172.21.15.46:56176) 2024-01-26T23:42:57.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f064affd700 1 -- 172.21.15.46:0/404306749 learned_addr learned my addr 172.21.15.46:0/404306749 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:57.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f064b7fe700 1 --2- 172.21.15.46:0/404306749 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f064c0aedd0 0x7f064c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:57.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f064a7fc700 1 --2- 172.21.15.46:0/404306749 >> v2:172.21.15.106:3300/0 conn(0x7f064c0ae960 0x7f064c0ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:57.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f064affd700 1 -- 172.21.15.46:0/404306749 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f064c0aedd0 msgr2=0x7f064c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:57.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f064affd700 1 --2- 172.21.15.46:0/404306749 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f064c0aedd0 0x7f064c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:57.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f064affd700 1 -- 172.21.15.46:0/404306749 >> v2:172.21.15.106:3300/0 conn(0x7f064c0ae960 msgr2=0x7f064c0ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:57.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f064affd700 1 --2- 172.21.15.46:0/404306749 >> v2:172.21.15.106:3300/0 conn(0x7f064c0ae960 0x7f064c0ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:57.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f064affd700 1 -- 172.21.15.46:0/404306749 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064c0ad8c0 con 0x7f064c0b2b40 2024-01-26T23:42:57.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f064b7fe700 1 --2- 172.21.15.46:0/404306749 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f064c0aedd0 0x7f064c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:42:57.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.848+0000 7f064a7fc700 1 --2- 172.21.15.46:0/404306749 >> v2:172.21.15.106:3300/0 conn(0x7f064c0ae960 0x7f064c0ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:42:57.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.849+0000 7f064affd700 1 --2- 172.21.15.46:0/404306749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f064c0b2b40 0x7f064c0ad1c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f064c0ce2f0 tx=0x7f063c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf551109d65803f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:57.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.849+0000 7f0649ffb700 1 -- 172.21.15.46:0/404306749 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c005c10 con 0x7f064c0b2b40 2024-01-26T23:42:57.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.849+0000 7f0649ffb700 1 -- 172.21.15.46:0/404306749 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f063c005d70 con 0x7f064c0b2b40 2024-01-26T23:42:57.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.849+0000 7f0649ffb700 1 -- 172.21.15.46:0/404306749 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c00b880 con 0x7f064c0b2b40 2024-01-26T23:42:57.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.849+0000 7f06516e6700 1 -- 172.21.15.46:0/404306749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f064c0b2b40 msgr2=0x7f064c0ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:57.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.849+0000 7f06516e6700 1 --2- 172.21.15.46:0/404306749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f064c0b2b40 0x7f064c0ad1c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f064c0ce2f0 tx=0x7f063c009220 comp rx=0 tx=0).stop 2024-01-26T23:42:57.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.849+0000 7f06516e6700 1 -- 172.21.15.46:0/404306749 shutdown_connections 2024-01-26T23:42:57.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.849+0000 7f06516e6700 1 --2- 172.21.15.46:0/404306749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f064c0b2b40 0x7f064c0ad1c0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:57.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.849+0000 7f06516e6700 1 --2- 172.21.15.46:0/404306749 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f064c0aedd0 0x7f064c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:57.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.849+0000 7f06516e6700 1 --2- 172.21.15.46:0/404306749 >> v2:172.21.15.106:3300/0 conn(0x7f064c0ae960 0x7f064c0ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:57.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.849+0000 7f06516e6700 1 -- 172.21.15.46:0/404306749 >> 172.21.15.46:0/404306749 conn(0x7f064c099a30 msgr2=0x7f064c06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:57.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.849+0000 7f06516e6700 1 -- 172.21.15.46:0/404306749 shutdown_connections 2024-01-26T23:42:57.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.850+0000 7f06516e6700 1 -- 172.21.15.46:0/404306749 wait complete. 2024-01-26T23:42:57.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.851+0000 7f06516e6700 1 Processor -- start 2024-01-26T23:42:57.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.851+0000 7f06516e6700 1 -- start start 2024-01-26T23:42:57.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f06516e6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f064c0ae960 0x7f064c0af430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:57.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f06516e6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f064c0b2b40 0x7f064c1472d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:57.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f06516e6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f064c0c9420 0x7f064c0c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:57.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f06516e6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c1573a0 con 0x7f064c0b2b40 2024-01-26T23:42:57.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f06516e6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c0ca6e0 con 0x7f064c0ae960 2024-01-26T23:42:57.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f06516e6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f064c0ca100 con 0x7f064c0c9420 2024-01-26T23:42:57.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f064a7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f064c0b2b40 0x7f064c1472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:57.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f064a7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f064c0b2b40 0x7f064c1472d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56180/0 (socket says 172.21.15.46:56180) 2024-01-26T23:42:57.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f064a7fc700 1 -- 172.21.15.46:0/586443406 learned_addr learned my addr 172.21.15.46:0/586443406 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:57.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f064b7fe700 1 --2- 172.21.15.46:0/586443406 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f064c0c9420 0x7f064c0c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:57.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f064affd700 1 --2- 172.21.15.46:0/586443406 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f064c0ae960 0x7f064c0af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:57.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f064a7fc700 1 -- 172.21.15.46:0/586443406 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f064c0c9420 msgr2=0x7f064c0c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:57.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f064a7fc700 1 --2- 172.21.15.46:0/586443406 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f064c0c9420 0x7f064c0c98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:57.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f064a7fc700 1 -- 172.21.15.46:0/586443406 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f064c0ae960 msgr2=0x7f064c0af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:57.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f064a7fc700 1 --2- 172.21.15.46:0/586443406 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f064c0ae960 0x7f064c0af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:57.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f064a7fc700 1 -- 172.21.15.46:0/586443406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063c009480 con 0x7f064c0b2b40 2024-01-26T23:42:57.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f064b7fe700 1 --2- 172.21.15.46:0/586443406 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f064c0c9420 0x7f064c0c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:57.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f064affd700 1 --2- 172.21.15.46:0/586443406 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f064c0ae960 0x7f064c0af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:57.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.852+0000 7f064a7fc700 1 --2- 172.21.15.46:0/586443406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f064c0b2b40 0x7f064c1472d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f064000c8b0 tx=0x7f0640008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:57.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.853+0000 7f0632ffd700 1 -- 172.21.15.46:0/586443406 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0640011600 con 0x7f064c0b2b40 2024-01-26T23:42:57.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.853+0000 7f0632ffd700 1 -- 172.21.15.46:0/586443406 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0640011760 con 0x7f064c0b2b40 2024-01-26T23:42:57.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.853+0000 7f06516e6700 1 -- 172.21.15.46:0/586443406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f064c0c7b70 con 0x7f064c0b2b40 2024-01-26T23:42:57.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.853+0000 7f0632ffd700 1 -- 172.21.15.46:0/586443406 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0640011b20 con 0x7f064c0b2b40 2024-01-26T23:42:57.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.853+0000 7f06516e6700 1 -- 172.21.15.46:0/586443406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f064c0ca420 con 0x7f064c0b2b40 2024-01-26T23:42:57.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.854+0000 7f06516e6700 1 -- 172.21.15.46:0/586443406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f064c0ca240 con 0x7f064c0b2b40 2024-01-26T23:42:57.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.854+0000 7f0632ffd700 1 -- 172.21.15.46:0/586443406 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f06400193b0 con 0x7f064c0b2b40 2024-01-26T23:42:57.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.854+0000 7f0632ffd700 1 --2- 172.21.15.46:0/586443406 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0638066f30 0x7f06380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:57.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.854+0000 7f0632ffd700 1 -- 172.21.15.46:0/586443406 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5537+0+0 (secure 0 0 0) 0x7f0640084400 con 0x7f064c0b2b40 2024-01-26T23:42:57.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.855+0000 7f064affd700 1 --2- 172.21.15.46:0/586443406 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0638066f30 0x7f06380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:57.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.855+0000 7f064affd700 1 --2- 172.21.15.46:0/586443406 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0638066f30 0x7f06380693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f063c009260 tx=0x7f063c016000 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:57.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:57.859+0000 7f0632ffd700 1 -- 172.21.15.46:0/586443406 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f06400551a0 con 0x7f064c0b2b40 2024-01-26T23:42:58.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.145+0000 7f06516e6700 1 -- 172.21.15.46:0/586443406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f064c04c710 con 0x7f064c0b2b40 2024-01-26T23:42:58.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.145+0000 7f0632ffd700 1 -- 172.21.15.46:0/586443406 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0640054d30 con 0x7f064c0b2b40 2024-01-26T23:42:58.147 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:58.147 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:42:58.147 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-01-26T23:42:58.147 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.147 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-01-26T23:42:58.147 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlM6dKKBAAPpeWeDZWrFgc/4zuxKCvHw==", 2024-01-26T23:42:58.147 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.147 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:42:58.147 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:42:58.147 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:42:58.148 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:42:58.148 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.148 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.148 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.148 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-01-26T23:42:58.148 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlmzHRNRAAflWYZ/p4AXeMy54tvLpugA==", 2024-01-26T23:42:58.148 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.148 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:42:58.148 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:42:58.148 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:42:58.148 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:42:58.148 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.148 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.148 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.148 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-01-26T23:42:58.149 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlMZxoMBAAG5kyfm+LLG86y/p2A8BfDQ==", 2024-01-26T23:42:58.149 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.149 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:42:58.149 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:42:58.149 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:42:58.149 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:42:58.149 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.149 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.149 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.149 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-01-26T23:42:58.149 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl5jx/ARAA3cwtLAOWQ/n2NG+xwEZ3QQ==", 2024-01-26T23:42:58.149 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.149 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:42:58.149 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:42:58.149 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:42:58.149 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:42:58.150 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.150 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.150 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.150 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-01-26T23:42:58.150 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4QLRlJ2OKJRAAWVhxEdQkUW20294w/KLL6g==", 2024-01-26T23:42:58.150 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.150 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:42:58.150 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:42:58.150 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:42:58.150 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.150 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.150 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.150 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-01-26T23:42:58.150 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7QLRlE2tAJRAAzNlTiCr6a2eWiabRy1SqSw==", 2024-01-26T23:42:58.150 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.151 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:42:58.151 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:42:58.151 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:42:58.151 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.151 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.151 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.151 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-01-26T23:42:58.151 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+QLRlyySaJBAA5072ba5HUNi2dYE3D02M/Q==", 2024-01-26T23:42:58.151 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.151 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:42:58.151 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:42:58.151 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:42:58.151 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.151 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.151 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.151 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-01-26T23:42:58.152 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBQLRl9+tAIxAApZk0btXIpQPWSb3BYUYsFQ==", 2024-01-26T23:42:58.152 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.152 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:42:58.152 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:42:58.152 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:42:58.152 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.152 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.152 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.152 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-01-26T23:42:58.152 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHQLRlurUtJxAAT9fwKrAWl62VOXwShg7NJg==", 2024-01-26T23:42:58.152 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.152 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:42:58.152 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:42:58.152 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:42:58.153 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.153 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.153 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.153 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-01-26T23:42:58.153 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKQLRl7aqoJRAAXu9fykWELsW3SwnahoIOLA==", 2024-01-26T23:42:58.153 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.153 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:42:58.153 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:42:58.153 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:42:58.153 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.153 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.153 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.153 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-01-26T23:42:58.153 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNQLRlWf7pJBAAopmFLIKb8s32YZs46VwbGg==", 2024-01-26T23:42:58.153 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.154 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:42:58.154 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:42:58.154 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:42:58.154 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.154 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.154 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.154 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-01-26T23:42:58.154 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQQLRlNGfhIxAAmCgmOB3QAGfSIYHBb5BUZw==", 2024-01-26T23:42:58.154 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.154 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:42:58.154 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:42:58.154 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:42:58.155 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.155 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.155 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.155 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-01-26T23:42:58.155 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRleO8wBhAA1018AcO3KcyOZxgB0Itxaw==", 2024-01-26T23:42:58.155 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.155 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:42:58.155 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-01-26T23:42:58.155 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-26T23:42:58.155 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.155 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.155 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.155 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-01-26T23:42:58.155 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlsiZEDRAAjXSamHe2bZcDqVsqXqIcGg==", 2024-01-26T23:42:58.156 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.156 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:42:58.156 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:42:58.156 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:42:58.156 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:42:58.156 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.156 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.156 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.156 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-01-26T23:42:58.156 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl6YT1CRAApXlN4j20xzwda0tDMy94Eg==", 2024-01-26T23:42:58.156 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.156 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:42:58.156 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:42:58.156 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:42:58.157 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:42:58.157 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.157 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.157 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.157 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-01-26T23:42:58.157 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlHvDQEhAAxMhgWJsZC5/6Af5zFV6XIA==", 2024-01-26T23:42:58.157 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.157 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:42:58.157 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:42:58.157 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:42:58.158 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:42:58.158 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.158 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.158 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.158 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-01-26T23:42:58.158 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlvMwDCBAAXnFyMduz7PfWdLWbjytIiQ==", 2024-01-26T23:42:58.158 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.158 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:42:58.158 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:42:58.158 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:42:58.158 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:42:58.159 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.159 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.159 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.159 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-01-26T23:42:58.159 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlhMNSJBAA38QU6DrCvFjO+u9zQ8bkEA==", 2024-01-26T23:42:58.159 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.159 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:42:58.159 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-26T23:42:58.159 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:42:58.159 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.160 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:42:58.160 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:58.160 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-01-26T23:42:58.160 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlI9EbIBAACTpzAO+8A/VFJZcT/WWBng==", 2024-01-26T23:42:58.160 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:42:58.160 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:42:58.160 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-26T23:42:58.160 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:42:58.160 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.160 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:58.160 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-01-26T23:42:58.160 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:42:58.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.148+0000 7f06516e6700 1 -- 172.21.15.46:0/586443406 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0638066f30 msgr2=0x7f06380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.148+0000 7f06516e6700 1 --2- 172.21.15.46:0/586443406 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0638066f30 0x7f06380693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f063c009260 tx=0x7f063c016000 comp rx=0 tx=0).stop 2024-01-26T23:42:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.148+0000 7f06516e6700 1 -- 172.21.15.46:0/586443406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f064c0b2b40 msgr2=0x7f064c1472d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.148+0000 7f06516e6700 1 --2- 172.21.15.46:0/586443406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f064c0b2b40 0x7f064c1472d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f064000c8b0 tx=0x7f0640008040 comp rx=0 tx=0).stop 2024-01-26T23:42:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.148+0000 7f06516e6700 1 -- 172.21.15.46:0/586443406 shutdown_connections 2024-01-26T23:42:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.148+0000 7f06516e6700 1 --2- 172.21.15.46:0/586443406 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0638066f30 0x7f06380693e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.148+0000 7f06516e6700 1 --2- 172.21.15.46:0/586443406 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f064c0c9420 0x7f064c0c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.148+0000 7f06516e6700 1 --2- 172.21.15.46:0/586443406 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f064c0ae960 0x7f064c0af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.148+0000 7f06516e6700 1 --2- 172.21.15.46:0/586443406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f064c0b2b40 0x7f064c1472d0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.148+0000 7f06516e6700 1 -- 172.21.15.46:0/586443406 >> 172.21.15.46:0/586443406 conn(0x7f064c099a30 msgr2=0x7f064c0cefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.148+0000 7f06516e6700 1 -- 172.21.15.46:0/586443406 shutdown_connections 2024-01-26T23:42:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.148+0000 7f06516e6700 1 -- 172.21.15.46:0/586443406 wait complete. 2024-01-26T23:42:58.164 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:42:58.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.294+0000 7f1ebc907700 1 Processor -- start 2024-01-26T23:42:58.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.294+0000 7f1ebc907700 1 -- start start 2024-01-26T23:42:58.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.295+0000 7f1ebc907700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb8159b30 0x7f1eb80b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:58.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.295+0000 7f1ebc907700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb806f940 con 0x7f1eb8159760 2024-01-26T23:42:58.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.295+0000 7f1ebc907700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb80b3230 con 0x7f1eb80b10f0 2024-01-26T23:42:58.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.295+0000 7f1ebc907700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb80b1570 con 0x7f1eb8159b30 2024-01-26T23:42:58.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.295+0000 7f1eb659c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1eb8159760 0x7f1eb80b2b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39900/0 (socket says 172.21.15.46:39900) 2024-01-26T23:42:58.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.295+0000 7f1eb659c700 1 -- 172.21.15.46:0/249792404 learned_addr learned my addr 172.21.15.46:0/249792404 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:58.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.295+0000 7f1eb5d9b700 1 --2- 172.21.15.46:0/249792404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb8159b30 0x7f1eb80b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:58.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.295+0000 7f1eb559a700 1 -- 172.21.15.46:0/249792404 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767480409 0 0) 0x7f1eb806f940 con 0x7f1eb8159760 2024-01-26T23:42:58.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.295+0000 7f1eb559a700 1 -- 172.21.15.46:0/249792404 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea0003570 con 0x7f1eb8159760 2024-01-26T23:42:58.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.295+0000 7f1eb559a700 1 -- 172.21.15.46:0/249792404 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1083392774 0 0) 0x7f1ea0003570 con 0x7f1eb8159760 2024-01-26T23:42:58.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.295+0000 7f1eb559a700 1 -- 172.21.15.46:0/249792404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb8159b30 msgr2=0x7f1eb80b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:58.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.295+0000 7f1eb559a700 1 --2- 172.21.15.46:0/249792404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb8159b30 0x7f1eb80b0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.295+0000 7f1eb559a700 1 -- 172.21.15.46:0/249792404 >> v1:172.21.15.106:6789/0 conn(0x7f1eb80b10f0 legacy=0x7f1eb80a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:58.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.295+0000 7f1eb559a700 1 -- 172.21.15.46:0/249792404 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb806ed30 con 0x7f1eb8159760 2024-01-26T23:42:58.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.295+0000 7f1eb559a700 1 -- 172.21.15.46:0/249792404 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f1ea8001470 con 0x7f1eb8159760 2024-01-26T23:42:58.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb559a700 1 -- 172.21.15.46:0/249792404 >> v1:172.21.15.46:6789/0 conn(0x7f1eb8159760 legacy=0x7f1eb80b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:58.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb559a700 1 --2- 172.21.15.46:0/249792404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea0003a90 0x7f1ea0003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:58.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb559a700 1 --2- 172.21.15.46:0/249792404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb8159b30 0x7f1ea0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:58.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb559a700 1 --2- 172.21.15.46:0/249792404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ea0004d50 0x7f1ea00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:58.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb559a700 1 -- 172.21.15.46:0/249792404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb80b1570 con 0x7f1ea0003a90 2024-01-26T23:42:58.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb559a700 1 -- 172.21.15.46:0/249792404 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea000bb40 con 0x7f1ea0004d50 2024-01-26T23:42:58.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb559a700 1 -- 172.21.15.46:0/249792404 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea000bd00 con 0x7f1eb8159b30 2024-01-26T23:42:58.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb659c700 1 --2- 172.21.15.46:0/249792404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea0003a90 0x7f1ea0003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:58.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb5d9b700 1 --2- 172.21.15.46:0/249792404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb8159b30 0x7f1ea0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:58.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb6d9d700 1 --2- 172.21.15.46:0/249792404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ea0004d50 0x7f1ea00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:58.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb659c700 1 -- 172.21.15.46:0/249792404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb8159b30 msgr2=0x7f1ea0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:58.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb659c700 1 --2- 172.21.15.46:0/249792404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb8159b30 0x7f1ea0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb659c700 1 -- 172.21.15.46:0/249792404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ea0004d50 msgr2=0x7f1ea00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:58.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb659c700 1 --2- 172.21.15.46:0/249792404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ea0004d50 0x7f1ea00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb659c700 1 -- 172.21.15.46:0/249792404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea0003570 con 0x7f1ea0003a90 2024-01-26T23:42:58.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb659c700 1 --2- 172.21.15.46:0/249792404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea0003a90 0x7f1ea0003e60 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1ea800c3c0 tx=0x7f1ea8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beddecabbc453eb1 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:58.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb559a700 1 -- 172.21.15.46:0/249792404 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea800cc90 con 0x7f1ea0003a90 2024-01-26T23:42:58.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb559a700 1 -- 172.21.15.46:0/249792404 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea800ce10 con 0x7f1ea0003a90 2024-01-26T23:42:58.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.296+0000 7f1eb559a700 1 -- 172.21.15.46:0/249792404 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8005b70 con 0x7f1ea0003a90 2024-01-26T23:42:58.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.297+0000 7f1ebc907700 1 -- 172.21.15.46:0/249792404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea0003a90 msgr2=0x7f1ea0003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:58.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.297+0000 7f1ebc907700 1 --2- 172.21.15.46:0/249792404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea0003a90 0x7f1ea0003e60 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1ea800c3c0 tx=0x7f1ea8001e70 comp rx=0 tx=0).stop 2024-01-26T23:42:58.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.297+0000 7f1eb6d9d700 1 -- 172.21.15.46:0/249792404 reap_dead start 2024-01-26T23:42:58.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.297+0000 7f1ebc907700 1 -- 172.21.15.46:0/249792404 shutdown_connections 2024-01-26T23:42:58.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.297+0000 7f1ebc907700 1 -- 172.21.15.46:0/249792404 >> 172.21.15.46:0/249792404 conn(0x7f1eb8069510 msgr2=0x7f1eb80aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:58.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.297+0000 7f1ebc907700 1 -- 172.21.15.46:0/249792404 shutdown_connections 2024-01-26T23:42:58.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.297+0000 7f1ebc907700 1 -- 172.21.15.46:0/249792404 wait complete. 2024-01-26T23:42:58.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.298+0000 7f1ebc907700 1 Processor -- start 2024-01-26T23:42:58.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.298+0000 7f1ebc907700 1 -- start start 2024-01-26T23:42:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.299+0000 7f1ebc907700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea0003a90 0x7f1eb8143d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.299+0000 7f1ebc907700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb8144240 0x7f1eb8155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.299+0000 7f1ebc907700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1eb8155b60 0x7f1eb815b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.299+0000 7f1ebc907700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb8157110 con 0x7f1eb8144240 2024-01-26T23:42:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.299+0000 7f1ebc907700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb8156070 con 0x7f1eb8155b60 2024-01-26T23:42:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.299+0000 7f1ebc907700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb8156350 con 0x7f1ea0003a90 2024-01-26T23:42:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.299+0000 7f1eb5d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb8144240 0x7f1eb8155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.299+0000 7f1eb5d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb8144240 0x7f1eb8155620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56188/0 (socket says 172.21.15.46:56188) 2024-01-26T23:42:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.299+0000 7f1eb5d9b700 1 -- 172.21.15.46:0/2471080630 learned_addr learned my addr 172.21.15.46:0/2471080630 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.299+0000 7f1eb659c700 1 --2- 172.21.15.46:0/2471080630 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea0003a90 0x7f1eb8143d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.299+0000 7f1eb6d9d700 1 --2- 172.21.15.46:0/2471080630 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1eb8155b60 0x7f1eb815b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.300+0000 7f1eb5d9b700 1 -- 172.21.15.46:0/2471080630 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea0003a90 msgr2=0x7f1eb8143d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.300+0000 7f1eb5d9b700 1 --2- 172.21.15.46:0/2471080630 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea0003a90 0x7f1eb8143d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.300+0000 7f1eb5d9b700 1 -- 172.21.15.46:0/2471080630 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1eb8155b60 msgr2=0x7f1eb815b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.300+0000 7f1eb5d9b700 1 --2- 172.21.15.46:0/2471080630 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1eb8155b60 0x7f1eb815b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.300+0000 7f1eb5d9b700 1 -- 172.21.15.46:0/2471080630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb8156630 con 0x7f1eb8144240 2024-01-26T23:42:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.300+0000 7f1eb6d9d700 1 --2- 172.21.15.46:0/2471080630 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1eb8155b60 0x7f1eb815b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:42:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.300+0000 7f1eb659c700 1 --2- 172.21.15.46:0/2471080630 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea0003a90 0x7f1eb8143d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:42:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.300+0000 7f1eb5d9b700 1 --2- 172.21.15.46:0/2471080630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb8144240 0x7f1eb8155620 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f1eac011d40 tx=0x7f1eac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.300+0000 7f1ea67fc700 1 -- 172.21.15.46:0/2471080630 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac00fac0 con 0x7f1eb8144240 2024-01-26T23:42:58.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.300+0000 7f1ea67fc700 1 -- 172.21.15.46:0/2471080630 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac00fc20 con 0x7f1eb8144240 2024-01-26T23:42:58.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.300+0000 7f1ea67fc700 1 -- 172.21.15.46:0/2471080630 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac014440 con 0x7f1eb8144240 2024-01-26T23:42:58.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.300+0000 7f1ebc907700 1 -- 172.21.15.46:0/2471080630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb8156910 con 0x7f1eb8144240 2024-01-26T23:42:58.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.300+0000 7f1ebc907700 1 -- 172.21.15.46:0/2471080630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb815a1c0 con 0x7f1eb8144240 2024-01-26T23:42:58.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.301+0000 7f1ebc907700 1 -- 172.21.15.46:0/2471080630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb804c710 con 0x7f1eb8144240 2024-01-26T23:42:58.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.302+0000 7f1ea67fc700 1 -- 172.21.15.46:0/2471080630 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1eac014870 con 0x7f1eb8144240 2024-01-26T23:42:58.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.302+0000 7f1ea67fc700 1 --2- 172.21.15.46:0/2471080630 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1e98066f30 0x7f1e980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:58.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.302+0000 7f1ea67fc700 1 -- 172.21.15.46:0/2471080630 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5537+0+0 (secure 0 0 0) 0x7f1eac018030 con 0x7f1eb8144240 2024-01-26T23:42:58.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.302+0000 7f1eb659c700 1 --2- 172.21.15.46:0/2471080630 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1e98066f30 0x7f1e980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:58.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.303+0000 7f1eb659c700 1 --2- 172.21.15.46:0/2471080630 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1e98066f30 0x7f1e980693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8001ed0 tx=0x7f1ea8015000 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:58.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.306+0000 7f1ea67fc700 1 -- 172.21.15.46:0/2471080630 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1eac052260 con 0x7f1eb8144240 2024-01-26T23:42:58.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.513+0000 7f1ebc907700 1 -- 172.21.15.46:0/2471080630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1eb80af6a0 con 0x7f1eb8144240 2024-01-26T23:42:58.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.514+0000 7f1ea67fc700 1 -- 172.21.15.46:0/2471080630 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+12889 (secure 0 0 0) 0x7f1eac055880 con 0x7f1eb8144240 2024-01-26T23:42:58.515 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:58.515 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":66,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:42:57.700573+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:42:58.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.516+0000 7f1ebc907700 1 -- 172.21.15.46:0/2471080630 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1e98066f30 msgr2=0x7f1e980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:58.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.516+0000 7f1ebc907700 1 --2- 172.21.15.46:0/2471080630 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1e98066f30 0x7f1e980693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8001ed0 tx=0x7f1ea8015000 comp rx=0 tx=0).stop 2024-01-26T23:42:58.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.516+0000 7f1ebc907700 1 -- 172.21.15.46:0/2471080630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb8144240 msgr2=0x7f1eb8155620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:58.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.516+0000 7f1ebc907700 1 --2- 172.21.15.46:0/2471080630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb8144240 0x7f1eb8155620 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f1eac011d40 tx=0x7f1eac0091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.516+0000 7f1ebc907700 1 -- 172.21.15.46:0/2471080630 shutdown_connections 2024-01-26T23:42:58.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.516+0000 7f1ebc907700 1 --2- 172.21.15.46:0/2471080630 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1e98066f30 0x7f1e980693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.516+0000 7f1ebc907700 1 --2- 172.21.15.46:0/2471080630 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1eb8155b60 0x7f1eb815b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.516+0000 7f1ebc907700 1 --2- 172.21.15.46:0/2471080630 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea0003a90 0x7f1eb8143d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.516+0000 7f1ebc907700 1 --2- 172.21.15.46:0/2471080630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb8144240 0x7f1eb8155620 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.516+0000 7f1ebc907700 1 -- 172.21.15.46:0/2471080630 >> 172.21.15.46:0/2471080630 conn(0x7f1eb8069510 msgr2=0x7f1eb80cd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:58.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.517+0000 7f1ebc907700 1 -- 172.21.15.46:0/2471080630 shutdown_connections 2024-01-26T23:42:58.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.517+0000 7f1ebc907700 1 -- 172.21.15.46:0/2471080630 wait complete. 2024-01-26T23:42:58.532 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-26T23:42:58.532 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:42:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.666+0000 7fdafad14700 1 Processor -- start 2024-01-26T23:42:58.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.666+0000 7fdafad14700 1 -- start start 2024-01-26T23:42:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.667+0000 7fdafad14700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdaf40c93a0 0x7fdaf40c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.667+0000 7fdafad14700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdaf4067d50 0x7fdaf4159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.667+0000 7fdafad14700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf415e390 con 0x7fdaf40681c0 2024-01-26T23:42:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.667+0000 7fdafad14700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf415d810 con 0x7fdaf4067d50 2024-01-26T23:42:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.667+0000 7fdafad14700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf415daf0 con 0x7fdaf40c93a0 2024-01-26T23:42:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.667+0000 7fdaf92b1700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdaf40681c0 0x7fdaf4159760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39908/0 (socket says 172.21.15.46:39908) 2024-01-26T23:42:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.667+0000 7fdaf92b1700 1 -- 172.21.15.46:0/1128517015 learned_addr learned my addr 172.21.15.46:0/1128517015 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:42:58.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.667+0000 7fdaf8ab0700 1 --2- 172.21.15.46:0/1128517015 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdaf40c93a0 0x7fdaf40c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:58.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.667+0000 7fdaf3fff700 1 --2- 172.21.15.46:0/1128517015 >> v2:172.21.15.106:3300/0 conn(0x7fdaf4067d50 0x7fdaf4159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:58.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.667+0000 7fdaf37fe700 1 -- 172.21.15.46:0/1128517015 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388237429 0 0) 0x7fdaf415e390 con 0x7fdaf40681c0 2024-01-26T23:42:58.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.667+0000 7fdaf37fe700 1 -- 172.21.15.46:0/1128517015 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdae4003570 con 0x7fdaf40681c0 2024-01-26T23:42:58.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.668+0000 7fdaf3fff700 1 -- 172.21.15.46:0/1128517015 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdaf40c93a0 msgr2=0x7fdaf40c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:58.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.668+0000 7fdaf3fff700 1 --2- 172.21.15.46:0/1128517015 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdaf40c93a0 0x7fdaf40c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.668+0000 7fdaf3fff700 1 -- 172.21.15.46:0/1128517015 >> v1:172.21.15.46:6789/0 conn(0x7fdaf40681c0 legacy=0x7fdaf4159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:58.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.668+0000 7fdaf3fff700 1 -- 172.21.15.46:0/1128517015 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf415ddd0 con 0x7fdaf4067d50 2024-01-26T23:42:58.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.668+0000 7fdaf8ab0700 1 --2- 172.21.15.46:0/1128517015 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdaf40c93a0 0x7fdaf40c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:58.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.668+0000 7fdaf3fff700 1 --2- 172.21.15.46:0/1128517015 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdaf4067d50 0x7fdaf4159220 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdae00098e0 tx=0x7fdae00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0e28a356e38cb21 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:58.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.668+0000 7fdaf37fe700 1 -- 172.21.15.46:0/1128517015 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae0011630 con 0x7fdaf4067d50 2024-01-26T23:42:58.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.668+0000 7fdaf37fe700 1 -- 172.21.15.46:0/1128517015 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae0011790 con 0x7fdaf4067d50 2024-01-26T23:42:58.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.668+0000 7fdaf37fe700 1 -- 172.21.15.46:0/1128517015 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae0011b30 con 0x7fdaf4067d50 2024-01-26T23:42:58.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.669+0000 7fdafad14700 1 -- 172.21.15.46:0/1128517015 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdaf4067d50 msgr2=0x7fdaf4159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:58.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.669+0000 7fdafad14700 1 --2- 172.21.15.46:0/1128517015 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdaf4067d50 0x7fdaf4159220 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdae00098e0 tx=0x7fdae00091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.669+0000 7fdafad14700 1 -- 172.21.15.46:0/1128517015 shutdown_connections 2024-01-26T23:42:58.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.669+0000 7fdafad14700 1 --2- 172.21.15.46:0/1128517015 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdaf40c93a0 0x7fdaf40c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.669+0000 7fdafad14700 1 --2- 172.21.15.46:0/1128517015 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdaf4067d50 0x7fdaf4159220 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.669+0000 7fdafad14700 1 -- 172.21.15.46:0/1128517015 >> 172.21.15.46:0/1128517015 conn(0x7fdaf40ca9a0 msgr2=0x7fdaf4068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:58.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.669+0000 7fdafad14700 1 -- 172.21.15.46:0/1128517015 shutdown_connections 2024-01-26T23:42:58.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.669+0000 7fdafad14700 1 -- 172.21.15.46:0/1128517015 wait complete. 2024-01-26T23:42:58.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.670+0000 7fdafad14700 1 Processor -- start 2024-01-26T23:42:58.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.670+0000 7fdafad14700 1 -- start start 2024-01-26T23:42:58.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.671+0000 7fdafad14700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdaf4067d50 0x7fdaf416a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:58.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.671+0000 7fdafad14700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaf40681c0 0x7fdaf414db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:58.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.671+0000 7fdafad14700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdaf40c93a0 0x7fdaf40c4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:58.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.671+0000 7fdafad14700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf416db30 con 0x7fdaf40681c0 2024-01-26T23:42:58.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.671+0000 7fdafad14700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf40b1d70 con 0x7fdaf40c93a0 2024-01-26T23:42:58.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.671+0000 7fdafad14700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf414e070 con 0x7fdaf4067d50 2024-01-26T23:42:58.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.671+0000 7fdaf3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaf40681c0 0x7fdaf414db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:58.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.671+0000 7fdaf3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaf40681c0 0x7fdaf414db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56202/0 (socket says 172.21.15.46:56202) 2024-01-26T23:42:58.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.671+0000 7fdaf3fff700 1 -- 172.21.15.46:0/451919693 learned_addr learned my addr 172.21.15.46:0/451919693 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:58.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.671+0000 7fdaf8ab0700 1 --2- 172.21.15.46:0/451919693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdaf4067d50 0x7fdaf416a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:58.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.671+0000 7fdaf92b1700 1 --2- 172.21.15.46:0/451919693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdaf40c93a0 0x7fdaf40c4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:58.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.671+0000 7fdaf3fff700 1 -- 172.21.15.46:0/451919693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdaf4067d50 msgr2=0x7fdaf416a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:58.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.671+0000 7fdaf3fff700 1 --2- 172.21.15.46:0/451919693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdaf4067d50 0x7fdaf416a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.671+0000 7fdaf3fff700 1 -- 172.21.15.46:0/451919693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdaf40c93a0 msgr2=0x7fdaf40c4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:58.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.671+0000 7fdaf3fff700 1 --2- 172.21.15.46:0/451919693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdaf40c93a0 0x7fdaf40c4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.671+0000 7fdaf3fff700 1 -- 172.21.15.46:0/451919693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae8009420 con 0x7fdaf40681c0 2024-01-26T23:42:58.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.672+0000 7fdaf3fff700 1 --2- 172.21.15.46:0/451919693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaf40681c0 0x7fdaf414db30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fdae0011f20 tx=0x7fdae0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:58.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.672+0000 7fdaf0ff9700 1 -- 172.21.15.46:0/451919693 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae0011900 con 0x7fdaf40681c0 2024-01-26T23:42:58.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.672+0000 7fdaf0ff9700 1 -- 172.21.15.46:0/451919693 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae0011aa0 con 0x7fdaf40681c0 2024-01-26T23:42:58.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.672+0000 7fdafad14700 1 -- 172.21.15.46:0/451919693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae0009420 con 0x7fdaf40681c0 2024-01-26T23:42:58.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.672+0000 7fdaf0ff9700 1 -- 172.21.15.46:0/451919693 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae0025640 con 0x7fdaf40681c0 2024-01-26T23:42:58.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.672+0000 7fdafad14700 1 -- 172.21.15.46:0/451919693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf40b1440 con 0x7fdaf40681c0 2024-01-26T23:42:58.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.673+0000 7fdafad14700 1 -- 172.21.15.46:0/451919693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf40b1f00 con 0x7fdaf40681c0 2024-01-26T23:42:58.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.673+0000 7fdaf0ff9700 1 -- 172.21.15.46:0/451919693 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fdae00259f0 con 0x7fdaf40681c0 2024-01-26T23:42:58.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.674+0000 7fdaf0ff9700 1 --2- 172.21.15.46:0/451919693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdad8066f30 0x7fdad80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:58.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.674+0000 7fdaf0ff9700 1 -- 172.21.15.46:0/451919693 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5537+0+0 (secure 0 0 0) 0x7fdae0085a60 con 0x7fdaf40681c0 2024-01-26T23:42:58.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.674+0000 7fdaf8ab0700 1 --2- 172.21.15.46:0/451919693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdad8066f30 0x7fdad80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:58.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.675+0000 7fdaf8ab0700 1 --2- 172.21.15.46:0/451919693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdad8066f30 0x7fdad80693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdaf40afaa0 tx=0x7fdae80091a0 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:58.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.678+0000 7fdaf0ff9700 1 -- 172.21.15.46:0/451919693 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdae0052d70 con 0x7fdaf40681c0 2024-01-26T23:42:58.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.882+0000 7fdafad14700 1 -- 172.21.15.46:0/451919693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdaf404c710 con 0x7fdaf40681c0 2024-01-26T23:42:58.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.882+0000 7fdaf0ff9700 1 -- 172.21.15.46:0/451919693 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdae0015020 con 0x7fdaf40681c0 2024-01-26T23:42:58.884 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:42:58.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.884+0000 7fdafad14700 1 -- 172.21.15.46:0/451919693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdad8066f30 msgr2=0x7fdad80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:58.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.884+0000 7fdafad14700 1 --2- 172.21.15.46:0/451919693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdad8066f30 0x7fdad80693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdaf40afaa0 tx=0x7fdae80091a0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.884+0000 7fdafad14700 1 -- 172.21.15.46:0/451919693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaf40681c0 msgr2=0x7fdaf414db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:58.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.884+0000 7fdafad14700 1 --2- 172.21.15.46:0/451919693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaf40681c0 0x7fdaf414db30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fdae0011f20 tx=0x7fdae0017000 comp rx=0 tx=0).stop 2024-01-26T23:42:58.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.884+0000 7fdafad14700 1 -- 172.21.15.46:0/451919693 shutdown_connections 2024-01-26T23:42:58.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.884+0000 7fdafad14700 1 --2- 172.21.15.46:0/451919693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdad8066f30 0x7fdad80693e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.884+0000 7fdafad14700 1 --2- 172.21.15.46:0/451919693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdaf40c93a0 0x7fdaf40c4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.884+0000 7fdafad14700 1 --2- 172.21.15.46:0/451919693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdaf4067d50 0x7fdaf416a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.884+0000 7fdafad14700 1 --2- 172.21.15.46:0/451919693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaf40681c0 0x7fdaf414db30 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:58.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.884+0000 7fdafad14700 1 -- 172.21.15.46:0/451919693 >> 172.21.15.46:0/451919693 conn(0x7fdaf40ca9a0 msgr2=0x7fdaf406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:58.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.885+0000 7fdafad14700 1 -- 172.21.15.46:0/451919693 shutdown_connections 2024-01-26T23:42:58.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:58.885+0000 7fdafad14700 1 -- 172.21.15.46:0/451919693 wait complete. 2024-01-26T23:42:58.900 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-26T23:42:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.031+0000 7f88449c3700 1 Processor -- start 2024-01-26T23:42:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.031+0000 7f88449c3700 1 -- start start 2024-01-26T23:42:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.032+0000 7f88449c3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8840158f50 0x7f88401593a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.032+0000 7f88449c3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8840157d80 0x7f88401581d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.032+0000 7f88449c3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f884006cae0 con 0x7f8840155290 2024-01-26T23:42:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.032+0000 7f88449c3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88401598e0 con 0x7f8840158f50 2024-01-26T23:42:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.032+0000 7f88449c3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8840158710 con 0x7f8840157d80 2024-01-26T23:42:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.032+0000 7f883dd9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8840158f50 0x7f88401593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.032+0000 7f883dd9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8840158f50 0x7f88401593a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56218/0 (socket says 172.21.15.46:56218) 2024-01-26T23:42:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.032+0000 7f883dd9b700 1 -- 172.21.15.46:0/2435279945 learned_addr learned my addr 172.21.15.46:0/2435279945 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.032+0000 7f883ed9d700 1 --2- 172.21.15.46:0/2435279945 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8840157d80 0x7f88401581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.032+0000 7f883dd9b700 1 -- 172.21.15.46:0/2435279945 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8840157d80 msgr2=0x7f88401581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.032+0000 7f883dd9b700 1 --2- 172.21.15.46:0/2435279945 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8840157d80 0x7f88401581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.032+0000 7f883dd9b700 1 -- 172.21.15.46:0/2435279945 >> v1:172.21.15.106:6789/0 conn(0x7f8840155290 legacy=0x7f884006cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.032+0000 7f883dd9b700 1 -- 172.21.15.46:0/2435279945 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88400ce010 con 0x7f8840158f50 2024-01-26T23:42:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.032+0000 7f883dd9b700 1 --2- 172.21.15.46:0/2435279945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8840158f50 0x7f88401593a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f88340099b0 tx=0x7f8834009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef82ea1ab9a702f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.033+0000 7f883d59a700 1 -- 172.21.15.46:0/2435279945 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883400b860 con 0x7f8840158f50 2024-01-26T23:42:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.033+0000 7f883d59a700 1 -- 172.21.15.46:0/2435279945 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f883400b9c0 con 0x7f8840158f50 2024-01-26T23:42:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.033+0000 7f883d59a700 1 -- 172.21.15.46:0/2435279945 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883400bd60 con 0x7f8840158f50 2024-01-26T23:42:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.033+0000 7f88449c3700 1 -- 172.21.15.46:0/2435279945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8840158f50 msgr2=0x7f88401593a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.033+0000 7f88449c3700 1 --2- 172.21.15.46:0/2435279945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8840158f50 0x7f88401593a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f88340099b0 tx=0x7f8834009250 comp rx=0 tx=0).stop 2024-01-26T23:42:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.033+0000 7f88449c3700 1 -- 172.21.15.46:0/2435279945 shutdown_connections 2024-01-26T23:42:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.033+0000 7f88449c3700 1 --2- 172.21.15.46:0/2435279945 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8840157d80 0x7f88401581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.033+0000 7f88449c3700 1 --2- 172.21.15.46:0/2435279945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8840158f50 0x7f88401593a0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.033+0000 7f88449c3700 1 -- 172.21.15.46:0/2435279945 >> 172.21.15.46:0/2435279945 conn(0x7f884006b820 msgr2=0x7f88400cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.033+0000 7f88449c3700 1 -- 172.21.15.46:0/2435279945 shutdown_connections 2024-01-26T23:42:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.033+0000 7f88449c3700 1 -- 172.21.15.46:0/2435279945 wait complete. 2024-01-26T23:42:59.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.035+0000 7f88449c3700 1 Processor -- start 2024-01-26T23:42:59.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.035+0000 7f88449c3700 1 -- start start 2024-01-26T23:42:59.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.035+0000 7f88449c3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8840155290 0x7f88400c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.035+0000 7f88449c3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8840157d80 0x7f884016ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.035+0000 7f88449c3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8840158f50 0x7f88400b23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.035+0000 7f88449c3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88400b3270 con 0x7f8840158f50 2024-01-26T23:42:59.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.035+0000 7f88449c3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88400b1a00 con 0x7f8840157d80 2024-01-26T23:42:59.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.035+0000 7f88449c3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88400b1570 con 0x7f8840155290 2024-01-26T23:42:59.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.036+0000 7f883ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8840158f50 0x7f88400b23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.036+0000 7f883ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8840158f50 0x7f88400b23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56230/0 (socket says 172.21.15.46:56230) 2024-01-26T23:42:59.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.036+0000 7f883ed9d700 1 -- 172.21.15.46:0/3882353231 learned_addr learned my addr 172.21.15.46:0/3882353231 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.036+0000 7f883dd9b700 1 --2- 172.21.15.46:0/3882353231 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8840157d80 0x7f884016ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.036+0000 7f883e59c700 1 --2- 172.21.15.46:0/3882353231 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8840155290 0x7f88400c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.036+0000 7f883ed9d700 1 -- 172.21.15.46:0/3882353231 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8840155290 msgr2=0x7f88400c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.036+0000 7f883ed9d700 1 --2- 172.21.15.46:0/3882353231 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8840155290 0x7f88400c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.036+0000 7f883ed9d700 1 -- 172.21.15.46:0/3882353231 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8840157d80 msgr2=0x7f884016ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.036+0000 7f883ed9d700 1 --2- 172.21.15.46:0/3882353231 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8840157d80 0x7f884016ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.036+0000 7f883ed9d700 1 -- 172.21.15.46:0/3882353231 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88340094f0 con 0x7f8840158f50 2024-01-26T23:42:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.036+0000 7f883dd9b700 1 --2- 172.21.15.46:0/3882353231 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8840157d80 0x7f884016ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.036+0000 7f883ed9d700 1 --2- 172.21.15.46:0/3882353231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8840158f50 0x7f88400b23b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8838009fd0 tx=0x7f8838009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.036+0000 7f882e7fc700 1 -- 172.21.15.46:0/3882353231 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883800eca0 con 0x7f8840158f50 2024-01-26T23:42:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.036+0000 7f882e7fc700 1 -- 172.21.15.46:0/3882353231 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f883800ee00 con 0x7f8840158f50 2024-01-26T23:42:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.036+0000 7f88449c3700 1 -- 172.21.15.46:0/3882353231 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f884016b2e0 con 0x7f8840158f50 2024-01-26T23:42:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.036+0000 7f882e7fc700 1 -- 172.21.15.46:0/3882353231 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8838014670 con 0x7f8840158f50 2024-01-26T23:42:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.036+0000 7f88449c3700 1 -- 172.21.15.46:0/3882353231 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88400b0c40 con 0x7f8840158f50 2024-01-26T23:42:59.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.037+0000 7f88449c3700 1 -- 172.21.15.46:0/3882353231 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88400b16e0 con 0x7f8840158f50 2024-01-26T23:42:59.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.038+0000 7f882e7fc700 1 -- 172.21.15.46:0/3882353231 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f88380149e0 con 0x7f8840158f50 2024-01-26T23:42:59.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.038+0000 7f882e7fc700 1 --2- 172.21.15.46:0/3882353231 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8820066f30 0x7f88200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.038+0000 7f882e7fc700 1 -- 172.21.15.46:0/3882353231 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5537+0+0 (secure 0 0 0) 0x7f8838084bc0 con 0x7f8840158f50 2024-01-26T23:42:59.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.039+0000 7f883e59c700 1 --2- 172.21.15.46:0/3882353231 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8820066f30 0x7f88200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.039+0000 7f883e59c700 1 --2- 172.21.15.46:0/3882353231 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8820066f30 0x7f88200693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8830003a80 tx=0x7f8830007ec0 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:59.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.043+0000 7f882e7fc700 1 -- 172.21.15.46:0/3882353231 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8838051ed0 con 0x7f8840158f50 2024-01-26T23:42:59.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.248+0000 7f88449c3700 1 -- 172.21.15.46:0/3882353231 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f884004c710 con 0x7f8840158f50 2024-01-26T23:42:59.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.249+0000 7f882e7fc700 1 -- 172.21.15.46:0/3882353231 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v66) v1 ==== 84+0+65 (secure 0 0 0) 0x7f88380554f0 con 0x7f8840158f50 2024-01-26T23:42:59.250 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:42:59.250 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-26T23:42:59.250 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:42:59.250 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-01-26T23:42:59.250 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-01-26T23:42:59.251 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:42:59.251 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-26T23:42:59.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.251+0000 7f88449c3700 1 -- 172.21.15.46:0/3882353231 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8820066f30 msgr2=0x7f88200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:59.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.251+0000 7f88449c3700 1 --2- 172.21.15.46:0/3882353231 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8820066f30 0x7f88200693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8830003a80 tx=0x7f8830007ec0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.251+0000 7f88449c3700 1 -- 172.21.15.46:0/3882353231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8840158f50 msgr2=0x7f88400b23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:59.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.251+0000 7f88449c3700 1 --2- 172.21.15.46:0/3882353231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8840158f50 0x7f88400b23b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8838009fd0 tx=0x7f8838009900 comp rx=0 tx=0).stop 2024-01-26T23:42:59.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.251+0000 7f88449c3700 1 -- 172.21.15.46:0/3882353231 shutdown_connections 2024-01-26T23:42:59.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.251+0000 7f88449c3700 1 --2- 172.21.15.46:0/3882353231 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8820066f30 0x7f88200693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.251+0000 7f88449c3700 1 --2- 172.21.15.46:0/3882353231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8840158f50 0x7f88400b23b0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.251+0000 7f88449c3700 1 --2- 172.21.15.46:0/3882353231 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8840155290 0x7f88400c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.251+0000 7f88449c3700 1 --2- 172.21.15.46:0/3882353231 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8840157d80 0x7f884016ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.251+0000 7f88449c3700 1 -- 172.21.15.46:0/3882353231 >> 172.21.15.46:0/3882353231 conn(0x7f884006b820 msgr2=0x7f88400c3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:59.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.251+0000 7f88449c3700 1 -- 172.21.15.46:0/3882353231 shutdown_connections 2024-01-26T23:42:59.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.251+0000 7f88449c3700 1 -- 172.21.15.46:0/3882353231 wait complete. 2024-01-26T23:42:59.266 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-26T23:42:59.266 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-26T23:42:59.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.401+0000 7f89156b0700 1 Processor -- start 2024-01-26T23:42:59.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.401+0000 7f89156b0700 1 -- start start 2024-01-26T23:42:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.402+0000 7f89156b0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89100ca4b0 0x7f89100afe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.402+0000 7f89156b0700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f89100b3a50 0x7f89100b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.402+0000 7f89156b0700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8910158010 con 0x7f89100ca0e0 2024-01-26T23:42:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.402+0000 7f89156b0700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f891006c7c0 con 0x7f89100b3a50 2024-01-26T23:42:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.402+0000 7f89156b0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89100cada0 con 0x7f89100ca4b0 2024-01-26T23:42:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.402+0000 7f890f7fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f89100b3a50 0x7f89100b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.402+0000 7f890e7fc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89100ca4b0 0x7f89100afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.403+0000 7f890f7fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f89100b3a50 0x7f89100b3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:43364/0 (socket says 172.21.15.46:43364) 2024-01-26T23:42:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.403+0000 7f890e7fc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89100ca4b0 0x7f89100afe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:47294/0 (socket says 172.21.15.46:47294) 2024-01-26T23:42:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.403+0000 7f890f7fe700 1 -- 172.21.15.46:0/2677066038 learned_addr learned my addr 172.21.15.46:0/2677066038 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.403+0000 7f890effd700 1 --1- 172.21.15.46:0/2677066038 >> v1:172.21.15.106:6789/0 conn(0x7f89100ca0e0 0x7f891006c0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:47006/0 (socket says 172.21.15.46:47006) 2024-01-26T23:42:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.403+0000 7f890f7fe700 1 -- 172.21.15.46:0/2677066038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89100ca4b0 msgr2=0x7f89100afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.403+0000 7f890f7fe700 1 --2- 172.21.15.46:0/2677066038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89100ca4b0 0x7f89100afe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.403+0000 7f890f7fe700 1 -- 172.21.15.46:0/2677066038 >> v1:172.21.15.106:6789/0 conn(0x7f89100ca0e0 legacy=0x7f891006c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.403+0000 7f890f7fe700 1 -- 172.21.15.46:0/2677066038 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89101582f0 con 0x7f89100b3a50 2024-01-26T23:42:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.403+0000 7f890dffb700 1 -- 172.21.15.46:0/2677066038 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173366813 0 0) 0x7f8910158010 con 0x7f89100ca0e0 2024-01-26T23:42:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.403+0000 7f890e7fc700 1 --2- 172.21.15.46:0/2677066038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89100ca4b0 0x7f89100afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:42:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.403+0000 7f890f7fe700 1 --2- 172.21.15.46:0/2677066038 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89100b3a50 0x7f89100b3ed0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8908009f20 tx=0x7f8908009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ae5f11f10db3f43 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.403+0000 7f890dffb700 1 -- 172.21.15.46:0/2677066038 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890800e070 con 0x7f89100b3a50 2024-01-26T23:42:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.403+0000 7f890dffb700 1 -- 172.21.15.46:0/2677066038 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890800b9d0 con 0x7f89100b3a50 2024-01-26T23:42:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.403+0000 7f890dffb700 1 -- 172.21.15.46:0/2677066038 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89080092e0 con 0x7f89100b3a50 2024-01-26T23:42:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.404+0000 7f89156b0700 1 -- 172.21.15.46:0/2677066038 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89100b3a50 msgr2=0x7f89100b3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.404+0000 7f89156b0700 1 --2- 172.21.15.46:0/2677066038 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89100b3a50 0x7f89100b3ed0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8908009f20 tx=0x7f8908009840 comp rx=0 tx=0).stop 2024-01-26T23:42:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.404+0000 7f89156b0700 1 -- 172.21.15.46:0/2677066038 shutdown_connections 2024-01-26T23:42:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.404+0000 7f89156b0700 1 --2- 172.21.15.46:0/2677066038 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89100b3a50 0x7f89100b3ed0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.404+0000 7f89156b0700 1 --2- 172.21.15.46:0/2677066038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89100ca4b0 0x7f89100afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.404+0000 7f89156b0700 1 -- 172.21.15.46:0/2677066038 >> 172.21.15.46:0/2677066038 conn(0x7f89100ce530 msgr2=0x7f89100ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.404+0000 7f89156b0700 1 -- 172.21.15.46:0/2677066038 shutdown_connections 2024-01-26T23:42:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.404+0000 7f89156b0700 1 -- 172.21.15.46:0/2677066038 wait complete. 2024-01-26T23:42:59.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.405+0000 7f89156b0700 1 Processor -- start 2024-01-26T23:42:59.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.405+0000 7f89156b0700 1 -- start start 2024-01-26T23:42:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.406+0000 7f89156b0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89100b3a50 0x7f8910149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.406+0000 7f89156b0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89100ca0e0 0x7f891014d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.406+0000 7f89156b0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89100ca4b0 0x7f89100c9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.406+0000 7f89156b0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891014ead0 con 0x7f89100ca4b0 2024-01-26T23:42:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.406+0000 7f89156b0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8910149f50 con 0x7f89100ca0e0 2024-01-26T23:42:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.406+0000 7f89156b0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891014dc20 con 0x7f89100b3a50 2024-01-26T23:42:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.406+0000 7f890f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89100ca4b0 0x7f89100c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.406+0000 7f890f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89100ca4b0 0x7f89100c9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56232/0 (socket says 172.21.15.46:56232) 2024-01-26T23:42:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.406+0000 7f890f7fe700 1 -- 172.21.15.46:0/1913695293 learned_addr learned my addr 172.21.15.46:0/1913695293 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.406+0000 7f890effd700 1 --2- 172.21.15.46:0/1913695293 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89100b3a50 0x7f8910149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.406+0000 7f890e7fc700 1 --2- 172.21.15.46:0/1913695293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89100ca0e0 0x7f891014d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.407+0000 7f890f7fe700 1 -- 172.21.15.46:0/1913695293 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89100b3a50 msgr2=0x7f8910149a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.407+0000 7f890f7fe700 1 --2- 172.21.15.46:0/1913695293 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89100b3a50 0x7f8910149a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.407+0000 7f890f7fe700 1 -- 172.21.15.46:0/1913695293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89100ca0e0 msgr2=0x7f891014d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.407+0000 7f890f7fe700 1 --2- 172.21.15.46:0/1913695293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89100ca0e0 0x7f891014d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.407+0000 7f890f7fe700 1 -- 172.21.15.46:0/1913695293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890400d8d0 con 0x7f89100ca4b0 2024-01-26T23:42:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.407+0000 7f890f7fe700 1 --2- 172.21.15.46:0/1913695293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89100ca4b0 0x7f89100c9240 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f890800b610 tx=0x7f8908018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.407+0000 7f88f6ffd700 1 -- 172.21.15.46:0/1913695293 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908008ad0 con 0x7f89100ca4b0 2024-01-26T23:42:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.407+0000 7f88f6ffd700 1 -- 172.21.15.46:0/1913695293 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8908008c50 con 0x7f89100ca4b0 2024-01-26T23:42:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.407+0000 7f89156b0700 1 -- 172.21.15.46:0/1913695293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8908009ae0 con 0x7f89100ca4b0 2024-01-26T23:42:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.407+0000 7f88f6ffd700 1 -- 172.21.15.46:0/1913695293 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908026460 con 0x7f89100ca4b0 2024-01-26T23:42:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.407+0000 7f89156b0700 1 -- 172.21.15.46:0/1913695293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f891014e1a0 con 0x7f89100ca4b0 2024-01-26T23:42:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.408+0000 7f89156b0700 1 -- 172.21.15.46:0/1913695293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f891004c710 con 0x7f89100ca4b0 2024-01-26T23:42:59.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.409+0000 7f88f6ffd700 1 -- 172.21.15.46:0/1913695293 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8908003040 con 0x7f89100ca4b0 2024-01-26T23:42:59.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.409+0000 7f88f6ffd700 1 --2- 172.21.15.46:0/1913695293 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f88fc066f30 0x7f88fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.409+0000 7f88f6ffd700 1 -- 172.21.15.46:0/1913695293 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5537+0+0 (secure 0 0 0) 0x7f8908086200 con 0x7f89100ca4b0 2024-01-26T23:42:59.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.409+0000 7f890effd700 1 --2- 172.21.15.46:0/1913695293 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f88fc066f30 0x7f88fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.410+0000 7f890effd700 1 --2- 172.21.15.46:0/1913695293 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f88fc066f30 0x7f88fc0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8900007fd0 tx=0x7f8900007d50 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:59.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.414+0000 7f88f6ffd700 1 -- 172.21.15.46:0/1913695293 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8908053510 con 0x7f89100ca4b0 2024-01-26T23:42:59.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.617+0000 7f89156b0700 1 -- 172.21.15.46:0/1913695293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f89100ce310 con 0x7f89100ca4b0 2024-01-26T23:42:59.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.695+0000 7f88f6ffd700 1 -- 172.21.15.46:0/1913695293 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v67) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8908056b30 con 0x7f89100ca4b0 2024-01-26T23:42:59.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.699+0000 7f89156b0700 1 -- 172.21.15.46:0/1913695293 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f88fc066f30 msgr2=0x7f88fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:59.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.699+0000 7f89156b0700 1 --2- 172.21.15.46:0/1913695293 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f88fc066f30 0x7f88fc0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8900007fd0 tx=0x7f8900007d50 comp rx=0 tx=0).stop 2024-01-26T23:42:59.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.699+0000 7f89156b0700 1 -- 172.21.15.46:0/1913695293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89100ca4b0 msgr2=0x7f89100c9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:59.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.699+0000 7f89156b0700 1 --2- 172.21.15.46:0/1913695293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89100ca4b0 0x7f89100c9240 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f890800b610 tx=0x7f8908018000 comp rx=0 tx=0).stop 2024-01-26T23:42:59.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.699+0000 7f89156b0700 1 -- 172.21.15.46:0/1913695293 shutdown_connections 2024-01-26T23:42:59.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.699+0000 7f89156b0700 1 --2- 172.21.15.46:0/1913695293 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f88fc066f30 0x7f88fc0693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.699+0000 7f89156b0700 1 --2- 172.21.15.46:0/1913695293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89100ca4b0 0x7f89100c9240 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.699+0000 7f89156b0700 1 --2- 172.21.15.46:0/1913695293 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89100b3a50 0x7f8910149a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.699+0000 7f89156b0700 1 --2- 172.21.15.46:0/1913695293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89100ca0e0 0x7f891014d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.699+0000 7f89156b0700 1 -- 172.21.15.46:0/1913695293 >> 172.21.15.46:0/1913695293 conn(0x7f89100ce530 msgr2=0x7f89100c4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:59.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.699+0000 7f89156b0700 1 -- 172.21.15.46:0/1913695293 shutdown_connections 2024-01-26T23:42:59.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.699+0000 7f89156b0700 1 -- 172.21.15.46:0/1913695293 wait complete. 2024-01-26T23:42:59.702 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-01-26T23:42:59.716 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-26T23:42:59.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.856+0000 7f7c0c826700 1 Processor -- start 2024-01-26T23:42:59.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.856+0000 7f7c0c826700 1 -- start start 2024-01-26T23:42:59.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.857+0000 7f7c0c826700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7c0806c450 0x7f7c08157e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.857+0000 7f7c0c826700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c08158340 0x7f7c080cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.857+0000 7f7c0c826700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0806c8f0 con 0x7f7c08157a30 2024-01-26T23:42:59.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.857+0000 7f7c0c826700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c081598a0 con 0x7f7c08158340 2024-01-26T23:42:59.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.857+0000 7f7c0c826700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c081587c0 con 0x7f7c0806c450 2024-01-26T23:42:59.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.857+0000 7f7c06d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c08158340 0x7f7c080cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.857+0000 7f7c06d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c08158340 0x7f7c080cd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56234/0 (socket says 172.21.15.46:56234) 2024-01-26T23:42:59.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.857+0000 7f7c06d9d700 1 -- 172.21.15.46:0/1994456050 learned_addr learned my addr 172.21.15.46:0/1994456050 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:59.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.857+0000 7f7c05d9b700 1 --2- 172.21.15.46:0/1994456050 >> v2:172.21.15.106:3300/0 conn(0x7f7c0806c450 0x7f7c08157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.857+0000 7f7c06d9d700 1 -- 172.21.15.46:0/1994456050 >> v2:172.21.15.106:3300/0 conn(0x7f7c0806c450 msgr2=0x7f7c08157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:59.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.857+0000 7f7c06d9d700 1 --2- 172.21.15.46:0/1994456050 >> v2:172.21.15.106:3300/0 conn(0x7f7c0806c450 0x7f7c08157e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.857+0000 7f7c06d9d700 1 -- 172.21.15.46:0/1994456050 >> v1:172.21.15.106:6789/0 conn(0x7f7c08157a30 legacy=0x7f7c08159190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:42:59.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.858+0000 7f7c06d9d700 1 -- 172.21.15.46:0/1994456050 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c08158aa0 con 0x7f7c08158340 2024-01-26T23:42:59.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.858+0000 7f7c06d9d700 1 --2- 172.21.15.46:0/1994456050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c08158340 0x7f7c080cd890 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7c0806e8d0 tx=0x7f7c00009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd90161826ffbfc server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:59.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.858+0000 7f7c0559a700 1 -- 172.21.15.46:0/1994456050 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0000e070 con 0x7f7c08158340 2024-01-26T23:42:59.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.858+0000 7f7c0559a700 1 -- 172.21.15.46:0/1994456050 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0000ba10 con 0x7f7c08158340 2024-01-26T23:42:59.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.858+0000 7f7c0559a700 1 -- 172.21.15.46:0/1994456050 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c000092e0 con 0x7f7c08158340 2024-01-26T23:42:59.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.858+0000 7f7c0c826700 1 -- 172.21.15.46:0/1994456050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c08158340 msgr2=0x7f7c080cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:59.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.858+0000 7f7c0c826700 1 --2- 172.21.15.46:0/1994456050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c08158340 0x7f7c080cd890 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7c0806e8d0 tx=0x7f7c00009840 comp rx=0 tx=0).stop 2024-01-26T23:42:59.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.859+0000 7f7c0c826700 1 -- 172.21.15.46:0/1994456050 shutdown_connections 2024-01-26T23:42:59.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.859+0000 7f7c0c826700 1 --2- 172.21.15.46:0/1994456050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c08158340 0x7f7c080cd890 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.859+0000 7f7c0c826700 1 --2- 172.21.15.46:0/1994456050 >> v2:172.21.15.106:3300/0 conn(0x7f7c0806c450 0x7f7c08157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.859+0000 7f7c0c826700 1 -- 172.21.15.46:0/1994456050 >> 172.21.15.46:0/1994456050 conn(0x7f7c080ce570 msgr2=0x7f7c080ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:42:59.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.859+0000 7f7c0c826700 1 -- 172.21.15.46:0/1994456050 shutdown_connections 2024-01-26T23:42:59.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.859+0000 7f7c0c826700 1 -- 172.21.15.46:0/1994456050 wait complete. 2024-01-26T23:42:59.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.860+0000 7f7c0c826700 1 Processor -- start 2024-01-26T23:42:59.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.860+0000 7f7c0c826700 1 -- start start 2024-01-26T23:42:59.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c0c826700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c0806c450 0x7f7c0814b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c0c826700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c08157a30 0x7f7c08069400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c0c826700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c0806e410 0x7f7c08150030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c0c826700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0806a940 con 0x7f7c0806e410 2024-01-26T23:42:59.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c0c826700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c08069d30 con 0x7f7c08157a30 2024-01-26T23:42:59.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c0c826700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0806a010 con 0x7f7c0806c450 2024-01-26T23:42:59.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c06d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c0806e410 0x7f7c08150030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c06d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c0806e410 0x7f7c08150030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56246/0 (socket says 172.21.15.46:56246) 2024-01-26T23:42:59.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c06d9d700 1 -- 172.21.15.46:0/4289098949 learned_addr learned my addr 172.21.15.46:0/4289098949 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:42:59.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c05d9b700 1 --2- 172.21.15.46:0/4289098949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c08157a30 0x7f7c08069400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c0659c700 1 --2- 172.21.15.46:0/4289098949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c0806c450 0x7f7c0814b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c06d9d700 1 -- 172.21.15.46:0/4289098949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c0806c450 msgr2=0x7f7c0814b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:59.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c06d9d700 1 --2- 172.21.15.46:0/4289098949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c0806c450 0x7f7c0814b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c06d9d700 1 -- 172.21.15.46:0/4289098949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c08157a30 msgr2=0x7f7c08069400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:42:59.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c06d9d700 1 --2- 172.21.15.46:0/4289098949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c08157a30 0x7f7c08069400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:42:59.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c06d9d700 1 -- 172.21.15.46:0/4289098949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c00009ae0 con 0x7f7c0806e410 2024-01-26T23:42:59.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c05d9b700 1 --2- 172.21.15.46:0/4289098949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c08157a30 0x7f7c08069400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:42:59.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.861+0000 7f7c06d9d700 1 --2- 172.21.15.46:0/4289098949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c0806e410 0x7f7c08150030 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7c0815b520 tx=0x7f7c00017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:59.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.862+0000 7f7bfe7fc700 1 -- 172.21.15.46:0/4289098949 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00012ea0 con 0x7f7c0806e410 2024-01-26T23:42:59.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.862+0000 7f7bfe7fc700 1 -- 172.21.15.46:0/4289098949 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c000092e0 con 0x7f7c0806e410 2024-01-26T23:42:59.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.862+0000 7f7bfe7fc700 1 -- 172.21.15.46:0/4289098949 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0001d670 con 0x7f7c0806e410 2024-01-26T23:42:59.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.862+0000 7f7c0c826700 1 -- 172.21.15.46:0/4289098949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0806a2f0 con 0x7f7c0806e410 2024-01-26T23:42:59.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.862+0000 7f7c0c826700 1 -- 172.21.15.46:0/4289098949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0814b7b0 con 0x7f7c0806e410 2024-01-26T23:42:59.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.863+0000 7f7c0c826700 1 -- 172.21.15.46:0/4289098949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c08070270 con 0x7f7c0806e410 2024-01-26T23:42:59.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.863+0000 7f7bfe7fc700 1 -- 172.21.15.46:0/4289098949 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7c0001daa0 con 0x7f7c0806e410 2024-01-26T23:42:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.864+0000 7f7bfe7fc700 1 --2- 172.21.15.46:0/4289098949 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7be8066f30 0x7f7be80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:42:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.864+0000 7f7bfe7fc700 1 -- 172.21.15.46:0/4289098949 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f7c00086170 con 0x7f7c0806e410 2024-01-26T23:42:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.864+0000 7f7c0659c700 1 --2- 172.21.15.46:0/4289098949 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7be8066f30 0x7f7be80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:42:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.864+0000 7f7c0659c700 1 --2- 172.21.15.46:0/4289098949 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7be8066f30 0x7f7be80693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8006fd0 tx=0x7f7bf8008040 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:42:59.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:42:59.869+0000 7f7bfe7fc700 1 -- 172.21.15.46:0/4289098949 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c00053270 con 0x7f7c0806e410 2024-01-26T23:43:00.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.072+0000 7f7c0c826700 1 -- 172.21.15.46:0/4289098949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7c0806c8f0 con 0x7f7c0806e410 2024-01-26T23:43:00.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.701+0000 7f7bfe7fc700 1 -- 172.21.15.46:0/4289098949 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v68) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7c0806c8f0 con 0x7f7c0806e410 2024-01-26T23:43:00.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.707+0000 7f7c0c826700 1 -- 172.21.15.46:0/4289098949 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7be8066f30 msgr2=0x7f7be80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:00.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.707+0000 7f7c0c826700 1 --2- 172.21.15.46:0/4289098949 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7be8066f30 0x7f7be80693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8006fd0 tx=0x7f7bf8008040 comp rx=0 tx=0).stop 2024-01-26T23:43:00.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.707+0000 7f7c0c826700 1 -- 172.21.15.46:0/4289098949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c0806e410 msgr2=0x7f7c08150030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:00.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.707+0000 7f7c0c826700 1 --2- 172.21.15.46:0/4289098949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c0806e410 0x7f7c08150030 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7c0815b520 tx=0x7f7c00017000 comp rx=0 tx=0).stop 2024-01-26T23:43:00.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.710+0000 7f7c0c826700 1 -- 172.21.15.46:0/4289098949 shutdown_connections 2024-01-26T23:43:00.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.710+0000 7f7c0c826700 1 --2- 172.21.15.46:0/4289098949 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7be8066f30 0x7f7be80693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:00.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.710+0000 7f7c0c826700 1 --2- 172.21.15.46:0/4289098949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c0806e410 0x7f7c08150030 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:00.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.710+0000 7f7c0c826700 1 --2- 172.21.15.46:0/4289098949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c0806c450 0x7f7c0814b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:00.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.710+0000 7f7c0c826700 1 --2- 172.21.15.46:0/4289098949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c08157a30 0x7f7c08069400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:00.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.710+0000 7f7c0c826700 1 -- 172.21.15.46:0/4289098949 >> 172.21.15.46:0/4289098949 conn(0x7f7c080ce570 msgr2=0x7f7c08159a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.714+0000 7f7c0c826700 1 -- 172.21.15.46:0/4289098949 shutdown_connections 2024-01-26T23:43:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.717+0000 7f7c0c826700 1 -- 172.21.15.46:0/4289098949 wait complete. 2024-01-26T23:43:00.725 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-01-26T23:43:00.740 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-26T23:43:00.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.877+0000 7f73e7c10700 1 Processor -- start 2024-01-26T23:43:00.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.878+0000 7f73e7c10700 1 -- start start 2024-01-26T23:43:00.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.878+0000 7f73e7c10700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73e01565a0 0x7f73e0156970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:00.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.878+0000 7f73e7c10700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e015a520 0x7f73e00acc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:00.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.878+0000 7f73e7c10700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e00b0830 con 0x7f73e00ae2f0 2024-01-26T23:43:00.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.878+0000 7f73e7c10700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e00ae130 con 0x7f73e01565a0 2024-01-26T23:43:00.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.879+0000 7f73e7c10700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00ad5b0 con 0x7f73e015a520 2024-01-26T23:43:00.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.879+0000 7f73e59ac700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73e01565a0 0x7f73e0156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:00.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.879+0000 7f73e61ad700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f73e00ae2f0 0x7f73e006ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37732/0 (socket says 172.21.15.46:37732) 2024-01-26T23:43:00.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.879+0000 7f73e61ad700 1 -- 172.21.15.46:0/1333922229 learned_addr learned my addr 172.21.15.46:0/1333922229 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:43:00.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.879+0000 7f73e51ab700 1 --2- 172.21.15.46:0/1333922229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e015a520 0x7f73e00acc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:00.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.879+0000 7f73e59ac700 1 -- 172.21.15.46:0/1333922229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e015a520 msgr2=0x7f73e00acc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:00.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.879+0000 7f73e59ac700 1 --2- 172.21.15.46:0/1333922229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e015a520 0x7f73e00acc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:00.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.879+0000 7f73e59ac700 1 -- 172.21.15.46:0/1333922229 >> v1:172.21.15.46:6789/0 conn(0x7f73e00ae2f0 legacy=0x7f73e006ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:00.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.879+0000 7f73e59ac700 1 -- 172.21.15.46:0/1333922229 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e00ad890 con 0x7f73e01565a0 2024-01-26T23:43:00.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.879+0000 7f73e49aa700 1 -- 172.21.15.46:0/1333922229 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426550715 0 0) 0x7f73e00b0830 con 0x7f73e00ae2f0 2024-01-26T23:43:00.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.879+0000 7f73e59ac700 1 --2- 172.21.15.46:0/1333922229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e01565a0 0x7f73e0156970 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f73d40098e0 tx=0x7f73d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2190b71fc18ab355 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:00.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.879+0000 7f73e49aa700 1 -- 172.21.15.46:0/1333922229 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d4009e80 con 0x7f73e01565a0 2024-01-26T23:43:00.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.879+0000 7f73e49aa700 1 -- 172.21.15.46:0/1333922229 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d400b860 con 0x7f73e01565a0 2024-01-26T23:43:00.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.879+0000 7f73e49aa700 1 -- 172.21.15.46:0/1333922229 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d400bbe0 con 0x7f73e01565a0 2024-01-26T23:43:00.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.880+0000 7f73e7c10700 1 -- 172.21.15.46:0/1333922229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e01565a0 msgr2=0x7f73e0156970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:00.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.880+0000 7f73e7c10700 1 --2- 172.21.15.46:0/1333922229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e01565a0 0x7f73e0156970 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f73d40098e0 tx=0x7f73d40091a0 comp rx=0 tx=0).stop 2024-01-26T23:43:00.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.880+0000 7f73e7c10700 1 -- 172.21.15.46:0/1333922229 shutdown_connections 2024-01-26T23:43:00.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.880+0000 7f73e7c10700 1 --2- 172.21.15.46:0/1333922229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e01565a0 0x7f73e0156970 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:00.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.880+0000 7f73e7c10700 1 --2- 172.21.15.46:0/1333922229 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e015a520 0x7f73e00acc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:00.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.880+0000 7f73e7c10700 1 -- 172.21.15.46:0/1333922229 >> 172.21.15.46:0/1333922229 conn(0x7f73e00cdf80 msgr2=0x7f73e006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:00.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.880+0000 7f73e7c10700 1 -- 172.21.15.46:0/1333922229 shutdown_connections 2024-01-26T23:43:00.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.880+0000 7f73e7c10700 1 -- 172.21.15.46:0/1333922229 wait complete. 2024-01-26T23:43:00.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.881+0000 7f73e7c10700 1 Processor -- start 2024-01-26T23:43:00.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.881+0000 7f73e7c10700 1 -- start start 2024-01-26T23:43:00.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.882+0000 7f73e7c10700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e00ae2f0 0x7f73e014d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:00.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.882+0000 7f73e7c10700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73e01565a0 0x7f73e0067930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:00.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.882+0000 7f73e7c10700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e015a520 0x7f73e0158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:00.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.882+0000 7f73e7c10700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e01551b0 con 0x7f73e00ae2f0 2024-01-26T23:43:00.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.882+0000 7f73e7c10700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e016b520 con 0x7f73e01565a0 2024-01-26T23:43:00.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.882+0000 7f73e7c10700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0155640 con 0x7f73e015a520 2024-01-26T23:43:00.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.882+0000 7f73e59ac700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e00ae2f0 0x7f73e014d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:00.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.882+0000 7f73e59ac700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e00ae2f0 0x7f73e014d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50642/0 (socket says 172.21.15.46:50642) 2024-01-26T23:43:00.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.882+0000 7f73e59ac700 1 -- 172.21.15.46:0/724613898 learned_addr learned my addr 172.21.15.46:0/724613898 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:00.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.882+0000 7f73e61ad700 1 --2- 172.21.15.46:0/724613898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e015a520 0x7f73e0158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:00.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.882+0000 7f73e51ab700 1 --2- 172.21.15.46:0/724613898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73e01565a0 0x7f73e0067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:00.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.883+0000 7f73e59ac700 1 -- 172.21.15.46:0/724613898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e015a520 msgr2=0x7f73e0158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:00.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.883+0000 7f73e59ac700 1 --2- 172.21.15.46:0/724613898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e015a520 0x7f73e0158cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:00.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.883+0000 7f73e59ac700 1 -- 172.21.15.46:0/724613898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73e01565a0 msgr2=0x7f73e0067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:00.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.883+0000 7f73e59ac700 1 --2- 172.21.15.46:0/724613898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73e01565a0 0x7f73e0067930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:00.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.883+0000 7f73e59ac700 1 -- 172.21.15.46:0/724613898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d4009420 con 0x7f73e00ae2f0 2024-01-26T23:43:00.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.883+0000 7f73e59ac700 1 --2- 172.21.15.46:0/724613898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e00ae2f0 0x7f73e014d910 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f73e0064440 tx=0x7f73d4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:00.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.883+0000 7f73d1ffb700 1 -- 172.21.15.46:0/724613898 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d400b9d0 con 0x7f73e00ae2f0 2024-01-26T23:43:00.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.883+0000 7f73d1ffb700 1 -- 172.21.15.46:0/724613898 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d400bb30 con 0x7f73e00ae2f0 2024-01-26T23:43:00.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.883+0000 7f73e7c10700 1 -- 172.21.15.46:0/724613898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e00683f0 con 0x7f73e00ae2f0 2024-01-26T23:43:00.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.883+0000 7f73d1ffb700 1 -- 172.21.15.46:0/724613898 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d4011670 con 0x7f73e00ae2f0 2024-01-26T23:43:00.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.883+0000 7f73e7c10700 1 -- 172.21.15.46:0/724613898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e0154e40 con 0x7f73e00ae2f0 2024-01-26T23:43:00.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.884+0000 7f73e7c10700 1 -- 172.21.15.46:0/724613898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e004c710 con 0x7f73e00ae2f0 2024-01-26T23:43:00.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.885+0000 7f73d1ffb700 1 -- 172.21.15.46:0/724613898 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f73d4011a20 con 0x7f73e00ae2f0 2024-01-26T23:43:00.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.885+0000 7f73d1ffb700 1 --2- 172.21.15.46:0/724613898 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73c4066f30 0x7f73c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:00.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.886+0000 7f73d1ffb700 1 -- 172.21.15.46:0/724613898 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f73d4086850 con 0x7f73e00ae2f0 2024-01-26T23:43:00.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.886+0000 7f73e51ab700 1 --2- 172.21.15.46:0/724613898 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73c4066f30 0x7f73c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:00.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.886+0000 7f73e51ab700 1 --2- 172.21.15.46:0/724613898 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73c4066f30 0x7f73c40693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f73e014ec50 tx=0x7f73dc008040 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:00.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:00.890+0000 7f73d1ffb700 1 -- 172.21.15.46:0/724613898 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f73d4053970 con 0x7f73e00ae2f0 2024-01-26T23:43:01.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.142+0000 7f73e7c10700 1 -- 172.21.15.46:0/724613898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f73e0064030 con 0x7f73e00ae2f0 2024-01-26T23:43:01.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.702+0000 7f73d1ffb700 1 -- 172.21.15.46:0/724613898 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v23) v1 ==== 172+0+0 (secure 0 0 0) 0x7f73d4056f10 con 0x7f73e00ae2f0 2024-01-26T23:43:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.709+0000 7f73e7c10700 1 -- 172.21.15.46:0/724613898 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73c4066f30 msgr2=0x7f73c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.709+0000 7f73e7c10700 1 --2- 172.21.15.46:0/724613898 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73c4066f30 0x7f73c40693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f73e014ec50 tx=0x7f73dc008040 comp rx=0 tx=0).stop 2024-01-26T23:43:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.709+0000 7f73e7c10700 1 -- 172.21.15.46:0/724613898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e00ae2f0 msgr2=0x7f73e014d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.709+0000 7f73e7c10700 1 --2- 172.21.15.46:0/724613898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e00ae2f0 0x7f73e014d910 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f73e0064440 tx=0x7f73d4017000 comp rx=0 tx=0).stop 2024-01-26T23:43:01.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.709+0000 7f73e7c10700 1 -- 172.21.15.46:0/724613898 shutdown_connections 2024-01-26T23:43:01.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.709+0000 7f73e7c10700 1 --2- 172.21.15.46:0/724613898 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73c4066f30 0x7f73c40693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:01.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.709+0000 7f73e7c10700 1 --2- 172.21.15.46:0/724613898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e00ae2f0 0x7f73e014d910 secure :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f73e0064440 tx=0x7f73d4017000 comp rx=0 tx=0).stop 2024-01-26T23:43:01.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.709+0000 7f73e7c10700 1 --2- 172.21.15.46:0/724613898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e015a520 0x7f73e0158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:01.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.709+0000 7f73e7c10700 1 --2- 172.21.15.46:0/724613898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73e01565a0 0x7f73e0067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:01.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.709+0000 7f73e7c10700 1 -- 172.21.15.46:0/724613898 >> 172.21.15.46:0/724613898 conn(0x7f73e00cdf80 msgr2=0x7f73e00b0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:01.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.710+0000 7f73e7c10700 1 -- 172.21.15.46:0/724613898 shutdown_connections 2024-01-26T23:43:01.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.710+0000 7f73e7c10700 1 -- 172.21.15.46:0/724613898 wait complete. 2024-01-26T23:43:01.753 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 7 and data pool 8 2024-01-26T23:43:01.753 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:43:01.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.889+0000 7f01a43dc700 1 Processor -- start 2024-01-26T23:43:01.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.889+0000 7f01a43dc700 1 -- start start 2024-01-26T23:43:01.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.890+0000 7f01a43dc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f019c0c4440 0x7f019c0cca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:01.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.890+0000 7f01a43dc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f019c159bd0 0x7f019c0c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:01.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.890+0000 7f01a43dc700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019c0c8ba0 con 0x7f019c159800 2024-01-26T23:43:01.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.890+0000 7f01a43dc700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f019c0c8e40 con 0x7f019c0c4440 2024-01-26T23:43:01.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.890+0000 7f01a43dc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019c0c9120 con 0x7f019c159bd0 2024-01-26T23:43:01.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.891+0000 7f01a1977700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f019c0c4440 0x7f019c0cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:01.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.891+0000 7f01a2178700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f019c159800 0x7f019c0c3d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37744/0 (socket says 172.21.15.46:37744) 2024-01-26T23:43:01.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.891+0000 7f01a1977700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f019c0c4440 0x7f019c0cca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50646/0 (socket says 172.21.15.46:50646) 2024-01-26T23:43:01.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.891+0000 7f01a2178700 1 -- 172.21.15.46:0/2471688159 learned_addr learned my addr 172.21.15.46:0/2471688159 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:43:01.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.891+0000 7f01a2979700 1 --2- 172.21.15.46:0/2471688159 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f019c159bd0 0x7f019c0c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:01.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.891+0000 7f01a1977700 1 -- 172.21.15.46:0/2471688159 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f019c159bd0 msgr2=0x7f019c0c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:01.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.891+0000 7f01a1977700 1 --2- 172.21.15.46:0/2471688159 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f019c159bd0 0x7f019c0c9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:01.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.891+0000 7f01a1977700 1 -- 172.21.15.46:0/2471688159 >> v1:172.21.15.46:6789/0 conn(0x7f019c159800 legacy=0x7f019c0c3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:01.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.891+0000 7f01a1977700 1 -- 172.21.15.46:0/2471688159 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c0c9400 con 0x7f019c0c4440 2024-01-26T23:43:01.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.891+0000 7f01a1176700 1 -- 172.21.15.46:0/2471688159 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774903028 0 0) 0x7f019c0c8ba0 con 0x7f019c159800 2024-01-26T23:43:01.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.891+0000 7f01a1977700 1 --2- 172.21.15.46:0/2471688159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f019c0c4440 0x7f019c0cca80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f018c0098e0 tx=0x7f018c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dce06c32b05763e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:01.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.891+0000 7f01a1176700 1 -- 172.21.15.46:0/2471688159 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018c00b640 con 0x7f019c0c4440 2024-01-26T23:43:01.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.891+0000 7f01a1176700 1 -- 172.21.15.46:0/2471688159 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018c00b7a0 con 0x7f019c0c4440 2024-01-26T23:43:01.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.891+0000 7f01a1176700 1 -- 172.21.15.46:0/2471688159 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018c00bb40 con 0x7f019c0c4440 2024-01-26T23:43:01.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.892+0000 7f01a43dc700 1 -- 172.21.15.46:0/2471688159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f019c0c4440 msgr2=0x7f019c0cca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.892+0000 7f01a43dc700 1 --2- 172.21.15.46:0/2471688159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f019c0c4440 0x7f019c0cca80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f018c0098e0 tx=0x7f018c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:43:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.892+0000 7f01a43dc700 1 -- 172.21.15.46:0/2471688159 shutdown_connections 2024-01-26T23:43:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.892+0000 7f01a43dc700 1 --2- 172.21.15.46:0/2471688159 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f019c159bd0 0x7f019c0c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.892+0000 7f01a43dc700 1 --2- 172.21.15.46:0/2471688159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f019c0c4440 0x7f019c0cca80 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.892+0000 7f01a43dc700 1 -- 172.21.15.46:0/2471688159 >> 172.21.15.46:0/2471688159 conn(0x7f019c097820 msgr2=0x7f019c068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.892+0000 7f01a43dc700 1 -- 172.21.15.46:0/2471688159 shutdown_connections 2024-01-26T23:43:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.892+0000 7f01a43dc700 1 -- 172.21.15.46:0/2471688159 wait complete. 2024-01-26T23:43:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.893+0000 7f01a43dc700 1 Processor -- start 2024-01-26T23:43:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.893+0000 7f01a43dc700 1 -- start start 2024-01-26T23:43:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.894+0000 7f01a43dc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f019c0c4440 0x7f019c14e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.894+0000 7f01a43dc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f019c159800 0x7f019c155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.894+0000 7f01a43dc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f019c159bd0 0x7f019c1562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.894+0000 7f01a43dc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019c0b1790 con 0x7f019c159bd0 2024-01-26T23:43:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.894+0000 7f01a43dc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019c0b1a70 con 0x7f019c159800 2024-01-26T23:43:01.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.894+0000 7f01a43dc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019c0b1d50 con 0x7f019c0c4440 2024-01-26T23:43:01.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.894+0000 7f01a2979700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f019c159bd0 0x7f019c1562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:01.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.894+0000 7f01a2979700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f019c159bd0 0x7f019c1562d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50662/0 (socket says 172.21.15.46:50662) 2024-01-26T23:43:01.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.894+0000 7f01a2979700 1 -- 172.21.15.46:0/2653501388 learned_addr learned my addr 172.21.15.46:0/2653501388 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:01.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.894+0000 7f01a2178700 1 --2- 172.21.15.46:0/2653501388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f019c0c4440 0x7f019c14e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:01.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.894+0000 7f01a1977700 1 --2- 172.21.15.46:0/2653501388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f019c159800 0x7f019c155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:01.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.895+0000 7f01a2979700 1 -- 172.21.15.46:0/2653501388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f019c0c4440 msgr2=0x7f019c14e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:01.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.895+0000 7f01a2979700 1 --2- 172.21.15.46:0/2653501388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f019c0c4440 0x7f019c14e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:01.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.895+0000 7f01a2979700 1 -- 172.21.15.46:0/2653501388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f019c159800 msgr2=0x7f019c155d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:01.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.895+0000 7f01a2979700 1 --2- 172.21.15.46:0/2653501388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f019c159800 0x7f019c155d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:01.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.895+0000 7f01a2979700 1 -- 172.21.15.46:0/2653501388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018c009420 con 0x7f019c159bd0 2024-01-26T23:43:01.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.895+0000 7f01a2979700 1 --2- 172.21.15.46:0/2653501388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f019c159bd0 0x7f019c1562d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0194009d40 tx=0x7f019400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:01.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.895+0000 7f01927fc700 1 -- 172.21.15.46:0/2653501388 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0194009ea0 con 0x7f019c159bd0 2024-01-26T23:43:01.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.895+0000 7f01927fc700 1 -- 172.21.15.46:0/2653501388 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019400f040 con 0x7f019c159bd0 2024-01-26T23:43:01.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.895+0000 7f01a43dc700 1 -- 172.21.15.46:0/2653501388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f019c0b2030 con 0x7f019c159bd0 2024-01-26T23:43:01.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.895+0000 7f01927fc700 1 -- 172.21.15.46:0/2653501388 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019400ccf0 con 0x7f019c159bd0 2024-01-26T23:43:01.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.895+0000 7f01a43dc700 1 -- 172.21.15.46:0/2653501388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f019c14e9f0 con 0x7f019c159bd0 2024-01-26T23:43:01.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.896+0000 7f01a43dc700 1 -- 172.21.15.46:0/2653501388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f019c04c710 con 0x7f019c159bd0 2024-01-26T23:43:01.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.897+0000 7f01927fc700 1 -- 172.21.15.46:0/2653501388 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0194014690 con 0x7f019c159bd0 2024-01-26T23:43:01.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.897+0000 7f01927fc700 1 --2- 172.21.15.46:0/2653501388 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0180066f30 0x7f01800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:01.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.897+0000 7f01927fc700 1 -- 172.21.15.46:0/2653501388 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f0194085260 con 0x7f019c159bd0 2024-01-26T23:43:01.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.897+0000 7f01a2178700 1 --2- 172.21.15.46:0/2653501388 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0180066f30 0x7f01800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:01.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.898+0000 7f01a2178700 1 --2- 172.21.15.46:0/2653501388 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0180066f30 0x7f01800693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0198001360 tx=0x7f0198009040 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:01.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:01.902+0000 7f01927fc700 1 -- 172.21.15.46:0/2653501388 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0194052220 con 0x7f019c159bd0 2024-01-26T23:43:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.105+0000 7f01a43dc700 1 -- 172.21.15.46:0/2653501388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f019c0c96e0 con 0x7f019c159bd0 2024-01-26T23:43:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.106+0000 7f01927fc700 1 -- 172.21.15.46:0/2653501388 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16118 (secure 0 0 0) 0x7f0194055840 con 0x7f019c159bd0 2024-01-26T23:43:02.108 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:43:02.108 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":69,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:43:01.143792+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:42:59.650806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-01-26T23:43:00.105558+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:43:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.109+0000 7f01a43dc700 1 -- 172.21.15.46:0/2653501388 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0180066f30 msgr2=0x7f01800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.109+0000 7f01a43dc700 1 --2- 172.21.15.46:0/2653501388 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0180066f30 0x7f01800693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0198001360 tx=0x7f0198009040 comp rx=0 tx=0).stop 2024-01-26T23:43:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.109+0000 7f01a43dc700 1 -- 172.21.15.46:0/2653501388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f019c159bd0 msgr2=0x7f019c1562d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.109+0000 7f01a43dc700 1 --2- 172.21.15.46:0/2653501388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f019c159bd0 0x7f019c1562d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0194009d40 tx=0x7f019400b040 comp rx=0 tx=0).stop 2024-01-26T23:43:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.109+0000 7f01a43dc700 1 -- 172.21.15.46:0/2653501388 shutdown_connections 2024-01-26T23:43:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.109+0000 7f01a43dc700 1 --2- 172.21.15.46:0/2653501388 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0180066f30 0x7f01800693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.109+0000 7f01a43dc700 1 --2- 172.21.15.46:0/2653501388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f019c159bd0 0x7f019c1562d0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.109+0000 7f01a43dc700 1 --2- 172.21.15.46:0/2653501388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f019c0c4440 0x7f019c14e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.109+0000 7f01a43dc700 1 --2- 172.21.15.46:0/2653501388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f019c159800 0x7f019c155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.109+0000 7f01a43dc700 1 -- 172.21.15.46:0/2653501388 >> 172.21.15.46:0/2653501388 conn(0x7f019c097820 msgr2=0x7f019c068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.109+0000 7f01a43dc700 1 -- 172.21.15.46:0/2653501388 shutdown_connections 2024-01-26T23:43:02.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.109+0000 7f01a43dc700 1 -- 172.21.15.46:0/2653501388 wait complete. 2024-01-26T23:43:02.124 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:43:02.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.254+0000 7ff04d9d9700 1 Processor -- start 2024-01-26T23:43:02.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.254+0000 7ff04d9d9700 1 -- start start 2024-01-26T23:43:02.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.255+0000 7ff04d9d9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff0480cbdf0 0x7ff0480ae610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:02.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.255+0000 7ff04d9d9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff0480ad070 0x7ff0480c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:02.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.255+0000 7ff04d9d9700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0480ae180 con 0x7ff0480cc1c0 2024-01-26T23:43:02.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.255+0000 7ff04d9d9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0480aeb50 con 0x7ff0480ad070 2024-01-26T23:43:02.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.255+0000 7ff04d9d9700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff048156c60 con 0x7ff0480cbdf0 2024-01-26T23:43:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.255+0000 7ff0477fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff0480ad070 0x7ff0480c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.255+0000 7ff0477fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff0480ad070 0x7ff0480c98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50672/0 (socket says 172.21.15.46:50672) 2024-01-26T23:43:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.255+0000 7ff0467fc700 1 --1- 172.21.15.46:0/3747041126 >> v1:172.21.15.46:6789/0 conn(0x7ff0480cc1c0 0x7ff0480b0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37750/0 (socket says 172.21.15.46:37750) 2024-01-26T23:43:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.255+0000 7ff0477fe700 1 -- 172.21.15.46:0/3747041126 learned_addr learned my addr 172.21.15.46:0/3747041126 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.255+0000 7ff046ffd700 1 --2- 172.21.15.46:0/3747041126 >> v2:172.21.15.106:3300/0 conn(0x7ff0480cbdf0 0x7ff0480ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.256+0000 7ff0477fe700 1 -- 172.21.15.46:0/3747041126 >> v2:172.21.15.106:3300/0 conn(0x7ff0480cbdf0 msgr2=0x7ff0480ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.256+0000 7ff0477fe700 1 --2- 172.21.15.46:0/3747041126 >> v2:172.21.15.106:3300/0 conn(0x7ff0480cbdf0 0x7ff0480ae610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.256+0000 7ff0477fe700 1 -- 172.21.15.46:0/3747041126 >> v1:172.21.15.46:6789/0 conn(0x7ff0480cc1c0 legacy=0x7ff0480b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.256+0000 7ff0477fe700 1 -- 172.21.15.46:0/3747041126 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0480ad570 con 0x7ff0480ad070 2024-01-26T23:43:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.256+0000 7ff045ffb700 1 -- 172.21.15.46:0/3747041126 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921859798 0 0) 0x7ff0480ae180 con 0x7ff0480cc1c0 2024-01-26T23:43:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.256+0000 7ff0477fe700 1 --2- 172.21.15.46:0/3747041126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0480ad070 0x7ff0480c98b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff04806ee20 tx=0x7ff040009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51711065661407fa server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.256+0000 7ff045ffb700 1 -- 172.21.15.46:0/3747041126 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04000e070 con 0x7ff0480ad070 2024-01-26T23:43:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.256+0000 7ff045ffb700 1 -- 172.21.15.46:0/3747041126 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff04000ba20 con 0x7ff0480ad070 2024-01-26T23:43:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.256+0000 7ff045ffb700 1 -- 172.21.15.46:0/3747041126 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0400092e0 con 0x7ff0480ad070 2024-01-26T23:43:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.257+0000 7ff04d9d9700 1 -- 172.21.15.46:0/3747041126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0480ad070 msgr2=0x7ff0480c98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:02.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.257+0000 7ff04d9d9700 1 --2- 172.21.15.46:0/3747041126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0480ad070 0x7ff0480c98b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff04806ee20 tx=0x7ff040009840 comp rx=0 tx=0).stop 2024-01-26T23:43:02.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.257+0000 7ff04d9d9700 1 -- 172.21.15.46:0/3747041126 shutdown_connections 2024-01-26T23:43:02.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.257+0000 7ff04d9d9700 1 --2- 172.21.15.46:0/3747041126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0480ad070 0x7ff0480c98b0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.257+0000 7ff04d9d9700 1 --2- 172.21.15.46:0/3747041126 >> v2:172.21.15.106:3300/0 conn(0x7ff0480cbdf0 0x7ff0480ae610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.257+0000 7ff04d9d9700 1 -- 172.21.15.46:0/3747041126 >> 172.21.15.46:0/3747041126 conn(0x7ff048068400 msgr2=0x7ff0480cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:02.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.257+0000 7ff04d9d9700 1 -- 172.21.15.46:0/3747041126 shutdown_connections 2024-01-26T23:43:02.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.257+0000 7ff04d9d9700 1 -- 172.21.15.46:0/3747041126 wait complete. 2024-01-26T23:43:02.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.258+0000 7ff04d9d9700 1 Processor -- start 2024-01-26T23:43:02.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.258+0000 7ff04d9d9700 1 -- start start 2024-01-26T23:43:02.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.259+0000 7ff04d9d9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0480ad070 0x7ff048165e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:02.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.259+0000 7ff04d9d9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0480cbdf0 0x7ff0481460d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:02.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.259+0000 7ff04d9d9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff048168c40 0x7ff04806a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:02.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.259+0000 7ff04d9d9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff048146610 con 0x7ff0480ad070 2024-01-26T23:43:02.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.259+0000 7ff04d9d9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04816a3b0 con 0x7ff0480cbdf0 2024-01-26T23:43:02.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.259+0000 7ff04d9d9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0481697a0 con 0x7ff048168c40 2024-01-26T23:43:02.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.259+0000 7ff046ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0480ad070 0x7ff048165e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:02.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.259+0000 7ff046ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0480ad070 0x7ff048165e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50674/0 (socket says 172.21.15.46:50674) 2024-01-26T23:43:02.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.259+0000 7ff046ffd700 1 -- 172.21.15.46:0/2576089048 learned_addr learned my addr 172.21.15.46:0/2576089048 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:02.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.259+0000 7ff0467fc700 1 --2- 172.21.15.46:0/2576089048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0480cbdf0 0x7ff0481460d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:02.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.259+0000 7ff0477fe700 1 --2- 172.21.15.46:0/2576089048 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff048168c40 0x7ff04806a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:02.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.259+0000 7ff046ffd700 1 -- 172.21.15.46:0/2576089048 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff048168c40 msgr2=0x7ff04806a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:02.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.259+0000 7ff046ffd700 1 --2- 172.21.15.46:0/2576089048 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff048168c40 0x7ff04806a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.259+0000 7ff046ffd700 1 -- 172.21.15.46:0/2576089048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0480cbdf0 msgr2=0x7ff0481460d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:02.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.259+0000 7ff046ffd700 1 --2- 172.21.15.46:0/2576089048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0480cbdf0 0x7ff0481460d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.259+0000 7ff046ffd700 1 -- 172.21.15.46:0/2576089048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff040009ae0 con 0x7ff0480ad070 2024-01-26T23:43:02.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.259+0000 7ff0467fc700 1 --2- 172.21.15.46:0/2576089048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0480cbdf0 0x7ff0481460d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:43:02.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.260+0000 7ff046ffd700 1 --2- 172.21.15.46:0/2576089048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0480ad070 0x7ff048165e30 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff03000ec80 tx=0x7ff0300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:02.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.260+0000 7ff03effd700 1 -- 172.21.15.46:0/2576089048 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03000ca00 con 0x7ff0480ad070 2024-01-26T23:43:02.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.260+0000 7ff03effd700 1 -- 172.21.15.46:0/2576089048 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03000cb60 con 0x7ff0480ad070 2024-01-26T23:43:02.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.260+0000 7ff04d9d9700 1 -- 172.21.15.46:0/2576089048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff048169a80 con 0x7ff0480ad070 2024-01-26T23:43:02.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.260+0000 7ff03effd700 1 -- 172.21.15.46:0/2576089048 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff030013470 con 0x7ff0480ad070 2024-01-26T23:43:02.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.260+0000 7ff04d9d9700 1 -- 172.21.15.46:0/2576089048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04816a040 con 0x7ff0480ad070 2024-01-26T23:43:02.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.261+0000 7ff04d9d9700 1 -- 172.21.15.46:0/2576089048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff04804c710 con 0x7ff0480ad070 2024-01-26T23:43:02.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.261+0000 7ff03effd700 1 -- 172.21.15.46:0/2576089048 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff0300137e0 con 0x7ff0480ad070 2024-01-26T23:43:02.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.262+0000 7ff03effd700 1 --2- 172.21.15.46:0/2576089048 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff028066f30 0x7ff0280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:02.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.262+0000 7ff03effd700 1 -- 172.21.15.46:0/2576089048 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6389+0+0 (secure 0 0 0) 0x7ff030084bc0 con 0x7ff0480ad070 2024-01-26T23:43:02.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.262+0000 7ff0467fc700 1 --2- 172.21.15.46:0/2576089048 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff028066f30 0x7ff0280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:02.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.262+0000 7ff0467fc700 1 --2- 172.21.15.46:0/2576089048 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff028066f30 0x7ff0280693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff0480c82a0 tx=0x7ff038009040 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:02.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.266+0000 7ff03effd700 1 -- 172.21.15.46:0/2576089048 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff030051b80 con 0x7ff0480ad070 2024-01-26T23:43:02.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.473+0000 7ff04d9d9700 1 -- 172.21.15.46:0/2576089048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff04816a690 con 0x7ff0480ad070 2024-01-26T23:43:02.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.474+0000 7ff03effd700 1 -- 172.21.15.46:0/2576089048 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16118 (secure 0 0 0) 0x7ff03006be60 con 0x7ff0480ad070 2024-01-26T23:43:02.475 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:43:02.475 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":69,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:43:01.143792+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:42:59.650806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-01-26T23:43:00.105558+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:43:02.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.476+0000 7ff04d9d9700 1 -- 172.21.15.46:0/2576089048 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff028066f30 msgr2=0x7ff0280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:02.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.476+0000 7ff04d9d9700 1 --2- 172.21.15.46:0/2576089048 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff028066f30 0x7ff0280693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff0480c82a0 tx=0x7ff038009040 comp rx=0 tx=0).stop 2024-01-26T23:43:02.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.476+0000 7ff04d9d9700 1 -- 172.21.15.46:0/2576089048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0480ad070 msgr2=0x7ff048165e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:02.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.476+0000 7ff04d9d9700 1 --2- 172.21.15.46:0/2576089048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0480ad070 0x7ff048165e30 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff03000ec80 tx=0x7ff0300091a0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.476+0000 7ff04d9d9700 1 -- 172.21.15.46:0/2576089048 shutdown_connections 2024-01-26T23:43:02.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.476+0000 7ff04d9d9700 1 --2- 172.21.15.46:0/2576089048 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff028066f30 0x7ff0280693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.476+0000 7ff04d9d9700 1 --2- 172.21.15.46:0/2576089048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0480ad070 0x7ff048165e30 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.476+0000 7ff04d9d9700 1 --2- 172.21.15.46:0/2576089048 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff048168c40 0x7ff04806a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.476+0000 7ff04d9d9700 1 --2- 172.21.15.46:0/2576089048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0480cbdf0 0x7ff0481460d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.476+0000 7ff04d9d9700 1 -- 172.21.15.46:0/2576089048 >> 172.21.15.46:0/2576089048 conn(0x7ff048068400 msgr2=0x7ff048155e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:02.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.476+0000 7ff04d9d9700 1 -- 172.21.15.46:0/2576089048 shutdown_connections 2024-01-26T23:43:02.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.476+0000 7ff04d9d9700 1 -- 172.21.15.46:0/2576089048 wait complete. 2024-01-26T23:43:02.491 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-26T23:43:02.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.626+0000 7fa078723700 1 Processor -- start 2024-01-26T23:43:02.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.626+0000 7fa078723700 1 -- start start 2024-01-26T23:43:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.627+0000 7fa078723700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0700c8da0 0x7fa0700c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.627+0000 7fa078723700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa070140af0 0x7fa070140f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.627+0000 7fa078723700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa0700caf30 0x7fa0700cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.627+0000 7fa078723700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0700cbbf0 con 0x7fa0700caf30 2024-01-26T23:43:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.627+0000 7fa078723700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0700ccc00 con 0x7fa070140af0 2024-01-26T23:43:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.627+0000 7fa078723700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0700cc310 con 0x7fa0700c8da0 2024-01-26T23:43:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.627+0000 7fa076cc0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa0700caf30 0x7fa0700cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.627+0000 7fa076cc0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa0700caf30 0x7fa0700cb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50676/0 (socket says 172.21.15.46:50676) 2024-01-26T23:43:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.627+0000 7fa076cc0700 1 -- 172.21.15.46:0/1211419330 learned_addr learned my addr 172.21.15.46:0/1211419330 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.627+0000 7fa0764bf700 1 --2- 172.21.15.46:0/1211419330 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0700c8da0 0x7fa0700c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.627+0000 7fa075cbe700 1 --2- 172.21.15.46:0/1211419330 >> v2:172.21.15.106:3300/0 conn(0x7fa070140af0 0x7fa070140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.627+0000 7fa076cc0700 1 -- 172.21.15.46:0/1211419330 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0700c8da0 msgr2=0x7fa0700c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.627+0000 7fa076cc0700 1 --2- 172.21.15.46:0/1211419330 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0700c8da0 0x7fa0700c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.627+0000 7fa076cc0700 1 -- 172.21.15.46:0/1211419330 >> v2:172.21.15.106:3300/0 conn(0x7fa070140af0 msgr2=0x7fa070140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.627+0000 7fa076cc0700 1 --2- 172.21.15.46:0/1211419330 >> v2:172.21.15.106:3300/0 conn(0x7fa070140af0 0x7fa070140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.627+0000 7fa076cc0700 1 -- 172.21.15.46:0/1211419330 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0700cc5b0 con 0x7fa0700caf30 2024-01-26T23:43:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.628+0000 7fa076cc0700 1 --2- 172.21.15.46:0/1211419330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0700caf30 0x7fa0700cb370 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa07006f9a0 tx=0x7fa068009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=795cb6c686562d9c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.628+0000 7fa0754bd700 1 -- 172.21.15.46:0/1211419330 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06800e070 con 0x7fa0700caf30 2024-01-26T23:43:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.628+0000 7fa0754bd700 1 -- 172.21.15.46:0/1211419330 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06800ba50 con 0x7fa0700caf30 2024-01-26T23:43:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.628+0000 7fa0754bd700 1 -- 172.21.15.46:0/1211419330 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0680092e0 con 0x7fa0700caf30 2024-01-26T23:43:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.628+0000 7fa078723700 1 -- 172.21.15.46:0/1211419330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0700caf30 msgr2=0x7fa0700cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.628+0000 7fa078723700 1 --2- 172.21.15.46:0/1211419330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0700caf30 0x7fa0700cb370 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa07006f9a0 tx=0x7fa068009840 comp rx=0 tx=0).stop 2024-01-26T23:43:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.628+0000 7fa078723700 1 -- 172.21.15.46:0/1211419330 shutdown_connections 2024-01-26T23:43:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.628+0000 7fa078723700 1 --2- 172.21.15.46:0/1211419330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0700caf30 0x7fa0700cb370 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.628+0000 7fa078723700 1 --2- 172.21.15.46:0/1211419330 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0700c8da0 0x7fa0700c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.628+0000 7fa078723700 1 --2- 172.21.15.46:0/1211419330 >> v2:172.21.15.106:3300/0 conn(0x7fa070140af0 0x7fa070140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.628+0000 7fa078723700 1 -- 172.21.15.46:0/1211419330 >> 172.21.15.46:0/1211419330 conn(0x7fa07006bd90 msgr2=0x7fa07006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.628+0000 7fa078723700 1 -- 172.21.15.46:0/1211419330 shutdown_connections 2024-01-26T23:43:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.628+0000 7fa078723700 1 -- 172.21.15.46:0/1211419330 wait complete. 2024-01-26T23:43:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.630+0000 7fa078723700 1 Processor -- start 2024-01-26T23:43:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.630+0000 7fa078723700 1 -- start start 2024-01-26T23:43:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.631+0000 7fa078723700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0700c8da0 0x7fa07016a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.631+0000 7fa078723700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0700caf30 0x7fa070069ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.631+0000 7fa078723700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa070140af0 0x7fa070157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.631+0000 7fa078723700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0701572f0 con 0x7fa0700caf30 2024-01-26T23:43:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.631+0000 7fa078723700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07015a630 con 0x7fa0700c8da0 2024-01-26T23:43:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.631+0000 7fa078723700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa07015a060 con 0x7fa070140af0 2024-01-26T23:43:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.631+0000 7fa075cbe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0700caf30 0x7fa070069ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.631+0000 7fa075cbe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0700caf30 0x7fa070069ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50686/0 (socket says 172.21.15.46:50686) 2024-01-26T23:43:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.631+0000 7fa075cbe700 1 -- 172.21.15.46:0/4061160703 learned_addr learned my addr 172.21.15.46:0/4061160703 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.631+0000 7fa0764bf700 1 --2- 172.21.15.46:0/4061160703 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0700c8da0 0x7fa07016a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.631+0000 7fa076cc0700 1 --2- 172.21.15.46:0/4061160703 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa070140af0 0x7fa070157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.632+0000 7fa075cbe700 1 -- 172.21.15.46:0/4061160703 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa070140af0 msgr2=0x7fa070157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.632+0000 7fa075cbe700 1 --2- 172.21.15.46:0/4061160703 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa070140af0 0x7fa070157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.632+0000 7fa075cbe700 1 -- 172.21.15.46:0/4061160703 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0700c8da0 msgr2=0x7fa07016a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.632+0000 7fa075cbe700 1 --2- 172.21.15.46:0/4061160703 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0700c8da0 0x7fa07016a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.632+0000 7fa075cbe700 1 -- 172.21.15.46:0/4061160703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa068009ae0 con 0x7fa0700caf30 2024-01-26T23:43:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.632+0000 7fa075cbe700 1 --2- 172.21.15.46:0/4061160703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0700caf30 0x7fa070069ed0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa06000d860 tx=0x7fa060008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.632+0000 7fa0667fc700 1 -- 172.21.15.46:0/4061160703 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0600125f0 con 0x7fa0700caf30 2024-01-26T23:43:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.632+0000 7fa0667fc700 1 -- 172.21.15.46:0/4061160703 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa060012750 con 0x7fa0700caf30 2024-01-26T23:43:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.632+0000 7fa078723700 1 -- 172.21.15.46:0/4061160703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa07015a2e0 con 0x7fa0700caf30 2024-01-26T23:43:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.632+0000 7fa0667fc700 1 -- 172.21.15.46:0/4061160703 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa060012b10 con 0x7fa0700caf30 2024-01-26T23:43:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.632+0000 7fa078723700 1 -- 172.21.15.46:0/4061160703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0700c67d0 con 0x7fa0700caf30 2024-01-26T23:43:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.633+0000 7fa078723700 1 -- 172.21.15.46:0/4061160703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa07015a7c0 con 0x7fa0700caf30 2024-01-26T23:43:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.634+0000 7fa0667fc700 1 -- 172.21.15.46:0/4061160703 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa06001e460 con 0x7fa0700caf30 2024-01-26T23:43:02.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.634+0000 7fa0667fc700 1 --2- 172.21.15.46:0/4061160703 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa054066f30 0x7fa0540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:02.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.634+0000 7fa0667fc700 1 -- 172.21.15.46:0/4061160703 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fa060084d10 con 0x7fa0700caf30 2024-01-26T23:43:02.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.634+0000 7fa0764bf700 1 --2- 172.21.15.46:0/4061160703 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa054066f30 0x7fa0540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:02.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.635+0000 7fa0764bf700 1 --2- 172.21.15.46:0/4061160703 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa054066f30 0x7fa0540693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa06c006fd0 tx=0x7fa06c008040 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:02.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.639+0000 7fa0667fc700 1 -- 172.21.15.46:0/4061160703 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa060055760 con 0x7fa0700caf30 2024-01-26T23:43:02.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:02.893+0000 7fa078723700 1 -- 172.21.15.46:0/4061160703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa07004c710 con 0x7fa0700caf30 2024-01-26T23:43:03.691 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-01-26T23:43:03.690+0000 7f587d54c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-26T23:43:03.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.718+0000 7fa0667fc700 1 -- 172.21.15.46:0/4061160703 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v26) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa0600552f0 con 0x7fa0700caf30 2024-01-26T23:43:03.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.721+0000 7fa078723700 1 -- 172.21.15.46:0/4061160703 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa054066f30 msgr2=0x7fa0540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:03.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.721+0000 7fa078723700 1 --2- 172.21.15.46:0/4061160703 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa054066f30 0x7fa0540693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa06c006fd0 tx=0x7fa06c008040 comp rx=0 tx=0).stop 2024-01-26T23:43:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.721+0000 7fa078723700 1 -- 172.21.15.46:0/4061160703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0700caf30 msgr2=0x7fa070069ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.721+0000 7fa078723700 1 --2- 172.21.15.46:0/4061160703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0700caf30 0x7fa070069ed0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa06000d860 tx=0x7fa060008040 comp rx=0 tx=0).stop 2024-01-26T23:43:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.721+0000 7fa078723700 1 -- 172.21.15.46:0/4061160703 shutdown_connections 2024-01-26T23:43:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.721+0000 7fa078723700 1 --2- 172.21.15.46:0/4061160703 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa054066f30 0x7fa0540693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.721+0000 7fa078723700 1 --2- 172.21.15.46:0/4061160703 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa070140af0 0x7fa070157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.721+0000 7fa078723700 1 --2- 172.21.15.46:0/4061160703 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0700c8da0 0x7fa07016a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.721+0000 7fa078723700 1 --2- 172.21.15.46:0/4061160703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0700caf30 0x7fa070069ed0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.721+0000 7fa078723700 1 -- 172.21.15.46:0/4061160703 >> 172.21.15.46:0/4061160703 conn(0x7fa07006bd90 msgr2=0x7fa0700c4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.721+0000 7fa078723700 1 -- 172.21.15.46:0/4061160703 shutdown_connections 2024-01-26T23:43:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.721+0000 7fa078723700 1 -- 172.21.15.46:0/4061160703 wait complete. 2024-01-26T23:43:03.738 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-26T23:43:03.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.877+0000 7f97f686c700 1 Processor -- start 2024-01-26T23:43:03.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.877+0000 7f97f686c700 1 -- start start 2024-01-26T23:43:03.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.878+0000 7f97f686c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97f0067af0 0x7f97f0067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:03.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.878+0000 7f97f686c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97f006ebe0 0x7f97f00ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:03.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.878+0000 7f97f686c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f97f01597e0 0x7f97f0159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:03.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.878+0000 7f97f686c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f006fdb0 con 0x7f97f0067af0 2024-01-26T23:43:03.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.878+0000 7f97f686c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f00ade10 con 0x7f97f01597e0 2024-01-26T23:43:03.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.878+0000 7f97f686c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f006f1a0 con 0x7f97f006ebe0 2024-01-26T23:43:03.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.878+0000 7f97effff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97f0067af0 0x7f97f0067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:03.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.878+0000 7f97effff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97f0067af0 0x7f97f0067ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50688/0 (socket says 172.21.15.46:50688) 2024-01-26T23:43:03.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.878+0000 7f97effff700 1 -- 172.21.15.46:0/104666839 learned_addr learned my addr 172.21.15.46:0/104666839 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:03.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.878+0000 7f97f4e09700 1 --2- 172.21.15.46:0/104666839 >> v2:172.21.15.106:3300/0 conn(0x7f97f01597e0 0x7f97f0159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:03.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.878+0000 7f97ef7fe700 1 --2- 172.21.15.46:0/104666839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97f006ebe0 0x7f97f00ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:03.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.878+0000 7f97effff700 1 -- 172.21.15.46:0/104666839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97f006ebe0 msgr2=0x7f97f00ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:03.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.878+0000 7f97effff700 1 --2- 172.21.15.46:0/104666839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97f006ebe0 0x7f97f00ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:03.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.878+0000 7f97effff700 1 -- 172.21.15.46:0/104666839 >> v2:172.21.15.106:3300/0 conn(0x7f97f01597e0 msgr2=0x7f97f0159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:03.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.878+0000 7f97effff700 1 --2- 172.21.15.46:0/104666839 >> v2:172.21.15.106:3300/0 conn(0x7f97f01597e0 0x7f97f0159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:03.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.878+0000 7f97effff700 1 -- 172.21.15.46:0/104666839 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f006f480 con 0x7f97f0067af0 2024-01-26T23:43:03.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.878+0000 7f97f4e09700 1 --2- 172.21.15.46:0/104666839 >> v2:172.21.15.106:3300/0 conn(0x7f97f01597e0 0x7f97f0159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:43:03.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.879+0000 7f97effff700 1 --2- 172.21.15.46:0/104666839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f0067af0 0x7f97f0067ec0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f97f00ce590 tx=0x7f97e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f7b14cd9556882 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:03.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.879+0000 7f97eeffd700 1 -- 172.21.15.46:0/104666839 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e000b860 con 0x7f97f0067af0 2024-01-26T23:43:03.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.879+0000 7f97eeffd700 1 -- 172.21.15.46:0/104666839 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e000b9c0 con 0x7f97f0067af0 2024-01-26T23:43:03.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.879+0000 7f97eeffd700 1 -- 172.21.15.46:0/104666839 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e000bd80 con 0x7f97f0067af0 2024-01-26T23:43:03.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.879+0000 7f97f686c700 1 -- 172.21.15.46:0/104666839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f0067af0 msgr2=0x7f97f0067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:03.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.879+0000 7f97f686c700 1 --2- 172.21.15.46:0/104666839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f0067af0 0x7f97f0067ec0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f97f00ce590 tx=0x7f97e0009250 comp rx=0 tx=0).stop 2024-01-26T23:43:03.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.879+0000 7f97f686c700 1 -- 172.21.15.46:0/104666839 shutdown_connections 2024-01-26T23:43:03.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.879+0000 7f97f686c700 1 --2- 172.21.15.46:0/104666839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f0067af0 0x7f97f0067ec0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:03.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.879+0000 7f97f686c700 1 --2- 172.21.15.46:0/104666839 >> v2:172.21.15.106:3300/0 conn(0x7f97f01597e0 0x7f97f0159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:03.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.879+0000 7f97f686c700 1 --2- 172.21.15.46:0/104666839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97f006ebe0 0x7f97f00ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:03.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.879+0000 7f97f686c700 1 -- 172.21.15.46:0/104666839 >> 172.21.15.46:0/104666839 conn(0x7f97f00cdf80 msgr2=0x7f97f006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:03.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.879+0000 7f97f686c700 1 -- 172.21.15.46:0/104666839 shutdown_connections 2024-01-26T23:43:03.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.879+0000 7f97f686c700 1 -- 172.21.15.46:0/104666839 wait complete. 2024-01-26T23:43:03.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.881+0000 7f97f686c700 1 Processor -- start 2024-01-26T23:43:03.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.881+0000 7f97f686c700 1 -- start start 2024-01-26T23:43:03.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97f686c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97f0067af0 0x7f97f014ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:03.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97f686c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97f006ebe0 0x7f97f0169dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:03.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97f686c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f01597e0 0x7f97f00b17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:03.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97f686c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f016a620 con 0x7f97f01597e0 2024-01-26T23:43:03.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97f686c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f016a900 con 0x7f97f0067af0 2024-01-26T23:43:03.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97f686c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f016abe0 con 0x7f97f006ebe0 2024-01-26T23:43:03.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97f4e09700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f01597e0 0x7f97f00b17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:03.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97f4e09700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f01597e0 0x7f97f00b17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50696/0 (socket says 172.21.15.46:50696) 2024-01-26T23:43:03.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97f4e09700 1 -- 172.21.15.46:0/2206015805 learned_addr learned my addr 172.21.15.46:0/2206015805 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:03.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97ef7fe700 1 --2- 172.21.15.46:0/2206015805 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97f006ebe0 0x7f97f0169dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:03.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97effff700 1 --2- 172.21.15.46:0/2206015805 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97f0067af0 0x7f97f014ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:03.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97f4e09700 1 -- 172.21.15.46:0/2206015805 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97f006ebe0 msgr2=0x7f97f0169dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:03.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97f4e09700 1 --2- 172.21.15.46:0/2206015805 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97f006ebe0 0x7f97f0169dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:03.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97f4e09700 1 -- 172.21.15.46:0/2206015805 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97f0067af0 msgr2=0x7f97f014ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:03.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97f4e09700 1 --2- 172.21.15.46:0/2206015805 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97f0067af0 0x7f97f014ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:03.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97f4e09700 1 -- 172.21.15.46:0/2206015805 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e00094f0 con 0x7f97f01597e0 2024-01-26T23:43:03.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97f4e09700 1 --2- 172.21.15.46:0/2206015805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f01597e0 0x7f97f00b17e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f97e800cef0 tx=0x7f97e8007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:03.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97d7fff700 1 -- 172.21.15.46:0/2206015805 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e800ae00 con 0x7f97f01597e0 2024-01-26T23:43:03.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97d7fff700 1 -- 172.21.15.46:0/2206015805 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e8016420 con 0x7f97f01597e0 2024-01-26T23:43:03.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97d7fff700 1 -- 172.21.15.46:0/2206015805 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e80167e0 con 0x7f97f01597e0 2024-01-26T23:43:03.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97f686c700 1 -- 172.21.15.46:0/2206015805 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f016af30 con 0x7f97f01597e0 2024-01-26T23:43:03.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.882+0000 7f97f686c700 1 -- 172.21.15.46:0/2206015805 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f014cf40 con 0x7f97f01597e0 2024-01-26T23:43:03.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.884+0000 7f97f686c700 1 -- 172.21.15.46:0/2206015805 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f004c710 con 0x7f97f01597e0 2024-01-26T23:43:03.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.884+0000 7f97d7fff700 1 -- 172.21.15.46:0/2206015805 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f97e8016bd0 con 0x7f97f01597e0 2024-01-26T23:43:03.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.884+0000 7f97d7fff700 1 --2- 172.21.15.46:0/2206015805 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f97dc066f30 0x7f97dc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:03.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.884+0000 7f97d7fff700 1 -- 172.21.15.46:0/2206015805 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f97e8012070 con 0x7f97f01597e0 2024-01-26T23:43:03.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.885+0000 7f97effff700 1 --2- 172.21.15.46:0/2206015805 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f97dc066f30 0x7f97dc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:03.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.885+0000 7f97effff700 1 --2- 172.21.15.46:0/2206015805 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f97dc066f30 0x7f97dc0693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f97f00ce590 tx=0x7f97e0017000 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:03.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:03.889+0000 7f97d7fff700 1 -- 172.21.15.46:0/2206015805 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f97e804e5a0 con 0x7f97f01597e0 2024-01-26T23:43:04.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.142+0000 7f97f686c700 1 -- 172.21.15.46:0/2206015805 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f97f0068690 con 0x7f97f01597e0 2024-01-26T23:43:04.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.720+0000 7f97d7fff700 1 -- 172.21.15.46:0/2206015805 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v27) v1 ==== 122+0+0 (secure 0 0 0) 0x7f97e8052140 con 0x7f97f01597e0 2024-01-26T23:43:04.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.723+0000 7f97f686c700 1 -- 172.21.15.46:0/2206015805 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f97dc066f30 msgr2=0x7f97dc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:04.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.723+0000 7f97f686c700 1 --2- 172.21.15.46:0/2206015805 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f97dc066f30 0x7f97dc0693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f97f00ce590 tx=0x7f97e0017000 comp rx=0 tx=0).stop 2024-01-26T23:43:04.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.723+0000 7f97f686c700 1 -- 172.21.15.46:0/2206015805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f01597e0 msgr2=0x7f97f00b17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:04.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.723+0000 7f97f686c700 1 --2- 172.21.15.46:0/2206015805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f01597e0 0x7f97f00b17e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f97e800cef0 tx=0x7f97e8007860 comp rx=0 tx=0).stop 2024-01-26T23:43:04.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.723+0000 7f97f686c700 1 -- 172.21.15.46:0/2206015805 shutdown_connections 2024-01-26T23:43:04.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.723+0000 7f97f686c700 1 --2- 172.21.15.46:0/2206015805 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f97dc066f30 0x7f97dc0693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:04.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.723+0000 7f97f686c700 1 --2- 172.21.15.46:0/2206015805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f01597e0 0x7f97f00b17e0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:04.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.723+0000 7f97f686c700 1 --2- 172.21.15.46:0/2206015805 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97f0067af0 0x7f97f014ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:04.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.723+0000 7f97f686c700 1 --2- 172.21.15.46:0/2206015805 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97f006ebe0 0x7f97f0169dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:04.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.723+0000 7f97f686c700 1 -- 172.21.15.46:0/2206015805 >> 172.21.15.46:0/2206015805 conn(0x7f97f00cdf80 msgr2=0x7f97f0155250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:04.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.723+0000 7f97f686c700 1 -- 172.21.15.46:0/2206015805 shutdown_connections 2024-01-26T23:43:04.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.724+0000 7f97f686c700 1 -- 172.21.15.46:0/2206015805 wait complete. 2024-01-26T23:43:04.740 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:43:04.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.875+0000 7facd89e3700 1 Processor -- start 2024-01-26T23:43:04.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.875+0000 7facd89e3700 1 -- start start 2024-01-26T23:43:04.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.876+0000 7facd89e3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7facd41596e0 0x7facd4159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.877+0000 7facd89e3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7facd40c4720 0x7facd4068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.877+0000 7facd89e3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd40c97f0 con 0x7facd40cadb0 2024-01-26T23:43:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.877+0000 7facd89e3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd40cbf90 con 0x7facd41596e0 2024-01-26T23:43:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.877+0000 7facd89e3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd40c8c20 con 0x7facd40c4720 2024-01-26T23:43:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.877+0000 7facd259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7facd41596e0 0x7facd4159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.877+0000 7facd259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7facd41596e0 0x7facd4159ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50708/0 (socket says 172.21.15.46:50708) 2024-01-26T23:43:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.877+0000 7facd259c700 1 -- 172.21.15.46:0/1191765645 learned_addr learned my addr 172.21.15.46:0/1191765645 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.877+0000 7facd1d9b700 1 --2- 172.21.15.46:0/1191765645 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7facd40c4720 0x7facd4068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.877+0000 7facd259c700 1 -- 172.21.15.46:0/1191765645 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7facd40c4720 msgr2=0x7facd4068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.877+0000 7facd259c700 1 --2- 172.21.15.46:0/1191765645 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7facd40c4720 0x7facd4068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.877+0000 7facd259c700 1 -- 172.21.15.46:0/1191765645 >> v1:172.21.15.106:6789/0 conn(0x7facd40cadb0 legacy=0x7facd40c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.877+0000 7facd259c700 1 -- 172.21.15.46:0/1191765645 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd40c8ec0 con 0x7facd41596e0 2024-01-26T23:43:04.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.877+0000 7facd259c700 1 --2- 172.21.15.46:0/1191765645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facd41596e0 0x7facd4159ab0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7facc40099b0 tx=0x7facc4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2686d43921932e4b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:04.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.877+0000 7facd159a700 1 -- 172.21.15.46:0/1191765645 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc4009ea0 con 0x7facd41596e0 2024-01-26T23:43:04.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.878+0000 7facd159a700 1 -- 172.21.15.46:0/1191765645 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc400b860 con 0x7facd41596e0 2024-01-26T23:43:04.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.878+0000 7facd159a700 1 -- 172.21.15.46:0/1191765645 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc400bc00 con 0x7facd41596e0 2024-01-26T23:43:04.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.878+0000 7facd89e3700 1 -- 172.21.15.46:0/1191765645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facd41596e0 msgr2=0x7facd4159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:04.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.878+0000 7facd89e3700 1 --2- 172.21.15.46:0/1191765645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facd41596e0 0x7facd4159ab0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7facc40099b0 tx=0x7facc4009250 comp rx=0 tx=0).stop 2024-01-26T23:43:04.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.878+0000 7facd89e3700 1 -- 172.21.15.46:0/1191765645 shutdown_connections 2024-01-26T23:43:04.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.878+0000 7facd89e3700 1 --2- 172.21.15.46:0/1191765645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facd41596e0 0x7facd4159ab0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:04.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.878+0000 7facd89e3700 1 --2- 172.21.15.46:0/1191765645 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7facd40c4720 0x7facd4068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:04.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.878+0000 7facd89e3700 1 -- 172.21.15.46:0/1191765645 >> 172.21.15.46:0/1191765645 conn(0x7facd4097820 msgr2=0x7facd4068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:04.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.878+0000 7facd89e3700 1 -- 172.21.15.46:0/1191765645 shutdown_connections 2024-01-26T23:43:04.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.878+0000 7facd89e3700 1 -- 172.21.15.46:0/1191765645 wait complete. 2024-01-26T23:43:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.879+0000 7facd89e3700 1 Processor -- start 2024-01-26T23:43:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.880+0000 7facd89e3700 1 -- start start 2024-01-26T23:43:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.880+0000 7facd89e3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7facd40c4720 0x7facd414e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.880+0000 7facd89e3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facd40cadb0 0x7facd4169ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.880+0000 7facd89e3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7facd41596e0 0x7facd40b1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.880+0000 7facd89e3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd416a430 con 0x7facd40cadb0 2024-01-26T23:43:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.880+0000 7facd89e3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd416aee0 con 0x7facd40c4720 2024-01-26T23:43:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.880+0000 7facd89e3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd40b1c80 con 0x7facd41596e0 2024-01-26T23:43:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.880+0000 7facd1d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facd40cadb0 0x7facd4169ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.880+0000 7facd1d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facd40cadb0 0x7facd4169ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50710/0 (socket says 172.21.15.46:50710) 2024-01-26T23:43:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.880+0000 7facd259c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7facd40c4720 0x7facd414e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.880+0000 7facd1d9b700 1 -- 172.21.15.46:0/1893993250 learned_addr learned my addr 172.21.15.46:0/1893993250 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.881+0000 7facd2d9d700 1 --2- 172.21.15.46:0/1893993250 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7facd41596e0 0x7facd40b1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.881+0000 7facd1d9b700 1 -- 172.21.15.46:0/1893993250 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7facd41596e0 msgr2=0x7facd40b1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.881+0000 7facd1d9b700 1 --2- 172.21.15.46:0/1893993250 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7facd41596e0 0x7facd40b1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.881+0000 7facd1d9b700 1 -- 172.21.15.46:0/1893993250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7facd40c4720 msgr2=0x7facd414e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.881+0000 7facd1d9b700 1 --2- 172.21.15.46:0/1893993250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7facd40c4720 0x7facd414e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.881+0000 7facd1d9b700 1 -- 172.21.15.46:0/1893993250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc40094f0 con 0x7facd40cadb0 2024-01-26T23:43:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.881+0000 7facd1d9b700 1 --2- 172.21.15.46:0/1893993250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facd40cadb0 0x7facd4169ef0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7facbc00ecb0 tx=0x7facbc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.881+0000 7facca7fc700 1 -- 172.21.15.46:0/1893993250 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc00c7f0 con 0x7facd40cadb0 2024-01-26T23:43:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.881+0000 7facca7fc700 1 -- 172.21.15.46:0/1893993250 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facbc00c950 con 0x7facd40cadb0 2024-01-26T23:43:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.881+0000 7facca7fc700 1 -- 172.21.15.46:0/1893993250 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc00ccf0 con 0x7facd40cadb0 2024-01-26T23:43:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.881+0000 7facd89e3700 1 -- 172.21.15.46:0/1893993250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd4153990 con 0x7facd40cadb0 2024-01-26T23:43:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.881+0000 7facd89e3700 1 -- 172.21.15.46:0/1893993250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd416a920 con 0x7facd40cadb0 2024-01-26T23:43:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.882+0000 7facd89e3700 1 -- 172.21.15.46:0/1893993250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd41537f0 con 0x7facd40cadb0 2024-01-26T23:43:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.883+0000 7facca7fc700 1 -- 172.21.15.46:0/1893993250 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7facbc013680 con 0x7facd40cadb0 2024-01-26T23:43:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.883+0000 7facca7fc700 1 --2- 172.21.15.46:0/1893993250 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7facb4066f30 0x7facb40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.883+0000 7facca7fc700 1 -- 172.21.15.46:0/1893993250 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7facbc084990 con 0x7facd40cadb0 2024-01-26T23:43:04.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.884+0000 7facd259c700 1 --2- 172.21.15.46:0/1893993250 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7facb4066f30 0x7facb40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:04.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.884+0000 7facd259c700 1 --2- 172.21.15.46:0/1893993250 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7facb4066f30 0x7facb40693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7facc40099b0 tx=0x7facc4017000 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:04.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:04.888+0000 7facca7fc700 1 -- 172.21.15.46:0/1893993250 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7facd41537f0 con 0x7facd40cadb0 2024-01-26T23:43:05.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.137+0000 7facd89e3700 1 -- 172.21.15.46:0/1893993250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facd41537f0 con 0x7facd40cadb0 2024-01-26T23:43:05.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.137+0000 7facca7fc700 1 -- 172.21.15.46:0/1893993250 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4603 (secure 0 0 0) 0x7facd41537f0 con 0x7facd40cadb0 2024-01-26T23:43:05.139 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:43:05.139 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3037972557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3037972557},{"type":"v1","addr":"172.21.15.106:6837","nonce":3037972557}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":5701,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3786017823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3786017823},{"type":"v1","addr":"172.21.15.106:6836","nonce":3786017823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":5704,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3329385210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3329385210},{"type":"v1","addr":"172.21.15.46:6835","nonce":3329385210}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:43:01.144129+0000","modified":"2024-01-26T23:43:04.718067+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6833/2801738403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2801738403},{"type":"v1","addr":"172.21.15.46:6833","nonce":2801738403}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2024-01-26T23:43:05.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.139+0000 7facd89e3700 1 -- 172.21.15.46:0/1893993250 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7facb4066f30 msgr2=0x7facb40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:05.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.139+0000 7facd89e3700 1 --2- 172.21.15.46:0/1893993250 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7facb4066f30 0x7facb40693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7facc40099b0 tx=0x7facc4017000 comp rx=0 tx=0).stop 2024-01-26T23:43:05.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.139+0000 7facd89e3700 1 -- 172.21.15.46:0/1893993250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facd40cadb0 msgr2=0x7facd4169ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:05.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.139+0000 7facd89e3700 1 --2- 172.21.15.46:0/1893993250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facd40cadb0 0x7facd4169ef0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7facbc00ecb0 tx=0x7facbc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.140+0000 7facd89e3700 1 -- 172.21.15.46:0/1893993250 shutdown_connections 2024-01-26T23:43:05.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.140+0000 7facd89e3700 1 --2- 172.21.15.46:0/1893993250 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7facb4066f30 0x7facb40693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.140+0000 7facd89e3700 1 --2- 172.21.15.46:0/1893993250 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7facd41596e0 0x7facd40b1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.140+0000 7facd89e3700 1 --2- 172.21.15.46:0/1893993250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7facd40c4720 0x7facd414e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.140+0000 7facd89e3700 1 --2- 172.21.15.46:0/1893993250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facd40cadb0 0x7facd4169ef0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.140+0000 7facd89e3700 1 -- 172.21.15.46:0/1893993250 >> 172.21.15.46:0/1893993250 conn(0x7facd4097820 msgr2=0x7facd4068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:05.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.140+0000 7facd89e3700 1 -- 172.21.15.46:0/1893993250 shutdown_connections 2024-01-26T23:43:05.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.140+0000 7facd89e3700 1 -- 172.21.15.46:0/1893993250 wait complete. 2024-01-26T23:43:05.142 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 27 2024-01-26T23:43:05.155 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:43:05.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.285+0000 7ff5cad14700 1 Processor -- start 2024-01-26T23:43:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.285+0000 7ff5cad14700 1 -- start start 2024-01-26T23:43:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.286+0000 7ff5cad14700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5c415ac70 0x7ff5c40c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.286+0000 7ff5cad14700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c40c44f0 con 0x7ff5c415b040 2024-01-26T23:43:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.286+0000 7ff5cad14700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c40c47d0 con 0x7ff5c406c770 2024-01-26T23:43:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.286+0000 7ff5cad14700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c40c9ab0 con 0x7ff5c415ac70 2024-01-26T23:43:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.286+0000 7ff5c3fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff5c415b040 0x7ff5c406c060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37756/0 (socket says 172.21.15.46:37756) 2024-01-26T23:43:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.286+0000 7ff5c3fff700 1 -- 172.21.15.46:0/2927332904 learned_addr learned my addr 172.21.15.46:0/2927332904 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:43:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.286+0000 7ff5c8ab0700 1 --2- 172.21.15.46:0/2927332904 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5c415ac70 0x7ff5c40c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.287+0000 7ff5c37fe700 1 -- 172.21.15.46:0/2927332904 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245004712 0 0) 0x7ff5c40c44f0 con 0x7ff5c415b040 2024-01-26T23:43:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.287+0000 7ff5c37fe700 1 -- 172.21.15.46:0/2927332904 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5b4003570 con 0x7ff5c415b040 2024-01-26T23:43:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.287+0000 7ff5c37fe700 1 -- 172.21.15.46:0/2927332904 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263227658 0 0) 0x7ff5c40c47d0 con 0x7ff5c406c770 2024-01-26T23:43:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.287+0000 7ff5c8ab0700 1 -- 172.21.15.46:0/2927332904 >> v1:172.21.15.106:6789/0 conn(0x7ff5c406c770 legacy=0x7ff5c40c4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.287+0000 7ff5c8ab0700 1 -- 172.21.15.46:0/2927332904 >> v1:172.21.15.46:6789/0 conn(0x7ff5c415b040 legacy=0x7ff5c406c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.287+0000 7ff5c8ab0700 1 -- 172.21.15.46:0/2927332904 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c406cd90 con 0x7ff5c415ac70 2024-01-26T23:43:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.287+0000 7ff5c8ab0700 1 --2- 172.21.15.46:0/2927332904 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5c415ac70 0x7ff5c40c7a80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8009910 tx=0x7ff5b80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35d75230bf4d8fb1 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.287+0000 7ff5c37fe700 1 -- 172.21.15.46:0/2927332904 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b800b680 con 0x7ff5c415ac70 2024-01-26T23:43:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.287+0000 7ff5c37fe700 1 -- 172.21.15.46:0/2927332904 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b800b7e0 con 0x7ff5c415ac70 2024-01-26T23:43:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.287+0000 7ff5c37fe700 1 -- 172.21.15.46:0/2927332904 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b800bba0 con 0x7ff5c415ac70 2024-01-26T23:43:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.288+0000 7ff5cad14700 1 -- 172.21.15.46:0/2927332904 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5c415ac70 msgr2=0x7ff5c40c7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.288+0000 7ff5cad14700 1 --2- 172.21.15.46:0/2927332904 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5c415ac70 0x7ff5c40c7a80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8009910 tx=0x7ff5b80091a0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.288+0000 7ff5cad14700 1 -- 172.21.15.46:0/2927332904 shutdown_connections 2024-01-26T23:43:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.288+0000 7ff5cad14700 1 --2- 172.21.15.46:0/2927332904 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5c415ac70 0x7ff5c40c7a80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.288+0000 7ff5cad14700 1 -- 172.21.15.46:0/2927332904 >> 172.21.15.46:0/2927332904 conn(0x7ff5c40cdeb0 msgr2=0x7ff5c40c9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.288+0000 7ff5cad14700 1 -- 172.21.15.46:0/2927332904 shutdown_connections 2024-01-26T23:43:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.288+0000 7ff5cad14700 1 -- 172.21.15.46:0/2927332904 wait complete. 2024-01-26T23:43:05.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.289+0000 7ff5cad14700 1 Processor -- start 2024-01-26T23:43:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.289+0000 7ff5cad14700 1 -- start start 2024-01-26T23:43:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.290+0000 7ff5cad14700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c406c770 0x7ff5c4157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.290+0000 7ff5cad14700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5c415ac70 0x7ff5c4169cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.290+0000 7ff5cad14700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5c416a210 0x7ff5c40caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.290+0000 7ff5cad14700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c415b910 con 0x7ff5c406c770 2024-01-26T23:43:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.290+0000 7ff5cad14700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c40cceb0 con 0x7ff5c416a210 2024-01-26T23:43:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.290+0000 7ff5cad14700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c40cba50 con 0x7ff5c415ac70 2024-01-26T23:43:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.290+0000 7ff5c8ab0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c406c770 0x7ff5c4157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.290+0000 7ff5c8ab0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c406c770 0x7ff5c4157260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50722/0 (socket says 172.21.15.46:50722) 2024-01-26T23:43:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.290+0000 7ff5c8ab0700 1 -- 172.21.15.46:0/3361693109 learned_addr learned my addr 172.21.15.46:0/3361693109 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.290+0000 7ff5c92b1700 1 --2- 172.21.15.46:0/3361693109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5c416a210 0x7ff5c40caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.290+0000 7ff5c8ab0700 1 -- 172.21.15.46:0/3361693109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5c415ac70 msgr2=0x7ff5c4169cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.290+0000 7ff5c3fff700 1 --2- 172.21.15.46:0/3361693109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5c415ac70 0x7ff5c4169cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.291+0000 7ff5c8ab0700 1 --2- 172.21.15.46:0/3361693109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5c415ac70 0x7ff5c4169cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.291+0000 7ff5c8ab0700 1 -- 172.21.15.46:0/3361693109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5c416a210 msgr2=0x7ff5c40caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.291+0000 7ff5c8ab0700 1 --2- 172.21.15.46:0/3361693109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5c416a210 0x7ff5c40caf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.291+0000 7ff5c8ab0700 1 -- 172.21.15.46:0/3361693109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b8009420 con 0x7ff5c406c770 2024-01-26T23:43:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.291+0000 7ff5c92b1700 1 --2- 172.21.15.46:0/3361693109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5c416a210 0x7ff5c40caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:43:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.291+0000 7ff5c8ab0700 1 --2- 172.21.15.46:0/3361693109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c406c770 0x7ff5c4157260 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff5c40cd220 tx=0x7ff5b8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.291+0000 7ff5c0ff9700 1 -- 172.21.15.46:0/3361693109 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b800b950 con 0x7ff5c406c770 2024-01-26T23:43:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.291+0000 7ff5c0ff9700 1 -- 172.21.15.46:0/3361693109 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b800bad0 con 0x7ff5c406c770 2024-01-26T23:43:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.291+0000 7ff5cad14700 1 -- 172.21.15.46:0/3361693109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c40cbd30 con 0x7ff5c406c770 2024-01-26T23:43:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.291+0000 7ff5c0ff9700 1 -- 172.21.15.46:0/3361693109 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b801a680 con 0x7ff5c406c770 2024-01-26T23:43:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.291+0000 7ff5cad14700 1 -- 172.21.15.46:0/3361693109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c41577a0 con 0x7ff5c406c770 2024-01-26T23:43:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.292+0000 7ff5cad14700 1 -- 172.21.15.46:0/3361693109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c404c710 con 0x7ff5c406c770 2024-01-26T23:43:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.292+0000 7ff5c0ff9700 1 -- 172.21.15.46:0/3361693109 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff5b801aa70 con 0x7ff5c406c770 2024-01-26T23:43:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.293+0000 7ff5c0ff9700 1 --2- 172.21.15.46:0/3361693109 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5a8066f30 0x7ff5a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.293+0000 7ff5c0ff9700 1 -- 172.21.15.46:0/3361693109 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7ff5b8086490 con 0x7ff5c406c770 2024-01-26T23:43:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.293+0000 7ff5c3fff700 1 --2- 172.21.15.46:0/3361693109 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5a8066f30 0x7ff5a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.294+0000 7ff5c3fff700 1 --2- 172.21.15.46:0/3361693109 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5a8066f30 0x7ff5a80693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00058d0 tx=0x7ff5b0009040 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:05.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.297+0000 7ff5c0ff9700 1 -- 172.21.15.46:0/3361693109 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff5b8053450 con 0x7ff5c406c770 2024-01-26T23:43:05.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.503+0000 7ff5cad14700 1 -- 172.21.15.46:0/3361693109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5c415a290 con 0x7ff5c406c770 2024-01-26T23:43:05.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.504+0000 7ff5c0ff9700 1 -- 172.21.15.46:0/3361693109 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16105 (secure 0 0 0) 0x7ff5b8056a70 con 0x7ff5c406c770 2024-01-26T23:43:05.505 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:43:05.506 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":70,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:43:02.599917+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:42:59.650806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-01-26T23:43:00.105558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:43:05.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.506+0000 7ff5cad14700 1 -- 172.21.15.46:0/3361693109 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5a8066f30 msgr2=0x7ff5a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:05.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.506+0000 7ff5cad14700 1 --2- 172.21.15.46:0/3361693109 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5a8066f30 0x7ff5a80693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00058d0 tx=0x7ff5b0009040 comp rx=0 tx=0).stop 2024-01-26T23:43:05.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.506+0000 7ff5cad14700 1 -- 172.21.15.46:0/3361693109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c406c770 msgr2=0x7ff5c4157260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:05.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.506+0000 7ff5cad14700 1 --2- 172.21.15.46:0/3361693109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c406c770 0x7ff5c4157260 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff5c40cd220 tx=0x7ff5b8017000 comp rx=0 tx=0).stop 2024-01-26T23:43:05.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.506+0000 7ff5cad14700 1 -- 172.21.15.46:0/3361693109 shutdown_connections 2024-01-26T23:43:05.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.506+0000 7ff5cad14700 1 --2- 172.21.15.46:0/3361693109 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff5a8066f30 0x7ff5a80693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.506+0000 7ff5cad14700 1 --2- 172.21.15.46:0/3361693109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c406c770 0x7ff5c4157260 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.506+0000 7ff5cad14700 1 --2- 172.21.15.46:0/3361693109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5c416a210 0x7ff5c40caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.506+0000 7ff5cad14700 1 --2- 172.21.15.46:0/3361693109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5c415ac70 0x7ff5c4169cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.506+0000 7ff5cad14700 1 -- 172.21.15.46:0/3361693109 >> 172.21.15.46:0/3361693109 conn(0x7ff5c40cdeb0 msgr2=0x7ff5c40c23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:05.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.506+0000 7ff5cad14700 1 -- 172.21.15.46:0/3361693109 shutdown_connections 2024-01-26T23:43:05.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.506+0000 7ff5cad14700 1 -- 172.21.15.46:0/3361693109 wait complete. 2024-01-26T23:43:05.521 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-26T23:43:05.521 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:43:05.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.651+0000 7ffa8fa27700 1 Processor -- start 2024-01-26T23:43:05.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.651+0000 7ffa8fa27700 1 -- start start 2024-01-26T23:43:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.652+0000 7ffa8fa27700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffa88154290 0x7ffa880b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.652+0000 7ffa8fa27700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa880b2560 con 0x7ffa88153ec0 2024-01-26T23:43:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.652+0000 7ffa8fa27700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa8815b140 con 0x7ffa880b2100 2024-01-26T23:43:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.652+0000 7ffa8fa27700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa8815a5c0 con 0x7ffa88154290 2024-01-26T23:43:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.652+0000 7ffa8cfc2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffa88154290 0x7ffa880b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.652+0000 7ffa8d7c3700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ffa88153ec0 0x7ffa88157630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37760/0 (socket says 172.21.15.46:37760) 2024-01-26T23:43:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.652+0000 7ffa8d7c3700 1 -- 172.21.15.46:0/1100323324 learned_addr learned my addr 172.21.15.46:0/1100323324 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:43:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.653+0000 7ffa8cfc2700 1 -- 172.21.15.46:0/1100323324 >> v1:172.21.15.106:6789/0 conn(0x7ffa880b2100 legacy=0x7ffa880b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.653+0000 7ffa8cfc2700 1 -- 172.21.15.46:0/1100323324 >> v1:172.21.15.46:6789/0 conn(0x7ffa88153ec0 legacy=0x7ffa88157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.653+0000 7ffa7ffff700 1 -- 172.21.15.46:0/1100323324 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140652242 0 0) 0x7ffa84001110 con 0x7ffa88153ec0 2024-01-26T23:43:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.653+0000 7ffa8cfc2700 1 -- 172.21.15.46:0/1100323324 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa8815a8a0 con 0x7ffa88154290 2024-01-26T23:43:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.653+0000 7ffa8cfc2700 1 --2- 172.21.15.46:0/1100323324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa88154290 0x7ffa880b1bc0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ffa780098e0 tx=0x7ffa780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7d5911dad01ba8e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.653+0000 7ffa7ffff700 1 -- 172.21.15.46:0/1100323324 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa78009e80 con 0x7ffa88154290 2024-01-26T23:43:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.653+0000 7ffa7ffff700 1 -- 172.21.15.46:0/1100323324 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7800b860 con 0x7ffa88154290 2024-01-26T23:43:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.653+0000 7ffa7ffff700 1 -- 172.21.15.46:0/1100323324 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7800bbe0 con 0x7ffa88154290 2024-01-26T23:43:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.654+0000 7ffa8fa27700 1 -- 172.21.15.46:0/1100323324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa88154290 msgr2=0x7ffa880b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.654+0000 7ffa8fa27700 1 --2- 172.21.15.46:0/1100323324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa88154290 0x7ffa880b1bc0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ffa780098e0 tx=0x7ffa780091a0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.654+0000 7ffa8fa27700 1 -- 172.21.15.46:0/1100323324 shutdown_connections 2024-01-26T23:43:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.654+0000 7ffa8fa27700 1 --2- 172.21.15.46:0/1100323324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa88154290 0x7ffa880b1bc0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.654+0000 7ffa8fa27700 1 -- 172.21.15.46:0/1100323324 >> 172.21.15.46:0/1100323324 conn(0x7ffa88069510 msgr2=0x7ffa880aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.654+0000 7ffa8fa27700 1 -- 172.21.15.46:0/1100323324 shutdown_connections 2024-01-26T23:43:05.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.654+0000 7ffa8fa27700 1 -- 172.21.15.46:0/1100323324 wait complete. 2024-01-26T23:43:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.655+0000 7ffa8fa27700 1 Processor -- start 2024-01-26T23:43:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.655+0000 7ffa8fa27700 1 -- start start 2024-01-26T23:43:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.656+0000 7ffa8fa27700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa880b2100 0x7ffa88146f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.656+0000 7ffa8fa27700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa88153ec0 0x7ffa880ae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.656+0000 7ffa8fa27700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa88154290 0x7ffa88157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.656+0000 7ffa8fa27700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa880adb70 con 0x7ffa88153ec0 2024-01-26T23:43:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.656+0000 7ffa8fa27700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa881474c0 con 0x7ffa88154290 2024-01-26T23:43:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.656+0000 7ffa8fa27700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa880a9d80 con 0x7ffa880b2100 2024-01-26T23:43:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.656+0000 7ffa8cfc2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa88153ec0 0x7ffa880ae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.656+0000 7ffa8cfc2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa88153ec0 0x7ffa880ae4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50736/0 (socket says 172.21.15.46:50736) 2024-01-26T23:43:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.656+0000 7ffa8cfc2700 1 -- 172.21.15.46:0/988900505 learned_addr learned my addr 172.21.15.46:0/988900505 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.656+0000 7ffa8d7c3700 1 --2- 172.21.15.46:0/988900505 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa880b2100 0x7ffa88146f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.656+0000 7ffa8dfc4700 1 --2- 172.21.15.46:0/988900505 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa88154290 0x7ffa88157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.656+0000 7ffa8cfc2700 1 -- 172.21.15.46:0/988900505 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa880b2100 msgr2=0x7ffa88146f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.656+0000 7ffa8cfc2700 1 --2- 172.21.15.46:0/988900505 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa880b2100 0x7ffa88146f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.656+0000 7ffa8cfc2700 1 -- 172.21.15.46:0/988900505 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa88154290 msgr2=0x7ffa88157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.656+0000 7ffa8cfc2700 1 --2- 172.21.15.46:0/988900505 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa88154290 0x7ffa88157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.657+0000 7ffa8cfc2700 1 -- 172.21.15.46:0/988900505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa78009420 con 0x7ffa88153ec0 2024-01-26T23:43:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.657+0000 7ffa8d7c3700 1 --2- 172.21.15.46:0/988900505 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa880b2100 0x7ffa88146f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:43:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.657+0000 7ffa8dfc4700 1 --2- 172.21.15.46:0/988900505 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa88154290 0x7ffa88157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:43:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.657+0000 7ffa8cfc2700 1 --2- 172.21.15.46:0/988900505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa88153ec0 0x7ffa880ae4d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ffa7800b3c0 tx=0x7ffa78017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.657+0000 7ffa7d7fa700 1 -- 172.21.15.46:0/988900505 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7801f070 con 0x7ffa88153ec0 2024-01-26T23:43:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.657+0000 7ffa7d7fa700 1 -- 172.21.15.46:0/988900505 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa78000cc0 con 0x7ffa88153ec0 2024-01-26T23:43:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.657+0000 7ffa8fa27700 1 -- 172.21.15.46:0/988900505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa880ae040 con 0x7ffa88153ec0 2024-01-26T23:43:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.657+0000 7ffa7d7fa700 1 -- 172.21.15.46:0/988900505 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7800bbe0 con 0x7ffa88153ec0 2024-01-26T23:43:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.657+0000 7ffa8fa27700 1 -- 172.21.15.46:0/988900505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa880ad5b0 con 0x7ffa88153ec0 2024-01-26T23:43:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.658+0000 7ffa8fa27700 1 -- 172.21.15.46:0/988900505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa880ae1d0 con 0x7ffa88153ec0 2024-01-26T23:43:05.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.659+0000 7ffa7d7fa700 1 -- 172.21.15.46:0/988900505 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ffa78011900 con 0x7ffa88153ec0 2024-01-26T23:43:05.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.659+0000 7ffa7d7fa700 1 --2- 172.21.15.46:0/988900505 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffa6c066f60 0x7ffa6c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:05.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.659+0000 7ffa7d7fa700 1 -- 172.21.15.46:0/988900505 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7ffa78085d20 con 0x7ffa88153ec0 2024-01-26T23:43:05.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.659+0000 7ffa8d7c3700 1 --2- 172.21.15.46:0/988900505 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffa6c066f60 0x7ffa6c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:05.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.660+0000 7ffa8d7c3700 1 --2- 172.21.15.46:0/988900505 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffa6c066f60 0x7ffa6c069410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffa84000c00 tx=0x7ffa84009040 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:05.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.664+0000 7ffa7d7fa700 1 -- 172.21.15.46:0/988900505 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffa78052ce0 con 0x7ffa88153ec0 2024-01-26T23:43:05.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.867+0000 7ffa8fa27700 1 -- 172.21.15.46:0/988900505 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffa8804c710 con 0x7ffa6c066f60 2024-01-26T23:43:05.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.876+0000 7ffa7d7fa700 1 -- 172.21.15.46:0/988900505 <== mgr.5267 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278586 (secure 0 0 0) 0x7ffa8804c710 con 0x7ffa6c066f60 2024-01-26T23:43:05.884 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:43:05.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.879+0000 7ffa8fa27700 1 -- 172.21.15.46:0/988900505 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffa6c066f60 msgr2=0x7ffa6c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:05.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.879+0000 7ffa8fa27700 1 --2- 172.21.15.46:0/988900505 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffa6c066f60 0x7ffa6c069410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffa84000c00 tx=0x7ffa84009040 comp rx=0 tx=0).stop 2024-01-26T23:43:05.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.879+0000 7ffa8fa27700 1 -- 172.21.15.46:0/988900505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa88153ec0 msgr2=0x7ffa880ae4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:05.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.879+0000 7ffa8fa27700 1 --2- 172.21.15.46:0/988900505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa88153ec0 0x7ffa880ae4d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ffa7800b3c0 tx=0x7ffa78017000 comp rx=0 tx=0).stop 2024-01-26T23:43:05.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.879+0000 7ffa8fa27700 1 -- 172.21.15.46:0/988900505 shutdown_connections 2024-01-26T23:43:05.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.879+0000 7ffa8fa27700 1 --2- 172.21.15.46:0/988900505 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ffa6c066f60 0x7ffa6c069410 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.879+0000 7ffa8fa27700 1 --2- 172.21.15.46:0/988900505 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa88154290 0x7ffa88157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.879+0000 7ffa8fa27700 1 --2- 172.21.15.46:0/988900505 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa880b2100 0x7ffa88146f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.879+0000 7ffa8fa27700 1 --2- 172.21.15.46:0/988900505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa88153ec0 0x7ffa880ae4d0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:05.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.879+0000 7ffa8fa27700 1 -- 172.21.15.46:0/988900505 >> 172.21.15.46:0/988900505 conn(0x7ffa88069510 msgr2=0x7ffa88156040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:05.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.879+0000 7ffa8fa27700 1 -- 172.21.15.46:0/988900505 shutdown_connections 2024-01-26T23:43:05.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:05.879+0000 7ffa8fa27700 1 -- 172.21.15.46:0/988900505 wait complete. 2024-01-26T23:43:05.888 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:43:05.894 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-26T23:43:05.642333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2418492,"kb_used_data":7420,"kb_used_omap":0,"kb_used_meta":2411008,"kb_avail":752556228,"statfs":{"total":773094113280,"available":770617577472,"internally_reserved":0,"allocated":7598080,"data_stored":4341407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2468872192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.946913"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.702730+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:55.750309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697701+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:55.750309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697807+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:15:45.000899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697169+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:00:27.828427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.705515+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:01:20.459485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.706547+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:48:06.519946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701638+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:07:07.891870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704532+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:03.697032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701642+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:24.351557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.706514+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:19.596229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.702053+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:34:08.288094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.696768+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:05.307132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704628+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:20:40.758787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.703147+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:22:30.811610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.702374+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:51:32.002538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.696777+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:08:26.147843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704560+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:41:09.634181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.699768+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:18:47.545463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701502+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:21:48.776990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701431+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:05:54.731798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697780+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:16:07.169692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.698213+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:36.728509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704561+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:38:03.176048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.699836+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:29:14.906700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.702155+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:29:51.950015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701472+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:31:22.376213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704797+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:55:52.786999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.696764+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:48:06.519946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.696894+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:19.596229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.706535+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:05.307132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697086+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:51:32.002538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704965+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:00:58.220861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"61'139","reported_seq":283,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-01-26T23:42:59.110217+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:42:59.110217+0000","last_peered":"2024-01-26T23:42:59.110217+0000","last_clean":"2024-01-26T23:42:59.110217+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:42:59.110217+0000","last_undegraded":"2024-01-26T23:42:59.110217+0000","last_fullsized":"2024-01-26T23:42:59.110217+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575011,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306844,"kb_used_data":1364,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064996,"statfs":{"total":96636764160,"available":96322555904,"internally_reserved":0,"allocated":1396736,"data_stored":986263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59599999999999997,"15min":0.56499999999999995},"min":{"1min":0.51100000000000001,"5min":0.371,"15min":0.317},"max":{"1min":0.78300000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61899999999999999,"15min":0.57899999999999996},"min":{"1min":0.50600000000000001,"5min":0.44500000000000001,"15min":0.34699999999999998},"max":{"1min":0.80300000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.32800000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.61299999999999999,"15min":0.56799999999999995},"min":{"1min":0.45500000000000002,"5min":0.43099999999999999,"15min":0.36399999999999999},"max":{"1min":0.81299999999999994,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.61899999999999999,"15min":0.58299999999999996},"min":{"1min":0.52300000000000002,"5min":0.41099999999999998,"15min":0.36099999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.85199999999999998},"last":0.46899999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.59799999999999998,"15min":0.56999999999999995},"min":{"1min":0.51000000000000001,"5min":0.47699999999999998,"15min":0.35899999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.81699999999999995},"last":0.378},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.61199999999999999,"15min":0.59199999999999997},"min":{"1min":0.50600000000000001,"5min":0.41699999999999998,"15min":0.38400000000000001},"max":{"1min":0.77300000000000002,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.44}]},{"osd":3,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60999999999999999,"15min":0.58599999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.35399999999999998},"max":{"1min":0.76900000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.621,"15min":0.60399999999999998},"min":{"1min":0.53500000000000003,"5min":0.433,"15min":0.373},"max":{"1min":0.79200000000000004,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.48699999999999999}]},{"osd":4,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46500000000000002,"15min":0.46899999999999997},"min":{"1min":0.33300000000000002,"5min":0.31900000000000001,"15min":0.28999999999999998},"max":{"1min":0.65600000000000003,"5min":0.72699999999999998,"15min":0.90100000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.49099999999999999,"15min":0.498},"min":{"1min":0.38500000000000001,"5min":0.33100000000000002,"15min":0.314},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.73799999999999999},"last":0.41799999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.498,"15min":0.50600000000000001},"min":{"1min":0.35399999999999998,"5min":0.32000000000000001,"15min":0.30599999999999999},"max":{"1min":0.71299999999999997,"5min":0.745,"15min":0.91700000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.40799999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.96599999999999997},"last":0.53200000000000003}]},{"osd":6,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56699999999999995,"15min":0.55900000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.33100000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.93500000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.432,"5min":0.38100000000000001,"15min":0.35299999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.86899999999999999},"last":0.499}]}]},{"osd":6,"up_from":13,"seq":55834575011,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302168,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069672,"statfs":{"total":96636764160,"available":96327344128,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.58599999999999997,"15min":0.56100000000000005},"min":{"1min":0.30199999999999999,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":0.71999999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60299999999999998,"15min":0.58399999999999996},"min":{"1min":0.41399999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71699999999999997,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.58199999999999996}]},{"osd":1,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.57999999999999996,"15min":0.56899999999999995},"min":{"1min":0.41699999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.60699999999999998,"15min":0.59099999999999997},"min":{"1min":0.38100000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73499999999999999,"5min":0.874,"15min":0.874},"last":0.54900000000000004}]},{"osd":2,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.58999999999999997,"15min":0.57799999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.29599999999999999},"max":{"1min":0.73299999999999998,"5min":0.78100000000000003,"15min":0.85799999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59799999999999998,"15min":0.58299999999999996},"min":{"1min":0.45300000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76200000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.61099999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60499999999999998,"15min":0.59399999999999997},"min":{"1min":0.435,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.80700000000000005,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.62,"15min":0.61499999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39000000000000001},"max":{"1min":0.71699999999999997,"5min":0.82299999999999995,"15min":0.84499999999999997},"last":0.628}]},{"osd":4,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.46999999999999997,"15min":0.47399999999999998},"min":{"1min":0.33300000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.70399999999999996,"5min":0.77800000000000002,"15min":0.90200000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.48799999999999999,"15min":0.498},"min":{"1min":0.32000000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.67400000000000004,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.53300000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.47999999999999998,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.69499999999999995,"5min":0.76500000000000001,"15min":0.78200000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51800000000000002,"15min":0.53000000000000003},"min":{"1min":0.34399999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.77900000000000003,"5min":0.83499999999999996,"15min":0.96499999999999997},"last":0.46400000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52200000000000002,"15min":0.54000000000000004},"min":{"1min":0.38600000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.73599999999999999,"5min":0.85899999999999999,"15min":0.97199999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.308},"max":{"1min":0.70399999999999996,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56200000000000006}]}]},{"osd":1,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302168,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069672,"statfs":{"total":96636764160,"available":96327344128,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39500000000000002,"15min":0.40600000000000003},"min":{"1min":0.26500000000000001,"5min":0.26200000000000001,"15min":0.23200000000000001},"max":{"1min":0.57799999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.621},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.441},"min":{"1min":0.32500000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.58199999999999996,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.41599999999999998,"15min":0.40699999999999997},"min":{"1min":0.33700000000000002,"5min":0.30599999999999999,"15min":0.26200000000000001},"max":{"1min":0.61399999999999999,"5min":0.63100000000000001,"15min":0.65900000000000003},"last":0.499},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.38600000000000001,"5min":0.32600000000000001,"15min":0.28999999999999998},"max":{"1min":0.69499999999999995,"5min":0.71899999999999997,"15min":0.79600000000000004},"last":0.51300000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.505,"15min":0.505},"min":{"1min":0.42499999999999999,"5min":0.33800000000000002,"15min":0.32100000000000001},"max":{"1min":0.69099999999999995,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.81999999999999995},"last":0.436}]},{"osd":4,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.66700000000000004,"15min":0.64000000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.39400000000000002},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.65900000000000003,"15min":0.66700000000000004},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.433},"max":{"1min":0.91900000000000004,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.82199999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.67900000000000005,"15min":0.67600000000000005},"min":{"1min":0.49299999999999999,"5min":0.45000000000000001,"15min":0.42799999999999999},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.67400000000000004,"15min":0.66600000000000004},"min":{"1min":0.57699999999999996,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.92200000000000004,"5min":0.98599999999999999,"15min":1.018},"last":0.63200000000000001}]},{"osd":6,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.68899999999999995,"15min":0.68600000000000005},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.45700000000000002},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.1279999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.69199999999999995,"15min":0.69099999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.42599999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.78500000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.72099999999999997,"15min":0.71399999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.49399999999999999},"max":{"1min":0.94699999999999995,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.59199999999999997,"5min":0.5,"15min":0.438},"max":{"1min":1.048,"5min":1.048,"15min":1.3049999999999999},"last":0.80200000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302736,"kb_used_data":1352,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069104,"statfs":{"total":96636764160,"available":96326762496,"internally_reserved":0,"allocated":1384448,"data_stored":979774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40100000000000002,"15min":0.41299999999999998},"min":{"1min":0.27800000000000002,"5min":0.26400000000000001,"15min":0.214},"max":{"1min":0.52200000000000002,"5min":0.64100000000000001,"15min":1.284},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46899999999999997,"15min":0.45900000000000002},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.27300000000000002},"max":{"1min":0.72699999999999998,"5min":0.75800000000000001,"15min":1.024},"last":0.30499999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.433,"15min":0.443},"min":{"1min":0.318,"5min":0.28199999999999997,"15min":0.26700000000000002},"max":{"1min":0.59699999999999998,"5min":0.66100000000000003,"15min":1.2230000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.68100000000000005,"5min":0.68300000000000005,"15min":1.2430000000000001},"last":0.375}]},{"osd":3,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.47999999999999998,"15min":0.47599999999999998},"min":{"1min":0.38800000000000001,"5min":0.317,"15min":0.26800000000000002},"max":{"1min":0.626,"5min":0.69899999999999995,"15min":1.0980000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.498,"15min":0.5},"min":{"1min":0.39500000000000002,"5min":0.34000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69799999999999995,"5min":0.74099999999999999,"15min":1.1850000000000001},"last":0.41299999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.65200000000000002,"15min":0.65700000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.372},"max":{"1min":0.86799999999999999,"5min":0.93300000000000005,"15min":1.133},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66600000000000004,"15min":0.66900000000000004},"min":{"1min":0.52900000000000003,"5min":0.443,"15min":0.443},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":1.0169999999999999},"last":0.55400000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.66100000000000003,"15min":0.66900000000000004},"min":{"1min":0.46500000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":1.0880000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68200000000000005,"15min":0.69199999999999995},"min":{"1min":0.504,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93300000000000005,"5min":0.98699999999999999,"15min":1.169},"last":0.58799999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67300000000000004,"15min":0.69399999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.44800000000000001},"max":{"1min":0.88400000000000001,"5min":0.97299999999999998,"15min":1.071},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69399999999999995,"15min":0.70999999999999996},"min":{"1min":0.55200000000000005,"5min":0.502,"15min":0.48599999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.1160000000000001},"last":0.60399999999999998}]},{"osd":7,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.72099999999999997,"15min":0.73999999999999999},"min":{"1min":0.51300000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.73699999999999999,"15min":0.749},"min":{"1min":0.58999999999999997,"5min":0.433,"15min":0.433},"max":{"1min":0.98699999999999999,"5min":1.028,"15min":1.2010000000000001},"last":0.68899999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306264,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065576,"statfs":{"total":96636764160,"available":96323149824,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.503},"min":{"1min":0.28399999999999997,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":0.53200000000000003,"5min":0.56699999999999995,"15min":9.0540000000000003},"last":0.442},{"interface":"front","average":{"1min":0.443,"5min":0.42199999999999999,"15min":0.46200000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.24299999999999999},"max":{"1min":0.57599999999999996,"5min":0.64000000000000001,"15min":4.1970000000000001},"last":0.307}]},{"osd":1,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.438,"15min":0.42299999999999999},"min":{"1min":0.28499999999999998,"5min":0.27400000000000002,"15min":0.221},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.82199999999999995},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.502,"15min":0.64800000000000002},"min":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.316},"max":{"1min":0.65900000000000003,"5min":0.72499999999999998,"15min":12.157},"last":0.45700000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.502,"15min":0.48599999999999999},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33500000000000002},"max":{"1min":0.65500000000000003,"5min":0.74099999999999999,"15min":0.86499999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47899999999999998,"15min":0.57699999999999996},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.30399999999999999},"max":{"1min":0.63600000000000001,"5min":0.73199999999999998,"15min":9.0879999999999992},"last":0.38300000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.65100000000000002,"15min":0.749},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.88500000000000001,"5min":1.048,"15min":9.0150000000000006},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.64600000000000002,"15min":0.68400000000000005},"min":{"1min":0.55700000000000005,"5min":0.371,"15min":0.371},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":4.2480000000000002},"last":0.63500000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66400000000000003,"15min":0.69799999999999995},"min":{"1min":0.55500000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.85899999999999999,"5min":1.008,"15min":4.3200000000000003},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.69199999999999995,"15min":0.67800000000000005},"min":{"1min":0.52000000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.95199999999999996,"5min":1.085,"15min":1.085},"last":0.621}]},{"osd":6,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.69899999999999995,"15min":0.68600000000000005},"min":{"1min":0.46999999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.505},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.68899999999999995,"15min":0.78800000000000003},"min":{"1min":0.53500000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.94699999999999995,"5min":1.111,"15min":8.9700000000000006},"last":0.64700000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.70799999999999996,"15min":0.752},"min":{"1min":0.61099999999999999,"5min":0.54100000000000004,"15min":0.45800000000000002},"max":{"1min":0.93700000000000006,"5min":1.127,"15min":4.282},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.746,"15min":0.73499999999999999},"min":{"1min":0.52900000000000003,"5min":0.46100000000000002,"15min":0.42099999999999999},"max":{"1min":1.0329999999999999,"5min":1.139,"15min":1.139},"last":0.66100000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302168,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069672,"statfs":{"total":96636764160,"available":96327344128,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.23999999999999999},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44900000000000001,"15min":0.56100000000000005},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":11.968},"last":0.504}]},{"osd":1,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.45600000000000002,"15min":0.57299999999999995},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.633,"5min":1.633,"15min":12.050000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48299999999999998,"15min":0.48999999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.2150000000000001},"last":0.378}]},{"osd":2,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.48899999999999999,"15min":0.59399999999999997},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.28999999999999998},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":12.234},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.46999999999999997,"15min":0.58699999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":11.925000000000001},"last":0.39700000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.70199999999999996,"15min":0.78200000000000003},"min":{"1min":0.46999999999999997,"5min":0.45500000000000002,"15min":0.40200000000000002},"max":{"1min":3.306,"5min":3.306,"15min":12.099},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68000000000000005,"15min":0.66900000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.379},"max":{"1min":0.89700000000000002,"5min":1.0009999999999999,"15min":1.0649999999999999},"last":0.46899999999999997}]},{"osd":5,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67700000000000005,"15min":0.79300000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.593,"5min":1.593,"15min":12.199999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.69099999999999995,"15min":0.68000000000000005},"min":{"1min":0.45300000000000001,"5min":0.439,"15min":0.436},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.56999999999999995}]},{"osd":6,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.70099999999999996,"15min":0.80000000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.506,"5min":1.506,"15min":12.132999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.70799999999999996,"15min":0.70199999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.53700000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.74099999999999999,"15min":0.70499999999999996},"min":{"1min":0.48499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.238,"5min":3.238,"15min":3.238},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.77200000000000002,"15min":0.86499999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.48699999999999999},"max":{"1min":3.351,"5min":3.351,"15min":12.167},"last":0.48899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298072,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073768,"statfs":{"total":96636764160,"available":96331538432,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58299999999999996,"15min":0.57399999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69699999999999995,"5min":0.79800000000000004,"15min":1.0209999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60199999999999998,"15min":0.58199999999999996},"min":{"1min":0.41499999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70099999999999996,"5min":0.84099999999999997,"15min":1.161},"last":0.622}]},{"osd":1,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.60799999999999998,"15min":0.59699999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73699999999999999,"5min":0.88,"15min":1.069},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.39100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72699999999999998,"5min":0.78800000000000003,"15min":1.085},"last":0.48499999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59199999999999997,"15min":0.57399999999999995},"min":{"1min":0.38400000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.77000000000000002,"5min":0.85999999999999999,"15min":1.052},"last":0.435},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61399999999999999,"15min":0.61299999999999999},"min":{"1min":0.45700000000000002,"5min":0.40899999999999997,"15min":0.373},"max":{"1min":0.77600000000000002,"5min":0.80100000000000005,"15min":1.111},"last":0.39900000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61099999999999999,"15min":0.59999999999999998},"min":{"1min":0.41699999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.71999999999999997,"5min":0.78300000000000003,"15min":1.1299999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.45200000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.69699999999999995,"5min":0.88800000000000001,"15min":1.1419999999999999},"last":0.60399999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47599999999999998,"15min":0.47899999999999998},"min":{"1min":0.39300000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.64400000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.496,"15min":0.51100000000000001},"min":{"1min":0.33400000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.69799999999999995,"5min":0.76700000000000002,"15min":0.77800000000000002},"last":0.66800000000000004}]},{"osd":6,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.38300000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53100000000000003,"15min":0.54100000000000004},"min":{"1min":0.35499999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.67200000000000004,"5min":0.78200000000000003,"15min":0.95699999999999996},"last":0.65700000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.42799999999999999,"5min":0.375,"15min":0.35799999999999998},"max":{"1min":0.745,"5min":0.75900000000000001,"15min":0.76300000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.501,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.74099999999999999,"5min":0.78400000000000003,"15min":0.879},"last":0.64500000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298072,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073768,"statfs":{"total":96636764160,"available":96331538432,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.57699999999999996,"15min":0.55800000000000005},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.26000000000000001},"max":{"1min":2.0289999999999999,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57299999999999995,"15min":0.57599999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.498}]},{"osd":1,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61399999999999999,"15min":0.60899999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.3319999999999999,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63700000000000001,"15min":0.61699999999999999},"min":{"1min":0.435,"5min":0.41799999999999998,"15min":0.379},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.59599999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55500000000000005,"15min":0.56399999999999995},"min":{"1min":0.38200000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72599999999999998,"5min":0.83699999999999997,"15min":0.83799999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.60299999999999998,"15min":0.64700000000000002},"min":{"1min":0.39400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":6.4589999999999996},"last":0.53400000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57499999999999996,"15min":0.57399999999999995},"min":{"1min":0.35999999999999999,"5min":0.35799999999999998,"15min":0.314},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.88400000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.59899999999999998,"15min":0.64700000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":6.0629999999999997},"last":0.51700000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.26300000000000001},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48999999999999999,"15min":0.497},"min":{"1min":0.34499999999999997,"5min":0.315,"15min":0.28100000000000003},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.621}]},{"osd":6,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.48999999999999999,"15min":0.48499999999999999},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.27000000000000002},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52600000000000002,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.29299999999999998},"max":{"1min":1.99,"5min":1.99,"15min":1.99},"last":0.63400000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.56000000000000005,"15min":0.54900000000000004},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.5270000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55700000000000005,"15min":0.57299999999999995},"min":{"1min":0.36899999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.624,"5min":0.82199999999999995,"15min":0.83099999999999996},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:43:05.900 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:43:06.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.067+0000 7f5b5008f700 1 Processor -- start 2024-01-26T23:43:06.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.067+0000 7f5b5008f700 1 -- start start 2024-01-26T23:43:06.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.068+0000 7f5b5008f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5b48157340 0x7f5b4815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:06.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.068+0000 7f5b5008f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b48158ae0 con 0x7f5b480b4180 2024-01-26T23:43:06.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.068+0000 7f5b5008f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b4815b650 con 0x7f5b480ae520 2024-01-26T23:43:06.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.068+0000 7f5b5008f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b4815c020 con 0x7f5b48157340 2024-01-26T23:43:06.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.069+0000 7f5b4de2b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5b48157340 0x7f5b4815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:06.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.069+0000 7f5b4e62c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5b480b4180 0x7f5b48157710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37766/0 (socket says 172.21.15.46:37766) 2024-01-26T23:43:06.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.069+0000 7f5b4e62c700 1 -- 172.21.15.46:0/1667872480 learned_addr learned my addr 172.21.15.46:0/1667872480 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:43:06.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.069+0000 7f5b4de2b700 1 -- 172.21.15.46:0/1667872480 >> v1:172.21.15.106:6789/0 conn(0x7f5b480ae520 legacy=0x7f5b480b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:06.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.069+0000 7f5b4de2b700 1 -- 172.21.15.46:0/1667872480 >> v1:172.21.15.46:6789/0 conn(0x7f5b480b4180 legacy=0x7f5b48157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:06.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.069+0000 7f5b4de2b700 1 -- 172.21.15.46:0/1667872480 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b48158520 con 0x7f5b48157340 2024-01-26T23:43:06.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.069+0000 7f5b4ce29700 1 -- 172.21.15.46:0/1667872480 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269972514 0 0) 0x7f5b48158ae0 con 0x7f5b480b4180 2024-01-26T23:43:06.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.069+0000 7f5b4de2b700 1 --2- 172.21.15.46:0/1667872480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b48157340 0x7f5b4815bae0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5b440098e0 tx=0x7f5b440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d098d62d81ca981c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:06.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.069+0000 7f5b4ce29700 1 -- 172.21.15.46:0/1667872480 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b44009e80 con 0x7f5b48157340 2024-01-26T23:43:06.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.069+0000 7f5b4ce29700 1 -- 172.21.15.46:0/1667872480 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4400b860 con 0x7f5b48157340 2024-01-26T23:43:06.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.069+0000 7f5b4ce29700 1 -- 172.21.15.46:0/1667872480 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4400bbe0 con 0x7f5b48157340 2024-01-26T23:43:06.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.070+0000 7f5b5008f700 1 -- 172.21.15.46:0/1667872480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b48157340 msgr2=0x7f5b4815bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:06.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.070+0000 7f5b5008f700 1 --2- 172.21.15.46:0/1667872480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b48157340 0x7f5b4815bae0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5b440098e0 tx=0x7f5b440091a0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.070+0000 7f5b5008f700 1 -- 172.21.15.46:0/1667872480 shutdown_connections 2024-01-26T23:43:06.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.070+0000 7f5b5008f700 1 --2- 172.21.15.46:0/1667872480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b48157340 0x7f5b4815bae0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.070+0000 7f5b5008f700 1 -- 172.21.15.46:0/1667872480 >> 172.21.15.46:0/1667872480 conn(0x7f5b480ce530 msgr2=0x7f5b480ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:06.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.070+0000 7f5b5008f700 1 -- 172.21.15.46:0/1667872480 shutdown_connections 2024-01-26T23:43:06.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.070+0000 7f5b5008f700 1 -- 172.21.15.46:0/1667872480 wait complete. 2024-01-26T23:43:06.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.071+0000 7f5b5008f700 1 Processor -- start 2024-01-26T23:43:06.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.071+0000 7f5b5008f700 1 -- start start 2024-01-26T23:43:06.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.072+0000 7f5b5008f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b480ae520 0x7f5b48146930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:06.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.072+0000 7f5b5008f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b480b4180 0x7f5b480b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:06.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.072+0000 7f5b5008f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b48157340 0x7f5b480cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:06.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.072+0000 7f5b5008f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b480c2aa0 con 0x7f5b480b4180 2024-01-26T23:43:06.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.072+0000 7f5b5008f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4816bd10 con 0x7f5b480ae520 2024-01-26T23:43:06.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.072+0000 7f5b5008f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4816b140 con 0x7f5b48157340 2024-01-26T23:43:06.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.072+0000 7f5b4d62a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b480b4180 0x7f5b480b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:06.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.072+0000 7f5b4d62a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b480b4180 0x7f5b480b2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50762/0 (socket says 172.21.15.46:50762) 2024-01-26T23:43:06.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.072+0000 7f5b4d62a700 1 -- 172.21.15.46:0/2307634155 learned_addr learned my addr 172.21.15.46:0/2307634155 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:06.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.072+0000 7f5b4de2b700 1 --2- 172.21.15.46:0/2307634155 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b480ae520 0x7f5b48146930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:06.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.072+0000 7f5b4e62c700 1 --2- 172.21.15.46:0/2307634155 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b48157340 0x7f5b480cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:06.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.072+0000 7f5b4d62a700 1 -- 172.21.15.46:0/2307634155 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b48157340 msgr2=0x7f5b480cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:06.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.072+0000 7f5b4d62a700 1 --2- 172.21.15.46:0/2307634155 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b48157340 0x7f5b480cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.072+0000 7f5b4d62a700 1 -- 172.21.15.46:0/2307634155 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b480ae520 msgr2=0x7f5b48146930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:06.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.072+0000 7f5b4d62a700 1 --2- 172.21.15.46:0/2307634155 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b480ae520 0x7f5b48146930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.072+0000 7f5b4d62a700 1 -- 172.21.15.46:0/2307634155 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b44009420 con 0x7f5b480b4180 2024-01-26T23:43:06.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.072+0000 7f5b4e62c700 1 --2- 172.21.15.46:0/2307634155 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b48157340 0x7f5b480cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:43:06.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.073+0000 7f5b4d62a700 1 --2- 172.21.15.46:0/2307634155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b480b4180 0x7f5b480b2b30 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800c4e0 tx=0x7f5b3800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:06.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.073+0000 7f5b3dffb700 1 -- 172.21.15.46:0/2307634155 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38011590 con 0x7f5b480b4180 2024-01-26T23:43:06.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.073+0000 7f5b3dffb700 1 -- 172.21.15.46:0/2307634155 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b38011710 con 0x7f5b480b4180 2024-01-26T23:43:06.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.073+0000 7f5b5008f700 1 -- 172.21.15.46:0/2307634155 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b48146e70 con 0x7f5b480b4180 2024-01-26T23:43:06.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.073+0000 7f5b3dffb700 1 -- 172.21.15.46:0/2307634155 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38011ad0 con 0x7f5b480b4180 2024-01-26T23:43:06.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.073+0000 7f5b5008f700 1 -- 172.21.15.46:0/2307634155 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4816b6c0 con 0x7f5b480b4180 2024-01-26T23:43:06.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.074+0000 7f5b5008f700 1 -- 172.21.15.46:0/2307634155 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b4816be80 con 0x7f5b480b4180 2024-01-26T23:43:06.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.075+0000 7f5b3dffb700 1 -- 172.21.15.46:0/2307634155 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5b3801b3b0 con 0x7f5b480b4180 2024-01-26T23:43:06.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.075+0000 7f5b3dffb700 1 --2- 172.21.15.46:0/2307634155 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5b2c066f30 0x7f5b2c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:06.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.075+0000 7f5b3dffb700 1 -- 172.21.15.46:0/2307634155 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f5b38083d10 con 0x7f5b480b4180 2024-01-26T23:43:06.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.075+0000 7f5b4de2b700 1 --2- 172.21.15.46:0/2307634155 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5b2c066f30 0x7f5b2c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:06.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.076+0000 7f5b4de2b700 1 --2- 172.21.15.46:0/2307634155 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5b2c066f30 0x7f5b2c0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5b440093f0 tx=0x7f5b44017000 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:06.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.080+0000 7f5b3dffb700 1 -- 172.21.15.46:0/2307634155 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5b38050cd0 con 0x7f5b480b4180 2024-01-26T23:43:06.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.282+0000 7f5b5008f700 1 -- 172.21.15.46:0/2307634155 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5b48158800 con 0x7f5b2c066f30 2024-01-26T23:43:06.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.290+0000 7f5b3dffb700 1 -- 172.21.15.46:0/2307634155 <== mgr.5267 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278586 (secure 0 0 0) 0x7f5b48158800 con 0x7f5b2c066f30 2024-01-26T23:43:06.295 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:43:06.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.293+0000 7f5b5008f700 1 -- 172.21.15.46:0/2307634155 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5b2c066f30 msgr2=0x7f5b2c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.293+0000 7f5b5008f700 1 --2- 172.21.15.46:0/2307634155 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5b2c066f30 0x7f5b2c0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5b440093f0 tx=0x7f5b44017000 comp rx=0 tx=0).stop 2024-01-26T23:43:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.293+0000 7f5b5008f700 1 -- 172.21.15.46:0/2307634155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b480b4180 msgr2=0x7f5b480b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.293+0000 7f5b5008f700 1 --2- 172.21.15.46:0/2307634155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b480b4180 0x7f5b480b2b30 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800c4e0 tx=0x7f5b3800c8a0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.294+0000 7f5b5008f700 1 -- 172.21.15.46:0/2307634155 shutdown_connections 2024-01-26T23:43:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.294+0000 7f5b5008f700 1 --2- 172.21.15.46:0/2307634155 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5b2c066f30 0x7f5b2c0693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.294+0000 7f5b5008f700 1 --2- 172.21.15.46:0/2307634155 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b48157340 0x7f5b480cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.294+0000 7f5b5008f700 1 --2- 172.21.15.46:0/2307634155 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b480ae520 0x7f5b48146930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.294+0000 7f5b5008f700 1 --2- 172.21.15.46:0/2307634155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b480b4180 0x7f5b480b2b30 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.294+0000 7f5b5008f700 1 -- 172.21.15.46:0/2307634155 >> 172.21.15.46:0/2307634155 conn(0x7f5b480ce530 msgr2=0x7f5b4806d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.294+0000 7f5b5008f700 1 -- 172.21.15.46:0/2307634155 shutdown_connections 2024-01-26T23:43:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.294+0000 7f5b5008f700 1 -- 172.21.15.46:0/2307634155 wait complete. 2024-01-26T23:43:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:43:06.309 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-26T23:43:05.642333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2418492,"kb_used_data":7420,"kb_used_omap":0,"kb_used_meta":2411008,"kb_avail":752556228,"statfs":{"total":773094113280,"available":770617577472,"internally_reserved":0,"allocated":7598080,"data_stored":4341407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2468872192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.946913"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.702730+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:55.750309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697701+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:55.750309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697807+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:15:45.000899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697169+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:00:27.828427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.705515+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:01:20.459485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.706547+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:48:06.519946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701638+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:07:07.891870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704532+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:03.697032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701642+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:24.351557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.706514+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:19.596229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.702053+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:34:08.288094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.696768+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:05.307132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704628+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:20:40.758787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.703147+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:22:30.811610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.702374+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:51:32.002538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.696777+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:08:26.147843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704560+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:41:09.634181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.699768+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:18:47.545463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701502+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:21:48.776990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701431+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:05:54.731798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697780+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:16:07.169692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.698213+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:36.728509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704561+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:38:03.176048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.699836+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:29:14.906700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.702155+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:29:51.950015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701472+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:31:22.376213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704797+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:55:52.786999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.696764+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:48:06.519946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.696894+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:19.596229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.706535+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:05.307132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697086+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:51:32.002538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704965+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:00:58.220861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"61'139","reported_seq":283,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-01-26T23:42:59.110217+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:42:59.110217+0000","last_peered":"2024-01-26T23:42:59.110217+0000","last_clean":"2024-01-26T23:42:59.110217+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:42:59.110217+0000","last_undegraded":"2024-01-26T23:42:59.110217+0000","last_fullsized":"2024-01-26T23:42:59.110217+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575011,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306844,"kb_used_data":1364,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064996,"statfs":{"total":96636764160,"available":96322555904,"internally_reserved":0,"allocated":1396736,"data_stored":986263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59599999999999997,"15min":0.56499999999999995},"min":{"1min":0.51100000000000001,"5min":0.371,"15min":0.317},"max":{"1min":0.78300000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61899999999999999,"15min":0.57899999999999996},"min":{"1min":0.50600000000000001,"5min":0.44500000000000001,"15min":0.34699999999999998},"max":{"1min":0.80300000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.32800000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.61299999999999999,"15min":0.56799999999999995},"min":{"1min":0.45500000000000002,"5min":0.43099999999999999,"15min":0.36399999999999999},"max":{"1min":0.81299999999999994,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.61899999999999999,"15min":0.58299999999999996},"min":{"1min":0.52300000000000002,"5min":0.41099999999999998,"15min":0.36099999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.85199999999999998},"last":0.46899999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.59799999999999998,"15min":0.56999999999999995},"min":{"1min":0.51000000000000001,"5min":0.47699999999999998,"15min":0.35899999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.81699999999999995},"last":0.378},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.61199999999999999,"15min":0.59199999999999997},"min":{"1min":0.50600000000000001,"5min":0.41699999999999998,"15min":0.38400000000000001},"max":{"1min":0.77300000000000002,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.44}]},{"osd":3,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60999999999999999,"15min":0.58599999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.35399999999999998},"max":{"1min":0.76900000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.621,"15min":0.60399999999999998},"min":{"1min":0.53500000000000003,"5min":0.433,"15min":0.373},"max":{"1min":0.79200000000000004,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.48699999999999999}]},{"osd":4,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46500000000000002,"15min":0.46899999999999997},"min":{"1min":0.33300000000000002,"5min":0.31900000000000001,"15min":0.28999999999999998},"max":{"1min":0.65600000000000003,"5min":0.72699999999999998,"15min":0.90100000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.49099999999999999,"15min":0.498},"min":{"1min":0.38500000000000001,"5min":0.33100000000000002,"15min":0.314},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.73799999999999999},"last":0.41799999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.498,"15min":0.50600000000000001},"min":{"1min":0.35399999999999998,"5min":0.32000000000000001,"15min":0.30599999999999999},"max":{"1min":0.71299999999999997,"5min":0.745,"15min":0.91700000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.40799999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.96599999999999997},"last":0.53200000000000003}]},{"osd":6,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56699999999999995,"15min":0.55900000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.33100000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.93500000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.432,"5min":0.38100000000000001,"15min":0.35299999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.86899999999999999},"last":0.499}]}]},{"osd":6,"up_from":13,"seq":55834575011,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302168,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069672,"statfs":{"total":96636764160,"available":96327344128,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.58599999999999997,"15min":0.56100000000000005},"min":{"1min":0.30199999999999999,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":0.71999999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60299999999999998,"15min":0.58399999999999996},"min":{"1min":0.41399999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71699999999999997,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.58199999999999996}]},{"osd":1,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.57999999999999996,"15min":0.56899999999999995},"min":{"1min":0.41699999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.60699999999999998,"15min":0.59099999999999997},"min":{"1min":0.38100000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73499999999999999,"5min":0.874,"15min":0.874},"last":0.54900000000000004}]},{"osd":2,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.58999999999999997,"15min":0.57799999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.29599999999999999},"max":{"1min":0.73299999999999998,"5min":0.78100000000000003,"15min":0.85799999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59799999999999998,"15min":0.58299999999999996},"min":{"1min":0.45300000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76200000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.61099999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60499999999999998,"15min":0.59399999999999997},"min":{"1min":0.435,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.80700000000000005,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.62,"15min":0.61499999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39000000000000001},"max":{"1min":0.71699999999999997,"5min":0.82299999999999995,"15min":0.84499999999999997},"last":0.628}]},{"osd":4,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.46999999999999997,"15min":0.47399999999999998},"min":{"1min":0.33300000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.70399999999999996,"5min":0.77800000000000002,"15min":0.90200000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.48799999999999999,"15min":0.498},"min":{"1min":0.32000000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.67400000000000004,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.53300000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.47999999999999998,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.69499999999999995,"5min":0.76500000000000001,"15min":0.78200000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51800000000000002,"15min":0.53000000000000003},"min":{"1min":0.34399999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.77900000000000003,"5min":0.83499999999999996,"15min":0.96499999999999997},"last":0.46400000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52200000000000002,"15min":0.54000000000000004},"min":{"1min":0.38600000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.73599999999999999,"5min":0.85899999999999999,"15min":0.97199999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.308},"max":{"1min":0.70399999999999996,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56200000000000006}]}]},{"osd":1,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302168,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069672,"statfs":{"total":96636764160,"available":96327344128,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39500000000000002,"15min":0.40600000000000003},"min":{"1min":0.26500000000000001,"5min":0.26200000000000001,"15min":0.23200000000000001},"max":{"1min":0.57799999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.621},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.441},"min":{"1min":0.32500000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.58199999999999996,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.41599999999999998,"15min":0.40699999999999997},"min":{"1min":0.33700000000000002,"5min":0.30599999999999999,"15min":0.26200000000000001},"max":{"1min":0.61399999999999999,"5min":0.63100000000000001,"15min":0.65900000000000003},"last":0.499},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.38600000000000001,"5min":0.32600000000000001,"15min":0.28999999999999998},"max":{"1min":0.69499999999999995,"5min":0.71899999999999997,"15min":0.79600000000000004},"last":0.51300000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.505,"15min":0.505},"min":{"1min":0.42499999999999999,"5min":0.33800000000000002,"15min":0.32100000000000001},"max":{"1min":0.69099999999999995,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.81999999999999995},"last":0.436}]},{"osd":4,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.66700000000000004,"15min":0.64000000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.39400000000000002},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.65900000000000003,"15min":0.66700000000000004},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.433},"max":{"1min":0.91900000000000004,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.82199999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.67900000000000005,"15min":0.67600000000000005},"min":{"1min":0.49299999999999999,"5min":0.45000000000000001,"15min":0.42799999999999999},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.67400000000000004,"15min":0.66600000000000004},"min":{"1min":0.57699999999999996,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.92200000000000004,"5min":0.98599999999999999,"15min":1.018},"last":0.63200000000000001}]},{"osd":6,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.68899999999999995,"15min":0.68600000000000005},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.45700000000000002},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.1279999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.69199999999999995,"15min":0.69099999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.42599999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.78500000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.72099999999999997,"15min":0.71399999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.49399999999999999},"max":{"1min":0.94699999999999995,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.59199999999999997,"5min":0.5,"15min":0.438},"max":{"1min":1.048,"5min":1.048,"15min":1.3049999999999999},"last":0.80200000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302736,"kb_used_data":1352,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069104,"statfs":{"total":96636764160,"available":96326762496,"internally_reserved":0,"allocated":1384448,"data_stored":979774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40100000000000002,"15min":0.41299999999999998},"min":{"1min":0.27800000000000002,"5min":0.26400000000000001,"15min":0.214},"max":{"1min":0.52200000000000002,"5min":0.64100000000000001,"15min":1.284},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46899999999999997,"15min":0.45900000000000002},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.27300000000000002},"max":{"1min":0.72699999999999998,"5min":0.75800000000000001,"15min":1.024},"last":0.30499999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.433,"15min":0.443},"min":{"1min":0.318,"5min":0.28199999999999997,"15min":0.26700000000000002},"max":{"1min":0.59699999999999998,"5min":0.66100000000000003,"15min":1.2230000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.68100000000000005,"5min":0.68300000000000005,"15min":1.2430000000000001},"last":0.375}]},{"osd":3,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.47999999999999998,"15min":0.47599999999999998},"min":{"1min":0.38800000000000001,"5min":0.317,"15min":0.26800000000000002},"max":{"1min":0.626,"5min":0.69899999999999995,"15min":1.0980000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.498,"15min":0.5},"min":{"1min":0.39500000000000002,"5min":0.34000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69799999999999995,"5min":0.74099999999999999,"15min":1.1850000000000001},"last":0.41299999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.65200000000000002,"15min":0.65700000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.372},"max":{"1min":0.86799999999999999,"5min":0.93300000000000005,"15min":1.133},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66600000000000004,"15min":0.66900000000000004},"min":{"1min":0.52900000000000003,"5min":0.443,"15min":0.443},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":1.0169999999999999},"last":0.55400000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.66100000000000003,"15min":0.66900000000000004},"min":{"1min":0.46500000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":1.0880000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68200000000000005,"15min":0.69199999999999995},"min":{"1min":0.504,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93300000000000005,"5min":0.98699999999999999,"15min":1.169},"last":0.58799999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67300000000000004,"15min":0.69399999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.44800000000000001},"max":{"1min":0.88400000000000001,"5min":0.97299999999999998,"15min":1.071},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69399999999999995,"15min":0.70999999999999996},"min":{"1min":0.55200000000000005,"5min":0.502,"15min":0.48599999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.1160000000000001},"last":0.60399999999999998}]},{"osd":7,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.72099999999999997,"15min":0.73999999999999999},"min":{"1min":0.51300000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.73699999999999999,"15min":0.749},"min":{"1min":0.58999999999999997,"5min":0.433,"15min":0.433},"max":{"1min":0.98699999999999999,"5min":1.028,"15min":1.2010000000000001},"last":0.68899999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306264,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065576,"statfs":{"total":96636764160,"available":96323149824,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.503},"min":{"1min":0.28399999999999997,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":0.53200000000000003,"5min":0.56699999999999995,"15min":9.0540000000000003},"last":0.442},{"interface":"front","average":{"1min":0.443,"5min":0.42199999999999999,"15min":0.46200000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.24299999999999999},"max":{"1min":0.57599999999999996,"5min":0.64000000000000001,"15min":4.1970000000000001},"last":0.307}]},{"osd":1,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.438,"15min":0.42299999999999999},"min":{"1min":0.28499999999999998,"5min":0.27400000000000002,"15min":0.221},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.82199999999999995},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.502,"15min":0.64800000000000002},"min":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.316},"max":{"1min":0.65900000000000003,"5min":0.72499999999999998,"15min":12.157},"last":0.45700000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.502,"15min":0.48599999999999999},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33500000000000002},"max":{"1min":0.65500000000000003,"5min":0.74099999999999999,"15min":0.86499999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47899999999999998,"15min":0.57699999999999996},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.30399999999999999},"max":{"1min":0.63600000000000001,"5min":0.73199999999999998,"15min":9.0879999999999992},"last":0.38300000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.65100000000000002,"15min":0.749},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.88500000000000001,"5min":1.048,"15min":9.0150000000000006},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.64600000000000002,"15min":0.68400000000000005},"min":{"1min":0.55700000000000005,"5min":0.371,"15min":0.371},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":4.2480000000000002},"last":0.63500000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66400000000000003,"15min":0.69799999999999995},"min":{"1min":0.55500000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.85899999999999999,"5min":1.008,"15min":4.3200000000000003},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.69199999999999995,"15min":0.67800000000000005},"min":{"1min":0.52000000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.95199999999999996,"5min":1.085,"15min":1.085},"last":0.621}]},{"osd":6,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.69899999999999995,"15min":0.68600000000000005},"min":{"1min":0.46999999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.505},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.68899999999999995,"15min":0.78800000000000003},"min":{"1min":0.53500000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.94699999999999995,"5min":1.111,"15min":8.9700000000000006},"last":0.64700000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.70799999999999996,"15min":0.752},"min":{"1min":0.61099999999999999,"5min":0.54100000000000004,"15min":0.45800000000000002},"max":{"1min":0.93700000000000006,"5min":1.127,"15min":4.282},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.746,"15min":0.73499999999999999},"min":{"1min":0.52900000000000003,"5min":0.46100000000000002,"15min":0.42099999999999999},"max":{"1min":1.0329999999999999,"5min":1.139,"15min":1.139},"last":0.66100000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302168,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069672,"statfs":{"total":96636764160,"available":96327344128,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.23999999999999999},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44900000000000001,"15min":0.56100000000000005},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":11.968},"last":0.504}]},{"osd":1,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.45600000000000002,"15min":0.57299999999999995},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.633,"5min":1.633,"15min":12.050000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48299999999999998,"15min":0.48999999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.2150000000000001},"last":0.378}]},{"osd":2,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.48899999999999999,"15min":0.59399999999999997},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.28999999999999998},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":12.234},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.46999999999999997,"15min":0.58699999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":11.925000000000001},"last":0.39700000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.70199999999999996,"15min":0.78200000000000003},"min":{"1min":0.46999999999999997,"5min":0.45500000000000002,"15min":0.40200000000000002},"max":{"1min":3.306,"5min":3.306,"15min":12.099},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68000000000000005,"15min":0.66900000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.379},"max":{"1min":0.89700000000000002,"5min":1.0009999999999999,"15min":1.0649999999999999},"last":0.46899999999999997}]},{"osd":5,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67700000000000005,"15min":0.79300000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.593,"5min":1.593,"15min":12.199999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.69099999999999995,"15min":0.68000000000000005},"min":{"1min":0.45300000000000001,"5min":0.439,"15min":0.436},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.56999999999999995}]},{"osd":6,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.70099999999999996,"15min":0.80000000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.506,"5min":1.506,"15min":12.132999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.70799999999999996,"15min":0.70199999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.53700000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.74099999999999999,"15min":0.70499999999999996},"min":{"1min":0.48499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.238,"5min":3.238,"15min":3.238},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.77200000000000002,"15min":0.86499999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.48699999999999999},"max":{"1min":3.351,"5min":3.351,"15min":12.167},"last":0.48899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298072,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073768,"statfs":{"total":96636764160,"available":96331538432,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58299999999999996,"15min":0.57399999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69699999999999995,"5min":0.79800000000000004,"15min":1.0209999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60199999999999998,"15min":0.58199999999999996},"min":{"1min":0.41499999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70099999999999996,"5min":0.84099999999999997,"15min":1.161},"last":0.622}]},{"osd":1,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.60799999999999998,"15min":0.59699999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73699999999999999,"5min":0.88,"15min":1.069},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.39100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72699999999999998,"5min":0.78800000000000003,"15min":1.085},"last":0.48499999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59199999999999997,"15min":0.57399999999999995},"min":{"1min":0.38400000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.77000000000000002,"5min":0.85999999999999999,"15min":1.052},"last":0.435},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61399999999999999,"15min":0.61299999999999999},"min":{"1min":0.45700000000000002,"5min":0.40899999999999997,"15min":0.373},"max":{"1min":0.77600000000000002,"5min":0.80100000000000005,"15min":1.111},"last":0.39900000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61099999999999999,"15min":0.59999999999999998},"min":{"1min":0.41699999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.71999999999999997,"5min":0.78300000000000003,"15min":1.1299999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.45200000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.69699999999999995,"5min":0.88800000000000001,"15min":1.1419999999999999},"last":0.60399999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47599999999999998,"15min":0.47899999999999998},"min":{"1min":0.39300000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.64400000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.496,"15min":0.51100000000000001},"min":{"1min":0.33400000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.69799999999999995,"5min":0.76700000000000002,"15min":0.77800000000000002},"last":0.66800000000000004}]},{"osd":6,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.38300000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53100000000000003,"15min":0.54100000000000004},"min":{"1min":0.35499999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.67200000000000004,"5min":0.78200000000000003,"15min":0.95699999999999996},"last":0.65700000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.42799999999999999,"5min":0.375,"15min":0.35799999999999998},"max":{"1min":0.745,"5min":0.75900000000000001,"15min":0.76300000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.501,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.74099999999999999,"5min":0.78400000000000003,"15min":0.879},"last":0.64500000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298072,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073768,"statfs":{"total":96636764160,"available":96331538432,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.57699999999999996,"15min":0.55800000000000005},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.26000000000000001},"max":{"1min":2.0289999999999999,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57299999999999995,"15min":0.57599999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.498}]},{"osd":1,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61399999999999999,"15min":0.60899999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.3319999999999999,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63700000000000001,"15min":0.61699999999999999},"min":{"1min":0.435,"5min":0.41799999999999998,"15min":0.379},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.59599999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55500000000000005,"15min":0.56399999999999995},"min":{"1min":0.38200000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72599999999999998,"5min":0.83699999999999997,"15min":0.83799999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.60299999999999998,"15min":0.64700000000000002},"min":{"1min":0.39400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":6.4589999999999996},"last":0.53400000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57499999999999996,"15min":0.57399999999999995},"min":{"1min":0.35999999999999999,"5min":0.35799999999999998,"15min":0.314},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.88400000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.59899999999999998,"15min":0.64700000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":6.0629999999999997},"last":0.51700000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.26300000000000001},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48999999999999999,"15min":0.497},"min":{"1min":0.34499999999999997,"5min":0.315,"15min":0.28100000000000003},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.621}]},{"osd":6,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.48999999999999999,"15min":0.48499999999999999},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.27000000000000002},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52600000000000002,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.29299999999999998},"max":{"1min":1.99,"5min":1.99,"15min":1.99},"last":0.63400000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.56000000000000005,"15min":0.54900000000000004},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.5270000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55700000000000005,"15min":0.57299999999999995},"min":{"1min":0.36899999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.624,"5min":0.82199999999999995,"15min":0.83099999999999996},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:43:06.313 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:43:06.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.482+0000 7fde11636700 1 Processor -- start 2024-01-26T23:43:06.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.482+0000 7fde11636700 1 -- start start 2024-01-26T23:43:06.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.483+0000 7fde11636700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde0c0cbdf0 0x7fde0c0ae610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:06.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.483+0000 7fde11636700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fde0c0cc1c0 0x7fde0c0b0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:06.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.483+0000 7fde11636700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde0c0ad070 0x7fde0c0b1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:06.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.483+0000 7fde11636700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde0c0aeb50 con 0x7fde0c0ad070 2024-01-26T23:43:06.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.483+0000 7fde11636700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde0c0ae180 con 0x7fde0c0cc1c0 2024-01-26T23:43:06.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.483+0000 7fde11636700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c156c60 con 0x7fde0c0cbdf0 2024-01-26T23:43:06.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.484+0000 7fde0b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde0c0ad070 0x7fde0c0b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:06.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.484+0000 7fde0b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde0c0ad070 0x7fde0c0b1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50774/0 (socket says 172.21.15.46:50774) 2024-01-26T23:43:06.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.484+0000 7fde0b7fe700 1 -- 172.21.15.46:0/1430183334 learned_addr learned my addr 172.21.15.46:0/1430183334 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:06.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.484+0000 7fde0a7fc700 1 --2- 172.21.15.46:0/1430183334 >> v2:172.21.15.106:3300/0 conn(0x7fde0c0cc1c0 0x7fde0c0b0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.484+0000 7fde0affd700 1 --2- 172.21.15.46:0/1430183334 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde0c0cbdf0 0x7fde0c0ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.484+0000 7fde0b7fe700 1 -- 172.21.15.46:0/1430183334 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde0c0cbdf0 msgr2=0x7fde0c0ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.484+0000 7fde0b7fe700 1 --2- 172.21.15.46:0/1430183334 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde0c0cbdf0 0x7fde0c0ae610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.484+0000 7fde0b7fe700 1 -- 172.21.15.46:0/1430183334 >> v2:172.21.15.106:3300/0 conn(0x7fde0c0cc1c0 msgr2=0x7fde0c0b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.484+0000 7fde0b7fe700 1 --2- 172.21.15.46:0/1430183334 >> v2:172.21.15.106:3300/0 conn(0x7fde0c0cc1c0 0x7fde0c0b0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.484+0000 7fde0b7fe700 1 -- 172.21.15.46:0/1430183334 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c0ad570 con 0x7fde0c0ad070 2024-01-26T23:43:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.484+0000 7fde0b7fe700 1 --2- 172.21.15.46:0/1430183334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde0c0ad070 0x7fde0c0b1350 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fde0c070390 tx=0x7fddfc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78663e6e9f829cd5 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.484+0000 7fde09ffb700 1 -- 172.21.15.46:0/1430183334 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc00b640 con 0x7fde0c0ad070 2024-01-26T23:43:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.484+0000 7fde09ffb700 1 -- 172.21.15.46:0/1430183334 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddfc00b7a0 con 0x7fde0c0ad070 2024-01-26T23:43:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.484+0000 7fde09ffb700 1 -- 172.21.15.46:0/1430183334 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc00bb60 con 0x7fde0c0ad070 2024-01-26T23:43:06.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.485+0000 7fde11636700 1 -- 172.21.15.46:0/1430183334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde0c0ad070 msgr2=0x7fde0c0b1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.485+0000 7fde11636700 1 --2- 172.21.15.46:0/1430183334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde0c0ad070 0x7fde0c0b1350 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fde0c070390 tx=0x7fddfc009250 comp rx=0 tx=0).stop 2024-01-26T23:43:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.485+0000 7fde11636700 1 -- 172.21.15.46:0/1430183334 shutdown_connections 2024-01-26T23:43:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.485+0000 7fde11636700 1 --2- 172.21.15.46:0/1430183334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde0c0ad070 0x7fde0c0b1350 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.485+0000 7fde11636700 1 --2- 172.21.15.46:0/1430183334 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde0c0cbdf0 0x7fde0c0ae610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.485+0000 7fde11636700 1 --2- 172.21.15.46:0/1430183334 >> v2:172.21.15.106:3300/0 conn(0x7fde0c0cc1c0 0x7fde0c0b0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.485+0000 7fde11636700 1 -- 172.21.15.46:0/1430183334 >> 172.21.15.46:0/1430183334 conn(0x7fde0c068400 msgr2=0x7fde0c0cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.485+0000 7fde11636700 1 -- 172.21.15.46:0/1430183334 shutdown_connections 2024-01-26T23:43:06.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.485+0000 7fde11636700 1 -- 172.21.15.46:0/1430183334 wait complete. 2024-01-26T23:43:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.486+0000 7fde11636700 1 Processor -- start 2024-01-26T23:43:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.486+0000 7fde11636700 1 -- start start 2024-01-26T23:43:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.487+0000 7fde11636700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde0c0ad070 0x7fde0c0c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.487+0000 7fde11636700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde0c0cbdf0 0x7fde0c0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.487+0000 7fde11636700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde0c148190 0x7fde0c154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.487+0000 7fde11636700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c0c7d80 con 0x7fde0c0ad070 2024-01-26T23:43:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.487+0000 7fde11636700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c0c5950 con 0x7fde0c0cbdf0 2024-01-26T23:43:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.487+0000 7fde11636700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c0c2b80 con 0x7fde0c148190 2024-01-26T23:43:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.487+0000 7fde0affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde0c0ad070 0x7fde0c0c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.487+0000 7fde0affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde0c0ad070 0x7fde0c0c7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50790/0 (socket says 172.21.15.46:50790) 2024-01-26T23:43:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.487+0000 7fde0affd700 1 -- 172.21.15.46:0/181653330 learned_addr learned my addr 172.21.15.46:0/181653330 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:06.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.487+0000 7fde0b7fe700 1 --2- 172.21.15.46:0/181653330 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde0c148190 0x7fde0c154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.487+0000 7fde0a7fc700 1 --2- 172.21.15.46:0/181653330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde0c0cbdf0 0x7fde0c0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.488+0000 7fde0affd700 1 -- 172.21.15.46:0/181653330 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde0c148190 msgr2=0x7fde0c154900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.488+0000 7fde0affd700 1 --2- 172.21.15.46:0/181653330 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde0c148190 0x7fde0c154900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.488+0000 7fde0affd700 1 -- 172.21.15.46:0/181653330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde0c0cbdf0 msgr2=0x7fde0c0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.488+0000 7fde0affd700 1 --2- 172.21.15.46:0/181653330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde0c0cbdf0 0x7fde0c0c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.488+0000 7fde0affd700 1 -- 172.21.15.46:0/181653330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddfc0094f0 con 0x7fde0c0ad070 2024-01-26T23:43:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.488+0000 7fde0affd700 1 --2- 172.21.15.46:0/181653330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde0c0ad070 0x7fde0c0c7840 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fde04009f20 tx=0x7fde0400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.488+0000 7fde02ffd700 1 -- 172.21.15.46:0/181653330 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0400cc80 con 0x7fde0c0ad070 2024-01-26T23:43:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.488+0000 7fde02ffd700 1 -- 172.21.15.46:0/181653330 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0400f040 con 0x7fde0c0ad070 2024-01-26T23:43:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.488+0000 7fde11636700 1 -- 172.21.15.46:0/181653330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0c0c7f70 con 0x7fde0c0ad070 2024-01-26T23:43:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.488+0000 7fde02ffd700 1 -- 172.21.15.46:0/181653330 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04014480 con 0x7fde0c0ad070 2024-01-26T23:43:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.488+0000 7fde11636700 1 -- 172.21.15.46:0/181653330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde0c0c2810 con 0x7fde0c0ad070 2024-01-26T23:43:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.489+0000 7fde11636700 1 -- 172.21.15.46:0/181653330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0c0c2390 con 0x7fde0c0ad070 2024-01-26T23:43:06.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.490+0000 7fde02ffd700 1 -- 172.21.15.46:0/181653330 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fde040147f0 con 0x7fde0c0ad070 2024-01-26T23:43:06.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.490+0000 7fde02ffd700 1 --2- 172.21.15.46:0/181653330 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fddec066f30 0x7fddec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:06.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.490+0000 7fde02ffd700 1 -- 172.21.15.46:0/181653330 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fde04085520 con 0x7fde0c0ad070 2024-01-26T23:43:06.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.490+0000 7fde0a7fc700 1 --2- 172.21.15.46:0/181653330 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fddec066f30 0x7fddec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:06.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.491+0000 7fde0a7fc700 1 --2- 172.21.15.46:0/181653330 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fddec066f30 0x7fddec0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fde0c15afb0 tx=0x7fddf4008040 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.494+0000 7fde02ffd700 1 -- 172.21.15.46:0/181653330 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fde0c0c2390 con 0x7fde0c0ad070 2024-01-26T23:43:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.696+0000 7fde11636700 1 -- 172.21.15.46:0/181653330 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fde0c0c2cc0 con 0x7fddec066f30 2024-01-26T23:43:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.705+0000 7fde02ffd700 1 -- 172.21.15.46:0/181653330 <== mgr.5267 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278586 (secure 0 0 0) 0x7fddf40098e0 con 0x7fddec066f30 2024-01-26T23:43:06.709 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:43:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.708+0000 7fde11636700 1 -- 172.21.15.46:0/181653330 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fddec066f30 msgr2=0x7fddec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.708+0000 7fde11636700 1 --2- 172.21.15.46:0/181653330 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fddec066f30 0x7fddec0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fde0c15afb0 tx=0x7fddf4008040 comp rx=0 tx=0).stop 2024-01-26T23:43:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.708+0000 7fde11636700 1 -- 172.21.15.46:0/181653330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde0c0ad070 msgr2=0x7fde0c0c7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.708+0000 7fde11636700 1 --2- 172.21.15.46:0/181653330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde0c0ad070 0x7fde0c0c7840 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fde04009f20 tx=0x7fde0400b040 comp rx=0 tx=0).stop 2024-01-26T23:43:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.708+0000 7fde11636700 1 -- 172.21.15.46:0/181653330 shutdown_connections 2024-01-26T23:43:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.708+0000 7fde11636700 1 --2- 172.21.15.46:0/181653330 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fddec066f30 0x7fddec0693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.708+0000 7fde11636700 1 --2- 172.21.15.46:0/181653330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde0c0ad070 0x7fde0c0c7840 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.708+0000 7fde11636700 1 --2- 172.21.15.46:0/181653330 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde0c148190 0x7fde0c154900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.708+0000 7fde11636700 1 --2- 172.21.15.46:0/181653330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde0c0cbdf0 0x7fde0c0c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.708+0000 7fde11636700 1 -- 172.21.15.46:0/181653330 >> 172.21.15.46:0/181653330 conn(0x7fde0c068400 msgr2=0x7fde0c06b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.708+0000 7fde11636700 1 -- 172.21.15.46:0/181653330 shutdown_connections 2024-01-26T23:43:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.708+0000 7fde11636700 1 -- 172.21.15.46:0/181653330 wait complete. 2024-01-26T23:43:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:43:06.723 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-26T23:43:05.642333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2418492,"kb_used_data":7420,"kb_used_omap":0,"kb_used_meta":2411008,"kb_avail":752556228,"statfs":{"total":773094113280,"available":770617577472,"internally_reserved":0,"allocated":7598080,"data_stored":4341407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2468872192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.946913"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.702730+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:55.750309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697701+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:55.750309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697807+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:15:45.000899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697169+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:00:27.828427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.705515+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:01:20.459485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.706547+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:48:06.519946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701638+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:07:07.891870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704532+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:03.697032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701642+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:24.351557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.706514+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:19.596229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.702053+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:34:08.288094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.696768+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:05.307132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704628+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:20:40.758787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.703147+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:22:30.811610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.702374+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:51:32.002538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.696777+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:08:26.147843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704560+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:41:09.634181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.699768+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:18:47.545463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701502+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:21:48.776990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701431+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:05:54.731798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697780+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:16:07.169692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.698213+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:36.728509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704561+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:38:03.176048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.699836+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:29:14.906700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.702155+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:29:51.950015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701472+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:31:22.376213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704797+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:55:52.786999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.696764+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:48:06.519946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.696894+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:19.596229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.706535+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:05.307132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697086+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:51:32.002538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704965+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:00:58.220861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"61'139","reported_seq":283,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-01-26T23:42:59.110217+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:42:59.110217+0000","last_peered":"2024-01-26T23:42:59.110217+0000","last_clean":"2024-01-26T23:42:59.110217+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:42:59.110217+0000","last_undegraded":"2024-01-26T23:42:59.110217+0000","last_fullsized":"2024-01-26T23:42:59.110217+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575011,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306844,"kb_used_data":1364,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064996,"statfs":{"total":96636764160,"available":96322555904,"internally_reserved":0,"allocated":1396736,"data_stored":986263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59599999999999997,"15min":0.56499999999999995},"min":{"1min":0.51100000000000001,"5min":0.371,"15min":0.317},"max":{"1min":0.78300000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61899999999999999,"15min":0.57899999999999996},"min":{"1min":0.50600000000000001,"5min":0.44500000000000001,"15min":0.34699999999999998},"max":{"1min":0.80300000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.32800000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.61299999999999999,"15min":0.56799999999999995},"min":{"1min":0.45500000000000002,"5min":0.43099999999999999,"15min":0.36399999999999999},"max":{"1min":0.81299999999999994,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.61899999999999999,"15min":0.58299999999999996},"min":{"1min":0.52300000000000002,"5min":0.41099999999999998,"15min":0.36099999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.85199999999999998},"last":0.46899999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.59799999999999998,"15min":0.56999999999999995},"min":{"1min":0.51000000000000001,"5min":0.47699999999999998,"15min":0.35899999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.81699999999999995},"last":0.378},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.61199999999999999,"15min":0.59199999999999997},"min":{"1min":0.50600000000000001,"5min":0.41699999999999998,"15min":0.38400000000000001},"max":{"1min":0.77300000000000002,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.44}]},{"osd":3,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60999999999999999,"15min":0.58599999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.35399999999999998},"max":{"1min":0.76900000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.621,"15min":0.60399999999999998},"min":{"1min":0.53500000000000003,"5min":0.433,"15min":0.373},"max":{"1min":0.79200000000000004,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.48699999999999999}]},{"osd":4,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46500000000000002,"15min":0.46899999999999997},"min":{"1min":0.33300000000000002,"5min":0.31900000000000001,"15min":0.28999999999999998},"max":{"1min":0.65600000000000003,"5min":0.72699999999999998,"15min":0.90100000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.49099999999999999,"15min":0.498},"min":{"1min":0.38500000000000001,"5min":0.33100000000000002,"15min":0.314},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.73799999999999999},"last":0.41799999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.498,"15min":0.50600000000000001},"min":{"1min":0.35399999999999998,"5min":0.32000000000000001,"15min":0.30599999999999999},"max":{"1min":0.71299999999999997,"5min":0.745,"15min":0.91700000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.40799999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.96599999999999997},"last":0.53200000000000003}]},{"osd":6,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56699999999999995,"15min":0.55900000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.33100000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.93500000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.432,"5min":0.38100000000000001,"15min":0.35299999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.86899999999999999},"last":0.499}]}]},{"osd":6,"up_from":13,"seq":55834575011,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302168,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069672,"statfs":{"total":96636764160,"available":96327344128,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.58599999999999997,"15min":0.56100000000000005},"min":{"1min":0.30199999999999999,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":0.71999999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60299999999999998,"15min":0.58399999999999996},"min":{"1min":0.41399999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71699999999999997,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.58199999999999996}]},{"osd":1,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.57999999999999996,"15min":0.56899999999999995},"min":{"1min":0.41699999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.60699999999999998,"15min":0.59099999999999997},"min":{"1min":0.38100000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73499999999999999,"5min":0.874,"15min":0.874},"last":0.54900000000000004}]},{"osd":2,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.58999999999999997,"15min":0.57799999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.29599999999999999},"max":{"1min":0.73299999999999998,"5min":0.78100000000000003,"15min":0.85799999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59799999999999998,"15min":0.58299999999999996},"min":{"1min":0.45300000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76200000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.61099999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60499999999999998,"15min":0.59399999999999997},"min":{"1min":0.435,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.80700000000000005,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.62,"15min":0.61499999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39000000000000001},"max":{"1min":0.71699999999999997,"5min":0.82299999999999995,"15min":0.84499999999999997},"last":0.628}]},{"osd":4,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.46999999999999997,"15min":0.47399999999999998},"min":{"1min":0.33300000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.70399999999999996,"5min":0.77800000000000002,"15min":0.90200000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.48799999999999999,"15min":0.498},"min":{"1min":0.32000000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.67400000000000004,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.53300000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.47999999999999998,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.69499999999999995,"5min":0.76500000000000001,"15min":0.78200000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51800000000000002,"15min":0.53000000000000003},"min":{"1min":0.34399999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.77900000000000003,"5min":0.83499999999999996,"15min":0.96499999999999997},"last":0.46400000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52200000000000002,"15min":0.54000000000000004},"min":{"1min":0.38600000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.73599999999999999,"5min":0.85899999999999999,"15min":0.97199999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.308},"max":{"1min":0.70399999999999996,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56200000000000006}]}]},{"osd":1,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302168,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069672,"statfs":{"total":96636764160,"available":96327344128,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39500000000000002,"15min":0.40600000000000003},"min":{"1min":0.26500000000000001,"5min":0.26200000000000001,"15min":0.23200000000000001},"max":{"1min":0.57799999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.621},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.441},"min":{"1min":0.32500000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.58199999999999996,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.41599999999999998,"15min":0.40699999999999997},"min":{"1min":0.33700000000000002,"5min":0.30599999999999999,"15min":0.26200000000000001},"max":{"1min":0.61399999999999999,"5min":0.63100000000000001,"15min":0.65900000000000003},"last":0.499},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.38600000000000001,"5min":0.32600000000000001,"15min":0.28999999999999998},"max":{"1min":0.69499999999999995,"5min":0.71899999999999997,"15min":0.79600000000000004},"last":0.51300000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.505,"15min":0.505},"min":{"1min":0.42499999999999999,"5min":0.33800000000000002,"15min":0.32100000000000001},"max":{"1min":0.69099999999999995,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.81999999999999995},"last":0.436}]},{"osd":4,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.66700000000000004,"15min":0.64000000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.39400000000000002},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.65900000000000003,"15min":0.66700000000000004},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.433},"max":{"1min":0.91900000000000004,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.82199999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.67900000000000005,"15min":0.67600000000000005},"min":{"1min":0.49299999999999999,"5min":0.45000000000000001,"15min":0.42799999999999999},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.67400000000000004,"15min":0.66600000000000004},"min":{"1min":0.57699999999999996,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.92200000000000004,"5min":0.98599999999999999,"15min":1.018},"last":0.63200000000000001}]},{"osd":6,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.68899999999999995,"15min":0.68600000000000005},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.45700000000000002},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.1279999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.69199999999999995,"15min":0.69099999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.42599999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.78500000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.72099999999999997,"15min":0.71399999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.49399999999999999},"max":{"1min":0.94699999999999995,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.59199999999999997,"5min":0.5,"15min":0.438},"max":{"1min":1.048,"5min":1.048,"15min":1.3049999999999999},"last":0.80200000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302736,"kb_used_data":1352,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069104,"statfs":{"total":96636764160,"available":96326762496,"internally_reserved":0,"allocated":1384448,"data_stored":979774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40100000000000002,"15min":0.41299999999999998},"min":{"1min":0.27800000000000002,"5min":0.26400000000000001,"15min":0.214},"max":{"1min":0.52200000000000002,"5min":0.64100000000000001,"15min":1.284},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46899999999999997,"15min":0.45900000000000002},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.27300000000000002},"max":{"1min":0.72699999999999998,"5min":0.75800000000000001,"15min":1.024},"last":0.30499999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.433,"15min":0.443},"min":{"1min":0.318,"5min":0.28199999999999997,"15min":0.26700000000000002},"max":{"1min":0.59699999999999998,"5min":0.66100000000000003,"15min":1.2230000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.68100000000000005,"5min":0.68300000000000005,"15min":1.2430000000000001},"last":0.375}]},{"osd":3,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.47999999999999998,"15min":0.47599999999999998},"min":{"1min":0.38800000000000001,"5min":0.317,"15min":0.26800000000000002},"max":{"1min":0.626,"5min":0.69899999999999995,"15min":1.0980000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.498,"15min":0.5},"min":{"1min":0.39500000000000002,"5min":0.34000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69799999999999995,"5min":0.74099999999999999,"15min":1.1850000000000001},"last":0.41299999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.65200000000000002,"15min":0.65700000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.372},"max":{"1min":0.86799999999999999,"5min":0.93300000000000005,"15min":1.133},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66600000000000004,"15min":0.66900000000000004},"min":{"1min":0.52900000000000003,"5min":0.443,"15min":0.443},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":1.0169999999999999},"last":0.55400000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.66100000000000003,"15min":0.66900000000000004},"min":{"1min":0.46500000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":1.0880000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68200000000000005,"15min":0.69199999999999995},"min":{"1min":0.504,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93300000000000005,"5min":0.98699999999999999,"15min":1.169},"last":0.58799999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67300000000000004,"15min":0.69399999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.44800000000000001},"max":{"1min":0.88400000000000001,"5min":0.97299999999999998,"15min":1.071},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69399999999999995,"15min":0.70999999999999996},"min":{"1min":0.55200000000000005,"5min":0.502,"15min":0.48599999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.1160000000000001},"last":0.60399999999999998}]},{"osd":7,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.72099999999999997,"15min":0.73999999999999999},"min":{"1min":0.51300000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.73699999999999999,"15min":0.749},"min":{"1min":0.58999999999999997,"5min":0.433,"15min":0.433},"max":{"1min":0.98699999999999999,"5min":1.028,"15min":1.2010000000000001},"last":0.68899999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306264,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065576,"statfs":{"total":96636764160,"available":96323149824,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.503},"min":{"1min":0.28399999999999997,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":0.53200000000000003,"5min":0.56699999999999995,"15min":9.0540000000000003},"last":0.442},{"interface":"front","average":{"1min":0.443,"5min":0.42199999999999999,"15min":0.46200000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.24299999999999999},"max":{"1min":0.57599999999999996,"5min":0.64000000000000001,"15min":4.1970000000000001},"last":0.307}]},{"osd":1,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.438,"15min":0.42299999999999999},"min":{"1min":0.28499999999999998,"5min":0.27400000000000002,"15min":0.221},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.82199999999999995},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.502,"15min":0.64800000000000002},"min":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.316},"max":{"1min":0.65900000000000003,"5min":0.72499999999999998,"15min":12.157},"last":0.45700000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.502,"15min":0.48599999999999999},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33500000000000002},"max":{"1min":0.65500000000000003,"5min":0.74099999999999999,"15min":0.86499999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47899999999999998,"15min":0.57699999999999996},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.30399999999999999},"max":{"1min":0.63600000000000001,"5min":0.73199999999999998,"15min":9.0879999999999992},"last":0.38300000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.65100000000000002,"15min":0.749},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.88500000000000001,"5min":1.048,"15min":9.0150000000000006},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.64600000000000002,"15min":0.68400000000000005},"min":{"1min":0.55700000000000005,"5min":0.371,"15min":0.371},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":4.2480000000000002},"last":0.63500000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66400000000000003,"15min":0.69799999999999995},"min":{"1min":0.55500000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.85899999999999999,"5min":1.008,"15min":4.3200000000000003},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.69199999999999995,"15min":0.67800000000000005},"min":{"1min":0.52000000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.95199999999999996,"5min":1.085,"15min":1.085},"last":0.621}]},{"osd":6,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.69899999999999995,"15min":0.68600000000000005},"min":{"1min":0.46999999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.505},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.68899999999999995,"15min":0.78800000000000003},"min":{"1min":0.53500000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.94699999999999995,"5min":1.111,"15min":8.9700000000000006},"last":0.64700000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.70799999999999996,"15min":0.752},"min":{"1min":0.61099999999999999,"5min":0.54100000000000004,"15min":0.45800000000000002},"max":{"1min":0.93700000000000006,"5min":1.127,"15min":4.282},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.746,"15min":0.73499999999999999},"min":{"1min":0.52900000000000003,"5min":0.46100000000000002,"15min":0.42099999999999999},"max":{"1min":1.0329999999999999,"5min":1.139,"15min":1.139},"last":0.66100000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302168,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069672,"statfs":{"total":96636764160,"available":96327344128,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.23999999999999999},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44900000000000001,"15min":0.56100000000000005},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":11.968},"last":0.504}]},{"osd":1,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.45600000000000002,"15min":0.57299999999999995},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.633,"5min":1.633,"15min":12.050000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48299999999999998,"15min":0.48999999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.2150000000000001},"last":0.378}]},{"osd":2,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.48899999999999999,"15min":0.59399999999999997},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.28999999999999998},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":12.234},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.46999999999999997,"15min":0.58699999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":11.925000000000001},"last":0.39700000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.70199999999999996,"15min":0.78200000000000003},"min":{"1min":0.46999999999999997,"5min":0.45500000000000002,"15min":0.40200000000000002},"max":{"1min":3.306,"5min":3.306,"15min":12.099},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68000000000000005,"15min":0.66900000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.379},"max":{"1min":0.89700000000000002,"5min":1.0009999999999999,"15min":1.0649999999999999},"last":0.46899999999999997}]},{"osd":5,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67700000000000005,"15min":0.79300000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.593,"5min":1.593,"15min":12.199999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.69099999999999995,"15min":0.68000000000000005},"min":{"1min":0.45300000000000001,"5min":0.439,"15min":0.436},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.56999999999999995}]},{"osd":6,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.70099999999999996,"15min":0.80000000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.506,"5min":1.506,"15min":12.132999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.70799999999999996,"15min":0.70199999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.53700000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.74099999999999999,"15min":0.70499999999999996},"min":{"1min":0.48499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.238,"5min":3.238,"15min":3.238},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.77200000000000002,"15min":0.86499999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.48699999999999999},"max":{"1min":3.351,"5min":3.351,"15min":12.167},"last":0.48899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298072,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073768,"statfs":{"total":96636764160,"available":96331538432,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58299999999999996,"15min":0.57399999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69699999999999995,"5min":0.79800000000000004,"15min":1.0209999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60199999999999998,"15min":0.58199999999999996},"min":{"1min":0.41499999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70099999999999996,"5min":0.84099999999999997,"15min":1.161},"last":0.622}]},{"osd":1,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.60799999999999998,"15min":0.59699999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73699999999999999,"5min":0.88,"15min":1.069},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.39100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72699999999999998,"5min":0.78800000000000003,"15min":1.085},"last":0.48499999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59199999999999997,"15min":0.57399999999999995},"min":{"1min":0.38400000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.77000000000000002,"5min":0.85999999999999999,"15min":1.052},"last":0.435},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61399999999999999,"15min":0.61299999999999999},"min":{"1min":0.45700000000000002,"5min":0.40899999999999997,"15min":0.373},"max":{"1min":0.77600000000000002,"5min":0.80100000000000005,"15min":1.111},"last":0.39900000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61099999999999999,"15min":0.59999999999999998},"min":{"1min":0.41699999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.71999999999999997,"5min":0.78300000000000003,"15min":1.1299999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.45200000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.69699999999999995,"5min":0.88800000000000001,"15min":1.1419999999999999},"last":0.60399999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47599999999999998,"15min":0.47899999999999998},"min":{"1min":0.39300000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.64400000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.496,"15min":0.51100000000000001},"min":{"1min":0.33400000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.69799999999999995,"5min":0.76700000000000002,"15min":0.77800000000000002},"last":0.66800000000000004}]},{"osd":6,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.38300000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53100000000000003,"15min":0.54100000000000004},"min":{"1min":0.35499999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.67200000000000004,"5min":0.78200000000000003,"15min":0.95699999999999996},"last":0.65700000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.42799999999999999,"5min":0.375,"15min":0.35799999999999998},"max":{"1min":0.745,"5min":0.75900000000000001,"15min":0.76300000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.501,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.74099999999999999,"5min":0.78400000000000003,"15min":0.879},"last":0.64500000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298072,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073768,"statfs":{"total":96636764160,"available":96331538432,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.57699999999999996,"15min":0.55800000000000005},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.26000000000000001},"max":{"1min":2.0289999999999999,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57299999999999995,"15min":0.57599999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.498}]},{"osd":1,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61399999999999999,"15min":0.60899999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.3319999999999999,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63700000000000001,"15min":0.61699999999999999},"min":{"1min":0.435,"5min":0.41799999999999998,"15min":0.379},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.59599999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55500000000000005,"15min":0.56399999999999995},"min":{"1min":0.38200000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72599999999999998,"5min":0.83699999999999997,"15min":0.83799999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.60299999999999998,"15min":0.64700000000000002},"min":{"1min":0.39400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":6.4589999999999996},"last":0.53400000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57499999999999996,"15min":0.57399999999999995},"min":{"1min":0.35999999999999999,"5min":0.35799999999999998,"15min":0.314},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.88400000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.59899999999999998,"15min":0.64700000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":6.0629999999999997},"last":0.51700000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.26300000000000001},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48999999999999999,"15min":0.497},"min":{"1min":0.34499999999999997,"5min":0.315,"15min":0.28100000000000003},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.621}]},{"osd":6,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.48999999999999999,"15min":0.48499999999999999},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.27000000000000002},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52600000000000002,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.29299999999999998},"max":{"1min":1.99,"5min":1.99,"15min":1.99},"last":0.63400000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.56000000000000005,"15min":0.54900000000000004},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.5270000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55700000000000005,"15min":0.57299999999999995},"min":{"1min":0.36899999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.624,"5min":0.82199999999999995,"15min":0.83099999999999996},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:43:06.729 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3f is not active+clean 2024-01-26T23:43:06.729 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.729 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3e is not active+clean 2024-01-26T23:43:06.729 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.729 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3d is not active+clean 2024-01-26T23:43:06.729 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.729 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3c is not active+clean 2024-01-26T23:43:06.729 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.729 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3b is not active+clean 2024-01-26T23:43:06.730 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.730 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3a is not active+clean 2024-01-26T23:43:06.730 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.730 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.39 is not active+clean 2024-01-26T23:43:06.730 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.730 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.38 is not active+clean 2024-01-26T23:43:06.730 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.730 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.37 is not active+clean 2024-01-26T23:43:06.730 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.730 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.36 is not active+clean 2024-01-26T23:43:06.730 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.731 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.35 is not active+clean 2024-01-26T23:43:06.731 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.731 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.34 is not active+clean 2024-01-26T23:43:06.731 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.731 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.33 is not active+clean 2024-01-26T23:43:06.731 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.731 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.32 is not active+clean 2024-01-26T23:43:06.731 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.731 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.31 is not active+clean 2024-01-26T23:43:06.731 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.732 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.30 is not active+clean 2024-01-26T23:43:06.732 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.732 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2f is not active+clean 2024-01-26T23:43:06.732 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.732 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2e is not active+clean 2024-01-26T23:43:06.732 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.732 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2d is not active+clean 2024-01-26T23:43:06.732 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.732 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2c is not active+clean 2024-01-26T23:43:06.732 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.733 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2b is not active+clean 2024-01-26T23:43:06.733 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.733 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2a is not active+clean 2024-01-26T23:43:06.733 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.733 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.29 is not active+clean 2024-01-26T23:43:06.733 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.734 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.28 is not active+clean 2024-01-26T23:43:06.734 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.734 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.27 is not active+clean 2024-01-26T23:43:06.734 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.734 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.26 is not active+clean 2024-01-26T23:43:06.734 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.734 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.25 is not active+clean 2024-01-26T23:43:06.734 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.734 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.24 is not active+clean 2024-01-26T23:43:06.735 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.735 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.23 is not active+clean 2024-01-26T23:43:06.735 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.735 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.22 is not active+clean 2024-01-26T23:43:06.735 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.735 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.21 is not active+clean 2024-01-26T23:43:06.735 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.735 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.20 is not active+clean 2024-01-26T23:43:06.735 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.735 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.10 is not active+clean 2024-01-26T23:43:06.736 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.736 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1f is not active+clean 2024-01-26T23:43:06.736 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.702730+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T01:42:55.750309+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-26T23:43:06.736 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.11 is not active+clean 2024-01-26T23:43:06.736 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.736 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1e is not active+clean 2024-01-26T23:43:06.737 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.697701+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T01:42:55.750309+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-26T23:43:06.737 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.12 is not active+clean 2024-01-26T23:43:06.737 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.737 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1d is not active+clean 2024-01-26T23:43:06.737 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.697807+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T11:15:45.000899+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-26T23:43:06.738 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.13 is not active+clean 2024-01-26T23:43:06.738 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.738 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1c is not active+clean 2024-01-26T23:43:06.738 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.697169+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T09:00:27.828427+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-26T23:43:06.738 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.14 is not active+clean 2024-01-26T23:43:06.738 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.738 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1b is not active+clean 2024-01-26T23:43:06.738 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.705515+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T02:01:20.459485+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-26T23:43:06.738 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.15 is not active+clean 2024-01-26T23:43:06.738 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.739 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1a is not active+clean 2024-01-26T23:43:06.739 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.706547+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T02:48:06.519946+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-26T23:43:06.739 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.16 is not active+clean 2024-01-26T23:43:06.739 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.739 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.19 is not active+clean 2024-01-26T23:43:06.739 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.701638+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T08:07:07.891870+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-26T23:43:06.739 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.17 is not active+clean 2024-01-26T23:43:06.739 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.739 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.18 is not active+clean 2024-01-26T23:43:06.739 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.704532+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T11:17:03.697032+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-26T23:43:06.740 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.18 is not active+clean 2024-01-26T23:43:06.740 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.740 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.17 is not active+clean 2024-01-26T23:43:06.740 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.701642+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T05:54:24.351557+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-26T23:43:06.740 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.19 is not active+clean 2024-01-26T23:43:06.740 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.740 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.16 is not active+clean 2024-01-26T23:43:06.740 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.706514+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T08:59:19.596229+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-26T23:43:06.740 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1a is not active+clean 2024-01-26T23:43:06.740 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.740 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.15 is not active+clean 2024-01-26T23:43:06.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.702053+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:34:08.288094+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-26T23:43:06.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1b is not active+clean 2024-01-26T23:43:06.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.14 is not active+clean 2024-01-26T23:43:06.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.696768+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T05:34:05.307132+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-26T23:43:06.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1c is not active+clean 2024-01-26T23:43:06.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.13 is not active+clean 2024-01-26T23:43:06.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.704628+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T07:20:40.758787+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-26T23:43:06.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1d is not active+clean 2024-01-26T23:43:06.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.12 is not active+clean 2024-01-26T23:43:06.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.703147+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T01:22:30.811610+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-26T23:43:06.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1e is not active+clean 2024-01-26T23:43:06.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.11 is not active+clean 2024-01-26T23:43:06.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.702374+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T07:51:32.002538+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-26T23:43:06.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1f is not active+clean 2024-01-26T23:43:06.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.10 is not active+clean 2024-01-26T23:43:06.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.696777+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T02:08:26.147843+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-26T23:43:06.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.0 is not active+clean 2024-01-26T23:43:06.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.f is not active+clean 2024-01-26T23:43:06.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.704560+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T05:41:09.634181+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-26T23:43:06.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1 is not active+clean 2024-01-26T23:43:06.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.e is not active+clean 2024-01-26T23:43:06.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.699768+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T05:18:47.545463+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-26T23:43:06.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2 is not active+clean 2024-01-26T23:43:06.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.d is not active+clean 2024-01-26T23:43:06.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.701502+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T10:21:48.776990+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-26T23:43:06.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3 is not active+clean 2024-01-26T23:43:06.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.c is not active+clean 2024-01-26T23:43:06.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.701431+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T01:05:54.731798+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-26T23:43:06.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.4 is not active+clean 2024-01-26T23:43:06.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.b is not active+clean 2024-01-26T23:43:06.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.697780+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T03:16:07.169692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-26T23:43:06.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.5 is not active+clean 2024-01-26T23:43:06.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.a is not active+clean 2024-01-26T23:43:06.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.698213+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T08:46:36.728509+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-26T23:43:06.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.6 is not active+clean 2024-01-26T23:43:06.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.9 is not active+clean 2024-01-26T23:43:06.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.704561+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T00:38:03.176048+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-26T23:43:06.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.7 is not active+clean 2024-01-26T23:43:06.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.8 is not active+clean 2024-01-26T23:43:06.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.699836+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T08:29:14.906700+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-26T23:43:06.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.8 is not active+clean 2024-01-26T23:43:06.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.7 is not active+clean 2024-01-26T23:43:06.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.702155+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:29:51.950015+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-26T23:43:06.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.a is not active+clean 2024-01-26T23:43:06.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.5 is not active+clean 2024-01-26T23:43:06.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.701472+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T10:31:22.376213+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-26T23:43:06.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.b is not active+clean 2024-01-26T23:43:06.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.4 is not active+clean 2024-01-26T23:43:06.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.704797+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T09:55:52.786999+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-26T23:43:06.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.c is not active+clean 2024-01-26T23:43:06.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3 is not active+clean 2024-01-26T23:43:06.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.696764+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T02:48:06.519946+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-26T23:43:06.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.d is not active+clean 2024-01-26T23:43:06.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2 is not active+clean 2024-01-26T23:43:06.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.696894+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T08:59:19.596229+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-26T23:43:06.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.e is not active+clean 2024-01-26T23:43:06.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1 is not active+clean 2024-01-26T23:43:06.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.706535+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T05:34:05.307132+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-26T23:43:06.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.f is not active+clean 2024-01-26T23:43:06.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.0 is not active+clean 2024-01-26T23:43:06.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.697086+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T07:51:32.002538+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-26T23:43:06.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.9 is not active+clean 2024-01-26T23:43:06.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:43:00.694334+0000', 'last_change': '2024-01-26T23:43:00.694334+0000', 'last_active': '2024-01-26T23:43:00.694334+0000', 'last_peered': '2024-01-26T23:43:00.694334+0000', 'last_clean': '2024-01-26T23:43:00.694334+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:43:00.694334+0000', 'last_undegraded': '2024-01-26T23:43:00.694334+0000', 'last_fullsized': '2024-01-26T23:43:00.694334+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'last_clean_scrub_stamp': '2024-01-26T23:43:00.694334+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:43:06.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.6 is not active+clean 2024-01-26T23:43:06.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:42:59.687004+0000', 'last_change': '2024-01-26T23:42:59.704965+0000', 'last_active': '2024-01-26T23:42:59.687004+0000', 'last_peered': '2024-01-26T23:42:59.687004+0000', 'last_clean': '2024-01-26T23:42:59.687004+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:42:59.687004+0000', 'last_undegraded': '2024-01-26T23:42:59.687004+0000', 'last_fullsized': '2024-01-26T23:42:59.687004+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'last_clean_scrub_stamp': '2024-01-26T23:42:59.687004+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T02:00:58.220861+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-26T23:43:06.749 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-26T23:43:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.896+0000 7f831446e700 1 Processor -- start 2024-01-26T23:43:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.896+0000 7f831446e700 1 -- start start 2024-01-26T23:43:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.897+0000 7f831446e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f830c1596e0 0x7f830c159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.897+0000 7f831446e700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830c0c97f0 con 0x7f830c0cb4c0 2024-01-26T23:43:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.897+0000 7f831446e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830c0c8c20 con 0x7f830c0c4720 2024-01-26T23:43:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.897+0000 7f831446e700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f830c0ce0d0 con 0x7f830c1596e0 2024-01-26T23:43:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.897+0000 7f8312a0b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f830c0cb4c0 0x7f830c0cd9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37776/0 (socket says 172.21.15.46:37776) 2024-01-26T23:43:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.898+0000 7f8312a0b700 1 -- 172.21.15.46:0/515862745 learned_addr learned my addr 172.21.15.46:0/515862745 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:43:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.898+0000 7f831220a700 1 --2- 172.21.15.46:0/515862745 >> v2:172.21.15.106:3300/0 conn(0x7f830c1596e0 0x7f830c159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.898+0000 7f8311208700 1 -- 172.21.15.46:0/515862745 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830471516 0 0) 0x7f830c0c97f0 con 0x7f830c0cb4c0 2024-01-26T23:43:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.898+0000 7f8311208700 1 -- 172.21.15.46:0/515862745 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82f8003570 con 0x7f830c0cb4c0 2024-01-26T23:43:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.898+0000 7f831220a700 1 -- 172.21.15.46:0/515862745 >> v1:172.21.15.106:6789/0 conn(0x7f830c0c4720 legacy=0x7f830c0cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.898+0000 7f831220a700 1 -- 172.21.15.46:0/515862745 >> v1:172.21.15.46:6789/0 conn(0x7f830c0cb4c0 legacy=0x7f830c0cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.898+0000 7f8311208700 1 -- 172.21.15.46:0/515862745 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180053010 0 0) 0x7f830c0c8c20 con 0x7f830c0c4720 2024-01-26T23:43:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.898+0000 7f831220a700 1 -- 172.21.15.46:0/515862745 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830c0c8ec0 con 0x7f830c1596e0 2024-01-26T23:43:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.898+0000 7f831220a700 1 --2- 172.21.15.46:0/515862745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f830c1596e0 0x7f830c159ab0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f83080098e0 tx=0x7f83080091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcbd5ffef6e11218 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.899+0000 7f8311208700 1 -- 172.21.15.46:0/515862745 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8308011630 con 0x7f830c1596e0 2024-01-26T23:43:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.899+0000 7f8311208700 1 -- 172.21.15.46:0/515862745 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8308011790 con 0x7f830c1596e0 2024-01-26T23:43:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.899+0000 7f8311208700 1 -- 172.21.15.46:0/515862745 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8308011b30 con 0x7f830c1596e0 2024-01-26T23:43:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.899+0000 7f831446e700 1 -- 172.21.15.46:0/515862745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f830c1596e0 msgr2=0x7f830c159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.899+0000 7f831446e700 1 --2- 172.21.15.46:0/515862745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f830c1596e0 0x7f830c159ab0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f83080098e0 tx=0x7f83080091a0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.899+0000 7f831446e700 1 -- 172.21.15.46:0/515862745 shutdown_connections 2024-01-26T23:43:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.899+0000 7f831446e700 1 --2- 172.21.15.46:0/515862745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f830c1596e0 0x7f830c159ab0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.899+0000 7f831446e700 1 -- 172.21.15.46:0/515862745 >> 172.21.15.46:0/515862745 conn(0x7f830c097820 msgr2=0x7f830c068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.899+0000 7f831446e700 1 -- 172.21.15.46:0/515862745 shutdown_connections 2024-01-26T23:43:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.899+0000 7f831446e700 1 -- 172.21.15.46:0/515862745 wait complete. 2024-01-26T23:43:06.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.900+0000 7f831446e700 1 Processor -- start 2024-01-26T23:43:06.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.900+0000 7f831446e700 1 -- start start 2024-01-26T23:43:06.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.901+0000 7f831446e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f830c0c4720 0x7f830c158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.901+0000 7f831446e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f830c0cb4c0 0x7f830c0c7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.901+0000 7f831446e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f830c1596e0 0x7f830c0c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.901+0000 7f831446e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c0c5ac0 con 0x7f830c1596e0 2024-01-26T23:43:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.901+0000 7f831446e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c0c5770 con 0x7f830c0c4720 2024-01-26T23:43:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.901+0000 7f831446e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f830c0b2ad0 con 0x7f830c0cb4c0 2024-01-26T23:43:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.901+0000 7f8312a0b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f830c1596e0 0x7f830c0c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.901+0000 7f8312a0b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f830c1596e0 0x7f830c0c7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50802/0 (socket says 172.21.15.46:50802) 2024-01-26T23:43:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.901+0000 7f8312a0b700 1 -- 172.21.15.46:0/141013998 learned_addr learned my addr 172.21.15.46:0/141013998 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.901+0000 7f831220a700 1 --2- 172.21.15.46:0/141013998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f830c0c4720 0x7f830c158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.901+0000 7f8311a09700 1 --2- 172.21.15.46:0/141013998 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f830c0cb4c0 0x7f830c0c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.902+0000 7f8312a0b700 1 -- 172.21.15.46:0/141013998 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f830c0cb4c0 msgr2=0x7f830c0c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.902+0000 7f8312a0b700 1 --2- 172.21.15.46:0/141013998 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f830c0cb4c0 0x7f830c0c7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.902+0000 7f8312a0b700 1 -- 172.21.15.46:0/141013998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f830c0c4720 msgr2=0x7f830c158210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.902+0000 7f8312a0b700 1 --2- 172.21.15.46:0/141013998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f830c0c4720 0x7f830c158210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.902+0000 7f8312a0b700 1 -- 172.21.15.46:0/141013998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8308009420 con 0x7f830c1596e0 2024-01-26T23:43:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.902+0000 7f8312a0b700 1 --2- 172.21.15.46:0/141013998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f830c1596e0 0x7f830c0c7d70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f830400cb60 tx=0x7f83040018a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.902+0000 7f83027fc700 1 -- 172.21.15.46:0/141013998 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8304004dc0 con 0x7f830c1596e0 2024-01-26T23:43:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.902+0000 7f83027fc700 1 -- 172.21.15.46:0/141013998 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f830400a5d0 con 0x7f830c1596e0 2024-01-26T23:43:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.902+0000 7f831446e700 1 -- 172.21.15.46:0/141013998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f830c0b2db0 con 0x7f830c1596e0 2024-01-26T23:43:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.902+0000 7f83027fc700 1 -- 172.21.15.46:0/141013998 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830400a990 con 0x7f830c1596e0 2024-01-26T23:43:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.902+0000 7f831446e700 1 -- 172.21.15.46:0/141013998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f830c0b1d20 con 0x7f830c1596e0 2024-01-26T23:43:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.903+0000 7f831446e700 1 -- 172.21.15.46:0/141013998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830c0c5c70 con 0x7f830c1596e0 2024-01-26T23:43:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.904+0000 7f83027fc700 1 -- 172.21.15.46:0/141013998 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8304007150 con 0x7f830c1596e0 2024-01-26T23:43:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.904+0000 7f83027fc700 1 --2- 172.21.15.46:0/141013998 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f82f0066f30 0x7f82f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.904+0000 7f83027fc700 1 -- 172.21.15.46:0/141013998 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f8304084150 con 0x7f830c1596e0 2024-01-26T23:43:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.904+0000 7f831220a700 1 --2- 172.21.15.46:0/141013998 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f82f0066f30 0x7f82f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.905+0000 7f831220a700 1 --2- 172.21.15.46:0/141013998 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f82f0066f30 0x7f82f00693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f83080091e0 tx=0x7f8308016000 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:06.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:06.908+0000 7f83027fc700 1 -- 172.21.15.46:0/141013998 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f830c0c5c70 con 0x7f830c1596e0 2024-01-26T23:43:07.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.193+0000 7f831446e700 1 -- 172.21.15.46:0/141013998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f830c0b2ff0 con 0x7f830c1596e0 2024-01-26T23:43:07.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.194+0000 7f83027fc700 1 -- 172.21.15.46:0/141013998 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1111 (secure 0 0 0) 0x7f8304018470 con 0x7f830c1596e0 2024-01-26T23:43:07.195 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:43:07.196 INFO:teuthology.orchestra.run.smithi046.stdout:{"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":820,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":70,"num_osds":8,"num_up_osds":8,"osd_up_since":1706311777,"num_in_osds":8,"osd_in_since":1706311769,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":2476535808,"bytes_avail":770617577472,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":27,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"c","status":"up:active","gid":5865}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-01-26T23:32:09.376584+0000","services":{}},"progress_events":{}} 2024-01-26T23:43:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.196+0000 7f831446e700 1 -- 172.21.15.46:0/141013998 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f82f0066f30 msgr2=0x7f82f00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.196+0000 7f831446e700 1 --2- 172.21.15.46:0/141013998 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f82f0066f30 0x7f82f00693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f83080091e0 tx=0x7f8308016000 comp rx=0 tx=0).stop 2024-01-26T23:43:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.196+0000 7f831446e700 1 -- 172.21.15.46:0/141013998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f830c1596e0 msgr2=0x7f830c0c7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.196+0000 7f831446e700 1 --2- 172.21.15.46:0/141013998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f830c1596e0 0x7f830c0c7d70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f830400cb60 tx=0x7f83040018a0 comp rx=0 tx=0).stop 2024-01-26T23:43:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.196+0000 7f831446e700 1 -- 172.21.15.46:0/141013998 shutdown_connections 2024-01-26T23:43:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.196+0000 7f831446e700 1 --2- 172.21.15.46:0/141013998 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f82f0066f30 0x7f82f00693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.196+0000 7f831446e700 1 --2- 172.21.15.46:0/141013998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f830c1596e0 0x7f830c0c7d70 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.196+0000 7f831446e700 1 --2- 172.21.15.46:0/141013998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f830c0c4720 0x7f830c158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.196+0000 7f831446e700 1 --2- 172.21.15.46:0/141013998 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f830c0cb4c0 0x7f830c0c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.196+0000 7f831446e700 1 -- 172.21.15.46:0/141013998 >> 172.21.15.46:0/141013998 conn(0x7f830c097820 msgr2=0x7f830c0cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.196+0000 7f831446e700 1 -- 172.21.15.46:0/141013998 shutdown_connections 2024-01-26T23:43:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.196+0000 7f831446e700 1 -- 172.21.15.46:0/141013998 wait complete. 2024-01-26T23:43:07.211 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-26T23:43:07.211 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:43:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.341+0000 7ff27b352700 1 Processor -- start 2024-01-26T23:43:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.341+0000 7ff27b352700 1 -- start start 2024-01-26T23:43:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.342+0000 7ff27b352700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2740ccc20 0x7ff274068340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.342+0000 7ff27b352700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff2740ccff0 0x7ff27406eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.342+0000 7ff27b352700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff274158020 con 0x7ff2740b2590 2024-01-26T23:43:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.342+0000 7ff27b352700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff274158300 con 0x7ff2740ccff0 2024-01-26T23:43:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.342+0000 7ff27b352700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2741585e0 con 0x7ff2740ccc20 2024-01-26T23:43:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.343+0000 7ff2790ee700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2740ccc20 0x7ff274068340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.343+0000 7ff2788ed700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff2740ccff0 0x7ff27406eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.343+0000 7ff2790ee700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2740ccc20 0x7ff274068340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:46614/0 (socket says 172.21.15.46:46614) 2024-01-26T23:43:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.343+0000 7ff2788ed700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff2740ccff0 0x7ff27406eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:47220/0 (socket says 172.21.15.46:47220) 2024-01-26T23:43:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.343+0000 7ff2790ee700 1 -- 172.21.15.46:0/3821782249 learned_addr learned my addr 172.21.15.46:0/3821782249 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.343+0000 7ff2798ef700 1 --1- 172.21.15.46:0/3821782249 >> v1:172.21.15.106:6789/0 conn(0x7ff2740b2590 0x7ff27415b5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:46976/0 (socket says 172.21.15.46:46976) 2024-01-26T23:43:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.343+0000 7ff2790ee700 1 -- 172.21.15.46:0/3821782249 >> v2:172.21.15.106:3300/0 conn(0x7ff2740ccff0 msgr2=0x7ff27406eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.343+0000 7ff2790ee700 1 --2- 172.21.15.46:0/3821782249 >> v2:172.21.15.106:3300/0 conn(0x7ff2740ccff0 0x7ff27406eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.343+0000 7ff2790ee700 1 -- 172.21.15.46:0/3821782249 >> v1:172.21.15.106:6789/0 conn(0x7ff2740b2590 legacy=0x7ff27415b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.343+0000 7ff26bfff700 1 -- 172.21.15.46:0/3821782249 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572543107 0 0) 0x7ff274158020 con 0x7ff2740b2590 2024-01-26T23:43:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.343+0000 7ff2790ee700 1 -- 172.21.15.46:0/3821782249 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2741588c0 con 0x7ff2740ccc20 2024-01-26T23:43:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.343+0000 7ff2788ed700 1 --2- 172.21.15.46:0/3821782249 >> v2:172.21.15.106:3300/0 conn(0x7ff2740ccff0 0x7ff27406eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:43:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.343+0000 7ff2790ee700 1 --2- 172.21.15.46:0/3821782249 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2740ccc20 0x7ff274068340 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff2640099b0 tx=0x7ff264009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b124d4550d5546c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.344+0000 7ff26bfff700 1 -- 172.21.15.46:0/3821782249 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff264009ea0 con 0x7ff2740ccc20 2024-01-26T23:43:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.344+0000 7ff26bfff700 1 -- 172.21.15.46:0/3821782249 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26400b860 con 0x7ff2740ccc20 2024-01-26T23:43:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.344+0000 7ff26bfff700 1 -- 172.21.15.46:0/3821782249 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26400bc00 con 0x7ff2740ccc20 2024-01-26T23:43:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.344+0000 7ff27b352700 1 -- 172.21.15.46:0/3821782249 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2740ccc20 msgr2=0x7ff274068340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.344+0000 7ff27b352700 1 --2- 172.21.15.46:0/3821782249 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2740ccc20 0x7ff274068340 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff2640099b0 tx=0x7ff264009250 comp rx=0 tx=0).stop 2024-01-26T23:43:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.344+0000 7ff27b352700 1 -- 172.21.15.46:0/3821782249 shutdown_connections 2024-01-26T23:43:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.344+0000 7ff27b352700 1 --2- 172.21.15.46:0/3821782249 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2740ccc20 0x7ff274068340 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.344+0000 7ff27b352700 1 --2- 172.21.15.46:0/3821782249 >> v2:172.21.15.106:3300/0 conn(0x7ff2740ccff0 0x7ff27406eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.344+0000 7ff27b352700 1 -- 172.21.15.46:0/3821782249 >> 172.21.15.46:0/3821782249 conn(0x7ff27406b630 msgr2=0x7ff2740aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.344+0000 7ff27b352700 1 -- 172.21.15.46:0/3821782249 shutdown_connections 2024-01-26T23:43:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.344+0000 7ff27b352700 1 -- 172.21.15.46:0/3821782249 wait complete. 2024-01-26T23:43:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.345+0000 7ff27b352700 1 Processor -- start 2024-01-26T23:43:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.345+0000 7ff27b352700 1 -- start start 2024-01-26T23:43:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.346+0000 7ff27b352700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740b2590 0x7ff2740c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.346+0000 7ff27b352700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2740ccc20 0x7ff27414ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.346+0000 7ff27b352700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2740ccff0 0x7ff27414b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.346+0000 7ff27b352700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff274168830 con 0x7ff2740b2590 2024-01-26T23:43:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.346+0000 7ff27b352700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2740c6640 con 0x7ff2740ccff0 2024-01-26T23:43:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.346+0000 7ff27b352700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2740c6920 con 0x7ff2740ccc20 2024-01-26T23:43:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.347+0000 7ff2790ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740b2590 0x7ff2740c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.347+0000 7ff2790ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740b2590 0x7ff2740c85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50816/0 (socket says 172.21.15.46:50816) 2024-01-26T23:43:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.347+0000 7ff2790ee700 1 -- 172.21.15.46:0/1272346299 learned_addr learned my addr 172.21.15.46:0/1272346299 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.347+0000 7ff2798ef700 1 --2- 172.21.15.46:0/1272346299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2740ccff0 0x7ff27414b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.347+0000 7ff2788ed700 1 --2- 172.21.15.46:0/1272346299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2740ccc20 0x7ff27414ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.347+0000 7ff2790ee700 1 -- 172.21.15.46:0/1272346299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2740ccc20 msgr2=0x7ff27414ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.347+0000 7ff2790ee700 1 --2- 172.21.15.46:0/1272346299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2740ccc20 0x7ff27414ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.347+0000 7ff2790ee700 1 -- 172.21.15.46:0/1272346299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2740ccff0 msgr2=0x7ff27414b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.347+0000 7ff2790ee700 1 --2- 172.21.15.46:0/1272346299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2740ccff0 0x7ff27414b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.347+0000 7ff2790ee700 1 -- 172.21.15.46:0/1272346299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff270009420 con 0x7ff2740b2590 2024-01-26T23:43:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.347+0000 7ff2788ed700 1 --2- 172.21.15.46:0/1272346299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2740ccc20 0x7ff27414ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:43:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.347+0000 7ff2790ee700 1 --2- 172.21.15.46:0/1272346299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740b2590 0x7ff2740c85a0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff274158720 tx=0x7ff264017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.347+0000 7ff2697fa700 1 -- 172.21.15.46:0/1272346299 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26400bbe0 con 0x7ff2740b2590 2024-01-26T23:43:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.347+0000 7ff2697fa700 1 -- 172.21.15.46:0/1272346299 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff264011420 con 0x7ff2740b2590 2024-01-26T23:43:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.347+0000 7ff27b352700 1 -- 172.21.15.46:0/1272346299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2640094f0 con 0x7ff2740b2590 2024-01-26T23:43:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.347+0000 7ff2697fa700 1 -- 172.21.15.46:0/1272346299 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2640117e0 con 0x7ff2740b2590 2024-01-26T23:43:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.347+0000 7ff27b352700 1 -- 172.21.15.46:0/1272346299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2740c6ee0 con 0x7ff2740b2590 2024-01-26T23:43:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.348+0000 7ff27b352700 1 -- 172.21.15.46:0/1272346299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27404c710 con 0x7ff2740b2590 2024-01-26T23:43:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.349+0000 7ff2697fa700 1 -- 172.21.15.46:0/1272346299 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff264011bd0 con 0x7ff2740b2590 2024-01-26T23:43:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.350+0000 7ff2697fa700 1 --2- 172.21.15.46:0/1272346299 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff258066f30 0x7ff2580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.350+0000 7ff2697fa700 1 -- 172.21.15.46:0/1272346299 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7ff2640860c0 con 0x7ff2740b2590 2024-01-26T23:43:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.350+0000 7ff2788ed700 1 --2- 172.21.15.46:0/1272346299 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff258066f30 0x7ff2580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.350+0000 7ff2788ed700 1 --2- 172.21.15.46:0/1272346299 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff258066f30 0x7ff2580693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff2700093f0 tx=0x7ff270008040 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.354+0000 7ff2697fa700 1 -- 172.21.15.46:0/1272346299 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff264053080 con 0x7ff2740b2590 2024-01-26T23:43:07.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.550+0000 7ff27b352700 1 -- 172.21.15.46:0/1272346299 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff27406f0b0 con 0x7ff258066f30 2024-01-26T23:43:07.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.558+0000 7ff2697fa700 1 -- 172.21.15.46:0/1272346299 <== mgr.5267 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278586 (secure 0 0 0) 0x7ff27406f0b0 con 0x7ff258066f30 2024-01-26T23:43:07.563 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:43:07.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.562+0000 7ff27b352700 1 -- 172.21.15.46:0/1272346299 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff258066f30 msgr2=0x7ff2580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:07.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.562+0000 7ff27b352700 1 --2- 172.21.15.46:0/1272346299 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff258066f30 0x7ff2580693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff2700093f0 tx=0x7ff270008040 comp rx=0 tx=0).stop 2024-01-26T23:43:07.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.562+0000 7ff27b352700 1 -- 172.21.15.46:0/1272346299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740b2590 msgr2=0x7ff2740c85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:07.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.562+0000 7ff27b352700 1 --2- 172.21.15.46:0/1272346299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740b2590 0x7ff2740c85a0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff274158720 tx=0x7ff264017000 comp rx=0 tx=0).stop 2024-01-26T23:43:07.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.562+0000 7ff27b352700 1 -- 172.21.15.46:0/1272346299 shutdown_connections 2024-01-26T23:43:07.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.562+0000 7ff27b352700 1 --2- 172.21.15.46:0/1272346299 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff258066f30 0x7ff2580693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:07.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.562+0000 7ff27b352700 1 --2- 172.21.15.46:0/1272346299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740b2590 0x7ff2740c85a0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:07.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.562+0000 7ff27b352700 1 --2- 172.21.15.46:0/1272346299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2740ccff0 0x7ff27414b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:07.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.562+0000 7ff27b352700 1 --2- 172.21.15.46:0/1272346299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2740ccc20 0x7ff27414ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:07.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.562+0000 7ff27b352700 1 -- 172.21.15.46:0/1272346299 >> 172.21.15.46:0/1272346299 conn(0x7ff27406b630 msgr2=0x7ff2740cb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:07.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.562+0000 7ff27b352700 1 -- 172.21.15.46:0/1272346299 shutdown_connections 2024-01-26T23:43:07.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:07.562+0000 7ff27b352700 1 -- 172.21.15.46:0/1272346299 wait complete. 2024-01-26T23:43:07.566 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:43:07.577 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-26T23:43:05.642333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2418492,"kb_used_data":7420,"kb_used_omap":0,"kb_used_meta":2411008,"kb_avail":752556228,"statfs":{"total":773094113280,"available":770617577472,"internally_reserved":0,"allocated":7598080,"data_stored":4341407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2468872192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.946913"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.702730+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:55.750309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697701+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:42:55.750309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697807+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:15:45.000899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697169+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:00:27.828427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.705515+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:01:20.459485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.706547+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:48:06.519946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701638+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:07:07.891870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704532+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:03.697032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701642+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:24.351557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.706514+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:19.596229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.702053+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:34:08.288094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.696768+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:05.307132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704628+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:20:40.758787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.703147+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:22:30.811610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.702374+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:51:32.002538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.696777+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:08:26.147843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704560+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:41:09.634181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.699768+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:18:47.545463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701502+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:21:48.776990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701431+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:05:54.731798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697780+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:16:07.169692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.698213+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:36.728509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704561+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:38:03.176048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.699836+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:29:14.906700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.702155+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:29:51.950015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.701472+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:31:22.376213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704797+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:55:52.786999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.696764+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:48:06.519946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.696894+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:19.596229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.706535+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:05.307132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.697086+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:51:32.002538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:43:00.694334+0000","last_change":"2024-01-26T23:43:00.694334+0000","last_active":"2024-01-26T23:43:00.694334+0000","last_peered":"2024-01-26T23:43:00.694334+0000","last_clean":"2024-01-26T23:43:00.694334+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:43:00.694334+0000","last_undegraded":"2024-01-26T23:43:00.694334+0000","last_fullsized":"2024-01-26T23:43:00.694334+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:42:59.687004+0000","last_change":"2024-01-26T23:42:59.704965+0000","last_active":"2024-01-26T23:42:59.687004+0000","last_peered":"2024-01-26T23:42:59.687004+0000","last_clean":"2024-01-26T23:42:59.687004+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:42:59.687004+0000","last_undegraded":"2024-01-26T23:42:59.687004+0000","last_fullsized":"2024-01-26T23:42:59.687004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:00:58.220861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"61'139","reported_seq":283,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-01-26T23:42:59.110217+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:42:59.110217+0000","last_peered":"2024-01-26T23:42:59.110217+0000","last_clean":"2024-01-26T23:42:59.110217+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:42:59.110217+0000","last_undegraded":"2024-01-26T23:42:59.110217+0000","last_fullsized":"2024-01-26T23:42:59.110217+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575011,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306844,"kb_used_data":1364,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064996,"statfs":{"total":96636764160,"available":96322555904,"internally_reserved":0,"allocated":1396736,"data_stored":986263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59599999999999997,"15min":0.56499999999999995},"min":{"1min":0.51100000000000001,"5min":0.371,"15min":0.317},"max":{"1min":0.78300000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61899999999999999,"15min":0.57899999999999996},"min":{"1min":0.50600000000000001,"5min":0.44500000000000001,"15min":0.34699999999999998},"max":{"1min":0.80300000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.32800000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.61299999999999999,"15min":0.56799999999999995},"min":{"1min":0.45500000000000002,"5min":0.43099999999999999,"15min":0.36399999999999999},"max":{"1min":0.81299999999999994,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.61899999999999999,"15min":0.58299999999999996},"min":{"1min":0.52300000000000002,"5min":0.41099999999999998,"15min":0.36099999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.85199999999999998},"last":0.46899999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.59799999999999998,"15min":0.56999999999999995},"min":{"1min":0.51000000000000001,"5min":0.47699999999999998,"15min":0.35899999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.81699999999999995},"last":0.378},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.61199999999999999,"15min":0.59199999999999997},"min":{"1min":0.50600000000000001,"5min":0.41699999999999998,"15min":0.38400000000000001},"max":{"1min":0.77300000000000002,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.44}]},{"osd":3,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60999999999999999,"15min":0.58599999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.35399999999999998},"max":{"1min":0.76900000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.621,"15min":0.60399999999999998},"min":{"1min":0.53500000000000003,"5min":0.433,"15min":0.373},"max":{"1min":0.79200000000000004,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.48699999999999999}]},{"osd":4,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46500000000000002,"15min":0.46899999999999997},"min":{"1min":0.33300000000000002,"5min":0.31900000000000001,"15min":0.28999999999999998},"max":{"1min":0.65600000000000003,"5min":0.72699999999999998,"15min":0.90100000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.49099999999999999,"15min":0.498},"min":{"1min":0.38500000000000001,"5min":0.33100000000000002,"15min":0.314},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.73799999999999999},"last":0.41799999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.498,"15min":0.50600000000000001},"min":{"1min":0.35399999999999998,"5min":0.32000000000000001,"15min":0.30599999999999999},"max":{"1min":0.71299999999999997,"5min":0.745,"15min":0.91700000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.40799999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.96599999999999997},"last":0.53200000000000003}]},{"osd":6,"last update":"Fri Jan 26 23:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56699999999999995,"15min":0.55900000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.33100000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.93500000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.432,"5min":0.38100000000000001,"15min":0.35299999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.86899999999999999},"last":0.499}]}]},{"osd":6,"up_from":13,"seq":55834575011,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302168,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069672,"statfs":{"total":96636764160,"available":96327344128,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.58599999999999997,"15min":0.56100000000000005},"min":{"1min":0.30199999999999999,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":0.71999999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60299999999999998,"15min":0.58399999999999996},"min":{"1min":0.41399999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71699999999999997,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.58199999999999996}]},{"osd":1,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.57999999999999996,"15min":0.56899999999999995},"min":{"1min":0.41699999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.60699999999999998,"15min":0.59099999999999997},"min":{"1min":0.38100000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73499999999999999,"5min":0.874,"15min":0.874},"last":0.54900000000000004}]},{"osd":2,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.58999999999999997,"15min":0.57799999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.29599999999999999},"max":{"1min":0.73299999999999998,"5min":0.78100000000000003,"15min":0.85799999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59799999999999998,"15min":0.58299999999999996},"min":{"1min":0.45300000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76200000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.61099999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60499999999999998,"15min":0.59399999999999997},"min":{"1min":0.435,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.80700000000000005,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.62,"15min":0.61499999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39000000000000001},"max":{"1min":0.71699999999999997,"5min":0.82299999999999995,"15min":0.84499999999999997},"last":0.628}]},{"osd":4,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.46999999999999997,"15min":0.47399999999999998},"min":{"1min":0.33300000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.70399999999999996,"5min":0.77800000000000002,"15min":0.90200000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.48799999999999999,"15min":0.498},"min":{"1min":0.32000000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.67400000000000004,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.53300000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.47999999999999998,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.69499999999999995,"5min":0.76500000000000001,"15min":0.78200000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51800000000000002,"15min":0.53000000000000003},"min":{"1min":0.34399999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.77900000000000003,"5min":0.83499999999999996,"15min":0.96499999999999997},"last":0.46400000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52200000000000002,"15min":0.54000000000000004},"min":{"1min":0.38600000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.73599999999999999,"5min":0.85899999999999999,"15min":0.97199999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.308},"max":{"1min":0.70399999999999996,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56200000000000006}]}]},{"osd":1,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302168,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069672,"statfs":{"total":96636764160,"available":96327344128,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39500000000000002,"15min":0.40600000000000003},"min":{"1min":0.26500000000000001,"5min":0.26200000000000001,"15min":0.23200000000000001},"max":{"1min":0.57799999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.621},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.441},"min":{"1min":0.32500000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.58199999999999996,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.41599999999999998,"15min":0.40699999999999997},"min":{"1min":0.33700000000000002,"5min":0.30599999999999999,"15min":0.26200000000000001},"max":{"1min":0.61399999999999999,"5min":0.63100000000000001,"15min":0.65900000000000003},"last":0.499},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.38600000000000001,"5min":0.32600000000000001,"15min":0.28999999999999998},"max":{"1min":0.69499999999999995,"5min":0.71899999999999997,"15min":0.79600000000000004},"last":0.51300000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.505,"15min":0.505},"min":{"1min":0.42499999999999999,"5min":0.33800000000000002,"15min":0.32100000000000001},"max":{"1min":0.69099999999999995,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.81999999999999995},"last":0.436}]},{"osd":4,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.66700000000000004,"15min":0.64000000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.39400000000000002},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.65900000000000003,"15min":0.66700000000000004},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.433},"max":{"1min":0.91900000000000004,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.82199999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.67900000000000005,"15min":0.67600000000000005},"min":{"1min":0.49299999999999999,"5min":0.45000000000000001,"15min":0.42799999999999999},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.67400000000000004,"15min":0.66600000000000004},"min":{"1min":0.57699999999999996,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.92200000000000004,"5min":0.98599999999999999,"15min":1.018},"last":0.63200000000000001}]},{"osd":6,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.68899999999999995,"15min":0.68600000000000005},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.45700000000000002},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.1279999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.69199999999999995,"15min":0.69099999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.42599999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.78500000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.72099999999999997,"15min":0.71399999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.49399999999999999},"max":{"1min":0.94699999999999995,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.59199999999999997,"5min":0.5,"15min":0.438},"max":{"1min":1.048,"5min":1.048,"15min":1.3049999999999999},"last":0.80200000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302736,"kb_used_data":1352,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069104,"statfs":{"total":96636764160,"available":96326762496,"internally_reserved":0,"allocated":1384448,"data_stored":979774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40100000000000002,"15min":0.41299999999999998},"min":{"1min":0.27800000000000002,"5min":0.26400000000000001,"15min":0.214},"max":{"1min":0.52200000000000002,"5min":0.64100000000000001,"15min":1.284},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46899999999999997,"15min":0.45900000000000002},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.27300000000000002},"max":{"1min":0.72699999999999998,"5min":0.75800000000000001,"15min":1.024},"last":0.30499999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.433,"15min":0.443},"min":{"1min":0.318,"5min":0.28199999999999997,"15min":0.26700000000000002},"max":{"1min":0.59699999999999998,"5min":0.66100000000000003,"15min":1.2230000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.68100000000000005,"5min":0.68300000000000005,"15min":1.2430000000000001},"last":0.375}]},{"osd":3,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.47999999999999998,"15min":0.47599999999999998},"min":{"1min":0.38800000000000001,"5min":0.317,"15min":0.26800000000000002},"max":{"1min":0.626,"5min":0.69899999999999995,"15min":1.0980000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.498,"15min":0.5},"min":{"1min":0.39500000000000002,"5min":0.34000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69799999999999995,"5min":0.74099999999999999,"15min":1.1850000000000001},"last":0.41299999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.65200000000000002,"15min":0.65700000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.372},"max":{"1min":0.86799999999999999,"5min":0.93300000000000005,"15min":1.133},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66600000000000004,"15min":0.66900000000000004},"min":{"1min":0.52900000000000003,"5min":0.443,"15min":0.443},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":1.0169999999999999},"last":0.55400000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.66100000000000003,"15min":0.66900000000000004},"min":{"1min":0.46500000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":1.0880000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68200000000000005,"15min":0.69199999999999995},"min":{"1min":0.504,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93300000000000005,"5min":0.98699999999999999,"15min":1.169},"last":0.58799999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67300000000000004,"15min":0.69399999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.44800000000000001},"max":{"1min":0.88400000000000001,"5min":0.97299999999999998,"15min":1.071},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69399999999999995,"15min":0.70999999999999996},"min":{"1min":0.55200000000000005,"5min":0.502,"15min":0.48599999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.1160000000000001},"last":0.60399999999999998}]},{"osd":7,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.72099999999999997,"15min":0.73999999999999999},"min":{"1min":0.51300000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.73699999999999999,"15min":0.749},"min":{"1min":0.58999999999999997,"5min":0.433,"15min":0.433},"max":{"1min":0.98699999999999999,"5min":1.028,"15min":1.2010000000000001},"last":0.68899999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306264,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065576,"statfs":{"total":96636764160,"available":96323149824,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.503},"min":{"1min":0.28399999999999997,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":0.53200000000000003,"5min":0.56699999999999995,"15min":9.0540000000000003},"last":0.442},{"interface":"front","average":{"1min":0.443,"5min":0.42199999999999999,"15min":0.46200000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.24299999999999999},"max":{"1min":0.57599999999999996,"5min":0.64000000000000001,"15min":4.1970000000000001},"last":0.307}]},{"osd":1,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.438,"15min":0.42299999999999999},"min":{"1min":0.28499999999999998,"5min":0.27400000000000002,"15min":0.221},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.82199999999999995},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.502,"15min":0.64800000000000002},"min":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.316},"max":{"1min":0.65900000000000003,"5min":0.72499999999999998,"15min":12.157},"last":0.45700000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.502,"15min":0.48599999999999999},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33500000000000002},"max":{"1min":0.65500000000000003,"5min":0.74099999999999999,"15min":0.86499999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47899999999999998,"15min":0.57699999999999996},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.30399999999999999},"max":{"1min":0.63600000000000001,"5min":0.73199999999999998,"15min":9.0879999999999992},"last":0.38300000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.65100000000000002,"15min":0.749},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.88500000000000001,"5min":1.048,"15min":9.0150000000000006},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.64600000000000002,"15min":0.68400000000000005},"min":{"1min":0.55700000000000005,"5min":0.371,"15min":0.371},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":4.2480000000000002},"last":0.63500000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66400000000000003,"15min":0.69799999999999995},"min":{"1min":0.55500000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.85899999999999999,"5min":1.008,"15min":4.3200000000000003},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.69199999999999995,"15min":0.67800000000000005},"min":{"1min":0.52000000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.95199999999999996,"5min":1.085,"15min":1.085},"last":0.621}]},{"osd":6,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.69899999999999995,"15min":0.68600000000000005},"min":{"1min":0.46999999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.505},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.68899999999999995,"15min":0.78800000000000003},"min":{"1min":0.53500000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.94699999999999995,"5min":1.111,"15min":8.9700000000000006},"last":0.64700000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.70799999999999996,"15min":0.752},"min":{"1min":0.61099999999999999,"5min":0.54100000000000004,"15min":0.45800000000000002},"max":{"1min":0.93700000000000006,"5min":1.127,"15min":4.282},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.746,"15min":0.73499999999999999},"min":{"1min":0.52900000000000003,"5min":0.46100000000000002,"15min":0.42099999999999999},"max":{"1min":1.0329999999999999,"5min":1.139,"15min":1.139},"last":0.66100000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302168,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069672,"statfs":{"total":96636764160,"available":96327344128,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.23999999999999999},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44900000000000001,"15min":0.56100000000000005},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":11.968},"last":0.504}]},{"osd":1,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.45600000000000002,"15min":0.57299999999999995},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.633,"5min":1.633,"15min":12.050000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48299999999999998,"15min":0.48999999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.2150000000000001},"last":0.378}]},{"osd":2,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.48899999999999999,"15min":0.59399999999999997},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.28999999999999998},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":12.234},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.46999999999999997,"15min":0.58699999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":11.925000000000001},"last":0.39700000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.70199999999999996,"15min":0.78200000000000003},"min":{"1min":0.46999999999999997,"5min":0.45500000000000002,"15min":0.40200000000000002},"max":{"1min":3.306,"5min":3.306,"15min":12.099},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68000000000000005,"15min":0.66900000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.379},"max":{"1min":0.89700000000000002,"5min":1.0009999999999999,"15min":1.0649999999999999},"last":0.46899999999999997}]},{"osd":5,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67700000000000005,"15min":0.79300000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.593,"5min":1.593,"15min":12.199999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.69099999999999995,"15min":0.68000000000000005},"min":{"1min":0.45300000000000001,"5min":0.439,"15min":0.436},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.56999999999999995}]},{"osd":6,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.70099999999999996,"15min":0.80000000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.506,"5min":1.506,"15min":12.132999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.70799999999999996,"15min":0.70199999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.53700000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.74099999999999999,"15min":0.70499999999999996},"min":{"1min":0.48499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.238,"5min":3.238,"15min":3.238},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.77200000000000002,"15min":0.86499999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.48699999999999999},"max":{"1min":3.351,"5min":3.351,"15min":12.167},"last":0.48899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298072,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073768,"statfs":{"total":96636764160,"available":96331538432,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58299999999999996,"15min":0.57399999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69699999999999995,"5min":0.79800000000000004,"15min":1.0209999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60199999999999998,"15min":0.58199999999999996},"min":{"1min":0.41499999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70099999999999996,"5min":0.84099999999999997,"15min":1.161},"last":0.622}]},{"osd":1,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.60799999999999998,"15min":0.59699999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73699999999999999,"5min":0.88,"15min":1.069},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.39100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72699999999999998,"5min":0.78800000000000003,"15min":1.085},"last":0.48499999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59199999999999997,"15min":0.57399999999999995},"min":{"1min":0.38400000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.77000000000000002,"5min":0.85999999999999999,"15min":1.052},"last":0.435},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61399999999999999,"15min":0.61299999999999999},"min":{"1min":0.45700000000000002,"5min":0.40899999999999997,"15min":0.373},"max":{"1min":0.77600000000000002,"5min":0.80100000000000005,"15min":1.111},"last":0.39900000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61099999999999999,"15min":0.59999999999999998},"min":{"1min":0.41699999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.71999999999999997,"5min":0.78300000000000003,"15min":1.1299999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.45200000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.69699999999999995,"5min":0.88800000000000001,"15min":1.1419999999999999},"last":0.60399999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47599999999999998,"15min":0.47899999999999998},"min":{"1min":0.39300000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.64400000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.496,"15min":0.51100000000000001},"min":{"1min":0.33400000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.69799999999999995,"5min":0.76700000000000002,"15min":0.77800000000000002},"last":0.66800000000000004}]},{"osd":6,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.38300000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53100000000000003,"15min":0.54100000000000004},"min":{"1min":0.35499999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.67200000000000004,"5min":0.78200000000000003,"15min":0.95699999999999996},"last":0.65700000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.42799999999999999,"5min":0.375,"15min":0.35799999999999998},"max":{"1min":0.745,"5min":0.75900000000000001,"15min":0.76300000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.501,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.74099999999999999,"5min":0.78400000000000003,"15min":0.879},"last":0.64500000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298072,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073768,"statfs":{"total":96636764160,"available":96331538432,"internally_reserved":0,"allocated":802816,"data_stored":395895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.57699999999999996,"15min":0.55800000000000005},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.26000000000000001},"max":{"1min":2.0289999999999999,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57299999999999995,"15min":0.57599999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.498}]},{"osd":1,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61399999999999999,"15min":0.60899999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.3319999999999999,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63700000000000001,"15min":0.61699999999999999},"min":{"1min":0.435,"5min":0.41799999999999998,"15min":0.379},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.59599999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55500000000000005,"15min":0.56399999999999995},"min":{"1min":0.38200000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72599999999999998,"5min":0.83699999999999997,"15min":0.83799999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.60299999999999998,"15min":0.64700000000000002},"min":{"1min":0.39400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":6.4589999999999996},"last":0.53400000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57499999999999996,"15min":0.57399999999999995},"min":{"1min":0.35999999999999999,"5min":0.35799999999999998,"15min":0.314},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.88400000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.59899999999999998,"15min":0.64700000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":6.0629999999999997},"last":0.51700000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.26300000000000001},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48999999999999999,"15min":0.497},"min":{"1min":0.34499999999999997,"5min":0.315,"15min":0.28100000000000003},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.621}]},{"osd":6,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.48999999999999999,"15min":0.48499999999999999},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.27000000000000002},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52600000000000002,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.29299999999999998},"max":{"1min":1.99,"5min":1.99,"15min":1.99},"last":0.63400000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.56000000000000005,"15min":0.54900000000000004},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.5270000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55700000000000005,"15min":0.57299999999999995},"min":{"1min":0.36899999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.624,"5min":0.82199999999999995,"15min":0.83099999999999996},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:43:10.582 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:43:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.724+0000 7f36ddf96700 1 Processor -- start 2024-01-26T23:43:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.724+0000 7f36ddf96700 1 -- start start 2024-01-26T23:43:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.725+0000 7f36ddf96700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36d80b2530 0x7f36d815a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.725+0000 7f36ddf96700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36d815da10 0x7f36d815de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.725+0000 7f36ddf96700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d80a9b90 con 0x7f36d80b2160 2024-01-26T23:43:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.725+0000 7f36ddf96700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d80aa210 con 0x7f36d80b2530 2024-01-26T23:43:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.725+0000 7f36ddf96700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d815d580 con 0x7f36d815da10 2024-01-26T23:43:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.725+0000 7f36d6ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36d80b2530 0x7f36d815a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.725+0000 7f36d6ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36d80b2530 0x7f36d815a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46820/0 (socket says 172.21.15.46:46820) 2024-01-26T23:43:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.725+0000 7f36d6ffd700 1 -- 172.21.15.46:0/4212368695 learned_addr learned my addr 172.21.15.46:0/4212368695 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.725+0000 7f36d7fff700 1 --2- 172.21.15.46:0/4212368695 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36d815da10 0x7f36d815de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.725+0000 7f36d6ffd700 1 -- 172.21.15.46:0/4212368695 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36d815da10 msgr2=0x7f36d815de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.725+0000 7f36d6ffd700 1 --2- 172.21.15.46:0/4212368695 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36d815da10 0x7f36d815de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.725+0000 7f36d6ffd700 1 -- 172.21.15.46:0/4212368695 >> v1:172.21.15.106:6789/0 conn(0x7f36d80b2160 legacy=0x7f36d80abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.725+0000 7f36d6ffd700 1 -- 172.21.15.46:0/4212368695 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d80aa460 con 0x7f36d80b2530 2024-01-26T23:43:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.725+0000 7f36d6ffd700 1 --2- 172.21.15.46:0/4212368695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d80b2530 0x7f36d815a7a0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0099b0 tx=0x7f36cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195b7a74a41ba07c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.726+0000 7f36d67fc700 1 -- 172.21.15.46:0/4212368695 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc00b860 con 0x7f36d80b2530 2024-01-26T23:43:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.726+0000 7f36d67fc700 1 -- 172.21.15.46:0/4212368695 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36cc00b9c0 con 0x7f36d80b2530 2024-01-26T23:43:10.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.726+0000 7f36d67fc700 1 -- 172.21.15.46:0/4212368695 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc00bd80 con 0x7f36d80b2530 2024-01-26T23:43:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.726+0000 7f36ddf96700 1 -- 172.21.15.46:0/4212368695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d80b2530 msgr2=0x7f36d815a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.726+0000 7f36ddf96700 1 --2- 172.21.15.46:0/4212368695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d80b2530 0x7f36d815a7a0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0099b0 tx=0x7f36cc009250 comp rx=0 tx=0).stop 2024-01-26T23:43:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.726+0000 7f36ddf96700 1 -- 172.21.15.46:0/4212368695 shutdown_connections 2024-01-26T23:43:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.726+0000 7f36ddf96700 1 --2- 172.21.15.46:0/4212368695 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36d815da10 0x7f36d815de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.726+0000 7f36ddf96700 1 --2- 172.21.15.46:0/4212368695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d80b2530 0x7f36d815a7a0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.726+0000 7f36ddf96700 1 -- 172.21.15.46:0/4212368695 >> 172.21.15.46:0/4212368695 conn(0x7f36d80ccbc0 msgr2=0x7f36d80ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.726+0000 7f36ddf96700 1 -- 172.21.15.46:0/4212368695 shutdown_connections 2024-01-26T23:43:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.726+0000 7f36ddf96700 1 -- 172.21.15.46:0/4212368695 wait complete. 2024-01-26T23:43:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.728+0000 7f36ddf96700 1 Processor -- start 2024-01-26T23:43:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.728+0000 7f36ddf96700 1 -- start start 2024-01-26T23:43:10.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.728+0000 7f36ddf96700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d80b2160 0x7f36d814a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:10.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.728+0000 7f36ddf96700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36d80b2530 0x7f36d80ac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:10.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.728+0000 7f36ddf96700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36d815da10 0x7f36d80afa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:10.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.728+0000 7f36ddf96700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d80adb40 con 0x7f36d80b2160 2024-01-26T23:43:10.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.728+0000 7f36ddf96700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d814ad80 con 0x7f36d80b2530 2024-01-26T23:43:10.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.728+0000 7f36ddf96700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d80acf30 con 0x7f36d815da10 2024-01-26T23:43:10.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.729+0000 7f36d77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d80b2160 0x7f36d814a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:10.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.729+0000 7f36d77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d80b2160 0x7f36d814a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46832/0 (socket says 172.21.15.46:46832) 2024-01-26T23:43:10.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.729+0000 7f36d77fe700 1 -- 172.21.15.46:0/1932867346 learned_addr learned my addr 172.21.15.46:0/1932867346 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:10.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.729+0000 7f36d6ffd700 1 --2- 172.21.15.46:0/1932867346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36d80b2530 0x7f36d80ac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:10.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.729+0000 7f36d7fff700 1 --2- 172.21.15.46:0/1932867346 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36d815da10 0x7f36d80afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:10.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.729+0000 7f36d77fe700 1 -- 172.21.15.46:0/1932867346 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36d815da10 msgr2=0x7f36d80afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:10.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.729+0000 7f36d77fe700 1 --2- 172.21.15.46:0/1932867346 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36d815da10 0x7f36d80afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:10.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.729+0000 7f36d77fe700 1 -- 172.21.15.46:0/1932867346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36d80b2530 msgr2=0x7f36d80ac6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:10.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.729+0000 7f36d77fe700 1 --2- 172.21.15.46:0/1932867346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36d80b2530 0x7f36d80ac6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:10.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.729+0000 7f36d77fe700 1 -- 172.21.15.46:0/1932867346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36cc0094f0 con 0x7f36d80b2160 2024-01-26T23:43:10.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.729+0000 7f36d7fff700 1 --2- 172.21.15.46:0/1932867346 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36d815da10 0x7f36d80afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:43:10.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.729+0000 7f36d77fe700 1 --2- 172.21.15.46:0/1932867346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d80b2160 0x7f36d814a840 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f36c800c590 tx=0x7f36c800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:10.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.729+0000 7f36bf7fe700 1 -- 172.21.15.46:0/1932867346 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c8011590 con 0x7f36d80b2160 2024-01-26T23:43:10.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.729+0000 7f36bf7fe700 1 -- 172.21.15.46:0/1932867346 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c8011710 con 0x7f36d80b2160 2024-01-26T23:43:10.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.729+0000 7f36ddf96700 1 -- 172.21.15.46:0/1932867346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d80ad210 con 0x7f36d80b2160 2024-01-26T23:43:10.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.729+0000 7f36bf7fe700 1 -- 172.21.15.46:0/1932867346 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c8011ad0 con 0x7f36d80b2160 2024-01-26T23:43:10.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.729+0000 7f36ddf96700 1 -- 172.21.15.46:0/1932867346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d80ad7d0 con 0x7f36d80b2160 2024-01-26T23:43:10.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.730+0000 7f36ddf96700 1 -- 172.21.15.46:0/1932867346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d804c710 con 0x7f36d80b2160 2024-01-26T23:43:10.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.731+0000 7f36bf7fe700 1 -- 172.21.15.46:0/1932867346 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f36c801b3b0 con 0x7f36d80b2160 2024-01-26T23:43:10.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.731+0000 7f36bf7fe700 1 --2- 172.21.15.46:0/1932867346 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36c4066f30 0x7f36c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:10.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.731+0000 7f36bf7fe700 1 -- 172.21.15.46:0/1932867346 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f36c8083dc0 con 0x7f36d80b2160 2024-01-26T23:43:10.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.732+0000 7f36d6ffd700 1 --2- 172.21.15.46:0/1932867346 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36c4066f30 0x7f36c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:10.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.732+0000 7f36d6ffd700 1 --2- 172.21.15.46:0/1932867346 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36c4066f30 0x7f36c40693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0099b0 tx=0x7f36cc017000 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:10.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.736+0000 7f36bf7fe700 1 -- 172.21.15.46:0/1932867346 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f36c8050ce0 con 0x7f36d80b2160 2024-01-26T23:43:10.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.935+0000 7f36ddf96700 1 -- 172.21.15.46:0/1932867346 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f36d814cde0 con 0x7f36c4066f30 2024-01-26T23:43:10.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.944+0000 7f36bf7fe700 1 -- 172.21.15.46:0/1932867346 <== mgr.5267 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289339 (secure 0 0 0) 0x7f36d814cde0 con 0x7f36c4066f30 2024-01-26T23:43:10.949 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:43:10.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.948+0000 7f36ddf96700 1 -- 172.21.15.46:0/1932867346 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36c4066f30 msgr2=0x7f36c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:10.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.948+0000 7f36ddf96700 1 --2- 172.21.15.46:0/1932867346 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36c4066f30 0x7f36c40693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0099b0 tx=0x7f36cc017000 comp rx=0 tx=0).stop 2024-01-26T23:43:10.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.948+0000 7f36ddf96700 1 -- 172.21.15.46:0/1932867346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d80b2160 msgr2=0x7f36d814a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:10.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.948+0000 7f36ddf96700 1 --2- 172.21.15.46:0/1932867346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d80b2160 0x7f36d814a840 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f36c800c590 tx=0x7f36c800c8a0 comp rx=0 tx=0).stop 2024-01-26T23:43:10.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.948+0000 7f36ddf96700 1 -- 172.21.15.46:0/1932867346 shutdown_connections 2024-01-26T23:43:10.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.948+0000 7f36ddf96700 1 --2- 172.21.15.46:0/1932867346 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36c4066f30 0x7f36c40693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:10.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.948+0000 7f36ddf96700 1 --2- 172.21.15.46:0/1932867346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36d80b2160 0x7f36d814a840 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:10.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.948+0000 7f36ddf96700 1 --2- 172.21.15.46:0/1932867346 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36d815da10 0x7f36d80afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:10.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.948+0000 7f36ddf96700 1 --2- 172.21.15.46:0/1932867346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36d80b2530 0x7f36d80ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:10.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.948+0000 7f36ddf96700 1 -- 172.21.15.46:0/1932867346 >> 172.21.15.46:0/1932867346 conn(0x7f36d80ccbc0 msgr2=0x7f36d814b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:10.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.948+0000 7f36ddf96700 1 -- 172.21.15.46:0/1932867346 shutdown_connections 2024-01-26T23:43:10.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:10.948+0000 7f36ddf96700 1 -- 172.21.15.46:0/1932867346 wait complete. 2024-01-26T23:43:10.952 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:43:10.963 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-26T23:43:09.643433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":176,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2422952,"kb_used_data":7784,"kb_used_omap":0,"kb_used_meta":2415104,"kb_avail":752551768,"statfs":{"total":773094113280,"available":770613010432,"internally_reserved":0,"allocated":7970816,"data_stored":4523436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2473066496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.040230"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.623496+0000","last_change":"2024-01-26T23:43:01.614122+0000","last_active":"2024-01-26T23:43:01.623496+0000","last_peered":"2024-01-26T23:43:01.623496+0000","last_clean":"2024-01-26T23:43:01.623496+0000","last_became_active":"2024-01-26T23:43:01.613826+0000","last_became_peered":"2024-01-26T23:43:01.613826+0000","last_unstale":"2024-01-26T23:43:01.623496+0000","last_undegraded":"2024-01-26T23:43:01.623496+0000","last_fullsized":"2024-01-26T23:43:01.623496+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:34:40.743435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.624892+0000","last_change":"2024-01-26T23:43:01.621478+0000","last_active":"2024-01-26T23:43:01.624892+0000","last_peered":"2024-01-26T23:43:01.624892+0000","last_clean":"2024-01-26T23:43:01.624892+0000","last_became_active":"2024-01-26T23:43:01.621172+0000","last_became_peered":"2024-01-26T23:43:01.621172+0000","last_unstale":"2024-01-26T23:43:01.624892+0000","last_undegraded":"2024-01-26T23:43:01.624892+0000","last_fullsized":"2024-01-26T23:43:01.624892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:07.433058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.622916+0000","last_change":"2024-01-26T23:43:01.619607+0000","last_active":"2024-01-26T23:43:01.622916+0000","last_peered":"2024-01-26T23:43:01.622916+0000","last_clean":"2024-01-26T23:43:01.622916+0000","last_became_active":"2024-01-26T23:43:01.619292+0000","last_became_peered":"2024-01-26T23:43:01.619292+0000","last_unstale":"2024-01-26T23:43:01.622916+0000","last_undegraded":"2024-01-26T23:43:01.622916+0000","last_fullsized":"2024-01-26T23:43:01.622916+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:00:43.150540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.630486+0000","last_change":"2024-01-26T23:43:01.626247+0000","last_active":"2024-01-26T23:43:01.630486+0000","last_peered":"2024-01-26T23:43:01.630486+0000","last_clean":"2024-01-26T23:43:01.630486+0000","last_became_active":"2024-01-26T23:43:01.625989+0000","last_became_peered":"2024-01-26T23:43:01.625989+0000","last_unstale":"2024-01-26T23:43:01.630486+0000","last_undegraded":"2024-01-26T23:43:01.630486+0000","last_fullsized":"2024-01-26T23:43:01.630486+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:20:41.766117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.627745+0000","last_change":"2024-01-26T23:43:01.620338+0000","last_active":"2024-01-26T23:43:01.627745+0000","last_peered":"2024-01-26T23:43:01.627745+0000","last_clean":"2024-01-26T23:43:01.627745+0000","last_became_active":"2024-01-26T23:43:01.620075+0000","last_became_peered":"2024-01-26T23:43:01.620075+0000","last_unstale":"2024-01-26T23:43:01.627745+0000","last_undegraded":"2024-01-26T23:43:01.627745+0000","last_fullsized":"2024-01-26T23:43:01.627745+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:55:53.794329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.623075+0000","last_change":"2024-01-26T23:43:01.613354+0000","last_active":"2024-01-26T23:43:01.623075+0000","last_peered":"2024-01-26T23:43:01.623075+0000","last_clean":"2024-01-26T23:43:01.623075+0000","last_became_active":"2024-01-26T23:43:01.613056+0000","last_became_peered":"2024-01-26T23:43:01.613056+0000","last_unstale":"2024-01-26T23:43:01.623075+0000","last_undegraded":"2024-01-26T23:43:01.623075+0000","last_fullsized":"2024-01-26T23:43:01.623075+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:04.704363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.624693+0000","last_change":"2024-01-26T23:43:01.623926+0000","last_active":"2024-01-26T23:43:01.624693+0000","last_peered":"2024-01-26T23:43:01.624693+0000","last_clean":"2024-01-26T23:43:01.624693+0000","last_became_active":"2024-01-26T23:43:01.623623+0000","last_became_peered":"2024-01-26T23:43:01.623623+0000","last_unstale":"2024-01-26T23:43:01.624693+0000","last_undegraded":"2024-01-26T23:43:01.624693+0000","last_fullsized":"2024-01-26T23:43:01.624693+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:51:49.236916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.625532+0000","last_change":"2024-01-26T23:43:01.619534+0000","last_active":"2024-01-26T23:43:01.625532+0000","last_peered":"2024-01-26T23:43:01.625532+0000","last_clean":"2024-01-26T23:43:01.625532+0000","last_became_active":"2024-01-26T23:43:01.619182+0000","last_became_peered":"2024-01-26T23:43:01.619182+0000","last_unstale":"2024-01-26T23:43:01.625532+0000","last_undegraded":"2024-01-26T23:43:01.625532+0000","last_fullsized":"2024-01-26T23:43:01.625532+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:43:53.575425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.628990+0000","last_change":"2024-01-26T23:43:01.628263+0000","last_active":"2024-01-26T23:43:01.628990+0000","last_peered":"2024-01-26T23:43:01.628990+0000","last_clean":"2024-01-26T23:43:01.628990+0000","last_became_active":"2024-01-26T23:43:01.626271+0000","last_became_peered":"2024-01-26T23:43:01.626271+0000","last_unstale":"2024-01-26T23:43:01.628990+0000","last_undegraded":"2024-01-26T23:43:01.628990+0000","last_fullsized":"2024-01-26T23:43:01.628990+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:56:28.246273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.627969+0000","last_change":"2024-01-26T23:43:01.625799+0000","last_active":"2024-01-26T23:43:01.627969+0000","last_peered":"2024-01-26T23:43:01.627969+0000","last_clean":"2024-01-26T23:43:01.627969+0000","last_became_active":"2024-01-26T23:43:01.625513+0000","last_became_peered":"2024-01-26T23:43:01.625513+0000","last_unstale":"2024-01-26T23:43:01.627969+0000","last_undegraded":"2024-01-26T23:43:01.627969+0000","last_fullsized":"2024-01-26T23:43:01.627969+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:32:56.856801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.623186+0000","last_change":"2024-01-26T23:43:01.615006+0000","last_active":"2024-01-26T23:43:01.623186+0000","last_peered":"2024-01-26T23:43:01.623186+0000","last_clean":"2024-01-26T23:43:01.623186+0000","last_became_active":"2024-01-26T23:43:01.614734+0000","last_became_peered":"2024-01-26T23:43:01.614734+0000","last_unstale":"2024-01-26T23:43:01.623186+0000","last_undegraded":"2024-01-26T23:43:01.623186+0000","last_fullsized":"2024-01-26T23:43:01.623186+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:03:10.320680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.627744+0000","last_change":"2024-01-26T23:43:01.625488+0000","last_active":"2024-01-26T23:43:01.627744+0000","last_peered":"2024-01-26T23:43:01.627744+0000","last_clean":"2024-01-26T23:43:01.627744+0000","last_became_active":"2024-01-26T23:43:01.625130+0000","last_became_peered":"2024-01-26T23:43:01.625130+0000","last_unstale":"2024-01-26T23:43:01.627744+0000","last_undegraded":"2024-01-26T23:43:01.627744+0000","last_fullsized":"2024-01-26T23:43:01.627744+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:25:03.496343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.625017+0000","last_change":"2024-01-26T23:43:01.621471+0000","last_active":"2024-01-26T23:43:01.625017+0000","last_peered":"2024-01-26T23:43:01.625017+0000","last_clean":"2024-01-26T23:43:01.625017+0000","last_became_active":"2024-01-26T23:43:01.621112+0000","last_became_peered":"2024-01-26T23:43:01.621112+0000","last_unstale":"2024-01-26T23:43:01.625017+0000","last_undegraded":"2024-01-26T23:43:01.625017+0000","last_fullsized":"2024-01-26T23:43:01.625017+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:38:04.183378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.622886+0000","last_change":"2024-01-26T23:43:01.616818+0000","last_active":"2024-01-26T23:43:01.622886+0000","last_peered":"2024-01-26T23:43:01.622886+0000","last_clean":"2024-01-26T23:43:01.622886+0000","last_became_active":"2024-01-26T23:43:01.616156+0000","last_became_peered":"2024-01-26T23:43:01.616156+0000","last_unstale":"2024-01-26T23:43:01.622886+0000","last_undegraded":"2024-01-26T23:43:01.622886+0000","last_fullsized":"2024-01-26T23:43:01.622886+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:12.130922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.622940+0000","last_change":"2024-01-26T23:43:01.618900+0000","last_active":"2024-01-26T23:43:01.622940+0000","last_peered":"2024-01-26T23:43:01.622940+0000","last_clean":"2024-01-26T23:43:01.622940+0000","last_became_active":"2024-01-26T23:43:01.618658+0000","last_became_peered":"2024-01-26T23:43:01.618658+0000","last_unstale":"2024-01-26T23:43:01.622940+0000","last_undegraded":"2024-01-26T23:43:01.622940+0000","last_fullsized":"2024-01-26T23:43:01.622940+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:04.704363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.627971+0000","last_change":"2024-01-26T23:43:01.626330+0000","last_active":"2024-01-26T23:43:01.627971+0000","last_peered":"2024-01-26T23:43:01.627971+0000","last_clean":"2024-01-26T23:43:01.627971+0000","last_became_active":"2024-01-26T23:43:01.626045+0000","last_became_peered":"2024-01-26T23:43:01.626045+0000","last_unstale":"2024-01-26T23:43:01.627971+0000","last_undegraded":"2024-01-26T23:43:01.627971+0000","last_fullsized":"2024-01-26T23:43:01.627971+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:39:44.562829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.622594+0000","last_change":"2024-01-26T23:43:01.615967+0000","last_active":"2024-01-26T23:43:01.622594+0000","last_peered":"2024-01-26T23:43:01.622594+0000","last_clean":"2024-01-26T23:43:01.622594+0000","last_became_active":"2024-01-26T23:43:01.615395+0000","last_became_peered":"2024-01-26T23:43:01.615395+0000","last_unstale":"2024-01-26T23:43:01.622594+0000","last_undegraded":"2024-01-26T23:43:01.622594+0000","last_fullsized":"2024-01-26T23:43:01.622594+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:42:22.378078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.624914+0000","last_change":"2024-01-26T23:43:01.621486+0000","last_active":"2024-01-26T23:43:01.624914+0000","last_peered":"2024-01-26T23:43:01.624914+0000","last_clean":"2024-01-26T23:43:01.624914+0000","last_became_active":"2024-01-26T23:43:01.621208+0000","last_became_peered":"2024-01-26T23:43:01.621208+0000","last_unstale":"2024-01-26T23:43:01.624914+0000","last_undegraded":"2024-01-26T23:43:01.624914+0000","last_fullsized":"2024-01-26T23:43:01.624914+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:55:53.794329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.629403+0000","last_change":"2024-01-26T23:43:01.627518+0000","last_active":"2024-01-26T23:43:01.629403+0000","last_peered":"2024-01-26T23:43:01.629403+0000","last_clean":"2024-01-26T23:43:01.629403+0000","last_became_active":"2024-01-26T23:43:01.627185+0000","last_became_peered":"2024-01-26T23:43:01.627185+0000","last_unstale":"2024-01-26T23:43:01.629403+0000","last_undegraded":"2024-01-26T23:43:01.629403+0000","last_fullsized":"2024-01-26T23:43:01.629403+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:29:29.278084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.625553+0000","last_change":"2024-01-26T23:43:01.620331+0000","last_active":"2024-01-26T23:43:01.625553+0000","last_peered":"2024-01-26T23:43:01.625553+0000","last_clean":"2024-01-26T23:43:01.625553+0000","last_became_active":"2024-01-26T23:43:01.620046+0000","last_became_peered":"2024-01-26T23:43:01.620046+0000","last_unstale":"2024-01-26T23:43:01.625553+0000","last_undegraded":"2024-01-26T23:43:01.625553+0000","last_fullsized":"2024-01-26T23:43:01.625553+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:41:14.525717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.622629+0000","last_change":"2024-01-26T23:43:01.617135+0000","last_active":"2024-01-26T23:43:01.622629+0000","last_peered":"2024-01-26T23:43:01.622629+0000","last_clean":"2024-01-26T23:43:01.622629+0000","last_became_active":"2024-01-26T23:43:01.616856+0000","last_became_peered":"2024-01-26T23:43:01.616856+0000","last_unstale":"2024-01-26T23:43:01.622629+0000","last_undegraded":"2024-01-26T23:43:01.622629+0000","last_fullsized":"2024-01-26T23:43:01.622629+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:39:44.562829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.621037+0000","last_change":"2024-01-26T23:43:01.618480+0000","last_active":"2024-01-26T23:43:01.621037+0000","last_peered":"2024-01-26T23:43:01.621037+0000","last_clean":"2024-01-26T23:43:01.621037+0000","last_became_active":"2024-01-26T23:43:01.617565+0000","last_became_peered":"2024-01-26T23:43:01.617565+0000","last_unstale":"2024-01-26T23:43:01.621037+0000","last_undegraded":"2024-01-26T23:43:01.621037+0000","last_fullsized":"2024-01-26T23:43:01.621037+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:03:58.836518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.622080+0000","last_change":"2024-01-26T23:43:01.614274+0000","last_active":"2024-01-26T23:43:01.622080+0000","last_peered":"2024-01-26T23:43:01.622080+0000","last_clean":"2024-01-26T23:43:01.622080+0000","last_became_active":"2024-01-26T23:43:01.613979+0000","last_became_peered":"2024-01-26T23:43:01.613979+0000","last_unstale":"2024-01-26T23:43:01.622080+0000","last_undegraded":"2024-01-26T23:43:01.622080+0000","last_fullsized":"2024-01-26T23:43:01.622080+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:36:00.989188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.622931+0000","last_change":"2024-01-26T23:43:01.617306+0000","last_active":"2024-01-26T23:43:01.622931+0000","last_peered":"2024-01-26T23:43:01.622931+0000","last_clean":"2024-01-26T23:43:01.622931+0000","last_became_active":"2024-01-26T23:43:01.617002+0000","last_became_peered":"2024-01-26T23:43:01.617002+0000","last_unstale":"2024-01-26T23:43:01.622931+0000","last_undegraded":"2024-01-26T23:43:01.622931+0000","last_fullsized":"2024-01-26T23:43:01.622931+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:41:14.932930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.630333+0000","last_change":"2024-01-26T23:43:01.629796+0000","last_active":"2024-01-26T23:43:01.630333+0000","last_peered":"2024-01-26T23:43:01.630333+0000","last_clean":"2024-01-26T23:43:01.630333+0000","last_became_active":"2024-01-26T23:43:01.629473+0000","last_became_peered":"2024-01-26T23:43:01.629473+0000","last_unstale":"2024-01-26T23:43:01.630333+0000","last_undegraded":"2024-01-26T23:43:01.630333+0000","last_fullsized":"2024-01-26T23:43:01.630333+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:07:08.954320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.627988+0000","last_change":"2024-01-26T23:43:01.619120+0000","last_active":"2024-01-26T23:43:01.627988+0000","last_peered":"2024-01-26T23:43:01.627988+0000","last_clean":"2024-01-26T23:43:01.627988+0000","last_became_active":"2024-01-26T23:43:01.617941+0000","last_became_peered":"2024-01-26T23:43:01.617941+0000","last_unstale":"2024-01-26T23:43:01.627988+0000","last_undegraded":"2024-01-26T23:43:01.627988+0000","last_fullsized":"2024-01-26T23:43:01.627988+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:29:58.419143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.625087+0000","last_change":"2024-01-26T23:43:01.621772+0000","last_active":"2024-01-26T23:43:01.625087+0000","last_peered":"2024-01-26T23:43:01.625087+0000","last_clean":"2024-01-26T23:43:01.625087+0000","last_became_active":"2024-01-26T23:43:01.621451+0000","last_became_peered":"2024-01-26T23:43:01.621451+0000","last_unstale":"2024-01-26T23:43:01.625087+0000","last_undegraded":"2024-01-26T23:43:01.625087+0000","last_fullsized":"2024-01-26T23:43:01.625087+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:00:59.183013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.621932+0000","last_change":"2024-01-26T23:43:01.617976+0000","last_active":"2024-01-26T23:43:01.621932+0000","last_peered":"2024-01-26T23:43:01.621932+0000","last_clean":"2024-01-26T23:43:01.621932+0000","last_became_active":"2024-01-26T23:43:01.617713+0000","last_became_peered":"2024-01-26T23:43:01.617713+0000","last_unstale":"2024-01-26T23:43:01.621932+0000","last_undegraded":"2024-01-26T23:43:01.621932+0000","last_fullsized":"2024-01-26T23:43:01.621932+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:51:47.149162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.622733+0000","last_change":"2024-01-26T23:43:01.615958+0000","last_active":"2024-01-26T23:43:01.622733+0000","last_peered":"2024-01-26T23:43:01.622733+0000","last_clean":"2024-01-26T23:43:01.622733+0000","last_became_active":"2024-01-26T23:43:01.615603+0000","last_became_peered":"2024-01-26T23:43:01.615603+0000","last_unstale":"2024-01-26T23:43:01.622733+0000","last_undegraded":"2024-01-26T23:43:01.622733+0000","last_fullsized":"2024-01-26T23:43:01.622733+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:17:05.392219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.627578+0000","last_change":"2024-01-26T23:43:01.623807+0000","last_active":"2024-01-26T23:43:01.627578+0000","last_peered":"2024-01-26T23:43:01.627578+0000","last_clean":"2024-01-26T23:43:01.627578+0000","last_became_active":"2024-01-26T23:43:01.623568+0000","last_became_peered":"2024-01-26T23:43:01.623568+0000","last_unstale":"2024-01-26T23:43:01.627578+0000","last_undegraded":"2024-01-26T23:43:01.627578+0000","last_fullsized":"2024-01-26T23:43:01.627578+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:38:04.183378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.622974+0000","last_change":"2024-01-26T23:43:01.617900+0000","last_active":"2024-01-26T23:43:01.622974+0000","last_peered":"2024-01-26T23:43:01.622974+0000","last_clean":"2024-01-26T23:43:01.622974+0000","last_became_active":"2024-01-26T23:43:01.617530+0000","last_became_peered":"2024-01-26T23:43:01.617530+0000","last_unstale":"2024-01-26T23:43:01.622974+0000","last_undegraded":"2024-01-26T23:43:01.622974+0000","last_fullsized":"2024-01-26T23:43:01.622974+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:19.830601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.625332+0000","last_change":"2024-01-26T23:43:01.620559+0000","last_active":"2024-01-26T23:43:01.625332+0000","last_peered":"2024-01-26T23:43:01.625332+0000","last_clean":"2024-01-26T23:43:01.625332+0000","last_became_active":"2024-01-26T23:43:01.619876+0000","last_became_peered":"2024-01-26T23:43:01.619876+0000","last_unstale":"2024-01-26T23:43:01.625332+0000","last_undegraded":"2024-01-26T23:43:01.625332+0000","last_fullsized":"2024-01-26T23:43:01.625332+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:51:47.149162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.622711+0000","last_change":"2024-01-26T23:43:01.618871+0000","last_active":"2024-01-26T23:43:01.622711+0000","last_peered":"2024-01-26T23:43:01.622711+0000","last_clean":"2024-01-26T23:43:01.622711+0000","last_became_active":"2024-01-26T23:43:01.618257+0000","last_became_peered":"2024-01-26T23:43:01.618257+0000","last_unstale":"2024-01-26T23:43:01.622711+0000","last_undegraded":"2024-01-26T23:43:01.622711+0000","last_fullsized":"2024-01-26T23:43:01.622711+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:07:34.029108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"69'4","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.714032+0000","last_change":"2024-01-26T23:43:00.714377+0000","last_active":"2024-01-26T23:43:01.714032+0000","last_peered":"2024-01-26T23:43:01.714032+0000","last_clean":"2024-01-26T23:43:01.714032+0000","last_became_active":"2024-01-26T23:43:00.714105+0000","last_became_peered":"2024-01-26T23:43:00.714105+0000","last_unstale":"2024-01-26T23:43:01.714032+0000","last_undegraded":"2024-01-26T23:43:01.714032+0000","last_fullsized":"2024-01-26T23:43:01.714032+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:31:22.376213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.620575+0000","last_change":"2024-01-26T23:43:01.615127+0000","last_active":"2024-01-26T23:43:01.620575+0000","last_peered":"2024-01-26T23:43:01.620575+0000","last_clean":"2024-01-26T23:43:01.620575+0000","last_became_active":"2024-01-26T23:43:01.614802+0000","last_became_peered":"2024-01-26T23:43:01.614802+0000","last_unstale":"2024-01-26T23:43:01.620575+0000","last_undegraded":"2024-01-26T23:43:01.620575+0000","last_fullsized":"2024-01-26T23:43:01.620575+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:00:59.228192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"69'1","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.710695+0000","last_change":"2024-01-26T23:43:00.719328+0000","last_active":"2024-01-26T23:43:01.710695+0000","last_peered":"2024-01-26T23:43:01.710695+0000","last_clean":"2024-01-26T23:43:01.710695+0000","last_became_active":"2024-01-26T23:43:00.719053+0000","last_became_peered":"2024-01-26T23:43:00.719053+0000","last_unstale":"2024-01-26T23:43:01.710695+0000","last_undegraded":"2024-01-26T23:43:01.710695+0000","last_fullsized":"2024-01-26T23:43:01.710695+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:41:51.340002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.619365+0000","last_change":"2024-01-26T23:43:01.616774+0000","last_active":"2024-01-26T23:43:01.619365+0000","last_peered":"2024-01-26T23:43:01.619365+0000","last_clean":"2024-01-26T23:43:01.619365+0000","last_became_active":"2024-01-26T23:43:01.616522+0000","last_became_peered":"2024-01-26T23:43:01.616522+0000","last_unstale":"2024-01-26T23:43:01.619365+0000","last_undegraded":"2024-01-26T23:43:01.619365+0000","last_fullsized":"2024-01-26T23:43:01.619365+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:32:56.856801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"69'2","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.712104+0000","last_change":"2024-01-26T23:43:00.715470+0000","last_active":"2024-01-26T23:43:01.712104+0000","last_peered":"2024-01-26T23:43:01.712104+0000","last_clean":"2024-01-26T23:43:01.712104+0000","last_became_active":"2024-01-26T23:43:00.715188+0000","last_became_peered":"2024-01-26T23:43:00.715188+0000","last_unstale":"2024-01-26T23:43:01.712104+0000","last_undegraded":"2024-01-26T23:43:01.712104+0000","last_fullsized":"2024-01-26T23:43:01.712104+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:31:22.376213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.625793+0000","last_change":"2024-01-26T23:43:01.619727+0000","last_active":"2024-01-26T23:43:01.625793+0000","last_peered":"2024-01-26T23:43:01.625793+0000","last_clean":"2024-01-26T23:43:01.625793+0000","last_became_active":"2024-01-26T23:43:01.619152+0000","last_became_peered":"2024-01-26T23:43:01.619152+0000","last_unstale":"2024-01-26T23:43:01.625793+0000","last_undegraded":"2024-01-26T23:43:01.625793+0000","last_fullsized":"2024-01-26T23:43:01.625793+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:35:19.813893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"69'2","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.713838+0000","last_change":"2024-01-26T23:43:00.717167+0000","last_active":"2024-01-26T23:43:01.713838+0000","last_peered":"2024-01-26T23:43:01.713838+0000","last_clean":"2024-01-26T23:43:01.713838+0000","last_became_active":"2024-01-26T23:43:00.716854+0000","last_became_peered":"2024-01-26T23:43:00.716854+0000","last_unstale":"2024-01-26T23:43:01.713838+0000","last_undegraded":"2024-01-26T23:43:01.713838+0000","last_fullsized":"2024-01-26T23:43:01.713838+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:08:26.147843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.627848+0000","last_change":"2024-01-26T23:43:01.623114+0000","last_active":"2024-01-26T23:43:01.627848+0000","last_peered":"2024-01-26T23:43:01.627848+0000","last_clean":"2024-01-26T23:43:01.627848+0000","last_became_active":"2024-01-26T23:43:01.622922+0000","last_became_peered":"2024-01-26T23:43:01.622922+0000","last_unstale":"2024-01-26T23:43:01.627848+0000","last_undegraded":"2024-01-26T23:43:01.627848+0000","last_fullsized":"2024-01-26T23:43:01.627848+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:04.704363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"69'1","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.710475+0000","last_change":"2024-01-26T23:43:00.724676+0000","last_active":"2024-01-26T23:43:01.710475+0000","last_peered":"2024-01-26T23:43:01.710475+0000","last_clean":"2024-01-26T23:43:01.710475+0000","last_became_active":"2024-01-26T23:43:00.724372+0000","last_became_peered":"2024-01-26T23:43:00.724372+0000","last_unstale":"2024-01-26T23:43:01.710475+0000","last_undegraded":"2024-01-26T23:43:01.710475+0000","last_fullsized":"2024-01-26T23:43:01.710475+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:23:54.142010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.627558+0000","last_change":"2024-01-26T23:43:01.623896+0000","last_active":"2024-01-26T23:43:01.627558+0000","last_peered":"2024-01-26T23:43:01.627558+0000","last_clean":"2024-01-26T23:43:01.627558+0000","last_became_active":"2024-01-26T23:43:01.623611+0000","last_became_peered":"2024-01-26T23:43:01.623611+0000","last_unstale":"2024-01-26T23:43:01.627558+0000","last_undegraded":"2024-01-26T23:43:01.627558+0000","last_fullsized":"2024-01-26T23:43:01.627558+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:00:59.228192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"69'1","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.711469+0000","last_change":"2024-01-26T23:43:00.714092+0000","last_active":"2024-01-26T23:43:01.711469+0000","last_peered":"2024-01-26T23:43:01.711469+0000","last_clean":"2024-01-26T23:43:01.711469+0000","last_became_active":"2024-01-26T23:43:00.713823+0000","last_became_peered":"2024-01-26T23:43:00.713823+0000","last_unstale":"2024-01-26T23:43:01.711469+0000","last_undegraded":"2024-01-26T23:43:01.711469+0000","last_fullsized":"2024-01-26T23:43:01.711469+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:36.728509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.628008+0000","last_change":"2024-01-26T23:43:01.620133+0000","last_active":"2024-01-26T23:43:01.628008+0000","last_peered":"2024-01-26T23:43:01.628008+0000","last_clean":"2024-01-26T23:43:01.628008+0000","last_became_active":"2024-01-26T23:43:01.619397+0000","last_became_peered":"2024-01-26T23:43:01.619397+0000","last_unstale":"2024-01-26T23:43:01.628008+0000","last_undegraded":"2024-01-26T23:43:01.628008+0000","last_fullsized":"2024-01-26T23:43:01.628008+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:31:01.785434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.607583+0000","last_change":"2024-01-26T23:43:00.713059+0000","last_active":"2024-01-26T23:43:01.607583+0000","last_peered":"2024-01-26T23:43:01.607583+0000","last_clean":"2024-01-26T23:43:01.607583+0000","last_became_active":"2024-01-26T23:43:00.712771+0000","last_became_peered":"2024-01-26T23:43:00.712771+0000","last_unstale":"2024-01-26T23:43:01.607583+0000","last_undegraded":"2024-01-26T23:43:01.607583+0000","last_fullsized":"2024-01-26T23:43:01.607583+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:15:45.000899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.624923+0000","last_change":"2024-01-26T23:43:01.621464+0000","last_active":"2024-01-26T23:43:01.624923+0000","last_peered":"2024-01-26T23:43:01.624923+0000","last_clean":"2024-01-26T23:43:01.624923+0000","last_became_active":"2024-01-26T23:43:01.621194+0000","last_became_peered":"2024-01-26T23:43:01.621194+0000","last_unstale":"2024-01-26T23:43:01.624923+0000","last_undegraded":"2024-01-26T23:43:01.624923+0000","last_fullsized":"2024-01-26T23:43:01.624923+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:41:10.641512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.601649+0000","last_change":"2024-01-26T23:43:00.713317+0000","last_active":"2024-01-26T23:43:01.601649+0000","last_peered":"2024-01-26T23:43:01.601649+0000","last_clean":"2024-01-26T23:43:01.601649+0000","last_became_active":"2024-01-26T23:43:00.713048+0000","last_became_peered":"2024-01-26T23:43:00.713048+0000","last_unstale":"2024-01-26T23:43:01.601649+0000","last_undegraded":"2024-01-26T23:43:01.601649+0000","last_fullsized":"2024-01-26T23:43:01.601649+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:06.425727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.633057+0000","last_change":"2024-01-26T23:43:01.625447+0000","last_active":"2024-01-26T23:43:01.633057+0000","last_peered":"2024-01-26T23:43:01.633057+0000","last_clean":"2024-01-26T23:43:01.633057+0000","last_became_active":"2024-01-26T23:43:01.625143+0000","last_became_peered":"2024-01-26T23:43:01.625143+0000","last_unstale":"2024-01-26T23:43:01.633057+0000","last_undegraded":"2024-01-26T23:43:01.633057+0000","last_fullsized":"2024-01-26T23:43:01.633057+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:00:59.228192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":10,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.610752+0000","last_change":"2024-01-26T23:43:00.721337+0000","last_active":"2024-01-26T23:43:01.610752+0000","last_peered":"2024-01-26T23:43:01.610752+0000","last_clean":"2024-01-26T23:43:01.610752+0000","last_became_active":"2024-01-26T23:43:00.721076+0000","last_became_peered":"2024-01-26T23:43:00.721076+0000","last_unstale":"2024-01-26T23:43:01.610752+0000","last_undegraded":"2024-01-26T23:43:01.610752+0000","last_fullsized":"2024-01-26T23:43:01.610752+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:00.104379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.630461+0000","last_change":"2024-01-26T23:43:01.626982+0000","last_active":"2024-01-26T23:43:01.630461+0000","last_peered":"2024-01-26T23:43:01.630461+0000","last_clean":"2024-01-26T23:43:01.630461+0000","last_became_active":"2024-01-26T23:43:01.626696+0000","last_became_peered":"2024-01-26T23:43:01.626696+0000","last_unstale":"2024-01-26T23:43:01.630461+0000","last_undegraded":"2024-01-26T23:43:01.630461+0000","last_fullsized":"2024-01-26T23:43:01.630461+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:38:04.183378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.617399+0000","last_change":"2024-01-26T23:43:00.720399+0000","last_active":"2024-01-26T23:43:01.617399+0000","last_peered":"2024-01-26T23:43:01.617399+0000","last_clean":"2024-01-26T23:43:01.617399+0000","last_became_active":"2024-01-26T23:43:00.720101+0000","last_became_peered":"2024-01-26T23:43:00.720101+0000","last_unstale":"2024-01-26T23:43:01.617399+0000","last_undegraded":"2024-01-26T23:43:01.617399+0000","last_fullsized":"2024-01-26T23:43:01.617399+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:41:13.518387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.622911+0000","last_change":"2024-01-26T23:43:01.616991+0000","last_active":"2024-01-26T23:43:01.622911+0000","last_peered":"2024-01-26T23:43:01.622911+0000","last_clean":"2024-01-26T23:43:01.622911+0000","last_became_active":"2024-01-26T23:43:01.616528+0000","last_became_peered":"2024-01-26T23:43:01.616528+0000","last_unstale":"2024-01-26T23:43:01.622911+0000","last_undegraded":"2024-01-26T23:43:01.622911+0000","last_fullsized":"2024-01-26T23:43:01.622911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:04:28.204578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"69'3","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.714297+0000","last_change":"2024-01-26T23:43:00.723756+0000","last_active":"2024-01-26T23:43:01.714297+0000","last_peered":"2024-01-26T23:43:01.714297+0000","last_clean":"2024-01-26T23:43:01.714297+0000","last_became_active":"2024-01-26T23:43:00.723533+0000","last_became_peered":"2024-01-26T23:43:00.723533+0000","last_unstale":"2024-01-26T23:43:01.714297+0000","last_undegraded":"2024-01-26T23:43:01.714297+0000","last_fullsized":"2024-01-26T23:43:01.714297+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:08:26.147843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.627814+0000","last_change":"2024-01-26T23:43:01.622676+0000","last_active":"2024-01-26T23:43:01.627814+0000","last_peered":"2024-01-26T23:43:01.627814+0000","last_clean":"2024-01-26T23:43:01.627814+0000","last_became_active":"2024-01-26T23:43:01.622409+0000","last_became_peered":"2024-01-26T23:43:01.622409+0000","last_unstale":"2024-01-26T23:43:01.627814+0000","last_undegraded":"2024-01-26T23:43:01.627814+0000","last_fullsized":"2024-01-26T23:43:01.627814+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:20:41.766117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"69'6","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.717212+0000","last_change":"2024-01-26T23:43:00.723406+0000","last_active":"2024-01-26T23:43:01.717212+0000","last_peered":"2024-01-26T23:43:01.717212+0000","last_clean":"2024-01-26T23:43:01.717212+0000","last_became_active":"2024-01-26T23:43:00.723103+0000","last_became_peered":"2024-01-26T23:43:00.723103+0000","last_unstale":"2024-01-26T23:43:01.717212+0000","last_undegraded":"2024-01-26T23:43:01.717212+0000","last_fullsized":"2024-01-26T23:43:01.717212+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:39:43.555499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.620762+0000","last_change":"2024-01-26T23:43:01.615337+0000","last_active":"2024-01-26T23:43:01.620762+0000","last_peered":"2024-01-26T23:43:01.620762+0000","last_clean":"2024-01-26T23:43:01.620762+0000","last_became_active":"2024-01-26T23:43:01.615020+0000","last_became_peered":"2024-01-26T23:43:01.615020+0000","last_unstale":"2024-01-26T23:43:01.620762+0000","last_undegraded":"2024-01-26T23:43:01.620762+0000","last_fullsized":"2024-01-26T23:43:01.620762+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:04.704363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"69'2","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.714191+0000","last_change":"2024-01-26T23:43:00.716155+0000","last_active":"2024-01-26T23:43:01.714191+0000","last_peered":"2024-01-26T23:43:01.714191+0000","last_clean":"2024-01-26T23:43:01.714191+0000","last_became_active":"2024-01-26T23:43:00.715808+0000","last_became_peered":"2024-01-26T23:43:00.715808+0000","last_unstale":"2024-01-26T23:43:01.714191+0000","last_undegraded":"2024-01-26T23:43:01.714191+0000","last_fullsized":"2024-01-26T23:43:01.714191+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:40:40.504486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.621724+0000","last_change":"2024-01-26T23:43:01.614854+0000","last_active":"2024-01-26T23:43:01.621724+0000","last_peered":"2024-01-26T23:43:01.621724+0000","last_clean":"2024-01-26T23:43:01.621724+0000","last_became_active":"2024-01-26T23:43:01.614554+0000","last_became_peered":"2024-01-26T23:43:01.614554+0000","last_unstale":"2024-01-26T23:43:01.621724+0000","last_undegraded":"2024-01-26T23:43:01.621724+0000","last_fullsized":"2024-01-26T23:43:01.621724+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:11:24.606563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.611032+0000","last_change":"2024-01-26T23:43:00.721192+0000","last_active":"2024-01-26T23:43:01.611032+0000","last_peered":"2024-01-26T23:43:01.611032+0000","last_clean":"2024-01-26T23:43:01.611032+0000","last_became_active":"2024-01-26T23:43:00.720882+0000","last_became_peered":"2024-01-26T23:43:00.720882+0000","last_unstale":"2024-01-26T23:43:01.611032+0000","last_undegraded":"2024-01-26T23:43:01.611032+0000","last_fullsized":"2024-01-26T23:43:01.611032+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:41:51.340002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.628024+0000","last_change":"2024-01-26T23:43:01.620353+0000","last_active":"2024-01-26T23:43:01.628024+0000","last_peered":"2024-01-26T23:43:01.628024+0000","last_clean":"2024-01-26T23:43:01.628024+0000","last_became_active":"2024-01-26T23:43:01.619747+0000","last_became_peered":"2024-01-26T23:43:01.619747+0000","last_unstale":"2024-01-26T23:43:01.628024+0000","last_undegraded":"2024-01-26T23:43:01.628024+0000","last_fullsized":"2024-01-26T23:43:01.628024+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:52:47.955317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":10,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.610827+0000","last_change":"2024-01-26T23:43:00.717737+0000","last_active":"2024-01-26T23:43:01.610827+0000","last_peered":"2024-01-26T23:43:01.610827+0000","last_clean":"2024-01-26T23:43:01.610827+0000","last_became_active":"2024-01-26T23:43:00.717486+0000","last_became_peered":"2024-01-26T23:43:00.717486+0000","last_unstale":"2024-01-26T23:43:01.610827+0000","last_undegraded":"2024-01-26T23:43:01.610827+0000","last_fullsized":"2024-01-26T23:43:01.610827+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:05.307132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.622572+0000","last_change":"2024-01-26T23:43:01.615742+0000","last_active":"2024-01-26T23:43:01.622572+0000","last_peered":"2024-01-26T23:43:01.622572+0000","last_clean":"2024-01-26T23:43:01.622572+0000","last_became_active":"2024-01-26T23:43:01.614939+0000","last_became_peered":"2024-01-26T23:43:01.614939+0000","last_unstale":"2024-01-26T23:43:01.622572+0000","last_undegraded":"2024-01-26T23:43:01.622572+0000","last_fullsized":"2024-01-26T23:43:01.622572+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:49:05.678365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"69'8","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.715908+0000","last_change":"2024-01-26T23:43:00.723821+0000","last_active":"2024-01-26T23:43:01.715908+0000","last_peered":"2024-01-26T23:43:01.715908+0000","last_clean":"2024-01-26T23:43:01.715908+0000","last_became_active":"2024-01-26T23:43:00.723545+0000","last_became_peered":"2024-01-26T23:43:00.723545+0000","last_unstale":"2024-01-26T23:43:01.715908+0000","last_undegraded":"2024-01-26T23:43:01.715908+0000","last_fullsized":"2024-01-26T23:43:01.715908+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:51:46.141832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.624916+0000","last_change":"2024-01-26T23:43:01.617890+0000","last_active":"2024-01-26T23:43:01.624916+0000","last_peered":"2024-01-26T23:43:01.624916+0000","last_clean":"2024-01-26T23:43:01.624916+0000","last_became_active":"2024-01-26T23:43:01.617556+0000","last_became_peered":"2024-01-26T23:43:01.617556+0000","last_unstale":"2024-01-26T23:43:01.624916+0000","last_undegraded":"2024-01-26T23:43:01.624916+0000","last_fullsized":"2024-01-26T23:43:01.624916+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:07:34.029108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"69'1","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.713753+0000","last_change":"2024-01-26T23:43:00.722023+0000","last_active":"2024-01-26T23:43:01.713753+0000","last_peered":"2024-01-26T23:43:01.713753+0000","last_clean":"2024-01-26T23:43:01.713753+0000","last_became_active":"2024-01-26T23:43:00.721724+0000","last_became_peered":"2024-01-26T23:43:00.721724+0000","last_unstale":"2024-01-26T23:43:01.713753+0000","last_undegraded":"2024-01-26T23:43:01.713753+0000","last_fullsized":"2024-01-26T23:43:01.713753+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:07:33.021778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.625054+0000","last_change":"2024-01-26T23:43:01.621862+0000","last_active":"2024-01-26T23:43:01.625054+0000","last_peered":"2024-01-26T23:43:01.625054+0000","last_clean":"2024-01-26T23:43:01.625054+0000","last_became_active":"2024-01-26T23:43:01.621551+0000","last_became_peered":"2024-01-26T23:43:01.621551+0000","last_unstale":"2024-01-26T23:43:01.625054+0000","last_undegraded":"2024-01-26T23:43:01.625054+0000","last_fullsized":"2024-01-26T23:43:01.625054+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:51:47.149162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"69'2","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.713992+0000","last_change":"2024-01-26T23:43:00.718605+0000","last_active":"2024-01-26T23:43:01.713992+0000","last_peered":"2024-01-26T23:43:01.713992+0000","last_clean":"2024-01-26T23:43:01.713992+0000","last_became_active":"2024-01-26T23:43:00.718330+0000","last_became_peered":"2024-01-26T23:43:00.718330+0000","last_unstale":"2024-01-26T23:43:01.713992+0000","last_undegraded":"2024-01-26T23:43:01.713992+0000","last_fullsized":"2024-01-26T23:43:01.713992+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:10:17.328392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.633069+0000","last_change":"2024-01-26T23:43:01.625497+0000","last_active":"2024-01-26T23:43:01.633069+0000","last_peered":"2024-01-26T23:43:01.633069+0000","last_clean":"2024-01-26T23:43:01.633069+0000","last_became_active":"2024-01-26T23:43:01.625247+0000","last_became_peered":"2024-01-26T23:43:01.625247+0000","last_unstale":"2024-01-26T23:43:01.633069+0000","last_undegraded":"2024-01-26T23:43:01.633069+0000","last_fullsized":"2024-01-26T23:43:01.633069+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:07.433058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"69'1","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.712089+0000","last_change":"2024-01-26T23:43:00.718379+0000","last_active":"2024-01-26T23:43:01.712089+0000","last_peered":"2024-01-26T23:43:01.712089+0000","last_clean":"2024-01-26T23:43:01.712089+0000","last_became_active":"2024-01-26T23:43:00.718108+0000","last_became_peered":"2024-01-26T23:43:00.718108+0000","last_unstale":"2024-01-26T23:43:01.712089+0000","last_undegraded":"2024-01-26T23:43:01.712089+0000","last_fullsized":"2024-01-26T23:43:01.712089+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:48:27.359511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.623100+0000","last_change":"2024-01-26T23:43:01.614209+0000","last_active":"2024-01-26T23:43:01.623100+0000","last_peered":"2024-01-26T23:43:01.623100+0000","last_clean":"2024-01-26T23:43:01.623100+0000","last_became_active":"2024-01-26T23:43:01.613913+0000","last_became_peered":"2024-01-26T23:43:01.613913+0000","last_unstale":"2024-01-26T23:43:01.623100+0000","last_undegraded":"2024-01-26T23:43:01.623100+0000","last_fullsized":"2024-01-26T23:43:01.623100+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:41:10.641512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.609234+0000","last_change":"2024-01-26T23:43:00.716960+0000","last_active":"2024-01-26T23:43:01.609234+0000","last_peered":"2024-01-26T23:43:01.609234+0000","last_clean":"2024-01-26T23:43:01.609234+0000","last_became_active":"2024-01-26T23:43:00.716672+0000","last_became_peered":"2024-01-26T23:43:00.716672+0000","last_unstale":"2024-01-26T23:43:01.609234+0000","last_undegraded":"2024-01-26T23:43:01.609234+0000","last_fullsized":"2024-01-26T23:43:01.609234+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:16:07.169692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.625415+0000","last_change":"2024-01-26T23:43:01.620683+0000","last_active":"2024-01-26T23:43:01.625415+0000","last_peered":"2024-01-26T23:43:01.625415+0000","last_clean":"2024-01-26T23:43:01.625415+0000","last_became_active":"2024-01-26T23:43:01.620423+0000","last_became_peered":"2024-01-26T23:43:01.620423+0000","last_unstale":"2024-01-26T23:43:01.625415+0000","last_undegraded":"2024-01-26T23:43:01.625415+0000","last_fullsized":"2024-01-26T23:43:01.625415+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:00:59.183013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"69'2","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.712342+0000","last_change":"2024-01-26T23:43:00.714818+0000","last_active":"2024-01-26T23:43:01.712342+0000","last_peered":"2024-01-26T23:43:01.712342+0000","last_clean":"2024-01-26T23:43:01.712342+0000","last_became_active":"2024-01-26T23:43:00.714507+0000","last_became_peered":"2024-01-26T23:43:00.714507+0000","last_unstale":"2024-01-26T23:43:01.712342+0000","last_undegraded":"2024-01-26T23:43:01.712342+0000","last_fullsized":"2024-01-26T23:43:01.712342+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:41:07.993261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.630495+0000","last_change":"2024-01-26T23:43:01.625961+0000","last_active":"2024-01-26T23:43:01.630495+0000","last_peered":"2024-01-26T23:43:01.630495+0000","last_clean":"2024-01-26T23:43:01.630495+0000","last_became_active":"2024-01-26T23:43:01.625715+0000","last_became_peered":"2024-01-26T23:43:01.625715+0000","last_unstale":"2024-01-26T23:43:01.630495+0000","last_undegraded":"2024-01-26T23:43:01.630495+0000","last_fullsized":"2024-01-26T23:43:01.630495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:37.735840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.607286+0000","last_change":"2024-01-26T23:43:00.723509+0000","last_active":"2024-01-26T23:43:01.607286+0000","last_peered":"2024-01-26T23:43:01.607286+0000","last_clean":"2024-01-26T23:43:01.607286+0000","last_became_active":"2024-01-26T23:43:00.723152+0000","last_became_peered":"2024-01-26T23:43:00.723152+0000","last_unstale":"2024-01-26T23:43:01.607286+0000","last_undegraded":"2024-01-26T23:43:01.607286+0000","last_fullsized":"2024-01-26T23:43:01.607286+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:34:39.736105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.623027+0000","last_change":"2024-01-26T23:43:01.616430+0000","last_active":"2024-01-26T23:43:01.623027+0000","last_peered":"2024-01-26T23:43:01.623027+0000","last_clean":"2024-01-26T23:43:01.623027+0000","last_became_active":"2024-01-26T23:43:01.615908+0000","last_became_peered":"2024-01-26T23:43:01.615908+0000","last_unstale":"2024-01-26T23:43:01.623027+0000","last_undegraded":"2024-01-26T23:43:01.623027+0000","last_fullsized":"2024-01-26T23:43:01.623027+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:55:25.268076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.602161+0000","last_change":"2024-01-26T23:43:00.717913+0000","last_active":"2024-01-26T23:43:01.602161+0000","last_peered":"2024-01-26T23:43:01.602161+0000","last_clean":"2024-01-26T23:43:01.602161+0000","last_became_active":"2024-01-26T23:43:00.717608+0000","last_became_peered":"2024-01-26T23:43:00.717608+0000","last_unstale":"2024-01-26T23:43:01.602161+0000","last_undegraded":"2024-01-26T23:43:01.602161+0000","last_fullsized":"2024-01-26T23:43:01.602161+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:25:02.489012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.628903+0000","last_change":"2024-01-26T23:43:01.628257+0000","last_active":"2024-01-26T23:43:01.628903+0000","last_peered":"2024-01-26T23:43:01.628903+0000","last_clean":"2024-01-26T23:43:01.628903+0000","last_became_active":"2024-01-26T23:43:01.627915+0000","last_became_peered":"2024-01-26T23:43:01.627915+0000","last_unstale":"2024-01-26T23:43:01.628903+0000","last_undegraded":"2024-01-26T23:43:01.628903+0000","last_fullsized":"2024-01-26T23:43:01.628903+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:00:59.228192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.610481+0000","last_change":"2024-01-26T23:43:00.723123+0000","last_active":"2024-01-26T23:43:01.610481+0000","last_peered":"2024-01-26T23:43:01.610481+0000","last_clean":"2024-01-26T23:43:01.610481+0000","last_became_active":"2024-01-26T23:43:00.722834+0000","last_became_peered":"2024-01-26T23:43:00.722834+0000","last_unstale":"2024-01-26T23:43:01.610481+0000","last_undegraded":"2024-01-26T23:43:01.610481+0000","last_fullsized":"2024-01-26T23:43:01.610481+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:41:55.276696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.625311+0000","last_change":"2024-01-26T23:43:01.620362+0000","last_active":"2024-01-26T23:43:01.625311+0000","last_peered":"2024-01-26T23:43:01.625311+0000","last_clean":"2024-01-26T23:43:01.625311+0000","last_became_active":"2024-01-26T23:43:01.619528+0000","last_became_peered":"2024-01-26T23:43:01.619528+0000","last_unstale":"2024-01-26T23:43:01.625311+0000","last_undegraded":"2024-01-26T23:43:01.625311+0000","last_fullsized":"2024-01-26T23:43:01.625311+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:41:10.641512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"69'2","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.710652+0000","last_change":"2024-01-26T23:43:00.720237+0000","last_active":"2024-01-26T23:43:01.710652+0000","last_peered":"2024-01-26T23:43:01.710652+0000","last_clean":"2024-01-26T23:43:01.710652+0000","last_became_active":"2024-01-26T23:43:00.719991+0000","last_became_peered":"2024-01-26T23:43:00.719991+0000","last_unstale":"2024-01-26T23:43:01.710652+0000","last_undegraded":"2024-01-26T23:43:01.710652+0000","last_fullsized":"2024-01-26T23:43:01.710652+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:41:07.993261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.625353+0000","last_change":"2024-01-26T23:43:01.616671+0000","last_active":"2024-01-26T23:43:01.625353+0000","last_peered":"2024-01-26T23:43:01.625353+0000","last_clean":"2024-01-26T23:43:01.625353+0000","last_became_active":"2024-01-26T23:43:01.616335+0000","last_became_peered":"2024-01-26T23:43:01.616335+0000","last_unstale":"2024-01-26T23:43:01.625353+0000","last_undegraded":"2024-01-26T23:43:01.625353+0000","last_fullsized":"2024-01-26T23:43:01.625353+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:20:41.766117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.610264+0000","last_change":"2024-01-26T23:43:00.723261+0000","last_active":"2024-01-26T23:43:01.610264+0000","last_peered":"2024-01-26T23:43:01.610264+0000","last_clean":"2024-01-26T23:43:01.610264+0000","last_became_active":"2024-01-26T23:43:00.722866+0000","last_became_peered":"2024-01-26T23:43:00.722866+0000","last_unstale":"2024-01-26T23:43:01.610264+0000","last_undegraded":"2024-01-26T23:43:01.610264+0000","last_fullsized":"2024-01-26T23:43:01.610264+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:00:27.828427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.627918+0000","last_change":"2024-01-26T23:43:01.625210+0000","last_active":"2024-01-26T23:43:01.627918+0000","last_peered":"2024-01-26T23:43:01.627918+0000","last_clean":"2024-01-26T23:43:01.627918+0000","last_became_active":"2024-01-26T23:43:01.624861+0000","last_became_peered":"2024-01-26T23:43:01.624861+0000","last_unstale":"2024-01-26T23:43:01.627918+0000","last_undegraded":"2024-01-26T23:43:01.627918+0000","last_fullsized":"2024-01-26T23:43:01.627918+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:34:06.780711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.601957+0000","last_change":"2024-01-26T23:43:00.716629+0000","last_active":"2024-01-26T23:43:01.601957+0000","last_peered":"2024-01-26T23:43:01.601957+0000","last_clean":"2024-01-26T23:43:01.601957+0000","last_became_active":"2024-01-26T23:43:00.715747+0000","last_became_peered":"2024-01-26T23:43:00.715747+0000","last_unstale":"2024-01-26T23:43:01.601957+0000","last_undegraded":"2024-01-26T23:43:01.601957+0000","last_fullsized":"2024-01-26T23:43:01.601957+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:38:48.810723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.627323+0000","last_change":"2024-01-26T23:43:01.626487+0000","last_active":"2024-01-26T23:43:01.627323+0000","last_peered":"2024-01-26T23:43:01.627323+0000","last_clean":"2024-01-26T23:43:01.627323+0000","last_became_active":"2024-01-26T23:43:01.626183+0000","last_became_peered":"2024-01-26T23:43:01.626183+0000","last_unstale":"2024-01-26T23:43:01.627323+0000","last_undegraded":"2024-01-26T23:43:01.627323+0000","last_fullsized":"2024-01-26T23:43:01.627323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:13:39.013119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.606778+0000","last_change":"2024-01-26T23:43:00.724459+0000","last_active":"2024-01-26T23:43:01.606778+0000","last_peered":"2024-01-26T23:43:01.606778+0000","last_clean":"2024-01-26T23:43:01.606778+0000","last_became_active":"2024-01-26T23:43:00.724174+0000","last_became_peered":"2024-01-26T23:43:00.724174+0000","last_unstale":"2024-01-26T23:43:01.606778+0000","last_undegraded":"2024-01-26T23:43:01.606778+0000","last_fullsized":"2024-01-26T23:43:01.606778+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:32:55.849471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.620793+0000","last_change":"2024-01-26T23:43:01.615478+0000","last_active":"2024-01-26T23:43:01.620793+0000","last_peered":"2024-01-26T23:43:01.620793+0000","last_clean":"2024-01-26T23:43:01.620793+0000","last_became_active":"2024-01-26T23:43:01.615221+0000","last_became_peered":"2024-01-26T23:43:01.615221+0000","last_unstale":"2024-01-26T23:43:01.620793+0000","last_undegraded":"2024-01-26T23:43:01.620793+0000","last_fullsized":"2024-01-26T23:43:01.620793+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:20:41.766117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.607402+0000","last_change":"2024-01-26T23:43:00.725344+0000","last_active":"2024-01-26T23:43:01.607402+0000","last_peered":"2024-01-26T23:43:01.607402+0000","last_clean":"2024-01-26T23:43:01.607402+0000","last_became_active":"2024-01-26T23:43:00.724788+0000","last_became_peered":"2024-01-26T23:43:00.724788+0000","last_unstale":"2024-01-26T23:43:01.607402+0000","last_undegraded":"2024-01-26T23:43:01.607402+0000","last_fullsized":"2024-01-26T23:43:01.607402+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:41:13.518387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.623007+0000","last_change":"2024-01-26T23:43:01.616226+0000","last_active":"2024-01-26T23:43:01.623007+0000","last_peered":"2024-01-26T23:43:01.623007+0000","last_clean":"2024-01-26T23:43:01.623007+0000","last_became_active":"2024-01-26T23:43:01.615438+0000","last_became_peered":"2024-01-26T23:43:01.615438+0000","last_unstale":"2024-01-26T23:43:01.623007+0000","last_undegraded":"2024-01-26T23:43:01.623007+0000","last_fullsized":"2024-01-26T23:43:01.623007+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:22:09.387956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.617857+0000","last_change":"2024-01-26T23:43:00.717095+0000","last_active":"2024-01-26T23:43:01.617857+0000","last_peered":"2024-01-26T23:43:01.617857+0000","last_clean":"2024-01-26T23:43:01.617857+0000","last_became_active":"2024-01-26T23:43:00.716830+0000","last_became_peered":"2024-01-26T23:43:00.716830+0000","last_unstale":"2024-01-26T23:43:01.617857+0000","last_undegraded":"2024-01-26T23:43:01.617857+0000","last_fullsized":"2024-01-26T23:43:01.617857+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:34:39.736105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.628172+0000","last_change":"2024-01-26T23:43:01.627301+0000","last_active":"2024-01-26T23:43:01.628172+0000","last_peered":"2024-01-26T23:43:01.628172+0000","last_clean":"2024-01-26T23:43:01.628172+0000","last_became_active":"2024-01-26T23:43:01.617977+0000","last_became_peered":"2024-01-26T23:43:01.617977+0000","last_unstale":"2024-01-26T23:43:01.628172+0000","last_undegraded":"2024-01-26T23:43:01.628172+0000","last_fullsized":"2024-01-26T23:43:01.628172+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:40:41.511816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.609480+0000","last_change":"2024-01-26T23:43:00.719444+0000","last_active":"2024-01-26T23:43:01.609480+0000","last_peered":"2024-01-26T23:43:01.609480+0000","last_clean":"2024-01-26T23:43:01.609480+0000","last_became_active":"2024-01-26T23:43:00.719223+0000","last_became_peered":"2024-01-26T23:43:00.719223+0000","last_unstale":"2024-01-26T23:43:01.609480+0000","last_undegraded":"2024-01-26T23:43:01.609480+0000","last_fullsized":"2024-01-26T23:43:01.609480+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:00.104379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.618763+0000","last_change":"2024-01-26T23:43:01.615549+0000","last_active":"2024-01-26T23:43:01.618763+0000","last_peered":"2024-01-26T23:43:01.618763+0000","last_clean":"2024-01-26T23:43:01.618763+0000","last_became_active":"2024-01-26T23:43:01.615290+0000","last_became_peered":"2024-01-26T23:43:01.615290+0000","last_unstale":"2024-01-26T23:43:01.618763+0000","last_undegraded":"2024-01-26T23:43:01.618763+0000","last_fullsized":"2024-01-26T23:43:01.618763+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:43:00.694334+0000","last_clean_scrub_stamp":"2024-01-26T23:43:00.694334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:34:40.743435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.603877+0000","last_change":"2024-01-26T23:43:00.716803+0000","last_active":"2024-01-26T23:43:01.603877+0000","last_peered":"2024-01-26T23:43:01.603877+0000","last_clean":"2024-01-26T23:43:01.603877+0000","last_became_active":"2024-01-26T23:43:00.716487+0000","last_became_peered":"2024-01-26T23:43:00.716487+0000","last_unstale":"2024-01-26T23:43:01.603877+0000","last_undegraded":"2024-01-26T23:43:01.603877+0000","last_fullsized":"2024-01-26T23:43:01.603877+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:42:59.687004+0000","last_clean_scrub_stamp":"2024-01-26T23:42:59.687004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:34:05.773381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"61'139","reported_seq":286,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-26T23:43:01.610456+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:43:01.610456+0000","last_peered":"2024-01-26T23:43:01.610456+0000","last_clean":"2024-01-26T23:43:01.610456+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:43:01.610456+0000","last_undegraded":"2024-01-26T23:43:01.610456+0000","last_fullsized":"2024-01-26T23:43:01.610456+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575012,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306888,"kb_used_data":1408,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064952,"statfs":{"total":96636764160,"available":96322510848,"internally_reserved":0,"allocated":1441792,"data_stored":1007753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:43:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.59099999999999997,"15min":0.56599999999999995},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.75700000000000001,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.60099999999999998,"15min":0.57999999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.68600000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.46300000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:43:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.59299999999999997,"15min":0.56899999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.72699999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.60799999999999998,"15min":0.58599999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.36099999999999999},"max":{"1min":0.71599999999999997,"5min":0.78900000000000003,"15min":0.85199999999999998},"last":0.54100000000000004}]},{"osd":2,"last update":"Fri Jan 26 23:43:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.58499999999999996,"15min":0.56999999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.81699999999999995},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75900000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.61899999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:43:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59899999999999998,"15min":0.58699999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.35399999999999998},"max":{"1min":0.74299999999999999,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.61199999999999999,"15min":0.60199999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.373},"max":{"1min":0.80600000000000005,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.51500000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:43:01 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.47199999999999998,"15min":0.46600000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.496,"15min":0.496},"min":{"1min":0.38100000000000001,"5min":0.33100000000000002,"15min":0.314},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.73799999999999999},"last":0.48799999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:43:01 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.70799999999999996,"5min":0.745,"15min":0.91700000000000004},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54600000000000004,"15min":0.53900000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.96599999999999997},"last":0.68600000000000005}]},{"osd":6,"last update":"Fri Jan 26 23:43:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57299999999999995,"15min":0.55800000000000005},"min":{"1min":0.379,"5min":0.375,"15min":0.33100000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.86899999999999999},"last":0.65900000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575012,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302212,"kb_used_data":828,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069628,"statfs":{"total":96636764160,"available":96327299072,"internally_reserved":0,"allocated":847872,"data_stored":417885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.58599999999999997,"15min":0.56100000000000005},"min":{"1min":0.30199999999999999,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":0.71999999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60299999999999998,"15min":0.58399999999999996},"min":{"1min":0.41399999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71699999999999997,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.53200000000000003}]},{"osd":1,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.57999999999999996,"15min":0.56899999999999995},"min":{"1min":0.41699999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.60699999999999998,"15min":0.59099999999999997},"min":{"1min":0.38100000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73499999999999999,"5min":0.874,"15min":0.874},"last":0.71299999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.58999999999999997,"15min":0.57799999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.29599999999999999},"max":{"1min":0.73299999999999998,"5min":0.78100000000000003,"15min":0.85799999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59799999999999998,"15min":0.58299999999999996},"min":{"1min":0.45300000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76200000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.63200000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60499999999999998,"15min":0.59399999999999997},"min":{"1min":0.435,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.80700000000000005,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.62,"15min":0.61499999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39000000000000001},"max":{"1min":0.71699999999999997,"5min":0.82299999999999995,"15min":0.84499999999999997},"last":0.64600000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.46999999999999997,"15min":0.47399999999999998},"min":{"1min":0.33300000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.70399999999999996,"5min":0.77800000000000002,"15min":0.90200000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.48799999999999999,"15min":0.498},"min":{"1min":0.32000000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.67400000000000004,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.59999999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.47999999999999998,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.69499999999999995,"5min":0.76500000000000001,"15min":0.78200000000000003},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51800000000000002,"15min":0.53000000000000003},"min":{"1min":0.34399999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.77900000000000003,"5min":0.83499999999999996,"15min":0.96499999999999997},"last":0.65800000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52200000000000002,"15min":0.54000000000000004},"min":{"1min":0.38600000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.73599999999999999,"5min":0.85899999999999999,"15min":0.97199999999999998},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.308},"max":{"1min":0.70399999999999996,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.67700000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575012,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302212,"kb_used_data":828,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069628,"statfs":{"total":96636764160,"available":96327299072,"internally_reserved":0,"allocated":847872,"data_stored":417421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39500000000000002,"15min":0.40600000000000003},"min":{"1min":0.26500000000000001,"5min":0.26200000000000001,"15min":0.23200000000000001},"max":{"1min":0.57799999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.441},"min":{"1min":0.32500000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.58199999999999996,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.371}]},{"osd":2,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.41599999999999998,"15min":0.40699999999999997},"min":{"1min":0.33700000000000002,"5min":0.30599999999999999,"15min":0.26200000000000001},"max":{"1min":0.61399999999999999,"5min":0.63100000000000001,"15min":0.65900000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.38600000000000001,"5min":0.32600000000000001,"15min":0.28999999999999998},"max":{"1min":0.69499999999999995,"5min":0.71899999999999997,"15min":0.79600000000000004},"last":0.38300000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.505,"15min":0.505},"min":{"1min":0.42499999999999999,"5min":0.33800000000000002,"15min":0.32100000000000001},"max":{"1min":0.69099999999999995,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.81999999999999995},"last":0.40300000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.66700000000000004,"15min":0.64000000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.39400000000000002},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.629},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.65900000000000003,"15min":0.66700000000000004},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.433},"max":{"1min":0.91900000000000004,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.71799999999999997}]},{"osd":5,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.67900000000000005,"15min":0.67600000000000005},"min":{"1min":0.49299999999999999,"5min":0.45000000000000001,"15min":0.42799999999999999},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.748,"5min":0.67400000000000004,"15min":0.66600000000000004},"min":{"1min":0.57699999999999996,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.92200000000000004,"5min":0.98599999999999999,"15min":1.018},"last":0.69599999999999995}]},{"osd":6,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.68899999999999995,"15min":0.68600000000000005},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.45700000000000002},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.1279999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.69199999999999995,"15min":0.69099999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.42599999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.56299999999999994}]},{"osd":7,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.72099999999999997,"15min":0.71399999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.49399999999999999},"max":{"1min":0.94699999999999995,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.59199999999999997,"5min":0.5,"15min":0.438},"max":{"1min":1.048,"5min":1.048,"15min":1.3049999999999999},"last":0.58799999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575011,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302792,"kb_used_data":1408,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069048,"statfs":{"total":96636764160,"available":96326705152,"internally_reserved":0,"allocated":1441792,"data_stored":1007685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40100000000000002,"15min":0.41299999999999998},"min":{"1min":0.27800000000000002,"5min":0.26400000000000001,"15min":0.214},"max":{"1min":0.52200000000000002,"5min":0.64100000000000001,"15min":1.284},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46899999999999997,"15min":0.45900000000000002},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.27300000000000002},"max":{"1min":0.72699999999999998,"5min":0.75800000000000001,"15min":1.024},"last":0.372}]},{"osd":2,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.433,"15min":0.443},"min":{"1min":0.318,"5min":0.28199999999999997,"15min":0.26700000000000002},"max":{"1min":0.59699999999999998,"5min":0.66100000000000003,"15min":1.2230000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.68100000000000005,"5min":0.68300000000000005,"15min":1.2430000000000001},"last":0.42299999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.47999999999999998,"15min":0.47599999999999998},"min":{"1min":0.38800000000000001,"5min":0.317,"15min":0.26800000000000002},"max":{"1min":0.626,"5min":0.69899999999999995,"15min":1.0980000000000001},"last":0.438},{"interface":"front","average":{"1min":0.501,"5min":0.498,"15min":0.5},"min":{"1min":0.39500000000000002,"5min":0.34000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69799999999999995,"5min":0.74099999999999999,"15min":1.1850000000000001},"last":0.45300000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.65200000000000002,"15min":0.65700000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.372},"max":{"1min":0.86799999999999999,"5min":0.93300000000000005,"15min":1.133},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66600000000000004,"15min":0.66900000000000004},"min":{"1min":0.52900000000000003,"5min":0.443,"15min":0.443},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":1.0169999999999999},"last":0.56499999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.66100000000000003,"15min":0.66900000000000004},"min":{"1min":0.46500000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":1.0880000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68200000000000005,"15min":0.69199999999999995},"min":{"1min":0.504,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93300000000000005,"5min":0.98699999999999999,"15min":1.169},"last":0.51800000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67300000000000004,"15min":0.69399999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.44800000000000001},"max":{"1min":0.88400000000000001,"5min":0.97299999999999998,"15min":1.071},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69399999999999995,"15min":0.70999999999999996},"min":{"1min":0.55200000000000005,"5min":0.502,"15min":0.48599999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.1160000000000001},"last":0.58499999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.72099999999999997,"15min":0.73999999999999999},"min":{"1min":0.51300000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.73699999999999999,"15min":0.749},"min":{"1min":0.58999999999999997,"5min":0.433,"15min":0.433},"max":{"1min":0.98699999999999999,"5min":1.028,"15min":1.2010000000000001},"last":0.626}]}]},{"osd":2,"up_from":13,"seq":55834575011,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306312,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065528,"statfs":{"total":96636764160,"available":96323100672,"internally_reserved":0,"allocated":851968,"data_stored":418797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.503},"min":{"1min":0.28399999999999997,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":0.53200000000000003,"5min":0.56699999999999995,"15min":9.0540000000000003},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.443,"5min":0.42199999999999999,"15min":0.46200000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.24299999999999999},"max":{"1min":0.57599999999999996,"5min":0.64000000000000001,"15min":4.1970000000000001},"last":0.47499999999999998}]},{"osd":1,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.438,"15min":0.42299999999999999},"min":{"1min":0.28499999999999998,"5min":0.27400000000000002,"15min":0.221},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.82199999999999995},"last":0.5},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.502,"15min":0.64800000000000002},"min":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.316},"max":{"1min":0.65900000000000003,"5min":0.72499999999999998,"15min":12.157},"last":0.52500000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.502,"15min":0.48599999999999999},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33500000000000002},"max":{"1min":0.65500000000000003,"5min":0.74099999999999999,"15min":0.86499999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47899999999999998,"15min":0.57699999999999996},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.30399999999999999},"max":{"1min":0.63600000000000001,"5min":0.73199999999999998,"15min":9.0879999999999992},"last":0.39900000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.65100000000000002,"15min":0.749},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.88500000000000001,"5min":1.048,"15min":9.0150000000000006},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.64600000000000002,"15min":0.68400000000000005},"min":{"1min":0.55700000000000005,"5min":0.371,"15min":0.371},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":4.2480000000000002},"last":0.70699999999999996}]},{"osd":5,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66400000000000003,"15min":0.69799999999999995},"min":{"1min":0.55500000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.85899999999999999,"5min":1.008,"15min":4.3200000000000003},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.69199999999999995,"15min":0.67800000000000005},"min":{"1min":0.52000000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.95199999999999996,"5min":1.085,"15min":1.085},"last":0.72499999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.69899999999999995,"15min":0.68600000000000005},"min":{"1min":0.46999999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.68899999999999995,"15min":0.78800000000000003},"min":{"1min":0.53500000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.94699999999999995,"5min":1.111,"15min":8.9700000000000006},"last":0.65500000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.70799999999999996,"15min":0.752},"min":{"1min":0.61099999999999999,"5min":0.54100000000000004,"15min":0.45800000000000002},"max":{"1min":0.93700000000000006,"5min":1.127,"15min":4.282},"last":0.747},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.746,"15min":0.73499999999999999},"min":{"1min":0.52900000000000003,"5min":0.46100000000000002,"15min":0.42099999999999999},"max":{"1min":1.0329999999999999,"5min":1.139,"15min":1.139},"last":0.67100000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575010,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306316,"kb_used_data":836,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065524,"statfs":{"total":96636764160,"available":96323096576,"internally_reserved":0,"allocated":856064,"data_stored":418819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.23999999999999999},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44900000000000001,"15min":0.56100000000000005},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":11.968},"last":0.312}]},{"osd":1,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.45600000000000002,"15min":0.57299999999999995},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.633,"5min":1.633,"15min":12.050000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48299999999999998,"15min":0.48999999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.2150000000000001},"last":0.35099999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.48899999999999999,"15min":0.59399999999999997},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.28999999999999998},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":12.234},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.46999999999999997,"15min":0.58699999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":11.925000000000001},"last":0.38300000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.70199999999999996,"15min":0.78200000000000003},"min":{"1min":0.46999999999999997,"5min":0.45500000000000002,"15min":0.40200000000000002},"max":{"1min":3.306,"5min":3.306,"15min":12.099},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68000000000000005,"15min":0.66900000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.379},"max":{"1min":0.89700000000000002,"5min":1.0009999999999999,"15min":1.0649999999999999},"last":0.70099999999999996}]},{"osd":5,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67700000000000005,"15min":0.79300000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.593,"5min":1.593,"15min":12.199999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.69099999999999995,"15min":0.68000000000000005},"min":{"1min":0.45300000000000001,"5min":0.439,"15min":0.436},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.79000000000000004}]},{"osd":6,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.70099999999999996,"15min":0.80000000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.506,"5min":1.506,"15min":12.132999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.70799999999999996,"15min":0.70199999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.753}]},{"osd":7,"last update":"Fri Jan 26 23:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.74099999999999999,"15min":0.70499999999999996},"min":{"1min":0.48499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.238,"5min":3.238,"15min":3.238},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.77200000000000002,"15min":0.86499999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.48699999999999999},"max":{"1min":3.351,"5min":3.351,"15min":12.167},"last":0.68300000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575012,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298112,"kb_used_data":824,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073728,"statfs":{"total":96636764160,"available":96331497472,"internally_reserved":0,"allocated":843776,"data_stored":417815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58299999999999996,"15min":0.57399999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69699999999999995,"5min":0.79800000000000004,"15min":1.0209999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60199999999999998,"15min":0.58199999999999996},"min":{"1min":0.41499999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70099999999999996,"5min":0.84099999999999997,"15min":1.161},"last":0.40500000000000003}]},{"osd":1,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.60799999999999998,"15min":0.59699999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73699999999999999,"5min":0.88,"15min":1.069},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.39100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72699999999999998,"5min":0.78800000000000003,"15min":1.085},"last":0.56999999999999995}]},{"osd":2,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59199999999999997,"15min":0.57399999999999995},"min":{"1min":0.38400000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.77000000000000002,"5min":0.85999999999999999,"15min":1.052},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61399999999999999,"15min":0.61299999999999999},"min":{"1min":0.45700000000000002,"5min":0.40899999999999997,"15min":0.373},"max":{"1min":0.77600000000000002,"5min":0.80100000000000005,"15min":1.111},"last":0.48199999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61099999999999999,"15min":0.59999999999999998},"min":{"1min":0.41699999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.71999999999999997,"5min":0.78300000000000003,"15min":1.1299999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.45200000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.69699999999999995,"5min":0.88800000000000001,"15min":1.1419999999999999},"last":0.50900000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47599999999999998,"15min":0.47899999999999998},"min":{"1min":0.39300000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.64400000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.496,"15min":0.51100000000000001},"min":{"1min":0.33400000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.69799999999999995,"5min":0.76700000000000002,"15min":0.77800000000000002},"last":0.59199999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.38300000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53100000000000003,"15min":0.54100000000000004},"min":{"1min":0.35499999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.67200000000000004,"5min":0.78200000000000003,"15min":0.95699999999999996},"last":0.52900000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.42799999999999999,"5min":0.375,"15min":0.35799999999999998},"max":{"1min":0.745,"5min":0.75900000000000001,"15min":0.76300000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.501,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.74099999999999999,"5min":0.78400000000000003,"15min":0.879},"last":0.55600000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575012,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298108,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073732,"statfs":{"total":96636764160,"available":96331501568,"internally_reserved":0,"allocated":839680,"data_stored":417261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.57699999999999996,"15min":0.55800000000000005},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.26000000000000001},"max":{"1min":2.0289999999999999,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57299999999999995,"15min":0.57599999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.41299999999999998}]},{"osd":1,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61399999999999999,"15min":0.60899999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.3319999999999999,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63700000000000001,"15min":0.61699999999999999},"min":{"1min":0.435,"5min":0.41799999999999998,"15min":0.379},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.53600000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55500000000000005,"15min":0.56399999999999995},"min":{"1min":0.38200000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72599999999999998,"5min":0.83699999999999997,"15min":0.83799999999999997},"last":0.436},{"interface":"front","average":{"1min":0.62,"5min":0.60299999999999998,"15min":0.64700000000000002},"min":{"1min":0.39400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":6.4589999999999996},"last":0.47999999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57499999999999996,"15min":0.57399999999999995},"min":{"1min":0.35999999999999999,"5min":0.35799999999999998,"15min":0.314},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.88400000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.59899999999999998,"15min":0.64700000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":6.0629999999999997},"last":0.505}]},{"osd":4,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.26300000000000001},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48999999999999999,"15min":0.497},"min":{"1min":0.34499999999999997,"5min":0.315,"15min":0.28100000000000003},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.67700000000000005}]},{"osd":6,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.48999999999999999,"15min":0.48499999999999999},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.27000000000000002},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52600000000000002,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.29299999999999998},"max":{"1min":1.99,"5min":1.99,"15min":1.99},"last":0.623}]},{"osd":7,"last update":"Fri Jan 26 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.56000000000000005,"15min":0.54900000000000004},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.5270000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55700000000000005,"15min":0.57299999999999995},"min":{"1min":0.36899999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.624,"5min":0.82199999999999995,"15min":0.83099999999999996},"last":0.60499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:43:10.968 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-26T23:43:10.968 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-26T23:43:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.142+0000 7f1b82081700 1 Processor -- start 2024-01-26T23:43:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.143+0000 7f1b82081700 1 -- start start 2024-01-26T23:43:11.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.143+0000 7f1b82081700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1b7c156940 0x7f1b7c06c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:11.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.143+0000 7f1b82081700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7c0c8b90 con 0x7f1b7c156570 2024-01-26T23:43:11.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b82081700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7c0c8e50 con 0x7f1b7c06f9a0 2024-01-26T23:43:11.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b82081700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b7c06d0b0 con 0x7f1b7c156940 2024-01-26T23:43:11.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7b7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1b7c156570 0x7f1b7c0c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32822/0 (socket says 172.21.15.46:32822) 2024-01-26T23:43:11.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7b7fe700 1 -- 172.21.15.46:0/2905854110 learned_addr learned my addr 172.21.15.46:0/2905854110 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:43:11.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7affd700 1 --2- 172.21.15.46:0/2905854110 >> v2:172.21.15.106:3300/0 conn(0x7f1b7c156940 0x7f1b7c06c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:11.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7a7fc700 1 -- 172.21.15.46:0/2905854110 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761969701 0 0) 0x7f1b7c0c8b90 con 0x7f1b7c156570 2024-01-26T23:43:11.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7a7fc700 1 -- 172.21.15.46:0/2905854110 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b64003570 con 0x7f1b7c156570 2024-01-26T23:43:11.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7a7fc700 1 -- 172.21.15.46:0/2905854110 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1489186195 0 0) 0x7f1b64003570 con 0x7f1b7c156570 2024-01-26T23:43:11.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7a7fc700 1 -- 172.21.15.46:0/2905854110 >> v2:172.21.15.106:3300/0 conn(0x7f1b7c156940 msgr2=0x7f1b7c06c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:11.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7a7fc700 1 --2- 172.21.15.46:0/2905854110 >> v2:172.21.15.106:3300/0 conn(0x7f1b7c156940 0x7f1b7c06c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7a7fc700 1 -- 172.21.15.46:0/2905854110 >> v1:172.21.15.106:6789/0 conn(0x7f1b7c06f9a0 legacy=0x7f1b7c06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:11.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7a7fc700 1 -- 172.21.15.46:0/2905854110 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c0c4c40 con 0x7f1b7c156570 2024-01-26T23:43:11.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7a7fc700 1 -- 172.21.15.46:0/2905854110 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f1b6c001470 con 0x7f1b7c156570 2024-01-26T23:43:11.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7a7fc700 1 -- 172.21.15.46:0/2905854110 >> v1:172.21.15.46:6789/0 conn(0x7f1b7c156570 legacy=0x7f1b7c0c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:11.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7a7fc700 1 --2- 172.21.15.46:0/2905854110 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b64003a60 0x7f1b64003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:11.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7a7fc700 1 --2- 172.21.15.46:0/2905854110 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b64004810 0x7f1b64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7a7fc700 1 --2- 172.21.15.46:0/2905854110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b640051c0 0x7f1b64010070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7a7fc700 1 -- 172.21.15.46:0/2905854110 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c06d0b0 con 0x7f1b640051c0 2024-01-26T23:43:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7a7fc700 1 -- 172.21.15.46:0/2905854110 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b640105b0 con 0x7f1b64003a60 2024-01-26T23:43:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.144+0000 7f1b7a7fc700 1 -- 172.21.15.46:0/2905854110 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b64010770 con 0x7f1b64004810 2024-01-26T23:43:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.145+0000 7f1b7bfff700 1 --2- 172.21.15.46:0/2905854110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b640051c0 0x7f1b64010070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.145+0000 7f1b7b7fe700 1 --2- 172.21.15.46:0/2905854110 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b64003a60 0x7f1b64003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.145+0000 7f1b7affd700 1 --2- 172.21.15.46:0/2905854110 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b64004810 0x7f1b64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.145+0000 7f1b7bfff700 1 -- 172.21.15.46:0/2905854110 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b64004810 msgr2=0x7f1b64004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.145+0000 7f1b7bfff700 1 --2- 172.21.15.46:0/2905854110 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b64004810 0x7f1b64004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.145+0000 7f1b7bfff700 1 -- 172.21.15.46:0/2905854110 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b64003a60 msgr2=0x7f1b64003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.145+0000 7f1b7bfff700 1 --2- 172.21.15.46:0/2905854110 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b64003a60 0x7f1b64003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.145+0000 7f1b7bfff700 1 -- 172.21.15.46:0/2905854110 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b64003570 con 0x7f1b640051c0 2024-01-26T23:43:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.145+0000 7f1b7bfff700 1 -- 172.21.15.46:0/2905854110 reap_dead start 2024-01-26T23:43:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.145+0000 7f1b7bfff700 1 --2- 172.21.15.46:0/2905854110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b640051c0 0x7f1b64010070 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1b74001fc0 tx=0x7f1b74001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=908bae5e429d22a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:11.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.145+0000 7f1b7a7fc700 1 -- 172.21.15.46:0/2905854110 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b74001950 con 0x7f1b640051c0 2024-01-26T23:43:11.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.145+0000 7f1b7a7fc700 1 -- 172.21.15.46:0/2905854110 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7400c8b0 con 0x7f1b640051c0 2024-01-26T23:43:11.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.145+0000 7f1b7a7fc700 1 -- 172.21.15.46:0/2905854110 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7400cc90 con 0x7f1b640051c0 2024-01-26T23:43:11.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.146+0000 7f1b82081700 1 -- 172.21.15.46:0/2905854110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b640051c0 msgr2=0x7f1b64010070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:11.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.146+0000 7f1b82081700 1 --2- 172.21.15.46:0/2905854110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b640051c0 0x7f1b64010070 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1b74001fc0 tx=0x7f1b74001640 comp rx=0 tx=0).stop 2024-01-26T23:43:11.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.146+0000 7f1b82081700 1 -- 172.21.15.46:0/2905854110 shutdown_connections 2024-01-26T23:43:11.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.146+0000 7f1b82081700 1 --2- 172.21.15.46:0/2905854110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b640051c0 0x7f1b64010070 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.146+0000 7f1b82081700 1 -- 172.21.15.46:0/2905854110 >> 172.21.15.46:0/2905854110 conn(0x7f1b7c0cdf50 msgr2=0x7f1b7c0c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:11.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.146+0000 7f1b82081700 1 -- 172.21.15.46:0/2905854110 shutdown_connections 2024-01-26T23:43:11.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.146+0000 7f1b82081700 1 -- 172.21.15.46:0/2905854110 wait complete. 2024-01-26T23:43:11.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.147+0000 7f1b82081700 1 Processor -- start 2024-01-26T23:43:11.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.147+0000 7f1b82081700 1 -- start start 2024-01-26T23:43:11.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.148+0000 7f1b82081700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b640051c0 0x7f1b7c0cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:11.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.148+0000 7f1b82081700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b7c0cb7f0 0x7f1b7c0cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:11.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.148+0000 7f1b82081700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b7c157c80 0x7f1b7c158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:11.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.148+0000 7f1b82081700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c1479a0 con 0x7f1b640051c0 2024-01-26T23:43:11.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.148+0000 7f1b82081700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c0c6a30 con 0x7f1b7c157c80 2024-01-26T23:43:11.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.148+0000 7f1b82081700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c1577f0 con 0x7f1b7c0cb7f0 2024-01-26T23:43:11.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.148+0000 7f1b7b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b640051c0 0x7f1b7c0cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:11.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.148+0000 7f1b7b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b640051c0 0x7f1b7c0cacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46856/0 (socket says 172.21.15.46:46856) 2024-01-26T23:43:11.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.148+0000 7f1b7b7fe700 1 -- 172.21.15.46:0/1560475399 learned_addr learned my addr 172.21.15.46:0/1560475399 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:11.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.148+0000 7f1b7affd700 1 --2- 172.21.15.46:0/1560475399 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b7c0cb7f0 0x7f1b7c0cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:11.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.148+0000 7f1b7bfff700 1 --2- 172.21.15.46:0/1560475399 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b7c157c80 0x7f1b7c158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:11.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.149+0000 7f1b7b7fe700 1 -- 172.21.15.46:0/1560475399 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b7c0cb7f0 msgr2=0x7f1b7c0cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:11.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.149+0000 7f1b7b7fe700 1 --2- 172.21.15.46:0/1560475399 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b7c0cb7f0 0x7f1b7c0cbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.149+0000 7f1b7b7fe700 1 -- 172.21.15.46:0/1560475399 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b7c157c80 msgr2=0x7f1b7c158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:11.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.149+0000 7f1b7b7fe700 1 --2- 172.21.15.46:0/1560475399 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b7c157c80 0x7f1b7c158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.149+0000 7f1b7b7fe700 1 -- 172.21.15.46:0/1560475399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c0cc1a0 con 0x7f1b640051c0 2024-01-26T23:43:11.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.149+0000 7f1b7b7fe700 1 --2- 172.21.15.46:0/1560475399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b640051c0 0x7f1b7c0cacd0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0073c0 tx=0x7f1b6c001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:11.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.149+0000 7f1b637fe700 1 -- 172.21.15.46:0/1560475399 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c011590 con 0x7f1b640051c0 2024-01-26T23:43:11.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.149+0000 7f1b637fe700 1 -- 172.21.15.46:0/1560475399 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6c0116f0 con 0x7f1b640051c0 2024-01-26T23:43:11.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.149+0000 7f1b82081700 1 -- 172.21.15.46:0/1560475399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7c156ec0 con 0x7f1b640051c0 2024-01-26T23:43:11.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.149+0000 7f1b637fe700 1 -- 172.21.15.46:0/1560475399 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c011a90 con 0x7f1b640051c0 2024-01-26T23:43:11.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.149+0000 7f1b82081700 1 -- 172.21.15.46:0/1560475399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7c157480 con 0x7f1b640051c0 2024-01-26T23:43:11.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.150+0000 7f1b82081700 1 -- 172.21.15.46:0/1560475399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b7c0c75d0 con 0x7f1b640051c0 2024-01-26T23:43:11.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.151+0000 7f1b637fe700 1 -- 172.21.15.46:0/1560475399 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1b6c01e3b0 con 0x7f1b640051c0 2024-01-26T23:43:11.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.151+0000 7f1b637fe700 1 --2- 172.21.15.46:0/1560475399 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1b68066f30 0x7f1b680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:11.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.151+0000 7f1b637fe700 1 -- 172.21.15.46:0/1560475399 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f1b6c0842f0 con 0x7f1b640051c0 2024-01-26T23:43:11.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.151+0000 7f1b7affd700 1 --2- 172.21.15.46:0/1560475399 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1b68066f30 0x7f1b680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:11.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.152+0000 7f1b7affd700 1 --2- 172.21.15.46:0/1560475399 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1b68066f30 0x7f1b680693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c14ea00 tx=0x7f1b700091a0 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:11.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.156+0000 7f1b637fe700 1 -- 172.21.15.46:0/1560475399 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1b6c0512b0 con 0x7f1b640051c0 2024-01-26T23:43:11.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.443+0000 7f1b82081700 1 -- 172.21.15.46:0/1560475399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f1b7c04c710 con 0x7f1b640051c0 2024-01-26T23:43:11.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.447+0000 7f1b637fe700 1 -- 172.21.15.46:0/1560475399 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+0 (secure 0 0 0) 0x7f1b6c0548d0 con 0x7f1b640051c0 2024-01-26T23:43:11.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.449+0000 7f1b82081700 1 -- 172.21.15.46:0/1560475399 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1b68066f30 msgr2=0x7f1b680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:11.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.449+0000 7f1b82081700 1 --2- 172.21.15.46:0/1560475399 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1b68066f30 0x7f1b680693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c14ea00 tx=0x7f1b700091a0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.449+0000 7f1b82081700 1 -- 172.21.15.46:0/1560475399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b640051c0 msgr2=0x7f1b7c0cacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:11.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.449+0000 7f1b82081700 1 --2- 172.21.15.46:0/1560475399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b640051c0 0x7f1b7c0cacd0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0073c0 tx=0x7f1b6c001e50 comp rx=0 tx=0).stop 2024-01-26T23:43:11.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.449+0000 7f1b82081700 1 -- 172.21.15.46:0/1560475399 shutdown_connections 2024-01-26T23:43:11.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.449+0000 7f1b82081700 1 --2- 172.21.15.46:0/1560475399 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1b68066f30 0x7f1b680693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.449+0000 7f1b82081700 1 --2- 172.21.15.46:0/1560475399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b640051c0 0x7f1b7c0cacd0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.449+0000 7f1b82081700 1 --2- 172.21.15.46:0/1560475399 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b7c157c80 0x7f1b7c158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.449+0000 7f1b82081700 1 --2- 172.21.15.46:0/1560475399 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b7c0cb7f0 0x7f1b7c0cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.449+0000 7f1b82081700 1 -- 172.21.15.46:0/1560475399 >> 172.21.15.46:0/1560475399 conn(0x7f1b7c0cdf50 msgr2=0x7f1b7c1543e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:11.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.449+0000 7f1b82081700 1 -- 172.21.15.46:0/1560475399 shutdown_connections 2024-01-26T23:43:11.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.449+0000 7f1b82081700 1 -- 172.21.15.46:0/1560475399 wait complete. 2024-01-26T23:43:11.452 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-01-26T23:43:11.464 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:43:11.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.595+0000 7f355f461700 1 Processor -- start 2024-01-26T23:43:11.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.595+0000 7f355f461700 1 -- start start 2024-01-26T23:43:11.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.596+0000 7f355f461700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3558067af0 0x7f3558067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:11.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.596+0000 7f355f461700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f355815da50 0x7f355815deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:11.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.596+0000 7f355f461700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355806cb40 con 0x7f355806f750 2024-01-26T23:43:11.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.596+0000 7f355f461700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355806d6e0 con 0x7f3558067af0 2024-01-26T23:43:11.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.596+0000 7f355f461700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355806c530 con 0x7f355815da50 2024-01-26T23:43:11.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.596+0000 7f355c9fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f355806f750 0x7f355806cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32836/0 (socket says 172.21.15.46:32836) 2024-01-26T23:43:11.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.596+0000 7f355c9fc700 1 -- 172.21.15.46:0/164018783 learned_addr learned my addr 172.21.15.46:0/164018783 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:43:11.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.597+0000 7f355d1fd700 1 --2- 172.21.15.46:0/164018783 >> v2:172.21.15.106:3300/0 conn(0x7f3558067af0 0x7f3558067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:11.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.597+0000 7f355d9fe700 1 --2- 172.21.15.46:0/164018783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f355815da50 0x7f355815deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:11.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.597+0000 7f354bfff700 1 -- 172.21.15.46:0/164018783 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770690643 0 0) 0x7f355806cb40 con 0x7f355806f750 2024-01-26T23:43:11.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.597+0000 7f354bfff700 1 -- 172.21.15.46:0/164018783 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3544003570 con 0x7f355806f750 2024-01-26T23:43:11.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.597+0000 7f355d1fd700 1 -- 172.21.15.46:0/164018783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f355815da50 msgr2=0x7f355815deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:11.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.597+0000 7f355d1fd700 1 --2- 172.21.15.46:0/164018783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f355815da50 0x7f355815deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.597+0000 7f355d1fd700 1 -- 172.21.15.46:0/164018783 >> v1:172.21.15.46:6789/0 conn(0x7f355806f750 legacy=0x7f355806cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:11.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.597+0000 7f355d1fd700 1 -- 172.21.15.46:0/164018783 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35580cbb70 con 0x7f3558067af0 2024-01-26T23:43:11.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.597+0000 7f355d9fe700 1 --2- 172.21.15.46:0/164018783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f355815da50 0x7f355815deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:43:11.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.597+0000 7f355d1fd700 1 --2- 172.21.15.46:0/164018783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3558067af0 0x7f3558067ec0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f354c0098e0 tx=0x7f354c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b428b5c0e8c912c7 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:11.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.597+0000 7f354bfff700 1 -- 172.21.15.46:0/164018783 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c011630 con 0x7f3558067af0 2024-01-26T23:43:11.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.597+0000 7f354bfff700 1 -- 172.21.15.46:0/164018783 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354c011790 con 0x7f3558067af0 2024-01-26T23:43:11.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.597+0000 7f354bfff700 1 -- 172.21.15.46:0/164018783 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c011b30 con 0x7f3558067af0 2024-01-26T23:43:11.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.598+0000 7f355f461700 1 -- 172.21.15.46:0/164018783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3558067af0 msgr2=0x7f3558067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:11.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.598+0000 7f355f461700 1 --2- 172.21.15.46:0/164018783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3558067af0 0x7f3558067ec0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f354c0098e0 tx=0x7f354c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.598+0000 7f355f461700 1 -- 172.21.15.46:0/164018783 shutdown_connections 2024-01-26T23:43:11.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.598+0000 7f355f461700 1 --2- 172.21.15.46:0/164018783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f355815da50 0x7f355815deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.598+0000 7f355f461700 1 --2- 172.21.15.46:0/164018783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3558067af0 0x7f3558067ec0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.598+0000 7f355f461700 1 -- 172.21.15.46:0/164018783 >> 172.21.15.46:0/164018783 conn(0x7f355806b850 msgr2=0x7f35580cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:11.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.598+0000 7f355f461700 1 -- 172.21.15.46:0/164018783 shutdown_connections 2024-01-26T23:43:11.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.598+0000 7f355f461700 1 -- 172.21.15.46:0/164018783 wait complete. 2024-01-26T23:43:11.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.600+0000 7f355f461700 1 Processor -- start 2024-01-26T23:43:11.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.600+0000 7f355f461700 1 -- start start 2024-01-26T23:43:11.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.601+0000 7f355f461700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3558067af0 0x7f355816e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:11.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.601+0000 7f355f461700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f355806f750 0x7f35580af930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:11.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.601+0000 7f355f461700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f355815da50 0x7f35580c7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.601+0000 7f355f461700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355816eec0 con 0x7f355815da50 2024-01-26T23:43:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.601+0000 7f355f461700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35580c6210 con 0x7f3558067af0 2024-01-26T23:43:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.601+0000 7f355f461700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35580afe70 con 0x7f355806f750 2024-01-26T23:43:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.601+0000 7f355d9fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f355815da50 0x7f35580c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.601+0000 7f355c9fc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f355806f750 0x7f35580af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.601+0000 7f355d9fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f355815da50 0x7f35580c7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46872/0 (socket says 172.21.15.46:46872) 2024-01-26T23:43:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.601+0000 7f355d1fd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3558067af0 0x7f355816e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.601+0000 7f355d9fe700 1 -- 172.21.15.46:0/489102126 learned_addr learned my addr 172.21.15.46:0/489102126 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.601+0000 7f355d9fe700 1 -- 172.21.15.46:0/489102126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f355806f750 msgr2=0x7f35580af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.601+0000 7f355d9fe700 1 --2- 172.21.15.46:0/489102126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f355806f750 0x7f35580af930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.601+0000 7f355d9fe700 1 -- 172.21.15.46:0/489102126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3558067af0 msgr2=0x7f355816e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.601+0000 7f355d9fe700 1 --2- 172.21.15.46:0/489102126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3558067af0 0x7f355816e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.601+0000 7f355d9fe700 1 -- 172.21.15.46:0/489102126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3550009a20 con 0x7f355815da50 2024-01-26T23:43:11.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.601+0000 7f355d9fe700 1 --2- 172.21.15.46:0/489102126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f355815da50 0x7f35580c7230 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3550009f20 tx=0x7f355000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:11.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.602+0000 7f35497fa700 1 -- 172.21.15.46:0/489102126 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355000cb70 con 0x7f355815da50 2024-01-26T23:43:11.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.602+0000 7f35497fa700 1 -- 172.21.15.46:0/489102126 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355000f040 con 0x7f355815da50 2024-01-26T23:43:11.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.602+0000 7f355f461700 1 -- 172.21.15.46:0/489102126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f354c009420 con 0x7f355815da50 2024-01-26T23:43:11.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.602+0000 7f35497fa700 1 -- 172.21.15.46:0/489102126 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3550014480 con 0x7f355815da50 2024-01-26T23:43:11.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.602+0000 7f355f461700 1 -- 172.21.15.46:0/489102126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35580c58e0 con 0x7f355815da50 2024-01-26T23:43:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.603+0000 7f355f461700 1 -- 172.21.15.46:0/489102126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355804c710 con 0x7f355815da50 2024-01-26T23:43:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.603+0000 7f35497fa700 1 -- 172.21.15.46:0/489102126 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3550003020 con 0x7f355815da50 2024-01-26T23:43:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.604+0000 7f35497fa700 1 --2- 172.21.15.46:0/489102126 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f353c066f30 0x7f353c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.604+0000 7f35497fa700 1 -- 172.21.15.46:0/489102126 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f35500850b0 con 0x7f355815da50 2024-01-26T23:43:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.604+0000 7f355d1fd700 1 --2- 172.21.15.46:0/489102126 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f353c066f30 0x7f353c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.604+0000 7f355d1fd700 1 --2- 172.21.15.46:0/489102126 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f353c066f30 0x7f353c0693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f354c0091e0 tx=0x7f354c017000 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.608+0000 7f35497fa700 1 -- 172.21.15.46:0/489102126 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3550052070 con 0x7f355815da50 2024-01-26T23:43:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.860+0000 7f355f461700 1 -- 172.21.15.46:0/489102126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35580cbe50 con 0x7f355815da50 2024-01-26T23:43:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.861+0000 7f35497fa700 1 -- 172.21.15.46:0/489102126 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4603 (secure 0 0 0) 0x7f3550055690 con 0x7f355815da50 2024-01-26T23:43:11.863 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:43:11.863 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3037972557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3037972557},{"type":"v1","addr":"172.21.15.106:6837","nonce":3037972557}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":5701,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3786017823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3786017823},{"type":"v1","addr":"172.21.15.106:6836","nonce":3786017823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":5704,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3329385210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3329385210},{"type":"v1","addr":"172.21.15.46:6835","nonce":3329385210}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:43:01.144129+0000","modified":"2024-01-26T23:43:04.718067+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6833/2801738403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2801738403},{"type":"v1","addr":"172.21.15.46:6833","nonce":2801738403}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2024-01-26T23:43:11.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.863+0000 7f355f461700 1 -- 172.21.15.46:0/489102126 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f353c066f30 msgr2=0x7f353c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:11.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.863+0000 7f355f461700 1 --2- 172.21.15.46:0/489102126 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f353c066f30 0x7f353c0693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f354c0091e0 tx=0x7f354c017000 comp rx=0 tx=0).stop 2024-01-26T23:43:11.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.863+0000 7f355f461700 1 -- 172.21.15.46:0/489102126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f355815da50 msgr2=0x7f35580c7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:11.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.863+0000 7f355f461700 1 --2- 172.21.15.46:0/489102126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f355815da50 0x7f35580c7230 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3550009f20 tx=0x7f355000b040 comp rx=0 tx=0).stop 2024-01-26T23:43:11.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.863+0000 7f355f461700 1 -- 172.21.15.46:0/489102126 shutdown_connections 2024-01-26T23:43:11.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.864+0000 7f355f461700 1 --2- 172.21.15.46:0/489102126 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f353c066f30 0x7f353c0693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.864+0000 7f355f461700 1 --2- 172.21.15.46:0/489102126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f355815da50 0x7f35580c7230 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.864+0000 7f355f461700 1 --2- 172.21.15.46:0/489102126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3558067af0 0x7f355816e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.864+0000 7f355f461700 1 --2- 172.21.15.46:0/489102126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f355806f750 0x7f35580af930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:11.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.864+0000 7f355f461700 1 -- 172.21.15.46:0/489102126 >> 172.21.15.46:0/489102126 conn(0x7f355806b850 msgr2=0x7f35580b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:11.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.864+0000 7f355f461700 1 -- 172.21.15.46:0/489102126 shutdown_connections 2024-01-26T23:43:11.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:11.864+0000 7f355f461700 1 -- 172.21.15.46:0/489102126 wait complete. 2024-01-26T23:43:11.866 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 27 2024-01-26T23:43:12.281 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-26T23:43:01.144129+0000', 'modified': '2024-01-26T23:43:04.718067+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5865}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5865': {'gid': 5865, 'name': 'c', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6833/2801738403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 2801738403}, {'type': 'v1', 'addr': '172.21.15.46:6833', 'nonce': 2801738403}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-26T23:43:12.282 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-26T23:43:12.282 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-26T23:43:12.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.423+0000 7f7974701700 1 Processor -- start 2024-01-26T23:43:12.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.423+0000 7f7974701700 1 -- start start 2024-01-26T23:43:12.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.424+0000 7f7974701700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f796c159910 0x7f796c159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.424+0000 7f7974701700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f796c068bf0 0x7f796c0cb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.424+0000 7f7974701700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f796c067930 0x7f796c067d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.424+0000 7f7974701700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f796c069340 con 0x7f796c068bf0 2024-01-26T23:43:12.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.424+0000 7f7974701700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f796c0af9d0 con 0x7f796c159910 2024-01-26T23:43:12.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.424+0000 7f7974701700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796c0cb770 con 0x7f796c067930 2024-01-26T23:43:12.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.424+0000 7f7971c9c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f796c068bf0 0x7f796c0cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.424+0000 7f7971c9c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f796c068bf0 0x7f796c0cb230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46878/0 (socket says 172.21.15.46:46878) 2024-01-26T23:43:12.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.424+0000 7f7971c9c700 1 -- 172.21.15.46:0/656373191 learned_addr learned my addr 172.21.15.46:0/656373191 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.424+0000 7f7972c9e700 1 --2- 172.21.15.46:0/656373191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f796c067930 0x7f796c067d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.424+0000 7f797249d700 1 --2- 172.21.15.46:0/656373191 >> v2:172.21.15.106:3300/0 conn(0x7f796c159910 0x7f796c159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.424+0000 7f7971c9c700 1 -- 172.21.15.46:0/656373191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f796c067930 msgr2=0x7f796c067d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.424+0000 7f7971c9c700 1 --2- 172.21.15.46:0/656373191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f796c067930 0x7f796c067d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.424+0000 7f7971c9c700 1 -- 172.21.15.46:0/656373191 >> v2:172.21.15.106:3300/0 conn(0x7f796c159910 msgr2=0x7f796c159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.424+0000 7f7971c9c700 1 --2- 172.21.15.46:0/656373191 >> v2:172.21.15.106:3300/0 conn(0x7f796c159910 0x7f796c159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.424+0000 7f7971c9c700 1 -- 172.21.15.46:0/656373191 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796c0c7e90 con 0x7f796c068bf0 2024-01-26T23:43:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.424+0000 7f7971c9c700 1 --2- 172.21.15.46:0/656373191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f796c068bf0 0x7f796c0cb230 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f796c06f800 tx=0x7f7968009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61382285d3e5f70e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.425+0000 7f797149b700 1 -- 172.21.15.46:0/656373191 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796800b860 con 0x7f796c068bf0 2024-01-26T23:43:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.425+0000 7f797149b700 1 -- 172.21.15.46:0/656373191 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796800b9c0 con 0x7f796c068bf0 2024-01-26T23:43:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.425+0000 7f797149b700 1 -- 172.21.15.46:0/656373191 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796800bd80 con 0x7f796c068bf0 2024-01-26T23:43:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.425+0000 7f7974701700 1 -- 172.21.15.46:0/656373191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f796c068bf0 msgr2=0x7f796c0cb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.425+0000 7f7974701700 1 --2- 172.21.15.46:0/656373191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f796c068bf0 0x7f796c0cb230 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f796c06f800 tx=0x7f7968009250 comp rx=0 tx=0).stop 2024-01-26T23:43:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.425+0000 7f7974701700 1 -- 172.21.15.46:0/656373191 shutdown_connections 2024-01-26T23:43:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.425+0000 7f7974701700 1 --2- 172.21.15.46:0/656373191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f796c067930 0x7f796c067d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.425+0000 7f7974701700 1 --2- 172.21.15.46:0/656373191 >> v2:172.21.15.106:3300/0 conn(0x7f796c159910 0x7f796c159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.425+0000 7f7974701700 1 --2- 172.21.15.46:0/656373191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f796c068bf0 0x7f796c0cb230 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.425+0000 7f7974701700 1 -- 172.21.15.46:0/656373191 >> 172.21.15.46:0/656373191 conn(0x7f796c0cd6e0 msgr2=0x7f796c0c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.425+0000 7f7974701700 1 -- 172.21.15.46:0/656373191 shutdown_connections 2024-01-26T23:43:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.425+0000 7f7974701700 1 -- 172.21.15.46:0/656373191 wait complete. 2024-01-26T23:43:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.426+0000 7f7974701700 1 Processor -- start 2024-01-26T23:43:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.427+0000 7f7974701700 1 -- start start 2024-01-26T23:43:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.427+0000 7f7974701700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f796c067930 0x7f796c0afb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.427+0000 7f7974701700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f796c068bf0 0x7f796c169b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.427+0000 7f7974701700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f796c159910 0x7f796c16a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.427+0000 7f7974701700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c0b14a0 con 0x7f796c067930 2024-01-26T23:43:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.427+0000 7f7974701700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c0b1010 con 0x7f796c068bf0 2024-01-26T23:43:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.427+0000 7f7974701700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796c0b0400 con 0x7f796c159910 2024-01-26T23:43:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.427+0000 7f797249d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f796c067930 0x7f796c0afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.427+0000 7f797249d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f796c067930 0x7f796c0afb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46886/0 (socket says 172.21.15.46:46886) 2024-01-26T23:43:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.427+0000 7f797249d700 1 -- 172.21.15.46:0/3186600893 learned_addr learned my addr 172.21.15.46:0/3186600893 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.427+0000 7f7971c9c700 1 --2- 172.21.15.46:0/3186600893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f796c068bf0 0x7f796c169b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.427+0000 7f7972c9e700 1 --2- 172.21.15.46:0/3186600893 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f796c159910 0x7f796c16a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.428+0000 7f797249d700 1 -- 172.21.15.46:0/3186600893 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f796c159910 msgr2=0x7f796c16a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.428+0000 7f797249d700 1 --2- 172.21.15.46:0/3186600893 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f796c159910 0x7f796c16a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.428+0000 7f797249d700 1 -- 172.21.15.46:0/3186600893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f796c068bf0 msgr2=0x7f796c169b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.428+0000 7f797249d700 1 --2- 172.21.15.46:0/3186600893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f796c068bf0 0x7f796c169b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.428+0000 7f797249d700 1 -- 172.21.15.46:0/3186600893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79680094f0 con 0x7f796c067930 2024-01-26T23:43:12.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.428+0000 7f797249d700 1 --2- 172.21.15.46:0/3186600893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f796c067930 0x7f796c0afb20 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f796000d8c0 tx=0x7f7960008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:12.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.428+0000 7f795e7fc700 1 -- 172.21.15.46:0/3186600893 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79600125f0 con 0x7f796c067930 2024-01-26T23:43:12.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.428+0000 7f795e7fc700 1 -- 172.21.15.46:0/3186600893 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7960012750 con 0x7f796c067930 2024-01-26T23:43:12.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.428+0000 7f7974701700 1 -- 172.21.15.46:0/3186600893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f796c0b06e0 con 0x7f796c067930 2024-01-26T23:43:12.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.428+0000 7f795e7fc700 1 -- 172.21.15.46:0/3186600893 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7960012af0 con 0x7f796c067930 2024-01-26T23:43:12.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.428+0000 7f7974701700 1 -- 172.21.15.46:0/3186600893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796c0b0ca0 con 0x7f796c067930 2024-01-26T23:43:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.430+0000 7f795e7fc700 1 -- 172.21.15.46:0/3186600893 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f796001e460 con 0x7f796c067930 2024-01-26T23:43:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.430+0000 7f795e7fc700 1 --2- 172.21.15.46:0/3186600893 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7950066f30 0x7f79500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.431+0000 7f795e7fc700 1 -- 172.21.15.46:0/3186600893 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f7960084d10 con 0x7f796c067930 2024-01-26T23:43:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.431+0000 7f7971c9c700 1 --2- 172.21.15.46:0/3186600893 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7950066f30 0x7f79500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.431+0000 7f7971c9c700 1 --2- 172.21.15.46:0/3186600893 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7950066f30 0x7f79500693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f796c06f800 tx=0x7f7968017000 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:12.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.433+0000 7f797da04b80 1 Processor -- start 2024-01-26T23:43:12.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.433+0000 7f797da04b80 1 -- start start 2024-01-26T23:43:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.433+0000 7f797da04b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65beb010 0x55af65beb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.433+0000 7f797da04b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65c33360 0x55af65c15840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.433+0000 7f797da04b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65bea5d0 0x55af65c15d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.433+0000 7f797da04b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af65aee9f0 con 0x55af65bea5d0 2024-01-26T23:43:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.433+0000 7f797da04b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af65b40130 con 0x55af65beb010 2024-01-26T23:43:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.433+0000 7f797da04b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55af65b3d930 con 0x55af65c33360 2024-01-26T23:43:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.434+0000 7f7972c9e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65bea5d0 0x55af65c15d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.434+0000 7f7972c9e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65bea5d0 0x55af65c15d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46896/0 (socket says 172.21.15.46:46896) 2024-01-26T23:43:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.434+0000 7f7972c9e700 1 -- 172.21.15.46:0/4170493776 learned_addr learned my addr 172.21.15.46:0/4170493776 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.434+0000 7f797249d700 1 --2- 172.21.15.46:0/4170493776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65beb010 0x55af65beb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.434+0000 7f7971c9c700 1 --2- 172.21.15.46:0/4170493776 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65c33360 0x55af65c15840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.434+0000 7f7972c9e700 1 -- 172.21.15.46:0/4170493776 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65c33360 msgr2=0x55af65c15840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.434+0000 7f7972c9e700 1 --2- 172.21.15.46:0/4170493776 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65c33360 0x55af65c15840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.434+0000 7f7972c9e700 1 -- 172.21.15.46:0/4170493776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65beb010 msgr2=0x55af65beb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.434+0000 7f7972c9e700 1 --2- 172.21.15.46:0/4170493776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65beb010 0x55af65beb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.434+0000 7f7972c9e700 1 -- 172.21.15.46:0/4170493776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af65978b10 con 0x55af65bea5d0 2024-01-26T23:43:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.434+0000 7f797249d700 1 --2- 172.21.15.46:0/4170493776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65beb010 0x55af65beb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:43:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.434+0000 7f7972c9e700 1 --2- 172.21.15.46:0/4170493776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65bea5d0 0x55af65c15d80 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f7964007370 tx=0x7f79640097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de931ce89e550b26 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.434+0000 7f794f7fe700 1 -- 172.21.15.46:0/4170493776 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7964007ea0 con 0x55af65bea5d0 2024-01-26T23:43:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.434+0000 7f794f7fe700 1 -- 172.21.15.46:0/4170493776 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7964015420 con 0x55af65bea5d0 2024-01-26T23:43:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.434+0000 7f794f7fe700 1 -- 172.21.15.46:0/4170493776 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79640157a0 con 0x55af65bea5d0 2024-01-26T23:43:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.435+0000 7f797da04b80 1 -- 172.21.15.46:0/4170493776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65bea5d0 msgr2=0x55af65c15d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.435+0000 7f797da04b80 1 --2- 172.21.15.46:0/4170493776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65bea5d0 0x55af65c15d80 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f7964007370 tx=0x7f79640097b0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.435+0000 7f797da04b80 1 -- 172.21.15.46:0/4170493776 shutdown_connections 2024-01-26T23:43:12.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.435+0000 7f797da04b80 1 --2- 172.21.15.46:0/4170493776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65bea5d0 0x55af65c15d80 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.435+0000 7f797da04b80 1 --2- 172.21.15.46:0/4170493776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65beb010 0x55af65beb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.435+0000 7f797da04b80 1 --2- 172.21.15.46:0/4170493776 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65c33360 0x55af65c15840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.435+0000 7f797da04b80 1 -- 172.21.15.46:0/4170493776 >> 172.21.15.46:0/4170493776 conn(0x55af65beaa70 msgr2=0x55af65c33820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:12.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.435+0000 7f797da04b80 1 -- 172.21.15.46:0/4170493776 shutdown_connections 2024-01-26T23:43:12.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.435+0000 7f797da04b80 1 -- 172.21.15.46:0/4170493776 wait complete. 2024-01-26T23:43:12.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.436+0000 7f797da04b80 1 -- start start 2024-01-26T23:43:12.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.436+0000 7f797da04b80 1 Processor -- start 2024-01-26T23:43:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.437+0000 7f797da04b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65bea5d0 0x55af65c2c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.437+0000 7f797da04b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65beb010 0x55af65be4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.437+0000 7f797da04b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65c33360 0x55af65be4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.437+0000 7f797da04b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af65b39720 con 0x55af65beb010 2024-01-26T23:43:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.437+0000 7f797da04b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af65ae51b0 con 0x55af65c33360 2024-01-26T23:43:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.437+0000 7f797da04b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55af65ae4e50 con 0x55af65bea5d0 2024-01-26T23:43:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.437+0000 7f797da04b80 10 client.0 _ll_register_callbacks cb 0x55af659477b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:43:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.437+0000 7f7971c9c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65beb010 0x55af65be4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.437+0000 7f7971c9c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65beb010 0x55af65be4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46898/0 (socket says 172.21.15.46:46898) 2024-01-26T23:43:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.437+0000 7f7971c9c700 1 -- 172.21.15.46:0/509330674 learned_addr learned my addr 172.21.15.46:0/509330674 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.437+0000 7f797249d700 1 --2- 172.21.15.46:0/509330674 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65bea5d0 0x55af65c2c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.437+0000 7f7972c9e700 1 --2- 172.21.15.46:0/509330674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65c33360 0x55af65be4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.437+0000 7f7971c9c700 1 -- 172.21.15.46:0/509330674 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65bea5d0 msgr2=0x55af65c2c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.437+0000 7f7971c9c700 1 --2- 172.21.15.46:0/509330674 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65bea5d0 0x55af65c2c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.437+0000 7f7971c9c700 1 -- 172.21.15.46:0/509330674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65c33360 msgr2=0x55af65be4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.437+0000 7f7971c9c700 1 --2- 172.21.15.46:0/509330674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65c33360 0x55af65be4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f7971c9c700 1 -- 172.21.15.46:0/509330674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796c0b0ca0 con 0x55af65beb010 2024-01-26T23:43:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f7972c9e700 1 --2- 172.21.15.46:0/509330674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65c33360 0x55af65be4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:43:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f7971c9c700 1 --2- 172.21.15.46:0/509330674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65beb010 0x55af65be4240 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f7968006fd0 tx=0x7f7968021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1aa5236238ab15 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f794cff9700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:43:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f794cff9700 1 -- 172.21.15.46:0/509330674 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796802c070 con 0x55af65beb010 2024-01-26T23:43:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f794cff9700 1 -- 172.21.15.46:0/509330674 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f7968011d70 con 0x55af65beb010 2024-01-26T23:43:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f797da04b80 1 -- 172.21.15.46:0/509330674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55af65bde190 con 0x55af65beb010 2024-01-26T23:43:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f794cff9700 1 -- 172.21.15.46:0/509330674 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7968026420 con 0x55af65beb010 2024-01-26T23:43:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f797da04b80 10 client.6027 fetch_fsmap learned FSMap version 27 2024-01-26T23:43:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f797da04b80 1 -- 172.21.15.46:0/509330674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x7f7964011040 con 0x55af65beb010 2024-01-26T23:43:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f794cff9700 1 -- 172.21.15.46:0/509330674 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55af65bde190 con 0x55af65beb010 2024-01-26T23:43:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f794cff9700 1 -- 172.21.15.46:0/509330674 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f7968026580 con 0x55af65beb010 2024-01-26T23:43:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f797da04b80 10 client.6027 fetch_fsmap finished waiting for FSMap version 27 2024-01-26T23:43:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f797da04b80 10 client.6027 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-26T23:43:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f797da04b80 20 client.6027 populate_metadata read hostname 'smithi046' 2024-01-26T23:43:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f797da04b80 1 --2- 172.21.15.46:0/509330674 >> [v2:172.21.15.46:6832/2801738403,v1:172.21.15.46:6833/2801738403] conn(0x55af65c41200 0x55af65c415d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f797da04b80 4 client.6027 mds_command: new command op to 5865 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-26T23:43:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.438+0000 7f797da04b80 1 -- 172.21.15.46:0/509330674 --> [v2:172.21.15.46:6832/2801738403,v1:172.21.15.46:6833/2801738403] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55af65adbee0 con 0x55af65c41200 2024-01-26T23:43:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.439+0000 7f797249d700 1 --2- 172.21.15.46:0/509330674 >> [v2:172.21.15.46:6832/2801738403,v1:172.21.15.46:6833/2801738403] conn(0x55af65c41200 0x55af65c415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.439+0000 7f794cff9700 10 client.6027 ms_handle_connect on v2:172.21.15.46:6832/2801738403 2024-01-26T23:43:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.439+0000 7f797249d700 1 --2- 172.21.15.46:0/509330674 >> [v2:172.21.15.46:6832/2801738403,v1:172.21.15.46:6833/2801738403] conn(0x55af65c41200 0x55af65c415d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b67fc69d239488c5 server_cookie=43188be674b0af8d in_seq=0 out_seq=0 2024-01-26T23:43:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.440+0000 7f794cff9700 1 -- 172.21.15.46:0/509330674 <== mds.0 v2:172.21.15.46:6832/2801738403 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x55af65adbee0 con 0x55af65c41200 2024-01-26T23:43:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.440+0000 7f794cff9700 10 client.6027 handle_command_reply: tid=0 2024-01-26T23:43:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.440+0000 7f797da04b80 1 client.6027 shutdown 2024-01-26T23:43:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.440+0000 7f797da04b80 1 -- 172.21.15.46:0/509330674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65beb010 msgr2=0x55af65be4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.440+0000 7f797da04b80 1 --2- 172.21.15.46:0/509330674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65beb010 0x55af65be4240 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f7968006fd0 tx=0x7f7968021000 comp rx=0 tx=0).stop 2024-01-26T23:43:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.440+0000 7f797da04b80 1 -- 172.21.15.46:0/509330674 shutdown_connections 2024-01-26T23:43:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.440+0000 7f797da04b80 1 --2- 172.21.15.46:0/509330674 >> [v2:172.21.15.46:6832/2801738403,v1:172.21.15.46:6833/2801738403] conn(0x55af65c41200 0x55af65c415d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.440+0000 7f797da04b80 1 --2- 172.21.15.46:0/509330674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65c33360 0x55af65be4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.440+0000 7f797da04b80 1 --2- 172.21.15.46:0/509330674 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65bea5d0 0x55af65c2c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.440+0000 7f794cff9700 0 client.6027 ms_handle_reset on v2:172.21.15.46:6832/2801738403 2024-01-26T23:43:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.440+0000 7f797da04b80 1 --2- 172.21.15.46:0/509330674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65beb010 0x55af65be4240 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.440+0000 7f797da04b80 1 -- 172.21.15.46:0/509330674 >> 172.21.15.46:0/509330674 conn(0x55af65beaa70 msgr2=0x55af65be5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.440+0000 7f797da04b80 1 -- 172.21.15.46:0/509330674 shutdown_connections 2024-01-26T23:43:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.440+0000 7f797da04b80 1 -- 172.21.15.46:0/509330674 wait complete. 2024-01-26T23:43:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.440+0000 7f797da04b80 20 client.6027 trim_cache size 0 max 16384 2024-01-26T23:43:12.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.458+0000 7f797da04b80 1 Processor -- start 2024-01-26T23:43:12.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.458+0000 7f797da04b80 1 -- start start 2024-01-26T23:43:12.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.459+0000 7f797da04b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65bea5d0 0x55af65c18d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.459+0000 7f797da04b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65beb010 0x55af65c2baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.459+0000 7f797da04b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65c33360 0x55af65be5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.459+0000 7f797da04b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af65a31e40 con 0x55af65c33360 2024-01-26T23:43:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.459+0000 7f797da04b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af65be3c50 con 0x55af65beb010 2024-01-26T23:43:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.459+0000 7f797da04b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55af65c2dbb0 con 0x55af65bea5d0 2024-01-26T23:43:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.459+0000 7f7972c9e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65c33360 0x55af65be5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.459+0000 7f7972c9e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65c33360 0x55af65be5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46910/0 (socket says 172.21.15.46:46910) 2024-01-26T23:43:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.459+0000 7f7972c9e700 1 -- 172.21.15.46:0/1253037227 learned_addr learned my addr 172.21.15.46:0/1253037227 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.459+0000 7f7972c9e700 1 -- 172.21.15.46:0/1253037227 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65bea5d0 msgr2=0x55af65c18d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.459+0000 7f7972c9e700 1 --2- 172.21.15.46:0/1253037227 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65bea5d0 0x55af65c18d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.459+0000 7f7972c9e700 1 -- 172.21.15.46:0/1253037227 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65beb010 msgr2=0x55af65c2baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.459+0000 7f7972c9e700 1 --2- 172.21.15.46:0/1253037227 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65beb010 0x55af65c2baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.459+0000 7f7972c9e700 1 -- 172.21.15.46:0/1253037227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af65bebe60 con 0x55af65c33360 2024-01-26T23:43:12.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.460+0000 7f7972c9e700 1 --2- 172.21.15.46:0/1253037227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65c33360 0x55af65be5ed0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f7964007340 tx=0x7f796401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe15f916b107a48 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:12.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.460+0000 7f794ffff700 1 -- 172.21.15.46:0/1253037227 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7964009ca0 con 0x55af65c33360 2024-01-26T23:43:12.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.460+0000 7f794ffff700 1 -- 172.21.15.46:0/1253037227 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7964000f40 con 0x55af65c33360 2024-01-26T23:43:12.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.460+0000 7f794ffff700 1 -- 172.21.15.46:0/1253037227 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79640157e0 con 0x55af65c33360 2024-01-26T23:43:12.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.460+0000 7f797da04b80 1 -- 172.21.15.46:0/1253037227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65c33360 msgr2=0x55af65be5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:12.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.460+0000 7f797da04b80 1 --2- 172.21.15.46:0/1253037227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65c33360 0x55af65be5ed0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f7964007340 tx=0x7f796401a000 comp rx=0 tx=0).stop 2024-01-26T23:43:12.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.460+0000 7f797da04b80 1 -- 172.21.15.46:0/1253037227 shutdown_connections 2024-01-26T23:43:12.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.460+0000 7f797da04b80 1 --2- 172.21.15.46:0/1253037227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65c33360 0x55af65be5ed0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.460+0000 7f797da04b80 1 --2- 172.21.15.46:0/1253037227 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65bea5d0 0x55af65c18d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.460+0000 7f797da04b80 1 --2- 172.21.15.46:0/1253037227 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65beb010 0x55af65c2baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.460+0000 7f797da04b80 1 -- 172.21.15.46:0/1253037227 >> 172.21.15.46:0/1253037227 conn(0x55af65beaa70 msgr2=0x55af65be9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:12.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.460+0000 7f797da04b80 1 -- 172.21.15.46:0/1253037227 shutdown_connections 2024-01-26T23:43:12.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.460+0000 7f797da04b80 1 -- 172.21.15.46:0/1253037227 wait complete. 2024-01-26T23:43:12.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.461+0000 7f797da04b80 1 -- start start 2024-01-26T23:43:12.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.461+0000 7f797da04b80 1 Processor -- start 2024-01-26T23:43:12.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.462+0000 7f797da04b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65bea5d0 0x55af65c32bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.462+0000 7f797da04b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65beb010 0x55af65c2a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.462+0000 7f797da04b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65c33360 0x55af65c97fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.462+0000 7f797da04b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af65c8c0b0 con 0x55af65beb010 2024-01-26T23:43:12.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.462+0000 7f797da04b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af65c97b90 con 0x55af65bea5d0 2024-01-26T23:43:12.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.462+0000 7f797da04b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55af65c8dc80 con 0x55af65c33360 2024-01-26T23:43:12.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.462+0000 7f797da04b80 10 client.0 _ll_register_callbacks cb 0x55af659477b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:43:12.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.462+0000 7f7971c9c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65beb010 0x55af65c2a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.462+0000 7f797249d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65bea5d0 0x55af65c32bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.462+0000 7f7971c9c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65beb010 0x55af65c2a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46922/0 (socket says 172.21.15.46:46922) 2024-01-26T23:43:12.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.462+0000 7f797249d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65bea5d0 0x55af65c32bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:36260/0 (socket says 172.21.15.46:36260) 2024-01-26T23:43:12.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.462+0000 7f7971c9c700 1 -- 172.21.15.46:0/2401950472 learned_addr learned my addr 172.21.15.46:0/2401950472 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:12.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.462+0000 7f7972c9e700 1 --2- 172.21.15.46:0/2401950472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65c33360 0x55af65c97fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f7971c9c700 1 -- 172.21.15.46:0/2401950472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65c33360 msgr2=0x55af65c97fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:12.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f7971c9c700 1 --2- 172.21.15.46:0/2401950472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65c33360 0x55af65c97fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f7971c9c700 1 -- 172.21.15.46:0/2401950472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65bea5d0 msgr2=0x55af65c32bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:12.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f7971c9c700 1 --2- 172.21.15.46:0/2401950472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65bea5d0 0x55af65c32bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f7971c9c700 1 -- 172.21.15.46:0/2401950472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af65978b10 con 0x55af65beb010 2024-01-26T23:43:12.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f797249d700 1 --2- 172.21.15.46:0/2401950472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65bea5d0 0x55af65c32bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:43:12.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f7971c9c700 1 --2- 172.21.15.46:0/2401950472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65beb010 0x55af65c2a5c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f79680090b0 tx=0x7f796800b050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f41e6af0218785ea server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:12.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f794cff9700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:43:12.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f794cff9700 1 -- 172.21.15.46:0/2401950472 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796802c040 con 0x55af65beb010 2024-01-26T23:43:12.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f797da04b80 1 -- 172.21.15.46:0/2401950472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f796c05e7d0 con 0x55af65beb010 2024-01-26T23:43:12.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f794cff9700 1 -- 172.21.15.46:0/2401950472 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f7968011d70 con 0x55af65beb010 2024-01-26T23:43:12.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f794cff9700 1 -- 172.21.15.46:0/2401950472 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f796801a420 con 0x55af65beb010 2024-01-26T23:43:12.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f797da04b80 10 client.6033 fetch_fsmap learned FSMap version 27 2024-01-26T23:43:12.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f797da04b80 1 -- 172.21.15.46:0/2401950472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x55af65b07290 con 0x55af65beb010 2024-01-26T23:43:12.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f794cff9700 1 -- 172.21.15.46:0/2401950472 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796c05e7d0 con 0x55af65beb010 2024-01-26T23:43:12.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f794cff9700 1 -- 172.21.15.46:0/2401950472 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f7968026420 con 0x55af65beb010 2024-01-26T23:43:12.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f797da04b80 10 client.6033 fetch_fsmap finished waiting for FSMap version 27 2024-01-26T23:43:12.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f797da04b80 10 client.6033 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-26T23:43:12.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f797da04b80 20 client.6033 populate_metadata read hostname 'smithi046' 2024-01-26T23:43:12.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f797da04b80 1 --2- 172.21.15.46:0/2401950472 >> [v2:172.21.15.46:6832/2801738403,v1:172.21.15.46:6833/2801738403] conn(0x55af65c8aaf0 0x55af65c8aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.463+0000 7f797da04b80 4 client.6033 mds_command: new command op to 5865 tid=0[{"prefix": "status"}] 2024-01-26T23:43:12.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.464+0000 7f797da04b80 1 -- 172.21.15.46:0/2401950472 --> [v2:172.21.15.46:6832/2801738403,v1:172.21.15.46:6833/2801738403] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f7964015420 con 0x55af65c8aaf0 2024-01-26T23:43:12.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.464+0000 7f797249d700 1 --2- 172.21.15.46:0/2401950472 >> [v2:172.21.15.46:6832/2801738403,v1:172.21.15.46:6833/2801738403] conn(0x55af65c8aaf0 0x55af65c8aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.464+0000 7f797249d700 1 --2- 172.21.15.46:0/2401950472 >> [v2:172.21.15.46:6832/2801738403,v1:172.21.15.46:6833/2801738403] conn(0x55af65c8aaf0 0x55af65c8aec0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10ac250ae8b7a8ef server_cookie=8bb9d976fe031c44 in_seq=0 out_seq=0 2024-01-26T23:43:12.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.464+0000 7f794cff9700 10 client.6033 ms_handle_connect on v2:172.21.15.46:6832/2801738403 2024-01-26T23:43:12.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.464+0000 7f794cff9700 1 -- 172.21.15.46:0/2401950472 <== mds.0 v2:172.21.15.46:6832/2801738403 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x7f7964015420 con 0x55af65c8aaf0 2024-01-26T23:43:12.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.464+0000 7f794cff9700 10 client.6033 handle_command_reply: tid=0 2024-01-26T23:43:12.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.464+0000 7f797da04b80 1 client.6033 shutdown 2024-01-26T23:43:12.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.464+0000 7f797da04b80 1 -- 172.21.15.46:0/2401950472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65beb010 msgr2=0x55af65c2a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:12.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.464+0000 7f797da04b80 1 --2- 172.21.15.46:0/2401950472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65beb010 0x55af65c2a5c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f79680090b0 tx=0x7f796800b050 comp rx=0 tx=0).stop 2024-01-26T23:43:12.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.464+0000 7f797da04b80 1 -- 172.21.15.46:0/2401950472 shutdown_connections 2024-01-26T23:43:12.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.464+0000 7f797da04b80 1 --2- 172.21.15.46:0/2401950472 >> [v2:172.21.15.46:6832/2801738403,v1:172.21.15.46:6833/2801738403] conn(0x55af65c8aaf0 0x55af65c8aec0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.464+0000 7f797da04b80 1 --2- 172.21.15.46:0/2401950472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55af65c33360 0x55af65c97fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.464+0000 7f797da04b80 1 --2- 172.21.15.46:0/2401950472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55af65bea5d0 0x55af65c32bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.464+0000 7f794cff9700 0 client.6033 ms_handle_reset on v2:172.21.15.46:6832/2801738403 2024-01-26T23:43:12.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.464+0000 7f797da04b80 1 --2- 172.21.15.46:0/2401950472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55af65beb010 0x55af65c2a5c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.464+0000 7f797da04b80 1 -- 172.21.15.46:0/2401950472 >> 172.21.15.46:0/2401950472 conn(0x55af65beaa70 msgr2=0x55af65c2ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:12.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.465+0000 7f797da04b80 1 -- 172.21.15.46:0/2401950472 shutdown_connections 2024-01-26T23:43:12.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.465+0000 7f797da04b80 1 -- 172.21.15.46:0/2401950472 wait complete. 2024-01-26T23:43:12.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.465+0000 7f797da04b80 20 client.6033 trim_cache size 0 max 16384 2024-01-26T23:43:12.472 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:43:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-26T23:43:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-26T23:43:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5865, 2024-01-26T23:43:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-26T23:43:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-26T23:43:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-01-26T23:43:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 10.764549758999999, 2024-01-26T23:43:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 27, 2024-01-26T23:43:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 70, 2024-01-26T23:43:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 70, 2024-01-26T23:43:12.473 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 15.729388416000001 2024-01-26T23:43:12.473 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:43:12.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.465+0000 7f7974701700 1 -- 172.21.15.46:0/3186600893 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7950066f30 msgr2=0x7f79500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:12.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.465+0000 7f7974701700 1 --2- 172.21.15.46:0/3186600893 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7950066f30 0x7f79500693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f796c06f800 tx=0x7f7968017000 comp rx=0 tx=0).stop 2024-01-26T23:43:12.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.465+0000 7f7974701700 1 -- 172.21.15.46:0/3186600893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f796c067930 msgr2=0x7f796c0afb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:12.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.465+0000 7f7974701700 1 --2- 172.21.15.46:0/3186600893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f796c067930 0x7f796c0afb20 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f796000d8c0 tx=0x7f7960008040 comp rx=0 tx=0).stop 2024-01-26T23:43:12.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.465+0000 7f7974701700 1 -- 172.21.15.46:0/3186600893 shutdown_connections 2024-01-26T23:43:12.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.465+0000 7f7974701700 1 --2- 172.21.15.46:0/3186600893 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7950066f30 0x7f79500693e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.465+0000 7f7974701700 1 --2- 172.21.15.46:0/3186600893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f796c067930 0x7f796c0afb20 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.465+0000 7f7974701700 1 --2- 172.21.15.46:0/3186600893 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f796c159910 0x7f796c16a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.465+0000 7f7974701700 1 --2- 172.21.15.46:0/3186600893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f796c068bf0 0x7f796c169b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.465+0000 7f7974701700 1 -- 172.21.15.46:0/3186600893 >> 172.21.15.46:0/3186600893 conn(0x7f796c0cd6e0 msgr2=0x7f796c0c97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:12.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.465+0000 7f7974701700 1 -- 172.21.15.46:0/3186600893 shutdown_connections 2024-01-26T23:43:12.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.465+0000 7f7974701700 1 -- 172.21.15.46:0/3186600893 wait complete. 2024-01-26T23:43:12.480 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-26T23:43:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.611+0000 7fb902ed4700 1 Processor -- start 2024-01-26T23:43:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.611+0000 7fb902ed4700 1 -- start start 2024-01-26T23:43:12.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.612+0000 7fb902ed4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb8fc159760 0x7fb8fc0b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.612+0000 7fb902ed4700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb8fc159b30 0x7fb8fc0b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.612+0000 7fb902ed4700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8fc15abc0 con 0x7fb8fc15afc0 2024-01-26T23:43:12.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.612+0000 7fb902ed4700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8fc0b2e90 con 0x7fb8fc159760 2024-01-26T23:43:12.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.612+0000 7fb902ed4700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8fc159fb0 con 0x7fb8fc159b30 2024-01-26T23:43:12.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.612+0000 7fb901471700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb8fc15afc0 0x7fb8fc067930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32840/0 (socket says 172.21.15.46:32840) 2024-01-26T23:43:12.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.612+0000 7fb900c70700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb8fc159760 0x7fb8fc0b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.612+0000 7fb901471700 1 -- 172.21.15.46:0/3082003909 learned_addr learned my addr 172.21.15.46:0/3082003909 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:43:12.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.612+0000 7fb8f3fff700 1 --2- 172.21.15.46:0/3082003909 >> v2:172.21.15.106:3300/0 conn(0x7fb8fc159b30 0x7fb8fc0b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.612+0000 7fb8f37fe700 1 -- 172.21.15.46:0/3082003909 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276661229 0 0) 0x7fb8fc15abc0 con 0x7fb8fc15afc0 2024-01-26T23:43:12.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.612+0000 7fb8f37fe700 1 -- 172.21.15.46:0/3082003909 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e4003570 con 0x7fb8fc15afc0 2024-01-26T23:43:12.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.612+0000 7fb900c70700 1 -- 172.21.15.46:0/3082003909 >> v2:172.21.15.106:3300/0 conn(0x7fb8fc159b30 msgr2=0x7fb8fc0b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:12.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.612+0000 7fb900c70700 1 --2- 172.21.15.46:0/3082003909 >> v2:172.21.15.106:3300/0 conn(0x7fb8fc159b30 0x7fb8fc0b11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.612+0000 7fb900c70700 1 -- 172.21.15.46:0/3082003909 >> v1:172.21.15.46:6789/0 conn(0x7fb8fc15afc0 legacy=0x7fb8fc067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:12.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.612+0000 7fb900c70700 1 -- 172.21.15.46:0/3082003909 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc15a290 con 0x7fb8fc159760 2024-01-26T23:43:12.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.612+0000 7fb900c70700 1 --2- 172.21.15.46:0/3082003909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8fc159760 0x7fb8fc0b0c70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc0b0970 tx=0x7fb8f80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b310925ebdb3b154 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:12.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.613+0000 7fb8f37fe700 1 -- 172.21.15.46:0/3082003909 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f8011630 con 0x7fb8fc159760 2024-01-26T23:43:12.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.613+0000 7fb8f37fe700 1 -- 172.21.15.46:0/3082003909 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f80117b0 con 0x7fb8fc159760 2024-01-26T23:43:12.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.613+0000 7fb8f37fe700 1 -- 172.21.15.46:0/3082003909 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f8011b70 con 0x7fb8fc159760 2024-01-26T23:43:12.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.613+0000 7fb902ed4700 1 -- 172.21.15.46:0/3082003909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8fc159760 msgr2=0x7fb8fc0b0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:12.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.613+0000 7fb902ed4700 1 --2- 172.21.15.46:0/3082003909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8fc159760 0x7fb8fc0b0c70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc0b0970 tx=0x7fb8f80091a0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.613+0000 7fb902ed4700 1 -- 172.21.15.46:0/3082003909 shutdown_connections 2024-01-26T23:43:12.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.613+0000 7fb902ed4700 1 --2- 172.21.15.46:0/3082003909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8fc159760 0x7fb8fc0b0c70 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.613+0000 7fb902ed4700 1 --2- 172.21.15.46:0/3082003909 >> v2:172.21.15.106:3300/0 conn(0x7fb8fc159b30 0x7fb8fc0b11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.613+0000 7fb902ed4700 1 -- 172.21.15.46:0/3082003909 >> 172.21.15.46:0/3082003909 conn(0x7fb8fc06b630 msgr2=0x7fb8fc0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:12.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.613+0000 7fb902ed4700 1 -- 172.21.15.46:0/3082003909 shutdown_connections 2024-01-26T23:43:12.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.613+0000 7fb902ed4700 1 -- 172.21.15.46:0/3082003909 wait complete. 2024-01-26T23:43:12.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.614+0000 7fb902ed4700 1 Processor -- start 2024-01-26T23:43:12.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.615+0000 7fb902ed4700 1 -- start start 2024-01-26T23:43:12.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.615+0000 7fb902ed4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8fc159760 0x7fb8fc14d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb902ed4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8fc15afc0 0x7fb8fc13ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb902ed4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8fc1695c0 0x7fb8fc0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb902ed4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc1517f0 con 0x7fb8fc1695c0 2024-01-26T23:43:12.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb902ed4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc1538e0 con 0x7fb8fc159760 2024-01-26T23:43:12.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb902ed4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc151510 con 0x7fb8fc15afc0 2024-01-26T23:43:12.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb901471700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8fc1695c0 0x7fb8fc0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb901471700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8fc1695c0 0x7fb8fc0ae5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46938/0 (socket says 172.21.15.46:46938) 2024-01-26T23:43:12.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb901471700 1 -- 172.21.15.46:0/3005362361 learned_addr learned my addr 172.21.15.46:0/3005362361 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:12.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb900c70700 1 --2- 172.21.15.46:0/3005362361 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8fc159760 0x7fb8fc14d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb8f3fff700 1 --2- 172.21.15.46:0/3005362361 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8fc15afc0 0x7fb8fc13ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb901471700 1 -- 172.21.15.46:0/3005362361 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8fc15afc0 msgr2=0x7fb8fc13ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:12.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb901471700 1 --2- 172.21.15.46:0/3005362361 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8fc15afc0 0x7fb8fc13ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb901471700 1 -- 172.21.15.46:0/3005362361 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8fc159760 msgr2=0x7fb8fc14d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:12.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb901471700 1 --2- 172.21.15.46:0/3005362361 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8fc159760 0x7fb8fc14d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb901471700 1 -- 172.21.15.46:0/3005362361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f8009420 con 0x7fb8fc1695c0 2024-01-26T23:43:12.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb901471700 1 --2- 172.21.15.46:0/3005362361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8fc1695c0 0x7fb8fc0ae5a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4001460 tx=0x7fb8f4001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:12.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb8f0ff9700 1 -- 172.21.15.46:0/3005362361 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f400a890 con 0x7fb8fc1695c0 2024-01-26T23:43:12.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb8f0ff9700 1 -- 172.21.15.46:0/3005362361 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f400a9f0 con 0x7fb8fc1695c0 2024-01-26T23:43:12.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb902ed4700 1 -- 172.21.15.46:0/3005362361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8fc153290 con 0x7fb8fc1695c0 2024-01-26T23:43:12.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.616+0000 7fb8f0ff9700 1 -- 172.21.15.46:0/3005362361 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f400ad70 con 0x7fb8fc1695c0 2024-01-26T23:43:12.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.617+0000 7fb902ed4700 1 -- 172.21.15.46:0/3005362361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8fc14d6c0 con 0x7fb8fc1695c0 2024-01-26T23:43:12.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.618+0000 7fb902ed4700 1 -- 172.21.15.46:0/3005362361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8fc1530f0 con 0x7fb8fc1695c0 2024-01-26T23:43:12.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.618+0000 7fb8f0ff9700 1 -- 172.21.15.46:0/3005362361 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb8f4007300 con 0x7fb8fc1695c0 2024-01-26T23:43:12.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.618+0000 7fb8f0ff9700 1 --2- 172.21.15.46:0/3005362361 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8e8066f30 0x7fb8e80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:12.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.618+0000 7fb8f0ff9700 1 -- 172.21.15.46:0/3005362361 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fb8f4084b80 con 0x7fb8fc1695c0 2024-01-26T23:43:12.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.619+0000 7fb900c70700 1 --2- 172.21.15.46:0/3005362361 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8e8066f30 0x7fb8e80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:12.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.619+0000 7fb900c70700 1 --2- 172.21.15.46:0/3005362361 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8e8066f30 0x7fb8e80693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc0b0970 tx=0x7fb8f8017000 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:12.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.623+0000 7fb8f0ff9700 1 -- 172.21.15.46:0/3005362361 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb8fc1530f0 con 0x7fb8fc1695c0 2024-01-26T23:43:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.878+0000 7fb902ed4700 1 -- 172.21.15.46:0/3005362361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb8fc1530f0 con 0x7fb8fc1695c0 2024-01-26T23:43:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.879+0000 7fb8f0ff9700 1 -- 172.21.15.46:0/3005362361 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb8fc1530f0 con 0x7fb8fc1695c0 2024-01-26T23:43:12.880 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-26T23:43:12.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.881+0000 7fb902ed4700 1 -- 172.21.15.46:0/3005362361 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8e8066f30 msgr2=0x7fb8e80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:12.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.881+0000 7fb902ed4700 1 --2- 172.21.15.46:0/3005362361 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8e8066f30 0x7fb8e80693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc0b0970 tx=0x7fb8f8017000 comp rx=0 tx=0).stop 2024-01-26T23:43:12.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.881+0000 7fb902ed4700 1 -- 172.21.15.46:0/3005362361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8fc1695c0 msgr2=0x7fb8fc0ae5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:12.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.881+0000 7fb902ed4700 1 --2- 172.21.15.46:0/3005362361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8fc1695c0 0x7fb8fc0ae5a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4001460 tx=0x7fb8f4001ae0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.881+0000 7fb902ed4700 1 -- 172.21.15.46:0/3005362361 shutdown_connections 2024-01-26T23:43:12.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.881+0000 7fb902ed4700 1 --2- 172.21.15.46:0/3005362361 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8e8066f30 0x7fb8e80693e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.881+0000 7fb902ed4700 1 --2- 172.21.15.46:0/3005362361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8fc1695c0 0x7fb8fc0ae5a0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.881+0000 7fb902ed4700 1 --2- 172.21.15.46:0/3005362361 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8fc159760 0x7fb8fc14d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.881+0000 7fb902ed4700 1 --2- 172.21.15.46:0/3005362361 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8fc15afc0 0x7fb8fc13ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:12.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.881+0000 7fb902ed4700 1 -- 172.21.15.46:0/3005362361 >> 172.21.15.46:0/3005362361 conn(0x7fb8fc06b630 msgr2=0x7fb8fc1569b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:12.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.881+0000 7fb902ed4700 1 -- 172.21.15.46:0/3005362361 shutdown_connections 2024-01-26T23:43:12.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:12.881+0000 7fb902ed4700 1 -- 172.21.15.46:0/3005362361 wait complete. 2024-01-26T23:43:12.896 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-26T23:43:12.896 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-26T23:43:12.896 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-26T23:43:12.896 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-26T23:43:12.896 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-26T23:43:12.896 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-26T23:43:12.896 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-26T23:43:12.897 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-26T23:43:12.897 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:43:12.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:43:12.897 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-26T23:43:12.916 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-26T23:43:12.916 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-26T23:43:12.916 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-26T23:43:12.916 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:43:12.916 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-26T23:43:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:43:12.917 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:43:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:43:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-26T23:43:12.917 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:43:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:43:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-26T23:43:12.917 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-26T23:43:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:43:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-26T23:43:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-26T23:43:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 41538sec preferred_lft 41538sec 2024-01-26T23:43:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-26T23:43:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:43:12.917 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:43:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:43:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-26T23:43:12.918 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-26T23:43:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:43:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-26T23:43:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:43:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::cce6:a7ff:fe4f:38d/64 scope link 2024-01-26T23:43:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:43:12.918 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:43:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 76:ef:90:4c:9d:6f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-26T23:43:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::74ef:90ff:fe4c:9d6f/64 scope link 2024-01-26T23:43:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:43:12.918 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:43:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-26T23:43:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::2c73:afff:fef0:6f39/64 scope link 2024-01-26T23:43:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:43:12.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:43:12.919 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-26T23:43:12.986 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-26T23:43:12.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:43:12.988 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-26T23:43:13.052 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-26T23:43:13.053 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-26T23:43:13.053 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-26T23:43:13.053 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-26T23:43:13.053 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-26T23:43:13.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:43:13.053 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:43:13.053 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:43:13.053 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-26T23:43:13.053 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-26T23:43:13.054 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-26T23:43:13.054 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:43:13.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:43:13.178 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-26T23:43:13.198 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-26T23:43:13.199 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-26T23:43:13.199 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:43:13.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:43:13.199 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-01-26T23:43:13.274 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-26T23:43:13.274 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:43:13.274 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-26T23:43:13.336 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:43:13.336 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:43:13.337 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:43:13.399 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:43:13.475 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:43:13.475 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:43:13.549 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:43:13.550 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:43:13.551 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:13.551 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:43:13.613 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-26T23:43:13.613 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-26T23:43:13.657 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:43:13.692 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:43:13.692 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:43:13.759 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-01-26T23:43:13.758+0000 7f824db4f540 -1 init, newargv = 0x55fbfcbfe1d0 newargc=15 2024-01-26T23:43:13.759 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[67642]: starting ceph client 2024-01-26T23:43:13.764 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:43:13.765 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[67642]: starting fuse 2024-01-26T23:43:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:43:13.766 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:13.766 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:43:13.827 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-26T23:43:13.827 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-26T23:43:13.827 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-26T23:43:13.827 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-26T23:43:13.827 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-26T23:43:13.827 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-26T23:43:13.827 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-26T23:43:13.827 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:43:13.828 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-26T23:43:13.828 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-26T23:43:13.828 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-26T23:43:13.828 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-26T23:43:13.828 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:43:13.828 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-26T23:43:13.828 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-26T23:43:13.828 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-26T23:43:13.828 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-26T23:43:13.828 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:43:13.828 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-26T23:43:13.828 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-26T23:43:13.828 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-26T23:43:13.828 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-26T23:43:13.828 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-26T23:43:13.829 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-26T23:43:13.829 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-26T23:43:13.829 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-26T23:43:13.829 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:43:13.829 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-01-26T23:43:13.829 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-26T23:43:13.969 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.67642.asok 2024-01-26T23:43:14.151 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.67642.asok 2024-01-26T23:43:14.151 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:14.151 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.67642.asok status 2024-01-26T23:43:14.328 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:43:14.328 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-26T23:43:14.328 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-26T23:43:14.328 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-26T23:43:14.328 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-01-26T23:43:14.328 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-26T23:43:14.328 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-26T23:43:14.329 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "67642", 2024-01-26T23:43:14.329 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-26T23:43:14.329 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:43:14.329 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-26T23:43:14.329 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-26T23:43:14.329 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 6048, 2024-01-26T23:43:14.329 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-26T23:43:14.329 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-26T23:43:14.329 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-26T23:43:14.329 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6048 2024-01-26T23:43:14.329 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:43:14.329 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:43:14.329 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:43:14.329 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:43:14.329 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 4272982984 2024-01-26T23:43:14.330 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:43:14.330 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:43:14.330 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:43:14.330 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:43:14.330 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:43:14.330 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 4272982984 2024-01-26T23:43:14.330 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:43:14.330 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.6048 192.168.0.1:0/4272982984", 2024-01-26T23:43:14.330 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/4272982984", 2024-01-26T23:43:14.330 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-26T23:43:14.330 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 27, 2024-01-26T23:43:14.330 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 70, 2024-01-26T23:43:14.330 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-26T23:43:14.330 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-26T23:43:14.330 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-26T23:43:14.330 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:43:14.339 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:14.339 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-26T23:43:14.339 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-26T23:43:14.358 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-26T23:43:14.358 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-26T23:43:14.359 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:14.359 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:43:14.432 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-26T23:43:14.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.611+0000 7f41ab9a9700 1 Processor -- start 2024-01-26T23:43:14.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.612+0000 7f41ab9a9700 1 -- start start 2024-01-26T23:43:14.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.613+0000 7f41ab9a9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f41a415ce40 0x7f41a40d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:14.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.613+0000 7f41ab9a9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41a40cc1c0 0x7f41a40ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:14.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.613+0000 7f41ab9a9700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a40cff70 con 0x7f41a415d210 2024-01-26T23:43:14.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.613+0000 7f41ab9a9700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41a40cf400 con 0x7f41a415ce40 2024-01-26T23:43:14.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.613+0000 7f41ab9a9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a40cf660 con 0x7f41a40cc1c0 2024-01-26T23:43:14.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.613+0000 7f41a9f46700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41a40cc1c0 0x7f41a40ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:14.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.613+0000 7f41a8f44700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f41a415d210 0x7f41a40cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:51196/0 (socket says 172.21.15.46:51196) 2024-01-26T23:43:14.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.613+0000 7f41a8f44700 1 -- 172.21.15.46:0/713169767 learned_addr learned my addr 172.21.15.46:0/713169767 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:43:14.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.613+0000 7f41a9745700 1 --2- 172.21.15.46:0/713169767 >> v2:172.21.15.106:3300/0 conn(0x7f41a415ce40 0x7f41a40d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:14.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.613+0000 7f41a9f46700 1 -- 172.21.15.46:0/713169767 >> v2:172.21.15.106:3300/0 conn(0x7f41a415ce40 msgr2=0x7f41a40d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:14.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.613+0000 7f41a9f46700 1 --2- 172.21.15.46:0/713169767 >> v2:172.21.15.106:3300/0 conn(0x7f41a415ce40 0x7f41a40d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:14.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.613+0000 7f41a9f46700 1 -- 172.21.15.46:0/713169767 >> v1:172.21.15.106:6789/0 conn(0x7f41a415d210 legacy=0x7f41a40cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:14.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.613+0000 7f41a9f46700 1 -- 172.21.15.46:0/713169767 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a40d0500 con 0x7f41a40cc1c0 2024-01-26T23:43:14.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.614+0000 7f41a9f46700 1 --2- 172.21.15.46:0/713169767 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41a40cc1c0 0x7f41a40ade30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f419c009e60 tx=0x7f419c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fea6b21ae97cd68 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:14.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.614+0000 7f419bfff700 1 -- 172.21.15.46:0/713169767 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c00e070 con 0x7f41a40cc1c0 2024-01-26T23:43:14.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.614+0000 7f419bfff700 1 -- 172.21.15.46:0/713169767 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419c00bb60 con 0x7f41a40cc1c0 2024-01-26T23:43:14.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.614+0000 7f419bfff700 1 -- 172.21.15.46:0/713169767 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c012430 con 0x7f41a40cc1c0 2024-01-26T23:43:14.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.615+0000 7f41ab9a9700 1 -- 172.21.15.46:0/713169767 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41a40cc1c0 msgr2=0x7f41a40ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:14.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.615+0000 7f41ab9a9700 1 --2- 172.21.15.46:0/713169767 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41a40cc1c0 0x7f41a40ade30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f419c009e60 tx=0x7f419c009780 comp rx=0 tx=0).stop 2024-01-26T23:43:14.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.615+0000 7f41ab9a9700 1 -- 172.21.15.46:0/713169767 shutdown_connections 2024-01-26T23:43:14.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.615+0000 7f41ab9a9700 1 --2- 172.21.15.46:0/713169767 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41a40cc1c0 0x7f41a40ade30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:14.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.615+0000 7f41ab9a9700 1 --2- 172.21.15.46:0/713169767 >> v2:172.21.15.106:3300/0 conn(0x7f41a415ce40 0x7f41a40d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:14.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.615+0000 7f41ab9a9700 1 -- 172.21.15.46:0/713169767 >> 172.21.15.46:0/713169767 conn(0x7f41a406f840 msgr2=0x7f41a40d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:14.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.615+0000 7f41ab9a9700 1 -- 172.21.15.46:0/713169767 shutdown_connections 2024-01-26T23:43:14.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.615+0000 7f41ab9a9700 1 -- 172.21.15.46:0/713169767 wait complete. 2024-01-26T23:43:14.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.617+0000 7f41ab9a9700 1 Processor -- start 2024-01-26T23:43:14.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.617+0000 7f41ab9a9700 1 -- start start 2024-01-26T23:43:14.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.618+0000 7f41ab9a9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41a40cc1c0 0x7f41a416c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:14.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.618+0000 7f41ab9a9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41a415ce40 0x7f41a415f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:14.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.618+0000 7f41ab9a9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a415d210 0x7f41a415fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:14.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.618+0000 7f41ab9a9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a416d6b0 con 0x7f41a415d210 2024-01-26T23:43:14.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.618+0000 7f41ab9a9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a416d990 con 0x7f41a40cc1c0 2024-01-26T23:43:14.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.618+0000 7f41ab9a9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a416dc70 con 0x7f41a415ce40 2024-01-26T23:43:14.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.618+0000 7f41a9f46700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a415d210 0x7f41a415fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:14.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.618+0000 7f41a9f46700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a415d210 0x7f41a415fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46940/0 (socket says 172.21.15.46:46940) 2024-01-26T23:43:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.618+0000 7f41a9f46700 1 -- 172.21.15.46:0/4284911199 learned_addr learned my addr 172.21.15.46:0/4284911199 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.618+0000 7f41a9745700 1 --2- 172.21.15.46:0/4284911199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41a40cc1c0 0x7f41a416c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.618+0000 7f41a8f44700 1 --2- 172.21.15.46:0/4284911199 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41a415ce40 0x7f41a415f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.618+0000 7f41a9f46700 1 -- 172.21.15.46:0/4284911199 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41a415ce40 msgr2=0x7f41a415f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.618+0000 7f41a9f46700 1 --2- 172.21.15.46:0/4284911199 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41a415ce40 0x7f41a415f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.618+0000 7f41a9f46700 1 -- 172.21.15.46:0/4284911199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41a40cc1c0 msgr2=0x7f41a416c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.618+0000 7f41a9f46700 1 --2- 172.21.15.46:0/4284911199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41a40cc1c0 0x7f41a416c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.618+0000 7f41a9f46700 1 -- 172.21.15.46:0/4284911199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c009a20 con 0x7f41a415d210 2024-01-26T23:43:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.618+0000 7f41a9f46700 1 --2- 172.21.15.46:0/4284911199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a415d210 0x7f41a415fce0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f419c013000 tx=0x7f419c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.619+0000 7f41997fa700 1 -- 172.21.15.46:0/4284911199 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c012ea0 con 0x7f41a415d210 2024-01-26T23:43:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.619+0000 7f41997fa700 1 -- 172.21.15.46:0/4284911199 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419c027470 con 0x7f41a415d210 2024-01-26T23:43:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.619+0000 7f41997fa700 1 -- 172.21.15.46:0/4284911199 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c027850 con 0x7f41a415d210 2024-01-26T23:43:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.619+0000 7f41ab9a9700 1 -- 172.21.15.46:0/4284911199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a416df50 con 0x7f41a415d210 2024-01-26T23:43:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.619+0000 7f41ab9a9700 1 -- 172.21.15.46:0/4284911199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a415bf80 con 0x7f41a415d210 2024-01-26T23:43:14.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.620+0000 7f41ab9a9700 1 -- 172.21.15.46:0/4284911199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a404c710 con 0x7f41a415d210 2024-01-26T23:43:14.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.620+0000 7f41997fa700 1 -- 172.21.15.46:0/4284911199 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f419c024070 con 0x7f41a415d210 2024-01-26T23:43:14.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.621+0000 7f41997fa700 1 --2- 172.21.15.46:0/4284911199 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4188066f30 0x7f41880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:14.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.621+0000 7f41997fa700 1 -- 172.21.15.46:0/4284911199 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f419c087210 con 0x7f41a415d210 2024-01-26T23:43:14.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.621+0000 7f41a9745700 1 --2- 172.21.15.46:0/4284911199 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4188066f30 0x7f41880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:14.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.621+0000 7f41a9745700 1 --2- 172.21.15.46:0/4284911199 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4188066f30 0x7f41880693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f41a0009da0 tx=0x7f41a0009120 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:14.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.625+0000 7f41997fa700 1 -- 172.21.15.46:0/4284911199 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f419c0541d0 con 0x7f41a415d210 2024-01-26T23:43:14.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.863+0000 7f41ab9a9700 1 -- 172.21.15.46:0/4284911199 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f41a40cf920 con 0x7f4188066f30 2024-01-26T23:43:14.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.865+0000 7f41997fa700 1 -- 172.21.15.46:0/4284911199 <== mgr.5267 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f41a40cf920 con 0x7f4188066f30 2024-01-26T23:43:14.866 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-26T23:43:14.866 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:43:14.866 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephfs" 2024-01-26T23:43:14.866 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:43:14.866 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-26T23:43:14.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.867+0000 7f41ab9a9700 1 -- 172.21.15.46:0/4284911199 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4188066f30 msgr2=0x7f41880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:14.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.867+0000 7f41ab9a9700 1 --2- 172.21.15.46:0/4284911199 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4188066f30 0x7f41880693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f41a0009da0 tx=0x7f41a0009120 comp rx=0 tx=0).stop 2024-01-26T23:43:14.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.867+0000 7f41ab9a9700 1 -- 172.21.15.46:0/4284911199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a415d210 msgr2=0x7f41a415fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:14.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.867+0000 7f41ab9a9700 1 --2- 172.21.15.46:0/4284911199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a415d210 0x7f41a415fce0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f419c013000 tx=0x7f419c019000 comp rx=0 tx=0).stop 2024-01-26T23:43:14.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.867+0000 7f41ab9a9700 1 -- 172.21.15.46:0/4284911199 shutdown_connections 2024-01-26T23:43:14.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.867+0000 7f41ab9a9700 1 --2- 172.21.15.46:0/4284911199 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4188066f30 0x7f41880693e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:14.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.867+0000 7f41ab9a9700 1 --2- 172.21.15.46:0/4284911199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a415d210 0x7f41a415fce0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:14.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.867+0000 7f41ab9a9700 1 --2- 172.21.15.46:0/4284911199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41a40cc1c0 0x7f41a416c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:14.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.867+0000 7f41ab9a9700 1 --2- 172.21.15.46:0/4284911199 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41a415ce40 0x7f41a415f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:14.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.867+0000 7f41ab9a9700 1 -- 172.21.15.46:0/4284911199 >> 172.21.15.46:0/4284911199 conn(0x7f41a406f840 msgr2=0x7f41a40c5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:14.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.868+0000 7f41ab9a9700 1 -- 172.21.15.46:0/4284911199 shutdown_connections 2024-01-26T23:43:14.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:14.868+0000 7f41ab9a9700 1 -- 172.21.15.46:0/4284911199 wait complete. 2024-01-26T23:43:14.882 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-26T23:43:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.014+0000 7f93427b0700 1 Processor -- start 2024-01-26T23:43:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.014+0000 7f93427b0700 1 -- start start 2024-01-26T23:43:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.015+0000 7f93427b0700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f933c141200 0x7f933c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.015+0000 7f93427b0700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c0ccc00 con 0x7f933c0c8da0 2024-01-26T23:43:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.015+0000 7f93427b0700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c0cc310 con 0x7f933c0caf30 2024-01-26T23:43:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.015+0000 7f93427b0700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f933c0cbbf0 con 0x7f933c141200 2024-01-26T23:43:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.015+0000 7f933bfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f933c0c8da0 0x7f933c140af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32846/0 (socket says 172.21.15.46:32846) 2024-01-26T23:43:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.015+0000 7f933bfff700 1 -- 172.21.15.46:0/3593995690 learned_addr learned my addr 172.21.15.46:0/3593995690 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:43:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.015+0000 7f933b7fe700 1 --2- 172.21.15.46:0/3593995690 >> v2:172.21.15.106:3300/0 conn(0x7f933c141200 0x7f933c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.016+0000 7f933affd700 1 -- 172.21.15.46:0/3593995690 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684189260 0 0) 0x7f933c0ccc00 con 0x7f933c0c8da0 2024-01-26T23:43:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.016+0000 7f933affd700 1 -- 172.21.15.46:0/3593995690 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9328003570 con 0x7f933c0c8da0 2024-01-26T23:43:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.016+0000 7f933affd700 1 -- 172.21.15.46:0/3593995690 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4005329013 0 0) 0x7f9328003570 con 0x7f933c0c8da0 2024-01-26T23:43:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.016+0000 7f933b7fe700 1 -- 172.21.15.46:0/3593995690 >> v1:172.21.15.106:6789/0 conn(0x7f933c0caf30 legacy=0x7f933c0cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.016+0000 7f933b7fe700 1 -- 172.21.15.46:0/3593995690 >> v1:172.21.15.46:6789/0 conn(0x7f933c0c8da0 legacy=0x7f933c140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.016+0000 7f933b7fe700 1 -- 172.21.15.46:0/3593995690 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c0cc5b0 con 0x7f933c141200 2024-01-26T23:43:15.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.016+0000 7f933b7fe700 1 --2- 172.21.15.46:0/3593995690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f933c141200 0x7f933c0c9170 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f93240098e0 tx=0x7f93240091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0029008b074bdb0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:15.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.016+0000 7f933affd700 1 -- 172.21.15.46:0/3593995690 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c001470 con 0x7f933c141200 2024-01-26T23:43:15.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.016+0000 7f933affd700 1 -- 172.21.15.46:0/3593995690 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9324011420 con 0x7f933c141200 2024-01-26T23:43:15.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.016+0000 7f933affd700 1 -- 172.21.15.46:0/3593995690 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93240117c0 con 0x7f933c141200 2024-01-26T23:43:15.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.017+0000 7f93427b0700 1 -- 172.21.15.46:0/3593995690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f933c141200 msgr2=0x7f933c0c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:15.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.017+0000 7f93427b0700 1 --2- 172.21.15.46:0/3593995690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f933c141200 0x7f933c0c9170 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f93240098e0 tx=0x7f93240091a0 comp rx=0 tx=0).stop 2024-01-26T23:43:15.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.017+0000 7f93427b0700 1 -- 172.21.15.46:0/3593995690 shutdown_connections 2024-01-26T23:43:15.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.017+0000 7f93427b0700 1 --2- 172.21.15.46:0/3593995690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f933c141200 0x7f933c0c9170 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:15.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.017+0000 7f93427b0700 1 -- 172.21.15.46:0/3593995690 >> 172.21.15.46:0/3593995690 conn(0x7f933c06bd90 msgr2=0x7f933c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:15.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.017+0000 7f93427b0700 1 -- 172.21.15.46:0/3593995690 shutdown_connections 2024-01-26T23:43:15.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.017+0000 7f93427b0700 1 -- 172.21.15.46:0/3593995690 wait complete. 2024-01-26T23:43:15.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.018+0000 7f93427b0700 1 Processor -- start 2024-01-26T23:43:15.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.018+0000 7f93427b0700 1 -- start start 2024-01-26T23:43:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.019+0000 7f93427b0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f933c0c8da0 0x7f933c0c5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.019+0000 7f93427b0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f933c0caf30 0x7f933c167ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.019+0000 7f93427b0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f933c141200 0x7f933c13c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.019+0000 7f93427b0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c0c8780 con 0x7f933c0caf30 2024-01-26T23:43:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.019+0000 7f93427b0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c168410 con 0x7f933c141200 2024-01-26T23:43:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.019+0000 7f93427b0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933c0b2d70 con 0x7f933c0c8da0 2024-01-26T23:43:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.019+0000 7f933b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f933c0caf30 0x7f933c167ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.019+0000 7f933b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f933c0caf30 0x7f933c167ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46944/0 (socket says 172.21.15.46:46944) 2024-01-26T23:43:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.019+0000 7f933b7fe700 1 -- 172.21.15.46:0/2095795713 learned_addr learned my addr 172.21.15.46:0/2095795713 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.019+0000 7f9340d4d700 1 --2- 172.21.15.46:0/2095795713 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f933c141200 0x7f933c13c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.019+0000 7f933bfff700 1 --2- 172.21.15.46:0/2095795713 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f933c0c8da0 0x7f933c0c5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.020+0000 7f933b7fe700 1 -- 172.21.15.46:0/2095795713 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f933c0c8da0 msgr2=0x7f933c0c5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.020+0000 7f933b7fe700 1 --2- 172.21.15.46:0/2095795713 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f933c0c8da0 0x7f933c0c5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.020+0000 7f933b7fe700 1 -- 172.21.15.46:0/2095795713 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f933c141200 msgr2=0x7f933c13c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.020+0000 7f933b7fe700 1 --2- 172.21.15.46:0/2095795713 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f933c141200 0x7f933c13c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.020+0000 7f933b7fe700 1 -- 172.21.15.46:0/2095795713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9324009420 con 0x7f933c0caf30 2024-01-26T23:43:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.020+0000 7f933b7fe700 1 --2- 172.21.15.46:0/2095795713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f933c0caf30 0x7f933c167ed0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f933c06f5a0 tx=0x7f9324017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.020+0000 7f9333fff700 1 -- 172.21.15.46:0/2095795713 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9324010ac0 con 0x7f933c0caf30 2024-01-26T23:43:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.020+0000 7f9333fff700 1 -- 172.21.15.46:0/2095795713 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9324010c20 con 0x7f933c0caf30 2024-01-26T23:43:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.020+0000 7f93427b0700 1 -- 172.21.15.46:0/2095795713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933c0b1840 con 0x7f933c0caf30 2024-01-26T23:43:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.020+0000 7f9333fff700 1 -- 172.21.15.46:0/2095795713 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932401d430 con 0x7f933c0caf30 2024-01-26T23:43:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.020+0000 7f93427b0700 1 -- 172.21.15.46:0/2095795713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933c0b1e00 con 0x7f933c0caf30 2024-01-26T23:43:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.021+0000 7f93427b0700 1 -- 172.21.15.46:0/2095795713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f933c04c710 con 0x7f933c0caf30 2024-01-26T23:43:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.022+0000 7f9333fff700 1 -- 172.21.15.46:0/2095795713 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f932401d7e0 con 0x7f933c0caf30 2024-01-26T23:43:15.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.022+0000 7f9333fff700 1 --2- 172.21.15.46:0/2095795713 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f931c066f30 0x7f931c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:15.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.022+0000 7f9333fff700 1 -- 172.21.15.46:0/2095795713 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f9324086380 con 0x7f933c0caf30 2024-01-26T23:43:15.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.022+0000 7f933bfff700 1 --2- 172.21.15.46:0/2095795713 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f931c066f30 0x7f931c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:15.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.023+0000 7f933bfff700 1 --2- 172.21.15.46:0/2095795713 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f931c066f30 0x7f931c0693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f933c1401c0 tx=0x7f932c000bc0 comp rx=0 tx=0).ready entity=mgr.5267 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:15.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.026+0000 7f9333fff700 1 -- 172.21.15.46:0/2095795713 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9324053340 con 0x7f933c0caf30 2024-01-26T23:43:15.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.289+0000 7f93427b0700 1 -- 172.21.15.46:0/2095795713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f933c0b2eb0 con 0x7f933c0caf30 2024-01-26T23:43:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.624+0000 7f9333fff700 1 -- 172.21.15.46:0/2095795713 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 19) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f9324053710 con 0x7f933c0caf30 2024-01-26T23:43:15.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.625+0000 7f9333fff700 1 -- 172.21.15.46:0/2095795713 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f9324053340 con 0x7f933c0caf30 2024-01-26T23:43:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.628+0000 7f93427b0700 1 -- 172.21.15.46:0/2095795713 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f931c066f30 msgr2=0x7f931c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.628+0000 7f93427b0700 1 --2- 172.21.15.46:0/2095795713 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f931c066f30 0x7f931c0693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f933c1401c0 tx=0x7f932c000bc0 comp rx=0 tx=0).stop 2024-01-26T23:43:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.628+0000 7f93427b0700 1 -- 172.21.15.46:0/2095795713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f933c0caf30 msgr2=0x7f933c167ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.628+0000 7f93427b0700 1 --2- 172.21.15.46:0/2095795713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f933c0caf30 0x7f933c167ed0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f933c06f5a0 tx=0x7f9324017000 comp rx=0 tx=0).stop 2024-01-26T23:43:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.628+0000 7f93427b0700 1 -- 172.21.15.46:0/2095795713 shutdown_connections 2024-01-26T23:43:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.628+0000 7f93427b0700 1 --2- 172.21.15.46:0/2095795713 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f931c066f30 0x7f931c0693e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.628+0000 7f93427b0700 1 --2- 172.21.15.46:0/2095795713 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f933c141200 0x7f933c13c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.628+0000 7f93427b0700 1 --2- 172.21.15.46:0/2095795713 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f933c0c8da0 0x7f933c0c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.628+0000 7f93427b0700 1 --2- 172.21.15.46:0/2095795713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f933c0caf30 0x7f933c167ed0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.628+0000 7f93427b0700 1 -- 172.21.15.46:0/2095795713 >> 172.21.15.46:0/2095795713 conn(0x7f933c06bd90 msgr2=0x7f933c0c3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.628+0000 7f93427b0700 1 -- 172.21.15.46:0/2095795713 shutdown_connections 2024-01-26T23:43:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.628+0000 7f93427b0700 1 -- 172.21.15.46:0/2095795713 wait complete. 2024-01-26T23:43:15.645 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-26T23:43:15.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.784+0000 7fe61d544700 1 Processor -- start 2024-01-26T23:43:15.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.784+0000 7fe61d544700 1 -- start start 2024-01-26T23:43:15.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.785+0000 7fe61d544700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe61806a570 0x7fe61806a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:15.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.785+0000 7fe61d544700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe61806b7f0 0x7fe61806e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:15.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.785+0000 7fe61d544700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61806db60 con 0x7fe6180a9980 2024-01-26T23:43:15.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.785+0000 7fe61d544700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6180a9e60 con 0x7fe61806b7f0 2024-01-26T23:43:15.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.785+0000 7fe61d544700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6180aa140 con 0x7fe61806a570 2024-01-26T23:43:15.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.785+0000 7fe60e7fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe61806b7f0 0x7fe61806e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:15.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.785+0000 7fe60e7fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe61806b7f0 0x7fe61806e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46960/0 (socket says 172.21.15.46:46960) 2024-01-26T23:43:15.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.785+0000 7fe60e7fc700 1 -- 172.21.15.46:0/1855360437 learned_addr learned my addr 172.21.15.46:0/1855360437 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:15.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.785+0000 7fe616ffd700 1 --2- 172.21.15.46:0/1855360437 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe61806a570 0x7fe61806a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:15.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.785+0000 7fe6167fc700 1 -- 172.21.15.46:0/1855360437 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497341602 0 0) 0x7fe61806db60 con 0x7fe6180a9980 2024-01-26T23:43:15.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.785+0000 7fe6167fc700 1 -- 172.21.15.46:0/1855360437 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe604003570 con 0x7fe6180a9980 2024-01-26T23:43:15.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.785+0000 7fe60e7fc700 1 -- 172.21.15.46:0/1855360437 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe61806a570 msgr2=0x7fe61806a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:15.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.785+0000 7fe60e7fc700 1 --2- 172.21.15.46:0/1855360437 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe61806a570 0x7fe61806a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:15.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.786+0000 7fe60e7fc700 1 -- 172.21.15.46:0/1855360437 >> v1:172.21.15.106:6789/0 conn(0x7fe6180a9980 legacy=0x7fe6180aba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:15.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.786+0000 7fe60e7fc700 1 -- 172.21.15.46:0/1855360437 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6180aa420 con 0x7fe61806b7f0 2024-01-26T23:43:15.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.786+0000 7fe60e7fc700 1 --2- 172.21.15.46:0/1855360437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe61806b7f0 0x7fe61806e2d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe6000099b0 tx=0x7fe600009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc08e54967e9b1d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:15.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.786+0000 7fe6167fc700 1 -- 172.21.15.46:0/1855360437 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60000b680 con 0x7fe61806b7f0 2024-01-26T23:43:15.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.786+0000 7fe6167fc700 1 -- 172.21.15.46:0/1855360437 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60000b7e0 con 0x7fe61806b7f0 2024-01-26T23:43:15.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.786+0000 7fe6167fc700 1 -- 172.21.15.46:0/1855360437 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60000bb60 con 0x7fe61806b7f0 2024-01-26T23:43:15.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.787+0000 7fe61d544700 1 -- 172.21.15.46:0/1855360437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe61806b7f0 msgr2=0x7fe61806e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:15.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.787+0000 7fe61d544700 1 --2- 172.21.15.46:0/1855360437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe61806b7f0 0x7fe61806e2d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe6000099b0 tx=0x7fe600009250 comp rx=0 tx=0).stop 2024-01-26T23:43:15.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.787+0000 7fe61d544700 1 -- 172.21.15.46:0/1855360437 shutdown_connections 2024-01-26T23:43:15.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.787+0000 7fe61d544700 1 --2- 172.21.15.46:0/1855360437 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe61806a570 0x7fe61806a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:15.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.787+0000 7fe61d544700 1 --2- 172.21.15.46:0/1855360437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe61806b7f0 0x7fe61806e2d0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:15.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.787+0000 7fe61d544700 1 -- 172.21.15.46:0/1855360437 >> 172.21.15.46:0/1855360437 conn(0x7fe618067930 msgr2=0x7fe618067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:15.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.787+0000 7fe61d544700 1 -- 172.21.15.46:0/1855360437 shutdown_connections 2024-01-26T23:43:15.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.787+0000 7fe61d544700 1 -- 172.21.15.46:0/1855360437 wait complete. 2024-01-26T23:43:15.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.789+0000 7fe61d544700 1 Processor -- start 2024-01-26T23:43:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.789+0000 7fe61d544700 1 -- start start 2024-01-26T23:43:15.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.790+0000 7fe61d544700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe61806a570 0x7fe618152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.790+0000 7fe61d544700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe61806b7f0 0x7fe618159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.790+0000 7fe61d544700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6180a9980 0x7fe61816d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.790+0000 7fe61d544700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe618161000 con 0x7fe61806a570 2024-01-26T23:43:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.790+0000 7fe61d544700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6181603f0 con 0x7fe6180a9980 2024-01-26T23:43:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.790+0000 7fe61d544700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6181606d0 con 0x7fe61806b7f0 2024-01-26T23:43:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.790+0000 7fe616ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe61806a570 0x7fe618152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.790+0000 7fe60e7fc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe61806b7f0 0x7fe618159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.790+0000 7fe60e7fc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe61806b7f0 0x7fe618159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:48194/0 (socket says 172.21.15.46:48194) 2024-01-26T23:43:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.790+0000 7fe616ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe61806a570 0x7fe618152e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46964/0 (socket says 172.21.15.46:46964) 2024-01-26T23:43:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.790+0000 7fe60e7fc700 1 -- 172.21.15.46:0/2073575239 learned_addr learned my addr 172.21.15.46:0/2073575239 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.790+0000 7fe6177fe700 1 --2- 172.21.15.46:0/2073575239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6180a9980 0x7fe61816d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.791+0000 7fe60e7fc700 1 -- 172.21.15.46:0/2073575239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6180a9980 msgr2=0x7fe61816d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.791+0000 7fe60e7fc700 1 --2- 172.21.15.46:0/2073575239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6180a9980 0x7fe61816d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.791+0000 7fe60e7fc700 1 -- 172.21.15.46:0/2073575239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe61806a570 msgr2=0x7fe618152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.791+0000 7fe60e7fc700 1 --2- 172.21.15.46:0/2073575239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe61806a570 0x7fe618152e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.791+0000 7fe60e7fc700 1 -- 172.21.15.46:0/2073575239 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6000094f0 con 0x7fe61806b7f0 2024-01-26T23:43:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.791+0000 7fe616ffd700 1 --2- 172.21.15.46:0/2073575239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe61806a570 0x7fe618152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:43:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.791+0000 7fe6177fe700 1 --2- 172.21.15.46:0/2073575239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6180a9980 0x7fe61816d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:43:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.791+0000 7fe60e7fc700 1 --2- 172.21.15.46:0/2073575239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe61806b7f0 0x7fe618159ce0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe618064090 tx=0x7fe600017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.791+0000 7fe60f7fe700 1 -- 172.21.15.46:0/2073575239 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe600002f90 con 0x7fe61806b7f0 2024-01-26T23:43:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.791+0000 7fe60f7fe700 1 -- 172.21.15.46:0/2073575239 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60000b950 con 0x7fe61806b7f0 2024-01-26T23:43:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.791+0000 7fe60f7fe700 1 -- 172.21.15.46:0/2073575239 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60001a660 con 0x7fe61806b7f0 2024-01-26T23:43:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.791+0000 7fe61d544700 1 -- 172.21.15.46:0/2073575239 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6181609b0 con 0x7fe61806b7f0 2024-01-26T23:43:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.791+0000 7fe61d544700 1 -- 172.21.15.46:0/2073575239 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6181533d0 con 0x7fe61806b7f0 2024-01-26T23:43:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.792+0000 7fe61d544700 1 -- 172.21.15.46:0/2073575239 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe61804c710 con 0x7fe61806b7f0 2024-01-26T23:43:15.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.793+0000 7fe60f7fe700 1 -- 172.21.15.46:0/2073575239 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 19) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fe60001aa50 con 0x7fe61806b7f0 2024-01-26T23:43:15.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.793+0000 7fe60f7fe700 1 --2- 172.21.15.46:0/2073575239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe5f8066fb0 0x7fe5f8069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:15.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.793+0000 7fe616ffd700 1 -- 172.21.15.46:0/2073575239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe5f8066fb0 msgr2=0x7fe5f8069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:43:15.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.793+0000 7fe60f7fe700 1 -- 172.21.15.46:0/2073575239 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fe6000863e0 con 0x7fe61806b7f0 2024-01-26T23:43:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.793+0000 7fe616ffd700 1 --2- 172.21.15.46:0/2073575239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe5f8066fb0 0x7fe5f8069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:43:15.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.798+0000 7fe60f7fe700 1 -- 172.21.15.46:0/2073575239 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe600053300 con 0x7fe61806b7f0 2024-01-26T23:43:15.961 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:15.960+0000 7feef38b91c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:43:15.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.994+0000 7fe616ffd700 1 -- 172.21.15.46:0/2073575239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe5f8066fb0 msgr2=0x7fe5f8069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:43:15.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:15.994+0000 7fe616ffd700 1 --2- 172.21.15.46:0/2073575239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe5f8066fb0 0x7fe5f8069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:43:16.037 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:16.036+0000 7feef38b91c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:43:16.056 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:16.055+0000 7f05bc5fe1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:43:16.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.066+0000 7fe61d544700 1 -- 172.21.15.46:0/2073575239 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fe618160810 con 0x7fe61806b7f0 2024-01-26T23:43:16.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.071+0000 7fe60f7fe700 1 -- 172.21.15.46:0/2073575239 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe618160810 con 0x7fe61806b7f0 2024-01-26T23:43:16.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.073+0000 7fe61d544700 1 -- 172.21.15.46:0/2073575239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe5f8066fb0 msgr2=0x7fe5f8069460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:43:16.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.073+0000 7fe61d544700 1 --2- 172.21.15.46:0/2073575239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe5f8066fb0 0x7fe5f8069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.073+0000 7fe61d544700 1 -- 172.21.15.46:0/2073575239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe61806b7f0 msgr2=0x7fe618159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:16.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.073+0000 7fe61d544700 1 --2- 172.21.15.46:0/2073575239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe61806b7f0 0x7fe618159ce0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe618064090 tx=0x7fe600017000 comp rx=0 tx=0).stop 2024-01-26T23:43:16.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.073+0000 7fe61d544700 1 -- 172.21.15.46:0/2073575239 shutdown_connections 2024-01-26T23:43:16.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.073+0000 7fe61d544700 1 --2- 172.21.15.46:0/2073575239 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe5f8066fb0 0x7fe5f8069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.073+0000 7fe61d544700 1 --2- 172.21.15.46:0/2073575239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe61806a570 0x7fe618152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.073+0000 7fe61d544700 1 --2- 172.21.15.46:0/2073575239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6180a9980 0x7fe61816d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.073+0000 7fe61d544700 1 --2- 172.21.15.46:0/2073575239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe61806b7f0 0x7fe618159ce0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.073+0000 7fe61d544700 1 -- 172.21.15.46:0/2073575239 >> 172.21.15.46:0/2073575239 conn(0x7fe618067930 msgr2=0x7fe6180afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:16.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.074+0000 7fe61d544700 1 -- 172.21.15.46:0/2073575239 shutdown_connections 2024-01-26T23:43:16.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.074+0000 7fe61d544700 1 -- 172.21.15.46:0/2073575239 wait complete. 2024-01-26T23:43:16.089 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-26T23:43:16.109 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:16.108+0000 7feef38b91c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:43:16.148 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:16.146+0000 7f05bc5fe1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:43:16.175 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:16.174+0000 7feef38b91c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:43:16.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.224+0000 7f749844b700 1 Processor -- start 2024-01-26T23:43:16.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.224+0000 7f749844b700 1 -- start start 2024-01-26T23:43:16.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.226+0000 7f749844b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f74900afb40 0x7f74900c5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:16.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.226+0000 7f749844b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74900cc100 con 0x7f74900cf050 2024-01-26T23:43:16.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.226+0000 7f749844b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74900cb850 con 0x7f7490159cb0 2024-01-26T23:43:16.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.226+0000 7f749844b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74900ccca0 con 0x7f74900afb40 2024-01-26T23:43:16.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.226+0000 7f74961e7700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f7490159cb0 0x7f74900cc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:51232/0 (socket says 172.21.15.46:51232) 2024-01-26T23:43:16.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.226+0000 7f74961e7700 1 -- 172.21.15.46:0/1465449985 learned_addr learned my addr 172.21.15.46:0/1465449985 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:43:16.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.226+0000 7f74969e8700 1 --2- 172.21.15.46:0/1465449985 >> v2:172.21.15.106:3300/0 conn(0x7f74900afb40 0x7f74900c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:16.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.226+0000 7f74951e5700 1 -- 172.21.15.46:0/1465449985 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 688297121 0 0) 0x7f74900cb850 con 0x7f7490159cb0 2024-01-26T23:43:16.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.226+0000 7f74951e5700 1 -- 172.21.15.46:0/1465449985 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f747c003570 con 0x7f7490159cb0 2024-01-26T23:43:16.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.226+0000 7f74951e5700 1 -- 172.21.15.46:0/1465449985 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917828988 0 0) 0x7f74900cc100 con 0x7f74900cf050 2024-01-26T23:43:16.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.226+0000 7f74951e5700 1 -- 172.21.15.46:0/1465449985 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74900cb850 con 0x7f74900cf050 2024-01-26T23:43:16.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.227+0000 7f74969e8700 1 -- 172.21.15.46:0/1465449985 >> v1:172.21.15.106:6789/0 conn(0x7f7490159cb0 legacy=0x7f74900cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:16.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.227+0000 7f74969e8700 1 -- 172.21.15.46:0/1465449985 >> v1:172.21.15.46:6789/0 conn(0x7f74900cf050 legacy=0x7f74900af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:16.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.227+0000 7f74969e8700 1 -- 172.21.15.46:0/1465449985 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74900cbaf0 con 0x7f74900afb40 2024-01-26T23:43:16.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.227+0000 7f74969e8700 1 --2- 172.21.15.46:0/1465449985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74900afb40 0x7f74900c5530 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f74880098e0 tx=0x7f74880091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bf5670da8ad58f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:16.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.227+0000 7f74951e5700 1 -- 172.21.15.46:0/1465449985 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488011630 con 0x7f74900afb40 2024-01-26T23:43:16.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.227+0000 7f74951e5700 1 -- 172.21.15.46:0/1465449985 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7488011790 con 0x7f74900afb40 2024-01-26T23:43:16.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.227+0000 7f74951e5700 1 -- 172.21.15.46:0/1465449985 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488011b30 con 0x7f74900afb40 2024-01-26T23:43:16.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.228+0000 7f749844b700 1 -- 172.21.15.46:0/1465449985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74900afb40 msgr2=0x7f74900c5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:16.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.228+0000 7f749844b700 1 --2- 172.21.15.46:0/1465449985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74900afb40 0x7f74900c5530 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f74880098e0 tx=0x7f74880091a0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.228+0000 7f749844b700 1 -- 172.21.15.46:0/1465449985 shutdown_connections 2024-01-26T23:43:16.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.228+0000 7f749844b700 1 --2- 172.21.15.46:0/1465449985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74900afb40 0x7f74900c5530 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.228+0000 7f749844b700 1 -- 172.21.15.46:0/1465449985 >> 172.21.15.46:0/1465449985 conn(0x7f7490087f60 msgr2=0x7f749006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:16.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.228+0000 7f749844b700 1 -- 172.21.15.46:0/1465449985 shutdown_connections 2024-01-26T23:43:16.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.228+0000 7f749844b700 1 -- 172.21.15.46:0/1465449985 wait complete. 2024-01-26T23:43:16.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.229+0000 7f749844b700 1 Processor -- start 2024-01-26T23:43:16.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.230+0000 7f749844b700 1 -- start start 2024-01-26T23:43:16.232 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:16.230+0000 7f05bc5fe1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:43:16.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f749844b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74900afb40 0x7f74900b2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:16.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f749844b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74900cf050 0x7f74900b5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:16.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f749844b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7490159cb0 0x7f74901537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:16.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f749844b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74900b5d20 con 0x7f7490159cb0 2024-01-26T23:43:16.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f749844b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74900b20c0 con 0x7f74900cf050 2024-01-26T23:43:16.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f749844b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74900b14b0 con 0x7f74900afb40 2024-01-26T23:43:16.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f74969e8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7490159cb0 0x7f74901537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:16.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f74969e8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7490159cb0 0x7f74901537a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46974/0 (socket says 172.21.15.46:46974) 2024-01-26T23:43:16.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f74969e8700 1 -- 172.21.15.46:0/1739065858 learned_addr learned my addr 172.21.15.46:0/1739065858 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:16.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f74961e7700 1 --2- 172.21.15.46:0/1739065858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74900afb40 0x7f74900b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:16.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f74959e6700 1 --2- 172.21.15.46:0/1739065858 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74900cf050 0x7f74900b5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:16.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f74969e8700 1 -- 172.21.15.46:0/1739065858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74900afb40 msgr2=0x7f74900b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:16.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f74969e8700 1 --2- 172.21.15.46:0/1739065858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74900afb40 0x7f74900b2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f74969e8700 1 -- 172.21.15.46:0/1739065858 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74900cf050 msgr2=0x7f74900b5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:16.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f74969e8700 1 --2- 172.21.15.46:0/1739065858 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74900cf050 0x7f74900b5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f74969e8700 1 -- 172.21.15.46:0/1739065858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7488009420 con 0x7f7490159cb0 2024-01-26T23:43:16.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f74961e7700 1 --2- 172.21.15.46:0/1739065858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74900afb40 0x7f74900b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:43:16.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.231+0000 7f74969e8700 1 --2- 172.21.15.46:0/1739065858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7490159cb0 0x7f74901537a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7488013000 tx=0x7f7488017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:16.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.232+0000 7f74867fc700 1 -- 172.21.15.46:0/1739065858 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748801f070 con 0x7f7490159cb0 2024-01-26T23:43:16.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.232+0000 7f74867fc700 1 -- 172.21.15.46:0/1739065858 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7488002f70 con 0x7f7490159cb0 2024-01-26T23:43:16.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.232+0000 7f749844b700 1 -- 172.21.15.46:0/1739065858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74900b1790 con 0x7f7490159cb0 2024-01-26T23:43:16.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.232+0000 7f74867fc700 1 -- 172.21.15.46:0/1739065858 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488023640 con 0x7f7490159cb0 2024-01-26T23:43:16.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.232+0000 7f749844b700 1 -- 172.21.15.46:0/1739065858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74900b1d50 con 0x7f7490159cb0 2024-01-26T23:43:16.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.233+0000 7f7473fff700 1 -- 172.21.15.46:0/1739065858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f746c0052b0 con 0x7f7490159cb0 2024-01-26T23:43:16.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.233+0000 7f74867fc700 1 -- 172.21.15.46:0/1739065858 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f748802c3b0 con 0x7f7490159cb0 2024-01-26T23:43:16.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.234+0000 7f74867fc700 1 --2- 172.21.15.46:0/1739065858 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7474066f80 0x7f7474069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:16.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.234+0000 7f74867fc700 1 -- 172.21.15.46:0/1739065858 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f7488085cd0 con 0x7f7490159cb0 2024-01-26T23:43:16.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.234+0000 7f74961e7700 1 -- 172.21.15.46:0/1739065858 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7474066f80 msgr2=0x7f7474069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:43:16.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.234+0000 7f74961e7700 1 --2- 172.21.15.46:0/1739065858 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7474066f80 0x7f7474069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:43:16.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.238+0000 7f74867fc700 1 -- 172.21.15.46:0/1739065858 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7488052c90 con 0x7f7490159cb0 2024-01-26T23:43:16.309 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:16.308+0000 7f05bc5fe1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:43:16.330 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:16.329+0000 7feef38b91c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:43:16.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.435+0000 7f74961e7700 1 -- 172.21.15.46:0/1739065858 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7474066f80 msgr2=0x7f7474069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:43:16.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.435+0000 7f74961e7700 1 --2- 172.21.15.46:0/1739065858 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7474066f80 0x7f7474069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:43:16.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.448+0000 7f7473fff700 1 -- 172.21.15.46:0/1739065858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f746c005830 con 0x7f7490159cb0 2024-01-26T23:43:16.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.451+0000 7f74867fc700 1 -- 172.21.15.46:0/1739065858 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v18)=0 v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7f74880150c0 con 0x7f7490159cb0 2024-01-26T23:43:16.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.454+0000 7f7473fff700 1 -- 172.21.15.46:0/1739065858 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7474066f80 msgr2=0x7f7474069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:43:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.454+0000 7f7473fff700 1 --2- 172.21.15.46:0/1739065858 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7474066f80 0x7f7474069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.454+0000 7f7473fff700 1 -- 172.21.15.46:0/1739065858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7490159cb0 msgr2=0x7f74901537a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.454+0000 7f7473fff700 1 --2- 172.21.15.46:0/1739065858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7490159cb0 0x7f74901537a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7488013000 tx=0x7f7488017000 comp rx=0 tx=0).stop 2024-01-26T23:43:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.454+0000 7f7473fff700 1 -- 172.21.15.46:0/1739065858 shutdown_connections 2024-01-26T23:43:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.454+0000 7f7473fff700 1 --2- 172.21.15.46:0/1739065858 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7474066f80 0x7f7474069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.454+0000 7f7473fff700 1 --2- 172.21.15.46:0/1739065858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7490159cb0 0x7f74901537a0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.454+0000 7f7473fff700 1 --2- 172.21.15.46:0/1739065858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74900afb40 0x7f74900b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.454+0000 7f7473fff700 1 --2- 172.21.15.46:0/1739065858 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74900cf050 0x7f74900b5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.454+0000 7f7473fff700 1 -- 172.21.15.46:0/1739065858 >> 172.21.15.46:0/1739065858 conn(0x7f7490087f60 msgr2=0x7f74900cc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.454+0000 7f7473fff700 1 -- 172.21.15.46:0/1739065858 shutdown_connections 2024-01-26T23:43:16.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.454+0000 7f7473fff700 1 -- 172.21.15.46:0/1739065858 wait complete. 2024-01-26T23:43:16.466 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:16.466+0000 7feef38b91c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:43:16.474 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:16.474 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-26T23:43:16.491 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:16.490+0000 7f05bc5fe1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:43:16.499 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-01-26T23:43:16.532 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:16.531+0000 7feef38b91c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:43:16.653 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:16.651+0000 7f05bc5fe1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:43:16.653 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:16.653+0000 7feef38b91c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:43:16.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.677+0000 7f7daafbb700 1 Processor -- start 2024-01-26T23:43:16.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.677+0000 7f7daafbb700 1 -- start start 2024-01-26T23:43:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.678+0000 7f7daafbb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da40cf050 0x7f7da40af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.678+0000 7f7daafbb700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7da40cf4a0 0x7f7da40af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.678+0000 7f7daafbb700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7da40cc100 con 0x7f7da4159cb0 2024-01-26T23:43:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.678+0000 7f7daafbb700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7da40ccca0 con 0x7f7da40cf4a0 2024-01-26T23:43:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.678+0000 7f7daafbb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da40cb850 con 0x7f7da40cf050 2024-01-26T23:43:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.678+0000 7f7da8d57700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7da4159cb0 0x7f7da40cc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32866/0 (socket says 172.21.15.46:32866) 2024-01-26T23:43:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.678+0000 7f7da8d57700 1 -- 172.21.15.46:0/3932936263 learned_addr learned my addr 172.21.15.46:0/3932936263 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:43:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.678+0000 7f7da9558700 1 --2- 172.21.15.46:0/3932936263 >> v2:172.21.15.106:3300/0 conn(0x7f7da40cf4a0 0x7f7da40af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.678+0000 7f7da3fff700 1 --2- 172.21.15.46:0/3932936263 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da40cf050 0x7f7da40af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.679+0000 7f7da37fe700 1 -- 172.21.15.46:0/3932936263 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698562123 0 0) 0x7f7da40cc100 con 0x7f7da4159cb0 2024-01-26T23:43:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.679+0000 7f7da37fe700 1 -- 172.21.15.46:0/3932936263 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d8c003570 con 0x7f7da4159cb0 2024-01-26T23:43:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.679+0000 7f7da3fff700 1 -- 172.21.15.46:0/3932936263 >> v2:172.21.15.106:3300/0 conn(0x7f7da40cf4a0 msgr2=0x7f7da40af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.679+0000 7f7da3fff700 1 --2- 172.21.15.46:0/3932936263 >> v2:172.21.15.106:3300/0 conn(0x7f7da40cf4a0 0x7f7da40af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.679+0000 7f7da3fff700 1 -- 172.21.15.46:0/3932936263 >> v1:172.21.15.46:6789/0 conn(0x7f7da4159cb0 legacy=0x7f7da40cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.679+0000 7f7da3fff700 1 -- 172.21.15.46:0/3932936263 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da40cbaf0 con 0x7f7da40cf050 2024-01-26T23:43:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.679+0000 7f7da9558700 1 --2- 172.21.15.46:0/3932936263 >> v2:172.21.15.106:3300/0 conn(0x7f7da40cf4a0 0x7f7da40af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:43:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.679+0000 7f7da3fff700 1 --2- 172.21.15.46:0/3932936263 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da40cf050 0x7f7da40af430 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7d980098e0 tx=0x7f7d980091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a0d7d5bef0cacf8 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.679+0000 7f7da37fe700 1 -- 172.21.15.46:0/3932936263 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98011630 con 0x7f7da40cf050 2024-01-26T23:43:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.679+0000 7f7da37fe700 1 -- 172.21.15.46:0/3932936263 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d98011790 con 0x7f7da40cf050 2024-01-26T23:43:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.679+0000 7f7da37fe700 1 -- 172.21.15.46:0/3932936263 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98011b30 con 0x7f7da40cf050 2024-01-26T23:43:16.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.680+0000 7f7daafbb700 1 -- 172.21.15.46:0/3932936263 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da40cf050 msgr2=0x7f7da40af430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:16.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.680+0000 7f7daafbb700 1 --2- 172.21.15.46:0/3932936263 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da40cf050 0x7f7da40af430 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7d980098e0 tx=0x7f7d980091a0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.680+0000 7f7daafbb700 1 -- 172.21.15.46:0/3932936263 shutdown_connections 2024-01-26T23:43:16.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.680+0000 7f7daafbb700 1 --2- 172.21.15.46:0/3932936263 >> v2:172.21.15.106:3300/0 conn(0x7f7da40cf4a0 0x7f7da40af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.680+0000 7f7daafbb700 1 --2- 172.21.15.46:0/3932936263 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da40cf050 0x7f7da40af430 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.680+0000 7f7daafbb700 1 -- 172.21.15.46:0/3932936263 >> 172.21.15.46:0/3932936263 conn(0x7f7da4087f60 msgr2=0x7f7da406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:16.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.680+0000 7f7daafbb700 1 -- 172.21.15.46:0/3932936263 shutdown_connections 2024-01-26T23:43:16.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.680+0000 7f7daafbb700 1 -- 172.21.15.46:0/3932936263 wait complete. 2024-01-26T23:43:16.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.682+0000 7f7daafbb700 1 Processor -- start 2024-01-26T23:43:16.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.682+0000 7f7daafbb700 1 -- start start 2024-01-26T23:43:16.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.683+0000 7f7daafbb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7da40cf050 0x7f7da40b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:16.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.683+0000 7f7daafbb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da40cf4a0 0x7f7da40b3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:16.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.683+0000 7f7daafbb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7da4159cb0 0x7f7da40b5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:16.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.683+0000 7f7daafbb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da40b1350 con 0x7f7da4159cb0 2024-01-26T23:43:16.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.683+0000 7f7daafbb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da40b0740 con 0x7f7da40cf050 2024-01-26T23:43:16.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.683+0000 7f7daafbb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da40b0a20 con 0x7f7da40cf4a0 2024-01-26T23:43:16.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.683+0000 7f7da9558700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7da4159cb0 0x7f7da40b5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:16.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.683+0000 7f7da8d57700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7da40cf050 0x7f7da40b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:16.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.683+0000 7f7da9558700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7da4159cb0 0x7f7da40b5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46986/0 (socket says 172.21.15.46:46986) 2024-01-26T23:43:16.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.683+0000 7f7da9558700 1 -- 172.21.15.46:0/3490910037 learned_addr learned my addr 172.21.15.46:0/3490910037 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:16.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.683+0000 7f7da3fff700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da40cf4a0 0x7f7da40b3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:16.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.683+0000 7f7da8d57700 1 -- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da40cf4a0 msgr2=0x7f7da40b3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:16.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.684+0000 7f7da8d57700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da40cf4a0 0x7f7da40b3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.684+0000 7f7da8d57700 1 -- 172.21.15.46:0/3490910037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7da4159cb0 msgr2=0x7f7da40b5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:16.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.684+0000 7f7da8d57700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7da4159cb0 0x7f7da40b5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:16.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.684+0000 7f7da8d57700 1 -- 172.21.15.46:0/3490910037 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9c009a20 con 0x7f7da40cf050 2024-01-26T23:43:16.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.684+0000 7f7da3fff700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da40cf4a0 0x7f7da40b3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:43:16.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.684+0000 7f7da8d57700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7da40cf050 0x7f7da40b3550 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400c790 tx=0x7f7d94007bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:16.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.684+0000 7f7da0ff9700 1 -- 172.21.15.46:0/3490910037 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d940072e0 con 0x7f7da40cf050 2024-01-26T23:43:16.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.684+0000 7f7da0ff9700 1 -- 172.21.15.46:0/3490910037 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d94007440 con 0x7f7da40cf050 2024-01-26T23:43:16.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.684+0000 7f7daafbb700 1 -- 172.21.15.46:0/3490910037 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d98009420 con 0x7f7da40cf050 2024-01-26T23:43:16.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.684+0000 7f7da0ff9700 1 -- 172.21.15.46:0/3490910037 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d940057f0 con 0x7f7da40cf050 2024-01-26T23:43:16.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.684+0000 7f7daafbb700 1 -- 172.21.15.46:0/3490910037 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da40b0fe0 con 0x7f7da40cf050 2024-01-26T23:43:16.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.685+0000 7f7daafbb700 1 -- 172.21.15.46:0/3490910037 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7da404c710 con 0x7f7da40cf050 2024-01-26T23:43:16.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.685+0000 7f7da0ff9700 1 -- 172.21.15.46:0/3490910037 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 19) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f7d94005c40 con 0x7f7da40cf050 2024-01-26T23:43:16.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.686+0000 7f7da0ff9700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 0x7f7d90069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:16.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.686+0000 7f7da0ff9700 1 -- 172.21.15.46:0/3490910037 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f7d94083c70 con 0x7f7da40cf050 2024-01-26T23:43:16.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.686+0000 7f7da3fff700 1 -- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 msgr2=0x7f7d90069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:43:16.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.686+0000 7f7da3fff700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 0x7f7d90069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:43:16.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.690+0000 7f7da0ff9700 1 -- 172.21.15.46:0/3490910037 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7d94050b90 con 0x7f7da40cf050 2024-01-26T23:43:16.728 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:16.727+0000 7feef38b91c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:43:16.732 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:16.731+0000 7f05bc5fe1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:43:16.868 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:16.868+0000 7feef38b91c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:43:16.872 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:16.871+0000 7f05bc5fe1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:43:16.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.887+0000 7f7da3fff700 1 -- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 msgr2=0x7f7d90069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:43:16.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.887+0000 7f7da3fff700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 0x7f7d90069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:43:16.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:16.927+0000 7f7daafbb700 1 -- 172.21.15.46:0/3490910037 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7da40cfc80 con 0x7f7d90066f80 2024-01-26T23:43:16.955 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:16.954+0000 7f05bc5fe1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:43:16.994 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:16.993+0000 7feef38b91c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:43:17.131 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:17.130+0000 7f05bc5fe1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:43:17.271 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:17.270+0000 7f05bc5fe1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:43:17.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:17.288+0000 7f7da3fff700 1 -- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 msgr2=0x7f7d90069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:43:17.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:17.288+0000 7f7da3fff700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 0x7f7d90069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-26T23:43:17.309 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:17.308+0000 7feef38b91c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:43:17.518 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:17.517+0000 7feef38b91c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:43:17.633 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:17.632+0000 7f05bc5fe1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:43:17.744 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:17.743+0000 7feef38b91c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:43:17.813 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:17.812+0000 7feef38b91c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:43:17.872 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:17.870+0000 7f05bc5fe1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:43:18.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:18.089+0000 7f7da3fff700 1 -- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 msgr2=0x7f7d90069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:43:18.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:18.089+0000 7f7da3fff700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 0x7f7d90069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-26T23:43:18.130 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:18.128+0000 7f05bc5fe1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:43:18.210 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:18.209+0000 7f05bc5fe1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:43:18.409 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:18.408+0000 7feef38b91c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:43:18.482 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:18.481+0000 7feef38b91c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:43:18.604 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:18.603+0000 7feef38b91c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:43:18.902 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:18.901+0000 7f05bc5fe1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:43:18.986 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:18.985+0000 7f05bc5fe1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:43:19.101 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:19.100+0000 7feef38b91c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:43:19.126 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:19.125+0000 7f05bc5fe1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:43:19.247 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:19.246+0000 7feef38b91c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:43:19.395 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:19.394+0000 7feef38b91c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:43:19.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:19.691+0000 7f7da3fff700 1 -- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 msgr2=0x7f7d90069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:43:19.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:19.691+0000 7f7da3fff700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 0x7f7d90069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-26T23:43:19.705 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:19.704+0000 7f05bc5fe1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:43:19.874 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:19.872+0000 7f05bc5fe1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:43:20.044 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:20.042+0000 7f05bc5fe1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:43:20.087 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:20.086+0000 7feef38b91c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:43:20.302 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:20.301+0000 7feef38b91c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:43:20.373 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:20.372+0000 7feef38b91c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:43:20.526 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:20.525+0000 7feef38b91c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:43:20.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:20.548+0000 7f7da0ff9700 1 -- 172.21.15.46:0/3490910037 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mgrmap(e 20) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f7d94050f60 con 0x7f7da40cf050 2024-01-26T23:43:20.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:20.549+0000 7f7da0ff9700 1 -- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 msgr2=0x7f7d90069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:43:20.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:20.549+0000 7f7da0ff9700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 0x7f7d90069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:20.652 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:20.649+0000 7fee6d15d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:43:20.652 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:20.649+0000 7fee6d15d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:43:20.653 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:20.649+0000 7fee6d15d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:43:20.653 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:20.649+0000 7fee6d15d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:43:20.653 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:20.649+0000 7fee6d15d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:43:20.653 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:20.650+0000 7fee6a157700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:43:20.653 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:20.650+0000 7fee6a157700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:43:20.653 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:20.650+0000 7fee6a157700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:43:20.653 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:20.650+0000 7fee6a157700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:43:20.653 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:43:20.650+0000 7fee6a157700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:43:20.859 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:20.857+0000 7f05bc5fe1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:43:21.103 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:21.102+0000 7f05bc5fe1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:43:21.184 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:21.183+0000 7f05bc5fe1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:43:21.361 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:43:21.360+0000 7f05bc5fe1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:43:21.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.552+0000 7f7da0ff9700 1 -- 172.21.15.46:0/3490910037 <== mon.1 v2:172.21.15.106:3300/0 8 ==== mgrmap(e 21) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f7d94054ab0 con 0x7f7da40cf050 2024-01-26T23:43:21.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.552+0000 7f7da0ff9700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 0x7f7d900693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:21.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.552+0000 7f7da0ff9700 1 -- 172.21.15.46:0/3490910037 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7da40cfc80 con 0x7f7d90066f80 2024-01-26T23:43:21.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.557+0000 7f7da3fff700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 0x7f7d900693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:21.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.560+0000 7f7da3fff700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 0x7f7d900693b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7d980091e0 tx=0x7f7d98017000 comp rx=0 tx=0).ready entity=mgr.5564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.619+0000 7f7da0ff9700 1 -- 172.21.15.46:0/3490910037 <== mgr.5564 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7da40cfc80 con 0x7f7d90066f80 2024-01-26T23:43:21.620 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path snap_test_dir1 2024-01-26T23:43:21.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.621+0000 7f7daafbb700 1 -- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 msgr2=0x7f7d900693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:21.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.621+0000 7f7daafbb700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 0x7f7d900693b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7d980091e0 tx=0x7f7d98017000 comp rx=0 tx=0).stop 2024-01-26T23:43:21.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.621+0000 7f7daafbb700 1 -- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7da40cf050 msgr2=0x7f7da40b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:21.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.621+0000 7f7daafbb700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7da40cf050 0x7f7da40b3550 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400c790 tx=0x7f7d94007bc0 comp rx=0 tx=0).stop 2024-01-26T23:43:21.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.622+0000 7f7daafbb700 1 -- 172.21.15.46:0/3490910037 shutdown_connections 2024-01-26T23:43:21.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.622+0000 7f7daafbb700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7d90066f80 0x7f7d900693b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:21.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.622+0000 7f7daafbb700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7da4159cb0 0x7f7da40b5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:21.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.622+0000 7f7daafbb700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7da40cf050 0x7f7da40b3550 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:21.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.622+0000 7f7daafbb700 1 --2- 172.21.15.46:0/3490910037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da40cf4a0 0x7f7da40b3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:21.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.622+0000 7f7daafbb700 1 -- 172.21.15.46:0/3490910037 >> 172.21.15.46:0/3490910037 conn(0x7f7da4087f60 msgr2=0x7f7da40c6680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:21.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.622+0000 7f7daafbb700 1 -- 172.21.15.46:0/3490910037 shutdown_connections 2024-01-26T23:43:21.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.622+0000 7f7daafbb700 1 -- 172.21.15.46:0/3490910037 wait complete. 2024-01-26T23:43:21.638 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-01-26T23:43:21.638 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 2m 2024-01-26T23:43:21.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.776+0000 7f0554be5700 1 Processor -- start 2024-01-26T23:43:21.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.776+0000 7f0554be5700 1 -- start start 2024-01-26T23:43:21.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f0554be5700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0550157750 0x7f0550157b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:21.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f0554be5700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f055006b3c0 0x7f05500af080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:21.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f0554be5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05500af5c0 0x7f0550067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:21.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f0554be5700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05500afa40 con 0x7f0550157750 2024-01-26T23:43:21.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f0554be5700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0550158f00 con 0x7f055006b3c0 2024-01-26T23:43:21.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f0554be5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0550158060 con 0x7f05500af5c0 2024-01-26T23:43:21.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f054e59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0550157750 0x7f0550157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:21.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f054e59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0550157750 0x7f0550157b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42016/0 (socket says 172.21.15.46:42016) 2024-01-26T23:43:21.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f054e59c700 1 -- 172.21.15.46:0/1737966430 learned_addr learned my addr 172.21.15.46:0/1737966430 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:21.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f054ed9d700 1 --2- 172.21.15.46:0/1737966430 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05500af5c0 0x7f0550067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:21.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f054dd9b700 1 --2- 172.21.15.46:0/1737966430 >> v2:172.21.15.106:3300/0 conn(0x7f055006b3c0 0x7f05500af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:21.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f054e59c700 1 -- 172.21.15.46:0/1737966430 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05500af5c0 msgr2=0x7f0550067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:21.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f054e59c700 1 --2- 172.21.15.46:0/1737966430 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05500af5c0 0x7f0550067930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:21.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f054e59c700 1 -- 172.21.15.46:0/1737966430 >> v2:172.21.15.106:3300/0 conn(0x7f055006b3c0 msgr2=0x7f05500af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:21.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f054e59c700 1 --2- 172.21.15.46:0/1737966430 >> v2:172.21.15.106:3300/0 conn(0x7f055006b3c0 0x7f05500af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:21.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f054e59c700 1 -- 172.21.15.46:0/1737966430 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055006fe90 con 0x7f0550157750 2024-01-26T23:43:21.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f054ed9d700 1 --2- 172.21.15.46:0/1737966430 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05500af5c0 0x7f0550067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:43:21.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.777+0000 7f054e59c700 1 --2- 172.21.15.46:0/1737966430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0550157750 0x7f0550157b20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f05500649b0 tx=0x7f0540009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba786111807ad005 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:21.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.778+0000 7f054d59a700 1 -- 172.21.15.46:0/1737966430 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054000b820 con 0x7f0550157750 2024-01-26T23:43:21.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.778+0000 7f054d59a700 1 -- 172.21.15.46:0/1737966430 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054000b980 con 0x7f0550157750 2024-01-26T23:43:21.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.778+0000 7f054d59a700 1 -- 172.21.15.46:0/1737966430 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054000bd40 con 0x7f0550157750 2024-01-26T23:43:21.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.778+0000 7f0554be5700 1 -- 172.21.15.46:0/1737966430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0550157750 msgr2=0x7f0550157b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:21.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.778+0000 7f0554be5700 1 --2- 172.21.15.46:0/1737966430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0550157750 0x7f0550157b20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f05500649b0 tx=0x7f0540009250 comp rx=0 tx=0).stop 2024-01-26T23:43:21.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.778+0000 7f0554be5700 1 -- 172.21.15.46:0/1737966430 shutdown_connections 2024-01-26T23:43:21.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.778+0000 7f0554be5700 1 --2- 172.21.15.46:0/1737966430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0550157750 0x7f0550157b20 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:21.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.778+0000 7f0554be5700 1 --2- 172.21.15.46:0/1737966430 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05500af5c0 0x7f0550067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:21.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.778+0000 7f0554be5700 1 --2- 172.21.15.46:0/1737966430 >> v2:172.21.15.106:3300/0 conn(0x7f055006b3c0 0x7f05500af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:21.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.778+0000 7f0554be5700 1 -- 172.21.15.46:0/1737966430 >> 172.21.15.46:0/1737966430 conn(0x7f05500cdf80 msgr2=0x7f055006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:21.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.778+0000 7f0554be5700 1 -- 172.21.15.46:0/1737966430 shutdown_connections 2024-01-26T23:43:21.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.778+0000 7f0554be5700 1 -- 172.21.15.46:0/1737966430 wait complete. 2024-01-26T23:43:21.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.780+0000 7f0554be5700 1 Processor -- start 2024-01-26T23:43:21.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.780+0000 7f0554be5700 1 -- start start 2024-01-26T23:43:21.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.780+0000 7f0554be5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f055006b3c0 0x7f05500cc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:21.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.780+0000 7f0554be5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05500af5c0 0x7f05500ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:21.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.780+0000 7f0554be5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0550157750 0x7f0550141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:21.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.780+0000 7f0554be5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05500aee80 con 0x7f055006b3c0 2024-01-26T23:43:21.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.780+0000 7f0554be5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055016d3c0 con 0x7f05500af5c0 2024-01-26T23:43:21.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.780+0000 7f0554be5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f055016d6a0 con 0x7f0550157750 2024-01-26T23:43:21.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.781+0000 7f054e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f055006b3c0 0x7f05500cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:21.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.781+0000 7f054e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f055006b3c0 0x7f05500cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42018/0 (socket says 172.21.15.46:42018) 2024-01-26T23:43:21.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.781+0000 7f054e59c700 1 -- 172.21.15.46:0/3651186269 learned_addr learned my addr 172.21.15.46:0/3651186269 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:21.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.781+0000 7f054ed9d700 1 --2- 172.21.15.46:0/3651186269 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0550157750 0x7f0550141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:21.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.781+0000 7f054dd9b700 1 --2- 172.21.15.46:0/3651186269 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05500af5c0 0x7f05500ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:21.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.781+0000 7f054e59c700 1 -- 172.21.15.46:0/3651186269 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0550157750 msgr2=0x7f0550141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:21.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.781+0000 7f054e59c700 1 --2- 172.21.15.46:0/3651186269 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0550157750 0x7f0550141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:21.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.781+0000 7f054e59c700 1 -- 172.21.15.46:0/3651186269 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05500af5c0 msgr2=0x7f05500ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:21.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.781+0000 7f054e59c700 1 --2- 172.21.15.46:0/3651186269 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05500af5c0 0x7f05500ccc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:21.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.781+0000 7f054e59c700 1 -- 172.21.15.46:0/3651186269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05400094f0 con 0x7f055006b3c0 2024-01-26T23:43:21.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.781+0000 7f054e59c700 1 --2- 172.21.15.46:0/3651186269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f055006b3c0 0x7f05500cc740 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f05400099b0 tx=0x7f0540017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:21.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.781+0000 7f053e7fc700 1 -- 172.21.15.46:0/3651186269 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540002f30 con 0x7f055006b3c0 2024-01-26T23:43:21.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.781+0000 7f053e7fc700 1 -- 172.21.15.46:0/3651186269 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0540006a90 con 0x7f055006b3c0 2024-01-26T23:43:21.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.781+0000 7f053e7fc700 1 -- 172.21.15.46:0/3651186269 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540011cf0 con 0x7f055006b3c0 2024-01-26T23:43:21.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.781+0000 7f0554be5700 1 -- 172.21.15.46:0/3651186269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f055016d980 con 0x7f055006b3c0 2024-01-26T23:43:21.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.781+0000 7f0554be5700 1 -- 172.21.15.46:0/3651186269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0550141d90 con 0x7f055006b3c0 2024-01-26T23:43:21.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.782+0000 7f0554be5700 1 -- 172.21.15.46:0/3651186269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05500b0ee0 con 0x7f055006b3c0 2024-01-26T23:43:21.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.783+0000 7f053e7fc700 1 -- 172.21.15.46:0/3651186269 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 21) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f0540023070 con 0x7f055006b3c0 2024-01-26T23:43:21.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.783+0000 7f053e7fc700 1 --2- 172.21.15.46:0/3651186269 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0530066dd0 0x7f0530069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:21.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.783+0000 7f053e7fc700 1 -- 172.21.15.46:0/3651186269 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6647+0+0 (secure 0 0 0) 0x7f0540085540 con 0x7f055006b3c0 2024-01-26T23:43:21.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.784+0000 7f054dd9b700 1 --2- 172.21.15.46:0/3651186269 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0530066dd0 0x7f0530069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:21.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.784+0000 7f054dd9b700 1 --2- 172.21.15.46:0/3651186269 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0530066dd0 0x7f0530069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f05500b30b0 tx=0x7f0544008040 comp rx=0 tx=0).ready entity=mgr.5564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:21.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:21.788+0000 7f053e7fc700 1 -- 172.21.15.46:0/3651186269 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f05400525e0 con 0x7f055006b3c0 2024-01-26T23:43:22.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.020+0000 7f0554be5700 1 -- 172.21.15.46:0/3651186269 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "2m", "target": ["mon-mgr", ""]}) v1 -- 0x7f055016e180 con 0x7f0530066dd0 2024-01-26T23:43:22.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.072+0000 7f053e7fc700 1 -- 172.21.15.46:0/3651186269 <== mgr.5564 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f055016e180 con 0x7f0530066dd0 2024-01-26T23:43:22.074 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path snap_test_dir1 2024-01-26T23:43:22.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.074+0000 7f0554be5700 1 -- 172.21.15.46:0/3651186269 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0530066dd0 msgr2=0x7f0530069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:22.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.074+0000 7f0554be5700 1 --2- 172.21.15.46:0/3651186269 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0530066dd0 0x7f0530069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f05500b30b0 tx=0x7f0544008040 comp rx=0 tx=0).stop 2024-01-26T23:43:22.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.074+0000 7f0554be5700 1 -- 172.21.15.46:0/3651186269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f055006b3c0 msgr2=0x7f05500cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:22.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.074+0000 7f0554be5700 1 --2- 172.21.15.46:0/3651186269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f055006b3c0 0x7f05500cc740 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f05400099b0 tx=0x7f0540017000 comp rx=0 tx=0).stop 2024-01-26T23:43:22.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.074+0000 7f0554be5700 1 -- 172.21.15.46:0/3651186269 shutdown_connections 2024-01-26T23:43:22.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.074+0000 7f0554be5700 1 --2- 172.21.15.46:0/3651186269 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0530066dd0 0x7f0530069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:22.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.074+0000 7f0554be5700 1 --2- 172.21.15.46:0/3651186269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f055006b3c0 0x7f05500cc740 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:22.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.074+0000 7f0554be5700 1 --2- 172.21.15.46:0/3651186269 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0550157750 0x7f0550141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:22.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.074+0000 7f0554be5700 1 --2- 172.21.15.46:0/3651186269 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05500af5c0 0x7f05500ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:22.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.074+0000 7f0554be5700 1 -- 172.21.15.46:0/3651186269 >> 172.21.15.46:0/3651186269 conn(0x7f05500cdf80 msgr2=0x7f0550154280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:22.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.074+0000 7f0554be5700 1 -- 172.21.15.46:0/3651186269 shutdown_connections 2024-01-26T23:43:22.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.075+0000 7f0554be5700 1 -- 172.21.15.46:0/3651186269 wait complete. 2024-01-26T23:43:22.090 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-01-26T23:43:22.090 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-01-26-23_44_00 in ~38s... 2024-01-26T23:43:22.090 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-01-26-23_45_00 in ~98s... 2024-01-26T23:43:22.091 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m', '2m'], expected_retention: [] 2024-01-26T23:43:22.091 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-01-26T23:43:22.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.226+0000 7f2b95cf1700 1 Processor -- start 2024-01-26T23:43:22.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.226+0000 7f2b95cf1700 1 -- start start 2024-01-26T23:43:22.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.227+0000 7f2b95cf1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2b90157490 0x7f2b900cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:22.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.227+0000 7f2b95cf1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b9006ddb0 0x7f2b900cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:22.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.227+0000 7f2b95cf1700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b900cea80 con 0x7f2b90157860 2024-01-26T23:43:22.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.227+0000 7f2b95cf1700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b900cdc60 con 0x7f2b90157490 2024-01-26T23:43:22.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.227+0000 7f2b95cf1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b900b2ae0 con 0x7f2b9006ddb0 2024-01-26T23:43:22.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.227+0000 7f2b8f7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2b90157490 0x7f2b900cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:22.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.228+0000 7f2b8f7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2b90157490 0x7f2b900cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42024/0 (socket says 172.21.15.46:42024) 2024-01-26T23:43:22.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.228+0000 7f2b8f7fe700 1 -- 172.21.15.46:0/2398712407 learned_addr learned my addr 172.21.15.46:0/2398712407 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:22.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.228+0000 7f2b8ffff700 1 --2- 172.21.15.46:0/2398712407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b9006ddb0 0x7f2b900cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:22.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.228+0000 7f2b8f7fe700 1 -- 172.21.15.46:0/2398712407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b9006ddb0 msgr2=0x7f2b900cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:43:22.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.228+0000 7f2b8f7fe700 1 --2- 172.21.15.46:0/2398712407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b9006ddb0 0x7f2b900cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:22.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.228+0000 7f2b8f7fe700 1 -- 172.21.15.46:0/2398712407 >> v1:172.21.15.106:6789/0 conn(0x7f2b90157860 legacy=0x7f2b9006d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:22.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.228+0000 7f2b8f7fe700 1 -- 172.21.15.46:0/2398712407 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b900cde70 con 0x7f2b90157490 2024-01-26T23:43:22.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.228+0000 7f2b8f7fe700 1 --2- 172.21.15.46:0/2398712407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b90157490 0x7f2b900cc600 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2b800099b0 tx=0x7f2b80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d01514d6efd7e7d server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:22.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.228+0000 7f2b8e7fc700 1 -- 172.21.15.46:0/2398712407 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8000b860 con 0x7f2b90157490 2024-01-26T23:43:22.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.228+0000 7f2b8e7fc700 1 -- 172.21.15.46:0/2398712407 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8000b9c0 con 0x7f2b90157490 2024-01-26T23:43:22.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.228+0000 7f2b8e7fc700 1 -- 172.21.15.46:0/2398712407 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8000bd60 con 0x7f2b90157490 2024-01-26T23:43:22.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.229+0000 7f2b95cf1700 1 -- 172.21.15.46:0/2398712407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b90157490 msgr2=0x7f2b900cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:22.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.229+0000 7f2b95cf1700 1 --2- 172.21.15.46:0/2398712407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b90157490 0x7f2b900cc600 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2b800099b0 tx=0x7f2b80009250 comp rx=0 tx=0).stop 2024-01-26T23:43:22.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.229+0000 7f2b95cf1700 1 -- 172.21.15.46:0/2398712407 shutdown_connections 2024-01-26T23:43:22.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.229+0000 7f2b95cf1700 1 --2- 172.21.15.46:0/2398712407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b9006ddb0 0x7f2b900cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:22.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.229+0000 7f2b95cf1700 1 --2- 172.21.15.46:0/2398712407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b90157490 0x7f2b900cc600 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:22.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.229+0000 7f2b95cf1700 1 -- 172.21.15.46:0/2398712407 >> 172.21.15.46:0/2398712407 conn(0x7f2b90067930 msgr2=0x7f2b90067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:22.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.229+0000 7f2b95cf1700 1 -- 172.21.15.46:0/2398712407 shutdown_connections 2024-01-26T23:43:22.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.229+0000 7f2b95cf1700 1 -- 172.21.15.46:0/2398712407 wait complete. 2024-01-26T23:43:22.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.230+0000 7f2b95cf1700 1 Processor -- start 2024-01-26T23:43:22.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.230+0000 7f2b95cf1700 1 -- start start 2024-01-26T23:43:22.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.231+0000 7f2b95cf1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b9006ddb0 0x7f2b901504b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:22.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.231+0000 7f2b95cf1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b90157490 0x7f2b900adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:22.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.231+0000 7f2b95cf1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b90157860 0x7f2b90152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:22.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.231+0000 7f2b95cf1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b90069c10 con 0x7f2b9006ddb0 2024-01-26T23:43:22.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.231+0000 7f2b95cf1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b900c8190 con 0x7f2b90157490 2024-01-26T23:43:22.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.231+0000 7f2b95cf1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b900c8470 con 0x7f2b90157860 2024-01-26T23:43:22.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.231+0000 7f2b8f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b9006ddb0 0x7f2b901504b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:22.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.231+0000 7f2b8f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b9006ddb0 0x7f2b901504b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42036/0 (socket says 172.21.15.46:42036) 2024-01-26T23:43:22.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.231+0000 7f2b8f7fe700 1 -- 172.21.15.46:0/3984057665 learned_addr learned my addr 172.21.15.46:0/3984057665 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:43:22.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.232+0000 7f2b8ffff700 1 --2- 172.21.15.46:0/3984057665 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b90157860 0x7f2b90152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:22.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.232+0000 7f2b8effd700 1 --2- 172.21.15.46:0/3984057665 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b90157490 0x7f2b900adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:22.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.232+0000 7f2b8f7fe700 1 -- 172.21.15.46:0/3984057665 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b90157860 msgr2=0x7f2b90152720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:22.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.232+0000 7f2b8f7fe700 1 --2- 172.21.15.46:0/3984057665 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b90157860 0x7f2b90152720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:22.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.232+0000 7f2b8f7fe700 1 -- 172.21.15.46:0/3984057665 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b90157490 msgr2=0x7f2b900adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:22.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.232+0000 7f2b8f7fe700 1 --2- 172.21.15.46:0/3984057665 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b90157490 0x7f2b900adf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:22.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.232+0000 7f2b8f7fe700 1 -- 172.21.15.46:0/3984057665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b800094f0 con 0x7f2b9006ddb0 2024-01-26T23:43:22.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.232+0000 7f2b8f7fe700 1 --2- 172.21.15.46:0/3984057665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b9006ddb0 0x7f2b901504b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2b9006d060 tx=0x7f2b80017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:22.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.232+0000 7f2b877fe700 1 -- 172.21.15.46:0/3984057665 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b80024440 con 0x7f2b9006ddb0 2024-01-26T23:43:22.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.232+0000 7f2b877fe700 1 -- 172.21.15.46:0/3984057665 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b800245c0 con 0x7f2b9006ddb0 2024-01-26T23:43:22.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.232+0000 7f2b877fe700 1 -- 172.21.15.46:0/3984057665 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b80024980 con 0x7f2b9006ddb0 2024-01-26T23:43:22.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.232+0000 7f2b95cf1700 1 -- 172.21.15.46:0/3984057665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b900c8750 con 0x7f2b9006ddb0 2024-01-26T23:43:22.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.232+0000 7f2b95cf1700 1 -- 172.21.15.46:0/3984057665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b900c8d10 con 0x7f2b9006ddb0 2024-01-26T23:43:22.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.233+0000 7f2b95cf1700 1 -- 172.21.15.46:0/3984057665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b9004c710 con 0x7f2b9006ddb0 2024-01-26T23:43:22.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.234+0000 7f2b877fe700 1 -- 172.21.15.46:0/3984057665 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 21) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f2b8001a420 con 0x7f2b9006ddb0 2024-01-26T23:43:22.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.234+0000 7f2b877fe700 1 --2- 172.21.15.46:0/3984057665 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2b70066dd0 0x7f2b70069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:43:22.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.234+0000 7f2b877fe700 1 -- 172.21.15.46:0/3984057665 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6647+0+0 (secure 0 0 0) 0x7f2b80086410 con 0x7f2b9006ddb0 2024-01-26T23:43:22.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.234+0000 7f2b8effd700 1 --2- 172.21.15.46:0/3984057665 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2b70066dd0 0x7f2b70069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:43:22.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.235+0000 7f2b8effd700 1 --2- 172.21.15.46:0/3984057665 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2b70066dd0 0x7f2b70069280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2b901533c0 tx=0x7f2b78009040 comp rx=0 tx=0).ready entity=mgr.5564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:43:22.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.239+0000 7f2b877fe700 1 -- 172.21.15.46:0/3984057665 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2b80053580 con 0x7f2b9006ddb0 2024-01-26T23:43:22.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.468+0000 7f2b95cf1700 1 -- 172.21.15.46:0/3984057665 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2b90069d50 con 0x7f2b70066dd0 2024-01-26T23:43:22.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.480+0000 7f2b877fe700 1 -- 172.21.15.46:0/3984057665 <== mgr.5564 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+76 (secure 0 0 0) 0x7f2b78013420 con 0x7f2b70066dd0 2024-01-26T23:43:22.481 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:43:22.481 INFO:teuthology.orchestra.run.smithi046.stdout:{"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-01-26T23:43:22.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.482+0000 7f2b95cf1700 1 -- 172.21.15.46:0/3984057665 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2b70066dd0 msgr2=0x7f2b70069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:22.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.482+0000 7f2b95cf1700 1 --2- 172.21.15.46:0/3984057665 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2b70066dd0 0x7f2b70069280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2b901533c0 tx=0x7f2b78009040 comp rx=0 tx=0).stop 2024-01-26T23:43:22.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.482+0000 7f2b95cf1700 1 -- 172.21.15.46:0/3984057665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b9006ddb0 msgr2=0x7f2b901504b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:43:22.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.482+0000 7f2b95cf1700 1 --2- 172.21.15.46:0/3984057665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b9006ddb0 0x7f2b901504b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2b9006d060 tx=0x7f2b80017000 comp rx=0 tx=0).stop 2024-01-26T23:43:22.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.482+0000 7f2b95cf1700 1 -- 172.21.15.46:0/3984057665 shutdown_connections 2024-01-26T23:43:22.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.482+0000 7f2b95cf1700 1 --2- 172.21.15.46:0/3984057665 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2b70066dd0 0x7f2b70069280 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:22.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.482+0000 7f2b95cf1700 1 --2- 172.21.15.46:0/3984057665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b9006ddb0 0x7f2b901504b0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:22.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.482+0000 7f2b95cf1700 1 --2- 172.21.15.46:0/3984057665 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b90157860 0x7f2b90152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:22.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.482+0000 7f2b95cf1700 1 --2- 172.21.15.46:0/3984057665 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b90157490 0x7f2b900adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:43:22.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.482+0000 7f2b95cf1700 1 -- 172.21.15.46:0/3984057665 >> 172.21.15.46:0/3984057665 conn(0x7f2b90067930 msgr2=0x7f2b901555f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:43:22.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.482+0000 7f2b95cf1700 1 -- 172.21.15.46:0/3984057665 shutdown_connections 2024-01-26T23:43:22.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:43:22.482+0000 7f2b95cf1700 1 -- 172.21.15.46:0/3984057665 wait complete. 2024-01-26T23:43:22.497 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-01-26T23:43:22.497 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m', '2m'], 'retention': [{}, {}]} 2024-01-26T23:43:22.498 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:22.498 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:22.521 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:22.521 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:22.521 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:23.522 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:23.523 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:23.550 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:23.550 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:23.550 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:24.324 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:43:24.327 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:43:24.551 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:24.552 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:24.580 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:24.580 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:24.580 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:25.581 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:25.581 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:25.608 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:25.609 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:25.609 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:26.609 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:26.609 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:26.637 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:26.637 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:26.637 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:27.639 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:27.639 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:27.667 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:27.667 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:27.668 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:28.669 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:28.669 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:28.696 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:28.697 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:28.697 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:29.697 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:29.698 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:29.725 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:29.726 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:29.726 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:30.726 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:30.727 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:30.755 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:30.755 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:30.755 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:31.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:31.756 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:31.784 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:31.784 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:31.784 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:32.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:32.785 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:32.813 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:32.813 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:32.813 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:33.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:33.814 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:33.842 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:33.842 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:33.842 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:34.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:34.843 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:34.871 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:34.871 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:34.871 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:35.872 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:35.872 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:35.900 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:35.900 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:35.900 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:36.901 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:36.901 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:36.928 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:36.929 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:36.929 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:37.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:37.930 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:37.957 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:37.957 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:37.957 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:38.958 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:38.958 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:38.986 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:38.986 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:38.986 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:39.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:39.987 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:40.015 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:40.015 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:40.016 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:41.017 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:41.017 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:41.046 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:41.047 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:41.047 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:42.047 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:42.048 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:42.075 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:42.075 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:42.075 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:43.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:43.077 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:43.104 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:43.105 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:43.105 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:44.105 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:44.106 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:44.134 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:44.134 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:44.134 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:45.135 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:45.135 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:45.162 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:45.163 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:45.163 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:46.164 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:46.164 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:46.192 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:46.192 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:46.192 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:47.193 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:47.193 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:47.221 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:47.221 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:47.221 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:48.222 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:48.222 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:48.249 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:48.250 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:48.250 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:49.251 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:49.251 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:49.278 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:49.278 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:49.278 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:50.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:50.279 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:50.306 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:50.306 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:50.306 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:51.307 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:51.308 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:51.335 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:51.335 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:51.335 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:52.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:52.337 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:52.364 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:52.364 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:52.364 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:53.366 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:53.366 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:53.393 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:53.393 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:53.394 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:54.394 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:54.395 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:54.402 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:43:54.423 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:54.423 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:54.424 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:54.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:43:55.425 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:55.425 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:55.452 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:55.452 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:55.452 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:56.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:56.453 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:56.480 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:56.480 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:56.480 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:57.481 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:57.481 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:57.509 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:57.509 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:57.509 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:58.510 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:58.510 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:58.540 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:58.540 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:58.540 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:43:59.541 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:43:59.541 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:43:59.569 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:43:59.569 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:43:59.569 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:00.084 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:44:00.083+0000 7fee5c13b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:44:00.084 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:44:00.083+0000 7fee5c13b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:44:00.084 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:44:00.083+0000 7fee5c13b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:44:00.084 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:44:00.083+0000 7fee5c13b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:44:00.084 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:44:00.083+0000 7fee5c13b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:44:00.570 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:00.570 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:00.597 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:00.597 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:00.598 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:00.598 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:00.598 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-01-26-23_44_00_UTC'] 2024-01-26T23:44:00.598 DEBUG:tasks.cephfs.test_snap_schedules:exec=1706312602.0907865, now = 1706312640.5983574, timo = 38 2024-01-26T23:44:01.599 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:01.599 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:01.625 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:01.626 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:01.626 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:01.626 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:02.627 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:02.627 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:02.654 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:02.654 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:02.654 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:02.654 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:03.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:03.655 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:03.684 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:03.684 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:03.684 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:03.684 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:04.685 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:04.685 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:04.712 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:04.712 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:04.713 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:04.713 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:05.713 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:05.714 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:05.740 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:05.741 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:05.741 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:05.741 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:06.742 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:06.742 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:06.769 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:06.769 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:06.770 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:06.770 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:07.770 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:07.771 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:07.798 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:07.798 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:07.798 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:07.798 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:08.799 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:08.799 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:08.825 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:08.826 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:08.826 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:08.827 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:09.827 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:09.827 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:09.854 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:09.854 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:09.855 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:09.855 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:10.856 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:10.856 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:10.882 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:10.883 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:10.883 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:10.884 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:11.884 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:11.884 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:11.911 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:11.912 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:11.912 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:11.912 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:12.913 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:12.913 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:12.940 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:12.941 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:12.941 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:12.941 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:13.942 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:13.942 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:13.968 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:13.969 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:13.969 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:13.969 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:14.970 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:14.970 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:14.997 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:14.998 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:14.998 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:14.998 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:15.999 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:15.999 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:16.025 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:16.026 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:16.026 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:16.026 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:17.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:17.027 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:17.054 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:17.055 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:17.055 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:17.055 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:18.056 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:18.056 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:18.082 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:18.083 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:18.083 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:18.083 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:19.084 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:19.084 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:19.111 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:19.112 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:19.112 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:19.112 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:20.113 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:20.114 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:20.142 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:20.142 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:20.142 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:20.142 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:21.143 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:21.143 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:21.171 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:21.171 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:21.171 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:21.171 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:22.172 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:22.172 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:22.200 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:22.200 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:22.200 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:22.200 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:23.201 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:23.201 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:23.229 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:23.230 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:23.230 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:23.230 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:24.231 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:24.232 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:24.258 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:24.259 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:24.259 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:24.259 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:24.543 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:44:24.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:44:25.260 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:25.260 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:25.287 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:25.288 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:25.288 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:25.288 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:26.289 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:26.289 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:26.316 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:26.317 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:26.317 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:26.317 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:27.318 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:27.318 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:27.344 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:27.345 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:27.345 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:27.345 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:28.346 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:28.347 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:28.374 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:28.374 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:28.374 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:28.374 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:29.375 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:29.375 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:29.403 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:29.403 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:29.403 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:29.403 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:30.404 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:30.404 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:30.430 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:30.431 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:30.431 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:30.431 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:31.432 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:31.432 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:31.458 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:31.459 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:31.459 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:31.459 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:32.460 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:32.460 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:32.487 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:32.487 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:32.488 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:32.488 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:33.489 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:33.489 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:33.516 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:33.516 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:33.516 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:33.516 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:34.517 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:34.517 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:34.543 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:34.544 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:34.545 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:34.545 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:35.545 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:35.546 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:35.572 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:35.573 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:35.573 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:35.573 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:36.574 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:36.574 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:36.601 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:36.602 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:36.602 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:36.602 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:37.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:37.603 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:37.631 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:37.632 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:37.632 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:37.632 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:38.633 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:38.634 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:38.661 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:38.662 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:38.662 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:38.662 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:39.663 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:39.664 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:39.691 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:39.692 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:39.692 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:39.692 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:40.693 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:40.693 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:40.723 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:40.724 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:40.724 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:40.724 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:41.725 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:41.725 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:41.752 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:41.753 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:41.753 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:41.753 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:42.753 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:42.754 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:42.780 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:42.781 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:42.781 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:42.781 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:43.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:43.782 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:43.809 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:43.809 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:43.809 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:43.809 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:44.811 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:44.811 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:44.838 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:44.839 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:44.839 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:44.839 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:45.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:45.843 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:45.869 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:45.870 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:45.870 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:45.870 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:46.871 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:46.872 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:46.899 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:46.899 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:46.899 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:46.899 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:47.901 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:47.901 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:47.927 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:47.928 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:47.928 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:47.928 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:48.929 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:48.930 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:48.956 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:48.957 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:48.957 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:48.957 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:49.958 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:49.959 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:49.987 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:49.987 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:49.987 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:49.988 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:50.989 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:50.989 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:51.016 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:51.016 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:51.017 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:51.017 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:52.018 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:52.018 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:52.045 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:52.046 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:52.046 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:52.046 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:53.047 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:53.048 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:53.075 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:53.076 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:53.076 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:53.076 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:54.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:54.077 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:54.104 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:54.105 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:54.105 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:54.105 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:54.685 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:44:54.688 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:44:55.106 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:55.106 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:55.135 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:55.135 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:55.135 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:55.135 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:56.136 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:56.136 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:56.164 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:56.164 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:56.164 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:56.164 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:57.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:57.166 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:57.192 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:57.193 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:57.193 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:57.193 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:58.194 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:58.194 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:58.222 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:58.222 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:58.222 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:58.222 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:44:59.223 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:44:59.223 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:44:59.249 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:44:59.250 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:44:59.250 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:44:59.250 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:45:00.251 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:45:00.251 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:45:00.279 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:45:00.279 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_45_00_UTC 2024-01-26T23:45:00.279 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_45_00_UTC', 'scheduled-2024-01-26-23_44_00_UTC'} 2024-01-26T23:45:00.279 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-01-26-23_45_00_UTC'} 2024-01-26T23:45:00.279 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:45:00.280 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-01-26-23_45_00_UTC'] 2024-01-26T23:45:00.280 DEBUG:tasks.cephfs.test_snap_schedules:exec=1706312602.0907865, now = 1706312700.2802014, timo = 98 2024-01-26T23:45:01.281 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-01-26T23:45:01.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.424+0000 7fcba1ffc700 1 Processor -- start 2024-01-26T23:45:01.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.424+0000 7fcba1ffc700 1 -- start start 2024-01-26T23:45:01.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.425+0000 7fcba1ffc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb9c0c8da0 0x7fcb9c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:01.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.425+0000 7fcba1ffc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fcb9c140af0 0x7fcb9c140f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:01.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.425+0000 7fcba1ffc700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c0ccc00 con 0x7fcb9c0caf30 2024-01-26T23:45:01.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.425+0000 7fcba1ffc700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb9c0cbbf0 con 0x7fcb9c140af0 2024-01-26T23:45:01.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.425+0000 7fcba1ffc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c0cc310 con 0x7fcb9c0c8da0 2024-01-26T23:45:01.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.425+0000 7fcb9bfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcb9c0caf30 0x7fcb9c0cb370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35028/0 (socket says 172.21.15.46:35028) 2024-01-26T23:45:01.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.425+0000 7fcb9bfff700 1 -- 172.21.15.46:0/4084074540 learned_addr learned my addr 172.21.15.46:0/4084074540 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.425+0000 7fcb9affd700 1 --2- 172.21.15.46:0/4084074540 >> v2:172.21.15.106:3300/0 conn(0x7fcb9c140af0 0x7fcb9c140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.425+0000 7fcb9b7fe700 1 --2- 172.21.15.46:0/4084074540 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb9c0c8da0 0x7fcb9c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.425+0000 7fcb9a7fc700 1 -- 172.21.15.46:0/4084074540 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269478976 0 0) 0x7fcb9c0ccc00 con 0x7fcb9c0caf30 2024-01-26T23:45:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.425+0000 7fcb9a7fc700 1 -- 172.21.15.46:0/4084074540 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb84003570 con 0x7fcb9c0caf30 2024-01-26T23:45:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.425+0000 7fcb9b7fe700 1 -- 172.21.15.46:0/4084074540 >> v2:172.21.15.106:3300/0 conn(0x7fcb9c140af0 msgr2=0x7fcb9c140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.425+0000 7fcb9b7fe700 1 --2- 172.21.15.46:0/4084074540 >> v2:172.21.15.106:3300/0 conn(0x7fcb9c140af0 0x7fcb9c140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.425+0000 7fcb9b7fe700 1 -- 172.21.15.46:0/4084074540 >> v1:172.21.15.46:6789/0 conn(0x7fcb9c0caf30 legacy=0x7fcb9c0cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.425+0000 7fcb9b7fe700 1 -- 172.21.15.46:0/4084074540 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c0cc5b0 con 0x7fcb9c0c8da0 2024-01-26T23:45:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.425+0000 7fcb9affd700 1 --2- 172.21.15.46:0/4084074540 >> v2:172.21.15.106:3300/0 conn(0x7fcb9c140af0 0x7fcb9c140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.426+0000 7fcb9b7fe700 1 --2- 172.21.15.46:0/4084074540 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb9c0c8da0 0x7fcb9c0c9170 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0098e0 tx=0x7fcb8c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fda92156def6fe7a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.426+0000 7fcb9a7fc700 1 -- 172.21.15.46:0/4084074540 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c011630 con 0x7fcb9c0c8da0 2024-01-26T23:45:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.426+0000 7fcb9a7fc700 1 -- 172.21.15.46:0/4084074540 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8c011790 con 0x7fcb9c0c8da0 2024-01-26T23:45:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.426+0000 7fcb9a7fc700 1 -- 172.21.15.46:0/4084074540 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c011b30 con 0x7fcb9c0c8da0 2024-01-26T23:45:01.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.426+0000 7fcba1ffc700 1 -- 172.21.15.46:0/4084074540 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb9c0c8da0 msgr2=0x7fcb9c0c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.426+0000 7fcba1ffc700 1 --2- 172.21.15.46:0/4084074540 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb9c0c8da0 0x7fcb9c0c9170 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0098e0 tx=0x7fcb8c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.426+0000 7fcba1ffc700 1 -- 172.21.15.46:0/4084074540 shutdown_connections 2024-01-26T23:45:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.427+0000 7fcba1ffc700 1 --2- 172.21.15.46:0/4084074540 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb9c0c8da0 0x7fcb9c0c9170 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.427+0000 7fcba1ffc700 1 --2- 172.21.15.46:0/4084074540 >> v2:172.21.15.106:3300/0 conn(0x7fcb9c140af0 0x7fcb9c140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.427+0000 7fcba1ffc700 1 -- 172.21.15.46:0/4084074540 >> 172.21.15.46:0/4084074540 conn(0x7fcb9c06bd90 msgr2=0x7fcb9c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.427+0000 7fcba1ffc700 1 -- 172.21.15.46:0/4084074540 shutdown_connections 2024-01-26T23:45:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.427+0000 7fcba1ffc700 1 -- 172.21.15.46:0/4084074540 wait complete. 2024-01-26T23:45:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.428+0000 7fcba1ffc700 1 Processor -- start 2024-01-26T23:45:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.428+0000 7fcba1ffc700 1 -- start start 2024-01-26T23:45:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.429+0000 7fcba1ffc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb9c0c8da0 0x7fcb9c0afcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.429+0000 7fcba1ffc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb9c0caf30 0x7fcb9c0c6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.429+0000 7fcba1ffc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb9c140af0 0x7fcb9c15ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.429+0000 7fcba1ffc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c157e00 con 0x7fcb9c140af0 2024-01-26T23:45:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.429+0000 7fcba1ffc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c0b0230 con 0x7fcb9c0caf30 2024-01-26T23:45:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.429+0000 7fcba1ffc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c0c6a40 con 0x7fcb9c0c8da0 2024-01-26T23:45:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.429+0000 7fcb9bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb9c140af0 0x7fcb9c15ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.429+0000 7fcb9bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb9c140af0 0x7fcb9c15ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35898/0 (socket says 172.21.15.46:35898) 2024-01-26T23:45:01.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.429+0000 7fcb9bfff700 1 -- 172.21.15.46:0/1715920832 learned_addr learned my addr 172.21.15.46:0/1715920832 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:01.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.429+0000 7fcb9affd700 1 --2- 172.21.15.46:0/1715920832 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb9c0caf30 0x7fcb9c0c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:01.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.429+0000 7fcb9b7fe700 1 --2- 172.21.15.46:0/1715920832 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb9c0c8da0 0x7fcb9c0afcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:01.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.429+0000 7fcb9bfff700 1 -- 172.21.15.46:0/1715920832 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb9c0c8da0 msgr2=0x7fcb9c0afcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:01.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.429+0000 7fcb9bfff700 1 --2- 172.21.15.46:0/1715920832 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb9c0c8da0 0x7fcb9c0afcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:01.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.429+0000 7fcb9bfff700 1 -- 172.21.15.46:0/1715920832 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb9c0caf30 msgr2=0x7fcb9c0c6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:01.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.429+0000 7fcb9bfff700 1 --2- 172.21.15.46:0/1715920832 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb9c0caf30 0x7fcb9c0c6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:01.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.429+0000 7fcb9bfff700 1 -- 172.21.15.46:0/1715920832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb90009420 con 0x7fcb9c140af0 2024-01-26T23:45:01.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.429+0000 7fcb9b7fe700 1 --2- 172.21.15.46:0/1715920832 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb9c0c8da0 0x7fcb9c0afcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:01.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.430+0000 7fcb9bfff700 1 --2- 172.21.15.46:0/1715920832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb9c140af0 0x7fcb9c15ada0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fcb94001460 tx=0x7fcb94001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:01.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.430+0000 7fcb837fe700 1 -- 172.21.15.46:0/1715920832 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9400ce80 con 0x7fcb9c140af0 2024-01-26T23:45:01.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.430+0000 7fcb837fe700 1 -- 172.21.15.46:0/1715920832 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9400ab70 con 0x7fcb9c140af0 2024-01-26T23:45:01.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.430+0000 7fcba1ffc700 1 -- 172.21.15.46:0/1715920832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb8c009420 con 0x7fcb9c140af0 2024-01-26T23:45:01.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.430+0000 7fcb837fe700 1 -- 172.21.15.46:0/1715920832 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb940070f0 con 0x7fcb9c140af0 2024-01-26T23:45:01.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.430+0000 7fcba1ffc700 1 -- 172.21.15.46:0/1715920832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9c0c7000 con 0x7fcb9c140af0 2024-01-26T23:45:01.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.431+0000 7fcba1ffc700 1 -- 172.21.15.46:0/1715920832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9c157320 con 0x7fcb9c140af0 2024-01-26T23:45:01.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.431+0000 7fcb837fe700 1 -- 172.21.15.46:0/1715920832 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) v1 ==== 84873+0+0 (secure 0 0 0) 0x7fcb94007520 con 0x7fcb9c140af0 2024-01-26T23:45:01.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.432+0000 7fcb837fe700 1 --2- 172.21.15.46:0/1715920832 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcb880670a0 0x7fcb88069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:01.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.432+0000 7fcb837fe700 1 -- 172.21.15.46:0/1715920832 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6705+0+0 (secure 0 0 0) 0x7fcb94085050 con 0x7fcb9c140af0 2024-01-26T23:45:01.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.432+0000 7fcb9b7fe700 1 --2- 172.21.15.46:0/1715920832 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcb880670a0 0x7fcb88069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:01.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.432+0000 7fcb9b7fe700 1 --2- 172.21.15.46:0/1715920832 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcb880670a0 0x7fcb88069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0091e0 tx=0x7fcb8c017000 comp rx=0 tx=0).ready entity=mgr.5564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:01.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.436+0000 7fcb837fe700 1 -- 172.21.15.46:0/1715920832 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcb94051dc0 con 0x7fcb9c140af0 2024-01-26T23:45:01.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.675+0000 7fcba1ffc700 1 -- 172.21.15.46:0/1715920832 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb9c06e690 con 0x7fcb880670a0 2024-01-26T23:45:01.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.732+0000 7fcb837fe700 1 -- 172.21.15.46:0/1715920832 <== mgr.5564 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fcb9c06e690 con 0x7fcb880670a0 2024-01-26T23:45:01.734 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path snap_test_dir1 2024-01-26T23:45:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.734+0000 7fcba1ffc700 1 -- 172.21.15.46:0/1715920832 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcb880670a0 msgr2=0x7fcb88069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.734+0000 7fcba1ffc700 1 --2- 172.21.15.46:0/1715920832 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcb880670a0 0x7fcb88069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0091e0 tx=0x7fcb8c017000 comp rx=0 tx=0).stop 2024-01-26T23:45:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.734+0000 7fcba1ffc700 1 -- 172.21.15.46:0/1715920832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb9c140af0 msgr2=0x7fcb9c15ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.734+0000 7fcba1ffc700 1 --2- 172.21.15.46:0/1715920832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb9c140af0 0x7fcb9c15ada0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fcb94001460 tx=0x7fcb94001ae0 comp rx=0 tx=0).stop 2024-01-26T23:45:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.735+0000 7fcba1ffc700 1 -- 172.21.15.46:0/1715920832 shutdown_connections 2024-01-26T23:45:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.735+0000 7fcba1ffc700 1 --2- 172.21.15.46:0/1715920832 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fcb880670a0 0x7fcb88069550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.735+0000 7fcba1ffc700 1 --2- 172.21.15.46:0/1715920832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb9c140af0 0x7fcb9c15ada0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.735+0000 7fcba1ffc700 1 --2- 172.21.15.46:0/1715920832 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb9c0c8da0 0x7fcb9c0afcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.735+0000 7fcba1ffc700 1 --2- 172.21.15.46:0/1715920832 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb9c0caf30 0x7fcb9c0c6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.735+0000 7fcba1ffc700 1 -- 172.21.15.46:0/1715920832 >> 172.21.15.46:0/1715920832 conn(0x7fcb9c06bd90 msgr2=0x7fcb9c156c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.735+0000 7fcba1ffc700 1 -- 172.21.15.46:0/1715920832 shutdown_connections 2024-01-26T23:45:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:01.735+0000 7fcba1ffc700 1 -- 172.21.15.46:0/1715920832 wait complete. 2024-01-26T23:45:01.751 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-01-26T23:45:01.751 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:45:01.751 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:45:01.775 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:45:01.775 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_45_00_UTC 2024-01-26T23:45:01.775 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-26-23_44_00_UTC 2024-01-26T23:45:01.775 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:45:01.775 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-26-23_44_00_UTC) 2024-01-26T23:45:01.844 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-26-23_45_00_UTC 2024-01-26T23:45:01.844 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:45:01.845 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-26-23_45_00_UTC) 2024-01-26T23:45:01.914 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:45:01.915 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-01-26T23:45:01.982 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-26T23:45:02.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.164+0000 7f2741f20700 1 Processor -- start 2024-01-26T23:45:02.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.164+0000 7f2741f20700 1 -- start start 2024-01-26T23:45:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.165+0000 7f2741f20700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f273c068ed0 0x7f273c06a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.165+0000 7f2741f20700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f273c0a9a80 con 0x7f273c068b00 2024-01-26T23:45:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.165+0000 7f2741f20700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f273c15a680 con 0x7f273c06add0 2024-01-26T23:45:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.165+0000 7f2741f20700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f273c069320 con 0x7f273c068ed0 2024-01-26T23:45:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.165+0000 7f273b7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f273c068b00 0x7f273c159ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35042/0 (socket says 172.21.15.46:35042) 2024-01-26T23:45:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.165+0000 7f273b7fe700 1 -- 172.21.15.46:0/3319916268 learned_addr learned my addr 172.21.15.46:0/3319916268 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.165+0000 7f273affd700 1 --2- 172.21.15.46:0/3319916268 >> v2:172.21.15.106:3300/0 conn(0x7f273c068ed0 0x7f273c06a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.165+0000 7f273a7fc700 1 -- 172.21.15.46:0/3319916268 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299479123 0 0) 0x7f273c0a9a80 con 0x7f273c068b00 2024-01-26T23:45:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.165+0000 7f273a7fc700 1 -- 172.21.15.46:0/3319916268 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2724003570 con 0x7f273c068b00 2024-01-26T23:45:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.165+0000 7f273a7fc700 1 -- 172.21.15.46:0/3319916268 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2122831422 0 0) 0x7f2724003570 con 0x7f273c068b00 2024-01-26T23:45:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.165+0000 7f273affd700 1 -- 172.21.15.46:0/3319916268 >> v1:172.21.15.106:6789/0 conn(0x7f273c06add0 legacy=0x7f273c0a9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.165+0000 7f273affd700 1 -- 172.21.15.46:0/3319916268 >> v1:172.21.15.46:6789/0 conn(0x7f273c068b00 legacy=0x7f273c159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.165+0000 7f273affd700 1 -- 172.21.15.46:0/3319916268 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273c15a960 con 0x7f273c068ed0 2024-01-26T23:45:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.166+0000 7f273affd700 1 --2- 172.21.15.46:0/3319916268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f273c068ed0 0x7f273c06a660 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f27300098e0 tx=0x7f27300091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32576411457b6ab server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.166+0000 7f273a7fc700 1 -- 172.21.15.46:0/3319916268 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c0015f0 con 0x7f273c068ed0 2024-01-26T23:45:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.166+0000 7f273a7fc700 1 -- 172.21.15.46:0/3319916268 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2730011420 con 0x7f273c068ed0 2024-01-26T23:45:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.166+0000 7f273a7fc700 1 -- 172.21.15.46:0/3319916268 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27300117a0 con 0x7f273c068ed0 2024-01-26T23:45:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.166+0000 7f2741f20700 1 -- 172.21.15.46:0/3319916268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f273c068ed0 msgr2=0x7f273c06a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.166+0000 7f2741f20700 1 --2- 172.21.15.46:0/3319916268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f273c068ed0 0x7f273c06a660 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f27300098e0 tx=0x7f27300091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.166+0000 7f2741f20700 1 -- 172.21.15.46:0/3319916268 shutdown_connections 2024-01-26T23:45:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.166+0000 7f2741f20700 1 --2- 172.21.15.46:0/3319916268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f273c068ed0 0x7f273c06a660 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.166+0000 7f2741f20700 1 -- 172.21.15.46:0/3319916268 >> 172.21.15.46:0/3319916268 conn(0x7f273c0ca100 msgr2=0x7f273c06e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.166+0000 7f2741f20700 1 -- 172.21.15.46:0/3319916268 shutdown_connections 2024-01-26T23:45:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.167+0000 7f2741f20700 1 -- 172.21.15.46:0/3319916268 wait complete. 2024-01-26T23:45:02.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.168+0000 7f2741f20700 1 Processor -- start 2024-01-26T23:45:02.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.168+0000 7f2741f20700 1 -- start start 2024-01-26T23:45:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.169+0000 7f2741f20700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f273c068b00 0x7f273c0afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.169+0000 7f2741f20700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f273c068ed0 0x7f273c0ac1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.169+0000 7f2741f20700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f273c06add0 0x7f273c0ac6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.169+0000 7f2741f20700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273c0b1450 con 0x7f273c068b00 2024-01-26T23:45:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.169+0000 7f2741f20700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273c0b0840 con 0x7f273c068ed0 2024-01-26T23:45:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.169+0000 7f2741f20700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f273c0b0b20 con 0x7f273c06add0 2024-01-26T23:45:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.169+0000 7f273b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f273c068b00 0x7f273c0afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.170+0000 7f273b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f273c068b00 0x7f273c0afe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35900/0 (socket says 172.21.15.46:35900) 2024-01-26T23:45:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.170+0000 7f273b7fe700 1 -- 172.21.15.46:0/1463809053 learned_addr learned my addr 172.21.15.46:0/1463809053 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.170+0000 7f273affd700 1 --2- 172.21.15.46:0/1463809053 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f273c068ed0 0x7f273c0ac1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.170+0000 7f273bfff700 1 --2- 172.21.15.46:0/1463809053 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f273c06add0 0x7f273c0ac6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.170+0000 7f273b7fe700 1 -- 172.21.15.46:0/1463809053 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f273c06add0 msgr2=0x7f273c0ac6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.170+0000 7f273b7fe700 1 --2- 172.21.15.46:0/1463809053 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f273c06add0 0x7f273c0ac6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.170+0000 7f273b7fe700 1 -- 172.21.15.46:0/1463809053 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f273c068ed0 msgr2=0x7f273c0ac1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.170+0000 7f273b7fe700 1 --2- 172.21.15.46:0/1463809053 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f273c068ed0 0x7f273c0ac1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.170+0000 7f273b7fe700 1 -- 172.21.15.46:0/1463809053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2730009420 con 0x7f273c068b00 2024-01-26T23:45:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.170+0000 7f273b7fe700 1 --2- 172.21.15.46:0/1463809053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f273c068b00 0x7f273c0afe80 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f272c00c810 tx=0x7f272c0016c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:02.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.170+0000 7f27237fe700 1 -- 172.21.15.46:0/1463809053 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c0072e0 con 0x7f273c068b00 2024-01-26T23:45:02.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.170+0000 7f27237fe700 1 -- 172.21.15.46:0/1463809053 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f272c007460 con 0x7f273c068b00 2024-01-26T23:45:02.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.170+0000 7f27237fe700 1 -- 172.21.15.46:0/1463809053 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c0057f0 con 0x7f273c068b00 2024-01-26T23:45:02.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.170+0000 7f2741f20700 1 -- 172.21.15.46:0/1463809053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f273c0b0e00 con 0x7f273c068b00 2024-01-26T23:45:02.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.170+0000 7f2741f20700 1 -- 172.21.15.46:0/1463809053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f273c0adb90 con 0x7f273c068b00 2024-01-26T23:45:02.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.171+0000 7f2741f20700 1 -- 172.21.15.46:0/1463809053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f273c04c710 con 0x7f273c068b00 2024-01-26T23:45:02.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.172+0000 7f27237fe700 1 -- 172.21.15.46:0/1463809053 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) v1 ==== 84873+0+0 (secure 0 0 0) 0x7f272c005be0 con 0x7f273c068b00 2024-01-26T23:45:02.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.172+0000 7f27237fe700 1 --2- 172.21.15.46:0/1463809053 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f27280670a0 0x7f2728069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:02.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.172+0000 7f27237fe700 1 -- 172.21.15.46:0/1463809053 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6705+0+0 (secure 0 0 0) 0x7f272c0841d0 con 0x7f273c068b00 2024-01-26T23:45:02.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.172+0000 7f273affd700 1 --2- 172.21.15.46:0/1463809053 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f27280670a0 0x7f2728069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:02.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.173+0000 7f273affd700 1 --2- 172.21.15.46:0/1463809053 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f27280670a0 0x7f2728069550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2730009fd0 tx=0x7f2730016000 comp rx=0 tx=0).ready entity=mgr.5564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:02.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.177+0000 7f27237fe700 1 -- 172.21.15.46:0/1463809053 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f272c050f40 con 0x7f273c068b00 2024-01-26T23:45:02.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.445+0000 7f2741f20700 1 -- 172.21.15.46:0/1463809053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f273c15a270 con 0x7f273c068b00 2024-01-26T23:45:02.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.689+0000 7f27237fe700 1 -- 172.21.15.46:0/1463809053 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f272c051310 con 0x7f273c068b00 2024-01-26T23:45:02.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.691+0000 7f27237fe700 1 -- 172.21.15.46:0/1463809053 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v26) v1 ==== 93+0+0 (secure 0 0 0) 0x7f272c0549d0 con 0x7f273c068b00 2024-01-26T23:45:02.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.694+0000 7f2741f20700 1 -- 172.21.15.46:0/1463809053 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f27280670a0 msgr2=0x7f2728069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:02.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.694+0000 7f2741f20700 1 --2- 172.21.15.46:0/1463809053 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f27280670a0 0x7f2728069550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2730009fd0 tx=0x7f2730016000 comp rx=0 tx=0).stop 2024-01-26T23:45:02.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.694+0000 7f2741f20700 1 -- 172.21.15.46:0/1463809053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f273c068b00 msgr2=0x7f273c0afe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:02.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.694+0000 7f2741f20700 1 --2- 172.21.15.46:0/1463809053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f273c068b00 0x7f273c0afe80 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f272c00c810 tx=0x7f272c0016c0 comp rx=0 tx=0).stop 2024-01-26T23:45:02.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.694+0000 7f2741f20700 1 -- 172.21.15.46:0/1463809053 shutdown_connections 2024-01-26T23:45:02.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.694+0000 7f2741f20700 1 --2- 172.21.15.46:0/1463809053 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f27280670a0 0x7f2728069550 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.694+0000 7f2741f20700 1 --2- 172.21.15.46:0/1463809053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f273c068b00 0x7f273c0afe80 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.694+0000 7f2741f20700 1 --2- 172.21.15.46:0/1463809053 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f273c06add0 0x7f273c0ac6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.694+0000 7f2741f20700 1 --2- 172.21.15.46:0/1463809053 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f273c068ed0 0x7f273c0ac1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.694+0000 7f2741f20700 1 -- 172.21.15.46:0/1463809053 >> 172.21.15.46:0/1463809053 conn(0x7f273c0ca100 msgr2=0x7f273c157050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.694+0000 7f2741f20700 1 -- 172.21.15.46:0/1463809053 shutdown_connections 2024-01-26T23:45:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:02.695+0000 7f2741f20700 1 -- 172.21.15.46:0/1463809053 wait complete. 2024-01-26T23:45:02.711 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-26T23:45:02.711 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:45:02.758 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:45:02.799 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-26T23:45:02.799 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-26T23:45:02.799 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-26T23:45:02.800 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-26T23:45:02.836 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[67642]: fuse finished with error 0 and tester_r 0 2024-01-26T23:45:02.837 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:45:02.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:45:02.838 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-26T23:45:03.011 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:45:03.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:45:03.011 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-26T23:45:03.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:45:03.028 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-26T23:45:03.028 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:45:03.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:45:03.028 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-26T23:45:03.031 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:03.030+0000 7fa6be3621c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:45:03.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:45:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-26T23:45:03.070 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:45:03.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:45:03.070 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-26T23:45:03.074 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:03.073+0000 7fe195ee01c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:45:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:45:03.087 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-26T23:45:03.087 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:45:03.111 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:03.110+0000 7fa6be3621c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:45:03.166 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:03.165+0000 7fe195ee01c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:45:03.182 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:03.181+0000 7fa6be3621c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:45:03.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.222+0000 7fd3b6852700 1 Processor -- start 2024-01-26T23:45:03.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.222+0000 7fd3b6852700 1 -- start start 2024-01-26T23:45:03.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.223+0000 7fd3b6852700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3b00b1fc0 0x7fd3b00b2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:03.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.223+0000 7fd3b6852700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd3b0068310 0x7fd3b0068780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:03.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.223+0000 7fd3b6852700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b00cac60 con 0x7fd3b0156f80 2024-01-26T23:45:03.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.223+0000 7fd3b6852700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3b0068dc0 con 0x7fd3b0068310 2024-01-26T23:45:03.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.223+0000 7fd3b6852700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b00690a0 con 0x7fd3b00b1fc0 2024-01-26T23:45:03.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.223+0000 7fd3affff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3b00b1fc0 0x7fd3b00b2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:03.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.223+0000 7fd3af7fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd3b0068310 0x7fd3b0068780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:03.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.223+0000 7fd3b4def700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fd3b0156f80 0x7fd3b01573b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:59482/0 (socket says 172.21.15.46:59482) 2024-01-26T23:45:03.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.223+0000 7fd3b4def700 1 -- 172.21.15.46:0/3503959363 learned_addr learned my addr 172.21.15.46:0/3503959363 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:03.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.223+0000 7fd3aeffd700 1 -- 172.21.15.46:0/3503959363 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890420126 0 0) 0x7fd3b00cac60 con 0x7fd3b0156f80 2024-01-26T23:45:03.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.223+0000 7fd3affff700 1 -- 172.21.15.46:0/3503959363 >> v2:172.21.15.106:3300/0 conn(0x7fd3b0068310 msgr2=0x7fd3b0068780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:03.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.224+0000 7fd3affff700 1 --2- 172.21.15.46:0/3503959363 >> v2:172.21.15.106:3300/0 conn(0x7fd3b0068310 0x7fd3b0068780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.224+0000 7fd3affff700 1 -- 172.21.15.46:0/3503959363 >> v1:172.21.15.106:6789/0 conn(0x7fd3b0156f80 legacy=0x7fd3b01573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:03.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.224+0000 7fd3affff700 1 -- 172.21.15.46:0/3503959363 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b00caee0 con 0x7fd3b00b1fc0 2024-01-26T23:45:03.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.224+0000 7fd3af7fe700 1 --2- 172.21.15.46:0/3503959363 >> v2:172.21.15.106:3300/0 conn(0x7fd3b0068310 0x7fd3b0068780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:03.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.224+0000 7fd3affff700 1 --2- 172.21.15.46:0/3503959363 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3b00b1fc0 0x7fd3b00b2390 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd3a00098e0 tx=0x7fd3a00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b4d4e1df3545984 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:03.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.224+0000 7fd3aeffd700 1 -- 172.21.15.46:0/3503959363 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a000b640 con 0x7fd3b00b1fc0 2024-01-26T23:45:03.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.224+0000 7fd3aeffd700 1 -- 172.21.15.46:0/3503959363 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a000b7a0 con 0x7fd3b00b1fc0 2024-01-26T23:45:03.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.224+0000 7fd3aeffd700 1 -- 172.21.15.46:0/3503959363 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a000bb40 con 0x7fd3b00b1fc0 2024-01-26T23:45:03.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.225+0000 7fd3b6852700 1 -- 172.21.15.46:0/3503959363 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3b00b1fc0 msgr2=0x7fd3b00b2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:03.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.225+0000 7fd3b6852700 1 --2- 172.21.15.46:0/3503959363 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3b00b1fc0 0x7fd3b00b2390 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd3a00098e0 tx=0x7fd3a00091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.225+0000 7fd3b6852700 1 -- 172.21.15.46:0/3503959363 shutdown_connections 2024-01-26T23:45:03.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.225+0000 7fd3b6852700 1 --2- 172.21.15.46:0/3503959363 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3b00b1fc0 0x7fd3b00b2390 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.225+0000 7fd3b6852700 1 --2- 172.21.15.46:0/3503959363 >> v2:172.21.15.106:3300/0 conn(0x7fd3b0068310 0x7fd3b0068780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.225+0000 7fd3b6852700 1 -- 172.21.15.46:0/3503959363 >> 172.21.15.46:0/3503959363 conn(0x7fd3b00ccbc0 msgr2=0x7fd3b00ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:03.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.225+0000 7fd3b6852700 1 -- 172.21.15.46:0/3503959363 shutdown_connections 2024-01-26T23:45:03.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.225+0000 7fd3b6852700 1 -- 172.21.15.46:0/3503959363 wait complete. 2024-01-26T23:45:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.226+0000 7fd3b6852700 1 Processor -- start 2024-01-26T23:45:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.226+0000 7fd3b6852700 1 -- start start 2024-01-26T23:45:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.227+0000 7fd3b6852700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b0068310 0x7fd3b013f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:03.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.227+0000 7fd3b6852700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3b00b1fc0 0x7fd3b00c29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:03.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.227+0000 7fd3b6852700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3b0156f80 0x7fd3b00c5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:03.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.227+0000 7fd3b6852700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b013f6f0 con 0x7fd3b0068310 2024-01-26T23:45:03.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.227+0000 7fd3b6852700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b015dc80 con 0x7fd3b00b1fc0 2024-01-26T23:45:03.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.227+0000 7fd3b6852700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b015d070 con 0x7fd3b0156f80 2024-01-26T23:45:03.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.227+0000 7fd3af7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3b00b1fc0 0x7fd3b00c29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:03.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.227+0000 7fd3af7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3b00b1fc0 0x7fd3b00c29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:59196/0 (socket says 172.21.15.46:59196) 2024-01-26T23:45:03.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.227+0000 7fd3af7fe700 1 -- 172.21.15.46:0/797302502 learned_addr learned my addr 172.21.15.46:0/797302502 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:03.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.227+0000 7fd3b4def700 1 --2- 172.21.15.46:0/797302502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3b0156f80 0x7fd3b00c5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:03.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.228+0000 7fd3b4def700 1 -- 172.21.15.46:0/797302502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3b00b1fc0 msgr2=0x7fd3b00c29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:03.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.228+0000 7fd3b4def700 1 --2- 172.21.15.46:0/797302502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3b00b1fc0 0x7fd3b00c29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.228+0000 7fd3b4def700 1 -- 172.21.15.46:0/797302502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b0068310 msgr2=0x7fd3b013f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:03.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.228+0000 7fd3b4def700 1 --2- 172.21.15.46:0/797302502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b0068310 0x7fd3b013f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.228+0000 7fd3b4def700 1 -- 172.21.15.46:0/797302502 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a4009420 con 0x7fd3b0156f80 2024-01-26T23:45:03.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.228+0000 7fd3af7fe700 1 --2- 172.21.15.46:0/797302502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3b00b1fc0 0x7fd3b00c29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:03.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.228+0000 7fd3b4def700 1 --2- 172.21.15.46:0/797302502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3b0156f80 0x7fd3b00c5390 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd3a800cb90 tx=0x7fd3a80018f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:03.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.228+0000 7fd397fff700 1 -- 172.21.15.46:0/797302502 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a800a850 con 0x7fd3b0156f80 2024-01-26T23:45:03.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.228+0000 7fd397fff700 1 -- 172.21.15.46:0/797302502 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a800a9b0 con 0x7fd3b0156f80 2024-01-26T23:45:03.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.228+0000 7fd3b6852700 1 -- 172.21.15.46:0/797302502 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a0009420 con 0x7fd3b0156f80 2024-01-26T23:45:03.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.228+0000 7fd397fff700 1 -- 172.21.15.46:0/797302502 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a800ad70 con 0x7fd3b0156f80 2024-01-26T23:45:03.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.228+0000 7fd3b6852700 1 -- 172.21.15.46:0/797302502 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b015d630 con 0x7fd3b0156f80 2024-01-26T23:45:03.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.229+0000 7fd3b6852700 1 -- 172.21.15.46:0/797302502 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b004c710 con 0x7fd3b0156f80 2024-01-26T23:45:03.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.229+0000 7fd397fff700 1 -- 172.21.15.46:0/797302502 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fd3a8007300 con 0x7fd3b0156f80 2024-01-26T23:45:03.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.230+0000 7fd397fff700 1 --2- 172.21.15.46:0/797302502 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd39c067050 0x7fd39c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:03.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.230+0000 7fd3affff700 1 -- 172.21.15.46:0/797302502 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd39c067050 msgr2=0x7fd39c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:03.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.230+0000 7fd397fff700 1 -- 172.21.15.46:0/797302502 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6705+0+0 (secure 0 0 0) 0x7fd3a8084d80 con 0x7fd3b0156f80 2024-01-26T23:45:03.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.230+0000 7fd3affff700 1 --2- 172.21.15.46:0/797302502 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd39c067050 0x7fd39c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.234+0000 7fd397fff700 1 -- 172.21.15.46:0/797302502 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd3a8051b70 con 0x7fd3b0156f80 2024-01-26T23:45:03.248 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:03.248+0000 7fa6be3621c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:45:03.249 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:03.248+0000 7fe195ee01c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:45:03.325 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:03.323+0000 7fe195ee01c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:45:03.404 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:03.403+0000 7fa6be3621c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:45:03.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.431+0000 7fd3affff700 1 -- 172.21.15.46:0/797302502 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd39c067050 msgr2=0x7fd39c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:03.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.431+0000 7fd3affff700 1 --2- 172.21.15.46:0/797302502 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd39c067050 0x7fd39c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:03.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.491+0000 7fd3b6852700 1 -- 172.21.15.46:0/797302502 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3b00cb1c0 con 0x7fd3b0156f80 2024-01-26T23:45:03.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.491+0000 7fd397fff700 1 -- 172.21.15.46:0/797302502 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4603 (secure 0 0 0) 0x7fd3a8055190 con 0x7fd3b0156f80 2024-01-26T23:45:03.493 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:03.493 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3037972557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3037972557},{"type":"v1","addr":"172.21.15.106:6837","nonce":3037972557}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":5701,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3786017823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3786017823},{"type":"v1","addr":"172.21.15.106:6836","nonce":3786017823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":5704,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3329385210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3329385210},{"type":"v1","addr":"172.21.15.46:6835","nonce":3329385210}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:43:01.144129+0000","modified":"2024-01-26T23:43:04.718067+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6833/2801738403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2801738403},{"type":"v1","addr":"172.21.15.46:6833","nonce":2801738403}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2024-01-26T23:45:03.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.494+0000 7fd3b6852700 1 -- 172.21.15.46:0/797302502 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd39c067050 msgr2=0x7fd39c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:03.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.494+0000 7fd3b6852700 1 --2- 172.21.15.46:0/797302502 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd39c067050 0x7fd39c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.494+0000 7fd3b6852700 1 -- 172.21.15.46:0/797302502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3b0156f80 msgr2=0x7fd3b00c5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:03.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.494+0000 7fd3b6852700 1 --2- 172.21.15.46:0/797302502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3b0156f80 0x7fd3b00c5390 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd3a800cb90 tx=0x7fd3a80018f0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.494+0000 7fd3b6852700 1 -- 172.21.15.46:0/797302502 shutdown_connections 2024-01-26T23:45:03.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.494+0000 7fd3b6852700 1 --2- 172.21.15.46:0/797302502 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd39c067050 0x7fd39c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.494+0000 7fd3b6852700 1 --2- 172.21.15.46:0/797302502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b0068310 0x7fd3b013f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.494+0000 7fd3b6852700 1 --2- 172.21.15.46:0/797302502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3b0156f80 0x7fd3b00c5390 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.494+0000 7fd3b6852700 1 --2- 172.21.15.46:0/797302502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3b00b1fc0 0x7fd3b00c29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.494+0000 7fd3b6852700 1 -- 172.21.15.46:0/797302502 >> 172.21.15.46:0/797302502 conn(0x7fd3b00ccbc0 msgr2=0x7fd3b0155e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:03.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.494+0000 7fd3b6852700 1 -- 172.21.15.46:0/797302502 shutdown_connections 2024-01-26T23:45:03.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.494+0000 7fd3b6852700 1 -- 172.21.15.46:0/797302502 wait complete. 2024-01-26T23:45:03.496 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 27 2024-01-26T23:45:03.501 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:03.500+0000 7fe195ee01c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:45:03.510 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:45:03.541 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:03.540+0000 7fa6be3621c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:45:03.607 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:03.606+0000 7fa6be3621c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:45:03.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.641+0000 7fddb75da700 1 Processor -- start 2024-01-26T23:45:03.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.642+0000 7fddb75da700 1 -- start start 2024-01-26T23:45:03.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.643+0000 7fddb75da700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fddb00cc1c0 0x7fddb0164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:03.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.643+0000 7fddb75da700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb00cff70 con 0x7fddb015ce40 2024-01-26T23:45:03.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.643+0000 7fddb75da700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb00cf400 con 0x7fddb015d210 2024-01-26T23:45:03.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.643+0000 7fddb75da700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb00cf660 con 0x7fddb00cc1c0 2024-01-26T23:45:03.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.643+0000 7fddb5b77700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fddb00cc1c0 0x7fddb0164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:03.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.643+0000 7fddb5376700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fddb015ce40 0x7fddb00d0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35056/0 (socket says 172.21.15.46:35056) 2024-01-26T23:45:03.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.643+0000 7fddb5376700 1 -- 172.21.15.46:0/1087990693 learned_addr learned my addr 172.21.15.46:0/1087990693 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:03.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.643+0000 7fddb5b77700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fddb00cc1c0 0x7fddb0164700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35942/0 (socket says 172.21.15.46:35942) 2024-01-26T23:45:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.643+0000 7fddb5b77700 1 -- 172.21.15.46:0/1087990693 >> v1:172.21.15.106:6789/0 conn(0x7fddb015d210 legacy=0x7fddb00cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.643+0000 7fddb5b77700 1 -- 172.21.15.46:0/1087990693 >> v1:172.21.15.46:6789/0 conn(0x7fddb015ce40 legacy=0x7fddb00d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.643+0000 7fddaffff700 1 -- 172.21.15.46:0/1087990693 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075399179 0 0) 0x7fddb00cff70 con 0x7fddb015ce40 2024-01-26T23:45:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.643+0000 7fddb5b77700 1 -- 172.21.15.46:0/1087990693 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb00d0500 con 0x7fddb00cc1c0 2024-01-26T23:45:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.643+0000 7fddb5b77700 1 --2- 172.21.15.46:0/1087990693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb00cc1c0 0x7fddb0164700 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fdda80098e0 tx=0x7fdda80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8dee7bf8109cd45 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.644+0000 7fddaffff700 1 -- 172.21.15.46:0/1087990693 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda8009e80 con 0x7fddb00cc1c0 2024-01-26T23:45:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.644+0000 7fddaffff700 1 -- 172.21.15.46:0/1087990693 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda800b860 con 0x7fddb00cc1c0 2024-01-26T23:45:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.644+0000 7fddaffff700 1 -- 172.21.15.46:0/1087990693 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda800bbe0 con 0x7fddb00cc1c0 2024-01-26T23:45:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.644+0000 7fddb75da700 1 -- 172.21.15.46:0/1087990693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb00cc1c0 msgr2=0x7fddb0164700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.644+0000 7fddb75da700 1 --2- 172.21.15.46:0/1087990693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb00cc1c0 0x7fddb0164700 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fdda80098e0 tx=0x7fdda80091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.644+0000 7fddb75da700 1 -- 172.21.15.46:0/1087990693 shutdown_connections 2024-01-26T23:45:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.644+0000 7fddb75da700 1 --2- 172.21.15.46:0/1087990693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb00cc1c0 0x7fddb0164700 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.644+0000 7fddb75da700 1 -- 172.21.15.46:0/1087990693 >> 172.21.15.46:0/1087990693 conn(0x7fddb006f840 msgr2=0x7fddb00d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.644+0000 7fddb75da700 1 -- 172.21.15.46:0/1087990693 shutdown_connections 2024-01-26T23:45:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.644+0000 7fddb75da700 1 -- 172.21.15.46:0/1087990693 wait complete. 2024-01-26T23:45:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.645+0000 7fddb75da700 1 Processor -- start 2024-01-26T23:45:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.646+0000 7fddb75da700 1 -- start start 2024-01-26T23:45:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.646+0000 7fddb75da700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb00cc1c0 0x7fddb016dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.646+0000 7fddb75da700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fddb015ce40 0x7fddb00d0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.646+0000 7fddb75da700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fddb015d210 0x7fddb015a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.646+0000 7fddb75da700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb00ca6c0 con 0x7fddb00cc1c0 2024-01-26T23:45:03.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.646+0000 7fddb75da700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb00c9180 con 0x7fddb015d210 2024-01-26T23:45:03.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.646+0000 7fddb75da700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb016e200 con 0x7fddb015ce40 2024-01-26T23:45:03.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.647+0000 7fddb5376700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb00cc1c0 0x7fddb016dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:03.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.647+0000 7fddb5376700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb00cc1c0 0x7fddb016dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35952/0 (socket says 172.21.15.46:35952) 2024-01-26T23:45:03.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.647+0000 7fddb5376700 1 -- 172.21.15.46:0/2823619093 learned_addr learned my addr 172.21.15.46:0/2823619093 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:03.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.647+0000 7fddb5b77700 1 --2- 172.21.15.46:0/2823619093 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fddb015d210 0x7fddb015a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:03.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.647+0000 7fddb4b75700 1 --2- 172.21.15.46:0/2823619093 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fddb015ce40 0x7fddb00d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:03.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.647+0000 7fddb5376700 1 -- 172.21.15.46:0/2823619093 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fddb015ce40 msgr2=0x7fddb00d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:03.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.647+0000 7fddb5376700 1 --2- 172.21.15.46:0/2823619093 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fddb015ce40 0x7fddb00d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.647+0000 7fddb5376700 1 -- 172.21.15.46:0/2823619093 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fddb015d210 msgr2=0x7fddb015a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:03.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.647+0000 7fddb5376700 1 --2- 172.21.15.46:0/2823619093 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fddb015d210 0x7fddb015a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.647+0000 7fddb5376700 1 -- 172.21.15.46:0/2823619093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda8009420 con 0x7fddb00cc1c0 2024-01-26T23:45:03.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.647+0000 7fddb5376700 1 --2- 172.21.15.46:0/2823619093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb00cc1c0 0x7fddb016dcc0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fdda000cb40 tx=0x7fdda00017b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:03.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.647+0000 7fddad7fa700 1 -- 172.21.15.46:0/2823619093 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda0002080 con 0x7fddb00cc1c0 2024-01-26T23:45:03.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.647+0000 7fddad7fa700 1 -- 172.21.15.46:0/2823619093 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda0005d80 con 0x7fddb00cc1c0 2024-01-26T23:45:03.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.647+0000 7fddad7fa700 1 -- 172.21.15.46:0/2823619093 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda000a7e0 con 0x7fddb00cc1c0 2024-01-26T23:45:03.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.647+0000 7fddb75da700 1 -- 172.21.15.46:0/2823619093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb00c9ba0 con 0x7fddb00cc1c0 2024-01-26T23:45:03.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.648+0000 7fddb75da700 1 -- 172.21.15.46:0/2823619093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb00ca3e0 con 0x7fddb00cc1c0 2024-01-26T23:45:03.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.648+0000 7fddb75da700 1 -- 172.21.15.46:0/2823619093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb015df80 con 0x7fddb00cc1c0 2024-01-26T23:45:03.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.649+0000 7fddad7fa700 1 -- 172.21.15.46:0/2823619093 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fdda000abd0 con 0x7fddb00cc1c0 2024-01-26T23:45:03.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.649+0000 7fddad7fa700 1 --2- 172.21.15.46:0/2823619093 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdd9c067050 0x7fdd9c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:03.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.650+0000 7fddb4b75700 1 -- 172.21.15.46:0/2823619093 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdd9c067050 msgr2=0x7fdd9c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:03.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.650+0000 7fddad7fa700 1 -- 172.21.15.46:0/2823619093 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6705+0+0 (secure 0 0 0) 0x7fdda0084680 con 0x7fddb00cc1c0 2024-01-26T23:45:03.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.650+0000 7fddb4b75700 1 --2- 172.21.15.46:0/2823619093 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdd9c067050 0x7fdd9c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:03.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.654+0000 7fddad7fa700 1 -- 172.21.15.46:0/2823619093 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdda0051470 con 0x7fddb00cc1c0 2024-01-26T23:45:03.662 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:03.661+0000 7fe195ee01c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:45:03.728 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:03.727+0000 7fa6be3621c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:45:03.739 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:03.738+0000 7fe195ee01c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:45:03.800 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:03.799+0000 7fa6be3621c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:45:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.851+0000 7fddb4b75700 1 -- 172.21.15.46:0/2823619093 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdd9c067050 msgr2=0x7fdd9c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.851+0000 7fddb4b75700 1 --2- 172.21.15.46:0/2823619093 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdd9c067050 0x7fdd9c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:03.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.864+0000 7fddb75da700 1 -- 172.21.15.46:0/2823619093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddb004c710 con 0x7fddb00cc1c0 2024-01-26T23:45:03.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.865+0000 7fddad7fa700 1 -- 172.21.15.46:0/2823619093 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+16571 (secure 0 0 0) 0x7fdda0054a90 con 0x7fddb00cc1c0 2024-01-26T23:45:03.867 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:03.867 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":73,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:44:21.651311+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:42:59.650806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-01-26T23:43:00.105558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2","mappings":[{"from":0,"to":6}]},{"pgid":"8.1d","mappings":[{"from":0,"to":6}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/1676475004":"2024-01-27T23:43:20.540793+0000","172.21.15.106:6833/43388":"2024-01-27T23:43:20.540793+0000","172.21.15.106:6832/43388":"2024-01-27T23:43:20.540793+0000","172.21.15.106:0/2801680181":"2024-01-27T23:43:20.540793+0000","172.21.15.106:0/677673260":"2024-01-27T23:43:20.540793+0000","172.21.15.106:0/904952331":"2024-01-27T23:43:20.540793+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:45:03.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.867+0000 7fddb75da700 1 -- 172.21.15.46:0/2823619093 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdd9c067050 msgr2=0x7fdd9c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:03.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.867+0000 7fddb75da700 1 --2- 172.21.15.46:0/2823619093 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdd9c067050 0x7fdd9c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.867+0000 7fddb75da700 1 -- 172.21.15.46:0/2823619093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb00cc1c0 msgr2=0x7fddb016dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:03.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.867+0000 7fddb75da700 1 --2- 172.21.15.46:0/2823619093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb00cc1c0 0x7fddb016dcc0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fdda000cb40 tx=0x7fdda00017b0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.868+0000 7fddb75da700 1 -- 172.21.15.46:0/2823619093 shutdown_connections 2024-01-26T23:45:03.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.868+0000 7fddb75da700 1 --2- 172.21.15.46:0/2823619093 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdd9c067050 0x7fdd9c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.868+0000 7fddb75da700 1 --2- 172.21.15.46:0/2823619093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb00cc1c0 0x7fddb016dcc0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.868+0000 7fddb75da700 1 --2- 172.21.15.46:0/2823619093 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fddb015d210 0x7fddb015a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.868+0000 7fddb75da700 1 --2- 172.21.15.46:0/2823619093 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fddb015ce40 0x7fddb00d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:03.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.868+0000 7fddb75da700 1 -- 172.21.15.46:0/2823619093 >> 172.21.15.46:0/2823619093 conn(0x7fddb006f840 msgr2=0x7fddb00689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:03.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.868+0000 7fddb75da700 1 -- 172.21.15.46:0/2823619093 shutdown_connections 2024-01-26T23:45:03.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:03.868+0000 7fddb75da700 1 -- 172.21.15.46:0/2823619093 wait complete. 2024-01-26T23:45:03.880 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:03.879+0000 7fe195ee01c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:45:03.883 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-26T23:42:59.650806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-01-26T23:43:00.105558+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-26T23:45:03.883 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:45:03.940 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:03.939+0000 7fa6be3621c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:45:03.966 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:03.965+0000 7fe195ee01c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:45:04.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.018+0000 7fac59fd1700 1 Processor -- start 2024-01-26T23:45:04.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.018+0000 7fac59fd1700 1 -- start start 2024-01-26T23:45:04.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.019+0000 7fac59fd1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fac540cc380 0x7fac540cc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:04.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.019+0000 7fac59fd1700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac54069260 con 0x7fac54159510 2024-01-26T23:45:04.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.019+0000 7fac59fd1700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5415d670 con 0x7fac541598e0 2024-01-26T23:45:04.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.019+0000 7fac59fd1700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac540ca930 con 0x7fac540cc380 2024-01-26T23:45:04.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.019+0000 7fac53fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fac540cc380 0x7fac540cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:04.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.019+0000 7fac537fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fac54159510 0x7fac540ca220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35066/0 (socket says 172.21.15.46:35066) 2024-01-26T23:45:04.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.019+0000 7fac537fe700 1 -- 172.21.15.46:0/1923006831 learned_addr learned my addr 172.21.15.46:0/1923006831 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:04.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.020+0000 7fac53fff700 1 -- 172.21.15.46:0/1923006831 >> v1:172.21.15.106:6789/0 conn(0x7fac541598e0 legacy=0x7fac540c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:04.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.020+0000 7fac53fff700 1 -- 172.21.15.46:0/1923006831 >> v1:172.21.15.46:6789/0 conn(0x7fac54159510 legacy=0x7fac540ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:04.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.020+0000 7fac53fff700 1 -- 172.21.15.46:0/1923006831 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac540c9750 con 0x7fac540cc380 2024-01-26T23:45:04.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.020+0000 7fac527fc700 1 -- 172.21.15.46:0/1923006831 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864226525 0 0) 0x7fac54069260 con 0x7fac54159510 2024-01-26T23:45:04.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.020+0000 7fac53fff700 1 --2- 172.21.15.46:0/1923006831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac540cc380 0x7fac540cc800 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fac4c009e60 tx=0x7fac4c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c0ca97b36b195 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:04.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.020+0000 7fac527fc700 1 -- 172.21.15.46:0/1923006831 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c00e070 con 0x7fac540cc380 2024-01-26T23:45:04.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.020+0000 7fac527fc700 1 -- 172.21.15.46:0/1923006831 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4c00bc30 con 0x7fac540cc380 2024-01-26T23:45:04.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.020+0000 7fac527fc700 1 -- 172.21.15.46:0/1923006831 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c012430 con 0x7fac540cc380 2024-01-26T23:45:04.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.021+0000 7fac59fd1700 1 -- 172.21.15.46:0/1923006831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac540cc380 msgr2=0x7fac540cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:04.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.021+0000 7fac59fd1700 1 --2- 172.21.15.46:0/1923006831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac540cc380 0x7fac540cc800 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fac4c009e60 tx=0x7fac4c009780 comp rx=0 tx=0).stop 2024-01-26T23:45:04.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.021+0000 7fac59fd1700 1 -- 172.21.15.46:0/1923006831 shutdown_connections 2024-01-26T23:45:04.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.021+0000 7fac59fd1700 1 --2- 172.21.15.46:0/1923006831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac540cc380 0x7fac540cc800 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.021+0000 7fac59fd1700 1 -- 172.21.15.46:0/1923006831 >> 172.21.15.46:0/1923006831 conn(0x7fac5406bd90 msgr2=0x7fac5406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:04.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.021+0000 7fac59fd1700 1 -- 172.21.15.46:0/1923006831 shutdown_connections 2024-01-26T23:45:04.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.021+0000 7fac59fd1700 1 -- 172.21.15.46:0/1923006831 wait complete. 2024-01-26T23:45:04.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.022+0000 7fac59fd1700 1 Processor -- start 2024-01-26T23:45:04.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.022+0000 7fac59fd1700 1 -- start start 2024-01-26T23:45:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.023+0000 7fac59fd1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac540cc380 0x7fac54154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.023+0000 7fac59fd1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac54159510 0x7fac54157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.023+0000 7fac59fd1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac541598e0 0x7fac540c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.023+0000 7fac59fd1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac54159d50 con 0x7fac541598e0 2024-01-26T23:45:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.023+0000 7fac59fd1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac540c8540 con 0x7fac54159510 2024-01-26T23:45:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.023+0000 7fac59fd1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac541578d0 con 0x7fac540cc380 2024-01-26T23:45:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.023+0000 7fac53fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac541598e0 0x7fac540c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.023+0000 7fac53fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac541598e0 0x7fac540c77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35978/0 (socket says 172.21.15.46:35978) 2024-01-26T23:45:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.023+0000 7fac53fff700 1 -- 172.21.15.46:0/3457006648 learned_addr learned my addr 172.21.15.46:0/3457006648 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.023+0000 7fac52ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac54159510 0x7fac54157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.023+0000 7fac537fe700 1 --2- 172.21.15.46:0/3457006648 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac540cc380 0x7fac54154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.024+0000 7fac53fff700 1 -- 172.21.15.46:0/3457006648 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac540cc380 msgr2=0x7fac54154240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.024+0000 7fac53fff700 1 --2- 172.21.15.46:0/3457006648 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac540cc380 0x7fac54154240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.024+0000 7fac53fff700 1 -- 172.21.15.46:0/3457006648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac54159510 msgr2=0x7fac54157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.024+0000 7fac53fff700 1 --2- 172.21.15.46:0/3457006648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac54159510 0x7fac54157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.024+0000 7fac53fff700 1 -- 172.21.15.46:0/3457006648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4c009a20 con 0x7fac541598e0 2024-01-26T23:45:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.024+0000 7fac53fff700 1 --2- 172.21.15.46:0/3457006648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac541598e0 0x7fac540c77a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0099f0 tx=0x7fac4c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.024+0000 7fac3b7fe700 1 -- 172.21.15.46:0/3457006648 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c012e10 con 0x7fac541598e0 2024-01-26T23:45:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.024+0000 7fac3b7fe700 1 -- 172.21.15.46:0/3457006648 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4c01d630 con 0x7fac541598e0 2024-01-26T23:45:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.024+0000 7fac59fd1700 1 -- 172.21.15.46:0/3457006648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac54154780 con 0x7fac541598e0 2024-01-26T23:45:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.024+0000 7fac3b7fe700 1 -- 172.21.15.46:0/3457006648 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c01d9f0 con 0x7fac541598e0 2024-01-26T23:45:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.024+0000 7fac59fd1700 1 -- 172.21.15.46:0/3457006648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac540c82e0 con 0x7fac541598e0 2024-01-26T23:45:04.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.026+0000 7fac59fd1700 1 -- 172.21.15.46:0/3457006648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5404c710 con 0x7fac541598e0 2024-01-26T23:45:04.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.026+0000 7fac3b7fe700 1 -- 172.21.15.46:0/3457006648 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fac4c024070 con 0x7fac541598e0 2024-01-26T23:45:04.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.026+0000 7fac3b7fe700 1 --2- 172.21.15.46:0/3457006648 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fac40067050 0x7fac40069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:04.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.026+0000 7fac3b7fe700 1 -- 172.21.15.46:0/3457006648 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6705+0+0 (secure 0 0 0) 0x7fac4c0872a0 con 0x7fac541598e0 2024-01-26T23:45:04.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.026+0000 7fac537fe700 1 -- 172.21.15.46:0/3457006648 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fac40067050 msgr2=0x7fac40069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:04.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.026+0000 7fac537fe700 1 --2- 172.21.15.46:0/3457006648 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fac40067050 0x7fac40069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:04.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.031+0000 7fac3b7fe700 1 -- 172.21.15.46:0/3457006648 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fac4c054010 con 0x7fac541598e0 2024-01-26T23:45:04.066 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:04.065+0000 7fa6be3621c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:45:04.139 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:04.137+0000 7fe195ee01c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:45:04.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.227+0000 7fac537fe700 1 -- 172.21.15.46:0/3457006648 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fac40067050 msgr2=0x7fac40069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:04.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.227+0000 7fac537fe700 1 --2- 172.21.15.46:0/3457006648 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fac40067050 0x7fac40069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:04.289 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:04.287+0000 7fe195ee01c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:45:04.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.291+0000 7fac59fd1700 1 -- 172.21.15.46:0/3457006648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fac54157a10 con 0x7fac541598e0 2024-01-26T23:45:04.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.291+0000 7fac3b7fe700 1 -- 172.21.15.46:0/3457006648 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) v1 ==== 93+0+10 (secure 0 0 0) 0x7fac4c057630 con 0x7fac541598e0 2024-01-26T23:45:04.293 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:45:04.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.293+0000 7fac59fd1700 1 -- 172.21.15.46:0/3457006648 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fac40067050 msgr2=0x7fac40069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:04.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.293+0000 7fac59fd1700 1 --2- 172.21.15.46:0/3457006648 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fac40067050 0x7fac40069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.294+0000 7fac59fd1700 1 -- 172.21.15.46:0/3457006648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac541598e0 msgr2=0x7fac540c77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:04.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.294+0000 7fac59fd1700 1 --2- 172.21.15.46:0/3457006648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac541598e0 0x7fac540c77a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0099f0 tx=0x7fac4c019000 comp rx=0 tx=0).stop 2024-01-26T23:45:04.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.294+0000 7fac59fd1700 1 -- 172.21.15.46:0/3457006648 shutdown_connections 2024-01-26T23:45:04.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.294+0000 7fac59fd1700 1 --2- 172.21.15.46:0/3457006648 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fac40067050 0x7fac40069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.294+0000 7fac59fd1700 1 --2- 172.21.15.46:0/3457006648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac541598e0 0x7fac540c77a0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.294+0000 7fac59fd1700 1 --2- 172.21.15.46:0/3457006648 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac540cc380 0x7fac54154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.294+0000 7fac59fd1700 1 --2- 172.21.15.46:0/3457006648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac54159510 0x7fac54157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.294+0000 7fac59fd1700 1 -- 172.21.15.46:0/3457006648 >> 172.21.15.46:0/3457006648 conn(0x7fac5406bd90 msgr2=0x7fac541551a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:04.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.294+0000 7fac59fd1700 1 -- 172.21.15.46:0/3457006648 shutdown_connections 2024-01-26T23:45:04.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.294+0000 7fac59fd1700 1 -- 172.21.15.46:0/3457006648 wait complete. 2024-01-26T23:45:04.309 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-26T23:45:04.380 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:04.379+0000 7fa6be3621c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:45:04.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.443+0000 7fb7ab673700 1 Processor -- start 2024-01-26T23:45:04.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.443+0000 7fb7ab673700 1 -- start start 2024-01-26T23:45:04.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.444+0000 7fb7ab673700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7a406ebe0 0x7fb7a4067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:04.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.444+0000 7fb7ab673700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb7a41597e0 0x7fb7a4159c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:04.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.444+0000 7fb7ab673700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a406fdb0 con 0x7fb7a4067af0 2024-01-26T23:45:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.444+0000 7fb7ab673700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a406f1a0 con 0x7fb7a41597e0 2024-01-26T23:45:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.444+0000 7fb7ab673700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a406f480 con 0x7fb7a406ebe0 2024-01-26T23:45:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.444+0000 7fb7a8c0e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7a406ebe0 0x7fb7a4067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.444+0000 7fb7a9c10700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb7a41597e0 0x7fb7a4159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.444+0000 7fb7a940f700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fb7a4067af0 0x7fb7a40ad8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:59504/0 (socket says 172.21.15.46:59504) 2024-01-26T23:45:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.444+0000 7fb7a940f700 1 -- 172.21.15.46:0/4249147348 learned_addr learned my addr 172.21.15.46:0/4249147348 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.445+0000 7fb7a8c0e700 1 -- 172.21.15.46:0/4249147348 >> v2:172.21.15.106:3300/0 conn(0x7fb7a41597e0 msgr2=0x7fb7a4159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.445+0000 7fb7a8c0e700 1 --2- 172.21.15.46:0/4249147348 >> v2:172.21.15.106:3300/0 conn(0x7fb7a41597e0 0x7fb7a4159c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.445+0000 7fb7a8c0e700 1 -- 172.21.15.46:0/4249147348 >> v1:172.21.15.106:6789/0 conn(0x7fb7a4067af0 legacy=0x7fb7a40ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.445+0000 7fb7a8c0e700 1 -- 172.21.15.46:0/4249147348 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a406f760 con 0x7fb7a406ebe0 2024-01-26T23:45:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.445+0000 7fb79bfff700 1 -- 172.21.15.46:0/4249147348 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087592635 0 0) 0x7fb7a406fdb0 con 0x7fb7a4067af0 2024-01-26T23:45:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.445+0000 7fb7a9c10700 1 --2- 172.21.15.46:0/4249147348 >> v2:172.21.15.106:3300/0 conn(0x7fb7a41597e0 0x7fb7a4159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:04.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.445+0000 7fb7a8c0e700 1 --2- 172.21.15.46:0/4249147348 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7a406ebe0 0x7fb7a4067ec0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb7940098e0 tx=0x7fb7940091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caad09f2221f1656 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.445+0000 7fb79bfff700 1 -- 172.21.15.46:0/4249147348 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79400b640 con 0x7fb7a406ebe0 2024-01-26T23:45:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.445+0000 7fb79bfff700 1 -- 172.21.15.46:0/4249147348 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79400b7a0 con 0x7fb7a406ebe0 2024-01-26T23:45:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.445+0000 7fb79bfff700 1 -- 172.21.15.46:0/4249147348 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79400bb40 con 0x7fb7a406ebe0 2024-01-26T23:45:04.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.446+0000 7fb7ab673700 1 -- 172.21.15.46:0/4249147348 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7a406ebe0 msgr2=0x7fb7a4067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:04.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.446+0000 7fb7ab673700 1 --2- 172.21.15.46:0/4249147348 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7a406ebe0 0x7fb7a4067ec0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb7940098e0 tx=0x7fb7940091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.446+0000 7fb7ab673700 1 -- 172.21.15.46:0/4249147348 shutdown_connections 2024-01-26T23:45:04.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.446+0000 7fb7ab673700 1 --2- 172.21.15.46:0/4249147348 >> v2:172.21.15.106:3300/0 conn(0x7fb7a41597e0 0x7fb7a4159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.446+0000 7fb7ab673700 1 --2- 172.21.15.46:0/4249147348 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7a406ebe0 0x7fb7a4067ec0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.446+0000 7fb7ab673700 1 -- 172.21.15.46:0/4249147348 >> 172.21.15.46:0/4249147348 conn(0x7fb7a40cdf80 msgr2=0x7fb7a406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:04.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.446+0000 7fb7ab673700 1 -- 172.21.15.46:0/4249147348 shutdown_connections 2024-01-26T23:45:04.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.446+0000 7fb7ab673700 1 -- 172.21.15.46:0/4249147348 wait complete. 2024-01-26T23:45:04.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.448+0000 7fb7ab673700 1 Processor -- start 2024-01-26T23:45:04.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.448+0000 7fb7ab673700 1 -- start start 2024-01-26T23:45:04.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.449+0000 7fb7ab673700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a4067af0 0x7fb7a414d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:04.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.449+0000 7fb7ab673700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7a406ebe0 0x7fb7a414fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:04.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.449+0000 7fb7ab673700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7a41597e0 0x7fb7a41503d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:04.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.449+0000 7fb7ab673700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a4150910 con 0x7fb7a4067af0 2024-01-26T23:45:04.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.449+0000 7fb7ab673700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a40b1e40 con 0x7fb7a41597e0 2024-01-26T23:45:04.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.449+0000 7fb7ab673700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a40b2120 con 0x7fb7a406ebe0 2024-01-26T23:45:04.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.449+0000 7fb7a940f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a4067af0 0x7fb7a414d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:04.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.449+0000 7fb7a8c0e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7a406ebe0 0x7fb7a414fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:04.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.449+0000 7fb7a940f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a4067af0 0x7fb7a414d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35992/0 (socket says 172.21.15.46:35992) 2024-01-26T23:45:04.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.449+0000 7fb7a940f700 1 -- 172.21.15.46:0/3064307627 learned_addr learned my addr 172.21.15.46:0/3064307627 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.449+0000 7fb7a9c10700 1 --2- 172.21.15.46:0/3064307627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7a41597e0 0x7fb7a41503d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.450+0000 7fb7a9c10700 1 -- 172.21.15.46:0/3064307627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7a406ebe0 msgr2=0x7fb7a414fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.450+0000 7fb7a9c10700 1 --2- 172.21.15.46:0/3064307627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7a406ebe0 0x7fb7a414fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.450+0000 7fb7a9c10700 1 -- 172.21.15.46:0/3064307627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a4067af0 msgr2=0x7fb7a414d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.450+0000 7fb7a9c10700 1 --2- 172.21.15.46:0/3064307627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a4067af0 0x7fb7a414d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.450+0000 7fb7a9c10700 1 -- 172.21.15.46:0/3064307627 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79c009a20 con 0x7fb7a41597e0 2024-01-26T23:45:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.450+0000 7fb7a9c10700 1 --2- 172.21.15.46:0/3064307627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7a41597e0 0x7fb7a41503d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009f10 tx=0x7fb79c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.450+0000 7fb7997fa700 1 -- 172.21.15.46:0/3064307627 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c00cb70 con 0x7fb7a41597e0 2024-01-26T23:45:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.450+0000 7fb7997fa700 1 -- 172.21.15.46:0/3064307627 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79c00f040 con 0x7fb7a41597e0 2024-01-26T23:45:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.450+0000 7fb7ab673700 1 -- 172.21.15.46:0/3064307627 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb794009420 con 0x7fb7a41597e0 2024-01-26T23:45:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.450+0000 7fb7997fa700 1 -- 172.21.15.46:0/3064307627 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c014480 con 0x7fb7a41597e0 2024-01-26T23:45:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.450+0000 7fb7ab673700 1 -- 172.21.15.46:0/3064307627 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a40b26e0 con 0x7fb7a41597e0 2024-01-26T23:45:04.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.451+0000 7fb7ab673700 1 -- 172.21.15.46:0/3064307627 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a404c710 con 0x7fb7a41597e0 2024-01-26T23:45:04.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.452+0000 7fb7997fa700 1 -- 172.21.15.46:0/3064307627 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fb79c003020 con 0x7fb7a41597e0 2024-01-26T23:45:04.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.452+0000 7fb7997fa700 1 --2- 172.21.15.46:0/3064307627 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb788067050 0x7fb788069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:04.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.452+0000 7fb7a940f700 1 -- 172.21.15.46:0/3064307627 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb788067050 msgr2=0x7fb788069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:04.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.452+0000 7fb7997fa700 1 -- 172.21.15.46:0/3064307627 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6705+0+0 (secure 0 0 0) 0x7fb79c085430 con 0x7fb7a41597e0 2024-01-26T23:45:04.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.452+0000 7fb7a940f700 1 --2- 172.21.15.46:0/3064307627 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb788067050 0x7fb788069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:04.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.457+0000 7fb7997fa700 1 -- 172.21.15.46:0/3064307627 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb79c052090 con 0x7fb7a41597e0 2024-01-26T23:45:04.591 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:04.590+0000 7fa6be3621c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:45:04.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.653+0000 7fb7a940f700 1 -- 172.21.15.46:0/3064307627 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb788067050 msgr2=0x7fb788069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:04.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.653+0000 7fb7a940f700 1 --2- 172.21.15.46:0/3064307627 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb788067050 0x7fb788069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:04.657 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:04.656+0000 7fe195ee01c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:45:04.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.662+0000 7fb7ab673700 1 -- 172.21.15.46:0/3064307627 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb7a4068690 con 0x7fb7a41597e0 2024-01-26T23:45:04.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.662+0000 7fb7997fa700 1 -- 172.21.15.46:0/3064307627 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v73) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb79c0556b0 con 0x7fb7a41597e0 2024-01-26T23:45:04.664 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:45:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.665+0000 7fb7ab673700 1 -- 172.21.15.46:0/3064307627 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb788067050 msgr2=0x7fb788069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.665+0000 7fb7ab673700 1 --2- 172.21.15.46:0/3064307627 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb788067050 0x7fb788069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.665+0000 7fb7ab673700 1 -- 172.21.15.46:0/3064307627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7a41597e0 msgr2=0x7fb7a41503d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.665+0000 7fb7ab673700 1 --2- 172.21.15.46:0/3064307627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7a41597e0 0x7fb7a41503d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009f10 tx=0x7fb79c00b040 comp rx=0 tx=0).stop 2024-01-26T23:45:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.665+0000 7fb7ab673700 1 -- 172.21.15.46:0/3064307627 shutdown_connections 2024-01-26T23:45:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.665+0000 7fb7ab673700 1 --2- 172.21.15.46:0/3064307627 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb788067050 0x7fb788069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.665+0000 7fb7ab673700 1 --2- 172.21.15.46:0/3064307627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a4067af0 0x7fb7a414d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.665+0000 7fb7ab673700 1 --2- 172.21.15.46:0/3064307627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7a41597e0 0x7fb7a41503d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.665+0000 7fb7ab673700 1 --2- 172.21.15.46:0/3064307627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7a406ebe0 0x7fb7a414fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.665+0000 7fb7ab673700 1 -- 172.21.15.46:0/3064307627 >> 172.21.15.46:0/3064307627 conn(0x7fb7a40cdf80 msgr2=0x7fb7a4155150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.665+0000 7fb7ab673700 1 -- 172.21.15.46:0/3064307627 shutdown_connections 2024-01-26T23:45:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.665+0000 7fb7ab673700 1 -- 172.21.15.46:0/3064307627 wait complete. 2024-01-26T23:45:04.680 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-26T23:45:04.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.815+0000 7fead7ea1700 1 Processor -- start 2024-01-26T23:45:04.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.816+0000 7fead7ea1700 1 -- start start 2024-01-26T23:45:04.817 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:04.816+0000 7fa6be3621c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:45:04.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.816+0000 7fead7ea1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fead00c8da0 0x7fead00c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.816+0000 7fead7ea1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fead0140af0 0x7fead0140f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.817+0000 7fead7ea1700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead00ccc00 con 0x7fead00caf30 2024-01-26T23:45:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.817+0000 7fead7ea1700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fead00cbbf0 con 0x7fead0140af0 2024-01-26T23:45:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.817+0000 7fead7ea1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead00cc310 con 0x7fead00c8da0 2024-01-26T23:45:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.817+0000 7fead543c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fead0140af0 0x7fead0140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.817+0000 7fead543c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fead0140af0 0x7fead0140f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36002/0 (socket says 172.21.15.46:36002) 2024-01-26T23:45:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.817+0000 7fead543c700 1 -- 172.21.15.46:0/3335903208 learned_addr learned my addr 172.21.15.46:0/3335903208 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.817+0000 7fead5c3d700 1 --2- 172.21.15.46:0/3335903208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fead00c8da0 0x7fead00c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.817+0000 7fead543c700 1 -- 172.21.15.46:0/3335903208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fead00c8da0 msgr2=0x7fead00c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.817+0000 7fead543c700 1 --2- 172.21.15.46:0/3335903208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fead00c8da0 0x7fead00c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.817+0000 7fead543c700 1 -- 172.21.15.46:0/3335903208 >> v1:172.21.15.106:6789/0 conn(0x7fead00caf30 legacy=0x7fead00cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.817+0000 7fead4c3b700 1 -- 172.21.15.46:0/3335903208 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890460054 0 0) 0x7fead00ccc00 con 0x7fead00caf30 2024-01-26T23:45:04.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.817+0000 7fead543c700 1 -- 172.21.15.46:0/3335903208 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead00cc5b0 con 0x7fead0140af0 2024-01-26T23:45:04.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.817+0000 7fead543c700 1 --2- 172.21.15.46:0/3335903208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead0140af0 0x7fead0140f60 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7feac00099b0 tx=0x7feac0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b1265badd89d05 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:04.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.817+0000 7fead4c3b700 1 -- 172.21.15.46:0/3335903208 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac0009ea0 con 0x7fead0140af0 2024-01-26T23:45:04.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.818+0000 7fead4c3b700 1 -- 172.21.15.46:0/3335903208 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac000b860 con 0x7fead0140af0 2024-01-26T23:45:04.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.818+0000 7fead4c3b700 1 -- 172.21.15.46:0/3335903208 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac000bc00 con 0x7fead0140af0 2024-01-26T23:45:04.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.818+0000 7fead7ea1700 1 -- 172.21.15.46:0/3335903208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead0140af0 msgr2=0x7fead0140f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:04.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.818+0000 7fead7ea1700 1 --2- 172.21.15.46:0/3335903208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead0140af0 0x7fead0140f60 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7feac00099b0 tx=0x7feac0009250 comp rx=0 tx=0).stop 2024-01-26T23:45:04.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.818+0000 7fead7ea1700 1 -- 172.21.15.46:0/3335903208 shutdown_connections 2024-01-26T23:45:04.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.818+0000 7fead7ea1700 1 --2- 172.21.15.46:0/3335903208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fead00c8da0 0x7fead00c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.818+0000 7fead7ea1700 1 --2- 172.21.15.46:0/3335903208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead0140af0 0x7fead0140f60 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.818+0000 7fead7ea1700 1 -- 172.21.15.46:0/3335903208 >> 172.21.15.46:0/3335903208 conn(0x7fead006bd90 msgr2=0x7fead006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:04.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.818+0000 7fead7ea1700 1 -- 172.21.15.46:0/3335903208 shutdown_connections 2024-01-26T23:45:04.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.818+0000 7fead7ea1700 1 -- 172.21.15.46:0/3335903208 wait complete. 2024-01-26T23:45:04.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.819+0000 7fead7ea1700 1 Processor -- start 2024-01-26T23:45:04.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.819+0000 7fead7ea1700 1 -- start start 2024-01-26T23:45:04.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.820+0000 7fead7ea1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fead00c8da0 0x7fead013c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:04.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.820+0000 7fead7ea1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fead00caf30 0x7fead013c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:04.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.820+0000 7fead7ea1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead0140af0 0x7fead00c5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:04.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.820+0000 7fead7ea1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead00c79c0 con 0x7fead0140af0 2024-01-26T23:45:04.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.820+0000 7fead7ea1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead00c7ca0 con 0x7fead00caf30 2024-01-26T23:45:04.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.820+0000 7fead7ea1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead00c7f80 con 0x7fead00c8da0 2024-01-26T23:45:04.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.820+0000 7fead643e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead0140af0 0x7fead00c5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:04.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.820+0000 7fead5c3d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fead00c8da0 0x7fead013c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:04.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.820+0000 7fead643e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead0140af0 0x7fead00c5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36014/0 (socket says 172.21.15.46:36014) 2024-01-26T23:45:04.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.820+0000 7fead643e700 1 -- 172.21.15.46:0/4210903231 learned_addr learned my addr 172.21.15.46:0/4210903231 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:04.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.820+0000 7fead543c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fead00caf30 0x7fead013c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:04.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.821+0000 7fead5c3d700 1 -- 172.21.15.46:0/4210903231 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fead00caf30 msgr2=0x7fead013c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.821+0000 7fead5c3d700 1 --2- 172.21.15.46:0/4210903231 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fead00caf30 0x7fead013c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.821+0000 7fead5c3d700 1 -- 172.21.15.46:0/4210903231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead0140af0 msgr2=0x7fead00c5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.821+0000 7fead5c3d700 1 --2- 172.21.15.46:0/4210903231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead0140af0 0x7fead00c5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.821+0000 7fead5c3d700 1 -- 172.21.15.46:0/4210903231 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac00094f0 con 0x7fead00c8da0 2024-01-26T23:45:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.821+0000 7fead5c3d700 1 --2- 172.21.15.46:0/4210903231 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fead00c8da0 0x7fead013c0a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7feacc00ea50 tx=0x7feacc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.821+0000 7feac5ffb700 1 -- 172.21.15.46:0/4210903231 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc00ee90 con 0x7fead00c8da0 2024-01-26T23:45:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.821+0000 7feac5ffb700 1 -- 172.21.15.46:0/4210903231 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feacc00c7f0 con 0x7fead00c8da0 2024-01-26T23:45:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.821+0000 7fead7ea1700 1 -- 172.21.15.46:0/4210903231 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead00c8260 con 0x7fead00c8da0 2024-01-26T23:45:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.821+0000 7feac5ffb700 1 -- 172.21.15.46:0/4210903231 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc00cb90 con 0x7fead00c8da0 2024-01-26T23:45:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.821+0000 7fead7ea1700 1 -- 172.21.15.46:0/4210903231 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead00b1360 con 0x7fead00c8da0 2024-01-26T23:45:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.823+0000 7fead7ea1700 1 -- 172.21.15.46:0/4210903231 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fead004c710 con 0x7fead00c8da0 2024-01-26T23:45:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.823+0000 7feac5ffb700 1 -- 172.21.15.46:0/4210903231 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7feacc013470 con 0x7fead00c8da0 2024-01-26T23:45:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.823+0000 7feac5ffb700 1 --2- 172.21.15.46:0/4210903231 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7feab4067050 0x7feab4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.823+0000 7fead543c700 1 -- 172.21.15.46:0/4210903231 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7feab4067050 msgr2=0x7feab4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.823+0000 7feac5ffb700 1 -- 172.21.15.46:0/4210903231 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6705+0+0 (secure 0 0 0) 0x7feacc084cd0 con 0x7fead00c8da0 2024-01-26T23:45:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.823+0000 7fead543c700 1 --2- 172.21.15.46:0/4210903231 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7feab4067050 0x7feab4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:04.828+0000 7feac5ffb700 1 -- 172.21.15.46:0/4210903231 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feacc0519a0 con 0x7fead00c8da0 2024-01-26T23:45:04.889 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:04.888+0000 7fa6be3621c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:45:04.904 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:04.903+0000 7fe195ee01c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:45:05.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.024+0000 7fead543c700 1 -- 172.21.15.46:0/4210903231 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7feab4067050 msgr2=0x7feab4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:05.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.024+0000 7fead543c700 1 --2- 172.21.15.46:0/4210903231 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7feab4067050 0x7feab4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:05.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.034+0000 7fead7ea1700 1 -- 172.21.15.46:0/4210903231 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fead01550b0 con 0x7fead00c8da0 2024-01-26T23:45:05.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.034+0000 7feac5ffb700 1 -- 172.21.15.46:0/4210903231 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v73) v1 ==== 100+0+11 (secure 0 0 0) 0x7feacc054fc0 con 0x7fead00c8da0 2024-01-26T23:45:05.036 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-26T23:45:05.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.037+0000 7fead7ea1700 1 -- 172.21.15.46:0/4210903231 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7feab4067050 msgr2=0x7feab4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.037+0000 7fead7ea1700 1 --2- 172.21.15.46:0/4210903231 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7feab4067050 0x7feab4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.037+0000 7fead7ea1700 1 -- 172.21.15.46:0/4210903231 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fead00c8da0 msgr2=0x7fead013c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.037+0000 7fead7ea1700 1 --2- 172.21.15.46:0/4210903231 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fead00c8da0 0x7fead013c0a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7feacc00ea50 tx=0x7feacc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.037+0000 7fead7ea1700 1 -- 172.21.15.46:0/4210903231 shutdown_connections 2024-01-26T23:45:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.037+0000 7fead7ea1700 1 --2- 172.21.15.46:0/4210903231 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7feab4067050 0x7feab4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.037+0000 7fead7ea1700 1 --2- 172.21.15.46:0/4210903231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead0140af0 0x7fead00c5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.037+0000 7fead7ea1700 1 --2- 172.21.15.46:0/4210903231 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fead00c8da0 0x7fead013c0a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.037+0000 7fead7ea1700 1 --2- 172.21.15.46:0/4210903231 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fead00caf30 0x7fead013c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.037+0000 7fead7ea1700 1 -- 172.21.15.46:0/4210903231 >> 172.21.15.46:0/4210903231 conn(0x7fead006bd90 msgr2=0x7fead013fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.037+0000 7fead7ea1700 1 -- 172.21.15.46:0/4210903231 shutdown_connections 2024-01-26T23:45:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.037+0000 7fead7ea1700 1 -- 172.21.15.46:0/4210903231 wait complete. 2024-01-26T23:45:05.053 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:45:05.169 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:05.168+0000 7fe195ee01c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:45:05.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.186+0000 7f2d8e92c700 1 Processor -- start 2024-01-26T23:45:05.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.186+0000 7f2d8e92c700 1 -- start start 2024-01-26T23:45:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.187+0000 7f2d8e92c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d8815ce40 0x7f2d880d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.187+0000 7f2d8e92c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d880cff70 con 0x7f2d880cc1c0 2024-01-26T23:45:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.187+0000 7f2d8e92c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d880cf400 con 0x7f2d8815d210 2024-01-26T23:45:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.187+0000 7f2d8e92c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d880cf660 con 0x7f2d8815ce40 2024-01-26T23:45:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.187+0000 7f2d87fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d8815ce40 0x7f2d880d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.187+0000 7f2d87fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d8815ce40 0x7f2d880d0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36020/0 (socket says 172.21.15.46:36020) 2024-01-26T23:45:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.187+0000 7f2d87fff700 1 -- 172.21.15.46:0/3191450669 learned_addr learned my addr 172.21.15.46:0/3191450669 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.187+0000 7f2d8cec9700 1 --1- 172.21.15.46:0/3191450669 >> v1:172.21.15.46:6789/0 conn(0x7f2d880cc1c0 0x7f2d880ade30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35078/0 (socket says 172.21.15.46:35078) 2024-01-26T23:45:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.188+0000 7f2d86ffd700 1 -- 172.21.15.46:0/3191450669 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448954413 0 0) 0x7f2d880cf400 con 0x7f2d8815d210 2024-01-26T23:45:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.188+0000 7f2d87fff700 1 -- 172.21.15.46:0/3191450669 >> v1:172.21.15.106:6789/0 conn(0x7f2d8815d210 legacy=0x7f2d880cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.188+0000 7f2d87fff700 1 -- 172.21.15.46:0/3191450669 >> v1:172.21.15.46:6789/0 conn(0x7f2d880cc1c0 legacy=0x7f2d880ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.188+0000 7f2d87fff700 1 -- 172.21.15.46:0/3191450669 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d880d0500 con 0x7f2d8815ce40 2024-01-26T23:45:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.188+0000 7f2d87fff700 1 --2- 172.21.15.46:0/3191450669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8815ce40 0x7f2d880d0e50 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009990 tx=0x7f2d78009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34af6016f8de44fd server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.188+0000 7f2d86ffd700 1 -- 172.21.15.46:0/3191450669 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7800b760 con 0x7f2d8815ce40 2024-01-26T23:45:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.188+0000 7f2d86ffd700 1 -- 172.21.15.46:0/3191450669 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7800b8c0 con 0x7f2d8815ce40 2024-01-26T23:45:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.188+0000 7f2d86ffd700 1 -- 172.21.15.46:0/3191450669 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7800bc80 con 0x7f2d8815ce40 2024-01-26T23:45:05.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.189+0000 7f2d8e92c700 1 -- 172.21.15.46:0/3191450669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8815ce40 msgr2=0x7f2d880d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:05.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.189+0000 7f2d8e92c700 1 --2- 172.21.15.46:0/3191450669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8815ce40 0x7f2d880d0e50 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009990 tx=0x7f2d78009220 comp rx=0 tx=0).stop 2024-01-26T23:45:05.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.189+0000 7f2d8e92c700 1 -- 172.21.15.46:0/3191450669 shutdown_connections 2024-01-26T23:45:05.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.189+0000 7f2d8e92c700 1 --2- 172.21.15.46:0/3191450669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8815ce40 0x7f2d880d0e50 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.189+0000 7f2d8e92c700 1 -- 172.21.15.46:0/3191450669 >> 172.21.15.46:0/3191450669 conn(0x7f2d8806f840 msgr2=0x7f2d880d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:05.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.189+0000 7f2d8e92c700 1 -- 172.21.15.46:0/3191450669 shutdown_connections 2024-01-26T23:45:05.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.189+0000 7f2d8e92c700 1 -- 172.21.15.46:0/3191450669 wait complete. 2024-01-26T23:45:05.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.191+0000 7f2d8e92c700 1 Processor -- start 2024-01-26T23:45:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.191+0000 7f2d8e92c700 1 -- start start 2024-01-26T23:45:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.192+0000 7f2d8e92c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2d880cc1c0 0x7f2d88154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.192+0000 7f2d8e92c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8815ce40 0x7f2d880c57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.192+0000 7f2d8e92c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2d8815d210 0x7f2d880c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.192+0000 7f2d8e92c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d880c85b0 con 0x7f2d8815ce40 2024-01-26T23:45:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.192+0000 7f2d8e92c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d88158760 con 0x7f2d8815d210 2024-01-26T23:45:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.192+0000 7f2d8e92c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d880c8120 con 0x7f2d880cc1c0 2024-01-26T23:45:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.192+0000 7f2d877fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8815ce40 0x7f2d880c57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.192+0000 7f2d877fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8815ce40 0x7f2d880c57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36030/0 (socket says 172.21.15.46:36030) 2024-01-26T23:45:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.192+0000 7f2d877fe700 1 -- 172.21.15.46:0/1140267923 learned_addr learned my addr 172.21.15.46:0/1140267923 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.192+0000 7f2d8cec9700 1 --2- 172.21.15.46:0/1140267923 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2d8815d210 0x7f2d880c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.192+0000 7f2d87fff700 1 --2- 172.21.15.46:0/1140267923 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2d880cc1c0 0x7f2d88154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.192+0000 7f2d877fe700 1 -- 172.21.15.46:0/1140267923 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2d880cc1c0 msgr2=0x7f2d88154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.192+0000 7f2d877fe700 1 --2- 172.21.15.46:0/1140267923 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2d880cc1c0 0x7f2d88154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.192+0000 7f2d877fe700 1 -- 172.21.15.46:0/1140267923 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2d8815d210 msgr2=0x7f2d880c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.192+0000 7f2d877fe700 1 --2- 172.21.15.46:0/1140267923 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2d8815d210 0x7f2d880c5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.192+0000 7f2d877fe700 1 -- 172.21.15.46:0/1140267923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d78009480 con 0x7f2d8815ce40 2024-01-26T23:45:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.193+0000 7f2d877fe700 1 --2- 172.21.15.46:0/1140267923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8815ce40 0x7f2d880c57f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00c590 tx=0x7f2d7c007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.193+0000 7f2d6ffff700 1 -- 172.21.15.46:0/1140267923 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c011590 con 0x7f2d8815ce40 2024-01-26T23:45:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.193+0000 7f2d6ffff700 1 -- 172.21.15.46:0/1140267923 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7c011710 con 0x7f2d8815ce40 2024-01-26T23:45:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.193+0000 7f2d8e92c700 1 -- 172.21.15.46:0/1140267923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d880c63d0 con 0x7f2d8815ce40 2024-01-26T23:45:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.193+0000 7f2d6ffff700 1 -- 172.21.15.46:0/1140267923 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c011ad0 con 0x7f2d8815ce40 2024-01-26T23:45:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.193+0000 7f2d8e92c700 1 -- 172.21.15.46:0/1140267923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d880c7ad0 con 0x7f2d8815ce40 2024-01-26T23:45:05.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.194+0000 7f2d8e92c700 1 -- 172.21.15.46:0/1140267923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d880c8290 con 0x7f2d8815ce40 2024-01-26T23:45:05.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.195+0000 7f2d6ffff700 1 -- 172.21.15.46:0/1140267923 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f2d7c01b3b0 con 0x7f2d8815ce40 2024-01-26T23:45:05.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.195+0000 7f2d6ffff700 1 --2- 172.21.15.46:0/1140267923 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d74067050 0x7f2d74069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:05.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.195+0000 7f2d6ffff700 1 -- 172.21.15.46:0/1140267923 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6705+0+0 (secure 0 0 0) 0x7f2d7c083f00 con 0x7f2d8815ce40 2024-01-26T23:45:05.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.195+0000 7f2d87fff700 1 -- 172.21.15.46:0/1140267923 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d74067050 msgr2=0x7f2d74069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:05.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.195+0000 7f2d87fff700 1 --2- 172.21.15.46:0/1140267923 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d74067050 0x7f2d74069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:05.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.199+0000 7f2d6ffff700 1 -- 172.21.15.46:0/1140267923 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2d7c050c70 con 0x7f2d8815ce40 2024-01-26T23:45:05.253 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:05.252+0000 7fe195ee01c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:45:05.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.396+0000 7f2d87fff700 1 -- 172.21.15.46:0/1140267923 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d74067050 msgr2=0x7f2d74069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:05.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.396+0000 7f2d87fff700 1 --2- 172.21.15.46:0/1140267923 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d74067050 0x7f2d74069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:05.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.458+0000 7f2d8e92c700 1 -- 172.21.15.46:0/1140267923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d8804c710 con 0x7f2d8815ce40 2024-01-26T23:45:05.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.458+0000 7f2d6ffff700 1 -- 172.21.15.46:0/1140267923 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4603 (secure 0 0 0) 0x7f2d7c017070 con 0x7f2d8815ce40 2024-01-26T23:45:05.460 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:05.460 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3037972557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3037972557},{"type":"v1","addr":"172.21.15.106:6837","nonce":3037972557}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":5701,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3786017823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3786017823},{"type":"v1","addr":"172.21.15.106:6836","nonce":3786017823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":5704,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3329385210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3329385210},{"type":"v1","addr":"172.21.15.46:6835","nonce":3329385210}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:43:01.144129+0000","modified":"2024-01-26T23:43:04.718067+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6833/2801738403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2801738403},{"type":"v1","addr":"172.21.15.46:6833","nonce":2801738403}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2024-01-26T23:45:05.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.461+0000 7f2d8e92c700 1 -- 172.21.15.46:0/1140267923 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d74067050 msgr2=0x7f2d74069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:05.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.461+0000 7f2d8e92c700 1 --2- 172.21.15.46:0/1140267923 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d74067050 0x7f2d74069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.461+0000 7f2d8e92c700 1 -- 172.21.15.46:0/1140267923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8815ce40 msgr2=0x7f2d880c57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.461+0000 7f2d8e92c700 1 --2- 172.21.15.46:0/1140267923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8815ce40 0x7f2d880c57f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00c590 tx=0x7f2d7c007f50 comp rx=0 tx=0).stop 2024-01-26T23:45:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.461+0000 7f2d8e92c700 1 -- 172.21.15.46:0/1140267923 shutdown_connections 2024-01-26T23:45:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.461+0000 7f2d8e92c700 1 --2- 172.21.15.46:0/1140267923 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2d74067050 0x7f2d74069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.461+0000 7f2d8e92c700 1 --2- 172.21.15.46:0/1140267923 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2d8815d210 0x7f2d880c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.461+0000 7f2d8e92c700 1 --2- 172.21.15.46:0/1140267923 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2d880cc1c0 0x7f2d88154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.461+0000 7f2d8e92c700 1 --2- 172.21.15.46:0/1140267923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8815ce40 0x7f2d880c57f0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.461+0000 7f2d8e92c700 1 -- 172.21.15.46:0/1140267923 >> 172.21.15.46:0/1140267923 conn(0x7f2d8806f840 msgr2=0x7f2d88154ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.461+0000 7f2d8e92c700 1 -- 172.21.15.46:0/1140267923 shutdown_connections 2024-01-26T23:45:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.461+0000 7f2d8e92c700 1 -- 172.21.15.46:0/1140267923 wait complete. 2024-01-26T23:45:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 27 2024-01-26T23:45:05.476 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:45:05.489 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:05.488+0000 7fa6be3621c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:45:05.563 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:05.562+0000 7fa6be3621c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:45:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.608+0000 7f8e522ad700 1 Processor -- start 2024-01-26T23:45:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.609+0000 7f8e522ad700 1 -- start start 2024-01-26T23:45:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.609+0000 7f8e522ad700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8e4c15ce40 0x7f8e4c0d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.609+0000 7f8e522ad700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e4c15d210 0x7f8e4c0cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.609+0000 7f8e522ad700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c0cff70 con 0x7f8e4c0d1390 2024-01-26T23:45:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.609+0000 7f8e522ad700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e4c0cf400 con 0x7f8e4c15ce40 2024-01-26T23:45:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.609+0000 7f8e522ad700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c0cf660 con 0x7f8e4c15d210 2024-01-26T23:45:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.610+0000 7f8e4bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8e4c15ce40 0x7f8e4c0d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.610+0000 7f8e4bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8e4c15ce40 0x7f8e4c0d0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36042/0 (socket says 172.21.15.46:36042) 2024-01-26T23:45:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.610+0000 7f8e4bfff700 1 -- 172.21.15.46:0/267577414 learned_addr learned my addr 172.21.15.46:0/267577414 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.610+0000 7f8e4b7fe700 1 --2- 172.21.15.46:0/267577414 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e4c15d210 0x7f8e4c0cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.610+0000 7f8e4bfff700 1 -- 172.21.15.46:0/267577414 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e4c15d210 msgr2=0x7f8e4c0cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.610+0000 7f8e4bfff700 1 --2- 172.21.15.46:0/267577414 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e4c15d210 0x7f8e4c0cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.610+0000 7f8e4bfff700 1 -- 172.21.15.46:0/267577414 >> v1:172.21.15.106:6789/0 conn(0x7f8e4c0d1390 legacy=0x7f8e4c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.610+0000 7f8e4bfff700 1 -- 172.21.15.46:0/267577414 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c0d0500 con 0x7f8e4c15ce40 2024-01-26T23:45:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.610+0000 7f8e4affd700 1 -- 172.21.15.46:0/267577414 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290430492 0 0) 0x7f8e4c0cff70 con 0x7f8e4c0d1390 2024-01-26T23:45:05.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.610+0000 7f8e4bfff700 1 --2- 172.21.15.46:0/267577414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c15ce40 0x7f8e4c0d0e50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c009990 tx=0x7f8e3c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc60480058b073c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:05.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.610+0000 7f8e4affd700 1 -- 172.21.15.46:0/267577414 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c009e80 con 0x7f8e4c15ce40 2024-01-26T23:45:05.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.610+0000 7f8e4affd700 1 -- 172.21.15.46:0/267577414 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3c005c10 con 0x7f8e4c15ce40 2024-01-26T23:45:05.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.610+0000 7f8e4affd700 1 -- 172.21.15.46:0/267577414 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c00b650 con 0x7f8e4c15ce40 2024-01-26T23:45:05.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.611+0000 7f8e522ad700 1 -- 172.21.15.46:0/267577414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c15ce40 msgr2=0x7f8e4c0d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:05.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.611+0000 7f8e522ad700 1 --2- 172.21.15.46:0/267577414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c15ce40 0x7f8e4c0d0e50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c009990 tx=0x7f8e3c009220 comp rx=0 tx=0).stop 2024-01-26T23:45:05.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.611+0000 7f8e522ad700 1 -- 172.21.15.46:0/267577414 shutdown_connections 2024-01-26T23:45:05.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.611+0000 7f8e522ad700 1 --2- 172.21.15.46:0/267577414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c15ce40 0x7f8e4c0d0e50 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.611+0000 7f8e522ad700 1 --2- 172.21.15.46:0/267577414 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e4c15d210 0x7f8e4c0cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.611+0000 7f8e522ad700 1 -- 172.21.15.46:0/267577414 >> 172.21.15.46:0/267577414 conn(0x7f8e4c06f840 msgr2=0x7f8e4c0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:05.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.611+0000 7f8e522ad700 1 -- 172.21.15.46:0/267577414 shutdown_connections 2024-01-26T23:45:05.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.611+0000 7f8e522ad700 1 -- 172.21.15.46:0/267577414 wait complete. 2024-01-26T23:45:05.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.612+0000 7f8e522ad700 1 Processor -- start 2024-01-26T23:45:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.613+0000 7f8e522ad700 1 -- start start 2024-01-26T23:45:05.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.613+0000 7f8e522ad700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8e4c0d1390 0x7f8e4c15c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:05.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.613+0000 7f8e522ad700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e4c15ce40 0x7f8e4c15f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.613+0000 7f8e522ad700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c15d210 0x7f8e4c09bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.613+0000 7f8e522ad700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c15e620 con 0x7f8e4c15d210 2024-01-26T23:45:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.613+0000 7f8e522ad700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c15e900 con 0x7f8e4c0d1390 2024-01-26T23:45:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.613+0000 7f8e522ad700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c15ebe0 con 0x7f8e4c15ce40 2024-01-26T23:45:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.614+0000 7f8e4bfff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8e4c0d1390 0x7f8e4c15c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.614+0000 7f8e4b7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e4c15ce40 0x7f8e4c15f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.614+0000 7f8e5084a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c15d210 0x7f8e4c09bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.614+0000 7f8e4bfff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8e4c0d1390 0x7f8e4c15c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:59232/0 (socket says 172.21.15.46:59232) 2024-01-26T23:45:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.614+0000 7f8e4bfff700 1 -- 172.21.15.46:0/3985229630 learned_addr learned my addr 172.21.15.46:0/3985229630 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.614+0000 7f8e4b7fe700 1 --2- 172.21.15.46:0/3985229630 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e4c15ce40 0x7f8e4c15f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:39294/0 (socket says 172.21.15.46:39294) 2024-01-26T23:45:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.614+0000 7f8e5084a700 1 -- 172.21.15.46:0/3985229630 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e4c15ce40 msgr2=0x7f8e4c15f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.614+0000 7f8e5084a700 1 --2- 172.21.15.46:0/3985229630 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e4c15ce40 0x7f8e4c15f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.614+0000 7f8e5084a700 1 -- 172.21.15.46:0/3985229630 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8e4c0d1390 msgr2=0x7f8e4c15c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:05.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.614+0000 7f8e5084a700 1 --2- 172.21.15.46:0/3985229630 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8e4c0d1390 0x7f8e4c15c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.614+0000 7f8e5084a700 1 -- 172.21.15.46:0/3985229630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3c009480 con 0x7f8e4c15d210 2024-01-26T23:45:05.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.614+0000 7f8e5084a700 1 --2- 172.21.15.46:0/3985229630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c15d210 0x7f8e4c09bde0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f8e44001560 tx=0x7f8e440016c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:05.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.614+0000 7f8e33fff700 1 -- 172.21.15.46:0/3985229630 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e44012630 con 0x7f8e4c15d210 2024-01-26T23:45:05.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.614+0000 7f8e33fff700 1 -- 172.21.15.46:0/3985229630 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e44012790 con 0x7f8e4c15d210 2024-01-26T23:45:05.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.614+0000 7f8e522ad700 1 -- 172.21.15.46:0/3985229630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e4c15eec0 con 0x7f8e4c15d210 2024-01-26T23:45:05.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.614+0000 7f8e33fff700 1 -- 172.21.15.46:0/3985229630 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e44012b50 con 0x7f8e4c15d210 2024-01-26T23:45:05.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.614+0000 7f8e522ad700 1 -- 172.21.15.46:0/3985229630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e4c15f480 con 0x7f8e4c15d210 2024-01-26T23:45:05.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.616+0000 7f8e522ad700 1 -- 172.21.15.46:0/3985229630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e4c04c710 con 0x7f8e4c15d210 2024-01-26T23:45:05.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.616+0000 7f8e33fff700 1 -- 172.21.15.46:0/3985229630 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f8e44003030 con 0x7f8e4c15d210 2024-01-26T23:45:05.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.616+0000 7f8e33fff700 1 --2- 172.21.15.46:0/3985229630 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8e38067050 0x7f8e38069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:05.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.616+0000 7f8e4bfff700 1 -- 172.21.15.46:0/3985229630 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8e38067050 msgr2=0x7f8e38069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:05.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.616+0000 7f8e33fff700 1 -- 172.21.15.46:0/3985229630 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6705+0+0 (secure 0 0 0) 0x7f8e440849e0 con 0x7f8e4c15d210 2024-01-26T23:45:05.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.616+0000 7f8e4bfff700 1 --2- 172.21.15.46:0/3985229630 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8e38067050 0x7f8e38069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:05.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.621+0000 7f8e33fff700 1 -- 172.21.15.46:0/3985229630 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8e44051750 con 0x7f8e4c15d210 2024-01-26T23:45:05.686 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:05.685+0000 7fa6be3621c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:45:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.818+0000 7f8e4bfff700 1 -- 172.21.15.46:0/3985229630 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8e38067050 msgr2=0x7f8e38069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.818+0000 7f8e4bfff700 1 --2- 172.21.15.46:0/3985229630 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8e38067050 0x7f8e38069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:05.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.831+0000 7f8e522ad700 1 -- 172.21.15.46:0/3985229630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e4c0cf920 con 0x7f8e4c15d210 2024-01-26T23:45:05.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.831+0000 7f8e33fff700 1 -- 172.21.15.46:0/3985229630 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+16571 (secure 0 0 0) 0x7f8e44054d70 con 0x7f8e4c15d210 2024-01-26T23:45:05.833 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:05.833 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":73,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:44:21.651311+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:42:59.650806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-01-26T23:43:00.105558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2","mappings":[{"from":0,"to":6}]},{"pgid":"8.1d","mappings":[{"from":0,"to":6}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/1676475004":"2024-01-27T23:43:20.540793+0000","172.21.15.106:6833/43388":"2024-01-27T23:43:20.540793+0000","172.21.15.106:6832/43388":"2024-01-27T23:43:20.540793+0000","172.21.15.106:0/2801680181":"2024-01-27T23:43:20.540793+0000","172.21.15.106:0/677673260":"2024-01-27T23:43:20.540793+0000","172.21.15.106:0/904952331":"2024-01-27T23:43:20.540793+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:45:05.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.834+0000 7f8e522ad700 1 -- 172.21.15.46:0/3985229630 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8e38067050 msgr2=0x7f8e38069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:05.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.834+0000 7f8e522ad700 1 --2- 172.21.15.46:0/3985229630 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8e38067050 0x7f8e38069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.834+0000 7f8e522ad700 1 -- 172.21.15.46:0/3985229630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c15d210 msgr2=0x7f8e4c09bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:05.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.834+0000 7f8e522ad700 1 --2- 172.21.15.46:0/3985229630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c15d210 0x7f8e4c09bde0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f8e44001560 tx=0x7f8e440016c0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.834+0000 7f8e522ad700 1 -- 172.21.15.46:0/3985229630 shutdown_connections 2024-01-26T23:45:05.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.834+0000 7f8e522ad700 1 --2- 172.21.15.46:0/3985229630 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8e38067050 0x7f8e38069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.834+0000 7f8e522ad700 1 --2- 172.21.15.46:0/3985229630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c15d210 0x7f8e4c09bde0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.834+0000 7f8e522ad700 1 --2- 172.21.15.46:0/3985229630 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8e4c0d1390 0x7f8e4c15c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.834+0000 7f8e522ad700 1 --2- 172.21.15.46:0/3985229630 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e4c15ce40 0x7f8e4c15f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.834+0000 7f8e522ad700 1 -- 172.21.15.46:0/3985229630 >> 172.21.15.46:0/3985229630 conn(0x7f8e4c06f840 msgr2=0x7f8e4c0b5910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:05.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.834+0000 7f8e522ad700 1 -- 172.21.15.46:0/3985229630 shutdown_connections 2024-01-26T23:45:05.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.834+0000 7f8e522ad700 1 -- 172.21.15.46:0/3985229630 wait complete. 2024-01-26T23:45:05.849 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-26T23:45:05.850 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:45:05.964 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:05.963+0000 7fe195ee01c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:45:05.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.981+0000 7f4e31001700 1 Processor -- start 2024-01-26T23:45:05.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.981+0000 7f4e31001700 1 -- start start 2024-01-26T23:45:05.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.983+0000 7f4e31001700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4e2c157550 0x7f4e2c06f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:05.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.983+0000 7f4e31001700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4e2c0b2c30 0x7f4e2c06fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:05.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.983+0000 7f4e31001700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c0cc550 con 0x7f4e2c157920 2024-01-26T23:45:05.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.983+0000 7f4e31001700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2c0ca020 con 0x7f4e2c0b2c30 2024-01-26T23:45:05.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.983+0000 7f4e31001700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2c0cb6a0 con 0x7f4e2c157550 2024-01-26T23:45:05.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.983+0000 7f4e2b59e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4e2c0b2c30 0x7f4e2c06fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:05.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.983+0000 7f4e2b59e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4e2c0b2c30 0x7f4e2c06fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36074/0 (socket says 172.21.15.46:36074) 2024-01-26T23:45:05.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.983+0000 7f4e2b59e700 1 -- 172.21.15.46:0/122725773 learned_addr learned my addr 172.21.15.46:0/122725773 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:05.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.983+0000 7f4e2ad9d700 1 --2- 172.21.15.46:0/122725773 >> v2:172.21.15.106:3300/0 conn(0x7f4e2c157550 0x7f4e2c06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:05.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.983+0000 7f4e2b59e700 1 -- 172.21.15.46:0/122725773 >> v2:172.21.15.106:3300/0 conn(0x7f4e2c157550 msgr2=0x7f4e2c06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:05.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.983+0000 7f4e2b59e700 1 --2- 172.21.15.46:0/122725773 >> v2:172.21.15.106:3300/0 conn(0x7f4e2c157550 0x7f4e2c06f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.983+0000 7f4e2b59e700 1 -- 172.21.15.46:0/122725773 >> v1:172.21.15.106:6789/0 conn(0x7f4e2c157920 legacy=0x7f4e2c0b2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:05.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.983+0000 7f4e2b59e700 1 -- 172.21.15.46:0/122725773 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c0cb940 con 0x7f4e2c0b2c30 2024-01-26T23:45:05.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.983+0000 7f4e2b59e700 1 --2- 172.21.15.46:0/122725773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c0b2c30 0x7f4e2c06fa90 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c06e2c0 tx=0x7f4e24009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e03954dea7041bf3 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:05.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.984+0000 7f4e29d9b700 1 -- 172.21.15.46:0/122725773 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2400e070 con 0x7f4e2c0b2c30 2024-01-26T23:45:05.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.984+0000 7f4e29d9b700 1 -- 172.21.15.46:0/122725773 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2400ba20 con 0x7f4e2c0b2c30 2024-01-26T23:45:05.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.984+0000 7f4e29d9b700 1 -- 172.21.15.46:0/122725773 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e240092e0 con 0x7f4e2c0b2c30 2024-01-26T23:45:05.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.984+0000 7f4e31001700 1 -- 172.21.15.46:0/122725773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c0b2c30 msgr2=0x7f4e2c06fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:05.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.984+0000 7f4e31001700 1 --2- 172.21.15.46:0/122725773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c0b2c30 0x7f4e2c06fa90 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c06e2c0 tx=0x7f4e24009840 comp rx=0 tx=0).stop 2024-01-26T23:45:05.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.984+0000 7f4e31001700 1 -- 172.21.15.46:0/122725773 shutdown_connections 2024-01-26T23:45:05.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.984+0000 7f4e31001700 1 --2- 172.21.15.46:0/122725773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c0b2c30 0x7f4e2c06fa90 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.984+0000 7f4e31001700 1 --2- 172.21.15.46:0/122725773 >> v2:172.21.15.106:3300/0 conn(0x7f4e2c157550 0x7f4e2c06f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.984+0000 7f4e31001700 1 -- 172.21.15.46:0/122725773 >> 172.21.15.46:0/122725773 conn(0x7f4e2c069510 msgr2=0x7f4e2c0aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:05.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.984+0000 7f4e31001700 1 -- 172.21.15.46:0/122725773 shutdown_connections 2024-01-26T23:45:05.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.984+0000 7f4e31001700 1 -- 172.21.15.46:0/122725773 wait complete. 2024-01-26T23:45:05.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.986+0000 7f4e31001700 1 Processor -- start 2024-01-26T23:45:05.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.986+0000 7f4e31001700 1 -- start start 2024-01-26T23:45:05.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e31001700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e2c0b2c30 0x7f4e2c1513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:05.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e31001700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4e2c157550 0x7f4e2c16d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:05.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e31001700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c157920 0x7f4e2c169620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:05.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e31001700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c169ea0 con 0x7f4e2c157920 2024-01-26T23:45:05.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e31001700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c16a180 con 0x7f4e2c0b2c30 2024-01-26T23:45:05.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e31001700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c16ccc0 con 0x7f4e2c157550 2024-01-26T23:45:05.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e2b59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c157920 0x7f4e2c169620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:05.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e2b59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c157920 0x7f4e2c169620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36080/0 (socket says 172.21.15.46:36080) 2024-01-26T23:45:05.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e2b59e700 1 -- 172.21.15.46:0/3588069377 learned_addr learned my addr 172.21.15.46:0/3588069377 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:05.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e2ad9d700 1 --2- 172.21.15.46:0/3588069377 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e2c0b2c30 0x7f4e2c1513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:05.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e2a59c700 1 --2- 172.21.15.46:0/3588069377 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4e2c157550 0x7f4e2c16d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:05.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e2b59e700 1 -- 172.21.15.46:0/3588069377 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4e2c157550 msgr2=0x7f4e2c16d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:05.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e2b59e700 1 --2- 172.21.15.46:0/3588069377 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4e2c157550 0x7f4e2c16d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e2b59e700 1 -- 172.21.15.46:0/3588069377 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e2c0b2c30 msgr2=0x7f4e2c1513b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:05.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e2b59e700 1 --2- 172.21.15.46:0/3588069377 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e2c0b2c30 0x7f4e2c1513b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:05.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e2b59e700 1 -- 172.21.15.46:0/3588069377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e24009ae0 con 0x7f4e2c157920 2024-01-26T23:45:05.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e2b59e700 1 --2- 172.21.15.46:0/3588069377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c157920 0x7f4e2c169620 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4e24012000 tx=0x7f4e24018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:05.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.987+0000 7f4e12ffd700 1 -- 172.21.15.46:0/3588069377 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e240070e0 con 0x7f4e2c157920 2024-01-26T23:45:05.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.988+0000 7f4e12ffd700 1 -- 172.21.15.46:0/3588069377 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e24003300 con 0x7f4e2c157920 2024-01-26T23:45:05.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.988+0000 7f4e31001700 1 -- 172.21.15.46:0/3588069377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2c16cfa0 con 0x7f4e2c157920 2024-01-26T23:45:05.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.988+0000 7f4e12ffd700 1 -- 172.21.15.46:0/3588069377 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2400be50 con 0x7f4e2c157920 2024-01-26T23:45:05.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.988+0000 7f4e31001700 1 -- 172.21.15.46:0/3588069377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2c169b60 con 0x7f4e2c157920 2024-01-26T23:45:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.989+0000 7f4e31001700 1 -- 172.21.15.46:0/3588069377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2c16d1e0 con 0x7f4e2c157920 2024-01-26T23:45:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.989+0000 7f4e12ffd700 1 -- 172.21.15.46:0/3588069377 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f4e24003050 con 0x7f4e2c157920 2024-01-26T23:45:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.990+0000 7f4e12ffd700 1 --2- 172.21.15.46:0/3588069377 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4e18067050 0x7f4e18069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.990+0000 7f4e2ad9d700 1 -- 172.21.15.46:0/3588069377 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4e18067050 msgr2=0x7f4e18069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.990+0000 7f4e12ffd700 1 -- 172.21.15.46:0/3588069377 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6705+0+0 (secure 0 0 0) 0x7f4e240863e0 con 0x7f4e2c157920 2024-01-26T23:45:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.990+0000 7f4e2ad9d700 1 --2- 172.21.15.46:0/3588069377 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4e18067050 0x7f4e18069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:05.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:05.994+0000 7f4e12ffd700 1 -- 172.21.15.46:0/3588069377 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4e2c16d1e0 con 0x7f4e2c157920 2024-01-26T23:45:06.053 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:06.051+0000 7fe195ee01c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:45:06.191 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:06.190+0000 7fa6be3621c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:45:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.191+0000 7f4e2ad9d700 1 -- 172.21.15.46:0/3588069377 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4e18067050 msgr2=0x7f4e18069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.191+0000 7f4e2ad9d700 1 --2- 172.21.15.46:0/3588069377 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4e18067050 0x7f4e18069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:06.194 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:06.193+0000 7fe195ee01c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:45:06.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.247+0000 7f4e31001700 1 -- 172.21.15.46:0/3588069377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e2c06ba40 con 0x7f4e2c157920 2024-01-26T23:45:06.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.248+0000 7f4e12ffd700 1 -- 172.21.15.46:0/3588069377 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4603 (secure 0 0 0) 0x7f4e24053040 con 0x7f4e2c157920 2024-01-26T23:45:06.249 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:06.249 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3037972557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3037972557},{"type":"v1","addr":"172.21.15.106:6837","nonce":3037972557}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":5701,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3786017823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3786017823},{"type":"v1","addr":"172.21.15.106:6836","nonce":3786017823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":5704,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3329385210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3329385210},{"type":"v1","addr":"172.21.15.46:6835","nonce":3329385210}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:43:01.144129+0000","modified":"2024-01-26T23:43:04.718067+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6833/2801738403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2801738403},{"type":"v1","addr":"172.21.15.46:6833","nonce":2801738403}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2024-01-26T23:45:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.250+0000 7f4e31001700 1 -- 172.21.15.46:0/3588069377 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4e18067050 msgr2=0x7f4e18069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.250+0000 7f4e31001700 1 --2- 172.21.15.46:0/3588069377 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4e18067050 0x7f4e18069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.250+0000 7f4e31001700 1 -- 172.21.15.46:0/3588069377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c157920 msgr2=0x7f4e2c169620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.250+0000 7f4e31001700 1 --2- 172.21.15.46:0/3588069377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c157920 0x7f4e2c169620 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4e24012000 tx=0x7f4e24018000 comp rx=0 tx=0).stop 2024-01-26T23:45:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.250+0000 7f4e31001700 1 -- 172.21.15.46:0/3588069377 shutdown_connections 2024-01-26T23:45:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.250+0000 7f4e31001700 1 --2- 172.21.15.46:0/3588069377 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4e18067050 0x7f4e18069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.250+0000 7f4e31001700 1 --2- 172.21.15.46:0/3588069377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c157920 0x7f4e2c169620 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.250+0000 7f4e31001700 1 --2- 172.21.15.46:0/3588069377 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e2c0b2c30 0x7f4e2c1513b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.250+0000 7f4e31001700 1 --2- 172.21.15.46:0/3588069377 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4e2c157550 0x7f4e2c16d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.250+0000 7f4e31001700 1 -- 172.21.15.46:0/3588069377 >> 172.21.15.46:0/3588069377 conn(0x7f4e2c069510 msgr2=0x7f4e2c06f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.250+0000 7f4e31001700 1 -- 172.21.15.46:0/3588069377 shutdown_connections 2024-01-26T23:45:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.250+0000 7f4e31001700 1 -- 172.21.15.46:0/3588069377 wait complete. 2024-01-26T23:45:06.253 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 27 2024-01-26T23:45:06.266 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:45:06.337 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:06.336+0000 7fa6be3621c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:45:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.397+0000 7fa833556700 1 Processor -- start 2024-01-26T23:45:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.397+0000 7fa833556700 1 -- start start 2024-01-26T23:45:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.398+0000 7fa833556700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa82c0b2b40 0x7fa82c0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.398+0000 7fa833556700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa82c0ae960 0x7fa82c0ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.399+0000 7fa833556700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa82c0aedd0 0x7fa82c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.399+0000 7fa833556700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82c099fc0 con 0x7fa82c0ae960 2024-01-26T23:45:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.399+0000 7fa833556700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82c15a0b0 con 0x7fa82c0aedd0 2024-01-26T23:45:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.399+0000 7fa833556700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0ae4d0 con 0x7fa82c0b2b40 2024-01-26T23:45:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.399+0000 7fa830af1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa82c0ae960 0x7fa82c0ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.399+0000 7fa830af1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa82c0ae960 0x7fa82c0ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36088/0 (socket says 172.21.15.46:36088) 2024-01-26T23:45:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.399+0000 7fa830af1700 1 -- 172.21.15.46:0/3657130592 learned_addr learned my addr 172.21.15.46:0/3657130592 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.399+0000 7fa831af3700 1 --2- 172.21.15.46:0/3657130592 >> v2:172.21.15.106:3300/0 conn(0x7fa82c0aedd0 0x7fa82c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.399+0000 7fa8312f2700 1 --2- 172.21.15.46:0/3657130592 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa82c0b2b40 0x7fa82c0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.399+0000 7fa830af1700 1 -- 172.21.15.46:0/3657130592 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa82c0b2b40 msgr2=0x7fa82c0ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.399+0000 7fa830af1700 1 --2- 172.21.15.46:0/3657130592 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa82c0b2b40 0x7fa82c0ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.399+0000 7fa830af1700 1 -- 172.21.15.46:0/3657130592 >> v2:172.21.15.106:3300/0 conn(0x7fa82c0aedd0 msgr2=0x7fa82c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.399+0000 7fa830af1700 1 --2- 172.21.15.46:0/3657130592 >> v2:172.21.15.106:3300/0 conn(0x7fa82c0aedd0 0x7fa82c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.399+0000 7fa830af1700 1 -- 172.21.15.46:0/3657130592 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82c0ad8c0 con 0x7fa82c0ae960 2024-01-26T23:45:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.399+0000 7fa830af1700 1 --2- 172.21.15.46:0/3657130592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c0ae960 0x7fa82c0ced50 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0ae440 tx=0x7fa81c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e793272abd97d175 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.399+0000 7fa823fff700 1 -- 172.21.15.46:0/3657130592 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81c00b640 con 0x7fa82c0ae960 2024-01-26T23:45:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.400+0000 7fa823fff700 1 -- 172.21.15.46:0/3657130592 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81c00b7a0 con 0x7fa82c0ae960 2024-01-26T23:45:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.400+0000 7fa823fff700 1 -- 172.21.15.46:0/3657130592 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81c00bb60 con 0x7fa82c0ae960 2024-01-26T23:45:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.400+0000 7fa833556700 1 -- 172.21.15.46:0/3657130592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c0ae960 msgr2=0x7fa82c0ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.400+0000 7fa833556700 1 --2- 172.21.15.46:0/3657130592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c0ae960 0x7fa82c0ced50 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0ae440 tx=0x7fa81c009250 comp rx=0 tx=0).stop 2024-01-26T23:45:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.400+0000 7fa833556700 1 -- 172.21.15.46:0/3657130592 shutdown_connections 2024-01-26T23:45:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.400+0000 7fa833556700 1 --2- 172.21.15.46:0/3657130592 >> v2:172.21.15.106:3300/0 conn(0x7fa82c0aedd0 0x7fa82c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.400+0000 7fa833556700 1 --2- 172.21.15.46:0/3657130592 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa82c0b2b40 0x7fa82c0ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.400+0000 7fa833556700 1 --2- 172.21.15.46:0/3657130592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c0ae960 0x7fa82c0ced50 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.400+0000 7fa833556700 1 -- 172.21.15.46:0/3657130592 >> 172.21.15.46:0/3657130592 conn(0x7fa82c099a30 msgr2=0x7fa82c06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.401+0000 7fa833556700 1 -- 172.21.15.46:0/3657130592 shutdown_connections 2024-01-26T23:45:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.401+0000 7fa833556700 1 -- 172.21.15.46:0/3657130592 wait complete. 2024-01-26T23:45:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.402+0000 7fa833556700 1 Processor -- start 2024-01-26T23:45:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.402+0000 7fa833556700 1 -- start start 2024-01-26T23:45:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa833556700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa82c0ae960 0x7fa82c0b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa833556700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c0b2b40 0x7fa82c0afd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa833556700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa82c0ca650 0x7fa82c156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa833556700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0b2160 con 0x7fa82c0b2b40 2024-01-26T23:45:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa833556700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c09e230 con 0x7fa82c0ca650 2024-01-26T23:45:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa833556700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0cabd0 con 0x7fa82c0ae960 2024-01-26T23:45:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa830af1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c0b2b40 0x7fa82c0afd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa830af1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c0b2b40 0x7fa82c0afd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36090/0 (socket says 172.21.15.46:36090) 2024-01-26T23:45:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa830af1700 1 -- 172.21.15.46:0/547718547 learned_addr learned my addr 172.21.15.46:0/547718547 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa8312f2700 1 --2- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa82c0ae960 0x7fa82c0b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa831af3700 1 --2- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa82c0ca650 0x7fa82c156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa831af3700 1 -- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa82c0ae960 msgr2=0x7fa82c0b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa831af3700 1 --2- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa82c0ae960 0x7fa82c0b1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa831af3700 1 -- 172.21.15.46:0/547718547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c0b2b40 msgr2=0x7fa82c0afd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa831af3700 1 --2- 172.21.15.46:0/547718547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c0b2b40 0x7fa82c0afd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa831af3700 1 -- 172.21.15.46:0/547718547 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c0094f0 con 0x7fa82c0ca650 2024-01-26T23:45:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa8312f2700 1 --2- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa82c0ae960 0x7fa82c0b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.404+0000 7fa830af1700 1 --2- 172.21.15.46:0/547718547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c0b2b40 0x7fa82c0afd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:06.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.405+0000 7fa831af3700 1 --2- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa82c0ca650 0x7fa82c156f90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0c77d0 tx=0x7fa82400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:06.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.405+0000 7fa8217fa700 1 -- 172.21.15.46:0/547718547 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82400c890 con 0x7fa82c0ca650 2024-01-26T23:45:06.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.405+0000 7fa8217fa700 1 -- 172.21.15.46:0/547718547 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82400f040 con 0x7fa82c0ca650 2024-01-26T23:45:06.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.405+0000 7fa833556700 1 -- 172.21.15.46:0/547718547 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82c0cc220 con 0x7fa82c0ca650 2024-01-26T23:45:06.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.405+0000 7fa8217fa700 1 -- 172.21.15.46:0/547718547 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82400cc60 con 0x7fa82c0ca650 2024-01-26T23:45:06.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.405+0000 7fa833556700 1 -- 172.21.15.46:0/547718547 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82c0cc7e0 con 0x7fa82c0ca650 2024-01-26T23:45:06.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.406+0000 7fa833556700 1 -- 172.21.15.46:0/547718547 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa82c04c710 con 0x7fa82c0ca650 2024-01-26T23:45:06.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.407+0000 7fa8217fa700 1 -- 172.21.15.46:0/547718547 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fa824014690 con 0x7fa82c0ca650 2024-01-26T23:45:06.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.407+0000 7fa8217fa700 1 --2- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa810067050 0x7fa810069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:06.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.407+0000 7fa8312f2700 1 -- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa810067050 msgr2=0x7fa810069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:06.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.407+0000 7fa8217fa700 1 -- 172.21.15.46:0/547718547 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6705+0+0 (secure 0 0 0) 0x7fa824085600 con 0x7fa82c0ca650 2024-01-26T23:45:06.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.407+0000 7fa8312f2700 1 --2- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa810067050 0x7fa810069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:06.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.415+0000 7fa8217fa700 1 -- 172.21.15.46:0/547718547 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa8240521c0 con 0x7fa82c0ca650 2024-01-26T23:45:06.490 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:06.489+0000 7fa6be3621c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:45:06.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.608+0000 7fa8312f2700 1 -- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa810067050 msgr2=0x7fa810069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:06.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.608+0000 7fa8312f2700 1 --2- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa810067050 0x7fa810069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:06.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.668+0000 7fa833556700 1 -- 172.21.15.46:0/547718547 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa82c0cf900 con 0x7fa82c0ca650 2024-01-26T23:45:06.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.669+0000 7fa8217fa700 1 -- 172.21.15.46:0/547718547 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4603 (secure 0 0 0) 0x7fa8240557e0 con 0x7fa82c0ca650 2024-01-26T23:45:06.670 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:06.670 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3037972557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3037972557},{"type":"v1","addr":"172.21.15.106:6837","nonce":3037972557}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":5701,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3786017823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3786017823},{"type":"v1","addr":"172.21.15.106:6836","nonce":3786017823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":5704,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3329385210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3329385210},{"type":"v1","addr":"172.21.15.46:6835","nonce":3329385210}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:43:01.144129+0000","modified":"2024-01-26T23:43:04.718067+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6833/2801738403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2801738403},{"type":"v1","addr":"172.21.15.46:6833","nonce":2801738403}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2024-01-26T23:45:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.671+0000 7fa833556700 1 -- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa810067050 msgr2=0x7fa810069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.671+0000 7fa833556700 1 --2- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa810067050 0x7fa810069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.671+0000 7fa833556700 1 -- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa82c0ca650 msgr2=0x7fa82c156f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.671+0000 7fa833556700 1 --2- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa82c0ca650 0x7fa82c156f90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0c77d0 tx=0x7fa82400b040 comp rx=0 tx=0).stop 2024-01-26T23:45:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.671+0000 7fa833556700 1 -- 172.21.15.46:0/547718547 shutdown_connections 2024-01-26T23:45:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.671+0000 7fa833556700 1 --2- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa810067050 0x7fa810069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.671+0000 7fa833556700 1 --2- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa82c0ca650 0x7fa82c156f90 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.671+0000 7fa833556700 1 --2- 172.21.15.46:0/547718547 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa82c0ae960 0x7fa82c0b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.671+0000 7fa833556700 1 --2- 172.21.15.46:0/547718547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c0b2b40 0x7fa82c0afd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.671+0000 7fa833556700 1 -- 172.21.15.46:0/547718547 >> 172.21.15.46:0/547718547 conn(0x7fa82c099a30 msgr2=0x7fa82c158f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.671+0000 7fa833556700 1 -- 172.21.15.46:0/547718547 shutdown_connections 2024-01-26T23:45:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.671+0000 7fa833556700 1 -- 172.21.15.46:0/547718547 wait complete. 2024-01-26T23:45:06.674 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 27 2024-01-26T23:45:06.687 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:45:06.793 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:06.792+0000 7fe195ee01c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:45:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.820+0000 7fab9a872700 1 Processor -- start 2024-01-26T23:45:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.820+0000 7fab9a872700 1 -- start start 2024-01-26T23:45:06.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.821+0000 7fab9a872700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fab9415ce40 0x7fab940d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:06.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.821+0000 7fab9a872700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab940cff70 con 0x7fab9415d210 2024-01-26T23:45:06.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.821+0000 7fab9a872700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab940cf400 con 0x7fab940cc1c0 2024-01-26T23:45:06.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.821+0000 7fab9a872700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab940cf660 con 0x7fab9415ce40 2024-01-26T23:45:06.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.821+0000 7fab93fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fab9415ce40 0x7fab940d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:06.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.821+0000 7fab93fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fab9415ce40 0x7fab940d0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36106/0 (socket says 172.21.15.46:36106) 2024-01-26T23:45:06.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.821+0000 7fab93fff700 1 -- 172.21.15.46:0/4254806188 learned_addr learned my addr 172.21.15.46:0/4254806188 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:06.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.821+0000 7fab937fe700 1 --1- 172.21.15.46:0/4254806188 >> v1:172.21.15.46:6789/0 conn(0x7fab9415d210 0x7fab940cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35082/0 (socket says 172.21.15.46:35082) 2024-01-26T23:45:06.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.821+0000 7fab93fff700 1 -- 172.21.15.46:0/4254806188 >> v1:172.21.15.106:6789/0 conn(0x7fab940cc1c0 legacy=0x7fab940ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:06.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.821+0000 7fab93fff700 1 -- 172.21.15.46:0/4254806188 >> v1:172.21.15.46:6789/0 conn(0x7fab9415d210 legacy=0x7fab940cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.821+0000 7fab92ffd700 1 -- 172.21.15.46:0/4254806188 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466936447 0 0) 0x7fab940cf400 con 0x7fab940cc1c0 2024-01-26T23:45:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.821+0000 7fab93fff700 1 -- 172.21.15.46:0/4254806188 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab940d0500 con 0x7fab9415ce40 2024-01-26T23:45:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.821+0000 7fab93fff700 1 --2- 172.21.15.46:0/4254806188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9415ce40 0x7fab940d0e50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fab84009990 tx=0x7fab84009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f6b2062868b7e81 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.822+0000 7fab92ffd700 1 -- 172.21.15.46:0/4254806188 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8400b760 con 0x7fab9415ce40 2024-01-26T23:45:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.822+0000 7fab92ffd700 1 -- 172.21.15.46:0/4254806188 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab8400b8c0 con 0x7fab9415ce40 2024-01-26T23:45:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.822+0000 7fab92ffd700 1 -- 172.21.15.46:0/4254806188 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8400bc80 con 0x7fab9415ce40 2024-01-26T23:45:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.822+0000 7fab9a872700 1 -- 172.21.15.46:0/4254806188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9415ce40 msgr2=0x7fab940d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.822+0000 7fab9a872700 1 --2- 172.21.15.46:0/4254806188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9415ce40 0x7fab940d0e50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fab84009990 tx=0x7fab84009220 comp rx=0 tx=0).stop 2024-01-26T23:45:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.822+0000 7fab9a872700 1 -- 172.21.15.46:0/4254806188 shutdown_connections 2024-01-26T23:45:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.822+0000 7fab9a872700 1 --2- 172.21.15.46:0/4254806188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9415ce40 0x7fab940d0e50 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.822+0000 7fab9a872700 1 -- 172.21.15.46:0/4254806188 >> 172.21.15.46:0/4254806188 conn(0x7fab9406f840 msgr2=0x7fab940d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.822+0000 7fab9a872700 1 -- 172.21.15.46:0/4254806188 shutdown_connections 2024-01-26T23:45:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.822+0000 7fab9a872700 1 -- 172.21.15.46:0/4254806188 wait complete. 2024-01-26T23:45:06.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.824+0000 7fab9a872700 1 Processor -- start 2024-01-26T23:45:06.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.824+0000 7fab9a872700 1 -- start start 2024-01-26T23:45:06.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.825+0000 7fab9a872700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab940cc1c0 0x7fab9415fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:06.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.825+0000 7fab9a872700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9415ce40 0x7fab940b5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:06.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.825+0000 7fab9a872700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab9415d210 0x7fab940b6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:06.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.825+0000 7fab9a872700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab940c5230 con 0x7fab9415ce40 2024-01-26T23:45:06.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.825+0000 7fab9a872700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab940c8a90 con 0x7fab9415d210 2024-01-26T23:45:06.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.825+0000 7fab9a872700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab940c8d70 con 0x7fab940cc1c0 2024-01-26T23:45:06.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.825+0000 7fab937fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9415ce40 0x7fab940b5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:06.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.825+0000 7fab937fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9415ce40 0x7fab940b5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36110/0 (socket says 172.21.15.46:36110) 2024-01-26T23:45:06.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.825+0000 7fab937fe700 1 -- 172.21.15.46:0/1108785628 learned_addr learned my addr 172.21.15.46:0/1108785628 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:06.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.826+0000 7fab93fff700 1 --2- 172.21.15.46:0/1108785628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab940cc1c0 0x7fab9415fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:06.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.826+0000 7fab98e0f700 1 --2- 172.21.15.46:0/1108785628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab9415d210 0x7fab940b6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:06.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.826+0000 7fab937fe700 1 -- 172.21.15.46:0/1108785628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab940cc1c0 msgr2=0x7fab9415fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:06.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.826+0000 7fab937fe700 1 --2- 172.21.15.46:0/1108785628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab940cc1c0 0x7fab9415fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.826+0000 7fab937fe700 1 -- 172.21.15.46:0/1108785628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab9415d210 msgr2=0x7fab940b6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:06.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.826+0000 7fab937fe700 1 --2- 172.21.15.46:0/1108785628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab9415d210 0x7fab940b6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:06.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.826+0000 7fab937fe700 1 -- 172.21.15.46:0/1108785628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab84009480 con 0x7fab9415ce40 2024-01-26T23:45:06.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.826+0000 7fab937fe700 1 --2- 172.21.15.46:0/1108785628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9415ce40 0x7fab940b5b00 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fab8800c810 tx=0x7fab88007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:06.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.826+0000 7fab7bfff700 1 -- 172.21.15.46:0/1108785628 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab88005a70 con 0x7fab9415ce40 2024-01-26T23:45:06.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.826+0000 7fab7bfff700 1 -- 172.21.15.46:0/1108785628 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab88005bf0 con 0x7fab9415ce40 2024-01-26T23:45:06.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.826+0000 7fab9a872700 1 -- 172.21.15.46:0/1108785628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab940c9050 con 0x7fab9415ce40 2024-01-26T23:45:06.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.826+0000 7fab7bfff700 1 -- 172.21.15.46:0/1108785628 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab880070d0 con 0x7fab9415ce40 2024-01-26T23:45:06.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.826+0000 7fab9a872700 1 -- 172.21.15.46:0/1108785628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab940c9610 con 0x7fab9415ce40 2024-01-26T23:45:06.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.827+0000 7fab9a872700 1 -- 172.21.15.46:0/1108785628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab940c9830 con 0x7fab9415ce40 2024-01-26T23:45:06.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.828+0000 7fab7bfff700 1 -- 172.21.15.46:0/1108785628 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fab88007500 con 0x7fab9415ce40 2024-01-26T23:45:06.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.828+0000 7fab7bfff700 1 --2- 172.21.15.46:0/1108785628 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fab80067050 0x7fab80069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:06.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.828+0000 7fab7bfff700 1 -- 172.21.15.46:0/1108785628 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6705+0+0 (secure 0 0 0) 0x7fab88083f10 con 0x7fab9415ce40 2024-01-26T23:45:06.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.828+0000 7fab93fff700 1 -- 172.21.15.46:0/1108785628 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fab80067050 msgr2=0x7fab80069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:06.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.828+0000 7fab93fff700 1 --2- 172.21.15.46:0/1108785628 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fab80067050 0x7fab80069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:06.837+0000 7fab7bfff700 1 -- 172.21.15.46:0/1108785628 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fab88050c80 con 0x7fab9415ce40 2024-01-26T23:45:06.970 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:06.969+0000 7fe195ee01c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:45:07.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.029+0000 7fab93fff700 1 -- 172.21.15.46:0/1108785628 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fab80067050 msgr2=0x7fab80069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:07.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.029+0000 7fab93fff700 1 --2- 172.21.15.46:0/1108785628 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fab80067050 0x7fab80069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.043+0000 7fab9a872700 1 -- 172.21.15.46:0/1108785628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab9404c710 con 0x7fab9415ce40 2024-01-26T23:45:07.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.044+0000 7fab7bfff700 1 -- 172.21.15.46:0/1108785628 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+16571 (secure 0 0 0) 0x7fab88017020 con 0x7fab9415ce40 2024-01-26T23:45:07.046 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:07.046 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":73,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:44:21.651311+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:42:59.650806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-01-26T23:43:00.105558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2","mappings":[{"from":0,"to":6}]},{"pgid":"8.1d","mappings":[{"from":0,"to":6}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/1676475004":"2024-01-27T23:43:20.540793+0000","172.21.15.106:6833/43388":"2024-01-27T23:43:20.540793+0000","172.21.15.106:6832/43388":"2024-01-27T23:43:20.540793+0000","172.21.15.106:0/2801680181":"2024-01-27T23:43:20.540793+0000","172.21.15.106:0/677673260":"2024-01-27T23:43:20.540793+0000","172.21.15.106:0/904952331":"2024-01-27T23:43:20.540793+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:45:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.046+0000 7fab9a872700 1 -- 172.21.15.46:0/1108785628 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fab80067050 msgr2=0x7fab80069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.046+0000 7fab9a872700 1 --2- 172.21.15.46:0/1108785628 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fab80067050 0x7fab80069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.046+0000 7fab9a872700 1 -- 172.21.15.46:0/1108785628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9415ce40 msgr2=0x7fab940b5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.046+0000 7fab9a872700 1 --2- 172.21.15.46:0/1108785628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9415ce40 0x7fab940b5b00 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fab8800c810 tx=0x7fab88007ec0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.047+0000 7fab9a872700 1 -- 172.21.15.46:0/1108785628 shutdown_connections 2024-01-26T23:45:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.047+0000 7fab9a872700 1 --2- 172.21.15.46:0/1108785628 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fab80067050 0x7fab80069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.047+0000 7fab9a872700 1 --2- 172.21.15.46:0/1108785628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab9415d210 0x7fab940b6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.047+0000 7fab9a872700 1 --2- 172.21.15.46:0/1108785628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab940cc1c0 0x7fab9415fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.047+0000 7fab9a872700 1 --2- 172.21.15.46:0/1108785628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9415ce40 0x7fab940b5b00 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.047+0000 7fab9a872700 1 -- 172.21.15.46:0/1108785628 >> 172.21.15.46:0/1108785628 conn(0x7fab9406f840 msgr2=0x7fab9415bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.047+0000 7fab9a872700 1 -- 172.21.15.46:0/1108785628 shutdown_connections 2024-01-26T23:45:07.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.047+0000 7fab9a872700 1 -- 172.21.15.46:0/1108785628 wait complete. 2024-01-26T23:45:07.062 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-26T23:45:07.146 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:07.145+0000 7fe195ee01c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:45:07.191 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:07.190+0000 7fa6be3621c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:45:07.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.195+0000 7fb4e659b700 1 Processor -- start 2024-01-26T23:45:07.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.195+0000 7fb4e659b700 1 -- start start 2024-01-26T23:45:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.195+0000 7fb4e659b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb4e0157340 0x7fb4e015bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.195+0000 7fb4e659b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4e00b4180 0x7fb4e0157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.196+0000 7fb4e659b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e0158ae0 con 0x7fb4e00ae520 2024-01-26T23:45:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.196+0000 7fb4e659b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4e015b650 con 0x7fb4e00b4180 2024-01-26T23:45:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.196+0000 7fb4e659b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4e015c020 con 0x7fb4e0157340 2024-01-26T23:45:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.196+0000 7fb4e4b38700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4e00b4180 0x7fb4e0157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.196+0000 7fb4e4b38700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4e00b4180 0x7fb4e0157710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36112/0 (socket says 172.21.15.46:36112) 2024-01-26T23:45:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.196+0000 7fb4e4b38700 1 -- 172.21.15.46:0/1867062728 learned_addr learned my addr 172.21.15.46:0/1867062728 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.196+0000 7fb4dffff700 1 --2- 172.21.15.46:0/1867062728 >> v2:172.21.15.106:3300/0 conn(0x7fb4e0157340 0x7fb4e015bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.196+0000 7fb4e4b38700 1 -- 172.21.15.46:0/1867062728 >> v2:172.21.15.106:3300/0 conn(0x7fb4e0157340 msgr2=0x7fb4e015bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:07.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.196+0000 7fb4e4b38700 1 --2- 172.21.15.46:0/1867062728 >> v2:172.21.15.106:3300/0 conn(0x7fb4e0157340 0x7fb4e015bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.196+0000 7fb4e4b38700 1 -- 172.21.15.46:0/1867062728 >> v1:172.21.15.106:6789/0 conn(0x7fb4e00ae520 legacy=0x7fb4e00b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:07.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.196+0000 7fb4e4b38700 1 -- 172.21.15.46:0/1867062728 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e0158520 con 0x7fb4e00b4180 2024-01-26T23:45:07.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.196+0000 7fb4e4b38700 1 --2- 172.21.15.46:0/1867062728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e00b4180 0x7fb4e0157710 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb4e006f980 tx=0x7fb4d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb6bf2dcea7e1852 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:07.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.196+0000 7fb4deffd700 1 -- 172.21.15.46:0/1867062728 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d8009e70 con 0x7fb4e00b4180 2024-01-26T23:45:07.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.197+0000 7fb4deffd700 1 -- 172.21.15.46:0/1867062728 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d800b860 con 0x7fb4e00b4180 2024-01-26T23:45:07.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.197+0000 7fb4deffd700 1 -- 172.21.15.46:0/1867062728 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d800bc20 con 0x7fb4e00b4180 2024-01-26T23:45:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.197+0000 7fb4e659b700 1 -- 172.21.15.46:0/1867062728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e00b4180 msgr2=0x7fb4e0157710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.197+0000 7fb4e659b700 1 --2- 172.21.15.46:0/1867062728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e00b4180 0x7fb4e0157710 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb4e006f980 tx=0x7fb4d8009250 comp rx=0 tx=0).stop 2024-01-26T23:45:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.197+0000 7fb4e659b700 1 -- 172.21.15.46:0/1867062728 shutdown_connections 2024-01-26T23:45:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.197+0000 7fb4e659b700 1 --2- 172.21.15.46:0/1867062728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e00b4180 0x7fb4e0157710 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.197+0000 7fb4e659b700 1 --2- 172.21.15.46:0/1867062728 >> v2:172.21.15.106:3300/0 conn(0x7fb4e0157340 0x7fb4e015bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.197+0000 7fb4e659b700 1 -- 172.21.15.46:0/1867062728 >> 172.21.15.46:0/1867062728 conn(0x7fb4e00ce530 msgr2=0x7fb4e00ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.197+0000 7fb4e659b700 1 -- 172.21.15.46:0/1867062728 shutdown_connections 2024-01-26T23:45:07.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.197+0000 7fb4e659b700 1 -- 172.21.15.46:0/1867062728 wait complete. 2024-01-26T23:45:07.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.198+0000 7fb4e659b700 1 Processor -- start 2024-01-26T23:45:07.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.199+0000 7fb4e659b700 1 -- start start 2024-01-26T23:45:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.199+0000 7fb4e659b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4e00ae520 0x7fb4e00c2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.199+0000 7fb4e659b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e00b4180 0x7fb4e00b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.199+0000 7fb4e659b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4e0157340 0x7fb4e00b1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.199+0000 7fb4e659b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e0152e40 con 0x7fb4e00b4180 2024-01-26T23:45:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.199+0000 7fb4e659b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e00c4e30 con 0x7fb4e0157340 2024-01-26T23:45:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.199+0000 7fb4e659b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e00b2bc0 con 0x7fb4e00ae520 2024-01-26T23:45:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.200+0000 7fb4df7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e00b4180 0x7fb4e00b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.200+0000 7fb4e4b38700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4e0157340 0x7fb4e00b1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.200+0000 7fb4df7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e00b4180 0x7fb4e00b12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36114/0 (socket says 172.21.15.46:36114) 2024-01-26T23:45:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.200+0000 7fb4dffff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4e00ae520 0x7fb4e00c2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.200+0000 7fb4df7fe700 1 -- 172.21.15.46:0/3420349747 learned_addr learned my addr 172.21.15.46:0/3420349747 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.200+0000 7fb4e4b38700 1 -- 172.21.15.46:0/3420349747 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4e00ae520 msgr2=0x7fb4e00c2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.200+0000 7fb4e4b38700 1 --2- 172.21.15.46:0/3420349747 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4e00ae520 0x7fb4e00c2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.200+0000 7fb4e4b38700 1 -- 172.21.15.46:0/3420349747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e00b4180 msgr2=0x7fb4e00b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.200+0000 7fb4e4b38700 1 --2- 172.21.15.46:0/3420349747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e00b4180 0x7fb4e00b12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.200+0000 7fb4e4b38700 1 -- 172.21.15.46:0/3420349747 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d80094f0 con 0x7fb4e0157340 2024-01-26T23:45:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.200+0000 7fb4e4b38700 1 --2- 172.21.15.46:0/3420349747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4e0157340 0x7fb4e00b1820 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb4e006e840 tx=0x7fb4d8017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.200+0000 7fb4d7fff700 1 -- 172.21.15.46:0/3420349747 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d801f070 con 0x7fb4e0157340 2024-01-26T23:45:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.200+0000 7fb4d7fff700 1 -- 172.21.15.46:0/3420349747 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d8006ad0 con 0x7fb4e0157340 2024-01-26T23:45:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.200+0000 7fb4e659b700 1 -- 172.21.15.46:0/3420349747 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e00c48b0 con 0x7fb4e0157340 2024-01-26T23:45:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.200+0000 7fb4d7fff700 1 -- 172.21.15.46:0/3420349747 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d8011690 con 0x7fb4e0157340 2024-01-26T23:45:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.200+0000 7fb4e659b700 1 -- 172.21.15.46:0/3420349747 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4e00c4b50 con 0x7fb4e0157340 2024-01-26T23:45:07.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.202+0000 7fb4e659b700 1 -- 172.21.15.46:0/3420349747 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4e00b2d70 con 0x7fb4e0157340 2024-01-26T23:45:07.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.202+0000 7fb4d7fff700 1 -- 172.21.15.46:0/3420349747 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fb4d8011a40 con 0x7fb4e0157340 2024-01-26T23:45:07.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.202+0000 7fb4d7fff700 1 --2- 172.21.15.46:0/3420349747 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb4c0067050 0x7fb4c0069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:07.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.202+0000 7fb4dffff700 1 -- 172.21.15.46:0/3420349747 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb4c0067050 msgr2=0x7fb4c0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:07.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.202+0000 7fb4d7fff700 1 -- 172.21.15.46:0/3420349747 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6705+0+0 (secure 0 0 0) 0x7fb4d8085d50 con 0x7fb4e0157340 2024-01-26T23:45:07.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.202+0000 7fb4dffff700 1 --2- 172.21.15.46:0/3420349747 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb4c0067050 0x7fb4c0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:07.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.207+0000 7fb4d7fff700 1 -- 172.21.15.46:0/3420349747 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb4e00b2d70 con 0x7fb4e0157340 2024-01-26T23:45:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.403+0000 7fb4dffff700 1 -- 172.21.15.46:0/3420349747 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb4c0067050 msgr2=0x7fb4c0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.403+0000 7fb4dffff700 1 --2- 172.21.15.46:0/3420349747 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb4c0067050 0x7fb4c0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:07.408 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:07.407+0000 7fa6be3621c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:45:07.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.463+0000 7fb4e659b700 1 -- 172.21.15.46:0/3420349747 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb4e00b2d70 con 0x7fb4e0157340 2024-01-26T23:45:07.465 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-26T23:45:07.464+0000 7f33ffe86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-26T23:45:07.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.478+0000 7fb4d7fff700 1 -- 172.21.15.46:0/3420349747 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v28) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb4e00b2d70 con 0x7fb4e0157340 2024-01-26T23:45:07.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.482+0000 7fb4e659b700 1 -- 172.21.15.46:0/3420349747 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb4c0067050 msgr2=0x7fb4c0069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:07.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.482+0000 7fb4e659b700 1 --2- 172.21.15.46:0/3420349747 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb4c0067050 0x7fb4c0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.482+0000 7fb4e659b700 1 -- 172.21.15.46:0/3420349747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4e0157340 msgr2=0x7fb4e00b1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.482+0000 7fb4e659b700 1 --2- 172.21.15.46:0/3420349747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4e0157340 0x7fb4e00b1820 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb4e006e840 tx=0x7fb4d8017000 comp rx=0 tx=0).stop 2024-01-26T23:45:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.482+0000 7fb4e659b700 1 -- 172.21.15.46:0/3420349747 shutdown_connections 2024-01-26T23:45:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.482+0000 7fb4e659b700 1 --2- 172.21.15.46:0/3420349747 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb4c0067050 0x7fb4c0069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.482+0000 7fb4e659b700 1 --2- 172.21.15.46:0/3420349747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4e0157340 0x7fb4e00b1820 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.482+0000 7fb4e659b700 1 --2- 172.21.15.46:0/3420349747 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4e00ae520 0x7fb4e00c2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.482+0000 7fb4e659b700 1 --2- 172.21.15.46:0/3420349747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e00b4180 0x7fb4e00b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.482+0000 7fb4e659b700 1 -- 172.21.15.46:0/3420349747 >> 172.21.15.46:0/3420349747 conn(0x7fb4e00ce530 msgr2=0x7fb4e0157710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.482+0000 7fb4e659b700 1 -- 172.21.15.46:0/3420349747 shutdown_connections 2024-01-26T23:45:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.482+0000 7fb4e659b700 1 -- 172.21.15.46:0/3420349747 wait complete. 2024-01-26T23:45:07.485 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:07.483+0000 7fa6be3621c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:45:07.485 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-26T23:45:07.499 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-26T23:45:07.514 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-26T23:45:07.641 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:07.640+0000 7fa6be3621c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:45:07.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.640+0000 7f37a214f700 1 Processor -- start 2024-01-26T23:45:07.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.640+0000 7f37a214f700 1 -- start start 2024-01-26T23:45:07.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.641+0000 7f37a214f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f379c157490 0x7f379c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:07.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.641+0000 7f37a214f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379c0cea80 con 0x7f379c157860 2024-01-26T23:45:07.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.641+0000 7f37a214f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379c0cde70 con 0x7f379c06ddb0 2024-01-26T23:45:07.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.641+0000 7f37a214f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379c0cdc60 con 0x7f379c157490 2024-01-26T23:45:07.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.641+0000 7f379affd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f379c157860 0x7f379c06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35100/0 (socket says 172.21.15.46:35100) 2024-01-26T23:45:07.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.641+0000 7f379affd700 1 -- 172.21.15.46:0/3978164607 learned_addr learned my addr 172.21.15.46:0/3978164607 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:07.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.641+0000 7f379b7fe700 1 --2- 172.21.15.46:0/3978164607 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f379c157490 0x7f379c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:07.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.642+0000 7f379a7fc700 1 -- 172.21.15.46:0/3978164607 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936056584 0 0) 0x7f379c0cea80 con 0x7f379c157860 2024-01-26T23:45:07.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.642+0000 7f379a7fc700 1 -- 172.21.15.46:0/3978164607 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3788003570 con 0x7f379c157860 2024-01-26T23:45:07.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.642+0000 7f379a7fc700 1 -- 172.21.15.46:0/3978164607 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735663166 0 0) 0x7f379c0cde70 con 0x7f379c06ddb0 2024-01-26T23:45:07.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.642+0000 7f379b7fe700 1 -- 172.21.15.46:0/3978164607 >> v1:172.21.15.106:6789/0 conn(0x7f379c06ddb0 legacy=0x7f379c0b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:07.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.642+0000 7f379b7fe700 1 -- 172.21.15.46:0/3978164607 >> v1:172.21.15.46:6789/0 conn(0x7f379c157860 legacy=0x7f379c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:07.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.642+0000 7f379b7fe700 1 -- 172.21.15.46:0/3978164607 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379c0ce150 con 0x7f379c157490 2024-01-26T23:45:07.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.642+0000 7f379b7fe700 1 --2- 172.21.15.46:0/3978164607 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f379c157490 0x7f379c0cc600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f379c0cea30 tx=0x7f378c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b78515552665b4a7 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:07.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.642+0000 7f379a7fc700 1 -- 172.21.15.46:0/3978164607 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c016070 con 0x7f379c157490 2024-01-26T23:45:07.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.642+0000 7f379a7fc700 1 -- 172.21.15.46:0/3978164607 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378c00bb60 con 0x7f379c157490 2024-01-26T23:45:07.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.642+0000 7f379a7fc700 1 -- 172.21.15.46:0/3978164607 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c00e070 con 0x7f379c157490 2024-01-26T23:45:07.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.643+0000 7f37a214f700 1 -- 172.21.15.46:0/3978164607 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f379c157490 msgr2=0x7f379c0cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:07.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.643+0000 7f37a214f700 1 --2- 172.21.15.46:0/3978164607 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f379c157490 0x7f379c0cc600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f379c0cea30 tx=0x7f378c009780 comp rx=0 tx=0).stop 2024-01-26T23:45:07.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.643+0000 7f37a214f700 1 -- 172.21.15.46:0/3978164607 shutdown_connections 2024-01-26T23:45:07.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.643+0000 7f37a214f700 1 --2- 172.21.15.46:0/3978164607 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f379c157490 0x7f379c0cc600 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.643+0000 7f37a214f700 1 -- 172.21.15.46:0/3978164607 >> 172.21.15.46:0/3978164607 conn(0x7f379c067930 msgr2=0x7f379c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:07.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.643+0000 7f37a214f700 1 -- 172.21.15.46:0/3978164607 shutdown_connections 2024-01-26T23:45:07.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.643+0000 7f37a214f700 1 -- 172.21.15.46:0/3978164607 wait complete. 2024-01-26T23:45:07.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.644+0000 7f37a214f700 1 Processor -- start 2024-01-26T23:45:07.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.645+0000 7f37a214f700 1 -- start start 2024-01-26T23:45:07.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.645+0000 7f37a214f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f379c06ddb0 0x7f379c0c3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:07.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.645+0000 7f37a214f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f379c157490 0x7f379c0c3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:07.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.645+0000 7f37a214f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f379c157860 0x7f379c0adf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:07.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.645+0000 7f37a214f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c0aec60 con 0x7f379c06ddb0 2024-01-26T23:45:07.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.645+0000 7f37a214f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c0aef40 con 0x7f379c157490 2024-01-26T23:45:07.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.645+0000 7f37a214f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379c0af220 con 0x7f379c157860 2024-01-26T23:45:07.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.645+0000 7f379b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f379c06ddb0 0x7f379c0c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:07.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.646+0000 7f379affd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f379c157490 0x7f379c0c3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.646+0000 7f379b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f379c06ddb0 0x7f379c0c3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36132/0 (socket says 172.21.15.46:36132) 2024-01-26T23:45:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.646+0000 7f379bfff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f379c157860 0x7f379c0adf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.646+0000 7f379b7fe700 1 -- 172.21.15.46:0/2755292180 learned_addr learned my addr 172.21.15.46:0/2755292180 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.646+0000 7f379b7fe700 1 -- 172.21.15.46:0/2755292180 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f379c157860 msgr2=0x7f379c0adf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.646+0000 7f379b7fe700 1 --2- 172.21.15.46:0/2755292180 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f379c157860 0x7f379c0adf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.646+0000 7f379b7fe700 1 -- 172.21.15.46:0/2755292180 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f379c157490 msgr2=0x7f379c0c3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.646+0000 7f379b7fe700 1 --2- 172.21.15.46:0/2755292180 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f379c157490 0x7f379c0c3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.646+0000 7f379b7fe700 1 -- 172.21.15.46:0/2755292180 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378c009a20 con 0x7f379c06ddb0 2024-01-26T23:45:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.646+0000 7f379affd700 1 --2- 172.21.15.46:0/2755292180 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f379c157490 0x7f379c0c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.646+0000 7f379bfff700 1 --2- 172.21.15.46:0/2755292180 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f379c157860 0x7f379c0adf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.646+0000 7f379b7fe700 1 --2- 172.21.15.46:0/2755292180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f379c06ddb0 0x7f379c0c3350 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f379c0adc40 tx=0x7f378c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.646+0000 7f37937fe700 1 -- 172.21.15.46:0/2755292180 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c009420 con 0x7f379c06ddb0 2024-01-26T23:45:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.646+0000 7f37937fe700 1 -- 172.21.15.46:0/2755292180 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378c012d00 con 0x7f379c06ddb0 2024-01-26T23:45:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.646+0000 7f37937fe700 1 -- 172.21.15.46:0/2755292180 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c01b660 con 0x7f379c06ddb0 2024-01-26T23:45:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.646+0000 7f37a214f700 1 -- 172.21.15.46:0/2755292180 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f379c0af500 con 0x7f379c06ddb0 2024-01-26T23:45:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.647+0000 7f37a214f700 1 -- 172.21.15.46:0/2755292180 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f379c0ae490 con 0x7f379c06ddb0 2024-01-26T23:45:07.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.648+0000 7f37a214f700 1 -- 172.21.15.46:0/2755292180 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f379c04c710 con 0x7f379c06ddb0 2024-01-26T23:45:07.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.648+0000 7f37937fe700 1 -- 172.21.15.46:0/2755292180 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f378c003020 con 0x7f379c06ddb0 2024-01-26T23:45:07.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.648+0000 7f37937fe700 1 --2- 172.21.15.46:0/2755292180 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f377c067050 0x7f377c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:07.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.649+0000 7f37937fe700 1 -- 172.21.15.46:0/2755292180 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6847+0+0 (secure 0 0 0) 0x7f378c086c50 con 0x7f379c06ddb0 2024-01-26T23:45:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.649+0000 7f379affd700 1 -- 172.21.15.46:0/2755292180 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f377c067050 msgr2=0x7f377c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.649+0000 7f379affd700 1 --2- 172.21.15.46:0/2755292180 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f377c067050 0x7f377c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.653+0000 7f37937fe700 1 -- 172.21.15.46:0/2755292180 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f378c08c020 con 0x7f379c06ddb0 2024-01-26T23:45:07.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.665+0000 7f37937fe700 1 -- 172.21.15.46:0/2755292180 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 27) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f378c053d10 con 0x7f379c06ddb0 2024-01-26T23:45:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.665+0000 7f37937fe700 1 -- 172.21.15.46:0/2755292180 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f377c067050 msgr2=0x7f377c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.665+0000 7f37937fe700 1 --2- 172.21.15.46:0/2755292180 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f377c067050 0x7f377c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:07.721 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:07.721+0000 7fa63d411700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:07.722 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:07.721+0000 7fa63d411700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:07.722 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:07.721+0000 7fa63d411700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:07.722 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:07.721+0000 7fa63d411700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:07.722 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:07.721+0000 7fa63d411700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:07.722 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:07.721+0000 7fa639c0a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:07.722 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:07.721+0000 7fa639c0a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:07.722 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:07.721+0000 7fa639c0a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:07.722 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:07.721+0000 7fa639c0a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:07.722 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:07.721+0000 7fa639c0a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:07.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:07.917+0000 7f37a214f700 1 -- 172.21.15.46:0/2755292180 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f379c0cbd50 con 0x7f379c06ddb0 2024-01-26T23:45:07.974 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:07.973+0000 7fe195ee01c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:45:08.228 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:08.227+0000 7fe195ee01c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:45:08.312 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:08.311+0000 7fe195ee01c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:45:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.469+0000 7f37937fe700 1 -- 172.21.15.46:0/2755292180 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f378c027760 con 0x7f379c06ddb0 2024-01-26T23:45:08.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.471+0000 7f37a214f700 1 -- 172.21.15.46:0/2755292180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f379c06ddb0 msgr2=0x7f379c0c3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:08.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.471+0000 7f37a214f700 1 --2- 172.21.15.46:0/2755292180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f379c06ddb0 0x7f379c0c3350 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f379c0adc40 tx=0x7f378c018000 comp rx=0 tx=0).stop 2024-01-26T23:45:08.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.471+0000 7f37a214f700 1 -- 172.21.15.46:0/2755292180 shutdown_connections 2024-01-26T23:45:08.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.471+0000 7f37a214f700 1 --2- 172.21.15.46:0/2755292180 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f377c067050 0x7f377c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:08.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.471+0000 7f37a214f700 1 --2- 172.21.15.46:0/2755292180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f379c06ddb0 0x7f379c0c3350 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:08.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.471+0000 7f37a214f700 1 --2- 172.21.15.46:0/2755292180 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f379c157860 0x7f379c0adf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:08.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.471+0000 7f37a214f700 1 --2- 172.21.15.46:0/2755292180 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f379c157490 0x7f379c0c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:08.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.471+0000 7f37a214f700 1 -- 172.21.15.46:0/2755292180 >> 172.21.15.46:0/2755292180 conn(0x7f379c067930 msgr2=0x7f379c154490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:08.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.471+0000 7f37a214f700 1 -- 172.21.15.46:0/2755292180 shutdown_connections 2024-01-26T23:45:08.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.471+0000 7f37a214f700 1 -- 172.21.15.46:0/2755292180 wait complete. 2024-01-26T23:45:08.487 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-26T23:45:08.490 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:08.489+0000 7fe195ee01c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:45:08.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.625+0000 7f507d2b4700 1 Processor -- start 2024-01-26T23:45:08.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.625+0000 7f507d2b4700 1 -- start start 2024-01-26T23:45:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.626+0000 7f507d2b4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50780cb790 0x7f50780c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.626+0000 7f507d2b4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f50780cae30 0x7f50780c4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.626+0000 7f507d2b4700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50780cdbc0 con 0x7f50780c4940 2024-01-26T23:45:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.626+0000 7f507d2b4700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50780ce050 con 0x7f50780cae30 2024-01-26T23:45:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.626+0000 7f507d2b4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50780c34b0 con 0x7f50780cb790 2024-01-26T23:45:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.626+0000 7f50767fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f50780cae30 0x7f50780c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.626+0000 7f50767fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f50780cae30 0x7f50780c4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36138/0 (socket says 172.21.15.46:36138) 2024-01-26T23:45:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.626+0000 7f50767fc700 1 -- 172.21.15.46:0/1358946820 learned_addr learned my addr 172.21.15.46:0/1358946820 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.626+0000 7f5076ffd700 1 --2- 172.21.15.46:0/1358946820 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50780cb790 0x7f50780c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.626+0000 7f50777fe700 1 --1- 172.21.15.46:0/1358946820 >> v1:172.21.15.106:6789/0 conn(0x7f50780c4940 0x7f5078068a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:59588/0 (socket says 172.21.15.46:59588) 2024-01-26T23:45:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.626+0000 7f50767fc700 1 -- 172.21.15.46:0/1358946820 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50780cb790 msgr2=0x7f50780c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.626+0000 7f50767fc700 1 --2- 172.21.15.46:0/1358946820 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50780cb790 0x7f50780c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.626+0000 7f50767fc700 1 -- 172.21.15.46:0/1358946820 >> v1:172.21.15.106:6789/0 conn(0x7f50780c4940 legacy=0x7f5078068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.626+0000 7f5075ffb700 1 -- 172.21.15.46:0/1358946820 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616165973 0 0) 0x7f50780cdbc0 con 0x7f50780c4940 2024-01-26T23:45:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.626+0000 7f50767fc700 1 -- 172.21.15.46:0/1358946820 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50780ccfb0 con 0x7f50780cae30 2024-01-26T23:45:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.626+0000 7f50767fc700 1 --2- 172.21.15.46:0/1358946820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50780cae30 0x7f50780c4400 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f506c0099b0 tx=0x7f506c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758ad46404d251b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.627+0000 7f5075ffb700 1 -- 172.21.15.46:0/1358946820 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c009ea0 con 0x7f50780cae30 2024-01-26T23:45:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.627+0000 7f5075ffb700 1 -- 172.21.15.46:0/1358946820 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506c00b860 con 0x7f50780cae30 2024-01-26T23:45:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.627+0000 7f5075ffb700 1 -- 172.21.15.46:0/1358946820 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c00bc20 con 0x7f50780cae30 2024-01-26T23:45:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.627+0000 7f507d2b4700 1 -- 172.21.15.46:0/1358946820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50780cae30 msgr2=0x7f50780c4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.627+0000 7f507d2b4700 1 --2- 172.21.15.46:0/1358946820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50780cae30 0x7f50780c4400 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f506c0099b0 tx=0x7f506c009250 comp rx=0 tx=0).stop 2024-01-26T23:45:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.627+0000 7f507d2b4700 1 -- 172.21.15.46:0/1358946820 shutdown_connections 2024-01-26T23:45:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.627+0000 7f507d2b4700 1 --2- 172.21.15.46:0/1358946820 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50780cb790 0x7f50780c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.627+0000 7f507d2b4700 1 --2- 172.21.15.46:0/1358946820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50780cae30 0x7f50780c4400 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.627+0000 7f507d2b4700 1 -- 172.21.15.46:0/1358946820 >> 172.21.15.46:0/1358946820 conn(0x7f5078087f60 msgr2=0x7f5078088330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.627+0000 7f507d2b4700 1 -- 172.21.15.46:0/1358946820 shutdown_connections 2024-01-26T23:45:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.627+0000 7f507d2b4700 1 -- 172.21.15.46:0/1358946820 wait complete. 2024-01-26T23:45:08.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.629+0000 7f507d2b4700 1 Processor -- start 2024-01-26T23:45:08.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.629+0000 7f507d2b4700 1 -- start start 2024-01-26T23:45:08.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f507d2b4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50780c4940 0x7f5078145540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:08.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f507d2b4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50780cae30 0x7f50780aedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f507d2b4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50780cb790 0x7f50780af2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f507d2b4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5078145a80 con 0x7f50780c4940 2024-01-26T23:45:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f507d2b4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50780b1720 con 0x7f50780cb790 2024-01-26T23:45:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f507d2b4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50780b1a00 con 0x7f50780cae30 2024-01-26T23:45:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f5076ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50780c4940 0x7f5078145540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f5076ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50780c4940 0x7f5078145540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36142/0 (socket says 172.21.15.46:36142) 2024-01-26T23:45:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f5076ffd700 1 -- 172.21.15.46:0/415626819 learned_addr learned my addr 172.21.15.46:0/415626819 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f50777fe700 1 --2- 172.21.15.46:0/415626819 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50780cb790 0x7f50780af2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f5076ffd700 1 -- 172.21.15.46:0/415626819 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50780cae30 msgr2=0x7f50780aedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f5076ffd700 1 --2- 172.21.15.46:0/415626819 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50780cae30 0x7f50780aedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f5076ffd700 1 -- 172.21.15.46:0/415626819 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50780cb790 msgr2=0x7f50780af2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f5076ffd700 1 --2- 172.21.15.46:0/415626819 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50780cb790 0x7f50780af2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f5076ffd700 1 -- 172.21.15.46:0/415626819 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c0094f0 con 0x7f50780c4940 2024-01-26T23:45:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f50777fe700 1 --2- 172.21.15.46:0/415626819 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50780cb790 0x7f50780af2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:45:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f5076ffd700 1 --2- 172.21.15.46:0/415626819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50780c4940 0x7f5078145540 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f506800ec80 tx=0x7f5068009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f505effd700 1 -- 172.21.15.46:0/415626819 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506800c9d0 con 0x7f50780c4940 2024-01-26T23:45:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f505effd700 1 -- 172.21.15.46:0/415626819 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506800cb30 con 0x7f50780c4940 2024-01-26T23:45:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f507d2b4700 1 -- 172.21.15.46:0/415626819 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50780b1ce0 con 0x7f50780c4940 2024-01-26T23:45:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f505effd700 1 -- 172.21.15.46:0/415626819 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068013470 con 0x7f50780c4940 2024-01-26T23:45:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.630+0000 7f507d2b4700 1 -- 172.21.15.46:0/415626819 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50780b1380 con 0x7f50780c4940 2024-01-26T23:45:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.632+0000 7f507d2b4700 1 -- 172.21.15.46:0/415626819 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50780b2200 con 0x7f50780c4940 2024-01-26T23:45:08.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.632+0000 7f505effd700 1 -- 172.21.15.46:0/415626819 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f5068013860 con 0x7f50780c4940 2024-01-26T23:45:08.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.633+0000 7f505effd700 1 -- 172.21.15.46:0/415626819 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7077+0+0 (secure 0 0 0) 0x7f5068084cf0 con 0x7f50780c4940 2024-01-26T23:45:08.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.637+0000 7f505effd700 1 -- 172.21.15.46:0/415626819 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f50780b2200 con 0x7f50780c4940 2024-01-26T23:45:08.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.662+0000 7f505effd700 1 -- 172.21.15.46:0/415626819 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 28) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f5068051870 con 0x7f50780c4940 2024-01-26T23:45:08.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.663+0000 7f505effd700 1 --2- 172.21.15.46:0/415626819 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f506406d020 0x7f506406f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:08.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.663+0000 7f50767fc700 1 --2- 172.21.15.46:0/415626819 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f506406d020 0x7f506406f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:08.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.667+0000 7f50767fc700 1 --2- 172.21.15.46:0/415626819 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f506406d020 0x7f506406f400 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f506c0099b0 tx=0x7f506c017000 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:08.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:08.848+0000 7f507d2b4700 1 -- 172.21.15.46:0/415626819 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5078152320 con 0x7f50780c4940 2024-01-26T23:45:09.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.708+0000 7f505effd700 1 -- 172.21.15.46:0/415626819 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v76) v1 ==== 181+0+0 (secure 0 0 0) 0x7f50680170d0 con 0x7f50780c4940 2024-01-26T23:45:09.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.712+0000 7f507d2b4700 1 -- 172.21.15.46:0/415626819 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f506406d020 msgr2=0x7f506406f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:09.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.712+0000 7f507d2b4700 1 --2- 172.21.15.46:0/415626819 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f506406d020 0x7f506406f400 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f506c0099b0 tx=0x7f506c017000 comp rx=0 tx=0).stop 2024-01-26T23:45:09.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.712+0000 7f507d2b4700 1 -- 172.21.15.46:0/415626819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50780c4940 msgr2=0x7f5078145540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:09.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.712+0000 7f507d2b4700 1 --2- 172.21.15.46:0/415626819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50780c4940 0x7f5078145540 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f506800ec80 tx=0x7f5068009120 comp rx=0 tx=0).stop 2024-01-26T23:45:09.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.712+0000 7f507d2b4700 1 -- 172.21.15.46:0/415626819 shutdown_connections 2024-01-26T23:45:09.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.712+0000 7f507d2b4700 1 --2- 172.21.15.46:0/415626819 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f506406d020 0x7f506406f400 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:09.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.712+0000 7f507d2b4700 1 --2- 172.21.15.46:0/415626819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50780c4940 0x7f5078145540 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:09.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.712+0000 7f507d2b4700 1 --2- 172.21.15.46:0/415626819 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50780cb790 0x7f50780af2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:09.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.712+0000 7f507d2b4700 1 --2- 172.21.15.46:0/415626819 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50780cae30 0x7f50780aedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:09.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.712+0000 7f507d2b4700 1 -- 172.21.15.46:0/415626819 >> 172.21.15.46:0/415626819 conn(0x7f5078087f60 msgr2=0x7f507806c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:09.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.713+0000 7f507d2b4700 1 -- 172.21.15.46:0/415626819 shutdown_connections 2024-01-26T23:45:09.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.713+0000 7f507d2b4700 1 -- 172.21.15.46:0/415626819 wait complete. 2024-01-26T23:45:09.716 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-01-26T23:45:09.731 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-26T23:45:09.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.870+0000 7f5f2e3e4700 1 Processor -- start 2024-01-26T23:45:09.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.870+0000 7f5f2e3e4700 1 -- start start 2024-01-26T23:45:09.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.871+0000 7f5f2e3e4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5f28159750 0x7f5f28069650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:09.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.871+0000 7f5f2e3e4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f28159b20 0x7f5f280ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:09.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.871+0000 7f5f2e3e4700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f280691c0 con 0x7f5f28069b90 2024-01-26T23:45:09.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.871+0000 7f5f2e3e4700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f2806a5e0 con 0x7f5f28159750 2024-01-26T23:45:09.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.871+0000 7f5f2e3e4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2815dfd0 con 0x7f5f28159b20 2024-01-26T23:45:09.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.871+0000 7f5f27fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5f28159750 0x7f5f28069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:09.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.871+0000 7f5f27fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5f28159750 0x7f5f28069650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36154/0 (socket says 172.21.15.46:36154) 2024-01-26T23:45:09.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.871+0000 7f5f27fff700 1 -- 172.21.15.46:0/4052378741 learned_addr learned my addr 172.21.15.46:0/4052378741 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.871+0000 7f5f277fe700 1 --2- 172.21.15.46:0/4052378741 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f28159b20 0x7f5f280ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.872+0000 7f5f27fff700 1 -- 172.21.15.46:0/4052378741 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f28159b20 msgr2=0x7f5f280ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.872+0000 7f5f27fff700 1 --2- 172.21.15.46:0/4052378741 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f28159b20 0x7f5f280ade00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.872+0000 7f5f27fff700 1 -- 172.21.15.46:0/4052378741 >> v1:172.21.15.106:6789/0 conn(0x7f5f28069b90 legacy=0x7f5f2815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.872+0000 7f5f27fff700 1 -- 172.21.15.46:0/4052378741 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f28068b70 con 0x7f5f28159750 2024-01-26T23:45:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.872+0000 7f5f27fff700 1 --2- 172.21.15.46:0/4052378741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f28159750 0x7f5f28069650 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f5f180099b0 tx=0x7f5f18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e79a76aafd69741 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.872+0000 7f5f26ffd700 1 -- 172.21.15.46:0/4052378741 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f18009ea0 con 0x7f5f28159750 2024-01-26T23:45:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.872+0000 7f5f26ffd700 1 -- 172.21.15.46:0/4052378741 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1800b860 con 0x7f5f28159750 2024-01-26T23:45:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.872+0000 7f5f26ffd700 1 -- 172.21.15.46:0/4052378741 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1800bc00 con 0x7f5f28159750 2024-01-26T23:45:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.873+0000 7f5f2e3e4700 1 -- 172.21.15.46:0/4052378741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f28159750 msgr2=0x7f5f28069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.873+0000 7f5f2e3e4700 1 --2- 172.21.15.46:0/4052378741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f28159750 0x7f5f28069650 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f5f180099b0 tx=0x7f5f18009250 comp rx=0 tx=0).stop 2024-01-26T23:45:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.873+0000 7f5f2e3e4700 1 -- 172.21.15.46:0/4052378741 shutdown_connections 2024-01-26T23:45:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.873+0000 7f5f2e3e4700 1 --2- 172.21.15.46:0/4052378741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f28159750 0x7f5f28069650 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.873+0000 7f5f2e3e4700 1 --2- 172.21.15.46:0/4052378741 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f28159b20 0x7f5f280ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:09.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.873+0000 7f5f2e3e4700 1 -- 172.21.15.46:0/4052378741 >> 172.21.15.46:0/4052378741 conn(0x7f5f280ce530 msgr2=0x7f5f280ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:09.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.873+0000 7f5f2e3e4700 1 -- 172.21.15.46:0/4052378741 shutdown_connections 2024-01-26T23:45:09.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.873+0000 7f5f2e3e4700 1 -- 172.21.15.46:0/4052378741 wait complete. 2024-01-26T23:45:09.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.874+0000 7f5f2e3e4700 1 Processor -- start 2024-01-26T23:45:09.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.874+0000 7f5f2e3e4700 1 -- start start 2024-01-26T23:45:09.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.875+0000 7f5f2e3e4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f28069b90 0x7f5f280b0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:09.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.875+0000 7f5f2e3e4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f28159750 0x7f5f2816ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:09.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.875+0000 7f5f2e3e4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f28159b20 0x7f5f2815a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:09.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.875+0000 7f5f2e3e4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f28169ec0 con 0x7f5f28069b90 2024-01-26T23:45:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.875+0000 7f5f2e3e4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2816a1a0 con 0x7f5f28159750 2024-01-26T23:45:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.875+0000 7f5f2e3e4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2816a480 con 0x7f5f28159b20 2024-01-26T23:45:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.875+0000 7f5f27fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f28069b90 0x7f5f280b0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.875+0000 7f5f27fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f28069b90 0x7f5f280b0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36168/0 (socket says 172.21.15.46:36168) 2024-01-26T23:45:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.875+0000 7f5f27fff700 1 -- 172.21.15.46:0/487879975 learned_addr learned my addr 172.21.15.46:0/487879975 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.875+0000 7f5f277fe700 1 --2- 172.21.15.46:0/487879975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f28159750 0x7f5f2816ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.875+0000 7f5f2c981700 1 --2- 172.21.15.46:0/487879975 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f28159b20 0x7f5f2815a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.875+0000 7f5f27fff700 1 -- 172.21.15.46:0/487879975 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f28159b20 msgr2=0x7f5f2815a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.875+0000 7f5f27fff700 1 --2- 172.21.15.46:0/487879975 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f28159b20 0x7f5f2815a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.875+0000 7f5f27fff700 1 -- 172.21.15.46:0/487879975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f28159750 msgr2=0x7f5f2816ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.875+0000 7f5f27fff700 1 --2- 172.21.15.46:0/487879975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f28159750 0x7f5f2816ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.875+0000 7f5f27fff700 1 -- 172.21.15.46:0/487879975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f180094f0 con 0x7f5f28069b90 2024-01-26T23:45:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.875+0000 7f5f2c981700 1 --2- 172.21.15.46:0/487879975 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f28159b20 0x7f5f2815a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.876+0000 7f5f27fff700 1 --2- 172.21.15.46:0/487879975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f28069b90 0x7f5f280b0740 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f5f2806f390 tx=0x7f5f18017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.876+0000 7f5f0ffff700 1 -- 172.21.15.46:0/487879975 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1800bbe0 con 0x7f5f28069b90 2024-01-26T23:45:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.876+0000 7f5f0ffff700 1 -- 172.21.15.46:0/487879975 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f18011420 con 0x7f5f28069b90 2024-01-26T23:45:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.876+0000 7f5f0ffff700 1 -- 172.21.15.46:0/487879975 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f180117c0 con 0x7f5f28069b90 2024-01-26T23:45:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.876+0000 7f5f2e3e4700 1 -- 172.21.15.46:0/487879975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f2816a760 con 0x7f5f28069b90 2024-01-26T23:45:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.876+0000 7f5f2e3e4700 1 -- 172.21.15.46:0/487879975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f28169b40 con 0x7f5f28069b90 2024-01-26T23:45:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.877+0000 7f5f2e3e4700 1 -- 172.21.15.46:0/487879975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f2816d300 con 0x7f5f28069b90 2024-01-26T23:45:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.877+0000 7f5f0ffff700 1 -- 172.21.15.46:0/487879975 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 28) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f5f18011bf0 con 0x7f5f28069b90 2024-01-26T23:45:09.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.878+0000 7f5f0ffff700 1 --2- 172.21.15.46:0/487879975 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5f14066d80 0x7f5f14069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:09.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.878+0000 7f5f0ffff700 1 -- 172.21.15.46:0/487879975 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6639+0+0 (secure 0 0 0) 0x7f5f18085a70 con 0x7f5f28069b90 2024-01-26T23:45:09.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.878+0000 7f5f277fe700 1 --2- 172.21.15.46:0/487879975 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5f14066d80 0x7f5f14069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:09.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.878+0000 7f5f277fe700 1 --2- 172.21.15.46:0/487879975 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5f14066d80 0x7f5f14069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5f2816d0a0 tx=0x7f5f1c0091a0 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:09.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:09.882+0000 7f5f0ffff700 1 -- 172.21.15.46:0/487879975 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5f18052c70 con 0x7f5f28069b90 2024-01-26T23:45:10.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.089+0000 7f5f2e3e4700 1 -- 172.21.15.46:0/487879975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5f2804c710 con 0x7f5f28069b90 2024-01-26T23:45:10.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.669+0000 7f5f0ffff700 1 -- 172.21.15.46:0/487879975 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 29) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f5f18053040 con 0x7f5f28069b90 2024-01-26T23:45:10.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.703+0000 7f5f0ffff700 1 -- 172.21.15.46:0/487879975 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v77) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5f18056700 con 0x7f5f28069b90 2024-01-26T23:45:10.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.707+0000 7f5f2e3e4700 1 -- 172.21.15.46:0/487879975 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5f14066d80 msgr2=0x7f5f14069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:10.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.707+0000 7f5f2e3e4700 1 --2- 172.21.15.46:0/487879975 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5f14066d80 0x7f5f14069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5f2816d0a0 tx=0x7f5f1c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:10.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.707+0000 7f5f2e3e4700 1 -- 172.21.15.46:0/487879975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f28069b90 msgr2=0x7f5f280b0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:10.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.707+0000 7f5f2e3e4700 1 --2- 172.21.15.46:0/487879975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f28069b90 0x7f5f280b0740 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f5f2806f390 tx=0x7f5f18017000 comp rx=0 tx=0).stop 2024-01-26T23:45:10.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.708+0000 7f5f2e3e4700 1 -- 172.21.15.46:0/487879975 shutdown_connections 2024-01-26T23:45:10.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.708+0000 7f5f2e3e4700 1 --2- 172.21.15.46:0/487879975 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5f14066d80 0x7f5f14069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:10.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.708+0000 7f5f2e3e4700 1 --2- 172.21.15.46:0/487879975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f28069b90 0x7f5f280b0740 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:10.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.708+0000 7f5f2e3e4700 1 --2- 172.21.15.46:0/487879975 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f28159b20 0x7f5f2815a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:10.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.708+0000 7f5f2e3e4700 1 --2- 172.21.15.46:0/487879975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f28159750 0x7f5f2816ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:10.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.708+0000 7f5f2e3e4700 1 -- 172.21.15.46:0/487879975 >> 172.21.15.46:0/487879975 conn(0x7f5f280ce530 msgr2=0x7f5f281584c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:10.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.709+0000 7f5f2e3e4700 1 -- 172.21.15.46:0/487879975 shutdown_connections 2024-01-26T23:45:10.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.709+0000 7f5f2e3e4700 1 -- 172.21.15.46:0/487879975 wait complete. 2024-01-26T23:45:10.718 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-01-26T23:45:10.730 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-26T23:45:10.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.865+0000 7f5819999700 1 Processor -- start 2024-01-26T23:45:10.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.865+0000 7f5819999700 1 -- start start 2024-01-26T23:45:10.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.866+0000 7f5819999700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58141599d0 0x7f5814159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:10.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.866+0000 7f5819999700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f581415d8c0 0x7f581415dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:10.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.866+0000 7f5819999700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f581415e260 con 0x7f58140ab1c0 2024-01-26T23:45:10.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.866+0000 7f5819999700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58140b1a60 con 0x7f581415d8c0 2024-01-26T23:45:10.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.866+0000 7f5819999700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58140ab5f0 con 0x7f58141599d0 2024-01-26T23:45:10.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.866+0000 7f58137fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f581415d8c0 0x7f581415dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:10.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.866+0000 7f58127fc700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f58140ab1c0 0x7f58140b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:53604/0 (socket says 172.21.15.46:53604) 2024-01-26T23:45:10.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.866+0000 7f5812ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58141599d0 0x7f5814159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:10.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.866+0000 7f58127fc700 1 -- 172.21.15.46:0/2950850360 learned_addr learned my addr 172.21.15.46:0/2950850360 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.866+0000 7f58137fe700 1 --2- 172.21.15.46:0/2950850360 >> v2:172.21.15.106:3300/0 conn(0x7f581415d8c0 0x7f581415dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:46422/0 (socket says 172.21.15.46:46422) 2024-01-26T23:45:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.867+0000 7f58137fe700 1 -- 172.21.15.46:0/2950850360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58141599d0 msgr2=0x7f5814159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.867+0000 7f58137fe700 1 --2- 172.21.15.46:0/2950850360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58141599d0 0x7f5814159da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.867+0000 7f58137fe700 1 -- 172.21.15.46:0/2950850360 >> v1:172.21.15.106:6789/0 conn(0x7f58140ab1c0 legacy=0x7f58140b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.867+0000 7f58137fe700 1 -- 172.21.15.46:0/2950850360 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58140b11c0 con 0x7f581415d8c0 2024-01-26T23:45:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.867+0000 7f5811ffb700 1 -- 172.21.15.46:0/2950850360 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397607124 0 0) 0x7f57fc001110 con 0x7f58140ab1c0 2024-01-26T23:45:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.867+0000 7f58137fe700 1 --2- 172.21.15.46:0/2950850360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f581415d8c0 0x7f581415dd20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f580c009e60 tx=0x7f580c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8325a914ae6aaf7e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.867+0000 7f5811ffb700 1 -- 172.21.15.46:0/2950850360 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c00e070 con 0x7f581415d8c0 2024-01-26T23:45:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.867+0000 7f5811ffb700 1 -- 172.21.15.46:0/2950850360 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580c00bb60 con 0x7f581415d8c0 2024-01-26T23:45:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.867+0000 7f5811ffb700 1 -- 172.21.15.46:0/2950850360 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c012430 con 0x7f581415d8c0 2024-01-26T23:45:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.868+0000 7f5819999700 1 -- 172.21.15.46:0/2950850360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f581415d8c0 msgr2=0x7f581415dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.868+0000 7f5819999700 1 --2- 172.21.15.46:0/2950850360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f581415d8c0 0x7f581415dd20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f580c009e60 tx=0x7f580c009780 comp rx=0 tx=0).stop 2024-01-26T23:45:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.868+0000 7f5819999700 1 -- 172.21.15.46:0/2950850360 shutdown_connections 2024-01-26T23:45:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.868+0000 7f5819999700 1 --2- 172.21.15.46:0/2950850360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f581415d8c0 0x7f581415dd20 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:10.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.868+0000 7f5819999700 1 --2- 172.21.15.46:0/2950850360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58141599d0 0x7f5814159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:10.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.868+0000 7f5819999700 1 -- 172.21.15.46:0/2950850360 >> 172.21.15.46:0/2950850360 conn(0x7f581406f840 msgr2=0x7f5814087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:10.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.868+0000 7f5819999700 1 -- 172.21.15.46:0/2950850360 shutdown_connections 2024-01-26T23:45:10.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.868+0000 7f5819999700 1 -- 172.21.15.46:0/2950850360 wait complete. 2024-01-26T23:45:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.869+0000 7f5819999700 1 Processor -- start 2024-01-26T23:45:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.869+0000 7f5819999700 1 -- start start 2024-01-26T23:45:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.870+0000 7f5819999700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58140ab1c0 0x7f581409c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.870+0000 7f5819999700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58141599d0 0x7f58140aef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.870+0000 7f5819999700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f581415d8c0 0x7f5814158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.870+0000 7f5819999700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581409ab00 con 0x7f58141599d0 2024-01-26T23:45:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.870+0000 7f5819999700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58140af920 con 0x7f58140ab1c0 2024-01-26T23:45:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.870+0000 7f5819999700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f581409a4b0 con 0x7f581415d8c0 2024-01-26T23:45:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.870+0000 7f58127fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58141599d0 0x7f58140aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.870+0000 7f58127fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58141599d0 0x7f58140aef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44522/0 (socket says 172.21.15.46:44522) 2024-01-26T23:45:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.870+0000 7f58127fc700 1 -- 172.21.15.46:0/3121293187 learned_addr learned my addr 172.21.15.46:0/3121293187 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.870+0000 7f5812ffd700 1 --2- 172.21.15.46:0/3121293187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58140ab1c0 0x7f581409c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.870+0000 7f58137fe700 1 --2- 172.21.15.46:0/3121293187 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f581415d8c0 0x7f5814158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:10.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.871+0000 7f58127fc700 1 -- 172.21.15.46:0/3121293187 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f581415d8c0 msgr2=0x7f5814158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:10.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.871+0000 7f58127fc700 1 --2- 172.21.15.46:0/3121293187 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f581415d8c0 0x7f5814158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:10.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.871+0000 7f58127fc700 1 -- 172.21.15.46:0/3121293187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58140ab1c0 msgr2=0x7f581409c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:10.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.871+0000 7f58127fc700 1 --2- 172.21.15.46:0/3121293187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58140ab1c0 0x7f581409c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:10.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.871+0000 7f58127fc700 1 -- 172.21.15.46:0/3121293187 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580c009a20 con 0x7f58141599d0 2024-01-26T23:45:10.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.871+0000 7f58127fc700 1 --2- 172.21.15.46:0/3121293187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58141599d0 0x7f58140aef40 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f57fc00c810 tx=0x7f57fc007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:10.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.871+0000 7f580affd700 1 -- 172.21.15.46:0/3121293187 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc005b30 con 0x7f58141599d0 2024-01-26T23:45:10.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.871+0000 7f580affd700 1 -- 172.21.15.46:0/3121293187 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57fc005c90 con 0x7f58141599d0 2024-01-26T23:45:10.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.871+0000 7f5819999700 1 -- 172.21.15.46:0/3121293187 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f581409a790 con 0x7f58141599d0 2024-01-26T23:45:10.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.871+0000 7f580affd700 1 -- 172.21.15.46:0/3121293187 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc007300 con 0x7f58141599d0 2024-01-26T23:45:10.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.871+0000 7f5819999700 1 -- 172.21.15.46:0/3121293187 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58140af480 con 0x7f58141599d0 2024-01-26T23:45:10.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.872+0000 7f5819999700 1 -- 172.21.15.46:0/3121293187 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581409a310 con 0x7f58141599d0 2024-01-26T23:45:10.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.873+0000 7f580affd700 1 -- 172.21.15.46:0/3121293187 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 29) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f57fc01b3b0 con 0x7f58141599d0 2024-01-26T23:45:10.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.873+0000 7f580affd700 1 --2- 172.21.15.46:0/3121293187 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57f4066ea0 0x7f57f4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:10.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.873+0000 7f580affd700 1 -- 172.21.15.46:0/3121293187 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6167+0+0 (secure 0 0 0) 0x7f57fc083950 con 0x7f58141599d0 2024-01-26T23:45:10.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.873+0000 7f5812ffd700 1 --2- 172.21.15.46:0/3121293187 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57f4066ea0 0x7f57f4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.874+0000 7f5812ffd700 1 --2- 172.21.15.46:0/3121293187 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57f4066ea0 0x7f57f4069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5804009da0 tx=0x7f5804009120 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:10.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:10.878+0000 7f580affd700 1 -- 172.21.15.46:0/3121293187 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f581409a310 con 0x7f58141599d0 2024-01-26T23:45:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.094+0000 7f5819999700 1 -- 172.21.15.46:0/3121293187 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f581409a310 con 0x7f58141599d0 2024-01-26T23:45:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.098+0000 7f580affd700 1 -- 172.21.15.46:0/3121293187 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f581409a310 con 0x7f58141599d0 2024-01-26T23:45:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.100+0000 7f5819999700 1 -- 172.21.15.46:0/3121293187 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57f4066ea0 msgr2=0x7f57f4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.100+0000 7f5819999700 1 --2- 172.21.15.46:0/3121293187 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57f4066ea0 0x7f57f4069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5804009da0 tx=0x7f5804009120 comp rx=0 tx=0).stop 2024-01-26T23:45:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.100+0000 7f5819999700 1 -- 172.21.15.46:0/3121293187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58141599d0 msgr2=0x7f58140aef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.100+0000 7f5819999700 1 --2- 172.21.15.46:0/3121293187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58141599d0 0x7f58140aef40 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f57fc00c810 tx=0x7f57fc007c50 comp rx=0 tx=0).stop 2024-01-26T23:45:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.100+0000 7f5819999700 1 -- 172.21.15.46:0/3121293187 shutdown_connections 2024-01-26T23:45:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.100+0000 7f5819999700 1 --2- 172.21.15.46:0/3121293187 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57f4066ea0 0x7f57f4069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.100+0000 7f5819999700 1 --2- 172.21.15.46:0/3121293187 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f581415d8c0 0x7f5814158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.100+0000 7f5819999700 1 --2- 172.21.15.46:0/3121293187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58140ab1c0 0x7f581409c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.100+0000 7f5819999700 1 --2- 172.21.15.46:0/3121293187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58141599d0 0x7f58140aef40 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.100+0000 7f5819999700 1 -- 172.21.15.46:0/3121293187 >> 172.21.15.46:0/3121293187 conn(0x7f581406f840 msgr2=0x7f58140b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:11.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.101+0000 7f5819999700 1 -- 172.21.15.46:0/3121293187 shutdown_connections 2024-01-26T23:45:11.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.101+0000 7f5819999700 1 -- 172.21.15.46:0/3121293187 wait complete. 2024-01-26T23:45:11.115 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-26T23:45:11.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.248+0000 7fe63ed6c700 1 Processor -- start 2024-01-26T23:45:11.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.248+0000 7fe63ed6c700 1 -- start start 2024-01-26T23:45:11.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.249+0000 7fe63ed6c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe638156570 0x7fe63806c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:11.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.249+0000 7fe63ed6c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe638156940 0x7fe63806f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:11.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.249+0000 7fe63ed6c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6380c8b90 con 0x7fe6380c4530 2024-01-26T23:45:11.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.249+0000 7fe63ed6c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe63806d0b0 con 0x7fe638156940 2024-01-26T23:45:11.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.249+0000 7fe63ed6c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6380c8e50 con 0x7fe638156570 2024-01-26T23:45:11.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.250+0000 7fe637fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe638156940 0x7fe63806f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:11.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.250+0000 7fe637fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe638156940 0x7fe63806f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44530/0 (socket says 172.21.15.46:44530) 2024-01-26T23:45:11.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.250+0000 7fe63d309700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe6380c4530 0x7fe63806eb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49024/0 (socket says 172.21.15.46:49024) 2024-01-26T23:45:11.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.250+0000 7fe637fff700 1 -- 172.21.15.46:0/1047962061 learned_addr learned my addr 172.21.15.46:0/1047962061 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:11.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.250+0000 7fe63cb08700 1 --2- 172.21.15.46:0/1047962061 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe638156570 0x7fe63806c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:11.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.250+0000 7fe637fff700 1 -- 172.21.15.46:0/1047962061 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe638156570 msgr2=0x7fe63806c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:11.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.250+0000 7fe637fff700 1 --2- 172.21.15.46:0/1047962061 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe638156570 0x7fe63806c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.250+0000 7fe637fff700 1 -- 172.21.15.46:0/1047962061 >> v1:172.21.15.46:6789/0 conn(0x7fe6380c4530 legacy=0x7fe63806eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.250+0000 7fe637fff700 1 -- 172.21.15.46:0/1047962061 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63806c810 con 0x7fe638156940 2024-01-26T23:45:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.250+0000 7fe6377fe700 1 -- 172.21.15.46:0/1047962061 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416073929 0 0) 0x7fe6380c8b90 con 0x7fe6380c4530 2024-01-26T23:45:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.250+0000 7fe637fff700 1 --2- 172.21.15.46:0/1047962061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe638156940 0x7fe63806f9a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fe6240099e0 tx=0x7fe624009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7180f487cc9f194 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.250+0000 7fe6377fe700 1 -- 172.21.15.46:0/1047962061 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62400b820 con 0x7fe638156940 2024-01-26T23:45:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.250+0000 7fe6377fe700 1 -- 172.21.15.46:0/1047962061 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62400b980 con 0x7fe638156940 2024-01-26T23:45:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.250+0000 7fe6377fe700 1 -- 172.21.15.46:0/1047962061 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62400bd20 con 0x7fe638156940 2024-01-26T23:45:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.251+0000 7fe63ed6c700 1 -- 172.21.15.46:0/1047962061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe638156940 msgr2=0x7fe63806f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.251+0000 7fe63ed6c700 1 --2- 172.21.15.46:0/1047962061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe638156940 0x7fe63806f9a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fe6240099e0 tx=0x7fe624009250 comp rx=0 tx=0).stop 2024-01-26T23:45:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.251+0000 7fe63ed6c700 1 -- 172.21.15.46:0/1047962061 shutdown_connections 2024-01-26T23:45:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.251+0000 7fe63ed6c700 1 --2- 172.21.15.46:0/1047962061 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe638156570 0x7fe63806c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.251+0000 7fe63ed6c700 1 --2- 172.21.15.46:0/1047962061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe638156940 0x7fe63806f9a0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.251+0000 7fe63ed6c700 1 -- 172.21.15.46:0/1047962061 >> 172.21.15.46:0/1047962061 conn(0x7fe6380cdf50 msgr2=0x7fe6380c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.251+0000 7fe63ed6c700 1 -- 172.21.15.46:0/1047962061 shutdown_connections 2024-01-26T23:45:11.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.251+0000 7fe63ed6c700 1 -- 172.21.15.46:0/1047962061 wait complete. 2024-01-26T23:45:11.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.253+0000 7fe63ed6c700 1 Processor -- start 2024-01-26T23:45:11.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.253+0000 7fe63ed6c700 1 -- start start 2024-01-26T23:45:11.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe63ed6c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6380c4530 0x7fe63814f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:11.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe63ed6c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe638156570 0x7fe6380cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:11.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe63ed6c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe638156940 0x7fe638148160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:11.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe63ed6c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6380ca500 con 0x7fe638156940 2024-01-26T23:45:11.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe63ed6c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6380aa180 con 0x7fe6380c4530 2024-01-26T23:45:11.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe63ed6c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6381486a0 con 0x7fe638156570 2024-01-26T23:45:11.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe63d309700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe638156940 0x7fe638148160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:11.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe63d309700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe638156940 0x7fe638148160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44540/0 (socket says 172.21.15.46:44540) 2024-01-26T23:45:11.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe63d309700 1 -- 172.21.15.46:0/2632297578 learned_addr learned my addr 172.21.15.46:0/2632297578 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:11.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe637fff700 1 --2- 172.21.15.46:0/2632297578 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe638156570 0x7fe6380cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:11.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe63cb08700 1 --2- 172.21.15.46:0/2632297578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6380c4530 0x7fe63814f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:11.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe63d309700 1 -- 172.21.15.46:0/2632297578 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe638156570 msgr2=0x7fe6380cb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:11.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe63d309700 1 --2- 172.21.15.46:0/2632297578 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe638156570 0x7fe6380cb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe63d309700 1 -- 172.21.15.46:0/2632297578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6380c4530 msgr2=0x7fe63814f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:11.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe63d309700 1 --2- 172.21.15.46:0/2632297578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6380c4530 0x7fe63814f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe63d309700 1 -- 172.21.15.46:0/2632297578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6240094f0 con 0x7fe638156940 2024-01-26T23:45:11.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe63d309700 1 --2- 172.21.15.46:0/2632297578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe638156940 0x7fe638148160 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe63000cd50 tx=0x7fe630001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:11.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe634ff9700 1 -- 172.21.15.46:0/2632297578 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630005df0 con 0x7fe638156940 2024-01-26T23:45:11.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.254+0000 7fe634ff9700 1 -- 172.21.15.46:0/2632297578 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63000a960 con 0x7fe638156940 2024-01-26T23:45:11.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.255+0000 7fe63ed6c700 1 -- 172.21.15.46:0/2632297578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6380cbfb0 con 0x7fe638156940 2024-01-26T23:45:11.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.255+0000 7fe634ff9700 1 -- 172.21.15.46:0/2632297578 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63000ad20 con 0x7fe638156940 2024-01-26T23:45:11.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.255+0000 7fe63ed6c700 1 -- 172.21.15.46:0/2632297578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6380c9f40 con 0x7fe638156940 2024-01-26T23:45:11.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.256+0000 7fe63ed6c700 1 -- 172.21.15.46:0/2632297578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6380aa2f0 con 0x7fe638156940 2024-01-26T23:45:11.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.256+0000 7fe634ff9700 1 -- 172.21.15.46:0/2632297578 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 29) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fe630007300 con 0x7fe638156940 2024-01-26T23:45:11.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.256+0000 7fe634ff9700 1 --2- 172.21.15.46:0/2632297578 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe61c066ea0 0x7fe61c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:11.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.257+0000 7fe634ff9700 1 -- 172.21.15.46:0/2632297578 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6167+0+0 (secure 0 0 0) 0x7fe630084800 con 0x7fe638156940 2024-01-26T23:45:11.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.257+0000 7fe63cb08700 1 --2- 172.21.15.46:0/2632297578 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe61c066ea0 0x7fe61c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:11.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.257+0000 7fe63cb08700 1 --2- 172.21.15.46:0/2632297578 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe61c066ea0 0x7fe61c069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe638153790 tx=0x7fe62c008040 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:11.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.261+0000 7fe634ff9700 1 -- 172.21.15.46:0/2632297578 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe630051960 con 0x7fe638156940 2024-01-26T23:45:11.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.468+0000 7fe63ed6c700 1 -- 172.21.15.46:0/2632297578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fe63804c710 con 0x7fe638156940 2024-01-26T23:45:11.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.472+0000 7fe634ff9700 1 -- 172.21.15.46:0/2632297578 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v20) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe630054f80 con 0x7fe638156940 2024-01-26T23:45:11.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.474+0000 7fe63ed6c700 1 -- 172.21.15.46:0/2632297578 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe61c066ea0 msgr2=0x7fe61c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:11.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.474+0000 7fe63ed6c700 1 --2- 172.21.15.46:0/2632297578 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe61c066ea0 0x7fe61c069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe638153790 tx=0x7fe62c008040 comp rx=0 tx=0).stop 2024-01-26T23:45:11.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.474+0000 7fe63ed6c700 1 -- 172.21.15.46:0/2632297578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe638156940 msgr2=0x7fe638148160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:11.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.474+0000 7fe63ed6c700 1 --2- 172.21.15.46:0/2632297578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe638156940 0x7fe638148160 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe63000cd50 tx=0x7fe630001b10 comp rx=0 tx=0).stop 2024-01-26T23:45:11.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.474+0000 7fe63ed6c700 1 -- 172.21.15.46:0/2632297578 shutdown_connections 2024-01-26T23:45:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.474+0000 7fe63ed6c700 1 --2- 172.21.15.46:0/2632297578 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe61c066ea0 0x7fe61c069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.474+0000 7fe63ed6c700 1 --2- 172.21.15.46:0/2632297578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe638156940 0x7fe638148160 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.474+0000 7fe63ed6c700 1 --2- 172.21.15.46:0/2632297578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6380c4530 0x7fe63814f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.474+0000 7fe63ed6c700 1 --2- 172.21.15.46:0/2632297578 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe638156570 0x7fe6380cb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.475+0000 7fe63ed6c700 1 -- 172.21.15.46:0/2632297578 >> 172.21.15.46:0/2632297578 conn(0x7fe6380cdf50 msgr2=0x7fe638153d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.475+0000 7fe63ed6c700 1 -- 172.21.15.46:0/2632297578 shutdown_connections 2024-01-26T23:45:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.475+0000 7fe63ed6c700 1 -- 172.21.15.46:0/2632297578 wait complete. 2024-01-26T23:45:11.489 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-26T23:45:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.623+0000 7fb2de270700 1 Processor -- start 2024-01-26T23:45:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.623+0000 7fb2de270700 1 -- start start 2024-01-26T23:45:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.624+0000 7fb2de270700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d80c8da0 0x7fb2d80c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.624+0000 7fb2de270700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2d80caf30 0x7fb2d80cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.624+0000 7fb2de270700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d80ccc00 con 0x7fb2d8140af0 2024-01-26T23:45:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.624+0000 7fb2de270700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d80cbbf0 con 0x7fb2d80caf30 2024-01-26T23:45:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.624+0000 7fb2de270700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d80cc310 con 0x7fb2d80c8da0 2024-01-26T23:45:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.624+0000 7fb2dc80d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2d80caf30 0x7fb2d80cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.624+0000 7fb2dc80d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2d80caf30 0x7fb2d80cb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44552/0 (socket says 172.21.15.46:44552) 2024-01-26T23:45:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.624+0000 7fb2dc80d700 1 -- 172.21.15.46:0/638342733 learned_addr learned my addr 172.21.15.46:0/638342733 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.624+0000 7fb2d7fff700 1 --2- 172.21.15.46:0/638342733 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d80c8da0 0x7fb2d80c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.624+0000 7fb2dc80d700 1 -- 172.21.15.46:0/638342733 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d80c8da0 msgr2=0x7fb2d80c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.624+0000 7fb2dc80d700 1 --2- 172.21.15.46:0/638342733 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d80c8da0 0x7fb2d80c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.624+0000 7fb2dc80d700 1 -- 172.21.15.46:0/638342733 >> v1:172.21.15.106:6789/0 conn(0x7fb2d8140af0 legacy=0x7fb2d8140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.624+0000 7fb2dc80d700 1 -- 172.21.15.46:0/638342733 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d80cc5b0 con 0x7fb2d80caf30 2024-01-26T23:45:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.624+0000 7fb2dc80d700 1 --2- 172.21.15.46:0/638342733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d80caf30 0x7fb2d80cb370 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009f20 tx=0x7fb2d0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4edb187968dc971b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.625+0000 7fb2d6ffd700 1 -- 172.21.15.46:0/638342733 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d000e070 con 0x7fb2d80caf30 2024-01-26T23:45:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.625+0000 7fb2d6ffd700 1 -- 172.21.15.46:0/638342733 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d000b9d0 con 0x7fb2d80caf30 2024-01-26T23:45:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.625+0000 7fb2d6ffd700 1 -- 172.21.15.46:0/638342733 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d00092e0 con 0x7fb2d80caf30 2024-01-26T23:45:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.625+0000 7fb2de270700 1 -- 172.21.15.46:0/638342733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d80caf30 msgr2=0x7fb2d80cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.625+0000 7fb2de270700 1 --2- 172.21.15.46:0/638342733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d80caf30 0x7fb2d80cb370 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009f20 tx=0x7fb2d0009840 comp rx=0 tx=0).stop 2024-01-26T23:45:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.625+0000 7fb2de270700 1 -- 172.21.15.46:0/638342733 shutdown_connections 2024-01-26T23:45:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.625+0000 7fb2de270700 1 --2- 172.21.15.46:0/638342733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d80caf30 0x7fb2d80cb370 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.625+0000 7fb2de270700 1 --2- 172.21.15.46:0/638342733 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d80c8da0 0x7fb2d80c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.625+0000 7fb2de270700 1 -- 172.21.15.46:0/638342733 >> 172.21.15.46:0/638342733 conn(0x7fb2d806bd90 msgr2=0x7fb2d806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.626+0000 7fb2de270700 1 -- 172.21.15.46:0/638342733 shutdown_connections 2024-01-26T23:45:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.626+0000 7fb2de270700 1 -- 172.21.15.46:0/638342733 wait complete. 2024-01-26T23:45:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.627+0000 7fb2de270700 1 Processor -- start 2024-01-26T23:45:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.627+0000 7fb2de270700 1 -- start start 2024-01-26T23:45:11.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.628+0000 7fb2de270700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2d80c8da0 0x7fb2d806a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:11.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.628+0000 7fb2de270700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d80caf30 0x7fb2d8068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:11.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.628+0000 7fb2de270700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d8140af0 0x7fb2d816a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:11.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.628+0000 7fb2de270700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d816acc0 con 0x7fb2d80caf30 2024-01-26T23:45:11.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.628+0000 7fb2de270700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d8069ce0 con 0x7fb2d80c8da0 2024-01-26T23:45:11.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.628+0000 7fb2de270700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d80690d0 con 0x7fb2d8140af0 2024-01-26T23:45:11.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.628+0000 7fb2d77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d80caf30 0x7fb2d8068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:11.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.628+0000 7fb2d77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d80caf30 0x7fb2d8068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44554/0 (socket says 172.21.15.46:44554) 2024-01-26T23:45:11.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.628+0000 7fb2d77fe700 1 -- 172.21.15.46:0/2421641133 learned_addr learned my addr 172.21.15.46:0/2421641133 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:11.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.628+0000 7fb2d7fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2d80c8da0 0x7fb2d806a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:11.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.628+0000 7fb2dc80d700 1 --2- 172.21.15.46:0/2421641133 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d8140af0 0x7fb2d816a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:11.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.628+0000 7fb2d77fe700 1 -- 172.21.15.46:0/2421641133 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d8140af0 msgr2=0x7fb2d816a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.628+0000 7fb2d77fe700 1 --2- 172.21.15.46:0/2421641133 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d8140af0 0x7fb2d816a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.628+0000 7fb2d77fe700 1 -- 172.21.15.46:0/2421641133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2d80c8da0 msgr2=0x7fb2d806a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.628+0000 7fb2d77fe700 1 --2- 172.21.15.46:0/2421641133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2d80c8da0 0x7fb2d806a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.628+0000 7fb2d77fe700 1 -- 172.21.15.46:0/2421641133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d0009ae0 con 0x7fb2d80caf30 2024-01-26T23:45:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.629+0000 7fb2d77fe700 1 --2- 172.21.15.46:0/2421641133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d80caf30 0x7fb2d8068a40 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00c4e0 tx=0x7fb2cc00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.629+0000 7fb2bffff700 1 -- 172.21.15.46:0/2421641133 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc011590 con 0x7fb2d80caf30 2024-01-26T23:45:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.629+0000 7fb2bffff700 1 -- 172.21.15.46:0/2421641133 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2cc011710 con 0x7fb2d80caf30 2024-01-26T23:45:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.629+0000 7fb2bffff700 1 -- 172.21.15.46:0/2421641133 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc011ad0 con 0x7fb2d80caf30 2024-01-26T23:45:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.629+0000 7fb2de270700 1 -- 172.21.15.46:0/2421641133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d80693b0 con 0x7fb2d80caf30 2024-01-26T23:45:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.629+0000 7fb2de270700 1 -- 172.21.15.46:0/2421641133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d8069970 con 0x7fb2d80caf30 2024-01-26T23:45:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.630+0000 7fb2de270700 1 -- 172.21.15.46:0/2421641133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d804c710 con 0x7fb2d80caf30 2024-01-26T23:45:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.630+0000 7fb2bffff700 1 -- 172.21.15.46:0/2421641133 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 29) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fb2cc01b3b0 con 0x7fb2d80caf30 2024-01-26T23:45:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.631+0000 7fb2bffff700 1 --2- 172.21.15.46:0/2421641133 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb2c4066ea0 0x7fb2c4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:11.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.631+0000 7fb2bffff700 1 -- 172.21.15.46:0/2421641133 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6167+0+0 (secure 0 0 0) 0x7fb2cc0839c0 con 0x7fb2d80caf30 2024-01-26T23:45:11.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.631+0000 7fb2d7fff700 1 --2- 172.21.15.46:0/2421641133 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb2c4066ea0 0x7fb2c4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:11.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.631+0000 7fb2d7fff700 1 --2- 172.21.15.46:0/2421641133 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb2c4066ea0 0x7fb2c4069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009db0 tx=0x7fb2c80091a0 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:11.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.635+0000 7fb2bffff700 1 -- 172.21.15.46:0/2421641133 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb2cc050aa0 con 0x7fb2d80caf30 2024-01-26T23:45:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.842+0000 7fb2de270700 1 -- 172.21.15.46:0/2421641133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fb2d80cc890 con 0x7fb2d80caf30 2024-01-26T23:45:11.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.846+0000 7fb2bffff700 1 -- 172.21.15.46:0/2421641133 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v21) v1 ==== 115+0+0 (secure 0 0 0) 0x7fb2cc017020 con 0x7fb2d80caf30 2024-01-26T23:45:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.849+0000 7fb2de270700 1 -- 172.21.15.46:0/2421641133 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb2c4066ea0 msgr2=0x7fb2c4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.849+0000 7fb2de270700 1 --2- 172.21.15.46:0/2421641133 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb2c4066ea0 0x7fb2c4069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009db0 tx=0x7fb2c80091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.849+0000 7fb2de270700 1 -- 172.21.15.46:0/2421641133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d80caf30 msgr2=0x7fb2d8068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.849+0000 7fb2de270700 1 --2- 172.21.15.46:0/2421641133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d80caf30 0x7fb2d8068a40 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00c4e0 tx=0x7fb2cc00c7f0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.849+0000 7fb2de270700 1 -- 172.21.15.46:0/2421641133 shutdown_connections 2024-01-26T23:45:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.849+0000 7fb2de270700 1 --2- 172.21.15.46:0/2421641133 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb2c4066ea0 0x7fb2c4069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.849+0000 7fb2de270700 1 --2- 172.21.15.46:0/2421641133 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d8140af0 0x7fb2d816a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.849+0000 7fb2de270700 1 --2- 172.21.15.46:0/2421641133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2d80c8da0 0x7fb2d806a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.849+0000 7fb2de270700 1 --2- 172.21.15.46:0/2421641133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d80caf30 0x7fb2d8068a40 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.849+0000 7fb2de270700 1 -- 172.21.15.46:0/2421641133 >> 172.21.15.46:0/2421641133 conn(0x7fb2d806bd90 msgr2=0x7fb2d80b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.849+0000 7fb2de270700 1 -- 172.21.15.46:0/2421641133 shutdown_connections 2024-01-26T23:45:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.849+0000 7fb2de270700 1 -- 172.21.15.46:0/2421641133 wait complete. 2024-01-26T23:45:11.864 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-01-26T23:45:11.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.994+0000 7fb94d88b700 1 Processor -- start 2024-01-26T23:45:11.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.995+0000 7fb94d88b700 1 -- start start 2024-01-26T23:45:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.995+0000 7fb94d88b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb948157340 0x7fb94815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.995+0000 7fb94d88b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb948158ae0 con 0x7fb9480b4180 2024-01-26T23:45:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.995+0000 7fb94d88b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94815b650 con 0x7fb9480ae520 2024-01-26T23:45:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.995+0000 7fb94d88b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb94815c020 con 0x7fb948157340 2024-01-26T23:45:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.996+0000 7fb9477fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb9480b4180 0x7fb948157710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49036/0 (socket says 172.21.15.46:49036) 2024-01-26T23:45:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.996+0000 7fb9477fe700 1 -- 172.21.15.46:0/3024575897 learned_addr learned my addr 172.21.15.46:0/3024575897 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.996+0000 7fb946ffd700 1 --2- 172.21.15.46:0/3024575897 >> v2:172.21.15.106:3300/0 conn(0x7fb948157340 0x7fb94815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.996+0000 7fb945ffb700 1 -- 172.21.15.46:0/3024575897 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350320271 0 0) 0x7fb948158ae0 con 0x7fb9480b4180 2024-01-26T23:45:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.996+0000 7fb945ffb700 1 -- 172.21.15.46:0/3024575897 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb930003570 con 0x7fb9480b4180 2024-01-26T23:45:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.996+0000 7fb946ffd700 1 -- 172.21.15.46:0/3024575897 >> v1:172.21.15.106:6789/0 conn(0x7fb9480ae520 legacy=0x7fb9480b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:11.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.996+0000 7fb946ffd700 1 -- 172.21.15.46:0/3024575897 >> v1:172.21.15.46:6789/0 conn(0x7fb9480b4180 legacy=0x7fb948157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:11.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.996+0000 7fb945ffb700 1 -- 172.21.15.46:0/3024575897 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412499700 0 0) 0x7fb94815b650 con 0x7fb9480ae520 2024-01-26T23:45:11.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.996+0000 7fb946ffd700 1 -- 172.21.15.46:0/3024575897 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb948158520 con 0x7fb948157340 2024-01-26T23:45:11.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.996+0000 7fb946ffd700 1 --2- 172.21.15.46:0/3024575897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb948157340 0x7fb94815bae0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb9380098e0 tx=0x7fb9380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6373ed7f1237a89 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:11.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.997+0000 7fb945ffb700 1 -- 172.21.15.46:0/3024575897 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93800b640 con 0x7fb948157340 2024-01-26T23:45:11.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.997+0000 7fb945ffb700 1 -- 172.21.15.46:0/3024575897 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93800b7a0 con 0x7fb948157340 2024-01-26T23:45:11.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.997+0000 7fb945ffb700 1 -- 172.21.15.46:0/3024575897 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93800bb40 con 0x7fb948157340 2024-01-26T23:45:12.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.997+0000 7fb94d88b700 1 -- 172.21.15.46:0/3024575897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb948157340 msgr2=0x7fb94815bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:12.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.997+0000 7fb94d88b700 1 --2- 172.21.15.46:0/3024575897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb948157340 0x7fb94815bae0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb9380098e0 tx=0x7fb9380091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:12.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.997+0000 7fb94d88b700 1 -- 172.21.15.46:0/3024575897 shutdown_connections 2024-01-26T23:45:12.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.997+0000 7fb94d88b700 1 --2- 172.21.15.46:0/3024575897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb948157340 0x7fb94815bae0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:12.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.997+0000 7fb94d88b700 1 -- 172.21.15.46:0/3024575897 >> 172.21.15.46:0/3024575897 conn(0x7fb9480ce530 msgr2=0x7fb9480ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:12.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.997+0000 7fb94d88b700 1 -- 172.21.15.46:0/3024575897 shutdown_connections 2024-01-26T23:45:12.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.997+0000 7fb94d88b700 1 -- 172.21.15.46:0/3024575897 wait complete. 2024-01-26T23:45:12.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.999+0000 7fb94d88b700 1 Processor -- start 2024-01-26T23:45:12.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.999+0000 7fb94d88b700 1 -- start start 2024-01-26T23:45:12.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.999+0000 7fb94d88b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9480ae520 0x7fb9480c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:12.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.999+0000 7fb94d88b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9480b4180 0x7fb9480c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:12.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.999+0000 7fb94d88b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb948157340 0x7fb9480ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:12.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.999+0000 7fb94d88b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9480c83c0 con 0x7fb9480ae520 2024-01-26T23:45:12.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:11.999+0000 7fb94d88b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9480c2850 con 0x7fb948157340 2024-01-26T23:45:12.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb94d88b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9480c4800 con 0x7fb9480b4180 2024-01-26T23:45:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb946ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9480ae520 0x7fb9480c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb946ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9480ae520 0x7fb9480c1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44564/0 (socket says 172.21.15.46:44564) 2024-01-26T23:45:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb946ffd700 1 -- 172.21.15.46:0/1813245188 learned_addr learned my addr 172.21.15.46:0/1813245188 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb9467fc700 1 --2- 172.21.15.46:0/1813245188 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9480b4180 0x7fb9480c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb9477fe700 1 --2- 172.21.15.46:0/1813245188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb948157340 0x7fb9480ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb946ffd700 1 -- 172.21.15.46:0/1813245188 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9480b4180 msgr2=0x7fb9480c9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb946ffd700 1 --2- 172.21.15.46:0/1813245188 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9480b4180 0x7fb9480c9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb946ffd700 1 -- 172.21.15.46:0/1813245188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb948157340 msgr2=0x7fb9480ca3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb946ffd700 1 --2- 172.21.15.46:0/1813245188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb948157340 0x7fb9480ca3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb946ffd700 1 -- 172.21.15.46:0/1813245188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb938009420 con 0x7fb9480ae520 2024-01-26T23:45:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb9467fc700 1 --2- 172.21.15.46:0/1813245188 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9480b4180 0x7fb9480c9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb946ffd700 1 --2- 172.21.15.46:0/1813245188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9480ae520 0x7fb9480c1f90 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb9380093f0 tx=0x7fb938017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb92effd700 1 -- 172.21.15.46:0/1813245188 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93801f070 con 0x7fb9480ae520 2024-01-26T23:45:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb92effd700 1 -- 172.21.15.46:0/1813245188 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb938006af0 con 0x7fb9480ae520 2024-01-26T23:45:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb94d88b700 1 -- 172.21.15.46:0/1813245188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9480c3c70 con 0x7fb9480ae520 2024-01-26T23:45:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb92effd700 1 -- 172.21.15.46:0/1813245188 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93800bb40 con 0x7fb9480ae520 2024-01-26T23:45:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.000+0000 7fb94d88b700 1 -- 172.21.15.46:0/1813245188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9480c41b0 con 0x7fb9480ae520 2024-01-26T23:45:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.002+0000 7fb94d88b700 1 -- 172.21.15.46:0/1813245188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94806bce0 con 0x7fb9480ae520 2024-01-26T23:45:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.002+0000 7fb92effd700 1 -- 172.21.15.46:0/1813245188 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 29) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fb938011630 con 0x7fb9480ae520 2024-01-26T23:45:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.002+0000 7fb92effd700 1 --2- 172.21.15.46:0/1813245188 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb934066ea0 0x7fb934069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:12.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.002+0000 7fb92effd700 1 -- 172.21.15.46:0/1813245188 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6167+0+0 (secure 0 0 0) 0x7fb9380857e0 con 0x7fb9480ae520 2024-01-26T23:45:12.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.003+0000 7fb9467fc700 1 --2- 172.21.15.46:0/1813245188 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb934066ea0 0x7fb934069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:12.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.003+0000 7fb9467fc700 1 --2- 172.21.15.46:0/1813245188 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb934066ea0 0x7fb934069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb93c007fd0 tx=0x7fb93c009040 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:12.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.007+0000 7fb92effd700 1 -- 172.21.15.46:0/1813245188 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb938052a10 con 0x7fb9480ae520 2024-01-26T23:45:12.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.293+0000 7fb94d88b700 1 -- 172.21.15.46:0/1813245188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7fb94804c710 con 0x7fb9480ae520 2024-01-26T23:45:12.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.634+0000 7fb92effd700 1 -- 172.21.15.46:0/1813245188 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb938052de0 con 0x7fb9480ae520 2024-01-26T23:45:12.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.715+0000 7fb92effd700 1 -- 172.21.15.46:0/1813245188 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v558) v1 ==== 154+0+0 (secure 0 0 0) 0x7fb9380564a0 con 0x7fb9480ae520 2024-01-26T23:45:12.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.718+0000 7fb94d88b700 1 -- 172.21.15.46:0/1813245188 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb934066ea0 msgr2=0x7fb934069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:12.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.718+0000 7fb94d88b700 1 --2- 172.21.15.46:0/1813245188 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb934066ea0 0x7fb934069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb93c007fd0 tx=0x7fb93c009040 comp rx=0 tx=0).stop 2024-01-26T23:45:12.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.718+0000 7fb94d88b700 1 -- 172.21.15.46:0/1813245188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9480ae520 msgr2=0x7fb9480c1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:12.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.718+0000 7fb94d88b700 1 --2- 172.21.15.46:0/1813245188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9480ae520 0x7fb9480c1f90 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb9380093f0 tx=0x7fb938017000 comp rx=0 tx=0).stop 2024-01-26T23:45:12.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.719+0000 7fb94d88b700 1 -- 172.21.15.46:0/1813245188 shutdown_connections 2024-01-26T23:45:12.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.719+0000 7fb94d88b700 1 --2- 172.21.15.46:0/1813245188 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb934066ea0 0x7fb934069350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:12.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.719+0000 7fb94d88b700 1 --2- 172.21.15.46:0/1813245188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9480ae520 0x7fb9480c1f90 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:12.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.719+0000 7fb94d88b700 1 --2- 172.21.15.46:0/1813245188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb948157340 0x7fb9480ca3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:12.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.719+0000 7fb94d88b700 1 --2- 172.21.15.46:0/1813245188 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9480b4180 0x7fb9480c9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:12.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.719+0000 7fb94d88b700 1 -- 172.21.15.46:0/1813245188 >> 172.21.15.46:0/1813245188 conn(0x7fb9480ce530 msgr2=0x7fb948156900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:12.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.719+0000 7fb94d88b700 1 -- 172.21.15.46:0/1813245188 shutdown_connections 2024-01-26T23:45:12.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.719+0000 7fb94d88b700 1 -- 172.21.15.46:0/1813245188 wait complete. 2024-01-26T23:45:12.735 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2024-01-26T23:45:12.735 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2024-01-26T23:45:12.736 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-26T23:45:12.736 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-01-26T23:45:12.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.870+0000 7fb588f38700 1 Processor -- start 2024-01-26T23:45:12.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.870+0000 7fb588f38700 1 -- start start 2024-01-26T23:45:12.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.871+0000 7fb588f38700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb584159b30 0x7fb5840b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:12.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.871+0000 7fb588f38700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb5840b10f0 0x7fb5840a9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:12.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.871+0000 7fb588f38700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb58406f940 con 0x7fb584159760 2024-01-26T23:45:12.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.871+0000 7fb588f38700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5840b1570 con 0x7fb5840b10f0 2024-01-26T23:45:12.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.871+0000 7fb588f38700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5840b3230 con 0x7fb584159b30 2024-01-26T23:45:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.871+0000 7fb58259c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb584159760 0x7fb5840b2b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49042/0 (socket says 172.21.15.46:49042) 2024-01-26T23:45:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.871+0000 7fb58259c700 1 -- 172.21.15.46:0/443921233 learned_addr learned my addr 172.21.15.46:0/443921233 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.871+0000 7fb581d9b700 1 --2- 172.21.15.46:0/443921233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb584159b30 0x7fb5840b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.871+0000 7fb582d9d700 1 --2- 172.21.15.46:0/443921233 >> v2:172.21.15.106:3300/0 conn(0x7fb5840b10f0 0x7fb5840a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.872+0000 7fb58159a700 1 -- 172.21.15.46:0/443921233 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176550187 0 0) 0x7fb58406f940 con 0x7fb584159760 2024-01-26T23:45:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.872+0000 7fb58159a700 1 -- 172.21.15.46:0/443921233 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb570003570 con 0x7fb584159760 2024-01-26T23:45:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.872+0000 7fb581d9b700 1 -- 172.21.15.46:0/443921233 >> v2:172.21.15.106:3300/0 conn(0x7fb5840b10f0 msgr2=0x7fb5840a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.872+0000 7fb581d9b700 1 --2- 172.21.15.46:0/443921233 >> v2:172.21.15.106:3300/0 conn(0x7fb5840b10f0 0x7fb5840a9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.872+0000 7fb581d9b700 1 -- 172.21.15.46:0/443921233 >> v1:172.21.15.46:6789/0 conn(0x7fb584159760 legacy=0x7fb5840b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.872+0000 7fb581d9b700 1 -- 172.21.15.46:0/443921233 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58406ed30 con 0x7fb584159b30 2024-01-26T23:45:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.872+0000 7fb582d9d700 1 --2- 172.21.15.46:0/443921233 >> v2:172.21.15.106:3300/0 conn(0x7fb5840b10f0 0x7fb5840a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.872+0000 7fb581d9b700 1 --2- 172.21.15.46:0/443921233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb584159b30 0x7fb5840b0bb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb56c0098e0 tx=0x7fb56c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33e5710c692ccba5 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.872+0000 7fb58159a700 1 -- 172.21.15.46:0/443921233 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c011630 con 0x7fb584159b30 2024-01-26T23:45:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.872+0000 7fb58159a700 1 -- 172.21.15.46:0/443921233 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56c011790 con 0x7fb584159b30 2024-01-26T23:45:12.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.872+0000 7fb58159a700 1 -- 172.21.15.46:0/443921233 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c011b30 con 0x7fb584159b30 2024-01-26T23:45:12.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.873+0000 7fb588f38700 1 -- 172.21.15.46:0/443921233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb584159b30 msgr2=0x7fb5840b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:12.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.873+0000 7fb588f38700 1 --2- 172.21.15.46:0/443921233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb584159b30 0x7fb5840b0bb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb56c0098e0 tx=0x7fb56c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:12.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.873+0000 7fb588f38700 1 -- 172.21.15.46:0/443921233 shutdown_connections 2024-01-26T23:45:12.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.873+0000 7fb588f38700 1 --2- 172.21.15.46:0/443921233 >> v2:172.21.15.106:3300/0 conn(0x7fb5840b10f0 0x7fb5840a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:12.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.873+0000 7fb588f38700 1 --2- 172.21.15.46:0/443921233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb584159b30 0x7fb5840b0bb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:12.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.873+0000 7fb588f38700 1 -- 172.21.15.46:0/443921233 >> 172.21.15.46:0/443921233 conn(0x7fb584069510 msgr2=0x7fb5840aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:12.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.873+0000 7fb588f38700 1 -- 172.21.15.46:0/443921233 shutdown_connections 2024-01-26T23:45:12.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.873+0000 7fb588f38700 1 -- 172.21.15.46:0/443921233 wait complete. 2024-01-26T23:45:12.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.874+0000 7fb588f38700 1 Processor -- start 2024-01-26T23:45:12.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.874+0000 7fb588f38700 1 -- start start 2024-01-26T23:45:12.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.875+0000 7fb588f38700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb5840b10f0 0x7fb58416a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:12.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.875+0000 7fb588f38700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb584159760 0x7fb58415efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:12.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.875+0000 7fb588f38700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb584159b30 0x7fb5841537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:12.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.875+0000 7fb588f38700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb584153ce0 con 0x7fb584159b30 2024-01-26T23:45:12.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.875+0000 7fb588f38700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58415eb30 con 0x7fb584159760 2024-01-26T23:45:12.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.875+0000 7fb588f38700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb584155570 con 0x7fb5840b10f0 2024-01-26T23:45:12.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.875+0000 7fb582d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb584159b30 0x7fb5841537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:12.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.875+0000 7fb582d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb584159b30 0x7fb5841537a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44580/0 (socket says 172.21.15.46:44580) 2024-01-26T23:45:12.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.875+0000 7fb582d9d700 1 -- 172.21.15.46:0/1624677651 learned_addr learned my addr 172.21.15.46:0/1624677651 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:12.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.875+0000 7fb58259c700 1 --2- 172.21.15.46:0/1624677651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb5840b10f0 0x7fb58416a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:12.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.875+0000 7fb581d9b700 1 --2- 172.21.15.46:0/1624677651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb584159760 0x7fb58415efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:12.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.875+0000 7fb582d9d700 1 -- 172.21.15.46:0/1624677651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb5840b10f0 msgr2=0x7fb58416a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:12.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.875+0000 7fb582d9d700 1 --2- 172.21.15.46:0/1624677651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb5840b10f0 0x7fb58416a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:12.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.875+0000 7fb582d9d700 1 -- 172.21.15.46:0/1624677651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb584159760 msgr2=0x7fb58415efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.875+0000 7fb582d9d700 1 --2- 172.21.15.46:0/1624677651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb584159760 0x7fb58415efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.875+0000 7fb582d9d700 1 -- 172.21.15.46:0/1624677651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57c009420 con 0x7fb584159b30 2024-01-26T23:45:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.875+0000 7fb581d9b700 1 --2- 172.21.15.46:0/1624677651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb584159760 0x7fb58415efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.876+0000 7fb582d9d700 1 --2- 172.21.15.46:0/1624677651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb584159b30 0x7fb5841537a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fb57c00ec80 tx=0x7fb57c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.876+0000 7fb57a7fc700 1 -- 172.21.15.46:0/1624677651 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c00c850 con 0x7fb584159b30 2024-01-26T23:45:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.876+0000 7fb57a7fc700 1 -- 172.21.15.46:0/1624677651 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57c00c9b0 con 0x7fb584159b30 2024-01-26T23:45:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.876+0000 7fb588f38700 1 -- 172.21.15.46:0/1624677651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56c009420 con 0x7fb584159b30 2024-01-26T23:45:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.876+0000 7fb57a7fc700 1 -- 172.21.15.46:0/1624677651 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c00cd70 con 0x7fb584159b30 2024-01-26T23:45:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.876+0000 7fb588f38700 1 -- 172.21.15.46:0/1624677651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb58415e4e0 con 0x7fb584159b30 2024-01-26T23:45:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.877+0000 7fb588f38700 1 -- 172.21.15.46:0/1624677651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb58404c710 con 0x7fb584159b30 2024-01-26T23:45:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.877+0000 7fb57a7fc700 1 -- 172.21.15.46:0/1624677651 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb57c013680 con 0x7fb584159b30 2024-01-26T23:45:12.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.878+0000 7fb57a7fc700 1 --2- 172.21.15.46:0/1624677651 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb564066f30 0x7fb5640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:12.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.878+0000 7fb57a7fc700 1 -- 172.21.15.46:0/1624677651 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6167+0+0 (secure 0 0 0) 0x7fb57c084be0 con 0x7fb584159b30 2024-01-26T23:45:12.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.878+0000 7fb58259c700 1 --2- 172.21.15.46:0/1624677651 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb564066f30 0x7fb5640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:12.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.879+0000 7fb58259c700 1 --2- 172.21.15.46:0/1624677651 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb564066f30 0x7fb5640693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb574005c50 tx=0x7fb574009040 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:12.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:12.882+0000 7fb57a7fc700 1 -- 172.21.15.46:0/1624677651 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb57c051be0 con 0x7fb584159b30 2024-01-26T23:45:13.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.174+0000 7fb588f38700 1 -- 172.21.15.46:0/1624677651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7fb58415ec70 con 0x7fb584159b30 2024-01-26T23:45:13.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.719+0000 7fb57a7fc700 1 -- 172.21.15.46:0/1624677651 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v559) v1 ==== 159+0+0 (secure 0 0 0) 0x7fb57c055200 con 0x7fb584159b30 2024-01-26T23:45:13.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.722+0000 7fb588f38700 1 -- 172.21.15.46:0/1624677651 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb564066f30 msgr2=0x7fb5640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:13.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.722+0000 7fb588f38700 1 --2- 172.21.15.46:0/1624677651 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb564066f30 0x7fb5640693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb574005c50 tx=0x7fb574009040 comp rx=0 tx=0).stop 2024-01-26T23:45:13.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.722+0000 7fb588f38700 1 -- 172.21.15.46:0/1624677651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb584159b30 msgr2=0x7fb5841537a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:13.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.722+0000 7fb588f38700 1 --2- 172.21.15.46:0/1624677651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb584159b30 0x7fb5841537a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fb57c00ec80 tx=0x7fb57c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:13.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.722+0000 7fb588f38700 1 -- 172.21.15.46:0/1624677651 shutdown_connections 2024-01-26T23:45:13.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.722+0000 7fb588f38700 1 --2- 172.21.15.46:0/1624677651 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb564066f30 0x7fb5640693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:13.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.722+0000 7fb588f38700 1 --2- 172.21.15.46:0/1624677651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb584159b30 0x7fb5841537a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:13.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.722+0000 7fb588f38700 1 --2- 172.21.15.46:0/1624677651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb5840b10f0 0x7fb58416a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:13.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.722+0000 7fb588f38700 1 --2- 172.21.15.46:0/1624677651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb584159760 0x7fb58415efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:13.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.722+0000 7fb588f38700 1 -- 172.21.15.46:0/1624677651 >> 172.21.15.46:0/1624677651 conn(0x7fb584069510 msgr2=0x7fb5840ad330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:13.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.722+0000 7fb588f38700 1 -- 172.21.15.46:0/1624677651 shutdown_connections 2024-01-26T23:45:13.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.723+0000 7fb588f38700 1 -- 172.21.15.46:0/1624677651 wait complete. 2024-01-26T23:45:13.739 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-26T23:45:13.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.877+0000 7f30a4627700 1 Processor -- start 2024-01-26T23:45:13.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.877+0000 7f30a4627700 1 -- start start 2024-01-26T23:45:13.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.878+0000 7f30a4627700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f309c0c8da0 0x7f309c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:13.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.878+0000 7f30a4627700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f309c140af0 0x7f309c140f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.878+0000 7f30a4627700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f309c0ccc00 con 0x7f309c0caf30 2024-01-26T23:45:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.878+0000 7f30a4627700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309c0cbbf0 con 0x7f309c140af0 2024-01-26T23:45:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.878+0000 7f30a4627700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0cc310 con 0x7f309c0c8da0 2024-01-26T23:45:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.878+0000 7f30a23c3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f309c0c8da0 0x7f309c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.878+0000 7f30a1bc2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f309c140af0 0x7f309c140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.879+0000 7f30a1bc2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f309c140af0 0x7f309c140f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:46492/0 (socket says 172.21.15.46:46492) 2024-01-26T23:45:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.879+0000 7f30a1bc2700 1 -- 172.21.15.46:0/1946038356 learned_addr learned my addr 172.21.15.46:0/1946038356 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.879+0000 7f30a2bc4700 1 --1- 172.21.15.46:0/1946038356 >> v1:172.21.15.106:6789/0 conn(0x7f309c0caf30 0x7f309c0cb370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:53634/0 (socket says 172.21.15.46:53634) 2024-01-26T23:45:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.879+0000 7f30a1bc2700 1 -- 172.21.15.46:0/1946038356 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f309c0c8da0 msgr2=0x7f309c0c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.879+0000 7f30a1bc2700 1 --2- 172.21.15.46:0/1946038356 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f309c0c8da0 0x7f309c0c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.879+0000 7f30a1bc2700 1 -- 172.21.15.46:0/1946038356 >> v1:172.21.15.106:6789/0 conn(0x7f309c0caf30 legacy=0x7f309c0cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.879+0000 7f30a1bc2700 1 -- 172.21.15.46:0/1946038356 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309c0cc5b0 con 0x7f309c140af0 2024-01-26T23:45:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.879+0000 7f30a13c1700 1 -- 172.21.15.46:0/1946038356 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378961529 0 0) 0x7f309c0ccc00 con 0x7f309c0caf30 2024-01-26T23:45:13.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.879+0000 7f30a1bc2700 1 --2- 172.21.15.46:0/1946038356 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f309c140af0 0x7f309c140f60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f30980099b0 tx=0x7f3098009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d592c46dd70ea58 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:13.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.879+0000 7f30a13c1700 1 -- 172.21.15.46:0/1946038356 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3098009ea0 con 0x7f309c140af0 2024-01-26T23:45:13.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.879+0000 7f30a13c1700 1 -- 172.21.15.46:0/1946038356 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309800b860 con 0x7f309c140af0 2024-01-26T23:45:13.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.879+0000 7f30a13c1700 1 -- 172.21.15.46:0/1946038356 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309800bc00 con 0x7f309c140af0 2024-01-26T23:45:13.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.880+0000 7f30a4627700 1 -- 172.21.15.46:0/1946038356 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f309c140af0 msgr2=0x7f309c140f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:13.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.880+0000 7f30a4627700 1 --2- 172.21.15.46:0/1946038356 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f309c140af0 0x7f309c140f60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f30980099b0 tx=0x7f3098009250 comp rx=0 tx=0).stop 2024-01-26T23:45:13.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.880+0000 7f30a4627700 1 -- 172.21.15.46:0/1946038356 shutdown_connections 2024-01-26T23:45:13.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.880+0000 7f30a4627700 1 --2- 172.21.15.46:0/1946038356 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f309c0c8da0 0x7f309c0c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:13.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.880+0000 7f30a4627700 1 --2- 172.21.15.46:0/1946038356 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f309c140af0 0x7f309c140f60 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:13.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.880+0000 7f30a4627700 1 -- 172.21.15.46:0/1946038356 >> 172.21.15.46:0/1946038356 conn(0x7f309c06bd90 msgr2=0x7f309c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:13.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.880+0000 7f30a4627700 1 -- 172.21.15.46:0/1946038356 shutdown_connections 2024-01-26T23:45:13.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.880+0000 7f30a4627700 1 -- 172.21.15.46:0/1946038356 wait complete. 2024-01-26T23:45:13.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.881+0000 7f30a4627700 1 Processor -- start 2024-01-26T23:45:13.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.881+0000 7f30a4627700 1 -- start start 2024-01-26T23:45:13.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.882+0000 7f30a4627700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0c8da0 0x7f309c13c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:13.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.882+0000 7f30a4627700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f309c0caf30 0x7f309c13c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:13.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.882+0000 7f30a4627700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f309c140af0 0x7f309c0c5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:13.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.882+0000 7f30a4627700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0c79c0 con 0x7f309c0c8da0 2024-01-26T23:45:13.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.882+0000 7f30a4627700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0c7ca0 con 0x7f309c140af0 2024-01-26T23:45:13.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.882+0000 7f30a4627700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0c7f80 con 0x7f309c0caf30 2024-01-26T23:45:13.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.883+0000 7f30a23c3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0c8da0 0x7f309c13c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:13.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.883+0000 7f30a23c3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0c8da0 0x7f309c13c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44586/0 (socket says 172.21.15.46:44586) 2024-01-26T23:45:13.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.883+0000 7f30a23c3700 1 -- 172.21.15.46:0/2932384978 learned_addr learned my addr 172.21.15.46:0/2932384978 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:13.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.883+0000 7f30a2bc4700 1 --2- 172.21.15.46:0/2932384978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f309c140af0 0x7f309c0c5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:13.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.883+0000 7f30a1bc2700 1 --2- 172.21.15.46:0/2932384978 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f309c0caf30 0x7f309c13c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:13.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.883+0000 7f30a23c3700 1 -- 172.21.15.46:0/2932384978 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f309c0caf30 msgr2=0x7f309c13c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:13.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.883+0000 7f30a23c3700 1 --2- 172.21.15.46:0/2932384978 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f309c0caf30 0x7f309c13c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:13.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.883+0000 7f30a23c3700 1 -- 172.21.15.46:0/2932384978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f309c140af0 msgr2=0x7f309c0c5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:13.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.883+0000 7f30a23c3700 1 --2- 172.21.15.46:0/2932384978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f309c140af0 0x7f309c0c5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:13.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.883+0000 7f30a23c3700 1 -- 172.21.15.46:0/2932384978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30980094f0 con 0x7f309c0c8da0 2024-01-26T23:45:13.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.883+0000 7f30a2bc4700 1 --2- 172.21.15.46:0/2932384978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f309c140af0 0x7f309c0c5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:13.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.883+0000 7f30a23c3700 1 --2- 172.21.15.46:0/2932384978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0c8da0 0x7f309c13c0a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f309000ec80 tx=0x7f30900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:13.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.883+0000 7f308e7fc700 1 -- 172.21.15.46:0/2932384978 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309000c850 con 0x7f309c0c8da0 2024-01-26T23:45:13.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.883+0000 7f308e7fc700 1 -- 172.21.15.46:0/2932384978 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309000c9b0 con 0x7f309c0c8da0 2024-01-26T23:45:13.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.883+0000 7f30a4627700 1 -- 172.21.15.46:0/2932384978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309c0c8260 con 0x7f309c0c8da0 2024-01-26T23:45:13.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.883+0000 7f308e7fc700 1 -- 172.21.15.46:0/2932384978 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309000cd50 con 0x7f309c0c8da0 2024-01-26T23:45:13.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.883+0000 7f30a4627700 1 -- 172.21.15.46:0/2932384978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309c0b1360 con 0x7f309c0c8da0 2024-01-26T23:45:13.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.884+0000 7f30a4627700 1 -- 172.21.15.46:0/2932384978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309c04c710 con 0x7f309c0c8da0 2024-01-26T23:45:13.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.885+0000 7f308e7fc700 1 -- 172.21.15.46:0/2932384978 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3090013680 con 0x7f309c0c8da0 2024-01-26T23:45:13.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.885+0000 7f308e7fc700 1 --2- 172.21.15.46:0/2932384978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3080066f30 0x7f30800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:13.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.886+0000 7f308e7fc700 1 -- 172.21.15.46:0/2932384978 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6167+0+0 (secure 0 0 0) 0x7f30900849c0 con 0x7f309c0c8da0 2024-01-26T23:45:13.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.886+0000 7f30a1bc2700 1 --2- 172.21.15.46:0/2932384978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3080066f30 0x7f30800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:13.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.886+0000 7f30a1bc2700 1 --2- 172.21.15.46:0/2932384978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3080066f30 0x7f30800693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f30980099b0 tx=0x7f3098017000 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:13.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:13.890+0000 7f308e7fc700 1 -- 172.21.15.46:0/2932384978 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3090051a60 con 0x7f309c0c8da0 2024-01-26T23:45:14.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.096+0000 7f30a4627700 1 -- 172.21.15.46:0/2932384978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f309c1550b0 con 0x7f309c0c8da0 2024-01-26T23:45:14.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.099+0000 7f308e7fc700 1 -- 172.21.15.46:0/2932384978 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3090055080 con 0x7f309c0c8da0 2024-01-26T23:45:14.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.102+0000 7f30a4627700 1 -- 172.21.15.46:0/2932384978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3080066f30 msgr2=0x7f30800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:14.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.102+0000 7f30a4627700 1 --2- 172.21.15.46:0/2932384978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3080066f30 0x7f30800693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f30980099b0 tx=0x7f3098017000 comp rx=0 tx=0).stop 2024-01-26T23:45:14.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.102+0000 7f30a4627700 1 -- 172.21.15.46:0/2932384978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0c8da0 msgr2=0x7f309c13c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:14.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.102+0000 7f30a4627700 1 --2- 172.21.15.46:0/2932384978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0c8da0 0x7f309c13c0a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f309000ec80 tx=0x7f30900091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:14.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.102+0000 7f30a4627700 1 -- 172.21.15.46:0/2932384978 shutdown_connections 2024-01-26T23:45:14.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.102+0000 7f30a4627700 1 --2- 172.21.15.46:0/2932384978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3080066f30 0x7f30800693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:14.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.102+0000 7f30a4627700 1 --2- 172.21.15.46:0/2932384978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0c8da0 0x7f309c13c0a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:14.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.102+0000 7f30a4627700 1 --2- 172.21.15.46:0/2932384978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f309c140af0 0x7f309c0c5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:14.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.102+0000 7f30a4627700 1 --2- 172.21.15.46:0/2932384978 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f309c0caf30 0x7f309c13c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:14.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.102+0000 7f30a4627700 1 -- 172.21.15.46:0/2932384978 >> 172.21.15.46:0/2932384978 conn(0x7f309c06bd90 msgr2=0x7f309c13fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:14.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.102+0000 7f30a4627700 1 -- 172.21.15.46:0/2932384978 shutdown_connections 2024-01-26T23:45:14.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.102+0000 7f30a4627700 1 -- 172.21.15.46:0/2932384978 wait complete. 2024-01-26T23:45:14.116 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-26T23:45:14.117 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:45:14.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:45:14.215 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-26T23:45:14.215 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-26T23:45:14.215 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:45:14.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.353+0000 7fc5c419d700 1 Processor -- start 2024-01-26T23:45:14.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.353+0000 7fc5c419d700 1 -- start start 2024-01-26T23:45:14.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.354+0000 7fc5c419d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5bc0cd250 0x7fc5bc162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:14.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.354+0000 7fc5c419d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5bc0cbf10 con 0x7fc5bc0acf30 2024-01-26T23:45:14.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.354+0000 7fc5c419d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5bc0c4460 con 0x7fc5bc0c3cf0 2024-01-26T23:45:14.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.354+0000 7fc5c419d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc0ada10 con 0x7fc5bc0cd250 2024-01-26T23:45:14.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.354+0000 7fc5c1f39700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc5bc0acf30 0x7fc5bc0ad300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49046/0 (socket says 172.21.15.46:49046) 2024-01-26T23:45:14.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.354+0000 7fc5c1f39700 1 -- 172.21.15.46:0/2036950917 learned_addr learned my addr 172.21.15.46:0/2036950917 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:14.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.354+0000 7fc5c273a700 1 --2- 172.21.15.46:0/2036950917 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5bc0cd250 0x7fc5bc162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:14.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.354+0000 7fc5c0f37700 1 -- 172.21.15.46:0/2036950917 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060036404 0 0) 0x7fc5bc0cbf10 con 0x7fc5bc0acf30 2024-01-26T23:45:14.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.354+0000 7fc5c0f37700 1 -- 172.21.15.46:0/2036950917 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a8003570 con 0x7fc5bc0acf30 2024-01-26T23:45:14.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.354+0000 7fc5c0f37700 1 -- 172.21.15.46:0/2036950917 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045556860 0 0) 0x7fc5bc0c4460 con 0x7fc5bc0c3cf0 2024-01-26T23:45:14.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.354+0000 7fc5c0f37700 1 -- 172.21.15.46:0/2036950917 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5bc0cbf10 con 0x7fc5bc0c3cf0 2024-01-26T23:45:14.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.354+0000 7fc5c0f37700 1 -- 172.21.15.46:0/2036950917 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3106539621 0 0) 0x7fc5a8003570 con 0x7fc5bc0acf30 2024-01-26T23:45:14.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.354+0000 7fc5c0f37700 1 -- 172.21.15.46:0/2036950917 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5bc0cd250 msgr2=0x7fc5bc162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:14.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c0f37700 1 --2- 172.21.15.46:0/2036950917 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5bc0cd250 0x7fc5bc162360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:14.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c0f37700 1 -- 172.21.15.46:0/2036950917 >> v1:172.21.15.106:6789/0 conn(0x7fc5bc0c3cf0 legacy=0x7fc5bc0ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:14.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c0f37700 1 -- 172.21.15.46:0/2036950917 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc0c4740 con 0x7fc5bc0acf30 2024-01-26T23:45:14.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c0f37700 1 -- 172.21.15.46:0/2036950917 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fc5b8001470 con 0x7fc5bc0acf30 2024-01-26T23:45:14.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c273a700 1 --2- 172.21.15.46:0/2036950917 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5bc0cd250 0x7fc5bc162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:14.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c0f37700 1 -- 172.21.15.46:0/2036950917 >> v1:172.21.15.46:6789/0 conn(0x7fc5bc0acf30 legacy=0x7fc5bc0ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:14.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c0f37700 1 --2- 172.21.15.46:0/2036950917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5a8003c20 0x7fc5a8004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:14.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c0f37700 1 --2- 172.21.15.46:0/2036950917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5a8004810 0x7fc5a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:14.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c0f37700 1 --2- 172.21.15.46:0/2036950917 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5bc0cd250 0x7fc5a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:14.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c0f37700 1 -- 172.21.15.46:0/2036950917 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc0ada10 con 0x7fc5a8003c20 2024-01-26T23:45:14.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c0f37700 1 -- 172.21.15.46:0/2036950917 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a800bb10 con 0x7fc5a8004810 2024-01-26T23:45:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c0f37700 1 -- 172.21.15.46:0/2036950917 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a800bcd0 con 0x7fc5bc0cd250 2024-01-26T23:45:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c1f39700 1 --2- 172.21.15.46:0/2036950917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5a8003c20 0x7fc5a8004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c1738700 1 --2- 172.21.15.46:0/2036950917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5a8004810 0x7fc5a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c273a700 1 --2- 172.21.15.46:0/2036950917 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5bc0cd250 0x7fc5a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c1f39700 1 -- 172.21.15.46:0/2036950917 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5bc0cd250 msgr2=0x7fc5a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c1f39700 1 --2- 172.21.15.46:0/2036950917 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5bc0cd250 0x7fc5a80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c1f39700 1 -- 172.21.15.46:0/2036950917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5a8004810 msgr2=0x7fc5a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c1f39700 1 --2- 172.21.15.46:0/2036950917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5a8004810 0x7fc5a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c1f39700 1 -- 172.21.15.46:0/2036950917 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b4009a20 con 0x7fc5a8003c20 2024-01-26T23:45:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c1f39700 1 --2- 172.21.15.46:0/2036950917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5a8003c20 0x7fc5a8004010 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800c2d0 tx=0x7fc5b8001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18d6597addc029af server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.355+0000 7fc5c0f37700 1 -- 172.21.15.46:0/2036950917 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b800cba0 con 0x7fc5a8003c20 2024-01-26T23:45:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.356+0000 7fc5c0f37700 1 -- 172.21.15.46:0/2036950917 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b800cd00 con 0x7fc5a8003c20 2024-01-26T23:45:14.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.356+0000 7fc5c0f37700 1 -- 172.21.15.46:0/2036950917 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8005b70 con 0x7fc5a8003c20 2024-01-26T23:45:14.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.356+0000 7fc5c419d700 1 -- 172.21.15.46:0/2036950917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5a8003c20 msgr2=0x7fc5a8004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:14.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.356+0000 7fc5c419d700 1 --2- 172.21.15.46:0/2036950917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5a8003c20 0x7fc5a8004010 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800c2d0 tx=0x7fc5b8001e10 comp rx=0 tx=0).stop 2024-01-26T23:45:14.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.356+0000 7fc5c273a700 1 -- 172.21.15.46:0/2036950917 reap_dead start 2024-01-26T23:45:14.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.356+0000 7fc5c419d700 1 -- 172.21.15.46:0/2036950917 shutdown_connections 2024-01-26T23:45:14.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.356+0000 7fc5c419d700 1 -- 172.21.15.46:0/2036950917 >> 172.21.15.46:0/2036950917 conn(0x7fc5bc097820 msgr2=0x7fc5bc068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:14.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.356+0000 7fc5c419d700 1 -- 172.21.15.46:0/2036950917 shutdown_connections 2024-01-26T23:45:14.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.356+0000 7fc5c419d700 1 -- 172.21.15.46:0/2036950917 wait complete. 2024-01-26T23:45:14.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.357+0000 7fc5c419d700 1 Processor -- start 2024-01-26T23:45:14.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.357+0000 7fc5c419d700 1 -- start start 2024-01-26T23:45:14.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.358+0000 7fc5c419d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5a8003c20 0x7fc5bc153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:14.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.358+0000 7fc5c419d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0cb550 0x7fc5bc16b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:14.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.358+0000 7fc5c419d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5bc0b0150 0x7fc5bc0b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:14.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.358+0000 7fc5c419d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc16c150 con 0x7fc5bc0cb550 2024-01-26T23:45:14.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.358+0000 7fc5c419d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc0afcc0 con 0x7fc5a8003c20 2024-01-26T23:45:14.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.358+0000 7fc5c419d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc16bb90 con 0x7fc5bc0b0150 2024-01-26T23:45:14.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.358+0000 7fc5c1738700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0cb550 0x7fc5bc16b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:14.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.358+0000 7fc5c1738700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0cb550 0x7fc5bc16b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44592/0 (socket says 172.21.15.46:44592) 2024-01-26T23:45:14.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.358+0000 7fc5c1738700 1 -- 172.21.15.46:0/3009959158 learned_addr learned my addr 172.21.15.46:0/3009959158 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:14.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.358+0000 7fc5c1f39700 1 --2- 172.21.15.46:0/3009959158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5a8003c20 0x7fc5bc153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:14.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.358+0000 7fc5c273a700 1 --2- 172.21.15.46:0/3009959158 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5bc0b0150 0x7fc5bc0b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:14.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.359+0000 7fc5c1738700 1 -- 172.21.15.46:0/3009959158 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5bc0b0150 msgr2=0x7fc5bc0b05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:14.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.359+0000 7fc5c1738700 1 --2- 172.21.15.46:0/3009959158 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5bc0b0150 0x7fc5bc0b05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:14.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.359+0000 7fc5c1738700 1 -- 172.21.15.46:0/3009959158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5a8003c20 msgr2=0x7fc5bc153b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:14.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.359+0000 7fc5c1738700 1 --2- 172.21.15.46:0/3009959158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5a8003c20 0x7fc5bc153b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:14.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.359+0000 7fc5c1738700 1 -- 172.21.15.46:0/3009959158 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc16be70 con 0x7fc5bc0cb550 2024-01-26T23:45:14.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.359+0000 7fc5c1738700 1 --2- 172.21.15.46:0/3009959158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0cb550 0x7fc5bc16b220 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac002ab0 tx=0x7fc5ac007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:14.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.359+0000 7fc5b1ffb700 1 -- 172.21.15.46:0/3009959158 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac006a30 con 0x7fc5bc0cb550 2024-01-26T23:45:14.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.359+0000 7fc5b1ffb700 1 -- 172.21.15.46:0/3009959158 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ac006bb0 con 0x7fc5bc0cb550 2024-01-26T23:45:14.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.359+0000 7fc5c419d700 1 -- 172.21.15.46:0/3009959158 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5bc0af950 con 0x7fc5bc0cb550 2024-01-26T23:45:14.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.359+0000 7fc5b1ffb700 1 -- 172.21.15.46:0/3009959158 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac007050 con 0x7fc5bc0cb550 2024-01-26T23:45:14.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.359+0000 7fc5c419d700 1 -- 172.21.15.46:0/3009959158 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5bc0addc0 con 0x7fc5bc0cb550 2024-01-26T23:45:14.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.360+0000 7fc5c419d700 1 -- 172.21.15.46:0/3009959158 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5bc04c710 con 0x7fc5bc0cb550 2024-01-26T23:45:14.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.361+0000 7fc5b1ffb700 1 -- 172.21.15.46:0/3009959158 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc5ac007440 con 0x7fc5bc0cb550 2024-01-26T23:45:14.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.361+0000 7fc5b1ffb700 1 --2- 172.21.15.46:0/3009959158 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc5a0066f30 0x7fc5a00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:14.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.361+0000 7fc5b1ffb700 1 -- 172.21.15.46:0/3009959158 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6167+0+0 (secure 0 0 0) 0x7fc5ac0848e0 con 0x7fc5bc0cb550 2024-01-26T23:45:14.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.361+0000 7fc5c1f39700 1 --2- 172.21.15.46:0/3009959158 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc5a0066f30 0x7fc5a00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:14.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.362+0000 7fc5c1f39700 1 --2- 172.21.15.46:0/3009959158 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc5a0066f30 0x7fc5a00693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8001e50 tx=0x7fc5b8015000 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:14.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.366+0000 7fc5b1ffb700 1 -- 172.21.15.46:0/3009959158 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc5ac051980 con 0x7fc5bc0cb550 2024-01-26T23:45:14.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.616+0000 7fc5c419d700 1 -- 172.21.15.46:0/3009959158 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5bc0afe00 con 0x7fc5bc0cb550 2024-01-26T23:45:14.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.617+0000 7fc5b1ffb700 1 -- 172.21.15.46:0/3009959158 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+3535 (secure 0 0 0) 0x7fc5ac054fa0 con 0x7fc5bc0cb550 2024-01-26T23:45:14.618 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:14.618 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":29,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3037972557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3037972557},{"type":"v1","addr":"172.21.15.106:6837","nonce":3037972557}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":5701,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/3786017823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3786017823},{"type":"v1","addr":"172.21.15.106:6836","nonce":3786017823}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":5704,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3329385210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3329385210},{"type":"v1","addr":"172.21.15.46:6835","nonce":3329385210}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21},{"gid":6058,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1399703845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1399703845},{"type":"v1","addr":"172.21.15.46:6833","nonce":1399703845}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[]} 2024-01-26T23:45:14.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.619+0000 7fc5c419d700 1 -- 172.21.15.46:0/3009959158 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc5a0066f30 msgr2=0x7fc5a00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:14.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.619+0000 7fc5c419d700 1 --2- 172.21.15.46:0/3009959158 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc5a0066f30 0x7fc5a00693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8001e50 tx=0x7fc5b8015000 comp rx=0 tx=0).stop 2024-01-26T23:45:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.619+0000 7fc5c419d700 1 -- 172.21.15.46:0/3009959158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0cb550 msgr2=0x7fc5bc16b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.619+0000 7fc5c419d700 1 --2- 172.21.15.46:0/3009959158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0cb550 0x7fc5bc16b220 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac002ab0 tx=0x7fc5ac007d30 comp rx=0 tx=0).stop 2024-01-26T23:45:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.619+0000 7fc5c419d700 1 -- 172.21.15.46:0/3009959158 shutdown_connections 2024-01-26T23:45:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.619+0000 7fc5c419d700 1 --2- 172.21.15.46:0/3009959158 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc5a0066f30 0x7fc5a00693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.619+0000 7fc5c419d700 1 --2- 172.21.15.46:0/3009959158 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5bc0b0150 0x7fc5bc0b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.619+0000 7fc5c419d700 1 --2- 172.21.15.46:0/3009959158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5a8003c20 0x7fc5bc153b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.619+0000 7fc5c419d700 1 --2- 172.21.15.46:0/3009959158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0cb550 0x7fc5bc16b220 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.619+0000 7fc5c419d700 1 -- 172.21.15.46:0/3009959158 >> 172.21.15.46:0/3009959158 conn(0x7fc5bc097820 msgr2=0x7fc5bc06bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.619+0000 7fc5c419d700 1 -- 172.21.15.46:0/3009959158 shutdown_connections 2024-01-26T23:45:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.619+0000 7fc5c419d700 1 -- 172.21.15.46:0/3009959158 wait complete. 2024-01-26T23:45:14.622 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 29 2024-01-26T23:45:14.635 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-26T23:45:14.635 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-26T23:45:14.635 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-26T23:45:14.635 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:45:14.635 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-26T23:45:14.635 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-26T23:45:14.636 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-26T23:45:14.636 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:45:14.636 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-26T23:45:14.636 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-26T23:45:14.636 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-26T23:45:14.636 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:45:14.636 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-26T23:45:14.637 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-26T23:45:14.637 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-26T23:45:14.637 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:45:14.645 INFO:tasks.ceph.mds.a:Stopped 2024-01-26T23:45:14.645 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-26T23:45:14.646 INFO:tasks.ceph.mds.c:Stopped 2024-01-26T23:45:14.646 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-26T23:45:14.647 INFO:tasks.ceph.mds.b:Stopped 2024-01-26T23:45:14.648 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-26T23:45:14.648 INFO:tasks.ceph.mds.d:Stopped 2024-01-26T23:45:14.648 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-26T23:45:14.689 INFO:tasks.ceph.mds.a:Started 2024-01-26T23:45:14.689 DEBUG:teuthology.parallel:result is None 2024-01-26T23:45:14.690 INFO:tasks.ceph.mds.c:Started 2024-01-26T23:45:14.690 DEBUG:teuthology.parallel:result is None 2024-01-26T23:45:14.690 INFO:tasks.ceph.mds.b:Started 2024-01-26T23:45:14.691 INFO:tasks.ceph.mds.d:Started 2024-01-26T23:45:14.691 DEBUG:teuthology.parallel:result is None 2024-01-26T23:45:14.691 DEBUG:teuthology.parallel:result is None 2024-01-26T23:45:14.691 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-26T23:45:14.772 INFO:tasks.ceph.mds.b.smithi106.stdout:starting mds.b at 2024-01-26T23:45:14.772 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-01-26T23:45:14.778 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-26T23:45:14.778 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-26T23:45:14.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.831+0000 7f54690b0700 1 Processor -- start 2024-01-26T23:45:15.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.831+0000 7f54690b0700 1 -- start start 2024-01-26T23:45:15.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.832+0000 7f54690b0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5464159760 0x7f54640b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:15.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.832+0000 7f54690b0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f54640b3060 0x7f54640b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:15.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.832+0000 7f54690b0700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f546406f940 con 0x7f5464159b30 2024-01-26T23:45:15.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.832+0000 7f54690b0700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546406ed30 con 0x7f54640b3060 2024-01-26T23:45:15.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.832+0000 7f54690b0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546406f010 con 0x7f5464159760 2024-01-26T23:45:15.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.832+0000 7f546259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f54640b3060 0x7f54640b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:15.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.832+0000 7f546359e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5464159b30 0x7f546415d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49066/0 (socket says 172.21.15.46:49066) 2024-01-26T23:45:15.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.832+0000 7f546259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f54640b3060 0x7f54640b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44636/0 (socket says 172.21.15.46:44636) 2024-01-26T23:45:15.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.832+0000 7f546359e700 1 -- 172.21.15.46:0/3019162929 learned_addr learned my addr 172.21.15.46:0/3019162929 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:15.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.832+0000 7f5462d9d700 1 --2- 172.21.15.46:0/3019162929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5464159760 0x7f54640b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:15.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.832+0000 7f546259c700 1 -- 172.21.15.46:0/3019162929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5464159760 msgr2=0x7f54640b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:15.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.832+0000 7f546259c700 1 --2- 172.21.15.46:0/3019162929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5464159760 0x7f54640b2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.832+0000 7f546259c700 1 -- 172.21.15.46:0/3019162929 >> v1:172.21.15.46:6789/0 conn(0x7f5464159b30 legacy=0x7f546415d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.832+0000 7f5461d9b700 1 -- 172.21.15.46:0/3019162929 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338210367 0 0) 0x7f546406f940 con 0x7f5464159b30 2024-01-26T23:45:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.832+0000 7f546259c700 1 -- 172.21.15.46:0/3019162929 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546406f2f0 con 0x7f54640b3060 2024-01-26T23:45:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.833+0000 7f546259c700 1 --2- 172.21.15.46:0/3019162929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54640b3060 0x7f54640b0bb0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f54580098e0 tx=0x7f54580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5cabea9ec55edd server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.833+0000 7f5461d9b700 1 -- 172.21.15.46:0/3019162929 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545800b640 con 0x7f54640b3060 2024-01-26T23:45:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.833+0000 7f5461d9b700 1 -- 172.21.15.46:0/3019162929 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545800b7a0 con 0x7f54640b3060 2024-01-26T23:45:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.833+0000 7f5461d9b700 1 -- 172.21.15.46:0/3019162929 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545800bb40 con 0x7f54640b3060 2024-01-26T23:45:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.833+0000 7f54690b0700 1 -- 172.21.15.46:0/3019162929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54640b3060 msgr2=0x7f54640b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.833+0000 7f54690b0700 1 --2- 172.21.15.46:0/3019162929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54640b3060 0x7f54640b0bb0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f54580098e0 tx=0x7f54580091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.833+0000 7f54690b0700 1 -- 172.21.15.46:0/3019162929 shutdown_connections 2024-01-26T23:45:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.833+0000 7f54690b0700 1 --2- 172.21.15.46:0/3019162929 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5464159760 0x7f54640b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.833+0000 7f54690b0700 1 --2- 172.21.15.46:0/3019162929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54640b3060 0x7f54640b0bb0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.833+0000 7f54690b0700 1 -- 172.21.15.46:0/3019162929 >> 172.21.15.46:0/3019162929 conn(0x7f5464069510 msgr2=0x7f54640aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.833+0000 7f54690b0700 1 -- 172.21.15.46:0/3019162929 shutdown_connections 2024-01-26T23:45:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.833+0000 7f54690b0700 1 -- 172.21.15.46:0/3019162929 wait complete. 2024-01-26T23:45:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.835+0000 7f54690b0700 1 Processor -- start 2024-01-26T23:45:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.835+0000 7f54690b0700 1 -- start start 2024-01-26T23:45:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f54690b0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54640b3060 0x7f5464151eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f54690b0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5464159760 0x7f546415aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f54690b0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5464159b30 0x7f54641322b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f54690b0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546415b530 con 0x7f5464159b30 2024-01-26T23:45:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f54690b0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546415b760 con 0x7f54640b3060 2024-01-26T23:45:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f54690b0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546415ba40 con 0x7f5464159760 2024-01-26T23:45:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f546359e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5464159b30 0x7f54641322b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f546359e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5464159b30 0x7f54641322b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44644/0 (socket says 172.21.15.46:44644) 2024-01-26T23:45:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f546359e700 1 -- 172.21.15.46:0/1261338170 learned_addr learned my addr 172.21.15.46:0/1261338170 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f546259c700 1 --2- 172.21.15.46:0/1261338170 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5464159760 0x7f546415aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f5462d9d700 1 --2- 172.21.15.46:0/1261338170 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54640b3060 0x7f5464151eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f546359e700 1 -- 172.21.15.46:0/1261338170 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5464159760 msgr2=0x7f546415aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f546359e700 1 --2- 172.21.15.46:0/1261338170 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5464159760 0x7f546415aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f546359e700 1 -- 172.21.15.46:0/1261338170 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54640b3060 msgr2=0x7f5464151eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f546359e700 1 --2- 172.21.15.46:0/1261338170 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54640b3060 0x7f5464151eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f546359e700 1 -- 172.21.15.46:0/1261338170 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5458009420 con 0x7f5464159b30 2024-01-26T23:45:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f546359e700 1 --2- 172.21.15.46:0/1261338170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5464159b30 0x7f54641322b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f545c00cbf0 tx=0x7f545c0019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f544affd700 1 -- 172.21.15.46:0/1261338170 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c00a8b0 con 0x7f5464159b30 2024-01-26T23:45:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f544affd700 1 -- 172.21.15.46:0/1261338170 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545c00aa10 con 0x7f5464159b30 2024-01-26T23:45:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f54690b0700 1 -- 172.21.15.46:0/1261338170 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546415bd20 con 0x7f5464159b30 2024-01-26T23:45:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.836+0000 7f544affd700 1 -- 172.21.15.46:0/1261338170 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c00add0 con 0x7f5464159b30 2024-01-26T23:45:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.837+0000 7f54690b0700 1 -- 172.21.15.46:0/1261338170 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546415c2e0 con 0x7f5464159b30 2024-01-26T23:45:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.838+0000 7f54690b0700 1 -- 172.21.15.46:0/1261338170 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546415c500 con 0x7f5464159b30 2024-01-26T23:45:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.838+0000 7f544affd700 1 -- 172.21.15.46:0/1261338170 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f545c007300 con 0x7f5464159b30 2024-01-26T23:45:15.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.839+0000 7f544affd700 1 --2- 172.21.15.46:0/1261338170 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5450066f30 0x7f54500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:15.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.839+0000 7f544affd700 1 -- 172.21.15.46:0/1261338170 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6167+0+0 (secure 0 0 0) 0x7f545c085500 con 0x7f5464159b30 2024-01-26T23:45:15.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.839+0000 7f5462d9d700 1 --2- 172.21.15.46:0/1261338170 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5450066f30 0x7f54500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:15.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.840+0000 7f5462d9d700 1 --2- 172.21.15.46:0/1261338170 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5450066f30 0x7f54500693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5454009db0 tx=0x7f54540091a0 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:15.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:14.843+0000 7f544affd700 1 -- 172.21.15.46:0/1261338170 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f545c052600 con 0x7f5464159b30 2024-01-26T23:45:15.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.065+0000 7f54690b0700 1 -- 172.21.15.46:0/1261338170 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f546404c710 con 0x7f5464159b30 2024-01-26T23:45:15.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.733+0000 7f544affd700 1 -- 172.21.15.46:0/1261338170 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v79) v1 ==== 97+0+0 (secure 0 0 0) 0x7f545c055c20 con 0x7f5464159b30 2024-01-26T23:45:15.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.739+0000 7f54690b0700 1 -- 172.21.15.46:0/1261338170 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5450066f30 msgr2=0x7f54500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:15.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.739+0000 7f54690b0700 1 --2- 172.21.15.46:0/1261338170 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5450066f30 0x7f54500693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5454009db0 tx=0x7f54540091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.739+0000 7f54690b0700 1 -- 172.21.15.46:0/1261338170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5464159b30 msgr2=0x7f54641322b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:15.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.739+0000 7f54690b0700 1 --2- 172.21.15.46:0/1261338170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5464159b30 0x7f54641322b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f545c00cbf0 tx=0x7f545c0019b0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.739+0000 7f54690b0700 1 -- 172.21.15.46:0/1261338170 shutdown_connections 2024-01-26T23:45:15.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.739+0000 7f54690b0700 1 --2- 172.21.15.46:0/1261338170 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5450066f30 0x7f54500693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.739+0000 7f54690b0700 1 --2- 172.21.15.46:0/1261338170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5464159b30 0x7f54641322b0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.739+0000 7f54690b0700 1 --2- 172.21.15.46:0/1261338170 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54640b3060 0x7f5464151eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.739+0000 7f54690b0700 1 --2- 172.21.15.46:0/1261338170 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5464159760 0x7f546415aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.739+0000 7f54690b0700 1 -- 172.21.15.46:0/1261338170 >> 172.21.15.46:0/1261338170 conn(0x7f5464069510 msgr2=0x7f54640ae720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:15.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.739+0000 7f54690b0700 1 -- 172.21.15.46:0/1261338170 shutdown_connections 2024-01-26T23:45:15.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.739+0000 7f54690b0700 1 -- 172.21.15.46:0/1261338170 wait complete. 2024-01-26T23:45:15.741 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-01-26T23:45:15.756 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-26T23:45:15.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.892+0000 7f5a0cc2a700 1 Processor -- start 2024-01-26T23:45:15.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.892+0000 7f5a0cc2a700 1 -- start start 2024-01-26T23:45:15.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.893+0000 7f5a0cc2a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a080b2b40 0x7f5a080ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:15.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.893+0000 7f5a0cc2a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5a080ae960 0x7f5a080ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:15.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.893+0000 7f5a0cc2a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a080aedd0 0x7f5a080cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:15.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.893+0000 7f5a0cc2a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a08099fc0 con 0x7f5a080b2b40 2024-01-26T23:45:15.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.893+0000 7f5a0cc2a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a0815a0b0 con 0x7f5a080ae960 2024-01-26T23:45:15.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.893+0000 7f5a0cc2a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a080ae4d0 con 0x7f5a080aedd0 2024-01-26T23:45:15.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.893+0000 7f5a0659c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a080b2b40 0x7f5a080ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:15.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.893+0000 7f5a0659c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a080b2b40 0x7f5a080ad1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44652/0 (socket says 172.21.15.46:44652) 2024-01-26T23:45:15.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.893+0000 7f5a0659c700 1 -- 172.21.15.46:0/3037681265 learned_addr learned my addr 172.21.15.46:0/3037681265 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:15.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.894+0000 7f5a06d9d700 1 --2- 172.21.15.46:0/3037681265 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a080aedd0 0x7f5a080cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:15.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.894+0000 7f5a05d9b700 1 --2- 172.21.15.46:0/3037681265 >> v2:172.21.15.106:3300/0 conn(0x7f5a080ae960 0x7f5a080ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:15.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.894+0000 7f5a0659c700 1 -- 172.21.15.46:0/3037681265 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a080aedd0 msgr2=0x7f5a080cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:15.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.894+0000 7f5a0659c700 1 --2- 172.21.15.46:0/3037681265 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a080aedd0 0x7f5a080cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.894+0000 7f5a0659c700 1 -- 172.21.15.46:0/3037681265 >> v2:172.21.15.106:3300/0 conn(0x7f5a080ae960 msgr2=0x7f5a080ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:15.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.894+0000 7f5a0659c700 1 --2- 172.21.15.46:0/3037681265 >> v2:172.21.15.106:3300/0 conn(0x7f5a080ae960 0x7f5a080ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.894+0000 7f5a0659c700 1 -- 172.21.15.46:0/3037681265 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a080ad8c0 con 0x7f5a080b2b40 2024-01-26T23:45:15.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.894+0000 7f5a0659c700 1 --2- 172.21.15.46:0/3037681265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a080b2b40 0x7f5a080ad1c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f5a080ce2f0 tx=0x7f59f8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744b1ed58e7fe9bf server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:15.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.894+0000 7f5a0559a700 1 -- 172.21.15.46:0/3037681265 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f8005c10 con 0x7f5a080b2b40 2024-01-26T23:45:15.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.894+0000 7f5a0559a700 1 -- 172.21.15.46:0/3037681265 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f8005d70 con 0x7f5a080b2b40 2024-01-26T23:45:15.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.894+0000 7f5a0559a700 1 -- 172.21.15.46:0/3037681265 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f800b880 con 0x7f5a080b2b40 2024-01-26T23:45:15.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.895+0000 7f5a0cc2a700 1 -- 172.21.15.46:0/3037681265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a080b2b40 msgr2=0x7f5a080ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:15.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.895+0000 7f5a0cc2a700 1 --2- 172.21.15.46:0/3037681265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a080b2b40 0x7f5a080ad1c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f5a080ce2f0 tx=0x7f59f8009220 comp rx=0 tx=0).stop 2024-01-26T23:45:15.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.895+0000 7f5a0cc2a700 1 -- 172.21.15.46:0/3037681265 shutdown_connections 2024-01-26T23:45:15.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.895+0000 7f5a0cc2a700 1 --2- 172.21.15.46:0/3037681265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a080b2b40 0x7f5a080ad1c0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.895+0000 7f5a0cc2a700 1 --2- 172.21.15.46:0/3037681265 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a080aedd0 0x7f5a080cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.895+0000 7f5a0cc2a700 1 --2- 172.21.15.46:0/3037681265 >> v2:172.21.15.106:3300/0 conn(0x7f5a080ae960 0x7f5a080ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.895+0000 7f5a0cc2a700 1 -- 172.21.15.46:0/3037681265 >> 172.21.15.46:0/3037681265 conn(0x7f5a08099a30 msgr2=0x7f5a0806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:15.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.895+0000 7f5a0cc2a700 1 -- 172.21.15.46:0/3037681265 shutdown_connections 2024-01-26T23:45:15.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.895+0000 7f5a0cc2a700 1 -- 172.21.15.46:0/3037681265 wait complete. 2024-01-26T23:45:15.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.896+0000 7f5a0cc2a700 1 Processor -- start 2024-01-26T23:45:15.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.896+0000 7f5a0cc2a700 1 -- start start 2024-01-26T23:45:15.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.897+0000 7f5a0cc2a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a080ae960 0x7f5a08151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:15.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.897+0000 7f5a0cc2a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a080b2b40 0x7f5a080c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:15.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.897+0000 7f5a0cc2a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a080c7d60 0x7f5a08158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:15.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.897+0000 7f5a0cc2a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a080c81e0 con 0x7f5a080ae960 2024-01-26T23:45:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.897+0000 7f5a0cc2a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a080c99b0 con 0x7f5a080b2b40 2024-01-26T23:45:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.897+0000 7f5a0cc2a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a080c9c90 con 0x7f5a080c7d60 2024-01-26T23:45:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.897+0000 7f5a0659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a080ae960 0x7f5a08151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.897+0000 7f5a0659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a080ae960 0x7f5a08151410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44660/0 (socket says 172.21.15.46:44660) 2024-01-26T23:45:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.897+0000 7f5a0659c700 1 -- 172.21.15.46:0/1453639965 learned_addr learned my addr 172.21.15.46:0/1453639965 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.897+0000 7f5a06d9d700 1 --2- 172.21.15.46:0/1453639965 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a080c7d60 0x7f5a08158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.897+0000 7f5a05d9b700 1 --2- 172.21.15.46:0/1453639965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a080b2b40 0x7f5a080c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.897+0000 7f5a0659c700 1 -- 172.21.15.46:0/1453639965 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a080c7d60 msgr2=0x7f5a08158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.897+0000 7f5a0659c700 1 --2- 172.21.15.46:0/1453639965 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a080c7d60 0x7f5a08158870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.897+0000 7f5a0659c700 1 -- 172.21.15.46:0/1453639965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a080b2b40 msgr2=0x7f5a080c7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.897+0000 7f5a0659c700 1 --2- 172.21.15.46:0/1453639965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a080b2b40 0x7f5a080c7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.897+0000 7f5a0659c700 1 -- 172.21.15.46:0/1453639965 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f8009480 con 0x7f5a080ae960 2024-01-26T23:45:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.898+0000 7f5a0659c700 1 --2- 172.21.15.46:0/1453639965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a080ae960 0x7f5a08151410 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f59f800b3c0 tx=0x7f59f8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:15.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.898+0000 7f59f67fc700 1 -- 172.21.15.46:0/1453639965 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f8011740 con 0x7f5a080ae960 2024-01-26T23:45:15.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.898+0000 7f59f67fc700 1 -- 172.21.15.46:0/1453639965 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f8020070 con 0x7f5a080ae960 2024-01-26T23:45:15.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.898+0000 7f5a0cc2a700 1 -- 172.21.15.46:0/1453639965 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a080c9f70 con 0x7f5a080ae960 2024-01-26T23:45:15.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.898+0000 7f59f67fc700 1 -- 172.21.15.46:0/1453639965 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f8011ad0 con 0x7f5a080ae960 2024-01-26T23:45:15.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.898+0000 7f5a0cc2a700 1 -- 172.21.15.46:0/1453639965 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a080ca530 con 0x7f5a080ae960 2024-01-26T23:45:15.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.899+0000 7f5a0cc2a700 1 -- 172.21.15.46:0/1453639965 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a080ca6c0 con 0x7f5a080ae960 2024-01-26T23:45:15.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.899+0000 7f59f67fc700 1 -- 172.21.15.46:0/1453639965 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f59f8022030 con 0x7f5a080ae960 2024-01-26T23:45:15.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.900+0000 7f59f67fc700 1 --2- 172.21.15.46:0/1453639965 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f59e8066f30 0x7f59e80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:15.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.900+0000 7f59f67fc700 1 -- 172.21.15.46:0/1453639965 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5565+0+0 (secure 0 0 0) 0x7f59f8085fe0 con 0x7f5a080ae960 2024-01-26T23:45:15.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.900+0000 7f5a05d9b700 1 --2- 172.21.15.46:0/1453639965 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f59e8066f30 0x7f59e80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:15.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.900+0000 7f5a05d9b700 1 --2- 172.21.15.46:0/1453639965 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f59e8066f30 0x7f59e80693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5a0806abc0 tx=0x7f59fc008040 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:15.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:15.904+0000 7f59f67fc700 1 -- 172.21.15.46:0/1453639965 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f59f8053240 con 0x7f5a080ae960 2024-01-26T23:45:16.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.194+0000 7f5a0cc2a700 1 -- 172.21.15.46:0/1453639965 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5a0804c710 con 0x7f5a080ae960 2024-01-26T23:45:16.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.194+0000 7f59f67fc700 1 -- 172.21.15.46:0/1453639965 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f59f8056860 con 0x7f5a080ae960 2024-01-26T23:45:16.196 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:16.196 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:45:16.196 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-01-26T23:45:16.196 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.196 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-01-26T23:45:16.196 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlM6dKKBAAPpeWeDZWrFgc/4zuxKCvHw==", 2024-01-26T23:45:16.197 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.197 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:45:16.197 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:45:16.197 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:45:16.197 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:16.197 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.197 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.197 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.197 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-01-26T23:45:16.197 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlmzHRNRAAflWYZ/p4AXeMy54tvLpugA==", 2024-01-26T23:45:16.197 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.197 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:45:16.197 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:45:16.198 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:45:16.198 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:16.198 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.198 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.198 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.198 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-01-26T23:45:16.198 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlMZxoMBAAG5kyfm+LLG86y/p2A8BfDQ==", 2024-01-26T23:45:16.198 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.198 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:45:16.198 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:45:16.198 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:45:16.198 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:16.198 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.198 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.199 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.199 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-01-26T23:45:16.199 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl5jx/ARAA3cwtLAOWQ/n2NG+xwEZ3QQ==", 2024-01-26T23:45:16.199 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.199 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:45:16.199 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:45:16.199 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:45:16.199 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:16.199 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.199 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.199 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.199 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-01-26T23:45:16.199 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4QLRlJ2OKJRAAWVhxEdQkUW20294w/KLL6g==", 2024-01-26T23:45:16.199 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.200 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:45:16.200 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:45:16.200 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:16.200 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.200 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.200 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.200 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-01-26T23:45:16.200 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7QLRlE2tAJRAAzNlTiCr6a2eWiabRy1SqSw==", 2024-01-26T23:45:16.200 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.200 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:45:16.200 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:45:16.200 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:16.201 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.201 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.201 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.201 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-01-26T23:45:16.201 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+QLRlyySaJBAA5072ba5HUNi2dYE3D02M/Q==", 2024-01-26T23:45:16.201 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.201 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:45:16.201 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:45:16.201 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:16.201 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.201 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.201 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.201 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-01-26T23:45:16.201 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBQLRl9+tAIxAApZk0btXIpQPWSb3BYUYsFQ==", 2024-01-26T23:45:16.202 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.202 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:45:16.202 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:45:16.202 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:16.202 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.202 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.202 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.202 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-01-26T23:45:16.202 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHQLRlurUtJxAAT9fwKrAWl62VOXwShg7NJg==", 2024-01-26T23:45:16.202 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.202 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:45:16.202 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:45:16.202 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:16.202 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.202 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.203 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.203 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-01-26T23:45:16.203 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKQLRl7aqoJRAAXu9fykWELsW3SwnahoIOLA==", 2024-01-26T23:45:16.203 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.203 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:45:16.203 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:45:16.203 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:16.203 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.203 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.203 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.203 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-01-26T23:45:16.203 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNQLRlWf7pJBAAopmFLIKb8s32YZs46VwbGg==", 2024-01-26T23:45:16.203 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.204 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:45:16.204 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:45:16.204 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:16.204 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.204 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.204 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.204 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-01-26T23:45:16.204 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQQLRlNGfhIxAAmCgmOB3QAGfSIYHBb5BUZw==", 2024-01-26T23:45:16.204 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.204 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:45:16.204 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:45:16.204 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:16.204 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.205 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.205 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.205 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-01-26T23:45:16.205 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRleO8wBhAA1018AcO3KcyOZxgB0Itxaw==", 2024-01-26T23:45:16.205 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.205 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:45:16.205 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-01-26T23:45:16.205 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-26T23:45:16.205 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.205 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.205 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.205 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-01-26T23:45:16.205 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlsiZEDRAAjXSamHe2bZcDqVsqXqIcGg==", 2024-01-26T23:45:16.206 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.206 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:45:16.206 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:45:16.206 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:45:16.206 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:45:16.206 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.206 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.206 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.206 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-01-26T23:45:16.206 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl6YT1CRAApXlN4j20xzwda0tDMy94Eg==", 2024-01-26T23:45:16.206 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.206 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:45:16.206 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:45:16.206 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:45:16.207 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:45:16.207 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.207 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.207 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.207 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-01-26T23:45:16.207 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlHvDQEhAAxMhgWJsZC5/6Af5zFV6XIA==", 2024-01-26T23:45:16.207 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.207 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:45:16.207 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:45:16.207 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:45:16.207 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:45:16.207 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.207 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.208 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.208 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-01-26T23:45:16.208 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlvMwDCBAAXnFyMduz7PfWdLWbjytIiQ==", 2024-01-26T23:45:16.208 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.208 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:45:16.208 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:45:16.208 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:45:16.208 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:16.208 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.208 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.208 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.208 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-01-26T23:45:16.208 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlhMNSJBAA38QU6DrCvFjO+u9zQ8bkEA==", 2024-01-26T23:45:16.209 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.209 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:45:16.209 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-26T23:45:16.209 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:16.209 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.209 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:16.209 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:16.209 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-01-26T23:45:16.209 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlI9EbIBAACTpzAO+8A/VFJZcT/WWBng==", 2024-01-26T23:45:16.209 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:16.209 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:45:16.209 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-26T23:45:16.209 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:16.210 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.210 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:16.210 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-01-26T23:45:16.210 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:45:16.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.196+0000 7f5a0cc2a700 1 -- 172.21.15.46:0/1453639965 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f59e8066f30 msgr2=0x7f59e80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.196+0000 7f5a0cc2a700 1 --2- 172.21.15.46:0/1453639965 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f59e8066f30 0x7f59e80693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5a0806abc0 tx=0x7f59fc008040 comp rx=0 tx=0).stop 2024-01-26T23:45:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.197+0000 7f5a0cc2a700 1 -- 172.21.15.46:0/1453639965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a080ae960 msgr2=0x7f5a08151410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.197+0000 7f5a0cc2a700 1 --2- 172.21.15.46:0/1453639965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a080ae960 0x7f5a08151410 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f59f800b3c0 tx=0x7f59f8016000 comp rx=0 tx=0).stop 2024-01-26T23:45:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.197+0000 7f5a0cc2a700 1 -- 172.21.15.46:0/1453639965 shutdown_connections 2024-01-26T23:45:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.197+0000 7f5a0cc2a700 1 --2- 172.21.15.46:0/1453639965 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f59e8066f30 0x7f59e80693e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.197+0000 7f5a0cc2a700 1 --2- 172.21.15.46:0/1453639965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a080ae960 0x7f5a08151410 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.197+0000 7f5a0cc2a700 1 --2- 172.21.15.46:0/1453639965 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a080c7d60 0x7f5a08158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.197+0000 7f5a0cc2a700 1 --2- 172.21.15.46:0/1453639965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a080b2b40 0x7f5a080c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.197+0000 7f5a0cc2a700 1 -- 172.21.15.46:0/1453639965 >> 172.21.15.46:0/1453639965 conn(0x7f5a08099a30 msgr2=0x7f5a08151c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.197+0000 7f5a0cc2a700 1 -- 172.21.15.46:0/1453639965 shutdown_connections 2024-01-26T23:45:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.197+0000 7f5a0cc2a700 1 -- 172.21.15.46:0/1453639965 wait complete. 2024-01-26T23:45:16.213 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:45:16.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.344+0000 7fa1b4ce9700 1 Processor -- start 2024-01-26T23:45:16.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.345+0000 7fa1b4ce9700 1 -- start start 2024-01-26T23:45:16.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.346+0000 7fa1b4ce9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa1b00cb930 0x7fa1b0159c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:16.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.346+0000 7fa1b4ce9700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b006eb20 con 0x7fa1b0068040 2024-01-26T23:45:16.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.346+0000 7fa1b4ce9700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b006f6a0 con 0x7fa1b01598a0 2024-01-26T23:45:16.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.346+0000 7fa1b4ce9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1b006ed70 con 0x7fa1b00cb930 2024-01-26T23:45:16.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.346+0000 7fa1add9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa1b0068040 0x7fa1b00cb220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49082/0 (socket says 172.21.15.46:49082) 2024-01-26T23:45:16.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.346+0000 7fa1add9b700 1 -- 172.21.15.46:0/1246755727 learned_addr learned my addr 172.21.15.46:0/1246755727 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:16.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.346+0000 7fa1aed9d700 1 --2- 172.21.15.46:0/1246755727 >> v2:172.21.15.46:3300/0 conn(0x7fa1b00cb930 0x7fa1b0159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:16.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.346+0000 7fa1aed9d700 1 -- 172.21.15.46:0/1246755727 >> v1:172.21.15.106:6789/0 conn(0x7fa1b01598a0 legacy=0x7fa1b0067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:16.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.346+0000 7fa1aed9d700 1 -- 172.21.15.46:0/1246755727 >> v1:172.21.15.46:6789/0 conn(0x7fa1b0068040 legacy=0x7fa1b00cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:16.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.346+0000 7fa1aed9d700 1 -- 172.21.15.46:0/1246755727 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b006f050 con 0x7fa1b00cb930 2024-01-26T23:45:16.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.346+0000 7fa1ad59a700 1 -- 172.21.15.46:0/1246755727 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738383827 0 0) 0x7fa1b006eb20 con 0x7fa1b0068040 2024-01-26T23:45:16.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.346+0000 7fa1aed9d700 1 --2- 172.21.15.46:0/1246755727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b00cb930 0x7fa1b0159c70 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8009e60 tx=0x7fa1a8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c2955126a93da7 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:16.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.346+0000 7fa1ad59a700 1 -- 172.21.15.46:0/1246755727 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a800e070 con 0x7fa1b00cb930 2024-01-26T23:45:16.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.347+0000 7fa1ad59a700 1 -- 172.21.15.46:0/1246755727 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a800bc30 con 0x7fa1b00cb930 2024-01-26T23:45:16.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.347+0000 7fa1ad59a700 1 -- 172.21.15.46:0/1246755727 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8012430 con 0x7fa1b00cb930 2024-01-26T23:45:16.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.347+0000 7fa1b4ce9700 1 -- 172.21.15.46:0/1246755727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b00cb930 msgr2=0x7fa1b0159c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:16.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.347+0000 7fa1b4ce9700 1 --2- 172.21.15.46:0/1246755727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b00cb930 0x7fa1b0159c70 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8009e60 tx=0x7fa1a8009780 comp rx=0 tx=0).stop 2024-01-26T23:45:16.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.347+0000 7fa1b4ce9700 1 -- 172.21.15.46:0/1246755727 shutdown_connections 2024-01-26T23:45:16.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.347+0000 7fa1b4ce9700 1 --2- 172.21.15.46:0/1246755727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b00cb930 0x7fa1b0159c70 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.347+0000 7fa1b4ce9700 1 -- 172.21.15.46:0/1246755727 >> 172.21.15.46:0/1246755727 conn(0x7fa1b00cdf10 msgr2=0x7fa1b00c9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:16.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.347+0000 7fa1b4ce9700 1 -- 172.21.15.46:0/1246755727 shutdown_connections 2024-01-26T23:45:16.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.347+0000 7fa1b4ce9700 1 -- 172.21.15.46:0/1246755727 wait complete. 2024-01-26T23:45:16.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.348+0000 7fa1b4ce9700 1 Processor -- start 2024-01-26T23:45:16.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.349+0000 7fa1b4ce9700 1 -- start start 2024-01-26T23:45:16.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.349+0000 7fa1b4ce9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1b0068040 0x7fa1b01690a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:16.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.349+0000 7fa1b4ce9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b00cb930 0x7fa1b0153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:16.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.349+0000 7fa1b4ce9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1b01598a0 0x7fa1b0152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:16.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.349+0000 7fa1b4ce9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b0155250 con 0x7fa1b00cb930 2024-01-26T23:45:16.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.349+0000 7fa1b4ce9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b0154640 con 0x7fa1b0068040 2024-01-26T23:45:16.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.349+0000 7fa1b4ce9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b0154920 con 0x7fa1b01598a0 2024-01-26T23:45:16.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.349+0000 7fa1add9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b00cb930 0x7fa1b0153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:16.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.349+0000 7fa1add9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b00cb930 0x7fa1b0153ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44680/0 (socket says 172.21.15.46:44680) 2024-01-26T23:45:16.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.349+0000 7fa1add9b700 1 -- 172.21.15.46:0/2024753493 learned_addr learned my addr 172.21.15.46:0/2024753493 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:16.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.349+0000 7fa1ae59c700 1 --2- 172.21.15.46:0/2024753493 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1b0068040 0x7fa1b01690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:16.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.350+0000 7fa1aed9d700 1 --2- 172.21.15.46:0/2024753493 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1b01598a0 0x7fa1b0152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:16.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.350+0000 7fa1add9b700 1 -- 172.21.15.46:0/2024753493 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1b01598a0 msgr2=0x7fa1b0152730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:16.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.350+0000 7fa1add9b700 1 --2- 172.21.15.46:0/2024753493 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1b01598a0 0x7fa1b0152730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.350+0000 7fa1add9b700 1 -- 172.21.15.46:0/2024753493 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1b0068040 msgr2=0x7fa1b01690a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:16.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.350+0000 7fa1add9b700 1 --2- 172.21.15.46:0/2024753493 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1b0068040 0x7fa1b01690a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.350+0000 7fa1add9b700 1 -- 172.21.15.46:0/2024753493 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a8009a20 con 0x7fa1b00cb930 2024-01-26T23:45:16.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.350+0000 7fa1add9b700 1 --2- 172.21.15.46:0/2024753493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b00cb930 0x7fa1b0153ba0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa1a000c810 tx=0x7fa1a0007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:16.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.350+0000 7fa1a67fc700 1 -- 172.21.15.46:0/2024753493 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0005c00 con 0x7fa1b00cb930 2024-01-26T23:45:16.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.350+0000 7fa1a67fc700 1 -- 172.21.15.46:0/2024753493 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a0005d60 con 0x7fa1b00cb930 2024-01-26T23:45:16.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.350+0000 7fa1b4ce9700 1 -- 172.21.15.46:0/2024753493 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b0154c00 con 0x7fa1b00cb930 2024-01-26T23:45:16.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.350+0000 7fa1a67fc700 1 -- 172.21.15.46:0/2024753493 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0007300 con 0x7fa1b00cb930 2024-01-26T23:45:16.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.350+0000 7fa1b4ce9700 1 -- 172.21.15.46:0/2024753493 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b01695e0 con 0x7fa1b00cb930 2024-01-26T23:45:16.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.351+0000 7fa1b4ce9700 1 -- 172.21.15.46:0/2024753493 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b004c710 con 0x7fa1b00cb930 2024-01-26T23:45:16.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.352+0000 7fa1a67fc700 1 -- 172.21.15.46:0/2024753493 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa1a001b3b0 con 0x7fa1b00cb930 2024-01-26T23:45:16.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.352+0000 7fa1a67fc700 1 --2- 172.21.15.46:0/2024753493 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa190066f30 0x7fa1900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:16.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.352+0000 7fa1a67fc700 1 -- 172.21.15.46:0/2024753493 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5565+0+0 (secure 0 0 0) 0x7fa1a0083960 con 0x7fa1b00cb930 2024-01-26T23:45:16.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.353+0000 7fa1ae59c700 1 --2- 172.21.15.46:0/2024753493 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa190066f30 0x7fa1900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:16.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.353+0000 7fa1ae59c700 1 --2- 172.21.15.46:0/2024753493 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa190066f30 0x7fa1900693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa1980039e0 tx=0x7fa198007ec0 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:16.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.357+0000 7fa1a67fc700 1 -- 172.21.15.46:0/2024753493 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa1a0050c60 con 0x7fa1b00cb930 2024-01-26T23:45:16.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.562+0000 7fa1b4ce9700 1 -- 172.21.15.46:0/2024753493 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1b00684d0 con 0x7fa1b00cb930 2024-01-26T23:45:16.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.563+0000 7fa1a67fc700 1 -- 172.21.15.46:0/2024753493 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+12933 (secure 0 0 0) 0x7fa1a0017020 con 0x7fa1b00cb930 2024-01-26T23:45:16.564 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:16.564 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":81,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:45:15.740058+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:45:16.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.565+0000 7fa1b4ce9700 1 -- 172.21.15.46:0/2024753493 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa190066f30 msgr2=0x7fa1900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.565+0000 7fa1b4ce9700 1 --2- 172.21.15.46:0/2024753493 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa190066f30 0x7fa1900693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa1980039e0 tx=0x7fa198007ec0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.565+0000 7fa1b4ce9700 1 -- 172.21.15.46:0/2024753493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b00cb930 msgr2=0x7fa1b0153ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.565+0000 7fa1b4ce9700 1 --2- 172.21.15.46:0/2024753493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b00cb930 0x7fa1b0153ba0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa1a000c810 tx=0x7fa1a0007db0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.565+0000 7fa1b4ce9700 1 -- 172.21.15.46:0/2024753493 shutdown_connections 2024-01-26T23:45:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.565+0000 7fa1b4ce9700 1 --2- 172.21.15.46:0/2024753493 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa190066f30 0x7fa1900693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.565+0000 7fa1b4ce9700 1 --2- 172.21.15.46:0/2024753493 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1b01598a0 0x7fa1b0152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.565+0000 7fa1b4ce9700 1 --2- 172.21.15.46:0/2024753493 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1b0068040 0x7fa1b01690a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.565+0000 7fa1b4ce9700 1 --2- 172.21.15.46:0/2024753493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b00cb930 0x7fa1b0153ba0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.565+0000 7fa1b4ce9700 1 -- 172.21.15.46:0/2024753493 >> 172.21.15.46:0/2024753493 conn(0x7fa1b00cdf10 msgr2=0x7fa1b00c9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.565+0000 7fa1b4ce9700 1 -- 172.21.15.46:0/2024753493 shutdown_connections 2024-01-26T23:45:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.565+0000 7fa1b4ce9700 1 -- 172.21.15.46:0/2024753493 wait complete. 2024-01-26T23:45:16.580 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-26T23:45:16.580 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:45:16.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.715+0000 7f2435e3c700 1 Processor -- start 2024-01-26T23:45:16.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.715+0000 7f2435e3c700 1 -- start start 2024-01-26T23:45:16.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.716+0000 7f2435e3c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2430067af0 0x7f2430067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:16.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.716+0000 7f2435e3c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f243006ebe0 0x7f24300ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:16.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.716+0000 7f2435e3c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f243006fdb0 con 0x7f24301597e0 2024-01-26T23:45:16.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.716+0000 7f2435e3c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24300ade10 con 0x7f243006ebe0 2024-01-26T23:45:16.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.716+0000 7f2435e3c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f243006f1a0 con 0x7f2430067af0 2024-01-26T23:45:16.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.716+0000 7f242effd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f243006ebe0 0x7f24300ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:16.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.716+0000 7f242f7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2430067af0 0x7f2430067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:16.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.716+0000 7f242ffff700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f24301597e0 0x7f2430159bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:53670/0 (socket says 172.21.15.46:53670) 2024-01-26T23:45:16.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.716+0000 7f242effd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f243006ebe0 0x7f24300ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:46594/0 (socket says 172.21.15.46:46594) 2024-01-26T23:45:16.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.716+0000 7f242ffff700 1 -- 172.21.15.46:0/1399686221 learned_addr learned my addr 172.21.15.46:0/1399686221 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:16.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.716+0000 7f242f7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2430067af0 0x7f2430067ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:45418/0 (socket says 172.21.15.46:45418) 2024-01-26T23:45:16.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.716+0000 7f242effd700 1 -- 172.21.15.46:0/1399686221 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2430067af0 msgr2=0x7f2430067ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:16.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.716+0000 7f242effd700 1 --2- 172.21.15.46:0/1399686221 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2430067af0 0x7f2430067ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.716+0000 7f242effd700 1 -- 172.21.15.46:0/1399686221 >> v1:172.21.15.106:6789/0 conn(0x7f24301597e0 legacy=0x7f2430159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:16.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.716+0000 7f242effd700 1 -- 172.21.15.46:0/1399686221 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243006f480 con 0x7f243006ebe0 2024-01-26T23:45:16.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.716+0000 7f242e7fc700 1 -- 172.21.15.46:0/1399686221 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285280593 0 0) 0x7f243006fdb0 con 0x7f24301597e0 2024-01-26T23:45:16.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.717+0000 7f242effd700 1 --2- 172.21.15.46:0/1399686221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f243006ebe0 0x7f24300ad8d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f24180098e0 tx=0x7f24180091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=442eb6d81d12aef9 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:16.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.717+0000 7f242e7fc700 1 -- 172.21.15.46:0/1399686221 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241800b640 con 0x7f243006ebe0 2024-01-26T23:45:16.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.717+0000 7f242e7fc700 1 -- 172.21.15.46:0/1399686221 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f241800b7a0 con 0x7f243006ebe0 2024-01-26T23:45:16.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.717+0000 7f242e7fc700 1 -- 172.21.15.46:0/1399686221 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241800bb40 con 0x7f243006ebe0 2024-01-26T23:45:16.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.717+0000 7f2435e3c700 1 -- 172.21.15.46:0/1399686221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f243006ebe0 msgr2=0x7f24300ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:16.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.717+0000 7f2435e3c700 1 --2- 172.21.15.46:0/1399686221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f243006ebe0 0x7f24300ad8d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f24180098e0 tx=0x7f24180091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.717+0000 7f2435e3c700 1 -- 172.21.15.46:0/1399686221 shutdown_connections 2024-01-26T23:45:16.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.717+0000 7f2435e3c700 1 --2- 172.21.15.46:0/1399686221 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2430067af0 0x7f2430067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.717+0000 7f2435e3c700 1 --2- 172.21.15.46:0/1399686221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f243006ebe0 0x7f24300ad8d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.717+0000 7f2435e3c700 1 -- 172.21.15.46:0/1399686221 >> 172.21.15.46:0/1399686221 conn(0x7f24300cdf80 msgr2=0x7f243006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:16.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.717+0000 7f2435e3c700 1 -- 172.21.15.46:0/1399686221 shutdown_connections 2024-01-26T23:45:16.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.717+0000 7f2435e3c700 1 -- 172.21.15.46:0/1399686221 wait complete. 2024-01-26T23:45:16.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.719+0000 7f2435e3c700 1 Processor -- start 2024-01-26T23:45:16.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.719+0000 7f2435e3c700 1 -- start start 2024-01-26T23:45:16.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.720+0000 7f2435e3c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2430067af0 0x7f243014dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:16.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.720+0000 7f2435e3c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f243006ebe0 0x7f243014d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:16.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.720+0000 7f2435e3c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f24301597e0 0x7f24300b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:16.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.720+0000 7f2435e3c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243014eef0 con 0x7f2430067af0 2024-01-26T23:45:16.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.720+0000 7f2435e3c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243014e2e0 con 0x7f243006ebe0 2024-01-26T23:45:16.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.720+0000 7f2435e3c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f243014e5c0 con 0x7f24301597e0 2024-01-26T23:45:16.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.720+0000 7f242f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2430067af0 0x7f243014dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:16.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.720+0000 7f242f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2430067af0 0x7f243014dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44694/0 (socket says 172.21.15.46:44694) 2024-01-26T23:45:16.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.720+0000 7f242f7fe700 1 -- 172.21.15.46:0/1029595670 learned_addr learned my addr 172.21.15.46:0/1029595670 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.720+0000 7f242effd700 1 --2- 172.21.15.46:0/1029595670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f243006ebe0 0x7f243014d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.720+0000 7f242ffff700 1 --2- 172.21.15.46:0/1029595670 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f24301597e0 0x7f24300b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.720+0000 7f242f7fe700 1 -- 172.21.15.46:0/1029595670 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f24301597e0 msgr2=0x7f24300b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.720+0000 7f242f7fe700 1 --2- 172.21.15.46:0/1029595670 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f24301597e0 0x7f24300b2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.720+0000 7f242f7fe700 1 -- 172.21.15.46:0/1029595670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f243006ebe0 msgr2=0x7f243014d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.720+0000 7f242f7fe700 1 --2- 172.21.15.46:0/1029595670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f243006ebe0 0x7f243014d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.720+0000 7f242f7fe700 1 -- 172.21.15.46:0/1029595670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2418009420 con 0x7f2430067af0 2024-01-26T23:45:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.720+0000 7f242f7fe700 1 --2- 172.21.15.46:0/1029595670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2430067af0 0x7f243014dab0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f242000e930 tx=0x7f24200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.721+0000 7f24277fe700 1 -- 172.21.15.46:0/1029595670 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420013680 con 0x7f2430067af0 2024-01-26T23:45:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.721+0000 7f24277fe700 1 -- 172.21.15.46:0/1029595670 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24200137e0 con 0x7f2430067af0 2024-01-26T23:45:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.721+0000 7f2435e3c700 1 -- 172.21.15.46:0/1029595670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f243014e8a0 con 0x7f2430067af0 2024-01-26T23:45:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.721+0000 7f24277fe700 1 -- 172.21.15.46:0/1029595670 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420013ba0 con 0x7f2430067af0 2024-01-26T23:45:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.721+0000 7f2435e3c700 1 -- 172.21.15.46:0/1029595670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f243014dff0 con 0x7f2430067af0 2024-01-26T23:45:16.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.722+0000 7f2435e3c700 1 -- 172.21.15.46:0/1029595670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f243004c710 con 0x7f2430067af0 2024-01-26T23:45:16.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.723+0000 7f24277fe700 1 -- 172.21.15.46:0/1029595670 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f242000c460 con 0x7f2430067af0 2024-01-26T23:45:16.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.723+0000 7f24277fe700 1 --2- 172.21.15.46:0/1029595670 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2410066f30 0x7f24100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:16.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.723+0000 7f24277fe700 1 -- 172.21.15.46:0/1029595670 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5565+0+0 (secure 0 0 0) 0x7f24200845a0 con 0x7f2430067af0 2024-01-26T23:45:16.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.723+0000 7f242effd700 1 --2- 172.21.15.46:0/1029595670 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2410066f30 0x7f24100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:16.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.724+0000 7f242effd700 1 --2- 172.21.15.46:0/1029595670 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2410066f30 0x7f24100693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f24180093f0 tx=0x7f2418017000 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:16.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.729+0000 7f24277fe700 1 -- 172.21.15.46:0/1029595670 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f24200518a0 con 0x7f2430067af0 2024-01-26T23:45:16.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.935+0000 7f2435e3c700 1 -- 172.21.15.46:0/1029595670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f243016a230 con 0x7f2430067af0 2024-01-26T23:45:16.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.935+0000 7f24277fe700 1 -- 172.21.15.46:0/1029595670 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2420054ec0 con 0x7f2430067af0 2024-01-26T23:45:16.937 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:45:16.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.937+0000 7f2435e3c700 1 -- 172.21.15.46:0/1029595670 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2410066f30 msgr2=0x7f24100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:16.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.937+0000 7f2435e3c700 1 --2- 172.21.15.46:0/1029595670 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2410066f30 0x7f24100693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f24180093f0 tx=0x7f2418017000 comp rx=0 tx=0).stop 2024-01-26T23:45:16.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.937+0000 7f2435e3c700 1 -- 172.21.15.46:0/1029595670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2430067af0 msgr2=0x7f243014dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:16.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.937+0000 7f2435e3c700 1 --2- 172.21.15.46:0/1029595670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2430067af0 0x7f243014dab0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f242000e930 tx=0x7f24200091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.937+0000 7f2435e3c700 1 -- 172.21.15.46:0/1029595670 shutdown_connections 2024-01-26T23:45:16.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.937+0000 7f2435e3c700 1 --2- 172.21.15.46:0/1029595670 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2410066f30 0x7f24100693e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.937+0000 7f2435e3c700 1 --2- 172.21.15.46:0/1029595670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2430067af0 0x7f243014dab0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.937+0000 7f2435e3c700 1 --2- 172.21.15.46:0/1029595670 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f24301597e0 0x7f24300b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.937+0000 7f2435e3c700 1 --2- 172.21.15.46:0/1029595670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f243006ebe0 0x7f243014d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:16.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.937+0000 7f2435e3c700 1 -- 172.21.15.46:0/1029595670 >> 172.21.15.46:0/1029595670 conn(0x7f24300cdf80 msgr2=0x7f24301551c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:16.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.937+0000 7f2435e3c700 1 -- 172.21.15.46:0/1029595670 shutdown_connections 2024-01-26T23:45:16.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:16.937+0000 7f2435e3c700 1 -- 172.21.15.46:0/1029595670 wait complete. 2024-01-26T23:45:16.952 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-26T23:45:17.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.083+0000 7fd0a3bdf700 1 Processor -- start 2024-01-26T23:45:17.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.083+0000 7fd0a3bdf700 1 -- start start 2024-01-26T23:45:17.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.084+0000 7fd0a3bdf700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd09c1596e0 0x7fd09c159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.084+0000 7fd0a3bdf700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd09c0c4720 0x7fd09c068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.084+0000 7fd0a3bdf700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd09c0c97f0 con 0x7fd09c0cadb0 2024-01-26T23:45:17.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.084+0000 7fd0a3bdf700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09c0cbf90 con 0x7fd09c1596e0 2024-01-26T23:45:17.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.084+0000 7fd0a3bdf700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c0c8c20 con 0x7fd09c0c4720 2024-01-26T23:45:17.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.084+0000 7fd0a217c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd09c0cadb0 0x7fd09c0c9c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49094/0 (socket says 172.21.15.46:49094) 2024-01-26T23:45:17.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.084+0000 7fd0a217c700 1 -- 172.21.15.46:0/1816667671 learned_addr learned my addr 172.21.15.46:0/1816667671 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.084+0000 7fd0a197b700 1 --2- 172.21.15.46:0/1816667671 >> v2:172.21.15.106:3300/0 conn(0x7fd09c1596e0 0x7fd09c159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.084+0000 7fd0a117a700 1 --2- 172.21.15.46:0/1816667671 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd09c0c4720 0x7fd09c068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a0979700 1 -- 172.21.15.46:0/1816667671 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293387070 0 0) 0x7fd09c0c97f0 con 0x7fd09c0cadb0 2024-01-26T23:45:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a0979700 1 -- 172.21.15.46:0/1816667671 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd088003570 con 0x7fd09c0cadb0 2024-01-26T23:45:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a0979700 1 -- 172.21.15.46:0/1816667671 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 327668419 0 0) 0x7fd088003570 con 0x7fd09c0cadb0 2024-01-26T23:45:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a0979700 1 -- 172.21.15.46:0/1816667671 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd09c0c4720 msgr2=0x7fd09c068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a0979700 1 --2- 172.21.15.46:0/1816667671 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd09c0c4720 0x7fd09c068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a0979700 1 -- 172.21.15.46:0/1816667671 >> v2:172.21.15.106:3300/0 conn(0x7fd09c1596e0 msgr2=0x7fd09c159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a0979700 1 --2- 172.21.15.46:0/1816667671 >> v2:172.21.15.106:3300/0 conn(0x7fd09c1596e0 0x7fd09c159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a0979700 1 -- 172.21.15.46:0/1816667671 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c0c8ec0 con 0x7fd09c0cadb0 2024-01-26T23:45:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a0979700 1 -- 172.21.15.46:0/1816667671 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fd0940017f0 con 0x7fd09c0cadb0 2024-01-26T23:45:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a197b700 1 --2- 172.21.15.46:0/1816667671 >> v2:172.21.15.106:3300/0 conn(0x7fd09c1596e0 0x7fd09c159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a0979700 1 -- 172.21.15.46:0/1816667671 >> v1:172.21.15.46:6789/0 conn(0x7fd09c0cadb0 legacy=0x7fd09c0c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a0979700 1 --2- 172.21.15.46:0/1816667671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd088003a60 0x7fd088003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a0979700 1 --2- 172.21.15.46:0/1816667671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd088004810 0x7fd088004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a0979700 1 --2- 172.21.15.46:0/1816667671 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd09c0c4720 0x7fd0880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a0979700 1 -- 172.21.15.46:0/1816667671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c0cbf90 con 0x7fd088004810 2024-01-26T23:45:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a0979700 1 -- 172.21.15.46:0/1816667671 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c0c8c20 con 0x7fd088003a60 2024-01-26T23:45:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a0979700 1 -- 172.21.15.46:0/1816667671 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd08800ba60 con 0x7fd09c0c4720 2024-01-26T23:45:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a117a700 1 --2- 172.21.15.46:0/1816667671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd088004810 0x7fd088004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a197b700 1 --2- 172.21.15.46:0/1816667671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd088003a60 0x7fd088003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a217c700 1 --2- 172.21.15.46:0/1816667671 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd09c0c4720 0x7fd0880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a117a700 1 -- 172.21.15.46:0/1816667671 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd09c0c4720 msgr2=0x7fd0880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a117a700 1 --2- 172.21.15.46:0/1816667671 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd09c0c4720 0x7fd0880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a117a700 1 -- 172.21.15.46:0/1816667671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd088003a60 msgr2=0x7fd088003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a117a700 1 --2- 172.21.15.46:0/1816667671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd088003a60 0x7fd088003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.085+0000 7fd0a117a700 1 -- 172.21.15.46:0/1816667671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd088003570 con 0x7fd088004810 2024-01-26T23:45:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.086+0000 7fd0a117a700 1 --2- 172.21.15.46:0/1816667671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd088004810 0x7fd088004c80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd08c009c30 tx=0x7fd08c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98068ea24ce4c9f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.086+0000 7fd0a0979700 1 -- 172.21.15.46:0/1816667671 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c00e710 con 0x7fd088004810 2024-01-26T23:45:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.086+0000 7fd0a0979700 1 -- 172.21.15.46:0/1816667671 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08c00e870 con 0x7fd088004810 2024-01-26T23:45:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.086+0000 7fd0a0979700 1 -- 172.21.15.46:0/1816667671 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c00ec30 con 0x7fd088004810 2024-01-26T23:45:17.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.086+0000 7fd0a3bdf700 1 -- 172.21.15.46:0/1816667671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd088004810 msgr2=0x7fd088004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:17.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.086+0000 7fd0a3bdf700 1 --2- 172.21.15.46:0/1816667671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd088004810 0x7fd088004c80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd08c009c30 tx=0x7fd08c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.086+0000 7fd0a217c700 1 -- 172.21.15.46:0/1816667671 reap_dead start 2024-01-26T23:45:17.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.086+0000 7fd0a3bdf700 1 -- 172.21.15.46:0/1816667671 shutdown_connections 2024-01-26T23:45:17.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.086+0000 7fd0a3bdf700 1 -- 172.21.15.46:0/1816667671 >> 172.21.15.46:0/1816667671 conn(0x7fd09c097820 msgr2=0x7fd09c068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:17.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.086+0000 7fd0a3bdf700 1 -- 172.21.15.46:0/1816667671 shutdown_connections 2024-01-26T23:45:17.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.087+0000 7fd0a3bdf700 1 -- 172.21.15.46:0/1816667671 wait complete. 2024-01-26T23:45:17.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.088+0000 7fd0a3bdf700 1 Processor -- start 2024-01-26T23:45:17.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.088+0000 7fd0a3bdf700 1 -- start start 2024-01-26T23:45:17.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.088+0000 7fd0a3bdf700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd088004810 0x7fd09c15c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.088+0000 7fd0a3bdf700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd09c0b12e0 0x7fd09c0b1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.088+0000 7fd0a3bdf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd09c153860 0x7fd09c153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.088+0000 7fd0a3bdf700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c0b1c90 con 0x7fd09c153860 2024-01-26T23:45:17.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd0a3bdf700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c0c62d0 con 0x7fd09c0b12e0 2024-01-26T23:45:17.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd0a3bdf700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c152d40 con 0x7fd088004810 2024-01-26T23:45:17.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd0a217c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd09c153860 0x7fd09c153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd0a217c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd09c153860 0x7fd09c153cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44716/0 (socket says 172.21.15.46:44716) 2024-01-26T23:45:17.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd0a217c700 1 -- 172.21.15.46:0/2912084413 learned_addr learned my addr 172.21.15.46:0/2912084413 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd0a117a700 1 --2- 172.21.15.46:0/2912084413 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd09c0b12e0 0x7fd09c0b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd0a197b700 1 --2- 172.21.15.46:0/2912084413 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd088004810 0x7fd09c15c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd0a217c700 1 -- 172.21.15.46:0/2912084413 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd088004810 msgr2=0x7fd09c15c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd0a217c700 1 --2- 172.21.15.46:0/2912084413 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd088004810 0x7fd09c15c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd0a217c700 1 -- 172.21.15.46:0/2912084413 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd09c0b12e0 msgr2=0x7fd09c0b1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd0a217c700 1 --2- 172.21.15.46:0/2912084413 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd09c0b12e0 0x7fd09c0b1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd0a217c700 1 -- 172.21.15.46:0/2912084413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c154580 con 0x7fd09c153860 2024-01-26T23:45:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd0a117a700 1 --2- 172.21.15.46:0/2912084413 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd09c0b12e0 0x7fd09c0b1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd0a217c700 1 --2- 172.21.15.46:0/2912084413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd09c153860 0x7fd09c153cb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0b4630 tx=0x7fd094002190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd091ffb700 1 -- 172.21.15.46:0/2912084413 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09400ebc0 con 0x7fd09c153860 2024-01-26T23:45:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd091ffb700 1 -- 172.21.15.46:0/2912084413 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09400ed40 con 0x7fd09c153860 2024-01-26T23:45:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd0a3bdf700 1 -- 172.21.15.46:0/2912084413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd09c154860 con 0x7fd09c153860 2024-01-26T23:45:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.089+0000 7fd091ffb700 1 -- 172.21.15.46:0/2912084413 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd094013670 con 0x7fd09c153860 2024-01-26T23:45:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.090+0000 7fd0a3bdf700 1 -- 172.21.15.46:0/2912084413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd09c0c5f60 con 0x7fd09c153860 2024-01-26T23:45:17.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.091+0000 7fd0a3bdf700 1 -- 172.21.15.46:0/2912084413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09c06db70 con 0x7fd09c153860 2024-01-26T23:45:17.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.091+0000 7fd091ffb700 1 -- 172.21.15.46:0/2912084413 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd094013aa0 con 0x7fd09c153860 2024-01-26T23:45:17.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.092+0000 7fd091ffb700 1 --2- 172.21.15.46:0/2912084413 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd080066f30 0x7fd0800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.092+0000 7fd091ffb700 1 -- 172.21.15.46:0/2912084413 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5565+0+0 (secure 0 0 0) 0x7fd094084b80 con 0x7fd09c153860 2024-01-26T23:45:17.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.092+0000 7fd0a197b700 1 --2- 172.21.15.46:0/2912084413 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd080066f30 0x7fd0800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.093+0000 7fd0a197b700 1 --2- 172.21.15.46:0/2912084413 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd080066f30 0x7fd0800693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd098006f00 tx=0x7fd098008040 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:17.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.096+0000 7fd091ffb700 1 -- 172.21.15.46:0/2912084413 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd094051de0 con 0x7fd09c153860 2024-01-26T23:45:17.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.300+0000 7fd0a3bdf700 1 -- 172.21.15.46:0/2912084413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd09c06db70 con 0x7fd09c153860 2024-01-26T23:45:17.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.301+0000 7fd091ffb700 1 -- 172.21.15.46:0/2912084413 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd094084e60 con 0x7fd09c153860 2024-01-26T23:45:17.302 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:17.302 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-26T23:45:17.302 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:17.302 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-01-26T23:45:17.302 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-01-26T23:45:17.302 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:17.302 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-26T23:45:17.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.303+0000 7fd0a3bdf700 1 -- 172.21.15.46:0/2912084413 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd080066f30 msgr2=0x7fd0800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:17.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.303+0000 7fd0a3bdf700 1 --2- 172.21.15.46:0/2912084413 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd080066f30 0x7fd0800693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd098006f00 tx=0x7fd098008040 comp rx=0 tx=0).stop 2024-01-26T23:45:17.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.303+0000 7fd0a3bdf700 1 -- 172.21.15.46:0/2912084413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd09c153860 msgr2=0x7fd09c153cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:17.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.303+0000 7fd0a3bdf700 1 --2- 172.21.15.46:0/2912084413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd09c153860 0x7fd09c153cb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0b4630 tx=0x7fd094002190 comp rx=0 tx=0).stop 2024-01-26T23:45:17.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.303+0000 7fd0a3bdf700 1 -- 172.21.15.46:0/2912084413 shutdown_connections 2024-01-26T23:45:17.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.303+0000 7fd0a3bdf700 1 --2- 172.21.15.46:0/2912084413 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd080066f30 0x7fd0800693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.303+0000 7fd0a3bdf700 1 --2- 172.21.15.46:0/2912084413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd09c153860 0x7fd09c153cb0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.303+0000 7fd0a3bdf700 1 --2- 172.21.15.46:0/2912084413 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd088004810 0x7fd09c15c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.303+0000 7fd0a3bdf700 1 --2- 172.21.15.46:0/2912084413 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd09c0b12e0 0x7fd09c0b1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.303+0000 7fd0a3bdf700 1 -- 172.21.15.46:0/2912084413 >> 172.21.15.46:0/2912084413 conn(0x7fd09c097820 msgr2=0x7fd09c06cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:17.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.303+0000 7fd0a3bdf700 1 -- 172.21.15.46:0/2912084413 shutdown_connections 2024-01-26T23:45:17.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.303+0000 7fd0a3bdf700 1 -- 172.21.15.46:0/2912084413 wait complete. 2024-01-26T23:45:17.318 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-26T23:45:17.318 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-26T23:45:17.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.453+0000 7f3c6ae3a700 1 Processor -- start 2024-01-26T23:45:17.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.453+0000 7f3c6ae3a700 1 -- start start 2024-01-26T23:45:17.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.454+0000 7f3c6ae3a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c640c9980 0x7f3c6415ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.454+0000 7f3c6ae3a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3c64156b00 0x7f3c6406eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.454+0000 7f3c6ae3a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c64157000 con 0x7f3c640c9d50 2024-01-26T23:45:17.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.454+0000 7f3c6ae3a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c6406f3e0 con 0x7f3c640c9980 2024-01-26T23:45:17.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.454+0000 7f3c6ae3a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c6406f6c0 con 0x7f3c64156b00 2024-01-26T23:45:17.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.454+0000 7f3c68bd6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c640c9980 0x7f3c6415ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.454+0000 7f3c68bd6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c640c9980 0x7f3c6415ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44718/0 (socket says 172.21.15.46:44718) 2024-01-26T23:45:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.454+0000 7f3c68bd6700 1 -- 172.21.15.46:0/2392291956 learned_addr learned my addr 172.21.15.46:0/2392291956 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.454+0000 7f3c63fff700 1 --2- 172.21.15.46:0/2392291956 >> v2:172.21.15.106:3300/0 conn(0x7f3c64156b00 0x7f3c6406eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.454+0000 7f3c68bd6700 1 -- 172.21.15.46:0/2392291956 >> v2:172.21.15.106:3300/0 conn(0x7f3c64156b00 msgr2=0x7f3c6406eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.454+0000 7f3c68bd6700 1 --2- 172.21.15.46:0/2392291956 >> v2:172.21.15.106:3300/0 conn(0x7f3c64156b00 0x7f3c6406eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.454+0000 7f3c68bd6700 1 -- 172.21.15.46:0/2392291956 >> v1:172.21.15.106:6789/0 conn(0x7f3c640c9d50 legacy=0x7f3c640b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.454+0000 7f3c68bd6700 1 -- 172.21.15.46:0/2392291956 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6406f9a0 con 0x7f3c640c9980 2024-01-26T23:45:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.454+0000 7f3c68bd6700 1 --2- 172.21.15.46:0/2392291956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c640c9980 0x7f3c6415ae10 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3c6415a190 tx=0x7f3c5800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ddf9958ce1ca816 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.455+0000 7f3c637fe700 1 -- 172.21.15.46:0/2392291956 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5800dc10 con 0x7f3c640c9980 2024-01-26T23:45:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.455+0000 7f3c637fe700 1 -- 172.21.15.46:0/2392291956 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5800dd70 con 0x7f3c640c9980 2024-01-26T23:45:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.455+0000 7f3c637fe700 1 -- 172.21.15.46:0/2392291956 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c580106c0 con 0x7f3c640c9980 2024-01-26T23:45:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.455+0000 7f3c6ae3a700 1 -- 172.21.15.46:0/2392291956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c640c9980 msgr2=0x7f3c6415ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.455+0000 7f3c6ae3a700 1 --2- 172.21.15.46:0/2392291956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c640c9980 0x7f3c6415ae10 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3c6415a190 tx=0x7f3c5800b4f0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.455+0000 7f3c6ae3a700 1 -- 172.21.15.46:0/2392291956 shutdown_connections 2024-01-26T23:45:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.455+0000 7f3c6ae3a700 1 --2- 172.21.15.46:0/2392291956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c640c9980 0x7f3c6415ae10 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.455+0000 7f3c6ae3a700 1 --2- 172.21.15.46:0/2392291956 >> v2:172.21.15.106:3300/0 conn(0x7f3c64156b00 0x7f3c6406eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.455+0000 7f3c6ae3a700 1 -- 172.21.15.46:0/2392291956 >> 172.21.15.46:0/2392291956 conn(0x7f3c64069510 msgr2=0x7f3c640aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.455+0000 7f3c6ae3a700 1 -- 172.21.15.46:0/2392291956 shutdown_connections 2024-01-26T23:45:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.455+0000 7f3c6ae3a700 1 -- 172.21.15.46:0/2392291956 wait complete. 2024-01-26T23:45:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.457+0000 7f3c6ae3a700 1 Processor -- start 2024-01-26T23:45:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.457+0000 7f3c6ae3a700 1 -- start start 2024-01-26T23:45:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.457+0000 7f3c6ae3a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c640c9980 0x7f3c64150c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.457+0000 7f3c6ae3a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c640c9d50 0x7f3c64151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c6ae3a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c64156b00 0x7f3c64152370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c6ae3a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c641670e0 con 0x7f3c640c9d50 2024-01-26T23:45:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c6ae3a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c64167350 con 0x7f3c640c9980 2024-01-26T23:45:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c6ae3a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c640cb5c0 con 0x7f3c64156b00 2024-01-26T23:45:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c63fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c640c9d50 0x7f3c64151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c63fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c640c9d50 0x7f3c64151e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44722/0 (socket says 172.21.15.46:44722) 2024-01-26T23:45:17.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c63fff700 1 -- 172.21.15.46:0/1330348400 learned_addr learned my addr 172.21.15.46:0/1330348400 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:17.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c68bd6700 1 --2- 172.21.15.46:0/1330348400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c640c9980 0x7f3c64150c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c693d7700 1 --2- 172.21.15.46:0/1330348400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c64156b00 0x7f3c64152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c63fff700 1 -- 172.21.15.46:0/1330348400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c64156b00 msgr2=0x7f3c64152370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:17.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c63fff700 1 --2- 172.21.15.46:0/1330348400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c64156b00 0x7f3c64152370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c63fff700 1 -- 172.21.15.46:0/1330348400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c640c9980 msgr2=0x7f3c64150c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:17.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c63fff700 1 --2- 172.21.15.46:0/1330348400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c640c9980 0x7f3c64150c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c63fff700 1 -- 172.21.15.46:0/1330348400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5800b790 con 0x7f3c640c9d50 2024-01-26T23:45:17.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c63fff700 1 --2- 172.21.15.46:0/1330348400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c640c9d50 0x7f3c64151e30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3c5000d860 tx=0x7f3c50008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:17.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c60ff9700 1 -- 172.21.15.46:0/1330348400 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c50009610 con 0x7f3c640c9d50 2024-01-26T23:45:17.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c60ff9700 1 -- 172.21.15.46:0/1330348400 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c50009770 con 0x7f3c640c9d50 2024-01-26T23:45:17.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c6ae3a700 1 -- 172.21.15.46:0/1330348400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c64151600 con 0x7f3c640c9d50 2024-01-26T23:45:17.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c60ff9700 1 -- 172.21.15.46:0/1330348400 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c50009b30 con 0x7f3c640c9d50 2024-01-26T23:45:17.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.458+0000 7f3c6ae3a700 1 -- 172.21.15.46:0/1330348400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c64166e00 con 0x7f3c640c9d50 2024-01-26T23:45:17.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.459+0000 7f3c6ae3a700 1 -- 172.21.15.46:0/1330348400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c640cb730 con 0x7f3c640c9d50 2024-01-26T23:45:17.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.460+0000 7f3c60ff9700 1 -- 172.21.15.46:0/1330348400 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3c5001e460 con 0x7f3c640c9d50 2024-01-26T23:45:17.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.460+0000 7f3c60ff9700 1 --2- 172.21.15.46:0/1330348400 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3c48066f30 0x7f3c480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.460+0000 7f3c60ff9700 1 -- 172.21.15.46:0/1330348400 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5565+0+0 (secure 0 0 0) 0x7f3c50084b10 con 0x7f3c640c9d50 2024-01-26T23:45:17.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.461+0000 7f3c68bd6700 1 --2- 172.21.15.46:0/1330348400 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3c48066f30 0x7f3c480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.461+0000 7f3c68bd6700 1 --2- 172.21.15.46:0/1330348400 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3c48066f30 0x7f3c480693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3c6415a190 tx=0x7f3c5801a000 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:17.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.465+0000 7f3c60ff9700 1 -- 172.21.15.46:0/1330348400 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3c50051e10 con 0x7f3c640c9d50 2024-01-26T23:45:17.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.669+0000 7f3c6ae3a700 1 -- 172.21.15.46:0/1330348400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3c6404c710 con 0x7f3c640c9d50 2024-01-26T23:45:17.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.739+0000 7f3c60ff9700 1 -- 172.21.15.46:0/1330348400 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3c50055430 con 0x7f3c640c9d50 2024-01-26T23:45:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.744+0000 7f3c6ae3a700 1 -- 172.21.15.46:0/1330348400 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3c48066f30 msgr2=0x7f3c480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.744+0000 7f3c6ae3a700 1 --2- 172.21.15.46:0/1330348400 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3c48066f30 0x7f3c480693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3c6415a190 tx=0x7f3c5801a000 comp rx=0 tx=0).stop 2024-01-26T23:45:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.744+0000 7f3c6ae3a700 1 -- 172.21.15.46:0/1330348400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c640c9d50 msgr2=0x7f3c64151e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.744+0000 7f3c6ae3a700 1 --2- 172.21.15.46:0/1330348400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c640c9d50 0x7f3c64151e30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3c5000d860 tx=0x7f3c50008040 comp rx=0 tx=0).stop 2024-01-26T23:45:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.744+0000 7f3c6ae3a700 1 -- 172.21.15.46:0/1330348400 shutdown_connections 2024-01-26T23:45:17.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.744+0000 7f3c6ae3a700 1 --2- 172.21.15.46:0/1330348400 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3c48066f30 0x7f3c480693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.744+0000 7f3c6ae3a700 1 --2- 172.21.15.46:0/1330348400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c64156b00 0x7f3c64152370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.744+0000 7f3c6ae3a700 1 --2- 172.21.15.46:0/1330348400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c640c9980 0x7f3c64150c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.744+0000 7f3c6ae3a700 1 --2- 172.21.15.46:0/1330348400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c640c9d50 0x7f3c64151e30 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.744+0000 7f3c6ae3a700 1 -- 172.21.15.46:0/1330348400 >> 172.21.15.46:0/1330348400 conn(0x7f3c64069510 msgr2=0x7f3c640b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:17.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.744+0000 7f3c6ae3a700 1 -- 172.21.15.46:0/1330348400 shutdown_connections 2024-01-26T23:45:17.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.745+0000 7f3c6ae3a700 1 -- 172.21.15.46:0/1330348400 wait complete. 2024-01-26T23:45:17.748 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-01-26T23:45:17.761 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-26T23:45:17.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.900+0000 7eff9a905700 1 Processor -- start 2024-01-26T23:45:17.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.900+0000 7eff9a905700 1 -- start start 2024-01-26T23:45:17.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.901+0000 7eff9a905700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff94067af0 0x7eff94067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.901+0000 7eff9a905700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff9406fdb0 con 0x7eff941597e0 2024-01-26T23:45:17.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.901+0000 7eff9a905700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff9406f1a0 con 0x7eff9406ebe0 2024-01-26T23:45:17.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.901+0000 7eff9a905700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff9406f480 con 0x7eff94067af0 2024-01-26T23:45:17.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.901+0000 7eff93fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff94067af0 0x7eff94067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.901+0000 7eff93fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff94067af0 0x7eff94067ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44724/0 (socket says 172.21.15.46:44724) 2024-01-26T23:45:17.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.901+0000 7eff98ea2700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7eff941597e0 0x7eff94159bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49098/0 (socket says 172.21.15.46:49098) 2024-01-26T23:45:17.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.901+0000 7eff93fff700 1 -- 172.21.15.46:0/4257035312 learned_addr learned my addr 172.21.15.46:0/4257035312 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:17.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.901+0000 7eff93fff700 1 -- 172.21.15.46:0/4257035312 >> v1:172.21.15.106:6789/0 conn(0x7eff9406ebe0 legacy=0x7eff940ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:17.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.901+0000 7eff93fff700 1 -- 172.21.15.46:0/4257035312 >> v1:172.21.15.46:6789/0 conn(0x7eff941597e0 legacy=0x7eff94159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:17.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.901+0000 7eff93fff700 1 -- 172.21.15.46:0/4257035312 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9406f760 con 0x7eff94067af0 2024-01-26T23:45:17.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.901+0000 7eff92ffd700 1 -- 172.21.15.46:0/4257035312 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120296340 0 0) 0x7eff9406fdb0 con 0x7eff941597e0 2024-01-26T23:45:17.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.902+0000 7eff93fff700 1 --2- 172.21.15.46:0/4257035312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff94067af0 0x7eff94067ec0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7eff840099b0 tx=0x7eff84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e582564809d7865e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:17.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.902+0000 7eff92ffd700 1 -- 172.21.15.46:0/4257035312 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8400b860 con 0x7eff94067af0 2024-01-26T23:45:17.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.902+0000 7eff92ffd700 1 -- 172.21.15.46:0/4257035312 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8400b9c0 con 0x7eff94067af0 2024-01-26T23:45:17.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.902+0000 7eff92ffd700 1 -- 172.21.15.46:0/4257035312 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8400bd80 con 0x7eff94067af0 2024-01-26T23:45:17.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.902+0000 7eff9a905700 1 -- 172.21.15.46:0/4257035312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff94067af0 msgr2=0x7eff94067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:17.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.902+0000 7eff9a905700 1 --2- 172.21.15.46:0/4257035312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff94067af0 0x7eff94067ec0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7eff840099b0 tx=0x7eff84009250 comp rx=0 tx=0).stop 2024-01-26T23:45:17.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.902+0000 7eff9a905700 1 -- 172.21.15.46:0/4257035312 shutdown_connections 2024-01-26T23:45:17.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.902+0000 7eff9a905700 1 --2- 172.21.15.46:0/4257035312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff94067af0 0x7eff94067ec0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.902+0000 7eff9a905700 1 -- 172.21.15.46:0/4257035312 >> 172.21.15.46:0/4257035312 conn(0x7eff940cdf80 msgr2=0x7eff9406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:17.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.903+0000 7eff9a905700 1 -- 172.21.15.46:0/4257035312 shutdown_connections 2024-01-26T23:45:17.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.903+0000 7eff9a905700 1 -- 172.21.15.46:0/4257035312 wait complete. 2024-01-26T23:45:17.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.904+0000 7eff9a905700 1 Processor -- start 2024-01-26T23:45:17.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.904+0000 7eff9a905700 1 -- start start 2024-01-26T23:45:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff9a905700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff94067af0 0x7eff9415b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff9a905700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9406ebe0 0x7eff941580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff9a905700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff941597e0 0x7eff940c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff9a905700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff94159bf0 con 0x7eff9406ebe0 2024-01-26T23:45:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff9a905700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9416e620 con 0x7eff94067af0 2024-01-26T23:45:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff9a905700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9416e900 con 0x7eff941597e0 2024-01-26T23:45:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff937fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9406ebe0 0x7eff941580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff937fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9406ebe0 0x7eff941580f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44730/0 (socket says 172.21.15.46:44730) 2024-01-26T23:45:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff937fe700 1 -- 172.21.15.46:0/2611374631 learned_addr learned my addr 172.21.15.46:0/2611374631 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff93fff700 1 --2- 172.21.15.46:0/2611374631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff94067af0 0x7eff9415b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff98ea2700 1 --2- 172.21.15.46:0/2611374631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff941597e0 0x7eff940c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff937fe700 1 -- 172.21.15.46:0/2611374631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff941597e0 msgr2=0x7eff940c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:17.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff937fe700 1 --2- 172.21.15.46:0/2611374631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff941597e0 0x7eff940c8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff937fe700 1 -- 172.21.15.46:0/2611374631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff94067af0 msgr2=0x7eff9415b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:17.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff937fe700 1 --2- 172.21.15.46:0/2611374631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff94067af0 0x7eff9415b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:17.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff937fe700 1 -- 172.21.15.46:0/2611374631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff840094f0 con 0x7eff9406ebe0 2024-01-26T23:45:17.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff98ea2700 1 --2- 172.21.15.46:0/2611374631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff941597e0 0x7eff940c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:17.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.905+0000 7eff937fe700 1 --2- 172.21.15.46:0/2611374631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9406ebe0 0x7eff941580f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7eff7c00c4e0 tx=0x7eff7c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:17.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.906+0000 7eff8bfff700 1 -- 172.21.15.46:0/2611374631 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c011590 con 0x7eff9406ebe0 2024-01-26T23:45:17.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.906+0000 7eff8bfff700 1 -- 172.21.15.46:0/2611374631 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff7c011710 con 0x7eff9406ebe0 2024-01-26T23:45:17.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.906+0000 7eff9a905700 1 -- 172.21.15.46:0/2611374631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff9416ebe0 con 0x7eff9406ebe0 2024-01-26T23:45:17.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.906+0000 7eff8bfff700 1 -- 172.21.15.46:0/2611374631 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c011ad0 con 0x7eff9406ebe0 2024-01-26T23:45:17.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.906+0000 7eff9a905700 1 -- 172.21.15.46:0/2611374631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff9415bd70 con 0x7eff9406ebe0 2024-01-26T23:45:17.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.907+0000 7eff9a905700 1 -- 172.21.15.46:0/2611374631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff94159d90 con 0x7eff9406ebe0 2024-01-26T23:45:17.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.907+0000 7eff8bfff700 1 -- 172.21.15.46:0/2611374631 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7eff7c01b3b0 con 0x7eff9406ebe0 2024-01-26T23:45:17.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.908+0000 7eff8bfff700 1 --2- 172.21.15.46:0/2611374631 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff74066f30 0x7eff740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:17.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.908+0000 7eff8bfff700 1 -- 172.21.15.46:0/2611374631 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5935+0+0 (secure 0 0 0) 0x7eff7c083aa0 con 0x7eff9406ebe0 2024-01-26T23:45:17.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.908+0000 7eff93fff700 1 --2- 172.21.15.46:0/2611374631 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff74066f30 0x7eff740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:17.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.908+0000 7eff93fff700 1 --2- 172.21.15.46:0/2611374631 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff74066f30 0x7eff740693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7eff840099b0 tx=0x7eff84017000 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:17.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:17.913+0000 7eff8bfff700 1 -- 172.21.15.46:0/2611374631 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7eff7c050c30 con 0x7eff9406ebe0 2024-01-26T23:45:18.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.115+0000 7eff9a905700 1 -- 172.21.15.46:0/2611374631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7eff9404c710 con 0x7eff9406ebe0 2024-01-26T23:45:18.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.743+0000 7eff8bfff700 1 -- 172.21.15.46:0/2611374631 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) v1 ==== 170+0+0 (secure 0 0 0) 0x7eff7c017070 con 0x7eff9406ebe0 2024-01-26T23:45:18.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.749+0000 7eff9a905700 1 -- 172.21.15.46:0/2611374631 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff74066f30 msgr2=0x7eff740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:18.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.749+0000 7eff9a905700 1 --2- 172.21.15.46:0/2611374631 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff74066f30 0x7eff740693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7eff840099b0 tx=0x7eff84017000 comp rx=0 tx=0).stop 2024-01-26T23:45:18.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.749+0000 7eff9a905700 1 -- 172.21.15.46:0/2611374631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9406ebe0 msgr2=0x7eff941580f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:18.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.749+0000 7eff9a905700 1 --2- 172.21.15.46:0/2611374631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9406ebe0 0x7eff941580f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7eff7c00c4e0 tx=0x7eff7c00c7f0 comp rx=0 tx=0).stop 2024-01-26T23:45:18.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.751+0000 7eff9a905700 1 -- 172.21.15.46:0/2611374631 shutdown_connections 2024-01-26T23:45:18.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.751+0000 7eff9a905700 1 --2- 172.21.15.46:0/2611374631 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7eff74066f30 0x7eff740693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:18.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.751+0000 7eff9a905700 1 --2- 172.21.15.46:0/2611374631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff941597e0 0x7eff940c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:18.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.751+0000 7eff9a905700 1 --2- 172.21.15.46:0/2611374631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff94067af0 0x7eff9415b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:18.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.751+0000 7eff9a905700 1 --2- 172.21.15.46:0/2611374631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9406ebe0 0x7eff941580f0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:18.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.751+0000 7eff9a905700 1 -- 172.21.15.46:0/2611374631 >> 172.21.15.46:0/2611374631 conn(0x7eff940cdf80 msgr2=0x7eff940cb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:18.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.753+0000 7eff9a905700 1 -- 172.21.15.46:0/2611374631 shutdown_connections 2024-01-26T23:45:18.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.756+0000 7eff9a905700 1 -- 172.21.15.46:0/2611374631 wait complete. 2024-01-26T23:45:18.767 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-01-26T23:45:18.781 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-26T23:45:18.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.918+0000 7f45198f9700 1 Processor -- start 2024-01-26T23:45:18.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.918+0000 7f45198f9700 1 -- start start 2024-01-26T23:45:18.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.919+0000 7f45198f9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f45141599d0 0x7f4514159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:18.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.919+0000 7f45198f9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f45140ab1c0 0x7f45140b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:18.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.919+0000 7f45198f9700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45140b1a60 con 0x7f451415d8c0 2024-01-26T23:45:18.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.919+0000 7f45198f9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45140ab5f0 con 0x7f45140ab1c0 2024-01-26T23:45:18.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.919+0000 7f45198f9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45140b0f40 con 0x7f45141599d0 2024-01-26T23:45:18.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.919+0000 7f45127fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f45140ab1c0 0x7f45140b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:18.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.919+0000 7f45137fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f451415d8c0 0x7f45140b20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49108/0 (socket says 172.21.15.46:49108) 2024-01-26T23:45:18.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.919+0000 7f45137fe700 1 -- 172.21.15.46:0/2941868450 learned_addr learned my addr 172.21.15.46:0/2941868450 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:18.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.919+0000 7f45127fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f45140ab1c0 0x7f45140b0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44740/0 (socket says 172.21.15.46:44740) 2024-01-26T23:45:18.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.919+0000 7f4512ffd700 1 --2- 172.21.15.46:0/2941868450 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f45141599d0 0x7f4514159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:18.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.919+0000 7f45127fc700 1 -- 172.21.15.46:0/2941868450 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f45141599d0 msgr2=0x7f4514159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:18.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.919+0000 7f45127fc700 1 --2- 172.21.15.46:0/2941868450 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f45141599d0 0x7f4514159da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:18.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.919+0000 7f45127fc700 1 -- 172.21.15.46:0/2941868450 >> v1:172.21.15.46:6789/0 conn(0x7f451415d8c0 legacy=0x7f45140b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:18.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.919+0000 7f45127fc700 1 -- 172.21.15.46:0/2941868450 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45140b11c0 con 0x7f45140ab1c0 2024-01-26T23:45:18.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.919+0000 7f4511ffb700 1 -- 172.21.15.46:0/2941868450 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695019082 0 0) 0x7f45140b1a60 con 0x7f451415d8c0 2024-01-26T23:45:18.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.920+0000 7f45127fc700 1 --2- 172.21.15.46:0/2941868450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45140ab1c0 0x7f45140b0a00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f45080098e0 tx=0x7f45080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebaaf24d196b3521 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:18.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.920+0000 7f4511ffb700 1 -- 172.21.15.46:0/2941868450 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4508009e80 con 0x7f45140ab1c0 2024-01-26T23:45:18.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.920+0000 7f4511ffb700 1 -- 172.21.15.46:0/2941868450 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450800b860 con 0x7f45140ab1c0 2024-01-26T23:45:18.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.920+0000 7f4511ffb700 1 -- 172.21.15.46:0/2941868450 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450800bbe0 con 0x7f45140ab1c0 2024-01-26T23:45:18.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.920+0000 7f45198f9700 1 -- 172.21.15.46:0/2941868450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45140ab1c0 msgr2=0x7f45140b0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:18.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.920+0000 7f45198f9700 1 --2- 172.21.15.46:0/2941868450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45140ab1c0 0x7f45140b0a00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f45080098e0 tx=0x7f45080091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:18.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.920+0000 7f45198f9700 1 -- 172.21.15.46:0/2941868450 shutdown_connections 2024-01-26T23:45:18.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.920+0000 7f45198f9700 1 --2- 172.21.15.46:0/2941868450 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f45141599d0 0x7f4514159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:18.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.920+0000 7f45198f9700 1 --2- 172.21.15.46:0/2941868450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45140ab1c0 0x7f45140b0a00 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:18.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.920+0000 7f45198f9700 1 -- 172.21.15.46:0/2941868450 >> 172.21.15.46:0/2941868450 conn(0x7f451406f840 msgr2=0x7f4514087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:18.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.921+0000 7f45198f9700 1 -- 172.21.15.46:0/2941868450 shutdown_connections 2024-01-26T23:45:18.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.921+0000 7f45198f9700 1 -- 172.21.15.46:0/2941868450 wait complete. 2024-01-26T23:45:18.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.922+0000 7f45198f9700 1 Processor -- start 2024-01-26T23:45:18.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.922+0000 7f45198f9700 1 -- start start 2024-01-26T23:45:18.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f45198f9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45140ab1c0 0x7f451415dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:18.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f45198f9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f45141599d0 0x7f4514158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:18.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f45198f9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f451415d8c0 0x7f451409c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:18.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f45198f9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451415f180 con 0x7f45140ab1c0 2024-01-26T23:45:18.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f45198f9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451415e570 con 0x7f451415d8c0 2024-01-26T23:45:18.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f45198f9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f451415e850 con 0x7f45141599d0 2024-01-26T23:45:18.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f4512ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45140ab1c0 0x7f451415dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:18.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f4512ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45140ab1c0 0x7f451415dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44752/0 (socket says 172.21.15.46:44752) 2024-01-26T23:45:18.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f4512ffd700 1 -- 172.21.15.46:0/2592334066 learned_addr learned my addr 172.21.15.46:0/2592334066 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:18.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f45127fc700 1 --2- 172.21.15.46:0/2592334066 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f45141599d0 0x7f4514158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:18.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f45137fe700 1 --2- 172.21.15.46:0/2592334066 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f451415d8c0 0x7f451409c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:18.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f4512ffd700 1 -- 172.21.15.46:0/2592334066 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f45141599d0 msgr2=0x7f4514158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:18.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f4512ffd700 1 --2- 172.21.15.46:0/2592334066 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f45141599d0 0x7f4514158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:18.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f4512ffd700 1 -- 172.21.15.46:0/2592334066 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f451415d8c0 msgr2=0x7f451409c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:18.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f4512ffd700 1 --2- 172.21.15.46:0/2592334066 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f451415d8c0 0x7f451409c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:18.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f4512ffd700 1 -- 172.21.15.46:0/2592334066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4508009420 con 0x7f45140ab1c0 2024-01-26T23:45:18.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f45137fe700 1 --2- 172.21.15.46:0/2592334066 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f451415d8c0 0x7f451409c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:45:18.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.923+0000 7f4512ffd700 1 --2- 172.21.15.46:0/2592334066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45140ab1c0 0x7f451415dd20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f450400d810 tx=0x7f4504008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:18.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.924+0000 7f44faffd700 1 -- 172.21.15.46:0/2592334066 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45040125f0 con 0x7f45140ab1c0 2024-01-26T23:45:18.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.924+0000 7f44faffd700 1 -- 172.21.15.46:0/2592334066 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4504012750 con 0x7f45140ab1c0 2024-01-26T23:45:18.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.924+0000 7f45198f9700 1 -- 172.21.15.46:0/2592334066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f451415eb30 con 0x7f45140ab1c0 2024-01-26T23:45:18.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.924+0000 7f44faffd700 1 -- 172.21.15.46:0/2592334066 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4504012b10 con 0x7f45140ab1c0 2024-01-26T23:45:18.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.924+0000 7f45198f9700 1 -- 172.21.15.46:0/2592334066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f451415e260 con 0x7f45140ab1c0 2024-01-26T23:45:18.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.925+0000 7f45198f9700 1 -- 172.21.15.46:0/2592334066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f451404c710 con 0x7f45140ab1c0 2024-01-26T23:45:18.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.925+0000 7f44faffd700 1 -- 172.21.15.46:0/2592334066 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f450401e460 con 0x7f45140ab1c0 2024-01-26T23:45:18.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.926+0000 7f44faffd700 1 --2- 172.21.15.46:0/2592334066 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4500066f30 0x7f45000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:18.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.926+0000 7f44faffd700 1 -- 172.21.15.46:0/2592334066 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f4504084e20 con 0x7f45140ab1c0 2024-01-26T23:45:18.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.926+0000 7f45127fc700 1 --2- 172.21.15.46:0/2592334066 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4500066f30 0x7f45000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:18.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.927+0000 7f45127fc700 1 --2- 172.21.15.46:0/2592334066 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4500066f30 0x7f45000693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f45080091e0 tx=0x7f4508017000 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:18.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:18.931+0000 7f44faffd700 1 -- 172.21.15.46:0/2592334066 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4504051e30 con 0x7f45140ab1c0 2024-01-26T23:45:19.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.185+0000 7f45198f9700 1 -- 172.21.15.46:0/2592334066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f45140b14a0 con 0x7f45140ab1c0 2024-01-26T23:45:19.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.773+0000 7f44faffd700 1 -- 172.21.15.46:0/2592334066 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v32) v1 ==== 173+0+0 (secure 0 0 0) 0x7f4504055450 con 0x7f45140ab1c0 2024-01-26T23:45:19.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.786+0000 7f45198f9700 1 -- 172.21.15.46:0/2592334066 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4500066f30 msgr2=0x7f45000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:19.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.786+0000 7f45198f9700 1 --2- 172.21.15.46:0/2592334066 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4500066f30 0x7f45000693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f45080091e0 tx=0x7f4508017000 comp rx=0 tx=0).stop 2024-01-26T23:45:19.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.786+0000 7f45198f9700 1 -- 172.21.15.46:0/2592334066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45140ab1c0 msgr2=0x7f451415dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:19.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.786+0000 7f45198f9700 1 --2- 172.21.15.46:0/2592334066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45140ab1c0 0x7f451415dd20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f450400d810 tx=0x7f4504008040 comp rx=0 tx=0).stop 2024-01-26T23:45:19.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.786+0000 7f45198f9700 1 -- 172.21.15.46:0/2592334066 shutdown_connections 2024-01-26T23:45:19.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.786+0000 7f45198f9700 1 --2- 172.21.15.46:0/2592334066 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4500066f30 0x7f45000693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:19.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.786+0000 7f45198f9700 1 --2- 172.21.15.46:0/2592334066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45140ab1c0 0x7f451415dd20 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:19.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.786+0000 7f45198f9700 1 --2- 172.21.15.46:0/2592334066 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f451415d8c0 0x7f451409c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:19.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.786+0000 7f45198f9700 1 --2- 172.21.15.46:0/2592334066 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f45141599d0 0x7f4514158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:19.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.786+0000 7f45198f9700 1 -- 172.21.15.46:0/2592334066 >> 172.21.15.46:0/2592334066 conn(0x7f451406f840 msgr2=0x7f451406a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:19.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.787+0000 7f45198f9700 1 -- 172.21.15.46:0/2592334066 shutdown_connections 2024-01-26T23:45:19.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.787+0000 7f45198f9700 1 -- 172.21.15.46:0/2592334066 wait complete. 2024-01-26T23:45:19.791 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 9 and data pool 10 2024-01-26T23:45:19.806 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:45:19.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.939+0000 7f142795b700 1 Processor -- start 2024-01-26T23:45:19.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.939+0000 7f142795b700 1 -- start start 2024-01-26T23:45:19.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.940+0000 7f142795b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f142015ce40 0x7f14200d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:19.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.940+0000 7f142795b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f14200cc1c0 0x7f14200ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:19.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.940+0000 7f142795b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14200cff70 con 0x7f142015d210 2024-01-26T23:45:19.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.940+0000 7f142795b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14200cf400 con 0x7f14200cc1c0 2024-01-26T23:45:19.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.940+0000 7f142795b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14200cf660 con 0x7f142015ce40 2024-01-26T23:45:19.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.940+0000 7f1425ef8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f14200cc1c0 0x7f14200ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:19.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.940+0000 7f1424ef6700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f142015d210 0x7f14200cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49112/0 (socket says 172.21.15.46:49112) 2024-01-26T23:45:19.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.940+0000 7f1424ef6700 1 -- 172.21.15.46:0/1428853016 learned_addr learned my addr 172.21.15.46:0/1428853016 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:19.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.941+0000 7f14256f7700 1 --2- 172.21.15.46:0/1428853016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f142015ce40 0x7f14200d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:19.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.941+0000 7f1425ef8700 1 -- 172.21.15.46:0/1428853016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f142015ce40 msgr2=0x7f14200d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:19.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.941+0000 7f1425ef8700 1 --2- 172.21.15.46:0/1428853016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f142015ce40 0x7f14200d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:19.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.941+0000 7f1425ef8700 1 -- 172.21.15.46:0/1428853016 >> v1:172.21.15.46:6789/0 conn(0x7f142015d210 legacy=0x7f14200cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:19.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.941+0000 7f1417fff700 1 -- 172.21.15.46:0/1428853016 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035470073 0 0) 0x7f14200cff70 con 0x7f142015d210 2024-01-26T23:45:19.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.941+0000 7f1425ef8700 1 -- 172.21.15.46:0/1428853016 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14200d0500 con 0x7f14200cc1c0 2024-01-26T23:45:19.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.941+0000 7f1425ef8700 1 --2- 172.21.15.46:0/1428853016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14200cc1c0 0x7f14200ade30 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f1418009e60 tx=0x7f1418009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1a73d7d4eec9ee server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:19.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.941+0000 7f1417fff700 1 -- 172.21.15.46:0/1428853016 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141800e070 con 0x7f14200cc1c0 2024-01-26T23:45:19.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.941+0000 7f1417fff700 1 -- 172.21.15.46:0/1428853016 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141800bb60 con 0x7f14200cc1c0 2024-01-26T23:45:19.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.941+0000 7f1417fff700 1 -- 172.21.15.46:0/1428853016 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1418012430 con 0x7f14200cc1c0 2024-01-26T23:45:19.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.942+0000 7f142795b700 1 -- 172.21.15.46:0/1428853016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14200cc1c0 msgr2=0x7f14200ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:19.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.942+0000 7f142795b700 1 --2- 172.21.15.46:0/1428853016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14200cc1c0 0x7f14200ade30 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f1418009e60 tx=0x7f1418009780 comp rx=0 tx=0).stop 2024-01-26T23:45:19.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.942+0000 7f142795b700 1 -- 172.21.15.46:0/1428853016 shutdown_connections 2024-01-26T23:45:19.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.942+0000 7f142795b700 1 --2- 172.21.15.46:0/1428853016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14200cc1c0 0x7f14200ade30 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:19.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.942+0000 7f142795b700 1 --2- 172.21.15.46:0/1428853016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f142015ce40 0x7f14200d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:19.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.942+0000 7f142795b700 1 -- 172.21.15.46:0/1428853016 >> 172.21.15.46:0/1428853016 conn(0x7f142006f840 msgr2=0x7f14200d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:19.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.942+0000 7f142795b700 1 -- 172.21.15.46:0/1428853016 shutdown_connections 2024-01-26T23:45:19.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.942+0000 7f142795b700 1 -- 172.21.15.46:0/1428853016 wait complete. 2024-01-26T23:45:19.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.943+0000 7f142795b700 1 Processor -- start 2024-01-26T23:45:19.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.943+0000 7f142795b700 1 -- start start 2024-01-26T23:45:19.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.944+0000 7f142795b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14200cc1c0 0x7f14200c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:19.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.944+0000 7f142795b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f142015ce40 0x7f142009d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:19.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.944+0000 7f142795b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f142015d210 0x7f14200c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:19.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.944+0000 7f142795b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14200c6160 con 0x7f14200cc1c0 2024-01-26T23:45:19.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.944+0000 7f142795b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14200c5550 con 0x7f142015ce40 2024-01-26T23:45:19.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.944+0000 7f142795b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14200c5830 con 0x7f142015d210 2024-01-26T23:45:19.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.944+0000 7f14256f7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14200cc1c0 0x7f14200c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:19.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.944+0000 7f14256f7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14200cc1c0 0x7f14200c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44772/0 (socket says 172.21.15.46:44772) 2024-01-26T23:45:19.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.944+0000 7f14256f7700 1 -- 172.21.15.46:0/1730362020 learned_addr learned my addr 172.21.15.46:0/1730362020 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:19.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.944+0000 7f1425ef8700 1 --2- 172.21.15.46:0/1730362020 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f142015d210 0x7f14200c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:19.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.944+0000 7f1424ef6700 1 --2- 172.21.15.46:0/1730362020 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f142015ce40 0x7f142009d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:19.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.944+0000 7f14256f7700 1 -- 172.21.15.46:0/1730362020 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f142015d210 msgr2=0x7f14200c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:19.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.944+0000 7f14256f7700 1 --2- 172.21.15.46:0/1730362020 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f142015d210 0x7f14200c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:19.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.944+0000 7f14256f7700 1 -- 172.21.15.46:0/1730362020 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f142015ce40 msgr2=0x7f142009d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:19.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.944+0000 7f14256f7700 1 --2- 172.21.15.46:0/1730362020 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f142015ce40 0x7f142009d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:19.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.944+0000 7f14256f7700 1 -- 172.21.15.46:0/1730362020 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1418009a20 con 0x7f14200cc1c0 2024-01-26T23:45:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.945+0000 7f1425ef8700 1 --2- 172.21.15.46:0/1730362020 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f142015d210 0x7f14200c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.945+0000 7f14256f7700 1 --2- 172.21.15.46:0/1730362020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14200cc1c0 0x7f14200c65f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f141c00ec80 tx=0x7f141c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.945+0000 7f14157fa700 1 -- 172.21.15.46:0/1730362020 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141c00c9d0 con 0x7f14200cc1c0 2024-01-26T23:45:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.945+0000 7f14157fa700 1 -- 172.21.15.46:0/1730362020 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141c00cb30 con 0x7f14200cc1c0 2024-01-26T23:45:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.945+0000 7f142795b700 1 -- 172.21.15.46:0/1730362020 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14200c5b10 con 0x7f14200cc1c0 2024-01-26T23:45:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.945+0000 7f14157fa700 1 -- 172.21.15.46:0/1730362020 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141c013470 con 0x7f14200cc1c0 2024-01-26T23:45:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.945+0000 7f142795b700 1 -- 172.21.15.46:0/1730362020 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f142009de10 con 0x7f14200cc1c0 2024-01-26T23:45:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.946+0000 7f142795b700 1 -- 172.21.15.46:0/1730362020 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f142004c710 con 0x7f14200cc1c0 2024-01-26T23:45:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.946+0000 7f14157fa700 1 -- 172.21.15.46:0/1730362020 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f141c013860 con 0x7f14200cc1c0 2024-01-26T23:45:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.947+0000 7f14157fa700 1 --2- 172.21.15.46:0/1730362020 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1404066f30 0x7f14040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:19.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.947+0000 7f14157fa700 1 -- 172.21.15.46:0/1730362020 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f141c084f50 con 0x7f14200cc1c0 2024-01-26T23:45:19.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.947+0000 7f1424ef6700 1 --2- 172.21.15.46:0/1730362020 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1404066f30 0x7f14040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:19.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.948+0000 7f1424ef6700 1 --2- 172.21.15.46:0/1730362020 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1404066f30 0x7f14040693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f14200cc010 tx=0x7f1410009040 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:19.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:19.952+0000 7f14157fa700 1 -- 172.21.15.46:0/1730362020 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f141c051ef0 con 0x7f14200cc1c0 2024-01-26T23:45:20.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.157+0000 7f142795b700 1 -- 172.21.15.46:0/1730362020 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14200c5fb0 con 0x7f14200cc1c0 2024-01-26T23:45:20.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.158+0000 7f14157fa700 1 -- 172.21.15.46:0/1730362020 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16164 (secure 0 0 0) 0x7f141c017090 con 0x7f14200cc1c0 2024-01-26T23:45:20.160 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:20.160 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":84,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:45:19.186919+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:45:17.703325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-01-26T23:45:18.149282+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:45:20.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.161+0000 7f142795b700 1 -- 172.21.15.46:0/1730362020 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1404066f30 msgr2=0x7f14040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:20.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.161+0000 7f142795b700 1 --2- 172.21.15.46:0/1730362020 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1404066f30 0x7f14040693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f14200cc010 tx=0x7f1410009040 comp rx=0 tx=0).stop 2024-01-26T23:45:20.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.161+0000 7f142795b700 1 -- 172.21.15.46:0/1730362020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14200cc1c0 msgr2=0x7f14200c65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:20.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.161+0000 7f142795b700 1 --2- 172.21.15.46:0/1730362020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14200cc1c0 0x7f14200c65f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f141c00ec80 tx=0x7f141c009120 comp rx=0 tx=0).stop 2024-01-26T23:45:20.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.161+0000 7f142795b700 1 -- 172.21.15.46:0/1730362020 shutdown_connections 2024-01-26T23:45:20.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.161+0000 7f142795b700 1 --2- 172.21.15.46:0/1730362020 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1404066f30 0x7f14040693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.161+0000 7f142795b700 1 --2- 172.21.15.46:0/1730362020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14200cc1c0 0x7f14200c65f0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.161+0000 7f142795b700 1 --2- 172.21.15.46:0/1730362020 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f142015d210 0x7f14200c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.161+0000 7f142795b700 1 --2- 172.21.15.46:0/1730362020 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f142015ce40 0x7f142009d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.161+0000 7f142795b700 1 -- 172.21.15.46:0/1730362020 >> 172.21.15.46:0/1730362020 conn(0x7f142006f840 msgr2=0x7f14200b34f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:20.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.161+0000 7f142795b700 1 -- 172.21.15.46:0/1730362020 shutdown_connections 2024-01-26T23:45:20.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.161+0000 7f142795b700 1 -- 172.21.15.46:0/1730362020 wait complete. 2024-01-26T23:45:20.177 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:45:20.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.309+0000 7fa6e6ce5700 1 Processor -- start 2024-01-26T23:45:20.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.309+0000 7fa6e6ce5700 1 -- start start 2024-01-26T23:45:20.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.309+0000 7fa6e6ce5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6e00b40a0 0x7fa6e00b1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:20.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.310+0000 7fa6e6ce5700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e00aedf0 con 0x7fa6e01573f0 2024-01-26T23:45:20.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.310+0000 7fa6e6ce5700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e00af0d0 con 0x7fa6e00ade30 2024-01-26T23:45:20.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.310+0000 7fa6e6ce5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e00b3510 con 0x7fa6e00b40a0 2024-01-26T23:45:20.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.310+0000 7fa6e4a81700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa6e01573f0 0x7fa6e01577c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49126/0 (socket says 172.21.15.46:49126) 2024-01-26T23:45:20.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.310+0000 7fa6e4a81700 1 -- 172.21.15.46:0/2434581782 learned_addr learned my addr 172.21.15.46:0/2434581782 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:20.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.310+0000 7fa6dffff700 1 --2- 172.21.15.46:0/2434581782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6e00b40a0 0x7fa6e00b1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:20.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.310+0000 7fa6df7fe700 1 -- 172.21.15.46:0/2434581782 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943353330 0 0) 0x7fa6e00aedf0 con 0x7fa6e01573f0 2024-01-26T23:45:20.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.310+0000 7fa6df7fe700 1 -- 172.21.15.46:0/2434581782 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c8003570 con 0x7fa6e01573f0 2024-01-26T23:45:20.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.310+0000 7fa6df7fe700 1 -- 172.21.15.46:0/2434581782 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200710841 0 0) 0x7fa6e00af0d0 con 0x7fa6e00ade30 2024-01-26T23:45:20.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.310+0000 7fa6df7fe700 1 -- 172.21.15.46:0/2434581782 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6e00aedf0 con 0x7fa6e00ade30 2024-01-26T23:45:20.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.310+0000 7fa6dffff700 1 -- 172.21.15.46:0/2434581782 >> v1:172.21.15.106:6789/0 conn(0x7fa6e00ade30 legacy=0x7fa6e01612e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:20.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.310+0000 7fa6dffff700 1 -- 172.21.15.46:0/2434581782 >> v1:172.21.15.46:6789/0 conn(0x7fa6e01573f0 legacy=0x7fa6e01577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:20.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.310+0000 7fa6df7fe700 1 -- 172.21.15.46:0/2434581782 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2817701495 0 0) 0x7fa6c8003570 con 0x7fa6e01573f0 2024-01-26T23:45:20.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.310+0000 7fa6dffff700 1 -- 172.21.15.46:0/2434581782 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e00af3b0 con 0x7fa6e00b40a0 2024-01-26T23:45:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.311+0000 7fa6dffff700 1 --2- 172.21.15.46:0/2434581782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6e00b40a0 0x7fa6e00b1e60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4009900 tx=0x7fa6d4009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd2381109752753d server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.311+0000 7fa6df7fe700 1 -- 172.21.15.46:0/2434581782 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0001470 con 0x7fa6e00b40a0 2024-01-26T23:45:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.311+0000 7fa6df7fe700 1 -- 172.21.15.46:0/2434581782 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d4011420 con 0x7fa6e00b40a0 2024-01-26T23:45:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.311+0000 7fa6df7fe700 1 -- 172.21.15.46:0/2434581782 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d40117c0 con 0x7fa6e00b40a0 2024-01-26T23:45:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.311+0000 7fa6e6ce5700 1 -- 172.21.15.46:0/2434581782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6e00b40a0 msgr2=0x7fa6e00b1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.311+0000 7fa6e6ce5700 1 --2- 172.21.15.46:0/2434581782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6e00b40a0 0x7fa6e00b1e60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4009900 tx=0x7fa6d4009120 comp rx=0 tx=0).stop 2024-01-26T23:45:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.311+0000 7fa6e6ce5700 1 -- 172.21.15.46:0/2434581782 shutdown_connections 2024-01-26T23:45:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.311+0000 7fa6e6ce5700 1 --2- 172.21.15.46:0/2434581782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6e00b40a0 0x7fa6e00b1e60 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.311+0000 7fa6e6ce5700 1 -- 172.21.15.46:0/2434581782 >> 172.21.15.46:0/2434581782 conn(0x7fa6e0087f60 msgr2=0x7fa6e0088330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.311+0000 7fa6e6ce5700 1 -- 172.21.15.46:0/2434581782 shutdown_connections 2024-01-26T23:45:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.311+0000 7fa6e6ce5700 1 -- 172.21.15.46:0/2434581782 wait complete. 2024-01-26T23:45:20.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.313+0000 7fa6e6ce5700 1 Processor -- start 2024-01-26T23:45:20.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.313+0000 7fa6e6ce5700 1 -- start start 2024-01-26T23:45:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6e6ce5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6e00ade30 0x7fa6e0154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6e6ce5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e00b40a0 0x7fa6e00981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6e6ce5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6e01573f0 0x7fa6e009b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6e6ce5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e009b1d0 con 0x7fa6e00b40a0 2024-01-26T23:45:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6e6ce5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e00a96e0 con 0x7fa6e00ade30 2024-01-26T23:45:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6e6ce5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e009a930 con 0x7fa6e01573f0 2024-01-26T23:45:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6dffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e00b40a0 0x7fa6e00981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6dffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e00b40a0 0x7fa6e00981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44780/0 (socket says 172.21.15.46:44780) 2024-01-26T23:45:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6dffff700 1 -- 172.21.15.46:0/2667636242 learned_addr learned my addr 172.21.15.46:0/2667636242 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6e4a81700 1 --2- 172.21.15.46:0/2667636242 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6e00ade30 0x7fa6e0154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6e5282700 1 --2- 172.21.15.46:0/2667636242 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6e01573f0 0x7fa6e009b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6dffff700 1 -- 172.21.15.46:0/2667636242 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6e01573f0 msgr2=0x7fa6e009b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6dffff700 1 --2- 172.21.15.46:0/2667636242 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6e01573f0 0x7fa6e009b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6dffff700 1 -- 172.21.15.46:0/2667636242 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6e00ade30 msgr2=0x7fa6e0154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6dffff700 1 --2- 172.21.15.46:0/2667636242 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6e00ade30 0x7fa6e0154b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6dffff700 1 -- 172.21.15.46:0/2667636242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d40093c0 con 0x7fa6e00b40a0 2024-01-26T23:45:20.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6e4a81700 1 --2- 172.21.15.46:0/2667636242 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6e00ade30 0x7fa6e0154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:20.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.314+0000 7fa6dffff700 1 --2- 172.21.15.46:0/2667636242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e00b40a0 0x7fa6e00981a0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4009160 tx=0x7fa6d4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:20.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.315+0000 7fa6dcff9700 1 -- 172.21.15.46:0/2667636242 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d401cc90 con 0x7fa6e00b40a0 2024-01-26T23:45:20.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.315+0000 7fa6dcff9700 1 -- 172.21.15.46:0/2667636242 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d401cdf0 con 0x7fa6e00b40a0 2024-01-26T23:45:20.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.315+0000 7fa6dcff9700 1 -- 172.21.15.46:0/2667636242 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d401a6a0 con 0x7fa6e00b40a0 2024-01-26T23:45:20.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.315+0000 7fa6e6ce5700 1 -- 172.21.15.46:0/2667636242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e009ac10 con 0x7fa6e00b40a0 2024-01-26T23:45:20.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.315+0000 7fa6e6ce5700 1 -- 172.21.15.46:0/2667636242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e01550b0 con 0x7fa6e00b40a0 2024-01-26T23:45:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.316+0000 7fa6e6ce5700 1 -- 172.21.15.46:0/2667636242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e004c710 con 0x7fa6e00b40a0 2024-01-26T23:45:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.316+0000 7fa6dcff9700 1 -- 172.21.15.46:0/2667636242 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa6d401aa50 con 0x7fa6e00b40a0 2024-01-26T23:45:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.317+0000 7fa6dcff9700 1 --2- 172.21.15.46:0/2667636242 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa6cc066f30 0x7fa6cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.317+0000 7fa6dcff9700 1 -- 172.21.15.46:0/2667636242 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fa6d4086110 con 0x7fa6e00b40a0 2024-01-26T23:45:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.317+0000 7fa6e4a81700 1 --2- 172.21.15.46:0/2667636242 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa6cc066f30 0x7fa6cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.317+0000 7fa6e4a81700 1 --2- 172.21.15.46:0/2667636242 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa6cc066f30 0x7fa6cc0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00011e0 tx=0x7fa6d0000bc0 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:20.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.321+0000 7fa6dcff9700 1 -- 172.21.15.46:0/2667636242 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa6d40530b0 con 0x7fa6e00b40a0 2024-01-26T23:45:20.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.528+0000 7fa6e6ce5700 1 -- 172.21.15.46:0/2667636242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6e015a610 con 0x7fa6e00b40a0 2024-01-26T23:45:20.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.529+0000 7fa6dcff9700 1 -- 172.21.15.46:0/2667636242 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16164 (secure 0 0 0) 0x7fa6d40566d0 con 0x7fa6e00b40a0 2024-01-26T23:45:20.530 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:20.531 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":84,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:45:19.186919+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:45:17.703325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-01-26T23:45:18.149282+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:45:20.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.531+0000 7fa6e6ce5700 1 -- 172.21.15.46:0/2667636242 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa6cc066f30 msgr2=0x7fa6cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:20.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.531+0000 7fa6e6ce5700 1 --2- 172.21.15.46:0/2667636242 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa6cc066f30 0x7fa6cc0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00011e0 tx=0x7fa6d0000bc0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.531+0000 7fa6e6ce5700 1 -- 172.21.15.46:0/2667636242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e00b40a0 msgr2=0x7fa6e00981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:20.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.532+0000 7fa6e6ce5700 1 --2- 172.21.15.46:0/2667636242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e00b40a0 0x7fa6e00981a0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4009160 tx=0x7fa6d4016000 comp rx=0 tx=0).stop 2024-01-26T23:45:20.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.532+0000 7fa6e6ce5700 1 -- 172.21.15.46:0/2667636242 shutdown_connections 2024-01-26T23:45:20.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.532+0000 7fa6e6ce5700 1 --2- 172.21.15.46:0/2667636242 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa6cc066f30 0x7fa6cc0693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.532+0000 7fa6e6ce5700 1 --2- 172.21.15.46:0/2667636242 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6e01573f0 0x7fa6e009b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.532+0000 7fa6e6ce5700 1 --2- 172.21.15.46:0/2667636242 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6e00ade30 0x7fa6e0154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.532+0000 7fa6e6ce5700 1 --2- 172.21.15.46:0/2667636242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e00b40a0 0x7fa6e00981a0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.532+0000 7fa6e6ce5700 1 -- 172.21.15.46:0/2667636242 >> 172.21.15.46:0/2667636242 conn(0x7fa6e0087f60 msgr2=0x7fa6e0156a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:20.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.532+0000 7fa6e6ce5700 1 -- 172.21.15.46:0/2667636242 shutdown_connections 2024-01-26T23:45:20.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.532+0000 7fa6e6ce5700 1 -- 172.21.15.46:0/2667636242 wait complete. 2024-01-26T23:45:20.547 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-26T23:45:20.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.681+0000 7fe6b5db0700 1 Processor -- start 2024-01-26T23:45:20.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.681+0000 7fe6b5db0700 1 -- start start 2024-01-26T23:45:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.682+0000 7fe6b5db0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe6b00c93a0 0x7fe6b00c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.682+0000 7fe6b5db0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6b0067d50 0x7fe6b0159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.682+0000 7fe6b5db0700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe6b00681c0 0x7fe6b0159760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.682+0000 7fe6b5db0700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b015e390 con 0x7fe6b00c93a0 2024-01-26T23:45:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.682+0000 7fe6b5db0700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b0159ca0 con 0x7fe6b00681c0 2024-01-26T23:45:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.682+0000 7fe6b5db0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b015d810 con 0x7fe6b0067d50 2024-01-26T23:45:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.682+0000 7fe6af7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe6b00c93a0 0x7fe6b00c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.682+0000 7fe6af7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe6b00c93a0 0x7fe6b00c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37132/0 (socket says 172.21.15.46:37132) 2024-01-26T23:45:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.682+0000 7fe6af7fe700 1 -- 172.21.15.46:0/3556014075 learned_addr learned my addr 172.21.15.46:0/3556014075 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.682+0000 7fe6affff700 1 --2- 172.21.15.46:0/3556014075 >> v2:172.21.15.106:3300/0 conn(0x7fe6b00681c0 0x7fe6b0159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.682+0000 7fe6aeffd700 1 --2- 172.21.15.46:0/3556014075 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6b0067d50 0x7fe6b0159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.682+0000 7fe6af7fe700 1 -- 172.21.15.46:0/3556014075 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6b0067d50 msgr2=0x7fe6b0159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.682+0000 7fe6af7fe700 1 --2- 172.21.15.46:0/3556014075 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6b0067d50 0x7fe6b0159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.682+0000 7fe6af7fe700 1 -- 172.21.15.46:0/3556014075 >> v2:172.21.15.106:3300/0 conn(0x7fe6b00681c0 msgr2=0x7fe6b0159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.682+0000 7fe6af7fe700 1 --2- 172.21.15.46:0/3556014075 >> v2:172.21.15.106:3300/0 conn(0x7fe6b00681c0 0x7fe6b0159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.682+0000 7fe6af7fe700 1 -- 172.21.15.46:0/3556014075 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b015daf0 con 0x7fe6b00c93a0 2024-01-26T23:45:20.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.682+0000 7fe6af7fe700 1 --2- 172.21.15.46:0/3556014075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b00c93a0 0x7fe6b00c9770 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fe6b00c7a30 tx=0x7fe698009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a81340f36fd01a7b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:20.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.683+0000 7fe6ae7fc700 1 -- 172.21.15.46:0/3556014075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69800b640 con 0x7fe6b00c93a0 2024-01-26T23:45:20.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.683+0000 7fe6ae7fc700 1 -- 172.21.15.46:0/3556014075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69800b7a0 con 0x7fe6b00c93a0 2024-01-26T23:45:20.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.683+0000 7fe6ae7fc700 1 -- 172.21.15.46:0/3556014075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69800bb40 con 0x7fe6b00c93a0 2024-01-26T23:45:20.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.683+0000 7fe6b5db0700 1 -- 172.21.15.46:0/3556014075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b00c93a0 msgr2=0x7fe6b00c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:20.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.683+0000 7fe6b5db0700 1 --2- 172.21.15.46:0/3556014075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b00c93a0 0x7fe6b00c9770 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fe6b00c7a30 tx=0x7fe698009250 comp rx=0 tx=0).stop 2024-01-26T23:45:20.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.683+0000 7fe6b5db0700 1 -- 172.21.15.46:0/3556014075 shutdown_connections 2024-01-26T23:45:20.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.683+0000 7fe6b5db0700 1 --2- 172.21.15.46:0/3556014075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b00c93a0 0x7fe6b00c9770 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.683+0000 7fe6b5db0700 1 --2- 172.21.15.46:0/3556014075 >> v2:172.21.15.106:3300/0 conn(0x7fe6b00681c0 0x7fe6b0159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.683+0000 7fe6b5db0700 1 --2- 172.21.15.46:0/3556014075 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6b0067d50 0x7fe6b0159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.683+0000 7fe6b5db0700 1 -- 172.21.15.46:0/3556014075 >> 172.21.15.46:0/3556014075 conn(0x7fe6b00ca9a0 msgr2=0x7fe6b0068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:20.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.683+0000 7fe6b5db0700 1 -- 172.21.15.46:0/3556014075 shutdown_connections 2024-01-26T23:45:20.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.683+0000 7fe6b5db0700 1 -- 172.21.15.46:0/3556014075 wait complete. 2024-01-26T23:45:20.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.685+0000 7fe6b5db0700 1 Processor -- start 2024-01-26T23:45:20.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.685+0000 7fe6b5db0700 1 -- start start 2024-01-26T23:45:20.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.685+0000 7fe6b5db0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6b0067d50 0x7fe6b014d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:20.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.685+0000 7fe6b5db0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b00681c0 0x7fe6b00c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:20.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.685+0000 7fe6b5db0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6b00c93a0 0x7fe6b00c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:20.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.685+0000 7fe6b5db0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b00b1160 con 0x7fe6b00681c0 2024-01-26T23:45:20.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.685+0000 7fe6b5db0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b00b0cd0 con 0x7fe6b0067d50 2024-01-26T23:45:20.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.685+0000 7fe6b5db0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b014d7f0 con 0x7fe6b00c93a0 2024-01-26T23:45:20.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.685+0000 7fe6aeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b00681c0 0x7fe6b00c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:20.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.686+0000 7fe6aeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b00681c0 0x7fe6b00c6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37136/0 (socket says 172.21.15.46:37136) 2024-01-26T23:45:20.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.686+0000 7fe6aeffd700 1 -- 172.21.15.46:0/1248140693 learned_addr learned my addr 172.21.15.46:0/1248140693 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:20.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.686+0000 7fe6af7fe700 1 --2- 172.21.15.46:0/1248140693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6b0067d50 0x7fe6b014d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:20.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.686+0000 7fe6affff700 1 --2- 172.21.15.46:0/1248140693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6b00c93a0 0x7fe6b00c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:20.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.686+0000 7fe6aeffd700 1 -- 172.21.15.46:0/1248140693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6b00c93a0 msgr2=0x7fe6b00c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:20.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.686+0000 7fe6aeffd700 1 --2- 172.21.15.46:0/1248140693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6b00c93a0 0x7fe6b00c84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.686+0000 7fe6aeffd700 1 -- 172.21.15.46:0/1248140693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6b0067d50 msgr2=0x7fe6b014d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:20.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.686+0000 7fe6aeffd700 1 --2- 172.21.15.46:0/1248140693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6b0067d50 0x7fe6b014d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:20.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.686+0000 7fe6aeffd700 1 -- 172.21.15.46:0/1248140693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6980094f0 con 0x7fe6b00681c0 2024-01-26T23:45:20.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.686+0000 7fe6aeffd700 1 --2- 172.21.15.46:0/1248140693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b00681c0 0x7fe6b00c6ab0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000e960 tx=0x7fe6a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:20.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.686+0000 7fe6a77fe700 1 -- 172.21.15.46:0/1248140693 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0013680 con 0x7fe6b00681c0 2024-01-26T23:45:20.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.686+0000 7fe6a77fe700 1 -- 172.21.15.46:0/1248140693 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a00137e0 con 0x7fe6b00681c0 2024-01-26T23:45:20.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.686+0000 7fe6b5db0700 1 -- 172.21.15.46:0/1248140693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b00b03a0 con 0x7fe6b00681c0 2024-01-26T23:45:20.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.686+0000 7fe6a77fe700 1 -- 172.21.15.46:0/1248140693 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0013b80 con 0x7fe6b00681c0 2024-01-26T23:45:20.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.686+0000 7fe6b5db0700 1 -- 172.21.15.46:0/1248140693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b00b0960 con 0x7fe6b00681c0 2024-01-26T23:45:20.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.688+0000 7fe6a77fe700 1 -- 172.21.15.46:0/1248140693 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe6a001c450 con 0x7fe6b00681c0 2024-01-26T23:45:20.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.688+0000 7fe6b5db0700 1 -- 172.21.15.46:0/1248140693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b00b0e80 con 0x7fe6b00681c0 2024-01-26T23:45:20.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.688+0000 7fe6a77fe700 1 --2- 172.21.15.46:0/1248140693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe690066f30 0x7fe6900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:20.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.688+0000 7fe6a77fe700 1 -- 172.21.15.46:0/1248140693 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fe6a0084900 con 0x7fe6b00681c0 2024-01-26T23:45:20.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.688+0000 7fe6af7fe700 1 --2- 172.21.15.46:0/1248140693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe690066f30 0x7fe6900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:20.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.689+0000 7fe6af7fe700 1 --2- 172.21.15.46:0/1248140693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe690066f30 0x7fe6900693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe6b00c7a30 tx=0x7fe698017000 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:20.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.693+0000 7fe6a77fe700 1 -- 172.21.15.46:0/1248140693 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe6b00b0e80 con 0x7fe6b00681c0 2024-01-26T23:45:20.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:20.945+0000 7fe6b5db0700 1 -- 172.21.15.46:0/1248140693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe6b00b0e40 con 0x7fe6b00681c0 2024-01-26T23:45:21.739 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-01-26T23:45:21.737+0000 7f3a3bca0700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-26T23:45:21.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.778+0000 7fe6a77fe700 1 -- 172.21.15.46:0/1248140693 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v35) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe6a0051800 con 0x7fe6b00681c0 2024-01-26T23:45:21.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.781+0000 7fe6b5db0700 1 -- 172.21.15.46:0/1248140693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe690066f30 msgr2=0x7fe6900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:21.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.781+0000 7fe6b5db0700 1 --2- 172.21.15.46:0/1248140693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe690066f30 0x7fe6900693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe6b00c7a30 tx=0x7fe698017000 comp rx=0 tx=0).stop 2024-01-26T23:45:21.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.781+0000 7fe6b5db0700 1 -- 172.21.15.46:0/1248140693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b00681c0 msgr2=0x7fe6b00c6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:21.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.781+0000 7fe6b5db0700 1 --2- 172.21.15.46:0/1248140693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b00681c0 0x7fe6b00c6ab0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000e960 tx=0x7fe6a00091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:21.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.781+0000 7fe6b5db0700 1 -- 172.21.15.46:0/1248140693 shutdown_connections 2024-01-26T23:45:21.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.781+0000 7fe6b5db0700 1 --2- 172.21.15.46:0/1248140693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe690066f30 0x7fe6900693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:21.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.781+0000 7fe6b5db0700 1 --2- 172.21.15.46:0/1248140693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6b00c93a0 0x7fe6b00c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:21.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.781+0000 7fe6b5db0700 1 --2- 172.21.15.46:0/1248140693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6b0067d50 0x7fe6b014d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:21.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.781+0000 7fe6b5db0700 1 --2- 172.21.15.46:0/1248140693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b00681c0 0x7fe6b00c6ab0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:21.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.781+0000 7fe6b5db0700 1 -- 172.21.15.46:0/1248140693 >> 172.21.15.46:0/1248140693 conn(0x7fe6b00ca9a0 msgr2=0x7fe6b006c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:21.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.781+0000 7fe6b5db0700 1 -- 172.21.15.46:0/1248140693 shutdown_connections 2024-01-26T23:45:21.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.781+0000 7fe6b5db0700 1 -- 172.21.15.46:0/1248140693 wait complete. 2024-01-26T23:45:21.798 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-26T23:45:21.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.935+0000 7f1ab7cb3700 1 Processor -- start 2024-01-26T23:45:21.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.936+0000 7f1ab7cb3700 1 -- start start 2024-01-26T23:45:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.936+0000 7f1ab7cb3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ab0157490 0x7f1ab00cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.936+0000 7f1ab7cb3700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab00cea80 con 0x7f1ab0157860 2024-01-26T23:45:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.936+0000 7f1ab7cb3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab00cde70 con 0x7f1ab006ddb0 2024-01-26T23:45:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.936+0000 7f1ab7cb3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab00cdc60 con 0x7f1ab0157490 2024-01-26T23:45:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.937+0000 7f1ab5a4f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ab0157490 0x7f1ab00cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.937+0000 7f1ab524e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1ab0157860 0x7f1ab006d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43330/0 (socket says 172.21.15.46:43330) 2024-01-26T23:45:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.937+0000 7f1ab524e700 1 -- 172.21.15.46:0/89115879 learned_addr learned my addr 172.21.15.46:0/89115879 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.937+0000 7f1ab5a4f700 1 -- 172.21.15.46:0/89115879 >> v1:172.21.15.106:6789/0 conn(0x7f1ab006ddb0 legacy=0x7f1ab00b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.937+0000 7f1ab5a4f700 1 -- 172.21.15.46:0/89115879 >> v1:172.21.15.46:6789/0 conn(0x7f1ab0157860 legacy=0x7f1ab006d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.937+0000 7f1ab5a4f700 1 -- 172.21.15.46:0/89115879 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab00ce150 con 0x7f1ab0157490 2024-01-26T23:45:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.937+0000 7f1ab4a4d700 1 -- 172.21.15.46:0/89115879 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291189447 0 0) 0x7f1ab00cea80 con 0x7f1ab0157860 2024-01-26T23:45:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.937+0000 7f1ab5a4f700 1 --2- 172.21.15.46:0/89115879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ab0157490 0x7f1ab00cc600 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00098e0 tx=0x7f1aa00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df873e54d225e6c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.937+0000 7f1ab4a4d700 1 -- 172.21.15.46:0/89115879 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0009e80 con 0x7f1ab0157490 2024-01-26T23:45:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.937+0000 7f1ab4a4d700 1 -- 172.21.15.46:0/89115879 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa000b860 con 0x7f1ab0157490 2024-01-26T23:45:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.937+0000 7f1ab4a4d700 1 -- 172.21.15.46:0/89115879 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa000bbe0 con 0x7f1ab0157490 2024-01-26T23:45:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.938+0000 7f1ab7cb3700 1 -- 172.21.15.46:0/89115879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ab0157490 msgr2=0x7f1ab00cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.938+0000 7f1ab7cb3700 1 --2- 172.21.15.46:0/89115879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ab0157490 0x7f1ab00cc600 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00098e0 tx=0x7f1aa00091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.938+0000 7f1ab7cb3700 1 -- 172.21.15.46:0/89115879 shutdown_connections 2024-01-26T23:45:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.938+0000 7f1ab7cb3700 1 --2- 172.21.15.46:0/89115879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ab0157490 0x7f1ab00cc600 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.938+0000 7f1ab7cb3700 1 -- 172.21.15.46:0/89115879 >> 172.21.15.46:0/89115879 conn(0x7f1ab0067930 msgr2=0x7f1ab0067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.938+0000 7f1ab7cb3700 1 -- 172.21.15.46:0/89115879 shutdown_connections 2024-01-26T23:45:21.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.938+0000 7f1ab7cb3700 1 -- 172.21.15.46:0/89115879 wait complete. 2024-01-26T23:45:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.939+0000 7f1ab7cb3700 1 Processor -- start 2024-01-26T23:45:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.939+0000 7f1ab7cb3700 1 -- start start 2024-01-26T23:45:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.940+0000 7f1ab7cb3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ab006ddb0 0x7f1ab01679b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.940+0000 7f1ab7cb3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ab0157490 0x7f1ab014fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.940+0000 7f1ab7cb3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ab0157860 0x7f1ab00b0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.940+0000 7f1ab7cb3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab016bd50 con 0x7f1ab0157860 2024-01-26T23:45:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.940+0000 7f1ab7cb3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab016b180 con 0x7f1ab0157490 2024-01-26T23:45:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.940+0000 7f1ab7cb3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab016b420 con 0x7f1ab006ddb0 2024-01-26T23:45:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.940+0000 7f1ab6250700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ab0157860 0x7f1ab00b0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.940+0000 7f1ab6250700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ab0157860 0x7f1ab00b0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37162/0 (socket says 172.21.15.46:37162) 2024-01-26T23:45:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.940+0000 7f1ab6250700 1 -- 172.21.15.46:0/3669798900 learned_addr learned my addr 172.21.15.46:0/3669798900 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.941+0000 7f1ab524e700 1 --2- 172.21.15.46:0/3669798900 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ab0157490 0x7f1ab014fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.941+0000 7f1ab5a4f700 1 --2- 172.21.15.46:0/3669798900 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ab006ddb0 0x7f1ab01679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.941+0000 7f1ab6250700 1 -- 172.21.15.46:0/3669798900 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ab006ddb0 msgr2=0x7f1ab01679b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.941+0000 7f1ab6250700 1 --2- 172.21.15.46:0/3669798900 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ab006ddb0 0x7f1ab01679b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.941+0000 7f1ab6250700 1 -- 172.21.15.46:0/3669798900 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ab0157490 msgr2=0x7f1ab014fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.941+0000 7f1ab6250700 1 --2- 172.21.15.46:0/3669798900 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ab0157490 0x7f1ab014fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.941+0000 7f1ab6250700 1 -- 172.21.15.46:0/3669798900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa0009420 con 0x7f1ab0157860 2024-01-26T23:45:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.941+0000 7f1ab5a4f700 1 --2- 172.21.15.46:0/3669798900 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ab006ddb0 0x7f1ab01679b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.941+0000 7f1ab6250700 1 --2- 172.21.15.46:0/3669798900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ab0157860 0x7f1ab00b0420 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800c910 tx=0x7f1aa80016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.941+0000 7f1aa5ffb700 1 -- 172.21.15.46:0/3669798900 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8012630 con 0x7f1ab0157860 2024-01-26T23:45:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.941+0000 7f1aa5ffb700 1 -- 172.21.15.46:0/3669798900 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa80127b0 con 0x7f1ab0157860 2024-01-26T23:45:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.941+0000 7f1aa5ffb700 1 -- 172.21.15.46:0/3669798900 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8012b70 con 0x7f1ab0157860 2024-01-26T23:45:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.941+0000 7f1ab7cb3700 1 -- 172.21.15.46:0/3669798900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab016b700 con 0x7f1ab0157860 2024-01-26T23:45:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.941+0000 7f1ab7cb3700 1 -- 172.21.15.46:0/3669798900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab0150170 con 0x7f1ab0157860 2024-01-26T23:45:21.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.943+0000 7f1ab7cb3700 1 -- 172.21.15.46:0/3669798900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab004c710 con 0x7f1ab0157860 2024-01-26T23:45:21.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.943+0000 7f1aa5ffb700 1 -- 172.21.15.46:0/3669798900 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1aa800a3b0 con 0x7f1ab0157860 2024-01-26T23:45:21.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.943+0000 7f1aa5ffb700 1 --2- 172.21.15.46:0/3669798900 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1a94066f30 0x7f1a940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:21.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.943+0000 7f1aa5ffb700 1 -- 172.21.15.46:0/3669798900 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f1aa8084af0 con 0x7f1ab0157860 2024-01-26T23:45:21.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.944+0000 7f1ab5a4f700 1 --2- 172.21.15.46:0/3669798900 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1a94066f30 0x7f1a940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:21.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.944+0000 7f1ab5a4f700 1 --2- 172.21.15.46:0/3669798900 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1a94066f30 0x7f1a940693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00093f0 tx=0x7f1aa0017000 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:21.948+0000 7f1aa5ffb700 1 -- 172.21.15.46:0/3669798900 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1aa80519f0 con 0x7f1ab0157860 2024-01-26T23:45:22.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.202+0000 7f1ab7cb3700 1 -- 172.21.15.46:0/3669798900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1ab016be90 con 0x7f1ab0157860 2024-01-26T23:45:22.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.780+0000 7f1aa5ffb700 1 -- 172.21.15.46:0/3669798900 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1ab016be90 con 0x7f1ab0157860 2024-01-26T23:45:22.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.783+0000 7f1ab7cb3700 1 -- 172.21.15.46:0/3669798900 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1a94066f30 msgr2=0x7f1a940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:22.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.783+0000 7f1ab7cb3700 1 --2- 172.21.15.46:0/3669798900 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1a94066f30 0x7f1a940693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00093f0 tx=0x7f1aa0017000 comp rx=0 tx=0).stop 2024-01-26T23:45:22.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.783+0000 7f1ab7cb3700 1 -- 172.21.15.46:0/3669798900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ab0157860 msgr2=0x7f1ab00b0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:22.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.783+0000 7f1ab7cb3700 1 --2- 172.21.15.46:0/3669798900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ab0157860 0x7f1ab00b0420 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800c910 tx=0x7f1aa80016f0 comp rx=0 tx=0).stop 2024-01-26T23:45:22.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.783+0000 7f1ab7cb3700 1 -- 172.21.15.46:0/3669798900 shutdown_connections 2024-01-26T23:45:22.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.783+0000 7f1ab7cb3700 1 --2- 172.21.15.46:0/3669798900 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1a94066f30 0x7f1a940693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:22.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.783+0000 7f1ab7cb3700 1 --2- 172.21.15.46:0/3669798900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ab0157860 0x7f1ab00b0420 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:22.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.783+0000 7f1ab7cb3700 1 --2- 172.21.15.46:0/3669798900 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ab006ddb0 0x7f1ab01679b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:22.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.783+0000 7f1ab7cb3700 1 --2- 172.21.15.46:0/3669798900 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ab0157490 0x7f1ab014fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:22.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.783+0000 7f1ab7cb3700 1 -- 172.21.15.46:0/3669798900 >> 172.21.15.46:0/3669798900 conn(0x7f1ab0067930 msgr2=0x7f1ab006d600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:22.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.783+0000 7f1ab7cb3700 1 -- 172.21.15.46:0/3669798900 shutdown_connections 2024-01-26T23:45:22.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.783+0000 7f1ab7cb3700 1 -- 172.21.15.46:0/3669798900 wait complete. 2024-01-26T23:45:22.800 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:45:22.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.935+0000 7f138dd8b700 1 Processor -- start 2024-01-26T23:45:22.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.935+0000 7f138dd8b700 1 -- start start 2024-01-26T23:45:22.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.936+0000 7f138dd8b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f13880cb3e0 0x7f1388159e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.936+0000 7f138dd8b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13880d15c0 con 0x7f1388159a30 2024-01-26T23:45:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.936+0000 7f138dd8b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13880cfcd0 con 0x7f13880d0e40 2024-01-26T23:45:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.936+0000 7f138dd8b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f138806fe70 con 0x7f13880cb3e0 2024-01-26T23:45:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.936+0000 7f13877fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1388159a30 0x7f13880cacd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43340/0 (socket says 172.21.15.46:43340) 2024-01-26T23:45:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.936+0000 7f13877fe700 1 -- 172.21.15.46:0/4196211497 learned_addr learned my addr 172.21.15.46:0/4196211497 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.937+0000 7f1387fff700 1 --2- 172.21.15.46:0/4196211497 >> v2:172.21.15.106:3300/0 conn(0x7f13880cb3e0 0x7f1388159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.937+0000 7f13867fc700 1 -- 172.21.15.46:0/4196211497 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045158737 0 0) 0x7f13880d15c0 con 0x7f1388159a30 2024-01-26T23:45:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.937+0000 7f13867fc700 1 -- 172.21.15.46:0/4196211497 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1370003570 con 0x7f1388159a30 2024-01-26T23:45:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.937+0000 7f1387fff700 1 -- 172.21.15.46:0/4196211497 >> v1:172.21.15.106:6789/0 conn(0x7f13880d0e40 legacy=0x7f13880ad7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.937+0000 7f1387fff700 1 -- 172.21.15.46:0/4196211497 >> v1:172.21.15.46:6789/0 conn(0x7f1388159a30 legacy=0x7f13880cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.937+0000 7f1387fff700 1 -- 172.21.15.46:0/4196211497 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13880cf3a0 con 0x7f13880cb3e0 2024-01-26T23:45:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.937+0000 7f1387fff700 1 --2- 172.21.15.46:0/4196211497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13880cb3e0 0x7f1388159e00 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1380009e60 tx=0x7f1380009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba17a79de59f19bf server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.937+0000 7f13867fc700 1 -- 172.21.15.46:0/4196211497 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138000e070 con 0x7f13880cb3e0 2024-01-26T23:45:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.937+0000 7f13867fc700 1 -- 172.21.15.46:0/4196211497 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138000bb60 con 0x7f13880cb3e0 2024-01-26T23:45:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.937+0000 7f13867fc700 1 -- 172.21.15.46:0/4196211497 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1380012430 con 0x7f13880cb3e0 2024-01-26T23:45:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.938+0000 7f138dd8b700 1 -- 172.21.15.46:0/4196211497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13880cb3e0 msgr2=0x7f1388159e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.938+0000 7f138dd8b700 1 --2- 172.21.15.46:0/4196211497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13880cb3e0 0x7f1388159e00 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1380009e60 tx=0x7f1380009780 comp rx=0 tx=0).stop 2024-01-26T23:45:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.938+0000 7f138dd8b700 1 -- 172.21.15.46:0/4196211497 shutdown_connections 2024-01-26T23:45:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.938+0000 7f138dd8b700 1 --2- 172.21.15.46:0/4196211497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13880cb3e0 0x7f1388159e00 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.938+0000 7f138dd8b700 1 -- 172.21.15.46:0/4196211497 >> 172.21.15.46:0/4196211497 conn(0x7f138806f840 msgr2=0x7f13880d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.938+0000 7f138dd8b700 1 -- 172.21.15.46:0/4196211497 shutdown_connections 2024-01-26T23:45:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.938+0000 7f138dd8b700 1 -- 172.21.15.46:0/4196211497 wait complete. 2024-01-26T23:45:22.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.939+0000 7f138dd8b700 1 Processor -- start 2024-01-26T23:45:22.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.939+0000 7f138dd8b700 1 -- start start 2024-01-26T23:45:22.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.940+0000 7f138dd8b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13880cb3e0 0x7f1388154e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:22.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.940+0000 7f138dd8b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13880d0e40 0x7f13880c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:22.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.940+0000 7f138dd8b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1388159a30 0x7f1388069ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:22.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.940+0000 7f138dd8b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1388169c20 con 0x7f13880cb3e0 2024-01-26T23:45:22.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.940+0000 7f138dd8b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138815e5c0 con 0x7f13880d0e40 2024-01-26T23:45:22.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.940+0000 7f138dd8b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13880ca2c0 con 0x7f1388159a30 2024-01-26T23:45:22.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.940+0000 7f13877fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13880cb3e0 0x7f1388154e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:22.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.940+0000 7f13877fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13880cb3e0 0x7f1388154e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37174/0 (socket says 172.21.15.46:37174) 2024-01-26T23:45:22.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.940+0000 7f13877fe700 1 -- 172.21.15.46:0/3699693556 learned_addr learned my addr 172.21.15.46:0/3699693556 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:22.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.940+0000 7f1387fff700 1 --2- 172.21.15.46:0/3699693556 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1388159a30 0x7f1388069ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:22.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.940+0000 7f1386ffd700 1 --2- 172.21.15.46:0/3699693556 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13880d0e40 0x7f13880c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:22.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.941+0000 7f13877fe700 1 -- 172.21.15.46:0/3699693556 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1388159a30 msgr2=0x7f1388069ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:22.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.941+0000 7f13877fe700 1 --2- 172.21.15.46:0/3699693556 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1388159a30 0x7f1388069ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:22.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.941+0000 7f13877fe700 1 -- 172.21.15.46:0/3699693556 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13880d0e40 msgr2=0x7f13880c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:22.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.941+0000 7f13877fe700 1 --2- 172.21.15.46:0/3699693556 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13880d0e40 0x7f13880c96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:22.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.941+0000 7f13877fe700 1 -- 172.21.15.46:0/3699693556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1380009a20 con 0x7f13880cb3e0 2024-01-26T23:45:22.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.941+0000 7f13877fe700 1 --2- 172.21.15.46:0/3699693556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13880cb3e0 0x7f1388154e00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f137800c790 tx=0x7f1378007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:22.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.941+0000 7f136f7fe700 1 -- 172.21.15.46:0/3699693556 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13780072e0 con 0x7f13880cb3e0 2024-01-26T23:45:22.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.941+0000 7f136f7fe700 1 -- 172.21.15.46:0/3699693556 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1378007440 con 0x7f13880cb3e0 2024-01-26T23:45:22.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.941+0000 7f138dd8b700 1 -- 172.21.15.46:0/3699693556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f138815dfb0 con 0x7f13880cb3e0 2024-01-26T23:45:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.941+0000 7f136f7fe700 1 -- 172.21.15.46:0/3699693556 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13780057f0 con 0x7f13880cb3e0 2024-01-26T23:45:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.941+0000 7f138dd8b700 1 -- 172.21.15.46:0/3699693556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f138815e250 con 0x7f13880cb3e0 2024-01-26T23:45:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.942+0000 7f138dd8b700 1 -- 172.21.15.46:0/3699693556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f138804c710 con 0x7f13880cb3e0 2024-01-26T23:45:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.943+0000 7f136f7fe700 1 -- 172.21.15.46:0/3699693556 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1378005be0 con 0x7f13880cb3e0 2024-01-26T23:45:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.943+0000 7f136f7fe700 1 --2- 172.21.15.46:0/3699693556 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1374066f30 0x7f13740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:22.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.944+0000 7f136f7fe700 1 -- 172.21.15.46:0/3699693556 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f1378083fe0 con 0x7f13880cb3e0 2024-01-26T23:45:22.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.944+0000 7f1386ffd700 1 --2- 172.21.15.46:0/3699693556 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1374066f30 0x7f13740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:22.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.944+0000 7f1386ffd700 1 --2- 172.21.15.46:0/3699693556 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1374066f30 0x7f13740693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f13880b3200 tx=0x7f137c000bc0 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:22.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:22.947+0000 7f136f7fe700 1 -- 172.21.15.46:0/3699693556 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1378051050 con 0x7f13880cb3e0 2024-01-26T23:45:23.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.204+0000 7f138dd8b700 1 -- 172.21.15.46:0/3699693556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13881553a0 con 0x7f13880cb3e0 2024-01-26T23:45:23.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.204+0000 7f136f7fe700 1 -- 172.21.15.46:0/3699693556 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f1378054670 con 0x7f13880cb3e0 2024-01-26T23:45:23.205 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:23.206 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/4083796808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":4083796808},{"type":"v1","addr":"172.21.15.106:6836","nonce":4083796808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":6172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/1769644590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1769644590},{"type":"v1","addr":"172.21.15.106:6837","nonce":1769644590}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":6184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1994423361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1994423361},{"type":"v1","addr":"172.21.15.46:6835","nonce":1994423361}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:45:19.187256+0000","modified":"2024-01-26T23:45:22.777939+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6366":{"gid":6366,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/1799649681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1799649681},{"type":"v1","addr":"172.21.15.46:6834","nonce":1799649681}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2024-01-26T23:45:23.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.206+0000 7f138dd8b700 1 -- 172.21.15.46:0/3699693556 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1374066f30 msgr2=0x7f13740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:23.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.206+0000 7f138dd8b700 1 --2- 172.21.15.46:0/3699693556 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1374066f30 0x7f13740693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f13880b3200 tx=0x7f137c000bc0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.206+0000 7f138dd8b700 1 -- 172.21.15.46:0/3699693556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13880cb3e0 msgr2=0x7f1388154e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:23.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.206+0000 7f138dd8b700 1 --2- 172.21.15.46:0/3699693556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13880cb3e0 0x7f1388154e00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f137800c790 tx=0x7f1378007c80 comp rx=0 tx=0).stop 2024-01-26T23:45:23.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.206+0000 7f138dd8b700 1 -- 172.21.15.46:0/3699693556 shutdown_connections 2024-01-26T23:45:23.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.206+0000 7f138dd8b700 1 --2- 172.21.15.46:0/3699693556 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1374066f30 0x7f13740693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.206+0000 7f138dd8b700 1 --2- 172.21.15.46:0/3699693556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13880cb3e0 0x7f1388154e00 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.206+0000 7f138dd8b700 1 --2- 172.21.15.46:0/3699693556 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1388159a30 0x7f1388069ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.206+0000 7f138dd8b700 1 --2- 172.21.15.46:0/3699693556 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13880d0e40 0x7f13880c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.206+0000 7f138dd8b700 1 -- 172.21.15.46:0/3699693556 >> 172.21.15.46:0/3699693556 conn(0x7f138806f840 msgr2=0x7f1388155e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:23.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.206+0000 7f138dd8b700 1 -- 172.21.15.46:0/3699693556 shutdown_connections 2024-01-26T23:45:23.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.206+0000 7f138dd8b700 1 -- 172.21.15.46:0/3699693556 wait complete. 2024-01-26T23:45:23.209 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 36 2024-01-26T23:45:23.221 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:45:23.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.351+0000 7f7a9928e700 1 Processor -- start 2024-01-26T23:45:23.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.351+0000 7f7a9928e700 1 -- start start 2024-01-26T23:45:23.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.352+0000 7f7a9928e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7a94159960 0x7f7a9406bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:23.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.352+0000 7f7a9928e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7a9406c1f0 0x7f7a94069370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:23.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.352+0000 7f7a9928e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9415dac0 con 0x7f7a94159590 2024-01-26T23:45:23.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.352+0000 7f7a9928e700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a9415e350 con 0x7f7a9406c1f0 2024-01-26T23:45:23.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.352+0000 7f7a9928e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a940c97c0 con 0x7f7a94159960 2024-01-26T23:45:23.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.353+0000 7f7a927fc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7a94159960 0x7f7a9406bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:23.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.353+0000 7f7a937fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7a9406c1f0 0x7f7a94069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:23.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.353+0000 7f7a927fc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7a94159960 0x7f7a9406bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:41552/0 (socket says 172.21.15.46:41552) 2024-01-26T23:45:23.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.353+0000 7f7a927fc700 1 -- 172.21.15.46:0/2551177071 learned_addr learned my addr 172.21.15.46:0/2551177071 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:23.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.353+0000 7f7a927fc700 1 -- 172.21.15.46:0/2551177071 >> v2:172.21.15.106:3300/0 conn(0x7f7a9406c1f0 msgr2=0x7f7a94069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:23.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.353+0000 7f7a927fc700 1 --2- 172.21.15.46:0/2551177071 >> v2:172.21.15.106:3300/0 conn(0x7f7a9406c1f0 0x7f7a94069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.353+0000 7f7a927fc700 1 -- 172.21.15.46:0/2551177071 >> v1:172.21.15.106:6789/0 conn(0x7f7a94159590 legacy=0x7f7a9415d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:23.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.353+0000 7f7a927fc700 1 -- 172.21.15.46:0/2551177071 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9415dd90 con 0x7f7a94159960 2024-01-26T23:45:23.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.353+0000 7f7a937fe700 1 --2- 172.21.15.46:0/2551177071 >> v2:172.21.15.106:3300/0 conn(0x7f7a9406c1f0 0x7f7a94069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:23.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.353+0000 7f7a927fc700 1 --2- 172.21.15.46:0/2551177071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7a94159960 0x7f7a9406bcb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c0099b0 tx=0x7f7a7c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1609262d0d2e6843 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:23.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.353+0000 7f7a91ffb700 1 -- 172.21.15.46:0/2551177071 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c009ea0 con 0x7f7a94159960 2024-01-26T23:45:23.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.354+0000 7f7a91ffb700 1 -- 172.21.15.46:0/2551177071 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7c00b860 con 0x7f7a94159960 2024-01-26T23:45:23.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.354+0000 7f7a91ffb700 1 -- 172.21.15.46:0/2551177071 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c00bc20 con 0x7f7a94159960 2024-01-26T23:45:23.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.354+0000 7f7a9928e700 1 -- 172.21.15.46:0/2551177071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7a94159960 msgr2=0x7f7a9406bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:23.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.354+0000 7f7a9928e700 1 --2- 172.21.15.46:0/2551177071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7a94159960 0x7f7a9406bcb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c0099b0 tx=0x7f7a7c009250 comp rx=0 tx=0).stop 2024-01-26T23:45:23.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.354+0000 7f7a9928e700 1 -- 172.21.15.46:0/2551177071 shutdown_connections 2024-01-26T23:45:23.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.354+0000 7f7a9928e700 1 --2- 172.21.15.46:0/2551177071 >> v2:172.21.15.106:3300/0 conn(0x7f7a9406c1f0 0x7f7a94069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.354+0000 7f7a9928e700 1 --2- 172.21.15.46:0/2551177071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7a94159960 0x7f7a9406bcb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.354+0000 7f7a9928e700 1 -- 172.21.15.46:0/2551177071 >> 172.21.15.46:0/2551177071 conn(0x7f7a940ce530 msgr2=0x7f7a940ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:23.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.354+0000 7f7a9928e700 1 -- 172.21.15.46:0/2551177071 shutdown_connections 2024-01-26T23:45:23.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.354+0000 7f7a9928e700 1 -- 172.21.15.46:0/2551177071 wait complete. 2024-01-26T23:45:23.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.356+0000 7f7a9928e700 1 Processor -- start 2024-01-26T23:45:23.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.356+0000 7f7a9928e700 1 -- start start 2024-01-26T23:45:23.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.356+0000 7f7a9928e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7a9406c1f0 0x7f7a94155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:23.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.356+0000 7f7a9928e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a94159590 0x7f7a940c38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:23.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.356+0000 7f7a9928e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a94159960 0x7f7a940c3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:23.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.356+0000 7f7a9928e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9416ae70 con 0x7f7a94159960 2024-01-26T23:45:23.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.356+0000 7f7a9928e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a940c2130 con 0x7f7a94159590 2024-01-26T23:45:23.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.356+0000 7f7a9928e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a940c2410 con 0x7f7a9406c1f0 2024-01-26T23:45:23.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.357+0000 7f7a937fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a94159960 0x7f7a940c3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:23.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.357+0000 7f7a937fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a94159960 0x7f7a940c3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37180/0 (socket says 172.21.15.46:37180) 2024-01-26T23:45:23.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.357+0000 7f7a937fe700 1 -- 172.21.15.46:0/4018112656 learned_addr learned my addr 172.21.15.46:0/4018112656 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:23.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.357+0000 7f7a927fc700 1 --2- 172.21.15.46:0/4018112656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a94159590 0x7f7a940c38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:23.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.357+0000 7f7a92ffd700 1 --2- 172.21.15.46:0/4018112656 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7a9406c1f0 0x7f7a94155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:23.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.357+0000 7f7a937fe700 1 -- 172.21.15.46:0/4018112656 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7a9406c1f0 msgr2=0x7f7a94155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:23.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.357+0000 7f7a937fe700 1 --2- 172.21.15.46:0/4018112656 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7a9406c1f0 0x7f7a94155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.357+0000 7f7a937fe700 1 -- 172.21.15.46:0/4018112656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a94159590 msgr2=0x7f7a940c38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:23.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.357+0000 7f7a937fe700 1 --2- 172.21.15.46:0/4018112656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a94159590 0x7f7a940c38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.357+0000 7f7a937fe700 1 -- 172.21.15.46:0/4018112656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8c009a20 con 0x7f7a94159960 2024-01-26T23:45:23.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.357+0000 7f7a937fe700 1 --2- 172.21.15.46:0/4018112656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a94159960 0x7f7a940c3de0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c009f20 tx=0x7f7a8c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:23.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.357+0000 7f7a8affd700 1 -- 172.21.15.46:0/4018112656 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c00cb70 con 0x7f7a94159960 2024-01-26T23:45:23.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.357+0000 7f7a8affd700 1 -- 172.21.15.46:0/4018112656 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8c00f040 con 0x7f7a94159960 2024-01-26T23:45:23.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.357+0000 7f7a9928e700 1 -- 172.21.15.46:0/4018112656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a7c0094f0 con 0x7f7a94159960 2024-01-26T23:45:23.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.357+0000 7f7a8affd700 1 -- 172.21.15.46:0/4018112656 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c014480 con 0x7f7a94159960 2024-01-26T23:45:23.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.357+0000 7f7a9928e700 1 -- 172.21.15.46:0/4018112656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a940c29d0 con 0x7f7a94159960 2024-01-26T23:45:23.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.358+0000 7f7a9928e700 1 -- 172.21.15.46:0/4018112656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a9404c710 con 0x7f7a94159960 2024-01-26T23:45:23.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.359+0000 7f7a8affd700 1 -- 172.21.15.46:0/4018112656 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7a8c003020 con 0x7f7a94159960 2024-01-26T23:45:23.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.359+0000 7f7a8affd700 1 --2- 172.21.15.46:0/4018112656 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7a74066f30 0x7f7a740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:23.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.359+0000 7f7a8affd700 1 -- 172.21.15.46:0/4018112656 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f7a8c0850c0 con 0x7f7a94159960 2024-01-26T23:45:23.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.359+0000 7f7a92ffd700 1 --2- 172.21.15.46:0/4018112656 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7a74066f30 0x7f7a740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:23.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.360+0000 7f7a92ffd700 1 --2- 172.21.15.46:0/4018112656 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7a74066f30 0x7f7a740693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7a940b3b70 tx=0x7f7a84009040 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:23.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.364+0000 7f7a8affd700 1 -- 172.21.15.46:0/4018112656 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7a8c052060 con 0x7f7a94159960 2024-01-26T23:45:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.570+0000 7f7a9928e700 1 -- 172.21.15.46:0/4018112656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a9406c9a0 con 0x7f7a94159960 2024-01-26T23:45:23.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.571+0000 7f7a8affd700 1 -- 172.21.15.46:0/4018112656 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16151 (secure 0 0 0) 0x7f7a8c055680 con 0x7f7a94159960 2024-01-26T23:45:23.572 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:23.572 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":85,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:45:20.752698+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:45:17.703325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-01-26T23:45:18.149282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:45:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.573+0000 7f7a9928e700 1 -- 172.21.15.46:0/4018112656 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7a74066f30 msgr2=0x7f7a740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.573+0000 7f7a9928e700 1 --2- 172.21.15.46:0/4018112656 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7a74066f30 0x7f7a740693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7a940b3b70 tx=0x7f7a84009040 comp rx=0 tx=0).stop 2024-01-26T23:45:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.573+0000 7f7a9928e700 1 -- 172.21.15.46:0/4018112656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a94159960 msgr2=0x7f7a940c3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.573+0000 7f7a9928e700 1 --2- 172.21.15.46:0/4018112656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a94159960 0x7f7a940c3de0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c009f20 tx=0x7f7a8c00b040 comp rx=0 tx=0).stop 2024-01-26T23:45:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.573+0000 7f7a9928e700 1 -- 172.21.15.46:0/4018112656 shutdown_connections 2024-01-26T23:45:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.573+0000 7f7a9928e700 1 --2- 172.21.15.46:0/4018112656 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7a74066f30 0x7f7a740693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.573+0000 7f7a9928e700 1 --2- 172.21.15.46:0/4018112656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a94159960 0x7f7a940c3de0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.573+0000 7f7a9928e700 1 --2- 172.21.15.46:0/4018112656 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7a9406c1f0 0x7f7a94155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.573+0000 7f7a9928e700 1 --2- 172.21.15.46:0/4018112656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7a94159590 0x7f7a940c38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.573+0000 7f7a9928e700 1 -- 172.21.15.46:0/4018112656 >> 172.21.15.46:0/4018112656 conn(0x7f7a940ce530 msgr2=0x7f7a94156870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.573+0000 7f7a9928e700 1 -- 172.21.15.46:0/4018112656 shutdown_connections 2024-01-26T23:45:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.573+0000 7f7a9928e700 1 -- 172.21.15.46:0/4018112656 wait complete. 2024-01-26T23:45:23.588 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-26T23:45:23.588 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:45:23.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.721+0000 7fa2c203f700 1 Processor -- start 2024-01-26T23:45:23.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.721+0000 7fa2c203f700 1 -- start start 2024-01-26T23:45:23.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.723+0000 7fa2c203f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2bc0cd250 0x7fa2bc162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:23.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.723+0000 7fa2c203f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2bc0cbf10 con 0x7fa2bc0c3cf0 2024-01-26T23:45:23.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.723+0000 7fa2c203f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2bc0c4460 con 0x7fa2bc0acf30 2024-01-26T23:45:23.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.723+0000 7fa2c203f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc0ada10 con 0x7fa2bc0cd250 2024-01-26T23:45:23.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.723+0000 7fa2baffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa2bc0c3cf0 0x7fa2bc0ccb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43346/0 (socket says 172.21.15.46:43346) 2024-01-26T23:45:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.723+0000 7fa2baffd700 1 -- 172.21.15.46:0/3010407683 learned_addr learned my addr 172.21.15.46:0/3010407683 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.723+0000 7fa2bbfff700 1 --2- 172.21.15.46:0/3010407683 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2bc0cd250 0x7fa2bc162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.723+0000 7fa2ba7fc700 1 -- 172.21.15.46:0/3010407683 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258031319 0 0) 0x7fa2bc0cbf10 con 0x7fa2bc0c3cf0 2024-01-26T23:45:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.723+0000 7fa2ba7fc700 1 -- 172.21.15.46:0/3010407683 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2a4003570 con 0x7fa2bc0c3cf0 2024-01-26T23:45:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.723+0000 7fa2ba7fc700 1 -- 172.21.15.46:0/3010407683 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104618 0 0) 0x7fa2bc0c4460 con 0x7fa2bc0acf30 2024-01-26T23:45:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.723+0000 7fa2ba7fc700 1 -- 172.21.15.46:0/3010407683 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2bc0cbf10 con 0x7fa2bc0acf30 2024-01-26T23:45:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.723+0000 7fa2bbfff700 1 -- 172.21.15.46:0/3010407683 >> v1:172.21.15.106:6789/0 conn(0x7fa2bc0acf30 legacy=0x7fa2bc0ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.723+0000 7fa2bbfff700 1 -- 172.21.15.46:0/3010407683 >> v1:172.21.15.46:6789/0 conn(0x7fa2bc0c3cf0 legacy=0x7fa2bc0ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.723+0000 7fa2bbfff700 1 -- 172.21.15.46:0/3010407683 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc0c4740 con 0x7fa2bc0cd250 2024-01-26T23:45:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.724+0000 7fa2bbfff700 1 --2- 172.21.15.46:0/3010407683 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2bc0cd250 0x7fa2bc162360 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4009e90 tx=0x7fa2b4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd60e6f45434af9f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.724+0000 7fa2ba7fc700 1 -- 172.21.15.46:0/3010407683 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b400e070 con 0x7fa2bc0cd250 2024-01-26T23:45:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.724+0000 7fa2ba7fc700 1 -- 172.21.15.46:0/3010407683 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b400bb80 con 0x7fa2bc0cd250 2024-01-26T23:45:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.724+0000 7fa2ba7fc700 1 -- 172.21.15.46:0/3010407683 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4012430 con 0x7fa2bc0cd250 2024-01-26T23:45:23.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.725+0000 7fa2c203f700 1 -- 172.21.15.46:0/3010407683 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2bc0cd250 msgr2=0x7fa2bc162360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:23.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.725+0000 7fa2c203f700 1 --2- 172.21.15.46:0/3010407683 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2bc0cd250 0x7fa2bc162360 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4009e90 tx=0x7fa2b4009780 comp rx=0 tx=0).stop 2024-01-26T23:45:23.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.725+0000 7fa2c203f700 1 -- 172.21.15.46:0/3010407683 shutdown_connections 2024-01-26T23:45:23.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.725+0000 7fa2c203f700 1 --2- 172.21.15.46:0/3010407683 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2bc0cd250 0x7fa2bc162360 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.725+0000 7fa2c203f700 1 -- 172.21.15.46:0/3010407683 >> 172.21.15.46:0/3010407683 conn(0x7fa2bc097820 msgr2=0x7fa2bc068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:23.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.725+0000 7fa2c203f700 1 -- 172.21.15.46:0/3010407683 shutdown_connections 2024-01-26T23:45:23.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.725+0000 7fa2c203f700 1 -- 172.21.15.46:0/3010407683 wait complete. 2024-01-26T23:45:23.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.727+0000 7fa2c203f700 1 Processor -- start 2024-01-26T23:45:23.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.727+0000 7fa2c203f700 1 -- start start 2024-01-26T23:45:23.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.728+0000 7fa2c203f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2bc0acf30 0x7fa2bc1529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:23.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.728+0000 7fa2c203f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2bc0c3cf0 0x7fa2bc155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:23.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.728+0000 7fa2c203f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2bc0cd250 0x7fa2bc156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.728+0000 7fa2c203f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc0c6e80 con 0x7fa2bc0cd250 2024-01-26T23:45:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.728+0000 7fa2c203f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc0c6270 con 0x7fa2bc0acf30 2024-01-26T23:45:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.728+0000 7fa2c203f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc0c6550 con 0x7fa2bc0c3cf0 2024-01-26T23:45:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.728+0000 7fa2bbfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2bc0cd250 0x7fa2bc156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.728+0000 7fa2bbfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2bc0cd250 0x7fa2bc156510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37192/0 (socket says 172.21.15.46:37192) 2024-01-26T23:45:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.728+0000 7fa2bbfff700 1 -- 172.21.15.46:0/4117882834 learned_addr learned my addr 172.21.15.46:0/4117882834 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.728+0000 7fa2bb7fe700 1 --2- 172.21.15.46:0/4117882834 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2bc0acf30 0x7fa2bc1529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.728+0000 7fa2baffd700 1 --2- 172.21.15.46:0/4117882834 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2bc0c3cf0 0x7fa2bc155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.728+0000 7fa2bbfff700 1 -- 172.21.15.46:0/4117882834 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2bc0c3cf0 msgr2=0x7fa2bc155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.728+0000 7fa2bbfff700 1 --2- 172.21.15.46:0/4117882834 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2bc0c3cf0 0x7fa2bc155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.728+0000 7fa2bbfff700 1 -- 172.21.15.46:0/4117882834 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2bc0acf30 msgr2=0x7fa2bc1529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.728+0000 7fa2bbfff700 1 --2- 172.21.15.46:0/4117882834 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2bc0acf30 0x7fa2bc1529b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.728+0000 7fa2bbfff700 1 -- 172.21.15.46:0/4117882834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b4009a20 con 0x7fa2bc0cd250 2024-01-26T23:45:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.729+0000 7fa2bbfff700 1 --2- 172.21.15.46:0/4117882834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2bc0cd250 0x7fa2bc156510 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4013000 tx=0x7fa2b4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:23.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.729+0000 7fa2a37fe700 1 -- 172.21.15.46:0/4117882834 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b401b420 con 0x7fa2bc0cd250 2024-01-26T23:45:23.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.729+0000 7fa2a37fe700 1 -- 172.21.15.46:0/4117882834 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b401b5a0 con 0x7fa2bc0cd250 2024-01-26T23:45:23.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.729+0000 7fa2a37fe700 1 -- 172.21.15.46:0/4117882834 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b401b960 con 0x7fa2bc0cd250 2024-01-26T23:45:23.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.729+0000 7fa2c203f700 1 -- 172.21.15.46:0/4117882834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2bc0c6830 con 0x7fa2bc0cd250 2024-01-26T23:45:23.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.729+0000 7fa2c203f700 1 -- 172.21.15.46:0/4117882834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2bc0c7310 con 0x7fa2bc0cd250 2024-01-26T23:45:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.730+0000 7fa2c203f700 1 -- 172.21.15.46:0/4117882834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2bc04c710 con 0x7fa2bc0cd250 2024-01-26T23:45:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.731+0000 7fa2a37fe700 1 -- 172.21.15.46:0/4117882834 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa2b401bd50 con 0x7fa2bc0cd250 2024-01-26T23:45:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.731+0000 7fa2a37fe700 1 --2- 172.21.15.46:0/4117882834 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa2a8066f30 0x7fa2a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.731+0000 7fa2a37fe700 1 -- 172.21.15.46:0/4117882834 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fa2b4086080 con 0x7fa2bc0cd250 2024-01-26T23:45:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.731+0000 7fa2bb7fe700 1 --2- 172.21.15.46:0/4117882834 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa2a8066f30 0x7fa2a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.732+0000 7fa2bb7fe700 1 --2- 172.21.15.46:0/4117882834 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa2a8066f30 0x7fa2a80693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac007fd0 tx=0x7fa2ac007d50 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:23.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.736+0000 7fa2a37fe700 1 -- 172.21.15.46:0/4117882834 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa2b40530d0 con 0x7fa2bc0cd250 2024-01-26T23:45:23.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.935+0000 7fa2c203f700 1 -- 172.21.15.46:0/4117882834 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa2bc0c6fc0 con 0x7fa2a8066f30 2024-01-26T23:45:23.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.944+0000 7fa2a37fe700 1 -- 172.21.15.46:0/4117882834 <== mgr.5660 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289575 (secure 0 0 0) 0x7fa2bc0c6fc0 con 0x7fa2a8066f30 2024-01-26T23:45:23.949 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:23.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.947+0000 7fa2c203f700 1 -- 172.21.15.46:0/4117882834 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa2a8066f30 msgr2=0x7fa2a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:23.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.947+0000 7fa2c203f700 1 --2- 172.21.15.46:0/4117882834 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa2a8066f30 0x7fa2a80693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac007fd0 tx=0x7fa2ac007d50 comp rx=0 tx=0).stop 2024-01-26T23:45:23.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.947+0000 7fa2c203f700 1 -- 172.21.15.46:0/4117882834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2bc0cd250 msgr2=0x7fa2bc156510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:23.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.947+0000 7fa2c203f700 1 --2- 172.21.15.46:0/4117882834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2bc0cd250 0x7fa2bc156510 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4013000 tx=0x7fa2b4019000 comp rx=0 tx=0).stop 2024-01-26T23:45:23.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.947+0000 7fa2c203f700 1 -- 172.21.15.46:0/4117882834 shutdown_connections 2024-01-26T23:45:23.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.947+0000 7fa2c203f700 1 --2- 172.21.15.46:0/4117882834 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa2a8066f30 0x7fa2a80693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.947+0000 7fa2c203f700 1 --2- 172.21.15.46:0/4117882834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2bc0cd250 0x7fa2bc156510 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.947+0000 7fa2c203f700 1 --2- 172.21.15.46:0/4117882834 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2bc0acf30 0x7fa2bc1529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.947+0000 7fa2c203f700 1 --2- 172.21.15.46:0/4117882834 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2bc0c3cf0 0x7fa2bc155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:23.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.947+0000 7fa2c203f700 1 -- 172.21.15.46:0/4117882834 >> 172.21.15.46:0/4117882834 conn(0x7fa2bc097820 msgr2=0x7fa2bc0ab040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:23.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.948+0000 7fa2c203f700 1 -- 172.21.15.46:0/4117882834 shutdown_connections 2024-01-26T23:45:23.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:23.948+0000 7fa2c203f700 1 -- 172.21.15.46:0/4117882834 wait complete. 2024-01-26T23:45:23.952 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:45:23.963 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-26T23:45:23.675532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2424444,"kb_used_data":9276,"kb_used_omap":0,"kb_used_meta":2415104,"kb_avail":752550276,"statfs":{"total":773094113280,"available":770611482624,"internally_reserved":0,"allocated":9498624,"data_stored":5334780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2473066496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.936137"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.772227+0000","last_change":"2024-01-26T23:45:19.769451+0000","last_active":"2024-01-26T23:45:19.772227+0000","last_peered":"2024-01-26T23:45:19.772227+0000","last_clean":"2024-01-26T23:45:19.772227+0000","last_became_active":"2024-01-26T23:45:19.769137+0000","last_became_peered":"2024-01-26T23:45:19.769137+0000","last_unstale":"2024-01-26T23:45:19.772227+0000","last_undegraded":"2024-01-26T23:45:19.772227+0000","last_fullsized":"2024-01-26T23:45:19.772227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:09:30.529877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.765394+0000","last_change":"2024-01-26T23:45:19.763048+0000","last_active":"2024-01-26T23:45:19.765394+0000","last_peered":"2024-01-26T23:45:19.765394+0000","last_clean":"2024-01-26T23:45:19.765394+0000","last_became_active":"2024-01-26T23:45:19.762735+0000","last_became_peered":"2024-01-26T23:45:19.762735+0000","last_unstale":"2024-01-26T23:45:19.765394+0000","last_undegraded":"2024-01-26T23:45:19.765394+0000","last_fullsized":"2024-01-26T23:45:19.765394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:39:33.812220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.790586+0000","last_change":"2024-01-26T23:45:19.786219+0000","last_active":"2024-01-26T23:45:19.790586+0000","last_peered":"2024-01-26T23:45:19.790586+0000","last_clean":"2024-01-26T23:45:19.790586+0000","last_became_active":"2024-01-26T23:45:19.785929+0000","last_became_peered":"2024-01-26T23:45:19.785929+0000","last_unstale":"2024-01-26T23:45:19.790586+0000","last_undegraded":"2024-01-26T23:45:19.790586+0000","last_fullsized":"2024-01-26T23:45:19.790586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:04:01.842189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.784596+0000","last_change":"2024-01-26T23:45:19.777614+0000","last_active":"2024-01-26T23:45:19.784596+0000","last_peered":"2024-01-26T23:45:19.784596+0000","last_clean":"2024-01-26T23:45:19.784596+0000","last_became_active":"2024-01-26T23:45:19.777257+0000","last_became_peered":"2024-01-26T23:45:19.777257+0000","last_unstale":"2024-01-26T23:45:19.784596+0000","last_undegraded":"2024-01-26T23:45:19.784596+0000","last_fullsized":"2024-01-26T23:45:19.784596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:54:09.399022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.782578+0000","last_change":"2024-01-26T23:45:19.777484+0000","last_active":"2024-01-26T23:45:19.782578+0000","last_peered":"2024-01-26T23:45:19.782578+0000","last_clean":"2024-01-26T23:45:19.782578+0000","last_became_active":"2024-01-26T23:45:19.777225+0000","last_became_peered":"2024-01-26T23:45:19.777225+0000","last_unstale":"2024-01-26T23:45:19.782578+0000","last_undegraded":"2024-01-26T23:45:19.782578+0000","last_fullsized":"2024-01-26T23:45:19.782578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:39:33.812220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.774917+0000","last_change":"2024-01-26T23:45:19.773714+0000","last_active":"2024-01-26T23:45:19.774917+0000","last_peered":"2024-01-26T23:45:19.774917+0000","last_clean":"2024-01-26T23:45:19.774917+0000","last_became_active":"2024-01-26T23:45:19.773375+0000","last_became_peered":"2024-01-26T23:45:19.773375+0000","last_unstale":"2024-01-26T23:45:19.774917+0000","last_undegraded":"2024-01-26T23:45:19.774917+0000","last_fullsized":"2024-01-26T23:45:19.774917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:52:25.112546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.784170+0000","last_change":"2024-01-26T23:45:19.780704+0000","last_active":"2024-01-26T23:45:19.784170+0000","last_peered":"2024-01-26T23:45:19.784170+0000","last_clean":"2024-01-26T23:45:19.784170+0000","last_became_active":"2024-01-26T23:45:19.780384+0000","last_became_peered":"2024-01-26T23:45:19.780384+0000","last_unstale":"2024-01-26T23:45:19.784170+0000","last_undegraded":"2024-01-26T23:45:19.784170+0000","last_fullsized":"2024-01-26T23:45:19.784170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:03:21.787299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.789515+0000","last_change":"2024-01-26T23:45:19.787991+0000","last_active":"2024-01-26T23:45:19.789515+0000","last_peered":"2024-01-26T23:45:19.789515+0000","last_clean":"2024-01-26T23:45:19.789515+0000","last_became_active":"2024-01-26T23:45:19.787701+0000","last_became_peered":"2024-01-26T23:45:19.787701+0000","last_unstale":"2024-01-26T23:45:19.789515+0000","last_undegraded":"2024-01-26T23:45:19.789515+0000","last_fullsized":"2024-01-26T23:45:19.789515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:39:33.812220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.790434+0000","last_change":"2024-01-26T23:45:19.786260+0000","last_active":"2024-01-26T23:45:19.790434+0000","last_peered":"2024-01-26T23:45:19.790434+0000","last_clean":"2024-01-26T23:45:19.790434+0000","last_became_active":"2024-01-26T23:45:19.785954+0000","last_became_peered":"2024-01-26T23:45:19.785954+0000","last_unstale":"2024-01-26T23:45:19.790434+0000","last_undegraded":"2024-01-26T23:45:19.790434+0000","last_fullsized":"2024-01-26T23:45:19.790434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:56:09.047270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.790253+0000","last_change":"2024-01-26T23:45:19.786309+0000","last_active":"2024-01-26T23:45:19.790253+0000","last_peered":"2024-01-26T23:45:19.790253+0000","last_clean":"2024-01-26T23:45:19.790253+0000","last_became_active":"2024-01-26T23:45:19.786033+0000","last_became_peered":"2024-01-26T23:45:19.786033+0000","last_unstale":"2024-01-26T23:45:19.790253+0000","last_undegraded":"2024-01-26T23:45:19.790253+0000","last_fullsized":"2024-01-26T23:45:19.790253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:05:43.312820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.784551+0000","last_change":"2024-01-26T23:45:19.778523+0000","last_active":"2024-01-26T23:45:19.784551+0000","last_peered":"2024-01-26T23:45:19.784551+0000","last_clean":"2024-01-26T23:45:19.784551+0000","last_became_active":"2024-01-26T23:45:19.778171+0000","last_became_peered":"2024-01-26T23:45:19.778171+0000","last_unstale":"2024-01-26T23:45:19.784551+0000","last_undegraded":"2024-01-26T23:45:19.784551+0000","last_fullsized":"2024-01-26T23:45:19.784551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:19:00.324107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.789962+0000","last_change":"2024-01-26T23:45:19.787971+0000","last_active":"2024-01-26T23:45:19.789962+0000","last_peered":"2024-01-26T23:45:19.789962+0000","last_clean":"2024-01-26T23:45:19.789962+0000","last_became_active":"2024-01-26T23:45:19.787669+0000","last_became_peered":"2024-01-26T23:45:19.787669+0000","last_unstale":"2024-01-26T23:45:19.789962+0000","last_undegraded":"2024-01-26T23:45:19.789962+0000","last_fullsized":"2024-01-26T23:45:19.789962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:09:50.055556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.779761+0000","last_change":"2024-01-26T23:45:19.776894+0000","last_active":"2024-01-26T23:45:19.779761+0000","last_peered":"2024-01-26T23:45:19.779761+0000","last_clean":"2024-01-26T23:45:19.779761+0000","last_became_active":"2024-01-26T23:45:19.776596+0000","last_became_peered":"2024-01-26T23:45:19.776596+0000","last_unstale":"2024-01-26T23:45:19.779761+0000","last_undegraded":"2024-01-26T23:45:19.779761+0000","last_fullsized":"2024-01-26T23:45:19.779761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:57:02.783696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.791675+0000","last_change":"2024-01-26T23:45:19.788073+0000","last_active":"2024-01-26T23:45:19.791675+0000","last_peered":"2024-01-26T23:45:19.791675+0000","last_clean":"2024-01-26T23:45:19.791675+0000","last_became_active":"2024-01-26T23:45:19.787750+0000","last_became_peered":"2024-01-26T23:45:19.787750+0000","last_unstale":"2024-01-26T23:45:19.791675+0000","last_undegraded":"2024-01-26T23:45:19.791675+0000","last_fullsized":"2024-01-26T23:45:19.791675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:39:38.641898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.793847+0000","last_change":"2024-01-26T23:45:19.789531+0000","last_active":"2024-01-26T23:45:19.793847+0000","last_peered":"2024-01-26T23:45:19.793847+0000","last_clean":"2024-01-26T23:45:19.793847+0000","last_became_active":"2024-01-26T23:45:19.789262+0000","last_became_peered":"2024-01-26T23:45:19.789262+0000","last_unstale":"2024-01-26T23:45:19.793847+0000","last_undegraded":"2024-01-26T23:45:19.793847+0000","last_fullsized":"2024-01-26T23:45:19.793847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:09:30.529877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.773591+0000","last_change":"2024-01-26T23:45:19.770070+0000","last_active":"2024-01-26T23:45:19.773591+0000","last_peered":"2024-01-26T23:45:19.773591+0000","last_clean":"2024-01-26T23:45:19.773591+0000","last_became_active":"2024-01-26T23:45:19.769337+0000","last_became_peered":"2024-01-26T23:45:19.769337+0000","last_unstale":"2024-01-26T23:45:19.773591+0000","last_undegraded":"2024-01-26T23:45:19.773591+0000","last_fullsized":"2024-01-26T23:45:19.773591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:02:52.183572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.785438+0000","last_change":"2024-01-26T23:45:19.783397+0000","last_active":"2024-01-26T23:45:19.785438+0000","last_peered":"2024-01-26T23:45:19.785438+0000","last_clean":"2024-01-26T23:45:19.785438+0000","last_became_active":"2024-01-26T23:45:19.781037+0000","last_became_peered":"2024-01-26T23:45:19.781037+0000","last_unstale":"2024-01-26T23:45:19.785438+0000","last_undegraded":"2024-01-26T23:45:19.785438+0000","last_fullsized":"2024-01-26T23:45:19.785438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:54:09.399022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778402+0000","last_change":"2024-01-26T23:45:19.776277+0000","last_active":"2024-01-26T23:45:19.778402+0000","last_peered":"2024-01-26T23:45:19.778402+0000","last_clean":"2024-01-26T23:45:19.778402+0000","last_became_active":"2024-01-26T23:45:19.775964+0000","last_became_peered":"2024-01-26T23:45:19.775964+0000","last_unstale":"2024-01-26T23:45:19.778402+0000","last_undegraded":"2024-01-26T23:45:19.778402+0000","last_fullsized":"2024-01-26T23:45:19.778402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:09:50.055556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778633+0000","last_change":"2024-01-26T23:45:19.774575+0000","last_active":"2024-01-26T23:45:19.778633+0000","last_peered":"2024-01-26T23:45:19.778633+0000","last_clean":"2024-01-26T23:45:19.778633+0000","last_became_active":"2024-01-26T23:45:19.774294+0000","last_became_peered":"2024-01-26T23:45:19.774294+0000","last_unstale":"2024-01-26T23:45:19.778633+0000","last_undegraded":"2024-01-26T23:45:19.778633+0000","last_fullsized":"2024-01-26T23:45:19.778633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:20:13.593058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.782044+0000","last_change":"2024-01-26T23:45:19.777529+0000","last_active":"2024-01-26T23:45:19.782044+0000","last_peered":"2024-01-26T23:45:19.782044+0000","last_clean":"2024-01-26T23:45:19.782044+0000","last_became_active":"2024-01-26T23:45:19.777190+0000","last_became_peered":"2024-01-26T23:45:19.777190+0000","last_unstale":"2024-01-26T23:45:19.782044+0000","last_undegraded":"2024-01-26T23:45:19.782044+0000","last_fullsized":"2024-01-26T23:45:19.782044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:52:35.855244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.777619+0000","last_change":"2024-01-26T23:45:19.768338+0000","last_active":"2024-01-26T23:45:19.777619+0000","last_peered":"2024-01-26T23:45:19.777619+0000","last_clean":"2024-01-26T23:45:19.777619+0000","last_became_active":"2024-01-26T23:45:19.768028+0000","last_became_peered":"2024-01-26T23:45:19.768028+0000","last_unstale":"2024-01-26T23:45:19.777619+0000","last_undegraded":"2024-01-26T23:45:19.777619+0000","last_fullsized":"2024-01-26T23:45:19.777619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:54:09.399022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.785709+0000","last_change":"2024-01-26T23:45:19.784566+0000","last_active":"2024-01-26T23:45:19.785709+0000","last_peered":"2024-01-26T23:45:19.785709+0000","last_clean":"2024-01-26T23:45:19.785709+0000","last_became_active":"2024-01-26T23:45:19.784229+0000","last_became_peered":"2024-01-26T23:45:19.784229+0000","last_unstale":"2024-01-26T23:45:19.785709+0000","last_undegraded":"2024-01-26T23:45:19.785709+0000","last_fullsized":"2024-01-26T23:45:19.785709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:35:48.959656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.782416+0000","last_change":"2024-01-26T23:45:19.781858+0000","last_active":"2024-01-26T23:45:19.782416+0000","last_peered":"2024-01-26T23:45:19.782416+0000","last_clean":"2024-01-26T23:45:19.782416+0000","last_became_active":"2024-01-26T23:45:19.781507+0000","last_became_peered":"2024-01-26T23:45:19.781507+0000","last_unstale":"2024-01-26T23:45:19.782416+0000","last_undegraded":"2024-01-26T23:45:19.782416+0000","last_fullsized":"2024-01-26T23:45:19.782416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:36.995172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.787633+0000","last_change":"2024-01-26T23:45:19.783280+0000","last_active":"2024-01-26T23:45:19.787633+0000","last_peered":"2024-01-26T23:45:19.787633+0000","last_clean":"2024-01-26T23:45:19.787633+0000","last_became_active":"2024-01-26T23:45:19.782929+0000","last_became_peered":"2024-01-26T23:45:19.782929+0000","last_unstale":"2024-01-26T23:45:19.787633+0000","last_undegraded":"2024-01-26T23:45:19.787633+0000","last_fullsized":"2024-01-26T23:45:19.787633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:55:05.236611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.779688+0000","last_change":"2024-01-26T23:45:19.773680+0000","last_active":"2024-01-26T23:45:19.779688+0000","last_peered":"2024-01-26T23:45:19.779688+0000","last_clean":"2024-01-26T23:45:19.779688+0000","last_became_active":"2024-01-26T23:45:19.773415+0000","last_became_peered":"2024-01-26T23:45:19.773415+0000","last_unstale":"2024-01-26T23:45:19.779688+0000","last_undegraded":"2024-01-26T23:45:19.779688+0000","last_fullsized":"2024-01-26T23:45:19.779688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:24.303858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.779840+0000","last_change":"2024-01-26T23:45:19.773746+0000","last_active":"2024-01-26T23:45:19.779840+0000","last_peered":"2024-01-26T23:45:19.779840+0000","last_clean":"2024-01-26T23:45:19.779840+0000","last_became_active":"2024-01-26T23:45:19.773469+0000","last_became_peered":"2024-01-26T23:45:19.773469+0000","last_unstale":"2024-01-26T23:45:19.779840+0000","last_undegraded":"2024-01-26T23:45:19.779840+0000","last_fullsized":"2024-01-26T23:45:19.779840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:19:00.324107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.784527+0000","last_change":"2024-01-26T23:45:19.778462+0000","last_active":"2024-01-26T23:45:19.784527+0000","last_peered":"2024-01-26T23:45:19.784527+0000","last_clean":"2024-01-26T23:45:19.784527+0000","last_became_active":"2024-01-26T23:45:19.778141+0000","last_became_peered":"2024-01-26T23:45:19.778141+0000","last_unstale":"2024-01-26T23:45:19.784527+0000","last_undegraded":"2024-01-26T23:45:19.784527+0000","last_fullsized":"2024-01-26T23:45:19.784527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:05:43.312820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.769929+0000","last_change":"2024-01-26T23:45:19.765821+0000","last_active":"2024-01-26T23:45:19.769929+0000","last_peered":"2024-01-26T23:45:19.769929+0000","last_clean":"2024-01-26T23:45:19.769929+0000","last_became_active":"2024-01-26T23:45:19.765517+0000","last_became_peered":"2024-01-26T23:45:19.765517+0000","last_unstale":"2024-01-26T23:45:19.769929+0000","last_undegraded":"2024-01-26T23:45:19.769929+0000","last_fullsized":"2024-01-26T23:45:19.769929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:29:01.653387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.791894+0000","last_change":"2024-01-26T23:45:19.788341+0000","last_active":"2024-01-26T23:45:19.791894+0000","last_peered":"2024-01-26T23:45:19.791894+0000","last_clean":"2024-01-26T23:45:19.791894+0000","last_became_active":"2024-01-26T23:45:19.788092+0000","last_became_peered":"2024-01-26T23:45:19.788092+0000","last_unstale":"2024-01-26T23:45:19.791894+0000","last_undegraded":"2024-01-26T23:45:19.791894+0000","last_fullsized":"2024-01-26T23:45:19.791894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:24.303858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.782081+0000","last_change":"2024-01-26T23:45:19.773137+0000","last_active":"2024-01-26T23:45:19.782081+0000","last_peered":"2024-01-26T23:45:19.782081+0000","last_clean":"2024-01-26T23:45:19.782081+0000","last_became_active":"2024-01-26T23:45:19.772839+0000","last_became_peered":"2024-01-26T23:45:19.772839+0000","last_unstale":"2024-01-26T23:45:19.782081+0000","last_undegraded":"2024-01-26T23:45:19.782081+0000","last_fullsized":"2024-01-26T23:45:19.782081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:05:43.312820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.779648+0000","last_change":"2024-01-26T23:45:19.776897+0000","last_active":"2024-01-26T23:45:19.779648+0000","last_peered":"2024-01-26T23:45:19.779648+0000","last_clean":"2024-01-26T23:45:19.779648+0000","last_became_active":"2024-01-26T23:45:19.776533+0000","last_became_peered":"2024-01-26T23:45:19.776533+0000","last_unstale":"2024-01-26T23:45:19.779648+0000","last_undegraded":"2024-01-26T23:45:19.779648+0000","last_fullsized":"2024-01-26T23:45:19.779648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:40.045595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.766807+0000","last_change":"2024-01-26T23:45:19.763609+0000","last_active":"2024-01-26T23:45:19.766807+0000","last_peered":"2024-01-26T23:45:19.766807+0000","last_clean":"2024-01-26T23:45:19.766807+0000","last_became_active":"2024-01-26T23:45:19.762617+0000","last_became_peered":"2024-01-26T23:45:19.762617+0000","last_unstale":"2024-01-26T23:45:19.766807+0000","last_undegraded":"2024-01-26T23:45:19.766807+0000","last_fullsized":"2024-01-26T23:45:19.766807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:40:16.213836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.776350+0000","last_change":"2024-01-26T23:45:19.771192+0000","last_active":"2024-01-26T23:45:19.776350+0000","last_peered":"2024-01-26T23:45:19.776350+0000","last_clean":"2024-01-26T23:45:19.776350+0000","last_became_active":"2024-01-26T23:45:19.770133+0000","last_became_peered":"2024-01-26T23:45:19.770133+0000","last_unstale":"2024-01-26T23:45:19.776350+0000","last_undegraded":"2024-01-26T23:45:19.776350+0000","last_fullsized":"2024-01-26T23:45:19.776350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:02:44.794463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"84'4","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.793960+0000","last_change":"2024-01-26T23:45:18.751254+0000","last_active":"2024-01-26T23:45:19.793960+0000","last_peered":"2024-01-26T23:45:19.793960+0000","last_clean":"2024-01-26T23:45:19.793960+0000","last_became_active":"2024-01-26T23:45:18.750982+0000","last_became_peered":"2024-01-26T23:45:18.750982+0000","last_unstale":"2024-01-26T23:45:19.793960+0000","last_undegraded":"2024-01-26T23:45:19.793960+0000","last_fullsized":"2024-01-26T23:45:19.793960+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:54:02.845982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.777971+0000","last_change":"2024-01-26T23:45:19.775570+0000","last_active":"2024-01-26T23:45:19.777971+0000","last_peered":"2024-01-26T23:45:19.777971+0000","last_clean":"2024-01-26T23:45:19.777971+0000","last_became_active":"2024-01-26T23:45:19.775301+0000","last_became_peered":"2024-01-26T23:45:19.775301+0000","last_unstale":"2024-01-26T23:45:19.777971+0000","last_undegraded":"2024-01-26T23:45:19.777971+0000","last_fullsized":"2024-01-26T23:45:19.777971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:09:30.529877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"84'1","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.788196+0000","last_change":"2024-01-26T23:45:18.767062+0000","last_active":"2024-01-26T23:45:19.788196+0000","last_peered":"2024-01-26T23:45:19.788196+0000","last_clean":"2024-01-26T23:45:19.788196+0000","last_became_active":"2024-01-26T23:45:18.766829+0000","last_became_peered":"2024-01-26T23:45:18.766829+0000","last_unstale":"2024-01-26T23:45:19.788196+0000","last_undegraded":"2024-01-26T23:45:19.788196+0000","last_fullsized":"2024-01-26T23:45:19.788196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:31:40.511537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.772356+0000","last_change":"2024-01-26T23:45:19.768373+0000","last_active":"2024-01-26T23:45:19.772356+0000","last_peered":"2024-01-26T23:45:19.772356+0000","last_clean":"2024-01-26T23:45:19.772356+0000","last_became_active":"2024-01-26T23:45:19.768093+0000","last_became_peered":"2024-01-26T23:45:19.768093+0000","last_unstale":"2024-01-26T23:45:19.772356+0000","last_undegraded":"2024-01-26T23:45:19.772356+0000","last_fullsized":"2024-01-26T23:45:19.772356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:21:33.110353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"84'2","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.785914+0000","last_change":"2024-01-26T23:45:18.765964+0000","last_active":"2024-01-26T23:45:19.785914+0000","last_peered":"2024-01-26T23:45:19.785914+0000","last_clean":"2024-01-26T23:45:19.785914+0000","last_became_active":"2024-01-26T23:45:18.765676+0000","last_became_peered":"2024-01-26T23:45:18.765676+0000","last_unstale":"2024-01-26T23:45:19.785914+0000","last_undegraded":"2024-01-26T23:45:19.785914+0000","last_fullsized":"2024-01-26T23:45:19.785914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:25:55.585050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.790453+0000","last_change":"2024-01-26T23:45:19.786738+0000","last_active":"2024-01-26T23:45:19.790453+0000","last_peered":"2024-01-26T23:45:19.790453+0000","last_clean":"2024-01-26T23:45:19.790453+0000","last_became_active":"2024-01-26T23:45:19.786474+0000","last_became_peered":"2024-01-26T23:45:19.786474+0000","last_unstale":"2024-01-26T23:45:19.790453+0000","last_undegraded":"2024-01-26T23:45:19.790453+0000","last_fullsized":"2024-01-26T23:45:19.790453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:40.045595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"84'2","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.789007+0000","last_change":"2024-01-26T23:45:18.758761+0000","last_active":"2024-01-26T23:45:19.789007+0000","last_peered":"2024-01-26T23:45:19.789007+0000","last_clean":"2024-01-26T23:45:19.789007+0000","last_became_active":"2024-01-26T23:45:18.758120+0000","last_became_peered":"2024-01-26T23:45:18.758120+0000","last_unstale":"2024-01-26T23:45:19.789007+0000","last_undegraded":"2024-01-26T23:45:19.789007+0000","last_fullsized":"2024-01-26T23:45:19.789007+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:08:05.114547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.771754+0000","last_change":"2024-01-26T23:45:19.763541+0000","last_active":"2024-01-26T23:45:19.771754+0000","last_peered":"2024-01-26T23:45:19.771754+0000","last_clean":"2024-01-26T23:45:19.771754+0000","last_became_active":"2024-01-26T23:45:19.763224+0000","last_became_peered":"2024-01-26T23:45:19.763224+0000","last_unstale":"2024-01-26T23:45:19.771754+0000","last_undegraded":"2024-01-26T23:45:19.771754+0000","last_fullsized":"2024-01-26T23:45:19.771754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:36:29.696055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"84'1","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.782207+0000","last_change":"2024-01-26T23:45:18.761520+0000","last_active":"2024-01-26T23:45:19.782207+0000","last_peered":"2024-01-26T23:45:19.782207+0000","last_clean":"2024-01-26T23:45:19.782207+0000","last_became_active":"2024-01-26T23:45:18.761198+0000","last_became_peered":"2024-01-26T23:45:18.761198+0000","last_unstale":"2024-01-26T23:45:19.782207+0000","last_undegraded":"2024-01-26T23:45:19.782207+0000","last_fullsized":"2024-01-26T23:45:19.782207+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:39:32.807018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.787507+0000","last_change":"2024-01-26T23:45:19.783460+0000","last_active":"2024-01-26T23:45:19.787507+0000","last_peered":"2024-01-26T23:45:19.787507+0000","last_clean":"2024-01-26T23:45:19.787507+0000","last_became_active":"2024-01-26T23:45:19.780983+0000","last_became_peered":"2024-01-26T23:45:19.780983+0000","last_unstale":"2024-01-26T23:45:19.787507+0000","last_undegraded":"2024-01-26T23:45:19.787507+0000","last_fullsized":"2024-01-26T23:45:19.787507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:57:02.783696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"84'1","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.782080+0000","last_change":"2024-01-26T23:45:18.764499+0000","last_active":"2024-01-26T23:45:19.782080+0000","last_peered":"2024-01-26T23:45:19.782080+0000","last_clean":"2024-01-26T23:45:19.782080+0000","last_became_active":"2024-01-26T23:45:18.764133+0000","last_became_peered":"2024-01-26T23:45:18.764133+0000","last_unstale":"2024-01-26T23:45:19.782080+0000","last_undegraded":"2024-01-26T23:45:19.782080+0000","last_fullsized":"2024-01-26T23:45:19.782080+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:18:59.318904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778542+0000","last_change":"2024-01-26T23:45:19.771709+0000","last_active":"2024-01-26T23:45:19.778542+0000","last_peered":"2024-01-26T23:45:19.778542+0000","last_clean":"2024-01-26T23:45:19.778542+0000","last_became_active":"2024-01-26T23:45:19.771443+0000","last_became_peered":"2024-01-26T23:45:19.771443+0000","last_unstale":"2024-01-26T23:45:19.778542+0000","last_undegraded":"2024-01-26T23:45:19.778542+0000","last_fullsized":"2024-01-26T23:45:19.778542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:54:09.399022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.767931+0000","last_change":"2024-01-26T23:45:18.762363+0000","last_active":"2024-01-26T23:45:19.767931+0000","last_peered":"2024-01-26T23:45:19.767931+0000","last_clean":"2024-01-26T23:45:19.767931+0000","last_became_active":"2024-01-26T23:45:18.761883+0000","last_became_peered":"2024-01-26T23:45:18.761883+0000","last_unstale":"2024-01-26T23:45:19.767931+0000","last_undegraded":"2024-01-26T23:45:19.767931+0000","last_fullsized":"2024-01-26T23:45:19.767931+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:05:42.307618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.782286+0000","last_change":"2024-01-26T23:45:19.779200+0000","last_active":"2024-01-26T23:45:19.782286+0000","last_peered":"2024-01-26T23:45:19.782286+0000","last_clean":"2024-01-26T23:45:19.782286+0000","last_became_active":"2024-01-26T23:45:19.777273+0000","last_became_peered":"2024-01-26T23:45:19.777273+0000","last_unstale":"2024-01-26T23:45:19.782286+0000","last_undegraded":"2024-01-26T23:45:19.782286+0000","last_fullsized":"2024-01-26T23:45:19.782286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:15:51.882694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.762310+0000","last_change":"2024-01-26T23:45:18.756051+0000","last_active":"2024-01-26T23:45:19.762310+0000","last_peered":"2024-01-26T23:45:19.762310+0000","last_clean":"2024-01-26T23:45:19.762310+0000","last_became_active":"2024-01-26T23:45:18.755595+0000","last_became_peered":"2024-01-26T23:45:18.755595+0000","last_unstale":"2024-01-26T23:45:19.762310+0000","last_undegraded":"2024-01-26T23:45:19.762310+0000","last_fullsized":"2024-01-26T23:45:19.762310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:41:01.991788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.765032+0000","last_change":"2024-01-26T23:45:19.763815+0000","last_active":"2024-01-26T23:45:19.765032+0000","last_peered":"2024-01-26T23:45:19.765032+0000","last_clean":"2024-01-26T23:45:19.765032+0000","last_became_active":"2024-01-26T23:45:19.763510+0000","last_became_peered":"2024-01-26T23:45:19.763510+0000","last_unstale":"2024-01-26T23:45:19.765032+0000","last_undegraded":"2024-01-26T23:45:19.765032+0000","last_fullsized":"2024-01-26T23:45:19.765032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:04:13.439721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.775992+0000","last_change":"2024-01-26T23:45:18.758624+0000","last_active":"2024-01-26T23:45:19.775992+0000","last_peered":"2024-01-26T23:45:19.775992+0000","last_clean":"2024-01-26T23:45:19.775992+0000","last_became_active":"2024-01-26T23:45:18.757214+0000","last_became_peered":"2024-01-26T23:45:18.757214+0000","last_unstale":"2024-01-26T23:45:19.775992+0000","last_undegraded":"2024-01-26T23:45:19.775992+0000","last_fullsized":"2024-01-26T23:45:19.775992+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:20:13.045863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.782134+0000","last_change":"2024-01-26T23:45:19.779183+0000","last_active":"2024-01-26T23:45:19.782134+0000","last_peered":"2024-01-26T23:45:19.782134+0000","last_clean":"2024-01-26T23:45:19.782134+0000","last_became_active":"2024-01-26T23:45:19.778797+0000","last_became_peered":"2024-01-26T23:45:19.778797+0000","last_unstale":"2024-01-26T23:45:19.782134+0000","last_undegraded":"2024-01-26T23:45:19.782134+0000","last_fullsized":"2024-01-26T23:45:19.782134+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:21:45.768113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.769736+0000","last_change":"2024-01-26T23:45:18.761124+0000","last_active":"2024-01-26T23:45:19.769736+0000","last_peered":"2024-01-26T23:45:19.769736+0000","last_clean":"2024-01-26T23:45:19.769736+0000","last_became_active":"2024-01-26T23:45:18.760861+0000","last_became_peered":"2024-01-26T23:45:18.760861+0000","last_unstale":"2024-01-26T23:45:19.769736+0000","last_undegraded":"2024-01-26T23:45:19.769736+0000","last_fullsized":"2024-01-26T23:45:19.769736+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:54:02.845982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.771954+0000","last_change":"2024-01-26T23:45:19.769326+0000","last_active":"2024-01-26T23:45:19.771954+0000","last_peered":"2024-01-26T23:45:19.771954+0000","last_clean":"2024-01-26T23:45:19.771954+0000","last_became_active":"2024-01-26T23:45:19.769016+0000","last_became_peered":"2024-01-26T23:45:19.769016+0000","last_unstale":"2024-01-26T23:45:19.771954+0000","last_undegraded":"2024-01-26T23:45:19.771954+0000","last_fullsized":"2024-01-26T23:45:19.771954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:57:02.783696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"84'3","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.794265+0000","last_change":"2024-01-26T23:45:18.762976+0000","last_active":"2024-01-26T23:45:19.794265+0000","last_peered":"2024-01-26T23:45:19.794265+0000","last_clean":"2024-01-26T23:45:19.794265+0000","last_became_active":"2024-01-26T23:45:18.762389+0000","last_became_peered":"2024-01-26T23:45:18.762389+0000","last_unstale":"2024-01-26T23:45:19.794265+0000","last_undegraded":"2024-01-26T23:45:19.794265+0000","last_fullsized":"2024-01-26T23:45:19.794265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:28:52.583643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.777464+0000","last_change":"2024-01-26T23:45:19.776012+0000","last_active":"2024-01-26T23:45:19.777464+0000","last_peered":"2024-01-26T23:45:19.777464+0000","last_clean":"2024-01-26T23:45:19.777464+0000","last_became_active":"2024-01-26T23:45:19.775193+0000","last_became_peered":"2024-01-26T23:45:19.775193+0000","last_unstale":"2024-01-26T23:45:19.777464+0000","last_undegraded":"2024-01-26T23:45:19.777464+0000","last_fullsized":"2024-01-26T23:45:19.777464+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:39:33.812220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"84'6","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.796892+0000","last_change":"2024-01-26T23:45:18.765358+0000","last_active":"2024-01-26T23:45:19.796892+0000","last_peered":"2024-01-26T23:45:19.796892+0000","last_clean":"2024-01-26T23:45:19.796892+0000","last_became_active":"2024-01-26T23:45:18.765058+0000","last_became_peered":"2024-01-26T23:45:18.765058+0000","last_unstale":"2024-01-26T23:45:19.796892+0000","last_undegraded":"2024-01-26T23:45:19.796892+0000","last_fullsized":"2024-01-26T23:45:19.796892+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:00.932196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.788853+0000","last_change":"2024-01-26T23:45:19.784757+0000","last_active":"2024-01-26T23:45:19.788853+0000","last_peered":"2024-01-26T23:45:19.788853+0000","last_clean":"2024-01-26T23:45:19.788853+0000","last_became_active":"2024-01-26T23:45:19.784467+0000","last_became_peered":"2024-01-26T23:45:19.784467+0000","last_unstale":"2024-01-26T23:45:19.788853+0000","last_undegraded":"2024-01-26T23:45:19.788853+0000","last_fullsized":"2024-01-26T23:45:19.788853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:24.303858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"84'2","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.794100+0000","last_change":"2024-01-26T23:45:18.766255+0000","last_active":"2024-01-26T23:45:19.794100+0000","last_peered":"2024-01-26T23:45:19.794100+0000","last_clean":"2024-01-26T23:45:19.794100+0000","last_became_active":"2024-01-26T23:45:18.765979+0000","last_became_peered":"2024-01-26T23:45:18.765979+0000","last_unstale":"2024-01-26T23:45:19.794100+0000","last_undegraded":"2024-01-26T23:45:19.794100+0000","last_fullsized":"2024-01-26T23:45:19.794100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:36:25.947995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.777988+0000","last_change":"2024-01-26T23:45:19.769598+0000","last_active":"2024-01-26T23:45:19.777988+0000","last_peered":"2024-01-26T23:45:19.777988+0000","last_clean":"2024-01-26T23:45:19.777988+0000","last_became_active":"2024-01-26T23:45:19.769297+0000","last_became_peered":"2024-01-26T23:45:19.769297+0000","last_unstale":"2024-01-26T23:45:19.777988+0000","last_undegraded":"2024-01-26T23:45:19.777988+0000","last_fullsized":"2024-01-26T23:45:19.777988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:29:01.653387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.764205+0000","last_change":"2024-01-26T23:45:18.764994+0000","last_active":"2024-01-26T23:45:19.764205+0000","last_peered":"2024-01-26T23:45:19.764205+0000","last_clean":"2024-01-26T23:45:19.764205+0000","last_became_active":"2024-01-26T23:45:18.764683+0000","last_became_peered":"2024-01-26T23:45:18.764683+0000","last_unstale":"2024-01-26T23:45:19.764205+0000","last_undegraded":"2024-01-26T23:45:19.764205+0000","last_fullsized":"2024-01-26T23:45:19.764205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:00.932196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778563+0000","last_change":"2024-01-26T23:45:19.773049+0000","last_active":"2024-01-26T23:45:19.778563+0000","last_peered":"2024-01-26T23:45:19.778563+0000","last_clean":"2024-01-26T23:45:19.778563+0000","last_became_active":"2024-01-26T23:45:19.772754+0000","last_became_peered":"2024-01-26T23:45:19.772754+0000","last_unstale":"2024-01-26T23:45:19.778563+0000","last_undegraded":"2024-01-26T23:45:19.778563+0000","last_fullsized":"2024-01-26T23:45:19.778563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:56:05.060730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.779784+0000","last_change":"2024-01-26T23:45:18.763694+0000","last_active":"2024-01-26T23:45:19.779784+0000","last_peered":"2024-01-26T23:45:19.779784+0000","last_clean":"2024-01-26T23:45:19.779784+0000","last_became_active":"2024-01-26T23:45:18.763345+0000","last_became_peered":"2024-01-26T23:45:18.763345+0000","last_unstale":"2024-01-26T23:45:19.779784+0000","last_undegraded":"2024-01-26T23:45:19.779784+0000","last_fullsized":"2024-01-26T23:45:19.779784+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:57:17.179137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778284+0000","last_change":"2024-01-26T23:45:19.775692+0000","last_active":"2024-01-26T23:45:19.778284+0000","last_peered":"2024-01-26T23:45:19.778284+0000","last_clean":"2024-01-26T23:45:19.778284+0000","last_became_active":"2024-01-26T23:45:19.775333+0000","last_became_peered":"2024-01-26T23:45:19.775333+0000","last_unstale":"2024-01-26T23:45:19.778284+0000","last_undegraded":"2024-01-26T23:45:19.778284+0000","last_fullsized":"2024-01-26T23:45:19.778284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:59.089607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"84'8","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.792435+0000","last_change":"2024-01-26T23:45:18.764634+0000","last_active":"2024-01-26T23:45:19.792435+0000","last_peered":"2024-01-26T23:45:19.792435+0000","last_clean":"2024-01-26T23:45:19.792435+0000","last_became_active":"2024-01-26T23:45:18.764318+0000","last_became_peered":"2024-01-26T23:45:18.764318+0000","last_unstale":"2024-01-26T23:45:19.792435+0000","last_undegraded":"2024-01-26T23:45:19.792435+0000","last_fullsized":"2024-01-26T23:45:19.792435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:00.932196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.791458+0000","last_change":"2024-01-26T23:45:19.787367+0000","last_active":"2024-01-26T23:45:19.791458+0000","last_peered":"2024-01-26T23:45:19.791458+0000","last_clean":"2024-01-26T23:45:19.791458+0000","last_became_active":"2024-01-26T23:45:19.787104+0000","last_became_peered":"2024-01-26T23:45:19.787104+0000","last_unstale":"2024-01-26T23:45:19.791458+0000","last_undegraded":"2024-01-26T23:45:19.791458+0000","last_fullsized":"2024-01-26T23:45:19.791458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:19:00.324107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"84'1","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.787862+0000","last_change":"2024-01-26T23:45:18.765531+0000","last_active":"2024-01-26T23:45:19.787862+0000","last_peered":"2024-01-26T23:45:19.787862+0000","last_clean":"2024-01-26T23:45:19.787862+0000","last_became_active":"2024-01-26T23:45:18.765288+0000","last_became_peered":"2024-01-26T23:45:18.765288+0000","last_unstale":"2024-01-26T23:45:19.787862+0000","last_undegraded":"2024-01-26T23:45:19.787862+0000","last_fullsized":"2024-01-26T23:45:19.787862+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:24:57.645890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.785942+0000","last_change":"2024-01-26T23:45:19.782678+0000","last_active":"2024-01-26T23:45:19.785942+0000","last_peered":"2024-01-26T23:45:19.785942+0000","last_clean":"2024-01-26T23:45:19.785942+0000","last_became_active":"2024-01-26T23:45:19.782422+0000","last_became_peered":"2024-01-26T23:45:19.782422+0000","last_unstale":"2024-01-26T23:45:19.785942+0000","last_undegraded":"2024-01-26T23:45:19.785942+0000","last_fullsized":"2024-01-26T23:45:19.785942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:09:30.529877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"84'2","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.794369+0000","last_change":"2024-01-26T23:45:18.768703+0000","last_active":"2024-01-26T23:45:19.794369+0000","last_peered":"2024-01-26T23:45:19.794369+0000","last_clean":"2024-01-26T23:45:19.794369+0000","last_became_active":"2024-01-26T23:45:18.768460+0000","last_became_peered":"2024-01-26T23:45:18.768460+0000","last_unstale":"2024-01-26T23:45:19.794369+0000","last_undegraded":"2024-01-26T23:45:19.794369+0000","last_fullsized":"2024-01-26T23:45:19.794369+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:54:02.845982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778375+0000","last_change":"2024-01-26T23:45:19.775743+0000","last_active":"2024-01-26T23:45:19.778375+0000","last_peered":"2024-01-26T23:45:19.778375+0000","last_clean":"2024-01-26T23:45:19.778375+0000","last_became_active":"2024-01-26T23:45:19.774642+0000","last_became_peered":"2024-01-26T23:45:19.774642+0000","last_unstale":"2024-01-26T23:45:19.778375+0000","last_undegraded":"2024-01-26T23:45:19.778375+0000","last_fullsized":"2024-01-26T23:45:19.778375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:04:13.439721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"84'1","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.794313+0000","last_change":"2024-01-26T23:45:18.763140+0000","last_active":"2024-01-26T23:45:19.794313+0000","last_peered":"2024-01-26T23:45:19.794313+0000","last_clean":"2024-01-26T23:45:19.794313+0000","last_became_active":"2024-01-26T23:45:18.762764+0000","last_became_peered":"2024-01-26T23:45:18.762764+0000","last_unstale":"2024-01-26T23:45:19.794313+0000","last_undegraded":"2024-01-26T23:45:19.794313+0000","last_fullsized":"2024-01-26T23:45:19.794313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:52:28.272099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778657+0000","last_change":"2024-01-26T23:45:19.774562+0000","last_active":"2024-01-26T23:45:19.778657+0000","last_peered":"2024-01-26T23:45:19.778657+0000","last_clean":"2024-01-26T23:45:19.778657+0000","last_became_active":"2024-01-26T23:45:19.774092+0000","last_became_peered":"2024-01-26T23:45:19.774092+0000","last_unstale":"2024-01-26T23:45:19.778657+0000","last_undegraded":"2024-01-26T23:45:19.778657+0000","last_fullsized":"2024-01-26T23:45:19.778657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:57:18.184339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.767799+0000","last_change":"2024-01-26T23:45:18.768278+0000","last_active":"2024-01-26T23:45:19.767799+0000","last_peered":"2024-01-26T23:45:19.767799+0000","last_clean":"2024-01-26T23:45:19.767799+0000","last_became_active":"2024-01-26T23:45:18.767952+0000","last_became_peered":"2024-01-26T23:45:18.767952+0000","last_unstale":"2024-01-26T23:45:19.767799+0000","last_undegraded":"2024-01-26T23:45:19.767799+0000","last_fullsized":"2024-01-26T23:45:19.767799+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:04:12.434519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.766828+0000","last_change":"2024-01-26T23:45:19.762912+0000","last_active":"2024-01-26T23:45:19.766828+0000","last_peered":"2024-01-26T23:45:19.766828+0000","last_clean":"2024-01-26T23:45:19.766828+0000","last_became_active":"2024-01-26T23:45:19.762207+0000","last_became_peered":"2024-01-26T23:45:19.762207+0000","last_unstale":"2024-01-26T23:45:19.766828+0000","last_undegraded":"2024-01-26T23:45:19.766828+0000","last_fullsized":"2024-01-26T23:45:19.766828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:51:03.633095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"84'2","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.786073+0000","last_change":"2024-01-26T23:45:18.761890+0000","last_active":"2024-01-26T23:45:19.786073+0000","last_peered":"2024-01-26T23:45:19.786073+0000","last_clean":"2024-01-26T23:45:19.786073+0000","last_became_active":"2024-01-26T23:45:18.760793+0000","last_became_peered":"2024-01-26T23:45:18.760793+0000","last_unstale":"2024-01-26T23:45:19.786073+0000","last_undegraded":"2024-01-26T23:45:19.786073+0000","last_fullsized":"2024-01-26T23:45:19.786073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:45:19.871782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.779715+0000","last_change":"2024-01-26T23:45:19.774087+0000","last_active":"2024-01-26T23:45:19.779715+0000","last_peered":"2024-01-26T23:45:19.779715+0000","last_clean":"2024-01-26T23:45:19.779715+0000","last_became_active":"2024-01-26T23:45:19.773249+0000","last_became_peered":"2024-01-26T23:45:19.773249+0000","last_unstale":"2024-01-26T23:45:19.779715+0000","last_undegraded":"2024-01-26T23:45:19.779715+0000","last_fullsized":"2024-01-26T23:45:19.779715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:04:13.439721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.764307+0000","last_change":"2024-01-26T23:45:18.764993+0000","last_active":"2024-01-26T23:45:19.764307+0000","last_peered":"2024-01-26T23:45:19.764307+0000","last_clean":"2024-01-26T23:45:19.764307+0000","last_became_active":"2024-01-26T23:45:18.764701+0000","last_became_peered":"2024-01-26T23:45:18.764701+0000","last_unstale":"2024-01-26T23:45:19.764307+0000","last_undegraded":"2024-01-26T23:45:19.764307+0000","last_fullsized":"2024-01-26T23:45:19.764307+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:57:07.774030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.770720+0000","last_change":"2024-01-26T23:45:19.767295+0000","last_active":"2024-01-26T23:45:19.770720+0000","last_peered":"2024-01-26T23:45:19.770720+0000","last_clean":"2024-01-26T23:45:19.770720+0000","last_became_active":"2024-01-26T23:45:19.767071+0000","last_became_peered":"2024-01-26T23:45:19.767071+0000","last_unstale":"2024-01-26T23:45:19.770720+0000","last_undegraded":"2024-01-26T23:45:19.770720+0000","last_fullsized":"2024-01-26T23:45:19.770720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:04:13.439721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.754675+0000","last_change":"2024-01-26T23:45:18.768303+0000","last_active":"2024-01-26T23:45:19.754675+0000","last_peered":"2024-01-26T23:45:19.754675+0000","last_clean":"2024-01-26T23:45:19.754675+0000","last_became_active":"2024-01-26T23:45:18.767961+0000","last_became_peered":"2024-01-26T23:45:18.767961+0000","last_unstale":"2024-01-26T23:45:19.754675+0000","last_undegraded":"2024-01-26T23:45:19.754675+0000","last_fullsized":"2024-01-26T23:45:19.754675+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:18:51.436569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.780733+0000","last_change":"2024-01-26T23:45:19.779663+0000","last_active":"2024-01-26T23:45:19.780733+0000","last_peered":"2024-01-26T23:45:19.780733+0000","last_clean":"2024-01-26T23:45:19.780733+0000","last_became_active":"2024-01-26T23:45:19.779091+0000","last_became_peered":"2024-01-26T23:45:19.779091+0000","last_unstale":"2024-01-26T23:45:19.780733+0000","last_undegraded":"2024-01-26T23:45:19.780733+0000","last_fullsized":"2024-01-26T23:45:19.780733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:41:17.135191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"84'2","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.792301+0000","last_change":"2024-01-26T23:45:18.767075+0000","last_active":"2024-01-26T23:45:19.792301+0000","last_peered":"2024-01-26T23:45:19.792301+0000","last_clean":"2024-01-26T23:45:19.792301+0000","last_became_active":"2024-01-26T23:45:18.766757+0000","last_became_peered":"2024-01-26T23:45:18.766757+0000","last_unstale":"2024-01-26T23:45:19.792301+0000","last_undegraded":"2024-01-26T23:45:19.792301+0000","last_fullsized":"2024-01-26T23:45:19.792301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:02:05.153467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.771608+0000","last_change":"2024-01-26T23:45:19.765627+0000","last_active":"2024-01-26T23:45:19.771608+0000","last_peered":"2024-01-26T23:45:19.771608+0000","last_clean":"2024-01-26T23:45:19.771608+0000","last_became_active":"2024-01-26T23:45:19.765096+0000","last_became_peered":"2024-01-26T23:45:19.765096+0000","last_unstale":"2024-01-26T23:45:19.771608+0000","last_undegraded":"2024-01-26T23:45:19.771608+0000","last_fullsized":"2024-01-26T23:45:19.771608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:55.510702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.764382+0000","last_change":"2024-01-26T23:45:18.763881+0000","last_active":"2024-01-26T23:45:19.764382+0000","last_peered":"2024-01-26T23:45:19.764382+0000","last_clean":"2024-01-26T23:45:19.764382+0000","last_became_active":"2024-01-26T23:45:18.763607+0000","last_became_peered":"2024-01-26T23:45:18.763607+0000","last_unstale":"2024-01-26T23:45:19.764382+0000","last_undegraded":"2024-01-26T23:45:19.764382+0000","last_fullsized":"2024-01-26T23:45:19.764382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:54:02.845982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778014+0000","last_change":"2024-01-26T23:45:19.768388+0000","last_active":"2024-01-26T23:45:19.778014+0000","last_peered":"2024-01-26T23:45:19.778014+0000","last_clean":"2024-01-26T23:45:19.778014+0000","last_became_active":"2024-01-26T23:45:19.768069+0000","last_became_peered":"2024-01-26T23:45:19.768069+0000","last_unstale":"2024-01-26T23:45:19.778014+0000","last_undegraded":"2024-01-26T23:45:19.778014+0000","last_fullsized":"2024-01-26T23:45:19.778014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:40.045595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.754836+0000","last_change":"2024-01-26T23:45:18.765412+0000","last_active":"2024-01-26T23:45:19.754836+0000","last_peered":"2024-01-26T23:45:19.754836+0000","last_clean":"2024-01-26T23:45:19.754836+0000","last_became_active":"2024-01-26T23:45:18.765125+0000","last_became_peered":"2024-01-26T23:45:18.765125+0000","last_unstale":"2024-01-26T23:45:19.754836+0000","last_undegraded":"2024-01-26T23:45:19.754836+0000","last_fullsized":"2024-01-26T23:45:19.754836+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:04:14.572796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778647+0000","last_change":"2024-01-26T23:45:19.774382+0000","last_active":"2024-01-26T23:45:19.778647+0000","last_peered":"2024-01-26T23:45:19.778647+0000","last_clean":"2024-01-26T23:45:19.778647+0000","last_became_active":"2024-01-26T23:45:19.773713+0000","last_became_peered":"2024-01-26T23:45:19.773713+0000","last_unstale":"2024-01-26T23:45:19.778647+0000","last_undegraded":"2024-01-26T23:45:19.778647+0000","last_fullsized":"2024-01-26T23:45:19.778647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:52:29.277301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.758507+0000","last_change":"2024-01-26T23:45:18.760331+0000","last_active":"2024-01-26T23:45:19.758507+0000","last_peered":"2024-01-26T23:45:19.758507+0000","last_clean":"2024-01-26T23:45:19.758507+0000","last_became_active":"2024-01-26T23:45:18.760024+0000","last_became_peered":"2024-01-26T23:45:18.760024+0000","last_unstale":"2024-01-26T23:45:19.758507+0000","last_undegraded":"2024-01-26T23:45:19.758507+0000","last_fullsized":"2024-01-26T23:45:19.758507+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:57:07.774030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.770462+0000","last_change":"2024-01-26T23:45:19.766917+0000","last_active":"2024-01-26T23:45:19.770462+0000","last_peered":"2024-01-26T23:45:19.770462+0000","last_clean":"2024-01-26T23:45:19.770462+0000","last_became_active":"2024-01-26T23:45:19.766638+0000","last_became_peered":"2024-01-26T23:45:19.766638+0000","last_unstale":"2024-01-26T23:45:19.770462+0000","last_undegraded":"2024-01-26T23:45:19.770462+0000","last_fullsized":"2024-01-26T23:45:19.770462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:59.089607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.775566+0000","last_change":"2024-01-26T23:45:18.763657+0000","last_active":"2024-01-26T23:45:19.775566+0000","last_peered":"2024-01-26T23:45:19.775566+0000","last_clean":"2024-01-26T23:45:19.775566+0000","last_became_active":"2024-01-26T23:45:18.763274+0000","last_became_peered":"2024-01-26T23:45:18.763274+0000","last_unstale":"2024-01-26T23:45:19.775566+0000","last_undegraded":"2024-01-26T23:45:19.775566+0000","last_fullsized":"2024-01-26T23:45:19.775566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:02:43.789261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.785440+0000","last_change":"2024-01-26T23:45:19.783363+0000","last_active":"2024-01-26T23:45:19.785440+0000","last_peered":"2024-01-26T23:45:19.785440+0000","last_clean":"2024-01-26T23:45:19.785440+0000","last_became_active":"2024-01-26T23:45:19.783118+0000","last_became_peered":"2024-01-26T23:45:19.783118+0000","last_unstale":"2024-01-26T23:45:19.785440+0000","last_undegraded":"2024-01-26T23:45:19.785440+0000","last_fullsized":"2024-01-26T23:45:19.785440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:57:02.783696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.767529+0000","last_change":"2024-01-26T23:45:18.764082+0000","last_active":"2024-01-26T23:45:19.767529+0000","last_peered":"2024-01-26T23:45:19.767529+0000","last_clean":"2024-01-26T23:45:19.767529+0000","last_became_active":"2024-01-26T23:45:18.763732+0000","last_became_peered":"2024-01-26T23:45:18.763732+0000","last_unstale":"2024-01-26T23:45:19.767529+0000","last_undegraded":"2024-01-26T23:45:19.767529+0000","last_fullsized":"2024-01-26T23:45:19.767529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:09:49.050354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.785139+0000","last_change":"2024-01-26T23:45:19.783970+0000","last_active":"2024-01-26T23:45:19.785139+0000","last_peered":"2024-01-26T23:45:19.785139+0000","last_clean":"2024-01-26T23:45:19.785139+0000","last_became_active":"2024-01-26T23:45:19.783042+0000","last_became_peered":"2024-01-26T23:45:19.783042+0000","last_unstale":"2024-01-26T23:45:19.785139+0000","last_undegraded":"2024-01-26T23:45:19.785139+0000","last_fullsized":"2024-01-26T23:45:19.785139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:56:05.060730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.767961+0000","last_change":"2024-01-26T23:45:18.762184+0000","last_active":"2024-01-26T23:45:19.767961+0000","last_peered":"2024-01-26T23:45:19.767961+0000","last_clean":"2024-01-26T23:45:19.767961+0000","last_became_active":"2024-01-26T23:45:18.761581+0000","last_became_peered":"2024-01-26T23:45:18.761581+0000","last_unstale":"2024-01-26T23:45:19.767961+0000","last_undegraded":"2024-01-26T23:45:19.767961+0000","last_fullsized":"2024-01-26T23:45:19.767961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:39.040393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.773724+0000","last_change":"2024-01-26T23:45:19.769873+0000","last_active":"2024-01-26T23:45:19.773724+0000","last_peered":"2024-01-26T23:45:19.773724+0000","last_clean":"2024-01-26T23:45:19.773724+0000","last_became_active":"2024-01-26T23:45:19.769571+0000","last_became_peered":"2024-01-26T23:45:19.769571+0000","last_unstale":"2024-01-26T23:45:19.773724+0000","last_undegraded":"2024-01-26T23:45:19.773724+0000","last_fullsized":"2024-01-26T23:45:19.773724+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:03:41.407804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.758583+0000","last_change":"2024-01-26T23:45:18.760794+0000","last_active":"2024-01-26T23:45:19.758583+0000","last_peered":"2024-01-26T23:45:19.758583+0000","last_clean":"2024-01-26T23:45:19.758583+0000","last_became_active":"2024-01-26T23:45:18.760500+0000","last_became_peered":"2024-01-26T23:45:18.760500+0000","last_unstale":"2024-01-26T23:45:19.758583+0000","last_undegraded":"2024-01-26T23:45:19.758583+0000","last_fullsized":"2024-01-26T23:45:19.758583+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:24:57.645890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.776234+0000","last_change":"2024-01-26T23:45:19.770513+0000","last_active":"2024-01-26T23:45:19.776234+0000","last_peered":"2024-01-26T23:45:19.776234+0000","last_clean":"2024-01-26T23:45:19.776234+0000","last_became_active":"2024-01-26T23:45:19.770231+0000","last_became_peered":"2024-01-26T23:45:19.770231+0000","last_unstale":"2024-01-26T23:45:19.776234+0000","last_undegraded":"2024-01-26T23:45:19.776234+0000","last_fullsized":"2024-01-26T23:45:19.776234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:49:56.096830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.767411+0000","last_change":"2024-01-26T23:45:18.763355+0000","last_active":"2024-01-26T23:45:19.767411+0000","last_peered":"2024-01-26T23:45:19.767411+0000","last_clean":"2024-01-26T23:45:19.767411+0000","last_became_active":"2024-01-26T23:45:18.763015+0000","last_became_peered":"2024-01-26T23:45:18.763015+0000","last_unstale":"2024-01-26T23:45:19.767411+0000","last_undegraded":"2024-01-26T23:45:19.767411+0000","last_fullsized":"2024-01-26T23:45:19.767411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:09:29.524675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'161","reported_seq":376,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.773981+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:45:19.773981+0000","last_peered":"2024-01-26T23:45:19.773981+0000","last_clean":"2024-01-26T23:45:19.773981+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:45:19.773981+0000","last_undegraded":"2024-01-26T23:45:19.773981+0000","last_fullsized":"2024-01-26T23:45:19.773981+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575038,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307084,"kb_used_data":1604,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064756,"statfs":{"total":96636764160,"available":96322310144,"internally_reserved":0,"allocated":1642496,"data_stored":1110670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57599999999999996,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.79500000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.35699999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80400000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.82099999999999995}]},{"osd":1,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57699999999999996,"15min":0.57799999999999996},"min":{"1min":0.42499999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77800000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.62},{"interface":"front","average":{"1min":0.62,"5min":0.61099999999999999,"15min":0.59899999999999998},"min":{"1min":0.46000000000000002,"5min":0.39200000000000002,"15min":0.36099999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.86499999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.38800000000000001,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.81699999999999995},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.48699999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77900000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.748}]},{"osd":3,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59999999999999998,"15min":0.59199999999999997},"min":{"1min":0.442,"5min":0.40500000000000003,"15min":0.35399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60599999999999998,"15min":0.60499999999999998},"min":{"1min":0.40100000000000002,"5min":0.39300000000000002,"15min":0.373},"max":{"1min":0.82699999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.67100000000000004}]},{"osd":4,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46899999999999997,"15min":0.46100000000000002},"min":{"1min":0.36399999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.66100000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.495,"15min":0.48799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.314},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.51400000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.504,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.91700000000000004},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.40000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.74199999999999999,"5min":0.90200000000000002,"15min":0.96599999999999997},"last":0.73599999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.57999999999999996,"15min":0.55700000000000005},"min":{"1min":0.46400000000000002,"5min":0.375,"15min":0.33100000000000002},"max":{"1min":0.77900000000000003,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56200000000000006,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76500000000000001,"5min":0.85199999999999998,"15min":0.86899999999999999},"last":0.68999999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575038,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302396,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069444,"statfs":{"total":96636764160,"available":96327110656,"internally_reserved":0,"allocated":1036288,"data_stored":519588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56899999999999995,"15min":0.57399999999999995},"min":{"1min":0.35999999999999999,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":0.752,"5min":0.85199999999999998,"15min":0.90200000000000002},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59099999999999997,"15min":0.59399999999999997},"min":{"1min":0.42899999999999999,"5min":0.38500000000000001,"15min":0.33600000000000002},"max":{"1min":0.79000000000000004,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.53700000000000003}]},{"osd":1,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.312,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.77100000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.622},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.34399999999999997},"max":{"1min":0.75700000000000001,"5min":0.78900000000000003,"15min":0.874},"last":0.70799999999999996}]},{"osd":2,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.40799999999999997,"5min":0.35499999999999998,"15min":0.29599999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.85799999999999998},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38600000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77000000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55100000000000005}]},{"osd":3,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.59499999999999997,"15min":0.59999999999999998},"min":{"1min":0.45500000000000002,"5min":0.38700000000000001,"15min":0.29999999999999999},"max":{"1min":0.79000000000000004,"5min":0.80700000000000005,"15min":0.92200000000000004},"last":0.745},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.622,"15min":0.624},"min":{"1min":0.55000000000000004,"5min":0.39500000000000002,"15min":0.39000000000000001},"max":{"1min":0.78400000000000003,"5min":0.81100000000000005,"15min":0.84499999999999997},"last":0.58499999999999996}]},{"osd":4,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.47799999999999998,"15min":0.47399999999999998},"min":{"1min":0.38100000000000001,"5min":0.29899999999999999,"15min":0.22},"max":{"1min":0.69899999999999995,"5min":0.84599999999999997,"15min":0.90200000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.501,"15min":0.499},"min":{"1min":0.44700000000000001,"5min":0.32000000000000001,"15min":0.28899999999999998},"max":{"1min":0.61899999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.46899999999999997}]},{"osd":5,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48499999999999999,"15min":0.48899999999999999},"min":{"1min":0.42499999999999999,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.53400000000000003,"15min":0.53300000000000003},"min":{"1min":0.46100000000000002,"5min":0.33700000000000002,"15min":0.26200000000000001},"max":{"1min":0.75700000000000001,"5min":0.872,"15min":0.96499999999999997},"last":0.60699999999999998}]},{"osd":7,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53500000000000003,"15min":0.53800000000000003},"min":{"1min":0.44400000000000001,"5min":0.32500000000000001,"15min":0.246},"max":{"1min":0.81799999999999995,"5min":0.82799999999999996,"15min":0.97199999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.308},"max":{"1min":0.73099999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.68600000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575038,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302420,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069420,"statfs":{"total":96636764160,"available":96327086080,"internally_reserved":0,"allocated":1060864,"data_stored":521826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29899999999999999,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":0.59699999999999998,"5min":0.621,"15min":0.72599999999999998},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.35799999999999998,"5min":0.28899999999999998,"15min":0.26000000000000001},"max":{"1min":0.73499999999999999,"5min":1.337,"15min":1.337},"last":0.41999999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41499999999999998},"min":{"1min":0.27700000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55900000000000005,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.379},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48599999999999999,"15min":0.47699999999999998},"min":{"1min":0.376,"5min":0.34699999999999998,"15min":0.28999999999999998},"max":{"1min":0.70799999999999996,"5min":0.77300000000000002,"15min":0.79600000000000004},"last":0.39500000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52200000000000002,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":0.75800000000000001,"5min":1.782,"15min":1.782},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.497,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.625,"5min":1.379,"15min":1.379},"last":0.32000000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65300000000000002,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":0.86499999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.435},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66200000000000003,"15min":0.66600000000000004},"min":{"1min":0.49399999999999999,"5min":0.45900000000000002,"15min":0.433},"max":{"1min":0.89900000000000002,"5min":0.91900000000000004,"15min":0.95699999999999996},"last":0.51600000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.50600000000000001,"5min":0.46800000000000003,"15min":0.42799999999999999},"max":{"1min":0.879,"5min":1.208,"15min":1.208},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68500000000000005,"15min":0.67100000000000004},"min":{"1min":0.53700000000000003,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":0.81599999999999995,"5min":0.96599999999999997,"15min":1.018},"last":0.46899999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.69399999999999995,"15min":0.68500000000000005},"min":{"1min":0.51800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85199999999999998,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.52900000000000003,"5min":0.437,"15min":0.42599999999999999},"max":{"1min":0.83699999999999997,"5min":1.022,"15min":1.022},"last":0.48899999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71599999999999997},"min":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.49399999999999999},"max":{"1min":0.93200000000000005,"5min":0.98199999999999998,"15min":1.0069999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.749,"15min":0.73199999999999998},"min":{"1min":0.53200000000000003,"5min":0.52200000000000002,"15min":0.438},"max":{"1min":0.90600000000000003,"5min":1.296,"15min":1.3049999999999999},"last":0.56999999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575037,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302972,"kb_used_data":1588,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068868,"statfs":{"total":96636764160,"available":96326520832,"internally_reserved":0,"allocated":1626112,"data_stored":1109922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.28499999999999998,"5min":0.23100000000000001,"15min":0.214},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":1.284},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.46000000000000002,"15min":0.45600000000000002},"min":{"1min":0.29099999999999998,"5min":0.28899999999999998,"15min":0.27300000000000002},"max":{"1min":0.60899999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.40000000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42099999999999999,"15min":0.42999999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.26700000000000002},"max":{"1min":0.57299999999999995,"5min":0.89600000000000002,"15min":1.2230000000000001},"last":0.315},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.38300000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":1.2430000000000001},"last":0.45600000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46899999999999997,"15min":0.46700000000000003},"min":{"1min":0.36799999999999999,"5min":0.30599999999999999,"15min":0.26800000000000002},"max":{"1min":0.68100000000000005,"5min":0.80700000000000005,"15min":1.0980000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.49399999999999999,"15min":0.49099999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.27000000000000002},"max":{"1min":0.64000000000000001,"5min":1.198,"15min":1.198},"last":0.439}]},{"osd":4,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.64300000000000002,"15min":0.65200000000000002},"min":{"1min":0.46899999999999997,"5min":0.42399999999999999,"15min":0.372},"max":{"1min":0.78700000000000003,"5min":0.93300000000000005,"15min":1.133},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64700000000000002,"15min":0.65900000000000003},"min":{"1min":0.438,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85299999999999998,"5min":0.94299999999999995,"15min":1.0169999999999999},"last":0.71199999999999997}]},{"osd":5,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65100000000000002,"15min":0.66300000000000003},"min":{"1min":0.52600000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79800000000000004,"5min":0.95799999999999996,"15min":1.0880000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67400000000000004,"15min":0.68200000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.41699999999999998},"max":{"1min":0.93999999999999995,"5min":1.012,"15min":1.169},"last":0.59299999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65500000000000003,"15min":0.67700000000000005},"min":{"1min":0.505,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89800000000000002,"5min":0.91600000000000004,"15min":1.071},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.68400000000000005,"15min":0.69999999999999996},"min":{"1min":0.48299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.94599999999999995,"5min":1.028,"15min":1.1160000000000001},"last":0.69899999999999995}]},{"osd":7,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":1.004,"15min":0.82599999999999996},"min":{"1min":0.54500000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.91500000000000004,"5min":25.965,"15min":25.965},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.73899999999999999,"5min":1.109,"15min":0.86599999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.433},"max":{"1min":0.93799999999999994,"5min":33.085000000000001,"15min":33.085000000000001},"last":0.68200000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575037,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306488,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065352,"statfs":{"total":96636764160,"available":96322920448,"internally_reserved":0,"allocated":1032192,"data_stored":519554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.39400000000000002,"15min":0.41999999999999998},"min":{"1min":0.29299999999999998,"5min":0.25800000000000001,"15min":0.22600000000000001},"max":{"1min":0.502,"5min":0.63400000000000001,"15min":9.0540000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.24299999999999999},"max":{"1min":0.56899999999999995,"5min":0.68500000000000005,"15min":4.1970000000000001},"last":0.78700000000000003}]},{"osd":1,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42699999999999999,"15min":0.42499999999999999},"min":{"1min":0.33300000000000002,"5min":0.248,"15min":0.221},"max":{"1min":0.55300000000000005,"5min":0.70499999999999996,"15min":0.82199999999999995},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.498,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":0.629,"5min":0.65900000000000003,"15min":12.157},"last":0.67200000000000004}]},{"osd":3,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47799999999999998,"15min":0.48599999999999999},"min":{"1min":0.38800000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.59299999999999997,"5min":0.65500000000000003,"15min":0.86499999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.47599999999999998,"15min":0.499},"min":{"1min":0.34599999999999997,"5min":0.32200000000000001,"15min":0.30399999999999999},"max":{"1min":0.60199999999999998,"5min":0.71099999999999997,"15min":9.0879999999999992},"last":0.749}]},{"osd":4,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65000000000000002,"15min":0.66600000000000004},"min":{"1min":0.499,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.86599999999999999,"5min":1.087,"15min":9.0150000000000006},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.65800000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.371},"max":{"1min":0.80200000000000005,"5min":0.98099999999999998,"15min":4.2480000000000002},"last":0.82799999999999996}]},{"osd":5,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78600000000000003,"5min":1.0680000000000001,"15min":4.3200000000000003},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68899999999999995,"15min":0.68200000000000005},"min":{"1min":0.58899999999999997,"5min":0.39800000000000002,"15min":0.35199999999999998},"max":{"1min":0.82399999999999995,"5min":0.95199999999999996,"15min":1.085},"last":0.876}]},{"osd":6,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68899999999999995,"15min":0.68400000000000005},"min":{"1min":0.51500000000000001,"5min":0.439,"15min":0.40899999999999997},"max":{"1min":0.79400000000000004,"5min":0.995,"15min":1.1080000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69199999999999995,"15min":0.71299999999999997},"min":{"1min":0.46200000000000002,"5min":0.443,"15min":0.38800000000000001},"max":{"1min":0.83999999999999997,"5min":1.1299999999999999,"15min":8.9700000000000006},"last":0.83799999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70799999999999996,"15min":0.71699999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.85599999999999998,"5min":1.0029999999999999,"15min":4.282},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.57499999999999996,"5min":0.48199999999999998,"15min":0.42099999999999999},"max":{"1min":0.87,"5min":1.052,"15min":1.139},"last":0.98199999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575036,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306488,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065352,"statfs":{"total":96636764160,"available":96322920448,"internally_reserved":0,"allocated":1032192,"data_stored":519554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40300000000000002,"15min":0.39800000000000002},"min":{"1min":0.32300000000000001,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.56999999999999995,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44900000000000001,"15min":0.48099999999999998},"min":{"1min":0.315,"5min":0.255,"15min":0.255},"max":{"1min":0.59299999999999997,"5min":1.2290000000000001,"15min":11.968},"last":0.32100000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.496},"min":{"1min":0.33200000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.623,"5min":1.633,"15min":12.050000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.48899999999999999,"15min":0.48599999999999999},"min":{"1min":0.378,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.59999999999999998,"5min":1.0760000000000001,"15min":1.2150000000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.49299999999999999,"15min":0.51900000000000002},"min":{"1min":0.39800000000000002,"5min":0.29699999999999999,"15min":0.28999999999999998},"max":{"1min":0.70399999999999996,"5min":1.4059999999999999,"15min":12.234},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46899999999999997,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.59499999999999997,"5min":1.1870000000000001,"15min":11.925000000000001},"last":0.435}]},{"osd":4,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69599999999999995,"15min":0.70599999999999996},"min":{"1min":0.56999999999999995,"5min":0.46999999999999997,"15min":0.40200000000000002},"max":{"1min":0.84899999999999998,"5min":3.306,"15min":12.099},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.66400000000000003,"15min":0.66600000000000004},"min":{"1min":0.59599999999999997,"5min":0.40200000000000002,"15min":0.379},"max":{"1min":0.82999999999999996,"5min":0.95799999999999996,"15min":1.0649999999999999},"last":0.65600000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.66500000000000004,"15min":0.70699999999999996},"min":{"1min":0.56299999999999994,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.79800000000000004,"5min":1.593,"15min":12.199999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68999999999999995,"15min":0.67800000000000005},"min":{"1min":0.53400000000000003,"5min":0.45300000000000001,"15min":0.436},"max":{"1min":0.92300000000000004,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.59099999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69499999999999995,"15min":0.72099999999999997},"min":{"1min":0.56100000000000005,"5min":0.442,"15min":0.442},"max":{"1min":0.88800000000000001,"5min":1.506,"15min":12.132999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70099999999999996,"15min":0.69799999999999995},"min":{"1min":0.54600000000000004,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.88200000000000001,"5min":1.427,"15min":1.427},"last":0.624}]},{"osd":7,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.72699999999999998,"15min":0.70299999999999996},"min":{"1min":0.52500000000000002,"5min":0.48499999999999999,"15min":0.42499999999999999},"max":{"1min":0.86199999999999999,"5min":3.238,"15min":3.238},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.78200000000000003},"min":{"1min":0.625,"5min":0.48899999999999999,"15min":0.48699999999999999},"max":{"1min":0.91300000000000003,"5min":3.351,"15min":12.167},"last":0.72199999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575038,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298296,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073544,"statfs":{"total":96636764160,"available":96331309056,"internally_reserved":0,"allocated":1032192,"data_stored":514090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57599999999999996,"15min":0.58399999999999996},"min":{"1min":0.439,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78300000000000003,"5min":1.21,"15min":1.21},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57499999999999996,"15min":0.58399999999999996},"min":{"1min":0.44900000000000001,"5min":0.309,"15min":0.28100000000000003},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":1.161},"last":0.49399999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.44900000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":1.069},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.59399999999999997},"min":{"1min":0.47199999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":1.085},"last":0.42199999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56299999999999994,"15min":0.57499999999999996},"min":{"1min":0.48499999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76500000000000001,"5min":0.81100000000000005,"15min":1.052},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60699999999999998,"15min":0.61199999999999999},"min":{"1min":0.41399999999999998,"5min":0.39900000000000002,"15min":0.373},"max":{"1min":0.81599999999999995,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.54700000000000004}]},{"osd":3,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58199999999999996,"15min":0.59399999999999997},"min":{"1min":0.41299999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.78300000000000003,"5min":0.83499999999999996,"15min":1.1299999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79100000000000004,"5min":1.05,"15min":1.1419999999999999},"last":0.56499999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.317,"5min":0.25700000000000001,"15min":0.247},"max":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.63900000000000001}]},{"osd":6,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48499999999999999,"15min":0.47799999999999998},"min":{"1min":0.35799999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.73899999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53300000000000003,"15min":0.53000000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.311},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.95699999999999996},"last":0.65900000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.35799999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.57699999999999996},"min":{"1min":0.42499999999999999,"5min":0.41799999999999998,"15min":0.32200000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.879},"last":0.61599999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575038,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298300,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073540,"statfs":{"total":96636764160,"available":96331304960,"internally_reserved":0,"allocated":1036288,"data_stored":519576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.58899999999999997,"15min":0.57599999999999996},"min":{"1min":0.46300000000000002,"5min":0.29999999999999999,"15min":0.26000000000000001},"max":{"1min":0.78900000000000003,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57199999999999995,"15min":0.58399999999999996},"min":{"1min":0.45400000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77400000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.64700000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.625,"15min":0.623},"min":{"1min":0.45500000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79500000000000004,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.64300000000000002,"15min":0.63300000000000001},"min":{"1min":0.499,"5min":0.435,"15min":0.379},"max":{"1min":0.80500000000000005,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.71199999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.46700000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77000000000000002,"5min":0.83699999999999997,"15min":0.83799999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60399999999999998,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81100000000000005,"5min":2.0600000000000001,"15min":6.4589999999999996},"last":0.59399999999999997}]},{"osd":3,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.50700000000000001,"5min":0.35999999999999999,"15min":0.314},"max":{"1min":0.755,"5min":0.86199999999999999,"15min":0.88400000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.60599999999999998,"15min":0.61499999999999999},"min":{"1min":0.47499999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77900000000000003,"5min":2.5630000000000002,"15min":6.0629999999999997},"last":0.52100000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.47399999999999998,"15min":0.46999999999999997},"min":{"1min":0.35599999999999998,"5min":0.28100000000000003,"15min":0.26300000000000001},"max":{"1min":0.56599999999999995,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.503,"15min":0.495},"min":{"1min":0.36899999999999999,"5min":0.33000000000000002,"15min":0.28100000000000003},"max":{"1min":0.69999999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.42699999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.31,"15min":0.27000000000000002},"max":{"1min":0.66600000000000004,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53200000000000003,"15min":0.52700000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.29299999999999998},"max":{"1min":0.71899999999999997,"5min":1.99,"15min":1.99},"last":0.45400000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.54900000000000004},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.76200000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33300000000000002},"max":{"1min":0.81499999999999995,"5min":0.82199999999999995,"15min":0.83099999999999996},"last":0.61899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:45:23.968 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:45:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.136+0000 7fc116746700 1 Processor -- start 2024-01-26T23:45:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.136+0000 7fc116746700 1 -- start start 2024-01-26T23:45:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.137+0000 7fc116746700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc110159b30 0x7fc1100b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.137+0000 7fc116746700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc1100b10f0 0x7fc1100a9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.137+0000 7fc116746700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11006f940 con 0x7fc110159760 2024-01-26T23:45:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.137+0000 7fc116746700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1100b1530 con 0x7fc110159b30 2024-01-26T23:45:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.137+0000 7fc116746700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1100b3230 con 0x7fc1100b10f0 2024-01-26T23:45:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.137+0000 7fc10f7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc110159b30 0x7fc1100b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.137+0000 7fc10f7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc110159b30 0x7fc1100b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37206/0 (socket says 172.21.15.46:37206) 2024-01-26T23:45:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.137+0000 7fc10f7fe700 1 -- 172.21.15.46:0/1114458958 learned_addr learned my addr 172.21.15.46:0/1114458958 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.138+0000 7fc114ce3700 1 --2- 172.21.15.46:0/1114458958 >> v2:172.21.15.106:3300/0 conn(0x7fc1100b10f0 0x7fc1100a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.138+0000 7fc10f7fe700 1 -- 172.21.15.46:0/1114458958 >> v2:172.21.15.106:3300/0 conn(0x7fc1100b10f0 msgr2=0x7fc1100a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.138+0000 7fc10f7fe700 1 --2- 172.21.15.46:0/1114458958 >> v2:172.21.15.106:3300/0 conn(0x7fc1100b10f0 0x7fc1100a9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.138+0000 7fc10f7fe700 1 -- 172.21.15.46:0/1114458958 >> v1:172.21.15.106:6789/0 conn(0x7fc110159760 legacy=0x7fc1100b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.138+0000 7fc10f7fe700 1 -- 172.21.15.46:0/1114458958 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11006ed30 con 0x7fc110159b30 2024-01-26T23:45:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.138+0000 7fc10f7fe700 1 --2- 172.21.15.46:0/1114458958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc110159b30 0x7fc1100b0bb0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc0f80099b0 tx=0x7fc0f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5cc0c4317452fbf server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.138+0000 7fc10effd700 1 -- 172.21.15.46:0/1114458958 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f800b820 con 0x7fc110159b30 2024-01-26T23:45:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.138+0000 7fc10effd700 1 -- 172.21.15.46:0/1114458958 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f800b980 con 0x7fc110159b30 2024-01-26T23:45:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.138+0000 7fc10effd700 1 -- 172.21.15.46:0/1114458958 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f800bd20 con 0x7fc110159b30 2024-01-26T23:45:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.139+0000 7fc116746700 1 -- 172.21.15.46:0/1114458958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc110159b30 msgr2=0x7fc1100b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.139+0000 7fc116746700 1 --2- 172.21.15.46:0/1114458958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc110159b30 0x7fc1100b0bb0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc0f80099b0 tx=0x7fc0f8009250 comp rx=0 tx=0).stop 2024-01-26T23:45:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.139+0000 7fc116746700 1 -- 172.21.15.46:0/1114458958 shutdown_connections 2024-01-26T23:45:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.139+0000 7fc116746700 1 --2- 172.21.15.46:0/1114458958 >> v2:172.21.15.106:3300/0 conn(0x7fc1100b10f0 0x7fc1100a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.139+0000 7fc116746700 1 --2- 172.21.15.46:0/1114458958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc110159b30 0x7fc1100b0bb0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.139+0000 7fc116746700 1 -- 172.21.15.46:0/1114458958 >> 172.21.15.46:0/1114458958 conn(0x7fc110069510 msgr2=0x7fc1100aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:24.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.139+0000 7fc116746700 1 -- 172.21.15.46:0/1114458958 shutdown_connections 2024-01-26T23:45:24.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.139+0000 7fc116746700 1 -- 172.21.15.46:0/1114458958 wait complete. 2024-01-26T23:45:24.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.140+0000 7fc116746700 1 Processor -- start 2024-01-26T23:45:24.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.140+0000 7fc116746700 1 -- start start 2024-01-26T23:45:24.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.141+0000 7fc116746700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1100b10f0 0x7fc1100c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.141+0000 7fc116746700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc110159760 0x7fc1100c4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.141+0000 7fc116746700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc110159b30 0x7fc1100c4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.141+0000 7fc116746700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11015d150 con 0x7fc110159760 2024-01-26T23:45:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.141+0000 7fc116746700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11015c540 con 0x7fc1100b10f0 2024-01-26T23:45:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.141+0000 7fc116746700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11015c820 con 0x7fc110159b30 2024-01-26T23:45:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.141+0000 7fc10f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc110159760 0x7fc1100c4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.141+0000 7fc10f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc110159760 0x7fc1100c4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37212/0 (socket says 172.21.15.46:37212) 2024-01-26T23:45:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.141+0000 7fc10f7fe700 1 -- 172.21.15.46:0/2742606522 learned_addr learned my addr 172.21.15.46:0/2742606522 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.141+0000 7fc10ffff700 1 --2- 172.21.15.46:0/2742606522 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1100b10f0 0x7fc1100c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.141+0000 7fc114ce3700 1 --2- 172.21.15.46:0/2742606522 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc110159b30 0x7fc1100c4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.141+0000 7fc10f7fe700 1 -- 172.21.15.46:0/2742606522 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc110159b30 msgr2=0x7fc1100c4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.141+0000 7fc10f7fe700 1 --2- 172.21.15.46:0/2742606522 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc110159b30 0x7fc1100c4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.141+0000 7fc10f7fe700 1 -- 172.21.15.46:0/2742606522 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1100b10f0 msgr2=0x7fc1100c2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:24.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.141+0000 7fc10f7fe700 1 --2- 172.21.15.46:0/2742606522 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1100b10f0 0x7fc1100c2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.141+0000 7fc10f7fe700 1 -- 172.21.15.46:0/2742606522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f80094f0 con 0x7fc110159760 2024-01-26T23:45:24.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.142+0000 7fc10f7fe700 1 --2- 172.21.15.46:0/2742606522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc110159760 0x7fc1100c4020 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc0f80094c0 tx=0x7fc0f8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:24.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.142+0000 7fc107fff700 1 -- 172.21.15.46:0/2742606522 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f801a450 con 0x7fc110159760 2024-01-26T23:45:24.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.142+0000 7fc107fff700 1 -- 172.21.15.46:0/2742606522 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f801a5b0 con 0x7fc110159760 2024-01-26T23:45:24.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.142+0000 7fc116746700 1 -- 172.21.15.46:0/2742606522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11015cb00 con 0x7fc110159760 2024-01-26T23:45:24.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.142+0000 7fc107fff700 1 -- 172.21.15.46:0/2742606522 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f801a970 con 0x7fc110159760 2024-01-26T23:45:24.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.142+0000 7fc116746700 1 -- 172.21.15.46:0/2742606522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1100c34b0 con 0x7fc110159760 2024-01-26T23:45:24.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.143+0000 7fc116746700 1 -- 172.21.15.46:0/2742606522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc11015d2c0 con 0x7fc110159760 2024-01-26T23:45:24.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.143+0000 7fc107fff700 1 -- 172.21.15.46:0/2742606522 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc0f801ad20 con 0x7fc110159760 2024-01-26T23:45:24.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.144+0000 7fc107fff700 1 --2- 172.21.15.46:0/2742606522 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc0f0066f60 0x7fc0f0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:24.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.144+0000 7fc107fff700 1 -- 172.21.15.46:0/2742606522 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fc0f80862d0 con 0x7fc110159760 2024-01-26T23:45:24.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.144+0000 7fc10ffff700 1 --2- 172.21.15.46:0/2742606522 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc0f0066f60 0x7fc0f0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:24.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.144+0000 7fc10ffff700 1 --2- 172.21.15.46:0/2742606522 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc0f0066f60 0x7fc0f0069410 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc100003860 tx=0x7fc100009040 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.148+0000 7fc107fff700 1 -- 172.21.15.46:0/2742606522 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc0f80532f0 con 0x7fc110159760 2024-01-26T23:45:24.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.350+0000 7fc116746700 1 -- 172.21.15.46:0/2742606522 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc11004c710 con 0x7fc0f0066f60 2024-01-26T23:45:24.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.358+0000 7fc107fff700 1 -- 172.21.15.46:0/2742606522 <== mgr.5660 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289575 (secure 0 0 0) 0x7fc11004c710 con 0x7fc0f0066f60 2024-01-26T23:45:24.364 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:24.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.361+0000 7fc116746700 1 -- 172.21.15.46:0/2742606522 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc0f0066f60 msgr2=0x7fc0f0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:24.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.361+0000 7fc116746700 1 --2- 172.21.15.46:0/2742606522 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc0f0066f60 0x7fc0f0069410 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc100003860 tx=0x7fc100009040 comp rx=0 tx=0).stop 2024-01-26T23:45:24.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.361+0000 7fc116746700 1 -- 172.21.15.46:0/2742606522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc110159760 msgr2=0x7fc1100c4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:24.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.361+0000 7fc116746700 1 --2- 172.21.15.46:0/2742606522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc110159760 0x7fc1100c4020 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc0f80094c0 tx=0x7fc0f8017000 comp rx=0 tx=0).stop 2024-01-26T23:45:24.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.362+0000 7fc116746700 1 -- 172.21.15.46:0/2742606522 shutdown_connections 2024-01-26T23:45:24.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.362+0000 7fc116746700 1 --2- 172.21.15.46:0/2742606522 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc0f0066f60 0x7fc0f0069410 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.362+0000 7fc116746700 1 --2- 172.21.15.46:0/2742606522 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc110159b30 0x7fc1100c4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.362+0000 7fc116746700 1 --2- 172.21.15.46:0/2742606522 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1100b10f0 0x7fc1100c2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.362+0000 7fc116746700 1 --2- 172.21.15.46:0/2742606522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc110159760 0x7fc1100c4020 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.362+0000 7fc116746700 1 -- 172.21.15.46:0/2742606522 >> 172.21.15.46:0/2742606522 conn(0x7fc110069510 msgr2=0x7fc110158900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:24.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.362+0000 7fc116746700 1 -- 172.21.15.46:0/2742606522 shutdown_connections 2024-01-26T23:45:24.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.362+0000 7fc116746700 1 -- 172.21.15.46:0/2742606522 wait complete. 2024-01-26T23:45:24.367 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:45:24.377 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-26T23:45:23.675532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2424444,"kb_used_data":9276,"kb_used_omap":0,"kb_used_meta":2415104,"kb_avail":752550276,"statfs":{"total":773094113280,"available":770611482624,"internally_reserved":0,"allocated":9498624,"data_stored":5334780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2473066496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.936137"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.772227+0000","last_change":"2024-01-26T23:45:19.769451+0000","last_active":"2024-01-26T23:45:19.772227+0000","last_peered":"2024-01-26T23:45:19.772227+0000","last_clean":"2024-01-26T23:45:19.772227+0000","last_became_active":"2024-01-26T23:45:19.769137+0000","last_became_peered":"2024-01-26T23:45:19.769137+0000","last_unstale":"2024-01-26T23:45:19.772227+0000","last_undegraded":"2024-01-26T23:45:19.772227+0000","last_fullsized":"2024-01-26T23:45:19.772227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:09:30.529877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.765394+0000","last_change":"2024-01-26T23:45:19.763048+0000","last_active":"2024-01-26T23:45:19.765394+0000","last_peered":"2024-01-26T23:45:19.765394+0000","last_clean":"2024-01-26T23:45:19.765394+0000","last_became_active":"2024-01-26T23:45:19.762735+0000","last_became_peered":"2024-01-26T23:45:19.762735+0000","last_unstale":"2024-01-26T23:45:19.765394+0000","last_undegraded":"2024-01-26T23:45:19.765394+0000","last_fullsized":"2024-01-26T23:45:19.765394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:39:33.812220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.790586+0000","last_change":"2024-01-26T23:45:19.786219+0000","last_active":"2024-01-26T23:45:19.790586+0000","last_peered":"2024-01-26T23:45:19.790586+0000","last_clean":"2024-01-26T23:45:19.790586+0000","last_became_active":"2024-01-26T23:45:19.785929+0000","last_became_peered":"2024-01-26T23:45:19.785929+0000","last_unstale":"2024-01-26T23:45:19.790586+0000","last_undegraded":"2024-01-26T23:45:19.790586+0000","last_fullsized":"2024-01-26T23:45:19.790586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:04:01.842189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.784596+0000","last_change":"2024-01-26T23:45:19.777614+0000","last_active":"2024-01-26T23:45:19.784596+0000","last_peered":"2024-01-26T23:45:19.784596+0000","last_clean":"2024-01-26T23:45:19.784596+0000","last_became_active":"2024-01-26T23:45:19.777257+0000","last_became_peered":"2024-01-26T23:45:19.777257+0000","last_unstale":"2024-01-26T23:45:19.784596+0000","last_undegraded":"2024-01-26T23:45:19.784596+0000","last_fullsized":"2024-01-26T23:45:19.784596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:54:09.399022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.782578+0000","last_change":"2024-01-26T23:45:19.777484+0000","last_active":"2024-01-26T23:45:19.782578+0000","last_peered":"2024-01-26T23:45:19.782578+0000","last_clean":"2024-01-26T23:45:19.782578+0000","last_became_active":"2024-01-26T23:45:19.777225+0000","last_became_peered":"2024-01-26T23:45:19.777225+0000","last_unstale":"2024-01-26T23:45:19.782578+0000","last_undegraded":"2024-01-26T23:45:19.782578+0000","last_fullsized":"2024-01-26T23:45:19.782578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:39:33.812220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.774917+0000","last_change":"2024-01-26T23:45:19.773714+0000","last_active":"2024-01-26T23:45:19.774917+0000","last_peered":"2024-01-26T23:45:19.774917+0000","last_clean":"2024-01-26T23:45:19.774917+0000","last_became_active":"2024-01-26T23:45:19.773375+0000","last_became_peered":"2024-01-26T23:45:19.773375+0000","last_unstale":"2024-01-26T23:45:19.774917+0000","last_undegraded":"2024-01-26T23:45:19.774917+0000","last_fullsized":"2024-01-26T23:45:19.774917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:52:25.112546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.784170+0000","last_change":"2024-01-26T23:45:19.780704+0000","last_active":"2024-01-26T23:45:19.784170+0000","last_peered":"2024-01-26T23:45:19.784170+0000","last_clean":"2024-01-26T23:45:19.784170+0000","last_became_active":"2024-01-26T23:45:19.780384+0000","last_became_peered":"2024-01-26T23:45:19.780384+0000","last_unstale":"2024-01-26T23:45:19.784170+0000","last_undegraded":"2024-01-26T23:45:19.784170+0000","last_fullsized":"2024-01-26T23:45:19.784170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:03:21.787299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.789515+0000","last_change":"2024-01-26T23:45:19.787991+0000","last_active":"2024-01-26T23:45:19.789515+0000","last_peered":"2024-01-26T23:45:19.789515+0000","last_clean":"2024-01-26T23:45:19.789515+0000","last_became_active":"2024-01-26T23:45:19.787701+0000","last_became_peered":"2024-01-26T23:45:19.787701+0000","last_unstale":"2024-01-26T23:45:19.789515+0000","last_undegraded":"2024-01-26T23:45:19.789515+0000","last_fullsized":"2024-01-26T23:45:19.789515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:39:33.812220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.790434+0000","last_change":"2024-01-26T23:45:19.786260+0000","last_active":"2024-01-26T23:45:19.790434+0000","last_peered":"2024-01-26T23:45:19.790434+0000","last_clean":"2024-01-26T23:45:19.790434+0000","last_became_active":"2024-01-26T23:45:19.785954+0000","last_became_peered":"2024-01-26T23:45:19.785954+0000","last_unstale":"2024-01-26T23:45:19.790434+0000","last_undegraded":"2024-01-26T23:45:19.790434+0000","last_fullsized":"2024-01-26T23:45:19.790434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:56:09.047270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.790253+0000","last_change":"2024-01-26T23:45:19.786309+0000","last_active":"2024-01-26T23:45:19.790253+0000","last_peered":"2024-01-26T23:45:19.790253+0000","last_clean":"2024-01-26T23:45:19.790253+0000","last_became_active":"2024-01-26T23:45:19.786033+0000","last_became_peered":"2024-01-26T23:45:19.786033+0000","last_unstale":"2024-01-26T23:45:19.790253+0000","last_undegraded":"2024-01-26T23:45:19.790253+0000","last_fullsized":"2024-01-26T23:45:19.790253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:05:43.312820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.784551+0000","last_change":"2024-01-26T23:45:19.778523+0000","last_active":"2024-01-26T23:45:19.784551+0000","last_peered":"2024-01-26T23:45:19.784551+0000","last_clean":"2024-01-26T23:45:19.784551+0000","last_became_active":"2024-01-26T23:45:19.778171+0000","last_became_peered":"2024-01-26T23:45:19.778171+0000","last_unstale":"2024-01-26T23:45:19.784551+0000","last_undegraded":"2024-01-26T23:45:19.784551+0000","last_fullsized":"2024-01-26T23:45:19.784551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:19:00.324107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.789962+0000","last_change":"2024-01-26T23:45:19.787971+0000","last_active":"2024-01-26T23:45:19.789962+0000","last_peered":"2024-01-26T23:45:19.789962+0000","last_clean":"2024-01-26T23:45:19.789962+0000","last_became_active":"2024-01-26T23:45:19.787669+0000","last_became_peered":"2024-01-26T23:45:19.787669+0000","last_unstale":"2024-01-26T23:45:19.789962+0000","last_undegraded":"2024-01-26T23:45:19.789962+0000","last_fullsized":"2024-01-26T23:45:19.789962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:09:50.055556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.779761+0000","last_change":"2024-01-26T23:45:19.776894+0000","last_active":"2024-01-26T23:45:19.779761+0000","last_peered":"2024-01-26T23:45:19.779761+0000","last_clean":"2024-01-26T23:45:19.779761+0000","last_became_active":"2024-01-26T23:45:19.776596+0000","last_became_peered":"2024-01-26T23:45:19.776596+0000","last_unstale":"2024-01-26T23:45:19.779761+0000","last_undegraded":"2024-01-26T23:45:19.779761+0000","last_fullsized":"2024-01-26T23:45:19.779761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:57:02.783696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.791675+0000","last_change":"2024-01-26T23:45:19.788073+0000","last_active":"2024-01-26T23:45:19.791675+0000","last_peered":"2024-01-26T23:45:19.791675+0000","last_clean":"2024-01-26T23:45:19.791675+0000","last_became_active":"2024-01-26T23:45:19.787750+0000","last_became_peered":"2024-01-26T23:45:19.787750+0000","last_unstale":"2024-01-26T23:45:19.791675+0000","last_undegraded":"2024-01-26T23:45:19.791675+0000","last_fullsized":"2024-01-26T23:45:19.791675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:39:38.641898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.793847+0000","last_change":"2024-01-26T23:45:19.789531+0000","last_active":"2024-01-26T23:45:19.793847+0000","last_peered":"2024-01-26T23:45:19.793847+0000","last_clean":"2024-01-26T23:45:19.793847+0000","last_became_active":"2024-01-26T23:45:19.789262+0000","last_became_peered":"2024-01-26T23:45:19.789262+0000","last_unstale":"2024-01-26T23:45:19.793847+0000","last_undegraded":"2024-01-26T23:45:19.793847+0000","last_fullsized":"2024-01-26T23:45:19.793847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:09:30.529877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.773591+0000","last_change":"2024-01-26T23:45:19.770070+0000","last_active":"2024-01-26T23:45:19.773591+0000","last_peered":"2024-01-26T23:45:19.773591+0000","last_clean":"2024-01-26T23:45:19.773591+0000","last_became_active":"2024-01-26T23:45:19.769337+0000","last_became_peered":"2024-01-26T23:45:19.769337+0000","last_unstale":"2024-01-26T23:45:19.773591+0000","last_undegraded":"2024-01-26T23:45:19.773591+0000","last_fullsized":"2024-01-26T23:45:19.773591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:02:52.183572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.785438+0000","last_change":"2024-01-26T23:45:19.783397+0000","last_active":"2024-01-26T23:45:19.785438+0000","last_peered":"2024-01-26T23:45:19.785438+0000","last_clean":"2024-01-26T23:45:19.785438+0000","last_became_active":"2024-01-26T23:45:19.781037+0000","last_became_peered":"2024-01-26T23:45:19.781037+0000","last_unstale":"2024-01-26T23:45:19.785438+0000","last_undegraded":"2024-01-26T23:45:19.785438+0000","last_fullsized":"2024-01-26T23:45:19.785438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:54:09.399022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778402+0000","last_change":"2024-01-26T23:45:19.776277+0000","last_active":"2024-01-26T23:45:19.778402+0000","last_peered":"2024-01-26T23:45:19.778402+0000","last_clean":"2024-01-26T23:45:19.778402+0000","last_became_active":"2024-01-26T23:45:19.775964+0000","last_became_peered":"2024-01-26T23:45:19.775964+0000","last_unstale":"2024-01-26T23:45:19.778402+0000","last_undegraded":"2024-01-26T23:45:19.778402+0000","last_fullsized":"2024-01-26T23:45:19.778402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:09:50.055556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778633+0000","last_change":"2024-01-26T23:45:19.774575+0000","last_active":"2024-01-26T23:45:19.778633+0000","last_peered":"2024-01-26T23:45:19.778633+0000","last_clean":"2024-01-26T23:45:19.778633+0000","last_became_active":"2024-01-26T23:45:19.774294+0000","last_became_peered":"2024-01-26T23:45:19.774294+0000","last_unstale":"2024-01-26T23:45:19.778633+0000","last_undegraded":"2024-01-26T23:45:19.778633+0000","last_fullsized":"2024-01-26T23:45:19.778633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:20:13.593058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.782044+0000","last_change":"2024-01-26T23:45:19.777529+0000","last_active":"2024-01-26T23:45:19.782044+0000","last_peered":"2024-01-26T23:45:19.782044+0000","last_clean":"2024-01-26T23:45:19.782044+0000","last_became_active":"2024-01-26T23:45:19.777190+0000","last_became_peered":"2024-01-26T23:45:19.777190+0000","last_unstale":"2024-01-26T23:45:19.782044+0000","last_undegraded":"2024-01-26T23:45:19.782044+0000","last_fullsized":"2024-01-26T23:45:19.782044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:52:35.855244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.777619+0000","last_change":"2024-01-26T23:45:19.768338+0000","last_active":"2024-01-26T23:45:19.777619+0000","last_peered":"2024-01-26T23:45:19.777619+0000","last_clean":"2024-01-26T23:45:19.777619+0000","last_became_active":"2024-01-26T23:45:19.768028+0000","last_became_peered":"2024-01-26T23:45:19.768028+0000","last_unstale":"2024-01-26T23:45:19.777619+0000","last_undegraded":"2024-01-26T23:45:19.777619+0000","last_fullsized":"2024-01-26T23:45:19.777619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:54:09.399022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.785709+0000","last_change":"2024-01-26T23:45:19.784566+0000","last_active":"2024-01-26T23:45:19.785709+0000","last_peered":"2024-01-26T23:45:19.785709+0000","last_clean":"2024-01-26T23:45:19.785709+0000","last_became_active":"2024-01-26T23:45:19.784229+0000","last_became_peered":"2024-01-26T23:45:19.784229+0000","last_unstale":"2024-01-26T23:45:19.785709+0000","last_undegraded":"2024-01-26T23:45:19.785709+0000","last_fullsized":"2024-01-26T23:45:19.785709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:35:48.959656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.782416+0000","last_change":"2024-01-26T23:45:19.781858+0000","last_active":"2024-01-26T23:45:19.782416+0000","last_peered":"2024-01-26T23:45:19.782416+0000","last_clean":"2024-01-26T23:45:19.782416+0000","last_became_active":"2024-01-26T23:45:19.781507+0000","last_became_peered":"2024-01-26T23:45:19.781507+0000","last_unstale":"2024-01-26T23:45:19.782416+0000","last_undegraded":"2024-01-26T23:45:19.782416+0000","last_fullsized":"2024-01-26T23:45:19.782416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:36.995172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.787633+0000","last_change":"2024-01-26T23:45:19.783280+0000","last_active":"2024-01-26T23:45:19.787633+0000","last_peered":"2024-01-26T23:45:19.787633+0000","last_clean":"2024-01-26T23:45:19.787633+0000","last_became_active":"2024-01-26T23:45:19.782929+0000","last_became_peered":"2024-01-26T23:45:19.782929+0000","last_unstale":"2024-01-26T23:45:19.787633+0000","last_undegraded":"2024-01-26T23:45:19.787633+0000","last_fullsized":"2024-01-26T23:45:19.787633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:55:05.236611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.779688+0000","last_change":"2024-01-26T23:45:19.773680+0000","last_active":"2024-01-26T23:45:19.779688+0000","last_peered":"2024-01-26T23:45:19.779688+0000","last_clean":"2024-01-26T23:45:19.779688+0000","last_became_active":"2024-01-26T23:45:19.773415+0000","last_became_peered":"2024-01-26T23:45:19.773415+0000","last_unstale":"2024-01-26T23:45:19.779688+0000","last_undegraded":"2024-01-26T23:45:19.779688+0000","last_fullsized":"2024-01-26T23:45:19.779688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:24.303858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.779840+0000","last_change":"2024-01-26T23:45:19.773746+0000","last_active":"2024-01-26T23:45:19.779840+0000","last_peered":"2024-01-26T23:45:19.779840+0000","last_clean":"2024-01-26T23:45:19.779840+0000","last_became_active":"2024-01-26T23:45:19.773469+0000","last_became_peered":"2024-01-26T23:45:19.773469+0000","last_unstale":"2024-01-26T23:45:19.779840+0000","last_undegraded":"2024-01-26T23:45:19.779840+0000","last_fullsized":"2024-01-26T23:45:19.779840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:19:00.324107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.784527+0000","last_change":"2024-01-26T23:45:19.778462+0000","last_active":"2024-01-26T23:45:19.784527+0000","last_peered":"2024-01-26T23:45:19.784527+0000","last_clean":"2024-01-26T23:45:19.784527+0000","last_became_active":"2024-01-26T23:45:19.778141+0000","last_became_peered":"2024-01-26T23:45:19.778141+0000","last_unstale":"2024-01-26T23:45:19.784527+0000","last_undegraded":"2024-01-26T23:45:19.784527+0000","last_fullsized":"2024-01-26T23:45:19.784527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:05:43.312820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.769929+0000","last_change":"2024-01-26T23:45:19.765821+0000","last_active":"2024-01-26T23:45:19.769929+0000","last_peered":"2024-01-26T23:45:19.769929+0000","last_clean":"2024-01-26T23:45:19.769929+0000","last_became_active":"2024-01-26T23:45:19.765517+0000","last_became_peered":"2024-01-26T23:45:19.765517+0000","last_unstale":"2024-01-26T23:45:19.769929+0000","last_undegraded":"2024-01-26T23:45:19.769929+0000","last_fullsized":"2024-01-26T23:45:19.769929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:29:01.653387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.791894+0000","last_change":"2024-01-26T23:45:19.788341+0000","last_active":"2024-01-26T23:45:19.791894+0000","last_peered":"2024-01-26T23:45:19.791894+0000","last_clean":"2024-01-26T23:45:19.791894+0000","last_became_active":"2024-01-26T23:45:19.788092+0000","last_became_peered":"2024-01-26T23:45:19.788092+0000","last_unstale":"2024-01-26T23:45:19.791894+0000","last_undegraded":"2024-01-26T23:45:19.791894+0000","last_fullsized":"2024-01-26T23:45:19.791894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:24.303858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.782081+0000","last_change":"2024-01-26T23:45:19.773137+0000","last_active":"2024-01-26T23:45:19.782081+0000","last_peered":"2024-01-26T23:45:19.782081+0000","last_clean":"2024-01-26T23:45:19.782081+0000","last_became_active":"2024-01-26T23:45:19.772839+0000","last_became_peered":"2024-01-26T23:45:19.772839+0000","last_unstale":"2024-01-26T23:45:19.782081+0000","last_undegraded":"2024-01-26T23:45:19.782081+0000","last_fullsized":"2024-01-26T23:45:19.782081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:05:43.312820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.779648+0000","last_change":"2024-01-26T23:45:19.776897+0000","last_active":"2024-01-26T23:45:19.779648+0000","last_peered":"2024-01-26T23:45:19.779648+0000","last_clean":"2024-01-26T23:45:19.779648+0000","last_became_active":"2024-01-26T23:45:19.776533+0000","last_became_peered":"2024-01-26T23:45:19.776533+0000","last_unstale":"2024-01-26T23:45:19.779648+0000","last_undegraded":"2024-01-26T23:45:19.779648+0000","last_fullsized":"2024-01-26T23:45:19.779648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:40.045595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.766807+0000","last_change":"2024-01-26T23:45:19.763609+0000","last_active":"2024-01-26T23:45:19.766807+0000","last_peered":"2024-01-26T23:45:19.766807+0000","last_clean":"2024-01-26T23:45:19.766807+0000","last_became_active":"2024-01-26T23:45:19.762617+0000","last_became_peered":"2024-01-26T23:45:19.762617+0000","last_unstale":"2024-01-26T23:45:19.766807+0000","last_undegraded":"2024-01-26T23:45:19.766807+0000","last_fullsized":"2024-01-26T23:45:19.766807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:40:16.213836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.776350+0000","last_change":"2024-01-26T23:45:19.771192+0000","last_active":"2024-01-26T23:45:19.776350+0000","last_peered":"2024-01-26T23:45:19.776350+0000","last_clean":"2024-01-26T23:45:19.776350+0000","last_became_active":"2024-01-26T23:45:19.770133+0000","last_became_peered":"2024-01-26T23:45:19.770133+0000","last_unstale":"2024-01-26T23:45:19.776350+0000","last_undegraded":"2024-01-26T23:45:19.776350+0000","last_fullsized":"2024-01-26T23:45:19.776350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:02:44.794463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"84'4","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.793960+0000","last_change":"2024-01-26T23:45:18.751254+0000","last_active":"2024-01-26T23:45:19.793960+0000","last_peered":"2024-01-26T23:45:19.793960+0000","last_clean":"2024-01-26T23:45:19.793960+0000","last_became_active":"2024-01-26T23:45:18.750982+0000","last_became_peered":"2024-01-26T23:45:18.750982+0000","last_unstale":"2024-01-26T23:45:19.793960+0000","last_undegraded":"2024-01-26T23:45:19.793960+0000","last_fullsized":"2024-01-26T23:45:19.793960+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:54:02.845982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.777971+0000","last_change":"2024-01-26T23:45:19.775570+0000","last_active":"2024-01-26T23:45:19.777971+0000","last_peered":"2024-01-26T23:45:19.777971+0000","last_clean":"2024-01-26T23:45:19.777971+0000","last_became_active":"2024-01-26T23:45:19.775301+0000","last_became_peered":"2024-01-26T23:45:19.775301+0000","last_unstale":"2024-01-26T23:45:19.777971+0000","last_undegraded":"2024-01-26T23:45:19.777971+0000","last_fullsized":"2024-01-26T23:45:19.777971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:09:30.529877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"84'1","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.788196+0000","last_change":"2024-01-26T23:45:18.767062+0000","last_active":"2024-01-26T23:45:19.788196+0000","last_peered":"2024-01-26T23:45:19.788196+0000","last_clean":"2024-01-26T23:45:19.788196+0000","last_became_active":"2024-01-26T23:45:18.766829+0000","last_became_peered":"2024-01-26T23:45:18.766829+0000","last_unstale":"2024-01-26T23:45:19.788196+0000","last_undegraded":"2024-01-26T23:45:19.788196+0000","last_fullsized":"2024-01-26T23:45:19.788196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:31:40.511537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.772356+0000","last_change":"2024-01-26T23:45:19.768373+0000","last_active":"2024-01-26T23:45:19.772356+0000","last_peered":"2024-01-26T23:45:19.772356+0000","last_clean":"2024-01-26T23:45:19.772356+0000","last_became_active":"2024-01-26T23:45:19.768093+0000","last_became_peered":"2024-01-26T23:45:19.768093+0000","last_unstale":"2024-01-26T23:45:19.772356+0000","last_undegraded":"2024-01-26T23:45:19.772356+0000","last_fullsized":"2024-01-26T23:45:19.772356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:21:33.110353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"84'2","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.785914+0000","last_change":"2024-01-26T23:45:18.765964+0000","last_active":"2024-01-26T23:45:19.785914+0000","last_peered":"2024-01-26T23:45:19.785914+0000","last_clean":"2024-01-26T23:45:19.785914+0000","last_became_active":"2024-01-26T23:45:18.765676+0000","last_became_peered":"2024-01-26T23:45:18.765676+0000","last_unstale":"2024-01-26T23:45:19.785914+0000","last_undegraded":"2024-01-26T23:45:19.785914+0000","last_fullsized":"2024-01-26T23:45:19.785914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:25:55.585050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.790453+0000","last_change":"2024-01-26T23:45:19.786738+0000","last_active":"2024-01-26T23:45:19.790453+0000","last_peered":"2024-01-26T23:45:19.790453+0000","last_clean":"2024-01-26T23:45:19.790453+0000","last_became_active":"2024-01-26T23:45:19.786474+0000","last_became_peered":"2024-01-26T23:45:19.786474+0000","last_unstale":"2024-01-26T23:45:19.790453+0000","last_undegraded":"2024-01-26T23:45:19.790453+0000","last_fullsized":"2024-01-26T23:45:19.790453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:40.045595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"84'2","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.789007+0000","last_change":"2024-01-26T23:45:18.758761+0000","last_active":"2024-01-26T23:45:19.789007+0000","last_peered":"2024-01-26T23:45:19.789007+0000","last_clean":"2024-01-26T23:45:19.789007+0000","last_became_active":"2024-01-26T23:45:18.758120+0000","last_became_peered":"2024-01-26T23:45:18.758120+0000","last_unstale":"2024-01-26T23:45:19.789007+0000","last_undegraded":"2024-01-26T23:45:19.789007+0000","last_fullsized":"2024-01-26T23:45:19.789007+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:08:05.114547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.771754+0000","last_change":"2024-01-26T23:45:19.763541+0000","last_active":"2024-01-26T23:45:19.771754+0000","last_peered":"2024-01-26T23:45:19.771754+0000","last_clean":"2024-01-26T23:45:19.771754+0000","last_became_active":"2024-01-26T23:45:19.763224+0000","last_became_peered":"2024-01-26T23:45:19.763224+0000","last_unstale":"2024-01-26T23:45:19.771754+0000","last_undegraded":"2024-01-26T23:45:19.771754+0000","last_fullsized":"2024-01-26T23:45:19.771754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:36:29.696055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"84'1","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.782207+0000","last_change":"2024-01-26T23:45:18.761520+0000","last_active":"2024-01-26T23:45:19.782207+0000","last_peered":"2024-01-26T23:45:19.782207+0000","last_clean":"2024-01-26T23:45:19.782207+0000","last_became_active":"2024-01-26T23:45:18.761198+0000","last_became_peered":"2024-01-26T23:45:18.761198+0000","last_unstale":"2024-01-26T23:45:19.782207+0000","last_undegraded":"2024-01-26T23:45:19.782207+0000","last_fullsized":"2024-01-26T23:45:19.782207+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:39:32.807018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.787507+0000","last_change":"2024-01-26T23:45:19.783460+0000","last_active":"2024-01-26T23:45:19.787507+0000","last_peered":"2024-01-26T23:45:19.787507+0000","last_clean":"2024-01-26T23:45:19.787507+0000","last_became_active":"2024-01-26T23:45:19.780983+0000","last_became_peered":"2024-01-26T23:45:19.780983+0000","last_unstale":"2024-01-26T23:45:19.787507+0000","last_undegraded":"2024-01-26T23:45:19.787507+0000","last_fullsized":"2024-01-26T23:45:19.787507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:57:02.783696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"84'1","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.782080+0000","last_change":"2024-01-26T23:45:18.764499+0000","last_active":"2024-01-26T23:45:19.782080+0000","last_peered":"2024-01-26T23:45:19.782080+0000","last_clean":"2024-01-26T23:45:19.782080+0000","last_became_active":"2024-01-26T23:45:18.764133+0000","last_became_peered":"2024-01-26T23:45:18.764133+0000","last_unstale":"2024-01-26T23:45:19.782080+0000","last_undegraded":"2024-01-26T23:45:19.782080+0000","last_fullsized":"2024-01-26T23:45:19.782080+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:18:59.318904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778542+0000","last_change":"2024-01-26T23:45:19.771709+0000","last_active":"2024-01-26T23:45:19.778542+0000","last_peered":"2024-01-26T23:45:19.778542+0000","last_clean":"2024-01-26T23:45:19.778542+0000","last_became_active":"2024-01-26T23:45:19.771443+0000","last_became_peered":"2024-01-26T23:45:19.771443+0000","last_unstale":"2024-01-26T23:45:19.778542+0000","last_undegraded":"2024-01-26T23:45:19.778542+0000","last_fullsized":"2024-01-26T23:45:19.778542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:54:09.399022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.767931+0000","last_change":"2024-01-26T23:45:18.762363+0000","last_active":"2024-01-26T23:45:19.767931+0000","last_peered":"2024-01-26T23:45:19.767931+0000","last_clean":"2024-01-26T23:45:19.767931+0000","last_became_active":"2024-01-26T23:45:18.761883+0000","last_became_peered":"2024-01-26T23:45:18.761883+0000","last_unstale":"2024-01-26T23:45:19.767931+0000","last_undegraded":"2024-01-26T23:45:19.767931+0000","last_fullsized":"2024-01-26T23:45:19.767931+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:05:42.307618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.782286+0000","last_change":"2024-01-26T23:45:19.779200+0000","last_active":"2024-01-26T23:45:19.782286+0000","last_peered":"2024-01-26T23:45:19.782286+0000","last_clean":"2024-01-26T23:45:19.782286+0000","last_became_active":"2024-01-26T23:45:19.777273+0000","last_became_peered":"2024-01-26T23:45:19.777273+0000","last_unstale":"2024-01-26T23:45:19.782286+0000","last_undegraded":"2024-01-26T23:45:19.782286+0000","last_fullsized":"2024-01-26T23:45:19.782286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:15:51.882694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.762310+0000","last_change":"2024-01-26T23:45:18.756051+0000","last_active":"2024-01-26T23:45:19.762310+0000","last_peered":"2024-01-26T23:45:19.762310+0000","last_clean":"2024-01-26T23:45:19.762310+0000","last_became_active":"2024-01-26T23:45:18.755595+0000","last_became_peered":"2024-01-26T23:45:18.755595+0000","last_unstale":"2024-01-26T23:45:19.762310+0000","last_undegraded":"2024-01-26T23:45:19.762310+0000","last_fullsized":"2024-01-26T23:45:19.762310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:41:01.991788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.765032+0000","last_change":"2024-01-26T23:45:19.763815+0000","last_active":"2024-01-26T23:45:19.765032+0000","last_peered":"2024-01-26T23:45:19.765032+0000","last_clean":"2024-01-26T23:45:19.765032+0000","last_became_active":"2024-01-26T23:45:19.763510+0000","last_became_peered":"2024-01-26T23:45:19.763510+0000","last_unstale":"2024-01-26T23:45:19.765032+0000","last_undegraded":"2024-01-26T23:45:19.765032+0000","last_fullsized":"2024-01-26T23:45:19.765032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:04:13.439721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.775992+0000","last_change":"2024-01-26T23:45:18.758624+0000","last_active":"2024-01-26T23:45:19.775992+0000","last_peered":"2024-01-26T23:45:19.775992+0000","last_clean":"2024-01-26T23:45:19.775992+0000","last_became_active":"2024-01-26T23:45:18.757214+0000","last_became_peered":"2024-01-26T23:45:18.757214+0000","last_unstale":"2024-01-26T23:45:19.775992+0000","last_undegraded":"2024-01-26T23:45:19.775992+0000","last_fullsized":"2024-01-26T23:45:19.775992+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:20:13.045863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.782134+0000","last_change":"2024-01-26T23:45:19.779183+0000","last_active":"2024-01-26T23:45:19.782134+0000","last_peered":"2024-01-26T23:45:19.782134+0000","last_clean":"2024-01-26T23:45:19.782134+0000","last_became_active":"2024-01-26T23:45:19.778797+0000","last_became_peered":"2024-01-26T23:45:19.778797+0000","last_unstale":"2024-01-26T23:45:19.782134+0000","last_undegraded":"2024-01-26T23:45:19.782134+0000","last_fullsized":"2024-01-26T23:45:19.782134+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:21:45.768113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.769736+0000","last_change":"2024-01-26T23:45:18.761124+0000","last_active":"2024-01-26T23:45:19.769736+0000","last_peered":"2024-01-26T23:45:19.769736+0000","last_clean":"2024-01-26T23:45:19.769736+0000","last_became_active":"2024-01-26T23:45:18.760861+0000","last_became_peered":"2024-01-26T23:45:18.760861+0000","last_unstale":"2024-01-26T23:45:19.769736+0000","last_undegraded":"2024-01-26T23:45:19.769736+0000","last_fullsized":"2024-01-26T23:45:19.769736+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:54:02.845982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.771954+0000","last_change":"2024-01-26T23:45:19.769326+0000","last_active":"2024-01-26T23:45:19.771954+0000","last_peered":"2024-01-26T23:45:19.771954+0000","last_clean":"2024-01-26T23:45:19.771954+0000","last_became_active":"2024-01-26T23:45:19.769016+0000","last_became_peered":"2024-01-26T23:45:19.769016+0000","last_unstale":"2024-01-26T23:45:19.771954+0000","last_undegraded":"2024-01-26T23:45:19.771954+0000","last_fullsized":"2024-01-26T23:45:19.771954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:57:02.783696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"84'3","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.794265+0000","last_change":"2024-01-26T23:45:18.762976+0000","last_active":"2024-01-26T23:45:19.794265+0000","last_peered":"2024-01-26T23:45:19.794265+0000","last_clean":"2024-01-26T23:45:19.794265+0000","last_became_active":"2024-01-26T23:45:18.762389+0000","last_became_peered":"2024-01-26T23:45:18.762389+0000","last_unstale":"2024-01-26T23:45:19.794265+0000","last_undegraded":"2024-01-26T23:45:19.794265+0000","last_fullsized":"2024-01-26T23:45:19.794265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:28:52.583643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.777464+0000","last_change":"2024-01-26T23:45:19.776012+0000","last_active":"2024-01-26T23:45:19.777464+0000","last_peered":"2024-01-26T23:45:19.777464+0000","last_clean":"2024-01-26T23:45:19.777464+0000","last_became_active":"2024-01-26T23:45:19.775193+0000","last_became_peered":"2024-01-26T23:45:19.775193+0000","last_unstale":"2024-01-26T23:45:19.777464+0000","last_undegraded":"2024-01-26T23:45:19.777464+0000","last_fullsized":"2024-01-26T23:45:19.777464+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:39:33.812220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"84'6","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.796892+0000","last_change":"2024-01-26T23:45:18.765358+0000","last_active":"2024-01-26T23:45:19.796892+0000","last_peered":"2024-01-26T23:45:19.796892+0000","last_clean":"2024-01-26T23:45:19.796892+0000","last_became_active":"2024-01-26T23:45:18.765058+0000","last_became_peered":"2024-01-26T23:45:18.765058+0000","last_unstale":"2024-01-26T23:45:19.796892+0000","last_undegraded":"2024-01-26T23:45:19.796892+0000","last_fullsized":"2024-01-26T23:45:19.796892+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:00.932196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.788853+0000","last_change":"2024-01-26T23:45:19.784757+0000","last_active":"2024-01-26T23:45:19.788853+0000","last_peered":"2024-01-26T23:45:19.788853+0000","last_clean":"2024-01-26T23:45:19.788853+0000","last_became_active":"2024-01-26T23:45:19.784467+0000","last_became_peered":"2024-01-26T23:45:19.784467+0000","last_unstale":"2024-01-26T23:45:19.788853+0000","last_undegraded":"2024-01-26T23:45:19.788853+0000","last_fullsized":"2024-01-26T23:45:19.788853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:24.303858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"84'2","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.794100+0000","last_change":"2024-01-26T23:45:18.766255+0000","last_active":"2024-01-26T23:45:19.794100+0000","last_peered":"2024-01-26T23:45:19.794100+0000","last_clean":"2024-01-26T23:45:19.794100+0000","last_became_active":"2024-01-26T23:45:18.765979+0000","last_became_peered":"2024-01-26T23:45:18.765979+0000","last_unstale":"2024-01-26T23:45:19.794100+0000","last_undegraded":"2024-01-26T23:45:19.794100+0000","last_fullsized":"2024-01-26T23:45:19.794100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:36:25.947995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.777988+0000","last_change":"2024-01-26T23:45:19.769598+0000","last_active":"2024-01-26T23:45:19.777988+0000","last_peered":"2024-01-26T23:45:19.777988+0000","last_clean":"2024-01-26T23:45:19.777988+0000","last_became_active":"2024-01-26T23:45:19.769297+0000","last_became_peered":"2024-01-26T23:45:19.769297+0000","last_unstale":"2024-01-26T23:45:19.777988+0000","last_undegraded":"2024-01-26T23:45:19.777988+0000","last_fullsized":"2024-01-26T23:45:19.777988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:29:01.653387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.764205+0000","last_change":"2024-01-26T23:45:18.764994+0000","last_active":"2024-01-26T23:45:19.764205+0000","last_peered":"2024-01-26T23:45:19.764205+0000","last_clean":"2024-01-26T23:45:19.764205+0000","last_became_active":"2024-01-26T23:45:18.764683+0000","last_became_peered":"2024-01-26T23:45:18.764683+0000","last_unstale":"2024-01-26T23:45:19.764205+0000","last_undegraded":"2024-01-26T23:45:19.764205+0000","last_fullsized":"2024-01-26T23:45:19.764205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:00.932196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778563+0000","last_change":"2024-01-26T23:45:19.773049+0000","last_active":"2024-01-26T23:45:19.778563+0000","last_peered":"2024-01-26T23:45:19.778563+0000","last_clean":"2024-01-26T23:45:19.778563+0000","last_became_active":"2024-01-26T23:45:19.772754+0000","last_became_peered":"2024-01-26T23:45:19.772754+0000","last_unstale":"2024-01-26T23:45:19.778563+0000","last_undegraded":"2024-01-26T23:45:19.778563+0000","last_fullsized":"2024-01-26T23:45:19.778563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:56:05.060730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.779784+0000","last_change":"2024-01-26T23:45:18.763694+0000","last_active":"2024-01-26T23:45:19.779784+0000","last_peered":"2024-01-26T23:45:19.779784+0000","last_clean":"2024-01-26T23:45:19.779784+0000","last_became_active":"2024-01-26T23:45:18.763345+0000","last_became_peered":"2024-01-26T23:45:18.763345+0000","last_unstale":"2024-01-26T23:45:19.779784+0000","last_undegraded":"2024-01-26T23:45:19.779784+0000","last_fullsized":"2024-01-26T23:45:19.779784+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:57:17.179137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778284+0000","last_change":"2024-01-26T23:45:19.775692+0000","last_active":"2024-01-26T23:45:19.778284+0000","last_peered":"2024-01-26T23:45:19.778284+0000","last_clean":"2024-01-26T23:45:19.778284+0000","last_became_active":"2024-01-26T23:45:19.775333+0000","last_became_peered":"2024-01-26T23:45:19.775333+0000","last_unstale":"2024-01-26T23:45:19.778284+0000","last_undegraded":"2024-01-26T23:45:19.778284+0000","last_fullsized":"2024-01-26T23:45:19.778284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:59.089607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"84'8","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.792435+0000","last_change":"2024-01-26T23:45:18.764634+0000","last_active":"2024-01-26T23:45:19.792435+0000","last_peered":"2024-01-26T23:45:19.792435+0000","last_clean":"2024-01-26T23:45:19.792435+0000","last_became_active":"2024-01-26T23:45:18.764318+0000","last_became_peered":"2024-01-26T23:45:18.764318+0000","last_unstale":"2024-01-26T23:45:19.792435+0000","last_undegraded":"2024-01-26T23:45:19.792435+0000","last_fullsized":"2024-01-26T23:45:19.792435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:00.932196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.791458+0000","last_change":"2024-01-26T23:45:19.787367+0000","last_active":"2024-01-26T23:45:19.791458+0000","last_peered":"2024-01-26T23:45:19.791458+0000","last_clean":"2024-01-26T23:45:19.791458+0000","last_became_active":"2024-01-26T23:45:19.787104+0000","last_became_peered":"2024-01-26T23:45:19.787104+0000","last_unstale":"2024-01-26T23:45:19.791458+0000","last_undegraded":"2024-01-26T23:45:19.791458+0000","last_fullsized":"2024-01-26T23:45:19.791458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:19:00.324107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"84'1","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.787862+0000","last_change":"2024-01-26T23:45:18.765531+0000","last_active":"2024-01-26T23:45:19.787862+0000","last_peered":"2024-01-26T23:45:19.787862+0000","last_clean":"2024-01-26T23:45:19.787862+0000","last_became_active":"2024-01-26T23:45:18.765288+0000","last_became_peered":"2024-01-26T23:45:18.765288+0000","last_unstale":"2024-01-26T23:45:19.787862+0000","last_undegraded":"2024-01-26T23:45:19.787862+0000","last_fullsized":"2024-01-26T23:45:19.787862+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:24:57.645890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.785942+0000","last_change":"2024-01-26T23:45:19.782678+0000","last_active":"2024-01-26T23:45:19.785942+0000","last_peered":"2024-01-26T23:45:19.785942+0000","last_clean":"2024-01-26T23:45:19.785942+0000","last_became_active":"2024-01-26T23:45:19.782422+0000","last_became_peered":"2024-01-26T23:45:19.782422+0000","last_unstale":"2024-01-26T23:45:19.785942+0000","last_undegraded":"2024-01-26T23:45:19.785942+0000","last_fullsized":"2024-01-26T23:45:19.785942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:09:30.529877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"84'2","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.794369+0000","last_change":"2024-01-26T23:45:18.768703+0000","last_active":"2024-01-26T23:45:19.794369+0000","last_peered":"2024-01-26T23:45:19.794369+0000","last_clean":"2024-01-26T23:45:19.794369+0000","last_became_active":"2024-01-26T23:45:18.768460+0000","last_became_peered":"2024-01-26T23:45:18.768460+0000","last_unstale":"2024-01-26T23:45:19.794369+0000","last_undegraded":"2024-01-26T23:45:19.794369+0000","last_fullsized":"2024-01-26T23:45:19.794369+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:54:02.845982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778375+0000","last_change":"2024-01-26T23:45:19.775743+0000","last_active":"2024-01-26T23:45:19.778375+0000","last_peered":"2024-01-26T23:45:19.778375+0000","last_clean":"2024-01-26T23:45:19.778375+0000","last_became_active":"2024-01-26T23:45:19.774642+0000","last_became_peered":"2024-01-26T23:45:19.774642+0000","last_unstale":"2024-01-26T23:45:19.778375+0000","last_undegraded":"2024-01-26T23:45:19.778375+0000","last_fullsized":"2024-01-26T23:45:19.778375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:04:13.439721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"84'1","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.794313+0000","last_change":"2024-01-26T23:45:18.763140+0000","last_active":"2024-01-26T23:45:19.794313+0000","last_peered":"2024-01-26T23:45:19.794313+0000","last_clean":"2024-01-26T23:45:19.794313+0000","last_became_active":"2024-01-26T23:45:18.762764+0000","last_became_peered":"2024-01-26T23:45:18.762764+0000","last_unstale":"2024-01-26T23:45:19.794313+0000","last_undegraded":"2024-01-26T23:45:19.794313+0000","last_fullsized":"2024-01-26T23:45:19.794313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:52:28.272099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778657+0000","last_change":"2024-01-26T23:45:19.774562+0000","last_active":"2024-01-26T23:45:19.778657+0000","last_peered":"2024-01-26T23:45:19.778657+0000","last_clean":"2024-01-26T23:45:19.778657+0000","last_became_active":"2024-01-26T23:45:19.774092+0000","last_became_peered":"2024-01-26T23:45:19.774092+0000","last_unstale":"2024-01-26T23:45:19.778657+0000","last_undegraded":"2024-01-26T23:45:19.778657+0000","last_fullsized":"2024-01-26T23:45:19.778657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:57:18.184339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.767799+0000","last_change":"2024-01-26T23:45:18.768278+0000","last_active":"2024-01-26T23:45:19.767799+0000","last_peered":"2024-01-26T23:45:19.767799+0000","last_clean":"2024-01-26T23:45:19.767799+0000","last_became_active":"2024-01-26T23:45:18.767952+0000","last_became_peered":"2024-01-26T23:45:18.767952+0000","last_unstale":"2024-01-26T23:45:19.767799+0000","last_undegraded":"2024-01-26T23:45:19.767799+0000","last_fullsized":"2024-01-26T23:45:19.767799+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:04:12.434519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.766828+0000","last_change":"2024-01-26T23:45:19.762912+0000","last_active":"2024-01-26T23:45:19.766828+0000","last_peered":"2024-01-26T23:45:19.766828+0000","last_clean":"2024-01-26T23:45:19.766828+0000","last_became_active":"2024-01-26T23:45:19.762207+0000","last_became_peered":"2024-01-26T23:45:19.762207+0000","last_unstale":"2024-01-26T23:45:19.766828+0000","last_undegraded":"2024-01-26T23:45:19.766828+0000","last_fullsized":"2024-01-26T23:45:19.766828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:51:03.633095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"84'2","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.786073+0000","last_change":"2024-01-26T23:45:18.761890+0000","last_active":"2024-01-26T23:45:19.786073+0000","last_peered":"2024-01-26T23:45:19.786073+0000","last_clean":"2024-01-26T23:45:19.786073+0000","last_became_active":"2024-01-26T23:45:18.760793+0000","last_became_peered":"2024-01-26T23:45:18.760793+0000","last_unstale":"2024-01-26T23:45:19.786073+0000","last_undegraded":"2024-01-26T23:45:19.786073+0000","last_fullsized":"2024-01-26T23:45:19.786073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:45:19.871782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.779715+0000","last_change":"2024-01-26T23:45:19.774087+0000","last_active":"2024-01-26T23:45:19.779715+0000","last_peered":"2024-01-26T23:45:19.779715+0000","last_clean":"2024-01-26T23:45:19.779715+0000","last_became_active":"2024-01-26T23:45:19.773249+0000","last_became_peered":"2024-01-26T23:45:19.773249+0000","last_unstale":"2024-01-26T23:45:19.779715+0000","last_undegraded":"2024-01-26T23:45:19.779715+0000","last_fullsized":"2024-01-26T23:45:19.779715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:04:13.439721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.764307+0000","last_change":"2024-01-26T23:45:18.764993+0000","last_active":"2024-01-26T23:45:19.764307+0000","last_peered":"2024-01-26T23:45:19.764307+0000","last_clean":"2024-01-26T23:45:19.764307+0000","last_became_active":"2024-01-26T23:45:18.764701+0000","last_became_peered":"2024-01-26T23:45:18.764701+0000","last_unstale":"2024-01-26T23:45:19.764307+0000","last_undegraded":"2024-01-26T23:45:19.764307+0000","last_fullsized":"2024-01-26T23:45:19.764307+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:57:07.774030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.770720+0000","last_change":"2024-01-26T23:45:19.767295+0000","last_active":"2024-01-26T23:45:19.770720+0000","last_peered":"2024-01-26T23:45:19.770720+0000","last_clean":"2024-01-26T23:45:19.770720+0000","last_became_active":"2024-01-26T23:45:19.767071+0000","last_became_peered":"2024-01-26T23:45:19.767071+0000","last_unstale":"2024-01-26T23:45:19.770720+0000","last_undegraded":"2024-01-26T23:45:19.770720+0000","last_fullsized":"2024-01-26T23:45:19.770720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:04:13.439721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.754675+0000","last_change":"2024-01-26T23:45:18.768303+0000","last_active":"2024-01-26T23:45:19.754675+0000","last_peered":"2024-01-26T23:45:19.754675+0000","last_clean":"2024-01-26T23:45:19.754675+0000","last_became_active":"2024-01-26T23:45:18.767961+0000","last_became_peered":"2024-01-26T23:45:18.767961+0000","last_unstale":"2024-01-26T23:45:19.754675+0000","last_undegraded":"2024-01-26T23:45:19.754675+0000","last_fullsized":"2024-01-26T23:45:19.754675+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:18:51.436569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.780733+0000","last_change":"2024-01-26T23:45:19.779663+0000","last_active":"2024-01-26T23:45:19.780733+0000","last_peered":"2024-01-26T23:45:19.780733+0000","last_clean":"2024-01-26T23:45:19.780733+0000","last_became_active":"2024-01-26T23:45:19.779091+0000","last_became_peered":"2024-01-26T23:45:19.779091+0000","last_unstale":"2024-01-26T23:45:19.780733+0000","last_undegraded":"2024-01-26T23:45:19.780733+0000","last_fullsized":"2024-01-26T23:45:19.780733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:41:17.135191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"84'2","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.792301+0000","last_change":"2024-01-26T23:45:18.767075+0000","last_active":"2024-01-26T23:45:19.792301+0000","last_peered":"2024-01-26T23:45:19.792301+0000","last_clean":"2024-01-26T23:45:19.792301+0000","last_became_active":"2024-01-26T23:45:18.766757+0000","last_became_peered":"2024-01-26T23:45:18.766757+0000","last_unstale":"2024-01-26T23:45:19.792301+0000","last_undegraded":"2024-01-26T23:45:19.792301+0000","last_fullsized":"2024-01-26T23:45:19.792301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:02:05.153467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.771608+0000","last_change":"2024-01-26T23:45:19.765627+0000","last_active":"2024-01-26T23:45:19.771608+0000","last_peered":"2024-01-26T23:45:19.771608+0000","last_clean":"2024-01-26T23:45:19.771608+0000","last_became_active":"2024-01-26T23:45:19.765096+0000","last_became_peered":"2024-01-26T23:45:19.765096+0000","last_unstale":"2024-01-26T23:45:19.771608+0000","last_undegraded":"2024-01-26T23:45:19.771608+0000","last_fullsized":"2024-01-26T23:45:19.771608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:55.510702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.764382+0000","last_change":"2024-01-26T23:45:18.763881+0000","last_active":"2024-01-26T23:45:19.764382+0000","last_peered":"2024-01-26T23:45:19.764382+0000","last_clean":"2024-01-26T23:45:19.764382+0000","last_became_active":"2024-01-26T23:45:18.763607+0000","last_became_peered":"2024-01-26T23:45:18.763607+0000","last_unstale":"2024-01-26T23:45:19.764382+0000","last_undegraded":"2024-01-26T23:45:19.764382+0000","last_fullsized":"2024-01-26T23:45:19.764382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:54:02.845982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778014+0000","last_change":"2024-01-26T23:45:19.768388+0000","last_active":"2024-01-26T23:45:19.778014+0000","last_peered":"2024-01-26T23:45:19.778014+0000","last_clean":"2024-01-26T23:45:19.778014+0000","last_became_active":"2024-01-26T23:45:19.768069+0000","last_became_peered":"2024-01-26T23:45:19.768069+0000","last_unstale":"2024-01-26T23:45:19.778014+0000","last_undegraded":"2024-01-26T23:45:19.778014+0000","last_fullsized":"2024-01-26T23:45:19.778014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:40.045595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.754836+0000","last_change":"2024-01-26T23:45:18.765412+0000","last_active":"2024-01-26T23:45:19.754836+0000","last_peered":"2024-01-26T23:45:19.754836+0000","last_clean":"2024-01-26T23:45:19.754836+0000","last_became_active":"2024-01-26T23:45:18.765125+0000","last_became_peered":"2024-01-26T23:45:18.765125+0000","last_unstale":"2024-01-26T23:45:19.754836+0000","last_undegraded":"2024-01-26T23:45:19.754836+0000","last_fullsized":"2024-01-26T23:45:19.754836+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:04:14.572796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.778647+0000","last_change":"2024-01-26T23:45:19.774382+0000","last_active":"2024-01-26T23:45:19.778647+0000","last_peered":"2024-01-26T23:45:19.778647+0000","last_clean":"2024-01-26T23:45:19.778647+0000","last_became_active":"2024-01-26T23:45:19.773713+0000","last_became_peered":"2024-01-26T23:45:19.773713+0000","last_unstale":"2024-01-26T23:45:19.778647+0000","last_undegraded":"2024-01-26T23:45:19.778647+0000","last_fullsized":"2024-01-26T23:45:19.778647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:52:29.277301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.758507+0000","last_change":"2024-01-26T23:45:18.760331+0000","last_active":"2024-01-26T23:45:19.758507+0000","last_peered":"2024-01-26T23:45:19.758507+0000","last_clean":"2024-01-26T23:45:19.758507+0000","last_became_active":"2024-01-26T23:45:18.760024+0000","last_became_peered":"2024-01-26T23:45:18.760024+0000","last_unstale":"2024-01-26T23:45:19.758507+0000","last_undegraded":"2024-01-26T23:45:19.758507+0000","last_fullsized":"2024-01-26T23:45:19.758507+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:57:07.774030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.770462+0000","last_change":"2024-01-26T23:45:19.766917+0000","last_active":"2024-01-26T23:45:19.770462+0000","last_peered":"2024-01-26T23:45:19.770462+0000","last_clean":"2024-01-26T23:45:19.770462+0000","last_became_active":"2024-01-26T23:45:19.766638+0000","last_became_peered":"2024-01-26T23:45:19.766638+0000","last_unstale":"2024-01-26T23:45:19.770462+0000","last_undegraded":"2024-01-26T23:45:19.770462+0000","last_fullsized":"2024-01-26T23:45:19.770462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:59.089607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.775566+0000","last_change":"2024-01-26T23:45:18.763657+0000","last_active":"2024-01-26T23:45:19.775566+0000","last_peered":"2024-01-26T23:45:19.775566+0000","last_clean":"2024-01-26T23:45:19.775566+0000","last_became_active":"2024-01-26T23:45:18.763274+0000","last_became_peered":"2024-01-26T23:45:18.763274+0000","last_unstale":"2024-01-26T23:45:19.775566+0000","last_undegraded":"2024-01-26T23:45:19.775566+0000","last_fullsized":"2024-01-26T23:45:19.775566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:02:43.789261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.785440+0000","last_change":"2024-01-26T23:45:19.783363+0000","last_active":"2024-01-26T23:45:19.785440+0000","last_peered":"2024-01-26T23:45:19.785440+0000","last_clean":"2024-01-26T23:45:19.785440+0000","last_became_active":"2024-01-26T23:45:19.783118+0000","last_became_peered":"2024-01-26T23:45:19.783118+0000","last_unstale":"2024-01-26T23:45:19.785440+0000","last_undegraded":"2024-01-26T23:45:19.785440+0000","last_fullsized":"2024-01-26T23:45:19.785440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:57:02.783696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.767529+0000","last_change":"2024-01-26T23:45:18.764082+0000","last_active":"2024-01-26T23:45:19.767529+0000","last_peered":"2024-01-26T23:45:19.767529+0000","last_clean":"2024-01-26T23:45:19.767529+0000","last_became_active":"2024-01-26T23:45:18.763732+0000","last_became_peered":"2024-01-26T23:45:18.763732+0000","last_unstale":"2024-01-26T23:45:19.767529+0000","last_undegraded":"2024-01-26T23:45:19.767529+0000","last_fullsized":"2024-01-26T23:45:19.767529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:09:49.050354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.785139+0000","last_change":"2024-01-26T23:45:19.783970+0000","last_active":"2024-01-26T23:45:19.785139+0000","last_peered":"2024-01-26T23:45:19.785139+0000","last_clean":"2024-01-26T23:45:19.785139+0000","last_became_active":"2024-01-26T23:45:19.783042+0000","last_became_peered":"2024-01-26T23:45:19.783042+0000","last_unstale":"2024-01-26T23:45:19.785139+0000","last_undegraded":"2024-01-26T23:45:19.785139+0000","last_fullsized":"2024-01-26T23:45:19.785139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:56:05.060730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.767961+0000","last_change":"2024-01-26T23:45:18.762184+0000","last_active":"2024-01-26T23:45:19.767961+0000","last_peered":"2024-01-26T23:45:19.767961+0000","last_clean":"2024-01-26T23:45:19.767961+0000","last_became_active":"2024-01-26T23:45:18.761581+0000","last_became_peered":"2024-01-26T23:45:18.761581+0000","last_unstale":"2024-01-26T23:45:19.767961+0000","last_undegraded":"2024-01-26T23:45:19.767961+0000","last_fullsized":"2024-01-26T23:45:19.767961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:39.040393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.773724+0000","last_change":"2024-01-26T23:45:19.769873+0000","last_active":"2024-01-26T23:45:19.773724+0000","last_peered":"2024-01-26T23:45:19.773724+0000","last_clean":"2024-01-26T23:45:19.773724+0000","last_became_active":"2024-01-26T23:45:19.769571+0000","last_became_peered":"2024-01-26T23:45:19.769571+0000","last_unstale":"2024-01-26T23:45:19.773724+0000","last_undegraded":"2024-01-26T23:45:19.773724+0000","last_fullsized":"2024-01-26T23:45:19.773724+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:03:41.407804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.758583+0000","last_change":"2024-01-26T23:45:18.760794+0000","last_active":"2024-01-26T23:45:19.758583+0000","last_peered":"2024-01-26T23:45:19.758583+0000","last_clean":"2024-01-26T23:45:19.758583+0000","last_became_active":"2024-01-26T23:45:18.760500+0000","last_became_peered":"2024-01-26T23:45:18.760500+0000","last_unstale":"2024-01-26T23:45:19.758583+0000","last_undegraded":"2024-01-26T23:45:19.758583+0000","last_fullsized":"2024-01-26T23:45:19.758583+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:24:57.645890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.776234+0000","last_change":"2024-01-26T23:45:19.770513+0000","last_active":"2024-01-26T23:45:19.776234+0000","last_peered":"2024-01-26T23:45:19.776234+0000","last_clean":"2024-01-26T23:45:19.776234+0000","last_became_active":"2024-01-26T23:45:19.770231+0000","last_became_peered":"2024-01-26T23:45:19.770231+0000","last_unstale":"2024-01-26T23:45:19.776234+0000","last_undegraded":"2024-01-26T23:45:19.776234+0000","last_fullsized":"2024-01-26T23:45:19.776234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:18.738707+0000","last_clean_scrub_stamp":"2024-01-26T23:45:18.738707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:49:56.096830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.767411+0000","last_change":"2024-01-26T23:45:18.763355+0000","last_active":"2024-01-26T23:45:19.767411+0000","last_peered":"2024-01-26T23:45:19.767411+0000","last_clean":"2024-01-26T23:45:19.767411+0000","last_became_active":"2024-01-26T23:45:18.763015+0000","last_became_peered":"2024-01-26T23:45:18.763015+0000","last_unstale":"2024-01-26T23:45:19.767411+0000","last_undegraded":"2024-01-26T23:45:19.767411+0000","last_fullsized":"2024-01-26T23:45:19.767411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:17.733505+0000","last_clean_scrub_stamp":"2024-01-26T23:45:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:09:29.524675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'161","reported_seq":376,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-26T23:45:19.773981+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:45:19.773981+0000","last_peered":"2024-01-26T23:45:19.773981+0000","last_clean":"2024-01-26T23:45:19.773981+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:45:19.773981+0000","last_undegraded":"2024-01-26T23:45:19.773981+0000","last_fullsized":"2024-01-26T23:45:19.773981+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575038,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307084,"kb_used_data":1604,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064756,"statfs":{"total":96636764160,"available":96322310144,"internally_reserved":0,"allocated":1642496,"data_stored":1110670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57599999999999996,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.79500000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.35699999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80400000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.82099999999999995}]},{"osd":1,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57699999999999996,"15min":0.57799999999999996},"min":{"1min":0.42499999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77800000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.62},{"interface":"front","average":{"1min":0.62,"5min":0.61099999999999999,"15min":0.59899999999999998},"min":{"1min":0.46000000000000002,"5min":0.39200000000000002,"15min":0.36099999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.86499999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.38800000000000001,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.81699999999999995},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.48699999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77900000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.748}]},{"osd":3,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59999999999999998,"15min":0.59199999999999997},"min":{"1min":0.442,"5min":0.40500000000000003,"15min":0.35399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60599999999999998,"15min":0.60499999999999998},"min":{"1min":0.40100000000000002,"5min":0.39300000000000002,"15min":0.373},"max":{"1min":0.82699999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.67100000000000004}]},{"osd":4,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46899999999999997,"15min":0.46100000000000002},"min":{"1min":0.36399999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.66100000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.495,"15min":0.48799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.314},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.51400000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.504,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.91700000000000004},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.40000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.74199999999999999,"5min":0.90200000000000002,"15min":0.96599999999999997},"last":0.73599999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.57999999999999996,"15min":0.55700000000000005},"min":{"1min":0.46400000000000002,"5min":0.375,"15min":0.33100000000000002},"max":{"1min":0.77900000000000003,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56200000000000006,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76500000000000001,"5min":0.85199999999999998,"15min":0.86899999999999999},"last":0.68999999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575038,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302396,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069444,"statfs":{"total":96636764160,"available":96327110656,"internally_reserved":0,"allocated":1036288,"data_stored":519588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56899999999999995,"15min":0.57399999999999995},"min":{"1min":0.35999999999999999,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":0.752,"5min":0.85199999999999998,"15min":0.90200000000000002},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59099999999999997,"15min":0.59399999999999997},"min":{"1min":0.42899999999999999,"5min":0.38500000000000001,"15min":0.33600000000000002},"max":{"1min":0.79000000000000004,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.53700000000000003}]},{"osd":1,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.312,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.77100000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.622},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.34399999999999997},"max":{"1min":0.75700000000000001,"5min":0.78900000000000003,"15min":0.874},"last":0.70799999999999996}]},{"osd":2,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.40799999999999997,"5min":0.35499999999999998,"15min":0.29599999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.85799999999999998},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38600000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77000000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55100000000000005}]},{"osd":3,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.59499999999999997,"15min":0.59999999999999998},"min":{"1min":0.45500000000000002,"5min":0.38700000000000001,"15min":0.29999999999999999},"max":{"1min":0.79000000000000004,"5min":0.80700000000000005,"15min":0.92200000000000004},"last":0.745},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.622,"15min":0.624},"min":{"1min":0.55000000000000004,"5min":0.39500000000000002,"15min":0.39000000000000001},"max":{"1min":0.78400000000000003,"5min":0.81100000000000005,"15min":0.84499999999999997},"last":0.58499999999999996}]},{"osd":4,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.47799999999999998,"15min":0.47399999999999998},"min":{"1min":0.38100000000000001,"5min":0.29899999999999999,"15min":0.22},"max":{"1min":0.69899999999999995,"5min":0.84599999999999997,"15min":0.90200000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.501,"15min":0.499},"min":{"1min":0.44700000000000001,"5min":0.32000000000000001,"15min":0.28899999999999998},"max":{"1min":0.61899999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.46899999999999997}]},{"osd":5,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48499999999999999,"15min":0.48899999999999999},"min":{"1min":0.42499999999999999,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.53400000000000003,"15min":0.53300000000000003},"min":{"1min":0.46100000000000002,"5min":0.33700000000000002,"15min":0.26200000000000001},"max":{"1min":0.75700000000000001,"5min":0.872,"15min":0.96499999999999997},"last":0.60699999999999998}]},{"osd":7,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53500000000000003,"15min":0.53800000000000003},"min":{"1min":0.44400000000000001,"5min":0.32500000000000001,"15min":0.246},"max":{"1min":0.81799999999999995,"5min":0.82799999999999996,"15min":0.97199999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.308},"max":{"1min":0.73099999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.68600000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575038,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302420,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069420,"statfs":{"total":96636764160,"available":96327086080,"internally_reserved":0,"allocated":1060864,"data_stored":521826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29899999999999999,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":0.59699999999999998,"5min":0.621,"15min":0.72599999999999998},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.35799999999999998,"5min":0.28899999999999998,"15min":0.26000000000000001},"max":{"1min":0.73499999999999999,"5min":1.337,"15min":1.337},"last":0.41999999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41499999999999998},"min":{"1min":0.27700000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55900000000000005,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.379},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48599999999999999,"15min":0.47699999999999998},"min":{"1min":0.376,"5min":0.34699999999999998,"15min":0.28999999999999998},"max":{"1min":0.70799999999999996,"5min":0.77300000000000002,"15min":0.79600000000000004},"last":0.39500000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52200000000000002,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":0.75800000000000001,"5min":1.782,"15min":1.782},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.497,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.625,"5min":1.379,"15min":1.379},"last":0.32000000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65300000000000002,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":0.86499999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.435},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66200000000000003,"15min":0.66600000000000004},"min":{"1min":0.49399999999999999,"5min":0.45900000000000002,"15min":0.433},"max":{"1min":0.89900000000000002,"5min":0.91900000000000004,"15min":0.95699999999999996},"last":0.51600000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.50600000000000001,"5min":0.46800000000000003,"15min":0.42799999999999999},"max":{"1min":0.879,"5min":1.208,"15min":1.208},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68500000000000005,"15min":0.67100000000000004},"min":{"1min":0.53700000000000003,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":0.81599999999999995,"5min":0.96599999999999997,"15min":1.018},"last":0.46899999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.69399999999999995,"15min":0.68500000000000005},"min":{"1min":0.51800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85199999999999998,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.52900000000000003,"5min":0.437,"15min":0.42599999999999999},"max":{"1min":0.83699999999999997,"5min":1.022,"15min":1.022},"last":0.48899999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71599999999999997},"min":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.49399999999999999},"max":{"1min":0.93200000000000005,"5min":0.98199999999999998,"15min":1.0069999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.749,"15min":0.73199999999999998},"min":{"1min":0.53200000000000003,"5min":0.52200000000000002,"15min":0.438},"max":{"1min":0.90600000000000003,"5min":1.296,"15min":1.3049999999999999},"last":0.56999999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575037,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302972,"kb_used_data":1588,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068868,"statfs":{"total":96636764160,"available":96326520832,"internally_reserved":0,"allocated":1626112,"data_stored":1109922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.28499999999999998,"5min":0.23100000000000001,"15min":0.214},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":1.284},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.46000000000000002,"15min":0.45600000000000002},"min":{"1min":0.29099999999999998,"5min":0.28899999999999998,"15min":0.27300000000000002},"max":{"1min":0.60899999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.40000000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42099999999999999,"15min":0.42999999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.26700000000000002},"max":{"1min":0.57299999999999995,"5min":0.89600000000000002,"15min":1.2230000000000001},"last":0.315},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.38300000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":1.2430000000000001},"last":0.45600000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46899999999999997,"15min":0.46700000000000003},"min":{"1min":0.36799999999999999,"5min":0.30599999999999999,"15min":0.26800000000000002},"max":{"1min":0.68100000000000005,"5min":0.80700000000000005,"15min":1.0980000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.49399999999999999,"15min":0.49099999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.27000000000000002},"max":{"1min":0.64000000000000001,"5min":1.198,"15min":1.198},"last":0.439}]},{"osd":4,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.64300000000000002,"15min":0.65200000000000002},"min":{"1min":0.46899999999999997,"5min":0.42399999999999999,"15min":0.372},"max":{"1min":0.78700000000000003,"5min":0.93300000000000005,"15min":1.133},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64700000000000002,"15min":0.65900000000000003},"min":{"1min":0.438,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85299999999999998,"5min":0.94299999999999995,"15min":1.0169999999999999},"last":0.71199999999999997}]},{"osd":5,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65100000000000002,"15min":0.66300000000000003},"min":{"1min":0.52600000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79800000000000004,"5min":0.95799999999999996,"15min":1.0880000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67400000000000004,"15min":0.68200000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.41699999999999998},"max":{"1min":0.93999999999999995,"5min":1.012,"15min":1.169},"last":0.59299999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65500000000000003,"15min":0.67700000000000005},"min":{"1min":0.505,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89800000000000002,"5min":0.91600000000000004,"15min":1.071},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.68400000000000005,"15min":0.69999999999999996},"min":{"1min":0.48299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.94599999999999995,"5min":1.028,"15min":1.1160000000000001},"last":0.69899999999999995}]},{"osd":7,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":1.004,"15min":0.82599999999999996},"min":{"1min":0.54500000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.91500000000000004,"5min":25.965,"15min":25.965},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.73899999999999999,"5min":1.109,"15min":0.86599999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.433},"max":{"1min":0.93799999999999994,"5min":33.085000000000001,"15min":33.085000000000001},"last":0.68200000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575037,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306488,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065352,"statfs":{"total":96636764160,"available":96322920448,"internally_reserved":0,"allocated":1032192,"data_stored":519554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.39400000000000002,"15min":0.41999999999999998},"min":{"1min":0.29299999999999998,"5min":0.25800000000000001,"15min":0.22600000000000001},"max":{"1min":0.502,"5min":0.63400000000000001,"15min":9.0540000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.24299999999999999},"max":{"1min":0.56899999999999995,"5min":0.68500000000000005,"15min":4.1970000000000001},"last":0.78700000000000003}]},{"osd":1,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42699999999999999,"15min":0.42499999999999999},"min":{"1min":0.33300000000000002,"5min":0.248,"15min":0.221},"max":{"1min":0.55300000000000005,"5min":0.70499999999999996,"15min":0.82199999999999995},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.498,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":0.629,"5min":0.65900000000000003,"15min":12.157},"last":0.67200000000000004}]},{"osd":3,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47799999999999998,"15min":0.48599999999999999},"min":{"1min":0.38800000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.59299999999999997,"5min":0.65500000000000003,"15min":0.86499999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.47599999999999998,"15min":0.499},"min":{"1min":0.34599999999999997,"5min":0.32200000000000001,"15min":0.30399999999999999},"max":{"1min":0.60199999999999998,"5min":0.71099999999999997,"15min":9.0879999999999992},"last":0.749}]},{"osd":4,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65000000000000002,"15min":0.66600000000000004},"min":{"1min":0.499,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.86599999999999999,"5min":1.087,"15min":9.0150000000000006},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.65800000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.371},"max":{"1min":0.80200000000000005,"5min":0.98099999999999998,"15min":4.2480000000000002},"last":0.82799999999999996}]},{"osd":5,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78600000000000003,"5min":1.0680000000000001,"15min":4.3200000000000003},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68899999999999995,"15min":0.68200000000000005},"min":{"1min":0.58899999999999997,"5min":0.39800000000000002,"15min":0.35199999999999998},"max":{"1min":0.82399999999999995,"5min":0.95199999999999996,"15min":1.085},"last":0.876}]},{"osd":6,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68899999999999995,"15min":0.68400000000000005},"min":{"1min":0.51500000000000001,"5min":0.439,"15min":0.40899999999999997},"max":{"1min":0.79400000000000004,"5min":0.995,"15min":1.1080000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69199999999999995,"15min":0.71299999999999997},"min":{"1min":0.46200000000000002,"5min":0.443,"15min":0.38800000000000001},"max":{"1min":0.83999999999999997,"5min":1.1299999999999999,"15min":8.9700000000000006},"last":0.83799999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70799999999999996,"15min":0.71699999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.85599999999999998,"5min":1.0029999999999999,"15min":4.282},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.57499999999999996,"5min":0.48199999999999998,"15min":0.42099999999999999},"max":{"1min":0.87,"5min":1.052,"15min":1.139},"last":0.98199999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575036,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306488,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065352,"statfs":{"total":96636764160,"available":96322920448,"internally_reserved":0,"allocated":1032192,"data_stored":519554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40300000000000002,"15min":0.39800000000000002},"min":{"1min":0.32300000000000001,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.56999999999999995,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44900000000000001,"15min":0.48099999999999998},"min":{"1min":0.315,"5min":0.255,"15min":0.255},"max":{"1min":0.59299999999999997,"5min":1.2290000000000001,"15min":11.968},"last":0.32100000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.496},"min":{"1min":0.33200000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.623,"5min":1.633,"15min":12.050000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.48899999999999999,"15min":0.48599999999999999},"min":{"1min":0.378,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.59999999999999998,"5min":1.0760000000000001,"15min":1.2150000000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.49299999999999999,"15min":0.51900000000000002},"min":{"1min":0.39800000000000002,"5min":0.29699999999999999,"15min":0.28999999999999998},"max":{"1min":0.70399999999999996,"5min":1.4059999999999999,"15min":12.234},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46899999999999997,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.59499999999999997,"5min":1.1870000000000001,"15min":11.925000000000001},"last":0.435}]},{"osd":4,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69599999999999995,"15min":0.70599999999999996},"min":{"1min":0.56999999999999995,"5min":0.46999999999999997,"15min":0.40200000000000002},"max":{"1min":0.84899999999999998,"5min":3.306,"15min":12.099},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.66400000000000003,"15min":0.66600000000000004},"min":{"1min":0.59599999999999997,"5min":0.40200000000000002,"15min":0.379},"max":{"1min":0.82999999999999996,"5min":0.95799999999999996,"15min":1.0649999999999999},"last":0.65600000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.66500000000000004,"15min":0.70699999999999996},"min":{"1min":0.56299999999999994,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.79800000000000004,"5min":1.593,"15min":12.199999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68999999999999995,"15min":0.67800000000000005},"min":{"1min":0.53400000000000003,"5min":0.45300000000000001,"15min":0.436},"max":{"1min":0.92300000000000004,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.59099999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69499999999999995,"15min":0.72099999999999997},"min":{"1min":0.56100000000000005,"5min":0.442,"15min":0.442},"max":{"1min":0.88800000000000001,"5min":1.506,"15min":12.132999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70099999999999996,"15min":0.69799999999999995},"min":{"1min":0.54600000000000004,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.88200000000000001,"5min":1.427,"15min":1.427},"last":0.624}]},{"osd":7,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.72699999999999998,"15min":0.70299999999999996},"min":{"1min":0.52500000000000002,"5min":0.48499999999999999,"15min":0.42499999999999999},"max":{"1min":0.86199999999999999,"5min":3.238,"15min":3.238},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.78200000000000003},"min":{"1min":0.625,"5min":0.48899999999999999,"15min":0.48699999999999999},"max":{"1min":0.91300000000000003,"5min":3.351,"15min":12.167},"last":0.72199999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575038,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298296,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073544,"statfs":{"total":96636764160,"available":96331309056,"internally_reserved":0,"allocated":1032192,"data_stored":514090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57599999999999996,"15min":0.58399999999999996},"min":{"1min":0.439,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78300000000000003,"5min":1.21,"15min":1.21},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57499999999999996,"15min":0.58399999999999996},"min":{"1min":0.44900000000000001,"5min":0.309,"15min":0.28100000000000003},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":1.161},"last":0.49399999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.44900000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":1.069},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.59399999999999997},"min":{"1min":0.47199999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":1.085},"last":0.42199999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56299999999999994,"15min":0.57499999999999996},"min":{"1min":0.48499999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76500000000000001,"5min":0.81100000000000005,"15min":1.052},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60699999999999998,"15min":0.61199999999999999},"min":{"1min":0.41399999999999998,"5min":0.39900000000000002,"15min":0.373},"max":{"1min":0.81599999999999995,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.54700000000000004}]},{"osd":3,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58199999999999996,"15min":0.59399999999999997},"min":{"1min":0.41299999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.78300000000000003,"5min":0.83499999999999996,"15min":1.1299999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79100000000000004,"5min":1.05,"15min":1.1419999999999999},"last":0.56499999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.317,"5min":0.25700000000000001,"15min":0.247},"max":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.63900000000000001}]},{"osd":6,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48499999999999999,"15min":0.47799999999999998},"min":{"1min":0.35799999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.73899999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53300000000000003,"15min":0.53000000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.311},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.95699999999999996},"last":0.65900000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.35799999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.57699999999999996},"min":{"1min":0.42499999999999999,"5min":0.41799999999999998,"15min":0.32200000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.879},"last":0.61599999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575038,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298300,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073540,"statfs":{"total":96636764160,"available":96331304960,"internally_reserved":0,"allocated":1036288,"data_stored":519576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.58899999999999997,"15min":0.57599999999999996},"min":{"1min":0.46300000000000002,"5min":0.29999999999999999,"15min":0.26000000000000001},"max":{"1min":0.78900000000000003,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57199999999999995,"15min":0.58399999999999996},"min":{"1min":0.45400000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77400000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.64700000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.625,"15min":0.623},"min":{"1min":0.45500000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79500000000000004,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.64300000000000002,"15min":0.63300000000000001},"min":{"1min":0.499,"5min":0.435,"15min":0.379},"max":{"1min":0.80500000000000005,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.71199999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.46700000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77000000000000002,"5min":0.83699999999999997,"15min":0.83799999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60399999999999998,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81100000000000005,"5min":2.0600000000000001,"15min":6.4589999999999996},"last":0.59399999999999997}]},{"osd":3,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.50700000000000001,"5min":0.35999999999999999,"15min":0.314},"max":{"1min":0.755,"5min":0.86199999999999999,"15min":0.88400000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.60599999999999998,"15min":0.61499999999999999},"min":{"1min":0.47499999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77900000000000003,"5min":2.5630000000000002,"15min":6.0629999999999997},"last":0.52100000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.47399999999999998,"15min":0.46999999999999997},"min":{"1min":0.35599999999999998,"5min":0.28100000000000003,"15min":0.26300000000000001},"max":{"1min":0.56599999999999995,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.503,"15min":0.495},"min":{"1min":0.36899999999999999,"5min":0.33000000000000002,"15min":0.28100000000000003},"max":{"1min":0.69999999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.42699999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.31,"15min":0.27000000000000002},"max":{"1min":0.66600000000000004,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53200000000000003,"15min":0.52700000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.29299999999999998},"max":{"1min":0.71899999999999997,"5min":1.99,"15min":1.99},"last":0.45400000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.54900000000000004},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.76200000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33300000000000002},"max":{"1min":0.81499999999999995,"5min":0.82199999999999995,"15min":0.83099999999999996},"last":0.61899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:45:24.381 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-26T23:45:24.382 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-26T23:45:24.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.559+0000 7fd9a1910700 1 Processor -- start 2024-01-26T23:45:24.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.559+0000 7fd9a1910700 1 -- start start 2024-01-26T23:45:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.560+0000 7fd9a1910700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd99c1599d0 0x7fd99c159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.560+0000 7fd9a1910700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd99c15d8c0 0x7fd99c15dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.560+0000 7fd9a1910700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd99c0ab620 con 0x7fd99c0ab1c0 2024-01-26T23:45:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.560+0000 7fd9a1910700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd99c15e260 con 0x7fd99c15d8c0 2024-01-26T23:45:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.560+0000 7fd9a1910700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd99c0b1a60 con 0x7fd99c1599d0 2024-01-26T23:45:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.560+0000 7fd99b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd99c15d8c0 0x7fd99c15dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.560+0000 7fd99a7fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd99c0ab1c0 0x7fd99c0b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43356/0 (socket says 172.21.15.46:43356) 2024-01-26T23:45:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.560+0000 7fd99a7fc700 1 -- 172.21.15.46:0/3707831479 learned_addr learned my addr 172.21.15.46:0/3707831479 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.560+0000 7fd99affd700 1 --2- 172.21.15.46:0/3707831479 >> v2:172.21.15.106:3300/0 conn(0x7fd99c1599d0 0x7fd99c159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.560+0000 7fd99b7fe700 1 -- 172.21.15.46:0/3707831479 >> v2:172.21.15.106:3300/0 conn(0x7fd99c1599d0 msgr2=0x7fd99c159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.560+0000 7fd99b7fe700 1 --2- 172.21.15.46:0/3707831479 >> v2:172.21.15.106:3300/0 conn(0x7fd99c1599d0 0x7fd99c159da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.560+0000 7fd99b7fe700 1 -- 172.21.15.46:0/3707831479 >> v1:172.21.15.46:6789/0 conn(0x7fd99c0ab1c0 legacy=0x7fd99c0b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.561+0000 7fd99b7fe700 1 -- 172.21.15.46:0/3707831479 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99c0b11c0 con 0x7fd99c15d8c0 2024-01-26T23:45:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.560+0000 7fd999ffb700 1 -- 172.21.15.46:0/3707831479 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164291818 0 0) 0x7fd984001110 con 0x7fd99c0ab1c0 2024-01-26T23:45:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.561+0000 7fd99affd700 1 --2- 172.21.15.46:0/3707831479 >> v2:172.21.15.106:3300/0 conn(0x7fd99c1599d0 0x7fd99c159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.561+0000 7fd99b7fe700 1 --2- 172.21.15.46:0/3707831479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c15d8c0 0x7fd99c15dd20 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd99c08aa50 tx=0x7fd994009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8d0a7f7524750df server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.561+0000 7fd999ffb700 1 -- 172.21.15.46:0/3707831479 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99400e070 con 0x7fd99c15d8c0 2024-01-26T23:45:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.561+0000 7fd999ffb700 1 -- 172.21.15.46:0/3707831479 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99400bc30 con 0x7fd99c15d8c0 2024-01-26T23:45:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.561+0000 7fd999ffb700 1 -- 172.21.15.46:0/3707831479 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd994012430 con 0x7fd99c15d8c0 2024-01-26T23:45:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.561+0000 7fd9a1910700 1 -- 172.21.15.46:0/3707831479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c15d8c0 msgr2=0x7fd99c15dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.561+0000 7fd9a1910700 1 --2- 172.21.15.46:0/3707831479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c15d8c0 0x7fd99c15dd20 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd99c08aa50 tx=0x7fd994009780 comp rx=0 tx=0).stop 2024-01-26T23:45:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.561+0000 7fd9a1910700 1 -- 172.21.15.46:0/3707831479 shutdown_connections 2024-01-26T23:45:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.561+0000 7fd9a1910700 1 --2- 172.21.15.46:0/3707831479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c15d8c0 0x7fd99c15dd20 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.561+0000 7fd9a1910700 1 --2- 172.21.15.46:0/3707831479 >> v2:172.21.15.106:3300/0 conn(0x7fd99c1599d0 0x7fd99c159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.561+0000 7fd9a1910700 1 -- 172.21.15.46:0/3707831479 >> 172.21.15.46:0/3707831479 conn(0x7fd99c06f840 msgr2=0x7fd99c087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:24.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.562+0000 7fd9a1910700 1 -- 172.21.15.46:0/3707831479 shutdown_connections 2024-01-26T23:45:24.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.562+0000 7fd9a1910700 1 -- 172.21.15.46:0/3707831479 wait complete. 2024-01-26T23:45:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.563+0000 7fd9a1910700 1 Processor -- start 2024-01-26T23:45:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.563+0000 7fd9a1910700 1 -- start start 2024-01-26T23:45:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd9a1910700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd99c0ab1c0 0x7fd99c158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd9a1910700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c1599d0 0x7fd99c0af460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd9a1910700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd99c15d8c0 0x7fd99c09d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd9a1910700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c0aefd0 con 0x7fd99c1599d0 2024-01-26T23:45:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd9a1910700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c159460 con 0x7fd99c0ab1c0 2024-01-26T23:45:24.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd9a1910700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c0ae6a0 con 0x7fd99c15d8c0 2024-01-26T23:45:24.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd99a7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c1599d0 0x7fd99c0af460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:24.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd99a7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c1599d0 0x7fd99c0af460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37234/0 (socket says 172.21.15.46:37234) 2024-01-26T23:45:24.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd99a7fc700 1 -- 172.21.15.46:0/97590940 learned_addr learned my addr 172.21.15.46:0/97590940 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:24.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd99affd700 1 --2- 172.21.15.46:0/97590940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd99c0ab1c0 0x7fd99c158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:24.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd99b7fe700 1 --2- 172.21.15.46:0/97590940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd99c15d8c0 0x7fd99c09d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:24.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd99a7fc700 1 -- 172.21.15.46:0/97590940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd99c15d8c0 msgr2=0x7fd99c09d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:24.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd99a7fc700 1 --2- 172.21.15.46:0/97590940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd99c15d8c0 0x7fd99c09d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd99a7fc700 1 -- 172.21.15.46:0/97590940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd99c0ab1c0 msgr2=0x7fd99c158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:24.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd99a7fc700 1 --2- 172.21.15.46:0/97590940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd99c0ab1c0 0x7fd99c158f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd99a7fc700 1 -- 172.21.15.46:0/97590940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd994009a20 con 0x7fd99c1599d0 2024-01-26T23:45:24.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd99b7fe700 1 --2- 172.21.15.46:0/97590940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd99c15d8c0 0x7fd99c09d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:45:24.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.564+0000 7fd99a7fc700 1 --2- 172.21.15.46:0/97590940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c1599d0 0x7fd99c0af460 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd98400c810 tx=0x7fd984007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:24.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.565+0000 7fd992ffd700 1 -- 172.21.15.46:0/97590940 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd984005b30 con 0x7fd99c1599d0 2024-01-26T23:45:24.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.565+0000 7fd992ffd700 1 -- 172.21.15.46:0/97590940 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd984005c90 con 0x7fd99c1599d0 2024-01-26T23:45:24.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.565+0000 7fd9a1910700 1 -- 172.21.15.46:0/97590940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd99c0ae980 con 0x7fd99c1599d0 2024-01-26T23:45:24.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.565+0000 7fd992ffd700 1 -- 172.21.15.46:0/97590940 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd984007300 con 0x7fd99c1599d0 2024-01-26T23:45:24.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.565+0000 7fd9a1910700 1 -- 172.21.15.46:0/97590940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd99c0ab9f0 con 0x7fd99c1599d0 2024-01-26T23:45:24.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.566+0000 7fd9a1910700 1 -- 172.21.15.46:0/97590940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd99c0af180 con 0x7fd99c1599d0 2024-01-26T23:45:24.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.566+0000 7fd992ffd700 1 -- 172.21.15.46:0/97590940 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd9840076f0 con 0x7fd99c1599d0 2024-01-26T23:45:24.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.566+0000 7fd992ffd700 1 --2- 172.21.15.46:0/97590940 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd97c066f30 0x7fd97c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:24.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.567+0000 7fd992ffd700 1 -- 172.21.15.46:0/97590940 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fd9840841f0 con 0x7fd99c1599d0 2024-01-26T23:45:24.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.567+0000 7fd99affd700 1 --2- 172.21.15.46:0/97590940 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd97c066f30 0x7fd97c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:24.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.567+0000 7fd99affd700 1 --2- 172.21.15.46:0/97590940 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd97c066f30 0x7fd97c0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd98c006ea0 tx=0x7fd98c008040 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:24.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.571+0000 7fd992ffd700 1 -- 172.21.15.46:0/97590940 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd99c0af180 con 0x7fd99c1599d0 2024-01-26T23:45:24.749 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:45:24.752 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:45:24.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.870+0000 7fd9a1910700 1 -- 172.21.15.46:0/97590940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd99c0ab620 con 0x7fd99c1599d0 2024-01-26T23:45:24.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.874+0000 7fd992ffd700 1 -- 172.21.15.46:0/97590940 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+0 (secure 0 0 0) 0x7fd99c0ab620 con 0x7fd99c1599d0 2024-01-26T23:45:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.877+0000 7fd9a1910700 1 -- 172.21.15.46:0/97590940 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd97c066f30 msgr2=0x7fd97c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.877+0000 7fd9a1910700 1 --2- 172.21.15.46:0/97590940 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd97c066f30 0x7fd97c0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd98c006ea0 tx=0x7fd98c008040 comp rx=0 tx=0).stop 2024-01-26T23:45:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.877+0000 7fd9a1910700 1 -- 172.21.15.46:0/97590940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c1599d0 msgr2=0x7fd99c0af460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.877+0000 7fd9a1910700 1 --2- 172.21.15.46:0/97590940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c1599d0 0x7fd99c0af460 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd98400c810 tx=0x7fd984007c50 comp rx=0 tx=0).stop 2024-01-26T23:45:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.877+0000 7fd9a1910700 1 -- 172.21.15.46:0/97590940 shutdown_connections 2024-01-26T23:45:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.877+0000 7fd9a1910700 1 --2- 172.21.15.46:0/97590940 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd97c066f30 0x7fd97c0693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.877+0000 7fd9a1910700 1 --2- 172.21.15.46:0/97590940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd99c15d8c0 0x7fd99c09d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.877+0000 7fd9a1910700 1 --2- 172.21.15.46:0/97590940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd99c0ab1c0 0x7fd99c158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.877+0000 7fd9a1910700 1 --2- 172.21.15.46:0/97590940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c1599d0 0x7fd99c0af460 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.877+0000 7fd9a1910700 1 -- 172.21.15.46:0/97590940 >> 172.21.15.46:0/97590940 conn(0x7fd99c06f840 msgr2=0x7fd99c15cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.877+0000 7fd9a1910700 1 -- 172.21.15.46:0/97590940 shutdown_connections 2024-01-26T23:45:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:24.877+0000 7fd9a1910700 1 -- 172.21.15.46:0/97590940 wait complete. 2024-01-26T23:45:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-01-26T23:45:24.891 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:45:25.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.066+0000 7fa0dc063700 1 Processor -- start 2024-01-26T23:45:25.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.066+0000 7fa0dc063700 1 -- start start 2024-01-26T23:45:25.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0dc063700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa0d40ccc20 0x7fa0d4068340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0dc063700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0d40ccff0 0x7fa0d406eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0dc063700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d4158020 con 0x7fa0d40b2590 2024-01-26T23:45:25.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0dc063700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0d4158300 con 0x7fa0d40ccc20 2024-01-26T23:45:25.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0dc063700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d41585e0 con 0x7fa0d40ccff0 2024-01-26T23:45:25.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0da600700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa0d40b2590 0x7fa0d415b5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43362/0 (socket says 172.21.15.46:43362) 2024-01-26T23:45:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0da600700 1 -- 172.21.15.46:0/3656080479 learned_addr learned my addr 172.21.15.46:0/3656080479 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d9dff700 1 --2- 172.21.15.46:0/3656080479 >> v2:172.21.15.106:3300/0 conn(0x7fa0d40ccc20 0x7fa0d4068340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d95fe700 1 --2- 172.21.15.46:0/3656080479 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0d40ccff0 0x7fa0d406eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d8dfd700 1 -- 172.21.15.46:0/3656080479 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974447365 0 0) 0x7fa0d4158020 con 0x7fa0d40b2590 2024-01-26T23:45:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d8dfd700 1 -- 172.21.15.46:0/3656080479 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0c0003570 con 0x7fa0d40b2590 2024-01-26T23:45:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d8dfd700 1 -- 172.21.15.46:0/3656080479 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2833583697 0 0) 0x7fa0c0003570 con 0x7fa0d40b2590 2024-01-26T23:45:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d8dfd700 1 -- 172.21.15.46:0/3656080479 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0d40ccff0 msgr2=0x7fa0d406eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d8dfd700 1 --2- 172.21.15.46:0/3656080479 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0d40ccff0 0x7fa0d406eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d8dfd700 1 -- 172.21.15.46:0/3656080479 >> v2:172.21.15.106:3300/0 conn(0x7fa0d40ccc20 msgr2=0x7fa0d4068340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d8dfd700 1 --2- 172.21.15.46:0/3656080479 >> v2:172.21.15.106:3300/0 conn(0x7fa0d40ccc20 0x7fa0d4068340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d8dfd700 1 -- 172.21.15.46:0/3656080479 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d41588c0 con 0x7fa0d40b2590 2024-01-26T23:45:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d8dfd700 1 -- 172.21.15.46:0/3656080479 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fa0cc0017f0 con 0x7fa0d40b2590 2024-01-26T23:45:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d9dff700 1 --2- 172.21.15.46:0/3656080479 >> v2:172.21.15.106:3300/0 conn(0x7fa0d40ccc20 0x7fa0d4068340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:25.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d8dfd700 1 -- 172.21.15.46:0/3656080479 >> v1:172.21.15.46:6789/0 conn(0x7fa0d40b2590 legacy=0x7fa0d415b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:25.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d8dfd700 1 --2- 172.21.15.46:0/3656080479 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0d40ccff0 0x7fa0c0003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d8dfd700 1 --2- 172.21.15.46:0/3656080479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c0003fe0 0x7fa0c0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d8dfd700 1 --2- 172.21.15.46:0/3656080479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0c0004d50 0x7fa0c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d8dfd700 1 -- 172.21.15.46:0/3656080479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d4158300 con 0x7fa0c0003fe0 2024-01-26T23:45:25.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d8dfd700 1 -- 172.21.15.46:0/3656080479 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d41585e0 con 0x7fa0c0004d50 2024-01-26T23:45:25.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d8dfd700 1 -- 172.21.15.46:0/3656080479 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c000ba90 con 0x7fa0d40ccff0 2024-01-26T23:45:25.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d95fe700 1 --2- 172.21.15.46:0/3656080479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c0003fe0 0x7fa0c0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d9dff700 1 --2- 172.21.15.46:0/3656080479 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0d40ccff0 0x7fa0c0003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0da600700 1 --2- 172.21.15.46:0/3656080479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0c0004d50 0x7fa0c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d95fe700 1 -- 172.21.15.46:0/3656080479 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0d40ccff0 msgr2=0x7fa0c0003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:25.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d95fe700 1 --2- 172.21.15.46:0/3656080479 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0d40ccff0 0x7fa0c0003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.068+0000 7fa0d95fe700 1 -- 172.21.15.46:0/3656080479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0c0004d50 msgr2=0x7fa0c00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:25.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.069+0000 7fa0d95fe700 1 --2- 172.21.15.46:0/3656080479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0c0004d50 0x7fa0c00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.069+0000 7fa0d95fe700 1 -- 172.21.15.46:0/3656080479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c0003570 con 0x7fa0c0003fe0 2024-01-26T23:45:25.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.069+0000 7fa0d95fe700 1 --2- 172.21.15.46:0/3656080479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c0003fe0 0x7fa0c0004810 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fa0d40cd450 tx=0x7fa0c40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d41a94c6ed02b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:25.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.069+0000 7fa0d8dfd700 1 -- 172.21.15.46:0/3656080479 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c400e6b0 con 0x7fa0c0003fe0 2024-01-26T23:45:25.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.069+0000 7fa0d8dfd700 1 -- 172.21.15.46:0/3656080479 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c400e810 con 0x7fa0c0003fe0 2024-01-26T23:45:25.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.069+0000 7fa0d8dfd700 1 -- 172.21.15.46:0/3656080479 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c400ebd0 con 0x7fa0c0003fe0 2024-01-26T23:45:25.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.070+0000 7fa0dc063700 1 -- 172.21.15.46:0/3656080479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c0003fe0 msgr2=0x7fa0c0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:25.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.070+0000 7fa0dc063700 1 --2- 172.21.15.46:0/3656080479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c0003fe0 0x7fa0c0004810 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fa0d40cd450 tx=0x7fa0c40091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.070+0000 7fa0da600700 1 -- 172.21.15.46:0/3656080479 reap_dead start 2024-01-26T23:45:25.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.070+0000 7fa0dc063700 1 -- 172.21.15.46:0/3656080479 shutdown_connections 2024-01-26T23:45:25.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.070+0000 7fa0dc063700 1 -- 172.21.15.46:0/3656080479 >> 172.21.15.46:0/3656080479 conn(0x7fa0d406b630 msgr2=0x7fa0d40aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:25.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.070+0000 7fa0dc063700 1 -- 172.21.15.46:0/3656080479 shutdown_connections 2024-01-26T23:45:25.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.070+0000 7fa0dc063700 1 -- 172.21.15.46:0/3656080479 wait complete. 2024-01-26T23:45:25.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.071+0000 7fa0dc063700 1 Processor -- start 2024-01-26T23:45:25.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.071+0000 7fa0dc063700 1 -- start start 2024-01-26T23:45:25.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0dc063700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0c0003fe0 0x7fa0d413c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0dc063700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0d40c71b0 0x7fa0d40c7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0dc063700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d415cb50 0x7fa0d415cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0dc063700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d415d500 con 0x7fa0d415cb50 2024-01-26T23:45:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0dc063700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d40c8940 con 0x7fa0d40c71b0 2024-01-26T23:45:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0dc063700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d413c1c0 con 0x7fa0c0003fe0 2024-01-26T23:45:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0da600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d415cb50 0x7fa0d415cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0da600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d415cb50 0x7fa0d415cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37244/0 (socket says 172.21.15.46:37244) 2024-01-26T23:45:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0da600700 1 -- 172.21.15.46:0/512180319 learned_addr learned my addr 172.21.15.46:0/512180319 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0d95fe700 1 --2- 172.21.15.46:0/512180319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0d40c71b0 0x7fa0d40c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0d9dff700 1 --2- 172.21.15.46:0/512180319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0c0003fe0 0x7fa0d413c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0da600700 1 -- 172.21.15.46:0/512180319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0c0003fe0 msgr2=0x7fa0d413c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0da600700 1 --2- 172.21.15.46:0/512180319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0c0003fe0 0x7fa0d413c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0da600700 1 -- 172.21.15.46:0/512180319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0d40c71b0 msgr2=0x7fa0d40c7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0da600700 1 --2- 172.21.15.46:0/512180319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0d40c71b0 0x7fa0d40c7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0da600700 1 -- 172.21.15.46:0/512180319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d40c7d30 con 0x7fa0d415cb50 2024-01-26T23:45:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0d95fe700 1 --2- 172.21.15.46:0/512180319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0d40c71b0 0x7fa0d40c7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.072+0000 7fa0da600700 1 --2- 172.21.15.46:0/512180319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d415cb50 0x7fa0d415cfc0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4068940 tx=0x7fa0cc002210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.073+0000 7fa0c9ffb700 1 -- 172.21.15.46:0/512180319 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc007b60 con 0x7fa0d415cb50 2024-01-26T23:45:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.073+0000 7fa0c9ffb700 1 -- 172.21.15.46:0/512180319 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0cc00ad80 con 0x7fa0d415cb50 2024-01-26T23:45:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.073+0000 7fa0dc063700 1 -- 172.21.15.46:0/512180319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d40c8010 con 0x7fa0d415cb50 2024-01-26T23:45:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.073+0000 7fa0c9ffb700 1 -- 172.21.15.46:0/512180319 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc013640 con 0x7fa0d415cb50 2024-01-26T23:45:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.073+0000 7fa0dc063700 1 -- 172.21.15.46:0/512180319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d40c85d0 con 0x7fa0d415cb50 2024-01-26T23:45:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.074+0000 7fa0dc063700 1 -- 172.21.15.46:0/512180319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d404c710 con 0x7fa0d415cb50 2024-01-26T23:45:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.074+0000 7fa0c9ffb700 1 -- 172.21.15.46:0/512180319 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa0cc013a30 con 0x7fa0d415cb50 2024-01-26T23:45:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.075+0000 7fa0c9ffb700 1 --2- 172.21.15.46:0/512180319 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa0b8066f30 0x7fa0b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.075+0000 7fa0c9ffb700 1 -- 172.21.15.46:0/512180319 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fa0cc085160 con 0x7fa0d415cb50 2024-01-26T23:45:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.075+0000 7fa0d9dff700 1 --2- 172.21.15.46:0/512180319 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa0b8066f30 0x7fa0b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.076+0000 7fa0d9dff700 1 --2- 172.21.15.46:0/512180319 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa0b8066f30 0x7fa0b80693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0006f00 tx=0x7fa0d00091a0 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:25.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.081+0000 7fa0c9ffb700 1 -- 172.21.15.46:0/512180319 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa0cc052100 con 0x7fa0d415cb50 2024-01-26T23:45:25.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.331+0000 7fa0dc063700 1 -- 172.21.15.46:0/512180319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0d40c9d60 con 0x7fa0d415cb50 2024-01-26T23:45:25.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.332+0000 7fa0c9ffb700 1 -- 172.21.15.46:0/512180319 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4604 (secure 0 0 0) 0x7fa0cc055720 con 0x7fa0d415cb50 2024-01-26T23:45:25.333 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:25.333 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/4083796808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":4083796808},{"type":"v1","addr":"172.21.15.106:6836","nonce":4083796808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":6172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/1769644590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1769644590},{"type":"v1","addr":"172.21.15.106:6837","nonce":1769644590}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":6184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1994423361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1994423361},{"type":"v1","addr":"172.21.15.46:6835","nonce":1994423361}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:45:19.187256+0000","modified":"2024-01-26T23:45:22.777939+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6366":{"gid":6366,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/1799649681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1799649681},{"type":"v1","addr":"172.21.15.46:6834","nonce":1799649681}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2024-01-26T23:45:25.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.334+0000 7fa0dc063700 1 -- 172.21.15.46:0/512180319 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa0b8066f30 msgr2=0x7fa0b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:25.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.334+0000 7fa0dc063700 1 --2- 172.21.15.46:0/512180319 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa0b8066f30 0x7fa0b80693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0006f00 tx=0x7fa0d00091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.334+0000 7fa0dc063700 1 -- 172.21.15.46:0/512180319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d415cb50 msgr2=0x7fa0d415cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:25.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.334+0000 7fa0dc063700 1 --2- 172.21.15.46:0/512180319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d415cb50 0x7fa0d415cfc0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4068940 tx=0x7fa0cc002210 comp rx=0 tx=0).stop 2024-01-26T23:45:25.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.334+0000 7fa0dc063700 1 -- 172.21.15.46:0/512180319 shutdown_connections 2024-01-26T23:45:25.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.334+0000 7fa0dc063700 1 --2- 172.21.15.46:0/512180319 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa0b8066f30 0x7fa0b80693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.334+0000 7fa0dc063700 1 --2- 172.21.15.46:0/512180319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d415cb50 0x7fa0d415cfc0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.334+0000 7fa0dc063700 1 --2- 172.21.15.46:0/512180319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0c0003fe0 0x7fa0d413c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.334+0000 7fa0dc063700 1 --2- 172.21.15.46:0/512180319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0d40c71b0 0x7fa0d40c7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.334+0000 7fa0dc063700 1 -- 172.21.15.46:0/512180319 >> 172.21.15.46:0/512180319 conn(0x7fa0d406b630 msgr2=0x7fa0d4159dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:25.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.334+0000 7fa0dc063700 1 -- 172.21.15.46:0/512180319 shutdown_connections 2024-01-26T23:45:25.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.334+0000 7fa0dc063700 1 -- 172.21.15.46:0/512180319 wait complete. 2024-01-26T23:45:25.337 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 36 2024-01-26T23:45:25.753 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-26T23:45:19.187256+0000', 'modified': '2024-01-26T23:45:22.777939+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6366}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6366': {'gid': 6366, 'name': 'a', 'rank': 0, 'incarnation': 32, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6834/1799649681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 1799649681}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 1799649681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-26T23:45:25.753 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-26T23:45:25.754 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-26T23:45:25.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.896+0000 7faae98c3700 1 Processor -- start 2024-01-26T23:45:25.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.896+0000 7faae98c3700 1 -- start start 2024-01-26T23:45:25.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae98c3700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7faae40ad070 0x7faae40ae610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae98c3700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae40ae180 con 0x7faae40cc1c0 2024-01-26T23:45:25.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae98c3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae4156c60 con 0x7faae40cbdf0 2024-01-26T23:45:25.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae98c3700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faae40aeb50 con 0x7faae40ad070 2024-01-26T23:45:25.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae27fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faae40cc1c0 0x7faae40c98b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43366/0 (socket says 172.21.15.46:43366) 2024-01-26T23:45:25.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae27fc700 1 -- 172.21.15.46:0/3278035958 learned_addr learned my addr 172.21.15.46:0/3278035958 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:25.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae37fe700 1 --2- 172.21.15.46:0/3278035958 >> v2:172.21.15.106:3300/0 conn(0x7faae40ad070 0x7faae40ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae1ffb700 1 -- 172.21.15.46:0/3278035958 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050568656 0 0) 0x7faae40ae180 con 0x7faae40cc1c0 2024-01-26T23:45:25.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae1ffb700 1 -- 172.21.15.46:0/3278035958 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faad0003570 con 0x7faae40cc1c0 2024-01-26T23:45:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae1ffb700 1 -- 172.21.15.46:0/3278035958 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4277788393 0 0) 0x7faad0003570 con 0x7faae40cc1c0 2024-01-26T23:45:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae1ffb700 1 -- 172.21.15.46:0/3278035958 >> v2:172.21.15.106:3300/0 conn(0x7faae40ad070 msgr2=0x7faae40ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae1ffb700 1 --2- 172.21.15.46:0/3278035958 >> v2:172.21.15.106:3300/0 conn(0x7faae40ad070 0x7faae40ae610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae1ffb700 1 -- 172.21.15.46:0/3278035958 >> v1:172.21.15.106:6789/0 conn(0x7faae40cbdf0 legacy=0x7faae40b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae1ffb700 1 -- 172.21.15.46:0/3278035958 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae40ad570 con 0x7faae40cc1c0 2024-01-26T23:45:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae1ffb700 1 -- 172.21.15.46:0/3278035958 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7faad4001470 con 0x7faae40cc1c0 2024-01-26T23:45:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae1ffb700 1 -- 172.21.15.46:0/3278035958 >> v1:172.21.15.46:6789/0 conn(0x7faae40cc1c0 legacy=0x7faae40c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae1ffb700 1 --2- 172.21.15.46:0/3278035958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faad0003a60 0x7faad0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae1ffb700 1 --2- 172.21.15.46:0/3278035958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faad0004810 0x7faad0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae1ffb700 1 --2- 172.21.15.46:0/3278035958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faad00051c0 0x7faad000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae1ffb700 1 -- 172.21.15.46:0/3278035958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae40aeb50 con 0x7faad00051c0 2024-01-26T23:45:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae1ffb700 1 -- 172.21.15.46:0/3278035958 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad000c160 con 0x7faad0004810 2024-01-26T23:45:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae1ffb700 1 -- 172.21.15.46:0/3278035958 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad000c320 con 0x7faad0003a60 2024-01-26T23:45:25.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae37fe700 1 --2- 172.21.15.46:0/3278035958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faad00051c0 0x7faad000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae27fc700 1 --2- 172.21.15.46:0/3278035958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faad0004810 0x7faad0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae2ffd700 1 --2- 172.21.15.46:0/3278035958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faad0003a60 0x7faad0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae37fe700 1 -- 172.21.15.46:0/3278035958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faad0003a60 msgr2=0x7faad0003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:25.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae37fe700 1 --2- 172.21.15.46:0/3278035958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faad0003a60 0x7faad0003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae37fe700 1 -- 172.21.15.46:0/3278035958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faad0004810 msgr2=0x7faad0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:25.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae37fe700 1 --2- 172.21.15.46:0/3278035958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faad0004810 0x7faad0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.898+0000 7faae37fe700 1 -- 172.21.15.46:0/3278035958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad0003570 con 0x7faad00051c0 2024-01-26T23:45:25.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.899+0000 7faae37fe700 1 -- 172.21.15.46:0/3278035958 reap_dead start 2024-01-26T23:45:25.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.899+0000 7faae37fe700 1 --2- 172.21.15.46:0/3278035958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faad00051c0 0x7faad000ba60 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7faadc00e210 tx=0x7faadc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28aa00403f457f82 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:25.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.899+0000 7faae1ffb700 1 -- 172.21.15.46:0/3278035958 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc00eac0 con 0x7faad00051c0 2024-01-26T23:45:25.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.899+0000 7faae1ffb700 1 -- 172.21.15.46:0/3278035958 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faadc00f040 con 0x7faad00051c0 2024-01-26T23:45:25.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.899+0000 7faae1ffb700 1 -- 172.21.15.46:0/3278035958 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc00ee70 con 0x7faad00051c0 2024-01-26T23:45:25.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.899+0000 7faae98c3700 1 -- 172.21.15.46:0/3278035958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faad00051c0 msgr2=0x7faad000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:25.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.899+0000 7faae98c3700 1 --2- 172.21.15.46:0/3278035958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faad00051c0 0x7faad000ba60 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7faadc00e210 tx=0x7faadc00b040 comp rx=0 tx=0).stop 2024-01-26T23:45:25.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.899+0000 7faae98c3700 1 -- 172.21.15.46:0/3278035958 shutdown_connections 2024-01-26T23:45:25.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.899+0000 7faae98c3700 1 --2- 172.21.15.46:0/3278035958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faad00051c0 0x7faad000ba60 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.899+0000 7faae98c3700 1 -- 172.21.15.46:0/3278035958 >> 172.21.15.46:0/3278035958 conn(0x7faae4068400 msgr2=0x7faae40cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:25.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.900+0000 7faae98c3700 1 -- 172.21.15.46:0/3278035958 shutdown_connections 2024-01-26T23:45:25.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.900+0000 7faae98c3700 1 -- 172.21.15.46:0/3278035958 wait complete. 2024-01-26T23:45:25.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.901+0000 7faae98c3700 1 Processor -- start 2024-01-26T23:45:25.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.901+0000 7faae98c3700 1 -- start start 2024-01-26T23:45:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.902+0000 7faae98c3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faad00051c0 0x7faae416e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.902+0000 7faae98c3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faae4159850 0x7faae40c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.902+0000 7faae98c3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faae4153a80 0x7faae416a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.902+0000 7faae98c3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae416dc60 con 0x7faad00051c0 2024-01-26T23:45:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.902+0000 7faae98c3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae4159d10 con 0x7faae4153a80 2024-01-26T23:45:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.902+0000 7faae98c3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae416d610 con 0x7faae4159850 2024-01-26T23:45:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.902+0000 7faae2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faad00051c0 0x7faae416e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.902+0000 7faae2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faad00051c0 0x7faae416e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37262/0 (socket says 172.21.15.46:37262) 2024-01-26T23:45:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.902+0000 7faae2ffd700 1 -- 172.21.15.46:0/4176165163 learned_addr learned my addr 172.21.15.46:0/4176165163 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.902+0000 7faae27fc700 1 --2- 172.21.15.46:0/4176165163 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faae4159850 0x7faae40c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.902+0000 7faae37fe700 1 --2- 172.21.15.46:0/4176165163 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faae4153a80 0x7faae416a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.902+0000 7faae2ffd700 1 -- 172.21.15.46:0/4176165163 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faae4159850 msgr2=0x7faae40c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.902+0000 7faae2ffd700 1 --2- 172.21.15.46:0/4176165163 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faae4159850 0x7faae40c9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.902+0000 7faae2ffd700 1 -- 172.21.15.46:0/4176165163 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faae4153a80 msgr2=0x7faae416a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:25.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.902+0000 7faae2ffd700 1 --2- 172.21.15.46:0/4176165163 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faae4153a80 0x7faae416a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.902+0000 7faae2ffd700 1 -- 172.21.15.46:0/4176165163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae416d8f0 con 0x7faad00051c0 2024-01-26T23:45:25.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.902+0000 7faae2ffd700 1 --2- 172.21.15.46:0/4176165163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faad00051c0 0x7faae416e0f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7faacc002950 tx=0x7faacc007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:25.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.903+0000 7faadaffd700 1 -- 172.21.15.46:0/4176165163 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faacc003390 con 0x7faad00051c0 2024-01-26T23:45:25.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.903+0000 7faadaffd700 1 -- 172.21.15.46:0/4176165163 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faacc013600 con 0x7faad00051c0 2024-01-26T23:45:25.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.903+0000 7faae98c3700 1 -- 172.21.15.46:0/4176165163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae4159ff0 con 0x7faad00051c0 2024-01-26T23:45:25.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.903+0000 7faadaffd700 1 -- 172.21.15.46:0/4176165163 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faacc0139a0 con 0x7faad00051c0 2024-01-26T23:45:25.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.903+0000 7faae98c3700 1 -- 172.21.15.46:0/4176165163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae4153f00 con 0x7faad00051c0 2024-01-26T23:45:25.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.904+0000 7faadaffd700 1 -- 172.21.15.46:0/4176165163 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7faacc013d50 con 0x7faad00051c0 2024-01-26T23:45:25.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.905+0000 7faadaffd700 1 --2- 172.21.15.46:0/4176165163 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faac4066f60 0x7faac4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.905+0000 7faadaffd700 1 -- 172.21.15.46:0/4176165163 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6417+0+0 (secure 0 0 0) 0x7faacc084510 con 0x7faad00051c0 2024-01-26T23:45:25.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.905+0000 7faae27fc700 1 --2- 172.21.15.46:0/4176165163 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faac4066f60 0x7faac4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.905+0000 7faae27fc700 1 --2- 172.21.15.46:0/4176165163 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faac4066f60 0x7faac4069410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faae4159c40 tx=0x7faad4001e50 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:25.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.907+0000 7faaf2bc6b80 1 Processor -- start 2024-01-26T23:45:25.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.907+0000 7faaf2bc6b80 1 -- start start 2024-01-26T23:45:25.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faaf2bc6b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x562293659010 0x5622936593e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faaf2bc6b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622936a1360 0x562293683840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faaf2bc6b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622936585d0 0x562293683d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faaf2bc6b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x56229355c9f0 con 0x5622936a1360 2024-01-26T23:45:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faaf2bc6b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622935ae130 con 0x5622936585d0 2024-01-26T23:45:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faaf2bc6b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5622935ab930 con 0x562293659010 2024-01-26T23:45:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faae27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622936a1360 0x562293683840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faae27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622936a1360 0x562293683840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37268/0 (socket says 172.21.15.46:37268) 2024-01-26T23:45:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faae27fc700 1 -- 172.21.15.46:0/3509286035 learned_addr learned my addr 172.21.15.46:0/3509286035 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faae2ffd700 1 --2- 172.21.15.46:0/3509286035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x562293659010 0x5622936593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faae37fe700 1 --2- 172.21.15.46:0/3509286035 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622936585d0 0x562293683d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faae27fc700 1 -- 172.21.15.46:0/3509286035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x562293659010 msgr2=0x5622936593e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faae27fc700 1 --2- 172.21.15.46:0/3509286035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x562293659010 0x5622936593e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faae27fc700 1 -- 172.21.15.46:0/3509286035 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622936585d0 msgr2=0x562293683d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faae27fc700 1 --2- 172.21.15.46:0/3509286035 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622936585d0 0x562293683d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faae27fc700 1 -- 172.21.15.46:0/3509286035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622933e6b10 con 0x5622936a1360 2024-01-26T23:45:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faae27fc700 1 --2- 172.21.15.46:0/3509286035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622936a1360 0x562293683840 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7faad4007460 tx=0x7faad4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7383bdb0ce2826e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faac3fff700 1 -- 172.21.15.46:0/3509286035 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad401a040 con 0x5622936a1360 2024-01-26T23:45:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faac3fff700 1 -- 172.21.15.46:0/3509286035 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad4015ae0 con 0x5622936a1360 2024-01-26T23:45:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.908+0000 7faac3fff700 1 -- 172.21.15.46:0/3509286035 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad4015e60 con 0x5622936a1360 2024-01-26T23:45:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.909+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/3509286035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622936a1360 msgr2=0x562293683840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.909+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/3509286035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622936a1360 0x562293683840 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7faad4007460 tx=0x7faad4017000 comp rx=0 tx=0).stop 2024-01-26T23:45:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.909+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/3509286035 shutdown_connections 2024-01-26T23:45:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.909+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/3509286035 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622936585d0 0x562293683d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.909+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/3509286035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x562293659010 0x5622936593e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.909+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/3509286035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622936a1360 0x562293683840 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.909+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/3509286035 >> 172.21.15.46:0/3509286035 conn(0x562293658a70 msgr2=0x5622936a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.909+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/3509286035 shutdown_connections 2024-01-26T23:45:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.909+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/3509286035 wait complete. 2024-01-26T23:45:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.910+0000 7faaf2bc6b80 1 -- start start 2024-01-26T23:45:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.910+0000 7faaf2bc6b80 1 Processor -- start 2024-01-26T23:45:25.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.911+0000 7faaf2bc6b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622936585d0 0x5622936add00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.911+0000 7faaf2bc6b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562293659010 0x562293683610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.911+0000 7faaf2bc6b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622936a1360 0x562293683b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.911+0000 7faaf2bc6b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622935a7720 con 0x562293659010 2024-01-26T23:45:25.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.911+0000 7faaf2bc6b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622935531b0 con 0x5622936585d0 2024-01-26T23:45:25.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.911+0000 7faaf2bc6b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x562293552e50 con 0x5622936a1360 2024-01-26T23:45:25.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.911+0000 7faaf2bc6b80 10 client.0 _ll_register_callbacks cb 0x5622933b57b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:45:25.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.911+0000 7faae27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562293659010 0x562293683610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.911+0000 7faae27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562293659010 0x562293683610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37278/0 (socket says 172.21.15.46:37278) 2024-01-26T23:45:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.911+0000 7faae27fc700 1 -- 172.21.15.46:0/1849989293 learned_addr learned my addr 172.21.15.46:0/1849989293 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.911+0000 7faae2ffd700 1 --2- 172.21.15.46:0/1849989293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622936585d0 0x5622936add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.911+0000 7faae37fe700 1 --2- 172.21.15.46:0/1849989293 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622936a1360 0x562293683b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.912+0000 7faae27fc700 1 -- 172.21.15.46:0/1849989293 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622936a1360 msgr2=0x562293683b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.912+0000 7faae27fc700 1 --2- 172.21.15.46:0/1849989293 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622936a1360 0x562293683b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.912+0000 7faae27fc700 1 -- 172.21.15.46:0/1849989293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622936585d0 msgr2=0x5622936add00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.912+0000 7faae27fc700 1 --2- 172.21.15.46:0/1849989293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622936585d0 0x5622936add00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.912+0000 7faae27fc700 1 -- 172.21.15.46:0/1849989293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae40ad570 con 0x562293659010 2024-01-26T23:45:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.912+0000 7faae2ffd700 1 --2- 172.21.15.46:0/1849989293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622936585d0 0x5622936add00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.912+0000 7faae27fc700 1 --2- 172.21.15.46:0/1849989293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562293659010 0x562293683610 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7faad4007350 tx=0x7faad4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710d5c0483f4a302 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.912+0000 7faac17fa700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:45:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.912+0000 7faac17fa700 1 -- 172.21.15.46:0/1849989293 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad4021d70 con 0x562293659010 2024-01-26T23:45:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.912+0000 7faac17fa700 1 -- 172.21.15.46:0/1849989293 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 6417+0+0 (secure 0 0 0) 0x7faad402edd0 con 0x562293659010 2024-01-26T23:45:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.912+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/1849989293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56229364c190 con 0x562293659010 2024-01-26T23:45:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.912+0000 7faac17fa700 1 -- 172.21.15.46:0/1849989293 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7faad4022630 con 0x562293659010 2024-01-26T23:45:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.912+0000 7faaf2bc6b80 10 client.6504 fetch_fsmap learned FSMap version 36 2024-01-26T23:45:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.912+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/1849989293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x562293575290 con 0x562293659010 2024-01-26T23:45:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.912+0000 7faac17fa700 1 -- 172.21.15.46:0/1849989293 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56229364c190 con 0x562293659010 2024-01-26T23:45:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.912+0000 7faac17fa700 1 -- 172.21.15.46:0/1849989293 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) v1 ==== 3100+0+0 (secure 0 0 0) 0x7faad4020040 con 0x562293659010 2024-01-26T23:45:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.913+0000 7faaf2bc6b80 10 client.6504 fetch_fsmap finished waiting for FSMap version 36 2024-01-26T23:45:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.913+0000 7faaf2bc6b80 10 client.6504 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-26T23:45:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.913+0000 7faaf2bc6b80 20 client.6504 populate_metadata read hostname 'smithi046' 2024-01-26T23:45:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.913+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/1849989293 >> [v2:172.21.15.46:6832/1799649681,v1:172.21.15.46:6834/1799649681] conn(0x562293685a80 0x562293684d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.913+0000 7faaf2bc6b80 4 client.6504 mds_command: new command op to 6366 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-26T23:45:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.913+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/1849989293 --> [v2:172.21.15.46:6832/1799649681,v1:172.21.15.46:6834/1799649681] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562293549ee0 con 0x562293685a80 2024-01-26T23:45:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.913+0000 7faae2ffd700 1 --2- 172.21.15.46:0/1849989293 >> [v2:172.21.15.46:6832/1799649681,v1:172.21.15.46:6834/1799649681] conn(0x562293685a80 0x562293684d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.913+0000 7faae2ffd700 1 --2- 172.21.15.46:0/1849989293 >> [v2:172.21.15.46:6832/1799649681,v1:172.21.15.46:6834/1799649681] conn(0x562293685a80 0x562293684d10 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c5be70d39cd1125 server_cookie=960749bf98dab54a in_seq=0 out_seq=0 2024-01-26T23:45:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.913+0000 7faac17fa700 10 client.6504 ms_handle_connect on v2:172.21.15.46:6832/1799649681 2024-01-26T23:45:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.914+0000 7faac17fa700 1 -- 172.21.15.46:0/1849989293 <== mds.0 v2:172.21.15.46:6832/1799649681 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x562293549ee0 con 0x562293685a80 2024-01-26T23:45:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.914+0000 7faac17fa700 10 client.6504 handle_command_reply: tid=0 2024-01-26T23:45:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.914+0000 7faaf2bc6b80 1 client.6504 shutdown 2024-01-26T23:45:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.914+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/1849989293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562293659010 msgr2=0x562293683610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.914+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/1849989293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562293659010 0x562293683610 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7faad4007350 tx=0x7faad4018000 comp rx=0 tx=0).stop 2024-01-26T23:45:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.914+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/1849989293 shutdown_connections 2024-01-26T23:45:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.914+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/1849989293 >> [v2:172.21.15.46:6832/1799649681,v1:172.21.15.46:6834/1799649681] conn(0x562293685a80 0x562293684d10 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.914+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/1849989293 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622936a1360 0x562293683b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.914+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/1849989293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622936585d0 0x5622936add00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.914+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/1849989293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562293659010 0x562293683610 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.914+0000 7faac17fa700 0 client.6504 ms_handle_reset on v2:172.21.15.46:6832/1799649681 2024-01-26T23:45:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.914+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/1849989293 >> 172.21.15.46:0/1849989293 conn(0x562293658a70 msgr2=0x562293652240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.914+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/1849989293 shutdown_connections 2024-01-26T23:45:25.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.914+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/1849989293 wait complete. 2024-01-26T23:45:25.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.914+0000 7faaf2bc6b80 20 client.6504 trim_cache size 0 max 16384 2024-01-26T23:45:25.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.933+0000 7faaf2bc6b80 1 Processor -- start 2024-01-26T23:45:25.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.933+0000 7faaf2bc6b80 1 -- start start 2024-01-26T23:45:25.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.933+0000 7faaf2bc6b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622936585d0 0x562293682940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.933+0000 7faaf2bc6b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562293659010 0x562293684d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.933+0000 7faaf2bc6b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622936a1360 0x562293685250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.933+0000 7faaf2bc6b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x56229349fe40 con 0x562293659010 2024-01-26T23:45:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.933+0000 7faaf2bc6b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x562293650010 con 0x5622936585d0 2024-01-26T23:45:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.933+0000 7faaf2bc6b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x562293687420 con 0x5622936a1360 2024-01-26T23:45:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.933+0000 7faae27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562293659010 0x562293684d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.933+0000 7faae27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562293659010 0x562293684d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37284/0 (socket says 172.21.15.46:37284) 2024-01-26T23:45:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.933+0000 7faae27fc700 1 -- 172.21.15.46:0/608631407 learned_addr learned my addr 172.21.15.46:0/608631407 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.933+0000 7faae2ffd700 1 --2- 172.21.15.46:0/608631407 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622936585d0 0x562293682940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.933+0000 7faae37fe700 1 --2- 172.21.15.46:0/608631407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622936a1360 0x562293685250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.934+0000 7faae27fc700 1 -- 172.21.15.46:0/608631407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622936a1360 msgr2=0x562293685250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.934+0000 7faae27fc700 1 --2- 172.21.15.46:0/608631407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622936a1360 0x562293685250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.934+0000 7faae27fc700 1 -- 172.21.15.46:0/608631407 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622936585d0 msgr2=0x562293682940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.934+0000 7faae27fc700 1 --2- 172.21.15.46:0/608631407 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622936585d0 0x562293682940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.934+0000 7faae27fc700 1 -- 172.21.15.46:0/608631407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56229343ec80 con 0x562293659010 2024-01-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.934+0000 7faae27fc700 1 --2- 172.21.15.46:0/608631407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562293659010 0x562293684d10 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7faad4007dd0 tx=0x7faad402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b8d56db0a8cc09 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.934+0000 7faad8ff9700 1 -- 172.21.15.46:0/608631407 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad4021d70 con 0x562293659010 2024-01-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.934+0000 7faad8ff9700 1 -- 172.21.15.46:0/608631407 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad4015850 con 0x562293659010 2024-01-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.934+0000 7faad8ff9700 1 -- 172.21.15.46:0/608631407 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad4022630 con 0x562293659010 2024-01-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.935+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/608631407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562293659010 msgr2=0x562293684d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.935+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/608631407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562293659010 0x562293684d10 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7faad4007dd0 tx=0x7faad402c000 comp rx=0 tx=0).stop 2024-01-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.935+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/608631407 shutdown_connections 2024-01-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.935+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/608631407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622936a1360 0x562293685250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.935+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/608631407 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5622936585d0 0x562293682940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.935+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/608631407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562293659010 0x562293684d10 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.935+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/608631407 >> 172.21.15.46:0/608631407 conn(0x562293658a70 msgr2=0x562293654780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.935+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/608631407 shutdown_connections 2024-01-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.935+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/608631407 wait complete. 2024-01-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.935+0000 7faaf2bc6b80 1 -- start start 2024-01-26T23:45:25.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.935+0000 7faaf2bc6b80 1 Processor -- start 2024-01-26T23:45:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.936+0000 7faaf2bc6b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622936585d0 0x562293701240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.936+0000 7faaf2bc6b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562293659010 0x562293701780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.936+0000 7faaf2bc6b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622936a1360 0x56229369d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.936+0000 7faaf2bc6b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x562293701d00 con 0x5622936585d0 2024-01-26T23:45:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.936+0000 7faaf2bc6b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x56229369a290 con 0x562293659010 2024-01-26T23:45:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.936+0000 7faaf2bc6b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x562293653ea0 con 0x5622936a1360 2024-01-26T23:45:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.936+0000 7faaf2bc6b80 10 client.0 _ll_register_callbacks cb 0x5622933b57b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:45:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.936+0000 7faae2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622936585d0 0x562293701240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.936+0000 7faae2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622936585d0 0x562293701240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37286/0 (socket says 172.21.15.46:37286) 2024-01-26T23:45:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.936+0000 7faae2ffd700 1 -- 172.21.15.46:0/524280472 learned_addr learned my addr 172.21.15.46:0/524280472 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.936+0000 7faae37fe700 1 --2- 172.21.15.46:0/524280472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622936a1360 0x56229369d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.937+0000 7faae27fc700 1 --2- 172.21.15.46:0/524280472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562293659010 0x562293701780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.937+0000 7faae2ffd700 1 -- 172.21.15.46:0/524280472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622936a1360 msgr2=0x56229369d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.937+0000 7faae2ffd700 1 --2- 172.21.15.46:0/524280472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622936a1360 0x56229369d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.937+0000 7faae2ffd700 1 -- 172.21.15.46:0/524280472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562293659010 msgr2=0x562293701780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.937+0000 7faae2ffd700 1 --2- 172.21.15.46:0/524280472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562293659010 0x562293701780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.937+0000 7faae2ffd700 1 -- 172.21.15.46:0/524280472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562293575290 con 0x5622936585d0 2024-01-26T23:45:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.937+0000 7faae2ffd700 1 --2- 172.21.15.46:0/524280472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622936585d0 0x562293701240 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7faacc002f40 tx=0x7faacc054fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6506aad59d5a5216 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.937+0000 7faac17fa700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:45:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.937+0000 7faac17fa700 1 -- 172.21.15.46:0/524280472 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faacc082850 con 0x5622936585d0 2024-01-26T23:45:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.937+0000 7faac17fa700 1 -- 172.21.15.46:0/524280472 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 6417+0+0 (secure 0 0 0) 0x7faacc0840b0 con 0x5622936585d0 2024-01-26T23:45:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.937+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/524280472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faae4157e90 con 0x5622936585d0 2024-01-26T23:45:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.937+0000 7faac17fa700 1 -- 172.21.15.46:0/524280472 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7faacc055330 con 0x5622936585d0 2024-01-26T23:45:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.937+0000 7faaf2bc6b80 10 client.6510 fetch_fsmap learned FSMap version 36 2024-01-26T23:45:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.937+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/524280472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x5622933e4fe0 con 0x5622936585d0 2024-01-26T23:45:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.937+0000 7faac17fa700 1 -- 172.21.15.46:0/524280472 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae4157e90 con 0x5622936585d0 2024-01-26T23:45:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.938+0000 7faac17fa700 1 -- 172.21.15.46:0/524280472 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) v1 ==== 3100+0+0 (secure 0 0 0) 0x7faacc0545c0 con 0x5622936585d0 2024-01-26T23:45:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.938+0000 7faaf2bc6b80 10 client.6510 fetch_fsmap finished waiting for FSMap version 36 2024-01-26T23:45:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.938+0000 7faaf2bc6b80 10 client.6510 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-26T23:45:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.938+0000 7faaf2bc6b80 20 client.6510 populate_metadata read hostname 'smithi046' 2024-01-26T23:45:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.938+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/524280472 >> [v2:172.21.15.46:6832/1799649681,v1:172.21.15.46:6834/1799649681] conn(0x562293705240 0x562293705610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.938+0000 7faaf2bc6b80 4 client.6510 mds_command: new command op to 6366 tid=0[{"prefix": "status"}] 2024-01-26T23:45:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.938+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/524280472 --> [v2:172.21.15.46:6832/1799649681,v1:172.21.15.46:6834/1799649681] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7faad4015ae0 con 0x562293705240 2024-01-26T23:45:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.938+0000 7faae27fc700 1 --2- 172.21.15.46:0/524280472 >> [v2:172.21.15.46:6832/1799649681,v1:172.21.15.46:6834/1799649681] conn(0x562293705240 0x562293705610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.938+0000 7faac17fa700 10 client.6510 ms_handle_connect on v2:172.21.15.46:6832/1799649681 2024-01-26T23:45:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.938+0000 7faae27fc700 1 --2- 172.21.15.46:0/524280472 >> [v2:172.21.15.46:6832/1799649681,v1:172.21.15.46:6834/1799649681] conn(0x562293705240 0x562293705610 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bf745515df48576 server_cookie=5d7e4a9ea6b25576 in_seq=0 out_seq=0 2024-01-26T23:45:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.938+0000 7faac17fa700 1 -- 172.21.15.46:0/524280472 <== mds.0 v2:172.21.15.46:6832/1799649681 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x7faad4015ae0 con 0x562293705240 2024-01-26T23:45:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.938+0000 7faac17fa700 10 client.6510 handle_command_reply: tid=0 2024-01-26T23:45:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.938+0000 7faaf2bc6b80 1 client.6510 shutdown 2024-01-26T23:45:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.938+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/524280472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622936585d0 msgr2=0x562293701240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.938+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/524280472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622936585d0 0x562293701240 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7faacc002f40 tx=0x7faacc054fd0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/524280472 shutdown_connections 2024-01-26T23:45:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/524280472 >> [v2:172.21.15.46:6832/1799649681,v1:172.21.15.46:6834/1799649681] conn(0x562293705240 0x562293705610 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/524280472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5622936585d0 0x562293701240 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/524280472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5622936a1360 0x56229369d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faac17fa700 0 client.6510 ms_handle_reset on v2:172.21.15.46:6832/1799649681 2024-01-26T23:45:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faaf2bc6b80 1 --2- 172.21.15.46:0/524280472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562293659010 0x562293701780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/524280472 >> 172.21.15.46:0/524280472 conn(0x562293658a70 msgr2=0x562293684d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/524280472 shutdown_connections 2024-01-26T23:45:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faaf2bc6b80 1 -- 172.21.15.46:0/524280472 wait complete. 2024-01-26T23:45:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faaf2bc6b80 20 client.6510 trim_cache size 0 max 16384 2024-01-26T23:45:25.947 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:45:25.947 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-26T23:45:25.947 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-26T23:45:25.947 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 6366, 2024-01-26T23:45:25.947 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-26T23:45:25.947 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-26T23:45:25.947 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-01-26T23:45:25.947 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.1713866399999997, 2024-01-26T23:45:25.947 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 36, 2024-01-26T23:45:25.947 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 85, 2024-01-26T23:45:25.947 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 85, 2024-01-26T23:45:25.948 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 11.159523751 2024-01-26T23:45:25.948 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:45:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faae98c3700 1 -- 172.21.15.46:0/4176165163 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faac4066f60 msgr2=0x7faac4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faae98c3700 1 --2- 172.21.15.46:0/4176165163 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faac4066f60 0x7faac4069410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faae4159c40 tx=0x7faad4001e50 comp rx=0 tx=0).stop 2024-01-26T23:45:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faae98c3700 1 -- 172.21.15.46:0/4176165163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faad00051c0 msgr2=0x7faae416e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faae98c3700 1 --2- 172.21.15.46:0/4176165163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faad00051c0 0x7faae416e0f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7faacc002950 tx=0x7faacc007d30 comp rx=0 tx=0).stop 2024-01-26T23:45:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faae98c3700 1 -- 172.21.15.46:0/4176165163 shutdown_connections 2024-01-26T23:45:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faae98c3700 1 --2- 172.21.15.46:0/4176165163 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faac4066f60 0x7faac4069410 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faae98c3700 1 --2- 172.21.15.46:0/4176165163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faad00051c0 0x7faae416e0f0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faae98c3700 1 --2- 172.21.15.46:0/4176165163 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faae4153a80 0x7faae416a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faae98c3700 1 --2- 172.21.15.46:0/4176165163 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faae4159850 0x7faae40c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.939+0000 7faae98c3700 1 -- 172.21.15.46:0/4176165163 >> 172.21.15.46:0/4176165163 conn(0x7faae4068400 msgr2=0x7faae40afde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.940+0000 7faae98c3700 1 -- 172.21.15.46:0/4176165163 shutdown_connections 2024-01-26T23:45:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:25.940+0000 7faae98c3700 1 -- 172.21.15.46:0/4176165163 wait complete. 2024-01-26T23:45:25.954 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-26T23:45:26.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.085+0000 7f830d375700 1 Processor -- start 2024-01-26T23:45:26.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.085+0000 7f830d375700 1 -- start start 2024-01-26T23:45:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.085+0000 7f830d375700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f83080cb110 0x7f830806a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.085+0000 7f830d375700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8308159cf0 0x7f830806aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.085+0000 7f830d375700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830806af60 con 0x7f8308159920 2024-01-26T23:45:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.085+0000 7f830d375700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f830806b1d0 con 0x7f83080cb110 2024-01-26T23:45:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.085+0000 7f830d375700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f830806b4b0 con 0x7f8308159cf0 2024-01-26T23:45:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.086+0000 7f8306ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8308159920 0x7f83080caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43372/0 (socket says 172.21.15.46:43372) 2024-01-26T23:45:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.086+0000 7f8306ffd700 1 -- 172.21.15.46:0/4284410398 learned_addr learned my addr 172.21.15.46:0/4284410398 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.086+0000 7f83077fe700 1 --2- 172.21.15.46:0/4284410398 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8308159cf0 0x7f830806aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.086+0000 7f83067fc700 1 --2- 172.21.15.46:0/4284410398 >> v2:172.21.15.106:3300/0 conn(0x7f83080cb110 0x7f830806a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.086+0000 7f8305ffb700 1 -- 172.21.15.46:0/4284410398 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705909331 0 0) 0x7f830806af60 con 0x7f8308159920 2024-01-26T23:45:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.086+0000 7f8305ffb700 1 -- 172.21.15.46:0/4284410398 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82f0003570 con 0x7f8308159920 2024-01-26T23:45:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.086+0000 7f83067fc700 1 -- 172.21.15.46:0/4284410398 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8308159cf0 msgr2=0x7f830806aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.086+0000 7f83067fc700 1 --2- 172.21.15.46:0/4284410398 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8308159cf0 0x7f830806aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.086+0000 7f83067fc700 1 -- 172.21.15.46:0/4284410398 >> v1:172.21.15.46:6789/0 conn(0x7f8308159920 legacy=0x7f83080caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.086+0000 7f83067fc700 1 -- 172.21.15.46:0/4284410398 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830806b790 con 0x7f83080cb110 2024-01-26T23:45:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.086+0000 7f83077fe700 1 --2- 172.21.15.46:0/4284410398 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8308159cf0 0x7f830806aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.086+0000 7f83067fc700 1 --2- 172.21.15.46:0/4284410398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83080cb110 0x7f830806a4e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f82f80098e0 tx=0x7f82f80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3badc44f2bdb015 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.087+0000 7f8305ffb700 1 -- 172.21.15.46:0/4284410398 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8011630 con 0x7f83080cb110 2024-01-26T23:45:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.087+0000 7f8305ffb700 1 -- 172.21.15.46:0/4284410398 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f8011790 con 0x7f83080cb110 2024-01-26T23:45:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.087+0000 7f8305ffb700 1 -- 172.21.15.46:0/4284410398 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8011b30 con 0x7f83080cb110 2024-01-26T23:45:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.087+0000 7f830d375700 1 -- 172.21.15.46:0/4284410398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83080cb110 msgr2=0x7f830806a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.087+0000 7f830d375700 1 --2- 172.21.15.46:0/4284410398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83080cb110 0x7f830806a4e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f82f80098e0 tx=0x7f82f80091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.087+0000 7f830d375700 1 -- 172.21.15.46:0/4284410398 shutdown_connections 2024-01-26T23:45:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.087+0000 7f830d375700 1 --2- 172.21.15.46:0/4284410398 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8308159cf0 0x7f830806aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.087+0000 7f830d375700 1 --2- 172.21.15.46:0/4284410398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83080cb110 0x7f830806a4e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.087+0000 7f830d375700 1 -- 172.21.15.46:0/4284410398 >> 172.21.15.46:0/4284410398 conn(0x7f83080cd6f0 msgr2=0x7f83080c9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.087+0000 7f830d375700 1 -- 172.21.15.46:0/4284410398 shutdown_connections 2024-01-26T23:45:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.087+0000 7f830d375700 1 -- 172.21.15.46:0/4284410398 wait complete. 2024-01-26T23:45:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.089+0000 7f830d375700 1 Processor -- start 2024-01-26T23:45:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.089+0000 7f830d375700 1 -- start start 2024-01-26T23:45:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.089+0000 7f830d375700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83080cb110 0x7f8308155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.089+0000 7f830d375700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8308159920 0x7f83080c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.089+0000 7f830d375700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8308159cf0 0x7f83080c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.089+0000 7f830d375700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830815b6f0 con 0x7f83080cb110 2024-01-26T23:45:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.089+0000 7f830d375700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83080ad620 con 0x7f8308159cf0 2024-01-26T23:45:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.089+0000 7f830d375700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83080ad900 con 0x7f8308159920 2024-01-26T23:45:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.090+0000 7f8306ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83080cb110 0x7f8308155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.090+0000 7f8306ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83080cb110 0x7f8308155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37302/0 (socket says 172.21.15.46:37302) 2024-01-26T23:45:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.090+0000 7f8306ffd700 1 -- 172.21.15.46:0/2473252548 learned_addr learned my addr 172.21.15.46:0/2473252548 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.090+0000 7f83067fc700 1 --2- 172.21.15.46:0/2473252548 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8308159920 0x7f83080c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.090+0000 7f83077fe700 1 --2- 172.21.15.46:0/2473252548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8308159cf0 0x7f83080c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.090+0000 7f8306ffd700 1 -- 172.21.15.46:0/2473252548 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8308159920 msgr2=0x7f83080c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.090+0000 7f8306ffd700 1 --2- 172.21.15.46:0/2473252548 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8308159920 0x7f83080c6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.090+0000 7f8306ffd700 1 -- 172.21.15.46:0/2473252548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8308159cf0 msgr2=0x7f83080c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.090+0000 7f8306ffd700 1 --2- 172.21.15.46:0/2473252548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8308159cf0 0x7f83080c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.090+0000 7f8306ffd700 1 -- 172.21.15.46:0/2473252548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8300009a20 con 0x7f83080cb110 2024-01-26T23:45:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.090+0000 7f83077fe700 1 --2- 172.21.15.46:0/2473252548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8308159cf0 0x7f83080c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.090+0000 7f8306ffd700 1 --2- 172.21.15.46:0/2473252548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83080cb110 0x7f8308155ca0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f82fc00c7c0 tx=0x7f82fc007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.090+0000 7f82eeffd700 1 -- 172.21.15.46:0/2473252548 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc005b30 con 0x7f83080cb110 2024-01-26T23:45:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.090+0000 7f82eeffd700 1 -- 172.21.15.46:0/2473252548 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82fc005cb0 con 0x7f83080cb110 2024-01-26T23:45:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.090+0000 7f830d375700 1 -- 172.21.15.46:0/2473252548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f8009420 con 0x7f83080cb110 2024-01-26T23:45:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.090+0000 7f82eeffd700 1 -- 172.21.15.46:0/2473252548 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc007320 con 0x7f83080cb110 2024-01-26T23:45:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.090+0000 7f830d375700 1 -- 172.21.15.46:0/2473252548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83080adec0 con 0x7f83080cb110 2024-01-26T23:45:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.091+0000 7f830d375700 1 -- 172.21.15.46:0/2473252548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830804c710 con 0x7f83080cb110 2024-01-26T23:45:26.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.092+0000 7f82eeffd700 1 -- 172.21.15.46:0/2473252548 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f82fc01b3b0 con 0x7f83080cb110 2024-01-26T23:45:26.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.092+0000 7f82eeffd700 1 --2- 172.21.15.46:0/2473252548 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f82f4066f30 0x7f82f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:26.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.092+0000 7f82eeffd700 1 -- 172.21.15.46:0/2473252548 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f82fc083cf0 con 0x7f83080cb110 2024-01-26T23:45:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.093+0000 7f83067fc700 1 --2- 172.21.15.46:0/2473252548 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f82f4066f30 0x7f82f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.093+0000 7f83067fc700 1 --2- 172.21.15.46:0/2473252548 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f82f4066f30 0x7f82f40693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f83080aea70 tx=0x7f82f8017000 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:26.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.098+0000 7f82eeffd700 1 -- 172.21.15.46:0/2473252548 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f82fc088070 con 0x7f83080cb110 2024-01-26T23:45:26.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.352+0000 7f830d375700 1 -- 172.21.15.46:0/2473252548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f830806ba70 con 0x7f83080cb110 2024-01-26T23:45:26.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.353+0000 7f82eeffd700 1 -- 172.21.15.46:0/2473252548 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v36) v1 ==== 53+0+73 (secure 0 0 0) 0x7f82fc017020 con 0x7f83080cb110 2024-01-26T23:45:26.353 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-26T23:45:26.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.355+0000 7f830d375700 1 -- 172.21.15.46:0/2473252548 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f82f4066f30 msgr2=0x7f82f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:26.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.355+0000 7f830d375700 1 --2- 172.21.15.46:0/2473252548 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f82f4066f30 0x7f82f40693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f83080aea70 tx=0x7f82f8017000 comp rx=0 tx=0).stop 2024-01-26T23:45:26.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.355+0000 7f830d375700 1 -- 172.21.15.46:0/2473252548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83080cb110 msgr2=0x7f8308155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:26.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.355+0000 7f830d375700 1 --2- 172.21.15.46:0/2473252548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83080cb110 0x7f8308155ca0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f82fc00c7c0 tx=0x7f82fc007e00 comp rx=0 tx=0).stop 2024-01-26T23:45:26.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.355+0000 7f830d375700 1 -- 172.21.15.46:0/2473252548 shutdown_connections 2024-01-26T23:45:26.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.355+0000 7f830d375700 1 --2- 172.21.15.46:0/2473252548 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f82f4066f30 0x7f82f40693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:26.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.355+0000 7f830d375700 1 --2- 172.21.15.46:0/2473252548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83080cb110 0x7f8308155ca0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:26.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.355+0000 7f830d375700 1 --2- 172.21.15.46:0/2473252548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8308159cf0 0x7f83080c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:26.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.355+0000 7f830d375700 1 --2- 172.21.15.46:0/2473252548 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8308159920 0x7f83080c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:26.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.355+0000 7f830d375700 1 -- 172.21.15.46:0/2473252548 >> 172.21.15.46:0/2473252548 conn(0x7f83080cd6f0 msgr2=0x7f83080c21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:26.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.355+0000 7f830d375700 1 -- 172.21.15.46:0/2473252548 shutdown_connections 2024-01-26T23:45:26.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:26.355+0000 7f830d375700 1 -- 172.21.15.46:0/2473252548 wait complete. 2024-01-26T23:45:26.369 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-26T23:45:26.369 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-26T23:45:26.369 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-26T23:45:26.369 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-26T23:45:26.370 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-26T23:45:26.370 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-26T23:45:26.370 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-26T23:45:26.370 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-26T23:45:26.370 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:45:26.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:45:26.370 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-26T23:45:26.390 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-26T23:45:26.390 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-26T23:45:26.390 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-26T23:45:26.390 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:45:26.390 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-26T23:45:26.391 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:45:26.391 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:45:26.391 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:45:26.391 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-26T23:45:26.391 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:45:26.391 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:45:26.391 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-26T23:45:26.391 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-26T23:45:26.391 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:45:26.391 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-26T23:45:26.391 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-26T23:45:26.391 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 41404sec preferred_lft 41404sec 2024-01-26T23:45:26.391 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-26T23:45:26.391 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:45:26.391 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:45:26.392 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:45:26.392 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-26T23:45:26.392 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-26T23:45:26.392 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:45:26.392 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-26T23:45:26.392 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:45:26.392 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::cce6:a7ff:fe4f:38d/64 scope link 2024-01-26T23:45:26.392 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:45:26.392 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:45:26.392 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 76:ef:90:4c:9d:6f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-26T23:45:26.392 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::74ef:90ff:fe4c:9d6f/64 scope link 2024-01-26T23:45:26.392 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:45:26.392 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:45:26.392 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-26T23:45:26.393 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::2c73:afff:fef0:6f39/64 scope link 2024-01-26T23:45:26.393 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:45:26.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:45:26.393 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-26T23:45:26.460 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-26T23:45:26.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:45:26.462 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-26T23:45:26.527 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-26T23:45:26.527 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-26T23:45:26.527 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-26T23:45:26.527 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-26T23:45:26.527 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-26T23:45:26.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:45:26.528 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:45:26.528 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:45:26.528 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-26T23:45:26.528 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-26T23:45:26.528 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-26T23:45:26.528 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:45:26.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:45:26.654 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-26T23:45:26.675 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-26T23:45:26.675 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-26T23:45:26.675 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:45:26.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:45:26.675 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-01-26T23:45:26.750 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-26T23:45:26.750 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:45:26.751 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-26T23:45:26.813 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:45:26.814 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:45:26.814 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:45:26.876 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:45:26.952 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:45:26.952 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:45:27.026 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:45:27.028 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:45:27.028 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:45:27.028 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:45:27.090 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-26T23:45:27.090 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-26T23:45:27.132 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:45:27.168 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:45:27.168 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:45:27.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-01-26T23:45:27.235+0000 7efde0b3b540 -1 init, newargv = 0x55ac0706a250 newargc=15 2024-01-26T23:45:27.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[72965]: starting ceph client 2024-01-26T23:45:27.239 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:45:27.241 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:45:27.241 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:45:27.242 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:45:27.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[72965]: starting fuse 2024-01-26T23:45:27.301 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-26T23:45:27.301 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-26T23:45:27.302 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-26T23:45:27.302 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-26T23:45:27.302 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-26T23:45:27.302 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-26T23:45:27.302 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-26T23:45:27.302 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:45:27.302 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-26T23:45:27.302 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-26T23:45:27.302 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-26T23:45:27.302 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-26T23:45:27.302 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:45:27.302 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-26T23:45:27.302 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-26T23:45:27.302 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-26T23:45:27.302 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-26T23:45:27.303 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:45:27.303 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-26T23:45:27.303 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-26T23:45:27.303 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-26T23:45:27.303 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-26T23:45:27.303 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-26T23:45:27.303 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-26T23:45:27.303 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-26T23:45:27.303 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-26T23:45:27.303 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:45:27.303 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-01-26T23:45:27.303 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-26T23:45:27.441 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.72965.asok 2024-01-26T23:45:27.622 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.72965.asok 2024-01-26T23:45:27.622 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:45:27.622 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.72965.asok status 2024-01-26T23:45:27.798 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:45:27.798 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-26T23:45:27.798 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-26T23:45:27.799 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-26T23:45:27.799 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-01-26T23:45:27.799 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-26T23:45:27.799 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-26T23:45:27.799 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "72965", 2024-01-26T23:45:27.799 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-26T23:45:27.799 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:27.799 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-26T23:45:27.799 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-26T23:45:27.799 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 6522, 2024-01-26T23:45:27.799 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-26T23:45:27.799 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-26T23:45:27.799 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-26T23:45:27.800 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6522 2024-01-26T23:45:27.800 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:27.800 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:45:27.800 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:45:27.800 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:45:27.800 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1606972779 2024-01-26T23:45:27.800 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:27.800 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:27.800 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:45:27.800 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:45:27.800 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:45:27.800 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1606972779 2024-01-26T23:45:27.800 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:27.801 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.6522 192.168.0.1:0/1606972779", 2024-01-26T23:45:27.801 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/1606972779", 2024-01-26T23:45:27.801 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-26T23:45:27.801 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 36, 2024-01-26T23:45:27.801 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 85, 2024-01-26T23:45:27.801 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-26T23:45:27.801 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-26T23:45:27.801 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-26T23:45:27.801 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:45:27.810 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:45:27.810 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-26T23:45:27.810 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-26T23:45:27.829 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-26T23:45:27.829 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-26T23:45:27.830 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:45:27.830 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:45:27.903 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-26T23:45:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.084+0000 7f26ee3b8700 1 Processor -- start 2024-01-26T23:45:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.084+0000 7f26ee3b8700 1 -- start start 2024-01-26T23:45:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.085+0000 7f26ee3b8700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f26e815d210 0x7f26e80cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.085+0000 7f26ee3b8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f26e80cbff0 0x7f26e80ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.085+0000 7f26ee3b8700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e80cc450 con 0x7f26e815ce40 2024-01-26T23:45:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.085+0000 7f26ee3b8700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e80cff70 con 0x7f26e80cbff0 2024-01-26T23:45:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.085+0000 7f26ee3b8700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e80cf400 con 0x7f26e815d210 2024-01-26T23:45:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.085+0000 7f26ec955700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f26e80cbff0 0x7f26e80ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.085+0000 7f26ec955700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f26e80cbff0 0x7f26e80ade30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37314/0 (socket says 172.21.15.46:37314) 2024-01-26T23:45:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.085+0000 7f26ec955700 1 -- 172.21.15.46:0/1538440462 learned_addr learned my addr 172.21.15.46:0/1538440462 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.085+0000 7f26e77fe700 1 --2- 172.21.15.46:0/1538440462 >> v2:172.21.15.106:3300/0 conn(0x7f26e815d210 0x7f26e80cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.085+0000 7f26ec955700 1 -- 172.21.15.46:0/1538440462 >> v2:172.21.15.106:3300/0 conn(0x7f26e815d210 msgr2=0x7f26e80cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.085+0000 7f26ec955700 1 --2- 172.21.15.46:0/1538440462 >> v2:172.21.15.106:3300/0 conn(0x7f26e815d210 0x7f26e80cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.085+0000 7f26ec955700 1 -- 172.21.15.46:0/1538440462 >> v1:172.21.15.106:6789/0 conn(0x7f26e815ce40 legacy=0x7f26e80d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.085+0000 7f26ec955700 1 -- 172.21.15.46:0/1538440462 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e80cf660 con 0x7f26e80cbff0 2024-01-26T23:45:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.086+0000 7f26e77fe700 1 --2- 172.21.15.46:0/1538440462 >> v2:172.21.15.106:3300/0 conn(0x7f26e815d210 0x7f26e80cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.086+0000 7f26ec955700 1 --2- 172.21.15.46:0/1538440462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26e80cbff0 0x7f26e80ade30 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f26e80895f0 tx=0x7f26e0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19935280c7ce382d server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.086+0000 7f26e6ffd700 1 -- 172.21.15.46:0/1538440462 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e000e070 con 0x7f26e80cbff0 2024-01-26T23:45:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.086+0000 7f26e6ffd700 1 -- 172.21.15.46:0/1538440462 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e000ba50 con 0x7f26e80cbff0 2024-01-26T23:45:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.086+0000 7f26e6ffd700 1 -- 172.21.15.46:0/1538440462 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e00092e0 con 0x7f26e80cbff0 2024-01-26T23:45:28.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.086+0000 7f26ee3b8700 1 -- 172.21.15.46:0/1538440462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26e80cbff0 msgr2=0x7f26e80ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:28.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.086+0000 7f26ee3b8700 1 --2- 172.21.15.46:0/1538440462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26e80cbff0 0x7f26e80ade30 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f26e80895f0 tx=0x7f26e0009840 comp rx=0 tx=0).stop 2024-01-26T23:45:28.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.086+0000 7f26ee3b8700 1 -- 172.21.15.46:0/1538440462 shutdown_connections 2024-01-26T23:45:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.086+0000 7f26ee3b8700 1 --2- 172.21.15.46:0/1538440462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26e80cbff0 0x7f26e80ade30 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.086+0000 7f26ee3b8700 1 --2- 172.21.15.46:0/1538440462 >> v2:172.21.15.106:3300/0 conn(0x7f26e815d210 0x7f26e80cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.086+0000 7f26ee3b8700 1 -- 172.21.15.46:0/1538440462 >> 172.21.15.46:0/1538440462 conn(0x7f26e806f840 msgr2=0x7f26e80d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.087+0000 7f26ee3b8700 1 -- 172.21.15.46:0/1538440462 shutdown_connections 2024-01-26T23:45:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.087+0000 7f26ee3b8700 1 -- 172.21.15.46:0/1538440462 wait complete. 2024-01-26T23:45:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.088+0000 7f26ee3b8700 1 Processor -- start 2024-01-26T23:45:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.088+0000 7f26ee3b8700 1 -- start start 2024-01-26T23:45:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.089+0000 7f26ee3b8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26e80cbff0 0x7f26e809dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.089+0000 7f26ee3b8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26e815ce40 0x7f26e8156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.089+0000 7f26ee3b8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26e815d210 0x7f26e815f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.089+0000 7f26ee3b8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8157990 con 0x7f26e80cbff0 2024-01-26T23:45:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.089+0000 7f26ee3b8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8156d80 con 0x7f26e815d210 2024-01-26T23:45:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.089+0000 7f26ee3b8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8157060 con 0x7f26e815ce40 2024-01-26T23:45:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.089+0000 7f26e7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26e80cbff0 0x7f26e809dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.089+0000 7f26e7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26e80cbff0 0x7f26e809dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37320/0 (socket says 172.21.15.46:37320) 2024-01-26T23:45:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.089+0000 7f26e7fff700 1 -- 172.21.15.46:0/979628634 learned_addr learned my addr 172.21.15.46:0/979628634 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.089+0000 7f26ec955700 1 --2- 172.21.15.46:0/979628634 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26e815d210 0x7f26e815f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.089+0000 7f26e77fe700 1 --2- 172.21.15.46:0/979628634 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26e815ce40 0x7f26e8156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.089+0000 7f26e7fff700 1 -- 172.21.15.46:0/979628634 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26e815ce40 msgr2=0x7f26e8156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.089+0000 7f26e7fff700 1 --2- 172.21.15.46:0/979628634 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26e815ce40 0x7f26e8156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.089+0000 7f26e7fff700 1 -- 172.21.15.46:0/979628634 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26e815d210 msgr2=0x7f26e815f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.089+0000 7f26e7fff700 1 --2- 172.21.15.46:0/979628634 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26e815d210 0x7f26e815f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.089+0000 7f26e7fff700 1 -- 172.21.15.46:0/979628634 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e0009ae0 con 0x7f26e80cbff0 2024-01-26T23:45:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.089+0000 7f26e7fff700 1 --2- 172.21.15.46:0/979628634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26e80cbff0 0x7f26e809dab0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f26d800c7c0 tx=0x7f26d8007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.090+0000 7f26cffff700 1 -- 172.21.15.46:0/979628634 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8011590 con 0x7f26e80cbff0 2024-01-26T23:45:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.090+0000 7f26cffff700 1 -- 172.21.15.46:0/979628634 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d80116f0 con 0x7f26e80cbff0 2024-01-26T23:45:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.090+0000 7f26ee3b8700 1 -- 172.21.15.46:0/979628634 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e8157340 con 0x7f26e80cbff0 2024-01-26T23:45:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.090+0000 7f26cffff700 1 -- 172.21.15.46:0/979628634 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8011ab0 con 0x7f26e80cbff0 2024-01-26T23:45:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.090+0000 7f26ee3b8700 1 -- 172.21.15.46:0/979628634 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e809dff0 con 0x7f26e80cbff0 2024-01-26T23:45:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.091+0000 7f26ee3b8700 1 -- 172.21.15.46:0/979628634 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e804c710 con 0x7f26e80cbff0 2024-01-26T23:45:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.091+0000 7f26cffff700 1 -- 172.21.15.46:0/979628634 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f26d801b3b0 con 0x7f26e80cbff0 2024-01-26T23:45:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.092+0000 7f26cffff700 1 --2- 172.21.15.46:0/979628634 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f26d4066f30 0x7f26d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.092+0000 7f26cffff700 1 -- 172.21.15.46:0/979628634 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f26d8083a10 con 0x7f26e80cbff0 2024-01-26T23:45:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.092+0000 7f26e77fe700 1 --2- 172.21.15.46:0/979628634 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f26d4066f30 0x7f26d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.093+0000 7f26e77fe700 1 --2- 172.21.15.46:0/979628634 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f26d4066f30 0x7f26d40693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f26e815c1a0 tx=0x7f26dc008040 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.096+0000 7f26cffff700 1 -- 172.21.15.46:0/979628634 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f26d8050a60 con 0x7f26e80cbff0 2024-01-26T23:45:28.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.336+0000 7f26ee3b8700 1 -- 172.21.15.46:0/979628634 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f26e81577e0 con 0x7f26d4066f30 2024-01-26T23:45:28.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.338+0000 7f26cffff700 1 -- 172.21.15.46:0/979628634 <== mgr.5660 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f26e81577e0 con 0x7f26d4066f30 2024-01-26T23:45:28.338 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-26T23:45:28.339 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:28.339 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephfs" 2024-01-26T23:45:28.339 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:28.339 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-26T23:45:28.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.340+0000 7f26ee3b8700 1 -- 172.21.15.46:0/979628634 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f26d4066f30 msgr2=0x7f26d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:28.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.340+0000 7f26ee3b8700 1 --2- 172.21.15.46:0/979628634 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f26d4066f30 0x7f26d40693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f26e815c1a0 tx=0x7f26dc008040 comp rx=0 tx=0).stop 2024-01-26T23:45:28.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.340+0000 7f26ee3b8700 1 -- 172.21.15.46:0/979628634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26e80cbff0 msgr2=0x7f26e809dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:28.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.340+0000 7f26ee3b8700 1 --2- 172.21.15.46:0/979628634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26e80cbff0 0x7f26e809dab0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f26d800c7c0 tx=0x7f26d8007f70 comp rx=0 tx=0).stop 2024-01-26T23:45:28.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.341+0000 7f26ee3b8700 1 -- 172.21.15.46:0/979628634 shutdown_connections 2024-01-26T23:45:28.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.341+0000 7f26ee3b8700 1 --2- 172.21.15.46:0/979628634 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f26d4066f30 0x7f26d40693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.341+0000 7f26ee3b8700 1 --2- 172.21.15.46:0/979628634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26e80cbff0 0x7f26e809dab0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.341+0000 7f26ee3b8700 1 --2- 172.21.15.46:0/979628634 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26e815d210 0x7f26e815f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.341+0000 7f26ee3b8700 1 --2- 172.21.15.46:0/979628634 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26e815ce40 0x7f26e8156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.341+0000 7f26ee3b8700 1 -- 172.21.15.46:0/979628634 >> 172.21.15.46:0/979628634 conn(0x7f26e806f840 msgr2=0x7f26e80b58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:28.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.341+0000 7f26ee3b8700 1 -- 172.21.15.46:0/979628634 shutdown_connections 2024-01-26T23:45:28.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.341+0000 7f26ee3b8700 1 -- 172.21.15.46:0/979628634 wait complete. 2024-01-26T23:45:28.354 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-26T23:45:28.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.488+0000 7f123a6ef700 1 Processor -- start 2024-01-26T23:45:28.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.488+0000 7f123a6ef700 1 -- start start 2024-01-26T23:45:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.489+0000 7f123a6ef700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1234159510 0x7f12341598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.489+0000 7f123a6ef700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12340ca220 0x7f12340c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.489+0000 7f123a6ef700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f12340ca690 0x7f12340c9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.489+0000 7f123a6ef700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1234069260 con 0x7f12340ca220 2024-01-26T23:45:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.489+0000 7f123a6ef700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f123415d670 con 0x7f12340ca690 2024-01-26T23:45:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.489+0000 7f123a6ef700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12340c9750 con 0x7f1234159510 2024-01-26T23:45:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.489+0000 7f12337fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12340ca220 0x7f12340c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.489+0000 7f12337fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12340ca220 0x7f12340c8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37336/0 (socket says 172.21.15.46:37336) 2024-01-26T23:45:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.489+0000 7f12337fe700 1 -- 172.21.15.46:0/521036057 learned_addr learned my addr 172.21.15.46:0/521036057 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.489+0000 7f1233fff700 1 --2- 172.21.15.46:0/521036057 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1234159510 0x7f12341598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.489+0000 7f1238c8c700 1 --2- 172.21.15.46:0/521036057 >> v2:172.21.15.106:3300/0 conn(0x7f12340ca690 0x7f12340c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.489+0000 7f12337fe700 1 -- 172.21.15.46:0/521036057 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1234159510 msgr2=0x7f12341598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.489+0000 7f12337fe700 1 --2- 172.21.15.46:0/521036057 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1234159510 0x7f12341598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.489+0000 7f12337fe700 1 -- 172.21.15.46:0/521036057 >> v2:172.21.15.106:3300/0 conn(0x7f12340ca690 msgr2=0x7f12340c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.489+0000 7f12337fe700 1 --2- 172.21.15.46:0/521036057 >> v2:172.21.15.106:3300/0 conn(0x7f12340ca690 0x7f12340c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.489+0000 7f12337fe700 1 -- 172.21.15.46:0/521036057 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12340c9a30 con 0x7f12340ca220 2024-01-26T23:45:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.489+0000 7f1238c8c700 1 --2- 172.21.15.46:0/521036057 >> v2:172.21.15.106:3300/0 conn(0x7f12340ca690 0x7f12340c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.490+0000 7f12337fe700 1 --2- 172.21.15.46:0/521036057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12340ca220 0x7f12340c8be0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f1234159e20 tx=0x7f121c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f03e3ef7cc3ae441 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.490+0000 7f1232ffd700 1 -- 172.21.15.46:0/521036057 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c009e70 con 0x7f12340ca220 2024-01-26T23:45:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.490+0000 7f1232ffd700 1 -- 172.21.15.46:0/521036057 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121c00b860 con 0x7f12340ca220 2024-01-26T23:45:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.490+0000 7f1232ffd700 1 -- 172.21.15.46:0/521036057 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c00bc00 con 0x7f12340ca220 2024-01-26T23:45:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.490+0000 7f123a6ef700 1 -- 172.21.15.46:0/521036057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12340ca220 msgr2=0x7f12340c8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.490+0000 7f123a6ef700 1 --2- 172.21.15.46:0/521036057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12340ca220 0x7f12340c8be0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f1234159e20 tx=0x7f121c009250 comp rx=0 tx=0).stop 2024-01-26T23:45:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.490+0000 7f123a6ef700 1 -- 172.21.15.46:0/521036057 shutdown_connections 2024-01-26T23:45:28.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.490+0000 7f123a6ef700 1 --2- 172.21.15.46:0/521036057 >> v2:172.21.15.106:3300/0 conn(0x7f12340ca690 0x7f12340c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.490+0000 7f123a6ef700 1 --2- 172.21.15.46:0/521036057 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1234159510 0x7f12341598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.490+0000 7f123a6ef700 1 --2- 172.21.15.46:0/521036057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12340ca220 0x7f12340c8be0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.490+0000 7f123a6ef700 1 -- 172.21.15.46:0/521036057 >> 172.21.15.46:0/521036057 conn(0x7f123406bd90 msgr2=0x7f123406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:28.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.491+0000 7f123a6ef700 1 -- 172.21.15.46:0/521036057 shutdown_connections 2024-01-26T23:45:28.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.491+0000 7f123a6ef700 1 -- 172.21.15.46:0/521036057 wait complete. 2024-01-26T23:45:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.492+0000 7f123a6ef700 1 Processor -- start 2024-01-26T23:45:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.492+0000 7f123a6ef700 1 -- start start 2024-01-26T23:45:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.493+0000 7f123a6ef700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12340ca220 0x7f123416a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.493+0000 7f123a6ef700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1234159510 0x7f12340c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.493+0000 7f123a6ef700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f123415ae20 0x7f12340c4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.493+0000 7f123a6ef700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123415b600 con 0x7f12340ca220 2024-01-26T23:45:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.493+0000 7f123a6ef700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123415b8e0 con 0x7f1234159510 2024-01-26T23:45:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.493+0000 7f123a6ef700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123415bbc0 con 0x7f123415ae20 2024-01-26T23:45:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.494+0000 7f1233fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12340ca220 0x7f123416a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.494+0000 7f1233fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12340ca220 0x7f123416a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37338/0 (socket says 172.21.15.46:37338) 2024-01-26T23:45:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.494+0000 7f1233fff700 1 -- 172.21.15.46:0/410347114 learned_addr learned my addr 172.21.15.46:0/410347114 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.494+0000 7f12337fe700 1 --2- 172.21.15.46:0/410347114 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1234159510 0x7f12340c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.494+0000 7f1238c8c700 1 --2- 172.21.15.46:0/410347114 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f123415ae20 0x7f12340c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.494+0000 7f1233fff700 1 -- 172.21.15.46:0/410347114 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f123415ae20 msgr2=0x7f12340c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.494+0000 7f1233fff700 1 --2- 172.21.15.46:0/410347114 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f123415ae20 0x7f12340c4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.494+0000 7f1233fff700 1 -- 172.21.15.46:0/410347114 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1234159510 msgr2=0x7f12340c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.494+0000 7f1233fff700 1 --2- 172.21.15.46:0/410347114 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1234159510 0x7f12340c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.494+0000 7f1233fff700 1 -- 172.21.15.46:0/410347114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121c0094f0 con 0x7f12340ca220 2024-01-26T23:45:28.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.494+0000 7f12337fe700 1 --2- 172.21.15.46:0/410347114 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1234159510 0x7f12340c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:28.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.494+0000 7f1233fff700 1 --2- 172.21.15.46:0/410347114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12340ca220 0x7f123416a2b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f122400e930 tx=0x7f12240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:28.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.494+0000 7f122bfff700 1 -- 172.21.15.46:0/410347114 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122400eed0 con 0x7f12340ca220 2024-01-26T23:45:28.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.494+0000 7f122bfff700 1 -- 172.21.15.46:0/410347114 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122400c850 con 0x7f12340ca220 2024-01-26T23:45:28.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.494+0000 7f123a6ef700 1 -- 172.21.15.46:0/410347114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123415bea0 con 0x7f12340ca220 2024-01-26T23:45:28.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.494+0000 7f122bfff700 1 -- 172.21.15.46:0/410347114 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122400cbf0 con 0x7f12340ca220 2024-01-26T23:45:28.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.494+0000 7f123a6ef700 1 -- 172.21.15.46:0/410347114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123415b290 con 0x7f12340ca220 2024-01-26T23:45:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.496+0000 7f123a6ef700 1 -- 172.21.15.46:0/410347114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f123404c710 con 0x7f12340ca220 2024-01-26T23:45:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.496+0000 7f122bfff700 1 -- 172.21.15.46:0/410347114 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1224013470 con 0x7f12340ca220 2024-01-26T23:45:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.496+0000 7f122bfff700 1 --2- 172.21.15.46:0/410347114 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1214066f30 0x7f12140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.496+0000 7f122bfff700 1 -- 172.21.15.46:0/410347114 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f1224084a90 con 0x7f12340ca220 2024-01-26T23:45:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.496+0000 7f12337fe700 1 --2- 172.21.15.46:0/410347114 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1214066f30 0x7f12140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.497+0000 7f12337fe700 1 --2- 172.21.15.46:0/410347114 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1214066f30 0x7f12140693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1234159e20 tx=0x7f121c017000 comp rx=0 tx=0).ready entity=mgr.5660 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:28.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.501+0000 7f122bfff700 1 -- 172.21.15.46:0/410347114 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1224051990 con 0x7f12340ca220 2024-01-26T23:45:28.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.765+0000 7f123a6ef700 1 -- 172.21.15.46:0/410347114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f12340c9d10 con 0x7f12340ca220 2024-01-26T23:45:28.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.886+0000 7f122bfff700 1 -- 172.21.15.46:0/410347114 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 31) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f1224051d60 con 0x7f12340ca220 2024-01-26T23:45:28.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.888+0000 7f122bfff700 1 -- 172.21.15.46:0/410347114 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v31) v1 ==== 92+0+0 (secure 0 0 0) 0x7f1224051990 con 0x7f12340ca220 2024-01-26T23:45:28.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.891+0000 7f123a6ef700 1 -- 172.21.15.46:0/410347114 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1214066f30 msgr2=0x7f12140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:28.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.891+0000 7f123a6ef700 1 --2- 172.21.15.46:0/410347114 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1214066f30 0x7f12140693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1234159e20 tx=0x7f121c017000 comp rx=0 tx=0).stop 2024-01-26T23:45:28.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.891+0000 7f123a6ef700 1 -- 172.21.15.46:0/410347114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12340ca220 msgr2=0x7f123416a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:28.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.891+0000 7f123a6ef700 1 --2- 172.21.15.46:0/410347114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12340ca220 0x7f123416a2b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f122400e930 tx=0x7f12240091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.891+0000 7f123a6ef700 1 -- 172.21.15.46:0/410347114 shutdown_connections 2024-01-26T23:45:28.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.891+0000 7f123a6ef700 1 --2- 172.21.15.46:0/410347114 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1214066f30 0x7f12140693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.891+0000 7f123a6ef700 1 --2- 172.21.15.46:0/410347114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12340ca220 0x7f123416a2b0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.891+0000 7f123a6ef700 1 --2- 172.21.15.46:0/410347114 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f123415ae20 0x7f12340c4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.891+0000 7f123a6ef700 1 --2- 172.21.15.46:0/410347114 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1234159510 0x7f12340c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:28.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.891+0000 7f123a6ef700 1 -- 172.21.15.46:0/410347114 >> 172.21.15.46:0/410347114 conn(0x7f123406bd90 msgr2=0x7f123406a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:28.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.891+0000 7f123a6ef700 1 -- 172.21.15.46:0/410347114 shutdown_connections 2024-01-26T23:45:28.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:28.891+0000 7f123a6ef700 1 -- 172.21.15.46:0/410347114 wait complete. 2024-01-26T23:45:28.907 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-26T23:45:29.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.046+0000 7f151a444700 1 Processor -- start 2024-01-26T23:45:29.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.046+0000 7f151a444700 1 -- start start 2024-01-26T23:45:29.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.047+0000 7f151a444700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1514157750 0x7f1514157b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:29.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.047+0000 7f151a444700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15140af790 0x7f15140b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:29.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.047+0000 7f151a444700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1514158f00 con 0x7f151406b3c0 2024-01-26T23:45:29.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.047+0000 7f151a444700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1514158060 con 0x7f1514157750 2024-01-26T23:45:29.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.047+0000 7f151a444700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151406fe90 con 0x7f15140af790 2024-01-26T23:45:29.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.047+0000 7f15137fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f151406b3c0 0x7f15140af080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43384/0 (socket says 172.21.15.46:43384) 2024-01-26T23:45:29.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.047+0000 7f15137fe700 1 -- 172.21.15.46:0/2421191614 learned_addr learned my addr 172.21.15.46:0/2421191614 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:29.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.047+0000 7f15189e1700 1 --2- 172.21.15.46:0/2421191614 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15140af790 0x7f15140b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:29.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.048+0000 7f1513fff700 1 --2- 172.21.15.46:0/2421191614 >> v2:172.21.15.106:3300/0 conn(0x7f1514157750 0x7f1514157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:29.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.048+0000 7f1512ffd700 1 -- 172.21.15.46:0/2421191614 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373564174 0 0) 0x7f1514158f00 con 0x7f151406b3c0 2024-01-26T23:45:29.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.048+0000 7f1512ffd700 1 -- 172.21.15.46:0/2421191614 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14fc003570 con 0x7f151406b3c0 2024-01-26T23:45:29.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.048+0000 7f15189e1700 1 -- 172.21.15.46:0/2421191614 >> v2:172.21.15.106:3300/0 conn(0x7f1514157750 msgr2=0x7f1514157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:29.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.048+0000 7f15189e1700 1 --2- 172.21.15.46:0/2421191614 >> v2:172.21.15.106:3300/0 conn(0x7f1514157750 0x7f1514157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.048+0000 7f15189e1700 1 -- 172.21.15.46:0/2421191614 >> v1:172.21.15.46:6789/0 conn(0x7f151406b3c0 legacy=0x7f15140af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:29.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.048+0000 7f15189e1700 1 -- 172.21.15.46:0/2421191614 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15141582f0 con 0x7f15140af790 2024-01-26T23:45:29.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.048+0000 7f1513fff700 1 --2- 172.21.15.46:0/2421191614 >> v2:172.21.15.106:3300/0 conn(0x7f1514157750 0x7f1514157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:29.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.048+0000 7f15189e1700 1 --2- 172.21.15.46:0/2421191614 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15140af790 0x7f15140b1c90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f150c009e60 tx=0x7f150c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69d16f12c26247a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:29.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.048+0000 7f1512ffd700 1 -- 172.21.15.46:0/2421191614 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c00e070 con 0x7f15140af790 2024-01-26T23:45:29.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.048+0000 7f1512ffd700 1 -- 172.21.15.46:0/2421191614 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150c00bab0 con 0x7f15140af790 2024-01-26T23:45:29.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.048+0000 7f1512ffd700 1 -- 172.21.15.46:0/2421191614 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c00be30 con 0x7f15140af790 2024-01-26T23:45:29.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.049+0000 7f151a444700 1 -- 172.21.15.46:0/2421191614 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15140af790 msgr2=0x7f15140b1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:29.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.049+0000 7f151a444700 1 --2- 172.21.15.46:0/2421191614 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15140af790 0x7f15140b1c90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f150c009e60 tx=0x7f150c009780 comp rx=0 tx=0).stop 2024-01-26T23:45:29.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.049+0000 7f151a444700 1 -- 172.21.15.46:0/2421191614 shutdown_connections 2024-01-26T23:45:29.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.049+0000 7f151a444700 1 --2- 172.21.15.46:0/2421191614 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15140af790 0x7f15140b1c90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.049+0000 7f151a444700 1 --2- 172.21.15.46:0/2421191614 >> v2:172.21.15.106:3300/0 conn(0x7f1514157750 0x7f1514157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.049+0000 7f151a444700 1 -- 172.21.15.46:0/2421191614 >> 172.21.15.46:0/2421191614 conn(0x7f15140cdf80 msgr2=0x7f151406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:29.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.049+0000 7f151a444700 1 -- 172.21.15.46:0/2421191614 shutdown_connections 2024-01-26T23:45:29.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.049+0000 7f151a444700 1 -- 172.21.15.46:0/2421191614 wait complete. 2024-01-26T23:45:29.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.050+0000 7f151a444700 1 Processor -- start 2024-01-26T23:45:29.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.051+0000 7f151a444700 1 -- start start 2024-01-26T23:45:29.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.051+0000 7f151a444700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f151406b3c0 0x7f15140abdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:29.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.051+0000 7f151a444700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15140af790 0x7f1514153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:29.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.051+0000 7f151a444700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1514157750 0x7f15140a8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:29.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.051+0000 7f151a444700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15140a9940 con 0x7f15140af790 2024-01-26T23:45:29.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.051+0000 7f151a444700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15140ab960 con 0x7f1514157750 2024-01-26T23:45:29.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.051+0000 7f151a444700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15141685c0 con 0x7f151406b3c0 2024-01-26T23:45:29.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.052+0000 7f15137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15140af790 0x7f1514153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:29.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.052+0000 7f1513fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f151406b3c0 0x7f15140abdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:29.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.052+0000 7f15137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15140af790 0x7f1514153c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37362/0 (socket says 172.21.15.46:37362) 2024-01-26T23:45:29.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.052+0000 7f15137fe700 1 -- 172.21.15.46:0/650999509 learned_addr learned my addr 172.21.15.46:0/650999509 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:29.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.052+0000 7f15189e1700 1 --2- 172.21.15.46:0/650999509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1514157750 0x7f15140a8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:29.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.052+0000 7f1513fff700 1 -- 172.21.15.46:0/650999509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1514157750 msgr2=0x7f15140a8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:29.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.052+0000 7f1513fff700 1 --2- 172.21.15.46:0/650999509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1514157750 0x7f15140a8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.052+0000 7f1513fff700 1 -- 172.21.15.46:0/650999509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15140af790 msgr2=0x7f1514153c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:29.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.052+0000 7f1513fff700 1 --2- 172.21.15.46:0/650999509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15140af790 0x7f1514153c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.052+0000 7f1513fff700 1 -- 172.21.15.46:0/650999509 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1504009420 con 0x7f151406b3c0 2024-01-26T23:45:29.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.052+0000 7f1513fff700 1 --2- 172.21.15.46:0/650999509 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f151406b3c0 0x7f15140abdf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f150400ec80 tx=0x7f15040091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:29.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.052+0000 7f14fbfff700 1 -- 172.21.15.46:0/650999509 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150400c850 con 0x7f151406b3c0 2024-01-26T23:45:29.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.052+0000 7f14fbfff700 1 -- 172.21.15.46:0/650999509 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150400c9b0 con 0x7f151406b3c0 2024-01-26T23:45:29.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.052+0000 7f14fbfff700 1 -- 172.21.15.46:0/650999509 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150400cd70 con 0x7f151406b3c0 2024-01-26T23:45:29.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.052+0000 7f151a444700 1 -- 172.21.15.46:0/650999509 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150c009a20 con 0x7f151406b3c0 2024-01-26T23:45:29.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.052+0000 7f151a444700 1 -- 172.21.15.46:0/650999509 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15140ab310 con 0x7f151406b3c0 2024-01-26T23:45:29.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.054+0000 7f151a444700 1 -- 172.21.15.46:0/650999509 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15140abc40 con 0x7f151406b3c0 2024-01-26T23:45:29.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.054+0000 7f14fbfff700 1 -- 172.21.15.46:0/650999509 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 31) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f1504013680 con 0x7f151406b3c0 2024-01-26T23:45:29.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.054+0000 7f14fbfff700 1 --2- 172.21.15.46:0/650999509 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1500066f80 0x7f1500069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:29.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.054+0000 7f15137fe700 1 -- 172.21.15.46:0/650999509 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1500066f80 msgr2=0x7f1500069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:29.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.054+0000 7f14fbfff700 1 -- 172.21.15.46:0/650999509 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f1504084b60 con 0x7f151406b3c0 2024-01-26T23:45:29.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.054+0000 7f15137fe700 1 --2- 172.21.15.46:0/650999509 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1500066f80 0x7f1500069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:29.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.063+0000 7f14fbfff700 1 -- 172.21.15.46:0/650999509 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1504051a60 con 0x7f151406b3c0 2024-01-26T23:45:29.226 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:29.225+0000 7f24075931c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:45:29.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.255+0000 7f15137fe700 1 -- 172.21.15.46:0/650999509 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1500066f80 msgr2=0x7f1500069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:29.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.255+0000 7f15137fe700 1 --2- 172.21.15.46:0/650999509 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1500066f80 0x7f1500069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:29.266 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:29.266+0000 7f4f434221c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:45:29.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.270+0000 7f151a444700 1 -- 172.21.15.46:0/650999509 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f1514157070 con 0x7f151406b3c0 2024-01-26T23:45:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.275+0000 7f14fbfff700 1 -- 172.21.15.46:0/650999509 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1504055080 con 0x7f151406b3c0 2024-01-26T23:45:29.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.278+0000 7f151a444700 1 -- 172.21.15.46:0/650999509 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1500066f80 msgr2=0x7f1500069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:29.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.278+0000 7f151a444700 1 --2- 172.21.15.46:0/650999509 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1500066f80 0x7f1500069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.278+0000 7f151a444700 1 -- 172.21.15.46:0/650999509 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f151406b3c0 msgr2=0x7f15140abdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:29.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.278+0000 7f151a444700 1 --2- 172.21.15.46:0/650999509 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f151406b3c0 0x7f15140abdf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f150400ec80 tx=0x7f15040091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.278+0000 7f151a444700 1 -- 172.21.15.46:0/650999509 shutdown_connections 2024-01-26T23:45:29.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.278+0000 7f151a444700 1 --2- 172.21.15.46:0/650999509 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1500066f80 0x7f1500069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.278+0000 7f151a444700 1 --2- 172.21.15.46:0/650999509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1514157750 0x7f15140a8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.278+0000 7f151a444700 1 --2- 172.21.15.46:0/650999509 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f151406b3c0 0x7f15140abdf0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.278+0000 7f151a444700 1 --2- 172.21.15.46:0/650999509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15140af790 0x7f1514153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.278+0000 7f151a444700 1 -- 172.21.15.46:0/650999509 >> 172.21.15.46:0/650999509 conn(0x7f15140cdf80 msgr2=0x7f1514155cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:29.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.278+0000 7f151a444700 1 -- 172.21.15.46:0/650999509 shutdown_connections 2024-01-26T23:45:29.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.278+0000 7f151a444700 1 -- 172.21.15.46:0/650999509 wait complete. 2024-01-26T23:45:29.292 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-26T23:45:29.304 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:29.304+0000 7f24075931c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:45:29.358 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:29.358+0000 7f4f434221c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:45:29.375 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:29.374+0000 7f24075931c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:45:29.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.429+0000 7f56e6bd0700 1 Processor -- start 2024-01-26T23:45:29.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.429+0000 7f56e6bd0700 1 -- start start 2024-01-26T23:45:29.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.430+0000 7f56e6bd0700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f56e01599d0 0x7f56e0159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:29.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.430+0000 7f56e6bd0700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e00ab620 con 0x7f56e00ab1c0 2024-01-26T23:45:29.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.430+0000 7f56e6bd0700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e00b1a60 con 0x7f56e015d8c0 2024-01-26T23:45:29.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.430+0000 7f56e6bd0700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e00b11c0 con 0x7f56e01599d0 2024-01-26T23:45:29.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.430+0000 7f56dffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f56e00ab1c0 0x7f56e00b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43394/0 (socket says 172.21.15.46:43394) 2024-01-26T23:45:29.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.430+0000 7f56dffff700 1 -- 172.21.15.46:0/2401193768 learned_addr learned my addr 172.21.15.46:0/2401193768 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:29.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.430+0000 7f56e496c700 1 --2- 172.21.15.46:0/2401193768 >> v2:172.21.15.106:3300/0 conn(0x7f56e01599d0 0x7f56e0159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:29.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.431+0000 7f56df7fe700 1 -- 172.21.15.46:0/2401193768 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573812177 0 0) 0x7f56cc001110 con 0x7f56e00ab1c0 2024-01-26T23:45:29.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.431+0000 7f56df7fe700 1 -- 172.21.15.46:0/2401193768 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d0003570 con 0x7f56e00ab1c0 2024-01-26T23:45:29.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.431+0000 7f56df7fe700 1 -- 172.21.15.46:0/2401193768 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083299240 0 0) 0x7f56e00b1a60 con 0x7f56e015d8c0 2024-01-26T23:45:29.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.431+0000 7f56e496c700 1 -- 172.21.15.46:0/2401193768 >> v1:172.21.15.106:6789/0 conn(0x7f56e015d8c0 legacy=0x7f56e00b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:29.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.431+0000 7f56e496c700 1 -- 172.21.15.46:0/2401193768 >> v1:172.21.15.46:6789/0 conn(0x7f56e00ab1c0 legacy=0x7f56e00b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:29.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.431+0000 7f56e496c700 1 -- 172.21.15.46:0/2401193768 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e00b14a0 con 0x7f56e01599d0 2024-01-26T23:45:29.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.431+0000 7f56e496c700 1 --2- 172.21.15.46:0/2401193768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56e01599d0 0x7f56e0159da0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f56d40098d0 tx=0x7f56d4009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c8e48caf063cac server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:29.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.431+0000 7f56df7fe700 1 -- 172.21.15.46:0/2401193768 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d4005c10 con 0x7f56e01599d0 2024-01-26T23:45:29.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.431+0000 7f56df7fe700 1 -- 172.21.15.46:0/2401193768 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d4005d70 con 0x7f56e01599d0 2024-01-26T23:45:29.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.431+0000 7f56df7fe700 1 -- 172.21.15.46:0/2401193768 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d4006280 con 0x7f56e01599d0 2024-01-26T23:45:29.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.432+0000 7f56e6bd0700 1 -- 172.21.15.46:0/2401193768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56e01599d0 msgr2=0x7f56e0159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:29.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.432+0000 7f56e6bd0700 1 --2- 172.21.15.46:0/2401193768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56e01599d0 0x7f56e0159da0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f56d40098d0 tx=0x7f56d4009120 comp rx=0 tx=0).stop 2024-01-26T23:45:29.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.432+0000 7f56e6bd0700 1 -- 172.21.15.46:0/2401193768 shutdown_connections 2024-01-26T23:45:29.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.432+0000 7f56e6bd0700 1 --2- 172.21.15.46:0/2401193768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56e01599d0 0x7f56e0159da0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.432+0000 7f56e6bd0700 1 -- 172.21.15.46:0/2401193768 >> 172.21.15.46:0/2401193768 conn(0x7f56e006f840 msgr2=0x7f56e0087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:29.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.432+0000 7f56e6bd0700 1 -- 172.21.15.46:0/2401193768 shutdown_connections 2024-01-26T23:45:29.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.432+0000 7f56e6bd0700 1 -- 172.21.15.46:0/2401193768 wait complete. 2024-01-26T23:45:29.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.433+0000 7f56e6bd0700 1 Processor -- start 2024-01-26T23:45:29.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.433+0000 7f56e6bd0700 1 -- start start 2024-01-26T23:45:29.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.434+0000 7f56e6bd0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56e00ab1c0 0x7f56e009b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:29.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.434+0000 7f56e6bd0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e01599d0 0x7f56e00aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:29.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.434+0000 7f56e6bd0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56e015d8c0 0x7f56e0157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:29.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.434+0000 7f56e6bd0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e00ae7c0 con 0x7f56e01599d0 2024-01-26T23:45:29.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.434+0000 7f56e6bd0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e00adb30 con 0x7f56e015d8c0 2024-01-26T23:45:29.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.434+0000 7f56e6bd0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e00ade10 con 0x7f56e00ab1c0 2024-01-26T23:45:29.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.434+0000 7f56dffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e01599d0 0x7f56e00aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:29.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.434+0000 7f56dffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e01599d0 0x7f56e00aec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37374/0 (socket says 172.21.15.46:37374) 2024-01-26T23:45:29.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.434+0000 7f56dffff700 1 -- 172.21.15.46:0/833794962 learned_addr learned my addr 172.21.15.46:0/833794962 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:29.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.434+0000 7f56e496c700 1 --2- 172.21.15.46:0/833794962 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56e00ab1c0 0x7f56e009b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:29.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.434+0000 7f56e516d700 1 --2- 172.21.15.46:0/833794962 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56e015d8c0 0x7f56e0157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:29.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.434+0000 7f56dffff700 1 -- 172.21.15.46:0/833794962 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56e00ab1c0 msgr2=0x7f56e009b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:29.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.434+0000 7f56dffff700 1 --2- 172.21.15.46:0/833794962 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56e00ab1c0 0x7f56e009b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.434+0000 7f56dffff700 1 -- 172.21.15.46:0/833794962 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56e015d8c0 msgr2=0x7f56e0157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:29.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.434+0000 7f56dffff700 1 --2- 172.21.15.46:0/833794962 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56e015d8c0 0x7f56e0157fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.434+0000 7f56dffff700 1 -- 172.21.15.46:0/833794962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d40093c0 con 0x7f56e01599d0 2024-01-26T23:45:29.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.435+0000 7f56dffff700 1 --2- 172.21.15.46:0/833794962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e01599d0 0x7f56e00aec50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00c7c0 tx=0x7f56cc007c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:29.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.435+0000 7f56dcff9700 1 -- 172.21.15.46:0/833794962 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc005b30 con 0x7f56e01599d0 2024-01-26T23:45:29.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.435+0000 7f56dcff9700 1 -- 172.21.15.46:0/833794962 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56cc005c90 con 0x7f56e01599d0 2024-01-26T23:45:29.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.435+0000 7f56e6bd0700 1 -- 172.21.15.46:0/833794962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e00ae170 con 0x7f56e01599d0 2024-01-26T23:45:29.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.435+0000 7f56dcff9700 1 -- 172.21.15.46:0/833794962 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc007300 con 0x7f56e01599d0 2024-01-26T23:45:29.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.435+0000 7f56e6bd0700 1 -- 172.21.15.46:0/833794962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e009b7a0 con 0x7f56e01599d0 2024-01-26T23:45:29.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.436+0000 7f56e6bd0700 1 -- 172.21.15.46:0/833794962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e00ae970 con 0x7f56e01599d0 2024-01-26T23:45:29.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.437+0000 7f56dcff9700 1 -- 172.21.15.46:0/833794962 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 31) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f56cc0076b0 con 0x7f56e01599d0 2024-01-26T23:45:29.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.437+0000 7f56dcff9700 1 --2- 172.21.15.46:0/833794962 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f56c4066f80 0x7f56c4069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:29.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.437+0000 7f56e496c700 1 -- 172.21.15.46:0/833794962 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f56c4066f80 msgr2=0x7f56c4069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:29.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.437+0000 7f56dcff9700 1 -- 172.21.15.46:0/833794962 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f56cc083b20 con 0x7f56e01599d0 2024-01-26T23:45:29.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.437+0000 7f56e496c700 1 --2- 172.21.15.46:0/833794962 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f56c4066f80 0x7f56c4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:29.440 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:29.439+0000 7f24075931c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:45:29.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.441+0000 7f56dcff9700 1 -- 172.21.15.46:0/833794962 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f56e00ae970 con 0x7f56e01599d0 2024-01-26T23:45:29.442 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:29.442+0000 7f4f434221c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:45:29.518 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:29.518+0000 7f4f434221c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:45:29.595 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:29.595+0000 7f24075931c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:45:29.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.638+0000 7f56e496c700 1 -- 172.21.15.46:0/833794962 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f56c4066f80 msgr2=0x7f56c4069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:29.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.638+0000 7f56e496c700 1 --2- 172.21.15.46:0/833794962 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f56c4066f80 0x7f56c4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:29.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.652+0000 7f56e6bd0700 1 -- 172.21.15.46:0/833794962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f56e004c780 con 0x7f56e01599d0 2024-01-26T23:45:29.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.656+0000 7f56dcff9700 1 -- 172.21.15.46:0/833794962 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7f56e004c780 con 0x7f56e01599d0 2024-01-26T23:45:29.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.658+0000 7f56e6bd0700 1 -- 172.21.15.46:0/833794962 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f56c4066f80 msgr2=0x7f56c4069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:29.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.658+0000 7f56e6bd0700 1 --2- 172.21.15.46:0/833794962 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f56c4066f80 0x7f56c4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.658+0000 7f56e6bd0700 1 -- 172.21.15.46:0/833794962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e01599d0 msgr2=0x7f56e00aec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.658+0000 7f56e6bd0700 1 --2- 172.21.15.46:0/833794962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e01599d0 0x7f56e00aec50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00c7c0 tx=0x7f56cc007c30 comp rx=0 tx=0).stop 2024-01-26T23:45:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.658+0000 7f56e6bd0700 1 -- 172.21.15.46:0/833794962 shutdown_connections 2024-01-26T23:45:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.658+0000 7f56e6bd0700 1 --2- 172.21.15.46:0/833794962 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f56c4066f80 0x7f56c4069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.658+0000 7f56e6bd0700 1 --2- 172.21.15.46:0/833794962 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56e015d8c0 0x7f56e0157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.658+0000 7f56e6bd0700 1 --2- 172.21.15.46:0/833794962 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56e00ab1c0 0x7f56e009b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.658+0000 7f56e6bd0700 1 --2- 172.21.15.46:0/833794962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e01599d0 0x7f56e00aec50 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.658+0000 7f56e6bd0700 1 -- 172.21.15.46:0/833794962 >> 172.21.15.46:0/833794962 conn(0x7f56e006f840 msgr2=0x7f56e00b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.658+0000 7f56e6bd0700 1 -- 172.21.15.46:0/833794962 shutdown_connections 2024-01-26T23:45:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.658+0000 7f56e6bd0700 1 -- 172.21.15.46:0/833794962 wait complete. 2024-01-26T23:45:29.673 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:45:29.673 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-26T23:45:29.697 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-01-26T23:45:29.699 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:29.699+0000 7f4f434221c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:45:29.733 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:29.732+0000 7f24075931c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:45:29.801 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:29.798+0000 7f24075931c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:45:29.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.835+0000 7f9a979cc700 1 Processor -- start 2024-01-26T23:45:29.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.835+0000 7f9a979cc700 1 -- start start 2024-01-26T23:45:29.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.836+0000 7f9a979cc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9a90155c00 0x7f9a9006bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:29.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.836+0000 7f9a979cc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9a90155fd0 0x7f9a9015a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:29.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.836+0000 7f9a979cc700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9006b740 con 0x7f9a9006a3c0 2024-01-26T23:45:29.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.836+0000 7f9a979cc700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a9006ab30 con 0x7f9a90155fd0 2024-01-26T23:45:29.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.836+0000 7f9a979cc700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a9006ae10 con 0x7f9a90155c00 2024-01-26T23:45:29.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.836+0000 7f9a94f67700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9a90155fd0 0x7f9a9015a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.836+0000 7f9a94f67700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9a90155fd0 0x7f9a9015a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37376/0 (socket says 172.21.15.46:37376) 2024-01-26T23:45:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.836+0000 7f9a94f67700 1 -- 172.21.15.46:0/2703344834 learned_addr learned my addr 172.21.15.46:0/2703344834 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.836+0000 7f9a95768700 1 --2- 172.21.15.46:0/2703344834 >> v2:172.21.15.106:3300/0 conn(0x7f9a90155c00 0x7f9a9006bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.836+0000 7f9a94f67700 1 -- 172.21.15.46:0/2703344834 >> v2:172.21.15.106:3300/0 conn(0x7f9a90155c00 msgr2=0x7f9a9006bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.836+0000 7f9a94f67700 1 --2- 172.21.15.46:0/2703344834 >> v2:172.21.15.106:3300/0 conn(0x7f9a90155c00 0x7f9a9006bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.836+0000 7f9a94f67700 1 -- 172.21.15.46:0/2703344834 >> v1:172.21.15.106:6789/0 conn(0x7f9a9006a3c0 legacy=0x7f9a90068940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.836+0000 7f9a94f67700 1 -- 172.21.15.46:0/2703344834 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9006b0f0 con 0x7f9a90155fd0 2024-01-26T23:45:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.837+0000 7f9a94f67700 1 --2- 172.21.15.46:0/2703344834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a90155fd0 0x7f9a9015a3b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f9a9006dd20 tx=0x7f9a80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35842c561cd07 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.837+0000 7f9a87fff700 1 -- 172.21.15.46:0/2703344834 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a80009e70 con 0x7f9a90155fd0 2024-01-26T23:45:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.837+0000 7f9a87fff700 1 -- 172.21.15.46:0/2703344834 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8000b860 con 0x7f9a90155fd0 2024-01-26T23:45:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.837+0000 7f9a87fff700 1 -- 172.21.15.46:0/2703344834 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8000bc00 con 0x7f9a90155fd0 2024-01-26T23:45:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.837+0000 7f9a979cc700 1 -- 172.21.15.46:0/2703344834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a90155fd0 msgr2=0x7f9a9015a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.837+0000 7f9a979cc700 1 --2- 172.21.15.46:0/2703344834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a90155fd0 0x7f9a9015a3b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f9a9006dd20 tx=0x7f9a80009250 comp rx=0 tx=0).stop 2024-01-26T23:45:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.838+0000 7f9a979cc700 1 -- 172.21.15.46:0/2703344834 shutdown_connections 2024-01-26T23:45:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.838+0000 7f9a979cc700 1 --2- 172.21.15.46:0/2703344834 >> v2:172.21.15.106:3300/0 conn(0x7f9a90155c00 0x7f9a9006bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.838+0000 7f9a979cc700 1 --2- 172.21.15.46:0/2703344834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a90155fd0 0x7f9a9015a3b0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.838+0000 7f9a979cc700 1 -- 172.21.15.46:0/2703344834 >> 172.21.15.46:0/2703344834 conn(0x7f9a900ca100 msgr2=0x7f9a9006e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.838+0000 7f9a979cc700 1 -- 172.21.15.46:0/2703344834 shutdown_connections 2024-01-26T23:45:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.838+0000 7f9a979cc700 1 -- 172.21.15.46:0/2703344834 wait complete. 2024-01-26T23:45:29.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.840+0000 7f9a979cc700 1 Processor -- start 2024-01-26T23:45:29.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.840+0000 7f9a979cc700 1 -- start start 2024-01-26T23:45:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.841+0000 7f9a979cc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9a9006a3c0 0x7f9a900cb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.841+0000 7f9a979cc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a90155c00 0x7f9a9014d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.841+0000 7f9a979cc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9a90155fd0 0x7f9a900b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.841+0000 7f9a979cc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a90159120 con 0x7f9a90155c00 2024-01-26T23:45:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.841+0000 7f9a979cc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a900cbbd0 con 0x7f9a90155fd0 2024-01-26T23:45:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.841+0000 7f9a979cc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9014d610 con 0x7f9a9006a3c0 2024-01-26T23:45:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.841+0000 7f9a94f67700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a90155c00 0x7f9a9014d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.841+0000 7f9a94f67700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a90155c00 0x7f9a9014d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37382/0 (socket says 172.21.15.46:37382) 2024-01-26T23:45:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.841+0000 7f9a94f67700 1 -- 172.21.15.46:0/4219308837 learned_addr learned my addr 172.21.15.46:0/4219308837 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.842+0000 7f9a95768700 1 --2- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9a9006a3c0 0x7f9a900cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.842+0000 7f9a95f69700 1 --2- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9a90155fd0 0x7f9a900b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.842+0000 7f9a94f67700 1 -- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9a9006a3c0 msgr2=0x7f9a900cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.842+0000 7f9a94f67700 1 --2- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9a9006a3c0 0x7f9a900cb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.842+0000 7f9a94f67700 1 -- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9a90155fd0 msgr2=0x7f9a900b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.842+0000 7f9a94f67700 1 --2- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9a90155fd0 0x7f9a900b1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.842+0000 7f9a94f67700 1 -- 172.21.15.46:0/4219308837 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a800094f0 con 0x7f9a90155c00 2024-01-26T23:45:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.842+0000 7f9a94f67700 1 --2- 172.21.15.46:0/4219308837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a90155c00 0x7f9a9014d0d0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f9a90069d70 tx=0x7f9a80017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.842+0000 7f9a857fa700 1 -- 172.21.15.46:0/4219308837 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8000bbe0 con 0x7f9a90155c00 2024-01-26T23:45:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.842+0000 7f9a857fa700 1 -- 172.21.15.46:0/4219308837 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a80011420 con 0x7f9a90155c00 2024-01-26T23:45:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.842+0000 7f9a857fa700 1 -- 172.21.15.46:0/4219308837 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a800117c0 con 0x7f9a90155c00 2024-01-26T23:45:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.842+0000 7f9a979cc700 1 -- 172.21.15.46:0/4219308837 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a901584a0 con 0x7f9a90155c00 2024-01-26T23:45:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.842+0000 7f9a979cc700 1 -- 172.21.15.46:0/4219308837 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a90158a60 con 0x7f9a90155c00 2024-01-26T23:45:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.843+0000 7f9a979cc700 1 -- 172.21.15.46:0/4219308837 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9006eb30 con 0x7f9a90155c00 2024-01-26T23:45:29.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.844+0000 7f9a857fa700 1 -- 172.21.15.46:0/4219308837 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 31) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f9a80011bb0 con 0x7f9a90155c00 2024-01-26T23:45:29.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.844+0000 7f9a857fa700 1 --2- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9a74066f80 0x7f9a74069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:29.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.844+0000 7f9a95768700 1 -- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9a74066f80 msgr2=0x7f9a74069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:29.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.844+0000 7f9a857fa700 1 -- 172.21.15.46:0/4219308837 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f9a80085c10 con 0x7f9a90155c00 2024-01-26T23:45:29.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.844+0000 7f9a95768700 1 --2- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9a74066f80 0x7f9a74069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:29.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:29.849+0000 7f9a857fa700 1 -- 172.21.15.46:0/4219308837 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9a80052bb0 con 0x7f9a90155c00 2024-01-26T23:45:29.867 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:29.867+0000 7f4f434221c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:45:29.922 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:29.921+0000 7f24075931c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:45:29.945 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:29.945+0000 7f4f434221c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:45:29.994 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:29.994+0000 7f24075931c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:45:30.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:30.045+0000 7f9a95768700 1 -- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9a74066f80 msgr2=0x7f9a74069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:30.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:30.045+0000 7f9a95768700 1 --2- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9a74066f80 0x7f9a74069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:30.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:30.083+0000 7f9a979cc700 1 -- 172.21.15.46:0/4219308837 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a9004c710 con 0x7f9a74066f80 2024-01-26T23:45:30.085 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:30.085+0000 7f4f434221c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:45:30.134 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:30.133+0000 7f24075931c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:45:30.168 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:30.168+0000 7f4f434221c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:45:30.260 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:30.259+0000 7f24075931c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:45:30.344 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:30.344+0000 7f4f434221c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:45:30.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:30.446+0000 7f9a95768700 1 -- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9a74066f80 msgr2=0x7f9a74069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:30.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:30.446+0000 7f9a95768700 1 --2- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9a74066f80 0x7f9a74069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-26T23:45:30.486 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:30.486+0000 7f4f434221c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:45:30.572 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:30.572+0000 7f24075931c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:45:30.783 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:30.782+0000 7f24075931c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:45:30.847 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:30.847+0000 7f4f434221c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:45:31.007 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:31.006+0000 7f24075931c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:45:31.077 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:31.076+0000 7f24075931c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:45:31.086 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:31.086+0000 7f4f434221c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:45:31.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:31.247+0000 7f9a95768700 1 -- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9a74066f80 msgr2=0x7f9a74069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:31.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:31.247+0000 7f9a95768700 1 --2- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9a74066f80 0x7f9a74069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-26T23:45:31.356 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:31.356+0000 7f4f434221c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:45:31.437 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:31.437+0000 7f4f434221c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:45:31.679 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:31.678+0000 7f24075931c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:45:31.752 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:31.752+0000 7f24075931c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:45:31.875 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:31.874+0000 7f24075931c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:45:32.133 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:32.132+0000 7f4f434221c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:45:32.217 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:32.217+0000 7f4f434221c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:45:32.359 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:32.359+0000 7f4f434221c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:45:32.375 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:32.374+0000 7f24075931c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:45:32.522 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:32.521+0000 7f24075931c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:45:32.671 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:32.670+0000 7f24075931c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:45:32.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:32.848+0000 7f9a95768700 1 -- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9a74066f80 msgr2=0x7f9a74069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:32.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:32.848+0000 7f9a95768700 1 --2- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9a74066f80 0x7f9a74069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-26T23:45:32.948 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:32.947+0000 7f4f434221c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:45:33.119 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:33.119+0000 7f4f434221c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:45:33.292 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:33.292+0000 7f4f434221c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:45:33.366 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:33.365+0000 7f24075931c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:45:33.583 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:33.582+0000 7f24075931c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:45:33.655 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:33.654+0000 7f24075931c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:45:33.807 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:33.806+0000 7f24075931c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:45:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:33.828+0000 7f9a857fa700 1 -- 172.21.15.46:0/4219308837 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 32) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f9a80052f80 con 0x7f9a90155c00 2024-01-26T23:45:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:33.828+0000 7f9a857fa700 1 -- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9a74066f80 msgr2=0x7f9a74069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:33.828+0000 7f9a857fa700 1 --2- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9a74066f80 0x7f9a74069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:33.938 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:33.937+0000 7f237de71700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:33.938 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:33.937+0000 7f237de71700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:33.938 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:33.937+0000 7f237de71700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:33.938 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:33.937+0000 7f237de71700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:33.939 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:33.937+0000 7f237de71700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:33.939 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:33.937+0000 7f2380e77700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:33.939 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:33.937+0000 7f2380e77700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:33.939 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:33.937+0000 7f2380e77700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:33.939 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:33.937+0000 7f2380e77700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:33.939 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:33.937+0000 7f2380e77700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:34.110 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:34.109+0000 7f4f434221c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:45:34.360 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:34.360+0000 7f4f434221c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:45:34.444 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:34.444+0000 7f4f434221c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:45:34.622 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:34.622+0000 7f4f434221c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:45:34.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:34.830+0000 7f9a857fa700 1 -- 172.21.15.46:0/4219308837 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 33) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f9a8005d380 con 0x7f9a90155c00 2024-01-26T23:45:34.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:34.831+0000 7f9a857fa700 1 --2- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9a74066f80 0x7f9a74069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:34.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:34.831+0000 7f9a857fa700 1 -- 172.21.15.46:0/4219308837 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a9004c710 con 0x7f9a74066f80 2024-01-26T23:45:34.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:34.831+0000 7f9a95768700 1 --2- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9a74066f80 0x7f9a74069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:34.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:34.838+0000 7f9a95768700 1 --2- 172.21.15.46:0/4219308837 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9a74066f80 0x7f9a74069430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c0097b0 tx=0x7f9a8c009120 comp rx=0 tx=0).ready entity=mgr.6346 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:34.844 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:34.843+0000 7f2387e85700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-01-26T23:45:34.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:34.844+0000 7f9a857fa700 1 -- 172.21.15.46:0/4219308837 <== mgr.6346 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f9a9004c710 con 0x7f9a74066f80 2024-01-26T23:45:34.847 INFO:teuthology.orchestra.run.smithi046.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-01-26T23:45:34.850 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-26T23:45:34.851 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:45:34.851 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-01-26T23:45:34.877 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-26T23:45:35.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.061+0000 7fc3e3af0700 1 Processor -- start 2024-01-26T23:45:35.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.061+0000 7fc3e3af0700 1 -- start start 2024-01-26T23:45:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.062+0000 7fc3e3af0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3dc0b2060 0x7fc3dc0b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.062+0000 7fc3e3af0700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc3dc0cec50 0x7fc3dc15e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.062+0000 7fc3e3af0700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3dc15f180 con 0x7fc3dc0b3ae0 2024-01-26T23:45:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.062+0000 7fc3e3af0700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3dc0cf0d0 con 0x7fc3dc0b2060 2024-01-26T23:45:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.062+0000 7fc3e3af0700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3dc15e8e0 con 0x7fc3dc0cec50 2024-01-26T23:45:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.062+0000 7fc3e108b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3dc0b2060 0x7fc3dc0b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:35.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.062+0000 7fc3e188c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc3dc0b3ae0 0x7fc3dc0b1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53498/0 (socket says 172.21.15.46:53498) 2024-01-26T23:45:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.062+0000 7fc3e108b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3dc0b2060 0x7fc3dc0b3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50540/0 (socket says 172.21.15.46:50540) 2024-01-26T23:45:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.062+0000 7fc3e188c700 1 -- 172.21.15.46:0/2307863333 learned_addr learned my addr 172.21.15.46:0/2307863333 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.062+0000 7fc3e208d700 1 --2- 172.21.15.46:0/2307863333 >> v2:172.21.15.106:3300/0 conn(0x7fc3dc0cec50 0x7fc3dc15e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.062+0000 7fc3e108b700 1 -- 172.21.15.46:0/2307863333 >> v2:172.21.15.106:3300/0 conn(0x7fc3dc0cec50 msgr2=0x7fc3dc15e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.062+0000 7fc3e108b700 1 --2- 172.21.15.46:0/2307863333 >> v2:172.21.15.106:3300/0 conn(0x7fc3dc0cec50 0x7fc3dc15e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.062+0000 7fc3e108b700 1 -- 172.21.15.46:0/2307863333 >> v1:172.21.15.46:6789/0 conn(0x7fc3dc0b3ae0 legacy=0x7fc3dc0b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.062+0000 7fc3e108b700 1 -- 172.21.15.46:0/2307863333 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc15ebc0 con 0x7fc3dc0b2060 2024-01-26T23:45:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.062+0000 7fc3e088a700 1 -- 172.21.15.46:0/2307863333 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176176329 0 0) 0x7fc3dc15f180 con 0x7fc3dc0b3ae0 2024-01-26T23:45:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.063+0000 7fc3e108b700 1 --2- 172.21.15.46:0/2307863333 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b2060 0x7fc3dc0b3eb0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0b3360 tx=0x7fc3cc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221aa8e50022a77a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.063+0000 7fc3e088a700 1 -- 172.21.15.46:0/2307863333 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc00b6b0 con 0x7fc3dc0b2060 2024-01-26T23:45:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.063+0000 7fc3e088a700 1 -- 172.21.15.46:0/2307863333 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3cc00b810 con 0x7fc3dc0b2060 2024-01-26T23:45:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.063+0000 7fc3e088a700 1 -- 172.21.15.46:0/2307863333 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc00bbb0 con 0x7fc3dc0b2060 2024-01-26T23:45:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.063+0000 7fc3e3af0700 1 -- 172.21.15.46:0/2307863333 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b2060 msgr2=0x7fc3dc0b3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.063+0000 7fc3e3af0700 1 --2- 172.21.15.46:0/2307863333 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b2060 0x7fc3dc0b3eb0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0b3360 tx=0x7fc3cc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.063+0000 7fc3e3af0700 1 -- 172.21.15.46:0/2307863333 shutdown_connections 2024-01-26T23:45:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.063+0000 7fc3e3af0700 1 --2- 172.21.15.46:0/2307863333 >> v2:172.21.15.106:3300/0 conn(0x7fc3dc0cec50 0x7fc3dc15e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.063+0000 7fc3e3af0700 1 --2- 172.21.15.46:0/2307863333 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b2060 0x7fc3dc0b3eb0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.063+0000 7fc3e3af0700 1 -- 172.21.15.46:0/2307863333 >> 172.21.15.46:0/2307863333 conn(0x7fc3dc067930 msgr2=0x7fc3dc067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.064+0000 7fc3e3af0700 1 -- 172.21.15.46:0/2307863333 shutdown_connections 2024-01-26T23:45:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.064+0000 7fc3e3af0700 1 -- 172.21.15.46:0/2307863333 wait complete. 2024-01-26T23:45:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.065+0000 7fc3e3af0700 1 Processor -- start 2024-01-26T23:45:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.065+0000 7fc3e3af0700 1 -- start start 2024-01-26T23:45:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e3af0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3dc0b2060 0x7fc3dc1485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e3af0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b3ae0 0x7fc3dc163d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e3af0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3dc0cec50 0x7fc3dc15d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e3af0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc148ae0 con 0x7fc3dc0b3ae0 2024-01-26T23:45:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e3af0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc159080 con 0x7fc3dc0b2060 2024-01-26T23:45:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e3af0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc159510 con 0x7fc3dc0cec50 2024-01-26T23:45:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e108b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b3ae0 0x7fc3dc163d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e108b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b3ae0 0x7fc3dc163d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50552/0 (socket says 172.21.15.46:50552) 2024-01-26T23:45:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e108b700 1 -- 172.21.15.46:0/2151584745 learned_addr learned my addr 172.21.15.46:0/2151584745 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e208d700 1 --2- 172.21.15.46:0/2151584745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3dc0cec50 0x7fc3dc15d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e188c700 1 --2- 172.21.15.46:0/2151584745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3dc0b2060 0x7fc3dc1485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e108b700 1 -- 172.21.15.46:0/2151584745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3dc0cec50 msgr2=0x7fc3dc15d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e108b700 1 --2- 172.21.15.46:0/2151584745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3dc0cec50 0x7fc3dc15d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e108b700 1 -- 172.21.15.46:0/2151584745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3dc0b2060 msgr2=0x7fc3dc1485a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e108b700 1 --2- 172.21.15.46:0/2151584745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3dc0b2060 0x7fc3dc1485a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e108b700 1 -- 172.21.15.46:0/2151584745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3cc009420 con 0x7fc3dc0b3ae0 2024-01-26T23:45:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e208d700 1 --2- 172.21.15.46:0/2151584745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3dc0cec50 0x7fc3dc15d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:45:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e188c700 1 --2- 172.21.15.46:0/2151584745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3dc0b2060 0x7fc3dc1485a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:45:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e108b700 1 --2- 172.21.15.46:0/2151584745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b3ae0 0x7fc3dc163d50 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc06d060 tx=0x7fc3cc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3d1ffb700 1 -- 172.21.15.46:0/2151584745 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc00bb90 con 0x7fc3dc0b3ae0 2024-01-26T23:45:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3d1ffb700 1 -- 172.21.15.46:0/2151584745 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3cc011420 con 0x7fc3dc0b3ae0 2024-01-26T23:45:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3e3af0700 1 -- 172.21.15.46:0/2151584745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3dc158470 con 0x7fc3dc0b3ae0 2024-01-26T23:45:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.066+0000 7fc3d1ffb700 1 -- 172.21.15.46:0/2151584745 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc011800 con 0x7fc3dc0b3ae0 2024-01-26T23:45:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.067+0000 7fc3e3af0700 1 -- 172.21.15.46:0/2151584745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3dc158a30 con 0x7fc3dc0b3ae0 2024-01-26T23:45:35.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.068+0000 7fc3e3af0700 1 -- 172.21.15.46:0/2151584745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3dc04c710 con 0x7fc3dc0b3ae0 2024-01-26T23:45:35.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.068+0000 7fc3d1ffb700 1 -- 172.21.15.46:0/2151584745 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 33) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fc3cc011c30 con 0x7fc3dc0b3ae0 2024-01-26T23:45:35.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.069+0000 7fc3d1ffb700 1 --2- 172.21.15.46:0/2151584745 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc3c0066dd0 0x7fc3c0069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:35.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.069+0000 7fc3d1ffb700 1 -- 172.21.15.46:0/2151584745 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fc3cc085ea0 con 0x7fc3dc0b3ae0 2024-01-26T23:45:35.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.069+0000 7fc3e188c700 1 --2- 172.21.15.46:0/2151584745 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc3c0066dd0 0x7fc3c0069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:35.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.069+0000 7fc3e188c700 1 --2- 172.21.15.46:0/2151584745 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc3c0066dd0 0x7fc3c0069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc3d80018d0 tx=0x7fc3d8009040 comp rx=0 tx=0).ready entity=mgr.6346 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:35.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.073+0000 7fc3d1ffb700 1 -- 172.21.15.46:0/2151584745 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc3cc052ff0 con 0x7fc3dc0b3ae0 2024-01-26T23:45:35.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.340+0000 7fc3e3af0700 1 -- 172.21.15.46:0/2151584745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fc3dc161230 con 0x7fc3dc0b3ae0 2024-01-26T23:45:35.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.882+0000 7fc3d1ffb700 1 -- 172.21.15.46:0/2151584745 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fc3cc0533c0 con 0x7fc3dc0b3ae0 2024-01-26T23:45:35.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.884+0000 7fc3d1ffb700 1 -- 172.21.15.46:0/2151584745 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v34) v1 ==== 93+0+0 (secure 0 0 0) 0x7fc3cc056a80 con 0x7fc3dc0b3ae0 2024-01-26T23:45:35.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.887+0000 7fc3e3af0700 1 -- 172.21.15.46:0/2151584745 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc3c0066dd0 msgr2=0x7fc3c0069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:35.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.887+0000 7fc3e3af0700 1 --2- 172.21.15.46:0/2151584745 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc3c0066dd0 0x7fc3c0069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc3d80018d0 tx=0x7fc3d8009040 comp rx=0 tx=0).stop 2024-01-26T23:45:35.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.887+0000 7fc3e3af0700 1 -- 172.21.15.46:0/2151584745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b3ae0 msgr2=0x7fc3dc163d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:35.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.887+0000 7fc3e3af0700 1 --2- 172.21.15.46:0/2151584745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b3ae0 0x7fc3dc163d50 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc06d060 tx=0x7fc3cc017000 comp rx=0 tx=0).stop 2024-01-26T23:45:35.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.887+0000 7fc3e3af0700 1 -- 172.21.15.46:0/2151584745 shutdown_connections 2024-01-26T23:45:35.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.887+0000 7fc3e3af0700 1 --2- 172.21.15.46:0/2151584745 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc3c0066dd0 0x7fc3c0069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:35.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.887+0000 7fc3e3af0700 1 --2- 172.21.15.46:0/2151584745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3dc0cec50 0x7fc3dc15d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:35.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.887+0000 7fc3e3af0700 1 --2- 172.21.15.46:0/2151584745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3dc0b2060 0x7fc3dc1485a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:35.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.887+0000 7fc3e3af0700 1 --2- 172.21.15.46:0/2151584745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b3ae0 0x7fc3dc163d50 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:35.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.887+0000 7fc3e3af0700 1 -- 172.21.15.46:0/2151584745 >> 172.21.15.46:0/2151584745 conn(0x7fc3dc067930 msgr2=0x7fc3dc06a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:35.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.887+0000 7fc3e3af0700 1 -- 172.21.15.46:0/2151584745 shutdown_connections 2024-01-26T23:45:35.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:35.887+0000 7fc3e3af0700 1 -- 172.21.15.46:0/2151584745 wait complete. 2024-01-26T23:45:35.904 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-26T23:45:35.904 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:45:35.950 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:45:35.986 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-26T23:45:35.986 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-26T23:45:35.987 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-26T23:45:35.987 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-26T23:45:36.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[72965]: fuse finished with error 0 and tester_r 0 2024-01-26T23:45:36.026 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:45:36.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:45:36.026 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-26T23:45:36.196 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:45:36.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:45:36.196 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-26T23:45:36.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:45:36.212 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-26T23:45:36.213 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:45:36.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:45:36.213 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-26T23:45:36.220 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:36.219+0000 7f70091c51c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:45:36.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:45:36.254 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-26T23:45:36.255 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:45:36.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:45:36.255 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-26T23:45:36.262 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:36.262+0000 7f023aa4e1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:45:36.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:45:36.271 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-26T23:45:36.272 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:45:36.299 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:36.298+0000 7f70091c51c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:45:36.352 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:36.352+0000 7f023aa4e1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:45:36.369 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:36.369+0000 7f70091c51c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:45:36.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.405+0000 7fd253c59700 1 Processor -- start 2024-01-26T23:45:36.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.405+0000 7fd253c59700 1 -- start start 2024-01-26T23:45:36.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.406+0000 7fd253c59700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd24c15d210 0x7fd24c0cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:36.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.406+0000 7fd253c59700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd24c0cbff0 0x7fd24c0ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:36.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.406+0000 7fd253c59700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24c0cff70 con 0x7fd24c15ce40 2024-01-26T23:45:36.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.406+0000 7fd253c59700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd24c0cc490 con 0x7fd24c15d210 2024-01-26T23:45:36.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.406+0000 7fd253c59700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c0cf400 con 0x7fd24c0cbff0 2024-01-26T23:45:36.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.406+0000 7fd2519f5700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd24c15ce40 0x7fd24c0d0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53514/0 (socket says 172.21.15.46:53514) 2024-01-26T23:45:36.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.406+0000 7fd2519f5700 1 -- 172.21.15.46:0/1645981781 learned_addr learned my addr 172.21.15.46:0/1645981781 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:36.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.406+0000 7fd2511f4700 1 --2- 172.21.15.46:0/1645981781 >> v2:172.21.15.106:3300/0 conn(0x7fd24c15d210 0x7fd24c0cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.406+0000 7fd2521f6700 1 --2- 172.21.15.46:0/1645981781 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd24c0cbff0 0x7fd24c0ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.406+0000 7fd2509f3700 1 -- 172.21.15.46:0/1645981781 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273782505 0 0) 0x7fd24c0cff70 con 0x7fd24c15ce40 2024-01-26T23:45:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.406+0000 7fd2509f3700 1 -- 172.21.15.46:0/1645981781 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd234003570 con 0x7fd24c15ce40 2024-01-26T23:45:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.406+0000 7fd2511f4700 1 -- 172.21.15.46:0/1645981781 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd24c0cbff0 msgr2=0x7fd24c0ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.406+0000 7fd2511f4700 1 --2- 172.21.15.46:0/1645981781 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd24c0cbff0 0x7fd24c0ade30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.406+0000 7fd2511f4700 1 -- 172.21.15.46:0/1645981781 >> v1:172.21.15.46:6789/0 conn(0x7fd24c15ce40 legacy=0x7fd24c0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.406+0000 7fd2511f4700 1 -- 172.21.15.46:0/1645981781 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c0cf660 con 0x7fd24c15d210 2024-01-26T23:45:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.407+0000 7fd2511f4700 1 --2- 172.21.15.46:0/1645981781 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd24c15d210 0x7fd24c0cbab0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd2400098e0 tx=0x7fd2400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44ee52ee61551019 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.407+0000 7fd2509f3700 1 -- 172.21.15.46:0/1645981781 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240011630 con 0x7fd24c15d210 2024-01-26T23:45:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.407+0000 7fd2509f3700 1 -- 172.21.15.46:0/1645981781 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd240011790 con 0x7fd24c15d210 2024-01-26T23:45:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.407+0000 7fd2509f3700 1 -- 172.21.15.46:0/1645981781 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240011b30 con 0x7fd24c15d210 2024-01-26T23:45:36.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.408+0000 7fd253c59700 1 -- 172.21.15.46:0/1645981781 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd24c15d210 msgr2=0x7fd24c0cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:36.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.408+0000 7fd253c59700 1 --2- 172.21.15.46:0/1645981781 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd24c15d210 0x7fd24c0cbab0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd2400098e0 tx=0x7fd2400091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:36.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.408+0000 7fd253c59700 1 -- 172.21.15.46:0/1645981781 shutdown_connections 2024-01-26T23:45:36.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.408+0000 7fd253c59700 1 --2- 172.21.15.46:0/1645981781 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd24c0cbff0 0x7fd24c0ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:36.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.408+0000 7fd253c59700 1 --2- 172.21.15.46:0/1645981781 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd24c15d210 0x7fd24c0cbab0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:36.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.408+0000 7fd253c59700 1 -- 172.21.15.46:0/1645981781 >> 172.21.15.46:0/1645981781 conn(0x7fd24c06f840 msgr2=0x7fd24c0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:36.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.408+0000 7fd253c59700 1 -- 172.21.15.46:0/1645981781 shutdown_connections 2024-01-26T23:45:36.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.408+0000 7fd253c59700 1 -- 172.21.15.46:0/1645981781 wait complete. 2024-01-26T23:45:36.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.409+0000 7fd253c59700 1 Processor -- start 2024-01-26T23:45:36.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.410+0000 7fd253c59700 1 -- start start 2024-01-26T23:45:36.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd253c59700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd24c0cbff0 0x7fd24c0c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:36.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd253c59700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd24c15ce40 0x7fd24c1698b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:36.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd253c59700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd24c15d210 0x7fd24c0cb100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:36.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd253c59700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c0c7d30 con 0x7fd24c15ce40 2024-01-26T23:45:36.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd253c59700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c0c8010 con 0x7fd24c15d210 2024-01-26T23:45:36.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd253c59700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c0c82f0 con 0x7fd24c0cbff0 2024-01-26T23:45:36.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd2511f4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd24c15ce40 0x7fd24c1698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:36.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd2519f5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd24c0cbff0 0x7fd24c0c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:36.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd2511f4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd24c15ce40 0x7fd24c1698b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50586/0 (socket says 172.21.15.46:50586) 2024-01-26T23:45:36.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd2519f5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd24c0cbff0 0x7fd24c0c7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:38570/0 (socket says 172.21.15.46:38570) 2024-01-26T23:45:36.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd2511f4700 1 -- 172.21.15.46:0/3014221799 learned_addr learned my addr 172.21.15.46:0/3014221799 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:36.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd2521f6700 1 --2- 172.21.15.46:0/3014221799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd24c15d210 0x7fd24c0cb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:36.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd2511f4700 1 -- 172.21.15.46:0/3014221799 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd24c0cbff0 msgr2=0x7fd24c0c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:36.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd2511f4700 1 --2- 172.21.15.46:0/3014221799 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd24c0cbff0 0x7fd24c0c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:36.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd2511f4700 1 -- 172.21.15.46:0/3014221799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd24c15d210 msgr2=0x7fd24c0cb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:36.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd2511f4700 1 --2- 172.21.15.46:0/3014221799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd24c15d210 0x7fd24c0cb100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:36.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd2511f4700 1 -- 172.21.15.46:0/3014221799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd240009420 con 0x7fd24c15ce40 2024-01-26T23:45:36.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd2511f4700 1 --2- 172.21.15.46:0/3014221799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd24c15ce40 0x7fd24c1698b0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd240013000 tx=0x7fd240017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:36.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.411+0000 7fd249ffb700 1 -- 172.21.15.46:0/3014221799 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240011900 con 0x7fd24c15ce40 2024-01-26T23:45:36.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.412+0000 7fd249ffb700 1 -- 172.21.15.46:0/3014221799 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd240011a80 con 0x7fd24c15ce40 2024-01-26T23:45:36.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.412+0000 7fd253c59700 1 -- 172.21.15.46:0/3014221799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24c0c85d0 con 0x7fd24c15ce40 2024-01-26T23:45:36.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.412+0000 7fd249ffb700 1 -- 172.21.15.46:0/3014221799 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240025640 con 0x7fd24c15ce40 2024-01-26T23:45:36.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.412+0000 7fd253c59700 1 -- 172.21.15.46:0/3014221799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd24c169df0 con 0x7fd24c15ce40 2024-01-26T23:45:36.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.413+0000 7fd253c59700 1 -- 172.21.15.46:0/3014221799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd24c0c8ad0 con 0x7fd24c15ce40 2024-01-26T23:45:36.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.413+0000 7fd249ffb700 1 -- 172.21.15.46:0/3014221799 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fd2400259f0 con 0x7fd24c15ce40 2024-01-26T23:45:36.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.414+0000 7fd249ffb700 1 --2- 172.21.15.46:0/3014221799 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd238067050 0x7fd238069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:36.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.414+0000 7fd249ffb700 1 -- 172.21.15.46:0/3014221799 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fd240086020 con 0x7fd24c15ce40 2024-01-26T23:45:36.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.414+0000 7fd2519f5700 1 -- 172.21.15.46:0/3014221799 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd238067050 msgr2=0x7fd238069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:36.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.414+0000 7fd2519f5700 1 --2- 172.21.15.46:0/3014221799 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd238067050 0x7fd238069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:36.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.418+0000 7fd249ffb700 1 -- 172.21.15.46:0/3014221799 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd240052db0 con 0x7fd24c15ce40 2024-01-26T23:45:36.435 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:36.434+0000 7f70091c51c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:45:36.436 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:36.436+0000 7f023aa4e1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:45:36.511 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:36.511+0000 7f023aa4e1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:45:36.590 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:36.589+0000 7f70091c51c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:45:36.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.615+0000 7fd2519f5700 1 -- 172.21.15.46:0/3014221799 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd238067050 msgr2=0x7fd238069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:36.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.615+0000 7fd2519f5700 1 --2- 172.21.15.46:0/3014221799 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd238067050 0x7fd238069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:36.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.675+0000 7fd253c59700 1 -- 172.21.15.46:0/3014221799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd24c04c710 con 0x7fd24c15ce40 2024-01-26T23:45:36.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.675+0000 7fd249ffb700 1 -- 172.21.15.46:0/3014221799 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4604 (secure 0 0 0) 0x7fd2400563d0 con 0x7fd24c15ce40 2024-01-26T23:45:36.676 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:36.676 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/4083796808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":4083796808},{"type":"v1","addr":"172.21.15.106:6836","nonce":4083796808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":6172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/1769644590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1769644590},{"type":"v1","addr":"172.21.15.106:6837","nonce":1769644590}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":6184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1994423361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1994423361},{"type":"v1","addr":"172.21.15.46:6835","nonce":1994423361}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:45:19.187256+0000","modified":"2024-01-26T23:45:22.777939+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6366":{"gid":6366,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/1799649681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1799649681},{"type":"v1","addr":"172.21.15.46:6834","nonce":1799649681}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2024-01-26T23:45:36.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.679+0000 7fd253c59700 1 -- 172.21.15.46:0/3014221799 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd238067050 msgr2=0x7fd238069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:36.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.679+0000 7fd253c59700 1 --2- 172.21.15.46:0/3014221799 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd238067050 0x7fd238069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:36.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.679+0000 7fd253c59700 1 -- 172.21.15.46:0/3014221799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd24c15ce40 msgr2=0x7fd24c1698b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:36.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.679+0000 7fd253c59700 1 --2- 172.21.15.46:0/3014221799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd24c15ce40 0x7fd24c1698b0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd240013000 tx=0x7fd240017000 comp rx=0 tx=0).stop 2024-01-26T23:45:36.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.679+0000 7fd253c59700 1 -- 172.21.15.46:0/3014221799 shutdown_connections 2024-01-26T23:45:36.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.679+0000 7fd253c59700 1 --2- 172.21.15.46:0/3014221799 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd238067050 0x7fd238069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:36.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.679+0000 7fd253c59700 1 --2- 172.21.15.46:0/3014221799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd24c15d210 0x7fd24c0cb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:36.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.679+0000 7fd253c59700 1 --2- 172.21.15.46:0/3014221799 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd24c0cbff0 0x7fd24c0c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:36.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.679+0000 7fd253c59700 1 --2- 172.21.15.46:0/3014221799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd24c15ce40 0x7fd24c1698b0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:36.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.679+0000 7fd253c59700 1 -- 172.21.15.46:0/3014221799 >> 172.21.15.46:0/3014221799 conn(0x7fd24c06f840 msgr2=0x7fd24c15bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:36.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.679+0000 7fd253c59700 1 -- 172.21.15.46:0/3014221799 shutdown_connections 2024-01-26T23:45:36.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.679+0000 7fd253c59700 1 -- 172.21.15.46:0/3014221799 wait complete. 2024-01-26T23:45:36.681 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 36 2024-01-26T23:45:36.697 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:36.697+0000 7f023aa4e1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:45:36.699 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:45:36.729 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:36.728+0000 7f70091c51c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:45:36.796 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:36.795+0000 7f70091c51c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:45:36.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.834+0000 7ff485ddb700 1 Processor -- start 2024-01-26T23:45:36.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.834+0000 7ff485ddb700 1 -- start start 2024-01-26T23:45:36.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.834+0000 7ff485ddb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff4800a8d10 0x7ff4800af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:36.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.834+0000 7ff485ddb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4800a90e0 0x7ff4800afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:36.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.834+0000 7ff485ddb700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4800b1c40 con 0x7ff480069910 2024-01-26T23:45:36.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.834+0000 7ff485ddb700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff480068d80 con 0x7ff4800a8d10 2024-01-26T23:45:36.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.834+0000 7ff485ddb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4800b21e0 con 0x7ff4800a90e0 2024-01-26T23:45:36.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.835+0000 7ff47f7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff4800a8d10 0x7ff4800af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:36.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.835+0000 7ff47ffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff480069910 0x7ff480069d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53526/0 (socket says 172.21.15.46:53526) 2024-01-26T23:45:36.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.835+0000 7ff47ffff700 1 -- 172.21.15.46:0/4128401630 learned_addr learned my addr 172.21.15.46:0/4128401630 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:36.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.835+0000 7ff47effd700 1 --2- 172.21.15.46:0/4128401630 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4800a90e0 0x7ff4800afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:36.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.835+0000 7ff47e7fc700 1 -- 172.21.15.46:0/4128401630 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501141536 0 0) 0x7ff4800b1c40 con 0x7ff480069910 2024-01-26T23:45:36.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.835+0000 7ff47e7fc700 1 -- 172.21.15.46:0/4128401630 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff468003570 con 0x7ff480069910 2024-01-26T23:45:36.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.835+0000 7ff47effd700 1 -- 172.21.15.46:0/4128401630 >> v2:172.21.15.46:3300/0 conn(0x7ff4800a8d10 msgr2=0x7ff4800af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:36.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.835+0000 7ff47effd700 1 --2- 172.21.15.46:0/4128401630 >> v2:172.21.15.46:3300/0 conn(0x7ff4800a8d10 0x7ff4800af7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:36.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.835+0000 7ff47effd700 1 -- 172.21.15.46:0/4128401630 >> v1:172.21.15.46:6789/0 conn(0x7ff480069910 legacy=0x7ff480069d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:36.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.835+0000 7ff47effd700 1 -- 172.21.15.46:0/4128401630 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48006a540 con 0x7ff4800a90e0 2024-01-26T23:45:36.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.835+0000 7ff47f7fe700 1 --2- 172.21.15.46:0/4128401630 >> v2:172.21.15.46:3300/0 conn(0x7ff4800a8d10 0x7ff4800af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:36.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.835+0000 7ff47effd700 1 --2- 172.21.15.46:0/4128401630 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4800a90e0 0x7ff4800afd20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff4740098d0 tx=0x7ff474009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b39439e2c28884 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:36.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.835+0000 7ff47e7fc700 1 -- 172.21.15.46:0/4128401630 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff474005c10 con 0x7ff4800a90e0 2024-01-26T23:45:36.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.835+0000 7ff47e7fc700 1 -- 172.21.15.46:0/4128401630 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff474005d70 con 0x7ff4800a90e0 2024-01-26T23:45:36.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.835+0000 7ff47e7fc700 1 -- 172.21.15.46:0/4128401630 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff474006240 con 0x7ff4800a90e0 2024-01-26T23:45:36.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.836+0000 7ff485ddb700 1 -- 172.21.15.46:0/4128401630 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4800a90e0 msgr2=0x7ff4800afd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:36.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.836+0000 7ff485ddb700 1 --2- 172.21.15.46:0/4128401630 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4800a90e0 0x7ff4800afd20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff4740098d0 tx=0x7ff474009120 comp rx=0 tx=0).stop 2024-01-26T23:45:36.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.836+0000 7ff485ddb700 1 -- 172.21.15.46:0/4128401630 shutdown_connections 2024-01-26T23:45:36.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.836+0000 7ff485ddb700 1 --2- 172.21.15.46:0/4128401630 >> v2:172.21.15.46:3300/0 conn(0x7ff4800a8d10 0x7ff4800af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:36.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.836+0000 7ff485ddb700 1 --2- 172.21.15.46:0/4128401630 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4800a90e0 0x7ff4800afd20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:36.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.836+0000 7ff485ddb700 1 -- 172.21.15.46:0/4128401630 >> 172.21.15.46:0/4128401630 conn(0x7ff480087f60 msgr2=0x7ff480088330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:36.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.836+0000 7ff485ddb700 1 -- 172.21.15.46:0/4128401630 shutdown_connections 2024-01-26T23:45:36.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.836+0000 7ff485ddb700 1 -- 172.21.15.46:0/4128401630 wait complete. 2024-01-26T23:45:36.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.838+0000 7ff485ddb700 1 Processor -- start 2024-01-26T23:45:36.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.838+0000 7ff485ddb700 1 -- start start 2024-01-26T23:45:36.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff485ddb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff480069910 0x7ff4800cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:36.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff485ddb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4800a8d10 0x7ff480169e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:36.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff485ddb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4800a90e0 0x7ff480098c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:36.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff485ddb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4800c3520 con 0x7ff4800a90e0 2024-01-26T23:45:36.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff485ddb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4800cd8f0 con 0x7ff480069910 2024-01-26T23:45:36.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff485ddb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff48009ac90 con 0x7ff4800a8d10 2024-01-26T23:45:36.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff47ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4800a90e0 0x7ff480098c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:36.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff47effd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4800a8d10 0x7ff480169e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:36.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff47ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4800a90e0 0x7ff480098c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50600/0 (socket says 172.21.15.46:50600) 2024-01-26T23:45:36.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff47f7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff480069910 0x7ff4800cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:36.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff47ffff700 1 -- 172.21.15.46:0/3740503726 learned_addr learned my addr 172.21.15.46:0/3740503726 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:36.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff47effd700 1 -- 172.21.15.46:0/3740503726 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff480069910 msgr2=0x7ff4800cd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff47effd700 1 --2- 172.21.15.46:0/3740503726 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff480069910 0x7ff4800cd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff47effd700 1 -- 172.21.15.46:0/3740503726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4800a90e0 msgr2=0x7ff480098c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff47effd700 1 --2- 172.21.15.46:0/3740503726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4800a90e0 0x7ff480098c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff47effd700 1 -- 172.21.15.46:0/3740503726 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff470009420 con 0x7ff4800a8d10 2024-01-26T23:45:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff47ffff700 1 --2- 172.21.15.46:0/3740503726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4800a90e0 0x7ff480098c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.839+0000 7ff47effd700 1 --2- 172.21.15.46:0/3740503726 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4800a8d10 0x7ff480169e00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff474015000 tx=0x7ff474016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.840+0000 7ff4677fe700 1 -- 172.21.15.46:0/3740503726 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff474005c10 con 0x7ff4800a8d10 2024-01-26T23:45:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.840+0000 7ff4677fe700 1 -- 172.21.15.46:0/3740503726 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47401a630 con 0x7ff4800a8d10 2024-01-26T23:45:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.840+0000 7ff485ddb700 1 -- 172.21.15.46:0/3740503726 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4740093c0 con 0x7ff4800a8d10 2024-01-26T23:45:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.840+0000 7ff4677fe700 1 -- 172.21.15.46:0/3740503726 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47401a9f0 con 0x7ff4800a8d10 2024-01-26T23:45:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.840+0000 7ff485ddb700 1 -- 172.21.15.46:0/3740503726 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4800981a0 con 0x7ff4800a8d10 2024-01-26T23:45:36.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.841+0000 7ff485ddb700 1 -- 172.21.15.46:0/3740503726 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff48004c710 con 0x7ff4800a8d10 2024-01-26T23:45:36.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.841+0000 7ff4677fe700 1 -- 172.21.15.46:0/3740503726 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7ff474011420 con 0x7ff4800a8d10 2024-01-26T23:45:36.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.842+0000 7ff4677fe700 1 --2- 172.21.15.46:0/3740503726 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff46c067050 0x7ff46c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:36.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.842+0000 7ff47f7fe700 1 -- 172.21.15.46:0/3740503726 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff46c067050 msgr2=0x7ff46c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:36.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.842+0000 7ff4677fe700 1 -- 172.21.15.46:0/3740503726 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6675+0+0 (secure 0 0 0) 0x7ff474086050 con 0x7ff4800a8d10 2024-01-26T23:45:36.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.842+0000 7ff47f7fe700 1 --2- 172.21.15.46:0/3740503726 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff46c067050 0x7ff46c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:36.850+0000 7ff4677fe700 1 -- 172.21.15.46:0/3740503726 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff474052e60 con 0x7ff4800a8d10 2024-01-26T23:45:36.859 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:36.859+0000 7f023aa4e1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:45:36.922 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:36.921+0000 7f70091c51c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:45:36.937 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:36.936+0000 7f023aa4e1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:45:36.994 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:36.993+0000 7f70091c51c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:45:37.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.043+0000 7ff47f7fe700 1 -- 172.21.15.46:0/3740503726 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff46c067050 msgr2=0x7ff46c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:37.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.043+0000 7ff47f7fe700 1 --2- 172.21.15.46:0/3740503726 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff46c067050 0x7ff46c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:37.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.061+0000 7ff485ddb700 1 -- 172.21.15.46:0/3740503726 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff48006a820 con 0x7ff4800a8d10 2024-01-26T23:45:37.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.063+0000 7ff4677fe700 1 -- 172.21.15.46:0/3740503726 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16524 (secure 0 0 0) 0x7ff47402b090 con 0x7ff4800a8d10 2024-01-26T23:45:37.063 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:37.063 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":86,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:45:33.820805+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:45:17.703325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-01-26T23:45:18.149282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/2697231774":"2024-01-27T23:45:33.820757+0000","172.21.15.106:0/2527421951":"2024-01-27T23:45:33.820757+0000","172.21.15.106:0/2110014618":"2024-01-27T23:45:33.820757+0000","172.21.15.106:6833/43388":"2024-01-27T23:45:33.820757+0000","172.21.15.106:6832/43388":"2024-01-27T23:45:33.820757+0000","172.21.15.106:0/2931190795":"2024-01-27T23:45:33.820757+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:45:37.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.065+0000 7ff485ddb700 1 -- 172.21.15.46:0/3740503726 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff46c067050 msgr2=0x7ff46c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:37.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.065+0000 7ff485ddb700 1 --2- 172.21.15.46:0/3740503726 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff46c067050 0x7ff46c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.065+0000 7ff485ddb700 1 -- 172.21.15.46:0/3740503726 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4800a8d10 msgr2=0x7ff480169e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:37.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.065+0000 7ff485ddb700 1 --2- 172.21.15.46:0/3740503726 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4800a8d10 0x7ff480169e00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff474015000 tx=0x7ff474016000 comp rx=0 tx=0).stop 2024-01-26T23:45:37.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.065+0000 7ff485ddb700 1 -- 172.21.15.46:0/3740503726 shutdown_connections 2024-01-26T23:45:37.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.065+0000 7ff485ddb700 1 --2- 172.21.15.46:0/3740503726 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff46c067050 0x7ff46c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.065+0000 7ff485ddb700 1 --2- 172.21.15.46:0/3740503726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4800a90e0 0x7ff480098c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.065+0000 7ff485ddb700 1 --2- 172.21.15.46:0/3740503726 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff480069910 0x7ff4800cd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.065+0000 7ff485ddb700 1 --2- 172.21.15.46:0/3740503726 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4800a8d10 0x7ff480169e00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.065+0000 7ff485ddb700 1 -- 172.21.15.46:0/3740503726 >> 172.21.15.46:0/3740503726 conn(0x7ff480087f60 msgr2=0x7ff480158f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:37.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.065+0000 7ff485ddb700 1 -- 172.21.15.46:0/3740503726 shutdown_connections 2024-01-26T23:45:37.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.065+0000 7ff485ddb700 1 -- 172.21.15.46:0/3740503726 wait complete. 2024-01-26T23:45:37.076 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:37.076+0000 7f023aa4e1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:45:37.080 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-26T23:45:17.703325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-01-26T23:45:18.149282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-26T23:45:37.080 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:45:37.135 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:37.134+0000 7f70091c51c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:45:37.164 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:37.164+0000 7f023aa4e1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:45:37.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.217+0000 7fc1e6870700 1 Processor -- start 2024-01-26T23:45:37.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.217+0000 7fc1e6870700 1 -- start start 2024-01-26T23:45:37.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.218+0000 7fc1e6870700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc1e0159510 0x7fc1e01598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:37.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.218+0000 7fc1e6870700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1e00ca690 0x7fc1e00cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:37.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.218+0000 7fc1e6870700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e0069260 con 0x7fc1e00ca220 2024-01-26T23:45:37.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.218+0000 7fc1e6870700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e015d670 con 0x7fc1e00ca690 2024-01-26T23:45:37.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.218+0000 7fc1e6870700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e00c9750 con 0x7fc1e0159510 2024-01-26T23:45:37.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.218+0000 7fc1e4e0d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1e00ca690 0x7fc1e00cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:37.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.218+0000 7fc1dffff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc1e0159510 0x7fc1e01598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:37.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.218+0000 7fc1e4e0d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1e00ca690 0x7fc1e00cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50602/0 (socket says 172.21.15.46:50602) 2024-01-26T23:45:37.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.218+0000 7fc1e4e0d700 1 -- 172.21.15.46:0/164599658 learned_addr learned my addr 172.21.15.46:0/164599658 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:37.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.219+0000 7fc1dffff700 1 -- 172.21.15.46:0/164599658 >> v2:172.21.15.46:3300/0 conn(0x7fc1e00ca690 msgr2=0x7fc1e00cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:37.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.219+0000 7fc1dffff700 1 --2- 172.21.15.46:0/164599658 >> v2:172.21.15.46:3300/0 conn(0x7fc1e00ca690 0x7fc1e00cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.219+0000 7fc1dffff700 1 -- 172.21.15.46:0/164599658 >> v1:172.21.15.106:6789/0 conn(0x7fc1e00ca220 legacy=0x7fc1e00c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:37.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.219+0000 7fc1dffff700 1 -- 172.21.15.46:0/164599658 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e00c9a30 con 0x7fc1e0159510 2024-01-26T23:45:37.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.219+0000 7fc1dffff700 1 --2- 172.21.15.46:0/164599658 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1e0159510 0x7fc1e01598e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc1e006f760 tx=0x7fc1d00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e8ff229fe89ec54 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:37.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.219+0000 7fc1deffd700 1 -- 172.21.15.46:0/164599658 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d0011630 con 0x7fc1e0159510 2024-01-26T23:45:37.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.219+0000 7fc1deffd700 1 -- 172.21.15.46:0/164599658 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d0011790 con 0x7fc1e0159510 2024-01-26T23:45:37.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.219+0000 7fc1deffd700 1 -- 172.21.15.46:0/164599658 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d0011b30 con 0x7fc1e0159510 2024-01-26T23:45:37.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.220+0000 7fc1e6870700 1 -- 172.21.15.46:0/164599658 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1e0159510 msgr2=0x7fc1e01598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:37.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.220+0000 7fc1e6870700 1 --2- 172.21.15.46:0/164599658 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1e0159510 0x7fc1e01598e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc1e006f760 tx=0x7fc1d00091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.220+0000 7fc1e6870700 1 -- 172.21.15.46:0/164599658 shutdown_connections 2024-01-26T23:45:37.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.220+0000 7fc1e6870700 1 --2- 172.21.15.46:0/164599658 >> v2:172.21.15.46:3300/0 conn(0x7fc1e00ca690 0x7fc1e00cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.220+0000 7fc1e6870700 1 --2- 172.21.15.46:0/164599658 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1e0159510 0x7fc1e01598e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.220+0000 7fc1e6870700 1 -- 172.21.15.46:0/164599658 >> 172.21.15.46:0/164599658 conn(0x7fc1e006bd90 msgr2=0x7fc1e006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:37.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.220+0000 7fc1e6870700 1 -- 172.21.15.46:0/164599658 shutdown_connections 2024-01-26T23:45:37.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.220+0000 7fc1e6870700 1 -- 172.21.15.46:0/164599658 wait complete. 2024-01-26T23:45:37.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.221+0000 7fc1e6870700 1 Processor -- start 2024-01-26T23:45:37.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.221+0000 7fc1e6870700 1 -- start start 2024-01-26T23:45:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.222+0000 7fc1e6870700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1e00ca220 0x7fc1e014c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.222+0000 7fc1e6870700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1e0159510 0x7fc1e014fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.222+0000 7fc1e6870700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1e014cd10 0x7fc1e00b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.222+0000 7fc1e6870700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e014fa40 con 0x7fc1e014cd10 2024-01-26T23:45:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.222+0000 7fc1e6870700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e014ee70 con 0x7fc1e00ca220 2024-01-26T23:45:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.222+0000 7fc1e6870700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e014f110 con 0x7fc1e0159510 2024-01-26T23:45:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.222+0000 7fc1e4e0d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1e014cd10 0x7fc1e00b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.222+0000 7fc1dffff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1e00ca220 0x7fc1e014c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.222+0000 7fc1dffff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1e00ca220 0x7fc1e014c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:59938/0 (socket says 172.21.15.46:59938) 2024-01-26T23:45:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.222+0000 7fc1dffff700 1 -- 172.21.15.46:0/2276824567 learned_addr learned my addr 172.21.15.46:0/2276824567 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.222+0000 7fc1df7fe700 1 --2- 172.21.15.46:0/2276824567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1e0159510 0x7fc1e014fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.223+0000 7fc1df7fe700 1 -- 172.21.15.46:0/2276824567 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1e00ca220 msgr2=0x7fc1e014c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.223+0000 7fc1df7fe700 1 --2- 172.21.15.46:0/2276824567 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1e00ca220 0x7fc1e014c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.223+0000 7fc1df7fe700 1 -- 172.21.15.46:0/2276824567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1e014cd10 msgr2=0x7fc1e00b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.223+0000 7fc1df7fe700 1 --2- 172.21.15.46:0/2276824567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1e014cd10 0x7fc1e00b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.223+0000 7fc1df7fe700 1 -- 172.21.15.46:0/2276824567 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d0009420 con 0x7fc1e0159510 2024-01-26T23:45:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.223+0000 7fc1dffff700 1 --2- 172.21.15.46:0/2276824567 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1e00ca220 0x7fc1e014c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.223+0000 7fc1df7fe700 1 --2- 172.21.15.46:0/2276824567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1e0159510 0x7fc1e014fed0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc1d400c590 tx=0x7fc1d4007ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.223+0000 7fc1c7fff700 1 -- 172.21.15.46:0/2276824567 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4011590 con 0x7fc1e0159510 2024-01-26T23:45:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.223+0000 7fc1c7fff700 1 -- 172.21.15.46:0/2276824567 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d4011710 con 0x7fc1e0159510 2024-01-26T23:45:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.223+0000 7fc1c7fff700 1 -- 172.21.15.46:0/2276824567 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4011ad0 con 0x7fc1e0159510 2024-01-26T23:45:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.223+0000 7fc1e6870700 1 -- 172.21.15.46:0/2276824567 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e014f3f0 con 0x7fc1e0159510 2024-01-26T23:45:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.223+0000 7fc1e6870700 1 -- 172.21.15.46:0/2276824567 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e00b3230 con 0x7fc1e0159510 2024-01-26T23:45:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.224+0000 7fc1e6870700 1 -- 172.21.15.46:0/2276824567 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e00b3450 con 0x7fc1e0159510 2024-01-26T23:45:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.225+0000 7fc1c7fff700 1 -- 172.21.15.46:0/2276824567 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fc1d401b3b0 con 0x7fc1e0159510 2024-01-26T23:45:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.225+0000 7fc1c7fff700 1 --2- 172.21.15.46:0/2276824567 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc1cc067080 0x7fc1cc069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.225+0000 7fc1dffff700 1 -- 172.21.15.46:0/2276824567 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc1cc067080 msgr2=0x7fc1cc069530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.225+0000 7fc1c7fff700 1 -- 172.21.15.46:0/2276824567 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fc1d4083f00 con 0x7fc1e0159510 2024-01-26T23:45:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.225+0000 7fc1dffff700 1 --2- 172.21.15.46:0/2276824567 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc1cc067080 0x7fc1cc069530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:37.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.233+0000 7fc1c7fff700 1 -- 172.21.15.46:0/2276824567 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc1d4050bf0 con 0x7fc1e0159510 2024-01-26T23:45:37.266 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:37.265+0000 7f70091c51c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:45:37.341 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:37.341+0000 7f023aa4e1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:45:37.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.426+0000 7fc1dffff700 1 -- 172.21.15.46:0/2276824567 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc1cc067080 msgr2=0x7fc1cc069530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:37.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.426+0000 7fc1dffff700 1 --2- 172.21.15.46:0/2276824567 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc1cc067080 0x7fc1cc069530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:37.483 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:37.482+0000 7f023aa4e1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:45:37.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.486+0000 7fc1e6870700 1 -- 172.21.15.46:0/2276824567 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc1e004c710 con 0x7fc1e0159510 2024-01-26T23:45:37.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.487+0000 7fc1c7fff700 1 -- 172.21.15.46:0/2276824567 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc1d4017070 con 0x7fc1e0159510 2024-01-26T23:45:37.487 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:45:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.489+0000 7fc1e6870700 1 -- 172.21.15.46:0/2276824567 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc1cc067080 msgr2=0x7fc1cc069530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.489+0000 7fc1e6870700 1 --2- 172.21.15.46:0/2276824567 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc1cc067080 0x7fc1cc069530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.489+0000 7fc1e6870700 1 -- 172.21.15.46:0/2276824567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1e0159510 msgr2=0x7fc1e014fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.489+0000 7fc1e6870700 1 --2- 172.21.15.46:0/2276824567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1e0159510 0x7fc1e014fed0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc1d400c590 tx=0x7fc1d4007ec0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.489+0000 7fc1e6870700 1 -- 172.21.15.46:0/2276824567 shutdown_connections 2024-01-26T23:45:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.489+0000 7fc1e6870700 1 --2- 172.21.15.46:0/2276824567 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc1cc067080 0x7fc1cc069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.489+0000 7fc1e6870700 1 --2- 172.21.15.46:0/2276824567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1e014cd10 0x7fc1e00b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.489+0000 7fc1e6870700 1 --2- 172.21.15.46:0/2276824567 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1e00ca220 0x7fc1e014c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.489+0000 7fc1e6870700 1 --2- 172.21.15.46:0/2276824567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1e0159510 0x7fc1e014fed0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.489+0000 7fc1e6870700 1 -- 172.21.15.46:0/2276824567 >> 172.21.15.46:0/2276824567 conn(0x7fc1e006bd90 msgr2=0x7fc1e0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.489+0000 7fc1e6870700 1 -- 172.21.15.46:0/2276824567 shutdown_connections 2024-01-26T23:45:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.489+0000 7fc1e6870700 1 -- 172.21.15.46:0/2276824567 wait complete. 2024-01-26T23:45:37.503 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-26T23:45:37.578 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:37.577+0000 7f70091c51c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:45:37.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.639+0000 7f3f2c664700 1 Processor -- start 2024-01-26T23:45:37.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.639+0000 7f3f2c664700 1 -- start start 2024-01-26T23:45:37.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.640+0000 7f3f2c664700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f240c9770 0x7f3f24159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:37.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.640+0000 7f3f2c664700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3f24159760 0x7f3f240c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:37.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.640+0000 7f3f2c664700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2415e390 con 0x7f3f240c93a0 2024-01-26T23:45:37.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.640+0000 7f3f2c664700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f240c9be0 con 0x7f3f24159760 2024-01-26T23:45:37.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.640+0000 7f3f2c664700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f24068460 con 0x7f3f240c9770 2024-01-26T23:45:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.640+0000 7f3f29bff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f240c9770 0x7f3f24159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.640+0000 7f3f2a400700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f3f240c93a0 0x7f3f24067d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:57842/0 (socket says 172.21.15.46:57842) 2024-01-26T23:45:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.640+0000 7f3f2a400700 1 -- 172.21.15.46:0/4242482423 learned_addr learned my addr 172.21.15.46:0/4242482423 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.640+0000 7f3f2ac01700 1 --2- 172.21.15.46:0/4242482423 >> v2:172.21.15.106:3300/0 conn(0x7f3f24159760 0x7f3f240c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.641+0000 7f3f29bff700 1 -- 172.21.15.46:0/4242482423 >> v2:172.21.15.106:3300/0 conn(0x7f3f24159760 msgr2=0x7f3f240c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.641+0000 7f3f29bff700 1 --2- 172.21.15.46:0/4242482423 >> v2:172.21.15.106:3300/0 conn(0x7f3f24159760 0x7f3f240c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.641+0000 7f3f29bff700 1 -- 172.21.15.46:0/4242482423 >> v1:172.21.15.106:6789/0 conn(0x7f3f240c93a0 legacy=0x7f3f24067d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.641+0000 7f3f293fe700 1 -- 172.21.15.46:0/4242482423 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3158494658 0 0) 0x7f3f2415e390 con 0x7f3f240c93a0 2024-01-26T23:45:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.641+0000 7f3f29bff700 1 -- 172.21.15.46:0/4242482423 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2415d810 con 0x7f3f240c9770 2024-01-26T23:45:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.641+0000 7f3f29bff700 1 --2- 172.21.15.46:0/4242482423 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f240c9770 0x7f3f24159220 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3f140098e0 tx=0x7f3f140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=557ecb9598d5b802 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.641+0000 7f3f293fe700 1 -- 172.21.15.46:0/4242482423 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1400b640 con 0x7f3f240c9770 2024-01-26T23:45:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.641+0000 7f3f293fe700 1 -- 172.21.15.46:0/4242482423 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1400b7a0 con 0x7f3f240c9770 2024-01-26T23:45:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.641+0000 7f3f293fe700 1 -- 172.21.15.46:0/4242482423 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1400bb40 con 0x7f3f240c9770 2024-01-26T23:45:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.642+0000 7f3f2c664700 1 -- 172.21.15.46:0/4242482423 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f240c9770 msgr2=0x7f3f24159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.642+0000 7f3f2c664700 1 --2- 172.21.15.46:0/4242482423 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f240c9770 0x7f3f24159220 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3f140098e0 tx=0x7f3f140091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.642+0000 7f3f2c664700 1 -- 172.21.15.46:0/4242482423 shutdown_connections 2024-01-26T23:45:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.642+0000 7f3f2c664700 1 --2- 172.21.15.46:0/4242482423 >> v2:172.21.15.106:3300/0 conn(0x7f3f24159760 0x7f3f240c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.642+0000 7f3f2c664700 1 --2- 172.21.15.46:0/4242482423 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f240c9770 0x7f3f24159220 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.642+0000 7f3f2c664700 1 -- 172.21.15.46:0/4242482423 >> 172.21.15.46:0/4242482423 conn(0x7f3f240ca9a0 msgr2=0x7f3f24068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.642+0000 7f3f2c664700 1 -- 172.21.15.46:0/4242482423 shutdown_connections 2024-01-26T23:45:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.642+0000 7f3f2c664700 1 -- 172.21.15.46:0/4242482423 wait complete. 2024-01-26T23:45:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.644+0000 7f3f2c664700 1 Processor -- start 2024-01-26T23:45:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.644+0000 7f3f2c664700 1 -- start start 2024-01-26T23:45:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.645+0000 7f3f2c664700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f240c93a0 0x7f3f24143a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.645+0000 7f3f2c664700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f240c9770 0x7f3f24159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.645+0000 7f3f2c664700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f24159760 0x7f3f2415a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.645+0000 7f3f2c664700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f240c5980 con 0x7f3f240c93a0 2024-01-26T23:45:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.645+0000 7f3f2c664700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2415ced0 con 0x7f3f24159760 2024-01-26T23:45:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.645+0000 7f3f2c664700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2415c910 con 0x7f3f240c9770 2024-01-26T23:45:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.645+0000 7f3f2a400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f240c93a0 0x7f3f24143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.645+0000 7f3f2a400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f240c93a0 0x7f3f24143a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50616/0 (socket says 172.21.15.46:50616) 2024-01-26T23:45:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.645+0000 7f3f29bff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f240c9770 0x7f3f24159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.645+0000 7f3f2a400700 1 -- 172.21.15.46:0/1870166798 learned_addr learned my addr 172.21.15.46:0/1870166798 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.645+0000 7f3f2ac01700 1 --2- 172.21.15.46:0/1870166798 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f24159760 0x7f3f2415a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.645+0000 7f3f29bff700 1 -- 172.21.15.46:0/1870166798 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f24159760 msgr2=0x7f3f2415a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.645+0000 7f3f29bff700 1 --2- 172.21.15.46:0/1870166798 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f24159760 0x7f3f2415a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.645+0000 7f3f29bff700 1 -- 172.21.15.46:0/1870166798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f240c93a0 msgr2=0x7f3f24143a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.645+0000 7f3f29bff700 1 --2- 172.21.15.46:0/1870166798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f240c93a0 0x7f3f24143a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.645+0000 7f3f29bff700 1 -- 172.21.15.46:0/1870166798 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f14009420 con 0x7f3f240c9770 2024-01-26T23:45:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.646+0000 7f3f29bff700 1 --2- 172.21.15.46:0/1870166798 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f240c9770 0x7f3f24159ba0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3f14016000 tx=0x7f3f14017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.646+0000 7f3f1a7fc700 1 -- 172.21.15.46:0/1870166798 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f14006af0 con 0x7f3f240c9770 2024-01-26T23:45:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.646+0000 7f3f1a7fc700 1 -- 172.21.15.46:0/1870166798 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1400b910 con 0x7f3f240c9770 2024-01-26T23:45:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.646+0000 7f3f2c664700 1 -- 172.21.15.46:0/1870166798 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2415cbf0 con 0x7f3f240c9770 2024-01-26T23:45:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.646+0000 7f3f1a7fc700 1 -- 172.21.15.46:0/1870166798 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f14011650 con 0x7f3f240c9770 2024-01-26T23:45:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.646+0000 7f3f2c664700 1 -- 172.21.15.46:0/1870166798 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2416aff0 con 0x7f3f240c9770 2024-01-26T23:45:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.647+0000 7f3f2c664700 1 -- 172.21.15.46:0/1870166798 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f240c5b30 con 0x7f3f240c9770 2024-01-26T23:45:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.647+0000 7f3f1a7fc700 1 -- 172.21.15.46:0/1870166798 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f3f14011a80 con 0x7f3f240c9770 2024-01-26T23:45:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.648+0000 7f3f1a7fc700 1 --2- 172.21.15.46:0/1870166798 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3f08067050 0x7f3f08069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.648+0000 7f3f2a400700 1 -- 172.21.15.46:0/1870166798 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3f08067050 msgr2=0x7f3f08069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.648+0000 7f3f1a7fc700 1 -- 172.21.15.46:0/1870166798 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f3f14086490 con 0x7f3f240c9770 2024-01-26T23:45:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.648+0000 7f3f2a400700 1 --2- 172.21.15.46:0/1870166798 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3f08067050 0x7f3f08069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:37.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.656+0000 7f3f1a7fc700 1 -- 172.21.15.46:0/1870166798 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3f240c5b30 con 0x7f3f240c9770 2024-01-26T23:45:37.793 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:37.793+0000 7f70091c51c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:45:37.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.849+0000 7f3f2a400700 1 -- 172.21.15.46:0/1870166798 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3f08067050 msgr2=0x7f3f08069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:37.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.849+0000 7f3f2a400700 1 --2- 172.21.15.46:0/1870166798 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3f08067050 0x7f3f08069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:37.855 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:37.855+0000 7f023aa4e1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:45:37.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.860+0000 7f3f2c664700 1 -- 172.21.15.46:0/1870166798 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3f240c5b30 con 0x7f3f240c9770 2024-01-26T23:45:37.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.861+0000 7f3f1a7fc700 1 -- 172.21.15.46:0/1870166798 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3f240c5b30 con 0x7f3f240c9770 2024-01-26T23:45:37.861 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:45:37.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.863+0000 7f3f2c664700 1 -- 172.21.15.46:0/1870166798 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3f08067050 msgr2=0x7f3f08069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:37.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.863+0000 7f3f2c664700 1 --2- 172.21.15.46:0/1870166798 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3f08067050 0x7f3f08069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.863+0000 7f3f2c664700 1 -- 172.21.15.46:0/1870166798 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f240c9770 msgr2=0x7f3f24159ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:37.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.863+0000 7f3f2c664700 1 --2- 172.21.15.46:0/1870166798 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f240c9770 0x7f3f24159ba0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3f14016000 tx=0x7f3f14017000 comp rx=0 tx=0).stop 2024-01-26T23:45:37.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.863+0000 7f3f2c664700 1 -- 172.21.15.46:0/1870166798 shutdown_connections 2024-01-26T23:45:37.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.863+0000 7f3f2c664700 1 --2- 172.21.15.46:0/1870166798 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3f08067050 0x7f3f08069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.863+0000 7f3f2c664700 1 --2- 172.21.15.46:0/1870166798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f240c93a0 0x7f3f24143a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.863+0000 7f3f2c664700 1 --2- 172.21.15.46:0/1870166798 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f24159760 0x7f3f2415a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.863+0000 7f3f2c664700 1 --2- 172.21.15.46:0/1870166798 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f240c9770 0x7f3f24159ba0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:37.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.863+0000 7f3f2c664700 1 -- 172.21.15.46:0/1870166798 >> 172.21.15.46:0/1870166798 conn(0x7f3f240ca9a0 msgr2=0x7f3f24155fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:37.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.864+0000 7f3f2c664700 1 -- 172.21.15.46:0/1870166798 shutdown_connections 2024-01-26T23:45:37.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:37.864+0000 7f3f2c664700 1 -- 172.21.15.46:0/1870166798 wait complete. 2024-01-26T23:45:37.878 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-26T23:45:38.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.014+0000 7f69c391c700 1 Processor -- start 2024-01-26T23:45:38.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.014+0000 7f69c391c700 1 -- start start 2024-01-26T23:45:38.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.015+0000 7f69c391c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69bc1599d0 0x7f69bc159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:38.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.015+0000 7f69c391c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f69bc0ab1c0 0x7f69bc0b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:38.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.015+0000 7f69c391c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69bc0b1a60 con 0x7f69bc15d8c0 2024-01-26T23:45:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.015+0000 7f69c391c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69bc0ab5f0 con 0x7f69bc0ab1c0 2024-01-26T23:45:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.015+0000 7f69c391c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc0b0f40 con 0x7f69bc1599d0 2024-01-26T23:45:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.015+0000 7f69c1eb9700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f69bc15d8c0 0x7f69bc0b20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53528/0 (socket says 172.21.15.46:53528) 2024-01-26T23:45:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.015+0000 7f69c1eb9700 1 -- 172.21.15.46:0/1259436562 learned_addr learned my addr 172.21.15.46:0/1259436562 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.015+0000 7f69c16b8700 1 --2- 172.21.15.46:0/1259436562 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69bc1599d0 0x7f69bc159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.015+0000 7f69c0eb7700 1 --2- 172.21.15.46:0/1259436562 >> v2:172.21.15.106:3300/0 conn(0x7f69bc0ab1c0 0x7f69bc0b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.016+0000 7f69b3fff700 1 -- 172.21.15.46:0/1259436562 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841044932 0 0) 0x7f69bc0b1a60 con 0x7f69bc15d8c0 2024-01-26T23:45:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.016+0000 7f69b3fff700 1 -- 172.21.15.46:0/1259436562 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69a8003570 con 0x7f69bc15d8c0 2024-01-26T23:45:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.016+0000 7f69c16b8700 1 -- 172.21.15.46:0/1259436562 >> v2:172.21.15.106:3300/0 conn(0x7f69bc0ab1c0 msgr2=0x7f69bc0b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.016+0000 7f69c16b8700 1 --2- 172.21.15.46:0/1259436562 >> v2:172.21.15.106:3300/0 conn(0x7f69bc0ab1c0 0x7f69bc0b0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.016+0000 7f69c16b8700 1 -- 172.21.15.46:0/1259436562 >> v1:172.21.15.46:6789/0 conn(0x7f69bc15d8c0 legacy=0x7f69bc0b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.016+0000 7f69c16b8700 1 -- 172.21.15.46:0/1259436562 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69bc0b11c0 con 0x7f69bc1599d0 2024-01-26T23:45:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.016+0000 7f69c16b8700 1 --2- 172.21.15.46:0/1259436562 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69bc1599d0 0x7f69bc159da0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f69b80098d0 tx=0x7f69b8009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e82553d234b8da3 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.016+0000 7f69b3fff700 1 -- 172.21.15.46:0/1259436562 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b8005c10 con 0x7f69bc1599d0 2024-01-26T23:45:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.016+0000 7f69b3fff700 1 -- 172.21.15.46:0/1259436562 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b8005d70 con 0x7f69bc1599d0 2024-01-26T23:45:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.016+0000 7f69b3fff700 1 -- 172.21.15.46:0/1259436562 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b8006260 con 0x7f69bc1599d0 2024-01-26T23:45:38.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.017+0000 7f69c391c700 1 -- 172.21.15.46:0/1259436562 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69bc1599d0 msgr2=0x7f69bc159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:38.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.017+0000 7f69c391c700 1 --2- 172.21.15.46:0/1259436562 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69bc1599d0 0x7f69bc159da0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f69b80098d0 tx=0x7f69b8009120 comp rx=0 tx=0).stop 2024-01-26T23:45:38.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.017+0000 7f69c391c700 1 -- 172.21.15.46:0/1259436562 shutdown_connections 2024-01-26T23:45:38.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.017+0000 7f69c391c700 1 --2- 172.21.15.46:0/1259436562 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69bc1599d0 0x7f69bc159da0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.017+0000 7f69c391c700 1 --2- 172.21.15.46:0/1259436562 >> v2:172.21.15.106:3300/0 conn(0x7f69bc0ab1c0 0x7f69bc0b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.017+0000 7f69c391c700 1 -- 172.21.15.46:0/1259436562 >> 172.21.15.46:0/1259436562 conn(0x7f69bc06f840 msgr2=0x7f69bc087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:38.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.017+0000 7f69c391c700 1 -- 172.21.15.46:0/1259436562 shutdown_connections 2024-01-26T23:45:38.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.017+0000 7f69c391c700 1 -- 172.21.15.46:0/1259436562 wait complete. 2024-01-26T23:45:38.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.018+0000 7f69c391c700 1 Processor -- start 2024-01-26T23:45:38.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.018+0000 7f69c391c700 1 -- start start 2024-01-26T23:45:38.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.019+0000 7f69c391c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69bc0ab1c0 0x7f69bc1589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:38.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.019+0000 7f69c391c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69bc1599d0 0x7f69bc0ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:38.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.019+0000 7f69c391c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69bc15d8c0 0x7f69bc0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:38.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.019+0000 7f69c391c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc169b30 con 0x7f69bc15d8c0 2024-01-26T23:45:38.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.019+0000 7f69c391c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc0aeda0 con 0x7f69bc0ab1c0 2024-01-26T23:45:38.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.019+0000 7f69c391c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc0af080 con 0x7f69bc1599d0 2024-01-26T23:45:38.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.019+0000 7f69c0eb7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69bc1599d0 0x7f69bc0ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:38.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.019+0000 7f69c1eb9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69bc15d8c0 0x7f69bc0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:38.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.019+0000 7f69c16b8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69bc0ab1c0 0x7f69bc1589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:38.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.019+0000 7f69c0eb7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69bc1599d0 0x7f69bc0ae2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:38624/0 (socket says 172.21.15.46:38624) 2024-01-26T23:45:38.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.019+0000 7f69c0eb7700 1 -- 172.21.15.46:0/1938225113 learned_addr learned my addr 172.21.15.46:0/1938225113 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:38.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.020+0000 7f69c0eb7700 1 -- 172.21.15.46:0/1938225113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69bc0ab1c0 msgr2=0x7f69bc1589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:38.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.020+0000 7f69c0eb7700 1 --2- 172.21.15.46:0/1938225113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69bc0ab1c0 0x7f69bc1589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.020+0000 7f69c0eb7700 1 -- 172.21.15.46:0/1938225113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69bc15d8c0 msgr2=0x7f69bc0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:38.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.020+0000 7f69c0eb7700 1 --2- 172.21.15.46:0/1938225113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69bc15d8c0 0x7f69bc0ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.020+0000 7f69c0eb7700 1 -- 172.21.15.46:0/1938225113 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b80093c0 con 0x7f69bc1599d0 2024-01-26T23:45:38.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.020+0000 7f69c16b8700 1 --2- 172.21.15.46:0/1938225113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69bc0ab1c0 0x7f69bc1589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:38.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.020+0000 7f69c0eb7700 1 --2- 172.21.15.46:0/1938225113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69bc1599d0 0x7f69bc0ae2d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00ec80 tx=0x7f69ac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:38.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.020+0000 7f69b17fa700 1 -- 172.21.15.46:0/1938225113 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac00c850 con 0x7f69bc1599d0 2024-01-26T23:45:38.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.020+0000 7f69b17fa700 1 -- 172.21.15.46:0/1938225113 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ac00c9b0 con 0x7f69bc1599d0 2024-01-26T23:45:38.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.020+0000 7f69c391c700 1 -- 172.21.15.46:0/1938225113 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69bc09a5d0 con 0x7f69bc1599d0 2024-01-26T23:45:38.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.020+0000 7f69b17fa700 1 -- 172.21.15.46:0/1938225113 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac00cd50 con 0x7f69bc1599d0 2024-01-26T23:45:38.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.020+0000 7f69c391c700 1 -- 172.21.15.46:0/1938225113 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69bc158f00 con 0x7f69bc1599d0 2024-01-26T23:45:38.023 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:38.018+0000 7f70091c51c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:45:38.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.021+0000 7f69c391c700 1 -- 172.21.15.46:0/1938225113 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69bc09a430 con 0x7f69bc1599d0 2024-01-26T23:45:38.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.022+0000 7f69b17fa700 1 -- 172.21.15.46:0/1938225113 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f69ac013680 con 0x7f69bc1599d0 2024-01-26T23:45:38.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.022+0000 7f69b17fa700 1 --2- 172.21.15.46:0/1938225113 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f69a0067050 0x7f69a0069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:38.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.022+0000 7f69c16b8700 1 -- 172.21.15.46:0/1938225113 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f69a0067050 msgr2=0x7f69a0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:38.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.022+0000 7f69b17fa700 1 -- 172.21.15.46:0/1938225113 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f69ac084ca0 con 0x7f69bc1599d0 2024-01-26T23:45:38.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.022+0000 7f69c16b8700 1 --2- 172.21.15.46:0/1938225113 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f69a0067050 0x7f69a0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:38.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.026+0000 7f69b17fa700 1 -- 172.21.15.46:0/1938225113 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f69bc09a430 con 0x7f69bc1599d0 2024-01-26T23:45:38.091 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:38.090+0000 7f70091c51c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:45:38.103 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:38.103+0000 7f023aa4e1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:45:38.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.223+0000 7f69c16b8700 1 -- 172.21.15.46:0/1938225113 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f69a0067050 msgr2=0x7f69a0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:38.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.223+0000 7f69c16b8700 1 --2- 172.21.15.46:0/1938225113 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f69a0067050 0x7f69a0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:38.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.232+0000 7f69c391c700 1 -- 172.21.15.46:0/1938225113 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f69bc09a430 con 0x7f69bc1599d0 2024-01-26T23:45:38.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.233+0000 7f69b17fa700 1 -- 172.21.15.46:0/1938225113 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) v1 ==== 100+0+11 (secure 0 0 0) 0x7f69bc09a430 con 0x7f69bc1599d0 2024-01-26T23:45:38.233 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-26T23:45:38.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.235+0000 7f69c391c700 1 -- 172.21.15.46:0/1938225113 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f69a0067050 msgr2=0x7f69a0069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:38.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.235+0000 7f69c391c700 1 --2- 172.21.15.46:0/1938225113 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f69a0067050 0x7f69a0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.235+0000 7f69c391c700 1 -- 172.21.15.46:0/1938225113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69bc1599d0 msgr2=0x7f69bc0ae2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:38.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.235+0000 7f69c391c700 1 --2- 172.21.15.46:0/1938225113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69bc1599d0 0x7f69bc0ae2d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00ec80 tx=0x7f69ac0091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.235+0000 7f69c391c700 1 -- 172.21.15.46:0/1938225113 shutdown_connections 2024-01-26T23:45:38.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.235+0000 7f69c391c700 1 --2- 172.21.15.46:0/1938225113 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f69a0067050 0x7f69a0069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.235+0000 7f69c391c700 1 --2- 172.21.15.46:0/1938225113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69bc15d8c0 0x7f69bc0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.235+0000 7f69c391c700 1 --2- 172.21.15.46:0/1938225113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69bc0ab1c0 0x7f69bc1589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.235+0000 7f69c391c700 1 --2- 172.21.15.46:0/1938225113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69bc1599d0 0x7f69bc0ae2d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.235+0000 7f69c391c700 1 -- 172.21.15.46:0/1938225113 >> 172.21.15.46:0/1938225113 conn(0x7f69bc06f840 msgr2=0x7f69bc0b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:38.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.236+0000 7f69c391c700 1 -- 172.21.15.46:0/1938225113 shutdown_connections 2024-01-26T23:45:38.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.236+0000 7f69c391c700 1 -- 172.21.15.46:0/1938225113 wait complete. 2024-01-26T23:45:38.251 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:45:38.364 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:38.364+0000 7f023aa4e1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:45:38.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.384+0000 7f2142d21700 1 Processor -- start 2024-01-26T23:45:38.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.384+0000 7f2142d21700 1 -- start start 2024-01-26T23:45:38.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.385+0000 7f2142d21700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f213c0cc1c0 0x7f213c164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.385+0000 7f2142d21700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f213c0cff70 con 0x7f213c15ce40 2024-01-26T23:45:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.385+0000 7f2142d21700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f213c0cf400 con 0x7f213c15d210 2024-01-26T23:45:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.385+0000 7f2142d21700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213c0cf660 con 0x7f213c0cc1c0 2024-01-26T23:45:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.385+0000 7f2140abd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f213c15ce40 0x7f213c0d0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53530/0 (socket says 172.21.15.46:53530) 2024-01-26T23:45:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.385+0000 7f2140abd700 1 -- 172.21.15.46:0/1173810183 learned_addr learned my addr 172.21.15.46:0/1173810183 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.385+0000 7f21412be700 1 --2- 172.21.15.46:0/1173810183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f213c0cc1c0 0x7f213c164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.386+0000 7f213b7fe700 1 -- 172.21.15.46:0/1173810183 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228731113 0 0) 0x7f213c0cff70 con 0x7f213c15ce40 2024-01-26T23:45:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.386+0000 7f213b7fe700 1 -- 172.21.15.46:0/1173810183 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2124003570 con 0x7f213c15ce40 2024-01-26T23:45:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.386+0000 7f213b7fe700 1 -- 172.21.15.46:0/1173810183 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483093379 0 0) 0x7f213c0cf400 con 0x7f213c15d210 2024-01-26T23:45:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.386+0000 7f213b7fe700 1 -- 172.21.15.46:0/1173810183 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f213c0cff70 con 0x7f213c15d210 2024-01-26T23:45:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.386+0000 7f21412be700 1 -- 172.21.15.46:0/1173810183 >> v1:172.21.15.106:6789/0 conn(0x7f213c15d210 legacy=0x7f213c0cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.386+0000 7f21412be700 1 -- 172.21.15.46:0/1173810183 >> v1:172.21.15.46:6789/0 conn(0x7f213c15ce40 legacy=0x7f213c0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.386+0000 7f21412be700 1 -- 172.21.15.46:0/1173810183 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f213c0d0500 con 0x7f213c0cc1c0 2024-01-26T23:45:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.386+0000 7f21412be700 1 --2- 172.21.15.46:0/1173810183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f213c0cc1c0 0x7f213c164700 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2134009e90 tx=0x7f2134009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=802bddaaff1ea1cc server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:38.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.386+0000 7f213b7fe700 1 -- 172.21.15.46:0/1173810183 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213400e070 con 0x7f213c0cc1c0 2024-01-26T23:45:38.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.386+0000 7f213b7fe700 1 -- 172.21.15.46:0/1173810183 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213400bb80 con 0x7f213c0cc1c0 2024-01-26T23:45:38.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.386+0000 7f213b7fe700 1 -- 172.21.15.46:0/1173810183 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2134012430 con 0x7f213c0cc1c0 2024-01-26T23:45:38.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.387+0000 7f2142d21700 1 -- 172.21.15.46:0/1173810183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f213c0cc1c0 msgr2=0x7f213c164700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:38.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.387+0000 7f2142d21700 1 --2- 172.21.15.46:0/1173810183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f213c0cc1c0 0x7f213c164700 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2134009e90 tx=0x7f2134009780 comp rx=0 tx=0).stop 2024-01-26T23:45:38.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.387+0000 7f2142d21700 1 -- 172.21.15.46:0/1173810183 shutdown_connections 2024-01-26T23:45:38.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.387+0000 7f2142d21700 1 --2- 172.21.15.46:0/1173810183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f213c0cc1c0 0x7f213c164700 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.387+0000 7f2142d21700 1 -- 172.21.15.46:0/1173810183 >> 172.21.15.46:0/1173810183 conn(0x7f213c06f840 msgr2=0x7f213c0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:38.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.387+0000 7f2142d21700 1 -- 172.21.15.46:0/1173810183 shutdown_connections 2024-01-26T23:45:38.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.387+0000 7f2142d21700 1 -- 172.21.15.46:0/1173810183 wait complete. 2024-01-26T23:45:38.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.388+0000 7f2142d21700 1 Processor -- start 2024-01-26T23:45:38.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.388+0000 7f2142d21700 1 -- start start 2024-01-26T23:45:38.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.389+0000 7f2142d21700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f213c0cc1c0 0x7f213c15f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:38.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.389+0000 7f2142d21700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f213c15ce40 0x7f213c16ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:38.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.389+0000 7f2142d21700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f213c15d210 0x7f213c16b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:38.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.389+0000 7f2142d21700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213c15ed90 con 0x7f213c15ce40 2024-01-26T23:45:38.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.389+0000 7f2142d21700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213c15e180 con 0x7f213c0cc1c0 2024-01-26T23:45:38.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.389+0000 7f2142d21700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213c15e460 con 0x7f213c15d210 2024-01-26T23:45:38.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.389+0000 7f213bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f213c15ce40 0x7f213c16ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:38.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.389+0000 7f2140abd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f213c0cc1c0 0x7f213c15f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:38.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.389+0000 7f213bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f213c15ce40 0x7f213c16ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50626/0 (socket says 172.21.15.46:50626) 2024-01-26T23:45:38.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.389+0000 7f213bfff700 1 -- 172.21.15.46:0/1392165689 learned_addr learned my addr 172.21.15.46:0/1392165689 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:38.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.389+0000 7f21412be700 1 --2- 172.21.15.46:0/1392165689 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f213c15d210 0x7f213c16b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:38.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.390+0000 7f21412be700 1 -- 172.21.15.46:0/1392165689 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f213c0cc1c0 msgr2=0x7f213c15f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:38.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.390+0000 7f21412be700 1 --2- 172.21.15.46:0/1392165689 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f213c0cc1c0 0x7f213c15f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.390+0000 7f21412be700 1 -- 172.21.15.46:0/1392165689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f213c15ce40 msgr2=0x7f213c16ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:38.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.390+0000 7f21412be700 1 --2- 172.21.15.46:0/1392165689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f213c15ce40 0x7f213c16ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.390+0000 7f21412be700 1 -- 172.21.15.46:0/1392165689 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2134009a20 con 0x7f213c15d210 2024-01-26T23:45:38.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.390+0000 7f213bfff700 1 --2- 172.21.15.46:0/1392165689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f213c15ce40 0x7f213c16ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:38.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.390+0000 7f2140abd700 1 --2- 172.21.15.46:0/1392165689 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f213c0cc1c0 0x7f213c15f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:38.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.390+0000 7f21412be700 1 --2- 172.21.15.46:0/1392165689 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f213c15d210 0x7f213c16b280 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2134016000 tx=0x7f2134019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:38.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.390+0000 7f2138ff9700 1 -- 172.21.15.46:0/1392165689 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213401b420 con 0x7f213c15d210 2024-01-26T23:45:38.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.390+0000 7f2138ff9700 1 -- 172.21.15.46:0/1392165689 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213401b5e0 con 0x7f213c15d210 2024-01-26T23:45:38.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.390+0000 7f2138ff9700 1 -- 172.21.15.46:0/1392165689 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213401b9a0 con 0x7f213c15d210 2024-01-26T23:45:38.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.390+0000 7f2142d21700 1 -- 172.21.15.46:0/1392165689 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f213c15e740 con 0x7f213c15d210 2024-01-26T23:45:38.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.390+0000 7f2142d21700 1 -- 172.21.15.46:0/1392165689 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f213c15f760 con 0x7f213c15d210 2024-01-26T23:45:38.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.391+0000 7f2142d21700 1 -- 172.21.15.46:0/1392165689 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f213c15ef00 con 0x7f213c15d210 2024-01-26T23:45:38.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.392+0000 7f2138ff9700 1 -- 172.21.15.46:0/1392165689 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f213401bd50 con 0x7f213c15d210 2024-01-26T23:45:38.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.392+0000 7f2138ff9700 1 --2- 172.21.15.46:0/1392165689 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2128067050 0x7f2128069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:38.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.393+0000 7f2140abd700 1 -- 172.21.15.46:0/1392165689 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2128067050 msgr2=0x7f2128069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:38.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.393+0000 7f2138ff9700 1 -- 172.21.15.46:0/1392165689 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f21340863e0 con 0x7f213c15d210 2024-01-26T23:45:38.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.393+0000 7f2140abd700 1 --2- 172.21.15.46:0/1392165689 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2128067050 0x7f2128069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:38.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.397+0000 7f2138ff9700 1 -- 172.21.15.46:0/1392165689 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2134056af0 con 0x7f213c15d210 2024-01-26T23:45:38.448 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:38.448+0000 7f023aa4e1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:45:38.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.594+0000 7f2140abd700 1 -- 172.21.15.46:0/1392165689 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2128067050 msgr2=0x7f2128069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:38.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.594+0000 7f2140abd700 1 --2- 172.21.15.46:0/1392165689 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2128067050 0x7f2128069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:38.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.651+0000 7f2142d21700 1 -- 172.21.15.46:0/1392165689 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f213c15e5a0 con 0x7f213c15d210 2024-01-26T23:45:38.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.652+0000 7f2138ff9700 1 -- 172.21.15.46:0/1392165689 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f213c15e5a0 con 0x7f213c15d210 2024-01-26T23:45:38.652 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:38.652 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/4083796808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":4083796808},{"type":"v1","addr":"172.21.15.106:6836","nonce":4083796808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":6172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/1769644590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1769644590},{"type":"v1","addr":"172.21.15.106:6837","nonce":1769644590}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":6184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1994423361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1994423361},{"type":"v1","addr":"172.21.15.46:6835","nonce":1994423361}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:45:19.187256+0000","modified":"2024-01-26T23:45:22.777939+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6366":{"gid":6366,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/1799649681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1799649681},{"type":"v1","addr":"172.21.15.46:6834","nonce":1799649681}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2024-01-26T23:45:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.654+0000 7f2142d21700 1 -- 172.21.15.46:0/1392165689 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2128067050 msgr2=0x7f2128069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.654+0000 7f2142d21700 1 --2- 172.21.15.46:0/1392165689 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2128067050 0x7f2128069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.654+0000 7f2142d21700 1 -- 172.21.15.46:0/1392165689 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f213c15d210 msgr2=0x7f213c16b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.654+0000 7f2142d21700 1 --2- 172.21.15.46:0/1392165689 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f213c15d210 0x7f213c16b280 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2134016000 tx=0x7f2134019000 comp rx=0 tx=0).stop 2024-01-26T23:45:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.654+0000 7f2142d21700 1 -- 172.21.15.46:0/1392165689 shutdown_connections 2024-01-26T23:45:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.654+0000 7f2142d21700 1 --2- 172.21.15.46:0/1392165689 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2128067050 0x7f2128069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.654+0000 7f2142d21700 1 --2- 172.21.15.46:0/1392165689 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f213c15d210 0x7f213c16b280 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.654+0000 7f2142d21700 1 --2- 172.21.15.46:0/1392165689 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f213c0cc1c0 0x7f213c15f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.654+0000 7f2142d21700 1 --2- 172.21.15.46:0/1392165689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f213c15ce40 0x7f213c16ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.654+0000 7f2142d21700 1 -- 172.21.15.46:0/1392165689 >> 172.21.15.46:0/1392165689 conn(0x7f213c06f840 msgr2=0x7f213c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.654+0000 7f2142d21700 1 -- 172.21.15.46:0/1392165689 shutdown_connections 2024-01-26T23:45:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.654+0000 7f2142d21700 1 -- 172.21.15.46:0/1392165689 wait complete. 2024-01-26T23:45:38.656 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 36 2024-01-26T23:45:38.669 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:45:38.690 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:38.690+0000 7f70091c51c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:45:38.764 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:38.764+0000 7f70091c51c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:45:38.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.800+0000 7ff2fa69d700 1 Processor -- start 2024-01-26T23:45:38.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.800+0000 7ff2fa69d700 1 -- start start 2024-01-26T23:45:38.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.801+0000 7ff2fa69d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff2f40c4720 0x7ff2f4159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:38.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.801+0000 7ff2fa69d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f40c97f0 con 0x7ff2f40cb4c0 2024-01-26T23:45:38.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.801+0000 7ff2fa69d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f40c8c20 con 0x7ff2f41596e0 2024-01-26T23:45:38.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.801+0000 7ff2fa69d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f40ce0d0 con 0x7ff2f40c4720 2024-01-26T23:45:38.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.801+0000 7ff2f37fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff2f40cb4c0 0x7ff2f40cd9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53532/0 (socket says 172.21.15.46:53532) 2024-01-26T23:45:38.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.801+0000 7ff2f8c3a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff2f40c4720 0x7ff2f4159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:38.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.801+0000 7ff2f37fe700 1 -- 172.21.15.46:0/2358735289 learned_addr learned my addr 172.21.15.46:0/2358735289 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:38.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.801+0000 7ff2f2ffd700 1 -- 172.21.15.46:0/2358735289 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190956873 0 0) 0x7ff2f40c97f0 con 0x7ff2f40cb4c0 2024-01-26T23:45:38.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.801+0000 7ff2f2ffd700 1 -- 172.21.15.46:0/2358735289 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2dc003570 con 0x7ff2f40cb4c0 2024-01-26T23:45:38.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.802+0000 7ff2f2ffd700 1 -- 172.21.15.46:0/2358735289 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449297429 0 0) 0x7ff2f40c8c20 con 0x7ff2f41596e0 2024-01-26T23:45:38.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.802+0000 7ff2f8c3a700 1 -- 172.21.15.46:0/2358735289 >> v1:172.21.15.106:6789/0 conn(0x7ff2f41596e0 legacy=0x7ff2f40cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:38.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.802+0000 7ff2f8c3a700 1 -- 172.21.15.46:0/2358735289 >> v1:172.21.15.46:6789/0 conn(0x7ff2f40cb4c0 legacy=0x7ff2f40cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:38.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.802+0000 7ff2f8c3a700 1 -- 172.21.15.46:0/2358735289 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f40c8ec0 con 0x7ff2f40c4720 2024-01-26T23:45:38.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.802+0000 7ff2f8c3a700 1 --2- 172.21.15.46:0/2358735289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2f40c4720 0x7ff2f4159ab0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec009e60 tx=0x7ff2ec009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97a5270558b6fbf server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:38.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.802+0000 7ff2f2ffd700 1 -- 172.21.15.46:0/2358735289 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec00e070 con 0x7ff2f40c4720 2024-01-26T23:45:38.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.802+0000 7ff2f2ffd700 1 -- 172.21.15.46:0/2358735289 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2ec00bb60 con 0x7ff2f40c4720 2024-01-26T23:45:38.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.802+0000 7ff2f2ffd700 1 -- 172.21.15.46:0/2358735289 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec00bee0 con 0x7ff2f40c4720 2024-01-26T23:45:38.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.803+0000 7ff2fa69d700 1 -- 172.21.15.46:0/2358735289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2f40c4720 msgr2=0x7ff2f4159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:38.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.803+0000 7ff2fa69d700 1 --2- 172.21.15.46:0/2358735289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2f40c4720 0x7ff2f4159ab0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec009e60 tx=0x7ff2ec009780 comp rx=0 tx=0).stop 2024-01-26T23:45:38.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.803+0000 7ff2fa69d700 1 -- 172.21.15.46:0/2358735289 shutdown_connections 2024-01-26T23:45:38.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.803+0000 7ff2fa69d700 1 --2- 172.21.15.46:0/2358735289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2f40c4720 0x7ff2f4159ab0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.803+0000 7ff2fa69d700 1 -- 172.21.15.46:0/2358735289 >> 172.21.15.46:0/2358735289 conn(0x7ff2f4097820 msgr2=0x7ff2f4068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:38.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.803+0000 7ff2fa69d700 1 -- 172.21.15.46:0/2358735289 shutdown_connections 2024-01-26T23:45:38.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.803+0000 7ff2fa69d700 1 -- 172.21.15.46:0/2358735289 wait complete. 2024-01-26T23:45:38.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.804+0000 7ff2fa69d700 1 Processor -- start 2024-01-26T23:45:38.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.804+0000 7ff2fa69d700 1 -- start start 2024-01-26T23:45:38.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.805+0000 7ff2fa69d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2f40c4720 0x7ff2f41548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:38.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.805+0000 7ff2fa69d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2f40cb4c0 0x7ff2f40c7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:38.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.805+0000 7ff2fa69d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2f41596e0 0x7ff2f40b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:38.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.805+0000 7ff2fa69d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f40b1ca0 con 0x7ff2f40cb4c0 2024-01-26T23:45:38.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.805+0000 7ff2fa69d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f40b2a60 con 0x7ff2f40c4720 2024-01-26T23:45:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.805+0000 7ff2fa69d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f40b1e90 con 0x7ff2f41596e0 2024-01-26T23:45:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.805+0000 7ff2f37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2f40cb4c0 0x7ff2f40c7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.805+0000 7ff2f37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2f40cb4c0 0x7ff2f40c7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50642/0 (socket says 172.21.15.46:50642) 2024-01-26T23:45:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.805+0000 7ff2f37fe700 1 -- 172.21.15.46:0/1516083238 learned_addr learned my addr 172.21.15.46:0/1516083238 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.805+0000 7ff2f3fff700 1 --2- 172.21.15.46:0/1516083238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2f40c4720 0x7ff2f41548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.805+0000 7ff2f8c3a700 1 --2- 172.21.15.46:0/1516083238 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2f41596e0 0x7ff2f40b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.805+0000 7ff2f37fe700 1 -- 172.21.15.46:0/1516083238 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2f41596e0 msgr2=0x7ff2f40b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.805+0000 7ff2f37fe700 1 --2- 172.21.15.46:0/1516083238 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2f41596e0 0x7ff2f40b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.805+0000 7ff2f37fe700 1 -- 172.21.15.46:0/1516083238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2f40c4720 msgr2=0x7ff2f41548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.805+0000 7ff2f37fe700 1 --2- 172.21.15.46:0/1516083238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2f40c4720 0x7ff2f41548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.805+0000 7ff2f37fe700 1 -- 172.21.15.46:0/1516083238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ec009a20 con 0x7ff2f40cb4c0 2024-01-26T23:45:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.806+0000 7ff2f37fe700 1 --2- 172.21.15.46:0/1516083238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2f40cb4c0 0x7ff2f40c7f30 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff2e800c7c0 tx=0x7ff2e8007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.806+0000 7ff2dbfff700 1 -- 172.21.15.46:0/1516083238 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8005c00 con 0x7ff2f40cb4c0 2024-01-26T23:45:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.806+0000 7ff2dbfff700 1 -- 172.21.15.46:0/1516083238 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e8005d60 con 0x7ff2f40cb4c0 2024-01-26T23:45:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.806+0000 7ff2fa69d700 1 -- 172.21.15.46:0/1516083238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f40b2130 con 0x7ff2f40cb4c0 2024-01-26T23:45:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.806+0000 7ff2dbfff700 1 -- 172.21.15.46:0/1516083238 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e80072e0 con 0x7ff2f40cb4c0 2024-01-26T23:45:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.806+0000 7ff2fa69d700 1 -- 172.21.15.46:0/1516083238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f40b26f0 con 0x7ff2f40cb4c0 2024-01-26T23:45:38.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.807+0000 7ff2fa69d700 1 -- 172.21.15.46:0/1516083238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f4063f90 con 0x7ff2f40cb4c0 2024-01-26T23:45:38.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.808+0000 7ff2dbfff700 1 -- 172.21.15.46:0/1516083238 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7ff2e8007690 con 0x7ff2f40cb4c0 2024-01-26T23:45:38.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.808+0000 7ff2dbfff700 1 --2- 172.21.15.46:0/1516083238 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff2e0067050 0x7ff2e0069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:38.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.808+0000 7ff2f3fff700 1 -- 172.21.15.46:0/1516083238 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff2e0067050 msgr2=0x7ff2e0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:38.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.808+0000 7ff2dbfff700 1 -- 172.21.15.46:0/1516083238 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6675+0+0 (secure 0 0 0) 0x7ff2e8083ce0 con 0x7ff2f40cb4c0 2024-01-26T23:45:38.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.808+0000 7ff2f3fff700 1 --2- 172.21.15.46:0/1516083238 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff2e0067050 0x7ff2e0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:38.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:38.813+0000 7ff2dbfff700 1 -- 172.21.15.46:0/1516083238 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff2e804b5a0 con 0x7ff2f40cb4c0 2024-01-26T23:45:38.888 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:38.888+0000 7f70091c51c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:45:39.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.009+0000 7ff2f3fff700 1 -- 172.21.15.46:0/1516083238 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff2e0067050 msgr2=0x7ff2e0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:39.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.009+0000 7ff2f3fff700 1 --2- 172.21.15.46:0/1516083238 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff2e0067050 0x7ff2e0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:39.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.022+0000 7ff2fa69d700 1 -- 172.21.15.46:0/1516083238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2f40b2bd0 con 0x7ff2f40cb4c0 2024-01-26T23:45:39.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.023+0000 7ff2dbfff700 1 -- 172.21.15.46:0/1516083238 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16524 (secure 0 0 0) 0x7ff2e8017020 con 0x7ff2f40cb4c0 2024-01-26T23:45:39.024 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:39.024 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":86,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:45:33.820805+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:45:17.703325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-01-26T23:45:18.149282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/2697231774":"2024-01-27T23:45:33.820757+0000","172.21.15.106:0/2527421951":"2024-01-27T23:45:33.820757+0000","172.21.15.106:0/2110014618":"2024-01-27T23:45:33.820757+0000","172.21.15.106:6833/43388":"2024-01-27T23:45:33.820757+0000","172.21.15.106:6832/43388":"2024-01-27T23:45:33.820757+0000","172.21.15.106:0/2931190795":"2024-01-27T23:45:33.820757+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:45:39.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.026+0000 7ff2fa69d700 1 -- 172.21.15.46:0/1516083238 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff2e0067050 msgr2=0x7ff2e0069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:39.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.026+0000 7ff2fa69d700 1 --2- 172.21.15.46:0/1516083238 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff2e0067050 0x7ff2e0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.026+0000 7ff2fa69d700 1 -- 172.21.15.46:0/1516083238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2f40cb4c0 msgr2=0x7ff2f40c7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:39.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.026+0000 7ff2fa69d700 1 --2- 172.21.15.46:0/1516083238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2f40cb4c0 0x7ff2f40c7f30 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff2e800c7c0 tx=0x7ff2e8007d90 comp rx=0 tx=0).stop 2024-01-26T23:45:39.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.026+0000 7ff2fa69d700 1 -- 172.21.15.46:0/1516083238 shutdown_connections 2024-01-26T23:45:39.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.026+0000 7ff2fa69d700 1 --2- 172.21.15.46:0/1516083238 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff2e0067050 0x7ff2e0069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.026+0000 7ff2fa69d700 1 --2- 172.21.15.46:0/1516083238 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2f41596e0 0x7ff2f40b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.026+0000 7ff2fa69d700 1 --2- 172.21.15.46:0/1516083238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2f40c4720 0x7ff2f41548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.026+0000 7ff2fa69d700 1 --2- 172.21.15.46:0/1516083238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2f40cb4c0 0x7ff2f40c7f30 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.026+0000 7ff2fa69d700 1 -- 172.21.15.46:0/1516083238 >> 172.21.15.46:0/1516083238 conn(0x7ff2f4097820 msgr2=0x7ff2f406c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:39.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.026+0000 7ff2fa69d700 1 -- 172.21.15.46:0/1516083238 shutdown_connections 2024-01-26T23:45:39.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.026+0000 7ff2fa69d700 1 -- 172.21.15.46:0/1516083238 wait complete. 2024-01-26T23:45:39.040 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-26T23:45:39.041 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:45:39.161 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:39.160+0000 7f023aa4e1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:45:39.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.175+0000 7ff8da413700 1 Processor -- start 2024-01-26T23:45:39.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.175+0000 7ff8da413700 1 -- start start 2024-01-26T23:45:39.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.176+0000 7ff8da413700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8d40ae520 0x7ff8d40b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:39.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.176+0000 7ff8da413700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff8d40b3fb0 0x7ff8d4157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:39.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.176+0000 7ff8da413700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d4158ae0 con 0x7ff8d4157340 2024-01-26T23:45:39.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.176+0000 7ff8da413700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d40b4430 con 0x7ff8d40b3fb0 2024-01-26T23:45:39.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.176+0000 7ff8da413700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d415b650 con 0x7ff8d40ae520 2024-01-26T23:45:39.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.176+0000 7ff8d3fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff8d4157340 0x7ff8d415bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53548/0 (socket says 172.21.15.46:53548) 2024-01-26T23:45:39.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.176+0000 7ff8d3fff700 1 -- 172.21.15.46:0/230535878 learned_addr learned my addr 172.21.15.46:0/230535878 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:39.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.176+0000 7ff8d37fe700 1 --2- 172.21.15.46:0/230535878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8d40ae520 0x7ff8d40b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:39.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.176+0000 7ff8d89b0700 1 --2- 172.21.15.46:0/230535878 >> v2:172.21.15.106:3300/0 conn(0x7ff8d40b3fb0 0x7ff8d4157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:39.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.177+0000 7ff8d2ffd700 1 -- 172.21.15.46:0/230535878 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786519336 0 0) 0x7ff8d4158ae0 con 0x7ff8d4157340 2024-01-26T23:45:39.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.177+0000 7ff8d2ffd700 1 -- 172.21.15.46:0/230535878 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8bc003570 con 0x7ff8d4157340 2024-01-26T23:45:39.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.177+0000 7ff8d37fe700 1 -- 172.21.15.46:0/230535878 >> v2:172.21.15.106:3300/0 conn(0x7ff8d40b3fb0 msgr2=0x7ff8d4157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:39.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.177+0000 7ff8d37fe700 1 --2- 172.21.15.46:0/230535878 >> v2:172.21.15.106:3300/0 conn(0x7ff8d40b3fb0 0x7ff8d4157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.177+0000 7ff8d37fe700 1 -- 172.21.15.46:0/230535878 >> v1:172.21.15.46:6789/0 conn(0x7ff8d4157340 legacy=0x7ff8d415bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:39.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.177+0000 7ff8d37fe700 1 -- 172.21.15.46:0/230535878 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d4158520 con 0x7ff8d40ae520 2024-01-26T23:45:39.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.177+0000 7ff8d89b0700 1 --2- 172.21.15.46:0/230535878 >> v2:172.21.15.106:3300/0 conn(0x7ff8d40b3fb0 0x7ff8d4157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:39.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.177+0000 7ff8d37fe700 1 --2- 172.21.15.46:0/230535878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8d40ae520 0x7ff8d40b3a70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80098e0 tx=0x7ff8c80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6594978e66af036 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:39.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.177+0000 7ff8d2ffd700 1 -- 172.21.15.46:0/230535878 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8011630 con 0x7ff8d40ae520 2024-01-26T23:45:39.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.177+0000 7ff8d2ffd700 1 -- 172.21.15.46:0/230535878 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c8011790 con 0x7ff8d40ae520 2024-01-26T23:45:39.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.177+0000 7ff8d2ffd700 1 -- 172.21.15.46:0/230535878 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8011b30 con 0x7ff8d40ae520 2024-01-26T23:45:39.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.178+0000 7ff8da413700 1 -- 172.21.15.46:0/230535878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8d40ae520 msgr2=0x7ff8d40b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:39.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.178+0000 7ff8da413700 1 --2- 172.21.15.46:0/230535878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8d40ae520 0x7ff8d40b3a70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80098e0 tx=0x7ff8c80091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.178+0000 7ff8da413700 1 -- 172.21.15.46:0/230535878 shutdown_connections 2024-01-26T23:45:39.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.178+0000 7ff8da413700 1 --2- 172.21.15.46:0/230535878 >> v2:172.21.15.106:3300/0 conn(0x7ff8d40b3fb0 0x7ff8d4157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.178+0000 7ff8da413700 1 --2- 172.21.15.46:0/230535878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8d40ae520 0x7ff8d40b3a70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.178+0000 7ff8da413700 1 -- 172.21.15.46:0/230535878 >> 172.21.15.46:0/230535878 conn(0x7ff8d40ce530 msgr2=0x7ff8d40ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:39.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.178+0000 7ff8da413700 1 -- 172.21.15.46:0/230535878 shutdown_connections 2024-01-26T23:45:39.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.178+0000 7ff8da413700 1 -- 172.21.15.46:0/230535878 wait complete. 2024-01-26T23:45:39.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.179+0000 7ff8da413700 1 Processor -- start 2024-01-26T23:45:39.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.179+0000 7ff8da413700 1 -- start start 2024-01-26T23:45:39.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.180+0000 7ff8da413700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8d40ae520 0x7ff8d40acec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:39.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.180+0000 7ff8da413700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8d40b3fb0 0x7ff8d40c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:39.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.180+0000 7ff8da413700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8d4157340 0x7ff8d40cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:39.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.180+0000 7ff8da413700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d406a7b0 con 0x7ff8d40ae520 2024-01-26T23:45:39.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.180+0000 7ff8da413700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d416a270 con 0x7ff8d4157340 2024-01-26T23:45:39.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.180+0000 7ff8da413700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d40c5920 con 0x7ff8d40b3fb0 2024-01-26T23:45:39.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.180+0000 7ff8d3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8d40ae520 0x7ff8d40acec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:39.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.180+0000 7ff8d3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8d40ae520 0x7ff8d40acec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50648/0 (socket says 172.21.15.46:50648) 2024-01-26T23:45:39.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.180+0000 7ff8d3fff700 1 -- 172.21.15.46:0/3474842327 learned_addr learned my addr 172.21.15.46:0/3474842327 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:39.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.180+0000 7ff8d89b0700 1 --2- 172.21.15.46:0/3474842327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8d4157340 0x7ff8d40cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:39.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.180+0000 7ff8d37fe700 1 --2- 172.21.15.46:0/3474842327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8d40b3fb0 0x7ff8d40c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:39.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.181+0000 7ff8d3fff700 1 -- 172.21.15.46:0/3474842327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8d40b3fb0 msgr2=0x7ff8d40c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:39.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.181+0000 7ff8d3fff700 1 --2- 172.21.15.46:0/3474842327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8d40b3fb0 0x7ff8d40c7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.181+0000 7ff8d3fff700 1 -- 172.21.15.46:0/3474842327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8d4157340 msgr2=0x7ff8d40cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.181+0000 7ff8d3fff700 1 --2- 172.21.15.46:0/3474842327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8d4157340 0x7ff8d40cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.181+0000 7ff8d3fff700 1 -- 172.21.15.46:0/3474842327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8cc009a20 con 0x7ff8d40ae520 2024-01-26T23:45:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.181+0000 7ff8d89b0700 1 --2- 172.21.15.46:0/3474842327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8d4157340 0x7ff8d40cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.181+0000 7ff8d3fff700 1 --2- 172.21.15.46:0/3474842327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8d40ae520 0x7ff8d40acec0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400c7c0 tx=0x7ff8c4007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.181+0000 7ff8bbfff700 1 -- 172.21.15.46:0/3474842327 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4005b30 con 0x7ff8d40ae520 2024-01-26T23:45:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.181+0000 7ff8bbfff700 1 -- 172.21.15.46:0/3474842327 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c4005cb0 con 0x7ff8d40ae520 2024-01-26T23:45:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.181+0000 7ff8da413700 1 -- 172.21.15.46:0/3474842327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c8009420 con 0x7ff8d40ae520 2024-01-26T23:45:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.181+0000 7ff8bbfff700 1 -- 172.21.15.46:0/3474842327 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4007320 con 0x7ff8d40ae520 2024-01-26T23:45:39.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.181+0000 7ff8da413700 1 -- 172.21.15.46:0/3474842327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d4169f00 con 0x7ff8d40ae520 2024-01-26T23:45:39.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.182+0000 7ff8da413700 1 -- 172.21.15.46:0/3474842327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d40cacf0 con 0x7ff8d40ae520 2024-01-26T23:45:39.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.183+0000 7ff8bbfff700 1 -- 172.21.15.46:0/3474842327 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7ff8c4007690 con 0x7ff8d40ae520 2024-01-26T23:45:39.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.183+0000 7ff8bbfff700 1 --2- 172.21.15.46:0/3474842327 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff8c0067050 0x7ff8c0069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:39.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.183+0000 7ff8bbfff700 1 -- 172.21.15.46:0/3474842327 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6675+0+0 (secure 0 0 0) 0x7ff8c4083e80 con 0x7ff8d40ae520 2024-01-26T23:45:39.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.183+0000 7ff8d37fe700 1 -- 172.21.15.46:0/3474842327 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff8c0067050 msgr2=0x7ff8c0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:39.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.183+0000 7ff8d37fe700 1 --2- 172.21.15.46:0/3474842327 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff8c0067050 0x7ff8c0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:39.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.188+0000 7ff8bbfff700 1 -- 172.21.15.46:0/3474842327 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff8c4050cd0 con 0x7ff8d40ae520 2024-01-26T23:45:39.248 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:39.248+0000 7f023aa4e1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:45:39.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.384+0000 7ff8d37fe700 1 -- 172.21.15.46:0/3474842327 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff8c0067050 msgr2=0x7ff8c0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:39.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.384+0000 7ff8d37fe700 1 --2- 172.21.15.46:0/3474842327 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff8c0067050 0x7ff8c0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:39.388 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:39.389+0000 7f70091c51c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:45:39.390 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:39.390+0000 7f023aa4e1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:45:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.441+0000 7ff8da413700 1 -- 172.21.15.46:0/3474842327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8d404c710 con 0x7ff8d40ae520 2024-01-26T23:45:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.442+0000 7ff8bbfff700 1 -- 172.21.15.46:0/3474842327 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4604 (secure 0 0 0) 0x7ff8c4017020 con 0x7ff8d40ae520 2024-01-26T23:45:39.442 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:39.442 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/4083796808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":4083796808},{"type":"v1","addr":"172.21.15.106:6836","nonce":4083796808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":6172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/1769644590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1769644590},{"type":"v1","addr":"172.21.15.106:6837","nonce":1769644590}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":6184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1994423361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1994423361},{"type":"v1","addr":"172.21.15.46:6835","nonce":1994423361}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:45:19.187256+0000","modified":"2024-01-26T23:45:22.777939+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6366":{"gid":6366,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/1799649681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1799649681},{"type":"v1","addr":"172.21.15.46:6834","nonce":1799649681}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2024-01-26T23:45:39.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.444+0000 7ff8da413700 1 -- 172.21.15.46:0/3474842327 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff8c0067050 msgr2=0x7ff8c0069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:39.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.444+0000 7ff8da413700 1 --2- 172.21.15.46:0/3474842327 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff8c0067050 0x7ff8c0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.444+0000 7ff8da413700 1 -- 172.21.15.46:0/3474842327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8d40ae520 msgr2=0x7ff8d40acec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:39.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.444+0000 7ff8da413700 1 --2- 172.21.15.46:0/3474842327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8d40ae520 0x7ff8d40acec0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400c7c0 tx=0x7ff8c4007e00 comp rx=0 tx=0).stop 2024-01-26T23:45:39.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.444+0000 7ff8da413700 1 -- 172.21.15.46:0/3474842327 shutdown_connections 2024-01-26T23:45:39.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.444+0000 7ff8da413700 1 --2- 172.21.15.46:0/3474842327 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff8c0067050 0x7ff8c0069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.444+0000 7ff8da413700 1 --2- 172.21.15.46:0/3474842327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8d40ae520 0x7ff8d40acec0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.444+0000 7ff8da413700 1 --2- 172.21.15.46:0/3474842327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8d4157340 0x7ff8d40cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.444+0000 7ff8da413700 1 --2- 172.21.15.46:0/3474842327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8d40b3fb0 0x7ff8d40c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.444+0000 7ff8da413700 1 -- 172.21.15.46:0/3474842327 >> 172.21.15.46:0/3474842327 conn(0x7ff8d40ce530 msgr2=0x7ff8d40b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:39.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.444+0000 7ff8da413700 1 -- 172.21.15.46:0/3474842327 shutdown_connections 2024-01-26T23:45:39.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.444+0000 7ff8da413700 1 -- 172.21.15.46:0/3474842327 wait complete. 2024-01-26T23:45:39.446 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 36 2024-01-26T23:45:39.459 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:45:39.535 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:39.535+0000 7f70091c51c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:45:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.591+0000 7f7c99e95700 1 Processor -- start 2024-01-26T23:45:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.591+0000 7f7c99e95700 1 -- start start 2024-01-26T23:45:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.591+0000 7f7c99e95700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7c940c6520 0x7f7c940ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.592+0000 7f7c99e95700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c9415a0b0 con 0x7f7c940b2b40 2024-01-26T23:45:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.592+0000 7f7c99e95700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c940ae4d0 con 0x7f7c940ced50 2024-01-26T23:45:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.592+0000 7f7c99e95700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c940af070 con 0x7f7c940c6520 2024-01-26T23:45:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.592+0000 7f7c937fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7c940b2b40 0x7f7c940ae960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53560/0 (socket says 172.21.15.46:53560) 2024-01-26T23:45:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.592+0000 7f7c937fe700 1 -- 172.21.15.46:0/4020169273 learned_addr learned my addr 172.21.15.46:0/4020169273 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.592+0000 7f7c92ffd700 1 --2- 172.21.15.46:0/4020169273 >> v2:172.21.15.106:3300/0 conn(0x7f7c940c6520 0x7f7c940ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.592+0000 7f7c927fc700 1 -- 172.21.15.46:0/4020169273 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111510860 0 0) 0x7f7c9415a0b0 con 0x7f7c940b2b40 2024-01-26T23:45:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.592+0000 7f7c927fc700 1 -- 172.21.15.46:0/4020169273 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c7c003570 con 0x7f7c940b2b40 2024-01-26T23:45:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.592+0000 7f7c92ffd700 1 -- 172.21.15.46:0/4020169273 >> v1:172.21.15.106:6789/0 conn(0x7f7c940ced50 legacy=0x7f7c940cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.592+0000 7f7c92ffd700 1 -- 172.21.15.46:0/4020169273 >> v1:172.21.15.46:6789/0 conn(0x7f7c940b2b40 legacy=0x7f7c940ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.592+0000 7f7c92ffd700 1 -- 172.21.15.46:0/4020169273 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c940ad8c0 con 0x7f7c940c6520 2024-01-26T23:45:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.593+0000 7f7c92ffd700 1 --2- 172.21.15.46:0/4020169273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c940c6520 0x7f7c940ad1c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7c880098e0 tx=0x7f7c880091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad52c0d9729dca59 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.593+0000 7f7c927fc700 1 -- 172.21.15.46:0/4020169273 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8800b640 con 0x7f7c940c6520 2024-01-26T23:45:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.593+0000 7f7c927fc700 1 -- 172.21.15.46:0/4020169273 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8800b7a0 con 0x7f7c940c6520 2024-01-26T23:45:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.593+0000 7f7c927fc700 1 -- 172.21.15.46:0/4020169273 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8800bb40 con 0x7f7c940c6520 2024-01-26T23:45:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.593+0000 7f7c99e95700 1 -- 172.21.15.46:0/4020169273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c940c6520 msgr2=0x7f7c940ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.593+0000 7f7c99e95700 1 --2- 172.21.15.46:0/4020169273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c940c6520 0x7f7c940ad1c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7c880098e0 tx=0x7f7c880091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.593+0000 7f7c99e95700 1 -- 172.21.15.46:0/4020169273 shutdown_connections 2024-01-26T23:45:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.593+0000 7f7c99e95700 1 --2- 172.21.15.46:0/4020169273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c940c6520 0x7f7c940ad1c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.593+0000 7f7c99e95700 1 -- 172.21.15.46:0/4020169273 >> 172.21.15.46:0/4020169273 conn(0x7f7c94099a30 msgr2=0x7f7c9406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.593+0000 7f7c99e95700 1 -- 172.21.15.46:0/4020169273 shutdown_connections 2024-01-26T23:45:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.593+0000 7f7c99e95700 1 -- 172.21.15.46:0/4020169273 wait complete. 2024-01-26T23:45:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.595+0000 7f7c99e95700 1 Processor -- start 2024-01-26T23:45:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.595+0000 7f7c99e95700 1 -- start start 2024-01-26T23:45:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.596+0000 7f7c99e95700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c940b2b40 0x7f7c940ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.596+0000 7f7c99e95700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c940c6520 0x7f7c940b5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.596+0000 7f7c99e95700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c940ced50 0x7f7c94157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.596+0000 7f7c99e95700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c940b6330 con 0x7f7c940ced50 2024-01-26T23:45:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.596+0000 7f7c99e95700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c94157b40 con 0x7f7c940c6520 2024-01-26T23:45:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.596+0000 7f7c99e95700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9415dcb0 con 0x7f7c940b2b40 2024-01-26T23:45:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.597+0000 7f7c93fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c940ced50 0x7f7c94157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.597+0000 7f7c93fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c940ced50 0x7f7c94157600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50650/0 (socket says 172.21.15.46:50650) 2024-01-26T23:45:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.597+0000 7f7c93fff700 1 -- 172.21.15.46:0/3252364504 learned_addr learned my addr 172.21.15.46:0/3252364504 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.597+0000 7f7c92ffd700 1 --2- 172.21.15.46:0/3252364504 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c940c6520 0x7f7c940b5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.597+0000 7f7c937fe700 1 --2- 172.21.15.46:0/3252364504 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c940b2b40 0x7f7c940ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.597+0000 7f7c92ffd700 1 -- 172.21.15.46:0/3252364504 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c940b2b40 msgr2=0x7f7c940ca170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.597+0000 7f7c92ffd700 1 --2- 172.21.15.46:0/3252364504 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c940b2b40 0x7f7c940ca170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.597+0000 7f7c92ffd700 1 -- 172.21.15.46:0/3252364504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c940ced50 msgr2=0x7f7c94157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.597+0000 7f7c92ffd700 1 --2- 172.21.15.46:0/3252364504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c940ced50 0x7f7c94157600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.597+0000 7f7c92ffd700 1 -- 172.21.15.46:0/3252364504 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c88009420 con 0x7f7c940c6520 2024-01-26T23:45:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.597+0000 7f7c92ffd700 1 --2- 172.21.15.46:0/3252364504 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c940c6520 0x7f7c940b5df0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f7c88016000 tx=0x7f7c88017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.597+0000 7f7c7b7fe700 1 -- 172.21.15.46:0/3252364504 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c88000d10 con 0x7f7c940c6520 2024-01-26T23:45:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.597+0000 7f7c7b7fe700 1 -- 172.21.15.46:0/3252364504 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8800bb20 con 0x7f7c940c6520 2024-01-26T23:45:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.597+0000 7f7c99e95700 1 -- 172.21.15.46:0/3252364504 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c9415df90 con 0x7f7c940c6520 2024-01-26T23:45:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.597+0000 7f7c7b7fe700 1 -- 172.21.15.46:0/3252364504 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c88011670 con 0x7f7c940c6520 2024-01-26T23:45:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.598+0000 7f7c99e95700 1 -- 172.21.15.46:0/3252364504 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c9415e550 con 0x7f7c940c6520 2024-01-26T23:45:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.599+0000 7f7c99e95700 1 -- 172.21.15.46:0/3252364504 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c9404c710 con 0x7f7c940c6520 2024-01-26T23:45:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.599+0000 7f7c7b7fe700 1 -- 172.21.15.46:0/3252364504 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f7c880119e0 con 0x7f7c940c6520 2024-01-26T23:45:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.599+0000 7f7c7b7fe700 1 --2- 172.21.15.46:0/3252364504 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7c80067050 0x7f7c80069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.599+0000 7f7c937fe700 1 -- 172.21.15.46:0/3252364504 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7c80067050 msgr2=0x7f7c80069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.599+0000 7f7c7b7fe700 1 -- 172.21.15.46:0/3252364504 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f7c88085fb0 con 0x7f7c940c6520 2024-01-26T23:45:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.599+0000 7f7c937fe700 1 --2- 172.21.15.46:0/3252364504 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7c80067050 0x7f7c80069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:39.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.604+0000 7f7c7b7fe700 1 -- 172.21.15.46:0/3252364504 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c88052ca0 con 0x7f7c940c6520 2024-01-26T23:45:39.684 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:39.684+0000 7f70091c51c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:45:39.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.800+0000 7f7c937fe700 1 -- 172.21.15.46:0/3252364504 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7c80067050 msgr2=0x7f7c80069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:39.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.800+0000 7f7c937fe700 1 --2- 172.21.15.46:0/3252364504 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7c80067050 0x7f7c80069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:39.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.858+0000 7f7c99e95700 1 -- 172.21.15.46:0/3252364504 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c940ade70 con 0x7f7c940c6520 2024-01-26T23:45:39.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.859+0000 7f7c7b7fe700 1 -- 172.21.15.46:0/3252364504 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f7c880562c0 con 0x7f7c940c6520 2024-01-26T23:45:39.859 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:39.859 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/4083796808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":4083796808},{"type":"v1","addr":"172.21.15.106:6836","nonce":4083796808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":6172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/1769644590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1769644590},{"type":"v1","addr":"172.21.15.106:6837","nonce":1769644590}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":6184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1994423361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1994423361},{"type":"v1","addr":"172.21.15.46:6835","nonce":1994423361}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:45:19.187256+0000","modified":"2024-01-26T23:45:22.777939+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6366":{"gid":6366,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/1799649681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1799649681},{"type":"v1","addr":"172.21.15.46:6834","nonce":1799649681}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2024-01-26T23:45:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.861+0000 7f7c99e95700 1 -- 172.21.15.46:0/3252364504 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7c80067050 msgr2=0x7f7c80069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.861+0000 7f7c99e95700 1 --2- 172.21.15.46:0/3252364504 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7c80067050 0x7f7c80069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.861+0000 7f7c99e95700 1 -- 172.21.15.46:0/3252364504 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c940c6520 msgr2=0x7f7c940b5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.861+0000 7f7c99e95700 1 --2- 172.21.15.46:0/3252364504 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c940c6520 0x7f7c940b5df0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f7c88016000 tx=0x7f7c88017000 comp rx=0 tx=0).stop 2024-01-26T23:45:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.861+0000 7f7c99e95700 1 -- 172.21.15.46:0/3252364504 shutdown_connections 2024-01-26T23:45:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.861+0000 7f7c99e95700 1 --2- 172.21.15.46:0/3252364504 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7c80067050 0x7f7c80069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.861+0000 7f7c99e95700 1 --2- 172.21.15.46:0/3252364504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c940ced50 0x7f7c94157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.861+0000 7f7c99e95700 1 --2- 172.21.15.46:0/3252364504 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c940b2b40 0x7f7c940ca170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.861+0000 7f7c99e95700 1 --2- 172.21.15.46:0/3252364504 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c940c6520 0x7f7c940b5df0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.861+0000 7f7c99e95700 1 -- 172.21.15.46:0/3252364504 >> 172.21.15.46:0/3252364504 conn(0x7f7c94099a30 msgr2=0x7f7c941593e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.861+0000 7f7c99e95700 1 -- 172.21.15.46:0/3252364504 shutdown_connections 2024-01-26T23:45:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:39.861+0000 7f7c99e95700 1 -- 172.21.15.46:0/3252364504 wait complete. 2024-01-26T23:45:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 36 2024-01-26T23:45:39.875 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:45:39.990 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:39.990+0000 7f023aa4e1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:45:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.008+0000 7f75b17c7700 1 Processor -- start 2024-01-26T23:45:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.008+0000 7f75b17c7700 1 -- start start 2024-01-26T23:45:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.009+0000 7f75b17c7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f75ac0c6520 0x7f75ac0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.009+0000 7f75b17c7700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f75ac0ced50 0x7f75ac0cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.009+0000 7f75b17c7700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75ac15a0b0 con 0x7f75ac0b2b40 2024-01-26T23:45:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.009+0000 7f75b17c7700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75ac0af070 con 0x7f75ac0ced50 2024-01-26T23:45:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.009+0000 7f75b17c7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac099fc0 con 0x7f75ac0c6520 2024-01-26T23:45:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.009+0000 7f75aaffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f75ac0b2b40 0x7f75ac0ae960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53566/0 (socket says 172.21.15.46:53566) 2024-01-26T23:45:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.009+0000 7f75aaffd700 1 -- 172.21.15.46:0/1797415081 learned_addr learned my addr 172.21.15.46:0/1797415081 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.009+0000 7f75ab7fe700 1 --2- 172.21.15.46:0/1797415081 >> v2:172.21.15.106:3300/0 conn(0x7f75ac0ced50 0x7f75ac0cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.009+0000 7f75a27fc700 1 --2- 172.21.15.46:0/1797415081 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f75ac0c6520 0x7f75ac0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.010+0000 7f75aa7fc700 1 -- 172.21.15.46:0/1797415081 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196594784 0 0) 0x7f75ac15a0b0 con 0x7f75ac0b2b40 2024-01-26T23:45:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.010+0000 7f75aa7fc700 1 -- 172.21.15.46:0/1797415081 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7598003570 con 0x7f75ac0b2b40 2024-01-26T23:45:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.010+0000 7f75ab7fe700 1 -- 172.21.15.46:0/1797415081 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f75ac0c6520 msgr2=0x7f75ac0ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.010+0000 7f75ab7fe700 1 --2- 172.21.15.46:0/1797415081 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f75ac0c6520 0x7f75ac0ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.010+0000 7f75ab7fe700 1 -- 172.21.15.46:0/1797415081 >> v1:172.21.15.46:6789/0 conn(0x7f75ac0b2b40 legacy=0x7f75ac0ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.010+0000 7f75ab7fe700 1 -- 172.21.15.46:0/1797415081 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac0ae4d0 con 0x7f75ac0ced50 2024-01-26T23:45:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.010+0000 7f75a27fc700 1 --2- 172.21.15.46:0/1797415081 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f75ac0c6520 0x7f75ac0ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.010+0000 7f75ab7fe700 1 --2- 172.21.15.46:0/1797415081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75ac0ced50 0x7f75ac0cf1d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f75a4009e60 tx=0x7f75a4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8c4e5df8a8f9201 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.010+0000 7f75aa7fc700 1 -- 172.21.15.46:0/1797415081 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a400e070 con 0x7f75ac0ced50 2024-01-26T23:45:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.010+0000 7f75aa7fc700 1 -- 172.21.15.46:0/1797415081 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a400bab0 con 0x7f75ac0ced50 2024-01-26T23:45:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.010+0000 7f75aa7fc700 1 -- 172.21.15.46:0/1797415081 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a400be70 con 0x7f75ac0ced50 2024-01-26T23:45:40.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.011+0000 7f75b17c7700 1 -- 172.21.15.46:0/1797415081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75ac0ced50 msgr2=0x7f75ac0cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:40.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.011+0000 7f75b17c7700 1 --2- 172.21.15.46:0/1797415081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75ac0ced50 0x7f75ac0cf1d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f75a4009e60 tx=0x7f75a4009780 comp rx=0 tx=0).stop 2024-01-26T23:45:40.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.011+0000 7f75b17c7700 1 -- 172.21.15.46:0/1797415081 shutdown_connections 2024-01-26T23:45:40.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.011+0000 7f75b17c7700 1 --2- 172.21.15.46:0/1797415081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75ac0ced50 0x7f75ac0cf1d0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.011+0000 7f75b17c7700 1 --2- 172.21.15.46:0/1797415081 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f75ac0c6520 0x7f75ac0ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.011+0000 7f75b17c7700 1 -- 172.21.15.46:0/1797415081 >> 172.21.15.46:0/1797415081 conn(0x7f75ac099a30 msgr2=0x7f75ac06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:40.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.011+0000 7f75b17c7700 1 -- 172.21.15.46:0/1797415081 shutdown_connections 2024-01-26T23:45:40.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.011+0000 7f75b17c7700 1 -- 172.21.15.46:0/1797415081 wait complete. 2024-01-26T23:45:40.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.012+0000 7f75b17c7700 1 Processor -- start 2024-01-26T23:45:40.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.013+0000 7f75b17c7700 1 -- start start 2024-01-26T23:45:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.013+0000 7f75b17c7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75ac0b2b40 0x7f75ac156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.013+0000 7f75b17c7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75ac0c6520 0x7f75ac0cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.013+0000 7f75b17c7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f75ac0ced50 0x7f75ac147080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.013+0000 7f75b17c7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac0c84b0 con 0x7f75ac0c6520 2024-01-26T23:45:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.013+0000 7f75b17c7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac0c8790 con 0x7f75ac0b2b40 2024-01-26T23:45:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.013+0000 7f75b17c7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac155fe0 con 0x7f75ac0ced50 2024-01-26T23:45:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.013+0000 7f75a27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75ac0c6520 0x7f75ac0cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.014+0000 7f75a27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75ac0c6520 0x7f75ac0cc6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50658/0 (socket says 172.21.15.46:50658) 2024-01-26T23:45:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.014+0000 7f75a27fc700 1 -- 172.21.15.46:0/2427517204 learned_addr learned my addr 172.21.15.46:0/2427517204 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.014+0000 7f75aaffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75ac0b2b40 0x7f75ac156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.014+0000 7f75ab7fe700 1 --2- 172.21.15.46:0/2427517204 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f75ac0ced50 0x7f75ac147080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:40.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.014+0000 7f75a27fc700 1 -- 172.21.15.46:0/2427517204 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f75ac0ced50 msgr2=0x7f75ac147080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:40.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.014+0000 7f75a27fc700 1 --2- 172.21.15.46:0/2427517204 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f75ac0ced50 0x7f75ac147080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.014+0000 7f75a27fc700 1 -- 172.21.15.46:0/2427517204 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75ac0b2b40 msgr2=0x7f75ac156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:40.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.014+0000 7f75a27fc700 1 --2- 172.21.15.46:0/2427517204 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75ac0b2b40 0x7f75ac156ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.014+0000 7f75a27fc700 1 -- 172.21.15.46:0/2427517204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7594009420 con 0x7f75ac0c6520 2024-01-26T23:45:40.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.014+0000 7f75aaffd700 1 --2- 172.21.15.46:0/2427517204 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75ac0b2b40 0x7f75ac156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:40.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.014+0000 7f75a27fc700 1 --2- 172.21.15.46:0/2427517204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75ac0c6520 0x7f75ac0cc6b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f759400ec80 tx=0x7f75940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.014+0000 7f75a37fe700 1 -- 172.21.15.46:0/2427517204 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759400c920 con 0x7f75ac0c6520 2024-01-26T23:45:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.014+0000 7f75a37fe700 1 -- 172.21.15.46:0/2427517204 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759400ca80 con 0x7f75ac0c6520 2024-01-26T23:45:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.014+0000 7f75b17c7700 1 -- 172.21.15.46:0/2427517204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a4009a20 con 0x7f75ac0c6520 2024-01-26T23:45:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.014+0000 7f75a37fe700 1 -- 172.21.15.46:0/2427517204 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759400ce00 con 0x7f75ac0c6520 2024-01-26T23:45:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.015+0000 7f75b17c7700 1 -- 172.21.15.46:0/2427517204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ac1565a0 con 0x7f75ac0c6520 2024-01-26T23:45:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.016+0000 7f75b17c7700 1 -- 172.21.15.46:0/2427517204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75ac1567c0 con 0x7f75ac0c6520 2024-01-26T23:45:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.016+0000 7f75a37fe700 1 -- 172.21.15.46:0/2427517204 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f7594013680 con 0x7f75ac0c6520 2024-01-26T23:45:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.017+0000 7f75a37fe700 1 --2- 172.21.15.46:0/2427517204 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f758c067050 0x7f758c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.017+0000 7f75a37fe700 1 -- 172.21.15.46:0/2427517204 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f7594084ef0 con 0x7f75ac0c6520 2024-01-26T23:45:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.017+0000 7f75aaffd700 1 -- 172.21.15.46:0/2427517204 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f758c067050 msgr2=0x7f758c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.017+0000 7f75aaffd700 1 --2- 172.21.15.46:0/2427517204 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f758c067050 0x7f758c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.021+0000 7f75a37fe700 1 -- 172.21.15.46:0/2427517204 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7594051be0 con 0x7f75ac0c6520 2024-01-26T23:45:40.166 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:40.165+0000 7f023aa4e1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:45:40.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.218+0000 7f75aaffd700 1 -- 172.21.15.46:0/2427517204 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f758c067050 msgr2=0x7f758c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:40.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.218+0000 7f75aaffd700 1 --2- 172.21.15.46:0/2427517204 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f758c067050 0x7f758c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:40.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.230+0000 7f75b17c7700 1 -- 172.21.15.46:0/2427517204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75ac04c710 con 0x7f75ac0c6520 2024-01-26T23:45:40.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.231+0000 7f75a37fe700 1 -- 172.21.15.46:0/2427517204 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16524 (secure 0 0 0) 0x7f7594055200 con 0x7f75ac0c6520 2024-01-26T23:45:40.231 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:40.231 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":86,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:45:33.820805+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:45:17.703325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-01-26T23:45:18.149282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/2697231774":"2024-01-27T23:45:33.820757+0000","172.21.15.106:0/2527421951":"2024-01-27T23:45:33.820757+0000","172.21.15.106:0/2110014618":"2024-01-27T23:45:33.820757+0000","172.21.15.106:6833/43388":"2024-01-27T23:45:33.820757+0000","172.21.15.106:6832/43388":"2024-01-27T23:45:33.820757+0000","172.21.15.106:0/2931190795":"2024-01-27T23:45:33.820757+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:45:40.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.233+0000 7f75b17c7700 1 -- 172.21.15.46:0/2427517204 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f758c067050 msgr2=0x7f758c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:40.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.233+0000 7f75b17c7700 1 --2- 172.21.15.46:0/2427517204 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f758c067050 0x7f758c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.233+0000 7f75b17c7700 1 -- 172.21.15.46:0/2427517204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75ac0c6520 msgr2=0x7f75ac0cc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:40.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.233+0000 7f75b17c7700 1 --2- 172.21.15.46:0/2427517204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75ac0c6520 0x7f75ac0cc6b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f759400ec80 tx=0x7f75940091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.233+0000 7f75b17c7700 1 -- 172.21.15.46:0/2427517204 shutdown_connections 2024-01-26T23:45:40.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.233+0000 7f75b17c7700 1 --2- 172.21.15.46:0/2427517204 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f758c067050 0x7f758c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.233+0000 7f75b17c7700 1 --2- 172.21.15.46:0/2427517204 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f75ac0ced50 0x7f75ac147080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.233+0000 7f75b17c7700 1 --2- 172.21.15.46:0/2427517204 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75ac0b2b40 0x7f75ac156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.233+0000 7f75b17c7700 1 --2- 172.21.15.46:0/2427517204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75ac0c6520 0x7f75ac0cc6b0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.233+0000 7f75b17c7700 1 -- 172.21.15.46:0/2427517204 >> 172.21.15.46:0/2427517204 conn(0x7f75ac099a30 msgr2=0x7f75ac155630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:40.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.233+0000 7f75b17c7700 1 -- 172.21.15.46:0/2427517204 shutdown_connections 2024-01-26T23:45:40.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.233+0000 7f75b17c7700 1 -- 172.21.15.46:0/2427517204 wait complete. 2024-01-26T23:45:40.248 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-26T23:45:40.344 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:40.344+0000 7f023aa4e1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:45:40.380 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:40.379+0000 7f70091c51c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:45:40.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.382+0000 7fdec9c78700 1 Processor -- start 2024-01-26T23:45:40.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.382+0000 7fdec9c78700 1 -- start start 2024-01-26T23:45:40.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.383+0000 7fdec9c78700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdec4159960 0x7fdec406bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.383+0000 7fdec9c78700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdec406c1f0 0x7fdec4069370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.383+0000 7fdec9c78700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec415dac0 con 0x7fdec4159590 2024-01-26T23:45:40.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.383+0000 7fdec9c78700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec415e350 con 0x7fdec406c1f0 2024-01-26T23:45:40.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.383+0000 7fdec9c78700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec40c97c0 con 0x7fdec4159960 2024-01-26T23:45:40.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.383+0000 7fdec3fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdec406c1f0 0x7fdec4069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:40.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.384+0000 7fdebaffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdec4159960 0x7fdec406bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:40.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.384+0000 7fdec37fe700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fdec4159590 0x7fdec415d3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:39686/0 (socket says 172.21.15.46:39686) 2024-01-26T23:45:40.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.384+0000 7fdec37fe700 1 -- 172.21.15.46:0/1584828020 learned_addr learned my addr 172.21.15.46:0/1584828020 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:40.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.384+0000 7fdec2ffd700 1 -- 172.21.15.46:0/1584828020 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226922159 0 0) 0x7fdec415dac0 con 0x7fdec4159590 2024-01-26T23:45:40.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.384+0000 7fdec3fff700 1 -- 172.21.15.46:0/1584828020 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdec4159960 msgr2=0x7fdec406bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:40.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.384+0000 7fdec3fff700 1 --2- 172.21.15.46:0/1584828020 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdec4159960 0x7fdec406bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.384+0000 7fdec3fff700 1 -- 172.21.15.46:0/1584828020 >> v1:172.21.15.106:6789/0 conn(0x7fdec4159590 legacy=0x7fdec415d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:40.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.384+0000 7fdec3fff700 1 -- 172.21.15.46:0/1584828020 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec415dd90 con 0x7fdec406c1f0 2024-01-26T23:45:40.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.384+0000 7fdec3fff700 1 --2- 172.21.15.46:0/1584828020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec406c1f0 0x7fdec4069370 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fdebc009e60 tx=0x7fdebc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=139538c5052f8daf server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:40.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.384+0000 7fdec2ffd700 1 -- 172.21.15.46:0/1584828020 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc00e070 con 0x7fdec406c1f0 2024-01-26T23:45:40.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.384+0000 7fdec2ffd700 1 -- 172.21.15.46:0/1584828020 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdebc00bb60 con 0x7fdec406c1f0 2024-01-26T23:45:40.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.384+0000 7fdec2ffd700 1 -- 172.21.15.46:0/1584828020 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc012430 con 0x7fdec406c1f0 2024-01-26T23:45:40.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.385+0000 7fdec9c78700 1 -- 172.21.15.46:0/1584828020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec406c1f0 msgr2=0x7fdec4069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:40.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.385+0000 7fdec9c78700 1 --2- 172.21.15.46:0/1584828020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec406c1f0 0x7fdec4069370 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fdebc009e60 tx=0x7fdebc009780 comp rx=0 tx=0).stop 2024-01-26T23:45:40.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.385+0000 7fdec9c78700 1 -- 172.21.15.46:0/1584828020 shutdown_connections 2024-01-26T23:45:40.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.385+0000 7fdec9c78700 1 --2- 172.21.15.46:0/1584828020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec406c1f0 0x7fdec4069370 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.385+0000 7fdec9c78700 1 --2- 172.21.15.46:0/1584828020 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdec4159960 0x7fdec406bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.385+0000 7fdec9c78700 1 -- 172.21.15.46:0/1584828020 >> 172.21.15.46:0/1584828020 conn(0x7fdec40ce530 msgr2=0x7fdec40ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:40.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.385+0000 7fdec9c78700 1 -- 172.21.15.46:0/1584828020 shutdown_connections 2024-01-26T23:45:40.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.385+0000 7fdec9c78700 1 -- 172.21.15.46:0/1584828020 wait complete. 2024-01-26T23:45:40.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.386+0000 7fdec9c78700 1 Processor -- start 2024-01-26T23:45:40.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.387+0000 7fdec9c78700 1 -- start start 2024-01-26T23:45:40.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.387+0000 7fdec9c78700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec406c1f0 0x7fdec4157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.387+0000 7fdec9c78700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdec4159590 0x7fdec40c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.387+0000 7fdec9c78700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdec4159960 0x7fdec40c6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.387+0000 7fdec9c78700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec415cc90 con 0x7fdec406c1f0 2024-01-26T23:45:40.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.387+0000 7fdec9c78700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec40c3670 con 0x7fdec4159960 2024-01-26T23:45:40.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.387+0000 7fdec9c78700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec415c9b0 con 0x7fdec4159590 2024-01-26T23:45:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.388+0000 7fdec37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec406c1f0 0x7fdec4157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.388+0000 7fdec37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec406c1f0 0x7fdec4157930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35844/0 (socket says 172.21.15.46:35844) 2024-01-26T23:45:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.388+0000 7fdec37fe700 1 -- 172.21.15.46:0/353488528 learned_addr learned my addr 172.21.15.46:0/353488528 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.388+0000 7fdec3fff700 1 --2- 172.21.15.46:0/353488528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdec4159960 0x7fdec40c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.388+0000 7fdebaffd700 1 --2- 172.21.15.46:0/353488528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdec4159590 0x7fdec40c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.388+0000 7fdec37fe700 1 -- 172.21.15.46:0/353488528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdec4159590 msgr2=0x7fdec40c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.388+0000 7fdec37fe700 1 --2- 172.21.15.46:0/353488528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdec4159590 0x7fdec40c5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.388+0000 7fdec37fe700 1 -- 172.21.15.46:0/353488528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdec4159960 msgr2=0x7fdec40c6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.388+0000 7fdec37fe700 1 --2- 172.21.15.46:0/353488528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdec4159960 0x7fdec40c6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.388+0000 7fdec37fe700 1 -- 172.21.15.46:0/353488528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdebc009a20 con 0x7fdec406c1f0 2024-01-26T23:45:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.388+0000 7fdec37fe700 1 --2- 172.21.15.46:0/353488528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec406c1f0 0x7fdec4157930 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400c5e0 tx=0x7fdeb4001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.388+0000 7fdebbfff700 1 -- 172.21.15.46:0/353488528 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4005b30 con 0x7fdec406c1f0 2024-01-26T23:45:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.388+0000 7fdebbfff700 1 -- 172.21.15.46:0/353488528 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb4005c90 con 0x7fdec406c1f0 2024-01-26T23:45:40.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.388+0000 7fdebbfff700 1 -- 172.21.15.46:0/353488528 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4007300 con 0x7fdec406c1f0 2024-01-26T23:45:40.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.388+0000 7fdec9c78700 1 -- 172.21.15.46:0/353488528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec40c3020 con 0x7fdec406c1f0 2024-01-26T23:45:40.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.388+0000 7fdec9c78700 1 -- 172.21.15.46:0/353488528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec40b12e0 con 0x7fdec406c1f0 2024-01-26T23:45:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.389+0000 7fdec9c78700 1 -- 172.21.15.46:0/353488528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec40c2e80 con 0x7fdec406c1f0 2024-01-26T23:45:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.390+0000 7fdebbfff700 1 -- 172.21.15.46:0/353488528 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fdeb401b3b0 con 0x7fdec406c1f0 2024-01-26T23:45:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.390+0000 7fdebbfff700 1 --2- 172.21.15.46:0/353488528 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdea4067050 0x7fdea4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.390+0000 7fdebbfff700 1 -- 172.21.15.46:0/353488528 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fdeb4083e80 con 0x7fdec406c1f0 2024-01-26T23:45:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.390+0000 7fdebaffd700 1 -- 172.21.15.46:0/353488528 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdea4067050 msgr2=0x7fdea4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.390+0000 7fdebaffd700 1 --2- 172.21.15.46:0/353488528 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdea4067050 0x7fdea4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:40.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.395+0000 7fdebbfff700 1 -- 172.21.15.46:0/353488528 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdec40c2e80 con 0x7fdec406c1f0 2024-01-26T23:45:40.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.591+0000 7fdebaffd700 1 -- 172.21.15.46:0/353488528 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdea4067050 msgr2=0x7fdea4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:40.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.591+0000 7fdebaffd700 1 --2- 172.21.15.46:0/353488528 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdea4067050 0x7fdea4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:45:40.599 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:40.598+0000 7f70091c51c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:45:40.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.650+0000 7fdec9c78700 1 -- 172.21.15.46:0/353488528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fdec40c2e80 con 0x7fdec406c1f0 2024-01-26T23:45:40.651 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-26T23:45:40.651+0000 7f33ffe86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-26T23:45:40.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.656+0000 7fdebbfff700 1 -- 172.21.15.46:0/353488528 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v37) v1 ==== 161+0+0 (secure 0 0 0) 0x7fdec40c2e80 con 0x7fdec406c1f0 2024-01-26T23:45:40.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.658+0000 7fdec9c78700 1 -- 172.21.15.46:0/353488528 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdea4067050 msgr2=0x7fdea4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:40.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.658+0000 7fdec9c78700 1 --2- 172.21.15.46:0/353488528 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdea4067050 0x7fdea4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.658+0000 7fdec9c78700 1 -- 172.21.15.46:0/353488528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec406c1f0 msgr2=0x7fdec4157930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:40.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.658+0000 7fdec9c78700 1 --2- 172.21.15.46:0/353488528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec406c1f0 0x7fdec4157930 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400c5e0 tx=0x7fdeb4001070 comp rx=0 tx=0).stop 2024-01-26T23:45:40.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.658+0000 7fdec9c78700 1 -- 172.21.15.46:0/353488528 shutdown_connections 2024-01-26T23:45:40.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.658+0000 7fdec9c78700 1 --2- 172.21.15.46:0/353488528 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fdea4067050 0x7fdea4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.658+0000 7fdec9c78700 1 --2- 172.21.15.46:0/353488528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec406c1f0 0x7fdec4157930 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.658+0000 7fdec9c78700 1 --2- 172.21.15.46:0/353488528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdec4159960 0x7fdec40c6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.658+0000 7fdec9c78700 1 --2- 172.21.15.46:0/353488528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdec4159590 0x7fdec40c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.658+0000 7fdec9c78700 1 -- 172.21.15.46:0/353488528 >> 172.21.15.46:0/353488528 conn(0x7fdec40ce530 msgr2=0x7fdec406e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:40.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.658+0000 7fdec9c78700 1 -- 172.21.15.46:0/353488528 shutdown_connections 2024-01-26T23:45:40.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.658+0000 7fdec9c78700 1 -- 172.21.15.46:0/353488528 wait complete. 2024-01-26T23:45:40.660 INFO:tasks.ceph.mds.a.smithi046.stderr: -2538> 2024-01-26T23:45:21.737+0000 7f3a3bca0700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-26T23:45:40.660 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-26T23:45:40.672 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:40.671+0000 7f70091c51c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:45:40.673 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-26T23:45:40.688 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-26T23:45:40.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.815+0000 7f6e44a72700 1 Processor -- start 2024-01-26T23:45:40.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.815+0000 7f6e44a72700 1 -- start start 2024-01-26T23:45:40.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.816+0000 7f6e44a72700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e401599d0 0x7f6e40159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.816+0000 7f6e44a72700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6e400ab1c0 0x7f6e400b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.816+0000 7f6e44a72700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e400b1a60 con 0x7f6e4015d8c0 2024-01-26T23:45:40.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.816+0000 7f6e44a72700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e400ab5f0 con 0x7f6e400ab1c0 2024-01-26T23:45:40.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.816+0000 7f6e44a72700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e400b0f40 con 0x7f6e401599d0 2024-01-26T23:45:40.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.816+0000 7f6e3ed9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6e4015d8c0 0x7f6e400b20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53128/0 (socket says 172.21.15.46:53128) 2024-01-26T23:45:40.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.816+0000 7f6e3ed9d700 1 -- 172.21.15.46:0/1722285662 learned_addr learned my addr 172.21.15.46:0/1722285662 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:40.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.817+0000 7f6e35d9b700 1 --2- 172.21.15.46:0/1722285662 >> v2:172.21.15.106:3300/0 conn(0x7f6e400ab1c0 0x7f6e400b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:40.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.817+0000 7f6e3e59c700 1 --2- 172.21.15.46:0/1722285662 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e401599d0 0x7f6e40159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:40.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.817+0000 7f6e3dd9b700 1 -- 172.21.15.46:0/1722285662 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086367766 0 0) 0x7f6e400b1a60 con 0x7f6e4015d8c0 2024-01-26T23:45:40.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.817+0000 7f6e3dd9b700 1 -- 172.21.15.46:0/1722285662 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e2c003570 con 0x7f6e4015d8c0 2024-01-26T23:45:40.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.817+0000 7f6e3e59c700 1 -- 172.21.15.46:0/1722285662 >> v2:172.21.15.106:3300/0 conn(0x7f6e400ab1c0 msgr2=0x7f6e400b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:40.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.817+0000 7f6e3e59c700 1 --2- 172.21.15.46:0/1722285662 >> v2:172.21.15.106:3300/0 conn(0x7f6e400ab1c0 0x7f6e400b0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.817+0000 7f6e3e59c700 1 -- 172.21.15.46:0/1722285662 >> v1:172.21.15.46:6789/0 conn(0x7f6e4015d8c0 legacy=0x7f6e400b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:40.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.817+0000 7f6e3e59c700 1 -- 172.21.15.46:0/1722285662 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e400b11c0 con 0x7f6e401599d0 2024-01-26T23:45:40.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.817+0000 7f6e35d9b700 1 --2- 172.21.15.46:0/1722285662 >> v2:172.21.15.106:3300/0 conn(0x7f6e400ab1c0 0x7f6e400b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:40.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.817+0000 7f6e3e59c700 1 --2- 172.21.15.46:0/1722285662 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e401599d0 0x7f6e40159da0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6e300098d0 tx=0x7f6e30009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7caad0e5f247dc9b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:40.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.817+0000 7f6e3dd9b700 1 -- 172.21.15.46:0/1722285662 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e30005c10 con 0x7f6e401599d0 2024-01-26T23:45:40.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.817+0000 7f6e3dd9b700 1 -- 172.21.15.46:0/1722285662 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e30005d70 con 0x7f6e401599d0 2024-01-26T23:45:40.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.817+0000 7f6e3dd9b700 1 -- 172.21.15.46:0/1722285662 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e30006240 con 0x7f6e401599d0 2024-01-26T23:45:40.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.818+0000 7f6e44a72700 1 -- 172.21.15.46:0/1722285662 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e401599d0 msgr2=0x7f6e40159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:40.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.818+0000 7f6e44a72700 1 --2- 172.21.15.46:0/1722285662 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e401599d0 0x7f6e40159da0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6e300098d0 tx=0x7f6e30009120 comp rx=0 tx=0).stop 2024-01-26T23:45:40.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.818+0000 7f6e44a72700 1 -- 172.21.15.46:0/1722285662 shutdown_connections 2024-01-26T23:45:40.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.818+0000 7f6e44a72700 1 --2- 172.21.15.46:0/1722285662 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e401599d0 0x7f6e40159da0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.818+0000 7f6e44a72700 1 --2- 172.21.15.46:0/1722285662 >> v2:172.21.15.106:3300/0 conn(0x7f6e400ab1c0 0x7f6e400b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.818+0000 7f6e44a72700 1 -- 172.21.15.46:0/1722285662 >> 172.21.15.46:0/1722285662 conn(0x7f6e4006f840 msgr2=0x7f6e40087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:40.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.818+0000 7f6e44a72700 1 -- 172.21.15.46:0/1722285662 shutdown_connections 2024-01-26T23:45:40.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.818+0000 7f6e44a72700 1 -- 172.21.15.46:0/1722285662 wait complete. 2024-01-26T23:45:40.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.820+0000 7f6e44a72700 1 Processor -- start 2024-01-26T23:45:40.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.820+0000 7f6e44a72700 1 -- start start 2024-01-26T23:45:40.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.820+0000 7f6e44a72700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6e400ab1c0 0x7f6e400ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.820+0000 7f6e44a72700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e401599d0 0x7f6e401580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.821+0000 7f6e44a72700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e4015d8c0 0x7f6e4015ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.821+0000 7f6e44a72700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4016b460 con 0x7f6e401599d0 2024-01-26T23:45:40.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.821+0000 7f6e44a72700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4009c970 con 0x7f6e400ab1c0 2024-01-26T23:45:40.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.821+0000 7f6e44a72700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e40158610 con 0x7f6e4015d8c0 2024-01-26T23:45:40.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.821+0000 7f6e35d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e401599d0 0x7f6e401580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:40.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.821+0000 7f6e35d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e401599d0 0x7f6e401580d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35852/0 (socket says 172.21.15.46:35852) 2024-01-26T23:45:40.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.821+0000 7f6e35d9b700 1 -- 172.21.15.46:0/506242541 learned_addr learned my addr 172.21.15.46:0/506242541 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:40.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.821+0000 7f6e3e59c700 1 --2- 172.21.15.46:0/506242541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6e400ab1c0 0x7f6e400ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:40.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.821+0000 7f6e3ed9d700 1 --2- 172.21.15.46:0/506242541 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e4015d8c0 0x7f6e4015ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:40.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.821+0000 7f6e3e59c700 1 -- 172.21.15.46:0/506242541 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e4015d8c0 msgr2=0x7f6e4015ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:40.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.821+0000 7f6e3e59c700 1 --2- 172.21.15.46:0/506242541 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e4015d8c0 0x7f6e4015ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.821+0000 7f6e3e59c700 1 -- 172.21.15.46:0/506242541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e401599d0 msgr2=0x7f6e401580d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:40.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.821+0000 7f6e3e59c700 1 --2- 172.21.15.46:0/506242541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e401599d0 0x7f6e401580d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.821+0000 7f6e3e59c700 1 -- 172.21.15.46:0/506242541 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e28009420 con 0x7f6e400ab1c0 2024-01-26T23:45:40.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.821+0000 7f6e3e59c700 1 --2- 172.21.15.46:0/506242541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6e400ab1c0 0x7f6e400ad5b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6e400686d0 tx=0x7f6e30016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:40.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.821+0000 7f6e36ffd700 1 -- 172.21.15.46:0/506242541 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3001a700 con 0x7f6e400ab1c0 2024-01-26T23:45:40.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.822+0000 7f6e36ffd700 1 -- 172.21.15.46:0/506242541 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3001a880 con 0x7f6e400ab1c0 2024-01-26T23:45:40.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.822+0000 7f6e36ffd700 1 -- 172.21.15.46:0/506242541 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3001ac40 con 0x7f6e400ab1c0 2024-01-26T23:45:40.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.822+0000 7f6e44a72700 1 -- 172.21.15.46:0/506242541 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e300093c0 con 0x7f6e400ab1c0 2024-01-26T23:45:40.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.822+0000 7f6e44a72700 1 -- 172.21.15.46:0/506242541 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4016ab30 con 0x7f6e400ab1c0 2024-01-26T23:45:40.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.823+0000 7f6e44a72700 1 -- 172.21.15.46:0/506242541 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4015d700 con 0x7f6e400ab1c0 2024-01-26T23:45:40.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.823+0000 7f6e36ffd700 1 -- 172.21.15.46:0/506242541 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f6e30011420 con 0x7f6e400ab1c0 2024-01-26T23:45:40.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.823+0000 7f6e36ffd700 1 --2- 172.21.15.46:0/506242541 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6e20067050 0x7f6e20069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:40.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.824+0000 7f6e35d9b700 1 -- 172.21.15.46:0/506242541 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6e20067050 msgr2=0x7f6e20069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:45:40.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.824+0000 7f6e36ffd700 1 -- 172.21.15.46:0/506242541 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6761+0+0 (secure 0 0 0) 0x7f6e30085f90 con 0x7f6e400ab1c0 2024-01-26T23:45:40.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.824+0000 7f6e35d9b700 1 --2- 172.21.15.46:0/506242541 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6e20067050 0x7f6e20069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:45:40.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.828+0000 7f6e36ffd700 1 -- 172.21.15.46:0/506242541 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6e300b75a0 con 0x7f6e400ab1c0 2024-01-26T23:45:40.829 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:40.828+0000 7f70091c51c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:45:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.850+0000 7f6e36ffd700 1 -- 172.21.15.46:0/506242541 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mgrmap(e 35) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f6e300530a0 con 0x7f6e400ab1c0 2024-01-26T23:45:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.851+0000 7f6e36ffd700 1 -- 172.21.15.46:0/506242541 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6e20067050 msgr2=0x7f6e20069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:45:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:40.851+0000 7f6e36ffd700 1 --2- 172.21.15.46:0/506242541 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6e20067050 0x7f6e20069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:40.909 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:40.909+0000 7f6f87ab3700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:40.909 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:40.909+0000 7f6f87ab3700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:40.909 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:40.909+0000 7f6f87ab3700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:40.910 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:40.909+0000 7f6f87ab3700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:40.910 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:40.909+0000 7f6f87ab3700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:40.910 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:40.910+0000 7f6f852ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:40.911 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:40.910+0000 7f6f852ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:40.911 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:40.910+0000 7f6f852ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:40.911 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:40.910+0000 7f6f852ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:40.911 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:40.910+0000 7f6f852ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:45:41.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.086+0000 7f6e44a72700 1 -- 172.21.15.46:0/506242541 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6e400b14a0 con 0x7f6e400ab1c0 2024-01-26T23:45:41.181 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:41.181+0000 7f023aa4e1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:45:41.432 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:41.432+0000 7f023aa4e1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:45:41.515 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:41.515+0000 7f023aa4e1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:45:41.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.656+0000 7f6e36ffd700 1 -- 172.21.15.46:0/506242541 <== mon.1 v2:172.21.15.106:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6e300138c0 con 0x7f6e400ab1c0 2024-01-26T23:45:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.658+0000 7f6e44a72700 1 -- 172.21.15.46:0/506242541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6e400ab1c0 msgr2=0x7f6e400ad5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.658+0000 7f6e44a72700 1 --2- 172.21.15.46:0/506242541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6e400ab1c0 0x7f6e400ad5b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6e400686d0 tx=0x7f6e30016000 comp rx=0 tx=0).stop 2024-01-26T23:45:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.658+0000 7f6e44a72700 1 -- 172.21.15.46:0/506242541 shutdown_connections 2024-01-26T23:45:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.658+0000 7f6e44a72700 1 --2- 172.21.15.46:0/506242541 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6e20067050 0x7f6e20069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.658+0000 7f6e44a72700 1 --2- 172.21.15.46:0/506242541 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e4015d8c0 0x7f6e4015ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.658+0000 7f6e44a72700 1 --2- 172.21.15.46:0/506242541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6e400ab1c0 0x7f6e400ad5b0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.658+0000 7f6e44a72700 1 --2- 172.21.15.46:0/506242541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e401599d0 0x7f6e401580d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.658+0000 7f6e44a72700 1 -- 172.21.15.46:0/506242541 >> 172.21.15.46:0/506242541 conn(0x7f6e4006f840 msgr2=0x7f6e40069cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.658+0000 7f6e44a72700 1 -- 172.21.15.46:0/506242541 shutdown_connections 2024-01-26T23:45:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.658+0000 7f6e44a72700 1 -- 172.21.15.46:0/506242541 wait complete. 2024-01-26T23:45:41.673 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-26T23:45:41.695 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:45:41.695+0000 7f023aa4e1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:45:41.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.830+0000 7fb4b18e6700 1 Processor -- start 2024-01-26T23:45:41.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.830+0000 7fb4b18e6700 1 -- start start 2024-01-26T23:45:41.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.831+0000 7fb4b18e6700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb4ac0ccf40 0x7fb4ac0c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:41.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.831+0000 7fb4b18e6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4ac069c10 0x7fb4ac06a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:41.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.831+0000 7fb4b18e6700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4ac0c9140 con 0x7fb4ac159730 2024-01-26T23:45:41.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.831+0000 7fb4b18e6700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4ac068b70 con 0x7fb4ac069c10 2024-01-26T23:45:41.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.831+0000 7fb4b18e6700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4ac068e50 con 0x7fb4ac0ccf40 2024-01-26T23:45:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.831+0000 7fb4ab7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4ac069c10 0x7fb4ac06a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.831+0000 7fb4ab7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4ac069c10 0x7fb4ac06a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35858/0 (socket says 172.21.15.46:35858) 2024-01-26T23:45:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.831+0000 7fb4ab7fe700 1 -- 172.21.15.46:0/3403151937 learned_addr learned my addr 172.21.15.46:0/3403151937 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.831+0000 7fb4aaffd700 1 --2- 172.21.15.46:0/3403151937 >> v2:172.21.15.106:3300/0 conn(0x7fb4ac0ccf40 0x7fb4ac0c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.832+0000 7fb4ab7fe700 1 -- 172.21.15.46:0/3403151937 >> v2:172.21.15.106:3300/0 conn(0x7fb4ac0ccf40 msgr2=0x7fb4ac0c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.832+0000 7fb4ab7fe700 1 --2- 172.21.15.46:0/3403151937 >> v2:172.21.15.106:3300/0 conn(0x7fb4ac0ccf40 0x7fb4ac0c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.832+0000 7fb4ab7fe700 1 -- 172.21.15.46:0/3403151937 >> v1:172.21.15.106:6789/0 conn(0x7fb4ac159730 legacy=0x7fb4ac159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.832+0000 7fb4ab7fe700 1 -- 172.21.15.46:0/3403151937 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ac069130 con 0x7fb4ac069c10 2024-01-26T23:45:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.832+0000 7fb4ab7fe700 1 --2- 172.21.15.46:0/3403151937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4ac069c10 0x7fb4ac06a050 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4009f20 tx=0x7fb4a4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1c41bdf3cf9c7c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.832+0000 7fb4a9ffb700 1 -- 172.21.15.46:0/3403151937 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a400e070 con 0x7fb4ac069c10 2024-01-26T23:45:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.832+0000 7fb4a9ffb700 1 -- 172.21.15.46:0/3403151937 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a400ba00 con 0x7fb4ac069c10 2024-01-26T23:45:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.832+0000 7fb4a9ffb700 1 -- 172.21.15.46:0/3403151937 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a40092e0 con 0x7fb4ac069c10 2024-01-26T23:45:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.832+0000 7fb4b18e6700 1 -- 172.21.15.46:0/3403151937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4ac069c10 msgr2=0x7fb4ac06a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:41.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.832+0000 7fb4b18e6700 1 --2- 172.21.15.46:0/3403151937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4ac069c10 0x7fb4ac06a050 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4009f20 tx=0x7fb4a4009840 comp rx=0 tx=0).stop 2024-01-26T23:45:41.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.833+0000 7fb4b18e6700 1 -- 172.21.15.46:0/3403151937 shutdown_connections 2024-01-26T23:45:41.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.833+0000 7fb4b18e6700 1 --2- 172.21.15.46:0/3403151937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4ac069c10 0x7fb4ac06a050 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:41.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.833+0000 7fb4b18e6700 1 --2- 172.21.15.46:0/3403151937 >> v2:172.21.15.106:3300/0 conn(0x7fb4ac0ccf40 0x7fb4ac0c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:41.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.833+0000 7fb4b18e6700 1 -- 172.21.15.46:0/3403151937 >> 172.21.15.46:0/3403151937 conn(0x7fb4ac087f60 msgr2=0x7fb4ac0ce5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:41.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.833+0000 7fb4b18e6700 1 -- 172.21.15.46:0/3403151937 shutdown_connections 2024-01-26T23:45:41.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.833+0000 7fb4b18e6700 1 -- 172.21.15.46:0/3403151937 wait complete. 2024-01-26T23:45:41.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.834+0000 7fb4b18e6700 1 Processor -- start 2024-01-26T23:45:41.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.834+0000 7fb4b18e6700 1 -- start start 2024-01-26T23:45:41.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4b18e6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4ac069c10 0x7fb4ac0b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4b18e6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4ac0ccf40 0x7fb4ac0b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4b18e6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4ac159730 0x7fb4ac169a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4b18e6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac0b2320 con 0x7fb4ac159730 2024-01-26T23:45:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4b18e6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac0b1a80 con 0x7fb4ac0ccf40 2024-01-26T23:45:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4b18e6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac0b1d60 con 0x7fb4ac069c10 2024-01-26T23:45:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4ab7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4ac159730 0x7fb4ac169a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4ab7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4ac159730 0x7fb4ac169a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35872/0 (socket says 172.21.15.46:35872) 2024-01-26T23:45:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4ab7fe700 1 -- 172.21.15.46:0/1173168526 learned_addr learned my addr 172.21.15.46:0/1173168526 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4aa7fc700 1 --2- 172.21.15.46:0/1173168526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4ac0ccf40 0x7fb4ac0b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4aaffd700 1 --2- 172.21.15.46:0/1173168526 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4ac069c10 0x7fb4ac0b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4ab7fe700 1 -- 172.21.15.46:0/1173168526 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4ac069c10 msgr2=0x7fb4ac0b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4ab7fe700 1 --2- 172.21.15.46:0/1173168526 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4ac069c10 0x7fb4ac0b3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4ab7fe700 1 -- 172.21.15.46:0/1173168526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4ac0ccf40 msgr2=0x7fb4ac0b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4ab7fe700 1 --2- 172.21.15.46:0/1173168526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4ac0ccf40 0x7fb4ac0b1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4ab7fe700 1 -- 172.21.15.46:0/1173168526 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a4009ae0 con 0x7fb4ac159730 2024-01-26T23:45:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4aa7fc700 1 --2- 172.21.15.46:0/1173168526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4ac0ccf40 0x7fb4ac0b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:45:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.835+0000 7fb4aaffd700 1 --2- 172.21.15.46:0/1173168526 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4ac069c10 0x7fb4ac0b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:45:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.836+0000 7fb4ab7fe700 1 --2- 172.21.15.46:0/1173168526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4ac159730 0x7fb4ac169a30 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac06eed0 tx=0x7fb4a4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.836+0000 7fb4a2ffd700 1 -- 172.21.15.46:0/1173168526 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4012ea0 con 0x7fb4ac159730 2024-01-26T23:45:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.836+0000 7fb4a2ffd700 1 -- 172.21.15.46:0/1173168526 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a400bed0 con 0x7fb4ac159730 2024-01-26T23:45:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.836+0000 7fb4b18e6700 1 -- 172.21.15.46:0/1173168526 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4ac0b2040 con 0x7fb4ac159730 2024-01-26T23:45:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.836+0000 7fb4a2ffd700 1 -- 172.21.15.46:0/1173168526 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a401d650 con 0x7fb4ac159730 2024-01-26T23:45:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.836+0000 7fb4b18e6700 1 -- 172.21.15.46:0/1173168526 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4ac169f70 con 0x7fb4ac159730 2024-01-26T23:45:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.837+0000 7fb4b18e6700 1 -- 172.21.15.46:0/1173168526 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4ac06f960 con 0x7fb4ac159730 2024-01-26T23:45:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.837+0000 7fb4a2ffd700 1 -- 172.21.15.46:0/1173168526 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 35) v1 ==== 84453+0+0 (secure 0 0 0) 0x7fb4a401da80 con 0x7fb4ac159730 2024-01-26T23:45:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.838+0000 7fb4a2ffd700 1 -- 172.21.15.46:0/1173168526 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 7019+0+0 (secure 0 0 0) 0x7fb4a4085e20 con 0x7fb4ac159730 2024-01-26T23:45:41.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.842+0000 7fb4a2ffd700 1 -- 172.21.15.46:0/1173168526 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb4a4052f30 con 0x7fb4ac159730 2024-01-26T23:45:41.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.854+0000 7fb4a2ffd700 1 -- 172.21.15.46:0/1173168526 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 36) v1 ==== 84580+0+0 (secure 0 0 0) 0x7fb4a40529e0 con 0x7fb4ac159730 2024-01-26T23:45:41.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.854+0000 7fb4a2ffd700 1 --2- 172.21.15.46:0/1173168526 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb48c06cf20 0x7fb48c06f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:41.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.854+0000 7fb4aaffd700 1 --2- 172.21.15.46:0/1173168526 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb48c06cf20 0x7fb48c06f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:41.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:41.855+0000 7fb4aaffd700 1 --2- 172.21.15.46:0/1173168526 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb48c06cf20 0x7fb48c06f300 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb49c006aa0 tx=0x7fb49c008040 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:42.051+0000 7fb4b18e6700 1 -- 172.21.15.46:0/1173168526 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb4ac04c710 con 0x7fb4ac159730 2024-01-26T23:45:42.889 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:42.888+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:42.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:42.893+0000 7fb4a2ffd700 1 -- 172.21.15.46:0/1173168526 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v89) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb4a406a590 con 0x7fb4ac159730 2024-01-26T23:45:42.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:42.899+0000 7fb4b18e6700 1 -- 172.21.15.46:0/1173168526 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb48c06cf20 msgr2=0x7fb48c06f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:42.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:42.899+0000 7fb4b18e6700 1 --2- 172.21.15.46:0/1173168526 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb48c06cf20 0x7fb48c06f300 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb49c006aa0 tx=0x7fb49c008040 comp rx=0 tx=0).stop 2024-01-26T23:45:42.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:42.899+0000 7fb4b18e6700 1 -- 172.21.15.46:0/1173168526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4ac159730 msgr2=0x7fb4ac169a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:42.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:42.899+0000 7fb4b18e6700 1 --2- 172.21.15.46:0/1173168526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4ac159730 0x7fb4ac169a30 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac06eed0 tx=0x7fb4a4017000 comp rx=0 tx=0).stop 2024-01-26T23:45:42.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:42.899+0000 7fb4b18e6700 1 -- 172.21.15.46:0/1173168526 shutdown_connections 2024-01-26T23:45:42.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:42.899+0000 7fb4b18e6700 1 --2- 172.21.15.46:0/1173168526 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb48c06cf20 0x7fb48c06f300 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:42.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:42.899+0000 7fb4b18e6700 1 --2- 172.21.15.46:0/1173168526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4ac159730 0x7fb4ac169a30 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:42.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:42.899+0000 7fb4b18e6700 1 --2- 172.21.15.46:0/1173168526 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4ac069c10 0x7fb4ac0b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:42.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:42.899+0000 7fb4b18e6700 1 --2- 172.21.15.46:0/1173168526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4ac0ccf40 0x7fb4ac0b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:42.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:42.899+0000 7fb4b18e6700 1 -- 172.21.15.46:0/1173168526 >> 172.21.15.46:0/1173168526 conn(0x7fb4ac087f60 msgr2=0x7fb4ac156040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:42.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:42.899+0000 7fb4b18e6700 1 -- 172.21.15.46:0/1173168526 shutdown_connections 2024-01-26T23:45:42.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:42.899+0000 7fb4b18e6700 1 -- 172.21.15.46:0/1173168526 wait complete. 2024-01-26T23:45:42.900 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-01-26T23:45:42.915 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-26T23:45:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.057+0000 7fc31e177700 1 Processor -- start 2024-01-26T23:45:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.057+0000 7fc31e177700 1 -- start start 2024-01-26T23:45:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc31e177700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc3180cc1c0 0x7fc318164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc31e177700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3180cff70 con 0x7fc31815ce40 2024-01-26T23:45:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc31e177700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3180cf400 con 0x7fc31815d210 2024-01-26T23:45:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc31e177700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3180cf660 con 0x7fc3180cc1c0 2024-01-26T23:45:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc3177fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc31815ce40 0x7fc3180d0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53132/0 (socket says 172.21.15.46:53132) 2024-01-26T23:45:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc3177fe700 1 -- 172.21.15.46:0/2159165438 learned_addr learned my addr 172.21.15.46:0/2159165438 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc317fff700 1 --2- 172.21.15.46:0/2159165438 >> v2:172.21.15.106:3300/0 conn(0x7fc3180cc1c0 0x7fc318164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc316ffd700 1 -- 172.21.15.46:0/2159165438 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094140121 0 0) 0x7fc3180cff70 con 0x7fc31815ce40 2024-01-26T23:45:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc316ffd700 1 -- 172.21.15.46:0/2159165438 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc304003570 con 0x7fc31815ce40 2024-01-26T23:45:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc316ffd700 1 -- 172.21.15.46:0/2159165438 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2000702623 0 0) 0x7fc304003570 con 0x7fc31815ce40 2024-01-26T23:45:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc316ffd700 1 -- 172.21.15.46:0/2159165438 >> v2:172.21.15.106:3300/0 conn(0x7fc3180cc1c0 msgr2=0x7fc318164700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc316ffd700 1 --2- 172.21.15.46:0/2159165438 >> v2:172.21.15.106:3300/0 conn(0x7fc3180cc1c0 0x7fc318164700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc316ffd700 1 -- 172.21.15.46:0/2159165438 >> v1:172.21.15.106:6789/0 conn(0x7fc31815d210 legacy=0x7fc3180cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc316ffd700 1 -- 172.21.15.46:0/2159165438 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3180d0500 con 0x7fc31815ce40 2024-01-26T23:45:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc316ffd700 1 -- 172.21.15.46:0/2159165438 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fc308001470 con 0x7fc31815ce40 2024-01-26T23:45:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc316ffd700 1 -- 172.21.15.46:0/2159165438 >> v1:172.21.15.46:6789/0 conn(0x7fc31815ce40 legacy=0x7fc3180d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc316ffd700 1 --2- 172.21.15.46:0/2159165438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc304003a60 0x7fc304003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc316ffd700 1 --2- 172.21.15.46:0/2159165438 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc304004810 0x7fc304004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc316ffd700 1 --2- 172.21.15.46:0/2159165438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3040051c0 0x7fc30400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc316ffd700 1 -- 172.21.15.46:0/2159165438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3180cf660 con 0x7fc3040051c0 2024-01-26T23:45:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.058+0000 7fc316ffd700 1 -- 172.21.15.46:0/2159165438 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30400c160 con 0x7fc304003a60 2024-01-26T23:45:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.059+0000 7fc316ffd700 1 -- 172.21.15.46:0/2159165438 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc30400c320 con 0x7fc304004810 2024-01-26T23:45:43.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.059+0000 7fc317fff700 1 --2- 172.21.15.46:0/2159165438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3040051c0 0x7fc30400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:43.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.059+0000 7fc3177fe700 1 --2- 172.21.15.46:0/2159165438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc304003a60 0x7fc304003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:43.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.059+0000 7fc30effd700 1 --2- 172.21.15.46:0/2159165438 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc304004810 0x7fc304004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:43.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.059+0000 7fc317fff700 1 -- 172.21.15.46:0/2159165438 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc304004810 msgr2=0x7fc304004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:43.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.059+0000 7fc317fff700 1 --2- 172.21.15.46:0/2159165438 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc304004810 0x7fc304004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:43.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.059+0000 7fc317fff700 1 -- 172.21.15.46:0/2159165438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc304003a60 msgr2=0x7fc304003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:43.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.059+0000 7fc317fff700 1 --2- 172.21.15.46:0/2159165438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc304003a60 0x7fc304003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:43.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.059+0000 7fc317fff700 1 -- 172.21.15.46:0/2159165438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc304003570 con 0x7fc3040051c0 2024-01-26T23:45:43.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.059+0000 7fc3177fe700 1 --2- 172.21.15.46:0/2159165438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc304003a60 0x7fc304003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:43.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.059+0000 7fc317fff700 1 -- 172.21.15.46:0/2159165438 reap_dead start 2024-01-26T23:45:43.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.059+0000 7fc317fff700 1 --2- 172.21.15.46:0/2159165438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3040051c0 0x7fc30400ba60 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc31806ac20 tx=0x7fc31000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f1c95ac9eecde6 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:43.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.059+0000 7fc316ffd700 1 -- 172.21.15.46:0/2159165438 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31000ea60 con 0x7fc3040051c0 2024-01-26T23:45:43.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.059+0000 7fc316ffd700 1 -- 172.21.15.46:0/2159165438 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc31000f040 con 0x7fc3040051c0 2024-01-26T23:45:43.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.059+0000 7fc316ffd700 1 -- 172.21.15.46:0/2159165438 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31000ee10 con 0x7fc3040051c0 2024-01-26T23:45:43.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.060+0000 7fc31e177700 1 -- 172.21.15.46:0/2159165438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3040051c0 msgr2=0x7fc30400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:43.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.060+0000 7fc31e177700 1 --2- 172.21.15.46:0/2159165438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3040051c0 0x7fc30400ba60 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc31806ac20 tx=0x7fc31000b040 comp rx=0 tx=0).stop 2024-01-26T23:45:43.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.060+0000 7fc31e177700 1 -- 172.21.15.46:0/2159165438 shutdown_connections 2024-01-26T23:45:43.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.060+0000 7fc31e177700 1 --2- 172.21.15.46:0/2159165438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3040051c0 0x7fc30400ba60 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:43.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.060+0000 7fc31e177700 1 -- 172.21.15.46:0/2159165438 >> 172.21.15.46:0/2159165438 conn(0x7fc31806f840 msgr2=0x7fc3180d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:43.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.060+0000 7fc31e177700 1 -- 172.21.15.46:0/2159165438 shutdown_connections 2024-01-26T23:45:43.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.060+0000 7fc31e177700 1 -- 172.21.15.46:0/2159165438 wait complete. 2024-01-26T23:45:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.061+0000 7fc31e177700 1 Processor -- start 2024-01-26T23:45:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.062+0000 7fc31e177700 1 -- start start 2024-01-26T23:45:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc31e177700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3040051c0 0x7fc3180c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc31e177700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3180c8600 0x7fc318152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc31e177700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3180b36f0 0x7fc3180b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc31e177700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3180b40d0 con 0x7fc3180b36f0 2024-01-26T23:45:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc31e177700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3180ca1f0 con 0x7fc3040051c0 2024-01-26T23:45:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc31e177700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3180c8a70 con 0x7fc3180c8600 2024-01-26T23:45:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc317fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3180b36f0 0x7fc3180b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc317fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3180b36f0 0x7fc3180b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35902/0 (socket says 172.21.15.46:35902) 2024-01-26T23:45:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc317fff700 1 -- 172.21.15.46:0/1876676372 learned_addr learned my addr 172.21.15.46:0/1876676372 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc30effd700 1 --2- 172.21.15.46:0/1876676372 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3180c8600 0x7fc318152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc3177fe700 1 --2- 172.21.15.46:0/1876676372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3040051c0 0x7fc3180c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc317fff700 1 -- 172.21.15.46:0/1876676372 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3180c8600 msgr2=0x7fc318152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc317fff700 1 --2- 172.21.15.46:0/1876676372 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3180c8600 0x7fc318152fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc317fff700 1 -- 172.21.15.46:0/1876676372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3040051c0 msgr2=0x7fc3180c8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc317fff700 1 --2- 172.21.15.46:0/1876676372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3040051c0 0x7fc3180c8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc317fff700 1 -- 172.21.15.46:0/1876676372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3180c95e0 con 0x7fc3180b36f0 2024-01-26T23:45:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc317fff700 1 --2- 172.21.15.46:0/1876676372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3180b36f0 0x7fc3180b3b90 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc31000afd0 tx=0x7fc310019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc30ffff700 1 -- 172.21.15.46:0/1876676372 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31001d660 con 0x7fc3180b36f0 2024-01-26T23:45:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc30ffff700 1 -- 172.21.15.46:0/1876676372 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc310022070 con 0x7fc3180b36f0 2024-01-26T23:45:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc31e177700 1 -- 172.21.15.46:0/1876676372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3180c98c0 con 0x7fc3180b36f0 2024-01-26T23:45:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.063+0000 7fc30ffff700 1 -- 172.21.15.46:0/1876676372 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31001da10 con 0x7fc3180b36f0 2024-01-26T23:45:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.064+0000 7fc31e177700 1 -- 172.21.15.46:0/1876676372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3180c9e80 con 0x7fc3180b36f0 2024-01-26T23:45:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.064+0000 7fc31e177700 1 -- 172.21.15.46:0/1876676372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc31804c710 con 0x7fc3180b36f0 2024-01-26T23:45:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.065+0000 7fc30ffff700 1 -- 172.21.15.46:0/1876676372 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 36) v1 ==== 84580+0+0 (secure 0 0 0) 0x7fc310003020 con 0x7fc3180b36f0 2024-01-26T23:45:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.065+0000 7fc30ffff700 1 --2- 172.21.15.46:0/1876676372 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc2f8066d80 0x7fc2f8069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.066+0000 7fc30ffff700 1 -- 172.21.15.46:0/1876676372 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6581+0+0 (secure 0 0 0) 0x7fc310024070 con 0x7fc3180b36f0 2024-01-26T23:45:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.066+0000 7fc3177fe700 1 --2- 172.21.15.46:0/1876676372 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc2f8066d80 0x7fc2f8069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.066+0000 7fc3177fe700 1 --2- 172.21.15.46:0/1876676372 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc2f8066d80 0x7fc2f8069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc308000fd0 tx=0x7fc308001e50 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:43.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.070+0000 7fc30ffff700 1 -- 172.21.15.46:0/1876676372 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc3100537b0 con 0x7fc3180b36f0 2024-01-26T23:45:43.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.282+0000 7fc31e177700 1 -- 172.21.15.46:0/1876676372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc3180cf920 con 0x7fc3180b36f0 2024-01-26T23:45:43.550 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:43.549+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:43.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.861+0000 7fc30ffff700 1 -- 172.21.15.46:0/1876676372 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fc310053b80 con 0x7fc3180b36f0 2024-01-26T23:45:43.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.895+0000 7fc30ffff700 1 -- 172.21.15.46:0/1876676372 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v90) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc310057240 con 0x7fc3180b36f0 2024-01-26T23:45:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.902+0000 7fc31e177700 1 -- 172.21.15.46:0/1876676372 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc2f8066d80 msgr2=0x7fc2f8069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.902+0000 7fc31e177700 1 --2- 172.21.15.46:0/1876676372 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc2f8066d80 0x7fc2f8069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc308000fd0 tx=0x7fc308001e50 comp rx=0 tx=0).stop 2024-01-26T23:45:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.902+0000 7fc31e177700 1 -- 172.21.15.46:0/1876676372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3180b36f0 msgr2=0x7fc3180b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.902+0000 7fc31e177700 1 --2- 172.21.15.46:0/1876676372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3180b36f0 0x7fc3180b3b90 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc31000afd0 tx=0x7fc310019000 comp rx=0 tx=0).stop 2024-01-26T23:45:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.902+0000 7fc31e177700 1 -- 172.21.15.46:0/1876676372 shutdown_connections 2024-01-26T23:45:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.902+0000 7fc31e177700 1 --2- 172.21.15.46:0/1876676372 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc2f8066d80 0x7fc2f8069230 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.902+0000 7fc31e177700 1 --2- 172.21.15.46:0/1876676372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3180b36f0 0x7fc3180b3b90 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.902+0000 7fc31e177700 1 --2- 172.21.15.46:0/1876676372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3040051c0 0x7fc3180c8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.902+0000 7fc31e177700 1 --2- 172.21.15.46:0/1876676372 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3180c8600 0x7fc318152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.902+0000 7fc31e177700 1 -- 172.21.15.46:0/1876676372 >> 172.21.15.46:0/1876676372 conn(0x7fc31806f840 msgr2=0x7fc3180b5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:43.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.903+0000 7fc31e177700 1 -- 172.21.15.46:0/1876676372 shutdown_connections 2024-01-26T23:45:43.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:43.904+0000 7fc31e177700 1 -- 172.21.15.46:0/1876676372 wait complete. 2024-01-26T23:45:43.906 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-01-26T23:45:43.921 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-26T23:45:44.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.062+0000 7f3710659700 1 Processor -- start 2024-01-26T23:45:44.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.062+0000 7f3710659700 1 -- start start 2024-01-26T23:45:44.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.063+0000 7f3710659700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f37080cf050 0x7f37080af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:44.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.063+0000 7f3710659700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f37080cf4a0 0x7f37080af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:44.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.063+0000 7f3710659700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37080cc100 con 0x7f3708159cb0 2024-01-26T23:45:44.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.063+0000 7f3710659700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37080ccca0 con 0x7f37080cf050 2024-01-26T23:45:44.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.063+0000 7f3710659700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37080cb850 con 0x7f37080cf4a0 2024-01-26T23:45:44.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.063+0000 7f370dbf4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f37080cf050 0x7f37080af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:44.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.063+0000 7f370ebf6700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f37080cf4a0 0x7f37080af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:44.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.063+0000 7f370dbf4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f37080cf050 0x7f37080af430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35918/0 (socket says 172.21.15.46:35918) 2024-01-26T23:45:44.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.063+0000 7f370e3f5700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f3708159cb0 0x7f37080cc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:39718/0 (socket says 172.21.15.46:39718) 2024-01-26T23:45:44.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.063+0000 7f370dbf4700 1 -- 172.21.15.46:0/184502343 learned_addr learned my addr 172.21.15.46:0/184502343 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:44.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.063+0000 7f370dbf4700 1 -- 172.21.15.46:0/184502343 >> v2:172.21.15.106:3300/0 conn(0x7f37080cf4a0 msgr2=0x7f37080af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:44.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.063+0000 7f370dbf4700 1 --2- 172.21.15.46:0/184502343 >> v2:172.21.15.106:3300/0 conn(0x7f37080cf4a0 0x7f37080af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.063+0000 7f370dbf4700 1 -- 172.21.15.46:0/184502343 >> v1:172.21.15.106:6789/0 conn(0x7f3708159cb0 legacy=0x7f37080cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:44.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.063+0000 7f370dbf4700 1 -- 172.21.15.46:0/184502343 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37080cbaf0 con 0x7f37080cf050 2024-01-26T23:45:44.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.064+0000 7f370dbf4700 1 --2- 172.21.15.46:0/184502343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37080cf050 0x7f37080af430 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f37080cef80 tx=0x7f36f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7b8fff1669dad3 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:44.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.064+0000 7f370d3f3700 1 -- 172.21.15.46:0/184502343 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f8009e70 con 0x7f37080cf050 2024-01-26T23:45:44.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.064+0000 7f370d3f3700 1 -- 172.21.15.46:0/184502343 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f800b860 con 0x7f37080cf050 2024-01-26T23:45:44.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.064+0000 7f370d3f3700 1 -- 172.21.15.46:0/184502343 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f800bc00 con 0x7f37080cf050 2024-01-26T23:45:44.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.064+0000 7f3710659700 1 -- 172.21.15.46:0/184502343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37080cf050 msgr2=0x7f37080af430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:44.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.064+0000 7f3710659700 1 --2- 172.21.15.46:0/184502343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37080cf050 0x7f37080af430 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f37080cef80 tx=0x7f36f8009250 comp rx=0 tx=0).stop 2024-01-26T23:45:44.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.064+0000 7f3710659700 1 -- 172.21.15.46:0/184502343 shutdown_connections 2024-01-26T23:45:44.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.064+0000 7f3710659700 1 --2- 172.21.15.46:0/184502343 >> v2:172.21.15.106:3300/0 conn(0x7f37080cf4a0 0x7f37080af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.064+0000 7f3710659700 1 --2- 172.21.15.46:0/184502343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37080cf050 0x7f37080af430 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.064+0000 7f3710659700 1 -- 172.21.15.46:0/184502343 >> 172.21.15.46:0/184502343 conn(0x7f3708087f60 msgr2=0x7f370806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:44.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.065+0000 7f3710659700 1 -- 172.21.15.46:0/184502343 shutdown_connections 2024-01-26T23:45:44.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.065+0000 7f3710659700 1 -- 172.21.15.46:0/184502343 wait complete. 2024-01-26T23:45:44.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.066+0000 7f3710659700 1 Processor -- start 2024-01-26T23:45:44.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.066+0000 7f3710659700 1 -- start start 2024-01-26T23:45:44.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.066+0000 7f3710659700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37080cf050 0x7f37080c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:44.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.066+0000 7f3710659700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37080cf4a0 0x7f370814ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:44.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f3710659700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3708159cb0 0x7f37080c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:44.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f3710659700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370814f100 con 0x7f37080cf4a0 2024-01-26T23:45:44.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f3710659700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37080c8950 con 0x7f37080cf050 2024-01-26T23:45:44.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f3710659700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370815cda0 con 0x7f3708159cb0 2024-01-26T23:45:44.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f370dbf4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37080cf4a0 0x7f370814ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:44.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f370dbf4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37080cf4a0 0x7f370814ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35922/0 (socket says 172.21.15.46:35922) 2024-01-26T23:45:44.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f370dbf4700 1 -- 172.21.15.46:0/1133453978 learned_addr learned my addr 172.21.15.46:0/1133453978 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:44.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f370e3f5700 1 --2- 172.21.15.46:0/1133453978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37080cf050 0x7f37080c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:44.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f370ebf6700 1 --2- 172.21.15.46:0/1133453978 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3708159cb0 0x7f37080c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:44.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f370dbf4700 1 -- 172.21.15.46:0/1133453978 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3708159cb0 msgr2=0x7f37080c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:44.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f370dbf4700 1 --2- 172.21.15.46:0/1133453978 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3708159cb0 0x7f37080c9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f370dbf4700 1 -- 172.21.15.46:0/1133453978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37080cf050 msgr2=0x7f37080c8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:44.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f370dbf4700 1 --2- 172.21.15.46:0/1133453978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37080cf050 0x7f37080c8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f370dbf4700 1 -- 172.21.15.46:0/1133453978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f80094f0 con 0x7f37080cf4a0 2024-01-26T23:45:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f370dbf4700 1 --2- 172.21.15.46:0/1133453978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37080cf4a0 0x7f370814ebc0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f3708088df0 tx=0x7f36f8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f36fe7fc700 1 -- 172.21.15.46:0/1133453978 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f800bbe0 con 0x7f37080cf4a0 2024-01-26T23:45:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f36fe7fc700 1 -- 172.21.15.46:0/1133453978 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f8000c90 con 0x7f37080cf4a0 2024-01-26T23:45:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f3710659700 1 -- 172.21.15.46:0/1133453978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37080c7d40 con 0x7f37080cf4a0 2024-01-26T23:45:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f36fe7fc700 1 -- 172.21.15.46:0/1133453978 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f8011650 con 0x7f37080cf4a0 2024-01-26T23:45:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.067+0000 7f3710659700 1 -- 172.21.15.46:0/1133453978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37080c8300 con 0x7f37080cf4a0 2024-01-26T23:45:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.068+0000 7f3710659700 1 -- 172.21.15.46:0/1133453978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37080c8ae0 con 0x7f37080cf4a0 2024-01-26T23:45:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.069+0000 7f36fe7fc700 1 -- 172.21.15.46:0/1133453978 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f36f8011a40 con 0x7f37080cf4a0 2024-01-26T23:45:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.069+0000 7f36fe7fc700 1 --2- 172.21.15.46:0/1133453978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36ec066ea0 0x7f36ec069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.069+0000 7f36fe7fc700 1 -- 172.21.15.46:0/1133453978 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6167+0+0 (secure 0 0 0) 0x7f36f8085bd0 con 0x7f37080cf4a0 2024-01-26T23:45:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.069+0000 7f370e3f5700 1 --2- 172.21.15.46:0/1133453978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36ec066ea0 0x7f36ec069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.070+0000 7f370e3f5700 1 --2- 172.21.15.46:0/1133453978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36ec066ea0 0x7f36ec069350 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3704000c00 tx=0x7f3704007ec0 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.074+0000 7f36fe7fc700 1 -- 172.21.15.46:0/1133453978 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f36f8052cb0 con 0x7f37080cf4a0 2024-01-26T23:45:44.216 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:44.215+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:44.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.281+0000 7f3710659700 1 -- 172.21.15.46:0/1133453978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f370804c710 con 0x7f37080cf4a0 2024-01-26T23:45:44.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.285+0000 7f36fe7fc700 1 -- 172.21.15.46:0/1133453978 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7f36f80562d0 con 0x7f37080cf4a0 2024-01-26T23:45:44.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.288+0000 7f3710659700 1 -- 172.21.15.46:0/1133453978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36ec066ea0 msgr2=0x7f36ec069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.288+0000 7f3710659700 1 --2- 172.21.15.46:0/1133453978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36ec066ea0 0x7f36ec069350 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3704000c00 tx=0x7f3704007ec0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.288+0000 7f3710659700 1 -- 172.21.15.46:0/1133453978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37080cf4a0 msgr2=0x7f370814ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.288+0000 7f3710659700 1 --2- 172.21.15.46:0/1133453978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37080cf4a0 0x7f370814ebc0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f3708088df0 tx=0x7f36f8017000 comp rx=0 tx=0).stop 2024-01-26T23:45:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.288+0000 7f3710659700 1 -- 172.21.15.46:0/1133453978 shutdown_connections 2024-01-26T23:45:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.288+0000 7f3710659700 1 --2- 172.21.15.46:0/1133453978 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f36ec066ea0 0x7f36ec069350 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.288+0000 7f3710659700 1 --2- 172.21.15.46:0/1133453978 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3708159cb0 0x7f37080c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.288+0000 7f3710659700 1 --2- 172.21.15.46:0/1133453978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37080cf050 0x7f37080c8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.288+0000 7f3710659700 1 --2- 172.21.15.46:0/1133453978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37080cf4a0 0x7f370814ebc0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.288+0000 7f3710659700 1 -- 172.21.15.46:0/1133453978 >> 172.21.15.46:0/1133453978 conn(0x7f3708087f60 msgr2=0x7f370815b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.288+0000 7f3710659700 1 -- 172.21.15.46:0/1133453978 shutdown_connections 2024-01-26T23:45:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.288+0000 7f3710659700 1 -- 172.21.15.46:0/1133453978 wait complete. 2024-01-26T23:45:44.302 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-26T23:45:44.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.436+0000 7f283d0d9700 1 Processor -- start 2024-01-26T23:45:44.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.436+0000 7f283d0d9700 1 -- start start 2024-01-26T23:45:44.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.437+0000 7f283d0d9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f283806ebe0 0x7f2838067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:44.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.437+0000 7f283d0d9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f28381597e0 0x7f2838159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:44.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.437+0000 7f283d0d9700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283806fdb0 con 0x7f2838067af0 2024-01-26T23:45:44.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.437+0000 7f283d0d9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f283806f1a0 con 0x7f283806ebe0 2024-01-26T23:45:44.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.437+0000 7f283d0d9700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f283806f480 con 0x7f28381597e0 2024-01-26T23:45:44.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.437+0000 7f283659c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f283806ebe0 0x7f2838067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:44.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.437+0000 7f2836d9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2838067af0 0x7f28380ad8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53146/0 (socket says 172.21.15.46:53146) 2024-01-26T23:45:44.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.437+0000 7f2836d9d700 1 -- 172.21.15.46:0/3434342135 learned_addr learned my addr 172.21.15.46:0/3434342135 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:44.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.437+0000 7f283759e700 1 --2- 172.21.15.46:0/3434342135 >> v2:172.21.15.106:3300/0 conn(0x7f28381597e0 0x7f2838159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:44.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.437+0000 7f283659c700 1 -- 172.21.15.46:0/3434342135 >> v2:172.21.15.106:3300/0 conn(0x7f28381597e0 msgr2=0x7f2838159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:44.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.437+0000 7f283659c700 1 --2- 172.21.15.46:0/3434342135 >> v2:172.21.15.106:3300/0 conn(0x7f28381597e0 0x7f2838159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.437+0000 7f283659c700 1 -- 172.21.15.46:0/3434342135 >> v1:172.21.15.46:6789/0 conn(0x7f2838067af0 legacy=0x7f28380ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:44.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.437+0000 7f283659c700 1 -- 172.21.15.46:0/3434342135 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283806f760 con 0x7f283806ebe0 2024-01-26T23:45:44.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.437+0000 7f2835d9b700 1 -- 172.21.15.46:0/3434342135 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330271523 0 0) 0x7f283806fdb0 con 0x7f2838067af0 2024-01-26T23:45:44.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.438+0000 7f283659c700 1 --2- 172.21.15.46:0/3434342135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f283806ebe0 0x7f2838067ec0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f283815eb20 tx=0x7f28200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc05993a370a0877 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:44.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.438+0000 7f2835d9b700 1 -- 172.21.15.46:0/3434342135 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2820011630 con 0x7f283806ebe0 2024-01-26T23:45:44.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.438+0000 7f2835d9b700 1 -- 172.21.15.46:0/3434342135 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2820011790 con 0x7f283806ebe0 2024-01-26T23:45:44.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.438+0000 7f2835d9b700 1 -- 172.21.15.46:0/3434342135 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2820011b30 con 0x7f283806ebe0 2024-01-26T23:45:44.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.438+0000 7f283d0d9700 1 -- 172.21.15.46:0/3434342135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f283806ebe0 msgr2=0x7f2838067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:44.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.438+0000 7f283d0d9700 1 --2- 172.21.15.46:0/3434342135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f283806ebe0 0x7f2838067ec0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f283815eb20 tx=0x7f28200091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.438+0000 7f283d0d9700 1 -- 172.21.15.46:0/3434342135 shutdown_connections 2024-01-26T23:45:44.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.438+0000 7f283d0d9700 1 --2- 172.21.15.46:0/3434342135 >> v2:172.21.15.106:3300/0 conn(0x7f28381597e0 0x7f2838159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.438+0000 7f283d0d9700 1 --2- 172.21.15.46:0/3434342135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f283806ebe0 0x7f2838067ec0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.438+0000 7f283d0d9700 1 -- 172.21.15.46:0/3434342135 >> 172.21.15.46:0/3434342135 conn(0x7f28380cdf80 msgr2=0x7f283806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:44.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.438+0000 7f283d0d9700 1 -- 172.21.15.46:0/3434342135 shutdown_connections 2024-01-26T23:45:44.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.438+0000 7f283d0d9700 1 -- 172.21.15.46:0/3434342135 wait complete. 2024-01-26T23:45:44.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.440+0000 7f283d0d9700 1 Processor -- start 2024-01-26T23:45:44.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.440+0000 7f283d0d9700 1 -- start start 2024-01-26T23:45:44.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f283d0d9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2838067af0 0x7f283814c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:44.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f283d0d9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f283806ebe0 0x7f283806c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:44.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f283d0d9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28381597e0 0x7f283806c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:44.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f283d0d9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283814d3d0 con 0x7f28381597e0 2024-01-26T23:45:44.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f283d0d9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283816d460 con 0x7f283806ebe0 2024-01-26T23:45:44.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f283d0d9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f283816a970 con 0x7f2838067af0 2024-01-26T23:45:44.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f283759e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28381597e0 0x7f283806c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:44.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f2836d9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2838067af0 0x7f283814c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:44.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f283759e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28381597e0 0x7f283806c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35946/0 (socket says 172.21.15.46:35946) 2024-01-26T23:45:44.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f283759e700 1 -- 172.21.15.46:0/2481237068 learned_addr learned my addr 172.21.15.46:0/2481237068 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:44.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f283659c700 1 --2- 172.21.15.46:0/2481237068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f283806ebe0 0x7f283806c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:44.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f283759e700 1 -- 172.21.15.46:0/2481237068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2838067af0 msgr2=0x7f283814c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:44.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f283759e700 1 --2- 172.21.15.46:0/2481237068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2838067af0 0x7f283814c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f283759e700 1 -- 172.21.15.46:0/2481237068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f283806ebe0 msgr2=0x7f283806c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:44.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f283759e700 1 --2- 172.21.15.46:0/2481237068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f283806ebe0 0x7f283806c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f283759e700 1 -- 172.21.15.46:0/2481237068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2820009420 con 0x7f28381597e0 2024-01-26T23:45:44.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f2836d9d700 1 --2- 172.21.15.46:0/2481237068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2838067af0 0x7f283814c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:44.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f283659c700 1 --2- 172.21.15.46:0/2481237068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f283806ebe0 0x7f283806c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:44.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f283759e700 1 --2- 172.21.15.46:0/2481237068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28381597e0 0x7f283806c840 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f2828009df0 tx=0x7f282800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:44.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.441+0000 7f282effd700 1 -- 172.21.15.46:0/2481237068 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282800caa0 con 0x7f28381597e0 2024-01-26T23:45:44.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.442+0000 7f282effd700 1 -- 172.21.15.46:0/2481237068 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282800f040 con 0x7f28381597e0 2024-01-26T23:45:44.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.442+0000 7f283d0d9700 1 -- 172.21.15.46:0/2481237068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283816d700 con 0x7f28381597e0 2024-01-26T23:45:44.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.442+0000 7f282effd700 1 -- 172.21.15.46:0/2481237068 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282800ce70 con 0x7f28381597e0 2024-01-26T23:45:44.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.442+0000 7f283d0d9700 1 -- 172.21.15.46:0/2481237068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f283816ac50 con 0x7f28381597e0 2024-01-26T23:45:44.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.443+0000 7f283d0d9700 1 -- 172.21.15.46:0/2481237068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f283804c710 con 0x7f28381597e0 2024-01-26T23:45:44.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.443+0000 7f282effd700 1 -- 172.21.15.46:0/2481237068 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f2828014690 con 0x7f28381597e0 2024-01-26T23:45:44.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.443+0000 7f282effd700 1 --2- 172.21.15.46:0/2481237068 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2818066ea0 0x7f2818069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:44.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.444+0000 7f282effd700 1 -- 172.21.15.46:0/2481237068 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6167+0+0 (secure 0 0 0) 0x7f2828084df0 con 0x7f28381597e0 2024-01-26T23:45:44.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.444+0000 7f2836d9d700 1 --2- 172.21.15.46:0/2481237068 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2818066ea0 0x7f2818069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:44.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.444+0000 7f2836d9d700 1 --2- 172.21.15.46:0/2481237068 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2818066ea0 0x7f2818069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f28300016f0 tx=0x7f2830007f50 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:44.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.448+0000 7f282effd700 1 -- 172.21.15.46:0/2481237068 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2828051f70 con 0x7f28381597e0 2024-01-26T23:45:44.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.661+0000 7f283d0d9700 1 -- 172.21.15.46:0/2481237068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f283814d6f0 con 0x7f28381597e0 2024-01-26T23:45:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.664+0000 7f282effd700 1 -- 172.21.15.46:0/2481237068 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v26) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2828055590 con 0x7f28381597e0 2024-01-26T23:45:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.667+0000 7f283d0d9700 1 -- 172.21.15.46:0/2481237068 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2818066ea0 msgr2=0x7f2818069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.667+0000 7f283d0d9700 1 --2- 172.21.15.46:0/2481237068 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2818066ea0 0x7f2818069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f28300016f0 tx=0x7f2830007f50 comp rx=0 tx=0).stop 2024-01-26T23:45:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.667+0000 7f283d0d9700 1 -- 172.21.15.46:0/2481237068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28381597e0 msgr2=0x7f283806c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.667+0000 7f283d0d9700 1 --2- 172.21.15.46:0/2481237068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28381597e0 0x7f283806c840 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f2828009df0 tx=0x7f282800b040 comp rx=0 tx=0).stop 2024-01-26T23:45:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.667+0000 7f283d0d9700 1 -- 172.21.15.46:0/2481237068 shutdown_connections 2024-01-26T23:45:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.667+0000 7f283d0d9700 1 --2- 172.21.15.46:0/2481237068 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2818066ea0 0x7f2818069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.667+0000 7f283d0d9700 1 --2- 172.21.15.46:0/2481237068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28381597e0 0x7f283806c840 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.667+0000 7f283d0d9700 1 --2- 172.21.15.46:0/2481237068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2838067af0 0x7f283814c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.667+0000 7f283d0d9700 1 --2- 172.21.15.46:0/2481237068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f283806ebe0 0x7f283806c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.667+0000 7f283d0d9700 1 -- 172.21.15.46:0/2481237068 >> 172.21.15.46:0/2481237068 conn(0x7f28380cdf80 msgr2=0x7f28380cbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.667+0000 7f283d0d9700 1 -- 172.21.15.46:0/2481237068 shutdown_connections 2024-01-26T23:45:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.667+0000 7f283d0d9700 1 -- 172.21.15.46:0/2481237068 wait complete. 2024-01-26T23:45:44.681 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-26T23:45:44.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.814+0000 7f8d30dec700 1 Processor -- start 2024-01-26T23:45:44.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.814+0000 7f8d30dec700 1 -- start start 2024-01-26T23:45:44.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.816+0000 7f8d30dec700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8d2c06a2f0 0x7f8d2c1647a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:44.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.816+0000 7f8d30dec700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2c15ad60 con 0x7f8d2c153c40 2024-01-26T23:45:44.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.816+0000 7f8d30dec700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2c06e4c0 con 0x7f8d2c154010 2024-01-26T23:45:44.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.816+0000 7f8d30dec700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d2c0b1d20 con 0x7f8d2c06a2f0 2024-01-26T23:45:44.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.816+0000 7f8d2a59c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8d2c153c40 0x7f8d2c06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53148/0 (socket says 172.21.15.46:53148) 2024-01-26T23:45:44.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.816+0000 7f8d2a59c700 1 -- 172.21.15.46:0/1641529899 learned_addr learned my addr 172.21.15.46:0/1641529899 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:44.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.816+0000 7f8d2ad9d700 1 --2- 172.21.15.46:0/1641529899 >> v2:172.21.15.106:3300/0 conn(0x7f8d2c06a2f0 0x7f8d2c1647a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:44.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.816+0000 7f8d2959a700 1 -- 172.21.15.46:0/1641529899 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639237574 0 0) 0x7f8d2c15ad60 con 0x7f8d2c153c40 2024-01-26T23:45:44.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.816+0000 7f8d2959a700 1 -- 172.21.15.46:0/1641529899 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d14003570 con 0x7f8d2c153c40 2024-01-26T23:45:44.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.816+0000 7f8d2ad9d700 1 -- 172.21.15.46:0/1641529899 >> v1:172.21.15.106:6789/0 conn(0x7f8d2c154010 legacy=0x7f8d2c06ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:44.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.816+0000 7f8d2ad9d700 1 -- 172.21.15.46:0/1641529899 >> v1:172.21.15.46:6789/0 conn(0x7f8d2c153c40 legacy=0x7f8d2c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:44.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.816+0000 7f8d2959a700 1 -- 172.21.15.46:0/1641529899 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913053051 0 0) 0x7f8d2c06e4c0 con 0x7f8d2c154010 2024-01-26T23:45:44.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.816+0000 7f8d2ad9d700 1 -- 172.21.15.46:0/1641529899 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c15a150 con 0x7f8d2c06a2f0 2024-01-26T23:45:44.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.816+0000 7f8d2ad9d700 1 --2- 172.21.15.46:0/1641529899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d2c06a2f0 0x7f8d2c1647a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8d24009e60 tx=0x7f8d24009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a70f66092bb483a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:44.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.817+0000 7f8d2959a700 1 -- 172.21.15.46:0/1641529899 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2400e070 con 0x7f8d2c06a2f0 2024-01-26T23:45:44.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.817+0000 7f8d2959a700 1 -- 172.21.15.46:0/1641529899 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2400bb60 con 0x7f8d2c06a2f0 2024-01-26T23:45:44.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.817+0000 7f8d2959a700 1 -- 172.21.15.46:0/1641529899 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d24012430 con 0x7f8d2c06a2f0 2024-01-26T23:45:44.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.817+0000 7f8d30dec700 1 -- 172.21.15.46:0/1641529899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d2c06a2f0 msgr2=0x7f8d2c1647a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:44.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.817+0000 7f8d30dec700 1 --2- 172.21.15.46:0/1641529899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d2c06a2f0 0x7f8d2c1647a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8d24009e60 tx=0x7f8d24009780 comp rx=0 tx=0).stop 2024-01-26T23:45:44.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.817+0000 7f8d30dec700 1 -- 172.21.15.46:0/1641529899 shutdown_connections 2024-01-26T23:45:44.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.817+0000 7f8d30dec700 1 --2- 172.21.15.46:0/1641529899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d2c06a2f0 0x7f8d2c1647a0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.817+0000 7f8d30dec700 1 -- 172.21.15.46:0/1641529899 >> 172.21.15.46:0/1641529899 conn(0x7f8d2c067930 msgr2=0x7f8d2c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:44.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.817+0000 7f8d30dec700 1 -- 172.21.15.46:0/1641529899 shutdown_connections 2024-01-26T23:45:44.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.817+0000 7f8d30dec700 1 -- 172.21.15.46:0/1641529899 wait complete. 2024-01-26T23:45:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.818+0000 7f8d30dec700 1 Processor -- start 2024-01-26T23:45:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.819+0000 7f8d30dec700 1 -- start start 2024-01-26T23:45:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.819+0000 7f8d30dec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d2c06a2f0 0x7f8d2c0cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.819+0000 7f8d30dec700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d2c153c40 0x7f8d2c0af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.819+0000 7f8d30dec700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d2c154010 0x7f8d2c0afae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.819+0000 7f8d30dec700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c0cdba0 con 0x7f8d2c06a2f0 2024-01-26T23:45:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.819+0000 7f8d30dec700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c16a350 con 0x7f8d2c153c40 2024-01-26T23:45:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.819+0000 7f8d30dec700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c0cd550 con 0x7f8d2c154010 2024-01-26T23:45:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.819+0000 7f8d2a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d2c06a2f0 0x7f8d2c0cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.819+0000 7f8d2a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d2c06a2f0 0x7f8d2c0cb6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35948/0 (socket says 172.21.15.46:35948) 2024-01-26T23:45:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.819+0000 7f8d2a59c700 1 -- 172.21.15.46:0/3207767364 learned_addr learned my addr 172.21.15.46:0/3207767364 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.819+0000 7f8d29d9b700 1 --2- 172.21.15.46:0/3207767364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d2c153c40 0x7f8d2c0af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:44.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.820+0000 7f8d2ad9d700 1 --2- 172.21.15.46:0/3207767364 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d2c154010 0x7f8d2c0afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:44.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.820+0000 7f8d2a59c700 1 -- 172.21.15.46:0/3207767364 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d2c154010 msgr2=0x7f8d2c0afae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:44.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.820+0000 7f8d2a59c700 1 --2- 172.21.15.46:0/3207767364 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d2c154010 0x7f8d2c0afae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.820+0000 7f8d2a59c700 1 -- 172.21.15.46:0/3207767364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d2c153c40 msgr2=0x7f8d2c0af5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:44.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.820+0000 7f8d2a59c700 1 --2- 172.21.15.46:0/3207767364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d2c153c40 0x7f8d2c0af5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:44.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.820+0000 7f8d2a59c700 1 -- 172.21.15.46:0/3207767364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d24009a20 con 0x7f8d2c06a2f0 2024-01-26T23:45:44.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.820+0000 7f8d2a59c700 1 --2- 172.21.15.46:0/3207767364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d2c06a2f0 0x7f8d2c0cb6e0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c0011e0 tx=0x7f8d1c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:44.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.820+0000 7f8d1a7fc700 1 -- 172.21.15.46:0/3207767364 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c005b30 con 0x7f8d2c06a2f0 2024-01-26T23:45:44.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.820+0000 7f8d1a7fc700 1 -- 172.21.15.46:0/3207767364 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1c005c90 con 0x7f8d2c06a2f0 2024-01-26T23:45:44.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.820+0000 7f8d30dec700 1 -- 172.21.15.46:0/3207767364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2c0cd830 con 0x7f8d2c06a2f0 2024-01-26T23:45:44.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.820+0000 7f8d1a7fc700 1 -- 172.21.15.46:0/3207767364 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c007300 con 0x7f8d2c06a2f0 2024-01-26T23:45:44.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.820+0000 7f8d30dec700 1 -- 172.21.15.46:0/3207767364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2c0cbc20 con 0x7f8d2c06a2f0 2024-01-26T23:45:44.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.821+0000 7f8d30dec700 1 -- 172.21.15.46:0/3207767364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2c0cd3c0 con 0x7f8d2c06a2f0 2024-01-26T23:45:44.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.822+0000 7f8d1a7fc700 1 -- 172.21.15.46:0/3207767364 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f8d1c007670 con 0x7f8d2c06a2f0 2024-01-26T23:45:44.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.822+0000 7f8d1a7fc700 1 --2- 172.21.15.46:0/3207767364 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d0c066ea0 0x7f8d0c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:44.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.822+0000 7f8d1a7fc700 1 -- 172.21.15.46:0/3207767364 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6167+0+0 (secure 0 0 0) 0x7f8d1c0839c0 con 0x7f8d2c06a2f0 2024-01-26T23:45:44.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.822+0000 7f8d29d9b700 1 --2- 172.21.15.46:0/3207767364 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d0c066ea0 0x7f8d0c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.823+0000 7f8d29d9b700 1 --2- 172.21.15.46:0/3207767364 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d0c066ea0 0x7f8d0c069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c153970 tx=0x7f8d20007ec0 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:44.827+0000 7f8d1a7fc700 1 -- 172.21.15.46:0/3207767364 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8d1c050b40 con 0x7f8d2c06a2f0 2024-01-26T23:45:44.869 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:44.868+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:45.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.037+0000 7f8d30dec700 1 -- 172.21.15.46:0/3207767364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f8d2c04c710 con 0x7f8d2c06a2f0 2024-01-26T23:45:45.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.041+0000 7f8d1a7fc700 1 -- 172.21.15.46:0/3207767364 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v27) v1 ==== 115+0+0 (secure 0 0 0) 0x7f8d1c017020 con 0x7f8d2c06a2f0 2024-01-26T23:45:45.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.043+0000 7f8d30dec700 1 -- 172.21.15.46:0/3207767364 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d0c066ea0 msgr2=0x7f8d0c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:45.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.043+0000 7f8d30dec700 1 --2- 172.21.15.46:0/3207767364 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d0c066ea0 0x7f8d0c069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c153970 tx=0x7f8d20007ec0 comp rx=0 tx=0).stop 2024-01-26T23:45:45.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.043+0000 7f8d30dec700 1 -- 172.21.15.46:0/3207767364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d2c06a2f0 msgr2=0x7f8d2c0cb6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:45.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.043+0000 7f8d30dec700 1 --2- 172.21.15.46:0/3207767364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d2c06a2f0 0x7f8d2c0cb6e0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c0011e0 tx=0x7f8d1c007df0 comp rx=0 tx=0).stop 2024-01-26T23:45:45.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.043+0000 7f8d30dec700 1 -- 172.21.15.46:0/3207767364 shutdown_connections 2024-01-26T23:45:45.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.043+0000 7f8d30dec700 1 --2- 172.21.15.46:0/3207767364 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d0c066ea0 0x7f8d0c069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:45.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.043+0000 7f8d30dec700 1 --2- 172.21.15.46:0/3207767364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d2c06a2f0 0x7f8d2c0cb6e0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:45.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.043+0000 7f8d30dec700 1 --2- 172.21.15.46:0/3207767364 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d2c154010 0x7f8d2c0afae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:45.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.043+0000 7f8d30dec700 1 --2- 172.21.15.46:0/3207767364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d2c153c40 0x7f8d2c0af5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:45.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.043+0000 7f8d30dec700 1 -- 172.21.15.46:0/3207767364 >> 172.21.15.46:0/3207767364 conn(0x7f8d2c067930 msgr2=0x7f8d2c15af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:45.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.043+0000 7f8d30dec700 1 -- 172.21.15.46:0/3207767364 shutdown_connections 2024-01-26T23:45:45.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.043+0000 7f8d30dec700 1 -- 172.21.15.46:0/3207767364 wait complete. 2024-01-26T23:45:45.057 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-01-26T23:45:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.190+0000 7fb8d954f700 1 Processor -- start 2024-01-26T23:45:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.190+0000 7fb8d954f700 1 -- start start 2024-01-26T23:45:45.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.191+0000 7fb8d954f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb8d40b2b40 0x7fb8d40ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:45.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.191+0000 7fb8d954f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb8d40ae960 0x7fb8d40ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:45.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.191+0000 7fb8d954f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8d415a0b0 con 0x7fb8d40aedd0 2024-01-26T23:45:45.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.191+0000 7fb8d954f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8d4099fc0 con 0x7fb8d40b2b40 2024-01-26T23:45:45.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.191+0000 7fb8d954f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8d40ae4d0 con 0x7fb8d40ae960 2024-01-26T23:45:45.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.191+0000 7fb8d27fc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb8d40ae960 0x7fb8d40ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:45.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.191+0000 7fb8d2ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb8d40b2b40 0x7fb8d40ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:45.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.191+0000 7fb8d37fe700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fb8d40aedd0 0x7fb8d40cb5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:39744/0 (socket says 172.21.15.46:39744) 2024-01-26T23:45:45.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.191+0000 7fb8d37fe700 1 -- 172.21.15.46:0/2278091739 learned_addr learned my addr 172.21.15.46:0/2278091739 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:45.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.191+0000 7fb8d27fc700 1 -- 172.21.15.46:0/2278091739 >> v2:172.21.15.46:3300/0 conn(0x7fb8d40b2b40 msgr2=0x7fb8d40ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:45.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.191+0000 7fb8d27fc700 1 --2- 172.21.15.46:0/2278091739 >> v2:172.21.15.46:3300/0 conn(0x7fb8d40b2b40 0x7fb8d40ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:45.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.191+0000 7fb8d27fc700 1 -- 172.21.15.46:0/2278091739 >> v1:172.21.15.106:6789/0 conn(0x7fb8d40aedd0 legacy=0x7fb8d40cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.191+0000 7fb8d27fc700 1 -- 172.21.15.46:0/2278091739 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d40ad8c0 con 0x7fb8d40ae960 2024-01-26T23:45:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.191+0000 7fb8d1ffb700 1 -- 172.21.15.46:0/2278091739 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 13868335 0 0) 0x7fb8d415a0b0 con 0x7fb8d40aedd0 2024-01-26T23:45:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.191+0000 7fb8d2ffd700 1 --2- 172.21.15.46:0/2278091739 >> v2:172.21.15.46:3300/0 conn(0x7fb8d40b2b40 0x7fb8d40ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.192+0000 7fb8d27fc700 1 --2- 172.21.15.46:0/2278091739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8d40ae960 0x7fb8d40ced50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb8d40ce2f0 tx=0x7fb8c80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cd6e5b5cdacf508 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.192+0000 7fb8d1ffb700 1 -- 172.21.15.46:0/2278091739 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c8011630 con 0x7fb8d40ae960 2024-01-26T23:45:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.192+0000 7fb8d1ffb700 1 -- 172.21.15.46:0/2278091739 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c8011790 con 0x7fb8d40ae960 2024-01-26T23:45:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.192+0000 7fb8d1ffb700 1 -- 172.21.15.46:0/2278091739 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c8011b30 con 0x7fb8d40ae960 2024-01-26T23:45:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.192+0000 7fb8d954f700 1 -- 172.21.15.46:0/2278091739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8d40ae960 msgr2=0x7fb8d40ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.192+0000 7fb8d954f700 1 --2- 172.21.15.46:0/2278091739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8d40ae960 0x7fb8d40ced50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb8d40ce2f0 tx=0x7fb8c80091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.192+0000 7fb8d954f700 1 -- 172.21.15.46:0/2278091739 shutdown_connections 2024-01-26T23:45:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.192+0000 7fb8d954f700 1 --2- 172.21.15.46:0/2278091739 >> v2:172.21.15.46:3300/0 conn(0x7fb8d40b2b40 0x7fb8d40ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.192+0000 7fb8d954f700 1 --2- 172.21.15.46:0/2278091739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8d40ae960 0x7fb8d40ced50 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.192+0000 7fb8d954f700 1 -- 172.21.15.46:0/2278091739 >> 172.21.15.46:0/2278091739 conn(0x7fb8d4099a30 msgr2=0x7fb8d406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.193+0000 7fb8d954f700 1 -- 172.21.15.46:0/2278091739 shutdown_connections 2024-01-26T23:45:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.193+0000 7fb8d954f700 1 -- 172.21.15.46:0/2278091739 wait complete. 2024-01-26T23:45:45.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.194+0000 7fb8d954f700 1 Processor -- start 2024-01-26T23:45:45.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.194+0000 7fb8d954f700 1 -- start start 2024-01-26T23:45:45.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8d954f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8d40ae960 0x7fb8d41569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:45.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8d954f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8d40b2b40 0x7fb8d40af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:45.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8d954f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8d4157250 0x7fb8d40b5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:45.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8d954f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d415ca70 con 0x7fb8d4157250 2024-01-26T23:45:45.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8d954f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d40b3510 con 0x7fb8d40ae960 2024-01-26T23:45:45.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8d954f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d4156f20 con 0x7fb8d40b2b40 2024-01-26T23:45:45.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8d37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8d4157250 0x7fb8d40b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:45.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8d37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8d4157250 0x7fb8d40b5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35958/0 (socket says 172.21.15.46:35958) 2024-01-26T23:45:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8d37fe700 1 -- 172.21.15.46:0/4087319821 learned_addr learned my addr 172.21.15.46:0/4087319821 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8d2ffd700 1 --2- 172.21.15.46:0/4087319821 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8d40ae960 0x7fb8d41569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8d27fc700 1 --2- 172.21.15.46:0/4087319821 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8d40b2b40 0x7fb8d40af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8d37fe700 1 -- 172.21.15.46:0/4087319821 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8d40b2b40 msgr2=0x7fb8d40af1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8d37fe700 1 --2- 172.21.15.46:0/4087319821 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8d40b2b40 0x7fb8d40af1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8d37fe700 1 -- 172.21.15.46:0/4087319821 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8d40ae960 msgr2=0x7fb8d41569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8d37fe700 1 --2- 172.21.15.46:0/4087319821 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8d40ae960 0x7fb8d41569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8d37fe700 1 -- 172.21.15.46:0/4087319821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c40093c0 con 0x7fb8d4157250 2024-01-26T23:45:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8d37fe700 1 --2- 172.21.15.46:0/4087319821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8d4157250 0x7fb8d40b5850 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc00ccc0 tx=0x7fb8cc001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.195+0000 7fb8baffd700 1 -- 172.21.15.46:0/4087319821 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc00a880 con 0x7fb8d4157250 2024-01-26T23:45:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.196+0000 7fb8baffd700 1 -- 172.21.15.46:0/4087319821 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8cc00a9e0 con 0x7fb8d4157250 2024-01-26T23:45:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.196+0000 7fb8d954f700 1 -- 172.21.15.46:0/4087319821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c8009420 con 0x7fb8d4157250 2024-01-26T23:45:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.196+0000 7fb8baffd700 1 -- 172.21.15.46:0/4087319821 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc00ad80 con 0x7fb8d4157250 2024-01-26T23:45:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.196+0000 7fb8d954f700 1 -- 172.21.15.46:0/4087319821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d40afe80 con 0x7fb8d4157250 2024-01-26T23:45:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.197+0000 7fb8d954f700 1 -- 172.21.15.46:0/4087319821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d40b0200 con 0x7fb8d4157250 2024-01-26T23:45:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.197+0000 7fb8baffd700 1 -- 172.21.15.46:0/4087319821 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fb8cc007300 con 0x7fb8d4157250 2024-01-26T23:45:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.198+0000 7fb8baffd700 1 --2- 172.21.15.46:0/4087319821 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8c0066ea0 0x7fb8c0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.198+0000 7fb8baffd700 1 -- 172.21.15.46:0/4087319821 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6167+0+0 (secure 0 0 0) 0x7fb8cc084900 con 0x7fb8d4157250 2024-01-26T23:45:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.198+0000 7fb8d2ffd700 1 --2- 172.21.15.46:0/4087319821 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8c0066ea0 0x7fb8c0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.198+0000 7fb8d2ffd700 1 --2- 172.21.15.46:0/4087319821 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8c0066ea0 0x7fb8c0069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009f30 tx=0x7fb8c4009120 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:45.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.202+0000 7fb8baffd700 1 -- 172.21.15.46:0/4087319821 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb8cc051b00 con 0x7fb8d4157250 2024-01-26T23:45:45.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.497+0000 7fb8d954f700 1 -- 172.21.15.46:0/4087319821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7fb8d40b2620 con 0x7fb8d4157250 2024-01-26T23:45:45.523 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:45.522+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:45.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.904+0000 7fb8baffd700 1 -- 172.21.15.46:0/4087319821 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v588) v1 ==== 156+0+0 (secure 0 0 0) 0x7fb8cc055120 con 0x7fb8d4157250 2024-01-26T23:45:45.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.907+0000 7fb8d954f700 1 -- 172.21.15.46:0/4087319821 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8c0066ea0 msgr2=0x7fb8c0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:45.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.907+0000 7fb8d954f700 1 --2- 172.21.15.46:0/4087319821 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8c0066ea0 0x7fb8c0069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009f30 tx=0x7fb8c4009120 comp rx=0 tx=0).stop 2024-01-26T23:45:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.907+0000 7fb8d954f700 1 -- 172.21.15.46:0/4087319821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8d4157250 msgr2=0x7fb8d40b5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.907+0000 7fb8d954f700 1 --2- 172.21.15.46:0/4087319821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8d4157250 0x7fb8d40b5850 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc00ccc0 tx=0x7fb8cc001a20 comp rx=0 tx=0).stop 2024-01-26T23:45:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.907+0000 7fb8d954f700 1 -- 172.21.15.46:0/4087319821 shutdown_connections 2024-01-26T23:45:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.907+0000 7fb8d954f700 1 --2- 172.21.15.46:0/4087319821 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb8c0066ea0 0x7fb8c0069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.907+0000 7fb8d954f700 1 --2- 172.21.15.46:0/4087319821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8d4157250 0x7fb8d40b5850 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.907+0000 7fb8d954f700 1 --2- 172.21.15.46:0/4087319821 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8d40ae960 0x7fb8d41569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.907+0000 7fb8d954f700 1 --2- 172.21.15.46:0/4087319821 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8d40b2b40 0x7fb8d40af1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.907+0000 7fb8d954f700 1 -- 172.21.15.46:0/4087319821 >> 172.21.15.46:0/4087319821 conn(0x7fb8d4099a30 msgr2=0x7fb8d4159120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.907+0000 7fb8d954f700 1 -- 172.21.15.46:0/4087319821 shutdown_connections 2024-01-26T23:45:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:45.907+0000 7fb8d954f700 1 -- 172.21.15.46:0/4087319821 wait complete. 2024-01-26T23:45:45.923 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2024-01-26T23:45:45.923 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2024-01-26T23:45:45.923 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-26T23:45:45.923 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-01-26T23:45:46.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.063+0000 7f182c5ed700 1 Processor -- start 2024-01-26T23:45:46.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.063+0000 7f182c5ed700 1 -- start start 2024-01-26T23:45:46.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.064+0000 7f182c5ed700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1824153ec0 0x7f1824154290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.064+0000 7f182c5ed700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1824157630 0x7f18240b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.064+0000 7f182c5ed700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18240b1500 con 0x7f18240b2100 2024-01-26T23:45:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.064+0000 7f182c5ed700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18240b2560 con 0x7f1824153ec0 2024-01-26T23:45:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.064+0000 7f182c5ed700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f182415b140 con 0x7f1824157630 2024-01-26T23:45:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.065+0000 7f182a389700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1824153ec0 0x7f1824154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.065+0000 7f182a389700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1824153ec0 0x7f1824154290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35962/0 (socket says 172.21.15.46:35962) 2024-01-26T23:45:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.065+0000 7f182a389700 1 -- 172.21.15.46:0/2721151359 learned_addr learned my addr 172.21.15.46:0/2721151359 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.065+0000 7f1829b88700 1 --2- 172.21.15.46:0/2721151359 >> v2:172.21.15.106:3300/0 conn(0x7f1824157630 0x7f18240b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.065+0000 7f182a389700 1 -- 172.21.15.46:0/2721151359 >> v2:172.21.15.106:3300/0 conn(0x7f1824157630 msgr2=0x7f18240b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.065+0000 7f182a389700 1 --2- 172.21.15.46:0/2721151359 >> v2:172.21.15.106:3300/0 conn(0x7f1824157630 0x7f18240b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.065+0000 7f182a389700 1 -- 172.21.15.46:0/2721151359 >> v1:172.21.15.106:6789/0 conn(0x7f18240b2100 legacy=0x7f182406eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.065+0000 7f182a389700 1 -- 172.21.15.46:0/2721151359 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182415a5c0 con 0x7f1824153ec0 2024-01-26T23:45:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.065+0000 7f182a389700 1 --2- 172.21.15.46:0/2721151359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1824153ec0 0x7f1824154290 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f182406e2c0 tx=0x7f1820009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8031b0c47c0010e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.065+0000 7f1829387700 1 -- 172.21.15.46:0/2721151359 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182000b820 con 0x7f1824153ec0 2024-01-26T23:45:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.065+0000 7f1829387700 1 -- 172.21.15.46:0/2721151359 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182000b980 con 0x7f1824153ec0 2024-01-26T23:45:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.065+0000 7f1829387700 1 -- 172.21.15.46:0/2721151359 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182000bd20 con 0x7f1824153ec0 2024-01-26T23:45:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.066+0000 7f182c5ed700 1 -- 172.21.15.46:0/2721151359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1824153ec0 msgr2=0x7f1824154290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.066+0000 7f182c5ed700 1 --2- 172.21.15.46:0/2721151359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1824153ec0 0x7f1824154290 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f182406e2c0 tx=0x7f1820009250 comp rx=0 tx=0).stop 2024-01-26T23:45:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.066+0000 7f182c5ed700 1 -- 172.21.15.46:0/2721151359 shutdown_connections 2024-01-26T23:45:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.066+0000 7f182c5ed700 1 --2- 172.21.15.46:0/2721151359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1824153ec0 0x7f1824154290 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.066+0000 7f182c5ed700 1 --2- 172.21.15.46:0/2721151359 >> v2:172.21.15.106:3300/0 conn(0x7f1824157630 0x7f18240b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.066+0000 7f182c5ed700 1 -- 172.21.15.46:0/2721151359 >> 172.21.15.46:0/2721151359 conn(0x7f1824069510 msgr2=0x7f18240aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.066+0000 7f182c5ed700 1 -- 172.21.15.46:0/2721151359 shutdown_connections 2024-01-26T23:45:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.066+0000 7f182c5ed700 1 -- 172.21.15.46:0/2721151359 wait complete. 2024-01-26T23:45:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.068+0000 7f182c5ed700 1 Processor -- start 2024-01-26T23:45:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.068+0000 7f182c5ed700 1 -- start start 2024-01-26T23:45:46.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f182c5ed700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18240b2100 0x7f18241596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f182c5ed700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1824153ec0 0x7f1824152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f182c5ed700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1824157630 0x7f1824152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f182c5ed700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1824149240 con 0x7f18240b2100 2024-01-26T23:45:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f182c5ed700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1824151c20 con 0x7f1824153ec0 2024-01-26T23:45:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f182c5ed700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824151010 con 0x7f1824157630 2024-01-26T23:45:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f182a389700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18240b2100 0x7f18241596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f182a389700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18240b2100 0x7f18241596c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35970/0 (socket says 172.21.15.46:35970) 2024-01-26T23:45:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f182a389700 1 -- 172.21.15.46:0/149057438 learned_addr learned my addr 172.21.15.46:0/149057438 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f1829b88700 1 --2- 172.21.15.46:0/149057438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1824153ec0 0x7f1824152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f182ab8a700 1 --2- 172.21.15.46:0/149057438 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1824157630 0x7f1824152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f182a389700 1 -- 172.21.15.46:0/149057438 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1824157630 msgr2=0x7f1824152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f182a389700 1 --2- 172.21.15.46:0/149057438 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1824157630 0x7f1824152e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f182a389700 1 -- 172.21.15.46:0/149057438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1824153ec0 msgr2=0x7f1824152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f182a389700 1 --2- 172.21.15.46:0/149057438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1824153ec0 0x7f1824152950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f182a389700 1 -- 172.21.15.46:0/149057438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18200094f0 con 0x7f18240b2100 2024-01-26T23:45:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f1829b88700 1 --2- 172.21.15.46:0/149057438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1824153ec0 0x7f1824152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f182a389700 1 --2- 172.21.15.46:0/149057438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18240b2100 0x7f18241596c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f18200030c0 tx=0x7f1820016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.069+0000 7f181a7fc700 1 -- 172.21.15.46:0/149057438 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1820011e60 con 0x7f18240b2100 2024-01-26T23:45:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.070+0000 7f181a7fc700 1 -- 172.21.15.46:0/149057438 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1820020070 con 0x7f18240b2100 2024-01-26T23:45:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.070+0000 7f182c5ed700 1 -- 172.21.15.46:0/149057438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18241512f0 con 0x7f18240b2100 2024-01-26T23:45:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.070+0000 7f181a7fc700 1 -- 172.21.15.46:0/149057438 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182001b6a0 con 0x7f18240b2100 2024-01-26T23:45:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.070+0000 7f182c5ed700 1 -- 172.21.15.46:0/149057438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18241518b0 con 0x7f18240b2100 2024-01-26T23:45:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.071+0000 7f182c5ed700 1 -- 172.21.15.46:0/149057438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1824151f00 con 0x7f18240b2100 2024-01-26T23:45:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.071+0000 7f181a7fc700 1 -- 172.21.15.46:0/149057438 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f1820022030 con 0x7f18240b2100 2024-01-26T23:45:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.072+0000 7f181a7fc700 1 --2- 172.21.15.46:0/149057438 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1808066ea0 0x7f1808069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.072+0000 7f181a7fc700 1 -- 172.21.15.46:0/149057438 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6167+0+0 (secure 0 0 0) 0x7f18200864e0 con 0x7f18240b2100 2024-01-26T23:45:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.072+0000 7f1829b88700 1 --2- 172.21.15.46:0/149057438 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1808066ea0 0x7f1808069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.072+0000 7f1829b88700 1 --2- 172.21.15.46:0/149057438 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1808066ea0 0x7f1808069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f182416d1a0 tx=0x7f1814008040 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:46.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.076+0000 7f181a7fc700 1 -- 172.21.15.46:0/149057438 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1820053660 con 0x7f18240b2100 2024-01-26T23:45:46.196 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:46.195+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:46.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.377+0000 7f182c5ed700 1 -- 172.21.15.46:0/149057438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f1824151f00 con 0x7f18240b2100 2024-01-26T23:45:46.874 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:46.873+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:46.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.905+0000 7f181a7fc700 1 -- 172.21.15.46:0/149057438 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v589) v1 ==== 169+0+0 (secure 0 0 0) 0x7f1820056c80 con 0x7f18240b2100 2024-01-26T23:45:46.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.908+0000 7f182c5ed700 1 -- 172.21.15.46:0/149057438 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1808066ea0 msgr2=0x7f1808069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:46.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.908+0000 7f182c5ed700 1 --2- 172.21.15.46:0/149057438 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1808066ea0 0x7f1808069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f182416d1a0 tx=0x7f1814008040 comp rx=0 tx=0).stop 2024-01-26T23:45:46.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.908+0000 7f182c5ed700 1 -- 172.21.15.46:0/149057438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18240b2100 msgr2=0x7f18241596c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:46.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.908+0000 7f182c5ed700 1 --2- 172.21.15.46:0/149057438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18240b2100 0x7f18241596c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f18200030c0 tx=0x7f1820016000 comp rx=0 tx=0).stop 2024-01-26T23:45:46.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.908+0000 7f182c5ed700 1 -- 172.21.15.46:0/149057438 shutdown_connections 2024-01-26T23:45:46.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.908+0000 7f182c5ed700 1 --2- 172.21.15.46:0/149057438 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1808066ea0 0x7f1808069350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:46.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.908+0000 7f182c5ed700 1 --2- 172.21.15.46:0/149057438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18240b2100 0x7f18241596c0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:46.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.908+0000 7f182c5ed700 1 --2- 172.21.15.46:0/149057438 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1824157630 0x7f1824152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:46.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.908+0000 7f182c5ed700 1 --2- 172.21.15.46:0/149057438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1824153ec0 0x7f1824152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:46.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.908+0000 7f182c5ed700 1 -- 172.21.15.46:0/149057438 >> 172.21.15.46:0/149057438 conn(0x7f1824069510 msgr2=0x7f18240681c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:46.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.908+0000 7f182c5ed700 1 -- 172.21.15.46:0/149057438 shutdown_connections 2024-01-26T23:45:46.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:46.908+0000 7f182c5ed700 1 -- 172.21.15.46:0/149057438 wait complete. 2024-01-26T23:45:46.924 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-26T23:45:47.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.065+0000 7f8d259da700 1 Processor -- start 2024-01-26T23:45:47.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.065+0000 7f8d259da700 1 -- start start 2024-01-26T23:45:47.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.066+0000 7f8d259da700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8d20159da0 0x7f8d200ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:47.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.066+0000 7f8d259da700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d2015d8c0 0x7f8d2015dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:47.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.066+0000 7f8d259da700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2015a200 con 0x7f8d201599d0 2024-01-26T23:45:47.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.066+0000 7f8d259da700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d200b1a60 con 0x7f8d20159da0 2024-01-26T23:45:47.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.066+0000 7f8d259da700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2015e230 con 0x7f8d2015d8c0 2024-01-26T23:45:47.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.067+0000 7f8d1f7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d2015d8c0 0x7f8d2015dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:47.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.067+0000 7f8d1e7fc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8d20159da0 0x7f8d200ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:47.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.067+0000 7f8d1effd700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f8d201599d0 0x7f8d200b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:39762/0 (socket says 172.21.15.46:39762) 2024-01-26T23:45:47.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.067+0000 7f8d1effd700 1 -- 172.21.15.46:0/1025530579 learned_addr learned my addr 172.21.15.46:0/1025530579 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.067+0000 7f8d1e7fc700 1 -- 172.21.15.46:0/1025530579 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d2015d8c0 msgr2=0x7f8d2015dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.067+0000 7f8d1e7fc700 1 --2- 172.21.15.46:0/1025530579 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d2015d8c0 0x7f8d2015dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.067+0000 7f8d1e7fc700 1 -- 172.21.15.46:0/1025530579 >> v1:172.21.15.106:6789/0 conn(0x7f8d201599d0 legacy=0x7f8d200b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.067+0000 7f8d1e7fc700 1 -- 172.21.15.46:0/1025530579 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d200b11c0 con 0x7f8d20159da0 2024-01-26T23:45:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.067+0000 7f8d1dffb700 1 -- 172.21.15.46:0/1025530579 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361239065 0 0) 0x7f8d2015a200 con 0x7f8d201599d0 2024-01-26T23:45:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.067+0000 7f8d1f7fe700 1 --2- 172.21.15.46:0/1025530579 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d2015d8c0 0x7f8d2015dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.067+0000 7f8d1e7fc700 1 --2- 172.21.15.46:0/1025530579 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d20159da0 0x7f8d200ab1c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8d080098e0 tx=0x7f8d080091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=441000aa5cf87b8d server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.067+0000 7f8d1dffb700 1 -- 172.21.15.46:0/1025530579 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0800b640 con 0x7f8d20159da0 2024-01-26T23:45:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.067+0000 7f8d1dffb700 1 -- 172.21.15.46:0/1025530579 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0800b7a0 con 0x7f8d20159da0 2024-01-26T23:45:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.067+0000 7f8d1dffb700 1 -- 172.21.15.46:0/1025530579 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0800bb40 con 0x7f8d20159da0 2024-01-26T23:45:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.068+0000 7f8d259da700 1 -- 172.21.15.46:0/1025530579 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d20159da0 msgr2=0x7f8d200ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.068+0000 7f8d259da700 1 --2- 172.21.15.46:0/1025530579 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d20159da0 0x7f8d200ab1c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8d080098e0 tx=0x7f8d080091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.068+0000 7f8d259da700 1 -- 172.21.15.46:0/1025530579 shutdown_connections 2024-01-26T23:45:47.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.068+0000 7f8d259da700 1 --2- 172.21.15.46:0/1025530579 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d2015d8c0 0x7f8d2015dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.068+0000 7f8d259da700 1 --2- 172.21.15.46:0/1025530579 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d20159da0 0x7f8d200ab1c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.068+0000 7f8d259da700 1 -- 172.21.15.46:0/1025530579 >> 172.21.15.46:0/1025530579 conn(0x7f8d2006f840 msgr2=0x7f8d20087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:47.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.068+0000 7f8d259da700 1 -- 172.21.15.46:0/1025530579 shutdown_connections 2024-01-26T23:45:47.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.068+0000 7f8d259da700 1 -- 172.21.15.46:0/1025530579 wait complete. 2024-01-26T23:45:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.069+0000 7f8d259da700 1 Processor -- start 2024-01-26T23:45:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.070+0000 7f8d259da700 1 -- start start 2024-01-26T23:45:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.070+0000 7f8d259da700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d201599d0 0x7f8d20156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.070+0000 7f8d259da700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d20159da0 0x7f8d2016a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.070+0000 7f8d259da700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d2015d8c0 0x7f8d200af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.070+0000 7f8d259da700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2009a2b0 con 0x7f8d201599d0 2024-01-26T23:45:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.070+0000 7f8d259da700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d20156780 con 0x7f8d2015d8c0 2024-01-26T23:45:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.070+0000 7f8d259da700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2009a520 con 0x7f8d20159da0 2024-01-26T23:45:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.071+0000 7f8d1effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d201599d0 0x7f8d20156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.071+0000 7f8d1effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d201599d0 0x7f8d20156240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35980/0 (socket says 172.21.15.46:35980) 2024-01-26T23:45:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.071+0000 7f8d1effd700 1 -- 172.21.15.46:0/1137217745 learned_addr learned my addr 172.21.15.46:0/1137217745 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.071+0000 7f8d1e7fc700 1 --2- 172.21.15.46:0/1137217745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d20159da0 0x7f8d2016a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.071+0000 7f8d1f7fe700 1 --2- 172.21.15.46:0/1137217745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d2015d8c0 0x7f8d200af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.071+0000 7f8d1effd700 1 -- 172.21.15.46:0/1137217745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d20159da0 msgr2=0x7f8d2016a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.071+0000 7f8d1effd700 1 --2- 172.21.15.46:0/1137217745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d20159da0 0x7f8d2016a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.071+0000 7f8d1effd700 1 -- 172.21.15.46:0/1137217745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d2015d8c0 msgr2=0x7f8d200af150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.071+0000 7f8d1effd700 1 --2- 172.21.15.46:0/1137217745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d2015d8c0 0x7f8d200af150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.071+0000 7f8d1effd700 1 -- 172.21.15.46:0/1137217745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d18009a20 con 0x7f8d201599d0 2024-01-26T23:45:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.071+0000 7f8d1effd700 1 --2- 172.21.15.46:0/1137217745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d201599d0 0x7f8d20156240 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000c7e0 tx=0x7f8d10007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.071+0000 7f8d16ffd700 1 -- 172.21.15.46:0/1137217745 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10005b80 con 0x7f8d201599d0 2024-01-26T23:45:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.071+0000 7f8d16ffd700 1 -- 172.21.15.46:0/1137217745 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d10005ce0 con 0x7f8d201599d0 2024-01-26T23:45:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.071+0000 7f8d16ffd700 1 -- 172.21.15.46:0/1137217745 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10007320 con 0x7f8d201599d0 2024-01-26T23:45:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.071+0000 7f8d259da700 1 -- 172.21.15.46:0/1137217745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d08009420 con 0x7f8d201599d0 2024-01-26T23:45:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.071+0000 7f8d259da700 1 -- 172.21.15.46:0/1137217745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2009ab60 con 0x7f8d201599d0 2024-01-26T23:45:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.073+0000 7f8d16ffd700 1 -- 172.21.15.46:0/1137217745 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f8d10007690 con 0x7f8d201599d0 2024-01-26T23:45:47.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.073+0000 7f8d259da700 1 -- 172.21.15.46:0/1137217745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2004c710 con 0x7f8d201599d0 2024-01-26T23:45:47.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.073+0000 7f8d16ffd700 1 --2- 172.21.15.46:0/1137217745 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d00066ea0 0x7f8d00069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:47.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.073+0000 7f8d16ffd700 1 -- 172.21.15.46:0/1137217745 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6167+0+0 (secure 0 0 0) 0x7f8d10083af0 con 0x7f8d201599d0 2024-01-26T23:45:47.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.074+0000 7f8d1e7fc700 1 --2- 172.21.15.46:0/1137217745 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d00066ea0 0x7f8d00069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:47.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.074+0000 7f8d1e7fc700 1 --2- 172.21.15.46:0/1137217745 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d00066ea0 0x7f8d00069350 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8d080093f0 tx=0x7f8d08017000 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.079+0000 7f8d16ffd700 1 -- 172.21.15.46:0/1137217745 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8d10050d30 con 0x7f8d201599d0 2024-01-26T23:45:47.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.299+0000 7f8d259da700 1 -- 172.21.15.46:0/1137217745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8d200b1510 con 0x7f8d201599d0 2024-01-26T23:45:47.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.302+0000 7f8d16ffd700 1 -- 172.21.15.46:0/1137217745 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8d10017020 con 0x7f8d201599d0 2024-01-26T23:45:47.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.305+0000 7f8d259da700 1 -- 172.21.15.46:0/1137217745 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d00066ea0 msgr2=0x7f8d00069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:47.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.305+0000 7f8d259da700 1 --2- 172.21.15.46:0/1137217745 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d00066ea0 0x7f8d00069350 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8d080093f0 tx=0x7f8d08017000 comp rx=0 tx=0).stop 2024-01-26T23:45:47.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.305+0000 7f8d259da700 1 -- 172.21.15.46:0/1137217745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d201599d0 msgr2=0x7f8d20156240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:47.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.305+0000 7f8d259da700 1 --2- 172.21.15.46:0/1137217745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d201599d0 0x7f8d20156240 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000c7e0 tx=0x7f8d10007e00 comp rx=0 tx=0).stop 2024-01-26T23:45:47.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.305+0000 7f8d259da700 1 -- 172.21.15.46:0/1137217745 shutdown_connections 2024-01-26T23:45:47.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.305+0000 7f8d259da700 1 --2- 172.21.15.46:0/1137217745 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d00066ea0 0x7f8d00069350 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.305+0000 7f8d259da700 1 --2- 172.21.15.46:0/1137217745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d201599d0 0x7f8d20156240 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.305+0000 7f8d259da700 1 --2- 172.21.15.46:0/1137217745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d2015d8c0 0x7f8d200af150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.305+0000 7f8d259da700 1 --2- 172.21.15.46:0/1137217745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d20159da0 0x7f8d2016a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.305+0000 7f8d259da700 1 -- 172.21.15.46:0/1137217745 >> 172.21.15.46:0/1137217745 conn(0x7f8d2006f840 msgr2=0x7f8d200b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:47.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.305+0000 7f8d259da700 1 -- 172.21.15.46:0/1137217745 shutdown_connections 2024-01-26T23:45:47.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.305+0000 7f8d259da700 1 -- 172.21.15.46:0/1137217745 wait complete. 2024-01-26T23:45:47.319 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-26T23:45:47.319 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:45:47.367 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:45:47.407 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-26T23:45:47.408 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-26T23:45:47.408 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:45:47.543 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:47.542+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:47.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.548+0000 7fce7b85d700 1 Processor -- start 2024-01-26T23:45:47.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.548+0000 7fce7b85d700 1 -- start start 2024-01-26T23:45:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.549+0000 7fce7b85d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce7415ce40 0x7fce740d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.549+0000 7fce7b85d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fce740cc1c0 0x7fce740ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.549+0000 7fce7b85d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce740cff70 con 0x7fce7415d210 2024-01-26T23:45:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.549+0000 7fce7b85d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce740cf400 con 0x7fce740cc1c0 2024-01-26T23:45:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.549+0000 7fce7b85d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce740cf660 con 0x7fce7415ce40 2024-01-26T23:45:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.549+0000 7fce78df8700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fce7415d210 0x7fce740cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53158/0 (socket says 172.21.15.46:53158) 2024-01-26T23:45:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.549+0000 7fce78df8700 1 -- 172.21.15.46:0/720625054 learned_addr learned my addr 172.21.15.46:0/720625054 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.549+0000 7fce795f9700 1 --2- 172.21.15.46:0/720625054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce7415ce40 0x7fce740d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.549+0000 7fce79dfa700 1 --2- 172.21.15.46:0/720625054 >> v2:172.21.15.106:3300/0 conn(0x7fce740cc1c0 0x7fce740ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.549+0000 7fce6bfff700 1 -- 172.21.15.46:0/720625054 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875760229 0 0) 0x7fce740cff70 con 0x7fce7415d210 2024-01-26T23:45:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.549+0000 7fce6bfff700 1 -- 172.21.15.46:0/720625054 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce60003570 con 0x7fce7415d210 2024-01-26T23:45:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.549+0000 7fce795f9700 1 -- 172.21.15.46:0/720625054 >> v2:172.21.15.106:3300/0 conn(0x7fce740cc1c0 msgr2=0x7fce740ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.549+0000 7fce795f9700 1 --2- 172.21.15.46:0/720625054 >> v2:172.21.15.106:3300/0 conn(0x7fce740cc1c0 0x7fce740ade30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.549+0000 7fce795f9700 1 -- 172.21.15.46:0/720625054 >> v1:172.21.15.46:6789/0 conn(0x7fce7415d210 legacy=0x7fce740cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.549+0000 7fce795f9700 1 -- 172.21.15.46:0/720625054 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce740d0500 con 0x7fce7415ce40 2024-01-26T23:45:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.549+0000 7fce79dfa700 1 --2- 172.21.15.46:0/720625054 >> v2:172.21.15.106:3300/0 conn(0x7fce740cc1c0 0x7fce740ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.550+0000 7fce795f9700 1 --2- 172.21.15.46:0/720625054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce7415ce40 0x7fce740d0e50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fce700098d0 tx=0x7fce70009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4902009f211274b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.550+0000 7fce6bfff700 1 -- 172.21.15.46:0/720625054 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce70005c10 con 0x7fce7415ce40 2024-01-26T23:45:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.550+0000 7fce6bfff700 1 -- 172.21.15.46:0/720625054 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce70005d70 con 0x7fce7415ce40 2024-01-26T23:45:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.550+0000 7fce6bfff700 1 -- 172.21.15.46:0/720625054 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce70006280 con 0x7fce7415ce40 2024-01-26T23:45:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.551+0000 7fce7b85d700 1 -- 172.21.15.46:0/720625054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce7415ce40 msgr2=0x7fce740d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.551+0000 7fce7b85d700 1 --2- 172.21.15.46:0/720625054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce7415ce40 0x7fce740d0e50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fce700098d0 tx=0x7fce70009120 comp rx=0 tx=0).stop 2024-01-26T23:45:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.551+0000 7fce7b85d700 1 -- 172.21.15.46:0/720625054 shutdown_connections 2024-01-26T23:45:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.551+0000 7fce7b85d700 1 --2- 172.21.15.46:0/720625054 >> v2:172.21.15.106:3300/0 conn(0x7fce740cc1c0 0x7fce740ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.551+0000 7fce7b85d700 1 --2- 172.21.15.46:0/720625054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce7415ce40 0x7fce740d0e50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.551+0000 7fce7b85d700 1 -- 172.21.15.46:0/720625054 >> 172.21.15.46:0/720625054 conn(0x7fce7406f840 msgr2=0x7fce740d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.551+0000 7fce7b85d700 1 -- 172.21.15.46:0/720625054 shutdown_connections 2024-01-26T23:45:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.551+0000 7fce7b85d700 1 -- 172.21.15.46:0/720625054 wait complete. 2024-01-26T23:45:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.552+0000 7fce7b85d700 1 Processor -- start 2024-01-26T23:45:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.552+0000 7fce7b85d700 1 -- start start 2024-01-26T23:45:47.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce7b85d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce740cc1c0 0x7fce7416a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:47.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce7b85d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce7415ce40 0x7fce7416ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:47.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce7b85d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce7415d210 0x7fce74152e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:47.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce7b85d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7416b040 con 0x7fce740cc1c0 2024-01-26T23:45:47.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce7b85d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7416b2c0 con 0x7fce7415d210 2024-01-26T23:45:47.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce7b85d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce7416b5a0 con 0x7fce7415ce40 2024-01-26T23:45:47.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce795f9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce740cc1c0 0x7fce7416a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce795f9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce740cc1c0 0x7fce7416a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35992/0 (socket says 172.21.15.46:35992) 2024-01-26T23:45:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce795f9700 1 -- 172.21.15.46:0/1284780581 learned_addr learned my addr 172.21.15.46:0/1284780581 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce79dfa700 1 --2- 172.21.15.46:0/1284780581 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce7415d210 0x7fce74152e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce78df8700 1 --2- 172.21.15.46:0/1284780581 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce7415ce40 0x7fce7416ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce795f9700 1 -- 172.21.15.46:0/1284780581 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce7415ce40 msgr2=0x7fce7416ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce795f9700 1 --2- 172.21.15.46:0/1284780581 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce7415ce40 0x7fce7416ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce795f9700 1 -- 172.21.15.46:0/1284780581 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce7415d210 msgr2=0x7fce74152e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce795f9700 1 --2- 172.21.15.46:0/1284780581 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce7415d210 0x7fce74152e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce795f9700 1 -- 172.21.15.46:0/1284780581 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c009a20 con 0x7fce740cc1c0 2024-01-26T23:45:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce78df8700 1 --2- 172.21.15.46:0/1284780581 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce7415ce40 0x7fce7416ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.554+0000 7fce795f9700 1 --2- 172.21.15.46:0/1284780581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce740cc1c0 0x7fce7416a570 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fce70009ec0 tx=0x7fce70016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.555+0000 7fce697fa700 1 -- 172.21.15.46:0/1284780581 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce70006a70 con 0x7fce740cc1c0 2024-01-26T23:45:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.555+0000 7fce697fa700 1 -- 172.21.15.46:0/1284780581 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7001d910 con 0x7fce740cc1c0 2024-01-26T23:45:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.555+0000 7fce7b85d700 1 -- 172.21.15.46:0/1284780581 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce700093c0 con 0x7fce740cc1c0 2024-01-26T23:45:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.555+0000 7fce697fa700 1 -- 172.21.15.46:0/1284780581 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7001dcd0 con 0x7fce740cc1c0 2024-01-26T23:45:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.555+0000 7fce7b85d700 1 -- 172.21.15.46:0/1284780581 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce7416bb60 con 0x7fce740cc1c0 2024-01-26T23:45:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.556+0000 7fce7b85d700 1 -- 172.21.15.46:0/1284780581 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce7404c710 con 0x7fce740cc1c0 2024-01-26T23:45:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.556+0000 7fce697fa700 1 -- 172.21.15.46:0/1284780581 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fce7001b690 con 0x7fce740cc1c0 2024-01-26T23:45:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.557+0000 7fce697fa700 1 --2- 172.21.15.46:0/1284780581 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fce58066ea0 0x7fce58069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.557+0000 7fce697fa700 1 -- 172.21.15.46:0/1284780581 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6167+0+0 (secure 0 0 0) 0x7fce700861b0 con 0x7fce740cc1c0 2024-01-26T23:45:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.557+0000 7fce78df8700 1 --2- 172.21.15.46:0/1284780581 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fce58066ea0 0x7fce58069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.557+0000 7fce78df8700 1 --2- 172.21.15.46:0/1284780581 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fce58066ea0 0x7fce58069350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fce74156b30 tx=0x7fce64009040 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.561+0000 7fce697fa700 1 -- 172.21.15.46:0/1284780581 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fce700533f0 con 0x7fce740cc1c0 2024-01-26T23:45:47.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.818+0000 7fce7b85d700 1 -- 172.21.15.46:0/1284780581 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce740cf920 con 0x7fce740cc1c0 2024-01-26T23:45:47.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.819+0000 7fce697fa700 1 -- 172.21.15.46:0/1284780581 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+3535 (secure 0 0 0) 0x7fce7002ade0 con 0x7fce740cc1c0 2024-01-26T23:45:47.819 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:47.819 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":38,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/4083796808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":4083796808},{"type":"v1","addr":"172.21.15.106:6836","nonce":4083796808}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":6172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/1769644590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1769644590},{"type":"v1","addr":"172.21.15.106:6837","nonce":1769644590}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":6184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1994423361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1994423361},{"type":"v1","addr":"172.21.15.46:6835","nonce":1994423361}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":6502,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2146923104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2146923104},{"type":"v1","addr":"172.21.15.46:6834","nonce":2146923104}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[]} 2024-01-26T23:45:47.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.821+0000 7fce7b85d700 1 -- 172.21.15.46:0/1284780581 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fce58066ea0 msgr2=0x7fce58069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:47.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.821+0000 7fce7b85d700 1 --2- 172.21.15.46:0/1284780581 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fce58066ea0 0x7fce58069350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fce74156b30 tx=0x7fce64009040 comp rx=0 tx=0).stop 2024-01-26T23:45:47.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.821+0000 7fce7b85d700 1 -- 172.21.15.46:0/1284780581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce740cc1c0 msgr2=0x7fce7416a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.821+0000 7fce7b85d700 1 --2- 172.21.15.46:0/1284780581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce740cc1c0 0x7fce7416a570 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fce70009ec0 tx=0x7fce70016000 comp rx=0 tx=0).stop 2024-01-26T23:45:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.821+0000 7fce7b85d700 1 -- 172.21.15.46:0/1284780581 shutdown_connections 2024-01-26T23:45:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.821+0000 7fce7b85d700 1 --2- 172.21.15.46:0/1284780581 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fce58066ea0 0x7fce58069350 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.821+0000 7fce7b85d700 1 --2- 172.21.15.46:0/1284780581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce740cc1c0 0x7fce7416a570 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.821+0000 7fce7b85d700 1 --2- 172.21.15.46:0/1284780581 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce7415d210 0x7fce74152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.821+0000 7fce7b85d700 1 --2- 172.21.15.46:0/1284780581 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce7415ce40 0x7fce7416ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.821+0000 7fce7b85d700 1 -- 172.21.15.46:0/1284780581 >> 172.21.15.46:0/1284780581 conn(0x7fce7406f840 msgr2=0x7fce740cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.821+0000 7fce7b85d700 1 -- 172.21.15.46:0/1284780581 shutdown_connections 2024-01-26T23:45:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:47.821+0000 7fce7b85d700 1 -- 172.21.15.46:0/1284780581 wait complete. 2024-01-26T23:45:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 38 2024-01-26T23:45:47.835 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-26T23:45:47.835 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-26T23:45:47.835 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-26T23:45:47.835 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:45:47.836 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-26T23:45:47.836 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-26T23:45:47.836 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-26T23:45:47.836 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:45:47.836 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-26T23:45:47.836 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-26T23:45:47.837 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-26T23:45:47.837 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:45:47.837 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-26T23:45:47.837 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-26T23:45:47.837 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-26T23:45:47.837 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:45:47.845 INFO:tasks.ceph.mds.b:Stopped 2024-01-26T23:45:47.845 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-26T23:45:47.846 INFO:tasks.ceph.mds.d:Stopped 2024-01-26T23:45:47.846 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-26T23:45:47.847 INFO:tasks.ceph.mds.a:Stopped 2024-01-26T23:45:47.847 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-26T23:45:47.848 INFO:tasks.ceph.mds.c:Stopped 2024-01-26T23:45:47.848 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-26T23:45:47.888 INFO:tasks.ceph.mds.b:Started 2024-01-26T23:45:47.888 DEBUG:teuthology.parallel:result is None 2024-01-26T23:45:47.888 INFO:tasks.ceph.mds.d:Started 2024-01-26T23:45:47.889 DEBUG:teuthology.parallel:result is None 2024-01-26T23:45:47.891 INFO:tasks.ceph.mds.a:Started 2024-01-26T23:45:47.891 DEBUG:teuthology.parallel:result is None 2024-01-26T23:45:47.891 INFO:tasks.ceph.mds.c:Started 2024-01-26T23:45:47.891 DEBUG:teuthology.parallel:result is None 2024-01-26T23:45:47.892 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-26T23:45:47.965 INFO:tasks.ceph.mds.b.smithi106.stdout:starting mds.b at 2024-01-26T23:45:47.966 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-01-26T23:45:47.980 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-26T23:45:47.980 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-26T23:45:48.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.036+0000 7fa73779d700 1 Processor -- start 2024-01-26T23:45:48.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.036+0000 7fa73779d700 1 -- start start 2024-01-26T23:45:48.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa73779d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7300b2b40 0x7fa7300ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:48.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa73779d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa7300ae960 0x7fa7300ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:48.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa73779d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73015a0b0 con 0x7fa7300aedd0 2024-01-26T23:45:48.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa73779d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa730099fc0 con 0x7fa7300ae960 2024-01-26T23:45:48.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa73779d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7300ae4d0 con 0x7fa7300b2b40 2024-01-26T23:45:48.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa735539700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7300b2b40 0x7fa7300ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:48.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa734d38700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa7300ae960 0x7fa7300ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:48.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa735d3a700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fa7300aedd0 0x7fa7300cb5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:39774/0 (socket says 172.21.15.46:39774) 2024-01-26T23:45:48.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa735d3a700 1 -- 172.21.15.46:0/1751439086 learned_addr learned my addr 172.21.15.46:0/1751439086 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:48.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa734d38700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa7300ae960 0x7fa7300ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36032/0 (socket says 172.21.15.46:36032) 2024-01-26T23:45:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa735539700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7300b2b40 0x7fa7300ad1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:56814/0 (socket says 172.21.15.46:56814) 2024-01-26T23:45:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa727fff700 1 -- 172.21.15.46:0/1751439086 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1878396010 0 0) 0x7fa73015a0b0 con 0x7fa7300aedd0 2024-01-26T23:45:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa734d38700 1 -- 172.21.15.46:0/1751439086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7300b2b40 msgr2=0x7fa7300ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa734d38700 1 --2- 172.21.15.46:0/1751439086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7300b2b40 0x7fa7300ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa734d38700 1 -- 172.21.15.46:0/1751439086 >> v1:172.21.15.106:6789/0 conn(0x7fa7300aedd0 legacy=0x7fa7300cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa734d38700 1 -- 172.21.15.46:0/1751439086 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7300ad8c0 con 0x7fa7300ae960 2024-01-26T23:45:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa735539700 1 --2- 172.21.15.46:0/1751439086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7300b2b40 0x7fa7300ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.038+0000 7fa734d38700 1 --2- 172.21.15.46:0/1751439086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7300ae960 0x7fa7300ced50 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fa7200098e0 tx=0x7fa7200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2665fe434a98730d server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.039+0000 7fa727fff700 1 -- 172.21.15.46:0/1751439086 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72000b640 con 0x7fa7300ae960 2024-01-26T23:45:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.039+0000 7fa727fff700 1 -- 172.21.15.46:0/1751439086 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72000b7a0 con 0x7fa7300ae960 2024-01-26T23:45:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.039+0000 7fa727fff700 1 -- 172.21.15.46:0/1751439086 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72000bb40 con 0x7fa7300ae960 2024-01-26T23:45:48.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.039+0000 7fa73779d700 1 -- 172.21.15.46:0/1751439086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7300ae960 msgr2=0x7fa7300ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:48.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.039+0000 7fa73779d700 1 --2- 172.21.15.46:0/1751439086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7300ae960 0x7fa7300ced50 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fa7200098e0 tx=0x7fa7200091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:48.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.039+0000 7fa73779d700 1 -- 172.21.15.46:0/1751439086 shutdown_connections 2024-01-26T23:45:48.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.039+0000 7fa73779d700 1 --2- 172.21.15.46:0/1751439086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7300b2b40 0x7fa7300ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:48.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.039+0000 7fa73779d700 1 --2- 172.21.15.46:0/1751439086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7300ae960 0x7fa7300ced50 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:48.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.039+0000 7fa73779d700 1 -- 172.21.15.46:0/1751439086 >> 172.21.15.46:0/1751439086 conn(0x7fa730099a30 msgr2=0x7fa73006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:48.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.039+0000 7fa73779d700 1 -- 172.21.15.46:0/1751439086 shutdown_connections 2024-01-26T23:45:48.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.039+0000 7fa73779d700 1 -- 172.21.15.46:0/1751439086 wait complete. 2024-01-26T23:45:48.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.041+0000 7fa73779d700 1 Processor -- start 2024-01-26T23:45:48.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.041+0000 7fa73779d700 1 -- start start 2024-01-26T23:45:48.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.042+0000 7fa73779d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7300ae960 0x7fa7300b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:48.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.042+0000 7fa73779d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7300b2b40 0x7fa73015ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:48.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.042+0000 7fa73779d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa73015cfe0 0x7fa7300b5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:48.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.042+0000 7fa73779d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7300b6160 con 0x7fa73015cfe0 2024-01-26T23:45:48.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.042+0000 7fa73779d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73015f240 con 0x7fa7300b2b40 2024-01-26T23:45:48.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.042+0000 7fa73779d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7300af9a0 con 0x7fa7300ae960 2024-01-26T23:45:48.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.042+0000 7fa735d3a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa73015cfe0 0x7fa7300b5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:48.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.042+0000 7fa735539700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7300ae960 0x7fa7300b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:48.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.042+0000 7fa734d38700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7300b2b40 0x7fa73015ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:48.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.042+0000 7fa735d3a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa73015cfe0 0x7fa7300b5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36034/0 (socket says 172.21.15.46:36034) 2024-01-26T23:45:48.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.042+0000 7fa735d3a700 1 -- 172.21.15.46:0/2745300567 learned_addr learned my addr 172.21.15.46:0/2745300567 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:48.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.043+0000 7fa735d3a700 1 -- 172.21.15.46:0/2745300567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7300ae960 msgr2=0x7fa7300b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:48.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.043+0000 7fa735d3a700 1 --2- 172.21.15.46:0/2745300567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7300ae960 0x7fa7300b1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:48.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.043+0000 7fa735d3a700 1 -- 172.21.15.46:0/2745300567 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7300b2b40 msgr2=0x7fa73015ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:48.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.043+0000 7fa735d3a700 1 --2- 172.21.15.46:0/2745300567 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7300b2b40 0x7fa73015ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:48.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.043+0000 7fa735d3a700 1 -- 172.21.15.46:0/2745300567 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72c0093c0 con 0x7fa73015cfe0 2024-01-26T23:45:48.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.043+0000 7fa735539700 1 --2- 172.21.15.46:0/2745300567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7300ae960 0x7fa7300b1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:48.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.043+0000 7fa735d3a700 1 --2- 172.21.15.46:0/2745300567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa73015cfe0 0x7fa7300b5600 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fa72800cbc0 tx=0x7fa728001910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:48.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.043+0000 7fa7257fa700 1 -- 172.21.15.46:0/2745300567 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72800a760 con 0x7fa73015cfe0 2024-01-26T23:45:48.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.043+0000 7fa7257fa700 1 -- 172.21.15.46:0/2745300567 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72800a8e0 con 0x7fa73015cfe0 2024-01-26T23:45:48.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.043+0000 7fa73779d700 1 -- 172.21.15.46:0/2745300567 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa720009420 con 0x7fa73015cfe0 2024-01-26T23:45:48.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.043+0000 7fa7257fa700 1 -- 172.21.15.46:0/2745300567 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72800aca0 con 0x7fa73015cfe0 2024-01-26T23:45:48.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.043+0000 7fa73779d700 1 -- 172.21.15.46:0/2745300567 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7300af350 con 0x7fa73015cfe0 2024-01-26T23:45:48.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.044+0000 7fa73779d700 1 -- 172.21.15.46:0/2745300567 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa73004c710 con 0x7fa73015cfe0 2024-01-26T23:45:48.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.045+0000 7fa7257fa700 1 -- 172.21.15.46:0/2745300567 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fa728007300 con 0x7fa73015cfe0 2024-01-26T23:45:48.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.045+0000 7fa7257fa700 1 --2- 172.21.15.46:0/2745300567 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa714066ea0 0x7fa714069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:48.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.045+0000 7fa7257fa700 1 -- 172.21.15.46:0/2745300567 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6167+0+0 (secure 0 0 0) 0x7fa7280848f0 con 0x7fa73015cfe0 2024-01-26T23:45:48.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.045+0000 7fa735539700 1 --2- 172.21.15.46:0/2745300567 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa714066ea0 0x7fa714069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:48.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.046+0000 7fa735539700 1 --2- 172.21.15.46:0/2745300567 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa714066ea0 0x7fa714069350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009fd0 tx=0x7fa72c009120 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:48.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.050+0000 7fa7257fa700 1 -- 172.21.15.46:0/2745300567 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa728051a60 con 0x7fa73015cfe0 2024-01-26T23:45:48.216 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:48.215+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.275+0000 7fa73779d700 1 -- 172.21.15.46:0/2745300567 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa7300afae0 con 0x7fa73015cfe0 2024-01-26T23:45:48.903 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:48.902+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:48.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.916+0000 7fa7257fa700 1 -- 172.21.15.46:0/2745300567 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v92) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa728055080 con 0x7fa73015cfe0 2024-01-26T23:45:48.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.919+0000 7fa73779d700 1 -- 172.21.15.46:0/2745300567 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa714066ea0 msgr2=0x7fa714069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:48.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.919+0000 7fa73779d700 1 --2- 172.21.15.46:0/2745300567 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa714066ea0 0x7fa714069350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009fd0 tx=0x7fa72c009120 comp rx=0 tx=0).stop 2024-01-26T23:45:48.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.920+0000 7fa73779d700 1 -- 172.21.15.46:0/2745300567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa73015cfe0 msgr2=0x7fa7300b5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:48.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.920+0000 7fa73779d700 1 --2- 172.21.15.46:0/2745300567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa73015cfe0 0x7fa7300b5600 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fa72800cbc0 tx=0x7fa728001910 comp rx=0 tx=0).stop 2024-01-26T23:45:48.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.920+0000 7fa73779d700 1 -- 172.21.15.46:0/2745300567 shutdown_connections 2024-01-26T23:45:48.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.920+0000 7fa73779d700 1 --2- 172.21.15.46:0/2745300567 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa714066ea0 0x7fa714069350 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:48.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.920+0000 7fa73779d700 1 --2- 172.21.15.46:0/2745300567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa73015cfe0 0x7fa7300b5600 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:48.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.920+0000 7fa73779d700 1 --2- 172.21.15.46:0/2745300567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7300ae960 0x7fa7300b1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:48.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.920+0000 7fa73779d700 1 --2- 172.21.15.46:0/2745300567 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7300b2b40 0x7fa73015ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:48.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.920+0000 7fa73779d700 1 -- 172.21.15.46:0/2745300567 >> 172.21.15.46:0/2745300567 conn(0x7fa730099a30 msgr2=0x7fa730159e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:48.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.920+0000 7fa73779d700 1 -- 172.21.15.46:0/2745300567 shutdown_connections 2024-01-26T23:45:48.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:48.920+0000 7fa73779d700 1 -- 172.21.15.46:0/2745300567 wait complete. 2024-01-26T23:45:48.921 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-01-26T23:45:48.937 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-26T23:45:49.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.074+0000 7faa5045f700 1 Processor -- start 2024-01-26T23:45:49.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.074+0000 7faa5045f700 1 -- start start 2024-01-26T23:45:49.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.075+0000 7faa5045f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7faa480cc1c0 0x7faa48164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:49.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.075+0000 7faa5045f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa480cff70 con 0x7faa4815ce40 2024-01-26T23:45:49.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.075+0000 7faa5045f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa480cf400 con 0x7faa4815d210 2024-01-26T23:45:49.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.075+0000 7faa5045f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa480cf660 con 0x7faa480cc1c0 2024-01-26T23:45:49.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.076+0000 7faa4e1fb700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faa4815ce40 0x7faa480d0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53184/0 (socket says 172.21.15.46:53184) 2024-01-26T23:45:49.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.076+0000 7faa4e1fb700 1 -- 172.21.15.46:0/817620245 learned_addr learned my addr 172.21.15.46:0/817620245 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:49.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.076+0000 7faa4e9fc700 1 --2- 172.21.15.46:0/817620245 >> v2:172.21.15.106:3300/0 conn(0x7faa480cc1c0 0x7faa48164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:49.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.076+0000 7faa4d1f9700 1 -- 172.21.15.46:0/817620245 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208949186 0 0) 0x7faa480cff70 con 0x7faa4815ce40 2024-01-26T23:45:49.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.076+0000 7faa4d1f9700 1 -- 172.21.15.46:0/817620245 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa34003570 con 0x7faa4815ce40 2024-01-26T23:45:49.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.076+0000 7faa4d1f9700 1 -- 172.21.15.46:0/817620245 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395637285 0 0) 0x7faa480cf400 con 0x7faa4815d210 2024-01-26T23:45:49.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.076+0000 7faa4e9fc700 1 -- 172.21.15.46:0/817620245 >> v1:172.21.15.106:6789/0 conn(0x7faa4815d210 legacy=0x7faa480cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:49.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.076+0000 7faa4e9fc700 1 -- 172.21.15.46:0/817620245 >> v1:172.21.15.46:6789/0 conn(0x7faa4815ce40 legacy=0x7faa480d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:49.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.076+0000 7faa4e9fc700 1 -- 172.21.15.46:0/817620245 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa480d0500 con 0x7faa480cc1c0 2024-01-26T23:45:49.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.076+0000 7faa4e9fc700 1 --2- 172.21.15.46:0/817620245 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa480cc1c0 0x7faa48164700 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7faa40009e60 tx=0x7faa40009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86404452318c0af4 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:49.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.076+0000 7faa4d1f9700 1 -- 172.21.15.46:0/817620245 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4000e070 con 0x7faa480cc1c0 2024-01-26T23:45:49.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.076+0000 7faa4d1f9700 1 -- 172.21.15.46:0/817620245 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4000bb60 con 0x7faa480cc1c0 2024-01-26T23:45:49.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.076+0000 7faa4d1f9700 1 -- 172.21.15.46:0/817620245 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4000bee0 con 0x7faa480cc1c0 2024-01-26T23:45:49.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.077+0000 7faa5045f700 1 -- 172.21.15.46:0/817620245 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa480cc1c0 msgr2=0x7faa48164700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:49.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.077+0000 7faa5045f700 1 --2- 172.21.15.46:0/817620245 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa480cc1c0 0x7faa48164700 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7faa40009e60 tx=0x7faa40009780 comp rx=0 tx=0).stop 2024-01-26T23:45:49.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.077+0000 7faa5045f700 1 -- 172.21.15.46:0/817620245 shutdown_connections 2024-01-26T23:45:49.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.077+0000 7faa5045f700 1 --2- 172.21.15.46:0/817620245 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa480cc1c0 0x7faa48164700 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.077+0000 7faa5045f700 1 -- 172.21.15.46:0/817620245 >> 172.21.15.46:0/817620245 conn(0x7faa4806f840 msgr2=0x7faa480d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:49.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.077+0000 7faa5045f700 1 -- 172.21.15.46:0/817620245 shutdown_connections 2024-01-26T23:45:49.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.077+0000 7faa5045f700 1 -- 172.21.15.46:0/817620245 wait complete. 2024-01-26T23:45:49.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.078+0000 7faa5045f700 1 Processor -- start 2024-01-26T23:45:49.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.079+0000 7faa5045f700 1 -- start start 2024-01-26T23:45:49.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.079+0000 7faa5045f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa480cc1c0 0x7faa4816d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:49.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.079+0000 7faa5045f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa4815ce40 0x7faa481595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:49.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.079+0000 7faa5045f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa4815d210 0x7faa4815eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:49.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.079+0000 7faa5045f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4815a5e0 con 0x7faa480cc1c0 2024-01-26T23:45:49.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.079+0000 7faa5045f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4816de20 con 0x7faa4815d210 2024-01-26T23:45:49.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.079+0000 7faa5045f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa48159b20 con 0x7faa4815ce40 2024-01-26T23:45:49.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.079+0000 7faa4e1fb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa480cc1c0 0x7faa4816d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:49.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.079+0000 7faa4e1fb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa480cc1c0 0x7faa4816d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36046/0 (socket says 172.21.15.46:36046) 2024-01-26T23:45:49.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.079+0000 7faa4e1fb700 1 -- 172.21.15.46:0/740341696 learned_addr learned my addr 172.21.15.46:0/740341696 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:49.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.080+0000 7faa4e9fc700 1 --2- 172.21.15.46:0/740341696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa4815d210 0x7faa4815eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:49.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.080+0000 7faa4d9fa700 1 --2- 172.21.15.46:0/740341696 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa4815ce40 0x7faa481595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:49.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.080+0000 7faa4e1fb700 1 -- 172.21.15.46:0/740341696 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa4815ce40 msgr2=0x7faa481595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:49.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.080+0000 7faa4e1fb700 1 --2- 172.21.15.46:0/740341696 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa4815ce40 0x7faa481595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.080+0000 7faa4e1fb700 1 -- 172.21.15.46:0/740341696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa4815d210 msgr2=0x7faa4815eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:49.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.080+0000 7faa4e1fb700 1 --2- 172.21.15.46:0/740341696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa4815d210 0x7faa4815eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.080+0000 7faa4e1fb700 1 -- 172.21.15.46:0/740341696 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa40009a20 con 0x7faa480cc1c0 2024-01-26T23:45:49.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.080+0000 7faa4e9fc700 1 --2- 172.21.15.46:0/740341696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa4815d210 0x7faa4815eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:49.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.080+0000 7faa4e1fb700 1 --2- 172.21.15.46:0/740341696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa480cc1c0 0x7faa4816d8e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7faa4400c790 tx=0x7faa44007c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:49.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.080+0000 7faa3e7fc700 1 -- 172.21.15.46:0/740341696 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa440072e0 con 0x7faa480cc1c0 2024-01-26T23:45:49.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.080+0000 7faa3e7fc700 1 -- 172.21.15.46:0/740341696 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa44007460 con 0x7faa480cc1c0 2024-01-26T23:45:49.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.080+0000 7faa3e7fc700 1 -- 172.21.15.46:0/740341696 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa440057f0 con 0x7faa480cc1c0 2024-01-26T23:45:49.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.080+0000 7faa5045f700 1 -- 172.21.15.46:0/740341696 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa480c7c10 con 0x7faa480cc1c0 2024-01-26T23:45:49.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.080+0000 7faa5045f700 1 -- 172.21.15.46:0/740341696 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa480c81d0 con 0x7faa480cc1c0 2024-01-26T23:45:49.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.081+0000 7faa5045f700 1 -- 172.21.15.46:0/740341696 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa4804c710 con 0x7faa480cc1c0 2024-01-26T23:45:49.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.082+0000 7faa3e7fc700 1 -- 172.21.15.46:0/740341696 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7faa44005bc0 con 0x7faa480cc1c0 2024-01-26T23:45:49.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.082+0000 7faa3e7fc700 1 --2- 172.21.15.46:0/740341696 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa2c066ea0 0x7faa2c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:49.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.083+0000 7faa3e7fc700 1 -- 172.21.15.46:0/740341696 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5565+0+0 (secure 0 0 0) 0x7faa44083810 con 0x7faa480cc1c0 2024-01-26T23:45:49.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.083+0000 7faa4d9fa700 1 --2- 172.21.15.46:0/740341696 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa2c066ea0 0x7faa2c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:49.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.083+0000 7faa4d9fa700 1 --2- 172.21.15.46:0/740341696 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa2c066ea0 0x7faa2c069350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faa4815b800 tx=0x7faa38007ec0 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:49.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.087+0000 7faa3e7fc700 1 -- 172.21.15.46:0/740341696 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faa44050bf0 con 0x7faa480cc1c0 2024-01-26T23:45:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.379+0000 7faa5045f700 1 -- 172.21.15.46:0/740341696 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7faa480cf920 con 0x7faa480cc1c0 2024-01-26T23:45:49.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.379+0000 7faa3e7fc700 1 -- 172.21.15.46:0/740341696 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7faa44017070 con 0x7faa480cc1c0 2024-01-26T23:45:49.380 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:49.380 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:45:49.380 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-01-26T23:45:49.380 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.380 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-01-26T23:45:49.380 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlM6dKKBAAPpeWeDZWrFgc/4zuxKCvHw==", 2024-01-26T23:45:49.380 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.381 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:45:49.381 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:45:49.381 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:45:49.381 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:49.381 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.381 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.381 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.381 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-01-26T23:45:49.381 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlmzHRNRAAflWYZ/p4AXeMy54tvLpugA==", 2024-01-26T23:45:49.381 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.381 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:45:49.381 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:45:49.381 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:45:49.381 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:49.382 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.382 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.382 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.382 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-01-26T23:45:49.382 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlMZxoMBAAG5kyfm+LLG86y/p2A8BfDQ==", 2024-01-26T23:45:49.382 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.382 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:45:49.382 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:45:49.382 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:45:49.382 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:49.382 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.382 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.382 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.382 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-01-26T23:45:49.382 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl5jx/ARAA3cwtLAOWQ/n2NG+xwEZ3QQ==", 2024-01-26T23:45:49.383 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.383 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:45:49.383 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:45:49.383 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:45:49.383 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:49.383 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.383 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.383 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.383 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-01-26T23:45:49.383 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4QLRlJ2OKJRAAWVhxEdQkUW20294w/KLL6g==", 2024-01-26T23:45:49.383 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.383 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:45:49.383 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:45:49.383 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:49.383 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.384 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.384 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.384 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-01-26T23:45:49.384 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7QLRlE2tAJRAAzNlTiCr6a2eWiabRy1SqSw==", 2024-01-26T23:45:49.384 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.384 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:45:49.384 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:45:49.384 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:49.384 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.384 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.384 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.384 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-01-26T23:45:49.384 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+QLRlyySaJBAA5072ba5HUNi2dYE3D02M/Q==", 2024-01-26T23:45:49.384 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.385 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:45:49.385 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:45:49.385 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:49.385 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.385 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.385 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.385 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-01-26T23:45:49.385 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBQLRl9+tAIxAApZk0btXIpQPWSb3BYUYsFQ==", 2024-01-26T23:45:49.385 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.385 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:45:49.385 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:45:49.385 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:49.385 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.386 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.386 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.386 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-01-26T23:45:49.386 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHQLRlurUtJxAAT9fwKrAWl62VOXwShg7NJg==", 2024-01-26T23:45:49.386 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.386 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:45:49.386 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:45:49.386 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:49.386 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.386 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.386 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.386 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-01-26T23:45:49.386 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKQLRl7aqoJRAAXu9fykWELsW3SwnahoIOLA==", 2024-01-26T23:45:49.387 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.387 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:45:49.387 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:45:49.387 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:49.387 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.387 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.387 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.387 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-01-26T23:45:49.387 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNQLRlWf7pJBAAopmFLIKb8s32YZs46VwbGg==", 2024-01-26T23:45:49.387 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.387 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:45:49.387 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:45:49.388 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:49.388 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.388 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.388 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.388 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-01-26T23:45:49.388 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQQLRlNGfhIxAAmCgmOB3QAGfSIYHBb5BUZw==", 2024-01-26T23:45:49.388 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.388 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:45:49.388 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:45:49.388 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:49.388 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.388 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.388 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.388 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-01-26T23:45:49.389 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRleO8wBhAA1018AcO3KcyOZxgB0Itxaw==", 2024-01-26T23:45:49.389 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.389 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:45:49.389 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-01-26T23:45:49.389 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-26T23:45:49.389 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.389 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.390 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.390 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-01-26T23:45:49.390 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlsiZEDRAAjXSamHe2bZcDqVsqXqIcGg==", 2024-01-26T23:45:49.390 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.390 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:45:49.390 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:45:49.390 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:45:49.390 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:45:49.390 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.390 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.390 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.390 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-01-26T23:45:49.390 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl6YT1CRAApXlN4j20xzwda0tDMy94Eg==", 2024-01-26T23:45:49.390 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.391 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:45:49.391 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:45:49.391 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:45:49.391 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:45:49.391 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.391 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.391 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.391 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-01-26T23:45:49.391 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlHvDQEhAAxMhgWJsZC5/6Af5zFV6XIA==", 2024-01-26T23:45:49.391 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.391 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:45:49.391 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:45:49.391 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:45:49.391 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:45:49.391 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.392 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.392 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.392 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-01-26T23:45:49.392 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlvMwDCBAAXnFyMduz7PfWdLWbjytIiQ==", 2024-01-26T23:45:49.392 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.392 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:45:49.392 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:45:49.392 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:45:49.392 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:49.392 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.392 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.392 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.392 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-01-26T23:45:49.392 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlhMNSJBAA38QU6DrCvFjO+u9zQ8bkEA==", 2024-01-26T23:45:49.392 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.393 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:45:49.393 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-26T23:45:49.393 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:49.393 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.393 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:45:49.393 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:49.393 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-01-26T23:45:49.393 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlI9EbIBAACTpzAO+8A/VFJZcT/WWBng==", 2024-01-26T23:45:49.393 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:45:49.393 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:45:49.393 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-26T23:45:49.393 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:45:49.393 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.393 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:49.394 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-01-26T23:45:49.394 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:45:49.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.382+0000 7faa5045f700 1 -- 172.21.15.46:0/740341696 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa2c066ea0 msgr2=0x7faa2c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:49.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.382+0000 7faa5045f700 1 --2- 172.21.15.46:0/740341696 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa2c066ea0 0x7faa2c069350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faa4815b800 tx=0x7faa38007ec0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.382+0000 7faa5045f700 1 -- 172.21.15.46:0/740341696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa480cc1c0 msgr2=0x7faa4816d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:49.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.382+0000 7faa5045f700 1 --2- 172.21.15.46:0/740341696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa480cc1c0 0x7faa4816d8e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7faa4400c790 tx=0x7faa44007c60 comp rx=0 tx=0).stop 2024-01-26T23:45:49.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.382+0000 7faa5045f700 1 -- 172.21.15.46:0/740341696 shutdown_connections 2024-01-26T23:45:49.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.382+0000 7faa5045f700 1 --2- 172.21.15.46:0/740341696 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faa2c066ea0 0x7faa2c069350 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.382+0000 7faa5045f700 1 --2- 172.21.15.46:0/740341696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa480cc1c0 0x7faa4816d8e0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.382+0000 7faa5045f700 1 --2- 172.21.15.46:0/740341696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa4815d210 0x7faa4815eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.382+0000 7faa5045f700 1 --2- 172.21.15.46:0/740341696 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa4815ce40 0x7faa481595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.382+0000 7faa5045f700 1 -- 172.21.15.46:0/740341696 >> 172.21.15.46:0/740341696 conn(0x7faa4806f840 msgr2=0x7faa480d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:49.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.382+0000 7faa5045f700 1 -- 172.21.15.46:0/740341696 shutdown_connections 2024-01-26T23:45:49.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.382+0000 7faa5045f700 1 -- 172.21.15.46:0/740341696 wait complete. 2024-01-26T23:45:49.396 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:45:49.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.531+0000 7fe2afd76700 1 Processor -- start 2024-01-26T23:45:49.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.531+0000 7fe2afd76700 1 -- start start 2024-01-26T23:45:49.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.532+0000 7fe2afd76700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe2a80ae520 0x7fe2a80b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:49.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.532+0000 7fe2afd76700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2a80b3fb0 0x7fe2a8157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:49.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.532+0000 7fe2afd76700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a8158ae0 con 0x7fe2a8157340 2024-01-26T23:45:49.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.532+0000 7fe2afd76700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2a815b650 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.532+0000 7fe2afd76700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2a80b43c0 con 0x7fe2a80ae520 2024-01-26T23:45:49.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.532+0000 7fe2ae313700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2a80b3fb0 0x7fe2a8157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:49.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.533+0000 7fe2ae313700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2a80b3fb0 0x7fe2a8157710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36052/0 (socket says 172.21.15.46:36052) 2024-01-26T23:45:49.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.533+0000 7fe2ae313700 1 -- 172.21.15.46:0/3294010496 learned_addr learned my addr 172.21.15.46:0/3294010496 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:49.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.533+0000 7fe2ad311700 1 --2- 172.21.15.46:0/3294010496 >> v2:172.21.15.106:3300/0 conn(0x7fe2a80ae520 0x7fe2a80b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:49.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.533+0000 7fe2ae313700 1 -- 172.21.15.46:0/3294010496 >> v2:172.21.15.106:3300/0 conn(0x7fe2a80ae520 msgr2=0x7fe2a80b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:49.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.533+0000 7fe2ae313700 1 --2- 172.21.15.46:0/3294010496 >> v2:172.21.15.106:3300/0 conn(0x7fe2a80ae520 0x7fe2a80b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.533+0000 7fe2ae313700 1 -- 172.21.15.46:0/3294010496 >> v1:172.21.15.106:6789/0 conn(0x7fe2a8157340 legacy=0x7fe2a815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:49.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.533+0000 7fe2ae313700 1 -- 172.21.15.46:0/3294010496 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a8158520 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.533+0000 7fe2ae313700 1 --2- 172.21.15.46:0/3294010496 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a80b3fb0 0x7fe2a8157710 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fe2a806e840 tx=0x7fe2a0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42888b781ad80388 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:49.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.533+0000 7fe2acb10700 1 -- 172.21.15.46:0/3294010496 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a000e070 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.533+0000 7fe2acb10700 1 -- 172.21.15.46:0/3294010496 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a000ba20 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.533+0000 7fe2acb10700 1 -- 172.21.15.46:0/3294010496 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a00092e0 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.534+0000 7fe2afd76700 1 -- 172.21.15.46:0/3294010496 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a80b3fb0 msgr2=0x7fe2a8157710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:49.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.534+0000 7fe2afd76700 1 --2- 172.21.15.46:0/3294010496 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a80b3fb0 0x7fe2a8157710 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fe2a806e840 tx=0x7fe2a0009840 comp rx=0 tx=0).stop 2024-01-26T23:45:49.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.534+0000 7fe2afd76700 1 -- 172.21.15.46:0/3294010496 shutdown_connections 2024-01-26T23:45:49.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.534+0000 7fe2afd76700 1 --2- 172.21.15.46:0/3294010496 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a80b3fb0 0x7fe2a8157710 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.534+0000 7fe2afd76700 1 --2- 172.21.15.46:0/3294010496 >> v2:172.21.15.106:3300/0 conn(0x7fe2a80ae520 0x7fe2a80b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.534+0000 7fe2afd76700 1 -- 172.21.15.46:0/3294010496 >> 172.21.15.46:0/3294010496 conn(0x7fe2a80ce530 msgr2=0x7fe2a80ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:49.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.534+0000 7fe2afd76700 1 -- 172.21.15.46:0/3294010496 shutdown_connections 2024-01-26T23:45:49.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.534+0000 7fe2afd76700 1 -- 172.21.15.46:0/3294010496 wait complete. 2024-01-26T23:45:49.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.535+0000 7fe2afd76700 1 Processor -- start 2024-01-26T23:45:49.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.535+0000 7fe2afd76700 1 -- start start 2024-01-26T23:45:49.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.536+0000 7fe2afd76700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2a80ae520 0x7fe2a80c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:49.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.536+0000 7fe2afd76700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a80b3fb0 0x7fe2a816a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:49.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.536+0000 7fe2afd76700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2a8157340 0x7fe2a816a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:49.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.536+0000 7fe2afd76700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a80b3a60 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.536+0000 7fe2afd76700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a814d280 con 0x7fe2a80ae520 2024-01-26T23:45:49.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.536+0000 7fe2afd76700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a8169e70 con 0x7fe2a8157340 2024-01-26T23:45:49.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.536+0000 7fe2ad311700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a80b3fb0 0x7fe2a816a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:49.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.536+0000 7fe2adb12700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2a80ae520 0x7fe2a80c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:49.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.536+0000 7fe2ad311700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a80b3fb0 0x7fe2a816a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36054/0 (socket says 172.21.15.46:36054) 2024-01-26T23:45:49.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.536+0000 7fe2ad311700 1 -- 172.21.15.46:0/3361728250 learned_addr learned my addr 172.21.15.46:0/3361728250 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:49.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.536+0000 7fe2ae313700 1 --2- 172.21.15.46:0/3361728250 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2a8157340 0x7fe2a816a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.537+0000 7fe2ad311700 1 -- 172.21.15.46:0/3361728250 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2a8157340 msgr2=0x7fe2a816a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.537+0000 7fe2ad311700 1 --2- 172.21.15.46:0/3361728250 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2a8157340 0x7fe2a816a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.537+0000 7fe2ad311700 1 -- 172.21.15.46:0/3361728250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2a80ae520 msgr2=0x7fe2a80c9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.537+0000 7fe2ad311700 1 --2- 172.21.15.46:0/3361728250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2a80ae520 0x7fe2a80c9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.537+0000 7fe2ad311700 1 -- 172.21.15.46:0/3361728250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a0009ae0 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.537+0000 7fe2ad311700 1 --2- 172.21.15.46:0/3361728250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a80b3fb0 0x7fe2a816a300 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe29800e930 tx=0x7fe2980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.537+0000 7fe29dffb700 1 -- 172.21.15.46:0/3361728250 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe298013680 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.537+0000 7fe29dffb700 1 -- 172.21.15.46:0/3361728250 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2980137e0 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.537+0000 7fe29dffb700 1 -- 172.21.15.46:0/3361728250 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe298013b80 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.537+0000 7fe2afd76700 1 -- 172.21.15.46:0/3361728250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a8169020 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.537+0000 7fe2afd76700 1 -- 172.21.15.46:0/3361728250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a8169820 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.538+0000 7fe2afd76700 1 -- 172.21.15.46:0/3361728250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a806bfe0 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.539+0000 7fe29dffb700 1 -- 172.21.15.46:0/3361728250 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fe29800c3e0 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.539+0000 7fe29dffb700 1 --2- 172.21.15.46:0/3361728250 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe28c066ea0 0x7fe28c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:49.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.539+0000 7fe29dffb700 1 -- 172.21.15.46:0/3361728250 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5565+0+0 (secure 0 0 0) 0x7fe2980842d0 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.539+0000 7fe2adb12700 1 --2- 172.21.15.46:0/3361728250 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe28c066ea0 0x7fe28c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:49.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.540+0000 7fe2adb12700 1 --2- 172.21.15.46:0/3361728250 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe28c066ea0 0x7fe28c069350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe2a80cc360 tx=0x7fe2a4009040 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:49.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.544+0000 7fe29dffb700 1 -- 172.21.15.46:0/3361728250 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe2980516b0 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.582 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:49.581+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:49.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.750+0000 7fe2afd76700 1 -- 172.21.15.46:0/3361728250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2a8168fc0 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.751+0000 7fe29dffb700 1 -- 172.21.15.46:0/3361728250 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+12934 (secure 0 0 0) 0x7fe298054cd0 con 0x7fe2a80b3fb0 2024-01-26T23:45:49.751 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":94,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:45:48.922295+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:45:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.753+0000 7fe2afd76700 1 -- 172.21.15.46:0/3361728250 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe28c066ea0 msgr2=0x7fe28c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.753+0000 7fe2afd76700 1 --2- 172.21.15.46:0/3361728250 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe28c066ea0 0x7fe28c069350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe2a80cc360 tx=0x7fe2a4009040 comp rx=0 tx=0).stop 2024-01-26T23:45:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.753+0000 7fe2afd76700 1 -- 172.21.15.46:0/3361728250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a80b3fb0 msgr2=0x7fe2a816a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.753+0000 7fe2afd76700 1 --2- 172.21.15.46:0/3361728250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a80b3fb0 0x7fe2a816a300 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe29800e930 tx=0x7fe2980091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.753+0000 7fe2afd76700 1 -- 172.21.15.46:0/3361728250 shutdown_connections 2024-01-26T23:45:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.753+0000 7fe2afd76700 1 --2- 172.21.15.46:0/3361728250 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe28c066ea0 0x7fe28c069350 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.753+0000 7fe2afd76700 1 --2- 172.21.15.46:0/3361728250 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2a8157340 0x7fe2a816a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.753+0000 7fe2afd76700 1 --2- 172.21.15.46:0/3361728250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2a80ae520 0x7fe2a80c9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.753+0000 7fe2afd76700 1 --2- 172.21.15.46:0/3361728250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a80b3fb0 0x7fe2a816a300 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.753+0000 7fe2afd76700 1 -- 172.21.15.46:0/3361728250 >> 172.21.15.46:0/3361728250 conn(0x7fe2a80ce530 msgr2=0x7fe2a8156a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:49.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.753+0000 7fe2afd76700 1 -- 172.21.15.46:0/3361728250 shutdown_connections 2024-01-26T23:45:49.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.753+0000 7fe2afd76700 1 -- 172.21.15.46:0/3361728250 wait complete. 2024-01-26T23:45:49.768 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-26T23:45:49.769 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:45:49.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.906+0000 7f7c3ce70700 1 Processor -- start 2024-01-26T23:45:49.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.906+0000 7f7c3ce70700 1 -- start start 2024-01-26T23:45:49.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.907+0000 7f7c3ce70700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7c38068b00 0x7f7c38068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:49.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.907+0000 7f7c3ce70700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c3806add0 0x7f7c3806b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:49.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.907+0000 7f7c3ce70700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c380a9a80 con 0x7f7c3806a660 2024-01-26T23:45:49.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.907+0000 7f7c3ce70700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c3806b790 con 0x7f7c3806add0 2024-01-26T23:45:49.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.907+0000 7f7c3ce70700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c3806aab0 con 0x7f7c38068b00 2024-01-26T23:45:49.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.907+0000 7f7c36d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c3806add0 0x7f7c3806b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:49.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.907+0000 7f7c36d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c3806add0 0x7f7c3806b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36060/0 (socket says 172.21.15.46:36060) 2024-01-26T23:45:49.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.907+0000 7f7c36d9d700 1 -- 172.21.15.46:0/2107179548 learned_addr learned my addr 172.21.15.46:0/2107179548 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:49.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.907+0000 7f7c3659c700 1 --2- 172.21.15.46:0/2107179548 >> v2:172.21.15.106:3300/0 conn(0x7f7c38068b00 0x7f7c38068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:49.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.908+0000 7f7c36d9d700 1 -- 172.21.15.46:0/2107179548 >> v2:172.21.15.106:3300/0 conn(0x7f7c38068b00 msgr2=0x7f7c38068ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:49.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.908+0000 7f7c36d9d700 1 --2- 172.21.15.46:0/2107179548 >> v2:172.21.15.106:3300/0 conn(0x7f7c38068b00 0x7f7c38068ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.908+0000 7f7c36d9d700 1 -- 172.21.15.46:0/2107179548 >> v1:172.21.15.106:6789/0 conn(0x7f7c3806a660 legacy=0x7f7c38159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:49.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.908+0000 7f7c36d9d700 1 -- 172.21.15.46:0/2107179548 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3815a680 con 0x7f7c3806add0 2024-01-26T23:45:49.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.908+0000 7f7c3559a700 1 -- 172.21.15.46:0/2107179548 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116057458 0 0) 0x7f7c380a9a80 con 0x7f7c3806a660 2024-01-26T23:45:49.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.908+0000 7f7c36d9d700 1 --2- 172.21.15.46:0/2107179548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c3806add0 0x7f7c3806b250 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f7c3806dc50 tx=0x7f7c30009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de39cb10cfa1da5a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:49.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.908+0000 7f7c3559a700 1 -- 172.21.15.46:0/2107179548 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3000e070 con 0x7f7c3806add0 2024-01-26T23:45:49.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.908+0000 7f7c3559a700 1 -- 172.21.15.46:0/2107179548 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3000ba20 con 0x7f7c3806add0 2024-01-26T23:45:49.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.908+0000 7f7c3559a700 1 -- 172.21.15.46:0/2107179548 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c300092e0 con 0x7f7c3806add0 2024-01-26T23:45:49.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.909+0000 7f7c3ce70700 1 -- 172.21.15.46:0/2107179548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c3806add0 msgr2=0x7f7c3806b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:49.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.909+0000 7f7c3ce70700 1 --2- 172.21.15.46:0/2107179548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c3806add0 0x7f7c3806b250 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f7c3806dc50 tx=0x7f7c30009840 comp rx=0 tx=0).stop 2024-01-26T23:45:49.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.909+0000 7f7c3ce70700 1 -- 172.21.15.46:0/2107179548 shutdown_connections 2024-01-26T23:45:49.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.909+0000 7f7c3ce70700 1 --2- 172.21.15.46:0/2107179548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c3806add0 0x7f7c3806b250 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.909+0000 7f7c3ce70700 1 --2- 172.21.15.46:0/2107179548 >> v2:172.21.15.106:3300/0 conn(0x7f7c38068b00 0x7f7c38068ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.909+0000 7f7c3ce70700 1 -- 172.21.15.46:0/2107179548 >> 172.21.15.46:0/2107179548 conn(0x7f7c380ca100 msgr2=0x7f7c3806e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:49.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.909+0000 7f7c3ce70700 1 -- 172.21.15.46:0/2107179548 shutdown_connections 2024-01-26T23:45:49.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.909+0000 7f7c3ce70700 1 -- 172.21.15.46:0/2107179548 wait complete. 2024-01-26T23:45:49.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.910+0000 7f7c3ce70700 1 Processor -- start 2024-01-26T23:45:49.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.910+0000 7f7c3ce70700 1 -- start start 2024-01-26T23:45:49.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.911+0000 7f7c3ce70700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c38068b00 0x7f7c381572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:49.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.911+0000 7f7c3ce70700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c3806a660 0x7f7c3815bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:49.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.911+0000 7f7c3ce70700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c3806add0 0x7f7c3815c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:49.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.911+0000 7f7c3ce70700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c380c68e0 con 0x7f7c38068b00 2024-01-26T23:45:49.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.911+0000 7f7c3ce70700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c380ae410 con 0x7f7c3806add0 2024-01-26T23:45:49.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.911+0000 7f7c3ce70700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c380ae6f0 con 0x7f7c3806a660 2024-01-26T23:45:49.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.911+0000 7f7c3659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c38068b00 0x7f7c381572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:49.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.911+0000 7f7c3659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c38068b00 0x7f7c381572a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36076/0 (socket says 172.21.15.46:36076) 2024-01-26T23:45:49.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.911+0000 7f7c3659c700 1 -- 172.21.15.46:0/2840648440 learned_addr learned my addr 172.21.15.46:0/2840648440 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:49.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.911+0000 7f7c36d9d700 1 --2- 172.21.15.46:0/2840648440 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c3806add0 0x7f7c3815c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:49.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.911+0000 7f7c35d9b700 1 --2- 172.21.15.46:0/2840648440 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c3806a660 0x7f7c3815bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:49.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.911+0000 7f7c3659c700 1 -- 172.21.15.46:0/2840648440 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c3806a660 msgr2=0x7f7c3815bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:49.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.911+0000 7f7c3659c700 1 --2- 172.21.15.46:0/2840648440 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c3806a660 0x7f7c3815bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.911+0000 7f7c3659c700 1 -- 172.21.15.46:0/2840648440 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c3806add0 msgr2=0x7f7c3815c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:49.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.911+0000 7f7c3659c700 1 --2- 172.21.15.46:0/2840648440 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c3806add0 0x7f7c3815c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:49.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.911+0000 7f7c3659c700 1 -- 172.21.15.46:0/2840648440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c30009ae0 con 0x7f7c38068b00 2024-01-26T23:45:49.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.911+0000 7f7c35d9b700 1 --2- 172.21.15.46:0/2840648440 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c3806a660 0x7f7c3815bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:49.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.912+0000 7f7c3659c700 1 --2- 172.21.15.46:0/2840648440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c38068b00 0x7f7c381572a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7c2800ea00 tx=0x7f7c28009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:49.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.914+0000 7f7c267fc700 1 -- 172.21.15.46:0/2840648440 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c280095e0 con 0x7f7c38068b00 2024-01-26T23:45:49.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.914+0000 7f7c3ce70700 1 -- 172.21.15.46:0/2840648440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c380ae9d0 con 0x7f7c38068b00 2024-01-26T23:45:49.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.914+0000 7f7c3ce70700 1 -- 172.21.15.46:0/2840648440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c381577e0 con 0x7f7c38068b00 2024-01-26T23:45:49.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.914+0000 7f7c267fc700 1 -- 172.21.15.46:0/2840648440 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2800ca60 con 0x7f7c38068b00 2024-01-26T23:45:49.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.914+0000 7f7c267fc700 1 -- 172.21.15.46:0/2840648440 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2800cbd0 con 0x7f7c38068b00 2024-01-26T23:45:49.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.915+0000 7f7c3ce70700 1 -- 172.21.15.46:0/2840648440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1c0052b0 con 0x7f7c38068b00 2024-01-26T23:45:49.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.916+0000 7f7c267fc700 1 -- 172.21.15.46:0/2840648440 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f7c2800cd30 con 0x7f7c38068b00 2024-01-26T23:45:49.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.917+0000 7f7c267fc700 1 --2- 172.21.15.46:0/2840648440 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7c18066ea0 0x7f7c18069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:49.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.917+0000 7f7c267fc700 1 -- 172.21.15.46:0/2840648440 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5565+0+0 (secure 0 0 0) 0x7f7c28084780 con 0x7f7c38068b00 2024-01-26T23:45:49.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.917+0000 7f7c35d9b700 1 --2- 172.21.15.46:0/2840648440 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7c18066ea0 0x7f7c18069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:49.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.917+0000 7f7c35d9b700 1 --2- 172.21.15.46:0/2840648440 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7c18066ea0 0x7f7c18069350 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c007fd0 tx=0x7f7c2c009040 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:49.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:49.923+0000 7f7c267fc700 1 -- 172.21.15.46:0/2840648440 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c28051b60 con 0x7f7c38068b00 2024-01-26T23:45:50.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.130+0000 7f7c3ce70700 1 -- 172.21.15.46:0/2840648440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7c1c005ae0 con 0x7f7c38068b00 2024-01-26T23:45:50.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.130+0000 7f7c267fc700 1 -- 172.21.15.46:0/2840648440 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7c280210a0 con 0x7f7c38068b00 2024-01-26T23:45:50.131 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:45:50.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.133+0000 7f7c3ce70700 1 -- 172.21.15.46:0/2840648440 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7c18066ea0 msgr2=0x7f7c18069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.133+0000 7f7c3ce70700 1 --2- 172.21.15.46:0/2840648440 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7c18066ea0 0x7f7c18069350 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c007fd0 tx=0x7f7c2c009040 comp rx=0 tx=0).stop 2024-01-26T23:45:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.133+0000 7f7c3ce70700 1 -- 172.21.15.46:0/2840648440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c38068b00 msgr2=0x7f7c381572a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.133+0000 7f7c3ce70700 1 --2- 172.21.15.46:0/2840648440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c38068b00 0x7f7c381572a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7c2800ea00 tx=0x7f7c28009120 comp rx=0 tx=0).stop 2024-01-26T23:45:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.133+0000 7f7c3ce70700 1 -- 172.21.15.46:0/2840648440 shutdown_connections 2024-01-26T23:45:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.133+0000 7f7c3ce70700 1 --2- 172.21.15.46:0/2840648440 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7c18066ea0 0x7f7c18069350 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.133+0000 7f7c3ce70700 1 --2- 172.21.15.46:0/2840648440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c38068b00 0x7f7c381572a0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.133+0000 7f7c3ce70700 1 --2- 172.21.15.46:0/2840648440 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c3806add0 0x7f7c3815c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.133+0000 7f7c3ce70700 1 --2- 172.21.15.46:0/2840648440 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c3806a660 0x7f7c3815bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.133+0000 7f7c3ce70700 1 -- 172.21.15.46:0/2840648440 >> 172.21.15.46:0/2840648440 conn(0x7f7c380ca100 msgr2=0x7f7c380c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.133+0000 7f7c3ce70700 1 -- 172.21.15.46:0/2840648440 shutdown_connections 2024-01-26T23:45:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.133+0000 7f7c3ce70700 1 -- 172.21.15.46:0/2840648440 wait complete. 2024-01-26T23:45:50.147 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-26T23:45:50.261 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:50.260+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:50.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.279+0000 7f1ea8f2b700 1 Processor -- start 2024-01-26T23:45:50.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.279+0000 7f1ea8f2b700 1 -- start start 2024-01-26T23:45:50.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.280+0000 7f1ea8f2b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea4157340 0x7f1ea415bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:50.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.280+0000 7f1ea8f2b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ea40ae520 0x7f1ea40b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:50.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.280+0000 7f1ea8f2b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea4158ae0 con 0x7f1ea40b3fb0 2024-01-26T23:45:50.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.280+0000 7f1ea8f2b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ea40b4430 con 0x7f1ea40ae520 2024-01-26T23:45:50.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.280+0000 7f1ea8f2b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea415b650 con 0x7f1ea4157340 2024-01-26T23:45:50.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.280+0000 7f1ea1d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ea40ae520 0x7f1ea40b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:50.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.280+0000 7f1ea2d9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1ea40b3fb0 0x7f1ea4157710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53196/0 (socket says 172.21.15.46:53196) 2024-01-26T23:45:50.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.280+0000 7f1ea1d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ea40ae520 0x7f1ea40b3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36082/0 (socket says 172.21.15.46:36082) 2024-01-26T23:45:50.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.280+0000 7f1ea2d9d700 1 -- 172.21.15.46:0/175577171 learned_addr learned my addr 172.21.15.46:0/175577171 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:50.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.280+0000 7f1ea259c700 1 --2- 172.21.15.46:0/175577171 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea4157340 0x7f1ea415bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:50.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.281+0000 7f1ea1d9b700 1 -- 172.21.15.46:0/175577171 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea4157340 msgr2=0x7f1ea415bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:50.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.281+0000 7f1ea1d9b700 1 --2- 172.21.15.46:0/175577171 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea4157340 0x7f1ea415bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.281+0000 7f1ea1d9b700 1 -- 172.21.15.46:0/175577171 >> v1:172.21.15.46:6789/0 conn(0x7f1ea40b3fb0 legacy=0x7f1ea4157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:50.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.281+0000 7f1ea1d9b700 1 -- 172.21.15.46:0/175577171 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea415c020 con 0x7f1ea40ae520 2024-01-26T23:45:50.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.281+0000 7f1ea159a700 1 -- 172.21.15.46:0/175577171 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415713740 0 0) 0x7f1ea4158ae0 con 0x7f1ea40b3fb0 2024-01-26T23:45:50.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.281+0000 7f1ea1d9b700 1 --2- 172.21.15.46:0/175577171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40ae520 0x7f1ea40b3a70 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f1e980098e0 tx=0x7f1e980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4024cd0b5fcf3d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:50.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.281+0000 7f1ea159a700 1 -- 172.21.15.46:0/175577171 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9800b640 con 0x7f1ea40ae520 2024-01-26T23:45:50.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.281+0000 7f1ea159a700 1 -- 172.21.15.46:0/175577171 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e9800b7a0 con 0x7f1ea40ae520 2024-01-26T23:45:50.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.281+0000 7f1ea159a700 1 -- 172.21.15.46:0/175577171 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9800bb40 con 0x7f1ea40ae520 2024-01-26T23:45:50.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.282+0000 7f1ea8f2b700 1 -- 172.21.15.46:0/175577171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40ae520 msgr2=0x7f1ea40b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:50.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.282+0000 7f1ea8f2b700 1 --2- 172.21.15.46:0/175577171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40ae520 0x7f1ea40b3a70 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f1e980098e0 tx=0x7f1e980091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.282+0000 7f1ea8f2b700 1 -- 172.21.15.46:0/175577171 shutdown_connections 2024-01-26T23:45:50.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.282+0000 7f1ea8f2b700 1 --2- 172.21.15.46:0/175577171 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea4157340 0x7f1ea415bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.282+0000 7f1ea8f2b700 1 --2- 172.21.15.46:0/175577171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40ae520 0x7f1ea40b3a70 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.282+0000 7f1ea8f2b700 1 -- 172.21.15.46:0/175577171 >> 172.21.15.46:0/175577171 conn(0x7f1ea40ce530 msgr2=0x7f1ea40ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:50.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.282+0000 7f1ea8f2b700 1 -- 172.21.15.46:0/175577171 shutdown_connections 2024-01-26T23:45:50.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.282+0000 7f1ea8f2b700 1 -- 172.21.15.46:0/175577171 wait complete. 2024-01-26T23:45:50.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.283+0000 7f1ea8f2b700 1 Processor -- start 2024-01-26T23:45:50.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.283+0000 7f1ea8f2b700 1 -- start start 2024-01-26T23:45:50.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.284+0000 7f1ea8f2b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea40ae520 0x7f1ea40b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:50.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.284+0000 7f1ea8f2b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b3fb0 0x7f1ea40c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:50.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.284+0000 7f1ea8f2b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ea4157340 0x7f1ea40c4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:50.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.284+0000 7f1ea8f2b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea40c7da0 con 0x7f1ea40b3fb0 2024-01-26T23:45:50.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.284+0000 7f1ea8f2b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea40c5040 con 0x7f1ea4157340 2024-01-26T23:45:50.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.284+0000 7f1ea8f2b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea40c7190 con 0x7f1ea40ae520 2024-01-26T23:45:50.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.284+0000 7f1ea1d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b3fb0 0x7f1ea40c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:50.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.284+0000 7f1ea1d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b3fb0 0x7f1ea40c6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36090/0 (socket says 172.21.15.46:36090) 2024-01-26T23:45:50.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.284+0000 7f1ea1d9b700 1 -- 172.21.15.46:0/3613093675 learned_addr learned my addr 172.21.15.46:0/3613093675 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:50.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.284+0000 7f1ea259c700 1 --2- 172.21.15.46:0/3613093675 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea40ae520 0x7f1ea40b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:50.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.284+0000 7f1ea2d9d700 1 --2- 172.21.15.46:0/3613093675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ea4157340 0x7f1ea40c4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:50.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.285+0000 7f1ea1d9b700 1 -- 172.21.15.46:0/3613093675 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea40ae520 msgr2=0x7f1ea40b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:50.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.285+0000 7f1ea1d9b700 1 --2- 172.21.15.46:0/3613093675 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea40ae520 0x7f1ea40b1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.285+0000 7f1ea1d9b700 1 -- 172.21.15.46:0/3613093675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ea4157340 msgr2=0x7f1ea40c4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:50.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.285+0000 7f1ea1d9b700 1 --2- 172.21.15.46:0/3613093675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ea4157340 0x7f1ea40c4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.285+0000 7f1ea1d9b700 1 -- 172.21.15.46:0/3613093675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e98009420 con 0x7f1ea40b3fb0 2024-01-26T23:45:50.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.285+0000 7f1ea259c700 1 --2- 172.21.15.46:0/3613093675 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea40ae520 0x7f1ea40b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:50.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.285+0000 7f1ea1d9b700 1 --2- 172.21.15.46:0/3613093675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b3fb0 0x7f1ea40c6a90 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f1e98009f80 tx=0x7f1e98017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:50.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.285+0000 7f1e927fc700 1 -- 172.21.15.46:0/3613093675 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9800bb20 con 0x7f1ea40b3fb0 2024-01-26T23:45:50.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.285+0000 7f1e927fc700 1 -- 172.21.15.46:0/3613093675 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e98011420 con 0x7f1ea40b3fb0 2024-01-26T23:45:50.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.285+0000 7f1e927fc700 1 -- 172.21.15.46:0/3613093675 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e98011820 con 0x7f1ea40b3fb0 2024-01-26T23:45:50.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.285+0000 7f1ea8f2b700 1 -- 172.21.15.46:0/3613093675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea40c7470 con 0x7f1ea40b3fb0 2024-01-26T23:45:50.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.285+0000 7f1ea8f2b700 1 -- 172.21.15.46:0/3613093675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea40c7a30 con 0x7f1ea40b3fb0 2024-01-26T23:45:50.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.286+0000 7f1ea8f2b700 1 -- 172.21.15.46:0/3613093675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea40c7ee0 con 0x7f1ea40b3fb0 2024-01-26T23:45:50.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.286+0000 7f1e927fc700 1 -- 172.21.15.46:0/3613093675 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f1e98011c50 con 0x7f1ea40b3fb0 2024-01-26T23:45:50.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.287+0000 7f1e927fc700 1 --2- 172.21.15.46:0/3613093675 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1e84066ea0 0x7f1e84069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:50.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.287+0000 7f1e927fc700 1 -- 172.21.15.46:0/3613093675 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5565+0+0 (secure 0 0 0) 0x7f1e98085db0 con 0x7f1ea40b3fb0 2024-01-26T23:45:50.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.287+0000 7f1ea259c700 1 --2- 172.21.15.46:0/3613093675 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1e84066ea0 0x7f1e84069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:50.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.288+0000 7f1ea259c700 1 --2- 172.21.15.46:0/3613093675 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1e84066ea0 0x7f1e84069350 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009d80 tx=0x7f1e94008040 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:50.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.291+0000 7f1e927fc700 1 -- 172.21.15.46:0/3613093675 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1e98053190 con 0x7f1ea40b3fb0 2024-01-26T23:45:50.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.507+0000 7f1ea8f2b700 1 -- 172.21.15.46:0/3613093675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1ea40c2b10 con 0x7f1ea40b3fb0 2024-01-26T23:45:50.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.508+0000 7f1e927fc700 1 -- 172.21.15.46:0/3613093675 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v94) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1e980567b0 con 0x7f1ea40b3fb0 2024-01-26T23:45:50.509 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:50.509 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-26T23:45:50.509 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:45:50.509 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-01-26T23:45:50.509 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-01-26T23:45:50.509 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:45:50.509 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-26T23:45:50.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.510+0000 7f1ea8f2b700 1 -- 172.21.15.46:0/3613093675 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1e84066ea0 msgr2=0x7f1e84069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:50.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.510+0000 7f1ea8f2b700 1 --2- 172.21.15.46:0/3613093675 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1e84066ea0 0x7f1e84069350 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009d80 tx=0x7f1e94008040 comp rx=0 tx=0).stop 2024-01-26T23:45:50.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.510+0000 7f1ea8f2b700 1 -- 172.21.15.46:0/3613093675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b3fb0 msgr2=0x7f1ea40c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:50.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.510+0000 7f1ea8f2b700 1 --2- 172.21.15.46:0/3613093675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b3fb0 0x7f1ea40c6a90 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f1e98009f80 tx=0x7f1e98017000 comp rx=0 tx=0).stop 2024-01-26T23:45:50.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.510+0000 7f1ea8f2b700 1 -- 172.21.15.46:0/3613093675 shutdown_connections 2024-01-26T23:45:50.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.510+0000 7f1ea8f2b700 1 --2- 172.21.15.46:0/3613093675 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f1e84066ea0 0x7f1e84069350 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.510+0000 7f1ea8f2b700 1 --2- 172.21.15.46:0/3613093675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ea4157340 0x7f1ea40c4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.510+0000 7f1ea8f2b700 1 --2- 172.21.15.46:0/3613093675 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea40ae520 0x7f1ea40b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.510+0000 7f1ea8f2b700 1 --2- 172.21.15.46:0/3613093675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b3fb0 0x7f1ea40c6a90 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.510+0000 7f1ea8f2b700 1 -- 172.21.15.46:0/3613093675 >> 172.21.15.46:0/3613093675 conn(0x7f1ea40ce530 msgr2=0x7f1ea40afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:50.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.510+0000 7f1ea8f2b700 1 -- 172.21.15.46:0/3613093675 shutdown_connections 2024-01-26T23:45:50.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.510+0000 7f1ea8f2b700 1 -- 172.21.15.46:0/3613093675 wait complete. 2024-01-26T23:45:50.524 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-26T23:45:50.524 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-26T23:45:50.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.662+0000 7f6f9d6b1700 1 Processor -- start 2024-01-26T23:45:50.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.662+0000 7f6f9d6b1700 1 -- start start 2024-01-26T23:45:50.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.663+0000 7f6f9d6b1700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6f9806a330 0x7f6f98159960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:50.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.663+0000 7f6f9d6b1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6f980b12e0 0x7f6f9806bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:50.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.663+0000 7f6f9d6b1700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f98068c50 con 0x7f6f98159590 2024-01-26T23:45:50.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.663+0000 7f6f9d6b1700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f980c98c0 con 0x7f6f980b12e0 2024-01-26T23:45:50.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.663+0000 7f6f9d6b1700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f9815dc90 con 0x7f6f9806a330 2024-01-26T23:45:50.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.663+0000 7f6f977fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6f980b12e0 0x7f6f9806bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:50.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.663+0000 7f6f977fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6f980b12e0 0x7f6f9806bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33268/0 (socket says 172.21.15.46:33268) 2024-01-26T23:45:50.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.663+0000 7f6f96ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6f98159590 0x7f6f9815d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52104/0 (socket says 172.21.15.46:52104) 2024-01-26T23:45:50.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.663+0000 7f6f977fe700 1 -- 172.21.15.46:0/305156075 learned_addr learned my addr 172.21.15.46:0/305156075 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:50.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.663+0000 7f6f967fc700 1 --2- 172.21.15.46:0/305156075 >> v2:172.21.15.106:3300/0 conn(0x7f6f9806a330 0x7f6f98159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:50.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.663+0000 7f6f977fe700 1 -- 172.21.15.46:0/305156075 >> v2:172.21.15.106:3300/0 conn(0x7f6f9806a330 msgr2=0x7f6f98159960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:50.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.663+0000 7f6f977fe700 1 --2- 172.21.15.46:0/305156075 >> v2:172.21.15.106:3300/0 conn(0x7f6f9806a330 0x7f6f98159960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.663+0000 7f6f977fe700 1 -- 172.21.15.46:0/305156075 >> v1:172.21.15.46:6789/0 conn(0x7f6f98159590 legacy=0x7f6f9815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:50.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.663+0000 7f6f977fe700 1 -- 172.21.15.46:0/305156075 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9815df70 con 0x7f6f980b12e0 2024-01-26T23:45:50.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.663+0000 7f6f95ffb700 1 -- 172.21.15.46:0/305156075 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298274851 0 0) 0x7f6f98068c50 con 0x7f6f98159590 2024-01-26T23:45:50.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.664+0000 7f6f977fe700 1 --2- 172.21.15.46:0/305156075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f980b12e0 0x7f6f9806bcd0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f6f9806cfd0 tx=0x7f6f90009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fced89bfaa2ae714 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:50.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.664+0000 7f6f95ffb700 1 -- 172.21.15.46:0/305156075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9000e070 con 0x7f6f980b12e0 2024-01-26T23:45:50.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.664+0000 7f6f95ffb700 1 -- 172.21.15.46:0/305156075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9000ba20 con 0x7f6f980b12e0 2024-01-26T23:45:50.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.664+0000 7f6f95ffb700 1 -- 172.21.15.46:0/305156075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f900092e0 con 0x7f6f980b12e0 2024-01-26T23:45:50.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.664+0000 7f6f9d6b1700 1 -- 172.21.15.46:0/305156075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f980b12e0 msgr2=0x7f6f9806bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:50.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.664+0000 7f6f9d6b1700 1 --2- 172.21.15.46:0/305156075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f980b12e0 0x7f6f9806bcd0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f6f9806cfd0 tx=0x7f6f90009840 comp rx=0 tx=0).stop 2024-01-26T23:45:50.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.664+0000 7f6f9d6b1700 1 -- 172.21.15.46:0/305156075 shutdown_connections 2024-01-26T23:45:50.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.665+0000 7f6f9d6b1700 1 --2- 172.21.15.46:0/305156075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f980b12e0 0x7f6f9806bcd0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.665+0000 7f6f9d6b1700 1 --2- 172.21.15.46:0/305156075 >> v2:172.21.15.106:3300/0 conn(0x7f6f9806a330 0x7f6f98159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.665+0000 7f6f9d6b1700 1 -- 172.21.15.46:0/305156075 >> 172.21.15.46:0/305156075 conn(0x7f6f980ce530 msgr2=0x7f6f980ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:50.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.665+0000 7f6f9d6b1700 1 -- 172.21.15.46:0/305156075 shutdown_connections 2024-01-26T23:45:50.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.665+0000 7f6f9d6b1700 1 -- 172.21.15.46:0/305156075 wait complete. 2024-01-26T23:45:50.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.666+0000 7f6f9d6b1700 1 Processor -- start 2024-01-26T23:45:50.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.666+0000 7f6f9d6b1700 1 -- start start 2024-01-26T23:45:50.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.667+0000 7f6f9d6b1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6f9806a330 0x7f6f98159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:50.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.667+0000 7f6f9d6b1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f980b12e0 0x7f6f980b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:50.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.667+0000 7f6f9d6b1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f98159590 0x7f6f980c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:50.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.667+0000 7f6f9d6b1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980c2ac0 con 0x7f6f98159590 2024-01-26T23:45:50.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.667+0000 7f6f9d6b1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980c2da0 con 0x7f6f9806a330 2024-01-26T23:45:50.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.667+0000 7f6f9d6b1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980c3080 con 0x7f6f980b12e0 2024-01-26T23:45:50.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.667+0000 7f6f977fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f98159590 0x7f6f980c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:50.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.667+0000 7f6f96ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6f9806a330 0x7f6f98159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:50.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.667+0000 7f6f977fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f98159590 0x7f6f980c3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33282/0 (socket says 172.21.15.46:33282) 2024-01-26T23:45:50.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.667+0000 7f6f977fe700 1 -- 172.21.15.46:0/1748252494 learned_addr learned my addr 172.21.15.46:0/1748252494 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:50.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.667+0000 7f6f967fc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f980b12e0 0x7f6f980b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:50.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.668+0000 7f6f96ffd700 1 -- 172.21.15.46:0/1748252494 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f980b12e0 msgr2=0x7f6f980b0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:50.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.668+0000 7f6f96ffd700 1 --2- 172.21.15.46:0/1748252494 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f980b12e0 0x7f6f980b0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.668+0000 7f6f96ffd700 1 -- 172.21.15.46:0/1748252494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f98159590 msgr2=0x7f6f980c3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:50.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.668+0000 7f6f96ffd700 1 --2- 172.21.15.46:0/1748252494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f98159590 0x7f6f980c3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.668+0000 7f6f96ffd700 1 -- 172.21.15.46:0/1748252494 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f90009ae0 con 0x7f6f9806a330 2024-01-26T23:45:50.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.668+0000 7f6f977fe700 1 --2- 172.21.15.46:0/1748252494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f98159590 0x7f6f980c3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:50.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.668+0000 7f6f96ffd700 1 --2- 172.21.15.46:0/1748252494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6f9806a330 0x7f6f98159f30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6f8800c5b0 tx=0x7f6f88007d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:50.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.668+0000 7f6f7effd700 1 -- 172.21.15.46:0/1748252494 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f88005b40 con 0x7f6f9806a330 2024-01-26T23:45:50.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.668+0000 7f6f7effd700 1 -- 172.21.15.46:0/1748252494 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f88005ca0 con 0x7f6f9806a330 2024-01-26T23:45:50.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.668+0000 7f6f7effd700 1 -- 172.21.15.46:0/1748252494 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f88007320 con 0x7f6f9806a330 2024-01-26T23:45:50.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.668+0000 7f6f9d6b1700 1 -- 172.21.15.46:0/1748252494 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f980c3360 con 0x7f6f9806a330 2024-01-26T23:45:50.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.668+0000 7f6f9d6b1700 1 -- 172.21.15.46:0/1748252494 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f980b05b0 con 0x7f6f9806a330 2024-01-26T23:45:50.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.669+0000 7f6f9d6b1700 1 -- 172.21.15.46:0/1748252494 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9804c710 con 0x7f6f9806a330 2024-01-26T23:45:50.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.669+0000 7f6f7effd700 1 -- 172.21.15.46:0/1748252494 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f6f8801b3b0 con 0x7f6f9806a330 2024-01-26T23:45:50.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.670+0000 7f6f7effd700 1 --2- 172.21.15.46:0/1748252494 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6f84066ea0 0x7f6f84069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:50.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.670+0000 7f6f7effd700 1 -- 172.21.15.46:0/1748252494 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5565+0+0 (secure 0 0 0) 0x7f6f88083780 con 0x7f6f9806a330 2024-01-26T23:45:50.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.670+0000 7f6f967fc700 1 --2- 172.21.15.46:0/1748252494 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6f84066ea0 0x7f6f84069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:50.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.671+0000 7f6f967fc700 1 --2- 172.21.15.46:0/1748252494 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6f84066ea0 0x7f6f84069350 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6f980ada00 tx=0x7f6f8c0091a0 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:50.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.675+0000 7f6f7effd700 1 -- 172.21.15.46:0/1748252494 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6f88050b60 con 0x7f6f9806a330 2024-01-26T23:45:50.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.887+0000 7f6f9d6b1700 1 -- 172.21.15.46:0/1748252494 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f6f980c2ee0 con 0x7f6f9806a330 2024-01-26T23:45:50.958 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:50.956+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:50.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.981+0000 7f6f7effd700 1 -- 172.21.15.46:0/1748252494 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v95) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6f980c2ee0 con 0x7f6f9806a330 2024-01-26T23:45:50.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.984+0000 7f6f9d6b1700 1 -- 172.21.15.46:0/1748252494 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6f84066ea0 msgr2=0x7f6f84069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:50.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.984+0000 7f6f9d6b1700 1 --2- 172.21.15.46:0/1748252494 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6f84066ea0 0x7f6f84069350 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6f980ada00 tx=0x7f6f8c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.984+0000 7f6f9d6b1700 1 -- 172.21.15.46:0/1748252494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6f9806a330 msgr2=0x7f6f98159f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:50.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.984+0000 7f6f9d6b1700 1 --2- 172.21.15.46:0/1748252494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6f9806a330 0x7f6f98159f30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6f8800c5b0 tx=0x7f6f88007d50 comp rx=0 tx=0).stop 2024-01-26T23:45:50.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.984+0000 7f6f9d6b1700 1 -- 172.21.15.46:0/1748252494 shutdown_connections 2024-01-26T23:45:50.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.984+0000 7f6f9d6b1700 1 --2- 172.21.15.46:0/1748252494 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6f84066ea0 0x7f6f84069350 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.984+0000 7f6f9d6b1700 1 --2- 172.21.15.46:0/1748252494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f98159590 0x7f6f980c3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.984+0000 7f6f9d6b1700 1 --2- 172.21.15.46:0/1748252494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6f9806a330 0x7f6f98159f30 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.984+0000 7f6f9d6b1700 1 --2- 172.21.15.46:0/1748252494 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f980b12e0 0x7f6f980b0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:50.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.984+0000 7f6f9d6b1700 1 -- 172.21.15.46:0/1748252494 >> 172.21.15.46:0/1748252494 conn(0x7f6f980ce530 msgr2=0x7f6f9806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:50.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.985+0000 7f6f9d6b1700 1 -- 172.21.15.46:0/1748252494 shutdown_connections 2024-01-26T23:45:50.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:50.985+0000 7f6f9d6b1700 1 -- 172.21.15.46:0/1748252494 wait complete. 2024-01-26T23:45:50.986 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-01-26T23:45:51.000 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-26T23:45:51.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.144+0000 7fc08b429700 1 Processor -- start 2024-01-26T23:45:51.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.145+0000 7fc08b429700 1 -- start start 2024-01-26T23:45:51.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.145+0000 7fc08b429700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc0840ab6e0 0x7fc08406a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:51.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.145+0000 7fc08b429700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc084068940 0x7fc084068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:51.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.145+0000 7fc08b429700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0840c7540 con 0x7fc0840ab310 2024-01-26T23:45:51.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.145+0000 7fc08b429700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc084069300 con 0x7fc0840ab6e0 2024-01-26T23:45:51.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.145+0000 7fc08b429700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0840c7820 con 0x7fc084068940 2024-01-26T23:45:51.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.146+0000 7fc0889c4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc0840ab6e0 0x7fc08406a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:51.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.146+0000 7fc0891c5700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc0840ab310 0x7fc0840a9850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52118/0 (socket says 172.21.15.46:52118) 2024-01-26T23:45:51.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.146+0000 7fc0891c5700 1 -- 172.21.15.46:0/3534048405 learned_addr learned my addr 172.21.15.46:0/3534048405 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:51.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.146+0000 7fc0899c6700 1 --2- 172.21.15.46:0/3534048405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc084068940 0x7fc084068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:51.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.146+0000 7fc0889c4700 1 -- 172.21.15.46:0/3534048405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc084068940 msgr2=0x7fc084068dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:51.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.146+0000 7fc0889c4700 1 --2- 172.21.15.46:0/3534048405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc084068940 0x7fc084068dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:51.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.146+0000 7fc0889c4700 1 -- 172.21.15.46:0/3534048405 >> v1:172.21.15.46:6789/0 conn(0x7fc0840ab310 legacy=0x7fc0840a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:51.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.146+0000 7fc0889c4700 1 -- 172.21.15.46:0/3534048405 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0840c7b00 con 0x7fc0840ab6e0 2024-01-26T23:45:51.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.146+0000 7fc07bfff700 1 -- 172.21.15.46:0/3534048405 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170035250 0 0) 0x7fc0840c7540 con 0x7fc0840ab310 2024-01-26T23:45:51.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.146+0000 7fc0889c4700 1 --2- 172.21.15.46:0/3534048405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0840ab6e0 0x7fc08406a3c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc0740098e0 tx=0x7fc0740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17a4051bc44057a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:51.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.146+0000 7fc07bfff700 1 -- 172.21.15.46:0/3534048405 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07400b640 con 0x7fc0840ab6e0 2024-01-26T23:45:51.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.146+0000 7fc07bfff700 1 -- 172.21.15.46:0/3534048405 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc07400b7a0 con 0x7fc0840ab6e0 2024-01-26T23:45:51.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.146+0000 7fc07bfff700 1 -- 172.21.15.46:0/3534048405 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07400bb40 con 0x7fc0840ab6e0 2024-01-26T23:45:51.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.147+0000 7fc08b429700 1 -- 172.21.15.46:0/3534048405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0840ab6e0 msgr2=0x7fc08406a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:51.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.147+0000 7fc08b429700 1 --2- 172.21.15.46:0/3534048405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0840ab6e0 0x7fc08406a3c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc0740098e0 tx=0x7fc0740091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:51.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.147+0000 7fc08b429700 1 -- 172.21.15.46:0/3534048405 shutdown_connections 2024-01-26T23:45:51.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.147+0000 7fc08b429700 1 --2- 172.21.15.46:0/3534048405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc084068940 0x7fc084068dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:51.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.147+0000 7fc08b429700 1 --2- 172.21.15.46:0/3534048405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0840ab6e0 0x7fc08406a3c0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:51.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.147+0000 7fc08b429700 1 -- 172.21.15.46:0/3534048405 >> 172.21.15.46:0/3534048405 conn(0x7fc0840ca100 msgr2=0x7fc08406e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:51.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.147+0000 7fc08b429700 1 -- 172.21.15.46:0/3534048405 shutdown_connections 2024-01-26T23:45:51.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.147+0000 7fc08b429700 1 -- 172.21.15.46:0/3534048405 wait complete. 2024-01-26T23:45:51.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.148+0000 7fc08b429700 1 Processor -- start 2024-01-26T23:45:51.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.148+0000 7fc08b429700 1 -- start start 2024-01-26T23:45:51.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.149+0000 7fc08b429700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc084068940 0x7fc084158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:51.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.149+0000 7fc08b429700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc0840ab310 0x7fc0840acab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:51.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.150+0000 7fc08b429700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0840ab6e0 0x7fc0840b1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:51.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.150+0000 7fc08b429700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0840abdc0 con 0x7fc084068940 2024-01-26T23:45:51.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.150+0000 7fc08b429700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0840ac620 con 0x7fc0840ab6e0 2024-01-26T23:45:51.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.150+0000 7fc08b429700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc084159530 con 0x7fc0840ab310 2024-01-26T23:45:51.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.150+0000 7fc0891c5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc084068940 0x7fc084158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:51.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.150+0000 7fc0889c4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc0840ab310 0x7fc0840acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:51.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.150+0000 7fc0891c5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc084068940 0x7fc084158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33308/0 (socket says 172.21.15.46:33308) 2024-01-26T23:45:51.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.150+0000 7fc0891c5700 1 -- 172.21.15.46:0/1434574264 learned_addr learned my addr 172.21.15.46:0/1434574264 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.150+0000 7fc0899c6700 1 --2- 172.21.15.46:0/1434574264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0840ab6e0 0x7fc0840b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.150+0000 7fc0899c6700 1 -- 172.21.15.46:0/1434574264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc0840ab310 msgr2=0x7fc0840acab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.150+0000 7fc0899c6700 1 --2- 172.21.15.46:0/1434574264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc0840ab310 0x7fc0840acab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.150+0000 7fc0899c6700 1 -- 172.21.15.46:0/1434574264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc084068940 msgr2=0x7fc084158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.150+0000 7fc0899c6700 1 --2- 172.21.15.46:0/1434574264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc084068940 0x7fc084158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.150+0000 7fc0899c6700 1 -- 172.21.15.46:0/1434574264 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc074009420 con 0x7fc0840ab6e0 2024-01-26T23:45:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.150+0000 7fc0891c5700 1 --2- 172.21.15.46:0/1434574264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc084068940 0x7fc084158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.150+0000 7fc0889c4700 1 --2- 172.21.15.46:0/1434574264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc0840ab310 0x7fc0840acab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.150+0000 7fc0899c6700 1 --2- 172.21.15.46:0/1434574264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0840ab6e0 0x7fc0840b1950 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc07c00ecc0 tx=0x7fc07c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.151+0000 7fc0797fa700 1 -- 172.21.15.46:0/1434574264 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07c00c8c0 con 0x7fc0840ab6e0 2024-01-26T23:45:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.151+0000 7fc0797fa700 1 -- 172.21.15.46:0/1434574264 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc07c00f040 con 0x7fc0840ab6e0 2024-01-26T23:45:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.151+0000 7fc0797fa700 1 -- 172.21.15.46:0/1434574264 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07c00cc90 con 0x7fc0840ab6e0 2024-01-26T23:45:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.151+0000 7fc08b429700 1 -- 172.21.15.46:0/1434574264 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08416d910 con 0x7fc0840ab6e0 2024-01-26T23:45:51.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.151+0000 7fc08b429700 1 -- 172.21.15.46:0/1434574264 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0840ac2b0 con 0x7fc0840ab6e0 2024-01-26T23:45:51.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.152+0000 7fc08b429700 1 -- 172.21.15.46:0/1434574264 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0840c6e10 con 0x7fc0840ab6e0 2024-01-26T23:45:51.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.152+0000 7fc0797fa700 1 -- 172.21.15.46:0/1434574264 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fc07c014690 con 0x7fc0840ab6e0 2024-01-26T23:45:51.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.153+0000 7fc0797fa700 1 --2- 172.21.15.46:0/1434574264 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc068066ea0 0x7fc068069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:51.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.153+0000 7fc0797fa700 1 -- 172.21.15.46:0/1434574264 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5935+0+0 (secure 0 0 0) 0x7fc07c084ec0 con 0x7fc0840ab6e0 2024-01-26T23:45:51.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.153+0000 7fc0891c5700 1 --2- 172.21.15.46:0/1434574264 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc068066ea0 0x7fc068069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:51.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.153+0000 7fc0891c5700 1 --2- 172.21.15.46:0/1434574264 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc068066ea0 0x7fc068069350 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc080001360 tx=0x7fc080009040 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:51.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.157+0000 7fc0797fa700 1 -- 172.21.15.46:0/1434574264 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc07c052130 con 0x7fc0840ab6e0 2024-01-26T23:45:51.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.364+0000 7fc08b429700 1 -- 172.21.15.46:0/1434574264 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc08404c710 con 0x7fc0840ab6e0 2024-01-26T23:45:51.652 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:51.650+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:51.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.983+0000 7fc0797fa700 1 -- 172.21.15.46:0/1434574264 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v96) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc07c055750 con 0x7fc0840ab6e0 2024-01-26T23:45:51.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.989+0000 7fc08b429700 1 -- 172.21.15.46:0/1434574264 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc068066ea0 msgr2=0x7fc068069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:51.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.989+0000 7fc08b429700 1 --2- 172.21.15.46:0/1434574264 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc068066ea0 0x7fc068069350 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc080001360 tx=0x7fc080009040 comp rx=0 tx=0).stop 2024-01-26T23:45:51.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.989+0000 7fc08b429700 1 -- 172.21.15.46:0/1434574264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0840ab6e0 msgr2=0x7fc0840b1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:51.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.989+0000 7fc08b429700 1 --2- 172.21.15.46:0/1434574264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0840ab6e0 0x7fc0840b1950 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc07c00ecc0 tx=0x7fc07c00b040 comp rx=0 tx=0).stop 2024-01-26T23:45:51.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.989+0000 7fc08b429700 1 -- 172.21.15.46:0/1434574264 shutdown_connections 2024-01-26T23:45:51.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.989+0000 7fc08b429700 1 --2- 172.21.15.46:0/1434574264 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc068066ea0 0x7fc068069350 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:51.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.989+0000 7fc08b429700 1 --2- 172.21.15.46:0/1434574264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc084068940 0x7fc084158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:51.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.989+0000 7fc08b429700 1 --2- 172.21.15.46:0/1434574264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0840ab6e0 0x7fc0840b1950 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:51.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.989+0000 7fc08b429700 1 --2- 172.21.15.46:0/1434574264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc0840ab310 0x7fc0840acab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:51.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.989+0000 7fc08b429700 1 -- 172.21.15.46:0/1434574264 >> 172.21.15.46:0/1434574264 conn(0x7fc0840ca100 msgr2=0x7fc084067930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:51.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.991+0000 7fc08b429700 1 -- 172.21.15.46:0/1434574264 shutdown_connections 2024-01-26T23:45:51.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:51.991+0000 7fc08b429700 1 -- 172.21.15.46:0/1434574264 wait complete. 2024-01-26T23:45:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-01-26T23:45:52.016 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-26T23:45:52.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.158+0000 7ff5774a2700 1 Processor -- start 2024-01-26T23:45:52.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.158+0000 7ff5774a2700 1 -- start start 2024-01-26T23:45:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.159+0000 7ff5774a2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff570067af0 0x7ff570067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.159+0000 7ff5774a2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff57006ebe0 0x7ff5700ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.159+0000 7ff5774a2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff5701597e0 0x7ff570159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.159+0000 7ff5774a2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff57006fdb0 con 0x7ff5701597e0 2024-01-26T23:45:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.159+0000 7ff5774a2700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5700ade10 con 0x7ff570067af0 2024-01-26T23:45:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.159+0000 7ff5774a2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff57006f1a0 con 0x7ff57006ebe0 2024-01-26T23:45:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.159+0000 7ff575a3f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff5701597e0 0x7ff570159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.159+0000 7ff575a3f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff5701597e0 0x7ff570159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33324/0 (socket says 172.21.15.46:33324) 2024-01-26T23:45:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.159+0000 7ff575a3f700 1 -- 172.21.15.46:0/1251856859 learned_addr learned my addr 172.21.15.46:0/1251856859 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.159+0000 7ff57523e700 1 --2- 172.21.15.46:0/1251856859 >> v2:172.21.15.106:3300/0 conn(0x7ff570067af0 0x7ff570067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.159+0000 7ff574a3d700 1 --2- 172.21.15.46:0/1251856859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff57006ebe0 0x7ff5700ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.159+0000 7ff575a3f700 1 -- 172.21.15.46:0/1251856859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff57006ebe0 msgr2=0x7ff5700ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.159+0000 7ff575a3f700 1 --2- 172.21.15.46:0/1251856859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff57006ebe0 0x7ff5700ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.159+0000 7ff575a3f700 1 -- 172.21.15.46:0/1251856859 >> v2:172.21.15.106:3300/0 conn(0x7ff570067af0 msgr2=0x7ff570067ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.159+0000 7ff575a3f700 1 --2- 172.21.15.46:0/1251856859 >> v2:172.21.15.106:3300/0 conn(0x7ff570067af0 0x7ff570067ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.159+0000 7ff575a3f700 1 -- 172.21.15.46:0/1251856859 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57006f480 con 0x7ff5701597e0 2024-01-26T23:45:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.160+0000 7ff575a3f700 1 --2- 172.21.15.46:0/1251856859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5701597e0 0x7ff570159bf0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ff570064050 tx=0x7ff568009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f25b0f8aa74a285 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.160+0000 7ff567fff700 1 -- 172.21.15.46:0/1251856859 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56800e070 con 0x7ff5701597e0 2024-01-26T23:45:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.160+0000 7ff567fff700 1 -- 172.21.15.46:0/1251856859 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56800ba50 con 0x7ff5701597e0 2024-01-26T23:45:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.160+0000 7ff567fff700 1 -- 172.21.15.46:0/1251856859 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5680092e0 con 0x7ff5701597e0 2024-01-26T23:45:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.160+0000 7ff5774a2700 1 -- 172.21.15.46:0/1251856859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5701597e0 msgr2=0x7ff570159bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.160+0000 7ff5774a2700 1 --2- 172.21.15.46:0/1251856859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5701597e0 0x7ff570159bf0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ff570064050 tx=0x7ff568009840 comp rx=0 tx=0).stop 2024-01-26T23:45:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.160+0000 7ff5774a2700 1 -- 172.21.15.46:0/1251856859 shutdown_connections 2024-01-26T23:45:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.160+0000 7ff5774a2700 1 --2- 172.21.15.46:0/1251856859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5701597e0 0x7ff570159bf0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.160+0000 7ff5774a2700 1 --2- 172.21.15.46:0/1251856859 >> v2:172.21.15.106:3300/0 conn(0x7ff570067af0 0x7ff570067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.161+0000 7ff5774a2700 1 --2- 172.21.15.46:0/1251856859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff57006ebe0 0x7ff5700ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.161+0000 7ff5774a2700 1 -- 172.21.15.46:0/1251856859 >> 172.21.15.46:0/1251856859 conn(0x7ff5700cdf80 msgr2=0x7ff57006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.161+0000 7ff5774a2700 1 -- 172.21.15.46:0/1251856859 shutdown_connections 2024-01-26T23:45:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.161+0000 7ff5774a2700 1 -- 172.21.15.46:0/1251856859 wait complete. 2024-01-26T23:45:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.162+0000 7ff5774a2700 1 Processor -- start 2024-01-26T23:45:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.162+0000 7ff5774a2700 1 -- start start 2024-01-26T23:45:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.163+0000 7ff5774a2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff570067af0 0x7ff57015cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.163+0000 7ff5774a2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff57006ebe0 0x7ff5701512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.163+0000 7ff5774a2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5701597e0 0x7ff570154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.163+0000 7ff5774a2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57016a9b0 con 0x7ff570067af0 2024-01-26T23:45:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.163+0000 7ff5774a2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff570169da0 con 0x7ff57006ebe0 2024-01-26T23:45:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.163+0000 7ff5774a2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff57016a080 con 0x7ff5701597e0 2024-01-26T23:45:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.163+0000 7ff57523e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff570067af0 0x7ff57015cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.163+0000 7ff57523e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff570067af0 0x7ff57015cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33340/0 (socket says 172.21.15.46:33340) 2024-01-26T23:45:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.163+0000 7ff57523e700 1 -- 172.21.15.46:0/2190307296 learned_addr learned my addr 172.21.15.46:0/2190307296 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.163+0000 7ff574a3d700 1 --2- 172.21.15.46:0/2190307296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff57006ebe0 0x7ff5701512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.163+0000 7ff575a3f700 1 --2- 172.21.15.46:0/2190307296 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5701597e0 0x7ff570154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.164+0000 7ff57523e700 1 -- 172.21.15.46:0/2190307296 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5701597e0 msgr2=0x7ff570154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.164+0000 7ff57523e700 1 --2- 172.21.15.46:0/2190307296 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5701597e0 0x7ff570154730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.164+0000 7ff57523e700 1 -- 172.21.15.46:0/2190307296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff57006ebe0 msgr2=0x7ff5701512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.164+0000 7ff57523e700 1 --2- 172.21.15.46:0/2190307296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff57006ebe0 0x7ff5701512b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.164+0000 7ff57523e700 1 -- 172.21.15.46:0/2190307296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff568009ae0 con 0x7ff570067af0 2024-01-26T23:45:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.164+0000 7ff57523e700 1 --2- 172.21.15.46:0/2190307296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff570067af0 0x7ff57015cdc0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7ff56c00d860 tx=0x7ff56c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.164+0000 7ff5657fa700 1 -- 172.21.15.46:0/2190307296 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c0125f0 con 0x7ff570067af0 2024-01-26T23:45:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.164+0000 7ff5657fa700 1 -- 172.21.15.46:0/2190307296 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56c012750 con 0x7ff570067af0 2024-01-26T23:45:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.164+0000 7ff5657fa700 1 -- 172.21.15.46:0/2190307296 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c012b10 con 0x7ff570067af0 2024-01-26T23:45:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.164+0000 7ff5774a2700 1 -- 172.21.15.46:0/2190307296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57016a360 con 0x7ff570067af0 2024-01-26T23:45:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.164+0000 7ff5774a2700 1 -- 172.21.15.46:0/2190307296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57015d300 con 0x7ff570067af0 2024-01-26T23:45:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.165+0000 7ff5774a2700 1 -- 172.21.15.46:0/2190307296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff570157620 con 0x7ff570067af0 2024-01-26T23:45:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.166+0000 7ff5657fa700 1 -- 172.21.15.46:0/2190307296 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7ff56c01e460 con 0x7ff570067af0 2024-01-26T23:45:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.166+0000 7ff5657fa700 1 --2- 172.21.15.46:0/2190307296 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff554066ea0 0x7ff554069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.166+0000 7ff5657fa700 1 -- 172.21.15.46:0/2190307296 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6317+0+0 (secure 0 0 0) 0x7ff56c084b60 con 0x7ff570067af0 2024-01-26T23:45:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.166+0000 7ff574a3d700 1 --2- 172.21.15.46:0/2190307296 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff554066ea0 0x7ff554069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.167+0000 7ff574a3d700 1 --2- 172.21.15.46:0/2190307296 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff554066ea0 0x7ff554069350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff570155a00 tx=0x7ff560008040 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:52.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.171+0000 7ff5657fa700 1 -- 172.21.15.46:0/2190307296 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff56c051c50 con 0x7ff570067af0 2024-01-26T23:45:52.362 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:52.361+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:52.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:52.425+0000 7ff5774a2700 1 -- 172.21.15.46:0/2190307296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff57004c710 con 0x7ff570067af0 2024-01-26T23:45:53.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.017+0000 7ff5657fa700 1 -- 172.21.15.46:0/2190307296 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v41) v1 ==== 174+0+0 (secure 0 0 0) 0x7ff56c055270 con 0x7ff570067af0 2024-01-26T23:45:53.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.022+0000 7ff5774a2700 1 -- 172.21.15.46:0/2190307296 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff554066ea0 msgr2=0x7ff554069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.022+0000 7ff5774a2700 1 --2- 172.21.15.46:0/2190307296 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff554066ea0 0x7ff554069350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff570155a00 tx=0x7ff560008040 comp rx=0 tx=0).stop 2024-01-26T23:45:53.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.022+0000 7ff5774a2700 1 -- 172.21.15.46:0/2190307296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff570067af0 msgr2=0x7ff57015cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.022+0000 7ff5774a2700 1 --2- 172.21.15.46:0/2190307296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff570067af0 0x7ff57015cdc0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7ff56c00d860 tx=0x7ff56c008040 comp rx=0 tx=0).stop 2024-01-26T23:45:53.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.022+0000 7ff5774a2700 1 -- 172.21.15.46:0/2190307296 shutdown_connections 2024-01-26T23:45:53.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.022+0000 7ff5774a2700 1 --2- 172.21.15.46:0/2190307296 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff554066ea0 0x7ff554069350 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.022+0000 7ff5774a2700 1 --2- 172.21.15.46:0/2190307296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff570067af0 0x7ff57015cdc0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.022+0000 7ff5774a2700 1 --2- 172.21.15.46:0/2190307296 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5701597e0 0x7ff570154730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.022+0000 7ff5774a2700 1 --2- 172.21.15.46:0/2190307296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff57006ebe0 0x7ff5701512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.022+0000 7ff5774a2700 1 -- 172.21.15.46:0/2190307296 >> 172.21.15.46:0/2190307296 conn(0x7ff5700cdf80 msgr2=0x7ff5700c9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:53.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.022+0000 7ff5774a2700 1 -- 172.21.15.46:0/2190307296 shutdown_connections 2024-01-26T23:45:53.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.022+0000 7ff5774a2700 1 -- 172.21.15.46:0/2190307296 wait complete. 2024-01-26T23:45:53.024 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 11 and data pool 12 2024-01-26T23:45:53.038 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:45:53.094 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:53.092+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:53.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.178+0000 7fac3eea0700 1 Processor -- start 2024-01-26T23:45:53.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.178+0000 7fac3eea0700 1 -- start start 2024-01-26T23:45:53.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.179+0000 7fac3eea0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac380c6520 0x7fac380ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.179+0000 7fac3eea0700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3815a0b0 con 0x7fac380b2b40 2024-01-26T23:45:53.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.179+0000 7fac3eea0700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac380ae4d0 con 0x7fac380ced50 2024-01-26T23:45:53.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.179+0000 7fac3eea0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac380af070 con 0x7fac380c6520 2024-01-26T23:45:53.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.179+0000 7fac3cc3c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fac380b2b40 0x7fac380ae960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52128/0 (socket says 172.21.15.46:52128) 2024-01-26T23:45:53.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.179+0000 7fac3cc3c700 1 -- 172.21.15.46:0/2405233071 learned_addr learned my addr 172.21.15.46:0/2405233071 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:53.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.179+0000 7fac37fff700 1 --2- 172.21.15.46:0/2405233071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac380c6520 0x7fac380ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.179+0000 7fac377fe700 1 -- 172.21.15.46:0/2405233071 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022355403 0 0) 0x7fac3815a0b0 con 0x7fac380b2b40 2024-01-26T23:45:53.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.179+0000 7fac377fe700 1 -- 172.21.15.46:0/2405233071 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac28003570 con 0x7fac380b2b40 2024-01-26T23:45:53.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.180+0000 7fac377fe700 1 -- 172.21.15.46:0/2405233071 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3348295014 0 0) 0x7fac380ae4d0 con 0x7fac380ced50 2024-01-26T23:45:53.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.180+0000 7fac377fe700 1 -- 172.21.15.46:0/2405233071 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac3815a0b0 con 0x7fac380ced50 2024-01-26T23:45:53.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.180+0000 7fac37fff700 1 -- 172.21.15.46:0/2405233071 >> v1:172.21.15.106:6789/0 conn(0x7fac380ced50 legacy=0x7fac380cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.180+0000 7fac37fff700 1 -- 172.21.15.46:0/2405233071 >> v1:172.21.15.46:6789/0 conn(0x7fac380b2b40 legacy=0x7fac380ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.180+0000 7fac37fff700 1 -- 172.21.15.46:0/2405233071 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac380ad8c0 con 0x7fac380c6520 2024-01-26T23:45:53.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.180+0000 7fac37fff700 1 --2- 172.21.15.46:0/2405233071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac380c6520 0x7fac380ad1c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fac24009910 tx=0x7fac240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c39466e1798e7aca server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:53.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.180+0000 7fac377fe700 1 -- 172.21.15.46:0/2405233071 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac24011630 con 0x7fac380c6520 2024-01-26T23:45:53.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.180+0000 7fac377fe700 1 -- 172.21.15.46:0/2405233071 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac24011790 con 0x7fac380c6520 2024-01-26T23:45:53.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.180+0000 7fac377fe700 1 -- 172.21.15.46:0/2405233071 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac24011b30 con 0x7fac380c6520 2024-01-26T23:45:53.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.181+0000 7fac3eea0700 1 -- 172.21.15.46:0/2405233071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac380c6520 msgr2=0x7fac380ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.181+0000 7fac3eea0700 1 --2- 172.21.15.46:0/2405233071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac380c6520 0x7fac380ad1c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fac24009910 tx=0x7fac240091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.181+0000 7fac3eea0700 1 -- 172.21.15.46:0/2405233071 shutdown_connections 2024-01-26T23:45:53.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.181+0000 7fac3eea0700 1 --2- 172.21.15.46:0/2405233071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac380c6520 0x7fac380ad1c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.181+0000 7fac3eea0700 1 -- 172.21.15.46:0/2405233071 >> 172.21.15.46:0/2405233071 conn(0x7fac38099a30 msgr2=0x7fac3806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:53.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.181+0000 7fac3eea0700 1 -- 172.21.15.46:0/2405233071 shutdown_connections 2024-01-26T23:45:53.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.181+0000 7fac3eea0700 1 -- 172.21.15.46:0/2405233071 wait complete. 2024-01-26T23:45:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.182+0000 7fac3eea0700 1 Processor -- start 2024-01-26T23:45:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.182+0000 7fac3eea0700 1 -- start start 2024-01-26T23:45:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.183+0000 7fac3eea0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac380b2b40 0x7fac380b3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.183+0000 7fac3eea0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac380c6520 0x7fac381516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.183+0000 7fac3eea0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac380ced50 0x7fac380b0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.183+0000 7fac3eea0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3816d210 con 0x7fac380ced50 2024-01-26T23:45:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.183+0000 7fac3eea0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3816d460 con 0x7fac380b2b40 2024-01-26T23:45:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.183+0000 7fac3eea0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac3816d740 con 0x7fac380c6520 2024-01-26T23:45:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.183+0000 7fac3cc3c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac380b2b40 0x7fac380b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.183+0000 7fac3d43d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac380ced50 0x7fac380b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.183+0000 7fac37fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac380c6520 0x7fac381516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.183+0000 7fac3cc3c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac380b2b40 0x7fac380b3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:57120/0 (socket says 172.21.15.46:57120) 2024-01-26T23:45:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.183+0000 7fac3d43d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac380ced50 0x7fac380b0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33348/0 (socket says 172.21.15.46:33348) 2024-01-26T23:45:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.183+0000 7fac37fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac380c6520 0x7fac381516d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:47648/0 (socket says 172.21.15.46:47648) 2024-01-26T23:45:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.183+0000 7fac3cc3c700 1 -- 172.21.15.46:0/2744198434 learned_addr learned my addr 172.21.15.46:0/2744198434 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.184+0000 7fac37fff700 1 -- 172.21.15.46:0/2744198434 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac380b2b40 msgr2=0x7fac380b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.184+0000 7fac37fff700 1 --2- 172.21.15.46:0/2744198434 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac380b2b40 0x7fac380b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.184+0000 7fac37fff700 1 -- 172.21.15.46:0/2744198434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac380ced50 msgr2=0x7fac380b0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.184+0000 7fac37fff700 1 --2- 172.21.15.46:0/2744198434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac380ced50 0x7fac380b0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.184+0000 7fac37fff700 1 -- 172.21.15.46:0/2744198434 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac24009420 con 0x7fac380c6520 2024-01-26T23:45:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.184+0000 7fac3cc3c700 1 --2- 172.21.15.46:0/2744198434 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac380b2b40 0x7fac380b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.184+0000 7fac3d43d700 1 --2- 172.21.15.46:0/2744198434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac380ced50 0x7fac380b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.184+0000 7fac37fff700 1 --2- 172.21.15.46:0/2744198434 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac380c6520 0x7fac381516d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fac24016000 tx=0x7fac24017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.184+0000 7fac34ff9700 1 -- 172.21.15.46:0/2744198434 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2401f070 con 0x7fac380c6520 2024-01-26T23:45:53.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.184+0000 7fac34ff9700 1 -- 172.21.15.46:0/2744198434 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac24002f70 con 0x7fac380c6520 2024-01-26T23:45:53.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.184+0000 7fac3eea0700 1 -- 172.21.15.46:0/2744198434 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac3816da20 con 0x7fac380c6520 2024-01-26T23:45:53.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.184+0000 7fac34ff9700 1 -- 172.21.15.46:0/2744198434 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac24011900 con 0x7fac380c6520 2024-01-26T23:45:53.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.184+0000 7fac3eea0700 1 -- 172.21.15.46:0/2744198434 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac3816dfe0 con 0x7fac380c6520 2024-01-26T23:45:53.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.185+0000 7fac3eea0700 1 -- 172.21.15.46:0/2744198434 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac380c7800 con 0x7fac380c6520 2024-01-26T23:45:53.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.186+0000 7fac34ff9700 1 -- 172.21.15.46:0/2744198434 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fac24023070 con 0x7fac380c6520 2024-01-26T23:45:53.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.186+0000 7fac34ff9700 1 --2- 172.21.15.46:0/2744198434 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fac1c066ea0 0x7fac1c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.186+0000 7fac34ff9700 1 -- 172.21.15.46:0/2744198434 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fac24085790 con 0x7fac380c6520 2024-01-26T23:45:53.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.186+0000 7fac3cc3c700 1 --2- 172.21.15.46:0/2744198434 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fac1c066ea0 0x7fac1c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.187+0000 7fac3cc3c700 1 --2- 172.21.15.46:0/2744198434 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fac1c066ea0 0x7fac1c069350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fac2c001360 tx=0x7fac2c009040 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:53.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.191+0000 7fac34ff9700 1 -- 172.21.15.46:0/2744198434 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fac24052810 con 0x7fac380c6520 2024-01-26T23:45:53.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.403+0000 7fac3eea0700 1 -- 172.21.15.46:0/2744198434 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac3804c710 con 0x7fac380c6520 2024-01-26T23:45:53.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.404+0000 7fac34ff9700 1 -- 172.21.15.46:0/2744198434 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16165 (secure 0 0 0) 0x7fac24055e30 con 0x7fac380c6520 2024-01-26T23:45:53.405 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:53.405 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":97,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:45:52.426643+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:45:50.922223+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-01-26T23:45:51.397995+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:45:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.407+0000 7fac3eea0700 1 -- 172.21.15.46:0/2744198434 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fac1c066ea0 msgr2=0x7fac1c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.407+0000 7fac3eea0700 1 --2- 172.21.15.46:0/2744198434 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fac1c066ea0 0x7fac1c069350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fac2c001360 tx=0x7fac2c009040 comp rx=0 tx=0).stop 2024-01-26T23:45:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.407+0000 7fac3eea0700 1 -- 172.21.15.46:0/2744198434 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac380c6520 msgr2=0x7fac381516d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.407+0000 7fac3eea0700 1 --2- 172.21.15.46:0/2744198434 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac380c6520 0x7fac381516d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fac24016000 tx=0x7fac24017000 comp rx=0 tx=0).stop 2024-01-26T23:45:53.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.407+0000 7fac3eea0700 1 -- 172.21.15.46:0/2744198434 shutdown_connections 2024-01-26T23:45:53.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.407+0000 7fac3eea0700 1 --2- 172.21.15.46:0/2744198434 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fac1c066ea0 0x7fac1c069350 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.407+0000 7fac3eea0700 1 --2- 172.21.15.46:0/2744198434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac380ced50 0x7fac380b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.407+0000 7fac3eea0700 1 --2- 172.21.15.46:0/2744198434 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac380b2b40 0x7fac380b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.407+0000 7fac3eea0700 1 --2- 172.21.15.46:0/2744198434 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac380c6520 0x7fac381516d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.407+0000 7fac3eea0700 1 -- 172.21.15.46:0/2744198434 >> 172.21.15.46:0/2744198434 conn(0x7fac38099a30 msgr2=0x7fac38158f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:53.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.407+0000 7fac3eea0700 1 -- 172.21.15.46:0/2744198434 shutdown_connections 2024-01-26T23:45:53.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.407+0000 7fac3eea0700 1 -- 172.21.15.46:0/2744198434 wait complete. 2024-01-26T23:45:53.421 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:45:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.557+0000 7f954faca700 1 Processor -- start 2024-01-26T23:45:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.557+0000 7f954faca700 1 -- start start 2024-01-26T23:45:53.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.558+0000 7f954faca700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9548157340 0x7f954815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.558+0000 7f954faca700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95480ae520 0x7f95480b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.558+0000 7f954faca700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f95480b3fb0 0x7f9548157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.558+0000 7f954faca700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95480b4430 con 0x7f9548157340 2024-01-26T23:45:53.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.558+0000 7f954faca700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9548158ae0 con 0x7f95480b3fb0 2024-01-26T23:45:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.558+0000 7f954faca700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954815b650 con 0x7f95480ae520 2024-01-26T23:45:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.558+0000 7f954d866700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9548157340 0x7f954815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.558+0000 7f954d866700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9548157340 0x7f954815bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33350/0 (socket says 172.21.15.46:33350) 2024-01-26T23:45:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.558+0000 7f954d866700 1 -- 172.21.15.46:0/2870161299 learned_addr learned my addr 172.21.15.46:0/2870161299 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.558+0000 7f954d065700 1 --2- 172.21.15.46:0/2870161299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95480ae520 0x7f95480b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.558+0000 7f954e067700 1 --2- 172.21.15.46:0/2870161299 >> v2:172.21.15.106:3300/0 conn(0x7f95480b3fb0 0x7f9548157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.558+0000 7f954d866700 1 -- 172.21.15.46:0/2870161299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95480ae520 msgr2=0x7f95480b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.558+0000 7f954d866700 1 --2- 172.21.15.46:0/2870161299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95480ae520 0x7f95480b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.559+0000 7f954d866700 1 -- 172.21.15.46:0/2870161299 >> v2:172.21.15.106:3300/0 conn(0x7f95480b3fb0 msgr2=0x7f9548157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.559+0000 7f954d866700 1 --2- 172.21.15.46:0/2870161299 >> v2:172.21.15.106:3300/0 conn(0x7f95480b3fb0 0x7f9548157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.559+0000 7f954d866700 1 -- 172.21.15.46:0/2870161299 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954815c020 con 0x7f9548157340 2024-01-26T23:45:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.559+0000 7f954d866700 1 --2- 172.21.15.46:0/2870161299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9548157340 0x7f954815bae0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f95480caee0 tx=0x7f9538009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03140a7a25528d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.559+0000 7f954c864700 1 -- 172.21.15.46:0/2870161299 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953800e070 con 0x7f9548157340 2024-01-26T23:45:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.559+0000 7f954c864700 1 -- 172.21.15.46:0/2870161299 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f953800ba50 con 0x7f9548157340 2024-01-26T23:45:53.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.559+0000 7f954c864700 1 -- 172.21.15.46:0/2870161299 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95380092e0 con 0x7f9548157340 2024-01-26T23:45:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.559+0000 7f954faca700 1 -- 172.21.15.46:0/2870161299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9548157340 msgr2=0x7f954815bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.559+0000 7f954faca700 1 --2- 172.21.15.46:0/2870161299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9548157340 0x7f954815bae0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f95480caee0 tx=0x7f9538009840 comp rx=0 tx=0).stop 2024-01-26T23:45:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.560+0000 7f954faca700 1 -- 172.21.15.46:0/2870161299 shutdown_connections 2024-01-26T23:45:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.560+0000 7f954faca700 1 --2- 172.21.15.46:0/2870161299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9548157340 0x7f954815bae0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.560+0000 7f954faca700 1 --2- 172.21.15.46:0/2870161299 >> v2:172.21.15.106:3300/0 conn(0x7f95480b3fb0 0x7f9548157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.560+0000 7f954faca700 1 --2- 172.21.15.46:0/2870161299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95480ae520 0x7f95480b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.560+0000 7f954faca700 1 -- 172.21.15.46:0/2870161299 >> 172.21.15.46:0/2870161299 conn(0x7f95480ce530 msgr2=0x7f95480ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.560+0000 7f954faca700 1 -- 172.21.15.46:0/2870161299 shutdown_connections 2024-01-26T23:45:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.560+0000 7f954faca700 1 -- 172.21.15.46:0/2870161299 wait complete. 2024-01-26T23:45:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.561+0000 7f954faca700 1 Processor -- start 2024-01-26T23:45:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.561+0000 7f954faca700 1 -- start start 2024-01-26T23:45:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.562+0000 7f954faca700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95480ae520 0x7f954815d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.562+0000 7f954faca700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95480b3fb0 0x7f95480c7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.562+0000 7f954faca700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9548157340 0x7f95480c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.562+0000 7f954faca700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95480c5540 con 0x7f95480ae520 2024-01-26T23:45:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.562+0000 7f954faca700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95480c5790 con 0x7f9548157340 2024-01-26T23:45:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.562+0000 7f954faca700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95480c5a70 con 0x7f95480b3fb0 2024-01-26T23:45:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.562+0000 7f954d866700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95480ae520 0x7f954815d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.562+0000 7f954d866700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95480ae520 0x7f954815d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33352/0 (socket says 172.21.15.46:33352) 2024-01-26T23:45:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.562+0000 7f954d866700 1 -- 172.21.15.46:0/719650554 learned_addr learned my addr 172.21.15.46:0/719650554 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.562+0000 7f954d065700 1 --2- 172.21.15.46:0/719650554 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95480b3fb0 0x7f95480c7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.562+0000 7f954e067700 1 --2- 172.21.15.46:0/719650554 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9548157340 0x7f95480c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.562+0000 7f954d866700 1 -- 172.21.15.46:0/719650554 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95480b3fb0 msgr2=0x7f95480c7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.562+0000 7f954d866700 1 --2- 172.21.15.46:0/719650554 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95480b3fb0 0x7f95480c7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.562+0000 7f954d866700 1 -- 172.21.15.46:0/719650554 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9548157340 msgr2=0x7f95480c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.562+0000 7f954d866700 1 --2- 172.21.15.46:0/719650554 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9548157340 0x7f95480c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.562+0000 7f954d866700 1 -- 172.21.15.46:0/719650554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9538009ae0 con 0x7f95480ae520 2024-01-26T23:45:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.562+0000 7f954d866700 1 --2- 172.21.15.46:0/719650554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95480ae520 0x7f954815d430 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f9538009fd0 tx=0x7f9538018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.563+0000 7f9545ffb700 1 -- 172.21.15.46:0/719650554 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9538007190 con 0x7f95480ae520 2024-01-26T23:45:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.563+0000 7f9545ffb700 1 -- 172.21.15.46:0/719650554 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9538007820 con 0x7f95480ae520 2024-01-26T23:45:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.563+0000 7f954faca700 1 -- 172.21.15.46:0/719650554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95480c5d50 con 0x7f95480ae520 2024-01-26T23:45:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.563+0000 7f9545ffb700 1 -- 172.21.15.46:0/719650554 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9538007be0 con 0x7f95480ae520 2024-01-26T23:45:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.563+0000 7f954faca700 1 -- 172.21.15.46:0/719650554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95480c6310 con 0x7f95480ae520 2024-01-26T23:45:53.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.564+0000 7f954faca700 1 -- 172.21.15.46:0/719650554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f954804c710 con 0x7f95480ae520 2024-01-26T23:45:53.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.564+0000 7f9545ffb700 1 -- 172.21.15.46:0/719650554 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f9538025460 con 0x7f95480ae520 2024-01-26T23:45:53.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.565+0000 7f9545ffb700 1 --2- 172.21.15.46:0/719650554 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9534066ea0 0x7f9534069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.565+0000 7f9545ffb700 1 -- 172.21.15.46:0/719650554 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f9538085fb0 con 0x7f95480ae520 2024-01-26T23:45:53.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.565+0000 7f954d065700 1 --2- 172.21.15.46:0/719650554 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9534066ea0 0x7f9534069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.566+0000 7f954d065700 1 --2- 172.21.15.46:0/719650554 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9534066ea0 0x7f9534069350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f953c006fd0 tx=0x7f953c008040 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.569+0000 7f9545ffb700 1 -- 172.21.15.46:0/719650554 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f95380b75a0 con 0x7f95480ae520 2024-01-26T23:45:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.781+0000 7f954faca700 1 -- 172.21.15.46:0/719650554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9548158520 con 0x7f95480ae520 2024-01-26T23:45:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.782+0000 7f9545ffb700 1 -- 172.21.15.46:0/719650554 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16165 (secure 0 0 0) 0x7f9538053030 con 0x7f95480ae520 2024-01-26T23:45:53.782 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:53.782 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":97,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:45:52.426643+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:45:50.922223+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-01-26T23:45:51.397995+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:45:53.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.784+0000 7f954faca700 1 -- 172.21.15.46:0/719650554 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9534066ea0 msgr2=0x7f9534069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.784+0000 7f954faca700 1 --2- 172.21.15.46:0/719650554 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9534066ea0 0x7f9534069350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f953c006fd0 tx=0x7f953c008040 comp rx=0 tx=0).stop 2024-01-26T23:45:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.784+0000 7f954faca700 1 -- 172.21.15.46:0/719650554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95480ae520 msgr2=0x7f954815d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.784+0000 7f954faca700 1 --2- 172.21.15.46:0/719650554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95480ae520 0x7f954815d430 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f9538009fd0 tx=0x7f9538018000 comp rx=0 tx=0).stop 2024-01-26T23:45:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.784+0000 7f954faca700 1 -- 172.21.15.46:0/719650554 shutdown_connections 2024-01-26T23:45:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.784+0000 7f954faca700 1 --2- 172.21.15.46:0/719650554 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9534066ea0 0x7f9534069350 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.784+0000 7f954faca700 1 --2- 172.21.15.46:0/719650554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95480ae520 0x7f954815d430 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.784+0000 7f954faca700 1 --2- 172.21.15.46:0/719650554 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9548157340 0x7f95480c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.784+0000 7f954faca700 1 --2- 172.21.15.46:0/719650554 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95480b3fb0 0x7f95480c7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.784+0000 7f954faca700 1 -- 172.21.15.46:0/719650554 >> 172.21.15.46:0/719650554 conn(0x7f95480ce530 msgr2=0x7f95481598d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.784+0000 7f954faca700 1 -- 172.21.15.46:0/719650554 shutdown_connections 2024-01-26T23:45:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.784+0000 7f954faca700 1 -- 172.21.15.46:0/719650554 wait complete. 2024-01-26T23:45:53.794 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:53.793+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:53.799 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-26T23:45:53.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.938+0000 7f6358e32700 1 Processor -- start 2024-01-26T23:45:53.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.938+0000 7f6358e32700 1 -- start start 2024-01-26T23:45:53.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.939+0000 7f6358e32700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f63540c93a0 0x7f63540c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.939+0000 7f6358e32700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6354067d50 0x7f6354159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.939+0000 7f6358e32700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635415e390 con 0x7f63540681f0 2024-01-26T23:45:53.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.939+0000 7f6358e32700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f635415d810 con 0x7f6354067d50 2024-01-26T23:45:53.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.939+0000 7f6358e32700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f635415daf0 con 0x7f63540c93a0 2024-01-26T23:45:53.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.939+0000 7f6351d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6354067d50 0x7f6354159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.939+0000 7f6352d9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f63540681f0 0x7f6354159760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52130/0 (socket says 172.21.15.46:52130) 2024-01-26T23:45:53.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.939+0000 7f6352d9d700 1 -- 172.21.15.46:0/2997907580 learned_addr learned my addr 172.21.15.46:0/2997907580 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:53.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.939+0000 7f635259c700 1 --2- 172.21.15.46:0/2997907580 >> v2:172.21.15.106:3300/0 conn(0x7f63540c93a0 0x7f63540c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.939+0000 7f6351d9b700 1 -- 172.21.15.46:0/2997907580 >> v2:172.21.15.106:3300/0 conn(0x7f63540c93a0 msgr2=0x7f63540c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:53.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.939+0000 7f6351d9b700 1 --2- 172.21.15.46:0/2997907580 >> v2:172.21.15.106:3300/0 conn(0x7f63540c93a0 0x7f63540c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.939+0000 7f6351d9b700 1 -- 172.21.15.46:0/2997907580 >> v1:172.21.15.46:6789/0 conn(0x7f63540681f0 legacy=0x7f6354159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.939+0000 7f6351d9b700 1 -- 172.21.15.46:0/2997907580 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635415ddd0 con 0x7f6354067d50 2024-01-26T23:45:53.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.939+0000 7f635159a700 1 -- 172.21.15.46:0/2997907580 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715900491 0 0) 0x7f635415e390 con 0x7f63540681f0 2024-01-26T23:45:53.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.940+0000 7f6351d9b700 1 --2- 172.21.15.46:0/2997907580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6354067d50 0x7f6354159220 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f635406e380 tx=0x7f63480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6c8404e2ecd38d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:53.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.940+0000 7f635159a700 1 -- 172.21.15.46:0/2997907580 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348011630 con 0x7f6354067d50 2024-01-26T23:45:53.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.940+0000 7f635159a700 1 -- 172.21.15.46:0/2997907580 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6348011790 con 0x7f6354067d50 2024-01-26T23:45:53.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.940+0000 7f635159a700 1 -- 172.21.15.46:0/2997907580 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348011b30 con 0x7f6354067d50 2024-01-26T23:45:53.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.940+0000 7f6358e32700 1 -- 172.21.15.46:0/2997907580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6354067d50 msgr2=0x7f6354159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.940+0000 7f6358e32700 1 --2- 172.21.15.46:0/2997907580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6354067d50 0x7f6354159220 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f635406e380 tx=0x7f63480091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.940+0000 7f6358e32700 1 -- 172.21.15.46:0/2997907580 shutdown_connections 2024-01-26T23:45:53.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.940+0000 7f6358e32700 1 --2- 172.21.15.46:0/2997907580 >> v2:172.21.15.106:3300/0 conn(0x7f63540c93a0 0x7f63540c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.940+0000 7f6358e32700 1 --2- 172.21.15.46:0/2997907580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6354067d50 0x7f6354159220 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.940+0000 7f6358e32700 1 -- 172.21.15.46:0/2997907580 >> 172.21.15.46:0/2997907580 conn(0x7f63540ca9a0 msgr2=0x7f6354068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:53.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.941+0000 7f6358e32700 1 -- 172.21.15.46:0/2997907580 shutdown_connections 2024-01-26T23:45:53.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.941+0000 7f6358e32700 1 -- 172.21.15.46:0/2997907580 wait complete. 2024-01-26T23:45:53.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.942+0000 7f6358e32700 1 Processor -- start 2024-01-26T23:45:53.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.942+0000 7f6358e32700 1 -- start start 2024-01-26T23:45:53.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f6358e32700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6354067d50 0x7f6354150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f6358e32700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63540681f0 0x7f63540c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f6358e32700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c93a0 0x7f63540c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f6358e32700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63540c4e10 con 0x7f63540c93a0 2024-01-26T23:45:53.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f6358e32700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63541508a0 con 0x7f63540681f0 2024-01-26T23:45:53.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f6358e32700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63540b1530 con 0x7f6354067d50 2024-01-26T23:45:53.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f6352d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c93a0 0x7f63540c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f6352d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c93a0 0x7f63540c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33362/0 (socket says 172.21.15.46:33362) 2024-01-26T23:45:53.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f6352d9d700 1 -- 172.21.15.46:0/2831151916 learned_addr learned my addr 172.21.15.46:0/2831151916 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:53.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f6351d9b700 1 --2- 172.21.15.46:0/2831151916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63540681f0 0x7f63540c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f635259c700 1 --2- 172.21.15.46:0/2831151916 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6354067d50 0x7f6354150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f6352d9d700 1 -- 172.21.15.46:0/2831151916 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6354067d50 msgr2=0x7f6354150360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f6352d9d700 1 --2- 172.21.15.46:0/2831151916 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6354067d50 0x7f6354150360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f6352d9d700 1 -- 172.21.15.46:0/2831151916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63540681f0 msgr2=0x7f63540c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f6352d9d700 1 --2- 172.21.15.46:0/2831151916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63540681f0 0x7f63540c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f6352d9d700 1 -- 172.21.15.46:0/2831151916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6348009420 con 0x7f63540c93a0 2024-01-26T23:45:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f6351d9b700 1 --2- 172.21.15.46:0/2831151916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63540681f0 0x7f63540c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f635259c700 1 --2- 172.21.15.46:0/2831151916 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6354067d50 0x7f6354150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f6352d9d700 1 --2- 172.21.15.46:0/2831151916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c93a0 0x7f63540c52a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f634c00ccc0 tx=0x7f634c001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f63427fc700 1 -- 172.21.15.46:0/2831151916 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c00ab10 con 0x7f63540c93a0 2024-01-26T23:45:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.943+0000 7f63427fc700 1 -- 172.21.15.46:0/2831151916 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f634c00ac70 con 0x7f63540c93a0 2024-01-26T23:45:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.944+0000 7f6358e32700 1 -- 172.21.15.46:0/2831151916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63540c47c0 con 0x7f63540c93a0 2024-01-26T23:45:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.944+0000 7f63427fc700 1 -- 172.21.15.46:0/2831151916 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c0070f0 con 0x7f63540c93a0 2024-01-26T23:45:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.944+0000 7f6358e32700 1 -- 172.21.15.46:0/2831151916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63540b1810 con 0x7f63540c93a0 2024-01-26T23:45:53.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.945+0000 7f6358e32700 1 -- 172.21.15.46:0/2831151916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63540c4f80 con 0x7f63540c93a0 2024-01-26T23:45:53.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.945+0000 7f63427fc700 1 -- 172.21.15.46:0/2831151916 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f634c007520 con 0x7f63540c93a0 2024-01-26T23:45:53.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.946+0000 7f63427fc700 1 --2- 172.21.15.46:0/2831151916 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6334066ea0 0x7f6334069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:53.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.946+0000 7f63427fc700 1 -- 172.21.15.46:0/2831151916 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f634c084b30 con 0x7f63540c93a0 2024-01-26T23:45:53.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.946+0000 7f635259c700 1 --2- 172.21.15.46:0/2831151916 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6334066ea0 0x7f6334069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:53.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.946+0000 7f635259c700 1 --2- 172.21.15.46:0/2831151916 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6334066ea0 0x7f6334069350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f635415d080 tx=0x7f6344008040 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:53.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:53.950+0000 7f63427fc700 1 -- 172.21.15.46:0/2831151916 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f634c051c30 con 0x7f63540c93a0 2024-01-26T23:45:54.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:54.208+0000 7f6358e32700 1 -- 172.21.15.46:0/2831151916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f635404c710 con 0x7f63540c93a0 2024-01-26T23:45:54.512 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:54.511+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:54.859 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:45:54.862 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:45:54.920 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-01-26T23:45:54.919+0000 7f274a58b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-26T23:45:55.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.005+0000 7f63427fc700 1 -- 172.21.15.46:0/2831151916 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v44) v1 ==== 118+0+0 (secure 0 0 0) 0x7f634c055250 con 0x7f63540c93a0 2024-01-26T23:45:55.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.008+0000 7f6358e32700 1 -- 172.21.15.46:0/2831151916 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6334066ea0 msgr2=0x7f6334069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:55.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.008+0000 7f6358e32700 1 --2- 172.21.15.46:0/2831151916 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6334066ea0 0x7f6334069350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f635415d080 tx=0x7f6344008040 comp rx=0 tx=0).stop 2024-01-26T23:45:55.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.008+0000 7f6358e32700 1 -- 172.21.15.46:0/2831151916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c93a0 msgr2=0x7f63540c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:55.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.008+0000 7f6358e32700 1 --2- 172.21.15.46:0/2831151916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c93a0 0x7f63540c52a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f634c00ccc0 tx=0x7f634c001a20 comp rx=0 tx=0).stop 2024-01-26T23:45:55.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.009+0000 7f6358e32700 1 -- 172.21.15.46:0/2831151916 shutdown_connections 2024-01-26T23:45:55.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.009+0000 7f6358e32700 1 --2- 172.21.15.46:0/2831151916 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6334066ea0 0x7f6334069350 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:55.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.009+0000 7f6358e32700 1 --2- 172.21.15.46:0/2831151916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c93a0 0x7f63540c52a0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:55.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.009+0000 7f6358e32700 1 --2- 172.21.15.46:0/2831151916 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6354067d50 0x7f6354150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:55.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.009+0000 7f6358e32700 1 --2- 172.21.15.46:0/2831151916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63540681f0 0x7f63540c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:55.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.009+0000 7f6358e32700 1 -- 172.21.15.46:0/2831151916 >> 172.21.15.46:0/2831151916 conn(0x7f63540ca9a0 msgr2=0x7f6354069360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:55.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.009+0000 7f6358e32700 1 -- 172.21.15.46:0/2831151916 shutdown_connections 2024-01-26T23:45:55.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.009+0000 7f6358e32700 1 -- 172.21.15.46:0/2831151916 wait complete. 2024-01-26T23:45:55.048 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-26T23:45:55.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.193+0000 7f581100b700 1 Processor -- start 2024-01-26T23:45:55.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.193+0000 7f581100b700 1 -- start start 2024-01-26T23:45:55.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.194+0000 7f581100b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f580c141200 0x7f580c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:55.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.194+0000 7f581100b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f580c0caf30 0x7f580c0cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:55.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.194+0000 7f581100b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580c0ccc00 con 0x7f580c0c8da0 2024-01-26T23:45:55.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.194+0000 7f581100b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f580c0cbbf0 con 0x7f580c141200 2024-01-26T23:45:55.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.194+0000 7f581100b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580c0cc310 con 0x7f580c0caf30 2024-01-26T23:45:55.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.194+0000 7f580a59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f580c141200 0x7f580c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:55.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.194+0000 7f580a59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f580c141200 0x7f580c0c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33372/0 (socket says 172.21.15.46:33372) 2024-01-26T23:45:55.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.194+0000 7f580a59c700 1 -- 172.21.15.46:0/64860808 learned_addr learned my addr 172.21.15.46:0/64860808 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:55.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.194+0000 7f580b59e700 1 --2- 172.21.15.46:0/64860808 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f580c0caf30 0x7f580c0cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:55.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.194+0000 7f580a59c700 1 -- 172.21.15.46:0/64860808 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f580c0caf30 msgr2=0x7f580c0cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:55.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.194+0000 7f580a59c700 1 --2- 172.21.15.46:0/64860808 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f580c0caf30 0x7f580c0cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:55.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.194+0000 7f580a59c700 1 -- 172.21.15.46:0/64860808 >> v1:172.21.15.106:6789/0 conn(0x7f580c0c8da0 legacy=0x7f580c140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:55.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.194+0000 7f580a59c700 1 -- 172.21.15.46:0/64860808 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580c0cc5b0 con 0x7f580c141200 2024-01-26T23:45:55.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.194+0000 7f580b59e700 1 --2- 172.21.15.46:0/64860808 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f580c0caf30 0x7f580c0cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:55.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.195+0000 7f580a59c700 1 --2- 172.21.15.46:0/64860808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f580c141200 0x7f580c0c9170 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f58000099b0 tx=0x7f5800009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b861d9c95ee5175 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:55.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.195+0000 7f5809d9b700 1 -- 172.21.15.46:0/64860808 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580000b860 con 0x7f580c141200 2024-01-26T23:45:55.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.195+0000 7f5809d9b700 1 -- 172.21.15.46:0/64860808 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580000b9c0 con 0x7f580c141200 2024-01-26T23:45:55.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.195+0000 7f5809d9b700 1 -- 172.21.15.46:0/64860808 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580000bd60 con 0x7f580c141200 2024-01-26T23:45:55.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.195+0000 7f581100b700 1 -- 172.21.15.46:0/64860808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f580c141200 msgr2=0x7f580c0c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:55.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.195+0000 7f581100b700 1 --2- 172.21.15.46:0/64860808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f580c141200 0x7f580c0c9170 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f58000099b0 tx=0x7f5800009250 comp rx=0 tx=0).stop 2024-01-26T23:45:55.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.195+0000 7f581100b700 1 -- 172.21.15.46:0/64860808 shutdown_connections 2024-01-26T23:45:55.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.195+0000 7f581100b700 1 --2- 172.21.15.46:0/64860808 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f580c0caf30 0x7f580c0cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:55.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.195+0000 7f581100b700 1 --2- 172.21.15.46:0/64860808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f580c141200 0x7f580c0c9170 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:55.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.195+0000 7f581100b700 1 -- 172.21.15.46:0/64860808 >> 172.21.15.46:0/64860808 conn(0x7f580c06bd90 msgr2=0x7f580c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:55.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.195+0000 7f581100b700 1 -- 172.21.15.46:0/64860808 shutdown_connections 2024-01-26T23:45:55.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.195+0000 7f581100b700 1 -- 172.21.15.46:0/64860808 wait complete. 2024-01-26T23:45:55.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.197+0000 7f581100b700 1 Processor -- start 2024-01-26T23:45:55.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.197+0000 7f581100b700 1 -- start start 2024-01-26T23:45:55.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f581100b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f580c0c8da0 0x7f580c157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:55.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f581100b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f580c0caf30 0x7f580c0c7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:55.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f581100b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f580c141200 0x7f580c0c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:55.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f581100b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c1576c0 con 0x7f580c141200 2024-01-26T23:45:55.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f581100b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c0c60a0 con 0x7f580c0caf30 2024-01-26T23:45:55.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f581100b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580c0c6380 con 0x7f580c0c8da0 2024-01-26T23:45:55.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f580a59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f580c0caf30 0x7f580c0c7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:55.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f580ad9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f580c0c8da0 0x7f580c157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:55.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f580b59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f580c141200 0x7f580c0c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:55.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f580a59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f580c0caf30 0x7f580c0c7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:57168/0 (socket says 172.21.15.46:57168) 2024-01-26T23:45:55.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f580a59c700 1 -- 172.21.15.46:0/2513880795 learned_addr learned my addr 172.21.15.46:0/2513880795 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:55.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f580ad9d700 1 -- 172.21.15.46:0/2513880795 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f580c0caf30 msgr2=0x7f580c0c7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:55.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f580ad9d700 1 --2- 172.21.15.46:0/2513880795 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f580c0caf30 0x7f580c0c7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:55.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f580ad9d700 1 -- 172.21.15.46:0/2513880795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f580c141200 msgr2=0x7f580c0c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:55.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f580ad9d700 1 --2- 172.21.15.46:0/2513880795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f580c141200 0x7f580c0c5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:55.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f580ad9d700 1 -- 172.21.15.46:0/2513880795 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58000094f0 con 0x7f580c0c8da0 2024-01-26T23:45:55.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f580b59e700 1 --2- 172.21.15.46:0/2513880795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f580c141200 0x7f580c0c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:55.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f580a59c700 1 --2- 172.21.15.46:0/2513880795 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f580c0caf30 0x7f580c0c7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:55.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.198+0000 7f580ad9d700 1 --2- 172.21.15.46:0/2513880795 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f580c0c8da0 0x7f580c157180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f57fc00c590 tx=0x7f57fc007ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:55.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.199+0000 7f57f2ffd700 1 -- 172.21.15.46:0/2513880795 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc011590 con 0x7f580c0c8da0 2024-01-26T23:45:55.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.199+0000 7f57f2ffd700 1 -- 172.21.15.46:0/2513880795 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57fc011710 con 0x7f580c0c8da0 2024-01-26T23:45:55.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.199+0000 7f581100b700 1 -- 172.21.15.46:0/2513880795 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f580c0c6660 con 0x7f580c0c8da0 2024-01-26T23:45:55.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.199+0000 7f57f2ffd700 1 -- 172.21.15.46:0/2513880795 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc011ad0 con 0x7f580c0c8da0 2024-01-26T23:45:55.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.199+0000 7f581100b700 1 -- 172.21.15.46:0/2513880795 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f580c0c6c20 con 0x7f580c0c8da0 2024-01-26T23:45:55.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.200+0000 7f581100b700 1 -- 172.21.15.46:0/2513880795 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f580c0c6e40 con 0x7f580c0c8da0 2024-01-26T23:45:55.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.200+0000 7f57f2ffd700 1 -- 172.21.15.46:0/2513880795 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f57fc01b3b0 con 0x7f580c0c8da0 2024-01-26T23:45:55.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.201+0000 7f57f2ffd700 1 --2- 172.21.15.46:0/2513880795 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57f8066ea0 0x7f57f8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:55.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.201+0000 7f57f2ffd700 1 -- 172.21.15.46:0/2513880795 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f57fc083b20 con 0x7f580c0c8da0 2024-01-26T23:45:55.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.201+0000 7f580a59c700 1 --2- 172.21.15.46:0/2513880795 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57f8066ea0 0x7f57f8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:55.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.201+0000 7f580a59c700 1 --2- 172.21.15.46:0/2513880795 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57f8066ea0 0x7f57f8069350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5800003080 tx=0x7f5800017000 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:55.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.205+0000 7f57f2ffd700 1 -- 172.21.15.46:0/2513880795 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f57fc050ba0 con 0x7f580c0c8da0 2024-01-26T23:45:55.241 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:55.240+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:55.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:55.461+0000 7f581100b700 1 -- 172.21.15.46:0/2513880795 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f580c04c710 con 0x7f580c0c8da0 2024-01-26T23:45:55.960 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:55.959+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:56.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.007+0000 7f57f2ffd700 1 -- 172.21.15.46:0/2513880795 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v45) v1 ==== 122+0+0 (secure 0 0 0) 0x7f57fc017070 con 0x7f580c0c8da0 2024-01-26T23:45:56.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.010+0000 7f581100b700 1 -- 172.21.15.46:0/2513880795 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57f8066ea0 msgr2=0x7f57f8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.010+0000 7f581100b700 1 --2- 172.21.15.46:0/2513880795 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57f8066ea0 0x7f57f8069350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5800003080 tx=0x7f5800017000 comp rx=0 tx=0).stop 2024-01-26T23:45:56.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.010+0000 7f581100b700 1 -- 172.21.15.46:0/2513880795 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f580c0c8da0 msgr2=0x7f580c157180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.010+0000 7f581100b700 1 --2- 172.21.15.46:0/2513880795 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f580c0c8da0 0x7f580c157180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f57fc00c590 tx=0x7f57fc007ec0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.010+0000 7f581100b700 1 -- 172.21.15.46:0/2513880795 shutdown_connections 2024-01-26T23:45:56.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.010+0000 7f581100b700 1 --2- 172.21.15.46:0/2513880795 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f57f8066ea0 0x7f57f8069350 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.010+0000 7f581100b700 1 --2- 172.21.15.46:0/2513880795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f580c141200 0x7f580c0c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.010+0000 7f581100b700 1 --2- 172.21.15.46:0/2513880795 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f580c0c8da0 0x7f580c157180 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.010+0000 7f581100b700 1 --2- 172.21.15.46:0/2513880795 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f580c0caf30 0x7f580c0c7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.010+0000 7f581100b700 1 -- 172.21.15.46:0/2513880795 >> 172.21.15.46:0/2513880795 conn(0x7f580c06bd90 msgr2=0x7f580c13f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:56.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.010+0000 7f581100b700 1 -- 172.21.15.46:0/2513880795 shutdown_connections 2024-01-26T23:45:56.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.010+0000 7f581100b700 1 -- 172.21.15.46:0/2513880795 wait complete. 2024-01-26T23:45:56.026 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:45:56.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.166+0000 7f5782cd6700 1 Processor -- start 2024-01-26T23:45:56.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.166+0000 7f5782cd6700 1 -- start start 2024-01-26T23:45:56.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.167+0000 7f5782cd6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f577c157340 0x7f577c15bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.167+0000 7f5782cd6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f577c0b4180 0x7f577c157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.167+0000 7f5782cd6700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f577c158ae0 con 0x7f577c0ae520 2024-01-26T23:45:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.167+0000 7f5782cd6700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f577c15b650 con 0x7f577c0b4180 2024-01-26T23:45:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.167+0000 7f5782cd6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577c15c020 con 0x7f577c157340 2024-01-26T23:45:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.167+0000 7f5781273700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f577c0b4180 0x7f577c157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.167+0000 7f5773fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f577c0ae520 0x7f577c0b3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52142/0 (socket says 172.21.15.46:52142) 2024-01-26T23:45:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.167+0000 7f5773fff700 1 -- 172.21.15.46:0/605047656 learned_addr learned my addr 172.21.15.46:0/605047656 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.167+0000 7f5780a72700 1 --2- 172.21.15.46:0/605047656 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f577c157340 0x7f577c15bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.167+0000 7f5781273700 1 -- 172.21.15.46:0/605047656 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f577c157340 msgr2=0x7f577c15bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.167+0000 7f5781273700 1 --2- 172.21.15.46:0/605047656 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f577c157340 0x7f577c15bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.167+0000 7f5781273700 1 -- 172.21.15.46:0/605047656 >> v1:172.21.15.46:6789/0 conn(0x7f577c0ae520 legacy=0x7f577c0b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.167+0000 7f5781273700 1 -- 172.21.15.46:0/605047656 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f577c158520 con 0x7f577c0b4180 2024-01-26T23:45:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.167+0000 7f57737fe700 1 -- 172.21.15.46:0/605047656 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817422620 0 0) 0x7f577c158ae0 con 0x7f577c0ae520 2024-01-26T23:45:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.167+0000 7f5781273700 1 --2- 172.21.15.46:0/605047656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f577c0b4180 0x7f577c157710 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f5774009e60 tx=0x7f5774009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f029103b6f30ec2e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.167+0000 7f57737fe700 1 -- 172.21.15.46:0/605047656 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577400e070 con 0x7f577c0b4180 2024-01-26T23:45:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.168+0000 7f57737fe700 1 -- 172.21.15.46:0/605047656 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f577400bb60 con 0x7f577c0b4180 2024-01-26T23:45:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.168+0000 7f57737fe700 1 -- 172.21.15.46:0/605047656 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5774012430 con 0x7f577c0b4180 2024-01-26T23:45:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.168+0000 7f5782cd6700 1 -- 172.21.15.46:0/605047656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f577c0b4180 msgr2=0x7f577c157710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.168+0000 7f5782cd6700 1 --2- 172.21.15.46:0/605047656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f577c0b4180 0x7f577c157710 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f5774009e60 tx=0x7f5774009780 comp rx=0 tx=0).stop 2024-01-26T23:45:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.168+0000 7f5782cd6700 1 -- 172.21.15.46:0/605047656 shutdown_connections 2024-01-26T23:45:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.168+0000 7f5782cd6700 1 --2- 172.21.15.46:0/605047656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f577c0b4180 0x7f577c157710 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.168+0000 7f5782cd6700 1 --2- 172.21.15.46:0/605047656 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f577c157340 0x7f577c15bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.168+0000 7f5782cd6700 1 -- 172.21.15.46:0/605047656 >> 172.21.15.46:0/605047656 conn(0x7f577c0ce530 msgr2=0x7f577c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.168+0000 7f5782cd6700 1 -- 172.21.15.46:0/605047656 shutdown_connections 2024-01-26T23:45:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.168+0000 7f5782cd6700 1 -- 172.21.15.46:0/605047656 wait complete. 2024-01-26T23:45:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.169+0000 7f5782cd6700 1 Processor -- start 2024-01-26T23:45:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.170+0000 7f5782cd6700 1 -- start start 2024-01-26T23:45:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.170+0000 7f5782cd6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f577c0ae520 0x7f577c150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.170+0000 7f5782cd6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f577c0b4180 0x7f577c0b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.170+0000 7f5782cd6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f577c157340 0x7f577c15d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.170+0000 7f5782cd6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c0c31c0 con 0x7f577c0b4180 2024-01-26T23:45:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5782cd6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c151160 con 0x7f577c0ae520 2024-01-26T23:45:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5782cd6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577c0b1da0 con 0x7f577c157340 2024-01-26T23:45:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5773fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f577c0b4180 0x7f577c0b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5773fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f577c0b4180 0x7f577c0b12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33398/0 (socket says 172.21.15.46:33398) 2024-01-26T23:45:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5773fff700 1 -- 172.21.15.46:0/2646473896 learned_addr learned my addr 172.21.15.46:0/2646473896 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5780a72700 1 --2- 172.21.15.46:0/2646473896 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f577c0ae520 0x7f577c150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5781273700 1 --2- 172.21.15.46:0/2646473896 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f577c157340 0x7f577c15d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5773fff700 1 -- 172.21.15.46:0/2646473896 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f577c157340 msgr2=0x7f577c15d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5773fff700 1 --2- 172.21.15.46:0/2646473896 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f577c157340 0x7f577c15d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5773fff700 1 -- 172.21.15.46:0/2646473896 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f577c0ae520 msgr2=0x7f577c150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5773fff700 1 --2- 172.21.15.46:0/2646473896 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f577c0ae520 0x7f577c150c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5773fff700 1 -- 172.21.15.46:0/2646473896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5774009a20 con 0x7f577c0b4180 2024-01-26T23:45:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5773fff700 1 --2- 172.21.15.46:0/2646473896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f577c0b4180 0x7f577c0b12e0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f576c00c5b0 tx=0x7f576c001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5770ff9700 1 -- 172.21.15.46:0/2646473896 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c005b30 con 0x7f577c0b4180 2024-01-26T23:45:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5770ff9700 1 -- 172.21.15.46:0/2646473896 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576c005c90 con 0x7f577c0b4180 2024-01-26T23:45:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5782cd6700 1 -- 172.21.15.46:0/2646473896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f577c0b2080 con 0x7f577c0b4180 2024-01-26T23:45:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5770ff9700 1 -- 172.21.15.46:0/2646473896 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c007300 con 0x7f577c0b4180 2024-01-26T23:45:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.171+0000 7f5782cd6700 1 -- 172.21.15.46:0/2646473896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f577c0c2e50 con 0x7f577c0b4180 2024-01-26T23:45:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.172+0000 7f5782cd6700 1 -- 172.21.15.46:0/2646473896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f577c0c2cb0 con 0x7f577c0b4180 2024-01-26T23:45:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.173+0000 7f5770ff9700 1 -- 172.21.15.46:0/2646473896 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f576c01b3b0 con 0x7f577c0b4180 2024-01-26T23:45:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.173+0000 7f5770ff9700 1 --2- 172.21.15.46:0/2646473896 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5768066ea0 0x7f5768069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.173+0000 7f5770ff9700 1 -- 172.21.15.46:0/2646473896 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f576c083ae0 con 0x7f577c0b4180 2024-01-26T23:45:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.174+0000 7f5780a72700 1 --2- 172.21.15.46:0/2646473896 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5768066ea0 0x7f5768069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.174+0000 7f5780a72700 1 --2- 172.21.15.46:0/2646473896 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5768066ea0 0x7f5768069350 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5778009db0 tx=0x7f57780091a0 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.178+0000 7f5770ff9700 1 -- 172.21.15.46:0/2646473896 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f577c0c2cb0 con 0x7f577c0b4180 2024-01-26T23:45:56.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.436+0000 7f5782cd6700 1 -- 172.21.15.46:0/2646473896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f577c0c2cb0 con 0x7f577c0b4180 2024-01-26T23:45:56.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.437+0000 7f5770ff9700 1 -- 172.21.15.46:0/2646473896 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f577c0c2cb0 con 0x7f577c0b4180 2024-01-26T23:45:56.437 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:56.438 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6137,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/1370365478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1370365478},{"type":"v1","addr":"172.21.15.106:6836","nonce":1370365478}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":6619,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/1980309094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1980309094},{"type":"v1","addr":"172.21.15.106:6837","nonce":1980309094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40},{"gid":6628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3910368747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3910368747},{"type":"v1","addr":"172.21.15.46:6834","nonce":3910368747}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:45:52.426977+0000","modified":"2024-01-26T23:45:56.004421+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6631":{"gid":6631,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3176096333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3176096333},{"type":"v1","addr":"172.21.15.46:6835","nonce":3176096333}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2024-01-26T23:45:56.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.439+0000 7f5782cd6700 1 -- 172.21.15.46:0/2646473896 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5768066ea0 msgr2=0x7f5768069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.439+0000 7f5782cd6700 1 --2- 172.21.15.46:0/2646473896 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5768066ea0 0x7f5768069350 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5778009db0 tx=0x7f57780091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.439+0000 7f5782cd6700 1 -- 172.21.15.46:0/2646473896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f577c0b4180 msgr2=0x7f577c0b12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.439+0000 7f5782cd6700 1 --2- 172.21.15.46:0/2646473896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f577c0b4180 0x7f577c0b12e0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f576c00c5b0 tx=0x7f576c001070 comp rx=0 tx=0).stop 2024-01-26T23:45:56.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.439+0000 7f5782cd6700 1 -- 172.21.15.46:0/2646473896 shutdown_connections 2024-01-26T23:45:56.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.439+0000 7f5782cd6700 1 --2- 172.21.15.46:0/2646473896 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5768066ea0 0x7f5768069350 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.439+0000 7f5782cd6700 1 --2- 172.21.15.46:0/2646473896 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f577c157340 0x7f577c15d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.439+0000 7f5782cd6700 1 --2- 172.21.15.46:0/2646473896 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f577c0ae520 0x7f577c150c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.439+0000 7f5782cd6700 1 --2- 172.21.15.46:0/2646473896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f577c0b4180 0x7f577c0b12e0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.439+0000 7f5782cd6700 1 -- 172.21.15.46:0/2646473896 >> 172.21.15.46:0/2646473896 conn(0x7f577c0ce530 msgr2=0x7f577c06d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:56.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.440+0000 7f5782cd6700 1 -- 172.21.15.46:0/2646473896 shutdown_connections 2024-01-26T23:45:56.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.440+0000 7f5782cd6700 1 -- 172.21.15.46:0/2646473896 wait complete. 2024-01-26T23:45:56.441 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 45 2024-01-26T23:45:56.454 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:45:56.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.586+0000 7f25ce9be700 1 Processor -- start 2024-01-26T23:45:56.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.586+0000 7f25ce9be700 1 -- start start 2024-01-26T23:45:56.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.587+0000 7f25ce9be700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25c80b1fc0 0x7f25c80b2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:56.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.587+0000 7f25ce9be700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f25c8068310 0x7f25c8068780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:56.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.587+0000 7f25ce9be700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c80cac60 con 0x7f25c8156f80 2024-01-26T23:45:56.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.587+0000 7f25ce9be700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c8068dc0 con 0x7f25c8068310 2024-01-26T23:45:56.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.587+0000 7f25ce9be700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c80690a0 con 0x7f25c80b1fc0 2024-01-26T23:45:56.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.587+0000 7f25c77fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f25c8068310 0x7f25c8068780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:56.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.587+0000 7f25c77fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f25c8068310 0x7f25c8068780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33412/0 (socket says 172.21.15.46:33412) 2024-01-26T23:45:56.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.587+0000 7f25c77fe700 1 -- 172.21.15.46:0/4015320851 learned_addr learned my addr 172.21.15.46:0/4015320851 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:56.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.587+0000 7f25c7fff700 1 --2- 172.21.15.46:0/4015320851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25c80b1fc0 0x7f25c80b2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:56.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.587+0000 7f25c77fe700 1 -- 172.21.15.46:0/4015320851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25c80b1fc0 msgr2=0x7f25c80b2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:56.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.587+0000 7f25c77fe700 1 --2- 172.21.15.46:0/4015320851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25c80b1fc0 0x7f25c80b2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.587+0000 7f25c77fe700 1 -- 172.21.15.46:0/4015320851 >> v1:172.21.15.106:6789/0 conn(0x7f25c8156f80 legacy=0x7f25c81573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:56.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.587+0000 7f25c77fe700 1 -- 172.21.15.46:0/4015320851 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c80caee0 con 0x7f25c8068310 2024-01-26T23:45:56.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.588+0000 7f25c77fe700 1 --2- 172.21.15.46:0/4015320851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25c8068310 0x7f25c8068780 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0099b0 tx=0x7f25bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29cb8ba2bd4b32a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:56.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.588+0000 7f25c6ffd700 1 -- 172.21.15.46:0/4015320851 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc00b860 con 0x7f25c8068310 2024-01-26T23:45:56.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.588+0000 7f25c6ffd700 1 -- 172.21.15.46:0/4015320851 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25bc00b9c0 con 0x7f25c8068310 2024-01-26T23:45:56.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.588+0000 7f25c6ffd700 1 -- 172.21.15.46:0/4015320851 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc00bd60 con 0x7f25c8068310 2024-01-26T23:45:56.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.588+0000 7f25ce9be700 1 -- 172.21.15.46:0/4015320851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25c8068310 msgr2=0x7f25c8068780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.588+0000 7f25ce9be700 1 --2- 172.21.15.46:0/4015320851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25c8068310 0x7f25c8068780 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0099b0 tx=0x7f25bc009250 comp rx=0 tx=0).stop 2024-01-26T23:45:56.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.588+0000 7f25ce9be700 1 -- 172.21.15.46:0/4015320851 shutdown_connections 2024-01-26T23:45:56.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.588+0000 7f25ce9be700 1 --2- 172.21.15.46:0/4015320851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25c80b1fc0 0x7f25c80b2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.588+0000 7f25ce9be700 1 --2- 172.21.15.46:0/4015320851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25c8068310 0x7f25c8068780 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.588+0000 7f25ce9be700 1 -- 172.21.15.46:0/4015320851 >> 172.21.15.46:0/4015320851 conn(0x7f25c80ccbc0 msgr2=0x7f25c80ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:56.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.588+0000 7f25ce9be700 1 -- 172.21.15.46:0/4015320851 shutdown_connections 2024-01-26T23:45:56.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.589+0000 7f25ce9be700 1 -- 172.21.15.46:0/4015320851 wait complete. 2024-01-26T23:45:56.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.590+0000 7f25ce9be700 1 Processor -- start 2024-01-26T23:45:56.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.590+0000 7f25ce9be700 1 -- start start 2024-01-26T23:45:56.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.591+0000 7f25ce9be700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25c8068310 0x7f25c80c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:56.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.591+0000 7f25ce9be700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25c80b1fc0 0x7f25c80c5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.591+0000 7f25ce9be700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25c8156f80 0x7f25c8155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.591+0000 7f25ce9be700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c80c6360 con 0x7f25c8068310 2024-01-26T23:45:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.591+0000 7f25ce9be700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c80c6600 con 0x7f25c8156f80 2024-01-26T23:45:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.591+0000 7f25ce9be700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c80c68e0 con 0x7f25c80b1fc0 2024-01-26T23:45:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.591+0000 7f25c7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25c8068310 0x7f25c80c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.591+0000 7f25c7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25c8068310 0x7f25c80c8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33426/0 (socket says 172.21.15.46:33426) 2024-01-26T23:45:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.591+0000 7f25c7fff700 1 -- 172.21.15.46:0/625878596 learned_addr learned my addr 172.21.15.46:0/625878596 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.591+0000 7f25c77fe700 1 --2- 172.21.15.46:0/625878596 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25c80b1fc0 0x7f25c80c5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.591+0000 7f25ccf5b700 1 --2- 172.21.15.46:0/625878596 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25c8156f80 0x7f25c8155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.591+0000 7f25c7fff700 1 -- 172.21.15.46:0/625878596 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25c80b1fc0 msgr2=0x7f25c80c5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.591+0000 7f25c7fff700 1 --2- 172.21.15.46:0/625878596 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25c80b1fc0 0x7f25c80c5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.591+0000 7f25c7fff700 1 -- 172.21.15.46:0/625878596 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25c8156f80 msgr2=0x7f25c8155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.591+0000 7f25c7fff700 1 --2- 172.21.15.46:0/625878596 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25c8156f80 0x7f25c8155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.591+0000 7f25c7fff700 1 -- 172.21.15.46:0/625878596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25bc0094f0 con 0x7f25c8068310 2024-01-26T23:45:56.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.591+0000 7f25c7fff700 1 --2- 172.21.15.46:0/625878596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25c8068310 0x7f25c80c8c70 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f25b800d890 tx=0x7f25b8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:56.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.592+0000 7f25affff700 1 -- 172.21.15.46:0/625878596 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b80125f0 con 0x7f25c8068310 2024-01-26T23:45:56.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.592+0000 7f25affff700 1 -- 172.21.15.46:0/625878596 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b8012750 con 0x7f25c8068310 2024-01-26T23:45:56.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.592+0000 7f25ce9be700 1 -- 172.21.15.46:0/625878596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c815ab30 con 0x7f25c8068310 2024-01-26T23:45:56.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.592+0000 7f25affff700 1 -- 172.21.15.46:0/625878596 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b8012af0 con 0x7f25c8068310 2024-01-26T23:45:56.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.592+0000 7f25ce9be700 1 -- 172.21.15.46:0/625878596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c815b0f0 con 0x7f25c8068310 2024-01-26T23:45:56.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.593+0000 7f25affff700 1 -- 172.21.15.46:0/625878596 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f25b801e460 con 0x7f25c8068310 2024-01-26T23:45:56.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.593+0000 7f25ce9be700 1 -- 172.21.15.46:0/625878596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c815b460 con 0x7f25c8068310 2024-01-26T23:45:56.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.593+0000 7f25affff700 1 --2- 172.21.15.46:0/625878596 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f25b4066ea0 0x7f25b4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:56.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.594+0000 7f25affff700 1 -- 172.21.15.46:0/625878596 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f25b8084ce0 con 0x7f25c8068310 2024-01-26T23:45:56.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.594+0000 7f25c77fe700 1 --2- 172.21.15.46:0/625878596 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f25b4066ea0 0x7f25b4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:56.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.594+0000 7f25c77fe700 1 --2- 172.21.15.46:0/625878596 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f25b4066ea0 0x7f25b4069350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0099b0 tx=0x7f25bc017000 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:56.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.598+0000 7f25affff700 1 -- 172.21.15.46:0/625878596 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f25b8051d60 con 0x7f25c8068310 2024-01-26T23:45:56.676 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:56.675+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:56.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.807+0000 7f25ce9be700 1 -- 172.21.15.46:0/625878596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25c804c710 con 0x7f25c8068310 2024-01-26T23:45:56.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.808+0000 7f25affff700 1 -- 172.21.15.46:0/625878596 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16152 (secure 0 0 0) 0x7f25b8055380 con 0x7f25c8068310 2024-01-26T23:45:56.808 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:56.809 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":98,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:45:53.989969+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:45:50.922223+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-01-26T23:45:51.397995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:45:56.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.810+0000 7f25ce9be700 1 -- 172.21.15.46:0/625878596 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f25b4066ea0 msgr2=0x7f25b4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.810+0000 7f25ce9be700 1 --2- 172.21.15.46:0/625878596 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f25b4066ea0 0x7f25b4069350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0099b0 tx=0x7f25bc017000 comp rx=0 tx=0).stop 2024-01-26T23:45:56.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.810+0000 7f25ce9be700 1 -- 172.21.15.46:0/625878596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25c8068310 msgr2=0x7f25c80c8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.810+0000 7f25ce9be700 1 --2- 172.21.15.46:0/625878596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25c8068310 0x7f25c80c8c70 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f25b800d890 tx=0x7f25b8008040 comp rx=0 tx=0).stop 2024-01-26T23:45:56.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.810+0000 7f25ce9be700 1 -- 172.21.15.46:0/625878596 shutdown_connections 2024-01-26T23:45:56.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.810+0000 7f25ce9be700 1 --2- 172.21.15.46:0/625878596 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f25b4066ea0 0x7f25b4069350 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.810+0000 7f25ce9be700 1 --2- 172.21.15.46:0/625878596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25c8068310 0x7f25c80c8c70 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.810+0000 7f25ce9be700 1 --2- 172.21.15.46:0/625878596 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25c8156f80 0x7f25c8155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.810+0000 7f25ce9be700 1 --2- 172.21.15.46:0/625878596 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25c80b1fc0 0x7f25c80c5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.810+0000 7f25ce9be700 1 -- 172.21.15.46:0/625878596 >> 172.21.15.46:0/625878596 conn(0x7f25c80ccbc0 msgr2=0x7f25c8153ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:56.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.810+0000 7f25ce9be700 1 -- 172.21.15.46:0/625878596 shutdown_connections 2024-01-26T23:45:56.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.810+0000 7f25ce9be700 1 -- 172.21.15.46:0/625878596 wait complete. 2024-01-26T23:45:56.825 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-26T23:45:56.825 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:45:56.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.960+0000 7ff70bf52700 1 Processor -- start 2024-01-26T23:45:56.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.960+0000 7ff70bf52700 1 -- start start 2024-01-26T23:45:56.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.961+0000 7ff70bf52700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff70406ab20 0x7ff7040b0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:56.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.961+0000 7ff70bf52700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7041597b0 con 0x7ff70414ffb0 2024-01-26T23:45:56.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.961+0000 7ff70bf52700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70415a330 con 0x7ff704150380 2024-01-26T23:45:56.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.961+0000 7ff70bf52700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff70415a7c0 con 0x7ff70406ab20 2024-01-26T23:45:56.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.961+0000 7ff7094ed700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff70406ab20 0x7ff7040b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:56.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.961+0000 7ff7094ed700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff70406ab20 0x7ff7040b0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33442/0 (socket says 172.21.15.46:33442) 2024-01-26T23:45:56.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.961+0000 7ff7094ed700 1 -- 172.21.15.46:0/1904142918 learned_addr learned my addr 172.21.15.46:0/1904142918 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:56.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.961+0000 7ff709cee700 1 --1- 172.21.15.46:0/1904142918 >> v1:172.21.15.46:6789/0 conn(0x7ff70414ffb0 0x7ff70406a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52144/0 (socket says 172.21.15.46:52144) 2024-01-26T23:45:56.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.962+0000 7ff708cec700 1 -- 172.21.15.46:0/1904142918 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22059962 0 0) 0x7ff70415a330 con 0x7ff704150380 2024-01-26T23:45:56.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.962+0000 7ff7094ed700 1 -- 172.21.15.46:0/1904142918 >> v1:172.21.15.106:6789/0 conn(0x7ff704150380 legacy=0x7ff7040b0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.962+0000 7ff7094ed700 1 -- 172.21.15.46:0/1904142918 >> v1:172.21.15.46:6789/0 conn(0x7ff70414ffb0 legacy=0x7ff70406a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.962+0000 7ff7094ed700 1 -- 172.21.15.46:0/1904142918 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff704159a00 con 0x7ff70406ab20 2024-01-26T23:45:56.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.962+0000 7ff7094ed700 1 --2- 172.21.15.46:0/1904142918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70406ab20 0x7ff7040b0890 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7ff70000bc50 tx=0x7ff70000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959e07299d5ab382 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:56.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.962+0000 7ff708cec700 1 -- 172.21.15.46:0/1904142918 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70000be60 con 0x7ff70406ab20 2024-01-26T23:45:56.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.962+0000 7ff708cec700 1 -- 172.21.15.46:0/1904142918 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff70000dd00 con 0x7ff70406ab20 2024-01-26T23:45:56.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.962+0000 7ff708cec700 1 -- 172.21.15.46:0/1904142918 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7000106c0 con 0x7ff70406ab20 2024-01-26T23:45:56.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.963+0000 7ff70bf52700 1 -- 172.21.15.46:0/1904142918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70406ab20 msgr2=0x7ff7040b0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.963+0000 7ff70bf52700 1 --2- 172.21.15.46:0/1904142918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70406ab20 0x7ff7040b0890 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7ff70000bc50 tx=0x7ff70000b4f0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.963+0000 7ff70bf52700 1 -- 172.21.15.46:0/1904142918 shutdown_connections 2024-01-26T23:45:56.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.963+0000 7ff70bf52700 1 --2- 172.21.15.46:0/1904142918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70406ab20 0x7ff7040b0890 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.963+0000 7ff70bf52700 1 -- 172.21.15.46:0/1904142918 >> 172.21.15.46:0/1904142918 conn(0x7ff7040ccbc0 msgr2=0x7ff7040ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:56.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.963+0000 7ff70bf52700 1 -- 172.21.15.46:0/1904142918 shutdown_connections 2024-01-26T23:45:56.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.963+0000 7ff70bf52700 1 -- 172.21.15.46:0/1904142918 wait complete. 2024-01-26T23:45:56.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.964+0000 7ff70bf52700 1 Processor -- start 2024-01-26T23:45:56.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.965+0000 7ff70bf52700 1 -- start start 2024-01-26T23:45:56.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.965+0000 7ff70bf52700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70406ab20 0x7ff7041415d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:56.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.965+0000 7ff70bf52700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff70414ffb0 0x7ff70416a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:56.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff70bf52700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff704150380 0x7ff704146520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:56.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff70bf52700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70416a7a0 con 0x7ff70406ab20 2024-01-26T23:45:56.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff70bf52700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff704146a60 con 0x7ff704150380 2024-01-26T23:45:56.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff70bf52700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff70416ce10 con 0x7ff70414ffb0 2024-01-26T23:45:56.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff709cee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70406ab20 0x7ff7041415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:56.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff709cee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70406ab20 0x7ff7041415d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33454/0 (socket says 172.21.15.46:33454) 2024-01-26T23:45:56.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff709cee700 1 -- 172.21.15.46:0/3279527080 learned_addr learned my addr 172.21.15.46:0/3279527080 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:56.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff7094ed700 1 --2- 172.21.15.46:0/3279527080 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff70414ffb0 0x7ff70416a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff70a4ef700 1 --2- 172.21.15.46:0/3279527080 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff704150380 0x7ff704146520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff709cee700 1 -- 172.21.15.46:0/3279527080 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff70414ffb0 msgr2=0x7ff70416a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff709cee700 1 --2- 172.21.15.46:0/3279527080 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff70414ffb0 0x7ff70416a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff709cee700 1 -- 172.21.15.46:0/3279527080 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff704150380 msgr2=0x7ff704146520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff709cee700 1 --2- 172.21.15.46:0/3279527080 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff704150380 0x7ff704146520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff709cee700 1 -- 172.21.15.46:0/3279527080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70000b790 con 0x7ff70406ab20 2024-01-26T23:45:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff7094ed700 1 --2- 172.21.15.46:0/3279527080 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff70414ffb0 0x7ff70416a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff70a4ef700 1 --2- 172.21.15.46:0/3279527080 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff704150380 0x7ff704146520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff709cee700 1 --2- 172.21.15.46:0/3279527080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70406ab20 0x7ff7041415d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800c810 tx=0x7ff6f8007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff6f5ffb700 1 -- 172.21.15.46:0/3279527080 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8007300 con 0x7ff70406ab20 2024-01-26T23:45:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff6f5ffb700 1 -- 172.21.15.46:0/3279527080 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f8007480 con 0x7ff70406ab20 2024-01-26T23:45:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff70bf52700 1 -- 172.21.15.46:0/3279527080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff70416d0f0 con 0x7ff70406ab20 2024-01-26T23:45:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.966+0000 7ff6f5ffb700 1 -- 172.21.15.46:0/3279527080 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8004830 con 0x7ff70406ab20 2024-01-26T23:45:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.967+0000 7ff70bf52700 1 -- 172.21.15.46:0/3279527080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff70416d6b0 con 0x7ff70406ab20 2024-01-26T23:45:56.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.968+0000 7ff70bf52700 1 -- 172.21.15.46:0/3279527080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff70404c710 con 0x7ff70406ab20 2024-01-26T23:45:56.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.968+0000 7ff6f5ffb700 1 -- 172.21.15.46:0/3279527080 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7ff6f8004ba0 con 0x7ff70406ab20 2024-01-26T23:45:56.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.968+0000 7ff6f5ffb700 1 --2- 172.21.15.46:0/3279527080 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff6e8066ea0 0x7ff6e8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:56.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.968+0000 7ff6f5ffb700 1 -- 172.21.15.46:0/3279527080 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6417+0+0 (secure 0 0 0) 0x7ff6f8083c90 con 0x7ff70406ab20 2024-01-26T23:45:56.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.968+0000 7ff7094ed700 1 --2- 172.21.15.46:0/3279527080 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff6e8066ea0 0x7ff6e8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:56.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.969+0000 7ff7094ed700 1 --2- 172.21.15.46:0/3279527080 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff6e8066ea0 0x7ff6e8069350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff70000bc50 tx=0x7ff70001a000 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:56.973+0000 7ff6f5ffb700 1 -- 172.21.15.46:0/3279527080 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff6f8089020 con 0x7ff70406ab20 2024-01-26T23:45:57.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.178+0000 7ff70bf52700 1 -- 172.21.15.46:0/3279527080 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff704067b70 con 0x7ff6e8066ea0 2024-01-26T23:45:57.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.187+0000 7ff6f5ffb700 1 -- 172.21.15.46:0/3279527080 <== mgr.6421 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281834 (secure 0 0 0) 0x7ff704067b70 con 0x7ff6e8066ea0 2024-01-26T23:45:57.191 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:57.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.191+0000 7ff70bf52700 1 -- 172.21.15.46:0/3279527080 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff6e8066ea0 msgr2=0x7ff6e8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:57.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.191+0000 7ff70bf52700 1 --2- 172.21.15.46:0/3279527080 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff6e8066ea0 0x7ff6e8069350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff70000bc50 tx=0x7ff70001a000 comp rx=0 tx=0).stop 2024-01-26T23:45:57.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.191+0000 7ff70bf52700 1 -- 172.21.15.46:0/3279527080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70406ab20 msgr2=0x7ff7041415d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:57.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.191+0000 7ff70bf52700 1 --2- 172.21.15.46:0/3279527080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70406ab20 0x7ff7041415d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800c810 tx=0x7ff6f8007f30 comp rx=0 tx=0).stop 2024-01-26T23:45:57.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.191+0000 7ff70bf52700 1 -- 172.21.15.46:0/3279527080 shutdown_connections 2024-01-26T23:45:57.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.191+0000 7ff70bf52700 1 --2- 172.21.15.46:0/3279527080 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff6e8066ea0 0x7ff6e8069350 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.191+0000 7ff70bf52700 1 --2- 172.21.15.46:0/3279527080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70406ab20 0x7ff7041415d0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.191+0000 7ff70bf52700 1 --2- 172.21.15.46:0/3279527080 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff704150380 0x7ff704146520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.191+0000 7ff70bf52700 1 --2- 172.21.15.46:0/3279527080 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff70414ffb0 0x7ff70416a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.191+0000 7ff70bf52700 1 -- 172.21.15.46:0/3279527080 >> 172.21.15.46:0/3279527080 conn(0x7ff7040ccbc0 msgr2=0x7ff7040ad810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:57.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.191+0000 7ff70bf52700 1 -- 172.21.15.46:0/3279527080 shutdown_connections 2024-01-26T23:45:57.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.191+0000 7ff70bf52700 1 -- 172.21.15.46:0/3279527080 wait complete. 2024-01-26T23:45:57.194 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:45:57.205 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-26T23:45:56.860876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":585,"num_read_kb":1124,"num_write":265,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1509,"ondisk_log_size":1509,"up":106,"acting":106,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2437784,"kb_used_data":10328,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752536936,"statfs":{"total":773094113280,"available":770597822464,"internally_reserved":0,"allocated":10575872,"data_stored":5911028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.883382"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.009542+0000","last_change":"2024-01-26T23:45:52.998874+0000","last_active":"2024-01-26T23:45:53.009542+0000","last_peered":"2024-01-26T23:45:53.009542+0000","last_clean":"2024-01-26T23:45:53.009542+0000","last_became_active":"2024-01-26T23:45:52.998533+0000","last_became_peered":"2024-01-26T23:45:52.998533+0000","last_unstale":"2024-01-26T23:45:53.009542+0000","last_undegraded":"2024-01-26T23:45:53.009542+0000","last_fullsized":"2024-01-26T23:45:53.009542+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:22:33.448776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.006656+0000","last_change":"2024-01-26T23:45:53.004780+0000","last_active":"2024-01-26T23:45:53.006656+0000","last_peered":"2024-01-26T23:45:53.006656+0000","last_clean":"2024-01-26T23:45:53.006656+0000","last_became_active":"2024-01-26T23:45:53.004486+0000","last_became_peered":"2024-01-26T23:45:53.004486+0000","last_unstale":"2024-01-26T23:45:53.006656+0000","last_undegraded":"2024-01-26T23:45:53.006656+0000","last_fullsized":"2024-01-26T23:45:53.006656+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:52:16.515887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.013956+0000","last_change":"2024-01-26T23:45:53.010404+0000","last_active":"2024-01-26T23:45:53.013956+0000","last_peered":"2024-01-26T23:45:53.013956+0000","last_clean":"2024-01-26T23:45:53.013956+0000","last_became_active":"2024-01-26T23:45:53.010120+0000","last_became_peered":"2024-01-26T23:45:53.010120+0000","last_unstale":"2024-01-26T23:45:53.013956+0000","last_undegraded":"2024-01-26T23:45:53.013956+0000","last_fullsized":"2024-01-26T23:45:53.013956+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:58:27.010243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001869+0000","last_change":"2024-01-26T23:45:52.993487+0000","last_active":"2024-01-26T23:45:53.001869+0000","last_peered":"2024-01-26T23:45:53.001869+0000","last_clean":"2024-01-26T23:45:53.001869+0000","last_became_active":"2024-01-26T23:45:52.993157+0000","last_became_peered":"2024-01-26T23:45:52.993157+0000","last_unstale":"2024-01-26T23:45:53.001869+0000","last_undegraded":"2024-01-26T23:45:53.001869+0000","last_fullsized":"2024-01-26T23:45:53.001869+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:44:24.788145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002692+0000","last_change":"2024-01-26T23:45:52.995031+0000","last_active":"2024-01-26T23:45:53.002692+0000","last_peered":"2024-01-26T23:45:53.002692+0000","last_clean":"2024-01-26T23:45:53.002692+0000","last_became_active":"2024-01-26T23:45:52.994711+0000","last_became_peered":"2024-01-26T23:45:52.994711+0000","last_unstale":"2024-01-26T23:45:53.002692+0000","last_undegraded":"2024-01-26T23:45:53.002692+0000","last_fullsized":"2024-01-26T23:45:53.002692+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:20.934227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010618+0000","last_change":"2024-01-26T23:45:53.002633+0000","last_active":"2024-01-26T23:45:53.010618+0000","last_peered":"2024-01-26T23:45:53.010618+0000","last_clean":"2024-01-26T23:45:53.010618+0000","last_became_active":"2024-01-26T23:45:53.002349+0000","last_became_peered":"2024-01-26T23:45:53.002349+0000","last_unstale":"2024-01-26T23:45:53.010618+0000","last_undegraded":"2024-01-26T23:45:53.010618+0000","last_fullsized":"2024-01-26T23:45:53.010618+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:48:28.528692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.011887+0000","last_change":"2024-01-26T23:45:53.006070+0000","last_active":"2024-01-26T23:45:53.011887+0000","last_peered":"2024-01-26T23:45:53.011887+0000","last_clean":"2024-01-26T23:45:53.011887+0000","last_became_active":"2024-01-26T23:45:53.005736+0000","last_became_peered":"2024-01-26T23:45:53.005736+0000","last_unstale":"2024-01-26T23:45:53.011887+0000","last_undegraded":"2024-01-26T23:45:53.011887+0000","last_fullsized":"2024-01-26T23:45:53.011887+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:57:13.832289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001218+0000","last_change":"2024-01-26T23:45:52.994893+0000","last_active":"2024-01-26T23:45:53.001218+0000","last_peered":"2024-01-26T23:45:53.001218+0000","last_clean":"2024-01-26T23:45:53.001218+0000","last_became_active":"2024-01-26T23:45:52.994613+0000","last_became_peered":"2024-01-26T23:45:52.994613+0000","last_unstale":"2024-01-26T23:45:53.001218+0000","last_undegraded":"2024-01-26T23:45:53.001218+0000","last_fullsized":"2024-01-26T23:45:53.001218+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:09:09.561092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002434+0000","last_change":"2024-01-26T23:45:52.997178+0000","last_active":"2024-01-26T23:45:53.002434+0000","last_peered":"2024-01-26T23:45:53.002434+0000","last_clean":"2024-01-26T23:45:53.002434+0000","last_became_active":"2024-01-26T23:45:52.996868+0000","last_became_peered":"2024-01-26T23:45:52.996868+0000","last_unstale":"2024-01-26T23:45:53.002434+0000","last_undegraded":"2024-01-26T23:45:53.002434+0000","last_fullsized":"2024-01-26T23:45:53.002434+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:07:38.558915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005249+0000","last_change":"2024-01-26T23:45:52.999818+0000","last_active":"2024-01-26T23:45:53.005249+0000","last_peered":"2024-01-26T23:45:53.005249+0000","last_clean":"2024-01-26T23:45:53.005249+0000","last_became_active":"2024-01-26T23:45:52.999503+0000","last_became_peered":"2024-01-26T23:45:52.999503+0000","last_unstale":"2024-01-26T23:45:53.005249+0000","last_undegraded":"2024-01-26T23:45:53.005249+0000","last_fullsized":"2024-01-26T23:45:53.005249+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:47.231142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979235+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:02:21.377733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.009449+0000","last_change":"2024-01-26T23:45:52.995466+0000","last_active":"2024-01-26T23:45:53.009449+0000","last_peered":"2024-01-26T23:45:53.009449+0000","last_clean":"2024-01-26T23:45:53.009449+0000","last_became_active":"2024-01-26T23:45:52.995141+0000","last_became_peered":"2024-01-26T23:45:52.995141+0000","last_unstale":"2024-01-26T23:45:53.009449+0000","last_undegraded":"2024-01-26T23:45:53.009449+0000","last_fullsized":"2024-01-26T23:45:53.009449+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:08.999844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.983371+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:46.226628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.983396+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:07:37.554401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.982175+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:50:39.847921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010005+0000","last_change":"2024-01-26T23:45:53.001042+0000","last_active":"2024-01-26T23:45:53.010005+0000","last_peered":"2024-01-26T23:45:53.010005+0000","last_clean":"2024-01-26T23:45:53.010005+0000","last_became_active":"2024-01-26T23:45:53.000759+0000","last_became_peered":"2024-01-26T23:45:53.000759+0000","last_unstale":"2024-01-26T23:45:53.010005+0000","last_undegraded":"2024-01-26T23:45:53.010005+0000","last_fullsized":"2024-01-26T23:45:53.010005+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:48:01.627096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.981367+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:08:58.845586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.013976+0000","last_change":"2024-01-26T23:45:53.011018+0000","last_active":"2024-01-26T23:45:53.013976+0000","last_peered":"2024-01-26T23:45:53.013976+0000","last_clean":"2024-01-26T23:45:53.013976+0000","last_became_active":"2024-01-26T23:45:53.010726+0000","last_became_peered":"2024-01-26T23:45:53.010726+0000","last_unstale":"2024-01-26T23:45:53.013976+0000","last_undegraded":"2024-01-26T23:45:53.013976+0000","last_fullsized":"2024-01-26T23:45:53.013976+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:50:13.686822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.985553+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:08:58.845586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.983528+0000","last_change":"2024-01-26T23:45:51.995344+0000","last_active":"2024-01-26T23:45:52.983528+0000","last_peered":"2024-01-26T23:45:52.983528+0000","last_clean":"2024-01-26T23:45:52.983528+0000","last_became_active":"2024-01-26T23:45:51.995068+0000","last_became_peered":"2024-01-26T23:45:51.995068+0000","last_unstale":"2024-01-26T23:45:52.983528+0000","last_undegraded":"2024-01-26T23:45:52.983528+0000","last_fullsized":"2024-01-26T23:45:52.983528+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:45:21.389917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979110+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:19:46.301437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010594+0000","last_change":"2024-01-26T23:45:52.999941+0000","last_active":"2024-01-26T23:45:53.010594+0000","last_peered":"2024-01-26T23:45:53.010594+0000","last_clean":"2024-01-26T23:45:53.010594+0000","last_became_active":"2024-01-26T23:45:52.999657+0000","last_became_peered":"2024-01-26T23:45:52.999657+0000","last_unstale":"2024-01-26T23:45:53.010594+0000","last_undegraded":"2024-01-26T23:45:53.010594+0000","last_fullsized":"2024-01-26T23:45:53.010594+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:21:56.682642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.983305+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:24:28.011807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.985296+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:37:57.061471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979863+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:57:41.012865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.006905+0000","last_change":"2024-01-26T23:45:53.004186+0000","last_active":"2024-01-26T23:45:53.006905+0000","last_peered":"2024-01-26T23:45:53.006905+0000","last_clean":"2024-01-26T23:45:53.006905+0000","last_became_active":"2024-01-26T23:45:53.003905+0000","last_became_peered":"2024-01-26T23:45:53.003905+0000","last_unstale":"2024-01-26T23:45:53.006905+0000","last_undegraded":"2024-01-26T23:45:53.006905+0000","last_fullsized":"2024-01-26T23:45:53.006905+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:24:29.016321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.981744+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:26.901131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.985171+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:21:06.318896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.983160+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:19.929713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979710+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:21:55.884844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.10","version":"97'8","reported_seq":18,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.020619+0000","last_change":"2024-01-26T23:45:51.994969+0000","last_active":"2024-01-26T23:45:53.020619+0000","last_peered":"2024-01-26T23:45:53.020619+0000","last_clean":"2024-01-26T23:45:53.020619+0000","last_became_active":"2024-01-26T23:45:51.994689+0000","last_became_peered":"2024-01-26T23:45:51.994689+0000","last_unstale":"2024-01-26T23:45:53.020619+0000","last_undegraded":"2024-01-26T23:45:53.020619+0000","last_fullsized":"2024-01-26T23:45:53.020619+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:01:04.622959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.982869+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:05:31.172516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.981303+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:37:57.061471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979140+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:32:28.510671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.987187+0000","last_change":"2024-01-26T23:45:52.007285+0000","last_active":"2024-01-26T23:45:52.987187+0000","last_peered":"2024-01-26T23:45:52.987187+0000","last_clean":"2024-01-26T23:45:52.987187+0000","last_became_active":"2024-01-26T23:45:52.006952+0000","last_became_peered":"2024-01-26T23:45:52.006952+0000","last_unstale":"2024-01-26T23:45:52.987187+0000","last_undegraded":"2024-01-26T23:45:52.987187+0000","last_fullsized":"2024-01-26T23:45:52.987187+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:50:39.847921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001264+0000","last_change":"2024-01-26T23:45:52.992221+0000","last_active":"2024-01-26T23:45:53.001264+0000","last_peered":"2024-01-26T23:45:53.001264+0000","last_clean":"2024-01-26T23:45:53.001264+0000","last_became_active":"2024-01-26T23:45:52.991944+0000","last_became_peered":"2024-01-26T23:45:52.991944+0000","last_unstale":"2024-01-26T23:45:53.001264+0000","last_undegraded":"2024-01-26T23:45:53.001264+0000","last_fullsized":"2024-01-26T23:45:53.001264+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:28:47.388074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979695+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:59.450936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001363+0000","last_change":"2024-01-26T23:45:52.994059+0000","last_active":"2024-01-26T23:45:53.001363+0000","last_peered":"2024-01-26T23:45:53.001363+0000","last_clean":"2024-01-26T23:45:53.001363+0000","last_became_active":"2024-01-26T23:45:52.993774+0000","last_became_peered":"2024-01-26T23:45:52.993774+0000","last_unstale":"2024-01-26T23:45:53.001363+0000","last_undegraded":"2024-01-26T23:45:53.001363+0000","last_fullsized":"2024-01-26T23:45:53.001363+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:37:33.720336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979861+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:22.235033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979243+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:52:15.511373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.985217+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:42:07.246862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.984063+0000","last_change":"2024-01-26T23:45:52.004361+0000","last_active":"2024-01-26T23:45:52.984063+0000","last_peered":"2024-01-26T23:45:52.984063+0000","last_clean":"2024-01-26T23:45:52.984063+0000","last_became_active":"2024-01-26T23:45:52.004048+0000","last_became_peered":"2024-01-26T23:45:52.004048+0000","last_unstale":"2024-01-26T23:45:52.984063+0000","last_undegraded":"2024-01-26T23:45:52.984063+0000","last_fullsized":"2024-01-26T23:45:52.984063+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:01:43.090741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.982946+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:26:59.702876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.009979+0000","last_change":"2024-01-26T23:45:52.999402+0000","last_active":"2024-01-26T23:45:53.009979+0000","last_peered":"2024-01-26T23:45:53.009979+0000","last_clean":"2024-01-26T23:45:53.009979+0000","last_became_active":"2024-01-26T23:45:52.999111+0000","last_became_peered":"2024-01-26T23:45:52.999111+0000","last_unstale":"2024-01-26T23:45:53.009979+0000","last_undegraded":"2024-01-26T23:45:53.009979+0000","last_fullsized":"2024-01-26T23:45:53.009979+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:15:38.716213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.982814+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:40:36.924823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.980793+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:21.591456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979645+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:02:38.392302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005600+0000","last_change":"2024-01-26T23:45:53.002599+0000","last_active":"2024-01-26T23:45:53.005600+0000","last_peered":"2024-01-26T23:45:53.005600+0000","last_clean":"2024-01-26T23:45:53.005600+0000","last_became_active":"2024-01-26T23:45:53.000819+0000","last_became_peered":"2024-01-26T23:45:53.000819+0000","last_unstale":"2024-01-26T23:45:53.005600+0000","last_undegraded":"2024-01-26T23:45:53.005600+0000","last_fullsized":"2024-01-26T23:45:53.005600+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:06.428462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.981239+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:54:17.875691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005498+0000","last_change":"2024-01-26T23:45:53.002124+0000","last_active":"2024-01-26T23:45:53.005498+0000","last_peered":"2024-01-26T23:45:53.005498+0000","last_clean":"2024-01-26T23:45:53.005498+0000","last_became_active":"2024-01-26T23:45:53.000783+0000","last_became_peered":"2024-01-26T23:45:53.000783+0000","last_unstale":"2024-01-26T23:45:53.005498+0000","last_undegraded":"2024-01-26T23:45:53.005498+0000","last_fullsized":"2024-01-26T23:45:53.005498+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:00:00.455450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.990097+0000","last_change":"2024-01-26T23:45:52.007142+0000","last_active":"2024-01-26T23:45:52.990097+0000","last_peered":"2024-01-26T23:45:52.990097+0000","last_clean":"2024-01-26T23:45:52.990097+0000","last_became_active":"2024-01-26T23:45:52.006791+0000","last_became_peered":"2024-01-26T23:45:52.006791+0000","last_unstale":"2024-01-26T23:45:52.990097+0000","last_undegraded":"2024-01-26T23:45:52.990097+0000","last_fullsized":"2024-01-26T23:45:52.990097+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:44.905097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979123+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:05.423948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"95'1501","reported_seq":1761,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-01-26T23:45:51.854728+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:45:51.854728+0000","last_peered":"2024-01-26T23:45:51.854728+0000","last_clean":"2024-01-26T23:45:51.854728+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:45:51.854728+0000","last_undegraded":"2024-01-26T23:45:51.854728+0000","last_fullsized":"2024-01-26T23:45:51.854728+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":1501,"log_dups_size":0,"ondisk_log_size":1501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":585,"num_read_kb":1124,"num_write":253,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":40,"acting":40,"num_store_stats":2},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":585,"num_read_kb":1124,"num_write":253,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1501,"ondisk_log_size":1501,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575043,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311292,"kb_used_data":1716,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060548,"statfs":{"total":96636764160,"available":96318001152,"internally_reserved":0,"allocated":1757184,"data_stored":1176278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57599999999999996,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.79500000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.314},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.35699999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80400000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.29999999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57699999999999996,"15min":0.57799999999999996},"min":{"1min":0.42499999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77800000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.61099999999999999,"15min":0.59899999999999998},"min":{"1min":0.46000000000000002,"5min":0.39200000000000002,"15min":0.36099999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.45000000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.38800000000000001,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.81699999999999995},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.48699999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77900000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.53000000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59999999999999998,"15min":0.59199999999999997},"min":{"1min":0.442,"5min":0.40500000000000003,"15min":0.35399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60599999999999998,"15min":0.60499999999999998},"min":{"1min":0.40100000000000002,"5min":0.39300000000000002,"15min":0.373},"max":{"1min":0.82699999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.46899999999999997}]},{"osd":4,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46899999999999997,"15min":0.46100000000000002},"min":{"1min":0.36399999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.66100000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.495,"15min":0.48799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.314},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.28499999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.504,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.91700000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.40000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.74199999999999999,"5min":0.90200000000000002,"15min":0.96599999999999997},"last":0.33300000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.57999999999999996,"15min":0.55700000000000005},"min":{"1min":0.46400000000000002,"5min":0.375,"15min":0.33100000000000002},"max":{"1min":0.77900000000000003,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.372},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56200000000000006,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76500000000000001,"5min":0.85199999999999998,"15min":0.86899999999999999},"last":0.434}]}]},{"osd":6,"up_from":13,"seq":55834575043,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302520,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069320,"statfs":{"total":96636764160,"available":96326983680,"internally_reserved":0,"allocated":1163264,"data_stored":585910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56899999999999995,"15min":0.57399999999999995},"min":{"1min":0.35999999999999999,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":0.752,"5min":0.85199999999999998,"15min":0.90200000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59099999999999997,"15min":0.59399999999999997},"min":{"1min":0.42899999999999999,"5min":0.38500000000000001,"15min":0.33600000000000002},"max":{"1min":0.79000000000000004,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.35499999999999998}]},{"osd":1,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.312,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.77100000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.34399999999999997},"max":{"1min":0.75700000000000001,"5min":0.78900000000000003,"15min":0.874},"last":0.40999999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.40799999999999997,"5min":0.35499999999999998,"15min":0.29599999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.85799999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38600000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77000000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.41899999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.59499999999999997,"15min":0.59999999999999998},"min":{"1min":0.45500000000000002,"5min":0.38700000000000001,"15min":0.29999999999999999},"max":{"1min":0.79000000000000004,"5min":0.80700000000000005,"15min":0.92200000000000004},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.622,"15min":0.624},"min":{"1min":0.55000000000000004,"5min":0.39500000000000002,"15min":0.39000000000000001},"max":{"1min":0.78400000000000003,"5min":0.81100000000000005,"15min":0.84499999999999997},"last":0.46500000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.47799999999999998,"15min":0.47399999999999998},"min":{"1min":0.38100000000000001,"5min":0.29899999999999999,"15min":0.22},"max":{"1min":0.69899999999999995,"5min":0.84599999999999997,"15min":0.90200000000000002},"last":0.375},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.501,"15min":0.499},"min":{"1min":0.44700000000000001,"5min":0.32000000000000001,"15min":0.28899999999999998},"max":{"1min":0.61899999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.43099999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48499999999999999,"15min":0.48899999999999999},"min":{"1min":0.42499999999999999,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.53400000000000003,"15min":0.53300000000000003},"min":{"1min":0.46100000000000002,"5min":0.33700000000000002,"15min":0.26200000000000001},"max":{"1min":0.75700000000000001,"5min":0.872,"15min":0.96499999999999997},"last":0.44400000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53500000000000003,"15min":0.53800000000000003},"min":{"1min":0.44400000000000001,"5min":0.32500000000000001,"15min":0.246},"max":{"1min":0.81799999999999995,"5min":0.82799999999999996,"15min":0.97199999999999998},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.308},"max":{"1min":0.73099999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.32200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575044,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302564,"kb_used_data":1180,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069276,"statfs":{"total":96636764160,"available":96326938624,"internally_reserved":0,"allocated":1208320,"data_stored":607472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29899999999999999,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":0.59699999999999998,"5min":0.621,"15min":0.72599999999999998},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.35799999999999998,"5min":0.28899999999999998,"15min":0.26000000000000001},"max":{"1min":0.73499999999999999,"5min":1.337,"15min":1.337},"last":0.29299999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41499999999999998},"min":{"1min":0.27700000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55900000000000005,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48599999999999999,"15min":0.47699999999999998},"min":{"1min":0.376,"5min":0.34699999999999998,"15min":0.28999999999999998},"max":{"1min":0.70799999999999996,"5min":0.77300000000000002,"15min":0.79600000000000004},"last":0.42999999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52200000000000002,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":0.75800000000000001,"5min":1.782,"15min":1.782},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.497,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.625,"5min":1.379,"15min":1.379},"last":0.54300000000000004}]},{"osd":4,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65300000000000002,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":0.86499999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66200000000000003,"15min":0.66600000000000004},"min":{"1min":0.49399999999999999,"5min":0.45900000000000002,"15min":0.433},"max":{"1min":0.89900000000000002,"5min":0.91900000000000004,"15min":0.95699999999999996},"last":0.56699999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.50600000000000001,"5min":0.46800000000000003,"15min":0.42799999999999999},"max":{"1min":0.879,"5min":1.208,"15min":1.208},"last":0.62},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68500000000000005,"15min":0.67100000000000004},"min":{"1min":0.53700000000000003,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":0.81599999999999995,"5min":0.96599999999999997,"15min":1.018},"last":0.48699999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.69399999999999995,"15min":0.68500000000000005},"min":{"1min":0.51800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85199999999999998,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.52900000000000003,"5min":0.437,"15min":0.42599999999999999},"max":{"1min":0.83699999999999997,"5min":1.022,"15min":1.022},"last":0.57799999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71599999999999997},"min":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.49399999999999999},"max":{"1min":0.93200000000000005,"5min":0.98199999999999998,"15min":1.0069999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.749,"15min":0.73199999999999998},"min":{"1min":0.53200000000000003,"5min":0.52200000000000002,"15min":0.438},"max":{"1min":0.90600000000000003,"5min":1.296,"15min":1.3049999999999999},"last":0.59799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575043,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311328,"kb_used_data":1752,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060512,"statfs":{"total":96636764160,"available":96317964288,"internally_reserved":0,"allocated":1794048,"data_stored":1197728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.28499999999999998,"5min":0.23100000000000001,"15min":0.214},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":1.284},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.46000000000000002,"15min":0.45600000000000002},"min":{"1min":0.29099999999999998,"5min":0.28899999999999998,"15min":0.27300000000000002},"max":{"1min":0.60899999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.308}]},{"osd":2,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42099999999999999,"15min":0.42999999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.26700000000000002},"max":{"1min":0.57299999999999995,"5min":0.89600000000000002,"15min":1.2230000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.38300000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":1.2430000000000001},"last":0.42099999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46899999999999997,"15min":0.46700000000000003},"min":{"1min":0.36799999999999999,"5min":0.30599999999999999,"15min":0.26800000000000002},"max":{"1min":0.68100000000000005,"5min":0.80700000000000005,"15min":1.0980000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.49399999999999999,"15min":0.49099999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.27000000000000002},"max":{"1min":0.64000000000000001,"5min":1.198,"15min":1.198},"last":0.436}]},{"osd":4,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.64300000000000002,"15min":0.65200000000000002},"min":{"1min":0.46899999999999997,"5min":0.42399999999999999,"15min":0.372},"max":{"1min":0.78700000000000003,"5min":0.93300000000000005,"15min":1.133},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64700000000000002,"15min":0.65900000000000003},"min":{"1min":0.438,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85299999999999998,"5min":0.94299999999999995,"15min":1.0169999999999999},"last":0.373}]},{"osd":5,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65100000000000002,"15min":0.66300000000000003},"min":{"1min":0.52600000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79800000000000004,"5min":0.95799999999999996,"15min":1.0880000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67400000000000004,"15min":0.68200000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.41699999999999998},"max":{"1min":0.93999999999999995,"5min":1.012,"15min":1.169},"last":0.47399999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65500000000000003,"15min":0.67700000000000005},"min":{"1min":0.505,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89800000000000002,"5min":0.91600000000000004,"15min":1.071},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.68400000000000005,"15min":0.69999999999999996},"min":{"1min":0.48299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.94599999999999995,"5min":1.028,"15min":1.1160000000000001},"last":0.55300000000000005}]},{"osd":7,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":1.004,"15min":0.82599999999999996},"min":{"1min":0.54500000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.91500000000000004,"5min":25.965,"15min":25.965},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.73899999999999999,"5min":1.109,"15min":0.86599999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.433},"max":{"1min":0.93799999999999994,"5min":33.085000000000001,"15min":33.085000000000001},"last":0.56299999999999994}]}]},{"osd":2,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306616,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065224,"statfs":{"total":96636764160,"available":96322789376,"internally_reserved":0,"allocated":1163264,"data_stored":585910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.39400000000000002,"15min":0.41999999999999998},"min":{"1min":0.29299999999999998,"5min":0.25800000000000001,"15min":0.22600000000000001},"max":{"1min":0.502,"5min":0.63400000000000001,"15min":9.0540000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.24299999999999999},"max":{"1min":0.56899999999999995,"5min":0.68500000000000005,"15min":4.1970000000000001},"last":0.54400000000000004}]},{"osd":1,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42699999999999999,"15min":0.42499999999999999},"min":{"1min":0.33300000000000002,"5min":0.248,"15min":0.221},"max":{"1min":0.55300000000000005,"5min":0.70499999999999996,"15min":0.82199999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.498,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":0.629,"5min":0.65900000000000003,"15min":12.157},"last":0.629}]},{"osd":3,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47799999999999998,"15min":0.48599999999999999},"min":{"1min":0.38800000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.59299999999999997,"5min":0.65500000000000003,"15min":0.86499999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.47599999999999998,"15min":0.499},"min":{"1min":0.34599999999999997,"5min":0.32200000000000001,"15min":0.30399999999999999},"max":{"1min":0.60199999999999998,"5min":0.71099999999999997,"15min":9.0879999999999992},"last":0.61299999999999999}]},{"osd":4,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65000000000000002,"15min":0.66600000000000004},"min":{"1min":0.499,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.86599999999999999,"5min":1.087,"15min":9.0150000000000006},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.65800000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.371},"max":{"1min":0.80200000000000005,"5min":0.98099999999999998,"15min":4.2480000000000002},"last":0.874}]},{"osd":5,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78600000000000003,"5min":1.0680000000000001,"15min":4.3200000000000003},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68899999999999995,"15min":0.68200000000000005},"min":{"1min":0.58899999999999997,"5min":0.39800000000000002,"15min":0.35199999999999998},"max":{"1min":0.82399999999999995,"5min":0.95199999999999996,"15min":1.085},"last":0.97699999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68899999999999995,"15min":0.68400000000000005},"min":{"1min":0.51500000000000001,"5min":0.439,"15min":0.40899999999999997},"max":{"1min":0.79400000000000004,"5min":0.995,"15min":1.1080000000000001},"last":1.022},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69199999999999995,"15min":0.71299999999999997},"min":{"1min":0.46200000000000002,"5min":0.443,"15min":0.38800000000000001},"max":{"1min":0.83999999999999997,"5min":1.1299999999999999,"15min":8.9700000000000006},"last":0.82899999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70799999999999996,"15min":0.71699999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.85599999999999998,"5min":1.0029999999999999,"15min":4.282},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.57499999999999996,"5min":0.48199999999999998,"15min":0.42099999999999999},"max":{"1min":0.87,"5min":1.052,"15min":1.139},"last":1.05}]}]},{"osd":3,"up_from":13,"seq":55834575041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306616,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065224,"statfs":{"total":96636764160,"available":96322789376,"internally_reserved":0,"allocated":1163264,"data_stored":585910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40300000000000002,"15min":0.39800000000000002},"min":{"1min":0.32300000000000001,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.56999999999999995,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44900000000000001,"15min":0.48099999999999998},"min":{"1min":0.315,"5min":0.255,"15min":0.255},"max":{"1min":0.59299999999999997,"5min":1.2290000000000001,"15min":11.968},"last":0.504}]},{"osd":1,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.496},"min":{"1min":0.33200000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.623,"5min":1.633,"15min":12.050000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.48899999999999999,"15min":0.48599999999999999},"min":{"1min":0.378,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.59999999999999998,"5min":1.0760000000000001,"15min":1.2150000000000001},"last":0.39400000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.49299999999999999,"15min":0.51900000000000002},"min":{"1min":0.39800000000000002,"5min":0.29699999999999999,"15min":0.28999999999999998},"max":{"1min":0.70399999999999996,"5min":1.4059999999999999,"15min":12.234},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46899999999999997,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.59499999999999997,"5min":1.1870000000000001,"15min":11.925000000000001},"last":0.371}]},{"osd":4,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69599999999999995,"15min":0.70599999999999996},"min":{"1min":0.56999999999999995,"5min":0.46999999999999997,"15min":0.40200000000000002},"max":{"1min":0.84899999999999998,"5min":3.306,"15min":12.099},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.66400000000000003,"15min":0.66600000000000004},"min":{"1min":0.59599999999999997,"5min":0.40200000000000002,"15min":0.379},"max":{"1min":0.82999999999999996,"5min":0.95799999999999996,"15min":1.0649999999999999},"last":0.48599999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.66500000000000004,"15min":0.70699999999999996},"min":{"1min":0.56299999999999994,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.79800000000000004,"5min":1.593,"15min":12.199999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68999999999999995,"15min":0.67800000000000005},"min":{"1min":0.53400000000000003,"5min":0.45300000000000001,"15min":0.436},"max":{"1min":0.92300000000000004,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.53000000000000003}]},{"osd":6,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69499999999999995,"15min":0.72099999999999997},"min":{"1min":0.56100000000000005,"5min":0.442,"15min":0.442},"max":{"1min":0.88800000000000001,"5min":1.506,"15min":12.132999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70099999999999996,"15min":0.69799999999999995},"min":{"1min":0.54600000000000004,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.88200000000000001,"5min":1.427,"15min":1.427},"last":0.58099999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.72699999999999998,"15min":0.70299999999999996},"min":{"1min":0.52500000000000002,"5min":0.48499999999999999,"15min":0.42499999999999999},"max":{"1min":0.86199999999999999,"5min":3.238,"15min":3.238},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.78200000000000003},"min":{"1min":0.625,"5min":0.48899999999999999,"15min":0.48699999999999999},"max":{"1min":0.91300000000000003,"5min":3.351,"15min":12.167},"last":0.628}]}]},{"osd":4,"up_from":13,"seq":55834575043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298424,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073416,"statfs":{"total":96636764160,"available":96331177984,"internally_reserved":0,"allocated":1163264,"data_stored":585910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57599999999999996,"15min":0.58399999999999996},"min":{"1min":0.439,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78300000000000003,"5min":1.21,"15min":1.21},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57499999999999996,"15min":0.58399999999999996},"min":{"1min":0.44900000000000001,"5min":0.309,"15min":0.28100000000000003},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":1.161},"last":0.35299999999999998}]},{"osd":1,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.44900000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":1.069},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.59399999999999997},"min":{"1min":0.47199999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":1.085},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56299999999999994,"15min":0.57499999999999996},"min":{"1min":0.48499999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76500000000000001,"5min":0.81100000000000005,"15min":1.052},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60699999999999998,"15min":0.61199999999999999},"min":{"1min":0.41399999999999998,"5min":0.39900000000000002,"15min":0.373},"max":{"1min":0.81599999999999995,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.54900000000000004}]},{"osd":3,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58199999999999996,"15min":0.59399999999999997},"min":{"1min":0.41299999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.78300000000000003,"5min":0.83499999999999996,"15min":1.1299999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79100000000000004,"5min":1.05,"15min":1.1419999999999999},"last":0.56499999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.317,"5min":0.25700000000000001,"15min":0.247},"max":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.437}]},{"osd":6,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48499999999999999,"15min":0.47799999999999998},"min":{"1min":0.35799999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.73899999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53300000000000003,"15min":0.53000000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.311},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.95699999999999996},"last":0.51100000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.35799999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.57699999999999996},"min":{"1min":0.42499999999999999,"5min":0.41799999999999998,"15min":0.32200000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.879},"last":0.60899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575043,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298424,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073416,"statfs":{"total":96636764160,"available":96331177984,"internally_reserved":0,"allocated":1163264,"data_stored":585910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.58899999999999997,"15min":0.57599999999999996},"min":{"1min":0.46300000000000002,"5min":0.29999999999999999,"15min":0.26000000000000001},"max":{"1min":0.78900000000000003,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.505},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57199999999999995,"15min":0.58399999999999996},"min":{"1min":0.45400000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77400000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.46999999999999997}]},{"osd":1,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.625,"15min":0.623},"min":{"1min":0.45500000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79500000000000004,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.64300000000000002,"15min":0.63300000000000001},"min":{"1min":0.499,"5min":0.435,"15min":0.379},"max":{"1min":0.80500000000000005,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.55800000000000005}]},{"osd":2,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.46700000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77000000000000002,"5min":0.83699999999999997,"15min":0.83799999999999997},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60399999999999998,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81100000000000005,"5min":2.0600000000000001,"15min":6.4589999999999996},"last":0.55000000000000004}]},{"osd":3,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.50700000000000001,"5min":0.35999999999999999,"15min":0.314},"max":{"1min":0.755,"5min":0.86199999999999999,"15min":0.88400000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.60599999999999998,"15min":0.61499999999999999},"min":{"1min":0.47499999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77900000000000003,"5min":2.5630000000000002,"15min":6.0629999999999997},"last":0.53900000000000003}]},{"osd":4,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.47399999999999998,"15min":0.46999999999999997},"min":{"1min":0.35599999999999998,"5min":0.28100000000000003,"15min":0.26300000000000001},"max":{"1min":0.56599999999999995,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.503,"15min":0.495},"min":{"1min":0.36899999999999999,"5min":0.33000000000000002,"15min":0.28100000000000003},"max":{"1min":0.69999999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.375}]},{"osd":6,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.31,"15min":0.27000000000000002},"max":{"1min":0.66600000000000004,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53200000000000003,"15min":0.52700000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.29299999999999998},"max":{"1min":0.71899999999999997,"5min":1.99,"15min":1.99},"last":0.32800000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.54900000000000004},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.76200000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33300000000000002},"max":{"1min":0.81499999999999995,"5min":0.82199999999999995,"15min":0.83099999999999996},"last":0.38400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:45:57.210 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:45:57.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.383+0000 7fa4e738c700 1 Processor -- start 2024-01-26T23:45:57.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.383+0000 7fa4e738c700 1 -- start start 2024-01-26T23:45:57.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.384+0000 7fa4e738c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa4e00acf30 0x7fa4e00c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:57.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.384+0000 7fa4e738c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa4e00ad300 0x7fa4e00ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:57.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.384+0000 7fa4e738c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e00c4230 con 0x7fa4e00ad780 2024-01-26T23:45:57.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.384+0000 7fa4e738c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4e0157be0 con 0x7fa4e00acf30 2024-01-26T23:45:57.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.384+0000 7fa4e738c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4e00cbf10 con 0x7fa4e00ad300 2024-01-26T23:45:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.384+0000 7fa4e5128700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa4e00acf30 0x7fa4e00c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.384+0000 7fa4e5128700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa4e00acf30 0x7fa4e00c3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33462/0 (socket says 172.21.15.46:33462) 2024-01-26T23:45:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.384+0000 7fa4e5929700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa4e00ad780 0x7fa4e015b270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52156/0 (socket says 172.21.15.46:52156) 2024-01-26T23:45:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.384+0000 7fa4e5128700 1 -- 172.21.15.46:0/4089064475 learned_addr learned my addr 172.21.15.46:0/4089064475 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.384+0000 7fa4e4927700 1 --2- 172.21.15.46:0/4089064475 >> v2:172.21.15.106:3300/0 conn(0x7fa4e00ad300 0x7fa4e00ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.385+0000 7fa4e5128700 1 -- 172.21.15.46:0/4089064475 >> v2:172.21.15.106:3300/0 conn(0x7fa4e00ad300 msgr2=0x7fa4e00ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.385+0000 7fa4e5128700 1 --2- 172.21.15.46:0/4089064475 >> v2:172.21.15.106:3300/0 conn(0x7fa4e00ad300 0x7fa4e00ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.385+0000 7fa4e5128700 1 -- 172.21.15.46:0/4089064475 >> v1:172.21.15.46:6789/0 conn(0x7fa4e00ad780 legacy=0x7fa4e015b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.385+0000 7fa4e5128700 1 -- 172.21.15.46:0/4089064475 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e00c4460 con 0x7fa4e00acf30 2024-01-26T23:45:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.385+0000 7fa4dffff700 1 -- 172.21.15.46:0/4089064475 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057150656 0 0) 0x7fa4e00c4230 con 0x7fa4e00ad780 2024-01-26T23:45:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.385+0000 7fa4e4927700 1 --2- 172.21.15.46:0/4089064475 >> v2:172.21.15.106:3300/0 conn(0x7fa4e00ad300 0x7fa4e00ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:45:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.385+0000 7fa4e5128700 1 --2- 172.21.15.46:0/4089064475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e00acf30 0x7fa4e00c3cf0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00b1060 tx=0x7fa4d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810885b9a8e6e405 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.385+0000 7fa4dffff700 1 -- 172.21.15.46:0/4089064475 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d800b860 con 0x7fa4e00acf30 2024-01-26T23:45:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.385+0000 7fa4dffff700 1 -- 172.21.15.46:0/4089064475 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d800b9c0 con 0x7fa4e00acf30 2024-01-26T23:45:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.385+0000 7fa4dffff700 1 -- 172.21.15.46:0/4089064475 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d800bd60 con 0x7fa4e00acf30 2024-01-26T23:45:57.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.386+0000 7fa4e738c700 1 -- 172.21.15.46:0/4089064475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e00acf30 msgr2=0x7fa4e00c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:57.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.386+0000 7fa4e738c700 1 --2- 172.21.15.46:0/4089064475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e00acf30 0x7fa4e00c3cf0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00b1060 tx=0x7fa4d8009250 comp rx=0 tx=0).stop 2024-01-26T23:45:57.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.386+0000 7fa4e738c700 1 -- 172.21.15.46:0/4089064475 shutdown_connections 2024-01-26T23:45:57.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.386+0000 7fa4e738c700 1 --2- 172.21.15.46:0/4089064475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e00acf30 0x7fa4e00c3cf0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.386+0000 7fa4e738c700 1 --2- 172.21.15.46:0/4089064475 >> v2:172.21.15.106:3300/0 conn(0x7fa4e00ad300 0x7fa4e00ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.386+0000 7fa4e738c700 1 -- 172.21.15.46:0/4089064475 >> 172.21.15.46:0/4089064475 conn(0x7fa4e0097820 msgr2=0x7fa4e0068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:57.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.386+0000 7fa4e738c700 1 -- 172.21.15.46:0/4089064475 shutdown_connections 2024-01-26T23:45:57.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.386+0000 7fa4e738c700 1 -- 172.21.15.46:0/4089064475 wait complete. 2024-01-26T23:45:57.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.387+0000 7fa4e738c700 1 Processor -- start 2024-01-26T23:45:57.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.387+0000 7fa4e738c700 1 -- start start 2024-01-26T23:45:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.388+0000 7fa4e738c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4e00acf30 0x7fa4e0154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.388+0000 7fa4e738c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4e00ad300 0x7fa4e0156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.388+0000 7fa4e738c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e00ad780 0x7fa4e0156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.388+0000 7fa4e738c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e00c80c0 con 0x7fa4e00ad780 2024-01-26T23:45:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.388+0000 7fa4e738c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e0156cc0 con 0x7fa4e00acf30 2024-01-26T23:45:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.388+0000 7fa4e738c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e00cae70 con 0x7fa4e00ad300 2024-01-26T23:45:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.388+0000 7fa4e5128700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4e00acf30 0x7fa4e0154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.388+0000 7fa4e4927700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4e00ad300 0x7fa4e0156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.388+0000 7fa4e5128700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4e00acf30 0x7fa4e0154580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:57192/0 (socket says 172.21.15.46:57192) 2024-01-26T23:45:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.388+0000 7fa4e5929700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e00ad780 0x7fa4e0156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.388+0000 7fa4e5128700 1 -- 172.21.15.46:0/88853271 learned_addr learned my addr 172.21.15.46:0/88853271 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.389+0000 7fa4e5929700 1 -- 172.21.15.46:0/88853271 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4e00ad300 msgr2=0x7fa4e0156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.389+0000 7fa4e5929700 1 --2- 172.21.15.46:0/88853271 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4e00ad300 0x7fa4e0156240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.389+0000 7fa4e5929700 1 -- 172.21.15.46:0/88853271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4e00acf30 msgr2=0x7fa4e0154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.389+0000 7fa4e5929700 1 --2- 172.21.15.46:0/88853271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4e00acf30 0x7fa4e0154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.389+0000 7fa4e5929700 1 -- 172.21.15.46:0/88853271 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d80094f0 con 0x7fa4e00ad780 2024-01-26T23:45:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.389+0000 7fa4e5128700 1 --2- 172.21.15.46:0/88853271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4e00acf30 0x7fa4e0154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.389+0000 7fa4e5929700 1 --2- 172.21.15.46:0/88853271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e00ad780 0x7fa4e0156780 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa4d000cbf0 tx=0x7fa4d00019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.389+0000 7fa4dd7fa700 1 -- 172.21.15.46:0/88853271 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d0005df0 con 0x7fa4e00ad780 2024-01-26T23:45:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.389+0000 7fa4dd7fa700 1 -- 172.21.15.46:0/88853271 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d000a6a0 con 0x7fa4e00ad780 2024-01-26T23:45:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.389+0000 7fa4dd7fa700 1 -- 172.21.15.46:0/88853271 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d000aa60 con 0x7fa4e00ad780 2024-01-26T23:45:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.389+0000 7fa4e738c700 1 -- 172.21.15.46:0/88853271 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e00cb150 con 0x7fa4e00ad780 2024-01-26T23:45:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.389+0000 7fa4e738c700 1 -- 172.21.15.46:0/88853271 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e00cab20 con 0x7fa4e00ad780 2024-01-26T23:45:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.390+0000 7fa4e738c700 1 -- 172.21.15.46:0/88853271 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e00cb950 con 0x7fa4e00ad780 2024-01-26T23:45:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.391+0000 7fa4dd7fa700 1 -- 172.21.15.46:0/88853271 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fa4d00070f0 con 0x7fa4e00ad780 2024-01-26T23:45:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.391+0000 7fa4dd7fa700 1 --2- 172.21.15.46:0/88853271 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa4cc066ea0 0x7fa4cc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.391+0000 7fa4dd7fa700 1 -- 172.21.15.46:0/88853271 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fa4d0084600 con 0x7fa4e00ad780 2024-01-26T23:45:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.391+0000 7fa4e5128700 1 --2- 172.21.15.46:0/88853271 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa4cc066ea0 0x7fa4cc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:57.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.392+0000 7fa4e5128700 1 --2- 172.21.15.46:0/88853271 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa4cc066ea0 0x7fa4cc069350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8009290 tx=0x7fa4d8017000 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:57.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.396+0000 7fa4dd7fa700 1 -- 172.21.15.46:0/88853271 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa4e00cb950 con 0x7fa4e00ad780 2024-01-26T23:45:57.397 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:57.396+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:57.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.597+0000 7fa4e738c700 1 -- 172.21.15.46:0/88853271 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa4e00cb950 con 0x7fa4cc066ea0 2024-01-26T23:45:57.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.605+0000 7fa4dd7fa700 1 -- 172.21.15.46:0/88853271 <== mgr.6421 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281834 (secure 0 0 0) 0x7fa4e00cb950 con 0x7fa4cc066ea0 2024-01-26T23:45:57.609 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.609+0000 7fa4e738c700 1 -- 172.21.15.46:0/88853271 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa4cc066ea0 msgr2=0x7fa4cc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.609+0000 7fa4e738c700 1 --2- 172.21.15.46:0/88853271 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa4cc066ea0 0x7fa4cc069350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8009290 tx=0x7fa4d8017000 comp rx=0 tx=0).stop 2024-01-26T23:45:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.609+0000 7fa4e738c700 1 -- 172.21.15.46:0/88853271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e00ad780 msgr2=0x7fa4e0156780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.609+0000 7fa4e738c700 1 --2- 172.21.15.46:0/88853271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e00ad780 0x7fa4e0156780 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa4d000cbf0 tx=0x7fa4d00019b0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.609+0000 7fa4e738c700 1 -- 172.21.15.46:0/88853271 shutdown_connections 2024-01-26T23:45:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.609+0000 7fa4e738c700 1 --2- 172.21.15.46:0/88853271 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa4cc066ea0 0x7fa4cc069350 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.609+0000 7fa4e738c700 1 --2- 172.21.15.46:0/88853271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e00ad780 0x7fa4e0156780 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.609+0000 7fa4e738c700 1 --2- 172.21.15.46:0/88853271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4e00acf30 0x7fa4e0154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.609+0000 7fa4e738c700 1 --2- 172.21.15.46:0/88853271 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4e00ad300 0x7fa4e0156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.609+0000 7fa4e738c700 1 -- 172.21.15.46:0/88853271 >> 172.21.15.46:0/88853271 conn(0x7fa4e0097820 msgr2=0x7fa4e006bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.609+0000 7fa4e738c700 1 -- 172.21.15.46:0/88853271 shutdown_connections 2024-01-26T23:45:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.609+0000 7fa4e738c700 1 -- 172.21.15.46:0/88853271 wait complete. 2024-01-26T23:45:57.613 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:45:57.623 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-26T23:45:56.860876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":585,"num_read_kb":1124,"num_write":265,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1509,"ondisk_log_size":1509,"up":106,"acting":106,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2437784,"kb_used_data":10328,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752536936,"statfs":{"total":773094113280,"available":770597822464,"internally_reserved":0,"allocated":10575872,"data_stored":5911028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.883382"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.009542+0000","last_change":"2024-01-26T23:45:52.998874+0000","last_active":"2024-01-26T23:45:53.009542+0000","last_peered":"2024-01-26T23:45:53.009542+0000","last_clean":"2024-01-26T23:45:53.009542+0000","last_became_active":"2024-01-26T23:45:52.998533+0000","last_became_peered":"2024-01-26T23:45:52.998533+0000","last_unstale":"2024-01-26T23:45:53.009542+0000","last_undegraded":"2024-01-26T23:45:53.009542+0000","last_fullsized":"2024-01-26T23:45:53.009542+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:22:33.448776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.006656+0000","last_change":"2024-01-26T23:45:53.004780+0000","last_active":"2024-01-26T23:45:53.006656+0000","last_peered":"2024-01-26T23:45:53.006656+0000","last_clean":"2024-01-26T23:45:53.006656+0000","last_became_active":"2024-01-26T23:45:53.004486+0000","last_became_peered":"2024-01-26T23:45:53.004486+0000","last_unstale":"2024-01-26T23:45:53.006656+0000","last_undegraded":"2024-01-26T23:45:53.006656+0000","last_fullsized":"2024-01-26T23:45:53.006656+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:52:16.515887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.013956+0000","last_change":"2024-01-26T23:45:53.010404+0000","last_active":"2024-01-26T23:45:53.013956+0000","last_peered":"2024-01-26T23:45:53.013956+0000","last_clean":"2024-01-26T23:45:53.013956+0000","last_became_active":"2024-01-26T23:45:53.010120+0000","last_became_peered":"2024-01-26T23:45:53.010120+0000","last_unstale":"2024-01-26T23:45:53.013956+0000","last_undegraded":"2024-01-26T23:45:53.013956+0000","last_fullsized":"2024-01-26T23:45:53.013956+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:58:27.010243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001869+0000","last_change":"2024-01-26T23:45:52.993487+0000","last_active":"2024-01-26T23:45:53.001869+0000","last_peered":"2024-01-26T23:45:53.001869+0000","last_clean":"2024-01-26T23:45:53.001869+0000","last_became_active":"2024-01-26T23:45:52.993157+0000","last_became_peered":"2024-01-26T23:45:52.993157+0000","last_unstale":"2024-01-26T23:45:53.001869+0000","last_undegraded":"2024-01-26T23:45:53.001869+0000","last_fullsized":"2024-01-26T23:45:53.001869+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:44:24.788145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002692+0000","last_change":"2024-01-26T23:45:52.995031+0000","last_active":"2024-01-26T23:45:53.002692+0000","last_peered":"2024-01-26T23:45:53.002692+0000","last_clean":"2024-01-26T23:45:53.002692+0000","last_became_active":"2024-01-26T23:45:52.994711+0000","last_became_peered":"2024-01-26T23:45:52.994711+0000","last_unstale":"2024-01-26T23:45:53.002692+0000","last_undegraded":"2024-01-26T23:45:53.002692+0000","last_fullsized":"2024-01-26T23:45:53.002692+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:20.934227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010618+0000","last_change":"2024-01-26T23:45:53.002633+0000","last_active":"2024-01-26T23:45:53.010618+0000","last_peered":"2024-01-26T23:45:53.010618+0000","last_clean":"2024-01-26T23:45:53.010618+0000","last_became_active":"2024-01-26T23:45:53.002349+0000","last_became_peered":"2024-01-26T23:45:53.002349+0000","last_unstale":"2024-01-26T23:45:53.010618+0000","last_undegraded":"2024-01-26T23:45:53.010618+0000","last_fullsized":"2024-01-26T23:45:53.010618+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:48:28.528692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.011887+0000","last_change":"2024-01-26T23:45:53.006070+0000","last_active":"2024-01-26T23:45:53.011887+0000","last_peered":"2024-01-26T23:45:53.011887+0000","last_clean":"2024-01-26T23:45:53.011887+0000","last_became_active":"2024-01-26T23:45:53.005736+0000","last_became_peered":"2024-01-26T23:45:53.005736+0000","last_unstale":"2024-01-26T23:45:53.011887+0000","last_undegraded":"2024-01-26T23:45:53.011887+0000","last_fullsized":"2024-01-26T23:45:53.011887+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:57:13.832289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001218+0000","last_change":"2024-01-26T23:45:52.994893+0000","last_active":"2024-01-26T23:45:53.001218+0000","last_peered":"2024-01-26T23:45:53.001218+0000","last_clean":"2024-01-26T23:45:53.001218+0000","last_became_active":"2024-01-26T23:45:52.994613+0000","last_became_peered":"2024-01-26T23:45:52.994613+0000","last_unstale":"2024-01-26T23:45:53.001218+0000","last_undegraded":"2024-01-26T23:45:53.001218+0000","last_fullsized":"2024-01-26T23:45:53.001218+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:09:09.561092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002434+0000","last_change":"2024-01-26T23:45:52.997178+0000","last_active":"2024-01-26T23:45:53.002434+0000","last_peered":"2024-01-26T23:45:53.002434+0000","last_clean":"2024-01-26T23:45:53.002434+0000","last_became_active":"2024-01-26T23:45:52.996868+0000","last_became_peered":"2024-01-26T23:45:52.996868+0000","last_unstale":"2024-01-26T23:45:53.002434+0000","last_undegraded":"2024-01-26T23:45:53.002434+0000","last_fullsized":"2024-01-26T23:45:53.002434+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:07:38.558915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005249+0000","last_change":"2024-01-26T23:45:52.999818+0000","last_active":"2024-01-26T23:45:53.005249+0000","last_peered":"2024-01-26T23:45:53.005249+0000","last_clean":"2024-01-26T23:45:53.005249+0000","last_became_active":"2024-01-26T23:45:52.999503+0000","last_became_peered":"2024-01-26T23:45:52.999503+0000","last_unstale":"2024-01-26T23:45:53.005249+0000","last_undegraded":"2024-01-26T23:45:53.005249+0000","last_fullsized":"2024-01-26T23:45:53.005249+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:47.231142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979235+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:02:21.377733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.009449+0000","last_change":"2024-01-26T23:45:52.995466+0000","last_active":"2024-01-26T23:45:53.009449+0000","last_peered":"2024-01-26T23:45:53.009449+0000","last_clean":"2024-01-26T23:45:53.009449+0000","last_became_active":"2024-01-26T23:45:52.995141+0000","last_became_peered":"2024-01-26T23:45:52.995141+0000","last_unstale":"2024-01-26T23:45:53.009449+0000","last_undegraded":"2024-01-26T23:45:53.009449+0000","last_fullsized":"2024-01-26T23:45:53.009449+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:08.999844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.983371+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:46.226628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.983396+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:07:37.554401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.982175+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:50:39.847921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010005+0000","last_change":"2024-01-26T23:45:53.001042+0000","last_active":"2024-01-26T23:45:53.010005+0000","last_peered":"2024-01-26T23:45:53.010005+0000","last_clean":"2024-01-26T23:45:53.010005+0000","last_became_active":"2024-01-26T23:45:53.000759+0000","last_became_peered":"2024-01-26T23:45:53.000759+0000","last_unstale":"2024-01-26T23:45:53.010005+0000","last_undegraded":"2024-01-26T23:45:53.010005+0000","last_fullsized":"2024-01-26T23:45:53.010005+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:48:01.627096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.981367+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:08:58.845586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.013976+0000","last_change":"2024-01-26T23:45:53.011018+0000","last_active":"2024-01-26T23:45:53.013976+0000","last_peered":"2024-01-26T23:45:53.013976+0000","last_clean":"2024-01-26T23:45:53.013976+0000","last_became_active":"2024-01-26T23:45:53.010726+0000","last_became_peered":"2024-01-26T23:45:53.010726+0000","last_unstale":"2024-01-26T23:45:53.013976+0000","last_undegraded":"2024-01-26T23:45:53.013976+0000","last_fullsized":"2024-01-26T23:45:53.013976+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:50:13.686822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.985553+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:08:58.845586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.983528+0000","last_change":"2024-01-26T23:45:51.995344+0000","last_active":"2024-01-26T23:45:52.983528+0000","last_peered":"2024-01-26T23:45:52.983528+0000","last_clean":"2024-01-26T23:45:52.983528+0000","last_became_active":"2024-01-26T23:45:51.995068+0000","last_became_peered":"2024-01-26T23:45:51.995068+0000","last_unstale":"2024-01-26T23:45:52.983528+0000","last_undegraded":"2024-01-26T23:45:52.983528+0000","last_fullsized":"2024-01-26T23:45:52.983528+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:45:21.389917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979110+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:19:46.301437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010594+0000","last_change":"2024-01-26T23:45:52.999941+0000","last_active":"2024-01-26T23:45:53.010594+0000","last_peered":"2024-01-26T23:45:53.010594+0000","last_clean":"2024-01-26T23:45:53.010594+0000","last_became_active":"2024-01-26T23:45:52.999657+0000","last_became_peered":"2024-01-26T23:45:52.999657+0000","last_unstale":"2024-01-26T23:45:53.010594+0000","last_undegraded":"2024-01-26T23:45:53.010594+0000","last_fullsized":"2024-01-26T23:45:53.010594+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:21:56.682642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.983305+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:24:28.011807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.985296+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:37:57.061471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979863+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:57:41.012865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.006905+0000","last_change":"2024-01-26T23:45:53.004186+0000","last_active":"2024-01-26T23:45:53.006905+0000","last_peered":"2024-01-26T23:45:53.006905+0000","last_clean":"2024-01-26T23:45:53.006905+0000","last_became_active":"2024-01-26T23:45:53.003905+0000","last_became_peered":"2024-01-26T23:45:53.003905+0000","last_unstale":"2024-01-26T23:45:53.006905+0000","last_undegraded":"2024-01-26T23:45:53.006905+0000","last_fullsized":"2024-01-26T23:45:53.006905+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:24:29.016321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.981744+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:26.901131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.985171+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:21:06.318896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.983160+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:19.929713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979710+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:21:55.884844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.10","version":"97'8","reported_seq":18,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.020619+0000","last_change":"2024-01-26T23:45:51.994969+0000","last_active":"2024-01-26T23:45:53.020619+0000","last_peered":"2024-01-26T23:45:53.020619+0000","last_clean":"2024-01-26T23:45:53.020619+0000","last_became_active":"2024-01-26T23:45:51.994689+0000","last_became_peered":"2024-01-26T23:45:51.994689+0000","last_unstale":"2024-01-26T23:45:53.020619+0000","last_undegraded":"2024-01-26T23:45:53.020619+0000","last_fullsized":"2024-01-26T23:45:53.020619+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:01:04.622959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.982869+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:05:31.172516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.981303+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:37:57.061471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979140+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:32:28.510671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.987187+0000","last_change":"2024-01-26T23:45:52.007285+0000","last_active":"2024-01-26T23:45:52.987187+0000","last_peered":"2024-01-26T23:45:52.987187+0000","last_clean":"2024-01-26T23:45:52.987187+0000","last_became_active":"2024-01-26T23:45:52.006952+0000","last_became_peered":"2024-01-26T23:45:52.006952+0000","last_unstale":"2024-01-26T23:45:52.987187+0000","last_undegraded":"2024-01-26T23:45:52.987187+0000","last_fullsized":"2024-01-26T23:45:52.987187+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:50:39.847921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001264+0000","last_change":"2024-01-26T23:45:52.992221+0000","last_active":"2024-01-26T23:45:53.001264+0000","last_peered":"2024-01-26T23:45:53.001264+0000","last_clean":"2024-01-26T23:45:53.001264+0000","last_became_active":"2024-01-26T23:45:52.991944+0000","last_became_peered":"2024-01-26T23:45:52.991944+0000","last_unstale":"2024-01-26T23:45:53.001264+0000","last_undegraded":"2024-01-26T23:45:53.001264+0000","last_fullsized":"2024-01-26T23:45:53.001264+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:28:47.388074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979695+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:59.450936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001363+0000","last_change":"2024-01-26T23:45:52.994059+0000","last_active":"2024-01-26T23:45:53.001363+0000","last_peered":"2024-01-26T23:45:53.001363+0000","last_clean":"2024-01-26T23:45:53.001363+0000","last_became_active":"2024-01-26T23:45:52.993774+0000","last_became_peered":"2024-01-26T23:45:52.993774+0000","last_unstale":"2024-01-26T23:45:53.001363+0000","last_undegraded":"2024-01-26T23:45:53.001363+0000","last_fullsized":"2024-01-26T23:45:53.001363+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:37:33.720336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979861+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:22.235033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979243+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:52:15.511373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.985217+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:42:07.246862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.984063+0000","last_change":"2024-01-26T23:45:52.004361+0000","last_active":"2024-01-26T23:45:52.984063+0000","last_peered":"2024-01-26T23:45:52.984063+0000","last_clean":"2024-01-26T23:45:52.984063+0000","last_became_active":"2024-01-26T23:45:52.004048+0000","last_became_peered":"2024-01-26T23:45:52.004048+0000","last_unstale":"2024-01-26T23:45:52.984063+0000","last_undegraded":"2024-01-26T23:45:52.984063+0000","last_fullsized":"2024-01-26T23:45:52.984063+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:01:43.090741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.982946+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:26:59.702876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.009979+0000","last_change":"2024-01-26T23:45:52.999402+0000","last_active":"2024-01-26T23:45:53.009979+0000","last_peered":"2024-01-26T23:45:53.009979+0000","last_clean":"2024-01-26T23:45:53.009979+0000","last_became_active":"2024-01-26T23:45:52.999111+0000","last_became_peered":"2024-01-26T23:45:52.999111+0000","last_unstale":"2024-01-26T23:45:53.009979+0000","last_undegraded":"2024-01-26T23:45:53.009979+0000","last_fullsized":"2024-01-26T23:45:53.009979+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:15:38.716213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.982814+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:40:36.924823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.980793+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:21.591456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979645+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:02:38.392302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005600+0000","last_change":"2024-01-26T23:45:53.002599+0000","last_active":"2024-01-26T23:45:53.005600+0000","last_peered":"2024-01-26T23:45:53.005600+0000","last_clean":"2024-01-26T23:45:53.005600+0000","last_became_active":"2024-01-26T23:45:53.000819+0000","last_became_peered":"2024-01-26T23:45:53.000819+0000","last_unstale":"2024-01-26T23:45:53.005600+0000","last_undegraded":"2024-01-26T23:45:53.005600+0000","last_fullsized":"2024-01-26T23:45:53.005600+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:06.428462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.981239+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:54:17.875691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005498+0000","last_change":"2024-01-26T23:45:53.002124+0000","last_active":"2024-01-26T23:45:53.005498+0000","last_peered":"2024-01-26T23:45:53.005498+0000","last_clean":"2024-01-26T23:45:53.005498+0000","last_became_active":"2024-01-26T23:45:53.000783+0000","last_became_peered":"2024-01-26T23:45:53.000783+0000","last_unstale":"2024-01-26T23:45:53.005498+0000","last_undegraded":"2024-01-26T23:45:53.005498+0000","last_fullsized":"2024-01-26T23:45:53.005498+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:00:00.455450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.990097+0000","last_change":"2024-01-26T23:45:52.007142+0000","last_active":"2024-01-26T23:45:52.990097+0000","last_peered":"2024-01-26T23:45:52.990097+0000","last_clean":"2024-01-26T23:45:52.990097+0000","last_became_active":"2024-01-26T23:45:52.006791+0000","last_became_peered":"2024-01-26T23:45:52.006791+0000","last_unstale":"2024-01-26T23:45:52.990097+0000","last_undegraded":"2024-01-26T23:45:52.990097+0000","last_fullsized":"2024-01-26T23:45:52.990097+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:44.905097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979123+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:05.423948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"95'1501","reported_seq":1761,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-01-26T23:45:51.854728+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:45:51.854728+0000","last_peered":"2024-01-26T23:45:51.854728+0000","last_clean":"2024-01-26T23:45:51.854728+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:45:51.854728+0000","last_undegraded":"2024-01-26T23:45:51.854728+0000","last_fullsized":"2024-01-26T23:45:51.854728+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":1501,"log_dups_size":0,"ondisk_log_size":1501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":585,"num_read_kb":1124,"num_write":253,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":40,"acting":40,"num_store_stats":2},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":585,"num_read_kb":1124,"num_write":253,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1501,"ondisk_log_size":1501,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575043,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311292,"kb_used_data":1716,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060548,"statfs":{"total":96636764160,"available":96318001152,"internally_reserved":0,"allocated":1757184,"data_stored":1176278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57599999999999996,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.79500000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.314},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.35699999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80400000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.29999999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57699999999999996,"15min":0.57799999999999996},"min":{"1min":0.42499999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77800000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.61099999999999999,"15min":0.59899999999999998},"min":{"1min":0.46000000000000002,"5min":0.39200000000000002,"15min":0.36099999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.45000000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.38800000000000001,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.81699999999999995},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.48699999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77900000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.53000000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59999999999999998,"15min":0.59199999999999997},"min":{"1min":0.442,"5min":0.40500000000000003,"15min":0.35399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60599999999999998,"15min":0.60499999999999998},"min":{"1min":0.40100000000000002,"5min":0.39300000000000002,"15min":0.373},"max":{"1min":0.82699999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.46899999999999997}]},{"osd":4,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46899999999999997,"15min":0.46100000000000002},"min":{"1min":0.36399999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.66100000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.495,"15min":0.48799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.314},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.28499999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.504,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.91700000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.40000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.74199999999999999,"5min":0.90200000000000002,"15min":0.96599999999999997},"last":0.33300000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.57999999999999996,"15min":0.55700000000000005},"min":{"1min":0.46400000000000002,"5min":0.375,"15min":0.33100000000000002},"max":{"1min":0.77900000000000003,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.372},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56200000000000006,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76500000000000001,"5min":0.85199999999999998,"15min":0.86899999999999999},"last":0.434}]}]},{"osd":6,"up_from":13,"seq":55834575043,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302520,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069320,"statfs":{"total":96636764160,"available":96326983680,"internally_reserved":0,"allocated":1163264,"data_stored":585910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56899999999999995,"15min":0.57399999999999995},"min":{"1min":0.35999999999999999,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":0.752,"5min":0.85199999999999998,"15min":0.90200000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59099999999999997,"15min":0.59399999999999997},"min":{"1min":0.42899999999999999,"5min":0.38500000000000001,"15min":0.33600000000000002},"max":{"1min":0.79000000000000004,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.35499999999999998}]},{"osd":1,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.312,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.77100000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.34399999999999997},"max":{"1min":0.75700000000000001,"5min":0.78900000000000003,"15min":0.874},"last":0.40999999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.40799999999999997,"5min":0.35499999999999998,"15min":0.29599999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.85799999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38600000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77000000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.41899999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.59499999999999997,"15min":0.59999999999999998},"min":{"1min":0.45500000000000002,"5min":0.38700000000000001,"15min":0.29999999999999999},"max":{"1min":0.79000000000000004,"5min":0.80700000000000005,"15min":0.92200000000000004},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.622,"15min":0.624},"min":{"1min":0.55000000000000004,"5min":0.39500000000000002,"15min":0.39000000000000001},"max":{"1min":0.78400000000000003,"5min":0.81100000000000005,"15min":0.84499999999999997},"last":0.46500000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.47799999999999998,"15min":0.47399999999999998},"min":{"1min":0.38100000000000001,"5min":0.29899999999999999,"15min":0.22},"max":{"1min":0.69899999999999995,"5min":0.84599999999999997,"15min":0.90200000000000002},"last":0.375},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.501,"15min":0.499},"min":{"1min":0.44700000000000001,"5min":0.32000000000000001,"15min":0.28899999999999998},"max":{"1min":0.61899999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.43099999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48499999999999999,"15min":0.48899999999999999},"min":{"1min":0.42499999999999999,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.53400000000000003,"15min":0.53300000000000003},"min":{"1min":0.46100000000000002,"5min":0.33700000000000002,"15min":0.26200000000000001},"max":{"1min":0.75700000000000001,"5min":0.872,"15min":0.96499999999999997},"last":0.44400000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53500000000000003,"15min":0.53800000000000003},"min":{"1min":0.44400000000000001,"5min":0.32500000000000001,"15min":0.246},"max":{"1min":0.81799999999999995,"5min":0.82799999999999996,"15min":0.97199999999999998},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.308},"max":{"1min":0.73099999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.32200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575044,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302564,"kb_used_data":1180,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069276,"statfs":{"total":96636764160,"available":96326938624,"internally_reserved":0,"allocated":1208320,"data_stored":607472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29899999999999999,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":0.59699999999999998,"5min":0.621,"15min":0.72599999999999998},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.35799999999999998,"5min":0.28899999999999998,"15min":0.26000000000000001},"max":{"1min":0.73499999999999999,"5min":1.337,"15min":1.337},"last":0.29299999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41499999999999998},"min":{"1min":0.27700000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55900000000000005,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48599999999999999,"15min":0.47699999999999998},"min":{"1min":0.376,"5min":0.34699999999999998,"15min":0.28999999999999998},"max":{"1min":0.70799999999999996,"5min":0.77300000000000002,"15min":0.79600000000000004},"last":0.42999999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52200000000000002,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":0.75800000000000001,"5min":1.782,"15min":1.782},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.497,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.625,"5min":1.379,"15min":1.379},"last":0.54300000000000004}]},{"osd":4,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65300000000000002,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":0.86499999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66200000000000003,"15min":0.66600000000000004},"min":{"1min":0.49399999999999999,"5min":0.45900000000000002,"15min":0.433},"max":{"1min":0.89900000000000002,"5min":0.91900000000000004,"15min":0.95699999999999996},"last":0.56699999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.50600000000000001,"5min":0.46800000000000003,"15min":0.42799999999999999},"max":{"1min":0.879,"5min":1.208,"15min":1.208},"last":0.62},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68500000000000005,"15min":0.67100000000000004},"min":{"1min":0.53700000000000003,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":0.81599999999999995,"5min":0.96599999999999997,"15min":1.018},"last":0.48699999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.69399999999999995,"15min":0.68500000000000005},"min":{"1min":0.51800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85199999999999998,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.52900000000000003,"5min":0.437,"15min":0.42599999999999999},"max":{"1min":0.83699999999999997,"5min":1.022,"15min":1.022},"last":0.57799999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71599999999999997},"min":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.49399999999999999},"max":{"1min":0.93200000000000005,"5min":0.98199999999999998,"15min":1.0069999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.749,"15min":0.73199999999999998},"min":{"1min":0.53200000000000003,"5min":0.52200000000000002,"15min":0.438},"max":{"1min":0.90600000000000003,"5min":1.296,"15min":1.3049999999999999},"last":0.59799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575043,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311328,"kb_used_data":1752,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060512,"statfs":{"total":96636764160,"available":96317964288,"internally_reserved":0,"allocated":1794048,"data_stored":1197728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.28499999999999998,"5min":0.23100000000000001,"15min":0.214},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":1.284},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.46000000000000002,"15min":0.45600000000000002},"min":{"1min":0.29099999999999998,"5min":0.28899999999999998,"15min":0.27300000000000002},"max":{"1min":0.60899999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.308}]},{"osd":2,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42099999999999999,"15min":0.42999999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.26700000000000002},"max":{"1min":0.57299999999999995,"5min":0.89600000000000002,"15min":1.2230000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.38300000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":1.2430000000000001},"last":0.42099999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46899999999999997,"15min":0.46700000000000003},"min":{"1min":0.36799999999999999,"5min":0.30599999999999999,"15min":0.26800000000000002},"max":{"1min":0.68100000000000005,"5min":0.80700000000000005,"15min":1.0980000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.49399999999999999,"15min":0.49099999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.27000000000000002},"max":{"1min":0.64000000000000001,"5min":1.198,"15min":1.198},"last":0.436}]},{"osd":4,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.64300000000000002,"15min":0.65200000000000002},"min":{"1min":0.46899999999999997,"5min":0.42399999999999999,"15min":0.372},"max":{"1min":0.78700000000000003,"5min":0.93300000000000005,"15min":1.133},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64700000000000002,"15min":0.65900000000000003},"min":{"1min":0.438,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85299999999999998,"5min":0.94299999999999995,"15min":1.0169999999999999},"last":0.373}]},{"osd":5,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65100000000000002,"15min":0.66300000000000003},"min":{"1min":0.52600000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79800000000000004,"5min":0.95799999999999996,"15min":1.0880000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67400000000000004,"15min":0.68200000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.41699999999999998},"max":{"1min":0.93999999999999995,"5min":1.012,"15min":1.169},"last":0.47399999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65500000000000003,"15min":0.67700000000000005},"min":{"1min":0.505,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89800000000000002,"5min":0.91600000000000004,"15min":1.071},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.68400000000000005,"15min":0.69999999999999996},"min":{"1min":0.48299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.94599999999999995,"5min":1.028,"15min":1.1160000000000001},"last":0.55300000000000005}]},{"osd":7,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":1.004,"15min":0.82599999999999996},"min":{"1min":0.54500000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.91500000000000004,"5min":25.965,"15min":25.965},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.73899999999999999,"5min":1.109,"15min":0.86599999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.433},"max":{"1min":0.93799999999999994,"5min":33.085000000000001,"15min":33.085000000000001},"last":0.56299999999999994}]}]},{"osd":2,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306616,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065224,"statfs":{"total":96636764160,"available":96322789376,"internally_reserved":0,"allocated":1163264,"data_stored":585910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.39400000000000002,"15min":0.41999999999999998},"min":{"1min":0.29299999999999998,"5min":0.25800000000000001,"15min":0.22600000000000001},"max":{"1min":0.502,"5min":0.63400000000000001,"15min":9.0540000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.24299999999999999},"max":{"1min":0.56899999999999995,"5min":0.68500000000000005,"15min":4.1970000000000001},"last":0.54400000000000004}]},{"osd":1,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42699999999999999,"15min":0.42499999999999999},"min":{"1min":0.33300000000000002,"5min":0.248,"15min":0.221},"max":{"1min":0.55300000000000005,"5min":0.70499999999999996,"15min":0.82199999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.498,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":0.629,"5min":0.65900000000000003,"15min":12.157},"last":0.629}]},{"osd":3,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47799999999999998,"15min":0.48599999999999999},"min":{"1min":0.38800000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.59299999999999997,"5min":0.65500000000000003,"15min":0.86499999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.47599999999999998,"15min":0.499},"min":{"1min":0.34599999999999997,"5min":0.32200000000000001,"15min":0.30399999999999999},"max":{"1min":0.60199999999999998,"5min":0.71099999999999997,"15min":9.0879999999999992},"last":0.61299999999999999}]},{"osd":4,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65000000000000002,"15min":0.66600000000000004},"min":{"1min":0.499,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.86599999999999999,"5min":1.087,"15min":9.0150000000000006},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.65800000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.371},"max":{"1min":0.80200000000000005,"5min":0.98099999999999998,"15min":4.2480000000000002},"last":0.874}]},{"osd":5,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78600000000000003,"5min":1.0680000000000001,"15min":4.3200000000000003},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68899999999999995,"15min":0.68200000000000005},"min":{"1min":0.58899999999999997,"5min":0.39800000000000002,"15min":0.35199999999999998},"max":{"1min":0.82399999999999995,"5min":0.95199999999999996,"15min":1.085},"last":0.97699999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68899999999999995,"15min":0.68400000000000005},"min":{"1min":0.51500000000000001,"5min":0.439,"15min":0.40899999999999997},"max":{"1min":0.79400000000000004,"5min":0.995,"15min":1.1080000000000001},"last":1.022},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69199999999999995,"15min":0.71299999999999997},"min":{"1min":0.46200000000000002,"5min":0.443,"15min":0.38800000000000001},"max":{"1min":0.83999999999999997,"5min":1.1299999999999999,"15min":8.9700000000000006},"last":0.82899999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70799999999999996,"15min":0.71699999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.85599999999999998,"5min":1.0029999999999999,"15min":4.282},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.57499999999999996,"5min":0.48199999999999998,"15min":0.42099999999999999},"max":{"1min":0.87,"5min":1.052,"15min":1.139},"last":1.05}]}]},{"osd":3,"up_from":13,"seq":55834575041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306616,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065224,"statfs":{"total":96636764160,"available":96322789376,"internally_reserved":0,"allocated":1163264,"data_stored":585910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40300000000000002,"15min":0.39800000000000002},"min":{"1min":0.32300000000000001,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.56999999999999995,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44900000000000001,"15min":0.48099999999999998},"min":{"1min":0.315,"5min":0.255,"15min":0.255},"max":{"1min":0.59299999999999997,"5min":1.2290000000000001,"15min":11.968},"last":0.504}]},{"osd":1,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.496},"min":{"1min":0.33200000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.623,"5min":1.633,"15min":12.050000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.48899999999999999,"15min":0.48599999999999999},"min":{"1min":0.378,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.59999999999999998,"5min":1.0760000000000001,"15min":1.2150000000000001},"last":0.39400000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.49299999999999999,"15min":0.51900000000000002},"min":{"1min":0.39800000000000002,"5min":0.29699999999999999,"15min":0.28999999999999998},"max":{"1min":0.70399999999999996,"5min":1.4059999999999999,"15min":12.234},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46899999999999997,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.59499999999999997,"5min":1.1870000000000001,"15min":11.925000000000001},"last":0.371}]},{"osd":4,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69599999999999995,"15min":0.70599999999999996},"min":{"1min":0.56999999999999995,"5min":0.46999999999999997,"15min":0.40200000000000002},"max":{"1min":0.84899999999999998,"5min":3.306,"15min":12.099},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.66400000000000003,"15min":0.66600000000000004},"min":{"1min":0.59599999999999997,"5min":0.40200000000000002,"15min":0.379},"max":{"1min":0.82999999999999996,"5min":0.95799999999999996,"15min":1.0649999999999999},"last":0.48599999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.66500000000000004,"15min":0.70699999999999996},"min":{"1min":0.56299999999999994,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.79800000000000004,"5min":1.593,"15min":12.199999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68999999999999995,"15min":0.67800000000000005},"min":{"1min":0.53400000000000003,"5min":0.45300000000000001,"15min":0.436},"max":{"1min":0.92300000000000004,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.53000000000000003}]},{"osd":6,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69499999999999995,"15min":0.72099999999999997},"min":{"1min":0.56100000000000005,"5min":0.442,"15min":0.442},"max":{"1min":0.88800000000000001,"5min":1.506,"15min":12.132999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70099999999999996,"15min":0.69799999999999995},"min":{"1min":0.54600000000000004,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.88200000000000001,"5min":1.427,"15min":1.427},"last":0.58099999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.72699999999999998,"15min":0.70299999999999996},"min":{"1min":0.52500000000000002,"5min":0.48499999999999999,"15min":0.42499999999999999},"max":{"1min":0.86199999999999999,"5min":3.238,"15min":3.238},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.78200000000000003},"min":{"1min":0.625,"5min":0.48899999999999999,"15min":0.48699999999999999},"max":{"1min":0.91300000000000003,"5min":3.351,"15min":12.167},"last":0.628}]}]},{"osd":4,"up_from":13,"seq":55834575043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298424,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073416,"statfs":{"total":96636764160,"available":96331177984,"internally_reserved":0,"allocated":1163264,"data_stored":585910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57599999999999996,"15min":0.58399999999999996},"min":{"1min":0.439,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78300000000000003,"5min":1.21,"15min":1.21},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57499999999999996,"15min":0.58399999999999996},"min":{"1min":0.44900000000000001,"5min":0.309,"15min":0.28100000000000003},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":1.161},"last":0.35299999999999998}]},{"osd":1,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.44900000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":1.069},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.59399999999999997},"min":{"1min":0.47199999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":1.085},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56299999999999994,"15min":0.57499999999999996},"min":{"1min":0.48499999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76500000000000001,"5min":0.81100000000000005,"15min":1.052},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60699999999999998,"15min":0.61199999999999999},"min":{"1min":0.41399999999999998,"5min":0.39900000000000002,"15min":0.373},"max":{"1min":0.81599999999999995,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.54900000000000004}]},{"osd":3,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58199999999999996,"15min":0.59399999999999997},"min":{"1min":0.41299999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.78300000000000003,"5min":0.83499999999999996,"15min":1.1299999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79100000000000004,"5min":1.05,"15min":1.1419999999999999},"last":0.56499999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.317,"5min":0.25700000000000001,"15min":0.247},"max":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.437}]},{"osd":6,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48499999999999999,"15min":0.47799999999999998},"min":{"1min":0.35799999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.73899999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53300000000000003,"15min":0.53000000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.311},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.95699999999999996},"last":0.51100000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.35799999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.57699999999999996},"min":{"1min":0.42499999999999999,"5min":0.41799999999999998,"15min":0.32200000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.879},"last":0.60899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575043,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298424,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073416,"statfs":{"total":96636764160,"available":96331177984,"internally_reserved":0,"allocated":1163264,"data_stored":585910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.58899999999999997,"15min":0.57599999999999996},"min":{"1min":0.46300000000000002,"5min":0.29999999999999999,"15min":0.26000000000000001},"max":{"1min":0.78900000000000003,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.505},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57199999999999995,"15min":0.58399999999999996},"min":{"1min":0.45400000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77400000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.46999999999999997}]},{"osd":1,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.625,"15min":0.623},"min":{"1min":0.45500000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79500000000000004,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.64300000000000002,"15min":0.63300000000000001},"min":{"1min":0.499,"5min":0.435,"15min":0.379},"max":{"1min":0.80500000000000005,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.55800000000000005}]},{"osd":2,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.46700000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77000000000000002,"5min":0.83699999999999997,"15min":0.83799999999999997},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60399999999999998,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81100000000000005,"5min":2.0600000000000001,"15min":6.4589999999999996},"last":0.55000000000000004}]},{"osd":3,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.50700000000000001,"5min":0.35999999999999999,"15min":0.314},"max":{"1min":0.755,"5min":0.86199999999999999,"15min":0.88400000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.60599999999999998,"15min":0.61499999999999999},"min":{"1min":0.47499999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77900000000000003,"5min":2.5630000000000002,"15min":6.0629999999999997},"last":0.53900000000000003}]},{"osd":4,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.47399999999999998,"15min":0.46999999999999997},"min":{"1min":0.35599999999999998,"5min":0.28100000000000003,"15min":0.26300000000000001},"max":{"1min":0.56599999999999995,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.503,"15min":0.495},"min":{"1min":0.36899999999999999,"5min":0.33000000000000002,"15min":0.28100000000000003},"max":{"1min":0.69999999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.375}]},{"osd":6,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.31,"15min":0.27000000000000002},"max":{"1min":0.66600000000000004,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53200000000000003,"15min":0.52700000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.29299999999999998},"max":{"1min":0.71899999999999997,"5min":1.99,"15min":1.99},"last":0.32800000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.54900000000000004},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.76200000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33300000000000002},"max":{"1min":0.81499999999999995,"5min":0.82199999999999995,"15min":0.83099999999999996},"last":0.38400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:45:57.628 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:45:57.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.799+0000 7faed51c6700 1 Processor -- start 2024-01-26T23:45:57.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.799+0000 7faed51c6700 1 -- start start 2024-01-26T23:45:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.799+0000 7faed51c6700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7faed01596c0 0x7faed006bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faed51c6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faed015d460 0x7faed006c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faed51c6700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed015d8f0 con 0x7faed0159a90 2024-01-26T23:45:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faed51c6700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed00c98c0 con 0x7faed015d460 2024-01-26T23:45:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faed51c6700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed015dbd0 con 0x7faed01596c0 2024-01-26T23:45:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faece59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faed015d460 0x7faed006c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faece59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faed015d460 0x7faed006c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33480/0 (socket says 172.21.15.46:33480) 2024-01-26T23:45:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faece59c700 1 -- 172.21.15.46:0/3304231989 learned_addr learned my addr 172.21.15.46:0/3304231989 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faecf59e700 1 --1- 172.21.15.46:0/3304231989 >> v1:172.21.15.46:6789/0 conn(0x7faed0159a90 0x7faed006d310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52172/0 (socket says 172.21.15.46:52172) 2024-01-26T23:45:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faeced9d700 1 --2- 172.21.15.46:0/3304231989 >> v2:172.21.15.106:3300/0 conn(0x7faed01596c0 0x7faed006bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faece59c700 1 -- 172.21.15.46:0/3304231989 >> v2:172.21.15.106:3300/0 conn(0x7faed01596c0 msgr2=0x7faed006bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faece59c700 1 --2- 172.21.15.46:0/3304231989 >> v2:172.21.15.106:3300/0 conn(0x7faed01596c0 0x7faed006bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faece59c700 1 -- 172.21.15.46:0/3304231989 >> v1:172.21.15.46:6789/0 conn(0x7faed0159a90 legacy=0x7faed006d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faecdd9b700 1 -- 172.21.15.46:0/3304231989 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210454317 0 0) 0x7faed015d8f0 con 0x7faed0159a90 2024-01-26T23:45:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faece59c700 1 -- 172.21.15.46:0/3304231989 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed015deb0 con 0x7faed015d460 2024-01-26T23:45:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faece59c700 1 --2- 172.21.15.46:0/3304231989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed015d460 0x7faed006c210 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7faed00cf480 tx=0x7faec4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fd64d56bd86f6fe server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faecdd9b700 1 -- 172.21.15.46:0/3304231989 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec400b860 con 0x7faed015d460 2024-01-26T23:45:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faecdd9b700 1 -- 172.21.15.46:0/3304231989 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec400b9c0 con 0x7faed015d460 2024-01-26T23:45:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.800+0000 7faecdd9b700 1 -- 172.21.15.46:0/3304231989 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec400bd60 con 0x7faed015d460 2024-01-26T23:45:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.801+0000 7faed51c6700 1 -- 172.21.15.46:0/3304231989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed015d460 msgr2=0x7faed006c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.801+0000 7faed51c6700 1 --2- 172.21.15.46:0/3304231989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed015d460 0x7faed006c210 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7faed00cf480 tx=0x7faec4009250 comp rx=0 tx=0).stop 2024-01-26T23:45:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.801+0000 7faed51c6700 1 -- 172.21.15.46:0/3304231989 shutdown_connections 2024-01-26T23:45:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.801+0000 7faed51c6700 1 --2- 172.21.15.46:0/3304231989 >> v2:172.21.15.106:3300/0 conn(0x7faed01596c0 0x7faed006bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.801+0000 7faed51c6700 1 --2- 172.21.15.46:0/3304231989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed015d460 0x7faed006c210 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.801+0000 7faed51c6700 1 -- 172.21.15.46:0/3304231989 >> 172.21.15.46:0/3304231989 conn(0x7faed00ce530 msgr2=0x7faed00ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.801+0000 7faed51c6700 1 -- 172.21.15.46:0/3304231989 shutdown_connections 2024-01-26T23:45:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.801+0000 7faed51c6700 1 -- 172.21.15.46:0/3304231989 wait complete. 2024-01-26T23:45:57.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.802+0000 7faed51c6700 1 Processor -- start 2024-01-26T23:45:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.803+0000 7faed51c6700 1 -- start start 2024-01-26T23:45:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.803+0000 7faed51c6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faed01596c0 0x7faed01563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.803+0000 7faed51c6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faed0159a90 0x7faed016a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.803+0000 7faed51c6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed015d460 0x7faed00af2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.803+0000 7faed51c6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed0156900 con 0x7faed015d460 2024-01-26T23:45:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.803+0000 7faed51c6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed00b18f0 con 0x7faed0159a90 2024-01-26T23:45:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.803+0000 7faed51c6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed016a900 con 0x7faed01596c0 2024-01-26T23:45:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.803+0000 7faecf59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed015d460 0x7faed00af2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.804+0000 7faeced9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faed01596c0 0x7faed01563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.804+0000 7faecf59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed015d460 0x7faed00af2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33492/0 (socket says 172.21.15.46:33492) 2024-01-26T23:45:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.804+0000 7faecf59e700 1 -- 172.21.15.46:0/1263185583 learned_addr learned my addr 172.21.15.46:0/1263185583 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.804+0000 7faece59c700 1 --2- 172.21.15.46:0/1263185583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faed0159a90 0x7faed016a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.804+0000 7faecf59e700 1 -- 172.21.15.46:0/1263185583 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faed01596c0 msgr2=0x7faed01563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.804+0000 7faecf59e700 1 --2- 172.21.15.46:0/1263185583 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faed01596c0 0x7faed01563c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.804+0000 7faecf59e700 1 -- 172.21.15.46:0/1263185583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faed0159a90 msgr2=0x7faed016a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.804+0000 7faecf59e700 1 --2- 172.21.15.46:0/1263185583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faed0159a90 0x7faed016a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.804+0000 7faecf59e700 1 -- 172.21.15.46:0/1263185583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec40094f0 con 0x7faed015d460 2024-01-26T23:45:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.804+0000 7faecf59e700 1 --2- 172.21.15.46:0/1263185583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed015d460 0x7faed00af2d0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7faec800cd30 tx=0x7faec8001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.804+0000 7faeb6ffd700 1 -- 172.21.15.46:0/1263185583 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec800ce90 con 0x7faed015d460 2024-01-26T23:45:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.804+0000 7faeb6ffd700 1 -- 172.21.15.46:0/1263185583 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec8005df0 con 0x7faed015d460 2024-01-26T23:45:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.804+0000 7faed51c6700 1 -- 172.21.15.46:0/1263185583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed00b12e0 con 0x7faed015d460 2024-01-26T23:45:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.804+0000 7faeb6ffd700 1 -- 172.21.15.46:0/1263185583 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec800aaa0 con 0x7faed015d460 2024-01-26T23:45:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.804+0000 7faed51c6700 1 -- 172.21.15.46:0/1263185583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed00b3bd0 con 0x7faed015d460 2024-01-26T23:45:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.805+0000 7faed51c6700 1 -- 172.21.15.46:0/1263185583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed00b1a80 con 0x7faed015d460 2024-01-26T23:45:57.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.806+0000 7faeb6ffd700 1 -- 172.21.15.46:0/1263185583 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7faec80070f0 con 0x7faed015d460 2024-01-26T23:45:57.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.806+0000 7faeb6ffd700 1 --2- 172.21.15.46:0/1263185583 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faebc066ea0 0x7faebc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:57.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.807+0000 7faeb6ffd700 1 -- 172.21.15.46:0/1263185583 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6417+0+0 (secure 0 0 0) 0x7faec8084740 con 0x7faed015d460 2024-01-26T23:45:57.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.807+0000 7faeced9d700 1 --2- 172.21.15.46:0/1263185583 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faebc066ea0 0x7faebc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:57.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.807+0000 7faeced9d700 1 --2- 172.21.15.46:0/1263185583 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faebc066ea0 0x7faebc069350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faec0006fd0 tx=0x7faec0008040 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:57.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:57.811+0000 7faeb6ffd700 1 -- 172.21.15.46:0/1263185583 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faec8051840 con 0x7faed015d460 2024-01-26T23:45:58.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.016+0000 7faed51c6700 1 -- 172.21.15.46:0/1263185583 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faed004c710 con 0x7faebc066ea0 2024-01-26T23:45:58.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.024+0000 7faeb6ffd700 1 -- 172.21.15.46:0/1263185583 <== mgr.6421 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281834 (secure 0 0 0) 0x7faed004c710 con 0x7faebc066ea0 2024-01-26T23:45:58.028 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:58.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.027+0000 7faed51c6700 1 -- 172.21.15.46:0/1263185583 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faebc066ea0 msgr2=0x7faebc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:58.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.027+0000 7faed51c6700 1 --2- 172.21.15.46:0/1263185583 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faebc066ea0 0x7faebc069350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faec0006fd0 tx=0x7faec0008040 comp rx=0 tx=0).stop 2024-01-26T23:45:58.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.027+0000 7faed51c6700 1 -- 172.21.15.46:0/1263185583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed015d460 msgr2=0x7faed00af2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.027+0000 7faed51c6700 1 --2- 172.21.15.46:0/1263185583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed015d460 0x7faed00af2d0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7faec800cd30 tx=0x7faec8001ac0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.028+0000 7faed51c6700 1 -- 172.21.15.46:0/1263185583 shutdown_connections 2024-01-26T23:45:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.028+0000 7faed51c6700 1 --2- 172.21.15.46:0/1263185583 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7faebc066ea0 0x7faebc069350 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.028+0000 7faed51c6700 1 --2- 172.21.15.46:0/1263185583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed015d460 0x7faed00af2d0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.028+0000 7faed51c6700 1 --2- 172.21.15.46:0/1263185583 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faed01596c0 0x7faed01563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.028+0000 7faed51c6700 1 --2- 172.21.15.46:0/1263185583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faed0159a90 0x7faed016a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.028+0000 7faed51c6700 1 -- 172.21.15.46:0/1263185583 >> 172.21.15.46:0/1263185583 conn(0x7faed00ce530 msgr2=0x7faed015bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.028+0000 7faed51c6700 1 -- 172.21.15.46:0/1263185583 shutdown_connections 2024-01-26T23:45:58.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.028+0000 7faed51c6700 1 -- 172.21.15.46:0/1263185583 wait complete. 2024-01-26T23:45:58.032 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:45:58.042 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-26T23:45:56.860876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":585,"num_read_kb":1124,"num_write":265,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1509,"ondisk_log_size":1509,"up":106,"acting":106,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2437784,"kb_used_data":10328,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752536936,"statfs":{"total":773094113280,"available":770597822464,"internally_reserved":0,"allocated":10575872,"data_stored":5911028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.883382"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.009542+0000","last_change":"2024-01-26T23:45:52.998874+0000","last_active":"2024-01-26T23:45:53.009542+0000","last_peered":"2024-01-26T23:45:53.009542+0000","last_clean":"2024-01-26T23:45:53.009542+0000","last_became_active":"2024-01-26T23:45:52.998533+0000","last_became_peered":"2024-01-26T23:45:52.998533+0000","last_unstale":"2024-01-26T23:45:53.009542+0000","last_undegraded":"2024-01-26T23:45:53.009542+0000","last_fullsized":"2024-01-26T23:45:53.009542+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:22:33.448776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.006656+0000","last_change":"2024-01-26T23:45:53.004780+0000","last_active":"2024-01-26T23:45:53.006656+0000","last_peered":"2024-01-26T23:45:53.006656+0000","last_clean":"2024-01-26T23:45:53.006656+0000","last_became_active":"2024-01-26T23:45:53.004486+0000","last_became_peered":"2024-01-26T23:45:53.004486+0000","last_unstale":"2024-01-26T23:45:53.006656+0000","last_undegraded":"2024-01-26T23:45:53.006656+0000","last_fullsized":"2024-01-26T23:45:53.006656+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:52:16.515887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.013956+0000","last_change":"2024-01-26T23:45:53.010404+0000","last_active":"2024-01-26T23:45:53.013956+0000","last_peered":"2024-01-26T23:45:53.013956+0000","last_clean":"2024-01-26T23:45:53.013956+0000","last_became_active":"2024-01-26T23:45:53.010120+0000","last_became_peered":"2024-01-26T23:45:53.010120+0000","last_unstale":"2024-01-26T23:45:53.013956+0000","last_undegraded":"2024-01-26T23:45:53.013956+0000","last_fullsized":"2024-01-26T23:45:53.013956+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:58:27.010243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001869+0000","last_change":"2024-01-26T23:45:52.993487+0000","last_active":"2024-01-26T23:45:53.001869+0000","last_peered":"2024-01-26T23:45:53.001869+0000","last_clean":"2024-01-26T23:45:53.001869+0000","last_became_active":"2024-01-26T23:45:52.993157+0000","last_became_peered":"2024-01-26T23:45:52.993157+0000","last_unstale":"2024-01-26T23:45:53.001869+0000","last_undegraded":"2024-01-26T23:45:53.001869+0000","last_fullsized":"2024-01-26T23:45:53.001869+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:44:24.788145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002692+0000","last_change":"2024-01-26T23:45:52.995031+0000","last_active":"2024-01-26T23:45:53.002692+0000","last_peered":"2024-01-26T23:45:53.002692+0000","last_clean":"2024-01-26T23:45:53.002692+0000","last_became_active":"2024-01-26T23:45:52.994711+0000","last_became_peered":"2024-01-26T23:45:52.994711+0000","last_unstale":"2024-01-26T23:45:53.002692+0000","last_undegraded":"2024-01-26T23:45:53.002692+0000","last_fullsized":"2024-01-26T23:45:53.002692+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:20.934227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010618+0000","last_change":"2024-01-26T23:45:53.002633+0000","last_active":"2024-01-26T23:45:53.010618+0000","last_peered":"2024-01-26T23:45:53.010618+0000","last_clean":"2024-01-26T23:45:53.010618+0000","last_became_active":"2024-01-26T23:45:53.002349+0000","last_became_peered":"2024-01-26T23:45:53.002349+0000","last_unstale":"2024-01-26T23:45:53.010618+0000","last_undegraded":"2024-01-26T23:45:53.010618+0000","last_fullsized":"2024-01-26T23:45:53.010618+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:48:28.528692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.011887+0000","last_change":"2024-01-26T23:45:53.006070+0000","last_active":"2024-01-26T23:45:53.011887+0000","last_peered":"2024-01-26T23:45:53.011887+0000","last_clean":"2024-01-26T23:45:53.011887+0000","last_became_active":"2024-01-26T23:45:53.005736+0000","last_became_peered":"2024-01-26T23:45:53.005736+0000","last_unstale":"2024-01-26T23:45:53.011887+0000","last_undegraded":"2024-01-26T23:45:53.011887+0000","last_fullsized":"2024-01-26T23:45:53.011887+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:57:13.832289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001218+0000","last_change":"2024-01-26T23:45:52.994893+0000","last_active":"2024-01-26T23:45:53.001218+0000","last_peered":"2024-01-26T23:45:53.001218+0000","last_clean":"2024-01-26T23:45:53.001218+0000","last_became_active":"2024-01-26T23:45:52.994613+0000","last_became_peered":"2024-01-26T23:45:52.994613+0000","last_unstale":"2024-01-26T23:45:53.001218+0000","last_undegraded":"2024-01-26T23:45:53.001218+0000","last_fullsized":"2024-01-26T23:45:53.001218+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:09:09.561092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002434+0000","last_change":"2024-01-26T23:45:52.997178+0000","last_active":"2024-01-26T23:45:53.002434+0000","last_peered":"2024-01-26T23:45:53.002434+0000","last_clean":"2024-01-26T23:45:53.002434+0000","last_became_active":"2024-01-26T23:45:52.996868+0000","last_became_peered":"2024-01-26T23:45:52.996868+0000","last_unstale":"2024-01-26T23:45:53.002434+0000","last_undegraded":"2024-01-26T23:45:53.002434+0000","last_fullsized":"2024-01-26T23:45:53.002434+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:07:38.558915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005249+0000","last_change":"2024-01-26T23:45:52.999818+0000","last_active":"2024-01-26T23:45:53.005249+0000","last_peered":"2024-01-26T23:45:53.005249+0000","last_clean":"2024-01-26T23:45:53.005249+0000","last_became_active":"2024-01-26T23:45:52.999503+0000","last_became_peered":"2024-01-26T23:45:52.999503+0000","last_unstale":"2024-01-26T23:45:53.005249+0000","last_undegraded":"2024-01-26T23:45:53.005249+0000","last_fullsized":"2024-01-26T23:45:53.005249+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:47.231142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979235+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:02:21.377733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.009449+0000","last_change":"2024-01-26T23:45:52.995466+0000","last_active":"2024-01-26T23:45:53.009449+0000","last_peered":"2024-01-26T23:45:53.009449+0000","last_clean":"2024-01-26T23:45:53.009449+0000","last_became_active":"2024-01-26T23:45:52.995141+0000","last_became_peered":"2024-01-26T23:45:52.995141+0000","last_unstale":"2024-01-26T23:45:53.009449+0000","last_undegraded":"2024-01-26T23:45:53.009449+0000","last_fullsized":"2024-01-26T23:45:53.009449+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:08.999844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.983371+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:46.226628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.983396+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:07:37.554401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.982175+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:50:39.847921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010005+0000","last_change":"2024-01-26T23:45:53.001042+0000","last_active":"2024-01-26T23:45:53.010005+0000","last_peered":"2024-01-26T23:45:53.010005+0000","last_clean":"2024-01-26T23:45:53.010005+0000","last_became_active":"2024-01-26T23:45:53.000759+0000","last_became_peered":"2024-01-26T23:45:53.000759+0000","last_unstale":"2024-01-26T23:45:53.010005+0000","last_undegraded":"2024-01-26T23:45:53.010005+0000","last_fullsized":"2024-01-26T23:45:53.010005+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:48:01.627096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.981367+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:08:58.845586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.013976+0000","last_change":"2024-01-26T23:45:53.011018+0000","last_active":"2024-01-26T23:45:53.013976+0000","last_peered":"2024-01-26T23:45:53.013976+0000","last_clean":"2024-01-26T23:45:53.013976+0000","last_became_active":"2024-01-26T23:45:53.010726+0000","last_became_peered":"2024-01-26T23:45:53.010726+0000","last_unstale":"2024-01-26T23:45:53.013976+0000","last_undegraded":"2024-01-26T23:45:53.013976+0000","last_fullsized":"2024-01-26T23:45:53.013976+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:50:13.686822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.985553+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:08:58.845586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.983528+0000","last_change":"2024-01-26T23:45:51.995344+0000","last_active":"2024-01-26T23:45:52.983528+0000","last_peered":"2024-01-26T23:45:52.983528+0000","last_clean":"2024-01-26T23:45:52.983528+0000","last_became_active":"2024-01-26T23:45:51.995068+0000","last_became_peered":"2024-01-26T23:45:51.995068+0000","last_unstale":"2024-01-26T23:45:52.983528+0000","last_undegraded":"2024-01-26T23:45:52.983528+0000","last_fullsized":"2024-01-26T23:45:52.983528+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:45:21.389917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979110+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:19:46.301437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010594+0000","last_change":"2024-01-26T23:45:52.999941+0000","last_active":"2024-01-26T23:45:53.010594+0000","last_peered":"2024-01-26T23:45:53.010594+0000","last_clean":"2024-01-26T23:45:53.010594+0000","last_became_active":"2024-01-26T23:45:52.999657+0000","last_became_peered":"2024-01-26T23:45:52.999657+0000","last_unstale":"2024-01-26T23:45:53.010594+0000","last_undegraded":"2024-01-26T23:45:53.010594+0000","last_fullsized":"2024-01-26T23:45:53.010594+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:21:56.682642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.983305+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:24:28.011807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.985296+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:37:57.061471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979863+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:57:41.012865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.006905+0000","last_change":"2024-01-26T23:45:53.004186+0000","last_active":"2024-01-26T23:45:53.006905+0000","last_peered":"2024-01-26T23:45:53.006905+0000","last_clean":"2024-01-26T23:45:53.006905+0000","last_became_active":"2024-01-26T23:45:53.003905+0000","last_became_peered":"2024-01-26T23:45:53.003905+0000","last_unstale":"2024-01-26T23:45:53.006905+0000","last_undegraded":"2024-01-26T23:45:53.006905+0000","last_fullsized":"2024-01-26T23:45:53.006905+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:24:29.016321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.981744+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:26.901131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.985171+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:21:06.318896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.983160+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:19.929713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979710+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:21:55.884844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.10","version":"97'8","reported_seq":18,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.020619+0000","last_change":"2024-01-26T23:45:51.994969+0000","last_active":"2024-01-26T23:45:53.020619+0000","last_peered":"2024-01-26T23:45:53.020619+0000","last_clean":"2024-01-26T23:45:53.020619+0000","last_became_active":"2024-01-26T23:45:51.994689+0000","last_became_peered":"2024-01-26T23:45:51.994689+0000","last_unstale":"2024-01-26T23:45:53.020619+0000","last_undegraded":"2024-01-26T23:45:53.020619+0000","last_fullsized":"2024-01-26T23:45:53.020619+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:01:04.622959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.982869+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:05:31.172516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.981303+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:37:57.061471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979140+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:32:28.510671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.987187+0000","last_change":"2024-01-26T23:45:52.007285+0000","last_active":"2024-01-26T23:45:52.987187+0000","last_peered":"2024-01-26T23:45:52.987187+0000","last_clean":"2024-01-26T23:45:52.987187+0000","last_became_active":"2024-01-26T23:45:52.006952+0000","last_became_peered":"2024-01-26T23:45:52.006952+0000","last_unstale":"2024-01-26T23:45:52.987187+0000","last_undegraded":"2024-01-26T23:45:52.987187+0000","last_fullsized":"2024-01-26T23:45:52.987187+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:50:39.847921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001264+0000","last_change":"2024-01-26T23:45:52.992221+0000","last_active":"2024-01-26T23:45:53.001264+0000","last_peered":"2024-01-26T23:45:53.001264+0000","last_clean":"2024-01-26T23:45:53.001264+0000","last_became_active":"2024-01-26T23:45:52.991944+0000","last_became_peered":"2024-01-26T23:45:52.991944+0000","last_unstale":"2024-01-26T23:45:53.001264+0000","last_undegraded":"2024-01-26T23:45:53.001264+0000","last_fullsized":"2024-01-26T23:45:53.001264+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:28:47.388074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979695+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:59.450936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001363+0000","last_change":"2024-01-26T23:45:52.994059+0000","last_active":"2024-01-26T23:45:53.001363+0000","last_peered":"2024-01-26T23:45:53.001363+0000","last_clean":"2024-01-26T23:45:53.001363+0000","last_became_active":"2024-01-26T23:45:52.993774+0000","last_became_peered":"2024-01-26T23:45:52.993774+0000","last_unstale":"2024-01-26T23:45:53.001363+0000","last_undegraded":"2024-01-26T23:45:53.001363+0000","last_fullsized":"2024-01-26T23:45:53.001363+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:37:33.720336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979861+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:22.235033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979243+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:52:15.511373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.985217+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:42:07.246862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.984063+0000","last_change":"2024-01-26T23:45:52.004361+0000","last_active":"2024-01-26T23:45:52.984063+0000","last_peered":"2024-01-26T23:45:52.984063+0000","last_clean":"2024-01-26T23:45:52.984063+0000","last_became_active":"2024-01-26T23:45:52.004048+0000","last_became_peered":"2024-01-26T23:45:52.004048+0000","last_unstale":"2024-01-26T23:45:52.984063+0000","last_undegraded":"2024-01-26T23:45:52.984063+0000","last_fullsized":"2024-01-26T23:45:52.984063+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:01:43.090741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.982946+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:26:59.702876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.009979+0000","last_change":"2024-01-26T23:45:52.999402+0000","last_active":"2024-01-26T23:45:53.009979+0000","last_peered":"2024-01-26T23:45:53.009979+0000","last_clean":"2024-01-26T23:45:53.009979+0000","last_became_active":"2024-01-26T23:45:52.999111+0000","last_became_peered":"2024-01-26T23:45:52.999111+0000","last_unstale":"2024-01-26T23:45:53.009979+0000","last_undegraded":"2024-01-26T23:45:53.009979+0000","last_fullsized":"2024-01-26T23:45:53.009979+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:15:38.716213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.982814+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:40:36.924823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.980793+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:21.591456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979645+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:02:38.392302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005600+0000","last_change":"2024-01-26T23:45:53.002599+0000","last_active":"2024-01-26T23:45:53.005600+0000","last_peered":"2024-01-26T23:45:53.005600+0000","last_clean":"2024-01-26T23:45:53.005600+0000","last_became_active":"2024-01-26T23:45:53.000819+0000","last_became_peered":"2024-01-26T23:45:53.000819+0000","last_unstale":"2024-01-26T23:45:53.005600+0000","last_undegraded":"2024-01-26T23:45:53.005600+0000","last_fullsized":"2024-01-26T23:45:53.005600+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:06.428462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.981239+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:54:17.875691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005498+0000","last_change":"2024-01-26T23:45:53.002124+0000","last_active":"2024-01-26T23:45:53.005498+0000","last_peered":"2024-01-26T23:45:53.005498+0000","last_clean":"2024-01-26T23:45:53.005498+0000","last_became_active":"2024-01-26T23:45:53.000783+0000","last_became_peered":"2024-01-26T23:45:53.000783+0000","last_unstale":"2024-01-26T23:45:53.005498+0000","last_undegraded":"2024-01-26T23:45:53.005498+0000","last_fullsized":"2024-01-26T23:45:53.005498+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:00:00.455450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.990097+0000","last_change":"2024-01-26T23:45:52.007142+0000","last_active":"2024-01-26T23:45:52.990097+0000","last_peered":"2024-01-26T23:45:52.990097+0000","last_clean":"2024-01-26T23:45:52.990097+0000","last_became_active":"2024-01-26T23:45:52.006791+0000","last_became_peered":"2024-01-26T23:45:52.006791+0000","last_unstale":"2024-01-26T23:45:52.990097+0000","last_undegraded":"2024-01-26T23:45:52.990097+0000","last_fullsized":"2024-01-26T23:45:52.990097+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:44.905097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:45:51.976854+0000","last_change":"2024-01-26T23:45:51.976854+0000","last_active":"2024-01-26T23:45:51.976854+0000","last_peered":"2024-01-26T23:45:51.976854+0000","last_clean":"2024-01-26T23:45:51.976854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:51.976854+0000","last_undegraded":"2024-01-26T23:45:51.976854+0000","last_fullsized":"2024-01-26T23:45:51.976854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:45:50.972340+0000","last_change":"2024-01-26T23:45:50.979123+0000","last_active":"2024-01-26T23:45:50.972340+0000","last_peered":"2024-01-26T23:45:50.972340+0000","last_clean":"2024-01-26T23:45:50.972340+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:45:50.972340+0000","last_undegraded":"2024-01-26T23:45:50.972340+0000","last_fullsized":"2024-01-26T23:45:50.972340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:05.423948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"95'1501","reported_seq":1761,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-01-26T23:45:51.854728+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:45:51.854728+0000","last_peered":"2024-01-26T23:45:51.854728+0000","last_clean":"2024-01-26T23:45:51.854728+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:45:51.854728+0000","last_undegraded":"2024-01-26T23:45:51.854728+0000","last_fullsized":"2024-01-26T23:45:51.854728+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":1501,"log_dups_size":0,"ondisk_log_size":1501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":585,"num_read_kb":1124,"num_write":253,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":40,"acting":40,"num_store_stats":2},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":585,"num_read_kb":1124,"num_write":253,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1501,"ondisk_log_size":1501,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575043,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311292,"kb_used_data":1716,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060548,"statfs":{"total":96636764160,"available":96318001152,"internally_reserved":0,"allocated":1757184,"data_stored":1176278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57599999999999996,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.79500000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.314},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.35699999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80400000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.29999999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57699999999999996,"15min":0.57799999999999996},"min":{"1min":0.42499999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77800000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.61099999999999999,"15min":0.59899999999999998},"min":{"1min":0.46000000000000002,"5min":0.39200000000000002,"15min":0.36099999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.45000000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.38800000000000001,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.81699999999999995},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.48699999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77900000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.53000000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59999999999999998,"15min":0.59199999999999997},"min":{"1min":0.442,"5min":0.40500000000000003,"15min":0.35399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60599999999999998,"15min":0.60499999999999998},"min":{"1min":0.40100000000000002,"5min":0.39300000000000002,"15min":0.373},"max":{"1min":0.82699999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.46899999999999997}]},{"osd":4,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46899999999999997,"15min":0.46100000000000002},"min":{"1min":0.36399999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.66100000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.495,"15min":0.48799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.314},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.28499999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.504,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.91700000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.40000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.74199999999999999,"5min":0.90200000000000002,"15min":0.96599999999999997},"last":0.33300000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.57999999999999996,"15min":0.55700000000000005},"min":{"1min":0.46400000000000002,"5min":0.375,"15min":0.33100000000000002},"max":{"1min":0.77900000000000003,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.372},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56200000000000006,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76500000000000001,"5min":0.85199999999999998,"15min":0.86899999999999999},"last":0.434}]}]},{"osd":6,"up_from":13,"seq":55834575043,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302520,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069320,"statfs":{"total":96636764160,"available":96326983680,"internally_reserved":0,"allocated":1163264,"data_stored":585910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56899999999999995,"15min":0.57399999999999995},"min":{"1min":0.35999999999999999,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":0.752,"5min":0.85199999999999998,"15min":0.90200000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59099999999999997,"15min":0.59399999999999997},"min":{"1min":0.42899999999999999,"5min":0.38500000000000001,"15min":0.33600000000000002},"max":{"1min":0.79000000000000004,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.35499999999999998}]},{"osd":1,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.312,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.77100000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.34399999999999997},"max":{"1min":0.75700000000000001,"5min":0.78900000000000003,"15min":0.874},"last":0.40999999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.40799999999999997,"5min":0.35499999999999998,"15min":0.29599999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.85799999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38600000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77000000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.41899999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.59499999999999997,"15min":0.59999999999999998},"min":{"1min":0.45500000000000002,"5min":0.38700000000000001,"15min":0.29999999999999999},"max":{"1min":0.79000000000000004,"5min":0.80700000000000005,"15min":0.92200000000000004},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.622,"15min":0.624},"min":{"1min":0.55000000000000004,"5min":0.39500000000000002,"15min":0.39000000000000001},"max":{"1min":0.78400000000000003,"5min":0.81100000000000005,"15min":0.84499999999999997},"last":0.46500000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.47799999999999998,"15min":0.47399999999999998},"min":{"1min":0.38100000000000001,"5min":0.29899999999999999,"15min":0.22},"max":{"1min":0.69899999999999995,"5min":0.84599999999999997,"15min":0.90200000000000002},"last":0.375},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.501,"15min":0.499},"min":{"1min":0.44700000000000001,"5min":0.32000000000000001,"15min":0.28899999999999998},"max":{"1min":0.61899999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.43099999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48499999999999999,"15min":0.48899999999999999},"min":{"1min":0.42499999999999999,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.53400000000000003,"15min":0.53300000000000003},"min":{"1min":0.46100000000000002,"5min":0.33700000000000002,"15min":0.26200000000000001},"max":{"1min":0.75700000000000001,"5min":0.872,"15min":0.96499999999999997},"last":0.44400000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53500000000000003,"15min":0.53800000000000003},"min":{"1min":0.44400000000000001,"5min":0.32500000000000001,"15min":0.246},"max":{"1min":0.81799999999999995,"5min":0.82799999999999996,"15min":0.97199999999999998},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.308},"max":{"1min":0.73099999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.32200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575044,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302564,"kb_used_data":1180,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069276,"statfs":{"total":96636764160,"available":96326938624,"internally_reserved":0,"allocated":1208320,"data_stored":607472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29899999999999999,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":0.59699999999999998,"5min":0.621,"15min":0.72599999999999998},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.35799999999999998,"5min":0.28899999999999998,"15min":0.26000000000000001},"max":{"1min":0.73499999999999999,"5min":1.337,"15min":1.337},"last":0.29299999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41499999999999998},"min":{"1min":0.27700000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55900000000000005,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48599999999999999,"15min":0.47699999999999998},"min":{"1min":0.376,"5min":0.34699999999999998,"15min":0.28999999999999998},"max":{"1min":0.70799999999999996,"5min":0.77300000000000002,"15min":0.79600000000000004},"last":0.42999999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52200000000000002,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":0.75800000000000001,"5min":1.782,"15min":1.782},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.497,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.625,"5min":1.379,"15min":1.379},"last":0.54300000000000004}]},{"osd":4,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65300000000000002,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":0.86499999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66200000000000003,"15min":0.66600000000000004},"min":{"1min":0.49399999999999999,"5min":0.45900000000000002,"15min":0.433},"max":{"1min":0.89900000000000002,"5min":0.91900000000000004,"15min":0.95699999999999996},"last":0.56699999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.50600000000000001,"5min":0.46800000000000003,"15min":0.42799999999999999},"max":{"1min":0.879,"5min":1.208,"15min":1.208},"last":0.62},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68500000000000005,"15min":0.67100000000000004},"min":{"1min":0.53700000000000003,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":0.81599999999999995,"5min":0.96599999999999997,"15min":1.018},"last":0.48699999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.69399999999999995,"15min":0.68500000000000005},"min":{"1min":0.51800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85199999999999998,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.52900000000000003,"5min":0.437,"15min":0.42599999999999999},"max":{"1min":0.83699999999999997,"5min":1.022,"15min":1.022},"last":0.57799999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71599999999999997},"min":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.49399999999999999},"max":{"1min":0.93200000000000005,"5min":0.98199999999999998,"15min":1.0069999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.749,"15min":0.73199999999999998},"min":{"1min":0.53200000000000003,"5min":0.52200000000000002,"15min":0.438},"max":{"1min":0.90600000000000003,"5min":1.296,"15min":1.3049999999999999},"last":0.59799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575043,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311328,"kb_used_data":1752,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060512,"statfs":{"total":96636764160,"available":96317964288,"internally_reserved":0,"allocated":1794048,"data_stored":1197728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.28499999999999998,"5min":0.23100000000000001,"15min":0.214},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":1.284},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.46000000000000002,"15min":0.45600000000000002},"min":{"1min":0.29099999999999998,"5min":0.28899999999999998,"15min":0.27300000000000002},"max":{"1min":0.60899999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.308}]},{"osd":2,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42099999999999999,"15min":0.42999999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.26700000000000002},"max":{"1min":0.57299999999999995,"5min":0.89600000000000002,"15min":1.2230000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.38300000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":1.2430000000000001},"last":0.42099999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46899999999999997,"15min":0.46700000000000003},"min":{"1min":0.36799999999999999,"5min":0.30599999999999999,"15min":0.26800000000000002},"max":{"1min":0.68100000000000005,"5min":0.80700000000000005,"15min":1.0980000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.49399999999999999,"15min":0.49099999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.27000000000000002},"max":{"1min":0.64000000000000001,"5min":1.198,"15min":1.198},"last":0.436}]},{"osd":4,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.64300000000000002,"15min":0.65200000000000002},"min":{"1min":0.46899999999999997,"5min":0.42399999999999999,"15min":0.372},"max":{"1min":0.78700000000000003,"5min":0.93300000000000005,"15min":1.133},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64700000000000002,"15min":0.65900000000000003},"min":{"1min":0.438,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85299999999999998,"5min":0.94299999999999995,"15min":1.0169999999999999},"last":0.373}]},{"osd":5,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65100000000000002,"15min":0.66300000000000003},"min":{"1min":0.52600000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79800000000000004,"5min":0.95799999999999996,"15min":1.0880000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67400000000000004,"15min":0.68200000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.41699999999999998},"max":{"1min":0.93999999999999995,"5min":1.012,"15min":1.169},"last":0.47399999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65500000000000003,"15min":0.67700000000000005},"min":{"1min":0.505,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89800000000000002,"5min":0.91600000000000004,"15min":1.071},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.68400000000000005,"15min":0.69999999999999996},"min":{"1min":0.48299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.94599999999999995,"5min":1.028,"15min":1.1160000000000001},"last":0.55300000000000005}]},{"osd":7,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":1.004,"15min":0.82599999999999996},"min":{"1min":0.54500000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.91500000000000004,"5min":25.965,"15min":25.965},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.73899999999999999,"5min":1.109,"15min":0.86599999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.433},"max":{"1min":0.93799999999999994,"5min":33.085000000000001,"15min":33.085000000000001},"last":0.56299999999999994}]}]},{"osd":2,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306616,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065224,"statfs":{"total":96636764160,"available":96322789376,"internally_reserved":0,"allocated":1163264,"data_stored":585910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.39400000000000002,"15min":0.41999999999999998},"min":{"1min":0.29299999999999998,"5min":0.25800000000000001,"15min":0.22600000000000001},"max":{"1min":0.502,"5min":0.63400000000000001,"15min":9.0540000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.24299999999999999},"max":{"1min":0.56899999999999995,"5min":0.68500000000000005,"15min":4.1970000000000001},"last":0.54400000000000004}]},{"osd":1,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42699999999999999,"15min":0.42499999999999999},"min":{"1min":0.33300000000000002,"5min":0.248,"15min":0.221},"max":{"1min":0.55300000000000005,"5min":0.70499999999999996,"15min":0.82199999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.498,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":0.629,"5min":0.65900000000000003,"15min":12.157},"last":0.629}]},{"osd":3,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47799999999999998,"15min":0.48599999999999999},"min":{"1min":0.38800000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.59299999999999997,"5min":0.65500000000000003,"15min":0.86499999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.47599999999999998,"15min":0.499},"min":{"1min":0.34599999999999997,"5min":0.32200000000000001,"15min":0.30399999999999999},"max":{"1min":0.60199999999999998,"5min":0.71099999999999997,"15min":9.0879999999999992},"last":0.61299999999999999}]},{"osd":4,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65000000000000002,"15min":0.66600000000000004},"min":{"1min":0.499,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.86599999999999999,"5min":1.087,"15min":9.0150000000000006},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.65800000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.371},"max":{"1min":0.80200000000000005,"5min":0.98099999999999998,"15min":4.2480000000000002},"last":0.874}]},{"osd":5,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78600000000000003,"5min":1.0680000000000001,"15min":4.3200000000000003},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68899999999999995,"15min":0.68200000000000005},"min":{"1min":0.58899999999999997,"5min":0.39800000000000002,"15min":0.35199999999999998},"max":{"1min":0.82399999999999995,"5min":0.95199999999999996,"15min":1.085},"last":0.97699999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68899999999999995,"15min":0.68400000000000005},"min":{"1min":0.51500000000000001,"5min":0.439,"15min":0.40899999999999997},"max":{"1min":0.79400000000000004,"5min":0.995,"15min":1.1080000000000001},"last":1.022},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69199999999999995,"15min":0.71299999999999997},"min":{"1min":0.46200000000000002,"5min":0.443,"15min":0.38800000000000001},"max":{"1min":0.83999999999999997,"5min":1.1299999999999999,"15min":8.9700000000000006},"last":0.82899999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70799999999999996,"15min":0.71699999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.85599999999999998,"5min":1.0029999999999999,"15min":4.282},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.57499999999999996,"5min":0.48199999999999998,"15min":0.42099999999999999},"max":{"1min":0.87,"5min":1.052,"15min":1.139},"last":1.05}]}]},{"osd":3,"up_from":13,"seq":55834575041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306616,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065224,"statfs":{"total":96636764160,"available":96322789376,"internally_reserved":0,"allocated":1163264,"data_stored":585910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40300000000000002,"15min":0.39800000000000002},"min":{"1min":0.32300000000000001,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.56999999999999995,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44900000000000001,"15min":0.48099999999999998},"min":{"1min":0.315,"5min":0.255,"15min":0.255},"max":{"1min":0.59299999999999997,"5min":1.2290000000000001,"15min":11.968},"last":0.504}]},{"osd":1,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.496},"min":{"1min":0.33200000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.623,"5min":1.633,"15min":12.050000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.48899999999999999,"15min":0.48599999999999999},"min":{"1min":0.378,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.59999999999999998,"5min":1.0760000000000001,"15min":1.2150000000000001},"last":0.39400000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.49299999999999999,"15min":0.51900000000000002},"min":{"1min":0.39800000000000002,"5min":0.29699999999999999,"15min":0.28999999999999998},"max":{"1min":0.70399999999999996,"5min":1.4059999999999999,"15min":12.234},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46899999999999997,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.59499999999999997,"5min":1.1870000000000001,"15min":11.925000000000001},"last":0.371}]},{"osd":4,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69599999999999995,"15min":0.70599999999999996},"min":{"1min":0.56999999999999995,"5min":0.46999999999999997,"15min":0.40200000000000002},"max":{"1min":0.84899999999999998,"5min":3.306,"15min":12.099},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.66400000000000003,"15min":0.66600000000000004},"min":{"1min":0.59599999999999997,"5min":0.40200000000000002,"15min":0.379},"max":{"1min":0.82999999999999996,"5min":0.95799999999999996,"15min":1.0649999999999999},"last":0.48599999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.66500000000000004,"15min":0.70699999999999996},"min":{"1min":0.56299999999999994,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.79800000000000004,"5min":1.593,"15min":12.199999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68999999999999995,"15min":0.67800000000000005},"min":{"1min":0.53400000000000003,"5min":0.45300000000000001,"15min":0.436},"max":{"1min":0.92300000000000004,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.53000000000000003}]},{"osd":6,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69499999999999995,"15min":0.72099999999999997},"min":{"1min":0.56100000000000005,"5min":0.442,"15min":0.442},"max":{"1min":0.88800000000000001,"5min":1.506,"15min":12.132999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70099999999999996,"15min":0.69799999999999995},"min":{"1min":0.54600000000000004,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.88200000000000001,"5min":1.427,"15min":1.427},"last":0.58099999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.72699999999999998,"15min":0.70299999999999996},"min":{"1min":0.52500000000000002,"5min":0.48499999999999999,"15min":0.42499999999999999},"max":{"1min":0.86199999999999999,"5min":3.238,"15min":3.238},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.78200000000000003},"min":{"1min":0.625,"5min":0.48899999999999999,"15min":0.48699999999999999},"max":{"1min":0.91300000000000003,"5min":3.351,"15min":12.167},"last":0.628}]}]},{"osd":4,"up_from":13,"seq":55834575043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298424,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073416,"statfs":{"total":96636764160,"available":96331177984,"internally_reserved":0,"allocated":1163264,"data_stored":585910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57599999999999996,"15min":0.58399999999999996},"min":{"1min":0.439,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78300000000000003,"5min":1.21,"15min":1.21},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57499999999999996,"15min":0.58399999999999996},"min":{"1min":0.44900000000000001,"5min":0.309,"15min":0.28100000000000003},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":1.161},"last":0.35299999999999998}]},{"osd":1,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.44900000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":1.069},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.59399999999999997},"min":{"1min":0.47199999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":1.085},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56299999999999994,"15min":0.57499999999999996},"min":{"1min":0.48499999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76500000000000001,"5min":0.81100000000000005,"15min":1.052},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60699999999999998,"15min":0.61199999999999999},"min":{"1min":0.41399999999999998,"5min":0.39900000000000002,"15min":0.373},"max":{"1min":0.81599999999999995,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.54900000000000004}]},{"osd":3,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58199999999999996,"15min":0.59399999999999997},"min":{"1min":0.41299999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.78300000000000003,"5min":0.83499999999999996,"15min":1.1299999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79100000000000004,"5min":1.05,"15min":1.1419999999999999},"last":0.56499999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.317,"5min":0.25700000000000001,"15min":0.247},"max":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.437}]},{"osd":6,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48499999999999999,"15min":0.47799999999999998},"min":{"1min":0.35799999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.73899999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53300000000000003,"15min":0.53000000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.311},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.95699999999999996},"last":0.51100000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.35799999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.57699999999999996},"min":{"1min":0.42499999999999999,"5min":0.41799999999999998,"15min":0.32200000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.879},"last":0.60899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575043,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298424,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073416,"statfs":{"total":96636764160,"available":96331177984,"internally_reserved":0,"allocated":1163264,"data_stored":585910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.58899999999999997,"15min":0.57599999999999996},"min":{"1min":0.46300000000000002,"5min":0.29999999999999999,"15min":0.26000000000000001},"max":{"1min":0.78900000000000003,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.505},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57199999999999995,"15min":0.58399999999999996},"min":{"1min":0.45400000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77400000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.46999999999999997}]},{"osd":1,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.625,"15min":0.623},"min":{"1min":0.45500000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79500000000000004,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.64300000000000002,"15min":0.63300000000000001},"min":{"1min":0.499,"5min":0.435,"15min":0.379},"max":{"1min":0.80500000000000005,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.55800000000000005}]},{"osd":2,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.46700000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77000000000000002,"5min":0.83699999999999997,"15min":0.83799999999999997},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60399999999999998,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81100000000000005,"5min":2.0600000000000001,"15min":6.4589999999999996},"last":0.55000000000000004}]},{"osd":3,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.50700000000000001,"5min":0.35999999999999999,"15min":0.314},"max":{"1min":0.755,"5min":0.86199999999999999,"15min":0.88400000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.60599999999999998,"15min":0.61499999999999999},"min":{"1min":0.47499999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77900000000000003,"5min":2.5630000000000002,"15min":6.0629999999999997},"last":0.53900000000000003}]},{"osd":4,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.47399999999999998,"15min":0.46999999999999997},"min":{"1min":0.35599999999999998,"5min":0.28100000000000003,"15min":0.26300000000000001},"max":{"1min":0.56599999999999995,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.503,"15min":0.495},"min":{"1min":0.36899999999999999,"5min":0.33000000000000002,"15min":0.28100000000000003},"max":{"1min":0.69999999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.375}]},{"osd":6,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.31,"15min":0.27000000000000002},"max":{"1min":0.66600000000000004,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53200000000000003,"15min":0.52700000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.29299999999999998},"max":{"1min":0.71899999999999997,"5min":1.99,"15min":1.99},"last":0.32800000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.54900000000000004},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.76200000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33300000000000002},"max":{"1min":0.81499999999999995,"5min":0.82199999999999995,"15min":0.83099999999999996},"last":0.38400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:45:58.047 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3f is not active+clean 2024-01-26T23:45:58.047 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.047 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3c is not active+clean 2024-01-26T23:45:58.047 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.047 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3b is not active+clean 2024-01-26T23:45:58.047 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.047 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3a is not active+clean 2024-01-26T23:45:58.047 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.047 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.39 is not active+clean 2024-01-26T23:45:58.048 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.048 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.38 is not active+clean 2024-01-26T23:45:58.048 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.048 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.37 is not active+clean 2024-01-26T23:45:58.048 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.048 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.36 is not active+clean 2024-01-26T23:45:58.048 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.048 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.34 is not active+clean 2024-01-26T23:45:58.048 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.33 is not active+clean 2024-01-26T23:45:58.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.32 is not active+clean 2024-01-26T23:45:58.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.30 is not active+clean 2024-01-26T23:45:58.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2f is not active+clean 2024-01-26T23:45:58.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2d is not active+clean 2024-01-26T23:45:58.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2c is not active+clean 2024-01-26T23:45:58.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2b is not active+clean 2024-01-26T23:45:58.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2a is not active+clean 2024-01-26T23:45:58.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.29 is not active+clean 2024-01-26T23:45:58.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.26 is not active+clean 2024-01-26T23:45:58.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.25 is not active+clean 2024-01-26T23:45:58.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.24 is not active+clean 2024-01-26T23:45:58.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.22 is not active+clean 2024-01-26T23:45:58.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.20 is not active+clean 2024-01-26T23:45:58.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1f is not active+clean 2024-01-26T23:45:58.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.979235+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T05:02:21.377733+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-26T23:45:58.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1e is not active+clean 2024-01-26T23:45:58.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.983371+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:31:46.226628+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-26T23:45:58.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1a is not active+clean 2024-01-26T23:45:58.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1d is not active+clean 2024-01-26T23:45:58.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.983396+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:07:37.554401+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-26T23:45:58.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1b is not active+clean 2024-01-26T23:45:58.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1c is not active+clean 2024-01-26T23:45:58.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.982175+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-27T23:50:39.847921+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-26T23:45:58.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1b is not active+clean 2024-01-26T23:45:58.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.981367+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T03:08:58.845586+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-26T23:45:58.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1a is not active+clean 2024-01-26T23:45:58.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.985553+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T03:08:58.845586+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-26T23:45:58.053 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1e is not active+clean 2024-01-26T23:45:58.053 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.053 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1f is not active+clean 2024-01-26T23:45:58.053 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.053 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.18 is not active+clean 2024-01-26T23:45:58.053 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.979110+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T10:19:46.301437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-26T23:45:58.053 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.17 is not active+clean 2024-01-26T23:45:58.053 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.983305+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T11:24:28.011807+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-26T23:45:58.053 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.11 is not active+clean 2024-01-26T23:45:58.053 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.054 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.16 is not active+clean 2024-01-26T23:45:58.054 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.985296+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:37:57.061471+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-26T23:45:58.054 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.12 is not active+clean 2024-01-26T23:45:58.054 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.054 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.15 is not active+clean 2024-01-26T23:45:58.054 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.979863+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:57:41.012865+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-26T23:45:58.054 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.14 is not active+clean 2024-01-26T23:45:58.054 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.981744+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T09:25:26.901131+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-26T23:45:58.054 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.14 is not active+clean 2024-01-26T23:45:58.054 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.054 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.13 is not active+clean 2024-01-26T23:45:58.055 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.985171+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T11:21:06.318896+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-26T23:45:58.055 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.15 is not active+clean 2024-01-26T23:45:58.055 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.055 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.12 is not active+clean 2024-01-26T23:45:58.055 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.983160+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T10:05:19.929713+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-26T23:45:58.055 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.16 is not active+clean 2024-01-26T23:45:58.055 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.055 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.11 is not active+clean 2024-01-26T23:45:58.055 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.979710+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T03:21:55.884844+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-26T23:45:58.055 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.17 is not active+clean 2024-01-26T23:45:58.055 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.056 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.8 is not active+clean 2024-01-26T23:45:58.056 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.056 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.f is not active+clean 2024-01-26T23:45:58.056 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.982869+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T07:05:31.172516+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-26T23:45:58.056 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.9 is not active+clean 2024-01-26T23:45:58.056 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.056 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.e is not active+clean 2024-01-26T23:45:58.056 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.981303+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:37:57.061471+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-26T23:45:58.056 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.a is not active+clean 2024-01-26T23:45:58.056 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.057 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.d is not active+clean 2024-01-26T23:45:58.057 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.979140+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T04:32:28.510671+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-26T23:45:58.057 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.b is not active+clean 2024-01-26T23:45:58.057 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.057 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.b is not active+clean 2024-01-26T23:45:58.057 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.979695+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T08:59:59.450936+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-26T23:45:58.057 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.9 is not active+clean 2024-01-26T23:45:58.057 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.979861+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T09:25:22.235033+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-26T23:45:58.057 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.f is not active+clean 2024-01-26T23:45:58.057 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.058 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.8 is not active+clean 2024-01-26T23:45:58.058 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.979243+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-27T23:52:15.511373+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-26T23:45:58.058 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.0 is not active+clean 2024-01-26T23:45:58.058 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.058 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.7 is not active+clean 2024-01-26T23:45:58.058 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.985217+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T09:42:07.246862+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-26T23:45:58.058 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1 is not active+clean 2024-01-26T23:45:58.058 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.058 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2 is not active+clean 2024-01-26T23:45:58.058 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.058 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.5 is not active+clean 2024-01-26T23:45:58.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.982946+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:26:59.702876+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-26T23:45:58.059 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.4 is not active+clean 2024-01-26T23:45:58.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.982814+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T02:40:36.924823+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-26T23:45:58.059 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.4 is not active+clean 2024-01-26T23:45:58.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.059 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3 is not active+clean 2024-01-26T23:45:58.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.980793+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T07:11:21.591456+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-26T23:45:58.059 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.5 is not active+clean 2024-01-26T23:45:58.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.059 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2 is not active+clean 2024-01-26T23:45:58.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.979645+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T07:02:38.392302+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-26T23:45:58.060 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1 is not active+clean 2024-01-26T23:45:58.060 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.981239+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T02:54:17.875691+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-26T23:45:58.060 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.d is not active+clean 2024-01-26T23:45:58.060 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:45:51.976854+0000', 'last_change': '2024-01-26T23:45:51.976854+0000', 'last_active': '2024-01-26T23:45:51.976854+0000', 'last_peered': '2024-01-26T23:45:51.976854+0000', 'last_clean': '2024-01-26T23:45:51.976854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:51.976854+0000', 'last_undegraded': '2024-01-26T23:45:51.976854+0000', 'last_fullsized': '2024-01-26T23:45:51.976854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:51.976854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:45:58.060 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.a is not active+clean 2024-01-26T23:45:58.060 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:45:50.972340+0000', 'last_change': '2024-01-26T23:45:50.979123+0000', 'last_active': '2024-01-26T23:45:50.972340+0000', 'last_peered': '2024-01-26T23:45:50.972340+0000', 'last_clean': '2024-01-26T23:45:50.972340+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:45:50.972340+0000', 'last_undegraded': '2024-01-26T23:45:50.972340+0000', 'last_fullsized': '2024-01-26T23:45:50.972340+0000', 'mapping_epoch': 95, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 95, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'last_clean_scrub_stamp': '2024-01-26T23:45:50.972340+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T08:03:05.423948+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-26T23:45:58.061 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-26T23:45:58.110 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:58.109+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:58.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.218+0000 7fb73e087700 1 Processor -- start 2024-01-26T23:45:58.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.218+0000 7fb73e087700 1 -- start start 2024-01-26T23:45:58.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.219+0000 7fb73e087700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb738153ec0 0x7fb738154290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:58.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.219+0000 7fb73e087700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73815b140 con 0x7fb7380b22d0 2024-01-26T23:45:58.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.219+0000 7fb73e087700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73815a5c0 con 0x7fb738157630 2024-01-26T23:45:58.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.219+0000 7fb73e087700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb73815a8a0 con 0x7fb738153ec0 2024-01-26T23:45:58.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.219+0000 7fb737fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb7380b22d0 0x7fb7380b04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52186/0 (socket says 172.21.15.46:52186) 2024-01-26T23:45:58.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.219+0000 7fb737fff700 1 -- 172.21.15.46:0/2821542546 learned_addr learned my addr 172.21.15.46:0/2821542546 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:45:58.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.219+0000 7fb7377fe700 1 --2- 172.21.15.46:0/2821542546 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb738153ec0 0x7fb738154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:58.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.219+0000 7fb736ffd700 1 -- 172.21.15.46:0/2821542546 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292451916 0 0) 0x7fb73815b140 con 0x7fb7380b22d0 2024-01-26T23:45:58.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.219+0000 7fb736ffd700 1 -- 172.21.15.46:0/2821542546 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb724003570 con 0x7fb7380b22d0 2024-01-26T23:45:58.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.219+0000 7fb736ffd700 1 -- 172.21.15.46:0/2821542546 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588275024 0 0) 0x7fb73815a5c0 con 0x7fb738157630 2024-01-26T23:45:58.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.219+0000 7fb736ffd700 1 -- 172.21.15.46:0/2821542546 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb73815b140 con 0x7fb738157630 2024-01-26T23:45:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.219+0000 7fb7377fe700 1 -- 172.21.15.46:0/2821542546 >> v1:172.21.15.106:6789/0 conn(0x7fb738157630 legacy=0x7fb7380b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.219+0000 7fb7377fe700 1 -- 172.21.15.46:0/2821542546 >> v1:172.21.15.46:6789/0 conn(0x7fb7380b22d0 legacy=0x7fb7380b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.219+0000 7fb7377fe700 1 -- 172.21.15.46:0/2821542546 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73815ab80 con 0x7fb738153ec0 2024-01-26T23:45:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.220+0000 7fb7377fe700 1 --2- 172.21.15.46:0/2821542546 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb738153ec0 0x7fb738154290 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb728009910 tx=0x7fb7280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dc51e759992a74e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.220+0000 7fb736ffd700 1 -- 172.21.15.46:0/2821542546 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb728011630 con 0x7fb738153ec0 2024-01-26T23:45:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.220+0000 7fb736ffd700 1 -- 172.21.15.46:0/2821542546 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb728011790 con 0x7fb738153ec0 2024-01-26T23:45:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.220+0000 7fb736ffd700 1 -- 172.21.15.46:0/2821542546 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb728011b30 con 0x7fb738153ec0 2024-01-26T23:45:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.220+0000 7fb73e087700 1 -- 172.21.15.46:0/2821542546 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb738153ec0 msgr2=0x7fb738154290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.220+0000 7fb73e087700 1 --2- 172.21.15.46:0/2821542546 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb738153ec0 0x7fb738154290 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb728009910 tx=0x7fb7280091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.221+0000 7fb73e087700 1 -- 172.21.15.46:0/2821542546 shutdown_connections 2024-01-26T23:45:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.221+0000 7fb73e087700 1 --2- 172.21.15.46:0/2821542546 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb738153ec0 0x7fb738154290 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.221+0000 7fb73e087700 1 -- 172.21.15.46:0/2821542546 >> 172.21.15.46:0/2821542546 conn(0x7fb738069510 msgr2=0x7fb7380aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.221+0000 7fb73e087700 1 -- 172.21.15.46:0/2821542546 shutdown_connections 2024-01-26T23:45:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.221+0000 7fb73e087700 1 -- 172.21.15.46:0/2821542546 wait complete. 2024-01-26T23:45:58.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.222+0000 7fb73e087700 1 Processor -- start 2024-01-26T23:45:58.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.222+0000 7fb73e087700 1 -- start start 2024-01-26T23:45:58.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb73e087700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7380b22d0 0x7fb738143e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:58.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb73e087700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb738153ec0 0x7fb738155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:58.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb73e087700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb738157630 0x7fb738155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:58.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb73e087700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380ca760 con 0x7fb738157630 2024-01-26T23:45:58.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb73e087700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380caa40 con 0x7fb7380b22d0 2024-01-26T23:45:58.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb73e087700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380cad20 con 0x7fb738153ec0 2024-01-26T23:45:58.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb737fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb738157630 0x7fb738155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:58.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb737fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb738157630 0x7fb738155db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33508/0 (socket says 172.21.15.46:33508) 2024-01-26T23:45:58.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb737fff700 1 -- 172.21.15.46:0/50879105 learned_addr learned my addr 172.21.15.46:0/50879105 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:58.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb7377fe700 1 --2- 172.21.15.46:0/50879105 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7380b22d0 0x7fb738143e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:58.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb72ffff700 1 --2- 172.21.15.46:0/50879105 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb738153ec0 0x7fb738155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:58.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb737fff700 1 -- 172.21.15.46:0/50879105 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb738153ec0 msgr2=0x7fb738155870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:58.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb737fff700 1 --2- 172.21.15.46:0/50879105 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb738153ec0 0x7fb738155870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb737fff700 1 -- 172.21.15.46:0/50879105 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7380b22d0 msgr2=0x7fb738143e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:58.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb737fff700 1 --2- 172.21.15.46:0/50879105 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7380b22d0 0x7fb738143e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb737fff700 1 -- 172.21.15.46:0/50879105 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb728009420 con 0x7fb738157630 2024-01-26T23:45:58.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb7377fe700 1 --2- 172.21.15.46:0/50879105 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7380b22d0 0x7fb738143e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:58.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb737fff700 1 --2- 172.21.15.46:0/50879105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb738157630 0x7fb738155db0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fb730002250 tx=0x7fb7300019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:58.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb72f7fe700 1 -- 172.21.15.46:0/50879105 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73000a7e0 con 0x7fb738157630 2024-01-26T23:45:58.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb72f7fe700 1 -- 172.21.15.46:0/50879105 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73000a940 con 0x7fb738157630 2024-01-26T23:45:58.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.223+0000 7fb73e087700 1 -- 172.21.15.46:0/50879105 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7380cb000 con 0x7fb738157630 2024-01-26T23:45:58.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.224+0000 7fb72f7fe700 1 -- 172.21.15.46:0/50879105 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73000ad00 con 0x7fb738157630 2024-01-26T23:45:58.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.224+0000 7fb73e087700 1 -- 172.21.15.46:0/50879105 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7381443b0 con 0x7fb738157630 2024-01-26T23:45:58.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.225+0000 7fb73e087700 1 -- 172.21.15.46:0/50879105 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7380cb520 con 0x7fb738157630 2024-01-26T23:45:58.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.225+0000 7fb72f7fe700 1 -- 172.21.15.46:0/50879105 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fb730007300 con 0x7fb738157630 2024-01-26T23:45:58.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.226+0000 7fb72f7fe700 1 --2- 172.21.15.46:0/50879105 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb718066ea0 0x7fb718069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:58.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.226+0000 7fb72f7fe700 1 -- 172.21.15.46:0/50879105 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fb730084900 con 0x7fb738157630 2024-01-26T23:45:58.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.226+0000 7fb7377fe700 1 --2- 172.21.15.46:0/50879105 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb718066ea0 0x7fb718069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:58.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.226+0000 7fb7377fe700 1 --2- 172.21.15.46:0/50879105 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb718066ea0 0x7fb718069350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb728009860 tx=0x7fb728017000 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:58.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.230+0000 7fb72f7fe700 1 -- 172.21.15.46:0/50879105 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb7380cb520 con 0x7fb738157630 2024-01-26T23:45:58.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.526+0000 7fb73e087700 1 -- 172.21.15.46:0/50879105 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fb7380cb520 con 0x7fb738157630 2024-01-26T23:45:58.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.527+0000 7fb72f7fe700 1 -- 172.21.15.46:0/50879105 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1195 (secure 0 0 0) 0x7fb7380cb520 con 0x7fb738157630 2024-01-26T23:45:58.527 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:58.527 INFO:teuthology.orchestra.run.smithi046.stdout:{"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":991,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":98,"num_osds":8,"num_up_osds":8,"osd_up_since":1706311777,"num_in_osds":8,"osd_in_since":1706311769,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":44},{"state_name":"creating+peering","count":27},{"state_name":"active+clean","count":26}],"num_pgs":97,"num_pools":3,"num_objects":6,"data_bytes":590368,"bytes_used":2496290816,"bytes_avail":770597822464,"bytes_total":773094113280,"unknown_pgs_ratio":0.45360824465751648,"inactive_pgs_ratio":0.27835050225257874,"read_bytes_sec":1392,"write_bytes_sec":0,"read_op_per_sec":1,"write_op_per_sec":2},"fsmap":{"epoch":45,"id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"c","status":"up:active","gid":6631}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-26T23:44:22.571535+0000","services":{}},"progress_events":{}} 2024-01-26T23:45:58.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.529+0000 7fb73e087700 1 -- 172.21.15.46:0/50879105 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb718066ea0 msgr2=0x7fb718069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:58.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.529+0000 7fb73e087700 1 --2- 172.21.15.46:0/50879105 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb718066ea0 0x7fb718069350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb728009860 tx=0x7fb728017000 comp rx=0 tx=0).stop 2024-01-26T23:45:58.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.529+0000 7fb73e087700 1 -- 172.21.15.46:0/50879105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb738157630 msgr2=0x7fb738155db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:58.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.529+0000 7fb73e087700 1 --2- 172.21.15.46:0/50879105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb738157630 0x7fb738155db0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fb730002250 tx=0x7fb7300019c0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.529+0000 7fb73e087700 1 -- 172.21.15.46:0/50879105 shutdown_connections 2024-01-26T23:45:58.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.529+0000 7fb73e087700 1 --2- 172.21.15.46:0/50879105 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb718066ea0 0x7fb718069350 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.529+0000 7fb73e087700 1 --2- 172.21.15.46:0/50879105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb738157630 0x7fb738155db0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.529+0000 7fb73e087700 1 --2- 172.21.15.46:0/50879105 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7380b22d0 0x7fb738143e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.529+0000 7fb73e087700 1 --2- 172.21.15.46:0/50879105 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb738153ec0 0x7fb738155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.529+0000 7fb73e087700 1 -- 172.21.15.46:0/50879105 >> 172.21.15.46:0/50879105 conn(0x7fb738069510 msgr2=0x7fb738151e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:58.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.530+0000 7fb73e087700 1 -- 172.21.15.46:0/50879105 shutdown_connections 2024-01-26T23:45:58.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.530+0000 7fb73e087700 1 -- 172.21.15.46:0/50879105 wait complete. 2024-01-26T23:45:58.543 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-26T23:45:58.544 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:45:58.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.681+0000 7f64901b6700 1 Processor -- start 2024-01-26T23:45:58.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.681+0000 7f64901b6700 1 -- start start 2024-01-26T23:45:58.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.682+0000 7f64901b6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6488157340 0x7f648815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:58.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.682+0000 7f64901b6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f64880ae520 0x7f64880b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:58.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.682+0000 7f64901b6700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6488158ae0 con 0x7f64880b3fb0 2024-01-26T23:45:58.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.682+0000 7f64901b6700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64880b4430 con 0x7f64880ae520 2024-01-26T23:45:58.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.682+0000 7f64901b6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648815b650 con 0x7f6488157340 2024-01-26T23:45:58.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.682+0000 7f648d751700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f64880ae520 0x7f64880b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:58.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.682+0000 7f648d751700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f64880ae520 0x7f64880b3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33512/0 (socket says 172.21.15.46:33512) 2024-01-26T23:45:58.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.682+0000 7f648d751700 1 -- 172.21.15.46:0/1825436816 learned_addr learned my addr 172.21.15.46:0/1825436816 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:58.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.682+0000 7f648df52700 1 --2- 172.21.15.46:0/1825436816 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6488157340 0x7f648815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:58.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.683+0000 7f648d751700 1 -- 172.21.15.46:0/1825436816 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6488157340 msgr2=0x7f648815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:45:58.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.683+0000 7f648d751700 1 --2- 172.21.15.46:0/1825436816 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6488157340 0x7f648815bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.683+0000 7f648cf50700 1 -- 172.21.15.46:0/1825436816 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154608773 0 0) 0x7f6488158ae0 con 0x7f64880b3fb0 2024-01-26T23:45:58.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.683+0000 7f648d751700 1 -- 172.21.15.46:0/1825436816 >> v1:172.21.15.106:6789/0 conn(0x7f64880b3fb0 legacy=0x7f6488157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:58.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.683+0000 7f648d751700 1 -- 172.21.15.46:0/1825436816 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648815c020 con 0x7f64880ae520 2024-01-26T23:45:58.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.683+0000 7f648d751700 1 --2- 172.21.15.46:0/1825436816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64880ae520 0x7f64880b3a70 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f64780099b0 tx=0x7f6478009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8362793dde83d696 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:58.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.683+0000 7f648cf50700 1 -- 172.21.15.46:0/1825436816 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6478009ea0 con 0x7f64880ae520 2024-01-26T23:45:58.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.683+0000 7f648cf50700 1 -- 172.21.15.46:0/1825436816 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647800b860 con 0x7f64880ae520 2024-01-26T23:45:58.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.683+0000 7f648cf50700 1 -- 172.21.15.46:0/1825436816 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647800bc20 con 0x7f64880ae520 2024-01-26T23:45:58.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.684+0000 7f64901b6700 1 -- 172.21.15.46:0/1825436816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64880ae520 msgr2=0x7f64880b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:58.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.684+0000 7f64901b6700 1 --2- 172.21.15.46:0/1825436816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64880ae520 0x7f64880b3a70 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f64780099b0 tx=0x7f6478009250 comp rx=0 tx=0).stop 2024-01-26T23:45:58.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.684+0000 7f64901b6700 1 -- 172.21.15.46:0/1825436816 shutdown_connections 2024-01-26T23:45:58.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.684+0000 7f64901b6700 1 --2- 172.21.15.46:0/1825436816 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6488157340 0x7f648815bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.684+0000 7f64901b6700 1 --2- 172.21.15.46:0/1825436816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64880ae520 0x7f64880b3a70 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.684+0000 7f64901b6700 1 -- 172.21.15.46:0/1825436816 >> 172.21.15.46:0/1825436816 conn(0x7f64880ce530 msgr2=0x7f64880ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:58.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.684+0000 7f64901b6700 1 -- 172.21.15.46:0/1825436816 shutdown_connections 2024-01-26T23:45:58.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.684+0000 7f64901b6700 1 -- 172.21.15.46:0/1825436816 wait complete. 2024-01-26T23:45:58.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.685+0000 7f64901b6700 1 Processor -- start 2024-01-26T23:45:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.685+0000 7f64901b6700 1 -- start start 2024-01-26T23:45:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.686+0000 7f64901b6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64880ae520 0x7f64881569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.686+0000 7f64901b6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64880b3fb0 0x7f6488069cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.686+0000 7f64901b6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6488157340 0x7f64880b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.686+0000 7f64901b6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648815cfc0 con 0x7f64880b3fb0 2024-01-26T23:45:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.686+0000 7f64901b6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648815c3b0 con 0x7f64880ae520 2024-01-26T23:45:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.686+0000 7f64901b6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648815c690 con 0x7f6488157340 2024-01-26T23:45:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.686+0000 7f648d751700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64880b3fb0 0x7f6488069cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.686+0000 7f648d751700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64880b3fb0 0x7f6488069cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33516/0 (socket says 172.21.15.46:33516) 2024-01-26T23:45:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.686+0000 7f648d751700 1 -- 172.21.15.46:0/2027873889 learned_addr learned my addr 172.21.15.46:0/2027873889 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:45:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.686+0000 7f648df52700 1 --2- 172.21.15.46:0/2027873889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64880ae520 0x7f64881569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.686+0000 7f648e753700 1 --2- 172.21.15.46:0/2027873889 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6488157340 0x7f64880b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.686+0000 7f648d751700 1 -- 172.21.15.46:0/2027873889 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6488157340 msgr2=0x7f64880b22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:58.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.686+0000 7f648d751700 1 --2- 172.21.15.46:0/2027873889 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6488157340 0x7f64880b22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.686+0000 7f648d751700 1 -- 172.21.15.46:0/2027873889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64880ae520 msgr2=0x7f64881569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:58.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.686+0000 7f648d751700 1 --2- 172.21.15.46:0/2027873889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64880ae520 0x7f64881569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.686+0000 7f648d751700 1 -- 172.21.15.46:0/2027873889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64780094f0 con 0x7f64880b3fb0 2024-01-26T23:45:58.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.686+0000 7f648df52700 1 --2- 172.21.15.46:0/2027873889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64880ae520 0x7f64881569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:45:58.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.687+0000 7f648d751700 1 --2- 172.21.15.46:0/2027873889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64880b3fb0 0x7f6488069cf0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f6478006fd0 tx=0x7f6478017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:58.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.687+0000 7f647dffb700 1 -- 172.21.15.46:0/2027873889 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647801f070 con 0x7f64880b3fb0 2024-01-26T23:45:58.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.687+0000 7f647dffb700 1 -- 172.21.15.46:0/2027873889 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647800b9d0 con 0x7f64880b3fb0 2024-01-26T23:45:58.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.687+0000 7f64901b6700 1 -- 172.21.15.46:0/2027873889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648815c970 con 0x7f64880b3fb0 2024-01-26T23:45:58.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.687+0000 7f647dffb700 1 -- 172.21.15.46:0/2027873889 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6478011650 con 0x7f64880b3fb0 2024-01-26T23:45:58.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.687+0000 7f64901b6700 1 -- 172.21.15.46:0/2027873889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6488156f20 con 0x7f64880b3fb0 2024-01-26T23:45:58.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.688+0000 7f64901b6700 1 -- 172.21.15.46:0/2027873889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648804c710 con 0x7f64880b3fb0 2024-01-26T23:45:58.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.689+0000 7f647dffb700 1 -- 172.21.15.46:0/2027873889 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f6478011a80 con 0x7f64880b3fb0 2024-01-26T23:45:58.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.689+0000 7f647dffb700 1 --2- 172.21.15.46:0/2027873889 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f646c066ea0 0x7f646c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:45:58.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.689+0000 7f647dffb700 1 -- 172.21.15.46:0/2027873889 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f6478085a30 con 0x7f64880b3fb0 2024-01-26T23:45:58.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.689+0000 7f648df52700 1 --2- 172.21.15.46:0/2027873889 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f646c066ea0 0x7f646c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:45:58.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.690+0000 7f648df52700 1 --2- 172.21.15.46:0/2027873889 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f646c066ea0 0x7f646c069350 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f64840113e0 tx=0x7f64840091a0 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:45:58.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.694+0000 7f647dffb700 1 -- 172.21.15.46:0/2027873889 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6478052b30 con 0x7f64880b3fb0 2024-01-26T23:45:58.832 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:58.831+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:45:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.900+0000 7f64901b6700 1 -- 172.21.15.46:0/2027873889 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f648815d100 con 0x7f646c066ea0 2024-01-26T23:45:58.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.908+0000 7f647dffb700 1 -- 172.21.15.46:0/2027873889 <== mgr.6421 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289596 (secure 0 0 0) 0x7f648815d100 con 0x7f646c066ea0 2024-01-26T23:45:58.912 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:45:58.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.913+0000 7f64901b6700 1 -- 172.21.15.46:0/2027873889 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f646c066ea0 msgr2=0x7f646c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:58.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.913+0000 7f64901b6700 1 --2- 172.21.15.46:0/2027873889 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f646c066ea0 0x7f646c069350 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f64840113e0 tx=0x7f64840091a0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.913+0000 7f64901b6700 1 -- 172.21.15.46:0/2027873889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64880b3fb0 msgr2=0x7f6488069cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:45:58.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.913+0000 7f64901b6700 1 --2- 172.21.15.46:0/2027873889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64880b3fb0 0x7f6488069cf0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f6478006fd0 tx=0x7f6478017000 comp rx=0 tx=0).stop 2024-01-26T23:45:58.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.913+0000 7f64901b6700 1 -- 172.21.15.46:0/2027873889 shutdown_connections 2024-01-26T23:45:58.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.913+0000 7f64901b6700 1 --2- 172.21.15.46:0/2027873889 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f646c066ea0 0x7f646c069350 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.913+0000 7f64901b6700 1 --2- 172.21.15.46:0/2027873889 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6488157340 0x7f64880b22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.913+0000 7f64901b6700 1 --2- 172.21.15.46:0/2027873889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64880ae520 0x7f64881569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.913+0000 7f64901b6700 1 --2- 172.21.15.46:0/2027873889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64880b3fb0 0x7f6488069cf0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:45:58.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.913+0000 7f64901b6700 1 -- 172.21.15.46:0/2027873889 >> 172.21.15.46:0/2027873889 conn(0x7f64880ce530 msgr2=0x7f64880afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:45:58.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.913+0000 7f64901b6700 1 -- 172.21.15.46:0/2027873889 shutdown_connections 2024-01-26T23:45:58.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:45:58.913+0000 7f64901b6700 1 -- 172.21.15.46:0/2027873889 wait complete. 2024-01-26T23:45:58.915 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:45:58.926 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-26T23:45:58.861195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":592,"num_read_kb":1131,"num_write":297,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2235,"ondisk_log_size":2235,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":184,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2429864,"kb_used_data":10600,"kb_used_omap":0,"kb_used_meta":2419200,"kb_avail":752544856,"statfs":{"total":773094113280,"available":770605932544,"internally_reserved":0,"allocated":10854400,"data_stored":6044228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2477260800},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.881546"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002073+0000","last_change":"2024-01-26T23:45:52.999075+0000","last_active":"2024-01-26T23:45:53.002073+0000","last_peered":"2024-01-26T23:45:53.002073+0000","last_clean":"2024-01-26T23:45:53.002073+0000","last_became_active":"2024-01-26T23:45:52.998797+0000","last_became_peered":"2024-01-26T23:45:52.998797+0000","last_unstale":"2024-01-26T23:45:53.002073+0000","last_undegraded":"2024-01-26T23:45:53.002073+0000","last_fullsized":"2024-01-26T23:45:53.002073+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:31:10.818408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.009542+0000","last_change":"2024-01-26T23:45:52.998874+0000","last_active":"2024-01-26T23:45:53.009542+0000","last_peered":"2024-01-26T23:45:53.009542+0000","last_clean":"2024-01-26T23:45:53.009542+0000","last_became_active":"2024-01-26T23:45:52.998533+0000","last_became_peered":"2024-01-26T23:45:52.998533+0000","last_unstale":"2024-01-26T23:45:53.009542+0000","last_undegraded":"2024-01-26T23:45:53.009542+0000","last_fullsized":"2024-01-26T23:45:53.009542+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:22:33.448776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.006656+0000","last_change":"2024-01-26T23:45:53.004780+0000","last_active":"2024-01-26T23:45:53.006656+0000","last_peered":"2024-01-26T23:45:53.006656+0000","last_clean":"2024-01-26T23:45:53.006656+0000","last_became_active":"2024-01-26T23:45:53.004486+0000","last_became_peered":"2024-01-26T23:45:53.004486+0000","last_unstale":"2024-01-26T23:45:53.006656+0000","last_undegraded":"2024-01-26T23:45:53.006656+0000","last_fullsized":"2024-01-26T23:45:53.006656+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:52:16.515887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.993881+0000","last_change":"2024-01-26T23:45:52.991250+0000","last_active":"2024-01-26T23:45:52.993881+0000","last_peered":"2024-01-26T23:45:52.993881+0000","last_clean":"2024-01-26T23:45:52.993881+0000","last_became_active":"2024-01-26T23:45:52.990993+0000","last_became_peered":"2024-01-26T23:45:52.990993+0000","last_unstale":"2024-01-26T23:45:52.993881+0000","last_undegraded":"2024-01-26T23:45:52.993881+0000","last_fullsized":"2024-01-26T23:45:52.993881+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:25.322381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.000409+0000","last_change":"2024-01-26T23:45:52.999059+0000","last_active":"2024-01-26T23:45:53.000409+0000","last_peered":"2024-01-26T23:45:53.000409+0000","last_clean":"2024-01-26T23:45:53.000409+0000","last_became_active":"2024-01-26T23:45:52.998752+0000","last_became_peered":"2024-01-26T23:45:52.998752+0000","last_unstale":"2024-01-26T23:45:53.000409+0000","last_undegraded":"2024-01-26T23:45:53.000409+0000","last_fullsized":"2024-01-26T23:45:53.000409+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:22:13.283742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005904+0000","last_change":"2024-01-26T23:45:53.003311+0000","last_active":"2024-01-26T23:45:53.005904+0000","last_peered":"2024-01-26T23:45:53.005904+0000","last_clean":"2024-01-26T23:45:53.005904+0000","last_became_active":"2024-01-26T23:45:53.003002+0000","last_became_peered":"2024-01-26T23:45:53.003002+0000","last_unstale":"2024-01-26T23:45:53.005904+0000","last_undegraded":"2024-01-26T23:45:53.005904+0000","last_fullsized":"2024-01-26T23:45:53.005904+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:33:04.855324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005541+0000","last_change":"2024-01-26T23:45:53.003341+0000","last_active":"2024-01-26T23:45:53.005541+0000","last_peered":"2024-01-26T23:45:53.005541+0000","last_clean":"2024-01-26T23:45:53.005541+0000","last_became_active":"2024-01-26T23:45:53.003049+0000","last_became_peered":"2024-01-26T23:45:53.003049+0000","last_unstale":"2024-01-26T23:45:53.005541+0000","last_undegraded":"2024-01-26T23:45:53.005541+0000","last_fullsized":"2024-01-26T23:45:53.005541+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:25.322381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.000275+0000","last_change":"2024-01-26T23:45:52.999099+0000","last_active":"2024-01-26T23:45:53.000275+0000","last_peered":"2024-01-26T23:45:53.000275+0000","last_clean":"2024-01-26T23:45:53.000275+0000","last_became_active":"2024-01-26T23:45:52.998795+0000","last_became_peered":"2024-01-26T23:45:52.998795+0000","last_unstale":"2024-01-26T23:45:53.000275+0000","last_undegraded":"2024-01-26T23:45:53.000275+0000","last_fullsized":"2024-01-26T23:45:53.000275+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:10:23.293703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010733+0000","last_change":"2024-01-26T23:45:53.005345+0000","last_active":"2024-01-26T23:45:53.010733+0000","last_peered":"2024-01-26T23:45:53.010733+0000","last_clean":"2024-01-26T23:45:53.010733+0000","last_became_active":"2024-01-26T23:45:53.005093+0000","last_became_peered":"2024-01-26T23:45:53.005093+0000","last_unstale":"2024-01-26T23:45:53.010733+0000","last_undegraded":"2024-01-26T23:45:53.010733+0000","last_fullsized":"2024-01-26T23:45:53.010733+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:53.299917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.997619+0000","last_change":"2024-01-26T23:45:52.993343+0000","last_active":"2024-01-26T23:45:52.997619+0000","last_peered":"2024-01-26T23:45:52.997619+0000","last_clean":"2024-01-26T23:45:52.997619+0000","last_became_active":"2024-01-26T23:45:52.993136+0000","last_became_peered":"2024-01-26T23:45:52.993136+0000","last_unstale":"2024-01-26T23:45:52.997619+0000","last_undegraded":"2024-01-26T23:45:52.997619+0000","last_fullsized":"2024-01-26T23:45:52.997619+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:21:21.636031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.013956+0000","last_change":"2024-01-26T23:45:53.010404+0000","last_active":"2024-01-26T23:45:53.013956+0000","last_peered":"2024-01-26T23:45:53.013956+0000","last_clean":"2024-01-26T23:45:53.013956+0000","last_became_active":"2024-01-26T23:45:53.010120+0000","last_became_peered":"2024-01-26T23:45:53.010120+0000","last_unstale":"2024-01-26T23:45:53.013956+0000","last_undegraded":"2024-01-26T23:45:53.013956+0000","last_fullsized":"2024-01-26T23:45:53.013956+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:58:27.010243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002076+0000","last_change":"2024-01-26T23:45:53.000714+0000","last_active":"2024-01-26T23:45:53.002076+0000","last_peered":"2024-01-26T23:45:53.002076+0000","last_clean":"2024-01-26T23:45:53.002076+0000","last_became_active":"2024-01-26T23:45:53.000514+0000","last_became_peered":"2024-01-26T23:45:53.000514+0000","last_unstale":"2024-01-26T23:45:53.002076+0000","last_undegraded":"2024-01-26T23:45:53.002076+0000","last_fullsized":"2024-01-26T23:45:53.002076+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:32:39.694027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.008249+0000","last_change":"2024-01-26T23:45:53.004100+0000","last_active":"2024-01-26T23:45:53.008249+0000","last_peered":"2024-01-26T23:45:53.008249+0000","last_clean":"2024-01-26T23:45:53.008249+0000","last_became_active":"2024-01-26T23:45:53.003792+0000","last_became_peered":"2024-01-26T23:45:53.003792+0000","last_unstale":"2024-01-26T23:45:53.008249+0000","last_undegraded":"2024-01-26T23:45:53.008249+0000","last_fullsized":"2024-01-26T23:45:53.008249+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:54:42.637169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005453+0000","last_change":"2024-01-26T23:45:53.002596+0000","last_active":"2024-01-26T23:45:53.005453+0000","last_peered":"2024-01-26T23:45:53.005453+0000","last_clean":"2024-01-26T23:45:53.005453+0000","last_became_active":"2024-01-26T23:45:53.002292+0000","last_became_peered":"2024-01-26T23:45:53.002292+0000","last_unstale":"2024-01-26T23:45:53.005453+0000","last_undegraded":"2024-01-26T23:45:53.005453+0000","last_fullsized":"2024-01-26T23:45:53.005453+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:01.271422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001869+0000","last_change":"2024-01-26T23:45:52.993487+0000","last_active":"2024-01-26T23:45:53.001869+0000","last_peered":"2024-01-26T23:45:53.001869+0000","last_clean":"2024-01-26T23:45:53.001869+0000","last_became_active":"2024-01-26T23:45:52.993157+0000","last_became_peered":"2024-01-26T23:45:52.993157+0000","last_unstale":"2024-01-26T23:45:53.001869+0000","last_undegraded":"2024-01-26T23:45:53.001869+0000","last_fullsized":"2024-01-26T23:45:53.001869+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:44:24.788145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002037+0000","last_change":"2024-01-26T23:45:52.999171+0000","last_active":"2024-01-26T23:45:53.002037+0000","last_peered":"2024-01-26T23:45:53.002037+0000","last_clean":"2024-01-26T23:45:53.002037+0000","last_became_active":"2024-01-26T23:45:52.998914+0000","last_became_peered":"2024-01-26T23:45:52.998914+0000","last_unstale":"2024-01-26T23:45:53.002037+0000","last_undegraded":"2024-01-26T23:45:53.002037+0000","last_fullsized":"2024-01-26T23:45:53.002037+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:01.271422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.999397+0000","last_change":"2024-01-26T23:45:52.996490+0000","last_active":"2024-01-26T23:45:52.999397+0000","last_peered":"2024-01-26T23:45:52.999397+0000","last_clean":"2024-01-26T23:45:52.999397+0000","last_became_active":"2024-01-26T23:45:52.996165+0000","last_became_peered":"2024-01-26T23:45:52.996165+0000","last_unstale":"2024-01-26T23:45:52.999397+0000","last_undegraded":"2024-01-26T23:45:52.999397+0000","last_fullsized":"2024-01-26T23:45:52.999397+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:10:03.768025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002692+0000","last_change":"2024-01-26T23:45:52.995031+0000","last_active":"2024-01-26T23:45:53.002692+0000","last_peered":"2024-01-26T23:45:53.002692+0000","last_clean":"2024-01-26T23:45:53.002692+0000","last_became_active":"2024-01-26T23:45:52.994711+0000","last_became_peered":"2024-01-26T23:45:52.994711+0000","last_unstale":"2024-01-26T23:45:53.002692+0000","last_undegraded":"2024-01-26T23:45:53.002692+0000","last_fullsized":"2024-01-26T23:45:53.002692+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:20.934227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.014705+0000","last_change":"2024-01-26T23:45:53.010620+0000","last_active":"2024-01-26T23:45:53.014705+0000","last_peered":"2024-01-26T23:45:53.014705+0000","last_clean":"2024-01-26T23:45:53.014705+0000","last_became_active":"2024-01-26T23:45:53.009789+0000","last_became_peered":"2024-01-26T23:45:53.009789+0000","last_unstale":"2024-01-26T23:45:53.014705+0000","last_undegraded":"2024-01-26T23:45:53.014705+0000","last_fullsized":"2024-01-26T23:45:53.014705+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:31:10.818408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.999364+0000","last_change":"2024-01-26T23:45:52.997158+0000","last_active":"2024-01-26T23:45:52.999364+0000","last_peered":"2024-01-26T23:45:52.999364+0000","last_clean":"2024-01-26T23:45:52.999364+0000","last_became_active":"2024-01-26T23:45:52.996822+0000","last_became_peered":"2024-01-26T23:45:52.996822+0000","last_unstale":"2024-01-26T23:45:52.999364+0000","last_undegraded":"2024-01-26T23:45:52.999364+0000","last_fullsized":"2024-01-26T23:45:52.999364+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:57.542006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002691+0000","last_change":"2024-01-26T23:45:53.000510+0000","last_active":"2024-01-26T23:45:53.002691+0000","last_peered":"2024-01-26T23:45:53.002691+0000","last_clean":"2024-01-26T23:45:53.002691+0000","last_became_active":"2024-01-26T23:45:53.000231+0000","last_became_peered":"2024-01-26T23:45:53.000231+0000","last_unstale":"2024-01-26T23:45:53.002691+0000","last_undegraded":"2024-01-26T23:45:53.002691+0000","last_fullsized":"2024-01-26T23:45:53.002691+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:12:46.926098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.009952+0000","last_change":"2024-01-26T23:45:53.006604+0000","last_active":"2024-01-26T23:45:53.009952+0000","last_peered":"2024-01-26T23:45:53.009952+0000","last_clean":"2024-01-26T23:45:53.009952+0000","last_became_active":"2024-01-26T23:45:53.006289+0000","last_became_peered":"2024-01-26T23:45:53.006289+0000","last_unstale":"2024-01-26T23:45:53.009952+0000","last_undegraded":"2024-01-26T23:45:53.009952+0000","last_fullsized":"2024-01-26T23:45:53.009952+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:19:33.562254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002680+0000","last_change":"2024-01-26T23:45:52.998559+0000","last_active":"2024-01-26T23:45:53.002680+0000","last_peered":"2024-01-26T23:45:53.002680+0000","last_clean":"2024-01-26T23:45:53.002680+0000","last_became_active":"2024-01-26T23:45:52.998284+0000","last_became_peered":"2024-01-26T23:45:52.998284+0000","last_unstale":"2024-01-26T23:45:53.002680+0000","last_undegraded":"2024-01-26T23:45:53.002680+0000","last_fullsized":"2024-01-26T23:45:53.002680+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:23.239547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010618+0000","last_change":"2024-01-26T23:45:53.002633+0000","last_active":"2024-01-26T23:45:53.010618+0000","last_peered":"2024-01-26T23:45:53.010618+0000","last_clean":"2024-01-26T23:45:53.010618+0000","last_became_active":"2024-01-26T23:45:53.002349+0000","last_became_peered":"2024-01-26T23:45:53.002349+0000","last_unstale":"2024-01-26T23:45:53.010618+0000","last_undegraded":"2024-01-26T23:45:53.010618+0000","last_fullsized":"2024-01-26T23:45:53.010618+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:48:28.528692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.011887+0000","last_change":"2024-01-26T23:45:53.006070+0000","last_active":"2024-01-26T23:45:53.011887+0000","last_peered":"2024-01-26T23:45:53.011887+0000","last_clean":"2024-01-26T23:45:53.011887+0000","last_became_active":"2024-01-26T23:45:53.005736+0000","last_became_peered":"2024-01-26T23:45:53.005736+0000","last_unstale":"2024-01-26T23:45:53.011887+0000","last_undegraded":"2024-01-26T23:45:53.011887+0000","last_fullsized":"2024-01-26T23:45:53.011887+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:57:13.832289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002050+0000","last_change":"2024-01-26T23:45:53.001275+0000","last_active":"2024-01-26T23:45:53.002050+0000","last_peered":"2024-01-26T23:45:53.002050+0000","last_clean":"2024-01-26T23:45:53.002050+0000","last_became_active":"2024-01-26T23:45:53.001040+0000","last_became_peered":"2024-01-26T23:45:53.001040+0000","last_unstale":"2024-01-26T23:45:53.002050+0000","last_undegraded":"2024-01-26T23:45:53.002050+0000","last_fullsized":"2024-01-26T23:45:53.002050+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:53.299917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.003152+0000","last_change":"2024-01-26T23:45:53.001405+0000","last_active":"2024-01-26T23:45:53.003152+0000","last_peered":"2024-01-26T23:45:53.003152+0000","last_clean":"2024-01-26T23:45:53.003152+0000","last_became_active":"2024-01-26T23:45:53.001141+0000","last_became_peered":"2024-01-26T23:45:53.001141+0000","last_unstale":"2024-01-26T23:45:53.003152+0000","last_undegraded":"2024-01-26T23:45:53.003152+0000","last_fullsized":"2024-01-26T23:45:53.003152+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:46:16.885651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.999343+0000","last_change":"2024-01-26T23:45:52.996567+0000","last_active":"2024-01-26T23:45:52.999343+0000","last_peered":"2024-01-26T23:45:52.999343+0000","last_clean":"2024-01-26T23:45:52.999343+0000","last_became_active":"2024-01-26T23:45:52.996241+0000","last_became_peered":"2024-01-26T23:45:52.996241+0000","last_unstale":"2024-01-26T23:45:52.999343+0000","last_undegraded":"2024-01-26T23:45:52.999343+0000","last_fullsized":"2024-01-26T23:45:52.999343+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:22:32.327755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001218+0000","last_change":"2024-01-26T23:45:52.994893+0000","last_active":"2024-01-26T23:45:53.001218+0000","last_peered":"2024-01-26T23:45:53.001218+0000","last_clean":"2024-01-26T23:45:53.001218+0000","last_became_active":"2024-01-26T23:45:52.994613+0000","last_became_peered":"2024-01-26T23:45:52.994613+0000","last_unstale":"2024-01-26T23:45:53.001218+0000","last_undegraded":"2024-01-26T23:45:53.001218+0000","last_fullsized":"2024-01-26T23:45:53.001218+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:09:09.561092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010916+0000","last_change":"2024-01-26T23:45:53.010217+0000","last_active":"2024-01-26T23:45:53.010916+0000","last_peered":"2024-01-26T23:45:53.010916+0000","last_clean":"2024-01-26T23:45:53.010916+0000","last_became_active":"2024-01-26T23:45:53.010005+0000","last_became_peered":"2024-01-26T23:45:53.010005+0000","last_unstale":"2024-01-26T23:45:53.010916+0000","last_undegraded":"2024-01-26T23:45:53.010916+0000","last_fullsized":"2024-01-26T23:45:53.010916+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:33:04.855324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002434+0000","last_change":"2024-01-26T23:45:52.997178+0000","last_active":"2024-01-26T23:45:53.002434+0000","last_peered":"2024-01-26T23:45:53.002434+0000","last_clean":"2024-01-26T23:45:53.002434+0000","last_became_active":"2024-01-26T23:45:52.996868+0000","last_became_peered":"2024-01-26T23:45:52.996868+0000","last_unstale":"2024-01-26T23:45:53.002434+0000","last_undegraded":"2024-01-26T23:45:53.002434+0000","last_fullsized":"2024-01-26T23:45:53.002434+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:07:38.558915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001885+0000","last_change":"2024-01-26T23:45:52.999010+0000","last_active":"2024-01-26T23:45:53.001885+0000","last_peered":"2024-01-26T23:45:53.001885+0000","last_clean":"2024-01-26T23:45:53.001885+0000","last_became_active":"2024-01-26T23:45:52.998758+0000","last_became_peered":"2024-01-26T23:45:52.998758+0000","last_unstale":"2024-01-26T23:45:53.001885+0000","last_undegraded":"2024-01-26T23:45:53.001885+0000","last_fullsized":"2024-01-26T23:45:53.001885+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:57:56.663833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005249+0000","last_change":"2024-01-26T23:45:52.999818+0000","last_active":"2024-01-26T23:45:53.005249+0000","last_peered":"2024-01-26T23:45:53.005249+0000","last_clean":"2024-01-26T23:45:53.005249+0000","last_became_active":"2024-01-26T23:45:52.999503+0000","last_became_peered":"2024-01-26T23:45:52.999503+0000","last_unstale":"2024-01-26T23:45:53.005249+0000","last_undegraded":"2024-01-26T23:45:53.005249+0000","last_fullsized":"2024-01-26T23:45:53.005249+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:47.231142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"97'4","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.018485+0000","last_change":"2024-01-26T23:45:51.994990+0000","last_active":"2024-01-26T23:45:53.018485+0000","last_peered":"2024-01-26T23:45:53.018485+0000","last_clean":"2024-01-26T23:45:53.018485+0000","last_became_active":"2024-01-26T23:45:51.994697+0000","last_became_peered":"2024-01-26T23:45:51.994697+0000","last_unstale":"2024-01-26T23:45:53.018485+0000","last_undegraded":"2024-01-26T23:45:53.018485+0000","last_fullsized":"2024-01-26T23:45:53.018485+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:25:30.884726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.009449+0000","last_change":"2024-01-26T23:45:52.995466+0000","last_active":"2024-01-26T23:45:53.009449+0000","last_peered":"2024-01-26T23:45:53.009449+0000","last_clean":"2024-01-26T23:45:53.009449+0000","last_became_active":"2024-01-26T23:45:52.995141+0000","last_became_peered":"2024-01-26T23:45:52.995141+0000","last_unstale":"2024-01-26T23:45:53.009449+0000","last_undegraded":"2024-01-26T23:45:53.009449+0000","last_fullsized":"2024-01-26T23:45:53.009449+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:08.999844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"97'1","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.015519+0000","last_change":"2024-01-26T23:45:51.996172+0000","last_active":"2024-01-26T23:45:53.015519+0000","last_peered":"2024-01-26T23:45:53.015519+0000","last_clean":"2024-01-26T23:45:53.015519+0000","last_became_active":"2024-01-26T23:45:51.995912+0000","last_became_peered":"2024-01-26T23:45:51.995912+0000","last_unstale":"2024-01-26T23:45:53.015519+0000","last_undegraded":"2024-01-26T23:45:53.015519+0000","last_fullsized":"2024-01-26T23:45:53.015519+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:59.450936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.008191+0000","last_change":"2024-01-26T23:45:53.004589+0000","last_active":"2024-01-26T23:45:53.008191+0000","last_peered":"2024-01-26T23:45:53.008191+0000","last_clean":"2024-01-26T23:45:53.008191+0000","last_became_active":"2024-01-26T23:45:53.004327+0000","last_became_peered":"2024-01-26T23:45:53.004327+0000","last_unstale":"2024-01-26T23:45:53.008191+0000","last_undegraded":"2024-01-26T23:45:53.008191+0000","last_fullsized":"2024-01-26T23:45:53.008191+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:27:37.169156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.020151+0000","last_change":"2024-01-26T23:45:51.997038+0000","last_active":"2024-01-26T23:45:53.020151+0000","last_peered":"2024-01-26T23:45:53.020151+0000","last_clean":"2024-01-26T23:45:53.020151+0000","last_became_active":"2024-01-26T23:45:51.996797+0000","last_became_peered":"2024-01-26T23:45:51.996797+0000","last_unstale":"2024-01-26T23:45:53.020151+0000","last_undegraded":"2024-01-26T23:45:53.020151+0000","last_fullsized":"2024-01-26T23:45:53.020151+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:24.317867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010756+0000","last_change":"2024-01-26T23:45:53.005919+0000","last_active":"2024-01-26T23:45:53.010756+0000","last_peered":"2024-01-26T23:45:53.010756+0000","last_clean":"2024-01-26T23:45:53.010756+0000","last_became_active":"2024-01-26T23:45:53.005721+0000","last_became_peered":"2024-01-26T23:45:53.005721+0000","last_unstale":"2024-01-26T23:45:53.010756+0000","last_undegraded":"2024-01-26T23:45:53.010756+0000","last_fullsized":"2024-01-26T23:45:53.010756+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:21:21.636031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.015980+0000","last_change":"2024-01-26T23:45:51.994444+0000","last_active":"2024-01-26T23:45:53.015980+0000","last_peered":"2024-01-26T23:45:53.015980+0000","last_clean":"2024-01-26T23:45:53.015980+0000","last_became_active":"2024-01-26T23:45:51.994036+0000","last_became_peered":"2024-01-26T23:45:51.994036+0000","last_unstale":"2024-01-26T23:45:53.015980+0000","last_undegraded":"2024-01-26T23:45:53.015980+0000","last_fullsized":"2024-01-26T23:45:53.015980+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:42:07.246862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010005+0000","last_change":"2024-01-26T23:45:53.001042+0000","last_active":"2024-01-26T23:45:53.010005+0000","last_peered":"2024-01-26T23:45:53.010005+0000","last_clean":"2024-01-26T23:45:53.010005+0000","last_became_active":"2024-01-26T23:45:53.000759+0000","last_became_peered":"2024-01-26T23:45:53.000759+0000","last_unstale":"2024-01-26T23:45:53.010005+0000","last_undegraded":"2024-01-26T23:45:53.010005+0000","last_fullsized":"2024-01-26T23:45:53.010005+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:48:01.627096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"97'1","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.015830+0000","last_change":"2024-01-26T23:45:51.993730+0000","last_active":"2024-01-26T23:45:53.015830+0000","last_peered":"2024-01-26T23:45:53.015830+0000","last_clean":"2024-01-26T23:45:53.015830+0000","last_became_active":"2024-01-26T23:45:51.993505+0000","last_became_peered":"2024-01-26T23:45:51.993505+0000","last_unstale":"2024-01-26T23:45:53.015830+0000","last_undegraded":"2024-01-26T23:45:53.015830+0000","last_fullsized":"2024-01-26T23:45:53.015830+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:05:29.034239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.013976+0000","last_change":"2024-01-26T23:45:53.011018+0000","last_active":"2024-01-26T23:45:53.013976+0000","last_peered":"2024-01-26T23:45:53.013976+0000","last_clean":"2024-01-26T23:45:53.013976+0000","last_became_active":"2024-01-26T23:45:53.010726+0000","last_became_peered":"2024-01-26T23:45:53.010726+0000","last_unstale":"2024-01-26T23:45:53.013976+0000","last_undegraded":"2024-01-26T23:45:53.013976+0000","last_fullsized":"2024-01-26T23:45:53.013976+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:50:13.686822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"97'1","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.015467+0000","last_change":"2024-01-26T23:45:51.998440+0000","last_active":"2024-01-26T23:45:53.015467+0000","last_peered":"2024-01-26T23:45:53.015467+0000","last_clean":"2024-01-26T23:45:53.015467+0000","last_became_active":"2024-01-26T23:45:51.998181+0000","last_became_peered":"2024-01-26T23:45:51.998181+0000","last_unstale":"2024-01-26T23:45:53.015467+0000","last_undegraded":"2024-01-26T23:45:53.015467+0000","last_fullsized":"2024-01-26T23:45:53.015467+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:46.226628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.008169+0000","last_change":"2024-01-26T23:45:53.004534+0000","last_active":"2024-01-26T23:45:53.008169+0000","last_peered":"2024-01-26T23:45:53.008169+0000","last_clean":"2024-01-26T23:45:53.008169+0000","last_became_active":"2024-01-26T23:45:53.004331+0000","last_became_peered":"2024-01-26T23:45:53.004331+0000","last_unstale":"2024-01-26T23:45:53.008169+0000","last_undegraded":"2024-01-26T23:45:53.008169+0000","last_fullsized":"2024-01-26T23:45:53.008169+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:32:39.694027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.983528+0000","last_change":"2024-01-26T23:45:51.995344+0000","last_active":"2024-01-26T23:45:52.983528+0000","last_peered":"2024-01-26T23:45:52.983528+0000","last_clean":"2024-01-26T23:45:52.983528+0000","last_became_active":"2024-01-26T23:45:51.995068+0000","last_became_peered":"2024-01-26T23:45:51.995068+0000","last_unstale":"2024-01-26T23:45:52.983528+0000","last_undegraded":"2024-01-26T23:45:52.983528+0000","last_fullsized":"2024-01-26T23:45:52.983528+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:45:21.389917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001964+0000","last_change":"2024-01-26T23:45:53.000934+0000","last_active":"2024-01-26T23:45:53.001964+0000","last_peered":"2024-01-26T23:45:53.001964+0000","last_clean":"2024-01-26T23:45:53.001964+0000","last_became_active":"2024-01-26T23:45:53.000698+0000","last_became_peered":"2024-01-26T23:45:53.000698+0000","last_unstale":"2024-01-26T23:45:53.001964+0000","last_undegraded":"2024-01-26T23:45:53.001964+0000","last_fullsized":"2024-01-26T23:45:53.001964+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:53.299917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.987463+0000","last_change":"2024-01-26T23:45:52.002908+0000","last_active":"2024-01-26T23:45:52.987463+0000","last_peered":"2024-01-26T23:45:52.987463+0000","last_clean":"2024-01-26T23:45:52.987463+0000","last_became_active":"2024-01-26T23:45:52.002606+0000","last_became_peered":"2024-01-26T23:45:52.002606+0000","last_unstale":"2024-01-26T23:45:52.987463+0000","last_undegraded":"2024-01-26T23:45:52.987463+0000","last_fullsized":"2024-01-26T23:45:52.987463+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:34.171032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010594+0000","last_change":"2024-01-26T23:45:52.999941+0000","last_active":"2024-01-26T23:45:53.010594+0000","last_peered":"2024-01-26T23:45:53.010594+0000","last_clean":"2024-01-26T23:45:53.010594+0000","last_became_active":"2024-01-26T23:45:52.999657+0000","last_became_peered":"2024-01-26T23:45:52.999657+0000","last_unstale":"2024-01-26T23:45:53.010594+0000","last_undegraded":"2024-01-26T23:45:53.010594+0000","last_fullsized":"2024-01-26T23:45:53.010594+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:21:56.682642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.996692+0000","last_change":"2024-01-26T23:45:51.996342+0000","last_active":"2024-01-26T23:45:52.996692+0000","last_peered":"2024-01-26T23:45:52.996692+0000","last_clean":"2024-01-26T23:45:52.996692+0000","last_became_active":"2024-01-26T23:45:51.996106+0000","last_became_peered":"2024-01-26T23:45:51.996106+0000","last_unstale":"2024-01-26T23:45:52.996692+0000","last_undegraded":"2024-01-26T23:45:52.996692+0000","last_fullsized":"2024-01-26T23:45:52.996692+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:52:15.511373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.997518+0000","last_change":"2024-01-26T23:45:52.992386+0000","last_active":"2024-01-26T23:45:52.997518+0000","last_peered":"2024-01-26T23:45:52.997518+0000","last_clean":"2024-01-26T23:45:52.997518+0000","last_became_active":"2024-01-26T23:45:52.992145+0000","last_became_peered":"2024-01-26T23:45:52.992145+0000","last_unstale":"2024-01-26T23:45:52.997518+0000","last_undegraded":"2024-01-26T23:45:52.997518+0000","last_fullsized":"2024-01-26T23:45:52.997518+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:12:46.926098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.987056+0000","last_change":"2024-01-26T23:45:51.993792+0000","last_active":"2024-01-26T23:45:52.987056+0000","last_peered":"2024-01-26T23:45:52.987056+0000","last_clean":"2024-01-26T23:45:52.987056+0000","last_became_active":"2024-01-26T23:45:51.993519+0000","last_became_peered":"2024-01-26T23:45:51.993519+0000","last_unstale":"2024-01-26T23:45:52.987056+0000","last_undegraded":"2024-01-26T23:45:52.987056+0000","last_fullsized":"2024-01-26T23:45:52.987056+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:05:31.172516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.013407+0000","last_change":"2024-01-26T23:45:53.009145+0000","last_active":"2024-01-26T23:45:53.013407+0000","last_peered":"2024-01-26T23:45:53.013407+0000","last_clean":"2024-01-26T23:45:53.013407+0000","last_became_active":"2024-01-26T23:45:53.008642+0000","last_became_peered":"2024-01-26T23:45:53.008642+0000","last_unstale":"2024-01-26T23:45:53.013407+0000","last_undegraded":"2024-01-26T23:45:53.013407+0000","last_fullsized":"2024-01-26T23:45:53.013407+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:27:27.092910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"97'3","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.017982+0000","last_change":"2024-01-26T23:45:52.000759+0000","last_active":"2024-01-26T23:45:53.017982+0000","last_peered":"2024-01-26T23:45:53.017982+0000","last_clean":"2024-01-26T23:45:53.017982+0000","last_became_active":"2024-01-26T23:45:52.000442+0000","last_became_peered":"2024-01-26T23:45:52.000442+0000","last_unstale":"2024-01-26T23:45:53.017982+0000","last_undegraded":"2024-01-26T23:45:53.017982+0000","last_fullsized":"2024-01-26T23:45:53.017982+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:36:59.186830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.006905+0000","last_change":"2024-01-26T23:45:53.004186+0000","last_active":"2024-01-26T23:45:53.006905+0000","last_peered":"2024-01-26T23:45:53.006905+0000","last_clean":"2024-01-26T23:45:53.006905+0000","last_became_active":"2024-01-26T23:45:53.003905+0000","last_became_peered":"2024-01-26T23:45:53.003905+0000","last_unstale":"2024-01-26T23:45:53.006905+0000","last_undegraded":"2024-01-26T23:45:53.006905+0000","last_fullsized":"2024-01-26T23:45:53.006905+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:24:29.016321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"97'6","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.019760+0000","last_change":"2024-01-26T23:45:51.990179+0000","last_active":"2024-01-26T23:45:53.019760+0000","last_peered":"2024-01-26T23:45:53.019760+0000","last_clean":"2024-01-26T23:45:53.019760+0000","last_became_active":"2024-01-26T23:45:51.989929+0000","last_became_peered":"2024-01-26T23:45:51.989929+0000","last_unstale":"2024-01-26T23:45:53.019760+0000","last_undegraded":"2024-01-26T23:45:53.019760+0000","last_fullsized":"2024-01-26T23:45:53.019760+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:44.905097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.008126+0000","last_change":"2024-01-26T23:45:53.004976+0000","last_active":"2024-01-26T23:45:53.008126+0000","last_peered":"2024-01-26T23:45:53.008126+0000","last_clean":"2024-01-26T23:45:53.008126+0000","last_became_active":"2024-01-26T23:45:53.004775+0000","last_became_peered":"2024-01-26T23:45:53.004775+0000","last_unstale":"2024-01-26T23:45:53.008126+0000","last_undegraded":"2024-01-26T23:45:53.008126+0000","last_fullsized":"2024-01-26T23:45:53.008126+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:27:06.530418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.015768+0000","last_change":"2024-01-26T23:45:51.989693+0000","last_active":"2024-01-26T23:45:53.015768+0000","last_peered":"2024-01-26T23:45:53.015768+0000","last_clean":"2024-01-26T23:45:53.015768+0000","last_became_active":"2024-01-26T23:45:51.989400+0000","last_became_peered":"2024-01-26T23:45:51.989400+0000","last_unstale":"2024-01-26T23:45:53.015768+0000","last_undegraded":"2024-01-26T23:45:53.015768+0000","last_fullsized":"2024-01-26T23:45:53.015768+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:40:36.924823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.996681+0000","last_change":"2024-01-26T23:45:52.992894+0000","last_active":"2024-01-26T23:45:52.996681+0000","last_peered":"2024-01-26T23:45:52.996681+0000","last_clean":"2024-01-26T23:45:52.996681+0000","last_became_active":"2024-01-26T23:45:52.992666+0000","last_became_peered":"2024-01-26T23:45:52.992666+0000","last_unstale":"2024-01-26T23:45:52.996681+0000","last_undegraded":"2024-01-26T23:45:52.996681+0000","last_fullsized":"2024-01-26T23:45:52.996681+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:27:06.530418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.995858+0000","last_change":"2024-01-26T23:45:52.004858+0000","last_active":"2024-01-26T23:45:52.995858+0000","last_peered":"2024-01-26T23:45:52.995858+0000","last_clean":"2024-01-26T23:45:52.995858+0000","last_became_active":"2024-01-26T23:45:52.004609+0000","last_became_peered":"2024-01-26T23:45:52.004609+0000","last_unstale":"2024-01-26T23:45:52.995858+0000","last_undegraded":"2024-01-26T23:45:52.995858+0000","last_fullsized":"2024-01-26T23:45:52.995858+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:12:45.921584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.997360+0000","last_change":"2024-01-26T23:45:52.992493+0000","last_active":"2024-01-26T23:45:52.997360+0000","last_peered":"2024-01-26T23:45:52.997360+0000","last_clean":"2024-01-26T23:45:52.997360+0000","last_became_active":"2024-01-26T23:45:52.992149+0000","last_became_peered":"2024-01-26T23:45:52.992149+0000","last_unstale":"2024-01-26T23:45:52.997360+0000","last_undegraded":"2024-01-26T23:45:52.997360+0000","last_fullsized":"2024-01-26T23:45:52.997360+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:06.428462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.986442+0000","last_change":"2024-01-26T23:45:51.995486+0000","last_active":"2024-01-26T23:45:52.986442+0000","last_peered":"2024-01-26T23:45:52.986442+0000","last_clean":"2024-01-26T23:45:52.986442+0000","last_became_active":"2024-01-26T23:45:51.995169+0000","last_became_peered":"2024-01-26T23:45:51.995169+0000","last_unstale":"2024-01-26T23:45:52.986442+0000","last_undegraded":"2024-01-26T23:45:52.986442+0000","last_fullsized":"2024-01-26T23:45:52.986442+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:31:47.780797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010570+0000","last_change":"2024-01-26T23:45:53.006345+0000","last_active":"2024-01-26T23:45:53.010570+0000","last_peered":"2024-01-26T23:45:53.010570+0000","last_clean":"2024-01-26T23:45:53.010570+0000","last_became_active":"2024-01-26T23:45:53.006080+0000","last_became_peered":"2024-01-26T23:45:53.006080+0000","last_unstale":"2024-01-26T23:45:53.010570+0000","last_undegraded":"2024-01-26T23:45:53.010570+0000","last_fullsized":"2024-01-26T23:45:53.010570+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:32:39.694027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"97'8","reported_seq":18,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.020619+0000","last_change":"2024-01-26T23:45:51.994969+0000","last_active":"2024-01-26T23:45:53.020619+0000","last_peered":"2024-01-26T23:45:53.020619+0000","last_clean":"2024-01-26T23:45:53.020619+0000","last_became_active":"2024-01-26T23:45:51.994689+0000","last_became_peered":"2024-01-26T23:45:51.994689+0000","last_unstale":"2024-01-26T23:45:53.020619+0000","last_undegraded":"2024-01-26T23:45:53.020619+0000","last_fullsized":"2024-01-26T23:45:53.020619+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:01:04.622959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.007301+0000","last_change":"2024-01-26T23:45:53.006377+0000","last_active":"2024-01-26T23:45:53.007301+0000","last_peered":"2024-01-26T23:45:53.007301+0000","last_clean":"2024-01-26T23:45:53.007301+0000","last_became_active":"2024-01-26T23:45:53.006050+0000","last_became_peered":"2024-01-26T23:45:53.006050+0000","last_unstale":"2024-01-26T23:45:53.007301+0000","last_undegraded":"2024-01-26T23:45:53.007301+0000","last_fullsized":"2024-01-26T23:45:53.007301+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:33:12.010368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"97'1","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.017369+0000","last_change":"2024-01-26T23:45:52.006988+0000","last_active":"2024-01-26T23:45:53.017369+0000","last_peered":"2024-01-26T23:45:53.017369+0000","last_clean":"2024-01-26T23:45:53.017369+0000","last_became_active":"2024-01-26T23:45:52.006827+0000","last_became_peered":"2024-01-26T23:45:52.006827+0000","last_unstale":"2024-01-26T23:45:53.017369+0000","last_undegraded":"2024-01-26T23:45:53.017369+0000","last_fullsized":"2024-01-26T23:45:53.017369+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:05.423948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.006509+0000","last_change":"2024-01-26T23:45:53.001552+0000","last_active":"2024-01-26T23:45:53.006509+0000","last_peered":"2024-01-26T23:45:53.006509+0000","last_clean":"2024-01-26T23:45:53.006509+0000","last_became_active":"2024-01-26T23:45:53.001246+0000","last_became_peered":"2024-01-26T23:45:53.001246+0000","last_unstale":"2024-01-26T23:45:53.006509+0000","last_undegraded":"2024-01-26T23:45:53.006509+0000","last_fullsized":"2024-01-26T23:45:53.006509+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:57:36.021844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.015324+0000","last_change":"2024-01-26T23:45:51.999517+0000","last_active":"2024-01-26T23:45:53.015324+0000","last_peered":"2024-01-26T23:45:53.015324+0000","last_clean":"2024-01-26T23:45:53.015324+0000","last_became_active":"2024-01-26T23:45:51.999248+0000","last_became_peered":"2024-01-26T23:45:51.999248+0000","last_unstale":"2024-01-26T23:45:53.015324+0000","last_undegraded":"2024-01-26T23:45:53.015324+0000","last_fullsized":"2024-01-26T23:45:53.015324+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:40:36.924823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.008199+0000","last_change":"2024-01-26T23:45:53.003010+0000","last_active":"2024-01-26T23:45:53.008199+0000","last_peered":"2024-01-26T23:45:53.008199+0000","last_clean":"2024-01-26T23:45:53.008199+0000","last_became_active":"2024-01-26T23:45:53.002641+0000","last_became_peered":"2024-01-26T23:45:53.002641+0000","last_unstale":"2024-01-26T23:45:53.008199+0000","last_undegraded":"2024-01-26T23:45:53.008199+0000","last_fullsized":"2024-01-26T23:45:53.008199+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:06:16.550968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"97'1","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.018074+0000","last_change":"2024-01-26T23:45:52.001332+0000","last_active":"2024-01-26T23:45:53.018074+0000","last_peered":"2024-01-26T23:45:53.018074+0000","last_clean":"2024-01-26T23:45:53.018074+0000","last_became_active":"2024-01-26T23:45:51.996690+0000","last_became_peered":"2024-01-26T23:45:51.996690+0000","last_unstale":"2024-01-26T23:45:53.018074+0000","last_undegraded":"2024-01-26T23:45:53.018074+0000","last_fullsized":"2024-01-26T23:45:53.018074+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:54:36.084818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.007512+0000","last_change":"2024-01-26T23:45:53.006606+0000","last_active":"2024-01-26T23:45:53.007512+0000","last_peered":"2024-01-26T23:45:53.007512+0000","last_clean":"2024-01-26T23:45:53.007512+0000","last_became_active":"2024-01-26T23:45:53.005949+0000","last_became_peered":"2024-01-26T23:45:53.005949+0000","last_unstale":"2024-01-26T23:45:53.007512+0000","last_undegraded":"2024-01-26T23:45:53.007512+0000","last_fullsized":"2024-01-26T23:45:53.007512+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:57:56.663833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.987187+0000","last_change":"2024-01-26T23:45:52.007285+0000","last_active":"2024-01-26T23:45:52.987187+0000","last_peered":"2024-01-26T23:45:52.987187+0000","last_clean":"2024-01-26T23:45:52.987187+0000","last_became_active":"2024-01-26T23:45:52.006952+0000","last_became_peered":"2024-01-26T23:45:52.006952+0000","last_unstale":"2024-01-26T23:45:52.987187+0000","last_undegraded":"2024-01-26T23:45:52.987187+0000","last_fullsized":"2024-01-26T23:45:52.987187+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:50:39.847921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001264+0000","last_change":"2024-01-26T23:45:52.992221+0000","last_active":"2024-01-26T23:45:53.001264+0000","last_peered":"2024-01-26T23:45:53.001264+0000","last_clean":"2024-01-26T23:45:53.001264+0000","last_became_active":"2024-01-26T23:45:52.991944+0000","last_became_peered":"2024-01-26T23:45:52.991944+0000","last_unstale":"2024-01-26T23:45:53.001264+0000","last_undegraded":"2024-01-26T23:45:53.001264+0000","last_fullsized":"2024-01-26T23:45:53.001264+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:28:47.388074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.017086+0000","last_change":"2024-01-26T23:45:52.004539+0000","last_active":"2024-01-26T23:45:53.017086+0000","last_peered":"2024-01-26T23:45:53.017086+0000","last_clean":"2024-01-26T23:45:53.017086+0000","last_became_active":"2024-01-26T23:45:52.004213+0000","last_became_peered":"2024-01-26T23:45:52.004213+0000","last_unstale":"2024-01-26T23:45:53.017086+0000","last_undegraded":"2024-01-26T23:45:53.017086+0000","last_fullsized":"2024-01-26T23:45:53.017086+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:32:38.689514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001363+0000","last_change":"2024-01-26T23:45:52.994059+0000","last_active":"2024-01-26T23:45:53.001363+0000","last_peered":"2024-01-26T23:45:53.001363+0000","last_clean":"2024-01-26T23:45:53.001363+0000","last_became_active":"2024-01-26T23:45:52.993774+0000","last_became_peered":"2024-01-26T23:45:52.993774+0000","last_unstale":"2024-01-26T23:45:53.001363+0000","last_undegraded":"2024-01-26T23:45:53.001363+0000","last_fullsized":"2024-01-26T23:45:53.001363+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:37:33.720336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.998320+0000","last_change":"2024-01-26T23:45:51.999488+0000","last_active":"2024-01-26T23:45:52.998320+0000","last_peered":"2024-01-26T23:45:52.998320+0000","last_clean":"2024-01-26T23:45:52.998320+0000","last_became_active":"2024-01-26T23:45:51.999147+0000","last_became_peered":"2024-01-26T23:45:51.999147+0000","last_unstale":"2024-01-26T23:45:52.998320+0000","last_undegraded":"2024-01-26T23:45:52.998320+0000","last_fullsized":"2024-01-26T23:45:52.998320+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:21:20.631517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.996791+0000","last_change":"2024-01-26T23:45:52.992528+0000","last_active":"2024-01-26T23:45:52.996791+0000","last_peered":"2024-01-26T23:45:52.996791+0000","last_clean":"2024-01-26T23:45:52.996791+0000","last_became_active":"2024-01-26T23:45:52.992194+0000","last_became_peered":"2024-01-26T23:45:52.992194+0000","last_unstale":"2024-01-26T23:45:52.996791+0000","last_undegraded":"2024-01-26T23:45:52.996791+0000","last_fullsized":"2024-01-26T23:45:52.996791+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:00:00.455450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.997703+0000","last_change":"2024-01-26T23:45:51.998439+0000","last_active":"2024-01-26T23:45:52.997703+0000","last_peered":"2024-01-26T23:45:52.997703+0000","last_clean":"2024-01-26T23:45:52.997703+0000","last_became_active":"2024-01-26T23:45:51.998055+0000","last_became_peered":"2024-01-26T23:45:51.998055+0000","last_unstale":"2024-01-26T23:45:52.997703+0000","last_undegraded":"2024-01-26T23:45:52.997703+0000","last_fullsized":"2024-01-26T23:45:52.997703+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:24.317867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.997887+0000","last_change":"2024-01-26T23:45:52.995272+0000","last_active":"2024-01-26T23:45:52.997887+0000","last_peered":"2024-01-26T23:45:52.997887+0000","last_clean":"2024-01-26T23:45:52.997887+0000","last_became_active":"2024-01-26T23:45:52.995010+0000","last_became_peered":"2024-01-26T23:45:52.995010+0000","last_unstale":"2024-01-26T23:45:52.997887+0000","last_undegraded":"2024-01-26T23:45:52.997887+0000","last_fullsized":"2024-01-26T23:45:52.997887+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:25.322381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.016573+0000","last_change":"2024-01-26T23:45:51.999087+0000","last_active":"2024-01-26T23:45:53.016573+0000","last_peered":"2024-01-26T23:45:53.016573+0000","last_clean":"2024-01-26T23:45:53.016573+0000","last_became_active":"2024-01-26T23:45:51.998809+0000","last_became_peered":"2024-01-26T23:45:51.998809+0000","last_unstale":"2024-01-26T23:45:53.016573+0000","last_undegraded":"2024-01-26T23:45:53.016573+0000","last_fullsized":"2024-01-26T23:45:53.016573+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:05:29.034239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005444+0000","last_change":"2024-01-26T23:45:53.002482+0000","last_active":"2024-01-26T23:45:53.005444+0000","last_peered":"2024-01-26T23:45:53.005444+0000","last_clean":"2024-01-26T23:45:53.005444+0000","last_became_active":"2024-01-26T23:45:53.002268+0000","last_became_peered":"2024-01-26T23:45:53.002268+0000","last_unstale":"2024-01-26T23:45:53.005444+0000","last_undegraded":"2024-01-26T23:45:53.005444+0000","last_fullsized":"2024-01-26T23:45:53.005444+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:21:21.636031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.984063+0000","last_change":"2024-01-26T23:45:52.004361+0000","last_active":"2024-01-26T23:45:52.984063+0000","last_peered":"2024-01-26T23:45:52.984063+0000","last_clean":"2024-01-26T23:45:52.984063+0000","last_became_active":"2024-01-26T23:45:52.004048+0000","last_became_peered":"2024-01-26T23:45:52.004048+0000","last_unstale":"2024-01-26T23:45:52.984063+0000","last_undegraded":"2024-01-26T23:45:52.984063+0000","last_fullsized":"2024-01-26T23:45:52.984063+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:01:43.090741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.008127+0000","last_change":"2024-01-26T23:45:53.003449+0000","last_active":"2024-01-26T23:45:53.008127+0000","last_peered":"2024-01-26T23:45:53.008127+0000","last_clean":"2024-01-26T23:45:53.008127+0000","last_became_active":"2024-01-26T23:45:53.003238+0000","last_became_peered":"2024-01-26T23:45:53.003238+0000","last_unstale":"2024-01-26T23:45:53.008127+0000","last_undegraded":"2024-01-26T23:45:53.008127+0000","last_fullsized":"2024-01-26T23:45:53.008127+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:27:06.530418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.997355+0000","last_change":"2024-01-26T23:45:51.994205+0000","last_active":"2024-01-26T23:45:52.997355+0000","last_peered":"2024-01-26T23:45:52.997355+0000","last_clean":"2024-01-26T23:45:52.997355+0000","last_became_active":"2024-01-26T23:45:51.993947+0000","last_became_peered":"2024-01-26T23:45:51.993947+0000","last_unstale":"2024-01-26T23:45:52.997355+0000","last_undegraded":"2024-01-26T23:45:52.997355+0000","last_fullsized":"2024-01-26T23:45:52.997355+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:21:20.631517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.009979+0000","last_change":"2024-01-26T23:45:52.999402+0000","last_active":"2024-01-26T23:45:53.009979+0000","last_peered":"2024-01-26T23:45:53.009979+0000","last_clean":"2024-01-26T23:45:53.009979+0000","last_became_active":"2024-01-26T23:45:52.999111+0000","last_became_peered":"2024-01-26T23:45:52.999111+0000","last_unstale":"2024-01-26T23:45:53.009979+0000","last_undegraded":"2024-01-26T23:45:53.009979+0000","last_fullsized":"2024-01-26T23:45:53.009979+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:15:38.716213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.996030+0000","last_change":"2024-01-26T23:45:51.994463+0000","last_active":"2024-01-26T23:45:52.996030+0000","last_peered":"2024-01-26T23:45:52.996030+0000","last_clean":"2024-01-26T23:45:52.996030+0000","last_became_active":"2024-01-26T23:45:51.994218+0000","last_became_peered":"2024-01-26T23:45:51.994218+0000","last_unstale":"2024-01-26T23:45:52.996030+0000","last_undegraded":"2024-01-26T23:45:52.996030+0000","last_fullsized":"2024-01-26T23:45:52.996030+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:22.235033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010568+0000","last_change":"2024-01-26T23:45:53.006350+0000","last_active":"2024-01-26T23:45:53.010568+0000","last_peered":"2024-01-26T23:45:53.010568+0000","last_clean":"2024-01-26T23:45:53.010568+0000","last_became_active":"2024-01-26T23:45:53.006074+0000","last_became_peered":"2024-01-26T23:45:53.006074+0000","last_unstale":"2024-01-26T23:45:53.010568+0000","last_undegraded":"2024-01-26T23:45:53.010568+0000","last_fullsized":"2024-01-26T23:45:53.010568+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:12:46.926098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.990311+0000","last_change":"2024-01-26T23:45:51.994063+0000","last_active":"2024-01-26T23:45:52.990311+0000","last_peered":"2024-01-26T23:45:52.990311+0000","last_clean":"2024-01-26T23:45:52.990311+0000","last_became_active":"2024-01-26T23:45:51.993780+0000","last_became_peered":"2024-01-26T23:45:51.993780+0000","last_unstale":"2024-01-26T23:45:52.990311+0000","last_undegraded":"2024-01-26T23:45:52.990311+0000","last_fullsized":"2024-01-26T23:45:52.990311+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:21:06.318896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010925+0000","last_change":"2024-01-26T23:45:53.009839+0000","last_active":"2024-01-26T23:45:53.010925+0000","last_peered":"2024-01-26T23:45:53.010925+0000","last_clean":"2024-01-26T23:45:53.010925+0000","last_became_active":"2024-01-26T23:45:53.009673+0000","last_became_peered":"2024-01-26T23:45:53.009673+0000","last_unstale":"2024-01-26T23:45:53.010925+0000","last_undegraded":"2024-01-26T23:45:53.010925+0000","last_fullsized":"2024-01-26T23:45:53.010925+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:23.239547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.987040+0000","last_change":"2024-01-26T23:45:52.001285+0000","last_active":"2024-01-26T23:45:52.987040+0000","last_peered":"2024-01-26T23:45:52.987040+0000","last_clean":"2024-01-26T23:45:52.987040+0000","last_became_active":"2024-01-26T23:45:52.000978+0000","last_became_peered":"2024-01-26T23:45:52.000978+0000","last_unstale":"2024-01-26T23:45:52.987040+0000","last_undegraded":"2024-01-26T23:45:52.987040+0000","last_fullsized":"2024-01-26T23:45:52.987040+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:53:01.510934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005600+0000","last_change":"2024-01-26T23:45:53.002599+0000","last_active":"2024-01-26T23:45:53.005600+0000","last_peered":"2024-01-26T23:45:53.005600+0000","last_clean":"2024-01-26T23:45:53.005600+0000","last_became_active":"2024-01-26T23:45:53.000819+0000","last_became_peered":"2024-01-26T23:45:53.000819+0000","last_unstale":"2024-01-26T23:45:53.005600+0000","last_undegraded":"2024-01-26T23:45:53.005600+0000","last_fullsized":"2024-01-26T23:45:53.005600+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:06.428462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.992489+0000","last_change":"2024-01-26T23:45:51.996049+0000","last_active":"2024-01-26T23:45:52.992489+0000","last_peered":"2024-01-26T23:45:52.992489+0000","last_clean":"2024-01-26T23:45:52.992489+0000","last_became_active":"2024-01-26T23:45:51.995820+0000","last_became_peered":"2024-01-26T23:45:51.995820+0000","last_unstale":"2024-01-26T23:45:52.992489+0000","last_undegraded":"2024-01-26T23:45:52.992489+0000","last_fullsized":"2024-01-26T23:45:52.992489+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:39:54.556222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005498+0000","last_change":"2024-01-26T23:45:53.002124+0000","last_active":"2024-01-26T23:45:53.005498+0000","last_peered":"2024-01-26T23:45:53.005498+0000","last_clean":"2024-01-26T23:45:53.005498+0000","last_became_active":"2024-01-26T23:45:53.000783+0000","last_became_peered":"2024-01-26T23:45:53.000783+0000","last_unstale":"2024-01-26T23:45:53.005498+0000","last_undegraded":"2024-01-26T23:45:53.005498+0000","last_fullsized":"2024-01-26T23:45:53.005498+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:00:00.455450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.990097+0000","last_change":"2024-01-26T23:45:52.007142+0000","last_active":"2024-01-26T23:45:52.990097+0000","last_peered":"2024-01-26T23:45:52.990097+0000","last_clean":"2024-01-26T23:45:52.990097+0000","last_became_active":"2024-01-26T23:45:52.006791+0000","last_became_peered":"2024-01-26T23:45:52.006791+0000","last_unstale":"2024-01-26T23:45:52.990097+0000","last_undegraded":"2024-01-26T23:45:52.990097+0000","last_fullsized":"2024-01-26T23:45:52.990097+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:44.905097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010714+0000","last_change":"2024-01-26T23:45:53.006271+0000","last_active":"2024-01-26T23:45:53.010714+0000","last_peered":"2024-01-26T23:45:53.010714+0000","last_clean":"2024-01-26T23:45:53.010714+0000","last_became_active":"2024-01-26T23:45:53.006012+0000","last_became_peered":"2024-01-26T23:45:53.006012+0000","last_unstale":"2024-01-26T23:45:53.010714+0000","last_undegraded":"2024-01-26T23:45:53.010714+0000","last_fullsized":"2024-01-26T23:45:53.010714+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:33:12.010368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.998723+0000","last_change":"2024-01-26T23:45:51.999496+0000","last_active":"2024-01-26T23:45:52.998723+0000","last_peered":"2024-01-26T23:45:52.998723+0000","last_clean":"2024-01-26T23:45:52.998723+0000","last_became_active":"2024-01-26T23:45:51.998966+0000","last_became_peered":"2024-01-26T23:45:51.998966+0000","last_unstale":"2024-01-26T23:45:52.998723+0000","last_undegraded":"2024-01-26T23:45:52.998723+0000","last_fullsized":"2024-01-26T23:45:52.998723+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:12:45.921584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"98'2197","reported_seq":2467,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-26T23:45:56.855491+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:45:56.855491+0000","last_peered":"2024-01-26T23:45:56.855491+0000","last_clean":"2024-01-26T23:45:56.855491+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:45:56.855491+0000","last_undegraded":"2024-01-26T23:45:56.855491+0000","last_fullsized":"2024-01-26T23:45:56.855491+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":2197,"log_dups_size":0,"ondisk_log_size":2197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":592,"num_read_kb":1131,"num_write":253,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":592,"num_read_kb":1131,"num_write":253,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2197,"ondisk_log_size":2197,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575044,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299044,"kb_used_data":1756,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072796,"statfs":{"total":96636764160,"available":96330543104,"internally_reserved":0,"allocated":1798144,"data_stored":1197750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57599999999999996,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.79500000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.35699999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80400000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.37}]},{"osd":1,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57699999999999996,"15min":0.57799999999999996},"min":{"1min":0.42499999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77800000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.443},{"interface":"front","average":{"1min":0.62,"5min":0.61099999999999999,"15min":0.59899999999999998},"min":{"1min":0.46000000000000002,"5min":0.39200000000000002,"15min":0.36099999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.47199999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.38800000000000001,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.81699999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.48699999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77900000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.51600000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59999999999999998,"15min":0.59199999999999997},"min":{"1min":0.442,"5min":0.40500000000000003,"15min":0.35399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60599999999999998,"15min":0.60499999999999998},"min":{"1min":0.40100000000000002,"5min":0.39300000000000002,"15min":0.373},"max":{"1min":0.82699999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.60199999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46899999999999997,"15min":0.46100000000000002},"min":{"1min":0.36399999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.66100000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.501,"5min":0.495,"15min":0.48799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.314},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.53700000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.504,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.91700000000000004},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.40000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.74199999999999999,"5min":0.90200000000000002,"15min":0.96599999999999997},"last":0.623}]},{"osd":6,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.57999999999999996,"15min":0.55700000000000005},"min":{"1min":0.46400000000000002,"5min":0.375,"15min":0.33100000000000002},"max":{"1min":0.77900000000000003,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56200000000000006,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76500000000000001,"5min":0.85199999999999998,"15min":0.86899999999999999},"last":0.61299999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575044,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302568,"kb_used_data":1184,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069272,"statfs":{"total":96636764160,"available":96326934528,"internally_reserved":0,"allocated":1212416,"data_stored":608896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56899999999999995,"15min":0.57399999999999995},"min":{"1min":0.35999999999999999,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":0.752,"5min":0.85199999999999998,"15min":0.90200000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59099999999999997,"15min":0.59399999999999997},"min":{"1min":0.42899999999999999,"5min":0.38500000000000001,"15min":0.33600000000000002},"max":{"1min":0.79000000000000004,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.33300000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.312,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.77100000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.34399999999999997},"max":{"1min":0.75700000000000001,"5min":0.78900000000000003,"15min":0.874},"last":0.36699999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.40799999999999997,"5min":0.35499999999999998,"15min":0.29599999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.85799999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38600000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77000000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.46600000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.59499999999999997,"15min":0.59999999999999998},"min":{"1min":0.45500000000000002,"5min":0.38700000000000001,"15min":0.29999999999999999},"max":{"1min":0.79000000000000004,"5min":0.80700000000000005,"15min":0.92200000000000004},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.622,"15min":0.624},"min":{"1min":0.55000000000000004,"5min":0.39500000000000002,"15min":0.39000000000000001},"max":{"1min":0.78400000000000003,"5min":0.81100000000000005,"15min":0.84499999999999997},"last":0.44500000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.47799999999999998,"15min":0.47399999999999998},"min":{"1min":0.38100000000000001,"5min":0.29899999999999999,"15min":0.22},"max":{"1min":0.69899999999999995,"5min":0.84599999999999997,"15min":0.90200000000000002},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.501,"15min":0.499},"min":{"1min":0.44700000000000001,"5min":0.32000000000000001,"15min":0.28899999999999998},"max":{"1min":0.61899999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.377}]},{"osd":5,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48499999999999999,"15min":0.48899999999999999},"min":{"1min":0.42499999999999999,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.53400000000000003,"15min":0.53300000000000003},"min":{"1min":0.46100000000000002,"5min":0.33700000000000002,"15min":0.26200000000000001},"max":{"1min":0.75700000000000001,"5min":0.872,"15min":0.96499999999999997},"last":0.39700000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53500000000000003,"15min":0.53800000000000003},"min":{"1min":0.44400000000000001,"5min":0.32500000000000001,"15min":0.246},"max":{"1min":0.81799999999999995,"5min":0.82799999999999996,"15min":0.97199999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.308},"max":{"1min":0.73099999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.41799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575044,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302564,"kb_used_data":1180,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069276,"statfs":{"total":96636764160,"available":96326938624,"internally_reserved":0,"allocated":1208320,"data_stored":607472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29899999999999999,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":0.59699999999999998,"5min":0.621,"15min":0.72599999999999998},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.35799999999999998,"5min":0.28899999999999998,"15min":0.26000000000000001},"max":{"1min":0.73499999999999999,"5min":1.337,"15min":1.337},"last":0.29299999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41499999999999998},"min":{"1min":0.27700000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55900000000000005,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48599999999999999,"15min":0.47699999999999998},"min":{"1min":0.376,"5min":0.34699999999999998,"15min":0.28999999999999998},"max":{"1min":0.70799999999999996,"5min":0.77300000000000002,"15min":0.79600000000000004},"last":0.42999999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52200000000000002,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":0.75800000000000001,"5min":1.782,"15min":1.782},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.497,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.625,"5min":1.379,"15min":1.379},"last":0.54300000000000004}]},{"osd":4,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65300000000000002,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":0.86499999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66200000000000003,"15min":0.66600000000000004},"min":{"1min":0.49399999999999999,"5min":0.45900000000000002,"15min":0.433},"max":{"1min":0.89900000000000002,"5min":0.91900000000000004,"15min":0.95699999999999996},"last":0.56699999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.50600000000000001,"5min":0.46800000000000003,"15min":0.42799999999999999},"max":{"1min":0.879,"5min":1.208,"15min":1.208},"last":0.62},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68500000000000005,"15min":0.67100000000000004},"min":{"1min":0.53700000000000003,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":0.81599999999999995,"5min":0.96599999999999997,"15min":1.018},"last":0.48699999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.69399999999999995,"15min":0.68500000000000005},"min":{"1min":0.51800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85199999999999998,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.52900000000000003,"5min":0.437,"15min":0.42599999999999999},"max":{"1min":0.83699999999999997,"5min":1.022,"15min":1.022},"last":0.57799999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71599999999999997},"min":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.49399999999999999},"max":{"1min":0.93200000000000005,"5min":0.98199999999999998,"15min":1.0069999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.749,"15min":0.73199999999999998},"min":{"1min":0.53200000000000003,"5min":0.52200000000000002,"15min":0.438},"max":{"1min":0.90600000000000003,"5min":1.296,"15min":1.3049999999999999},"last":0.59799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575043,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311328,"kb_used_data":1752,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060512,"statfs":{"total":96636764160,"available":96317964288,"internally_reserved":0,"allocated":1794048,"data_stored":1197728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.28499999999999998,"5min":0.23100000000000001,"15min":0.214},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":1.284},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.46000000000000002,"15min":0.45600000000000002},"min":{"1min":0.29099999999999998,"5min":0.28899999999999998,"15min":0.27300000000000002},"max":{"1min":0.60899999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.308}]},{"osd":2,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42099999999999999,"15min":0.42999999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.26700000000000002},"max":{"1min":0.57299999999999995,"5min":0.89600000000000002,"15min":1.2230000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.38300000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":1.2430000000000001},"last":0.42099999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46899999999999997,"15min":0.46700000000000003},"min":{"1min":0.36799999999999999,"5min":0.30599999999999999,"15min":0.26800000000000002},"max":{"1min":0.68100000000000005,"5min":0.80700000000000005,"15min":1.0980000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.49399999999999999,"15min":0.49099999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.27000000000000002},"max":{"1min":0.64000000000000001,"5min":1.198,"15min":1.198},"last":0.436}]},{"osd":4,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.64300000000000002,"15min":0.65200000000000002},"min":{"1min":0.46899999999999997,"5min":0.42399999999999999,"15min":0.372},"max":{"1min":0.78700000000000003,"5min":0.93300000000000005,"15min":1.133},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64700000000000002,"15min":0.65900000000000003},"min":{"1min":0.438,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85299999999999998,"5min":0.94299999999999995,"15min":1.0169999999999999},"last":0.373}]},{"osd":5,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65100000000000002,"15min":0.66300000000000003},"min":{"1min":0.52600000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79800000000000004,"5min":0.95799999999999996,"15min":1.0880000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67400000000000004,"15min":0.68200000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.41699999999999998},"max":{"1min":0.93999999999999995,"5min":1.012,"15min":1.169},"last":0.47399999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65500000000000003,"15min":0.67700000000000005},"min":{"1min":0.505,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89800000000000002,"5min":0.91600000000000004,"15min":1.071},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.68400000000000005,"15min":0.69999999999999996},"min":{"1min":0.48299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.94599999999999995,"5min":1.028,"15min":1.1160000000000001},"last":0.55300000000000005}]},{"osd":7,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":1.004,"15min":0.82599999999999996},"min":{"1min":0.54500000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.91500000000000004,"5min":25.965,"15min":25.965},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.73899999999999999,"5min":1.109,"15min":0.86599999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.433},"max":{"1min":0.93799999999999994,"5min":33.085000000000001,"15min":33.085000000000001},"last":0.56299999999999994}]}]},{"osd":2,"up_from":13,"seq":55834575043,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306664,"kb_used_data":1184,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065176,"statfs":{"total":96636764160,"available":96322740224,"internally_reserved":0,"allocated":1212416,"data_stored":608074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.39400000000000002,"15min":0.41999999999999998},"min":{"1min":0.29299999999999998,"5min":0.25800000000000001,"15min":0.22600000000000001},"max":{"1min":0.502,"5min":0.63400000000000001,"15min":9.0540000000000003},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.24299999999999999},"max":{"1min":0.56899999999999995,"5min":0.68500000000000005,"15min":4.1970000000000001},"last":0.40899999999999997}]},{"osd":1,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42699999999999999,"15min":0.42499999999999999},"min":{"1min":0.33300000000000002,"5min":0.248,"15min":0.221},"max":{"1min":0.55300000000000005,"5min":0.70499999999999996,"15min":0.82199999999999995},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.498,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":0.629,"5min":0.65900000000000003,"15min":12.157},"last":0.65700000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47799999999999998,"15min":0.48599999999999999},"min":{"1min":0.38800000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.59299999999999997,"5min":0.65500000000000003,"15min":0.86499999999999999},"last":0.432},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.47599999999999998,"15min":0.499},"min":{"1min":0.34599999999999997,"5min":0.32200000000000001,"15min":0.30399999999999999},"max":{"1min":0.60199999999999998,"5min":0.71099999999999997,"15min":9.0879999999999992},"last":0.53000000000000003}]},{"osd":4,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65000000000000002,"15min":0.66600000000000004},"min":{"1min":0.499,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.86599999999999999,"5min":1.087,"15min":9.0150000000000006},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.65800000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.371},"max":{"1min":0.80200000000000005,"5min":0.98099999999999998,"15min":4.2480000000000002},"last":0.496}]},{"osd":5,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78600000000000003,"5min":1.0680000000000001,"15min":4.3200000000000003},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68899999999999995,"15min":0.68200000000000005},"min":{"1min":0.58899999999999997,"5min":0.39800000000000002,"15min":0.35199999999999998},"max":{"1min":0.82399999999999995,"5min":0.95199999999999996,"15min":1.085},"last":0.60599999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68899999999999995,"15min":0.68400000000000005},"min":{"1min":0.51500000000000001,"5min":0.439,"15min":0.40899999999999997},"max":{"1min":0.79400000000000004,"5min":0.995,"15min":1.1080000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69199999999999995,"15min":0.71299999999999997},"min":{"1min":0.46200000000000002,"5min":0.443,"15min":0.38800000000000001},"max":{"1min":0.83999999999999997,"5min":1.1299999999999999,"15min":8.9700000000000006},"last":0.56999999999999995}]},{"osd":7,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70799999999999996,"15min":0.71699999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.85599999999999998,"5min":1.0029999999999999,"15min":4.282},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.57499999999999996,"5min":0.48199999999999998,"15min":0.42099999999999999},"max":{"1min":0.87,"5min":1.052,"15min":1.139},"last":0.72599999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575042,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306656,"kb_used_data":1176,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065184,"statfs":{"total":96636764160,"available":96322748416,"internally_reserved":0,"allocated":1204224,"data_stored":607914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40300000000000002,"15min":0.39800000000000002},"min":{"1min":0.32300000000000001,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.56999999999999995,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44900000000000001,"15min":0.48099999999999998},"min":{"1min":0.315,"5min":0.255,"15min":0.255},"max":{"1min":0.59299999999999997,"5min":1.2290000000000001,"15min":11.968},"last":0.33700000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.496},"min":{"1min":0.33200000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.623,"5min":1.633,"15min":12.050000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.495,"5min":0.48899999999999999,"15min":0.48599999999999999},"min":{"1min":0.378,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.59999999999999998,"5min":1.0760000000000001,"15min":1.2150000000000001},"last":0.54900000000000004}]},{"osd":2,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.49299999999999999,"15min":0.51900000000000002},"min":{"1min":0.39800000000000002,"5min":0.29699999999999999,"15min":0.28999999999999998},"max":{"1min":0.70399999999999996,"5min":1.4059999999999999,"15min":12.234},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46899999999999997,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.59499999999999997,"5min":1.1870000000000001,"15min":11.925000000000001},"last":0.38300000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69599999999999995,"15min":0.70599999999999996},"min":{"1min":0.56999999999999995,"5min":0.46999999999999997,"15min":0.40200000000000002},"max":{"1min":0.84899999999999998,"5min":3.306,"15min":12.099},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.66400000000000003,"15min":0.66600000000000004},"min":{"1min":0.59599999999999997,"5min":0.40200000000000002,"15min":0.379},"max":{"1min":0.82999999999999996,"5min":0.95799999999999996,"15min":1.0649999999999999},"last":0.52100000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.66500000000000004,"15min":0.70699999999999996},"min":{"1min":0.56299999999999994,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.79800000000000004,"5min":1.593,"15min":12.199999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68999999999999995,"15min":0.67800000000000005},"min":{"1min":0.53400000000000003,"5min":0.45300000000000001,"15min":0.436},"max":{"1min":0.92300000000000004,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.44600000000000001}]},{"osd":6,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69499999999999995,"15min":0.72099999999999997},"min":{"1min":0.56100000000000005,"5min":0.442,"15min":0.442},"max":{"1min":0.88800000000000001,"5min":1.506,"15min":12.132999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70099999999999996,"15min":0.69799999999999995},"min":{"1min":0.54600000000000004,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.88200000000000001,"5min":1.427,"15min":1.427},"last":0.48799999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.72699999999999998,"15min":0.70299999999999996},"min":{"1min":0.52500000000000002,"5min":0.48499999999999999,"15min":0.42499999999999999},"max":{"1min":0.86199999999999999,"5min":3.238,"15min":3.238},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.78200000000000003},"min":{"1min":0.625,"5min":0.48899999999999999,"15min":0.48699999999999999},"max":{"1min":0.91300000000000003,"5min":3.351,"15min":12.167},"last":0.55600000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575044,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302576,"kb_used_data":1192,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069264,"statfs":{"total":96636764160,"available":96326926336,"internally_reserved":0,"allocated":1220608,"data_stored":609000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57599999999999996,"15min":0.58399999999999996},"min":{"1min":0.439,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78300000000000003,"5min":1.21,"15min":1.21},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57499999999999996,"15min":0.58399999999999996},"min":{"1min":0.44900000000000001,"5min":0.309,"15min":0.28100000000000003},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":1.161},"last":0.39600000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.44900000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":1.069},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.59399999999999997},"min":{"1min":0.47199999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":1.085},"last":0.373}]},{"osd":2,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56299999999999994,"15min":0.57499999999999996},"min":{"1min":0.48499999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76500000000000001,"5min":0.81100000000000005,"15min":1.052},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60699999999999998,"15min":0.61199999999999999},"min":{"1min":0.41399999999999998,"5min":0.39900000000000002,"15min":0.373},"max":{"1min":0.81599999999999995,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.48599999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58199999999999996,"15min":0.59399999999999997},"min":{"1min":0.41299999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.78300000000000003,"5min":0.83499999999999996,"15min":1.1299999999999999},"last":0.502},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79100000000000004,"5min":1.05,"15min":1.1419999999999999},"last":0.441}]},{"osd":5,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.317,"5min":0.25700000000000001,"15min":0.247},"max":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.40799999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48499999999999999,"15min":0.47799999999999998},"min":{"1min":0.35799999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.73899999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53300000000000003,"15min":0.53000000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.311},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.95699999999999996},"last":0.42999999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.35799999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.57699999999999996},"min":{"1min":0.42499999999999999,"5min":0.41799999999999998,"15min":0.32200000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.879},"last":0.42199999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575044,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298464,"kb_used_data":1176,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073376,"statfs":{"total":96636764160,"available":96331137024,"internally_reserved":0,"allocated":1204224,"data_stored":607394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.58899999999999997,"15min":0.57599999999999996},"min":{"1min":0.46300000000000002,"5min":0.29999999999999999,"15min":0.26000000000000001},"max":{"1min":0.78900000000000003,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57199999999999995,"15min":0.58399999999999996},"min":{"1min":0.45400000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77400000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.5}]},{"osd":1,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.625,"15min":0.623},"min":{"1min":0.45500000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79500000000000004,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.64300000000000002,"15min":0.63300000000000001},"min":{"1min":0.499,"5min":0.435,"15min":0.379},"max":{"1min":0.80500000000000005,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.54100000000000004}]},{"osd":2,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.46700000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77000000000000002,"5min":0.83699999999999997,"15min":0.83799999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60399999999999998,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81100000000000005,"5min":2.0600000000000001,"15min":6.4589999999999996},"last":0.41499999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.50700000000000001,"5min":0.35999999999999999,"15min":0.314},"max":{"1min":0.755,"5min":0.86199999999999999,"15min":0.88400000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.60599999999999998,"15min":0.61499999999999999},"min":{"1min":0.47499999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77900000000000003,"5min":2.5630000000000002,"15min":6.0629999999999997},"last":0.48199999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.47399999999999998,"15min":0.46999999999999997},"min":{"1min":0.35599999999999998,"5min":0.28100000000000003,"15min":0.26300000000000001},"max":{"1min":0.56599999999999995,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.503,"15min":0.495},"min":{"1min":0.36899999999999999,"5min":0.33000000000000002,"15min":0.28100000000000003},"max":{"1min":0.69999999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.42799999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.31,"15min":0.27000000000000002},"max":{"1min":0.66600000000000004,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53200000000000003,"15min":0.52700000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.29299999999999998},"max":{"1min":0.71899999999999997,"5min":1.99,"15min":1.99},"last":0.49099999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.54900000000000004},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.76200000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33300000000000002},"max":{"1min":0.81499999999999995,"5min":0.82199999999999995,"15min":0.83099999999999996},"last":0.50700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:45:59.571 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:45:59.570+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:46:00.323 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:00.322+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:46:01.082 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:01.081+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:46:01.832 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:01.831+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:46:01.932 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:46:02.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.074+0000 7f99fbb0f700 1 Processor -- start 2024-01-26T23:46:02.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.074+0000 7f99fbb0f700 1 -- start start 2024-01-26T23:46:02.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.075+0000 7f99fbb0f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99f415e010 0x7f99f4067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.075+0000 7f99fbb0f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f99f415f490 0x7f99f4067e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.075+0000 7f99fbb0f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f406fb60 con 0x7f99f415dc40 2024-01-26T23:46:02.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.075+0000 7f99fbb0f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f406ef50 con 0x7f99f415f490 2024-01-26T23:46:02.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.075+0000 7f99fbb0f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f406f230 con 0x7f99f415e010 2024-01-26T23:46:02.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.075+0000 7f99fa0ac700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f99f415f490 0x7f99f4067e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.075+0000 7f99fa0ac700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f99f415f490 0x7f99f4067e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58748/0 (socket says 172.21.15.46:58748) 2024-01-26T23:46:02.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.075+0000 7f99f98ab700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f99f415dc40 0x7f99f40b2b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41784/0 (socket says 172.21.15.46:41784) 2024-01-26T23:46:02.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.075+0000 7f99fa0ac700 1 -- 172.21.15.46:0/439770357 learned_addr learned my addr 172.21.15.46:0/439770357 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:02.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.075+0000 7f99f90aa700 1 --2- 172.21.15.46:0/439770357 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99f415e010 0x7f99f4067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.075+0000 7f99fa0ac700 1 -- 172.21.15.46:0/439770357 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99f415e010 msgr2=0x7f99f4067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.075+0000 7f99fa0ac700 1 --2- 172.21.15.46:0/439770357 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99f415e010 0x7f99f4067930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.075+0000 7f99fa0ac700 1 -- 172.21.15.46:0/439770357 >> v1:172.21.15.46:6789/0 conn(0x7f99f415dc40 legacy=0x7f99f40b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.075+0000 7f99fa0ac700 1 -- 172.21.15.46:0/439770357 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f406f510 con 0x7f99f415f490 2024-01-26T23:46:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.075+0000 7f99f88a9700 1 -- 172.21.15.46:0/439770357 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136527878 0 0) 0x7f99f406fb60 con 0x7f99f415dc40 2024-01-26T23:46:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.076+0000 7f99fa0ac700 1 --2- 172.21.15.46:0/439770357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99f415f490 0x7f99f4067e70 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f99ec009f20 tx=0x7f99ec009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ccaf349cd2ae93b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.076+0000 7f99f88a9700 1 -- 172.21.15.46:0/439770357 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec00e070 con 0x7f99f415f490 2024-01-26T23:46:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.076+0000 7f99f88a9700 1 -- 172.21.15.46:0/439770357 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ec00b9d0 con 0x7f99f415f490 2024-01-26T23:46:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.076+0000 7f99f88a9700 1 -- 172.21.15.46:0/439770357 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec0092e0 con 0x7f99f415f490 2024-01-26T23:46:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.076+0000 7f99fbb0f700 1 -- 172.21.15.46:0/439770357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99f415f490 msgr2=0x7f99f4067e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.076+0000 7f99fbb0f700 1 --2- 172.21.15.46:0/439770357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99f415f490 0x7f99f4067e70 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f99ec009f20 tx=0x7f99ec009840 comp rx=0 tx=0).stop 2024-01-26T23:46:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.076+0000 7f99fbb0f700 1 -- 172.21.15.46:0/439770357 shutdown_connections 2024-01-26T23:46:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.076+0000 7f99fbb0f700 1 --2- 172.21.15.46:0/439770357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99f415f490 0x7f99f4067e70 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.076+0000 7f99fbb0f700 1 --2- 172.21.15.46:0/439770357 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99f415e010 0x7f99f4067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.076+0000 7f99fbb0f700 1 -- 172.21.15.46:0/439770357 >> 172.21.15.46:0/439770357 conn(0x7f99f4069510 msgr2=0x7f99f40aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:02.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.077+0000 7f99fbb0f700 1 -- 172.21.15.46:0/439770357 shutdown_connections 2024-01-26T23:46:02.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.077+0000 7f99fbb0f700 1 -- 172.21.15.46:0/439770357 wait complete. 2024-01-26T23:46:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.078+0000 7f99fbb0f700 1 Processor -- start 2024-01-26T23:46:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.078+0000 7f99fbb0f700 1 -- start start 2024-01-26T23:46:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.079+0000 7f99fbb0f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99f415dc40 0x7f99f4163940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.079+0000 7f99fbb0f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99f415e010 0x7f99f4162260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.079+0000 7f99fbb0f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99f415f490 0x7f99f4160b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.079+0000 7f99fbb0f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f40a5e90 con 0x7f99f415f490 2024-01-26T23:46:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.079+0000 7f99fbb0f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f4162a30 con 0x7f99f415e010 2024-01-26T23:46:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.079+0000 7f99fbb0f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f41627a0 con 0x7f99f415dc40 2024-01-26T23:46:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.079+0000 7f99fa0ac700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99f415f490 0x7f99f4160b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.079+0000 7f99fa0ac700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99f415f490 0x7f99f4160b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58752/0 (socket says 172.21.15.46:58752) 2024-01-26T23:46:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.079+0000 7f99fa0ac700 1 -- 172.21.15.46:0/4009311269 learned_addr learned my addr 172.21.15.46:0/4009311269 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.079+0000 7f99f90aa700 1 --2- 172.21.15.46:0/4009311269 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99f415e010 0x7f99f4162260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.079+0000 7f99f98ab700 1 --2- 172.21.15.46:0/4009311269 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99f415dc40 0x7f99f4163940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.079+0000 7f99fa0ac700 1 -- 172.21.15.46:0/4009311269 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99f415dc40 msgr2=0x7f99f4163940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.079+0000 7f99fa0ac700 1 --2- 172.21.15.46:0/4009311269 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99f415dc40 0x7f99f4163940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.079+0000 7f99fa0ac700 1 -- 172.21.15.46:0/4009311269 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99f415e010 msgr2=0x7f99f4162260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.079+0000 7f99fa0ac700 1 --2- 172.21.15.46:0/4009311269 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99f415e010 0x7f99f4162260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.079+0000 7f99fa0ac700 1 -- 172.21.15.46:0/4009311269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ec009ae0 con 0x7f99f415f490 2024-01-26T23:46:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.079+0000 7f99fa0ac700 1 --2- 172.21.15.46:0/4009311269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99f415f490 0x7f99f4160b40 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f99f40ce460 tx=0x7f99ec018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:02.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.080+0000 7f99e9ffb700 1 -- 172.21.15.46:0/4009311269 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec008c70 con 0x7f99f415f490 2024-01-26T23:46:02.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.080+0000 7f99e9ffb700 1 -- 172.21.15.46:0/4009311269 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ec008dd0 con 0x7f99f415f490 2024-01-26T23:46:02.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.080+0000 7f99fbb0f700 1 -- 172.21.15.46:0/4009311269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f40a5560 con 0x7f99f415f490 2024-01-26T23:46:02.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.080+0000 7f99e9ffb700 1 -- 172.21.15.46:0/4009311269 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec0266b0 con 0x7f99f415f490 2024-01-26T23:46:02.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.080+0000 7f99fbb0f700 1 -- 172.21.15.46:0/4009311269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f40a5b20 con 0x7f99f415f490 2024-01-26T23:46:02.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.081+0000 7f99fbb0f700 1 -- 172.21.15.46:0/4009311269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f40a6040 con 0x7f99f415f490 2024-01-26T23:46:02.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.081+0000 7f99e9ffb700 1 -- 172.21.15.46:0/4009311269 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f99ec003040 con 0x7f99f415f490 2024-01-26T23:46:02.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.082+0000 7f99e9ffb700 1 --2- 172.21.15.46:0/4009311269 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f99d8066ea0 0x7f99d8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.082+0000 7f99e9ffb700 1 -- 172.21.15.46:0/4009311269 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f99ec0865f0 con 0x7f99f415f490 2024-01-26T23:46:02.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.082+0000 7f99f98ab700 1 --2- 172.21.15.46:0/4009311269 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f99d8066ea0 0x7f99d8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.082+0000 7f99f98ab700 1 --2- 172.21.15.46:0/4009311269 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f99d8066ea0 0x7f99d8069350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f99f0007fd0 tx=0x7f99f0007e10 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:02.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.086+0000 7f99e9ffb700 1 -- 172.21.15.46:0/4009311269 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f99f40a6040 con 0x7f99f415f490 2024-01-26T23:46:02.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.291+0000 7f99fbb0f700 1 -- 172.21.15.46:0/4009311269 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f99f404c710 con 0x7f99d8066ea0 2024-01-26T23:46:02.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.300+0000 7f99e9ffb700 1 -- 172.21.15.46:0/4009311269 <== mgr.6421 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289596 (secure 0 0 0) 0x7f99f404c710 con 0x7f99d8066ea0 2024-01-26T23:46:02.304 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:02.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.303+0000 7f99fbb0f700 1 -- 172.21.15.46:0/4009311269 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f99d8066ea0 msgr2=0x7f99d8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:02.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.303+0000 7f99fbb0f700 1 --2- 172.21.15.46:0/4009311269 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f99d8066ea0 0x7f99d8069350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f99f0007fd0 tx=0x7f99f0007e10 comp rx=0 tx=0).stop 2024-01-26T23:46:02.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.303+0000 7f99fbb0f700 1 -- 172.21.15.46:0/4009311269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99f415f490 msgr2=0x7f99f4160b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:02.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.303+0000 7f99fbb0f700 1 --2- 172.21.15.46:0/4009311269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99f415f490 0x7f99f4160b40 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f99f40ce460 tx=0x7f99ec018000 comp rx=0 tx=0).stop 2024-01-26T23:46:02.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.303+0000 7f99fbb0f700 1 -- 172.21.15.46:0/4009311269 shutdown_connections 2024-01-26T23:46:02.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.303+0000 7f99fbb0f700 1 --2- 172.21.15.46:0/4009311269 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f99d8066ea0 0x7f99d8069350 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.303+0000 7f99fbb0f700 1 --2- 172.21.15.46:0/4009311269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99f415f490 0x7f99f4160b40 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.303+0000 7f99fbb0f700 1 --2- 172.21.15.46:0/4009311269 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99f415dc40 0x7f99f4163940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.303+0000 7f99fbb0f700 1 --2- 172.21.15.46:0/4009311269 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99f415e010 0x7f99f4162260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.303+0000 7f99fbb0f700 1 -- 172.21.15.46:0/4009311269 >> 172.21.15.46:0/4009311269 conn(0x7f99f4069510 msgr2=0x7f99f406abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:02.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.303+0000 7f99fbb0f700 1 -- 172.21.15.46:0/4009311269 shutdown_connections 2024-01-26T23:46:02.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.303+0000 7f99fbb0f700 1 -- 172.21.15.46:0/4009311269 wait complete. 2024-01-26T23:46:02.307 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:46:02.317 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-26T23:46:00.861574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":592,"num_read_kb":1131,"num_write":297,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2235,"ondisk_log_size":2235,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":184,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2429864,"kb_used_data":10600,"kb_used_omap":0,"kb_used_meta":2419200,"kb_avail":752544856,"statfs":{"total":773094113280,"available":770605932544,"internally_reserved":0,"allocated":10854400,"data_stored":6044228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2477260800},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001319"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002073+0000","last_change":"2024-01-26T23:45:52.999075+0000","last_active":"2024-01-26T23:45:53.002073+0000","last_peered":"2024-01-26T23:45:53.002073+0000","last_clean":"2024-01-26T23:45:53.002073+0000","last_became_active":"2024-01-26T23:45:52.998797+0000","last_became_peered":"2024-01-26T23:45:52.998797+0000","last_unstale":"2024-01-26T23:45:53.002073+0000","last_undegraded":"2024-01-26T23:45:53.002073+0000","last_fullsized":"2024-01-26T23:45:53.002073+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:31:10.818408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.009542+0000","last_change":"2024-01-26T23:45:52.998874+0000","last_active":"2024-01-26T23:45:53.009542+0000","last_peered":"2024-01-26T23:45:53.009542+0000","last_clean":"2024-01-26T23:45:53.009542+0000","last_became_active":"2024-01-26T23:45:52.998533+0000","last_became_peered":"2024-01-26T23:45:52.998533+0000","last_unstale":"2024-01-26T23:45:53.009542+0000","last_undegraded":"2024-01-26T23:45:53.009542+0000","last_fullsized":"2024-01-26T23:45:53.009542+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:22:33.448776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.006656+0000","last_change":"2024-01-26T23:45:53.004780+0000","last_active":"2024-01-26T23:45:53.006656+0000","last_peered":"2024-01-26T23:45:53.006656+0000","last_clean":"2024-01-26T23:45:53.006656+0000","last_became_active":"2024-01-26T23:45:53.004486+0000","last_became_peered":"2024-01-26T23:45:53.004486+0000","last_unstale":"2024-01-26T23:45:53.006656+0000","last_undegraded":"2024-01-26T23:45:53.006656+0000","last_fullsized":"2024-01-26T23:45:53.006656+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:52:16.515887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.993881+0000","last_change":"2024-01-26T23:45:52.991250+0000","last_active":"2024-01-26T23:45:52.993881+0000","last_peered":"2024-01-26T23:45:52.993881+0000","last_clean":"2024-01-26T23:45:52.993881+0000","last_became_active":"2024-01-26T23:45:52.990993+0000","last_became_peered":"2024-01-26T23:45:52.990993+0000","last_unstale":"2024-01-26T23:45:52.993881+0000","last_undegraded":"2024-01-26T23:45:52.993881+0000","last_fullsized":"2024-01-26T23:45:52.993881+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:25.322381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.000409+0000","last_change":"2024-01-26T23:45:52.999059+0000","last_active":"2024-01-26T23:45:53.000409+0000","last_peered":"2024-01-26T23:45:53.000409+0000","last_clean":"2024-01-26T23:45:53.000409+0000","last_became_active":"2024-01-26T23:45:52.998752+0000","last_became_peered":"2024-01-26T23:45:52.998752+0000","last_unstale":"2024-01-26T23:45:53.000409+0000","last_undegraded":"2024-01-26T23:45:53.000409+0000","last_fullsized":"2024-01-26T23:45:53.000409+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:22:13.283742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005904+0000","last_change":"2024-01-26T23:45:53.003311+0000","last_active":"2024-01-26T23:45:53.005904+0000","last_peered":"2024-01-26T23:45:53.005904+0000","last_clean":"2024-01-26T23:45:53.005904+0000","last_became_active":"2024-01-26T23:45:53.003002+0000","last_became_peered":"2024-01-26T23:45:53.003002+0000","last_unstale":"2024-01-26T23:45:53.005904+0000","last_undegraded":"2024-01-26T23:45:53.005904+0000","last_fullsized":"2024-01-26T23:45:53.005904+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:33:04.855324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005541+0000","last_change":"2024-01-26T23:45:53.003341+0000","last_active":"2024-01-26T23:45:53.005541+0000","last_peered":"2024-01-26T23:45:53.005541+0000","last_clean":"2024-01-26T23:45:53.005541+0000","last_became_active":"2024-01-26T23:45:53.003049+0000","last_became_peered":"2024-01-26T23:45:53.003049+0000","last_unstale":"2024-01-26T23:45:53.005541+0000","last_undegraded":"2024-01-26T23:45:53.005541+0000","last_fullsized":"2024-01-26T23:45:53.005541+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:25.322381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.000275+0000","last_change":"2024-01-26T23:45:52.999099+0000","last_active":"2024-01-26T23:45:53.000275+0000","last_peered":"2024-01-26T23:45:53.000275+0000","last_clean":"2024-01-26T23:45:53.000275+0000","last_became_active":"2024-01-26T23:45:52.998795+0000","last_became_peered":"2024-01-26T23:45:52.998795+0000","last_unstale":"2024-01-26T23:45:53.000275+0000","last_undegraded":"2024-01-26T23:45:53.000275+0000","last_fullsized":"2024-01-26T23:45:53.000275+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:10:23.293703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010733+0000","last_change":"2024-01-26T23:45:53.005345+0000","last_active":"2024-01-26T23:45:53.010733+0000","last_peered":"2024-01-26T23:45:53.010733+0000","last_clean":"2024-01-26T23:45:53.010733+0000","last_became_active":"2024-01-26T23:45:53.005093+0000","last_became_peered":"2024-01-26T23:45:53.005093+0000","last_unstale":"2024-01-26T23:45:53.010733+0000","last_undegraded":"2024-01-26T23:45:53.010733+0000","last_fullsized":"2024-01-26T23:45:53.010733+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:53.299917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.997619+0000","last_change":"2024-01-26T23:45:52.993343+0000","last_active":"2024-01-26T23:45:52.997619+0000","last_peered":"2024-01-26T23:45:52.997619+0000","last_clean":"2024-01-26T23:45:52.997619+0000","last_became_active":"2024-01-26T23:45:52.993136+0000","last_became_peered":"2024-01-26T23:45:52.993136+0000","last_unstale":"2024-01-26T23:45:52.997619+0000","last_undegraded":"2024-01-26T23:45:52.997619+0000","last_fullsized":"2024-01-26T23:45:52.997619+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:21:21.636031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.013956+0000","last_change":"2024-01-26T23:45:53.010404+0000","last_active":"2024-01-26T23:45:53.013956+0000","last_peered":"2024-01-26T23:45:53.013956+0000","last_clean":"2024-01-26T23:45:53.013956+0000","last_became_active":"2024-01-26T23:45:53.010120+0000","last_became_peered":"2024-01-26T23:45:53.010120+0000","last_unstale":"2024-01-26T23:45:53.013956+0000","last_undegraded":"2024-01-26T23:45:53.013956+0000","last_fullsized":"2024-01-26T23:45:53.013956+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:58:27.010243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002076+0000","last_change":"2024-01-26T23:45:53.000714+0000","last_active":"2024-01-26T23:45:53.002076+0000","last_peered":"2024-01-26T23:45:53.002076+0000","last_clean":"2024-01-26T23:45:53.002076+0000","last_became_active":"2024-01-26T23:45:53.000514+0000","last_became_peered":"2024-01-26T23:45:53.000514+0000","last_unstale":"2024-01-26T23:45:53.002076+0000","last_undegraded":"2024-01-26T23:45:53.002076+0000","last_fullsized":"2024-01-26T23:45:53.002076+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:32:39.694027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.008249+0000","last_change":"2024-01-26T23:45:53.004100+0000","last_active":"2024-01-26T23:45:53.008249+0000","last_peered":"2024-01-26T23:45:53.008249+0000","last_clean":"2024-01-26T23:45:53.008249+0000","last_became_active":"2024-01-26T23:45:53.003792+0000","last_became_peered":"2024-01-26T23:45:53.003792+0000","last_unstale":"2024-01-26T23:45:53.008249+0000","last_undegraded":"2024-01-26T23:45:53.008249+0000","last_fullsized":"2024-01-26T23:45:53.008249+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:54:42.637169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005453+0000","last_change":"2024-01-26T23:45:53.002596+0000","last_active":"2024-01-26T23:45:53.005453+0000","last_peered":"2024-01-26T23:45:53.005453+0000","last_clean":"2024-01-26T23:45:53.005453+0000","last_became_active":"2024-01-26T23:45:53.002292+0000","last_became_peered":"2024-01-26T23:45:53.002292+0000","last_unstale":"2024-01-26T23:45:53.005453+0000","last_undegraded":"2024-01-26T23:45:53.005453+0000","last_fullsized":"2024-01-26T23:45:53.005453+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:01.271422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001869+0000","last_change":"2024-01-26T23:45:52.993487+0000","last_active":"2024-01-26T23:45:53.001869+0000","last_peered":"2024-01-26T23:45:53.001869+0000","last_clean":"2024-01-26T23:45:53.001869+0000","last_became_active":"2024-01-26T23:45:52.993157+0000","last_became_peered":"2024-01-26T23:45:52.993157+0000","last_unstale":"2024-01-26T23:45:53.001869+0000","last_undegraded":"2024-01-26T23:45:53.001869+0000","last_fullsized":"2024-01-26T23:45:53.001869+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:44:24.788145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002037+0000","last_change":"2024-01-26T23:45:52.999171+0000","last_active":"2024-01-26T23:45:53.002037+0000","last_peered":"2024-01-26T23:45:53.002037+0000","last_clean":"2024-01-26T23:45:53.002037+0000","last_became_active":"2024-01-26T23:45:52.998914+0000","last_became_peered":"2024-01-26T23:45:52.998914+0000","last_unstale":"2024-01-26T23:45:53.002037+0000","last_undegraded":"2024-01-26T23:45:53.002037+0000","last_fullsized":"2024-01-26T23:45:53.002037+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:39:01.271422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.999397+0000","last_change":"2024-01-26T23:45:52.996490+0000","last_active":"2024-01-26T23:45:52.999397+0000","last_peered":"2024-01-26T23:45:52.999397+0000","last_clean":"2024-01-26T23:45:52.999397+0000","last_became_active":"2024-01-26T23:45:52.996165+0000","last_became_peered":"2024-01-26T23:45:52.996165+0000","last_unstale":"2024-01-26T23:45:52.999397+0000","last_undegraded":"2024-01-26T23:45:52.999397+0000","last_fullsized":"2024-01-26T23:45:52.999397+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:10:03.768025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002692+0000","last_change":"2024-01-26T23:45:52.995031+0000","last_active":"2024-01-26T23:45:53.002692+0000","last_peered":"2024-01-26T23:45:53.002692+0000","last_clean":"2024-01-26T23:45:53.002692+0000","last_became_active":"2024-01-26T23:45:52.994711+0000","last_became_peered":"2024-01-26T23:45:52.994711+0000","last_unstale":"2024-01-26T23:45:53.002692+0000","last_undegraded":"2024-01-26T23:45:53.002692+0000","last_fullsized":"2024-01-26T23:45:53.002692+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:20.934227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.014705+0000","last_change":"2024-01-26T23:45:53.010620+0000","last_active":"2024-01-26T23:45:53.014705+0000","last_peered":"2024-01-26T23:45:53.014705+0000","last_clean":"2024-01-26T23:45:53.014705+0000","last_became_active":"2024-01-26T23:45:53.009789+0000","last_became_peered":"2024-01-26T23:45:53.009789+0000","last_unstale":"2024-01-26T23:45:53.014705+0000","last_undegraded":"2024-01-26T23:45:53.014705+0000","last_fullsized":"2024-01-26T23:45:53.014705+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:31:10.818408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.999364+0000","last_change":"2024-01-26T23:45:52.997158+0000","last_active":"2024-01-26T23:45:52.999364+0000","last_peered":"2024-01-26T23:45:52.999364+0000","last_clean":"2024-01-26T23:45:52.999364+0000","last_became_active":"2024-01-26T23:45:52.996822+0000","last_became_peered":"2024-01-26T23:45:52.996822+0000","last_unstale":"2024-01-26T23:45:52.999364+0000","last_undegraded":"2024-01-26T23:45:52.999364+0000","last_fullsized":"2024-01-26T23:45:52.999364+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:57.542006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002691+0000","last_change":"2024-01-26T23:45:53.000510+0000","last_active":"2024-01-26T23:45:53.002691+0000","last_peered":"2024-01-26T23:45:53.002691+0000","last_clean":"2024-01-26T23:45:53.002691+0000","last_became_active":"2024-01-26T23:45:53.000231+0000","last_became_peered":"2024-01-26T23:45:53.000231+0000","last_unstale":"2024-01-26T23:45:53.002691+0000","last_undegraded":"2024-01-26T23:45:53.002691+0000","last_fullsized":"2024-01-26T23:45:53.002691+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:12:46.926098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.009952+0000","last_change":"2024-01-26T23:45:53.006604+0000","last_active":"2024-01-26T23:45:53.009952+0000","last_peered":"2024-01-26T23:45:53.009952+0000","last_clean":"2024-01-26T23:45:53.009952+0000","last_became_active":"2024-01-26T23:45:53.006289+0000","last_became_peered":"2024-01-26T23:45:53.006289+0000","last_unstale":"2024-01-26T23:45:53.009952+0000","last_undegraded":"2024-01-26T23:45:53.009952+0000","last_fullsized":"2024-01-26T23:45:53.009952+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:19:33.562254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002680+0000","last_change":"2024-01-26T23:45:52.998559+0000","last_active":"2024-01-26T23:45:53.002680+0000","last_peered":"2024-01-26T23:45:53.002680+0000","last_clean":"2024-01-26T23:45:53.002680+0000","last_became_active":"2024-01-26T23:45:52.998284+0000","last_became_peered":"2024-01-26T23:45:52.998284+0000","last_unstale":"2024-01-26T23:45:53.002680+0000","last_undegraded":"2024-01-26T23:45:53.002680+0000","last_fullsized":"2024-01-26T23:45:53.002680+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:23.239547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010618+0000","last_change":"2024-01-26T23:45:53.002633+0000","last_active":"2024-01-26T23:45:53.010618+0000","last_peered":"2024-01-26T23:45:53.010618+0000","last_clean":"2024-01-26T23:45:53.010618+0000","last_became_active":"2024-01-26T23:45:53.002349+0000","last_became_peered":"2024-01-26T23:45:53.002349+0000","last_unstale":"2024-01-26T23:45:53.010618+0000","last_undegraded":"2024-01-26T23:45:53.010618+0000","last_fullsized":"2024-01-26T23:45:53.010618+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:48:28.528692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.011887+0000","last_change":"2024-01-26T23:45:53.006070+0000","last_active":"2024-01-26T23:45:53.011887+0000","last_peered":"2024-01-26T23:45:53.011887+0000","last_clean":"2024-01-26T23:45:53.011887+0000","last_became_active":"2024-01-26T23:45:53.005736+0000","last_became_peered":"2024-01-26T23:45:53.005736+0000","last_unstale":"2024-01-26T23:45:53.011887+0000","last_undegraded":"2024-01-26T23:45:53.011887+0000","last_fullsized":"2024-01-26T23:45:53.011887+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:57:13.832289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002050+0000","last_change":"2024-01-26T23:45:53.001275+0000","last_active":"2024-01-26T23:45:53.002050+0000","last_peered":"2024-01-26T23:45:53.002050+0000","last_clean":"2024-01-26T23:45:53.002050+0000","last_became_active":"2024-01-26T23:45:53.001040+0000","last_became_peered":"2024-01-26T23:45:53.001040+0000","last_unstale":"2024-01-26T23:45:53.002050+0000","last_undegraded":"2024-01-26T23:45:53.002050+0000","last_fullsized":"2024-01-26T23:45:53.002050+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:53.299917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.003152+0000","last_change":"2024-01-26T23:45:53.001405+0000","last_active":"2024-01-26T23:45:53.003152+0000","last_peered":"2024-01-26T23:45:53.003152+0000","last_clean":"2024-01-26T23:45:53.003152+0000","last_became_active":"2024-01-26T23:45:53.001141+0000","last_became_peered":"2024-01-26T23:45:53.001141+0000","last_unstale":"2024-01-26T23:45:53.003152+0000","last_undegraded":"2024-01-26T23:45:53.003152+0000","last_fullsized":"2024-01-26T23:45:53.003152+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:46:16.885651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.999343+0000","last_change":"2024-01-26T23:45:52.996567+0000","last_active":"2024-01-26T23:45:52.999343+0000","last_peered":"2024-01-26T23:45:52.999343+0000","last_clean":"2024-01-26T23:45:52.999343+0000","last_became_active":"2024-01-26T23:45:52.996241+0000","last_became_peered":"2024-01-26T23:45:52.996241+0000","last_unstale":"2024-01-26T23:45:52.999343+0000","last_undegraded":"2024-01-26T23:45:52.999343+0000","last_fullsized":"2024-01-26T23:45:52.999343+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:22:32.327755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001218+0000","last_change":"2024-01-26T23:45:52.994893+0000","last_active":"2024-01-26T23:45:53.001218+0000","last_peered":"2024-01-26T23:45:53.001218+0000","last_clean":"2024-01-26T23:45:53.001218+0000","last_became_active":"2024-01-26T23:45:52.994613+0000","last_became_peered":"2024-01-26T23:45:52.994613+0000","last_unstale":"2024-01-26T23:45:53.001218+0000","last_undegraded":"2024-01-26T23:45:53.001218+0000","last_fullsized":"2024-01-26T23:45:53.001218+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:09:09.561092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010916+0000","last_change":"2024-01-26T23:45:53.010217+0000","last_active":"2024-01-26T23:45:53.010916+0000","last_peered":"2024-01-26T23:45:53.010916+0000","last_clean":"2024-01-26T23:45:53.010916+0000","last_became_active":"2024-01-26T23:45:53.010005+0000","last_became_peered":"2024-01-26T23:45:53.010005+0000","last_unstale":"2024-01-26T23:45:53.010916+0000","last_undegraded":"2024-01-26T23:45:53.010916+0000","last_fullsized":"2024-01-26T23:45:53.010916+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:33:04.855324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.002434+0000","last_change":"2024-01-26T23:45:52.997178+0000","last_active":"2024-01-26T23:45:53.002434+0000","last_peered":"2024-01-26T23:45:53.002434+0000","last_clean":"2024-01-26T23:45:53.002434+0000","last_became_active":"2024-01-26T23:45:52.996868+0000","last_became_peered":"2024-01-26T23:45:52.996868+0000","last_unstale":"2024-01-26T23:45:53.002434+0000","last_undegraded":"2024-01-26T23:45:53.002434+0000","last_fullsized":"2024-01-26T23:45:53.002434+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:07:38.558915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001885+0000","last_change":"2024-01-26T23:45:52.999010+0000","last_active":"2024-01-26T23:45:53.001885+0000","last_peered":"2024-01-26T23:45:53.001885+0000","last_clean":"2024-01-26T23:45:53.001885+0000","last_became_active":"2024-01-26T23:45:52.998758+0000","last_became_peered":"2024-01-26T23:45:52.998758+0000","last_unstale":"2024-01-26T23:45:53.001885+0000","last_undegraded":"2024-01-26T23:45:53.001885+0000","last_fullsized":"2024-01-26T23:45:53.001885+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:57:56.663833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005249+0000","last_change":"2024-01-26T23:45:52.999818+0000","last_active":"2024-01-26T23:45:53.005249+0000","last_peered":"2024-01-26T23:45:53.005249+0000","last_clean":"2024-01-26T23:45:53.005249+0000","last_became_active":"2024-01-26T23:45:52.999503+0000","last_became_peered":"2024-01-26T23:45:52.999503+0000","last_unstale":"2024-01-26T23:45:53.005249+0000","last_undegraded":"2024-01-26T23:45:53.005249+0000","last_fullsized":"2024-01-26T23:45:53.005249+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:47.231142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"97'4","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.018485+0000","last_change":"2024-01-26T23:45:51.994990+0000","last_active":"2024-01-26T23:45:53.018485+0000","last_peered":"2024-01-26T23:45:53.018485+0000","last_clean":"2024-01-26T23:45:53.018485+0000","last_became_active":"2024-01-26T23:45:51.994697+0000","last_became_peered":"2024-01-26T23:45:51.994697+0000","last_unstale":"2024-01-26T23:45:53.018485+0000","last_undegraded":"2024-01-26T23:45:53.018485+0000","last_fullsized":"2024-01-26T23:45:53.018485+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:25:30.884726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.009449+0000","last_change":"2024-01-26T23:45:52.995466+0000","last_active":"2024-01-26T23:45:53.009449+0000","last_peered":"2024-01-26T23:45:53.009449+0000","last_clean":"2024-01-26T23:45:53.009449+0000","last_became_active":"2024-01-26T23:45:52.995141+0000","last_became_peered":"2024-01-26T23:45:52.995141+0000","last_unstale":"2024-01-26T23:45:53.009449+0000","last_undegraded":"2024-01-26T23:45:53.009449+0000","last_fullsized":"2024-01-26T23:45:53.009449+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:08.999844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"97'1","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.015519+0000","last_change":"2024-01-26T23:45:51.996172+0000","last_active":"2024-01-26T23:45:53.015519+0000","last_peered":"2024-01-26T23:45:53.015519+0000","last_clean":"2024-01-26T23:45:53.015519+0000","last_became_active":"2024-01-26T23:45:51.995912+0000","last_became_peered":"2024-01-26T23:45:51.995912+0000","last_unstale":"2024-01-26T23:45:53.015519+0000","last_undegraded":"2024-01-26T23:45:53.015519+0000","last_fullsized":"2024-01-26T23:45:53.015519+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:59.450936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.008191+0000","last_change":"2024-01-26T23:45:53.004589+0000","last_active":"2024-01-26T23:45:53.008191+0000","last_peered":"2024-01-26T23:45:53.008191+0000","last_clean":"2024-01-26T23:45:53.008191+0000","last_became_active":"2024-01-26T23:45:53.004327+0000","last_became_peered":"2024-01-26T23:45:53.004327+0000","last_unstale":"2024-01-26T23:45:53.008191+0000","last_undegraded":"2024-01-26T23:45:53.008191+0000","last_fullsized":"2024-01-26T23:45:53.008191+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:27:37.169156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.020151+0000","last_change":"2024-01-26T23:45:51.997038+0000","last_active":"2024-01-26T23:45:53.020151+0000","last_peered":"2024-01-26T23:45:53.020151+0000","last_clean":"2024-01-26T23:45:53.020151+0000","last_became_active":"2024-01-26T23:45:51.996797+0000","last_became_peered":"2024-01-26T23:45:51.996797+0000","last_unstale":"2024-01-26T23:45:53.020151+0000","last_undegraded":"2024-01-26T23:45:53.020151+0000","last_fullsized":"2024-01-26T23:45:53.020151+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:24.317867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010756+0000","last_change":"2024-01-26T23:45:53.005919+0000","last_active":"2024-01-26T23:45:53.010756+0000","last_peered":"2024-01-26T23:45:53.010756+0000","last_clean":"2024-01-26T23:45:53.010756+0000","last_became_active":"2024-01-26T23:45:53.005721+0000","last_became_peered":"2024-01-26T23:45:53.005721+0000","last_unstale":"2024-01-26T23:45:53.010756+0000","last_undegraded":"2024-01-26T23:45:53.010756+0000","last_fullsized":"2024-01-26T23:45:53.010756+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:21:21.636031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.015980+0000","last_change":"2024-01-26T23:45:51.994444+0000","last_active":"2024-01-26T23:45:53.015980+0000","last_peered":"2024-01-26T23:45:53.015980+0000","last_clean":"2024-01-26T23:45:53.015980+0000","last_became_active":"2024-01-26T23:45:51.994036+0000","last_became_peered":"2024-01-26T23:45:51.994036+0000","last_unstale":"2024-01-26T23:45:53.015980+0000","last_undegraded":"2024-01-26T23:45:53.015980+0000","last_fullsized":"2024-01-26T23:45:53.015980+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:42:07.246862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010005+0000","last_change":"2024-01-26T23:45:53.001042+0000","last_active":"2024-01-26T23:45:53.010005+0000","last_peered":"2024-01-26T23:45:53.010005+0000","last_clean":"2024-01-26T23:45:53.010005+0000","last_became_active":"2024-01-26T23:45:53.000759+0000","last_became_peered":"2024-01-26T23:45:53.000759+0000","last_unstale":"2024-01-26T23:45:53.010005+0000","last_undegraded":"2024-01-26T23:45:53.010005+0000","last_fullsized":"2024-01-26T23:45:53.010005+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:48:01.627096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"97'1","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.015830+0000","last_change":"2024-01-26T23:45:51.993730+0000","last_active":"2024-01-26T23:45:53.015830+0000","last_peered":"2024-01-26T23:45:53.015830+0000","last_clean":"2024-01-26T23:45:53.015830+0000","last_became_active":"2024-01-26T23:45:51.993505+0000","last_became_peered":"2024-01-26T23:45:51.993505+0000","last_unstale":"2024-01-26T23:45:53.015830+0000","last_undegraded":"2024-01-26T23:45:53.015830+0000","last_fullsized":"2024-01-26T23:45:53.015830+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:05:29.034239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.013976+0000","last_change":"2024-01-26T23:45:53.011018+0000","last_active":"2024-01-26T23:45:53.013976+0000","last_peered":"2024-01-26T23:45:53.013976+0000","last_clean":"2024-01-26T23:45:53.013976+0000","last_became_active":"2024-01-26T23:45:53.010726+0000","last_became_peered":"2024-01-26T23:45:53.010726+0000","last_unstale":"2024-01-26T23:45:53.013976+0000","last_undegraded":"2024-01-26T23:45:53.013976+0000","last_fullsized":"2024-01-26T23:45:53.013976+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:50:13.686822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"97'1","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.015467+0000","last_change":"2024-01-26T23:45:51.998440+0000","last_active":"2024-01-26T23:45:53.015467+0000","last_peered":"2024-01-26T23:45:53.015467+0000","last_clean":"2024-01-26T23:45:53.015467+0000","last_became_active":"2024-01-26T23:45:51.998181+0000","last_became_peered":"2024-01-26T23:45:51.998181+0000","last_unstale":"2024-01-26T23:45:53.015467+0000","last_undegraded":"2024-01-26T23:45:53.015467+0000","last_fullsized":"2024-01-26T23:45:53.015467+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:46.226628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.008169+0000","last_change":"2024-01-26T23:45:53.004534+0000","last_active":"2024-01-26T23:45:53.008169+0000","last_peered":"2024-01-26T23:45:53.008169+0000","last_clean":"2024-01-26T23:45:53.008169+0000","last_became_active":"2024-01-26T23:45:53.004331+0000","last_became_peered":"2024-01-26T23:45:53.004331+0000","last_unstale":"2024-01-26T23:45:53.008169+0000","last_undegraded":"2024-01-26T23:45:53.008169+0000","last_fullsized":"2024-01-26T23:45:53.008169+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:32:39.694027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.983528+0000","last_change":"2024-01-26T23:45:51.995344+0000","last_active":"2024-01-26T23:45:52.983528+0000","last_peered":"2024-01-26T23:45:52.983528+0000","last_clean":"2024-01-26T23:45:52.983528+0000","last_became_active":"2024-01-26T23:45:51.995068+0000","last_became_peered":"2024-01-26T23:45:51.995068+0000","last_unstale":"2024-01-26T23:45:52.983528+0000","last_undegraded":"2024-01-26T23:45:52.983528+0000","last_fullsized":"2024-01-26T23:45:52.983528+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:45:21.389917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001964+0000","last_change":"2024-01-26T23:45:53.000934+0000","last_active":"2024-01-26T23:45:53.001964+0000","last_peered":"2024-01-26T23:45:53.001964+0000","last_clean":"2024-01-26T23:45:53.001964+0000","last_became_active":"2024-01-26T23:45:53.000698+0000","last_became_peered":"2024-01-26T23:45:53.000698+0000","last_unstale":"2024-01-26T23:45:53.001964+0000","last_undegraded":"2024-01-26T23:45:53.001964+0000","last_fullsized":"2024-01-26T23:45:53.001964+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:53.299917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.987463+0000","last_change":"2024-01-26T23:45:52.002908+0000","last_active":"2024-01-26T23:45:52.987463+0000","last_peered":"2024-01-26T23:45:52.987463+0000","last_clean":"2024-01-26T23:45:52.987463+0000","last_became_active":"2024-01-26T23:45:52.002606+0000","last_became_peered":"2024-01-26T23:45:52.002606+0000","last_unstale":"2024-01-26T23:45:52.987463+0000","last_undegraded":"2024-01-26T23:45:52.987463+0000","last_fullsized":"2024-01-26T23:45:52.987463+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:34.171032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010594+0000","last_change":"2024-01-26T23:45:52.999941+0000","last_active":"2024-01-26T23:45:53.010594+0000","last_peered":"2024-01-26T23:45:53.010594+0000","last_clean":"2024-01-26T23:45:53.010594+0000","last_became_active":"2024-01-26T23:45:52.999657+0000","last_became_peered":"2024-01-26T23:45:52.999657+0000","last_unstale":"2024-01-26T23:45:53.010594+0000","last_undegraded":"2024-01-26T23:45:53.010594+0000","last_fullsized":"2024-01-26T23:45:53.010594+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:21:56.682642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.996692+0000","last_change":"2024-01-26T23:45:51.996342+0000","last_active":"2024-01-26T23:45:52.996692+0000","last_peered":"2024-01-26T23:45:52.996692+0000","last_clean":"2024-01-26T23:45:52.996692+0000","last_became_active":"2024-01-26T23:45:51.996106+0000","last_became_peered":"2024-01-26T23:45:51.996106+0000","last_unstale":"2024-01-26T23:45:52.996692+0000","last_undegraded":"2024-01-26T23:45:52.996692+0000","last_fullsized":"2024-01-26T23:45:52.996692+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:52:15.511373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.997518+0000","last_change":"2024-01-26T23:45:52.992386+0000","last_active":"2024-01-26T23:45:52.997518+0000","last_peered":"2024-01-26T23:45:52.997518+0000","last_clean":"2024-01-26T23:45:52.997518+0000","last_became_active":"2024-01-26T23:45:52.992145+0000","last_became_peered":"2024-01-26T23:45:52.992145+0000","last_unstale":"2024-01-26T23:45:52.997518+0000","last_undegraded":"2024-01-26T23:45:52.997518+0000","last_fullsized":"2024-01-26T23:45:52.997518+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:12:46.926098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.987056+0000","last_change":"2024-01-26T23:45:51.993792+0000","last_active":"2024-01-26T23:45:52.987056+0000","last_peered":"2024-01-26T23:45:52.987056+0000","last_clean":"2024-01-26T23:45:52.987056+0000","last_became_active":"2024-01-26T23:45:51.993519+0000","last_became_peered":"2024-01-26T23:45:51.993519+0000","last_unstale":"2024-01-26T23:45:52.987056+0000","last_undegraded":"2024-01-26T23:45:52.987056+0000","last_fullsized":"2024-01-26T23:45:52.987056+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:05:31.172516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.013407+0000","last_change":"2024-01-26T23:45:53.009145+0000","last_active":"2024-01-26T23:45:53.013407+0000","last_peered":"2024-01-26T23:45:53.013407+0000","last_clean":"2024-01-26T23:45:53.013407+0000","last_became_active":"2024-01-26T23:45:53.008642+0000","last_became_peered":"2024-01-26T23:45:53.008642+0000","last_unstale":"2024-01-26T23:45:53.013407+0000","last_undegraded":"2024-01-26T23:45:53.013407+0000","last_fullsized":"2024-01-26T23:45:53.013407+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:27:27.092910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"97'3","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.017982+0000","last_change":"2024-01-26T23:45:52.000759+0000","last_active":"2024-01-26T23:45:53.017982+0000","last_peered":"2024-01-26T23:45:53.017982+0000","last_clean":"2024-01-26T23:45:53.017982+0000","last_became_active":"2024-01-26T23:45:52.000442+0000","last_became_peered":"2024-01-26T23:45:52.000442+0000","last_unstale":"2024-01-26T23:45:53.017982+0000","last_undegraded":"2024-01-26T23:45:53.017982+0000","last_fullsized":"2024-01-26T23:45:53.017982+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:36:59.186830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.006905+0000","last_change":"2024-01-26T23:45:53.004186+0000","last_active":"2024-01-26T23:45:53.006905+0000","last_peered":"2024-01-26T23:45:53.006905+0000","last_clean":"2024-01-26T23:45:53.006905+0000","last_became_active":"2024-01-26T23:45:53.003905+0000","last_became_peered":"2024-01-26T23:45:53.003905+0000","last_unstale":"2024-01-26T23:45:53.006905+0000","last_undegraded":"2024-01-26T23:45:53.006905+0000","last_fullsized":"2024-01-26T23:45:53.006905+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:24:29.016321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"97'6","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.019760+0000","last_change":"2024-01-26T23:45:51.990179+0000","last_active":"2024-01-26T23:45:53.019760+0000","last_peered":"2024-01-26T23:45:53.019760+0000","last_clean":"2024-01-26T23:45:53.019760+0000","last_became_active":"2024-01-26T23:45:51.989929+0000","last_became_peered":"2024-01-26T23:45:51.989929+0000","last_unstale":"2024-01-26T23:45:53.019760+0000","last_undegraded":"2024-01-26T23:45:53.019760+0000","last_fullsized":"2024-01-26T23:45:53.019760+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:44.905097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.008126+0000","last_change":"2024-01-26T23:45:53.004976+0000","last_active":"2024-01-26T23:45:53.008126+0000","last_peered":"2024-01-26T23:45:53.008126+0000","last_clean":"2024-01-26T23:45:53.008126+0000","last_became_active":"2024-01-26T23:45:53.004775+0000","last_became_peered":"2024-01-26T23:45:53.004775+0000","last_unstale":"2024-01-26T23:45:53.008126+0000","last_undegraded":"2024-01-26T23:45:53.008126+0000","last_fullsized":"2024-01-26T23:45:53.008126+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:27:06.530418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.015768+0000","last_change":"2024-01-26T23:45:51.989693+0000","last_active":"2024-01-26T23:45:53.015768+0000","last_peered":"2024-01-26T23:45:53.015768+0000","last_clean":"2024-01-26T23:45:53.015768+0000","last_became_active":"2024-01-26T23:45:51.989400+0000","last_became_peered":"2024-01-26T23:45:51.989400+0000","last_unstale":"2024-01-26T23:45:53.015768+0000","last_undegraded":"2024-01-26T23:45:53.015768+0000","last_fullsized":"2024-01-26T23:45:53.015768+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:40:36.924823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.996681+0000","last_change":"2024-01-26T23:45:52.992894+0000","last_active":"2024-01-26T23:45:52.996681+0000","last_peered":"2024-01-26T23:45:52.996681+0000","last_clean":"2024-01-26T23:45:52.996681+0000","last_became_active":"2024-01-26T23:45:52.992666+0000","last_became_peered":"2024-01-26T23:45:52.992666+0000","last_unstale":"2024-01-26T23:45:52.996681+0000","last_undegraded":"2024-01-26T23:45:52.996681+0000","last_fullsized":"2024-01-26T23:45:52.996681+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:27:06.530418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.995858+0000","last_change":"2024-01-26T23:45:52.004858+0000","last_active":"2024-01-26T23:45:52.995858+0000","last_peered":"2024-01-26T23:45:52.995858+0000","last_clean":"2024-01-26T23:45:52.995858+0000","last_became_active":"2024-01-26T23:45:52.004609+0000","last_became_peered":"2024-01-26T23:45:52.004609+0000","last_unstale":"2024-01-26T23:45:52.995858+0000","last_undegraded":"2024-01-26T23:45:52.995858+0000","last_fullsized":"2024-01-26T23:45:52.995858+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:12:45.921584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.997360+0000","last_change":"2024-01-26T23:45:52.992493+0000","last_active":"2024-01-26T23:45:52.997360+0000","last_peered":"2024-01-26T23:45:52.997360+0000","last_clean":"2024-01-26T23:45:52.997360+0000","last_became_active":"2024-01-26T23:45:52.992149+0000","last_became_peered":"2024-01-26T23:45:52.992149+0000","last_unstale":"2024-01-26T23:45:52.997360+0000","last_undegraded":"2024-01-26T23:45:52.997360+0000","last_fullsized":"2024-01-26T23:45:52.997360+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:06.428462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.986442+0000","last_change":"2024-01-26T23:45:51.995486+0000","last_active":"2024-01-26T23:45:52.986442+0000","last_peered":"2024-01-26T23:45:52.986442+0000","last_clean":"2024-01-26T23:45:52.986442+0000","last_became_active":"2024-01-26T23:45:51.995169+0000","last_became_peered":"2024-01-26T23:45:51.995169+0000","last_unstale":"2024-01-26T23:45:52.986442+0000","last_undegraded":"2024-01-26T23:45:52.986442+0000","last_fullsized":"2024-01-26T23:45:52.986442+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:31:47.780797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010570+0000","last_change":"2024-01-26T23:45:53.006345+0000","last_active":"2024-01-26T23:45:53.010570+0000","last_peered":"2024-01-26T23:45:53.010570+0000","last_clean":"2024-01-26T23:45:53.010570+0000","last_became_active":"2024-01-26T23:45:53.006080+0000","last_became_peered":"2024-01-26T23:45:53.006080+0000","last_unstale":"2024-01-26T23:45:53.010570+0000","last_undegraded":"2024-01-26T23:45:53.010570+0000","last_fullsized":"2024-01-26T23:45:53.010570+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:32:39.694027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"97'8","reported_seq":18,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.020619+0000","last_change":"2024-01-26T23:45:51.994969+0000","last_active":"2024-01-26T23:45:53.020619+0000","last_peered":"2024-01-26T23:45:53.020619+0000","last_clean":"2024-01-26T23:45:53.020619+0000","last_became_active":"2024-01-26T23:45:51.994689+0000","last_became_peered":"2024-01-26T23:45:51.994689+0000","last_unstale":"2024-01-26T23:45:53.020619+0000","last_undegraded":"2024-01-26T23:45:53.020619+0000","last_fullsized":"2024-01-26T23:45:53.020619+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:01:04.622959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.007301+0000","last_change":"2024-01-26T23:45:53.006377+0000","last_active":"2024-01-26T23:45:53.007301+0000","last_peered":"2024-01-26T23:45:53.007301+0000","last_clean":"2024-01-26T23:45:53.007301+0000","last_became_active":"2024-01-26T23:45:53.006050+0000","last_became_peered":"2024-01-26T23:45:53.006050+0000","last_unstale":"2024-01-26T23:45:53.007301+0000","last_undegraded":"2024-01-26T23:45:53.007301+0000","last_fullsized":"2024-01-26T23:45:53.007301+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:33:12.010368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"97'1","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.017369+0000","last_change":"2024-01-26T23:45:52.006988+0000","last_active":"2024-01-26T23:45:53.017369+0000","last_peered":"2024-01-26T23:45:53.017369+0000","last_clean":"2024-01-26T23:45:53.017369+0000","last_became_active":"2024-01-26T23:45:52.006827+0000","last_became_peered":"2024-01-26T23:45:52.006827+0000","last_unstale":"2024-01-26T23:45:53.017369+0000","last_undegraded":"2024-01-26T23:45:53.017369+0000","last_fullsized":"2024-01-26T23:45:53.017369+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:05.423948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.006509+0000","last_change":"2024-01-26T23:45:53.001552+0000","last_active":"2024-01-26T23:45:53.006509+0000","last_peered":"2024-01-26T23:45:53.006509+0000","last_clean":"2024-01-26T23:45:53.006509+0000","last_became_active":"2024-01-26T23:45:53.001246+0000","last_became_peered":"2024-01-26T23:45:53.001246+0000","last_unstale":"2024-01-26T23:45:53.006509+0000","last_undegraded":"2024-01-26T23:45:53.006509+0000","last_fullsized":"2024-01-26T23:45:53.006509+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:57:36.021844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.015324+0000","last_change":"2024-01-26T23:45:51.999517+0000","last_active":"2024-01-26T23:45:53.015324+0000","last_peered":"2024-01-26T23:45:53.015324+0000","last_clean":"2024-01-26T23:45:53.015324+0000","last_became_active":"2024-01-26T23:45:51.999248+0000","last_became_peered":"2024-01-26T23:45:51.999248+0000","last_unstale":"2024-01-26T23:45:53.015324+0000","last_undegraded":"2024-01-26T23:45:53.015324+0000","last_fullsized":"2024-01-26T23:45:53.015324+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:40:36.924823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.008199+0000","last_change":"2024-01-26T23:45:53.003010+0000","last_active":"2024-01-26T23:45:53.008199+0000","last_peered":"2024-01-26T23:45:53.008199+0000","last_clean":"2024-01-26T23:45:53.008199+0000","last_became_active":"2024-01-26T23:45:53.002641+0000","last_became_peered":"2024-01-26T23:45:53.002641+0000","last_unstale":"2024-01-26T23:45:53.008199+0000","last_undegraded":"2024-01-26T23:45:53.008199+0000","last_fullsized":"2024-01-26T23:45:53.008199+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:06:16.550968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"97'1","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.018074+0000","last_change":"2024-01-26T23:45:52.001332+0000","last_active":"2024-01-26T23:45:53.018074+0000","last_peered":"2024-01-26T23:45:53.018074+0000","last_clean":"2024-01-26T23:45:53.018074+0000","last_became_active":"2024-01-26T23:45:51.996690+0000","last_became_peered":"2024-01-26T23:45:51.996690+0000","last_unstale":"2024-01-26T23:45:53.018074+0000","last_undegraded":"2024-01-26T23:45:53.018074+0000","last_fullsized":"2024-01-26T23:45:53.018074+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:54:36.084818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.007512+0000","last_change":"2024-01-26T23:45:53.006606+0000","last_active":"2024-01-26T23:45:53.007512+0000","last_peered":"2024-01-26T23:45:53.007512+0000","last_clean":"2024-01-26T23:45:53.007512+0000","last_became_active":"2024-01-26T23:45:53.005949+0000","last_became_peered":"2024-01-26T23:45:53.005949+0000","last_unstale":"2024-01-26T23:45:53.007512+0000","last_undegraded":"2024-01-26T23:45:53.007512+0000","last_fullsized":"2024-01-26T23:45:53.007512+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:57:56.663833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.987187+0000","last_change":"2024-01-26T23:45:52.007285+0000","last_active":"2024-01-26T23:45:52.987187+0000","last_peered":"2024-01-26T23:45:52.987187+0000","last_clean":"2024-01-26T23:45:52.987187+0000","last_became_active":"2024-01-26T23:45:52.006952+0000","last_became_peered":"2024-01-26T23:45:52.006952+0000","last_unstale":"2024-01-26T23:45:52.987187+0000","last_undegraded":"2024-01-26T23:45:52.987187+0000","last_fullsized":"2024-01-26T23:45:52.987187+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:50:39.847921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001264+0000","last_change":"2024-01-26T23:45:52.992221+0000","last_active":"2024-01-26T23:45:53.001264+0000","last_peered":"2024-01-26T23:45:53.001264+0000","last_clean":"2024-01-26T23:45:53.001264+0000","last_became_active":"2024-01-26T23:45:52.991944+0000","last_became_peered":"2024-01-26T23:45:52.991944+0000","last_unstale":"2024-01-26T23:45:53.001264+0000","last_undegraded":"2024-01-26T23:45:53.001264+0000","last_fullsized":"2024-01-26T23:45:53.001264+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:28:47.388074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.017086+0000","last_change":"2024-01-26T23:45:52.004539+0000","last_active":"2024-01-26T23:45:53.017086+0000","last_peered":"2024-01-26T23:45:53.017086+0000","last_clean":"2024-01-26T23:45:53.017086+0000","last_became_active":"2024-01-26T23:45:52.004213+0000","last_became_peered":"2024-01-26T23:45:52.004213+0000","last_unstale":"2024-01-26T23:45:53.017086+0000","last_undegraded":"2024-01-26T23:45:53.017086+0000","last_fullsized":"2024-01-26T23:45:53.017086+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:32:38.689514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.001363+0000","last_change":"2024-01-26T23:45:52.994059+0000","last_active":"2024-01-26T23:45:53.001363+0000","last_peered":"2024-01-26T23:45:53.001363+0000","last_clean":"2024-01-26T23:45:53.001363+0000","last_became_active":"2024-01-26T23:45:52.993774+0000","last_became_peered":"2024-01-26T23:45:52.993774+0000","last_unstale":"2024-01-26T23:45:53.001363+0000","last_undegraded":"2024-01-26T23:45:53.001363+0000","last_fullsized":"2024-01-26T23:45:53.001363+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:37:33.720336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.998320+0000","last_change":"2024-01-26T23:45:51.999488+0000","last_active":"2024-01-26T23:45:52.998320+0000","last_peered":"2024-01-26T23:45:52.998320+0000","last_clean":"2024-01-26T23:45:52.998320+0000","last_became_active":"2024-01-26T23:45:51.999147+0000","last_became_peered":"2024-01-26T23:45:51.999147+0000","last_unstale":"2024-01-26T23:45:52.998320+0000","last_undegraded":"2024-01-26T23:45:52.998320+0000","last_fullsized":"2024-01-26T23:45:52.998320+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:21:20.631517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.996791+0000","last_change":"2024-01-26T23:45:52.992528+0000","last_active":"2024-01-26T23:45:52.996791+0000","last_peered":"2024-01-26T23:45:52.996791+0000","last_clean":"2024-01-26T23:45:52.996791+0000","last_became_active":"2024-01-26T23:45:52.992194+0000","last_became_peered":"2024-01-26T23:45:52.992194+0000","last_unstale":"2024-01-26T23:45:52.996791+0000","last_undegraded":"2024-01-26T23:45:52.996791+0000","last_fullsized":"2024-01-26T23:45:52.996791+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:00:00.455450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.997703+0000","last_change":"2024-01-26T23:45:51.998439+0000","last_active":"2024-01-26T23:45:52.997703+0000","last_peered":"2024-01-26T23:45:52.997703+0000","last_clean":"2024-01-26T23:45:52.997703+0000","last_became_active":"2024-01-26T23:45:51.998055+0000","last_became_peered":"2024-01-26T23:45:51.998055+0000","last_unstale":"2024-01-26T23:45:52.997703+0000","last_undegraded":"2024-01-26T23:45:52.997703+0000","last_fullsized":"2024-01-26T23:45:52.997703+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:24.317867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.997887+0000","last_change":"2024-01-26T23:45:52.995272+0000","last_active":"2024-01-26T23:45:52.997887+0000","last_peered":"2024-01-26T23:45:52.997887+0000","last_clean":"2024-01-26T23:45:52.997887+0000","last_became_active":"2024-01-26T23:45:52.995010+0000","last_became_peered":"2024-01-26T23:45:52.995010+0000","last_unstale":"2024-01-26T23:45:52.997887+0000","last_undegraded":"2024-01-26T23:45:52.997887+0000","last_fullsized":"2024-01-26T23:45:52.997887+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:56:25.322381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"97'2","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.016573+0000","last_change":"2024-01-26T23:45:51.999087+0000","last_active":"2024-01-26T23:45:53.016573+0000","last_peered":"2024-01-26T23:45:53.016573+0000","last_clean":"2024-01-26T23:45:53.016573+0000","last_became_active":"2024-01-26T23:45:51.998809+0000","last_became_peered":"2024-01-26T23:45:51.998809+0000","last_unstale":"2024-01-26T23:45:53.016573+0000","last_undegraded":"2024-01-26T23:45:53.016573+0000","last_fullsized":"2024-01-26T23:45:53.016573+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:05:29.034239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005444+0000","last_change":"2024-01-26T23:45:53.002482+0000","last_active":"2024-01-26T23:45:53.005444+0000","last_peered":"2024-01-26T23:45:53.005444+0000","last_clean":"2024-01-26T23:45:53.005444+0000","last_became_active":"2024-01-26T23:45:53.002268+0000","last_became_peered":"2024-01-26T23:45:53.002268+0000","last_unstale":"2024-01-26T23:45:53.005444+0000","last_undegraded":"2024-01-26T23:45:53.005444+0000","last_fullsized":"2024-01-26T23:45:53.005444+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:21:21.636031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.984063+0000","last_change":"2024-01-26T23:45:52.004361+0000","last_active":"2024-01-26T23:45:52.984063+0000","last_peered":"2024-01-26T23:45:52.984063+0000","last_clean":"2024-01-26T23:45:52.984063+0000","last_became_active":"2024-01-26T23:45:52.004048+0000","last_became_peered":"2024-01-26T23:45:52.004048+0000","last_unstale":"2024-01-26T23:45:52.984063+0000","last_undegraded":"2024-01-26T23:45:52.984063+0000","last_fullsized":"2024-01-26T23:45:52.984063+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:01:43.090741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.008127+0000","last_change":"2024-01-26T23:45:53.003449+0000","last_active":"2024-01-26T23:45:53.008127+0000","last_peered":"2024-01-26T23:45:53.008127+0000","last_clean":"2024-01-26T23:45:53.008127+0000","last_became_active":"2024-01-26T23:45:53.003238+0000","last_became_peered":"2024-01-26T23:45:53.003238+0000","last_unstale":"2024-01-26T23:45:53.008127+0000","last_undegraded":"2024-01-26T23:45:53.008127+0000","last_fullsized":"2024-01-26T23:45:53.008127+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:27:06.530418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.997355+0000","last_change":"2024-01-26T23:45:51.994205+0000","last_active":"2024-01-26T23:45:52.997355+0000","last_peered":"2024-01-26T23:45:52.997355+0000","last_clean":"2024-01-26T23:45:52.997355+0000","last_became_active":"2024-01-26T23:45:51.993947+0000","last_became_peered":"2024-01-26T23:45:51.993947+0000","last_unstale":"2024-01-26T23:45:52.997355+0000","last_undegraded":"2024-01-26T23:45:52.997355+0000","last_fullsized":"2024-01-26T23:45:52.997355+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:21:20.631517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.009979+0000","last_change":"2024-01-26T23:45:52.999402+0000","last_active":"2024-01-26T23:45:53.009979+0000","last_peered":"2024-01-26T23:45:53.009979+0000","last_clean":"2024-01-26T23:45:53.009979+0000","last_became_active":"2024-01-26T23:45:52.999111+0000","last_became_peered":"2024-01-26T23:45:52.999111+0000","last_unstale":"2024-01-26T23:45:53.009979+0000","last_undegraded":"2024-01-26T23:45:53.009979+0000","last_fullsized":"2024-01-26T23:45:53.009979+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:15:38.716213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.996030+0000","last_change":"2024-01-26T23:45:51.994463+0000","last_active":"2024-01-26T23:45:52.996030+0000","last_peered":"2024-01-26T23:45:52.996030+0000","last_clean":"2024-01-26T23:45:52.996030+0000","last_became_active":"2024-01-26T23:45:51.994218+0000","last_became_peered":"2024-01-26T23:45:51.994218+0000","last_unstale":"2024-01-26T23:45:52.996030+0000","last_undegraded":"2024-01-26T23:45:52.996030+0000","last_fullsized":"2024-01-26T23:45:52.996030+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:22.235033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010568+0000","last_change":"2024-01-26T23:45:53.006350+0000","last_active":"2024-01-26T23:45:53.010568+0000","last_peered":"2024-01-26T23:45:53.010568+0000","last_clean":"2024-01-26T23:45:53.010568+0000","last_became_active":"2024-01-26T23:45:53.006074+0000","last_became_peered":"2024-01-26T23:45:53.006074+0000","last_unstale":"2024-01-26T23:45:53.010568+0000","last_undegraded":"2024-01-26T23:45:53.010568+0000","last_fullsized":"2024-01-26T23:45:53.010568+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:12:46.926098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.990311+0000","last_change":"2024-01-26T23:45:51.994063+0000","last_active":"2024-01-26T23:45:52.990311+0000","last_peered":"2024-01-26T23:45:52.990311+0000","last_clean":"2024-01-26T23:45:52.990311+0000","last_became_active":"2024-01-26T23:45:51.993780+0000","last_became_peered":"2024-01-26T23:45:51.993780+0000","last_unstale":"2024-01-26T23:45:52.990311+0000","last_undegraded":"2024-01-26T23:45:52.990311+0000","last_fullsized":"2024-01-26T23:45:52.990311+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:21:06.318896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010925+0000","last_change":"2024-01-26T23:45:53.009839+0000","last_active":"2024-01-26T23:45:53.010925+0000","last_peered":"2024-01-26T23:45:53.010925+0000","last_clean":"2024-01-26T23:45:53.010925+0000","last_became_active":"2024-01-26T23:45:53.009673+0000","last_became_peered":"2024-01-26T23:45:53.009673+0000","last_unstale":"2024-01-26T23:45:53.010925+0000","last_undegraded":"2024-01-26T23:45:53.010925+0000","last_fullsized":"2024-01-26T23:45:53.010925+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:23.239547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.987040+0000","last_change":"2024-01-26T23:45:52.001285+0000","last_active":"2024-01-26T23:45:52.987040+0000","last_peered":"2024-01-26T23:45:52.987040+0000","last_clean":"2024-01-26T23:45:52.987040+0000","last_became_active":"2024-01-26T23:45:52.000978+0000","last_became_peered":"2024-01-26T23:45:52.000978+0000","last_unstale":"2024-01-26T23:45:52.987040+0000","last_undegraded":"2024-01-26T23:45:52.987040+0000","last_fullsized":"2024-01-26T23:45:52.987040+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:53:01.510934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005600+0000","last_change":"2024-01-26T23:45:53.002599+0000","last_active":"2024-01-26T23:45:53.005600+0000","last_peered":"2024-01-26T23:45:53.005600+0000","last_clean":"2024-01-26T23:45:53.005600+0000","last_became_active":"2024-01-26T23:45:53.000819+0000","last_became_peered":"2024-01-26T23:45:53.000819+0000","last_unstale":"2024-01-26T23:45:53.005600+0000","last_undegraded":"2024-01-26T23:45:53.005600+0000","last_fullsized":"2024-01-26T23:45:53.005600+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:06.428462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.992489+0000","last_change":"2024-01-26T23:45:51.996049+0000","last_active":"2024-01-26T23:45:52.992489+0000","last_peered":"2024-01-26T23:45:52.992489+0000","last_clean":"2024-01-26T23:45:52.992489+0000","last_became_active":"2024-01-26T23:45:51.995820+0000","last_became_peered":"2024-01-26T23:45:51.995820+0000","last_unstale":"2024-01-26T23:45:52.992489+0000","last_undegraded":"2024-01-26T23:45:52.992489+0000","last_fullsized":"2024-01-26T23:45:52.992489+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:39:54.556222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.005498+0000","last_change":"2024-01-26T23:45:53.002124+0000","last_active":"2024-01-26T23:45:53.005498+0000","last_peered":"2024-01-26T23:45:53.005498+0000","last_clean":"2024-01-26T23:45:53.005498+0000","last_became_active":"2024-01-26T23:45:53.000783+0000","last_became_peered":"2024-01-26T23:45:53.000783+0000","last_unstale":"2024-01-26T23:45:53.005498+0000","last_undegraded":"2024-01-26T23:45:53.005498+0000","last_fullsized":"2024-01-26T23:45:53.005498+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:00:00.455450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.990097+0000","last_change":"2024-01-26T23:45:52.007142+0000","last_active":"2024-01-26T23:45:52.990097+0000","last_peered":"2024-01-26T23:45:52.990097+0000","last_clean":"2024-01-26T23:45:52.990097+0000","last_became_active":"2024-01-26T23:45:52.006791+0000","last_became_peered":"2024-01-26T23:45:52.006791+0000","last_unstale":"2024-01-26T23:45:52.990097+0000","last_undegraded":"2024-01-26T23:45:52.990097+0000","last_fullsized":"2024-01-26T23:45:52.990097+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:30:44.905097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:53.010714+0000","last_change":"2024-01-26T23:45:53.006271+0000","last_active":"2024-01-26T23:45:53.010714+0000","last_peered":"2024-01-26T23:45:53.010714+0000","last_clean":"2024-01-26T23:45:53.010714+0000","last_became_active":"2024-01-26T23:45:53.006012+0000","last_became_peered":"2024-01-26T23:45:53.006012+0000","last_unstale":"2024-01-26T23:45:53.010714+0000","last_undegraded":"2024-01-26T23:45:53.010714+0000","last_fullsized":"2024-01-26T23:45:53.010714+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:51.976854+0000","last_clean_scrub_stamp":"2024-01-26T23:45:51.976854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:33:12.010368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-26T23:45:52.998723+0000","last_change":"2024-01-26T23:45:51.999496+0000","last_active":"2024-01-26T23:45:52.998723+0000","last_peered":"2024-01-26T23:45:52.998723+0000","last_clean":"2024-01-26T23:45:52.998723+0000","last_became_active":"2024-01-26T23:45:51.998966+0000","last_became_peered":"2024-01-26T23:45:51.998966+0000","last_unstale":"2024-01-26T23:45:52.998723+0000","last_undegraded":"2024-01-26T23:45:52.998723+0000","last_fullsized":"2024-01-26T23:45:52.998723+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:45:50.972340+0000","last_clean_scrub_stamp":"2024-01-26T23:45:50.972340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:12:45.921584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"98'2197","reported_seq":2467,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-26T23:45:56.855491+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:45:56.855491+0000","last_peered":"2024-01-26T23:45:56.855491+0000","last_clean":"2024-01-26T23:45:56.855491+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:45:56.855491+0000","last_undegraded":"2024-01-26T23:45:56.855491+0000","last_fullsized":"2024-01-26T23:45:56.855491+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":2197,"log_dups_size":0,"ondisk_log_size":2197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":592,"num_read_kb":1131,"num_write":253,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":592,"num_read_kb":1131,"num_write":253,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2197,"ondisk_log_size":2197,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575044,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299044,"kb_used_data":1756,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072796,"statfs":{"total":96636764160,"available":96330543104,"internally_reserved":0,"allocated":1798144,"data_stored":1197750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57599999999999996,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.79500000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.35699999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80400000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.37}]},{"osd":1,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57699999999999996,"15min":0.57799999999999996},"min":{"1min":0.42499999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77800000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.443},{"interface":"front","average":{"1min":0.62,"5min":0.61099999999999999,"15min":0.59899999999999998},"min":{"1min":0.46000000000000002,"5min":0.39200000000000002,"15min":0.36099999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.47199999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.38800000000000001,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.81699999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.48699999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77900000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.51600000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59999999999999998,"15min":0.59199999999999997},"min":{"1min":0.442,"5min":0.40500000000000003,"15min":0.35399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60599999999999998,"15min":0.60499999999999998},"min":{"1min":0.40100000000000002,"5min":0.39300000000000002,"15min":0.373},"max":{"1min":0.82699999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.60199999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46899999999999997,"15min":0.46100000000000002},"min":{"1min":0.36399999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.66100000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.501,"5min":0.495,"15min":0.48799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.314},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.53700000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.504,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.91700000000000004},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.40000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.74199999999999999,"5min":0.90200000000000002,"15min":0.96599999999999997},"last":0.623}]},{"osd":6,"last update":"Fri Jan 26 23:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.57999999999999996,"15min":0.55700000000000005},"min":{"1min":0.46400000000000002,"5min":0.375,"15min":0.33100000000000002},"max":{"1min":0.77900000000000003,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56200000000000006,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76500000000000001,"5min":0.85199999999999998,"15min":0.86899999999999999},"last":0.61299999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575044,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302568,"kb_used_data":1184,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069272,"statfs":{"total":96636764160,"available":96326934528,"internally_reserved":0,"allocated":1212416,"data_stored":608896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56899999999999995,"15min":0.57399999999999995},"min":{"1min":0.35999999999999999,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":0.752,"5min":0.85199999999999998,"15min":0.90200000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59099999999999997,"15min":0.59399999999999997},"min":{"1min":0.42899999999999999,"5min":0.38500000000000001,"15min":0.33600000000000002},"max":{"1min":0.79000000000000004,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.33300000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.312,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.77100000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.34399999999999997},"max":{"1min":0.75700000000000001,"5min":0.78900000000000003,"15min":0.874},"last":0.36699999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.40799999999999997,"5min":0.35499999999999998,"15min":0.29599999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.85799999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38600000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77000000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.46600000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.59499999999999997,"15min":0.59999999999999998},"min":{"1min":0.45500000000000002,"5min":0.38700000000000001,"15min":0.29999999999999999},"max":{"1min":0.79000000000000004,"5min":0.80700000000000005,"15min":0.92200000000000004},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.622,"15min":0.624},"min":{"1min":0.55000000000000004,"5min":0.39500000000000002,"15min":0.39000000000000001},"max":{"1min":0.78400000000000003,"5min":0.81100000000000005,"15min":0.84499999999999997},"last":0.44500000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.47799999999999998,"15min":0.47399999999999998},"min":{"1min":0.38100000000000001,"5min":0.29899999999999999,"15min":0.22},"max":{"1min":0.69899999999999995,"5min":0.84599999999999997,"15min":0.90200000000000002},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.501,"15min":0.499},"min":{"1min":0.44700000000000001,"5min":0.32000000000000001,"15min":0.28899999999999998},"max":{"1min":0.61899999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.377}]},{"osd":5,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48499999999999999,"15min":0.48899999999999999},"min":{"1min":0.42499999999999999,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.53400000000000003,"15min":0.53300000000000003},"min":{"1min":0.46100000000000002,"5min":0.33700000000000002,"15min":0.26200000000000001},"max":{"1min":0.75700000000000001,"5min":0.872,"15min":0.96499999999999997},"last":0.39700000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53500000000000003,"15min":0.53800000000000003},"min":{"1min":0.44400000000000001,"5min":0.32500000000000001,"15min":0.246},"max":{"1min":0.81799999999999995,"5min":0.82799999999999996,"15min":0.97199999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.308},"max":{"1min":0.73099999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.41799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575044,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302564,"kb_used_data":1180,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069276,"statfs":{"total":96636764160,"available":96326938624,"internally_reserved":0,"allocated":1208320,"data_stored":607472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29899999999999999,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":0.59699999999999998,"5min":0.621,"15min":0.72599999999999998},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.35799999999999998,"5min":0.28899999999999998,"15min":0.26000000000000001},"max":{"1min":0.73499999999999999,"5min":1.337,"15min":1.337},"last":0.29299999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41499999999999998},"min":{"1min":0.27700000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55900000000000005,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48599999999999999,"15min":0.47699999999999998},"min":{"1min":0.376,"5min":0.34699999999999998,"15min":0.28999999999999998},"max":{"1min":0.70799999999999996,"5min":0.77300000000000002,"15min":0.79600000000000004},"last":0.42999999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52200000000000002,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":0.75800000000000001,"5min":1.782,"15min":1.782},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.497,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.625,"5min":1.379,"15min":1.379},"last":0.54300000000000004}]},{"osd":4,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65300000000000002,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":0.86499999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66200000000000003,"15min":0.66600000000000004},"min":{"1min":0.49399999999999999,"5min":0.45900000000000002,"15min":0.433},"max":{"1min":0.89900000000000002,"5min":0.91900000000000004,"15min":0.95699999999999996},"last":0.56699999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.50600000000000001,"5min":0.46800000000000003,"15min":0.42799999999999999},"max":{"1min":0.879,"5min":1.208,"15min":1.208},"last":0.62},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68500000000000005,"15min":0.67100000000000004},"min":{"1min":0.53700000000000003,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":0.81599999999999995,"5min":0.96599999999999997,"15min":1.018},"last":0.48699999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.69399999999999995,"15min":0.68500000000000005},"min":{"1min":0.51800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85199999999999998,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.52900000000000003,"5min":0.437,"15min":0.42599999999999999},"max":{"1min":0.83699999999999997,"5min":1.022,"15min":1.022},"last":0.57799999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:45:13 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71599999999999997},"min":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.49399999999999999},"max":{"1min":0.93200000000000005,"5min":0.98199999999999998,"15min":1.0069999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.749,"15min":0.73199999999999998},"min":{"1min":0.53200000000000003,"5min":0.52200000000000002,"15min":0.438},"max":{"1min":0.90600000000000003,"5min":1.296,"15min":1.3049999999999999},"last":0.59799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575043,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311328,"kb_used_data":1752,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060512,"statfs":{"total":96636764160,"available":96317964288,"internally_reserved":0,"allocated":1794048,"data_stored":1197728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.28499999999999998,"5min":0.23100000000000001,"15min":0.214},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":1.284},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.46000000000000002,"15min":0.45600000000000002},"min":{"1min":0.29099999999999998,"5min":0.28899999999999998,"15min":0.27300000000000002},"max":{"1min":0.60899999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.308}]},{"osd":2,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42099999999999999,"15min":0.42999999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.26700000000000002},"max":{"1min":0.57299999999999995,"5min":0.89600000000000002,"15min":1.2230000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.38300000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":1.2430000000000001},"last":0.42099999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46899999999999997,"15min":0.46700000000000003},"min":{"1min":0.36799999999999999,"5min":0.30599999999999999,"15min":0.26800000000000002},"max":{"1min":0.68100000000000005,"5min":0.80700000000000005,"15min":1.0980000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.49399999999999999,"15min":0.49099999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.27000000000000002},"max":{"1min":0.64000000000000001,"5min":1.198,"15min":1.198},"last":0.436}]},{"osd":4,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.64300000000000002,"15min":0.65200000000000002},"min":{"1min":0.46899999999999997,"5min":0.42399999999999999,"15min":0.372},"max":{"1min":0.78700000000000003,"5min":0.93300000000000005,"15min":1.133},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64700000000000002,"15min":0.65900000000000003},"min":{"1min":0.438,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85299999999999998,"5min":0.94299999999999995,"15min":1.0169999999999999},"last":0.373}]},{"osd":5,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65100000000000002,"15min":0.66300000000000003},"min":{"1min":0.52600000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79800000000000004,"5min":0.95799999999999996,"15min":1.0880000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67400000000000004,"15min":0.68200000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.41699999999999998},"max":{"1min":0.93999999999999995,"5min":1.012,"15min":1.169},"last":0.47399999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65500000000000003,"15min":0.67700000000000005},"min":{"1min":0.505,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89800000000000002,"5min":0.91600000000000004,"15min":1.071},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.68400000000000005,"15min":0.69999999999999996},"min":{"1min":0.48299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.94599999999999995,"5min":1.028,"15min":1.1160000000000001},"last":0.55300000000000005}]},{"osd":7,"last update":"Fri Jan 26 23:45:12 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":1.004,"15min":0.82599999999999996},"min":{"1min":0.54500000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.91500000000000004,"5min":25.965,"15min":25.965},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.73899999999999999,"5min":1.109,"15min":0.86599999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.433},"max":{"1min":0.93799999999999994,"5min":33.085000000000001,"15min":33.085000000000001},"last":0.56299999999999994}]}]},{"osd":2,"up_from":13,"seq":55834575043,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306664,"kb_used_data":1184,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065176,"statfs":{"total":96636764160,"available":96322740224,"internally_reserved":0,"allocated":1212416,"data_stored":608074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.39400000000000002,"15min":0.41999999999999998},"min":{"1min":0.29299999999999998,"5min":0.25800000000000001,"15min":0.22600000000000001},"max":{"1min":0.502,"5min":0.63400000000000001,"15min":9.0540000000000003},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.24299999999999999},"max":{"1min":0.56899999999999995,"5min":0.68500000000000005,"15min":4.1970000000000001},"last":0.40899999999999997}]},{"osd":1,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42699999999999999,"15min":0.42499999999999999},"min":{"1min":0.33300000000000002,"5min":0.248,"15min":0.221},"max":{"1min":0.55300000000000005,"5min":0.70499999999999996,"15min":0.82199999999999995},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.498,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":0.629,"5min":0.65900000000000003,"15min":12.157},"last":0.65700000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47799999999999998,"15min":0.48599999999999999},"min":{"1min":0.38800000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.59299999999999997,"5min":0.65500000000000003,"15min":0.86499999999999999},"last":0.432},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.47599999999999998,"15min":0.499},"min":{"1min":0.34599999999999997,"5min":0.32200000000000001,"15min":0.30399999999999999},"max":{"1min":0.60199999999999998,"5min":0.71099999999999997,"15min":9.0879999999999992},"last":0.53000000000000003}]},{"osd":4,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65000000000000002,"15min":0.66600000000000004},"min":{"1min":0.499,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.86599999999999999,"5min":1.087,"15min":9.0150000000000006},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.65800000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.371},"max":{"1min":0.80200000000000005,"5min":0.98099999999999998,"15min":4.2480000000000002},"last":0.496}]},{"osd":5,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78600000000000003,"5min":1.0680000000000001,"15min":4.3200000000000003},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68899999999999995,"15min":0.68200000000000005},"min":{"1min":0.58899999999999997,"5min":0.39800000000000002,"15min":0.35199999999999998},"max":{"1min":0.82399999999999995,"5min":0.95199999999999996,"15min":1.085},"last":0.60599999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68899999999999995,"15min":0.68400000000000005},"min":{"1min":0.51500000000000001,"5min":0.439,"15min":0.40899999999999997},"max":{"1min":0.79400000000000004,"5min":0.995,"15min":1.1080000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69199999999999995,"15min":0.71299999999999997},"min":{"1min":0.46200000000000002,"5min":0.443,"15min":0.38800000000000001},"max":{"1min":0.83999999999999997,"5min":1.1299999999999999,"15min":8.9700000000000006},"last":0.56999999999999995}]},{"osd":7,"last update":"Fri Jan 26 23:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70799999999999996,"15min":0.71699999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.85599999999999998,"5min":1.0029999999999999,"15min":4.282},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.57499999999999996,"5min":0.48199999999999998,"15min":0.42099999999999999},"max":{"1min":0.87,"5min":1.052,"15min":1.139},"last":0.72599999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575042,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306656,"kb_used_data":1176,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065184,"statfs":{"total":96636764160,"available":96322748416,"internally_reserved":0,"allocated":1204224,"data_stored":607914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40300000000000002,"15min":0.39800000000000002},"min":{"1min":0.32300000000000001,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.56999999999999995,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44900000000000001,"15min":0.48099999999999998},"min":{"1min":0.315,"5min":0.255,"15min":0.255},"max":{"1min":0.59299999999999997,"5min":1.2290000000000001,"15min":11.968},"last":0.33700000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.496},"min":{"1min":0.33200000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.623,"5min":1.633,"15min":12.050000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.495,"5min":0.48899999999999999,"15min":0.48599999999999999},"min":{"1min":0.378,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.59999999999999998,"5min":1.0760000000000001,"15min":1.2150000000000001},"last":0.54900000000000004}]},{"osd":2,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.49299999999999999,"15min":0.51900000000000002},"min":{"1min":0.39800000000000002,"5min":0.29699999999999999,"15min":0.28999999999999998},"max":{"1min":0.70399999999999996,"5min":1.4059999999999999,"15min":12.234},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46899999999999997,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.59499999999999997,"5min":1.1870000000000001,"15min":11.925000000000001},"last":0.38300000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69599999999999995,"15min":0.70599999999999996},"min":{"1min":0.56999999999999995,"5min":0.46999999999999997,"15min":0.40200000000000002},"max":{"1min":0.84899999999999998,"5min":3.306,"15min":12.099},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.66400000000000003,"15min":0.66600000000000004},"min":{"1min":0.59599999999999997,"5min":0.40200000000000002,"15min":0.379},"max":{"1min":0.82999999999999996,"5min":0.95799999999999996,"15min":1.0649999999999999},"last":0.52100000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.66500000000000004,"15min":0.70699999999999996},"min":{"1min":0.56299999999999994,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.79800000000000004,"5min":1.593,"15min":12.199999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68999999999999995,"15min":0.67800000000000005},"min":{"1min":0.53400000000000003,"5min":0.45300000000000001,"15min":0.436},"max":{"1min":0.92300000000000004,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.44600000000000001}]},{"osd":6,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69499999999999995,"15min":0.72099999999999997},"min":{"1min":0.56100000000000005,"5min":0.442,"15min":0.442},"max":{"1min":0.88800000000000001,"5min":1.506,"15min":12.132999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70099999999999996,"15min":0.69799999999999995},"min":{"1min":0.54600000000000004,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.88200000000000001,"5min":1.427,"15min":1.427},"last":0.48799999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.72699999999999998,"15min":0.70299999999999996},"min":{"1min":0.52500000000000002,"5min":0.48499999999999999,"15min":0.42499999999999999},"max":{"1min":0.86199999999999999,"5min":3.238,"15min":3.238},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.78200000000000003},"min":{"1min":0.625,"5min":0.48899999999999999,"15min":0.48699999999999999},"max":{"1min":0.91300000000000003,"5min":3.351,"15min":12.167},"last":0.55600000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575044,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302576,"kb_used_data":1192,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069264,"statfs":{"total":96636764160,"available":96326926336,"internally_reserved":0,"allocated":1220608,"data_stored":609000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57599999999999996,"15min":0.58399999999999996},"min":{"1min":0.439,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78300000000000003,"5min":1.21,"15min":1.21},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57499999999999996,"15min":0.58399999999999996},"min":{"1min":0.44900000000000001,"5min":0.309,"15min":0.28100000000000003},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":1.161},"last":0.39600000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.44900000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":1.069},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.59399999999999997},"min":{"1min":0.47199999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":1.085},"last":0.373}]},{"osd":2,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56299999999999994,"15min":0.57499999999999996},"min":{"1min":0.48499999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76500000000000001,"5min":0.81100000000000005,"15min":1.052},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60699999999999998,"15min":0.61199999999999999},"min":{"1min":0.41399999999999998,"5min":0.39900000000000002,"15min":0.373},"max":{"1min":0.81599999999999995,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.48599999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58199999999999996,"15min":0.59399999999999997},"min":{"1min":0.41299999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.78300000000000003,"5min":0.83499999999999996,"15min":1.1299999999999999},"last":0.502},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79100000000000004,"5min":1.05,"15min":1.1419999999999999},"last":0.441}]},{"osd":5,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.317,"5min":0.25700000000000001,"15min":0.247},"max":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.40799999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48499999999999999,"15min":0.47799999999999998},"min":{"1min":0.35799999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.73899999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53300000000000003,"15min":0.53000000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.311},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.95699999999999996},"last":0.42999999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:45:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.35799999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.57699999999999996},"min":{"1min":0.42499999999999999,"5min":0.41799999999999998,"15min":0.32200000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.879},"last":0.42199999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575044,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298464,"kb_used_data":1176,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073376,"statfs":{"total":96636764160,"available":96331137024,"internally_reserved":0,"allocated":1204224,"data_stored":607394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.58899999999999997,"15min":0.57599999999999996},"min":{"1min":0.46300000000000002,"5min":0.29999999999999999,"15min":0.26000000000000001},"max":{"1min":0.78900000000000003,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57199999999999995,"15min":0.58399999999999996},"min":{"1min":0.45400000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77400000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.5}]},{"osd":1,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.625,"15min":0.623},"min":{"1min":0.45500000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79500000000000004,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.64300000000000002,"15min":0.63300000000000001},"min":{"1min":0.499,"5min":0.435,"15min":0.379},"max":{"1min":0.80500000000000005,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.54100000000000004}]},{"osd":2,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.46700000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77000000000000002,"5min":0.83699999999999997,"15min":0.83799999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60399999999999998,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81100000000000005,"5min":2.0600000000000001,"15min":6.4589999999999996},"last":0.41499999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.50700000000000001,"5min":0.35999999999999999,"15min":0.314},"max":{"1min":0.755,"5min":0.86199999999999999,"15min":0.88400000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.60599999999999998,"15min":0.61499999999999999},"min":{"1min":0.47499999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77900000000000003,"5min":2.5630000000000002,"15min":6.0629999999999997},"last":0.48199999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.47399999999999998,"15min":0.46999999999999997},"min":{"1min":0.35599999999999998,"5min":0.28100000000000003,"15min":0.26300000000000001},"max":{"1min":0.56599999999999995,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.503,"15min":0.495},"min":{"1min":0.36899999999999999,"5min":0.33000000000000002,"15min":0.28100000000000003},"max":{"1min":0.69999999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.42799999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.31,"15min":0.27000000000000002},"max":{"1min":0.66600000000000004,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53200000000000003,"15min":0.52700000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.29299999999999998},"max":{"1min":0.71899999999999997,"5min":1.99,"15min":1.99},"last":0.49099999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.54900000000000004},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.76200000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33300000000000002},"max":{"1min":0.81499999999999995,"5min":0.82199999999999995,"15min":0.83099999999999996},"last":0.50700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:46:02.322 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-26T23:46:02.322 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-26T23:46:02.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.501+0000 7f4ab5dca700 1 Processor -- start 2024-01-26T23:46:02.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.501+0000 7f4ab5dca700 1 -- start start 2024-01-26T23:46:02.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.501+0000 7f4ab5dca700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4ab0068b00 0x7f4ab0068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.501+0000 7f4ab5dca700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4ab006b910 0x7f4ab015aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.501+0000 7f4ab5dca700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab00ad380 0x7f4ab00ad800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.501+0000 7f4ab5dca700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab015a620 con 0x7f4ab0068b00 2024-01-26T23:46:02.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.501+0000 7f4ab5dca700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab0156b80 con 0x7f4ab006b910 2024-01-26T23:46:02.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.501+0000 7f4ab5dca700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab015a2b0 con 0x7f4ab00ad380 2024-01-26T23:46:02.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.502+0000 7f4aaf7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4ab0068b00 0x7f4ab0068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.502+0000 7f4aaf7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4ab0068b00 0x7f4ab0068ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58764/0 (socket says 172.21.15.46:58764) 2024-01-26T23:46:02.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.502+0000 7f4aaf7fe700 1 -- 172.21.15.46:0/1082142360 learned_addr learned my addr 172.21.15.46:0/1082142360 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:02.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.502+0000 7f4aaeffd700 1 --2- 172.21.15.46:0/1082142360 >> v2:172.21.15.106:3300/0 conn(0x7f4ab006b910 0x7f4ab015aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.502+0000 7f4aaffff700 1 --2- 172.21.15.46:0/1082142360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab00ad380 0x7f4ab00ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.502+0000 7f4aaf7fe700 1 -- 172.21.15.46:0/1082142360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab00ad380 msgr2=0x7f4ab00ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:02.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.502+0000 7f4aaf7fe700 1 --2- 172.21.15.46:0/1082142360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab00ad380 0x7f4ab00ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.502+0000 7f4aaf7fe700 1 -- 172.21.15.46:0/1082142360 >> v2:172.21.15.106:3300/0 conn(0x7f4ab006b910 msgr2=0x7f4ab015aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:02.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.502+0000 7f4aaf7fe700 1 --2- 172.21.15.46:0/1082142360 >> v2:172.21.15.106:3300/0 conn(0x7f4ab006b910 0x7f4ab015aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.502+0000 7f4aaf7fe700 1 -- 172.21.15.46:0/1082142360 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab006c140 con 0x7f4ab0068b00 2024-01-26T23:46:02.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.502+0000 7f4aaffff700 1 --2- 172.21.15.46:0/1082142360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab00ad380 0x7f4ab00ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:46:02.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.502+0000 7f4aaf7fe700 1 --2- 172.21.15.46:0/1082142360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab0068b00 0x7f4ab0068ed0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0063b80 tx=0x7f4aa0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffd918628c3f8e31 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:02.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.502+0000 7f4aae7fc700 1 -- 172.21.15.46:0/1082142360 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0005c10 con 0x7f4ab0068b00 2024-01-26T23:46:02.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.502+0000 7f4aae7fc700 1 -- 172.21.15.46:0/1082142360 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa0005d70 con 0x7f4ab0068b00 2024-01-26T23:46:02.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.502+0000 7f4aae7fc700 1 -- 172.21.15.46:0/1082142360 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0006280 con 0x7f4ab0068b00 2024-01-26T23:46:02.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.503+0000 7f4ab5dca700 1 -- 172.21.15.46:0/1082142360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab0068b00 msgr2=0x7f4ab0068ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:02.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.503+0000 7f4ab5dca700 1 --2- 172.21.15.46:0/1082142360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab0068b00 0x7f4ab0068ed0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0063b80 tx=0x7f4aa0009220 comp rx=0 tx=0).stop 2024-01-26T23:46:02.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.503+0000 7f4ab5dca700 1 -- 172.21.15.46:0/1082142360 shutdown_connections 2024-01-26T23:46:02.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.503+0000 7f4ab5dca700 1 --2- 172.21.15.46:0/1082142360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab0068b00 0x7f4ab0068ed0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.503+0000 7f4ab5dca700 1 --2- 172.21.15.46:0/1082142360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab00ad380 0x7f4ab00ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.503+0000 7f4ab5dca700 1 --2- 172.21.15.46:0/1082142360 >> v2:172.21.15.106:3300/0 conn(0x7f4ab006b910 0x7f4ab015aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.503+0000 7f4ab5dca700 1 -- 172.21.15.46:0/1082142360 >> 172.21.15.46:0/1082142360 conn(0x7f4ab00ca100 msgr2=0x7f4ab006e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:02.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.503+0000 7f4ab5dca700 1 -- 172.21.15.46:0/1082142360 shutdown_connections 2024-01-26T23:46:02.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.503+0000 7f4ab5dca700 1 -- 172.21.15.46:0/1082142360 wait complete. 2024-01-26T23:46:02.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.504+0000 7f4ab5dca700 1 Processor -- start 2024-01-26T23:46:02.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.505+0000 7f4ab5dca700 1 -- start start 2024-01-26T23:46:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.505+0000 7f4ab5dca700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab0068b00 0x7f4ab014df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.505+0000 7f4ab5dca700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ab006b910 0x7f4ab0157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.505+0000 7f4ab5dca700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab00ad380 0x7f4ab00c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.505+0000 7f4ab5dca700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab014e990 con 0x7f4ab00ad380 2024-01-26T23:46:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.505+0000 7f4ab5dca700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab00c0e70 con 0x7f4ab006b910 2024-01-26T23:46:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.505+0000 7f4ab5dca700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab0158590 con 0x7f4ab0068b00 2024-01-26T23:46:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.506+0000 7f4aaf7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab0068b00 0x7f4ab014df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.506+0000 7f4aaffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab00ad380 0x7f4ab00c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.506+0000 7f4aaeffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ab006b910 0x7f4ab0157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.506+0000 7f4aaf7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab0068b00 0x7f4ab014df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:53168/0 (socket says 172.21.15.46:53168) 2024-01-26T23:46:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.506+0000 7f4aaf7fe700 1 -- 172.21.15.46:0/1199048829 learned_addr learned my addr 172.21.15.46:0/1199048829 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.506+0000 7f4aaeffd700 1 -- 172.21.15.46:0/1199048829 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab0068b00 msgr2=0x7f4ab014df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.506+0000 7f4aaeffd700 1 --2- 172.21.15.46:0/1199048829 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab0068b00 0x7f4ab014df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.506+0000 7f4aaeffd700 1 -- 172.21.15.46:0/1199048829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab00ad380 msgr2=0x7f4ab00c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.506+0000 7f4aaeffd700 1 --2- 172.21.15.46:0/1199048829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab00ad380 0x7f4ab00c7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.506+0000 7f4aaeffd700 1 -- 172.21.15.46:0/1199048829 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa0009480 con 0x7f4ab006b910 2024-01-26T23:46:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.506+0000 7f4aaf7fe700 1 --2- 172.21.15.46:0/1199048829 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab0068b00 0x7f4ab014df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.506+0000 7f4aaeffd700 1 --2- 172.21.15.46:0/1199048829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ab006b910 0x7f4ab0157c90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4aa400ea50 tx=0x7f4aa40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.506+0000 7f4a977fe700 1 -- 172.21.15.46:0/1199048829 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa400ee90 con 0x7f4ab006b910 2024-01-26T23:46:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.506+0000 7f4a977fe700 1 -- 172.21.15.46:0/1199048829 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa400c7f0 con 0x7f4ab006b910 2024-01-26T23:46:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.506+0000 7f4ab5dca700 1 -- 172.21.15.46:0/1199048829 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab0158870 con 0x7f4ab006b910 2024-01-26T23:46:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.506+0000 7f4a977fe700 1 -- 172.21.15.46:0/1199048829 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa400cb90 con 0x7f4ab006b910 2024-01-26T23:46:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.506+0000 7f4ab5dca700 1 -- 172.21.15.46:0/1199048829 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab014e620 con 0x7f4ab006b910 2024-01-26T23:46:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.507+0000 7f4ab5dca700 1 -- 172.21.15.46:0/1199048829 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab014e480 con 0x7f4ab006b910 2024-01-26T23:46:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.508+0000 7f4a977fe700 1 -- 172.21.15.46:0/1199048829 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f4aa4013470 con 0x7f4ab006b910 2024-01-26T23:46:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.508+0000 7f4a977fe700 1 --2- 172.21.15.46:0/1199048829 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4a9c066ea0 0x7f4a9c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.508+0000 7f4a977fe700 1 -- 172.21.15.46:0/1199048829 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f4aa4084900 con 0x7f4ab006b910 2024-01-26T23:46:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.508+0000 7f4aaf7fe700 1 --2- 172.21.15.46:0/1199048829 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4a9c066ea0 0x7f4a9c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.509+0000 7f4aaf7fe700 1 --2- 172.21.15.46:0/1199048829 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4a9c066ea0 0x7f4a9c069350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009370 tx=0x7f4aa0016000 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:02.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.513+0000 7f4a977fe700 1 -- 172.21.15.46:0/1199048829 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4ab014e480 con 0x7f4ab006b910 2024-01-26T23:46:02.587 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:02.586+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:46:02.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.805+0000 7f4ab5dca700 1 -- 172.21.15.46:0/1199048829 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f4ab014e480 con 0x7f4ab006b910 2024-01-26T23:46:02.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.810+0000 7f4a977fe700 1 -- 172.21.15.46:0/1199048829 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+0 (secure 0 0 0) 0x7f4ab014e480 con 0x7f4ab006b910 2024-01-26T23:46:02.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.812+0000 7f4ab5dca700 1 -- 172.21.15.46:0/1199048829 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4a9c066ea0 msgr2=0x7f4a9c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:02.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.812+0000 7f4ab5dca700 1 --2- 172.21.15.46:0/1199048829 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4a9c066ea0 0x7f4a9c069350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009370 tx=0x7f4aa0016000 comp rx=0 tx=0).stop 2024-01-26T23:46:02.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.812+0000 7f4ab5dca700 1 -- 172.21.15.46:0/1199048829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ab006b910 msgr2=0x7f4ab0157c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:02.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.812+0000 7f4ab5dca700 1 --2- 172.21.15.46:0/1199048829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ab006b910 0x7f4ab0157c90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4aa400ea50 tx=0x7f4aa40091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.812+0000 7f4ab5dca700 1 -- 172.21.15.46:0/1199048829 shutdown_connections 2024-01-26T23:46:02.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.812+0000 7f4ab5dca700 1 --2- 172.21.15.46:0/1199048829 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4a9c066ea0 0x7f4a9c069350 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.812+0000 7f4ab5dca700 1 --2- 172.21.15.46:0/1199048829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab00ad380 0x7f4ab00c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.812+0000 7f4ab5dca700 1 --2- 172.21.15.46:0/1199048829 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab0068b00 0x7f4ab014df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.812+0000 7f4ab5dca700 1 --2- 172.21.15.46:0/1199048829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ab006b910 0x7f4ab0157c90 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.812+0000 7f4ab5dca700 1 -- 172.21.15.46:0/1199048829 >> 172.21.15.46:0/1199048829 conn(0x7f4ab00ca100 msgr2=0x7f4ab00ab870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:02.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.812+0000 7f4ab5dca700 1 -- 172.21.15.46:0/1199048829 shutdown_connections 2024-01-26T23:46:02.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.812+0000 7f4ab5dca700 1 -- 172.21.15.46:0/1199048829 wait complete. 2024-01-26T23:46:02.814 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-01-26T23:46:02.826 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:46:02.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.961+0000 7fadcd8c7700 1 Processor -- start 2024-01-26T23:46:02.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.961+0000 7fadcd8c7700 1 -- start start 2024-01-26T23:46:02.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.962+0000 7fadcd8c7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadc81596c0 0x7fadc806bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.962+0000 7fadcd8c7700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fadc815d460 0x7fadc806c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.962+0000 7fadcd8c7700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc815d8f0 con 0x7fadc8159a90 2024-01-26T23:46:02.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.962+0000 7fadcd8c7700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadc80c98c0 con 0x7fadc815d460 2024-01-26T23:46:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.962+0000 7fadcd8c7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc815dbd0 con 0x7fadc81596c0 2024-01-26T23:46:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.962+0000 7fadc67fc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fadc815d460 0x7fadc806c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.962+0000 7fadc6ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadc81596c0 0x7fadc806bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.962+0000 7fadc77fe700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fadc8159a90 0x7fadc806d310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:55854/0 (socket says 172.21.15.46:55854) 2024-01-26T23:46:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.962+0000 7fadc77fe700 1 -- 172.21.15.46:0/1036680349 learned_addr learned my addr 172.21.15.46:0/1036680349 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.963+0000 7fadc67fc700 1 -- 172.21.15.46:0/1036680349 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadc81596c0 msgr2=0x7fadc806bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.963+0000 7fadc67fc700 1 --2- 172.21.15.46:0/1036680349 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadc81596c0 0x7fadc806bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.963+0000 7fadc67fc700 1 -- 172.21.15.46:0/1036680349 >> v1:172.21.15.106:6789/0 conn(0x7fadc8159a90 legacy=0x7fadc806d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.963+0000 7fadc67fc700 1 -- 172.21.15.46:0/1036680349 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc815deb0 con 0x7fadc815d460 2024-01-26T23:46:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.963+0000 7fadc6ffd700 1 --2- 172.21.15.46:0/1036680349 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadc81596c0 0x7fadc806bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.963+0000 7fadc5ffb700 1 -- 172.21.15.46:0/1036680349 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220255238 0 0) 0x7fadc815d8f0 con 0x7fadc8159a90 2024-01-26T23:46:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.963+0000 7fadc67fc700 1 --2- 172.21.15.46:0/1036680349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadc815d460 0x7fadc806c210 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0098e0 tx=0x7fadbc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ccf77c1fe4249e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.963+0000 7fadc5ffb700 1 -- 172.21.15.46:0/1036680349 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc00b640 con 0x7fadc815d460 2024-01-26T23:46:02.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.963+0000 7fadc5ffb700 1 -- 172.21.15.46:0/1036680349 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadbc00b7a0 con 0x7fadc815d460 2024-01-26T23:46:02.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.963+0000 7fadc5ffb700 1 -- 172.21.15.46:0/1036680349 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc00bb40 con 0x7fadc815d460 2024-01-26T23:46:02.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.964+0000 7fadcd8c7700 1 -- 172.21.15.46:0/1036680349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadc815d460 msgr2=0x7fadc806c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:02.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.964+0000 7fadcd8c7700 1 --2- 172.21.15.46:0/1036680349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadc815d460 0x7fadc806c210 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0098e0 tx=0x7fadbc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.964+0000 7fadcd8c7700 1 -- 172.21.15.46:0/1036680349 shutdown_connections 2024-01-26T23:46:02.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.964+0000 7fadcd8c7700 1 --2- 172.21.15.46:0/1036680349 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadc81596c0 0x7fadc806bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.964+0000 7fadcd8c7700 1 --2- 172.21.15.46:0/1036680349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadc815d460 0x7fadc806c210 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.964+0000 7fadcd8c7700 1 -- 172.21.15.46:0/1036680349 >> 172.21.15.46:0/1036680349 conn(0x7fadc80ce530 msgr2=0x7fadc80ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:02.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.964+0000 7fadcd8c7700 1 -- 172.21.15.46:0/1036680349 shutdown_connections 2024-01-26T23:46:02.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.964+0000 7fadcd8c7700 1 -- 172.21.15.46:0/1036680349 wait complete. 2024-01-26T23:46:02.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.965+0000 7fadcd8c7700 1 Processor -- start 2024-01-26T23:46:02.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.965+0000 7fadcd8c7700 1 -- start start 2024-01-26T23:46:02.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.966+0000 7fadcd8c7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadc81596c0 0x7fadc8169cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.966+0000 7fadcd8c7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadc8159a90 0x7fadc80b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.966+0000 7fadcd8c7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadc815d460 0x7fadc81576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.966+0000 7fadcd8c7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc80af000 con 0x7fadc815d460 2024-01-26T23:46:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.966+0000 7fadcd8c7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc80af5c0 con 0x7fadc8159a90 2024-01-26T23:46:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.966+0000 7fadcd8c7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc80b34e0 con 0x7fadc81596c0 2024-01-26T23:46:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.966+0000 7fadc77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadc815d460 0x7fadc81576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.966+0000 7fadc77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadc815d460 0x7fadc81576a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58782/0 (socket says 172.21.15.46:58782) 2024-01-26T23:46:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.966+0000 7fadc77fe700 1 -- 172.21.15.46:0/733103427 learned_addr learned my addr 172.21.15.46:0/733103427 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.966+0000 7fadc67fc700 1 --2- 172.21.15.46:0/733103427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadc8159a90 0x7fadc80b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.966+0000 7fadc6ffd700 1 --2- 172.21.15.46:0/733103427 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadc81596c0 0x7fadc8169cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.966+0000 7fadc77fe700 1 -- 172.21.15.46:0/733103427 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadc81596c0 msgr2=0x7fadc8169cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.966+0000 7fadc77fe700 1 --2- 172.21.15.46:0/733103427 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadc81596c0 0x7fadc8169cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.966+0000 7fadc77fe700 1 -- 172.21.15.46:0/733103427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadc8159a90 msgr2=0x7fadc80b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.966+0000 7fadc77fe700 1 --2- 172.21.15.46:0/733103427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadc8159a90 0x7fadc80b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.967+0000 7fadc77fe700 1 -- 172.21.15.46:0/733103427 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb8009420 con 0x7fadc815d460 2024-01-26T23:46:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.967+0000 7fadc67fc700 1 --2- 172.21.15.46:0/733103427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadc8159a90 0x7fadc80b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.967+0000 7fadc77fe700 1 --2- 172.21.15.46:0/733103427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadc815d460 0x7fadc81576a0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fadc000cc90 tx=0x7fadc0001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.967+0000 7fadaeffd700 1 -- 172.21.15.46:0/733103427 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc000a8b0 con 0x7fadc815d460 2024-01-26T23:46:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.967+0000 7fadaeffd700 1 -- 172.21.15.46:0/733103427 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc000aa10 con 0x7fadc815d460 2024-01-26T23:46:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.967+0000 7fadaeffd700 1 -- 172.21.15.46:0/733103427 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc000add0 con 0x7fadc815d460 2024-01-26T23:46:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.967+0000 7fadcd8c7700 1 -- 172.21.15.46:0/733103427 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadbc009420 con 0x7fadc815d460 2024-01-26T23:46:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.967+0000 7fadcd8c7700 1 -- 172.21.15.46:0/733103427 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc80af250 con 0x7fadc815d460 2024-01-26T23:46:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.968+0000 7fadcd8c7700 1 -- 172.21.15.46:0/733103427 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadc80af730 con 0x7fadc815d460 2024-01-26T23:46:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.968+0000 7fadaeffd700 1 -- 172.21.15.46:0/733103427 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fadc0007300 con 0x7fadc815d460 2024-01-26T23:46:02.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.969+0000 7fadaeffd700 1 --2- 172.21.15.46:0/733103427 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadb4066ea0 0x7fadb4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:02.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.969+0000 7fadaeffd700 1 -- 172.21.15.46:0/733103427 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fadc0084a00 con 0x7fadc815d460 2024-01-26T23:46:02.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.969+0000 7fadc6ffd700 1 --2- 172.21.15.46:0/733103427 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadb4066ea0 0x7fadb4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:02.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.969+0000 7fadc6ffd700 1 --2- 172.21.15.46:0/733103427 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadb4066ea0 0x7fadb4069350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fadc816d1d0 tx=0x7fadb80091a0 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:02.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:02.973+0000 7fadaeffd700 1 -- 172.21.15.46:0/733103427 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fadc0051b00 con 0x7fadc815d460 2024-01-26T23:46:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.230+0000 7fadcd8c7700 1 -- 172.21.15.46:0/733103427 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadc804c710 con 0x7fadc815d460 2024-01-26T23:46:03.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.231+0000 7fadaeffd700 1 -- 172.21.15.46:0/733103427 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4605 (secure 0 0 0) 0x7fadc0055120 con 0x7fadc815d460 2024-01-26T23:46:03.231 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:03.231 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6137,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/1370365478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1370365478},{"type":"v1","addr":"172.21.15.106:6836","nonce":1370365478}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":6619,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/1980309094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1980309094},{"type":"v1","addr":"172.21.15.106:6837","nonce":1980309094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40},{"gid":6628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3910368747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3910368747},{"type":"v1","addr":"172.21.15.46:6834","nonce":3910368747}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:45:52.426977+0000","modified":"2024-01-26T23:45:56.004421+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6631":{"gid":6631,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3176096333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3176096333},{"type":"v1","addr":"172.21.15.46:6835","nonce":3176096333}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2024-01-26T23:46:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.233+0000 7fadcd8c7700 1 -- 172.21.15.46:0/733103427 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadb4066ea0 msgr2=0x7fadb4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.233+0000 7fadcd8c7700 1 --2- 172.21.15.46:0/733103427 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadb4066ea0 0x7fadb4069350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fadc816d1d0 tx=0x7fadb80091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.233+0000 7fadcd8c7700 1 -- 172.21.15.46:0/733103427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadc815d460 msgr2=0x7fadc81576a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.233+0000 7fadcd8c7700 1 --2- 172.21.15.46:0/733103427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadc815d460 0x7fadc81576a0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fadc000cc90 tx=0x7fadc0001a80 comp rx=0 tx=0).stop 2024-01-26T23:46:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.233+0000 7fadcd8c7700 1 -- 172.21.15.46:0/733103427 shutdown_connections 2024-01-26T23:46:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.233+0000 7fadcd8c7700 1 --2- 172.21.15.46:0/733103427 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadb4066ea0 0x7fadb4069350 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.233+0000 7fadcd8c7700 1 --2- 172.21.15.46:0/733103427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadc815d460 0x7fadc81576a0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.233+0000 7fadcd8c7700 1 --2- 172.21.15.46:0/733103427 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadc81596c0 0x7fadc8169cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.233+0000 7fadcd8c7700 1 --2- 172.21.15.46:0/733103427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadc8159a90 0x7fadc80b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.233+0000 7fadcd8c7700 1 -- 172.21.15.46:0/733103427 >> 172.21.15.46:0/733103427 conn(0x7fadc80ce530 msgr2=0x7fadc806dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.233+0000 7fadcd8c7700 1 -- 172.21.15.46:0/733103427 shutdown_connections 2024-01-26T23:46:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.233+0000 7fadcd8c7700 1 -- 172.21.15.46:0/733103427 wait complete. 2024-01-26T23:46:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 45 2024-01-26T23:46:03.326 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:03.325+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:46:03.650 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 45, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-26T23:45:52.426977+0000', 'modified': '2024-01-26T23:45:56.004421+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6631}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6631': {'gid': 6631, 'name': 'c', 'rank': 0, 'incarnation': 41, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6835/3176096333', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 3176096333}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 3176096333}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-26T23:46:03.650 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-26T23:46:03.650 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-26T23:46:03.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.792+0000 7f927e8f7700 1 Processor -- start 2024-01-26T23:46:03.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.792+0000 7f927e8f7700 1 -- start start 2024-01-26T23:46:03.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.793+0000 7f927e8f7700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9278159590 0x7f927815d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.793+0000 7f927e8f7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f927806c3c0 0x7f9278069370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.793+0000 7f927e8f7700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927815dd90 con 0x7f927815d8f0 2024-01-26T23:46:03.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.793+0000 7f927e8f7700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f927815e350 con 0x7f927806c3c0 2024-01-26T23:46:03.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.793+0000 7f927e8f7700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92780c97c0 con 0x7f9278159590 2024-01-26T23:46:03.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.793+0000 7f927ce94700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f927806c3c0 0x7f9278069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.793+0000 7f927ce94700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f927806c3c0 0x7f9278069370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58788/0 (socket says 172.21.15.46:58788) 2024-01-26T23:46:03.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.793+0000 7f9277fff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9278159590 0x7f927815d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.794+0000 7f927ce94700 1 -- 172.21.15.46:0/2411970198 learned_addr learned my addr 172.21.15.46:0/2411970198 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:03.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.794+0000 7f927ce94700 1 -- 172.21.15.46:0/2411970198 >> v2:172.21.15.106:3300/0 conn(0x7f9278159590 msgr2=0x7f927815d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:03.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.794+0000 7f927ce94700 1 --2- 172.21.15.46:0/2411970198 >> v2:172.21.15.106:3300/0 conn(0x7f9278159590 0x7f927815d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.794+0000 7f927ce94700 1 -- 172.21.15.46:0/2411970198 >> v1:172.21.15.106:6789/0 conn(0x7f927815d8f0 legacy=0x7f927806bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:03.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.794+0000 7f927ce94700 1 -- 172.21.15.46:0/2411970198 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927815e070 con 0x7f927806c3c0 2024-01-26T23:46:03.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.794+0000 7f9277fff700 1 --2- 172.21.15.46:0/2411970198 >> v2:172.21.15.106:3300/0 conn(0x7f9278159590 0x7f927815d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:03.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.794+0000 7f9276ffd700 1 -- 172.21.15.46:0/2411970198 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968730662 0 0) 0x7f927815dd90 con 0x7f927815d8f0 2024-01-26T23:46:03.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.794+0000 7f927ce94700 1 --2- 172.21.15.46:0/2411970198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927806c3c0 0x7f9278069370 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f92780c1f60 tx=0x7f9270009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=344926740bd33276 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:03.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.794+0000 7f9276ffd700 1 -- 172.21.15.46:0/2411970198 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927000e070 con 0x7f927806c3c0 2024-01-26T23:46:03.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.794+0000 7f9276ffd700 1 -- 172.21.15.46:0/2411970198 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927000b990 con 0x7f927806c3c0 2024-01-26T23:46:03.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.794+0000 7f9276ffd700 1 -- 172.21.15.46:0/2411970198 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92700092e0 con 0x7f927806c3c0 2024-01-26T23:46:03.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.795+0000 7f927e8f7700 1 -- 172.21.15.46:0/2411970198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927806c3c0 msgr2=0x7f9278069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:03.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.795+0000 7f927e8f7700 1 --2- 172.21.15.46:0/2411970198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927806c3c0 0x7f9278069370 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f92780c1f60 tx=0x7f9270009840 comp rx=0 tx=0).stop 2024-01-26T23:46:03.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.795+0000 7f927e8f7700 1 -- 172.21.15.46:0/2411970198 shutdown_connections 2024-01-26T23:46:03.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.795+0000 7f927e8f7700 1 --2- 172.21.15.46:0/2411970198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927806c3c0 0x7f9278069370 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.795+0000 7f927e8f7700 1 --2- 172.21.15.46:0/2411970198 >> v2:172.21.15.106:3300/0 conn(0x7f9278159590 0x7f927815d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.795+0000 7f927e8f7700 1 -- 172.21.15.46:0/2411970198 >> 172.21.15.46:0/2411970198 conn(0x7f92780ce530 msgr2=0x7f92780ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:03.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.795+0000 7f927e8f7700 1 -- 172.21.15.46:0/2411970198 shutdown_connections 2024-01-26T23:46:03.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.795+0000 7f927e8f7700 1 -- 172.21.15.46:0/2411970198 wait complete. 2024-01-26T23:46:03.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.797+0000 7f927e8f7700 1 Processor -- start 2024-01-26T23:46:03.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.797+0000 7f927e8f7700 1 -- start start 2024-01-26T23:46:03.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.798+0000 7f927e8f7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f927806c3c0 0x7f927815c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.798+0000 7f927e8f7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9278159590 0x7f92780b2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.798+0000 7f927e8f7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f927815d8f0 0x7f92780c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.798+0000 7f927e8f7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92780b2270 con 0x7f9278159590 2024-01-26T23:46:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.798+0000 7f927e8f7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927815cf10 con 0x7f927806c3c0 2024-01-26T23:46:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.798+0000 7f927e8f7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92780b1660 con 0x7f927815d8f0 2024-01-26T23:46:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.798+0000 7f92777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9278159590 0x7f92780b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.798+0000 7f92777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9278159590 0x7f92780b2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58802/0 (socket says 172.21.15.46:58802) 2024-01-26T23:46:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.798+0000 7f92777fe700 1 -- 172.21.15.46:0/2240702335 learned_addr learned my addr 172.21.15.46:0/2240702335 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.798+0000 7f9277fff700 1 --2- 172.21.15.46:0/2240702335 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f927806c3c0 0x7f927815c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.798+0000 7f927ce94700 1 --2- 172.21.15.46:0/2240702335 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f927815d8f0 0x7f92780c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.798+0000 7f92777fe700 1 -- 172.21.15.46:0/2240702335 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f927815d8f0 msgr2=0x7f92780c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.798+0000 7f92777fe700 1 --2- 172.21.15.46:0/2240702335 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f927815d8f0 0x7f92780c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.798+0000 7f92777fe700 1 -- 172.21.15.46:0/2240702335 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f927806c3c0 msgr2=0x7f927815c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.798+0000 7f92777fe700 1 --2- 172.21.15.46:0/2240702335 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f927806c3c0 0x7f927815c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.798+0000 7f92777fe700 1 -- 172.21.15.46:0/2240702335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9268009420 con 0x7f9278159590 2024-01-26T23:46:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.799+0000 7f92777fe700 1 --2- 172.21.15.46:0/2240702335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9278159590 0x7f92780b2700 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f926c00c590 tx=0x7f926c007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.799+0000 7f925ffff700 1 -- 172.21.15.46:0/2240702335 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c011590 con 0x7f9278159590 2024-01-26T23:46:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.799+0000 7f925ffff700 1 -- 172.21.15.46:0/2240702335 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926c011710 con 0x7f9278159590 2024-01-26T23:46:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.799+0000 7f927e8f7700 1 -- 172.21.15.46:0/2240702335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9270009ae0 con 0x7f9278159590 2024-01-26T23:46:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.799+0000 7f925ffff700 1 -- 172.21.15.46:0/2240702335 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c011ad0 con 0x7f9278159590 2024-01-26T23:46:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.799+0000 7f927e8f7700 1 -- 172.21.15.46:0/2240702335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92780b1c20 con 0x7f9278159590 2024-01-26T23:46:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.800+0000 7f925ffff700 1 -- 172.21.15.46:0/2240702335 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f926c01b3b0 con 0x7f9278159590 2024-01-26T23:46:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.801+0000 7f925ffff700 1 --2- 172.21.15.46:0/2240702335 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9264066ea0 0x7f9264069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.801+0000 7f925ffff700 1 -- 172.21.15.46:0/2240702335 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f926c083bb0 con 0x7f9278159590 2024-01-26T23:46:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.801+0000 7f9277fff700 1 --2- 172.21.15.46:0/2240702335 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9264066ea0 0x7f9264069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.802+0000 7f9277fff700 1 --2- 172.21.15.46:0/2240702335 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9264066ea0 0x7f9264069350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9268009f40 tx=0x7f92680091a0 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:03.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.803+0000 7f9287bfab80 1 Processor -- start 2024-01-26T23:46:03.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.803+0000 7f9287bfab80 1 -- start start 2024-01-26T23:46:03.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.804+0000 7f9287bfab80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e53bf010 0x5572e53bf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.804+0000 7f9287bfab80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e5407360 0x5572e53e9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.804+0000 7f9287bfab80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53be5d0 0x5572e53e9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.804+0000 7f9287bfab80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e52c29f0 con 0x5572e53be5d0 2024-01-26T23:46:03.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.804+0000 7f9287bfab80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e5314130 con 0x5572e5407360 2024-01-26T23:46:03.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.804+0000 7f9287bfab80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572e5311930 con 0x5572e53bf010 2024-01-26T23:46:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.804+0000 7f927ce94700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53be5d0 0x5572e53e9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.804+0000 7f927ce94700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53be5d0 0x5572e53e9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58818/0 (socket says 172.21.15.46:58818) 2024-01-26T23:46:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.804+0000 7f927ce94700 1 -- 172.21.15.46:0/3654333704 learned_addr learned my addr 172.21.15.46:0/3654333704 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.804+0000 7f92777fe700 1 --2- 172.21.15.46:0/3654333704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e5407360 0x5572e53e9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.804+0000 7f9277fff700 1 --2- 172.21.15.46:0/3654333704 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e53bf010 0x5572e53bf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.804+0000 7f92777fe700 1 -- 172.21.15.46:0/3654333704 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e53bf010 msgr2=0x5572e53bf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.804+0000 7f92777fe700 1 --2- 172.21.15.46:0/3654333704 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e53bf010 0x5572e53bf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.804+0000 7f92777fe700 1 -- 172.21.15.46:0/3654333704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53be5d0 msgr2=0x5572e53e9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.804+0000 7f92777fe700 1 --2- 172.21.15.46:0/3654333704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53be5d0 0x5572e53e9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.804+0000 7f92777fe700 1 -- 172.21.15.46:0/3654333704 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572e514cb10 con 0x5572e5407360 2024-01-26T23:46:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.805+0000 7f927ce94700 1 --2- 172.21.15.46:0/3654333704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53be5d0 0x5572e53e9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.805+0000 7f92777fe700 1 --2- 172.21.15.46:0/3654333704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e5407360 0x5572e53e9840 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f926c081e60 tx=0x7f926c001210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5493cc681487fbc server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.805+0000 7f925d7fa700 1 -- 172.21.15.46:0/3654333704 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c054ff0 con 0x5572e5407360 2024-01-26T23:46:03.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.805+0000 7f925d7fa700 1 -- 172.21.15.46:0/3654333704 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926c068070 con 0x5572e5407360 2024-01-26T23:46:03.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.805+0000 7f925d7fa700 1 -- 172.21.15.46:0/3654333704 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c0548f0 con 0x5572e5407360 2024-01-26T23:46:03.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.806+0000 7f9287bfab80 1 -- 172.21.15.46:0/3654333704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e5407360 msgr2=0x5572e53e9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:03.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.806+0000 7f9287bfab80 1 --2- 172.21.15.46:0/3654333704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e5407360 0x5572e53e9840 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f926c081e60 tx=0x7f926c001210 comp rx=0 tx=0).stop 2024-01-26T23:46:03.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.806+0000 7f9287bfab80 1 -- 172.21.15.46:0/3654333704 shutdown_connections 2024-01-26T23:46:03.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.806+0000 7f9287bfab80 1 --2- 172.21.15.46:0/3654333704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53be5d0 0x5572e53e9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.806+0000 7f9287bfab80 1 --2- 172.21.15.46:0/3654333704 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e53bf010 0x5572e53bf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.806+0000 7f9287bfab80 1 --2- 172.21.15.46:0/3654333704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e5407360 0x5572e53e9840 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.806+0000 7f9287bfab80 1 -- 172.21.15.46:0/3654333704 >> 172.21.15.46:0/3654333704 conn(0x5572e53bea70 msgr2=0x5572e5407820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:03.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.806+0000 7f9287bfab80 1 -- 172.21.15.46:0/3654333704 shutdown_connections 2024-01-26T23:46:03.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.806+0000 7f9287bfab80 1 -- 172.21.15.46:0/3654333704 wait complete. 2024-01-26T23:46:03.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.806+0000 7f9287bfab80 1 -- start start 2024-01-26T23:46:03.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.806+0000 7f9287bfab80 1 Processor -- start 2024-01-26T23:46:03.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.807+0000 7f9287bfab80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e53be5d0 0x5572e54135c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.807+0000 7f9287bfab80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53bf010 0x5572e5413b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.807+0000 7f9287bfab80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e5407360 0x5572e53b8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.807+0000 7f9287bfab80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e530d720 con 0x5572e53bf010 2024-01-26T23:46:03.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.807+0000 7f9287bfab80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e52b91b0 con 0x5572e5407360 2024-01-26T23:46:03.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.807+0000 7f9287bfab80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572e52b8e50 con 0x5572e53be5d0 2024-01-26T23:46:03.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.807+0000 7f9287bfab80 10 client.0 _ll_register_callbacks cb 0x5572e511b7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:46:03.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.807+0000 7f92777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53bf010 0x5572e5413b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.807+0000 7f92777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53bf010 0x5572e5413b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58830/0 (socket says 172.21.15.46:58830) 2024-01-26T23:46:03.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.808+0000 7f92777fe700 1 -- 172.21.15.46:0/3189033302 learned_addr learned my addr 172.21.15.46:0/3189033302 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:03.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.808+0000 7f9277fff700 1 --2- 172.21.15.46:0/3189033302 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e53be5d0 0x5572e54135c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.808+0000 7f927ce94700 1 --2- 172.21.15.46:0/3189033302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e5407360 0x5572e53b8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.808+0000 7f92777fe700 1 -- 172.21.15.46:0/3189033302 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e53be5d0 msgr2=0x5572e54135c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:03.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.808+0000 7f92777fe700 1 --2- 172.21.15.46:0/3189033302 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e53be5d0 0x5572e54135c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.808+0000 7f92777fe700 1 -- 172.21.15.46:0/3189033302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e5407360 msgr2=0x5572e53b8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:03.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.808+0000 7f92777fe700 1 --2- 172.21.15.46:0/3189033302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e5407360 0x5572e53b8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.808+0000 7f92777fe700 1 -- 172.21.15.46:0/3189033302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927815e070 con 0x5572e53bf010 2024-01-26T23:46:03.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.808+0000 7f92777fe700 1 --2- 172.21.15.46:0/3189033302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53bf010 0x5572e5413b00 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f926c0545f0 tx=0x7f926c003000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff3f5a2b33782b08 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:03.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.808+0000 7f92567fc700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:46:03.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.808+0000 7f92567fc700 1 -- 172.21.15.46:0/3189033302 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c05b890 con 0x5572e53bf010 2024-01-26T23:46:03.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.808+0000 7f9287bfab80 1 -- 172.21.15.46:0/3189033302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5572e53b2190 con 0x5572e53bf010 2024-01-26T23:46:03.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.808+0000 7f92567fc700 1 -- 172.21.15.46:0/3189033302 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f926c0548f0 con 0x5572e53bf010 2024-01-26T23:46:03.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.810+0000 7f92567fc700 1 -- 172.21.15.46:0/3189033302 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7f926c053ff0 con 0x5572e53bf010 2024-01-26T23:46:03.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.810+0000 7f9287bfab80 10 client.6984 fetch_fsmap learned FSMap version 45 2024-01-26T23:46:03.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.810+0000 7f9287bfab80 1 -- 172.21.15.46:0/3189033302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) v3 -- 0x7f92780b1c20 con 0x5572e53bf010 2024-01-26T23:46:03.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.811+0000 7f92567fc700 1 -- 172.21.15.46:0/3189033302 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5572e53b2190 con 0x5572e53bf010 2024-01-26T23:46:03.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.811+0000 7f92567fc700 1 -- 172.21.15.46:0/3189033302 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 45) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f926c05b400 con 0x5572e53bf010 2024-01-26T23:46:03.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.811+0000 7f9287bfab80 10 client.6984 fetch_fsmap finished waiting for FSMap version 45 2024-01-26T23:46:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.811+0000 7f9287bfab80 10 client.6984 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-26T23:46:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.811+0000 7f9287bfab80 20 client.6984 populate_metadata read hostname 'smithi046' 2024-01-26T23:46:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.811+0000 7f9287bfab80 1 --2- 172.21.15.46:0/3189033302 >> [v2:172.21.15.46:6833/3176096333,v1:172.21.15.46:6835/3176096333] conn(0x5572e54067b0 0x5572e5406b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.811+0000 7f9287bfab80 4 client.6984 mds_command: new command op to 6631 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-26T23:46:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.811+0000 7f9287bfab80 1 -- 172.21.15.46:0/3189033302 --> [v2:172.21.15.46:6833/3176096333,v1:172.21.15.46:6835/3176096333] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5572e52afee0 con 0x5572e54067b0 2024-01-26T23:46:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.811+0000 7f9277fff700 1 --2- 172.21.15.46:0/3189033302 >> [v2:172.21.15.46:6833/3176096333,v1:172.21.15.46:6835/3176096333] conn(0x5572e54067b0 0x5572e5406b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.811+0000 7f92567fc700 10 client.6984 ms_handle_connect on v2:172.21.15.46:6833/3176096333 2024-01-26T23:46:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.811+0000 7f9277fff700 1 --2- 172.21.15.46:0/3189033302 >> [v2:172.21.15.46:6833/3176096333,v1:172.21.15.46:6835/3176096333] conn(0x5572e54067b0 0x5572e5406b80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ac20824a641b735 server_cookie=21d5f018c1b49696 in_seq=0 out_seq=0 2024-01-26T23:46:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.812+0000 7f92567fc700 1 -- 172.21.15.46:0/3189033302 <== mds.0 v2:172.21.15.46:6833/3176096333 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x5572e52afee0 con 0x5572e54067b0 2024-01-26T23:46:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.812+0000 7f92567fc700 10 client.6984 handle_command_reply: tid=0 2024-01-26T23:46:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.812+0000 7f9287bfab80 1 client.6984 shutdown 2024-01-26T23:46:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.812+0000 7f9287bfab80 1 -- 172.21.15.46:0/3189033302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53bf010 msgr2=0x5572e5413b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.812+0000 7f9287bfab80 1 --2- 172.21.15.46:0/3189033302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53bf010 0x5572e5413b00 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f926c0545f0 tx=0x7f926c003000 comp rx=0 tx=0).stop 2024-01-26T23:46:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.812+0000 7f9287bfab80 1 -- 172.21.15.46:0/3189033302 shutdown_connections 2024-01-26T23:46:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.812+0000 7f9287bfab80 1 --2- 172.21.15.46:0/3189033302 >> [v2:172.21.15.46:6833/3176096333,v1:172.21.15.46:6835/3176096333] conn(0x5572e54067b0 0x5572e5406b80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.812+0000 7f9287bfab80 1 --2- 172.21.15.46:0/3189033302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e5407360 0x5572e53b8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.812+0000 7f9287bfab80 1 --2- 172.21.15.46:0/3189033302 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e53be5d0 0x5572e54135c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.812+0000 7f9287bfab80 1 --2- 172.21.15.46:0/3189033302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53bf010 0x5572e5413b00 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.812+0000 7f92567fc700 0 client.6984 ms_handle_reset on v2:172.21.15.46:6833/3176096333 2024-01-26T23:46:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.812+0000 7f9287bfab80 1 -- 172.21.15.46:0/3189033302 >> 172.21.15.46:0/3189033302 conn(0x5572e53bea70 msgr2=0x5572e53b9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.812+0000 7f9287bfab80 1 -- 172.21.15.46:0/3189033302 shutdown_connections 2024-01-26T23:46:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.812+0000 7f9287bfab80 1 -- 172.21.15.46:0/3189033302 wait complete. 2024-01-26T23:46:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.812+0000 7f9287bfab80 20 client.6984 trim_cache size 0 max 16384 2024-01-26T23:46:03.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.831+0000 7f9287bfab80 1 Processor -- start 2024-01-26T23:46:03.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.831+0000 7f9287bfab80 1 -- start start 2024-01-26T23:46:03.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f9287bfab80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e53be5d0 0x5572e53d68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f9287bfab80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53bf010 0x5572e53e9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f9287bfab80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e5407360 0x5572e53e88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f9287bfab80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e5205e40 con 0x5572e53bf010 2024-01-26T23:46:03.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f9287bfab80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e53e8600 con 0x5572e53be5d0 2024-01-26T23:46:03.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f9287bfab80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572e53ec990 con 0x5572e5407360 2024-01-26T23:46:03.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f92777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53bf010 0x5572e53e9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f92777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53bf010 0x5572e53e9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58842/0 (socket says 172.21.15.46:58842) 2024-01-26T23:46:03.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f92777fe700 1 -- 172.21.15.46:0/2393947713 learned_addr learned my addr 172.21.15.46:0/2393947713 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:03.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f9277fff700 1 --2- 172.21.15.46:0/2393947713 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e53be5d0 0x5572e53d68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f927ce94700 1 --2- 172.21.15.46:0/2393947713 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e5407360 0x5572e53e88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f92777fe700 1 -- 172.21.15.46:0/2393947713 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e5407360 msgr2=0x5572e53e88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:03.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f92777fe700 1 --2- 172.21.15.46:0/2393947713 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e5407360 0x5572e53e88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f92777fe700 1 -- 172.21.15.46:0/2393947713 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e53be5d0 msgr2=0x5572e53d68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:03.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f92777fe700 1 --2- 172.21.15.46:0/2393947713 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e53be5d0 0x5572e53d68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f92777fe700 1 -- 172.21.15.46:0/2393947713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572e52db290 con 0x5572e53bf010 2024-01-26T23:46:03.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f92777fe700 1 --2- 172.21.15.46:0/2393947713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53bf010 0x5572e53e9c90 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f926c00c560 tx=0x7f926c07e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06edc515acc784a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:03.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f925dffb700 1 -- 172.21.15.46:0/2393947713 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c05b890 con 0x5572e53bf010 2024-01-26T23:46:03.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f925dffb700 1 -- 172.21.15.46:0/2393947713 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926c0548f0 con 0x5572e53bf010 2024-01-26T23:46:03.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.832+0000 7f925dffb700 1 -- 172.21.15.46:0/2393947713 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c057b40 con 0x5572e53bf010 2024-01-26T23:46:03.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.833+0000 7f9287bfab80 1 -- 172.21.15.46:0/2393947713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53bf010 msgr2=0x5572e53e9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:03.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.833+0000 7f9287bfab80 1 --2- 172.21.15.46:0/2393947713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53bf010 0x5572e53e9c90 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f926c00c560 tx=0x7f926c07e070 comp rx=0 tx=0).stop 2024-01-26T23:46:03.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.833+0000 7f9287bfab80 1 -- 172.21.15.46:0/2393947713 shutdown_connections 2024-01-26T23:46:03.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.833+0000 7f9287bfab80 1 --2- 172.21.15.46:0/2393947713 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e5407360 0x5572e53e88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.833+0000 7f9287bfab80 1 --2- 172.21.15.46:0/2393947713 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e53be5d0 0x5572e53d68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.833+0000 7f9287bfab80 1 --2- 172.21.15.46:0/2393947713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e53bf010 0x5572e53e9c90 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.833+0000 7f9287bfab80 1 -- 172.21.15.46:0/2393947713 >> 172.21.15.46:0/2393947713 conn(0x5572e53bea70 msgr2=0x5572e5406b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:03.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.833+0000 7f9287bfab80 1 -- 172.21.15.46:0/2393947713 shutdown_connections 2024-01-26T23:46:03.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.833+0000 7f9287bfab80 1 -- 172.21.15.46:0/2393947713 wait complete. 2024-01-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.834+0000 7f9287bfab80 1 -- start start 2024-01-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.834+0000 7f9287bfab80 1 Processor -- start 2024-01-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.835+0000 7f9287bfab80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e53be5d0 0x5572e5467f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.835+0000 7f9287bfab80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e53bf010 0x5572e53e8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.835+0000 7f9287bfab80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e5407360 0x5572e53e8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.835+0000 7f9287bfab80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e5468460 con 0x5572e5407360 2024-01-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.835+0000 7f9287bfab80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e53e69a0 con 0x5572e53be5d0 2024-01-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.835+0000 7f9287bfab80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572e53b9790 con 0x5572e53bf010 2024-01-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.835+0000 7f9287bfab80 10 client.0 _ll_register_callbacks cb 0x5572e511b7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.835+0000 7f927ce94700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e5407360 0x5572e53e8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.835+0000 7f927ce94700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e5407360 0x5572e53e8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58844/0 (socket says 172.21.15.46:58844) 2024-01-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.835+0000 7f927ce94700 1 -- 172.21.15.46:0/1428824177 learned_addr learned my addr 172.21.15.46:0/1428824177 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.835+0000 7f9277fff700 1 --2- 172.21.15.46:0/1428824177 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e53be5d0 0x5572e5467f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.835+0000 7f92777fe700 1 --2- 172.21.15.46:0/1428824177 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e53bf010 0x5572e53e8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.835+0000 7f927ce94700 1 -- 172.21.15.46:0/1428824177 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e53bf010 msgr2=0x5572e53e8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.835+0000 7f927ce94700 1 --2- 172.21.15.46:0/1428824177 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e53bf010 0x5572e53e8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.835+0000 7f927ce94700 1 -- 172.21.15.46:0/1428824177 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e53be5d0 msgr2=0x5572e5467f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.835+0000 7f927ce94700 1 --2- 172.21.15.46:0/1428824177 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e53be5d0 0x5572e5467f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.835+0000 7f927ce94700 1 -- 172.21.15.46:0/1428824177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92780b1c20 con 0x5572e5407360 2024-01-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.836+0000 7f927ce94700 1 --2- 172.21.15.46:0/1428824177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e5407360 0x5572e53e8db0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f927806f390 tx=0x7f9270018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=300d5f5c3baa592e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.836+0000 7f92567fc700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:46:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.836+0000 7f92567fc700 1 -- 172.21.15.46:0/1428824177 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9270029410 con 0x5572e5407360 2024-01-26T23:46:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.836+0000 7f9287bfab80 1 -- 172.21.15.46:0/1428824177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f927815a4d0 con 0x5572e5407360 2024-01-26T23:46:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.836+0000 7f92567fc700 1 -- 172.21.15.46:0/1428824177 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f9270029570 con 0x5572e5407360 2024-01-26T23:46:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.836+0000 7f92567fc700 1 -- 172.21.15.46:0/1428824177 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9270029940 con 0x5572e5407360 2024-01-26T23:46:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.836+0000 7f9287bfab80 10 client.6990 fetch_fsmap learned FSMap version 45 2024-01-26T23:46:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.836+0000 7f9287bfab80 1 -- 172.21.15.46:0/1428824177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) v3 -- 0x5572e51a4c80 con 0x5572e5407360 2024-01-26T23:46:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.836+0000 7f92567fc700 1 -- 172.21.15.46:0/1428824177 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927815a4d0 con 0x5572e5407360 2024-01-26T23:46:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.836+0000 7f92567fc700 1 -- 172.21.15.46:0/1428824177 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 45) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f927000b050 con 0x5572e5407360 2024-01-26T23:46:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.836+0000 7f9287bfab80 10 client.6990 fetch_fsmap finished waiting for FSMap version 45 2024-01-26T23:46:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.836+0000 7f9287bfab80 10 client.6990 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-26T23:46:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.836+0000 7f9287bfab80 20 client.6990 populate_metadata read hostname 'smithi046' 2024-01-26T23:46:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.836+0000 7f9287bfab80 1 --2- 172.21.15.46:0/1428824177 >> [v2:172.21.15.46:6833/3176096333,v1:172.21.15.46:6835/3176096333] conn(0x5572e53e6df0 0x5572e53e71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.836+0000 7f9287bfab80 4 client.6990 mds_command: new command op to 6631 tid=0[{"prefix": "status"}] 2024-01-26T23:46:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.836+0000 7f9287bfab80 1 -- 172.21.15.46:0/1428824177 --> [v2:172.21.15.46:6833/3176096333,v1:172.21.15.46:6835/3176096333] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f926c068070 con 0x5572e53e6df0 2024-01-26T23:46:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.836+0000 7f9277fff700 1 --2- 172.21.15.46:0/1428824177 >> [v2:172.21.15.46:6833/3176096333,v1:172.21.15.46:6835/3176096333] conn(0x5572e53e6df0 0x5572e53e71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.837+0000 7f9277fff700 1 --2- 172.21.15.46:0/1428824177 >> [v2:172.21.15.46:6833/3176096333,v1:172.21.15.46:6835/3176096333] conn(0x5572e53e6df0 0x5572e53e71c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c929edae9ac25ec6 server_cookie=b77529397bf9207e in_seq=0 out_seq=0 2024-01-26T23:46:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.837+0000 7f92567fc700 10 client.6990 ms_handle_connect on v2:172.21.15.46:6833/3176096333 2024-01-26T23:46:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.837+0000 7f92567fc700 1 -- 172.21.15.46:0/1428824177 <== mds.0 v2:172.21.15.46:6833/3176096333 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+318 (crc 0 0 0) 0x7f926c068070 con 0x5572e53e6df0 2024-01-26T23:46:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.837+0000 7f92567fc700 10 client.6990 handle_command_reply: tid=0 2024-01-26T23:46:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.837+0000 7f9287bfab80 1 client.6990 shutdown 2024-01-26T23:46:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.837+0000 7f9287bfab80 1 -- 172.21.15.46:0/1428824177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e5407360 msgr2=0x5572e53e8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.837+0000 7f9287bfab80 1 --2- 172.21.15.46:0/1428824177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e5407360 0x5572e53e8db0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f927806f390 tx=0x7f9270018000 comp rx=0 tx=0).stop 2024-01-26T23:46:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.837+0000 7f9287bfab80 1 -- 172.21.15.46:0/1428824177 shutdown_connections 2024-01-26T23:46:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.837+0000 7f9287bfab80 1 --2- 172.21.15.46:0/1428824177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572e5407360 0x5572e53e8db0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.837+0000 7f9287bfab80 1 --2- 172.21.15.46:0/1428824177 >> [v2:172.21.15.46:6833/3176096333,v1:172.21.15.46:6835/3176096333] conn(0x5572e53e6df0 0x5572e53e71c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.837+0000 7f9287bfab80 1 --2- 172.21.15.46:0/1428824177 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5572e53be5d0 0x5572e5467f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.837+0000 7f9287bfab80 1 --2- 172.21.15.46:0/1428824177 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5572e53bf010 0x5572e53e8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.837+0000 7f92567fc700 0 client.6990 ms_handle_reset on v2:172.21.15.46:6833/3176096333 2024-01-26T23:46:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.837+0000 7f9287bfab80 1 -- 172.21.15.46:0/1428824177 >> 172.21.15.46:0/1428824177 conn(0x5572e53bea70 msgr2=0x5572e5406b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.837+0000 7f9287bfab80 1 -- 172.21.15.46:0/1428824177 shutdown_connections 2024-01-26T23:46:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.837+0000 7f9287bfab80 1 -- 172.21.15.46:0/1428824177 wait complete. 2024-01-26T23:46:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.837+0000 7f9287bfab80 20 client.6990 trim_cache size 0 max 16384 2024-01-26T23:46:03.844 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:46:03.844 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-26T23:46:03.844 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-26T23:46:03.844 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 6631, 2024-01-26T23:46:03.845 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-26T23:46:03.845 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-26T23:46:03.845 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-01-26T23:46:03.845 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 10.84618612, 2024-01-26T23:46:03.845 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 45, 2024-01-26T23:46:03.845 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 98, 2024-01-26T23:46:03.845 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 98, 2024-01-26T23:46:03.845 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 15.854820087 2024-01-26T23:46:03.845 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:46:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.838+0000 7f927e8f7700 1 -- 172.21.15.46:0/2240702335 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9264066ea0 msgr2=0x7f9264069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.838+0000 7f927e8f7700 1 --2- 172.21.15.46:0/2240702335 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9264066ea0 0x7f9264069350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9268009f40 tx=0x7f92680091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.838+0000 7f927e8f7700 1 -- 172.21.15.46:0/2240702335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9278159590 msgr2=0x7f92780b2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.838+0000 7f927e8f7700 1 --2- 172.21.15.46:0/2240702335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9278159590 0x7f92780b2700 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f926c00c590 tx=0x7f926c007f50 comp rx=0 tx=0).stop 2024-01-26T23:46:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.838+0000 7f927e8f7700 1 -- 172.21.15.46:0/2240702335 shutdown_connections 2024-01-26T23:46:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.838+0000 7f927e8f7700 1 --2- 172.21.15.46:0/2240702335 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9264066ea0 0x7f9264069350 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.838+0000 7f927e8f7700 1 --2- 172.21.15.46:0/2240702335 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f927815d8f0 0x7f92780c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.838+0000 7f927e8f7700 1 --2- 172.21.15.46:0/2240702335 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f927806c3c0 0x7f927815c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.838+0000 7f927e8f7700 1 --2- 172.21.15.46:0/2240702335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9278159590 0x7f92780b2700 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.838+0000 7f927e8f7700 1 -- 172.21.15.46:0/2240702335 >> 172.21.15.46:0/2240702335 conn(0x7f92780ce530 msgr2=0x7f92780cb830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.838+0000 7f927e8f7700 1 -- 172.21.15.46:0/2240702335 shutdown_connections 2024-01-26T23:46:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.838+0000 7f927e8f7700 1 -- 172.21.15.46:0/2240702335 wait complete. 2024-01-26T23:46:03.852 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-26T23:46:03.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.987+0000 7f64bcce3700 1 Processor -- start 2024-01-26T23:46:03.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.987+0000 7f64bcce3700 1 -- start start 2024-01-26T23:46:03.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.988+0000 7f64bcce3700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f64b8157340 0x7f64b815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.988+0000 7f64bcce3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f64b80ae520 0x7f64b80b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.988+0000 7f64bcce3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64b80b3fb0 0x7f64b8157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.988+0000 7f64bcce3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64b80b4410 con 0x7f64b80ae520 2024-01-26T23:46:03.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.988+0000 7f64bcce3700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64b8158ae0 con 0x7f64b8157340 2024-01-26T23:46:03.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.988+0000 7f64bcce3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b815b650 con 0x7f64b80b3fb0 2024-01-26T23:46:03.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.988+0000 7f64b5d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f64b80ae520 0x7f64b80b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.988+0000 7f64b6d9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64b80b3fb0 0x7f64b8157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.988+0000 7f64b659c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f64b8157340 0x7f64b815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.988+0000 7f64b6d9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64b80b3fb0 0x7f64b8157710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:53234/0 (socket says 172.21.15.46:53234) 2024-01-26T23:46:03.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.988+0000 7f64b5d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f64b80ae520 0x7f64b80b3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58852/0 (socket says 172.21.15.46:58852) 2024-01-26T23:46:03.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.988+0000 7f64b6d9d700 1 -- 172.21.15.46:0/1498075138 learned_addr learned my addr 172.21.15.46:0/1498075138 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:03.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.988+0000 7f64b659c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f64b8157340 0x7f64b815bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56362/0 (socket says 172.21.15.46:56362) 2024-01-26T23:46:03.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.989+0000 7f64b659c700 1 -- 172.21.15.46:0/1498075138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64b80b3fb0 msgr2=0x7f64b8157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:03.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.989+0000 7f64b659c700 1 --2- 172.21.15.46:0/1498075138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64b80b3fb0 0x7f64b8157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.989+0000 7f64b659c700 1 -- 172.21.15.46:0/1498075138 >> v2:172.21.15.46:3300/0 conn(0x7f64b80ae520 msgr2=0x7f64b80b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:03.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.989+0000 7f64b659c700 1 --2- 172.21.15.46:0/1498075138 >> v2:172.21.15.46:3300/0 conn(0x7f64b80ae520 0x7f64b80b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.989+0000 7f64b659c700 1 -- 172.21.15.46:0/1498075138 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b815c020 con 0x7f64b8157340 2024-01-26T23:46:03.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.989+0000 7f64b5d9b700 1 --2- 172.21.15.46:0/1498075138 >> v2:172.21.15.46:3300/0 conn(0x7f64b80ae520 0x7f64b80b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:03.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.989+0000 7f64b6d9d700 1 --2- 172.21.15.46:0/1498075138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64b80b3fb0 0x7f64b8157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:03.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.989+0000 7f64b659c700 1 --2- 172.21.15.46:0/1498075138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64b8157340 0x7f64b815bae0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f64b806f980 tx=0x7f64a80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99525699742187a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:03.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.989+0000 7f64b559a700 1 -- 172.21.15.46:0/1498075138 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a8011630 con 0x7f64b8157340 2024-01-26T23:46:03.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.989+0000 7f64b559a700 1 -- 172.21.15.46:0/1498075138 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a80117b0 con 0x7f64b8157340 2024-01-26T23:46:03.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.989+0000 7f64b559a700 1 -- 172.21.15.46:0/1498075138 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a8011b70 con 0x7f64b8157340 2024-01-26T23:46:03.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.990+0000 7f64bcce3700 1 -- 172.21.15.46:0/1498075138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64b8157340 msgr2=0x7f64b815bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:03.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.990+0000 7f64bcce3700 1 --2- 172.21.15.46:0/1498075138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64b8157340 0x7f64b815bae0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f64b806f980 tx=0x7f64a80091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.990+0000 7f64bcce3700 1 -- 172.21.15.46:0/1498075138 shutdown_connections 2024-01-26T23:46:03.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.990+0000 7f64bcce3700 1 --2- 172.21.15.46:0/1498075138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64b80b3fb0 0x7f64b8157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.990+0000 7f64bcce3700 1 --2- 172.21.15.46:0/1498075138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64b8157340 0x7f64b815bae0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.990+0000 7f64bcce3700 1 --2- 172.21.15.46:0/1498075138 >> v2:172.21.15.46:3300/0 conn(0x7f64b80ae520 0x7f64b80b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.990+0000 7f64bcce3700 1 -- 172.21.15.46:0/1498075138 >> 172.21.15.46:0/1498075138 conn(0x7f64b80ce530 msgr2=0x7f64b80ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:03.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.990+0000 7f64bcce3700 1 -- 172.21.15.46:0/1498075138 shutdown_connections 2024-01-26T23:46:03.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.990+0000 7f64bcce3700 1 -- 172.21.15.46:0/1498075138 wait complete. 2024-01-26T23:46:03.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.991+0000 7f64bcce3700 1 Processor -- start 2024-01-26T23:46:03.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.991+0000 7f64bcce3700 1 -- start start 2024-01-26T23:46:03.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.992+0000 7f64bcce3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64b80ae520 0x7f64b80b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.992+0000 7f64bcce3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64b80b3fb0 0x7f64b80b3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.992+0000 7f64bcce3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64b8157340 0x7f64b80c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:03.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.992+0000 7f64bcce3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b8069230 con 0x7f64b80ae520 2024-01-26T23:46:03.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.992+0000 7f64bcce3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b816deb0 con 0x7f64b8157340 2024-01-26T23:46:03.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.992+0000 7f64bcce3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b8068be0 con 0x7f64b80b3fb0 2024-01-26T23:46:03.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.992+0000 7f64b659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64b80ae520 0x7f64b80b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.992+0000 7f64b5d9b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64b80b3fb0 0x7f64b80b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.992+0000 7f64b659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64b80ae520 0x7f64b80b1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58864/0 (socket says 172.21.15.46:58864) 2024-01-26T23:46:03.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.992+0000 7f64b659c700 1 -- 172.21.15.46:0/3967850899 learned_addr learned my addr 172.21.15.46:0/3967850899 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:03.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.992+0000 7f64b6d9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64b8157340 0x7f64b80c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:03.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.992+0000 7f64b5d9b700 1 -- 172.21.15.46:0/3967850899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64b8157340 msgr2=0x7f64b80c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:03.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.992+0000 7f64b5d9b700 1 --2- 172.21.15.46:0/3967850899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64b8157340 0x7f64b80c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.992+0000 7f64b5d9b700 1 -- 172.21.15.46:0/3967850899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64b80ae520 msgr2=0x7f64b80b1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:03.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.992+0000 7f64b5d9b700 1 --2- 172.21.15.46:0/3967850899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64b80ae520 0x7f64b80b1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:03.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.992+0000 7f64b5d9b700 1 -- 172.21.15.46:0/3967850899 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ac009420 con 0x7f64b80b3fb0 2024-01-26T23:46:03.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.993+0000 7f64b659c700 1 --2- 172.21.15.46:0/3967850899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64b80ae520 0x7f64b80b1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:03.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.993+0000 7f64b5d9b700 1 --2- 172.21.15.46:0/3967850899 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64b80b3fb0 0x7f64b80b3700 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00ea00 tx=0x7f64ac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:03.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.993+0000 7f64a67fc700 1 -- 172.21.15.46:0/3967850899 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac00c6a0 con 0x7f64b80b3fb0 2024-01-26T23:46:03.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.993+0000 7f64a67fc700 1 -- 172.21.15.46:0/3967850899 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ac00c800 con 0x7f64b80b3fb0 2024-01-26T23:46:03.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.993+0000 7f64bcce3700 1 -- 172.21.15.46:0/3967850899 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b0009ae0 con 0x7f64b80b3fb0 2024-01-26T23:46:03.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.993+0000 7f64a67fc700 1 -- 172.21.15.46:0/3967850899 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac00cba0 con 0x7f64b80b3fb0 2024-01-26T23:46:04.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.993+0000 7f64bcce3700 1 -- 172.21.15.46:0/3967850899 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a8009420 con 0x7f64b80b3fb0 2024-01-26T23:46:04.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.994+0000 7f64bcce3700 1 -- 172.21.15.46:0/3967850899 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b804c710 con 0x7f64b80b3fb0 2024-01-26T23:46:04.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.994+0000 7f64a67fc700 1 -- 172.21.15.46:0/3967850899 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f64ac013470 con 0x7f64b80b3fb0 2024-01-26T23:46:04.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.995+0000 7f64a67fc700 1 --2- 172.21.15.46:0/3967850899 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6498066ea0 0x7f6498069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:04.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.995+0000 7f64a67fc700 1 -- 172.21.15.46:0/3967850899 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f64ac084850 con 0x7f64b80b3fb0 2024-01-26T23:46:04.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.995+0000 7f64b659c700 1 --2- 172.21.15.46:0/3967850899 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6498066ea0 0x7f6498069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:04.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.995+0000 7f64b659c700 1 --2- 172.21.15.46:0/3967850899 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6498066ea0 0x7f6498069350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f64a80093c0 tx=0x7f64a8017000 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:04.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:03.999+0000 7f64a67fc700 1 -- 172.21.15.46:0/3967850899 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f64ac0518d0 con 0x7f64b80b3fb0 2024-01-26T23:46:04.092 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:04.091+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:46:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:04.257+0000 7f64bcce3700 1 -- 172.21.15.46:0/3967850899 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f64b8158520 con 0x7f64b80b3fb0 2024-01-26T23:46:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:04.258+0000 7f64a67fc700 1 -- 172.21.15.46:0/3967850899 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v45) v1 ==== 53+0+73 (secure 0 0 0) 0x7f64ac054ef0 con 0x7f64b80b3fb0 2024-01-26T23:46:04.258 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-26T23:46:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:04.260+0000 7f64bcce3700 1 -- 172.21.15.46:0/3967850899 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6498066ea0 msgr2=0x7f6498069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:04.260+0000 7f64bcce3700 1 --2- 172.21.15.46:0/3967850899 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6498066ea0 0x7f6498069350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f64a80093c0 tx=0x7f64a8017000 comp rx=0 tx=0).stop 2024-01-26T23:46:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:04.260+0000 7f64bcce3700 1 -- 172.21.15.46:0/3967850899 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64b80b3fb0 msgr2=0x7f64b80b3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:04.260+0000 7f64bcce3700 1 --2- 172.21.15.46:0/3967850899 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64b80b3fb0 0x7f64b80b3700 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00ea00 tx=0x7f64ac0091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:04.260+0000 7f64bcce3700 1 -- 172.21.15.46:0/3967850899 shutdown_connections 2024-01-26T23:46:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:04.260+0000 7f64bcce3700 1 --2- 172.21.15.46:0/3967850899 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6498066ea0 0x7f6498069350 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:04.260+0000 7f64bcce3700 1 --2- 172.21.15.46:0/3967850899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64b80ae520 0x7f64b80b1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:04.260+0000 7f64bcce3700 1 --2- 172.21.15.46:0/3967850899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64b8157340 0x7f64b80c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:04.260+0000 7f64bcce3700 1 --2- 172.21.15.46:0/3967850899 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64b80b3fb0 0x7f64b80b3700 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:04.260+0000 7f64bcce3700 1 -- 172.21.15.46:0/3967850899 >> 172.21.15.46:0/3967850899 conn(0x7f64b80ce530 msgr2=0x7f64b8155a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:04.260+0000 7f64bcce3700 1 -- 172.21.15.46:0/3967850899 shutdown_connections 2024-01-26T23:46:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:04.260+0000 7f64bcce3700 1 -- 172.21.15.46:0/3967850899 wait complete. 2024-01-26T23:46:04.274 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-26T23:46:04.274 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-26T23:46:04.274 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-26T23:46:04.274 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-26T23:46:04.274 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-26T23:46:04.274 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-26T23:46:04.274 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-26T23:46:04.274 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-26T23:46:04.274 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:46:04.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:46:04.275 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-26T23:46:04.294 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-26T23:46:04.295 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-26T23:46:04.295 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-26T23:46:04.295 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:46:04.295 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-26T23:46:04.295 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:46:04.295 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:46:04.295 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:46:04.295 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-26T23:46:04.295 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:46:04.295 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:46:04.295 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-26T23:46:04.295 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-26T23:46:04.296 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:46:04.296 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-26T23:46:04.296 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-26T23:46:04.296 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 41366sec preferred_lft 41366sec 2024-01-26T23:46:04.296 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-26T23:46:04.296 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:46:04.296 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:46:04.296 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:46:04.296 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-26T23:46:04.296 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-26T23:46:04.296 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:46:04.296 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-26T23:46:04.296 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:46:04.296 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::cce6:a7ff:fe4f:38d/64 scope link 2024-01-26T23:46:04.297 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:46:04.297 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:46:04.297 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 76:ef:90:4c:9d:6f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-26T23:46:04.297 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::74ef:90ff:fe4c:9d6f/64 scope link 2024-01-26T23:46:04.297 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:46:04.297 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:46:04.297 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-26T23:46:04.297 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::2c73:afff:fef0:6f39/64 scope link 2024-01-26T23:46:04.297 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:46:04.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:46:04.297 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-26T23:46:04.366 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-26T23:46:04.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:46:04.368 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-26T23:46:04.432 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-26T23:46:04.432 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-26T23:46:04.432 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-26T23:46:04.432 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-26T23:46:04.432 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-26T23:46:04.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:46:04.432 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:46:04.433 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:46:04.433 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-26T23:46:04.433 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-26T23:46:04.433 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-26T23:46:04.433 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:46:04.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:46:04.559 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-26T23:46:04.578 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-26T23:46:04.578 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-26T23:46:04.578 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:46:04.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:46:04.579 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-01-26T23:46:04.652 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-26T23:46:04.653 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:46:04.653 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-26T23:46:04.715 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:46:04.715 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:46:04.715 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:46:04.777 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:46:04.853 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:46:04.853 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:46:04.860 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:04.859+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:46:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:46:04.930 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:46:04.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:46:04.930 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:46:04.992 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-26T23:46:04.992 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-26T23:46:05.035 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:46:05.069 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:46:05.069 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:46:05.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-01-26T23:46:05.138+0000 7fcd2d133540 -1 init, newargv = 0x555c37466de0 newargc=15 2024-01-26T23:46:05.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[76245]: starting ceph client 2024-01-26T23:46:05.142 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:46:05.143 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:46:05.144 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:46:05.144 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:46:05.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[76245]: starting fuse 2024-01-26T23:46:05.204 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-26T23:46:05.204 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-26T23:46:05.204 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-26T23:46:05.204 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-26T23:46:05.204 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-26T23:46:05.205 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-26T23:46:05.205 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-26T23:46:05.205 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:46:05.205 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-26T23:46:05.205 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-26T23:46:05.205 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-26T23:46:05.205 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-26T23:46:05.205 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:46:05.205 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-26T23:46:05.205 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-26T23:46:05.205 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-26T23:46:05.205 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-26T23:46:05.205 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:46:05.205 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-26T23:46:05.205 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-26T23:46:05.205 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-26T23:46:05.206 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-26T23:46:05.206 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-26T23:46:05.206 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-26T23:46:05.206 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-26T23:46:05.206 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-26T23:46:05.206 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:46:05.206 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-01-26T23:46:05.206 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-26T23:46:05.346 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.76245.asok 2024-01-26T23:46:05.527 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.76245.asok 2024-01-26T23:46:05.527 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:46:05.527 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.76245.asok status 2024-01-26T23:46:05.640 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:05.639+0000 7f6f9cb5d700 -1 client.6080: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.5966:afe67fe6-b866-473e-b85f-a6c2f23a0abb:v1:172.21.15.106:0/3064987359] 2024-01-26T23:46:05.704 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:46:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-26T23:46:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-26T23:46:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-26T23:46:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-01-26T23:46:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-26T23:46:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-26T23:46:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "76245", 2024-01-26T23:46:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-26T23:46:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-26T23:46:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-26T23:46:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 7005, 2024-01-26T23:46:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-26T23:46:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-26T23:46:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-26T23:46:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 7005 2024-01-26T23:46:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:46:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:46:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:46:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2595332689 2024-01-26T23:46:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:46:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:46:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:46:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2595332689 2024-01-26T23:46:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.7005 192.168.0.1:0/2595332689", 2024-01-26T23:46:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/2595332689", 2024-01-26T23:46:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-26T23:46:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 45, 2024-01-26T23:46:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 98, 2024-01-26T23:46:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-26T23:46:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-26T23:46:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-26T23:46:05.706 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:46:05.715 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:46:05.715 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-26T23:46:05.715 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-26T23:46:05.734 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-26T23:46:05.734 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-26T23:46:05.735 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:46:05.735 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:46:05.809 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-26T23:46:05.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.993+0000 7efea4480700 1 Processor -- start 2024-01-26T23:46:05.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.993+0000 7efea4480700 1 -- start start 2024-01-26T23:46:05.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.994+0000 7efea4480700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe9c159590 0x7efe9c15d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:05.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.994+0000 7efea4480700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7efe9c15d8f0 0x7efe9c06bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:05.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.994+0000 7efea4480700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c15dd90 con 0x7efe9c159960 2024-01-26T23:46:05.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.994+0000 7efea4480700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe9c15e350 con 0x7efe9c15d8f0 2024-01-26T23:46:05.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.994+0000 7efea4480700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c0c97c0 con 0x7efe9c159590 2024-01-26T23:46:05.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.994+0000 7efea1a1b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7efe9c15d8f0 0x7efe9c06bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:05.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.994+0000 7efea221c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe9c159590 0x7efe9c15d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:05.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.995+0000 7efea2a1d700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7efe9c159960 0x7efe9c06d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:55870/0 (socket says 172.21.15.46:55870) 2024-01-26T23:46:05.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.995+0000 7efea1a1b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7efe9c15d8f0 0x7efe9c06bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56378/0 (socket says 172.21.15.46:56378) 2024-01-26T23:46:05.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.995+0000 7efea2a1d700 1 -- 172.21.15.46:0/2522517400 learned_addr learned my addr 172.21.15.46:0/2522517400 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:05.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.995+0000 7efea121a700 1 -- 172.21.15.46:0/2522517400 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960849666 0 0) 0x7efe9c15dd90 con 0x7efe9c159960 2024-01-26T23:46:05.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.995+0000 7efea121a700 1 -- 172.21.15.46:0/2522517400 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe88003570 con 0x7efe9c159960 2024-01-26T23:46:05.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.995+0000 7efea1a1b700 1 -- 172.21.15.46:0/2522517400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe9c159590 msgr2=0x7efe9c15d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:05.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.995+0000 7efea1a1b700 1 --2- 172.21.15.46:0/2522517400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe9c159590 0x7efe9c15d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:05.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.995+0000 7efea1a1b700 1 -- 172.21.15.46:0/2522517400 >> v1:172.21.15.106:6789/0 conn(0x7efe9c159960 legacy=0x7efe9c06d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:05.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.995+0000 7efea1a1b700 1 -- 172.21.15.46:0/2522517400 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c15e070 con 0x7efe9c15d8f0 2024-01-26T23:46:05.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.995+0000 7efea221c700 1 --2- 172.21.15.46:0/2522517400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe9c159590 0x7efe9c15d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:05.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.995+0000 7efea1a1b700 1 --2- 172.21.15.46:0/2522517400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe9c15d8f0 0x7efe9c06bcb0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009910 tx=0x7efe8c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c43a9cd4b74a324 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:05.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.995+0000 7efea121a700 1 -- 172.21.15.46:0/2522517400 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c011630 con 0x7efe9c15d8f0 2024-01-26T23:46:05.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.995+0000 7efea121a700 1 -- 172.21.15.46:0/2522517400 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8c011790 con 0x7efe9c15d8f0 2024-01-26T23:46:05.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.995+0000 7efea121a700 1 -- 172.21.15.46:0/2522517400 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c011b30 con 0x7efe9c15d8f0 2024-01-26T23:46:05.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.996+0000 7efea4480700 1 -- 172.21.15.46:0/2522517400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe9c15d8f0 msgr2=0x7efe9c06bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:05.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.996+0000 7efea4480700 1 --2- 172.21.15.46:0/2522517400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe9c15d8f0 0x7efe9c06bcb0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009910 tx=0x7efe8c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:05.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.996+0000 7efea4480700 1 -- 172.21.15.46:0/2522517400 shutdown_connections 2024-01-26T23:46:05.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.996+0000 7efea4480700 1 --2- 172.21.15.46:0/2522517400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe9c159590 0x7efe9c15d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:05.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.996+0000 7efea4480700 1 --2- 172.21.15.46:0/2522517400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe9c15d8f0 0x7efe9c06bcb0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:05.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.996+0000 7efea4480700 1 -- 172.21.15.46:0/2522517400 >> 172.21.15.46:0/2522517400 conn(0x7efe9c0ce530 msgr2=0x7efe9c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:05.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.996+0000 7efea4480700 1 -- 172.21.15.46:0/2522517400 shutdown_connections 2024-01-26T23:46:05.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.996+0000 7efea4480700 1 -- 172.21.15.46:0/2522517400 wait complete. 2024-01-26T23:46:06.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.997+0000 7efea4480700 1 Processor -- start 2024-01-26T23:46:06.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.997+0000 7efea4480700 1 -- start start 2024-01-26T23:46:06.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.998+0000 7efea4480700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe9c159590 0x7efe9c158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:06.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.998+0000 7efea4480700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe9c159960 0x7efe9c0c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:06.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.998+0000 7efea4480700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe9c15d8f0 0x7efe9c0b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:06.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.998+0000 7efea4480700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c0c5f20 con 0x7efe9c159590 2024-01-26T23:46:06.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.998+0000 7efea4480700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c0c6200 con 0x7efe9c15d8f0 2024-01-26T23:46:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.998+0000 7efea4480700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c0c64e0 con 0x7efe9c159960 2024-01-26T23:46:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.998+0000 7efea221c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe9c159590 0x7efe9c158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.998+0000 7efea221c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe9c159590 0x7efe9c158a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58878/0 (socket says 172.21.15.46:58878) 2024-01-26T23:46:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.998+0000 7efea221c700 1 -- 172.21.15.46:0/1391679084 learned_addr learned my addr 172.21.15.46:0/1391679084 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.998+0000 7efea1a1b700 1 --2- 172.21.15.46:0/1391679084 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe9c159960 0x7efe9c0c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.998+0000 7efea2a1d700 1 --2- 172.21.15.46:0/1391679084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe9c15d8f0 0x7efe9c0b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.999+0000 7efea221c700 1 -- 172.21.15.46:0/1391679084 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe9c159960 msgr2=0x7efe9c0c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.999+0000 7efea221c700 1 --2- 172.21.15.46:0/1391679084 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe9c159960 0x7efe9c0c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.999+0000 7efea221c700 1 -- 172.21.15.46:0/1391679084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe9c15d8f0 msgr2=0x7efe9c0b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.999+0000 7efea221c700 1 --2- 172.21.15.46:0/1391679084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe9c15d8f0 0x7efe9c0b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.999+0000 7efea221c700 1 -- 172.21.15.46:0/1391679084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe98009420 con 0x7efe9c159590 2024-01-26T23:46:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.999+0000 7efea221c700 1 --2- 172.21.15.46:0/1391679084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe9c159590 0x7efe9c158a70 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7efe9800ec80 tx=0x7efe980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.999+0000 7efe927fc700 1 -- 172.21.15.46:0/1391679084 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9800c920 con 0x7efe9c159590 2024-01-26T23:46:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.999+0000 7efe927fc700 1 -- 172.21.15.46:0/1391679084 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9800ca80 con 0x7efe9c159590 2024-01-26T23:46:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.999+0000 7efea4480700 1 -- 172.21.15.46:0/1391679084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe8c009420 con 0x7efe9c159590 2024-01-26T23:46:06.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.999+0000 7efe927fc700 1 -- 172.21.15.46:0/1391679084 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9800ce00 con 0x7efe9c159590 2024-01-26T23:46:06.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:05.999+0000 7efea4480700 1 -- 172.21.15.46:0/1391679084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9c158fb0 con 0x7efe9c159590 2024-01-26T23:46:06.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.000+0000 7efea4480700 1 -- 172.21.15.46:0/1391679084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9c15af60 con 0x7efe9c159590 2024-01-26T23:46:06.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.001+0000 7efe927fc700 1 -- 172.21.15.46:0/1391679084 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7efe98013680 con 0x7efe9c159590 2024-01-26T23:46:06.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.001+0000 7efe927fc700 1 --2- 172.21.15.46:0/1391679084 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efe80066ea0 0x7efe80069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:06.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.001+0000 7efe927fc700 1 -- 172.21.15.46:0/1391679084 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6460+0+0 (secure 0 0 0) 0x7efe98084b00 con 0x7efe9c159590 2024-01-26T23:46:06.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.001+0000 7efea1a1b700 1 --2- 172.21.15.46:0/1391679084 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efe80066ea0 0x7efe80069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:06.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.002+0000 7efea1a1b700 1 --2- 172.21.15.46:0/1391679084 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efe80066ea0 0x7efe80069350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efe9c15dcc0 tx=0x7efe8c017000 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:06.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.006+0000 7efe927fc700 1 -- 172.21.15.46:0/1391679084 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efe98051b60 con 0x7efe9c159590 2024-01-26T23:46:06.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.241+0000 7efea4480700 1 -- 172.21.15.46:0/1391679084 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7efe9c0c1a40 con 0x7efe80066ea0 2024-01-26T23:46:06.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.243+0000 7efe927fc700 1 -- 172.21.15.46:0/1391679084 <== mgr.6421 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7efe9c0c1a40 con 0x7efe80066ea0 2024-01-26T23:46:06.244 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-26T23:46:06.244 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:06.244 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephfs" 2024-01-26T23:46:06.244 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:06.245 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-26T23:46:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.245+0000 7efea4480700 1 -- 172.21.15.46:0/1391679084 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efe80066ea0 msgr2=0x7efe80069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.245+0000 7efea4480700 1 --2- 172.21.15.46:0/1391679084 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efe80066ea0 0x7efe80069350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efe9c15dcc0 tx=0x7efe8c017000 comp rx=0 tx=0).stop 2024-01-26T23:46:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.245+0000 7efea4480700 1 -- 172.21.15.46:0/1391679084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe9c159590 msgr2=0x7efe9c158a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.245+0000 7efea4480700 1 --2- 172.21.15.46:0/1391679084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe9c159590 0x7efe9c158a70 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7efe9800ec80 tx=0x7efe980091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.245+0000 7efea4480700 1 -- 172.21.15.46:0/1391679084 shutdown_connections 2024-01-26T23:46:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.245+0000 7efea4480700 1 --2- 172.21.15.46:0/1391679084 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efe80066ea0 0x7efe80069350 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.245+0000 7efea4480700 1 --2- 172.21.15.46:0/1391679084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe9c159590 0x7efe9c158a70 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.245+0000 7efea4480700 1 --2- 172.21.15.46:0/1391679084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe9c15d8f0 0x7efe9c0b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.245+0000 7efea4480700 1 --2- 172.21.15.46:0/1391679084 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe9c159960 0x7efe9c0c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:06.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.245+0000 7efea4480700 1 -- 172.21.15.46:0/1391679084 >> 172.21.15.46:0/1391679084 conn(0x7efe9c0ce530 msgr2=0x7efe9c0ce900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:06.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.246+0000 7efea4480700 1 -- 172.21.15.46:0/1391679084 shutdown_connections 2024-01-26T23:46:06.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.246+0000 7efea4480700 1 -- 172.21.15.46:0/1391679084 wait complete. 2024-01-26T23:46:06.260 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-26T23:46:06.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.393+0000 7fc8d643a700 1 Processor -- start 2024-01-26T23:46:06.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.393+0000 7fc8d643a700 1 -- start start 2024-01-26T23:46:06.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.394+0000 7fc8d643a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc8d006ebe0 0x7fc8d0067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:06.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.394+0000 7fc8d643a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc8d01597e0 0x7fc8d0159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:06.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.394+0000 7fc8d643a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d006fdb0 con 0x7fc8d0067af0 2024-01-26T23:46:06.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.394+0000 7fc8d643a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8d006f1a0 con 0x7fc8d01597e0 2024-01-26T23:46:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.394+0000 7fc8d643a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8d006f480 con 0x7fc8d006ebe0 2024-01-26T23:46:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.395+0000 7fc8d49d7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc8d01597e0 0x7fc8d0159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.395+0000 7fc8d49d7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc8d01597e0 0x7fc8d0159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58882/0 (socket says 172.21.15.46:58882) 2024-01-26T23:46:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.395+0000 7fc8d49d7700 1 -- 172.21.15.46:0/2579704809 learned_addr learned my addr 172.21.15.46:0/2579704809 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.395+0000 7fc8cf7fe700 1 --2- 172.21.15.46:0/2579704809 >> v2:172.21.15.106:3300/0 conn(0x7fc8d006ebe0 0x7fc8d0067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.395+0000 7fc8d49d7700 1 -- 172.21.15.46:0/2579704809 >> v2:172.21.15.106:3300/0 conn(0x7fc8d006ebe0 msgr2=0x7fc8d0067ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.395+0000 7fc8d49d7700 1 --2- 172.21.15.46:0/2579704809 >> v2:172.21.15.106:3300/0 conn(0x7fc8d006ebe0 0x7fc8d0067ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.395+0000 7fc8d49d7700 1 -- 172.21.15.46:0/2579704809 >> v1:172.21.15.106:6789/0 conn(0x7fc8d0067af0 legacy=0x7fc8d00ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.395+0000 7fc8d49d7700 1 -- 172.21.15.46:0/2579704809 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d006f760 con 0x7fc8d01597e0 2024-01-26T23:46:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.395+0000 7fc8cf7fe700 1 --2- 172.21.15.46:0/2579704809 >> v2:172.21.15.106:3300/0 conn(0x7fc8d006ebe0 0x7fc8d0067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:46:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.395+0000 7fc8d49d7700 1 --2- 172.21.15.46:0/2579704809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8d01597e0 0x7fc8d0159bf0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc8d00ce590 tx=0x7fc8c8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7605512dd615c29a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.395+0000 7fc8ceffd700 1 -- 172.21.15.46:0/2579704809 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c800e070 con 0x7fc8d01597e0 2024-01-26T23:46:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.395+0000 7fc8ceffd700 1 -- 172.21.15.46:0/2579704809 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c800ba50 con 0x7fc8d01597e0 2024-01-26T23:46:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.395+0000 7fc8ceffd700 1 -- 172.21.15.46:0/2579704809 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c80092e0 con 0x7fc8d01597e0 2024-01-26T23:46:06.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.396+0000 7fc8d643a700 1 -- 172.21.15.46:0/2579704809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8d01597e0 msgr2=0x7fc8d0159bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:06.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.396+0000 7fc8d643a700 1 --2- 172.21.15.46:0/2579704809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8d01597e0 0x7fc8d0159bf0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc8d00ce590 tx=0x7fc8c8009840 comp rx=0 tx=0).stop 2024-01-26T23:46:06.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.396+0000 7fc8d643a700 1 -- 172.21.15.46:0/2579704809 shutdown_connections 2024-01-26T23:46:06.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.396+0000 7fc8d643a700 1 --2- 172.21.15.46:0/2579704809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8d01597e0 0x7fc8d0159bf0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:06.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.396+0000 7fc8d643a700 1 --2- 172.21.15.46:0/2579704809 >> v2:172.21.15.106:3300/0 conn(0x7fc8d006ebe0 0x7fc8d0067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:06.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.396+0000 7fc8d643a700 1 -- 172.21.15.46:0/2579704809 >> 172.21.15.46:0/2579704809 conn(0x7fc8d00cdf80 msgr2=0x7fc8d006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:06.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.396+0000 7fc8d643a700 1 -- 172.21.15.46:0/2579704809 shutdown_connections 2024-01-26T23:46:06.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.396+0000 7fc8d643a700 1 -- 172.21.15.46:0/2579704809 wait complete. 2024-01-26T23:46:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.397+0000 7fc8d643a700 1 Processor -- start 2024-01-26T23:46:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.398+0000 7fc8d643a700 1 -- start start 2024-01-26T23:46:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.398+0000 7fc8d643a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8d0067af0 0x7fc8d0140ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.398+0000 7fc8d643a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8d006ebe0 0x7fc8d0156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.398+0000 7fc8d643a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8d01597e0 0x7fc8d0156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.398+0000 7fc8d643a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d0142500 con 0x7fc8d0067af0 2024-01-26T23:46:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.398+0000 7fc8d643a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d01418f0 con 0x7fc8d01597e0 2024-01-26T23:46:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.398+0000 7fc8d643a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d0141bd0 con 0x7fc8d006ebe0 2024-01-26T23:46:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.399+0000 7fc8cffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8d0067af0 0x7fc8d0140ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.399+0000 7fc8cffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8d0067af0 0x7fc8d0140ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58892/0 (socket says 172.21.15.46:58892) 2024-01-26T23:46:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.399+0000 7fc8cffff700 1 -- 172.21.15.46:0/2851169115 learned_addr learned my addr 172.21.15.46:0/2851169115 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.399+0000 7fc8cf7fe700 1 --2- 172.21.15.46:0/2851169115 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8d006ebe0 0x7fc8d0156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.399+0000 7fc8d49d7700 1 --2- 172.21.15.46:0/2851169115 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8d01597e0 0x7fc8d0156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.399+0000 7fc8cffff700 1 -- 172.21.15.46:0/2851169115 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8d006ebe0 msgr2=0x7fc8d0156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.399+0000 7fc8cffff700 1 --2- 172.21.15.46:0/2851169115 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8d006ebe0 0x7fc8d0156a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.399+0000 7fc8cffff700 1 -- 172.21.15.46:0/2851169115 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8d01597e0 msgr2=0x7fc8d0156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.399+0000 7fc8cffff700 1 --2- 172.21.15.46:0/2851169115 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8d01597e0 0x7fc8d0156f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.399+0000 7fc8cffff700 1 -- 172.21.15.46:0/2851169115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c8009ae0 con 0x7fc8d0067af0 2024-01-26T23:46:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.399+0000 7fc8cffff700 1 --2- 172.21.15.46:0/2851169115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8d0067af0 0x7fc8d0140ec0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000c4e0 tx=0x7fc8c000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.399+0000 7fc8c7fff700 1 -- 172.21.15.46:0/2851169115 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0011590 con 0x7fc8d0067af0 2024-01-26T23:46:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.399+0000 7fc8c7fff700 1 -- 172.21.15.46:0/2851169115 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c0011710 con 0x7fc8d0067af0 2024-01-26T23:46:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.399+0000 7fc8d643a700 1 -- 172.21.15.46:0/2851169115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d0141eb0 con 0x7fc8d0067af0 2024-01-26T23:46:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.399+0000 7fc8c7fff700 1 -- 172.21.15.46:0/2851169115 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0011ad0 con 0x7fc8d0067af0 2024-01-26T23:46:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.399+0000 7fc8d643a700 1 -- 172.21.15.46:0/2851169115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d0152720 con 0x7fc8d0067af0 2024-01-26T23:46:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.400+0000 7fc8d643a700 1 -- 172.21.15.46:0/2851169115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d004c710 con 0x7fc8d0067af0 2024-01-26T23:46:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.401+0000 7fc8c7fff700 1 -- 172.21.15.46:0/2851169115 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fc8c001b3b0 con 0x7fc8d0067af0 2024-01-26T23:46:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.402+0000 7fc8c7fff700 1 --2- 172.21.15.46:0/2851169115 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc8b0066ea0 0x7fc8b0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.402+0000 7fc8c7fff700 1 -- 172.21.15.46:0/2851169115 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc8c0083ea0 con 0x7fc8d0067af0 2024-01-26T23:46:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.402+0000 7fc8cf7fe700 1 --2- 172.21.15.46:0/2851169115 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc8b0066ea0 0x7fc8b0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.402+0000 7fc8cf7fe700 1 --2- 172.21.15.46:0/2851169115 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc8b0066ea0 0x7fc8b0069350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0152d60 tx=0x7fc8b8008040 comp rx=0 tx=0).ready entity=mgr.6421 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.406+0000 7fc8c7fff700 1 -- 172.21.15.46:0/2851169115 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc8c0050f80 con 0x7fc8d0067af0 2024-01-26T23:46:06.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:06.673+0000 7fc8d643a700 1 -- 172.21.15.46:0/2851169115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fc8d00aec10 con 0x7fc8d0067af0 2024-01-26T23:46:07.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.646+0000 7fc8c7fff700 1 -- 172.21.15.46:0/2851169115 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 38) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fc8c0051350 con 0x7fc8d0067af0 2024-01-26T23:46:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.648+0000 7fc8c7fff700 1 -- 172.21.15.46:0/2851169115 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v38) v1 ==== 92+0+0 (secure 0 0 0) 0x7fc8c0054a10 con 0x7fc8d0067af0 2024-01-26T23:46:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.651+0000 7fc8d643a700 1 -- 172.21.15.46:0/2851169115 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc8b0066ea0 msgr2=0x7fc8b0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.651+0000 7fc8d643a700 1 --2- 172.21.15.46:0/2851169115 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc8b0066ea0 0x7fc8b0069350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0152d60 tx=0x7fc8b8008040 comp rx=0 tx=0).stop 2024-01-26T23:46:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.651+0000 7fc8d643a700 1 -- 172.21.15.46:0/2851169115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8d0067af0 msgr2=0x7fc8d0140ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.651+0000 7fc8d643a700 1 --2- 172.21.15.46:0/2851169115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8d0067af0 0x7fc8d0140ec0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000c4e0 tx=0x7fc8c000c7f0 comp rx=0 tx=0).stop 2024-01-26T23:46:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.651+0000 7fc8d643a700 1 -- 172.21.15.46:0/2851169115 shutdown_connections 2024-01-26T23:46:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.651+0000 7fc8d643a700 1 --2- 172.21.15.46:0/2851169115 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc8b0066ea0 0x7fc8b0069350 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.651+0000 7fc8d643a700 1 --2- 172.21.15.46:0/2851169115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8d0067af0 0x7fc8d0140ec0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.651+0000 7fc8d643a700 1 --2- 172.21.15.46:0/2851169115 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8d01597e0 0x7fc8d0156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.651+0000 7fc8d643a700 1 --2- 172.21.15.46:0/2851169115 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8d006ebe0 0x7fc8d0156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.651+0000 7fc8d643a700 1 -- 172.21.15.46:0/2851169115 >> 172.21.15.46:0/2851169115 conn(0x7fc8d00cdf80 msgr2=0x7fc8d00b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.652+0000 7fc8d643a700 1 -- 172.21.15.46:0/2851169115 shutdown_connections 2024-01-26T23:46:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.652+0000 7fc8d643a700 1 -- 172.21.15.46:0/2851169115 wait complete. 2024-01-26T23:46:07.669 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-26T23:46:07.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.807+0000 7f30a1e8d700 1 Processor -- start 2024-01-26T23:46:07.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.807+0000 7f30a1e8d700 1 -- start start 2024-01-26T23:46:07.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.808+0000 7f30a1e8d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f309c0c8da0 0x7f309c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:07.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.808+0000 7f30a1e8d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f309c0caf30 0x7f309c0cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:07.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.808+0000 7f30a1e8d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f309c0ccc00 con 0x7f309c140af0 2024-01-26T23:46:07.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.808+0000 7f30a1e8d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309c0cbbf0 con 0x7f309c0caf30 2024-01-26T23:46:07.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.808+0000 7f30a1e8d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309c0cc310 con 0x7f309c0c8da0 2024-01-26T23:46:07.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.808+0000 7f309bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f309c0caf30 0x7f309c0cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:07.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.808+0000 7f309bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f309c0caf30 0x7f309c0cb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58926/0 (socket says 172.21.15.46:58926) 2024-01-26T23:46:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.808+0000 7f309bfff700 1 -- 172.21.15.46:0/1425362920 learned_addr learned my addr 172.21.15.46:0/1425362920 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.808+0000 7f309b7fe700 1 --2- 172.21.15.46:0/1425362920 >> v2:172.21.15.106:3300/0 conn(0x7f309c0c8da0 0x7f309c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.809+0000 7f309bfff700 1 -- 172.21.15.46:0/1425362920 >> v2:172.21.15.106:3300/0 conn(0x7f309c0c8da0 msgr2=0x7f309c0c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.809+0000 7f309bfff700 1 --2- 172.21.15.46:0/1425362920 >> v2:172.21.15.106:3300/0 conn(0x7f309c0c8da0 0x7f309c0c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.809+0000 7f309bfff700 1 -- 172.21.15.46:0/1425362920 >> v1:172.21.15.106:6789/0 conn(0x7f309c140af0 legacy=0x7f309c140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.809+0000 7f309affd700 1 -- 172.21.15.46:0/1425362920 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322198551 0 0) 0x7f309c0ccc00 con 0x7f309c140af0 2024-01-26T23:46:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.809+0000 7f309bfff700 1 -- 172.21.15.46:0/1425362920 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309c0cc5b0 con 0x7f309c0caf30 2024-01-26T23:46:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.809+0000 7f309bfff700 1 --2- 172.21.15.46:0/1425362920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0caf30 0x7f309c0cb370 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f309c06f100 tx=0x7f3094009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3126579753e88e02 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.809+0000 7f309affd700 1 -- 172.21.15.46:0/1425362920 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309400e070 con 0x7f309c0caf30 2024-01-26T23:46:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.809+0000 7f309affd700 1 -- 172.21.15.46:0/1425362920 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309400ba20 con 0x7f309c0caf30 2024-01-26T23:46:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.809+0000 7f309affd700 1 -- 172.21.15.46:0/1425362920 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30940092e0 con 0x7f309c0caf30 2024-01-26T23:46:07.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.810+0000 7f30a1e8d700 1 -- 172.21.15.46:0/1425362920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0caf30 msgr2=0x7f309c0cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.810+0000 7f30a1e8d700 1 --2- 172.21.15.46:0/1425362920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0caf30 0x7f309c0cb370 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f309c06f100 tx=0x7f3094009840 comp rx=0 tx=0).stop 2024-01-26T23:46:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.810+0000 7f30a1e8d700 1 -- 172.21.15.46:0/1425362920 shutdown_connections 2024-01-26T23:46:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.810+0000 7f30a1e8d700 1 --2- 172.21.15.46:0/1425362920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0caf30 0x7f309c0cb370 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.810+0000 7f30a1e8d700 1 --2- 172.21.15.46:0/1425362920 >> v2:172.21.15.106:3300/0 conn(0x7f309c0c8da0 0x7f309c0c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.810+0000 7f30a1e8d700 1 -- 172.21.15.46:0/1425362920 >> 172.21.15.46:0/1425362920 conn(0x7f309c06bd90 msgr2=0x7f309c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.810+0000 7f30a1e8d700 1 -- 172.21.15.46:0/1425362920 shutdown_connections 2024-01-26T23:46:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.810+0000 7f30a1e8d700 1 -- 172.21.15.46:0/1425362920 wait complete. 2024-01-26T23:46:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.811+0000 7f30a1e8d700 1 Processor -- start 2024-01-26T23:46:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.811+0000 7f30a1e8d700 1 -- start start 2024-01-26T23:46:07.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.812+0000 7f30a1e8d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0c8da0 0x7f309c0c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:07.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.812+0000 7f30a1e8d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f309c0caf30 0x7f309c157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:07.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.812+0000 7f30a1e8d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f309c140af0 0x7f309c167f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:07.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.812+0000 7f30a1e8d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c168440 con 0x7f309c0c8da0 2024-01-26T23:46:07.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.812+0000 7f30a1e8d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0c7710 con 0x7f309c140af0 2024-01-26T23:46:07.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.812+0000 7f30a1e8d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0b0af0 con 0x7f309c0caf30 2024-01-26T23:46:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.812+0000 7f309b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0c8da0 0x7f309c0c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.812+0000 7f309b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0c8da0 0x7f309c0c7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58930/0 (socket says 172.21.15.46:58930) 2024-01-26T23:46:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.812+0000 7f309b7fe700 1 -- 172.21.15.46:0/4080740195 learned_addr learned my addr 172.21.15.46:0/4080740195 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.812+0000 7f309bfff700 1 --2- 172.21.15.46:0/4080740195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f309c140af0 0x7f309c167f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.812+0000 7f3092ffd700 1 --2- 172.21.15.46:0/4080740195 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f309c0caf30 0x7f309c157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.812+0000 7f309b7fe700 1 -- 172.21.15.46:0/4080740195 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f309c0caf30 msgr2=0x7f309c157220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.812+0000 7f309b7fe700 1 --2- 172.21.15.46:0/4080740195 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f309c0caf30 0x7f309c157220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.812+0000 7f309b7fe700 1 -- 172.21.15.46:0/4080740195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f309c140af0 msgr2=0x7f309c167f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.812+0000 7f309b7fe700 1 --2- 172.21.15.46:0/4080740195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f309c140af0 0x7f309c167f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.813+0000 7f309b7fe700 1 -- 172.21.15.46:0/4080740195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3094009ae0 con 0x7f309c0c8da0 2024-01-26T23:46:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.813+0000 7f309b7fe700 1 --2- 172.21.15.46:0/4080740195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0c8da0 0x7f309c0c7ba0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f308c00ec80 tx=0x7f308c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.813+0000 7f3093fff700 1 -- 172.21.15.46:0/4080740195 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c00c850 con 0x7f309c0c8da0 2024-01-26T23:46:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.813+0000 7f3093fff700 1 -- 172.21.15.46:0/4080740195 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308c00c9b0 con 0x7f309c0c8da0 2024-01-26T23:46:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.813+0000 7f30a1e8d700 1 -- 172.21.15.46:0/4080740195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309c0c6b00 con 0x7f309c0c8da0 2024-01-26T23:46:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.813+0000 7f3093fff700 1 -- 172.21.15.46:0/4080740195 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c00cd50 con 0x7f309c0c8da0 2024-01-26T23:46:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.813+0000 7f30a1e8d700 1 -- 172.21.15.46:0/4080740195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309c0c70c0 con 0x7f309c0c8da0 2024-01-26T23:46:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.814+0000 7f30a1e8d700 1 -- 172.21.15.46:0/4080740195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309c04c710 con 0x7f309c0c8da0 2024-01-26T23:46:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.815+0000 7f3093fff700 1 -- 172.21.15.46:0/4080740195 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 38) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f308c013680 con 0x7f309c0c8da0 2024-01-26T23:46:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.815+0000 7f3093fff700 1 --2- 172.21.15.46:0/4080740195 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f307c066f80 0x7f307c069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.815+0000 7f3092ffd700 1 -- 172.21.15.46:0/4080740195 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f307c066f80 msgr2=0x7f307c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.815+0000 7f3093fff700 1 -- 172.21.15.46:0/4080740195 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f308c084b80 con 0x7f309c0c8da0 2024-01-26T23:46:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.815+0000 7f3092ffd700 1 --2- 172.21.15.46:0/4080740195 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f307c066f80 0x7f307c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:46:07.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:07.820+0000 7f3093fff700 1 -- 172.21.15.46:0/4080740195 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f308c051a60 con 0x7f309c0c8da0 2024-01-26T23:46:07.987 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:07.986+0000 7f16b26b21c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:46:08.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.016+0000 7f3092ffd700 1 -- 172.21.15.46:0/4080740195 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f307c066f80 msgr2=0x7f307c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:08.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.016+0000 7f3092ffd700 1 --2- 172.21.15.46:0/4080740195 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f307c066f80 0x7f307c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:46:08.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.028+0000 7f30a1e8d700 1 -- 172.21.15.46:0/4080740195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f309c0cc890 con 0x7f309c0c8da0 2024-01-26T23:46:08.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.033+0000 7f3093fff700 1 -- 172.21.15.46:0/4080740195 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7f308c055080 con 0x7f309c0c8da0 2024-01-26T23:46:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.035+0000 7f30a1e8d700 1 -- 172.21.15.46:0/4080740195 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f307c066f80 msgr2=0x7f307c069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:46:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.035+0000 7f30a1e8d700 1 --2- 172.21.15.46:0/4080740195 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f307c066f80 0x7f307c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.035+0000 7f30a1e8d700 1 -- 172.21.15.46:0/4080740195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0c8da0 msgr2=0x7f309c0c7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.035+0000 7f30a1e8d700 1 --2- 172.21.15.46:0/4080740195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0c8da0 0x7f309c0c7ba0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f308c00ec80 tx=0x7f308c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.035+0000 7f30a1e8d700 1 -- 172.21.15.46:0/4080740195 shutdown_connections 2024-01-26T23:46:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.035+0000 7f30a1e8d700 1 --2- 172.21.15.46:0/4080740195 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f307c066f80 0x7f307c069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.035+0000 7f30a1e8d700 1 --2- 172.21.15.46:0/4080740195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f309c0c8da0 0x7f309c0c7ba0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.035+0000 7f30a1e8d700 1 --2- 172.21.15.46:0/4080740195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f309c140af0 0x7f309c167f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.035+0000 7f30a1e8d700 1 --2- 172.21.15.46:0/4080740195 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f309c0caf30 0x7f309c157220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.035+0000 7f30a1e8d700 1 -- 172.21.15.46:0/4080740195 >> 172.21.15.46:0/4080740195 conn(0x7f309c06bd90 msgr2=0x7f309c0c41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.035+0000 7f30a1e8d700 1 -- 172.21.15.46:0/4080740195 shutdown_connections 2024-01-26T23:46:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.035+0000 7f30a1e8d700 1 -- 172.21.15.46:0/4080740195 wait complete. 2024-01-26T23:46:08.038 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:08.036+0000 7fd30e06d1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:46:08.051 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-26T23:46:08.067 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:08.066+0000 7f16b26b21c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:46:08.128 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:08.127+0000 7fd30e06d1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:46:08.137 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:08.136+0000 7f16b26b21c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:46:08.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.186+0000 7fb93625a700 1 Processor -- start 2024-01-26T23:46:08.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.186+0000 7fb93625a700 1 -- start start 2024-01-26T23:46:08.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.187+0000 7fb93625a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9300cf050 0x7fb9300af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:08.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.187+0000 7fb93625a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb9300cf4a0 0x7fb9300af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:08.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.187+0000 7fb93625a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9300cc100 con 0x7fb930159cb0 2024-01-26T23:46:08.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.187+0000 7fb93625a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9300ccca0 con 0x7fb9300cf4a0 2024-01-26T23:46:08.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.187+0000 7fb93625a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9300cb850 con 0x7fb9300cf050 2024-01-26T23:46:08.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.187+0000 7fb92ffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb9300cf4a0 0x7fb9300af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:08.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.187+0000 7fb92ffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb9300cf4a0 0x7fb9300af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58938/0 (socket says 172.21.15.46:58938) 2024-01-26T23:46:08.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.187+0000 7fb92ffff700 1 -- 172.21.15.46:0/4040364694 learned_addr learned my addr 172.21.15.46:0/4040364694 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:08.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.187+0000 7fb92effd700 1 --2- 172.21.15.46:0/4040364694 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9300cf050 0x7fb9300af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:08.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.188+0000 7fb92e7fc700 1 -- 172.21.15.46:0/4040364694 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089895456 0 0) 0x7fb9300cc100 con 0x7fb930159cb0 2024-01-26T23:46:08.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.188+0000 7fb92ffff700 1 -- 172.21.15.46:0/4040364694 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9300cf050 msgr2=0x7fb9300af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:08.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.188+0000 7fb92ffff700 1 --2- 172.21.15.46:0/4040364694 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9300cf050 0x7fb9300af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.188+0000 7fb92ffff700 1 -- 172.21.15.46:0/4040364694 >> v1:172.21.15.106:6789/0 conn(0x7fb930159cb0 legacy=0x7fb9300cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:08.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.188+0000 7fb92ffff700 1 -- 172.21.15.46:0/4040364694 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9300cbaf0 con 0x7fb9300cf4a0 2024-01-26T23:46:08.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.188+0000 7fb92ffff700 1 --2- 172.21.15.46:0/4040364694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9300cf4a0 0x7fb9300af970 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fb928009f20 tx=0x7fb928009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f904fbc9896e7e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:08.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.188+0000 7fb92e7fc700 1 -- 172.21.15.46:0/4040364694 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92800e070 con 0x7fb9300cf4a0 2024-01-26T23:46:08.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.188+0000 7fb92e7fc700 1 -- 172.21.15.46:0/4040364694 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92800b9d0 con 0x7fb9300cf4a0 2024-01-26T23:46:08.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.188+0000 7fb92e7fc700 1 -- 172.21.15.46:0/4040364694 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9280092e0 con 0x7fb9300cf4a0 2024-01-26T23:46:08.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.189+0000 7fb93625a700 1 -- 172.21.15.46:0/4040364694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9300cf4a0 msgr2=0x7fb9300af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:08.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.189+0000 7fb93625a700 1 --2- 172.21.15.46:0/4040364694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9300cf4a0 0x7fb9300af970 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fb928009f20 tx=0x7fb928009840 comp rx=0 tx=0).stop 2024-01-26T23:46:08.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.189+0000 7fb93625a700 1 -- 172.21.15.46:0/4040364694 shutdown_connections 2024-01-26T23:46:08.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.189+0000 7fb93625a700 1 --2- 172.21.15.46:0/4040364694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9300cf4a0 0x7fb9300af970 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.189+0000 7fb93625a700 1 --2- 172.21.15.46:0/4040364694 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9300cf050 0x7fb9300af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.189+0000 7fb93625a700 1 -- 172.21.15.46:0/4040364694 >> 172.21.15.46:0/4040364694 conn(0x7fb930087f60 msgr2=0x7fb93006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:08.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.189+0000 7fb93625a700 1 -- 172.21.15.46:0/4040364694 shutdown_connections 2024-01-26T23:46:08.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.189+0000 7fb93625a700 1 -- 172.21.15.46:0/4040364694 wait complete. 2024-01-26T23:46:08.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.190+0000 7fb93625a700 1 Processor -- start 2024-01-26T23:46:08.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.190+0000 7fb93625a700 1 -- start start 2024-01-26T23:46:08.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.191+0000 7fb93625a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9300cf050 0x7fb9300b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:08.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.191+0000 7fb93625a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9300cf4a0 0x7fb9300b0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.191+0000 7fb93625a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb930159cb0 0x7fb9300b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.191+0000 7fb93625a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93016a880 con 0x7fb930159cb0 2024-01-26T23:46:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.191+0000 7fb93625a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb930169c70 con 0x7fb9300cf4a0 2024-01-26T23:46:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.191+0000 7fb93625a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb930169f50 con 0x7fb9300cf050 2024-01-26T23:46:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.191+0000 7fb92ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb930159cb0 0x7fb9300b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.191+0000 7fb92effd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9300cf4a0 0x7fb9300b0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.191+0000 7fb92f7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9300cf050 0x7fb9300b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.191+0000 7fb92effd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9300cf4a0 0x7fb9300b0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56474/0 (socket says 172.21.15.46:56474) 2024-01-26T23:46:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.191+0000 7fb92effd700 1 -- 172.21.15.46:0/227782685 learned_addr learned my addr 172.21.15.46:0/227782685 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.192+0000 7fb92ffff700 1 -- 172.21.15.46:0/227782685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9300cf050 msgr2=0x7fb9300b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.192+0000 7fb92ffff700 1 --2- 172.21.15.46:0/227782685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9300cf050 0x7fb9300b29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.192+0000 7fb92ffff700 1 -- 172.21.15.46:0/227782685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9300cf4a0 msgr2=0x7fb9300b0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.192+0000 7fb92ffff700 1 --2- 172.21.15.46:0/227782685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9300cf4a0 0x7fb9300b0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.192+0000 7fb92ffff700 1 -- 172.21.15.46:0/227782685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb928009ae0 con 0x7fb930159cb0 2024-01-26T23:46:08.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.192+0000 7fb92effd700 1 --2- 172.21.15.46:0/227782685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9300cf4a0 0x7fb9300b0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:08.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.192+0000 7fb92f7fe700 1 --2- 172.21.15.46:0/227782685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9300cf050 0x7fb9300b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:08.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.192+0000 7fb92ffff700 1 --2- 172.21.15.46:0/227782685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb930159cb0 0x7fb9300b5a30 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fb928015000 tx=0x7fb928018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:08.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.192+0000 7fb9177fe700 1 -- 172.21.15.46:0/227782685 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb928008b30 con 0x7fb930159cb0 2024-01-26T23:46:08.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.192+0000 7fb9177fe700 1 -- 172.21.15.46:0/227782685 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb928008cb0 con 0x7fb930159cb0 2024-01-26T23:46:08.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.192+0000 7fb93625a700 1 -- 172.21.15.46:0/227782685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb93016a230 con 0x7fb930159cb0 2024-01-26T23:46:08.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.192+0000 7fb9177fe700 1 -- 172.21.15.46:0/227782685 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9280266b0 con 0x7fb930159cb0 2024-01-26T23:46:08.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.192+0000 7fb93625a700 1 -- 172.21.15.46:0/227782685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9300b40f0 con 0x7fb930159cb0 2024-01-26T23:46:08.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.193+0000 7fb93625a700 1 -- 172.21.15.46:0/227782685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb93004c710 con 0x7fb930159cb0 2024-01-26T23:46:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.194+0000 7fb9177fe700 1 -- 172.21.15.46:0/227782685 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 38) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fb928003040 con 0x7fb930159cb0 2024-01-26T23:46:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.195+0000 7fb9177fe700 1 --2- 172.21.15.46:0/227782685 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb91c066f80 0x7fb91c069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.195+0000 7fb92f7fe700 1 -- 172.21.15.46:0/227782685 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb91c066f80 msgr2=0x7fb91c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.195+0000 7fb9177fe700 1 -- 172.21.15.46:0/227782685 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb9280866e0 con 0x7fb930159cb0 2024-01-26T23:46:08.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.195+0000 7fb92f7fe700 1 --2- 172.21.15.46:0/227782685 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb91c066f80 0x7fb91c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:46:08.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.199+0000 7fb9177fe700 1 -- 172.21.15.46:0/227782685 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb928053660 con 0x7fb930159cb0 2024-01-26T23:46:08.203 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:08.202+0000 7f16b26b21c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:46:08.212 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:08.211+0000 7fd30e06d1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:46:08.288 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:08.286+0000 7fd30e06d1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:46:08.358 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:08.357+0000 7f16b26b21c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:46:08.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.396+0000 7fb92f7fe700 1 -- 172.21.15.46:0/227782685 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb91c066f80 msgr2=0x7fb91c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:08.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.396+0000 7fb92f7fe700 1 --2- 172.21.15.46:0/227782685 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb91c066f80 0x7fb91c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:46:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.406+0000 7fb93625a700 1 -- 172.21.15.46:0/227782685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fb93016a9c0 con 0x7fb930159cb0 2024-01-26T23:46:08.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.409+0000 7fb9177fe700 1 -- 172.21.15.46:0/227782685 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb928056c80 con 0x7fb930159cb0 2024-01-26T23:46:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.411+0000 7fb93625a700 1 -- 172.21.15.46:0/227782685 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb91c066f80 msgr2=0x7fb91c069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:46:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.411+0000 7fb93625a700 1 --2- 172.21.15.46:0/227782685 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb91c066f80 0x7fb91c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.411+0000 7fb93625a700 1 -- 172.21.15.46:0/227782685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb930159cb0 msgr2=0x7fb9300b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.411+0000 7fb93625a700 1 --2- 172.21.15.46:0/227782685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb930159cb0 0x7fb9300b5a30 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fb928015000 tx=0x7fb928018000 comp rx=0 tx=0).stop 2024-01-26T23:46:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.412+0000 7fb93625a700 1 -- 172.21.15.46:0/227782685 shutdown_connections 2024-01-26T23:46:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.412+0000 7fb93625a700 1 --2- 172.21.15.46:0/227782685 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fb91c066f80 0x7fb91c069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.412+0000 7fb93625a700 1 --2- 172.21.15.46:0/227782685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb930159cb0 0x7fb9300b5a30 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.412+0000 7fb93625a700 1 --2- 172.21.15.46:0/227782685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9300cf050 0x7fb9300b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.412+0000 7fb93625a700 1 --2- 172.21.15.46:0/227782685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9300cf4a0 0x7fb9300b0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.412+0000 7fb93625a700 1 -- 172.21.15.46:0/227782685 >> 172.21.15.46:0/227782685 conn(0x7fb930087f60 msgr2=0x7fb9301554b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.412+0000 7fb93625a700 1 -- 172.21.15.46:0/227782685 shutdown_connections 2024-01-26T23:46:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.412+0000 7fb93625a700 1 -- 172.21.15.46:0/227782685 wait complete. 2024-01-26T23:46:08.427 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-01-26T23:46:08.469 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:08.468+0000 7fd30e06d1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:46:08.497 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:08.496+0000 7f16b26b21c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:46:08.563 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:08.562+0000 7f16b26b21c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:46:08.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.562+0000 7efed3567700 1 Processor -- start 2024-01-26T23:46:08.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.562+0000 7efed3567700 1 -- start start 2024-01-26T23:46:08.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.563+0000 7efed3567700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efecc1599d0 0x7efecc159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:08.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.564+0000 7efed3567700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7efecc15d8c0 0x7efecc15dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.564+0000 7efed3567700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efecc0ab620 con 0x7efecc0ab1c0 2024-01-26T23:46:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.564+0000 7efed3567700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efecc15e260 con 0x7efecc1599d0 2024-01-26T23:46:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.564+0000 7efed3567700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efecc0b1a60 con 0x7efecc15d8c0 2024-01-26T23:46:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.564+0000 7efed1303700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efecc1599d0 0x7efecc159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.564+0000 7efed1303700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efecc1599d0 0x7efecc159da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58958/0 (socket says 172.21.15.46:58958) 2024-01-26T23:46:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.564+0000 7efed1303700 1 -- 172.21.15.46:0/193588467 learned_addr learned my addr 172.21.15.46:0/193588467 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.564+0000 7efed0b02700 1 --1- 172.21.15.46:0/193588467 >> v1:172.21.15.106:6789/0 conn(0x7efecc0ab1c0 0x7efecc0b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:55920/0 (socket says 172.21.15.46:55920) 2024-01-26T23:46:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.564+0000 7efed1b04700 1 --2- 172.21.15.46:0/193588467 >> v2:172.21.15.106:3300/0 conn(0x7efecc15d8c0 0x7efecc15dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.564+0000 7efed1303700 1 -- 172.21.15.46:0/193588467 >> v2:172.21.15.106:3300/0 conn(0x7efecc15d8c0 msgr2=0x7efecc15dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.564+0000 7efed1303700 1 --2- 172.21.15.46:0/193588467 >> v2:172.21.15.106:3300/0 conn(0x7efecc15d8c0 0x7efecc15dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.564+0000 7efed1303700 1 -- 172.21.15.46:0/193588467 >> v1:172.21.15.106:6789/0 conn(0x7efecc0ab1c0 legacy=0x7efecc0b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.564+0000 7efed1303700 1 -- 172.21.15.46:0/193588467 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efecc0b11c0 con 0x7efecc1599d0 2024-01-26T23:46:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.564+0000 7efed1303700 1 --2- 172.21.15.46:0/193588467 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efecc1599d0 0x7efecc159da0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7efecc068140 tx=0x7efec8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c61d6e1953dbf9ee server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.564+0000 7efec3fff700 1 -- 172.21.15.46:0/193588467 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec8005c10 con 0x7efecc1599d0 2024-01-26T23:46:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.565+0000 7efec3fff700 1 -- 172.21.15.46:0/193588467 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec8005d70 con 0x7efecc1599d0 2024-01-26T23:46:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.565+0000 7efec3fff700 1 -- 172.21.15.46:0/193588467 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec8006260 con 0x7efecc1599d0 2024-01-26T23:46:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.565+0000 7efed3567700 1 -- 172.21.15.46:0/193588467 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efecc1599d0 msgr2=0x7efecc159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.565+0000 7efed3567700 1 --2- 172.21.15.46:0/193588467 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efecc1599d0 0x7efecc159da0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7efecc068140 tx=0x7efec8009220 comp rx=0 tx=0).stop 2024-01-26T23:46:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.565+0000 7efed3567700 1 -- 172.21.15.46:0/193588467 shutdown_connections 2024-01-26T23:46:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.565+0000 7efed3567700 1 --2- 172.21.15.46:0/193588467 >> v2:172.21.15.106:3300/0 conn(0x7efecc15d8c0 0x7efecc15dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.565+0000 7efed3567700 1 --2- 172.21.15.46:0/193588467 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efecc1599d0 0x7efecc159da0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.565+0000 7efed3567700 1 -- 172.21.15.46:0/193588467 >> 172.21.15.46:0/193588467 conn(0x7efecc06f840 msgr2=0x7efecc087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.565+0000 7efed3567700 1 -- 172.21.15.46:0/193588467 shutdown_connections 2024-01-26T23:46:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.565+0000 7efed3567700 1 -- 172.21.15.46:0/193588467 wait complete. 2024-01-26T23:46:08.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.567+0000 7efed3567700 1 Processor -- start 2024-01-26T23:46:08.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.567+0000 7efed3567700 1 -- start start 2024-01-26T23:46:08.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.569+0000 7efed3567700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efecc0ab1c0 0x7efecc0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:08.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.569+0000 7efed3567700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efecc1599d0 0x7efecc0ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.569+0000 7efed3567700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efecc15d8c0 0x7efecc169e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.569+0000 7efed3567700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc09c4a0 con 0x7efecc15d8c0 2024-01-26T23:46:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.569+0000 7efed3567700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc09ca60 con 0x7efecc0ab1c0 2024-01-26T23:46:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.569+0000 7efed3567700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efecc16a360 con 0x7efecc1599d0 2024-01-26T23:46:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.569+0000 7efed1b04700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efecc15d8c0 0x7efecc169e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.569+0000 7efed1303700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efecc0ab1c0 0x7efecc0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.569+0000 7efed1b04700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efecc15d8c0 0x7efecc169e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58964/0 (socket says 172.21.15.46:58964) 2024-01-26T23:46:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.569+0000 7efed1b04700 1 -- 172.21.15.46:0/3011422863 learned_addr learned my addr 172.21.15.46:0/3011422863 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.569+0000 7efed0b02700 1 --2- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efecc1599d0 0x7efecc0ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.569+0000 7efed1b04700 1 -- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efecc1599d0 msgr2=0x7efecc0ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.569+0000 7efed1b04700 1 --2- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efecc1599d0 0x7efecc0ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.569+0000 7efed1b04700 1 -- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efecc0ab1c0 msgr2=0x7efecc0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.569+0000 7efed1b04700 1 --2- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efecc0ab1c0 0x7efecc0ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.569+0000 7efed1b04700 1 -- 172.21.15.46:0/3011422863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec8009480 con 0x7efecc15d8c0 2024-01-26T23:46:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.570+0000 7efed1b04700 1 --2- 172.21.15.46:0/3011422863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efecc15d8c0 0x7efecc169e20 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7efec4009df0 tx=0x7efec400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.570+0000 7efec17fa700 1 -- 172.21.15.46:0/3011422863 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec400caa0 con 0x7efecc15d8c0 2024-01-26T23:46:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.570+0000 7efec17fa700 1 -- 172.21.15.46:0/3011422863 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec400f040 con 0x7efecc15d8c0 2024-01-26T23:46:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.570+0000 7efed3567700 1 -- 172.21.15.46:0/3011422863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efecc158b20 con 0x7efecc15d8c0 2024-01-26T23:46:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.570+0000 7efec17fa700 1 -- 172.21.15.46:0/3011422863 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec400ce50 con 0x7efecc15d8c0 2024-01-26T23:46:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.570+0000 7efed3567700 1 -- 172.21.15.46:0/3011422863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efecc0ad1c0 con 0x7efecc15d8c0 2024-01-26T23:46:08.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.571+0000 7efed3567700 1 -- 172.21.15.46:0/3011422863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efecc09cc10 con 0x7efecc15d8c0 2024-01-26T23:46:08.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.571+0000 7efec17fa700 1 -- 172.21.15.46:0/3011422863 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 38) v1 ==== 84759+0+0 (secure 0 0 0) 0x7efec4014690 con 0x7efecc15d8c0 2024-01-26T23:46:08.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.572+0000 7efec17fa700 1 --2- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efeb0066f80 0x7efeb0069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:08.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.572+0000 7efec17fa700 1 -- 172.21.15.46:0/3011422863 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6460+0+0 (secure 0 0 0) 0x7efec4085310 con 0x7efecc15d8c0 2024-01-26T23:46:08.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.572+0000 7efed1303700 1 -- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efeb0066f80 msgr2=0x7efeb0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:08.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.572+0000 7efed1303700 1 --2- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efeb0066f80 0x7efeb0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:46:08.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.576+0000 7efec17fa700 1 -- 172.21.15.46:0/3011422863 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efecc09cc10 con 0x7efecc15d8c0 2024-01-26T23:46:08.630 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:08.628+0000 7fd30e06d1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:46:08.685 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:08.684+0000 7f16b26b21c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:46:08.707 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:08.706+0000 7fd30e06d1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:46:08.757 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:08.756+0000 7f16b26b21c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:46:08.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.773+0000 7efed1303700 1 -- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efeb0066f80 msgr2=0x7efeb0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:08.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.773+0000 7efed1303700 1 --2- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efeb0066f80 0x7efeb0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:46:08.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:08.809+0000 7efed3567700 1 -- 172.21.15.46:0/3011422863 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7efecc09cbd0 con 0x7efeb0066f80 2024-01-26T23:46:08.846 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:08.845+0000 7fd30e06d1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:46:08.896 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:08.895+0000 7f16b26b21c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:46:08.929 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:08.928+0000 7fd30e06d1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:46:09.025 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:09.024+0000 7f16b26b21c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:46:09.108 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:09.107+0000 7fd30e06d1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:46:09.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:09.174+0000 7efed1303700 1 -- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efeb0066f80 msgr2=0x7efeb0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:09.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:09.174+0000 7efed1303700 1 --2- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efeb0066f80 0x7efeb0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-26T23:46:09.250 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:09.249+0000 7fd30e06d1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:46:09.338 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:09.338+0000 7f16b26b21c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:46:09.549 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:09.548+0000 7f16b26b21c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:46:09.612 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:09.611+0000 7fd30e06d1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:46:09.774 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:09.773+0000 7f16b26b21c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:46:09.844 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:09.844+0000 7f16b26b21c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:46:09.852 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:09.851+0000 7fd30e06d1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:46:09.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:09.975+0000 7efed1303700 1 -- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efeb0066f80 msgr2=0x7efeb0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:09.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:09.975+0000 7efed1303700 1 --2- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efeb0066f80 0x7efeb0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-26T23:46:10.112 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:10.111+0000 7fd30e06d1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:46:10.193 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:10.191+0000 7fd30e06d1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:46:10.441 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:10.441+0000 7f16b26b21c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:46:10.515 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:10.514+0000 7f16b26b21c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:46:10.638 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:10.637+0000 7f16b26b21c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:46:10.887 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:10.886+0000 7fd30e06d1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:46:10.972 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:10.970+0000 7fd30e06d1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:46:11.112 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:11.111+0000 7fd30e06d1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:46:11.138 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:11.137+0000 7f16b26b21c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:46:11.287 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:11.286+0000 7f16b26b21c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:46:11.437 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:11.436+0000 7f16b26b21c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:46:11.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:11.577+0000 7efed1303700 1 -- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efeb0066f80 msgr2=0x7efeb0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:11.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:11.577+0000 7efed1303700 1 --2- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efeb0066f80 0x7efeb0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-26T23:46:11.698 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:11.697+0000 7fd30e06d1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:46:11.868 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:11.867+0000 7fd30e06d1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:46:12.039 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:12.038+0000 7fd30e06d1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:46:12.137 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:12.137+0000 7f16b26b21c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:46:12.354 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:12.353+0000 7f16b26b21c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:46:12.427 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:12.426+0000 7f16b26b21c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:46:12.583 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:12.582+0000 7f16b26b21c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:46:12.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:12.602+0000 7efec17fa700 1 -- 172.21.15.46:0/3011422863 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 39) v1 ==== 84470+0+0 (secure 0 0 0) 0x7efec4052660 con 0x7efecc15d8c0 2024-01-26T23:46:12.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:12.602+0000 7efec17fa700 1 -- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efeb0066f80 msgr2=0x7efeb0069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:46:12.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:12.602+0000 7efec17fa700 1 --2- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efeb0066f80 0x7efeb0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:12.707 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:12.706+0000 7f162bf56700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:12.707 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:12.706+0000 7f162bf56700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:12.708 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:12.706+0000 7f162bf56700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:12.708 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:12.706+0000 7f162bf56700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:12.708 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:12.706+0000 7f162bf56700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:12.708 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:12.706+0000 7f162874f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:12.708 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:12.706+0000 7f162874f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:12.708 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:12.706+0000 7f162874f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:12.708 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:12.706+0000 7f162874f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:12.708 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:12.706+0000 7f162874f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:12.863 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:12.862+0000 7fd30e06d1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:46:13.114 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:13.113+0000 7fd30e06d1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:46:13.198 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:13.196+0000 7fd30e06d1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:46:13.375 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:13.374+0000 7fd30e06d1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:46:13.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.605+0000 7efec17fa700 1 -- 172.21.15.46:0/3011422863 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 40) v1 ==== 84597+0+0 (secure 0 0 0) 0x7efec4024300 con 0x7efecc15d8c0 2024-01-26T23:46:13.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.605+0000 7efec17fa700 1 --2- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efeb0066f80 0x7efeb0069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:13.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.605+0000 7efec17fa700 1 -- 172.21.15.46:0/3011422863 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7efecc09cbd0 con 0x7efeb0066f80 2024-01-26T23:46:13.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.606+0000 7efed1303700 1 --2- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efeb0066f80 0x7efeb0069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:13.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.607+0000 7efed1303700 1 --2- 172.21.15.46:0/3011422863 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efeb0066f80 0x7efeb0069430 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7efec8009260 tx=0x7efec8016000 comp rx=0 tx=0).ready entity=mgr.6823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:13.618 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:13.617+0000 7f1632f64700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-01-26T23:46:13.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.617+0000 7efec17fa700 1 -- 172.21.15.46:0/3011422863 <== mgr.6823 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7efecc09cbd0 con 0x7efeb0066f80 2024-01-26T23:46:13.621 INFO:teuthology.orchestra.run.smithi046.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-01-26T23:46:13.625 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-26T23:46:13.625 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-26T23:46:13.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.807+0000 7f66d502c700 1 Processor -- start 2024-01-26T23:46:13.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.807+0000 7f66d502c700 1 -- start start 2024-01-26T23:46:13.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66d502c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f66d00c8da0 0x7f66d00c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:13.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66d502c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66d0140af0 0x7f66d0140f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:13.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66d502c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f66d00caf30 0x7f66d00cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:13.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66d502c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d00cbbf0 con 0x7f66d00caf30 2024-01-26T23:46:13.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66d502c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d00ccc00 con 0x7f66d00c8da0 2024-01-26T23:46:13.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66d502c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d00cc310 con 0x7f66d0140af0 2024-01-26T23:46:13.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66cf59e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f66d00caf30 0x7f66d00cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:13.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66cf59e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f66d00caf30 0x7f66d00cb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41086/0 (socket says 172.21.15.46:41086) 2024-01-26T23:46:13.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66cf59e700 1 -- 172.21.15.46:0/3824368545 learned_addr learned my addr 172.21.15.46:0/3824368545 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:13.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66ce59c700 1 --2- 172.21.15.46:0/3824368545 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66d0140af0 0x7f66d0140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:13.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66ced9d700 1 --2- 172.21.15.46:0/3824368545 >> v2:172.21.15.106:3300/0 conn(0x7f66d00c8da0 0x7f66d00c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:13.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66cf59e700 1 -- 172.21.15.46:0/3824368545 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66d0140af0 msgr2=0x7f66d0140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:13.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66cf59e700 1 --2- 172.21.15.46:0/3824368545 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66d0140af0 0x7f66d0140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:13.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66cf59e700 1 -- 172.21.15.46:0/3824368545 >> v2:172.21.15.106:3300/0 conn(0x7f66d00c8da0 msgr2=0x7f66d00c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:13.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66cf59e700 1 --2- 172.21.15.46:0/3824368545 >> v2:172.21.15.106:3300/0 conn(0x7f66d00c8da0 0x7f66d00c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:13.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66cf59e700 1 -- 172.21.15.46:0/3824368545 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d00cc5b0 con 0x7f66d00caf30 2024-01-26T23:46:13.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66cf59e700 1 --2- 172.21.15.46:0/3824368545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66d00caf30 0x7f66d00cb370 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f66d006f100 tx=0x7f66c8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1d6d5ba57b8450 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:13.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.808+0000 7f66cdd9b700 1 -- 172.21.15.46:0/3824368545 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c800e070 con 0x7f66d00caf30 2024-01-26T23:46:13.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.809+0000 7f66cdd9b700 1 -- 172.21.15.46:0/3824368545 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c800ba50 con 0x7f66d00caf30 2024-01-26T23:46:13.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.809+0000 7f66cdd9b700 1 -- 172.21.15.46:0/3824368545 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c80092e0 con 0x7f66d00caf30 2024-01-26T23:46:13.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.809+0000 7f66d502c700 1 -- 172.21.15.46:0/3824368545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66d00caf30 msgr2=0x7f66d00cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:13.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.809+0000 7f66d502c700 1 --2- 172.21.15.46:0/3824368545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66d00caf30 0x7f66d00cb370 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f66d006f100 tx=0x7f66c8009840 comp rx=0 tx=0).stop 2024-01-26T23:46:13.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.809+0000 7f66d502c700 1 -- 172.21.15.46:0/3824368545 shutdown_connections 2024-01-26T23:46:13.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.809+0000 7f66d502c700 1 --2- 172.21.15.46:0/3824368545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66d00caf30 0x7f66d00cb370 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:13.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.809+0000 7f66d502c700 1 --2- 172.21.15.46:0/3824368545 >> v2:172.21.15.106:3300/0 conn(0x7f66d00c8da0 0x7f66d00c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:13.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.809+0000 7f66d502c700 1 --2- 172.21.15.46:0/3824368545 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66d0140af0 0x7f66d0140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:13.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.809+0000 7f66d502c700 1 -- 172.21.15.46:0/3824368545 >> 172.21.15.46:0/3824368545 conn(0x7f66d006bd90 msgr2=0x7f66d006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:13.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.809+0000 7f66d502c700 1 -- 172.21.15.46:0/3824368545 shutdown_connections 2024-01-26T23:46:13.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.809+0000 7f66d502c700 1 -- 172.21.15.46:0/3824368545 wait complete. 2024-01-26T23:46:13.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.811+0000 7f66d502c700 1 Processor -- start 2024-01-26T23:46:13.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.811+0000 7f66d502c700 1 -- start start 2024-01-26T23:46:13.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.811+0000 7f66d502c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66d00c8da0 0x7f66d00c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:13.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.811+0000 7f66d502c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66d00caf30 0x7f66d00c6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:13.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.811+0000 7f66d502c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66d0140af0 0x7f66d00b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:13.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.811+0000 7f66d502c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d00b3180 con 0x7f66d00c8da0 2024-01-26T23:46:13.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.811+0000 7f66d502c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d00b28c0 con 0x7f66d00caf30 2024-01-26T23:46:13.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.811+0000 7f66d502c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d00b2b30 con 0x7f66d0140af0 2024-01-26T23:46:13.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.811+0000 7f66ced9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66d00c8da0 0x7f66d00c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:13.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.812+0000 7f66ced9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66d00c8da0 0x7f66d00c64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41092/0 (socket says 172.21.15.46:41092) 2024-01-26T23:46:13.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.812+0000 7f66ced9d700 1 -- 172.21.15.46:0/3323424814 learned_addr learned my addr 172.21.15.46:0/3323424814 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:13.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.812+0000 7f66ce59c700 1 --2- 172.21.15.46:0/3323424814 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66d00caf30 0x7f66d00c6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:13.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.812+0000 7f66cf59e700 1 --2- 172.21.15.46:0/3323424814 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66d0140af0 0x7f66d00b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:13.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.812+0000 7f66ced9d700 1 -- 172.21.15.46:0/3323424814 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66d0140af0 msgr2=0x7f66d00b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:13.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.812+0000 7f66ced9d700 1 --2- 172.21.15.46:0/3323424814 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66d0140af0 0x7f66d00b3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:13.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.812+0000 7f66ced9d700 1 -- 172.21.15.46:0/3323424814 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66d00caf30 msgr2=0x7f66d00c6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:13.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.812+0000 7f66ced9d700 1 --2- 172.21.15.46:0/3323424814 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66d00caf30 0x7f66d00c6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:13.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.812+0000 7f66ced9d700 1 -- 172.21.15.46:0/3323424814 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c8009ae0 con 0x7f66d00c8da0 2024-01-26T23:46:13.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.812+0000 7f66ced9d700 1 --2- 172.21.15.46:0/3323424814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66d00c8da0 0x7f66d00c64e0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f66b800d8c0 tx=0x7f66b8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:13.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.812+0000 7f66c6ffd700 1 -- 172.21.15.46:0/3323424814 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b800de60 con 0x7f66d00c8da0 2024-01-26T23:46:13.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.812+0000 7f66c6ffd700 1 -- 172.21.15.46:0/3323424814 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b800b780 con 0x7f66d00c8da0 2024-01-26T23:46:13.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.812+0000 7f66d502c700 1 -- 172.21.15.46:0/3323424814 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d00b2e10 con 0x7f66d00c8da0 2024-01-26T23:46:13.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.812+0000 7f66c6ffd700 1 -- 172.21.15.46:0/3323424814 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b800bb20 con 0x7f66d00c8da0 2024-01-26T23:46:13.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.812+0000 7f66d502c700 1 -- 172.21.15.46:0/3323424814 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d00c5700 con 0x7f66d00c8da0 2024-01-26T23:46:13.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.813+0000 7f66d502c700 1 -- 172.21.15.46:0/3323424814 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d004c710 con 0x7f66d00c8da0 2024-01-26T23:46:13.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.814+0000 7f66c6ffd700 1 -- 172.21.15.46:0/3323424814 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 40) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f66b801e460 con 0x7f66d00c8da0 2024-01-26T23:46:13.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.814+0000 7f66c6ffd700 1 --2- 172.21.15.46:0/3323424814 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f66b0066dd0 0x7f66b0069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:13.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.814+0000 7f66c6ffd700 1 -- 172.21.15.46:0/3323424814 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f66b8084cb0 con 0x7f66d00c8da0 2024-01-26T23:46:13.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.814+0000 7f66ce59c700 1 --2- 172.21.15.46:0/3323424814 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f66b0066dd0 0x7f66b0069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:13.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.815+0000 7f66ce59c700 1 --2- 172.21.15.46:0/3323424814 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f66b0066dd0 0x7f66b0069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f66c0006fd0 tx=0x7f66c0008040 comp rx=0 tx=0).ready entity=mgr.6823 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:13.819+0000 7f66c6ffd700 1 -- 172.21.15.46:0/3323424814 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f66b8051de0 con 0x7f66d00c8da0 2024-01-26T23:46:14.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:14.084+0000 7f66d502c700 1 -- 172.21.15.46:0/3323424814 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f66d00b3460 con 0x7f66d00c8da0 2024-01-26T23:46:14.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:14.648+0000 7f66c6ffd700 1 -- 172.21.15.46:0/3323424814 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 41) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f66b80521b0 con 0x7f66d00c8da0 2024-01-26T23:46:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:14.649+0000 7f66c6ffd700 1 -- 172.21.15.46:0/3323424814 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v41) v1 ==== 93+0+0 (secure 0 0 0) 0x7f66b8051de0 con 0x7f66d00c8da0 2024-01-26T23:46:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:14.652+0000 7f66d502c700 1 -- 172.21.15.46:0/3323424814 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f66b0066dd0 msgr2=0x7f66b0069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:14.652+0000 7f66d502c700 1 --2- 172.21.15.46:0/3323424814 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f66b0066dd0 0x7f66b0069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f66c0006fd0 tx=0x7f66c0008040 comp rx=0 tx=0).stop 2024-01-26T23:46:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:14.652+0000 7f66d502c700 1 -- 172.21.15.46:0/3323424814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66d00c8da0 msgr2=0x7f66d00c64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:14.652+0000 7f66d502c700 1 --2- 172.21.15.46:0/3323424814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66d00c8da0 0x7f66d00c64e0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f66b800d8c0 tx=0x7f66b8008040 comp rx=0 tx=0).stop 2024-01-26T23:46:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:14.652+0000 7f66d502c700 1 -- 172.21.15.46:0/3323424814 shutdown_connections 2024-01-26T23:46:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:14.652+0000 7f66d502c700 1 --2- 172.21.15.46:0/3323424814 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f66b0066dd0 0x7f66b0069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:14.652+0000 7f66d502c700 1 --2- 172.21.15.46:0/3323424814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66d00c8da0 0x7f66d00c64e0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:14.652+0000 7f66d502c700 1 --2- 172.21.15.46:0/3323424814 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66d0140af0 0x7f66d00b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:14.652+0000 7f66d502c700 1 --2- 172.21.15.46:0/3323424814 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66d00caf30 0x7f66d00c6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:14.652+0000 7f66d502c700 1 -- 172.21.15.46:0/3323424814 >> 172.21.15.46:0/3323424814 conn(0x7f66d006bd90 msgr2=0x7f66d00c3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:14.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:14.652+0000 7f66d502c700 1 -- 172.21.15.46:0/3323424814 shutdown_connections 2024-01-26T23:46:14.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:14.652+0000 7f66d502c700 1 -- 172.21.15.46:0/3323424814 wait complete. 2024-01-26T23:46:14.671 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-26T23:46:14.672 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:46:14.716 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:46:14.766 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-26T23:46:14.767 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-26T23:46:14.767 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-26T23:46:14.767 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-26T23:46:14.807 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[76245]: fuse finished with error 0 and tester_r 0 2024-01-26T23:46:14.808 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:46:14.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:46:14.809 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-26T23:46:14.941 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:46:14.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:46:14.943 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-26T23:46:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:46:14.959 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-26T23:46:14.960 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:46:14.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:46:14.960 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-26T23:46:14.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:46:14.978 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-26T23:46:14.978 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:46:14.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:46:14.978 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-26T23:46:14.996 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:14.995+0000 7fafd9abe1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:46:15.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:46:15.012 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-26T23:46:15.013 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:46:15.037 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:15.036+0000 7f84319161c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:46:15.075 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:15.074+0000 7fafd9abe1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:46:15.128 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:15.127+0000 7f84319161c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:46:15.145 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:15.144+0000 7fafd9abe1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:46:15.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.150+0000 7f24833f5700 1 Processor -- start 2024-01-26T23:46:15.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.151+0000 7f24833f5700 1 -- start start 2024-01-26T23:46:15.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.152+0000 7f24833f5700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f247c0b2b40 0x7f247c0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:15.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.152+0000 7f24833f5700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f247c0ae960 0x7f247c0ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:15.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.152+0000 7f24833f5700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247c15a0b0 con 0x7f247c0aedd0 2024-01-26T23:46:15.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.152+0000 7f24833f5700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f247c099fc0 con 0x7f247c0ae960 2024-01-26T23:46:15.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.152+0000 7f24833f5700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f247c0ae4d0 con 0x7f247c0b2b40 2024-01-26T23:46:15.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.152+0000 7f2480990700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f247c0ae960 0x7f247c0ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:15.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.152+0000 7f2480990700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f247c0ae960 0x7f247c0ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41120/0 (socket says 172.21.15.46:41120) 2024-01-26T23:46:15.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.152+0000 7f2480990700 1 -- 172.21.15.46:0/3212441442 learned_addr learned my addr 172.21.15.46:0/3212441442 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:15.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.152+0000 7f2481191700 1 --2- 172.21.15.46:0/3212441442 >> v2:172.21.15.106:3300/0 conn(0x7f247c0b2b40 0x7f247c0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:15.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.153+0000 7f2480990700 1 -- 172.21.15.46:0/3212441442 >> v2:172.21.15.106:3300/0 conn(0x7f247c0b2b40 msgr2=0x7f247c0ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:15.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.153+0000 7f2480990700 1 --2- 172.21.15.46:0/3212441442 >> v2:172.21.15.106:3300/0 conn(0x7f247c0b2b40 0x7f247c0ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.153+0000 7f2480990700 1 -- 172.21.15.46:0/3212441442 >> v1:172.21.15.106:6789/0 conn(0x7f247c0aedd0 legacy=0x7f247c0cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:15.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.153+0000 7f2480990700 1 -- 172.21.15.46:0/3212441442 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c0ad8c0 con 0x7f247c0ae960 2024-01-26T23:46:15.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.153+0000 7f2480990700 1 --2- 172.21.15.46:0/3212441442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f247c0ae960 0x7f247c0ced50 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f247c088850 tx=0x7f2470009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1f83d08b917389 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:15.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.153+0000 7f247bfff700 1 -- 172.21.15.46:0/3212441442 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2470009e70 con 0x7f247c0ae960 2024-01-26T23:46:15.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.153+0000 7f247bfff700 1 -- 172.21.15.46:0/3212441442 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247000b860 con 0x7f247c0ae960 2024-01-26T23:46:15.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.153+0000 7f247bfff700 1 -- 172.21.15.46:0/3212441442 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247000bc00 con 0x7f247c0ae960 2024-01-26T23:46:15.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.154+0000 7f24833f5700 1 -- 172.21.15.46:0/3212441442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f247c0ae960 msgr2=0x7f247c0ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:15.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.154+0000 7f24833f5700 1 --2- 172.21.15.46:0/3212441442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f247c0ae960 0x7f247c0ced50 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f247c088850 tx=0x7f2470009250 comp rx=0 tx=0).stop 2024-01-26T23:46:15.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.154+0000 7f24833f5700 1 -- 172.21.15.46:0/3212441442 shutdown_connections 2024-01-26T23:46:15.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.154+0000 7f24833f5700 1 --2- 172.21.15.46:0/3212441442 >> v2:172.21.15.106:3300/0 conn(0x7f247c0b2b40 0x7f247c0ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.154+0000 7f24833f5700 1 --2- 172.21.15.46:0/3212441442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f247c0ae960 0x7f247c0ced50 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.154+0000 7f24833f5700 1 -- 172.21.15.46:0/3212441442 >> 172.21.15.46:0/3212441442 conn(0x7f247c099a30 msgr2=0x7f247c06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:15.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.154+0000 7f24833f5700 1 -- 172.21.15.46:0/3212441442 shutdown_connections 2024-01-26T23:46:15.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.154+0000 7f24833f5700 1 -- 172.21.15.46:0/3212441442 wait complete. 2024-01-26T23:46:15.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.156+0000 7f24833f5700 1 Processor -- start 2024-01-26T23:46:15.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.156+0000 7f24833f5700 1 -- start start 2024-01-26T23:46:15.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.157+0000 7f24833f5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f247c0ae960 0x7f247c0c9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:15.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.157+0000 7f24833f5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f247c0b2b40 0x7f247c069640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:15.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.157+0000 7f24833f5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f247c151360 0x7f247c1517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:15.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.157+0000 7f24833f5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c151d20 con 0x7f247c0b2b40 2024-01-26T23:46:15.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.157+0000 7f24833f5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c0c7820 con 0x7f247c0ae960 2024-01-26T23:46:15.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.157+0000 7f24833f5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247c0b3510 con 0x7f247c151360 2024-01-26T23:46:15.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.158+0000 7f2480990700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f247c0b2b40 0x7f247c069640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:15.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.158+0000 7f2480990700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f247c0b2b40 0x7f247c069640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41130/0 (socket says 172.21.15.46:41130) 2024-01-26T23:46:15.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.158+0000 7f2480990700 1 -- 172.21.15.46:0/1129254043 learned_addr learned my addr 172.21.15.46:0/1129254043 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:15.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.158+0000 7f2481191700 1 --2- 172.21.15.46:0/1129254043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f247c0ae960 0x7f247c0c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:15.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.158+0000 7f2481992700 1 --2- 172.21.15.46:0/1129254043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f247c151360 0x7f247c1517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:15.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.158+0000 7f2480990700 1 -- 172.21.15.46:0/1129254043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f247c151360 msgr2=0x7f247c1517e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:15.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.158+0000 7f2480990700 1 --2- 172.21.15.46:0/1129254043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f247c151360 0x7f247c1517e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.158+0000 7f2480990700 1 -- 172.21.15.46:0/1129254043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f247c0ae960 msgr2=0x7f247c0c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:15.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.158+0000 7f2480990700 1 --2- 172.21.15.46:0/1129254043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f247c0ae960 0x7f247c0c9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.158+0000 7f2480990700 1 -- 172.21.15.46:0/1129254043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24700094f0 con 0x7f247c0b2b40 2024-01-26T23:46:15.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.158+0000 7f2481191700 1 --2- 172.21.15.46:0/1129254043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f247c0ae960 0x7f247c0c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:15.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.158+0000 7f2480990700 1 --2- 172.21.15.46:0/1129254043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f247c0b2b40 0x7f247c069640 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2470006fd0 tx=0x7f2470017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:15.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.158+0000 7f24797fa700 1 -- 172.21.15.46:0/1129254043 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247000bbe0 con 0x7f247c0b2b40 2024-01-26T23:46:15.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.158+0000 7f24797fa700 1 -- 172.21.15.46:0/1129254043 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2470011420 con 0x7f247c0b2b40 2024-01-26T23:46:15.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.158+0000 7f24797fa700 1 -- 172.21.15.46:0/1129254043 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24700117e0 con 0x7f247c0b2b40 2024-01-26T23:46:15.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.158+0000 7f24833f5700 1 -- 172.21.15.46:0/1129254043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247c0c98b0 con 0x7f247c0b2b40 2024-01-26T23:46:15.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.158+0000 7f24833f5700 1 -- 172.21.15.46:0/1129254043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247c0b3a40 con 0x7f247c0b2b40 2024-01-26T23:46:15.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.160+0000 7f24833f5700 1 -- 172.21.15.46:0/1129254043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247c04c710 con 0x7f247c0b2b40 2024-01-26T23:46:15.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.160+0000 7f24797fa700 1 -- 172.21.15.46:0/1129254043 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 41) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f2470011bd0 con 0x7f247c0b2b40 2024-01-26T23:46:15.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.160+0000 7f24797fa700 1 --2- 172.21.15.46:0/1129254043 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2468067050 0x7f2468069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:15.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.160+0000 7f24797fa700 1 -- 172.21.15.46:0/1129254043 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f2470086200 con 0x7f247c0b2b40 2024-01-26T23:46:15.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.160+0000 7f2481191700 1 -- 172.21.15.46:0/1129254043 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2468067050 msgr2=0x7f2468069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:15.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.161+0000 7f2481191700 1 --2- 172.21.15.46:0/1129254043 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2468067050 0x7f2468069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:46:15.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.165+0000 7f24797fa700 1 -- 172.21.15.46:0/1129254043 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2470052f70 con 0x7f247c0b2b40 2024-01-26T23:46:15.212 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:15.211+0000 7fafd9abe1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:46:15.215 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:15.214+0000 7f84319161c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:46:15.291 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:15.290+0000 7f84319161c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:46:15.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.362+0000 7f2481191700 1 -- 172.21.15.46:0/1129254043 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2468067050 msgr2=0x7f2468069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:15.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.362+0000 7f2481191700 1 --2- 172.21.15.46:0/1129254043 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2468067050 0x7f2468069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:46:15.366 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:15.365+0000 7fafd9abe1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:46:15.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.419+0000 7f24833f5700 1 -- 172.21.15.46:0/1129254043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f247c0682c0 con 0x7f247c0b2b40 2024-01-26T23:46:15.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.419+0000 7f24797fa700 1 -- 172.21.15.46:0/1129254043 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f2470056590 con 0x7f247c0b2b40 2024-01-26T23:46:15.421 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:15.421 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6137,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/1370365478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1370365478},{"type":"v1","addr":"172.21.15.106:6836","nonce":1370365478}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":6619,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/1980309094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1980309094},{"type":"v1","addr":"172.21.15.106:6837","nonce":1980309094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40},{"gid":6628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3910368747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3910368747},{"type":"v1","addr":"172.21.15.46:6834","nonce":3910368747}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:45:52.426977+0000","modified":"2024-01-26T23:45:56.004421+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6631":{"gid":6631,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3176096333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3176096333},{"type":"v1","addr":"172.21.15.46:6835","nonce":3176096333}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2024-01-26T23:46:15.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.421+0000 7f24833f5700 1 -- 172.21.15.46:0/1129254043 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2468067050 msgr2=0x7f2468069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:46:15.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.421+0000 7f24833f5700 1 --2- 172.21.15.46:0/1129254043 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2468067050 0x7f2468069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.421+0000 7f24833f5700 1 -- 172.21.15.46:0/1129254043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f247c0b2b40 msgr2=0x7f247c069640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:15.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.421+0000 7f24833f5700 1 --2- 172.21.15.46:0/1129254043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f247c0b2b40 0x7f247c069640 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2470006fd0 tx=0x7f2470017000 comp rx=0 tx=0).stop 2024-01-26T23:46:15.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.421+0000 7f24833f5700 1 -- 172.21.15.46:0/1129254043 shutdown_connections 2024-01-26T23:46:15.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.421+0000 7f24833f5700 1 --2- 172.21.15.46:0/1129254043 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2468067050 0x7f2468069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.421+0000 7f24833f5700 1 --2- 172.21.15.46:0/1129254043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f247c151360 0x7f247c1517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.421+0000 7f24833f5700 1 --2- 172.21.15.46:0/1129254043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f247c0ae960 0x7f247c0c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.421+0000 7f24833f5700 1 --2- 172.21.15.46:0/1129254043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f247c0b2b40 0x7f247c069640 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.421+0000 7f24833f5700 1 -- 172.21.15.46:0/1129254043 >> 172.21.15.46:0/1129254043 conn(0x7f247c099a30 msgr2=0x7f247c158f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:15.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.422+0000 7f24833f5700 1 -- 172.21.15.46:0/1129254043 shutdown_connections 2024-01-26T23:46:15.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.422+0000 7f24833f5700 1 -- 172.21.15.46:0/1129254043 wait complete. 2024-01-26T23:46:15.424 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 45 2024-01-26T23:46:15.438 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:46:15.470 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:15.468+0000 7f84319161c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:46:15.502 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:15.501+0000 7fafd9abe1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:46:15.568 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:15.567+0000 7fafd9abe1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:46:15.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.601+0000 7f0c2d24a700 1 Processor -- start 2024-01-26T23:46:15.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.601+0000 7f0c2d24a700 1 -- start start 2024-01-26T23:46:15.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.602+0000 7f0c2d24a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0c28157340 0x7f0c2815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:15.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.602+0000 7f0c2d24a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c280b4180 0x7f0c28157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:15.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.602+0000 7f0c2d24a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c28158ae0 con 0x7f0c280ae520 2024-01-26T23:46:15.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.602+0000 7f0c2d24a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c2815b650 con 0x7f0c28157340 2024-01-26T23:46:15.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.602+0000 7f0c2d24a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2815c020 con 0x7f0c280b4180 2024-01-26T23:46:15.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.602+0000 7f0c2659c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0c280ae520 0x7f0c280b3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59624/0 (socket says 172.21.15.46:59624) 2024-01-26T23:46:15.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.602+0000 7f0c2659c700 1 -- 172.21.15.46:0/3448183473 learned_addr learned my addr 172.21.15.46:0/3448183473 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:15.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.602+0000 7f0c26d9d700 1 --2- 172.21.15.46:0/3448183473 >> v2:172.21.15.106:3300/0 conn(0x7f0c28157340 0x7f0c2815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:15.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.602+0000 7f0c2759e700 1 --2- 172.21.15.46:0/3448183473 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c280b4180 0x7f0c28157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:15.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.602+0000 7f0c25d9b700 1 -- 172.21.15.46:0/3448183473 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462747790 0 0) 0x7f0c28158ae0 con 0x7f0c280ae520 2024-01-26T23:46:15.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.602+0000 7f0c25d9b700 1 -- 172.21.15.46:0/3448183473 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c10003570 con 0x7f0c280ae520 2024-01-26T23:46:15.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.603+0000 7f0c26d9d700 1 -- 172.21.15.46:0/3448183473 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c280b4180 msgr2=0x7f0c28157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:15.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.603+0000 7f0c26d9d700 1 --2- 172.21.15.46:0/3448183473 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c280b4180 0x7f0c28157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.603+0000 7f0c26d9d700 1 -- 172.21.15.46:0/3448183473 >> v1:172.21.15.46:6789/0 conn(0x7f0c280ae520 legacy=0x7f0c280b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:15.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.603+0000 7f0c26d9d700 1 -- 172.21.15.46:0/3448183473 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c28158520 con 0x7f0c28157340 2024-01-26T23:46:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.603+0000 7f0c2759e700 1 --2- 172.21.15.46:0/3448183473 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c280b4180 0x7f0c28157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.603+0000 7f0c26d9d700 1 --2- 172.21.15.46:0/3448183473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c28157340 0x7f0c2815bae0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0c180098e0 tx=0x7f0c180091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=672d3ab5781f8b0b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.603+0000 7f0c25d9b700 1 -- 172.21.15.46:0/3448183473 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c18011630 con 0x7f0c28157340 2024-01-26T23:46:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.603+0000 7f0c25d9b700 1 -- 172.21.15.46:0/3448183473 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c18011790 con 0x7f0c28157340 2024-01-26T23:46:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.603+0000 7f0c25d9b700 1 -- 172.21.15.46:0/3448183473 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c18011b30 con 0x7f0c28157340 2024-01-26T23:46:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.604+0000 7f0c2d24a700 1 -- 172.21.15.46:0/3448183473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c28157340 msgr2=0x7f0c2815bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.604+0000 7f0c2d24a700 1 --2- 172.21.15.46:0/3448183473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c28157340 0x7f0c2815bae0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0c180098e0 tx=0x7f0c180091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.604+0000 7f0c2d24a700 1 -- 172.21.15.46:0/3448183473 shutdown_connections 2024-01-26T23:46:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.604+0000 7f0c2d24a700 1 --2- 172.21.15.46:0/3448183473 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c280b4180 0x7f0c28157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.604+0000 7f0c2d24a700 1 --2- 172.21.15.46:0/3448183473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c28157340 0x7f0c2815bae0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.604+0000 7f0c2d24a700 1 -- 172.21.15.46:0/3448183473 >> 172.21.15.46:0/3448183473 conn(0x7f0c280ce530 msgr2=0x7f0c280ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.604+0000 7f0c2d24a700 1 -- 172.21.15.46:0/3448183473 shutdown_connections 2024-01-26T23:46:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.604+0000 7f0c2d24a700 1 -- 172.21.15.46:0/3448183473 wait complete. 2024-01-26T23:46:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.606+0000 7f0c2d24a700 1 Processor -- start 2024-01-26T23:46:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.606+0000 7f0c2d24a700 1 -- start start 2024-01-26T23:46:15.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.607+0000 7f0c2d24a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c280ae520 0x7f0c280c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.607+0000 7f0c2d24a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c280b4180 0x7f0c281596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.607+0000 7f0c2d24a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c28157340 0x7f0c28155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.607+0000 7f0c2d24a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c28159c20 con 0x7f0c280ae520 2024-01-26T23:46:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.607+0000 7f0c2d24a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c280c5bb0 con 0x7f0c280b4180 2024-01-26T23:46:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.607+0000 7f0c2d24a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c280c5e90 con 0x7f0c28157340 2024-01-26T23:46:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.607+0000 7f0c26d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c280ae520 0x7f0c280c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.607+0000 7f0c26d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c280ae520 0x7f0c280c5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41132/0 (socket says 172.21.15.46:41132) 2024-01-26T23:46:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.607+0000 7f0c26d9d700 1 -- 172.21.15.46:0/2225844890 learned_addr learned my addr 172.21.15.46:0/2225844890 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.607+0000 7f0c2759e700 1 --2- 172.21.15.46:0/2225844890 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c28157340 0x7f0c28155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.607+0000 7f0c2659c700 1 --2- 172.21.15.46:0/2225844890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c280b4180 0x7f0c281596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.607+0000 7f0c26d9d700 1 -- 172.21.15.46:0/2225844890 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c28157340 msgr2=0x7f0c28155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.607+0000 7f0c26d9d700 1 --2- 172.21.15.46:0/2225844890 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c28157340 0x7f0c28155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.607+0000 7f0c26d9d700 1 -- 172.21.15.46:0/2225844890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c280b4180 msgr2=0x7f0c281596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:15.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.607+0000 7f0c26d9d700 1 --2- 172.21.15.46:0/2225844890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c280b4180 0x7f0c281596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.607+0000 7f0c26d9d700 1 -- 172.21.15.46:0/2225844890 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c20009a20 con 0x7f0c280ae520 2024-01-26T23:46:15.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.607+0000 7f0c26d9d700 1 --2- 172.21.15.46:0/2225844890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c280ae520 0x7f0c280c5540 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0c18009ed0 tx=0x7f0c18017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:15.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.608+0000 7f0c0effd700 1 -- 172.21.15.46:0/2225844890 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c18011900 con 0x7f0c280ae520 2024-01-26T23:46:15.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.608+0000 7f0c0effd700 1 -- 172.21.15.46:0/2225844890 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c18011aa0 con 0x7f0c280ae520 2024-01-26T23:46:15.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.608+0000 7f0c2d24a700 1 -- 172.21.15.46:0/2225844890 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c18009420 con 0x7f0c280ae520 2024-01-26T23:46:15.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.608+0000 7f0c0effd700 1 -- 172.21.15.46:0/2225844890 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c18025640 con 0x7f0c280ae520 2024-01-26T23:46:15.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.608+0000 7f0c2d24a700 1 -- 172.21.15.46:0/2225844890 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c280c6450 con 0x7f0c280ae520 2024-01-26T23:46:15.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.609+0000 7f0c2d24a700 1 -- 172.21.15.46:0/2225844890 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c2804c710 con 0x7f0c280ae520 2024-01-26T23:46:15.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.609+0000 7f0c0effd700 1 -- 172.21.15.46:0/2225844890 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 41) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f0c18025a30 con 0x7f0c280ae520 2024-01-26T23:46:15.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.610+0000 7f0c0effd700 1 --2- 172.21.15.46:0/2225844890 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c14067050 0x7f0c14069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:15.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.610+0000 7f0c2659c700 1 -- 172.21.15.46:0/2225844890 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c14067050 msgr2=0x7f0c14069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:15.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.610+0000 7f0c0effd700 1 -- 172.21.15.46:0/2225844890 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f0c18086080 con 0x7f0c280ae520 2024-01-26T23:46:15.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.610+0000 7f0c2659c700 1 --2- 172.21.15.46:0/2225844890 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c14067050 0x7f0c14069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:46:15.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.614+0000 7f0c0effd700 1 -- 172.21.15.46:0/2225844890 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0c180b75a0 con 0x7f0c280ae520 2024-01-26T23:46:15.633 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:15.632+0000 7f84319161c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:46:15.689 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:15.689+0000 7fafd9abe1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:46:15.711 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:15.710+0000 7f84319161c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:46:15.761 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:15.760+0000 7fafd9abe1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:46:15.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.811+0000 7f0c2659c700 1 -- 172.21.15.46:0/2225844890 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c14067050 msgr2=0x7f0c14069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:15.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.811+0000 7f0c2659c700 1 --2- 172.21.15.46:0/2225844890 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c14067050 0x7f0c14069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:46:15.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.821+0000 7f0c2d24a700 1 -- 172.21.15.46:0/2225844890 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c28158800 con 0x7f0c280ae520 2024-01-26T23:46:15.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.822+0000 7f0c0effd700 1 -- 172.21.15.46:0/2225844890 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16587 (secure 0 0 0) 0x7f0c18015090 con 0x7f0c280ae520 2024-01-26T23:46:15.824 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:15.824 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":100,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:46:12.597656+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:45:50.922223+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-01-26T23:45:51.397995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/2701612997":"2024-01-27T23:46:12.597621+0000","172.21.15.106:0/1894478014":"2024-01-27T23:46:12.597621+0000","172.21.15.106:0/623871419":"2024-01-27T23:46:12.597621+0000","172.21.15.106:6833/43388":"2024-01-27T23:46:12.597621+0000","172.21.15.106:6832/43388":"2024-01-27T23:46:12.597621+0000","172.21.15.106:0/243105601":"2024-01-27T23:46:12.597621+0000","172.21.15.106:0/3064987359":"2024-01-27T00:46:05.889834+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:46:15.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.824+0000 7f0c2d24a700 1 -- 172.21.15.46:0/2225844890 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c14067050 msgr2=0x7f0c14069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:46:15.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.824+0000 7f0c2d24a700 1 --2- 172.21.15.46:0/2225844890 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c14067050 0x7f0c14069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.824+0000 7f0c2d24a700 1 -- 172.21.15.46:0/2225844890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c280ae520 msgr2=0x7f0c280c5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:15.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.824+0000 7f0c2d24a700 1 --2- 172.21.15.46:0/2225844890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c280ae520 0x7f0c280c5540 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0c18009ed0 tx=0x7f0c18017000 comp rx=0 tx=0).stop 2024-01-26T23:46:15.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.824+0000 7f0c2d24a700 1 -- 172.21.15.46:0/2225844890 shutdown_connections 2024-01-26T23:46:15.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.824+0000 7f0c2d24a700 1 --2- 172.21.15.46:0/2225844890 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c14067050 0x7f0c14069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.824+0000 7f0c2d24a700 1 --2- 172.21.15.46:0/2225844890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c280ae520 0x7f0c280c5540 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.824+0000 7f0c2d24a700 1 --2- 172.21.15.46:0/2225844890 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c28157340 0x7f0c28155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.824+0000 7f0c2d24a700 1 --2- 172.21.15.46:0/2225844890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c280b4180 0x7f0c281596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.824+0000 7f0c2d24a700 1 -- 172.21.15.46:0/2225844890 >> 172.21.15.46:0/2225844890 conn(0x7f0c280ce530 msgr2=0x7f0c280ade00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:15.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.825+0000 7f0c2d24a700 1 -- 172.21.15.46:0/2225844890 shutdown_connections 2024-01-26T23:46:15.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.825+0000 7f0c2d24a700 1 -- 172.21.15.46:0/2225844890 wait complete. 2024-01-26T23:46:15.841 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-26T23:45:50.922223+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-01-26T23:45:51.397995+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-26T23:46:15.841 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:46:15.855 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:15.854+0000 7f84319161c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:46:15.902 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:15.901+0000 7fafd9abe1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:46:15.941 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:15.940+0000 7f84319161c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:46:15.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.977+0000 7f8ac8ca9700 1 Processor -- start 2024-01-26T23:46:15.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.977+0000 7f8ac8ca9700 1 -- start start 2024-01-26T23:46:15.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.978+0000 7f8ac8ca9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8ac40afb40 0x7f8ac40c5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:15.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.978+0000 7f8ac8ca9700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac40cc100 con 0x7f8ac40cf050 2024-01-26T23:46:15.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.978+0000 7f8ac8ca9700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac40cb850 con 0x7f8ac4159cb0 2024-01-26T23:46:15.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.978+0000 7f8ac8ca9700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac40ccca0 con 0x7f8ac40afb40 2024-01-26T23:46:15.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.979+0000 7f8ac1d9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8ac40cf050 0x7f8ac40af430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59634/0 (socket says 172.21.15.46:59634) 2024-01-26T23:46:15.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.979+0000 7f8ac1d9b700 1 -- 172.21.15.46:0/2780670537 learned_addr learned my addr 172.21.15.46:0/2780670537 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:15.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.979+0000 7f8ac2d9d700 1 --2- 172.21.15.46:0/2780670537 >> v2:172.21.15.106:3300/0 conn(0x7f8ac40afb40 0x7f8ac40c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:15.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.979+0000 7f8ac159a700 1 -- 172.21.15.46:0/2780670537 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2857187262 0 0) 0x7f8ac40cc100 con 0x7f8ac40cf050 2024-01-26T23:46:15.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.979+0000 7f8ac159a700 1 -- 172.21.15.46:0/2780670537 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aac003570 con 0x7f8ac40cf050 2024-01-26T23:46:15.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.979+0000 7f8ac159a700 1 -- 172.21.15.46:0/2780670537 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792188474 0 0) 0x7f8ac40cb850 con 0x7f8ac4159cb0 2024-01-26T23:46:15.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.979+0000 7f8ac159a700 1 -- 172.21.15.46:0/2780670537 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac40cc100 con 0x7f8ac4159cb0 2024-01-26T23:46:15.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.979+0000 7f8ac2d9d700 1 -- 172.21.15.46:0/2780670537 >> v1:172.21.15.106:6789/0 conn(0x7f8ac4159cb0 legacy=0x7f8ac40cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:15.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.979+0000 7f8ac2d9d700 1 -- 172.21.15.46:0/2780670537 >> v1:172.21.15.46:6789/0 conn(0x7f8ac40cf050 legacy=0x7f8ac40af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:15.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.979+0000 7f8ac2d9d700 1 -- 172.21.15.46:0/2780670537 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac40cbaf0 con 0x7f8ac40afb40 2024-01-26T23:46:15.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.979+0000 7f8ac2d9d700 1 --2- 172.21.15.46:0/2780670537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ac40afb40 0x7f8ac40c5530 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8abc0098e0 tx=0x7f8abc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ffbb7778da7e056 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:15.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.980+0000 7f8ac159a700 1 -- 172.21.15.46:0/2780670537 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc011630 con 0x7f8ac40afb40 2024-01-26T23:46:15.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.980+0000 7f8ac159a700 1 -- 172.21.15.46:0/2780670537 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8abc011790 con 0x7f8ac40afb40 2024-01-26T23:46:15.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.980+0000 7f8ac159a700 1 -- 172.21.15.46:0/2780670537 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc011b30 con 0x7f8ac40afb40 2024-01-26T23:46:15.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.980+0000 7f8ac8ca9700 1 -- 172.21.15.46:0/2780670537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ac40afb40 msgr2=0x7f8ac40c5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:15.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.980+0000 7f8ac8ca9700 1 --2- 172.21.15.46:0/2780670537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ac40afb40 0x7f8ac40c5530 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8abc0098e0 tx=0x7f8abc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.980+0000 7f8ac8ca9700 1 -- 172.21.15.46:0/2780670537 shutdown_connections 2024-01-26T23:46:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.981+0000 7f8ac8ca9700 1 --2- 172.21.15.46:0/2780670537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ac40afb40 0x7f8ac40c5530 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.981+0000 7f8ac8ca9700 1 -- 172.21.15.46:0/2780670537 >> 172.21.15.46:0/2780670537 conn(0x7f8ac4087f60 msgr2=0x7f8ac406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.981+0000 7f8ac8ca9700 1 -- 172.21.15.46:0/2780670537 shutdown_connections 2024-01-26T23:46:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.981+0000 7f8ac8ca9700 1 -- 172.21.15.46:0/2780670537 wait complete. 2024-01-26T23:46:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.982+0000 7f8ac8ca9700 1 Processor -- start 2024-01-26T23:46:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.982+0000 7f8ac8ca9700 1 -- start start 2024-01-26T23:46:15.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.983+0000 7f8ac8ca9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ac40afb40 0x7f8ac40ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:15.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.983+0000 7f8ac8ca9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ac40cf050 0x7f8ac4153710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:15.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.983+0000 7f8ac8ca9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ac4159cb0 0x7f8ac40b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:15.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.983+0000 7f8ac8ca9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac40b1de0 con 0x7f8ac4159cb0 2024-01-26T23:46:15.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.983+0000 7f8ac8ca9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac40b1b00 con 0x7f8ac40cf050 2024-01-26T23:46:15.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.983+0000 7f8ac8ca9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac40c7a70 con 0x7f8ac40afb40 2024-01-26T23:46:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.983+0000 7f8ac2d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ac4159cb0 0x7f8ac40b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.983+0000 7f8ac2d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ac4159cb0 0x7f8ac40b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41140/0 (socket says 172.21.15.46:41140) 2024-01-26T23:46:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.983+0000 7f8ac2d9d700 1 -- 172.21.15.46:0/3637925853 learned_addr learned my addr 172.21.15.46:0/3637925853 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.983+0000 7f8ac1d9b700 1 --2- 172.21.15.46:0/3637925853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ac40cf050 0x7f8ac4153710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.983+0000 7f8ac259c700 1 --2- 172.21.15.46:0/3637925853 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ac40afb40 0x7f8ac40ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.983+0000 7f8ac1d9b700 1 -- 172.21.15.46:0/3637925853 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ac40afb40 msgr2=0x7f8ac40ca320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.983+0000 7f8ac1d9b700 1 --2- 172.21.15.46:0/3637925853 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ac40afb40 0x7f8ac40ca320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.983+0000 7f8ac1d9b700 1 -- 172.21.15.46:0/3637925853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ac4159cb0 msgr2=0x7f8ac40b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.983+0000 7f8ac1d9b700 1 --2- 172.21.15.46:0/3637925853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ac4159cb0 0x7f8ac40b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.983+0000 7f8ac1d9b700 1 -- 172.21.15.46:0/3637925853 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8abc009420 con 0x7f8ac40cf050 2024-01-26T23:46:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.983+0000 7f8ac1d9b700 1 --2- 172.21.15.46:0/3637925853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ac40cf050 0x7f8ac4153710 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8ab800c580 tx=0x7f8ab8007d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.984+0000 7f8ab27fc700 1 -- 172.21.15.46:0/3637925853 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab8005b30 con 0x7f8ac40cf050 2024-01-26T23:46:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.984+0000 7f8ab27fc700 1 -- 172.21.15.46:0/3637925853 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ab8005c90 con 0x7f8ac40cf050 2024-01-26T23:46:15.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.984+0000 7f8ac8ca9700 1 -- 172.21.15.46:0/3637925853 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac40c7d50 con 0x7f8ac40cf050 2024-01-26T23:46:15.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.984+0000 7f8ab27fc700 1 -- 172.21.15.46:0/3637925853 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab8007320 con 0x7f8ac40cf050 2024-01-26T23:46:15.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.984+0000 7f8ac8ca9700 1 -- 172.21.15.46:0/3637925853 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac4153c50 con 0x7f8ac40cf050 2024-01-26T23:46:15.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.985+0000 7f8ac8ca9700 1 -- 172.21.15.46:0/3637925853 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac404c710 con 0x7f8ac40cf050 2024-01-26T23:46:15.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.985+0000 7f8ab27fc700 1 -- 172.21.15.46:0/3637925853 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 41) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f8ab801b3b0 con 0x7f8ac40cf050 2024-01-26T23:46:15.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.986+0000 7f8ab27fc700 1 --2- 172.21.15.46:0/3637925853 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8aa4067050 0x7f8aa4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:15.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.986+0000 7f8ac259c700 1 -- 172.21.15.46:0/3637925853 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8aa4067050 msgr2=0x7f8aa4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:15.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.986+0000 7f8ab27fc700 1 -- 172.21.15.46:0/3637925853 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f8ab8083ea0 con 0x7f8ac40cf050 2024-01-26T23:46:15.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.986+0000 7f8ac259c700 1 --2- 172.21.15.46:0/3637925853 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8aa4067050 0x7f8aa4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:46:15.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:15.990+0000 7f8ab27fc700 1 -- 172.21.15.46:0/3637925853 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8ab8050c10 con 0x7f8ac40cf050 2024-01-26T23:46:16.031 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:16.031+0000 7fafd9abe1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:46:16.136 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:16.135+0000 7f84319161c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:46:16.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.187+0000 7f8ac259c700 1 -- 172.21.15.46:0/3637925853 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8aa4067050 msgr2=0x7f8aa4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:16.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.187+0000 7f8ac259c700 1 --2- 172.21.15.46:0/3637925853 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8aa4067050 0x7f8aa4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:46:16.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.264+0000 7f8ac8ca9700 1 -- 172.21.15.46:0/3637925853 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8ac40cfc80 con 0x7f8ac40cf050 2024-01-26T23:46:16.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.265+0000 7f8ab27fc700 1 -- 172.21.15.46:0/3637925853 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8ab8017020 con 0x7f8ac40cf050 2024-01-26T23:46:16.266 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:46:16.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.267+0000 7f8ac8ca9700 1 -- 172.21.15.46:0/3637925853 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8aa4067050 msgr2=0x7f8aa4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:46:16.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.267+0000 7f8ac8ca9700 1 --2- 172.21.15.46:0/3637925853 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8aa4067050 0x7f8aa4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.267+0000 7f8ac8ca9700 1 -- 172.21.15.46:0/3637925853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ac40cf050 msgr2=0x7f8ac4153710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:16.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.267+0000 7f8ac8ca9700 1 --2- 172.21.15.46:0/3637925853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ac40cf050 0x7f8ac4153710 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8ab800c580 tx=0x7f8ab8007d70 comp rx=0 tx=0).stop 2024-01-26T23:46:16.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.267+0000 7f8ac8ca9700 1 -- 172.21.15.46:0/3637925853 shutdown_connections 2024-01-26T23:46:16.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.267+0000 7f8ac8ca9700 1 --2- 172.21.15.46:0/3637925853 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8aa4067050 0x7f8aa4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.267+0000 7f8ac8ca9700 1 --2- 172.21.15.46:0/3637925853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ac4159cb0 0x7f8ac40b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.267+0000 7f8ac8ca9700 1 --2- 172.21.15.46:0/3637925853 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ac40afb40 0x7f8ac40ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.267+0000 7f8ac8ca9700 1 --2- 172.21.15.46:0/3637925853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ac40cf050 0x7f8ac4153710 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.267+0000 7f8ac8ca9700 1 -- 172.21.15.46:0/3637925853 >> 172.21.15.46:0/3637925853 conn(0x7f8ac4087f60 msgr2=0x7f8ac4155560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:16.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.268+0000 7f8ac8ca9700 1 -- 172.21.15.46:0/3637925853 shutdown_connections 2024-01-26T23:46:16.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.268+0000 7f8ac8ca9700 1 -- 172.21.15.46:0/3637925853 wait complete. 2024-01-26T23:46:16.280 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:16.279+0000 7f84319161c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:46:16.282 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-26T23:46:16.345 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:16.344+0000 7fafd9abe1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:46:16.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.459+0000 7fd8f7d98700 1 Processor -- start 2024-01-26T23:46:16.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.459+0000 7fd8f7d98700 1 -- start start 2024-01-26T23:46:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.460+0000 7fd8f7d98700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8f00cf050 0x7fd8f00af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.460+0000 7fd8f7d98700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8f00cf4a0 0x7fd8f00af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.460+0000 7fd8f7d98700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f00cc100 con 0x7fd8f0159cb0 2024-01-26T23:46:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.460+0000 7fd8f7d98700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f00ccca0 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.460+0000 7fd8f7d98700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00cb850 con 0x7fd8f00cf050 2024-01-26T23:46:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.460+0000 7fd8f6335700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8f00cf4a0 0x7fd8f00af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.460+0000 7fd8f5b34700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd8f0159cb0 0x7fd8f00cc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59648/0 (socket says 172.21.15.46:59648) 2024-01-26T23:46:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.460+0000 7fd8f5b34700 1 -- 172.21.15.46:0/981159363 learned_addr learned my addr 172.21.15.46:0/981159363 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.460+0000 7fd8f5333700 1 --2- 172.21.15.46:0/981159363 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8f00cf050 0x7fd8f00af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.460+0000 7fd8f6335700 1 -- 172.21.15.46:0/981159363 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8f00cf050 msgr2=0x7fd8f00af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.460+0000 7fd8f6335700 1 --2- 172.21.15.46:0/981159363 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8f00cf050 0x7fd8f00af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.460+0000 7fd8f6335700 1 -- 172.21.15.46:0/981159363 >> v1:172.21.15.46:6789/0 conn(0x7fd8f0159cb0 legacy=0x7fd8f00cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.461+0000 7fd8f6335700 1 -- 172.21.15.46:0/981159363 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f00cbaf0 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.460+0000 7fd8f4b32700 1 -- 172.21.15.46:0/981159363 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214329176 0 0) 0x7fd8f00cc100 con 0x7fd8f0159cb0 2024-01-26T23:46:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.461+0000 7fd8f6335700 1 --2- 172.21.15.46:0/981159363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f00cf4a0 0x7fd8f00af970 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8009e60 tx=0x7fd8e8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62bfe0d63b862d65 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.461+0000 7fd8f4b32700 1 -- 172.21.15.46:0/981159363 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e800e070 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.461+0000 7fd8f4b32700 1 -- 172.21.15.46:0/981159363 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e800bb60 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.461+0000 7fd8f4b32700 1 -- 172.21.15.46:0/981159363 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8012430 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.462+0000 7fd8f7d98700 1 -- 172.21.15.46:0/981159363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f00cf4a0 msgr2=0x7fd8f00af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:16.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.462+0000 7fd8f7d98700 1 --2- 172.21.15.46:0/981159363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f00cf4a0 0x7fd8f00af970 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8009e60 tx=0x7fd8e8009780 comp rx=0 tx=0).stop 2024-01-26T23:46:16.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.462+0000 7fd8f7d98700 1 -- 172.21.15.46:0/981159363 shutdown_connections 2024-01-26T23:46:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.462+0000 7fd8f7d98700 1 --2- 172.21.15.46:0/981159363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f00cf4a0 0x7fd8f00af970 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.462+0000 7fd8f7d98700 1 --2- 172.21.15.46:0/981159363 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8f00cf050 0x7fd8f00af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.462+0000 7fd8f7d98700 1 -- 172.21.15.46:0/981159363 >> 172.21.15.46:0/981159363 conn(0x7fd8f0087f60 msgr2=0x7fd8f006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.462+0000 7fd8f7d98700 1 -- 172.21.15.46:0/981159363 shutdown_connections 2024-01-26T23:46:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.462+0000 7fd8f7d98700 1 -- 172.21.15.46:0/981159363 wait complete. 2024-01-26T23:46:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.463+0000 7fd8f7d98700 1 Processor -- start 2024-01-26T23:46:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.463+0000 7fd8f7d98700 1 -- start start 2024-01-26T23:46:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.464+0000 7fd8f7d98700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8f00cf050 0x7fd8f00b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.464+0000 7fd8f7d98700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f00cf4a0 0x7fd8f00b0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.464+0000 7fd8f7d98700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8f0159cb0 0x7fd8f00b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.464+0000 7fd8f7d98700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f016a880 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.464+0000 7fd8f7d98700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f0169c70 con 0x7fd8f0159cb0 2024-01-26T23:46:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.464+0000 7fd8f7d98700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f0169f50 con 0x7fd8f00cf050 2024-01-26T23:46:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.464+0000 7fd8f5333700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f00cf4a0 0x7fd8f00b0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.464+0000 7fd8f5333700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f00cf4a0 0x7fd8f00b0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41158/0 (socket says 172.21.15.46:41158) 2024-01-26T23:46:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.464+0000 7fd8f5333700 1 -- 172.21.15.46:0/2824051181 learned_addr learned my addr 172.21.15.46:0/2824051181 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.464+0000 7fd8f5b34700 1 --2- 172.21.15.46:0/2824051181 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8f00cf050 0x7fd8f00b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.464+0000 7fd8f6335700 1 --2- 172.21.15.46:0/2824051181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8f0159cb0 0x7fd8f00b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.464+0000 7fd8f5333700 1 -- 172.21.15.46:0/2824051181 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8f00cf050 msgr2=0x7fd8f00b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.464+0000 7fd8f5333700 1 --2- 172.21.15.46:0/2824051181 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8f00cf050 0x7fd8f00b29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.464+0000 7fd8f5333700 1 -- 172.21.15.46:0/2824051181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8f0159cb0 msgr2=0x7fd8f00b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.464+0000 7fd8f5333700 1 --2- 172.21.15.46:0/2824051181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8f0159cb0 0x7fd8f00b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.464+0000 7fd8f5333700 1 -- 172.21.15.46:0/2824051181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e8009a20 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.465+0000 7fd8f5333700 1 --2- 172.21.15.46:0/2824051181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f00cf4a0 0x7fd8f00b0ad0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000ec80 tx=0x7fd8e00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.465+0000 7fd8e5ffb700 1 -- 172.21.15.46:0/2824051181 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e000c850 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.465+0000 7fd8e5ffb700 1 -- 172.21.15.46:0/2824051181 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e000c9b0 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.465+0000 7fd8e5ffb700 1 -- 172.21.15.46:0/2824051181 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e000cd50 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.465+0000 7fd8f7d98700 1 -- 172.21.15.46:0/2824051181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f016a230 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.465+0000 7fd8f7d98700 1 -- 172.21.15.46:0/2824051181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f00b40f0 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.466+0000 7fd8f7d98700 1 -- 172.21.15.46:0/2824051181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f004c710 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.468+0000 7fd8e5ffb700 1 -- 172.21.15.46:0/2824051181 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 41) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fd8e0013680 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.468+0000 7fd8e5ffb700 1 --2- 172.21.15.46:0/2824051181 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd8d4067050 0x7fd8d4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.468+0000 7fd8e5ffb700 1 -- 172.21.15.46:0/2824051181 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fd8e00854f0 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.468+0000 7fd8f5b34700 1 -- 172.21.15.46:0/2824051181 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd8d4067050 msgr2=0x7fd8d4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.468+0000 7fd8f5b34700 1 --2- 172.21.15.46:0/2824051181 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd8d4067050 0x7fd8d4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:46:16.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.472+0000 7fd8e5ffb700 1 -- 172.21.15.46:0/2824051181 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd8e00522e0 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.555 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:16.554+0000 7fafd9abe1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:46:16.721 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:16.655+0000 7f84319161c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:46:16.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.669+0000 7fd8f5b34700 1 -- 172.21.15.46:0/2824051181 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd8d4067050 msgr2=0x7fd8d4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:16.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.669+0000 7fd8f5b34700 1 --2- 172.21.15.46:0/2824051181 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd8d4067050 0x7fd8d4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:46:16.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.678+0000 7fd8f7d98700 1 -- 172.21.15.46:0/2824051181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd8f016a9c0 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.678+0000 7fd8e5ffb700 1 -- 172.21.15.46:0/2824051181 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd8e0055900 con 0x7fd8f00cf4a0 2024-01-26T23:46:16.722 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:46:16.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.680+0000 7fd8f7d98700 1 -- 172.21.15.46:0/2824051181 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd8d4067050 msgr2=0x7fd8d4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:46:16.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.680+0000 7fd8f7d98700 1 --2- 172.21.15.46:0/2824051181 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd8d4067050 0x7fd8d4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.680+0000 7fd8f7d98700 1 -- 172.21.15.46:0/2824051181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f00cf4a0 msgr2=0x7fd8f00b0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.680+0000 7fd8f7d98700 1 --2- 172.21.15.46:0/2824051181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f00cf4a0 0x7fd8f00b0ad0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000ec80 tx=0x7fd8e00091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.680+0000 7fd8f7d98700 1 -- 172.21.15.46:0/2824051181 shutdown_connections 2024-01-26T23:46:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.680+0000 7fd8f7d98700 1 --2- 172.21.15.46:0/2824051181 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd8d4067050 0x7fd8d4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.680+0000 7fd8f7d98700 1 --2- 172.21.15.46:0/2824051181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8f0159cb0 0x7fd8f00b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.680+0000 7fd8f7d98700 1 --2- 172.21.15.46:0/2824051181 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8f00cf050 0x7fd8f00b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.680+0000 7fd8f7d98700 1 --2- 172.21.15.46:0/2824051181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f00cf4a0 0x7fd8f00b0ad0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.680+0000 7fd8f7d98700 1 -- 172.21.15.46:0/2824051181 >> 172.21.15.46:0/2824051181 conn(0x7fd8f0087f60 msgr2=0x7fd8f01554b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.681+0000 7fd8f7d98700 1 -- 172.21.15.46:0/2824051181 shutdown_connections 2024-01-26T23:46:16.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.681+0000 7fd8f7d98700 1 -- 172.21.15.46:0/2824051181 wait complete. 2024-01-26T23:46:16.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-26T23:46:16.783 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:16.781+0000 7fafd9abe1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:46:16.852 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:16.851+0000 7fafd9abe1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:46:16.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.859+0000 7fee6bab9700 1 Processor -- start 2024-01-26T23:46:16.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.859+0000 7fee6bab9700 1 -- start start 2024-01-26T23:46:16.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.861+0000 7fee6bab9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee640cf050 0x7fee640af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:16.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.861+0000 7fee6bab9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fee640cf4a0 0x7fee640af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:16.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.861+0000 7fee6bab9700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee640cc100 con 0x7fee64159cb0 2024-01-26T23:46:16.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.861+0000 7fee6bab9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee640ccca0 con 0x7fee640cf4a0 2024-01-26T23:46:16.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.861+0000 7fee6bab9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee640cb850 con 0x7fee640cf050 2024-01-26T23:46:16.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.861+0000 7fee6a056700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fee640cf4a0 0x7fee640af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:16.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.861+0000 7fee69054700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee640cf050 0x7fee640af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:16.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.861+0000 7fee69855700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fee64159cb0 0x7fee640cc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:59400/0 (socket says 172.21.15.46:59400) 2024-01-26T23:46:16.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.861+0000 7fee69855700 1 -- 172.21.15.46:0/731436728 learned_addr learned my addr 172.21.15.46:0/731436728 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:16.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.862+0000 7fee6a056700 1 -- 172.21.15.46:0/731436728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee640cf050 msgr2=0x7fee640af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:16.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.861+0000 7fee68853700 1 -- 172.21.15.46:0/731436728 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821829759 0 0) 0x7fee640cc100 con 0x7fee64159cb0 2024-01-26T23:46:16.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.862+0000 7fee6a056700 1 --2- 172.21.15.46:0/731436728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee640cf050 0x7fee640af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.862+0000 7fee6a056700 1 -- 172.21.15.46:0/731436728 >> v1:172.21.15.106:6789/0 conn(0x7fee64159cb0 legacy=0x7fee640cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:16.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.862+0000 7fee6a056700 1 -- 172.21.15.46:0/731436728 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee640cbaf0 con 0x7fee640cf4a0 2024-01-26T23:46:16.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.862+0000 7fee69054700 1 --2- 172.21.15.46:0/731436728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee640cf050 0x7fee640af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:16.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.862+0000 7fee6a056700 1 --2- 172.21.15.46:0/731436728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee640cf4a0 0x7fee640af970 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fee5c009e60 tx=0x7fee5c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20647f894a509d20 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:16.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.862+0000 7fee68853700 1 -- 172.21.15.46:0/731436728 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5c00e070 con 0x7fee640cf4a0 2024-01-26T23:46:16.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.862+0000 7fee68853700 1 -- 172.21.15.46:0/731436728 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5c00bb60 con 0x7fee640cf4a0 2024-01-26T23:46:16.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.862+0000 7fee68853700 1 -- 172.21.15.46:0/731436728 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5c012430 con 0x7fee640cf4a0 2024-01-26T23:46:16.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.863+0000 7fee6bab9700 1 -- 172.21.15.46:0/731436728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee640cf4a0 msgr2=0x7fee640af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:16.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.863+0000 7fee6bab9700 1 --2- 172.21.15.46:0/731436728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee640cf4a0 0x7fee640af970 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fee5c009e60 tx=0x7fee5c009780 comp rx=0 tx=0).stop 2024-01-26T23:46:16.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.863+0000 7fee6bab9700 1 -- 172.21.15.46:0/731436728 shutdown_connections 2024-01-26T23:46:16.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.863+0000 7fee6bab9700 1 --2- 172.21.15.46:0/731436728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee640cf4a0 0x7fee640af970 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.863+0000 7fee6bab9700 1 --2- 172.21.15.46:0/731436728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee640cf050 0x7fee640af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.863+0000 7fee6bab9700 1 -- 172.21.15.46:0/731436728 >> 172.21.15.46:0/731436728 conn(0x7fee64087f60 msgr2=0x7fee6406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:16.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.863+0000 7fee6bab9700 1 -- 172.21.15.46:0/731436728 shutdown_connections 2024-01-26T23:46:16.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.863+0000 7fee6bab9700 1 -- 172.21.15.46:0/731436728 wait complete. 2024-01-26T23:46:16.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.865+0000 7fee6bab9700 1 Processor -- start 2024-01-26T23:46:16.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.865+0000 7fee6bab9700 1 -- start start 2024-01-26T23:46:16.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.866+0000 7fee6bab9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee640cf050 0x7fee640b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.866+0000 7fee6bab9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee640cf4a0 0x7fee640b0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.866+0000 7fee6bab9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee64159cb0 0x7fee640b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.866+0000 7fee6bab9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6416a880 con 0x7fee640cf050 2024-01-26T23:46:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.866+0000 7fee6bab9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee64169c70 con 0x7fee64159cb0 2024-01-26T23:46:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.866+0000 7fee6bab9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee64169f50 con 0x7fee640cf4a0 2024-01-26T23:46:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.866+0000 7fee69855700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee640cf050 0x7fee640b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.866+0000 7fee69855700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee640cf050 0x7fee640b29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41174/0 (socket says 172.21.15.46:41174) 2024-01-26T23:46:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.866+0000 7fee69855700 1 -- 172.21.15.46:0/1549952374 learned_addr learned my addr 172.21.15.46:0/1549952374 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.866+0000 7fee69054700 1 --2- 172.21.15.46:0/1549952374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee640cf4a0 0x7fee640b0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.866+0000 7fee6a056700 1 --2- 172.21.15.46:0/1549952374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee64159cb0 0x7fee640b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.866+0000 7fee69855700 1 -- 172.21.15.46:0/1549952374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee640cf4a0 msgr2=0x7fee640b0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.866+0000 7fee69855700 1 --2- 172.21.15.46:0/1549952374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee640cf4a0 0x7fee640b0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.866+0000 7fee69855700 1 -- 172.21.15.46:0/1549952374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee64159cb0 msgr2=0x7fee640b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:16.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.866+0000 7fee69855700 1 --2- 172.21.15.46:0/1549952374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee64159cb0 0x7fee640b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:16.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.866+0000 7fee69855700 1 -- 172.21.15.46:0/1549952374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee54009420 con 0x7fee640cf050 2024-01-26T23:46:16.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.867+0000 7fee69855700 1 --2- 172.21.15.46:0/1549952374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee640cf050 0x7fee640b29f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fee6000c5e0 tx=0x7fee60001030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:16.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.867+0000 7fee59ffb700 1 -- 172.21.15.46:0/1549952374 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee60005b80 con 0x7fee640cf050 2024-01-26T23:46:16.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.867+0000 7fee59ffb700 1 -- 172.21.15.46:0/1549952374 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee60005d00 con 0x7fee640cf050 2024-01-26T23:46:16.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.867+0000 7fee6bab9700 1 -- 172.21.15.46:0/1549952374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee5c009a20 con 0x7fee640cf050 2024-01-26T23:46:16.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.867+0000 7fee59ffb700 1 -- 172.21.15.46:0/1549952374 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee60007320 con 0x7fee640cf050 2024-01-26T23:46:16.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.867+0000 7fee6bab9700 1 -- 172.21.15.46:0/1549952374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee6416a510 con 0x7fee640cf050 2024-01-26T23:46:16.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.868+0000 7fee6bab9700 1 -- 172.21.15.46:0/1549952374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee6404c710 con 0x7fee640cf050 2024-01-26T23:46:16.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.869+0000 7fee59ffb700 1 -- 172.21.15.46:0/1549952374 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 41) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fee6001b3b0 con 0x7fee640cf050 2024-01-26T23:46:16.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.869+0000 7fee59ffb700 1 --2- 172.21.15.46:0/1549952374 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fee48067050 0x7fee48069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:16.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.869+0000 7fee59ffb700 1 -- 172.21.15.46:0/1549952374 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fee60083eb0 con 0x7fee640cf050 2024-01-26T23:46:16.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.869+0000 7fee69054700 1 -- 172.21.15.46:0/1549952374 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fee48067050 msgr2=0x7fee48069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:16.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.869+0000 7fee69054700 1 --2- 172.21.15.46:0/1549952374 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fee48067050 0x7fee48069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:46:16.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:16.874+0000 7fee59ffb700 1 -- 172.21.15.46:0/1549952374 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fee60050c20 con 0x7fee640cf050 2024-01-26T23:46:16.909 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:16.908+0000 7f84319161c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:46:17.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.070+0000 7fee69054700 1 -- 172.21.15.46:0/1549952374 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fee48067050 msgr2=0x7fee48069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:17.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.070+0000 7fee69054700 1 --2- 172.21.15.46:0/1549952374 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fee48067050 0x7fee48069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:46:17.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.081+0000 7fee6bab9700 1 -- 172.21.15.46:0/1549952374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fee640cfc80 con 0x7fee640cf050 2024-01-26T23:46:17.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.081+0000 7fee59ffb700 1 -- 172.21.15.46:0/1549952374 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+11 (secure 0 0 0) 0x7fee60017070 con 0x7fee640cf050 2024-01-26T23:46:17.083 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-26T23:46:17.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.084+0000 7fee6bab9700 1 -- 172.21.15.46:0/1549952374 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fee48067050 msgr2=0x7fee48069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:46:17.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.084+0000 7fee6bab9700 1 --2- 172.21.15.46:0/1549952374 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fee48067050 0x7fee48069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.084+0000 7fee6bab9700 1 -- 172.21.15.46:0/1549952374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee640cf050 msgr2=0x7fee640b29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:17.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.084+0000 7fee6bab9700 1 --2- 172.21.15.46:0/1549952374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee640cf050 0x7fee640b29f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fee6000c5e0 tx=0x7fee60001030 comp rx=0 tx=0).stop 2024-01-26T23:46:17.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.084+0000 7fee6bab9700 1 -- 172.21.15.46:0/1549952374 shutdown_connections 2024-01-26T23:46:17.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.084+0000 7fee6bab9700 1 --2- 172.21.15.46:0/1549952374 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fee48067050 0x7fee48069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.084+0000 7fee6bab9700 1 --2- 172.21.15.46:0/1549952374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee640cf050 0x7fee640b29f0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.084+0000 7fee6bab9700 1 --2- 172.21.15.46:0/1549952374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee64159cb0 0x7fee640b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.084+0000 7fee6bab9700 1 --2- 172.21.15.46:0/1549952374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee640cf4a0 0x7fee640b0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.084+0000 7fee6bab9700 1 -- 172.21.15.46:0/1549952374 >> 172.21.15.46:0/1549952374 conn(0x7fee64087f60 msgr2=0x7fee641554b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:17.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.084+0000 7fee6bab9700 1 -- 172.21.15.46:0/1549952374 shutdown_connections 2024-01-26T23:46:17.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.084+0000 7fee6bab9700 1 -- 172.21.15.46:0/1549952374 wait complete. 2024-01-26T23:46:17.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:46:17.176 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:17.175+0000 7f84319161c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:46:17.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.231+0000 7efd3f5de700 1 Processor -- start 2024-01-26T23:46:17.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.231+0000 7efd3f5de700 1 -- start start 2024-01-26T23:46:17.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.232+0000 7efd3f5de700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efd38157340 0x7efd3815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:17.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.232+0000 7efd3f5de700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd380ae520 0x7efd380b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:17.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.232+0000 7efd3f5de700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7efd380b3fb0 0x7efd38157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:17.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.232+0000 7efd3f5de700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd380b4430 con 0x7efd38157340 2024-01-26T23:46:17.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.232+0000 7efd3f5de700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd38158ae0 con 0x7efd380b3fb0 2024-01-26T23:46:17.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.232+0000 7efd3f5de700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3815b650 con 0x7efd380ae520 2024-01-26T23:46:17.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.232+0000 7efd3d37a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efd38157340 0x7efd3815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:17.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.232+0000 7efd3d37a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efd38157340 0x7efd3815bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41180/0 (socket says 172.21.15.46:41180) 2024-01-26T23:46:17.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.232+0000 7efd3d37a700 1 -- 172.21.15.46:0/2085397549 learned_addr learned my addr 172.21.15.46:0/2085397549 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:17.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.232+0000 7efd3cb79700 1 --2- 172.21.15.46:0/2085397549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd380ae520 0x7efd380b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:17.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.232+0000 7efd3db7b700 1 --2- 172.21.15.46:0/2085397549 >> v2:172.21.15.106:3300/0 conn(0x7efd380b3fb0 0x7efd38157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:17.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.232+0000 7efd3d37a700 1 -- 172.21.15.46:0/2085397549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd380ae520 msgr2=0x7efd380b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:17.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.232+0000 7efd3d37a700 1 --2- 172.21.15.46:0/2085397549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd380ae520 0x7efd380b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.232+0000 7efd3d37a700 1 -- 172.21.15.46:0/2085397549 >> v2:172.21.15.106:3300/0 conn(0x7efd380b3fb0 msgr2=0x7efd38157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:17.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.232+0000 7efd3d37a700 1 --2- 172.21.15.46:0/2085397549 >> v2:172.21.15.106:3300/0 conn(0x7efd380b3fb0 0x7efd38157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.232+0000 7efd3d37a700 1 -- 172.21.15.46:0/2085397549 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3815c020 con 0x7efd38157340 2024-01-26T23:46:17.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.233+0000 7efd3d37a700 1 --2- 172.21.15.46:0/2085397549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd38157340 0x7efd3815bae0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7efd380caee0 tx=0x7efd34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77dc7a5ff85c42f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:17.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.233+0000 7efd2ffff700 1 -- 172.21.15.46:0/2085397549 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3400b640 con 0x7efd38157340 2024-01-26T23:46:17.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.233+0000 7efd2ffff700 1 -- 172.21.15.46:0/2085397549 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3400b7a0 con 0x7efd38157340 2024-01-26T23:46:17.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.233+0000 7efd2ffff700 1 -- 172.21.15.46:0/2085397549 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3400bb40 con 0x7efd38157340 2024-01-26T23:46:17.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.233+0000 7efd3f5de700 1 -- 172.21.15.46:0/2085397549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd38157340 msgr2=0x7efd3815bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.233+0000 7efd3f5de700 1 --2- 172.21.15.46:0/2085397549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd38157340 0x7efd3815bae0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7efd380caee0 tx=0x7efd34009250 comp rx=0 tx=0).stop 2024-01-26T23:46:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.234+0000 7efd3f5de700 1 -- 172.21.15.46:0/2085397549 shutdown_connections 2024-01-26T23:46:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.234+0000 7efd3f5de700 1 --2- 172.21.15.46:0/2085397549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd38157340 0x7efd3815bae0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.234+0000 7efd3f5de700 1 --2- 172.21.15.46:0/2085397549 >> v2:172.21.15.106:3300/0 conn(0x7efd380b3fb0 0x7efd38157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.234+0000 7efd3f5de700 1 --2- 172.21.15.46:0/2085397549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd380ae520 0x7efd380b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.234+0000 7efd3f5de700 1 -- 172.21.15.46:0/2085397549 >> 172.21.15.46:0/2085397549 conn(0x7efd380ce530 msgr2=0x7efd380ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.234+0000 7efd3f5de700 1 -- 172.21.15.46:0/2085397549 shutdown_connections 2024-01-26T23:46:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.234+0000 7efd3f5de700 1 -- 172.21.15.46:0/2085397549 wait complete. 2024-01-26T23:46:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.235+0000 7efd3f5de700 1 Processor -- start 2024-01-26T23:46:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.235+0000 7efd3f5de700 1 -- start start 2024-01-26T23:46:17.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.236+0000 7efd3f5de700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd380ae520 0x7efd380b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:17.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.236+0000 7efd3f5de700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd380b3fb0 0x7efd381569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:17.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.236+0000 7efd3f5de700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd38157340 0x7efd380c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:17.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.236+0000 7efd3f5de700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd380b1c10 con 0x7efd380b3fb0 2024-01-26T23:46:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.236+0000 7efd3f5de700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd38156f10 con 0x7efd380ae520 2024-01-26T23:46:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.236+0000 7efd3f5de700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd380c7760 con 0x7efd38157340 2024-01-26T23:46:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.236+0000 7efd3cb79700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd380b3fb0 0x7efd381569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.236+0000 7efd3cb79700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd380b3fb0 0x7efd381569d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41186/0 (socket says 172.21.15.46:41186) 2024-01-26T23:46:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.236+0000 7efd3d37a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd380ae520 0x7efd380b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.236+0000 7efd3cb79700 1 -- 172.21.15.46:0/2874198742 learned_addr learned my addr 172.21.15.46:0/2874198742 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.236+0000 7efd3db7b700 1 --2- 172.21.15.46:0/2874198742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd38157340 0x7efd380c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.236+0000 7efd3cb79700 1 -- 172.21.15.46:0/2874198742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd38157340 msgr2=0x7efd380c6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.236+0000 7efd3cb79700 1 --2- 172.21.15.46:0/2874198742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd38157340 0x7efd380c6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.236+0000 7efd3cb79700 1 -- 172.21.15.46:0/2874198742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd380ae520 msgr2=0x7efd380b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.236+0000 7efd3cb79700 1 --2- 172.21.15.46:0/2874198742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd380ae520 0x7efd380b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.237+0000 7efd3cb79700 1 -- 172.21.15.46:0/2874198742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd340094f0 con 0x7efd380b3fb0 2024-01-26T23:46:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.237+0000 7efd3d37a700 1 --2- 172.21.15.46:0/2874198742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd380ae520 0x7efd380b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.237+0000 7efd3cb79700 1 --2- 172.21.15.46:0/2874198742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd380b3fb0 0x7efd381569d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7efd2800ec80 tx=0x7efd280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.237+0000 7efd2d7fa700 1 -- 172.21.15.46:0/2874198742 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2800c850 con 0x7efd380b3fb0 2024-01-26T23:46:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.237+0000 7efd2d7fa700 1 -- 172.21.15.46:0/2874198742 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2800c9b0 con 0x7efd380b3fb0 2024-01-26T23:46:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.237+0000 7efd2d7fa700 1 -- 172.21.15.46:0/2874198742 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2800cd50 con 0x7efd380b3fb0 2024-01-26T23:46:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.237+0000 7efd3f5de700 1 -- 172.21.15.46:0/2874198742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd380b12e0 con 0x7efd380b3fb0 2024-01-26T23:46:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.237+0000 7efd3f5de700 1 -- 172.21.15.46:0/2874198742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd380b18a0 con 0x7efd380b3fb0 2024-01-26T23:46:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.238+0000 7efd3f5de700 1 -- 172.21.15.46:0/2874198742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3804c710 con 0x7efd380b3fb0 2024-01-26T23:46:17.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.239+0000 7efd2d7fa700 1 -- 172.21.15.46:0/2874198742 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 41) v1 ==== 84845+0+0 (secure 0 0 0) 0x7efd28013680 con 0x7efd380b3fb0 2024-01-26T23:46:17.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.240+0000 7efd2d7fa700 1 --2- 172.21.15.46:0/2874198742 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd1c067050 0x7efd1c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:17.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.240+0000 7efd2d7fa700 1 -- 172.21.15.46:0/2874198742 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6718+0+0 (secure 0 0 0) 0x7efd28085590 con 0x7efd380b3fb0 2024-01-26T23:46:17.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.240+0000 7efd3d37a700 1 -- 172.21.15.46:0/2874198742 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd1c067050 msgr2=0x7efd1c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:17.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.240+0000 7efd3d37a700 1 --2- 172.21.15.46:0/2874198742 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd1c067050 0x7efd1c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:46:17.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.243+0000 7efd2d7fa700 1 -- 172.21.15.46:0/2874198742 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efd280522e0 con 0x7efd380b3fb0 2024-01-26T23:46:17.273 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:17.271+0000 7f84319161c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:46:17.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.441+0000 7efd3d37a700 1 -- 172.21.15.46:0/2874198742 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd1c067050 msgr2=0x7efd1c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:17.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.441+0000 7efd3d37a700 1 --2- 172.21.15.46:0/2874198742 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd1c067050 0x7efd1c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:46:17.455 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:17.455+0000 7fafd9abe1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:46:17.529 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:17.528+0000 7fafd9abe1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:46:17.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.574+0000 7efd3f5de700 1 -- 172.21.15.46:0/2874198742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd380b1d50 con 0x7efd380b3fb0 2024-01-26T23:46:17.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.575+0000 7efd2d7fa700 1 -- 172.21.15.46:0/2874198742 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4605 (secure 0 0 0) 0x7efd28055900 con 0x7efd380b3fb0 2024-01-26T23:46:17.576 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:17.576 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6137,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/1370365478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1370365478},{"type":"v1","addr":"172.21.15.106:6836","nonce":1370365478}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":6619,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/1980309094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1980309094},{"type":"v1","addr":"172.21.15.106:6837","nonce":1980309094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40},{"gid":6628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3910368747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3910368747},{"type":"v1","addr":"172.21.15.46:6834","nonce":3910368747}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:45:52.426977+0000","modified":"2024-01-26T23:45:56.004421+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6631":{"gid":6631,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3176096333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3176096333},{"type":"v1","addr":"172.21.15.46:6835","nonce":3176096333}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2024-01-26T23:46:17.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.577+0000 7efd3f5de700 1 -- 172.21.15.46:0/2874198742 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd1c067050 msgr2=0x7efd1c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:46:17.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.577+0000 7efd3f5de700 1 --2- 172.21.15.46:0/2874198742 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd1c067050 0x7efd1c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.577+0000 7efd3f5de700 1 -- 172.21.15.46:0/2874198742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd380b3fb0 msgr2=0x7efd381569d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.577+0000 7efd3f5de700 1 --2- 172.21.15.46:0/2874198742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd380b3fb0 0x7efd381569d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7efd2800ec80 tx=0x7efd280091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.577+0000 7efd3f5de700 1 -- 172.21.15.46:0/2874198742 shutdown_connections 2024-01-26T23:46:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.577+0000 7efd3f5de700 1 --2- 172.21.15.46:0/2874198742 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efd1c067050 0x7efd1c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.577+0000 7efd3f5de700 1 --2- 172.21.15.46:0/2874198742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd38157340 0x7efd380c6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.577+0000 7efd3f5de700 1 --2- 172.21.15.46:0/2874198742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd380ae520 0x7efd380b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.577+0000 7efd3f5de700 1 --2- 172.21.15.46:0/2874198742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd380b3fb0 0x7efd381569d0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.577+0000 7efd3f5de700 1 -- 172.21.15.46:0/2874198742 >> 172.21.15.46:0/2874198742 conn(0x7efd380ce530 msgr2=0x7efd3806d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.577+0000 7efd3f5de700 1 -- 172.21.15.46:0/2874198742 shutdown_connections 2024-01-26T23:46:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.577+0000 7efd3f5de700 1 -- 172.21.15.46:0/2874198742 wait complete. 2024-01-26T23:46:17.580 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 45 2024-01-26T23:46:17.592 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:46:17.652 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:17.651+0000 7fafd9abe1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:46:17.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.725+0000 7f2def734700 1 Processor -- start 2024-01-26T23:46:17.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.725+0000 7f2def734700 1 -- start start 2024-01-26T23:46:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.726+0000 7f2def734700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2de81599e0 0x7f2de8067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.726+0000 7f2def734700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de80ce2b0 con 0x7f2de8159610 2024-01-26T23:46:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.726+0000 7f2def734700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de80afeb0 con 0x7f2de8067e70 2024-01-26T23:46:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.726+0000 7f2def734700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de80682f0 con 0x7f2de81599e0 2024-01-26T23:46:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.726+0000 7f2ded4d0700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2de8159610 0x7f2de80b2a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59656/0 (socket says 172.21.15.46:59656) 2024-01-26T23:46:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.726+0000 7f2ded4d0700 1 -- 172.21.15.46:0/3666223162 learned_addr learned my addr 172.21.15.46:0/3666223162 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.726+0000 7f2decccf700 1 --2- 172.21.15.46:0/3666223162 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2de81599e0 0x7f2de8067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.727+0000 7f2ddffff700 1 -- 172.21.15.46:0/3666223162 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650835749 0 0) 0x7f2de80ce2b0 con 0x7f2de8159610 2024-01-26T23:46:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.727+0000 7f2ddffff700 1 -- 172.21.15.46:0/3666223162 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd4003570 con 0x7f2de8159610 2024-01-26T23:46:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.727+0000 7f2ddffff700 1 -- 172.21.15.46:0/3666223162 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3238010709 0 0) 0x7f2de80afeb0 con 0x7f2de8067e70 2024-01-26T23:46:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.727+0000 7f2ddffff700 1 -- 172.21.15.46:0/3666223162 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2de80ce2b0 con 0x7f2de8067e70 2024-01-26T23:46:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.727+0000 7f2decccf700 1 -- 172.21.15.46:0/3666223162 >> v1:172.21.15.106:6789/0 conn(0x7f2de8067e70 legacy=0x7f2de815a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.727+0000 7f2decccf700 1 -- 172.21.15.46:0/3666223162 >> v1:172.21.15.46:6789/0 conn(0x7f2de8159610 legacy=0x7f2de80b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.727+0000 7f2decccf700 1 -- 172.21.15.46:0/3666223162 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de80cd3c0 con 0x7f2de81599e0 2024-01-26T23:46:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.727+0000 7f2decccf700 1 --2- 172.21.15.46:0/3666223162 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2de81599e0 0x7f2de8067930 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8009910 tx=0x7f2dd80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9af734334b64bcb server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.727+0000 7f2ddffff700 1 -- 172.21.15.46:0/3666223162 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8011630 con 0x7f2de81599e0 2024-01-26T23:46:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.727+0000 7f2ddffff700 1 -- 172.21.15.46:0/3666223162 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd8011790 con 0x7f2de81599e0 2024-01-26T23:46:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.727+0000 7f2ddffff700 1 -- 172.21.15.46:0/3666223162 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8011b30 con 0x7f2de81599e0 2024-01-26T23:46:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.728+0000 7f2def734700 1 -- 172.21.15.46:0/3666223162 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2de81599e0 msgr2=0x7f2de8067930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.728+0000 7f2def734700 1 --2- 172.21.15.46:0/3666223162 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2de81599e0 0x7f2de8067930 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8009910 tx=0x7f2dd80091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.728+0000 7f2def734700 1 -- 172.21.15.46:0/3666223162 shutdown_connections 2024-01-26T23:46:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.728+0000 7f2def734700 1 --2- 172.21.15.46:0/3666223162 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2de81599e0 0x7f2de8067930 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.728+0000 7f2def734700 1 -- 172.21.15.46:0/3666223162 >> 172.21.15.46:0/3666223162 conn(0x7f2de80694d0 msgr2=0x7f2de80aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.728+0000 7f2def734700 1 -- 172.21.15.46:0/3666223162 shutdown_connections 2024-01-26T23:46:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.728+0000 7f2def734700 1 -- 172.21.15.46:0/3666223162 wait complete. 2024-01-26T23:46:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.729+0000 7f2def734700 1 Processor -- start 2024-01-26T23:46:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.729+0000 7f2def734700 1 -- start start 2024-01-26T23:46:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.730+0000 7f2def734700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2de8067e70 0x7f2de814b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.730+0000 7f2def734700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2de8159610 0x7f2de814d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.730+0000 7f2def734700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de81599e0 0x7f2de80ad850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.730+0000 7f2def734700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de8150890 con 0x7f2de81599e0 2024-01-26T23:46:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.730+0000 7f2def734700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de814fca0 con 0x7f2de8159610 2024-01-26T23:46:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.730+0000 7f2def734700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de814ff60 con 0x7f2de8067e70 2024-01-26T23:46:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.730+0000 7f2dedcd1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de81599e0 0x7f2de80ad850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.730+0000 7f2ded4d0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2de8067e70 0x7f2de814b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.730+0000 7f2decccf700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2de8159610 0x7f2de814d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.730+0000 7f2dedcd1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de81599e0 0x7f2de80ad850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41194/0 (socket says 172.21.15.46:41194) 2024-01-26T23:46:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.730+0000 7f2dedcd1700 1 -- 172.21.15.46:0/2208196376 learned_addr learned my addr 172.21.15.46:0/2208196376 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.731+0000 7f2dedcd1700 1 -- 172.21.15.46:0/2208196376 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2de8067e70 msgr2=0x7f2de814b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.731+0000 7f2dedcd1700 1 --2- 172.21.15.46:0/2208196376 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2de8067e70 0x7f2de814b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.731+0000 7f2dedcd1700 1 -- 172.21.15.46:0/2208196376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2de8159610 msgr2=0x7f2de814d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.731+0000 7f2dedcd1700 1 --2- 172.21.15.46:0/2208196376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2de8159610 0x7f2de814d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.731+0000 7f2dedcd1700 1 -- 172.21.15.46:0/2208196376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd8009420 con 0x7f2de81599e0 2024-01-26T23:46:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.731+0000 7f2ded4d0700 1 --2- 172.21.15.46:0/2208196376 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2de8067e70 0x7f2de814b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.731+0000 7f2dedcd1700 1 --2- 172.21.15.46:0/2208196376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de81599e0 0x7f2de80ad850 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f2de000c910 tx=0x7f2de00027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.731+0000 7f2ddd7fa700 1 -- 172.21.15.46:0/2208196376 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de0012630 con 0x7f2de81599e0 2024-01-26T23:46:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.731+0000 7f2ddd7fa700 1 -- 172.21.15.46:0/2208196376 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de00127b0 con 0x7f2de81599e0 2024-01-26T23:46:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.731+0000 7f2ddd7fa700 1 -- 172.21.15.46:0/2208196376 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de0012b70 con 0x7f2de81599e0 2024-01-26T23:46:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.731+0000 7f2def734700 1 -- 172.21.15.46:0/2208196376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de8150240 con 0x7f2de81599e0 2024-01-26T23:46:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.731+0000 7f2def734700 1 -- 172.21.15.46:0/2208196376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de80ac1a0 con 0x7f2de81599e0 2024-01-26T23:46:17.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.732+0000 7f2def734700 1 -- 172.21.15.46:0/2208196376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de8150a00 con 0x7f2de81599e0 2024-01-26T23:46:17.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.733+0000 7f2ddd7fa700 1 -- 172.21.15.46:0/2208196376 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 41) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f2de001b400 con 0x7f2de81599e0 2024-01-26T23:46:17.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.733+0000 7f2ddd7fa700 1 --2- 172.21.15.46:0/2208196376 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2dcc067050 0x7f2dcc069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.733+0000 7f2ded4d0700 1 -- 172.21.15.46:0/2208196376 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2dcc067050 msgr2=0x7f2dcc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.734+0000 7f2ddd7fa700 1 -- 172.21.15.46:0/2208196376 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f2de0084da0 con 0x7f2de81599e0 2024-01-26T23:46:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.734+0000 7f2ded4d0700 1 --2- 172.21.15.46:0/2208196376 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2dcc067050 0x7f2dcc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:46:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.738+0000 7f2ddd7fa700 1 -- 172.21.15.46:0/2208196376 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2de0051b10 con 0x7f2de81599e0 2024-01-26T23:46:17.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.935+0000 7f2ded4d0700 1 -- 172.21.15.46:0/2208196376 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2dcc067050 msgr2=0x7f2dcc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:17.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.935+0000 7f2ded4d0700 1 --2- 172.21.15.46:0/2208196376 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2dcc067050 0x7f2dcc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:46:17.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.944+0000 7f2def734700 1 -- 172.21.15.46:0/2208196376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2de804c710 con 0x7f2de81599e0 2024-01-26T23:46:17.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.945+0000 7f2ddd7fa700 1 -- 172.21.15.46:0/2208196376 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16587 (secure 0 0 0) 0x7f2de0055130 con 0x7f2de81599e0 2024-01-26T23:46:17.947 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:17.947 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":100,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:46:12.597656+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:45:50.922223+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-01-26T23:45:51.397995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/2701612997":"2024-01-27T23:46:12.597621+0000","172.21.15.106:0/1894478014":"2024-01-27T23:46:12.597621+0000","172.21.15.106:0/623871419":"2024-01-27T23:46:12.597621+0000","172.21.15.106:6833/43388":"2024-01-27T23:46:12.597621+0000","172.21.15.106:6832/43388":"2024-01-27T23:46:12.597621+0000","172.21.15.106:0/243105601":"2024-01-27T23:46:12.597621+0000","172.21.15.106:0/3064987359":"2024-01-27T00:46:05.889834+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:46:17.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.947+0000 7f2def734700 1 -- 172.21.15.46:0/2208196376 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2dcc067050 msgr2=0x7f2dcc069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:46:17.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.947+0000 7f2def734700 1 --2- 172.21.15.46:0/2208196376 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2dcc067050 0x7f2dcc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.948+0000 7f2def734700 1 -- 172.21.15.46:0/2208196376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de81599e0 msgr2=0x7f2de80ad850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:17.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.948+0000 7f2def734700 1 --2- 172.21.15.46:0/2208196376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de81599e0 0x7f2de80ad850 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f2de000c910 tx=0x7f2de00027a0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.948+0000 7f2def734700 1 -- 172.21.15.46:0/2208196376 shutdown_connections 2024-01-26T23:46:17.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.948+0000 7f2def734700 1 --2- 172.21.15.46:0/2208196376 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2dcc067050 0x7f2dcc069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.948+0000 7f2def734700 1 --2- 172.21.15.46:0/2208196376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de81599e0 0x7f2de80ad850 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.948+0000 7f2def734700 1 --2- 172.21.15.46:0/2208196376 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2de8067e70 0x7f2de814b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.948+0000 7f2def734700 1 --2- 172.21.15.46:0/2208196376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2de8159610 0x7f2de814d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:17.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.948+0000 7f2def734700 1 -- 172.21.15.46:0/2208196376 >> 172.21.15.46:0/2208196376 conn(0x7f2de80694d0 msgr2=0x7f2de81562a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:17.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.948+0000 7f2def734700 1 -- 172.21.15.46:0/2208196376 shutdown_connections 2024-01-26T23:46:17.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:17.948+0000 7f2def734700 1 -- 172.21.15.46:0/2208196376 wait complete. 2024-01-26T23:46:17.963 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-26T23:46:17.964 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:46:18.039 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:18.038+0000 7f84319161c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:46:18.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.096+0000 7f2499bda700 1 Processor -- start 2024-01-26T23:46:18.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.096+0000 7f2499bda700 1 -- start start 2024-01-26T23:46:18.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.097+0000 7f2499bda700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f249415ce40 0x7f24940d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.097+0000 7f2499bda700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f24940cc1c0 0x7f24940ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.097+0000 7f2499bda700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24940cff70 con 0x7f249415d210 2024-01-26T23:46:18.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.097+0000 7f2499bda700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24940cf400 con 0x7f249415ce40 2024-01-26T23:46:18.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.097+0000 7f2499bda700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24940cf660 con 0x7f24940cc1c0 2024-01-26T23:46:18.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.097+0000 7f24937fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f249415ce40 0x7f24940d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:18.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.097+0000 7f2493fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f24940cc1c0 0x7f24940ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:18.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.097+0000 7f2492ffd700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f249415d210 0x7f24940cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:59418/0 (socket says 172.21.15.46:59418) 2024-01-26T23:46:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.097+0000 7f2492ffd700 1 -- 172.21.15.46:0/1072702175 learned_addr learned my addr 172.21.15.46:0/1072702175 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.097+0000 7f2493fff700 1 -- 172.21.15.46:0/1072702175 >> v2:172.21.15.106:3300/0 conn(0x7f249415ce40 msgr2=0x7f24940d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.097+0000 7f2493fff700 1 --2- 172.21.15.46:0/1072702175 >> v2:172.21.15.106:3300/0 conn(0x7f249415ce40 0x7f24940d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.097+0000 7f2493fff700 1 -- 172.21.15.46:0/1072702175 >> v1:172.21.15.106:6789/0 conn(0x7f249415d210 legacy=0x7f24940cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.097+0000 7f2493fff700 1 -- 172.21.15.46:0/1072702175 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24940d0500 con 0x7f24940cc1c0 2024-01-26T23:46:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.097+0000 7f24927fc700 1 -- 172.21.15.46:0/1072702175 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147120791 0 0) 0x7f24940cff70 con 0x7f249415d210 2024-01-26T23:46:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.097+0000 7f24937fe700 1 --2- 172.21.15.46:0/1072702175 >> v2:172.21.15.106:3300/0 conn(0x7f249415ce40 0x7f24940d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.098+0000 7f2493fff700 1 --2- 172.21.15.46:0/1072702175 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f24940cc1c0 0x7f24940ade30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f248c009e60 tx=0x7f248c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a32ddc1422e28e5a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.098+0000 7f24927fc700 1 -- 172.21.15.46:0/1072702175 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c00e070 con 0x7f24940cc1c0 2024-01-26T23:46:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.098+0000 7f24927fc700 1 -- 172.21.15.46:0/1072702175 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248c00bb60 con 0x7f24940cc1c0 2024-01-26T23:46:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.098+0000 7f24927fc700 1 -- 172.21.15.46:0/1072702175 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c012430 con 0x7f24940cc1c0 2024-01-26T23:46:18.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.098+0000 7f2499bda700 1 -- 172.21.15.46:0/1072702175 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f24940cc1c0 msgr2=0x7f24940ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:18.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.098+0000 7f2499bda700 1 --2- 172.21.15.46:0/1072702175 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f24940cc1c0 0x7f24940ade30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f248c009e60 tx=0x7f248c009780 comp rx=0 tx=0).stop 2024-01-26T23:46:18.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.098+0000 7f2499bda700 1 -- 172.21.15.46:0/1072702175 shutdown_connections 2024-01-26T23:46:18.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.098+0000 7f2499bda700 1 --2- 172.21.15.46:0/1072702175 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f24940cc1c0 0x7f24940ade30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.098+0000 7f2499bda700 1 --2- 172.21.15.46:0/1072702175 >> v2:172.21.15.106:3300/0 conn(0x7f249415ce40 0x7f24940d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.099+0000 7f2499bda700 1 -- 172.21.15.46:0/1072702175 >> 172.21.15.46:0/1072702175 conn(0x7f249406f840 msgr2=0x7f24940d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:18.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.099+0000 7f2499bda700 1 -- 172.21.15.46:0/1072702175 shutdown_connections 2024-01-26T23:46:18.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.099+0000 7f2499bda700 1 -- 172.21.15.46:0/1072702175 wait complete. 2024-01-26T23:46:18.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.100+0000 7f2499bda700 1 Processor -- start 2024-01-26T23:46:18.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.100+0000 7f2499bda700 1 -- start start 2024-01-26T23:46:18.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f2499bda700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24940cc1c0 0x7f2494157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f2499bda700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f249415ce40 0x7f24940c5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f2499bda700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f249415d210 0x7f24940c6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f2499bda700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249415d720 con 0x7f249415d210 2024-01-26T23:46:18.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f2499bda700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2494157c50 con 0x7f24940cc1c0 2024-01-26T23:46:18.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f2499bda700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2494157f30 con 0x7f249415ce40 2024-01-26T23:46:18.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f2493fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f249415d210 0x7f24940c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:18.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f2492ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f249415ce40 0x7f24940c5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:18.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f24937fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24940cc1c0 0x7f2494157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f2493fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f249415d210 0x7f24940c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41198/0 (socket says 172.21.15.46:41198) 2024-01-26T23:46:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f2493fff700 1 -- 172.21.15.46:0/2204512194 learned_addr learned my addr 172.21.15.46:0/2204512194 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f2492ffd700 1 -- 172.21.15.46:0/2204512194 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24940cc1c0 msgr2=0x7f2494157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f2492ffd700 1 --2- 172.21.15.46:0/2204512194 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24940cc1c0 0x7f2494157640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f2492ffd700 1 -- 172.21.15.46:0/2204512194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f249415d210 msgr2=0x7f24940c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f2492ffd700 1 --2- 172.21.15.46:0/2204512194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f249415d210 0x7f24940c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f2492ffd700 1 -- 172.21.15.46:0/2204512194 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24840093c0 con 0x7f249415ce40 2024-01-26T23:46:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f24937fe700 1 --2- 172.21.15.46:0/2204512194 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24940cc1c0 0x7f2494157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.101+0000 7f2493fff700 1 --2- 172.21.15.46:0/2204512194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f249415d210 0x7f24940c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.102+0000 7f2492ffd700 1 --2- 172.21.15.46:0/2204512194 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f249415ce40 0x7f24940c5240 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f248800c4c0 tx=0x7f2488007f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.102+0000 7f247b7fe700 1 -- 172.21.15.46:0/2204512194 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248800ce70 con 0x7f249415ce40 2024-01-26T23:46:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.102+0000 7f247b7fe700 1 -- 172.21.15.46:0/2204512194 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2488005b30 con 0x7f249415ce40 2024-01-26T23:46:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.102+0000 7f2499bda700 1 -- 172.21.15.46:0/2204512194 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248c009a20 con 0x7f249415ce40 2024-01-26T23:46:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.102+0000 7f247b7fe700 1 -- 172.21.15.46:0/2204512194 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24880070d0 con 0x7f249415ce40 2024-01-26T23:46:18.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.102+0000 7f2499bda700 1 -- 172.21.15.46:0/2204512194 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24941584f0 con 0x7f249415ce40 2024-01-26T23:46:18.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.103+0000 7f2499bda700 1 -- 172.21.15.46:0/2204512194 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f249415d8d0 con 0x7f249415ce40 2024-01-26T23:46:18.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.103+0000 7f247b7fe700 1 -- 172.21.15.46:0/2204512194 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 41) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f24880074c0 con 0x7f249415ce40 2024-01-26T23:46:18.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.104+0000 7f247b7fe700 1 --2- 172.21.15.46:0/2204512194 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2480067050 0x7f2480069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.104+0000 7f24937fe700 1 -- 172.21.15.46:0/2204512194 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2480067050 msgr2=0x7f2480069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:18.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.104+0000 7f247b7fe700 1 -- 172.21.15.46:0/2204512194 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f2488083e00 con 0x7f249415ce40 2024-01-26T23:46:18.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.104+0000 7f24937fe700 1 --2- 172.21.15.46:0/2204512194 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2480067050 0x7f2480069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:46:18.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.108+0000 7f247b7fe700 1 -- 172.21.15.46:0/2204512194 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f249415d8d0 con 0x7f249415ce40 2024-01-26T23:46:18.128 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:18.127+0000 7f84319161c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:46:18.156 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:18.155+0000 7fafd9abe1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:46:18.271 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:18.270+0000 7f84319161c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:46:18.303 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:18.302+0000 7fafd9abe1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:46:18.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.305+0000 7f24937fe700 1 -- 172.21.15.46:0/2204512194 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2480067050 msgr2=0x7f2480069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:18.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.305+0000 7f24937fe700 1 --2- 172.21.15.46:0/2204512194 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2480067050 0x7f2480069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:46:18.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.359+0000 7f2499bda700 1 -- 172.21.15.46:0/2204512194 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f249415d890 con 0x7f249415ce40 2024-01-26T23:46:18.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.360+0000 7f247b7fe700 1 -- 172.21.15.46:0/2204512194 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f2488017090 con 0x7f249415ce40 2024-01-26T23:46:18.361 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:18.361 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6137,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/1370365478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1370365478},{"type":"v1","addr":"172.21.15.106:6836","nonce":1370365478}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":6619,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/1980309094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1980309094},{"type":"v1","addr":"172.21.15.106:6837","nonce":1980309094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40},{"gid":6628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3910368747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3910368747},{"type":"v1","addr":"172.21.15.46:6834","nonce":3910368747}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:45:52.426977+0000","modified":"2024-01-26T23:45:56.004421+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6631":{"gid":6631,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3176096333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3176096333},{"type":"v1","addr":"172.21.15.46:6835","nonce":3176096333}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2024-01-26T23:46:18.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.363+0000 7f2499bda700 1 -- 172.21.15.46:0/2204512194 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2480067050 msgr2=0x7f2480069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:46:18.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.363+0000 7f2499bda700 1 --2- 172.21.15.46:0/2204512194 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2480067050 0x7f2480069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.363+0000 7f2499bda700 1 -- 172.21.15.46:0/2204512194 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f249415ce40 msgr2=0x7f24940c5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:18.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.363+0000 7f2499bda700 1 --2- 172.21.15.46:0/2204512194 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f249415ce40 0x7f24940c5240 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f248800c4c0 tx=0x7f2488007f50 comp rx=0 tx=0).stop 2024-01-26T23:46:18.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.363+0000 7f2499bda700 1 -- 172.21.15.46:0/2204512194 shutdown_connections 2024-01-26T23:46:18.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.363+0000 7f2499bda700 1 --2- 172.21.15.46:0/2204512194 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2480067050 0x7f2480069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.363+0000 7f2499bda700 1 --2- 172.21.15.46:0/2204512194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f249415d210 0x7f24940c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.363+0000 7f2499bda700 1 --2- 172.21.15.46:0/2204512194 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24940cc1c0 0x7f2494157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.363+0000 7f2499bda700 1 --2- 172.21.15.46:0/2204512194 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f249415ce40 0x7f24940c5240 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.363+0000 7f2499bda700 1 -- 172.21.15.46:0/2204512194 >> 172.21.15.46:0/2204512194 conn(0x7f249406f840 msgr2=0x7f24940b5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:18.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.363+0000 7f2499bda700 1 -- 172.21.15.46:0/2204512194 shutdown_connections 2024-01-26T23:46:18.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.364+0000 7f2499bda700 1 -- 172.21.15.46:0/2204512194 wait complete. 2024-01-26T23:46:18.366 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 45 2024-01-26T23:46:18.383 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:46:18.452 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:18.451+0000 7fafd9abe1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:46:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.515+0000 7f40353b0700 1 Processor -- start 2024-01-26T23:46:18.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.515+0000 7f40353b0700 1 -- start start 2024-01-26T23:46:18.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.516+0000 7f40353b0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4030159920 0x7f40300caa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.516+0000 7f40353b0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f403006abf0 0x7f40300caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.516+0000 7f40353b0700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403006b1d0 con 0x7f4030159cf0 2024-01-26T23:46:18.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.516+0000 7f40353b0700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f403006b4b0 con 0x7f403006abf0 2024-01-26T23:46:18.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.516+0000 7f40353b0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403006b790 con 0x7f4030159920 2024-01-26T23:46:18.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.516+0000 7f402f7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f403006abf0 0x7f40300caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:18.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.516+0000 7f402e7fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4030159cf0 0x7f403006a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59662/0 (socket says 172.21.15.46:59662) 2024-01-26T23:46:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.516+0000 7f402e7fc700 1 -- 172.21.15.46:0/1782746719 learned_addr learned my addr 172.21.15.46:0/1782746719 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.516+0000 7f402effd700 1 --2- 172.21.15.46:0/1782746719 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4030159920 0x7f40300caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.516+0000 7f402f7fe700 1 -- 172.21.15.46:0/1782746719 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4030159920 msgr2=0x7f40300caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.516+0000 7f402f7fe700 1 --2- 172.21.15.46:0/1782746719 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4030159920 0x7f40300caa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.516+0000 7f402f7fe700 1 -- 172.21.15.46:0/1782746719 >> v1:172.21.15.46:6789/0 conn(0x7f4030159cf0 legacy=0x7f403006a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.517+0000 7f402f7fe700 1 -- 172.21.15.46:0/1782746719 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403006ba70 con 0x7f403006abf0 2024-01-26T23:46:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.516+0000 7f402dffb700 1 -- 172.21.15.46:0/1782746719 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697168358 0 0) 0x7f403006b1d0 con 0x7f4030159cf0 2024-01-26T23:46:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.517+0000 7f402f7fe700 1 --2- 172.21.15.46:0/1782746719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f403006abf0 0x7f40300caf40 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4028009e60 tx=0x7f4028009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=821288469519c8d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.517+0000 7f402dffb700 1 -- 172.21.15.46:0/1782746719 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402800e070 con 0x7f403006abf0 2024-01-26T23:46:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.517+0000 7f402dffb700 1 -- 172.21.15.46:0/1782746719 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402800bb60 con 0x7f403006abf0 2024-01-26T23:46:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.517+0000 7f402dffb700 1 -- 172.21.15.46:0/1782746719 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4028012430 con 0x7f403006abf0 2024-01-26T23:46:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.517+0000 7f40353b0700 1 -- 172.21.15.46:0/1782746719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f403006abf0 msgr2=0x7f40300caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.517+0000 7f40353b0700 1 --2- 172.21.15.46:0/1782746719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f403006abf0 0x7f40300caf40 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4028009e60 tx=0x7f4028009780 comp rx=0 tx=0).stop 2024-01-26T23:46:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.518+0000 7f40353b0700 1 -- 172.21.15.46:0/1782746719 shutdown_connections 2024-01-26T23:46:18.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.518+0000 7f40353b0700 1 --2- 172.21.15.46:0/1782746719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f403006abf0 0x7f40300caf40 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.518+0000 7f40353b0700 1 --2- 172.21.15.46:0/1782746719 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4030159920 0x7f40300caa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.518+0000 7f40353b0700 1 -- 172.21.15.46:0/1782746719 >> 172.21.15.46:0/1782746719 conn(0x7f40300cd6f0 msgr2=0x7f40300c9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:18.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.518+0000 7f40353b0700 1 -- 172.21.15.46:0/1782746719 shutdown_connections 2024-01-26T23:46:18.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.518+0000 7f40353b0700 1 -- 172.21.15.46:0/1782746719 wait complete. 2024-01-26T23:46:18.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.519+0000 7f40353b0700 1 Processor -- start 2024-01-26T23:46:18.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.519+0000 7f40353b0700 1 -- start start 2024-01-26T23:46:18.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.520+0000 7f40353b0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f403006abf0 0x7f4030158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.520+0000 7f40353b0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4030159920 0x7f40300b2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.520+0000 7f40353b0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4030159cf0 0x7f40300ad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.520+0000 7f40353b0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403016a440 con 0x7f4030159920 2024-01-26T23:46:18.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.520+0000 7f40353b0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40300b0110 con 0x7f403006abf0 2024-01-26T23:46:18.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.520+0000 7f40353b0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40300aca10 con 0x7f4030159cf0 2024-01-26T23:46:18.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.520+0000 7f402e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4030159920 0x7f40300b2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:18.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.520+0000 7f402e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4030159920 0x7f40300b2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41208/0 (socket says 172.21.15.46:41208) 2024-01-26T23:46:18.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.520+0000 7f402e7fc700 1 -- 172.21.15.46:0/3057284109 learned_addr learned my addr 172.21.15.46:0/3057284109 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:18.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.520+0000 7f402f7fe700 1 --2- 172.21.15.46:0/3057284109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4030159cf0 0x7f40300ad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:18.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.520+0000 7f402effd700 1 --2- 172.21.15.46:0/3057284109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f403006abf0 0x7f4030158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:18.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.520+0000 7f402f7fe700 1 -- 172.21.15.46:0/3057284109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f403006abf0 msgr2=0x7f4030158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:18.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.520+0000 7f402f7fe700 1 --2- 172.21.15.46:0/3057284109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f403006abf0 0x7f4030158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.520+0000 7f402f7fe700 1 -- 172.21.15.46:0/3057284109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4030159920 msgr2=0x7f40300b2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:18.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.520+0000 7f402f7fe700 1 --2- 172.21.15.46:0/3057284109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4030159920 0x7f40300b2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.520+0000 7f402f7fe700 1 -- 172.21.15.46:0/3057284109 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4028009a20 con 0x7f4030159cf0 2024-01-26T23:46:18.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.520+0000 7f402e7fc700 1 --2- 172.21.15.46:0/3057284109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4030159920 0x7f40300b2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:18.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.521+0000 7f402f7fe700 1 --2- 172.21.15.46:0/3057284109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4030159cf0 0x7f40300ad9c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4028016000 tx=0x7f4028019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:18.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.521+0000 7f4026ffd700 1 -- 172.21.15.46:0/3057284109 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4028012e60 con 0x7f4030159cf0 2024-01-26T23:46:18.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.521+0000 7f4026ffd700 1 -- 172.21.15.46:0/3057284109 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4028025440 con 0x7f4030159cf0 2024-01-26T23:46:18.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.521+0000 7f4026ffd700 1 -- 172.21.15.46:0/3057284109 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40280257e0 con 0x7f4030159cf0 2024-01-26T23:46:18.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.521+0000 7f40353b0700 1 -- 172.21.15.46:0/3057284109 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40300accf0 con 0x7f4030159cf0 2024-01-26T23:46:18.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.521+0000 7f40353b0700 1 -- 172.21.15.46:0/3057284109 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40300afda0 con 0x7f4030159cf0 2024-01-26T23:46:18.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.522+0000 7f40353b0700 1 -- 172.21.15.46:0/3057284109 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403004c710 con 0x7f4030159cf0 2024-01-26T23:46:18.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.522+0000 7f4026ffd700 1 -- 172.21.15.46:0/3057284109 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 41) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f4028003050 con 0x7f4030159cf0 2024-01-26T23:46:18.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.523+0000 7f4026ffd700 1 --2- 172.21.15.46:0/3057284109 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4010067050 0x7f4010069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.523+0000 7f402effd700 1 -- 172.21.15.46:0/3057284109 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4010067050 msgr2=0x7f4010069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:18.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.523+0000 7f4026ffd700 1 -- 172.21.15.46:0/3057284109 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f4028086760 con 0x7f4030159cf0 2024-01-26T23:46:18.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.523+0000 7f402effd700 1 --2- 172.21.15.46:0/3057284109 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4010067050 0x7f4010069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:46:18.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.527+0000 7f4026ffd700 1 -- 172.21.15.46:0/3057284109 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4028053320 con 0x7f4030159cf0 2024-01-26T23:46:18.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.724+0000 7f402effd700 1 -- 172.21.15.46:0/3057284109 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4010067050 msgr2=0x7f4010069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:18.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.724+0000 7f402effd700 1 --2- 172.21.15.46:0/3057284109 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4010067050 0x7f4010069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:46:18.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.782+0000 7f40353b0700 1 -- 172.21.15.46:0/3057284109 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40300a9c20 con 0x7f4030159cf0 2024-01-26T23:46:18.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.783+0000 7f4026ffd700 1 -- 172.21.15.46:0/3057284109 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f4028056940 con 0x7f4030159cf0 2024-01-26T23:46:18.784 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:18.784 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6137,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/1370365478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1370365478},{"type":"v1","addr":"172.21.15.106:6836","nonce":1370365478}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":6619,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/1980309094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1980309094},{"type":"v1","addr":"172.21.15.106:6837","nonce":1980309094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40},{"gid":6628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3910368747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3910368747},{"type":"v1","addr":"172.21.15.46:6834","nonce":3910368747}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:45:52.426977+0000","modified":"2024-01-26T23:45:56.004421+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6631":{"gid":6631,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3176096333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3176096333},{"type":"v1","addr":"172.21.15.46:6835","nonce":3176096333}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2024-01-26T23:46:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.785+0000 7f40353b0700 1 -- 172.21.15.46:0/3057284109 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4010067050 msgr2=0x7f4010069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:46:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.785+0000 7f40353b0700 1 --2- 172.21.15.46:0/3057284109 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4010067050 0x7f4010069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.785+0000 7f40353b0700 1 -- 172.21.15.46:0/3057284109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4030159cf0 msgr2=0x7f40300ad9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.785+0000 7f40353b0700 1 --2- 172.21.15.46:0/3057284109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4030159cf0 0x7f40300ad9c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4028016000 tx=0x7f4028019000 comp rx=0 tx=0).stop 2024-01-26T23:46:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.785+0000 7f40353b0700 1 -- 172.21.15.46:0/3057284109 shutdown_connections 2024-01-26T23:46:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.785+0000 7f40353b0700 1 --2- 172.21.15.46:0/3057284109 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4010067050 0x7f4010069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.785+0000 7f40353b0700 1 --2- 172.21.15.46:0/3057284109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4030159cf0 0x7f40300ad9c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.785+0000 7f40353b0700 1 --2- 172.21.15.46:0/3057284109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f403006abf0 0x7f4030158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.785+0000 7f40353b0700 1 --2- 172.21.15.46:0/3057284109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4030159920 0x7f40300b2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.785+0000 7f40353b0700 1 -- 172.21.15.46:0/3057284109 >> 172.21.15.46:0/3057284109 conn(0x7f40300cd6f0 msgr2=0x7f40300c1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.785+0000 7f40353b0700 1 -- 172.21.15.46:0/3057284109 shutdown_connections 2024-01-26T23:46:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.785+0000 7f40353b0700 1 -- 172.21.15.46:0/3057284109 wait complete. 2024-01-26T23:46:18.788 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 45 2024-01-26T23:46:18.801 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:46:18.876 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:18.875+0000 7f84319161c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:46:18.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.932+0000 7f9da0e39700 1 Processor -- start 2024-01-26T23:46:18.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.932+0000 7f9da0e39700 1 -- start start 2024-01-26T23:46:18.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.934+0000 7f9da0e39700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9d9c0b2b40 0x7f9d9c0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.934+0000 7f9da0e39700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d9c0ae960 0x7f9d9c0ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.934+0000 7f9da0e39700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9c15a0b0 con 0x7f9d9c0aedd0 2024-01-26T23:46:18.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.934+0000 7f9da0e39700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d9c099fc0 con 0x7f9d9c0ae960 2024-01-26T23:46:18.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.934+0000 7f9da0e39700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d9c0ae4d0 con 0x7f9d9c0b2b40 2024-01-26T23:46:18.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.934+0000 7f9d99d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d9c0ae960 0x7f9d9c0ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:18.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.934+0000 7f9d9ad9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9d9c0aedd0 0x7f9d9c0cb5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59678/0 (socket says 172.21.15.46:59678) 2024-01-26T23:46:18.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.934+0000 7f9d9ad9d700 1 -- 172.21.15.46:0/27579492 learned_addr learned my addr 172.21.15.46:0/27579492 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:18.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.934+0000 7f9d9a59c700 1 --2- 172.21.15.46:0/27579492 >> v2:172.21.15.106:3300/0 conn(0x7f9d9c0b2b40 0x7f9d9c0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:18.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.934+0000 7f9d99d9b700 1 -- 172.21.15.46:0/27579492 >> v2:172.21.15.106:3300/0 conn(0x7f9d9c0b2b40 msgr2=0x7f9d9c0ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:18.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.934+0000 7f9d99d9b700 1 --2- 172.21.15.46:0/27579492 >> v2:172.21.15.106:3300/0 conn(0x7f9d9c0b2b40 0x7f9d9c0ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.934+0000 7f9d99d9b700 1 -- 172.21.15.46:0/27579492 >> v1:172.21.15.46:6789/0 conn(0x7f9d9c0aedd0 legacy=0x7f9d9c0cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:18.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.934+0000 7f9d99d9b700 1 -- 172.21.15.46:0/27579492 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c0ad8c0 con 0x7f9d9c0ae960 2024-01-26T23:46:18.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.934+0000 7f9d99d9b700 1 --2- 172.21.15.46:0/27579492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c0ae960 0x7f9d9c0ced50 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c088850 tx=0x7f9d900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b980720859977ea server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:18.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.935+0000 7f9d9959a700 1 -- 172.21.15.46:0/27579492 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d90011630 con 0x7f9d9c0ae960 2024-01-26T23:46:18.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.935+0000 7f9d9959a700 1 -- 172.21.15.46:0/27579492 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d90011790 con 0x7f9d9c0ae960 2024-01-26T23:46:18.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.935+0000 7f9d9959a700 1 -- 172.21.15.46:0/27579492 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d90011b30 con 0x7f9d9c0ae960 2024-01-26T23:46:18.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.935+0000 7f9da0e39700 1 -- 172.21.15.46:0/27579492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c0ae960 msgr2=0x7f9d9c0ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:18.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.935+0000 7f9da0e39700 1 --2- 172.21.15.46:0/27579492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c0ae960 0x7f9d9c0ced50 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c088850 tx=0x7f9d900091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.936+0000 7f9da0e39700 1 -- 172.21.15.46:0/27579492 shutdown_connections 2024-01-26T23:46:18.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.936+0000 7f9da0e39700 1 --2- 172.21.15.46:0/27579492 >> v2:172.21.15.106:3300/0 conn(0x7f9d9c0b2b40 0x7f9d9c0ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.936+0000 7f9da0e39700 1 --2- 172.21.15.46:0/27579492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c0ae960 0x7f9d9c0ced50 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.936+0000 7f9da0e39700 1 -- 172.21.15.46:0/27579492 >> 172.21.15.46:0/27579492 conn(0x7f9d9c099a30 msgr2=0x7f9d9c06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:18.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.936+0000 7f9da0e39700 1 -- 172.21.15.46:0/27579492 shutdown_connections 2024-01-26T23:46:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.936+0000 7f9da0e39700 1 -- 172.21.15.46:0/27579492 wait complete. 2024-01-26T23:46:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.937+0000 7f9da0e39700 1 Processor -- start 2024-01-26T23:46:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.937+0000 7f9da0e39700 1 -- start start 2024-01-26T23:46:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9da0e39700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c0ae960 0x7f9d9c0c9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9da0e39700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d9c0b2b40 0x7f9d9c069640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9da0e39700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d9c151360 0x7f9d9c1517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9da0e39700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c151d20 con 0x7f9d9c0ae960 2024-01-26T23:46:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9da0e39700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0c7820 con 0x7f9d9c151360 2024-01-26T23:46:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9da0e39700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c0b3510 con 0x7f9d9c0b2b40 2024-01-26T23:46:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9d9a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c0ae960 0x7f9d9c0c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9d9a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c0ae960 0x7f9d9c0c9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41218/0 (socket says 172.21.15.46:41218) 2024-01-26T23:46:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9d9a59c700 1 -- 172.21.15.46:0/3080347642 learned_addr learned my addr 172.21.15.46:0/3080347642 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9d99d9b700 1 --2- 172.21.15.46:0/3080347642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d9c0b2b40 0x7f9d9c069640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9d9ad9d700 1 --2- 172.21.15.46:0/3080347642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d9c151360 0x7f9d9c1517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9d9a59c700 1 -- 172.21.15.46:0/3080347642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d9c0b2b40 msgr2=0x7f9d9c069640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9d9a59c700 1 --2- 172.21.15.46:0/3080347642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d9c0b2b40 0x7f9d9c069640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9d9a59c700 1 -- 172.21.15.46:0/3080347642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d9c151360 msgr2=0x7f9d9c1517e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9d9a59c700 1 --2- 172.21.15.46:0/3080347642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d9c151360 0x7f9d9c1517e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9d9a59c700 1 -- 172.21.15.46:0/3080347642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d90009420 con 0x7f9d9c0ae960 2024-01-26T23:46:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9d99d9b700 1 --2- 172.21.15.46:0/3080347642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d9c0b2b40 0x7f9d9c069640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.938+0000 7f9d9a59c700 1 --2- 172.21.15.46:0/3080347642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c0ae960 0x7f9d9c0c9370 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00ecb0 tx=0x7f9d8c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.939+0000 7f9d8a7fc700 1 -- 172.21.15.46:0/3080347642 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c00c850 con 0x7f9d9c0ae960 2024-01-26T23:46:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.939+0000 7f9d8a7fc700 1 -- 172.21.15.46:0/3080347642 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c00c9b0 con 0x7f9d9c0ae960 2024-01-26T23:46:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.939+0000 7f9d8a7fc700 1 -- 172.21.15.46:0/3080347642 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c00cd30 con 0x7f9d9c0ae960 2024-01-26T23:46:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.939+0000 7f9da0e39700 1 -- 172.21.15.46:0/3080347642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d9c0c98b0 con 0x7f9d9c0ae960 2024-01-26T23:46:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.939+0000 7f9da0e39700 1 -- 172.21.15.46:0/3080347642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9c0b3a40 con 0x7f9d9c0ae960 2024-01-26T23:46:18.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.940+0000 7f9d8a7fc700 1 -- 172.21.15.46:0/3080347642 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 41) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f9d8c013680 con 0x7f9d9c0ae960 2024-01-26T23:46:18.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.941+0000 7f9d8a7fc700 1 --2- 172.21.15.46:0/3080347642 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d7c067050 0x7f9d7c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:18.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.941+0000 7f9d99d9b700 1 -- 172.21.15.46:0/3080347642 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d7c067050 msgr2=0x7f9d7c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:18.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.941+0000 7f9d8a7fc700 1 -- 172.21.15.46:0/3080347642 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f9d8c085130 con 0x7f9d9c0ae960 2024-01-26T23:46:18.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.941+0000 7f9da0e39700 1 -- 172.21.15.46:0/3080347642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9c04c710 con 0x7f9d9c0ae960 2024-01-26T23:46:18.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.941+0000 7f9d99d9b700 1 --2- 172.21.15.46:0/3080347642 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d7c067050 0x7f9d7c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:46:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:18.946+0000 7f9d8a7fc700 1 -- 172.21.15.46:0/3080347642 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9d8c021080 con 0x7f9d9c0ae960 2024-01-26T23:46:19.053 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:19.052+0000 7f84319161c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:46:19.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.142+0000 7f9d99d9b700 1 -- 172.21.15.46:0/3080347642 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d7c067050 msgr2=0x7f9d7c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:19.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.142+0000 7f9d99d9b700 1 --2- 172.21.15.46:0/3080347642 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d7c067050 0x7f9d7c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:46:19.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.150+0000 7f9da0e39700 1 -- 172.21.15.46:0/3080347642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d9c0682c0 con 0x7f9d9c0ae960 2024-01-26T23:46:19.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.151+0000 7f9d8a7fc700 1 -- 172.21.15.46:0/3080347642 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16587 (secure 0 0 0) 0x7f9d8c017020 con 0x7f9d9c0ae960 2024-01-26T23:46:19.152 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:19.152 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":100,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:46:12.597656+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:45:50.922223+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-01-26T23:45:51.397995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/2701612997":"2024-01-27T23:46:12.597621+0000","172.21.15.106:0/1894478014":"2024-01-27T23:46:12.597621+0000","172.21.15.106:0/623871419":"2024-01-27T23:46:12.597621+0000","172.21.15.106:6833/43388":"2024-01-27T23:46:12.597621+0000","172.21.15.106:6832/43388":"2024-01-27T23:46:12.597621+0000","172.21.15.106:0/243105601":"2024-01-27T23:46:12.597621+0000","172.21.15.106:0/3064987359":"2024-01-27T00:46:05.889834+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:46:19.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.153+0000 7f9da0e39700 1 -- 172.21.15.46:0/3080347642 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d7c067050 msgr2=0x7f9d7c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:46:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.153+0000 7f9da0e39700 1 --2- 172.21.15.46:0/3080347642 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d7c067050 0x7f9d7c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.153+0000 7f9da0e39700 1 -- 172.21.15.46:0/3080347642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c0ae960 msgr2=0x7f9d9c0c9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.153+0000 7f9da0e39700 1 --2- 172.21.15.46:0/3080347642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c0ae960 0x7f9d9c0c9370 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00ecb0 tx=0x7f9d8c009120 comp rx=0 tx=0).stop 2024-01-26T23:46:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.153+0000 7f9da0e39700 1 -- 172.21.15.46:0/3080347642 shutdown_connections 2024-01-26T23:46:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.153+0000 7f9da0e39700 1 --2- 172.21.15.46:0/3080347642 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d7c067050 0x7f9d7c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.153+0000 7f9da0e39700 1 --2- 172.21.15.46:0/3080347642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c0ae960 0x7f9d9c0c9370 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.153+0000 7f9da0e39700 1 --2- 172.21.15.46:0/3080347642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d9c151360 0x7f9d9c1517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.153+0000 7f9da0e39700 1 --2- 172.21.15.46:0/3080347642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d9c0b2b40 0x7f9d9c069640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.153+0000 7f9da0e39700 1 -- 172.21.15.46:0/3080347642 >> 172.21.15.46:0/3080347642 conn(0x7f9d9c099a30 msgr2=0x7f9d9c158f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.153+0000 7f9da0e39700 1 -- 172.21.15.46:0/3080347642 shutdown_connections 2024-01-26T23:46:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.153+0000 7f9da0e39700 1 -- 172.21.15.46:0/3080347642 wait complete. 2024-01-26T23:46:19.156 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:19.155+0000 7fafd9abe1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:46:19.169 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-26T23:46:19.231 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:19.230+0000 7f84319161c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:46:19.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.302+0000 7fc8c45bb700 1 Processor -- start 2024-01-26T23:46:19.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.302+0000 7fc8c45bb700 1 -- start start 2024-01-26T23:46:19.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.303+0000 7fc8c45bb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc8bc0c7740 0x7fc8bc0c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:19.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.303+0000 7fc8c45bb700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc0cf650 con 0x7fc8bc159bd0 2024-01-26T23:46:19.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.303+0000 7fc8c45bb700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc0cea40 con 0x7fc8bc0cce30 2024-01-26T23:46:19.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.303+0000 7fc8c45bb700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8bc0ced20 con 0x7fc8bc0c7740 2024-01-26T23:46:19.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.304+0000 7fc8c2b58700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc8bc0c7740 0x7fc8bc0c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:19.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.304+0000 7fc8c2357700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc8bc159bd0 0x7fc8bc0ce260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59686/0 (socket says 172.21.15.46:59686) 2024-01-26T23:46:19.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.304+0000 7fc8c2357700 1 -- 172.21.15.46:0/3103327716 learned_addr learned my addr 172.21.15.46:0/3103327716 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:19.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.304+0000 7fc8c2b58700 1 -- 172.21.15.46:0/3103327716 >> v1:172.21.15.106:6789/0 conn(0x7fc8bc0cce30 legacy=0x7fc8bc0cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:19.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.304+0000 7fc8c2b58700 1 -- 172.21.15.46:0/3103327716 >> v1:172.21.15.46:6789/0 conn(0x7fc8bc159bd0 legacy=0x7fc8bc0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:19.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.304+0000 7fc8c2b58700 1 -- 172.21.15.46:0/3103327716 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc0cf000 con 0x7fc8bc0c7740 2024-01-26T23:46:19.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.304+0000 7fc8c2b58700 1 --2- 172.21.15.46:0/3103327716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8bc0c7740 0x7fc8bc0c7b60 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4009e60 tx=0x7fc8b4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c6c236a2d762c2 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:19.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.304+0000 7fc8c1355700 1 -- 172.21.15.46:0/3103327716 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b400e070 con 0x7fc8bc0c7740 2024-01-26T23:46:19.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.304+0000 7fc8c1355700 1 -- 172.21.15.46:0/3103327716 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b400bc30 con 0x7fc8bc0c7740 2024-01-26T23:46:19.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.304+0000 7fc8c1355700 1 -- 172.21.15.46:0/3103327716 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b4012430 con 0x7fc8bc0c7740 2024-01-26T23:46:19.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.305+0000 7fc8c45bb700 1 -- 172.21.15.46:0/3103327716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8bc0c7740 msgr2=0x7fc8bc0c7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:19.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.305+0000 7fc8c45bb700 1 --2- 172.21.15.46:0/3103327716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8bc0c7740 0x7fc8bc0c7b60 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4009e60 tx=0x7fc8b4009780 comp rx=0 tx=0).stop 2024-01-26T23:46:19.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.305+0000 7fc8c45bb700 1 -- 172.21.15.46:0/3103327716 shutdown_connections 2024-01-26T23:46:19.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.305+0000 7fc8c45bb700 1 --2- 172.21.15.46:0/3103327716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8bc0c7740 0x7fc8bc0c7b60 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.305+0000 7fc8c45bb700 1 -- 172.21.15.46:0/3103327716 >> 172.21.15.46:0/3103327716 conn(0x7fc8bc0999e0 msgr2=0x7fc8bc06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:19.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.305+0000 7fc8c45bb700 1 -- 172.21.15.46:0/3103327716 shutdown_connections 2024-01-26T23:46:19.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.305+0000 7fc8c45bb700 1 -- 172.21.15.46:0/3103327716 wait complete. 2024-01-26T23:46:19.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.307+0000 7fc8c45bb700 1 Processor -- start 2024-01-26T23:46:19.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.307+0000 7fc8c45bb700 1 -- start start 2024-01-26T23:46:19.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.308+0000 7fc8c45bb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8bc0c7740 0x7fc8bc0b5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:19.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.308+0000 7fc8c45bb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8bc0cce30 0x7fc8bc16ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:19.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.308+0000 7fc8c45bb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8bc159bd0 0x7fc8bc1575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:19.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.308+0000 7fc8c45bb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc1514a0 con 0x7fc8bc0c7740 2024-01-26T23:46:19.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.308+0000 7fc8c45bb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc156ad0 con 0x7fc8bc0cce30 2024-01-26T23:46:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.308+0000 7fc8c45bb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc155c50 con 0x7fc8bc159bd0 2024-01-26T23:46:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.308+0000 7fc8c2357700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8bc0c7740 0x7fc8bc0b5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.308+0000 7fc8c2357700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8bc0c7740 0x7fc8bc0b5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41232/0 (socket says 172.21.15.46:41232) 2024-01-26T23:46:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.308+0000 7fc8c2357700 1 -- 172.21.15.46:0/3131333119 learned_addr learned my addr 172.21.15.46:0/3131333119 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.308+0000 7fc8c1b56700 1 --2- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8bc0cce30 0x7fc8bc16ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.308+0000 7fc8c2b58700 1 --2- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8bc159bd0 0x7fc8bc1575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.309+0000 7fc8c1b56700 1 -- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8bc159bd0 msgr2=0x7fc8bc1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.309+0000 7fc8c1b56700 1 --2- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8bc159bd0 0x7fc8bc1575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.309+0000 7fc8c1b56700 1 -- 172.21.15.46:0/3131333119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8bc0c7740 msgr2=0x7fc8bc0b5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.309+0000 7fc8c1b56700 1 --2- 172.21.15.46:0/3131333119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8bc0c7740 0x7fc8bc0b5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.309+0000 7fc8c1b56700 1 -- 172.21.15.46:0/3131333119 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b4009a20 con 0x7fc8bc0cce30 2024-01-26T23:46:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.309+0000 7fc8c2b58700 1 --2- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8bc159bd0 0x7fc8bc1575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.309+0000 7fc8c1b56700 1 --2- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8bc0cce30 0x7fc8bc16ab60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac00c5c0 tx=0x7fc8ac007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:19.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.309+0000 7fc8b27fc700 1 -- 172.21.15.46:0/3131333119 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac011590 con 0x7fc8bc0cce30 2024-01-26T23:46:19.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.309+0000 7fc8b27fc700 1 -- 172.21.15.46:0/3131333119 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ac011710 con 0x7fc8bc0cce30 2024-01-26T23:46:19.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.309+0000 7fc8c45bb700 1 -- 172.21.15.46:0/3131333119 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8bc16b0a0 con 0x7fc8bc0cce30 2024-01-26T23:46:19.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.309+0000 7fc8b27fc700 1 -- 172.21.15.46:0/3131333119 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac011ad0 con 0x7fc8bc0cce30 2024-01-26T23:46:19.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.309+0000 7fc8c45bb700 1 -- 172.21.15.46:0/3131333119 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8bc1561a0 con 0x7fc8bc0cce30 2024-01-26T23:46:19.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.310+0000 7fc8c45bb700 1 -- 172.21.15.46:0/3131333119 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8bc04c710 con 0x7fc8bc0cce30 2024-01-26T23:46:19.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.311+0000 7fc8b27fc700 1 -- 172.21.15.46:0/3131333119 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 41) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fc8ac01b3b0 con 0x7fc8bc0cce30 2024-01-26T23:46:19.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.311+0000 7fc8b27fc700 1 --2- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc8a0067080 0x7fc8a0069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:19.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.311+0000 7fc8b27fc700 1 -- 172.21.15.46:0/3131333119 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fc8ac083e40 con 0x7fc8bc0cce30 2024-01-26T23:46:19.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.311+0000 7fc8c2357700 1 -- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc8a0067080 msgr2=0x7fc8a0069530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:19.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.311+0000 7fc8c2357700 1 --2- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc8a0067080 0x7fc8a0069530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:46:19.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.316+0000 7fc8b27fc700 1 -- 172.21.15.46:0/3131333119 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc8ac050bb0 con 0x7fc8bc0cce30 2024-01-26T23:46:19.374 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:19.375+0000 7fafd9abe1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:46:19.448 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:19.448+0000 7fafd9abe1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:46:19.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.512+0000 7fc8c2357700 1 -- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc8a0067080 msgr2=0x7fc8a0069530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:19.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.512+0000 7fc8c2357700 1 --2- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc8a0067080 0x7fc8a0069530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:46:19.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.584+0000 7fc8c45bb700 1 -- 172.21.15.46:0/3131333119 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc8bc0c88e0 con 0x7fc8bc0cce30 2024-01-26T23:46:19.586 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-26T23:46:19.585+0000 7f33ffe86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-26T23:46:19.592 INFO:tasks.ceph.mds.c.smithi046.stderr: -1374> 2024-01-26T23:45:54.919+0000 7f274a58b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-26T23:46:19.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.592+0000 7fc8b27fc700 1 -- 172.21.15.46:0/3131333119 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v46) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc8ac017070 con 0x7fc8bc0cce30 2024-01-26T23:46:19.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.594+0000 7fc8c45bb700 1 -- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc8a0067080 msgr2=0x7fc8a0069530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:46:19.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.594+0000 7fc8c45bb700 1 --2- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc8a0067080 0x7fc8a0069530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.594+0000 7fc8c45bb700 1 -- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8bc0cce30 msgr2=0x7fc8bc16ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:19.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.594+0000 7fc8c45bb700 1 --2- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8bc0cce30 0x7fc8bc16ab60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac00c5c0 tx=0x7fc8ac007ec0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.595+0000 7fc8c45bb700 1 -- 172.21.15.46:0/3131333119 shutdown_connections 2024-01-26T23:46:19.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.595+0000 7fc8c45bb700 1 --2- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc8a0067080 0x7fc8a0069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.595+0000 7fc8c45bb700 1 --2- 172.21.15.46:0/3131333119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8bc0c7740 0x7fc8bc0b5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.595+0000 7fc8c45bb700 1 --2- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8bc159bd0 0x7fc8bc1575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.595+0000 7fc8c45bb700 1 --2- 172.21.15.46:0/3131333119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8bc0cce30 0x7fc8bc16ab60 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.595+0000 7fc8c45bb700 1 -- 172.21.15.46:0/3131333119 >> 172.21.15.46:0/3131333119 conn(0x7fc8bc0999e0 msgr2=0x7fc8bc0ca980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:19.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.595+0000 7fc8c45bb700 1 -- 172.21.15.46:0/3131333119 shutdown_connections 2024-01-26T23:46:19.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.595+0000 7fc8c45bb700 1 -- 172.21.15.46:0/3131333119 wait complete. 2024-01-26T23:46:19.597 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-26T23:46:19.604 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:19.604+0000 7fafd9abe1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:46:19.611 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-26T23:46:19.617 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-26T23:46:19.688 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:19.688+0000 7faf58b6d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:19.688 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:19.688+0000 7faf58b6d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:19.688 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:19.688+0000 7faf58b6d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:19.688 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:19.688+0000 7faf58b6d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:19.688 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:19.688+0000 7faf58b6d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:19.689 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:19.689+0000 7faf55b67700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:19.689 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:19.689+0000 7faf55b67700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:19.689 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:19.689+0000 7faf55b67700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:19.689 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:19.689+0000 7faf55b67700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:19.689 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:19.689+0000 7faf55b67700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:19.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.761+0000 7f9757179700 1 Processor -- start 2024-01-26T23:46:19.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.761+0000 7f9757179700 1 -- start start 2024-01-26T23:46:19.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.762+0000 7f9757179700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97500cf050 0x7f97500af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:19.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.762+0000 7f9757179700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f97500cf4a0 0x7f97500af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:19.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.762+0000 7f9757179700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97500cc100 con 0x7f9750159cb0 2024-01-26T23:46:19.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.762+0000 7f9757179700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97500ccca0 con 0x7f97500cf4a0 2024-01-26T23:46:19.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.762+0000 7f9757179700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97500cb850 con 0x7f97500cf050 2024-01-26T23:46:19.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.762+0000 7f9754f15700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9750159cb0 0x7f97500cc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59702/0 (socket says 172.21.15.46:59702) 2024-01-26T23:46:19.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.762+0000 7f9754f15700 1 -- 172.21.15.46:0/1031690367 learned_addr learned my addr 172.21.15.46:0/1031690367 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:19.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.762+0000 7f9747fff700 1 --2- 172.21.15.46:0/1031690367 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97500cf050 0x7f97500af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:19.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.762+0000 7f9755716700 1 --2- 172.21.15.46:0/1031690367 >> v2:172.21.15.106:3300/0 conn(0x7f97500cf4a0 0x7f97500af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:19.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.763+0000 7f97477fe700 1 -- 172.21.15.46:0/1031690367 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801113239 0 0) 0x7f97500cc100 con 0x7f9750159cb0 2024-01-26T23:46:19.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.763+0000 7f97477fe700 1 -- 172.21.15.46:0/1031690367 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9738003570 con 0x7f9750159cb0 2024-01-26T23:46:19.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.763+0000 7f9755716700 1 -- 172.21.15.46:0/1031690367 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97500cf050 msgr2=0x7f97500af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:19.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.763+0000 7f9755716700 1 --2- 172.21.15.46:0/1031690367 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97500cf050 0x7f97500af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.763+0000 7f9755716700 1 -- 172.21.15.46:0/1031690367 >> v1:172.21.15.46:6789/0 conn(0x7f9750159cb0 legacy=0x7f97500cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:19.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.763+0000 7f9755716700 1 -- 172.21.15.46:0/1031690367 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97500cbaf0 con 0x7f97500cf4a0 2024-01-26T23:46:19.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.763+0000 7f9747fff700 1 --2- 172.21.15.46:0/1031690367 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97500cf050 0x7f97500af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:19.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.763+0000 7f9755716700 1 --2- 172.21.15.46:0/1031690367 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97500cf4a0 0x7f97500af970 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9748009e60 tx=0x7f9748009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25f5df8eca7934ed server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:19.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.763+0000 7f97477fe700 1 -- 172.21.15.46:0/1031690367 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974800e070 con 0x7f97500cf4a0 2024-01-26T23:46:19.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.763+0000 7f97477fe700 1 -- 172.21.15.46:0/1031690367 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974800bab0 con 0x7f97500cf4a0 2024-01-26T23:46:19.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.763+0000 7f97477fe700 1 -- 172.21.15.46:0/1031690367 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974800be70 con 0x7f97500cf4a0 2024-01-26T23:46:19.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.764+0000 7f9757179700 1 -- 172.21.15.46:0/1031690367 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97500cf4a0 msgr2=0x7f97500af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:19.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.764+0000 7f9757179700 1 --2- 172.21.15.46:0/1031690367 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97500cf4a0 0x7f97500af970 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9748009e60 tx=0x7f9748009780 comp rx=0 tx=0).stop 2024-01-26T23:46:19.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.764+0000 7f9757179700 1 -- 172.21.15.46:0/1031690367 shutdown_connections 2024-01-26T23:46:19.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.764+0000 7f9757179700 1 --2- 172.21.15.46:0/1031690367 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97500cf4a0 0x7f97500af970 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.764+0000 7f9757179700 1 --2- 172.21.15.46:0/1031690367 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97500cf050 0x7f97500af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.764+0000 7f9757179700 1 -- 172.21.15.46:0/1031690367 >> 172.21.15.46:0/1031690367 conn(0x7f9750087f60 msgr2=0x7f975006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:19.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.764+0000 7f9757179700 1 -- 172.21.15.46:0/1031690367 shutdown_connections 2024-01-26T23:46:19.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.764+0000 7f9757179700 1 -- 172.21.15.46:0/1031690367 wait complete. 2024-01-26T23:46:19.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.765+0000 7f9757179700 1 Processor -- start 2024-01-26T23:46:19.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.765+0000 7f9757179700 1 -- start start 2024-01-26T23:46:19.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.766+0000 7f9757179700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97500cf050 0x7f97500b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:19.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.766+0000 7f9757179700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97500cf4a0 0x7f9750153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:19.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.766+0000 7f9757179700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9750159cb0 0x7f97500c5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:19.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.766+0000 7f9757179700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97500b3810 con 0x7f97500cf4a0 2024-01-26T23:46:19.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.766+0000 7f9757179700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97500b5270 con 0x7f97500cf050 2024-01-26T23:46:19.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.766+0000 7f9757179700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97500b5530 con 0x7f9750159cb0 2024-01-26T23:46:19.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.766+0000 7f9747fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97500cf4a0 0x7f9750153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:19.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.766+0000 7f9754f15700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97500cf050 0x7f97500b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:19.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.766+0000 7f9747fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97500cf4a0 0x7f9750153240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41250/0 (socket says 172.21.15.46:41250) 2024-01-26T23:46:19.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.766+0000 7f9747fff700 1 -- 172.21.15.46:0/2973235065 learned_addr learned my addr 172.21.15.46:0/2973235065 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:19.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.766+0000 7f9755716700 1 --2- 172.21.15.46:0/2973235065 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9750159cb0 0x7f97500c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:19.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.767+0000 7f9754f15700 1 -- 172.21.15.46:0/2973235065 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9750159cb0 msgr2=0x7f97500c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:19.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.767+0000 7f9754f15700 1 --2- 172.21.15.46:0/2973235065 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9750159cb0 0x7f97500c5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.767+0000 7f9754f15700 1 -- 172.21.15.46:0/2973235065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97500cf4a0 msgr2=0x7f9750153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:19.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.767+0000 7f9754f15700 1 --2- 172.21.15.46:0/2973235065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97500cf4a0 0x7f9750153240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:19.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.767+0000 7f9754f15700 1 -- 172.21.15.46:0/2973235065 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9740009420 con 0x7f97500cf050 2024-01-26T23:46:19.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.767+0000 7f9755716700 1 --2- 172.21.15.46:0/2973235065 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9750159cb0 0x7f97500c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:19.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.767+0000 7f9747fff700 1 --2- 172.21.15.46:0/2973235065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97500cf4a0 0x7f9750153240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:19.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.767+0000 7f9754f15700 1 --2- 172.21.15.46:0/2973235065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97500cf050 0x7f97500b4d30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f974c00c790 tx=0x7f974c007bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:19.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.767+0000 7f9744ff9700 1 -- 172.21.15.46:0/2973235065 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c0072e0 con 0x7f97500cf050 2024-01-26T23:46:19.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.767+0000 7f9744ff9700 1 -- 172.21.15.46:0/2973235065 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974c007440 con 0x7f97500cf050 2024-01-26T23:46:19.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.767+0000 7f9757179700 1 -- 172.21.15.46:0/2973235065 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9748009a20 con 0x7f97500cf050 2024-01-26T23:46:19.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.767+0000 7f9744ff9700 1 -- 172.21.15.46:0/2973235065 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c0057f0 con 0x7f97500cf050 2024-01-26T23:46:19.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.767+0000 7f9757179700 1 -- 172.21.15.46:0/2973235065 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97500b5af0 con 0x7f97500cf050 2024-01-26T23:46:19.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.768+0000 7f9757179700 1 -- 172.21.15.46:0/2973235065 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97500aff40 con 0x7f97500cf050 2024-01-26T23:46:19.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.769+0000 7f9744ff9700 1 -- 172.21.15.46:0/2973235065 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 42) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f974c005c40 con 0x7f97500cf050 2024-01-26T23:46:19.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.769+0000 7f9744ff9700 1 -- 172.21.15.46:0/2973235065 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 7062+0+0 (secure 0 0 0) 0x7f974c0838f0 con 0x7f97500cf050 2024-01-26T23:46:19.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:19.774+0000 7f9744ff9700 1 -- 172.21.15.46:0/2973235065 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f97500aff40 con 0x7f97500cf050 2024-01-26T23:46:20.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.028+0000 7f9757179700 1 -- 172.21.15.46:0/2973235065 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f97500cfd10 con 0x7f97500cf050 2024-01-26T23:46:20.085 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:20.084+0000 7f84319161c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:46:20.338 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:20.337+0000 7f84319161c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:46:20.423 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:20.421+0000 7f84319161c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:46:20.603 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:20.601+0000 7f84319161c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:46:20.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.622+0000 7f9744ff9700 1 -- 172.21.15.46:0/2973235065 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v47) v1 ==== 104+0+0 (secure 0 0 0) 0x7f974c0509d0 con 0x7f97500cf050 2024-01-26T23:46:20.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.625+0000 7f9757179700 1 -- 172.21.15.46:0/2973235065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97500cf050 msgr2=0x7f97500b4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:20.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.625+0000 7f9757179700 1 --2- 172.21.15.46:0/2973235065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97500cf050 0x7f97500b4d30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f974c00c790 tx=0x7f974c007bc0 comp rx=0 tx=0).stop 2024-01-26T23:46:20.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.625+0000 7f9757179700 1 -- 172.21.15.46:0/2973235065 shutdown_connections 2024-01-26T23:46:20.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.625+0000 7f9757179700 1 --2- 172.21.15.46:0/2973235065 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9750159cb0 0x7f97500c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:20.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.625+0000 7f9757179700 1 --2- 172.21.15.46:0/2973235065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97500cf050 0x7f97500b4d30 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:20.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.625+0000 7f9757179700 1 --2- 172.21.15.46:0/2973235065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97500cf4a0 0x7f9750153240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:20.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.625+0000 7f9757179700 1 -- 172.21.15.46:0/2973235065 >> 172.21.15.46:0/2973235065 conn(0x7f9750087f60 msgr2=0x7f97500af210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:20.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.625+0000 7f9757179700 1 -- 172.21.15.46:0/2973235065 shutdown_connections 2024-01-26T23:46:20.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.625+0000 7f9757179700 1 -- 172.21.15.46:0/2973235065 wait complete. 2024-01-26T23:46:20.644 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-26T23:46:20.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.821+0000 7fa9c62da700 1 Processor -- start 2024-01-26T23:46:20.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.821+0000 7fa9c62da700 1 -- start start 2024-01-26T23:46:20.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.822+0000 7fa9c62da700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa9c0159630 0x7fa9c0159a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.822+0000 7fa9c62da700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c015ab40 con 0x7fa9c0067930 2024-01-26T23:46:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.822+0000 7fa9c62da700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c0159f40 con 0x7fa9c00af570 2024-01-26T23:46:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.822+0000 7fa9c62da700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c015a210 con 0x7fa9c0159630 2024-01-26T23:46:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.822+0000 7fa9c4877700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa9c0067930 0x7fa9c0067d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37626/0 (socket says 172.21.15.46:37626) 2024-01-26T23:46:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.822+0000 7fa9c4877700 1 -- 172.21.15.46:0/3326732756 learned_addr learned my addr 172.21.15.46:0/3326732756 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.823+0000 7fa9bffff700 1 --2- 172.21.15.46:0/3326732756 >> v2:172.21.15.106:3300/0 conn(0x7fa9c0159630 0x7fa9c0159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.823+0000 7fa9beffd700 1 -- 172.21.15.46:0/3326732756 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202682642 0 0) 0x7fa9c015ab40 con 0x7fa9c0067930 2024-01-26T23:46:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.823+0000 7fa9beffd700 1 -- 172.21.15.46:0/3326732756 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a8003570 con 0x7fa9c0067930 2024-01-26T23:46:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.823+0000 7fa9beffd700 1 -- 172.21.15.46:0/3326732756 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561541560 0 0) 0x7fa9c0159f40 con 0x7fa9c00af570 2024-01-26T23:46:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.823+0000 7fa9beffd700 1 -- 172.21.15.46:0/3326732756 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c015ab40 con 0x7fa9c00af570 2024-01-26T23:46:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.823+0000 7fa9bffff700 1 -- 172.21.15.46:0/3326732756 >> v1:172.21.15.106:6789/0 conn(0x7fa9c00af570 legacy=0x7fa9c00b2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.823+0000 7fa9bffff700 1 -- 172.21.15.46:0/3326732756 >> v1:172.21.15.46:6789/0 conn(0x7fa9c0067930 legacy=0x7fa9c0067d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:20.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.823+0000 7fa9bffff700 1 -- 172.21.15.46:0/3326732756 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c015a4f0 con 0x7fa9c0159630 2024-01-26T23:46:20.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.823+0000 7fa9bffff700 1 --2- 172.21.15.46:0/3326732756 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9c0159630 0x7fa9c0159a00 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa9b00098e0 tx=0x7fa9b00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fd10888a9cbfb7c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:20.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.823+0000 7fa9beffd700 1 -- 172.21.15.46:0/3326732756 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0011630 con 0x7fa9c0159630 2024-01-26T23:46:20.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.823+0000 7fa9beffd700 1 -- 172.21.15.46:0/3326732756 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b0011790 con 0x7fa9c0159630 2024-01-26T23:46:20.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.823+0000 7fa9beffd700 1 -- 172.21.15.46:0/3326732756 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0011b30 con 0x7fa9c0159630 2024-01-26T23:46:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.824+0000 7fa9c62da700 1 -- 172.21.15.46:0/3326732756 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9c0159630 msgr2=0x7fa9c0159a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.824+0000 7fa9c62da700 1 --2- 172.21.15.46:0/3326732756 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9c0159630 0x7fa9c0159a00 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa9b00098e0 tx=0x7fa9b00091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.824+0000 7fa9c62da700 1 -- 172.21.15.46:0/3326732756 shutdown_connections 2024-01-26T23:46:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.824+0000 7fa9c62da700 1 --2- 172.21.15.46:0/3326732756 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9c0159630 0x7fa9c0159a00 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.824+0000 7fa9c62da700 1 -- 172.21.15.46:0/3326732756 >> 172.21.15.46:0/3326732756 conn(0x7fa9c0069510 msgr2=0x7fa9c00aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.824+0000 7fa9c62da700 1 -- 172.21.15.46:0/3326732756 shutdown_connections 2024-01-26T23:46:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.824+0000 7fa9c62da700 1 -- 172.21.15.46:0/3326732756 wait complete. 2024-01-26T23:46:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.825+0000 7fa9c62da700 1 Processor -- start 2024-01-26T23:46:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.825+0000 7fa9c62da700 1 -- start start 2024-01-26T23:46:20.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.826+0000 7fa9c62da700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9c0067930 0x7fa9c00ab980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:20.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.826+0000 7fa9c62da700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9c00af570 0x7fa9c0168eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:20.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.826+0000 7fa9c62da700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9c0159630 0x7fa9c0158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:20.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.826+0000 7fa9c62da700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c00ab4f0 con 0x7fa9c0159630 2024-01-26T23:46:20.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.826+0000 7fa9c62da700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c00abec0 con 0x7fa9c00af570 2024-01-26T23:46:20.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.826+0000 7fa9c62da700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c00aa8e0 con 0x7fa9c0067930 2024-01-26T23:46:20.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.826+0000 7fa9c4877700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9c0159630 0x7fa9c0158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:20.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.826+0000 7fa9bffff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9c0067930 0x7fa9c00ab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:20.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.826+0000 7fa9bffff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9c0067930 0x7fa9c00ab980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:32992/0 (socket says 172.21.15.46:32992) 2024-01-26T23:46:20.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.826+0000 7fa9c4877700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9c0159630 0x7fa9c0158300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46430/0 (socket says 172.21.15.46:46430) 2024-01-26T23:46:20.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.826+0000 7fa9bffff700 1 -- 172.21.15.46:0/854908163 learned_addr learned my addr 172.21.15.46:0/854908163 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:20.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.826+0000 7fa9bf7fe700 1 --2- 172.21.15.46:0/854908163 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9c00af570 0x7fa9c0168eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:20.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.827+0000 7fa9c4877700 1 -- 172.21.15.46:0/854908163 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9c0067930 msgr2=0x7fa9c00ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:20.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.827+0000 7fa9c4877700 1 --2- 172.21.15.46:0/854908163 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9c0067930 0x7fa9c00ab980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:20.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.827+0000 7fa9c4877700 1 -- 172.21.15.46:0/854908163 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9c00af570 msgr2=0x7fa9c0168eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:20.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.827+0000 7fa9c4877700 1 --2- 172.21.15.46:0/854908163 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9c00af570 0x7fa9c0168eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:20.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.827+0000 7fa9c4877700 1 -- 172.21.15.46:0/854908163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b0009420 con 0x7fa9c0159630 2024-01-26T23:46:20.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.827+0000 7fa9bffff700 1 --2- 172.21.15.46:0/854908163 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9c0067930 0x7fa9c00ab980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:20.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.827+0000 7fa9c4877700 1 --2- 172.21.15.46:0/854908163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9c0159630 0x7fa9c0158300 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8001460 tx=0x7fa9b8001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:20.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.827+0000 7fa9a7fff700 1 -- 172.21.15.46:0/854908163 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b800a840 con 0x7fa9c0159630 2024-01-26T23:46:20.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.827+0000 7fa9a7fff700 1 -- 172.21.15.46:0/854908163 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b800a9a0 con 0x7fa9c0159630 2024-01-26T23:46:20.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.827+0000 7fa9c62da700 1 -- 172.21.15.46:0/854908163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c00aabc0 con 0x7fa9c0159630 2024-01-26T23:46:20.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.827+0000 7fa9a7fff700 1 -- 172.21.15.46:0/854908163 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b800ad60 con 0x7fa9c0159630 2024-01-26T23:46:20.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.827+0000 7fa9c62da700 1 -- 172.21.15.46:0/854908163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c00ab180 con 0x7fa9c0159630 2024-01-26T23:46:20.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.828+0000 7fa9c62da700 1 -- 172.21.15.46:0/854908163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c004c710 con 0x7fa9c0159630 2024-01-26T23:46:20.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.829+0000 7fa9a7fff700 1 -- 172.21.15.46:0/854908163 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 43) v1 ==== 84580+0+0 (secure 0 0 0) 0x7fa9b8007300 con 0x7fa9c0159630 2024-01-26T23:46:20.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.829+0000 7fa9a7fff700 1 --2- 172.21.15.46:0/854908163 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9ac066d80 0x7fa9ac069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:20.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.829+0000 7fa9a7fff700 1 -- 172.21.15.46:0/854908163 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 7062+0+0 (secure 0 0 0) 0x7fa9b8084a40 con 0x7fa9c0159630 2024-01-26T23:46:20.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.830+0000 7fa9bffff700 1 --2- 172.21.15.46:0/854908163 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9ac066d80 0x7fa9ac069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:20.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.830+0000 7fa9bffff700 1 --2- 172.21.15.46:0/854908163 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9ac066d80 0x7fa9ac069230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0011fd0 tx=0x7fa9b0017000 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:20.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:20.834+0000 7fa9a7fff700 1 -- 172.21.15.46:0/854908163 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa9b8051a10 con 0x7fa9c0159630 2024-01-26T23:46:21.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.038+0000 7fa9c62da700 1 -- 172.21.15.46:0/854908163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa9c006a310 con 0x7fa9c0159630 2024-01-26T23:46:21.666 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:21.665+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:21.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.673+0000 7fa9a7fff700 1 -- 172.21.15.46:0/854908163 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v103) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa9b8055030 con 0x7fa9c0159630 2024-01-26T23:46:21.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.677+0000 7fa9c62da700 1 -- 172.21.15.46:0/854908163 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9ac066d80 msgr2=0x7fa9ac069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:21.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.677+0000 7fa9c62da700 1 --2- 172.21.15.46:0/854908163 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9ac066d80 0x7fa9ac069230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0011fd0 tx=0x7fa9b0017000 comp rx=0 tx=0).stop 2024-01-26T23:46:21.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.677+0000 7fa9c62da700 1 -- 172.21.15.46:0/854908163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9c0159630 msgr2=0x7fa9c0158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:21.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.677+0000 7fa9c62da700 1 --2- 172.21.15.46:0/854908163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9c0159630 0x7fa9c0158300 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8001460 tx=0x7fa9b8001a40 comp rx=0 tx=0).stop 2024-01-26T23:46:21.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.678+0000 7fa9c62da700 1 -- 172.21.15.46:0/854908163 shutdown_connections 2024-01-26T23:46:21.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.678+0000 7fa9c62da700 1 --2- 172.21.15.46:0/854908163 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa9ac066d80 0x7fa9ac069230 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:21.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.678+0000 7fa9c62da700 1 --2- 172.21.15.46:0/854908163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9c0159630 0x7fa9c0158300 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:21.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.678+0000 7fa9c62da700 1 --2- 172.21.15.46:0/854908163 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa9c0067930 0x7fa9c00ab980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:21.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.678+0000 7fa9c62da700 1 --2- 172.21.15.46:0/854908163 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9c00af570 0x7fa9c0168eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:21.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.678+0000 7fa9c62da700 1 -- 172.21.15.46:0/854908163 >> 172.21.15.46:0/854908163 conn(0x7fa9c0069510 msgr2=0x7fa9c006eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:21.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.678+0000 7fa9c62da700 1 -- 172.21.15.46:0/854908163 shutdown_connections 2024-01-26T23:46:21.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.678+0000 7fa9c62da700 1 -- 172.21.15.46:0/854908163 wait complete. 2024-01-26T23:46:21.681 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-01-26T23:46:21.696 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-26T23:46:21.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.840+0000 7f8155aa3700 1 Processor -- start 2024-01-26T23:46:21.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.840+0000 7f8155aa3700 1 -- start start 2024-01-26T23:46:21.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.841+0000 7f8155aa3700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f81500ca0e0 0x7f81500ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:21.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.841+0000 7f8155aa3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f815006c0b0 0x7f81500afe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:21.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.841+0000 7f8155aa3700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8150158010 con 0x7f815006c520 2024-01-26T23:46:21.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.841+0000 7f8155aa3700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81500cada0 con 0x7f81500ca0e0 2024-01-26T23:46:21.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.841+0000 7f8155aa3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81501582f0 con 0x7f815006c0b0 2024-01-26T23:46:21.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.841+0000 7f814ffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f815006c520 0x7f81500b3a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37636/0 (socket says 172.21.15.46:37636) 2024-01-26T23:46:21.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.841+0000 7f814ffff700 1 -- 172.21.15.46:0/1204781715 learned_addr learned my addr 172.21.15.46:0/1204781715 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:21.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.841+0000 7f814f7fe700 1 --2- 172.21.15.46:0/1204781715 >> v2:172.21.15.106:3300/0 conn(0x7f81500ca0e0 0x7f81500ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:21.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.841+0000 7f814effd700 1 --2- 172.21.15.46:0/1204781715 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f815006c0b0 0x7f81500afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:21.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.841+0000 7f814e7fc700 1 -- 172.21.15.46:0/1204781715 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618258258 0 0) 0x7f8150158010 con 0x7f815006c520 2024-01-26T23:46:21.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.841+0000 7f814e7fc700 1 -- 172.21.15.46:0/1204781715 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8138003570 con 0x7f815006c520 2024-01-26T23:46:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.841+0000 7f814f7fe700 1 -- 172.21.15.46:0/1204781715 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f815006c0b0 msgr2=0x7f81500afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.842+0000 7f814f7fe700 1 --2- 172.21.15.46:0/1204781715 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f815006c0b0 0x7f81500afe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.842+0000 7f814f7fe700 1 -- 172.21.15.46:0/1204781715 >> v1:172.21.15.46:6789/0 conn(0x7f815006c520 legacy=0x7f81500b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.842+0000 7f814f7fe700 1 -- 172.21.15.46:0/1204781715 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81501585d0 con 0x7f81500ca0e0 2024-01-26T23:46:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.842+0000 7f814effd700 1 --2- 172.21.15.46:0/1204781715 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f815006c0b0 0x7f81500afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.842+0000 7f814f7fe700 1 --2- 172.21.15.46:0/1204781715 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81500ca0e0 0x7f81500ca4b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f81400098e0 tx=0x7f81400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc92df9aeae4668c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.842+0000 7f814e7fc700 1 -- 172.21.15.46:0/1204781715 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8140011630 con 0x7f81500ca0e0 2024-01-26T23:46:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.842+0000 7f814e7fc700 1 -- 172.21.15.46:0/1204781715 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8140011790 con 0x7f81500ca0e0 2024-01-26T23:46:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.842+0000 7f814e7fc700 1 -- 172.21.15.46:0/1204781715 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8140011b30 con 0x7f81500ca0e0 2024-01-26T23:46:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.842+0000 7f8155aa3700 1 -- 172.21.15.46:0/1204781715 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81500ca0e0 msgr2=0x7f81500ca4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.842+0000 7f8155aa3700 1 --2- 172.21.15.46:0/1204781715 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81500ca0e0 0x7f81500ca4b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f81400098e0 tx=0x7f81400091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:21.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.843+0000 7f8155aa3700 1 -- 172.21.15.46:0/1204781715 shutdown_connections 2024-01-26T23:46:21.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.843+0000 7f8155aa3700 1 --2- 172.21.15.46:0/1204781715 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81500ca0e0 0x7f81500ca4b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:21.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.843+0000 7f8155aa3700 1 --2- 172.21.15.46:0/1204781715 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f815006c0b0 0x7f81500afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:21.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.843+0000 7f8155aa3700 1 -- 172.21.15.46:0/1204781715 >> 172.21.15.46:0/1204781715 conn(0x7f81500ce530 msgr2=0x7f81500ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:21.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.843+0000 7f8155aa3700 1 -- 172.21.15.46:0/1204781715 shutdown_connections 2024-01-26T23:46:21.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.843+0000 7f8155aa3700 1 -- 172.21.15.46:0/1204781715 wait complete. 2024-01-26T23:46:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.844+0000 7f8155aa3700 1 Processor -- start 2024-01-26T23:46:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.844+0000 7f8155aa3700 1 -- start start 2024-01-26T23:46:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f8155aa3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f815006c0b0 0x7f81500b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f8155aa3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f815006c520 0x7f8150151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f8155aa3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81500ca0e0 0x7f8150151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f8155aa3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8150168640 con 0x7f815006c0b0 2024-01-26T23:46:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f8155aa3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8150167d50 con 0x7f815006c520 2024-01-26T23:46:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f8155aa3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8150167ff0 con 0x7f81500ca0e0 2024-01-26T23:46:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f814f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f815006c0b0 0x7f81500b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f814f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f815006c0b0 0x7f81500b2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46442/0 (socket says 172.21.15.46:46442) 2024-01-26T23:46:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f814f7fe700 1 -- 172.21.15.46:0/3278142539 learned_addr learned my addr 172.21.15.46:0/3278142539 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f814ffff700 1 --2- 172.21.15.46:0/3278142539 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81500ca0e0 0x7f8150151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f814effd700 1 --2- 172.21.15.46:0/3278142539 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f815006c520 0x7f8150151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f814f7fe700 1 -- 172.21.15.46:0/3278142539 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81500ca0e0 msgr2=0x7f8150151740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f814f7fe700 1 --2- 172.21.15.46:0/3278142539 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81500ca0e0 0x7f8150151740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f814f7fe700 1 -- 172.21.15.46:0/3278142539 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f815006c520 msgr2=0x7f8150151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f814f7fe700 1 --2- 172.21.15.46:0/3278142539 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f815006c520 0x7f8150151200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f814f7fe700 1 -- 172.21.15.46:0/3278142539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8144009420 con 0x7f815006c0b0 2024-01-26T23:46:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f814effd700 1 --2- 172.21.15.46:0/3278142539 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f815006c520 0x7f8150151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:46:21.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.845+0000 7f814f7fe700 1 --2- 172.21.15.46:0/3278142539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f815006c0b0 0x7f81500b2cd0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f815006f390 tx=0x7f8140017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:21.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.846+0000 7f81377fe700 1 -- 172.21.15.46:0/3278142539 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8140011b10 con 0x7f815006c0b0 2024-01-26T23:46:21.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.846+0000 7f81377fe700 1 -- 172.21.15.46:0/3278142539 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8140000cc0 con 0x7f815006c0b0 2024-01-26T23:46:21.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.846+0000 7f8155aa3700 1 -- 172.21.15.46:0/3278142539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8140009420 con 0x7f815006c0b0 2024-01-26T23:46:21.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.846+0000 7f81377fe700 1 -- 172.21.15.46:0/3278142539 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8140023600 con 0x7f815006c0b0 2024-01-26T23:46:21.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.846+0000 7f8155aa3700 1 -- 172.21.15.46:0/3278142539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81500c7980 con 0x7f815006c0b0 2024-01-26T23:46:21.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.847+0000 7f8155aa3700 1 -- 172.21.15.46:0/3278142539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8150064c40 con 0x7f815006c0b0 2024-01-26T23:46:21.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.847+0000 7f81377fe700 1 -- 172.21.15.46:0/3278142539 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 43) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f814002c3b0 con 0x7f815006c0b0 2024-01-26T23:46:21.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.848+0000 7f81377fe700 1 --2- 172.21.15.46:0/3278142539 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f813c066d80 0x7f813c069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:21.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.848+0000 7f81377fe700 1 -- 172.21.15.46:0/3278142539 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6624+0+0 (secure 0 0 0) 0x7f8140085950 con 0x7f815006c0b0 2024-01-26T23:46:21.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.848+0000 7f814effd700 1 --2- 172.21.15.46:0/3278142539 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f813c066d80 0x7f813c069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:21.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.848+0000 7f814effd700 1 --2- 172.21.15.46:0/3278142539 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f813c066d80 0x7f813c069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f81440093f0 tx=0x7f8144008040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:21.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:21.852+0000 7f81377fe700 1 -- 172.21.15.46:0/3278142539 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8140052ae0 con 0x7f815006c0b0 2024-01-26T23:46:22.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.069+0000 7f8155aa3700 1 -- 172.21.15.46:0/3278142539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f815004c710 con 0x7f815006c0b0 2024-01-26T23:46:22.460 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:22.458+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:22.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.638+0000 7f81377fe700 1 -- 172.21.15.46:0/3278142539 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f8140052eb0 con 0x7f815006c0b0 2024-01-26T23:46:22.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.678+0000 7f81377fe700 1 -- 172.21.15.46:0/3278142539 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v104) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8140056570 con 0x7f815006c0b0 2024-01-26T23:46:22.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.684+0000 7f8155aa3700 1 -- 172.21.15.46:0/3278142539 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f813c066d80 msgr2=0x7f813c069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:22.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.684+0000 7f8155aa3700 1 --2- 172.21.15.46:0/3278142539 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f813c066d80 0x7f813c069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f81440093f0 tx=0x7f8144008040 comp rx=0 tx=0).stop 2024-01-26T23:46:22.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.684+0000 7f8155aa3700 1 -- 172.21.15.46:0/3278142539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f815006c0b0 msgr2=0x7f81500b2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:22.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.684+0000 7f8155aa3700 1 --2- 172.21.15.46:0/3278142539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f815006c0b0 0x7f81500b2cd0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f815006f390 tx=0x7f8140017000 comp rx=0 tx=0).stop 2024-01-26T23:46:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.684+0000 7f8155aa3700 1 -- 172.21.15.46:0/3278142539 shutdown_connections 2024-01-26T23:46:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.684+0000 7f8155aa3700 1 --2- 172.21.15.46:0/3278142539 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f813c066d80 0x7f813c069230 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.684+0000 7f8155aa3700 1 --2- 172.21.15.46:0/3278142539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f815006c0b0 0x7f81500b2cd0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.684+0000 7f8155aa3700 1 --2- 172.21.15.46:0/3278142539 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81500ca0e0 0x7f8150151740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.684+0000 7f8155aa3700 1 --2- 172.21.15.46:0/3278142539 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f815006c520 0x7f8150151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.684+0000 7f8155aa3700 1 -- 172.21.15.46:0/3278142539 >> 172.21.15.46:0/3278142539 conn(0x7f81500ce530 msgr2=0x7f81500cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.685+0000 7f8155aa3700 1 -- 172.21.15.46:0/3278142539 shutdown_connections 2024-01-26T23:46:22.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.687+0000 7f8155aa3700 1 -- 172.21.15.46:0/3278142539 wait complete. 2024-01-26T23:46:22.690 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-01-26T23:46:22.705 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-26T23:46:22.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.843+0000 7f699bc29700 1 Processor -- start 2024-01-26T23:46:22.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.844+0000 7f699bc29700 1 -- start start 2024-01-26T23:46:22.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.845+0000 7f699bc29700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69940ab6e0 0x7f699406a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:22.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.845+0000 7f699bc29700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6994068940 0x7f6994068d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:22.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.845+0000 7f699bc29700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69940692c0 con 0x7f69940ab310 2024-01-26T23:46:22.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.845+0000 7f699bc29700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69940c7540 con 0x7f69940ab6e0 2024-01-26T23:46:22.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.845+0000 7f699bc29700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69940c7820 con 0x7f6994068940 2024-01-26T23:46:22.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.845+0000 7f69991c4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69940ab6e0 0x7f699406a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:22.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.845+0000 7f69999c5700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f69940ab310 0x7f69940a9850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37640/0 (socket says 172.21.15.46:37640) 2024-01-26T23:46:22.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.845+0000 7f69999c5700 1 -- 172.21.15.46:0/3734788824 learned_addr learned my addr 172.21.15.46:0/3734788824 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:22.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.845+0000 7f699a1c6700 1 --2- 172.21.15.46:0/3734788824 >> v2:172.21.15.106:3300/0 conn(0x7f6994068940 0x7f6994068d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:22.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.845+0000 7f69991c4700 1 -- 172.21.15.46:0/3734788824 >> v2:172.21.15.106:3300/0 conn(0x7f6994068940 msgr2=0x7f6994068d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:22.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.845+0000 7f69991c4700 1 --2- 172.21.15.46:0/3734788824 >> v2:172.21.15.106:3300/0 conn(0x7f6994068940 0x7f6994068d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:22.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.845+0000 7f69991c4700 1 -- 172.21.15.46:0/3734788824 >> v1:172.21.15.46:6789/0 conn(0x7f69940ab310 legacy=0x7f69940a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:22.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.845+0000 7f69991c4700 1 -- 172.21.15.46:0/3734788824 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69940c7b00 con 0x7f69940ab6e0 2024-01-26T23:46:22.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.845+0000 7f699a1c6700 1 --2- 172.21.15.46:0/3734788824 >> v2:172.21.15.106:3300/0 conn(0x7f6994068940 0x7f6994068d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:22.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.845+0000 7f69989c3700 1 -- 172.21.15.46:0/3734788824 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086137083 0 0) 0x7f69940692c0 con 0x7f69940ab310 2024-01-26T23:46:22.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.845+0000 7f69991c4700 1 --2- 172.21.15.46:0/3734788824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69940ab6e0 0x7f699406a3c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f699415e530 tx=0x7f69840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=910acfbb756f5f05 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:22.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.846+0000 7f69989c3700 1 -- 172.21.15.46:0/3734788824 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6984011630 con 0x7f69940ab6e0 2024-01-26T23:46:22.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.846+0000 7f69989c3700 1 -- 172.21.15.46:0/3734788824 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6984011790 con 0x7f69940ab6e0 2024-01-26T23:46:22.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.846+0000 7f69989c3700 1 -- 172.21.15.46:0/3734788824 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6984011b30 con 0x7f69940ab6e0 2024-01-26T23:46:22.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.846+0000 7f699bc29700 1 -- 172.21.15.46:0/3734788824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69940ab6e0 msgr2=0x7f699406a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:22.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.846+0000 7f699bc29700 1 --2- 172.21.15.46:0/3734788824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69940ab6e0 0x7f699406a3c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f699415e530 tx=0x7f69840091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:22.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.846+0000 7f699bc29700 1 -- 172.21.15.46:0/3734788824 shutdown_connections 2024-01-26T23:46:22.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.846+0000 7f699bc29700 1 --2- 172.21.15.46:0/3734788824 >> v2:172.21.15.106:3300/0 conn(0x7f6994068940 0x7f6994068d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:22.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.846+0000 7f699bc29700 1 --2- 172.21.15.46:0/3734788824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69940ab6e0 0x7f699406a3c0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:22.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.846+0000 7f699bc29700 1 -- 172.21.15.46:0/3734788824 >> 172.21.15.46:0/3734788824 conn(0x7f69940ca100 msgr2=0x7f699406e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:22.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.847+0000 7f699bc29700 1 -- 172.21.15.46:0/3734788824 shutdown_connections 2024-01-26T23:46:22.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.847+0000 7f699bc29700 1 -- 172.21.15.46:0/3734788824 wait complete. 2024-01-26T23:46:22.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.848+0000 7f699bc29700 1 Processor -- start 2024-01-26T23:46:22.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.848+0000 7f699bc29700 1 -- start start 2024-01-26T23:46:22.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.849+0000 7f699bc29700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6994068940 0x7f699414df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:22.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.849+0000 7f699bc29700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69940ab310 0x7f69940abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:22.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.849+0000 7f699bc29700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69940ab6e0 0x7f69940ad6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:22.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.849+0000 7f699bc29700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699415c950 con 0x7f6994068940 2024-01-26T23:46:22.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.849+0000 7f699bc29700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69940accd0 con 0x7f69940ab6e0 2024-01-26T23:46:22.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.849+0000 7f699bc29700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699415ced0 con 0x7f69940ab310 2024-01-26T23:46:22.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.849+0000 7f69999c5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6994068940 0x7f699414df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:22.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.849+0000 7f69999c5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6994068940 0x7f699414df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46464/0 (socket says 172.21.15.46:46464) 2024-01-26T23:46:22.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.849+0000 7f69999c5700 1 -- 172.21.15.46:0/3530649235 learned_addr learned my addr 172.21.15.46:0/3530649235 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:22.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.850+0000 7f699a1c6700 1 --2- 172.21.15.46:0/3530649235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69940ab6e0 0x7f69940ad6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:22.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.850+0000 7f69991c4700 1 --2- 172.21.15.46:0/3530649235 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69940ab310 0x7f69940abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:22.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.850+0000 7f69999c5700 1 -- 172.21.15.46:0/3530649235 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69940ab310 msgr2=0x7f69940abdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:22.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.850+0000 7f69999c5700 1 --2- 172.21.15.46:0/3530649235 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69940ab310 0x7f69940abdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:22.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.850+0000 7f69999c5700 1 -- 172.21.15.46:0/3530649235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69940ab6e0 msgr2=0x7f69940ad6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:22.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.850+0000 7f69999c5700 1 --2- 172.21.15.46:0/3530649235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69940ab6e0 0x7f69940ad6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:22.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.850+0000 7f69999c5700 1 -- 172.21.15.46:0/3530649235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698c009a20 con 0x7f6994068940 2024-01-26T23:46:22.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.850+0000 7f69999c5700 1 --2- 172.21.15.46:0/3530649235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6994068940 0x7f699414df40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f699000c7e0 tx=0x7f6990007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:22.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.850+0000 7f6989ffb700 1 -- 172.21.15.46:0/3530649235 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69900072e0 con 0x7f6994068940 2024-01-26T23:46:22.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.850+0000 7f6989ffb700 1 -- 172.21.15.46:0/3530649235 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6990007440 con 0x7f6994068940 2024-01-26T23:46:22.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.850+0000 7f699bc29700 1 -- 172.21.15.46:0/3530649235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6984009420 con 0x7f6994068940 2024-01-26T23:46:22.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.850+0000 7f6989ffb700 1 -- 172.21.15.46:0/3530649235 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69900057f0 con 0x7f6994068940 2024-01-26T23:46:22.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.850+0000 7f699bc29700 1 -- 172.21.15.46:0/3530649235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f699415cb60 con 0x7f6994068940 2024-01-26T23:46:22.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.852+0000 7f699bc29700 1 -- 172.21.15.46:0/3530649235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6994068f90 con 0x7f6994068940 2024-01-26T23:46:22.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.852+0000 7f6989ffb700 1 -- 172.21.15.46:0/3530649235 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f6990005be0 con 0x7f6994068940 2024-01-26T23:46:22.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.852+0000 7f6989ffb700 1 --2- 172.21.15.46:0/3530649235 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6978066ea0 0x7f6978069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:22.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.852+0000 7f6989ffb700 1 -- 172.21.15.46:0/3530649235 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f6990083ae0 con 0x7f6994068940 2024-01-26T23:46:22.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.852+0000 7f69991c4700 1 --2- 172.21.15.46:0/3530649235 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6978066ea0 0x7f6978069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:22.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.853+0000 7f69991c4700 1 --2- 172.21.15.46:0/3530649235 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6978066ea0 0x7f6978069350 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f69940abad0 tx=0x7f6984017000 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:22.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:22.857+0000 7f6989ffb700 1 -- 172.21.15.46:0/3530649235 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6990050cf0 con 0x7f6994068940 2024-01-26T23:46:23.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.065+0000 7f699bc29700 1 -- 172.21.15.46:0/3530649235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f699404c710 con 0x7f6994068940 2024-01-26T23:46:23.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.069+0000 7f6989ffb700 1 -- 172.21.15.46:0/3530649235 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f69900170a0 con 0x7f6994068940 2024-01-26T23:46:23.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.072+0000 7f699bc29700 1 -- 172.21.15.46:0/3530649235 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6978066ea0 msgr2=0x7f6978069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.072+0000 7f699bc29700 1 --2- 172.21.15.46:0/3530649235 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6978066ea0 0x7f6978069350 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f69940abad0 tx=0x7f6984017000 comp rx=0 tx=0).stop 2024-01-26T23:46:23.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.072+0000 7f699bc29700 1 -- 172.21.15.46:0/3530649235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6994068940 msgr2=0x7f699414df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.072+0000 7f699bc29700 1 --2- 172.21.15.46:0/3530649235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6994068940 0x7f699414df40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f699000c7e0 tx=0x7f6990007c80 comp rx=0 tx=0).stop 2024-01-26T23:46:23.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.072+0000 7f699bc29700 1 -- 172.21.15.46:0/3530649235 shutdown_connections 2024-01-26T23:46:23.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.072+0000 7f699bc29700 1 --2- 172.21.15.46:0/3530649235 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f6978066ea0 0x7f6978069350 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.072+0000 7f699bc29700 1 --2- 172.21.15.46:0/3530649235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6994068940 0x7f699414df40 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.072+0000 7f699bc29700 1 --2- 172.21.15.46:0/3530649235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69940ab6e0 0x7f69940ad6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.072+0000 7f699bc29700 1 --2- 172.21.15.46:0/3530649235 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69940ab310 0x7f69940abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.072+0000 7f699bc29700 1 -- 172.21.15.46:0/3530649235 >> 172.21.15.46:0/3530649235 conn(0x7f69940ca100 msgr2=0x7f69940a9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:23.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.072+0000 7f699bc29700 1 -- 172.21.15.46:0/3530649235 shutdown_connections 2024-01-26T23:46:23.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.072+0000 7f699bc29700 1 -- 172.21.15.46:0/3530649235 wait complete. 2024-01-26T23:46:23.087 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-26T23:46:23.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.221+0000 7efeb89a2700 1 Processor -- start 2024-01-26T23:46:23.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.221+0000 7efeb89a2700 1 -- start start 2024-01-26T23:46:23.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.222+0000 7efeb89a2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeb4159970 0x7efeb40b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.222+0000 7efeb89a2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb40c6140 con 0x7efeb40ccfb0 2024-01-26T23:46:23.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.222+0000 7efeb89a2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb40b38f0 con 0x7efeb40c6ce0 2024-01-26T23:46:23.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.222+0000 7efeb89a2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb40c5530 con 0x7efeb4159970 2024-01-26T23:46:23.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.222+0000 7efeb2ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efeb40ccfb0 0x7efeb40c65d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37652/0 (socket says 172.21.15.46:37652) 2024-01-26T23:46:23.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.222+0000 7efeb2ffd700 1 -- 172.21.15.46:0/1059184416 learned_addr learned my addr 172.21.15.46:0/1059184416 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:23.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.222+0000 7efeb37fe700 1 --2- 172.21.15.46:0/1059184416 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeb4159970 0x7efeb40b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.223+0000 7efeb27fc700 1 -- 172.21.15.46:0/1059184416 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383018138 0 0) 0x7efeb40c6140 con 0x7efeb40ccfb0 2024-01-26T23:46:23.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.223+0000 7efeb27fc700 1 -- 172.21.15.46:0/1059184416 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe9c003570 con 0x7efeb40ccfb0 2024-01-26T23:46:23.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.223+0000 7efeb27fc700 1 -- 172.21.15.46:0/1059184416 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516119278 0 0) 0x7efeb40b38f0 con 0x7efeb40c6ce0 2024-01-26T23:46:23.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.223+0000 7efeb27fc700 1 -- 172.21.15.46:0/1059184416 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb40c6140 con 0x7efeb40c6ce0 2024-01-26T23:46:23.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.223+0000 7efeb37fe700 1 -- 172.21.15.46:0/1059184416 >> v1:172.21.15.106:6789/0 conn(0x7efeb40c6ce0 legacy=0x7efeb40cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.223+0000 7efeb37fe700 1 -- 172.21.15.46:0/1059184416 >> v1:172.21.15.46:6789/0 conn(0x7efeb40ccfb0 legacy=0x7efeb40c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.223+0000 7efeb37fe700 1 -- 172.21.15.46:0/1059184416 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb40c5810 con 0x7efeb4159970 2024-01-26T23:46:23.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.223+0000 7efeb37fe700 1 --2- 172.21.15.46:0/1059184416 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeb4159970 0x7efeb40b33b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efea4009900 tx=0x7efea4009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bdfc9de52fdf688 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:23.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.223+0000 7efeb27fc700 1 -- 172.21.15.46:0/1059184416 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea4005c10 con 0x7efeb4159970 2024-01-26T23:46:23.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.223+0000 7efeb27fc700 1 -- 172.21.15.46:0/1059184416 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea4005d70 con 0x7efeb4159970 2024-01-26T23:46:23.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.223+0000 7efeb27fc700 1 -- 172.21.15.46:0/1059184416 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea4006260 con 0x7efeb4159970 2024-01-26T23:46:23.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.224+0000 7efeb89a2700 1 -- 172.21.15.46:0/1059184416 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeb4159970 msgr2=0x7efeb40b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.224+0000 7efeb89a2700 1 --2- 172.21.15.46:0/1059184416 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeb4159970 0x7efeb40b33b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efea4009900 tx=0x7efea4009120 comp rx=0 tx=0).stop 2024-01-26T23:46:23.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.224+0000 7efeb89a2700 1 -- 172.21.15.46:0/1059184416 shutdown_connections 2024-01-26T23:46:23.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.224+0000 7efeb89a2700 1 --2- 172.21.15.46:0/1059184416 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeb4159970 0x7efeb40b33b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.224+0000 7efeb89a2700 1 -- 172.21.15.46:0/1059184416 >> 172.21.15.46:0/1059184416 conn(0x7efeb4099950 msgr2=0x7efeb406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:23.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.224+0000 7efeb89a2700 1 -- 172.21.15.46:0/1059184416 shutdown_connections 2024-01-26T23:46:23.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.224+0000 7efeb89a2700 1 -- 172.21.15.46:0/1059184416 wait complete. 2024-01-26T23:46:23.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.225+0000 7efeb89a2700 1 Processor -- start 2024-01-26T23:46:23.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.225+0000 7efeb89a2700 1 -- start start 2024-01-26T23:46:23.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.226+0000 7efeb89a2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeb40c6ce0 0x7efeb415b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.226+0000 7efeb89a2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeb40ccfb0 0x7efeb40c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.226+0000 7efeb89a2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efeb4159970 0x7efeb40c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.226+0000 7efeb89a2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb40c77c0 con 0x7efeb40c6ce0 2024-01-26T23:46:23.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.226+0000 7efeb89a2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb40c7a30 con 0x7efeb4159970 2024-01-26T23:46:23.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.226+0000 7efeb89a2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb40c7d10 con 0x7efeb40ccfb0 2024-01-26T23:46:23.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.226+0000 7efeb37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeb40c6ce0 0x7efeb415b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.226+0000 7efeb37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeb40c6ce0 0x7efeb415b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46466/0 (socket says 172.21.15.46:46466) 2024-01-26T23:46:23.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.226+0000 7efeb37fe700 1 -- 172.21.15.46:0/3540818598 learned_addr learned my addr 172.21.15.46:0/3540818598 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:23.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.226+0000 7efeb2ffd700 1 --2- 172.21.15.46:0/3540818598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeb40ccfb0 0x7efeb40c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.226+0000 7efeb3fff700 1 --2- 172.21.15.46:0/3540818598 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efeb4159970 0x7efeb40c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.227+0000 7efeb37fe700 1 -- 172.21.15.46:0/3540818598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeb40ccfb0 msgr2=0x7efeb40c87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.227+0000 7efeb37fe700 1 --2- 172.21.15.46:0/3540818598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeb40ccfb0 0x7efeb40c87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.227+0000 7efeb37fe700 1 -- 172.21.15.46:0/3540818598 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efeb4159970 msgr2=0x7efeb40c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.227+0000 7efeb37fe700 1 --2- 172.21.15.46:0/3540818598 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efeb4159970 0x7efeb40c8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.227+0000 7efeb37fe700 1 -- 172.21.15.46:0/3540818598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea40093c0 con 0x7efeb40c6ce0 2024-01-26T23:46:23.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.227+0000 7efeb2ffd700 1 --2- 172.21.15.46:0/3540818598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeb40ccfb0 0x7efeb40c87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:46:23.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.227+0000 7efeb3fff700 1 --2- 172.21.15.46:0/3540818598 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efeb4159970 0x7efeb40c8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:46:23.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.227+0000 7efeb37fe700 1 --2- 172.21.15.46:0/3540818598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeb40c6ce0 0x7efeb415b0b0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7efea4003080 tx=0x7efea4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:23.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.227+0000 7efe9b7fe700 1 -- 172.21.15.46:0/3540818598 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea401d970 con 0x7efeb40c6ce0 2024-01-26T23:46:23.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.227+0000 7efe9b7fe700 1 -- 172.21.15.46:0/3540818598 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea401dad0 con 0x7efeb40c6ce0 2024-01-26T23:46:23.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.227+0000 7efeb89a2700 1 -- 172.21.15.46:0/3540818598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb40c7ff0 con 0x7efeb40c6ce0 2024-01-26T23:46:23.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.227+0000 7efe9b7fe700 1 -- 172.21.15.46:0/3540818598 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea401de90 con 0x7efeb40c6ce0 2024-01-26T23:46:23.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.227+0000 7efeb89a2700 1 -- 172.21.15.46:0/3540818598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb4156710 con 0x7efeb40c6ce0 2024-01-26T23:46:23.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.228+0000 7efeb89a2700 1 -- 172.21.15.46:0/3540818598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb4068700 con 0x7efeb40c6ce0 2024-01-26T23:46:23.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.229+0000 7efe9b7fe700 1 -- 172.21.15.46:0/3540818598 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7efea401b690 con 0x7efeb40c6ce0 2024-01-26T23:46:23.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.229+0000 7efe9b7fe700 1 --2- 172.21.15.46:0/3540818598 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efea0066ea0 0x7efea0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.230+0000 7efe9b7fe700 1 -- 172.21.15.46:0/3540818598 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6210+0+0 (secure 0 0 0) 0x7efea4086580 con 0x7efeb40c6ce0 2024-01-26T23:46:23.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.230+0000 7efeb2ffd700 1 --2- 172.21.15.46:0/3540818598 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efea0066ea0 0x7efea0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.230+0000 7efeb2ffd700 1 --2- 172.21.15.46:0/3540818598 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efea0066ea0 0x7efea0069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efea80058d0 tx=0x7efea8009040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:23.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.234+0000 7efe9b7fe700 1 -- 172.21.15.46:0/3540818598 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efea4053750 con 0x7efeb40c6ce0 2024-01-26T23:46:23.257 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:23.256+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:23.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.449+0000 7efeb89a2700 1 -- 172.21.15.46:0/3540818598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7efeb404bed0 con 0x7efeb40c6ce0 2024-01-26T23:46:23.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.452+0000 7efe9b7fe700 1 -- 172.21.15.46:0/3540818598 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v32) v1 ==== 120+0+0 (secure 0 0 0) 0x7efea4009220 con 0x7efeb40c6ce0 2024-01-26T23:46:23.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.455+0000 7efeb89a2700 1 -- 172.21.15.46:0/3540818598 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efea0066ea0 msgr2=0x7efea0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.455+0000 7efeb89a2700 1 --2- 172.21.15.46:0/3540818598 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efea0066ea0 0x7efea0069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efea80058d0 tx=0x7efea8009040 comp rx=0 tx=0).stop 2024-01-26T23:46:23.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.455+0000 7efeb89a2700 1 -- 172.21.15.46:0/3540818598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeb40c6ce0 msgr2=0x7efeb415b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.455+0000 7efeb89a2700 1 --2- 172.21.15.46:0/3540818598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeb40c6ce0 0x7efeb415b0b0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7efea4003080 tx=0x7efea4016000 comp rx=0 tx=0).stop 2024-01-26T23:46:23.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.455+0000 7efeb89a2700 1 -- 172.21.15.46:0/3540818598 shutdown_connections 2024-01-26T23:46:23.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.455+0000 7efeb89a2700 1 --2- 172.21.15.46:0/3540818598 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7efea0066ea0 0x7efea0069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.455+0000 7efeb89a2700 1 --2- 172.21.15.46:0/3540818598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeb40c6ce0 0x7efeb415b0b0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.455+0000 7efeb89a2700 1 --2- 172.21.15.46:0/3540818598 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efeb4159970 0x7efeb40c8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.455+0000 7efeb89a2700 1 --2- 172.21.15.46:0/3540818598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeb40ccfb0 0x7efeb40c87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.455+0000 7efeb89a2700 1 -- 172.21.15.46:0/3540818598 >> 172.21.15.46:0/3540818598 conn(0x7efeb4099950 msgr2=0x7efeb40698b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:23.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.455+0000 7efeb89a2700 1 -- 172.21.15.46:0/3540818598 shutdown_connections 2024-01-26T23:46:23.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.455+0000 7efeb89a2700 1 -- 172.21.15.46:0/3540818598 wait complete. 2024-01-26T23:46:23.470 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-26T23:46:23.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.608+0000 7f9d3cb27700 1 Processor -- start 2024-01-26T23:46:23.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.608+0000 7f9d3cb27700 1 -- start start 2024-01-26T23:46:23.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.609+0000 7f9d3cb27700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d38068b00 0x7f9d3815a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.609+0000 7f9d3cb27700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9d38068ed0 0x7f9d3806a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.609+0000 7f9d3cb27700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d3806a900 0x7f9d38155810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.609+0000 7f9d3cb27700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d38069320 con 0x7f9d3806a900 2024-01-26T23:46:23.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.609+0000 7f9d3cb27700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d3806b210 con 0x7f9d38068ed0 2024-01-26T23:46:23.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.609+0000 7f9d3cb27700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3815a0c0 con 0x7f9d38068b00 2024-01-26T23:46:23.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.609+0000 7f9d36d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d3806a900 0x7f9d38155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.609+0000 7f9d36d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d3806a900 0x7f9d38155810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46474/0 (socket says 172.21.15.46:46474) 2024-01-26T23:46:23.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.609+0000 7f9d36d9d700 1 -- 172.21.15.46:0/2457573492 learned_addr learned my addr 172.21.15.46:0/2457573492 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:23.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.609+0000 7f9d3659c700 1 --2- 172.21.15.46:0/2457573492 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d38068b00 0x7f9d3815a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.609+0000 7f9d35d9b700 1 --2- 172.21.15.46:0/2457573492 >> v2:172.21.15.106:3300/0 conn(0x7f9d38068ed0 0x7f9d3806a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.610+0000 7f9d36d9d700 1 -- 172.21.15.46:0/2457573492 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d38068b00 msgr2=0x7f9d3815a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:23.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.610+0000 7f9d36d9d700 1 --2- 172.21.15.46:0/2457573492 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d38068b00 0x7f9d3815a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.610+0000 7f9d36d9d700 1 -- 172.21.15.46:0/2457573492 >> v2:172.21.15.106:3300/0 conn(0x7f9d38068ed0 msgr2=0x7f9d3806a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:23.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.610+0000 7f9d36d9d700 1 --2- 172.21.15.46:0/2457573492 >> v2:172.21.15.106:3300/0 conn(0x7f9d38068ed0 0x7f9d3806a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.610+0000 7f9d36d9d700 1 -- 172.21.15.46:0/2457573492 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3806af30 con 0x7f9d3806a900 2024-01-26T23:46:23.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.610+0000 7f9d36d9d700 1 --2- 172.21.15.46:0/2457573492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d3806a900 0x7f9d38155810 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f9d3806dc90 tx=0x7f9d2800ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31d4bbea43e1baa server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:23.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.610+0000 7f9d3559a700 1 -- 172.21.15.46:0/2457573492 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d28010070 con 0x7f9d3806a900 2024-01-26T23:46:23.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.610+0000 7f9d3559a700 1 -- 172.21.15.46:0/2457573492 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2800dd30 con 0x7f9d3806a900 2024-01-26T23:46:23.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.610+0000 7f9d3559a700 1 -- 172.21.15.46:0/2457573492 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2800dea0 con 0x7f9d3806a900 2024-01-26T23:46:23.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.611+0000 7f9d3cb27700 1 -- 172.21.15.46:0/2457573492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d3806a900 msgr2=0x7f9d38155810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.611+0000 7f9d3cb27700 1 --2- 172.21.15.46:0/2457573492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d3806a900 0x7f9d38155810 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f9d3806dc90 tx=0x7f9d2800ba50 comp rx=0 tx=0).stop 2024-01-26T23:46:23.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.611+0000 7f9d3cb27700 1 -- 172.21.15.46:0/2457573492 shutdown_connections 2024-01-26T23:46:23.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.611+0000 7f9d3cb27700 1 --2- 172.21.15.46:0/2457573492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d3806a900 0x7f9d38155810 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.611+0000 7f9d3cb27700 1 --2- 172.21.15.46:0/2457573492 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d38068b00 0x7f9d3815a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.611+0000 7f9d3cb27700 1 --2- 172.21.15.46:0/2457573492 >> v2:172.21.15.106:3300/0 conn(0x7f9d38068ed0 0x7f9d3806a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.611+0000 7f9d3cb27700 1 -- 172.21.15.46:0/2457573492 >> 172.21.15.46:0/2457573492 conn(0x7f9d380ca100 msgr2=0x7f9d3806e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:23.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.611+0000 7f9d3cb27700 1 -- 172.21.15.46:0/2457573492 shutdown_connections 2024-01-26T23:46:23.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.611+0000 7f9d3cb27700 1 -- 172.21.15.46:0/2457573492 wait complete. 2024-01-26T23:46:23.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.612+0000 7f9d3cb27700 1 Processor -- start 2024-01-26T23:46:23.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.612+0000 7f9d3cb27700 1 -- start start 2024-01-26T23:46:23.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.613+0000 7f9d3cb27700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d38068b00 0x7f9d38151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.613+0000 7f9d3cb27700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d38068ed0 0x7f9d3816a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.613+0000 7f9d3cb27700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d3806a900 0x7f9d380b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.613+0000 7f9d3cb27700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380b17a0 con 0x7f9d38068b00 2024-01-26T23:46:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.613+0000 7f9d3cb27700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d38151650 con 0x7f9d3806a900 2024-01-26T23:46:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.613+0000 7f9d3cb27700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380b0b90 con 0x7f9d38068ed0 2024-01-26T23:46:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.613+0000 7f9d3659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d38068b00 0x7f9d38151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.613+0000 7f9d3659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d38068b00 0x7f9d38151110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46476/0 (socket says 172.21.15.46:46476) 2024-01-26T23:46:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.613+0000 7f9d3659c700 1 -- 172.21.15.46:0/3665756701 learned_addr learned my addr 172.21.15.46:0/3665756701 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.613+0000 7f9d36d9d700 1 --2- 172.21.15.46:0/3665756701 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d3806a900 0x7f9d380b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.613+0000 7f9d35d9b700 1 --2- 172.21.15.46:0/3665756701 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d38068ed0 0x7f9d3816a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.613+0000 7f9d3659c700 1 -- 172.21.15.46:0/3665756701 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d38068ed0 msgr2=0x7f9d3816a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.613+0000 7f9d3659c700 1 --2- 172.21.15.46:0/3665756701 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d38068ed0 0x7f9d3816a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.613+0000 7f9d3659c700 1 -- 172.21.15.46:0/3665756701 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d3806a900 msgr2=0x7f9d380b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.613+0000 7f9d3659c700 1 --2- 172.21.15.46:0/3665756701 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d3806a900 0x7f9d380b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.613+0000 7f9d3659c700 1 -- 172.21.15.46:0/3665756701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2800bcf0 con 0x7f9d38068b00 2024-01-26T23:46:23.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.613+0000 7f9d3659c700 1 --2- 172.21.15.46:0/3665756701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d38068b00 0x7f9d38151110 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000ec80 tx=0x7f9d30009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:23.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.614+0000 7f9d2e7fc700 1 -- 172.21.15.46:0/3665756701 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3000c850 con 0x7f9d38068b00 2024-01-26T23:46:23.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.614+0000 7f9d2e7fc700 1 -- 172.21.15.46:0/3665756701 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3000c9b0 con 0x7f9d38068b00 2024-01-26T23:46:23.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.614+0000 7f9d3cb27700 1 -- 172.21.15.46:0/3665756701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d380b0e70 con 0x7f9d38068b00 2024-01-26T23:46:23.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.614+0000 7f9d2e7fc700 1 -- 172.21.15.46:0/3665756701 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3000cd30 con 0x7f9d38068b00 2024-01-26T23:46:23.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.614+0000 7f9d3cb27700 1 -- 172.21.15.46:0/3665756701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d380b1430 con 0x7f9d38068b00 2024-01-26T23:46:23.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.615+0000 7f9d3cb27700 1 -- 172.21.15.46:0/3665756701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d380b1a80 con 0x7f9d38068b00 2024-01-26T23:46:23.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.615+0000 7f9d2e7fc700 1 -- 172.21.15.46:0/3665756701 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f9d30013680 con 0x7f9d38068b00 2024-01-26T23:46:23.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.616+0000 7f9d2e7fc700 1 --2- 172.21.15.46:0/3665756701 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d18066ea0 0x7f9d18069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.616+0000 7f9d2e7fc700 1 -- 172.21.15.46:0/3665756701 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f9d30084b90 con 0x7f9d38068b00 2024-01-26T23:46:23.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.616+0000 7f9d35d9b700 1 --2- 172.21.15.46:0/3665756701 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d18066ea0 0x7f9d18069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.617+0000 7f9d35d9b700 1 --2- 172.21.15.46:0/3665756701 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d18066ea0 0x7f9d18069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9d380b2840 tx=0x7f9d200091a0 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:23.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.620+0000 7f9d2e7fc700 1 -- 172.21.15.46:0/3665756701 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9d30051ce0 con 0x7f9d38068b00 2024-01-26T23:46:23.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.828+0000 7f9d3cb27700 1 -- 172.21.15.46:0/3665756701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f9d3804c710 con 0x7f9d38068b00 2024-01-26T23:46:23.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.832+0000 7f9d2e7fc700 1 -- 172.21.15.46:0/3665756701 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v33) v1 ==== 115+0+0 (secure 0 0 0) 0x7f9d300210b0 con 0x7f9d38068b00 2024-01-26T23:46:23.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.835+0000 7f9d3cb27700 1 -- 172.21.15.46:0/3665756701 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d18066ea0 msgr2=0x7f9d18069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.835+0000 7f9d3cb27700 1 --2- 172.21.15.46:0/3665756701 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d18066ea0 0x7f9d18069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9d380b2840 tx=0x7f9d200091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.835+0000 7f9d3cb27700 1 -- 172.21.15.46:0/3665756701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d38068b00 msgr2=0x7f9d38151110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.835+0000 7f9d3cb27700 1 --2- 172.21.15.46:0/3665756701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d38068b00 0x7f9d38151110 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000ec80 tx=0x7f9d30009120 comp rx=0 tx=0).stop 2024-01-26T23:46:23.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.835+0000 7f9d3cb27700 1 -- 172.21.15.46:0/3665756701 shutdown_connections 2024-01-26T23:46:23.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.835+0000 7f9d3cb27700 1 --2- 172.21.15.46:0/3665756701 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9d18066ea0 0x7f9d18069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.835+0000 7f9d3cb27700 1 --2- 172.21.15.46:0/3665756701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d38068b00 0x7f9d38151110 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.835+0000 7f9d3cb27700 1 --2- 172.21.15.46:0/3665756701 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d3806a900 0x7f9d380b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.835+0000 7f9d3cb27700 1 --2- 172.21.15.46:0/3665756701 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d38068ed0 0x7f9d3816a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.835+0000 7f9d3cb27700 1 -- 172.21.15.46:0/3665756701 >> 172.21.15.46:0/3665756701 conn(0x7f9d380ca100 msgr2=0x7f9d380adc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:23.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.836+0000 7f9d3cb27700 1 -- 172.21.15.46:0/3665756701 shutdown_connections 2024-01-26T23:46:23.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.836+0000 7f9d3cb27700 1 -- 172.21.15.46:0/3665756701 wait complete. 2024-01-26T23:46:23.851 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-01-26T23:46:23.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.984+0000 7f0267822700 1 Processor -- start 2024-01-26T23:46:23.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.984+0000 7f0267822700 1 -- start start 2024-01-26T23:46:23.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.985+0000 7f0267822700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02600cb4c0 0x7f0260159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.985+0000 7f0267822700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02600c4720 0x7f0260068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.985+0000 7f0267822700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02600c97f0 con 0x7f02601596e0 2024-01-26T23:46:23.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.985+0000 7f0267822700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02600cbf90 con 0x7f02600cb4c0 2024-01-26T23:46:23.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.985+0000 7f0267822700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02600c8c20 con 0x7f02600c4720 2024-01-26T23:46:23.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.985+0000 7f0264dbd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02600cb4c0 0x7f0260159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.985+0000 7f02655be700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f02601596e0 0x7f02600cadb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37656/0 (socket says 172.21.15.46:37656) 2024-01-26T23:46:23.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.985+0000 7f02655be700 1 -- 172.21.15.46:0/2858447454 learned_addr learned my addr 172.21.15.46:0/2858447454 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:23.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.985+0000 7f0265dbf700 1 --2- 172.21.15.46:0/2858447454 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02600c4720 0x7f0260068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.986+0000 7f0257fff700 1 -- 172.21.15.46:0/2858447454 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457070835 0 0) 0x7f02600c97f0 con 0x7f02601596e0 2024-01-26T23:46:23.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.986+0000 7f0257fff700 1 -- 172.21.15.46:0/2858447454 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f024c003570 con 0x7f02601596e0 2024-01-26T23:46:23.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.986+0000 7f0264dbd700 1 -- 172.21.15.46:0/2858447454 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02600c4720 msgr2=0x7f0260068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:23.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.986+0000 7f0264dbd700 1 --2- 172.21.15.46:0/2858447454 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02600c4720 0x7f0260068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.986+0000 7f0264dbd700 1 -- 172.21.15.46:0/2858447454 >> v1:172.21.15.46:6789/0 conn(0x7f02601596e0 legacy=0x7f02600cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.986+0000 7f0264dbd700 1 -- 172.21.15.46:0/2858447454 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02600c8ec0 con 0x7f02600cb4c0 2024-01-26T23:46:23.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.986+0000 7f0265dbf700 1 --2- 172.21.15.46:0/2858447454 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02600c4720 0x7f0260068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:23.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.986+0000 7f0264dbd700 1 --2- 172.21.15.46:0/2858447454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02600cb4c0 0x7f0260159ab0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f02500098e0 tx=0x7f02500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af7af226a39a190 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:23.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.986+0000 7f0257fff700 1 -- 172.21.15.46:0/2858447454 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0250011630 con 0x7f02600cb4c0 2024-01-26T23:46:23.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.986+0000 7f0257fff700 1 -- 172.21.15.46:0/2858447454 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0250011790 con 0x7f02600cb4c0 2024-01-26T23:46:23.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.986+0000 7f0257fff700 1 -- 172.21.15.46:0/2858447454 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0250011b30 con 0x7f02600cb4c0 2024-01-26T23:46:23.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.987+0000 7f0267822700 1 -- 172.21.15.46:0/2858447454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02600cb4c0 msgr2=0x7f0260159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.987+0000 7f0267822700 1 --2- 172.21.15.46:0/2858447454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02600cb4c0 0x7f0260159ab0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f02500098e0 tx=0x7f02500091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.987+0000 7f0267822700 1 -- 172.21.15.46:0/2858447454 shutdown_connections 2024-01-26T23:46:23.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.987+0000 7f0267822700 1 --2- 172.21.15.46:0/2858447454 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02600c4720 0x7f0260068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.987+0000 7f0267822700 1 --2- 172.21.15.46:0/2858447454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02600cb4c0 0x7f0260159ab0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.987+0000 7f0267822700 1 -- 172.21.15.46:0/2858447454 >> 172.21.15.46:0/2858447454 conn(0x7f0260097820 msgr2=0x7f0260068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:23.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.987+0000 7f0267822700 1 -- 172.21.15.46:0/2858447454 shutdown_connections 2024-01-26T23:46:23.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.987+0000 7f0267822700 1 -- 172.21.15.46:0/2858447454 wait complete. 2024-01-26T23:46:23.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.988+0000 7f0267822700 1 Processor -- start 2024-01-26T23:46:23.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.988+0000 7f0267822700 1 -- start start 2024-01-26T23:46:23.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.989+0000 7f0267822700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02600c4720 0x7f026015a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.989+0000 7f0267822700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02600cb4c0 0x7f02600b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.989+0000 7f0267822700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02601596e0 0x7f026014d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.989+0000 7f0267822700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026015b2b0 con 0x7f02600c4720 2024-01-26T23:46:23.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.989+0000 7f0267822700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02600b24a0 con 0x7f02600cb4c0 2024-01-26T23:46:23.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.989+0000 7f0267822700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026015a6a0 con 0x7f02601596e0 2024-01-26T23:46:23.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.989+0000 7f02655be700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02600c4720 0x7f026015a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.989+0000 7f02655be700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02600c4720 0x7f026015a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46484/0 (socket says 172.21.15.46:46484) 2024-01-26T23:46:23.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.989+0000 7f02655be700 1 -- 172.21.15.46:0/1887139720 learned_addr learned my addr 172.21.15.46:0/1887139720 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:23.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.990+0000 7f0264dbd700 1 --2- 172.21.15.46:0/1887139720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02600cb4c0 0x7f02600b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.990+0000 7f0265dbf700 1 --2- 172.21.15.46:0/1887139720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02601596e0 0x7f026014d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.990+0000 7f02655be700 1 -- 172.21.15.46:0/1887139720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02601596e0 msgr2=0x7f026014d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.990+0000 7f02655be700 1 --2- 172.21.15.46:0/1887139720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02601596e0 0x7f026014d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.990+0000 7f02655be700 1 -- 172.21.15.46:0/1887139720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02600cb4c0 msgr2=0x7f02600b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.990+0000 7f02655be700 1 --2- 172.21.15.46:0/1887139720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02600cb4c0 0x7f02600b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.990+0000 7f02655be700 1 -- 172.21.15.46:0/1887139720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0258009a20 con 0x7f02600c4720 2024-01-26T23:46:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.990+0000 7f0264dbd700 1 --2- 172.21.15.46:0/1887139720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02600cb4c0 0x7f02600b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.990+0000 7f02655be700 1 --2- 172.21.15.46:0/1887139720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02600c4720 0x7f026015a050 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f025c00c5c0 tx=0x7f025c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.990+0000 7f02557fa700 1 -- 172.21.15.46:0/1887139720 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c005b30 con 0x7f02600c4720 2024-01-26T23:46:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.990+0000 7f02557fa700 1 -- 172.21.15.46:0/1887139720 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025c005cb0 con 0x7f02600c4720 2024-01-26T23:46:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.990+0000 7f0267822700 1 -- 172.21.15.46:0/1887139720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0250009420 con 0x7f02600c4720 2024-01-26T23:46:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.990+0000 7f02557fa700 1 -- 172.21.15.46:0/1887139720 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c007320 con 0x7f02600c4720 2024-01-26T23:46:23.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.990+0000 7f0267822700 1 -- 172.21.15.46:0/1887139720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f026015ac60 con 0x7f02600c4720 2024-01-26T23:46:23.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.991+0000 7f0267822700 1 -- 172.21.15.46:0/1887139720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f026004c710 con 0x7f02600c4720 2024-01-26T23:46:23.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.992+0000 7f02557fa700 1 -- 172.21.15.46:0/1887139720 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f025c01b3b0 con 0x7f02600c4720 2024-01-26T23:46:23.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.992+0000 7f02557fa700 1 --2- 172.21.15.46:0/1887139720 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0244066ea0 0x7f0244069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:23.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.992+0000 7f02557fa700 1 -- 172.21.15.46:0/1887139720 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f025c083a00 con 0x7f02600c4720 2024-01-26T23:46:23.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.992+0000 7f0264dbd700 1 --2- 172.21.15.46:0/1887139720 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0244066ea0 0x7f0244069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:23.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.993+0000 7f0264dbd700 1 --2- 172.21.15.46:0/1887139720 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0244066ea0 0x7f0244069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f02600cb8c0 tx=0x7f0250017000 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:23.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:23.997+0000 7f02557fa700 1 -- 172.21.15.46:0/1887139720 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f025c050c10 con 0x7f02600c4720 2024-01-26T23:46:24.029 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:24.028+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:24.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.286+0000 7f0267822700 1 -- 172.21.15.46:0/1887139720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f02600c91a0 con 0x7f02600c4720 2024-01-26T23:46:24.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.687+0000 7f02557fa700 1 -- 172.21.15.46:0/1887139720 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v621) v1 ==== 166+0+0 (secure 0 0 0) 0x7f025c017020 con 0x7f02600c4720 2024-01-26T23:46:24.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.690+0000 7f0267822700 1 -- 172.21.15.46:0/1887139720 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0244066ea0 msgr2=0x7f0244069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:24.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.690+0000 7f0267822700 1 --2- 172.21.15.46:0/1887139720 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0244066ea0 0x7f0244069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f02600cb8c0 tx=0x7f0250017000 comp rx=0 tx=0).stop 2024-01-26T23:46:24.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.690+0000 7f0267822700 1 -- 172.21.15.46:0/1887139720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02600c4720 msgr2=0x7f026015a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:24.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.690+0000 7f0267822700 1 --2- 172.21.15.46:0/1887139720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02600c4720 0x7f026015a050 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f025c00c5c0 tx=0x7f025c007e00 comp rx=0 tx=0).stop 2024-01-26T23:46:24.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.690+0000 7f0267822700 1 -- 172.21.15.46:0/1887139720 shutdown_connections 2024-01-26T23:46:24.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.690+0000 7f0267822700 1 --2- 172.21.15.46:0/1887139720 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0244066ea0 0x7f0244069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:24.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.690+0000 7f0267822700 1 --2- 172.21.15.46:0/1887139720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02600c4720 0x7f026015a050 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:24.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.690+0000 7f0267822700 1 --2- 172.21.15.46:0/1887139720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02601596e0 0x7f026014d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:24.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.690+0000 7f0267822700 1 --2- 172.21.15.46:0/1887139720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02600cb4c0 0x7f02600b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:24.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.690+0000 7f0267822700 1 -- 172.21.15.46:0/1887139720 >> 172.21.15.46:0/1887139720 conn(0x7f0260097820 msgr2=0x7f0260158520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:24.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.691+0000 7f0267822700 1 -- 172.21.15.46:0/1887139720 shutdown_connections 2024-01-26T23:46:24.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.691+0000 7f0267822700 1 -- 172.21.15.46:0/1887139720 wait complete. 2024-01-26T23:46:24.707 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2024-01-26T23:46:24.707 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2024-01-26T23:46:24.707 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-26T23:46:24.708 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-01-26T23:46:24.809 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:24.808+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:24.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.845+0000 7f0c3c899700 1 Processor -- start 2024-01-26T23:46:24.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.845+0000 7f0c3c899700 1 -- start start 2024-01-26T23:46:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.846+0000 7f0c3c899700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c380acf30 0x7f0c380c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.846+0000 7f0c3c899700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c380c4230 con 0x7f0c380ad300 2024-01-26T23:46:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.846+0000 7f0c3c899700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c380cbf10 con 0x7f0c380ad770 2024-01-26T23:46:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.846+0000 7f0c3c899700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c38157be0 con 0x7f0c380acf30 2024-01-26T23:46:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.846+0000 7f0c35d9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0c380ad300 0x7f0c380ccb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37664/0 (socket says 172.21.15.46:37664) 2024-01-26T23:46:24.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.846+0000 7f0c35d9b700 1 -- 172.21.15.46:0/3260530075 learned_addr learned my addr 172.21.15.46:0/3260530075 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.846+0000 7f0c3659c700 1 --2- 172.21.15.46:0/3260530075 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c380acf30 0x7f0c380c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.847+0000 7f0c3559a700 1 -- 172.21.15.46:0/3260530075 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023639451 0 0) 0x7f0c380cbf10 con 0x7f0c380ad770 2024-01-26T23:46:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.847+0000 7f0c3559a700 1 -- 172.21.15.46:0/3260530075 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c24003570 con 0x7f0c380ad770 2024-01-26T23:46:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.847+0000 7f0c3559a700 1 -- 172.21.15.46:0/3260530075 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210292421 0 0) 0x7f0c380c4230 con 0x7f0c380ad300 2024-01-26T23:46:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.847+0000 7f0c3559a700 1 -- 172.21.15.46:0/3260530075 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c380cbf10 con 0x7f0c380ad300 2024-01-26T23:46:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.847+0000 7f0c3659c700 1 -- 172.21.15.46:0/3260530075 >> v1:172.21.15.106:6789/0 conn(0x7f0c380ad770 legacy=0x7f0c380ab1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.847+0000 7f0c3659c700 1 -- 172.21.15.46:0/3260530075 >> v1:172.21.15.46:6789/0 conn(0x7f0c380ad300 legacy=0x7f0c380ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.847+0000 7f0c3659c700 1 -- 172.21.15.46:0/3260530075 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c380c4460 con 0x7f0c380acf30 2024-01-26T23:46:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.847+0000 7f0c3659c700 1 --2- 172.21.15.46:0/3260530075 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c380acf30 0x7f0c380c3cf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0c28009910 tx=0x7f0c280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92dd060a8dfd80b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.847+0000 7f0c3559a700 1 -- 172.21.15.46:0/3260530075 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c28011630 con 0x7f0c380acf30 2024-01-26T23:46:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.847+0000 7f0c3559a700 1 -- 172.21.15.46:0/3260530075 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c28011790 con 0x7f0c380acf30 2024-01-26T23:46:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.847+0000 7f0c3559a700 1 -- 172.21.15.46:0/3260530075 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c28011b30 con 0x7f0c380acf30 2024-01-26T23:46:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.848+0000 7f0c3c899700 1 -- 172.21.15.46:0/3260530075 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c380acf30 msgr2=0x7f0c380c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.848+0000 7f0c3c899700 1 --2- 172.21.15.46:0/3260530075 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c380acf30 0x7f0c380c3cf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0c28009910 tx=0x7f0c280091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.848+0000 7f0c3c899700 1 -- 172.21.15.46:0/3260530075 shutdown_connections 2024-01-26T23:46:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.848+0000 7f0c3c899700 1 --2- 172.21.15.46:0/3260530075 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c380acf30 0x7f0c380c3cf0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.848+0000 7f0c3c899700 1 -- 172.21.15.46:0/3260530075 >> 172.21.15.46:0/3260530075 conn(0x7f0c38097820 msgr2=0x7f0c38068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.848+0000 7f0c3c899700 1 -- 172.21.15.46:0/3260530075 shutdown_connections 2024-01-26T23:46:24.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.848+0000 7f0c3c899700 1 -- 172.21.15.46:0/3260530075 wait complete. 2024-01-26T23:46:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.849+0000 7f0c3c899700 1 Processor -- start 2024-01-26T23:46:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.849+0000 7f0c3c899700 1 -- start start 2024-01-26T23:46:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.850+0000 7f0c3c899700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c380acf30 0x7f0c38148270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.850+0000 7f0c3c899700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c380ad300 0x7f0c3806a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.850+0000 7f0c3c899700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c38156180 0x7f0c38156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.850+0000 7f0c3c899700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c380b09f0 con 0x7f0c38156180 2024-01-26T23:46:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.850+0000 7f0c3c899700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c38156b60 con 0x7f0c380ad300 2024-01-26T23:46:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.850+0000 7f0c3c899700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c380b0170 con 0x7f0c380acf30 2024-01-26T23:46:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.850+0000 7f0c36d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c38156180 0x7f0c38156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.850+0000 7f0c36d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c38156180 0x7f0c38156620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46494/0 (socket says 172.21.15.46:46494) 2024-01-26T23:46:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.850+0000 7f0c36d9d700 1 -- 172.21.15.46:0/4206364716 learned_addr learned my addr 172.21.15.46:0/4206364716 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.850+0000 7f0c35d9b700 1 --2- 172.21.15.46:0/4206364716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c380ad300 0x7f0c3806a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.850+0000 7f0c3659c700 1 --2- 172.21.15.46:0/4206364716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c380acf30 0x7f0c38148270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.850+0000 7f0c36d9d700 1 -- 172.21.15.46:0/4206364716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c380acf30 msgr2=0x7f0c38148270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.850+0000 7f0c36d9d700 1 --2- 172.21.15.46:0/4206364716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c380acf30 0x7f0c38148270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.850+0000 7f0c36d9d700 1 -- 172.21.15.46:0/4206364716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c380ad300 msgr2=0x7f0c3806a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.850+0000 7f0c36d9d700 1 --2- 172.21.15.46:0/4206364716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c380ad300 0x7f0c3806a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.851+0000 7f0c36d9d700 1 -- 172.21.15.46:0/4206364716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c28009420 con 0x7f0c38156180 2024-01-26T23:46:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.851+0000 7f0c36d9d700 1 --2- 172.21.15.46:0/4206364716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c38156180 0x7f0c38156620 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f0c3000c860 tx=0x7f0c30002700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.851+0000 7f0c2e7fc700 1 -- 172.21.15.46:0/4206364716 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c30012630 con 0x7f0c38156180 2024-01-26T23:46:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.851+0000 7f0c2e7fc700 1 -- 172.21.15.46:0/4206364716 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c300127b0 con 0x7f0c38156180 2024-01-26T23:46:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.851+0000 7f0c3c899700 1 -- 172.21.15.46:0/4206364716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c380b0430 con 0x7f0c38156180 2024-01-26T23:46:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.851+0000 7f0c2e7fc700 1 -- 172.21.15.46:0/4206364716 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c30012b70 con 0x7f0c38156180 2024-01-26T23:46:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.851+0000 7f0c3c899700 1 -- 172.21.15.46:0/4206364716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c380c9550 con 0x7f0c38156180 2024-01-26T23:46:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.852+0000 7f0c3c899700 1 -- 172.21.15.46:0/4206364716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c380c93b0 con 0x7f0c38156180 2024-01-26T23:46:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.852+0000 7f0c2e7fc700 1 -- 172.21.15.46:0/4206364716 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f0c3001b400 con 0x7f0c38156180 2024-01-26T23:46:24.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.853+0000 7f0c2e7fc700 1 --2- 172.21.15.46:0/4206364716 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c18066ea0 0x7f0c18069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:24.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.853+0000 7f0c2e7fc700 1 -- 172.21.15.46:0/4206364716 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f0c30084850 con 0x7f0c38156180 2024-01-26T23:46:24.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.853+0000 7f0c3659c700 1 --2- 172.21.15.46:0/4206364716 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c18066ea0 0x7f0c18069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:24.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.854+0000 7f0c3659c700 1 --2- 172.21.15.46:0/4206364716 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c18066ea0 0x7f0c18069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0c28009860 tx=0x7f0c28017000 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:24.858+0000 7f0c2e7fc700 1 -- 172.21.15.46:0/4206364716 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0c380c93b0 con 0x7f0c38156180 2024-01-26T23:46:25.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.159+0000 7f0c3c899700 1 -- 172.21.15.46:0/4206364716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f0c380c93b0 con 0x7f0c38156180 2024-01-26T23:46:25.246 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:46:25.248 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:46:25.590 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:25.589+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:25.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.691+0000 7f0c2e7fc700 1 -- 172.21.15.46:0/4206364716 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v622) v1 ==== 186+0+0 (secure 0 0 0) 0x7f0c380c93b0 con 0x7f0c38156180 2024-01-26T23:46:25.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.693+0000 7f0c3c899700 1 -- 172.21.15.46:0/4206364716 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c18066ea0 msgr2=0x7f0c18069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:25.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.693+0000 7f0c3c899700 1 --2- 172.21.15.46:0/4206364716 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c18066ea0 0x7f0c18069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0c28009860 tx=0x7f0c28017000 comp rx=0 tx=0).stop 2024-01-26T23:46:25.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.693+0000 7f0c3c899700 1 -- 172.21.15.46:0/4206364716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c38156180 msgr2=0x7f0c38156620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:25.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.693+0000 7f0c3c899700 1 --2- 172.21.15.46:0/4206364716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c38156180 0x7f0c38156620 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f0c3000c860 tx=0x7f0c30002700 comp rx=0 tx=0).stop 2024-01-26T23:46:25.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.694+0000 7f0c3c899700 1 -- 172.21.15.46:0/4206364716 shutdown_connections 2024-01-26T23:46:25.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.694+0000 7f0c3c899700 1 --2- 172.21.15.46:0/4206364716 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c18066ea0 0x7f0c18069350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:25.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.694+0000 7f0c3c899700 1 --2- 172.21.15.46:0/4206364716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c38156180 0x7f0c38156620 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:25.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.694+0000 7f0c3c899700 1 --2- 172.21.15.46:0/4206364716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c380acf30 0x7f0c38148270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:25.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.694+0000 7f0c3c899700 1 --2- 172.21.15.46:0/4206364716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c380ad300 0x7f0c3806a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:25.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.694+0000 7f0c3c899700 1 -- 172.21.15.46:0/4206364716 >> 172.21.15.46:0/4206364716 conn(0x7f0c38097820 msgr2=0x7f0c3815ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:25.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.694+0000 7f0c3c899700 1 -- 172.21.15.46:0/4206364716 shutdown_connections 2024-01-26T23:46:25.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.694+0000 7f0c3c899700 1 -- 172.21.15.46:0/4206364716 wait complete. 2024-01-26T23:46:25.711 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-26T23:46:25.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.850+0000 7fe584adb700 1 Processor -- start 2024-01-26T23:46:25.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.850+0000 7fe584adb700 1 -- start start 2024-01-26T23:46:25.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.851+0000 7fe584adb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe580157840 0x7fe5800ca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:25.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.851+0000 7fe584adb700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe58006c110 0x7fe580067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:25.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.851+0000 7fe584adb700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5800b02d0 con 0x7fe580067e70 2024-01-26T23:46:25.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.851+0000 7fe584adb700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe580068310 con 0x7fe580157840 2024-01-26T23:46:25.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.851+0000 7fe584adb700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58006c680 con 0x7fe58006c110 2024-01-26T23:46:25.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.851+0000 7fe57e59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe580157840 0x7fe5800ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:25.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.851+0000 7fe57ed9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe580067e70 0x7fe58006eb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37666/0 (socket says 172.21.15.46:37666) 2024-01-26T23:46:25.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.851+0000 7fe57e59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe580157840 0x7fe5800ca810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46506/0 (socket says 172.21.15.46:46506) 2024-01-26T23:46:25.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.851+0000 7fe57ed9d700 1 -- 172.21.15.46:0/3447877153 learned_addr learned my addr 172.21.15.46:0/3447877153 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:25.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.851+0000 7fe57dd9b700 1 --2- 172.21.15.46:0/3447877153 >> v2:172.21.15.106:3300/0 conn(0x7fe58006c110 0x7fe580067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.851+0000 7fe57e59c700 1 -- 172.21.15.46:0/3447877153 >> v2:172.21.15.106:3300/0 conn(0x7fe58006c110 msgr2=0x7fe580067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.851+0000 7fe57e59c700 1 --2- 172.21.15.46:0/3447877153 >> v2:172.21.15.106:3300/0 conn(0x7fe58006c110 0x7fe580067930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.851+0000 7fe57e59c700 1 -- 172.21.15.46:0/3447877153 >> v1:172.21.15.46:6789/0 conn(0x7fe580067e70 legacy=0x7fe58006eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.851+0000 7fe57e59c700 1 -- 172.21.15.46:0/3447877153 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58006c960 con 0x7fe580157840 2024-01-26T23:46:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.851+0000 7fe57d59a700 1 -- 172.21.15.46:0/3447877153 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187660141 0 0) 0x7fe5800b02d0 con 0x7fe580067e70 2024-01-26T23:46:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.852+0000 7fe57e59c700 1 --2- 172.21.15.46:0/3447877153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe580157840 0x7fe5800ca810 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fe5800c9ec0 tx=0x7fe5680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4be37f9b5a10c9e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.852+0000 7fe57d59a700 1 -- 172.21.15.46:0/3447877153 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe568011630 con 0x7fe580157840 2024-01-26T23:46:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.852+0000 7fe57d59a700 1 -- 172.21.15.46:0/3447877153 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe568011790 con 0x7fe580157840 2024-01-26T23:46:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.852+0000 7fe57d59a700 1 -- 172.21.15.46:0/3447877153 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe568011b30 con 0x7fe580157840 2024-01-26T23:46:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.852+0000 7fe584adb700 1 -- 172.21.15.46:0/3447877153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe580157840 msgr2=0x7fe5800ca810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.852+0000 7fe584adb700 1 --2- 172.21.15.46:0/3447877153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe580157840 0x7fe5800ca810 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fe5800c9ec0 tx=0x7fe5680091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.852+0000 7fe584adb700 1 -- 172.21.15.46:0/3447877153 shutdown_connections 2024-01-26T23:46:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.852+0000 7fe584adb700 1 --2- 172.21.15.46:0/3447877153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe580157840 0x7fe5800ca810 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.852+0000 7fe584adb700 1 --2- 172.21.15.46:0/3447877153 >> v2:172.21.15.106:3300/0 conn(0x7fe58006c110 0x7fe580067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:25.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.852+0000 7fe584adb700 1 -- 172.21.15.46:0/3447877153 >> 172.21.15.46:0/3447877153 conn(0x7fe5800cdf50 msgr2=0x7fe5800c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:25.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.852+0000 7fe584adb700 1 -- 172.21.15.46:0/3447877153 shutdown_connections 2024-01-26T23:46:25.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.852+0000 7fe584adb700 1 -- 172.21.15.46:0/3447877153 wait complete. 2024-01-26T23:46:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.854+0000 7fe584adb700 1 Processor -- start 2024-01-26T23:46:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.854+0000 7fe584adb700 1 -- start start 2024-01-26T23:46:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.854+0000 7fe584adb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe580067e70 0x7fe580145110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.854+0000 7fe584adb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe58006c110 0x7fe5800ac8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.854+0000 7fe584adb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe580157840 0x7fe58015c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.854+0000 7fe584adb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe580145650 con 0x7fe580157840 2024-01-26T23:46:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.854+0000 7fe584adb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5800ad290 con 0x7fe58006c110 2024-01-26T23:46:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.854+0000 7fe584adb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5800ad570 con 0x7fe580067e70 2024-01-26T23:46:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.855+0000 7fe57ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe580157840 0x7fe58015c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.855+0000 7fe57ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe580157840 0x7fe58015c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46508/0 (socket says 172.21.15.46:46508) 2024-01-26T23:46:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.855+0000 7fe57ed9d700 1 -- 172.21.15.46:0/1224598762 learned_addr learned my addr 172.21.15.46:0/1224598762 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.855+0000 7fe57e59c700 1 --2- 172.21.15.46:0/1224598762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe580067e70 0x7fe580145110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.855+0000 7fe57dd9b700 1 --2- 172.21.15.46:0/1224598762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe58006c110 0x7fe5800ac8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.855+0000 7fe57ed9d700 1 -- 172.21.15.46:0/1224598762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe580067e70 msgr2=0x7fe580145110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.855+0000 7fe57ed9d700 1 --2- 172.21.15.46:0/1224598762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe580067e70 0x7fe580145110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.855+0000 7fe57ed9d700 1 -- 172.21.15.46:0/1224598762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe58006c110 msgr2=0x7fe5800ac8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.855+0000 7fe57ed9d700 1 --2- 172.21.15.46:0/1224598762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe58006c110 0x7fe5800ac8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.855+0000 7fe57ed9d700 1 -- 172.21.15.46:0/1224598762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe568009420 con 0x7fe580157840 2024-01-26T23:46:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.855+0000 7fe57dd9b700 1 --2- 172.21.15.46:0/1224598762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe58006c110 0x7fe5800ac8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:46:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.855+0000 7fe57ed9d700 1 --2- 172.21.15.46:0/1224598762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe580157840 0x7fe58015c020 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fe57800cc90 tx=0x7fe578001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.855+0000 7fe5767fc700 1 -- 172.21.15.46:0/1224598762 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57800a960 con 0x7fe580157840 2024-01-26T23:46:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.855+0000 7fe5767fc700 1 -- 172.21.15.46:0/1224598762 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57800aac0 con 0x7fe580157840 2024-01-26T23:46:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.855+0000 7fe5767fc700 1 -- 172.21.15.46:0/1224598762 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57800ae80 con 0x7fe580157840 2024-01-26T23:46:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.855+0000 7fe584adb700 1 -- 172.21.15.46:0/1224598762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5800ad850 con 0x7fe580157840 2024-01-26T23:46:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.855+0000 7fe584adb700 1 -- 172.21.15.46:0/1224598762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5800ade10 con 0x7fe580157840 2024-01-26T23:46:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.856+0000 7fe584adb700 1 -- 172.21.15.46:0/1224598762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5800ae030 con 0x7fe580157840 2024-01-26T23:46:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.857+0000 7fe5767fc700 1 -- 172.21.15.46:0/1224598762 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fe578007300 con 0x7fe580157840 2024-01-26T23:46:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.857+0000 7fe5767fc700 1 --2- 172.21.15.46:0/1224598762 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe560066ea0 0x7fe560069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.858+0000 7fe5767fc700 1 -- 172.21.15.46:0/1224598762 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6210+0+0 (secure 0 0 0) 0x7fe578084a90 con 0x7fe580157840 2024-01-26T23:46:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.858+0000 7fe57e59c700 1 --2- 172.21.15.46:0/1224598762 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe560066ea0 0x7fe560069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:25.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.858+0000 7fe57e59c700 1 --2- 172.21.15.46:0/1224598762 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe560066ea0 0x7fe560069350 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe5680091e0 tx=0x7fe568017000 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:25.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:25.862+0000 7fe5767fc700 1 -- 172.21.15.46:0/1224598762 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe578051bc0 con 0x7fe580157840 2024-01-26T23:46:26.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.069+0000 7fe584adb700 1 -- 172.21.15.46:0/1224598762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe58004c710 con 0x7fe580157840 2024-01-26T23:46:26.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.073+0000 7fe5767fc700 1 -- 172.21.15.46:0/1224598762 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe5780551e0 con 0x7fe580157840 2024-01-26T23:46:26.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.075+0000 7fe584adb700 1 -- 172.21.15.46:0/1224598762 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe560066ea0 msgr2=0x7fe560069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:26.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.075+0000 7fe584adb700 1 --2- 172.21.15.46:0/1224598762 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe560066ea0 0x7fe560069350 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe5680091e0 tx=0x7fe568017000 comp rx=0 tx=0).stop 2024-01-26T23:46:26.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.075+0000 7fe584adb700 1 -- 172.21.15.46:0/1224598762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe580157840 msgr2=0x7fe58015c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:26.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.075+0000 7fe584adb700 1 --2- 172.21.15.46:0/1224598762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe580157840 0x7fe58015c020 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fe57800cc90 tx=0x7fe578001a80 comp rx=0 tx=0).stop 2024-01-26T23:46:26.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.075+0000 7fe584adb700 1 -- 172.21.15.46:0/1224598762 shutdown_connections 2024-01-26T23:46:26.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.075+0000 7fe584adb700 1 --2- 172.21.15.46:0/1224598762 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fe560066ea0 0x7fe560069350 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.075+0000 7fe584adb700 1 --2- 172.21.15.46:0/1224598762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe580157840 0x7fe58015c020 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.075+0000 7fe584adb700 1 --2- 172.21.15.46:0/1224598762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe580067e70 0x7fe580145110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.075+0000 7fe584adb700 1 --2- 172.21.15.46:0/1224598762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe58006c110 0x7fe5800ac8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.075+0000 7fe584adb700 1 -- 172.21.15.46:0/1224598762 >> 172.21.15.46:0/1224598762 conn(0x7fe5800cdf50 msgr2=0x7fe580156aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:26.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.075+0000 7fe584adb700 1 -- 172.21.15.46:0/1224598762 shutdown_connections 2024-01-26T23:46:26.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.076+0000 7fe584adb700 1 -- 172.21.15.46:0/1224598762 wait complete. 2024-01-26T23:46:26.090 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-26T23:46:26.091 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:46:26.137 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:46:26.179 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-26T23:46:26.180 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-26T23:46:26.180 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:46:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.317+0000 7f5eaa9a7700 1 Processor -- start 2024-01-26T23:46:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.317+0000 7f5eaa9a7700 1 -- start start 2024-01-26T23:46:26.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.318+0000 7f5eaa9a7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ea4159760 0x7f5ea40b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:26.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.318+0000 7f5eaa9a7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5ea40b12c0 0x7f5ea40a9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:26.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.318+0000 7f5eaa9a7700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea406f940 con 0x7f5ea40b3060 2024-01-26T23:46:26.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.318+0000 7f5eaa9a7700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ea406ed30 con 0x7f5ea40b12c0 2024-01-26T23:46:26.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.318+0000 7f5eaa9a7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea406f010 con 0x7f5ea4159760 2024-01-26T23:46:26.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.318+0000 7f5ea8f44700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5ea40b12c0 0x7f5ea40a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:26.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.318+0000 7f5ea8f44700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5ea40b12c0 0x7f5ea40a9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46516/0 (socket says 172.21.15.46:46516) 2024-01-26T23:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.318+0000 7f5ea8f44700 1 -- 172.21.15.46:0/2129095443 learned_addr learned my addr 172.21.15.46:0/2129095443 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.318+0000 7f5ea3fff700 1 --2- 172.21.15.46:0/2129095443 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ea4159760 0x7f5ea40b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.318+0000 7f5ea8f44700 1 -- 172.21.15.46:0/2129095443 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ea4159760 msgr2=0x7f5ea40b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.318+0000 7f5ea8f44700 1 --2- 172.21.15.46:0/2129095443 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ea4159760 0x7f5ea40b2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.318+0000 7f5ea8f44700 1 -- 172.21.15.46:0/2129095443 >> v1:172.21.15.106:6789/0 conn(0x7f5ea40b3060 legacy=0x7f5ea40b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.318+0000 7f5ea8f44700 1 -- 172.21.15.46:0/2129095443 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea406f2f0 con 0x7f5ea40b12c0 2024-01-26T23:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.318+0000 7f5ea8f44700 1 --2- 172.21.15.46:0/2129095443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea40b12c0 0x7f5ea40a9840 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c009f20 tx=0x7f5e9c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=973f6717aa177ba0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.319+0000 7f5ea2ffd700 1 -- 172.21.15.46:0/2129095443 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c00e070 con 0x7f5ea40b12c0 2024-01-26T23:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.319+0000 7f5ea2ffd700 1 -- 172.21.15.46:0/2129095443 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9c00b9d0 con 0x7f5ea40b12c0 2024-01-26T23:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.319+0000 7f5ea2ffd700 1 -- 172.21.15.46:0/2129095443 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c0092e0 con 0x7f5ea40b12c0 2024-01-26T23:46:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.319+0000 7f5eaa9a7700 1 -- 172.21.15.46:0/2129095443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea40b12c0 msgr2=0x7f5ea40a9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.319+0000 7f5eaa9a7700 1 --2- 172.21.15.46:0/2129095443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea40b12c0 0x7f5ea40a9840 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c009f20 tx=0x7f5e9c009840 comp rx=0 tx=0).stop 2024-01-26T23:46:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.319+0000 7f5eaa9a7700 1 -- 172.21.15.46:0/2129095443 shutdown_connections 2024-01-26T23:46:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.319+0000 7f5eaa9a7700 1 --2- 172.21.15.46:0/2129095443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea40b12c0 0x7f5ea40a9840 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.319+0000 7f5eaa9a7700 1 --2- 172.21.15.46:0/2129095443 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ea4159760 0x7f5ea40b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.319+0000 7f5eaa9a7700 1 -- 172.21.15.46:0/2129095443 >> 172.21.15.46:0/2129095443 conn(0x7f5ea4069510 msgr2=0x7f5ea40aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:26.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.319+0000 7f5eaa9a7700 1 -- 172.21.15.46:0/2129095443 shutdown_connections 2024-01-26T23:46:26.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.319+0000 7f5eaa9a7700 1 -- 172.21.15.46:0/2129095443 wait complete. 2024-01-26T23:46:26.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.321+0000 7f5eaa9a7700 1 Processor -- start 2024-01-26T23:46:26.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.321+0000 7f5eaa9a7700 1 -- start start 2024-01-26T23:46:26.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5eaa9a7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ea40b12c0 0x7f5ea4155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:26.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5eaa9a7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ea40b3060 0x7f5ea416cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:26.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5eaa9a7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4159760 0x7f5ea416d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:26.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5eaa9a7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea415b490 con 0x7f5ea4159760 2024-01-26T23:46:26.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5eaa9a7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea4150eb0 con 0x7f5ea40b3060 2024-01-26T23:46:26.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5eaa9a7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea4151190 con 0x7f5ea40b12c0 2024-01-26T23:46:26.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5ea8f44700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4159760 0x7f5ea416d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:26.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5ea8f44700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4159760 0x7f5ea416d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46524/0 (socket says 172.21.15.46:46524) 2024-01-26T23:46:26.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5ea8f44700 1 -- 172.21.15.46:0/1938651065 learned_addr learned my addr 172.21.15.46:0/1938651065 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:26.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5ea3fff700 1 --2- 172.21.15.46:0/1938651065 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ea40b12c0 0x7f5ea4155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:26.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5ea37fe700 1 --2- 172.21.15.46:0/1938651065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ea40b3060 0x7f5ea416cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:26.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5ea8f44700 1 -- 172.21.15.46:0/1938651065 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ea40b12c0 msgr2=0x7f5ea4155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:26.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5ea8f44700 1 --2- 172.21.15.46:0/1938651065 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ea40b12c0 0x7f5ea4155270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5ea8f44700 1 -- 172.21.15.46:0/1938651065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ea40b3060 msgr2=0x7f5ea416cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:26.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5ea8f44700 1 --2- 172.21.15.46:0/1938651065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ea40b3060 0x7f5ea416cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5ea8f44700 1 -- 172.21.15.46:0/1938651065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c009ae0 con 0x7f5ea4159760 2024-01-26T23:46:26.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5ea8f44700 1 --2- 172.21.15.46:0/1938651065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4159760 0x7f5ea416d010 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f5ea40ce460 tx=0x7f5e9c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:26.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.322+0000 7f5e8bfff700 1 -- 172.21.15.46:0/1938651065 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c008ad0 con 0x7f5ea4159760 2024-01-26T23:46:26.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.323+0000 7f5e8bfff700 1 -- 172.21.15.46:0/1938651065 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9c008c30 con 0x7f5ea4159760 2024-01-26T23:46:26.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.323+0000 7f5eaa9a7700 1 -- 172.21.15.46:0/1938651065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea4151470 con 0x7f5ea4159760 2024-01-26T23:46:26.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.323+0000 7f5e8bfff700 1 -- 172.21.15.46:0/1938651065 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c026460 con 0x7f5ea4159760 2024-01-26T23:46:26.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.323+0000 7f5eaa9a7700 1 -- 172.21.15.46:0/1938651065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea414ffa0 con 0x7f5ea4159760 2024-01-26T23:46:26.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.324+0000 7f5eaa9a7700 1 -- 172.21.15.46:0/1938651065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea4151660 con 0x7f5ea4159760 2024-01-26T23:46:26.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.324+0000 7f5e8bfff700 1 -- 172.21.15.46:0/1938651065 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f5e9c003040 con 0x7f5ea4159760 2024-01-26T23:46:26.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.325+0000 7f5e8bfff700 1 --2- 172.21.15.46:0/1938651065 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5e90066ea0 0x7f5e90069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:26.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.325+0000 7f5e8bfff700 1 -- 172.21.15.46:0/1938651065 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f5e9c086460 con 0x7f5ea4159760 2024-01-26T23:46:26.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.325+0000 7f5ea3fff700 1 --2- 172.21.15.46:0/1938651065 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5e90066ea0 0x7f5e90069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:26.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.325+0000 7f5ea3fff700 1 --2- 172.21.15.46:0/1938651065 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5e90066ea0 0x7f5e90069350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009d10 tx=0x7f5e940091a0 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:26.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.329+0000 7f5e8bfff700 1 -- 172.21.15.46:0/1938651065 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5e9c0535b0 con 0x7f5ea4159760 2024-01-26T23:46:26.391 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:26.390+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:26.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.583+0000 7f5eaa9a7700 1 -- 172.21.15.46:0/1938651065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ea404c710 con 0x7f5ea4159760 2024-01-26T23:46:26.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.584+0000 7f5e8bfff700 1 -- 172.21.15.46:0/1938651065 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f5e9c021de0 con 0x7f5ea4159760 2024-01-26T23:46:26.585 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:26.585 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":47,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6137,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/1370365478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1370365478},{"type":"v1","addr":"172.21.15.106:6836","nonce":1370365478}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":39},{"gid":6619,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/1980309094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1980309094},{"type":"v1","addr":"172.21.15.106:6837","nonce":1980309094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40},{"gid":6628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3910368747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3910368747},{"type":"v1","addr":"172.21.15.46:6834","nonce":3910368747}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40},{"gid":7179,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3220187957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3220187957},{"type":"v1","addr":"172.21.15.46:6835","nonce":3220187957}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":47}],"filesystems":[]} 2024-01-26T23:46:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.586+0000 7f5eaa9a7700 1 -- 172.21.15.46:0/1938651065 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5e90066ea0 msgr2=0x7f5e90069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.586+0000 7f5eaa9a7700 1 --2- 172.21.15.46:0/1938651065 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5e90066ea0 0x7f5e90069350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009d10 tx=0x7f5e940091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.586+0000 7f5eaa9a7700 1 -- 172.21.15.46:0/1938651065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4159760 msgr2=0x7f5ea416d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.586+0000 7f5eaa9a7700 1 --2- 172.21.15.46:0/1938651065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4159760 0x7f5ea416d010 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f5ea40ce460 tx=0x7f5e9c018000 comp rx=0 tx=0).stop 2024-01-26T23:46:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.586+0000 7f5eaa9a7700 1 -- 172.21.15.46:0/1938651065 shutdown_connections 2024-01-26T23:46:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.586+0000 7f5eaa9a7700 1 --2- 172.21.15.46:0/1938651065 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5e90066ea0 0x7f5e90069350 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.586+0000 7f5eaa9a7700 1 --2- 172.21.15.46:0/1938651065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4159760 0x7f5ea416d010 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.586+0000 7f5eaa9a7700 1 --2- 172.21.15.46:0/1938651065 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ea40b12c0 0x7f5ea4155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.586+0000 7f5eaa9a7700 1 --2- 172.21.15.46:0/1938651065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ea40b3060 0x7f5ea416cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.586+0000 7f5eaa9a7700 1 -- 172.21.15.46:0/1938651065 >> 172.21.15.46:0/1938651065 conn(0x7f5ea4069510 msgr2=0x7f5ea40b04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.587+0000 7f5eaa9a7700 1 -- 172.21.15.46:0/1938651065 shutdown_connections 2024-01-26T23:46:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.587+0000 7f5eaa9a7700 1 -- 172.21.15.46:0/1938651065 wait complete. 2024-01-26T23:46:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 47 2024-01-26T23:46:26.602 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-26T23:46:26.602 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-26T23:46:26.602 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-26T23:46:26.602 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:46:26.602 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-26T23:46:26.602 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-26T23:46:26.603 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-26T23:46:26.603 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:46:26.603 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-26T23:46:26.603 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-26T23:46:26.603 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-26T23:46:26.603 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:46:26.603 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-26T23:46:26.603 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-26T23:46:26.604 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-26T23:46:26.604 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:46:26.612 INFO:tasks.ceph.mds.b:Stopped 2024-01-26T23:46:26.612 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-26T23:46:26.613 INFO:tasks.ceph.mds.d:Stopped 2024-01-26T23:46:26.613 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-26T23:46:26.614 INFO:tasks.ceph.mds.c:Stopped 2024-01-26T23:46:26.614 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-26T23:46:26.615 INFO:tasks.ceph.mds.a:Stopped 2024-01-26T23:46:26.615 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-26T23:46:26.655 INFO:tasks.ceph.mds.b:Started 2024-01-26T23:46:26.655 DEBUG:teuthology.parallel:result is None 2024-01-26T23:46:26.656 INFO:tasks.ceph.mds.d:Started 2024-01-26T23:46:26.657 DEBUG:teuthology.parallel:result is None 2024-01-26T23:46:26.657 INFO:tasks.ceph.mds.c:Started 2024-01-26T23:46:26.658 DEBUG:teuthology.parallel:result is None 2024-01-26T23:46:26.658 INFO:tasks.ceph.mds.a:Started 2024-01-26T23:46:26.658 DEBUG:teuthology.parallel:result is None 2024-01-26T23:46:26.659 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-26T23:46:26.732 INFO:tasks.ceph.mds.b.smithi106.stdout:starting mds.b at 2024-01-26T23:46:26.732 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-01-26T23:46:26.746 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-26T23:46:26.746 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-26T23:46:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.803+0000 7f43395fd700 1 Processor -- start 2024-01-26T23:46:26.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.803+0000 7f43395fd700 1 -- start start 2024-01-26T23:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.804+0000 7f43395fd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f43340b2b40 0x7f43340ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.804+0000 7f43395fd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43340aedd0 0x7f43340cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.804+0000 7f43395fd700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f433415a0b0 con 0x7f43340ae960 2024-01-26T23:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.804+0000 7f43395fd700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4334099fc0 con 0x7f43340b2b40 2024-01-26T23:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.804+0000 7f43395fd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43340ae4d0 con 0x7f43340aedd0 2024-01-26T23:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.804+0000 7f4332ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f43340b2b40 0x7f43340ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.804+0000 7f4332ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f43340b2b40 0x7f43340ad1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46578/0 (socket says 172.21.15.46:46578) 2024-01-26T23:46:26.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.804+0000 7f4332ffd700 1 -- 172.21.15.46:0/1145810383 learned_addr learned my addr 172.21.15.46:0/1145810383 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:26.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.805+0000 7f43337fe700 1 --2- 172.21.15.46:0/1145810383 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43340aedd0 0x7f43340cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:26.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.805+0000 7f4332ffd700 1 -- 172.21.15.46:0/1145810383 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43340aedd0 msgr2=0x7f43340cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:26.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.805+0000 7f4332ffd700 1 --2- 172.21.15.46:0/1145810383 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43340aedd0 0x7f43340cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.805+0000 7f4332ffd700 1 -- 172.21.15.46:0/1145810383 >> v1:172.21.15.106:6789/0 conn(0x7f43340ae960 legacy=0x7f43340ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:26.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.805+0000 7f4332ffd700 1 -- 172.21.15.46:0/1145810383 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43340ad8c0 con 0x7f43340b2b40 2024-01-26T23:46:26.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.805+0000 7f4332ffd700 1 --2- 172.21.15.46:0/1145810383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43340b2b40 0x7f43340ad1c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f4324009990 tx=0x7f4324009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555b0b0142f45116 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:26.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.805+0000 7f4331ffb700 1 -- 172.21.15.46:0/1145810383 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324009e80 con 0x7f43340b2b40 2024-01-26T23:46:26.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.805+0000 7f4331ffb700 1 -- 172.21.15.46:0/1145810383 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4324005c10 con 0x7f43340b2b40 2024-01-26T23:46:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.805+0000 7f4331ffb700 1 -- 172.21.15.46:0/1145810383 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432400b650 con 0x7f43340b2b40 2024-01-26T23:46:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.806+0000 7f43395fd700 1 -- 172.21.15.46:0/1145810383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43340b2b40 msgr2=0x7f43340ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.806+0000 7f43395fd700 1 --2- 172.21.15.46:0/1145810383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43340b2b40 0x7f43340ad1c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f4324009990 tx=0x7f4324009220 comp rx=0 tx=0).stop 2024-01-26T23:46:26.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.806+0000 7f43395fd700 1 -- 172.21.15.46:0/1145810383 shutdown_connections 2024-01-26T23:46:26.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.806+0000 7f43395fd700 1 --2- 172.21.15.46:0/1145810383 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43340aedd0 0x7f43340cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.806+0000 7f43395fd700 1 --2- 172.21.15.46:0/1145810383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43340b2b40 0x7f43340ad1c0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.806+0000 7f43395fd700 1 -- 172.21.15.46:0/1145810383 >> 172.21.15.46:0/1145810383 conn(0x7f4334099a30 msgr2=0x7f433406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.806+0000 7f43395fd700 1 -- 172.21.15.46:0/1145810383 shutdown_connections 2024-01-26T23:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.806+0000 7f43395fd700 1 -- 172.21.15.46:0/1145810383 wait complete. 2024-01-26T23:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.807+0000 7f43395fd700 1 Processor -- start 2024-01-26T23:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.808+0000 7f43395fd700 1 -- start start 2024-01-26T23:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.808+0000 7f43395fd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43340ae960 0x7f4334158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.808+0000 7f43395fd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43340b2b40 0x7f43340b4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.808+0000 7f43395fd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43341537f0 0x7f4334068910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.808+0000 7f43395fd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4334153c90 con 0x7f43340ae960 2024-01-26T23:46:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.808+0000 7f43395fd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43340c81e0 con 0x7f43341537f0 2024-01-26T23:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.808+0000 7f43395fd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43340c8480 con 0x7f43340b2b40 2024-01-26T23:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.808+0000 7f4332ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43340ae960 0x7f4334158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.808+0000 7f4332ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43340ae960 0x7f4334158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46586/0 (socket says 172.21.15.46:46586) 2024-01-26T23:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.808+0000 7f4332ffd700 1 -- 172.21.15.46:0/2239084384 learned_addr learned my addr 172.21.15.46:0/2239084384 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.808+0000 7f43327fc700 1 --2- 172.21.15.46:0/2239084384 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43340b2b40 0x7f43340b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.809+0000 7f43337fe700 1 --2- 172.21.15.46:0/2239084384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43341537f0 0x7f4334068910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.809+0000 7f4332ffd700 1 -- 172.21.15.46:0/2239084384 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43340b2b40 msgr2=0x7f43340b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.809+0000 7f4332ffd700 1 --2- 172.21.15.46:0/2239084384 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43340b2b40 0x7f43340b4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.809+0000 7f4332ffd700 1 -- 172.21.15.46:0/2239084384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43341537f0 msgr2=0x7f4334068910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.809+0000 7f4332ffd700 1 --2- 172.21.15.46:0/2239084384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43341537f0 0x7f4334068910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:26.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.809+0000 7f4332ffd700 1 -- 172.21.15.46:0/2239084384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4324009480 con 0x7f43340ae960 2024-01-26T23:46:26.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.809+0000 7f4332ffd700 1 --2- 172.21.15.46:0/2239084384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43340ae960 0x7f4334158d80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f432400b3c0 tx=0x7f4324016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:26.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.809+0000 7f432affd700 1 -- 172.21.15.46:0/2239084384 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324011b90 con 0x7f43340ae960 2024-01-26T23:46:26.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.809+0000 7f432affd700 1 -- 172.21.15.46:0/2239084384 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4324011cf0 con 0x7f43340ae960 2024-01-26T23:46:26.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.809+0000 7f43395fd700 1 -- 172.21.15.46:0/2239084384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43340c8760 con 0x7f43340ae960 2024-01-26T23:46:26.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.809+0000 7f432affd700 1 -- 172.21.15.46:0/2239084384 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432401b660 con 0x7f43340ae960 2024-01-26T23:46:26.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.809+0000 7f43395fd700 1 -- 172.21.15.46:0/2239084384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4334159360 con 0x7f43340ae960 2024-01-26T23:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.810+0000 7f43395fd700 1 -- 172.21.15.46:0/2239084384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43340cf9e0 con 0x7f43340ae960 2024-01-26T23:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.811+0000 7f432affd700 1 -- 172.21.15.46:0/2239084384 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f432401ba90 con 0x7f43340ae960 2024-01-26T23:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.811+0000 7f432affd700 1 --2- 172.21.15.46:0/2239084384 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4314066ea0 0x7f4314069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.811+0000 7f432affd700 1 -- 172.21.15.46:0/2239084384 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f4324085e10 con 0x7f43340ae960 2024-01-26T23:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.811+0000 7f43327fc700 1 --2- 172.21.15.46:0/2239084384 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4314066ea0 0x7f4314069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.812+0000 7f43327fc700 1 --2- 172.21.15.46:0/2239084384 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4314066ea0 0x7f4314069350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f431c003a80 tx=0x7f431c007ec0 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:26.816+0000 7f432affd700 1 -- 172.21.15.46:0/2239084384 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4324052f60 con 0x7f43340ae960 2024-01-26T23:46:26.821 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-01-26T23:46:27.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.044+0000 7f43395fd700 1 -- 172.21.15.46:0/2239084384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f43340ad650 con 0x7f43340ae960 2024-01-26T23:46:27.192 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:27.190+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:27.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.707+0000 7f432affd700 1 -- 172.21.15.46:0/2239084384 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v106) v1 ==== 97+0+0 (secure 0 0 0) 0x7f432402a080 con 0x7f43340ae960 2024-01-26T23:46:27.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.711+0000 7f43395fd700 1 -- 172.21.15.46:0/2239084384 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4314066ea0 msgr2=0x7f4314069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:27.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.711+0000 7f43395fd700 1 --2- 172.21.15.46:0/2239084384 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4314066ea0 0x7f4314069350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f431c003a80 tx=0x7f431c007ec0 comp rx=0 tx=0).stop 2024-01-26T23:46:27.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.711+0000 7f43395fd700 1 -- 172.21.15.46:0/2239084384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43340ae960 msgr2=0x7f4334158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:27.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.711+0000 7f43395fd700 1 --2- 172.21.15.46:0/2239084384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43340ae960 0x7f4334158d80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f432400b3c0 tx=0x7f4324016000 comp rx=0 tx=0).stop 2024-01-26T23:46:27.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.711+0000 7f43395fd700 1 -- 172.21.15.46:0/2239084384 shutdown_connections 2024-01-26T23:46:27.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.711+0000 7f43395fd700 1 --2- 172.21.15.46:0/2239084384 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4314066ea0 0x7f4314069350 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:27.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.711+0000 7f43395fd700 1 --2- 172.21.15.46:0/2239084384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43340ae960 0x7f4334158d80 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:27.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.711+0000 7f43395fd700 1 --2- 172.21.15.46:0/2239084384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43341537f0 0x7f4334068910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:27.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.711+0000 7f43395fd700 1 --2- 172.21.15.46:0/2239084384 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43340b2b40 0x7f43340b4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:27.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.711+0000 7f43395fd700 1 -- 172.21.15.46:0/2239084384 >> 172.21.15.46:0/2239084384 conn(0x7f4334099a30 msgr2=0x7f4334155530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:27.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.711+0000 7f43395fd700 1 -- 172.21.15.46:0/2239084384 shutdown_connections 2024-01-26T23:46:27.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.711+0000 7f43395fd700 1 -- 172.21.15.46:0/2239084384 wait complete. 2024-01-26T23:46:27.714 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-01-26T23:46:27.730 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-26T23:46:27.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.865+0000 7fa05142c700 1 Processor -- start 2024-01-26T23:46:27.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.865+0000 7fa05142c700 1 -- start start 2024-01-26T23:46:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.866+0000 7fa05142c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa04c1596c0 0x7fa04c06bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.866+0000 7fa05142c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04c06e130 con 0x7fa04c15d460 2024-01-26T23:46:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.866+0000 7fa05142c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04c15d8f0 con 0x7fa04c159a90 2024-01-26T23:46:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.866+0000 7fa05142c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa04c0c98c0 con 0x7fa04c1596c0 2024-01-26T23:46:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.866+0000 7fa04a7fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa04c15d460 0x7fa04c06d310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37680/0 (socket says 172.21.15.46:37680) 2024-01-26T23:46:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.866+0000 7fa04a7fc700 1 -- 172.21.15.46:0/2103122222 learned_addr learned my addr 172.21.15.46:0/2103122222 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.866+0000 7fa04affd700 1 --2- 172.21.15.46:0/2103122222 >> v2:172.21.15.106:3300/0 conn(0x7fa04c1596c0 0x7fa04c06bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.867+0000 7fa049ffb700 1 -- 172.21.15.46:0/2103122222 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 241693588 0 0) 0x7fa04c06e130 con 0x7fa04c15d460 2024-01-26T23:46:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.867+0000 7fa049ffb700 1 -- 172.21.15.46:0/2103122222 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa034003570 con 0x7fa04c15d460 2024-01-26T23:46:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.867+0000 7fa049ffb700 1 -- 172.21.15.46:0/2103122222 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914034104 0 0) 0x7fa04c15d8f0 con 0x7fa04c159a90 2024-01-26T23:46:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.867+0000 7fa049ffb700 1 -- 172.21.15.46:0/2103122222 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa04c06e130 con 0x7fa04c159a90 2024-01-26T23:46:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.867+0000 7fa04affd700 1 -- 172.21.15.46:0/2103122222 >> v1:172.21.15.106:6789/0 conn(0x7fa04c159a90 legacy=0x7fa04c06da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.867+0000 7fa04affd700 1 -- 172.21.15.46:0/2103122222 >> v1:172.21.15.46:6789/0 conn(0x7fa04c15d460 legacy=0x7fa04c06d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.867+0000 7fa04affd700 1 -- 172.21.15.46:0/2103122222 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04c15dbd0 con 0x7fa04c1596c0 2024-01-26T23:46:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.867+0000 7fa04affd700 1 --2- 172.21.15.46:0/2103122222 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa04c1596c0 0x7fa04c06bcd0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa03c0098e0 tx=0x7fa03c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f0d90ba8afbf8f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.867+0000 7fa049ffb700 1 -- 172.21.15.46:0/2103122222 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c011630 con 0x7fa04c1596c0 2024-01-26T23:46:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.867+0000 7fa049ffb700 1 -- 172.21.15.46:0/2103122222 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03c011790 con 0x7fa04c1596c0 2024-01-26T23:46:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.867+0000 7fa049ffb700 1 -- 172.21.15.46:0/2103122222 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c011b30 con 0x7fa04c1596c0 2024-01-26T23:46:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.868+0000 7fa05142c700 1 -- 172.21.15.46:0/2103122222 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa04c1596c0 msgr2=0x7fa04c06bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.868+0000 7fa05142c700 1 --2- 172.21.15.46:0/2103122222 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa04c1596c0 0x7fa04c06bcd0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa03c0098e0 tx=0x7fa03c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.868+0000 7fa05142c700 1 -- 172.21.15.46:0/2103122222 shutdown_connections 2024-01-26T23:46:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.868+0000 7fa05142c700 1 --2- 172.21.15.46:0/2103122222 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa04c1596c0 0x7fa04c06bcd0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:27.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.868+0000 7fa05142c700 1 -- 172.21.15.46:0/2103122222 >> 172.21.15.46:0/2103122222 conn(0x7fa04c0ce530 msgr2=0x7fa04c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:27.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.868+0000 7fa05142c700 1 -- 172.21.15.46:0/2103122222 shutdown_connections 2024-01-26T23:46:27.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.868+0000 7fa05142c700 1 -- 172.21.15.46:0/2103122222 wait complete. 2024-01-26T23:46:27.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.869+0000 7fa05142c700 1 Processor -- start 2024-01-26T23:46:27.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.869+0000 7fa05142c700 1 -- start start 2024-01-26T23:46:27.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.870+0000 7fa05142c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa04c1596c0 0x7fa04c14e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:27.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.870+0000 7fa05142c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa04c159a90 0x7fa04c0af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:27.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.870+0000 7fa05142c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa04c15d460 0x7fa04c0af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:27.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.870+0000 7fa05142c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c0b40c0 con 0x7fa04c1596c0 2024-01-26T23:46:27.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.870+0000 7fa05142c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c0b28d0 con 0x7fa04c15d460 2024-01-26T23:46:27.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.870+0000 7fa05142c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c0b2bb0 con 0x7fa04c159a90 2024-01-26T23:46:27.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.870+0000 7fa04affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa04c1596c0 0x7fa04c14e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:27.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.870+0000 7fa04affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa04c1596c0 0x7fa04c14e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46592/0 (socket says 172.21.15.46:46592) 2024-01-26T23:46:27.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.870+0000 7fa04affd700 1 -- 172.21.15.46:0/1776850492 learned_addr learned my addr 172.21.15.46:0/1776850492 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:27.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.871+0000 7fa04a7fc700 1 --2- 172.21.15.46:0/1776850492 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa04c159a90 0x7fa04c0af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:27.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.871+0000 7fa04b7fe700 1 --2- 172.21.15.46:0/1776850492 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa04c15d460 0x7fa04c0af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:27.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.871+0000 7fa04affd700 1 -- 172.21.15.46:0/1776850492 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa04c159a90 msgr2=0x7fa04c0af1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:27.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.871+0000 7fa04affd700 1 --2- 172.21.15.46:0/1776850492 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa04c159a90 0x7fa04c0af1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:27.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.871+0000 7fa04affd700 1 -- 172.21.15.46:0/1776850492 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa04c15d460 msgr2=0x7fa04c0af6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:27.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.871+0000 7fa04affd700 1 --2- 172.21.15.46:0/1776850492 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa04c15d460 0x7fa04c0af6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:27.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.871+0000 7fa04affd700 1 -- 172.21.15.46:0/1776850492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa03c009420 con 0x7fa04c1596c0 2024-01-26T23:46:27.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.871+0000 7fa04a7fc700 1 --2- 172.21.15.46:0/1776850492 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa04c159a90 0x7fa04c0af1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:46:27.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.871+0000 7fa04affd700 1 --2- 172.21.15.46:0/1776850492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa04c1596c0 0x7fa04c14e9d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa04c06f790 tx=0x7fa03c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:27.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.871+0000 7fa032ffd700 1 -- 172.21.15.46:0/1776850492 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c01f070 con 0x7fa04c1596c0 2024-01-26T23:46:27.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.871+0000 7fa032ffd700 1 -- 172.21.15.46:0/1776850492 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03c000bc0 con 0x7fa04c1596c0 2024-01-26T23:46:27.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.871+0000 7fa05142c700 1 -- 172.21.15.46:0/1776850492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04c0b2e90 con 0x7fa04c1596c0 2024-01-26T23:46:27.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.871+0000 7fa032ffd700 1 -- 172.21.15.46:0/1776850492 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c025420 con 0x7fa04c1596c0 2024-01-26T23:46:27.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.871+0000 7fa05142c700 1 -- 172.21.15.46:0/1776850492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04c14ef10 con 0x7fa04c1596c0 2024-01-26T23:46:27.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.873+0000 7fa05142c700 1 -- 172.21.15.46:0/1776850492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa04c0b34a0 con 0x7fa04c1596c0 2024-01-26T23:46:27.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.873+0000 7fa032ffd700 1 -- 172.21.15.46:0/1776850492 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fa03c023070 con 0x7fa04c1596c0 2024-01-26T23:46:27.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.873+0000 7fa032ffd700 1 --2- 172.21.15.46:0/1776850492 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa038066ea0 0x7fa038069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:27.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.873+0000 7fa032ffd700 1 -- 172.21.15.46:0/1776850492 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5565+0+0 (secure 0 0 0) 0x7fa03c085680 con 0x7fa04c1596c0 2024-01-26T23:46:27.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.873+0000 7fa04a7fc700 1 --2- 172.21.15.46:0/1776850492 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa038066ea0 0x7fa038069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:27.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.874+0000 7fa04a7fc700 1 --2- 172.21.15.46:0/1776850492 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa038066ea0 0x7fa038069350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0c9690 tx=0x7fa040009040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:27.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:27.878+0000 7fa032ffd700 1 -- 172.21.15.46:0/1776850492 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa03c052a60 con 0x7fa04c1596c0 2024-01-26T23:46:28.002 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:28.001+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:28.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.172+0000 7fa05142c700 1 -- 172.21.15.46:0/1776850492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa04c04c710 con 0x7fa04c1596c0 2024-01-26T23:46:28.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.173+0000 7fa032ffd700 1 -- 172.21.15.46:0/1776850492 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fa03c02a090 con 0x7fa04c1596c0 2024-01-26T23:46:28.174 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:28.174 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:46:28.174 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-01-26T23:46:28.174 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.174 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-01-26T23:46:28.175 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlM6dKKBAAPpeWeDZWrFgc/4zuxKCvHw==", 2024-01-26T23:46:28.175 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.175 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:46:28.175 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:46:28.175 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:46:28.175 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:46:28.175 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.175 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.175 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.175 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-01-26T23:46:28.175 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlmzHRNRAAflWYZ/p4AXeMy54tvLpugA==", 2024-01-26T23:46:28.175 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.175 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:46:28.175 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:46:28.176 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:46:28.176 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:46:28.176 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.176 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.176 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.176 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-01-26T23:46:28.176 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlMZxoMBAAG5kyfm+LLG86y/p2A8BfDQ==", 2024-01-26T23:46:28.176 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.176 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:46:28.176 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:46:28.176 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:46:28.176 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:46:28.176 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.176 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.177 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.177 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-01-26T23:46:28.177 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl5jx/ARAA3cwtLAOWQ/n2NG+xwEZ3QQ==", 2024-01-26T23:46:28.177 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.177 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:46:28.177 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:46:28.177 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:46:28.177 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:46:28.177 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.177 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.177 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.177 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-01-26T23:46:28.177 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4QLRlJ2OKJRAAWVhxEdQkUW20294w/KLL6g==", 2024-01-26T23:46:28.177 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.178 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:46:28.178 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:46:28.178 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:46:28.178 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.178 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.178 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.178 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-01-26T23:46:28.178 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7QLRlE2tAJRAAzNlTiCr6a2eWiabRy1SqSw==", 2024-01-26T23:46:28.178 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.178 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:46:28.178 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:46:28.178 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:46:28.178 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.178 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.178 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.179 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-01-26T23:46:28.179 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+QLRlyySaJBAA5072ba5HUNi2dYE3D02M/Q==", 2024-01-26T23:46:28.179 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.179 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:46:28.179 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:46:28.179 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:46:28.179 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.179 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.179 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.179 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-01-26T23:46:28.179 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBQLRl9+tAIxAApZk0btXIpQPWSb3BYUYsFQ==", 2024-01-26T23:46:28.179 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.179 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:46:28.179 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:46:28.180 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:46:28.180 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.180 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.180 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.180 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-01-26T23:46:28.180 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHQLRlurUtJxAAT9fwKrAWl62VOXwShg7NJg==", 2024-01-26T23:46:28.180 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.180 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:46:28.180 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:46:28.180 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:46:28.180 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.180 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.180 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.180 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-01-26T23:46:28.181 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKQLRl7aqoJRAAXu9fykWELsW3SwnahoIOLA==", 2024-01-26T23:46:28.181 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.181 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:46:28.181 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:46:28.181 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:46:28.181 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.181 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.181 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.181 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-01-26T23:46:28.181 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNQLRlWf7pJBAAopmFLIKb8s32YZs46VwbGg==", 2024-01-26T23:46:28.181 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.181 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:46:28.181 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:46:28.181 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:46:28.181 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.181 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.182 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.182 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-01-26T23:46:28.182 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQQLRlNGfhIxAAmCgmOB3QAGfSIYHBb5BUZw==", 2024-01-26T23:46:28.182 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.182 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:46:28.182 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:46:28.182 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:46:28.182 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.182 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.182 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.182 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-01-26T23:46:28.182 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRleO8wBhAA1018AcO3KcyOZxgB0Itxaw==", 2024-01-26T23:46:28.182 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.182 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:46:28.182 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-01-26T23:46:28.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-26T23:46:28.183 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.183 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.183 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.183 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-01-26T23:46:28.183 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlsiZEDRAAjXSamHe2bZcDqVsqXqIcGg==", 2024-01-26T23:46:28.183 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.183 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:46:28.183 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:46:28.183 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:46:28.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:46:28.183 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.183 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.183 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.183 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-01-26T23:46:28.184 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl6YT1CRAApXlN4j20xzwda0tDMy94Eg==", 2024-01-26T23:46:28.184 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.184 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:46:28.184 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:46:28.184 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:46:28.184 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:46:28.184 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.184 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.184 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.184 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-01-26T23:46:28.184 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlHvDQEhAAxMhgWJsZC5/6Af5zFV6XIA==", 2024-01-26T23:46:28.184 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.184 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:46:28.184 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:46:28.185 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:46:28.185 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:46:28.185 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.185 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.185 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.185 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-01-26T23:46:28.185 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlvMwDCBAAXnFyMduz7PfWdLWbjytIiQ==", 2024-01-26T23:46:28.185 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.185 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:46:28.185 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:46:28.185 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:46:28.185 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:46:28.185 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.185 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.185 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.186 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-01-26T23:46:28.186 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlhMNSJBAA38QU6DrCvFjO+u9zQ8bkEA==", 2024-01-26T23:46:28.186 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.186 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:46:28.186 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-26T23:46:28.186 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:46:28.186 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.186 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:28.186 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:28.186 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-01-26T23:46:28.186 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlI9EbIBAACTpzAO+8A/VFJZcT/WWBng==", 2024-01-26T23:46:28.186 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:46:28.186 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:46:28.186 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-26T23:46:28.187 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:46:28.187 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.187 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:28.187 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-01-26T23:46:28.187 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:46:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.175+0000 7fa05142c700 1 -- 172.21.15.46:0/1776850492 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa038066ea0 msgr2=0x7fa038069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.175+0000 7fa05142c700 1 --2- 172.21.15.46:0/1776850492 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa038066ea0 0x7fa038069350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0c9690 tx=0x7fa040009040 comp rx=0 tx=0).stop 2024-01-26T23:46:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.175+0000 7fa05142c700 1 -- 172.21.15.46:0/1776850492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa04c1596c0 msgr2=0x7fa04c14e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.175+0000 7fa05142c700 1 --2- 172.21.15.46:0/1776850492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa04c1596c0 0x7fa04c14e9d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa04c06f790 tx=0x7fa03c017000 comp rx=0 tx=0).stop 2024-01-26T23:46:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.175+0000 7fa05142c700 1 -- 172.21.15.46:0/1776850492 shutdown_connections 2024-01-26T23:46:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.175+0000 7fa05142c700 1 --2- 172.21.15.46:0/1776850492 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa038066ea0 0x7fa038069350 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.175+0000 7fa05142c700 1 --2- 172.21.15.46:0/1776850492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa04c1596c0 0x7fa04c14e9d0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.175+0000 7fa05142c700 1 --2- 172.21.15.46:0/1776850492 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa04c15d460 0x7fa04c0af6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.175+0000 7fa05142c700 1 --2- 172.21.15.46:0/1776850492 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa04c159a90 0x7fa04c0af1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.175+0000 7fa05142c700 1 -- 172.21.15.46:0/1776850492 >> 172.21.15.46:0/1776850492 conn(0x7fa04c0ce530 msgr2=0x7fa04c1587a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.175+0000 7fa05142c700 1 -- 172.21.15.46:0/1776850492 shutdown_connections 2024-01-26T23:46:28.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.175+0000 7fa05142c700 1 -- 172.21.15.46:0/1776850492 wait complete. 2024-01-26T23:46:28.190 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:46:28.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.322+0000 7f19e359f700 1 Processor -- start 2024-01-26T23:46:28.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.322+0000 7f19e359f700 1 -- start start 2024-01-26T23:46:28.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.323+0000 7f19e359f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f19dc15e010 0x7f19dc067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:28.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.323+0000 7f19e359f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc06fb60 con 0x7f19dc15dc40 2024-01-26T23:46:28.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.323+0000 7f19e359f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc06ef50 con 0x7f19dc15f490 2024-01-26T23:46:28.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.323+0000 7f19e359f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19dc06f230 con 0x7f19dc15e010 2024-01-26T23:46:28.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.323+0000 7f19e133b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f19dc15dc40 0x7f19dc0b2b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37684/0 (socket says 172.21.15.46:37684) 2024-01-26T23:46:28.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.323+0000 7f19e133b700 1 -- 172.21.15.46:0/4106925034 learned_addr learned my addr 172.21.15.46:0/4106925034 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:28.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.323+0000 7f19e0b3a700 1 --2- 172.21.15.46:0/4106925034 >> v2:172.21.15.106:3300/0 conn(0x7f19dc15e010 0x7f19dc067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:28.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.324+0000 7f19d3fff700 1 -- 172.21.15.46:0/4106925034 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489310135 0 0) 0x7f19dc06fb60 con 0x7f19dc15dc40 2024-01-26T23:46:28.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.324+0000 7f19d3fff700 1 -- 172.21.15.46:0/4106925034 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c8003570 con 0x7f19dc15dc40 2024-01-26T23:46:28.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.324+0000 7f19d3fff700 1 -- 172.21.15.46:0/4106925034 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387210591 0 0) 0x7f19dc06ef50 con 0x7f19dc15f490 2024-01-26T23:46:28.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.324+0000 7f19d3fff700 1 -- 172.21.15.46:0/4106925034 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19dc06fb60 con 0x7f19dc15f490 2024-01-26T23:46:28.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.324+0000 7f19e0b3a700 1 -- 172.21.15.46:0/4106925034 >> v1:172.21.15.106:6789/0 conn(0x7f19dc15f490 legacy=0x7f19dc167950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:28.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.324+0000 7f19e0b3a700 1 -- 172.21.15.46:0/4106925034 >> v1:172.21.15.46:6789/0 conn(0x7f19dc15dc40 legacy=0x7f19dc0b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:28.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.324+0000 7f19e0b3a700 1 -- 172.21.15.46:0/4106925034 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc06f510 con 0x7f19dc15e010 2024-01-26T23:46:28.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.324+0000 7f19e0b3a700 1 --2- 172.21.15.46:0/4106925034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19dc15e010 0x7f19dc067930 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0098e0 tx=0x7f19cc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ff39f1ae6d7313b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:28.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.324+0000 7f19d3fff700 1 -- 172.21.15.46:0/4106925034 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc011630 con 0x7f19dc15e010 2024-01-26T23:46:28.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.324+0000 7f19d3fff700 1 -- 172.21.15.46:0/4106925034 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc011790 con 0x7f19dc15e010 2024-01-26T23:46:28.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.324+0000 7f19d3fff700 1 -- 172.21.15.46:0/4106925034 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc011b30 con 0x7f19dc15e010 2024-01-26T23:46:28.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.325+0000 7f19e359f700 1 -- 172.21.15.46:0/4106925034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19dc15e010 msgr2=0x7f19dc067930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:28.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.325+0000 7f19e359f700 1 --2- 172.21.15.46:0/4106925034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19dc15e010 0x7f19dc067930 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0098e0 tx=0x7f19cc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.325+0000 7f19e359f700 1 -- 172.21.15.46:0/4106925034 shutdown_connections 2024-01-26T23:46:28.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.325+0000 7f19e359f700 1 --2- 172.21.15.46:0/4106925034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19dc15e010 0x7f19dc067930 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.325+0000 7f19e359f700 1 -- 172.21.15.46:0/4106925034 >> 172.21.15.46:0/4106925034 conn(0x7f19dc069510 msgr2=0x7f19dc0aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:28.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.325+0000 7f19e359f700 1 -- 172.21.15.46:0/4106925034 shutdown_connections 2024-01-26T23:46:28.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.325+0000 7f19e359f700 1 -- 172.21.15.46:0/4106925034 wait complete. 2024-01-26T23:46:28.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.326+0000 7f19e359f700 1 Processor -- start 2024-01-26T23:46:28.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.327+0000 7f19e359f700 1 -- start start 2024-01-26T23:46:28.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.327+0000 7f19e359f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19dc15dc40 0x7f19dc152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:28.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.327+0000 7f19e359f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19dc15e010 0x7f19dc151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:28.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.327+0000 7f19e359f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19dc15f490 0x7f19dc157610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:28.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.327+0000 7f19e359f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc14fcd0 con 0x7f19dc15e010 2024-01-26T23:46:28.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.327+0000 7f19e359f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc153070 con 0x7f19dc15f490 2024-01-26T23:46:28.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.327+0000 7f19e359f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc15d420 con 0x7f19dc15dc40 2024-01-26T23:46:28.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.328+0000 7f19e0b3a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19dc15e010 0x7f19dc151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:28.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.328+0000 7f19e0b3a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19dc15e010 0x7f19dc151340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46606/0 (socket says 172.21.15.46:46606) 2024-01-26T23:46:28.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.328+0000 7f19e0b3a700 1 -- 172.21.15.46:0/2603475942 learned_addr learned my addr 172.21.15.46:0/2603475942 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:28.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.328+0000 7f19e133b700 1 --2- 172.21.15.46:0/2603475942 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19dc15dc40 0x7f19dc152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:28.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.328+0000 7f19e1b3c700 1 --2- 172.21.15.46:0/2603475942 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19dc15f490 0x7f19dc157610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:28.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.328+0000 7f19e0b3a700 1 -- 172.21.15.46:0/2603475942 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19dc15dc40 msgr2=0x7f19dc152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:28.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.328+0000 7f19e0b3a700 1 --2- 172.21.15.46:0/2603475942 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19dc15dc40 0x7f19dc152b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.328+0000 7f19e0b3a700 1 -- 172.21.15.46:0/2603475942 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19dc15f490 msgr2=0x7f19dc157610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:28.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.328+0000 7f19e0b3a700 1 --2- 172.21.15.46:0/2603475942 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19dc15f490 0x7f19dc157610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.328+0000 7f19e0b3a700 1 -- 172.21.15.46:0/2603475942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc009420 con 0x7f19dc15e010 2024-01-26T23:46:28.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.328+0000 7f19e0b3a700 1 --2- 172.21.15.46:0/2603475942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19dc15e010 0x7f19dc151340 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f19cc009ed0 tx=0x7f19cc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:28.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.328+0000 7f19d17fa700 1 -- 172.21.15.46:0/2603475942 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc01f070 con 0x7f19dc15e010 2024-01-26T23:46:28.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.328+0000 7f19d17fa700 1 -- 172.21.15.46:0/2603475942 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc000bc0 con 0x7f19dc15e010 2024-01-26T23:46:28.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.328+0000 7f19e359f700 1 -- 172.21.15.46:0/2603475942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19dc156140 con 0x7f19dc15e010 2024-01-26T23:46:28.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.328+0000 7f19d17fa700 1 -- 172.21.15.46:0/2603475942 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc025420 con 0x7f19dc15e010 2024-01-26T23:46:28.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.328+0000 7f19e359f700 1 -- 172.21.15.46:0/2603475942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19dc151880 con 0x7f19dc15e010 2024-01-26T23:46:28.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.329+0000 7f19e359f700 1 -- 172.21.15.46:0/2603475942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19dc14fe40 con 0x7f19dc15e010 2024-01-26T23:46:28.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.331+0000 7f19d17fa700 1 -- 172.21.15.46:0/2603475942 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f19cc023070 con 0x7f19dc15e010 2024-01-26T23:46:28.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.331+0000 7f19d17fa700 1 --2- 172.21.15.46:0/2603475942 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f19c0066ea0 0x7f19c0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:28.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.331+0000 7f19d17fa700 1 -- 172.21.15.46:0/2603475942 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5565+0+0 (secure 0 0 0) 0x7f19cc085740 con 0x7f19dc15e010 2024-01-26T23:46:28.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.331+0000 7f19e133b700 1 --2- 172.21.15.46:0/2603475942 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f19c0066ea0 0x7f19c0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:28.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.332+0000 7f19e133b700 1 --2- 172.21.15.46:0/2603475942 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f19c0066ea0 0x7f19c0069350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f19d8007fd0 tx=0x7f19d8009040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:28.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.334+0000 7f19d17fa700 1 -- 172.21.15.46:0/2603475942 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f19cc052ba0 con 0x7f19dc15e010 2024-01-26T23:46:28.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.542+0000 7f19e359f700 1 -- 172.21.15.46:0/2603475942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19dc14fe10 con 0x7f19dc15e010 2024-01-26T23:46:28.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.543+0000 7f19d17fa700 1 -- 172.21.15.46:0/2603475942 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+12935 (secure 0 0 0) 0x7f19cc02a020 con 0x7f19dc15e010 2024-01-26T23:46:28.545 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:28.545 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":108,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:46:27.713856+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:46:28.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.546+0000 7f19e359f700 1 -- 172.21.15.46:0/2603475942 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f19c0066ea0 msgr2=0x7f19c0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:28.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.546+0000 7f19e359f700 1 --2- 172.21.15.46:0/2603475942 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f19c0066ea0 0x7f19c0069350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f19d8007fd0 tx=0x7f19d8009040 comp rx=0 tx=0).stop 2024-01-26T23:46:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.546+0000 7f19e359f700 1 -- 172.21.15.46:0/2603475942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19dc15e010 msgr2=0x7f19dc151340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.546+0000 7f19e359f700 1 --2- 172.21.15.46:0/2603475942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19dc15e010 0x7f19dc151340 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f19cc009ed0 tx=0x7f19cc017000 comp rx=0 tx=0).stop 2024-01-26T23:46:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.546+0000 7f19e359f700 1 -- 172.21.15.46:0/2603475942 shutdown_connections 2024-01-26T23:46:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.546+0000 7f19e359f700 1 --2- 172.21.15.46:0/2603475942 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f19c0066ea0 0x7f19c0069350 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.546+0000 7f19e359f700 1 --2- 172.21.15.46:0/2603475942 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19dc15f490 0x7f19dc157610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.546+0000 7f19e359f700 1 --2- 172.21.15.46:0/2603475942 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19dc15dc40 0x7f19dc152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.546+0000 7f19e359f700 1 --2- 172.21.15.46:0/2603475942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19dc15e010 0x7f19dc151340 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.546+0000 7f19e359f700 1 -- 172.21.15.46:0/2603475942 >> 172.21.15.46:0/2603475942 conn(0x7f19dc069510 msgr2=0x7f19dc0ad440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.546+0000 7f19e359f700 1 -- 172.21.15.46:0/2603475942 shutdown_connections 2024-01-26T23:46:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.546+0000 7f19e359f700 1 -- 172.21.15.46:0/2603475942 wait complete. 2024-01-26T23:46:28.561 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-26T23:46:28.561 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:46:28.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.699+0000 7f2a2518d700 1 Processor -- start 2024-01-26T23:46:28.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.699+0000 7f2a2518d700 1 -- start start 2024-01-26T23:46:28.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.701+0000 7f2a2518d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2a200c93a0 0x7f2a200c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:28.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.701+0000 7f2a2518d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a2015e390 con 0x7f2a200681f0 2024-01-26T23:46:28.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.701+0000 7f2a2518d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a2015d810 con 0x7f2a20067d50 2024-01-26T23:46:28.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.701+0000 7f2a2518d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a2015daf0 con 0x7f2a200c93a0 2024-01-26T23:46:28.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.701+0000 7f2a1f59e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2a200681f0 0x7f2a200c7e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37690/0 (socket says 172.21.15.46:37690) 2024-01-26T23:46:28.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.701+0000 7f2a1f59e700 1 -- 172.21.15.46:0/3407682963 learned_addr learned my addr 172.21.15.46:0/3407682963 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:28.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.701+0000 7f2a1ed9d700 1 --2- 172.21.15.46:0/3407682963 >> v2:172.21.15.106:3300/0 conn(0x7f2a200c93a0 0x7f2a200c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:28.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.701+0000 7f2a1ed9d700 1 -- 172.21.15.46:0/3407682963 >> v1:172.21.15.106:6789/0 conn(0x7f2a20067d50 legacy=0x7f2a20159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:28.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.701+0000 7f2a1ed9d700 1 -- 172.21.15.46:0/3407682963 >> v1:172.21.15.46:6789/0 conn(0x7f2a200681f0 legacy=0x7f2a200c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:28.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.701+0000 7f2a1ed9d700 1 -- 172.21.15.46:0/3407682963 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a2015ddd0 con 0x7f2a200c93a0 2024-01-26T23:46:28.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.702+0000 7f2a1ed9d700 1 --2- 172.21.15.46:0/3407682963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a200c93a0 0x7f2a200c9770 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2a100098e0 tx=0x7f2a100091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8cef8cb15273a8f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:28.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.704+0000 7f2a1dd9b700 1 -- 172.21.15.46:0/3407682963 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a10009e80 con 0x7f2a200c93a0 2024-01-26T23:46:28.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.704+0000 7f2a1dd9b700 1 -- 172.21.15.46:0/3407682963 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1000b860 con 0x7f2a200c93a0 2024-01-26T23:46:28.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.704+0000 7f2a1dd9b700 1 -- 172.21.15.46:0/3407682963 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1000bbe0 con 0x7f2a200c93a0 2024-01-26T23:46:28.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.704+0000 7f2a2518d700 1 -- 172.21.15.46:0/3407682963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a200c93a0 msgr2=0x7f2a200c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:28.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.704+0000 7f2a2518d700 1 --2- 172.21.15.46:0/3407682963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a200c93a0 0x7f2a200c9770 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2a100098e0 tx=0x7f2a100091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.705+0000 7f2a2518d700 1 -- 172.21.15.46:0/3407682963 shutdown_connections 2024-01-26T23:46:28.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.705+0000 7f2a2518d700 1 --2- 172.21.15.46:0/3407682963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a200c93a0 0x7f2a200c9770 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.705+0000 7f2a2518d700 1 -- 172.21.15.46:0/3407682963 >> 172.21.15.46:0/3407682963 conn(0x7f2a200ca9a0 msgr2=0x7f2a20068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:28.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.705+0000 7f2a2518d700 1 -- 172.21.15.46:0/3407682963 shutdown_connections 2024-01-26T23:46:28.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.705+0000 7f2a2518d700 1 -- 172.21.15.46:0/3407682963 wait complete. 2024-01-26T23:46:28.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.706+0000 7f2a2518d700 1 Processor -- start 2024-01-26T23:46:28.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.706+0000 7f2a2518d700 1 -- start start 2024-01-26T23:46:28.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.707+0000 7f2a2518d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a20067d50 0x7f2a20144e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:28.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.707+0000 7f2a2518d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a200681f0 0x7f2a20157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:28.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.707+0000 7f2a2518d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a200c93a0 0x7f2a20157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:28.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.707+0000 7f2a2518d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a201558d0 con 0x7f2a20067d50 2024-01-26T23:46:28.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.707+0000 7f2a2518d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a20155010 con 0x7f2a200681f0 2024-01-26T23:46:28.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.707+0000 7f2a2518d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a20155280 con 0x7f2a200c93a0 2024-01-26T23:46:28.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.707+0000 7f2a1ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a20067d50 0x7f2a20144e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:28.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.707+0000 7f2a1ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a20067d50 0x7f2a20144e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46616/0 (socket says 172.21.15.46:46616) 2024-01-26T23:46:28.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.707+0000 7f2a1ed9d700 1 -- 172.21.15.46:0/276760456 learned_addr learned my addr 172.21.15.46:0/276760456 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:28.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.707+0000 7f2a1f59e700 1 --2- 172.21.15.46:0/276760456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a200c93a0 0x7f2a20157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:28.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.707+0000 7f2a1e59c700 1 --2- 172.21.15.46:0/276760456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a200681f0 0x7f2a20157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:28.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.708+0000 7f2a1ed9d700 1 -- 172.21.15.46:0/276760456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a200c93a0 msgr2=0x7f2a20157e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:28.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.708+0000 7f2a1ed9d700 1 --2- 172.21.15.46:0/276760456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a200c93a0 0x7f2a20157e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.708+0000 7f2a1ed9d700 1 -- 172.21.15.46:0/276760456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a200681f0 msgr2=0x7f2a20157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:28.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.708+0000 7f2a1ed9d700 1 --2- 172.21.15.46:0/276760456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a200681f0 0x7f2a20157900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.708+0000 7f2a1ed9d700 1 -- 172.21.15.46:0/276760456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a10009420 con 0x7f2a20067d50 2024-01-26T23:46:28.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.708+0000 7f2a1ed9d700 1 --2- 172.21.15.46:0/276760456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a20067d50 0x7f2a20144e70 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f2a1000b3c0 tx=0x7f2a10017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:28.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.708+0000 7f2a06ffd700 1 -- 172.21.15.46:0/276760456 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1000bbe0 con 0x7f2a20067d50 2024-01-26T23:46:28.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.708+0000 7f2a06ffd700 1 -- 172.21.15.46:0/276760456 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a10011420 con 0x7f2a20067d50 2024-01-26T23:46:28.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.708+0000 7f2a2518d700 1 -- 172.21.15.46:0/276760456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a20155560 con 0x7f2a20067d50 2024-01-26T23:46:28.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.708+0000 7f2a06ffd700 1 -- 172.21.15.46:0/276760456 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a100117e0 con 0x7f2a20067d50 2024-01-26T23:46:28.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.708+0000 7f2a2518d700 1 -- 172.21.15.46:0/276760456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a200c7090 con 0x7f2a20067d50 2024-01-26T23:46:28.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.709+0000 7f2a2518d700 1 -- 172.21.15.46:0/276760456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2004c710 con 0x7f2a20067d50 2024-01-26T23:46:28.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.710+0000 7f2a06ffd700 1 -- 172.21.15.46:0/276760456 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f2a10011bd0 con 0x7f2a20067d50 2024-01-26T23:46:28.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.711+0000 7f2a06ffd700 1 --2- 172.21.15.46:0/276760456 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a0c066ea0 0x7f2a0c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:28.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.711+0000 7f2a06ffd700 1 -- 172.21.15.46:0/276760456 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5565+0+0 (secure 0 0 0) 0x7f2a10085cd0 con 0x7f2a20067d50 2024-01-26T23:46:28.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.711+0000 7f2a1e59c700 1 --2- 172.21.15.46:0/276760456 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a0c066ea0 0x7f2a0c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:28.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.711+0000 7f2a1e59c700 1 --2- 172.21.15.46:0/276760456 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a0c066ea0 0x7f2a0c069350 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2a20154310 tx=0x7f2a14007ec0 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:28.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.715+0000 7f2a06ffd700 1 -- 172.21.15.46:0/276760456 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2a10053010 con 0x7f2a20067d50 2024-01-26T23:46:28.799 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:28.798+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:28.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.919+0000 7f2a2518d700 1 -- 172.21.15.46:0/276760456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2a2015e0b0 con 0x7f2a20067d50 2024-01-26T23:46:28.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.920+0000 7f2a06ffd700 1 -- 172.21.15.46:0/276760456 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v108) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2a10056630 con 0x7f2a20067d50 2024-01-26T23:46:28.921 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:46:28.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.922+0000 7f2a2518d700 1 -- 172.21.15.46:0/276760456 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a0c066ea0 msgr2=0x7f2a0c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.922+0000 7f2a2518d700 1 --2- 172.21.15.46:0/276760456 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a0c066ea0 0x7f2a0c069350 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2a20154310 tx=0x7f2a14007ec0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.922+0000 7f2a2518d700 1 -- 172.21.15.46:0/276760456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a20067d50 msgr2=0x7f2a20144e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.922+0000 7f2a2518d700 1 --2- 172.21.15.46:0/276760456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a20067d50 0x7f2a20144e70 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f2a1000b3c0 tx=0x7f2a10017000 comp rx=0 tx=0).stop 2024-01-26T23:46:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.922+0000 7f2a2518d700 1 -- 172.21.15.46:0/276760456 shutdown_connections 2024-01-26T23:46:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.922+0000 7f2a2518d700 1 --2- 172.21.15.46:0/276760456 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a0c066ea0 0x7f2a0c069350 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.922+0000 7f2a2518d700 1 --2- 172.21.15.46:0/276760456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a20067d50 0x7f2a20144e70 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.922+0000 7f2a2518d700 1 --2- 172.21.15.46:0/276760456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a200c93a0 0x7f2a20157e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.922+0000 7f2a2518d700 1 --2- 172.21.15.46:0/276760456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a200681f0 0x7f2a20157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.922+0000 7f2a2518d700 1 -- 172.21.15.46:0/276760456 >> 172.21.15.46:0/276760456 conn(0x7f2a200ca9a0 msgr2=0x7f2a200688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.922+0000 7f2a2518d700 1 -- 172.21.15.46:0/276760456 shutdown_connections 2024-01-26T23:46:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:28.922+0000 7f2a2518d700 1 -- 172.21.15.46:0/276760456 wait complete. 2024-01-26T23:46:28.938 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-26T23:46:29.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.070+0000 7f47c56bf700 1 Processor -- start 2024-01-26T23:46:29.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.070+0000 7f47c56bf700 1 -- start start 2024-01-26T23:46:29.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.071+0000 7f47c56bf700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f47c0068040 0x7f47c00cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:29.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.071+0000 7f47c56bf700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f47c0159d30 0x7f47c00ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:29.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.071+0000 7f47c56bf700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c006d760 con 0x7f47c0159960 2024-01-26T23:46:29.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.071+0000 7f47c56bf700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c006f820 con 0x7f47c0068040 2024-01-26T23:46:29.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.071+0000 7f47c56bf700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c00c9f00 con 0x7f47c0159d30 2024-01-26T23:46:29.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.071+0000 7f47be7fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f47c0068040 0x7f47c00cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:29.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.071+0000 7f47beffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f47c0159960 0x7f47c0067930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37692/0 (socket says 172.21.15.46:37692) 2024-01-26T23:46:29.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.071+0000 7f47be7fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f47c0068040 0x7f47c00cbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46626/0 (socket says 172.21.15.46:46626) 2024-01-26T23:46:29.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.071+0000 7f47beffd700 1 -- 172.21.15.46:0/3718700890 learned_addr learned my addr 172.21.15.46:0/3718700890 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:29.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.071+0000 7f47bf7fe700 1 --2- 172.21.15.46:0/3718700890 >> v2:172.21.15.106:3300/0 conn(0x7f47c0159d30 0x7f47c00ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:29.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.071+0000 7f47be7fc700 1 -- 172.21.15.46:0/3718700890 >> v2:172.21.15.106:3300/0 conn(0x7f47c0159d30 msgr2=0x7f47c00ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:29.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.071+0000 7f47be7fc700 1 --2- 172.21.15.46:0/3718700890 >> v2:172.21.15.106:3300/0 conn(0x7f47c0159d30 0x7f47c00ce1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.071+0000 7f47be7fc700 1 -- 172.21.15.46:0/3718700890 >> v1:172.21.15.46:6789/0 conn(0x7f47c0159960 legacy=0x7f47c0067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:29.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.071+0000 7f47bdffb700 1 -- 172.21.15.46:0/3718700890 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398194705 0 0) 0x7f47c006d760 con 0x7f47c0159960 2024-01-26T23:46:29.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.071+0000 7f47be7fc700 1 -- 172.21.15.46:0/3718700890 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c006fcb0 con 0x7f47c0068040 2024-01-26T23:46:29.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.071+0000 7f47be7fc700 1 --2- 172.21.15.46:0/3718700890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c0068040 0x7f47c00cbc30 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f47c015e7e0 tx=0x7f47b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a4b18be876005c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:29.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.072+0000 7f47bdffb700 1 -- 172.21.15.46:0/3718700890 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b4011630 con 0x7f47c0068040 2024-01-26T23:46:29.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.072+0000 7f47bdffb700 1 -- 172.21.15.46:0/3718700890 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b4011790 con 0x7f47c0068040 2024-01-26T23:46:29.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.072+0000 7f47bdffb700 1 -- 172.21.15.46:0/3718700890 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b4011b30 con 0x7f47c0068040 2024-01-26T23:46:29.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.072+0000 7f47c56bf700 1 -- 172.21.15.46:0/3718700890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c0068040 msgr2=0x7f47c00cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:29.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.072+0000 7f47c56bf700 1 --2- 172.21.15.46:0/3718700890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c0068040 0x7f47c00cbc30 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f47c015e7e0 tx=0x7f47b40091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.072+0000 7f47c56bf700 1 -- 172.21.15.46:0/3718700890 shutdown_connections 2024-01-26T23:46:29.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.072+0000 7f47c56bf700 1 --2- 172.21.15.46:0/3718700890 >> v2:172.21.15.106:3300/0 conn(0x7f47c0159d30 0x7f47c00ce1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.072+0000 7f47c56bf700 1 --2- 172.21.15.46:0/3718700890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c0068040 0x7f47c00cbc30 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.072+0000 7f47c56bf700 1 -- 172.21.15.46:0/3718700890 >> 172.21.15.46:0/3718700890 conn(0x7f47c006b850 msgr2=0x7f47c00cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:29.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.072+0000 7f47c56bf700 1 -- 172.21.15.46:0/3718700890 shutdown_connections 2024-01-26T23:46:29.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.072+0000 7f47c56bf700 1 -- 172.21.15.46:0/3718700890 wait complete. 2024-01-26T23:46:29.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.074+0000 7f47c56bf700 1 Processor -- start 2024-01-26T23:46:29.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.074+0000 7f47c56bf700 1 -- start start 2024-01-26T23:46:29.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47c56bf700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47c0068040 0x7f47c016d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:29.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47c56bf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c0159960 0x7f47c015db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:29.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47c56bf700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47c0159d30 0x7f47c0155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:29.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47c56bf700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c015e0d0 con 0x7f47c0159960 2024-01-26T23:46:29.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47c56bf700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c016a050 con 0x7f47c0068040 2024-01-26T23:46:29.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47c56bf700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c016e5f0 con 0x7f47c0159d30 2024-01-26T23:46:29.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47be7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c0159960 0x7f47c015db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:29.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47be7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c0159960 0x7f47c015db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46640/0 (socket says 172.21.15.46:46640) 2024-01-26T23:46:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47be7fc700 1 -- 172.21.15.46:0/2245670168 learned_addr learned my addr 172.21.15.46:0/2245670168 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47bf7fe700 1 --2- 172.21.15.46:0/2245670168 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47c0159d30 0x7f47c0155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47beffd700 1 --2- 172.21.15.46:0/2245670168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47c0068040 0x7f47c016d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47be7fc700 1 -- 172.21.15.46:0/2245670168 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47c0159d30 msgr2=0x7f47c0155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47be7fc700 1 --2- 172.21.15.46:0/2245670168 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47c0159d30 0x7f47c0155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47be7fc700 1 -- 172.21.15.46:0/2245670168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47c0068040 msgr2=0x7f47c016d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47be7fc700 1 --2- 172.21.15.46:0/2245670168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47c0068040 0x7f47c016d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47be7fc700 1 -- 172.21.15.46:0/2245670168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b4009420 con 0x7f47c0159960 2024-01-26T23:46:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47bf7fe700 1 --2- 172.21.15.46:0/2245670168 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47c0159d30 0x7f47c0155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47be7fc700 1 --2- 172.21.15.46:0/2245670168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c0159960 0x7f47c015db90 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f47b4013000 tx=0x7f47b4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.075+0000 7f47a6ffd700 1 -- 172.21.15.46:0/2245670168 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b4000d10 con 0x7f47c0159960 2024-01-26T23:46:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.076+0000 7f47a6ffd700 1 -- 172.21.15.46:0/2245670168 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b4011b10 con 0x7f47c0159960 2024-01-26T23:46:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.076+0000 7f47a6ffd700 1 -- 172.21.15.46:0/2245670168 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b4025640 con 0x7f47c0159960 2024-01-26T23:46:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.076+0000 7f47c56bf700 1 -- 172.21.15.46:0/2245670168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c016dd50 con 0x7f47c0159960 2024-01-26T23:46:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.076+0000 7f47c56bf700 1 -- 172.21.15.46:0/2245670168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c016e310 con 0x7f47c0159960 2024-01-26T23:46:29.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.077+0000 7f47c56bf700 1 -- 172.21.15.46:0/2245670168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c004c710 con 0x7f47c0159960 2024-01-26T23:46:29.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.077+0000 7f47a6ffd700 1 -- 172.21.15.46:0/2245670168 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f47b4025a30 con 0x7f47c0159960 2024-01-26T23:46:29.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.077+0000 7f47a6ffd700 1 --2- 172.21.15.46:0/2245670168 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f47ac066ea0 0x7f47ac069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:29.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.078+0000 7f47a6ffd700 1 -- 172.21.15.46:0/2245670168 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5565+0+0 (secure 0 0 0) 0x7f47b4085bb0 con 0x7f47c0159960 2024-01-26T23:46:29.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.078+0000 7f47beffd700 1 --2- 172.21.15.46:0/2245670168 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f47ac066ea0 0x7f47ac069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:29.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.078+0000 7f47beffd700 1 --2- 172.21.15.46:0/2245670168 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f47ac066ea0 0x7f47ac069350 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f47b0007fd0 tx=0x7f47b0007d30 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:29.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.082+0000 7f47a6ffd700 1 -- 172.21.15.46:0/2245670168 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f47b4052f90 con 0x7f47c0159960 2024-01-26T23:46:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.292+0000 7f47c56bf700 1 -- 172.21.15.46:0/2245670168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f47c00cc290 con 0x7f47c0159960 2024-01-26T23:46:29.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.292+0000 7f47a6ffd700 1 -- 172.21.15.46:0/2245670168 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v108) v1 ==== 84+0+65 (secure 0 0 0) 0x7f47b4015090 con 0x7f47c0159960 2024-01-26T23:46:29.294 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:29.294 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-26T23:46:29.294 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:29.294 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-01-26T23:46:29.294 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-01-26T23:46:29.294 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:29.294 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-26T23:46:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.295+0000 7f47c56bf700 1 -- 172.21.15.46:0/2245670168 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f47ac066ea0 msgr2=0x7f47ac069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.295+0000 7f47c56bf700 1 --2- 172.21.15.46:0/2245670168 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f47ac066ea0 0x7f47ac069350 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f47b0007fd0 tx=0x7f47b0007d30 comp rx=0 tx=0).stop 2024-01-26T23:46:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.295+0000 7f47c56bf700 1 -- 172.21.15.46:0/2245670168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c0159960 msgr2=0x7f47c015db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.295+0000 7f47c56bf700 1 --2- 172.21.15.46:0/2245670168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c0159960 0x7f47c015db90 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f47b4013000 tx=0x7f47b4017000 comp rx=0 tx=0).stop 2024-01-26T23:46:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.295+0000 7f47c56bf700 1 -- 172.21.15.46:0/2245670168 shutdown_connections 2024-01-26T23:46:29.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.295+0000 7f47c56bf700 1 --2- 172.21.15.46:0/2245670168 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f47ac066ea0 0x7f47ac069350 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.295+0000 7f47c56bf700 1 --2- 172.21.15.46:0/2245670168 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47c0159d30 0x7f47c0155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.295+0000 7f47c56bf700 1 --2- 172.21.15.46:0/2245670168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47c0068040 0x7f47c016d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.295+0000 7f47c56bf700 1 --2- 172.21.15.46:0/2245670168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c0159960 0x7f47c015db90 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.295+0000 7f47c56bf700 1 -- 172.21.15.46:0/2245670168 >> 172.21.15.46:0/2245670168 conn(0x7f47c006b850 msgr2=0x7f47c0067930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:29.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.295+0000 7f47c56bf700 1 -- 172.21.15.46:0/2245670168 shutdown_connections 2024-01-26T23:46:29.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.295+0000 7f47c56bf700 1 -- 172.21.15.46:0/2245670168 wait complete. 2024-01-26T23:46:29.310 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-26T23:46:29.310 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-26T23:46:29.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.446+0000 7fd9b3ef1700 1 Processor -- start 2024-01-26T23:46:29.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.447+0000 7fd9b3ef1700 1 -- start start 2024-01-26T23:46:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.447+0000 7fd9b3ef1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac159750 0x7fd9ac069650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.447+0000 7fd9b3ef1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd9ac069b90 0x7fd9ac15d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.447+0000 7fd9b3ef1700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ac0691c0 con 0x7fd9ac159b20 2024-01-26T23:46:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.447+0000 7fd9b3ef1700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ac06a5e0 con 0x7fd9ac069b90 2024-01-26T23:46:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.447+0000 7fd9b3ef1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac15dfd0 con 0x7fd9ac159750 2024-01-26T23:46:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.448+0000 7fd9b248e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd9ac069b90 0x7fd9ac15d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.448+0000 7fd9b148c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd9ac159b20 0x7fd9ac0ade00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37694/0 (socket says 172.21.15.46:37694) 2024-01-26T23:46:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.448+0000 7fd9b148c700 1 -- 172.21.15.46:0/2102878614 learned_addr learned my addr 172.21.15.46:0/2102878614 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.448+0000 7fd9b1c8d700 1 --2- 172.21.15.46:0/2102878614 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac159750 0x7fd9ac069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.448+0000 7fd9b0c8b700 1 -- 172.21.15.46:0/2102878614 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736193985 0 0) 0x7fd9ac0691c0 con 0x7fd9ac159b20 2024-01-26T23:46:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.448+0000 7fd9b0c8b700 1 -- 172.21.15.46:0/2102878614 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd998003570 con 0x7fd9ac159b20 2024-01-26T23:46:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.448+0000 7fd9b248e700 1 -- 172.21.15.46:0/2102878614 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac159750 msgr2=0x7fd9ac069650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.448+0000 7fd9b248e700 1 --2- 172.21.15.46:0/2102878614 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac159750 0x7fd9ac069650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.448+0000 7fd9b248e700 1 -- 172.21.15.46:0/2102878614 >> v1:172.21.15.46:6789/0 conn(0x7fd9ac159b20 legacy=0x7fd9ac0ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:29.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.448+0000 7fd9b248e700 1 -- 172.21.15.46:0/2102878614 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac068b70 con 0x7fd9ac069b90 2024-01-26T23:46:29.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.448+0000 7fd9b1c8d700 1 --2- 172.21.15.46:0/2102878614 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac159750 0x7fd9ac069650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:29.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.448+0000 7fd9b248e700 1 --2- 172.21.15.46:0/2102878614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9ac069b90 0x7fd9ac15d460 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4009e60 tx=0x7fd9a4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2379e8fe2377fb1c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:29.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.448+0000 7fd9b0c8b700 1 -- 172.21.15.46:0/2102878614 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a400e070 con 0x7fd9ac069b90 2024-01-26T23:46:29.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.448+0000 7fd9b0c8b700 1 -- 172.21.15.46:0/2102878614 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a400bab0 con 0x7fd9ac069b90 2024-01-26T23:46:29.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.448+0000 7fd9b0c8b700 1 -- 172.21.15.46:0/2102878614 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a400be70 con 0x7fd9ac069b90 2024-01-26T23:46:29.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.449+0000 7fd9b3ef1700 1 -- 172.21.15.46:0/2102878614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9ac069b90 msgr2=0x7fd9ac15d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:29.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.449+0000 7fd9b3ef1700 1 --2- 172.21.15.46:0/2102878614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9ac069b90 0x7fd9ac15d460 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4009e60 tx=0x7fd9a4009780 comp rx=0 tx=0).stop 2024-01-26T23:46:29.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.449+0000 7fd9b3ef1700 1 -- 172.21.15.46:0/2102878614 shutdown_connections 2024-01-26T23:46:29.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.449+0000 7fd9b3ef1700 1 --2- 172.21.15.46:0/2102878614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9ac069b90 0x7fd9ac15d460 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.449+0000 7fd9b3ef1700 1 --2- 172.21.15.46:0/2102878614 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac159750 0x7fd9ac069650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.449+0000 7fd9b3ef1700 1 -- 172.21.15.46:0/2102878614 >> 172.21.15.46:0/2102878614 conn(0x7fd9ac0ce530 msgr2=0x7fd9ac0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:29.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.449+0000 7fd9b3ef1700 1 -- 172.21.15.46:0/2102878614 shutdown_connections 2024-01-26T23:46:29.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.449+0000 7fd9b3ef1700 1 -- 172.21.15.46:0/2102878614 wait complete. 2024-01-26T23:46:29.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.450+0000 7fd9b3ef1700 1 Processor -- start 2024-01-26T23:46:29.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.451+0000 7fd9b3ef1700 1 -- start start 2024-01-26T23:46:29.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.451+0000 7fd9b3ef1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9ac069b90 0x7fd9ac0c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:29.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.451+0000 7fd9b3ef1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9ac159750 0x7fd9ac15a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:29.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.451+0000 7fd9b3ef1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac159b20 0x7fd9ac15d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:29.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.451+0000 7fd9b3ef1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac0c8830 con 0x7fd9ac069b90 2024-01-26T23:46:29.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.451+0000 7fd9b3ef1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac15aca0 con 0x7fd9ac159750 2024-01-26T23:46:29.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.451+0000 7fd9b3ef1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac0c7f90 con 0x7fd9ac159b20 2024-01-26T23:46:29.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.451+0000 7fd9b1c8d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9ac069b90 0x7fd9ac0c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:29.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.452+0000 7fd9b1c8d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9ac069b90 0x7fd9ac0c7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46658/0 (socket says 172.21.15.46:46658) 2024-01-26T23:46:29.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.452+0000 7fd9b1c8d700 1 -- 172.21.15.46:0/193360299 learned_addr learned my addr 172.21.15.46:0/193360299 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:29.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.452+0000 7fd9b248e700 1 --2- 172.21.15.46:0/193360299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac159b20 0x7fd9ac15d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:29.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.452+0000 7fd9b148c700 1 --2- 172.21.15.46:0/193360299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9ac159750 0x7fd9ac15a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:29.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.452+0000 7fd9b1c8d700 1 -- 172.21.15.46:0/193360299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac159b20 msgr2=0x7fd9ac15d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:29.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.452+0000 7fd9b1c8d700 1 --2- 172.21.15.46:0/193360299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac159b20 0x7fd9ac15d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.452+0000 7fd9b1c8d700 1 -- 172.21.15.46:0/193360299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9ac159750 msgr2=0x7fd9ac15a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:29.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.452+0000 7fd9b1c8d700 1 --2- 172.21.15.46:0/193360299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9ac159750 0x7fd9ac15a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.452+0000 7fd9b1c8d700 1 -- 172.21.15.46:0/193360299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a8009420 con 0x7fd9ac069b90 2024-01-26T23:46:29.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.452+0000 7fd9b148c700 1 --2- 172.21.15.46:0/193360299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9ac159750 0x7fd9ac15a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:46:29.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.452+0000 7fd9b1c8d700 1 --2- 172.21.15.46:0/193360299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9ac069b90 0x7fd9ac0c7960 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800ec80 tx=0x7fd9a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:29.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.452+0000 7fd9a1ffb700 1 -- 172.21.15.46:0/193360299 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a800c8f0 con 0x7fd9ac069b90 2024-01-26T23:46:29.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.452+0000 7fd9a1ffb700 1 -- 172.21.15.46:0/193360299 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a800ca50 con 0x7fd9ac069b90 2024-01-26T23:46:29.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.452+0000 7fd9a1ffb700 1 -- 172.21.15.46:0/193360299 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a800cdd0 con 0x7fd9ac069b90 2024-01-26T23:46:29.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.452+0000 7fd9b3ef1700 1 -- 172.21.15.46:0/193360299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a4009a20 con 0x7fd9ac069b90 2024-01-26T23:46:29.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.452+0000 7fd9b3ef1700 1 -- 172.21.15.46:0/193360299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ac0c8550 con 0x7fd9ac069b90 2024-01-26T23:46:29.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.453+0000 7fd9b3ef1700 1 -- 172.21.15.46:0/193360299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ac15af80 con 0x7fd9ac069b90 2024-01-26T23:46:29.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.454+0000 7fd9a1ffb700 1 -- 172.21.15.46:0/193360299 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fd9a8013680 con 0x7fd9ac069b90 2024-01-26T23:46:29.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.454+0000 7fd9a1ffb700 1 --2- 172.21.15.46:0/193360299 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd990066ea0 0x7fd990069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:29.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.454+0000 7fd9a1ffb700 1 -- 172.21.15.46:0/193360299 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5565+0+0 (secure 0 0 0) 0x7fd9a8084a50 con 0x7fd9ac069b90 2024-01-26T23:46:29.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.454+0000 7fd9b148c700 1 --2- 172.21.15.46:0/193360299 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd990066ea0 0x7fd990069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:29.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.455+0000 7fd9b148c700 1 --2- 172.21.15.46:0/193360299 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd990066ea0 0x7fd990069350 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac0b3660 tx=0x7fd99c009040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:29.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.460+0000 7fd9a1ffb700 1 -- 172.21.15.46:0/193360299 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd9a8051e30 con 0x7fd9ac069b90 2024-01-26T23:46:29.606 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:29.605+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:29.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.668+0000 7fd9b3ef1700 1 -- 172.21.15.46:0/193360299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd9ac04c710 con 0x7fd9ac069b90 2024-01-26T23:46:29.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.755+0000 7fd9a1ffb700 1 -- 172.21.15.46:0/193360299 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v109) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd9a80170b0 con 0x7fd9ac069b90 2024-01-26T23:46:29.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.760+0000 7fd9b3ef1700 1 -- 172.21.15.46:0/193360299 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd990066ea0 msgr2=0x7fd990069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:29.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.760+0000 7fd9b3ef1700 1 --2- 172.21.15.46:0/193360299 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd990066ea0 0x7fd990069350 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac0b3660 tx=0x7fd99c009040 comp rx=0 tx=0).stop 2024-01-26T23:46:29.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.760+0000 7fd9b3ef1700 1 -- 172.21.15.46:0/193360299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9ac069b90 msgr2=0x7fd9ac0c7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:29.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.760+0000 7fd9b3ef1700 1 --2- 172.21.15.46:0/193360299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9ac069b90 0x7fd9ac0c7960 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800ec80 tx=0x7fd9a80091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.760+0000 7fd9b3ef1700 1 -- 172.21.15.46:0/193360299 shutdown_connections 2024-01-26T23:46:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.760+0000 7fd9b3ef1700 1 --2- 172.21.15.46:0/193360299 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd990066ea0 0x7fd990069350 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac0b3660 tx=0x7fd99c009040 comp rx=0 tx=0).stop 2024-01-26T23:46:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.760+0000 7fd9b3ef1700 1 --2- 172.21.15.46:0/193360299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9ac069b90 0x7fd9ac0c7960 secure :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800ec80 tx=0x7fd9a80091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.760+0000 7fd9b3ef1700 1 --2- 172.21.15.46:0/193360299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac159b20 0x7fd9ac15d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.760+0000 7fd9b3ef1700 1 --2- 172.21.15.46:0/193360299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9ac159750 0x7fd9ac15a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.760+0000 7fd9b3ef1700 1 -- 172.21.15.46:0/193360299 >> 172.21.15.46:0/193360299 conn(0x7fd9ac0ce530 msgr2=0x7fd9ac0cd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:29.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.761+0000 7fd9b3ef1700 1 -- 172.21.15.46:0/193360299 shutdown_connections 2024-01-26T23:46:29.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.761+0000 7fd9b3ef1700 1 -- 172.21.15.46:0/193360299 wait complete. 2024-01-26T23:46:29.763 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-01-26T23:46:29.777 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-26T23:46:29.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.918+0000 7f93f5fab700 1 Processor -- start 2024-01-26T23:46:29.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.918+0000 7f93f5fab700 1 -- start start 2024-01-26T23:46:29.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.919+0000 7f93f5fab700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f93f00c8da0 0x7f93f00c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:29.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.919+0000 7f93f5fab700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f00ccc00 con 0x7f93f00caf30 2024-01-26T23:46:29.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.919+0000 7f93f5fab700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f00cc310 con 0x7f93f0140af0 2024-01-26T23:46:29.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.919+0000 7f93f5fab700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93f00cbbf0 con 0x7f93f00c8da0 2024-01-26T23:46:29.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.919+0000 7f93effff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f93f00caf30 0x7f93f00cb370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37708/0 (socket says 172.21.15.46:37708) 2024-01-26T23:46:29.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.919+0000 7f93effff700 1 -- 172.21.15.46:0/2940425302 learned_addr learned my addr 172.21.15.46:0/2940425302 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:29.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.919+0000 7f93ef7fe700 1 --2- 172.21.15.46:0/2940425302 >> v2:172.21.15.106:3300/0 conn(0x7f93f00c8da0 0x7f93f00c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:29.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.920+0000 7f93ee7fc700 1 -- 172.21.15.46:0/2940425302 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871370607 0 0) 0x7f93f00cc310 con 0x7f93f0140af0 2024-01-26T23:46:29.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.920+0000 7f93ee7fc700 1 -- 172.21.15.46:0/2940425302 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93d8003570 con 0x7f93f0140af0 2024-01-26T23:46:29.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.920+0000 7f93ee7fc700 1 -- 172.21.15.46:0/2940425302 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661619569 0 0) 0x7f93f00ccc00 con 0x7f93f00caf30 2024-01-26T23:46:29.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.920+0000 7f93ee7fc700 1 -- 172.21.15.46:0/2940425302 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93f00cc310 con 0x7f93f00caf30 2024-01-26T23:46:29.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.920+0000 7f93ef7fe700 1 -- 172.21.15.46:0/2940425302 >> v1:172.21.15.106:6789/0 conn(0x7f93f0140af0 legacy=0x7f93f0140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:29.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.920+0000 7f93ef7fe700 1 -- 172.21.15.46:0/2940425302 >> v1:172.21.15.46:6789/0 conn(0x7f93f00caf30 legacy=0x7f93f00cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:29.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.920+0000 7f93ef7fe700 1 -- 172.21.15.46:0/2940425302 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f00cc5b0 con 0x7f93f00c8da0 2024-01-26T23:46:29.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.920+0000 7f93ef7fe700 1 --2- 172.21.15.46:0/2940425302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93f00c8da0 0x7f93f00c9170 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f93e00098e0 tx=0x7f93e00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a78d8487d9d01db server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:29.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.920+0000 7f93ee7fc700 1 -- 172.21.15.46:0/2940425302 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0011630 con 0x7f93f00c8da0 2024-01-26T23:46:29.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.920+0000 7f93ee7fc700 1 -- 172.21.15.46:0/2940425302 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e0011790 con 0x7f93f00c8da0 2024-01-26T23:46:29.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.920+0000 7f93ee7fc700 1 -- 172.21.15.46:0/2940425302 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0011b30 con 0x7f93f00c8da0 2024-01-26T23:46:29.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.921+0000 7f93f5fab700 1 -- 172.21.15.46:0/2940425302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93f00c8da0 msgr2=0x7f93f00c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:29.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.921+0000 7f93f5fab700 1 --2- 172.21.15.46:0/2940425302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93f00c8da0 0x7f93f00c9170 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f93e00098e0 tx=0x7f93e00091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.921+0000 7f93f5fab700 1 -- 172.21.15.46:0/2940425302 shutdown_connections 2024-01-26T23:46:29.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.921+0000 7f93f5fab700 1 --2- 172.21.15.46:0/2940425302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93f00c8da0 0x7f93f00c9170 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.921+0000 7f93f5fab700 1 -- 172.21.15.46:0/2940425302 >> 172.21.15.46:0/2940425302 conn(0x7f93f006bd90 msgr2=0x7f93f006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:29.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.921+0000 7f93f5fab700 1 -- 172.21.15.46:0/2940425302 shutdown_connections 2024-01-26T23:46:29.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.921+0000 7f93f5fab700 1 -- 172.21.15.46:0/2940425302 wait complete. 2024-01-26T23:46:29.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.922+0000 7f93f5fab700 1 Processor -- start 2024-01-26T23:46:29.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.922+0000 7f93f5fab700 1 -- start start 2024-01-26T23:46:29.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.923+0000 7f93f5fab700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93f00c8da0 0x7f93f0156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.923+0000 7f93f5fab700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93f00caf30 0x7f93f00b2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.923+0000 7f93f5fab700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f93f0140af0 0x7f93f00b3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.923+0000 7f93f5fab700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f0159600 con 0x7f93f00c8da0 2024-01-26T23:46:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.923+0000 7f93f5fab700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f0157520 con 0x7f93f00caf30 2024-01-26T23:46:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.923+0000 7f93f5fab700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f00b3e00 con 0x7f93f0140af0 2024-01-26T23:46:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.923+0000 7f93ef7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93f00c8da0 0x7f93f0156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.923+0000 7f93ef7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93f00c8da0 0x7f93f0156a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46668/0 (socket says 172.21.15.46:46668) 2024-01-26T23:46:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.923+0000 7f93ef7fe700 1 -- 172.21.15.46:0/2474082977 learned_addr learned my addr 172.21.15.46:0/2474082977 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.923+0000 7f93eeffd700 1 --2- 172.21.15.46:0/2474082977 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93f00caf30 0x7f93f00b2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.923+0000 7f93effff700 1 --2- 172.21.15.46:0/2474082977 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f93f0140af0 0x7f93f00b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.924+0000 7f93ef7fe700 1 -- 172.21.15.46:0/2474082977 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f93f0140af0 msgr2=0x7f93f00b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.924+0000 7f93ef7fe700 1 --2- 172.21.15.46:0/2474082977 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f93f0140af0 0x7f93f00b3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.924+0000 7f93ef7fe700 1 -- 172.21.15.46:0/2474082977 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93f00caf30 msgr2=0x7f93f00b2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.924+0000 7f93ef7fe700 1 --2- 172.21.15.46:0/2474082977 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93f00caf30 0x7f93f00b2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.924+0000 7f93ef7fe700 1 -- 172.21.15.46:0/2474082977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e0009420 con 0x7f93f00c8da0 2024-01-26T23:46:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.924+0000 7f93ef7fe700 1 --2- 172.21.15.46:0/2474082977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93f00c8da0 0x7f93f0156a20 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f93e0009ed0 tx=0x7f93e0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.924+0000 7f93d77fe700 1 -- 172.21.15.46:0/2474082977 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e001f070 con 0x7f93f00c8da0 2024-01-26T23:46:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.924+0000 7f93d77fe700 1 -- 172.21.15.46:0/2474082977 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e0011900 con 0x7f93f00c8da0 2024-01-26T23:46:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.924+0000 7f93f5fab700 1 -- 172.21.15.46:0/2474082977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f015a2f0 con 0x7f93f00c8da0 2024-01-26T23:46:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.924+0000 7f93d77fe700 1 -- 172.21.15.46:0/2474082977 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0023410 con 0x7f93f00c8da0 2024-01-26T23:46:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.924+0000 7f93f5fab700 1 -- 172.21.15.46:0/2474082977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f0159ca0 con 0x7f93f00c8da0 2024-01-26T23:46:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.925+0000 7f93f5fab700 1 -- 172.21.15.46:0/2474082977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f004c710 con 0x7f93f00c8da0 2024-01-26T23:46:29.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.926+0000 7f93d77fe700 1 -- 172.21.15.46:0/2474082977 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f93e002c3b0 con 0x7f93f00c8da0 2024-01-26T23:46:29.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.926+0000 7f93d77fe700 1 --2- 172.21.15.46:0/2474082977 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f93dc066ea0 0x7f93dc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:29.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.926+0000 7f93d77fe700 1 -- 172.21.15.46:0/2474082977 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5935+0+0 (secure 0 0 0) 0x7f93e0085700 con 0x7f93f00c8da0 2024-01-26T23:46:29.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.926+0000 7f93eeffd700 1 --2- 172.21.15.46:0/2474082977 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f93dc066ea0 0x7f93dc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:29.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.927+0000 7f93eeffd700 1 --2- 172.21.15.46:0/2474082977 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f93dc066ea0 0x7f93dc069350 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f93f00cce20 tx=0x7f93e4007d50 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:29.931+0000 7f93d77fe700 1 -- 172.21.15.46:0/2474082977 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f93e00529b0 con 0x7f93f00c8da0 2024-01-26T23:46:30.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.143+0000 7f93f5fab700 1 -- 172.21.15.46:0/2474082977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f93f00cc890 con 0x7f93f00c8da0 2024-01-26T23:46:30.424 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:30.423+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.759+0000 7f93d77fe700 1 -- 172.21.15.46:0/2474082977 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v110) v1 ==== 170+0+0 (secure 0 0 0) 0x7f93e0055f90 con 0x7f93f00c8da0 2024-01-26T23:46:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.768+0000 7f93f5fab700 1 -- 172.21.15.46:0/2474082977 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f93dc066ea0 msgr2=0x7f93dc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.768+0000 7f93f5fab700 1 --2- 172.21.15.46:0/2474082977 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f93dc066ea0 0x7f93dc069350 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f93f00cce20 tx=0x7f93e4007d50 comp rx=0 tx=0).stop 2024-01-26T23:46:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.768+0000 7f93f5fab700 1 -- 172.21.15.46:0/2474082977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93f00c8da0 msgr2=0x7f93f0156a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.768+0000 7f93f5fab700 1 --2- 172.21.15.46:0/2474082977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93f00c8da0 0x7f93f0156a20 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f93e0009ed0 tx=0x7f93e0017000 comp rx=0 tx=0).stop 2024-01-26T23:46:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.769+0000 7f93f5fab700 1 -- 172.21.15.46:0/2474082977 shutdown_connections 2024-01-26T23:46:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.769+0000 7f93f5fab700 1 --2- 172.21.15.46:0/2474082977 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f93dc066ea0 0x7f93dc069350 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.769+0000 7f93f5fab700 1 --2- 172.21.15.46:0/2474082977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93f00c8da0 0x7f93f0156a20 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:30.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.769+0000 7f93f5fab700 1 --2- 172.21.15.46:0/2474082977 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f93f0140af0 0x7f93f00b3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:30.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.769+0000 7f93f5fab700 1 --2- 172.21.15.46:0/2474082977 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93f00caf30 0x7f93f00b2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:30.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.769+0000 7f93f5fab700 1 -- 172.21.15.46:0/2474082977 >> 172.21.15.46:0/2474082977 conn(0x7f93f006bd90 msgr2=0x7f93f00c3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:30.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.771+0000 7f93f5fab700 1 -- 172.21.15.46:0/2474082977 shutdown_connections 2024-01-26T23:46:30.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.771+0000 7f93f5fab700 1 -- 172.21.15.46:0/2474082977 wait complete. 2024-01-26T23:46:30.776 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-01-26T23:46:30.793 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-26T23:46:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.932+0000 7f17ba9bc700 1 Processor -- start 2024-01-26T23:46:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.932+0000 7f17ba9bc700 1 -- start start 2024-01-26T23:46:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.933+0000 7f17ba9bc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17b4141200 0x7f17b40c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.933+0000 7f17ba9bc700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b40ccc00 con 0x7f17b40c8da0 2024-01-26T23:46:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.933+0000 7f17ba9bc700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b40cc310 con 0x7f17b40caf30 2024-01-26T23:46:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.933+0000 7f17ba9bc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17b40cbbf0 con 0x7f17b4141200 2024-01-26T23:46:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.934+0000 7f17b3fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f17b40c8da0 0x7f17b4140af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40818/0 (socket says 172.21.15.46:40818) 2024-01-26T23:46:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.934+0000 7f17b3fff700 1 -- 172.21.15.46:0/1031492183 learned_addr learned my addr 172.21.15.46:0/1031492183 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.934+0000 7f17b37fe700 1 --2- 172.21.15.46:0/1031492183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17b4141200 0x7f17b40c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.934+0000 7f17b2ffd700 1 -- 172.21.15.46:0/1031492183 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135595938 0 0) 0x7f17b40cc310 con 0x7f17b40caf30 2024-01-26T23:46:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.934+0000 7f17b2ffd700 1 -- 172.21.15.46:0/1031492183 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f179c003570 con 0x7f17b40caf30 2024-01-26T23:46:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.934+0000 7f17b2ffd700 1 -- 172.21.15.46:0/1031492183 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643040082 0 0) 0x7f17b40ccc00 con 0x7f17b40c8da0 2024-01-26T23:46:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.934+0000 7f17b37fe700 1 -- 172.21.15.46:0/1031492183 >> v1:172.21.15.106:6789/0 conn(0x7f17b40caf30 legacy=0x7f17b40cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.934+0000 7f17b37fe700 1 -- 172.21.15.46:0/1031492183 >> v1:172.21.15.46:6789/0 conn(0x7f17b40c8da0 legacy=0x7f17b4140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.934+0000 7f17b37fe700 1 -- 172.21.15.46:0/1031492183 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b40cc5b0 con 0x7f17b4141200 2024-01-26T23:46:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.934+0000 7f17b37fe700 1 --2- 172.21.15.46:0/1031492183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17b4141200 0x7f17b40c9170 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f17a8009910 tx=0x7f17a80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3473fb2d830f9cc1 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.934+0000 7f17b2ffd700 1 -- 172.21.15.46:0/1031492183 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a800b680 con 0x7f17b4141200 2024-01-26T23:46:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.934+0000 7f17b2ffd700 1 -- 172.21.15.46:0/1031492183 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a800b7e0 con 0x7f17b4141200 2024-01-26T23:46:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.934+0000 7f17b2ffd700 1 -- 172.21.15.46:0/1031492183 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a800bba0 con 0x7f17b4141200 2024-01-26T23:46:30.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.935+0000 7f17ba9bc700 1 -- 172.21.15.46:0/1031492183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17b4141200 msgr2=0x7f17b40c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:30.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.935+0000 7f17ba9bc700 1 --2- 172.21.15.46:0/1031492183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17b4141200 0x7f17b40c9170 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f17a8009910 tx=0x7f17a80091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.935+0000 7f17ba9bc700 1 -- 172.21.15.46:0/1031492183 shutdown_connections 2024-01-26T23:46:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.935+0000 7f17ba9bc700 1 --2- 172.21.15.46:0/1031492183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17b4141200 0x7f17b40c9170 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.935+0000 7f17ba9bc700 1 -- 172.21.15.46:0/1031492183 >> 172.21.15.46:0/1031492183 conn(0x7f17b406bd90 msgr2=0x7f17b406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.935+0000 7f17ba9bc700 1 -- 172.21.15.46:0/1031492183 shutdown_connections 2024-01-26T23:46:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.935+0000 7f17ba9bc700 1 -- 172.21.15.46:0/1031492183 wait complete. 2024-01-26T23:46:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.937+0000 7f17ba9bc700 1 Processor -- start 2024-01-26T23:46:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.937+0000 7f17ba9bc700 1 -- start start 2024-01-26T23:46:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.938+0000 7f17ba9bc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17b40c8da0 0x7f17b40c6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.938+0000 7f17ba9bc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17b40caf30 0x7f17b40b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.938+0000 7f17ba9bc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b4141200 0x7f17b4156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.938+0000 7f17ba9bc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b40c6570 con 0x7f17b4141200 2024-01-26T23:46:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.938+0000 7f17ba9bc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b40c7370 con 0x7f17b40caf30 2024-01-26T23:46:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.938+0000 7f17ba9bc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17b40c5030 con 0x7f17b40c8da0 2024-01-26T23:46:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.938+0000 7f17b8f59700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b4141200 0x7f17b4156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.938+0000 7f17b8f59700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b4141200 0x7f17b4156850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33526/0 (socket says 172.21.15.46:33526) 2024-01-26T23:46:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.938+0000 7f17b8f59700 1 -- 172.21.15.46:0/3744655810 learned_addr learned my addr 172.21.15.46:0/3744655810 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.938+0000 7f17b37fe700 1 --2- 172.21.15.46:0/3744655810 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17b40caf30 0x7f17b40b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.938+0000 7f17b3fff700 1 --2- 172.21.15.46:0/3744655810 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17b40c8da0 0x7f17b40c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.939+0000 7f17b8f59700 1 -- 172.21.15.46:0/3744655810 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17b40c8da0 msgr2=0x7f17b40c6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.939+0000 7f17b8f59700 1 --2- 172.21.15.46:0/3744655810 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17b40c8da0 0x7f17b40c6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.939+0000 7f17b8f59700 1 -- 172.21.15.46:0/3744655810 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17b40caf30 msgr2=0x7f17b40b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.939+0000 7f17b8f59700 1 --2- 172.21.15.46:0/3744655810 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17b40caf30 0x7f17b40b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.939+0000 7f17b8f59700 1 -- 172.21.15.46:0/3744655810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a8009420 con 0x7f17b4141200 2024-01-26T23:46:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.939+0000 7f17b8f59700 1 --2- 172.21.15.46:0/3744655810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b4141200 0x7f17b4156850 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f17ac00c910 tx=0x7f17ac0027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.939+0000 7f179bfff700 1 -- 172.21.15.46:0/3744655810 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ac012630 con 0x7f17b4141200 2024-01-26T23:46:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.939+0000 7f179bfff700 1 -- 172.21.15.46:0/3744655810 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17ac0127b0 con 0x7f17b4141200 2024-01-26T23:46:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.939+0000 7f17ba9bc700 1 -- 172.21.15.46:0/3744655810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b40c6760 con 0x7f17b4141200 2024-01-26T23:46:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.939+0000 7f179bfff700 1 -- 172.21.15.46:0/3744655810 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ac012b70 con 0x7f17b4141200 2024-01-26T23:46:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.939+0000 7f17ba9bc700 1 -- 172.21.15.46:0/3744655810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b40c6d20 con 0x7f17b4141200 2024-01-26T23:46:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.940+0000 7f17ba9bc700 1 -- 172.21.15.46:0/3744655810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17b404c710 con 0x7f17b4141200 2024-01-26T23:46:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.941+0000 7f179bfff700 1 -- 172.21.15.46:0/3744655810 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f17ac01b400 con 0x7f17b4141200 2024-01-26T23:46:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.941+0000 7f179bfff700 1 --2- 172.21.15.46:0/3744655810 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f17a0066ea0 0x7f17a0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.942+0000 7f179bfff700 1 -- 172.21.15.46:0/3744655810 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f17ac084910 con 0x7f17b4141200 2024-01-26T23:46:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.942+0000 7f17b3fff700 1 --2- 172.21.15.46:0/3744655810 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f17a0066ea0 0x7f17a0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.942+0000 7f17b3fff700 1 --2- 172.21.15.46:0/3744655810 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f17a0066ea0 0x7f17a0069350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f17a4007fd0 tx=0x7f17a4007d10 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:30.946+0000 7f179bfff700 1 -- 172.21.15.46:0/3744655810 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f17ac051a00 con 0x7f17b4141200 2024-01-26T23:46:31.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.206+0000 7f17ba9bc700 1 -- 172.21.15.46:0/3744655810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f17b40cc890 con 0x7f17b4141200 2024-01-26T23:46:31.263 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:31.262+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:31.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.765+0000 7f179bfff700 1 -- 172.21.15.46:0/3744655810 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v50) v1 ==== 174+0+0 (secure 0 0 0) 0x7f17ac055020 con 0x7f17b4141200 2024-01-26T23:46:31.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.768+0000 7f17ba9bc700 1 -- 172.21.15.46:0/3744655810 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f17a0066ea0 msgr2=0x7f17a0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:31.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.768+0000 7f17ba9bc700 1 --2- 172.21.15.46:0/3744655810 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f17a0066ea0 0x7f17a0069350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f17a4007fd0 tx=0x7f17a4007d10 comp rx=0 tx=0).stop 2024-01-26T23:46:31.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.768+0000 7f17ba9bc700 1 -- 172.21.15.46:0/3744655810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b4141200 msgr2=0x7f17b4156850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:31.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.768+0000 7f17ba9bc700 1 --2- 172.21.15.46:0/3744655810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b4141200 0x7f17b4156850 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f17ac00c910 tx=0x7f17ac0027a0 comp rx=0 tx=0).stop 2024-01-26T23:46:31.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.768+0000 7f17ba9bc700 1 -- 172.21.15.46:0/3744655810 shutdown_connections 2024-01-26T23:46:31.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.768+0000 7f17ba9bc700 1 --2- 172.21.15.46:0/3744655810 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f17a0066ea0 0x7f17a0069350 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:31.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.768+0000 7f17ba9bc700 1 --2- 172.21.15.46:0/3744655810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b4141200 0x7f17b4156850 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:31.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.768+0000 7f17ba9bc700 1 --2- 172.21.15.46:0/3744655810 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17b40c8da0 0x7f17b40c6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:31.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.768+0000 7f17ba9bc700 1 --2- 172.21.15.46:0/3744655810 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17b40caf30 0x7f17b40b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:31.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.768+0000 7f17ba9bc700 1 -- 172.21.15.46:0/3744655810 >> 172.21.15.46:0/3744655810 conn(0x7f17b406bd90 msgr2=0x7f17b40cb370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:31.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.769+0000 7f17ba9bc700 1 -- 172.21.15.46:0/3744655810 shutdown_connections 2024-01-26T23:46:31.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.769+0000 7f17ba9bc700 1 -- 172.21.15.46:0/3744655810 wait complete. 2024-01-26T23:46:31.776 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 13 and data pool 14 2024-01-26T23:46:31.789 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:46:31.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.927+0000 7faddb6df700 1 Processor -- start 2024-01-26T23:46:31.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.927+0000 7faddb6df700 1 -- start start 2024-01-26T23:46:31.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.928+0000 7faddb6df700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadd40b2530 0x7fadd415a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:31.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.928+0000 7faddb6df700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd40a9b90 con 0x7fadd40b2160 2024-01-26T23:46:31.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.928+0000 7faddb6df700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd40aa210 con 0x7fadd415da10 2024-01-26T23:46:31.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.928+0000 7faddb6df700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd415d580 con 0x7fadd40b2530 2024-01-26T23:46:31.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.928+0000 7fadd947b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fadd40b2160 0x7fadd40abe50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40834/0 (socket says 172.21.15.46:40834) 2024-01-26T23:46:31.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.928+0000 7fadd947b700 1 -- 172.21.15.46:0/3235258658 learned_addr learned my addr 172.21.15.46:0/3235258658 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:31.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.928+0000 7fadd8c7a700 1 --2- 172.21.15.46:0/3235258658 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadd40b2530 0x7fadd415a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:31.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.928+0000 7fadcbfff700 1 -- 172.21.15.46:0/3235258658 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024661628 0 0) 0x7fadd40a9b90 con 0x7fadd40b2160 2024-01-26T23:46:31.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.928+0000 7fadcbfff700 1 -- 172.21.15.46:0/3235258658 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc0003570 con 0x7fadd40b2160 2024-01-26T23:46:31.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.929+0000 7fadcbfff700 1 -- 172.21.15.46:0/3235258658 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320983196 0 0) 0x7fadd40aa210 con 0x7fadd415da10 2024-01-26T23:46:31.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.929+0000 7fadcbfff700 1 -- 172.21.15.46:0/3235258658 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd40a9b90 con 0x7fadd415da10 2024-01-26T23:46:31.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.929+0000 7fadd8c7a700 1 -- 172.21.15.46:0/3235258658 >> v1:172.21.15.106:6789/0 conn(0x7fadd415da10 legacy=0x7fadd406a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:31.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.929+0000 7fadd8c7a700 1 -- 172.21.15.46:0/3235258658 >> v1:172.21.15.46:6789/0 conn(0x7fadd40b2160 legacy=0x7fadd40abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:31.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.929+0000 7fadd8c7a700 1 -- 172.21.15.46:0/3235258658 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd40aa460 con 0x7fadd40b2530 2024-01-26T23:46:31.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.929+0000 7fadd8c7a700 1 --2- 172.21.15.46:0/3235258658 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadd40b2530 0x7fadd415a7a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fadc4009910 tx=0x7fadc40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cd7ed3254cccdee server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:31.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.929+0000 7fadcbfff700 1 -- 172.21.15.46:0/3235258658 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc4011630 con 0x7fadd40b2530 2024-01-26T23:46:31.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.929+0000 7fadcbfff700 1 -- 172.21.15.46:0/3235258658 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc4011790 con 0x7fadd40b2530 2024-01-26T23:46:31.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.929+0000 7fadcbfff700 1 -- 172.21.15.46:0/3235258658 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc4011b30 con 0x7fadd40b2530 2024-01-26T23:46:31.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.930+0000 7faddb6df700 1 -- 172.21.15.46:0/3235258658 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadd40b2530 msgr2=0x7fadd415a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:31.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.930+0000 7faddb6df700 1 --2- 172.21.15.46:0/3235258658 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadd40b2530 0x7fadd415a7a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fadc4009910 tx=0x7fadc40091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:31.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.930+0000 7faddb6df700 1 -- 172.21.15.46:0/3235258658 shutdown_connections 2024-01-26T23:46:31.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.930+0000 7faddb6df700 1 --2- 172.21.15.46:0/3235258658 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadd40b2530 0x7fadd415a7a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:31.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.930+0000 7faddb6df700 1 -- 172.21.15.46:0/3235258658 >> 172.21.15.46:0/3235258658 conn(0x7fadd40ccbc0 msgr2=0x7fadd40ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:31.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.930+0000 7faddb6df700 1 -- 172.21.15.46:0/3235258658 shutdown_connections 2024-01-26T23:46:31.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.930+0000 7faddb6df700 1 -- 172.21.15.46:0/3235258658 wait complete. 2024-01-26T23:46:31.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.931+0000 7faddb6df700 1 Processor -- start 2024-01-26T23:46:31.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.931+0000 7faddb6df700 1 -- start start 2024-01-26T23:46:31.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.932+0000 7faddb6df700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadd40b2160 0x7fadd413f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:31.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.932+0000 7faddb6df700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd40b2530 0x7fadd40aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:31.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.932+0000 7faddb6df700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadd415da10 0x7fadd4155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:31.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.932+0000 7faddb6df700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd413fbc0 con 0x7fadd40b2530 2024-01-26T23:46:31.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.932+0000 7faddb6df700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd41553e0 con 0x7fadd40b2160 2024-01-26T23:46:31.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.932+0000 7faddb6df700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd41547d0 con 0x7fadd415da10 2024-01-26T23:46:31.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.932+0000 7fadd8c7a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd40b2530 0x7fadd40aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:31.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.932+0000 7fadd8c7a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd40b2530 0x7fadd40aede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33536/0 (socket says 172.21.15.46:33536) 2024-01-26T23:46:31.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.932+0000 7fadd9c7c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadd415da10 0x7fadd4155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:31.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.932+0000 7fadd947b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadd40b2160 0x7fadd413f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:31.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.932+0000 7fadd8c7a700 1 -- 172.21.15.46:0/2852461693 learned_addr learned my addr 172.21.15.46:0/2852461693 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:31.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.932+0000 7fadd8c7a700 1 -- 172.21.15.46:0/2852461693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadd415da10 msgr2=0x7fadd4155870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:31.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.933+0000 7fadd8c7a700 1 --2- 172.21.15.46:0/2852461693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadd415da10 0x7fadd4155870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:31.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.933+0000 7fadd8c7a700 1 -- 172.21.15.46:0/2852461693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadd40b2160 msgr2=0x7fadd413f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:31.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.933+0000 7fadd8c7a700 1 --2- 172.21.15.46:0/2852461693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadd40b2160 0x7fadd413f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:31.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.933+0000 7fadd8c7a700 1 -- 172.21.15.46:0/2852461693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc4009420 con 0x7fadd40b2530 2024-01-26T23:46:31.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.933+0000 7fadd947b700 1 --2- 172.21.15.46:0/2852461693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadd40b2160 0x7fadd413f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:31.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.933+0000 7fadd8c7a700 1 --2- 172.21.15.46:0/2852461693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd40b2530 0x7fadd40aede0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fadc4009f00 tx=0x7fadc4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:31.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.933+0000 7fadc97fa700 1 -- 172.21.15.46:0/2852461693 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc401f070 con 0x7fadd40b2530 2024-01-26T23:46:31.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.933+0000 7fadc97fa700 1 -- 172.21.15.46:0/2852461693 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc4002f70 con 0x7fadd40b2530 2024-01-26T23:46:31.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.933+0000 7faddb6df700 1 -- 172.21.15.46:0/2852461693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd4154ab0 con 0x7fadd40b2530 2024-01-26T23:46:31.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.933+0000 7fadc97fa700 1 -- 172.21.15.46:0/2852461693 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc4006af0 con 0x7fadd40b2530 2024-01-26T23:46:31.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.933+0000 7faddb6df700 1 -- 172.21.15.46:0/2852461693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd4155070 con 0x7fadd40b2530 2024-01-26T23:46:31.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.934+0000 7faddb6df700 1 -- 172.21.15.46:0/2852461693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd404c710 con 0x7fadd40b2530 2024-01-26T23:46:31.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.935+0000 7fadc97fa700 1 -- 172.21.15.46:0/2852461693 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fadc4023070 con 0x7fadd40b2530 2024-01-26T23:46:31.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.935+0000 7fadc97fa700 1 --2- 172.21.15.46:0/2852461693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadb8066ea0 0x7fadb8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:31.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.935+0000 7fadc97fa700 1 -- 172.21.15.46:0/2852461693 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fadc4085810 con 0x7fadd40b2530 2024-01-26T23:46:31.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.935+0000 7fadd947b700 1 --2- 172.21.15.46:0/2852461693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadb8066ea0 0x7fadb8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:31.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.936+0000 7fadd947b700 1 --2- 172.21.15.46:0/2852461693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadb8066ea0 0x7fadb8069350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fadd0007fd0 tx=0x7fadd0009040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:31.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:31.940+0000 7fadc97fa700 1 -- 172.21.15.46:0/2852461693 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fadc4052890 con 0x7fadd40b2530 2024-01-26T23:46:32.117 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:32.116+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:32.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.158+0000 7faddb6df700 1 -- 172.21.15.46:0/2852461693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadd40aa740 con 0x7fadd40b2530 2024-01-26T23:46:32.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.159+0000 7fadc97fa700 1 -- 172.21.15.46:0/2852461693 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16176 (secure 0 0 0) 0x7fadc402a090 con 0x7fadd40b2530 2024-01-26T23:46:32.160 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:32.160 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":111,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:46:31.208514+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:46:29.702709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-01-26T23:46:30.178825+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:46:32.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.161+0000 7faddb6df700 1 -- 172.21.15.46:0/2852461693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadb8066ea0 msgr2=0x7fadb8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:32.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.161+0000 7faddb6df700 1 --2- 172.21.15.46:0/2852461693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadb8066ea0 0x7fadb8069350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fadd0007fd0 tx=0x7fadd0009040 comp rx=0 tx=0).stop 2024-01-26T23:46:32.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.161+0000 7faddb6df700 1 -- 172.21.15.46:0/2852461693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd40b2530 msgr2=0x7fadd40aede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:32.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.161+0000 7faddb6df700 1 --2- 172.21.15.46:0/2852461693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd40b2530 0x7fadd40aede0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fadc4009f00 tx=0x7fadc4017000 comp rx=0 tx=0).stop 2024-01-26T23:46:32.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.161+0000 7faddb6df700 1 -- 172.21.15.46:0/2852461693 shutdown_connections 2024-01-26T23:46:32.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.161+0000 7faddb6df700 1 --2- 172.21.15.46:0/2852461693 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fadb8066ea0 0x7fadb8069350 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:32.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.161+0000 7faddb6df700 1 --2- 172.21.15.46:0/2852461693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fadd415da10 0x7fadd4155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:32.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.161+0000 7faddb6df700 1 --2- 172.21.15.46:0/2852461693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadd40b2160 0x7fadd413f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:32.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.161+0000 7faddb6df700 1 --2- 172.21.15.46:0/2852461693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd40b2530 0x7fadd40aede0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:32.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.161+0000 7faddb6df700 1 -- 172.21.15.46:0/2852461693 >> 172.21.15.46:0/2852461693 conn(0x7fadd40ccbc0 msgr2=0x7fadd4150e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:32.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.161+0000 7faddb6df700 1 -- 172.21.15.46:0/2852461693 shutdown_connections 2024-01-26T23:46:32.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.161+0000 7faddb6df700 1 -- 172.21.15.46:0/2852461693 wait complete. 2024-01-26T23:46:32.176 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:46:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.313+0000 7f4b7317f700 1 Processor -- start 2024-01-26T23:46:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.313+0000 7f4b7317f700 1 -- start start 2024-01-26T23:46:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.314+0000 7f4b7317f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b6c0cd250 0x7f4b6c162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.314+0000 7f4b7317f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b6c0cbf10 con 0x7f4b6c0acf30 2024-01-26T23:46:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.314+0000 7f4b7317f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b6c0c4460 con 0x7f4b6c0c3cf0 2024-01-26T23:46:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.314+0000 7f4b7317f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b6c0ada10 con 0x7f4b6c0cd250 2024-01-26T23:46:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.314+0000 7f4b7171c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b6c0cd250 0x7f4b6c162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.314+0000 7f4b7171c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b6c0cd250 0x7f4b6c162360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33538/0 (socket says 172.21.15.46:33538) 2024-01-26T23:46:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.314+0000 7f4b7171c700 1 -- 172.21.15.46:0/4252273730 learned_addr learned my addr 172.21.15.46:0/4252273730 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:32.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.314+0000 7f4b70f1b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4b6c0acf30 0x7f4b6c0ad300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40848/0 (socket says 172.21.15.46:40848) 2024-01-26T23:46:32.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.315+0000 7f4b6b7fe700 1 -- 172.21.15.46:0/4252273730 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649551096 0 0) 0x7f4b6c0c4460 con 0x7f4b6c0c3cf0 2024-01-26T23:46:32.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.315+0000 7f4b7171c700 1 -- 172.21.15.46:0/4252273730 >> v1:172.21.15.106:6789/0 conn(0x7f4b6c0c3cf0 legacy=0x7f4b6c0ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:32.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.315+0000 7f4b7171c700 1 -- 172.21.15.46:0/4252273730 >> v1:172.21.15.46:6789/0 conn(0x7f4b6c0acf30 legacy=0x7f4b6c0ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:32.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.315+0000 7f4b7171c700 1 -- 172.21.15.46:0/4252273730 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6c0c4740 con 0x7f4b6c0cd250 2024-01-26T23:46:32.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.315+0000 7f4b7171c700 1 --2- 172.21.15.46:0/4252273730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b6c0cd250 0x7f4b6c162360 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f4b64009f20 tx=0x7f4b64009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c31acec21d9062b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:32.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.315+0000 7f4b6b7fe700 1 -- 172.21.15.46:0/4252273730 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6400e070 con 0x7f4b6c0cd250 2024-01-26T23:46:32.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.315+0000 7f4b6b7fe700 1 -- 172.21.15.46:0/4252273730 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6400b980 con 0x7f4b6c0cd250 2024-01-26T23:46:32.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.315+0000 7f4b6b7fe700 1 -- 172.21.15.46:0/4252273730 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b640092e0 con 0x7f4b6c0cd250 2024-01-26T23:46:32.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.316+0000 7f4b7317f700 1 -- 172.21.15.46:0/4252273730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b6c0cd250 msgr2=0x7f4b6c162360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:32.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.316+0000 7f4b7317f700 1 --2- 172.21.15.46:0/4252273730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b6c0cd250 0x7f4b6c162360 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f4b64009f20 tx=0x7f4b64009840 comp rx=0 tx=0).stop 2024-01-26T23:46:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.316+0000 7f4b7317f700 1 -- 172.21.15.46:0/4252273730 shutdown_connections 2024-01-26T23:46:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.316+0000 7f4b7317f700 1 --2- 172.21.15.46:0/4252273730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b6c0cd250 0x7f4b6c162360 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.316+0000 7f4b7317f700 1 -- 172.21.15.46:0/4252273730 >> 172.21.15.46:0/4252273730 conn(0x7f4b6c097820 msgr2=0x7f4b6c068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.316+0000 7f4b7317f700 1 -- 172.21.15.46:0/4252273730 shutdown_connections 2024-01-26T23:46:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.316+0000 7f4b7317f700 1 -- 172.21.15.46:0/4252273730 wait complete. 2024-01-26T23:46:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.317+0000 7f4b7317f700 1 Processor -- start 2024-01-26T23:46:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.317+0000 7f4b7317f700 1 -- start start 2024-01-26T23:46:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.318+0000 7f4b7317f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b6c0acf30 0x7f4b6c152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.318+0000 7f4b7317f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b6c0c3cf0 0x7f4b6c168380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.318+0000 7f4b7317f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b6c0cd250 0x7f4b6c158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.318+0000 7f4b7317f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c152560 con 0x7f4b6c0acf30 2024-01-26T23:46:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.318+0000 7f4b7317f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c0b35d0 con 0x7f4b6c0cd250 2024-01-26T23:46:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.318+0000 7f4b7317f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c152ce0 con 0x7f4b6c0c3cf0 2024-01-26T23:46:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.318+0000 7f4b70f1b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b6c0acf30 0x7f4b6c152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.318+0000 7f4b70f1b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b6c0acf30 0x7f4b6c152020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33554/0 (socket says 172.21.15.46:33554) 2024-01-26T23:46:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.318+0000 7f4b70f1b700 1 -- 172.21.15.46:0/3186964307 learned_addr learned my addr 172.21.15.46:0/3186964307 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.318+0000 7f4b7171c700 1 --2- 172.21.15.46:0/3186964307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b6c0cd250 0x7f4b6c158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.318+0000 7f4b6bfff700 1 --2- 172.21.15.46:0/3186964307 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b6c0c3cf0 0x7f4b6c168380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.319+0000 7f4b70f1b700 1 -- 172.21.15.46:0/3186964307 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b6c0c3cf0 msgr2=0x7f4b6c168380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.319+0000 7f4b70f1b700 1 --2- 172.21.15.46:0/3186964307 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b6c0c3cf0 0x7f4b6c168380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.319+0000 7f4b70f1b700 1 -- 172.21.15.46:0/3186964307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b6c0cd250 msgr2=0x7f4b6c158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.319+0000 7f4b70f1b700 1 --2- 172.21.15.46:0/3186964307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b6c0cd250 0x7f4b6c158770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.319+0000 7f4b70f1b700 1 -- 172.21.15.46:0/3186964307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b64009ae0 con 0x7f4b6c0acf30 2024-01-26T23:46:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.319+0000 7f4b7171c700 1 --2- 172.21.15.46:0/3186964307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b6c0cd250 0x7f4b6c158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.319+0000 7f4b70f1b700 1 --2- 172.21.15.46:0/3186964307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b6c0acf30 0x7f4b6c152020 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f4b6000c810 tx=0x7f4b60001100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.319+0000 7f4b68ff9700 1 -- 172.21.15.46:0/3186964307 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b60005a70 con 0x7f4b6c0acf30 2024-01-26T23:46:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.319+0000 7f4b68ff9700 1 -- 172.21.15.46:0/3186964307 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b60005bd0 con 0x7f4b6c0acf30 2024-01-26T23:46:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.319+0000 7f4b68ff9700 1 -- 172.21.15.46:0/3186964307 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b600070d0 con 0x7f4b6c0acf30 2024-01-26T23:46:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.319+0000 7f4b7317f700 1 -- 172.21.15.46:0/3186964307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b6c1688c0 con 0x7f4b6c0acf30 2024-01-26T23:46:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.319+0000 7f4b7317f700 1 -- 172.21.15.46:0/3186964307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b6c0b3b00 con 0x7f4b6c0acf30 2024-01-26T23:46:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.320+0000 7f4b7317f700 1 -- 172.21.15.46:0/3186964307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b6c04c710 con 0x7f4b6c0acf30 2024-01-26T23:46:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.321+0000 7f4b68ff9700 1 -- 172.21.15.46:0/3186964307 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f4b60007480 con 0x7f4b6c0acf30 2024-01-26T23:46:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.321+0000 7f4b68ff9700 1 --2- 172.21.15.46:0/3186964307 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4b50066ea0 0x7f4b50069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.321+0000 7f4b68ff9700 1 -- 172.21.15.46:0/3186964307 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f4b60083c60 con 0x7f4b6c0acf30 2024-01-26T23:46:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.321+0000 7f4b6bfff700 1 --2- 172.21.15.46:0/3186964307 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4b50066ea0 0x7f4b50069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.322+0000 7f4b6bfff700 1 --2- 172.21.15.46:0/3186964307 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4b50066ea0 0x7f4b50069350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4b58007fd0 tx=0x7f4b58007d30 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:32.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.326+0000 7f4b68ff9700 1 -- 172.21.15.46:0/3186964307 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4b60050ce0 con 0x7f4b6c0acf30 2024-01-26T23:46:32.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.534+0000 7f4b7317f700 1 -- 172.21.15.46:0/3186964307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b6c0b4330 con 0x7f4b6c0acf30 2024-01-26T23:46:32.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.535+0000 7f4b68ff9700 1 -- 172.21.15.46:0/3186964307 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16176 (secure 0 0 0) 0x7f4b60017070 con 0x7f4b6c0acf30 2024-01-26T23:46:32.536 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:32.536 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":111,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:46:31.208514+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:46:29.702709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-01-26T23:46:30.178825+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:46:32.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.537+0000 7f4b7317f700 1 -- 172.21.15.46:0/3186964307 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4b50066ea0 msgr2=0x7f4b50069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:32.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.537+0000 7f4b7317f700 1 --2- 172.21.15.46:0/3186964307 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4b50066ea0 0x7f4b50069350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4b58007fd0 tx=0x7f4b58007d30 comp rx=0 tx=0).stop 2024-01-26T23:46:32.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.537+0000 7f4b7317f700 1 -- 172.21.15.46:0/3186964307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b6c0acf30 msgr2=0x7f4b6c152020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:32.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.537+0000 7f4b7317f700 1 --2- 172.21.15.46:0/3186964307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b6c0acf30 0x7f4b6c152020 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f4b6000c810 tx=0x7f4b60001100 comp rx=0 tx=0).stop 2024-01-26T23:46:32.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.537+0000 7f4b7317f700 1 -- 172.21.15.46:0/3186964307 shutdown_connections 2024-01-26T23:46:32.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.537+0000 7f4b7317f700 1 --2- 172.21.15.46:0/3186964307 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f4b50066ea0 0x7f4b50069350 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:32.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.537+0000 7f4b7317f700 1 --2- 172.21.15.46:0/3186964307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b6c0acf30 0x7f4b6c152020 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:32.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.537+0000 7f4b7317f700 1 --2- 172.21.15.46:0/3186964307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b6c0cd250 0x7f4b6c158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:32.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.537+0000 7f4b7317f700 1 --2- 172.21.15.46:0/3186964307 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b6c0c3cf0 0x7f4b6c168380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:32.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.537+0000 7f4b7317f700 1 -- 172.21.15.46:0/3186964307 >> 172.21.15.46:0/3186964307 conn(0x7f4b6c097820 msgr2=0x7f4b6c15b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:32.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.537+0000 7f4b7317f700 1 -- 172.21.15.46:0/3186964307 shutdown_connections 2024-01-26T23:46:32.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.537+0000 7f4b7317f700 1 -- 172.21.15.46:0/3186964307 wait complete. 2024-01-26T23:46:32.553 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-26T23:46:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.690+0000 7fd6da5f8700 1 Processor -- start 2024-01-26T23:46:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.690+0000 7fd6da5f8700 1 -- start start 2024-01-26T23:46:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.691+0000 7fd6da5f8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6d4141200 0x7fd6d40c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.691+0000 7fd6da5f8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d40ccc00 con 0x7fd6d40caf30 2024-01-26T23:46:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.691+0000 7fd6da5f8700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d40cc310 con 0x7fd6d40c8da0 2024-01-26T23:46:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.691+0000 7fd6da5f8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d40cbbf0 con 0x7fd6d4141200 2024-01-26T23:46:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.691+0000 7fd6d8b95700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd6d40caf30 0x7fd6d40cb370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40854/0 (socket says 172.21.15.46:40854) 2024-01-26T23:46:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.691+0000 7fd6d8b95700 1 -- 172.21.15.46:0/3478070895 learned_addr learned my addr 172.21.15.46:0/3478070895 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.691+0000 7fd6d37fe700 1 --2- 172.21.15.46:0/3478070895 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6d4141200 0x7fd6d40c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.691+0000 7fd6d2ffd700 1 -- 172.21.15.46:0/3478070895 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685930772 0 0) 0x7fd6d40ccc00 con 0x7fd6d40caf30 2024-01-26T23:46:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.691+0000 7fd6d2ffd700 1 -- 172.21.15.46:0/3478070895 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6c0003570 con 0x7fd6d40caf30 2024-01-26T23:46:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.691+0000 7fd6d2ffd700 1 -- 172.21.15.46:0/3478070895 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718871377 0 0) 0x7fd6d40cc310 con 0x7fd6d40c8da0 2024-01-26T23:46:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.691+0000 7fd6d2ffd700 1 -- 172.21.15.46:0/3478070895 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6d40ccc00 con 0x7fd6d40c8da0 2024-01-26T23:46:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.692+0000 7fd6d37fe700 1 -- 172.21.15.46:0/3478070895 >> v1:172.21.15.106:6789/0 conn(0x7fd6d40c8da0 legacy=0x7fd6d4140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.692+0000 7fd6d37fe700 1 -- 172.21.15.46:0/3478070895 >> v1:172.21.15.46:6789/0 conn(0x7fd6d40caf30 legacy=0x7fd6d40cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.692+0000 7fd6d37fe700 1 -- 172.21.15.46:0/3478070895 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d40cc5b0 con 0x7fd6d4141200 2024-01-26T23:46:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.692+0000 7fd6d37fe700 1 --2- 172.21.15.46:0/3478070895 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6d4141200 0x7fd6d40c9170 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009910 tx=0x7fd6bc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3122098e9ebb6463 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.692+0000 7fd6d2ffd700 1 -- 172.21.15.46:0/3478070895 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc011630 con 0x7fd6d4141200 2024-01-26T23:46:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.692+0000 7fd6d2ffd700 1 -- 172.21.15.46:0/3478070895 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6bc011790 con 0x7fd6d4141200 2024-01-26T23:46:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.692+0000 7fd6d2ffd700 1 -- 172.21.15.46:0/3478070895 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc011b30 con 0x7fd6d4141200 2024-01-26T23:46:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.693+0000 7fd6da5f8700 1 -- 172.21.15.46:0/3478070895 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6d4141200 msgr2=0x7fd6d40c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.693+0000 7fd6da5f8700 1 --2- 172.21.15.46:0/3478070895 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6d4141200 0x7fd6d40c9170 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009910 tx=0x7fd6bc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.693+0000 7fd6da5f8700 1 -- 172.21.15.46:0/3478070895 shutdown_connections 2024-01-26T23:46:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.693+0000 7fd6da5f8700 1 --2- 172.21.15.46:0/3478070895 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6d4141200 0x7fd6d40c9170 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.693+0000 7fd6da5f8700 1 -- 172.21.15.46:0/3478070895 >> 172.21.15.46:0/3478070895 conn(0x7fd6d406bd90 msgr2=0x7fd6d406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.693+0000 7fd6da5f8700 1 -- 172.21.15.46:0/3478070895 shutdown_connections 2024-01-26T23:46:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.693+0000 7fd6da5f8700 1 -- 172.21.15.46:0/3478070895 wait complete. 2024-01-26T23:46:32.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.694+0000 7fd6da5f8700 1 Processor -- start 2024-01-26T23:46:32.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.694+0000 7fd6da5f8700 1 -- start start 2024-01-26T23:46:32.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.695+0000 7fd6da5f8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6d40c8da0 0x7fd6d413d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:32.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.695+0000 7fd6da5f8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6d40caf30 0x7fd6d4069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:32.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.695+0000 7fd6da5f8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6d4141200 0x7fd6d41570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:32.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.695+0000 7fd6da5f8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d406a3f0 con 0x7fd6d4141200 2024-01-26T23:46:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.695+0000 7fd6da5f8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d4159e90 con 0x7fd6d40c8da0 2024-01-26T23:46:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.695+0000 7fd6da5f8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d4159b60 con 0x7fd6d40caf30 2024-01-26T23:46:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.695+0000 7fd6d8b95700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6d4141200 0x7fd6d41570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.695+0000 7fd6d37fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6d40caf30 0x7fd6d4069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.695+0000 7fd6d8b95700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6d4141200 0x7fd6d41570c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33560/0 (socket says 172.21.15.46:33560) 2024-01-26T23:46:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.695+0000 7fd6d8b95700 1 -- 172.21.15.46:0/842632378 learned_addr learned my addr 172.21.15.46:0/842632378 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.695+0000 7fd6d3fff700 1 --2- 172.21.15.46:0/842632378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6d40c8da0 0x7fd6d413d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.695+0000 7fd6d37fe700 1 -- 172.21.15.46:0/842632378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6d40c8da0 msgr2=0x7fd6d413d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.695+0000 7fd6d37fe700 1 --2- 172.21.15.46:0/842632378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6d40c8da0 0x7fd6d413d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.695+0000 7fd6d37fe700 1 -- 172.21.15.46:0/842632378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6d4141200 msgr2=0x7fd6d41570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.695+0000 7fd6d37fe700 1 --2- 172.21.15.46:0/842632378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6d4141200 0x7fd6d41570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.695+0000 7fd6d37fe700 1 -- 172.21.15.46:0/842632378 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc009420 con 0x7fd6d40caf30 2024-01-26T23:46:32.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.696+0000 7fd6d37fe700 1 --2- 172.21.15.46:0/842632378 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6d40caf30 0x7fd6d4069450 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc016000 tx=0x7fd6bc017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:32.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.696+0000 7fd6cbfff700 1 -- 172.21.15.46:0/842632378 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc01f070 con 0x7fd6d40caf30 2024-01-26T23:46:32.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.696+0000 7fd6cbfff700 1 -- 172.21.15.46:0/842632378 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6bc002f70 con 0x7fd6d40caf30 2024-01-26T23:46:32.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.696+0000 7fd6da5f8700 1 -- 172.21.15.46:0/842632378 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d4069b50 con 0x7fd6d40caf30 2024-01-26T23:46:32.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.696+0000 7fd6cbfff700 1 -- 172.21.15.46:0/842632378 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc011900 con 0x7fd6d40caf30 2024-01-26T23:46:32.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.696+0000 7fd6da5f8700 1 -- 172.21.15.46:0/842632378 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d406a110 con 0x7fd6d40caf30 2024-01-26T23:46:32.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.697+0000 7fd6da5f8700 1 -- 172.21.15.46:0/842632378 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d415a040 con 0x7fd6d40caf30 2024-01-26T23:46:32.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.697+0000 7fd6cbfff700 1 -- 172.21.15.46:0/842632378 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fd6bc023070 con 0x7fd6d40caf30 2024-01-26T23:46:32.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.698+0000 7fd6cbfff700 1 --2- 172.21.15.46:0/842632378 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd6b4066ea0 0x7fd6b4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:32.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.698+0000 7fd6cbfff700 1 -- 172.21.15.46:0/842632378 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fd6bc0857d0 con 0x7fd6d40caf30 2024-01-26T23:46:32.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.698+0000 7fd6d3fff700 1 --2- 172.21.15.46:0/842632378 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd6b4066ea0 0x7fd6b4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:32.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.698+0000 7fd6d3fff700 1 --2- 172.21.15.46:0/842632378 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd6b4066ea0 0x7fd6b4069350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4007fd0 tx=0x7fd6c4007d50 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:32.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.702+0000 7fd6cbfff700 1 -- 172.21.15.46:0/842632378 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd6d415a040 con 0x7fd6d40caf30 2024-01-26T23:46:32.960 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:32.959+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:32.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:32.981+0000 7fd6da5f8700 1 -- 172.21.15.46:0/842632378 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd6d415a000 con 0x7fd6d40caf30 2024-01-26T23:46:33.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-01-26T23:46:33.707 INFO:tasks.ceph.mds.d.smithi106.stderr:2024-01-26T23:46:33.706+0000 7f9fb9a8e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-26T23:46:33.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.773+0000 7fd6cbfff700 1 -- 172.21.15.46:0/842632378 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v53) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd6bc02a0b0 con 0x7fd6d40caf30 2024-01-26T23:46:33.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.776+0000 7fd6da5f8700 1 -- 172.21.15.46:0/842632378 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd6b4066ea0 msgr2=0x7fd6b4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:33.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.776+0000 7fd6da5f8700 1 --2- 172.21.15.46:0/842632378 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd6b4066ea0 0x7fd6b4069350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4007fd0 tx=0x7fd6c4007d50 comp rx=0 tx=0).stop 2024-01-26T23:46:33.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.776+0000 7fd6da5f8700 1 -- 172.21.15.46:0/842632378 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6d40caf30 msgr2=0x7fd6d4069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:33.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.776+0000 7fd6da5f8700 1 --2- 172.21.15.46:0/842632378 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6d40caf30 0x7fd6d4069450 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc016000 tx=0x7fd6bc017000 comp rx=0 tx=0).stop 2024-01-26T23:46:33.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.777+0000 7fd6da5f8700 1 -- 172.21.15.46:0/842632378 shutdown_connections 2024-01-26T23:46:33.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.777+0000 7fd6da5f8700 1 --2- 172.21.15.46:0/842632378 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd6b4066ea0 0x7fd6b4069350 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:33.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.777+0000 7fd6da5f8700 1 --2- 172.21.15.46:0/842632378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6d4141200 0x7fd6d41570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:33.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.777+0000 7fd6da5f8700 1 --2- 172.21.15.46:0/842632378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6d40c8da0 0x7fd6d413d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:33.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.777+0000 7fd6da5f8700 1 --2- 172.21.15.46:0/842632378 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6d40caf30 0x7fd6d4069450 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:33.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.777+0000 7fd6da5f8700 1 -- 172.21.15.46:0/842632378 >> 172.21.15.46:0/842632378 conn(0x7fd6d406bd90 msgr2=0x7fd6d413f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:33.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.777+0000 7fd6da5f8700 1 -- 172.21.15.46:0/842632378 shutdown_connections 2024-01-26T23:46:33.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.777+0000 7fd6da5f8700 1 -- 172.21.15.46:0/842632378 wait complete. 2024-01-26T23:46:33.792 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:33.791+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:33.794 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-26T23:46:33.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.934+0000 7f2c14154700 1 Processor -- start 2024-01-26T23:46:33.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.934+0000 7f2c14154700 1 -- start start 2024-01-26T23:46:33.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.935+0000 7f2c14154700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c0c068b00 0x7f2c0c068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:33.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.935+0000 7f2c14154700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c0c0a9a80 con 0x7f2c0c06add0 2024-01-26T23:46:33.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.935+0000 7f2c14154700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c0c06aab0 con 0x7f2c0c06a660 2024-01-26T23:46:33.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.935+0000 7f2c14154700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c0c15a680 con 0x7f2c0c068b00 2024-01-26T23:46:33.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.935+0000 7f2c11ef0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c0c068b00 0x7f2c0c068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:33.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.935+0000 7f2c126f1700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2c0c06add0 0x7f2c0c0a9f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40868/0 (socket says 172.21.15.46:40868) 2024-01-26T23:46:33.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.935+0000 7f2c126f1700 1 -- 172.21.15.46:0/3237422408 learned_addr learned my addr 172.21.15.46:0/3237422408 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:33.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.935+0000 7f2c11ef0700 1 -- 172.21.15.46:0/3237422408 >> v1:172.21.15.106:6789/0 conn(0x7f2c0c06a660 legacy=0x7f2c0c159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:33.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.935+0000 7f2c11ef0700 1 -- 172.21.15.46:0/3237422408 >> v1:172.21.15.46:6789/0 conn(0x7f2c0c06add0 legacy=0x7f2c0c0a9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:33.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.935+0000 7f2c10eee700 1 -- 172.21.15.46:0/3237422408 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771222821 0 0) 0x7f2c0c0a9a80 con 0x7f2c0c06add0 2024-01-26T23:46:33.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.935+0000 7f2c11ef0700 1 -- 172.21.15.46:0/3237422408 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0c15a960 con 0x7f2c0c068b00 2024-01-26T23:46:33.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.936+0000 7f2c11ef0700 1 --2- 172.21.15.46:0/3237422408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0c068b00 0x7f2c0c068ed0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2c080098d0 tx=0x7f2c08009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998d6cf841a71262 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:33.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.936+0000 7f2c10eee700 1 -- 172.21.15.46:0/3237422408 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c08009e70 con 0x7f2c0c068b00 2024-01-26T23:46:33.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.936+0000 7f2c10eee700 1 -- 172.21.15.46:0/3237422408 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c08005c10 con 0x7f2c0c068b00 2024-01-26T23:46:33.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.936+0000 7f2c10eee700 1 -- 172.21.15.46:0/3237422408 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0800b610 con 0x7f2c0c068b00 2024-01-26T23:46:33.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.936+0000 7f2c14154700 1 -- 172.21.15.46:0/3237422408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0c068b00 msgr2=0x7f2c0c068ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:33.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.936+0000 7f2c14154700 1 --2- 172.21.15.46:0/3237422408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0c068b00 0x7f2c0c068ed0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2c080098d0 tx=0x7f2c08009120 comp rx=0 tx=0).stop 2024-01-26T23:46:33.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.936+0000 7f2c14154700 1 -- 172.21.15.46:0/3237422408 shutdown_connections 2024-01-26T23:46:33.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.936+0000 7f2c14154700 1 --2- 172.21.15.46:0/3237422408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0c068b00 0x7f2c0c068ed0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:33.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.936+0000 7f2c14154700 1 -- 172.21.15.46:0/3237422408 >> 172.21.15.46:0/3237422408 conn(0x7f2c0c0ca100 msgr2=0x7f2c0c06e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:33.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.937+0000 7f2c14154700 1 -- 172.21.15.46:0/3237422408 shutdown_connections 2024-01-26T23:46:33.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.937+0000 7f2c14154700 1 -- 172.21.15.46:0/3237422408 wait complete. 2024-01-26T23:46:33.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.938+0000 7f2c14154700 1 Processor -- start 2024-01-26T23:46:33.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.938+0000 7f2c14154700 1 -- start start 2024-01-26T23:46:33.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c14154700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2c0c068b00 0x7f2c0c0ab940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:33.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c14154700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c0c06a660 0x7f2c0c156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:33.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c14154700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0c06add0 0x7f2c0c0c5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:33.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c14154700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c0c6590 con 0x7f2c0c06add0 2024-01-26T23:46:33.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c14154700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c0c6870 con 0x7f2c0c068b00 2024-01-26T23:46:33.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c14154700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c0c6b50 con 0x7f2c0c06a660 2024-01-26T23:46:33.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c126f1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0c06add0 0x7f2c0c0c5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:33.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c11ef0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2c0c068b00 0x7f2c0c0ab940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:33.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c126f1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0c06add0 0x7f2c0c0c5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33576/0 (socket says 172.21.15.46:33576) 2024-01-26T23:46:33.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c126f1700 1 -- 172.21.15.46:0/2409921386 learned_addr learned my addr 172.21.15.46:0/2409921386 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:33.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c116ef700 1 --2- 172.21.15.46:0/2409921386 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c0c06a660 0x7f2c0c156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:33.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c11ef0700 1 -- 172.21.15.46:0/2409921386 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c0c06a660 msgr2=0x7f2c0c156140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:33.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c11ef0700 1 --2- 172.21.15.46:0/2409921386 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c0c06a660 0x7f2c0c156140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:33.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c11ef0700 1 -- 172.21.15.46:0/2409921386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0c06add0 msgr2=0x7f2c0c0c5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:33.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c11ef0700 1 --2- 172.21.15.46:0/2409921386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0c06add0 0x7f2c0c0c5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:33.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c11ef0700 1 -- 172.21.15.46:0/2409921386 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c080093c0 con 0x7f2c0c068b00 2024-01-26T23:46:33.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c126f1700 1 --2- 172.21.15.46:0/2409921386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0c06add0 0x7f2c0c0c5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:33.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.939+0000 7f2c11ef0700 1 --2- 172.21.15.46:0/2409921386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2c0c068b00 0x7f2c0c0ab940 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2c08014000 tx=0x7f2c08015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:33.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.940+0000 7f2bfdffb700 1 -- 172.21.15.46:0/2409921386 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c080064d0 con 0x7f2c0c068b00 2024-01-26T23:46:33.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.940+0000 7f2bfdffb700 1 -- 172.21.15.46:0/2409921386 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c08006630 con 0x7f2c0c068b00 2024-01-26T23:46:33.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.940+0000 7f2c14154700 1 -- 172.21.15.46:0/2409921386 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c0c0c6e30 con 0x7f2c0c068b00 2024-01-26T23:46:33.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.940+0000 7f2bfdffb700 1 -- 172.21.15.46:0/2409921386 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0801c430 con 0x7f2c0c068b00 2024-01-26T23:46:33.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.940+0000 7f2c14154700 1 -- 172.21.15.46:0/2409921386 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c0c0abe80 con 0x7f2c0c068b00 2024-01-26T23:46:33.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.941+0000 7f2c14154700 1 -- 172.21.15.46:0/2409921386 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c0c0c7330 con 0x7f2c0c068b00 2024-01-26T23:46:33.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.941+0000 7f2bfdffb700 1 -- 172.21.15.46:0/2409921386 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f2c08023070 con 0x7f2c0c068b00 2024-01-26T23:46:33.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.942+0000 7f2bfdffb700 1 --2- 172.21.15.46:0/2409921386 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2bf0066ea0 0x7f2bf0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:33.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.942+0000 7f2bfdffb700 1 -- 172.21.15.46:0/2409921386 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f2c080864f0 con 0x7f2c0c068b00 2024-01-26T23:46:33.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.942+0000 7f2c116ef700 1 --2- 172.21.15.46:0/2409921386 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2bf0066ea0 0x7f2bf0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:33.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.942+0000 7f2c116ef700 1 --2- 172.21.15.46:0/2409921386 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2bf0066ea0 0x7f2bf0069350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0cc7b0 tx=0x7f2c00007ea0 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:33.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:33.946+0000 7f2bfdffb700 1 -- 172.21.15.46:0/2409921386 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2c08053570 con 0x7f2c0c068b00 2024-01-26T23:46:34.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.203+0000 7f2c14154700 1 -- 172.21.15.46:0/2409921386 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2c0c0c69b0 con 0x7f2c0c068b00 2024-01-26T23:46:34.625 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:34.624+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:34.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.778+0000 7f2bfdffb700 1 -- 172.21.15.46:0/2409921386 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v54) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2c0c0c69b0 con 0x7f2c0c068b00 2024-01-26T23:46:34.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.781+0000 7f2c14154700 1 -- 172.21.15.46:0/2409921386 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2bf0066ea0 msgr2=0x7f2bf0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:34.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.781+0000 7f2c14154700 1 --2- 172.21.15.46:0/2409921386 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2bf0066ea0 0x7f2bf0069350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0cc7b0 tx=0x7f2c00007ea0 comp rx=0 tx=0).stop 2024-01-26T23:46:34.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.781+0000 7f2c14154700 1 -- 172.21.15.46:0/2409921386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2c0c068b00 msgr2=0x7f2c0c0ab940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:34.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.781+0000 7f2c14154700 1 --2- 172.21.15.46:0/2409921386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2c0c068b00 0x7f2c0c0ab940 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2c08014000 tx=0x7f2c08015000 comp rx=0 tx=0).stop 2024-01-26T23:46:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.781+0000 7f2c14154700 1 -- 172.21.15.46:0/2409921386 shutdown_connections 2024-01-26T23:46:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.781+0000 7f2c14154700 1 --2- 172.21.15.46:0/2409921386 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2bf0066ea0 0x7f2bf0069350 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.781+0000 7f2c14154700 1 --2- 172.21.15.46:0/2409921386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0c06add0 0x7f2c0c0c5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.781+0000 7f2c14154700 1 --2- 172.21.15.46:0/2409921386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2c0c068b00 0x7f2c0c0ab940 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.781+0000 7f2c14154700 1 --2- 172.21.15.46:0/2409921386 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c0c06a660 0x7f2c0c156140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.781+0000 7f2c14154700 1 -- 172.21.15.46:0/2409921386 >> 172.21.15.46:0/2409921386 conn(0x7f2c0c0ca100 msgr2=0x7f2c0c157030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.781+0000 7f2c14154700 1 -- 172.21.15.46:0/2409921386 shutdown_connections 2024-01-26T23:46:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.781+0000 7f2c14154700 1 -- 172.21.15.46:0/2409921386 wait complete. 2024-01-26T23:46:34.798 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:46:34.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.936+0000 7f1402f78700 1 Processor -- start 2024-01-26T23:46:34.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.937+0000 7f1402f78700 1 -- start start 2024-01-26T23:46:34.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.937+0000 7f1402f78700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13fc15ce40 0x7f13fc0d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:34.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.937+0000 7f1402f78700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f13fc15d210 0x7f13fc0cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:34.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.937+0000 7f1402f78700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13fc0d1390 0x7f13fc0cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.937+0000 7f1402f78700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13fc0cff70 con 0x7f13fc0d1390 2024-01-26T23:46:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.937+0000 7f1402f78700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13fc0cf400 con 0x7f13fc15d210 2024-01-26T23:46:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.937+0000 7f1402f78700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13fc0cf660 con 0x7f13fc15ce40 2024-01-26T23:46:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.938+0000 7f1401515700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13fc0d1390 0x7f13fc0cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.938+0000 7f1401515700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13fc0d1390 0x7f13fc0cbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33590/0 (socket says 172.21.15.46:33590) 2024-01-26T23:46:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.938+0000 7f1401515700 1 -- 172.21.15.46:0/3349650315 learned_addr learned my addr 172.21.15.46:0/3349650315 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.938+0000 7f1400d14700 1 --2- 172.21.15.46:0/3349650315 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13fc15ce40 0x7f13fc0d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.938+0000 7f13fbfff700 1 --2- 172.21.15.46:0/3349650315 >> v2:172.21.15.106:3300/0 conn(0x7f13fc15d210 0x7f13fc0cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.938+0000 7f1401515700 1 -- 172.21.15.46:0/3349650315 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13fc15ce40 msgr2=0x7f13fc0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.938+0000 7f1401515700 1 --2- 172.21.15.46:0/3349650315 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13fc15ce40 0x7f13fc0d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.938+0000 7f1401515700 1 -- 172.21.15.46:0/3349650315 >> v2:172.21.15.106:3300/0 conn(0x7f13fc15d210 msgr2=0x7f13fc0cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.938+0000 7f1401515700 1 --2- 172.21.15.46:0/3349650315 >> v2:172.21.15.106:3300/0 conn(0x7f13fc15d210 0x7f13fc0cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.938+0000 7f1401515700 1 -- 172.21.15.46:0/3349650315 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13fc0d0500 con 0x7f13fc0d1390 2024-01-26T23:46:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.938+0000 7f1400d14700 1 --2- 172.21.15.46:0/3349650315 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13fc15ce40 0x7f13fc0d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:46:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.938+0000 7f1401515700 1 --2- 172.21.15.46:0/3349650315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13fc0d1390 0x7f13fc0cbff0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f13fc088cb0 tx=0x7f13f4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18993ffb080a4a70 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.938+0000 7f13fb7fe700 1 -- 172.21.15.46:0/3349650315 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f400e070 con 0x7f13fc0d1390 2024-01-26T23:46:34.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.938+0000 7f13fb7fe700 1 -- 172.21.15.46:0/3349650315 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f400ba50 con 0x7f13fc0d1390 2024-01-26T23:46:34.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.938+0000 7f13fb7fe700 1 -- 172.21.15.46:0/3349650315 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f40092e0 con 0x7f13fc0d1390 2024-01-26T23:46:34.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.939+0000 7f1402f78700 1 -- 172.21.15.46:0/3349650315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13fc0d1390 msgr2=0x7f13fc0cbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:34.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.939+0000 7f1402f78700 1 --2- 172.21.15.46:0/3349650315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13fc0d1390 0x7f13fc0cbff0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f13fc088cb0 tx=0x7f13f4009840 comp rx=0 tx=0).stop 2024-01-26T23:46:34.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.939+0000 7f1402f78700 1 -- 172.21.15.46:0/3349650315 shutdown_connections 2024-01-26T23:46:34.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.939+0000 7f1402f78700 1 --2- 172.21.15.46:0/3349650315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13fc0d1390 0x7f13fc0cbff0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:34.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.939+0000 7f1402f78700 1 --2- 172.21.15.46:0/3349650315 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13fc15ce40 0x7f13fc0d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:34.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.939+0000 7f1402f78700 1 --2- 172.21.15.46:0/3349650315 >> v2:172.21.15.106:3300/0 conn(0x7f13fc15d210 0x7f13fc0cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:34.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.939+0000 7f1402f78700 1 -- 172.21.15.46:0/3349650315 >> 172.21.15.46:0/3349650315 conn(0x7f13fc06f840 msgr2=0x7f13fc0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:34.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.939+0000 7f1402f78700 1 -- 172.21.15.46:0/3349650315 shutdown_connections 2024-01-26T23:46:34.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.939+0000 7f1402f78700 1 -- 172.21.15.46:0/3349650315 wait complete. 2024-01-26T23:46:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.940+0000 7f1402f78700 1 Processor -- start 2024-01-26T23:46:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.941+0000 7f1402f78700 1 -- start start 2024-01-26T23:46:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.941+0000 7f1402f78700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13fc0d1390 0x7f13fc1547f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.941+0000 7f1402f78700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13fc15ce40 0x7f13fc144da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f1402f78700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13fc15d210 0x7f13fc0c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f1402f78700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13fc154d30 con 0x7f13fc0d1390 2024-01-26T23:46:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f1402f78700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13fc142be0 con 0x7f13fc15d210 2024-01-26T23:46:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f1402f78700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13fc142ec0 con 0x7f13fc15ce40 2024-01-26T23:46:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f1400d14700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13fc0d1390 0x7f13fc1547f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f1400d14700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13fc0d1390 0x7f13fc1547f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33604/0 (socket says 172.21.15.46:33604) 2024-01-26T23:46:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f1400d14700 1 -- 172.21.15.46:0/2661706289 learned_addr learned my addr 172.21.15.46:0/2661706289 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f13fbfff700 1 --2- 172.21.15.46:0/2661706289 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13fc15ce40 0x7f13fc144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f1401515700 1 --2- 172.21.15.46:0/2661706289 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13fc15d210 0x7f13fc0c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:34.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f1400d14700 1 -- 172.21.15.46:0/2661706289 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13fc15ce40 msgr2=0x7f13fc144da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:34.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f1400d14700 1 --2- 172.21.15.46:0/2661706289 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13fc15ce40 0x7f13fc144da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:34.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f1400d14700 1 -- 172.21.15.46:0/2661706289 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13fc15d210 msgr2=0x7f13fc0c6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:34.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f1400d14700 1 --2- 172.21.15.46:0/2661706289 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13fc15d210 0x7f13fc0c6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:34.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f1400d14700 1 -- 172.21.15.46:0/2661706289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f4009ae0 con 0x7f13fc0d1390 2024-01-26T23:46:34.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f13fbfff700 1 --2- 172.21.15.46:0/2661706289 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13fc15ce40 0x7f13fc144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:34.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f1400d14700 1 --2- 172.21.15.46:0/2661706289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13fc0d1390 0x7f13fc1547f0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f13f000e920 tx=0x7f13f0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:34.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f13f8ff9700 1 -- 172.21.15.46:0/2661706289 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0005a00 con 0x7f13fc0d1390 2024-01-26T23:46:34.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f13f8ff9700 1 -- 172.21.15.46:0/2661706289 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f0005b60 con 0x7f13fc0d1390 2024-01-26T23:46:34.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f1402f78700 1 -- 172.21.15.46:0/2661706289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13fc1431a0 con 0x7f13fc0d1390 2024-01-26T23:46:34.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.942+0000 7f13f8ff9700 1 -- 172.21.15.46:0/2661706289 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0006020 con 0x7f13fc0d1390 2024-01-26T23:46:34.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.943+0000 7f1402f78700 1 -- 172.21.15.46:0/2661706289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13fc1437e0 con 0x7f13fc0d1390 2024-01-26T23:46:34.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.943+0000 7f1402f78700 1 -- 172.21.15.46:0/2661706289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13fc143ad0 con 0x7f13fc0d1390 2024-01-26T23:46:34.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.944+0000 7f13f8ff9700 1 -- 172.21.15.46:0/2661706289 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f13f00063d0 con 0x7f13fc0d1390 2024-01-26T23:46:34.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.944+0000 7f13f8ff9700 1 --2- 172.21.15.46:0/2661706289 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f13e0066ea0 0x7f13e0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:34.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.945+0000 7f13f8ff9700 1 -- 172.21.15.46:0/2661706289 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f13f00846e0 con 0x7f13fc0d1390 2024-01-26T23:46:34.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.945+0000 7f13fbfff700 1 --2- 172.21.15.46:0/2661706289 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f13e0066ea0 0x7f13e0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:34.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.945+0000 7f13fbfff700 1 --2- 172.21.15.46:0/2661706289 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f13e0066ea0 0x7f13e0069350 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f13fc141750 tx=0x7f13e8008040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:34.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:34.949+0000 7f13f8ff9700 1 -- 172.21.15.46:0/2661706289 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f13f0051810 con 0x7f13fc0d1390 2024-01-26T23:46:35.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.203+0000 7f1402f78700 1 -- 172.21.15.46:0/2661706289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13fc0cce00 con 0x7f13fc0d1390 2024-01-26T23:46:35.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.203+0000 7f13f8ff9700 1 -- 172.21.15.46:0/2661706289 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f13f0054e30 con 0x7f13fc0d1390 2024-01-26T23:46:35.205 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:35.205 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6518,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/4188139264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":4188139264},{"type":"v1","addr":"172.21.15.106:6836","nonce":4188139264}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":48},{"gid":6527,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/221301328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":221301328},{"type":"v1","addr":"172.21.15.46:6833","nonce":221301328}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":6530,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1531399912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":1531399912},{"type":"v1","addr":"172.21.15.46:6835","nonce":1531399912}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:46:31.208843+0000","modified":"2024-01-26T23:46:34.775841+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7084":{"gid":7084,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6837/1643596655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1643596655},{"type":"v1","addr":"172.21.15.106:6837","nonce":1643596655}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2024-01-26T23:46:35.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.206+0000 7f1402f78700 1 -- 172.21.15.46:0/2661706289 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f13e0066ea0 msgr2=0x7f13e0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:35.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.206+0000 7f1402f78700 1 --2- 172.21.15.46:0/2661706289 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f13e0066ea0 0x7f13e0069350 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f13fc141750 tx=0x7f13e8008040 comp rx=0 tx=0).stop 2024-01-26T23:46:35.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.206+0000 7f1402f78700 1 -- 172.21.15.46:0/2661706289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13fc0d1390 msgr2=0x7f13fc1547f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:35.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.206+0000 7f1402f78700 1 --2- 172.21.15.46:0/2661706289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13fc0d1390 0x7f13fc1547f0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f13f000e920 tx=0x7f13f0008040 comp rx=0 tx=0).stop 2024-01-26T23:46:35.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.206+0000 7f1402f78700 1 -- 172.21.15.46:0/2661706289 shutdown_connections 2024-01-26T23:46:35.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.206+0000 7f1402f78700 1 --2- 172.21.15.46:0/2661706289 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f13e0066ea0 0x7f13e0069350 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.206+0000 7f1402f78700 1 --2- 172.21.15.46:0/2661706289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13fc0d1390 0x7f13fc1547f0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.206+0000 7f1402f78700 1 --2- 172.21.15.46:0/2661706289 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13fc15d210 0x7f13fc0c6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.206+0000 7f1402f78700 1 --2- 172.21.15.46:0/2661706289 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13fc15ce40 0x7f13fc144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.206+0000 7f1402f78700 1 -- 172.21.15.46:0/2661706289 >> 172.21.15.46:0/2661706289 conn(0x7f13fc06f840 msgr2=0x7f13fc0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:35.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.206+0000 7f1402f78700 1 -- 172.21.15.46:0/2661706289 shutdown_connections 2024-01-26T23:46:35.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.206+0000 7f1402f78700 1 -- 172.21.15.46:0/2661706289 wait complete. 2024-01-26T23:46:35.209 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 54 2024-01-26T23:46:35.221 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:46:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.352+0000 7fd19f3d2700 1 Processor -- start 2024-01-26T23:46:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.353+0000 7fd19f3d2700 1 -- start start 2024-01-26T23:46:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.354+0000 7fd19f3d2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd198157920 0x7fd1980b2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.354+0000 7fd19f3d2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd1980b2a60 0x7fd1980cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.354+0000 7fd19f3d2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1980cc550 con 0x7fd198157550 2024-01-26T23:46:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.354+0000 7fd19f3d2700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1980ca020 con 0x7fd1980b2a60 2024-01-26T23:46:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.354+0000 7fd19f3d2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1980b2e90 con 0x7fd198157920 2024-01-26T23:46:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.354+0000 7fd19c96d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd198157920 0x7fd1980b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.354+0000 7fd19d96f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd1980b2a60 0x7fd1980cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.354+0000 7fd19d16e700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fd198157550 0x7fd19806f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:45754/0 (socket says 172.21.15.46:45754) 2024-01-26T23:46:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.354+0000 7fd19d16e700 1 -- 172.21.15.46:0/3363766487 learned_addr learned my addr 172.21.15.46:0/3363766487 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.354+0000 7fd19c96d700 1 -- 172.21.15.46:0/3363766487 >> v2:172.21.15.106:3300/0 conn(0x7fd1980b2a60 msgr2=0x7fd1980cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.354+0000 7fd18ffff700 1 -- 172.21.15.46:0/3363766487 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230343060 0 0) 0x7fd1980cc550 con 0x7fd198157550 2024-01-26T23:46:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.354+0000 7fd19c96d700 1 --2- 172.21.15.46:0/3363766487 >> v2:172.21.15.106:3300/0 conn(0x7fd1980b2a60 0x7fd1980cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.354+0000 7fd19c96d700 1 -- 172.21.15.46:0/3363766487 >> v1:172.21.15.106:6789/0 conn(0x7fd198157550 legacy=0x7fd19806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.354+0000 7fd19c96d700 1 -- 172.21.15.46:0/3363766487 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1980cb6a0 con 0x7fd198157920 2024-01-26T23:46:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.355+0000 7fd19c96d700 1 --2- 172.21.15.46:0/3363766487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd198157920 0x7fd1980b2520 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd1880098e0 tx=0x7fd1880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31f818e766765ff9 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.355+0000 7fd18ffff700 1 -- 172.21.15.46:0/3363766487 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18800b640 con 0x7fd198157920 2024-01-26T23:46:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.355+0000 7fd18ffff700 1 -- 172.21.15.46:0/3363766487 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18800b7a0 con 0x7fd198157920 2024-01-26T23:46:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.355+0000 7fd18ffff700 1 -- 172.21.15.46:0/3363766487 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18800bb40 con 0x7fd198157920 2024-01-26T23:46:35.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.355+0000 7fd19f3d2700 1 -- 172.21.15.46:0/3363766487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd198157920 msgr2=0x7fd1980b2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:35.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.355+0000 7fd19f3d2700 1 --2- 172.21.15.46:0/3363766487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd198157920 0x7fd1980b2520 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd1880098e0 tx=0x7fd1880091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.355+0000 7fd19f3d2700 1 -- 172.21.15.46:0/3363766487 shutdown_connections 2024-01-26T23:46:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.355+0000 7fd19f3d2700 1 --2- 172.21.15.46:0/3363766487 >> v2:172.21.15.106:3300/0 conn(0x7fd1980b2a60 0x7fd1980cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.355+0000 7fd19f3d2700 1 --2- 172.21.15.46:0/3363766487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd198157920 0x7fd1980b2520 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.355+0000 7fd19f3d2700 1 -- 172.21.15.46:0/3363766487 >> 172.21.15.46:0/3363766487 conn(0x7fd198069510 msgr2=0x7fd1980aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.355+0000 7fd19f3d2700 1 -- 172.21.15.46:0/3363766487 shutdown_connections 2024-01-26T23:46:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.355+0000 7fd19f3d2700 1 -- 172.21.15.46:0/3363766487 wait complete. 2024-01-26T23:46:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.357+0000 7fd19f3d2700 1 Processor -- start 2024-01-26T23:46:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.357+0000 7fd19f3d2700 1 -- start start 2024-01-26T23:46:35.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.358+0000 7fd19f3d2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd1980b2a60 0x7fd198159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:35.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.358+0000 7fd19f3d2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd198157550 0x7fd19816af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:35.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.358+0000 7fd19f3d2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd198157920 0x7fd1981667a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:35.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.358+0000 7fd19f3d2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19815cd70 con 0x7fd198157920 2024-01-26T23:46:35.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.358+0000 7fd19f3d2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19816b440 con 0x7fd1980b2a60 2024-01-26T23:46:35.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.358+0000 7fd19f3d2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd198167050 con 0x7fd198157550 2024-01-26T23:46:35.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.358+0000 7fd19d16e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd1980b2a60 0x7fd198159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:35.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.358+0000 7fd19c96d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd198157550 0x7fd19816af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:35.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.358+0000 7fd19d16e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd1980b2a60 0x7fd198159f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:37316/0 (socket says 172.21.15.46:37316) 2024-01-26T23:46:35.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.358+0000 7fd19c96d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd198157550 0x7fd19816af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:46998/0 (socket says 172.21.15.46:46998) 2024-01-26T23:46:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.358+0000 7fd19d16e700 1 -- 172.21.15.46:0/2333784129 learned_addr learned my addr 172.21.15.46:0/2333784129 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.358+0000 7fd19d96f700 1 --2- 172.21.15.46:0/2333784129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd198157920 0x7fd1981667a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.358+0000 7fd19c96d700 1 -- 172.21.15.46:0/2333784129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd1980b2a60 msgr2=0x7fd198159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.358+0000 7fd19c96d700 1 --2- 172.21.15.46:0/2333784129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd1980b2a60 0x7fd198159f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.358+0000 7fd19c96d700 1 -- 172.21.15.46:0/2333784129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd198157920 msgr2=0x7fd1981667a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.358+0000 7fd19c96d700 1 --2- 172.21.15.46:0/2333784129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd198157920 0x7fd1981667a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.358+0000 7fd19c96d700 1 -- 172.21.15.46:0/2333784129 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd188009420 con 0x7fd198157550 2024-01-26T23:46:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.359+0000 7fd19c96d700 1 --2- 172.21.15.46:0/2333784129 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd198157550 0x7fd19816af00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd188016000 tx=0x7fd188017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.359+0000 7fd18d7fa700 1 -- 172.21.15.46:0/2333784129 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd188006af0 con 0x7fd198157550 2024-01-26T23:46:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.359+0000 7fd18d7fa700 1 -- 172.21.15.46:0/2333784129 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18800b910 con 0x7fd198157550 2024-01-26T23:46:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.359+0000 7fd18d7fa700 1 -- 172.21.15.46:0/2333784129 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd188011650 con 0x7fd198157550 2024-01-26T23:46:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.359+0000 7fd19f3d2700 1 -- 172.21.15.46:0/2333784129 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd198167330 con 0x7fd198157550 2024-01-26T23:46:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.359+0000 7fd19f3d2700 1 -- 172.21.15.46:0/2333784129 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1981678f0 con 0x7fd198157550 2024-01-26T23:46:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.360+0000 7fd19f3d2700 1 -- 172.21.15.46:0/2333784129 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd198167ae0 con 0x7fd198157550 2024-01-26T23:46:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.360+0000 7fd18d7fa700 1 -- 172.21.15.46:0/2333784129 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fd188011a80 con 0x7fd198157550 2024-01-26T23:46:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.361+0000 7fd18d7fa700 1 --2- 172.21.15.46:0/2333784129 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd17c066ea0 0x7fd17c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.361+0000 7fd18d7fa700 1 -- 172.21.15.46:0/2333784129 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fd188085d70 con 0x7fd198157550 2024-01-26T23:46:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.361+0000 7fd19d16e700 1 --2- 172.21.15.46:0/2333784129 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd17c066ea0 0x7fd17c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:35.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.361+0000 7fd19d16e700 1 --2- 172.21.15.46:0/2333784129 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd17c066ea0 0x7fd17c069350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd194007fd0 tx=0x7fd194007df0 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:35.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.365+0000 7fd18d7fa700 1 -- 172.21.15.46:0/2333784129 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd188052df0 con 0x7fd198157550 2024-01-26T23:46:35.469 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:35.468+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:35.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.578+0000 7fd19f3d2700 1 -- 172.21.15.46:0/2333784129 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd19804c710 con 0x7fd198157550 2024-01-26T23:46:35.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.580+0000 7fd18d7fa700 1 -- 172.21.15.46:0/2333784129 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16163 (secure 0 0 0) 0x7fd188015090 con 0x7fd198157550 2024-01-26T23:46:35.581 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:35.581 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":112,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:46:32.645638+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:46:29.702709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-01-26T23:46:30.178825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:46:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.581+0000 7fd19f3d2700 1 -- 172.21.15.46:0/2333784129 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd17c066ea0 msgr2=0x7fd17c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.581+0000 7fd19f3d2700 1 --2- 172.21.15.46:0/2333784129 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd17c066ea0 0x7fd17c069350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd194007fd0 tx=0x7fd194007df0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.581+0000 7fd19f3d2700 1 -- 172.21.15.46:0/2333784129 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd198157550 msgr2=0x7fd19816af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.581+0000 7fd19f3d2700 1 --2- 172.21.15.46:0/2333784129 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd198157550 0x7fd19816af00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd188016000 tx=0x7fd188017000 comp rx=0 tx=0).stop 2024-01-26T23:46:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.582+0000 7fd19f3d2700 1 -- 172.21.15.46:0/2333784129 shutdown_connections 2024-01-26T23:46:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.582+0000 7fd19f3d2700 1 --2- 172.21.15.46:0/2333784129 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd17c066ea0 0x7fd17c069350 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.582+0000 7fd19f3d2700 1 --2- 172.21.15.46:0/2333784129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd198157920 0x7fd1981667a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.582+0000 7fd19f3d2700 1 --2- 172.21.15.46:0/2333784129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd1980b2a60 0x7fd198159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.582+0000 7fd19f3d2700 1 --2- 172.21.15.46:0/2333784129 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd198157550 0x7fd19816af00 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.582+0000 7fd19f3d2700 1 -- 172.21.15.46:0/2333784129 >> 172.21.15.46:0/2333784129 conn(0x7fd198069510 msgr2=0x7fd1980cc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.582+0000 7fd19f3d2700 1 -- 172.21.15.46:0/2333784129 shutdown_connections 2024-01-26T23:46:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.582+0000 7fd19f3d2700 1 -- 172.21.15.46:0/2333784129 wait complete. 2024-01-26T23:46:35.597 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-26T23:46:35.597 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:46:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.735+0000 7f26d047c700 1 Processor -- start 2024-01-26T23:46:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.735+0000 7f26d047c700 1 -- start start 2024-01-26T23:46:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.736+0000 7f26d047c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f26c8159b30 0x7f26c80b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.736+0000 7f26d047c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c806f940 con 0x7f26c8159760 2024-01-26T23:46:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.736+0000 7f26d047c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c80b1530 con 0x7f26c80b10f0 2024-01-26T23:46:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.736+0000 7f26d047c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26c80b3230 con 0x7f26c8159b30 2024-01-26T23:46:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.736+0000 7f26cda17700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f26c8159b30 0x7f26c80b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.736+0000 7f26ce218700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f26c8159760 0x7f26c80b2b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40880/0 (socket says 172.21.15.46:40880) 2024-01-26T23:46:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.736+0000 7f26ce218700 1 -- 172.21.15.46:0/274843274 learned_addr learned my addr 172.21.15.46:0/274843274 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.736+0000 7f26cda17700 1 -- 172.21.15.46:0/274843274 >> v1:172.21.15.106:6789/0 conn(0x7f26c80b10f0 legacy=0x7f26c80a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.736+0000 7f26cda17700 1 -- 172.21.15.46:0/274843274 >> v1:172.21.15.46:6789/0 conn(0x7f26c8159760 legacy=0x7f26c80b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:35.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.736+0000 7f26cda17700 1 -- 172.21.15.46:0/274843274 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c806ed30 con 0x7f26c8159b30 2024-01-26T23:46:35.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.737+0000 7f26cda17700 1 --2- 172.21.15.46:0/274843274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c8159b30 0x7f26c80b0bb0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f26b8009910 tx=0x7f26b80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928de7dc7aad3d1c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:35.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.737+0000 7f26cd216700 1 -- 172.21.15.46:0/274843274 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b800b820 con 0x7f26c8159b30 2024-01-26T23:46:35.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.737+0000 7f26cd216700 1 -- 172.21.15.46:0/274843274 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b800b980 con 0x7f26c8159b30 2024-01-26T23:46:35.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.737+0000 7f26cd216700 1 -- 172.21.15.46:0/274843274 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b800bd20 con 0x7f26c8159b30 2024-01-26T23:46:35.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.737+0000 7f26d047c700 1 -- 172.21.15.46:0/274843274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c8159b30 msgr2=0x7f26c80b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:35.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.737+0000 7f26d047c700 1 --2- 172.21.15.46:0/274843274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c8159b30 0x7f26c80b0bb0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f26b8009910 tx=0x7f26b80091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.738+0000 7f26d047c700 1 -- 172.21.15.46:0/274843274 shutdown_connections 2024-01-26T23:46:35.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.738+0000 7f26d047c700 1 --2- 172.21.15.46:0/274843274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c8159b30 0x7f26c80b0bb0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.738+0000 7f26d047c700 1 -- 172.21.15.46:0/274843274 >> 172.21.15.46:0/274843274 conn(0x7f26c8069510 msgr2=0x7f26c80aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:35.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.738+0000 7f26d047c700 1 -- 172.21.15.46:0/274843274 shutdown_connections 2024-01-26T23:46:35.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.738+0000 7f26d047c700 1 -- 172.21.15.46:0/274843274 wait complete. 2024-01-26T23:46:35.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.739+0000 7f26d047c700 1 Processor -- start 2024-01-26T23:46:35.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.739+0000 7f26d047c700 1 -- start start 2024-01-26T23:46:35.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26d047c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80b10f0 0x7f26c8146000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:35.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26d047c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26c8159760 0x7f26c815e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:35.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26d047c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26c8159b30 0x7f26c80c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26d047c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c815f0b0 con 0x7f26c80b10f0 2024-01-26T23:46:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26d047c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c815b450 con 0x7f26c8159b30 2024-01-26T23:46:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26d047c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c815eaf0 con 0x7f26c8159760 2024-01-26T23:46:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26ce218700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80b10f0 0x7f26c8146000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26ce218700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80b10f0 0x7f26c8146000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33646/0 (socket says 172.21.15.46:33646) 2024-01-26T23:46:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26ce218700 1 -- 172.21.15.46:0/133563089 learned_addr learned my addr 172.21.15.46:0/133563089 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26cda17700 1 --2- 172.21.15.46:0/133563089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26c8159760 0x7f26c815e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26cea19700 1 --2- 172.21.15.46:0/133563089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26c8159b30 0x7f26c80c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26ce218700 1 -- 172.21.15.46:0/133563089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26c8159760 msgr2=0x7f26c815e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26ce218700 1 --2- 172.21.15.46:0/133563089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26c8159760 0x7f26c815e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26ce218700 1 -- 172.21.15.46:0/133563089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26c8159b30 msgr2=0x7f26c80c2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26ce218700 1 --2- 172.21.15.46:0/133563089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26c8159b30 0x7f26c80c2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26ce218700 1 -- 172.21.15.46:0/133563089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b8009420 con 0x7f26c80b10f0 2024-01-26T23:46:35.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26ce218700 1 --2- 172.21.15.46:0/133563089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80b10f0 0x7f26c8146000 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f26c400c7c0 tx=0x7f26c4007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:35.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26be7fc700 1 -- 172.21.15.46:0/133563089 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c4005b30 con 0x7f26c80b10f0 2024-01-26T23:46:35.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26be7fc700 1 -- 172.21.15.46:0/133563089 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c4005c90 con 0x7f26c80b10f0 2024-01-26T23:46:35.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26d047c700 1 -- 172.21.15.46:0/133563089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c815edd0 con 0x7f26c80b10f0 2024-01-26T23:46:35.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.740+0000 7f26be7fc700 1 -- 172.21.15.46:0/133563089 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c4004a80 con 0x7f26c80b10f0 2024-01-26T23:46:35.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.741+0000 7f26d047c700 1 -- 172.21.15.46:0/133563089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c8156490 con 0x7f26c80b10f0 2024-01-26T23:46:35.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.742+0000 7f26d047c700 1 -- 172.21.15.46:0/133563089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c815af40 con 0x7f26c80b10f0 2024-01-26T23:46:35.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.742+0000 7f26be7fc700 1 -- 172.21.15.46:0/133563089 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f26c40070d0 con 0x7f26c80b10f0 2024-01-26T23:46:35.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.742+0000 7f26be7fc700 1 --2- 172.21.15.46:0/133563089 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f26ac066ea0 0x7f26ac069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:35.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.743+0000 7f26be7fc700 1 -- 172.21.15.46:0/133563089 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f26c40839d0 con 0x7f26c80b10f0 2024-01-26T23:46:35.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.743+0000 7f26cda17700 1 --2- 172.21.15.46:0/133563089 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f26ac066ea0 0x7f26ac069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:35.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.743+0000 7f26cda17700 1 --2- 172.21.15.46:0/133563089 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f26ac066ea0 0x7f26ac069350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f26b80091e0 tx=0x7f26b8017000 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:35.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.747+0000 7f26be7fc700 1 -- 172.21.15.46:0/133563089 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f26c404b5a0 con 0x7f26c80b10f0 2024-01-26T23:46:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.966+0000 7f26d047c700 1 -- 172.21.15.46:0/133563089 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f26c815b590 con 0x7f26ac066ea0 2024-01-26T23:46:35.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.975+0000 7f26be7fc700 1 -- 172.21.15.46:0/133563089 <== mgr.6365 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+280417 (secure 0 0 0) 0x7f26b800baf0 con 0x7f26ac066ea0 2024-01-26T23:46:35.980 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:35.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.978+0000 7f26d047c700 1 -- 172.21.15.46:0/133563089 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f26ac066ea0 msgr2=0x7f26ac069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:35.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.978+0000 7f26d047c700 1 --2- 172.21.15.46:0/133563089 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f26ac066ea0 0x7f26ac069350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f26b80091e0 tx=0x7f26b8017000 comp rx=0 tx=0).stop 2024-01-26T23:46:35.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.978+0000 7f26d047c700 1 -- 172.21.15.46:0/133563089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80b10f0 msgr2=0x7f26c8146000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:35.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.978+0000 7f26d047c700 1 --2- 172.21.15.46:0/133563089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80b10f0 0x7f26c8146000 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f26c400c7c0 tx=0x7f26c4007f70 comp rx=0 tx=0).stop 2024-01-26T23:46:35.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.978+0000 7f26d047c700 1 -- 172.21.15.46:0/133563089 shutdown_connections 2024-01-26T23:46:35.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.978+0000 7f26d047c700 1 --2- 172.21.15.46:0/133563089 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f26ac066ea0 0x7f26ac069350 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.978+0000 7f26d047c700 1 --2- 172.21.15.46:0/133563089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80b10f0 0x7f26c8146000 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.978+0000 7f26d047c700 1 --2- 172.21.15.46:0/133563089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26c8159b30 0x7f26c80c2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.978+0000 7f26d047c700 1 --2- 172.21.15.46:0/133563089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26c8159760 0x7f26c815e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:35.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.978+0000 7f26d047c700 1 -- 172.21.15.46:0/133563089 >> 172.21.15.46:0/133563089 conn(0x7f26c8069510 msgr2=0x7f26c80ad3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:35.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.978+0000 7f26d047c700 1 -- 172.21.15.46:0/133563089 shutdown_connections 2024-01-26T23:46:35.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:35.978+0000 7f26d047c700 1 -- 172.21.15.46:0/133563089 wait complete. 2024-01-26T23:46:35.983 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:46:35.993 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-26T23:46:35.636475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591904,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":737,"num_read_kb":1446,"num_write":285,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4552,"ondisk_log_size":4552,"up":94,"acting":94,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":46,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2439160,"kb_used_data":11704,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752535560,"statfs":{"total":773094113280,"available":770596413440,"internally_reserved":0,"allocated":11984896,"data_stored":6643515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.880831"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678729+0000","last_change":"2024-01-26T23:46:31.677430+0000","last_active":"2024-01-26T23:46:31.678729+0000","last_peered":"2024-01-26T23:46:31.678729+0000","last_clean":"2024-01-26T23:46:31.678729+0000","last_became_active":"2024-01-26T23:46:31.677151+0000","last_became_peered":"2024-01-26T23:46:31.677151+0000","last_unstale":"2024-01-26T23:46:31.678729+0000","last_undegraded":"2024-01-26T23:46:31.678729+0000","last_fullsized":"2024-01-26T23:46:31.678729+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:15:00.551460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671925+0000","last_change":"2024-01-26T23:46:31.666995+0000","last_active":"2024-01-26T23:46:31.671925+0000","last_peered":"2024-01-26T23:46:31.671925+0000","last_clean":"2024-01-26T23:46:31.671925+0000","last_became_active":"2024-01-26T23:46:31.666507+0000","last_became_peered":"2024-01-26T23:46:31.666507+0000","last_unstale":"2024-01-26T23:46:31.671925+0000","last_undegraded":"2024-01-26T23:46:31.671925+0000","last_fullsized":"2024-01-26T23:46:31.671925+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:35:52.815366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.672072+0000","last_change":"2024-01-26T23:46:31.665567+0000","last_active":"2024-01-26T23:46:31.672072+0000","last_peered":"2024-01-26T23:46:31.672072+0000","last_clean":"2024-01-26T23:46:31.672072+0000","last_became_active":"2024-01-26T23:46:31.665250+0000","last_became_peered":"2024-01-26T23:46:31.665250+0000","last_unstale":"2024-01-26T23:46:31.672072+0000","last_undegraded":"2024-01-26T23:46:31.672072+0000","last_fullsized":"2024-01-26T23:46:31.672072+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:44.847438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671844+0000","last_change":"2024-01-26T23:46:31.666793+0000","last_active":"2024-01-26T23:46:31.671844+0000","last_peered":"2024-01-26T23:46:31.671844+0000","last_clean":"2024-01-26T23:46:31.671844+0000","last_became_active":"2024-01-26T23:46:31.666112+0000","last_became_peered":"2024-01-26T23:46:31.666112+0000","last_unstale":"2024-01-26T23:46:31.671844+0000","last_undegraded":"2024-01-26T23:46:31.671844+0000","last_fullsized":"2024-01-26T23:46:31.671844+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:38.074736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675273+0000","last_change":"2024-01-26T23:46:31.668562+0000","last_active":"2024-01-26T23:46:31.675273+0000","last_peered":"2024-01-26T23:46:31.675273+0000","last_clean":"2024-01-26T23:46:31.675273+0000","last_became_active":"2024-01-26T23:46:31.668253+0000","last_became_peered":"2024-01-26T23:46:31.668253+0000","last_unstale":"2024-01-26T23:46:31.675273+0000","last_undegraded":"2024-01-26T23:46:31.675273+0000","last_fullsized":"2024-01-26T23:46:31.675273+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:43:18.393081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761872+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:06:33.819261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675662+0000","last_change":"2024-01-26T23:46:31.672354+0000","last_active":"2024-01-26T23:46:31.675662+0000","last_peered":"2024-01-26T23:46:31.675662+0000","last_clean":"2024-01-26T23:46:31.675662+0000","last_became_active":"2024-01-26T23:46:31.672029+0000","last_became_peered":"2024-01-26T23:46:31.672029+0000","last_unstale":"2024-01-26T23:46:31.675662+0000","last_undegraded":"2024-01-26T23:46:31.675662+0000","last_fullsized":"2024-01-26T23:46:31.675662+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:16:43.142449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.762969+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:46.577047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760270+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:07.859918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.670354+0000","last_change":"2024-01-26T23:46:31.662791+0000","last_active":"2024-01-26T23:46:31.670354+0000","last_peered":"2024-01-26T23:46:31.670354+0000","last_clean":"2024-01-26T23:46:31.670354+0000","last_became_active":"2024-01-26T23:46:31.662506+0000","last_became_peered":"2024-01-26T23:46:31.662506+0000","last_unstale":"2024-01-26T23:46:31.670354+0000","last_undegraded":"2024-01-26T23:46:31.670354+0000","last_fullsized":"2024-01-26T23:46:31.670354+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:22.391377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761816+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:25:23.538512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761960+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:34:01.252528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760688+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:19:58.164346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678716+0000","last_change":"2024-01-26T23:46:31.677394+0000","last_active":"2024-01-26T23:46:31.678716+0000","last_peered":"2024-01-26T23:46:31.678716+0000","last_clean":"2024-01-26T23:46:31.678716+0000","last_became_active":"2024-01-26T23:46:31.677090+0000","last_became_peered":"2024-01-26T23:46:31.677090+0000","last_unstale":"2024-01-26T23:46:31.678716+0000","last_undegraded":"2024-01-26T23:46:31.678716+0000","last_fullsized":"2024-01-26T23:46:31.678716+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:28:36.539497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.757478+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:16:42.463283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760298+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:16:42.463283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.758550+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:06:33.819261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.757542+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:04:08.622450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.762929+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:37.479741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.14","version":"111'6","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.776077+0000","last_change":"2024-01-26T23:46:30.772469+0000","last_active":"2024-01-26T23:46:31.776077+0000","last_peered":"2024-01-26T23:46:31.776077+0000","last_clean":"2024-01-26T23:46:31.776077+0000","last_became_active":"2024-01-26T23:46:30.772188+0000","last_became_peered":"2024-01-26T23:46:30.772188+0000","last_unstale":"2024-01-26T23:46:31.776077+0000","last_undegraded":"2024-01-26T23:46:31.776077+0000","last_fullsized":"2024-01-26T23:46:31.776077+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:04:26.732373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.759862+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:37.479741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.657510+0000","last_change":"2024-01-26T23:46:30.783543+0000","last_active":"2024-01-26T23:46:31.657510+0000","last_peered":"2024-01-26T23:46:31.657510+0000","last_clean":"2024-01-26T23:46:31.657510+0000","last_became_active":"2024-01-26T23:46:30.774051+0000","last_became_peered":"2024-01-26T23:46:30.774051+0000","last_unstale":"2024-01-26T23:46:31.657510+0000","last_undegraded":"2024-01-26T23:46:31.657510+0000","last_fullsized":"2024-01-26T23:46:31.657510+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:18:11.911206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760169+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:45:02.564138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761051+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:00:54.682539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.672965+0000","last_change":"2024-01-26T23:46:31.667032+0000","last_active":"2024-01-26T23:46:31.672965+0000","last_peered":"2024-01-26T23:46:31.672965+0000","last_clean":"2024-01-26T23:46:31.672965+0000","last_became_active":"2024-01-26T23:46:31.666719+0000","last_became_peered":"2024-01-26T23:46:31.666719+0000","last_unstale":"2024-01-26T23:46:31.672965+0000","last_undegraded":"2024-01-26T23:46:31.672965+0000","last_fullsized":"2024-01-26T23:46:31.672965+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:47:07.666792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761349+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:15:25.108006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.759973+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:46.577047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.762459+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:06:33.819261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671822+0000","last_change":"2024-01-26T23:46:31.665817+0000","last_active":"2024-01-26T23:46:31.671822+0000","last_peered":"2024-01-26T23:46:31.671822+0000","last_clean":"2024-01-26T23:46:31.671822+0000","last_became_active":"2024-01-26T23:46:31.665513+0000","last_became_peered":"2024-01-26T23:46:31.665513+0000","last_unstale":"2024-01-26T23:46:31.671822+0000","last_undegraded":"2024-01-26T23:46:31.671822+0000","last_fullsized":"2024-01-26T23:46:31.671822+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:56:41.826944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.758590+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:25:23.538512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.756936+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:58.710220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.758449+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:15:25.108006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675654+0000","last_change":"2024-01-26T23:46:31.672269+0000","last_active":"2024-01-26T23:46:31.675654+0000","last_peered":"2024-01-26T23:46:31.675654+0000","last_clean":"2024-01-26T23:46:31.675654+0000","last_became_active":"2024-01-26T23:46:31.671924+0000","last_became_peered":"2024-01-26T23:46:31.671924+0000","last_unstale":"2024-01-26T23:46:31.675654+0000","last_undegraded":"2024-01-26T23:46:31.675654+0000","last_fullsized":"2024-01-26T23:46:31.675654+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:36:26.521199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.756907+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:27:38.483382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760450+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:04:08.622450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675423+0000","last_change":"2024-01-26T23:46:31.668761+0000","last_active":"2024-01-26T23:46:31.675423+0000","last_peered":"2024-01-26T23:46:31.675423+0000","last_clean":"2024-01-26T23:46:31.675423+0000","last_became_active":"2024-01-26T23:46:31.668456+0000","last_became_peered":"2024-01-26T23:46:31.668456+0000","last_unstale":"2024-01-26T23:46:31.675423+0000","last_undegraded":"2024-01-26T23:46:31.675423+0000","last_fullsized":"2024-01-26T23:46:31.675423+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:51:54.286344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.762515+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:46:39.085189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675695+0000","last_change":"2024-01-26T23:46:31.672735+0000","last_active":"2024-01-26T23:46:31.675695+0000","last_peered":"2024-01-26T23:46:31.675695+0000","last_clean":"2024-01-26T23:46:31.675695+0000","last_became_active":"2024-01-26T23:46:31.672465+0000","last_became_peered":"2024-01-26T23:46:31.672465+0000","last_unstale":"2024-01-26T23:46:31.675695+0000","last_undegraded":"2024-01-26T23:46:31.675695+0000","last_fullsized":"2024-01-26T23:46:31.675695+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:57:26.854523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.762075+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:25:23.538512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761378+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:34:01.252528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.656731+0000","last_change":"2024-01-26T23:46:30.777287+0000","last_active":"2024-01-26T23:46:31.656731+0000","last_peered":"2024-01-26T23:46:31.656731+0000","last_clean":"2024-01-26T23:46:31.656731+0000","last_became_active":"2024-01-26T23:46:30.773908+0000","last_became_peered":"2024-01-26T23:46:30.773908+0000","last_unstale":"2024-01-26T23:46:31.656731+0000","last_undegraded":"2024-01-26T23:46:31.656731+0000","last_fullsized":"2024-01-26T23:46:31.656731+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:28.962140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.758491+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:34:01.252528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761333+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:28.581902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675370+0000","last_change":"2024-01-26T23:46:31.668096+0000","last_active":"2024-01-26T23:46:31.675370+0000","last_peered":"2024-01-26T23:46:31.675370+0000","last_clean":"2024-01-26T23:46:31.675370+0000","last_became_active":"2024-01-26T23:46:31.665095+0000","last_became_peered":"2024-01-26T23:46:31.665095+0000","last_unstale":"2024-01-26T23:46:31.675370+0000","last_undegraded":"2024-01-26T23:46:31.675370+0000","last_fullsized":"2024-01-26T23:46:31.675370+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:53:38.299979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.759827+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:41:51.840628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760147+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:23:11.224769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"109'4546","reported_seq":4885,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-26T23:46:30.629658+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:46:30.629658+0000","last_peered":"2024-01-26T23:46:30.629658+0000","last_clean":"2024-01-26T23:46:30.629658+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:46:30.629658+0000","last_undegraded":"2024-01-26T23:46:30.629658+0000","last_fullsized":"2024-01-26T23:46:30.629658+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":4546,"log_dups_size":0,"ondisk_log_size":4546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":737,"num_read_kb":1446,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":28,"acting":28,"num_store_stats":1},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":737,"num_read_kb":1446,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4546,"ondisk_log_size":4546,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575050,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311468,"kb_used_data":1892,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060372,"statfs":{"total":96636764160,"available":96317820928,"internally_reserved":0,"allocated":1937408,"data_stored":1271072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.54100000000000004,"15min":0.56499999999999995},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.74099999999999999,"5min":0.79500000000000004,"15min":0.82299999999999995},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.55600000000000005,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.83999999999999997},"last":0.28000000000000003}]},{"osd":1,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.55200000000000005,"15min":0.57899999999999996},"min":{"1min":0.36699999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67400000000000004,"5min":0.81299999999999994,"15min":0.82899999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.34999999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.56299999999999994,"15min":0.57699999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.81699999999999995},"last":0.38},{"interface":"front","average":{"1min":0.504,"5min":0.58299999999999996,"15min":0.59799999999999998},"min":{"1min":0.30199999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75700000000000001,"5min":0.81899999999999995,"15min":0.88600000000000001},"last":0.47299999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.77100000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.58699999999999997,"15min":0.60099999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71699999999999997,"5min":0.83899999999999997,"15min":0.86399999999999999},"last":0.439}]},{"osd":4,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46100000000000002,"15min":0.45600000000000002},"min":{"1min":0.252,"5min":0.247,"15min":0.247},"max":{"1min":0.64300000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.315},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.48799999999999999,"15min":0.48199999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.66400000000000003,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.42999999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.48899999999999999},"min":{"1min":0.27200000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.70599999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54200000000000004,"15min":0.52700000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.73599999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.39100000000000001}]},{"osd":6,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.55300000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78700000000000003,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55700000000000005,"15min":0.54900000000000004},"min":{"1min":0.32000000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70599999999999996,"5min":0.85199999999999998,"15min":0.86899999999999999},"last":0.40799999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575050,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302696,"kb_used_data":1312,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069144,"statfs":{"total":96636764160,"available":96326803456,"internally_reserved":0,"allocated":1343488,"data_stored":680704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55800000000000005,"15min":0.57699999999999996},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26300000000000001},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.255},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.57199999999999995,"15min":0.59199999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.68300000000000005,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.36599999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.54300000000000004,"15min":0.57199999999999995},"min":{"1min":0.317,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.622,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.34399999999999997},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.38700000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55900000000000005,"15min":0.58499999999999996},"min":{"1min":0.39600000000000002,"5min":0.35499999999999998,"15min":0.29599999999999999},"max":{"1min":0.67200000000000004,"5min":0.74199999999999999,"15min":0.85799999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.56999999999999995,"15min":0.58899999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.33900000000000002},"max":{"1min":0.621,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.47899999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.29999999999999999},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60999999999999999,"15min":0.622},"min":{"1min":0.41899999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.47299999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47699999999999998,"15min":0.47299999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.22},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.498,"15min":0.495},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.64600000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.435}]},{"osd":5,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48599999999999999,"15min":0.48499999999999999},"min":{"1min":0.316,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.66100000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54000000000000004,"15min":0.53100000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26200000000000001},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":0.46300000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.246},"max":{"1min":0.64100000000000001,"5min":0.82799999999999996,"15min":0.85899999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.53500000000000003,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.308},"max":{"1min":0.68600000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.44400000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575050,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302696,"kb_used_data":1312,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069144,"statfs":{"total":96636764160,"available":96326803456,"internally_reserved":0,"allocated":1343488,"data_stored":680704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41599999999999998,"15min":0.40899999999999997},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.72599999999999998},"last":0.315},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.44900000000000001,"15min":0.44600000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.78600000000000003,"5min":1.337,"15min":1.337},"last":0.28999999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.41699999999999998,"15min":0.41499999999999998},"min":{"1min":0.254,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.68799999999999994,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.48199999999999998,"15min":0.47499999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.28999999999999998},"max":{"1min":0.74099999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.42199999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52500000000000002,"15min":0.51400000000000001},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":0.75900000000000001,"5min":1.782,"15min":1.782},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.48899999999999999,"15min":0.47999999999999998},"min":{"1min":0.32000000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.71499999999999997,"5min":1.379,"15min":1.379},"last":0.38800000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.64200000000000002,"15min":0.64900000000000002},"min":{"1min":0.435,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":0.81699999999999995,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.65700000000000003,"15min":0.66200000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.84299999999999997,"5min":0.91900000000000004,"15min":0.95699999999999996},"last":0.56100000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.67700000000000005,"15min":0.67300000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":0.86499999999999999,"5min":1.208,"15min":1.208},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.67700000000000005,"15min":0.66800000000000004},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.42499999999999999},"max":{"1min":0.82899999999999996,"5min":0.96599999999999997,"15min":1.018},"last":0.61299999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.68000000000000005,"15min":0.67800000000000005},"min":{"1min":0.47799999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91900000000000004,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.68000000000000005,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.78300000000000003,"5min":1.022,"15min":1.022},"last":0.504}]},{"osd":7,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.70499999999999996,"15min":0.71099999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89200000000000002,"5min":0.98199999999999998,"15min":1.0069999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.73099999999999998,"15min":0.72299999999999998},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.438},"max":{"1min":0.95399999999999996,"5min":1.296,"15min":1.296},"last":0.496}]}]},{"osd":0,"up_from":13,"seq":55834575050,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307432,"kb_used_data":1952,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064408,"statfs":{"total":96636764160,"available":96321953792,"internally_reserved":0,"allocated":1998848,"data_stored":1294736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.41999999999999998,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.45100000000000001,"15min":0.45500000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27300000000000002},"max":{"1min":1.363,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41199999999999998,"15min":0.42499999999999999},"min":{"1min":0.27900000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.57499999999999996,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.45700000000000002,"15min":0.47099999999999997},"min":{"1min":0.32700000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63700000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.36699999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.47399999999999998,"15min":0.46899999999999997},"min":{"1min":0.32300000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.145,"5min":2.145,"15min":2.145},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48599999999999999,"15min":0.48799999999999999},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64300000000000002,"5min":1.198,"15min":1.198},"last":0.32100000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.64400000000000002,"15min":0.65200000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.625,"15min":0.65000000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.752,"5min":0.94299999999999995,"15min":0.98899999999999999},"last":0.42499999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64200000000000002,"15min":0.65900000000000003},"min":{"1min":0.42199999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.67500000000000004,"15min":0.68100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.70799999999999996}]},{"osd":6,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.63,"15min":0.66700000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80200000000000005,"5min":0.89800000000000002,"15min":1.032},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.67400000000000004,"15min":0.69499999999999995},"min":{"1min":0.48299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.61299999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.028,"15min":0.83199999999999996},"min":{"1min":0.501,"5min":0.435,"15min":0.435},"max":{"1min":4.8710000000000004,"5min":25.965,"15min":25.965},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.1379999999999999,"15min":0.875},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.433},"max":{"1min":4.9199999999999999,"5min":33.085000000000001,"15min":33.085000000000001},"last":0.64400000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575049,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306792,"kb_used_data":1312,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065048,"statfs":{"total":96636764160,"available":96322609152,"internally_reserved":0,"allocated":1343488,"data_stored":680704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.51700000000000002,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.437,"15min":0.42299999999999999},"min":{"1min":0.33500000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.308}]},{"osd":1,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.432,"15min":0.42799999999999999},"min":{"1min":0.26900000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.61899999999999999,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.377,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.42299999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46999999999999997,"15min":0.48599999999999999},"min":{"1min":0.35999999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.64800000000000002,"5min":0.65500000000000003,"15min":0.74099999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.48099999999999998,"15min":0.47499999999999998},"min":{"1min":0.35599999999999998,"5min":0.32200000000000001,"15min":0.307},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.45400000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.63200000000000001,"15min":0.63400000000000001},"min":{"1min":0.41499999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.80900000000000005,"5min":1.087,"15min":1.087},"last":0.499},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.65200000000000002,"15min":0.64400000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.874,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.53200000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.65100000000000002,"15min":0.65200000000000002},"min":{"1min":0.375,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.90300000000000002,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.67300000000000004,"15min":0.67900000000000005},"min":{"1min":0.41799999999999998,"5min":0.39800000000000002,"15min":0.35199999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.085},"last":0.52000000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67900000000000005,"15min":0.68100000000000005},"min":{"1min":0.47699999999999998,"5min":0.439,"15min":0.40899999999999997},"max":{"1min":1.022,"5min":1.022,"15min":1.1080000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.67700000000000005,"15min":0.68300000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.38800000000000001},"max":{"1min":0.83799999999999997,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.69499999999999995,"15min":0.70299999999999996},"min":{"1min":0.52100000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94699999999999995,"5min":1.0029999999999999,"15min":1.127},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.72299999999999998,"15min":0.73299999999999998},"min":{"1min":0.48899999999999999,"5min":0.48199999999999998,"15min":0.42099999999999999},"max":{"1min":1.05,"5min":1.052,"15min":1.139},"last":0.58499999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575048,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306780,"kb_used_data":1300,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065060,"statfs":{"total":96636764160,"available":96322621440,"internally_reserved":0,"allocated":1331200,"data_stored":674187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.39900000000000002,"15min":0.39400000000000002},"min":{"1min":0.26500000000000001,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.45000000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.309},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.44,"15min":0.439},"min":{"1min":0.25800000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.58699999999999997,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.38900000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.46600000000000003,"15min":0.45500000000000002},"min":{"1min":0.27900000000000003,"5min":0.253,"15min":0.253},"max":{"1min":0.57799999999999996,"5min":1.633,"15min":1.633},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.48399999999999999,"15min":0.48099999999999998},"min":{"1min":0.35199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.54900000000000004,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.48899999999999999,"15min":0.47899999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.54600000000000004,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.33200000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.53700000000000003,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.33700000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.68200000000000005,"15min":0.66300000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.878,"5min":3.306,"15min":3.306},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.41499999999999998,"5min":0.40200000000000002,"15min":0.379},"max":{"1min":0.85199999999999998,"5min":0.95799999999999996,"15min":1.008},"last":0.55700000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.65700000000000003,"15min":0.66100000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82499999999999996,"5min":1.593,"15min":1.593},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.67500000000000004,"15min":0.67300000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.84399999999999997,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.52300000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.67100000000000004,"15min":0.67700000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.77800000000000002,"5min":1.506,"15min":1.506},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.69099999999999995,"15min":0.69299999999999995},"min":{"1min":0.48799999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.82599999999999996,"5min":1.427,"15min":1.427},"last":0.59599999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.70799999999999996,"15min":0.69799999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.42499999999999999},"max":{"1min":0.73899999999999999,"5min":3.238,"15min":3.238},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.747,"15min":0.73899999999999999},"min":{"1min":0.54100000000000004,"5min":0.48899999999999999,"15min":0.48699999999999999},"max":{"1min":0.86699999999999999,"5min":3.351,"15min":3.351},"last":0.66600000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575050,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302696,"kb_used_data":1312,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069144,"statfs":{"total":96636764160,"available":96326803456,"internally_reserved":0,"allocated":1343488,"data_stored":680704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.54800000000000004,"15min":0.57799999999999996},"min":{"1min":0.28999999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57699999999999996,"5min":1.21,"15min":1.21},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.58499999999999996,"15min":0.58999999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.367,"5min":5.367,"15min":5.367},"last":0.48799999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.55700000000000005,"15min":0.59299999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68899999999999995,"5min":0.77500000000000002,"15min":0.88},"last":0.442},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71499999999999997,"5min":0.80600000000000005,"15min":0.81699999999999995},"last":0.501}]},{"osd":2,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53500000000000003,"15min":0.56799999999999995},"min":{"1min":0.36699999999999999,"5min":0.33300000000000002,"15min":0.32700000000000001},"max":{"1min":0.56599999999999995,"5min":0.81100000000000005,"15min":0.85999999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.58899999999999997,"15min":0.60499999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.69599999999999995,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.50700000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.56000000000000005,"15min":0.58699999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.318},"max":{"1min":0.66900000000000004,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.434},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56699999999999995,"15min":0.58599999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.35699999999999998},"max":{"1min":0.64900000000000002,"5min":1.05,"15min":1.05},"last":0.51300000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.46000000000000002,"15min":0.46300000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.63100000000000001,"5min":0.76200000000000001,"15min":0.77600000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.48499999999999999,"15min":0.48199999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68300000000000005,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.47599999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.47099999999999997,"15min":0.47199999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.27200000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.73899999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.311},"max":{"1min":0.72899999999999998,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.374}]},{"osd":7,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.36699999999999999,"5min":0.36499999999999999,"15min":0.35799999999999998},"max":{"1min":0.753,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.57099999999999995,"15min":0.56899999999999995},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81000000000000005,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.45600000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575050,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298600,"kb_used_data":1312,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073240,"statfs":{"total":96636764160,"available":96330997760,"internally_reserved":0,"allocated":1343488,"data_stored":680704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.26000000000000001},"max":{"1min":0.64700000000000002,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.246},{"interface":"front","average":{"1min":0.502,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.315,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.65800000000000003,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.35999999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.61199999999999999,"15min":0.622},"min":{"1min":0.34799999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66300000000000003,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.629,"15min":0.63400000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.39300000000000002},"max":{"1min":0.71199999999999997,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.46800000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.55900000000000005,"15min":0.56999999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.30099999999999999},"max":{"1min":0.67400000000000004,"5min":0.83699999999999997,"15min":0.83799999999999997},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.58999999999999997,"15min":0.59699999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69699999999999995,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":0.42699999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.57099999999999995,"15min":0.58399999999999996},"min":{"1min":0.38100000000000001,"5min":0.35999999999999999,"15min":0.314},"max":{"1min":0.68799999999999994,"5min":0.86199999999999999,"15min":0.88400000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.59899999999999998,"15min":0.59199999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73599999999999999,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.437}]},{"osd":4,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.48899999999999999,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.48999999999999999,"15min":0.48699999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":0.626,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.41499999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.48799999999999999,"15min":0.47699999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65400000000000003,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.52100000000000002,"15min":0.51900000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.76200000000000001,"5min":1.99,"15min":1.99},"last":0.45200000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.312,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71399999999999997,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.443},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79300000000000004,"5min":0.82199999999999995,"15min":0.83099999999999996},"last":0.45700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:46:36.001 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:46:36.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.172+0000 7f73bffff700 1 Processor -- start 2024-01-26T23:46:36.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.172+0000 7f73bffff700 1 -- start start 2024-01-26T23:46:36.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.173+0000 7f73bffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73c00c9100 0x7f73c0157a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:36.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.173+0000 7f73bffff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73c00cb6b0 0x7f73c0157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:36.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.173+0000 7f73bffff700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c015b7c0 con 0x7f73c00cb2e0 2024-01-26T23:46:36.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.173+0000 7f73bffff700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c00ca410 con 0x7f73c00c9100 2024-01-26T23:46:36.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.173+0000 7f73bffff700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c01584e0 con 0x7f73c00cb6b0 2024-01-26T23:46:36.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.173+0000 7f73be7fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73c00c9100 0x7f73c0157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:36.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.173+0000 7f73beffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f73c00cb2e0 0x7f73c00c89f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40888/0 (socket says 172.21.15.46:40888) 2024-01-26T23:46:36.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.173+0000 7f73beffd700 1 -- 172.21.15.46:0/2595327252 learned_addr learned my addr 172.21.15.46:0/2595327252 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:36.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.174+0000 7f73bf7fe700 1 --2- 172.21.15.46:0/2595327252 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73c00cb6b0 0x7f73c0157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:36.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.174+0000 7f73be7fc700 1 -- 172.21.15.46:0/2595327252 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73c00cb6b0 msgr2=0x7f73c0157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:36.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.174+0000 7f73be7fc700 1 --2- 172.21.15.46:0/2595327252 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73c00cb6b0 0x7f73c0157fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.174+0000 7f73be7fc700 1 -- 172.21.15.46:0/2595327252 >> v1:172.21.15.46:6789/0 conn(0x7f73c00cb2e0 legacy=0x7f73c00c89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:36.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.174+0000 7f73bdffb700 1 -- 172.21.15.46:0/2595327252 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086636559 0 0) 0x7f73c015b7c0 con 0x7f73c00cb2e0 2024-01-26T23:46:36.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.174+0000 7f73be7fc700 1 -- 172.21.15.46:0/2595327252 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c01589a0 con 0x7f73c00c9100 2024-01-26T23:46:36.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.174+0000 7f73be7fc700 1 --2- 172.21.15.46:0/2595327252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c00c9100 0x7f73c0157a60 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f73a80098e0 tx=0x7f73a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c4ac2062ee7311e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:36.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.174+0000 7f73bdffb700 1 -- 172.21.15.46:0/2595327252 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a800b640 con 0x7f73c00c9100 2024-01-26T23:46:36.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.174+0000 7f73bdffb700 1 -- 172.21.15.46:0/2595327252 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a800b7a0 con 0x7f73c00c9100 2024-01-26T23:46:36.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.174+0000 7f73bdffb700 1 -- 172.21.15.46:0/2595327252 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a800bb40 con 0x7f73c00c9100 2024-01-26T23:46:36.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.175+0000 7f73bffff700 1 -- 172.21.15.46:0/2595327252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c00c9100 msgr2=0x7f73c0157a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:36.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.175+0000 7f73bffff700 1 --2- 172.21.15.46:0/2595327252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c00c9100 0x7f73c0157a60 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f73a80098e0 tx=0x7f73a80091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.175+0000 7f73bffff700 1 -- 172.21.15.46:0/2595327252 shutdown_connections 2024-01-26T23:46:36.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.175+0000 7f73bffff700 1 --2- 172.21.15.46:0/2595327252 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73c00cb6b0 0x7f73c0157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.175+0000 7f73bffff700 1 --2- 172.21.15.46:0/2595327252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c00c9100 0x7f73c0157a60 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.175+0000 7f73bffff700 1 -- 172.21.15.46:0/2595327252 >> 172.21.15.46:0/2595327252 conn(0x7f73c00683c0 msgr2=0x7f73c006bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:36.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.175+0000 7f73bffff700 1 -- 172.21.15.46:0/2595327252 shutdown_connections 2024-01-26T23:46:36.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.175+0000 7f73bffff700 1 -- 172.21.15.46:0/2595327252 wait complete. 2024-01-26T23:46:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.177+0000 7f73bffff700 1 Processor -- start 2024-01-26T23:46:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.177+0000 7f73bffff700 1 -- start start 2024-01-26T23:46:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73bffff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73c00c9100 0x7f73c0150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73bffff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73c00cb2e0 0x7f73c01575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73bffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c00cb6b0 0x7f73c0157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73bffff700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c0150a90 con 0x7f73c00cb6b0 2024-01-26T23:46:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73bffff700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c00b32d0 con 0x7f73c00cb2e0 2024-01-26T23:46:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73bffff700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c00b2230 con 0x7f73c00c9100 2024-01-26T23:46:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73bf7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c00cb6b0 0x7f73c0157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73beffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73c00c9100 0x7f73c0150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73bf7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c00cb6b0 0x7f73c0157af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33658/0 (socket says 172.21.15.46:33658) 2024-01-26T23:46:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73bf7fe700 1 -- 172.21.15.46:0/2538742784 learned_addr learned my addr 172.21.15.46:0/2538742784 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73be7fc700 1 --2- 172.21.15.46:0/2538742784 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73c00cb2e0 0x7f73c01575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:36.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73bf7fe700 1 -- 172.21.15.46:0/2538742784 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73c00c9100 msgr2=0x7f73c0150550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:36.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73bf7fe700 1 --2- 172.21.15.46:0/2538742784 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73c00c9100 0x7f73c0150550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73bf7fe700 1 -- 172.21.15.46:0/2538742784 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73c00cb2e0 msgr2=0x7f73c01575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:36.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73bf7fe700 1 --2- 172.21.15.46:0/2538742784 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73c00cb2e0 0x7f73c01575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73bf7fe700 1 -- 172.21.15.46:0/2538742784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a8009420 con 0x7f73c00cb6b0 2024-01-26T23:46:36.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73beffd700 1 --2- 172.21.15.46:0/2538742784 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73c00c9100 0x7f73c0150550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:36.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.178+0000 7f73be7fc700 1 --2- 172.21.15.46:0/2538742784 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73c00cb2e0 0x7f73c01575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:36.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.179+0000 7f73bf7fe700 1 --2- 172.21.15.46:0/2538742784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c00cb6b0 0x7f73c0157af0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f73b8009df0 tx=0x7f73b800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:36.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.179+0000 7f73b6ffd700 1 -- 172.21.15.46:0/2538742784 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b800caa0 con 0x7f73c00cb6b0 2024-01-26T23:46:36.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.179+0000 7f73b6ffd700 1 -- 172.21.15.46:0/2538742784 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b800f040 con 0x7f73c00cb6b0 2024-01-26T23:46:36.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.179+0000 7f73bffff700 1 -- 172.21.15.46:0/2538742784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c00b2510 con 0x7f73c00cb6b0 2024-01-26T23:46:36.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.179+0000 7f73b6ffd700 1 -- 172.21.15.46:0/2538742784 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b800ce70 con 0x7f73c00cb6b0 2024-01-26T23:46:36.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.179+0000 7f73bffff700 1 -- 172.21.15.46:0/2538742784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c00b2ad0 con 0x7f73c00cb6b0 2024-01-26T23:46:36.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.180+0000 7f73bffff700 1 -- 172.21.15.46:0/2538742784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c006ed60 con 0x7f73c00cb6b0 2024-01-26T23:46:36.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.180+0000 7f73b6ffd700 1 -- 172.21.15.46:0/2538742784 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f73b8014690 con 0x7f73c00cb6b0 2024-01-26T23:46:36.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.181+0000 7f73b6ffd700 1 --2- 172.21.15.46:0/2538742784 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73a0066ea0 0x7f73a0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:36.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.181+0000 7f73b6ffd700 1 -- 172.21.15.46:0/2538742784 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f73b8084ef0 con 0x7f73c00cb6b0 2024-01-26T23:46:36.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.181+0000 7f73beffd700 1 --2- 172.21.15.46:0/2538742784 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73a0066ea0 0x7f73a0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:36.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.181+0000 7f73beffd700 1 --2- 172.21.15.46:0/2538742784 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73a0066ea0 0x7f73a0069350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f73b0007fd0 tx=0x7f73b0007d30 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.185+0000 7f73b6ffd700 1 -- 172.21.15.46:0/2538742784 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f73b8051f70 con 0x7f73c00cb6b0 2024-01-26T23:46:36.339 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:36.338+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:36.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.403+0000 7f73bffff700 1 -- 172.21.15.46:0/2538742784 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f73c004bed0 con 0x7f73a0066ea0 2024-01-26T23:46:36.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.412+0000 7f73b6ffd700 1 -- 172.21.15.46:0/2538742784 <== mgr.6365 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+280417 (secure 0 0 0) 0x7f73c004bed0 con 0x7f73a0066ea0 2024-01-26T23:46:36.416 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:36.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.415+0000 7f73bffff700 1 -- 172.21.15.46:0/2538742784 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73a0066ea0 msgr2=0x7f73a0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:36.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.415+0000 7f73bffff700 1 --2- 172.21.15.46:0/2538742784 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73a0066ea0 0x7f73a0069350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f73b0007fd0 tx=0x7f73b0007d30 comp rx=0 tx=0).stop 2024-01-26T23:46:36.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.415+0000 7f73bffff700 1 -- 172.21.15.46:0/2538742784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c00cb6b0 msgr2=0x7f73c0157af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:36.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.415+0000 7f73bffff700 1 --2- 172.21.15.46:0/2538742784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c00cb6b0 0x7f73c0157af0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f73b8009df0 tx=0x7f73b800b040 comp rx=0 tx=0).stop 2024-01-26T23:46:36.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.415+0000 7f73bffff700 1 -- 172.21.15.46:0/2538742784 shutdown_connections 2024-01-26T23:46:36.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.415+0000 7f73bffff700 1 --2- 172.21.15.46:0/2538742784 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f73a0066ea0 0x7f73a0069350 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.415+0000 7f73bffff700 1 --2- 172.21.15.46:0/2538742784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c00cb6b0 0x7f73c0157af0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.415+0000 7f73bffff700 1 --2- 172.21.15.46:0/2538742784 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73c00c9100 0x7f73c0150550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.415+0000 7f73bffff700 1 --2- 172.21.15.46:0/2538742784 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73c00cb2e0 0x7f73c01575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.415+0000 7f73bffff700 1 -- 172.21.15.46:0/2538742784 >> 172.21.15.46:0/2538742784 conn(0x7f73c00683c0 msgr2=0x7f73c01517f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:36.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.415+0000 7f73bffff700 1 -- 172.21.15.46:0/2538742784 shutdown_connections 2024-01-26T23:46:36.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.415+0000 7f73bffff700 1 -- 172.21.15.46:0/2538742784 wait complete. 2024-01-26T23:46:36.422 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:46:36.430 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-26T23:46:35.636475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591904,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":737,"num_read_kb":1446,"num_write":285,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4552,"ondisk_log_size":4552,"up":94,"acting":94,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":46,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2439160,"kb_used_data":11704,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752535560,"statfs":{"total":773094113280,"available":770596413440,"internally_reserved":0,"allocated":11984896,"data_stored":6643515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.880831"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678729+0000","last_change":"2024-01-26T23:46:31.677430+0000","last_active":"2024-01-26T23:46:31.678729+0000","last_peered":"2024-01-26T23:46:31.678729+0000","last_clean":"2024-01-26T23:46:31.678729+0000","last_became_active":"2024-01-26T23:46:31.677151+0000","last_became_peered":"2024-01-26T23:46:31.677151+0000","last_unstale":"2024-01-26T23:46:31.678729+0000","last_undegraded":"2024-01-26T23:46:31.678729+0000","last_fullsized":"2024-01-26T23:46:31.678729+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:15:00.551460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671925+0000","last_change":"2024-01-26T23:46:31.666995+0000","last_active":"2024-01-26T23:46:31.671925+0000","last_peered":"2024-01-26T23:46:31.671925+0000","last_clean":"2024-01-26T23:46:31.671925+0000","last_became_active":"2024-01-26T23:46:31.666507+0000","last_became_peered":"2024-01-26T23:46:31.666507+0000","last_unstale":"2024-01-26T23:46:31.671925+0000","last_undegraded":"2024-01-26T23:46:31.671925+0000","last_fullsized":"2024-01-26T23:46:31.671925+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:35:52.815366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.672072+0000","last_change":"2024-01-26T23:46:31.665567+0000","last_active":"2024-01-26T23:46:31.672072+0000","last_peered":"2024-01-26T23:46:31.672072+0000","last_clean":"2024-01-26T23:46:31.672072+0000","last_became_active":"2024-01-26T23:46:31.665250+0000","last_became_peered":"2024-01-26T23:46:31.665250+0000","last_unstale":"2024-01-26T23:46:31.672072+0000","last_undegraded":"2024-01-26T23:46:31.672072+0000","last_fullsized":"2024-01-26T23:46:31.672072+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:44.847438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671844+0000","last_change":"2024-01-26T23:46:31.666793+0000","last_active":"2024-01-26T23:46:31.671844+0000","last_peered":"2024-01-26T23:46:31.671844+0000","last_clean":"2024-01-26T23:46:31.671844+0000","last_became_active":"2024-01-26T23:46:31.666112+0000","last_became_peered":"2024-01-26T23:46:31.666112+0000","last_unstale":"2024-01-26T23:46:31.671844+0000","last_undegraded":"2024-01-26T23:46:31.671844+0000","last_fullsized":"2024-01-26T23:46:31.671844+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:38.074736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675273+0000","last_change":"2024-01-26T23:46:31.668562+0000","last_active":"2024-01-26T23:46:31.675273+0000","last_peered":"2024-01-26T23:46:31.675273+0000","last_clean":"2024-01-26T23:46:31.675273+0000","last_became_active":"2024-01-26T23:46:31.668253+0000","last_became_peered":"2024-01-26T23:46:31.668253+0000","last_unstale":"2024-01-26T23:46:31.675273+0000","last_undegraded":"2024-01-26T23:46:31.675273+0000","last_fullsized":"2024-01-26T23:46:31.675273+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:43:18.393081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761872+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:06:33.819261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675662+0000","last_change":"2024-01-26T23:46:31.672354+0000","last_active":"2024-01-26T23:46:31.675662+0000","last_peered":"2024-01-26T23:46:31.675662+0000","last_clean":"2024-01-26T23:46:31.675662+0000","last_became_active":"2024-01-26T23:46:31.672029+0000","last_became_peered":"2024-01-26T23:46:31.672029+0000","last_unstale":"2024-01-26T23:46:31.675662+0000","last_undegraded":"2024-01-26T23:46:31.675662+0000","last_fullsized":"2024-01-26T23:46:31.675662+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:16:43.142449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.762969+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:46.577047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760270+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:07.859918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.670354+0000","last_change":"2024-01-26T23:46:31.662791+0000","last_active":"2024-01-26T23:46:31.670354+0000","last_peered":"2024-01-26T23:46:31.670354+0000","last_clean":"2024-01-26T23:46:31.670354+0000","last_became_active":"2024-01-26T23:46:31.662506+0000","last_became_peered":"2024-01-26T23:46:31.662506+0000","last_unstale":"2024-01-26T23:46:31.670354+0000","last_undegraded":"2024-01-26T23:46:31.670354+0000","last_fullsized":"2024-01-26T23:46:31.670354+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:22.391377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761816+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:25:23.538512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761960+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:34:01.252528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760688+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:19:58.164346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678716+0000","last_change":"2024-01-26T23:46:31.677394+0000","last_active":"2024-01-26T23:46:31.678716+0000","last_peered":"2024-01-26T23:46:31.678716+0000","last_clean":"2024-01-26T23:46:31.678716+0000","last_became_active":"2024-01-26T23:46:31.677090+0000","last_became_peered":"2024-01-26T23:46:31.677090+0000","last_unstale":"2024-01-26T23:46:31.678716+0000","last_undegraded":"2024-01-26T23:46:31.678716+0000","last_fullsized":"2024-01-26T23:46:31.678716+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:28:36.539497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.757478+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:16:42.463283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760298+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:16:42.463283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.758550+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:06:33.819261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.757542+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:04:08.622450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.762929+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:37.479741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.14","version":"111'6","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.776077+0000","last_change":"2024-01-26T23:46:30.772469+0000","last_active":"2024-01-26T23:46:31.776077+0000","last_peered":"2024-01-26T23:46:31.776077+0000","last_clean":"2024-01-26T23:46:31.776077+0000","last_became_active":"2024-01-26T23:46:30.772188+0000","last_became_peered":"2024-01-26T23:46:30.772188+0000","last_unstale":"2024-01-26T23:46:31.776077+0000","last_undegraded":"2024-01-26T23:46:31.776077+0000","last_fullsized":"2024-01-26T23:46:31.776077+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:04:26.732373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.759862+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:37.479741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.657510+0000","last_change":"2024-01-26T23:46:30.783543+0000","last_active":"2024-01-26T23:46:31.657510+0000","last_peered":"2024-01-26T23:46:31.657510+0000","last_clean":"2024-01-26T23:46:31.657510+0000","last_became_active":"2024-01-26T23:46:30.774051+0000","last_became_peered":"2024-01-26T23:46:30.774051+0000","last_unstale":"2024-01-26T23:46:31.657510+0000","last_undegraded":"2024-01-26T23:46:31.657510+0000","last_fullsized":"2024-01-26T23:46:31.657510+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:18:11.911206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760169+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:45:02.564138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761051+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:00:54.682539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.672965+0000","last_change":"2024-01-26T23:46:31.667032+0000","last_active":"2024-01-26T23:46:31.672965+0000","last_peered":"2024-01-26T23:46:31.672965+0000","last_clean":"2024-01-26T23:46:31.672965+0000","last_became_active":"2024-01-26T23:46:31.666719+0000","last_became_peered":"2024-01-26T23:46:31.666719+0000","last_unstale":"2024-01-26T23:46:31.672965+0000","last_undegraded":"2024-01-26T23:46:31.672965+0000","last_fullsized":"2024-01-26T23:46:31.672965+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:47:07.666792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761349+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:15:25.108006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.759973+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:46.577047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.762459+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:06:33.819261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671822+0000","last_change":"2024-01-26T23:46:31.665817+0000","last_active":"2024-01-26T23:46:31.671822+0000","last_peered":"2024-01-26T23:46:31.671822+0000","last_clean":"2024-01-26T23:46:31.671822+0000","last_became_active":"2024-01-26T23:46:31.665513+0000","last_became_peered":"2024-01-26T23:46:31.665513+0000","last_unstale":"2024-01-26T23:46:31.671822+0000","last_undegraded":"2024-01-26T23:46:31.671822+0000","last_fullsized":"2024-01-26T23:46:31.671822+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:56:41.826944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.758590+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:25:23.538512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.756936+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:58.710220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.758449+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:15:25.108006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675654+0000","last_change":"2024-01-26T23:46:31.672269+0000","last_active":"2024-01-26T23:46:31.675654+0000","last_peered":"2024-01-26T23:46:31.675654+0000","last_clean":"2024-01-26T23:46:31.675654+0000","last_became_active":"2024-01-26T23:46:31.671924+0000","last_became_peered":"2024-01-26T23:46:31.671924+0000","last_unstale":"2024-01-26T23:46:31.675654+0000","last_undegraded":"2024-01-26T23:46:31.675654+0000","last_fullsized":"2024-01-26T23:46:31.675654+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:36:26.521199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.756907+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:27:38.483382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760450+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:04:08.622450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675423+0000","last_change":"2024-01-26T23:46:31.668761+0000","last_active":"2024-01-26T23:46:31.675423+0000","last_peered":"2024-01-26T23:46:31.675423+0000","last_clean":"2024-01-26T23:46:31.675423+0000","last_became_active":"2024-01-26T23:46:31.668456+0000","last_became_peered":"2024-01-26T23:46:31.668456+0000","last_unstale":"2024-01-26T23:46:31.675423+0000","last_undegraded":"2024-01-26T23:46:31.675423+0000","last_fullsized":"2024-01-26T23:46:31.675423+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:51:54.286344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.762515+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:46:39.085189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675695+0000","last_change":"2024-01-26T23:46:31.672735+0000","last_active":"2024-01-26T23:46:31.675695+0000","last_peered":"2024-01-26T23:46:31.675695+0000","last_clean":"2024-01-26T23:46:31.675695+0000","last_became_active":"2024-01-26T23:46:31.672465+0000","last_became_peered":"2024-01-26T23:46:31.672465+0000","last_unstale":"2024-01-26T23:46:31.675695+0000","last_undegraded":"2024-01-26T23:46:31.675695+0000","last_fullsized":"2024-01-26T23:46:31.675695+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:57:26.854523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.762075+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:25:23.538512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761378+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:34:01.252528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.656731+0000","last_change":"2024-01-26T23:46:30.777287+0000","last_active":"2024-01-26T23:46:31.656731+0000","last_peered":"2024-01-26T23:46:31.656731+0000","last_clean":"2024-01-26T23:46:31.656731+0000","last_became_active":"2024-01-26T23:46:30.773908+0000","last_became_peered":"2024-01-26T23:46:30.773908+0000","last_unstale":"2024-01-26T23:46:31.656731+0000","last_undegraded":"2024-01-26T23:46:31.656731+0000","last_fullsized":"2024-01-26T23:46:31.656731+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:28.962140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.758491+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:34:01.252528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761333+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:28.581902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675370+0000","last_change":"2024-01-26T23:46:31.668096+0000","last_active":"2024-01-26T23:46:31.675370+0000","last_peered":"2024-01-26T23:46:31.675370+0000","last_clean":"2024-01-26T23:46:31.675370+0000","last_became_active":"2024-01-26T23:46:31.665095+0000","last_became_peered":"2024-01-26T23:46:31.665095+0000","last_unstale":"2024-01-26T23:46:31.675370+0000","last_undegraded":"2024-01-26T23:46:31.675370+0000","last_fullsized":"2024-01-26T23:46:31.675370+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:53:38.299979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.759827+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:41:51.840628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760147+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:23:11.224769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"109'4546","reported_seq":4885,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-26T23:46:30.629658+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:46:30.629658+0000","last_peered":"2024-01-26T23:46:30.629658+0000","last_clean":"2024-01-26T23:46:30.629658+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:46:30.629658+0000","last_undegraded":"2024-01-26T23:46:30.629658+0000","last_fullsized":"2024-01-26T23:46:30.629658+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":4546,"log_dups_size":0,"ondisk_log_size":4546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":737,"num_read_kb":1446,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":28,"acting":28,"num_store_stats":1},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":737,"num_read_kb":1446,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4546,"ondisk_log_size":4546,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575050,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311468,"kb_used_data":1892,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060372,"statfs":{"total":96636764160,"available":96317820928,"internally_reserved":0,"allocated":1937408,"data_stored":1271072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.54100000000000004,"15min":0.56499999999999995},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.74099999999999999,"5min":0.79500000000000004,"15min":0.82299999999999995},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.55600000000000005,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.83999999999999997},"last":0.28000000000000003}]},{"osd":1,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.55200000000000005,"15min":0.57899999999999996},"min":{"1min":0.36699999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67400000000000004,"5min":0.81299999999999994,"15min":0.82899999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.34999999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.56299999999999994,"15min":0.57699999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.81699999999999995},"last":0.38},{"interface":"front","average":{"1min":0.504,"5min":0.58299999999999996,"15min":0.59799999999999998},"min":{"1min":0.30199999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75700000000000001,"5min":0.81899999999999995,"15min":0.88600000000000001},"last":0.47299999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.77100000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.58699999999999997,"15min":0.60099999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71699999999999997,"5min":0.83899999999999997,"15min":0.86399999999999999},"last":0.439}]},{"osd":4,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46100000000000002,"15min":0.45600000000000002},"min":{"1min":0.252,"5min":0.247,"15min":0.247},"max":{"1min":0.64300000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.315},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.48799999999999999,"15min":0.48199999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.66400000000000003,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.42999999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.48899999999999999},"min":{"1min":0.27200000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.70599999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54200000000000004,"15min":0.52700000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.73599999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.39100000000000001}]},{"osd":6,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.55300000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78700000000000003,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55700000000000005,"15min":0.54900000000000004},"min":{"1min":0.32000000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70599999999999996,"5min":0.85199999999999998,"15min":0.86899999999999999},"last":0.40799999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575050,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302696,"kb_used_data":1312,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069144,"statfs":{"total":96636764160,"available":96326803456,"internally_reserved":0,"allocated":1343488,"data_stored":680704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55800000000000005,"15min":0.57699999999999996},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26300000000000001},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.255},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.57199999999999995,"15min":0.59199999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.68300000000000005,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.36599999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.54300000000000004,"15min":0.57199999999999995},"min":{"1min":0.317,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.622,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.34399999999999997},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.38700000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55900000000000005,"15min":0.58499999999999996},"min":{"1min":0.39600000000000002,"5min":0.35499999999999998,"15min":0.29599999999999999},"max":{"1min":0.67200000000000004,"5min":0.74199999999999999,"15min":0.85799999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.56999999999999995,"15min":0.58899999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.33900000000000002},"max":{"1min":0.621,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.47899999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.29999999999999999},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60999999999999999,"15min":0.622},"min":{"1min":0.41899999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.47299999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47699999999999998,"15min":0.47299999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.22},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.498,"15min":0.495},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.64600000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.435}]},{"osd":5,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48599999999999999,"15min":0.48499999999999999},"min":{"1min":0.316,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.66100000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54000000000000004,"15min":0.53100000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26200000000000001},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":0.46300000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.246},"max":{"1min":0.64100000000000001,"5min":0.82799999999999996,"15min":0.85899999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.53500000000000003,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.308},"max":{"1min":0.68600000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.44400000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575050,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302696,"kb_used_data":1312,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069144,"statfs":{"total":96636764160,"available":96326803456,"internally_reserved":0,"allocated":1343488,"data_stored":680704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41599999999999998,"15min":0.40899999999999997},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.72599999999999998},"last":0.315},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.44900000000000001,"15min":0.44600000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.78600000000000003,"5min":1.337,"15min":1.337},"last":0.28999999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.41699999999999998,"15min":0.41499999999999998},"min":{"1min":0.254,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.68799999999999994,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.48199999999999998,"15min":0.47499999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.28999999999999998},"max":{"1min":0.74099999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.42199999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52500000000000002,"15min":0.51400000000000001},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":0.75900000000000001,"5min":1.782,"15min":1.782},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.48899999999999999,"15min":0.47999999999999998},"min":{"1min":0.32000000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.71499999999999997,"5min":1.379,"15min":1.379},"last":0.38800000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.64200000000000002,"15min":0.64900000000000002},"min":{"1min":0.435,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":0.81699999999999995,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.65700000000000003,"15min":0.66200000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.84299999999999997,"5min":0.91900000000000004,"15min":0.95699999999999996},"last":0.56100000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.67700000000000005,"15min":0.67300000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":0.86499999999999999,"5min":1.208,"15min":1.208},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.67700000000000005,"15min":0.66800000000000004},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.42499999999999999},"max":{"1min":0.82899999999999996,"5min":0.96599999999999997,"15min":1.018},"last":0.61299999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.68000000000000005,"15min":0.67800000000000005},"min":{"1min":0.47799999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91900000000000004,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.68000000000000005,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.78300000000000003,"5min":1.022,"15min":1.022},"last":0.504}]},{"osd":7,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.70499999999999996,"15min":0.71099999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89200000000000002,"5min":0.98199999999999998,"15min":1.0069999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.73099999999999998,"15min":0.72299999999999998},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.438},"max":{"1min":0.95399999999999996,"5min":1.296,"15min":1.296},"last":0.496}]}]},{"osd":0,"up_from":13,"seq":55834575050,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307432,"kb_used_data":1952,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064408,"statfs":{"total":96636764160,"available":96321953792,"internally_reserved":0,"allocated":1998848,"data_stored":1294736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.41999999999999998,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.45100000000000001,"15min":0.45500000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27300000000000002},"max":{"1min":1.363,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41199999999999998,"15min":0.42499999999999999},"min":{"1min":0.27900000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.57499999999999996,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.45700000000000002,"15min":0.47099999999999997},"min":{"1min":0.32700000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63700000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.36699999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.47399999999999998,"15min":0.46899999999999997},"min":{"1min":0.32300000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.145,"5min":2.145,"15min":2.145},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48599999999999999,"15min":0.48799999999999999},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64300000000000002,"5min":1.198,"15min":1.198},"last":0.32100000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.64400000000000002,"15min":0.65200000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.625,"15min":0.65000000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.752,"5min":0.94299999999999995,"15min":0.98899999999999999},"last":0.42499999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64200000000000002,"15min":0.65900000000000003},"min":{"1min":0.42199999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.67500000000000004,"15min":0.68100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.70799999999999996}]},{"osd":6,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.63,"15min":0.66700000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80200000000000005,"5min":0.89800000000000002,"15min":1.032},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.67400000000000004,"15min":0.69499999999999995},"min":{"1min":0.48299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.61299999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.028,"15min":0.83199999999999996},"min":{"1min":0.501,"5min":0.435,"15min":0.435},"max":{"1min":4.8710000000000004,"5min":25.965,"15min":25.965},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.1379999999999999,"15min":0.875},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.433},"max":{"1min":4.9199999999999999,"5min":33.085000000000001,"15min":33.085000000000001},"last":0.64400000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575049,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306792,"kb_used_data":1312,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065048,"statfs":{"total":96636764160,"available":96322609152,"internally_reserved":0,"allocated":1343488,"data_stored":680704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.51700000000000002,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.437,"15min":0.42299999999999999},"min":{"1min":0.33500000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.308}]},{"osd":1,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.432,"15min":0.42799999999999999},"min":{"1min":0.26900000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.61899999999999999,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.377,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.42299999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46999999999999997,"15min":0.48599999999999999},"min":{"1min":0.35999999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.64800000000000002,"5min":0.65500000000000003,"15min":0.74099999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.48099999999999998,"15min":0.47499999999999998},"min":{"1min":0.35599999999999998,"5min":0.32200000000000001,"15min":0.307},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.45400000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.63200000000000001,"15min":0.63400000000000001},"min":{"1min":0.41499999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.80900000000000005,"5min":1.087,"15min":1.087},"last":0.499},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.65200000000000002,"15min":0.64400000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.874,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.53200000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.65100000000000002,"15min":0.65200000000000002},"min":{"1min":0.375,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.90300000000000002,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.67300000000000004,"15min":0.67900000000000005},"min":{"1min":0.41799999999999998,"5min":0.39800000000000002,"15min":0.35199999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.085},"last":0.52000000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67900000000000005,"15min":0.68100000000000005},"min":{"1min":0.47699999999999998,"5min":0.439,"15min":0.40899999999999997},"max":{"1min":1.022,"5min":1.022,"15min":1.1080000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.67700000000000005,"15min":0.68300000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.38800000000000001},"max":{"1min":0.83799999999999997,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.69499999999999995,"15min":0.70299999999999996},"min":{"1min":0.52100000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94699999999999995,"5min":1.0029999999999999,"15min":1.127},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.72299999999999998,"15min":0.73299999999999998},"min":{"1min":0.48899999999999999,"5min":0.48199999999999998,"15min":0.42099999999999999},"max":{"1min":1.05,"5min":1.052,"15min":1.139},"last":0.58499999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575048,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306780,"kb_used_data":1300,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065060,"statfs":{"total":96636764160,"available":96322621440,"internally_reserved":0,"allocated":1331200,"data_stored":674187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.39900000000000002,"15min":0.39400000000000002},"min":{"1min":0.26500000000000001,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.45000000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.309},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.44,"15min":0.439},"min":{"1min":0.25800000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.58699999999999997,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.38900000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.46600000000000003,"15min":0.45500000000000002},"min":{"1min":0.27900000000000003,"5min":0.253,"15min":0.253},"max":{"1min":0.57799999999999996,"5min":1.633,"15min":1.633},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.48399999999999999,"15min":0.48099999999999998},"min":{"1min":0.35199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.54900000000000004,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.48899999999999999,"15min":0.47899999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.54600000000000004,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.33200000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.53700000000000003,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.33700000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.68200000000000005,"15min":0.66300000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.878,"5min":3.306,"15min":3.306},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.41499999999999998,"5min":0.40200000000000002,"15min":0.379},"max":{"1min":0.85199999999999998,"5min":0.95799999999999996,"15min":1.008},"last":0.55700000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.65700000000000003,"15min":0.66100000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82499999999999996,"5min":1.593,"15min":1.593},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.67500000000000004,"15min":0.67300000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.84399999999999997,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.52300000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.67100000000000004,"15min":0.67700000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.77800000000000002,"5min":1.506,"15min":1.506},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.69099999999999995,"15min":0.69299999999999995},"min":{"1min":0.48799999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.82599999999999996,"5min":1.427,"15min":1.427},"last":0.59599999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.70799999999999996,"15min":0.69799999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.42499999999999999},"max":{"1min":0.73899999999999999,"5min":3.238,"15min":3.238},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.747,"15min":0.73899999999999999},"min":{"1min":0.54100000000000004,"5min":0.48899999999999999,"15min":0.48699999999999999},"max":{"1min":0.86699999999999999,"5min":3.351,"15min":3.351},"last":0.66600000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575050,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302696,"kb_used_data":1312,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069144,"statfs":{"total":96636764160,"available":96326803456,"internally_reserved":0,"allocated":1343488,"data_stored":680704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.54800000000000004,"15min":0.57799999999999996},"min":{"1min":0.28999999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57699999999999996,"5min":1.21,"15min":1.21},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.58499999999999996,"15min":0.58999999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.367,"5min":5.367,"15min":5.367},"last":0.48799999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.55700000000000005,"15min":0.59299999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68899999999999995,"5min":0.77500000000000002,"15min":0.88},"last":0.442},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71499999999999997,"5min":0.80600000000000005,"15min":0.81699999999999995},"last":0.501}]},{"osd":2,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53500000000000003,"15min":0.56799999999999995},"min":{"1min":0.36699999999999999,"5min":0.33300000000000002,"15min":0.32700000000000001},"max":{"1min":0.56599999999999995,"5min":0.81100000000000005,"15min":0.85999999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.58899999999999997,"15min":0.60499999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.69599999999999995,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.50700000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.56000000000000005,"15min":0.58699999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.318},"max":{"1min":0.66900000000000004,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.434},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56699999999999995,"15min":0.58599999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.35699999999999998},"max":{"1min":0.64900000000000002,"5min":1.05,"15min":1.05},"last":0.51300000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.46000000000000002,"15min":0.46300000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.63100000000000001,"5min":0.76200000000000001,"15min":0.77600000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.48499999999999999,"15min":0.48199999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68300000000000005,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.47599999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.47099999999999997,"15min":0.47199999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.27200000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.73899999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.311},"max":{"1min":0.72899999999999998,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.374}]},{"osd":7,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.36699999999999999,"5min":0.36499999999999999,"15min":0.35799999999999998},"max":{"1min":0.753,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.57099999999999995,"15min":0.56899999999999995},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81000000000000005,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.45600000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575050,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298600,"kb_used_data":1312,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073240,"statfs":{"total":96636764160,"available":96330997760,"internally_reserved":0,"allocated":1343488,"data_stored":680704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.26000000000000001},"max":{"1min":0.64700000000000002,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.246},{"interface":"front","average":{"1min":0.502,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.315,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.65800000000000003,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.35999999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.61199999999999999,"15min":0.622},"min":{"1min":0.34799999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66300000000000003,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.629,"15min":0.63400000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.39300000000000002},"max":{"1min":0.71199999999999997,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.46800000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.55900000000000005,"15min":0.56999999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.30099999999999999},"max":{"1min":0.67400000000000004,"5min":0.83699999999999997,"15min":0.83799999999999997},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.58999999999999997,"15min":0.59699999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69699999999999995,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":0.42699999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.57099999999999995,"15min":0.58399999999999996},"min":{"1min":0.38100000000000001,"5min":0.35999999999999999,"15min":0.314},"max":{"1min":0.68799999999999994,"5min":0.86199999999999999,"15min":0.88400000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.59899999999999998,"15min":0.59199999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73599999999999999,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.437}]},{"osd":4,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.48899999999999999,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.48999999999999999,"15min":0.48699999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":0.626,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.41499999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.48799999999999999,"15min":0.47699999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65400000000000003,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.52100000000000002,"15min":0.51900000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.76200000000000001,"5min":1.99,"15min":1.99},"last":0.45200000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.312,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71399999999999997,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.443},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79300000000000004,"5min":0.82199999999999995,"15min":0.83099999999999996},"last":0.45700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:46:36.435 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:46:36.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.604+0000 7fc775072700 1 Processor -- start 2024-01-26T23:46:36.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.604+0000 7fc775072700 1 -- start start 2024-01-26T23:46:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.605+0000 7fc775072700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7701596c0 0x7fc77006bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.605+0000 7fc775072700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc770159a90 0x7fc77006c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.605+0000 7fc775072700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77015d8f0 con 0x7fc77015d460 2024-01-26T23:46:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.605+0000 7fc775072700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7700c98c0 con 0x7fc770159a90 2024-01-26T23:46:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.605+0000 7fc775072700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc77015dbd0 con 0x7fc7701596c0 2024-01-26T23:46:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.605+0000 7fc76f59e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc770159a90 0x7fc77006c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.605+0000 7fc76f59e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc770159a90 0x7fc77006c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33670/0 (socket says 172.21.15.46:33670) 2024-01-26T23:46:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.605+0000 7fc76f59e700 1 -- 172.21.15.46:0/4089867699 learned_addr learned my addr 172.21.15.46:0/4089867699 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.605+0000 7fc76e59c700 1 --1- 172.21.15.46:0/4089867699 >> v1:172.21.15.46:6789/0 conn(0x7fc77015d460 0x7fc77006d310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40892/0 (socket says 172.21.15.46:40892) 2024-01-26T23:46:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.605+0000 7fc76ed9d700 1 --2- 172.21.15.46:0/4089867699 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7701596c0 0x7fc77006bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:36.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.605+0000 7fc76f59e700 1 -- 172.21.15.46:0/4089867699 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7701596c0 msgr2=0x7fc77006bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.605+0000 7fc76f59e700 1 --2- 172.21.15.46:0/4089867699 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7701596c0 0x7fc77006bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.605+0000 7fc76f59e700 1 -- 172.21.15.46:0/4089867699 >> v1:172.21.15.46:6789/0 conn(0x7fc77015d460 legacy=0x7fc77006d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.605+0000 7fc76f59e700 1 -- 172.21.15.46:0/4089867699 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77015deb0 con 0x7fc770159a90 2024-01-26T23:46:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.606+0000 7fc76f59e700 1 --2- 172.21.15.46:0/4089867699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770159a90 0x7fc77006c210 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fc768009f20 tx=0x7fc768009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3305541a7996e5b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.606+0000 7fc76dd9b700 1 -- 172.21.15.46:0/4089867699 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76800e070 con 0x7fc770159a90 2024-01-26T23:46:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.606+0000 7fc76dd9b700 1 -- 172.21.15.46:0/4089867699 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76800b9d0 con 0x7fc770159a90 2024-01-26T23:46:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.606+0000 7fc76dd9b700 1 -- 172.21.15.46:0/4089867699 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7680092e0 con 0x7fc770159a90 2024-01-26T23:46:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.606+0000 7fc775072700 1 -- 172.21.15.46:0/4089867699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770159a90 msgr2=0x7fc77006c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.606+0000 7fc775072700 1 --2- 172.21.15.46:0/4089867699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770159a90 0x7fc77006c210 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fc768009f20 tx=0x7fc768009840 comp rx=0 tx=0).stop 2024-01-26T23:46:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.606+0000 7fc775072700 1 -- 172.21.15.46:0/4089867699 shutdown_connections 2024-01-26T23:46:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.606+0000 7fc775072700 1 --2- 172.21.15.46:0/4089867699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770159a90 0x7fc77006c210 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.606+0000 7fc775072700 1 --2- 172.21.15.46:0/4089867699 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7701596c0 0x7fc77006bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.607+0000 7fc775072700 1 -- 172.21.15.46:0/4089867699 >> 172.21.15.46:0/4089867699 conn(0x7fc7700ce530 msgr2=0x7fc7700ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.607+0000 7fc775072700 1 -- 172.21.15.46:0/4089867699 shutdown_connections 2024-01-26T23:46:36.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.607+0000 7fc775072700 1 -- 172.21.15.46:0/4089867699 wait complete. 2024-01-26T23:46:36.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.608+0000 7fc775072700 1 Processor -- start 2024-01-26T23:46:36.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.608+0000 7fc775072700 1 -- start start 2024-01-26T23:46:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.609+0000 7fc775072700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7701596c0 0x7fc770157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.609+0000 7fc775072700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770159a90 0x7fc77015f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc775072700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc77015d460 0x7fc77015fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc775072700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7700aeb90 con 0x7fc770159a90 2024-01-26T23:46:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc775072700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7700adf80 con 0x7fc7701596c0 2024-01-26T23:46:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc775072700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7700ae260 con 0x7fc77015d460 2024-01-26T23:46:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc76e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770159a90 0x7fc77015f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc76e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770159a90 0x7fc77015f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33676/0 (socket says 172.21.15.46:33676) 2024-01-26T23:46:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc76e59c700 1 -- 172.21.15.46:0/3039833739 learned_addr learned my addr 172.21.15.46:0/3039833739 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc76f59e700 1 --2- 172.21.15.46:0/3039833739 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc77015d460 0x7fc77015fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc76ed9d700 1 --2- 172.21.15.46:0/3039833739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7701596c0 0x7fc770157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc76e59c700 1 -- 172.21.15.46:0/3039833739 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc77015d460 msgr2=0x7fc77015fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc76e59c700 1 --2- 172.21.15.46:0/3039833739 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc77015d460 0x7fc77015fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc76e59c700 1 -- 172.21.15.46:0/3039833739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7701596c0 msgr2=0x7fc770157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc76e59c700 1 --2- 172.21.15.46:0/3039833739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7701596c0 0x7fc770157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc76e59c700 1 -- 172.21.15.46:0/3039833739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc768009ae0 con 0x7fc770159a90 2024-01-26T23:46:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc76f59e700 1 --2- 172.21.15.46:0/3039833739 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc77015d460 0x7fc77015fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc76e59c700 1 --2- 172.21.15.46:0/3039833739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770159a90 0x7fc77015f8b0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fc76400c560 tx=0x7fc764007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc756ffd700 1 -- 172.21.15.46:0/3039833739 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc764005a70 con 0x7fc770159a90 2024-01-26T23:46:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc756ffd700 1 -- 172.21.15.46:0/3039833739 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc764005bd0 con 0x7fc770159a90 2024-01-26T23:46:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc756ffd700 1 -- 172.21.15.46:0/3039833739 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7640070d0 con 0x7fc770159a90 2024-01-26T23:46:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.610+0000 7fc775072700 1 -- 172.21.15.46:0/3039833739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7700ae540 con 0x7fc770159a90 2024-01-26T23:46:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.611+0000 7fc775072700 1 -- 172.21.15.46:0/3039833739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7700af020 con 0x7fc770159a90 2024-01-26T23:46:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.611+0000 7fc775072700 1 -- 172.21.15.46:0/3039833739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7700af240 con 0x7fc770159a90 2024-01-26T23:46:36.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.612+0000 7fc756ffd700 1 -- 172.21.15.46:0/3039833739 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fc764007480 con 0x7fc770159a90 2024-01-26T23:46:36.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.612+0000 7fc756ffd700 1 --2- 172.21.15.46:0/3039833739 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc75c066ea0 0x7fc75c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:36.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.612+0000 7fc756ffd700 1 -- 172.21.15.46:0/3039833739 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fc764083ae0 con 0x7fc770159a90 2024-01-26T23:46:36.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.613+0000 7fc76ed9d700 1 --2- 172.21.15.46:0/3039833739 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc75c066ea0 0x7fc75c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:36.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.613+0000 7fc76ed9d700 1 --2- 172.21.15.46:0/3039833739 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc75c066ea0 0x7fc75c069350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc760009db0 tx=0x7fc7600091a0 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.617+0000 7fc756ffd700 1 -- 172.21.15.46:0/3039833739 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc764050ac0 con 0x7fc770159a90 2024-01-26T23:46:36.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.818+0000 7fc775072700 1 -- 172.21.15.46:0/3039833739 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc77004c710 con 0x7fc75c066ea0 2024-01-26T23:46:36.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.827+0000 7fc756ffd700 1 -- 172.21.15.46:0/3039833739 <== mgr.6365 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+280417 (secure 0 0 0) 0x7fc77004c710 con 0x7fc75c066ea0 2024-01-26T23:46:36.832 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:36.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.830+0000 7fc775072700 1 -- 172.21.15.46:0/3039833739 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc75c066ea0 msgr2=0x7fc75c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:36.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.830+0000 7fc775072700 1 --2- 172.21.15.46:0/3039833739 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc75c066ea0 0x7fc75c069350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc760009db0 tx=0x7fc7600091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.830+0000 7fc775072700 1 -- 172.21.15.46:0/3039833739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770159a90 msgr2=0x7fc77015f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:36.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.830+0000 7fc775072700 1 --2- 172.21.15.46:0/3039833739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770159a90 0x7fc77015f8b0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fc76400c560 tx=0x7fc764007f70 comp rx=0 tx=0).stop 2024-01-26T23:46:36.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.830+0000 7fc775072700 1 -- 172.21.15.46:0/3039833739 shutdown_connections 2024-01-26T23:46:36.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.830+0000 7fc775072700 1 --2- 172.21.15.46:0/3039833739 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc75c066ea0 0x7fc75c069350 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.830+0000 7fc775072700 1 --2- 172.21.15.46:0/3039833739 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc77015d460 0x7fc77015fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.830+0000 7fc775072700 1 --2- 172.21.15.46:0/3039833739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7701596c0 0x7fc770157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.830+0000 7fc775072700 1 --2- 172.21.15.46:0/3039833739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770159a90 0x7fc77015f8b0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:36.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.830+0000 7fc775072700 1 -- 172.21.15.46:0/3039833739 >> 172.21.15.46:0/3039833739 conn(0x7fc7700ce530 msgr2=0x7fc7700cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:36.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.830+0000 7fc775072700 1 -- 172.21.15.46:0/3039833739 shutdown_connections 2024-01-26T23:46:36.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:36.830+0000 7fc775072700 1 -- 172.21.15.46:0/3039833739 wait complete. 2024-01-26T23:46:36.835 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:46:36.845 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-26T23:46:35.636475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591904,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":737,"num_read_kb":1446,"num_write":285,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4552,"ondisk_log_size":4552,"up":94,"acting":94,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":46,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2439160,"kb_used_data":11704,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752535560,"statfs":{"total":773094113280,"available":770596413440,"internally_reserved":0,"allocated":11984896,"data_stored":6643515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.880831"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678729+0000","last_change":"2024-01-26T23:46:31.677430+0000","last_active":"2024-01-26T23:46:31.678729+0000","last_peered":"2024-01-26T23:46:31.678729+0000","last_clean":"2024-01-26T23:46:31.678729+0000","last_became_active":"2024-01-26T23:46:31.677151+0000","last_became_peered":"2024-01-26T23:46:31.677151+0000","last_unstale":"2024-01-26T23:46:31.678729+0000","last_undegraded":"2024-01-26T23:46:31.678729+0000","last_fullsized":"2024-01-26T23:46:31.678729+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:15:00.551460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671925+0000","last_change":"2024-01-26T23:46:31.666995+0000","last_active":"2024-01-26T23:46:31.671925+0000","last_peered":"2024-01-26T23:46:31.671925+0000","last_clean":"2024-01-26T23:46:31.671925+0000","last_became_active":"2024-01-26T23:46:31.666507+0000","last_became_peered":"2024-01-26T23:46:31.666507+0000","last_unstale":"2024-01-26T23:46:31.671925+0000","last_undegraded":"2024-01-26T23:46:31.671925+0000","last_fullsized":"2024-01-26T23:46:31.671925+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:35:52.815366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.672072+0000","last_change":"2024-01-26T23:46:31.665567+0000","last_active":"2024-01-26T23:46:31.672072+0000","last_peered":"2024-01-26T23:46:31.672072+0000","last_clean":"2024-01-26T23:46:31.672072+0000","last_became_active":"2024-01-26T23:46:31.665250+0000","last_became_peered":"2024-01-26T23:46:31.665250+0000","last_unstale":"2024-01-26T23:46:31.672072+0000","last_undegraded":"2024-01-26T23:46:31.672072+0000","last_fullsized":"2024-01-26T23:46:31.672072+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:44.847438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671844+0000","last_change":"2024-01-26T23:46:31.666793+0000","last_active":"2024-01-26T23:46:31.671844+0000","last_peered":"2024-01-26T23:46:31.671844+0000","last_clean":"2024-01-26T23:46:31.671844+0000","last_became_active":"2024-01-26T23:46:31.666112+0000","last_became_peered":"2024-01-26T23:46:31.666112+0000","last_unstale":"2024-01-26T23:46:31.671844+0000","last_undegraded":"2024-01-26T23:46:31.671844+0000","last_fullsized":"2024-01-26T23:46:31.671844+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:38.074736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675273+0000","last_change":"2024-01-26T23:46:31.668562+0000","last_active":"2024-01-26T23:46:31.675273+0000","last_peered":"2024-01-26T23:46:31.675273+0000","last_clean":"2024-01-26T23:46:31.675273+0000","last_became_active":"2024-01-26T23:46:31.668253+0000","last_became_peered":"2024-01-26T23:46:31.668253+0000","last_unstale":"2024-01-26T23:46:31.675273+0000","last_undegraded":"2024-01-26T23:46:31.675273+0000","last_fullsized":"2024-01-26T23:46:31.675273+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:43:18.393081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761872+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:06:33.819261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675662+0000","last_change":"2024-01-26T23:46:31.672354+0000","last_active":"2024-01-26T23:46:31.675662+0000","last_peered":"2024-01-26T23:46:31.675662+0000","last_clean":"2024-01-26T23:46:31.675662+0000","last_became_active":"2024-01-26T23:46:31.672029+0000","last_became_peered":"2024-01-26T23:46:31.672029+0000","last_unstale":"2024-01-26T23:46:31.675662+0000","last_undegraded":"2024-01-26T23:46:31.675662+0000","last_fullsized":"2024-01-26T23:46:31.675662+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:16:43.142449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.762969+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:46.577047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760270+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:07.859918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.670354+0000","last_change":"2024-01-26T23:46:31.662791+0000","last_active":"2024-01-26T23:46:31.670354+0000","last_peered":"2024-01-26T23:46:31.670354+0000","last_clean":"2024-01-26T23:46:31.670354+0000","last_became_active":"2024-01-26T23:46:31.662506+0000","last_became_peered":"2024-01-26T23:46:31.662506+0000","last_unstale":"2024-01-26T23:46:31.670354+0000","last_undegraded":"2024-01-26T23:46:31.670354+0000","last_fullsized":"2024-01-26T23:46:31.670354+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:22.391377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761816+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:25:23.538512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761960+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:34:01.252528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760688+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:19:58.164346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678716+0000","last_change":"2024-01-26T23:46:31.677394+0000","last_active":"2024-01-26T23:46:31.678716+0000","last_peered":"2024-01-26T23:46:31.678716+0000","last_clean":"2024-01-26T23:46:31.678716+0000","last_became_active":"2024-01-26T23:46:31.677090+0000","last_became_peered":"2024-01-26T23:46:31.677090+0000","last_unstale":"2024-01-26T23:46:31.678716+0000","last_undegraded":"2024-01-26T23:46:31.678716+0000","last_fullsized":"2024-01-26T23:46:31.678716+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:28:36.539497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.757478+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:16:42.463283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760298+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:16:42.463283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.758550+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:06:33.819261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.757542+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:04:08.622450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.762929+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:37.479741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.14","version":"111'6","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.776077+0000","last_change":"2024-01-26T23:46:30.772469+0000","last_active":"2024-01-26T23:46:31.776077+0000","last_peered":"2024-01-26T23:46:31.776077+0000","last_clean":"2024-01-26T23:46:31.776077+0000","last_became_active":"2024-01-26T23:46:30.772188+0000","last_became_peered":"2024-01-26T23:46:30.772188+0000","last_unstale":"2024-01-26T23:46:31.776077+0000","last_undegraded":"2024-01-26T23:46:31.776077+0000","last_fullsized":"2024-01-26T23:46:31.776077+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:04:26.732373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.759862+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:37.479741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.657510+0000","last_change":"2024-01-26T23:46:30.783543+0000","last_active":"2024-01-26T23:46:31.657510+0000","last_peered":"2024-01-26T23:46:31.657510+0000","last_clean":"2024-01-26T23:46:31.657510+0000","last_became_active":"2024-01-26T23:46:30.774051+0000","last_became_peered":"2024-01-26T23:46:30.774051+0000","last_unstale":"2024-01-26T23:46:31.657510+0000","last_undegraded":"2024-01-26T23:46:31.657510+0000","last_fullsized":"2024-01-26T23:46:31.657510+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:18:11.911206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760169+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:45:02.564138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761051+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:00:54.682539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.672965+0000","last_change":"2024-01-26T23:46:31.667032+0000","last_active":"2024-01-26T23:46:31.672965+0000","last_peered":"2024-01-26T23:46:31.672965+0000","last_clean":"2024-01-26T23:46:31.672965+0000","last_became_active":"2024-01-26T23:46:31.666719+0000","last_became_peered":"2024-01-26T23:46:31.666719+0000","last_unstale":"2024-01-26T23:46:31.672965+0000","last_undegraded":"2024-01-26T23:46:31.672965+0000","last_fullsized":"2024-01-26T23:46:31.672965+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:47:07.666792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761349+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:15:25.108006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.759973+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:46.577047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.762459+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:06:33.819261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671822+0000","last_change":"2024-01-26T23:46:31.665817+0000","last_active":"2024-01-26T23:46:31.671822+0000","last_peered":"2024-01-26T23:46:31.671822+0000","last_clean":"2024-01-26T23:46:31.671822+0000","last_became_active":"2024-01-26T23:46:31.665513+0000","last_became_peered":"2024-01-26T23:46:31.665513+0000","last_unstale":"2024-01-26T23:46:31.671822+0000","last_undegraded":"2024-01-26T23:46:31.671822+0000","last_fullsized":"2024-01-26T23:46:31.671822+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:56:41.826944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.758590+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:25:23.538512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.756936+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:58.710220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.758449+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:15:25.108006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675654+0000","last_change":"2024-01-26T23:46:31.672269+0000","last_active":"2024-01-26T23:46:31.675654+0000","last_peered":"2024-01-26T23:46:31.675654+0000","last_clean":"2024-01-26T23:46:31.675654+0000","last_became_active":"2024-01-26T23:46:31.671924+0000","last_became_peered":"2024-01-26T23:46:31.671924+0000","last_unstale":"2024-01-26T23:46:31.675654+0000","last_undegraded":"2024-01-26T23:46:31.675654+0000","last_fullsized":"2024-01-26T23:46:31.675654+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:36:26.521199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.756907+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:27:38.483382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760450+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:04:08.622450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675423+0000","last_change":"2024-01-26T23:46:31.668761+0000","last_active":"2024-01-26T23:46:31.675423+0000","last_peered":"2024-01-26T23:46:31.675423+0000","last_clean":"2024-01-26T23:46:31.675423+0000","last_became_active":"2024-01-26T23:46:31.668456+0000","last_became_peered":"2024-01-26T23:46:31.668456+0000","last_unstale":"2024-01-26T23:46:31.675423+0000","last_undegraded":"2024-01-26T23:46:31.675423+0000","last_fullsized":"2024-01-26T23:46:31.675423+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:51:54.286344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.762515+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:46:39.085189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675695+0000","last_change":"2024-01-26T23:46:31.672735+0000","last_active":"2024-01-26T23:46:31.675695+0000","last_peered":"2024-01-26T23:46:31.675695+0000","last_clean":"2024-01-26T23:46:31.675695+0000","last_became_active":"2024-01-26T23:46:31.672465+0000","last_became_peered":"2024-01-26T23:46:31.672465+0000","last_unstale":"2024-01-26T23:46:31.675695+0000","last_undegraded":"2024-01-26T23:46:31.675695+0000","last_fullsized":"2024-01-26T23:46:31.675695+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:57:26.854523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.762075+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:25:23.538512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761378+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:34:01.252528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.656731+0000","last_change":"2024-01-26T23:46:30.777287+0000","last_active":"2024-01-26T23:46:31.656731+0000","last_peered":"2024-01-26T23:46:31.656731+0000","last_clean":"2024-01-26T23:46:31.656731+0000","last_became_active":"2024-01-26T23:46:30.773908+0000","last_became_peered":"2024-01-26T23:46:30.773908+0000","last_unstale":"2024-01-26T23:46:31.656731+0000","last_undegraded":"2024-01-26T23:46:31.656731+0000","last_fullsized":"2024-01-26T23:46:31.656731+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:28.962140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.758491+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:34:01.252528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.761333+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:28.581902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675370+0000","last_change":"2024-01-26T23:46:31.668096+0000","last_active":"2024-01-26T23:46:31.675370+0000","last_peered":"2024-01-26T23:46:31.675370+0000","last_clean":"2024-01-26T23:46:31.675370+0000","last_became_active":"2024-01-26T23:46:31.665095+0000","last_became_peered":"2024-01-26T23:46:31.665095+0000","last_unstale":"2024-01-26T23:46:31.675370+0000","last_undegraded":"2024-01-26T23:46:31.675370+0000","last_fullsized":"2024-01-26T23:46:31.675370+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:53:38.299979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.759827+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:41:51.840628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:46:30.754190+0000","last_change":"2024-01-26T23:46:30.754190+0000","last_active":"2024-01-26T23:46:30.754190+0000","last_peered":"2024-01-26T23:46:30.754190+0000","last_clean":"2024-01-26T23:46:30.754190+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:30.754190+0000","last_undegraded":"2024-01-26T23:46:30.754190+0000","last_fullsized":"2024-01-26T23:46:30.754190+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-26T23:46:29.752847+0000","last_change":"2024-01-26T23:46:29.760147+0000","last_active":"2024-01-26T23:46:29.752847+0000","last_peered":"2024-01-26T23:46:29.752847+0000","last_clean":"2024-01-26T23:46:29.752847+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:46:29.752847+0000","last_undegraded":"2024-01-26T23:46:29.752847+0000","last_fullsized":"2024-01-26T23:46:29.752847+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:23:11.224769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"109'4546","reported_seq":4885,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-26T23:46:30.629658+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:46:30.629658+0000","last_peered":"2024-01-26T23:46:30.629658+0000","last_clean":"2024-01-26T23:46:30.629658+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:46:30.629658+0000","last_undegraded":"2024-01-26T23:46:30.629658+0000","last_fullsized":"2024-01-26T23:46:30.629658+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":4546,"log_dups_size":0,"ondisk_log_size":4546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":737,"num_read_kb":1446,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":28,"acting":28,"num_store_stats":1},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":737,"num_read_kb":1446,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4546,"ondisk_log_size":4546,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575050,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311468,"kb_used_data":1892,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060372,"statfs":{"total":96636764160,"available":96317820928,"internally_reserved":0,"allocated":1937408,"data_stored":1271072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.54100000000000004,"15min":0.56499999999999995},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.74099999999999999,"5min":0.79500000000000004,"15min":0.82299999999999995},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.55600000000000005,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.83999999999999997},"last":0.28000000000000003}]},{"osd":1,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.55200000000000005,"15min":0.57899999999999996},"min":{"1min":0.36699999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67400000000000004,"5min":0.81299999999999994,"15min":0.82899999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.34999999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.56299999999999994,"15min":0.57699999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.81699999999999995},"last":0.38},{"interface":"front","average":{"1min":0.504,"5min":0.58299999999999996,"15min":0.59799999999999998},"min":{"1min":0.30199999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75700000000000001,"5min":0.81899999999999995,"15min":0.88600000000000001},"last":0.47299999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.77100000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.58699999999999997,"15min":0.60099999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71699999999999997,"5min":0.83899999999999997,"15min":0.86399999999999999},"last":0.439}]},{"osd":4,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46100000000000002,"15min":0.45600000000000002},"min":{"1min":0.252,"5min":0.247,"15min":0.247},"max":{"1min":0.64300000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.315},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.48799999999999999,"15min":0.48199999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.66400000000000003,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.42999999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.48899999999999999},"min":{"1min":0.27200000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.70599999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54200000000000004,"15min":0.52700000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.73599999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.39100000000000001}]},{"osd":6,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.55300000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78700000000000003,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55700000000000005,"15min":0.54900000000000004},"min":{"1min":0.32000000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70599999999999996,"5min":0.85199999999999998,"15min":0.86899999999999999},"last":0.40799999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575050,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302696,"kb_used_data":1312,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069144,"statfs":{"total":96636764160,"available":96326803456,"internally_reserved":0,"allocated":1343488,"data_stored":680704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55800000000000005,"15min":0.57699999999999996},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26300000000000001},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.255},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.57199999999999995,"15min":0.59199999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.68300000000000005,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.36599999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.54300000000000004,"15min":0.57199999999999995},"min":{"1min":0.317,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.622,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.34399999999999997},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.38700000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55900000000000005,"15min":0.58499999999999996},"min":{"1min":0.39600000000000002,"5min":0.35499999999999998,"15min":0.29599999999999999},"max":{"1min":0.67200000000000004,"5min":0.74199999999999999,"15min":0.85799999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.56999999999999995,"15min":0.58899999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.33900000000000002},"max":{"1min":0.621,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.47899999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.29999999999999999},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60999999999999999,"15min":0.622},"min":{"1min":0.41899999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.47299999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47699999999999998,"15min":0.47299999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.22},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.498,"15min":0.495},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.64600000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.435}]},{"osd":5,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48599999999999999,"15min":0.48499999999999999},"min":{"1min":0.316,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.66100000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54000000000000004,"15min":0.53100000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26200000000000001},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":0.46300000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.246},"max":{"1min":0.64100000000000001,"5min":0.82799999999999996,"15min":0.85899999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.53500000000000003,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.308},"max":{"1min":0.68600000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.44400000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575050,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302696,"kb_used_data":1312,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069144,"statfs":{"total":96636764160,"available":96326803456,"internally_reserved":0,"allocated":1343488,"data_stored":680704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41599999999999998,"15min":0.40899999999999997},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.72599999999999998},"last":0.315},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.44900000000000001,"15min":0.44600000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.78600000000000003,"5min":1.337,"15min":1.337},"last":0.28999999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.41699999999999998,"15min":0.41499999999999998},"min":{"1min":0.254,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.68799999999999994,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.48199999999999998,"15min":0.47499999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.28999999999999998},"max":{"1min":0.74099999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.42199999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52500000000000002,"15min":0.51400000000000001},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":0.75900000000000001,"5min":1.782,"15min":1.782},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.48899999999999999,"15min":0.47999999999999998},"min":{"1min":0.32000000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.71499999999999997,"5min":1.379,"15min":1.379},"last":0.38800000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.64200000000000002,"15min":0.64900000000000002},"min":{"1min":0.435,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":0.81699999999999995,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.65700000000000003,"15min":0.66200000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.84299999999999997,"5min":0.91900000000000004,"15min":0.95699999999999996},"last":0.56100000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.67700000000000005,"15min":0.67300000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":0.86499999999999999,"5min":1.208,"15min":1.208},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.67700000000000005,"15min":0.66800000000000004},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.42499999999999999},"max":{"1min":0.82899999999999996,"5min":0.96599999999999997,"15min":1.018},"last":0.61299999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.68000000000000005,"15min":0.67800000000000005},"min":{"1min":0.47799999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91900000000000004,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.68000000000000005,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.78300000000000003,"5min":1.022,"15min":1.022},"last":0.504}]},{"osd":7,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.70499999999999996,"15min":0.71099999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89200000000000002,"5min":0.98199999999999998,"15min":1.0069999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.73099999999999998,"15min":0.72299999999999998},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.438},"max":{"1min":0.95399999999999996,"5min":1.296,"15min":1.296},"last":0.496}]}]},{"osd":0,"up_from":13,"seq":55834575050,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307432,"kb_used_data":1952,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064408,"statfs":{"total":96636764160,"available":96321953792,"internally_reserved":0,"allocated":1998848,"data_stored":1294736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.41999999999999998,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.45100000000000001,"15min":0.45500000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27300000000000002},"max":{"1min":1.363,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41199999999999998,"15min":0.42499999999999999},"min":{"1min":0.27900000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.57499999999999996,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.45700000000000002,"15min":0.47099999999999997},"min":{"1min":0.32700000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63700000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.36699999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.47399999999999998,"15min":0.46899999999999997},"min":{"1min":0.32300000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.145,"5min":2.145,"15min":2.145},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48599999999999999,"15min":0.48799999999999999},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64300000000000002,"5min":1.198,"15min":1.198},"last":0.32100000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.64400000000000002,"15min":0.65200000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.625,"15min":0.65000000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.752,"5min":0.94299999999999995,"15min":0.98899999999999999},"last":0.42499999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64200000000000002,"15min":0.65900000000000003},"min":{"1min":0.42199999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.67500000000000004,"15min":0.68100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.70799999999999996}]},{"osd":6,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.63,"15min":0.66700000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80200000000000005,"5min":0.89800000000000002,"15min":1.032},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.67400000000000004,"15min":0.69499999999999995},"min":{"1min":0.48299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.61299999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.028,"15min":0.83199999999999996},"min":{"1min":0.501,"5min":0.435,"15min":0.435},"max":{"1min":4.8710000000000004,"5min":25.965,"15min":25.965},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.1379999999999999,"15min":0.875},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.433},"max":{"1min":4.9199999999999999,"5min":33.085000000000001,"15min":33.085000000000001},"last":0.64400000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575049,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306792,"kb_used_data":1312,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065048,"statfs":{"total":96636764160,"available":96322609152,"internally_reserved":0,"allocated":1343488,"data_stored":680704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.51700000000000002,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.437,"15min":0.42299999999999999},"min":{"1min":0.33500000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.308}]},{"osd":1,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.432,"15min":0.42799999999999999},"min":{"1min":0.26900000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.61899999999999999,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.377,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.42299999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46999999999999997,"15min":0.48599999999999999},"min":{"1min":0.35999999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.64800000000000002,"5min":0.65500000000000003,"15min":0.74099999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.48099999999999998,"15min":0.47499999999999998},"min":{"1min":0.35599999999999998,"5min":0.32200000000000001,"15min":0.307},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.45400000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.63200000000000001,"15min":0.63400000000000001},"min":{"1min":0.41499999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.80900000000000005,"5min":1.087,"15min":1.087},"last":0.499},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.65200000000000002,"15min":0.64400000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.874,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.53200000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.65100000000000002,"15min":0.65200000000000002},"min":{"1min":0.375,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.90300000000000002,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.67300000000000004,"15min":0.67900000000000005},"min":{"1min":0.41799999999999998,"5min":0.39800000000000002,"15min":0.35199999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.085},"last":0.52000000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67900000000000005,"15min":0.68100000000000005},"min":{"1min":0.47699999999999998,"5min":0.439,"15min":0.40899999999999997},"max":{"1min":1.022,"5min":1.022,"15min":1.1080000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.67700000000000005,"15min":0.68300000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.38800000000000001},"max":{"1min":0.83799999999999997,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.69499999999999995,"15min":0.70299999999999996},"min":{"1min":0.52100000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94699999999999995,"5min":1.0029999999999999,"15min":1.127},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.72299999999999998,"15min":0.73299999999999998},"min":{"1min":0.48899999999999999,"5min":0.48199999999999998,"15min":0.42099999999999999},"max":{"1min":1.05,"5min":1.052,"15min":1.139},"last":0.58499999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575048,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306780,"kb_used_data":1300,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065060,"statfs":{"total":96636764160,"available":96322621440,"internally_reserved":0,"allocated":1331200,"data_stored":674187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.39900000000000002,"15min":0.39400000000000002},"min":{"1min":0.26500000000000001,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.45000000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.309},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.44,"15min":0.439},"min":{"1min":0.25800000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.58699999999999997,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.38900000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.46600000000000003,"15min":0.45500000000000002},"min":{"1min":0.27900000000000003,"5min":0.253,"15min":0.253},"max":{"1min":0.57799999999999996,"5min":1.633,"15min":1.633},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.48399999999999999,"15min":0.48099999999999998},"min":{"1min":0.35199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.54900000000000004,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.48899999999999999,"15min":0.47899999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.54600000000000004,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.33200000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.53700000000000003,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.33700000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.68200000000000005,"15min":0.66300000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.878,"5min":3.306,"15min":3.306},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.41499999999999998,"5min":0.40200000000000002,"15min":0.379},"max":{"1min":0.85199999999999998,"5min":0.95799999999999996,"15min":1.008},"last":0.55700000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.65700000000000003,"15min":0.66100000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82499999999999996,"5min":1.593,"15min":1.593},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.67500000000000004,"15min":0.67300000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.84399999999999997,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.52300000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.67100000000000004,"15min":0.67700000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.77800000000000002,"5min":1.506,"15min":1.506},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.69099999999999995,"15min":0.69299999999999995},"min":{"1min":0.48799999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.82599999999999996,"5min":1.427,"15min":1.427},"last":0.59599999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.70799999999999996,"15min":0.69799999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.42499999999999999},"max":{"1min":0.73899999999999999,"5min":3.238,"15min":3.238},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.747,"15min":0.73899999999999999},"min":{"1min":0.54100000000000004,"5min":0.48899999999999999,"15min":0.48699999999999999},"max":{"1min":0.86699999999999999,"5min":3.351,"15min":3.351},"last":0.66600000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575050,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302696,"kb_used_data":1312,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069144,"statfs":{"total":96636764160,"available":96326803456,"internally_reserved":0,"allocated":1343488,"data_stored":680704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.54800000000000004,"15min":0.57799999999999996},"min":{"1min":0.28999999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57699999999999996,"5min":1.21,"15min":1.21},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.58499999999999996,"15min":0.58999999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.367,"5min":5.367,"15min":5.367},"last":0.48799999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.55700000000000005,"15min":0.59299999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68899999999999995,"5min":0.77500000000000002,"15min":0.88},"last":0.442},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71499999999999997,"5min":0.80600000000000005,"15min":0.81699999999999995},"last":0.501}]},{"osd":2,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53500000000000003,"15min":0.56799999999999995},"min":{"1min":0.36699999999999999,"5min":0.33300000000000002,"15min":0.32700000000000001},"max":{"1min":0.56599999999999995,"5min":0.81100000000000005,"15min":0.85999999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.58899999999999997,"15min":0.60499999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.69599999999999995,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.50700000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.56000000000000005,"15min":0.58699999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.318},"max":{"1min":0.66900000000000004,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.434},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56699999999999995,"15min":0.58599999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.35699999999999998},"max":{"1min":0.64900000000000002,"5min":1.05,"15min":1.05},"last":0.51300000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.46000000000000002,"15min":0.46300000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.63100000000000001,"5min":0.76200000000000001,"15min":0.77600000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.48499999999999999,"15min":0.48199999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68300000000000005,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.47599999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.47099999999999997,"15min":0.47199999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.27200000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.73899999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.311},"max":{"1min":0.72899999999999998,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.374}]},{"osd":7,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.36699999999999999,"5min":0.36499999999999999,"15min":0.35799999999999998},"max":{"1min":0.753,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.57099999999999995,"15min":0.56899999999999995},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81000000000000005,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.45600000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575050,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298600,"kb_used_data":1312,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073240,"statfs":{"total":96636764160,"available":96330997760,"internally_reserved":0,"allocated":1343488,"data_stored":680704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.26000000000000001},"max":{"1min":0.64700000000000002,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.246},{"interface":"front","average":{"1min":0.502,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.315,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.65800000000000003,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.35999999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.61199999999999999,"15min":0.622},"min":{"1min":0.34799999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66300000000000003,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.629,"15min":0.63400000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.39300000000000002},"max":{"1min":0.71199999999999997,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.46800000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.55900000000000005,"15min":0.56999999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.30099999999999999},"max":{"1min":0.67400000000000004,"5min":0.83699999999999997,"15min":0.83799999999999997},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.58999999999999997,"15min":0.59699999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69699999999999995,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":0.42699999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.57099999999999995,"15min":0.58399999999999996},"min":{"1min":0.38100000000000001,"5min":0.35999999999999999,"15min":0.314},"max":{"1min":0.68799999999999994,"5min":0.86199999999999999,"15min":0.88400000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.59899999999999998,"15min":0.59199999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73599999999999999,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.437}]},{"osd":4,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.48899999999999999,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.48999999999999999,"15min":0.48699999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":0.626,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.41499999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.48799999999999999,"15min":0.47699999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65400000000000003,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.52100000000000002,"15min":0.51900000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.76200000000000001,"5min":1.99,"15min":1.99},"last":0.45200000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.312,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71399999999999997,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.443},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79300000000000004,"5min":0.82199999999999995,"15min":0.83099999999999996},"last":0.45700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:46:36.850 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3f is not active+clean 2024-01-26T23:46:36.850 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.850 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3e is not active+clean 2024-01-26T23:46:36.850 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.850 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3d is not active+clean 2024-01-26T23:46:36.850 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.851 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3c is not active+clean 2024-01-26T23:46:36.851 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.851 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3b is not active+clean 2024-01-26T23:46:36.851 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.851 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.39 is not active+clean 2024-01-26T23:46:36.851 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.851 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.38 is not active+clean 2024-01-26T23:46:36.851 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.852 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.37 is not active+clean 2024-01-26T23:46:36.852 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.852 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.36 is not active+clean 2024-01-26T23:46:36.852 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.852 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.35 is not active+clean 2024-01-26T23:46:36.852 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.852 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.34 is not active+clean 2024-01-26T23:46:36.852 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.852 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.33 is not active+clean 2024-01-26T23:46:36.853 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.853 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.32 is not active+clean 2024-01-26T23:46:36.853 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.853 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.31 is not active+clean 2024-01-26T23:46:36.853 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.853 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2e is not active+clean 2024-01-26T23:46:36.853 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.853 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2d is not active+clean 2024-01-26T23:46:36.853 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.853 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2c is not active+clean 2024-01-26T23:46:36.853 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.854 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2b is not active+clean 2024-01-26T23:46:36.854 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.854 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2a is not active+clean 2024-01-26T23:46:36.854 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.854 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.29 is not active+clean 2024-01-26T23:46:36.854 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.854 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.28 is not active+clean 2024-01-26T23:46:36.854 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.854 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.27 is not active+clean 2024-01-26T23:46:36.854 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.854 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.26 is not active+clean 2024-01-26T23:46:36.855 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.855 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.25 is not active+clean 2024-01-26T23:46:36.855 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.855 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.24 is not active+clean 2024-01-26T23:46:36.855 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.855 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.23 is not active+clean 2024-01-26T23:46:36.855 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.855 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.22 is not active+clean 2024-01-26T23:46:36.855 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.855 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.21 is not active+clean 2024-01-26T23:46:36.855 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.856 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1f is not active+clean 2024-01-26T23:46:36.856 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.761872+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T05:06:33.819261+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-26T23:46:36.856 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1e is not active+clean 2024-01-26T23:46:36.856 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.762969+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T00:34:46.577047+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-26T23:46:36.856 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1e is not active+clean 2024-01-26T23:46:36.856 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.856 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1d is not active+clean 2024-01-26T23:46:36.856 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.760270+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T00:45:07.859918+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-26T23:46:36.856 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1c is not active+clean 2024-01-26T23:46:36.856 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.761816+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T11:25:23.538512+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-26T23:46:36.856 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.18 is not active+clean 2024-01-26T23:46:36.857 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.857 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1b is not active+clean 2024-01-26T23:46:36.857 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.761960+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T07:34:01.252528+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-26T23:46:36.857 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.19 is not active+clean 2024-01-26T23:46:36.857 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.857 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1a is not active+clean 2024-01-26T23:46:36.857 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.760688+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T05:19:58.164346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-26T23:46:36.857 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.19 is not active+clean 2024-01-26T23:46:36.857 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.757478+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:16:42.463283+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-26T23:46:36.857 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1b is not active+clean 2024-01-26T23:46:36.857 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.858 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.18 is not active+clean 2024-01-26T23:46:36.858 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.760298+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:16:42.463283+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-26T23:46:36.858 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.14 is not active+clean 2024-01-26T23:46:36.858 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.858 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.17 is not active+clean 2024-01-26T23:46:36.858 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.758550+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T05:06:33.819261+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-26T23:46:36.858 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.15 is not active+clean 2024-01-26T23:46:36.858 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.858 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.16 is not active+clean 2024-01-26T23:46:36.858 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.757542+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T01:04:08.622450+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-26T23:46:36.858 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.16 is not active+clean 2024-01-26T23:46:36.858 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.859 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.15 is not active+clean 2024-01-26T23:46:36.859 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.762929+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T04:24:37.479741+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-26T23:46:36.859 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.17 is not active+clean 2024-01-26T23:46:36.859 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.859 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.10 is not active+clean 2024-01-26T23:46:36.859 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.859 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.13 is not active+clean 2024-01-26T23:46:36.859 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.759862+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T04:24:37.479741+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-26T23:46:36.859 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.11 is not active+clean 2024-01-26T23:46:36.860 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.860 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.12 is not active+clean 2024-01-26T23:46:36.860 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.860 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.11 is not active+clean 2024-01-26T23:46:36.860 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.760169+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T08:45:02.564138+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-26T23:46:36.860 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.13 is not active+clean 2024-01-26T23:46:36.860 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.860 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.10 is not active+clean 2024-01-26T23:46:36.860 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.761051+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T10:00:54.682539+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-26T23:46:36.860 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.f is not active+clean 2024-01-26T23:46:36.860 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.761349+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T02:15:25.108006+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-26T23:46:36.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.d is not active+clean 2024-01-26T23:46:36.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.e is not active+clean 2024-01-26T23:46:36.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.759973+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T00:34:46.577047+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-26T23:46:36.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.e is not active+clean 2024-01-26T23:46:36.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.d is not active+clean 2024-01-26T23:46:36.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.762459+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T05:06:33.819261+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-26T23:46:36.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.b is not active+clean 2024-01-26T23:46:36.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.758590+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T11:25:23.538512+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-26T23:46:36.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.9 is not active+clean 2024-01-26T23:46:36.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.a is not active+clean 2024-01-26T23:46:36.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.756936+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T10:05:58.710220+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-26T23:46:36.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.a is not active+clean 2024-01-26T23:46:36.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.9 is not active+clean 2024-01-26T23:46:36.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.758449+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T02:15:25.108006+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-26T23:46:36.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.8 is not active+clean 2024-01-26T23:46:36.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.756907+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:27:38.483382+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-26T23:46:36.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.4 is not active+clean 2024-01-26T23:46:36.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.7 is not active+clean 2024-01-26T23:46:36.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.760450+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T01:04:08.622450+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-26T23:46:36.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.6 is not active+clean 2024-01-26T23:46:36.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.762515+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T02:46:39.085189+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-26T23:46:36.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.5 is not active+clean 2024-01-26T23:46:36.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.762075+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T11:25:23.538512+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-26T23:46:36.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.7 is not active+clean 2024-01-26T23:46:36.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.4 is not active+clean 2024-01-26T23:46:36.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.761378+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T07:34:01.252528+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-26T23:46:36.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.0 is not active+clean 2024-01-26T23:46:36.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1 is not active+clean 2024-01-26T23:46:36.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2 is not active+clean 2024-01-26T23:46:36.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.758491+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T07:34:01.252528+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-26T23:46:36.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2 is not active+clean 2024-01-26T23:46:36.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1 is not active+clean 2024-01-26T23:46:36.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.761333+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T02:19:28.581902+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-26T23:46:36.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.0 is not active+clean 2024-01-26T23:46:36.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.759827+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T03:41:51.840628+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-26T23:46:36.865 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.f is not active+clean 2024-01-26T23:46:36.865 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-26T23:46:30.754190+0000', 'last_change': '2024-01-26T23:46:30.754190+0000', 'last_active': '2024-01-26T23:46:30.754190+0000', 'last_peered': '2024-01-26T23:46:30.754190+0000', 'last_clean': '2024-01-26T23:46:30.754190+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:30.754190+0000', 'last_undegraded': '2024-01-26T23:46:30.754190+0000', 'last_fullsized': '2024-01-26T23:46:30.754190+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:30.754190+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-26T23:46:36.865 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.c is not active+clean 2024-01-26T23:46:36.865 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-26T23:46:29.752847+0000', 'last_change': '2024-01-26T23:46:29.760147+0000', 'last_active': '2024-01-26T23:46:29.752847+0000', 'last_peered': '2024-01-26T23:46:29.752847+0000', 'last_clean': '2024-01-26T23:46:29.752847+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-26T23:46:29.752847+0000', 'last_undegraded': '2024-01-26T23:46:29.752847+0000', 'last_fullsized': '2024-01-26T23:46:29.752847+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'last_clean_scrub_stamp': '2024-01-26T23:46:29.752847+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T09:23:11.224769+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-26T23:46:36.865 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-26T23:46:37.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.020+0000 7fa52c5dc700 1 Processor -- start 2024-01-26T23:46:37.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.020+0000 7fa52c5dc700 1 -- start start 2024-01-26T23:46:37.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.021+0000 7fa52c5dc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5240cb5a0 0x7fa5240ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:37.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.021+0000 7fa52c5dc700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5240b0df0 con 0x7fa5240c4410 2024-01-26T23:46:37.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.021+0000 7fa52c5dc700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5240c9df0 con 0x7fa5240b0a20 2024-01-26T23:46:37.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.021+0000 7fa52c5dc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa524157c00 con 0x7fa5240cb5a0 2024-01-26T23:46:37.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.021+0000 7fa52ab79700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa5240c4410 0x7fa524161300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40894/0 (socket says 172.21.15.46:40894) 2024-01-26T23:46:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.021+0000 7fa52ab79700 1 -- 172.21.15.46:0/1349568236 learned_addr learned my addr 172.21.15.46:0/1349568236 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.021+0000 7fa529b77700 1 --2- 172.21.15.46:0/1349568236 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5240cb5a0 0x7fa5240ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.021+0000 7fa529376700 1 -- 172.21.15.46:0/1349568236 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776333212 0 0) 0x7fa5240b0df0 con 0x7fa5240c4410 2024-01-26T23:46:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.021+0000 7fa529376700 1 -- 172.21.15.46:0/1349568236 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa510003570 con 0x7fa5240c4410 2024-01-26T23:46:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.021+0000 7fa529376700 1 -- 172.21.15.46:0/1349568236 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667180314 0 0) 0x7fa5240c9df0 con 0x7fa5240b0a20 2024-01-26T23:46:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.021+0000 7fa529376700 1 -- 172.21.15.46:0/1349568236 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5240b0df0 con 0x7fa5240b0a20 2024-01-26T23:46:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.021+0000 7fa529b77700 1 -- 172.21.15.46:0/1349568236 >> v1:172.21.15.106:6789/0 conn(0x7fa5240b0a20 legacy=0x7fa5240cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.021+0000 7fa529b77700 1 -- 172.21.15.46:0/1349568236 >> v1:172.21.15.46:6789/0 conn(0x7fa5240c4410 legacy=0x7fa524161300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.021+0000 7fa529b77700 1 -- 172.21.15.46:0/1349568236 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5240ca0a0 con 0x7fa5240cb5a0 2024-01-26T23:46:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.022+0000 7fa529b77700 1 --2- 172.21.15.46:0/1349568236 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5240cb5a0 0x7fa5240ccc00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa514009910 tx=0x7fa5140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30e5a0cd23aa17b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.022+0000 7fa529376700 1 -- 172.21.15.46:0/1349568236 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa514011630 con 0x7fa5240cb5a0 2024-01-26T23:46:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.022+0000 7fa529376700 1 -- 172.21.15.46:0/1349568236 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa514011790 con 0x7fa5240cb5a0 2024-01-26T23:46:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.022+0000 7fa529376700 1 -- 172.21.15.46:0/1349568236 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa514011b10 con 0x7fa5240cb5a0 2024-01-26T23:46:37.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.022+0000 7fa52c5dc700 1 -- 172.21.15.46:0/1349568236 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5240cb5a0 msgr2=0x7fa5240ccc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:37.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.022+0000 7fa52c5dc700 1 --2- 172.21.15.46:0/1349568236 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5240cb5a0 0x7fa5240ccc00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa514009910 tx=0x7fa5140091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.023+0000 7fa52c5dc700 1 -- 172.21.15.46:0/1349568236 shutdown_connections 2024-01-26T23:46:37.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.023+0000 7fa52c5dc700 1 --2- 172.21.15.46:0/1349568236 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5240cb5a0 0x7fa5240ccc00 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.023+0000 7fa52c5dc700 1 -- 172.21.15.46:0/1349568236 >> 172.21.15.46:0/1349568236 conn(0x7fa524097820 msgr2=0x7fa524068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:37.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.023+0000 7fa52c5dc700 1 -- 172.21.15.46:0/1349568236 shutdown_connections 2024-01-26T23:46:37.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.023+0000 7fa52c5dc700 1 -- 172.21.15.46:0/1349568236 wait complete. 2024-01-26T23:46:37.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.024+0000 7fa52c5dc700 1 Processor -- start 2024-01-26T23:46:37.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.024+0000 7fa52c5dc700 1 -- start start 2024-01-26T23:46:37.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.025+0000 7fa52c5dc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5240b0a20 0x7fa524153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:37.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.025+0000 7fa52c5dc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5240c4410 0x7fa5240b1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:37.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.025+0000 7fa52c5dc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5240cb5a0 0x7fa5240b1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:37.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.025+0000 7fa52c5dc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5241539a0 con 0x7fa5240cb5a0 2024-01-26T23:46:37.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.025+0000 7fa52c5dc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5240b1e70 con 0x7fa5240c4410 2024-01-26T23:46:37.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.025+0000 7fa52c5dc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5240b2150 con 0x7fa5240b0a20 2024-01-26T23:46:37.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.025+0000 7fa52ab79700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5240cb5a0 0x7fa5240b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:37.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.025+0000 7fa529b77700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5240c4410 0x7fa5240b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:37.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.025+0000 7fa52ab79700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5240cb5a0 0x7fa5240b1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33692/0 (socket says 172.21.15.46:33692) 2024-01-26T23:46:37.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.025+0000 7fa52a378700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5240b0a20 0x7fa524153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:37.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.025+0000 7fa52ab79700 1 -- 172.21.15.46:0/4217858584 learned_addr learned my addr 172.21.15.46:0/4217858584 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:37.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.025+0000 7fa52ab79700 1 -- 172.21.15.46:0/4217858584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5240b0a20 msgr2=0x7fa524153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:37.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.025+0000 7fa52ab79700 1 --2- 172.21.15.46:0/4217858584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5240b0a20 0x7fa524153460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.025+0000 7fa52ab79700 1 -- 172.21.15.46:0/4217858584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5240c4410 msgr2=0x7fa5240b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:37.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.025+0000 7fa52ab79700 1 --2- 172.21.15.46:0/4217858584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5240c4410 0x7fa5240b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.026+0000 7fa52ab79700 1 -- 172.21.15.46:0/4217858584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa514009420 con 0x7fa5240cb5a0 2024-01-26T23:46:37.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.026+0000 7fa52ab79700 1 --2- 172.21.15.46:0/4217858584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5240cb5a0 0x7fa5240b1840 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fa51c00cc20 tx=0x7fa51c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:37.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.026+0000 7fa51a7fc700 1 -- 172.21.15.46:0/4217858584 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c00a850 con 0x7fa5240cb5a0 2024-01-26T23:46:37.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.026+0000 7fa51a7fc700 1 -- 172.21.15.46:0/4217858584 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51c00a9d0 con 0x7fa5240cb5a0 2024-01-26T23:46:37.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.026+0000 7fa51a7fc700 1 -- 172.21.15.46:0/4217858584 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c00ad90 con 0x7fa5240cb5a0 2024-01-26T23:46:37.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.026+0000 7fa52c5dc700 1 -- 172.21.15.46:0/4217858584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5240b2430 con 0x7fa5240cb5a0 2024-01-26T23:46:37.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.026+0000 7fa52c5dc700 1 -- 172.21.15.46:0/4217858584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5240b2a70 con 0x7fa5240cb5a0 2024-01-26T23:46:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.027+0000 7fa52c5dc700 1 -- 172.21.15.46:0/4217858584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5240b2c90 con 0x7fa5240cb5a0 2024-01-26T23:46:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.028+0000 7fa51a7fc700 1 -- 172.21.15.46:0/4217858584 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fa51c007300 con 0x7fa5240cb5a0 2024-01-26T23:46:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.028+0000 7fa51a7fc700 1 --2- 172.21.15.46:0/4217858584 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa508066ea0 0x7fa508069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.028+0000 7fa51a7fc700 1 -- 172.21.15.46:0/4217858584 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fa51c084940 con 0x7fa5240cb5a0 2024-01-26T23:46:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.028+0000 7fa52a378700 1 --2- 172.21.15.46:0/4217858584 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa508066ea0 0x7fa508069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.029+0000 7fa52a378700 1 --2- 172.21.15.46:0/4217858584 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa508066ea0 0x7fa508069350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa520001130 tx=0x7fa520007d00 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:37.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.033+0000 7fa51a7fc700 1 -- 172.21.15.46:0/4217858584 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa51c051a40 con 0x7fa5240cb5a0 2024-01-26T23:46:37.184 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:37.183+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:37.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.325+0000 7fa52c5dc700 1 -- 172.21.15.46:0/4217858584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa52404c710 con 0x7fa5240cb5a0 2024-01-26T23:46:37.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.325+0000 7fa51a7fc700 1 -- 172.21.15.46:0/4217858584 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1199 (secure 0 0 0) 0x7fa51c055060 con 0x7fa5240cb5a0 2024-01-26T23:46:37.327 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:37.327 INFO:teuthology.orchestra.run.smithi046.stdout:{"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1030,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":112,"num_osds":8,"num_up_osds":8,"osd_up_since":1706311777,"num_in_osds":8,"osd_in_since":1706311769,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":50},{"state_name":"creating+peering","count":29},{"state_name":"active+clean","count":18}],"num_pgs":97,"num_pools":3,"num_objects":6,"data_bytes":591904,"bytes_used":2497699840,"bytes_avail":770596413440,"bytes_total":773094113280,"unknown_pgs_ratio":0.51546388864517212,"inactive_pgs_ratio":0.29896906018257141,"read_bytes_sec":1044,"write_bytes_sec":522,"read_op_per_sec":1,"write_op_per_sec":1},"fsmap":{"epoch":54,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"d","status":"up:active","gid":7084}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-26T23:44:22.571535+0000","services":{}},"progress_events":{}} 2024-01-26T23:46:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.327+0000 7fa52c5dc700 1 -- 172.21.15.46:0/4217858584 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa508066ea0 msgr2=0x7fa508069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.327+0000 7fa52c5dc700 1 --2- 172.21.15.46:0/4217858584 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa508066ea0 0x7fa508069350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa520001130 tx=0x7fa520007d00 comp rx=0 tx=0).stop 2024-01-26T23:46:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.327+0000 7fa52c5dc700 1 -- 172.21.15.46:0/4217858584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5240cb5a0 msgr2=0x7fa5240b1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.327+0000 7fa52c5dc700 1 --2- 172.21.15.46:0/4217858584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5240cb5a0 0x7fa5240b1840 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fa51c00cc20 tx=0x7fa51c001a40 comp rx=0 tx=0).stop 2024-01-26T23:46:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.327+0000 7fa52c5dc700 1 -- 172.21.15.46:0/4217858584 shutdown_connections 2024-01-26T23:46:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.327+0000 7fa52c5dc700 1 --2- 172.21.15.46:0/4217858584 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa508066ea0 0x7fa508069350 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.327+0000 7fa52c5dc700 1 --2- 172.21.15.46:0/4217858584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5240cb5a0 0x7fa5240b1840 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.327+0000 7fa52c5dc700 1 --2- 172.21.15.46:0/4217858584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5240b0a20 0x7fa524153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.327+0000 7fa52c5dc700 1 --2- 172.21.15.46:0/4217858584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5240c4410 0x7fa5240b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.327+0000 7fa52c5dc700 1 -- 172.21.15.46:0/4217858584 >> 172.21.15.46:0/4217858584 conn(0x7fa524097820 msgr2=0x7fa5240c93d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:37.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.328+0000 7fa52c5dc700 1 -- 172.21.15.46:0/4217858584 shutdown_connections 2024-01-26T23:46:37.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.328+0000 7fa52c5dc700 1 -- 172.21.15.46:0/4217858584 wait complete. 2024-01-26T23:46:37.343 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-26T23:46:37.343 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:46:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.474+0000 7fbbda0aa700 1 Processor -- start 2024-01-26T23:46:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.474+0000 7fbbda0aa700 1 -- start start 2024-01-26T23:46:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.475+0000 7fbbda0aa700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbbd4153ec0 0x7fbbd4154290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.475+0000 7fbbda0aa700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbbd40b22d0 0x7fbbd40b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.475+0000 7fbbda0aa700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd415b140 con 0x7fbbd4157630 2024-01-26T23:46:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.475+0000 7fbbda0aa700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbd415a5c0 con 0x7fbbd40b22d0 2024-01-26T23:46:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.475+0000 7fbbda0aa700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd415a8a0 con 0x7fbbd4153ec0 2024-01-26T23:46:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.476+0000 7fbbd3fff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbbd40b22d0 0x7fbbd40b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.476+0000 7fbbd37fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbbd4153ec0 0x7fbbd4154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.476+0000 7fbbd2ffd700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fbbd4157630 0x7fbbd40b1bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:45778/0 (socket says 172.21.15.46:45778) 2024-01-26T23:46:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.476+0000 7fbbd2ffd700 1 -- 172.21.15.46:0/414453435 learned_addr learned my addr 172.21.15.46:0/414453435 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.476+0000 7fbbd37fe700 1 -- 172.21.15.46:0/414453435 >> v2:172.21.15.106:3300/0 conn(0x7fbbd40b22d0 msgr2=0x7fbbd40b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.476+0000 7fbbd37fe700 1 --2- 172.21.15.46:0/414453435 >> v2:172.21.15.106:3300/0 conn(0x7fbbd40b22d0 0x7fbbd40b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.476+0000 7fbbd37fe700 1 -- 172.21.15.46:0/414453435 >> v1:172.21.15.106:6789/0 conn(0x7fbbd4157630 legacy=0x7fbbd40b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.476+0000 7fbbd27fc700 1 -- 172.21.15.46:0/414453435 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273164414 0 0) 0x7fbbd415b140 con 0x7fbbd4157630 2024-01-26T23:46:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.476+0000 7fbbd37fe700 1 -- 172.21.15.46:0/414453435 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd415ab80 con 0x7fbbd4153ec0 2024-01-26T23:46:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.476+0000 7fbbd3fff700 1 --2- 172.21.15.46:0/414453435 >> v2:172.21.15.106:3300/0 conn(0x7fbbd40b22d0 0x7fbbd40b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.476+0000 7fbbd37fe700 1 --2- 172.21.15.46:0/414453435 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbbd4153ec0 0x7fbbd4154290 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbbc400d0d0 tx=0x7fbbc400b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1b5f10e58f591b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.476+0000 7fbbd27fc700 1 -- 172.21.15.46:0/414453435 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc4010070 con 0x7fbbd4153ec0 2024-01-26T23:46:37.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.476+0000 7fbbd27fc700 1 -- 172.21.15.46:0/414453435 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc400b2e0 con 0x7fbbd4153ec0 2024-01-26T23:46:37.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.476+0000 7fbbd27fc700 1 -- 172.21.15.46:0/414453435 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc400dd80 con 0x7fbbd4153ec0 2024-01-26T23:46:37.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.477+0000 7fbbda0aa700 1 -- 172.21.15.46:0/414453435 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbbd4153ec0 msgr2=0x7fbbd4154290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:37.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.477+0000 7fbbda0aa700 1 --2- 172.21.15.46:0/414453435 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbbd4153ec0 0x7fbbd4154290 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbbc400d0d0 tx=0x7fbbc400b990 comp rx=0 tx=0).stop 2024-01-26T23:46:37.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.477+0000 7fbbda0aa700 1 -- 172.21.15.46:0/414453435 shutdown_connections 2024-01-26T23:46:37.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.477+0000 7fbbda0aa700 1 --2- 172.21.15.46:0/414453435 >> v2:172.21.15.106:3300/0 conn(0x7fbbd40b22d0 0x7fbbd40b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.477+0000 7fbbda0aa700 1 --2- 172.21.15.46:0/414453435 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbbd4153ec0 0x7fbbd4154290 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.477+0000 7fbbda0aa700 1 -- 172.21.15.46:0/414453435 >> 172.21.15.46:0/414453435 conn(0x7fbbd4069510 msgr2=0x7fbbd40aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:37.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.477+0000 7fbbda0aa700 1 -- 172.21.15.46:0/414453435 shutdown_connections 2024-01-26T23:46:37.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.477+0000 7fbbda0aa700 1 -- 172.21.15.46:0/414453435 wait complete. 2024-01-26T23:46:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.478+0000 7fbbda0aa700 1 Processor -- start 2024-01-26T23:46:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.479+0000 7fbbda0aa700 1 -- start start 2024-01-26T23:46:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.479+0000 7fbbda0aa700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbd40b22d0 0x7fbbd414e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.479+0000 7fbbda0aa700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbbd4153ec0 0x7fbbd40c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.479+0000 7fbbda0aa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbd4157630 0x7fbbd40ae810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.479+0000 7fbbda0aa700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd414fe20 con 0x7fbbd4157630 2024-01-26T23:46:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.479+0000 7fbbda0aa700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd414ef30 con 0x7fbbd40b22d0 2024-01-26T23:46:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.479+0000 7fbbda0aa700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd414f210 con 0x7fbbd4153ec0 2024-01-26T23:46:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.480+0000 7fbbd3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbd4157630 0x7fbbd40ae810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.480+0000 7fbbd3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbd4157630 0x7fbbd40ae810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33706/0 (socket says 172.21.15.46:33706) 2024-01-26T23:46:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.480+0000 7fbbd3fff700 1 -- 172.21.15.46:0/3719355873 learned_addr learned my addr 172.21.15.46:0/3719355873 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.480+0000 7fbbd2ffd700 1 --2- 172.21.15.46:0/3719355873 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbbd4153ec0 0x7fbbd40c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.480+0000 7fbbd37fe700 1 --2- 172.21.15.46:0/3719355873 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbd40b22d0 0x7fbbd414e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.480+0000 7fbbd3fff700 1 -- 172.21.15.46:0/3719355873 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbbd4153ec0 msgr2=0x7fbbd40c6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.480+0000 7fbbd3fff700 1 --2- 172.21.15.46:0/3719355873 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbbd4153ec0 0x7fbbd40c6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.480+0000 7fbbd3fff700 1 -- 172.21.15.46:0/3719355873 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbd40b22d0 msgr2=0x7fbbd414e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.480+0000 7fbbd3fff700 1 --2- 172.21.15.46:0/3719355873 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbd40b22d0 0x7fbbd414e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.480+0000 7fbbd3fff700 1 -- 172.21.15.46:0/3719355873 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbcc009420 con 0x7fbbd4157630 2024-01-26T23:46:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.480+0000 7fbbd3fff700 1 --2- 172.21.15.46:0/3719355873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbd4157630 0x7fbbd40ae810 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc00ea50 tx=0x7fbbcc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.480+0000 7fbbbb7fe700 1 -- 172.21.15.46:0/3719355873 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc00c640 con 0x7fbbd4157630 2024-01-26T23:46:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.480+0000 7fbbbb7fe700 1 -- 172.21.15.46:0/3719355873 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbcc00c7c0 con 0x7fbbd4157630 2024-01-26T23:46:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.480+0000 7fbbbb7fe700 1 -- 172.21.15.46:0/3719355873 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc00cb80 con 0x7fbbd4157630 2024-01-26T23:46:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.480+0000 7fbbda0aa700 1 -- 172.21.15.46:0/3719355873 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc400bc30 con 0x7fbbd4157630 2024-01-26T23:46:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.480+0000 7fbbda0aa700 1 -- 172.21.15.46:0/3719355873 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd414f7d0 con 0x7fbbd4157630 2024-01-26T23:46:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.482+0000 7fbbda0aa700 1 -- 172.21.15.46:0/3719355873 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbd414ff90 con 0x7fbbd4157630 2024-01-26T23:46:37.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.482+0000 7fbbbb7fe700 1 -- 172.21.15.46:0/3719355873 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fbbcc013470 con 0x7fbbd4157630 2024-01-26T23:46:37.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.482+0000 7fbbbb7fe700 1 --2- 172.21.15.46:0/3719355873 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbbc0066ea0 0x7fbbc0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:37.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.482+0000 7fbbbb7fe700 1 -- 172.21.15.46:0/3719355873 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fbbcc084850 con 0x7fbbd4157630 2024-01-26T23:46:37.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.483+0000 7fbbd37fe700 1 --2- 172.21.15.46:0/3719355873 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbbc0066ea0 0x7fbbc0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:37.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.483+0000 7fbbd37fe700 1 --2- 172.21.15.46:0/3719355873 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbbc0066ea0 0x7fbbc0069350 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbbc400b9d0 tx=0x7fbbc401a000 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.488+0000 7fbbbb7fe700 1 -- 172.21.15.46:0/3719355873 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbbcc0518d0 con 0x7fbbd4157630 2024-01-26T23:46:37.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.702+0000 7fbbda0aa700 1 -- 172.21.15.46:0/3719355873 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbbd404c710 con 0x7fbbc0066ea0 2024-01-26T23:46:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.710+0000 7fbbbb7fe700 1 -- 172.21.15.46:0/3719355873 <== mgr.6365 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289409 (secure 0 0 0) 0x7fbbd404c710 con 0x7fbbc0066ea0 2024-01-26T23:46:37.715 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.713+0000 7fbbda0aa700 1 -- 172.21.15.46:0/3719355873 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbbc0066ea0 msgr2=0x7fbbc0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.713+0000 7fbbda0aa700 1 --2- 172.21.15.46:0/3719355873 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbbc0066ea0 0x7fbbc0069350 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbbc400b9d0 tx=0x7fbbc401a000 comp rx=0 tx=0).stop 2024-01-26T23:46:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.713+0000 7fbbda0aa700 1 -- 172.21.15.46:0/3719355873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbd4157630 msgr2=0x7fbbd40ae810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.713+0000 7fbbda0aa700 1 --2- 172.21.15.46:0/3719355873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbd4157630 0x7fbbd40ae810 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc00ea50 tx=0x7fbbcc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.713+0000 7fbbda0aa700 1 -- 172.21.15.46:0/3719355873 shutdown_connections 2024-01-26T23:46:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.713+0000 7fbbda0aa700 1 --2- 172.21.15.46:0/3719355873 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbbc0066ea0 0x7fbbc0069350 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.713+0000 7fbbda0aa700 1 --2- 172.21.15.46:0/3719355873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbd4157630 0x7fbbd40ae810 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.713+0000 7fbbda0aa700 1 --2- 172.21.15.46:0/3719355873 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbd40b22d0 0x7fbbd414e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.713+0000 7fbbda0aa700 1 --2- 172.21.15.46:0/3719355873 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbbd4153ec0 0x7fbbd40c6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.713+0000 7fbbda0aa700 1 -- 172.21.15.46:0/3719355873 >> 172.21.15.46:0/3719355873 conn(0x7fbbd4069510 msgr2=0x7fbbd4067930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.713+0000 7fbbda0aa700 1 -- 172.21.15.46:0/3719355873 shutdown_connections 2024-01-26T23:46:37.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:37.713+0000 7fbbda0aa700 1 -- 172.21.15.46:0/3719355873 wait complete. 2024-01-26T23:46:37.719 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:46:37.729 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-26T23:46:37.636828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":743,"num_read_kb":1452,"num_write":323,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5178,"ondisk_log_size":5178,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2439464,"kb_used_data":12008,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752535256,"statfs":{"total":773094113280,"available":770596102144,"internally_reserved":0,"allocated":12296192,"data_stored":6802580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.879424"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.670514+0000","last_change":"2024-01-26T23:46:31.666841+0000","last_active":"2024-01-26T23:46:31.670514+0000","last_peered":"2024-01-26T23:46:31.670514+0000","last_clean":"2024-01-26T23:46:31.670514+0000","last_became_active":"2024-01-26T23:46:31.666343+0000","last_became_peered":"2024-01-26T23:46:31.666343+0000","last_unstale":"2024-01-26T23:46:31.670514+0000","last_undegraded":"2024-01-26T23:46:31.670514+0000","last_fullsized":"2024-01-26T23:46:31.670514+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:20:40.199495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.663655+0000","last_change":"2024-01-26T23:46:31.660798+0000","last_active":"2024-01-26T23:46:31.663655+0000","last_peered":"2024-01-26T23:46:31.663655+0000","last_clean":"2024-01-26T23:46:31.663655+0000","last_became_active":"2024-01-26T23:46:31.660389+0000","last_became_peered":"2024-01-26T23:46:31.660389+0000","last_unstale":"2024-01-26T23:46:31.663655+0000","last_undegraded":"2024-01-26T23:46:31.663655+0000","last_fullsized":"2024-01-26T23:46:31.663655+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:15:01.090798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674189+0000","last_change":"2024-01-26T23:46:31.669466+0000","last_active":"2024-01-26T23:46:31.674189+0000","last_peered":"2024-01-26T23:46:31.674189+0000","last_clean":"2024-01-26T23:46:31.674189+0000","last_became_active":"2024-01-26T23:46:31.669137+0000","last_became_peered":"2024-01-26T23:46:31.669137+0000","last_unstale":"2024-01-26T23:46:31.674189+0000","last_undegraded":"2024-01-26T23:46:31.674189+0000","last_fullsized":"2024-01-26T23:46:31.674189+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:49:03.876856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671210+0000","last_change":"2024-01-26T23:46:31.664196+0000","last_active":"2024-01-26T23:46:31.671210+0000","last_peered":"2024-01-26T23:46:31.671210+0000","last_clean":"2024-01-26T23:46:31.671210+0000","last_became_active":"2024-01-26T23:46:31.663567+0000","last_became_peered":"2024-01-26T23:46:31.663567+0000","last_unstale":"2024-01-26T23:46:31.671210+0000","last_undegraded":"2024-01-26T23:46:31.671210+0000","last_fullsized":"2024-01-26T23:46:31.671210+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:18:30.549273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674416+0000","last_change":"2024-01-26T23:46:31.666630+0000","last_active":"2024-01-26T23:46:31.674416+0000","last_peered":"2024-01-26T23:46:31.674416+0000","last_clean":"2024-01-26T23:46:31.674416+0000","last_became_active":"2024-01-26T23:46:31.666304+0000","last_became_peered":"2024-01-26T23:46:31.666304+0000","last_unstale":"2024-01-26T23:46:31.674416+0000","last_undegraded":"2024-01-26T23:46:31.674416+0000","last_fullsized":"2024-01-26T23:46:31.674416+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:17:40.471277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678729+0000","last_change":"2024-01-26T23:46:31.677430+0000","last_active":"2024-01-26T23:46:31.678729+0000","last_peered":"2024-01-26T23:46:31.678729+0000","last_clean":"2024-01-26T23:46:31.678729+0000","last_became_active":"2024-01-26T23:46:31.677151+0000","last_became_peered":"2024-01-26T23:46:31.677151+0000","last_unstale":"2024-01-26T23:46:31.678729+0000","last_undegraded":"2024-01-26T23:46:31.678729+0000","last_fullsized":"2024-01-26T23:46:31.678729+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:15:00.551460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.677758+0000","last_change":"2024-01-26T23:46:31.674897+0000","last_active":"2024-01-26T23:46:31.677758+0000","last_peered":"2024-01-26T23:46:31.677758+0000","last_clean":"2024-01-26T23:46:31.677758+0000","last_became_active":"2024-01-26T23:46:31.674663+0000","last_became_peered":"2024-01-26T23:46:31.674663+0000","last_unstale":"2024-01-26T23:46:31.677758+0000","last_undegraded":"2024-01-26T23:46:31.677758+0000","last_fullsized":"2024-01-26T23:46:31.677758+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:38:57.979602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.669177+0000","last_change":"2024-01-26T23:46:31.659845+0000","last_active":"2024-01-26T23:46:31.669177+0000","last_peered":"2024-01-26T23:46:31.669177+0000","last_clean":"2024-01-26T23:46:31.669177+0000","last_became_active":"2024-01-26T23:46:31.659539+0000","last_became_peered":"2024-01-26T23:46:31.659539+0000","last_unstale":"2024-01-26T23:46:31.669177+0000","last_undegraded":"2024-01-26T23:46:31.669177+0000","last_fullsized":"2024-01-26T23:46:31.669177+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:26:22.946213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674606+0000","last_change":"2024-01-26T23:46:31.666924+0000","last_active":"2024-01-26T23:46:31.674606+0000","last_peered":"2024-01-26T23:46:31.674606+0000","last_clean":"2024-01-26T23:46:31.674606+0000","last_became_active":"2024-01-26T23:46:31.666670+0000","last_became_peered":"2024-01-26T23:46:31.666670+0000","last_unstale":"2024-01-26T23:46:31.674606+0000","last_undegraded":"2024-01-26T23:46:31.674606+0000","last_fullsized":"2024-01-26T23:46:31.674606+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:12:26.391081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.676434+0000","last_change":"2024-01-26T23:46:31.672168+0000","last_active":"2024-01-26T23:46:31.676434+0000","last_peered":"2024-01-26T23:46:31.676434+0000","last_clean":"2024-01-26T23:46:31.676434+0000","last_became_active":"2024-01-26T23:46:31.671762+0000","last_became_peered":"2024-01-26T23:46:31.671762+0000","last_unstale":"2024-01-26T23:46:31.676434+0000","last_undegraded":"2024-01-26T23:46:31.676434+0000","last_fullsized":"2024-01-26T23:46:31.676434+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:00:04.119988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674650+0000","last_change":"2024-01-26T23:46:31.667223+0000","last_active":"2024-01-26T23:46:31.674650+0000","last_peered":"2024-01-26T23:46:31.674650+0000","last_clean":"2024-01-26T23:46:31.674650+0000","last_became_active":"2024-01-26T23:46:31.666318+0000","last_became_peered":"2024-01-26T23:46:31.666318+0000","last_unstale":"2024-01-26T23:46:31.674650+0000","last_undegraded":"2024-01-26T23:46:31.674650+0000","last_fullsized":"2024-01-26T23:46:31.674650+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:09:35.596681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671217+0000","last_change":"2024-01-26T23:46:31.666856+0000","last_active":"2024-01-26T23:46:31.671217+0000","last_peered":"2024-01-26T23:46:31.671217+0000","last_clean":"2024-01-26T23:46:31.671217+0000","last_became_active":"2024-01-26T23:46:31.666519+0000","last_became_peered":"2024-01-26T23:46:31.666519+0000","last_unstale":"2024-01-26T23:46:31.671217+0000","last_undegraded":"2024-01-26T23:46:31.671217+0000","last_fullsized":"2024-01-26T23:46:31.671217+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:28:15.946492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.669162+0000","last_change":"2024-01-26T23:46:31.659612+0000","last_active":"2024-01-26T23:46:31.669162+0000","last_peered":"2024-01-26T23:46:31.669162+0000","last_clean":"2024-01-26T23:46:31.669162+0000","last_became_active":"2024-01-26T23:46:31.659271+0000","last_became_peered":"2024-01-26T23:46:31.659271+0000","last_unstale":"2024-01-26T23:46:31.669162+0000","last_undegraded":"2024-01-26T23:46:31.669162+0000","last_fullsized":"2024-01-26T23:46:31.669162+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:49:03.876856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678127+0000","last_change":"2024-01-26T23:46:31.677644+0000","last_active":"2024-01-26T23:46:31.678127+0000","last_peered":"2024-01-26T23:46:31.678127+0000","last_clean":"2024-01-26T23:46:31.678127+0000","last_became_active":"2024-01-26T23:46:31.677289+0000","last_became_peered":"2024-01-26T23:46:31.677289+0000","last_unstale":"2024-01-26T23:46:31.678127+0000","last_undegraded":"2024-01-26T23:46:31.678127+0000","last_fullsized":"2024-01-26T23:46:31.678127+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:44:47.619765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671107+0000","last_change":"2024-01-26T23:46:31.664774+0000","last_active":"2024-01-26T23:46:31.671107+0000","last_peered":"2024-01-26T23:46:31.671107+0000","last_clean":"2024-01-26T23:46:31.671107+0000","last_became_active":"2024-01-26T23:46:31.664456+0000","last_became_peered":"2024-01-26T23:46:31.664456+0000","last_unstale":"2024-01-26T23:46:31.671107+0000","last_undegraded":"2024-01-26T23:46:31.671107+0000","last_fullsized":"2024-01-26T23:46:31.671107+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:15:01.090798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671925+0000","last_change":"2024-01-26T23:46:31.666995+0000","last_active":"2024-01-26T23:46:31.671925+0000","last_peered":"2024-01-26T23:46:31.671925+0000","last_clean":"2024-01-26T23:46:31.671925+0000","last_became_active":"2024-01-26T23:46:31.666507+0000","last_became_peered":"2024-01-26T23:46:31.666507+0000","last_unstale":"2024-01-26T23:46:31.671925+0000","last_undegraded":"2024-01-26T23:46:31.671925+0000","last_fullsized":"2024-01-26T23:46:31.671925+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:35:52.815366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.672072+0000","last_change":"2024-01-26T23:46:31.665567+0000","last_active":"2024-01-26T23:46:31.672072+0000","last_peered":"2024-01-26T23:46:31.672072+0000","last_clean":"2024-01-26T23:46:31.672072+0000","last_became_active":"2024-01-26T23:46:31.665250+0000","last_became_peered":"2024-01-26T23:46:31.665250+0000","last_unstale":"2024-01-26T23:46:31.672072+0000","last_undegraded":"2024-01-26T23:46:31.672072+0000","last_fullsized":"2024-01-26T23:46:31.672072+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:44.847438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.670423+0000","last_change":"2024-01-26T23:46:31.659801+0000","last_active":"2024-01-26T23:46:31.670423+0000","last_peered":"2024-01-26T23:46:31.670423+0000","last_clean":"2024-01-26T23:46:31.670423+0000","last_became_active":"2024-01-26T23:46:31.659436+0000","last_became_peered":"2024-01-26T23:46:31.659436+0000","last_unstale":"2024-01-26T23:46:31.670423+0000","last_undegraded":"2024-01-26T23:46:31.670423+0000","last_fullsized":"2024-01-26T23:46:31.670423+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:26:22.946213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.677680+0000","last_change":"2024-01-26T23:46:31.671836+0000","last_active":"2024-01-26T23:46:31.677680+0000","last_peered":"2024-01-26T23:46:31.677680+0000","last_clean":"2024-01-26T23:46:31.677680+0000","last_became_active":"2024-01-26T23:46:31.671411+0000","last_became_peered":"2024-01-26T23:46:31.671411+0000","last_unstale":"2024-01-26T23:46:31.677680+0000","last_undegraded":"2024-01-26T23:46:31.677680+0000","last_fullsized":"2024-01-26T23:46:31.677680+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:06:39.892851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.670670+0000","last_change":"2024-01-26T23:46:31.660955+0000","last_active":"2024-01-26T23:46:31.670670+0000","last_peered":"2024-01-26T23:46:31.670670+0000","last_clean":"2024-01-26T23:46:31.670670+0000","last_became_active":"2024-01-26T23:46:31.660600+0000","last_became_peered":"2024-01-26T23:46:31.660600+0000","last_unstale":"2024-01-26T23:46:31.670670+0000","last_undegraded":"2024-01-26T23:46:31.670670+0000","last_fullsized":"2024-01-26T23:46:31.670670+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:20:40.199495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674437+0000","last_change":"2024-01-26T23:46:31.666637+0000","last_active":"2024-01-26T23:46:31.674437+0000","last_peered":"2024-01-26T23:46:31.674437+0000","last_clean":"2024-01-26T23:46:31.674437+0000","last_became_active":"2024-01-26T23:46:31.666340+0000","last_became_peered":"2024-01-26T23:46:31.666340+0000","last_unstale":"2024-01-26T23:46:31.674437+0000","last_undegraded":"2024-01-26T23:46:31.674437+0000","last_fullsized":"2024-01-26T23:46:31.674437+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:35.826457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678116+0000","last_change":"2024-01-26T23:46:31.677579+0000","last_active":"2024-01-26T23:46:31.678116+0000","last_peered":"2024-01-26T23:46:31.678116+0000","last_clean":"2024-01-26T23:46:31.678116+0000","last_became_active":"2024-01-26T23:46:31.677261+0000","last_became_peered":"2024-01-26T23:46:31.677261+0000","last_unstale":"2024-01-26T23:46:31.678116+0000","last_undegraded":"2024-01-26T23:46:31.678116+0000","last_fullsized":"2024-01-26T23:46:31.678116+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:19:41.093581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674555+0000","last_change":"2024-01-26T23:46:31.672211+0000","last_active":"2024-01-26T23:46:31.674555+0000","last_peered":"2024-01-26T23:46:31.674555+0000","last_clean":"2024-01-26T23:46:31.674555+0000","last_became_active":"2024-01-26T23:46:31.671945+0000","last_became_peered":"2024-01-26T23:46:31.671945+0000","last_unstale":"2024-01-26T23:46:31.674555+0000","last_undegraded":"2024-01-26T23:46:31.674555+0000","last_fullsized":"2024-01-26T23:46:31.674555+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:15:01.090798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674185+0000","last_change":"2024-01-26T23:46:31.669513+0000","last_active":"2024-01-26T23:46:31.674185+0000","last_peered":"2024-01-26T23:46:31.674185+0000","last_clean":"2024-01-26T23:46:31.674185+0000","last_became_active":"2024-01-26T23:46:31.669214+0000","last_became_peered":"2024-01-26T23:46:31.669214+0000","last_unstale":"2024-01-26T23:46:31.674185+0000","last_undegraded":"2024-01-26T23:46:31.674185+0000","last_fullsized":"2024-01-26T23:46:31.674185+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:15:01.090798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.663696+0000","last_change":"2024-01-26T23:46:31.661379+0000","last_active":"2024-01-26T23:46:31.663696+0000","last_peered":"2024-01-26T23:46:31.663696+0000","last_clean":"2024-01-26T23:46:31.663696+0000","last_became_active":"2024-01-26T23:46:31.661139+0000","last_became_peered":"2024-01-26T23:46:31.661139+0000","last_unstale":"2024-01-26T23:46:31.663696+0000","last_undegraded":"2024-01-26T23:46:31.663696+0000","last_fullsized":"2024-01-26T23:46:31.663696+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:14.351439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.677651+0000","last_change":"2024-01-26T23:46:31.672080+0000","last_active":"2024-01-26T23:46:31.677651+0000","last_peered":"2024-01-26T23:46:31.677651+0000","last_clean":"2024-01-26T23:46:31.677651+0000","last_became_active":"2024-01-26T23:46:31.671662+0000","last_became_peered":"2024-01-26T23:46:31.671662+0000","last_unstale":"2024-01-26T23:46:31.677651+0000","last_undegraded":"2024-01-26T23:46:31.677651+0000","last_fullsized":"2024-01-26T23:46:31.677651+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:21:18.772481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671143+0000","last_change":"2024-01-26T23:46:31.668130+0000","last_active":"2024-01-26T23:46:31.671143+0000","last_peered":"2024-01-26T23:46:31.671143+0000","last_clean":"2024-01-26T23:46:31.671143+0000","last_became_active":"2024-01-26T23:46:31.667823+0000","last_became_peered":"2024-01-26T23:46:31.667823+0000","last_unstale":"2024-01-26T23:46:31.671143+0000","last_undegraded":"2024-01-26T23:46:31.671143+0000","last_fullsized":"2024-01-26T23:46:31.671143+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:33:18.471363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671230+0000","last_change":"2024-01-26T23:46:31.667876+0000","last_active":"2024-01-26T23:46:31.671230+0000","last_peered":"2024-01-26T23:46:31.671230+0000","last_clean":"2024-01-26T23:46:31.671230+0000","last_became_active":"2024-01-26T23:46:31.667555+0000","last_became_peered":"2024-01-26T23:46:31.667555+0000","last_unstale":"2024-01-26T23:46:31.671230+0000","last_undegraded":"2024-01-26T23:46:31.671230+0000","last_fullsized":"2024-01-26T23:46:31.671230+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:14.351439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674525+0000","last_change":"2024-01-26T23:46:31.671962+0000","last_active":"2024-01-26T23:46:31.674525+0000","last_peered":"2024-01-26T23:46:31.674525+0000","last_clean":"2024-01-26T23:46:31.674525+0000","last_became_active":"2024-01-26T23:46:31.671717+0000","last_became_peered":"2024-01-26T23:46:31.671717+0000","last_unstale":"2024-01-26T23:46:31.674525+0000","last_undegraded":"2024-01-26T23:46:31.674525+0000","last_fullsized":"2024-01-26T23:46:31.674525+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:19:13.010766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674578+0000","last_change":"2024-01-26T23:46:31.667039+0000","last_active":"2024-01-26T23:46:31.674578+0000","last_peered":"2024-01-26T23:46:31.674578+0000","last_clean":"2024-01-26T23:46:31.674578+0000","last_became_active":"2024-01-26T23:46:31.666790+0000","last_became_peered":"2024-01-26T23:46:31.666790+0000","last_unstale":"2024-01-26T23:46:31.674578+0000","last_undegraded":"2024-01-26T23:46:31.674578+0000","last_fullsized":"2024-01-26T23:46:31.674578+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:25.220136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.663634+0000","last_change":"2024-01-26T23:46:31.657261+0000","last_active":"2024-01-26T23:46:31.663634+0000","last_peered":"2024-01-26T23:46:31.663634+0000","last_clean":"2024-01-26T23:46:31.663634+0000","last_became_active":"2024-01-26T23:46:31.657084+0000","last_became_peered":"2024-01-26T23:46:31.657084+0000","last_unstale":"2024-01-26T23:46:31.663634+0000","last_undegraded":"2024-01-26T23:46:31.663634+0000","last_fullsized":"2024-01-26T23:46:31.663634+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:46:27.325038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671844+0000","last_change":"2024-01-26T23:46:31.666793+0000","last_active":"2024-01-26T23:46:31.671844+0000","last_peered":"2024-01-26T23:46:31.671844+0000","last_clean":"2024-01-26T23:46:31.671844+0000","last_became_active":"2024-01-26T23:46:31.666112+0000","last_became_peered":"2024-01-26T23:46:31.666112+0000","last_unstale":"2024-01-26T23:46:31.671844+0000","last_undegraded":"2024-01-26T23:46:31.671844+0000","last_fullsized":"2024-01-26T23:46:31.671844+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:38.074736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675273+0000","last_change":"2024-01-26T23:46:31.668562+0000","last_active":"2024-01-26T23:46:31.675273+0000","last_peered":"2024-01-26T23:46:31.675273+0000","last_clean":"2024-01-26T23:46:31.675273+0000","last_became_active":"2024-01-26T23:46:31.668253+0000","last_became_peered":"2024-01-26T23:46:31.668253+0000","last_unstale":"2024-01-26T23:46:31.675273+0000","last_undegraded":"2024-01-26T23:46:31.675273+0000","last_fullsized":"2024-01-26T23:46:31.675273+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:43:18.393081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"111'4","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.774608+0000","last_change":"2024-01-26T23:46:30.779482+0000","last_active":"2024-01-26T23:46:31.774608+0000","last_peered":"2024-01-26T23:46:31.774608+0000","last_clean":"2024-01-26T23:46:31.774608+0000","last_became_active":"2024-01-26T23:46:30.779245+0000","last_became_peered":"2024-01-26T23:46:30.779245+0000","last_unstale":"2024-01-26T23:46:31.774608+0000","last_undegraded":"2024-01-26T23:46:31.774608+0000","last_fullsized":"2024-01-26T23:46:31.774608+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:37.479741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675662+0000","last_change":"2024-01-26T23:46:31.672354+0000","last_active":"2024-01-26T23:46:31.675662+0000","last_peered":"2024-01-26T23:46:31.675662+0000","last_clean":"2024-01-26T23:46:31.675662+0000","last_became_active":"2024-01-26T23:46:31.672029+0000","last_became_peered":"2024-01-26T23:46:31.672029+0000","last_unstale":"2024-01-26T23:46:31.675662+0000","last_undegraded":"2024-01-26T23:46:31.675662+0000","last_fullsized":"2024-01-26T23:46:31.675662+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:16:43.142449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"111'1","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.772979+0000","last_change":"2024-01-26T23:46:30.769470+0000","last_active":"2024-01-26T23:46:31.772979+0000","last_peered":"2024-01-26T23:46:31.772979+0000","last_clean":"2024-01-26T23:46:31.772979+0000","last_became_active":"2024-01-26T23:46:30.769231+0000","last_became_peered":"2024-01-26T23:46:30.769231+0000","last_unstale":"2024-01-26T23:46:31.772979+0000","last_undegraded":"2024-01-26T23:46:31.772979+0000","last_fullsized":"2024-01-26T23:46:31.772979+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:16:42.463283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.672601+0000","last_change":"2024-01-26T23:46:31.669777+0000","last_active":"2024-01-26T23:46:31.672601+0000","last_peered":"2024-01-26T23:46:31.672601+0000","last_clean":"2024-01-26T23:46:31.672601+0000","last_became_active":"2024-01-26T23:46:31.669538+0000","last_became_peered":"2024-01-26T23:46:31.669538+0000","last_unstale":"2024-01-26T23:46:31.672601+0000","last_undegraded":"2024-01-26T23:46:31.672601+0000","last_fullsized":"2024-01-26T23:46:31.672601+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:17:40.471277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"111'2","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.773859+0000","last_change":"2024-01-26T23:46:30.776273+0000","last_active":"2024-01-26T23:46:31.773859+0000","last_peered":"2024-01-26T23:46:31.773859+0000","last_clean":"2024-01-26T23:46:31.773859+0000","last_became_active":"2024-01-26T23:46:30.775989+0000","last_became_peered":"2024-01-26T23:46:30.775989+0000","last_unstale":"2024-01-26T23:46:31.773859+0000","last_undegraded":"2024-01-26T23:46:31.773859+0000","last_fullsized":"2024-01-26T23:46:31.773859+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:00:45.452291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.670354+0000","last_change":"2024-01-26T23:46:31.662791+0000","last_active":"2024-01-26T23:46:31.670354+0000","last_peered":"2024-01-26T23:46:31.670354+0000","last_clean":"2024-01-26T23:46:31.670354+0000","last_became_active":"2024-01-26T23:46:31.662506+0000","last_became_peered":"2024-01-26T23:46:31.662506+0000","last_unstale":"2024-01-26T23:46:31.670354+0000","last_undegraded":"2024-01-26T23:46:31.670354+0000","last_fullsized":"2024-01-26T23:46:31.670354+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:22.391377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"111'2","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.772905+0000","last_change":"2024-01-26T23:46:30.776989+0000","last_active":"2024-01-26T23:46:31.772905+0000","last_peered":"2024-01-26T23:46:31.772905+0000","last_clean":"2024-01-26T23:46:31.772905+0000","last_became_active":"2024-01-26T23:46:30.776785+0000","last_became_peered":"2024-01-26T23:46:30.776785+0000","last_unstale":"2024-01-26T23:46:31.772905+0000","last_undegraded":"2024-01-26T23:46:31.772905+0000","last_fullsized":"2024-01-26T23:46:31.772905+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:07.859918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.670702+0000","last_change":"2024-01-26T23:46:31.661125+0000","last_active":"2024-01-26T23:46:31.670702+0000","last_peered":"2024-01-26T23:46:31.670702+0000","last_clean":"2024-01-26T23:46:31.670702+0000","last_became_active":"2024-01-26T23:46:31.660752+0000","last_became_peered":"2024-01-26T23:46:31.660752+0000","last_unstale":"2024-01-26T23:46:31.670702+0000","last_undegraded":"2024-01-26T23:46:31.670702+0000","last_fullsized":"2024-01-26T23:46:31.670702+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:46:27.325038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"111'1","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.771129+0000","last_change":"2024-01-26T23:46:30.777476+0000","last_active":"2024-01-26T23:46:31.771129+0000","last_peered":"2024-01-26T23:46:31.771129+0000","last_clean":"2024-01-26T23:46:31.771129+0000","last_became_active":"2024-01-26T23:46:30.777188+0000","last_became_peered":"2024-01-26T23:46:30.777188+0000","last_unstale":"2024-01-26T23:46:31.771129+0000","last_undegraded":"2024-01-26T23:46:31.771129+0000","last_fullsized":"2024-01-26T23:46:31.771129+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:41:51.840628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.666024+0000","last_change":"2024-01-26T23:46:31.662764+0000","last_active":"2024-01-26T23:46:31.666024+0000","last_peered":"2024-01-26T23:46:31.666024+0000","last_clean":"2024-01-26T23:46:31.666024+0000","last_became_active":"2024-01-26T23:46:31.662519+0000","last_became_peered":"2024-01-26T23:46:31.662519+0000","last_unstale":"2024-01-26T23:46:31.666024+0000","last_undegraded":"2024-01-26T23:46:31.666024+0000","last_fullsized":"2024-01-26T23:46:31.666024+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:14.351439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"111'1","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.777701+0000","last_change":"2024-01-26T23:46:30.780739+0000","last_active":"2024-01-26T23:46:31.777701+0000","last_peered":"2024-01-26T23:46:31.777701+0000","last_clean":"2024-01-26T23:46:31.777701+0000","last_became_active":"2024-01-26T23:46:30.780309+0000","last_became_peered":"2024-01-26T23:46:30.780309+0000","last_unstale":"2024-01-26T23:46:31.777701+0000","last_undegraded":"2024-01-26T23:46:31.777701+0000","last_fullsized":"2024-01-26T23:46:31.777701+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:28.581902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678716+0000","last_change":"2024-01-26T23:46:31.677394+0000","last_active":"2024-01-26T23:46:31.678716+0000","last_peered":"2024-01-26T23:46:31.678716+0000","last_clean":"2024-01-26T23:46:31.678716+0000","last_became_active":"2024-01-26T23:46:31.677090+0000","last_became_peered":"2024-01-26T23:46:31.677090+0000","last_unstale":"2024-01-26T23:46:31.678716+0000","last_undegraded":"2024-01-26T23:46:31.678716+0000","last_fullsized":"2024-01-26T23:46:31.678716+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:28:36.539497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.653719+0000","last_change":"2024-01-26T23:46:30.771550+0000","last_active":"2024-01-26T23:46:31.653719+0000","last_peered":"2024-01-26T23:46:31.653719+0000","last_clean":"2024-01-26T23:46:31.653719+0000","last_became_active":"2024-01-26T23:46:30.771291+0000","last_became_peered":"2024-01-26T23:46:30.771291+0000","last_unstale":"2024-01-26T23:46:31.653719+0000","last_undegraded":"2024-01-26T23:46:31.653719+0000","last_fullsized":"2024-01-26T23:46:31.653719+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:30:09.071510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.673494+0000","last_change":"2024-01-26T23:46:31.663969+0000","last_active":"2024-01-26T23:46:31.673494+0000","last_peered":"2024-01-26T23:46:31.673494+0000","last_clean":"2024-01-26T23:46:31.673494+0000","last_became_active":"2024-01-26T23:46:31.663602+0000","last_became_peered":"2024-01-26T23:46:31.663602+0000","last_unstale":"2024-01-26T23:46:31.673494+0000","last_undegraded":"2024-01-26T23:46:31.673494+0000","last_fullsized":"2024-01-26T23:46:31.673494+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:23:09.740800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.654348+0000","last_change":"2024-01-26T23:46:30.782673+0000","last_active":"2024-01-26T23:46:31.654348+0000","last_peered":"2024-01-26T23:46:31.654348+0000","last_clean":"2024-01-26T23:46:31.654348+0000","last_became_active":"2024-01-26T23:46:30.782464+0000","last_became_peered":"2024-01-26T23:46:30.782464+0000","last_unstale":"2024-01-26T23:46:31.654348+0000","last_undegraded":"2024-01-26T23:46:31.654348+0000","last_fullsized":"2024-01-26T23:46:31.654348+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:13.350096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674733+0000","last_change":"2024-01-26T23:46:31.669467+0000","last_active":"2024-01-26T23:46:31.674733+0000","last_peered":"2024-01-26T23:46:31.674733+0000","last_clean":"2024-01-26T23:46:31.674733+0000","last_became_active":"2024-01-26T23:46:31.669212+0000","last_became_peered":"2024-01-26T23:46:31.669212+0000","last_unstale":"2024-01-26T23:46:31.674733+0000","last_undegraded":"2024-01-26T23:46:31.674733+0000","last_fullsized":"2024-01-26T23:46:31.674733+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:24:19.005800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.652762+0000","last_change":"2024-01-26T23:46:30.773338+0000","last_active":"2024-01-26T23:46:31.652762+0000","last_peered":"2024-01-26T23:46:31.652762+0000","last_clean":"2024-01-26T23:46:31.652762+0000","last_became_active":"2024-01-26T23:46:30.773156+0000","last_became_peered":"2024-01-26T23:46:30.773156+0000","last_unstale":"2024-01-26T23:46:31.652762+0000","last_undegraded":"2024-01-26T23:46:31.652762+0000","last_fullsized":"2024-01-26T23:46:31.652762+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:46.577047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671908+0000","last_change":"2024-01-26T23:46:31.667991+0000","last_active":"2024-01-26T23:46:31.671908+0000","last_peered":"2024-01-26T23:46:31.671908+0000","last_clean":"2024-01-26T23:46:31.671908+0000","last_became_active":"2024-01-26T23:46:31.667740+0000","last_became_peered":"2024-01-26T23:46:31.667740+0000","last_unstale":"2024-01-26T23:46:31.671908+0000","last_undegraded":"2024-01-26T23:46:31.671908+0000","last_fullsized":"2024-01-26T23:46:31.671908+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:45.348777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.653142+0000","last_change":"2024-01-26T23:46:30.776658+0000","last_active":"2024-01-26T23:46:31.653142+0000","last_peered":"2024-01-26T23:46:31.653142+0000","last_clean":"2024-01-26T23:46:31.653142+0000","last_became_active":"2024-01-26T23:46:30.776294+0000","last_became_peered":"2024-01-26T23:46:30.776294+0000","last_unstale":"2024-01-26T23:46:31.653142+0000","last_undegraded":"2024-01-26T23:46:31.653142+0000","last_fullsized":"2024-01-26T23:46:31.653142+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:00:45.452291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.670487+0000","last_change":"2024-01-26T23:46:31.665924+0000","last_active":"2024-01-26T23:46:31.670487+0000","last_peered":"2024-01-26T23:46:31.670487+0000","last_clean":"2024-01-26T23:46:31.670487+0000","last_became_active":"2024-01-26T23:46:31.665551+0000","last_became_peered":"2024-01-26T23:46:31.665551+0000","last_unstale":"2024-01-26T23:46:31.670487+0000","last_undegraded":"2024-01-26T23:46:31.670487+0000","last_fullsized":"2024-01-26T23:46:31.670487+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:38:57.979602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"111'3","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.775090+0000","last_change":"2024-01-26T23:46:30.784086+0000","last_active":"2024-01-26T23:46:31.775090+0000","last_peered":"2024-01-26T23:46:31.775090+0000","last_clean":"2024-01-26T23:46:31.775090+0000","last_became_active":"2024-01-26T23:46:30.783840+0000","last_became_peered":"2024-01-26T23:46:30.783840+0000","last_unstale":"2024-01-26T23:46:31.775090+0000","last_undegraded":"2024-01-26T23:46:31.775090+0000","last_fullsized":"2024-01-26T23:46:31.775090+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:00:45.452291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.668429+0000","last_change":"2024-01-26T23:46:31.655793+0000","last_active":"2024-01-26T23:46:31.668429+0000","last_peered":"2024-01-26T23:46:31.668429+0000","last_clean":"2024-01-26T23:46:31.668429+0000","last_became_active":"2024-01-26T23:46:31.655460+0000","last_became_peered":"2024-01-26T23:46:31.655460+0000","last_unstale":"2024-01-26T23:46:31.668429+0000","last_undegraded":"2024-01-26T23:46:31.668429+0000","last_fullsized":"2024-01-26T23:46:31.668429+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:22:27.095843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"111'6","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.776077+0000","last_change":"2024-01-26T23:46:30.772469+0000","last_active":"2024-01-26T23:46:31.776077+0000","last_peered":"2024-01-26T23:46:31.776077+0000","last_clean":"2024-01-26T23:46:31.776077+0000","last_became_active":"2024-01-26T23:46:30.772188+0000","last_became_peered":"2024-01-26T23:46:30.772188+0000","last_unstale":"2024-01-26T23:46:31.776077+0000","last_undegraded":"2024-01-26T23:46:31.776077+0000","last_fullsized":"2024-01-26T23:46:31.776077+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:04:26.732373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.669153+0000","last_change":"2024-01-26T23:46:31.659621+0000","last_active":"2024-01-26T23:46:31.669153+0000","last_peered":"2024-01-26T23:46:31.669153+0000","last_clean":"2024-01-26T23:46:31.669153+0000","last_became_active":"2024-01-26T23:46:31.658380+0000","last_became_peered":"2024-01-26T23:46:31.658380+0000","last_unstale":"2024-01-26T23:46:31.669153+0000","last_undegraded":"2024-01-26T23:46:31.669153+0000","last_fullsized":"2024-01-26T23:46:31.669153+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:48.805791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"111'2","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.775041+0000","last_change":"2024-01-26T23:46:30.779281+0000","last_active":"2024-01-26T23:46:31.775041+0000","last_peered":"2024-01-26T23:46:31.775041+0000","last_clean":"2024-01-26T23:46:31.775041+0000","last_became_active":"2024-01-26T23:46:30.779013+0000","last_became_peered":"2024-01-26T23:46:30.779013+0000","last_unstale":"2024-01-26T23:46:31.775041+0000","last_undegraded":"2024-01-26T23:46:31.775041+0000","last_fullsized":"2024-01-26T23:46:31.775041+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:22:26.094500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671594+0000","last_change":"2024-01-26T23:46:31.663372+0000","last_active":"2024-01-26T23:46:31.671594+0000","last_peered":"2024-01-26T23:46:31.671594+0000","last_clean":"2024-01-26T23:46:31.671594+0000","last_became_active":"2024-01-26T23:46:31.663122+0000","last_became_peered":"2024-01-26T23:46:31.663122+0000","last_unstale":"2024-01-26T23:46:31.671594+0000","last_undegraded":"2024-01-26T23:46:31.671594+0000","last_fullsized":"2024-01-26T23:46:31.671594+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:33:37.940458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.657510+0000","last_change":"2024-01-26T23:46:30.783543+0000","last_active":"2024-01-26T23:46:31.657510+0000","last_peered":"2024-01-26T23:46:31.657510+0000","last_clean":"2024-01-26T23:46:31.657510+0000","last_became_active":"2024-01-26T23:46:30.774051+0000","last_became_peered":"2024-01-26T23:46:30.774051+0000","last_unstale":"2024-01-26T23:46:31.657510+0000","last_undegraded":"2024-01-26T23:46:31.657510+0000","last_fullsized":"2024-01-26T23:46:31.657510+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:18:11.911206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678158+0000","last_change":"2024-01-26T23:46:31.677498+0000","last_active":"2024-01-26T23:46:31.678158+0000","last_peered":"2024-01-26T23:46:31.678158+0000","last_clean":"2024-01-26T23:46:31.678158+0000","last_became_active":"2024-01-26T23:46:31.677210+0000","last_became_peered":"2024-01-26T23:46:31.677210+0000","last_unstale":"2024-01-26T23:46:31.678158+0000","last_undegraded":"2024-01-26T23:46:31.678158+0000","last_fullsized":"2024-01-26T23:46:31.678158+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:28.759656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.648299+0000","last_change":"2024-01-26T23:46:30.779455+0000","last_active":"2024-01-26T23:46:31.648299+0000","last_peered":"2024-01-26T23:46:31.648299+0000","last_clean":"2024-01-26T23:46:31.648299+0000","last_became_active":"2024-01-26T23:46:30.779141+0000","last_became_peered":"2024-01-26T23:46:30.779141+0000","last_unstale":"2024-01-26T23:46:31.648299+0000","last_undegraded":"2024-01-26T23:46:31.648299+0000","last_fullsized":"2024-01-26T23:46:31.648299+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:34:01.252528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674140+0000","last_change":"2024-01-26T23:46:31.670025+0000","last_active":"2024-01-26T23:46:31.674140+0000","last_peered":"2024-01-26T23:46:31.674140+0000","last_clean":"2024-01-26T23:46:31.674140+0000","last_became_active":"2024-01-26T23:46:31.669744+0000","last_became_peered":"2024-01-26T23:46:31.669744+0000","last_unstale":"2024-01-26T23:46:31.674140+0000","last_undegraded":"2024-01-26T23:46:31.674140+0000","last_fullsized":"2024-01-26T23:46:31.674140+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:27:45.307754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"111'8","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.778380+0000","last_change":"2024-01-26T23:46:30.780133+0000","last_active":"2024-01-26T23:46:31.778380+0000","last_peered":"2024-01-26T23:46:31.778380+0000","last_clean":"2024-01-26T23:46:31.778380+0000","last_became_active":"2024-01-26T23:46:30.779792+0000","last_became_peered":"2024-01-26T23:46:30.779792+0000","last_unstale":"2024-01-26T23:46:31.778380+0000","last_undegraded":"2024-01-26T23:46:31.778380+0000","last_fullsized":"2024-01-26T23:46:31.778380+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:15:25.108006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.672965+0000","last_change":"2024-01-26T23:46:31.667032+0000","last_active":"2024-01-26T23:46:31.672965+0000","last_peered":"2024-01-26T23:46:31.672965+0000","last_clean":"2024-01-26T23:46:31.672965+0000","last_became_active":"2024-01-26T23:46:31.666719+0000","last_became_peered":"2024-01-26T23:46:31.666719+0000","last_unstale":"2024-01-26T23:46:31.672965+0000","last_undegraded":"2024-01-26T23:46:31.672965+0000","last_fullsized":"2024-01-26T23:46:31.672965+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:47:07.666792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"111'1","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.774378+0000","last_change":"2024-01-26T23:46:30.768481+0000","last_active":"2024-01-26T23:46:31.774378+0000","last_peered":"2024-01-26T23:46:31.774378+0000","last_clean":"2024-01-26T23:46:31.774378+0000","last_became_active":"2024-01-26T23:46:30.768174+0000","last_became_peered":"2024-01-26T23:46:30.768174+0000","last_unstale":"2024-01-26T23:46:31.774378+0000","last_undegraded":"2024-01-26T23:46:31.774378+0000","last_fullsized":"2024-01-26T23:46:31.774378+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:46.577047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.663243+0000","last_change":"2024-01-26T23:46:31.660241+0000","last_active":"2024-01-26T23:46:31.663243+0000","last_peered":"2024-01-26T23:46:31.663243+0000","last_clean":"2024-01-26T23:46:31.663243+0000","last_became_active":"2024-01-26T23:46:31.659972+0000","last_became_peered":"2024-01-26T23:46:31.659972+0000","last_unstale":"2024-01-26T23:46:31.663243+0000","last_undegraded":"2024-01-26T23:46:31.663243+0000","last_fullsized":"2024-01-26T23:46:31.663243+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:23:09.740800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"111'2","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.774206+0000","last_change":"2024-01-26T23:46:30.768055+0000","last_active":"2024-01-26T23:46:31.774206+0000","last_peered":"2024-01-26T23:46:31.774206+0000","last_clean":"2024-01-26T23:46:31.774206+0000","last_became_active":"2024-01-26T23:46:30.767751+0000","last_became_peered":"2024-01-26T23:46:30.767751+0000","last_unstale":"2024-01-26T23:46:31.774206+0000","last_undegraded":"2024-01-26T23:46:31.774206+0000","last_fullsized":"2024-01-26T23:46:31.774206+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:30:09.071510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.677633+0000","last_change":"2024-01-26T23:46:31.671797+0000","last_active":"2024-01-26T23:46:31.677633+0000","last_peered":"2024-01-26T23:46:31.677633+0000","last_clean":"2024-01-26T23:46:31.677633+0000","last_became_active":"2024-01-26T23:46:31.671535+0000","last_became_peered":"2024-01-26T23:46:31.671535+0000","last_unstale":"2024-01-26T23:46:31.677633+0000","last_undegraded":"2024-01-26T23:46:31.677633+0000","last_fullsized":"2024-01-26T23:46:31.677633+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:21:09.440118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"111'1","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.772424+0000","last_change":"2024-01-26T23:46:30.775593+0000","last_active":"2024-01-26T23:46:31.772424+0000","last_peered":"2024-01-26T23:46:31.772424+0000","last_clean":"2024-01-26T23:46:31.772424+0000","last_became_active":"2024-01-26T23:46:30.775280+0000","last_became_peered":"2024-01-26T23:46:30.775280+0000","last_unstale":"2024-01-26T23:46:31.772424+0000","last_undegraded":"2024-01-26T23:46:31.772424+0000","last_fullsized":"2024-01-26T23:46:31.772424+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:30:09.071510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671822+0000","last_change":"2024-01-26T23:46:31.665817+0000","last_active":"2024-01-26T23:46:31.671822+0000","last_peered":"2024-01-26T23:46:31.671822+0000","last_clean":"2024-01-26T23:46:31.671822+0000","last_became_active":"2024-01-26T23:46:31.665513+0000","last_became_peered":"2024-01-26T23:46:31.665513+0000","last_unstale":"2024-01-26T23:46:31.671822+0000","last_undegraded":"2024-01-26T23:46:31.671822+0000","last_fullsized":"2024-01-26T23:46:31.671822+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:56:41.826944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"111'2","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.774170+0000","last_change":"2024-01-26T23:46:30.768575+0000","last_active":"2024-01-26T23:46:31.774170+0000","last_peered":"2024-01-26T23:46:31.774170+0000","last_clean":"2024-01-26T23:46:31.774170+0000","last_became_active":"2024-01-26T23:46:30.768327+0000","last_became_peered":"2024-01-26T23:46:30.768327+0000","last_unstale":"2024-01-26T23:46:31.774170+0000","last_undegraded":"2024-01-26T23:46:31.774170+0000","last_fullsized":"2024-01-26T23:46:31.774170+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:41:51.840628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.666067+0000","last_change":"2024-01-26T23:46:31.663557+0000","last_active":"2024-01-26T23:46:31.666067+0000","last_peered":"2024-01-26T23:46:31.666067+0000","last_clean":"2024-01-26T23:46:31.666067+0000","last_became_active":"2024-01-26T23:46:31.663319+0000","last_became_peered":"2024-01-26T23:46:31.663319+0000","last_unstale":"2024-01-26T23:46:31.666067+0000","last_undegraded":"2024-01-26T23:46:31.666067+0000","last_fullsized":"2024-01-26T23:46:31.666067+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:48.805791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.658837+0000","last_change":"2024-01-26T23:46:30.777564+0000","last_active":"2024-01-26T23:46:31.658837+0000","last_peered":"2024-01-26T23:46:31.658837+0000","last_clean":"2024-01-26T23:46:31.658837+0000","last_became_active":"2024-01-26T23:46:30.777242+0000","last_became_peered":"2024-01-26T23:46:30.777242+0000","last_unstale":"2024-01-26T23:46:31.658837+0000","last_undegraded":"2024-01-26T23:46:31.658837+0000","last_fullsized":"2024-01-26T23:46:31.658837+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:08:16.334908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.663666+0000","last_change":"2024-01-26T23:46:31.657382+0000","last_active":"2024-01-26T23:46:31.663666+0000","last_peered":"2024-01-26T23:46:31.663666+0000","last_clean":"2024-01-26T23:46:31.663666+0000","last_became_active":"2024-01-26T23:46:31.657142+0000","last_became_peered":"2024-01-26T23:46:31.657142+0000","last_unstale":"2024-01-26T23:46:31.663666+0000","last_undegraded":"2024-01-26T23:46:31.663666+0000","last_fullsized":"2024-01-26T23:46:31.663666+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:18:30.549273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.652138+0000","last_change":"2024-01-26T23:46:30.777901+0000","last_active":"2024-01-26T23:46:31.652138+0000","last_peered":"2024-01-26T23:46:31.652138+0000","last_clean":"2024-01-26T23:46:31.652138+0000","last_became_active":"2024-01-26T23:46:30.777649+0000","last_became_peered":"2024-01-26T23:46:30.777649+0000","last_unstale":"2024-01-26T23:46:31.652138+0000","last_undegraded":"2024-01-26T23:46:31.652138+0000","last_fullsized":"2024-01-26T23:46:31.652138+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:46:39.085189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675654+0000","last_change":"2024-01-26T23:46:31.672269+0000","last_active":"2024-01-26T23:46:31.675654+0000","last_peered":"2024-01-26T23:46:31.675654+0000","last_clean":"2024-01-26T23:46:31.675654+0000","last_became_active":"2024-01-26T23:46:31.671924+0000","last_became_peered":"2024-01-26T23:46:31.671924+0000","last_unstale":"2024-01-26T23:46:31.675654+0000","last_undegraded":"2024-01-26T23:46:31.675654+0000","last_fullsized":"2024-01-26T23:46:31.675654+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:36:26.521199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.660165+0000","last_change":"2024-01-26T23:46:30.771183+0000","last_active":"2024-01-26T23:46:31.660165+0000","last_peered":"2024-01-26T23:46:31.660165+0000","last_clean":"2024-01-26T23:46:31.660165+0000","last_became_active":"2024-01-26T23:46:30.770867+0000","last_became_peered":"2024-01-26T23:46:30.770867+0000","last_unstale":"2024-01-26T23:46:31.660165+0000","last_undegraded":"2024-01-26T23:46:31.660165+0000","last_fullsized":"2024-01-26T23:46:31.660165+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:25:06.792314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.665959+0000","last_change":"2024-01-26T23:46:31.661160+0000","last_active":"2024-01-26T23:46:31.665959+0000","last_peered":"2024-01-26T23:46:31.665959+0000","last_clean":"2024-01-26T23:46:31.665959+0000","last_became_active":"2024-01-26T23:46:31.660925+0000","last_became_peered":"2024-01-26T23:46:31.660925+0000","last_unstale":"2024-01-26T23:46:31.665959+0000","last_undegraded":"2024-01-26T23:46:31.665959+0000","last_fullsized":"2024-01-26T23:46:31.665959+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:48.805791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"111'2","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.773899+0000","last_change":"2024-01-26T23:46:30.768210+0000","last_active":"2024-01-26T23:46:31.773899+0000","last_peered":"2024-01-26T23:46:31.773899+0000","last_clean":"2024-01-26T23:46:31.773899+0000","last_became_active":"2024-01-26T23:46:30.767953+0000","last_became_peered":"2024-01-26T23:46:30.767953+0000","last_unstale":"2024-01-26T23:46:31.773899+0000","last_undegraded":"2024-01-26T23:46:31.773899+0000","last_fullsized":"2024-01-26T23:46:31.773899+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:23:08.739457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675423+0000","last_change":"2024-01-26T23:46:31.668761+0000","last_active":"2024-01-26T23:46:31.675423+0000","last_peered":"2024-01-26T23:46:31.675423+0000","last_clean":"2024-01-26T23:46:31.675423+0000","last_became_active":"2024-01-26T23:46:31.668456+0000","last_became_peered":"2024-01-26T23:46:31.668456+0000","last_unstale":"2024-01-26T23:46:31.675423+0000","last_undegraded":"2024-01-26T23:46:31.675423+0000","last_fullsized":"2024-01-26T23:46:31.675423+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:51:54.286344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.659239+0000","last_change":"2024-01-26T23:46:30.771548+0000","last_active":"2024-01-26T23:46:31.659239+0000","last_peered":"2024-01-26T23:46:31.659239+0000","last_clean":"2024-01-26T23:46:31.659239+0000","last_became_active":"2024-01-26T23:46:30.771300+0000","last_became_peered":"2024-01-26T23:46:30.771300+0000","last_unstale":"2024-01-26T23:46:31.659239+0000","last_undegraded":"2024-01-26T23:46:31.659239+0000","last_fullsized":"2024-01-26T23:46:31.659239+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:07.859918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675695+0000","last_change":"2024-01-26T23:46:31.672735+0000","last_active":"2024-01-26T23:46:31.675695+0000","last_peered":"2024-01-26T23:46:31.675695+0000","last_clean":"2024-01-26T23:46:31.675695+0000","last_became_active":"2024-01-26T23:46:31.672465+0000","last_became_peered":"2024-01-26T23:46:31.672465+0000","last_unstale":"2024-01-26T23:46:31.675695+0000","last_undegraded":"2024-01-26T23:46:31.675695+0000","last_fullsized":"2024-01-26T23:46:31.675695+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:57:26.854523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.659544+0000","last_change":"2024-01-26T23:46:30.784052+0000","last_active":"2024-01-26T23:46:31.659544+0000","last_peered":"2024-01-26T23:46:31.659544+0000","last_clean":"2024-01-26T23:46:31.659544+0000","last_became_active":"2024-01-26T23:46:30.783827+0000","last_became_peered":"2024-01-26T23:46:30.783827+0000","last_unstale":"2024-01-26T23:46:31.659544+0000","last_undegraded":"2024-01-26T23:46:31.659544+0000","last_fullsized":"2024-01-26T23:46:31.659544+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:04:08.622450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.668796+0000","last_change":"2024-01-26T23:46:31.667235+0000","last_active":"2024-01-26T23:46:31.668796+0000","last_peered":"2024-01-26T23:46:31.668796+0000","last_clean":"2024-01-26T23:46:31.668796+0000","last_became_active":"2024-01-26T23:46:31.666976+0000","last_became_peered":"2024-01-26T23:46:31.666976+0000","last_unstale":"2024-01-26T23:46:31.668796+0000","last_undegraded":"2024-01-26T23:46:31.668796+0000","last_fullsized":"2024-01-26T23:46:31.668796+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:23:09.740800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.655190+0000","last_change":"2024-01-26T23:46:30.771368+0000","last_active":"2024-01-26T23:46:31.655190+0000","last_peered":"2024-01-26T23:46:31.655190+0000","last_clean":"2024-01-26T23:46:31.655190+0000","last_became_active":"2024-01-26T23:46:30.771131+0000","last_became_peered":"2024-01-26T23:46:30.771131+0000","last_unstale":"2024-01-26T23:46:31.655190+0000","last_undegraded":"2024-01-26T23:46:31.655190+0000","last_fullsized":"2024-01-26T23:46:31.655190+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:46:39.085189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671187+0000","last_change":"2024-01-26T23:46:31.669379+0000","last_active":"2024-01-26T23:46:31.671187+0000","last_peered":"2024-01-26T23:46:31.671187+0000","last_clean":"2024-01-26T23:46:31.671187+0000","last_became_active":"2024-01-26T23:46:31.669103+0000","last_became_peered":"2024-01-26T23:46:31.669103+0000","last_unstale":"2024-01-26T23:46:31.671187+0000","last_undegraded":"2024-01-26T23:46:31.671187+0000","last_fullsized":"2024-01-26T23:46:31.671187+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:20:32.077253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.656731+0000","last_change":"2024-01-26T23:46:30.777287+0000","last_active":"2024-01-26T23:46:31.656731+0000","last_peered":"2024-01-26T23:46:31.656731+0000","last_clean":"2024-01-26T23:46:31.656731+0000","last_became_active":"2024-01-26T23:46:30.773908+0000","last_became_peered":"2024-01-26T23:46:30.773908+0000","last_unstale":"2024-01-26T23:46:31.656731+0000","last_undegraded":"2024-01-26T23:46:31.656731+0000","last_fullsized":"2024-01-26T23:46:31.656731+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:28.962140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.676474+0000","last_change":"2024-01-26T23:46:31.673074+0000","last_active":"2024-01-26T23:46:31.676474+0000","last_peered":"2024-01-26T23:46:31.676474+0000","last_clean":"2024-01-26T23:46:31.676474+0000","last_became_active":"2024-01-26T23:46:31.668846+0000","last_became_peered":"2024-01-26T23:46:31.668846+0000","last_unstale":"2024-01-26T23:46:31.676474+0000","last_undegraded":"2024-01-26T23:46:31.676474+0000","last_fullsized":"2024-01-26T23:46:31.676474+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:45.348777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.652367+0000","last_change":"2024-01-26T23:46:30.772801+0000","last_active":"2024-01-26T23:46:31.652367+0000","last_peered":"2024-01-26T23:46:31.652367+0000","last_clean":"2024-01-26T23:46:31.652367+0000","last_became_active":"2024-01-26T23:46:30.772337+0000","last_became_peered":"2024-01-26T23:46:30.772337+0000","last_unstale":"2024-01-26T23:46:31.652367+0000","last_undegraded":"2024-01-26T23:46:31.652367+0000","last_fullsized":"2024-01-26T23:46:31.652367+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:37.479741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678017+0000","last_change":"2024-01-26T23:46:31.677471+0000","last_active":"2024-01-26T23:46:31.678017+0000","last_peered":"2024-01-26T23:46:31.678017+0000","last_clean":"2024-01-26T23:46:31.678017+0000","last_became_active":"2024-01-26T23:46:31.677223+0000","last_became_peered":"2024-01-26T23:46:31.677223+0000","last_unstale":"2024-01-26T23:46:31.678017+0000","last_undegraded":"2024-01-26T23:46:31.678017+0000","last_fullsized":"2024-01-26T23:46:31.678017+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:46:27.325038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.654023+0000","last_change":"2024-01-26T23:46:30.763987+0000","last_active":"2024-01-26T23:46:31.654023+0000","last_peered":"2024-01-26T23:46:31.654023+0000","last_clean":"2024-01-26T23:46:31.654023+0000","last_became_active":"2024-01-26T23:46:30.763719+0000","last_became_peered":"2024-01-26T23:46:30.763719+0000","last_unstale":"2024-01-26T23:46:31.654023+0000","last_undegraded":"2024-01-26T23:46:31.654023+0000","last_fullsized":"2024-01-26T23:46:31.654023+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:41:51.840628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675370+0000","last_change":"2024-01-26T23:46:31.668096+0000","last_active":"2024-01-26T23:46:31.675370+0000","last_peered":"2024-01-26T23:46:31.675370+0000","last_clean":"2024-01-26T23:46:31.675370+0000","last_became_active":"2024-01-26T23:46:31.665095+0000","last_became_peered":"2024-01-26T23:46:31.665095+0000","last_unstale":"2024-01-26T23:46:31.675370+0000","last_undegraded":"2024-01-26T23:46:31.675370+0000","last_fullsized":"2024-01-26T23:46:31.675370+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:53:38.299979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.654449+0000","last_change":"2024-01-26T23:46:30.781240+0000","last_active":"2024-01-26T23:46:31.654449+0000","last_peered":"2024-01-26T23:46:31.654449+0000","last_clean":"2024-01-26T23:46:31.654449+0000","last_became_active":"2024-01-26T23:46:30.781056+0000","last_became_peered":"2024-01-26T23:46:30.781056+0000","last_unstale":"2024-01-26T23:46:31.654449+0000","last_undegraded":"2024-01-26T23:46:31.654449+0000","last_fullsized":"2024-01-26T23:46:31.654449+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:04.033669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.665879+0000","last_change":"2024-01-26T23:46:31.662608+0000","last_active":"2024-01-26T23:46:31.665879+0000","last_peered":"2024-01-26T23:46:31.665879+0000","last_clean":"2024-01-26T23:46:31.665879+0000","last_became_active":"2024-01-26T23:46:31.662342+0000","last_became_peered":"2024-01-26T23:46:31.662342+0000","last_unstale":"2024-01-26T23:46:31.665879+0000","last_undegraded":"2024-01-26T23:46:31.665879+0000","last_fullsized":"2024-01-26T23:46:31.665879+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:12:26.391081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.649742+0000","last_change":"2024-01-26T23:46:30.775933+0000","last_active":"2024-01-26T23:46:31.649742+0000","last_peered":"2024-01-26T23:46:31.649742+0000","last_clean":"2024-01-26T23:46:31.649742+0000","last_became_active":"2024-01-26T23:46:30.775286+0000","last_became_peered":"2024-01-26T23:46:30.775286+0000","last_unstale":"2024-01-26T23:46:31.649742+0000","last_undegraded":"2024-01-26T23:46:31.649742+0000","last_fullsized":"2024-01-26T23:46:31.649742+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:24.875367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"112'5140","reported_seq":5488,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-26T23:46:35.629783+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:46:35.629783+0000","last_peered":"2024-01-26T23:46:35.629783+0000","last_clean":"2024-01-26T23:46:35.629783+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:46:35.629783+0000","last_undegraded":"2024-01-26T23:46:35.629783+0000","last_fullsized":"2024-01-26T23:46:35.629783+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":5140,"log_dups_size":0,"ondisk_log_size":5140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":743,"num_read_kb":1452,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":743,"num_read_kb":1452,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5140,"ondisk_log_size":5140,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575051,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311512,"kb_used_data":1936,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060328,"statfs":{"total":96636764160,"available":96317775872,"internally_reserved":0,"allocated":1982464,"data_stored":1292578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.54100000000000004,"15min":0.56499999999999995},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.74099999999999999,"5min":0.79500000000000004,"15min":0.82299999999999995},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.55600000000000005,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.83999999999999997},"last":0.47099999999999997}]},{"osd":1,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.55200000000000005,"15min":0.57899999999999996},"min":{"1min":0.36699999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67400000000000004,"5min":0.81299999999999994,"15min":0.82899999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.44500000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.56299999999999994,"15min":0.57699999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.81699999999999995},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.58299999999999996,"15min":0.59799999999999998},"min":{"1min":0.30199999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75700000000000001,"5min":0.81899999999999995,"15min":0.88600000000000001},"last":0.40200000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.77100000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.497},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.58699999999999997,"15min":0.60099999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71699999999999997,"5min":0.83899999999999997,"15min":0.86399999999999999},"last":0.441}]},{"osd":4,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46100000000000002,"15min":0.45600000000000002},"min":{"1min":0.252,"5min":0.247,"15min":0.247},"max":{"1min":0.64300000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.48799999999999999,"15min":0.48199999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.66400000000000003,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.378}]},{"osd":5,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.48899999999999999},"min":{"1min":0.27200000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.70599999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54200000000000004,"15min":0.52700000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.73599999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.41199999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.55300000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78700000000000003,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55700000000000005,"15min":0.54900000000000004},"min":{"1min":0.32000000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70599999999999996,"5min":0.85199999999999998,"15min":0.86899999999999999},"last":0.43099999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575051,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302752,"kb_used_data":1368,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069088,"statfs":{"total":96636764160,"available":96326746112,"internally_reserved":0,"allocated":1400832,"data_stored":704266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55800000000000005,"15min":0.57699999999999996},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26300000000000001},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.57199999999999995,"15min":0.59199999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.68300000000000005,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.45300000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.54300000000000004,"15min":0.57199999999999995},"min":{"1min":0.317,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.622,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.505},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.34399999999999997},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.34100000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55900000000000005,"15min":0.58499999999999996},"min":{"1min":0.39600000000000002,"5min":0.35499999999999998,"15min":0.29599999999999999},"max":{"1min":0.67200000000000004,"5min":0.74199999999999999,"15min":0.85799999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.56999999999999995,"15min":0.58899999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.33900000000000002},"max":{"1min":0.621,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.48399999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.29999999999999999},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60999999999999999,"15min":0.622},"min":{"1min":0.41899999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.58199999999999996}]},{"osd":4,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47699999999999998,"15min":0.47299999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.22},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.498,"15min":0.495},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.64600000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48599999999999999,"15min":0.48499999999999999},"min":{"1min":0.316,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.66100000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54000000000000004,"15min":0.53100000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26200000000000001},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":0.49399999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.246},"max":{"1min":0.64100000000000001,"5min":0.82799999999999996,"15min":0.85899999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.53500000000000003,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.308},"max":{"1min":0.68600000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.378}]}]},{"osd":1,"up_from":13,"seq":55834575051,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302736,"kb_used_data":1352,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069104,"statfs":{"total":96636764160,"available":96326762496,"internally_reserved":0,"allocated":1384448,"data_stored":702224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41599999999999998,"15min":0.40899999999999997},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.72599999999999998},"last":0.315},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.44900000000000001,"15min":0.44600000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.78600000000000003,"5min":1.337,"15min":1.337},"last":0.28999999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.41699999999999998,"15min":0.41499999999999998},"min":{"1min":0.254,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.68799999999999994,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.48199999999999998,"15min":0.47499999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.28999999999999998},"max":{"1min":0.74099999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.42199999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52500000000000002,"15min":0.51400000000000001},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":0.75900000000000001,"5min":1.782,"15min":1.782},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.48899999999999999,"15min":0.47999999999999998},"min":{"1min":0.32000000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.71499999999999997,"5min":1.379,"15min":1.379},"last":0.38800000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.64200000000000002,"15min":0.64900000000000002},"min":{"1min":0.435,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":0.81699999999999995,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.65700000000000003,"15min":0.66200000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.84299999999999997,"5min":0.91900000000000004,"15min":0.95699999999999996},"last":0.56100000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.67700000000000005,"15min":0.67300000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":0.86499999999999999,"5min":1.208,"15min":1.208},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.67700000000000005,"15min":0.66800000000000004},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.42499999999999999},"max":{"1min":0.82899999999999996,"5min":0.96599999999999997,"15min":1.018},"last":0.61299999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.68000000000000005,"15min":0.67800000000000005},"min":{"1min":0.47799999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91900000000000004,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.68000000000000005,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.78300000000000003,"5min":1.022,"15min":1.022},"last":0.504}]},{"osd":7,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.70499999999999996,"15min":0.71099999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89200000000000002,"5min":0.98199999999999998,"15min":1.0069999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.73099999999999998,"15min":0.72299999999999998},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.438},"max":{"1min":0.95399999999999996,"5min":1.296,"15min":1.296},"last":0.496}]}]},{"osd":0,"up_from":13,"seq":55834575050,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307432,"kb_used_data":1952,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064408,"statfs":{"total":96636764160,"available":96321953792,"internally_reserved":0,"allocated":1998848,"data_stored":1294736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.41999999999999998,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.45100000000000001,"15min":0.45500000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27300000000000002},"max":{"1min":1.363,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41199999999999998,"15min":0.42499999999999999},"min":{"1min":0.27900000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.57499999999999996,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.45700000000000002,"15min":0.47099999999999997},"min":{"1min":0.32700000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63700000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.36699999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.47399999999999998,"15min":0.46899999999999997},"min":{"1min":0.32300000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.145,"5min":2.145,"15min":2.145},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48599999999999999,"15min":0.48799999999999999},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64300000000000002,"5min":1.198,"15min":1.198},"last":0.32100000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.64400000000000002,"15min":0.65200000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.625,"15min":0.65000000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.752,"5min":0.94299999999999995,"15min":0.98899999999999999},"last":0.42499999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64200000000000002,"15min":0.65900000000000003},"min":{"1min":0.42199999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.67500000000000004,"15min":0.68100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.70799999999999996}]},{"osd":6,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.63,"15min":0.66700000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80200000000000005,"5min":0.89800000000000002,"15min":1.032},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.67400000000000004,"15min":0.69499999999999995},"min":{"1min":0.48299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.61299999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.028,"15min":0.83199999999999996},"min":{"1min":0.501,"5min":0.435,"15min":0.435},"max":{"1min":4.8710000000000004,"5min":25.965,"15min":25.965},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.1379999999999999,"15min":0.875},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.433},"max":{"1min":4.9199999999999999,"5min":33.085000000000001,"15min":33.085000000000001},"last":0.64400000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575050,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306828,"kb_used_data":1348,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065012,"statfs":{"total":96636764160,"available":96322572288,"internally_reserved":0,"allocated":1380352,"data_stored":702154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.51700000000000002,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.437,"15min":0.42299999999999999},"min":{"1min":0.33500000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.28299999999999997}]},{"osd":1,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.432,"15min":0.42799999999999999},"min":{"1min":0.26900000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.61899999999999999,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.377,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.38900000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46999999999999997,"15min":0.48599999999999999},"min":{"1min":0.35999999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.64800000000000002,"5min":0.65500000000000003,"15min":0.74099999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.48099999999999998,"15min":0.47499999999999998},"min":{"1min":0.35599999999999998,"5min":0.32200000000000001,"15min":0.307},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.63600000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.63200000000000001,"15min":0.63400000000000001},"min":{"1min":0.41499999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.80900000000000005,"5min":1.087,"15min":1.087},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.65200000000000002,"15min":0.64400000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.874,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.68200000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.65100000000000002,"15min":0.65200000000000002},"min":{"1min":0.375,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.90300000000000002,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.67300000000000004,"15min":0.67900000000000005},"min":{"1min":0.41799999999999998,"5min":0.39800000000000002,"15min":0.35199999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.085},"last":0.70099999999999996}]},{"osd":6,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67900000000000005,"15min":0.68100000000000005},"min":{"1min":0.47699999999999998,"5min":0.439,"15min":0.40899999999999997},"max":{"1min":1.022,"5min":1.022,"15min":1.1080000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.67700000000000005,"15min":0.68300000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.38800000000000001},"max":{"1min":0.83799999999999997,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.71699999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.69499999999999995,"15min":0.70299999999999996},"min":{"1min":0.52100000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94699999999999995,"5min":1.0029999999999999,"15min":1.127},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.72299999999999998,"15min":0.73299999999999998},"min":{"1min":0.48899999999999999,"5min":0.48199999999999998,"15min":0.42099999999999999},"max":{"1min":1.05,"5min":1.052,"15min":1.139},"last":0.66000000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575049,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306828,"kb_used_data":1348,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065012,"statfs":{"total":96636764160,"available":96322572288,"internally_reserved":0,"allocated":1380352,"data_stored":702154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.39900000000000002,"15min":0.39400000000000002},"min":{"1min":0.26500000000000001,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.45000000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.44,"15min":0.439},"min":{"1min":0.25800000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.58699999999999997,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.42399999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.46600000000000003,"15min":0.45500000000000002},"min":{"1min":0.27900000000000003,"5min":0.253,"15min":0.253},"max":{"1min":0.57799999999999996,"5min":1.633,"15min":1.633},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.48399999999999999,"15min":0.48099999999999998},"min":{"1min":0.35199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.54900000000000004,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58099999999999996}]},{"osd":2,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.48899999999999999,"15min":0.47899999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.54600000000000004,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.33200000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.53700000000000003,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.503}]},{"osd":4,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.68200000000000005,"15min":0.66300000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.878,"5min":3.306,"15min":3.306},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.41499999999999998,"5min":0.40200000000000002,"15min":0.379},"max":{"1min":0.85199999999999998,"5min":0.95799999999999996,"15min":1.008},"last":0.40100000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.65700000000000003,"15min":0.66100000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82499999999999996,"5min":1.593,"15min":1.593},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.67500000000000004,"15min":0.67300000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.84399999999999997,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.46200000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.67100000000000004,"15min":0.67700000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.77800000000000002,"5min":1.506,"15min":1.506},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.69099999999999995,"15min":0.69299999999999995},"min":{"1min":0.48799999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.82599999999999996,"5min":1.427,"15min":1.427},"last":0.53100000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.70799999999999996,"15min":0.69799999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.42499999999999999},"max":{"1min":0.73899999999999999,"5min":3.238,"15min":3.238},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.747,"15min":0.73899999999999999},"min":{"1min":0.54100000000000004,"5min":0.48899999999999999,"15min":0.48699999999999999},"max":{"1min":0.86699999999999999,"5min":3.351,"15min":3.351},"last":0.64100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575051,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302740,"kb_used_data":1356,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069100,"statfs":{"total":96636764160,"available":96326758400,"internally_reserved":0,"allocated":1388544,"data_stored":702314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.54800000000000004,"15min":0.57799999999999996},"min":{"1min":0.28999999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57699999999999996,"5min":1.21,"15min":1.21},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.58499999999999996,"15min":0.58999999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.367,"5min":5.367,"15min":5.367},"last":0.48799999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.55700000000000005,"15min":0.59299999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68899999999999995,"5min":0.77500000000000002,"15min":0.88},"last":0.442},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71499999999999997,"5min":0.80600000000000005,"15min":0.81699999999999995},"last":0.501}]},{"osd":2,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53500000000000003,"15min":0.56799999999999995},"min":{"1min":0.36699999999999999,"5min":0.33300000000000002,"15min":0.32700000000000001},"max":{"1min":0.56599999999999995,"5min":0.81100000000000005,"15min":0.85999999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.58899999999999997,"15min":0.60499999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.69599999999999995,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.50700000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.56000000000000005,"15min":0.58699999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.318},"max":{"1min":0.66900000000000004,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.434},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56699999999999995,"15min":0.58599999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.35699999999999998},"max":{"1min":0.64900000000000002,"5min":1.05,"15min":1.05},"last":0.51300000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.46000000000000002,"15min":0.46300000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.63100000000000001,"5min":0.76200000000000001,"15min":0.77600000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.48499999999999999,"15min":0.48199999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68300000000000005,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.47599999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.47099999999999997,"15min":0.47199999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.27200000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.73899999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.311},"max":{"1min":0.72899999999999998,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.374}]},{"osd":7,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.36699999999999999,"5min":0.36499999999999999,"15min":0.35799999999999998},"max":{"1min":0.753,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.57099999999999995,"15min":0.56899999999999995},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81000000000000005,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.45600000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575051,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298636,"kb_used_data":1348,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073204,"statfs":{"total":96636764160,"available":96330960896,"internally_reserved":0,"allocated":1380352,"data_stored":702154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.26000000000000001},"max":{"1min":0.64700000000000002,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.502,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.315,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.65800000000000003,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.63}]},{"osd":1,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.61199999999999999,"15min":0.622},"min":{"1min":0.34799999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66300000000000003,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.629,"15min":0.63400000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.39300000000000002},"max":{"1min":0.71199999999999997,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.55900000000000005,"15min":0.56999999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.30099999999999999},"max":{"1min":0.67400000000000004,"5min":0.83699999999999997,"15min":0.83799999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.58999999999999997,"15min":0.59699999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69699999999999995,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":0.59199999999999997}]},{"osd":3,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.57099999999999995,"15min":0.58399999999999996},"min":{"1min":0.38100000000000001,"5min":0.35999999999999999,"15min":0.314},"max":{"1min":0.68799999999999994,"5min":0.86199999999999999,"15min":0.88400000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.59899999999999998,"15min":0.59199999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73599999999999999,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.72399999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.48899999999999999,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.48999999999999999,"15min":0.48699999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":0.626,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.44}]},{"osd":6,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.48799999999999999,"15min":0.47699999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65400000000000003,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.52100000000000002,"15min":0.51900000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.76200000000000001,"5min":1.99,"15min":1.99},"last":0.50600000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.312,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71399999999999997,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79300000000000004,"5min":0.82199999999999995,"15min":0.83099999999999996},"last":0.51600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:46:38.035 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:38.034+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:38.889 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:38.888+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:39.753 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:39.751+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:40.604 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:40.603+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:40.734 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:46:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.877+0000 7fc498d58700 1 Processor -- start 2024-01-26T23:46:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.877+0000 7fc498d58700 1 -- start start 2024-01-26T23:46:40.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.878+0000 7fc498d58700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc494159590 0x7fc49415d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:40.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.878+0000 7fc498d58700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc49415dd90 con 0x7fc49406c3c0 2024-01-26T23:46:40.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.878+0000 7fc498d58700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc49415e070 con 0x7fc49415d8f0 2024-01-26T23:46:40.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.878+0000 7fc498d58700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc49415e350 con 0x7fc494159590 2024-01-26T23:46:40.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.878+0000 7fc492d9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc49406c3c0 0x7fc494069370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34894/0 (socket says 172.21.15.46:34894) 2024-01-26T23:46:40.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.878+0000 7fc492d9d700 1 -- 172.21.15.46:0/2593870696 learned_addr learned my addr 172.21.15.46:0/2593870696 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.878+0000 7fc49259c700 1 --2- 172.21.15.46:0/2593870696 >> v2:172.21.15.106:3300/0 conn(0x7fc494159590 0x7fc49415d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.878+0000 7fc49159a700 1 -- 172.21.15.46:0/2593870696 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770383407 0 0) 0x7fc49415dd90 con 0x7fc49406c3c0 2024-01-26T23:46:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.878+0000 7fc49159a700 1 -- 172.21.15.46:0/2593870696 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc47c003570 con 0x7fc49406c3c0 2024-01-26T23:46:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.879+0000 7fc49259c700 1 -- 172.21.15.46:0/2593870696 >> v1:172.21.15.106:6789/0 conn(0x7fc49415d8f0 legacy=0x7fc49406bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.879+0000 7fc49159a700 1 -- 172.21.15.46:0/2593870696 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4172232652 0 0) 0x7fc47c003570 con 0x7fc49406c3c0 2024-01-26T23:46:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.879+0000 7fc49259c700 1 -- 172.21.15.46:0/2593870696 >> v1:172.21.15.46:6789/0 conn(0x7fc49406c3c0 legacy=0x7fc494069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.879+0000 7fc49259c700 1 -- 172.21.15.46:0/2593870696 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc494068b70 con 0x7fc494159590 2024-01-26T23:46:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.879+0000 7fc49259c700 1 --2- 172.21.15.46:0/2593870696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc494159590 0x7fc49415d3b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc484009860 tx=0x7fc4840091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d54723850dda639 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.879+0000 7fc49159a700 1 -- 172.21.15.46:0/2593870696 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c0017f0 con 0x7fc494159590 2024-01-26T23:46:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.879+0000 7fc49159a700 1 -- 172.21.15.46:0/2593870696 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc484011420 con 0x7fc494159590 2024-01-26T23:46:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.879+0000 7fc49159a700 1 -- 172.21.15.46:0/2593870696 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4840117c0 con 0x7fc494159590 2024-01-26T23:46:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.879+0000 7fc498d58700 1 -- 172.21.15.46:0/2593870696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc494159590 msgr2=0x7fc49415d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.879+0000 7fc498d58700 1 --2- 172.21.15.46:0/2593870696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc494159590 0x7fc49415d3b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc484009860 tx=0x7fc4840091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.880+0000 7fc498d58700 1 -- 172.21.15.46:0/2593870696 shutdown_connections 2024-01-26T23:46:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.880+0000 7fc498d58700 1 --2- 172.21.15.46:0/2593870696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc494159590 0x7fc49415d3b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.880+0000 7fc498d58700 1 -- 172.21.15.46:0/2593870696 >> 172.21.15.46:0/2593870696 conn(0x7fc4940ce530 msgr2=0x7fc4940ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:40.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.880+0000 7fc498d58700 1 -- 172.21.15.46:0/2593870696 shutdown_connections 2024-01-26T23:46:40.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.880+0000 7fc498d58700 1 -- 172.21.15.46:0/2593870696 wait complete. 2024-01-26T23:46:40.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.881+0000 7fc498d58700 1 Processor -- start 2024-01-26T23:46:40.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.881+0000 7fc498d58700 1 -- start start 2024-01-26T23:46:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.882+0000 7fc498d58700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49406c3c0 0x7fc49415c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.882+0000 7fc498d58700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc494159590 0x7fc49406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.882+0000 7fc498d58700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc49415d8f0 0x7fc49406dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.882+0000 7fc498d58700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4940c35c0 con 0x7fc49406c3c0 2024-01-26T23:46:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.882+0000 7fc498d58700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49416a990 con 0x7fc49415d8f0 2024-01-26T23:46:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.882+0000 7fc498d58700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc49415cf20 con 0x7fc494159590 2024-01-26T23:46:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.882+0000 7fc49259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49406c3c0 0x7fc49415c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.882+0000 7fc49259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49406c3c0 0x7fc49415c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35398/0 (socket says 172.21.15.46:35398) 2024-01-26T23:46:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.882+0000 7fc49259c700 1 -- 172.21.15.46:0/3016915124 learned_addr learned my addr 172.21.15.46:0/3016915124 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.882+0000 7fc491d9b700 1 --2- 172.21.15.46:0/3016915124 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc494159590 0x7fc49406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.882+0000 7fc492d9d700 1 --2- 172.21.15.46:0/3016915124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc49415d8f0 0x7fc49406dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.882+0000 7fc49259c700 1 -- 172.21.15.46:0/3016915124 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc494159590 msgr2=0x7fc49406d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.882+0000 7fc49259c700 1 --2- 172.21.15.46:0/3016915124 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc494159590 0x7fc49406d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.882+0000 7fc49259c700 1 -- 172.21.15.46:0/3016915124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc49415d8f0 msgr2=0x7fc49406dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.882+0000 7fc49259c700 1 --2- 172.21.15.46:0/3016915124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc49415d8f0 0x7fc49406dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.882+0000 7fc49259c700 1 -- 172.21.15.46:0/3016915124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc484009420 con 0x7fc49406c3c0 2024-01-26T23:46:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.882+0000 7fc49259c700 1 --2- 172.21.15.46:0/3016915124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49406c3c0 0x7fc49415c9e0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fc484009f00 tx=0x7fc484017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.883+0000 7fc4827fc700 1 -- 172.21.15.46:0/3016915124 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc484000d10 con 0x7fc49406c3c0 2024-01-26T23:46:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.883+0000 7fc4827fc700 1 -- 172.21.15.46:0/3016915124 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc484010ab0 con 0x7fc49406c3c0 2024-01-26T23:46:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.883+0000 7fc498d58700 1 -- 172.21.15.46:0/3016915124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc49416d160 con 0x7fc49406c3c0 2024-01-26T23:46:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.883+0000 7fc4827fc700 1 -- 172.21.15.46:0/3016915124 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc484010e50 con 0x7fc49406c3c0 2024-01-26T23:46:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.883+0000 7fc498d58700 1 -- 172.21.15.46:0/3016915124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc49416d720 con 0x7fc49406c3c0 2024-01-26T23:46:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.884+0000 7fc498d58700 1 -- 172.21.15.46:0/3016915124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc49404c710 con 0x7fc49406c3c0 2024-01-26T23:46:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.884+0000 7fc4827fc700 1 -- 172.21.15.46:0/3016915124 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fc48401d640 con 0x7fc49406c3c0 2024-01-26T23:46:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.885+0000 7fc4827fc700 1 --2- 172.21.15.46:0/3016915124 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc474066ea0 0x7fc474069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.885+0000 7fc4827fc700 1 -- 172.21.15.46:0/3016915124 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fc484086030 con 0x7fc49406c3c0 2024-01-26T23:46:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.885+0000 7fc491d9b700 1 --2- 172.21.15.46:0/3016915124 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc474066ea0 0x7fc474069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.885+0000 7fc491d9b700 1 --2- 172.21.15.46:0/3016915124 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc474066ea0 0x7fc474069350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc49415dcc0 tx=0x7fc488007e40 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:40.889+0000 7fc4827fc700 1 -- 172.21.15.46:0/3016915124 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc4840530b0 con 0x7fc49406c3c0 2024-01-26T23:46:41.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.092+0000 7fc498d58700 1 -- 172.21.15.46:0/3016915124 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc49406c9a0 con 0x7fc474066ea0 2024-01-26T23:46:41.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.100+0000 7fc4827fc700 1 -- 172.21.15.46:0/3016915124 <== mgr.6365 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289409 (secure 0 0 0) 0x7fc49406c9a0 con 0x7fc474066ea0 2024-01-26T23:46:41.105 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:41.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.103+0000 7fc498d58700 1 -- 172.21.15.46:0/3016915124 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc474066ea0 msgr2=0x7fc474069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:41.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.103+0000 7fc498d58700 1 --2- 172.21.15.46:0/3016915124 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc474066ea0 0x7fc474069350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc49415dcc0 tx=0x7fc488007e40 comp rx=0 tx=0).stop 2024-01-26T23:46:41.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.103+0000 7fc498d58700 1 -- 172.21.15.46:0/3016915124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49406c3c0 msgr2=0x7fc49415c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.103+0000 7fc498d58700 1 --2- 172.21.15.46:0/3016915124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49406c3c0 0x7fc49415c9e0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fc484009f00 tx=0x7fc484017000 comp rx=0 tx=0).stop 2024-01-26T23:46:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.103+0000 7fc498d58700 1 -- 172.21.15.46:0/3016915124 shutdown_connections 2024-01-26T23:46:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.103+0000 7fc498d58700 1 --2- 172.21.15.46:0/3016915124 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fc474066ea0 0x7fc474069350 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.103+0000 7fc498d58700 1 --2- 172.21.15.46:0/3016915124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49406c3c0 0x7fc49415c9e0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.103+0000 7fc498d58700 1 --2- 172.21.15.46:0/3016915124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc49415d8f0 0x7fc49406dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.103+0000 7fc498d58700 1 --2- 172.21.15.46:0/3016915124 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc494159590 0x7fc49406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.103+0000 7fc498d58700 1 -- 172.21.15.46:0/3016915124 >> 172.21.15.46:0/3016915124 conn(0x7fc4940ce530 msgr2=0x7fc4940ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.104+0000 7fc498d58700 1 -- 172.21.15.46:0/3016915124 shutdown_connections 2024-01-26T23:46:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.104+0000 7fc498d58700 1 -- 172.21.15.46:0/3016915124 wait complete. 2024-01-26T23:46:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:46:41.118 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-26T23:46:39.637218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":743,"num_read_kb":1452,"num_write":323,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5178,"ondisk_log_size":5178,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2439464,"kb_used_data":12008,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752535256,"statfs":{"total":773094113280,"available":770596102144,"internally_reserved":0,"allocated":12296192,"data_stored":6802580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001420"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.670514+0000","last_change":"2024-01-26T23:46:31.666841+0000","last_active":"2024-01-26T23:46:31.670514+0000","last_peered":"2024-01-26T23:46:31.670514+0000","last_clean":"2024-01-26T23:46:31.670514+0000","last_became_active":"2024-01-26T23:46:31.666343+0000","last_became_peered":"2024-01-26T23:46:31.666343+0000","last_unstale":"2024-01-26T23:46:31.670514+0000","last_undegraded":"2024-01-26T23:46:31.670514+0000","last_fullsized":"2024-01-26T23:46:31.670514+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:20:40.199495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.663655+0000","last_change":"2024-01-26T23:46:31.660798+0000","last_active":"2024-01-26T23:46:31.663655+0000","last_peered":"2024-01-26T23:46:31.663655+0000","last_clean":"2024-01-26T23:46:31.663655+0000","last_became_active":"2024-01-26T23:46:31.660389+0000","last_became_peered":"2024-01-26T23:46:31.660389+0000","last_unstale":"2024-01-26T23:46:31.663655+0000","last_undegraded":"2024-01-26T23:46:31.663655+0000","last_fullsized":"2024-01-26T23:46:31.663655+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:15:01.090798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674189+0000","last_change":"2024-01-26T23:46:31.669466+0000","last_active":"2024-01-26T23:46:31.674189+0000","last_peered":"2024-01-26T23:46:31.674189+0000","last_clean":"2024-01-26T23:46:31.674189+0000","last_became_active":"2024-01-26T23:46:31.669137+0000","last_became_peered":"2024-01-26T23:46:31.669137+0000","last_unstale":"2024-01-26T23:46:31.674189+0000","last_undegraded":"2024-01-26T23:46:31.674189+0000","last_fullsized":"2024-01-26T23:46:31.674189+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:49:03.876856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671210+0000","last_change":"2024-01-26T23:46:31.664196+0000","last_active":"2024-01-26T23:46:31.671210+0000","last_peered":"2024-01-26T23:46:31.671210+0000","last_clean":"2024-01-26T23:46:31.671210+0000","last_became_active":"2024-01-26T23:46:31.663567+0000","last_became_peered":"2024-01-26T23:46:31.663567+0000","last_unstale":"2024-01-26T23:46:31.671210+0000","last_undegraded":"2024-01-26T23:46:31.671210+0000","last_fullsized":"2024-01-26T23:46:31.671210+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:18:30.549273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674416+0000","last_change":"2024-01-26T23:46:31.666630+0000","last_active":"2024-01-26T23:46:31.674416+0000","last_peered":"2024-01-26T23:46:31.674416+0000","last_clean":"2024-01-26T23:46:31.674416+0000","last_became_active":"2024-01-26T23:46:31.666304+0000","last_became_peered":"2024-01-26T23:46:31.666304+0000","last_unstale":"2024-01-26T23:46:31.674416+0000","last_undegraded":"2024-01-26T23:46:31.674416+0000","last_fullsized":"2024-01-26T23:46:31.674416+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:17:40.471277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678729+0000","last_change":"2024-01-26T23:46:31.677430+0000","last_active":"2024-01-26T23:46:31.678729+0000","last_peered":"2024-01-26T23:46:31.678729+0000","last_clean":"2024-01-26T23:46:31.678729+0000","last_became_active":"2024-01-26T23:46:31.677151+0000","last_became_peered":"2024-01-26T23:46:31.677151+0000","last_unstale":"2024-01-26T23:46:31.678729+0000","last_undegraded":"2024-01-26T23:46:31.678729+0000","last_fullsized":"2024-01-26T23:46:31.678729+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:15:00.551460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.677758+0000","last_change":"2024-01-26T23:46:31.674897+0000","last_active":"2024-01-26T23:46:31.677758+0000","last_peered":"2024-01-26T23:46:31.677758+0000","last_clean":"2024-01-26T23:46:31.677758+0000","last_became_active":"2024-01-26T23:46:31.674663+0000","last_became_peered":"2024-01-26T23:46:31.674663+0000","last_unstale":"2024-01-26T23:46:31.677758+0000","last_undegraded":"2024-01-26T23:46:31.677758+0000","last_fullsized":"2024-01-26T23:46:31.677758+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:38:57.979602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.669177+0000","last_change":"2024-01-26T23:46:31.659845+0000","last_active":"2024-01-26T23:46:31.669177+0000","last_peered":"2024-01-26T23:46:31.669177+0000","last_clean":"2024-01-26T23:46:31.669177+0000","last_became_active":"2024-01-26T23:46:31.659539+0000","last_became_peered":"2024-01-26T23:46:31.659539+0000","last_unstale":"2024-01-26T23:46:31.669177+0000","last_undegraded":"2024-01-26T23:46:31.669177+0000","last_fullsized":"2024-01-26T23:46:31.669177+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:26:22.946213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674606+0000","last_change":"2024-01-26T23:46:31.666924+0000","last_active":"2024-01-26T23:46:31.674606+0000","last_peered":"2024-01-26T23:46:31.674606+0000","last_clean":"2024-01-26T23:46:31.674606+0000","last_became_active":"2024-01-26T23:46:31.666670+0000","last_became_peered":"2024-01-26T23:46:31.666670+0000","last_unstale":"2024-01-26T23:46:31.674606+0000","last_undegraded":"2024-01-26T23:46:31.674606+0000","last_fullsized":"2024-01-26T23:46:31.674606+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:12:26.391081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.676434+0000","last_change":"2024-01-26T23:46:31.672168+0000","last_active":"2024-01-26T23:46:31.676434+0000","last_peered":"2024-01-26T23:46:31.676434+0000","last_clean":"2024-01-26T23:46:31.676434+0000","last_became_active":"2024-01-26T23:46:31.671762+0000","last_became_peered":"2024-01-26T23:46:31.671762+0000","last_unstale":"2024-01-26T23:46:31.676434+0000","last_undegraded":"2024-01-26T23:46:31.676434+0000","last_fullsized":"2024-01-26T23:46:31.676434+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:00:04.119988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674650+0000","last_change":"2024-01-26T23:46:31.667223+0000","last_active":"2024-01-26T23:46:31.674650+0000","last_peered":"2024-01-26T23:46:31.674650+0000","last_clean":"2024-01-26T23:46:31.674650+0000","last_became_active":"2024-01-26T23:46:31.666318+0000","last_became_peered":"2024-01-26T23:46:31.666318+0000","last_unstale":"2024-01-26T23:46:31.674650+0000","last_undegraded":"2024-01-26T23:46:31.674650+0000","last_fullsized":"2024-01-26T23:46:31.674650+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:09:35.596681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671217+0000","last_change":"2024-01-26T23:46:31.666856+0000","last_active":"2024-01-26T23:46:31.671217+0000","last_peered":"2024-01-26T23:46:31.671217+0000","last_clean":"2024-01-26T23:46:31.671217+0000","last_became_active":"2024-01-26T23:46:31.666519+0000","last_became_peered":"2024-01-26T23:46:31.666519+0000","last_unstale":"2024-01-26T23:46:31.671217+0000","last_undegraded":"2024-01-26T23:46:31.671217+0000","last_fullsized":"2024-01-26T23:46:31.671217+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:28:15.946492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.669162+0000","last_change":"2024-01-26T23:46:31.659612+0000","last_active":"2024-01-26T23:46:31.669162+0000","last_peered":"2024-01-26T23:46:31.669162+0000","last_clean":"2024-01-26T23:46:31.669162+0000","last_became_active":"2024-01-26T23:46:31.659271+0000","last_became_peered":"2024-01-26T23:46:31.659271+0000","last_unstale":"2024-01-26T23:46:31.669162+0000","last_undegraded":"2024-01-26T23:46:31.669162+0000","last_fullsized":"2024-01-26T23:46:31.669162+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:49:03.876856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678127+0000","last_change":"2024-01-26T23:46:31.677644+0000","last_active":"2024-01-26T23:46:31.678127+0000","last_peered":"2024-01-26T23:46:31.678127+0000","last_clean":"2024-01-26T23:46:31.678127+0000","last_became_active":"2024-01-26T23:46:31.677289+0000","last_became_peered":"2024-01-26T23:46:31.677289+0000","last_unstale":"2024-01-26T23:46:31.678127+0000","last_undegraded":"2024-01-26T23:46:31.678127+0000","last_fullsized":"2024-01-26T23:46:31.678127+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:44:47.619765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671107+0000","last_change":"2024-01-26T23:46:31.664774+0000","last_active":"2024-01-26T23:46:31.671107+0000","last_peered":"2024-01-26T23:46:31.671107+0000","last_clean":"2024-01-26T23:46:31.671107+0000","last_became_active":"2024-01-26T23:46:31.664456+0000","last_became_peered":"2024-01-26T23:46:31.664456+0000","last_unstale":"2024-01-26T23:46:31.671107+0000","last_undegraded":"2024-01-26T23:46:31.671107+0000","last_fullsized":"2024-01-26T23:46:31.671107+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:15:01.090798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671925+0000","last_change":"2024-01-26T23:46:31.666995+0000","last_active":"2024-01-26T23:46:31.671925+0000","last_peered":"2024-01-26T23:46:31.671925+0000","last_clean":"2024-01-26T23:46:31.671925+0000","last_became_active":"2024-01-26T23:46:31.666507+0000","last_became_peered":"2024-01-26T23:46:31.666507+0000","last_unstale":"2024-01-26T23:46:31.671925+0000","last_undegraded":"2024-01-26T23:46:31.671925+0000","last_fullsized":"2024-01-26T23:46:31.671925+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:35:52.815366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.672072+0000","last_change":"2024-01-26T23:46:31.665567+0000","last_active":"2024-01-26T23:46:31.672072+0000","last_peered":"2024-01-26T23:46:31.672072+0000","last_clean":"2024-01-26T23:46:31.672072+0000","last_became_active":"2024-01-26T23:46:31.665250+0000","last_became_peered":"2024-01-26T23:46:31.665250+0000","last_unstale":"2024-01-26T23:46:31.672072+0000","last_undegraded":"2024-01-26T23:46:31.672072+0000","last_fullsized":"2024-01-26T23:46:31.672072+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:44.847438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.670423+0000","last_change":"2024-01-26T23:46:31.659801+0000","last_active":"2024-01-26T23:46:31.670423+0000","last_peered":"2024-01-26T23:46:31.670423+0000","last_clean":"2024-01-26T23:46:31.670423+0000","last_became_active":"2024-01-26T23:46:31.659436+0000","last_became_peered":"2024-01-26T23:46:31.659436+0000","last_unstale":"2024-01-26T23:46:31.670423+0000","last_undegraded":"2024-01-26T23:46:31.670423+0000","last_fullsized":"2024-01-26T23:46:31.670423+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:26:22.946213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.677680+0000","last_change":"2024-01-26T23:46:31.671836+0000","last_active":"2024-01-26T23:46:31.677680+0000","last_peered":"2024-01-26T23:46:31.677680+0000","last_clean":"2024-01-26T23:46:31.677680+0000","last_became_active":"2024-01-26T23:46:31.671411+0000","last_became_peered":"2024-01-26T23:46:31.671411+0000","last_unstale":"2024-01-26T23:46:31.677680+0000","last_undegraded":"2024-01-26T23:46:31.677680+0000","last_fullsized":"2024-01-26T23:46:31.677680+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:06:39.892851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.670670+0000","last_change":"2024-01-26T23:46:31.660955+0000","last_active":"2024-01-26T23:46:31.670670+0000","last_peered":"2024-01-26T23:46:31.670670+0000","last_clean":"2024-01-26T23:46:31.670670+0000","last_became_active":"2024-01-26T23:46:31.660600+0000","last_became_peered":"2024-01-26T23:46:31.660600+0000","last_unstale":"2024-01-26T23:46:31.670670+0000","last_undegraded":"2024-01-26T23:46:31.670670+0000","last_fullsized":"2024-01-26T23:46:31.670670+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:20:40.199495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674437+0000","last_change":"2024-01-26T23:46:31.666637+0000","last_active":"2024-01-26T23:46:31.674437+0000","last_peered":"2024-01-26T23:46:31.674437+0000","last_clean":"2024-01-26T23:46:31.674437+0000","last_became_active":"2024-01-26T23:46:31.666340+0000","last_became_peered":"2024-01-26T23:46:31.666340+0000","last_unstale":"2024-01-26T23:46:31.674437+0000","last_undegraded":"2024-01-26T23:46:31.674437+0000","last_fullsized":"2024-01-26T23:46:31.674437+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:35.826457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678116+0000","last_change":"2024-01-26T23:46:31.677579+0000","last_active":"2024-01-26T23:46:31.678116+0000","last_peered":"2024-01-26T23:46:31.678116+0000","last_clean":"2024-01-26T23:46:31.678116+0000","last_became_active":"2024-01-26T23:46:31.677261+0000","last_became_peered":"2024-01-26T23:46:31.677261+0000","last_unstale":"2024-01-26T23:46:31.678116+0000","last_undegraded":"2024-01-26T23:46:31.678116+0000","last_fullsized":"2024-01-26T23:46:31.678116+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:19:41.093581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674555+0000","last_change":"2024-01-26T23:46:31.672211+0000","last_active":"2024-01-26T23:46:31.674555+0000","last_peered":"2024-01-26T23:46:31.674555+0000","last_clean":"2024-01-26T23:46:31.674555+0000","last_became_active":"2024-01-26T23:46:31.671945+0000","last_became_peered":"2024-01-26T23:46:31.671945+0000","last_unstale":"2024-01-26T23:46:31.674555+0000","last_undegraded":"2024-01-26T23:46:31.674555+0000","last_fullsized":"2024-01-26T23:46:31.674555+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:15:01.090798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674185+0000","last_change":"2024-01-26T23:46:31.669513+0000","last_active":"2024-01-26T23:46:31.674185+0000","last_peered":"2024-01-26T23:46:31.674185+0000","last_clean":"2024-01-26T23:46:31.674185+0000","last_became_active":"2024-01-26T23:46:31.669214+0000","last_became_peered":"2024-01-26T23:46:31.669214+0000","last_unstale":"2024-01-26T23:46:31.674185+0000","last_undegraded":"2024-01-26T23:46:31.674185+0000","last_fullsized":"2024-01-26T23:46:31.674185+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:15:01.090798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.663696+0000","last_change":"2024-01-26T23:46:31.661379+0000","last_active":"2024-01-26T23:46:31.663696+0000","last_peered":"2024-01-26T23:46:31.663696+0000","last_clean":"2024-01-26T23:46:31.663696+0000","last_became_active":"2024-01-26T23:46:31.661139+0000","last_became_peered":"2024-01-26T23:46:31.661139+0000","last_unstale":"2024-01-26T23:46:31.663696+0000","last_undegraded":"2024-01-26T23:46:31.663696+0000","last_fullsized":"2024-01-26T23:46:31.663696+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:14.351439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.677651+0000","last_change":"2024-01-26T23:46:31.672080+0000","last_active":"2024-01-26T23:46:31.677651+0000","last_peered":"2024-01-26T23:46:31.677651+0000","last_clean":"2024-01-26T23:46:31.677651+0000","last_became_active":"2024-01-26T23:46:31.671662+0000","last_became_peered":"2024-01-26T23:46:31.671662+0000","last_unstale":"2024-01-26T23:46:31.677651+0000","last_undegraded":"2024-01-26T23:46:31.677651+0000","last_fullsized":"2024-01-26T23:46:31.677651+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:21:18.772481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671143+0000","last_change":"2024-01-26T23:46:31.668130+0000","last_active":"2024-01-26T23:46:31.671143+0000","last_peered":"2024-01-26T23:46:31.671143+0000","last_clean":"2024-01-26T23:46:31.671143+0000","last_became_active":"2024-01-26T23:46:31.667823+0000","last_became_peered":"2024-01-26T23:46:31.667823+0000","last_unstale":"2024-01-26T23:46:31.671143+0000","last_undegraded":"2024-01-26T23:46:31.671143+0000","last_fullsized":"2024-01-26T23:46:31.671143+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:33:18.471363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671230+0000","last_change":"2024-01-26T23:46:31.667876+0000","last_active":"2024-01-26T23:46:31.671230+0000","last_peered":"2024-01-26T23:46:31.671230+0000","last_clean":"2024-01-26T23:46:31.671230+0000","last_became_active":"2024-01-26T23:46:31.667555+0000","last_became_peered":"2024-01-26T23:46:31.667555+0000","last_unstale":"2024-01-26T23:46:31.671230+0000","last_undegraded":"2024-01-26T23:46:31.671230+0000","last_fullsized":"2024-01-26T23:46:31.671230+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:14.351439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674525+0000","last_change":"2024-01-26T23:46:31.671962+0000","last_active":"2024-01-26T23:46:31.674525+0000","last_peered":"2024-01-26T23:46:31.674525+0000","last_clean":"2024-01-26T23:46:31.674525+0000","last_became_active":"2024-01-26T23:46:31.671717+0000","last_became_peered":"2024-01-26T23:46:31.671717+0000","last_unstale":"2024-01-26T23:46:31.674525+0000","last_undegraded":"2024-01-26T23:46:31.674525+0000","last_fullsized":"2024-01-26T23:46:31.674525+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:19:13.010766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674578+0000","last_change":"2024-01-26T23:46:31.667039+0000","last_active":"2024-01-26T23:46:31.674578+0000","last_peered":"2024-01-26T23:46:31.674578+0000","last_clean":"2024-01-26T23:46:31.674578+0000","last_became_active":"2024-01-26T23:46:31.666790+0000","last_became_peered":"2024-01-26T23:46:31.666790+0000","last_unstale":"2024-01-26T23:46:31.674578+0000","last_undegraded":"2024-01-26T23:46:31.674578+0000","last_fullsized":"2024-01-26T23:46:31.674578+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:25.220136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.663634+0000","last_change":"2024-01-26T23:46:31.657261+0000","last_active":"2024-01-26T23:46:31.663634+0000","last_peered":"2024-01-26T23:46:31.663634+0000","last_clean":"2024-01-26T23:46:31.663634+0000","last_became_active":"2024-01-26T23:46:31.657084+0000","last_became_peered":"2024-01-26T23:46:31.657084+0000","last_unstale":"2024-01-26T23:46:31.663634+0000","last_undegraded":"2024-01-26T23:46:31.663634+0000","last_fullsized":"2024-01-26T23:46:31.663634+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:46:27.325038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671844+0000","last_change":"2024-01-26T23:46:31.666793+0000","last_active":"2024-01-26T23:46:31.671844+0000","last_peered":"2024-01-26T23:46:31.671844+0000","last_clean":"2024-01-26T23:46:31.671844+0000","last_became_active":"2024-01-26T23:46:31.666112+0000","last_became_peered":"2024-01-26T23:46:31.666112+0000","last_unstale":"2024-01-26T23:46:31.671844+0000","last_undegraded":"2024-01-26T23:46:31.671844+0000","last_fullsized":"2024-01-26T23:46:31.671844+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:38.074736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675273+0000","last_change":"2024-01-26T23:46:31.668562+0000","last_active":"2024-01-26T23:46:31.675273+0000","last_peered":"2024-01-26T23:46:31.675273+0000","last_clean":"2024-01-26T23:46:31.675273+0000","last_became_active":"2024-01-26T23:46:31.668253+0000","last_became_peered":"2024-01-26T23:46:31.668253+0000","last_unstale":"2024-01-26T23:46:31.675273+0000","last_undegraded":"2024-01-26T23:46:31.675273+0000","last_fullsized":"2024-01-26T23:46:31.675273+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:43:18.393081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"111'4","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.774608+0000","last_change":"2024-01-26T23:46:30.779482+0000","last_active":"2024-01-26T23:46:31.774608+0000","last_peered":"2024-01-26T23:46:31.774608+0000","last_clean":"2024-01-26T23:46:31.774608+0000","last_became_active":"2024-01-26T23:46:30.779245+0000","last_became_peered":"2024-01-26T23:46:30.779245+0000","last_unstale":"2024-01-26T23:46:31.774608+0000","last_undegraded":"2024-01-26T23:46:31.774608+0000","last_fullsized":"2024-01-26T23:46:31.774608+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:37.479741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675662+0000","last_change":"2024-01-26T23:46:31.672354+0000","last_active":"2024-01-26T23:46:31.675662+0000","last_peered":"2024-01-26T23:46:31.675662+0000","last_clean":"2024-01-26T23:46:31.675662+0000","last_became_active":"2024-01-26T23:46:31.672029+0000","last_became_peered":"2024-01-26T23:46:31.672029+0000","last_unstale":"2024-01-26T23:46:31.675662+0000","last_undegraded":"2024-01-26T23:46:31.675662+0000","last_fullsized":"2024-01-26T23:46:31.675662+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:16:43.142449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"111'1","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.772979+0000","last_change":"2024-01-26T23:46:30.769470+0000","last_active":"2024-01-26T23:46:31.772979+0000","last_peered":"2024-01-26T23:46:31.772979+0000","last_clean":"2024-01-26T23:46:31.772979+0000","last_became_active":"2024-01-26T23:46:30.769231+0000","last_became_peered":"2024-01-26T23:46:30.769231+0000","last_unstale":"2024-01-26T23:46:31.772979+0000","last_undegraded":"2024-01-26T23:46:31.772979+0000","last_fullsized":"2024-01-26T23:46:31.772979+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:16:42.463283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.672601+0000","last_change":"2024-01-26T23:46:31.669777+0000","last_active":"2024-01-26T23:46:31.672601+0000","last_peered":"2024-01-26T23:46:31.672601+0000","last_clean":"2024-01-26T23:46:31.672601+0000","last_became_active":"2024-01-26T23:46:31.669538+0000","last_became_peered":"2024-01-26T23:46:31.669538+0000","last_unstale":"2024-01-26T23:46:31.672601+0000","last_undegraded":"2024-01-26T23:46:31.672601+0000","last_fullsized":"2024-01-26T23:46:31.672601+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:17:40.471277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"111'2","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.773859+0000","last_change":"2024-01-26T23:46:30.776273+0000","last_active":"2024-01-26T23:46:31.773859+0000","last_peered":"2024-01-26T23:46:31.773859+0000","last_clean":"2024-01-26T23:46:31.773859+0000","last_became_active":"2024-01-26T23:46:30.775989+0000","last_became_peered":"2024-01-26T23:46:30.775989+0000","last_unstale":"2024-01-26T23:46:31.773859+0000","last_undegraded":"2024-01-26T23:46:31.773859+0000","last_fullsized":"2024-01-26T23:46:31.773859+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:00:45.452291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.670354+0000","last_change":"2024-01-26T23:46:31.662791+0000","last_active":"2024-01-26T23:46:31.670354+0000","last_peered":"2024-01-26T23:46:31.670354+0000","last_clean":"2024-01-26T23:46:31.670354+0000","last_became_active":"2024-01-26T23:46:31.662506+0000","last_became_peered":"2024-01-26T23:46:31.662506+0000","last_unstale":"2024-01-26T23:46:31.670354+0000","last_undegraded":"2024-01-26T23:46:31.670354+0000","last_fullsized":"2024-01-26T23:46:31.670354+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:22.391377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"111'2","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.772905+0000","last_change":"2024-01-26T23:46:30.776989+0000","last_active":"2024-01-26T23:46:31.772905+0000","last_peered":"2024-01-26T23:46:31.772905+0000","last_clean":"2024-01-26T23:46:31.772905+0000","last_became_active":"2024-01-26T23:46:30.776785+0000","last_became_peered":"2024-01-26T23:46:30.776785+0000","last_unstale":"2024-01-26T23:46:31.772905+0000","last_undegraded":"2024-01-26T23:46:31.772905+0000","last_fullsized":"2024-01-26T23:46:31.772905+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:07.859918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.670702+0000","last_change":"2024-01-26T23:46:31.661125+0000","last_active":"2024-01-26T23:46:31.670702+0000","last_peered":"2024-01-26T23:46:31.670702+0000","last_clean":"2024-01-26T23:46:31.670702+0000","last_became_active":"2024-01-26T23:46:31.660752+0000","last_became_peered":"2024-01-26T23:46:31.660752+0000","last_unstale":"2024-01-26T23:46:31.670702+0000","last_undegraded":"2024-01-26T23:46:31.670702+0000","last_fullsized":"2024-01-26T23:46:31.670702+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:46:27.325038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"111'1","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.771129+0000","last_change":"2024-01-26T23:46:30.777476+0000","last_active":"2024-01-26T23:46:31.771129+0000","last_peered":"2024-01-26T23:46:31.771129+0000","last_clean":"2024-01-26T23:46:31.771129+0000","last_became_active":"2024-01-26T23:46:30.777188+0000","last_became_peered":"2024-01-26T23:46:30.777188+0000","last_unstale":"2024-01-26T23:46:31.771129+0000","last_undegraded":"2024-01-26T23:46:31.771129+0000","last_fullsized":"2024-01-26T23:46:31.771129+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:41:51.840628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.666024+0000","last_change":"2024-01-26T23:46:31.662764+0000","last_active":"2024-01-26T23:46:31.666024+0000","last_peered":"2024-01-26T23:46:31.666024+0000","last_clean":"2024-01-26T23:46:31.666024+0000","last_became_active":"2024-01-26T23:46:31.662519+0000","last_became_peered":"2024-01-26T23:46:31.662519+0000","last_unstale":"2024-01-26T23:46:31.666024+0000","last_undegraded":"2024-01-26T23:46:31.666024+0000","last_fullsized":"2024-01-26T23:46:31.666024+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:14.351439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"111'1","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.777701+0000","last_change":"2024-01-26T23:46:30.780739+0000","last_active":"2024-01-26T23:46:31.777701+0000","last_peered":"2024-01-26T23:46:31.777701+0000","last_clean":"2024-01-26T23:46:31.777701+0000","last_became_active":"2024-01-26T23:46:30.780309+0000","last_became_peered":"2024-01-26T23:46:30.780309+0000","last_unstale":"2024-01-26T23:46:31.777701+0000","last_undegraded":"2024-01-26T23:46:31.777701+0000","last_fullsized":"2024-01-26T23:46:31.777701+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:28.581902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678716+0000","last_change":"2024-01-26T23:46:31.677394+0000","last_active":"2024-01-26T23:46:31.678716+0000","last_peered":"2024-01-26T23:46:31.678716+0000","last_clean":"2024-01-26T23:46:31.678716+0000","last_became_active":"2024-01-26T23:46:31.677090+0000","last_became_peered":"2024-01-26T23:46:31.677090+0000","last_unstale":"2024-01-26T23:46:31.678716+0000","last_undegraded":"2024-01-26T23:46:31.678716+0000","last_fullsized":"2024-01-26T23:46:31.678716+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:28:36.539497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.653719+0000","last_change":"2024-01-26T23:46:30.771550+0000","last_active":"2024-01-26T23:46:31.653719+0000","last_peered":"2024-01-26T23:46:31.653719+0000","last_clean":"2024-01-26T23:46:31.653719+0000","last_became_active":"2024-01-26T23:46:30.771291+0000","last_became_peered":"2024-01-26T23:46:30.771291+0000","last_unstale":"2024-01-26T23:46:31.653719+0000","last_undegraded":"2024-01-26T23:46:31.653719+0000","last_fullsized":"2024-01-26T23:46:31.653719+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:30:09.071510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.673494+0000","last_change":"2024-01-26T23:46:31.663969+0000","last_active":"2024-01-26T23:46:31.673494+0000","last_peered":"2024-01-26T23:46:31.673494+0000","last_clean":"2024-01-26T23:46:31.673494+0000","last_became_active":"2024-01-26T23:46:31.663602+0000","last_became_peered":"2024-01-26T23:46:31.663602+0000","last_unstale":"2024-01-26T23:46:31.673494+0000","last_undegraded":"2024-01-26T23:46:31.673494+0000","last_fullsized":"2024-01-26T23:46:31.673494+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:23:09.740800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.654348+0000","last_change":"2024-01-26T23:46:30.782673+0000","last_active":"2024-01-26T23:46:31.654348+0000","last_peered":"2024-01-26T23:46:31.654348+0000","last_clean":"2024-01-26T23:46:31.654348+0000","last_became_active":"2024-01-26T23:46:30.782464+0000","last_became_peered":"2024-01-26T23:46:30.782464+0000","last_unstale":"2024-01-26T23:46:31.654348+0000","last_undegraded":"2024-01-26T23:46:31.654348+0000","last_fullsized":"2024-01-26T23:46:31.654348+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:13.350096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674733+0000","last_change":"2024-01-26T23:46:31.669467+0000","last_active":"2024-01-26T23:46:31.674733+0000","last_peered":"2024-01-26T23:46:31.674733+0000","last_clean":"2024-01-26T23:46:31.674733+0000","last_became_active":"2024-01-26T23:46:31.669212+0000","last_became_peered":"2024-01-26T23:46:31.669212+0000","last_unstale":"2024-01-26T23:46:31.674733+0000","last_undegraded":"2024-01-26T23:46:31.674733+0000","last_fullsized":"2024-01-26T23:46:31.674733+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:24:19.005800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.652762+0000","last_change":"2024-01-26T23:46:30.773338+0000","last_active":"2024-01-26T23:46:31.652762+0000","last_peered":"2024-01-26T23:46:31.652762+0000","last_clean":"2024-01-26T23:46:31.652762+0000","last_became_active":"2024-01-26T23:46:30.773156+0000","last_became_peered":"2024-01-26T23:46:30.773156+0000","last_unstale":"2024-01-26T23:46:31.652762+0000","last_undegraded":"2024-01-26T23:46:31.652762+0000","last_fullsized":"2024-01-26T23:46:31.652762+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:46.577047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671908+0000","last_change":"2024-01-26T23:46:31.667991+0000","last_active":"2024-01-26T23:46:31.671908+0000","last_peered":"2024-01-26T23:46:31.671908+0000","last_clean":"2024-01-26T23:46:31.671908+0000","last_became_active":"2024-01-26T23:46:31.667740+0000","last_became_peered":"2024-01-26T23:46:31.667740+0000","last_unstale":"2024-01-26T23:46:31.671908+0000","last_undegraded":"2024-01-26T23:46:31.671908+0000","last_fullsized":"2024-01-26T23:46:31.671908+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:45.348777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.653142+0000","last_change":"2024-01-26T23:46:30.776658+0000","last_active":"2024-01-26T23:46:31.653142+0000","last_peered":"2024-01-26T23:46:31.653142+0000","last_clean":"2024-01-26T23:46:31.653142+0000","last_became_active":"2024-01-26T23:46:30.776294+0000","last_became_peered":"2024-01-26T23:46:30.776294+0000","last_unstale":"2024-01-26T23:46:31.653142+0000","last_undegraded":"2024-01-26T23:46:31.653142+0000","last_fullsized":"2024-01-26T23:46:31.653142+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:00:45.452291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.670487+0000","last_change":"2024-01-26T23:46:31.665924+0000","last_active":"2024-01-26T23:46:31.670487+0000","last_peered":"2024-01-26T23:46:31.670487+0000","last_clean":"2024-01-26T23:46:31.670487+0000","last_became_active":"2024-01-26T23:46:31.665551+0000","last_became_peered":"2024-01-26T23:46:31.665551+0000","last_unstale":"2024-01-26T23:46:31.670487+0000","last_undegraded":"2024-01-26T23:46:31.670487+0000","last_fullsized":"2024-01-26T23:46:31.670487+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:38:57.979602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"111'3","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.775090+0000","last_change":"2024-01-26T23:46:30.784086+0000","last_active":"2024-01-26T23:46:31.775090+0000","last_peered":"2024-01-26T23:46:31.775090+0000","last_clean":"2024-01-26T23:46:31.775090+0000","last_became_active":"2024-01-26T23:46:30.783840+0000","last_became_peered":"2024-01-26T23:46:30.783840+0000","last_unstale":"2024-01-26T23:46:31.775090+0000","last_undegraded":"2024-01-26T23:46:31.775090+0000","last_fullsized":"2024-01-26T23:46:31.775090+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:00:45.452291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.668429+0000","last_change":"2024-01-26T23:46:31.655793+0000","last_active":"2024-01-26T23:46:31.668429+0000","last_peered":"2024-01-26T23:46:31.668429+0000","last_clean":"2024-01-26T23:46:31.668429+0000","last_became_active":"2024-01-26T23:46:31.655460+0000","last_became_peered":"2024-01-26T23:46:31.655460+0000","last_unstale":"2024-01-26T23:46:31.668429+0000","last_undegraded":"2024-01-26T23:46:31.668429+0000","last_fullsized":"2024-01-26T23:46:31.668429+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:22:27.095843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"111'6","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.776077+0000","last_change":"2024-01-26T23:46:30.772469+0000","last_active":"2024-01-26T23:46:31.776077+0000","last_peered":"2024-01-26T23:46:31.776077+0000","last_clean":"2024-01-26T23:46:31.776077+0000","last_became_active":"2024-01-26T23:46:30.772188+0000","last_became_peered":"2024-01-26T23:46:30.772188+0000","last_unstale":"2024-01-26T23:46:31.776077+0000","last_undegraded":"2024-01-26T23:46:31.776077+0000","last_fullsized":"2024-01-26T23:46:31.776077+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:04:26.732373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.669153+0000","last_change":"2024-01-26T23:46:31.659621+0000","last_active":"2024-01-26T23:46:31.669153+0000","last_peered":"2024-01-26T23:46:31.669153+0000","last_clean":"2024-01-26T23:46:31.669153+0000","last_became_active":"2024-01-26T23:46:31.658380+0000","last_became_peered":"2024-01-26T23:46:31.658380+0000","last_unstale":"2024-01-26T23:46:31.669153+0000","last_undegraded":"2024-01-26T23:46:31.669153+0000","last_fullsized":"2024-01-26T23:46:31.669153+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:48.805791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"111'2","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.775041+0000","last_change":"2024-01-26T23:46:30.779281+0000","last_active":"2024-01-26T23:46:31.775041+0000","last_peered":"2024-01-26T23:46:31.775041+0000","last_clean":"2024-01-26T23:46:31.775041+0000","last_became_active":"2024-01-26T23:46:30.779013+0000","last_became_peered":"2024-01-26T23:46:30.779013+0000","last_unstale":"2024-01-26T23:46:31.775041+0000","last_undegraded":"2024-01-26T23:46:31.775041+0000","last_fullsized":"2024-01-26T23:46:31.775041+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:22:26.094500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671594+0000","last_change":"2024-01-26T23:46:31.663372+0000","last_active":"2024-01-26T23:46:31.671594+0000","last_peered":"2024-01-26T23:46:31.671594+0000","last_clean":"2024-01-26T23:46:31.671594+0000","last_became_active":"2024-01-26T23:46:31.663122+0000","last_became_peered":"2024-01-26T23:46:31.663122+0000","last_unstale":"2024-01-26T23:46:31.671594+0000","last_undegraded":"2024-01-26T23:46:31.671594+0000","last_fullsized":"2024-01-26T23:46:31.671594+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:33:37.940458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.657510+0000","last_change":"2024-01-26T23:46:30.783543+0000","last_active":"2024-01-26T23:46:31.657510+0000","last_peered":"2024-01-26T23:46:31.657510+0000","last_clean":"2024-01-26T23:46:31.657510+0000","last_became_active":"2024-01-26T23:46:30.774051+0000","last_became_peered":"2024-01-26T23:46:30.774051+0000","last_unstale":"2024-01-26T23:46:31.657510+0000","last_undegraded":"2024-01-26T23:46:31.657510+0000","last_fullsized":"2024-01-26T23:46:31.657510+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:18:11.911206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678158+0000","last_change":"2024-01-26T23:46:31.677498+0000","last_active":"2024-01-26T23:46:31.678158+0000","last_peered":"2024-01-26T23:46:31.678158+0000","last_clean":"2024-01-26T23:46:31.678158+0000","last_became_active":"2024-01-26T23:46:31.677210+0000","last_became_peered":"2024-01-26T23:46:31.677210+0000","last_unstale":"2024-01-26T23:46:31.678158+0000","last_undegraded":"2024-01-26T23:46:31.678158+0000","last_fullsized":"2024-01-26T23:46:31.678158+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:28.759656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.648299+0000","last_change":"2024-01-26T23:46:30.779455+0000","last_active":"2024-01-26T23:46:31.648299+0000","last_peered":"2024-01-26T23:46:31.648299+0000","last_clean":"2024-01-26T23:46:31.648299+0000","last_became_active":"2024-01-26T23:46:30.779141+0000","last_became_peered":"2024-01-26T23:46:30.779141+0000","last_unstale":"2024-01-26T23:46:31.648299+0000","last_undegraded":"2024-01-26T23:46:31.648299+0000","last_fullsized":"2024-01-26T23:46:31.648299+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:34:01.252528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.674140+0000","last_change":"2024-01-26T23:46:31.670025+0000","last_active":"2024-01-26T23:46:31.674140+0000","last_peered":"2024-01-26T23:46:31.674140+0000","last_clean":"2024-01-26T23:46:31.674140+0000","last_became_active":"2024-01-26T23:46:31.669744+0000","last_became_peered":"2024-01-26T23:46:31.669744+0000","last_unstale":"2024-01-26T23:46:31.674140+0000","last_undegraded":"2024-01-26T23:46:31.674140+0000","last_fullsized":"2024-01-26T23:46:31.674140+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:27:45.307754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"111'8","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.778380+0000","last_change":"2024-01-26T23:46:30.780133+0000","last_active":"2024-01-26T23:46:31.778380+0000","last_peered":"2024-01-26T23:46:31.778380+0000","last_clean":"2024-01-26T23:46:31.778380+0000","last_became_active":"2024-01-26T23:46:30.779792+0000","last_became_peered":"2024-01-26T23:46:30.779792+0000","last_unstale":"2024-01-26T23:46:31.778380+0000","last_undegraded":"2024-01-26T23:46:31.778380+0000","last_fullsized":"2024-01-26T23:46:31.778380+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:15:25.108006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.672965+0000","last_change":"2024-01-26T23:46:31.667032+0000","last_active":"2024-01-26T23:46:31.672965+0000","last_peered":"2024-01-26T23:46:31.672965+0000","last_clean":"2024-01-26T23:46:31.672965+0000","last_became_active":"2024-01-26T23:46:31.666719+0000","last_became_peered":"2024-01-26T23:46:31.666719+0000","last_unstale":"2024-01-26T23:46:31.672965+0000","last_undegraded":"2024-01-26T23:46:31.672965+0000","last_fullsized":"2024-01-26T23:46:31.672965+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:47:07.666792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"111'1","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.774378+0000","last_change":"2024-01-26T23:46:30.768481+0000","last_active":"2024-01-26T23:46:31.774378+0000","last_peered":"2024-01-26T23:46:31.774378+0000","last_clean":"2024-01-26T23:46:31.774378+0000","last_became_active":"2024-01-26T23:46:30.768174+0000","last_became_peered":"2024-01-26T23:46:30.768174+0000","last_unstale":"2024-01-26T23:46:31.774378+0000","last_undegraded":"2024-01-26T23:46:31.774378+0000","last_fullsized":"2024-01-26T23:46:31.774378+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:34:46.577047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.663243+0000","last_change":"2024-01-26T23:46:31.660241+0000","last_active":"2024-01-26T23:46:31.663243+0000","last_peered":"2024-01-26T23:46:31.663243+0000","last_clean":"2024-01-26T23:46:31.663243+0000","last_became_active":"2024-01-26T23:46:31.659972+0000","last_became_peered":"2024-01-26T23:46:31.659972+0000","last_unstale":"2024-01-26T23:46:31.663243+0000","last_undegraded":"2024-01-26T23:46:31.663243+0000","last_fullsized":"2024-01-26T23:46:31.663243+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:23:09.740800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"111'2","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.774206+0000","last_change":"2024-01-26T23:46:30.768055+0000","last_active":"2024-01-26T23:46:31.774206+0000","last_peered":"2024-01-26T23:46:31.774206+0000","last_clean":"2024-01-26T23:46:31.774206+0000","last_became_active":"2024-01-26T23:46:30.767751+0000","last_became_peered":"2024-01-26T23:46:30.767751+0000","last_unstale":"2024-01-26T23:46:31.774206+0000","last_undegraded":"2024-01-26T23:46:31.774206+0000","last_fullsized":"2024-01-26T23:46:31.774206+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:30:09.071510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.677633+0000","last_change":"2024-01-26T23:46:31.671797+0000","last_active":"2024-01-26T23:46:31.677633+0000","last_peered":"2024-01-26T23:46:31.677633+0000","last_clean":"2024-01-26T23:46:31.677633+0000","last_became_active":"2024-01-26T23:46:31.671535+0000","last_became_peered":"2024-01-26T23:46:31.671535+0000","last_unstale":"2024-01-26T23:46:31.677633+0000","last_undegraded":"2024-01-26T23:46:31.677633+0000","last_fullsized":"2024-01-26T23:46:31.677633+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:21:09.440118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"111'1","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.772424+0000","last_change":"2024-01-26T23:46:30.775593+0000","last_active":"2024-01-26T23:46:31.772424+0000","last_peered":"2024-01-26T23:46:31.772424+0000","last_clean":"2024-01-26T23:46:31.772424+0000","last_became_active":"2024-01-26T23:46:30.775280+0000","last_became_peered":"2024-01-26T23:46:30.775280+0000","last_unstale":"2024-01-26T23:46:31.772424+0000","last_undegraded":"2024-01-26T23:46:31.772424+0000","last_fullsized":"2024-01-26T23:46:31.772424+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:30:09.071510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671822+0000","last_change":"2024-01-26T23:46:31.665817+0000","last_active":"2024-01-26T23:46:31.671822+0000","last_peered":"2024-01-26T23:46:31.671822+0000","last_clean":"2024-01-26T23:46:31.671822+0000","last_became_active":"2024-01-26T23:46:31.665513+0000","last_became_peered":"2024-01-26T23:46:31.665513+0000","last_unstale":"2024-01-26T23:46:31.671822+0000","last_undegraded":"2024-01-26T23:46:31.671822+0000","last_fullsized":"2024-01-26T23:46:31.671822+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:56:41.826944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"111'2","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.774170+0000","last_change":"2024-01-26T23:46:30.768575+0000","last_active":"2024-01-26T23:46:31.774170+0000","last_peered":"2024-01-26T23:46:31.774170+0000","last_clean":"2024-01-26T23:46:31.774170+0000","last_became_active":"2024-01-26T23:46:30.768327+0000","last_became_peered":"2024-01-26T23:46:30.768327+0000","last_unstale":"2024-01-26T23:46:31.774170+0000","last_undegraded":"2024-01-26T23:46:31.774170+0000","last_fullsized":"2024-01-26T23:46:31.774170+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:41:51.840628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.666067+0000","last_change":"2024-01-26T23:46:31.663557+0000","last_active":"2024-01-26T23:46:31.666067+0000","last_peered":"2024-01-26T23:46:31.666067+0000","last_clean":"2024-01-26T23:46:31.666067+0000","last_became_active":"2024-01-26T23:46:31.663319+0000","last_became_peered":"2024-01-26T23:46:31.663319+0000","last_unstale":"2024-01-26T23:46:31.666067+0000","last_undegraded":"2024-01-26T23:46:31.666067+0000","last_fullsized":"2024-01-26T23:46:31.666067+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:48.805791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.658837+0000","last_change":"2024-01-26T23:46:30.777564+0000","last_active":"2024-01-26T23:46:31.658837+0000","last_peered":"2024-01-26T23:46:31.658837+0000","last_clean":"2024-01-26T23:46:31.658837+0000","last_became_active":"2024-01-26T23:46:30.777242+0000","last_became_peered":"2024-01-26T23:46:30.777242+0000","last_unstale":"2024-01-26T23:46:31.658837+0000","last_undegraded":"2024-01-26T23:46:31.658837+0000","last_fullsized":"2024-01-26T23:46:31.658837+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:08:16.334908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.663666+0000","last_change":"2024-01-26T23:46:31.657382+0000","last_active":"2024-01-26T23:46:31.663666+0000","last_peered":"2024-01-26T23:46:31.663666+0000","last_clean":"2024-01-26T23:46:31.663666+0000","last_became_active":"2024-01-26T23:46:31.657142+0000","last_became_peered":"2024-01-26T23:46:31.657142+0000","last_unstale":"2024-01-26T23:46:31.663666+0000","last_undegraded":"2024-01-26T23:46:31.663666+0000","last_fullsized":"2024-01-26T23:46:31.663666+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:18:30.549273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.652138+0000","last_change":"2024-01-26T23:46:30.777901+0000","last_active":"2024-01-26T23:46:31.652138+0000","last_peered":"2024-01-26T23:46:31.652138+0000","last_clean":"2024-01-26T23:46:31.652138+0000","last_became_active":"2024-01-26T23:46:30.777649+0000","last_became_peered":"2024-01-26T23:46:30.777649+0000","last_unstale":"2024-01-26T23:46:31.652138+0000","last_undegraded":"2024-01-26T23:46:31.652138+0000","last_fullsized":"2024-01-26T23:46:31.652138+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:46:39.085189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675654+0000","last_change":"2024-01-26T23:46:31.672269+0000","last_active":"2024-01-26T23:46:31.675654+0000","last_peered":"2024-01-26T23:46:31.675654+0000","last_clean":"2024-01-26T23:46:31.675654+0000","last_became_active":"2024-01-26T23:46:31.671924+0000","last_became_peered":"2024-01-26T23:46:31.671924+0000","last_unstale":"2024-01-26T23:46:31.675654+0000","last_undegraded":"2024-01-26T23:46:31.675654+0000","last_fullsized":"2024-01-26T23:46:31.675654+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:36:26.521199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.660165+0000","last_change":"2024-01-26T23:46:30.771183+0000","last_active":"2024-01-26T23:46:31.660165+0000","last_peered":"2024-01-26T23:46:31.660165+0000","last_clean":"2024-01-26T23:46:31.660165+0000","last_became_active":"2024-01-26T23:46:30.770867+0000","last_became_peered":"2024-01-26T23:46:30.770867+0000","last_unstale":"2024-01-26T23:46:31.660165+0000","last_undegraded":"2024-01-26T23:46:31.660165+0000","last_fullsized":"2024-01-26T23:46:31.660165+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:25:06.792314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.665959+0000","last_change":"2024-01-26T23:46:31.661160+0000","last_active":"2024-01-26T23:46:31.665959+0000","last_peered":"2024-01-26T23:46:31.665959+0000","last_clean":"2024-01-26T23:46:31.665959+0000","last_became_active":"2024-01-26T23:46:31.660925+0000","last_became_peered":"2024-01-26T23:46:31.660925+0000","last_unstale":"2024-01-26T23:46:31.665959+0000","last_undegraded":"2024-01-26T23:46:31.665959+0000","last_fullsized":"2024-01-26T23:46:31.665959+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:48.805791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"111'2","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.773899+0000","last_change":"2024-01-26T23:46:30.768210+0000","last_active":"2024-01-26T23:46:31.773899+0000","last_peered":"2024-01-26T23:46:31.773899+0000","last_clean":"2024-01-26T23:46:31.773899+0000","last_became_active":"2024-01-26T23:46:30.767953+0000","last_became_peered":"2024-01-26T23:46:30.767953+0000","last_unstale":"2024-01-26T23:46:31.773899+0000","last_undegraded":"2024-01-26T23:46:31.773899+0000","last_fullsized":"2024-01-26T23:46:31.773899+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:23:08.739457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675423+0000","last_change":"2024-01-26T23:46:31.668761+0000","last_active":"2024-01-26T23:46:31.675423+0000","last_peered":"2024-01-26T23:46:31.675423+0000","last_clean":"2024-01-26T23:46:31.675423+0000","last_became_active":"2024-01-26T23:46:31.668456+0000","last_became_peered":"2024-01-26T23:46:31.668456+0000","last_unstale":"2024-01-26T23:46:31.675423+0000","last_undegraded":"2024-01-26T23:46:31.675423+0000","last_fullsized":"2024-01-26T23:46:31.675423+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:51:54.286344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.659239+0000","last_change":"2024-01-26T23:46:30.771548+0000","last_active":"2024-01-26T23:46:31.659239+0000","last_peered":"2024-01-26T23:46:31.659239+0000","last_clean":"2024-01-26T23:46:31.659239+0000","last_became_active":"2024-01-26T23:46:30.771300+0000","last_became_peered":"2024-01-26T23:46:30.771300+0000","last_unstale":"2024-01-26T23:46:31.659239+0000","last_undegraded":"2024-01-26T23:46:31.659239+0000","last_fullsized":"2024-01-26T23:46:31.659239+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:07.859918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675695+0000","last_change":"2024-01-26T23:46:31.672735+0000","last_active":"2024-01-26T23:46:31.675695+0000","last_peered":"2024-01-26T23:46:31.675695+0000","last_clean":"2024-01-26T23:46:31.675695+0000","last_became_active":"2024-01-26T23:46:31.672465+0000","last_became_peered":"2024-01-26T23:46:31.672465+0000","last_unstale":"2024-01-26T23:46:31.675695+0000","last_undegraded":"2024-01-26T23:46:31.675695+0000","last_fullsized":"2024-01-26T23:46:31.675695+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:57:26.854523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.659544+0000","last_change":"2024-01-26T23:46:30.784052+0000","last_active":"2024-01-26T23:46:31.659544+0000","last_peered":"2024-01-26T23:46:31.659544+0000","last_clean":"2024-01-26T23:46:31.659544+0000","last_became_active":"2024-01-26T23:46:30.783827+0000","last_became_peered":"2024-01-26T23:46:30.783827+0000","last_unstale":"2024-01-26T23:46:31.659544+0000","last_undegraded":"2024-01-26T23:46:31.659544+0000","last_fullsized":"2024-01-26T23:46:31.659544+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:04:08.622450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.668796+0000","last_change":"2024-01-26T23:46:31.667235+0000","last_active":"2024-01-26T23:46:31.668796+0000","last_peered":"2024-01-26T23:46:31.668796+0000","last_clean":"2024-01-26T23:46:31.668796+0000","last_became_active":"2024-01-26T23:46:31.666976+0000","last_became_peered":"2024-01-26T23:46:31.666976+0000","last_unstale":"2024-01-26T23:46:31.668796+0000","last_undegraded":"2024-01-26T23:46:31.668796+0000","last_fullsized":"2024-01-26T23:46:31.668796+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:23:09.740800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.655190+0000","last_change":"2024-01-26T23:46:30.771368+0000","last_active":"2024-01-26T23:46:31.655190+0000","last_peered":"2024-01-26T23:46:31.655190+0000","last_clean":"2024-01-26T23:46:31.655190+0000","last_became_active":"2024-01-26T23:46:30.771131+0000","last_became_peered":"2024-01-26T23:46:30.771131+0000","last_unstale":"2024-01-26T23:46:31.655190+0000","last_undegraded":"2024-01-26T23:46:31.655190+0000","last_fullsized":"2024-01-26T23:46:31.655190+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:46:39.085189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.671187+0000","last_change":"2024-01-26T23:46:31.669379+0000","last_active":"2024-01-26T23:46:31.671187+0000","last_peered":"2024-01-26T23:46:31.671187+0000","last_clean":"2024-01-26T23:46:31.671187+0000","last_became_active":"2024-01-26T23:46:31.669103+0000","last_became_peered":"2024-01-26T23:46:31.669103+0000","last_unstale":"2024-01-26T23:46:31.671187+0000","last_undegraded":"2024-01-26T23:46:31.671187+0000","last_fullsized":"2024-01-26T23:46:31.671187+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:20:32.077253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.656731+0000","last_change":"2024-01-26T23:46:30.777287+0000","last_active":"2024-01-26T23:46:31.656731+0000","last_peered":"2024-01-26T23:46:31.656731+0000","last_clean":"2024-01-26T23:46:31.656731+0000","last_became_active":"2024-01-26T23:46:30.773908+0000","last_became_peered":"2024-01-26T23:46:30.773908+0000","last_unstale":"2024-01-26T23:46:31.656731+0000","last_undegraded":"2024-01-26T23:46:31.656731+0000","last_fullsized":"2024-01-26T23:46:31.656731+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:28.962140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.676474+0000","last_change":"2024-01-26T23:46:31.673074+0000","last_active":"2024-01-26T23:46:31.676474+0000","last_peered":"2024-01-26T23:46:31.676474+0000","last_clean":"2024-01-26T23:46:31.676474+0000","last_became_active":"2024-01-26T23:46:31.668846+0000","last_became_peered":"2024-01-26T23:46:31.668846+0000","last_unstale":"2024-01-26T23:46:31.676474+0000","last_undegraded":"2024-01-26T23:46:31.676474+0000","last_fullsized":"2024-01-26T23:46:31.676474+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:45.348777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.652367+0000","last_change":"2024-01-26T23:46:30.772801+0000","last_active":"2024-01-26T23:46:31.652367+0000","last_peered":"2024-01-26T23:46:31.652367+0000","last_clean":"2024-01-26T23:46:31.652367+0000","last_became_active":"2024-01-26T23:46:30.772337+0000","last_became_peered":"2024-01-26T23:46:30.772337+0000","last_unstale":"2024-01-26T23:46:31.652367+0000","last_undegraded":"2024-01-26T23:46:31.652367+0000","last_fullsized":"2024-01-26T23:46:31.652367+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:37.479741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.678017+0000","last_change":"2024-01-26T23:46:31.677471+0000","last_active":"2024-01-26T23:46:31.678017+0000","last_peered":"2024-01-26T23:46:31.678017+0000","last_clean":"2024-01-26T23:46:31.678017+0000","last_became_active":"2024-01-26T23:46:31.677223+0000","last_became_peered":"2024-01-26T23:46:31.677223+0000","last_unstale":"2024-01-26T23:46:31.678017+0000","last_undegraded":"2024-01-26T23:46:31.678017+0000","last_fullsized":"2024-01-26T23:46:31.678017+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:46:27.325038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.654023+0000","last_change":"2024-01-26T23:46:30.763987+0000","last_active":"2024-01-26T23:46:31.654023+0000","last_peered":"2024-01-26T23:46:31.654023+0000","last_clean":"2024-01-26T23:46:31.654023+0000","last_became_active":"2024-01-26T23:46:30.763719+0000","last_became_peered":"2024-01-26T23:46:30.763719+0000","last_unstale":"2024-01-26T23:46:31.654023+0000","last_undegraded":"2024-01-26T23:46:31.654023+0000","last_fullsized":"2024-01-26T23:46:31.654023+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:41:51.840628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.675370+0000","last_change":"2024-01-26T23:46:31.668096+0000","last_active":"2024-01-26T23:46:31.675370+0000","last_peered":"2024-01-26T23:46:31.675370+0000","last_clean":"2024-01-26T23:46:31.675370+0000","last_became_active":"2024-01-26T23:46:31.665095+0000","last_became_peered":"2024-01-26T23:46:31.665095+0000","last_unstale":"2024-01-26T23:46:31.675370+0000","last_undegraded":"2024-01-26T23:46:31.675370+0000","last_fullsized":"2024-01-26T23:46:31.675370+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:53:38.299979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.654449+0000","last_change":"2024-01-26T23:46:30.781240+0000","last_active":"2024-01-26T23:46:31.654449+0000","last_peered":"2024-01-26T23:46:31.654449+0000","last_clean":"2024-01-26T23:46:31.654449+0000","last_became_active":"2024-01-26T23:46:30.781056+0000","last_became_peered":"2024-01-26T23:46:30.781056+0000","last_unstale":"2024-01-26T23:46:31.654449+0000","last_undegraded":"2024-01-26T23:46:31.654449+0000","last_fullsized":"2024-01-26T23:46:31.654449+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:04.033669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.665879+0000","last_change":"2024-01-26T23:46:31.662608+0000","last_active":"2024-01-26T23:46:31.665879+0000","last_peered":"2024-01-26T23:46:31.665879+0000","last_clean":"2024-01-26T23:46:31.665879+0000","last_became_active":"2024-01-26T23:46:31.662342+0000","last_became_peered":"2024-01-26T23:46:31.662342+0000","last_unstale":"2024-01-26T23:46:31.665879+0000","last_undegraded":"2024-01-26T23:46:31.665879+0000","last_fullsized":"2024-01-26T23:46:31.665879+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:30.754190+0000","last_clean_scrub_stamp":"2024-01-26T23:46:30.754190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:12:26.391081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-01-26T23:46:31.649742+0000","last_change":"2024-01-26T23:46:30.775933+0000","last_active":"2024-01-26T23:46:31.649742+0000","last_peered":"2024-01-26T23:46:31.649742+0000","last_clean":"2024-01-26T23:46:31.649742+0000","last_became_active":"2024-01-26T23:46:30.775286+0000","last_became_peered":"2024-01-26T23:46:30.775286+0000","last_unstale":"2024-01-26T23:46:31.649742+0000","last_undegraded":"2024-01-26T23:46:31.649742+0000","last_fullsized":"2024-01-26T23:46:31.649742+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:46:29.752847+0000","last_clean_scrub_stamp":"2024-01-26T23:46:29.752847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:24.875367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"112'5140","reported_seq":5488,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-26T23:46:35.629783+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:46:35.629783+0000","last_peered":"2024-01-26T23:46:35.629783+0000","last_clean":"2024-01-26T23:46:35.629783+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:46:35.629783+0000","last_undegraded":"2024-01-26T23:46:35.629783+0000","last_fullsized":"2024-01-26T23:46:35.629783+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":5140,"log_dups_size":0,"ondisk_log_size":5140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":743,"num_read_kb":1452,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":743,"num_read_kb":1452,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5140,"ondisk_log_size":5140,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575051,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311512,"kb_used_data":1936,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060328,"statfs":{"total":96636764160,"available":96317775872,"internally_reserved":0,"allocated":1982464,"data_stored":1292578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.54100000000000004,"15min":0.56499999999999995},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.74099999999999999,"5min":0.79500000000000004,"15min":0.82299999999999995},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.55600000000000005,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.83999999999999997},"last":0.47099999999999997}]},{"osd":1,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.55200000000000005,"15min":0.57899999999999996},"min":{"1min":0.36699999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67400000000000004,"5min":0.81299999999999994,"15min":0.82899999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.44500000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.56299999999999994,"15min":0.57699999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.81699999999999995},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.58299999999999996,"15min":0.59799999999999998},"min":{"1min":0.30199999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75700000000000001,"5min":0.81899999999999995,"15min":0.88600000000000001},"last":0.40200000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.77100000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.497},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.58699999999999997,"15min":0.60099999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71699999999999997,"5min":0.83899999999999997,"15min":0.86399999999999999},"last":0.441}]},{"osd":4,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46100000000000002,"15min":0.45600000000000002},"min":{"1min":0.252,"5min":0.247,"15min":0.247},"max":{"1min":0.64300000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.48799999999999999,"15min":0.48199999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.66400000000000003,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.378}]},{"osd":5,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.48899999999999999},"min":{"1min":0.27200000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.70599999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54200000000000004,"15min":0.52700000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.73599999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.41199999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.55300000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78700000000000003,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55700000000000005,"15min":0.54900000000000004},"min":{"1min":0.32000000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70599999999999996,"5min":0.85199999999999998,"15min":0.86899999999999999},"last":0.43099999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575051,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302752,"kb_used_data":1368,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069088,"statfs":{"total":96636764160,"available":96326746112,"internally_reserved":0,"allocated":1400832,"data_stored":704266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55800000000000005,"15min":0.57699999999999996},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26300000000000001},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.57199999999999995,"15min":0.59199999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.68300000000000005,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.45300000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.54300000000000004,"15min":0.57199999999999995},"min":{"1min":0.317,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.622,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.505},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.34399999999999997},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.34100000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55900000000000005,"15min":0.58499999999999996},"min":{"1min":0.39600000000000002,"5min":0.35499999999999998,"15min":0.29599999999999999},"max":{"1min":0.67200000000000004,"5min":0.74199999999999999,"15min":0.85799999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.56999999999999995,"15min":0.58899999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.33900000000000002},"max":{"1min":0.621,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.48399999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.29999999999999999},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60999999999999999,"15min":0.622},"min":{"1min":0.41899999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.58199999999999996}]},{"osd":4,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47699999999999998,"15min":0.47299999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.22},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.498,"15min":0.495},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.64600000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48599999999999999,"15min":0.48499999999999999},"min":{"1min":0.316,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.66100000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54000000000000004,"15min":0.53100000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26200000000000001},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":0.49399999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.246},"max":{"1min":0.64100000000000001,"5min":0.82799999999999996,"15min":0.85899999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.53500000000000003,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.308},"max":{"1min":0.68600000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.378}]}]},{"osd":1,"up_from":13,"seq":55834575051,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302736,"kb_used_data":1352,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069104,"statfs":{"total":96636764160,"available":96326762496,"internally_reserved":0,"allocated":1384448,"data_stored":702224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41599999999999998,"15min":0.40899999999999997},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.72599999999999998},"last":0.315},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.44900000000000001,"15min":0.44600000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.78600000000000003,"5min":1.337,"15min":1.337},"last":0.28999999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.41699999999999998,"15min":0.41499999999999998},"min":{"1min":0.254,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.68799999999999994,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.48199999999999998,"15min":0.47499999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.28999999999999998},"max":{"1min":0.74099999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.42199999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52500000000000002,"15min":0.51400000000000001},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":0.75900000000000001,"5min":1.782,"15min":1.782},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.48899999999999999,"15min":0.47999999999999998},"min":{"1min":0.32000000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.71499999999999997,"5min":1.379,"15min":1.379},"last":0.38800000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.64200000000000002,"15min":0.64900000000000002},"min":{"1min":0.435,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":0.81699999999999995,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.65700000000000003,"15min":0.66200000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.84299999999999997,"5min":0.91900000000000004,"15min":0.95699999999999996},"last":0.56100000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.67700000000000005,"15min":0.67300000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":0.86499999999999999,"5min":1.208,"15min":1.208},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.67700000000000005,"15min":0.66800000000000004},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.42499999999999999},"max":{"1min":0.82899999999999996,"5min":0.96599999999999997,"15min":1.018},"last":0.61299999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.68000000000000005,"15min":0.67800000000000005},"min":{"1min":0.47799999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91900000000000004,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.68000000000000005,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.78300000000000003,"5min":1.022,"15min":1.022},"last":0.504}]},{"osd":7,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.70499999999999996,"15min":0.71099999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89200000000000002,"5min":0.98199999999999998,"15min":1.0069999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.73099999999999998,"15min":0.72299999999999998},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.438},"max":{"1min":0.95399999999999996,"5min":1.296,"15min":1.296},"last":0.496}]}]},{"osd":0,"up_from":13,"seq":55834575050,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307432,"kb_used_data":1952,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064408,"statfs":{"total":96636764160,"available":96321953792,"internally_reserved":0,"allocated":1998848,"data_stored":1294736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.41999999999999998,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.45100000000000001,"15min":0.45500000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27300000000000002},"max":{"1min":1.363,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41199999999999998,"15min":0.42499999999999999},"min":{"1min":0.27900000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.57499999999999996,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.45700000000000002,"15min":0.47099999999999997},"min":{"1min":0.32700000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63700000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.36699999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.47399999999999998,"15min":0.46899999999999997},"min":{"1min":0.32300000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.145,"5min":2.145,"15min":2.145},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48599999999999999,"15min":0.48799999999999999},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64300000000000002,"5min":1.198,"15min":1.198},"last":0.32100000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.64400000000000002,"15min":0.65200000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.625,"15min":0.65000000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.752,"5min":0.94299999999999995,"15min":0.98899999999999999},"last":0.42499999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64200000000000002,"15min":0.65900000000000003},"min":{"1min":0.42199999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.67500000000000004,"15min":0.68100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.70799999999999996}]},{"osd":6,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.63,"15min":0.66700000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80200000000000005,"5min":0.89800000000000002,"15min":1.032},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.67400000000000004,"15min":0.69499999999999995},"min":{"1min":0.48299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.61299999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.028,"15min":0.83199999999999996},"min":{"1min":0.501,"5min":0.435,"15min":0.435},"max":{"1min":4.8710000000000004,"5min":25.965,"15min":25.965},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.1379999999999999,"15min":0.875},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.433},"max":{"1min":4.9199999999999999,"5min":33.085000000000001,"15min":33.085000000000001},"last":0.64400000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575050,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306828,"kb_used_data":1348,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065012,"statfs":{"total":96636764160,"available":96322572288,"internally_reserved":0,"allocated":1380352,"data_stored":702154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.51700000000000002,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.437,"15min":0.42299999999999999},"min":{"1min":0.33500000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.28299999999999997}]},{"osd":1,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.432,"15min":0.42799999999999999},"min":{"1min":0.26900000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.61899999999999999,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.377,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.38900000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46999999999999997,"15min":0.48599999999999999},"min":{"1min":0.35999999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.64800000000000002,"5min":0.65500000000000003,"15min":0.74099999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.48099999999999998,"15min":0.47499999999999998},"min":{"1min":0.35599999999999998,"5min":0.32200000000000001,"15min":0.307},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.63600000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.63200000000000001,"15min":0.63400000000000001},"min":{"1min":0.41499999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.80900000000000005,"5min":1.087,"15min":1.087},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.65200000000000002,"15min":0.64400000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.874,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.68200000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.65100000000000002,"15min":0.65200000000000002},"min":{"1min":0.375,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.90300000000000002,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.67300000000000004,"15min":0.67900000000000005},"min":{"1min":0.41799999999999998,"5min":0.39800000000000002,"15min":0.35199999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.085},"last":0.70099999999999996}]},{"osd":6,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67900000000000005,"15min":0.68100000000000005},"min":{"1min":0.47699999999999998,"5min":0.439,"15min":0.40899999999999997},"max":{"1min":1.022,"5min":1.022,"15min":1.1080000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.67700000000000005,"15min":0.68300000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.38800000000000001},"max":{"1min":0.83799999999999997,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.71699999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.69499999999999995,"15min":0.70299999999999996},"min":{"1min":0.52100000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94699999999999995,"5min":1.0029999999999999,"15min":1.127},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.72299999999999998,"15min":0.73299999999999998},"min":{"1min":0.48899999999999999,"5min":0.48199999999999998,"15min":0.42099999999999999},"max":{"1min":1.05,"5min":1.052,"15min":1.139},"last":0.66000000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575049,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306828,"kb_used_data":1348,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065012,"statfs":{"total":96636764160,"available":96322572288,"internally_reserved":0,"allocated":1380352,"data_stored":702154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.39900000000000002,"15min":0.39400000000000002},"min":{"1min":0.26500000000000001,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.45000000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.44,"15min":0.439},"min":{"1min":0.25800000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.58699999999999997,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.42399999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.46600000000000003,"15min":0.45500000000000002},"min":{"1min":0.27900000000000003,"5min":0.253,"15min":0.253},"max":{"1min":0.57799999999999996,"5min":1.633,"15min":1.633},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.48399999999999999,"15min":0.48099999999999998},"min":{"1min":0.35199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.54900000000000004,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58099999999999996}]},{"osd":2,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.48899999999999999,"15min":0.47899999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.54600000000000004,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.33200000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.53700000000000003,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.503}]},{"osd":4,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.68200000000000005,"15min":0.66300000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.878,"5min":3.306,"15min":3.306},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.41499999999999998,"5min":0.40200000000000002,"15min":0.379},"max":{"1min":0.85199999999999998,"5min":0.95799999999999996,"15min":1.008},"last":0.40100000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.65700000000000003,"15min":0.66100000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82499999999999996,"5min":1.593,"15min":1.593},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.67500000000000004,"15min":0.67300000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.84399999999999997,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.46200000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.67100000000000004,"15min":0.67700000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.77800000000000002,"5min":1.506,"15min":1.506},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.69099999999999995,"15min":0.69299999999999995},"min":{"1min":0.48799999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.82599999999999996,"5min":1.427,"15min":1.427},"last":0.53100000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.70799999999999996,"15min":0.69799999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.42499999999999999},"max":{"1min":0.73899999999999999,"5min":3.238,"15min":3.238},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.747,"15min":0.73899999999999999},"min":{"1min":0.54100000000000004,"5min":0.48899999999999999,"15min":0.48699999999999999},"max":{"1min":0.86699999999999999,"5min":3.351,"15min":3.351},"last":0.64100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575051,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302740,"kb_used_data":1356,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069100,"statfs":{"total":96636764160,"available":96326758400,"internally_reserved":0,"allocated":1388544,"data_stored":702314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.54800000000000004,"15min":0.57799999999999996},"min":{"1min":0.28999999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57699999999999996,"5min":1.21,"15min":1.21},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.58499999999999996,"15min":0.58999999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.367,"5min":5.367,"15min":5.367},"last":0.48799999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.55700000000000005,"15min":0.59299999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68899999999999995,"5min":0.77500000000000002,"15min":0.88},"last":0.442},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71499999999999997,"5min":0.80600000000000005,"15min":0.81699999999999995},"last":0.501}]},{"osd":2,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53500000000000003,"15min":0.56799999999999995},"min":{"1min":0.36699999999999999,"5min":0.33300000000000002,"15min":0.32700000000000001},"max":{"1min":0.56599999999999995,"5min":0.81100000000000005,"15min":0.85999999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.58899999999999997,"15min":0.60499999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.69599999999999995,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.50700000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.56000000000000005,"15min":0.58699999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.318},"max":{"1min":0.66900000000000004,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.434},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56699999999999995,"15min":0.58599999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.35699999999999998},"max":{"1min":0.64900000000000002,"5min":1.05,"15min":1.05},"last":0.51300000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.46000000000000002,"15min":0.46300000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.63100000000000001,"5min":0.76200000000000001,"15min":0.77600000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.48499999999999999,"15min":0.48199999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68300000000000005,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.47599999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.47099999999999997,"15min":0.47199999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.27200000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.73899999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.311},"max":{"1min":0.72899999999999998,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.374}]},{"osd":7,"last update":"Fri Jan 26 23:46:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.36699999999999999,"5min":0.36499999999999999,"15min":0.35799999999999998},"max":{"1min":0.753,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.57099999999999995,"15min":0.56899999999999995},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81000000000000005,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.45600000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575051,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298636,"kb_used_data":1348,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073204,"statfs":{"total":96636764160,"available":96330960896,"internally_reserved":0,"allocated":1380352,"data_stored":702154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.26000000000000001},"max":{"1min":0.64700000000000002,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.502,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.315,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.65800000000000003,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.63}]},{"osd":1,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.61199999999999999,"15min":0.622},"min":{"1min":0.34799999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66300000000000003,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.629,"15min":0.63400000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.39300000000000002},"max":{"1min":0.71199999999999997,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.55900000000000005,"15min":0.56999999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.30099999999999999},"max":{"1min":0.67400000000000004,"5min":0.83699999999999997,"15min":0.83799999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.58999999999999997,"15min":0.59699999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69699999999999995,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":0.59199999999999997}]},{"osd":3,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.57099999999999995,"15min":0.58399999999999996},"min":{"1min":0.38100000000000001,"5min":0.35999999999999999,"15min":0.314},"max":{"1min":0.68799999999999994,"5min":0.86199999999999999,"15min":0.88400000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.59899999999999998,"15min":0.59199999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73599999999999999,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.72399999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.48899999999999999,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.48999999999999999,"15min":0.48699999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":0.626,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.44}]},{"osd":6,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.48799999999999999,"15min":0.47699999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65400000000000003,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.52100000000000002,"15min":0.51900000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.76200000000000001,"5min":1.99,"15min":1.99},"last":0.50600000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.312,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71399999999999997,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79300000000000004,"5min":0.82199999999999995,"15min":0.83099999999999996},"last":0.51600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:46:41.123 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-26T23:46:41.123 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-26T23:46:41.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.300+0000 7fbcae3f0700 1 Processor -- start 2024-01-26T23:46:41.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.301+0000 7fbcae3f0700 1 -- start start 2024-01-26T23:46:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.301+0000 7fbcae3f0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbca8141200 0x7fbca80c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.301+0000 7fbcae3f0700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca80ccc00 con 0x7fbca80c8da0 2024-01-26T23:46:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.301+0000 7fbcae3f0700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca80cc310 con 0x7fbca80caf30 2024-01-26T23:46:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.301+0000 7fbcae3f0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca80cbbf0 con 0x7fbca8141200 2024-01-26T23:46:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.302+0000 7fbca7fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbca80c8da0 0x7fbca8140af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34902/0 (socket says 172.21.15.46:34902) 2024-01-26T23:46:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.302+0000 7fbca7fff700 1 -- 172.21.15.46:0/3690694361 learned_addr learned my addr 172.21.15.46:0/3690694361 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.302+0000 7fbca77fe700 1 --2- 172.21.15.46:0/3690694361 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbca8141200 0x7fbca80c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.302+0000 7fbca6ffd700 1 -- 172.21.15.46:0/3690694361 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32793106 0 0) 0x7fbca80cc310 con 0x7fbca80caf30 2024-01-26T23:46:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.302+0000 7fbca6ffd700 1 -- 172.21.15.46:0/3690694361 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc94003570 con 0x7fbca80caf30 2024-01-26T23:46:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.302+0000 7fbca6ffd700 1 -- 172.21.15.46:0/3690694361 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054942066 0 0) 0x7fbca80ccc00 con 0x7fbca80c8da0 2024-01-26T23:46:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.302+0000 7fbca6ffd700 1 -- 172.21.15.46:0/3690694361 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca80cc310 con 0x7fbca80c8da0 2024-01-26T23:46:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.302+0000 7fbca77fe700 1 -- 172.21.15.46:0/3690694361 >> v1:172.21.15.106:6789/0 conn(0x7fbca80caf30 legacy=0x7fbca80cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.302+0000 7fbca77fe700 1 -- 172.21.15.46:0/3690694361 >> v1:172.21.15.46:6789/0 conn(0x7fbca80c8da0 legacy=0x7fbca8140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.302+0000 7fbca77fe700 1 -- 172.21.15.46:0/3690694361 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca80cc5b0 con 0x7fbca8141200 2024-01-26T23:46:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.302+0000 7fbca77fe700 1 --2- 172.21.15.46:0/3690694361 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbca8141200 0x7fbca80c9170 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009910 tx=0x7fbc900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ca3cd619974f08c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.303+0000 7fbca6ffd700 1 -- 172.21.15.46:0/3690694361 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc90011630 con 0x7fbca8141200 2024-01-26T23:46:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.303+0000 7fbca6ffd700 1 -- 172.21.15.46:0/3690694361 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc90011790 con 0x7fbca8141200 2024-01-26T23:46:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.303+0000 7fbca6ffd700 1 -- 172.21.15.46:0/3690694361 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc90011b30 con 0x7fbca8141200 2024-01-26T23:46:41.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.303+0000 7fbcae3f0700 1 -- 172.21.15.46:0/3690694361 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbca8141200 msgr2=0x7fbca80c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:41.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.303+0000 7fbcae3f0700 1 --2- 172.21.15.46:0/3690694361 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbca8141200 0x7fbca80c9170 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009910 tx=0x7fbc900091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.303+0000 7fbcae3f0700 1 -- 172.21.15.46:0/3690694361 shutdown_connections 2024-01-26T23:46:41.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.303+0000 7fbcae3f0700 1 --2- 172.21.15.46:0/3690694361 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbca8141200 0x7fbca80c9170 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.303+0000 7fbcae3f0700 1 -- 172.21.15.46:0/3690694361 >> 172.21.15.46:0/3690694361 conn(0x7fbca806bd90 msgr2=0x7fbca806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:41.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.303+0000 7fbcae3f0700 1 -- 172.21.15.46:0/3690694361 shutdown_connections 2024-01-26T23:46:41.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.303+0000 7fbcae3f0700 1 -- 172.21.15.46:0/3690694361 wait complete. 2024-01-26T23:46:41.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.305+0000 7fbcae3f0700 1 Processor -- start 2024-01-26T23:46:41.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.305+0000 7fbcae3f0700 1 -- start start 2024-01-26T23:46:41.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.306+0000 7fbcae3f0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbca80c8da0 0x7fbca80ce530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:41.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.306+0000 7fbcae3f0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbca80caf30 0x7fbca80cea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:41.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.306+0000 7fbcae3f0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbca8141200 0x7fbca8159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:41.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.306+0000 7fbcae3f0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca80b4320 con 0x7fbca80caf30 2024-01-26T23:46:41.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.306+0000 7fbcae3f0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca815b7d0 con 0x7fbca80c8da0 2024-01-26T23:46:41.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.306+0000 7fbcae3f0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca815abc0 con 0x7fbca8141200 2024-01-26T23:46:41.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.306+0000 7fbca77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbca80caf30 0x7fbca80cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:41.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.306+0000 7fbca7fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbca80c8da0 0x7fbca80ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:41.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.306+0000 7fbca7fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbca80c8da0 0x7fbca80ce530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:48766/0 (socket says 172.21.15.46:48766) 2024-01-26T23:46:41.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.306+0000 7fbca77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbca80caf30 0x7fbca80cea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35406/0 (socket says 172.21.15.46:35406) 2024-01-26T23:46:41.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.306+0000 7fbca7fff700 1 -- 172.21.15.46:0/2696351760 learned_addr learned my addr 172.21.15.46:0/2696351760 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:41.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.306+0000 7fbcac98d700 1 --2- 172.21.15.46:0/2696351760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbca8141200 0x7fbca8159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:41.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.307+0000 7fbca7fff700 1 -- 172.21.15.46:0/2696351760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbca8141200 msgr2=0x7fbca8159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:41.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.307+0000 7fbca7fff700 1 --2- 172.21.15.46:0/2696351760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbca8141200 0x7fbca8159eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.307+0000 7fbca7fff700 1 -- 172.21.15.46:0/2696351760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbca80caf30 msgr2=0x7fbca80cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:41.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.307+0000 7fbca7fff700 1 --2- 172.21.15.46:0/2696351760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbca80caf30 0x7fbca80cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.307+0000 7fbca7fff700 1 -- 172.21.15.46:0/2696351760 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc90009420 con 0x7fbca80c8da0 2024-01-26T23:46:41.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.307+0000 7fbca77fe700 1 --2- 172.21.15.46:0/2696351760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbca80caf30 0x7fbca80cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:41.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.307+0000 7fbca7fff700 1 --2- 172.21.15.46:0/2696351760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbca80c8da0 0x7fbca80ce530 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fbc9800c7c0 tx=0x7fbc98007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:41.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.307+0000 7fbc9ffff700 1 -- 172.21.15.46:0/2696351760 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc98005cd0 con 0x7fbca80c8da0 2024-01-26T23:46:41.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.307+0000 7fbc9ffff700 1 -- 172.21.15.46:0/2696351760 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc98005e30 con 0x7fbca80c8da0 2024-01-26T23:46:41.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.307+0000 7fbcae3f0700 1 -- 172.21.15.46:0/2696351760 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca815aea0 con 0x7fbca80c8da0 2024-01-26T23:46:41.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.307+0000 7fbc9ffff700 1 -- 172.21.15.46:0/2696351760 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc98007320 con 0x7fbca80c8da0 2024-01-26T23:46:41.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.307+0000 7fbcae3f0700 1 -- 172.21.15.46:0/2696351760 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca815b460 con 0x7fbca80c8da0 2024-01-26T23:46:41.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.308+0000 7fbcae3f0700 1 -- 172.21.15.46:0/2696351760 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca804c710 con 0x7fbca80c8da0 2024-01-26T23:46:41.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.309+0000 7fbc9ffff700 1 -- 172.21.15.46:0/2696351760 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fbc9801b3b0 con 0x7fbca80c8da0 2024-01-26T23:46:41.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.309+0000 7fbc9ffff700 1 --2- 172.21.15.46:0/2696351760 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbc88066ea0 0x7fbc88069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:41.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.309+0000 7fbc9ffff700 1 -- 172.21.15.46:0/2696351760 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fbc98083d80 con 0x7fbca80c8da0 2024-01-26T23:46:41.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.309+0000 7fbca77fe700 1 --2- 172.21.15.46:0/2696351760 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbc88066ea0 0x7fbc88069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:41.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.310+0000 7fbca77fe700 1 --2- 172.21.15.46:0/2696351760 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbc88066ea0 0x7fbc88069350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009860 tx=0x7fbc90017000 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:41.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.314+0000 7fbc9ffff700 1 -- 172.21.15.46:0/2696351760 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbc98050e00 con 0x7fbca80c8da0 2024-01-26T23:46:41.467 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:41.466+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:41.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.623+0000 7fbcae3f0700 1 -- 172.21.15.46:0/2696351760 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fbca80cc890 con 0x7fbca80c8da0 2024-01-26T23:46:41.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.627+0000 7fbc9ffff700 1 -- 172.21.15.46:0/2696351760 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+0 (secure 0 0 0) 0x7fbc98017020 con 0x7fbca80c8da0 2024-01-26T23:46:41.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.629+0000 7fbcae3f0700 1 -- 172.21.15.46:0/2696351760 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbc88066ea0 msgr2=0x7fbc88069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:41.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.629+0000 7fbcae3f0700 1 --2- 172.21.15.46:0/2696351760 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbc88066ea0 0x7fbc88069350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009860 tx=0x7fbc90017000 comp rx=0 tx=0).stop 2024-01-26T23:46:41.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.629+0000 7fbcae3f0700 1 -- 172.21.15.46:0/2696351760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbca80c8da0 msgr2=0x7fbca80ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:41.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.629+0000 7fbcae3f0700 1 --2- 172.21.15.46:0/2696351760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbca80c8da0 0x7fbca80ce530 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fbc9800c7c0 tx=0x7fbc98007e00 comp rx=0 tx=0).stop 2024-01-26T23:46:41.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.629+0000 7fbcae3f0700 1 -- 172.21.15.46:0/2696351760 shutdown_connections 2024-01-26T23:46:41.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.629+0000 7fbcae3f0700 1 --2- 172.21.15.46:0/2696351760 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fbc88066ea0 0x7fbc88069350 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.629+0000 7fbcae3f0700 1 --2- 172.21.15.46:0/2696351760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbca8141200 0x7fbca8159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.629+0000 7fbcae3f0700 1 --2- 172.21.15.46:0/2696351760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbca80c8da0 0x7fbca80ce530 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.629+0000 7fbcae3f0700 1 --2- 172.21.15.46:0/2696351760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbca80caf30 0x7fbca80cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.629+0000 7fbcae3f0700 1 -- 172.21.15.46:0/2696351760 >> 172.21.15.46:0/2696351760 conn(0x7fbca806bd90 msgr2=0x7fbca813d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.630+0000 7fbcae3f0700 1 -- 172.21.15.46:0/2696351760 shutdown_connections 2024-01-26T23:46:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.630+0000 7fbcae3f0700 1 -- 172.21.15.46:0/2696351760 wait complete. 2024-01-26T23:46:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-01-26T23:46:41.645 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:46:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.776+0000 7f3143079700 1 Processor -- start 2024-01-26T23:46:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.776+0000 7f3143079700 1 -- start start 2024-01-26T23:46:41.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.777+0000 7f3143079700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f313c157620 0x7f313c0c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:41.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.777+0000 7f3143079700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f313c1579f0 0x7f313c0c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:41.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.777+0000 7f3143079700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f313c0c78b0 0x7f313c0af9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:41.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.777+0000 7f3143079700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f313c06bce0 con 0x7f313c1579f0 2024-01-26T23:46:41.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.777+0000 7f3143079700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f313c06b850 con 0x7f313c0c78b0 2024-01-26T23:46:41.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.777+0000 7f3143079700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313c06af90 con 0x7f313c157620 2024-01-26T23:46:41.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.777+0000 7f313bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f313c1579f0 0x7f313c0c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:41.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.778+0000 7f313bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f313c1579f0 0x7f313c0c7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35418/0 (socket says 172.21.15.46:35418) 2024-01-26T23:46:41.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.778+0000 7f313bfff700 1 -- 172.21.15.46:0/3384823264 learned_addr learned my addr 172.21.15.46:0/3384823264 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:41.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.778+0000 7f3141616700 1 --2- 172.21.15.46:0/3384823264 >> v2:172.21.15.106:3300/0 conn(0x7f313c0c78b0 0x7f313c0af9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:41.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.778+0000 7f3140e15700 1 --2- 172.21.15.46:0/3384823264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f313c157620 0x7f313c0c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:41.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.778+0000 7f313bfff700 1 -- 172.21.15.46:0/3384823264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f313c157620 msgr2=0x7f313c0c9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:41.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.778+0000 7f313bfff700 1 --2- 172.21.15.46:0/3384823264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f313c157620 0x7f313c0c9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.778+0000 7f313bfff700 1 -- 172.21.15.46:0/3384823264 >> v2:172.21.15.106:3300/0 conn(0x7f313c0c78b0 msgr2=0x7f313c0af9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:41.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.778+0000 7f313bfff700 1 --2- 172.21.15.46:0/3384823264 >> v2:172.21.15.106:3300/0 conn(0x7f313c0c78b0 0x7f313c0af9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.778+0000 7f313bfff700 1 -- 172.21.15.46:0/3384823264 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313c0c7d10 con 0x7f313c1579f0 2024-01-26T23:46:41.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.778+0000 7f313bfff700 1 --2- 172.21.15.46:0/3384823264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f313c1579f0 0x7f313c0c7370 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f313c0c71e0 tx=0x7f3128009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d780391ae84cc058 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:41.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.778+0000 7f313b7fe700 1 -- 172.21.15.46:0/3384823264 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312800b640 con 0x7f313c1579f0 2024-01-26T23:46:41.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.778+0000 7f313b7fe700 1 -- 172.21.15.46:0/3384823264 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312800b7a0 con 0x7f313c1579f0 2024-01-26T23:46:41.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.778+0000 7f313b7fe700 1 -- 172.21.15.46:0/3384823264 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312800bb60 con 0x7f313c1579f0 2024-01-26T23:46:41.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.779+0000 7f3143079700 1 -- 172.21.15.46:0/3384823264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f313c1579f0 msgr2=0x7f313c0c7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:41.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.779+0000 7f3143079700 1 --2- 172.21.15.46:0/3384823264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f313c1579f0 0x7f313c0c7370 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f313c0c71e0 tx=0x7f3128009250 comp rx=0 tx=0).stop 2024-01-26T23:46:41.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.779+0000 7f3143079700 1 -- 172.21.15.46:0/3384823264 shutdown_connections 2024-01-26T23:46:41.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.779+0000 7f3143079700 1 --2- 172.21.15.46:0/3384823264 >> v2:172.21.15.106:3300/0 conn(0x7f313c0c78b0 0x7f313c0af9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.779+0000 7f3143079700 1 --2- 172.21.15.46:0/3384823264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f313c157620 0x7f313c0c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.779+0000 7f3143079700 1 --2- 172.21.15.46:0/3384823264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f313c1579f0 0x7f313c0c7370 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.779+0000 7f3143079700 1 -- 172.21.15.46:0/3384823264 >> 172.21.15.46:0/3384823264 conn(0x7f313c0cd6e0 msgr2=0x7f313c0c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:41.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.779+0000 7f3143079700 1 -- 172.21.15.46:0/3384823264 shutdown_connections 2024-01-26T23:46:41.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.779+0000 7f3143079700 1 -- 172.21.15.46:0/3384823264 wait complete. 2024-01-26T23:46:41.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.780+0000 7f3143079700 1 Processor -- start 2024-01-26T23:46:41.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.780+0000 7f3143079700 1 -- start start 2024-01-26T23:46:41.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.781+0000 7f3143079700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f313c0c78b0 0x7f313c0adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:41.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.781+0000 7f3143079700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f313c157620 0x7f313c1530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:41.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.781+0000 7f3143079700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f313c1579f0 0x7f313c067930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:41.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.781+0000 7f3143079700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313c0ae430 con 0x7f313c0c78b0 2024-01-26T23:46:41.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.781+0000 7f3143079700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313c169660 con 0x7f313c157620 2024-01-26T23:46:41.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.781+0000 7f3143079700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313c169940 con 0x7f313c1579f0 2024-01-26T23:46:41.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.781+0000 7f3140e15700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f313c0c78b0 0x7f313c0adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:41.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.781+0000 7f3140e15700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f313c0c78b0 0x7f313c0adef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35424/0 (socket says 172.21.15.46:35424) 2024-01-26T23:46:41.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.781+0000 7f3140e15700 1 -- 172.21.15.46:0/3605038262 learned_addr learned my addr 172.21.15.46:0/3605038262 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:41.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.781+0000 7f313bfff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f313c157620 0x7f313c1530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:41.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.781+0000 7f3141616700 1 --2- 172.21.15.46:0/3605038262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f313c1579f0 0x7f313c067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.782+0000 7f3140e15700 1 -- 172.21.15.46:0/3605038262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f313c1579f0 msgr2=0x7f313c067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.782+0000 7f3140e15700 1 --2- 172.21.15.46:0/3605038262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f313c1579f0 0x7f313c067930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.782+0000 7f3140e15700 1 -- 172.21.15.46:0/3605038262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f313c157620 msgr2=0x7f313c1530f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.782+0000 7f3140e15700 1 --2- 172.21.15.46:0/3605038262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f313c157620 0x7f313c1530f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.782+0000 7f3140e15700 1 -- 172.21.15.46:0/3605038262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31280094f0 con 0x7f313c0c78b0 2024-01-26T23:46:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.782+0000 7f313bfff700 1 --2- 172.21.15.46:0/3605038262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f313c157620 0x7f313c1530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.782+0000 7f3140e15700 1 --2- 172.21.15.46:0/3605038262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f313c0c78b0 0x7f313c0adef0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f313000ec80 tx=0x7f31300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.782+0000 7f3138ff9700 1 -- 172.21.15.46:0/3605038262 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313000c850 con 0x7f313c0c78b0 2024-01-26T23:46:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.782+0000 7f3138ff9700 1 -- 172.21.15.46:0/3605038262 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313000c9b0 con 0x7f313c0c78b0 2024-01-26T23:46:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.782+0000 7f3138ff9700 1 -- 172.21.15.46:0/3605038262 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313000cd50 con 0x7f313c0c78b0 2024-01-26T23:46:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.782+0000 7f3143079700 1 -- 172.21.15.46:0/3605038262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313c169c20 con 0x7f313c0c78b0 2024-01-26T23:46:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.782+0000 7f3143079700 1 -- 172.21.15.46:0/3605038262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313c169fb0 con 0x7f313c0c78b0 2024-01-26T23:46:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.783+0000 7f3143079700 1 -- 172.21.15.46:0/3605038262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313c169420 con 0x7f313c0c78b0 2024-01-26T23:46:41.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.784+0000 7f3138ff9700 1 -- 172.21.15.46:0/3605038262 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f3130013680 con 0x7f313c0c78b0 2024-01-26T23:46:41.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.784+0000 7f3138ff9700 1 --2- 172.21.15.46:0/3605038262 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3120066ea0 0x7f3120069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:41.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.784+0000 7f3138ff9700 1 -- 172.21.15.46:0/3605038262 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f3130084940 con 0x7f313c0c78b0 2024-01-26T23:46:41.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.785+0000 7f313bfff700 1 --2- 172.21.15.46:0/3605038262 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3120066ea0 0x7f3120069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:41.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.785+0000 7f313bfff700 1 --2- 172.21.15.46:0/3605038262 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3120066ea0 0x7f3120069350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f313c0aed50 tx=0x7f3128017000 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:41.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:41.789+0000 7f3138ff9700 1 -- 172.21.15.46:0/3605038262 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f313c169420 con 0x7f313c0c78b0 2024-01-26T23:46:42.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.047+0000 7f3143079700 1 -- 172.21.15.46:0/3605038262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f313c169420 con 0x7f313c0c78b0 2024-01-26T23:46:42.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.047+0000 7f3138ff9700 1 -- 172.21.15.46:0/3605038262 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f313c169420 con 0x7f313c0c78b0 2024-01-26T23:46:42.049 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:42.049 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6518,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/4188139264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":4188139264},{"type":"v1","addr":"172.21.15.106:6836","nonce":4188139264}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":48},{"gid":6527,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/221301328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":221301328},{"type":"v1","addr":"172.21.15.46:6833","nonce":221301328}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":6530,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1531399912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":1531399912},{"type":"v1","addr":"172.21.15.46:6835","nonce":1531399912}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:46:31.208843+0000","modified":"2024-01-26T23:46:34.775841+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7084":{"gid":7084,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6837/1643596655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1643596655},{"type":"v1","addr":"172.21.15.106:6837","nonce":1643596655}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2024-01-26T23:46:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.050+0000 7f3143079700 1 -- 172.21.15.46:0/3605038262 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3120066ea0 msgr2=0x7f3120069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.050+0000 7f3143079700 1 --2- 172.21.15.46:0/3605038262 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3120066ea0 0x7f3120069350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f313c0aed50 tx=0x7f3128017000 comp rx=0 tx=0).stop 2024-01-26T23:46:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.050+0000 7f3143079700 1 -- 172.21.15.46:0/3605038262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f313c0c78b0 msgr2=0x7f313c0adef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.050+0000 7f3143079700 1 --2- 172.21.15.46:0/3605038262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f313c0c78b0 0x7f313c0adef0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f313000ec80 tx=0x7f31300091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.050+0000 7f3143079700 1 -- 172.21.15.46:0/3605038262 shutdown_connections 2024-01-26T23:46:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.050+0000 7f3143079700 1 --2- 172.21.15.46:0/3605038262 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f3120066ea0 0x7f3120069350 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.050+0000 7f3143079700 1 --2- 172.21.15.46:0/3605038262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f313c0c78b0 0x7f313c0adef0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.050+0000 7f3143079700 1 --2- 172.21.15.46:0/3605038262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f313c1579f0 0x7f313c067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.050+0000 7f3143079700 1 --2- 172.21.15.46:0/3605038262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f313c157620 0x7f313c1530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.050+0000 7f3143079700 1 -- 172.21.15.46:0/3605038262 >> 172.21.15.46:0/3605038262 conn(0x7f313c0cd6e0 msgr2=0x7f313c0c3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:42.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.050+0000 7f3143079700 1 -- 172.21.15.46:0/3605038262 shutdown_connections 2024-01-26T23:46:42.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.051+0000 7f3143079700 1 -- 172.21.15.46:0/3605038262 wait complete. 2024-01-26T23:46:42.053 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 54 2024-01-26T23:46:42.339 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:42.337+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:42.469 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 54, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-26T23:46:31.208843+0000', 'modified': '2024-01-26T23:46:34.775841+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7084}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7084': {'gid': 7084, 'name': 'd', 'rank': 0, 'incarnation': 50, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.106:6837/1643596655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6835', 'nonce': 1643596655}, {'type': 'v1', 'addr': '172.21.15.106:6837', 'nonce': 1643596655}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-26T23:46:42.469 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-26T23:46:42.469 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-26T23:46:42.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.613+0000 7fae8fcb8700 1 Processor -- start 2024-01-26T23:46:42.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.613+0000 7fae8fcb8700 1 -- start start 2024-01-26T23:46:42.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.614+0000 7fae8fcb8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae88159590 0x7fae8815d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.614+0000 7fae8fcb8700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fae8815d8f0 0x7fae8806bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.614+0000 7fae8fcb8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8815dd90 con 0x7fae88159960 2024-01-26T23:46:42.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.614+0000 7fae8fcb8700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8815e350 con 0x7fae8815d8f0 2024-01-26T23:46:42.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.614+0000 7fae8fcb8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae880c97c0 con 0x7fae88159590 2024-01-26T23:46:42.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.614+0000 7fae8d253700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fae8815d8f0 0x7fae8806bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.614+0000 7fae8da54700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae88159590 0x7fae8815d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.614+0000 7fae8e255700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fae88159960 0x7fae8806d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34910/0 (socket says 172.21.15.46:34910) 2024-01-26T23:46:42.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.614+0000 7fae8e255700 1 -- 172.21.15.46:0/3760101664 learned_addr learned my addr 172.21.15.46:0/3760101664 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:42.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.615+0000 7fae8d253700 1 -- 172.21.15.46:0/3760101664 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae88159590 msgr2=0x7fae8815d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:42.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.615+0000 7fae8d253700 1 --2- 172.21.15.46:0/3760101664 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae88159590 0x7fae8815d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.615+0000 7fae8d253700 1 -- 172.21.15.46:0/3760101664 >> v1:172.21.15.46:6789/0 conn(0x7fae88159960 legacy=0x7fae8806d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:42.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.615+0000 7fae8d253700 1 -- 172.21.15.46:0/3760101664 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8815e070 con 0x7fae8815d8f0 2024-01-26T23:46:42.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.615+0000 7fae8ca52700 1 -- 172.21.15.46:0/3760101664 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582773906 0 0) 0x7fae8815dd90 con 0x7fae88159960 2024-01-26T23:46:42.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.615+0000 7fae8da54700 1 --2- 172.21.15.46:0/3760101664 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae88159590 0x7fae8815d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:42.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.615+0000 7fae8d253700 1 --2- 172.21.15.46:0/3760101664 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae8815d8f0 0x7fae8806bcb0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fae78009910 tx=0x7fae780091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20eb4e9d6d3fb572 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:42.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.615+0000 7fae8ca52700 1 -- 172.21.15.46:0/3760101664 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7800b640 con 0x7fae8815d8f0 2024-01-26T23:46:42.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.615+0000 7fae8ca52700 1 -- 172.21.15.46:0/3760101664 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7800b7a0 con 0x7fae8815d8f0 2024-01-26T23:46:42.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.615+0000 7fae8ca52700 1 -- 172.21.15.46:0/3760101664 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7800bb60 con 0x7fae8815d8f0 2024-01-26T23:46:42.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.616+0000 7fae8fcb8700 1 -- 172.21.15.46:0/3760101664 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae8815d8f0 msgr2=0x7fae8806bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:42.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.616+0000 7fae8fcb8700 1 --2- 172.21.15.46:0/3760101664 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae8815d8f0 0x7fae8806bcb0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fae78009910 tx=0x7fae780091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.616+0000 7fae8fcb8700 1 -- 172.21.15.46:0/3760101664 shutdown_connections 2024-01-26T23:46:42.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.616+0000 7fae8fcb8700 1 --2- 172.21.15.46:0/3760101664 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae88159590 0x7fae8815d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.616+0000 7fae8fcb8700 1 --2- 172.21.15.46:0/3760101664 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae8815d8f0 0x7fae8806bcb0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.616+0000 7fae8fcb8700 1 -- 172.21.15.46:0/3760101664 >> 172.21.15.46:0/3760101664 conn(0x7fae880ce530 msgr2=0x7fae880ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:42.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.616+0000 7fae8fcb8700 1 -- 172.21.15.46:0/3760101664 shutdown_connections 2024-01-26T23:46:42.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.616+0000 7fae8fcb8700 1 -- 172.21.15.46:0/3760101664 wait complete. 2024-01-26T23:46:42.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.618+0000 7fae8fcb8700 1 Processor -- start 2024-01-26T23:46:42.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.618+0000 7fae8fcb8700 1 -- start start 2024-01-26T23:46:42.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.619+0000 7fae8fcb8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae88159590 0x7fae88069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.619+0000 7fae8fcb8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae88159960 0x7fae880c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.619+0000 7fae8fcb8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae8815d8f0 0x7fae8814a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.619+0000 7fae8fcb8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae880c7a00 con 0x7fae88159960 2024-01-26T23:46:42.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.619+0000 7fae8fcb8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae880b1ca0 con 0x7fae8815d8f0 2024-01-26T23:46:42.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.619+0000 7fae8fcb8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae880b1f40 con 0x7fae88159590 2024-01-26T23:46:42.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.619+0000 7fae8d253700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae88159960 0x7fae880c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.619+0000 7fae8d253700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae88159960 0x7fae880c6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35434/0 (socket says 172.21.15.46:35434) 2024-01-26T23:46:42.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.619+0000 7fae8d253700 1 -- 172.21.15.46:0/3556324498 learned_addr learned my addr 172.21.15.46:0/3556324498 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:42.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.619+0000 7fae8e255700 1 --2- 172.21.15.46:0/3556324498 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae8815d8f0 0x7fae8814a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.619+0000 7fae8da54700 1 --2- 172.21.15.46:0/3556324498 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae88159590 0x7fae88069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.619+0000 7fae8d253700 1 -- 172.21.15.46:0/3556324498 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae88159590 msgr2=0x7fae88069a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:42.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.619+0000 7fae8d253700 1 --2- 172.21.15.46:0/3556324498 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae88159590 0x7fae88069a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.619+0000 7fae8d253700 1 -- 172.21.15.46:0/3556324498 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae8815d8f0 msgr2=0x7fae8814a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:42.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.619+0000 7fae8d253700 1 --2- 172.21.15.46:0/3556324498 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae8815d8f0 0x7fae8814a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.619+0000 7fae8d253700 1 -- 172.21.15.46:0/3556324498 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae84009420 con 0x7fae88159960 2024-01-26T23:46:42.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.620+0000 7fae8d253700 1 --2- 172.21.15.46:0/3556324498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae88159960 0x7fae880c6d40 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fae8806f390 tx=0x7fae78017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:42.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.620+0000 7fae7dffb700 1 -- 172.21.15.46:0/3556324498 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7801f070 con 0x7fae88159960 2024-01-26T23:46:42.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.620+0000 7fae7dffb700 1 -- 172.21.15.46:0/3556324498 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7800b910 con 0x7fae88159960 2024-01-26T23:46:42.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.620+0000 7fae8fcb8700 1 -- 172.21.15.46:0/3556324498 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae78009420 con 0x7fae88159960 2024-01-26T23:46:42.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.620+0000 7fae7dffb700 1 -- 172.21.15.46:0/3556324498 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae78011420 con 0x7fae88159960 2024-01-26T23:46:42.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.620+0000 7fae8fcb8700 1 -- 172.21.15.46:0/3556324498 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae880b2500 con 0x7fae88159960 2024-01-26T23:46:42.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.622+0000 7fae7dffb700 1 -- 172.21.15.46:0/3556324498 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fae78011810 con 0x7fae88159960 2024-01-26T23:46:42.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.622+0000 7fae7dffb700 1 --2- 172.21.15.46:0/3556324498 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fae6c066ea0 0x7fae6c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.622+0000 7fae7dffb700 1 -- 172.21.15.46:0/3556324498 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fae78085960 con 0x7fae88159960 2024-01-26T23:46:42.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.622+0000 7fae8da54700 1 --2- 172.21.15.46:0/3556324498 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fae6c066ea0 0x7fae6c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.623+0000 7fae8da54700 1 --2- 172.21.15.46:0/3556324498 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fae6c066ea0 0x7fae6c069350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fae84009f40 tx=0x7fae840091a0 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:42.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.624+0000 7fae98fbbb80 1 Processor -- start 2024-01-26T23:46:42.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.624+0000 7fae98fbbb80 1 -- start start 2024-01-26T23:46:42.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.625+0000 7fae98fbbb80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acde010 0x55b93acde3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.625+0000 7fae98fbbb80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93ad26360 0x55b93ad08840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.625+0000 7fae98fbbb80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acdd5d0 0x55b93ad08d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.625+0000 7fae98fbbb80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93abe19f0 con 0x55b93acde010 2024-01-26T23:46:42.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.625+0000 7fae98fbbb80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93ac33130 con 0x55b93acdd5d0 2024-01-26T23:46:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.625+0000 7fae98fbbb80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b93ac30930 con 0x55b93ad26360 2024-01-26T23:46:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.625+0000 7fae8da54700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acde010 0x55b93acde3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.626+0000 7fae8da54700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acde010 0x55b93acde3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35436/0 (socket says 172.21.15.46:35436) 2024-01-26T23:46:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.626+0000 7fae8d253700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93ad26360 0x55b93ad08840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.626+0000 7fae8da54700 1 -- 172.21.15.46:0/2255247928 learned_addr learned my addr 172.21.15.46:0/2255247928 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.626+0000 7fae8e255700 1 --2- 172.21.15.46:0/2255247928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acdd5d0 0x55b93ad08d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.626+0000 7fae8d253700 1 -- 172.21.15.46:0/2255247928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acdd5d0 msgr2=0x55b93ad08d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.626+0000 7fae8d253700 1 --2- 172.21.15.46:0/2255247928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acdd5d0 0x55b93ad08d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.626+0000 7fae8d253700 1 -- 172.21.15.46:0/2255247928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acde010 msgr2=0x55b93acde3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.626+0000 7fae8d253700 1 --2- 172.21.15.46:0/2255247928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acde010 0x55b93acde3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.626+0000 7fae8d253700 1 -- 172.21.15.46:0/2255247928 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b93aa6bb10 con 0x55b93ad26360 2024-01-26T23:46:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.626+0000 7fae8d253700 1 --2- 172.21.15.46:0/2255247928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93ad26360 0x55b93ad08840 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fae78056380 tx=0x7fae78056450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db5b3904168f3a4c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.629+0000 7fae72ffd700 1 -- 172.21.15.46:0/2255247928 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae78055fe0 con 0x55b93ad26360 2024-01-26T23:46:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.629+0000 7fae72ffd700 1 -- 172.21.15.46:0/2255247928 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae78056d80 con 0x55b93ad26360 2024-01-26T23:46:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.629+0000 7fae72ffd700 1 -- 172.21.15.46:0/2255247928 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae78056680 con 0x55b93ad26360 2024-01-26T23:46:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.629+0000 7fae98fbbb80 1 -- 172.21.15.46:0/2255247928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93ad26360 msgr2=0x55b93ad08840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.629+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/2255247928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93ad26360 0x55b93ad08840 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fae78056380 tx=0x7fae78056450 comp rx=0 tx=0).stop 2024-01-26T23:46:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.629+0000 7fae98fbbb80 1 -- 172.21.15.46:0/2255247928 shutdown_connections 2024-01-26T23:46:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.629+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/2255247928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acde010 0x55b93acde3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.629+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/2255247928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acdd5d0 0x55b93ad08d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.629+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/2255247928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93ad26360 0x55b93ad08840 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.629+0000 7fae98fbbb80 1 -- 172.21.15.46:0/2255247928 >> 172.21.15.46:0/2255247928 conn(0x55b93acdda70 msgr2=0x55b93ad26820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:42.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.629+0000 7fae98fbbb80 1 -- 172.21.15.46:0/2255247928 shutdown_connections 2024-01-26T23:46:42.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.630+0000 7fae98fbbb80 1 -- 172.21.15.46:0/2255247928 wait complete. 2024-01-26T23:46:42.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.630+0000 7fae98fbbb80 1 -- start start 2024-01-26T23:46:42.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.630+0000 7fae98fbbb80 1 Processor -- start 2024-01-26T23:46:42.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae98fbbb80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acdd5d0 0x55b93ad055e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae98fbbb80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acde010 0x55b93ad22090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae98fbbb80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93ad26360 0x55b93ad0bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae98fbbb80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93ac2c720 con 0x55b93acdd5d0 2024-01-26T23:46:42.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae98fbbb80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93abd81b0 con 0x55b93acde010 2024-01-26T23:46:42.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae98fbbb80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b93abd7e50 con 0x55b93ad26360 2024-01-26T23:46:42.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae98fbbb80 10 client.0 _ll_register_callbacks cb 0x55b93aa3a7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:46:42.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae8da54700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acdd5d0 0x55b93ad055e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae8da54700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acdd5d0 0x55b93ad055e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35452/0 (socket says 172.21.15.46:35452) 2024-01-26T23:46:42.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae8da54700 1 -- 172.21.15.46:0/3672342802 learned_addr learned my addr 172.21.15.46:0/3672342802 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:42.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae8e255700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93ad26360 0x55b93ad0bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae8d253700 1 --2- 172.21.15.46:0/3672342802 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acde010 0x55b93ad22090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae8da54700 1 -- 172.21.15.46:0/3672342802 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93ad26360 msgr2=0x55b93ad0bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:42.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae8da54700 1 --2- 172.21.15.46:0/3672342802 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93ad26360 0x55b93ad0bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae8da54700 1 -- 172.21.15.46:0/3672342802 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acde010 msgr2=0x55b93ad22090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:42.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae8da54700 1 --2- 172.21.15.46:0/3672342802 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acde010 0x55b93ad22090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae8da54700 1 -- 172.21.15.46:0/3672342802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae880b2500 con 0x55b93acdd5d0 2024-01-26T23:46:42.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.632+0000 7fae8d253700 1 --2- 172.21.15.46:0/3672342802 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acde010 0x55b93ad22090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:42.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.633+0000 7fae8da54700 1 --2- 172.21.15.46:0/3672342802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acdd5d0 0x55b93ad055e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fae84014f10 tx=0x7fae84016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb569017d2c5b279 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:42.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.633+0000 7fae67fff700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:46:42.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.633+0000 7fae67fff700 1 -- 172.21.15.46:0/3672342802 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae84013b10 con 0x55b93acdd5d0 2024-01-26T23:46:42.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.633+0000 7fae67fff700 1 -- 172.21.15.46:0/3672342802 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fae84013c70 con 0x55b93acdd5d0 2024-01-26T23:46:42.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.633+0000 7fae98fbbb80 1 -- 172.21.15.46:0/3672342802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b93acd1190 con 0x55b93acdd5d0 2024-01-26T23:46:42.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.633+0000 7fae67fff700 1 -- 172.21.15.46:0/3672342802 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae84013e30 con 0x55b93acdd5d0 2024-01-26T23:46:42.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.633+0000 7fae98fbbb80 10 client.7455 fetch_fsmap learned FSMap version 54 2024-01-26T23:46:42.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.633+0000 7fae98fbbb80 1 -- 172.21.15.46:0/3672342802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) v3 -- 0x55b93abfa290 con 0x55b93acdd5d0 2024-01-26T23:46:42.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.633+0000 7fae67fff700 1 -- 172.21.15.46:0/3672342802 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b93acd1190 con 0x55b93acdd5d0 2024-01-26T23:46:42.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.633+0000 7fae67fff700 1 -- 172.21.15.46:0/3672342802 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 54) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fae84023680 con 0x55b93acdd5d0 2024-01-26T23:46:42.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.633+0000 7fae98fbbb80 10 client.7455 fetch_fsmap finished waiting for FSMap version 54 2024-01-26T23:46:42.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.633+0000 7fae98fbbb80 10 client.7455 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-26T23:46:42.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.633+0000 7fae98fbbb80 20 client.7455 populate_metadata read hostname 'smithi046' 2024-01-26T23:46:42.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.633+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/3672342802 >> [v2:172.21.15.106:6835/1643596655,v1:172.21.15.106:6837/1643596655] conn(0x55b93ad0c2b0 0x55b93ad24d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.633+0000 7fae98fbbb80 4 client.7455 mds_command: new command op to 7084 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-26T23:46:42.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.633+0000 7fae98fbbb80 1 -- 172.21.15.46:0/3672342802 --> [v2:172.21.15.106:6835/1643596655,v1:172.21.15.106:6837/1643596655] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b93abceee0 con 0x55b93ad0c2b0 2024-01-26T23:46:42.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.634+0000 7fae8d253700 1 --2- 172.21.15.46:0/3672342802 >> [v2:172.21.15.106:6835/1643596655,v1:172.21.15.106:6837/1643596655] conn(0x55b93ad0c2b0 0x55b93ad24d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.634+0000 7fae67fff700 10 client.7455 ms_handle_connect on v2:172.21.15.106:6835/1643596655 2024-01-26T23:46:42.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.634+0000 7fae8d253700 1 --2- 172.21.15.46:0/3672342802 >> [v2:172.21.15.106:6835/1643596655,v1:172.21.15.106:6837/1643596655] conn(0x55b93ad0c2b0 0x55b93ad24d40 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64ed3991110986af server_cookie=29e2f6bcbfde7640 in_seq=0 out_seq=0 2024-01-26T23:46:42.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.635+0000 7fae67fff700 1 -- 172.21.15.46:0/3672342802 <== mds.0 v2:172.21.15.106:6835/1643596655 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x55b93abceee0 con 0x55b93ad0c2b0 2024-01-26T23:46:42.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.635+0000 7fae67fff700 10 client.7455 handle_command_reply: tid=0 2024-01-26T23:46:42.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.635+0000 7fae98fbbb80 1 client.7455 shutdown 2024-01-26T23:46:42.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.635+0000 7fae98fbbb80 1 -- 172.21.15.46:0/3672342802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acdd5d0 msgr2=0x55b93ad055e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:42.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.635+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/3672342802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acdd5d0 0x55b93ad055e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fae84014f10 tx=0x7fae84016000 comp rx=0 tx=0).stop 2024-01-26T23:46:42.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.635+0000 7fae98fbbb80 1 -- 172.21.15.46:0/3672342802 shutdown_connections 2024-01-26T23:46:42.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.635+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/3672342802 >> [v2:172.21.15.106:6835/1643596655,v1:172.21.15.106:6837/1643596655] conn(0x55b93ad0c2b0 0x55b93ad24d40 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.635+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/3672342802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acdd5d0 0x55b93ad055e0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.635+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/3672342802 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93ad26360 0x55b93ad0bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.635+0000 7fae67fff700 0 client.7455 ms_handle_reset on v2:172.21.15.106:6835/1643596655 2024-01-26T23:46:42.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.635+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/3672342802 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acde010 0x55b93ad22090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.635+0000 7fae98fbbb80 1 -- 172.21.15.46:0/3672342802 >> 172.21.15.46:0/3672342802 conn(0x55b93acdda70 msgr2=0x55b93acd7240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:42.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.635+0000 7fae98fbbb80 1 -- 172.21.15.46:0/3672342802 shutdown_connections 2024-01-26T23:46:42.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.635+0000 7fae98fbbb80 1 -- 172.21.15.46:0/3672342802 wait complete. 2024-01-26T23:46:42.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.635+0000 7fae98fbbb80 20 client.7455 trim_cache size 0 max 16384 2024-01-26T23:46:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.653+0000 7fae98fbbb80 1 Processor -- start 2024-01-26T23:46:42.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.653+0000 7fae98fbbb80 1 -- start start 2024-01-26T23:46:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.654+0000 7fae98fbbb80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acdd5d0 0x55b93acd4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.654+0000 7fae98fbbb80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acde010 0x55b93ad07f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.654+0000 7fae98fbbb80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93ad26360 0x55b93ad084a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.654+0000 7fae98fbbb80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93ab24e40 con 0x55b93acdd5d0 2024-01-26T23:46:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.654+0000 7fae98fbbb80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93ad25440 con 0x55b93acde010 2024-01-26T23:46:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.654+0000 7fae98fbbb80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b93ad089e0 con 0x55b93ad26360 2024-01-26T23:46:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.654+0000 7fae8da54700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acdd5d0 0x55b93acd4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.654+0000 7fae8d253700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acde010 0x55b93ad07f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.654+0000 7fae8e255700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93ad26360 0x55b93ad084a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.654+0000 7fae8da54700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acdd5d0 0x55b93acd4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35458/0 (socket says 172.21.15.46:35458) 2024-01-26T23:46:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.654+0000 7fae8da54700 1 -- 172.21.15.46:0/2348058247 learned_addr learned my addr 172.21.15.46:0/2348058247 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.655+0000 7fae8d253700 1 -- 172.21.15.46:0/2348058247 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93ad26360 msgr2=0x55b93ad084a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.655+0000 7fae8d253700 1 --2- 172.21.15.46:0/2348058247 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93ad26360 0x55b93ad084a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.655+0000 7fae8d253700 1 -- 172.21.15.46:0/2348058247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acdd5d0 msgr2=0x55b93acd4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.655+0000 7fae8d253700 1 --2- 172.21.15.46:0/2348058247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acdd5d0 0x55b93acd4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.655+0000 7fae8d253700 1 -- 172.21.15.46:0/2348058247 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b93aac3c80 con 0x55b93acde010 2024-01-26T23:46:42.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.655+0000 7fae8e255700 1 --2- 172.21.15.46:0/2348058247 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93ad26360 0x55b93ad084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.655+0000 7fae8da54700 1 --2- 172.21.15.46:0/2348058247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acdd5d0 0x55b93acd4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.655+0000 7fae8d253700 1 --2- 172.21.15.46:0/2348058247 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acde010 0x55b93ad07f60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fae78083c30 tx=0x7fae7806a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cdacad02d08658e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.655+0000 7fae737fe700 1 -- 172.21.15.46:0/2348058247 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae78056450 con 0x55b93acde010 2024-01-26T23:46:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.655+0000 7fae737fe700 1 -- 172.21.15.46:0/2348058247 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae78055fe0 con 0x55b93acde010 2024-01-26T23:46:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.655+0000 7fae737fe700 1 -- 172.21.15.46:0/2348058247 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae780598d0 con 0x55b93acde010 2024-01-26T23:46:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.656+0000 7fae98fbbb80 1 -- 172.21.15.46:0/2348058247 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acde010 msgr2=0x55b93ad07f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.656+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/2348058247 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acde010 0x55b93ad07f60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fae78083c30 tx=0x7fae7806a040 comp rx=0 tx=0).stop 2024-01-26T23:46:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.656+0000 7fae98fbbb80 1 -- 172.21.15.46:0/2348058247 shutdown_connections 2024-01-26T23:46:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.656+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/2348058247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93acdd5d0 0x55b93acd4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.656+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/2348058247 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93ad26360 0x55b93ad084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.656+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/2348058247 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acde010 0x55b93ad07f60 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.656+0000 7fae98fbbb80 1 -- 172.21.15.46:0/2348058247 >> 172.21.15.46:0/2348058247 conn(0x55b93acdda70 msgr2=0x55b93ad24bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.656+0000 7fae98fbbb80 1 -- 172.21.15.46:0/2348058247 shutdown_connections 2024-01-26T23:46:42.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.656+0000 7fae98fbbb80 1 -- 172.21.15.46:0/2348058247 wait complete. 2024-01-26T23:46:42.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.656+0000 7fae98fbbb80 1 -- start start 2024-01-26T23:46:42.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.656+0000 7fae98fbbb80 1 Processor -- start 2024-01-26T23:46:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.657+0000 7fae98fbbb80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93acdd5d0 0x55b93ad7d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.657+0000 7fae98fbbb80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acde010 0x55b93ad35690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.657+0000 7fae98fbbb80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93ad26360 0x55b93ad07d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.657+0000 7fae98fbbb80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93ad04d60 con 0x55b93ad26360 2024-01-26T23:46:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.657+0000 7fae98fbbb80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93ad36180 con 0x55b93acde010 2024-01-26T23:46:42.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.657+0000 7fae98fbbb80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b93ad04b10 con 0x55b93acdd5d0 2024-01-26T23:46:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.657+0000 7fae98fbbb80 10 client.0 _ll_register_callbacks cb 0x55b93aa3a7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:46:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.658+0000 7fae8e255700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93ad26360 0x55b93ad07d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.658+0000 7fae8e255700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93ad26360 0x55b93ad07d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35474/0 (socket says 172.21.15.46:35474) 2024-01-26T23:46:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.658+0000 7fae8e255700 1 -- 172.21.15.46:0/1734480063 learned_addr learned my addr 172.21.15.46:0/1734480063 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.658+0000 7fae8d253700 1 --2- 172.21.15.46:0/1734480063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acde010 0x55b93ad35690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.658+0000 7fae8da54700 1 --2- 172.21.15.46:0/1734480063 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93acdd5d0 0x55b93ad7d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.658+0000 7fae8e255700 1 -- 172.21.15.46:0/1734480063 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93acdd5d0 msgr2=0x55b93ad7d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.658+0000 7fae8e255700 1 --2- 172.21.15.46:0/1734480063 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93acdd5d0 0x55b93ad7d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.658+0000 7fae8e255700 1 -- 172.21.15.46:0/1734480063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acde010 msgr2=0x55b93ad35690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.658+0000 7fae8e255700 1 --2- 172.21.15.46:0/1734480063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acde010 0x55b93ad35690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.658+0000 7fae8e255700 1 -- 172.21.15.46:0/1734480063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b93abfa290 con 0x55b93ad26360 2024-01-26T23:46:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.658+0000 7fae8e255700 1 --2- 172.21.15.46:0/1734480063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93ad26360 0x55b93ad07d40 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fae80016e20 tx=0x7fae80006040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5efbed2a5f597bf server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.658+0000 7fae67fff700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:46:42.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.658+0000 7fae67fff700 1 -- 172.21.15.46:0/1734480063 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8001b070 con 0x55b93ad26360 2024-01-26T23:46:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.658+0000 7fae67fff700 1 -- 172.21.15.46:0/1734480063 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fae80007aa0 con 0x55b93ad26360 2024-01-26T23:46:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.658+0000 7fae98fbbb80 1 -- 172.21.15.46:0/1734480063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fae880c6390 con 0x55b93ad26360 2024-01-26T23:46:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.659+0000 7fae67fff700 1 -- 172.21.15.46:0/1734480063 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae8001de10 con 0x55b93ad26360 2024-01-26T23:46:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.659+0000 7fae98fbbb80 10 client.7461 fetch_fsmap learned FSMap version 54 2024-01-26T23:46:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.659+0000 7fae98fbbb80 1 -- 172.21.15.46:0/1734480063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) v3 -- 0x7fae80001e70 con 0x55b93ad26360 2024-01-26T23:46:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.659+0000 7fae67fff700 1 -- 172.21.15.46:0/1734480063 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae880c6390 con 0x55b93ad26360 2024-01-26T23:46:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.659+0000 7fae67fff700 1 -- 172.21.15.46:0/1734480063 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 54) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fae80012070 con 0x55b93ad26360 2024-01-26T23:46:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.659+0000 7fae98fbbb80 10 client.7461 fetch_fsmap finished waiting for FSMap version 54 2024-01-26T23:46:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.659+0000 7fae98fbbb80 10 client.7461 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-26T23:46:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.659+0000 7fae98fbbb80 20 client.7461 populate_metadata read hostname 'smithi046' 2024-01-26T23:46:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.659+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/1734480063 >> [v2:172.21.15.106:6835/1643596655,v1:172.21.15.106:6837/1643596655] conn(0x55b93acdbf80 0x55b93acdc350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.659+0000 7fae98fbbb80 4 client.7461 mds_command: new command op to 7084 tid=0[{"prefix": "status"}] 2024-01-26T23:46:42.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.659+0000 7fae98fbbb80 1 -- 172.21.15.46:0/1734480063 --> [v2:172.21.15.106:6835/1643596655,v1:172.21.15.106:6837/1643596655] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fae78056d80 con 0x55b93acdbf80 2024-01-26T23:46:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.659+0000 7fae8da54700 1 --2- 172.21.15.46:0/1734480063 >> [v2:172.21.15.106:6835/1643596655,v1:172.21.15.106:6837/1643596655] conn(0x55b93acdbf80 0x55b93acdc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.659+0000 7fae67fff700 10 client.7461 ms_handle_connect on v2:172.21.15.106:6835/1643596655 2024-01-26T23:46:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.659+0000 7fae8da54700 1 --2- 172.21.15.46:0/1734480063 >> [v2:172.21.15.106:6835/1643596655,v1:172.21.15.106:6837/1643596655] conn(0x55b93acdbf80 0x55b93acdc350 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=340c41e9514914e server_cookie=1094210897bccfc in_seq=0 out_seq=0 2024-01-26T23:46:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.660+0000 7fae67fff700 1 -- 172.21.15.46:0/1734480063 <== mds.0 v2:172.21.15.106:6835/1643596655 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+320 (crc 0 0 0) 0x7fae78056d80 con 0x55b93acdbf80 2024-01-26T23:46:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.660+0000 7fae67fff700 10 client.7461 handle_command_reply: tid=0 2024-01-26T23:46:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.660+0000 7fae98fbbb80 1 client.7461 shutdown 2024-01-26T23:46:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.660+0000 7fae98fbbb80 1 -- 172.21.15.46:0/1734480063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93ad26360 msgr2=0x55b93ad07d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.660+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/1734480063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93ad26360 0x55b93ad07d40 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fae80016e20 tx=0x7fae80006040 comp rx=0 tx=0).stop 2024-01-26T23:46:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.660+0000 7fae98fbbb80 1 -- 172.21.15.46:0/1734480063 shutdown_connections 2024-01-26T23:46:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.660+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/1734480063 >> [v2:172.21.15.106:6835/1643596655,v1:172.21.15.106:6837/1643596655] conn(0x55b93acdbf80 0x55b93acdc350 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.660+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/1734480063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b93ad26360 0x55b93ad07d40 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.660+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/1734480063 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55b93acdd5d0 0x55b93ad7d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.660+0000 7fae67fff700 0 client.7461 ms_handle_reset on v2:172.21.15.106:6835/1643596655 2024-01-26T23:46:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.660+0000 7fae98fbbb80 1 --2- 172.21.15.46:0/1734480063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b93acde010 0x55b93ad35690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.660+0000 7fae98fbbb80 1 -- 172.21.15.46:0/1734480063 >> 172.21.15.46:0/1734480063 conn(0x55b93acdda70 msgr2=0x55b93ad22c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.660+0000 7fae98fbbb80 1 -- 172.21.15.46:0/1734480063 shutdown_connections 2024-01-26T23:46:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.660+0000 7fae98fbbb80 1 -- 172.21.15.46:0/1734480063 wait complete. 2024-01-26T23:46:42.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.660+0000 7fae98fbbb80 20 client.7461 trim_cache size 0 max 16384 2024-01-26T23:46:42.667 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:46:42.667 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-26T23:46:42.667 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-26T23:46:42.667 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 7084, 2024-01-26T23:46:42.667 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-26T23:46:42.667 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-26T23:46:42.667 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-01-26T23:46:42.668 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 10.896856837, 2024-01-26T23:46:42.668 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 54, 2024-01-26T23:46:42.668 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 112, 2024-01-26T23:46:42.668 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 112, 2024-01-26T23:46:42.668 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 15.92774818 2024-01-26T23:46:42.668 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:46:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.661+0000 7fae8fcb8700 1 -- 172.21.15.46:0/3556324498 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fae6c066ea0 msgr2=0x7fae6c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.661+0000 7fae8fcb8700 1 --2- 172.21.15.46:0/3556324498 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fae6c066ea0 0x7fae6c069350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fae84009f40 tx=0x7fae840091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.661+0000 7fae8fcb8700 1 -- 172.21.15.46:0/3556324498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae88159960 msgr2=0x7fae880c6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.661+0000 7fae8fcb8700 1 --2- 172.21.15.46:0/3556324498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae88159960 0x7fae880c6d40 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fae8806f390 tx=0x7fae78017000 comp rx=0 tx=0).stop 2024-01-26T23:46:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.661+0000 7fae8fcb8700 1 -- 172.21.15.46:0/3556324498 shutdown_connections 2024-01-26T23:46:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.661+0000 7fae8fcb8700 1 --2- 172.21.15.46:0/3556324498 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fae6c066ea0 0x7fae6c069350 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.661+0000 7fae8fcb8700 1 --2- 172.21.15.46:0/3556324498 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae8815d8f0 0x7fae8814a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.661+0000 7fae8fcb8700 1 --2- 172.21.15.46:0/3556324498 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae88159590 0x7fae88069a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.661+0000 7fae8fcb8700 1 --2- 172.21.15.46:0/3556324498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae88159960 0x7fae880c6d40 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.661+0000 7fae8fcb8700 1 -- 172.21.15.46:0/3556324498 >> 172.21.15.46:0/3556324498 conn(0x7fae880ce530 msgr2=0x7fae88156660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.661+0000 7fae8fcb8700 1 -- 172.21.15.46:0/3556324498 shutdown_connections 2024-01-26T23:46:42.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.661+0000 7fae8fcb8700 1 -- 172.21.15.46:0/3556324498 wait complete. 2024-01-26T23:46:42.677 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-26T23:46:42.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.812+0000 7f0e8e24f700 1 Processor -- start 2024-01-26T23:46:42.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.812+0000 7f0e8e24f700 1 -- start start 2024-01-26T23:46:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.813+0000 7f0e8e24f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e88157340 0x7f0e8815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.813+0000 7f0e8e24f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e880ae520 0x7f0e880b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.813+0000 7f0e8e24f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e88158ae0 con 0x7f0e880b3fb0 2024-01-26T23:46:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.813+0000 7f0e8e24f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e880b4430 con 0x7f0e880ae520 2024-01-26T23:46:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.813+0000 7f0e8e24f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8815b650 con 0x7f0e88157340 2024-01-26T23:46:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.813+0000 7f0e86ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e880ae520 0x7f0e880b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.813+0000 7f0e86ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e880ae520 0x7f0e880b3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35488/0 (socket says 172.21.15.46:35488) 2024-01-26T23:46:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.813+0000 7f0e86ffd700 1 -- 172.21.15.46:0/2443614211 learned_addr learned my addr 172.21.15.46:0/2443614211 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.813+0000 7f0e877fe700 1 --2- 172.21.15.46:0/2443614211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e88157340 0x7f0e8815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.813+0000 7f0e86ffd700 1 -- 172.21.15.46:0/2443614211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e88157340 msgr2=0x7f0e8815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.813+0000 7f0e86ffd700 1 --2- 172.21.15.46:0/2443614211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e88157340 0x7f0e8815bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.813+0000 7f0e86ffd700 1 -- 172.21.15.46:0/2443614211 >> v1:172.21.15.106:6789/0 conn(0x7f0e880b3fb0 legacy=0x7f0e88157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.813+0000 7f0e86ffd700 1 -- 172.21.15.46:0/2443614211 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8815c020 con 0x7f0e880ae520 2024-01-26T23:46:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.813+0000 7f0e86ffd700 1 --2- 172.21.15.46:0/2443614211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880ae520 0x7f0e880b3a70 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c0099b0 tx=0x7f0e7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7537eadf81c793 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.814+0000 7f0e867fc700 1 -- 172.21.15.46:0/2443614211 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c00b860 con 0x7f0e880ae520 2024-01-26T23:46:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.814+0000 7f0e867fc700 1 -- 172.21.15.46:0/2443614211 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7c00b9c0 con 0x7f0e880ae520 2024-01-26T23:46:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.814+0000 7f0e867fc700 1 -- 172.21.15.46:0/2443614211 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c00bd80 con 0x7f0e880ae520 2024-01-26T23:46:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.814+0000 7f0e8e24f700 1 -- 172.21.15.46:0/2443614211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880ae520 msgr2=0x7f0e880b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.814+0000 7f0e8e24f700 1 --2- 172.21.15.46:0/2443614211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880ae520 0x7f0e880b3a70 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c0099b0 tx=0x7f0e7c009250 comp rx=0 tx=0).stop 2024-01-26T23:46:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.814+0000 7f0e8e24f700 1 -- 172.21.15.46:0/2443614211 shutdown_connections 2024-01-26T23:46:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.814+0000 7f0e8e24f700 1 --2- 172.21.15.46:0/2443614211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e88157340 0x7f0e8815bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.814+0000 7f0e8e24f700 1 --2- 172.21.15.46:0/2443614211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880ae520 0x7f0e880b3a70 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.814+0000 7f0e8e24f700 1 -- 172.21.15.46:0/2443614211 >> 172.21.15.46:0/2443614211 conn(0x7f0e880ce530 msgr2=0x7f0e880ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.814+0000 7f0e8e24f700 1 -- 172.21.15.46:0/2443614211 shutdown_connections 2024-01-26T23:46:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.814+0000 7f0e8e24f700 1 -- 172.21.15.46:0/2443614211 wait complete. 2024-01-26T23:46:42.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.816+0000 7f0e8e24f700 1 Processor -- start 2024-01-26T23:46:42.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.816+0000 7f0e8e24f700 1 -- start start 2024-01-26T23:46:42.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.817+0000 7f0e8e24f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880ae520 0x7f0e880b2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.817+0000 7f0e8e24f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e880b3fb0 0x7f0e880acd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.817+0000 7f0e8e24f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e88157340 0x7f0e881684b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.817+0000 7f0e8e24f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e880b2490 con 0x7f0e880ae520 2024-01-26T23:46:42.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.817+0000 7f0e8e24f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e880ad280 con 0x7f0e88157340 2024-01-26T23:46:42.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.817+0000 7f0e8e24f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e880b1880 con 0x7f0e880b3fb0 2024-01-26T23:46:42.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.817+0000 7f0e877fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880ae520 0x7f0e880b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.817+0000 7f0e877fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880ae520 0x7f0e880b2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35492/0 (socket says 172.21.15.46:35492) 2024-01-26T23:46:42.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.817+0000 7f0e877fe700 1 -- 172.21.15.46:0/4200909542 learned_addr learned my addr 172.21.15.46:0/4200909542 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:42.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.817+0000 7f0e86ffd700 1 --2- 172.21.15.46:0/4200909542 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e880b3fb0 0x7f0e880acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.817+0000 7f0e87fff700 1 --2- 172.21.15.46:0/4200909542 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e88157340 0x7f0e881684b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.817+0000 7f0e877fe700 1 -- 172.21.15.46:0/4200909542 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e880b3fb0 msgr2=0x7f0e880acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:42.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.817+0000 7f0e877fe700 1 --2- 172.21.15.46:0/4200909542 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e880b3fb0 0x7f0e880acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.817+0000 7f0e877fe700 1 -- 172.21.15.46:0/4200909542 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e88157340 msgr2=0x7f0e881684b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:42.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.817+0000 7f0e877fe700 1 --2- 172.21.15.46:0/4200909542 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e88157340 0x7f0e881684b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:42.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.817+0000 7f0e877fe700 1 -- 172.21.15.46:0/4200909542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7c0094f0 con 0x7f0e880ae520 2024-01-26T23:46:42.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.817+0000 7f0e877fe700 1 --2- 172.21.15.46:0/4200909542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880ae520 0x7f0e880b2920 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0e7800d860 tx=0x7f0e78008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:42.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.818+0000 7f0e6f7fe700 1 -- 172.21.15.46:0/4200909542 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e780125f0 con 0x7f0e880ae520 2024-01-26T23:46:42.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.818+0000 7f0e6f7fe700 1 -- 172.21.15.46:0/4200909542 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e78012750 con 0x7f0e880ae520 2024-01-26T23:46:42.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.818+0000 7f0e6f7fe700 1 -- 172.21.15.46:0/4200909542 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e78012b10 con 0x7f0e880ae520 2024-01-26T23:46:42.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.818+0000 7f0e8e24f700 1 -- 172.21.15.46:0/4200909542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e880b1b60 con 0x7f0e880ae520 2024-01-26T23:46:42.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.818+0000 7f0e8e24f700 1 -- 172.21.15.46:0/4200909542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e880b2120 con 0x7f0e880ae520 2024-01-26T23:46:42.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.819+0000 7f0e8e24f700 1 -- 172.21.15.46:0/4200909542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e880cacf0 con 0x7f0e880ae520 2024-01-26T23:46:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.819+0000 7f0e6f7fe700 1 -- 172.21.15.46:0/4200909542 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f0e7801e460 con 0x7f0e880ae520 2024-01-26T23:46:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.820+0000 7f0e6f7fe700 1 --2- 172.21.15.46:0/4200909542 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0e74066ea0 0x7f0e74069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.820+0000 7f0e6f7fe700 1 -- 172.21.15.46:0/4200909542 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f0e78084bd0 con 0x7f0e880ae520 2024-01-26T23:46:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.820+0000 7f0e86ffd700 1 --2- 172.21.15.46:0/4200909542 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0e74066ea0 0x7f0e74069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.820+0000 7f0e86ffd700 1 --2- 172.21.15.46:0/4200909542 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0e74066ea0 0x7f0e74069350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c0099b0 tx=0x7f0e7c017000 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:42.824+0000 7f0e6f7fe700 1 -- 172.21.15.46:0/4200909542 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0e78051c50 con 0x7f0e880ae520 2024-01-26T23:46:43.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:43.107+0000 7f0e8e24f700 1 -- 172.21.15.46:0/4200909542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0e8804c710 con 0x7f0e880ae520 2024-01-26T23:46:43.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:43.107+0000 7f0e6f7fe700 1 -- 172.21.15.46:0/4200909542 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v54) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0e78055270 con 0x7f0e880ae520 2024-01-26T23:46:43.109 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-26T23:46:43.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:43.110+0000 7f0e8e24f700 1 -- 172.21.15.46:0/4200909542 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0e74066ea0 msgr2=0x7f0e74069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:43.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:43.110+0000 7f0e8e24f700 1 --2- 172.21.15.46:0/4200909542 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0e74066ea0 0x7f0e74069350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c0099b0 tx=0x7f0e7c017000 comp rx=0 tx=0).stop 2024-01-26T23:46:43.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:43.110+0000 7f0e8e24f700 1 -- 172.21.15.46:0/4200909542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880ae520 msgr2=0x7f0e880b2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:43.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:43.110+0000 7f0e8e24f700 1 --2- 172.21.15.46:0/4200909542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880ae520 0x7f0e880b2920 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0e7800d860 tx=0x7f0e78008040 comp rx=0 tx=0).stop 2024-01-26T23:46:43.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:43.110+0000 7f0e8e24f700 1 -- 172.21.15.46:0/4200909542 shutdown_connections 2024-01-26T23:46:43.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:43.110+0000 7f0e8e24f700 1 --2- 172.21.15.46:0/4200909542 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0e74066ea0 0x7f0e74069350 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:43.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:43.110+0000 7f0e8e24f700 1 --2- 172.21.15.46:0/4200909542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880ae520 0x7f0e880b2920 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:43.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:43.110+0000 7f0e8e24f700 1 --2- 172.21.15.46:0/4200909542 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e88157340 0x7f0e881684b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:43.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:43.110+0000 7f0e8e24f700 1 --2- 172.21.15.46:0/4200909542 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e880b3fb0 0x7f0e880acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:43.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:43.110+0000 7f0e8e24f700 1 -- 172.21.15.46:0/4200909542 >> 172.21.15.46:0/4200909542 conn(0x7f0e880ce530 msgr2=0x7f0e8815a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:43.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:43.110+0000 7f0e8e24f700 1 -- 172.21.15.46:0/4200909542 shutdown_connections 2024-01-26T23:46:43.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:43.110+0000 7f0e8e24f700 1 -- 172.21.15.46:0/4200909542 wait complete. 2024-01-26T23:46:43.125 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-26T23:46:43.125 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-26T23:46:43.125 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-26T23:46:43.125 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-26T23:46:43.125 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-26T23:46:43.125 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-26T23:46:43.125 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-26T23:46:43.125 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-26T23:46:43.125 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:46:43.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:46:43.126 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-26T23:46:43.145 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-26T23:46:43.146 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-26T23:46:43.146 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-26T23:46:43.146 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:46:43.146 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-26T23:46:43.146 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:46:43.146 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:46:43.146 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:46:43.146 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-26T23:46:43.146 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:46:43.146 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:46:43.146 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-26T23:46:43.146 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-26T23:46:43.146 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:46:43.146 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-26T23:46:43.147 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-26T23:46:43.147 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 41327sec preferred_lft 41327sec 2024-01-26T23:46:43.147 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-26T23:46:43.147 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:46:43.147 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:46:43.147 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:46:43.147 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-26T23:46:43.147 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-26T23:46:43.147 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:46:43.147 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-26T23:46:43.147 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:46:43.147 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::cce6:a7ff:fe4f:38d/64 scope link 2024-01-26T23:46:43.147 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:46:43.147 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:46:43.148 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 76:ef:90:4c:9d:6f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-26T23:46:43.148 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::74ef:90ff:fe4c:9d6f/64 scope link 2024-01-26T23:46:43.148 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:46:43.148 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:46:43.148 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-26T23:46:43.148 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::2c73:afff:fef0:6f39/64 scope link 2024-01-26T23:46:43.148 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:46:43.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:46:43.148 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-26T23:46:43.217 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-26T23:46:43.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:46:43.219 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-26T23:46:43.240 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:43.239+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:43.284 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-26T23:46:43.284 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-26T23:46:43.284 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-26T23:46:43.284 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-26T23:46:43.284 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-26T23:46:43.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:46:43.284 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:46:43.284 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:46:43.284 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-26T23:46:43.285 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-26T23:46:43.285 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-26T23:46:43.285 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:46:43.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:46:43.410 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-26T23:46:43.430 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-26T23:46:43.430 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-26T23:46:43.430 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:46:43.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:46:43.430 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-01-26T23:46:43.504 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-26T23:46:43.505 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:46:43.505 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-26T23:46:43.566 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:46:43.566 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:46:43.566 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:46:43.628 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:46:43.705 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:46:43.705 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:46:43.781 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:46:43.782 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:46:43.783 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:46:43.783 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:46:43.845 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-26T23:46:43.845 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-26T23:46:43.887 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:46:43.923 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:46:43.923 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:46:43.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-01-26T23:46:43.991+0000 7ffa9cdcd540 -1 init, newargv = 0x55f6b65daf70 newargc=15 2024-01-26T23:46:43.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[79483]: starting ceph client 2024-01-26T23:46:43.995 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:46:43.996 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:46:43.997 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:46:43.997 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:46:43.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[79483]: starting fuse 2024-01-26T23:46:44.017 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-26T23:46:44.017 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-26T23:46:44.017 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-26T23:46:44.017 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-26T23:46:44.017 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-26T23:46:44.017 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-26T23:46:44.017 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-26T23:46:44.017 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:46:44.017 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-26T23:46:44.017 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-26T23:46:44.017 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-26T23:46:44.018 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-26T23:46:44.018 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:46:44.018 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-26T23:46:44.018 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-26T23:46:44.018 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-26T23:46:44.018 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-26T23:46:44.018 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:46:44.018 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-26T23:46:44.018 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-26T23:46:44.018 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-26T23:46:44.018 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-26T23:46:44.018 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-26T23:46:44.018 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-26T23:46:44.018 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-26T23:46:44.018 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-26T23:46:44.018 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:46:44.018 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-01-26T23:46:44.019 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-26T23:46:44.150 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:44.149+0000 7faf6d416700 -1 client.6991: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6859:b859190d-bc6b-49af-80e0-ed2c89f765c6:v1:172.21.15.106:0/1323704418] 2024-01-26T23:46:44.153 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.79483.asok 2024-01-26T23:46:44.334 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.79483.asok 2024-01-26T23:46:44.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:46:44.335 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.79483.asok status 2024-01-26T23:46:44.509 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:46:44.509 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-26T23:46:44.510 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-26T23:46:44.510 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-26T23:46:44.510 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-01-26T23:46:44.510 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-26T23:46:44.510 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-26T23:46:44.510 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "79483", 2024-01-26T23:46:44.510 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-26T23:46:44.510 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:44.510 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-26T23:46:44.510 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-26T23:46:44.510 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 7264, 2024-01-26T23:46:44.510 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-26T23:46:44.511 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-26T23:46:44.511 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-26T23:46:44.511 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 7264 2024-01-26T23:46:44.511 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:44.511 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:46:44.511 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:46:44.511 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:46:44.511 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3010173125 2024-01-26T23:46:44.511 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:44.511 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:44.511 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:46:44.511 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:46:44.511 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:46:44.511 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3010173125 2024-01-26T23:46:44.512 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:46:44.512 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.7264 192.168.0.1:0/3010173125", 2024-01-26T23:46:44.512 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/3010173125", 2024-01-26T23:46:44.512 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-26T23:46:44.512 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 54, 2024-01-26T23:46:44.512 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 112, 2024-01-26T23:46:44.512 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-26T23:46:44.512 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-26T23:46:44.512 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-26T23:46:44.512 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:46:44.521 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:46:44.521 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-26T23:46:44.521 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-26T23:46:44.540 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-26T23:46:44.540 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-26T23:46:44.541 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:46:44.541 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:46:44.614 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-26T23:46:44.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.791+0000 7f180c9eb700 1 Processor -- start 2024-01-26T23:46:44.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.791+0000 7f180c9eb700 1 -- start start 2024-01-26T23:46:44.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.792+0000 7f180c9eb700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1808159590 0x7f180815d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:44.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.792+0000 7f180c9eb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f180806c3c0 0x7f1808069370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:44.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.792+0000 7f180c9eb700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f180815dd90 con 0x7f180815d8f0 2024-01-26T23:46:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.792+0000 7f180c9eb700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f180815e350 con 0x7f180806c3c0 2024-01-26T23:46:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.792+0000 7f180c9eb700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18080c97c0 con 0x7f1808159590 2024-01-26T23:46:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.792+0000 7f1806d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f180806c3c0 0x7f1808069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.792+0000 7f1806d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f180806c3c0 0x7f1808069370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35500/0 (socket says 172.21.15.46:35500) 2024-01-26T23:46:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.792+0000 7f180659c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1808159590 0x7f180815d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.792+0000 7f1806d9d700 1 -- 172.21.15.46:0/3872497883 learned_addr learned my addr 172.21.15.46:0/3872497883 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.792+0000 7f1806d9d700 1 -- 172.21.15.46:0/3872497883 >> v2:172.21.15.106:3300/0 conn(0x7f1808159590 msgr2=0x7f180815d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.792+0000 7f1806d9d700 1 --2- 172.21.15.46:0/3872497883 >> v2:172.21.15.106:3300/0 conn(0x7f1808159590 0x7f180815d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.792+0000 7f1806d9d700 1 -- 172.21.15.46:0/3872497883 >> v1:172.21.15.106:6789/0 conn(0x7f180815d8f0 legacy=0x7f180806bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.792+0000 7f180559a700 1 -- 172.21.15.46:0/3872497883 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349168060 0 0) 0x7f180815dd90 con 0x7f180815d8f0 2024-01-26T23:46:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.792+0000 7f1806d9d700 1 -- 172.21.15.46:0/3872497883 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180815e070 con 0x7f180806c3c0 2024-01-26T23:46:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.792+0000 7f1806d9d700 1 --2- 172.21.15.46:0/3872497883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f180806c3c0 0x7f1808069370 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f18080c1f60 tx=0x7f1800009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dbcc655b20fb704 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.793+0000 7f180559a700 1 -- 172.21.15.46:0/3872497883 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180000e070 con 0x7f180806c3c0 2024-01-26T23:46:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.793+0000 7f180559a700 1 -- 172.21.15.46:0/3872497883 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180000b990 con 0x7f180806c3c0 2024-01-26T23:46:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.793+0000 7f180559a700 1 -- 172.21.15.46:0/3872497883 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18000092e0 con 0x7f180806c3c0 2024-01-26T23:46:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.793+0000 7f180c9eb700 1 -- 172.21.15.46:0/3872497883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f180806c3c0 msgr2=0x7f1808069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.793+0000 7f180c9eb700 1 --2- 172.21.15.46:0/3872497883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f180806c3c0 0x7f1808069370 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f18080c1f60 tx=0x7f1800009840 comp rx=0 tx=0).stop 2024-01-26T23:46:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.793+0000 7f180c9eb700 1 -- 172.21.15.46:0/3872497883 shutdown_connections 2024-01-26T23:46:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.793+0000 7f180c9eb700 1 --2- 172.21.15.46:0/3872497883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f180806c3c0 0x7f1808069370 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.793+0000 7f180c9eb700 1 --2- 172.21.15.46:0/3872497883 >> v2:172.21.15.106:3300/0 conn(0x7f1808159590 0x7f180815d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.793+0000 7f180c9eb700 1 -- 172.21.15.46:0/3872497883 >> 172.21.15.46:0/3872497883 conn(0x7f18080ce530 msgr2=0x7f18080ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.793+0000 7f180c9eb700 1 -- 172.21.15.46:0/3872497883 shutdown_connections 2024-01-26T23:46:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.793+0000 7f180c9eb700 1 -- 172.21.15.46:0/3872497883 wait complete. 2024-01-26T23:46:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.794+0000 7f180c9eb700 1 Processor -- start 2024-01-26T23:46:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.795+0000 7f180c9eb700 1 -- start start 2024-01-26T23:46:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.795+0000 7f180c9eb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f180806c3c0 0x7f180815c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.795+0000 7f180c9eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1808159590 0x7f18080b2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.795+0000 7f180c9eb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f180815d8f0 0x7f18080c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.795+0000 7f180c9eb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18080b2270 con 0x7f1808159590 2024-01-26T23:46:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.795+0000 7f180c9eb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180815cf10 con 0x7f180815d8f0 2024-01-26T23:46:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.795+0000 7f180c9eb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18080b1660 con 0x7f180806c3c0 2024-01-26T23:46:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.795+0000 7f1805d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1808159590 0x7f18080b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.796+0000 7f1805d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1808159590 0x7f18080b2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35514/0 (socket says 172.21.15.46:35514) 2024-01-26T23:46:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.796+0000 7f1805d9b700 1 -- 172.21.15.46:0/2188017712 learned_addr learned my addr 172.21.15.46:0/2188017712 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.796+0000 7f180659c700 1 --2- 172.21.15.46:0/2188017712 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f180806c3c0 0x7f180815c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.796+0000 7f1806d9d700 1 --2- 172.21.15.46:0/2188017712 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f180815d8f0 0x7f18080c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.796+0000 7f1805d9b700 1 -- 172.21.15.46:0/2188017712 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f180806c3c0 msgr2=0x7f180815c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.796+0000 7f1805d9b700 1 --2- 172.21.15.46:0/2188017712 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f180806c3c0 0x7f180815c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.796+0000 7f1805d9b700 1 -- 172.21.15.46:0/2188017712 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f180815d8f0 msgr2=0x7f18080c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.796+0000 7f1805d9b700 1 --2- 172.21.15.46:0/2188017712 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f180815d8f0 0x7f18080c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.796+0000 7f1805d9b700 1 -- 172.21.15.46:0/2188017712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1800009ae0 con 0x7f1808159590 2024-01-26T23:46:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.796+0000 7f1805d9b700 1 --2- 172.21.15.46:0/2188017712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1808159590 0x7f18080b2700 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f17fc00c590 tx=0x7f17fc007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:44.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.796+0000 7f17f67fc700 1 -- 172.21.15.46:0/2188017712 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17fc011590 con 0x7f1808159590 2024-01-26T23:46:44.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.796+0000 7f17f67fc700 1 -- 172.21.15.46:0/2188017712 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17fc011710 con 0x7f1808159590 2024-01-26T23:46:44.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.796+0000 7f180c9eb700 1 -- 172.21.15.46:0/2188017712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18080b1940 con 0x7f1808159590 2024-01-26T23:46:44.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.796+0000 7f17f67fc700 1 -- 172.21.15.46:0/2188017712 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17fc011ad0 con 0x7f1808159590 2024-01-26T23:46:44.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.796+0000 7f180c9eb700 1 -- 172.21.15.46:0/2188017712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18080b1f00 con 0x7f1808159590 2024-01-26T23:46:44.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.797+0000 7f180c9eb700 1 -- 172.21.15.46:0/2188017712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f180804c710 con 0x7f1808159590 2024-01-26T23:46:44.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.798+0000 7f17f67fc700 1 -- 172.21.15.46:0/2188017712 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f17fc01b3b0 con 0x7f1808159590 2024-01-26T23:46:44.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.798+0000 7f17f67fc700 1 --2- 172.21.15.46:0/2188017712 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f17e8066ea0 0x7f17e8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:44.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.798+0000 7f17f67fc700 1 -- 172.21.15.46:0/2188017712 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f17fc083cf0 con 0x7f1808159590 2024-01-26T23:46:44.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.798+0000 7f180659c700 1 --2- 172.21.15.46:0/2188017712 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f17e8066ea0 0x7f17e8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:44.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.799+0000 7f180659c700 1 --2- 172.21.15.46:0/2188017712 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f17e8066ea0 0x7f17e8069350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f17f8009d40 tx=0x7f17f8008040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:44.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:44.802+0000 7f17f67fc700 1 -- 172.21.15.46:0/2188017712 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f17fc050d70 con 0x7f1808159590 2024-01-26T23:46:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.039+0000 7f180c9eb700 1 -- 172.21.15.46:0/2188017712 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f1808068b70 con 0x7f17e8066ea0 2024-01-26T23:46:45.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.041+0000 7f17f67fc700 1 -- 172.21.15.46:0/2188017712 <== mgr.6365 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f1808068b70 con 0x7f17e8066ea0 2024-01-26T23:46:45.042 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-26T23:46:45.042 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:46:45.042 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephfs" 2024-01-26T23:46:45.042 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:46:45.042 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-26T23:46:45.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.043+0000 7f180c9eb700 1 -- 172.21.15.46:0/2188017712 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f17e8066ea0 msgr2=0x7f17e8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:45.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.043+0000 7f180c9eb700 1 --2- 172.21.15.46:0/2188017712 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f17e8066ea0 0x7f17e8069350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f17f8009d40 tx=0x7f17f8008040 comp rx=0 tx=0).stop 2024-01-26T23:46:45.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.043+0000 7f180c9eb700 1 -- 172.21.15.46:0/2188017712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1808159590 msgr2=0x7f18080b2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:45.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.043+0000 7f180c9eb700 1 --2- 172.21.15.46:0/2188017712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1808159590 0x7f18080b2700 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f17fc00c590 tx=0x7f17fc007f50 comp rx=0 tx=0).stop 2024-01-26T23:46:45.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.043+0000 7f180c9eb700 1 -- 172.21.15.46:0/2188017712 shutdown_connections 2024-01-26T23:46:45.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.043+0000 7f180c9eb700 1 --2- 172.21.15.46:0/2188017712 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f17e8066ea0 0x7f17e8069350 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.043+0000 7f180c9eb700 1 --2- 172.21.15.46:0/2188017712 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f180815d8f0 0x7f18080c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.043+0000 7f180c9eb700 1 --2- 172.21.15.46:0/2188017712 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f180806c3c0 0x7f180815c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.043+0000 7f180c9eb700 1 --2- 172.21.15.46:0/2188017712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1808159590 0x7f18080b2700 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.043+0000 7f180c9eb700 1 -- 172.21.15.46:0/2188017712 >> 172.21.15.46:0/2188017712 conn(0x7f18080ce530 msgr2=0x7f18080cb830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:45.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.043+0000 7f180c9eb700 1 -- 172.21.15.46:0/2188017712 shutdown_connections 2024-01-26T23:46:45.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.043+0000 7f180c9eb700 1 -- 172.21.15.46:0/2188017712 wait complete. 2024-01-26T23:46:45.057 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-26T23:46:45.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.190+0000 7f43061ab700 1 Processor -- start 2024-01-26T23:46:45.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.190+0000 7f43061ab700 1 -- start start 2024-01-26T23:46:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.191+0000 7f43061ab700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f43000c8da0 0x7f43000c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.191+0000 7f43061ab700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f43000caf30 0x7f43000cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.191+0000 7f43061ab700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43000ccc00 con 0x7f4300140af0 2024-01-26T23:46:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.191+0000 7f43061ab700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43000cbbf0 con 0x7f43000c8da0 2024-01-26T23:46:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.191+0000 7f43061ab700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43000cc310 con 0x7f43000caf30 2024-01-26T23:46:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.191+0000 7f42ff7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f43000c8da0 0x7f43000c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.191+0000 7f42feffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4300140af0 0x7f4300140f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34920/0 (socket says 172.21.15.46:34920) 2024-01-26T23:46:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.191+0000 7f42feffd700 1 -- 172.21.15.46:0/3708471250 learned_addr learned my addr 172.21.15.46:0/3708471250 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.191+0000 7f42fffff700 1 --2- 172.21.15.46:0/3708471250 >> v2:172.21.15.106:3300/0 conn(0x7f43000caf30 0x7f43000cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.191+0000 7f42ff7fe700 1 -- 172.21.15.46:0/3708471250 >> v2:172.21.15.106:3300/0 conn(0x7f43000caf30 msgr2=0x7f43000cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.191+0000 7f42ff7fe700 1 --2- 172.21.15.46:0/3708471250 >> v2:172.21.15.106:3300/0 conn(0x7f43000caf30 0x7f43000cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.191+0000 7f42ff7fe700 1 -- 172.21.15.46:0/3708471250 >> v1:172.21.15.46:6789/0 conn(0x7f4300140af0 legacy=0x7f4300140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.191+0000 7f42ff7fe700 1 -- 172.21.15.46:0/3708471250 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43000cc5b0 con 0x7f43000c8da0 2024-01-26T23:46:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.191+0000 7f42fe7fc700 1 -- 172.21.15.46:0/3708471250 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824203771 0 0) 0x7f43000ccc00 con 0x7f4300140af0 2024-01-26T23:46:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.192+0000 7f42ff7fe700 1 --2- 172.21.15.46:0/3708471250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43000c8da0 0x7f43000c9170 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f43001416d0 tx=0x7f42f00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b92d6795c6d224 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.192+0000 7f42fe7fc700 1 -- 172.21.15.46:0/3708471250 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f000b6b0 con 0x7f43000c8da0 2024-01-26T23:46:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.192+0000 7f42fe7fc700 1 -- 172.21.15.46:0/3708471250 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f000b810 con 0x7f43000c8da0 2024-01-26T23:46:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.192+0000 7f42fe7fc700 1 -- 172.21.15.46:0/3708471250 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f000bbb0 con 0x7f43000c8da0 2024-01-26T23:46:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.192+0000 7f43061ab700 1 -- 172.21.15.46:0/3708471250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43000c8da0 msgr2=0x7f43000c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.192+0000 7f43061ab700 1 --2- 172.21.15.46:0/3708471250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43000c8da0 0x7f43000c9170 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f43001416d0 tx=0x7f42f00091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.192+0000 7f43061ab700 1 -- 172.21.15.46:0/3708471250 shutdown_connections 2024-01-26T23:46:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.192+0000 7f43061ab700 1 --2- 172.21.15.46:0/3708471250 >> v2:172.21.15.106:3300/0 conn(0x7f43000caf30 0x7f43000cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.192+0000 7f43061ab700 1 --2- 172.21.15.46:0/3708471250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43000c8da0 0x7f43000c9170 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.192+0000 7f43061ab700 1 -- 172.21.15.46:0/3708471250 >> 172.21.15.46:0/3708471250 conn(0x7f430006bd90 msgr2=0x7f430006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.193+0000 7f43061ab700 1 -- 172.21.15.46:0/3708471250 shutdown_connections 2024-01-26T23:46:45.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.193+0000 7f43061ab700 1 -- 172.21.15.46:0/3708471250 wait complete. 2024-01-26T23:46:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.194+0000 7f43061ab700 1 Processor -- start 2024-01-26T23:46:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.194+0000 7f43061ab700 1 -- start start 2024-01-26T23:46:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f43061ab700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43000c8da0 0x7f430013c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f43061ab700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43000caf30 0x7f430016a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f43061ab700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4300140af0 0x7f430016aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f43061ab700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430013ce60 con 0x7f43000c8da0 2024-01-26T23:46:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f43061ab700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43000b39c0 con 0x7f43000caf30 2024-01-26T23:46:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f43061ab700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43000b3ca0 con 0x7f4300140af0 2024-01-26T23:46:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f42ff7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43000c8da0 0x7f430013c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:45.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f42ff7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43000c8da0 0x7f430013c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35534/0 (socket says 172.21.15.46:35534) 2024-01-26T23:46:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f42ff7fe700 1 -- 172.21.15.46:0/276581093 learned_addr learned my addr 172.21.15.46:0/276581093 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f42feffd700 1 --2- 172.21.15.46:0/276581093 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43000caf30 0x7f430016a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f42ff7fe700 1 -- 172.21.15.46:0/276581093 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4300140af0 msgr2=0x7f430016aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f42fffff700 1 --2- 172.21.15.46:0/276581093 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4300140af0 0x7f430016aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f42ff7fe700 1 --2- 172.21.15.46:0/276581093 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4300140af0 0x7f430016aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f42ff7fe700 1 -- 172.21.15.46:0/276581093 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43000caf30 msgr2=0x7f430016a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f42ff7fe700 1 --2- 172.21.15.46:0/276581093 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43000caf30 0x7f430016a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f42ff7fe700 1 -- 172.21.15.46:0/276581093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f0009420 con 0x7f43000c8da0 2024-01-26T23:46:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f42fffff700 1 --2- 172.21.15.46:0/276581093 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4300140af0 0x7f430016aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:46:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f42feffd700 1 --2- 172.21.15.46:0/276581093 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43000caf30 0x7f430016a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:46:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.195+0000 7f42ff7fe700 1 --2- 172.21.15.46:0/276581093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43000c8da0 0x7f430013c920 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f430006f5a0 tx=0x7f42f0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.196+0000 7f42e77fe700 1 -- 172.21.15.46:0/276581093 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f0011420 con 0x7f43000c8da0 2024-01-26T23:46:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.196+0000 7f42e77fe700 1 -- 172.21.15.46:0/276581093 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f0011580 con 0x7f43000c8da0 2024-01-26T23:46:45.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.196+0000 7f43061ab700 1 -- 172.21.15.46:0/276581093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4300168660 con 0x7f43000c8da0 2024-01-26T23:46:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.196+0000 7f42e77fe700 1 -- 172.21.15.46:0/276581093 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f0011940 con 0x7f43000c8da0 2024-01-26T23:46:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.196+0000 7f43061ab700 1 -- 172.21.15.46:0/276581093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43000b2cf0 con 0x7f43000c8da0 2024-01-26T23:46:45.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.197+0000 7f43061ab700 1 -- 172.21.15.46:0/276581093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4300168b80 con 0x7f43000c8da0 2024-01-26T23:46:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.197+0000 7f42e77fe700 1 -- 172.21.15.46:0/276581093 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 44) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f42f0011d30 con 0x7f43000c8da0 2024-01-26T23:46:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.198+0000 7f42e77fe700 1 --2- 172.21.15.46:0/276581093 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f42ec066ea0 0x7f42ec069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.198+0000 7f42e77fe700 1 -- 172.21.15.46:0/276581093 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f42f0085b90 con 0x7f43000c8da0 2024-01-26T23:46:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.198+0000 7f42feffd700 1 --2- 172.21.15.46:0/276581093 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f42ec066ea0 0x7f42ec069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.199+0000 7f42feffd700 1 --2- 172.21.15.46:0/276581093 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f42ec066ea0 0x7f42ec069350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f43000c4a80 tx=0x7f42f4009040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.202+0000 7f42e77fe700 1 -- 172.21.15.46:0/276581093 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4300168b80 con 0x7f43000c8da0 2024-01-26T23:46:45.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.464+0000 7f43061ab700 1 -- 172.21.15.46:0/276581093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f43000b3ec0 con 0x7f43000c8da0 2024-01-26T23:46:45.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.675+0000 7f42e77fe700 1 -- 172.21.15.46:0/276581093 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 45) v1 ==== 84703+0+0 (secure 0 0 0) 0x7f42f0052fe0 con 0x7f43000c8da0 2024-01-26T23:46:45.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.677+0000 7f42e77fe700 1 -- 172.21.15.46:0/276581093 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v45) v1 ==== 92+0+0 (secure 0 0 0) 0x7f42f006cef0 con 0x7f43000c8da0 2024-01-26T23:46:45.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.680+0000 7f43061ab700 1 -- 172.21.15.46:0/276581093 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f42ec066ea0 msgr2=0x7f42ec069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:45.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.680+0000 7f43061ab700 1 --2- 172.21.15.46:0/276581093 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f42ec066ea0 0x7f42ec069350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f43000c4a80 tx=0x7f42f4009040 comp rx=0 tx=0).stop 2024-01-26T23:46:45.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.680+0000 7f43061ab700 1 -- 172.21.15.46:0/276581093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43000c8da0 msgr2=0x7f430013c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:45.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.680+0000 7f43061ab700 1 --2- 172.21.15.46:0/276581093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43000c8da0 0x7f430013c920 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f430006f5a0 tx=0x7f42f0017000 comp rx=0 tx=0).stop 2024-01-26T23:46:45.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.680+0000 7f43061ab700 1 -- 172.21.15.46:0/276581093 shutdown_connections 2024-01-26T23:46:45.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.680+0000 7f43061ab700 1 --2- 172.21.15.46:0/276581093 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f42ec066ea0 0x7f42ec069350 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.680+0000 7f43061ab700 1 --2- 172.21.15.46:0/276581093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43000c8da0 0x7f430013c920 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.680+0000 7f43061ab700 1 --2- 172.21.15.46:0/276581093 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4300140af0 0x7f430016aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.680+0000 7f43061ab700 1 --2- 172.21.15.46:0/276581093 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43000caf30 0x7f430016a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.680+0000 7f43061ab700 1 -- 172.21.15.46:0/276581093 >> 172.21.15.46:0/276581093 conn(0x7f430006bd90 msgr2=0x7f430013f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:45.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.680+0000 7f43061ab700 1 -- 172.21.15.46:0/276581093 shutdown_connections 2024-01-26T23:46:45.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.680+0000 7f43061ab700 1 -- 172.21.15.46:0/276581093 wait complete. 2024-01-26T23:46:45.697 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-26T23:46:45.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.838+0000 7effcdb8a700 1 Processor -- start 2024-01-26T23:46:45.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.838+0000 7effcdb8a700 1 -- start start 2024-01-26T23:46:45.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.839+0000 7effcdb8a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7effc8159510 0x7effc81598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:45.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.839+0000 7effcdb8a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effc80ca220 0x7effc80c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:45.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.839+0000 7effcdb8a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc8069260 con 0x7effc80ca690 2024-01-26T23:46:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.839+0000 7effcdb8a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc815d670 con 0x7effc8159510 2024-01-26T23:46:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.839+0000 7effcdb8a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc80c9750 con 0x7effc80ca220 2024-01-26T23:46:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.839+0000 7effc7fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7effc80ca690 0x7effc80cc380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34932/0 (socket says 172.21.15.46:34932) 2024-01-26T23:46:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.839+0000 7effc7fff700 1 -- 172.21.15.46:0/3572196262 learned_addr learned my addr 172.21.15.46:0/3572196262 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.839+0000 7effc77fe700 1 --2- 172.21.15.46:0/3572196262 >> v2:172.21.15.106:3300/0 conn(0x7effc8159510 0x7effc81598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.839+0000 7effc6ffd700 1 --2- 172.21.15.46:0/3572196262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effc80ca220 0x7effc80c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.839+0000 7effc67fc700 1 -- 172.21.15.46:0/3572196262 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079886871 0 0) 0x7effc8069260 con 0x7effc80ca690 2024-01-26T23:46:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.839+0000 7effc67fc700 1 -- 172.21.15.46:0/3572196262 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effb0003570 con 0x7effc80ca690 2024-01-26T23:46:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.839+0000 7effc77fe700 1 -- 172.21.15.46:0/3572196262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effc80ca220 msgr2=0x7effc80c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.839+0000 7effc77fe700 1 --2- 172.21.15.46:0/3572196262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effc80ca220 0x7effc80c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.839+0000 7effc77fe700 1 -- 172.21.15.46:0/3572196262 >> v1:172.21.15.46:6789/0 conn(0x7effc80ca690 legacy=0x7effc80cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.839+0000 7effc77fe700 1 -- 172.21.15.46:0/3572196262 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc80c9a30 con 0x7effc8159510 2024-01-26T23:46:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.839+0000 7effc6ffd700 1 --2- 172.21.15.46:0/3572196262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effc80ca220 0x7effc80c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:45.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.840+0000 7effc77fe700 1 --2- 172.21.15.46:0/3572196262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effc8159510 0x7effc81598e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7effb8009910 tx=0x7effb80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3395c00b0a8fdc64 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:45.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.840+0000 7effc67fc700 1 -- 172.21.15.46:0/3572196262 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb8011630 con 0x7effc8159510 2024-01-26T23:46:45.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.840+0000 7effc67fc700 1 -- 172.21.15.46:0/3572196262 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb8011790 con 0x7effc8159510 2024-01-26T23:46:45.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.840+0000 7effc67fc700 1 -- 172.21.15.46:0/3572196262 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb8011b30 con 0x7effc8159510 2024-01-26T23:46:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.841+0000 7effcdb8a700 1 -- 172.21.15.46:0/3572196262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effc8159510 msgr2=0x7effc81598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.841+0000 7effcdb8a700 1 --2- 172.21.15.46:0/3572196262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effc8159510 0x7effc81598e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7effb8009910 tx=0x7effb80091a0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.841+0000 7effcdb8a700 1 -- 172.21.15.46:0/3572196262 shutdown_connections 2024-01-26T23:46:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.841+0000 7effcdb8a700 1 --2- 172.21.15.46:0/3572196262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effc8159510 0x7effc81598e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.841+0000 7effcdb8a700 1 --2- 172.21.15.46:0/3572196262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effc80ca220 0x7effc80c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.841+0000 7effcdb8a700 1 -- 172.21.15.46:0/3572196262 >> 172.21.15.46:0/3572196262 conn(0x7effc806bd90 msgr2=0x7effc806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.841+0000 7effcdb8a700 1 -- 172.21.15.46:0/3572196262 shutdown_connections 2024-01-26T23:46:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.841+0000 7effcdb8a700 1 -- 172.21.15.46:0/3572196262 wait complete. 2024-01-26T23:46:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.842+0000 7effcdb8a700 1 Processor -- start 2024-01-26T23:46:45.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.842+0000 7effcdb8a700 1 -- start start 2024-01-26T23:46:45.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.843+0000 7effcdb8a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effc80ca220 0x7effc80b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:45.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.843+0000 7effcdb8a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effc8159510 0x7effc814cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:45.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.843+0000 7effcdb8a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effc8159e50 0x7effc815b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:45.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.843+0000 7effcdb8a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc815afd0 con 0x7effc80ca220 2024-01-26T23:46:45.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.843+0000 7effcdb8a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc815a3c0 con 0x7effc8159e50 2024-01-26T23:46:45.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.843+0000 7effcdb8a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc815a6a0 con 0x7effc8159510 2024-01-26T23:46:45.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.843+0000 7effc77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effc80ca220 0x7effc80b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:45.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.843+0000 7effc77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effc80ca220 0x7effc80b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35550/0 (socket says 172.21.15.46:35550) 2024-01-26T23:46:45.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.843+0000 7effc77fe700 1 -- 172.21.15.46:0/912980050 learned_addr learned my addr 172.21.15.46:0/912980050 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:45.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.843+0000 7effc6ffd700 1 --2- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effc8159510 0x7effc814cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:45.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.843+0000 7effc7fff700 1 --2- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effc8159e50 0x7effc815b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:45.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.844+0000 7effc77fe700 1 -- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effc8159510 msgr2=0x7effc814cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:45.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.844+0000 7effc77fe700 1 --2- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effc8159510 0x7effc814cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.844+0000 7effc77fe700 1 -- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effc8159e50 msgr2=0x7effc815b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:45.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.844+0000 7effc77fe700 1 --2- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effc8159e50 0x7effc815b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:45.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.844+0000 7effc77fe700 1 -- 172.21.15.46:0/912980050 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effbc009420 con 0x7effc80ca220 2024-01-26T23:46:45.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.844+0000 7effc6ffd700 1 --2- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effc8159510 0x7effc814cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:45.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.844+0000 7effc7fff700 1 --2- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effc8159e50 0x7effc815b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:45.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.844+0000 7effc77fe700 1 --2- 172.21.15.46:0/912980050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effc80ca220 0x7effc80b3b90 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7effb8009f00 tx=0x7effb8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:45.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.844+0000 7effaf7fe700 1 -- 172.21.15.46:0/912980050 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb8011900 con 0x7effc80ca220 2024-01-26T23:46:45.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.844+0000 7effaf7fe700 1 -- 172.21.15.46:0/912980050 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb8011ac0 con 0x7effc80ca220 2024-01-26T23:46:45.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.844+0000 7effaf7fe700 1 -- 172.21.15.46:0/912980050 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb8025640 con 0x7effc80ca220 2024-01-26T23:46:45.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.844+0000 7effcdb8a700 1 -- 172.21.15.46:0/912980050 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb8009420 con 0x7effc80ca220 2024-01-26T23:46:45.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.844+0000 7effcdb8a700 1 -- 172.21.15.46:0/912980050 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc815ac60 con 0x7effc80ca220 2024-01-26T23:46:45.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.845+0000 7effcdb8a700 1 -- 172.21.15.46:0/912980050 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc804c710 con 0x7effc80ca220 2024-01-26T23:46:45.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.846+0000 7effaf7fe700 1 -- 172.21.15.46:0/912980050 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 45) v1 ==== 84703+0+0 (secure 0 0 0) 0x7effb8025a30 con 0x7effc80ca220 2024-01-26T23:46:45.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.846+0000 7effaf7fe700 1 --2- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7effb4066ef0 0x7effb40693a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:45.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.846+0000 7effc6ffd700 1 -- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7effb4066ef0 msgr2=0x7effb40693a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:45.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.846+0000 7effaf7fe700 1 -- 172.21.15.46:0/912980050 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6460+0+0 (secure 0 0 0) 0x7effb8085d40 con 0x7effc80ca220 2024-01-26T23:46:45.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.846+0000 7effc6ffd700 1 --2- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7effb4066ef0 0x7effb40693a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:46:45.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:45.851+0000 7effaf7fe700 1 -- 172.21.15.46:0/912980050 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7effb8052da0 con 0x7effc80ca220 2024-01-26T23:46:46.013 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:46.012+0000 7f595a1b91c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:46:46.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.048+0000 7effc6ffd700 1 -- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7effb4066ef0 msgr2=0x7effb40693a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:46.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.048+0000 7effc6ffd700 1 --2- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7effb4066ef0 0x7effb40693a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:46:46.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.059+0000 7effcdb8a700 1 -- 172.21.15.46:0/912980050 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7effc80c9d10 con 0x7effc80ca220 2024-01-26T23:46:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.065+0000 7effaf7fe700 1 -- 172.21.15.46:0/912980050 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v35)=0 v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7effb80563c0 con 0x7effc80ca220 2024-01-26T23:46:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.067+0000 7effcdb8a700 1 -- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7effb4066ef0 msgr2=0x7effb40693a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:46:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.067+0000 7effcdb8a700 1 --2- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7effb4066ef0 0x7effb40693a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.067+0000 7effcdb8a700 1 -- 172.21.15.46:0/912980050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effc80ca220 msgr2=0x7effc80b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.067+0000 7effcdb8a700 1 --2- 172.21.15.46:0/912980050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effc80ca220 0x7effc80b3b90 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7effb8009f00 tx=0x7effb8017000 comp rx=0 tx=0).stop 2024-01-26T23:46:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.067+0000 7effcdb8a700 1 -- 172.21.15.46:0/912980050 shutdown_connections 2024-01-26T23:46:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.067+0000 7effcdb8a700 1 --2- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7effb4066ef0 0x7effb40693a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.067+0000 7effcdb8a700 1 --2- 172.21.15.46:0/912980050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effc80ca220 0x7effc80b3b90 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.067+0000 7effcdb8a700 1 --2- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effc8159e50 0x7effc815b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.067+0000 7effcdb8a700 1 --2- 172.21.15.46:0/912980050 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effc8159510 0x7effc814cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.067+0000 7effcdb8a700 1 -- 172.21.15.46:0/912980050 >> 172.21.15.46:0/912980050 conn(0x7effc806bd90 msgr2=0x7effc8069d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:46.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.067+0000 7effcdb8a700 1 -- 172.21.15.46:0/912980050 shutdown_connections 2024-01-26T23:46:46.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.067+0000 7effcdb8a700 1 -- 172.21.15.46:0/912980050 wait complete. 2024-01-26T23:46:46.075 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:46.074+0000 7f9def0491c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:46:46.083 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-26T23:46:46.092 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:46.091+0000 7f595a1b91c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:46:46.161 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:46.161+0000 7f595a1b91c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:46:46.165 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:46.164+0000 7f9def0491c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:46:46.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.218+0000 7f7bd8af3700 1 Processor -- start 2024-01-26T23:46:46.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.218+0000 7f7bd8af3700 1 -- start start 2024-01-26T23:46:46.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.219+0000 7f7bd8af3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7bd40cf050 0x7f7bd40af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:46.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.219+0000 7f7bd8af3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bd40cf4a0 0x7f7bd40af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:46.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.219+0000 7f7bd8af3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd40cc100 con 0x7f7bd4159cb0 2024-01-26T23:46:46.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.219+0000 7f7bd8af3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bd40ccca0 con 0x7f7bd40cf050 2024-01-26T23:46:46.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.219+0000 7f7bd8af3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd40cb850 con 0x7f7bd40cf4a0 2024-01-26T23:46:46.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.219+0000 7f7bd1d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7bd40cf050 0x7f7bd40af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:46.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.219+0000 7f7bd1d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7bd40cf050 0x7f7bd40af430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35560/0 (socket says 172.21.15.46:35560) 2024-01-26T23:46:46.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.219+0000 7f7bd1d9b700 1 -- 172.21.15.46:0/2075575628 learned_addr learned my addr 172.21.15.46:0/2075575628 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:46.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.219+0000 7f7bd2d9d700 1 --2- 172.21.15.46:0/2075575628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bd40cf4a0 0x7f7bd40af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:46.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.220+0000 7f7bd159a700 1 -- 172.21.15.46:0/2075575628 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717758512 0 0) 0x7f7bd40cc100 con 0x7f7bd4159cb0 2024-01-26T23:46:46.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.220+0000 7f7bd1d9b700 1 -- 172.21.15.46:0/2075575628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bd40cf4a0 msgr2=0x7f7bd40af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:46.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.220+0000 7f7bd1d9b700 1 --2- 172.21.15.46:0/2075575628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bd40cf4a0 0x7f7bd40af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.220+0000 7f7bd1d9b700 1 -- 172.21.15.46:0/2075575628 >> v1:172.21.15.106:6789/0 conn(0x7f7bd4159cb0 legacy=0x7f7bd40cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:46.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.220+0000 7f7bd1d9b700 1 -- 172.21.15.46:0/2075575628 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd40cbaf0 con 0x7f7bd40cf050 2024-01-26T23:46:46.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.220+0000 7f7bd1d9b700 1 --2- 172.21.15.46:0/2075575628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd40cf050 0x7f7bd40af430 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f7bc80099b0 tx=0x7f7bc8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9ee6b8cc02ce3e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:46.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.220+0000 7f7bd159a700 1 -- 172.21.15.46:0/2075575628 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8009ea0 con 0x7f7bd40cf050 2024-01-26T23:46:46.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.220+0000 7f7bd159a700 1 -- 172.21.15.46:0/2075575628 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc800b860 con 0x7f7bd40cf050 2024-01-26T23:46:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.220+0000 7f7bd159a700 1 -- 172.21.15.46:0/2075575628 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc800bc00 con 0x7f7bd40cf050 2024-01-26T23:46:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.221+0000 7f7bd8af3700 1 -- 172.21.15.46:0/2075575628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd40cf050 msgr2=0x7f7bd40af430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.221+0000 7f7bd8af3700 1 --2- 172.21.15.46:0/2075575628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd40cf050 0x7f7bd40af430 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f7bc80099b0 tx=0x7f7bc8009250 comp rx=0 tx=0).stop 2024-01-26T23:46:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.221+0000 7f7bd8af3700 1 -- 172.21.15.46:0/2075575628 shutdown_connections 2024-01-26T23:46:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.221+0000 7f7bd8af3700 1 --2- 172.21.15.46:0/2075575628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bd40cf4a0 0x7f7bd40af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.221+0000 7f7bd8af3700 1 --2- 172.21.15.46:0/2075575628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd40cf050 0x7f7bd40af430 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.221+0000 7f7bd8af3700 1 -- 172.21.15.46:0/2075575628 >> 172.21.15.46:0/2075575628 conn(0x7f7bd4087f60 msgr2=0x7f7bd406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.221+0000 7f7bd8af3700 1 -- 172.21.15.46:0/2075575628 shutdown_connections 2024-01-26T23:46:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.221+0000 7f7bd8af3700 1 -- 172.21.15.46:0/2075575628 wait complete. 2024-01-26T23:46:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.222+0000 7f7bd8af3700 1 Processor -- start 2024-01-26T23:46:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.222+0000 7f7bd8af3700 1 -- start start 2024-01-26T23:46:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.223+0000 7f7bd8af3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bd40cf050 0x7f7bd40c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.223+0000 7f7bd8af3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd40cf4a0 0x7f7bd40c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.223+0000 7f7bd8af3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bd4159cb0 0x7f7bd40ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.223+0000 7f7bd8af3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd40ca640 con 0x7f7bd40cf4a0 2024-01-26T23:46:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.223+0000 7f7bd8af3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd4156620 con 0x7f7bd40cf050 2024-01-26T23:46:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.223+0000 7f7bd8af3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd4169f70 con 0x7f7bd4159cb0 2024-01-26T23:46:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.223+0000 7f7bd1d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd40cf4a0 0x7f7bd40c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.223+0000 7f7bd259c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bd40cf050 0x7f7bd40c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.223+0000 7f7bd1d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd40cf4a0 0x7f7bd40c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35574/0 (socket says 172.21.15.46:35574) 2024-01-26T23:46:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.223+0000 7f7bd2d9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bd4159cb0 0x7f7bd40ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.223+0000 7f7bd259c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bd40cf050 0x7f7bd40c9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:48924/0 (socket says 172.21.15.46:48924) 2024-01-26T23:46:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.223+0000 7f7bd1d9b700 1 -- 172.21.15.46:0/3240871972 learned_addr learned my addr 172.21.15.46:0/3240871972 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.224+0000 7f7bd1d9b700 1 -- 172.21.15.46:0/3240871972 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bd4159cb0 msgr2=0x7f7bd40ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.224+0000 7f7bd1d9b700 1 --2- 172.21.15.46:0/3240871972 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bd4159cb0 0x7f7bd40ae240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.224+0000 7f7bd1d9b700 1 -- 172.21.15.46:0/3240871972 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bd40cf050 msgr2=0x7f7bd40c9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.224+0000 7f7bd1d9b700 1 --2- 172.21.15.46:0/3240871972 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bd40cf050 0x7f7bd40c9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.224+0000 7f7bd1d9b700 1 -- 172.21.15.46:0/3240871972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc80094f0 con 0x7f7bd40cf4a0 2024-01-26T23:46:46.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.224+0000 7f7bd259c700 1 --2- 172.21.15.46:0/3240871972 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bd40cf050 0x7f7bd40c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:46:46.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.224+0000 7f7bd1d9b700 1 --2- 172.21.15.46:0/3240871972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd40cf4a0 0x7f7bd40c9770 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4088df0 tx=0x7f7bc8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:46.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.224+0000 7f7bc27fc700 1 -- 172.21.15.46:0/3240871972 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8000d10 con 0x7f7bd40cf4a0 2024-01-26T23:46:46.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.224+0000 7f7bc27fc700 1 -- 172.21.15.46:0/3240871972 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc800bbe0 con 0x7f7bd40cf4a0 2024-01-26T23:46:46.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.224+0000 7f7bc27fc700 1 -- 172.21.15.46:0/3240871972 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8011670 con 0x7f7bd40cf4a0 2024-01-26T23:46:46.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.224+0000 7f7bd8af3700 1 -- 172.21.15.46:0/3240871972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd416ae30 con 0x7f7bd40cf4a0 2024-01-26T23:46:46.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.224+0000 7f7bd8af3700 1 -- 172.21.15.46:0/3240871972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd40ca080 con 0x7f7bd40cf4a0 2024-01-26T23:46:46.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.225+0000 7f7bd8af3700 1 -- 172.21.15.46:0/3240871972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd404c710 con 0x7f7bd40cf4a0 2024-01-26T23:46:46.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.226+0000 7f7bc27fc700 1 -- 172.21.15.46:0/3240871972 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 45) v1 ==== 84703+0+0 (secure 0 0 0) 0x7f7bc8011aa0 con 0x7f7bd40cf4a0 2024-01-26T23:46:46.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.226+0000 7f7bc27fc700 1 --2- 172.21.15.46:0/3240871972 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7bb4066ef0 0x7f7bb40693a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:46.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.226+0000 7f7bc27fc700 1 -- 172.21.15.46:0/3240871972 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7bc8085be0 con 0x7f7bd40cf4a0 2024-01-26T23:46:46.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.226+0000 7f7bd259c700 1 -- 172.21.15.46:0/3240871972 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7bb4066ef0 msgr2=0x7f7bb40693a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:46.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.226+0000 7f7bd259c700 1 --2- 172.21.15.46:0/3240871972 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7bb4066ef0 0x7f7bb40693a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:46:46.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.231+0000 7f7bc27fc700 1 -- 172.21.15.46:0/3240871972 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7bc8052cc0 con 0x7f7bd40cf4a0 2024-01-26T23:46:46.232 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:46.227+0000 7f595a1b91c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:46:46.250 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:46.249+0000 7f9def0491c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:46:46.326 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:46.325+0000 7f9def0491c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:46:46.384 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:46.383+0000 7f595a1b91c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:46:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.427+0000 7f7bd259c700 1 -- 172.21.15.46:0/3240871972 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7bb4066ef0 msgr2=0x7f7bb40693a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.427+0000 7f7bd259c700 1 --2- 172.21.15.46:0/3240871972 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7bb4066ef0 0x7f7bb40693a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:46:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.438+0000 7f7bd8af3700 1 -- 172.21.15.46:0/3240871972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f7bd40cfc80 con 0x7f7bd40cf4a0 2024-01-26T23:46:46.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.441+0000 7f7bc27fc700 1 -- 172.21.15.46:0/3240871972 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v36)=0 v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7f7bc80150c0 con 0x7f7bd40cf4a0 2024-01-26T23:46:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.444+0000 7f7bd8af3700 1 -- 172.21.15.46:0/3240871972 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7bb4066ef0 msgr2=0x7f7bb40693a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:46:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.444+0000 7f7bd8af3700 1 --2- 172.21.15.46:0/3240871972 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7bb4066ef0 0x7f7bb40693a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.444+0000 7f7bd8af3700 1 -- 172.21.15.46:0/3240871972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd40cf4a0 msgr2=0x7f7bd40c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.444+0000 7f7bd8af3700 1 --2- 172.21.15.46:0/3240871972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd40cf4a0 0x7f7bd40c9770 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4088df0 tx=0x7f7bc8017000 comp rx=0 tx=0).stop 2024-01-26T23:46:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.444+0000 7f7bd8af3700 1 -- 172.21.15.46:0/3240871972 shutdown_connections 2024-01-26T23:46:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.444+0000 7f7bd8af3700 1 --2- 172.21.15.46:0/3240871972 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7bb4066ef0 0x7f7bb40693a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.444+0000 7f7bd8af3700 1 --2- 172.21.15.46:0/3240871972 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bd4159cb0 0x7f7bd40ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.444+0000 7f7bd8af3700 1 --2- 172.21.15.46:0/3240871972 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bd40cf050 0x7f7bd40c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.444+0000 7f7bd8af3700 1 --2- 172.21.15.46:0/3240871972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd40cf4a0 0x7f7bd40c9770 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.444+0000 7f7bd8af3700 1 -- 172.21.15.46:0/3240871972 >> 172.21.15.46:0/3240871972 conn(0x7f7bd4087f60 msgr2=0x7f7bd40c6970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.444+0000 7f7bd8af3700 1 -- 172.21.15.46:0/3240871972 shutdown_connections 2024-01-26T23:46:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.444+0000 7f7bd8af3700 1 -- 172.21.15.46:0/3240871972 wait complete. 2024-01-26T23:46:46.459 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /bad-path 1m 2024-01-26T23:46:46.509 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:46.508+0000 7f9def0491c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:46:46.520 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:46.519+0000 7f595a1b91c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:46:46.586 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:46.585+0000 7f595a1b91c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:46:46.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.598+0000 7fd2ff333700 1 Processor -- start 2024-01-26T23:46:46.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.598+0000 7fd2ff333700 1 -- start start 2024-01-26T23:46:46.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.599+0000 7fd2ff333700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd2f8159cb0 0x7fd2f80cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:46.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.599+0000 7fd2ff333700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f80cc100 con 0x7fd2f80afb40 2024-01-26T23:46:46.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.599+0000 7fd2ff333700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f80cb850 con 0x7fd2f80cf050 2024-01-26T23:46:46.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.599+0000 7fd2ff333700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f80cbaf0 con 0x7fd2f8159cb0 2024-01-26T23:46:46.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.599+0000 7fd2fd8d0700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd2f80afb40 0x7fd2f80c5530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34948/0 (socket says 172.21.15.46:34948) 2024-01-26T23:46:46.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.600+0000 7fd2fd8d0700 1 -- 172.21.15.46:0/1838736139 learned_addr learned my addr 172.21.15.46:0/1838736139 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:46:46.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.600+0000 7fd2fd0cf700 1 --2- 172.21.15.46:0/1838736139 >> v2:172.21.15.106:3300/0 conn(0x7fd2f8159cb0 0x7fd2f80cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:46.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.600+0000 7fd2effff700 1 -- 172.21.15.46:0/1838736139 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266412133 0 0) 0x7fd2f80cc100 con 0x7fd2f80afb40 2024-01-26T23:46:46.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.600+0000 7fd2effff700 1 -- 172.21.15.46:0/1838736139 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e4003570 con 0x7fd2f80afb40 2024-01-26T23:46:46.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.600+0000 7fd2effff700 1 -- 172.21.15.46:0/1838736139 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222454521 0 0) 0x7fd2f80cb850 con 0x7fd2f80cf050 2024-01-26T23:46:46.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.600+0000 7fd2effff700 1 -- 172.21.15.46:0/1838736139 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f80cc100 con 0x7fd2f80cf050 2024-01-26T23:46:46.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.600+0000 7fd2fd0cf700 1 -- 172.21.15.46:0/1838736139 >> v1:172.21.15.106:6789/0 conn(0x7fd2f80cf050 legacy=0x7fd2f80af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:46.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.600+0000 7fd2fd0cf700 1 -- 172.21.15.46:0/1838736139 >> v1:172.21.15.46:6789/0 conn(0x7fd2f80afb40 legacy=0x7fd2f80c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:46.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.600+0000 7fd2fd0cf700 1 -- 172.21.15.46:0/1838736139 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f80cfc80 con 0x7fd2f8159cb0 2024-01-26T23:46:46.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.600+0000 7fd2fd0cf700 1 --2- 172.21.15.46:0/1838736139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f8159cb0 0x7fd2f80cc590 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009eb0 tx=0x7fd2f4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df5e7080e4e1fde6 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:46.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.600+0000 7fd2effff700 1 -- 172.21.15.46:0/1838736139 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f400e070 con 0x7fd2f8159cb0 2024-01-26T23:46:46.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.600+0000 7fd2effff700 1 -- 172.21.15.46:0/1838736139 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f400bb80 con 0x7fd2f8159cb0 2024-01-26T23:46:46.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.600+0000 7fd2effff700 1 -- 172.21.15.46:0/1838736139 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4012430 con 0x7fd2f8159cb0 2024-01-26T23:46:46.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.601+0000 7fd2ff333700 1 -- 172.21.15.46:0/1838736139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f8159cb0 msgr2=0x7fd2f80cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:46.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.601+0000 7fd2ff333700 1 --2- 172.21.15.46:0/1838736139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f8159cb0 0x7fd2f80cc590 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009eb0 tx=0x7fd2f4009780 comp rx=0 tx=0).stop 2024-01-26T23:46:46.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.601+0000 7fd2ff333700 1 -- 172.21.15.46:0/1838736139 shutdown_connections 2024-01-26T23:46:46.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.601+0000 7fd2ff333700 1 --2- 172.21.15.46:0/1838736139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f8159cb0 0x7fd2f80cc590 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.601+0000 7fd2ff333700 1 -- 172.21.15.46:0/1838736139 >> 172.21.15.46:0/1838736139 conn(0x7fd2f8087f60 msgr2=0x7fd2f806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:46.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.601+0000 7fd2ff333700 1 -- 172.21.15.46:0/1838736139 shutdown_connections 2024-01-26T23:46:46.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.601+0000 7fd2ff333700 1 -- 172.21.15.46:0/1838736139 wait complete. 2024-01-26T23:46:46.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.603+0000 7fd2ff333700 1 Processor -- start 2024-01-26T23:46:46.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.603+0000 7fd2ff333700 1 -- start start 2024-01-26T23:46:46.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.604+0000 7fd2ff333700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f80afb40 0x7fd2f8153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.604+0000 7fd2ff333700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f80cf050 0x7fd2f80c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.604+0000 7fd2ff333700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f8159cb0 0x7fd2f80b2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.604+0000 7fd2ff333700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80c8080 con 0x7fd2f8159cb0 2024-01-26T23:46:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.604+0000 7fd2ff333700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80c8360 con 0x7fd2f80cf050 2024-01-26T23:46:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.604+0000 7fd2ff333700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80c8640 con 0x7fd2f80afb40 2024-01-26T23:46:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.604+0000 7fd2fd8d0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f8159cb0 0x7fd2f80b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.604+0000 7fd2fd0cf700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f80afb40 0x7fd2f8153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.604+0000 7fd2fc8ce700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f80cf050 0x7fd2f80c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.604+0000 7fd2fd0cf700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f80afb40 0x7fd2f8153290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:46128/0 (socket says 172.21.15.46:46128) 2024-01-26T23:46:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.604+0000 7fd2fd8d0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f8159cb0 0x7fd2f80b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35582/0 (socket says 172.21.15.46:35582) 2024-01-26T23:46:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.604+0000 7fd2fd0cf700 1 -- 172.21.15.46:0/2640407934 learned_addr learned my addr 172.21.15.46:0/2640407934 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.605+0000 7fd2fd8d0700 1 -- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f80afb40 msgr2=0x7fd2f8153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.605+0000 7fd2fd8d0700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f80afb40 0x7fd2f8153290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.605+0000 7fd2fd8d0700 1 -- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f80cf050 msgr2=0x7fd2f80c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.605+0000 7fd2fd8d0700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f80cf050 0x7fd2f80c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.605+0000 7fd2fd8d0700 1 -- 172.21.15.46:0/2640407934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f4009a20 con 0x7fd2f8159cb0 2024-01-26T23:46:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.605+0000 7fd2fd8d0700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f8159cb0 0x7fd2f80b2830 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00011e0 tx=0x7fd2f0007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.605+0000 7fd2ed7fa700 1 -- 172.21.15.46:0/2640407934 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0005b30 con 0x7fd2f8159cb0 2024-01-26T23:46:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.605+0000 7fd2ed7fa700 1 -- 172.21.15.46:0/2640407934 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f0005c90 con 0x7fd2f8159cb0 2024-01-26T23:46:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.605+0000 7fd2ff333700 1 -- 172.21.15.46:0/2640407934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f80c8920 con 0x7fd2f8159cb0 2024-01-26T23:46:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.605+0000 7fd2ed7fa700 1 -- 172.21.15.46:0/2640407934 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0007320 con 0x7fd2f8159cb0 2024-01-26T23:46:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.605+0000 7fd2ff333700 1 -- 172.21.15.46:0/2640407934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f81537d0 con 0x7fd2f8159cb0 2024-01-26T23:46:46.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.606+0000 7fd2ff333700 1 -- 172.21.15.46:0/2640407934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f80c8e40 con 0x7fd2f8159cb0 2024-01-26T23:46:46.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.607+0000 7fd2ed7fa700 1 -- 172.21.15.46:0/2640407934 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 45) v1 ==== 84703+0+0 (secure 0 0 0) 0x7fd2f001b3b0 con 0x7fd2f8159cb0 2024-01-26T23:46:46.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.607+0000 7fd2ed7fa700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 0x7fd2dc0693a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:46.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.607+0000 7fd2ed7fa700 1 -- 172.21.15.46:0/2640407934 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd2f0083b70 con 0x7fd2f8159cb0 2024-01-26T23:46:46.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.607+0000 7fd2fd0cf700 1 -- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 msgr2=0x7fd2dc0693a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:46.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.607+0000 7fd2fd0cf700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 0x7fd2dc0693a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:46:46.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.612+0000 7fd2ed7fa700 1 -- 172.21.15.46:0/2640407934 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd2f0050bd0 con 0x7fd2f8159cb0 2024-01-26T23:46:46.680 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:46.679+0000 7f9def0491c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:46:46.710 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:46.709+0000 7f595a1b91c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:46:46.757 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:46.756+0000 7f9def0491c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:46:46.784 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:46.784+0000 7f595a1b91c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:46:46.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.808+0000 7fd2fd0cf700 1 -- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 msgr2=0x7fd2dc0693a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:46.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.808+0000 7fd2fd0cf700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 0x7fd2dc0693a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:46:46.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:46.864+0000 7fd2ff333700 1 -- 172.21.15.46:0/2640407934 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2f804c710 con 0x7fd2dc066ef0 2024-01-26T23:46:46.898 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:46.897+0000 7f9def0491c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:46:46.924 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:46.923+0000 7f595a1b91c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:46:46.981 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:46.980+0000 7f9def0491c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:46:47.050 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:47.050+0000 7f595a1b91c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:46:47.157 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:47.156+0000 7f9def0491c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:46:47.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:47.209+0000 7fd2fd0cf700 1 -- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 msgr2=0x7fd2dc0693a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:47.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:47.209+0000 7fd2fd0cf700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 0x7fd2dc0693a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-26T23:46:47.299 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:47.298+0000 7f9def0491c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:46:47.362 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:47.361+0000 7f595a1b91c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:46:47.571 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:47.571+0000 7f595a1b91c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:46:47.662 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:47.661+0000 7f9def0491c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:46:47.799 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:47.798+0000 7f595a1b91c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:46:47.869 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:47.867+0000 7f595a1b91c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:46:47.905 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:47.904+0000 7f9def0491c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:46:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:48.010+0000 7fd2fd0cf700 1 -- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 msgr2=0x7fd2dc0693a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:48.010+0000 7fd2fd0cf700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 0x7fd2dc0693a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-26T23:46:48.165 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:48.164+0000 7f9def0491c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:46:48.246 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:48.245+0000 7f9def0491c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:46:48.465 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:48.464+0000 7f595a1b91c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:46:48.539 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:48.538+0000 7f595a1b91c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:46:48.660 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:48.660+0000 7f595a1b91c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:46:48.943 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:48.942+0000 7f9def0491c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:46:49.028 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:49.027+0000 7f9def0491c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:46:49.159 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:49.158+0000 7f595a1b91c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:46:49.169 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:49.168+0000 7f9def0491c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:46:49.311 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:49.310+0000 7f595a1b91c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:46:49.460 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:49.459+0000 7f595a1b91c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:46:49.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:49.612+0000 7fd2fd0cf700 1 -- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 msgr2=0x7fd2dc0693a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:46:49.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:49.612+0000 7fd2fd0cf700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 0x7fd2dc0693a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-26T23:46:49.757 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:49.756+0000 7f9def0491c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:46:49.929 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:49.928+0000 7f9def0491c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:46:50.104 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:50.103+0000 7f9def0491c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:46:50.158 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:50.157+0000 7f595a1b91c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:46:50.376 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:50.376+0000 7f595a1b91c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:46:50.448 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:50.448+0000 7f595a1b91c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:46:50.603 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:50.602+0000 7f595a1b91c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:46:50.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:50.622+0000 7fd2ed7fa700 1 -- 172.21.15.46:0/2640407934 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 46) v1 ==== 84470+0+0 (secure 0 0 0) 0x7fd2f0050fa0 con 0x7fd2f8159cb0 2024-01-26T23:46:50.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:50.623+0000 7fd2ed7fa700 1 -- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 msgr2=0x7fd2dc0693a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:46:50.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:50.623+0000 7fd2ed7fa700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 0x7fd2dc0693a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:50.732 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:50.731+0000 7f58d3a9d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:50.733 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:50.731+0000 7f58d3a9d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:50.733 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:50.731+0000 7f58d3a9d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:50.733 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:50.731+0000 7f58d3a9d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:50.733 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:50.731+0000 7f58d3a9d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:50.733 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:50.732+0000 7f58d0296700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:50.733 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:50.732+0000 7f58d0296700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:50.733 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:50.732+0000 7f58d0296700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:50.734 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:50.732+0000 7f58d0296700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:50.734 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:50.732+0000 7f58d0296700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:46:50.924 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:50.923+0000 7f9def0491c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:46:51.177 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:51.176+0000 7f9def0491c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:46:51.260 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:51.258+0000 7f9def0491c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:46:51.439 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:46:51.438+0000 7f9def0491c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:46:51.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.626+0000 7fd2ed7fa700 1 -- 172.21.15.46:0/2640407934 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 47) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fd2f0054af0 con 0x7fd2f8159cb0 2024-01-26T23:46:51.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.627+0000 7fd2ed7fa700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 0x7fd2dc0693a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:51.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.627+0000 7fd2ed7fa700 1 -- 172.21.15.46:0/2640407934 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2f804c710 con 0x7fd2dc066ef0 2024-01-26T23:46:51.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.629+0000 7fd2fd0cf700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 0x7fd2dc0693a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:51.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.632+0000 7fd2fd0cf700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 0x7fd2dc0693a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80c9a30 tx=0x7fd2f4019000 comp rx=0 tx=0).ready entity=mgr.7285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:51.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.709+0000 7fd2ed7fa700 1 -- 172.21.15.46:0/2640407934 <== mgr.7285 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fd2f804c710 con 0x7fd2dc066ef0 2024-01-26T23:46:51.710 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path /bad-path 2024-01-26T23:46:51.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.712+0000 7fd2ff333700 1 -- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 msgr2=0x7fd2dc0693a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:51.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.712+0000 7fd2ff333700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 0x7fd2dc0693a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80c9a30 tx=0x7fd2f4019000 comp rx=0 tx=0).stop 2024-01-26T23:46:51.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.712+0000 7fd2ff333700 1 -- 172.21.15.46:0/2640407934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f8159cb0 msgr2=0x7fd2f80b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:51.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.712+0000 7fd2ff333700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f8159cb0 0x7fd2f80b2830 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00011e0 tx=0x7fd2f0007d30 comp rx=0 tx=0).stop 2024-01-26T23:46:51.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.712+0000 7fd2ff333700 1 -- 172.21.15.46:0/2640407934 shutdown_connections 2024-01-26T23:46:51.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.712+0000 7fd2ff333700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2dc066ef0 0x7fd2dc0693a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:51.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.712+0000 7fd2ff333700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f8159cb0 0x7fd2f80b2830 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:51.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.712+0000 7fd2ff333700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f80afb40 0x7fd2f8153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:51.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.712+0000 7fd2ff333700 1 --2- 172.21.15.46:0/2640407934 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f80cf050 0x7fd2f80c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:51.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.712+0000 7fd2ff333700 1 -- 172.21.15.46:0/2640407934 >> 172.21.15.46:0/2640407934 conn(0x7fd2f8087f60 msgr2=0x7fd2f80c65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:51.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.712+0000 7fd2ff333700 1 -- 172.21.15.46:0/2640407934 shutdown_connections 2024-01-26T23:46:51.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.712+0000 7fd2ff333700 1 -- 172.21.15.46:0/2640407934 wait complete. 2024-01-26T23:46:51.729 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2024-01-26T23:46:51.729 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-01-26T23:46:51.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.867+0000 7f16b52e2700 1 Processor -- start 2024-01-26T23:46:51.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.867+0000 7f16b52e2700 1 -- start start 2024-01-26T23:46:51.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.868+0000 7f16b52e2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f16b01599d0 0x7f16b0159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:51.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.868+0000 7f16b52e2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16b00ab1c0 0x7f16b00b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:51.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.868+0000 7f16b52e2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16b015d8c0 0x7f16b015dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:51.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.868+0000 7f16b52e2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b015e260 con 0x7f16b01599d0 2024-01-26T23:46:51.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.868+0000 7f16b52e2700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b00b1a60 con 0x7f16b015d8c0 2024-01-26T23:46:51.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.868+0000 7f16b52e2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b00ab5f0 con 0x7f16b00ab1c0 2024-01-26T23:46:51.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.868+0000 7f16aeffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f16b01599d0 0x7f16b0159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:51.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.868+0000 7f16aeffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f16b01599d0 0x7f16b0159da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55032/0 (socket says 172.21.15.46:55032) 2024-01-26T23:46:51.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.868+0000 7f16aeffd700 1 -- 172.21.15.46:0/200196291 learned_addr learned my addr 172.21.15.46:0/200196291 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:51.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.868+0000 7f16af7fe700 1 --2- 172.21.15.46:0/200196291 >> v2:172.21.15.106:3300/0 conn(0x7f16b015d8c0 0x7f16b015dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:51.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.868+0000 7f16ae7fc700 1 --2- 172.21.15.46:0/200196291 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16b00ab1c0 0x7f16b00b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:51.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.868+0000 7f16aeffd700 1 -- 172.21.15.46:0/200196291 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16b00ab1c0 msgr2=0x7f16b00b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:51.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.868+0000 7f16aeffd700 1 --2- 172.21.15.46:0/200196291 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16b00ab1c0 0x7f16b00b0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:51.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.868+0000 7f16aeffd700 1 -- 172.21.15.46:0/200196291 >> v2:172.21.15.106:3300/0 conn(0x7f16b015d8c0 msgr2=0x7f16b015dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:46:51.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.868+0000 7f16aeffd700 1 --2- 172.21.15.46:0/200196291 >> v2:172.21.15.106:3300/0 conn(0x7f16b015d8c0 0x7f16b015dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:51.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.868+0000 7f16aeffd700 1 -- 172.21.15.46:0/200196291 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b00b0f40 con 0x7f16b01599d0 2024-01-26T23:46:51.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.869+0000 7f16aeffd700 1 --2- 172.21.15.46:0/200196291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b01599d0 0x7f16b0159da0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f16b00686d0 tx=0x7f16a0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0d1206ca017e96 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:51.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.869+0000 7f16adffb700 1 -- 172.21.15.46:0/200196291 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a0005c10 con 0x7f16b01599d0 2024-01-26T23:46:51.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.869+0000 7f16adffb700 1 -- 172.21.15.46:0/200196291 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a0005d70 con 0x7f16b01599d0 2024-01-26T23:46:51.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.869+0000 7f16adffb700 1 -- 172.21.15.46:0/200196291 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a000b880 con 0x7f16b01599d0 2024-01-26T23:46:51.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.869+0000 7f16b52e2700 1 -- 172.21.15.46:0/200196291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b01599d0 msgr2=0x7f16b0159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:51.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.869+0000 7f16b52e2700 1 --2- 172.21.15.46:0/200196291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b01599d0 0x7f16b0159da0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f16b00686d0 tx=0x7f16a0009220 comp rx=0 tx=0).stop 2024-01-26T23:46:51.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.869+0000 7f16b52e2700 1 -- 172.21.15.46:0/200196291 shutdown_connections 2024-01-26T23:46:51.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.869+0000 7f16b52e2700 1 --2- 172.21.15.46:0/200196291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b01599d0 0x7f16b0159da0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:51.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.869+0000 7f16b52e2700 1 --2- 172.21.15.46:0/200196291 >> v2:172.21.15.106:3300/0 conn(0x7f16b015d8c0 0x7f16b015dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:51.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.869+0000 7f16b52e2700 1 --2- 172.21.15.46:0/200196291 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16b00ab1c0 0x7f16b00b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:51.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.869+0000 7f16b52e2700 1 -- 172.21.15.46:0/200196291 >> 172.21.15.46:0/200196291 conn(0x7f16b006f840 msgr2=0x7f16b0087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:51.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.869+0000 7f16b52e2700 1 -- 172.21.15.46:0/200196291 shutdown_connections 2024-01-26T23:46:51.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.870+0000 7f16b52e2700 1 -- 172.21.15.46:0/200196291 wait complete. 2024-01-26T23:46:51.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.871+0000 7f16b52e2700 1 Processor -- start 2024-01-26T23:46:51.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.871+0000 7f16b52e2700 1 -- start start 2024-01-26T23:46:51.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.872+0000 7f16b52e2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b00ab1c0 0x7f16b016ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:51.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.872+0000 7f16b52e2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16b01599d0 0x7f16b00cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:51.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.872+0000 7f16b52e2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16b015d8c0 0x7f16b00ac180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:51.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.872+0000 7f16b52e2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b00cf5f0 con 0x7f16b00ab1c0 2024-01-26T23:46:51.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.872+0000 7f16b52e2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b009bd40 con 0x7f16b01599d0 2024-01-26T23:46:51.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.872+0000 7f16b52e2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b00ac9c0 con 0x7f16b015d8c0 2024-01-26T23:46:51.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.872+0000 7f16aeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b00ab1c0 0x7f16b016ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:51.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.872+0000 7f16aeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b00ab1c0 0x7f16b016ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55038/0 (socket says 172.21.15.46:55038) 2024-01-26T23:46:51.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.872+0000 7f16aeffd700 1 -- 172.21.15.46:0/2593516673 learned_addr learned my addr 172.21.15.46:0/2593516673 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:46:51.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.872+0000 7f16ae7fc700 1 --2- 172.21.15.46:0/2593516673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16b01599d0 0x7f16b00cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:51.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.872+0000 7f16af7fe700 1 --2- 172.21.15.46:0/2593516673 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16b015d8c0 0x7f16b00ac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:51.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.872+0000 7f16aeffd700 1 -- 172.21.15.46:0/2593516673 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16b015d8c0 msgr2=0x7f16b00ac180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:51.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.872+0000 7f16aeffd700 1 --2- 172.21.15.46:0/2593516673 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16b015d8c0 0x7f16b00ac180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:51.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.872+0000 7f16aeffd700 1 -- 172.21.15.46:0/2593516673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16b01599d0 msgr2=0x7f16b00cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:51.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.872+0000 7f16aeffd700 1 --2- 172.21.15.46:0/2593516673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16b01599d0 0x7f16b00cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:51.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.872+0000 7f16aeffd700 1 -- 172.21.15.46:0/2593516673 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a0009480 con 0x7f16b00ab1c0 2024-01-26T23:46:51.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.873+0000 7f16aeffd700 1 --2- 172.21.15.46:0/2593516673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b00ab1c0 0x7f16b016ad00 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f16a000b3c0 tx=0x7f16a0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:51.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.873+0000 7f1696ffd700 1 -- 172.21.15.46:0/2593516673 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a0011740 con 0x7f16b00ab1c0 2024-01-26T23:46:51.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.873+0000 7f1696ffd700 1 -- 172.21.15.46:0/2593516673 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a0020070 con 0x7f16b00ab1c0 2024-01-26T23:46:51.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.873+0000 7f16b52e2700 1 -- 172.21.15.46:0/2593516673 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b00acca0 con 0x7f16b00ab1c0 2024-01-26T23:46:51.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.873+0000 7f1696ffd700 1 -- 172.21.15.46:0/2593516673 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a0011ad0 con 0x7f16b00ab1c0 2024-01-26T23:46:51.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.873+0000 7f16b52e2700 1 -- 172.21.15.46:0/2593516673 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b00ad260 con 0x7f16b00ab1c0 2024-01-26T23:46:51.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.874+0000 7f16b52e2700 1 -- 172.21.15.46:0/2593516673 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b00ad3f0 con 0x7f16b00ab1c0 2024-01-26T23:46:51.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.874+0000 7f1696ffd700 1 -- 172.21.15.46:0/2593516673 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 47) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f16a0022030 con 0x7f16b00ab1c0 2024-01-26T23:46:51.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.875+0000 7f1696ffd700 1 --2- 172.21.15.46:0/2593516673 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f169c066dd0 0x7f169c069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:46:51.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.875+0000 7f1696ffd700 1 -- 172.21.15.46:0/2593516673 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f16a0086130 con 0x7f16b00ab1c0 2024-01-26T23:46:51.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.875+0000 7f16ae7fc700 1 --2- 172.21.15.46:0/2593516673 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f169c066dd0 0x7f169c069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:46:51.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.875+0000 7f16ae7fc700 1 --2- 172.21.15.46:0/2593516673 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f169c066dd0 0x7f169c069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f16b0158ec0 tx=0x7f16a4008040 comp rx=0 tx=0).ready entity=mgr.7285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:46:51.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:51.879+0000 7f1696ffd700 1 -- 172.21.15.46:0/2593516673 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f16a0053280 con 0x7f16b00ab1c0 2024-01-26T23:46:52.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:52.110+0000 7f16b52e2700 1 -- 172.21.15.46:0/2593516673 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16b00b11c0 con 0x7f169c066dd0 2024-01-26T23:46:52.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:52.121+0000 7f1696ffd700 1 -- 172.21.15.46:0/2593516673 <== mgr.7285 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+287 (secure 0 0 0) 0x7f16b00b11c0 con 0x7f169c066dd0 2024-01-26T23:46:52.121 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:46:52.122 INFO:teuthology.orchestra.run.smithi046.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:46:51", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-01-26T23:46:52.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:52.122+0000 7f16b52e2700 1 -- 172.21.15.46:0/2593516673 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f169c066dd0 msgr2=0x7f169c069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:52.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:52.122+0000 7f16b52e2700 1 --2- 172.21.15.46:0/2593516673 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f169c066dd0 0x7f169c069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f16b0158ec0 tx=0x7f16a4008040 comp rx=0 tx=0).stop 2024-01-26T23:46:52.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:52.122+0000 7f16b52e2700 1 -- 172.21.15.46:0/2593516673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b00ab1c0 msgr2=0x7f16b016ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:46:52.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:52.122+0000 7f16b52e2700 1 --2- 172.21.15.46:0/2593516673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b00ab1c0 0x7f16b016ad00 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f16a000b3c0 tx=0x7f16a0016000 comp rx=0 tx=0).stop 2024-01-26T23:46:52.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:52.122+0000 7f16b52e2700 1 -- 172.21.15.46:0/2593516673 shutdown_connections 2024-01-26T23:46:52.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:52.122+0000 7f16b52e2700 1 --2- 172.21.15.46:0/2593516673 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f169c066dd0 0x7f169c069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:52.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:52.122+0000 7f16b52e2700 1 --2- 172.21.15.46:0/2593516673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b00ab1c0 0x7f16b016ad00 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:52.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:52.122+0000 7f16b52e2700 1 --2- 172.21.15.46:0/2593516673 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16b015d8c0 0x7f16b00ac180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:52.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:52.122+0000 7f16b52e2700 1 --2- 172.21.15.46:0/2593516673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16b01599d0 0x7f16b00cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:46:52.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:52.122+0000 7f16b52e2700 1 -- 172.21.15.46:0/2593516673 >> 172.21.15.46:0/2593516673 conn(0x7f16b006f840 msgr2=0x7f16b00d0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:46:52.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:52.122+0000 7f16b52e2700 1 -- 172.21.15.46:0/2593516673 shutdown_connections 2024-01-26T23:46:52.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:46:52.123+0000 7f16b52e2700 1 -- 172.21.15.46:0/2593516673 wait complete. 2024-01-26T23:46:52.138 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:46:51", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-01-26T23:46:52.671 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:52.670+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:46:53.579 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:53.578+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:46:54.477 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:54.476+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:46:55.344 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:46:55.347 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:46:55.384 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:55.383+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:46:56.279 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:56.278+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:46:57.183 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:57.181+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:46:58.083 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:58.083+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:46:58.988 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:58.987+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:46:59.897 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:46:59.897+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:47:00.717 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:00.717+0000 7f58c327c700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:47:00.717 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:00.717+0000 7f58c327c700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:47:00.717 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:00.717+0000 7f58c327c700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:47:00.718 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:00.717+0000 7f58c327c700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:47:00.718 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:00.717+0000 7f58c327c700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:47:00.821 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:00.821+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:47:01.744 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:01.744+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:47:02.667 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:02.666+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:47:03.596 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:03.595+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:47:04.527 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:04.526+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:47:05.466 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:05.465+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:47:06.410 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:06.409+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:47:07.374 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:07.372+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:47:08.318 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:08.317+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:47:09.269 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:09.268+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:47:10.224 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:10.223+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:47:11.183 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:11.180+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:47:12.156 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:12.155+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:47:13.124 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:13.123+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:47:14.096 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:14.095+0000 7f58edb51700 -1 client.6698: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.6991:ee1dbb23-9240-44fb-96c1-efc90410383e:v1:172.21.15.106:0/2301684208] 2024-01-26T23:47:25.417 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:47:25.421 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:47:52.139 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-01-26T23:47:52.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.281+0000 7f3542e82700 1 Processor -- start 2024-01-26T23:47:52.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.281+0000 7f3542e82700 1 -- start start 2024-01-26T23:47:52.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.282+0000 7f3542e82700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f353c141100 0x7f353c140af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:52.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.282+0000 7f3542e82700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353c0cf2b0 con 0x7f353c0b34e0 2024-01-26T23:47:52.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.282+0000 7f3542e82700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353c0b3050 con 0x7f353c140720 2024-01-26T23:47:52.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.282+0000 7f3542e82700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f353c0b2440 con 0x7f353c141100 2024-01-26T23:47:52.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.282+0000 7f354141f700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f353c0b34e0 0x7f353c06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59496/0 (socket says 172.21.15.46:59496) 2024-01-26T23:47:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.282+0000 7f354141f700 1 -- 172.21.15.46:0/3988907631 learned_addr learned my addr 172.21.15.46:0/3988907631 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:47:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.282+0000 7f353bfff700 1 --2- 172.21.15.46:0/3988907631 >> v2:172.21.15.106:3300/0 conn(0x7f353c141100 0x7f353c140af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.282+0000 7f353b7fe700 1 -- 172.21.15.46:0/3988907631 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212920390 0 0) 0x7f353c0cf2b0 con 0x7f353c0b34e0 2024-01-26T23:47:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.282+0000 7f353b7fe700 1 -- 172.21.15.46:0/3988907631 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f352c003570 con 0x7f353c0b34e0 2024-01-26T23:47:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.283+0000 7f353bfff700 1 -- 172.21.15.46:0/3988907631 >> v1:172.21.15.106:6789/0 conn(0x7f353c140720 legacy=0x7f353c0b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.283+0000 7f353b7fe700 1 -- 172.21.15.46:0/3988907631 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3626682607 0 0) 0x7f352c003570 con 0x7f353c0b34e0 2024-01-26T23:47:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.283+0000 7f353bfff700 1 -- 172.21.15.46:0/3988907631 >> v1:172.21.15.46:6789/0 conn(0x7f353c0b34e0 legacy=0x7f353c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.283+0000 7f353bfff700 1 -- 172.21.15.46:0/3988907631 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353c0b2720 con 0x7f353c141100 2024-01-26T23:47:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.283+0000 7f353bfff700 1 --2- 172.21.15.46:0/3988907631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f353c141100 0x7f353c140af0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f35280098e0 tx=0x7f35280091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=997f3aa179ca002d server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.283+0000 7f353b7fe700 1 -- 172.21.15.46:0/3988907631 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35340017f0 con 0x7f353c141100 2024-01-26T23:47:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.283+0000 7f353b7fe700 1 -- 172.21.15.46:0/3988907631 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3528011420 con 0x7f353c141100 2024-01-26T23:47:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.283+0000 7f353b7fe700 1 -- 172.21.15.46:0/3988907631 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35280117c0 con 0x7f353c141100 2024-01-26T23:47:52.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.284+0000 7f3542e82700 1 -- 172.21.15.46:0/3988907631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f353c141100 msgr2=0x7f353c140af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:52.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.284+0000 7f3542e82700 1 --2- 172.21.15.46:0/3988907631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f353c141100 0x7f353c140af0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f35280098e0 tx=0x7f35280091a0 comp rx=0 tx=0).stop 2024-01-26T23:47:52.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.284+0000 7f3542e82700 1 -- 172.21.15.46:0/3988907631 shutdown_connections 2024-01-26T23:47:52.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.284+0000 7f3542e82700 1 --2- 172.21.15.46:0/3988907631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f353c141100 0x7f353c140af0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:52.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.284+0000 7f3542e82700 1 -- 172.21.15.46:0/3988907631 >> 172.21.15.46:0/3988907631 conn(0x7f353c067930 msgr2=0x7f353c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:52.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.284+0000 7f3542e82700 1 -- 172.21.15.46:0/3988907631 shutdown_connections 2024-01-26T23:47:52.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.284+0000 7f3542e82700 1 -- 172.21.15.46:0/3988907631 wait complete. 2024-01-26T23:47:52.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.285+0000 7f3542e82700 1 Processor -- start 2024-01-26T23:47:52.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.286+0000 7f3542e82700 1 -- start start 2024-01-26T23:47:52.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.286+0000 7f3542e82700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f353c0b34e0 0x7f353c13ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:52.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f3542e82700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353c140720 0x7f353c145980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:52.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f3542e82700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f353c141100 0x7f353c13cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:52.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f3542e82700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353c146420 con 0x7f353c140720 2024-01-26T23:47:52.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f3542e82700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353c146700 con 0x7f353c141100 2024-01-26T23:47:52.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f3542e82700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f353c1469e0 con 0x7f353c0b34e0 2024-01-26T23:47:52.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f353bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353c140720 0x7f353c145980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:52.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f353bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353c140720 0x7f353c145980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41090/0 (socket says 172.21.15.46:41090) 2024-01-26T23:47:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f353bfff700 1 -- 172.21.15.46:0/4294432327 learned_addr learned my addr 172.21.15.46:0/4294432327 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f3540c1e700 1 --2- 172.21.15.46:0/4294432327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f353c0b34e0 0x7f353c13ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f354141f700 1 --2- 172.21.15.46:0/4294432327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f353c141100 0x7f353c13cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f353bfff700 1 -- 172.21.15.46:0/4294432327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f353c0b34e0 msgr2=0x7f353c13ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f353bfff700 1 --2- 172.21.15.46:0/4294432327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f353c0b34e0 0x7f353c13ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f353bfff700 1 -- 172.21.15.46:0/4294432327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f353c141100 msgr2=0x7f353c13cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f353bfff700 1 --2- 172.21.15.46:0/4294432327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f353c141100 0x7f353c13cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f353bfff700 1 -- 172.21.15.46:0/4294432327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3528009420 con 0x7f353c140720 2024-01-26T23:47:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f353bfff700 1 --2- 172.21.15.46:0/4294432327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353c140720 0x7f353c145980 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f3528002000 tx=0x7f3528017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f3538ff9700 1 -- 172.21.15.46:0/4294432327 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35280091a0 con 0x7f353c140720 2024-01-26T23:47:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.287+0000 7f3538ff9700 1 -- 172.21.15.46:0/4294432327 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3528010ac0 con 0x7f353c140720 2024-01-26T23:47:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.288+0000 7f3542e82700 1 -- 172.21.15.46:0/4294432327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f353c146cc0 con 0x7f353c140720 2024-01-26T23:47:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.288+0000 7f3538ff9700 1 -- 172.21.15.46:0/4294432327 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3528010e80 con 0x7f353c140720 2024-01-26T23:47:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.288+0000 7f3542e82700 1 -- 172.21.15.46:0/4294432327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f353c1592c0 con 0x7f353c140720 2024-01-26T23:47:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.288+0000 7f3542e82700 1 -- 172.21.15.46:0/4294432327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f353c1471e0 con 0x7f353c140720 2024-01-26T23:47:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.289+0000 7f3538ff9700 1 -- 172.21.15.46:0/4294432327 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 50) v1 ==== 84975+0+0 (secure 0 0 0) 0x7f352801d640 con 0x7f353c140720 2024-01-26T23:47:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.289+0000 7f3538ff9700 1 --2- 172.21.15.46:0/4294432327 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f35200671c0 0x7f3520069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.290+0000 7f3538ff9700 1 -- 172.21.15.46:0/4294432327 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6791+0+0 (secure 0 0 0) 0x7f35280869f0 con 0x7f353c140720 2024-01-26T23:47:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.290+0000 7f3540c1e700 1 --2- 172.21.15.46:0/4294432327 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f35200671c0 0x7f3520069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.290+0000 7f3540c1e700 1 --2- 172.21.15.46:0/4294432327 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f35200671c0 0x7f3520069670 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3530000f60 tx=0x7f3530007ea0 comp rx=0 tx=0).ready entity=mgr.7285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:52.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.294+0000 7f3538ff9700 1 -- 172.21.15.46:0/4294432327 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f353c1471e0 con 0x7f353c140720 2024-01-26T23:47:52.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.529+0000 7f3542e82700 1 -- 172.21.15.46:0/4294432327 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f353c1471e0 con 0x7f35200671c0 2024-01-26T23:47:52.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.536+0000 7f3538ff9700 1 -- 172.21.15.46:0/4294432327 <== mgr.7285 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+288 (secure 0 0 0) 0x7f353c1471e0 con 0x7f35200671c0 2024-01-26T23:47:52.537 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:47:52.537 INFO:teuthology.orchestra.run.smithi046.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:46:51", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-01-26T23:47:52.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.538+0000 7f3542e82700 1 -- 172.21.15.46:0/4294432327 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f35200671c0 msgr2=0x7f3520069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:52.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.538+0000 7f3542e82700 1 --2- 172.21.15.46:0/4294432327 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f35200671c0 0x7f3520069670 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3530000f60 tx=0x7f3530007ea0 comp rx=0 tx=0).stop 2024-01-26T23:47:52.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.538+0000 7f3542e82700 1 -- 172.21.15.46:0/4294432327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353c140720 msgr2=0x7f353c145980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:52.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.538+0000 7f3542e82700 1 --2- 172.21.15.46:0/4294432327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353c140720 0x7f353c145980 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f3528002000 tx=0x7f3528017000 comp rx=0 tx=0).stop 2024-01-26T23:47:52.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.538+0000 7f3542e82700 1 -- 172.21.15.46:0/4294432327 shutdown_connections 2024-01-26T23:47:52.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.538+0000 7f3542e82700 1 --2- 172.21.15.46:0/4294432327 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f35200671c0 0x7f3520069670 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:52.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.538+0000 7f3542e82700 1 --2- 172.21.15.46:0/4294432327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f353c141100 0x7f353c13cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:52.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.538+0000 7f3542e82700 1 --2- 172.21.15.46:0/4294432327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f353c0b34e0 0x7f353c13ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:52.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.538+0000 7f3542e82700 1 --2- 172.21.15.46:0/4294432327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353c140720 0x7f353c145980 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:52.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.538+0000 7f3542e82700 1 -- 172.21.15.46:0/4294432327 >> 172.21.15.46:0/4294432327 conn(0x7f353c067930 msgr2=0x7f353c15ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:52.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.539+0000 7f3542e82700 1 -- 172.21.15.46:0/4294432327 shutdown_connections 2024-01-26T23:47:52.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.539+0000 7f3542e82700 1 -- 172.21.15.46:0/4294432327 wait complete. 2024-01-26T23:47:52.553 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:46:51", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-01-26T23:47:52.553 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /bad-path 2024-01-26T23:47:52.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.685+0000 7f2a2a4ba700 1 Processor -- start 2024-01-26T23:47:52.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.685+0000 7f2a2a4ba700 1 -- start start 2024-01-26T23:47:52.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.686+0000 7f2a2a4ba700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2a241565a0 0x7f2a24156970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:52.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.686+0000 7f2a2a4ba700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a2415a520 0x7f2a240acc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:52.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.686+0000 7f2a2a4ba700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a240ae2f0 0x7f2a240ae750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:52.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.686+0000 7f2a2a4ba700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a240ae130 con 0x7f2a2415a520 2024-01-26T23:47:52.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.686+0000 7f2a2a4ba700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a240b0830 con 0x7f2a241565a0 2024-01-26T23:47:52.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.686+0000 7f2a2a4ba700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a240aec90 con 0x7f2a240ae2f0 2024-01-26T23:47:52.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.686+0000 7f2a237fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a2415a520 0x7f2a240acc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:52.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.686+0000 7f2a237fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a2415a520 0x7f2a240acc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41102/0 (socket says 172.21.15.46:41102) 2024-01-26T23:47:52.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.686+0000 7f2a237fe700 1 -- 172.21.15.46:0/1246529234 learned_addr learned my addr 172.21.15.46:0/1246529234 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:52.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.686+0000 7f2a23fff700 1 --2- 172.21.15.46:0/1246529234 >> v2:172.21.15.106:3300/0 conn(0x7f2a241565a0 0x7f2a24156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:52.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.686+0000 7f2a28a57700 1 --2- 172.21.15.46:0/1246529234 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a240ae2f0 0x7f2a240ae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:52.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.686+0000 7f2a237fe700 1 -- 172.21.15.46:0/1246529234 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a240ae2f0 msgr2=0x7f2a240ae750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:47:52.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.686+0000 7f2a237fe700 1 --2- 172.21.15.46:0/1246529234 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a240ae2f0 0x7f2a240ae750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:52.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.686+0000 7f2a237fe700 1 -- 172.21.15.46:0/1246529234 >> v2:172.21.15.106:3300/0 conn(0x7f2a241565a0 msgr2=0x7f2a24156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:47:52.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.686+0000 7f2a237fe700 1 --2- 172.21.15.46:0/1246529234 >> v2:172.21.15.106:3300/0 conn(0x7f2a241565a0 0x7f2a24156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:52.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.686+0000 7f2a237fe700 1 -- 172.21.15.46:0/1246529234 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a240ad5b0 con 0x7f2a2415a520 2024-01-26T23:47:52.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.686+0000 7f2a237fe700 1 --2- 172.21.15.46:0/1246529234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a2415a520 0x7f2a240acc40 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f2a24064a50 tx=0x7f2a18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1374de4ced89fede server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:52.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.687+0000 7f2a22ffd700 1 -- 172.21.15.46:0/1246529234 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1800b640 con 0x7f2a2415a520 2024-01-26T23:47:52.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.687+0000 7f2a22ffd700 1 -- 172.21.15.46:0/1246529234 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1800b7a0 con 0x7f2a2415a520 2024-01-26T23:47:52.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.687+0000 7f2a22ffd700 1 -- 172.21.15.46:0/1246529234 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1800bb60 con 0x7f2a2415a520 2024-01-26T23:47:52.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.687+0000 7f2a2a4ba700 1 -- 172.21.15.46:0/1246529234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a2415a520 msgr2=0x7f2a240acc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:52.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.687+0000 7f2a2a4ba700 1 --2- 172.21.15.46:0/1246529234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a2415a520 0x7f2a240acc40 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f2a24064a50 tx=0x7f2a18009250 comp rx=0 tx=0).stop 2024-01-26T23:47:52.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.687+0000 7f2a2a4ba700 1 -- 172.21.15.46:0/1246529234 shutdown_connections 2024-01-26T23:47:52.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.687+0000 7f2a2a4ba700 1 --2- 172.21.15.46:0/1246529234 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a240ae2f0 0x7f2a240ae750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:52.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.687+0000 7f2a2a4ba700 1 --2- 172.21.15.46:0/1246529234 >> v2:172.21.15.106:3300/0 conn(0x7f2a241565a0 0x7f2a24156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:52.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.687+0000 7f2a2a4ba700 1 --2- 172.21.15.46:0/1246529234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a2415a520 0x7f2a240acc40 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:52.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.687+0000 7f2a2a4ba700 1 -- 172.21.15.46:0/1246529234 >> 172.21.15.46:0/1246529234 conn(0x7f2a240cdf80 msgr2=0x7f2a2406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:52.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.687+0000 7f2a2a4ba700 1 -- 172.21.15.46:0/1246529234 shutdown_connections 2024-01-26T23:47:52.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.687+0000 7f2a2a4ba700 1 -- 172.21.15.46:0/1246529234 wait complete. 2024-01-26T23:47:52.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.689+0000 7f2a2a4ba700 1 Processor -- start 2024-01-26T23:47:52.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.689+0000 7f2a2a4ba700 1 -- start start 2024-01-26T23:47:52.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a2a4ba700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a240ae2f0 0x7f2a24150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:52.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a2a4ba700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a241565a0 0x7f2a24156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:52.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a2a4ba700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a2415a520 0x7f2a24153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:52.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a2a4ba700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a241578a0 con 0x7f2a240ae2f0 2024-01-26T23:47:52.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a2a4ba700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a24157f20 con 0x7f2a241565a0 2024-01-26T23:47:52.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a2a4ba700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a241572e0 con 0x7f2a2415a520 2024-01-26T23:47:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a23fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a240ae2f0 0x7f2a24150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a23fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a240ae2f0 0x7f2a24150870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41116/0 (socket says 172.21.15.46:41116) 2024-01-26T23:47:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a23fff700 1 -- 172.21.15.46:0/4232448308 learned_addr learned my addr 172.21.15.46:0/4232448308 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a28a57700 1 --2- 172.21.15.46:0/4232448308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a2415a520 0x7f2a24153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a237fe700 1 --2- 172.21.15.46:0/4232448308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a241565a0 0x7f2a24156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a23fff700 1 -- 172.21.15.46:0/4232448308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a2415a520 msgr2=0x7f2a24153c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a23fff700 1 --2- 172.21.15.46:0/4232448308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a2415a520 0x7f2a24153c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a23fff700 1 -- 172.21.15.46:0/4232448308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a241565a0 msgr2=0x7f2a24156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a23fff700 1 --2- 172.21.15.46:0/4232448308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a241565a0 0x7f2a24156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a23fff700 1 -- 172.21.15.46:0/4232448308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a180094f0 con 0x7f2a240ae2f0 2024-01-26T23:47:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a28a57700 1 --2- 172.21.15.46:0/4232448308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a2415a520 0x7f2a24153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:47:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a23fff700 1 --2- 172.21.15.46:0/4232448308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a240ae2f0 0x7f2a24150870 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f2a1400ecb0 tx=0x7f2a140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a0bfff700 1 -- 172.21.15.46:0/4232448308 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1400c920 con 0x7f2a240ae2f0 2024-01-26T23:47:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a0bfff700 1 -- 172.21.15.46:0/4232448308 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1400ca80 con 0x7f2a240ae2f0 2024-01-26T23:47:52.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a2a4ba700 1 -- 172.21.15.46:0/4232448308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a241575c0 con 0x7f2a240ae2f0 2024-01-26T23:47:52.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.690+0000 7f2a0bfff700 1 -- 172.21.15.46:0/4232448308 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1400ce20 con 0x7f2a240ae2f0 2024-01-26T23:47:52.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.691+0000 7f2a2a4ba700 1 -- 172.21.15.46:0/4232448308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a24150db0 con 0x7f2a240ae2f0 2024-01-26T23:47:52.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.691+0000 7f2a2a4ba700 1 -- 172.21.15.46:0/4232448308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2404c710 con 0x7f2a240ae2f0 2024-01-26T23:47:52.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.692+0000 7f2a0bfff700 1 -- 172.21.15.46:0/4232448308 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 50) v1 ==== 84975+0+0 (secure 0 0 0) 0x7f2a14013680 con 0x7f2a240ae2f0 2024-01-26T23:47:52.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.692+0000 7f2a0bfff700 1 --2- 172.21.15.46:0/4232448308 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a100671c0 0x7f2a10069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:52.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.692+0000 7f2a0bfff700 1 -- 172.21.15.46:0/4232448308 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6791+0+0 (secure 0 0 0) 0x7f2a14085000 con 0x7f2a240ae2f0 2024-01-26T23:47:52.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.693+0000 7f2a237fe700 1 --2- 172.21.15.46:0/4232448308 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a100671c0 0x7f2a10069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:52.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.693+0000 7f2a237fe700 1 --2- 172.21.15.46:0/4232448308 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a100671c0 0x7f2a10069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2a24064a50 tx=0x7f2a18017000 comp rx=0 tx=0).ready entity=mgr.7285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:52.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.697+0000 7f2a0bfff700 1 -- 172.21.15.46:0/4232448308 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2a14051ba0 con 0x7f2a240ae2f0 2024-01-26T23:47:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:52.930+0000 7f2a2a4ba700 1 -- 172.21.15.46:0/4232448308 --> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/bad-path", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a24158060 con 0x7f2a100671c0 2024-01-26T23:47:53.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.001+0000 7f2a0bfff700 1 -- 172.21.15.46:0/4232448308 <== mgr.7285 v2:172.21.15.106:6832/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+36 (secure 0 0 0) 0x7f2a18000c50 con 0x7f2a100671c0 2024-01-26T23:47:53.002 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path /bad-path 2024-01-26T23:47:53.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.003+0000 7f2a2a4ba700 1 -- 172.21.15.46:0/4232448308 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a100671c0 msgr2=0x7f2a10069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:53.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.003+0000 7f2a2a4ba700 1 --2- 172.21.15.46:0/4232448308 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a100671c0 0x7f2a10069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2a24064a50 tx=0x7f2a18017000 comp rx=0 tx=0).stop 2024-01-26T23:47:53.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.003+0000 7f2a2a4ba700 1 -- 172.21.15.46:0/4232448308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a240ae2f0 msgr2=0x7f2a24150870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:53.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.003+0000 7f2a2a4ba700 1 --2- 172.21.15.46:0/4232448308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a240ae2f0 0x7f2a24150870 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f2a1400ecb0 tx=0x7f2a140091a0 comp rx=0 tx=0).stop 2024-01-26T23:47:53.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.003+0000 7f2a2a4ba700 1 -- 172.21.15.46:0/4232448308 shutdown_connections 2024-01-26T23:47:53.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.003+0000 7f2a2a4ba700 1 --2- 172.21.15.46:0/4232448308 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f2a100671c0 0x7f2a10069670 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:53.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.003+0000 7f2a2a4ba700 1 --2- 172.21.15.46:0/4232448308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a240ae2f0 0x7f2a24150870 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:53.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.003+0000 7f2a2a4ba700 1 --2- 172.21.15.46:0/4232448308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a2415a520 0x7f2a24153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:53.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.003+0000 7f2a2a4ba700 1 --2- 172.21.15.46:0/4232448308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a241565a0 0x7f2a24156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:53.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.003+0000 7f2a2a4ba700 1 -- 172.21.15.46:0/4232448308 >> 172.21.15.46:0/4232448308 conn(0x7f2a240cdf80 msgr2=0x7f2a240cb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:53.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.004+0000 7f2a2a4ba700 1 -- 172.21.15.46:0/4232448308 shutdown_connections 2024-01-26T23:47:53.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.004+0000 7f2a2a4ba700 1 -- 172.21.15.46:0/4232448308 wait complete. 2024-01-26T23:47:53.019 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2024-01-26T23:47:53.019 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-26T23:47:53.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.154+0000 7fa3ac3cd700 1 Processor -- start 2024-01-26T23:47:53.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.154+0000 7fa3ac3cd700 1 -- start start 2024-01-26T23:47:53.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.155+0000 7fa3ac3cd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3a4159aa0 0x7fa3a40cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:53.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.155+0000 7fa3ac3cd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa3a4159e70 0x7fa3a4067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:53.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.155+0000 7fa3ac3cd700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a40cce30 con 0x7fa3a40cd2c0 2024-01-26T23:47:53.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.155+0000 7fa3ac3cd700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a40c9970 con 0x7fa3a4159aa0 2024-01-26T23:47:53.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.155+0000 7fa3ac3cd700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a40c9c50 con 0x7fa3a4159e70 2024-01-26T23:47:53.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.155+0000 7fa3aa169700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3a4159aa0 0x7fa3a40cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:53.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.155+0000 7fa3aa169700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3a4159aa0 0x7fa3a40cdff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41122/0 (socket says 172.21.15.46:41122) 2024-01-26T23:47:53.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.155+0000 7fa3aa169700 1 -- 172.21.15.46:0/1173552907 learned_addr learned my addr 172.21.15.46:0/1173552907 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:53.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.155+0000 7fa3aa96a700 1 --2- 172.21.15.46:0/1173552907 >> v2:172.21.15.106:3300/0 conn(0x7fa3a4159e70 0x7fa3a4067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:53.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.155+0000 7fa3aa169700 1 -- 172.21.15.46:0/1173552907 >> v2:172.21.15.106:3300/0 conn(0x7fa3a4159e70 msgr2=0x7fa3a4067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:47:53.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.155+0000 7fa3aa169700 1 --2- 172.21.15.46:0/1173552907 >> v2:172.21.15.106:3300/0 conn(0x7fa3a4159e70 0x7fa3a4067930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.156+0000 7fa3aa169700 1 -- 172.21.15.46:0/1173552907 >> v1:172.21.15.106:6789/0 conn(0x7fa3a40cd2c0 legacy=0x7fa3a40c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.156+0000 7fa3aa169700 1 -- 172.21.15.46:0/1173552907 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a40c9620 con 0x7fa3a4159aa0 2024-01-26T23:47:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.156+0000 7fa3aa96a700 1 --2- 172.21.15.46:0/1173552907 >> v2:172.21.15.106:3300/0 conn(0x7fa3a4159e70 0x7fa3a4067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:47:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.156+0000 7fa3aa169700 1 --2- 172.21.15.46:0/1173552907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a4159aa0 0x7fa3a40cdff0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fa3a406e1f0 tx=0x7fa3a0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582b7c4bbd2e3302 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.156+0000 7fa3a9167700 1 -- 172.21.15.46:0/1173552907 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0005c10 con 0x7fa3a4159aa0 2024-01-26T23:47:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.156+0000 7fa3a9167700 1 -- 172.21.15.46:0/1173552907 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a0005d70 con 0x7fa3a4159aa0 2024-01-26T23:47:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.156+0000 7fa3a9167700 1 -- 172.21.15.46:0/1173552907 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a000b8a0 con 0x7fa3a4159aa0 2024-01-26T23:47:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.156+0000 7fa3ac3cd700 1 -- 172.21.15.46:0/1173552907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a4159aa0 msgr2=0x7fa3a40cdff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.156+0000 7fa3ac3cd700 1 --2- 172.21.15.46:0/1173552907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a4159aa0 0x7fa3a40cdff0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fa3a406e1f0 tx=0x7fa3a0009220 comp rx=0 tx=0).stop 2024-01-26T23:47:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.157+0000 7fa3ac3cd700 1 -- 172.21.15.46:0/1173552907 shutdown_connections 2024-01-26T23:47:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.157+0000 7fa3ac3cd700 1 --2- 172.21.15.46:0/1173552907 >> v2:172.21.15.106:3300/0 conn(0x7fa3a4159e70 0x7fa3a4067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.157+0000 7fa3ac3cd700 1 --2- 172.21.15.46:0/1173552907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a4159aa0 0x7fa3a40cdff0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.157+0000 7fa3ac3cd700 1 -- 172.21.15.46:0/1173552907 >> 172.21.15.46:0/1173552907 conn(0x7fa3a4069220 msgr2=0x7fa3a40cd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:53.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.157+0000 7fa3ac3cd700 1 -- 172.21.15.46:0/1173552907 shutdown_connections 2024-01-26T23:47:53.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.157+0000 7fa3ac3cd700 1 -- 172.21.15.46:0/1173552907 wait complete. 2024-01-26T23:47:53.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.158+0000 7fa3ac3cd700 1 Processor -- start 2024-01-26T23:47:53.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.158+0000 7fa3ac3cd700 1 -- start start 2024-01-26T23:47:53.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa3ac3cd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a40cd2c0 0x7fa3a4156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:53.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa3ac3cd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3a4159aa0 0x7fa3a406c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:53.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa3ac3cd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3a4159e70 0x7fa3a406c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:53.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa3ac3cd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a415b510 con 0x7fa3a40cd2c0 2024-01-26T23:47:53.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa3ac3cd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a415b080 con 0x7fa3a4159aa0 2024-01-26T23:47:53.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa3ac3cd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a415a470 con 0x7fa3a4159e70 2024-01-26T23:47:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa3aa169700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a40cd2c0 0x7fa3a4156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa3aa169700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a40cd2c0 0x7fa3a4156b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41128/0 (socket says 172.21.15.46:41128) 2024-01-26T23:47:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa3aa169700 1 -- 172.21.15.46:0/2447781102 learned_addr learned my addr 172.21.15.46:0/2447781102 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa3aa96a700 1 --2- 172.21.15.46:0/2447781102 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3a4159e70 0x7fa3a406c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa3a9968700 1 --2- 172.21.15.46:0/2447781102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3a4159aa0 0x7fa3a406c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa3aa169700 1 -- 172.21.15.46:0/2447781102 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3a4159e70 msgr2=0x7fa3a406c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa3aa169700 1 --2- 172.21.15.46:0/2447781102 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3a4159e70 0x7fa3a406c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa3aa169700 1 -- 172.21.15.46:0/2447781102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3a4159aa0 msgr2=0x7fa3a406c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa3aa169700 1 --2- 172.21.15.46:0/2447781102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3a4159aa0 0x7fa3a406c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa3aa169700 1 -- 172.21.15.46:0/2447781102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a0009480 con 0x7fa3a40cd2c0 2024-01-26T23:47:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa3aa169700 1 --2- 172.21.15.46:0/2447781102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a40cd2c0 0x7fa3a4156b70 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fa3a406dfe0 tx=0x7fa3a0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.159+0000 7fa39a7fc700 1 -- 172.21.15.46:0/2447781102 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a001a420 con 0x7fa3a40cd2c0 2024-01-26T23:47:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.160+0000 7fa39a7fc700 1 -- 172.21.15.46:0/2447781102 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a001a580 con 0x7fa3a40cd2c0 2024-01-26T23:47:53.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.160+0000 7fa3ac3cd700 1 -- 172.21.15.46:0/2447781102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a415a750 con 0x7fa3a40cd2c0 2024-01-26T23:47:53.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.160+0000 7fa39a7fc700 1 -- 172.21.15.46:0/2447781102 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a001a940 con 0x7fa3a40cd2c0 2024-01-26T23:47:53.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.160+0000 7fa3ac3cd700 1 -- 172.21.15.46:0/2447781102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a415ad10 con 0x7fa3a40cd2c0 2024-01-26T23:47:53.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.161+0000 7fa3ac3cd700 1 -- 172.21.15.46:0/2447781102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a415b360 con 0x7fa3a40cd2c0 2024-01-26T23:47:53.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.161+0000 7fa39a7fc700 1 -- 172.21.15.46:0/2447781102 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 50) v1 ==== 84975+0+0 (secure 0 0 0) 0x7fa3a0011420 con 0x7fa3a40cd2c0 2024-01-26T23:47:53.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.162+0000 7fa39a7fc700 1 --2- 172.21.15.46:0/2447781102 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa3880671c0 0x7fa388069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:53.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.162+0000 7fa39a7fc700 1 -- 172.21.15.46:0/2447781102 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6791+0+0 (secure 0 0 0) 0x7fa3a0086270 con 0x7fa3a40cd2c0 2024-01-26T23:47:53.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.162+0000 7fa3a9968700 1 --2- 172.21.15.46:0/2447781102 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa3880671c0 0x7fa388069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:53.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.163+0000 7fa3a9968700 1 --2- 172.21.15.46:0/2447781102 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa3880671c0 0x7fa388069670 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa3a409b390 tx=0x7fa394007ec0 comp rx=0 tx=0).ready entity=mgr.7285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:53.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.166+0000 7fa39a7fc700 1 -- 172.21.15.46:0/2447781102 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa3a0052d40 con 0x7fa3a40cd2c0 2024-01-26T23:47:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.426+0000 7fa3ac3cd700 1 -- 172.21.15.46:0/2447781102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fa3a404c710 con 0x7fa3a40cd2c0 2024-01-26T23:47:53.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.733+0000 7fa39a7fc700 1 -- 172.21.15.46:0/2447781102 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 51) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fa3a0053110 con 0x7fa3a40cd2c0 2024-01-26T23:47:53.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.734+0000 7fa39a7fc700 1 -- 172.21.15.46:0/2447781102 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v51) v1 ==== 93+0+0 (secure 0 0 0) 0x7fa3a00567d0 con 0x7fa3a40cd2c0 2024-01-26T23:47:53.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.737+0000 7fa3ac3cd700 1 -- 172.21.15.46:0/2447781102 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa3880671c0 msgr2=0x7fa388069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:53.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.737+0000 7fa3ac3cd700 1 --2- 172.21.15.46:0/2447781102 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa3880671c0 0x7fa388069670 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa3a409b390 tx=0x7fa394007ec0 comp rx=0 tx=0).stop 2024-01-26T23:47:53.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.737+0000 7fa3ac3cd700 1 -- 172.21.15.46:0/2447781102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a40cd2c0 msgr2=0x7fa3a4156b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:53.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.737+0000 7fa3ac3cd700 1 --2- 172.21.15.46:0/2447781102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a40cd2c0 0x7fa3a4156b70 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fa3a406dfe0 tx=0x7fa3a0016000 comp rx=0 tx=0).stop 2024-01-26T23:47:53.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.737+0000 7fa3ac3cd700 1 -- 172.21.15.46:0/2447781102 shutdown_connections 2024-01-26T23:47:53.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.737+0000 7fa3ac3cd700 1 --2- 172.21.15.46:0/2447781102 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fa3880671c0 0x7fa388069670 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:53.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.737+0000 7fa3ac3cd700 1 --2- 172.21.15.46:0/2447781102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a40cd2c0 0x7fa3a4156b70 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:53.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.737+0000 7fa3ac3cd700 1 --2- 172.21.15.46:0/2447781102 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3a4159e70 0x7fa3a406c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:53.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.737+0000 7fa3ac3cd700 1 --2- 172.21.15.46:0/2447781102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3a4159aa0 0x7fa3a406c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:53.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.737+0000 7fa3ac3cd700 1 -- 172.21.15.46:0/2447781102 >> 172.21.15.46:0/2447781102 conn(0x7fa3a4069220 msgr2=0x7fa3a40aa830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:53.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.737+0000 7fa3ac3cd700 1 -- 172.21.15.46:0/2447781102 shutdown_connections 2024-01-26T23:47:53.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:53.737+0000 7fa3ac3cd700 1 -- 172.21.15.46:0/2447781102 wait complete. 2024-01-26T23:47:53.755 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-26T23:47:53.755 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:47:53.802 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:47:53.839 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-26T23:47:53.839 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-26T23:47:53.839 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-26T23:47:53.839 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-26T23:47:53.885 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[79483]: fuse finished with error 0 and tester_r 0 2024-01-26T23:47:53.887 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:47:53.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:47:53.888 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-26T23:47:54.017 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:47:54.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:47:54.018 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-26T23:47:54.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:47:54.034 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-26T23:47:54.034 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:47:54.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:47:54.034 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-26T23:47:54.076 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:54.075+0000 7fca3d6501c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:47:54.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:47:54.077 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-26T23:47:54.077 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:47:54.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:47:54.077 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-26T23:47:54.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:47:54.094 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-26T23:47:54.094 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:47:54.128 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:54.127+0000 7f3fc81491c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:47:54.155 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:54.154+0000 7fca3d6501c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:47:54.220 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:54.219+0000 7f3fc81491c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:47:54.225 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:54.225+0000 7fca3d6501c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:47:54.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.250+0000 7ff72d15d700 1 Processor -- start 2024-01-26T23:47:54.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.250+0000 7ff72d15d700 1 -- start start 2024-01-26T23:47:54.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.251+0000 7ff72d15d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff728157340 0x7ff72815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:54.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.251+0000 7ff72d15d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff7280b4180 0x7ff728157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:54.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.251+0000 7ff72d15d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff728158ae0 con 0x7ff7280ae520 2024-01-26T23:47:54.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.251+0000 7ff72d15d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff72815b650 con 0x7ff728157340 2024-01-26T23:47:54.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.251+0000 7ff72d15d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff72815c020 con 0x7ff7280b4180 2024-01-26T23:47:54.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.251+0000 7ff726d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff728157340 0x7ff72815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:54.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.251+0000 7ff726d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff728157340 0x7ff72815bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41166/0 (socket says 172.21.15.46:41166) 2024-01-26T23:47:54.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.251+0000 7ff726d9d700 1 -- 172.21.15.46:0/2797370005 learned_addr learned my addr 172.21.15.46:0/2797370005 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:54.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.251+0000 7ff72759e700 1 --2- 172.21.15.46:0/2797370005 >> v2:172.21.15.106:3300/0 conn(0x7ff7280b4180 0x7ff728157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:54.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.251+0000 7ff726d9d700 1 -- 172.21.15.46:0/2797370005 >> v2:172.21.15.106:3300/0 conn(0x7ff7280b4180 msgr2=0x7ff728157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:47:54.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.251+0000 7ff726d9d700 1 --2- 172.21.15.46:0/2797370005 >> v2:172.21.15.106:3300/0 conn(0x7ff7280b4180 0x7ff728157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.251+0000 7ff726d9d700 1 -- 172.21.15.46:0/2797370005 >> v1:172.21.15.106:6789/0 conn(0x7ff7280ae520 legacy=0x7ff7280b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:54.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.251+0000 7ff726d9d700 1 -- 172.21.15.46:0/2797370005 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff728158520 con 0x7ff728157340 2024-01-26T23:47:54.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.252+0000 7ff726d9d700 1 --2- 172.21.15.46:0/2797370005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff728157340 0x7ff72815bae0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7ff7280caeb0 tx=0x7ff718009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27026a96033e3cac server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:54.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.252+0000 7ff725d9b700 1 -- 172.21.15.46:0/2797370005 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff718009e70 con 0x7ff728157340 2024-01-26T23:47:54.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.252+0000 7ff725d9b700 1 -- 172.21.15.46:0/2797370005 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71800b860 con 0x7ff728157340 2024-01-26T23:47:54.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.252+0000 7ff725d9b700 1 -- 172.21.15.46:0/2797370005 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff71800bc20 con 0x7ff728157340 2024-01-26T23:47:54.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.252+0000 7ff72d15d700 1 -- 172.21.15.46:0/2797370005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff728157340 msgr2=0x7ff72815bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:54.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.252+0000 7ff72d15d700 1 --2- 172.21.15.46:0/2797370005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff728157340 0x7ff72815bae0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7ff7280caeb0 tx=0x7ff718009250 comp rx=0 tx=0).stop 2024-01-26T23:47:54.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.252+0000 7ff72d15d700 1 -- 172.21.15.46:0/2797370005 shutdown_connections 2024-01-26T23:47:54.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.252+0000 7ff72d15d700 1 --2- 172.21.15.46:0/2797370005 >> v2:172.21.15.106:3300/0 conn(0x7ff7280b4180 0x7ff728157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.252+0000 7ff72d15d700 1 --2- 172.21.15.46:0/2797370005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff728157340 0x7ff72815bae0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.252+0000 7ff72d15d700 1 -- 172.21.15.46:0/2797370005 >> 172.21.15.46:0/2797370005 conn(0x7ff7280ce530 msgr2=0x7ff7280ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:54.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.252+0000 7ff72d15d700 1 -- 172.21.15.46:0/2797370005 shutdown_connections 2024-01-26T23:47:54.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.252+0000 7ff72d15d700 1 -- 172.21.15.46:0/2797370005 wait complete. 2024-01-26T23:47:54.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.254+0000 7ff72d15d700 1 Processor -- start 2024-01-26T23:47:54.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.254+0000 7ff72d15d700 1 -- start start 2024-01-26T23:47:54.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.255+0000 7ff72d15d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7280ae520 0x7ff728144930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:54.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.255+0000 7ff72d15d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff7280b4180 0x7ff7280c4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:54.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.255+0000 7ff72d15d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff728157340 0x7ff7280c4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:54.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.255+0000 7ff72d15d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7280c4f70 con 0x7ff728157340 2024-01-26T23:47:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.255+0000 7ff72d15d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff728153bf0 con 0x7ff7280ae520 2024-01-26T23:47:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.255+0000 7ff72d15d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff728144e70 con 0x7ff7280b4180 2024-01-26T23:47:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.255+0000 7ff726d9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7280ae520 0x7ff728144930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.256+0000 7ff72759e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff728157340 0x7ff7280c4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.256+0000 7ff72659c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff7280b4180 0x7ff7280c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.256+0000 7ff726d9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7280ae520 0x7ff728144930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:55390/0 (socket says 172.21.15.46:55390) 2024-01-26T23:47:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.256+0000 7ff72759e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff728157340 0x7ff7280c4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41180/0 (socket says 172.21.15.46:41180) 2024-01-26T23:47:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.256+0000 7ff726d9d700 1 -- 172.21.15.46:0/1271326294 learned_addr learned my addr 172.21.15.46:0/1271326294 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.256+0000 7ff72659c700 1 -- 172.21.15.46:0/1271326294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7280ae520 msgr2=0x7ff728144930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.256+0000 7ff72659c700 1 --2- 172.21.15.46:0/1271326294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7280ae520 0x7ff728144930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.256+0000 7ff72659c700 1 -- 172.21.15.46:0/1271326294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff728157340 msgr2=0x7ff7280c4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.256+0000 7ff72659c700 1 --2- 172.21.15.46:0/1271326294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff728157340 0x7ff7280c4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.256+0000 7ff72659c700 1 -- 172.21.15.46:0/1271326294 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7180094f0 con 0x7ff7280b4180 2024-01-26T23:47:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.256+0000 7ff726d9d700 1 --2- 172.21.15.46:0/1271326294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7280ae520 0x7ff728144930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:47:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.256+0000 7ff72659c700 1 --2- 172.21.15.46:0/1271326294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff7280b4180 0x7ff7280c4110 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff71000c4c0 tx=0x7ff710007ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:54.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.256+0000 7ff71effd700 1 -- 172.21.15.46:0/1271326294 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff710012420 con 0x7ff7280b4180 2024-01-26T23:47:54.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.256+0000 7ff71effd700 1 -- 172.21.15.46:0/1271326294 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7100125a0 con 0x7ff7280b4180 2024-01-26T23:47:54.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.256+0000 7ff72d15d700 1 -- 172.21.15.46:0/1271326294 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7280c4c90 con 0x7ff7280b4180 2024-01-26T23:47:54.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.256+0000 7ff71effd700 1 -- 172.21.15.46:0/1271326294 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff710012960 con 0x7ff7280b4180 2024-01-26T23:47:54.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.256+0000 7ff72d15d700 1 -- 172.21.15.46:0/1271326294 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff728153880 con 0x7ff7280b4180 2024-01-26T23:47:54.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.257+0000 7ff72d15d700 1 -- 172.21.15.46:0/1271326294 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff728153400 con 0x7ff7280b4180 2024-01-26T23:47:54.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.258+0000 7ff71effd700 1 -- 172.21.15.46:0/1271326294 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 51) v1 ==== 84958+0+0 (secure 0 0 0) 0x7ff71001b3b0 con 0x7ff7280b4180 2024-01-26T23:47:54.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.258+0000 7ff71effd700 1 --2- 172.21.15.46:0/1271326294 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff708067170 0x7ff708069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:54.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.258+0000 7ff71effd700 1 -- 172.21.15.46:0/1271326294 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6791+0+0 (secure 0 0 0) 0x7ff710084220 con 0x7ff7280b4180 2024-01-26T23:47:54.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.258+0000 7ff726d9d700 1 -- 172.21.15.46:0/1271326294 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff708067170 msgr2=0x7ff708069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:54.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.258+0000 7ff726d9d700 1 --2- 172.21.15.46:0/1271326294 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff708067170 0x7ff708069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:47:54.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.263+0000 7ff71effd700 1 -- 172.21.15.46:0/1271326294 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff728153400 con 0x7ff7280b4180 2024-01-26T23:47:54.291 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:54.290+0000 7fca3d6501c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:47:54.304 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:54.303+0000 7f3fc81491c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:47:54.379 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:54.378+0000 7f3fc81491c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:47:54.445 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:54.444+0000 7fca3d6501c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:47:54.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.459+0000 7ff726d9d700 1 -- 172.21.15.46:0/1271326294 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff708067170 msgr2=0x7ff708069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:54.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.459+0000 7ff726d9d700 1 --2- 172.21.15.46:0/1271326294 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff708067170 0x7ff708069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:47:54.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.515+0000 7ff72d15d700 1 -- 172.21.15.46:0/1271326294 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff728153400 con 0x7ff7280b4180 2024-01-26T23:47:54.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.516+0000 7ff71effd700 1 -- 172.21.15.46:0/1271326294 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4602 (secure 0 0 0) 0x7ff728153400 con 0x7ff7280b4180 2024-01-26T23:47:54.518 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:47:54.518 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6518,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/4188139264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":4188139264},{"type":"v1","addr":"172.21.15.106:6836","nonce":4188139264}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":48},{"gid":6527,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/221301328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":221301328},{"type":"v1","addr":"172.21.15.46:6833","nonce":221301328}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":6530,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1531399912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":1531399912},{"type":"v1","addr":"172.21.15.46:6835","nonce":1531399912}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:46:31.208843+0000","modified":"2024-01-26T23:46:34.775841+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7084":{"gid":7084,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6837/1643596655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1643596655},{"type":"v1","addr":"172.21.15.106:6837","nonce":1643596655}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2024-01-26T23:47:54.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.519+0000 7ff72d15d700 1 -- 172.21.15.46:0/1271326294 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff708067170 msgr2=0x7ff708069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:47:54.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.519+0000 7ff72d15d700 1 --2- 172.21.15.46:0/1271326294 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff708067170 0x7ff708069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.519+0000 7ff72d15d700 1 -- 172.21.15.46:0/1271326294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff7280b4180 msgr2=0x7ff7280c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:54.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.519+0000 7ff72d15d700 1 --2- 172.21.15.46:0/1271326294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff7280b4180 0x7ff7280c4110 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff71000c4c0 tx=0x7ff710007ec0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.519+0000 7ff72d15d700 1 -- 172.21.15.46:0/1271326294 shutdown_connections 2024-01-26T23:47:54.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.519+0000 7ff72d15d700 1 --2- 172.21.15.46:0/1271326294 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff708067170 0x7ff708069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.519+0000 7ff72d15d700 1 --2- 172.21.15.46:0/1271326294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff728157340 0x7ff7280c4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.519+0000 7ff72d15d700 1 --2- 172.21.15.46:0/1271326294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7280ae520 0x7ff728144930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.519+0000 7ff72d15d700 1 --2- 172.21.15.46:0/1271326294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff7280b4180 0x7ff7280c4110 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.519+0000 7ff72d15d700 1 -- 172.21.15.46:0/1271326294 >> 172.21.15.46:0/1271326294 conn(0x7ff7280ce530 msgr2=0x7ff7281564d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:54.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.519+0000 7ff72d15d700 1 -- 172.21.15.46:0/1271326294 shutdown_connections 2024-01-26T23:47:54.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.519+0000 7ff72d15d700 1 -- 172.21.15.46:0/1271326294 wait complete. 2024-01-26T23:47:54.522 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 54 2024-01-26T23:47:54.535 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:47:54.555 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:54.554+0000 7f3fc81491c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:47:54.581 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:54.580+0000 7fca3d6501c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:47:54.647 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:54.646+0000 7fca3d6501c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:47:54.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.686+0000 7f669840d700 1 Processor -- start 2024-01-26T23:47:54.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.686+0000 7f669840d700 1 -- start start 2024-01-26T23:47:54.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.687+0000 7f669840d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f66900b2160 0x7f669015a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:54.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.688+0000 7f669840d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f66900b2530 0x7f66900abe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:54.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.688+0000 7f669840d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66900a9b90 con 0x7f669015da10 2024-01-26T23:47:54.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.688+0000 7f669840d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66900aa210 con 0x7f66900b2530 2024-01-26T23:47:54.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.688+0000 7f669840d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f669015d580 con 0x7f66900b2160 2024-01-26T23:47:54.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.688+0000 7f66959a8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f66900b2530 0x7f66900abe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:54.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.688+0000 7f66969aa700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f669015da10 0x7f669006a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59506/0 (socket says 172.21.15.46:59506) 2024-01-26T23:47:54.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.688+0000 7f66969aa700 1 -- 172.21.15.46:0/1732065487 learned_addr learned my addr 172.21.15.46:0/1732065487 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:47:54.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.688+0000 7f66961a9700 1 --2- 172.21.15.46:0/1732065487 >> v2:172.21.15.106:3300/0 conn(0x7f66900b2160 0x7f669015a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:54.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.688+0000 7f66959a8700 1 -- 172.21.15.46:0/1732065487 >> v2:172.21.15.106:3300/0 conn(0x7f66900b2160 msgr2=0x7f669015a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:47:54.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.688+0000 7f66959a8700 1 --2- 172.21.15.46:0/1732065487 >> v2:172.21.15.106:3300/0 conn(0x7f66900b2160 0x7f669015a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.688+0000 7f66959a8700 1 -- 172.21.15.46:0/1732065487 >> v1:172.21.15.46:6789/0 conn(0x7f669015da10 legacy=0x7f669006a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:54.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.688+0000 7f66959a8700 1 -- 172.21.15.46:0/1732065487 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66900ac390 con 0x7f66900b2530 2024-01-26T23:47:54.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.688+0000 7f66959a8700 1 --2- 172.21.15.46:0/1732065487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66900b2530 0x7f66900abe50 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f6690063ca0 tx=0x7f66840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc643b85df7228e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:54.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.688+0000 7f66951a7700 1 -- 172.21.15.46:0/1732065487 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6684011630 con 0x7f66900b2530 2024-01-26T23:47:54.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.689+0000 7f66951a7700 1 -- 172.21.15.46:0/1732065487 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6684011790 con 0x7f66900b2530 2024-01-26T23:47:54.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.689+0000 7f66951a7700 1 -- 172.21.15.46:0/1732065487 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6684011b30 con 0x7f66900b2530 2024-01-26T23:47:54.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.689+0000 7f669840d700 1 -- 172.21.15.46:0/1732065487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66900b2530 msgr2=0x7f66900abe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:54.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.689+0000 7f669840d700 1 --2- 172.21.15.46:0/1732065487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66900b2530 0x7f66900abe50 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f6690063ca0 tx=0x7f66840091a0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.689+0000 7f669840d700 1 -- 172.21.15.46:0/1732065487 shutdown_connections 2024-01-26T23:47:54.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.689+0000 7f669840d700 1 --2- 172.21.15.46:0/1732065487 >> v2:172.21.15.106:3300/0 conn(0x7f66900b2160 0x7f669015a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.689+0000 7f669840d700 1 --2- 172.21.15.46:0/1732065487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66900b2530 0x7f66900abe50 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.689+0000 7f669840d700 1 -- 172.21.15.46:0/1732065487 >> 172.21.15.46:0/1732065487 conn(0x7f66900ccbc0 msgr2=0x7f66900ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:54.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.689+0000 7f669840d700 1 -- 172.21.15.46:0/1732065487 shutdown_connections 2024-01-26T23:47:54.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.689+0000 7f669840d700 1 -- 172.21.15.46:0/1732065487 wait complete. 2024-01-26T23:47:54.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.691+0000 7f669840d700 1 Processor -- start 2024-01-26T23:47:54.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.691+0000 7f669840d700 1 -- start start 2024-01-26T23:47:54.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.692+0000 7f669840d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66900b2160 0x7f6690148020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:54.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.692+0000 7f669840d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66900b2530 0x7f6690149220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:54.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.692+0000 7f669840d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f669015da10 0x7f6690149760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:54.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.692+0000 7f669840d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66900ae210 con 0x7f66900b2160 2024-01-26T23:47:54.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.692+0000 7f669840d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669015c3e0 con 0x7f66900b2530 2024-01-26T23:47:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.692+0000 7f669840d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669015c6c0 con 0x7f669015da10 2024-01-26T23:47:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.692+0000 7f66961a9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66900b2160 0x7f6690148020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.692+0000 7f66961a9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66900b2160 0x7f6690148020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41194/0 (socket says 172.21.15.46:41194) 2024-01-26T23:47:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.692+0000 7f66961a9700 1 -- 172.21.15.46:0/3890121348 learned_addr learned my addr 172.21.15.46:0/3890121348 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.692+0000 7f66959a8700 1 --2- 172.21.15.46:0/3890121348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66900b2530 0x7f6690149220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.692+0000 7f66969aa700 1 --2- 172.21.15.46:0/3890121348 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f669015da10 0x7f6690149760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.693+0000 7f66961a9700 1 -- 172.21.15.46:0/3890121348 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f669015da10 msgr2=0x7f6690149760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.693+0000 7f66961a9700 1 --2- 172.21.15.46:0/3890121348 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f669015da10 0x7f6690149760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.693+0000 7f66961a9700 1 -- 172.21.15.46:0/3890121348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66900b2530 msgr2=0x7f6690149220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.693+0000 7f66961a9700 1 --2- 172.21.15.46:0/3890121348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66900b2530 0x7f6690149220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.693+0000 7f66961a9700 1 -- 172.21.15.46:0/3890121348 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6684009420 con 0x7f66900b2160 2024-01-26T23:47:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.693+0000 7f66961a9700 1 --2- 172.21.15.46:0/3890121348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66900b2160 0x7f6690148020 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f6680009ef0 tx=0x7f668000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.693+0000 7f668e7fc700 1 -- 172.21.15.46:0/3890121348 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668000cb70 con 0x7f66900b2160 2024-01-26T23:47:54.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.693+0000 7f668e7fc700 1 -- 172.21.15.46:0/3890121348 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668000f040 con 0x7f66900b2160 2024-01-26T23:47:54.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.693+0000 7f669840d700 1 -- 172.21.15.46:0/3890121348 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66900adea0 con 0x7f66900b2160 2024-01-26T23:47:54.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.693+0000 7f668e7fc700 1 -- 172.21.15.46:0/3890121348 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6680014480 con 0x7f66900b2160 2024-01-26T23:47:54.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.693+0000 7f669840d700 1 -- 172.21.15.46:0/3890121348 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6690148a80 con 0x7f66900b2160 2024-01-26T23:47:54.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.694+0000 7f669840d700 1 -- 172.21.15.46:0/3890121348 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f669004c710 con 0x7f66900b2160 2024-01-26T23:47:54.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.695+0000 7f668e7fc700 1 -- 172.21.15.46:0/3890121348 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 51) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f66800147f0 con 0x7f66900b2160 2024-01-26T23:47:54.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.696+0000 7f668e7fc700 1 --2- 172.21.15.46:0/3890121348 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f667c067170 0x7f667c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:54.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.696+0000 7f668e7fc700 1 -- 172.21.15.46:0/3890121348 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6791+0+0 (secure 0 0 0) 0x7f6680085e60 con 0x7f66900b2160 2024-01-26T23:47:54.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.696+0000 7f66959a8700 1 -- 172.21.15.46:0/3890121348 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f667c067170 msgr2=0x7f667c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:54.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.696+0000 7f66959a8700 1 --2- 172.21.15.46:0/3890121348 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f667c067170 0x7f667c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:47:54.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.700+0000 7f668e7fc700 1 -- 172.21.15.46:0/3890121348 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6680052970 con 0x7f66900b2160 2024-01-26T23:47:54.715 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:54.714+0000 7f3fc81491c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:47:54.769 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:54.768+0000 7fca3d6501c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:47:54.793 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:54.792+0000 7f3fc81491c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:47:54.842 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:54.841+0000 7fca3d6501c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:47:54.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.897+0000 7f66959a8700 1 -- 172.21.15.46:0/3890121348 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f667c067170 msgr2=0x7f667c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:54.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.897+0000 7f66959a8700 1 --2- 172.21.15.46:0/3890121348 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f667c067170 0x7f667c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:47:54.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.908+0000 7f669840d700 1 -- 172.21.15.46:0/3890121348 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66900aa460 con 0x7f66900b2160 2024-01-26T23:47:54.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.909+0000 7f668e7fc700 1 -- 172.21.15.46:0/3890121348 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+16729 (secure 0 0 0) 0x7f6680055f90 con 0x7f66900b2160 2024-01-26T23:47:54.911 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:47:54.911 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":117,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:47:51.713551+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:46:29.702709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-01-26T23:46:30.178825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.2c","mappings":[{"from":1,"to":2}]},{"pgid":"14.33","mappings":[{"from":1,"to":2}]},{"pgid":"14.3a","mappings":[{"from":1,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/4144593394":"2024-01-27T23:46:50.618051+0000","172.21.15.106:0/2301684208":"2024-01-27T23:46:50.618051+0000","172.21.15.106:0/1989924028":"2024-01-27T23:46:50.618051+0000","172.21.15.106:6833/43388":"2024-01-27T23:46:50.618051+0000","172.21.15.106:6832/43388":"2024-01-27T23:46:50.618051+0000","172.21.15.106:0/1323704418":"2024-01-27T00:46:44.670099+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:47:54.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.912+0000 7f669840d700 1 -- 172.21.15.46:0/3890121348 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f667c067170 msgr2=0x7f667c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:47:54.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.912+0000 7f669840d700 1 --2- 172.21.15.46:0/3890121348 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f667c067170 0x7f667c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.912+0000 7f669840d700 1 -- 172.21.15.46:0/3890121348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66900b2160 msgr2=0x7f6690148020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:54.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.912+0000 7f669840d700 1 --2- 172.21.15.46:0/3890121348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66900b2160 0x7f6690148020 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f6680009ef0 tx=0x7f668000b040 comp rx=0 tx=0).stop 2024-01-26T23:47:54.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.912+0000 7f669840d700 1 -- 172.21.15.46:0/3890121348 shutdown_connections 2024-01-26T23:47:54.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.912+0000 7f669840d700 1 --2- 172.21.15.46:0/3890121348 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f667c067170 0x7f667c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.912+0000 7f669840d700 1 --2- 172.21.15.46:0/3890121348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66900b2160 0x7f6690148020 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.912+0000 7f669840d700 1 --2- 172.21.15.46:0/3890121348 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f669015da10 0x7f6690149760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.912+0000 7f669840d700 1 --2- 172.21.15.46:0/3890121348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66900b2530 0x7f6690149220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:54.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.912+0000 7f669840d700 1 -- 172.21.15.46:0/3890121348 >> 172.21.15.46:0/3890121348 conn(0x7f66900ccbc0 msgr2=0x7f66900b0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:54.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.912+0000 7f669840d700 1 -- 172.21.15.46:0/3890121348 shutdown_connections 2024-01-26T23:47:54.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:54.912+0000 7f669840d700 1 -- 172.21.15.46:0/3890121348 wait complete. 2024-01-26T23:47:54.932 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-26T23:46:29.702709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-01-26T23:46:30.178825+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-26T23:47:54.933 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:47:54.940 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:54.939+0000 7f3fc81491c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:47:54.980 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:54.980+0000 7fca3d6501c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:47:55.025 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:55.024+0000 7f3fc81491c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:47:55.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.068+0000 7fd317c96700 1 Processor -- start 2024-01-26T23:47:55.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.068+0000 7fd317c96700 1 -- start start 2024-01-26T23:47:55.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.069+0000 7fd317c96700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3100ccc50 0x7fd3100ca350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:55.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.069+0000 7fd317c96700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3100c8f70 con 0x7fd3100ca890 2024-01-26T23:47:55.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.069+0000 7fd317c96700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3100cd0c0 con 0x7fd3101599c0 2024-01-26T23:47:55.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.069+0000 7fd317c96700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3100cad10 con 0x7fd3100ccc50 2024-01-26T23:47:55.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.069+0000 7fd315231700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3100ccc50 0x7fd3100ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:55.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.069+0000 7fd316233700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd3100ca890 0x7fd310159d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59520/0 (socket says 172.21.15.46:59520) 2024-01-26T23:47:55.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.069+0000 7fd316233700 1 -- 172.21.15.46:0/62580169 learned_addr learned my addr 172.21.15.46:0/62580169 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:47:55.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.069+0000 7fd315231700 1 -- 172.21.15.46:0/62580169 >> v1:172.21.15.106:6789/0 conn(0x7fd3101599c0 legacy=0x7fd310068c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:55.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.069+0000 7fd315231700 1 -- 172.21.15.46:0/62580169 >> v1:172.21.15.46:6789/0 conn(0x7fd3100ca890 legacy=0x7fd310159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:55.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.070+0000 7fd315231700 1 -- 172.21.15.46:0/62580169 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31015a9c0 con 0x7fd3100ccc50 2024-01-26T23:47:55.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.070+0000 7fd315231700 1 --2- 172.21.15.46:0/62580169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3100ccc50 0x7fd3100ca350 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fd3000098e0 tx=0x7fd3000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea360708593b2456 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:55.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.070+0000 7fd314a30700 1 -- 172.21.15.46:0/62580169 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd300009e80 con 0x7fd3100ccc50 2024-01-26T23:47:55.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.070+0000 7fd314a30700 1 -- 172.21.15.46:0/62580169 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30000b860 con 0x7fd3100ccc50 2024-01-26T23:47:55.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.070+0000 7fd314a30700 1 -- 172.21.15.46:0/62580169 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30000bbe0 con 0x7fd3100ccc50 2024-01-26T23:47:55.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.071+0000 7fd317c96700 1 -- 172.21.15.46:0/62580169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3100ccc50 msgr2=0x7fd3100ca350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:55.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.071+0000 7fd317c96700 1 --2- 172.21.15.46:0/62580169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3100ccc50 0x7fd3100ca350 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fd3000098e0 tx=0x7fd3000091a0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.071+0000 7fd317c96700 1 -- 172.21.15.46:0/62580169 shutdown_connections 2024-01-26T23:47:55.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.071+0000 7fd317c96700 1 --2- 172.21.15.46:0/62580169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3100ccc50 0x7fd3100ca350 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.071+0000 7fd317c96700 1 -- 172.21.15.46:0/62580169 >> 172.21.15.46:0/62580169 conn(0x7fd3100ce570 msgr2=0x7fd3100ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:55.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.071+0000 7fd317c96700 1 -- 172.21.15.46:0/62580169 shutdown_connections 2024-01-26T23:47:55.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.071+0000 7fd317c96700 1 -- 172.21.15.46:0/62580169 wait complete. 2024-01-26T23:47:55.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.072+0000 7fd317c96700 1 Processor -- start 2024-01-26T23:47:55.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.072+0000 7fd317c96700 1 -- start start 2024-01-26T23:47:55.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd317c96700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3100ca890 0x7fd310154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:55.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd317c96700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3100ccc50 0x7fd3100b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:55.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd317c96700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3101599c0 0x7fd310159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:55.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd317c96700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd310159070 con 0x7fd3101599c0 2024-01-26T23:47:55.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd317c96700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31016ad00 con 0x7fd3100ca890 2024-01-26T23:47:55.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd317c96700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd310158d90 con 0x7fd3100ccc50 2024-01-26T23:47:55.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd315a32700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3100ca890 0x7fd310154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:55.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd316233700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3101599c0 0x7fd310159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:55.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd315231700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3100ccc50 0x7fd3100b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:55.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd315a32700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3100ca890 0x7fd310154300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:55410/0 (socket says 172.21.15.46:55410) 2024-01-26T23:47:55.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd315a32700 1 -- 172.21.15.46:0/2109340086 learned_addr learned my addr 172.21.15.46:0/2109340086 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:55.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd315231700 1 -- 172.21.15.46:0/2109340086 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3100ca890 msgr2=0x7fd310154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:55.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd315231700 1 --2- 172.21.15.46:0/2109340086 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3100ca890 0x7fd310154300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd315231700 1 -- 172.21.15.46:0/2109340086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3101599c0 msgr2=0x7fd310159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:55.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd315231700 1 --2- 172.21.15.46:0/2109340086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3101599c0 0x7fd310159d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd315231700 1 -- 172.21.15.46:0/2109340086 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd300009420 con 0x7fd3100ccc50 2024-01-26T23:47:55.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd316233700 1 --2- 172.21.15.46:0/2109340086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3101599c0 0x7fd310159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:47:55.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.073+0000 7fd315a32700 1 --2- 172.21.15.46:0/2109340086 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3100ca890 0x7fd310154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:47:55.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.074+0000 7fd315231700 1 --2- 172.21.15.46:0/2109340086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3100ccc50 0x7fd3100b31a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd300016000 tx=0x7fd300017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:55.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.074+0000 7fd305ffb700 1 -- 172.21.15.46:0/2109340086 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30000b9d0 con 0x7fd3100ccc50 2024-01-26T23:47:55.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.074+0000 7fd305ffb700 1 -- 172.21.15.46:0/2109340086 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30000bb30 con 0x7fd3100ccc50 2024-01-26T23:47:55.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.074+0000 7fd317c96700 1 -- 172.21.15.46:0/2109340086 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd31016a6b0 con 0x7fd3100ccc50 2024-01-26T23:47:55.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.074+0000 7fd305ffb700 1 -- 172.21.15.46:0/2109340086 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd300025630 con 0x7fd3100ccc50 2024-01-26T23:47:55.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.074+0000 7fd317c96700 1 -- 172.21.15.46:0/2109340086 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd310154840 con 0x7fd3100ccc50 2024-01-26T23:47:55.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.075+0000 7fd317c96700 1 -- 172.21.15.46:0/2109340086 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd31016a510 con 0x7fd3100ccc50 2024-01-26T23:47:55.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.076+0000 7fd305ffb700 1 -- 172.21.15.46:0/2109340086 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 51) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fd300023070 con 0x7fd3100ccc50 2024-01-26T23:47:55.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.076+0000 7fd305ffb700 1 --2- 172.21.15.46:0/2109340086 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2f4067170 0x7fd2f4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:55.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.076+0000 7fd305ffb700 1 -- 172.21.15.46:0/2109340086 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6791+0+0 (secure 0 0 0) 0x7fd300086260 con 0x7fd3100ccc50 2024-01-26T23:47:55.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.076+0000 7fd315a32700 1 -- 172.21.15.46:0/2109340086 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2f4067170 msgr2=0x7fd2f4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:55.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.076+0000 7fd315a32700 1 --2- 172.21.15.46:0/2109340086 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2f4067170 0x7fd2f4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:47:55.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.080+0000 7fd305ffb700 1 -- 172.21.15.46:0/2109340086 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd31016a510 con 0x7fd3100ccc50 2024-01-26T23:47:55.107 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:55.107+0000 7fca3d6501c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:47:55.200 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:55.199+0000 7f3fc81491c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:47:55.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.277+0000 7fd315a32700 1 -- 172.21.15.46:0/2109340086 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2f4067170 msgr2=0x7fd2f4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:55.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.277+0000 7fd315a32700 1 --2- 172.21.15.46:0/2109340086 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2f4067170 0x7fd2f4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:47:55.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.330+0000 7fd317c96700 1 -- 172.21.15.46:0/2109340086 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd31016a510 con 0x7fd3100ccc50 2024-01-26T23:47:55.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.330+0000 7fd305ffb700 1 -- 172.21.15.46:0/2109340086 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd31016a510 con 0x7fd3100ccc50 2024-01-26T23:47:55.332 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:47:55.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.333+0000 7fd317c96700 1 -- 172.21.15.46:0/2109340086 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2f4067170 msgr2=0x7fd2f4069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:47:55.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.333+0000 7fd317c96700 1 --2- 172.21.15.46:0/2109340086 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2f4067170 0x7fd2f4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.333+0000 7fd317c96700 1 -- 172.21.15.46:0/2109340086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3100ccc50 msgr2=0x7fd3100b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:55.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.333+0000 7fd317c96700 1 --2- 172.21.15.46:0/2109340086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3100ccc50 0x7fd3100b31a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd300016000 tx=0x7fd300017000 comp rx=0 tx=0).stop 2024-01-26T23:47:55.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.333+0000 7fd317c96700 1 -- 172.21.15.46:0/2109340086 shutdown_connections 2024-01-26T23:47:55.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.333+0000 7fd317c96700 1 --2- 172.21.15.46:0/2109340086 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fd2f4067170 0x7fd2f4069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.333+0000 7fd317c96700 1 --2- 172.21.15.46:0/2109340086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3101599c0 0x7fd310159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.333+0000 7fd317c96700 1 --2- 172.21.15.46:0/2109340086 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3100ca890 0x7fd310154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.333+0000 7fd317c96700 1 --2- 172.21.15.46:0/2109340086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3100ccc50 0x7fd3100b31a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.333+0000 7fd317c96700 1 -- 172.21.15.46:0/2109340086 >> 172.21.15.46:0/2109340086 conn(0x7fd3100ce570 msgr2=0x7fd3100c4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:55.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.333+0000 7fd317c96700 1 -- 172.21.15.46:0/2109340086 shutdown_connections 2024-01-26T23:47:55.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.333+0000 7fd317c96700 1 -- 172.21.15.46:0/2109340086 wait complete. 2024-01-26T23:47:55.342 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:55.341+0000 7f3fc81491c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:47:55.348 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-26T23:47:55.422 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:55.421+0000 7fca3d6501c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:47:55.482 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:47:55.485 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:47:55.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.528+0000 7f9b40abf700 1 Processor -- start 2024-01-26T23:47:55.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.528+0000 7f9b40abf700 1 -- start start 2024-01-26T23:47:55.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.529+0000 7f9b40abf700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b3c068ed0 0x7f9b3c06a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:55.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.529+0000 7f9b40abf700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c0a9a80 con 0x7f9b3c06add0 2024-01-26T23:47:55.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.529+0000 7f9b40abf700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c15a680 con 0x7f9b3c068b00 2024-01-26T23:47:55.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.529+0000 7f9b40abf700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c069320 con 0x7f9b3c068ed0 2024-01-26T23:47:55.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.529+0000 7f9b3ad9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9b3c06add0 0x7f9b3c0a9f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59526/0 (socket says 172.21.15.46:59526) 2024-01-26T23:47:55.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.529+0000 7f9b3ad9d700 1 -- 172.21.15.46:0/2153027271 learned_addr learned my addr 172.21.15.46:0/2153027271 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:47:55.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.529+0000 7f9b39d9b700 1 --2- 172.21.15.46:0/2153027271 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b3c068ed0 0x7f9b3c06a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:55.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.530+0000 7f9b3959a700 1 -- 172.21.15.46:0/2153027271 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462973267 0 0) 0x7f9b3c15a680 con 0x7f9b3c068b00 2024-01-26T23:47:55.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.530+0000 7f9b3959a700 1 -- 172.21.15.46:0/2153027271 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b28003570 con 0x7f9b3c068b00 2024-01-26T23:47:55.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.530+0000 7f9b3959a700 1 -- 172.21.15.46:0/2153027271 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333167124 0 0) 0x7f9b3c0a9a80 con 0x7f9b3c06add0 2024-01-26T23:47:55.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.530+0000 7f9b3959a700 1 -- 172.21.15.46:0/2153027271 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b3c15a680 con 0x7f9b3c06add0 2024-01-26T23:47:55.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.530+0000 7f9b39d9b700 1 -- 172.21.15.46:0/2153027271 >> v1:172.21.15.106:6789/0 conn(0x7f9b3c068b00 legacy=0x7f9b3c159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:55.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.530+0000 7f9b39d9b700 1 -- 172.21.15.46:0/2153027271 >> v1:172.21.15.46:6789/0 conn(0x7f9b3c06add0 legacy=0x7f9b3c0a9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:55.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.530+0000 7f9b39d9b700 1 -- 172.21.15.46:0/2153027271 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3c15a960 con 0x7f9b3c068ed0 2024-01-26T23:47:55.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.530+0000 7f9b39d9b700 1 --2- 172.21.15.46:0/2153027271 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b3c068ed0 0x7f9b3c06a660 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009910 tx=0x7f9b2c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c448dc7a05e6067 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:55.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.530+0000 7f9b3959a700 1 -- 172.21.15.46:0/2153027271 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c011630 con 0x7f9b3c068ed0 2024-01-26T23:47:55.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.530+0000 7f9b3959a700 1 -- 172.21.15.46:0/2153027271 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2c011790 con 0x7f9b3c068ed0 2024-01-26T23:47:55.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.530+0000 7f9b3959a700 1 -- 172.21.15.46:0/2153027271 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c011b30 con 0x7f9b3c068ed0 2024-01-26T23:47:55.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.531+0000 7f9b40abf700 1 -- 172.21.15.46:0/2153027271 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b3c068ed0 msgr2=0x7f9b3c06a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:55.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.531+0000 7f9b40abf700 1 --2- 172.21.15.46:0/2153027271 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b3c068ed0 0x7f9b3c06a660 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009910 tx=0x7f9b2c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.531+0000 7f9b40abf700 1 -- 172.21.15.46:0/2153027271 shutdown_connections 2024-01-26T23:47:55.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.531+0000 7f9b40abf700 1 --2- 172.21.15.46:0/2153027271 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b3c068ed0 0x7f9b3c06a660 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.531+0000 7f9b40abf700 1 -- 172.21.15.46:0/2153027271 >> 172.21.15.46:0/2153027271 conn(0x7f9b3c0ca100 msgr2=0x7f9b3c06e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:55.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.531+0000 7f9b40abf700 1 -- 172.21.15.46:0/2153027271 shutdown_connections 2024-01-26T23:47:55.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.531+0000 7f9b40abf700 1 -- 172.21.15.46:0/2153027271 wait complete. 2024-01-26T23:47:55.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.532+0000 7f9b40abf700 1 Processor -- start 2024-01-26T23:47:55.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.532+0000 7f9b40abf700 1 -- start start 2024-01-26T23:47:55.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.533+0000 7f9b40abf700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b3c068b00 0x7f9b3c0c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:55.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.533+0000 7f9b40abf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b3c068ed0 0x7f9b3c0ab940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:55.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.533+0000 7f9b40abf700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b3c06add0 0x7f9b3c0b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:55.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.533+0000 7f9b40abf700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c0c8ac0 con 0x7f9b3c068ed0 2024-01-26T23:47:55.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.533+0000 7f9b40abf700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c0ac870 con 0x7f9b3c068b00 2024-01-26T23:47:55.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.533+0000 7f9b40abf700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c0ac2b0 con 0x7f9b3c06add0 2024-01-26T23:47:55.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.533+0000 7f9b39d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b3c068ed0 0x7f9b3c0ab940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:55.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.533+0000 7f9b39d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b3c068ed0 0x7f9b3c0ab940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41220/0 (socket says 172.21.15.46:41220) 2024-01-26T23:47:55.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.533+0000 7f9b39d9b700 1 -- 172.21.15.46:0/2248691068 learned_addr learned my addr 172.21.15.46:0/2248691068 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:55.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.533+0000 7f9b3a59c700 1 --2- 172.21.15.46:0/2248691068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b3c068b00 0x7f9b3c0c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:55.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.533+0000 7f9b3ad9d700 1 --2- 172.21.15.46:0/2248691068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b3c06add0 0x7f9b3c0b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:55.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.534+0000 7f9b39d9b700 1 -- 172.21.15.46:0/2248691068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b3c06add0 msgr2=0x7f9b3c0b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:55.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.534+0000 7f9b39d9b700 1 --2- 172.21.15.46:0/2248691068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b3c06add0 0x7f9b3c0b1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.534+0000 7f9b39d9b700 1 -- 172.21.15.46:0/2248691068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b3c068b00 msgr2=0x7f9b3c0c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:55.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.534+0000 7f9b39d9b700 1 --2- 172.21.15.46:0/2248691068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b3c068b00 0x7f9b3c0c8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.534+0000 7f9b39d9b700 1 -- 172.21.15.46:0/2248691068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2c009420 con 0x7f9b3c068ed0 2024-01-26T23:47:55.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.534+0000 7f9b39d9b700 1 --2- 172.21.15.46:0/2248691068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b3c068ed0 0x7f9b3c0ab940 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009f00 tx=0x7f9b2c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:55.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.534+0000 7f9b327fc700 1 -- 172.21.15.46:0/2248691068 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c01f070 con 0x7f9b3c068ed0 2024-01-26T23:47:55.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.534+0000 7f9b327fc700 1 -- 172.21.15.46:0/2248691068 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2c002f70 con 0x7f9b3c068ed0 2024-01-26T23:47:55.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.534+0000 7f9b40abf700 1 -- 172.21.15.46:0/2248691068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3c0ac590 con 0x7f9b3c068ed0 2024-01-26T23:47:55.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.534+0000 7f9b327fc700 1 -- 172.21.15.46:0/2248691068 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c011900 con 0x7f9b3c068ed0 2024-01-26T23:47:55.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.534+0000 7f9b40abf700 1 -- 172.21.15.46:0/2248691068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b3c0ad9b0 con 0x7f9b3c068ed0 2024-01-26T23:47:55.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.535+0000 7f9b40abf700 1 -- 172.21.15.46:0/2248691068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b3c1578a0 con 0x7f9b3c068ed0 2024-01-26T23:47:55.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.536+0000 7f9b327fc700 1 -- 172.21.15.46:0/2248691068 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 51) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f9b2c023070 con 0x7f9b3c068ed0 2024-01-26T23:47:55.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.536+0000 7f9b327fc700 1 --2- 172.21.15.46:0/2248691068 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9b1c067170 0x7f9b1c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:55.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.536+0000 7f9b3a59c700 1 -- 172.21.15.46:0/2248691068 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9b1c067170 msgr2=0x7f9b1c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:55.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.536+0000 7f9b327fc700 1 -- 172.21.15.46:0/2248691068 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6791+0+0 (secure 0 0 0) 0x7f9b2c0860c0 con 0x7f9b3c068ed0 2024-01-26T23:47:55.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.536+0000 7f9b3a59c700 1 --2- 172.21.15.46:0/2248691068 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9b1c067170 0x7f9b1c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:47:55.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.541+0000 7f9b327fc700 1 -- 172.21.15.46:0/2248691068 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9b2c052b90 con 0x7f9b3c068ed0 2024-01-26T23:47:55.641 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:55.640+0000 7fca3d6501c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:47:55.719 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:55.718+0000 7f3fc81491c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:47:55.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.737+0000 7f9b3a59c700 1 -- 172.21.15.46:0/2248691068 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9b1c067170 msgr2=0x7f9b1c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:55.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.737+0000 7f9b3a59c700 1 --2- 172.21.15.46:0/2248691068 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9b1c067170 0x7f9b1c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:47:55.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.748+0000 7f9b40abf700 1 -- 172.21.15.46:0/2248691068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9b3c0adba0 con 0x7f9b3c068ed0 2024-01-26T23:47:55.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.748+0000 7f9b327fc700 1 -- 172.21.15.46:0/2248691068 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v117) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9b2c0150a0 con 0x7f9b3c068ed0 2024-01-26T23:47:55.750 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:47:55.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.750+0000 7f9b40abf700 1 -- 172.21.15.46:0/2248691068 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9b1c067170 msgr2=0x7f9b1c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:47:55.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.750+0000 7f9b40abf700 1 --2- 172.21.15.46:0/2248691068 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9b1c067170 0x7f9b1c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.750+0000 7f9b40abf700 1 -- 172.21.15.46:0/2248691068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b3c068ed0 msgr2=0x7f9b3c0ab940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:55.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.750+0000 7f9b40abf700 1 --2- 172.21.15.46:0/2248691068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b3c068ed0 0x7f9b3c0ab940 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009f00 tx=0x7f9b2c017000 comp rx=0 tx=0).stop 2024-01-26T23:47:55.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.751+0000 7f9b40abf700 1 -- 172.21.15.46:0/2248691068 shutdown_connections 2024-01-26T23:47:55.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.751+0000 7f9b40abf700 1 --2- 172.21.15.46:0/2248691068 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f9b1c067170 0x7f9b1c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.751+0000 7f9b40abf700 1 --2- 172.21.15.46:0/2248691068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b3c06add0 0x7f9b3c0b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.751+0000 7f9b40abf700 1 --2- 172.21.15.46:0/2248691068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b3c068b00 0x7f9b3c0c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.751+0000 7f9b40abf700 1 --2- 172.21.15.46:0/2248691068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b3c068ed0 0x7f9b3c0ab940 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.751+0000 7f9b40abf700 1 -- 172.21.15.46:0/2248691068 >> 172.21.15.46:0/2248691068 conn(0x7f9b3c0ca100 msgr2=0x7f9b3c157040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:55.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.751+0000 7f9b40abf700 1 -- 172.21.15.46:0/2248691068 shutdown_connections 2024-01-26T23:47:55.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.751+0000 7f9b40abf700 1 -- 172.21.15.46:0/2248691068 wait complete. 2024-01-26T23:47:55.766 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-26T23:47:55.866 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:55.866+0000 7fca3d6501c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:47:55.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.901+0000 7f8edfb94700 1 Processor -- start 2024-01-26T23:47:55.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.901+0000 7f8edfb94700 1 -- start start 2024-01-26T23:47:55.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.902+0000 7f8edfb94700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ed80ab310 0x7f8ed80a9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:55.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.902+0000 7f8edfb94700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed806aad0 con 0x7f8ed80ab6e0 2024-01-26T23:47:55.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.902+0000 7f8edfb94700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed80c7540 con 0x7f8ed8068940 2024-01-26T23:47:55.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.902+0000 7f8edfb94700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed80c7820 con 0x7f8ed80ab310 2024-01-26T23:47:55.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.902+0000 7f8edd12f700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8ed80ab6e0 0x7f8ed806a3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59534/0 (socket says 172.21.15.46:59534) 2024-01-26T23:47:55.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.902+0000 7f8edd12f700 1 -- 172.21.15.46:0/1415060086 learned_addr learned my addr 172.21.15.46:0/1415060086 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:47:55.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.902+0000 7f8edd930700 1 --2- 172.21.15.46:0/1415060086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ed80ab310 0x7f8ed80a9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:55.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.902+0000 7f8edd930700 1 -- 172.21.15.46:0/1415060086 >> v1:172.21.15.106:6789/0 conn(0x7f8ed8068940 legacy=0x7f8ed80c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:55.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.902+0000 7f8edc92e700 1 -- 172.21.15.46:0/1415060086 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740296003 0 0) 0x7f8ed806aad0 con 0x7f8ed80ab6e0 2024-01-26T23:47:55.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.903+0000 7f8edd930700 1 -- 172.21.15.46:0/1415060086 >> v1:172.21.15.46:6789/0 conn(0x7f8ed80ab6e0 legacy=0x7f8ed806a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:55.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.903+0000 7f8edd930700 1 -- 172.21.15.46:0/1415060086 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed80c7b00 con 0x7f8ed80ab310 2024-01-26T23:47:55.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.903+0000 7f8edd930700 1 --2- 172.21.15.46:0/1415060086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ed80ab310 0x7f8ed80a9850 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4009880 tx=0x7f8ed4009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2f2c3efd472e610 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:55.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.903+0000 7f8edc92e700 1 -- 172.21.15.46:0/1415060086 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed4011630 con 0x7f8ed80ab310 2024-01-26T23:47:55.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.903+0000 7f8edc92e700 1 -- 172.21.15.46:0/1415060086 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed4011790 con 0x7f8ed80ab310 2024-01-26T23:47:55.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.903+0000 7f8edc92e700 1 -- 172.21.15.46:0/1415060086 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed4011b50 con 0x7f8ed80ab310 2024-01-26T23:47:55.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.904+0000 7f8edfb94700 1 -- 172.21.15.46:0/1415060086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ed80ab310 msgr2=0x7f8ed80a9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:55.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.904+0000 7f8edfb94700 1 --2- 172.21.15.46:0/1415060086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ed80ab310 0x7f8ed80a9850 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4009880 tx=0x7f8ed4009120 comp rx=0 tx=0).stop 2024-01-26T23:47:55.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.904+0000 7f8edfb94700 1 -- 172.21.15.46:0/1415060086 shutdown_connections 2024-01-26T23:47:55.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.904+0000 7f8edfb94700 1 --2- 172.21.15.46:0/1415060086 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ed80ab310 0x7f8ed80a9850 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.904+0000 7f8edfb94700 1 -- 172.21.15.46:0/1415060086 >> 172.21.15.46:0/1415060086 conn(0x7f8ed80ca100 msgr2=0x7f8ed806e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:55.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.904+0000 7f8edfb94700 1 -- 172.21.15.46:0/1415060086 shutdown_connections 2024-01-26T23:47:55.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.904+0000 7f8edfb94700 1 -- 172.21.15.46:0/1415060086 wait complete. 2024-01-26T23:47:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.905+0000 7f8edfb94700 1 Processor -- start 2024-01-26T23:47:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.905+0000 7f8edfb94700 1 -- start start 2024-01-26T23:47:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.906+0000 7f8edfb94700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8068940 0x7f8ed80c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.906+0000 7f8edfb94700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed80ab310 0x7f8ed8169da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.906+0000 7f8edfb94700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ed80ab6e0 0x7f8ed816a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.906+0000 7f8edfb94700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed80b01f0 con 0x7f8ed8068940 2024-01-26T23:47:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.906+0000 7f8edfb94700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed80c5e00 con 0x7f8ed80ab310 2024-01-26T23:47:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.906+0000 7f8edfb94700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed80c60e0 con 0x7f8ed80ab6e0 2024-01-26T23:47:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.906+0000 7f8edd930700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8068940 0x7f8ed80c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.906+0000 7f8edd930700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8068940 0x7f8ed80c5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41222/0 (socket says 172.21.15.46:41222) 2024-01-26T23:47:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.906+0000 7f8edd930700 1 -- 172.21.15.46:0/4022841594 learned_addr learned my addr 172.21.15.46:0/4022841594 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.906+0000 7f8edd12f700 1 --2- 172.21.15.46:0/4022841594 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed80ab310 0x7f8ed8169da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.906+0000 7f8ede131700 1 --2- 172.21.15.46:0/4022841594 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ed80ab6e0 0x7f8ed816a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.906+0000 7f8edd930700 1 -- 172.21.15.46:0/4022841594 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ed80ab6e0 msgr2=0x7f8ed816a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.906+0000 7f8edd930700 1 --2- 172.21.15.46:0/4022841594 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ed80ab6e0 0x7f8ed816a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.906+0000 7f8edd930700 1 -- 172.21.15.46:0/4022841594 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed80ab310 msgr2=0x7f8ed8169da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:55.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.906+0000 7f8edd930700 1 --2- 172.21.15.46:0/4022841594 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed80ab310 0x7f8ed8169da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:55.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.906+0000 7f8edd930700 1 -- 172.21.15.46:0/4022841594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed40093c0 con 0x7f8ed8068940 2024-01-26T23:47:55.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.906+0000 7f8edd930700 1 --2- 172.21.15.46:0/4022841594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8068940 0x7f8ed80c5580 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8069dc0 tx=0x7f8ed4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:55.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.907+0000 7f8ecdffb700 1 -- 172.21.15.46:0/4022841594 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed4000d10 con 0x7f8ed8068940 2024-01-26T23:47:55.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.907+0000 7f8ecdffb700 1 -- 172.21.15.46:0/4022841594 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed401ce70 con 0x7f8ed8068940 2024-01-26T23:47:55.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.907+0000 7f8ecdffb700 1 -- 172.21.15.46:0/4022841594 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed401a660 con 0x7f8ed8068940 2024-01-26T23:47:55.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.907+0000 7f8edfb94700 1 -- 172.21.15.46:0/4022841594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed80c63c0 con 0x7f8ed8068940 2024-01-26T23:47:55.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.907+0000 7f8edfb94700 1 -- 172.21.15.46:0/4022841594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed8158560 con 0x7f8ed8068940 2024-01-26T23:47:55.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.908+0000 7f8edfb94700 1 -- 172.21.15.46:0/4022841594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed804c710 con 0x7f8ed8068940 2024-01-26T23:47:55.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.908+0000 7f8ecdffb700 1 -- 172.21.15.46:0/4022841594 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 51) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f8ed401aa50 con 0x7f8ed8068940 2024-01-26T23:47:55.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.909+0000 7f8ecdffb700 1 --2- 172.21.15.46:0/4022841594 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8ebc067170 0x7f8ebc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:55.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.909+0000 7f8ecdffb700 1 -- 172.21.15.46:0/4022841594 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6791+0+0 (secure 0 0 0) 0x7f8ed4086960 con 0x7f8ed8068940 2024-01-26T23:47:55.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.909+0000 7f8edd12f700 1 -- 172.21.15.46:0/4022841594 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8ebc067170 msgr2=0x7f8ebc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:55.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.909+0000 7f8edd12f700 1 --2- 172.21.15.46:0/4022841594 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8ebc067170 0x7f8ebc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:47:55.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:55.913+0000 7f8ecdffb700 1 -- 172.21.15.46:0/4022841594 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8ed4053460 con 0x7f8ed8068940 2024-01-26T23:47:55.938 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:55.937+0000 7fca3d6501c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:47:55.970 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:55.969+0000 7f3fc81491c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:47:56.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.110+0000 7f8edd12f700 1 -- 172.21.15.46:0/4022841594 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8ebc067170 msgr2=0x7f8ebc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:56.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.110+0000 7f8edd12f700 1 --2- 172.21.15.46:0/4022841594 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8ebc067170 0x7f8ebc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:47:56.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.120+0000 7f8edfb94700 1 -- 172.21.15.46:0/4022841594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8ed80cc830 con 0x7f8ed8068940 2024-01-26T23:47:56.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.121+0000 7f8ecdffb700 1 -- 172.21.15.46:0/4022841594 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v117) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8ed4029020 con 0x7f8ed8068940 2024-01-26T23:47:56.122 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-26T23:47:56.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.123+0000 7f8edfb94700 1 -- 172.21.15.46:0/4022841594 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8ebc067170 msgr2=0x7f8ebc069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:47:56.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.123+0000 7f8edfb94700 1 --2- 172.21.15.46:0/4022841594 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8ebc067170 0x7f8ebc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.123+0000 7f8edfb94700 1 -- 172.21.15.46:0/4022841594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8068940 msgr2=0x7f8ed80c5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:56.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.123+0000 7f8edfb94700 1 --2- 172.21.15.46:0/4022841594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8068940 0x7f8ed80c5580 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8069dc0 tx=0x7f8ed4016000 comp rx=0 tx=0).stop 2024-01-26T23:47:56.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.123+0000 7f8edfb94700 1 -- 172.21.15.46:0/4022841594 shutdown_connections 2024-01-26T23:47:56.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.123+0000 7f8edfb94700 1 --2- 172.21.15.46:0/4022841594 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8ebc067170 0x7f8ebc069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.123+0000 7f8edfb94700 1 --2- 172.21.15.46:0/4022841594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8068940 0x7f8ed80c5580 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.123+0000 7f8edfb94700 1 --2- 172.21.15.46:0/4022841594 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ed80ab6e0 0x7f8ed816a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.123+0000 7f8edfb94700 1 --2- 172.21.15.46:0/4022841594 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed80ab310 0x7f8ed8169da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.123+0000 7f8edfb94700 1 -- 172.21.15.46:0/4022841594 >> 172.21.15.46:0/4022841594 conn(0x7f8ed80ca100 msgr2=0x7f8ed8067f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:56.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.123+0000 7f8edfb94700 1 -- 172.21.15.46:0/4022841594 shutdown_connections 2024-01-26T23:47:56.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.123+0000 7f8edfb94700 1 -- 172.21.15.46:0/4022841594 wait complete. 2024-01-26T23:47:56.138 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:47:56.234 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:56.233+0000 7f3fc81491c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:47:56.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.270+0000 7f0c789fd700 1 Processor -- start 2024-01-26T23:47:56.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.270+0000 7f0c789fd700 1 -- start start 2024-01-26T23:47:56.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.271+0000 7f0c789fd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c740c3cf0 0x7f0c740ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.271+0000 7f0c789fd700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c740cbf10 con 0x7f0c740acf30 2024-01-26T23:47:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.271+0000 7f0c789fd700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c740cd4a0 con 0x7f0c740cd080 2024-01-26T23:47:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.271+0000 7f0c789fd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c740ada10 con 0x7f0c740c3cf0 2024-01-26T23:47:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.271+0000 7f0c7259c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0c740acf30 0x7f0c740ad300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59548/0 (socket says 172.21.15.46:59548) 2024-01-26T23:47:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.271+0000 7f0c7259c700 1 -- 172.21.15.46:0/608090962 learned_addr learned my addr 172.21.15.46:0/608090962 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:47:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.271+0000 7f0c71d9b700 1 --2- 172.21.15.46:0/608090962 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c740c3cf0 0x7f0c740ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.272+0000 7f0c7159a700 1 -- 172.21.15.46:0/608090962 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177460954 0 0) 0x7f0c740cbf10 con 0x7f0c740acf30 2024-01-26T23:47:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.272+0000 7f0c7159a700 1 -- 172.21.15.46:0/608090962 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c60003570 con 0x7f0c740acf30 2024-01-26T23:47:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.272+0000 7f0c71d9b700 1 -- 172.21.15.46:0/608090962 >> v1:172.21.15.106:6789/0 conn(0x7f0c740cd080 legacy=0x7f0c740ab1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.272+0000 7f0c7159a700 1 -- 172.21.15.46:0/608090962 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282501524 0 0) 0x7f0c740cd4a0 con 0x7f0c740cd080 2024-01-26T23:47:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.272+0000 7f0c71d9b700 1 -- 172.21.15.46:0/608090962 >> v1:172.21.15.46:6789/0 conn(0x7f0c740acf30 legacy=0x7f0c740ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.272+0000 7f0c71d9b700 1 -- 172.21.15.46:0/608090962 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c740c4460 con 0x7f0c740c3cf0 2024-01-26T23:47:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.272+0000 7f0c71d9b700 1 --2- 172.21.15.46:0/608090962 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c740c3cf0 0x7f0c740ccb40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c009910 tx=0x7f0c5c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e47ef40f839a377f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:56.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.272+0000 7f0c7159a700 1 -- 172.21.15.46:0/608090962 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c00b680 con 0x7f0c740c3cf0 2024-01-26T23:47:56.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.272+0000 7f0c7159a700 1 -- 172.21.15.46:0/608090962 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5c00b7e0 con 0x7f0c740c3cf0 2024-01-26T23:47:56.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.272+0000 7f0c7159a700 1 -- 172.21.15.46:0/608090962 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c00bba0 con 0x7f0c740c3cf0 2024-01-26T23:47:56.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.273+0000 7f0c789fd700 1 -- 172.21.15.46:0/608090962 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c740c3cf0 msgr2=0x7f0c740ccb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:56.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.273+0000 7f0c789fd700 1 --2- 172.21.15.46:0/608090962 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c740c3cf0 0x7f0c740ccb40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c009910 tx=0x7f0c5c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.273+0000 7f0c789fd700 1 -- 172.21.15.46:0/608090962 shutdown_connections 2024-01-26T23:47:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.273+0000 7f0c789fd700 1 --2- 172.21.15.46:0/608090962 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c740c3cf0 0x7f0c740ccb40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.273+0000 7f0c789fd700 1 -- 172.21.15.46:0/608090962 >> 172.21.15.46:0/608090962 conn(0x7f0c74097820 msgr2=0x7f0c74068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.273+0000 7f0c789fd700 1 -- 172.21.15.46:0/608090962 shutdown_connections 2024-01-26T23:47:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.273+0000 7f0c789fd700 1 -- 172.21.15.46:0/608090962 wait complete. 2024-01-26T23:47:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.274+0000 7f0c789fd700 1 Processor -- start 2024-01-26T23:47:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.274+0000 7f0c789fd700 1 -- start start 2024-01-26T23:47:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.275+0000 7f0c789fd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c740acf30 0x7f0c740cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.275+0000 7f0c789fd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c740c3cf0 0x7f0c74156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.275+0000 7f0c789fd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c740cd080 0x7f0c741567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.275+0000 7f0c789fd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c74153b30 con 0x7f0c740acf30 2024-01-26T23:47:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.275+0000 7f0c789fd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c740c9530 con 0x7f0c740c3cf0 2024-01-26T23:47:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.275+0000 7f0c789fd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c740c9810 con 0x7f0c740cd080 2024-01-26T23:47:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.275+0000 7f0c7259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c740acf30 0x7f0c740cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:56.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.275+0000 7f0c7259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c740acf30 0x7f0c740cb0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41232/0 (socket says 172.21.15.46:41232) 2024-01-26T23:47:56.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.275+0000 7f0c7259c700 1 -- 172.21.15.46:0/3243328205 learned_addr learned my addr 172.21.15.46:0/3243328205 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:56.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.275+0000 7f0c72d9d700 1 --2- 172.21.15.46:0/3243328205 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c740cd080 0x7f0c741567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:56.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.275+0000 7f0c71d9b700 1 --2- 172.21.15.46:0/3243328205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c740c3cf0 0x7f0c74156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:56.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.275+0000 7f0c7259c700 1 -- 172.21.15.46:0/3243328205 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c740cd080 msgr2=0x7f0c741567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:56.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.275+0000 7f0c7259c700 1 --2- 172.21.15.46:0/3243328205 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c740cd080 0x7f0c741567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.275+0000 7f0c7259c700 1 -- 172.21.15.46:0/3243328205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c740c3cf0 msgr2=0x7f0c74156260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:56.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.275+0000 7f0c7259c700 1 --2- 172.21.15.46:0/3243328205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c740c3cf0 0x7f0c74156260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.275+0000 7f0c7259c700 1 -- 172.21.15.46:0/3243328205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5c009420 con 0x7f0c740acf30 2024-01-26T23:47:56.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.276+0000 7f0c7259c700 1 --2- 172.21.15.46:0/3243328205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c740acf30 0x7f0c740cb0e0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f0c6400cc40 tx=0x7f0c64001a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:56.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.276+0000 7f0c6a7fc700 1 -- 172.21.15.46:0/3243328205 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6400a870 con 0x7f0c740acf30 2024-01-26T23:47:56.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.276+0000 7f0c6a7fc700 1 -- 172.21.15.46:0/3243328205 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6400a9d0 con 0x7f0c740acf30 2024-01-26T23:47:56.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.276+0000 7f0c789fd700 1 -- 172.21.15.46:0/3243328205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c740c9af0 con 0x7f0c740acf30 2024-01-26T23:47:56.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.276+0000 7f0c6a7fc700 1 -- 172.21.15.46:0/3243328205 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6400ad70 con 0x7f0c740acf30 2024-01-26T23:47:56.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.276+0000 7f0c789fd700 1 -- 172.21.15.46:0/3243328205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c740ca2f0 con 0x7f0c740acf30 2024-01-26T23:47:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.277+0000 7f0c789fd700 1 -- 172.21.15.46:0/3243328205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c740ca010 con 0x7f0c740acf30 2024-01-26T23:47:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.277+0000 7f0c6a7fc700 1 -- 172.21.15.46:0/3243328205 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 51) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f0c64007300 con 0x7f0c740acf30 2024-01-26T23:47:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.278+0000 7f0c6a7fc700 1 --2- 172.21.15.46:0/3243328205 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c54067170 0x7f0c54069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:56.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.278+0000 7f0c6a7fc700 1 -- 172.21.15.46:0/3243328205 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6791+0+0 (secure 0 0 0) 0x7f0c64084fe0 con 0x7f0c740acf30 2024-01-26T23:47:56.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.278+0000 7f0c71d9b700 1 -- 172.21.15.46:0/3243328205 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c54067170 msgr2=0x7f0c54069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:56.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.278+0000 7f0c71d9b700 1 --2- 172.21.15.46:0/3243328205 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c54067170 0x7f0c54069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:47:56.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.286+0000 7f0c6a7fc700 1 -- 172.21.15.46:0/3243328205 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0c64051c00 con 0x7f0c740acf30 2024-01-26T23:47:56.321 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:56.320+0000 7f3fc81491c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:47:56.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.479+0000 7f0c71d9b700 1 -- 172.21.15.46:0/3243328205 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c54067170 msgr2=0x7f0c54069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:56.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.479+0000 7f0c71d9b700 1 --2- 172.21.15.46:0/3243328205 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c54067170 0x7f0c54069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:47:56.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.538+0000 7f0c789fd700 1 -- 172.21.15.46:0/3243328205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c7404c710 con 0x7f0c740acf30 2024-01-26T23:47:56.540 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:56.538+0000 7fca3d6501c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:47:56.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.538+0000 7f0c6a7fc700 1 -- 172.21.15.46:0/3243328205 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f0c64055220 con 0x7f0c740acf30 2024-01-26T23:47:56.540 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:47:56.540 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6518,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/4188139264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":4188139264},{"type":"v1","addr":"172.21.15.106:6836","nonce":4188139264}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":48},{"gid":6527,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/221301328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":221301328},{"type":"v1","addr":"172.21.15.46:6833","nonce":221301328}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":6530,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1531399912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":1531399912},{"type":"v1","addr":"172.21.15.46:6835","nonce":1531399912}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:46:31.208843+0000","modified":"2024-01-26T23:46:34.775841+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7084":{"gid":7084,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6837/1643596655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1643596655},{"type":"v1","addr":"172.21.15.106:6837","nonce":1643596655}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2024-01-26T23:47:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.540+0000 7f0c789fd700 1 -- 172.21.15.46:0/3243328205 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c54067170 msgr2=0x7f0c54069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:47:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.540+0000 7f0c789fd700 1 --2- 172.21.15.46:0/3243328205 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c54067170 0x7f0c54069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.540+0000 7f0c789fd700 1 -- 172.21.15.46:0/3243328205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c740acf30 msgr2=0x7f0c740cb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.540+0000 7f0c789fd700 1 --2- 172.21.15.46:0/3243328205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c740acf30 0x7f0c740cb0e0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f0c6400cc40 tx=0x7f0c64001a60 comp rx=0 tx=0).stop 2024-01-26T23:47:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.541+0000 7f0c789fd700 1 -- 172.21.15.46:0/3243328205 shutdown_connections 2024-01-26T23:47:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.541+0000 7f0c789fd700 1 --2- 172.21.15.46:0/3243328205 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f0c54067170 0x7f0c54069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.541+0000 7f0c789fd700 1 --2- 172.21.15.46:0/3243328205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c740acf30 0x7f0c740cb0e0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.541+0000 7f0c789fd700 1 --2- 172.21.15.46:0/3243328205 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c740cd080 0x7f0c741567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.541+0000 7f0c789fd700 1 --2- 172.21.15.46:0/3243328205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c740c3cf0 0x7f0c74156260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.541+0000 7f0c789fd700 1 -- 172.21.15.46:0/3243328205 >> 172.21.15.46:0/3243328205 conn(0x7f0c74097820 msgr2=0x7f0c7406c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:56.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.541+0000 7f0c789fd700 1 -- 172.21.15.46:0/3243328205 shutdown_connections 2024-01-26T23:47:56.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.541+0000 7f0c789fd700 1 -- 172.21.15.46:0/3243328205 wait complete. 2024-01-26T23:47:56.543 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 54 2024-01-26T23:47:56.556 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:47:56.613 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:56.612+0000 7fca3d6501c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:47:56.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.687+0000 7f8d1dfa2700 1 Processor -- start 2024-01-26T23:47:56.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.687+0000 7f8d1dfa2700 1 -- start start 2024-01-26T23:47:56.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.688+0000 7f8d1dfa2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8d180c6520 0x7f8d180ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:56.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.688+0000 7f8d1dfa2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d180ced50 0x7f8d180cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:56.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.688+0000 7f8d1dfa2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1815a0b0 con 0x7f8d180b2b40 2024-01-26T23:47:56.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.688+0000 7f8d1dfa2700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d180af070 con 0x7f8d180c6520 2024-01-26T23:47:56.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.688+0000 7f8d1dfa2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d18099fc0 con 0x7f8d180ced50 2024-01-26T23:47:56.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.689+0000 7f8d17fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d180ced50 0x7f8d180cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:56.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.689+0000 7f8d16ffd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8d180c6520 0x7f8d180ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:56.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.689+0000 7f8d16ffd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8d180c6520 0x7f8d180ad1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:55448/0 (socket says 172.21.15.46:55448) 2024-01-26T23:47:56.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.689+0000 7f8d17fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d180ced50 0x7f8d180cf1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:42626/0 (socket says 172.21.15.46:42626) 2024-01-26T23:47:56.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.689+0000 7f8d16ffd700 1 -- 172.21.15.46:0/3823887355 learned_addr learned my addr 172.21.15.46:0/3823887355 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:56.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.689+0000 7f8d17fff700 1 -- 172.21.15.46:0/3823887355 >> v2:172.21.15.106:3300/0 conn(0x7f8d180c6520 msgr2=0x7f8d180ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:47:56.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.689+0000 7f8d17fff700 1 --2- 172.21.15.46:0/3823887355 >> v2:172.21.15.106:3300/0 conn(0x7f8d180c6520 0x7f8d180ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.689+0000 7f8d17fff700 1 -- 172.21.15.46:0/3823887355 >> v1:172.21.15.106:6789/0 conn(0x7f8d180b2b40 legacy=0x7f8d180ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:56.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.689+0000 7f8d17fff700 1 -- 172.21.15.46:0/3823887355 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d180ae4d0 con 0x7f8d180ced50 2024-01-26T23:47:56.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.689+0000 7f8d17fff700 1 --2- 172.21.15.46:0/3823887355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d180ced50 0x7f8d180cf1d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009e60 tx=0x7f8d10009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1ffb2aa8c5a8e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:56.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.689+0000 7f8d167fc700 1 -- 172.21.15.46:0/3823887355 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1000e070 con 0x7f8d180ced50 2024-01-26T23:47:56.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.689+0000 7f8d167fc700 1 -- 172.21.15.46:0/3823887355 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1000bb60 con 0x7f8d180ced50 2024-01-26T23:47:56.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.689+0000 7f8d167fc700 1 -- 172.21.15.46:0/3823887355 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10012430 con 0x7f8d180ced50 2024-01-26T23:47:56.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.690+0000 7f8d1dfa2700 1 -- 172.21.15.46:0/3823887355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d180ced50 msgr2=0x7f8d180cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:56.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.690+0000 7f8d1dfa2700 1 --2- 172.21.15.46:0/3823887355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d180ced50 0x7f8d180cf1d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009e60 tx=0x7f8d10009780 comp rx=0 tx=0).stop 2024-01-26T23:47:56.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.690+0000 7f8d1dfa2700 1 -- 172.21.15.46:0/3823887355 shutdown_connections 2024-01-26T23:47:56.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.690+0000 7f8d1dfa2700 1 --2- 172.21.15.46:0/3823887355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d180ced50 0x7f8d180cf1d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.690+0000 7f8d1dfa2700 1 --2- 172.21.15.46:0/3823887355 >> v2:172.21.15.106:3300/0 conn(0x7f8d180c6520 0x7f8d180ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.690+0000 7f8d1dfa2700 1 -- 172.21.15.46:0/3823887355 >> 172.21.15.46:0/3823887355 conn(0x7f8d18099a30 msgr2=0x7f8d1806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:56.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.690+0000 7f8d1dfa2700 1 -- 172.21.15.46:0/3823887355 shutdown_connections 2024-01-26T23:47:56.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.690+0000 7f8d1dfa2700 1 -- 172.21.15.46:0/3823887355 wait complete. 2024-01-26T23:47:56.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.692+0000 7f8d1dfa2700 1 Processor -- start 2024-01-26T23:47:56.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.692+0000 7f8d1dfa2700 1 -- start start 2024-01-26T23:47:56.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.692+0000 7f8d1dfa2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d180b2b40 0x7f8d1814b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:56.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.692+0000 7f8d1dfa2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d180c6520 0x7f8d180c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:56.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.692+0000 7f8d1dfa2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d180ced50 0x7f8d180c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.692+0000 7f8d1dfa2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1814b9f0 con 0x7f8d180ced50 2024-01-26T23:47:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.692+0000 7f8d1dfa2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d180af1b0 con 0x7f8d180b2b40 2024-01-26T23:47:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.692+0000 7f8d1dfa2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d180c86a0 con 0x7f8d180c6520 2024-01-26T23:47:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.693+0000 7f8d17fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d180ced50 0x7f8d180c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.693+0000 7f8d177fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d180b2b40 0x7f8d1814b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.693+0000 7f8d17fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d180ced50 0x7f8d180c9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41234/0 (socket says 172.21.15.46:41234) 2024-01-26T23:47:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.693+0000 7f8d16ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d180c6520 0x7f8d180c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.693+0000 7f8d17fff700 1 -- 172.21.15.46:0/1273361822 learned_addr learned my addr 172.21.15.46:0/1273361822 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.693+0000 7f8d17fff700 1 -- 172.21.15.46:0/1273361822 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d180c6520 msgr2=0x7f8d180c95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.693+0000 7f8d17fff700 1 --2- 172.21.15.46:0/1273361822 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d180c6520 0x7f8d180c95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.693+0000 7f8d17fff700 1 -- 172.21.15.46:0/1273361822 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d180b2b40 msgr2=0x7f8d1814b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.693+0000 7f8d17fff700 1 --2- 172.21.15.46:0/1273361822 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d180b2b40 0x7f8d1814b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.693+0000 7f8d17fff700 1 -- 172.21.15.46:0/1273361822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d10009a20 con 0x7f8d180ced50 2024-01-26T23:47:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.693+0000 7f8d177fe700 1 --2- 172.21.15.46:0/1273361822 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d180b2b40 0x7f8d1814b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:47:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.693+0000 7f8d16ffd700 1 --2- 172.21.15.46:0/1273361822 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d180c6520 0x7f8d180c95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:47:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.693+0000 7f8d17fff700 1 --2- 172.21.15.46:0/1273361822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d180ced50 0x7f8d180c9af0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000b6c0 tx=0x7f8d10019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.693+0000 7f8cff7fe700 1 -- 172.21.15.46:0/1273361822 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10012e90 con 0x7f8d180ced50 2024-01-26T23:47:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.694+0000 7f8cff7fe700 1 -- 172.21.15.46:0/1273361822 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d10027470 con 0x7f8d180ced50 2024-01-26T23:47:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.694+0000 7f8d1dfa2700 1 -- 172.21.15.46:0/1273361822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d180c8980 con 0x7f8d180ced50 2024-01-26T23:47:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.694+0000 7f8cff7fe700 1 -- 172.21.15.46:0/1273361822 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10027810 con 0x7f8d180ced50 2024-01-26T23:47:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.694+0000 7f8d1dfa2700 1 -- 172.21.15.46:0/1273361822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d180af400 con 0x7f8d180ced50 2024-01-26T23:47:56.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.695+0000 7f8d1dfa2700 1 -- 172.21.15.46:0/1273361822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d180af620 con 0x7f8d180ced50 2024-01-26T23:47:56.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.695+0000 7f8cff7fe700 1 -- 172.21.15.46:0/1273361822 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 51) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f8d10024070 con 0x7f8d180ced50 2024-01-26T23:47:56.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.696+0000 7f8cff7fe700 1 --2- 172.21.15.46:0/1273361822 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d04067170 0x7f8d04069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:56.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.696+0000 7f8d177fe700 1 -- 172.21.15.46:0/1273361822 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d04067170 msgr2=0x7f8d04069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:56.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.696+0000 7f8cff7fe700 1 -- 172.21.15.46:0/1273361822 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6791+0+0 (secure 0 0 0) 0x7f8d100877c0 con 0x7f8d180ced50 2024-01-26T23:47:56.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.696+0000 7f8d177fe700 1 --2- 172.21.15.46:0/1273361822 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d04067170 0x7f8d04069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:47:56.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.700+0000 7f8cff7fe700 1 -- 172.21.15.46:0/1273361822 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8d100542c0 con 0x7f8d180ced50 2024-01-26T23:47:56.738 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:56.738+0000 7fca3d6501c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:47:56.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.897+0000 7f8d177fe700 1 -- 172.21.15.46:0/1273361822 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d04067170 msgr2=0x7f8d04069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:56.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.897+0000 7f8d177fe700 1 --2- 172.21.15.46:0/1273361822 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d04067170 0x7f8d04069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:47:56.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.911+0000 7f8d1dfa2700 1 -- 172.21.15.46:0/1273361822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d1804c710 con 0x7f8d180ced50 2024-01-26T23:47:56.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.912+0000 7f8cff7fe700 1 -- 172.21.15.46:0/1273361822 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+16729 (secure 0 0 0) 0x7f8d100578e0 con 0x7f8d180ced50 2024-01-26T23:47:56.913 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:47:56.914 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":117,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:47:51.713551+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:46:29.702709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-01-26T23:46:30.178825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.2c","mappings":[{"from":1,"to":2}]},{"pgid":"14.33","mappings":[{"from":1,"to":2}]},{"pgid":"14.3a","mappings":[{"from":1,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/4144593394":"2024-01-27T23:46:50.618051+0000","172.21.15.106:0/2301684208":"2024-01-27T23:46:50.618051+0000","172.21.15.106:0/1989924028":"2024-01-27T23:46:50.618051+0000","172.21.15.106:6833/43388":"2024-01-27T23:46:50.618051+0000","172.21.15.106:6832/43388":"2024-01-27T23:46:50.618051+0000","172.21.15.106:0/1323704418":"2024-01-27T00:46:44.670099+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:47:56.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.914+0000 7f8d1dfa2700 1 -- 172.21.15.46:0/1273361822 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d04067170 msgr2=0x7f8d04069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:47:56.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.914+0000 7f8d1dfa2700 1 --2- 172.21.15.46:0/1273361822 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d04067170 0x7f8d04069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.914+0000 7f8d1dfa2700 1 -- 172.21.15.46:0/1273361822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d180ced50 msgr2=0x7f8d180c9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:56.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.914+0000 7f8d1dfa2700 1 --2- 172.21.15.46:0/1273361822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d180ced50 0x7f8d180c9af0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000b6c0 tx=0x7f8d10019000 comp rx=0 tx=0).stop 2024-01-26T23:47:56.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.914+0000 7f8d1dfa2700 1 -- 172.21.15.46:0/1273361822 shutdown_connections 2024-01-26T23:47:56.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.914+0000 7f8d1dfa2700 1 --2- 172.21.15.46:0/1273361822 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f8d04067170 0x7f8d04069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.914+0000 7f8d1dfa2700 1 --2- 172.21.15.46:0/1273361822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d180ced50 0x7f8d180c9af0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.914+0000 7f8d1dfa2700 1 --2- 172.21.15.46:0/1273361822 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d180b2b40 0x7f8d1814b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.914+0000 7f8d1dfa2700 1 --2- 172.21.15.46:0/1273361822 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d180c6520 0x7f8d180c95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:56.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.914+0000 7f8d1dfa2700 1 -- 172.21.15.46:0/1273361822 >> 172.21.15.46:0/1273361822 conn(0x7f8d18099a30 msgr2=0x7f8d18069bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:56.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.914+0000 7f8d1dfa2700 1 -- 172.21.15.46:0/1273361822 shutdown_connections 2024-01-26T23:47:56.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:56.914+0000 7f8d1dfa2700 1 -- 172.21.15.46:0/1273361822 wait complete. 2024-01-26T23:47:56.930 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-26T23:47:56.930 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:47:57.034 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:57.033+0000 7f3fc81491c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:47:57.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.062+0000 7fea979fd700 1 Processor -- start 2024-01-26T23:47:57.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.062+0000 7fea979fd700 1 -- start start 2024-01-26T23:47:57.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.063+0000 7fea979fd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea90154290 0x7fea900b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:57.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.063+0000 7fea979fd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fea900b2100 0x7fea900b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:57.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.063+0000 7fea979fd700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea9015b140 con 0x7fea90153ec0 2024-01-26T23:47:57.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.063+0000 7fea979fd700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea901546e0 con 0x7fea900b2100 2024-01-26T23:47:57.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.063+0000 7fea979fd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea900b2580 con 0x7fea90154290 2024-01-26T23:47:57.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.063+0000 7fea94f98700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea90154290 0x7fea900b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.063+0000 7fea95f9a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fea900b2100 0x7fea900b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.063+0000 7fea94f98700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea90154290 0x7fea900b1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:42632/0 (socket says 172.21.15.46:42632) 2024-01-26T23:47:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.063+0000 7fea94f98700 1 -- 172.21.15.46:0/3892524412 learned_addr learned my addr 172.21.15.46:0/3892524412 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.064+0000 7fea94f98700 1 -- 172.21.15.46:0/3892524412 >> v2:172.21.15.106:3300/0 conn(0x7fea900b2100 msgr2=0x7fea900b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:47:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.064+0000 7fea94f98700 1 --2- 172.21.15.46:0/3892524412 >> v2:172.21.15.106:3300/0 conn(0x7fea900b2100 0x7fea900b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.064+0000 7fea94f98700 1 -- 172.21.15.46:0/3892524412 >> v1:172.21.15.106:6789/0 conn(0x7fea90153ec0 legacy=0x7fea90157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.064+0000 7fea94f98700 1 -- 172.21.15.46:0/3892524412 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea9015a5c0 con 0x7fea90154290 2024-01-26T23:47:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.064+0000 7fea94f98700 1 --2- 172.21.15.46:0/3892524412 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea90154290 0x7fea900b1bc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fea8000bc50 tx=0x7fea8000b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=222d2583fb400a83 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.064+0000 7fea87fff700 1 -- 172.21.15.46:0/3892524412 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8000dbf0 con 0x7fea90154290 2024-01-26T23:47:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.064+0000 7fea87fff700 1 -- 172.21.15.46:0/3892524412 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8000dd50 con 0x7fea90154290 2024-01-26T23:47:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.064+0000 7fea87fff700 1 -- 172.21.15.46:0/3892524412 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea800196b0 con 0x7fea90154290 2024-01-26T23:47:57.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.065+0000 7fea979fd700 1 -- 172.21.15.46:0/3892524412 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea90154290 msgr2=0x7fea900b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:57.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.065+0000 7fea979fd700 1 --2- 172.21.15.46:0/3892524412 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea90154290 0x7fea900b1bc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fea8000bc50 tx=0x7fea8000b4f0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.065+0000 7fea979fd700 1 -- 172.21.15.46:0/3892524412 shutdown_connections 2024-01-26T23:47:57.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.065+0000 7fea979fd700 1 --2- 172.21.15.46:0/3892524412 >> v2:172.21.15.106:3300/0 conn(0x7fea900b2100 0x7fea900b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.065+0000 7fea979fd700 1 --2- 172.21.15.46:0/3892524412 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea90154290 0x7fea900b1bc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.065+0000 7fea979fd700 1 -- 172.21.15.46:0/3892524412 >> 172.21.15.46:0/3892524412 conn(0x7fea90069510 msgr2=0x7fea900aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:57.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.065+0000 7fea979fd700 1 -- 172.21.15.46:0/3892524412 shutdown_connections 2024-01-26T23:47:57.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.065+0000 7fea979fd700 1 -- 172.21.15.46:0/3892524412 wait complete. 2024-01-26T23:47:57.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.066+0000 7fea979fd700 1 Processor -- start 2024-01-26T23:47:57.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.066+0000 7fea979fd700 1 -- start start 2024-01-26T23:47:57.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.067+0000 7fea979fd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea900b2100 0x7fea90143db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:57.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.067+0000 7fea979fd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea90153ec0 0x7fea900aac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:57.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.067+0000 7fea979fd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea90154290 0x7fea900ab180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:57.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.067+0000 7fea979fd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9016d400 con 0x7fea900b2100 2024-01-26T23:47:57.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.067+0000 7fea979fd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9016df80 con 0x7fea90153ec0 2024-01-26T23:47:57.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.067+0000 7fea979fd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9016d650 con 0x7fea90154290 2024-01-26T23:47:57.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.067+0000 7fea95799700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea900b2100 0x7fea90143db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:57.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.067+0000 7fea94f98700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea90153ec0 0x7fea900aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:57.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.067+0000 7fea95799700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea900b2100 0x7fea90143db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41244/0 (socket says 172.21.15.46:41244) 2024-01-26T23:47:57.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.067+0000 7fea95f9a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea90154290 0x7fea900ab180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:57.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.067+0000 7fea95799700 1 -- 172.21.15.46:0/2463619945 learned_addr learned my addr 172.21.15.46:0/2463619945 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:57.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.068+0000 7fea94f98700 1 -- 172.21.15.46:0/2463619945 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea90154290 msgr2=0x7fea900ab180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:57.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.068+0000 7fea94f98700 1 --2- 172.21.15.46:0/2463619945 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea90154290 0x7fea900ab180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.068+0000 7fea94f98700 1 -- 172.21.15.46:0/2463619945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea900b2100 msgr2=0x7fea90143db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:57.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.068+0000 7fea94f98700 1 --2- 172.21.15.46:0/2463619945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea900b2100 0x7fea90143db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.068+0000 7fea94f98700 1 -- 172.21.15.46:0/2463619945 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8000b790 con 0x7fea90153ec0 2024-01-26T23:47:57.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.068+0000 7fea95f9a700 1 --2- 172.21.15.46:0/2463619945 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea90154290 0x7fea900ab180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:47:57.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.068+0000 7fea95799700 1 --2- 172.21.15.46:0/2463619945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea900b2100 0x7fea90143db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:47:57.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.068+0000 7fea94f98700 1 --2- 172.21.15.46:0/2463619945 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea90153ec0 0x7fea900aac40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fea80018000 tx=0x7fea8001b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:57.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.068+0000 7fea857fa700 1 -- 172.21.15.46:0/2463619945 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80010680 con 0x7fea90153ec0 2024-01-26T23:47:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.068+0000 7fea857fa700 1 -- 172.21.15.46:0/2463619945 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea80010820 con 0x7fea90153ec0 2024-01-26T23:47:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.068+0000 7fea857fa700 1 -- 172.21.15.46:0/2463619945 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80010be0 con 0x7fea90153ec0 2024-01-26T23:47:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.068+0000 7fea979fd700 1 -- 172.21.15.46:0/2463619945 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea9016d930 con 0x7fea90153ec0 2024-01-26T23:47:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.068+0000 7fea979fd700 1 -- 172.21.15.46:0/2463619945 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea901442f0 con 0x7fea90153ec0 2024-01-26T23:47:57.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.069+0000 7fea979fd700 1 -- 172.21.15.46:0/2463619945 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea90064be0 con 0x7fea90153ec0 2024-01-26T23:47:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.070+0000 7fea857fa700 1 -- 172.21.15.46:0/2463619945 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 51) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fea8002c440 con 0x7fea90153ec0 2024-01-26T23:47:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.070+0000 7fea857fa700 1 --2- 172.21.15.46:0/2463619945 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fea74067170 0x7fea74069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.070+0000 7fea857fa700 1 -- 172.21.15.46:0/2463619945 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6791+0+0 (secure 0 0 0) 0x7fea80014070 con 0x7fea90153ec0 2024-01-26T23:47:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.070+0000 7fea95799700 1 -- 172.21.15.46:0/2463619945 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fea74067170 msgr2=0x7fea74069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.070+0000 7fea95799700 1 --2- 172.21.15.46:0/2463619945 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fea74067170 0x7fea74069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:47:57.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.074+0000 7fea857fa700 1 -- 172.21.15.46:0/2463619945 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fea80053b20 con 0x7fea90153ec0 2024-01-26T23:47:57.123 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:57.122+0000 7f3fc81491c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:47:57.238 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:57.237+0000 7fca3d6501c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:47:57.265 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:57.264+0000 7f3fc81491c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:47:57.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.271+0000 7fea95799700 1 -- 172.21.15.46:0/2463619945 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fea74067170 msgr2=0x7fea74069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:57.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.271+0000 7fea95799700 1 --2- 172.21.15.46:0/2463619945 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fea74067170 0x7fea74069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:47:57.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.331+0000 7fea979fd700 1 -- 172.21.15.46:0/2463619945 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea9004c710 con 0x7fea90153ec0 2024-01-26T23:47:57.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.331+0000 7fea857fa700 1 -- 172.21.15.46:0/2463619945 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4602 (secure 0 0 0) 0x7fea80057140 con 0x7fea90153ec0 2024-01-26T23:47:57.333 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:47:57.333 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6518,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/4188139264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":4188139264},{"type":"v1","addr":"172.21.15.106:6836","nonce":4188139264}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":48},{"gid":6527,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/221301328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":221301328},{"type":"v1","addr":"172.21.15.46:6833","nonce":221301328}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":6530,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1531399912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":1531399912},{"type":"v1","addr":"172.21.15.46:6835","nonce":1531399912}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:46:31.208843+0000","modified":"2024-01-26T23:46:34.775841+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7084":{"gid":7084,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6837/1643596655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1643596655},{"type":"v1","addr":"172.21.15.106:6837","nonce":1643596655}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2024-01-26T23:47:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.334+0000 7fea979fd700 1 -- 172.21.15.46:0/2463619945 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fea74067170 msgr2=0x7fea74069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:47:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.334+0000 7fea979fd700 1 --2- 172.21.15.46:0/2463619945 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fea74067170 0x7fea74069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.334+0000 7fea979fd700 1 -- 172.21.15.46:0/2463619945 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea90153ec0 msgr2=0x7fea900aac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.334+0000 7fea979fd700 1 --2- 172.21.15.46:0/2463619945 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea90153ec0 0x7fea900aac40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fea80018000 tx=0x7fea8001b000 comp rx=0 tx=0).stop 2024-01-26T23:47:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.334+0000 7fea979fd700 1 -- 172.21.15.46:0/2463619945 shutdown_connections 2024-01-26T23:47:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.334+0000 7fea979fd700 1 --2- 172.21.15.46:0/2463619945 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7fea74067170 0x7fea74069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.334+0000 7fea979fd700 1 --2- 172.21.15.46:0/2463619945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea900b2100 0x7fea90143db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.334+0000 7fea979fd700 1 --2- 172.21.15.46:0/2463619945 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea90154290 0x7fea900ab180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.334+0000 7fea979fd700 1 --2- 172.21.15.46:0/2463619945 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea90153ec0 0x7fea900aac40 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.334+0000 7fea979fd700 1 -- 172.21.15.46:0/2463619945 >> 172.21.15.46:0/2463619945 conn(0x7fea90069510 msgr2=0x7fea90154820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.334+0000 7fea979fd700 1 -- 172.21.15.46:0/2463619945 shutdown_connections 2024-01-26T23:47:57.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.334+0000 7fea979fd700 1 -- 172.21.15.46:0/2463619945 wait complete. 2024-01-26T23:47:57.337 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 54 2024-01-26T23:47:57.349 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:47:57.388 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:57.387+0000 7fca3d6501c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:47:57.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.481+0000 7f7b890ac700 1 Processor -- start 2024-01-26T23:47:57.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.481+0000 7f7b890ac700 1 -- start start 2024-01-26T23:47:57.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.483+0000 7f7b890ac700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7b8415ce40 0x7f7b840d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.483+0000 7f7b890ac700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7b8415d210 0x7f7b840cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.483+0000 7f7b890ac700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b840cff70 con 0x7f7b840d1390 2024-01-26T23:47:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.483+0000 7f7b890ac700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b840cf400 con 0x7f7b8415d210 2024-01-26T23:47:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.483+0000 7f7b890ac700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b840cf660 con 0x7f7b8415ce40 2024-01-26T23:47:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.483+0000 7f7b8359e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7b840d1390 0x7f7b840ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59564/0 (socket says 172.21.15.46:59564) 2024-01-26T23:47:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.483+0000 7f7b8359e700 1 -- 172.21.15.46:0/3510186406 learned_addr learned my addr 172.21.15.46:0/3510186406 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:47:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.483+0000 7f7b82d9d700 1 --2- 172.21.15.46:0/3510186406 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7b8415ce40 0x7f7b840d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.483+0000 7f7b8259c700 1 --2- 172.21.15.46:0/3510186406 >> v2:172.21.15.106:3300/0 conn(0x7f7b8415d210 0x7f7b840cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.483+0000 7f7b81d9b700 1 -- 172.21.15.46:0/3510186406 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852957251 0 0) 0x7f7b840cff70 con 0x7f7b840d1390 2024-01-26T23:47:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.483+0000 7f7b81d9b700 1 -- 172.21.15.46:0/3510186406 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b70003570 con 0x7f7b840d1390 2024-01-26T23:47:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.483+0000 7f7b82d9d700 1 -- 172.21.15.46:0/3510186406 >> v2:172.21.15.106:3300/0 conn(0x7f7b8415d210 msgr2=0x7f7b840cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:47:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.483+0000 7f7b82d9d700 1 --2- 172.21.15.46:0/3510186406 >> v2:172.21.15.106:3300/0 conn(0x7f7b8415d210 0x7f7b840cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.483+0000 7f7b82d9d700 1 -- 172.21.15.46:0/3510186406 >> v1:172.21.15.46:6789/0 conn(0x7f7b840d1390 legacy=0x7f7b840ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:57.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.483+0000 7f7b82d9d700 1 -- 172.21.15.46:0/3510186406 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b840d0500 con 0x7f7b8415ce40 2024-01-26T23:47:57.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.483+0000 7f7b82d9d700 1 --2- 172.21.15.46:0/3510186406 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7b8415ce40 0x7f7b840d0e50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009eb0 tx=0x7f7b74009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=269d10d9090694a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:57.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.484+0000 7f7b81d9b700 1 -- 172.21.15.46:0/3510186406 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7400e070 con 0x7f7b8415ce40 2024-01-26T23:47:57.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.484+0000 7f7b81d9b700 1 -- 172.21.15.46:0/3510186406 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b7400bb80 con 0x7f7b8415ce40 2024-01-26T23:47:57.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.484+0000 7f7b81d9b700 1 -- 172.21.15.46:0/3510186406 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b74012430 con 0x7f7b8415ce40 2024-01-26T23:47:57.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.484+0000 7f7b890ac700 1 -- 172.21.15.46:0/3510186406 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7b8415ce40 msgr2=0x7f7b840d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:57.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.484+0000 7f7b890ac700 1 --2- 172.21.15.46:0/3510186406 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7b8415ce40 0x7f7b840d0e50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009eb0 tx=0x7f7b74009780 comp rx=0 tx=0).stop 2024-01-26T23:47:57.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.484+0000 7f7b890ac700 1 -- 172.21.15.46:0/3510186406 shutdown_connections 2024-01-26T23:47:57.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.484+0000 7f7b890ac700 1 --2- 172.21.15.46:0/3510186406 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7b8415ce40 0x7f7b840d0e50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.484+0000 7f7b890ac700 1 --2- 172.21.15.46:0/3510186406 >> v2:172.21.15.106:3300/0 conn(0x7f7b8415d210 0x7f7b840cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.484+0000 7f7b890ac700 1 -- 172.21.15.46:0/3510186406 >> 172.21.15.46:0/3510186406 conn(0x7f7b8406f840 msgr2=0x7f7b840d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:57.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.484+0000 7f7b890ac700 1 -- 172.21.15.46:0/3510186406 shutdown_connections 2024-01-26T23:47:57.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.484+0000 7f7b890ac700 1 -- 172.21.15.46:0/3510186406 wait complete. 2024-01-26T23:47:57.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.486+0000 7f7b890ac700 1 Processor -- start 2024-01-26T23:47:57.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.486+0000 7f7b890ac700 1 -- start start 2024-01-26T23:47:57.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.487+0000 7f7b890ac700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b840d1390 0x7f7b840c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:57.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.487+0000 7f7b890ac700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7b8415ce40 0x7f7b840c9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:57.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.487+0000 7f7b890ac700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7b8415d210 0x7f7b8416d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:57.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.487+0000 7f7b890ac700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8415c540 con 0x7f7b840d1390 2024-01-26T23:47:57.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.487+0000 7f7b890ac700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8409b7b0 con 0x7f7b8415ce40 2024-01-26T23:47:57.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.487+0000 7f7b890ac700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8409ba90 con 0x7f7b8415d210 2024-01-26T23:47:57.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.487+0000 7f7b82d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b840d1390 0x7f7b840c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:57.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.487+0000 7f7b8359e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7b8415d210 0x7f7b8416d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:57.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.487+0000 7f7b82d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b840d1390 0x7f7b840c7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41256/0 (socket says 172.21.15.46:41256) 2024-01-26T23:47:57.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.487+0000 7f7b8259c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7b8415ce40 0x7f7b840c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:57.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.487+0000 7f7b82d9d700 1 -- 172.21.15.46:0/7386805 learned_addr learned my addr 172.21.15.46:0/7386805 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:57.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.487+0000 7f7b8359e700 1 -- 172.21.15.46:0/7386805 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7b8415ce40 msgr2=0x7f7b840c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.487+0000 7f7b8359e700 1 --2- 172.21.15.46:0/7386805 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7b8415ce40 0x7f7b840c9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.487+0000 7f7b8359e700 1 -- 172.21.15.46:0/7386805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b840d1390 msgr2=0x7f7b840c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.487+0000 7f7b8359e700 1 --2- 172.21.15.46:0/7386805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b840d1390 0x7f7b840c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.487+0000 7f7b8359e700 1 -- 172.21.15.46:0/7386805 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b74009a20 con 0x7f7b8415d210 2024-01-26T23:47:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.487+0000 7f7b8359e700 1 --2- 172.21.15.46:0/7386805 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7b8415d210 0x7f7b8416d5d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0058d0 tx=0x7f7b7c007d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.488+0000 7f7b7affd700 1 -- 172.21.15.46:0/7386805 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c005b30 con 0x7f7b8415d210 2024-01-26T23:47:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.488+0000 7f7b7affd700 1 -- 172.21.15.46:0/7386805 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b7c005c90 con 0x7f7b8415d210 2024-01-26T23:47:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.488+0000 7f7b7affd700 1 -- 172.21.15.46:0/7386805 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c0072e0 con 0x7f7b8415d210 2024-01-26T23:47:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.488+0000 7f7b890ac700 1 -- 172.21.15.46:0/7386805 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8409bd70 con 0x7f7b8415d210 2024-01-26T23:47:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.488+0000 7f7b890ac700 1 -- 172.21.15.46:0/7386805 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8409c330 con 0x7f7b8415d210 2024-01-26T23:47:57.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.489+0000 7f7b7affd700 1 -- 172.21.15.46:0/7386805 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 51) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f7b7c007690 con 0x7f7b8415d210 2024-01-26T23:47:57.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.489+0000 7f7b890ac700 1 -- 172.21.15.46:0/7386805 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8404c710 con 0x7f7b8415d210 2024-01-26T23:47:57.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.490+0000 7f7b7affd700 1 --2- 172.21.15.46:0/7386805 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7b64067170 0x7f7b64069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:57.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.490+0000 7f7b82d9d700 1 -- 172.21.15.46:0/7386805 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7b64067170 msgr2=0x7f7b64069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:57.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.490+0000 7f7b7affd700 1 -- 172.21.15.46:0/7386805 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6791+0+0 (secure 0 0 0) 0x7f7b7c083fb0 con 0x7f7b8415d210 2024-01-26T23:47:57.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.490+0000 7f7b82d9d700 1 --2- 172.21.15.46:0/7386805 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7b64067170 0x7f7b64069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:47:57.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.494+0000 7f7b7affd700 1 -- 172.21.15.46:0/7386805 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b7c050ad0 con 0x7f7b8415d210 2024-01-26T23:47:57.540 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:57.539+0000 7fca3d6501c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:47:57.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.691+0000 7f7b82d9d700 1 -- 172.21.15.46:0/7386805 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7b64067170 msgr2=0x7f7b64069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:57.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.691+0000 7f7b82d9d700 1 --2- 172.21.15.46:0/7386805 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7b64067170 0x7f7b64069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:47:57.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.782+0000 7f7b890ac700 1 -- 172.21.15.46:0/7386805 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b8409bbd0 con 0x7f7b8415d210 2024-01-26T23:47:57.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.783+0000 7f7b7affd700 1 -- 172.21.15.46:0/7386805 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f7b8409bbd0 con 0x7f7b8415d210 2024-01-26T23:47:57.784 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:47:57.784 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6518,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/4188139264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":4188139264},{"type":"v1","addr":"172.21.15.106:6836","nonce":4188139264}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":48},{"gid":6527,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/221301328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":221301328},{"type":"v1","addr":"172.21.15.46:6833","nonce":221301328}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":6530,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1531399912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":1531399912},{"type":"v1","addr":"172.21.15.46:6835","nonce":1531399912}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:46:31.208843+0000","modified":"2024-01-26T23:46:34.775841+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7084":{"gid":7084,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6837/1643596655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":1643596655},{"type":"v1","addr":"172.21.15.106:6837","nonce":1643596655}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2024-01-26T23:47:57.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.785+0000 7f7b890ac700 1 -- 172.21.15.46:0/7386805 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7b64067170 msgr2=0x7f7b64069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:47:57.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.785+0000 7f7b890ac700 1 --2- 172.21.15.46:0/7386805 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7b64067170 0x7f7b64069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.785+0000 7f7b890ac700 1 -- 172.21.15.46:0/7386805 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7b8415d210 msgr2=0x7f7b8416d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:57.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.785+0000 7f7b890ac700 1 --2- 172.21.15.46:0/7386805 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7b8415d210 0x7f7b8416d5d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0058d0 tx=0x7f7b7c007d10 comp rx=0 tx=0).stop 2024-01-26T23:47:57.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.785+0000 7f7b890ac700 1 -- 172.21.15.46:0/7386805 shutdown_connections 2024-01-26T23:47:57.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.785+0000 7f7b890ac700 1 --2- 172.21.15.46:0/7386805 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f7b64067170 0x7f7b64069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.785+0000 7f7b890ac700 1 --2- 172.21.15.46:0/7386805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b840d1390 0x7f7b840c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.785+0000 7f7b890ac700 1 --2- 172.21.15.46:0/7386805 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7b8415d210 0x7f7b8416d5d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.785+0000 7f7b890ac700 1 --2- 172.21.15.46:0/7386805 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7b8415ce40 0x7f7b840c9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.785+0000 7f7b890ac700 1 -- 172.21.15.46:0/7386805 >> 172.21.15.46:0/7386805 conn(0x7f7b8406f840 msgr2=0x7f7b840689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:57.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.785+0000 7f7b890ac700 1 -- 172.21.15.46:0/7386805 shutdown_connections 2024-01-26T23:47:57.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.785+0000 7f7b890ac700 1 -- 172.21.15.46:0/7386805 wait complete. 2024-01-26T23:47:57.788 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 54 2024-01-26T23:47:57.801 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:47:57.882 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:57.881+0000 7f3fc81491c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:47:57.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.932+0000 7f5531676700 1 Processor -- start 2024-01-26T23:47:57.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.932+0000 7f5531676700 1 -- start start 2024-01-26T23:47:57.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.933+0000 7f5531676700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f552c0cbdf0 0x7f552c0ae610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:57.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.933+0000 7f5531676700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f552c0ae180 con 0x7f552c0ad070 2024-01-26T23:47:57.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.933+0000 7f5531676700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f552c156c60 con 0x7f552c0cc1c0 2024-01-26T23:47:57.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.933+0000 7f5531676700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f552c0aeb50 con 0x7f552c0cbdf0 2024-01-26T23:47:57.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.933+0000 7f552affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f552c0cbdf0 0x7f552c0ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:57.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.933+0000 7f552b7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f552c0ad070 0x7f552c0c98b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59574/0 (socket says 172.21.15.46:59574) 2024-01-26T23:47:57.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.933+0000 7f552b7fe700 1 -- 172.21.15.46:0/3284275311 learned_addr learned my addr 172.21.15.46:0/3284275311 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:47:57.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.933+0000 7f552affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f552c0cbdf0 0x7f552c0ae610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41270/0 (socket says 172.21.15.46:41270) 2024-01-26T23:47:57.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.934+0000 7f552affd700 1 -- 172.21.15.46:0/3284275311 >> v1:172.21.15.106:6789/0 conn(0x7f552c0cc1c0 legacy=0x7f552c0b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:57.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.934+0000 7f552affd700 1 -- 172.21.15.46:0/3284275311 >> v1:172.21.15.46:6789/0 conn(0x7f552c0ad070 legacy=0x7f552c0c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:57.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.934+0000 7f552affd700 1 -- 172.21.15.46:0/3284275311 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f552c0ad570 con 0x7f552c0cbdf0 2024-01-26T23:47:57.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.934+0000 7f552affd700 1 --2- 172.21.15.46:0/3284275311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f552c0cbdf0 0x7f552c0ae610 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f551c00bb90 tx=0x7f551c00b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71bf4556c1c74d12 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:57.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.934+0000 7f5529ffb700 1 -- 172.21.15.46:0/3284275311 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c00dba0 con 0x7f552c0cbdf0 2024-01-26T23:47:57.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.934+0000 7f5529ffb700 1 -- 172.21.15.46:0/3284275311 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551c00dd00 con 0x7f552c0cbdf0 2024-01-26T23:47:57.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.934+0000 7f5529ffb700 1 -- 172.21.15.46:0/3284275311 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c0196b0 con 0x7f552c0cbdf0 2024-01-26T23:47:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.935+0000 7f5531676700 1 -- 172.21.15.46:0/3284275311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f552c0cbdf0 msgr2=0x7f552c0ae610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.935+0000 7f5531676700 1 --2- 172.21.15.46:0/3284275311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f552c0cbdf0 0x7f552c0ae610 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f551c00bb90 tx=0x7f551c00b430 comp rx=0 tx=0).stop 2024-01-26T23:47:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.935+0000 7f5531676700 1 -- 172.21.15.46:0/3284275311 shutdown_connections 2024-01-26T23:47:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.935+0000 7f5531676700 1 --2- 172.21.15.46:0/3284275311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f552c0cbdf0 0x7f552c0ae610 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.935+0000 7f5531676700 1 -- 172.21.15.46:0/3284275311 >> 172.21.15.46:0/3284275311 conn(0x7f552c068400 msgr2=0x7f552c0cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.935+0000 7f5531676700 1 -- 172.21.15.46:0/3284275311 shutdown_connections 2024-01-26T23:47:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.935+0000 7f5531676700 1 -- 172.21.15.46:0/3284275311 wait complete. 2024-01-26T23:47:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.936+0000 7f5531676700 1 Processor -- start 2024-01-26T23:47:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.936+0000 7f5531676700 1 -- start start 2024-01-26T23:47:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.937+0000 7f5531676700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f552c0ad070 0x7f552c1541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.937+0000 7f5531676700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f552c0cbdf0 0x7f552c0aa760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.937+0000 7f5531676700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f552c0a98d0 0x7f552c16a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.937+0000 7f5531676700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552c154710 con 0x7f552c0a98d0 2024-01-26T23:47:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.937+0000 7f5531676700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552c0c14e0 con 0x7f552c0cbdf0 2024-01-26T23:47:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.937+0000 7f5531676700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f552c0c17c0 con 0x7f552c0ad070 2024-01-26T23:47:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.937+0000 7f552b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f552c0a98d0 0x7f552c16a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.937+0000 7f552b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f552c0a98d0 0x7f552c16a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41274/0 (socket says 172.21.15.46:41274) 2024-01-26T23:47:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.937+0000 7f552b7fe700 1 -- 172.21.15.46:0/3182620783 learned_addr learned my addr 172.21.15.46:0/3182620783 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:57.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.937+0000 7f552affd700 1 --2- 172.21.15.46:0/3182620783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f552c0ad070 0x7f552c1541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:57.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.937+0000 7f552a7fc700 1 --2- 172.21.15.46:0/3182620783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f552c0cbdf0 0x7f552c0aa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:57.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.938+0000 7f552affd700 1 -- 172.21.15.46:0/3182620783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f552c0cbdf0 msgr2=0x7f552c0aa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:57.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.938+0000 7f552affd700 1 --2- 172.21.15.46:0/3182620783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f552c0cbdf0 0x7f552c0aa760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.938+0000 7f552affd700 1 -- 172.21.15.46:0/3182620783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f552c0a98d0 msgr2=0x7f552c16a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:57.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.938+0000 7f552affd700 1 --2- 172.21.15.46:0/3182620783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f552c0a98d0 0x7f552c16a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:57.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.938+0000 7f552affd700 1 -- 172.21.15.46:0/3182620783 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551c00b6d0 con 0x7f552c0ad070 2024-01-26T23:47:57.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.938+0000 7f552b7fe700 1 --2- 172.21.15.46:0/3182620783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f552c0a98d0 0x7f552c16a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:47:57.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.938+0000 7f552affd700 1 --2- 172.21.15.46:0/3182620783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f552c0ad070 0x7f552c1541d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f551c018000 tx=0x7f551c01b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:57.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.938+0000 7f5512ffd700 1 -- 172.21.15.46:0/3182620783 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c010470 con 0x7f552c0ad070 2024-01-26T23:47:57.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.938+0000 7f5512ffd700 1 -- 172.21.15.46:0/3182620783 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551c0105d0 con 0x7f552c0ad070 2024-01-26T23:47:57.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.938+0000 7f5512ffd700 1 -- 172.21.15.46:0/3182620783 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c010990 con 0x7f552c0ad070 2024-01-26T23:47:57.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.938+0000 7f5531676700 1 -- 172.21.15.46:0/3182620783 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f552c0c1aa0 con 0x7f552c0ad070 2024-01-26T23:47:57.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.938+0000 7f5531676700 1 -- 172.21.15.46:0/3182620783 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f552c0c2060 con 0x7f552c0ad070 2024-01-26T23:47:57.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.939+0000 7f5531676700 1 -- 172.21.15.46:0/3182620783 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f552c04c710 con 0x7f552c0ad070 2024-01-26T23:47:57.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.940+0000 7f5512ffd700 1 -- 172.21.15.46:0/3182620783 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 51) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f551c02d440 con 0x7f552c0ad070 2024-01-26T23:47:57.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.940+0000 7f5512ffd700 1 --2- 172.21.15.46:0/3182620783 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5518067170 0x7f5518069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:57.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.940+0000 7f5512ffd700 1 -- 172.21.15.46:0/3182620783 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6791+0+0 (secure 0 0 0) 0x7f551c020070 con 0x7f552c0ad070 2024-01-26T23:47:57.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.940+0000 7f552a7fc700 1 -- 172.21.15.46:0/3182620783 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5518067170 msgr2=0x7f5518069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:57.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.940+0000 7f552a7fc700 1 --2- 172.21.15.46:0/3182620783 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5518067170 0x7f5518069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:47:57.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:57.945+0000 7f5512ffd700 1 -- 172.21.15.46:0/3182620783 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f551c053860 con 0x7f552c0ad070 2024-01-26T23:47:58.057 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:58.056+0000 7f3fc81491c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:47:58.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.141+0000 7f552a7fc700 1 -- 172.21.15.46:0/3182620783 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5518067170 msgr2=0x7f5518069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:58.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.141+0000 7f552a7fc700 1 --2- 172.21.15.46:0/3182620783 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5518067170 0x7f5518069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:47:58.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.150+0000 7f5531676700 1 -- 172.21.15.46:0/3182620783 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f552c0c2250 con 0x7f552c0ad070 2024-01-26T23:47:58.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.152+0000 7f5512ffd700 1 -- 172.21.15.46:0/3182620783 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+16729 (secure 0 0 0) 0x7f551c056e80 con 0x7f552c0ad070 2024-01-26T23:47:58.153 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:47:58.153 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":117,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:47:51.713551+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:46:29.702709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-01-26T23:46:30.178825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.2c","mappings":[{"from":1,"to":2}]},{"pgid":"14.33","mappings":[{"from":1,"to":2}]},{"pgid":"14.3a","mappings":[{"from":1,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/4144593394":"2024-01-27T23:46:50.618051+0000","172.21.15.106:0/2301684208":"2024-01-27T23:46:50.618051+0000","172.21.15.106:0/1989924028":"2024-01-27T23:46:50.618051+0000","172.21.15.106:6833/43388":"2024-01-27T23:46:50.618051+0000","172.21.15.106:6832/43388":"2024-01-27T23:46:50.618051+0000","172.21.15.106:0/1323704418":"2024-01-27T00:46:44.670099+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:47:58.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.154+0000 7f5531676700 1 -- 172.21.15.46:0/3182620783 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5518067170 msgr2=0x7f5518069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:47:58.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.154+0000 7f5531676700 1 --2- 172.21.15.46:0/3182620783 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5518067170 0x7f5518069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.154+0000 7f5531676700 1 -- 172.21.15.46:0/3182620783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f552c0ad070 msgr2=0x7f552c1541d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:58.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.154+0000 7f5531676700 1 --2- 172.21.15.46:0/3182620783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f552c0ad070 0x7f552c1541d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f551c018000 tx=0x7f551c01b000 comp rx=0 tx=0).stop 2024-01-26T23:47:58.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.154+0000 7f5531676700 1 -- 172.21.15.46:0/3182620783 shutdown_connections 2024-01-26T23:47:58.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.154+0000 7f5531676700 1 --2- 172.21.15.46:0/3182620783 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7f5518067170 0x7f5518069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.154+0000 7f5531676700 1 --2- 172.21.15.46:0/3182620783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f552c0a98d0 0x7f552c16a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.154+0000 7f5531676700 1 --2- 172.21.15.46:0/3182620783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f552c0ad070 0x7f552c1541d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.154+0000 7f5531676700 1 --2- 172.21.15.46:0/3182620783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f552c0cbdf0 0x7f552c0aa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.154+0000 7f5531676700 1 -- 172.21.15.46:0/3182620783 >> 172.21.15.46:0/3182620783 conn(0x7f552c068400 msgr2=0x7f552c0ac7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:58.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.154+0000 7f5531676700 1 -- 172.21.15.46:0/3182620783 shutdown_connections 2024-01-26T23:47:58.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.154+0000 7f5531676700 1 -- 172.21.15.46:0/3182620783 wait complete. 2024-01-26T23:47:58.169 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-26T23:47:58.235 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:58.234+0000 7f3fc81491c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:47:58.239 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:58.238+0000 7fca3d6501c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:47:58.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.303+0000 7ff7a414c700 1 Processor -- start 2024-01-26T23:47:58.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.303+0000 7ff7a414c700 1 -- start start 2024-01-26T23:47:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.304+0000 7ff7a414c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff79c0b2160 0x7ff79c15a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.304+0000 7ff7a414c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff79c0b2530 0x7ff79c0abe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.304+0000 7ff7a414c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff79c0a9b90 con 0x7ff79c15da10 2024-01-26T23:47:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.304+0000 7ff7a414c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff79c0aa210 con 0x7ff79c0b2160 2024-01-26T23:47:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.304+0000 7ff7a414c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c15d580 con 0x7ff79c0b2530 2024-01-26T23:47:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.304+0000 7ff7a1ee8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff79c0b2160 0x7ff79c15a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.304+0000 7ff7a1ee8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff79c0b2160 0x7ff79c15a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41280/0 (socket says 172.21.15.46:41280) 2024-01-26T23:47:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.304+0000 7ff7a1ee8700 1 -- 172.21.15.46:0/3185527299 learned_addr learned my addr 172.21.15.46:0/3185527299 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.304+0000 7ff7a16e7700 1 --2- 172.21.15.46:0/3185527299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff79c0b2530 0x7ff79c0abe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.305+0000 7ff7a1ee8700 1 -- 172.21.15.46:0/3185527299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff79c0b2530 msgr2=0x7ff79c0abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:47:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.305+0000 7ff7a1ee8700 1 --2- 172.21.15.46:0/3185527299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff79c0b2530 0x7ff79c0abe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.305+0000 7ff7a1ee8700 1 -- 172.21.15.46:0/3185527299 >> v1:172.21.15.106:6789/0 conn(0x7ff79c15da10 legacy=0x7ff79c06a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.305+0000 7ff7a1ee8700 1 -- 172.21.15.46:0/3185527299 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c0ac390 con 0x7ff79c0b2160 2024-01-26T23:47:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.305+0000 7ff7a0ee6700 1 -- 172.21.15.46:0/3185527299 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670371059 0 0) 0x7ff79c0a9b90 con 0x7ff79c15da10 2024-01-26T23:47:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.305+0000 7ff7a1ee8700 1 --2- 172.21.15.46:0/3185527299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff79c0b2160 0x7ff79c15a7a0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7ff7980099b0 tx=0x7ff798009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ae0c7996b47bae server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.305+0000 7ff7a0ee6700 1 -- 172.21.15.46:0/3185527299 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff798009ea0 con 0x7ff79c0b2160 2024-01-26T23:47:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.305+0000 7ff7a0ee6700 1 -- 172.21.15.46:0/3185527299 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79800b860 con 0x7ff79c0b2160 2024-01-26T23:47:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.305+0000 7ff7a0ee6700 1 -- 172.21.15.46:0/3185527299 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79800bc20 con 0x7ff79c0b2160 2024-01-26T23:47:58.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.306+0000 7ff7a414c700 1 -- 172.21.15.46:0/3185527299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff79c0b2160 msgr2=0x7ff79c15a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:58.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.306+0000 7ff7a414c700 1 --2- 172.21.15.46:0/3185527299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff79c0b2160 0x7ff79c15a7a0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7ff7980099b0 tx=0x7ff798009250 comp rx=0 tx=0).stop 2024-01-26T23:47:58.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.306+0000 7ff7a414c700 1 -- 172.21.15.46:0/3185527299 shutdown_connections 2024-01-26T23:47:58.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.306+0000 7ff7a414c700 1 --2- 172.21.15.46:0/3185527299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff79c0b2160 0x7ff79c15a7a0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.306+0000 7ff7a414c700 1 --2- 172.21.15.46:0/3185527299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff79c0b2530 0x7ff79c0abe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.306+0000 7ff7a414c700 1 -- 172.21.15.46:0/3185527299 >> 172.21.15.46:0/3185527299 conn(0x7ff79c0ccbc0 msgr2=0x7ff79c0ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:58.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.306+0000 7ff7a414c700 1 -- 172.21.15.46:0/3185527299 shutdown_connections 2024-01-26T23:47:58.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.306+0000 7ff7a414c700 1 -- 172.21.15.46:0/3185527299 wait complete. 2024-01-26T23:47:58.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.307+0000 7ff7a414c700 1 Processor -- start 2024-01-26T23:47:58.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.307+0000 7ff7a414c700 1 -- start start 2024-01-26T23:47:58.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.308+0000 7ff7a414c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff79c0b2160 0x7ff79c1478c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:58.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.308+0000 7ff7a414c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff79c0b2530 0x7ff79c0aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:58.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.308+0000 7ff7a414c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff79c15da10 0x7ff79c0b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:58.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.308+0000 7ff7a414c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c149270 con 0x7ff79c0b2160 2024-01-26T23:47:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.308+0000 7ff7a414c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c147e00 con 0x7ff79c15da10 2024-01-26T23:47:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.308+0000 7ff7a414c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c0ad470 con 0x7ff79c0b2530 2024-01-26T23:47:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.308+0000 7ff7a1ee8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff79c0b2160 0x7ff79c1478c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.308+0000 7ff7a16e7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff79c0b2530 0x7ff79c0aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.308+0000 7ff7a1ee8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff79c0b2160 0x7ff79c1478c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41288/0 (socket says 172.21.15.46:41288) 2024-01-26T23:47:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.308+0000 7ff7a1ee8700 1 -- 172.21.15.46:0/3839462643 learned_addr learned my addr 172.21.15.46:0/3839462643 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.308+0000 7ff7a26e9700 1 --2- 172.21.15.46:0/3839462643 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff79c15da10 0x7ff79c0b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.308+0000 7ff7a16e7700 1 -- 172.21.15.46:0/3839462643 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff79c15da10 msgr2=0x7ff79c0b0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.308+0000 7ff7a16e7700 1 --2- 172.21.15.46:0/3839462643 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff79c15da10 0x7ff79c0b0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.309+0000 7ff7a16e7700 1 -- 172.21.15.46:0/3839462643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff79c0b2160 msgr2=0x7ff79c1478c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.309+0000 7ff7a16e7700 1 --2- 172.21.15.46:0/3839462643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff79c0b2160 0x7ff79c1478c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.309+0000 7ff7a16e7700 1 -- 172.21.15.46:0/3839462643 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7980094f0 con 0x7ff79c0b2530 2024-01-26T23:47:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.309+0000 7ff7a16e7700 1 --2- 172.21.15.46:0/3839462643 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff79c0b2530 0x7ff79c0aeaf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff78c00ea50 tx=0x7ff78c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.309+0000 7ff791ffb700 1 -- 172.21.15.46:0/3839462643 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c00ee90 con 0x7ff79c0b2530 2024-01-26T23:47:58.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.309+0000 7ff791ffb700 1 -- 172.21.15.46:0/3839462643 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78c00c880 con 0x7ff79c0b2530 2024-01-26T23:47:58.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.309+0000 7ff7a414c700 1 -- 172.21.15.46:0/3839462643 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79c0ad750 con 0x7ff79c0b2530 2024-01-26T23:47:58.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.309+0000 7ff791ffb700 1 -- 172.21.15.46:0/3839462643 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c00cc20 con 0x7ff79c0b2530 2024-01-26T23:47:58.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.309+0000 7ff7a414c700 1 -- 172.21.15.46:0/3839462643 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79c148f00 con 0x7ff79c0b2530 2024-01-26T23:47:58.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.310+0000 7ff7a414c700 1 -- 172.21.15.46:0/3839462643 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff79c149420 con 0x7ff79c0b2530 2024-01-26T23:47:58.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.311+0000 7ff791ffb700 1 -- 172.21.15.46:0/3839462643 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 51) v1 ==== 84958+0+0 (secure 0 0 0) 0x7ff78c013470 con 0x7ff79c0b2530 2024-01-26T23:47:58.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.311+0000 7ff791ffb700 1 --2- 172.21.15.46:0/3839462643 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff780067170 0x7ff780069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:58.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.311+0000 7ff7a1ee8700 1 -- 172.21.15.46:0/3839462643 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff780067170 msgr2=0x7ff780069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:58.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.311+0000 7ff791ffb700 1 -- 172.21.15.46:0/3839462643 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6791+0+0 (secure 0 0 0) 0x7ff78c084ec0 con 0x7ff79c0b2530 2024-01-26T23:47:58.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.311+0000 7ff7a1ee8700 1 --2- 172.21.15.46:0/3839462643 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff780067170 0x7ff780069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:47:58.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.315+0000 7ff791ffb700 1 -- 172.21.15.46:0/3839462643 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff79c149420 con 0x7ff79c0b2530 2024-01-26T23:47:58.462 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:58.461+0000 7fca3d6501c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:47:58.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.512+0000 7ff7a1ee8700 1 -- 172.21.15.46:0/3839462643 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff780067170 msgr2=0x7ff780069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6832/43388 2024-01-26T23:47:58.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.512+0000 7ff7a1ee8700 1 --2- 172.21.15.46:0/3839462643 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff780067170 0x7ff780069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:47:58.535 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:58.534+0000 7fca3d6501c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:47:58.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.569+0000 7ff7a414c700 1 -- 172.21.15.46:0/3839462643 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff79c1493e0 con 0x7ff79c0b2530 2024-01-26T23:47:58.572 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-26T23:47:58.571+0000 7f33ffe86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-26T23:47:58.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.575+0000 7ff791ffb700 1 -- 172.21.15.46:0/3839462643 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v55) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff78c055260 con 0x7ff79c0b2530 2024-01-26T23:47:58.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.577+0000 7ff7a414c700 1 -- 172.21.15.46:0/3839462643 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff780067170 msgr2=0x7ff780069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:47:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.577+0000 7ff7a414c700 1 --2- 172.21.15.46:0/3839462643 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff780067170 0x7ff780069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.578+0000 7ff7a414c700 1 -- 172.21.15.46:0/3839462643 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff79c0b2530 msgr2=0x7ff79c0aeaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.578+0000 7ff7a414c700 1 --2- 172.21.15.46:0/3839462643 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff79c0b2530 0x7ff79c0aeaf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff78c00ea50 tx=0x7ff78c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.578+0000 7ff7a414c700 1 -- 172.21.15.46:0/3839462643 shutdown_connections 2024-01-26T23:47:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.578+0000 7ff7a414c700 1 --2- 172.21.15.46:0/3839462643 >> [v2:172.21.15.106:6832/43388,v1:172.21.15.106:6833/43388] conn(0x7ff780067170 0x7ff780069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.578+0000 7ff7a414c700 1 --2- 172.21.15.46:0/3839462643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff79c0b2160 0x7ff79c1478c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.578+0000 7ff7a414c700 1 --2- 172.21.15.46:0/3839462643 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff79c15da10 0x7ff79c0b0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.578+0000 7ff7a414c700 1 --2- 172.21.15.46:0/3839462643 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff79c0b2530 0x7ff79c0aeaf0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.578+0000 7ff7a414c700 1 -- 172.21.15.46:0/3839462643 >> 172.21.15.46:0/3839462643 conn(0x7ff79c0ccbc0 msgr2=0x7ff79c15aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:58.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.578+0000 7ff7a414c700 1 -- 172.21.15.46:0/3839462643 shutdown_connections 2024-01-26T23:47:58.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.578+0000 7ff7a414c700 1 -- 172.21.15.46:0/3839462643 wait complete. 2024-01-26T23:47:58.581 INFO:tasks.ceph.mds.d.smithi106.stderr: -6211> 2024-01-26T23:46:33.706+0000 7f9fb9a8e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-26T23:47:58.581 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-26T23:47:58.593 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-26T23:47:58.605 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-01-26T23:47:58.696 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:58.696+0000 7fca3d6501c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:47:58.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.742+0000 7f461deb8700 1 Processor -- start 2024-01-26T23:47:58.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.742+0000 7f461deb8700 1 -- start start 2024-01-26T23:47:58.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.743+0000 7f461deb8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f46180cb640 0x7f46180c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:58.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.743+0000 7f461deb8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46180ccc00 con 0x7f46180c8da0 2024-01-26T23:47:58.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.743+0000 7f461deb8700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46180cc310 con 0x7f4618141200 2024-01-26T23:47:58.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.743+0000 7f461deb8700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46180cbbf0 con 0x7f46180cb640 2024-01-26T23:47:58.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.743+0000 7f4617fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f46180cb640 0x7f46180c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:58.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.743+0000 7f4616ffd700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f4618141200 0x7f46180caf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:41216/0 (socket says 172.21.15.46:41216) 2024-01-26T23:47:58.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.743+0000 7f4616ffd700 1 -- 172.21.15.46:0/768502509 learned_addr learned my addr 172.21.15.46:0/768502509 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:47:58.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.743+0000 7f46177fe700 1 --1- 172.21.15.46:0/768502509 >> v1:172.21.15.46:6789/0 conn(0x7f46180c8da0 0x7f4618140af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59588/0 (socket says 172.21.15.46:59588) 2024-01-26T23:47:58.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.743+0000 7f46167fc700 1 -- 172.21.15.46:0/768502509 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209360372 0 0) 0x7f46180cc310 con 0x7f4618141200 2024-01-26T23:47:58.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.743+0000 7f46167fc700 1 -- 172.21.15.46:0/768502509 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4600003570 con 0x7f4618141200 2024-01-26T23:47:58.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.743+0000 7f4617fff700 1 -- 172.21.15.46:0/768502509 >> v1:172.21.15.106:6789/0 conn(0x7f4618141200 legacy=0x7f46180caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:58.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.743+0000 7f46167fc700 1 -- 172.21.15.46:0/768502509 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555979716 0 0) 0x7f46180ccc00 con 0x7f46180c8da0 2024-01-26T23:47:58.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.743+0000 7f4617fff700 1 -- 172.21.15.46:0/768502509 >> v1:172.21.15.46:6789/0 conn(0x7f46180c8da0 legacy=0x7f4618140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:58.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.743+0000 7f4617fff700 1 -- 172.21.15.46:0/768502509 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46180cc5b0 con 0x7f46180cb640 2024-01-26T23:47:58.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.744+0000 7f4617fff700 1 --2- 172.21.15.46:0/768502509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46180cb640 0x7f46180c9170 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f4610009e60 tx=0x7f4610009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4431724c03c93a84 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:58.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.744+0000 7f46167fc700 1 -- 172.21.15.46:0/768502509 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461000e070 con 0x7f46180cb640 2024-01-26T23:47:58.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.744+0000 7f46167fc700 1 -- 172.21.15.46:0/768502509 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461000bb60 con 0x7f46180cb640 2024-01-26T23:47:58.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.744+0000 7f46167fc700 1 -- 172.21.15.46:0/768502509 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4610012430 con 0x7f46180cb640 2024-01-26T23:47:58.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.744+0000 7f461deb8700 1 -- 172.21.15.46:0/768502509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46180cb640 msgr2=0x7f46180c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:58.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.744+0000 7f461deb8700 1 --2- 172.21.15.46:0/768502509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46180cb640 0x7f46180c9170 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f4610009e60 tx=0x7f4610009780 comp rx=0 tx=0).stop 2024-01-26T23:47:58.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.744+0000 7f461deb8700 1 -- 172.21.15.46:0/768502509 shutdown_connections 2024-01-26T23:47:58.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.744+0000 7f461deb8700 1 --2- 172.21.15.46:0/768502509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46180cb640 0x7f46180c9170 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.744+0000 7f461deb8700 1 -- 172.21.15.46:0/768502509 >> 172.21.15.46:0/768502509 conn(0x7f461806bd90 msgr2=0x7f461806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:58.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.745+0000 7f461deb8700 1 -- 172.21.15.46:0/768502509 shutdown_connections 2024-01-26T23:47:58.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.745+0000 7f461deb8700 1 -- 172.21.15.46:0/768502509 wait complete. 2024-01-26T23:47:58.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.746+0000 7f461deb8700 1 Processor -- start 2024-01-26T23:47:58.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.746+0000 7f461deb8700 1 -- start start 2024-01-26T23:47:58.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.747+0000 7f461deb8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f46180c8da0 0x7f4618157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:58.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.747+0000 7f461deb8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46180cb640 0x7f46180c4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:58.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.747+0000 7f461deb8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4618141200 0x7f461813ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:58.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f461deb8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4618069ae0 con 0x7f4618141200 2024-01-26T23:47:58.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f461deb8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4618068ed0 con 0x7f46180cb640 2024-01-26T23:47:58.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f461deb8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46180691b0 con 0x7f46180c8da0 2024-01-26T23:47:58.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f4617fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4618141200 0x7f461813ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:58.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f46177fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f46180c8da0 0x7f4618157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:58.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f4616ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46180cb640 0x7f46180c4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:58.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f46177fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f46180c8da0 0x7f4618157c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:42692/0 (socket says 172.21.15.46:42692) 2024-01-26T23:47:58.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f4617fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4618141200 0x7f461813ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41296/0 (socket says 172.21.15.46:41296) 2024-01-26T23:47:58.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f46177fe700 1 -- 172.21.15.46:0/209953541 learned_addr learned my addr 172.21.15.46:0/209953541 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:58.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f46177fe700 1 -- 172.21.15.46:0/209953541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46180cb640 msgr2=0x7f46180c4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:58.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f46177fe700 1 --2- 172.21.15.46:0/209953541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46180cb640 0x7f46180c4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f46177fe700 1 -- 172.21.15.46:0/209953541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4618141200 msgr2=0x7f461813ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:58.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f46177fe700 1 --2- 172.21.15.46:0/209953541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4618141200 0x7f461813ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:58.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f46177fe700 1 -- 172.21.15.46:0/209953541 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4610009a20 con 0x7f46180c8da0 2024-01-26T23:47:58.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f4617fff700 1 --2- 172.21.15.46:0/209953541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4618141200 0x7f461813ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:47:58.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f4616ffd700 1 --2- 172.21.15.46:0/209953541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46180cb640 0x7f46180c4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:47:58.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f46177fe700 1 --2- 172.21.15.46:0/209953541 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f46180c8da0 0x7f4618157c90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f46080011e0 tx=0x7f4608007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:58.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.748+0000 7f45ff7fe700 1 -- 172.21.15.46:0/209953541 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4608005a70 con 0x7f46180c8da0 2024-01-26T23:47:58.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.749+0000 7f45ff7fe700 1 -- 172.21.15.46:0/209953541 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4608005bd0 con 0x7f46180c8da0 2024-01-26T23:47:58.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.749+0000 7f461deb8700 1 -- 172.21.15.46:0/209953541 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4618069490 con 0x7f46180c8da0 2024-01-26T23:47:58.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.749+0000 7f45ff7fe700 1 -- 172.21.15.46:0/209953541 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46080070d0 con 0x7f46180c8da0 2024-01-26T23:47:58.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.749+0000 7f461deb8700 1 -- 172.21.15.46:0/209953541 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46181581d0 con 0x7f46180c8da0 2024-01-26T23:47:58.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.750+0000 7f461deb8700 1 -- 172.21.15.46:0/209953541 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f461804c710 con 0x7f46180c8da0 2024-01-26T23:47:58.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.750+0000 7f45ff7fe700 1 -- 172.21.15.46:0/209953541 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 52) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f4608007480 con 0x7f46180c8da0 2024-01-26T23:47:58.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.751+0000 7f45ff7fe700 1 -- 172.21.15.46:0/209953541 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 7221+0+0 (secure 0 0 0) 0x7f4608083b50 con 0x7f46180c8da0 2024-01-26T23:47:58.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:58.755+0000 7f45ff7fe700 1 -- 172.21.15.46:0/209953541 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4608050b30 con 0x7f46180c8da0 2024-01-26T23:47:58.766 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:58.765+0000 7fc9b96f9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:47:58.766 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:58.765+0000 7fc9b96f9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:47:58.766 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:58.765+0000 7fc9b96f9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:47:58.766 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:58.765+0000 7fc9b96f9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:47:58.766 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:58.765+0000 7fc9b96f9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:47:58.767 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:58.766+0000 7fc9bbefe700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:47:58.767 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:58.766+0000 7fc9bbefe700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:47:58.767 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:58.766+0000 7fc9bbefe700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:47:58.767 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:58.766+0000 7fc9bbefe700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:47:58.767 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:47:58.766+0000 7fc9bbefe700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:47:59.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.013+0000 7f461deb8700 1 -- 172.21.15.46:0/209953541 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f46180cc890 con 0x7f46180c8da0 2024-01-26T23:47:59.065 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:59.064+0000 7f3fc81491c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:47:59.317 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:59.315+0000 7f3fc81491c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:47:59.402 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:59.400+0000 7f3fc81491c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:47:59.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.576+0000 7f45ff7fe700 1 -- 172.21.15.46:0/209953541 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v56) v1 ==== 104+0+0 (secure 0 0 0) 0x7f46080170a0 con 0x7f46180c8da0 2024-01-26T23:47:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.578+0000 7f461deb8700 1 -- 172.21.15.46:0/209953541 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f46180c8da0 msgr2=0x7f4618157c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.578+0000 7f461deb8700 1 --2- 172.21.15.46:0/209953541 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f46180c8da0 0x7f4618157c90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f46080011e0 tx=0x7f4608007e10 comp rx=0 tx=0).stop 2024-01-26T23:47:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.578+0000 7f461deb8700 1 -- 172.21.15.46:0/209953541 shutdown_connections 2024-01-26T23:47:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.578+0000 7f461deb8700 1 --2- 172.21.15.46:0/209953541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4618141200 0x7f461813ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.578+0000 7f461deb8700 1 --2- 172.21.15.46:0/209953541 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f46180c8da0 0x7f4618157c90 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.578+0000 7f461deb8700 1 --2- 172.21.15.46:0/209953541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46180cb640 0x7f46180c4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.578+0000 7f461deb8700 1 -- 172.21.15.46:0/209953541 >> 172.21.15.46:0/209953541 conn(0x7f461806bd90 msgr2=0x7f461813f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.578+0000 7f461deb8700 1 -- 172.21.15.46:0/209953541 shutdown_connections 2024-01-26T23:47:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.579+0000 7f461deb8700 1 -- 172.21.15.46:0/209953541 wait complete. 2024-01-26T23:47:59.583 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:47:59.582+0000 7f3fc81491c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:47:59.593 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-26T23:47:59.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.734+0000 7f569a15f700 1 Processor -- start 2024-01-26T23:47:59.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.734+0000 7f569a15f700 1 -- start start 2024-01-26T23:47:59.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.735+0000 7f569a15f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56940b26e0 0x7f56940b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:59.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.735+0000 7f569a15f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f569415bd50 0x7f56941571f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:59.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.735+0000 7f569a15f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5694157b80 con 0x7f5694157730 2024-01-26T23:47:59.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.735+0000 7f569a15f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56940cd4f0 con 0x7f569415bd50 2024-01-26T23:47:59.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.735+0000 7f569a15f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56940cd7d0 con 0x7f56940b26e0 2024-01-26T23:47:59.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.735+0000 7f5692ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f569415bd50 0x7f56941571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:59.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.735+0000 7f5693fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5694157730 0x7f5694067930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59590/0 (socket says 172.21.15.46:59590) 2024-01-26T23:47:59.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.735+0000 7f5693fff700 1 -- 172.21.15.46:0/587037822 learned_addr learned my addr 172.21.15.46:0/587037822 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:47:59.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.735+0000 7f56937fe700 1 --2- 172.21.15.46:0/587037822 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56940b26e0 0x7f56940b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:59.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.736+0000 7f5692ffd700 1 -- 172.21.15.46:0/587037822 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56940b26e0 msgr2=0x7f56940b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:47:59.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.736+0000 7f5692ffd700 1 --2- 172.21.15.46:0/587037822 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56940b26e0 0x7f56940b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:59.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.736+0000 7f5692ffd700 1 -- 172.21.15.46:0/587037822 >> v1:172.21.15.46:6789/0 conn(0x7f5694157730 legacy=0x7f5694067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:59.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.736+0000 7f5692ffd700 1 -- 172.21.15.46:0/587037822 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56940cdab0 con 0x7f569415bd50 2024-01-26T23:47:59.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.736+0000 7f56927fc700 1 -- 172.21.15.46:0/587037822 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219193909 0 0) 0x7f5694157b80 con 0x7f5694157730 2024-01-26T23:47:59.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.736+0000 7f5692ffd700 1 --2- 172.21.15.46:0/587037822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f569415bd50 0x7f56941571f0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f56880098e0 tx=0x7f56880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=131bdfd7c99a91b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:59.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.736+0000 7f56927fc700 1 -- 172.21.15.46:0/587037822 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568800b640 con 0x7f569415bd50 2024-01-26T23:47:59.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.736+0000 7f56927fc700 1 -- 172.21.15.46:0/587037822 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568800b7a0 con 0x7f569415bd50 2024-01-26T23:47:59.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.736+0000 7f56927fc700 1 -- 172.21.15.46:0/587037822 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568800bb40 con 0x7f569415bd50 2024-01-26T23:47:59.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.737+0000 7f569a15f700 1 -- 172.21.15.46:0/587037822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f569415bd50 msgr2=0x7f56941571f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:59.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.737+0000 7f569a15f700 1 --2- 172.21.15.46:0/587037822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f569415bd50 0x7f56941571f0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f56880098e0 tx=0x7f56880091a0 comp rx=0 tx=0).stop 2024-01-26T23:47:59.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.737+0000 7f569a15f700 1 -- 172.21.15.46:0/587037822 shutdown_connections 2024-01-26T23:47:59.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.737+0000 7f569a15f700 1 --2- 172.21.15.46:0/587037822 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56940b26e0 0x7f56940b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:59.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.737+0000 7f569a15f700 1 --2- 172.21.15.46:0/587037822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f569415bd50 0x7f56941571f0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:59.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.737+0000 7f569a15f700 1 -- 172.21.15.46:0/587037822 >> 172.21.15.46:0/587037822 conn(0x7f5694069510 msgr2=0x7f56940aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:47:59.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.737+0000 7f569a15f700 1 -- 172.21.15.46:0/587037822 shutdown_connections 2024-01-26T23:47:59.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.737+0000 7f569a15f700 1 -- 172.21.15.46:0/587037822 wait complete. 2024-01-26T23:47:59.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.739+0000 7f569a15f700 1 Processor -- start 2024-01-26T23:47:59.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.739+0000 7f569a15f700 1 -- start start 2024-01-26T23:47:59.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.740+0000 7f569a15f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56940b26e0 0x7f5694145f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:59.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.740+0000 7f569a15f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5694157730 0x7f5694157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:59.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.740+0000 7f569a15f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f569415bd50 0x7f56940ab990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:59.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.740+0000 7f569a15f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56940aa8b0 con 0x7f56940b26e0 2024-01-26T23:47:59.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.740+0000 7f569a15f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56941518a0 con 0x7f5694157730 2024-01-26T23:47:59.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.740+0000 7f569a15f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5694146440 con 0x7f569415bd50 2024-01-26T23:47:59.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.740+0000 7f56937fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56940b26e0 0x7f5694145f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:59.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.740+0000 7f56937fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56940b26e0 0x7f5694145f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41318/0 (socket says 172.21.15.46:41318) 2024-01-26T23:47:59.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.740+0000 7f56937fe700 1 -- 172.21.15.46:0/3935333935 learned_addr learned my addr 172.21.15.46:0/3935333935 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:47:59.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.740+0000 7f5693fff700 1 --2- 172.21.15.46:0/3935333935 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f569415bd50 0x7f56940ab990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:59.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.740+0000 7f5692ffd700 1 --2- 172.21.15.46:0/3935333935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5694157730 0x7f5694157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:59.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.741+0000 7f56937fe700 1 -- 172.21.15.46:0/3935333935 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f569415bd50 msgr2=0x7f56940ab990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:59.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.741+0000 7f56937fe700 1 --2- 172.21.15.46:0/3935333935 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f569415bd50 0x7f56940ab990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:59.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.741+0000 7f56937fe700 1 -- 172.21.15.46:0/3935333935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5694157730 msgr2=0x7f5694157db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:47:59.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.741+0000 7f56937fe700 1 --2- 172.21.15.46:0/3935333935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5694157730 0x7f5694157db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:47:59.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.741+0000 7f56937fe700 1 -- 172.21.15.46:0/3935333935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5688009420 con 0x7f56940b26e0 2024-01-26T23:47:59.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.741+0000 7f56937fe700 1 --2- 172.21.15.46:0/3935333935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56940b26e0 0x7f5694145f00 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f568400e930 tx=0x7f56840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:59.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.741+0000 7f567b7fe700 1 -- 172.21.15.46:0/3935333935 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684013680 con 0x7f56940b26e0 2024-01-26T23:47:59.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.741+0000 7f567b7fe700 1 -- 172.21.15.46:0/3935333935 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56840137e0 con 0x7f56940b26e0 2024-01-26T23:47:59.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.741+0000 7f569a15f700 1 -- 172.21.15.46:0/3935333935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f569415d090 con 0x7f56940b26e0 2024-01-26T23:47:59.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.741+0000 7f567b7fe700 1 -- 172.21.15.46:0/3935333935 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684013b80 con 0x7f56940b26e0 2024-01-26T23:47:59.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.741+0000 7f569a15f700 1 -- 172.21.15.46:0/3935333935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56940ab470 con 0x7f56940b26e0 2024-01-26T23:47:59.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.742+0000 7f569a15f700 1 -- 172.21.15.46:0/3935333935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f569404c710 con 0x7f56940b26e0 2024-01-26T23:47:59.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.743+0000 7f567b7fe700 1 -- 172.21.15.46:0/3935333935 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 53) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f568400c3e0 con 0x7f56940b26e0 2024-01-26T23:47:59.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.743+0000 7f567b7fe700 1 --2- 172.21.15.46:0/3935333935 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5680066d80 0x7f5680069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:47:59.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.743+0000 7f567b7fe700 1 -- 172.21.15.46:0/3935333935 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 7221+0+0 (secure 0 0 0) 0x7f56840848c0 con 0x7f56940b26e0 2024-01-26T23:47:59.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.743+0000 7f5692ffd700 1 --2- 172.21.15.46:0/3935333935 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5680066d80 0x7f5680069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:47:59.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.744+0000 7f5692ffd700 1 --2- 172.21.15.46:0/3935333935 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5680066d80 0x7f5680069230 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f56880093f0 tx=0x7f5688017000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:47:59.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.748+0000 7f567b7fe700 1 -- 172.21.15.46:0/3935333935 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f56840517f0 con 0x7f56940b26e0 2024-01-26T23:47:59.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:47:59.955+0000 7f569a15f700 1 -- 172.21.15.46:0/3935333935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f56940c9a80 con 0x7f56940b26e0 2024-01-26T23:48:00.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.750+0000 7f567b7fe700 1 -- 172.21.15.46:0/3935333935 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v120) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5684054e10 con 0x7f56940b26e0 2024-01-26T23:48:00.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.754+0000 7f569a15f700 1 -- 172.21.15.46:0/3935333935 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5680066d80 msgr2=0x7f5680069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:00.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.754+0000 7f569a15f700 1 --2- 172.21.15.46:0/3935333935 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5680066d80 0x7f5680069230 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f56880093f0 tx=0x7f5688017000 comp rx=0 tx=0).stop 2024-01-26T23:48:00.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.754+0000 7f569a15f700 1 -- 172.21.15.46:0/3935333935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56940b26e0 msgr2=0x7f5694145f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:00.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.754+0000 7f569a15f700 1 --2- 172.21.15.46:0/3935333935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56940b26e0 0x7f5694145f00 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f568400e930 tx=0x7f56840091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:00.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.754+0000 7f569a15f700 1 -- 172.21.15.46:0/3935333935 shutdown_connections 2024-01-26T23:48:00.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.754+0000 7f569a15f700 1 --2- 172.21.15.46:0/3935333935 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5680066d80 0x7f5680069230 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:00.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.754+0000 7f569a15f700 1 --2- 172.21.15.46:0/3935333935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56940b26e0 0x7f5694145f00 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:00.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.754+0000 7f569a15f700 1 --2- 172.21.15.46:0/3935333935 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f569415bd50 0x7f56940ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:00.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.754+0000 7f569a15f700 1 --2- 172.21.15.46:0/3935333935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5694157730 0x7f5694157db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:00.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.754+0000 7f569a15f700 1 -- 172.21.15.46:0/3935333935 >> 172.21.15.46:0/3935333935 conn(0x7f5694069510 msgr2=0x7f5694070b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:00.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.754+0000 7f569a15f700 1 -- 172.21.15.46:0/3935333935 shutdown_connections 2024-01-26T23:48:00.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.754+0000 7f569a15f700 1 -- 172.21.15.46:0/3935333935 wait complete. 2024-01-26T23:48:00.757 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-01-26T23:48:00.771 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-26T23:48:00.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.912+0000 7f31dad51700 1 Processor -- start 2024-01-26T23:48:00.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.913+0000 7f31dad51700 1 -- start start 2024-01-26T23:48:00.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.913+0000 7f31dad51700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31d4159b20 0x7f31d40ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.913+0000 7f31dad51700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d40691c0 con 0x7f31d40ae340 2024-01-26T23:48:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.913+0000 7f31dad51700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d406a5e0 con 0x7f31d4159750 2024-01-26T23:48:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.913+0000 7f31dad51700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d415dfd0 con 0x7f31d4159b20 2024-01-26T23:48:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.914+0000 7f31d92ee700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f31d40ae340 0x7f31d415d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34356/0 (socket says 172.21.15.46:34356) 2024-01-26T23:48:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.914+0000 7f31d92ee700 1 -- 172.21.15.46:0/2782692930 learned_addr learned my addr 172.21.15.46:0/2782692930 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.914+0000 7f31d3fff700 1 --2- 172.21.15.46:0/2782692930 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31d4159b20 0x7f31d40ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.914+0000 7f31d37fe700 1 -- 172.21.15.46:0/2782692930 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017408780 0 0) 0x7f31d40691c0 con 0x7f31d40ae340 2024-01-26T23:48:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.914+0000 7f31d37fe700 1 -- 172.21.15.46:0/2782692930 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31bc003570 con 0x7f31d40ae340 2024-01-26T23:48:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.914+0000 7f31d37fe700 1 -- 172.21.15.46:0/2782692930 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476135141 0 0) 0x7f31d406a5e0 con 0x7f31d4159750 2024-01-26T23:48:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.914+0000 7f31d37fe700 1 -- 172.21.15.46:0/2782692930 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31d40691c0 con 0x7f31d4159750 2024-01-26T23:48:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.914+0000 7f31d3fff700 1 -- 172.21.15.46:0/2782692930 >> v1:172.21.15.106:6789/0 conn(0x7f31d4159750 legacy=0x7f31d4069650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.914+0000 7f31d3fff700 1 -- 172.21.15.46:0/2782692930 >> v1:172.21.15.46:6789/0 conn(0x7f31d40ae340 legacy=0x7f31d415d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.914+0000 7f31d3fff700 1 -- 172.21.15.46:0/2782692930 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d4068b70 con 0x7f31d4159b20 2024-01-26T23:48:00.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.914+0000 7f31d3fff700 1 --2- 172.21.15.46:0/2782692930 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31d4159b20 0x7f31d40ade00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009910 tx=0x7f31c80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2df733d348fae816 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:00.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.915+0000 7f31d37fe700 1 -- 172.21.15.46:0/2782692930 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c8011630 con 0x7f31d4159b20 2024-01-26T23:48:00.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.915+0000 7f31d37fe700 1 -- 172.21.15.46:0/2782692930 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c8011790 con 0x7f31d4159b20 2024-01-26T23:48:00.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.915+0000 7f31d37fe700 1 -- 172.21.15.46:0/2782692930 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c8011b30 con 0x7f31d4159b20 2024-01-26T23:48:00.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.915+0000 7f31dad51700 1 -- 172.21.15.46:0/2782692930 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31d4159b20 msgr2=0x7f31d40ade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:00.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.915+0000 7f31dad51700 1 --2- 172.21.15.46:0/2782692930 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31d4159b20 0x7f31d40ade00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009910 tx=0x7f31c80091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:00.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.915+0000 7f31dad51700 1 -- 172.21.15.46:0/2782692930 shutdown_connections 2024-01-26T23:48:00.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.915+0000 7f31dad51700 1 --2- 172.21.15.46:0/2782692930 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31d4159b20 0x7f31d40ade00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:00.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.915+0000 7f31dad51700 1 -- 172.21.15.46:0/2782692930 >> 172.21.15.46:0/2782692930 conn(0x7f31d40ce530 msgr2=0x7f31d40ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:00.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.915+0000 7f31dad51700 1 -- 172.21.15.46:0/2782692930 shutdown_connections 2024-01-26T23:48:00.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.915+0000 7f31dad51700 1 -- 172.21.15.46:0/2782692930 wait complete. 2024-01-26T23:48:00.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.916+0000 7f31dad51700 1 Processor -- start 2024-01-26T23:48:00.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.917+0000 7f31dad51700 1 -- start start 2024-01-26T23:48:00.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.917+0000 7f31dad51700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f31d40ae340 0x7f31d40b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:00.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.917+0000 7f31dad51700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31d4159750 0x7f31d4155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:00.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.917+0000 7f31dad51700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31d4159b20 0x7f31d415aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:00.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.917+0000 7f31dad51700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d415ba70 con 0x7f31d4159b20 2024-01-26T23:48:00.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.917+0000 7f31dad51700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d40c6e30 con 0x7f31d40ae340 2024-01-26T23:48:00.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.917+0000 7f31dad51700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d415b790 con 0x7f31d4159750 2024-01-26T23:48:00.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.918+0000 7f31d92ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31d4159b20 0x7f31d415aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:00.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.918+0000 7f31d92ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31d4159b20 0x7f31d415aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39486/0 (socket says 172.21.15.46:39486) 2024-01-26T23:48:00.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.918+0000 7f31d92ee700 1 -- 172.21.15.46:0/3550335475 learned_addr learned my addr 172.21.15.46:0/3550335475 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:00.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.918+0000 7f31d8aed700 1 --2- 172.21.15.46:0/3550335475 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f31d40ae340 0x7f31d40b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:00.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.918+0000 7f31d3fff700 1 --2- 172.21.15.46:0/3550335475 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31d4159750 0x7f31d4155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:00.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.918+0000 7f31d92ee700 1 -- 172.21.15.46:0/3550335475 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31d4159750 msgr2=0x7f31d4155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:00.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.918+0000 7f31d92ee700 1 --2- 172.21.15.46:0/3550335475 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31d4159750 0x7f31d4155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:00.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.918+0000 7f31d92ee700 1 -- 172.21.15.46:0/3550335475 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f31d40ae340 msgr2=0x7f31d40b22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:00.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.918+0000 7f31d92ee700 1 --2- 172.21.15.46:0/3550335475 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f31d40ae340 0x7f31d40b22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:00.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.918+0000 7f31d92ee700 1 -- 172.21.15.46:0/3550335475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c8009420 con 0x7f31d4159b20 2024-01-26T23:48:00.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.918+0000 7f31d92ee700 1 --2- 172.21.15.46:0/3550335475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31d4159b20 0x7f31d415aca0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00cd00 tx=0x7f31cc001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:00.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.918+0000 7f31d0ff9700 1 -- 172.21.15.46:0/3550335475 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc00ce00 con 0x7f31d4159b20 2024-01-26T23:48:00.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.918+0000 7f31d0ff9700 1 -- 172.21.15.46:0/3550335475 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31cc00aac0 con 0x7f31d4159b20 2024-01-26T23:48:00.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.918+0000 7f31dad51700 1 -- 172.21.15.46:0/3550335475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d40c70d0 con 0x7f31d4159b20 2024-01-26T23:48:00.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.918+0000 7f31d0ff9700 1 -- 172.21.15.46:0/3550335475 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc00ae80 con 0x7f31d4159b20 2024-01-26T23:48:00.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.918+0000 7f31dad51700 1 -- 172.21.15.46:0/3550335475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d40c7690 con 0x7f31d4159b20 2024-01-26T23:48:00.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.919+0000 7f31dad51700 1 -- 172.21.15.46:0/3550335475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d404c710 con 0x7f31d4159b20 2024-01-26T23:48:00.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.920+0000 7f31d0ff9700 1 -- 172.21.15.46:0/3550335475 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 53) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f31cc007300 con 0x7f31d4159b20 2024-01-26T23:48:00.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.920+0000 7f31d0ff9700 1 --2- 172.21.15.46:0/3550335475 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f31c0066d80 0x7f31c0069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:00.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.920+0000 7f31d0ff9700 1 -- 172.21.15.46:0/3550335475 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6783+0+0 (secure 0 0 0) 0x7f31cc084940 con 0x7f31d4159b20 2024-01-26T23:48:00.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.920+0000 7f31d8aed700 1 --2- 172.21.15.46:0/3550335475 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f31c0066d80 0x7f31c0069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:00.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.921+0000 7f31d8aed700 1 --2- 172.21.15.46:0/3550335475 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f31c0066d80 0x7f31c0069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f31c4007fd0 tx=0x7f31c4007d50 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:00.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:00.925+0000 7f31d0ff9700 1 -- 172.21.15.46:0/3550335475 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f31cc051ab0 con 0x7f31d4159b20 2024-01-26T23:48:01.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.134+0000 7f31dad51700 1 -- 172.21.15.46:0/3550335475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f31d40c7880 con 0x7f31d4159b20 2024-01-26T23:48:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.719+0000 7f31d0ff9700 1 -- 172.21.15.46:0/3550335475 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 54) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f31cc051e80 con 0x7f31d4159b20 2024-01-26T23:48:01.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.755+0000 7f31d0ff9700 1 -- 172.21.15.46:0/3550335475 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v121) v1 ==== 169+0+0 (secure 0 0 0) 0x7f31cc055540 con 0x7f31d4159b20 2024-01-26T23:48:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.759+0000 7f31dad51700 1 -- 172.21.15.46:0/3550335475 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f31c0066d80 msgr2=0x7f31c0069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.759+0000 7f31dad51700 1 --2- 172.21.15.46:0/3550335475 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f31c0066d80 0x7f31c0069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f31c4007fd0 tx=0x7f31c4007d50 comp rx=0 tx=0).stop 2024-01-26T23:48:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.759+0000 7f31dad51700 1 -- 172.21.15.46:0/3550335475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31d4159b20 msgr2=0x7f31d415aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.759+0000 7f31dad51700 1 --2- 172.21.15.46:0/3550335475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31d4159b20 0x7f31d415aca0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00cd00 tx=0x7f31cc001ac0 comp rx=0 tx=0).stop 2024-01-26T23:48:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.759+0000 7f31dad51700 1 -- 172.21.15.46:0/3550335475 shutdown_connections 2024-01-26T23:48:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.759+0000 7f31dad51700 1 --2- 172.21.15.46:0/3550335475 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f31c0066d80 0x7f31c0069230 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.759+0000 7f31dad51700 1 --2- 172.21.15.46:0/3550335475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31d4159b20 0x7f31d415aca0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.759+0000 7f31dad51700 1 --2- 172.21.15.46:0/3550335475 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f31d40ae340 0x7f31d40b22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.759+0000 7f31dad51700 1 --2- 172.21.15.46:0/3550335475 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f31d4159750 0x7f31d4155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.759+0000 7f31dad51700 1 -- 172.21.15.46:0/3550335475 >> 172.21.15.46:0/3550335475 conn(0x7f31d40ce530 msgr2=0x7f31d406bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.760+0000 7f31dad51700 1 -- 172.21.15.46:0/3550335475 shutdown_connections 2024-01-26T23:48:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.760+0000 7f31dad51700 1 -- 172.21.15.46:0/3550335475 wait complete. 2024-01-26T23:48:01.764 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-01-26T23:48:01.779 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-26T23:48:01.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.915+0000 7fc777ab9700 1 Processor -- start 2024-01-26T23:48:01.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.915+0000 7fc777ab9700 1 -- start start 2024-01-26T23:48:01.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.916+0000 7fc777ab9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc770140720 0x7fc770140af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:01.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.916+0000 7fc777ab9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc770141100 0x7fc7700b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:01.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.916+0000 7fc777ab9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7700b1b80 0x7fc77006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:01.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.916+0000 7fc777ab9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7700b3a20 con 0x7fc7700b1b80 2024-01-26T23:48:01.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.916+0000 7fc777ab9700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7700cf2b0 con 0x7fc770140720 2024-01-26T23:48:01.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.916+0000 7fc777ab9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7700b3050 con 0x7fc770141100 2024-01-26T23:48:01.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.916+0000 7fc776056700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7700b1b80 0x7fc77006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:01.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.916+0000 7fc776056700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7700b1b80 0x7fc77006d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39494/0 (socket says 172.21.15.46:39494) 2024-01-26T23:48:01.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.916+0000 7fc776056700 1 -- 172.21.15.46:0/287060262 learned_addr learned my addr 172.21.15.46:0/287060262 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:01.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.916+0000 7fc775855700 1 --2- 172.21.15.46:0/287060262 >> v2:172.21.15.106:3300/0 conn(0x7fc770140720 0x7fc770140af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:01.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.916+0000 7fc775054700 1 --2- 172.21.15.46:0/287060262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc770141100 0x7fc7700b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:01.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.917+0000 7fc776056700 1 -- 172.21.15.46:0/287060262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc770141100 msgr2=0x7fc7700b34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:01.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.917+0000 7fc776056700 1 --2- 172.21.15.46:0/287060262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc770141100 0x7fc7700b34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:01.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.917+0000 7fc776056700 1 -- 172.21.15.46:0/287060262 >> v2:172.21.15.106:3300/0 conn(0x7fc770140720 msgr2=0x7fc770140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:01.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.917+0000 7fc776056700 1 --2- 172.21.15.46:0/287060262 >> v2:172.21.15.106:3300/0 conn(0x7fc770140720 0x7fc770140af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:01.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.917+0000 7fc776056700 1 -- 172.21.15.46:0/287060262 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7700b2440 con 0x7fc7700b1b80 2024-01-26T23:48:01.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.917+0000 7fc775054700 1 --2- 172.21.15.46:0/287060262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc770141100 0x7fc7700b34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:01.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.917+0000 7fc776056700 1 --2- 172.21.15.46:0/287060262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7700b1b80 0x7fc77006d6a0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fc7700b3c00 tx=0x7fc768009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9f368368eb5c23 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:01.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.917+0000 7fc774853700 1 -- 172.21.15.46:0/287060262 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76800e070 con 0x7fc7700b1b80 2024-01-26T23:48:01.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.917+0000 7fc774853700 1 -- 172.21.15.46:0/287060262 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76800b990 con 0x7fc7700b1b80 2024-01-26T23:48:01.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.917+0000 7fc774853700 1 -- 172.21.15.46:0/287060262 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7680092e0 con 0x7fc7700b1b80 2024-01-26T23:48:01.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.917+0000 7fc777ab9700 1 -- 172.21.15.46:0/287060262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7700b1b80 msgr2=0x7fc77006d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:01.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.917+0000 7fc777ab9700 1 --2- 172.21.15.46:0/287060262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7700b1b80 0x7fc77006d6a0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fc7700b3c00 tx=0x7fc768009840 comp rx=0 tx=0).stop 2024-01-26T23:48:01.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.918+0000 7fc777ab9700 1 -- 172.21.15.46:0/287060262 shutdown_connections 2024-01-26T23:48:01.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.918+0000 7fc777ab9700 1 --2- 172.21.15.46:0/287060262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7700b1b80 0x7fc77006d6a0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:01.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.918+0000 7fc777ab9700 1 --2- 172.21.15.46:0/287060262 >> v2:172.21.15.106:3300/0 conn(0x7fc770140720 0x7fc770140af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:01.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.918+0000 7fc777ab9700 1 --2- 172.21.15.46:0/287060262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc770141100 0x7fc7700b34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:01.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.918+0000 7fc777ab9700 1 -- 172.21.15.46:0/287060262 >> 172.21.15.46:0/287060262 conn(0x7fc770067930 msgr2=0x7fc770067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:01.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.918+0000 7fc777ab9700 1 -- 172.21.15.46:0/287060262 shutdown_connections 2024-01-26T23:48:01.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.918+0000 7fc777ab9700 1 -- 172.21.15.46:0/287060262 wait complete. 2024-01-26T23:48:01.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.919+0000 7fc777ab9700 1 Processor -- start 2024-01-26T23:48:01.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.919+0000 7fc777ab9700 1 -- start start 2024-01-26T23:48:01.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.920+0000 7fc777ab9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7700b1b80 0x7fc7700ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:01.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.920+0000 7fc777ab9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770140720 0x7fc77013e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:01.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.920+0000 7fc777ab9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc770141100 0x7fc7700c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:01.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.920+0000 7fc777ab9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7700a6450 con 0x7fc770140720 2024-01-26T23:48:01.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.920+0000 7fc777ab9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc770159390 con 0x7fc770141100 2024-01-26T23:48:01.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.920+0000 7fc777ab9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7700c8d50 con 0x7fc7700b1b80 2024-01-26T23:48:01.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.920+0000 7fc775054700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770140720 0x7fc77013e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:01.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.921+0000 7fc775054700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770140720 0x7fc77013e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39500/0 (socket says 172.21.15.46:39500) 2024-01-26T23:48:01.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.921+0000 7fc775054700 1 -- 172.21.15.46:0/4203597790 learned_addr learned my addr 172.21.15.46:0/4203597790 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:01.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.921+0000 7fc776056700 1 --2- 172.21.15.46:0/4203597790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc770141100 0x7fc7700c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:01.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.921+0000 7fc775855700 1 --2- 172.21.15.46:0/4203597790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7700b1b80 0x7fc7700ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:01.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.921+0000 7fc775054700 1 -- 172.21.15.46:0/4203597790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7700b1b80 msgr2=0x7fc7700ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:01.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.921+0000 7fc775054700 1 --2- 172.21.15.46:0/4203597790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7700b1b80 0x7fc7700ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:01.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.921+0000 7fc775054700 1 -- 172.21.15.46:0/4203597790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc770141100 msgr2=0x7fc7700c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:01.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.921+0000 7fc775054700 1 --2- 172.21.15.46:0/4203597790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc770141100 0x7fc7700c5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:01.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.921+0000 7fc775054700 1 -- 172.21.15.46:0/4203597790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc768009ae0 con 0x7fc770140720 2024-01-26T23:48:01.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.921+0000 7fc775054700 1 --2- 172.21.15.46:0/4203597790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770140720 0x7fc77013e040 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc76400d860 tx=0x7fc764008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:01.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.921+0000 7fc761ffb700 1 -- 172.21.15.46:0/4203597790 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7640125f0 con 0x7fc770140720 2024-01-26T23:48:01.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.921+0000 7fc761ffb700 1 -- 172.21.15.46:0/4203597790 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc764012750 con 0x7fc770140720 2024-01-26T23:48:01.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.921+0000 7fc777ab9700 1 -- 172.21.15.46:0/4203597790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc77013e580 con 0x7fc770140720 2024-01-26T23:48:01.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.921+0000 7fc761ffb700 1 -- 172.21.15.46:0/4203597790 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc764012af0 con 0x7fc770140720 2024-01-26T23:48:01.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.922+0000 7fc777ab9700 1 -- 172.21.15.46:0/4203597790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7700a6170 con 0x7fc770140720 2024-01-26T23:48:01.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.923+0000 7fc777ab9700 1 -- 172.21.15.46:0/4203597790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7700c8ec0 con 0x7fc770140720 2024-01-26T23:48:01.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.923+0000 7fc761ffb700 1 -- 172.21.15.46:0/4203597790 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 54) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fc764013480 con 0x7fc770140720 2024-01-26T23:48:01.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.923+0000 7fc761ffb700 1 --2- 172.21.15.46:0/4203597790 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc754066ea0 0x7fc754069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:01.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.923+0000 7fc761ffb700 1 -- 172.21.15.46:0/4203597790 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6253+0+0 (secure 0 0 0) 0x7fc764084770 con 0x7fc770140720 2024-01-26T23:48:01.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.924+0000 7fc775855700 1 --2- 172.21.15.46:0/4203597790 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc754066ea0 0x7fc754069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:01.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.924+0000 7fc775855700 1 --2- 172.21.15.46:0/4203597790 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc754066ea0 0x7fc754069350 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc7701597c0 tx=0x7fc76c0091a0 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:01.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:01.928+0000 7fc761ffb700 1 -- 172.21.15.46:0/4203597790 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc7640518a0 con 0x7fc770140720 2024-01-26T23:48:02.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.133+0000 7fc777ab9700 1 -- 172.21.15.46:0/4203597790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc77004c710 con 0x7fc770140720 2024-01-26T23:48:02.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.136+0000 7fc761ffb700 1 -- 172.21.15.46:0/4203597790 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v37) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc764054ec0 con 0x7fc770140720 2024-01-26T23:48:02.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.139+0000 7fc777ab9700 1 -- 172.21.15.46:0/4203597790 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc754066ea0 msgr2=0x7fc754069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:02.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.139+0000 7fc777ab9700 1 --2- 172.21.15.46:0/4203597790 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc754066ea0 0x7fc754069350 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc7701597c0 tx=0x7fc76c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.139+0000 7fc777ab9700 1 -- 172.21.15.46:0/4203597790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770140720 msgr2=0x7fc77013e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:02.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.139+0000 7fc777ab9700 1 --2- 172.21.15.46:0/4203597790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770140720 0x7fc77013e040 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc76400d860 tx=0x7fc764008040 comp rx=0 tx=0).stop 2024-01-26T23:48:02.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.139+0000 7fc777ab9700 1 -- 172.21.15.46:0/4203597790 shutdown_connections 2024-01-26T23:48:02.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.139+0000 7fc777ab9700 1 --2- 172.21.15.46:0/4203597790 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc754066ea0 0x7fc754069350 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.139+0000 7fc777ab9700 1 --2- 172.21.15.46:0/4203597790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc770141100 0x7fc7700c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.139+0000 7fc777ab9700 1 --2- 172.21.15.46:0/4203597790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7700b1b80 0x7fc7700ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.139+0000 7fc777ab9700 1 --2- 172.21.15.46:0/4203597790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc770140720 0x7fc77013e040 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.139+0000 7fc777ab9700 1 -- 172.21.15.46:0/4203597790 >> 172.21.15.46:0/4203597790 conn(0x7fc770067930 msgr2=0x7fc77013fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:02.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.139+0000 7fc777ab9700 1 -- 172.21.15.46:0/4203597790 shutdown_connections 2024-01-26T23:48:02.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.139+0000 7fc777ab9700 1 -- 172.21.15.46:0/4203597790 wait complete. 2024-01-26T23:48:02.153 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-26T23:48:02.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.289+0000 7f1aee873700 1 Processor -- start 2024-01-26T23:48:02.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.290+0000 7f1aee873700 1 -- start start 2024-01-26T23:48:02.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.291+0000 7f1aee873700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ae8159800 0x7f1ae8159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:02.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.291+0000 7f1aee873700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae806a910 con 0x7f1ae80cba30 2024-01-26T23:48:02.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.291+0000 7f1aee873700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae806abf0 con 0x7f1ae80a98e0 2024-01-26T23:48:02.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.291+0000 7f1aee873700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae806aed0 con 0x7f1ae8159800 2024-01-26T23:48:02.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.291+0000 7f1ae7fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ae8159800 0x7f1ae8159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:02.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.291+0000 7f1ae77fe700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f1ae80a98e0 0x7f1ae80cb320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:60976/0 (socket says 172.21.15.46:60976) 2024-01-26T23:48:02.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.291+0000 7f1ae77fe700 1 -- 172.21.15.46:0/1276002405 learned_addr learned my addr 172.21.15.46:0/1276002405 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:02.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.292+0000 7f1ae7fff700 1 -- 172.21.15.46:0/1276002405 >> v1:172.21.15.106:6789/0 conn(0x7f1ae80a98e0 legacy=0x7f1ae80cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:02.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.292+0000 7f1ae7fff700 1 -- 172.21.15.46:0/1276002405 >> v1:172.21.15.46:6789/0 conn(0x7f1ae80cba30 legacy=0x7f1ae806e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:02.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.292+0000 7f1ae7fff700 1 -- 172.21.15.46:0/1276002405 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae806b1b0 con 0x7f1ae8159800 2024-01-26T23:48:02.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.292+0000 7f1ae7fff700 1 --2- 172.21.15.46:0/1276002405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ae8159800 0x7f1ae8159bd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8009910 tx=0x7f1ad80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=247f07160d58f718 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:02.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.293+0000 7f1ae6ffd700 1 -- 172.21.15.46:0/1276002405 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad800b860 con 0x7f1ae8159800 2024-01-26T23:48:02.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.293+0000 7f1ae6ffd700 1 -- 172.21.15.46:0/1276002405 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad800b9c0 con 0x7f1ae8159800 2024-01-26T23:48:02.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.293+0000 7f1ae6ffd700 1 -- 172.21.15.46:0/1276002405 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad800bd60 con 0x7f1ae8159800 2024-01-26T23:48:02.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.294+0000 7f1aee873700 1 -- 172.21.15.46:0/1276002405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ae8159800 msgr2=0x7f1ae8159bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:02.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.294+0000 7f1aee873700 1 --2- 172.21.15.46:0/1276002405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ae8159800 0x7f1ae8159bd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8009910 tx=0x7f1ad80091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.294+0000 7f1aee873700 1 -- 172.21.15.46:0/1276002405 shutdown_connections 2024-01-26T23:48:02.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.294+0000 7f1aee873700 1 --2- 172.21.15.46:0/1276002405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ae8159800 0x7f1ae8159bd0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.294+0000 7f1aee873700 1 -- 172.21.15.46:0/1276002405 >> 172.21.15.46:0/1276002405 conn(0x7f1ae8067930 msgr2=0x7f1ae8067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:02.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.294+0000 7f1aee873700 1 -- 172.21.15.46:0/1276002405 shutdown_connections 2024-01-26T23:48:02.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.295+0000 7f1aee873700 1 -- 172.21.15.46:0/1276002405 wait complete. 2024-01-26T23:48:02.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.297+0000 7f1aee873700 1 Processor -- start 2024-01-26T23:48:02.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.298+0000 7f1aee873700 1 -- start start 2024-01-26T23:48:02.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.299+0000 7f1aee873700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ae80a98e0 0x7f1ae81727f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:02.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.299+0000 7f1aee873700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ae80cba30 0x7f1ae80ab560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:02.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.299+0000 7f1aee873700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ae8159800 0x7f1ae80abaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:02.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.299+0000 7f1aee873700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80ae900 con 0x7f1ae80a98e0 2024-01-26T23:48:02.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.299+0000 7f1aee873700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae8172d30 con 0x7f1ae8159800 2024-01-26T23:48:02.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.299+0000 7f1aee873700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80adcf0 con 0x7f1ae80cba30 2024-01-26T23:48:02.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.299+0000 7f1ae7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ae80a98e0 0x7f1ae81727f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:02.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.299+0000 7f1ae7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ae80a98e0 0x7f1ae81727f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39514/0 (socket says 172.21.15.46:39514) 2024-01-26T23:48:02.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.299+0000 7f1ae7fff700 1 -- 172.21.15.46:0/3904190444 learned_addr learned my addr 172.21.15.46:0/3904190444 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:02.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.299+0000 7f1ae77fe700 1 --2- 172.21.15.46:0/3904190444 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ae80cba30 0x7f1ae80ab560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:02.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.299+0000 7f1aece10700 1 --2- 172.21.15.46:0/3904190444 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ae8159800 0x7f1ae80abaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:02.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.299+0000 7f1ae7fff700 1 -- 172.21.15.46:0/3904190444 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ae80cba30 msgr2=0x7f1ae80ab560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:02.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.299+0000 7f1ae7fff700 1 --2- 172.21.15.46:0/3904190444 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ae80cba30 0x7f1ae80ab560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.299+0000 7f1ae7fff700 1 -- 172.21.15.46:0/3904190444 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ae8159800 msgr2=0x7f1ae80abaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:02.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.299+0000 7f1ae7fff700 1 --2- 172.21.15.46:0/3904190444 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ae8159800 0x7f1ae80abaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.299+0000 7f1ae7fff700 1 -- 172.21.15.46:0/3904190444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad8009420 con 0x7f1ae80a98e0 2024-01-26T23:48:02.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.300+0000 7f1ae7fff700 1 --2- 172.21.15.46:0/3904190444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ae80a98e0 0x7f1ae81727f0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f1ae80cd440 tx=0x7f1ad8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:02.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.300+0000 7f1acffff700 1 -- 172.21.15.46:0/3904190444 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad801c070 con 0x7f1ae80a98e0 2024-01-26T23:48:02.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.300+0000 7f1acffff700 1 -- 172.21.15.46:0/3904190444 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad8018b40 con 0x7f1ae80a98e0 2024-01-26T23:48:02.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.300+0000 7f1acffff700 1 -- 172.21.15.46:0/3904190444 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8022420 con 0x7f1ae80a98e0 2024-01-26T23:48:02.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.300+0000 7f1aee873700 1 -- 172.21.15.46:0/3904190444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae80adfd0 con 0x7f1ae80a98e0 2024-01-26T23:48:02.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.300+0000 7f1aee873700 1 -- 172.21.15.46:0/3904190444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae80ae590 con 0x7f1ae80a98e0 2024-01-26T23:48:02.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.301+0000 7f1aee873700 1 -- 172.21.15.46:0/3904190444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae804c710 con 0x7f1ae80a98e0 2024-01-26T23:48:02.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.302+0000 7f1acffff700 1 -- 172.21.15.46:0/3904190444 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 54) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f1ad8020070 con 0x7f1ae80a98e0 2024-01-26T23:48:02.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.302+0000 7f1acffff700 1 --2- 172.21.15.46:0/3904190444 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1ad4066ea0 0x7f1ad4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:02.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.302+0000 7f1acffff700 1 -- 172.21.15.46:0/3904190444 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f1ad8085790 con 0x7f1ae80a98e0 2024-01-26T23:48:02.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.302+0000 7f1ae77fe700 1 --2- 172.21.15.46:0/3904190444 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1ad4066ea0 0x7f1ad4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:02.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.303+0000 7f1ae77fe700 1 --2- 172.21.15.46:0/3904190444 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1ad4066ea0 0x7f1ad4069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1ae815d710 tx=0x7f1adc004e70 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:02.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.307+0000 7f1acffff700 1 -- 172.21.15.46:0/3904190444 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1ad8052970 con 0x7f1ae80a98e0 2024-01-26T23:48:02.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.514+0000 7f1aee873700 1 -- 172.21.15.46:0/3904190444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f1ae806df70 con 0x7f1ae80a98e0 2024-01-26T23:48:02.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.518+0000 7f1acffff700 1 -- 172.21.15.46:0/3904190444 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v38) v1 ==== 120+0+0 (secure 0 0 0) 0x7f1ad8055f90 con 0x7f1ae80a98e0 2024-01-26T23:48:02.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.520+0000 7f1aee873700 1 -- 172.21.15.46:0/3904190444 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1ad4066ea0 msgr2=0x7f1ad4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:02.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.520+0000 7f1aee873700 1 --2- 172.21.15.46:0/3904190444 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1ad4066ea0 0x7f1ad4069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1ae815d710 tx=0x7f1adc004e70 comp rx=0 tx=0).stop 2024-01-26T23:48:02.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.520+0000 7f1aee873700 1 -- 172.21.15.46:0/3904190444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ae80a98e0 msgr2=0x7f1ae81727f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:02.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.520+0000 7f1aee873700 1 --2- 172.21.15.46:0/3904190444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ae80a98e0 0x7f1ae81727f0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f1ae80cd440 tx=0x7f1ad8018000 comp rx=0 tx=0).stop 2024-01-26T23:48:02.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.521+0000 7f1aee873700 1 -- 172.21.15.46:0/3904190444 shutdown_connections 2024-01-26T23:48:02.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.521+0000 7f1aee873700 1 --2- 172.21.15.46:0/3904190444 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1ad4066ea0 0x7f1ad4069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.521+0000 7f1aee873700 1 --2- 172.21.15.46:0/3904190444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ae80a98e0 0x7f1ae81727f0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.521+0000 7f1aee873700 1 --2- 172.21.15.46:0/3904190444 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ae8159800 0x7f1ae80abaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.521+0000 7f1aee873700 1 --2- 172.21.15.46:0/3904190444 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ae80cba30 0x7f1ae80ab560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.521+0000 7f1aee873700 1 -- 172.21.15.46:0/3904190444 >> 172.21.15.46:0/3904190444 conn(0x7f1ae8067930 msgr2=0x7f1ae80afbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:02.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.521+0000 7f1aee873700 1 -- 172.21.15.46:0/3904190444 shutdown_connections 2024-01-26T23:48:02.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.521+0000 7f1aee873700 1 -- 172.21.15.46:0/3904190444 wait complete. 2024-01-26T23:48:02.535 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-26T23:48:02.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.668+0000 7f1dfdf99700 1 Processor -- start 2024-01-26T23:48:02.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.668+0000 7f1dfdf99700 1 -- start start 2024-01-26T23:48:02.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.669+0000 7f1dfdf99700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1df8156940 0x7f1df806c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:02.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.669+0000 7f1dfdf99700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df80c8b90 con 0x7f1df806f9a0 2024-01-26T23:48:02.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.669+0000 7f1dfdf99700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df80c8e50 con 0x7f1df8156570 2024-01-26T23:48:02.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.669+0000 7f1dfdf99700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df806d0b0 con 0x7f1df8156940 2024-01-26T23:48:02.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.669+0000 7f1df7fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1df806f9a0 0x7f1df806eb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34370/0 (socket says 172.21.15.46:34370) 2024-01-26T23:48:02.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.669+0000 7f1df7fff700 1 -- 172.21.15.46:0/3456383061 learned_addr learned my addr 172.21.15.46:0/3456383061 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:02.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df6ffd700 1 --2- 172.21.15.46:0/3456383061 >> v2:172.21.15.106:3300/0 conn(0x7f1df8156940 0x7f1df806c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:02.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df67fc700 1 -- 172.21.15.46:0/3456383061 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669370181 0 0) 0x7f1df80c8b90 con 0x7f1df806f9a0 2024-01-26T23:48:02.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df67fc700 1 -- 172.21.15.46:0/3456383061 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1de4003570 con 0x7f1df806f9a0 2024-01-26T23:48:02.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df67fc700 1 -- 172.21.15.46:0/3456383061 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 998511185 0 0) 0x7f1de4003570 con 0x7f1df806f9a0 2024-01-26T23:48:02.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df67fc700 1 -- 172.21.15.46:0/3456383061 >> v2:172.21.15.106:3300/0 conn(0x7f1df8156940 msgr2=0x7f1df806c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:02.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df67fc700 1 --2- 172.21.15.46:0/3456383061 >> v2:172.21.15.106:3300/0 conn(0x7f1df8156940 0x7f1df806c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df67fc700 1 -- 172.21.15.46:0/3456383061 >> v1:172.21.15.106:6789/0 conn(0x7f1df8156570 legacy=0x7f1df80c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:02.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df67fc700 1 -- 172.21.15.46:0/3456383061 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df80c4c40 con 0x7f1df806f9a0 2024-01-26T23:48:02.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df67fc700 1 -- 172.21.15.46:0/3456383061 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f1df00017f0 con 0x7f1df806f9a0 2024-01-26T23:48:02.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df67fc700 1 -- 172.21.15.46:0/3456383061 >> v1:172.21.15.46:6789/0 conn(0x7f1df806f9a0 legacy=0x7f1df806eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:02.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df67fc700 1 --2- 172.21.15.46:0/3456383061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1de4003a60 0x7f1de4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:02.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df67fc700 1 --2- 172.21.15.46:0/3456383061 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1de4004810 0x7f1de4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:02.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df67fc700 1 --2- 172.21.15.46:0/3456383061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1de40051c0 0x7f1de400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:02.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df67fc700 1 -- 172.21.15.46:0/3456383061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df806d0b0 con 0x7f1de4003a60 2024-01-26T23:48:02.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df67fc700 1 -- 172.21.15.46:0/3456383061 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de400c160 con 0x7f1de40051c0 2024-01-26T23:48:02.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df67fc700 1 -- 172.21.15.46:0/3456383061 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de400c320 con 0x7f1de4004810 2024-01-26T23:48:02.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df77fe700 1 --2- 172.21.15.46:0/3456383061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1de4003a60 0x7f1de4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:02.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df7fff700 1 --2- 172.21.15.46:0/3456383061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1de40051c0 0x7f1de400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:02.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.670+0000 7f1df6ffd700 1 --2- 172.21.15.46:0/3456383061 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1de4004810 0x7f1de4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:02.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.671+0000 7f1df77fe700 1 -- 172.21.15.46:0/3456383061 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1de4004810 msgr2=0x7f1de4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:02.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.671+0000 7f1df77fe700 1 --2- 172.21.15.46:0/3456383061 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1de4004810 0x7f1de4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.671+0000 7f1df77fe700 1 -- 172.21.15.46:0/3456383061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1de40051c0 msgr2=0x7f1de400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:02.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.671+0000 7f1df77fe700 1 --2- 172.21.15.46:0/3456383061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1de40051c0 0x7f1de400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.671+0000 7f1df77fe700 1 -- 172.21.15.46:0/3456383061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de4003570 con 0x7f1de4003a60 2024-01-26T23:48:02.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.671+0000 7f1df7fff700 1 -- 172.21.15.46:0/3456383061 reap_dead start 2024-01-26T23:48:02.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.671+0000 7f1df77fe700 1 --2- 172.21.15.46:0/3456383061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1de4003a60 0x7f1de4003e70 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f1de80018b0 tx=0x7f1de8007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df5cb3b700c9215 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:02.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.671+0000 7f1df67fc700 1 -- 172.21.15.46:0/3456383061 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8012420 con 0x7f1de4003a60 2024-01-26T23:48:02.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.671+0000 7f1df67fc700 1 -- 172.21.15.46:0/3456383061 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de80125a0 con 0x7f1de4003a60 2024-01-26T23:48:02.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.671+0000 7f1df67fc700 1 -- 172.21.15.46:0/3456383061 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8012960 con 0x7f1de4003a60 2024-01-26T23:48:02.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.672+0000 7f1dfdf99700 1 -- 172.21.15.46:0/3456383061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1de4003a60 msgr2=0x7f1de4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:02.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.672+0000 7f1dfdf99700 1 --2- 172.21.15.46:0/3456383061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1de4003a60 0x7f1de4003e70 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f1de80018b0 tx=0x7f1de8007d30 comp rx=0 tx=0).stop 2024-01-26T23:48:02.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.672+0000 7f1dfdf99700 1 -- 172.21.15.46:0/3456383061 shutdown_connections 2024-01-26T23:48:02.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.672+0000 7f1dfdf99700 1 --2- 172.21.15.46:0/3456383061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1de4003a60 0x7f1de4003e70 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.672+0000 7f1dfdf99700 1 -- 172.21.15.46:0/3456383061 >> 172.21.15.46:0/3456383061 conn(0x7f1df80cdf50 msgr2=0x7f1df80c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:02.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.672+0000 7f1dfdf99700 1 -- 172.21.15.46:0/3456383061 shutdown_connections 2024-01-26T23:48:02.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.672+0000 7f1dfdf99700 1 -- 172.21.15.46:0/3456383061 wait complete. 2024-01-26T23:48:02.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.673+0000 7f1dfdf99700 1 Processor -- start 2024-01-26T23:48:02.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.673+0000 7f1dfdf99700 1 -- start start 2024-01-26T23:48:02.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1dfdf99700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1de4003a60 0x7f1df8152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:02.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1dfdf99700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1df80b2740 0x7f1df80b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:02.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1dfdf99700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1df81574a0 0x7f1df8157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:02.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1dfdf99700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df8158720 con 0x7f1df80b2740 2024-01-26T23:48:02.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1dfdf99700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df80b30f0 con 0x7f1df81574a0 2024-01-26T23:48:02.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1dfdf99700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df8167b90 con 0x7f1de4003a60 2024-01-26T23:48:02.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1df6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1df80b2740 0x7f1df80b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:02.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1df6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1df80b2740 0x7f1df80b2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39540/0 (socket says 172.21.15.46:39540) 2024-01-26T23:48:02.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1df6ffd700 1 -- 172.21.15.46:0/2511577098 learned_addr learned my addr 172.21.15.46:0/2511577098 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:02.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1df77fe700 1 --2- 172.21.15.46:0/2511577098 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1de4003a60 0x7f1df8152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:02.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1df7fff700 1 --2- 172.21.15.46:0/2511577098 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1df81574a0 0x7f1df8157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:02.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1df6ffd700 1 -- 172.21.15.46:0/2511577098 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1de4003a60 msgr2=0x7f1df8152640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:02.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1df6ffd700 1 --2- 172.21.15.46:0/2511577098 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1de4003a60 0x7f1df8152640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1df6ffd700 1 -- 172.21.15.46:0/2511577098 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1df81574a0 msgr2=0x7f1df8157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:02.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1df6ffd700 1 --2- 172.21.15.46:0/2511577098 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1df81574a0 0x7f1df8157910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1df6ffd700 1 -- 172.21.15.46:0/2511577098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df8158160 con 0x7f1df80b2740 2024-01-26T23:48:02.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1df7fff700 1 --2- 172.21.15.46:0/2511577098 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1df81574a0 0x7f1df8157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:02.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.674+0000 7f1df77fe700 1 --2- 172.21.15.46:0/2511577098 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1de4003a60 0x7f1df8152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:02.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.675+0000 7f1df6ffd700 1 --2- 172.21.15.46:0/2511577098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1df80b2740 0x7f1df80b2bb0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f1de0011d40 tx=0x7f1de00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:02.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.675+0000 7f1def7fe700 1 -- 172.21.15.46:0/2511577098 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de000fa60 con 0x7f1df80b2740 2024-01-26T23:48:02.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.675+0000 7f1def7fe700 1 -- 172.21.15.46:0/2511577098 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de000fbc0 con 0x7f1df80b2740 2024-01-26T23:48:02.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.675+0000 7f1dfdf99700 1 -- 172.21.15.46:0/2511577098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df8158440 con 0x7f1df80b2740 2024-01-26T23:48:02.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.675+0000 7f1def7fe700 1 -- 172.21.15.46:0/2511577098 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de0014440 con 0x7f1df80b2740 2024-01-26T23:48:02.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.675+0000 7f1dfdf99700 1 -- 172.21.15.46:0/2511577098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df8157e50 con 0x7f1df80b2740 2024-01-26T23:48:02.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.676+0000 7f1dfdf99700 1 -- 172.21.15.46:0/2511577098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df8167680 con 0x7f1df80b2740 2024-01-26T23:48:02.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.676+0000 7f1def7fe700 1 -- 172.21.15.46:0/2511577098 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 54) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f1de0014a80 con 0x7f1df80b2740 2024-01-26T23:48:02.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.677+0000 7f1def7fe700 1 --2- 172.21.15.46:0/2511577098 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1dd8066ea0 0x7f1dd8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:02.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.677+0000 7f1def7fe700 1 -- 172.21.15.46:0/2511577098 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f1de0018030 con 0x7f1df80b2740 2024-01-26T23:48:02.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.677+0000 7f1df77fe700 1 --2- 172.21.15.46:0/2511577098 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1dd8066ea0 0x7f1dd8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:02.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.678+0000 7f1df77fe700 1 --2- 172.21.15.46:0/2511577098 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1dd8066ea0 0x7f1dd8069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1de80018b0 tx=0x7f1de8015000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:02.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.681+0000 7f1def7fe700 1 -- 172.21.15.46:0/2511577098 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1df8167680 con 0x7f1df80b2740 2024-01-26T23:48:02.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.890+0000 7f1dfdf99700 1 -- 172.21.15.46:0/2511577098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f1df8167680 con 0x7f1df80b2740 2024-01-26T23:48:02.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.895+0000 7f1def7fe700 1 -- 172.21.15.46:0/2511577098 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v39) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1df8167680 con 0x7f1df80b2740 2024-01-26T23:48:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.897+0000 7f1dfdf99700 1 -- 172.21.15.46:0/2511577098 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1dd8066ea0 msgr2=0x7f1dd8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.897+0000 7f1dfdf99700 1 --2- 172.21.15.46:0/2511577098 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1dd8066ea0 0x7f1dd8069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1de80018b0 tx=0x7f1de8015000 comp rx=0 tx=0).stop 2024-01-26T23:48:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.897+0000 7f1dfdf99700 1 -- 172.21.15.46:0/2511577098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1df80b2740 msgr2=0x7f1df80b2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.897+0000 7f1dfdf99700 1 --2- 172.21.15.46:0/2511577098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1df80b2740 0x7f1df80b2bb0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f1de0011d40 tx=0x7f1de00091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.897+0000 7f1dfdf99700 1 -- 172.21.15.46:0/2511577098 shutdown_connections 2024-01-26T23:48:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.897+0000 7f1dfdf99700 1 --2- 172.21.15.46:0/2511577098 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1dd8066ea0 0x7f1dd8069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.897+0000 7f1dfdf99700 1 --2- 172.21.15.46:0/2511577098 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1df81574a0 0x7f1df8157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.897+0000 7f1dfdf99700 1 --2- 172.21.15.46:0/2511577098 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1de4003a60 0x7f1df8152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.897+0000 7f1dfdf99700 1 --2- 172.21.15.46:0/2511577098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1df80b2740 0x7f1df80b2bb0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.897+0000 7f1dfdf99700 1 -- 172.21.15.46:0/2511577098 >> 172.21.15.46:0/2511577098 conn(0x7f1df80cdf50 msgr2=0x7f1df80c4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:02.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.897+0000 7f1dfdf99700 1 -- 172.21.15.46:0/2511577098 shutdown_connections 2024-01-26T23:48:02.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:02.897+0000 7f1dfdf99700 1 -- 172.21.15.46:0/2511577098 wait complete. 2024-01-26T23:48:02.912 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-01-26T23:48:03.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.042+0000 7fbf30b2a700 1 Processor -- start 2024-01-26T23:48:03.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.042+0000 7fbf30b2a700 1 -- start start 2024-01-26T23:48:03.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.043+0000 7fbf30b2a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbf2c0b12c0 0x7fbf2c1646a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:03.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.043+0000 7fbf30b2a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf2c06f940 con 0x7fbf2c159b30 2024-01-26T23:48:03.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.043+0000 7fbf30b2a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf2c0b3230 con 0x7fbf2c159760 2024-01-26T23:48:03.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.043+0000 7fbf30b2a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf2c06ed30 con 0x7fbf2c0b12c0 2024-01-26T23:48:03.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf29d9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbf2c159b30 0x7fbf2c0b0bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34380/0 (socket says 172.21.15.46:34380) 2024-01-26T23:48:03.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf29d9b700 1 -- 172.21.15.46:0/830677970 learned_addr learned my addr 172.21.15.46:0/830677970 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:03.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2ad9d700 1 --2- 172.21.15.46:0/830677970 >> v2:172.21.15.106:3300/0 conn(0x7fbf2c0b12c0 0x7fbf2c1646a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:03.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2959a700 1 -- 172.21.15.46:0/830677970 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190183578 0 0) 0x7fbf2c06f940 con 0x7fbf2c159b30 2024-01-26T23:48:03.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2959a700 1 -- 172.21.15.46:0/830677970 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf14003570 con 0x7fbf2c159b30 2024-01-26T23:48:03.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2959a700 1 -- 172.21.15.46:0/830677970 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363541469 0 0) 0x7fbf2c0b3230 con 0x7fbf2c159760 2024-01-26T23:48:03.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2959a700 1 -- 172.21.15.46:0/830677970 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf2c06f940 con 0x7fbf2c159760 2024-01-26T23:48:03.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2959a700 1 -- 172.21.15.46:0/830677970 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 154542944 0 0) 0x7fbf14003570 con 0x7fbf2c159b30 2024-01-26T23:48:03.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2959a700 1 -- 172.21.15.46:0/830677970 >> v2:172.21.15.106:3300/0 conn(0x7fbf2c0b12c0 msgr2=0x7fbf2c1646a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:03.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2959a700 1 --2- 172.21.15.46:0/830677970 >> v2:172.21.15.106:3300/0 conn(0x7fbf2c0b12c0 0x7fbf2c1646a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2959a700 1 -- 172.21.15.46:0/830677970 >> v1:172.21.15.106:6789/0 conn(0x7fbf2c159760 legacy=0x7fbf2c0b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:03.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2959a700 1 -- 172.21.15.46:0/830677970 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2c06f010 con 0x7fbf2c159b30 2024-01-26T23:48:03.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2959a700 1 -- 172.21.15.46:0/830677970 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fbf20001470 con 0x7fbf2c159b30 2024-01-26T23:48:03.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2959a700 1 -- 172.21.15.46:0/830677970 >> v1:172.21.15.46:6789/0 conn(0x7fbf2c159b30 legacy=0x7fbf2c0b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:03.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2959a700 1 --2- 172.21.15.46:0/830677970 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf14003bf0 0x7fbf14003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:03.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2959a700 1 --2- 172.21.15.46:0/830677970 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf14004810 0x7fbf14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:03.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2959a700 1 --2- 172.21.15.46:0/830677970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf140051c0 0x7fbf1400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:03.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2959a700 1 -- 172.21.15.46:0/830677970 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c06ed30 con 0x7fbf140051c0 2024-01-26T23:48:03.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2959a700 1 -- 172.21.15.46:0/830677970 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1400c160 con 0x7fbf14003bf0 2024-01-26T23:48:03.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.044+0000 7fbf2959a700 1 -- 172.21.15.46:0/830677970 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1400c320 con 0x7fbf14004810 2024-01-26T23:48:03.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.045+0000 7fbf2a59c700 1 --2- 172.21.15.46:0/830677970 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf14003bf0 0x7fbf14003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:03.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.045+0000 7fbf29d9b700 1 --2- 172.21.15.46:0/830677970 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf14004810 0x7fbf14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:03.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.045+0000 7fbf2ad9d700 1 --2- 172.21.15.46:0/830677970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf140051c0 0x7fbf1400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:03.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.045+0000 7fbf2a59c700 1 -- 172.21.15.46:0/830677970 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf14004810 msgr2=0x7fbf14004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:03.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.045+0000 7fbf2a59c700 1 --2- 172.21.15.46:0/830677970 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf14004810 0x7fbf14004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.045+0000 7fbf2a59c700 1 -- 172.21.15.46:0/830677970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf140051c0 msgr2=0x7fbf1400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:03.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.045+0000 7fbf2a59c700 1 --2- 172.21.15.46:0/830677970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf140051c0 0x7fbf1400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.045+0000 7fbf2a59c700 1 -- 172.21.15.46:0/830677970 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf14003570 con 0x7fbf14003bf0 2024-01-26T23:48:03.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.045+0000 7fbf2ad9d700 1 --2- 172.21.15.46:0/830677970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf140051c0 0x7fbf1400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:48:03.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.045+0000 7fbf2ad9d700 1 -- 172.21.15.46:0/830677970 reap_dead start 2024-01-26T23:48:03.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.045+0000 7fbf2a59c700 1 --2- 172.21.15.46:0/830677970 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf14003bf0 0x7fbf14003fe0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c0019e0 tx=0x7fbf1c007d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1c4c050514c306e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:03.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.045+0000 7fbf2959a700 1 -- 172.21.15.46:0/830677970 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c00c5d0 con 0x7fbf14003bf0 2024-01-26T23:48:03.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.045+0000 7fbf2959a700 1 -- 172.21.15.46:0/830677970 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1c00c730 con 0x7fbf14003bf0 2024-01-26T23:48:03.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.045+0000 7fbf2959a700 1 -- 172.21.15.46:0/830677970 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c00caf0 con 0x7fbf14003bf0 2024-01-26T23:48:03.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.046+0000 7fbf30b2a700 1 -- 172.21.15.46:0/830677970 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf14003bf0 msgr2=0x7fbf14003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:03.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.046+0000 7fbf30b2a700 1 --2- 172.21.15.46:0/830677970 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf14003bf0 0x7fbf14003fe0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c0019e0 tx=0x7fbf1c007d30 comp rx=0 tx=0).stop 2024-01-26T23:48:03.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.046+0000 7fbf30b2a700 1 -- 172.21.15.46:0/830677970 shutdown_connections 2024-01-26T23:48:03.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.046+0000 7fbf30b2a700 1 --2- 172.21.15.46:0/830677970 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf14003bf0 0x7fbf14003fe0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.046+0000 7fbf30b2a700 1 -- 172.21.15.46:0/830677970 >> 172.21.15.46:0/830677970 conn(0x7fbf2c069510 msgr2=0x7fbf2c0aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:03.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.046+0000 7fbf30b2a700 1 -- 172.21.15.46:0/830677970 shutdown_connections 2024-01-26T23:48:03.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.046+0000 7fbf30b2a700 1 -- 172.21.15.46:0/830677970 wait complete. 2024-01-26T23:48:03.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.048+0000 7fbf30b2a700 1 Processor -- start 2024-01-26T23:48:03.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.048+0000 7fbf30b2a700 1 -- start start 2024-01-26T23:48:03.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf30b2a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf14003bf0 0x7fbf2c15ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:03.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf30b2a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf2c157fc0 0x7fbf2c14ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:03.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf30b2a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf2c0c4620 0x7fbf2c0c4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:03.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf30b2a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c0c4fd0 con 0x7fbf2c157fc0 2024-01-26T23:48:03.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf30b2a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c15f990 con 0x7fbf14003bf0 2024-01-26T23:48:03.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf30b2a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c158850 con 0x7fbf2c0c4620 2024-01-26T23:48:03.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf29d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf2c157fc0 0x7fbf2c14ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:03.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf29d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf2c157fc0 0x7fbf2c14ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39550/0 (socket says 172.21.15.46:39550) 2024-01-26T23:48:03.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf29d9b700 1 -- 172.21.15.46:0/3497250604 learned_addr learned my addr 172.21.15.46:0/3497250604 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:03.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf2a59c700 1 --2- 172.21.15.46:0/3497250604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf14003bf0 0x7fbf2c15ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:03.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf2ad9d700 1 --2- 172.21.15.46:0/3497250604 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf2c0c4620 0x7fbf2c0c4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:03.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf29d9b700 1 -- 172.21.15.46:0/3497250604 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf2c0c4620 msgr2=0x7fbf2c0c4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:03.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf29d9b700 1 --2- 172.21.15.46:0/3497250604 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf2c0c4620 0x7fbf2c0c4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf29d9b700 1 -- 172.21.15.46:0/3497250604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf14003bf0 msgr2=0x7fbf2c15ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:03.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf29d9b700 1 --2- 172.21.15.46:0/3497250604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf14003bf0 0x7fbf2c15ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf29d9b700 1 -- 172.21.15.46:0/3497250604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2c15f0f0 con 0x7fbf2c157fc0 2024-01-26T23:48:03.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf2a59c700 1 --2- 172.21.15.46:0/3497250604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf14003bf0 0x7fbf2c15ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:03.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.049+0000 7fbf29d9b700 1 --2- 172.21.15.46:0/3497250604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf2c157fc0 0x7fbf2c14ae50 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fbf20007390 tx=0x7fbf20001df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:03.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.050+0000 7fbf1a7fc700 1 -- 172.21.15.46:0/3497250604 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20018030 con 0x7fbf2c157fc0 2024-01-26T23:48:03.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.050+0000 7fbf1a7fc700 1 -- 172.21.15.46:0/3497250604 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2000f850 con 0x7fbf2c157fc0 2024-01-26T23:48:03.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.050+0000 7fbf30b2a700 1 -- 172.21.15.46:0/3497250604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf2c15f3d0 con 0x7fbf2c157fc0 2024-01-26T23:48:03.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.050+0000 7fbf1a7fc700 1 -- 172.21.15.46:0/3497250604 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2000fc10 con 0x7fbf2c157fc0 2024-01-26T23:48:03.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.050+0000 7fbf30b2a700 1 -- 172.21.15.46:0/3497250604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf2c0c2290 con 0x7fbf2c157fc0 2024-01-26T23:48:03.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.051+0000 7fbf30b2a700 1 -- 172.21.15.46:0/3497250604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf2c0685a0 con 0x7fbf2c157fc0 2024-01-26T23:48:03.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.051+0000 7fbf1a7fc700 1 -- 172.21.15.46:0/3497250604 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 54) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fbf2001b070 con 0x7fbf2c157fc0 2024-01-26T23:48:03.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.052+0000 7fbf1a7fc700 1 --2- 172.21.15.46:0/3497250604 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf0c066ea0 0x7fbf0c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:03.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.052+0000 7fbf1a7fc700 1 -- 172.21.15.46:0/3497250604 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6253+0+0 (secure 0 0 0) 0x7fbf20019070 con 0x7fbf2c157fc0 2024-01-26T23:48:03.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.052+0000 7fbf2a59c700 1 --2- 172.21.15.46:0/3497250604 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf0c066ea0 0x7fbf0c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:03.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.052+0000 7fbf2a59c700 1 --2- 172.21.15.46:0/3497250604 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf0c066ea0 0x7fbf0c069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c000c00 tx=0x7fbf1c015000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:03.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.056+0000 7fbf1a7fc700 1 -- 172.21.15.46:0/3497250604 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbf20051720 con 0x7fbf2c157fc0 2024-01-26T23:48:03.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.345+0000 7fbf30b2a700 1 -- 172.21.15.46:0/3497250604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7fbf2c04c710 con 0x7fbf2c157fc0 2024-01-26T23:48:03.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.523+0000 7fbf1a7fc700 1 -- 172.21.15.46:0/3497250604 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbf20051af0 con 0x7fbf2c157fc0 2024-01-26T23:48:03.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.757+0000 7fbf1a7fc700 1 -- 172.21.15.46:0/3497250604 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v689) v1 ==== 183+0+0 (secure 0 0 0) 0x7fbf20051720 con 0x7fbf2c157fc0 2024-01-26T23:48:03.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.760+0000 7fbf30b2a700 1 -- 172.21.15.46:0/3497250604 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf0c066ea0 msgr2=0x7fbf0c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:03.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.760+0000 7fbf30b2a700 1 --2- 172.21.15.46:0/3497250604 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf0c066ea0 0x7fbf0c069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c000c00 tx=0x7fbf1c015000 comp rx=0 tx=0).stop 2024-01-26T23:48:03.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.760+0000 7fbf30b2a700 1 -- 172.21.15.46:0/3497250604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf2c157fc0 msgr2=0x7fbf2c14ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:03.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.760+0000 7fbf30b2a700 1 --2- 172.21.15.46:0/3497250604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf2c157fc0 0x7fbf2c14ae50 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fbf20007390 tx=0x7fbf20001df0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.760+0000 7fbf30b2a700 1 -- 172.21.15.46:0/3497250604 shutdown_connections 2024-01-26T23:48:03.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.760+0000 7fbf30b2a700 1 --2- 172.21.15.46:0/3497250604 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf0c066ea0 0x7fbf0c069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.760+0000 7fbf30b2a700 1 --2- 172.21.15.46:0/3497250604 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf2c0c4620 0x7fbf2c0c4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.760+0000 7fbf30b2a700 1 --2- 172.21.15.46:0/3497250604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf14003bf0 0x7fbf2c15ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.760+0000 7fbf30b2a700 1 --2- 172.21.15.46:0/3497250604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf2c157fc0 0x7fbf2c14ae50 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.760+0000 7fbf30b2a700 1 -- 172.21.15.46:0/3497250604 >> 172.21.15.46:0/3497250604 conn(0x7fbf2c069510 msgr2=0x7fbf2c067930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:03.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.760+0000 7fbf30b2a700 1 -- 172.21.15.46:0/3497250604 shutdown_connections 2024-01-26T23:48:03.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.760+0000 7fbf30b2a700 1 -- 172.21.15.46:0/3497250604 wait complete. 2024-01-26T23:48:03.777 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2024-01-26T23:48:03.777 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2024-01-26T23:48:03.777 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-26T23:48:03.778 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-01-26T23:48:03.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.914+0000 7fcbeeab6700 1 Processor -- start 2024-01-26T23:48:03.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.914+0000 7fcbeeab6700 1 -- start start 2024-01-26T23:48:03.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.915+0000 7fcbeeab6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbe806d310 0x7fcbe80b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:03.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbeeab6700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fcbe806c290 0x7fcbe80c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:03.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbeeab6700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe806d840 con 0x7fcbe80b3bf0 2024-01-26T23:48:03.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbeeab6700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe815b910 con 0x7fcbe806c290 2024-01-26T23:48:03.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbeeab6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe8157d00 con 0x7fcbe806d310 2024-01-26T23:48:03.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbec852700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcbe80b3bf0 0x7fcbe815b200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34396/0 (socket says 172.21.15.46:34396) 2024-01-26T23:48:03.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbec852700 1 -- 172.21.15.46:0/2652540642 learned_addr learned my addr 172.21.15.46:0/2652540642 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:03.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe7fff700 1 --2- 172.21.15.46:0/2652540642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbe806d310 0x7fcbe80b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:03.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbed053700 1 --2- 172.21.15.46:0/2652540642 >> v2:172.21.15.106:3300/0 conn(0x7fcbe806c290 0x7fcbe80c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:03.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe77fe700 1 -- 172.21.15.46:0/2652540642 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261634589 0 0) 0x7fcbe806d840 con 0x7fcbe80b3bf0 2024-01-26T23:48:03.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe77fe700 1 -- 172.21.15.46:0/2652540642 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd4003570 con 0x7fcbe80b3bf0 2024-01-26T23:48:03.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe77fe700 1 -- 172.21.15.46:0/2652540642 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3225883860 0 0) 0x7fcbd4003570 con 0x7fcbe80b3bf0 2024-01-26T23:48:03.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe77fe700 1 -- 172.21.15.46:0/2652540642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbe806d310 msgr2=0x7fcbe80b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:03.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe77fe700 1 --2- 172.21.15.46:0/2652540642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbe806d310 0x7fcbe80b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe77fe700 1 -- 172.21.15.46:0/2652540642 >> v2:172.21.15.106:3300/0 conn(0x7fcbe806c290 msgr2=0x7fcbe80c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:03.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe77fe700 1 --2- 172.21.15.46:0/2652540642 >> v2:172.21.15.106:3300/0 conn(0x7fcbe806c290 0x7fcbe80c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe77fe700 1 -- 172.21.15.46:0/2652540642 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe806db20 con 0x7fcbe80b3bf0 2024-01-26T23:48:03.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe77fe700 1 -- 172.21.15.46:0/2652540642 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fcbd8001470 con 0x7fcbe80b3bf0 2024-01-26T23:48:03.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe77fe700 1 -- 172.21.15.46:0/2652540642 >> v1:172.21.15.46:6789/0 conn(0x7fcbe80b3bf0 legacy=0x7fcbe815b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:03.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe77fe700 1 --2- 172.21.15.46:0/2652540642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbd4003a60 0x7fcbd4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:03.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe77fe700 1 --2- 172.21.15.46:0/2652540642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbd4004810 0x7fcbd4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:03.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe77fe700 1 --2- 172.21.15.46:0/2652540642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbe806d310 0x7fcbd40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:03.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe77fe700 1 -- 172.21.15.46:0/2652540642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe815b910 con 0x7fcbd4004810 2024-01-26T23:48:03.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe77fe700 1 -- 172.21.15.46:0/2652540642 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe8157d00 con 0x7fcbd4003a60 2024-01-26T23:48:03.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe77fe700 1 -- 172.21.15.46:0/2652540642 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd400ba60 con 0x7fcbe806d310 2024-01-26T23:48:03.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.916+0000 7fcbe7fff700 1 --2- 172.21.15.46:0/2652540642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbd4004810 0x7fcbd4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:03.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.917+0000 7fcbec852700 1 --2- 172.21.15.46:0/2652540642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbd4003a60 0x7fcbd4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:03.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.917+0000 7fcbed053700 1 --2- 172.21.15.46:0/2652540642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbe806d310 0x7fcbd40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:03.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.917+0000 7fcbe7fff700 1 -- 172.21.15.46:0/2652540642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbe806d310 msgr2=0x7fcbd40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:03.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.917+0000 7fcbe7fff700 1 --2- 172.21.15.46:0/2652540642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbe806d310 0x7fcbd40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.917+0000 7fcbe7fff700 1 -- 172.21.15.46:0/2652540642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbd4003a60 msgr2=0x7fcbd4003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:03.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.917+0000 7fcbe7fff700 1 --2- 172.21.15.46:0/2652540642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbd4003a60 0x7fcbd4003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.917+0000 7fcbe7fff700 1 -- 172.21.15.46:0/2652540642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd4003570 con 0x7fcbd4004810 2024-01-26T23:48:03.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.917+0000 7fcbe7fff700 1 --2- 172.21.15.46:0/2652540642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbd4004810 0x7fcbd4004c80 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0009c30 tx=0x7fcbd00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df5bce368b164a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:03.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.917+0000 7fcbe77fe700 1 -- 172.21.15.46:0/2652540642 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd000e710 con 0x7fcbd4004810 2024-01-26T23:48:03.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.917+0000 7fcbe77fe700 1 -- 172.21.15.46:0/2652540642 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd000e870 con 0x7fcbd4004810 2024-01-26T23:48:03.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.917+0000 7fcbe77fe700 1 -- 172.21.15.46:0/2652540642 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd000ec30 con 0x7fcbd4004810 2024-01-26T23:48:03.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.918+0000 7fcbeeab6700 1 -- 172.21.15.46:0/2652540642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbd4004810 msgr2=0x7fcbd4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:03.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.918+0000 7fcbeeab6700 1 --2- 172.21.15.46:0/2652540642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbd4004810 0x7fcbd4004c80 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0009c30 tx=0x7fcbd00091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.918+0000 7fcbed053700 1 -- 172.21.15.46:0/2652540642 reap_dead start 2024-01-26T23:48:03.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.918+0000 7fcbeeab6700 1 -- 172.21.15.46:0/2652540642 shutdown_connections 2024-01-26T23:48:03.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.918+0000 7fcbeeab6700 1 -- 172.21.15.46:0/2652540642 >> 172.21.15.46:0/2652540642 conn(0x7fcbe80ce530 msgr2=0x7fcbe80ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:03.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.918+0000 7fcbeeab6700 1 -- 172.21.15.46:0/2652540642 shutdown_connections 2024-01-26T23:48:03.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.918+0000 7fcbeeab6700 1 -- 172.21.15.46:0/2652540642 wait complete. 2024-01-26T23:48:03.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.919+0000 7fcbeeab6700 1 Processor -- start 2024-01-26T23:48:03.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.920+0000 7fcbeeab6700 1 -- start start 2024-01-26T23:48:03.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbeeab6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbd4004810 0x7fcbe80690a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:03.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbeeab6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbe816a0f0 0x7fcbe80c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:03.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbeeab6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbe816ca90 0x7fcbe816cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:03.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbeeab6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe816d440 con 0x7fcbe816ca90 2024-01-26T23:48:03.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbeeab6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe806a600 con 0x7fcbd4004810 2024-01-26T23:48:03.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbeeab6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe806aa90 con 0x7fcbe816a0f0 2024-01-26T23:48:03.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbed053700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbe816ca90 0x7fcbe816cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:03.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbed053700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbe816ca90 0x7fcbe816cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39564/0 (socket says 172.21.15.46:39564) 2024-01-26T23:48:03.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbed053700 1 -- 172.21.15.46:0/2506144182 learned_addr learned my addr 172.21.15.46:0/2506144182 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:03.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbec852700 1 --2- 172.21.15.46:0/2506144182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbd4004810 0x7fcbe80690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:03.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbe7fff700 1 --2- 172.21.15.46:0/2506144182 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbe816a0f0 0x7fcbe80c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:03.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbed053700 1 -- 172.21.15.46:0/2506144182 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbe816a0f0 msgr2=0x7fcbe80c7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:03.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbed053700 1 --2- 172.21.15.46:0/2506144182 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbe816a0f0 0x7fcbe80c7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbed053700 1 -- 172.21.15.46:0/2506144182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbd4004810 msgr2=0x7fcbe80690a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:03.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbed053700 1 --2- 172.21.15.46:0/2506144182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbd4004810 0x7fcbe80690a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:03.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbed053700 1 -- 172.21.15.46:0/2506144182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe80699f0 con 0x7fcbe816ca90 2024-01-26T23:48:03.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbec852700 1 --2- 172.21.15.46:0/2506144182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbd4004810 0x7fcbe80690a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:03.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbe7fff700 1 --2- 172.21.15.46:0/2506144182 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbe816a0f0 0x7fcbe80c7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:03.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.921+0000 7fcbed053700 1 --2- 172.21.15.46:0/2506144182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbe816ca90 0x7fcbe816cf00 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fcbe806fc40 tx=0x7fcbe000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:03.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.922+0000 7fcbe4ff9700 1 -- 172.21.15.46:0/2506144182 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe000e040 con 0x7fcbe816ca90 2024-01-26T23:48:03.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.922+0000 7fcbe4ff9700 1 -- 172.21.15.46:0/2506144182 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe000f040 con 0x7fcbe816ca90 2024-01-26T23:48:03.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.922+0000 7fcbeeab6700 1 -- 172.21.15.46:0/2506144182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe8069cd0 con 0x7fcbe816ca90 2024-01-26T23:48:03.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.922+0000 7fcbe4ff9700 1 -- 172.21.15.46:0/2506144182 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe000fe20 con 0x7fcbe816ca90 2024-01-26T23:48:03.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.922+0000 7fcbeeab6700 1 -- 172.21.15.46:0/2506144182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe806a290 con 0x7fcbe816ca90 2024-01-26T23:48:03.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.923+0000 7fcbeeab6700 1 -- 172.21.15.46:0/2506144182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe804c710 con 0x7fcbe816ca90 2024-01-26T23:48:03.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.923+0000 7fcbe4ff9700 1 -- 172.21.15.46:0/2506144182 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fcbe0015640 con 0x7fcbe816ca90 2024-01-26T23:48:03.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.924+0000 7fcbe4ff9700 1 --2- 172.21.15.46:0/2506144182 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fcbc8066f60 0x7fcbc8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:03.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.924+0000 7fcbe4ff9700 1 -- 172.21.15.46:0/2506144182 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6253+0+0 (secure 0 0 0) 0x7fcbe0085020 con 0x7fcbe816ca90 2024-01-26T23:48:03.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.924+0000 7fcbec852700 1 --2- 172.21.15.46:0/2506144182 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fcbc8066f60 0x7fcbc8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:03.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.924+0000 7fcbec852700 1 --2- 172.21.15.46:0/2506144182 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fcbc8066f60 0x7fcbc8069410 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8007f10 tx=0x7fcbd8009040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:03.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:03.928+0000 7fcbe4ff9700 1 -- 172.21.15.46:0/2506144182 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcbe0052070 con 0x7fcbe816ca90 2024-01-26T23:48:04.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.215+0000 7fcbeeab6700 1 -- 172.21.15.46:0/2506144182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7fcbe806c9c0 con 0x7fcbe816ca90 2024-01-26T23:48:04.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.761+0000 7fcbe4ff9700 1 -- 172.21.15.46:0/2506144182 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v690) v1 ==== 151+0+0 (secure 0 0 0) 0x7fcbe0055690 con 0x7fcbe816ca90 2024-01-26T23:48:04.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.764+0000 7fcbeeab6700 1 -- 172.21.15.46:0/2506144182 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fcbc8066f60 msgr2=0x7fcbc8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:04.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.764+0000 7fcbeeab6700 1 --2- 172.21.15.46:0/2506144182 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fcbc8066f60 0x7fcbc8069410 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8007f10 tx=0x7fcbd8009040 comp rx=0 tx=0).stop 2024-01-26T23:48:04.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.764+0000 7fcbeeab6700 1 -- 172.21.15.46:0/2506144182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbe816ca90 msgr2=0x7fcbe816cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:04.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.764+0000 7fcbeeab6700 1 --2- 172.21.15.46:0/2506144182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbe816ca90 0x7fcbe816cf00 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fcbe806fc40 tx=0x7fcbe000b040 comp rx=0 tx=0).stop 2024-01-26T23:48:04.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.764+0000 7fcbeeab6700 1 -- 172.21.15.46:0/2506144182 shutdown_connections 2024-01-26T23:48:04.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.764+0000 7fcbeeab6700 1 --2- 172.21.15.46:0/2506144182 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fcbc8066f60 0x7fcbc8069410 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:04.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.764+0000 7fcbeeab6700 1 --2- 172.21.15.46:0/2506144182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbe816ca90 0x7fcbe816cf00 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:04.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.764+0000 7fcbeeab6700 1 --2- 172.21.15.46:0/2506144182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbd4004810 0x7fcbe80690a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:04.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.764+0000 7fcbeeab6700 1 --2- 172.21.15.46:0/2506144182 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbe816a0f0 0x7fcbe80c7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:04.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.764+0000 7fcbeeab6700 1 -- 172.21.15.46:0/2506144182 >> 172.21.15.46:0/2506144182 conn(0x7fcbe80ce530 msgr2=0x7fcbe80b12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:04.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.764+0000 7fcbeeab6700 1 -- 172.21.15.46:0/2506144182 shutdown_connections 2024-01-26T23:48:04.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.764+0000 7fcbeeab6700 1 -- 172.21.15.46:0/2506144182 wait complete. 2024-01-26T23:48:04.781 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-26T23:48:04.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.919+0000 7f0de73c3700 1 Processor -- start 2024-01-26T23:48:04.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.920+0000 7f0de73c3700 1 -- start start 2024-01-26T23:48:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.920+0000 7f0de73c3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0de0157750 0x7f0de0157b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.920+0000 7f0de73c3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0de006b3c0 0x7f0de00af080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.920+0000 7f0de73c3700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0de00af5c0 0x7f0de0067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.920+0000 7f0de73c3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0de0158f00 con 0x7f0de006b3c0 2024-01-26T23:48:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.920+0000 7f0de73c3700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0de00afa20 con 0x7f0de00af5c0 2024-01-26T23:48:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.920+0000 7f0de73c3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de0158060 con 0x7f0de0157750 2024-01-26T23:48:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.921+0000 7f0de495e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0de006b3c0 0x7f0de00af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.921+0000 7f0de495e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0de006b3c0 0x7f0de00af080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39574/0 (socket says 172.21.15.46:39574) 2024-01-26T23:48:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.921+0000 7f0de495e700 1 -- 172.21.15.46:0/4044730068 learned_addr learned my addr 172.21.15.46:0/4044730068 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.921+0000 7f0de515f700 1 --2- 172.21.15.46:0/4044730068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0de0157750 0x7f0de0157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.921+0000 7f0de5960700 1 --2- 172.21.15.46:0/4044730068 >> v2:172.21.15.106:3300/0 conn(0x7f0de00af5c0 0x7f0de0067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.921+0000 7f0de495e700 1 -- 172.21.15.46:0/4044730068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0de0157750 msgr2=0x7f0de0157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.921+0000 7f0de495e700 1 --2- 172.21.15.46:0/4044730068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0de0157750 0x7f0de0157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.921+0000 7f0de495e700 1 -- 172.21.15.46:0/4044730068 >> v2:172.21.15.106:3300/0 conn(0x7f0de00af5c0 msgr2=0x7f0de0067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.921+0000 7f0de495e700 1 --2- 172.21.15.46:0/4044730068 >> v2:172.21.15.106:3300/0 conn(0x7f0de00af5c0 0x7f0de0067930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.921+0000 7f0de495e700 1 -- 172.21.15.46:0/4044730068 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de006fe90 con 0x7f0de006b3c0 2024-01-26T23:48:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.921+0000 7f0de495e700 1 --2- 172.21.15.46:0/4044730068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0de006b3c0 0x7f0de00af080 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f0de00649b0 tx=0x7f0ddc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b34876e6e7cd58dd server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.921+0000 7f0dd7fff700 1 -- 172.21.15.46:0/4044730068 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc00b860 con 0x7f0de006b3c0 2024-01-26T23:48:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.921+0000 7f0dd7fff700 1 -- 172.21.15.46:0/4044730068 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ddc00b9c0 con 0x7f0de006b3c0 2024-01-26T23:48:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.921+0000 7f0dd7fff700 1 -- 172.21.15.46:0/4044730068 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc00bd80 con 0x7f0de006b3c0 2024-01-26T23:48:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.922+0000 7f0de73c3700 1 -- 172.21.15.46:0/4044730068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0de006b3c0 msgr2=0x7f0de00af080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.922+0000 7f0de73c3700 1 --2- 172.21.15.46:0/4044730068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0de006b3c0 0x7f0de00af080 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f0de00649b0 tx=0x7f0ddc009250 comp rx=0 tx=0).stop 2024-01-26T23:48:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.922+0000 7f0de73c3700 1 -- 172.21.15.46:0/4044730068 shutdown_connections 2024-01-26T23:48:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.922+0000 7f0de73c3700 1 --2- 172.21.15.46:0/4044730068 >> v2:172.21.15.106:3300/0 conn(0x7f0de00af5c0 0x7f0de0067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.922+0000 7f0de73c3700 1 --2- 172.21.15.46:0/4044730068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0de0157750 0x7f0de0157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.922+0000 7f0de73c3700 1 --2- 172.21.15.46:0/4044730068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0de006b3c0 0x7f0de00af080 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.922+0000 7f0de73c3700 1 -- 172.21.15.46:0/4044730068 >> 172.21.15.46:0/4044730068 conn(0x7f0de00cdf80 msgr2=0x7f0de006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:04.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.922+0000 7f0de73c3700 1 -- 172.21.15.46:0/4044730068 shutdown_connections 2024-01-26T23:48:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.922+0000 7f0de73c3700 1 -- 172.21.15.46:0/4044730068 wait complete. 2024-01-26T23:48:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.924+0000 7f0de73c3700 1 Processor -- start 2024-01-26T23:48:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.924+0000 7f0de73c3700 1 -- start start 2024-01-26T23:48:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.925+0000 7f0de73c3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0de006b3c0 0x7f0de0144020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.925+0000 7f0de73c3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0de00af5c0 0x7f0de00a8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.925+0000 7f0de73c3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0de0157750 0x7f0de00a8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.925+0000 7f0de73c3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de00ac270 con 0x7f0de0157750 2024-01-26T23:48:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.925+0000 7f0de73c3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de00ab660 con 0x7f0de006b3c0 2024-01-26T23:48:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.925+0000 7f0de73c3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de00ab940 con 0x7f0de00af5c0 2024-01-26T23:48:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.925+0000 7f0de5960700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0de0157750 0x7f0de00a8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.925+0000 7f0de5960700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0de0157750 0x7f0de00a8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39582/0 (socket says 172.21.15.46:39582) 2024-01-26T23:48:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.925+0000 7f0de5960700 1 -- 172.21.15.46:0/3029970710 learned_addr learned my addr 172.21.15.46:0/3029970710 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.925+0000 7f0de515f700 1 --2- 172.21.15.46:0/3029970710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0de006b3c0 0x7f0de0144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.925+0000 7f0de495e700 1 --2- 172.21.15.46:0/3029970710 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0de00af5c0 0x7f0de00a8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.925+0000 7f0de5960700 1 -- 172.21.15.46:0/3029970710 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0de00af5c0 msgr2=0x7f0de00a8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.925+0000 7f0de5960700 1 --2- 172.21.15.46:0/3029970710 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0de00af5c0 0x7f0de00a8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.925+0000 7f0de5960700 1 -- 172.21.15.46:0/3029970710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0de006b3c0 msgr2=0x7f0de0144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.925+0000 7f0de5960700 1 --2- 172.21.15.46:0/3029970710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0de006b3c0 0x7f0de0144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.926+0000 7f0de5960700 1 -- 172.21.15.46:0/3029970710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ddc0094f0 con 0x7f0de0157750 2024-01-26T23:48:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.926+0000 7f0de5960700 1 --2- 172.21.15.46:0/3029970710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0de0157750 0x7f0de00a8d50 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8007f20 tx=0x7f0dd8009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.926+0000 7f0dd57fa700 1 -- 172.21.15.46:0/3029970710 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd800eca0 con 0x7f0de0157750 2024-01-26T23:48:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.926+0000 7f0dd57fa700 1 -- 172.21.15.46:0/3029970710 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd800ee20 con 0x7f0de0157750 2024-01-26T23:48:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.926+0000 7f0de73c3700 1 -- 172.21.15.46:0/3029970710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0de00abc20 con 0x7f0de0157750 2024-01-26T23:48:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.926+0000 7f0dd57fa700 1 -- 172.21.15.46:0/3029970710 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8014670 con 0x7f0de0157750 2024-01-26T23:48:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.926+0000 7f0de73c3700 1 -- 172.21.15.46:0/3029970710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de0144560 con 0x7f0de0157750 2024-01-26T23:48:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.927+0000 7f0de73c3700 1 -- 172.21.15.46:0/3029970710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de00a5e20 con 0x7f0de0157750 2024-01-26T23:48:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.927+0000 7f0dd57fa700 1 -- 172.21.15.46:0/3029970710 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0dd80149e0 con 0x7f0de0157750 2024-01-26T23:48:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.928+0000 7f0dd57fa700 1 --2- 172.21.15.46:0/3029970710 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0dc4066f30 0x7f0dc40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:04.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.928+0000 7f0dd57fa700 1 -- 172.21.15.46:0/3029970710 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f0dd8084e80 con 0x7f0de0157750 2024-01-26T23:48:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.928+0000 7f0de515f700 1 --2- 172.21.15.46:0/3029970710 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0dc4066f30 0x7f0dc40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.929+0000 7f0de515f700 1 --2- 172.21.15.46:0/3029970710 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0dc4066f30 0x7f0dc40693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0006fd0 tx=0x7f0dd0008040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:04.933+0000 7f0dd57fa700 1 -- 172.21.15.46:0/3029970710 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0de00a5e20 con 0x7f0de0157750 2024-01-26T23:48:05.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.137+0000 7f0de73c3700 1 -- 172.21.15.46:0/3029970710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0de00a5e20 con 0x7f0de0157750 2024-01-26T23:48:05.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.142+0000 7f0dd57fa700 1 -- 172.21.15.46:0/3029970710 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0de00a5e20 con 0x7f0de0157750 2024-01-26T23:48:05.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.144+0000 7f0de73c3700 1 -- 172.21.15.46:0/3029970710 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0dc4066f30 msgr2=0x7f0dc40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:05.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.144+0000 7f0de73c3700 1 --2- 172.21.15.46:0/3029970710 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0dc4066f30 0x7f0dc40693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0006fd0 tx=0x7f0dd0008040 comp rx=0 tx=0).stop 2024-01-26T23:48:05.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.144+0000 7f0de73c3700 1 -- 172.21.15.46:0/3029970710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0de0157750 msgr2=0x7f0de00a8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:05.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.144+0000 7f0de73c3700 1 --2- 172.21.15.46:0/3029970710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0de0157750 0x7f0de00a8d50 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8007f20 tx=0x7f0dd8009860 comp rx=0 tx=0).stop 2024-01-26T23:48:05.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.144+0000 7f0de73c3700 1 -- 172.21.15.46:0/3029970710 shutdown_connections 2024-01-26T23:48:05.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.144+0000 7f0de73c3700 1 --2- 172.21.15.46:0/3029970710 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0dc4066f30 0x7f0dc40693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.144+0000 7f0de73c3700 1 --2- 172.21.15.46:0/3029970710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0de0157750 0x7f0de00a8d50 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.144+0000 7f0de73c3700 1 --2- 172.21.15.46:0/3029970710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0de006b3c0 0x7f0de0144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.144+0000 7f0de73c3700 1 --2- 172.21.15.46:0/3029970710 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0de00af5c0 0x7f0de00a8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.144+0000 7f0de73c3700 1 -- 172.21.15.46:0/3029970710 >> 172.21.15.46:0/3029970710 conn(0x7f0de00cdf80 msgr2=0x7f0de00cb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:05.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.144+0000 7f0de73c3700 1 -- 172.21.15.46:0/3029970710 shutdown_connections 2024-01-26T23:48:05.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.144+0000 7f0de73c3700 1 -- 172.21.15.46:0/3029970710 wait complete. 2024-01-26T23:48:05.159 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-26T23:48:05.160 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:48:05.208 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:48:05.255 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-26T23:48:05.255 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-26T23:48:05.255 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:48:05.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.391+0000 7fc8bd236700 1 Processor -- start 2024-01-26T23:48:05.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.391+0000 7fc8bd236700 1 -- start start 2024-01-26T23:48:05.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8bd236700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc8b8159760 0x7fc8b80b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:05.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8bd236700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8b8159b30 0x7fc8b80b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:05.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8bd236700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc8b815afc0 0x7fc8b8067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:05.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8bd236700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b815abc0 con 0x7fc8b8159760 2024-01-26T23:48:05.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8bd236700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b80b2e90 con 0x7fc8b815afc0 2024-01-26T23:48:05.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8bd236700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b8159fb0 con 0x7fc8b8159b30 2024-01-26T23:48:05.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8b6d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc8b8159760 0x7fc8b80b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:05.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8b6d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc8b8159760 0x7fc8b80b0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39588/0 (socket says 172.21.15.46:39588) 2024-01-26T23:48:05.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8b6d9d700 1 -- 172.21.15.46:0/3686986597 learned_addr learned my addr 172.21.15.46:0/3686986597 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:05.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8b659c700 1 --2- 172.21.15.46:0/3686986597 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8b8159b30 0x7fc8b80b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:05.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8b759e700 1 --2- 172.21.15.46:0/3686986597 >> v2:172.21.15.106:3300/0 conn(0x7fc8b815afc0 0x7fc8b8067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:05.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8b6d9d700 1 -- 172.21.15.46:0/3686986597 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8b8159b30 msgr2=0x7fc8b80b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:05.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8b6d9d700 1 --2- 172.21.15.46:0/3686986597 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8b8159b30 0x7fc8b80b11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8b6d9d700 1 -- 172.21.15.46:0/3686986597 >> v2:172.21.15.106:3300/0 conn(0x7fc8b815afc0 msgr2=0x7fc8b8067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:05.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8b6d9d700 1 --2- 172.21.15.46:0/3686986597 >> v2:172.21.15.106:3300/0 conn(0x7fc8b815afc0 0x7fc8b8067930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8b6d9d700 1 -- 172.21.15.46:0/3686986597 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b815a290 con 0x7fc8b8159760 2024-01-26T23:48:05.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8b659c700 1 --2- 172.21.15.46:0/3686986597 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8b8159b30 0x7fc8b80b11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:48:05.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8b6d9d700 1 --2- 172.21.15.46:0/3686986597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8b8159760 0x7fc8b80b0c70 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8068500 tx=0x7fc8a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945ff1bcfc71fb5f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:05.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.392+0000 7fc8b5d9b700 1 -- 172.21.15.46:0/3686986597 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a800b860 con 0x7fc8b8159760 2024-01-26T23:48:05.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.393+0000 7fc8b5d9b700 1 -- 172.21.15.46:0/3686986597 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a800b9c0 con 0x7fc8b8159760 2024-01-26T23:48:05.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.393+0000 7fc8b5d9b700 1 -- 172.21.15.46:0/3686986597 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a800bd80 con 0x7fc8b8159760 2024-01-26T23:48:05.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.393+0000 7fc8bd236700 1 -- 172.21.15.46:0/3686986597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8b8159760 msgr2=0x7fc8b80b0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:05.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.393+0000 7fc8bd236700 1 --2- 172.21.15.46:0/3686986597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8b8159760 0x7fc8b80b0c70 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8068500 tx=0x7fc8a8009250 comp rx=0 tx=0).stop 2024-01-26T23:48:05.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.393+0000 7fc8bd236700 1 -- 172.21.15.46:0/3686986597 shutdown_connections 2024-01-26T23:48:05.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.393+0000 7fc8bd236700 1 --2- 172.21.15.46:0/3686986597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8b8159760 0x7fc8b80b0c70 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.393+0000 7fc8bd236700 1 --2- 172.21.15.46:0/3686986597 >> v2:172.21.15.106:3300/0 conn(0x7fc8b815afc0 0x7fc8b8067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.393+0000 7fc8bd236700 1 --2- 172.21.15.46:0/3686986597 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8b8159b30 0x7fc8b80b11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.393+0000 7fc8bd236700 1 -- 172.21.15.46:0/3686986597 >> 172.21.15.46:0/3686986597 conn(0x7fc8b806b630 msgr2=0x7fc8b80aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:05.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.393+0000 7fc8bd236700 1 -- 172.21.15.46:0/3686986597 shutdown_connections 2024-01-26T23:48:05.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.393+0000 7fc8bd236700 1 -- 172.21.15.46:0/3686986597 wait complete. 2024-01-26T23:48:05.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.395+0000 7fc8bd236700 1 Processor -- start 2024-01-26T23:48:05.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.395+0000 7fc8bd236700 1 -- start start 2024-01-26T23:48:05.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.396+0000 7fc8bd236700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8b8159760 0x7fc8b814f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:05.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.396+0000 7fc8bd236700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8b815afc0 0x7fc8b80ad710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:05.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.396+0000 7fc8bd236700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8b814b900 0x7fc8b8155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:05.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.396+0000 7fc8bd236700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b80adf80 con 0x7fc8b815afc0 2024-01-26T23:48:05.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.396+0000 7fc8bd236700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b80ae540 con 0x7fc8b8159760 2024-01-26T23:48:05.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.396+0000 7fc8bd236700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b80af980 con 0x7fc8b814b900 2024-01-26T23:48:05.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.396+0000 7fc8b659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8b815afc0 0x7fc8b80ad710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:05.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.396+0000 7fc8b659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8b815afc0 0x7fc8b80ad710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39592/0 (socket says 172.21.15.46:39592) 2024-01-26T23:48:05.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.396+0000 7fc8b659c700 1 -- 172.21.15.46:0/3291089220 learned_addr learned my addr 172.21.15.46:0/3291089220 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:05.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.396+0000 7fc8b759e700 1 --2- 172.21.15.46:0/3291089220 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8b814b900 0x7fc8b8155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:05.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.396+0000 7fc8b6d9d700 1 --2- 172.21.15.46:0/3291089220 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8b8159760 0x7fc8b814f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:05.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.397+0000 7fc8b659c700 1 -- 172.21.15.46:0/3291089220 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8b814b900 msgr2=0x7fc8b8155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:05.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.397+0000 7fc8b659c700 1 --2- 172.21.15.46:0/3291089220 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8b814b900 0x7fc8b8155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.397+0000 7fc8b659c700 1 -- 172.21.15.46:0/3291089220 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8b8159760 msgr2=0x7fc8b814f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:05.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.397+0000 7fc8b659c700 1 --2- 172.21.15.46:0/3291089220 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8b8159760 0x7fc8b814f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.397+0000 7fc8b659c700 1 -- 172.21.15.46:0/3291089220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a80094f0 con 0x7fc8b815afc0 2024-01-26T23:48:05.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.397+0000 7fc8b659c700 1 --2- 172.21.15.46:0/3291089220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8b815afc0 0x7fc8b80ad710 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac00c860 tx=0x7fc8ac008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:05.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.397+0000 7fc89effd700 1 -- 172.21.15.46:0/3291089220 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac011600 con 0x7fc8b815afc0 2024-01-26T23:48:05.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.397+0000 7fc89effd700 1 -- 172.21.15.46:0/3291089220 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ac011760 con 0x7fc8b815afc0 2024-01-26T23:48:05.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.397+0000 7fc8bd236700 1 -- 172.21.15.46:0/3291089220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b815dd70 con 0x7fc8b815afc0 2024-01-26T23:48:05.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.397+0000 7fc89effd700 1 -- 172.21.15.46:0/3291089220 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac011b00 con 0x7fc8b815afc0 2024-01-26T23:48:05.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.397+0000 7fc8bd236700 1 -- 172.21.15.46:0/3291089220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b815e2f0 con 0x7fc8b815afc0 2024-01-26T23:48:05.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.398+0000 7fc8bd236700 1 -- 172.21.15.46:0/3291089220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b80ae6d0 con 0x7fc8b815afc0 2024-01-26T23:48:05.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.398+0000 7fc89effd700 1 -- 172.21.15.46:0/3291089220 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc8ac00a3e0 con 0x7fc8b815afc0 2024-01-26T23:48:05.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.399+0000 7fc89effd700 1 --2- 172.21.15.46:0/3291089220 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc8a4066f30 0x7fc8a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:05.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.399+0000 7fc89effd700 1 -- 172.21.15.46:0/3291089220 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6253+0+0 (secure 0 0 0) 0x7fc8ac0847e0 con 0x7fc8b815afc0 2024-01-26T23:48:05.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.399+0000 7fc8b6d9d700 1 --2- 172.21.15.46:0/3291089220 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc8a4066f30 0x7fc8a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:05.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.400+0000 7fc8b6d9d700 1 --2- 172.21.15.46:0/3291089220 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc8a4066f30 0x7fc8a40693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8068500 tx=0x7fc8a8017000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:05.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.403+0000 7fc89effd700 1 -- 172.21.15.46:0/3291089220 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc8ac051830 con 0x7fc8b815afc0 2024-01-26T23:48:05.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.655+0000 7fc8bd236700 1 -- 172.21.15.46:0/3291089220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8b815a570 con 0x7fc8b815afc0 2024-01-26T23:48:05.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.655+0000 7fc89effd700 1 -- 172.21.15.46:0/3291089220 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+3529 (secure 0 0 0) 0x7fc8ac015de0 con 0x7fc8b815afc0 2024-01-26T23:48:05.657 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:48:05.657 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":56,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6518,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/4188139264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":4188139264},{"type":"v1","addr":"172.21.15.106:6836","nonce":4188139264}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":48},{"gid":6527,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/221301328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":221301328},{"type":"v1","addr":"172.21.15.46:6833","nonce":221301328}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":6530,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1531399912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":1531399912},{"type":"v1","addr":"172.21.15.46:6835","nonce":1531399912}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":6827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/744210265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":744210265},{"type":"v1","addr":"172.21.15.106:6833","nonce":744210265}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":56}],"filesystems":[]} 2024-01-26T23:48:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.657+0000 7fc8bd236700 1 -- 172.21.15.46:0/3291089220 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc8a4066f30 msgr2=0x7fc8a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.657+0000 7fc8bd236700 1 --2- 172.21.15.46:0/3291089220 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc8a4066f30 0x7fc8a40693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8068500 tx=0x7fc8a8017000 comp rx=0 tx=0).stop 2024-01-26T23:48:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.657+0000 7fc8bd236700 1 -- 172.21.15.46:0/3291089220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8b815afc0 msgr2=0x7fc8b80ad710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.657+0000 7fc8bd236700 1 --2- 172.21.15.46:0/3291089220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8b815afc0 0x7fc8b80ad710 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac00c860 tx=0x7fc8ac008040 comp rx=0 tx=0).stop 2024-01-26T23:48:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.658+0000 7fc8bd236700 1 -- 172.21.15.46:0/3291089220 shutdown_connections 2024-01-26T23:48:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.658+0000 7fc8bd236700 1 --2- 172.21.15.46:0/3291089220 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc8a4066f30 0x7fc8a40693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.658+0000 7fc8bd236700 1 --2- 172.21.15.46:0/3291089220 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc8b814b900 0x7fc8b8155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.658+0000 7fc8bd236700 1 --2- 172.21.15.46:0/3291089220 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8b8159760 0x7fc8b814f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.658+0000 7fc8bd236700 1 --2- 172.21.15.46:0/3291089220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8b815afc0 0x7fc8b80ad710 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.658+0000 7fc8bd236700 1 -- 172.21.15.46:0/3291089220 >> 172.21.15.46:0/3291089220 conn(0x7fc8b806b630 msgr2=0x7fc8b80b0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.658+0000 7fc8bd236700 1 -- 172.21.15.46:0/3291089220 shutdown_connections 2024-01-26T23:48:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.658+0000 7fc8bd236700 1 -- 172.21.15.46:0/3291089220 wait complete. 2024-01-26T23:48:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 56 2024-01-26T23:48:05.673 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-26T23:48:05.674 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-26T23:48:05.674 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-26T23:48:05.674 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:48:05.674 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-26T23:48:05.674 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-26T23:48:05.675 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-26T23:48:05.675 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:48:05.675 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-26T23:48:05.675 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-26T23:48:05.675 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-26T23:48:05.675 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:48:05.676 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-26T23:48:05.676 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-26T23:48:05.677 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-26T23:48:05.677 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:48:05.684 INFO:tasks.ceph.mds.a:Stopped 2024-01-26T23:48:05.684 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-26T23:48:05.685 INFO:tasks.ceph.mds.c:Stopped 2024-01-26T23:48:05.685 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-26T23:48:05.687 INFO:tasks.ceph.mds.b:Stopped 2024-01-26T23:48:05.687 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-26T23:48:05.688 INFO:tasks.ceph.mds.d:Stopped 2024-01-26T23:48:05.689 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-26T23:48:05.727 INFO:tasks.ceph.mds.a:Started 2024-01-26T23:48:05.727 DEBUG:teuthology.parallel:result is None 2024-01-26T23:48:05.729 INFO:tasks.ceph.mds.c:Started 2024-01-26T23:48:05.729 DEBUG:teuthology.parallel:result is None 2024-01-26T23:48:05.730 INFO:tasks.ceph.mds.b:Started 2024-01-26T23:48:05.731 DEBUG:teuthology.parallel:result is None 2024-01-26T23:48:05.731 INFO:tasks.ceph.mds.d:Started 2024-01-26T23:48:05.731 DEBUG:teuthology.parallel:result is None 2024-01-26T23:48:05.731 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-26T23:48:05.816 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-26T23:48:05.816 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-26T23:48:05.820 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-01-26T23:48:05.820 INFO:tasks.ceph.mds.b.smithi106.stdout:starting mds.b at 2024-01-26T23:48:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.872+0000 7f97f1820700 1 Processor -- start 2024-01-26T23:48:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.872+0000 7f97f1820700 1 -- start start 2024-01-26T23:48:05.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.873+0000 7f97f1820700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97ec0b1d80 0x7f97ec0ae800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.873+0000 7f97f1820700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ec0cbf30 con 0x7f97ec0adbe0 2024-01-26T23:48:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.873+0000 7f97f1820700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ec0cb600 con 0x7f97ec0ad810 2024-01-26T23:48:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.873+0000 7f97f1820700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec0aeea0 con 0x7f97ec0b1d80 2024-01-26T23:48:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.873+0000 7f97ea7fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f97ec0adbe0 0x7f97ec0b1670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34428/0 (socket says 172.21.15.46:34428) 2024-01-26T23:48:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.873+0000 7f97ea7fc700 1 -- 172.21.15.46:0/308560935 learned_addr learned my addr 172.21.15.46:0/308560935 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.873+0000 7f97eb7fe700 1 --2- 172.21.15.46:0/308560935 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97ec0b1d80 0x7f97ec0ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.874+0000 7f97e9ffb700 1 -- 172.21.15.46:0/308560935 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966291559 0 0) 0x7f97ec0cbf30 con 0x7f97ec0adbe0 2024-01-26T23:48:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.874+0000 7f97e9ffb700 1 -- 172.21.15.46:0/308560935 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97d8003570 con 0x7f97ec0adbe0 2024-01-26T23:48:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.874+0000 7f97e9ffb700 1 -- 172.21.15.46:0/308560935 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692148811 0 0) 0x7f97ec0cb600 con 0x7f97ec0ad810 2024-01-26T23:48:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.874+0000 7f97e9ffb700 1 -- 172.21.15.46:0/308560935 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97ec0cbf30 con 0x7f97ec0ad810 2024-01-26T23:48:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.874+0000 7f97eb7fe700 1 -- 172.21.15.46:0/308560935 >> v1:172.21.15.106:6789/0 conn(0x7f97ec0ad810 legacy=0x7f97ec0c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.874+0000 7f97eb7fe700 1 -- 172.21.15.46:0/308560935 >> v1:172.21.15.46:6789/0 conn(0x7f97ec0adbe0 legacy=0x7f97ec0b1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.874+0000 7f97eb7fe700 1 -- 172.21.15.46:0/308560935 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec0cb8e0 con 0x7f97ec0b1d80 2024-01-26T23:48:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.874+0000 7f97eb7fe700 1 --2- 172.21.15.46:0/308560935 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97ec0b1d80 0x7f97ec0ae800 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009e90 tx=0x7f97e4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89033d62435b3c69 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.874+0000 7f97e9ffb700 1 -- 172.21.15.46:0/308560935 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e400e070 con 0x7f97ec0b1d80 2024-01-26T23:48:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.874+0000 7f97e9ffb700 1 -- 172.21.15.46:0/308560935 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e400bb80 con 0x7f97ec0b1d80 2024-01-26T23:48:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.874+0000 7f97e9ffb700 1 -- 172.21.15.46:0/308560935 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e4012430 con 0x7f97ec0b1d80 2024-01-26T23:48:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.875+0000 7f97f1820700 1 -- 172.21.15.46:0/308560935 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97ec0b1d80 msgr2=0x7f97ec0ae800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.875+0000 7f97f1820700 1 --2- 172.21.15.46:0/308560935 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97ec0b1d80 0x7f97ec0ae800 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009e90 tx=0x7f97e4009780 comp rx=0 tx=0).stop 2024-01-26T23:48:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.875+0000 7f97f1820700 1 -- 172.21.15.46:0/308560935 shutdown_connections 2024-01-26T23:48:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.875+0000 7f97f1820700 1 --2- 172.21.15.46:0/308560935 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97ec0b1d80 0x7f97ec0ae800 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.875+0000 7f97f1820700 1 -- 172.21.15.46:0/308560935 >> 172.21.15.46:0/308560935 conn(0x7f97ec068400 msgr2=0x7f97ec0cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.875+0000 7f97f1820700 1 -- 172.21.15.46:0/308560935 shutdown_connections 2024-01-26T23:48:05.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.875+0000 7f97f1820700 1 -- 172.21.15.46:0/308560935 wait complete. 2024-01-26T23:48:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.876+0000 7f97f1820700 1 Processor -- start 2024-01-26T23:48:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.876+0000 7f97f1820700 1 -- start start 2024-01-26T23:48:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.877+0000 7f97f1820700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97ec0ad810 0x7f97ec0c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.877+0000 7f97f1820700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97ec0adbe0 0x7f97ec140960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.877+0000 7f97f1820700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97ec0b1d80 0x7f97ec15c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.877+0000 7f97f1820700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec1419f0 con 0x7f97ec0ad810 2024-01-26T23:48:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.877+0000 7f97f1820700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec141cd0 con 0x7f97ec0b1d80 2024-01-26T23:48:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.877+0000 7f97f1820700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec141fb0 con 0x7f97ec0adbe0 2024-01-26T23:48:05.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.877+0000 7f97eaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97ec0ad810 0x7f97ec0c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.877+0000 7f97eaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97ec0ad810 0x7f97ec0c87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39660/0 (socket says 172.21.15.46:39660) 2024-01-26T23:48:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.877+0000 7f97eaffd700 1 -- 172.21.15.46:0/3691123636 learned_addr learned my addr 172.21.15.46:0/3691123636 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.877+0000 7f97eb7fe700 1 --2- 172.21.15.46:0/3691123636 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97ec0b1d80 0x7f97ec15c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.877+0000 7f97ea7fc700 1 --2- 172.21.15.46:0/3691123636 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97ec0adbe0 0x7f97ec140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.878+0000 7f97eaffd700 1 -- 172.21.15.46:0/3691123636 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97ec0adbe0 msgr2=0x7f97ec140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.878+0000 7f97eaffd700 1 --2- 172.21.15.46:0/3691123636 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97ec0adbe0 0x7f97ec140960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.878+0000 7f97eaffd700 1 -- 172.21.15.46:0/3691123636 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97ec0b1d80 msgr2=0x7f97ec15c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.878+0000 7f97eaffd700 1 --2- 172.21.15.46:0/3691123636 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97ec0b1d80 0x7f97ec15c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.878+0000 7f97eaffd700 1 -- 172.21.15.46:0/3691123636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e4009a20 con 0x7f97ec0ad810 2024-01-26T23:48:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.878+0000 7f97eaffd700 1 --2- 172.21.15.46:0/3691123636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97ec0ad810 0x7f97ec0c87b0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f97d400c7c0 tx=0x7f97d4007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.878+0000 7f97e2ffd700 1 -- 172.21.15.46:0/3691123636 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d4016070 con 0x7f97ec0ad810 2024-01-26T23:48:05.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.878+0000 7f97e2ffd700 1 -- 172.21.15.46:0/3691123636 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d4011680 con 0x7f97ec0ad810 2024-01-26T23:48:05.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.878+0000 7f97f1820700 1 -- 172.21.15.46:0/3691123636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ec142290 con 0x7f97ec0ad810 2024-01-26T23:48:05.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.878+0000 7f97e2ffd700 1 -- 172.21.15.46:0/3691123636 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d4011a20 con 0x7f97ec0ad810 2024-01-26T23:48:05.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.878+0000 7f97f1820700 1 -- 172.21.15.46:0/3691123636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec0c8cf0 con 0x7f97ec0ad810 2024-01-26T23:48:05.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.879+0000 7f97f1820700 1 -- 172.21.15.46:0/3691123636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ec1427b0 con 0x7f97ec0ad810 2024-01-26T23:48:05.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.880+0000 7f97e2ffd700 1 -- 172.21.15.46:0/3691123636 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f97d4012440 con 0x7f97ec0ad810 2024-01-26T23:48:05.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.880+0000 7f97e2ffd700 1 --2- 172.21.15.46:0/3691123636 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f97cc066f00 0x7f97cc0693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:05.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.880+0000 7f97e2ffd700 1 -- 172.21.15.46:0/3691123636 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f97d40845a0 con 0x7f97ec0ad810 2024-01-26T23:48:05.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.880+0000 7f97ea7fc700 1 --2- 172.21.15.46:0/3691123636 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f97cc066f00 0x7f97cc0693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:05.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.881+0000 7f97ea7fc700 1 --2- 172.21.15.46:0/3691123636 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f97cc066f00 0x7f97cc0693b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97dc0058d0 tx=0x7f97dc009040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:05.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:05.885+0000 7f97e2ffd700 1 -- 172.21.15.46:0/3691123636 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f97d40515f0 con 0x7f97ec0ad810 2024-01-26T23:48:06.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:06.107+0000 7f97f1820700 1 -- 172.21.15.46:0/3691123636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f97ec04c710 con 0x7f97ec0ad810 2024-01-26T23:48:07.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.699+0000 7f97e2ffd700 1 -- 172.21.15.46:0/3691123636 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v126) v1 ==== 97+0+0 (secure 0 0 0) 0x7f97d4054c10 con 0x7f97ec0ad810 2024-01-26T23:48:07.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.702+0000 7f97f1820700 1 -- 172.21.15.46:0/3691123636 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f97cc066f00 msgr2=0x7f97cc0693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.702+0000 7f97f1820700 1 --2- 172.21.15.46:0/3691123636 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f97cc066f00 0x7f97cc0693b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97dc0058d0 tx=0x7f97dc009040 comp rx=0 tx=0).stop 2024-01-26T23:48:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.702+0000 7f97f1820700 1 -- 172.21.15.46:0/3691123636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97ec0ad810 msgr2=0x7f97ec0c87b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.702+0000 7f97f1820700 1 --2- 172.21.15.46:0/3691123636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97ec0ad810 0x7f97ec0c87b0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f97d400c7c0 tx=0x7f97d4007dd0 comp rx=0 tx=0).stop 2024-01-26T23:48:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.702+0000 7f97f1820700 1 -- 172.21.15.46:0/3691123636 shutdown_connections 2024-01-26T23:48:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.702+0000 7f97f1820700 1 --2- 172.21.15.46:0/3691123636 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f97cc066f00 0x7f97cc0693b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.702+0000 7f97f1820700 1 --2- 172.21.15.46:0/3691123636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97ec0ad810 0x7f97ec0c87b0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.702+0000 7f97f1820700 1 --2- 172.21.15.46:0/3691123636 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97ec0b1d80 0x7f97ec15c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.702+0000 7f97f1820700 1 --2- 172.21.15.46:0/3691123636 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97ec0adbe0 0x7f97ec140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.702+0000 7f97f1820700 1 -- 172.21.15.46:0/3691123636 >> 172.21.15.46:0/3691123636 conn(0x7f97ec068400 msgr2=0x7f97ec158600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.702+0000 7f97f1820700 1 -- 172.21.15.46:0/3691123636 shutdown_connections 2024-01-26T23:48:07.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.702+0000 7f97f1820700 1 -- 172.21.15.46:0/3691123636 wait complete. 2024-01-26T23:48:07.705 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-01-26T23:48:07.719 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-26T23:48:07.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.857+0000 7ff3b8a06700 1 Processor -- start 2024-01-26T23:48:07.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.857+0000 7ff3b8a06700 1 -- start start 2024-01-26T23:48:07.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.858+0000 7ff3b8a06700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff3b40b3bf0 0x7ff3b40b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:07.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.858+0000 7ff3b8a06700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3b406d310 0x7ff3b406c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:07.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.858+0000 7ff3b8a06700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b415b200 0x7ff3b40c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:07.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.858+0000 7ff3b8a06700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b4157d00 con 0x7ff3b406d310 2024-01-26T23:48:07.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.858+0000 7ff3b8a06700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b406c7d0 con 0x7ff3b40b3bf0 2024-01-26T23:48:07.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.858+0000 7ff3b8a06700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b406d840 con 0x7ff3b415b200 2024-01-26T23:48:07.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.858+0000 7ff3b1d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3b406d310 0x7ff3b406c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:07.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.858+0000 7ff3b1d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3b406d310 0x7ff3b406c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39666/0 (socket says 172.21.15.46:39666) 2024-01-26T23:48:07.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.858+0000 7ff3b1d9b700 1 -- 172.21.15.46:0/3638140690 learned_addr learned my addr 172.21.15.46:0/3638140690 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:07.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.858+0000 7ff3b2d9d700 1 --2- 172.21.15.46:0/3638140690 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b415b200 0x7ff3b40c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:07.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.858+0000 7ff3b259c700 1 --2- 172.21.15.46:0/3638140690 >> v2:172.21.15.106:3300/0 conn(0x7ff3b40b3bf0 0x7ff3b40b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:07.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.858+0000 7ff3b1d9b700 1 -- 172.21.15.46:0/3638140690 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b415b200 msgr2=0x7ff3b40c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:07.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.858+0000 7ff3b1d9b700 1 --2- 172.21.15.46:0/3638140690 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b415b200 0x7ff3b40c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:07.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.858+0000 7ff3b1d9b700 1 -- 172.21.15.46:0/3638140690 >> v2:172.21.15.106:3300/0 conn(0x7ff3b40b3bf0 msgr2=0x7ff3b40b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:07.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.858+0000 7ff3b1d9b700 1 --2- 172.21.15.46:0/3638140690 >> v2:172.21.15.106:3300/0 conn(0x7ff3b40b3bf0 0x7ff3b40b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:07.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.858+0000 7ff3b1d9b700 1 -- 172.21.15.46:0/3638140690 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b406db20 con 0x7ff3b406d310 2024-01-26T23:48:07.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.858+0000 7ff3b1d9b700 1 --2- 172.21.15.46:0/3638140690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3b406d310 0x7ff3b406c290 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7ff3b406b900 tx=0x7ff3a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b598a07f985c515 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:07.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.859+0000 7ff3b159a700 1 -- 172.21.15.46:0/3638140690 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a800b640 con 0x7ff3b406d310 2024-01-26T23:48:07.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.859+0000 7ff3b159a700 1 -- 172.21.15.46:0/3638140690 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a800b7a0 con 0x7ff3b406d310 2024-01-26T23:48:07.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.859+0000 7ff3b159a700 1 -- 172.21.15.46:0/3638140690 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a800bb60 con 0x7ff3b406d310 2024-01-26T23:48:07.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.859+0000 7ff3b8a06700 1 -- 172.21.15.46:0/3638140690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3b406d310 msgr2=0x7ff3b406c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:07.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.859+0000 7ff3b8a06700 1 --2- 172.21.15.46:0/3638140690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3b406d310 0x7ff3b406c290 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7ff3b406b900 tx=0x7ff3a8009250 comp rx=0 tx=0).stop 2024-01-26T23:48:07.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.859+0000 7ff3b8a06700 1 -- 172.21.15.46:0/3638140690 shutdown_connections 2024-01-26T23:48:07.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.859+0000 7ff3b8a06700 1 --2- 172.21.15.46:0/3638140690 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b415b200 0x7ff3b40c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:07.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.859+0000 7ff3b8a06700 1 --2- 172.21.15.46:0/3638140690 >> v2:172.21.15.106:3300/0 conn(0x7ff3b40b3bf0 0x7ff3b40b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:07.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.859+0000 7ff3b8a06700 1 --2- 172.21.15.46:0/3638140690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3b406d310 0x7ff3b406c290 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:07.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.859+0000 7ff3b8a06700 1 -- 172.21.15.46:0/3638140690 >> 172.21.15.46:0/3638140690 conn(0x7ff3b40ce530 msgr2=0x7ff3b40ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:07.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.859+0000 7ff3b8a06700 1 -- 172.21.15.46:0/3638140690 shutdown_connections 2024-01-26T23:48:07.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.859+0000 7ff3b8a06700 1 -- 172.21.15.46:0/3638140690 wait complete. 2024-01-26T23:48:07.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.860+0000 7ff3b8a06700 1 Processor -- start 2024-01-26T23:48:07.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.861+0000 7ff3b8a06700 1 -- start start 2024-01-26T23:48:07.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.861+0000 7ff3b8a06700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3b406d310 0x7ff3b40c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:07.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.861+0000 7ff3b8a06700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b40b3bf0 0x7ff3b41592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:07.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.861+0000 7ff3b8a06700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3b415b200 0x7ff3b40c1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:07.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.861+0000 7ff3b8a06700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b40c67e0 con 0x7ff3b406d310 2024-01-26T23:48:07.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.861+0000 7ff3b8a06700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b40c6a80 con 0x7ff3b415b200 2024-01-26T23:48:07.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.861+0000 7ff3b8a06700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b40c6d60 con 0x7ff3b40b3bf0 2024-01-26T23:48:07.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.861+0000 7ff3b259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3b406d310 0x7ff3b40c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:07.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.861+0000 7ff3b259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3b406d310 0x7ff3b40c7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39668/0 (socket says 172.21.15.46:39668) 2024-01-26T23:48:07.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.861+0000 7ff3b259c700 1 -- 172.21.15.46:0/4010776898 learned_addr learned my addr 172.21.15.46:0/4010776898 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:07.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.862+0000 7ff3b2d9d700 1 --2- 172.21.15.46:0/4010776898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3b415b200 0x7ff3b40c1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:07.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.862+0000 7ff3b1d9b700 1 --2- 172.21.15.46:0/4010776898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b40b3bf0 0x7ff3b41592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:07.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.862+0000 7ff3b259c700 1 -- 172.21.15.46:0/4010776898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b40b3bf0 msgr2=0x7ff3b41592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:07.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.862+0000 7ff3b259c700 1 --2- 172.21.15.46:0/4010776898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b40b3bf0 0x7ff3b41592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:07.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.862+0000 7ff3b259c700 1 -- 172.21.15.46:0/4010776898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3b415b200 msgr2=0x7ff3b40c1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:07.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.862+0000 7ff3b259c700 1 --2- 172.21.15.46:0/4010776898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3b415b200 0x7ff3b40c1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:07.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.862+0000 7ff3b259c700 1 -- 172.21.15.46:0/4010776898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a80094f0 con 0x7ff3b406d310 2024-01-26T23:48:07.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.862+0000 7ff3b1d9b700 1 --2- 172.21.15.46:0/4010776898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b40b3bf0 0x7ff3b41592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:07.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.862+0000 7ff3b259c700 1 --2- 172.21.15.46:0/4010776898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3b406d310 0x7ff3b40c7840 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7ff3a400e930 tx=0x7ff3a40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:07.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.862+0000 7ff3a27fc700 1 -- 172.21.15.46:0/4010776898 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4013680 con 0x7ff3b406d310 2024-01-26T23:48:07.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.862+0000 7ff3a27fc700 1 -- 172.21.15.46:0/4010776898 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a40137e0 con 0x7ff3b406d310 2024-01-26T23:48:07.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.862+0000 7ff3b8a06700 1 -- 172.21.15.46:0/4010776898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b40c7040 con 0x7ff3b406d310 2024-01-26T23:48:07.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.862+0000 7ff3a27fc700 1 -- 172.21.15.46:0/4010776898 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4013b80 con 0x7ff3b406d310 2024-01-26T23:48:07.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.862+0000 7ff3b8a06700 1 -- 172.21.15.46:0/4010776898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b40c4c70 con 0x7ff3b406d310 2024-01-26T23:48:07.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.863+0000 7ff3b8a06700 1 -- 172.21.15.46:0/4010776898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b404c710 con 0x7ff3b406d310 2024-01-26T23:48:07.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.864+0000 7ff3a27fc700 1 -- 172.21.15.46:0/4010776898 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff3a400c460 con 0x7ff3b406d310 2024-01-26T23:48:07.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.865+0000 7ff3a27fc700 1 --2- 172.21.15.46:0/4010776898 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff394066f30 0x7ff3940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:07.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.865+0000 7ff3a27fc700 1 -- 172.21.15.46:0/4010776898 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5565+0+0 (secure 0 0 0) 0x7ff3a4084d90 con 0x7ff3b406d310 2024-01-26T23:48:07.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.865+0000 7ff3b1d9b700 1 --2- 172.21.15.46:0/4010776898 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff394066f30 0x7ff3940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:07.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.866+0000 7ff3b1d9b700 1 --2- 172.21.15.46:0/4010776898 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff394066f30 0x7ff3940693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff3b406b900 tx=0x7ff3a8017000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:07.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:07.869+0000 7ff3a27fc700 1 -- 172.21.15.46:0/4010776898 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff3a4052110 con 0x7ff3b406d310 2024-01-26T23:48:08.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.158+0000 7ff3b8a06700 1 -- 172.21.15.46:0/4010776898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff3b40c7690 con 0x7ff3b406d310 2024-01-26T23:48:08.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.158+0000 7ff3a27fc700 1 -- 172.21.15.46:0/4010776898 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ff3a4055730 con 0x7ff3b406d310 2024-01-26T23:48:08.159 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:48:08.160 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:48:08.160 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-01-26T23:48:08.160 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.160 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-01-26T23:48:08.160 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlM6dKKBAAPpeWeDZWrFgc/4zuxKCvHw==", 2024-01-26T23:48:08.160 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.160 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:48:08.160 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:48:08.160 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:48:08.160 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:48:08.160 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.160 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.160 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.161 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-01-26T23:48:08.161 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlmzHRNRAAflWYZ/p4AXeMy54tvLpugA==", 2024-01-26T23:48:08.161 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.161 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:48:08.161 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:48:08.161 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:48:08.161 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:48:08.161 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.161 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.161 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.161 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-01-26T23:48:08.161 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlMZxoMBAAG5kyfm+LLG86y/p2A8BfDQ==", 2024-01-26T23:48:08.161 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.161 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:48:08.162 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:48:08.162 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:48:08.162 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:48:08.162 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.162 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.162 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.162 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-01-26T23:48:08.162 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl5jx/ARAA3cwtLAOWQ/n2NG+xwEZ3QQ==", 2024-01-26T23:48:08.162 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.162 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:48:08.162 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:48:08.162 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:48:08.162 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:48:08.162 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.162 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.163 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.163 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-01-26T23:48:08.163 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4QLRlJ2OKJRAAWVhxEdQkUW20294w/KLL6g==", 2024-01-26T23:48:08.163 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.163 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:48:08.163 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:48:08.163 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:48:08.163 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.163 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.163 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.163 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-01-26T23:48:08.163 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7QLRlE2tAJRAAzNlTiCr6a2eWiabRy1SqSw==", 2024-01-26T23:48:08.163 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.163 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:48:08.164 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:48:08.164 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:48:08.164 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.164 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.164 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.164 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-01-26T23:48:08.164 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+QLRlyySaJBAA5072ba5HUNi2dYE3D02M/Q==", 2024-01-26T23:48:08.164 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.164 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:48:08.164 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:48:08.164 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:48:08.164 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.164 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.164 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.165 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-01-26T23:48:08.165 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBQLRl9+tAIxAApZk0btXIpQPWSb3BYUYsFQ==", 2024-01-26T23:48:08.165 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.165 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:48:08.165 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:48:08.165 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:48:08.165 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.165 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.165 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.165 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-01-26T23:48:08.165 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHQLRlurUtJxAAT9fwKrAWl62VOXwShg7NJg==", 2024-01-26T23:48:08.165 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.165 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:48:08.165 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:48:08.165 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:48:08.166 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.166 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.166 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.166 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-01-26T23:48:08.166 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKQLRl7aqoJRAAXu9fykWELsW3SwnahoIOLA==", 2024-01-26T23:48:08.166 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.166 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:48:08.166 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:48:08.166 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:48:08.166 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.166 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.166 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.166 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-01-26T23:48:08.166 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNQLRlWf7pJBAAopmFLIKb8s32YZs46VwbGg==", 2024-01-26T23:48:08.167 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.167 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:48:08.167 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:48:08.167 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:48:08.167 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.167 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.167 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.167 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-01-26T23:48:08.167 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQQLRlNGfhIxAAmCgmOB3QAGfSIYHBb5BUZw==", 2024-01-26T23:48:08.167 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.167 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:48:08.167 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:48:08.167 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:48:08.167 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.168 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.168 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.168 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-01-26T23:48:08.168 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRleO8wBhAA1018AcO3KcyOZxgB0Itxaw==", 2024-01-26T23:48:08.168 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.168 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:48:08.168 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-01-26T23:48:08.168 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-26T23:48:08.168 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.168 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.168 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.168 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-01-26T23:48:08.168 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlsiZEDRAAjXSamHe2bZcDqVsqXqIcGg==", 2024-01-26T23:48:08.168 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.168 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:48:08.169 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:48:08.169 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:48:08.169 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:48:08.169 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.169 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.169 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.169 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-01-26T23:48:08.169 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl6YT1CRAApXlN4j20xzwda0tDMy94Eg==", 2024-01-26T23:48:08.169 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.169 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:48:08.169 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:48:08.169 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:48:08.169 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:48:08.169 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.169 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.170 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.170 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-01-26T23:48:08.170 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlHvDQEhAAxMhgWJsZC5/6Af5zFV6XIA==", 2024-01-26T23:48:08.170 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.170 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:48:08.170 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:48:08.170 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:48:08.170 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:48:08.170 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.170 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.170 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.170 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-01-26T23:48:08.170 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlvMwDCBAAXnFyMduz7PfWdLWbjytIiQ==", 2024-01-26T23:48:08.170 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.170 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:48:08.171 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:48:08.171 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:48:08.171 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:48:08.171 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.171 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.171 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.171 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-01-26T23:48:08.171 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlhMNSJBAA38QU6DrCvFjO+u9zQ8bkEA==", 2024-01-26T23:48:08.171 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.171 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:48:08.171 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-26T23:48:08.171 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:48:08.171 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.171 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:08.171 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:08.172 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-01-26T23:48:08.172 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlI9EbIBAACTpzAO+8A/VFJZcT/WWBng==", 2024-01-26T23:48:08.172 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:48:08.172 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:48:08.172 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-26T23:48:08.172 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:48:08.172 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.172 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:08.172 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-01-26T23:48:08.172 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:48:08.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.161+0000 7ff3b8a06700 1 -- 172.21.15.46:0/4010776898 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff394066f30 msgr2=0x7ff3940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:08.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.161+0000 7ff3b8a06700 1 --2- 172.21.15.46:0/4010776898 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff394066f30 0x7ff3940693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff3b406b900 tx=0x7ff3a8017000 comp rx=0 tx=0).stop 2024-01-26T23:48:08.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.161+0000 7ff3b8a06700 1 -- 172.21.15.46:0/4010776898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3b406d310 msgr2=0x7ff3b40c7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:08.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.161+0000 7ff3b8a06700 1 --2- 172.21.15.46:0/4010776898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3b406d310 0x7ff3b40c7840 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7ff3a400e930 tx=0x7ff3a40091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.161+0000 7ff3b8a06700 1 -- 172.21.15.46:0/4010776898 shutdown_connections 2024-01-26T23:48:08.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.161+0000 7ff3b8a06700 1 --2- 172.21.15.46:0/4010776898 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff394066f30 0x7ff3940693e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.161+0000 7ff3b8a06700 1 --2- 172.21.15.46:0/4010776898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3b406d310 0x7ff3b40c7840 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.161+0000 7ff3b8a06700 1 --2- 172.21.15.46:0/4010776898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3b415b200 0x7ff3b40c1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.161+0000 7ff3b8a06700 1 --2- 172.21.15.46:0/4010776898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b40b3bf0 0x7ff3b41592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.161+0000 7ff3b8a06700 1 -- 172.21.15.46:0/4010776898 >> 172.21.15.46:0/4010776898 conn(0x7ff3b40ce530 msgr2=0x7ff3b40b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:08.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.161+0000 7ff3b8a06700 1 -- 172.21.15.46:0/4010776898 shutdown_connections 2024-01-26T23:48:08.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.161+0000 7ff3b8a06700 1 -- 172.21.15.46:0/4010776898 wait complete. 2024-01-26T23:48:08.176 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:48:08.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.305+0000 7fde22039700 1 Processor -- start 2024-01-26T23:48:08.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.305+0000 7fde22039700 1 -- start start 2024-01-26T23:48:08.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.306+0000 7fde22039700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde1c157920 0x7fde1c0b2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:08.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.306+0000 7fde22039700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fde1c0b2a60 0x7fde1c0cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:08.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.306+0000 7fde22039700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1c0cc550 con 0x7fde1c157550 2024-01-26T23:48:08.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.306+0000 7fde22039700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1c0ca020 con 0x7fde1c157920 2024-01-26T23:48:08.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.306+0000 7fde22039700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1c0b2ee0 con 0x7fde1c0b2a60 2024-01-26T23:48:08.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.306+0000 7fde1affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde1c157920 0x7fde1c0b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:08.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.307+0000 7fde1b7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fde1c157550 0x7fde1c06f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34438/0 (socket says 172.21.15.46:34438) 2024-01-26T23:48:08.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.307+0000 7fde1b7fe700 1 -- 172.21.15.46:0/2014335611 learned_addr learned my addr 172.21.15.46:0/2014335611 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:08.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.307+0000 7fde1bfff700 1 --2- 172.21.15.46:0/2014335611 >> v2:172.21.15.106:3300/0 conn(0x7fde1c0b2a60 0x7fde1c0cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:08.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.307+0000 7fde1affd700 1 -- 172.21.15.46:0/2014335611 >> v2:172.21.15.106:3300/0 conn(0x7fde1c0b2a60 msgr2=0x7fde1c0cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:08.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.307+0000 7fde1affd700 1 --2- 172.21.15.46:0/2014335611 >> v2:172.21.15.106:3300/0 conn(0x7fde1c0b2a60 0x7fde1c0cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.307+0000 7fde1affd700 1 -- 172.21.15.46:0/2014335611 >> v1:172.21.15.46:6789/0 conn(0x7fde1c157550 legacy=0x7fde1c06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:08.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.307+0000 7fde1affd700 1 -- 172.21.15.46:0/2014335611 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1c0cb6a0 con 0x7fde1c157920 2024-01-26T23:48:08.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.307+0000 7fde1a7fc700 1 -- 172.21.15.46:0/2014335611 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716489977 0 0) 0x7fde1c0cc550 con 0x7fde1c157550 2024-01-26T23:48:08.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.307+0000 7fde1affd700 1 --2- 172.21.15.46:0/2014335611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1c157920 0x7fde1c0b2520 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fde1c0cb1e0 tx=0x7fde040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41195732c8f5211 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:08.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.307+0000 7fde1a7fc700 1 -- 172.21.15.46:0/2014335611 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04011630 con 0x7fde1c157920 2024-01-26T23:48:08.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.307+0000 7fde1a7fc700 1 -- 172.21.15.46:0/2014335611 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde04011790 con 0x7fde1c157920 2024-01-26T23:48:08.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.307+0000 7fde1a7fc700 1 -- 172.21.15.46:0/2014335611 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04011b30 con 0x7fde1c157920 2024-01-26T23:48:08.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.308+0000 7fde22039700 1 -- 172.21.15.46:0/2014335611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1c157920 msgr2=0x7fde1c0b2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:08.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.308+0000 7fde22039700 1 --2- 172.21.15.46:0/2014335611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1c157920 0x7fde1c0b2520 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fde1c0cb1e0 tx=0x7fde040091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.308+0000 7fde22039700 1 -- 172.21.15.46:0/2014335611 shutdown_connections 2024-01-26T23:48:08.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.308+0000 7fde22039700 1 --2- 172.21.15.46:0/2014335611 >> v2:172.21.15.106:3300/0 conn(0x7fde1c0b2a60 0x7fde1c0cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.308+0000 7fde22039700 1 --2- 172.21.15.46:0/2014335611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1c157920 0x7fde1c0b2520 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.308+0000 7fde22039700 1 -- 172.21.15.46:0/2014335611 >> 172.21.15.46:0/2014335611 conn(0x7fde1c069510 msgr2=0x7fde1c0aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:08.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.308+0000 7fde22039700 1 -- 172.21.15.46:0/2014335611 shutdown_connections 2024-01-26T23:48:08.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.308+0000 7fde22039700 1 -- 172.21.15.46:0/2014335611 wait complete. 2024-01-26T23:48:08.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.309+0000 7fde22039700 1 Processor -- start 2024-01-26T23:48:08.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.309+0000 7fde22039700 1 -- start start 2024-01-26T23:48:08.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.310+0000 7fde22039700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1c0b2a60 0x7fde1c1526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:08.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.310+0000 7fde22039700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde1c157550 0x7fde1c150b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:08.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.310+0000 7fde22039700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde1c157920 0x7fde1c167520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:08.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.310+0000 7fde22039700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c152210 con 0x7fde1c0b2a60 2024-01-26T23:48:08.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.310+0000 7fde22039700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c151600 con 0x7fde1c157920 2024-01-26T23:48:08.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.310+0000 7fde22039700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c1518e0 con 0x7fde1c157550 2024-01-26T23:48:08.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.310+0000 7fde1b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1c0b2a60 0x7fde1c1526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:08.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.310+0000 7fde1b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1c0b2a60 0x7fde1c1526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39678/0 (socket says 172.21.15.46:39678) 2024-01-26T23:48:08.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.310+0000 7fde1b7fe700 1 -- 172.21.15.46:0/1819720685 learned_addr learned my addr 172.21.15.46:0/1819720685 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:08.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.310+0000 7fde1bfff700 1 --2- 172.21.15.46:0/1819720685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde1c157920 0x7fde1c167520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:08.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.310+0000 7fde1affd700 1 --2- 172.21.15.46:0/1819720685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde1c157550 0x7fde1c150b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:08.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.311+0000 7fde1b7fe700 1 -- 172.21.15.46:0/1819720685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde1c157550 msgr2=0x7fde1c150b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:08.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.311+0000 7fde1b7fe700 1 --2- 172.21.15.46:0/1819720685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde1c157550 0x7fde1c150b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.311+0000 7fde1b7fe700 1 -- 172.21.15.46:0/1819720685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde1c157920 msgr2=0x7fde1c167520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:08.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.311+0000 7fde1b7fe700 1 --2- 172.21.15.46:0/1819720685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde1c157920 0x7fde1c167520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.311+0000 7fde1b7fe700 1 -- 172.21.15.46:0/1819720685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde04009420 con 0x7fde1c0b2a60 2024-01-26T23:48:08.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.311+0000 7fde1bfff700 1 --2- 172.21.15.46:0/1819720685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde1c157920 0x7fde1c167520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:48:08.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.311+0000 7fde1affd700 1 --2- 172.21.15.46:0/1819720685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde1c157550 0x7fde1c150b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:48:08.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.311+0000 7fde1b7fe700 1 --2- 172.21.15.46:0/1819720685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1c0b2a60 0x7fde1c1526a0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fde0c00c7e0 tx=0x7fde0c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:08.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.311+0000 7fde137fe700 1 -- 172.21.15.46:0/1819720685 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c005b30 con 0x7fde1c0b2a60 2024-01-26T23:48:08.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.311+0000 7fde137fe700 1 -- 172.21.15.46:0/1819720685 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0c005c90 con 0x7fde1c0b2a60 2024-01-26T23:48:08.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.311+0000 7fde22039700 1 -- 172.21.15.46:0/1819720685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1c151bc0 con 0x7fde1c0b2a60 2024-01-26T23:48:08.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.311+0000 7fde137fe700 1 -- 172.21.15.46:0/1819720685 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c007300 con 0x7fde1c0b2a60 2024-01-26T23:48:08.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.311+0000 7fde22039700 1 -- 172.21.15.46:0/1819720685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1c14fa40 con 0x7fde1c0b2a60 2024-01-26T23:48:08.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.312+0000 7fde22039700 1 -- 172.21.15.46:0/1819720685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1c1524f0 con 0x7fde1c0b2a60 2024-01-26T23:48:08.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.313+0000 7fde137fe700 1 -- 172.21.15.46:0/1819720685 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fde0c01b400 con 0x7fde1c0b2a60 2024-01-26T23:48:08.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.313+0000 7fde137fe700 1 --2- 172.21.15.46:0/1819720685 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fddfc066f30 0x7fddfc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:08.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.313+0000 7fde137fe700 1 -- 172.21.15.46:0/1819720685 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5565+0+0 (secure 0 0 0) 0x7fde0c084070 con 0x7fde1c0b2a60 2024-01-26T23:48:08.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.313+0000 7fde1affd700 1 --2- 172.21.15.46:0/1819720685 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fddfc066f30 0x7fddfc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:08.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.314+0000 7fde1affd700 1 --2- 172.21.15.46:0/1819720685 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fddfc066f30 0x7fddfc0693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fde1c0cb1e0 tx=0x7fde04017000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:08.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.317+0000 7fde137fe700 1 -- 172.21.15.46:0/1819720685 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fde0c051370 con 0x7fde1c0b2a60 2024-01-26T23:48:08.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.524+0000 7fde22039700 1 -- 172.21.15.46:0/1819720685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde1c04c710 con 0x7fde1c0b2a60 2024-01-26T23:48:08.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.524+0000 7fde137fe700 1 -- 172.21.15.46:0/1819720685 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+12943 (secure 0 0 0) 0x7fde0c054990 con 0x7fde1c0b2a60 2024-01-26T23:48:08.526 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:48:08.526 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":126,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:48:07.692297+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:48:08.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.527+0000 7fde22039700 1 -- 172.21.15.46:0/1819720685 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fddfc066f30 msgr2=0x7fddfc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:08.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.527+0000 7fde22039700 1 --2- 172.21.15.46:0/1819720685 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fddfc066f30 0x7fddfc0693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fde1c0cb1e0 tx=0x7fde04017000 comp rx=0 tx=0).stop 2024-01-26T23:48:08.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.527+0000 7fde22039700 1 -- 172.21.15.46:0/1819720685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1c0b2a60 msgr2=0x7fde1c1526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:08.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.527+0000 7fde22039700 1 --2- 172.21.15.46:0/1819720685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1c0b2a60 0x7fde1c1526a0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fde0c00c7e0 tx=0x7fde0c007df0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.527+0000 7fde22039700 1 -- 172.21.15.46:0/1819720685 shutdown_connections 2024-01-26T23:48:08.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.527+0000 7fde22039700 1 --2- 172.21.15.46:0/1819720685 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fddfc066f30 0x7fddfc0693e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.527+0000 7fde22039700 1 --2- 172.21.15.46:0/1819720685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1c0b2a60 0x7fde1c1526a0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.527+0000 7fde22039700 1 --2- 172.21.15.46:0/1819720685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde1c157920 0x7fde1c167520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.527+0000 7fde22039700 1 --2- 172.21.15.46:0/1819720685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde1c157550 0x7fde1c150b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.527+0000 7fde22039700 1 -- 172.21.15.46:0/1819720685 >> 172.21.15.46:0/1819720685 conn(0x7fde1c069510 msgr2=0x7fde1c0aed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:08.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.527+0000 7fde22039700 1 -- 172.21.15.46:0/1819720685 shutdown_connections 2024-01-26T23:48:08.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.527+0000 7fde22039700 1 -- 172.21.15.46:0/1819720685 wait complete. 2024-01-26T23:48:08.542 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-26T23:48:08.542 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:48:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.675+0000 7fe944109700 1 Processor -- start 2024-01-26T23:48:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.675+0000 7fe944109700 1 -- start start 2024-01-26T23:48:08.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.676+0000 7fe944109700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe93c154010 0x7fe93c06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:08.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.676+0000 7fe944109700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe93c0b1e80 0x7fe93c0692e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:08.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.676+0000 7fe944109700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93c15ad60 con 0x7fe93c153c40 2024-01-26T23:48:08.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.676+0000 7fe944109700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe93c0b1d20 con 0x7fe93c0b1e80 2024-01-26T23:48:08.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.676+0000 7fe944109700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c15a150 con 0x7fe93c154010 2024-01-26T23:48:08.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.676+0000 7fe941ea5700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe93c153c40 0x7fe93c06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34454/0 (socket says 172.21.15.46:34454) 2024-01-26T23:48:08.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.676+0000 7fe9426a6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe93c0b1e80 0x7fe93c0692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:08.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.676+0000 7fe941ea5700 1 -- 172.21.15.46:0/1693883992 learned_addr learned my addr 172.21.15.46:0/1693883992 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:08.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.676+0000 7fe9416a4700 1 --2- 172.21.15.46:0/1693883992 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe93c154010 0x7fe93c06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:08.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.677+0000 7fe9426a6700 1 -- 172.21.15.46:0/1693883992 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe93c154010 msgr2=0x7fe93c06ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:08.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.677+0000 7fe9426a6700 1 --2- 172.21.15.46:0/1693883992 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe93c154010 0x7fe93c06ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.677+0000 7fe9426a6700 1 -- 172.21.15.46:0/1693883992 >> v1:172.21.15.46:6789/0 conn(0x7fe93c153c40 legacy=0x7fe93c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:08.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.677+0000 7fe940ea3700 1 -- 172.21.15.46:0/1693883992 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320304285 0 0) 0x7fe93c15ad60 con 0x7fe93c153c40 2024-01-26T23:48:08.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.677+0000 7fe9426a6700 1 -- 172.21.15.46:0/1693883992 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c15a430 con 0x7fe93c0b1e80 2024-01-26T23:48:08.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.677+0000 7fe9426a6700 1 --2- 172.21.15.46:0/1693883992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93c0b1e80 0x7fe93c0692e0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fe934009e60 tx=0x7fe934009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbbcaf53f2460bc1 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:08.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.677+0000 7fe940ea3700 1 -- 172.21.15.46:0/1693883992 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93400e070 con 0x7fe93c0b1e80 2024-01-26T23:48:08.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.677+0000 7fe940ea3700 1 -- 172.21.15.46:0/1693883992 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93400bb60 con 0x7fe93c0b1e80 2024-01-26T23:48:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.677+0000 7fe940ea3700 1 -- 172.21.15.46:0/1693883992 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934012430 con 0x7fe93c0b1e80 2024-01-26T23:48:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.678+0000 7fe944109700 1 -- 172.21.15.46:0/1693883992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93c0b1e80 msgr2=0x7fe93c0692e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.678+0000 7fe944109700 1 --2- 172.21.15.46:0/1693883992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93c0b1e80 0x7fe93c0692e0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fe934009e60 tx=0x7fe934009780 comp rx=0 tx=0).stop 2024-01-26T23:48:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.678+0000 7fe944109700 1 -- 172.21.15.46:0/1693883992 shutdown_connections 2024-01-26T23:48:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.678+0000 7fe944109700 1 --2- 172.21.15.46:0/1693883992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93c0b1e80 0x7fe93c0692e0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.678+0000 7fe944109700 1 --2- 172.21.15.46:0/1693883992 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe93c154010 0x7fe93c06ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.678+0000 7fe944109700 1 -- 172.21.15.46:0/1693883992 >> 172.21.15.46:0/1693883992 conn(0x7fe93c067930 msgr2=0x7fe93c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.678+0000 7fe944109700 1 -- 172.21.15.46:0/1693883992 shutdown_connections 2024-01-26T23:48:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.678+0000 7fe944109700 1 -- 172.21.15.46:0/1693883992 wait complete. 2024-01-26T23:48:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.679+0000 7fe944109700 1 Processor -- start 2024-01-26T23:48:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.679+0000 7fe944109700 1 -- start start 2024-01-26T23:48:08.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.680+0000 7fe944109700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93c0b1e80 0x7fe93c0c9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.680+0000 7fe944109700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe93c153c40 0x7fe93c159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.680+0000 7fe944109700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe93c154010 0x7fe93c169a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.680+0000 7fe944109700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c1535b0 con 0x7fe93c0b1e80 2024-01-26T23:48:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.680+0000 7fe944109700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0ca0e0 con 0x7fe93c153c40 2024-01-26T23:48:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.680+0000 7fe944109700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c159670 con 0x7fe93c154010 2024-01-26T23:48:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.680+0000 7fe941ea5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93c0b1e80 0x7fe93c0c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.680+0000 7fe941ea5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93c0b1e80 0x7fe93c0c9140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39690/0 (socket says 172.21.15.46:39690) 2024-01-26T23:48:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.680+0000 7fe941ea5700 1 -- 172.21.15.46:0/578431427 learned_addr learned my addr 172.21.15.46:0/578431427 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.680+0000 7fe9426a6700 1 --2- 172.21.15.46:0/578431427 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe93c154010 0x7fe93c169a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.680+0000 7fe9416a4700 1 --2- 172.21.15.46:0/578431427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe93c153c40 0x7fe93c159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.680+0000 7fe941ea5700 1 -- 172.21.15.46:0/578431427 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe93c154010 msgr2=0x7fe93c169a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.680+0000 7fe941ea5700 1 --2- 172.21.15.46:0/578431427 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe93c154010 0x7fe93c169a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.680+0000 7fe941ea5700 1 -- 172.21.15.46:0/578431427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe93c153c40 msgr2=0x7fe93c159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.680+0000 7fe941ea5700 1 --2- 172.21.15.46:0/578431427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe93c153c40 0x7fe93c159130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.680+0000 7fe941ea5700 1 -- 172.21.15.46:0/578431427 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe934009a20 con 0x7fe93c0b1e80 2024-01-26T23:48:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.681+0000 7fe941ea5700 1 --2- 172.21.15.46:0/578431427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93c0b1e80 0x7fe93c0c9140 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fe92c00c7e0 tx=0x7fe92c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.681+0000 7fe931ffb700 1 -- 172.21.15.46:0/578431427 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c005b30 con 0x7fe93c0b1e80 2024-01-26T23:48:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.681+0000 7fe931ffb700 1 -- 172.21.15.46:0/578431427 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92c005c90 con 0x7fe93c0b1e80 2024-01-26T23:48:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.681+0000 7fe931ffb700 1 -- 172.21.15.46:0/578431427 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c007300 con 0x7fe93c0b1e80 2024-01-26T23:48:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.681+0000 7fe944109700 1 -- 172.21.15.46:0/578431427 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93c0c9840 con 0x7fe93c0b1e80 2024-01-26T23:48:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.681+0000 7fe944109700 1 -- 172.21.15.46:0/578431427 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93c0c9e00 con 0x7fe93c0b1e80 2024-01-26T23:48:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.682+0000 7fe944109700 1 -- 172.21.15.46:0/578431427 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93c04c710 con 0x7fe93c0b1e80 2024-01-26T23:48:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.683+0000 7fe931ffb700 1 -- 172.21.15.46:0/578431427 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe92c01b3b0 con 0x7fe93c0b1e80 2024-01-26T23:48:08.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.683+0000 7fe931ffb700 1 --2- 172.21.15.46:0/578431427 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe920066f30 0x7fe9200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:08.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.683+0000 7fe931ffb700 1 -- 172.21.15.46:0/578431427 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5565+0+0 (secure 0 0 0) 0x7fe92c0839c0 con 0x7fe93c0b1e80 2024-01-26T23:48:08.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.683+0000 7fe9416a4700 1 --2- 172.21.15.46:0/578431427 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe920066f30 0x7fe9200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:08.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.684+0000 7fe9416a4700 1 --2- 172.21.15.46:0/578431427 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe920066f30 0x7fe9200693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe93c15b190 tx=0x7fe9380091a0 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:08.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.687+0000 7fe931ffb700 1 -- 172.21.15.46:0/578431427 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe92c050cc0 con 0x7fe93c0b1e80 2024-01-26T23:48:08.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.891+0000 7fe944109700 1 -- 172.21.15.46:0/578431427 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe93c15a710 con 0x7fe93c0b1e80 2024-01-26T23:48:08.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.891+0000 7fe931ffb700 1 -- 172.21.15.46:0/578431427 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe92c017020 con 0x7fe93c0b1e80 2024-01-26T23:48:08.893 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:48:08.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.894+0000 7fe944109700 1 -- 172.21.15.46:0/578431427 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe920066f30 msgr2=0x7fe9200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:08.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.894+0000 7fe944109700 1 --2- 172.21.15.46:0/578431427 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe920066f30 0x7fe9200693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe93c15b190 tx=0x7fe9380091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.894+0000 7fe944109700 1 -- 172.21.15.46:0/578431427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93c0b1e80 msgr2=0x7fe93c0c9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:08.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.894+0000 7fe944109700 1 --2- 172.21.15.46:0/578431427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93c0b1e80 0x7fe93c0c9140 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fe92c00c7e0 tx=0x7fe92c007df0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.894+0000 7fe944109700 1 -- 172.21.15.46:0/578431427 shutdown_connections 2024-01-26T23:48:08.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.894+0000 7fe944109700 1 --2- 172.21.15.46:0/578431427 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe920066f30 0x7fe9200693e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.894+0000 7fe944109700 1 --2- 172.21.15.46:0/578431427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93c0b1e80 0x7fe93c0c9140 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.894+0000 7fe944109700 1 --2- 172.21.15.46:0/578431427 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe93c154010 0x7fe93c169a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.894+0000 7fe944109700 1 --2- 172.21.15.46:0/578431427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe93c153c40 0x7fe93c159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:08.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.894+0000 7fe944109700 1 -- 172.21.15.46:0/578431427 >> 172.21.15.46:0/578431427 conn(0x7fe93c067930 msgr2=0x7fe93c0692e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:08.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.894+0000 7fe944109700 1 -- 172.21.15.46:0/578431427 shutdown_connections 2024-01-26T23:48:08.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:08.894+0000 7fe944109700 1 -- 172.21.15.46:0/578431427 wait complete. 2024-01-26T23:48:08.909 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-26T23:48:09.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.039+0000 7f3275aed700 1 Processor -- start 2024-01-26T23:48:09.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.039+0000 7f3275aed700 1 -- start start 2024-01-26T23:48:09.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.040+0000 7f3275aed700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32700c3cf0 0x7f32700ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.040+0000 7f3275aed700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f32700cd080 0x7f32700ab1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.040+0000 7f3275aed700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32700cbf10 con 0x7f32700acf30 2024-01-26T23:48:09.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.040+0000 7f3275aed700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32700ada10 con 0x7f32700c3cf0 2024-01-26T23:48:09.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.040+0000 7f3275aed700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3270157be0 con 0x7f32700cd080 2024-01-26T23:48:09.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.040+0000 7f326effd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32700c3cf0 0x7f32700ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.040+0000 7f326f7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f32700acf30 0x7f32700ad300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34468/0 (socket says 172.21.15.46:34468) 2024-01-26T23:48:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.040+0000 7f326f7fe700 1 -- 172.21.15.46:0/3635188143 learned_addr learned my addr 172.21.15.46:0/3635188143 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.041+0000 7f326ffff700 1 --2- 172.21.15.46:0/3635188143 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f32700cd080 0x7f32700ab1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.041+0000 7f326effd700 1 -- 172.21.15.46:0/3635188143 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f32700cd080 msgr2=0x7f32700ab1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.041+0000 7f326effd700 1 --2- 172.21.15.46:0/3635188143 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f32700cd080 0x7f32700ab1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.041+0000 7f326effd700 1 -- 172.21.15.46:0/3635188143 >> v1:172.21.15.46:6789/0 conn(0x7f32700acf30 legacy=0x7f32700ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.041+0000 7f326effd700 1 -- 172.21.15.46:0/3635188143 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32700cd480 con 0x7f32700c3cf0 2024-01-26T23:48:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.041+0000 7f326e7fc700 1 -- 172.21.15.46:0/3635188143 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793076289 0 0) 0x7f32700cbf10 con 0x7f32700acf30 2024-01-26T23:48:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.041+0000 7f326effd700 1 --2- 172.21.15.46:0/3635188143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32700c3cf0 0x7f32700ccb40 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f32640098e0 tx=0x7f32640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27e3d06e9b8377a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.041+0000 7f326e7fc700 1 -- 172.21.15.46:0/3635188143 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264009e80 con 0x7f32700c3cf0 2024-01-26T23:48:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.041+0000 7f326e7fc700 1 -- 172.21.15.46:0/3635188143 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326400b860 con 0x7f32700c3cf0 2024-01-26T23:48:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.041+0000 7f326e7fc700 1 -- 172.21.15.46:0/3635188143 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326400bbe0 con 0x7f32700c3cf0 2024-01-26T23:48:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.042+0000 7f3275aed700 1 -- 172.21.15.46:0/3635188143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32700c3cf0 msgr2=0x7f32700ccb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.042+0000 7f3275aed700 1 --2- 172.21.15.46:0/3635188143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32700c3cf0 0x7f32700ccb40 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f32640098e0 tx=0x7f32640091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.042+0000 7f3275aed700 1 -- 172.21.15.46:0/3635188143 shutdown_connections 2024-01-26T23:48:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.042+0000 7f3275aed700 1 --2- 172.21.15.46:0/3635188143 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f32700cd080 0x7f32700ab1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.042+0000 7f3275aed700 1 --2- 172.21.15.46:0/3635188143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32700c3cf0 0x7f32700ccb40 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.042+0000 7f3275aed700 1 -- 172.21.15.46:0/3635188143 >> 172.21.15.46:0/3635188143 conn(0x7f3270097820 msgr2=0x7f3270068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.042+0000 7f3275aed700 1 -- 172.21.15.46:0/3635188143 shutdown_connections 2024-01-26T23:48:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.042+0000 7f3275aed700 1 -- 172.21.15.46:0/3635188143 wait complete. 2024-01-26T23:48:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.043+0000 7f3275aed700 1 Processor -- start 2024-01-26T23:48:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.043+0000 7f3275aed700 1 -- start start 2024-01-26T23:48:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.044+0000 7f3275aed700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32700acf30 0x7f3270153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.044+0000 7f3275aed700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f32700c3cf0 0x7f3270159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.044+0000 7f3275aed700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f32700cd080 0x7f3270158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.044+0000 7f3275aed700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32701581c0 con 0x7f32700acf30 2024-01-26T23:48:09.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.044+0000 7f3275aed700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32700cb880 con 0x7f32700c3cf0 2024-01-26T23:48:09.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.044+0000 7f3275aed700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3270159e50 con 0x7f32700cd080 2024-01-26T23:48:09.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.044+0000 7f326f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32700acf30 0x7f3270153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.044+0000 7f326f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32700acf30 0x7f3270153570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39716/0 (socket says 172.21.15.46:39716) 2024-01-26T23:48:09.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.044+0000 7f326f7fe700 1 -- 172.21.15.46:0/1641243495 learned_addr learned my addr 172.21.15.46:0/1641243495 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:09.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.044+0000 7f326effd700 1 --2- 172.21.15.46:0/1641243495 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f32700c3cf0 0x7f3270159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.044+0000 7f326ffff700 1 --2- 172.21.15.46:0/1641243495 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f32700cd080 0x7f3270158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.044+0000 7f326f7fe700 1 -- 172.21.15.46:0/1641243495 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f32700cd080 msgr2=0x7f3270158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:09.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.044+0000 7f326f7fe700 1 --2- 172.21.15.46:0/1641243495 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f32700cd080 0x7f3270158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.044+0000 7f326f7fe700 1 -- 172.21.15.46:0/1641243495 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f32700c3cf0 msgr2=0x7f3270159750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:09.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.044+0000 7f326f7fe700 1 --2- 172.21.15.46:0/1641243495 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f32700c3cf0 0x7f3270159750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.045+0000 7f326f7fe700 1 -- 172.21.15.46:0/1641243495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3264009420 con 0x7f32700acf30 2024-01-26T23:48:09.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.045+0000 7f326f7fe700 1 --2- 172.21.15.46:0/1641243495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32700acf30 0x7f3270153570 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f326000c810 tx=0x7f3260007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.045+0000 7f32577fe700 1 -- 172.21.15.46:0/1641243495 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3260005b30 con 0x7f32700acf30 2024-01-26T23:48:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.045+0000 7f32577fe700 1 -- 172.21.15.46:0/1641243495 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3260005c90 con 0x7f32700acf30 2024-01-26T23:48:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.045+0000 7f3275aed700 1 -- 172.21.15.46:0/1641243495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f327015a130 con 0x7f32700acf30 2024-01-26T23:48:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.045+0000 7f32577fe700 1 -- 172.21.15.46:0/1641243495 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3260007300 con 0x7f32700acf30 2024-01-26T23:48:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.045+0000 7f3275aed700 1 -- 172.21.15.46:0/1641243495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f327015a410 con 0x7f32700acf30 2024-01-26T23:48:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.046+0000 7f3275aed700 1 -- 172.21.15.46:0/1641243495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32701584a0 con 0x7f32700acf30 2024-01-26T23:48:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.046+0000 7f32577fe700 1 -- 172.21.15.46:0/1641243495 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f326001b3b0 con 0x7f32700acf30 2024-01-26T23:48:09.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.047+0000 7f32577fe700 1 --2- 172.21.15.46:0/1641243495 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f325c066f30 0x7f325c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.047+0000 7f32577fe700 1 -- 172.21.15.46:0/1641243495 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5565+0+0 (secure 0 0 0) 0x7f3260083950 con 0x7f32700acf30 2024-01-26T23:48:09.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.047+0000 7f326effd700 1 --2- 172.21.15.46:0/1641243495 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f325c066f30 0x7f325c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.048+0000 7f326effd700 1 --2- 172.21.15.46:0/1641243495 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f325c066f30 0x7f325c0693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f32640091e0 tx=0x7f3264017000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:09.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.051+0000 7f32577fe700 1 -- 172.21.15.46:0/1641243495 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3260050c50 con 0x7f32700acf30 2024-01-26T23:48:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.259+0000 7f3275aed700 1 -- 172.21.15.46:0/1641243495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f327004c710 con 0x7f32700acf30 2024-01-26T23:48:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.259+0000 7f32577fe700 1 -- 172.21.15.46:0/1641243495 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v126) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3260017090 con 0x7f32700acf30 2024-01-26T23:48:09.260 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:48:09.261 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-26T23:48:09.261 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:09.261 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-01-26T23:48:09.261 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-01-26T23:48:09.261 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:09.261 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-26T23:48:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.261+0000 7f3275aed700 1 -- 172.21.15.46:0/1641243495 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f325c066f30 msgr2=0x7f325c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.261+0000 7f3275aed700 1 --2- 172.21.15.46:0/1641243495 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f325c066f30 0x7f325c0693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f32640091e0 tx=0x7f3264017000 comp rx=0 tx=0).stop 2024-01-26T23:48:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.261+0000 7f3275aed700 1 -- 172.21.15.46:0/1641243495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32700acf30 msgr2=0x7f3270153570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.261+0000 7f3275aed700 1 --2- 172.21.15.46:0/1641243495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32700acf30 0x7f3270153570 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f326000c810 tx=0x7f3260007df0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.261+0000 7f3275aed700 1 -- 172.21.15.46:0/1641243495 shutdown_connections 2024-01-26T23:48:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.261+0000 7f3275aed700 1 --2- 172.21.15.46:0/1641243495 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f325c066f30 0x7f325c0693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.261+0000 7f3275aed700 1 --2- 172.21.15.46:0/1641243495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32700acf30 0x7f3270153570 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.261+0000 7f3275aed700 1 --2- 172.21.15.46:0/1641243495 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f32700cd080 0x7f3270158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.261+0000 7f3275aed700 1 --2- 172.21.15.46:0/1641243495 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f32700c3cf0 0x7f3270159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.261+0000 7f3275aed700 1 -- 172.21.15.46:0/1641243495 >> 172.21.15.46:0/1641243495 conn(0x7f3270097820 msgr2=0x7f32700cebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.261+0000 7f3275aed700 1 -- 172.21.15.46:0/1641243495 shutdown_connections 2024-01-26T23:48:09.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.261+0000 7f3275aed700 1 -- 172.21.15.46:0/1641243495 wait complete. 2024-01-26T23:48:09.276 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-26T23:48:09.277 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-26T23:48:09.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.411+0000 7f6a73b1e700 1 Processor -- start 2024-01-26T23:48:09.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.411+0000 7f6a73b1e700 1 -- start start 2024-01-26T23:48:09.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.412+0000 7f6a73b1e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a6c159630 0x7f6a6c159a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.412+0000 7f6a73b1e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6a6c067930 0x7f6a6c067d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.412+0000 7f6a73b1e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a6c15ab40 con 0x7f6a6c0af570 2024-01-26T23:48:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.412+0000 7f6a73b1e700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a6c0682b0 con 0x7f6a6c067930 2024-01-26T23:48:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.412+0000 7f6a73b1e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c159f40 con 0x7f6a6c159630 2024-01-26T23:48:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.412+0000 7f6a720bb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6a6c067930 0x7f6a6c067d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.412+0000 7f6a720bb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6a6c067930 0x7f6a6c067d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39732/0 (socket says 172.21.15.46:39732) 2024-01-26T23:48:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.412+0000 7f6a720bb700 1 -- 172.21.15.46:0/2480025218 learned_addr learned my addr 172.21.15.46:0/2480025218 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.413+0000 7f6a718ba700 1 --2- 172.21.15.46:0/2480025218 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a6c159630 0x7f6a6c159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.413+0000 7f6a720bb700 1 -- 172.21.15.46:0/2480025218 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a6c159630 msgr2=0x7f6a6c159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.413+0000 7f6a720bb700 1 --2- 172.21.15.46:0/2480025218 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a6c159630 0x7f6a6c159a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.413+0000 7f6a720bb700 1 -- 172.21.15.46:0/2480025218 >> v1:172.21.15.106:6789/0 conn(0x7f6a6c0af570 legacy=0x7f6a6c0b2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.413+0000 7f6a720bb700 1 -- 172.21.15.46:0/2480025218 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c15a210 con 0x7f6a6c067930 2024-01-26T23:48:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.413+0000 7f6a720bb700 1 --2- 172.21.15.46:0/2480025218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a6c067930 0x7f6a6c067d70 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f6a64009f20 tx=0x7f6a64009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d91be3d807bcc8 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.413+0000 7f6a708b8700 1 -- 172.21.15.46:0/2480025218 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6400e070 con 0x7f6a6c067930 2024-01-26T23:48:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.413+0000 7f6a708b8700 1 -- 172.21.15.46:0/2480025218 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6400ba80 con 0x7f6a6c067930 2024-01-26T23:48:09.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.413+0000 7f6a708b8700 1 -- 172.21.15.46:0/2480025218 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a640092e0 con 0x7f6a6c067930 2024-01-26T23:48:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.414+0000 7f6a73b1e700 1 -- 172.21.15.46:0/2480025218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a6c067930 msgr2=0x7f6a6c067d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.414+0000 7f6a73b1e700 1 --2- 172.21.15.46:0/2480025218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a6c067930 0x7f6a6c067d70 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f6a64009f20 tx=0x7f6a64009840 comp rx=0 tx=0).stop 2024-01-26T23:48:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.414+0000 7f6a73b1e700 1 -- 172.21.15.46:0/2480025218 shutdown_connections 2024-01-26T23:48:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.414+0000 7f6a73b1e700 1 --2- 172.21.15.46:0/2480025218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a6c067930 0x7f6a6c067d70 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.414+0000 7f6a73b1e700 1 --2- 172.21.15.46:0/2480025218 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a6c159630 0x7f6a6c159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.414+0000 7f6a73b1e700 1 -- 172.21.15.46:0/2480025218 >> 172.21.15.46:0/2480025218 conn(0x7f6a6c069510 msgr2=0x7f6a6c0aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.414+0000 7f6a73b1e700 1 -- 172.21.15.46:0/2480025218 shutdown_connections 2024-01-26T23:48:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.414+0000 7f6a73b1e700 1 -- 172.21.15.46:0/2480025218 wait complete. 2024-01-26T23:48:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.415+0000 7f6a73b1e700 1 Processor -- start 2024-01-26T23:48:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.415+0000 7f6a73b1e700 1 -- start start 2024-01-26T23:48:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a73b1e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a6c067930 0x7f6a6c0abca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a73b1e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a6c0af570 0x7f6a6c13b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a73b1e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a6c159630 0x7f6a6c169d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a73b1e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c150620 con 0x7f6a6c159630 2024-01-26T23:48:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a73b1e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c0ac4d0 con 0x7f6a6c0af570 2024-01-26T23:48:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a73b1e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c0ac7b0 con 0x7f6a6c067930 2024-01-26T23:48:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a720bb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a6c159630 0x7f6a6c169d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a720bb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a6c159630 0x7f6a6c169d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39736/0 (socket says 172.21.15.46:39736) 2024-01-26T23:48:09.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a720bb700 1 -- 172.21.15.46:0/2957089419 learned_addr learned my addr 172.21.15.46:0/2957089419 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a718ba700 1 --2- 172.21.15.46:0/2957089419 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a6c067930 0x7f6a6c0abca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a710b9700 1 --2- 172.21.15.46:0/2957089419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a6c0af570 0x7f6a6c13b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a720bb700 1 -- 172.21.15.46:0/2957089419 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a6c067930 msgr2=0x7f6a6c0abca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a720bb700 1 --2- 172.21.15.46:0/2957089419 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a6c067930 0x7f6a6c0abca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a720bb700 1 -- 172.21.15.46:0/2957089419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a6c0af570 msgr2=0x7f6a6c13b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a720bb700 1 --2- 172.21.15.46:0/2957089419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a6c0af570 0x7f6a6c13b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a720bb700 1 -- 172.21.15.46:0/2957089419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a64009ae0 con 0x7f6a6c159630 2024-01-26T23:48:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a710b9700 1 --2- 172.21.15.46:0/2957089419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a6c0af570 0x7f6a6c13b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:48:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a718ba700 1 --2- 172.21.15.46:0/2957089419 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a6c067930 0x7f6a6c0abca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:48:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.416+0000 7f6a720bb700 1 --2- 172.21.15.46:0/2957089419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a6c159630 0x7f6a6c169d20 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400b6c0 tx=0x7f6a64018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.417+0000 7f6a5dffb700 1 -- 172.21.15.46:0/2957089419 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a640094e0 con 0x7f6a6c159630 2024-01-26T23:48:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.417+0000 7f6a5dffb700 1 -- 172.21.15.46:0/2957089419 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a640032f0 con 0x7f6a6c159630 2024-01-26T23:48:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.417+0000 7f6a5dffb700 1 -- 172.21.15.46:0/2957089419 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a64026690 con 0x7f6a6c159630 2024-01-26T23:48:09.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.417+0000 7f6a73b1e700 1 -- 172.21.15.46:0/2957089419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6c14ffd0 con 0x7f6a6c159630 2024-01-26T23:48:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.417+0000 7f6a73b1e700 1 -- 172.21.15.46:0/2957089419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a6c0ac1e0 con 0x7f6a6c159630 2024-01-26T23:48:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.418+0000 7f6a73b1e700 1 -- 172.21.15.46:0/2957089419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6c150790 con 0x7f6a6c159630 2024-01-26T23:48:09.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.419+0000 7f6a5dffb700 1 -- 172.21.15.46:0/2957089419 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6a64003040 con 0x7f6a6c159630 2024-01-26T23:48:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.419+0000 7f6a5dffb700 1 --2- 172.21.15.46:0/2957089419 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6a50066f30 0x7f6a500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.419+0000 7f6a5dffb700 1 -- 172.21.15.46:0/2957089419 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5565+0+0 (secure 0 0 0) 0x7f6a64086670 con 0x7f6a6c159630 2024-01-26T23:48:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.420+0000 7f6a718ba700 1 --2- 172.21.15.46:0/2957089419 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6a50066f30 0x7f6a500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.420+0000 7f6a718ba700 1 --2- 172.21.15.46:0/2957089419 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6a50066f30 0x7f6a500693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009990 tx=0x7f6a68008040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:09.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.424+0000 7f6a5dffb700 1 -- 172.21.15.46:0/2957089419 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6a64053990 con 0x7f6a6c159630 2024-01-26T23:48:09.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.628+0000 7f6a73b1e700 1 -- 172.21.15.46:0/2957089419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f6a6c04c710 con 0x7f6a6c159630 2024-01-26T23:48:09.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.785+0000 7f6a5dffb700 1 -- 172.21.15.46:0/2957089419 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v127) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6a64056fb0 con 0x7f6a6c159630 2024-01-26T23:48:09.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.789+0000 7f6a73b1e700 1 -- 172.21.15.46:0/2957089419 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6a50066f30 msgr2=0x7f6a500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:09.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.789+0000 7f6a73b1e700 1 --2- 172.21.15.46:0/2957089419 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6a50066f30 0x7f6a500693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009990 tx=0x7f6a68008040 comp rx=0 tx=0).stop 2024-01-26T23:48:09.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.789+0000 7f6a73b1e700 1 -- 172.21.15.46:0/2957089419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a6c159630 msgr2=0x7f6a6c169d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:09.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.789+0000 7f6a73b1e700 1 --2- 172.21.15.46:0/2957089419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a6c159630 0x7f6a6c169d20 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400b6c0 tx=0x7f6a64018000 comp rx=0 tx=0).stop 2024-01-26T23:48:09.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.790+0000 7f6a73b1e700 1 -- 172.21.15.46:0/2957089419 shutdown_connections 2024-01-26T23:48:09.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.790+0000 7f6a73b1e700 1 --2- 172.21.15.46:0/2957089419 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6a50066f30 0x7f6a500693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.790+0000 7f6a73b1e700 1 --2- 172.21.15.46:0/2957089419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a6c159630 0x7f6a6c169d20 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.790+0000 7f6a73b1e700 1 --2- 172.21.15.46:0/2957089419 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a6c067930 0x7f6a6c0abca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.790+0000 7f6a73b1e700 1 --2- 172.21.15.46:0/2957089419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a6c0af570 0x7f6a6c13b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.790+0000 7f6a73b1e700 1 -- 172.21.15.46:0/2957089419 >> 172.21.15.46:0/2957089419 conn(0x7f6a6c069510 msgr2=0x7f6a6c06eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:09.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.791+0000 7f6a73b1e700 1 -- 172.21.15.46:0/2957089419 shutdown_connections 2024-01-26T23:48:09.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.791+0000 7f6a73b1e700 1 -- 172.21.15.46:0/2957089419 wait complete. 2024-01-26T23:48:09.796 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-01-26T23:48:09.809 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-26T23:48:09.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.949+0000 7f6b3ddc8700 1 Processor -- start 2024-01-26T23:48:09.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.950+0000 7f6b3ddc8700 1 -- start start 2024-01-26T23:48:09.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.950+0000 7f6b3ddc8700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6b38067af0 0x7f6b38067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.950+0000 7f6b3ddc8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b3806ebe0 0x7f6b380ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.950+0000 7f6b3ddc8700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3806fdb0 con 0x7f6b381597e0 2024-01-26T23:48:09.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.950+0000 7f6b3ddc8700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b380ade10 con 0x7f6b3806ebe0 2024-01-26T23:48:09.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.950+0000 7f6b3ddc8700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b3806f1a0 con 0x7f6b38067af0 2024-01-26T23:48:09.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.951+0000 7f6b36ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b3806ebe0 0x7f6b380ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.951+0000 7f6b36ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b3806ebe0 0x7f6b380ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39742/0 (socket says 172.21.15.46:39742) 2024-01-26T23:48:09.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.951+0000 7f6b36ffd700 1 -- 172.21.15.46:0/2020268610 learned_addr learned my addr 172.21.15.46:0/2020268610 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:09.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.951+0000 7f6b377fe700 1 --2- 172.21.15.46:0/2020268610 >> v2:172.21.15.106:3300/0 conn(0x7f6b38067af0 0x7f6b38067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.951+0000 7f6b36ffd700 1 -- 172.21.15.46:0/2020268610 >> v2:172.21.15.106:3300/0 conn(0x7f6b38067af0 msgr2=0x7f6b38067ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:09.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.951+0000 7f6b36ffd700 1 --2- 172.21.15.46:0/2020268610 >> v2:172.21.15.106:3300/0 conn(0x7f6b38067af0 0x7f6b38067ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.951+0000 7f6b36ffd700 1 -- 172.21.15.46:0/2020268610 >> v1:172.21.15.106:6789/0 conn(0x7f6b381597e0 legacy=0x7f6b38159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:09.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.951+0000 7f6b36ffd700 1 -- 172.21.15.46:0/2020268610 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3806f480 con 0x7f6b3806ebe0 2024-01-26T23:48:09.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.951+0000 7f6b36ffd700 1 --2- 172.21.15.46:0/2020268610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b3806ebe0 0x7f6b380ad8d0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f6b38064050 tx=0x7f6b2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a8b239d9f72e506 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:09.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.951+0000 7f6b367fc700 1 -- 172.21.15.46:0/2020268610 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c00b860 con 0x7f6b3806ebe0 2024-01-26T23:48:09.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.951+0000 7f6b367fc700 1 -- 172.21.15.46:0/2020268610 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2c00b9c0 con 0x7f6b3806ebe0 2024-01-26T23:48:09.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.951+0000 7f6b367fc700 1 -- 172.21.15.46:0/2020268610 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c00bd80 con 0x7f6b3806ebe0 2024-01-26T23:48:09.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.952+0000 7f6b3ddc8700 1 -- 172.21.15.46:0/2020268610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b3806ebe0 msgr2=0x7f6b380ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:09.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.952+0000 7f6b3ddc8700 1 --2- 172.21.15.46:0/2020268610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b3806ebe0 0x7f6b380ad8d0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f6b38064050 tx=0x7f6b2c009250 comp rx=0 tx=0).stop 2024-01-26T23:48:09.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.952+0000 7f6b3ddc8700 1 -- 172.21.15.46:0/2020268610 shutdown_connections 2024-01-26T23:48:09.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.952+0000 7f6b3ddc8700 1 --2- 172.21.15.46:0/2020268610 >> v2:172.21.15.106:3300/0 conn(0x7f6b38067af0 0x7f6b38067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.952+0000 7f6b3ddc8700 1 --2- 172.21.15.46:0/2020268610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b3806ebe0 0x7f6b380ad8d0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.952+0000 7f6b3ddc8700 1 -- 172.21.15.46:0/2020268610 >> 172.21.15.46:0/2020268610 conn(0x7f6b380cdf80 msgr2=0x7f6b3806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:09.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.952+0000 7f6b3ddc8700 1 -- 172.21.15.46:0/2020268610 shutdown_connections 2024-01-26T23:48:09.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.952+0000 7f6b3ddc8700 1 -- 172.21.15.46:0/2020268610 wait complete. 2024-01-26T23:48:09.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.954+0000 7f6b3ddc8700 1 Processor -- start 2024-01-26T23:48:09.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.954+0000 7f6b3ddc8700 1 -- start start 2024-01-26T23:48:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.955+0000 7f6b3ddc8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b38067af0 0x7f6b3814c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.955+0000 7f6b3ddc8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b3806ebe0 0x7f6b38152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.955+0000 7f6b3ddc8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b381597e0 0x7f6b38159c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.955+0000 7f6b3ddc8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b380b31e0 con 0x7f6b381597e0 2024-01-26T23:48:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.955+0000 7f6b3ddc8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b380b2c20 con 0x7f6b3806ebe0 2024-01-26T23:48:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.955+0000 7f6b3ddc8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b380b2f00 con 0x7f6b38067af0 2024-01-26T23:48:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.955+0000 7f6b37fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b381597e0 0x7f6b38159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.955+0000 7f6b37fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b381597e0 0x7f6b38159c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39758/0 (socket says 172.21.15.46:39758) 2024-01-26T23:48:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.955+0000 7f6b37fff700 1 -- 172.21.15.46:0/1434311594 learned_addr learned my addr 172.21.15.46:0/1434311594 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.955+0000 7f6b36ffd700 1 --2- 172.21.15.46:0/1434311594 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b3806ebe0 0x7f6b38152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.955+0000 7f6b377fe700 1 --2- 172.21.15.46:0/1434311594 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b38067af0 0x7f6b3814c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.955+0000 7f6b37fff700 1 -- 172.21.15.46:0/1434311594 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b38067af0 msgr2=0x7f6b3814c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.956+0000 7f6b37fff700 1 --2- 172.21.15.46:0/1434311594 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b38067af0 0x7f6b3814c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.956+0000 7f6b37fff700 1 -- 172.21.15.46:0/1434311594 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b3806ebe0 msgr2=0x7f6b38152740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.956+0000 7f6b37fff700 1 --2- 172.21.15.46:0/1434311594 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b3806ebe0 0x7f6b38152740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.956+0000 7f6b37fff700 1 -- 172.21.15.46:0/1434311594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c0094f0 con 0x7f6b381597e0 2024-01-26T23:48:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.956+0000 7f6b377fe700 1 --2- 172.21.15.46:0/1434311594 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b38067af0 0x7f6b3814c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.956+0000 7f6b37fff700 1 --2- 172.21.15.46:0/1434311594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b381597e0 0x7f6b38159c20 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000c840 tx=0x7f6b30001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.956+0000 7f6b1f7fe700 1 -- 172.21.15.46:0/1434311594 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b30012630 con 0x7f6b381597e0 2024-01-26T23:48:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.956+0000 7f6b1f7fe700 1 -- 172.21.15.46:0/1434311594 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b300127b0 con 0x7f6b381597e0 2024-01-26T23:48:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.956+0000 7f6b3ddc8700 1 -- 172.21.15.46:0/1434311594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b38151f40 con 0x7f6b381597e0 2024-01-26T23:48:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.956+0000 7f6b1f7fe700 1 -- 172.21.15.46:0/1434311594 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b30012b70 con 0x7f6b381597e0 2024-01-26T23:48:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.956+0000 7f6b3ddc8700 1 -- 172.21.15.46:0/1434311594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b380b28d0 con 0x7f6b381597e0 2024-01-26T23:48:09.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.957+0000 7f6b3ddc8700 1 -- 172.21.15.46:0/1434311594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3804c710 con 0x7f6b381597e0 2024-01-26T23:48:09.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.958+0000 7f6b1f7fe700 1 -- 172.21.15.46:0/1434311594 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6b30003030 con 0x7f6b381597e0 2024-01-26T23:48:09.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.958+0000 7f6b1f7fe700 1 --2- 172.21.15.46:0/1434311594 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b24066f60 0x7f6b24069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:09.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.958+0000 7f6b1f7fe700 1 -- 172.21.15.46:0/1434311594 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5935+0+0 (secure 0 0 0) 0x7f6b30084580 con 0x7f6b381597e0 2024-01-26T23:48:09.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.958+0000 7f6b377fe700 1 --2- 172.21.15.46:0/1434311594 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b24066f60 0x7f6b24069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:09.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.959+0000 7f6b377fe700 1 --2- 172.21.15.46:0/1434311594 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b24066f60 0x7f6b24069410 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6b28006fd0 tx=0x7f6b28008040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:09.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:09.963+0000 7f6b1f7fe700 1 -- 172.21.15.46:0/1434311594 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6b30051710 con 0x7f6b381597e0 2024-01-26T23:48:10.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.166+0000 7f6b3ddc8700 1 -- 172.21.15.46:0/1434311594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f6b380cc140 con 0x7f6b381597e0 2024-01-26T23:48:10.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.793+0000 7f6b1f7fe700 1 -- 172.21.15.46:0/1434311594 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v128) v1 ==== 170+0+0 (secure 0 0 0) 0x7f6b30054d30 con 0x7f6b381597e0 2024-01-26T23:48:10.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.800+0000 7f6b3ddc8700 1 -- 172.21.15.46:0/1434311594 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b24066f60 msgr2=0x7f6b24069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:10.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.800+0000 7f6b3ddc8700 1 --2- 172.21.15.46:0/1434311594 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b24066f60 0x7f6b24069410 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6b28006fd0 tx=0x7f6b28008040 comp rx=0 tx=0).stop 2024-01-26T23:48:10.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.800+0000 7f6b3ddc8700 1 -- 172.21.15.46:0/1434311594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b381597e0 msgr2=0x7f6b38159c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:10.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.800+0000 7f6b3ddc8700 1 --2- 172.21.15.46:0/1434311594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b381597e0 0x7f6b38159c20 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000c840 tx=0x7f6b30001670 comp rx=0 tx=0).stop 2024-01-26T23:48:10.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.801+0000 7f6b3ddc8700 1 -- 172.21.15.46:0/1434311594 shutdown_connections 2024-01-26T23:48:10.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.801+0000 7f6b3ddc8700 1 --2- 172.21.15.46:0/1434311594 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b24066f60 0x7f6b24069410 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:10.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.801+0000 7f6b3ddc8700 1 --2- 172.21.15.46:0/1434311594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b381597e0 0x7f6b38159c20 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:10.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.801+0000 7f6b3ddc8700 1 --2- 172.21.15.46:0/1434311594 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b38067af0 0x7f6b3814c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:10.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.801+0000 7f6b3ddc8700 1 --2- 172.21.15.46:0/1434311594 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b3806ebe0 0x7f6b38152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:10.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.801+0000 7f6b3ddc8700 1 -- 172.21.15.46:0/1434311594 >> 172.21.15.46:0/1434311594 conn(0x7f6b380cdf80 msgr2=0x7f6b3806c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:10.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.801+0000 7f6b3ddc8700 1 -- 172.21.15.46:0/1434311594 shutdown_connections 2024-01-26T23:48:10.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.802+0000 7f6b3ddc8700 1 -- 172.21.15.46:0/1434311594 wait complete. 2024-01-26T23:48:10.810 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-01-26T23:48:10.828 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-26T23:48:10.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.965+0000 7fd71cc31700 1 Processor -- start 2024-01-26T23:48:10.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.965+0000 7fd71cc31700 1 -- start start 2024-01-26T23:48:10.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.966+0000 7fd71cc31700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd718157860 0x7fd7180cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:10.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.966+0000 7fd71cc31700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7180cea80 con 0x7fd718157490 2024-01-26T23:48:10.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.966+0000 7fd71cc31700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7180cde70 con 0x7fd71806ddb0 2024-01-26T23:48:10.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.966+0000 7fd71cc31700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7180cdc60 con 0x7fd718157860 2024-01-26T23:48:10.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.966+0000 7fd71659c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd718157490 0x7fd71806d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40700/0 (socket says 172.21.15.46:40700) 2024-01-26T23:48:10.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.967+0000 7fd71659c700 1 -- 172.21.15.46:0/4010849189 learned_addr learned my addr 172.21.15.46:0/4010849189 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:10.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.967+0000 7fd715d9b700 1 --2- 172.21.15.46:0/4010849189 >> v2:172.21.15.106:3300/0 conn(0x7fd718157860 0x7fd7180cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:10.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.967+0000 7fd71559a700 1 -- 172.21.15.46:0/4010849189 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494748060 0 0) 0x7fd7180cea80 con 0x7fd718157490 2024-01-26T23:48:10.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.967+0000 7fd71559a700 1 -- 172.21.15.46:0/4010849189 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd704003570 con 0x7fd718157490 2024-01-26T23:48:10.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.967+0000 7fd71559a700 1 -- 172.21.15.46:0/4010849189 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502635897 0 0) 0x7fd7180cde70 con 0x7fd71806ddb0 2024-01-26T23:48:10.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.967+0000 7fd71559a700 1 -- 172.21.15.46:0/4010849189 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7180cea80 con 0x7fd71806ddb0 2024-01-26T23:48:10.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.967+0000 7fd715d9b700 1 -- 172.21.15.46:0/4010849189 >> v1:172.21.15.106:6789/0 conn(0x7fd71806ddb0 legacy=0x7fd7180b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:10.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.967+0000 7fd715d9b700 1 -- 172.21.15.46:0/4010849189 >> v1:172.21.15.46:6789/0 conn(0x7fd718157490 legacy=0x7fd71806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:10.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.967+0000 7fd715d9b700 1 -- 172.21.15.46:0/4010849189 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7180ce150 con 0x7fd718157860 2024-01-26T23:48:10.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.967+0000 7fd715d9b700 1 --2- 172.21.15.46:0/4010849189 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd718157860 0x7fd7180cc600 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd7000098e0 tx=0x7fd7000091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=906d0f4f0e5e3986 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:10.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.967+0000 7fd71559a700 1 -- 172.21.15.46:0/4010849189 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd700011630 con 0x7fd718157860 2024-01-26T23:48:10.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.968+0000 7fd71559a700 1 -- 172.21.15.46:0/4010849189 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd700011790 con 0x7fd718157860 2024-01-26T23:48:10.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.968+0000 7fd71559a700 1 -- 172.21.15.46:0/4010849189 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd700011b30 con 0x7fd718157860 2024-01-26T23:48:10.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.968+0000 7fd71cc31700 1 -- 172.21.15.46:0/4010849189 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd718157860 msgr2=0x7fd7180cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:10.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.968+0000 7fd71cc31700 1 --2- 172.21.15.46:0/4010849189 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd718157860 0x7fd7180cc600 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd7000098e0 tx=0x7fd7000091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:10.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.968+0000 7fd71cc31700 1 -- 172.21.15.46:0/4010849189 shutdown_connections 2024-01-26T23:48:10.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.968+0000 7fd71cc31700 1 --2- 172.21.15.46:0/4010849189 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd718157860 0x7fd7180cc600 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:10.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.968+0000 7fd71cc31700 1 -- 172.21.15.46:0/4010849189 >> 172.21.15.46:0/4010849189 conn(0x7fd718067930 msgr2=0x7fd718067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:10.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.968+0000 7fd71cc31700 1 -- 172.21.15.46:0/4010849189 shutdown_connections 2024-01-26T23:48:10.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.968+0000 7fd71cc31700 1 -- 172.21.15.46:0/4010849189 wait complete. 2024-01-26T23:48:10.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.970+0000 7fd71cc31700 1 Processor -- start 2024-01-26T23:48:10.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.970+0000 7fd71cc31700 1 -- start start 2024-01-26T23:48:10.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.971+0000 7fd71cc31700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd71806ddb0 0x7fd71813e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:10.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.971+0000 7fd71cc31700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd718157490 0x7fd7180c3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:10.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.971+0000 7fd71cc31700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd718157860 0x7fd7180c3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:10.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.971+0000 7fd71cc31700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7181537f0 con 0x7fd718157490 2024-01-26T23:48:10.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.971+0000 7fd71cc31700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd718152be0 con 0x7fd71806ddb0 2024-01-26T23:48:10.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.971+0000 7fd71cc31700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd718152ec0 con 0x7fd718157860 2024-01-26T23:48:10.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.971+0000 7fd715d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd718157490 0x7fd7180c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:10.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.971+0000 7fd715d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd718157490 0x7fd7180c3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38874/0 (socket says 172.21.15.46:38874) 2024-01-26T23:48:10.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.971+0000 7fd715d9b700 1 -- 172.21.15.46:0/898451702 learned_addr learned my addr 172.21.15.46:0/898451702 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:10.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.971+0000 7fd71659c700 1 --2- 172.21.15.46:0/898451702 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd71806ddb0 0x7fd71813e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:10.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.971+0000 7fd716d9d700 1 --2- 172.21.15.46:0/898451702 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd718157860 0x7fd7180c3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:10.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.971+0000 7fd715d9b700 1 -- 172.21.15.46:0/898451702 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd718157860 msgr2=0x7fd7180c3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:10.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.971+0000 7fd715d9b700 1 --2- 172.21.15.46:0/898451702 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd718157860 0x7fd7180c3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:10.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.971+0000 7fd715d9b700 1 -- 172.21.15.46:0/898451702 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd71806ddb0 msgr2=0x7fd71813e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:10.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.971+0000 7fd715d9b700 1 --2- 172.21.15.46:0/898451702 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd71806ddb0 0x7fd71813e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:10.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.971+0000 7fd715d9b700 1 -- 172.21.15.46:0/898451702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd700009420 con 0x7fd718157490 2024-01-26T23:48:10.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.971+0000 7fd71659c700 1 --2- 172.21.15.46:0/898451702 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd71806ddb0 0x7fd71813e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:48:10.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.972+0000 7fd715d9b700 1 --2- 172.21.15.46:0/898451702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd718157490 0x7fd7180c3350 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fd7000093f0 tx=0x7fd700017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:10.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.972+0000 7fd70e7fc700 1 -- 172.21.15.46:0/898451702 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70001f070 con 0x7fd718157490 2024-01-26T23:48:10.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.972+0000 7fd70e7fc700 1 -- 172.21.15.46:0/898451702 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd700000c90 con 0x7fd718157490 2024-01-26T23:48:10.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.972+0000 7fd71cc31700 1 -- 172.21.15.46:0/898451702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7181531a0 con 0x7fd718157490 2024-01-26T23:48:10.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.972+0000 7fd70e7fc700 1 -- 172.21.15.46:0/898451702 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd700025420 con 0x7fd718157490 2024-01-26T23:48:10.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.972+0000 7fd71cc31700 1 -- 172.21.15.46:0/898451702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd71813ec40 con 0x7fd718157490 2024-01-26T23:48:10.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.973+0000 7fd71cc31700 1 -- 172.21.15.46:0/898451702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd718153960 con 0x7fd718157490 2024-01-26T23:48:10.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.974+0000 7fd70e7fc700 1 -- 172.21.15.46:0/898451702 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd700023070 con 0x7fd718157490 2024-01-26T23:48:10.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.974+0000 7fd70e7fc700 1 --2- 172.21.15.46:0/898451702 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd6f8066f30 0x7fd6f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:10.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.974+0000 7fd70e7fc700 1 -- 172.21.15.46:0/898451702 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fd700085950 con 0x7fd718157490 2024-01-26T23:48:10.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.974+0000 7fd71659c700 1 --2- 172.21.15.46:0/898451702 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd6f8066f30 0x7fd6f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:10.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.975+0000 7fd71659c700 1 --2- 172.21.15.46:0/898451702 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd6f8066f30 0x7fd6f80693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd71815d000 tx=0x7fd708009040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:10.979+0000 7fd70e7fc700 1 -- 172.21.15.46:0/898451702 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd7000528c0 con 0x7fd718157490 2024-01-26T23:48:11.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.232+0000 7fd71cc31700 1 -- 172.21.15.46:0/898451702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd71804c710 con 0x7fd718157490 2024-01-26T23:48:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.794+0000 7fd70e7fc700 1 -- 172.21.15.46:0/898451702 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v58) v1 ==== 174+0+0 (secure 0 0 0) 0x7fd700055ee0 con 0x7fd718157490 2024-01-26T23:48:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.800+0000 7fd71cc31700 1 -- 172.21.15.46:0/898451702 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd6f8066f30 msgr2=0x7fd6f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.800+0000 7fd71cc31700 1 --2- 172.21.15.46:0/898451702 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd6f8066f30 0x7fd6f80693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd71815d000 tx=0x7fd708009040 comp rx=0 tx=0).stop 2024-01-26T23:48:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.800+0000 7fd71cc31700 1 -- 172.21.15.46:0/898451702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd718157490 msgr2=0x7fd7180c3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.800+0000 7fd71cc31700 1 --2- 172.21.15.46:0/898451702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd718157490 0x7fd7180c3350 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fd7000093f0 tx=0x7fd700017000 comp rx=0 tx=0).stop 2024-01-26T23:48:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.800+0000 7fd71cc31700 1 -- 172.21.15.46:0/898451702 shutdown_connections 2024-01-26T23:48:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.800+0000 7fd71cc31700 1 --2- 172.21.15.46:0/898451702 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd6f8066f30 0x7fd6f80693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.800+0000 7fd71cc31700 1 --2- 172.21.15.46:0/898451702 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd718157860 0x7fd7180c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.800+0000 7fd71cc31700 1 --2- 172.21.15.46:0/898451702 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd71806ddb0 0x7fd71813e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.800+0000 7fd71cc31700 1 --2- 172.21.15.46:0/898451702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd718157490 0x7fd7180c3350 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.800+0000 7fd71cc31700 1 -- 172.21.15.46:0/898451702 >> 172.21.15.46:0/898451702 conn(0x7fd718067930 msgr2=0x7fd718151c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.801+0000 7fd71cc31700 1 -- 172.21.15.46:0/898451702 shutdown_connections 2024-01-26T23:48:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.801+0000 7fd71cc31700 1 -- 172.21.15.46:0/898451702 wait complete. 2024-01-26T23:48:11.804 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 15 and data pool 16 2024-01-26T23:48:11.820 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:48:11.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.954+0000 7fd9e953a700 1 Processor -- start 2024-01-26T23:48:11.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.954+0000 7fd9e953a700 1 -- start start 2024-01-26T23:48:11.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.954+0000 7fd9e953a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd9e415a1d0 0x7fd9e40afc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:11.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.954+0000 7fd9e953a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd9e40b0160 0x7fd9e4157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:11.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.954+0000 7fd9e953a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e41509d0 con 0x7fd9e40af850 2024-01-26T23:48:11.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.954+0000 7fd9e953a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e4157820 con 0x7fd9e40b0160 2024-01-26T23:48:11.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.954+0000 7fd9e953a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e40b10d0 con 0x7fd9e415a1d0 2024-01-26T23:48:11.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.955+0000 7fd9e37fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd9e40b0160 0x7fd9e4157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:11.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.955+0000 7fd9e2ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd9e40af850 0x7fd9e4159ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40706/0 (socket says 172.21.15.46:40706) 2024-01-26T23:48:11.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.955+0000 7fd9e2ffd700 1 -- 172.21.15.46:0/448837335 learned_addr learned my addr 172.21.15.46:0/448837335 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:11.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.955+0000 7fd9e27fc700 1 --2- 172.21.15.46:0/448837335 >> v2:172.21.15.106:3300/0 conn(0x7fd9e415a1d0 0x7fd9e40afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:11.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.955+0000 7fd9e37fe700 1 -- 172.21.15.46:0/448837335 >> v2:172.21.15.106:3300/0 conn(0x7fd9e415a1d0 msgr2=0x7fd9e40afc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:11.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.955+0000 7fd9e37fe700 1 --2- 172.21.15.46:0/448837335 >> v2:172.21.15.106:3300/0 conn(0x7fd9e415a1d0 0x7fd9e40afc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:11.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.955+0000 7fd9e37fe700 1 -- 172.21.15.46:0/448837335 >> v1:172.21.15.46:6789/0 conn(0x7fd9e40af850 legacy=0x7fd9e4159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:11.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.955+0000 7fd9e1ffb700 1 -- 172.21.15.46:0/448837335 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486410465 0 0) 0x7fd9d4001490 con 0x7fd9e40af850 2024-01-26T23:48:11.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.955+0000 7fd9e37fe700 1 -- 172.21.15.46:0/448837335 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e40b0590 con 0x7fd9e40b0160 2024-01-26T23:48:11.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.955+0000 7fd9e37fe700 1 --2- 172.21.15.46:0/448837335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9e40b0160 0x7fd9e4157cb0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fd9e406ee40 tx=0x7fd9dc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b285a5987e61abe4 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:11.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.955+0000 7fd9e1ffb700 1 -- 172.21.15.46:0/448837335 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc00b6b0 con 0x7fd9e40b0160 2024-01-26T23:48:11.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.955+0000 7fd9e1ffb700 1 -- 172.21.15.46:0/448837335 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc00b810 con 0x7fd9e40b0160 2024-01-26T23:48:11.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.955+0000 7fd9e1ffb700 1 -- 172.21.15.46:0/448837335 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc00bbb0 con 0x7fd9e40b0160 2024-01-26T23:48:11.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.956+0000 7fd9e953a700 1 -- 172.21.15.46:0/448837335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9e40b0160 msgr2=0x7fd9e4157cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:11.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.956+0000 7fd9e953a700 1 --2- 172.21.15.46:0/448837335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9e40b0160 0x7fd9e4157cb0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fd9e406ee40 tx=0x7fd9dc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:11.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.956+0000 7fd9e953a700 1 -- 172.21.15.46:0/448837335 shutdown_connections 2024-01-26T23:48:11.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.956+0000 7fd9e953a700 1 --2- 172.21.15.46:0/448837335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9e40b0160 0x7fd9e4157cb0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:11.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.956+0000 7fd9e953a700 1 --2- 172.21.15.46:0/448837335 >> v2:172.21.15.106:3300/0 conn(0x7fd9e415a1d0 0x7fd9e40afc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:11.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.956+0000 7fd9e953a700 1 -- 172.21.15.46:0/448837335 >> 172.21.15.46:0/448837335 conn(0x7fd9e4068400 msgr2=0x7fd9e40cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:11.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.956+0000 7fd9e953a700 1 -- 172.21.15.46:0/448837335 shutdown_connections 2024-01-26T23:48:11.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.956+0000 7fd9e953a700 1 -- 172.21.15.46:0/448837335 wait complete. 2024-01-26T23:48:11.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.958+0000 7fd9e953a700 1 Processor -- start 2024-01-26T23:48:11.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.958+0000 7fd9e953a700 1 -- start start 2024-01-26T23:48:11.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.958+0000 7fd9e953a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9e40af850 0x7fd9e414ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:11.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.958+0000 7fd9e953a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9e40b0160 0x7fd9e4154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:11.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.958+0000 7fd9e953a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9e415a1d0 0x7fd9e4154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:11.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.958+0000 7fd9e953a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e415a8e0 con 0x7fd9e415a1d0 2024-01-26T23:48:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.958+0000 7fd9e953a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e4160a60 con 0x7fd9e40b0160 2024-01-26T23:48:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.958+0000 7fd9e953a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e41601c0 con 0x7fd9e40af850 2024-01-26T23:48:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.959+0000 7fd9e37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9e415a1d0 0x7fd9e4154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.959+0000 7fd9e37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9e415a1d0 0x7fd9e4154a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38888/0 (socket says 172.21.15.46:38888) 2024-01-26T23:48:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.959+0000 7fd9e37fe700 1 -- 172.21.15.46:0/3371080850 learned_addr learned my addr 172.21.15.46:0/3371080850 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.959+0000 7fd9e27fc700 1 --2- 172.21.15.46:0/3371080850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9e40b0160 0x7fd9e4154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.959+0000 7fd9e2ffd700 1 --2- 172.21.15.46:0/3371080850 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9e40af850 0x7fd9e414ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.959+0000 7fd9e37fe700 1 -- 172.21.15.46:0/3371080850 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9e40af850 msgr2=0x7fd9e414ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.959+0000 7fd9e37fe700 1 --2- 172.21.15.46:0/3371080850 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9e40af850 0x7fd9e414ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.959+0000 7fd9e37fe700 1 -- 172.21.15.46:0/3371080850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9e40b0160 msgr2=0x7fd9e4154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.959+0000 7fd9e37fe700 1 --2- 172.21.15.46:0/3371080850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9e40b0160 0x7fd9e4154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.959+0000 7fd9e37fe700 1 -- 172.21.15.46:0/3371080850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9dc009420 con 0x7fd9e415a1d0 2024-01-26T23:48:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.959+0000 7fd9e37fe700 1 --2- 172.21.15.46:0/3371080850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9e415a1d0 0x7fd9e4154a50 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4070360 tx=0x7fd9dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.959+0000 7fd9caffd700 1 -- 172.21.15.46:0/3371080850 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc011420 con 0x7fd9e415a1d0 2024-01-26T23:48:11.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.959+0000 7fd9caffd700 1 -- 172.21.15.46:0/3371080850 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc011580 con 0x7fd9e415a1d0 2024-01-26T23:48:11.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.959+0000 7fd9caffd700 1 -- 172.21.15.46:0/3371080850 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc011940 con 0x7fd9e415a1d0 2024-01-26T23:48:11.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.959+0000 7fd9e953a700 1 -- 172.21.15.46:0/3371080850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e41604a0 con 0x7fd9e415a1d0 2024-01-26T23:48:11.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.959+0000 7fd9e953a700 1 -- 172.21.15.46:0/3371080850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e406be20 con 0x7fd9e415a1d0 2024-01-26T23:48:11.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.960+0000 7fd9e953a700 1 -- 172.21.15.46:0/3371080850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e404c710 con 0x7fd9e415a1d0 2024-01-26T23:48:11.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.961+0000 7fd9caffd700 1 -- 172.21.15.46:0/3371080850 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd9dc011cb0 con 0x7fd9e415a1d0 2024-01-26T23:48:11.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.962+0000 7fd9caffd700 1 --2- 172.21.15.46:0/3371080850 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd9d0066f30 0x7fd9d00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:11.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.962+0000 7fd9caffd700 1 -- 172.21.15.46:0/3371080850 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fd9dc0867e0 con 0x7fd9e415a1d0 2024-01-26T23:48:11.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.962+0000 7fd9e2ffd700 1 --2- 172.21.15.46:0/3371080850 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd9d0066f30 0x7fd9d00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:11.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.963+0000 7fd9e2ffd700 1 --2- 172.21.15.46:0/3371080850 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd9d0066f30 0x7fd9d00693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4007240 tx=0x7fd9d4001970 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:11.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:11.966+0000 7fd9caffd700 1 -- 172.21.15.46:0/3371080850 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd9dc053760 con 0x7fd9e415a1d0 2024-01-26T23:48:12.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.171+0000 7fd9e953a700 1 -- 172.21.15.46:0/3371080850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9e4068800 con 0x7fd9e415a1d0 2024-01-26T23:48:12.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.172+0000 7fd9caffd700 1 -- 172.21.15.46:0/3371080850 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16176 (secure 0 0 0) 0x7fd9dc056d80 con 0x7fd9e415a1d0 2024-01-26T23:48:12.173 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:48:12.173 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":129,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:48:11.234196+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:48:09.662590+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-01-26T23:48:10.200635+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:48:12.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.174+0000 7fd9e953a700 1 -- 172.21.15.46:0/3371080850 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd9d0066f30 msgr2=0x7fd9d00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:12.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.174+0000 7fd9e953a700 1 --2- 172.21.15.46:0/3371080850 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd9d0066f30 0x7fd9d00693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4007240 tx=0x7fd9d4001970 comp rx=0 tx=0).stop 2024-01-26T23:48:12.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.174+0000 7fd9e953a700 1 -- 172.21.15.46:0/3371080850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9e415a1d0 msgr2=0x7fd9e4154a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:12.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.174+0000 7fd9e953a700 1 --2- 172.21.15.46:0/3371080850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9e415a1d0 0x7fd9e4154a50 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4070360 tx=0x7fd9dc017000 comp rx=0 tx=0).stop 2024-01-26T23:48:12.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.174+0000 7fd9e953a700 1 -- 172.21.15.46:0/3371080850 shutdown_connections 2024-01-26T23:48:12.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.174+0000 7fd9e953a700 1 --2- 172.21.15.46:0/3371080850 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd9d0066f30 0x7fd9d00693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.174+0000 7fd9e953a700 1 --2- 172.21.15.46:0/3371080850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9e415a1d0 0x7fd9e4154a50 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.174+0000 7fd9e953a700 1 --2- 172.21.15.46:0/3371080850 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9e40af850 0x7fd9e414ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.174+0000 7fd9e953a700 1 --2- 172.21.15.46:0/3371080850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9e40b0160 0x7fd9e4154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.174+0000 7fd9e953a700 1 -- 172.21.15.46:0/3371080850 >> 172.21.15.46:0/3371080850 conn(0x7fd9e4068400 msgr2=0x7fd9e40cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:12.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.174+0000 7fd9e953a700 1 -- 172.21.15.46:0/3371080850 shutdown_connections 2024-01-26T23:48:12.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.174+0000 7fd9e953a700 1 -- 172.21.15.46:0/3371080850 wait complete. 2024-01-26T23:48:12.189 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:48:12.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.320+0000 7f33465b8700 1 Processor -- start 2024-01-26T23:48:12.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.320+0000 7f33465b8700 1 -- start start 2024-01-26T23:48:12.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.321+0000 7f33465b8700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3340157340 0x7f334015bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:12.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.321+0000 7f33465b8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33400ae520 0x7f33400b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:12.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.321+0000 7f33465b8700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3340158ae0 con 0x7f33400b3fb0 2024-01-26T23:48:12.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.321+0000 7f33465b8700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33400b4430 con 0x7f3340157340 2024-01-26T23:48:12.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.321+0000 7f33465b8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334015b650 con 0x7f33400ae520 2024-01-26T23:48:12.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.321+0000 7f333ffff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3340157340 0x7f334015bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:12.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.321+0000 7f3344b55700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f33400b3fb0 0x7f3340157710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:42516/0 (socket says 172.21.15.46:42516) 2024-01-26T23:48:12.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.321+0000 7f333f7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33400ae520 0x7f33400b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:12.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.321+0000 7f3344b55700 1 -- 172.21.15.46:0/429944351 learned_addr learned my addr 172.21.15.46:0/429944351 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:12.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.322+0000 7f333ffff700 1 -- 172.21.15.46:0/429944351 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33400ae520 msgr2=0x7f33400b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:12.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.322+0000 7f333ffff700 1 --2- 172.21.15.46:0/429944351 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33400ae520 0x7f33400b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.322+0000 7f333ffff700 1 -- 172.21.15.46:0/429944351 >> v1:172.21.15.106:6789/0 conn(0x7f33400b3fb0 legacy=0x7f3340157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:12.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.322+0000 7f333effd700 1 -- 172.21.15.46:0/429944351 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392638410 0 0) 0x7f3340158ae0 con 0x7f33400b3fb0 2024-01-26T23:48:12.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.322+0000 7f333ffff700 1 -- 172.21.15.46:0/429944351 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334015c020 con 0x7f3340157340 2024-01-26T23:48:12.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.322+0000 7f333ffff700 1 --2- 172.21.15.46:0/429944351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3340157340 0x7f334015bae0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f33300098e0 tx=0x7f33300091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81bbe18ba4bbe571 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:12.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.322+0000 7f333effd700 1 -- 172.21.15.46:0/429944351 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333000b640 con 0x7f3340157340 2024-01-26T23:48:12.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.322+0000 7f333effd700 1 -- 172.21.15.46:0/429944351 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333000b7a0 con 0x7f3340157340 2024-01-26T23:48:12.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.322+0000 7f333effd700 1 -- 172.21.15.46:0/429944351 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333000bb40 con 0x7f3340157340 2024-01-26T23:48:12.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.323+0000 7f33465b8700 1 -- 172.21.15.46:0/429944351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3340157340 msgr2=0x7f334015bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:12.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.323+0000 7f33465b8700 1 --2- 172.21.15.46:0/429944351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3340157340 0x7f334015bae0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f33300098e0 tx=0x7f33300091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.323+0000 7f33465b8700 1 -- 172.21.15.46:0/429944351 shutdown_connections 2024-01-26T23:48:12.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.323+0000 7f33465b8700 1 --2- 172.21.15.46:0/429944351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3340157340 0x7f334015bae0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.323+0000 7f33465b8700 1 --2- 172.21.15.46:0/429944351 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33400ae520 0x7f33400b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.323+0000 7f33465b8700 1 -- 172.21.15.46:0/429944351 >> 172.21.15.46:0/429944351 conn(0x7f33400ce530 msgr2=0x7f33400ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:12.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.323+0000 7f33465b8700 1 -- 172.21.15.46:0/429944351 shutdown_connections 2024-01-26T23:48:12.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.323+0000 7f33465b8700 1 -- 172.21.15.46:0/429944351 wait complete. 2024-01-26T23:48:12.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.324+0000 7f33465b8700 1 Processor -- start 2024-01-26T23:48:12.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.324+0000 7f33465b8700 1 -- start start 2024-01-26T23:48:12.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.325+0000 7f33465b8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33400ae520 0x7f33400c6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:12.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.325+0000 7f33465b8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33400b3fb0 0x7f3340155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:12.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.325+0000 7f33465b8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3340157340 0x7f33400ad6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:12.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.325+0000 7f33465b8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33400c8120 con 0x7f3340157340 2024-01-26T23:48:12.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.325+0000 7f33465b8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33400c7c90 con 0x7f33400ae520 2024-01-26T23:48:12.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.325+0000 7f33465b8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33400c6db0 con 0x7f33400b3fb0 2024-01-26T23:48:12.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.325+0000 7f3344b55700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3340157340 0x7f33400ad6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:12.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.325+0000 7f3344b55700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3340157340 0x7f33400ad6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38900/0 (socket says 172.21.15.46:38900) 2024-01-26T23:48:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.325+0000 7f3344b55700 1 -- 172.21.15.46:0/134951228 learned_addr learned my addr 172.21.15.46:0/134951228 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.325+0000 7f333ffff700 1 --2- 172.21.15.46:0/134951228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33400ae520 0x7f33400c6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.325+0000 7f333f7fe700 1 --2- 172.21.15.46:0/134951228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33400b3fb0 0x7f3340155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.326+0000 7f3344b55700 1 -- 172.21.15.46:0/134951228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33400b3fb0 msgr2=0x7f3340155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.326+0000 7f3344b55700 1 --2- 172.21.15.46:0/134951228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33400b3fb0 0x7f3340155c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.326+0000 7f3344b55700 1 -- 172.21.15.46:0/134951228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33400ae520 msgr2=0x7f33400c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.326+0000 7f3344b55700 1 --2- 172.21.15.46:0/134951228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33400ae520 0x7f33400c6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.326+0000 7f3344b55700 1 -- 172.21.15.46:0/134951228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3330009420 con 0x7f3340157340 2024-01-26T23:48:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.326+0000 7f3344b55700 1 --2- 172.21.15.46:0/134951228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3340157340 0x7f33400ad6f0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f333800cc90 tx=0x7f3338001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.326+0000 7f3327fff700 1 -- 172.21.15.46:0/134951228 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333800a870 con 0x7f3340157340 2024-01-26T23:48:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.326+0000 7f3327fff700 1 -- 172.21.15.46:0/134951228 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333800a9d0 con 0x7f3340157340 2024-01-26T23:48:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.326+0000 7f33465b8700 1 -- 172.21.15.46:0/134951228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33400c7080 con 0x7f3340157340 2024-01-26T23:48:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.326+0000 7f3327fff700 1 -- 172.21.15.46:0/134951228 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333800ad90 con 0x7f3340157340 2024-01-26T23:48:12.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.326+0000 7f33465b8700 1 -- 172.21.15.46:0/134951228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33400c7640 con 0x7f3340157340 2024-01-26T23:48:12.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.327+0000 7f33465b8700 1 -- 172.21.15.46:0/134951228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33400c7e00 con 0x7f3340157340 2024-01-26T23:48:12.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.328+0000 7f3327fff700 1 -- 172.21.15.46:0/134951228 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3338007300 con 0x7f3340157340 2024-01-26T23:48:12.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.328+0000 7f3327fff700 1 --2- 172.21.15.46:0/134951228 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f332c066f30 0x7f332c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:12.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.328+0000 7f3327fff700 1 -- 172.21.15.46:0/134951228 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f3338084b80 con 0x7f3340157340 2024-01-26T23:48:12.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.328+0000 7f333ffff700 1 --2- 172.21.15.46:0/134951228 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f332c066f30 0x7f332c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:12.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.329+0000 7f333ffff700 1 --2- 172.21.15.46:0/134951228 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f332c066f30 0x7f332c0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f333000a010 tx=0x7f3330017000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:12.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.333+0000 7f3327fff700 1 -- 172.21.15.46:0/134951228 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3338051b10 con 0x7f3340157340 2024-01-26T23:48:12.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.539+0000 7f33465b8700 1 -- 172.21.15.46:0/134951228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f334004c710 con 0x7f3340157340 2024-01-26T23:48:12.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.540+0000 7f3327fff700 1 -- 172.21.15.46:0/134951228 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16176 (secure 0 0 0) 0x7f3338055130 con 0x7f3340157340 2024-01-26T23:48:12.541 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:48:12.541 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":129,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:48:11.234196+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:48:09.662590+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-01-26T23:48:10.200635+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:48:12.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.542+0000 7f33465b8700 1 -- 172.21.15.46:0/134951228 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f332c066f30 msgr2=0x7f332c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:12.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.542+0000 7f33465b8700 1 --2- 172.21.15.46:0/134951228 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f332c066f30 0x7f332c0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f333000a010 tx=0x7f3330017000 comp rx=0 tx=0).stop 2024-01-26T23:48:12.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.542+0000 7f33465b8700 1 -- 172.21.15.46:0/134951228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3340157340 msgr2=0x7f33400ad6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:12.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.542+0000 7f33465b8700 1 --2- 172.21.15.46:0/134951228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3340157340 0x7f33400ad6f0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f333800cc90 tx=0x7f3338001a20 comp rx=0 tx=0).stop 2024-01-26T23:48:12.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.542+0000 7f33465b8700 1 -- 172.21.15.46:0/134951228 shutdown_connections 2024-01-26T23:48:12.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.542+0000 7f33465b8700 1 --2- 172.21.15.46:0/134951228 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f332c066f30 0x7f332c0693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.542+0000 7f33465b8700 1 --2- 172.21.15.46:0/134951228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3340157340 0x7f33400ad6f0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.542+0000 7f33465b8700 1 --2- 172.21.15.46:0/134951228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33400ae520 0x7f33400c6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.542+0000 7f33465b8700 1 --2- 172.21.15.46:0/134951228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33400b3fb0 0x7f3340155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.542+0000 7f33465b8700 1 -- 172.21.15.46:0/134951228 >> 172.21.15.46:0/134951228 conn(0x7f33400ce530 msgr2=0x7f3340159c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:12.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.542+0000 7f33465b8700 1 -- 172.21.15.46:0/134951228 shutdown_connections 2024-01-26T23:48:12.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.542+0000 7f33465b8700 1 -- 172.21.15.46:0/134951228 wait complete. 2024-01-26T23:48:12.557 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-26T23:48:12.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.691+0000 7fb44d61a700 1 Processor -- start 2024-01-26T23:48:12.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.692+0000 7fb44d61a700 1 -- start start 2024-01-26T23:48:12.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.692+0000 7fb44d61a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb44806deb0 0x7fb4480ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:12.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.692+0000 7fb44d61a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4481599b0 0x7fb44815de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:12.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.692+0000 7fb44d61a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44815e3d0 con 0x7fb4480ca4f0 2024-01-26T23:48:12.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.692+0000 7fb44d61a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb448159e30 con 0x7fb4481599b0 2024-01-26T23:48:12.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.692+0000 7fb44d61a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb44806ef50 con 0x7fb44806deb0 2024-01-26T23:48:12.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.693+0000 7fb4477fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4481599b0 0x7fb44815de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:12.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.693+0000 7fb4477fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4481599b0 0x7fb44815de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38908/0 (socket says 172.21.15.46:38908) 2024-01-26T23:48:12.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.693+0000 7fb4477fe700 1 -- 172.21.15.46:0/1777546666 learned_addr learned my addr 172.21.15.46:0/1777546666 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:12.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.693+0000 7fb4467fc700 1 --2- 172.21.15.46:0/1777546666 >> v2:172.21.15.106:3300/0 conn(0x7fb44806deb0 0x7fb4480ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:12.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.693+0000 7fb4477fe700 1 -- 172.21.15.46:0/1777546666 >> v2:172.21.15.106:3300/0 conn(0x7fb44806deb0 msgr2=0x7fb4480ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:12.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.693+0000 7fb4477fe700 1 --2- 172.21.15.46:0/1777546666 >> v2:172.21.15.106:3300/0 conn(0x7fb44806deb0 0x7fb4480ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.693+0000 7fb4477fe700 1 -- 172.21.15.46:0/1777546666 >> v1:172.21.15.106:6789/0 conn(0x7fb4480ca4f0 legacy=0x7fb4481592a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:12.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.693+0000 7fb4477fe700 1 -- 172.21.15.46:0/1777546666 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44806f3e0 con 0x7fb4481599b0 2024-01-26T23:48:12.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.693+0000 7fb4477fe700 1 --2- 172.21.15.46:0/1777546666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4481599b0 0x7fb44815de90 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fb448064170 tx=0x7fb440009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29bf209a5eef2a48 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:12.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.693+0000 7fb445ffb700 1 -- 172.21.15.46:0/1777546666 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44000e070 con 0x7fb4481599b0 2024-01-26T23:48:12.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.693+0000 7fb445ffb700 1 -- 172.21.15.46:0/1777546666 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44000ba20 con 0x7fb4481599b0 2024-01-26T23:48:12.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.693+0000 7fb445ffb700 1 -- 172.21.15.46:0/1777546666 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4400092e0 con 0x7fb4481599b0 2024-01-26T23:48:12.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.694+0000 7fb44d61a700 1 -- 172.21.15.46:0/1777546666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4481599b0 msgr2=0x7fb44815de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:12.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.694+0000 7fb44d61a700 1 --2- 172.21.15.46:0/1777546666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4481599b0 0x7fb44815de90 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fb448064170 tx=0x7fb440009840 comp rx=0 tx=0).stop 2024-01-26T23:48:12.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.694+0000 7fb44d61a700 1 -- 172.21.15.46:0/1777546666 shutdown_connections 2024-01-26T23:48:12.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.694+0000 7fb44d61a700 1 --2- 172.21.15.46:0/1777546666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4481599b0 0x7fb44815de90 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.694+0000 7fb44d61a700 1 --2- 172.21.15.46:0/1777546666 >> v2:172.21.15.106:3300/0 conn(0x7fb44806deb0 0x7fb4480ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.694+0000 7fb44d61a700 1 -- 172.21.15.46:0/1777546666 >> 172.21.15.46:0/1777546666 conn(0x7fb448063580 msgr2=0x7fb448063980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:12.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.694+0000 7fb44d61a700 1 -- 172.21.15.46:0/1777546666 shutdown_connections 2024-01-26T23:48:12.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.694+0000 7fb44d61a700 1 -- 172.21.15.46:0/1777546666 wait complete. 2024-01-26T23:48:12.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.695+0000 7fb44d61a700 1 Processor -- start 2024-01-26T23:48:12.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.695+0000 7fb44d61a700 1 -- start start 2024-01-26T23:48:12.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.696+0000 7fb44d61a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb44806deb0 0x7fb4480c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:12.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.696+0000 7fb44d61a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4480ca4f0 0x7fb4480c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:12.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.696+0000 7fb44d61a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4481599b0 0x7fb4480a9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:12.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.696+0000 7fb44d61a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480b0720 con 0x7fb4480ca4f0 2024-01-26T23:48:12.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.696+0000 7fb44d61a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480b1260 con 0x7fb44806deb0 2024-01-26T23:48:12.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.696+0000 7fb44d61a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480b0930 con 0x7fb4481599b0 2024-01-26T23:48:12.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.696+0000 7fb4467fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4480ca4f0 0x7fb4480c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:12.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.696+0000 7fb4467fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4480ca4f0 0x7fb4480c6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38920/0 (socket says 172.21.15.46:38920) 2024-01-26T23:48:12.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.696+0000 7fb4467fc700 1 -- 172.21.15.46:0/3224684011 learned_addr learned my addr 172.21.15.46:0/3224684011 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:12.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.696+0000 7fb446ffd700 1 --2- 172.21.15.46:0/3224684011 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb44806deb0 0x7fb4480c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:12.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.696+0000 7fb4477fe700 1 --2- 172.21.15.46:0/3224684011 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4481599b0 0x7fb4480a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:12.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.697+0000 7fb4467fc700 1 -- 172.21.15.46:0/3224684011 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4481599b0 msgr2=0x7fb4480a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:12.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.697+0000 7fb4467fc700 1 --2- 172.21.15.46:0/3224684011 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4481599b0 0x7fb4480a9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.697+0000 7fb4467fc700 1 -- 172.21.15.46:0/3224684011 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb44806deb0 msgr2=0x7fb4480c96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:12.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.697+0000 7fb4467fc700 1 --2- 172.21.15.46:0/3224684011 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb44806deb0 0x7fb4480c96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:12.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.697+0000 7fb4467fc700 1 -- 172.21.15.46:0/3224684011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb440009ae0 con 0x7fb4480ca4f0 2024-01-26T23:48:12.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.697+0000 7fb4467fc700 1 --2- 172.21.15.46:0/3224684011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4480ca4f0 0x7fb4480c6970 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fb43800ec50 tx=0x7fb4380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:12.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.697+0000 7fb43effd700 1 -- 172.21.15.46:0/3224684011 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43800c850 con 0x7fb4480ca4f0 2024-01-26T23:48:12.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.697+0000 7fb43effd700 1 -- 172.21.15.46:0/3224684011 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43800c9b0 con 0x7fb4480ca4f0 2024-01-26T23:48:12.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.697+0000 7fb44d61a700 1 -- 172.21.15.46:0/3224684011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4480b0c10 con 0x7fb4480ca4f0 2024-01-26T23:48:12.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.697+0000 7fb43effd700 1 -- 172.21.15.46:0/3224684011 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43800cd70 con 0x7fb4480ca4f0 2024-01-26T23:48:12.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.697+0000 7fb44d61a700 1 -- 172.21.15.46:0/3224684011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4481582c0 con 0x7fb4480ca4f0 2024-01-26T23:48:12.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.698+0000 7fb44d61a700 1 -- 172.21.15.46:0/3224684011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb44804c710 con 0x7fb4480ca4f0 2024-01-26T23:48:12.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.699+0000 7fb43effd700 1 -- 172.21.15.46:0/3224684011 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb438013680 con 0x7fb4480ca4f0 2024-01-26T23:48:12.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.699+0000 7fb43effd700 1 --2- 172.21.15.46:0/3224684011 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb428066f30 0x7fb4280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:12.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.699+0000 7fb43effd700 1 -- 172.21.15.46:0/3224684011 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fb438084b20 con 0x7fb4480ca4f0 2024-01-26T23:48:12.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.700+0000 7fb446ffd700 1 --2- 172.21.15.46:0/3224684011 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb428066f30 0x7fb4280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:12.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.700+0000 7fb446ffd700 1 --2- 172.21.15.46:0/3224684011 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb428066f30 0x7fb4280693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb430003a80 tx=0x7fb430007ec0 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:12.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.704+0000 7fb43effd700 1 -- 172.21.15.46:0/3224684011 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb438051a30 con 0x7fb4480ca4f0 2024-01-26T23:48:12.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:12.960+0000 7fb44d61a700 1 -- 172.21.15.46:0/3224684011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb4480b13a0 con 0x7fb4480ca4f0 2024-01-26T23:48:13.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.815+0000 7fb43effd700 1 -- 172.21.15.46:0/3224684011 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v61) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb4380170b0 con 0x7fb4480ca4f0 2024-01-26T23:48:13.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.818+0000 7fb44d61a700 1 -- 172.21.15.46:0/3224684011 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb428066f30 msgr2=0x7fb4280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.818+0000 7fb44d61a700 1 --2- 172.21.15.46:0/3224684011 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb428066f30 0x7fb4280693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb430003a80 tx=0x7fb430007ec0 comp rx=0 tx=0).stop 2024-01-26T23:48:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.818+0000 7fb44d61a700 1 -- 172.21.15.46:0/3224684011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4480ca4f0 msgr2=0x7fb4480c6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.818+0000 7fb44d61a700 1 --2- 172.21.15.46:0/3224684011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4480ca4f0 0x7fb4480c6970 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fb43800ec50 tx=0x7fb4380091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.818+0000 7fb44d61a700 1 -- 172.21.15.46:0/3224684011 shutdown_connections 2024-01-26T23:48:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.818+0000 7fb44d61a700 1 --2- 172.21.15.46:0/3224684011 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb428066f30 0x7fb4280693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.818+0000 7fb44d61a700 1 --2- 172.21.15.46:0/3224684011 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4481599b0 0x7fb4480a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.818+0000 7fb44d61a700 1 --2- 172.21.15.46:0/3224684011 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb44806deb0 0x7fb4480c96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.818+0000 7fb44d61a700 1 --2- 172.21.15.46:0/3224684011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4480ca4f0 0x7fb4480c6970 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.818+0000 7fb44d61a700 1 -- 172.21.15.46:0/3224684011 >> 172.21.15.46:0/3224684011 conn(0x7fb448063580 msgr2=0x7fb44815ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.818+0000 7fb44d61a700 1 -- 172.21.15.46:0/3224684011 shutdown_connections 2024-01-26T23:48:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.818+0000 7fb44d61a700 1 -- 172.21.15.46:0/3224684011 wait complete. 2024-01-26T23:48:13.837 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-26T23:48:13.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.974+0000 7fa7e49a2700 1 Processor -- start 2024-01-26T23:48:13.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.974+0000 7fa7e49a2700 1 -- start start 2024-01-26T23:48:13.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.975+0000 7fa7e49a2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7e0157840 0x7fa7e00ca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:13.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.975+0000 7fa7e49a2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa7e0068040 0x7fa7e00b0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:13.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.975+0000 7fa7e49a2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e00b02d0 con 0x7fa7e006c110 2024-01-26T23:48:13.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.975+0000 7fa7e49a2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e006c680 con 0x7fa7e0068040 2024-01-26T23:48:13.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.975+0000 7fa7e49a2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e006c960 con 0x7fa7e0157840 2024-01-26T23:48:13.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.975+0000 7fa7ded9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa7e0068040 0x7fa7e00b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:13.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.975+0000 7fa7ded9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa7e0068040 0x7fa7e00b0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38924/0 (socket says 172.21.15.46:38924) 2024-01-26T23:48:13.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.975+0000 7fa7ded9d700 1 -- 172.21.15.46:0/576838480 learned_addr learned my addr 172.21.15.46:0/576838480 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:13.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.975+0000 7fa7de59c700 1 --2- 172.21.15.46:0/576838480 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7e0157840 0x7fa7e00ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:13.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.976+0000 7fa7ded9d700 1 -- 172.21.15.46:0/576838480 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7e0157840 msgr2=0x7fa7e00ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:13.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.976+0000 7fa7ded9d700 1 --2- 172.21.15.46:0/576838480 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7e0157840 0x7fa7e00ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:13.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.976+0000 7fa7ded9d700 1 -- 172.21.15.46:0/576838480 >> v1:172.21.15.106:6789/0 conn(0x7fa7e006c110 legacy=0x7fa7e0067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:13.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.976+0000 7fa7ded9d700 1 -- 172.21.15.46:0/576838480 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e006cc40 con 0x7fa7e0068040 2024-01-26T23:48:13.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.976+0000 7fa7ded9d700 1 --2- 172.21.15.46:0/576838480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7e0068040 0x7fa7e00b0760 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8009f20 tx=0x7fa7d8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ebe6d4854fa1ed5 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:13.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.976+0000 7fa7dd59a700 1 -- 172.21.15.46:0/576838480 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d800e070 con 0x7fa7e0068040 2024-01-26T23:48:13.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.976+0000 7fa7dd59a700 1 -- 172.21.15.46:0/576838480 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d800ba40 con 0x7fa7e0068040 2024-01-26T23:48:13.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.976+0000 7fa7dd59a700 1 -- 172.21.15.46:0/576838480 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d80092e0 con 0x7fa7e0068040 2024-01-26T23:48:13.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.976+0000 7fa7e49a2700 1 -- 172.21.15.46:0/576838480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7e0068040 msgr2=0x7fa7e00b0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:13.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.976+0000 7fa7e49a2700 1 --2- 172.21.15.46:0/576838480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7e0068040 0x7fa7e00b0760 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8009f20 tx=0x7fa7d8009840 comp rx=0 tx=0).stop 2024-01-26T23:48:13.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.977+0000 7fa7e49a2700 1 -- 172.21.15.46:0/576838480 shutdown_connections 2024-01-26T23:48:13.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.977+0000 7fa7e49a2700 1 --2- 172.21.15.46:0/576838480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7e0068040 0x7fa7e00b0760 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:13.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.977+0000 7fa7e49a2700 1 --2- 172.21.15.46:0/576838480 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7e0157840 0x7fa7e00ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:13.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.977+0000 7fa7e49a2700 1 -- 172.21.15.46:0/576838480 >> 172.21.15.46:0/576838480 conn(0x7fa7e00cdf50 msgr2=0x7fa7e00c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:13.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.977+0000 7fa7e49a2700 1 -- 172.21.15.46:0/576838480 shutdown_connections 2024-01-26T23:48:13.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.977+0000 7fa7e49a2700 1 -- 172.21.15.46:0/576838480 wait complete. 2024-01-26T23:48:13.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.978+0000 7fa7e49a2700 1 Processor -- start 2024-01-26T23:48:13.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.978+0000 7fa7e49a2700 1 -- start start 2024-01-26T23:48:13.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7e49a2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7e0068040 0x7fa7e00c4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:13.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7e49a2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7e006c110 0x7fa7e00c4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:13.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7e49a2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7e0157840 0x7fa7e00c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:13.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7e49a2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e00aeff0 con 0x7fa7e0157840 2024-01-26T23:48:13.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7e49a2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e00ae3e0 con 0x7fa7e006c110 2024-01-26T23:48:13.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7e49a2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e00ae6c0 con 0x7fa7e0068040 2024-01-26T23:48:13.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7ded9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7e0157840 0x7fa7e00c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:13.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7ded9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7e0157840 0x7fa7e00c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38940/0 (socket says 172.21.15.46:38940) 2024-01-26T23:48:13.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7ded9d700 1 -- 172.21.15.46:0/1529059996 learned_addr learned my addr 172.21.15.46:0/1529059996 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:13.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7de59c700 1 --2- 172.21.15.46:0/1529059996 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7e0068040 0x7fa7e00c4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:13.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7ddd9b700 1 --2- 172.21.15.46:0/1529059996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7e006c110 0x7fa7e00c4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:13.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7ded9d700 1 -- 172.21.15.46:0/1529059996 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7e0068040 msgr2=0x7fa7e00c4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:13.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7ded9d700 1 --2- 172.21.15.46:0/1529059996 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7e0068040 0x7fa7e00c4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:13.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7ded9d700 1 -- 172.21.15.46:0/1529059996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7e006c110 msgr2=0x7fa7e00c4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:13.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7ded9d700 1 --2- 172.21.15.46:0/1529059996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7e006c110 0x7fa7e00c4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:13.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7ded9d700 1 -- 172.21.15.46:0/1529059996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d8009ae0 con 0x7fa7e0157840 2024-01-26T23:48:13.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7de59c700 1 --2- 172.21.15.46:0/1529059996 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7e0068040 0x7fa7e00c4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:13.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7ded9d700 1 --2- 172.21.15.46:0/1529059996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7e0157840 0x7fa7e00c5410 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8003000 tx=0x7fa7d8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:13.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7d67fc700 1 -- 172.21.15.46:0/1529059996 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d80092e0 con 0x7fa7e0157840 2024-01-26T23:48:13.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7d67fc700 1 -- 172.21.15.46:0/1529059996 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d8012db0 con 0x7fa7e0157840 2024-01-26T23:48:13.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.979+0000 7fa7e49a2700 1 -- 172.21.15.46:0/1529059996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e00ae9a0 con 0x7fa7e0157840 2024-01-26T23:48:13.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.980+0000 7fa7d67fc700 1 -- 172.21.15.46:0/1529059996 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d801e660 con 0x7fa7e0157840 2024-01-26T23:48:13.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.980+0000 7fa7e49a2700 1 -- 172.21.15.46:0/1529059996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e00c75e0 con 0x7fa7e0157840 2024-01-26T23:48:13.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.981+0000 7fa7e49a2700 1 -- 172.21.15.46:0/1529059996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e004c710 con 0x7fa7e0157840 2024-01-26T23:48:13.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.981+0000 7fa7d67fc700 1 -- 172.21.15.46:0/1529059996 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa7d801ea50 con 0x7fa7e0157840 2024-01-26T23:48:13.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.982+0000 7fa7d67fc700 1 --2- 172.21.15.46:0/1529059996 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa7c0066f30 0x7fa7c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:13.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.982+0000 7fa7d67fc700 1 -- 172.21.15.46:0/1529059996 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fa7d8086db0 con 0x7fa7e0157840 2024-01-26T23:48:13.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.982+0000 7fa7de59c700 1 --2- 172.21.15.46:0/1529059996 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa7c0066f30 0x7fa7c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:13.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.983+0000 7fa7de59c700 1 --2- 172.21.15.46:0/1529059996 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa7c0066f30 0x7fa7c00693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0067e60 tx=0x7fa7d0008040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:13.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:13.986+0000 7fa7d67fc700 1 -- 172.21.15.46:0/1529059996 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa7d8053dd0 con 0x7fa7e0157840 2024-01-26T23:48:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.241+0000 7fa7e49a2700 1 -- 172.21.15.46:0/1529059996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa7e00af130 con 0x7fa7e0157840 2024-01-26T23:48:14.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.820+0000 7fa7d67fc700 1 -- 172.21.15.46:0/1529059996 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v62) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa7d80573f0 con 0x7fa7e0157840 2024-01-26T23:48:14.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.824+0000 7fa7e49a2700 1 -- 172.21.15.46:0/1529059996 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa7c0066f30 msgr2=0x7fa7c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:14.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.824+0000 7fa7e49a2700 1 --2- 172.21.15.46:0/1529059996 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa7c0066f30 0x7fa7c00693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0067e60 tx=0x7fa7d0008040 comp rx=0 tx=0).stop 2024-01-26T23:48:14.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.824+0000 7fa7e49a2700 1 -- 172.21.15.46:0/1529059996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7e0157840 msgr2=0x7fa7e00c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:14.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.824+0000 7fa7e49a2700 1 --2- 172.21.15.46:0/1529059996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7e0157840 0x7fa7e00c5410 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8003000 tx=0x7fa7d8017000 comp rx=0 tx=0).stop 2024-01-26T23:48:14.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.824+0000 7fa7e49a2700 1 -- 172.21.15.46:0/1529059996 shutdown_connections 2024-01-26T23:48:14.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.824+0000 7fa7e49a2700 1 --2- 172.21.15.46:0/1529059996 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa7c0066f30 0x7fa7c00693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:14.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.824+0000 7fa7e49a2700 1 --2- 172.21.15.46:0/1529059996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7e0157840 0x7fa7e00c5410 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:14.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.824+0000 7fa7e49a2700 1 --2- 172.21.15.46:0/1529059996 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7e0068040 0x7fa7e00c4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:14.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.824+0000 7fa7e49a2700 1 --2- 172.21.15.46:0/1529059996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7e006c110 0x7fa7e00c4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:14.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.824+0000 7fa7e49a2700 1 -- 172.21.15.46:0/1529059996 >> 172.21.15.46:0/1529059996 conn(0x7fa7e00cdf50 msgr2=0x7fa7e015d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:14.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.824+0000 7fa7e49a2700 1 -- 172.21.15.46:0/1529059996 shutdown_connections 2024-01-26T23:48:14.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.824+0000 7fa7e49a2700 1 -- 172.21.15.46:0/1529059996 wait complete. 2024-01-26T23:48:14.840 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:48:14.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.975+0000 7f6b5e310700 1 Processor -- start 2024-01-26T23:48:14.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.975+0000 7f6b5e310700 1 -- start start 2024-01-26T23:48:14.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.976+0000 7f6b5e310700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b580acf30 0x7f6b580c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:14.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.976+0000 7f6b5e310700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b580ad780 0x7f6b580ab1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:14.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.976+0000 7f6b5e310700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b580c4230 con 0x7f6b580ad300 2024-01-26T23:48:14.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.976+0000 7f6b5e310700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b58157be0 con 0x7f6b580acf30 2024-01-26T23:48:14.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.976+0000 7f6b5e310700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b580cbf10 con 0x7f6b580ad780 2024-01-26T23:48:14.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.976+0000 7f6b57fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b580acf30 0x7f6b580c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:14.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.976+0000 7f6b57fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b580acf30 0x7f6b580c3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38950/0 (socket says 172.21.15.46:38950) 2024-01-26T23:48:14.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.976+0000 7f6b57fff700 1 -- 172.21.15.46:0/2322540960 learned_addr learned my addr 172.21.15.46:0/2322540960 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:14.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.976+0000 7f6b5c8ad700 1 --2- 172.21.15.46:0/2322540960 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b580ad780 0x7f6b580ab1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:14.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.976+0000 7f6b57fff700 1 -- 172.21.15.46:0/2322540960 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b580ad780 msgr2=0x7f6b580ab1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:14.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.976+0000 7f6b57fff700 1 --2- 172.21.15.46:0/2322540960 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b580ad780 0x7f6b580ab1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:14.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.976+0000 7f6b57fff700 1 -- 172.21.15.46:0/2322540960 >> v1:172.21.15.106:6789/0 conn(0x7f6b580ad300 legacy=0x7f6b580ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:14.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.976+0000 7f6b57fff700 1 -- 172.21.15.46:0/2322540960 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b580c4460 con 0x7f6b580acf30 2024-01-26T23:48:14.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.977+0000 7f6b57fff700 1 --2- 172.21.15.46:0/2322540960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b580acf30 0x7f6b580c3cf0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f6b480099b0 tx=0x7f6b48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f334f54267a914 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:14.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.977+0000 7f6b56ffd700 1 -- 172.21.15.46:0/2322540960 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b48009ea0 con 0x7f6b580acf30 2024-01-26T23:48:14.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.977+0000 7f6b56ffd700 1 -- 172.21.15.46:0/2322540960 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4800b860 con 0x7f6b580acf30 2024-01-26T23:48:14.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.977+0000 7f6b56ffd700 1 -- 172.21.15.46:0/2322540960 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4800bc00 con 0x7f6b580acf30 2024-01-26T23:48:14.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.977+0000 7f6b5e310700 1 -- 172.21.15.46:0/2322540960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b580acf30 msgr2=0x7f6b580c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:14.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.977+0000 7f6b5e310700 1 --2- 172.21.15.46:0/2322540960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b580acf30 0x7f6b580c3cf0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f6b480099b0 tx=0x7f6b48009250 comp rx=0 tx=0).stop 2024-01-26T23:48:14.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.977+0000 7f6b5e310700 1 -- 172.21.15.46:0/2322540960 shutdown_connections 2024-01-26T23:48:14.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.977+0000 7f6b5e310700 1 --2- 172.21.15.46:0/2322540960 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b580ad780 0x7f6b580ab1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:14.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.977+0000 7f6b5e310700 1 --2- 172.21.15.46:0/2322540960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b580acf30 0x7f6b580c3cf0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:14.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.977+0000 7f6b5e310700 1 -- 172.21.15.46:0/2322540960 >> 172.21.15.46:0/2322540960 conn(0x7f6b58097820 msgr2=0x7f6b58068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:14.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.978+0000 7f6b5e310700 1 -- 172.21.15.46:0/2322540960 shutdown_connections 2024-01-26T23:48:14.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.978+0000 7f6b5e310700 1 -- 172.21.15.46:0/2322540960 wait complete. 2024-01-26T23:48:14.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.979+0000 7f6b5e310700 1 Processor -- start 2024-01-26T23:48:14.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.979+0000 7f6b5e310700 1 -- start start 2024-01-26T23:48:14.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.980+0000 7f6b5e310700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b580acf30 0x7f6b580cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:14.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.980+0000 7f6b5e310700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b580ad300 0x7f6b58144020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:14.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.980+0000 7f6b5e310700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b580ad780 0x7f6b580ab3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:14.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.980+0000 7f6b5e310700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b58153fa0 con 0x7f6b580acf30 2024-01-26T23:48:14.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.980+0000 7f6b5e310700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b581537d0 con 0x7f6b580ad780 2024-01-26T23:48:14.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.980+0000 7f6b5e310700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b580c7f30 con 0x7f6b580ad300 2024-01-26T23:48:14.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.981+0000 7f6b57fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b580acf30 0x7f6b580cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:14.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.981+0000 7f6b57fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b580acf30 0x7f6b580cb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38966/0 (socket says 172.21.15.46:38966) 2024-01-26T23:48:14.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.981+0000 7f6b57fff700 1 -- 172.21.15.46:0/95131071 learned_addr learned my addr 172.21.15.46:0/95131071 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:14.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.981+0000 7f6b577fe700 1 --2- 172.21.15.46:0/95131071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b580ad300 0x7f6b58144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:14.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.981+0000 7f6b5c8ad700 1 --2- 172.21.15.46:0/95131071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b580ad780 0x7f6b580ab3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:14.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.981+0000 7f6b57fff700 1 -- 172.21.15.46:0/95131071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b580ad300 msgr2=0x7f6b58144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:14.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.981+0000 7f6b57fff700 1 --2- 172.21.15.46:0/95131071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b580ad300 0x7f6b58144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:14.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.981+0000 7f6b57fff700 1 -- 172.21.15.46:0/95131071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b580ad780 msgr2=0x7f6b580ab3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:14.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.981+0000 7f6b57fff700 1 --2- 172.21.15.46:0/95131071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b580ad780 0x7f6b580ab3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:14.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.981+0000 7f6b57fff700 1 -- 172.21.15.46:0/95131071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b480094f0 con 0x7f6b580acf30 2024-01-26T23:48:14.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.981+0000 7f6b57fff700 1 --2- 172.21.15.46:0/95131071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b580acf30 0x7f6b580cb3b0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f6b5806ef50 tx=0x7f6b48017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:14.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.981+0000 7f6b4ffff700 1 -- 172.21.15.46:0/95131071 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b48000d10 con 0x7f6b580acf30 2024-01-26T23:48:14.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.981+0000 7f6b4ffff700 1 -- 172.21.15.46:0/95131071 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4800bbe0 con 0x7f6b580acf30 2024-01-26T23:48:14.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.981+0000 7f6b4ffff700 1 -- 172.21.15.46:0/95131071 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b48011670 con 0x7f6b580acf30 2024-01-26T23:48:14.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.981+0000 7f6b5e310700 1 -- 172.21.15.46:0/95131071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b580cb8f0 con 0x7f6b580acf30 2024-01-26T23:48:14.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.981+0000 7f6b5e310700 1 -- 172.21.15.46:0/95131071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b581539e0 con 0x7f6b580acf30 2024-01-26T23:48:14.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.982+0000 7f6b5e310700 1 -- 172.21.15.46:0/95131071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b5804c710 con 0x7f6b580acf30 2024-01-26T23:48:14.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.983+0000 7f6b4ffff700 1 -- 172.21.15.46:0/95131071 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6b48011aa0 con 0x7f6b580acf30 2024-01-26T23:48:14.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.983+0000 7f6b4ffff700 1 --2- 172.21.15.46:0/95131071 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b38066f30 0x7f6b380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:14.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.983+0000 7f6b4ffff700 1 -- 172.21.15.46:0/95131071 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f6b48085ce0 con 0x7f6b580acf30 2024-01-26T23:48:14.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.984+0000 7f6b577fe700 1 --2- 172.21.15.46:0/95131071 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b38066f30 0x7f6b380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:14.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.984+0000 7f6b577fe700 1 --2- 172.21.15.46:0/95131071 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b38066f30 0x7f6b380693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6b580caf90 tx=0x7f6b40007ec0 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:14.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:14.988+0000 7f6b4ffff700 1 -- 172.21.15.46:0/95131071 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6b48052c60 con 0x7f6b580acf30 2024-01-26T23:48:15.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.241+0000 7f6b5e310700 1 -- 172.21.15.46:0/95131071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b580c4740 con 0x7f6b580acf30 2024-01-26T23:48:15.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.242+0000 7f6b4ffff700 1 -- 172.21.15.46:0/95131071 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f6b48015090 con 0x7f6b580acf30 2024-01-26T23:48:15.243 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:48:15.244 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":62,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6923,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/4123353304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4123353304},{"type":"v1","addr":"172.21.15.106:6834","nonce":4123353304}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/690662483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":690662483},{"type":"v1","addr":"172.21.15.106:6836","nonce":690662483}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7791,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3035969379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3035969379},{"type":"v1","addr":"172.21.15.46:6834","nonce":3035969379}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:48:11.234632+0000","modified":"2024-01-26T23:48:14.815720+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7794},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7794":{"gid":7794,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/783623731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":783623731},{"type":"v1","addr":"172.21.15.46:6835","nonce":783623731}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2024-01-26T23:48:15.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.244+0000 7f6b5e310700 1 -- 172.21.15.46:0/95131071 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b38066f30 msgr2=0x7f6b380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:15.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.244+0000 7f6b5e310700 1 --2- 172.21.15.46:0/95131071 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b38066f30 0x7f6b380693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6b580caf90 tx=0x7f6b40007ec0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.244+0000 7f6b5e310700 1 -- 172.21.15.46:0/95131071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b580acf30 msgr2=0x7f6b580cb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:15.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.244+0000 7f6b5e310700 1 --2- 172.21.15.46:0/95131071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b580acf30 0x7f6b580cb3b0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f6b5806ef50 tx=0x7f6b48017000 comp rx=0 tx=0).stop 2024-01-26T23:48:15.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.244+0000 7f6b5e310700 1 -- 172.21.15.46:0/95131071 shutdown_connections 2024-01-26T23:48:15.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.244+0000 7f6b5e310700 1 --2- 172.21.15.46:0/95131071 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b38066f30 0x7f6b380693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.244+0000 7f6b5e310700 1 --2- 172.21.15.46:0/95131071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b580acf30 0x7f6b580cb3b0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.244+0000 7f6b5e310700 1 --2- 172.21.15.46:0/95131071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b580ad780 0x7f6b580ab3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.244+0000 7f6b5e310700 1 --2- 172.21.15.46:0/95131071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b580ad300 0x7f6b58144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.244+0000 7f6b5e310700 1 -- 172.21.15.46:0/95131071 >> 172.21.15.46:0/95131071 conn(0x7f6b58097820 msgr2=0x7f6b5806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:15.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.244+0000 7f6b5e310700 1 -- 172.21.15.46:0/95131071 shutdown_connections 2024-01-26T23:48:15.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.244+0000 7f6b5e310700 1 -- 172.21.15.46:0/95131071 wait complete. 2024-01-26T23:48:15.247 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 62 2024-01-26T23:48:15.260 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:48:15.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.393+0000 7f4d4c28e700 1 Processor -- start 2024-01-26T23:48:15.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.393+0000 7f4d4c28e700 1 -- start start 2024-01-26T23:48:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.394+0000 7f4d4c28e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4d44157340 0x7f4d4415bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.394+0000 7f4d4c28e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d440ae520 0x7f4d440b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.394+0000 7f4d4c28e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4d440b3fb0 0x7f4d44157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.394+0000 7f4d4c28e700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d440b4430 con 0x7f4d44157340 2024-01-26T23:48:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.394+0000 7f4d4c28e700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d44158ae0 con 0x7f4d440b3fb0 2024-01-26T23:48:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.394+0000 7f4d4c28e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4415b650 con 0x7f4d440ae520 2024-01-26T23:48:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.394+0000 7f4d4a02a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4d44157340 0x7f4d4415bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.394+0000 7f4d4a02a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4d44157340 0x7f4d4415bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38980/0 (socket says 172.21.15.46:38980) 2024-01-26T23:48:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.394+0000 7f4d4a02a700 1 -- 172.21.15.46:0/359755259 learned_addr learned my addr 172.21.15.46:0/359755259 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.394+0000 7f4d4a82b700 1 --2- 172.21.15.46:0/359755259 >> v2:172.21.15.106:3300/0 conn(0x7f4d440b3fb0 0x7f4d44157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.394+0000 7f4d49829700 1 --2- 172.21.15.46:0/359755259 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d440ae520 0x7f4d440b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.394+0000 7f4d4a02a700 1 -- 172.21.15.46:0/359755259 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d440ae520 msgr2=0x7f4d440b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.394+0000 7f4d4a02a700 1 --2- 172.21.15.46:0/359755259 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d440ae520 0x7f4d440b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.394+0000 7f4d4a02a700 1 -- 172.21.15.46:0/359755259 >> v2:172.21.15.106:3300/0 conn(0x7f4d440b3fb0 msgr2=0x7f4d44157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.394+0000 7f4d4a02a700 1 --2- 172.21.15.46:0/359755259 >> v2:172.21.15.106:3300/0 conn(0x7f4d440b3fb0 0x7f4d44157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.395+0000 7f4d4a02a700 1 -- 172.21.15.46:0/359755259 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4415c020 con 0x7f4d44157340 2024-01-26T23:48:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.395+0000 7f4d49829700 1 --2- 172.21.15.46:0/359755259 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d440ae520 0x7f4d440b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.395+0000 7f4d4a02a700 1 --2- 172.21.15.46:0/359755259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d44157340 0x7f4d4415bae0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f4d440caee0 tx=0x7f4d40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59055a877915759 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.395+0000 7f4d49028700 1 -- 172.21.15.46:0/359755259 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4000b860 con 0x7f4d44157340 2024-01-26T23:48:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.395+0000 7f4d49028700 1 -- 172.21.15.46:0/359755259 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4000b9c0 con 0x7f4d44157340 2024-01-26T23:48:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.395+0000 7f4d49028700 1 -- 172.21.15.46:0/359755259 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4000bd80 con 0x7f4d44157340 2024-01-26T23:48:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.395+0000 7f4d4c28e700 1 -- 172.21.15.46:0/359755259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d44157340 msgr2=0x7f4d4415bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.395+0000 7f4d4c28e700 1 --2- 172.21.15.46:0/359755259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d44157340 0x7f4d4415bae0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f4d440caee0 tx=0x7f4d40009250 comp rx=0 tx=0).stop 2024-01-26T23:48:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.396+0000 7f4d4c28e700 1 -- 172.21.15.46:0/359755259 shutdown_connections 2024-01-26T23:48:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.396+0000 7f4d4c28e700 1 --2- 172.21.15.46:0/359755259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d44157340 0x7f4d4415bae0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.396+0000 7f4d4c28e700 1 --2- 172.21.15.46:0/359755259 >> v2:172.21.15.106:3300/0 conn(0x7f4d440b3fb0 0x7f4d44157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.396+0000 7f4d4c28e700 1 --2- 172.21.15.46:0/359755259 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d440ae520 0x7f4d440b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.396+0000 7f4d4c28e700 1 -- 172.21.15.46:0/359755259 >> 172.21.15.46:0/359755259 conn(0x7f4d440ce530 msgr2=0x7f4d440ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:15.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.396+0000 7f4d4c28e700 1 -- 172.21.15.46:0/359755259 shutdown_connections 2024-01-26T23:48:15.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.396+0000 7f4d4c28e700 1 -- 172.21.15.46:0/359755259 wait complete. 2024-01-26T23:48:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.397+0000 7f4d4c28e700 1 Processor -- start 2024-01-26T23:48:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.397+0000 7f4d4c28e700 1 -- start start 2024-01-26T23:48:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.398+0000 7f4d4c28e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d440ae520 0x7f4d440c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.398+0000 7f4d4c28e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d440b3fb0 0x7f4d44167d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.398+0000 7f4d4c28e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d44157340 0x7f4d441684b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.398+0000 7f4d4c28e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d441682c0 con 0x7f4d440ae520 2024-01-26T23:48:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.398+0000 7f4d4c28e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d440b23e0 con 0x7f4d440b3fb0 2024-01-26T23:48:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.398+0000 7f4d4c28e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d441689f0 con 0x7f4d44157340 2024-01-26T23:48:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.398+0000 7f4d4a02a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d440ae520 0x7f4d440c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.399+0000 7f4d4a02a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d440ae520 0x7f4d440c6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38982/0 (socket says 172.21.15.46:38982) 2024-01-26T23:48:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.399+0000 7f4d4a02a700 1 -- 172.21.15.46:0/1213722202 learned_addr learned my addr 172.21.15.46:0/1213722202 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.399+0000 7f4d49829700 1 --2- 172.21.15.46:0/1213722202 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d440b3fb0 0x7f4d44167d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.399+0000 7f4d4a82b700 1 --2- 172.21.15.46:0/1213722202 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d44157340 0x7f4d441684b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.399+0000 7f4d4a02a700 1 -- 172.21.15.46:0/1213722202 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d44157340 msgr2=0x7f4d441684b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.399+0000 7f4d4a02a700 1 --2- 172.21.15.46:0/1213722202 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d44157340 0x7f4d441684b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.399+0000 7f4d4a02a700 1 -- 172.21.15.46:0/1213722202 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d440b3fb0 msgr2=0x7f4d44167d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.399+0000 7f4d4a02a700 1 --2- 172.21.15.46:0/1213722202 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d440b3fb0 0x7f4d44167d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.399+0000 7f4d4a02a700 1 -- 172.21.15.46:0/1213722202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d400094f0 con 0x7f4d440ae520 2024-01-26T23:48:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.399+0000 7f4d4a02a700 1 --2- 172.21.15.46:0/1213722202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d440ae520 0x7f4d440c6d40 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000b3c0 tx=0x7f4d40017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.399+0000 7f4d3a7fc700 1 -- 172.21.15.46:0/1213722202 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d40011e80 con 0x7f4d440ae520 2024-01-26T23:48:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.399+0000 7f4d3a7fc700 1 -- 172.21.15.46:0/1213722202 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d40026450 con 0x7f4d440ae520 2024-01-26T23:48:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.399+0000 7f4d4c28e700 1 -- 172.21.15.46:0/1213722202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d440c7440 con 0x7f4d440ae520 2024-01-26T23:48:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.399+0000 7f4d3a7fc700 1 -- 172.21.15.46:0/1213722202 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d400267f0 con 0x7f4d440ae520 2024-01-26T23:48:15.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.399+0000 7f4d4c28e700 1 -- 172.21.15.46:0/1213722202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d440c7a00 con 0x7f4d440ae520 2024-01-26T23:48:15.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.400+0000 7f4d4c28e700 1 -- 172.21.15.46:0/1213722202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4404c710 con 0x7f4d440ae520 2024-01-26T23:48:15.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.401+0000 7f4d3a7fc700 1 -- 172.21.15.46:0/1213722202 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4d40026b60 con 0x7f4d440ae520 2024-01-26T23:48:15.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.401+0000 7f4d3a7fc700 1 --2- 172.21.15.46:0/1213722202 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4d28066f30 0x7f4d280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:15.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.401+0000 7f4d3a7fc700 1 -- 172.21.15.46:0/1213722202 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f4d40086310 con 0x7f4d440ae520 2024-01-26T23:48:15.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.401+0000 7f4d49829700 1 --2- 172.21.15.46:0/1213722202 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4d28066f30 0x7f4d280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:15.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.402+0000 7f4d49829700 1 --2- 172.21.15.46:0/1213722202 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4d28066f30 0x7f4d280693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4d34006f00 tx=0x7f4d34008040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.406+0000 7f4d3a7fc700 1 -- 172.21.15.46:0/1213722202 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4d40053290 con 0x7f4d440ae520 2024-01-26T23:48:15.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.612+0000 7f4d4c28e700 1 -- 172.21.15.46:0/1213722202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d44158520 con 0x7f4d440ae520 2024-01-26T23:48:15.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.613+0000 7f4d3a7fc700 1 -- 172.21.15.46:0/1213722202 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16163 (secure 0 0 0) 0x7f4d400568b0 con 0x7f4d440ae520 2024-01-26T23:48:15.614 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:48:15.614 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":130,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:48:12.671788+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:48:09.662590+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-01-26T23:48:10.200635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:48:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.615+0000 7f4d4c28e700 1 -- 172.21.15.46:0/1213722202 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4d28066f30 msgr2=0x7f4d280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.615+0000 7f4d4c28e700 1 --2- 172.21.15.46:0/1213722202 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4d28066f30 0x7f4d280693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4d34006f00 tx=0x7f4d34008040 comp rx=0 tx=0).stop 2024-01-26T23:48:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.615+0000 7f4d4c28e700 1 -- 172.21.15.46:0/1213722202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d440ae520 msgr2=0x7f4d440c6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.615+0000 7f4d4c28e700 1 --2- 172.21.15.46:0/1213722202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d440ae520 0x7f4d440c6d40 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000b3c0 tx=0x7f4d40017000 comp rx=0 tx=0).stop 2024-01-26T23:48:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.615+0000 7f4d4c28e700 1 -- 172.21.15.46:0/1213722202 shutdown_connections 2024-01-26T23:48:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.615+0000 7f4d4c28e700 1 --2- 172.21.15.46:0/1213722202 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4d28066f30 0x7f4d280693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.615+0000 7f4d4c28e700 1 --2- 172.21.15.46:0/1213722202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d440ae520 0x7f4d440c6d40 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.615+0000 7f4d4c28e700 1 --2- 172.21.15.46:0/1213722202 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d44157340 0x7f4d441684b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.615+0000 7f4d4c28e700 1 --2- 172.21.15.46:0/1213722202 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d440b3fb0 0x7f4d44167d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.615+0000 7f4d4c28e700 1 -- 172.21.15.46:0/1213722202 >> 172.21.15.46:0/1213722202 conn(0x7f4d440ce530 msgr2=0x7f4d441599f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:15.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.615+0000 7f4d4c28e700 1 -- 172.21.15.46:0/1213722202 shutdown_connections 2024-01-26T23:48:15.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.615+0000 7f4d4c28e700 1 -- 172.21.15.46:0/1213722202 wait complete. 2024-01-26T23:48:15.631 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-26T23:48:15.631 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:48:15.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.762+0000 7f54ea8cd700 1 Processor -- start 2024-01-26T23:48:15.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.762+0000 7f54ea8cd700 1 -- start start 2024-01-26T23:48:15.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.763+0000 7f54ea8cd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f54e40ccff0 0x7f54e4068340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:15.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.763+0000 7f54ea8cd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f54e40b2590 0x7f54e40b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:15.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.763+0000 7f54ea8cd700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e40b2f10 con 0x7f54e40ccc20 2024-01-26T23:48:15.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.763+0000 7f54ea8cd700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e4158020 con 0x7f54e40ccff0 2024-01-26T23:48:15.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.763+0000 7f54ea8cd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e4158300 con 0x7f54e40b2590 2024-01-26T23:48:15.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.763+0000 7f54e37fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f54e40ccff0 0x7f54e4068340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:15.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.763+0000 7f54e3fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f54e40ccc20 0x7f54e406eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40722/0 (socket says 172.21.15.46:40722) 2024-01-26T23:48:15.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.763+0000 7f54e3fff700 1 -- 172.21.15.46:0/3299485154 learned_addr learned my addr 172.21.15.46:0/3299485154 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:15.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.763+0000 7f54e8e6a700 1 --2- 172.21.15.46:0/3299485154 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f54e40b2590 0x7f54e40b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:15.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.763+0000 7f54e37fe700 1 -- 172.21.15.46:0/3299485154 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f54e40b2590 msgr2=0x7f54e40b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:15.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.763+0000 7f54e37fe700 1 --2- 172.21.15.46:0/3299485154 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f54e40b2590 0x7f54e40b29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.763+0000 7f54e37fe700 1 -- 172.21.15.46:0/3299485154 >> v1:172.21.15.46:6789/0 conn(0x7f54e40ccc20 legacy=0x7f54e406eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:15.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.763+0000 7f54e37fe700 1 -- 172.21.15.46:0/3299485154 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e41585e0 con 0x7f54e40ccff0 2024-01-26T23:48:15.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.763+0000 7f54e2ffd700 1 -- 172.21.15.46:0/3299485154 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409432927 0 0) 0x7f54e40b2f10 con 0x7f54e40ccc20 2024-01-26T23:48:15.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.764+0000 7f54e37fe700 1 --2- 172.21.15.46:0/3299485154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54e40ccff0 0x7f54e4068340 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0098e0 tx=0x7f54cc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3fc5760a7aa6934 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:15.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.764+0000 7f54e2ffd700 1 -- 172.21.15.46:0/3299485154 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc009e80 con 0x7f54e40ccff0 2024-01-26T23:48:15.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.764+0000 7f54e2ffd700 1 -- 172.21.15.46:0/3299485154 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54cc00b860 con 0x7f54e40ccff0 2024-01-26T23:48:15.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.764+0000 7f54e2ffd700 1 -- 172.21.15.46:0/3299485154 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc00bbe0 con 0x7f54e40ccff0 2024-01-26T23:48:15.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.764+0000 7f54ea8cd700 1 -- 172.21.15.46:0/3299485154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54e40ccff0 msgr2=0x7f54e4068340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:15.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.764+0000 7f54ea8cd700 1 --2- 172.21.15.46:0/3299485154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54e40ccff0 0x7f54e4068340 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0098e0 tx=0x7f54cc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.764+0000 7f54ea8cd700 1 -- 172.21.15.46:0/3299485154 shutdown_connections 2024-01-26T23:48:15.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.764+0000 7f54ea8cd700 1 --2- 172.21.15.46:0/3299485154 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f54e40b2590 0x7f54e40b29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.764+0000 7f54ea8cd700 1 --2- 172.21.15.46:0/3299485154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54e40ccff0 0x7f54e4068340 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.764+0000 7f54ea8cd700 1 -- 172.21.15.46:0/3299485154 >> 172.21.15.46:0/3299485154 conn(0x7f54e406b630 msgr2=0x7f54e40aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:15.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.764+0000 7f54ea8cd700 1 -- 172.21.15.46:0/3299485154 shutdown_connections 2024-01-26T23:48:15.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.764+0000 7f54ea8cd700 1 -- 172.21.15.46:0/3299485154 wait complete. 2024-01-26T23:48:15.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.766+0000 7f54ea8cd700 1 Processor -- start 2024-01-26T23:48:15.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.766+0000 7f54ea8cd700 1 -- start start 2024-01-26T23:48:15.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54ea8cd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54e40b2590 0x7f54e4147fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:15.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54ea8cd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f54e40ccc20 0x7f54e40c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:15.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54ea8cd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54e40ccff0 0x7f54e40c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:15.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54ea8cd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e40c5e90 con 0x7f54e40b2590 2024-01-26T23:48:15.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54ea8cd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e40c6a10 con 0x7f54e40ccff0 2024-01-26T23:48:15.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54ea8cd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e41677b0 con 0x7f54e40ccc20 2024-01-26T23:48:15.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54e3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54e40b2590 0x7f54e4147fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:15.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54e3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54e40b2590 0x7f54e4147fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38992/0 (socket says 172.21.15.46:38992) 2024-01-26T23:48:15.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54e3fff700 1 -- 172.21.15.46:0/617239030 learned_addr learned my addr 172.21.15.46:0/617239030 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:15.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54e8e6a700 1 --2- 172.21.15.46:0/617239030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54e40ccff0 0x7f54e40c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:15.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54e37fe700 1 --2- 172.21.15.46:0/617239030 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f54e40ccc20 0x7f54e40c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:15.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54e3fff700 1 -- 172.21.15.46:0/617239030 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f54e40ccc20 msgr2=0x7f54e40c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:15.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54e3fff700 1 --2- 172.21.15.46:0/617239030 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f54e40ccc20 0x7f54e40c8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54e3fff700 1 -- 172.21.15.46:0/617239030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54e40ccff0 msgr2=0x7f54e40c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:15.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54e3fff700 1 --2- 172.21.15.46:0/617239030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54e40ccff0 0x7f54e40c8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54e3fff700 1 -- 172.21.15.46:0/617239030 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54cc009420 con 0x7f54e40b2590 2024-01-26T23:48:15.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54e3fff700 1 --2- 172.21.15.46:0/617239030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54e40b2590 0x7f54e4147fc0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f54d400c7e0 tx=0x7f54d4007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:15.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54dbfff700 1 -- 172.21.15.46:0/617239030 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4005b30 con 0x7f54e40b2590 2024-01-26T23:48:15.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54dbfff700 1 -- 172.21.15.46:0/617239030 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d4005c90 con 0x7f54e40b2590 2024-01-26T23:48:15.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.767+0000 7f54ea8cd700 1 -- 172.21.15.46:0/617239030 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e40c60e0 con 0x7f54e40b2590 2024-01-26T23:48:15.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.768+0000 7f54dbfff700 1 -- 172.21.15.46:0/617239030 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4007300 con 0x7f54e40b2590 2024-01-26T23:48:15.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.768+0000 7f54ea8cd700 1 -- 172.21.15.46:0/617239030 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e40c66a0 con 0x7f54e40b2590 2024-01-26T23:48:15.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.769+0000 7f54ea8cd700 1 -- 172.21.15.46:0/617239030 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e404c710 con 0x7f54e40b2590 2024-01-26T23:48:15.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.769+0000 7f54dbfff700 1 -- 172.21.15.46:0/617239030 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f54d401b400 con 0x7f54e40b2590 2024-01-26T23:48:15.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.769+0000 7f54dbfff700 1 --2- 172.21.15.46:0/617239030 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f54c4066f30 0x7f54c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:15.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.769+0000 7f54dbfff700 1 -- 172.21.15.46:0/617239030 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f54d4084420 con 0x7f54e40b2590 2024-01-26T23:48:15.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.770+0000 7f54e37fe700 1 --2- 172.21.15.46:0/617239030 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f54c4066f30 0x7f54c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:15.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.770+0000 7f54e37fe700 1 --2- 172.21.15.46:0/617239030 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f54c4066f30 0x7f54c40693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0098e0 tx=0x7f54cc017000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:15.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.774+0000 7f54dbfff700 1 -- 172.21.15.46:0/617239030 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f54d40513a0 con 0x7f54e40b2590 2024-01-26T23:48:15.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.976+0000 7f54ea8cd700 1 -- 172.21.15.46:0/617239030 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f54e41588c0 con 0x7f54c4066f30 2024-01-26T23:48:15.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.984+0000 7f54dbfff700 1 -- 172.21.15.46:0/617239030 <== mgr.7396 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277082 (secure 0 0 0) 0x7f54e41588c0 con 0x7f54c4066f30 2024-01-26T23:48:15.989 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:48:15.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.988+0000 7f54ea8cd700 1 -- 172.21.15.46:0/617239030 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f54c4066f30 msgr2=0x7f54c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:15.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.988+0000 7f54ea8cd700 1 --2- 172.21.15.46:0/617239030 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f54c4066f30 0x7f54c40693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0098e0 tx=0x7f54cc017000 comp rx=0 tx=0).stop 2024-01-26T23:48:15.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.988+0000 7f54ea8cd700 1 -- 172.21.15.46:0/617239030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54e40b2590 msgr2=0x7f54e4147fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:15.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.988+0000 7f54ea8cd700 1 --2- 172.21.15.46:0/617239030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54e40b2590 0x7f54e4147fc0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f54d400c7e0 tx=0x7f54d4007df0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.988+0000 7f54ea8cd700 1 -- 172.21.15.46:0/617239030 shutdown_connections 2024-01-26T23:48:15.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.988+0000 7f54ea8cd700 1 --2- 172.21.15.46:0/617239030 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f54c4066f30 0x7f54c40693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.988+0000 7f54ea8cd700 1 --2- 172.21.15.46:0/617239030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54e40b2590 0x7f54e4147fc0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.988+0000 7f54ea8cd700 1 --2- 172.21.15.46:0/617239030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54e40ccff0 0x7f54e40c8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.988+0000 7f54ea8cd700 1 --2- 172.21.15.46:0/617239030 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f54e40ccc20 0x7f54e40c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:15.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.988+0000 7f54ea8cd700 1 -- 172.21.15.46:0/617239030 >> 172.21.15.46:0/617239030 conn(0x7f54e406b630 msgr2=0x7f54e4156640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:15.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.988+0000 7f54ea8cd700 1 -- 172.21.15.46:0/617239030 shutdown_connections 2024-01-26T23:48:15.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:15.988+0000 7f54ea8cd700 1 -- 172.21.15.46:0/617239030 wait complete. 2024-01-26T23:48:15.992 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:48:16.003 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-26T23:48:14.716316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":971,"num_read_kb":1902,"num_write":324,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8567,"ondisk_log_size":8567,"up":40,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":754974720,"kb_used":2444956,"kb_used_data":13404,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752529764,"statfs":{"total":773094113280,"available":770590478336,"internally_reserved":0,"allocated":13725696,"data_stored":7595921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.927983"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702212+0000","last_change":"2024-01-26T23:48:11.695520+0000","last_active":"2024-01-26T23:48:11.702212+0000","last_peered":"2024-01-26T23:48:11.702212+0000","last_clean":"2024-01-26T23:48:11.702212+0000","last_became_active":"2024-01-26T23:48:11.695205+0000","last_became_peered":"2024-01-26T23:48:11.695205+0000","last_unstale":"2024-01-26T23:48:11.702212+0000","last_undegraded":"2024-01-26T23:48:11.702212+0000","last_fullsized":"2024-01-26T23:48:11.702212+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:36:53.012187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.690830+0000","last_change":"2024-01-26T23:48:11.689380+0000","last_active":"2024-01-26T23:48:11.690830+0000","last_peered":"2024-01-26T23:48:11.690830+0000","last_clean":"2024-01-26T23:48:11.690830+0000","last_became_active":"2024-01-26T23:48:11.688997+0000","last_became_peered":"2024-01-26T23:48:11.688997+0000","last_unstale":"2024-01-26T23:48:11.690830+0000","last_undegraded":"2024-01-26T23:48:11.690830+0000","last_fullsized":"2024-01-26T23:48:11.690830+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:20.796536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702333+0000","last_change":"2024-01-26T23:48:11.700735+0000","last_active":"2024-01-26T23:48:11.702333+0000","last_peered":"2024-01-26T23:48:11.702333+0000","last_clean":"2024-01-26T23:48:11.702333+0000","last_became_active":"2024-01-26T23:48:11.700416+0000","last_became_peered":"2024-01-26T23:48:11.700416+0000","last_unstale":"2024-01-26T23:48:11.702333+0000","last_undegraded":"2024-01-26T23:48:11.702333+0000","last_fullsized":"2024-01-26T23:48:11.702333+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:03:16.363445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699884+0000","last_change":"2024-01-26T23:48:11.694600+0000","last_active":"2024-01-26T23:48:11.699884+0000","last_peered":"2024-01-26T23:48:11.699884+0000","last_clean":"2024-01-26T23:48:11.699884+0000","last_became_active":"2024-01-26T23:48:11.694195+0000","last_became_peered":"2024-01-26T23:48:11.694195+0000","last_unstale":"2024-01-26T23:48:11.699884+0000","last_undegraded":"2024-01-26T23:48:11.699884+0000","last_fullsized":"2024-01-26T23:48:11.699884+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:54:52.459709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702321+0000","last_change":"2024-01-26T23:48:11.699321+0000","last_active":"2024-01-26T23:48:11.702321+0000","last_peered":"2024-01-26T23:48:11.702321+0000","last_clean":"2024-01-26T23:48:11.702321+0000","last_became_active":"2024-01-26T23:48:11.699085+0000","last_became_peered":"2024-01-26T23:48:11.699085+0000","last_unstale":"2024-01-26T23:48:11.702321+0000","last_undegraded":"2024-01-26T23:48:11.702321+0000","last_fullsized":"2024-01-26T23:48:11.702321+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:49:57.681990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"129'1","reported_seq":11,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.805807+0000","last_change":"2024-01-26T23:48:10.816548+0000","last_active":"2024-01-26T23:48:11.805807+0000","last_peered":"2024-01-26T23:48:11.805807+0000","last_clean":"2024-01-26T23:48:11.805807+0000","last_became_active":"2024-01-26T23:48:10.816271+0000","last_became_peered":"2024-01-26T23:48:10.816271+0000","last_unstale":"2024-01-26T23:48:11.805807+0000","last_undegraded":"2024-01-26T23:48:11.805807+0000","last_fullsized":"2024-01-26T23:48:11.805807+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:10.976028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702237+0000","last_change":"2024-01-26T23:48:11.695812+0000","last_active":"2024-01-26T23:48:11.702237+0000","last_peered":"2024-01-26T23:48:11.702237+0000","last_clean":"2024-01-26T23:48:11.702237+0000","last_became_active":"2024-01-26T23:48:11.695535+0000","last_became_peered":"2024-01-26T23:48:11.695535+0000","last_unstale":"2024-01-26T23:48:11.702237+0000","last_undegraded":"2024-01-26T23:48:11.702237+0000","last_fullsized":"2024-01-26T23:48:11.702237+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:12:39.189557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699954+0000","last_change":"2024-01-26T23:48:11.694802+0000","last_active":"2024-01-26T23:48:11.699954+0000","last_peered":"2024-01-26T23:48:11.699954+0000","last_clean":"2024-01-26T23:48:11.699954+0000","last_became_active":"2024-01-26T23:48:11.694277+0000","last_became_peered":"2024-01-26T23:48:11.694277+0000","last_unstale":"2024-01-26T23:48:11.699954+0000","last_undegraded":"2024-01-26T23:48:11.699954+0000","last_fullsized":"2024-01-26T23:48:11.699954+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:28:58.191038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.684638+0000","last_change":"2024-01-26T23:48:10.813933+0000","last_active":"2024-01-26T23:48:11.684638+0000","last_peered":"2024-01-26T23:48:11.684638+0000","last_clean":"2024-01-26T23:48:11.684638+0000","last_became_active":"2024-01-26T23:48:10.813646+0000","last_became_peered":"2024-01-26T23:48:10.813646+0000","last_unstale":"2024-01-26T23:48:11.684638+0000","last_undegraded":"2024-01-26T23:48:11.684638+0000","last_fullsized":"2024-01-26T23:48:11.684638+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:20:47.863263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699904+0000","last_change":"2024-01-26T23:48:11.694643+0000","last_active":"2024-01-26T23:48:11.699904+0000","last_peered":"2024-01-26T23:48:11.699904+0000","last_clean":"2024-01-26T23:48:11.699904+0000","last_became_active":"2024-01-26T23:48:11.694296+0000","last_became_peered":"2024-01-26T23:48:11.694296+0000","last_unstale":"2024-01-26T23:48:11.699904+0000","last_undegraded":"2024-01-26T23:48:11.699904+0000","last_fullsized":"2024-01-26T23:48:11.699904+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:53:21.191149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.b","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.803125+0000","last_change":"2024-01-26T23:48:10.815064+0000","last_active":"2024-01-26T23:48:11.803125+0000","last_peered":"2024-01-26T23:48:11.803125+0000","last_clean":"2024-01-26T23:48:11.803125+0000","last_became_active":"2024-01-26T23:48:10.814778+0000","last_became_peered":"2024-01-26T23:48:10.814778+0000","last_unstale":"2024-01-26T23:48:11.803125+0000","last_undegraded":"2024-01-26T23:48:11.803125+0000","last_fullsized":"2024-01-26T23:48:11.803125+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:17.080283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.686302+0000","last_change":"2024-01-26T23:48:10.813851+0000","last_active":"2024-01-26T23:48:11.686302+0000","last_peered":"2024-01-26T23:48:11.686302+0000","last_clean":"2024-01-26T23:48:11.686302+0000","last_became_active":"2024-01-26T23:48:10.813510+0000","last_became_peered":"2024-01-26T23:48:10.813510+0000","last_unstale":"2024-01-26T23:48:11.686302+0000","last_undegraded":"2024-01-26T23:48:11.686302+0000","last_fullsized":"2024-01-26T23:48:11.686302+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:26:11.395397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698608+0000","last_change":"2024-01-26T23:48:11.692941+0000","last_active":"2024-01-26T23:48:11.698608+0000","last_peered":"2024-01-26T23:48:11.698608+0000","last_clean":"2024-01-26T23:48:11.698608+0000","last_became_active":"2024-01-26T23:48:11.692605+0000","last_became_peered":"2024-01-26T23:48:11.692605+0000","last_unstale":"2024-01-26T23:48:11.698608+0000","last_undegraded":"2024-01-26T23:48:11.698608+0000","last_fullsized":"2024-01-26T23:48:11.698608+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:01:01.083122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685230+0000","last_change":"2024-01-26T23:48:10.816002+0000","last_active":"2024-01-26T23:48:11.685230+0000","last_peered":"2024-01-26T23:48:11.685230+0000","last_clean":"2024-01-26T23:48:11.685230+0000","last_became_active":"2024-01-26T23:48:10.815730+0000","last_became_peered":"2024-01-26T23:48:10.815730+0000","last_unstale":"2024-01-26T23:48:11.685230+0000","last_undegraded":"2024-01-26T23:48:11.685230+0000","last_fullsized":"2024-01-26T23:48:11.685230+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:59.034288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698643+0000","last_change":"2024-01-26T23:48:11.692758+0000","last_active":"2024-01-26T23:48:11.698643+0000","last_peered":"2024-01-26T23:48:11.698643+0000","last_clean":"2024-01-26T23:48:11.698643+0000","last_became_active":"2024-01-26T23:48:11.692446+0000","last_became_peered":"2024-01-26T23:48:11.692446+0000","last_unstale":"2024-01-26T23:48:11.698643+0000","last_undegraded":"2024-01-26T23:48:11.698643+0000","last_fullsized":"2024-01-26T23:48:11.698643+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:54:10.694699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702415+0000","last_change":"2024-01-26T23:48:11.696828+0000","last_active":"2024-01-26T23:48:11.702415+0000","last_peered":"2024-01-26T23:48:11.702415+0000","last_clean":"2024-01-26T23:48:11.702415+0000","last_became_active":"2024-01-26T23:48:11.695115+0000","last_became_peered":"2024-01-26T23:48:11.695115+0000","last_unstale":"2024-01-26T23:48:11.702415+0000","last_undegraded":"2024-01-26T23:48:11.702415+0000","last_fullsized":"2024-01-26T23:48:11.702415+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:25.597544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.687089+0000","last_change":"2024-01-26T23:48:10.815435+0000","last_active":"2024-01-26T23:48:11.687089+0000","last_peered":"2024-01-26T23:48:11.687089+0000","last_clean":"2024-01-26T23:48:11.687089+0000","last_became_active":"2024-01-26T23:48:10.815136+0000","last_became_peered":"2024-01-26T23:48:10.815136+0000","last_unstale":"2024-01-26T23:48:11.687089+0000","last_undegraded":"2024-01-26T23:48:11.687089+0000","last_fullsized":"2024-01-26T23:48:11.687089+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:36:28.278118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685337+0000","last_change":"2024-01-26T23:48:10.817416+0000","last_active":"2024-01-26T23:48:11.685337+0000","last_peered":"2024-01-26T23:48:11.685337+0000","last_clean":"2024-01-26T23:48:11.685337+0000","last_became_active":"2024-01-26T23:48:10.817077+0000","last_became_peered":"2024-01-26T23:48:10.817077+0000","last_unstale":"2024-01-26T23:48:11.685337+0000","last_undegraded":"2024-01-26T23:48:11.685337+0000","last_fullsized":"2024-01-26T23:48:11.685337+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:05.190741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.e","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.804097+0000","last_change":"2024-01-26T23:48:10.814833+0000","last_active":"2024-01-26T23:48:11.804097+0000","last_peered":"2024-01-26T23:48:11.804097+0000","last_clean":"2024-01-26T23:48:11.804097+0000","last_became_active":"2024-01-26T23:48:10.814499+0000","last_became_peered":"2024-01-26T23:48:10.814499+0000","last_unstale":"2024-01-26T23:48:11.804097+0000","last_undegraded":"2024-01-26T23:48:11.804097+0000","last_fullsized":"2024-01-26T23:48:11.804097+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:27:02.987035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"120'8562","reported_seq":9013,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-01-26T23:48:06.755362+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:48:06.755362+0000","last_peered":"2024-01-26T23:48:06.755362+0000","last_clean":"2024-01-26T23:48:06.755362+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:48:06.755362+0000","last_undegraded":"2024-01-26T23:48:06.755362+0000","last_fullsized":"2024-01-26T23:48:06.755362+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8562,"log_dups_size":0,"ondisk_log_size":8562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":971,"num_read_kb":1902,"num_write":320,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":22,"acting":22,"num_store_stats":1},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":1},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":971,"num_read_kb":1902,"num_write":320,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8562,"ondisk_log_size":8562,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575069,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311680,"kb_used_data":2104,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060160,"statfs":{"total":96636764160,"available":96317603840,"internally_reserved":0,"allocated":2154496,"data_stored":1388766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.503,"15min":0.56100000000000005},"min":{"1min":0.307,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65200000000000002,"5min":0.79500000000000004,"15min":0.82299999999999995},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.52000000000000002,"15min":0.57899999999999996},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60599999999999998,"5min":0.82099999999999995,"15min":0.83999999999999997},"last":0.38}]},{"osd":1,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.52000000000000002,"15min":0.57499999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.68000000000000005,"5min":0.77800000000000002,"15min":0.82899999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.55600000000000005,"15min":0.59599999999999997},"min":{"1min":0.34999999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.64000000000000001,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.46200000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54000000000000004,"15min":0.57599999999999996},"min":{"1min":0.38,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.66400000000000003,"5min":0.80000000000000004,"15min":0.81699999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.54800000000000004,"15min":0.59099999999999997},"min":{"1min":0.34000000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72699999999999998,"5min":0.81899999999999995,"15min":0.88600000000000001},"last":0.435}]},{"osd":3,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.38900000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.66200000000000003,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.56299999999999994,"15min":0.59699999999999998},"min":{"1min":0.38100000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69799999999999995,"5min":0.83899999999999997,"15min":0.86399999999999999},"last":0.626}]},{"osd":4,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.43099999999999999,"15min":0.44600000000000001},"min":{"1min":0.28699999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.44600000000000001,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.45700000000000002,"15min":0.47199999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.45000000000000001,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.54700000000000004}]},{"osd":5,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.47299999999999998,"15min":0.48099999999999998},"min":{"1min":0.27300000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.61099999999999999,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.377,"5min":0.501,"15min":0.51600000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.47299999999999998,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.59899999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.53300000000000003,"15min":0.54200000000000004},"min":{"1min":0.34399999999999997,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.56899999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.52100000000000002,"15min":0.53700000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55800000000000005,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.61499999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":302908,"kb_used_data":1524,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068932,"statfs":{"total":96636764160,"available":96326586368,"internally_reserved":0,"allocated":1560576,"data_stored":798398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.53300000000000003,"15min":0.56799999999999995},"min":{"1min":0.379,"5min":0.255,"15min":0.255},"max":{"1min":0.78700000000000003,"5min":1.423,"15min":1.423},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.53800000000000003,"15min":0.58399999999999996},"min":{"1min":0.45000000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.72999999999999998,"5min":0.79000000000000004,"15min":0.91100000000000003},"last":0.50700000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.52400000000000002,"15min":0.56399999999999995},"min":{"1min":0.39800000000000002,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.76900000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55300000000000005,"15min":0.58999999999999997},"min":{"1min":0.432,"5min":0.312,"15min":0.312},"max":{"1min":0.749,"5min":1.498,"15min":1.498},"last":0.59999999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53100000000000003,"15min":0.57399999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72699999999999998,"5min":0.74199999999999999,"15min":0.85799999999999998},"last":0.436},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53500000000000003,"15min":0.57799999999999996},"min":{"1min":0.39400000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71799999999999997,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.48799999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57199999999999995,"15min":0.59399999999999997},"min":{"1min":0.37,"5min":0.33800000000000002,"15min":0.29999999999999999},"max":{"1min":0.745,"5min":1.595,"15min":1.595},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.61199999999999999},"min":{"1min":0.39000000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.80300000000000005,"5min":1.369,"15min":1.369},"last":0.65800000000000003}]},{"osd":4,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.45100000000000001,"15min":0.46000000000000002},"min":{"1min":0.32200000000000001,"5min":0.245,"15min":0.22},"max":{"1min":0.64400000000000002,"5min":1.556,"15min":1.556},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.47499999999999998,"15min":0.48299999999999998},"min":{"1min":0.32000000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65900000000000003,"5min":0.753,"15min":0.86199999999999999},"last":0.627}]},{"osd":5,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45900000000000002,"15min":0.47299999999999998},"min":{"1min":0.32600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.59699999999999998,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52000000000000002,"15min":0.51900000000000002},"min":{"1min":0.27200000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.72699999999999998,"5min":1.47,"15min":1.47},"last":0.64200000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.505,"15min":0.51800000000000002},"min":{"1min":0.41299999999999998,"5min":0.26700000000000002,"15min":0.246},"max":{"1min":0.68700000000000006,"5min":0.82199999999999995,"15min":0.85899999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.374,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.67400000000000004,"5min":0.78500000000000003,"15min":0.80400000000000005},"last":0.54700000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307004,"kb_used_data":1524,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064836,"statfs":{"total":96636764160,"available":96322392064,"internally_reserved":0,"allocated":1560576,"data_stored":798398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.41699999999999998,"15min":0.40699999999999997},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61799999999999999,"5min":0.65200000000000002,"15min":0.72599999999999998},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.442,"15min":0.443},"min":{"1min":0.28999999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.63500000000000001,"5min":1.337,"15min":1.337},"last":0.373}]},{"osd":2,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40400000000000003,"15min":0.41099999999999998},"min":{"1min":0.28000000000000003,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.60199999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.71999999999999997,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.35399999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.51500000000000001},"min":{"1min":0.40000000000000002,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":0.77900000000000003,"5min":1.782,"15min":1.782},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48599999999999999,"15min":0.47999999999999998},"min":{"1min":0.38800000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.69899999999999995,"5min":1.379,"15min":1.379},"last":0.42199999999999999}]},{"osd":4,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.61199999999999999,"15min":0.64100000000000001},"min":{"1min":0.40799999999999997,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":0.73299999999999998,"5min":0.94099999999999995,"15min":1.0820000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63900000000000001,"15min":0.65700000000000003},"min":{"1min":0.443,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.872,"5min":0.89900000000000002,"15min":0.95699999999999996},"last":0.55900000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.65600000000000003,"15min":0.66900000000000004},"min":{"1min":0.501,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":0.68899999999999995,"5min":1.208,"15min":1.208},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.47099999999999997,"5min":0.45100000000000001,"15min":0.42499999999999999},"max":{"1min":0.77700000000000002,"5min":0.96599999999999997,"15min":1.018},"last":0.67600000000000005}]},{"osd":6,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.65900000000000003,"15min":0.67300000000000004},"min":{"1min":0.498,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.78400000000000003,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.73099999999999998,"5min":1.01,"15min":1.022},"last":0.70499999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.68400000000000005,"15min":0.70599999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.875,"5min":0.98199999999999998,"15min":1.0069999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.70399999999999996,"15min":0.71799999999999997},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.753,"5min":1.296,"15min":1.296},"last":0.54600000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575068,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307584,"kb_used_data":2104,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064256,"statfs":{"total":96636764160,"available":96321798144,"internally_reserved":0,"allocated":2154496,"data_stored":1388766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.41699999999999998,"15min":0.40600000000000003},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.50800000000000001,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.44,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.27400000000000002,"15min":0.27300000000000002},"max":{"1min":0.58899999999999997,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.34499999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40600000000000003,"15min":0.42399999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.67600000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45300000000000001,"15min":0.47199999999999998},"min":{"1min":0.34100000000000003,"5min":0.32500000000000001,"15min":0.28000000000000003},"max":{"1min":0.64300000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.46300000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47699999999999998,"15min":0.47199999999999998},"min":{"1min":0.35199999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.623,"5min":2.145,"15min":2.145},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48099999999999998,"15min":0.48799999999999999},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.63100000000000001,"5min":1.198,"15min":1.198},"last":0.54000000000000004}]},{"osd":4,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.621,"15min":0.64800000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.81399999999999995,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.59699999999999998,"15min":0.64300000000000002},"min":{"1min":0.38500000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.72099999999999997,"5min":0.88,"15min":0.98899999999999999},"last":0.77300000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.628,"15min":0.65400000000000003},"min":{"1min":0.45000000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78800000000000003,"5min":1.329,"15min":1.329},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.66400000000000003,"15min":0.68100000000000005},"min":{"1min":0.47199999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93100000000000005,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.627}]},{"osd":6,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.61099999999999999,"15min":0.66100000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80000000000000004,"5min":0.89800000000000002,"15min":1.032},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.65900000000000003,"15min":0.68999999999999995},"min":{"1min":0.48999999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82399999999999995,"5min":1.286,"15min":1.286},"last":0.78900000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":1.012,"15min":0.82699999999999996},"min":{"1min":0.498,"5min":0.435,"15min":0.435},"max":{"1min":0.76900000000000002,"5min":25.965,"15min":25.965},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":1.123,"15min":0.873},"min":{"1min":0.5,"5min":0.48799999999999999,"15min":0.433},"max":{"1min":0.89000000000000001,"5min":33.085000000000001,"15min":33.085000000000001},"last":0.83099999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575069,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307056,"kb_used_data":1576,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064784,"statfs":{"total":96636764160,"available":96322338816,"internally_reserved":0,"allocated":1613824,"data_stored":826399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.39100000000000001},"min":{"1min":0.27200000000000002,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.434}]},{"osd":1,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.42899999999999999,"15min":0.42799999999999999},"min":{"1min":0.27400000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.57999999999999996,"5min":0.61899999999999999,"15min":0.70499999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.50900000000000001,"15min":0.50600000000000001},"min":{"1min":0.379,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.51000000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46300000000000002,"15min":0.48599999999999999},"min":{"1min":0.32500000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.74099999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47899999999999998,"15min":0.47599999999999998},"min":{"1min":0.33500000000000002,"5min":0.32200000000000001,"15min":0.307},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.47799999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.60099999999999998,"15min":0.629},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.75800000000000001,"5min":1.087,"15min":1.087},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.628,"15min":0.64000000000000001},"min":{"1min":0.374,"5min":0.371,"15min":0.371},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.70599999999999996}]},{"osd":5,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63600000000000001,"15min":0.65400000000000003},"min":{"1min":0.44500000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.65000000000000002,"15min":0.67400000000000004},"min":{"1min":0.48199999999999998,"5min":0.39800000000000002,"15min":0.35199999999999998},"max":{"1min":0.80600000000000005,"5min":0.97699999999999998,"15min":1.085},"last":0.56000000000000005}]},{"osd":6,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65900000000000003,"15min":0.67800000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89600000000000002,"5min":1.022,"15min":1.1080000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.65700000000000003,"15min":0.67900000000000005},"min":{"1min":0.49099999999999999,"5min":0.42799999999999999,"15min":0.38800000000000001},"max":{"1min":0.88,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.58299999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68400000000000005,"15min":0.70199999999999996},"min":{"1min":0.53100000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.69799999999999995,"15min":0.72899999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.42099999999999999},"max":{"1min":0.84199999999999997,"5min":1.052,"15min":1.139},"last":0.61899999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575067,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307004,"kb_used_data":1524,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064836,"statfs":{"total":96636764160,"available":96322392064,"internally_reserved":0,"allocated":1560576,"data_stored":798398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.38200000000000001,"15min":0.39100000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":1.8540000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.441,"5min":0.42099999999999999,"15min":0.44},"min":{"1min":0.30599999999999999,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":1.2290000000000001},"last":0.33000000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44500000000000001,"15min":0.45400000000000001},"min":{"1min":0.33100000000000002,"5min":0.27900000000000003,"15min":0.253},"max":{"1min":0.63200000000000001,"5min":0.63800000000000001,"15min":1.633},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.48099999999999998,"15min":0.48499999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":0.65900000000000003,"5min":0.67800000000000005,"15min":1.0760000000000001},"last":0.315}]},{"osd":2,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.47399999999999998,"15min":0.47999999999999998},"min":{"1min":0.31900000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68600000000000005,"5min":0.70399999999999996,"15min":1.4059999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.439,"15min":0.45900000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28299999999999997},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":1.1870000000000001},"last":0.29499999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.61699999999999999,"15min":0.65800000000000003},"min":{"1min":0.45400000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.86799999999999999,"5min":0.878,"15min":3.306},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.628,"15min":0.65700000000000003},"min":{"1min":0.50700000000000001,"5min":0.40100000000000002,"15min":0.379},"max":{"1min":0.83899999999999997,"5min":0.85199999999999998,"15min":1.008},"last":0.73299999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.62,"15min":0.65300000000000002},"min":{"1min":0.49199999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":1.593},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63500000000000001,"15min":0.66600000000000004},"min":{"1min":0.499,"5min":0.379,"15min":0.379},"max":{"1min":0.79500000000000004,"5min":0.92300000000000004,"15min":1.3819999999999999},"last":0.57099999999999995}]},{"osd":6,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63400000000000001,"15min":0.66900000000000004},"min":{"1min":0.49099999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85599999999999998,"5min":0.88800000000000001,"15min":1.506},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.66000000000000003,"15min":0.68799999999999994},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.44400000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":1.427},"last":0.60099999999999998}]},{"osd":7,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.63700000000000001,"15min":0.68799999999999994},"min":{"1min":0.442,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":3.238},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.69699999999999995,"15min":0.73299999999999998},"min":{"1min":0.55000000000000004,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":3.351},"last":0.61199999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":302908,"kb_used_data":1524,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068932,"statfs":{"total":96636764160,"available":96326586368,"internally_reserved":0,"allocated":1560576,"data_stored":798398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53400000000000003,"15min":0.57099999999999995},"min":{"1min":0.34699999999999998,"5min":0.28999999999999998,"15min":0.28699999999999998},"max":{"1min":0.71199999999999997,"5min":1.21,"15min":1.21},"last":0.443},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.35999999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.76400000000000001,"5min":5.367,"15min":5.367},"last":0.48499999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55700000000000005,"15min":0.58899999999999997},"min":{"1min":0.38800000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.88},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.53400000000000003,"15min":0.57699999999999996},"min":{"1min":0.35199999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.621,"5min":0.80600000000000005,"15min":0.81699999999999995},"last":0.40699999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52700000000000002,"15min":0.56699999999999995},"min":{"1min":0.33900000000000002,"5min":0.33300000000000002,"15min":0.32700000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.501},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.57199999999999995,"15min":0.59699999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80500000000000005,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.43099999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55100000000000005,"15min":0.58399999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.68500000000000005,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56200000000000006,"15min":0.58299999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.35699999999999998},"max":{"1min":0.94199999999999995,"5min":1.05,"15min":1.05},"last":0.45500000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.44400000000000001,"15min":0.45800000000000002},"min":{"1min":0.29199999999999998,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.68300000000000005,"5min":0.76200000000000001,"15min":0.77600000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.441,"5min":0.47499999999999998,"15min":0.47699999999999998},"min":{"1min":0.29099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72499999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.51700000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45200000000000001,"15min":0.46700000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27200000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.78900000000000003,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.58699999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.51400000000000001,"15min":0.53400000000000003},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.63700000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.5,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.36899999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.55700000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":298812,"kb_used_data":1524,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073028,"statfs":{"total":96636764160,"available":96330780672,"internally_reserved":0,"allocated":1560576,"data_stored":798398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.55100000000000005,"15min":0.56699999999999995},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.65600000000000003,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.29999999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63600000000000001,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.35799999999999998}]},{"osd":1,"last update":"Fri Jan 26 23:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.58299999999999996,"15min":0.61199999999999999},"min":{"1min":0.32100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.61899999999999999,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.60099999999999998,"15min":0.629},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69999999999999996,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.47199999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.53500000000000003,"15min":0.56499999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.30099999999999999},"max":{"1min":0.64800000000000002,"5min":0.77000000000000002,"15min":0.83799999999999997},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.56299999999999994,"15min":0.58799999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68899999999999995,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":0.52900000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.58199999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.314},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.439},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58799999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.72399999999999998,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.54600000000000004}]},{"osd":4,"last update":"Fri Jan 26 23:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.441,"15min":0.45200000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67200000000000004,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.623},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.46800000000000003,"15min":0.47899999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":0.65300000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.40300000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.46700000000000003,"15min":0.47199999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.59699999999999998,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.505,"15min":0.51100000000000001},"min":{"1min":0.33600000000000002,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.69499999999999995,"5min":1.99,"15min":1.99},"last":0.60499999999999998}]},{"osd":7,"last update":"Fri Jan 26 23:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.54600000000000004,"15min":0.53800000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63400000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.504},{"interface":"front","average":{"1min":0.503,"5min":0.53000000000000003,"15min":0.55700000000000005},"min":{"1min":0.38300000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.67700000000000005,"5min":0.81499999999999995,"15min":0.82199999999999995},"last":0.51300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:48:16.008 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:48:16.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.175+0000 7f5af844e700 1 Processor -- start 2024-01-26T23:48:16.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.176+0000 7f5af844e700 1 -- start start 2024-01-26T23:48:16.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.177+0000 7f5af844e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5af00acf30 0x7f5af00c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:16.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.177+0000 7f5af844e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5af00ad300 0x7f5af00ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:16.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.177+0000 7f5af844e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af00c4230 con 0x7f5af00ad750 2024-01-26T23:48:16.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.177+0000 7f5af844e700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af0157be0 con 0x7f5af00acf30 2024-01-26T23:48:16.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.177+0000 7f5af844e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af00cbf10 con 0x7f5af00ad300 2024-01-26T23:48:16.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.177+0000 7f5af61ea700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5af00acf30 0x7f5af00c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:16.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.177+0000 7f5af59e9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5af00ad300 0x7f5af00ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:16.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.177+0000 7f5af69eb700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f5af00ad750 0x7f5af015b270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:42550/0 (socket says 172.21.15.46:42550) 2024-01-26T23:48:16.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.177+0000 7f5af59e9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5af00ad300 0x7f5af00ccb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:59674/0 (socket says 172.21.15.46:59674) 2024-01-26T23:48:16.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.177+0000 7f5af69eb700 1 -- 172.21.15.46:0/3186309894 learned_addr learned my addr 172.21.15.46:0/3186309894 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:16.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.177+0000 7f5af61ea700 1 --2- 172.21.15.46:0/3186309894 >> v2:172.21.15.106:3300/0 conn(0x7f5af00acf30 0x7f5af00c3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:34554/0 (socket says 172.21.15.46:34554) 2024-01-26T23:48:16.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.177+0000 7f5af51e8700 1 -- 172.21.15.46:0/3186309894 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564360988 0 0) 0x7f5af00c4230 con 0x7f5af00ad750 2024-01-26T23:48:16.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.177+0000 7f5af59e9700 1 -- 172.21.15.46:0/3186309894 >> v2:172.21.15.106:3300/0 conn(0x7f5af00acf30 msgr2=0x7f5af00c3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:16.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.177+0000 7f5af59e9700 1 --2- 172.21.15.46:0/3186309894 >> v2:172.21.15.106:3300/0 conn(0x7f5af00acf30 0x7f5af00c3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.177+0000 7f5af59e9700 1 -- 172.21.15.46:0/3186309894 >> v1:172.21.15.106:6789/0 conn(0x7f5af00ad750 legacy=0x7f5af015b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:16.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.177+0000 7f5af59e9700 1 -- 172.21.15.46:0/3186309894 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af00c4460 con 0x7f5af00ad300 2024-01-26T23:48:16.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.177+0000 7f5af61ea700 1 --2- 172.21.15.46:0/3186309894 >> v2:172.21.15.106:3300/0 conn(0x7f5af00acf30 0x7f5af00c3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:48:16.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.178+0000 7f5af59e9700 1 --2- 172.21.15.46:0/3186309894 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5af00ad300 0x7f5af00ccb40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5ae00098e0 tx=0x7f5ae00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8277b74cf969b4a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:16.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.178+0000 7f5af51e8700 1 -- 172.21.15.46:0/3186309894 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae000b640 con 0x7f5af00ad300 2024-01-26T23:48:16.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.178+0000 7f5af51e8700 1 -- 172.21.15.46:0/3186309894 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae000b7a0 con 0x7f5af00ad300 2024-01-26T23:48:16.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.178+0000 7f5af51e8700 1 -- 172.21.15.46:0/3186309894 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae000bb40 con 0x7f5af00ad300 2024-01-26T23:48:16.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.179+0000 7f5af844e700 1 -- 172.21.15.46:0/3186309894 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5af00ad300 msgr2=0x7f5af00ccb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:16.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.179+0000 7f5af844e700 1 --2- 172.21.15.46:0/3186309894 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5af00ad300 0x7f5af00ccb40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5ae00098e0 tx=0x7f5ae00091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.179+0000 7f5af844e700 1 -- 172.21.15.46:0/3186309894 shutdown_connections 2024-01-26T23:48:16.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.179+0000 7f5af844e700 1 --2- 172.21.15.46:0/3186309894 >> v2:172.21.15.106:3300/0 conn(0x7f5af00acf30 0x7f5af00c3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.179+0000 7f5af844e700 1 --2- 172.21.15.46:0/3186309894 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5af00ad300 0x7f5af00ccb40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.179+0000 7f5af844e700 1 -- 172.21.15.46:0/3186309894 >> 172.21.15.46:0/3186309894 conn(0x7f5af0097820 msgr2=0x7f5af0068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:16.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.179+0000 7f5af844e700 1 -- 172.21.15.46:0/3186309894 shutdown_connections 2024-01-26T23:48:16.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.179+0000 7f5af844e700 1 -- 172.21.15.46:0/3186309894 wait complete. 2024-01-26T23:48:16.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.180+0000 7f5af844e700 1 Processor -- start 2024-01-26T23:48:16.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.180+0000 7f5af844e700 1 -- start start 2024-01-26T23:48:16.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.181+0000 7f5af844e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af00acf30 0x7f5af0153780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:16.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.181+0000 7f5af844e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5af00ad300 0x7f5af0069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:16.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.181+0000 7f5af844e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5af00ad750 0x7f5af00c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:16.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.181+0000 7f5af844e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af00cb7e0 con 0x7f5af00acf30 2024-01-26T23:48:16.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.181+0000 7f5af844e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af00c9850 con 0x7f5af00ad300 2024-01-26T23:48:16.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.181+0000 7f5af844e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af00caf80 con 0x7f5af00ad750 2024-01-26T23:48:16.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.181+0000 7f5af61ea700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af00acf30 0x7f5af0153780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:16.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.181+0000 7f5af61ea700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af00acf30 0x7f5af0153780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39006/0 (socket says 172.21.15.46:39006) 2024-01-26T23:48:16.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.181+0000 7f5af61ea700 1 -- 172.21.15.46:0/3926264 learned_addr learned my addr 172.21.15.46:0/3926264 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:16.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.181+0000 7f5af59e9700 1 --2- 172.21.15.46:0/3926264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5af00ad300 0x7f5af0069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:16.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.181+0000 7f5af69eb700 1 --2- 172.21.15.46:0/3926264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5af00ad750 0x7f5af00c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:16.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.181+0000 7f5af61ea700 1 -- 172.21.15.46:0/3926264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5af00ad750 msgr2=0x7f5af00c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:16.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.181+0000 7f5af61ea700 1 --2- 172.21.15.46:0/3926264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5af00ad750 0x7f5af00c9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.181+0000 7f5af61ea700 1 -- 172.21.15.46:0/3926264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5af00ad300 msgr2=0x7f5af0069450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:16.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.181+0000 7f5af61ea700 1 --2- 172.21.15.46:0/3926264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5af00ad300 0x7f5af0069450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.181+0000 7f5af61ea700 1 -- 172.21.15.46:0/3926264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec009420 con 0x7f5af00acf30 2024-01-26T23:48:16.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.182+0000 7f5af61ea700 1 --2- 172.21.15.46:0/3926264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af00acf30 0x7f5af0153780 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00ecb0 tx=0x7f5aec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:16.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.182+0000 7f5ae67fc700 1 -- 172.21.15.46:0/3926264 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec00c920 con 0x7f5af00acf30 2024-01-26T23:48:16.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.182+0000 7f5af844e700 1 -- 172.21.15.46:0/3926264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae0009420 con 0x7f5af00acf30 2024-01-26T23:48:16.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.182+0000 7f5ae67fc700 1 -- 172.21.15.46:0/3926264 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aec00ca80 con 0x7f5af00acf30 2024-01-26T23:48:16.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.182+0000 7f5ae67fc700 1 -- 172.21.15.46:0/3926264 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec00ce00 con 0x7f5af00acf30 2024-01-26T23:48:16.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.182+0000 7f5af844e700 1 -- 172.21.15.46:0/3926264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af00c94e0 con 0x7f5af00acf30 2024-01-26T23:48:16.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.183+0000 7f5af844e700 1 -- 172.21.15.46:0/3926264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af00c9b30 con 0x7f5af00acf30 2024-01-26T23:48:16.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.183+0000 7f5ae67fc700 1 -- 172.21.15.46:0/3926264 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5aec013680 con 0x7f5af00acf30 2024-01-26T23:48:16.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.184+0000 7f5ae67fc700 1 --2- 172.21.15.46:0/3926264 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5ad4066f30 0x7f5ad40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:16.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.184+0000 7f5ae67fc700 1 -- 172.21.15.46:0/3926264 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f5aec084b50 con 0x7f5af00acf30 2024-01-26T23:48:16.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.184+0000 7f5af59e9700 1 --2- 172.21.15.46:0/3926264 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5ad4066f30 0x7f5ad40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:16.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.184+0000 7f5af59e9700 1 --2- 172.21.15.46:0/3926264 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5ad4066f30 0x7f5ad40693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5ae00093f0 tx=0x7f5ae0017000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.188+0000 7f5ae67fc700 1 -- 172.21.15.46:0/3926264 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5aec051af0 con 0x7f5af00acf30 2024-01-26T23:48:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.388+0000 7f5af844e700 1 -- 172.21.15.46:0/3926264 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5af00cb220 con 0x7f5ad4066f30 2024-01-26T23:48:16.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.396+0000 7f5ae67fc700 1 -- 172.21.15.46:0/3926264 <== mgr.7396 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277082 (secure 0 0 0) 0x7f5af00cb220 con 0x7f5ad4066f30 2024-01-26T23:48:16.400 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:48:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.399+0000 7f5af844e700 1 -- 172.21.15.46:0/3926264 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5ad4066f30 msgr2=0x7f5ad40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.399+0000 7f5af844e700 1 --2- 172.21.15.46:0/3926264 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5ad4066f30 0x7f5ad40693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5ae00093f0 tx=0x7f5ae0017000 comp rx=0 tx=0).stop 2024-01-26T23:48:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.399+0000 7f5af844e700 1 -- 172.21.15.46:0/3926264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af00acf30 msgr2=0x7f5af0153780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.399+0000 7f5af844e700 1 --2- 172.21.15.46:0/3926264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af00acf30 0x7f5af0153780 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00ecb0 tx=0x7f5aec0091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.399+0000 7f5af844e700 1 -- 172.21.15.46:0/3926264 shutdown_connections 2024-01-26T23:48:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.399+0000 7f5af844e700 1 --2- 172.21.15.46:0/3926264 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5ad4066f30 0x7f5ad40693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.399+0000 7f5af844e700 1 --2- 172.21.15.46:0/3926264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af00acf30 0x7f5af0153780 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.399+0000 7f5af844e700 1 --2- 172.21.15.46:0/3926264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5af00ad750 0x7f5af00c9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.399+0000 7f5af844e700 1 --2- 172.21.15.46:0/3926264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5af00ad300 0x7f5af0069450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.399+0000 7f5af844e700 1 -- 172.21.15.46:0/3926264 >> 172.21.15.46:0/3926264 conn(0x7f5af0097820 msgr2=0x7f5af00cc330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.400+0000 7f5af844e700 1 -- 172.21.15.46:0/3926264 shutdown_connections 2024-01-26T23:48:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.400+0000 7f5af844e700 1 -- 172.21.15.46:0/3926264 wait complete. 2024-01-26T23:48:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:48:16.414 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-26T23:48:14.716316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":971,"num_read_kb":1902,"num_write":324,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8567,"ondisk_log_size":8567,"up":40,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":754974720,"kb_used":2444956,"kb_used_data":13404,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752529764,"statfs":{"total":773094113280,"available":770590478336,"internally_reserved":0,"allocated":13725696,"data_stored":7595921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.927983"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702212+0000","last_change":"2024-01-26T23:48:11.695520+0000","last_active":"2024-01-26T23:48:11.702212+0000","last_peered":"2024-01-26T23:48:11.702212+0000","last_clean":"2024-01-26T23:48:11.702212+0000","last_became_active":"2024-01-26T23:48:11.695205+0000","last_became_peered":"2024-01-26T23:48:11.695205+0000","last_unstale":"2024-01-26T23:48:11.702212+0000","last_undegraded":"2024-01-26T23:48:11.702212+0000","last_fullsized":"2024-01-26T23:48:11.702212+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:36:53.012187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.690830+0000","last_change":"2024-01-26T23:48:11.689380+0000","last_active":"2024-01-26T23:48:11.690830+0000","last_peered":"2024-01-26T23:48:11.690830+0000","last_clean":"2024-01-26T23:48:11.690830+0000","last_became_active":"2024-01-26T23:48:11.688997+0000","last_became_peered":"2024-01-26T23:48:11.688997+0000","last_unstale":"2024-01-26T23:48:11.690830+0000","last_undegraded":"2024-01-26T23:48:11.690830+0000","last_fullsized":"2024-01-26T23:48:11.690830+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:20.796536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702333+0000","last_change":"2024-01-26T23:48:11.700735+0000","last_active":"2024-01-26T23:48:11.702333+0000","last_peered":"2024-01-26T23:48:11.702333+0000","last_clean":"2024-01-26T23:48:11.702333+0000","last_became_active":"2024-01-26T23:48:11.700416+0000","last_became_peered":"2024-01-26T23:48:11.700416+0000","last_unstale":"2024-01-26T23:48:11.702333+0000","last_undegraded":"2024-01-26T23:48:11.702333+0000","last_fullsized":"2024-01-26T23:48:11.702333+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:03:16.363445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699884+0000","last_change":"2024-01-26T23:48:11.694600+0000","last_active":"2024-01-26T23:48:11.699884+0000","last_peered":"2024-01-26T23:48:11.699884+0000","last_clean":"2024-01-26T23:48:11.699884+0000","last_became_active":"2024-01-26T23:48:11.694195+0000","last_became_peered":"2024-01-26T23:48:11.694195+0000","last_unstale":"2024-01-26T23:48:11.699884+0000","last_undegraded":"2024-01-26T23:48:11.699884+0000","last_fullsized":"2024-01-26T23:48:11.699884+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:54:52.459709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702321+0000","last_change":"2024-01-26T23:48:11.699321+0000","last_active":"2024-01-26T23:48:11.702321+0000","last_peered":"2024-01-26T23:48:11.702321+0000","last_clean":"2024-01-26T23:48:11.702321+0000","last_became_active":"2024-01-26T23:48:11.699085+0000","last_became_peered":"2024-01-26T23:48:11.699085+0000","last_unstale":"2024-01-26T23:48:11.702321+0000","last_undegraded":"2024-01-26T23:48:11.702321+0000","last_fullsized":"2024-01-26T23:48:11.702321+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:49:57.681990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"129'1","reported_seq":11,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.805807+0000","last_change":"2024-01-26T23:48:10.816548+0000","last_active":"2024-01-26T23:48:11.805807+0000","last_peered":"2024-01-26T23:48:11.805807+0000","last_clean":"2024-01-26T23:48:11.805807+0000","last_became_active":"2024-01-26T23:48:10.816271+0000","last_became_peered":"2024-01-26T23:48:10.816271+0000","last_unstale":"2024-01-26T23:48:11.805807+0000","last_undegraded":"2024-01-26T23:48:11.805807+0000","last_fullsized":"2024-01-26T23:48:11.805807+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:10.976028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702237+0000","last_change":"2024-01-26T23:48:11.695812+0000","last_active":"2024-01-26T23:48:11.702237+0000","last_peered":"2024-01-26T23:48:11.702237+0000","last_clean":"2024-01-26T23:48:11.702237+0000","last_became_active":"2024-01-26T23:48:11.695535+0000","last_became_peered":"2024-01-26T23:48:11.695535+0000","last_unstale":"2024-01-26T23:48:11.702237+0000","last_undegraded":"2024-01-26T23:48:11.702237+0000","last_fullsized":"2024-01-26T23:48:11.702237+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:12:39.189557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699954+0000","last_change":"2024-01-26T23:48:11.694802+0000","last_active":"2024-01-26T23:48:11.699954+0000","last_peered":"2024-01-26T23:48:11.699954+0000","last_clean":"2024-01-26T23:48:11.699954+0000","last_became_active":"2024-01-26T23:48:11.694277+0000","last_became_peered":"2024-01-26T23:48:11.694277+0000","last_unstale":"2024-01-26T23:48:11.699954+0000","last_undegraded":"2024-01-26T23:48:11.699954+0000","last_fullsized":"2024-01-26T23:48:11.699954+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:28:58.191038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.684638+0000","last_change":"2024-01-26T23:48:10.813933+0000","last_active":"2024-01-26T23:48:11.684638+0000","last_peered":"2024-01-26T23:48:11.684638+0000","last_clean":"2024-01-26T23:48:11.684638+0000","last_became_active":"2024-01-26T23:48:10.813646+0000","last_became_peered":"2024-01-26T23:48:10.813646+0000","last_unstale":"2024-01-26T23:48:11.684638+0000","last_undegraded":"2024-01-26T23:48:11.684638+0000","last_fullsized":"2024-01-26T23:48:11.684638+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:20:47.863263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699904+0000","last_change":"2024-01-26T23:48:11.694643+0000","last_active":"2024-01-26T23:48:11.699904+0000","last_peered":"2024-01-26T23:48:11.699904+0000","last_clean":"2024-01-26T23:48:11.699904+0000","last_became_active":"2024-01-26T23:48:11.694296+0000","last_became_peered":"2024-01-26T23:48:11.694296+0000","last_unstale":"2024-01-26T23:48:11.699904+0000","last_undegraded":"2024-01-26T23:48:11.699904+0000","last_fullsized":"2024-01-26T23:48:11.699904+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:53:21.191149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.b","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.803125+0000","last_change":"2024-01-26T23:48:10.815064+0000","last_active":"2024-01-26T23:48:11.803125+0000","last_peered":"2024-01-26T23:48:11.803125+0000","last_clean":"2024-01-26T23:48:11.803125+0000","last_became_active":"2024-01-26T23:48:10.814778+0000","last_became_peered":"2024-01-26T23:48:10.814778+0000","last_unstale":"2024-01-26T23:48:11.803125+0000","last_undegraded":"2024-01-26T23:48:11.803125+0000","last_fullsized":"2024-01-26T23:48:11.803125+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:17.080283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.686302+0000","last_change":"2024-01-26T23:48:10.813851+0000","last_active":"2024-01-26T23:48:11.686302+0000","last_peered":"2024-01-26T23:48:11.686302+0000","last_clean":"2024-01-26T23:48:11.686302+0000","last_became_active":"2024-01-26T23:48:10.813510+0000","last_became_peered":"2024-01-26T23:48:10.813510+0000","last_unstale":"2024-01-26T23:48:11.686302+0000","last_undegraded":"2024-01-26T23:48:11.686302+0000","last_fullsized":"2024-01-26T23:48:11.686302+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:26:11.395397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698608+0000","last_change":"2024-01-26T23:48:11.692941+0000","last_active":"2024-01-26T23:48:11.698608+0000","last_peered":"2024-01-26T23:48:11.698608+0000","last_clean":"2024-01-26T23:48:11.698608+0000","last_became_active":"2024-01-26T23:48:11.692605+0000","last_became_peered":"2024-01-26T23:48:11.692605+0000","last_unstale":"2024-01-26T23:48:11.698608+0000","last_undegraded":"2024-01-26T23:48:11.698608+0000","last_fullsized":"2024-01-26T23:48:11.698608+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:01:01.083122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685230+0000","last_change":"2024-01-26T23:48:10.816002+0000","last_active":"2024-01-26T23:48:11.685230+0000","last_peered":"2024-01-26T23:48:11.685230+0000","last_clean":"2024-01-26T23:48:11.685230+0000","last_became_active":"2024-01-26T23:48:10.815730+0000","last_became_peered":"2024-01-26T23:48:10.815730+0000","last_unstale":"2024-01-26T23:48:11.685230+0000","last_undegraded":"2024-01-26T23:48:11.685230+0000","last_fullsized":"2024-01-26T23:48:11.685230+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:59.034288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698643+0000","last_change":"2024-01-26T23:48:11.692758+0000","last_active":"2024-01-26T23:48:11.698643+0000","last_peered":"2024-01-26T23:48:11.698643+0000","last_clean":"2024-01-26T23:48:11.698643+0000","last_became_active":"2024-01-26T23:48:11.692446+0000","last_became_peered":"2024-01-26T23:48:11.692446+0000","last_unstale":"2024-01-26T23:48:11.698643+0000","last_undegraded":"2024-01-26T23:48:11.698643+0000","last_fullsized":"2024-01-26T23:48:11.698643+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:54:10.694699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702415+0000","last_change":"2024-01-26T23:48:11.696828+0000","last_active":"2024-01-26T23:48:11.702415+0000","last_peered":"2024-01-26T23:48:11.702415+0000","last_clean":"2024-01-26T23:48:11.702415+0000","last_became_active":"2024-01-26T23:48:11.695115+0000","last_became_peered":"2024-01-26T23:48:11.695115+0000","last_unstale":"2024-01-26T23:48:11.702415+0000","last_undegraded":"2024-01-26T23:48:11.702415+0000","last_fullsized":"2024-01-26T23:48:11.702415+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:25.597544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.687089+0000","last_change":"2024-01-26T23:48:10.815435+0000","last_active":"2024-01-26T23:48:11.687089+0000","last_peered":"2024-01-26T23:48:11.687089+0000","last_clean":"2024-01-26T23:48:11.687089+0000","last_became_active":"2024-01-26T23:48:10.815136+0000","last_became_peered":"2024-01-26T23:48:10.815136+0000","last_unstale":"2024-01-26T23:48:11.687089+0000","last_undegraded":"2024-01-26T23:48:11.687089+0000","last_fullsized":"2024-01-26T23:48:11.687089+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:36:28.278118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:09.778766+0000","last_change":"2024-01-26T23:48:09.778766+0000","last_active":"2024-01-26T23:48:09.778766+0000","last_peered":"2024-01-26T23:48:09.778766+0000","last_clean":"2024-01-26T23:48:09.778766+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:09.778766+0000","last_undegraded":"2024-01-26T23:48:09.778766+0000","last_fullsized":"2024-01-26T23:48:09.778766+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685337+0000","last_change":"2024-01-26T23:48:10.817416+0000","last_active":"2024-01-26T23:48:11.685337+0000","last_peered":"2024-01-26T23:48:11.685337+0000","last_clean":"2024-01-26T23:48:11.685337+0000","last_became_active":"2024-01-26T23:48:10.817077+0000","last_became_peered":"2024-01-26T23:48:10.817077+0000","last_unstale":"2024-01-26T23:48:11.685337+0000","last_undegraded":"2024-01-26T23:48:11.685337+0000","last_fullsized":"2024-01-26T23:48:11.685337+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:05.190741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:48:10.785466+0000","last_change":"2024-01-26T23:48:10.785466+0000","last_active":"2024-01-26T23:48:10.785466+0000","last_peered":"2024-01-26T23:48:10.785466+0000","last_clean":"2024-01-26T23:48:10.785466+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:48:10.785466+0000","last_undegraded":"2024-01-26T23:48:10.785466+0000","last_fullsized":"2024-01-26T23:48:10.785466+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.e","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.804097+0000","last_change":"2024-01-26T23:48:10.814833+0000","last_active":"2024-01-26T23:48:11.804097+0000","last_peered":"2024-01-26T23:48:11.804097+0000","last_clean":"2024-01-26T23:48:11.804097+0000","last_became_active":"2024-01-26T23:48:10.814499+0000","last_became_peered":"2024-01-26T23:48:10.814499+0000","last_unstale":"2024-01-26T23:48:11.804097+0000","last_undegraded":"2024-01-26T23:48:11.804097+0000","last_fullsized":"2024-01-26T23:48:11.804097+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:27:02.987035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"120'8562","reported_seq":9013,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-01-26T23:48:06.755362+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:48:06.755362+0000","last_peered":"2024-01-26T23:48:06.755362+0000","last_clean":"2024-01-26T23:48:06.755362+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:48:06.755362+0000","last_undegraded":"2024-01-26T23:48:06.755362+0000","last_fullsized":"2024-01-26T23:48:06.755362+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8562,"log_dups_size":0,"ondisk_log_size":8562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":971,"num_read_kb":1902,"num_write":320,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":22,"acting":22,"num_store_stats":1},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":1},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":971,"num_read_kb":1902,"num_write":320,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8562,"ondisk_log_size":8562,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575069,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311680,"kb_used_data":2104,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060160,"statfs":{"total":96636764160,"available":96317603840,"internally_reserved":0,"allocated":2154496,"data_stored":1388766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.503,"15min":0.56100000000000005},"min":{"1min":0.307,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65200000000000002,"5min":0.79500000000000004,"15min":0.82299999999999995},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.52000000000000002,"15min":0.57899999999999996},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60599999999999998,"5min":0.82099999999999995,"15min":0.83999999999999997},"last":0.38}]},{"osd":1,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.52000000000000002,"15min":0.57499999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.68000000000000005,"5min":0.77800000000000002,"15min":0.82899999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.55600000000000005,"15min":0.59599999999999997},"min":{"1min":0.34999999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.64000000000000001,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.46200000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54000000000000004,"15min":0.57599999999999996},"min":{"1min":0.38,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.66400000000000003,"5min":0.80000000000000004,"15min":0.81699999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.54800000000000004,"15min":0.59099999999999997},"min":{"1min":0.34000000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72699999999999998,"5min":0.81899999999999995,"15min":0.88600000000000001},"last":0.435}]},{"osd":3,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.38900000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.66200000000000003,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.56299999999999994,"15min":0.59699999999999998},"min":{"1min":0.38100000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69799999999999995,"5min":0.83899999999999997,"15min":0.86399999999999999},"last":0.626}]},{"osd":4,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.43099999999999999,"15min":0.44600000000000001},"min":{"1min":0.28699999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.44600000000000001,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.45700000000000002,"15min":0.47199999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.45000000000000001,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.54700000000000004}]},{"osd":5,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.47299999999999998,"15min":0.48099999999999998},"min":{"1min":0.27300000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.61099999999999999,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.377,"5min":0.501,"15min":0.51600000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.47299999999999998,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.59899999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.53300000000000003,"15min":0.54200000000000004},"min":{"1min":0.34399999999999997,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.56899999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.52100000000000002,"15min":0.53700000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55800000000000005,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.61499999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":302908,"kb_used_data":1524,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068932,"statfs":{"total":96636764160,"available":96326586368,"internally_reserved":0,"allocated":1560576,"data_stored":798398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.53300000000000003,"15min":0.56799999999999995},"min":{"1min":0.379,"5min":0.255,"15min":0.255},"max":{"1min":0.78700000000000003,"5min":1.423,"15min":1.423},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.53800000000000003,"15min":0.58399999999999996},"min":{"1min":0.45000000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.72999999999999998,"5min":0.79000000000000004,"15min":0.91100000000000003},"last":0.50700000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.52400000000000002,"15min":0.56399999999999995},"min":{"1min":0.39800000000000002,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.76900000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55300000000000005,"15min":0.58999999999999997},"min":{"1min":0.432,"5min":0.312,"15min":0.312},"max":{"1min":0.749,"5min":1.498,"15min":1.498},"last":0.59999999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53100000000000003,"15min":0.57399999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72699999999999998,"5min":0.74199999999999999,"15min":0.85799999999999998},"last":0.436},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53500000000000003,"15min":0.57799999999999996},"min":{"1min":0.39400000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71799999999999997,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.48799999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57199999999999995,"15min":0.59399999999999997},"min":{"1min":0.37,"5min":0.33800000000000002,"15min":0.29999999999999999},"max":{"1min":0.745,"5min":1.595,"15min":1.595},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.61199999999999999},"min":{"1min":0.39000000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.80300000000000005,"5min":1.369,"15min":1.369},"last":0.65800000000000003}]},{"osd":4,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.45100000000000001,"15min":0.46000000000000002},"min":{"1min":0.32200000000000001,"5min":0.245,"15min":0.22},"max":{"1min":0.64400000000000002,"5min":1.556,"15min":1.556},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.47499999999999998,"15min":0.48299999999999998},"min":{"1min":0.32000000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65900000000000003,"5min":0.753,"15min":0.86199999999999999},"last":0.627}]},{"osd":5,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45900000000000002,"15min":0.47299999999999998},"min":{"1min":0.32600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.59699999999999998,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52000000000000002,"15min":0.51900000000000002},"min":{"1min":0.27200000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.72699999999999998,"5min":1.47,"15min":1.47},"last":0.64200000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.505,"15min":0.51800000000000002},"min":{"1min":0.41299999999999998,"5min":0.26700000000000002,"15min":0.246},"max":{"1min":0.68700000000000006,"5min":0.82199999999999995,"15min":0.85899999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.374,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.67400000000000004,"5min":0.78500000000000003,"15min":0.80400000000000005},"last":0.54700000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307004,"kb_used_data":1524,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064836,"statfs":{"total":96636764160,"available":96322392064,"internally_reserved":0,"allocated":1560576,"data_stored":798398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.41699999999999998,"15min":0.40699999999999997},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61799999999999999,"5min":0.65200000000000002,"15min":0.72599999999999998},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.442,"15min":0.443},"min":{"1min":0.28999999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.63500000000000001,"5min":1.337,"15min":1.337},"last":0.373}]},{"osd":2,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40400000000000003,"15min":0.41099999999999998},"min":{"1min":0.28000000000000003,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.60199999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.71999999999999997,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.35399999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.51500000000000001},"min":{"1min":0.40000000000000002,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":0.77900000000000003,"5min":1.782,"15min":1.782},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48599999999999999,"15min":0.47999999999999998},"min":{"1min":0.38800000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.69899999999999995,"5min":1.379,"15min":1.379},"last":0.42199999999999999}]},{"osd":4,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.61199999999999999,"15min":0.64100000000000001},"min":{"1min":0.40799999999999997,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":0.73299999999999998,"5min":0.94099999999999995,"15min":1.0820000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63900000000000001,"15min":0.65700000000000003},"min":{"1min":0.443,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.872,"5min":0.89900000000000002,"15min":0.95699999999999996},"last":0.55900000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.65600000000000003,"15min":0.66900000000000004},"min":{"1min":0.501,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":0.68899999999999995,"5min":1.208,"15min":1.208},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.47099999999999997,"5min":0.45100000000000001,"15min":0.42499999999999999},"max":{"1min":0.77700000000000002,"5min":0.96599999999999997,"15min":1.018},"last":0.67600000000000005}]},{"osd":6,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.65900000000000003,"15min":0.67300000000000004},"min":{"1min":0.498,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.78400000000000003,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.73099999999999998,"5min":1.01,"15min":1.022},"last":0.70499999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.68400000000000005,"15min":0.70599999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.875,"5min":0.98199999999999998,"15min":1.0069999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.70399999999999996,"15min":0.71799999999999997},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.753,"5min":1.296,"15min":1.296},"last":0.54600000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575068,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307584,"kb_used_data":2104,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064256,"statfs":{"total":96636764160,"available":96321798144,"internally_reserved":0,"allocated":2154496,"data_stored":1388766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.41699999999999998,"15min":0.40600000000000003},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.50800000000000001,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.44,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.27400000000000002,"15min":0.27300000000000002},"max":{"1min":0.58899999999999997,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.34499999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40600000000000003,"15min":0.42399999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.67600000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45300000000000001,"15min":0.47199999999999998},"min":{"1min":0.34100000000000003,"5min":0.32500000000000001,"15min":0.28000000000000003},"max":{"1min":0.64300000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.46300000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47699999999999998,"15min":0.47199999999999998},"min":{"1min":0.35199999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.623,"5min":2.145,"15min":2.145},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48099999999999998,"15min":0.48799999999999999},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.63100000000000001,"5min":1.198,"15min":1.198},"last":0.54000000000000004}]},{"osd":4,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.621,"15min":0.64800000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.81399999999999995,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.59699999999999998,"15min":0.64300000000000002},"min":{"1min":0.38500000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.72099999999999997,"5min":0.88,"15min":0.98899999999999999},"last":0.77300000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.628,"15min":0.65400000000000003},"min":{"1min":0.45000000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78800000000000003,"5min":1.329,"15min":1.329},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.66400000000000003,"15min":0.68100000000000005},"min":{"1min":0.47199999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93100000000000005,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.627}]},{"osd":6,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.61099999999999999,"15min":0.66100000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80000000000000004,"5min":0.89800000000000002,"15min":1.032},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.65900000000000003,"15min":0.68999999999999995},"min":{"1min":0.48999999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82399999999999995,"5min":1.286,"15min":1.286},"last":0.78900000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":1.012,"15min":0.82699999999999996},"min":{"1min":0.498,"5min":0.435,"15min":0.435},"max":{"1min":0.76900000000000002,"5min":25.965,"15min":25.965},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":1.123,"15min":0.873},"min":{"1min":0.5,"5min":0.48799999999999999,"15min":0.433},"max":{"1min":0.89000000000000001,"5min":33.085000000000001,"15min":33.085000000000001},"last":0.83099999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575069,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307056,"kb_used_data":1576,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064784,"statfs":{"total":96636764160,"available":96322338816,"internally_reserved":0,"allocated":1613824,"data_stored":826399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.39100000000000001},"min":{"1min":0.27200000000000002,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.434}]},{"osd":1,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.42899999999999999,"15min":0.42799999999999999},"min":{"1min":0.27400000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.57999999999999996,"5min":0.61899999999999999,"15min":0.70499999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.50900000000000001,"15min":0.50600000000000001},"min":{"1min":0.379,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.51000000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46300000000000002,"15min":0.48599999999999999},"min":{"1min":0.32500000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.74099999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47899999999999998,"15min":0.47599999999999998},"min":{"1min":0.33500000000000002,"5min":0.32200000000000001,"15min":0.307},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.47799999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.60099999999999998,"15min":0.629},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.75800000000000001,"5min":1.087,"15min":1.087},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.628,"15min":0.64000000000000001},"min":{"1min":0.374,"5min":0.371,"15min":0.371},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.70599999999999996}]},{"osd":5,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63600000000000001,"15min":0.65400000000000003},"min":{"1min":0.44500000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.65000000000000002,"15min":0.67400000000000004},"min":{"1min":0.48199999999999998,"5min":0.39800000000000002,"15min":0.35199999999999998},"max":{"1min":0.80600000000000005,"5min":0.97699999999999998,"15min":1.085},"last":0.56000000000000005}]},{"osd":6,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65900000000000003,"15min":0.67800000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89600000000000002,"5min":1.022,"15min":1.1080000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.65700000000000003,"15min":0.67900000000000005},"min":{"1min":0.49099999999999999,"5min":0.42799999999999999,"15min":0.38800000000000001},"max":{"1min":0.88,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.58299999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68400000000000005,"15min":0.70199999999999996},"min":{"1min":0.53100000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.69799999999999995,"15min":0.72899999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.42099999999999999},"max":{"1min":0.84199999999999997,"5min":1.052,"15min":1.139},"last":0.61899999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575067,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307004,"kb_used_data":1524,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064836,"statfs":{"total":96636764160,"available":96322392064,"internally_reserved":0,"allocated":1560576,"data_stored":798398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.38200000000000001,"15min":0.39100000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":1.8540000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.441,"5min":0.42099999999999999,"15min":0.44},"min":{"1min":0.30599999999999999,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":1.2290000000000001},"last":0.33000000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44500000000000001,"15min":0.45400000000000001},"min":{"1min":0.33100000000000002,"5min":0.27900000000000003,"15min":0.253},"max":{"1min":0.63200000000000001,"5min":0.63800000000000001,"15min":1.633},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.48099999999999998,"15min":0.48499999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":0.65900000000000003,"5min":0.67800000000000005,"15min":1.0760000000000001},"last":0.315}]},{"osd":2,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.47399999999999998,"15min":0.47999999999999998},"min":{"1min":0.31900000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68600000000000005,"5min":0.70399999999999996,"15min":1.4059999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.439,"15min":0.45900000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28299999999999997},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":1.1870000000000001},"last":0.29499999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.61699999999999999,"15min":0.65800000000000003},"min":{"1min":0.45400000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.86799999999999999,"5min":0.878,"15min":3.306},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.628,"15min":0.65700000000000003},"min":{"1min":0.50700000000000001,"5min":0.40100000000000002,"15min":0.379},"max":{"1min":0.83899999999999997,"5min":0.85199999999999998,"15min":1.008},"last":0.73299999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.62,"15min":0.65300000000000002},"min":{"1min":0.49199999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":1.593},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63500000000000001,"15min":0.66600000000000004},"min":{"1min":0.499,"5min":0.379,"15min":0.379},"max":{"1min":0.79500000000000004,"5min":0.92300000000000004,"15min":1.3819999999999999},"last":0.57099999999999995}]},{"osd":6,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63400000000000001,"15min":0.66900000000000004},"min":{"1min":0.49099999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85599999999999998,"5min":0.88800000000000001,"15min":1.506},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.66000000000000003,"15min":0.68799999999999994},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.44400000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":1.427},"last":0.60099999999999998}]},{"osd":7,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.63700000000000001,"15min":0.68799999999999994},"min":{"1min":0.442,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":3.238},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.69699999999999995,"15min":0.73299999999999998},"min":{"1min":0.55000000000000004,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":3.351},"last":0.61199999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":302908,"kb_used_data":1524,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068932,"statfs":{"total":96636764160,"available":96326586368,"internally_reserved":0,"allocated":1560576,"data_stored":798398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53400000000000003,"15min":0.57099999999999995},"min":{"1min":0.34699999999999998,"5min":0.28999999999999998,"15min":0.28699999999999998},"max":{"1min":0.71199999999999997,"5min":1.21,"15min":1.21},"last":0.443},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.35999999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.76400000000000001,"5min":5.367,"15min":5.367},"last":0.48499999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55700000000000005,"15min":0.58899999999999997},"min":{"1min":0.38800000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.88},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.53400000000000003,"15min":0.57699999999999996},"min":{"1min":0.35199999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.621,"5min":0.80600000000000005,"15min":0.81699999999999995},"last":0.40699999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52700000000000002,"15min":0.56699999999999995},"min":{"1min":0.33900000000000002,"5min":0.33300000000000002,"15min":0.32700000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.501},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.57199999999999995,"15min":0.59699999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80500000000000005,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.43099999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55100000000000005,"15min":0.58399999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.68500000000000005,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56200000000000006,"15min":0.58299999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.35699999999999998},"max":{"1min":0.94199999999999995,"5min":1.05,"15min":1.05},"last":0.45500000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.44400000000000001,"15min":0.45800000000000002},"min":{"1min":0.29199999999999998,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.68300000000000005,"5min":0.76200000000000001,"15min":0.77600000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.441,"5min":0.47499999999999998,"15min":0.47699999999999998},"min":{"1min":0.29099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72499999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.51700000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45200000000000001,"15min":0.46700000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27200000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.78900000000000003,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.58699999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.51400000000000001,"15min":0.53400000000000003},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.63700000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.5,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.36899999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.55700000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":298812,"kb_used_data":1524,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073028,"statfs":{"total":96636764160,"available":96330780672,"internally_reserved":0,"allocated":1560576,"data_stored":798398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.55100000000000005,"15min":0.56699999999999995},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.65600000000000003,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.29999999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63600000000000001,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.35799999999999998}]},{"osd":1,"last update":"Fri Jan 26 23:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.58299999999999996,"15min":0.61199999999999999},"min":{"1min":0.32100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.61899999999999999,"5min":2.3319999999999999,"15min":2.3319999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.60099999999999998,"15min":0.629},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69999999999999996,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.47199999999999998}]},{"osd":2,"last update":"Fri Jan 26 23:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.53500000000000003,"15min":0.56499999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.30099999999999999},"max":{"1min":0.64800000000000002,"5min":0.77000000000000002,"15min":0.83799999999999997},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.56299999999999994,"15min":0.58799999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68899999999999995,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":0.52900000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.58199999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.314},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.439},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58799999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.72399999999999998,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.54600000000000004}]},{"osd":4,"last update":"Fri Jan 26 23:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.441,"15min":0.45200000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67200000000000004,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.623},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.46800000000000003,"15min":0.47899999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":0.65300000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.40300000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.46700000000000003,"15min":0.47199999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.59699999999999998,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.505,"15min":0.51100000000000001},"min":{"1min":0.33600000000000002,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":0.69499999999999995,"5min":1.99,"15min":1.99},"last":0.60499999999999998}]},{"osd":7,"last update":"Fri Jan 26 23:47:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.54600000000000004,"15min":0.53800000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63400000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":0.504},{"interface":"front","average":{"1min":0.503,"5min":0.53000000000000003,"15min":0.55700000000000005},"min":{"1min":0.38300000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.67700000000000005,"5min":0.81499999999999995,"15min":0.82199999999999995},"last":0.51300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:48:16.419 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:48:16.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.588+0000 7fda865ae700 1 Processor -- start 2024-01-26T23:48:16.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.588+0000 7fda865ae700 1 -- start start 2024-01-26T23:48:16.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.589+0000 7fda865ae700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda800cd3f0 0x7fda800cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:16.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.589+0000 7fda865ae700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda800cbb70 con 0x7fda801574a0 2024-01-26T23:48:16.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.589+0000 7fda865ae700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda800c9570 con 0x7fda80157870 2024-01-26T23:48:16.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.589+0000 7fda865ae700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda800c9850 con 0x7fda800cd3f0 2024-01-26T23:48:16.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.589+0000 7fda7ffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fda801574a0 0x7fda800c3300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40732/0 (socket says 172.21.15.46:40732) 2024-01-26T23:48:16.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.589+0000 7fda7ffff700 1 -- 172.21.15.46:0/205379041 learned_addr learned my addr 172.21.15.46:0/205379041 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:16.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.589+0000 7fda7effd700 1 -- 172.21.15.46:0/205379041 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636142215 0 0) 0x7fda800cbb70 con 0x7fda801574a0 2024-01-26T23:48:16.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.589+0000 7fda84b4b700 1 --2- 172.21.15.46:0/205379041 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda800cd3f0 0x7fda800cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.589+0000 7fda7effd700 1 -- 172.21.15.46:0/205379041 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda68003570 con 0x7fda801574a0 2024-01-26T23:48:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.589+0000 7fda7effd700 1 -- 172.21.15.46:0/205379041 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2933289837 0 0) 0x7fda68003570 con 0x7fda801574a0 2024-01-26T23:48:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.589+0000 7fda7effd700 1 -- 172.21.15.46:0/205379041 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda800cd3f0 msgr2=0x7fda800cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.589+0000 7fda7effd700 1 --2- 172.21.15.46:0/205379041 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda800cd3f0 0x7fda800cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.589+0000 7fda7effd700 1 -- 172.21.15.46:0/205379041 >> v1:172.21.15.106:6789/0 conn(0x7fda80157870 legacy=0x7fda800ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.589+0000 7fda7effd700 1 -- 172.21.15.46:0/205379041 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda800c9b30 con 0x7fda801574a0 2024-01-26T23:48:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.589+0000 7fda7effd700 1 -- 172.21.15.46:0/205379041 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fda700015f0 con 0x7fda801574a0 2024-01-26T23:48:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7effd700 1 -- 172.21.15.46:0/205379041 >> v1:172.21.15.46:6789/0 conn(0x7fda801574a0 legacy=0x7fda800c3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7effd700 1 --2- 172.21.15.46:0/205379041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda68003a90 0x7fda68003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7effd700 1 --2- 172.21.15.46:0/205379041 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda800cd3f0 0x7fda68004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7effd700 1 --2- 172.21.15.46:0/205379041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda68004d50 0x7fda680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7effd700 1 -- 172.21.15.46:0/205379041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda800c9850 con 0x7fda68003a90 2024-01-26T23:48:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7effd700 1 -- 172.21.15.46:0/205379041 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6800bb70 con 0x7fda68004d50 2024-01-26T23:48:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7effd700 1 -- 172.21.15.46:0/205379041 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6800bd30 con 0x7fda800cd3f0 2024-01-26T23:48:16.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7ffff700 1 --2- 172.21.15.46:0/205379041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda68003a90 0x7fda68003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:16.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7f7fe700 1 --2- 172.21.15.46:0/205379041 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda800cd3f0 0x7fda68004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:16.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda84b4b700 1 --2- 172.21.15.46:0/205379041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda68004d50 0x7fda680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:16.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7ffff700 1 -- 172.21.15.46:0/205379041 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda800cd3f0 msgr2=0x7fda68004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:16.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7ffff700 1 --2- 172.21.15.46:0/205379041 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda800cd3f0 0x7fda68004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7ffff700 1 -- 172.21.15.46:0/205379041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda68004d50 msgr2=0x7fda680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:16.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7ffff700 1 --2- 172.21.15.46:0/205379041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda68004d50 0x7fda680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7ffff700 1 -- 172.21.15.46:0/205379041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda68003570 con 0x7fda68003a90 2024-01-26T23:48:16.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7f7fe700 1 --2- 172.21.15.46:0/205379041 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda800cd3f0 0x7fda68004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:16.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda84b4b700 1 --2- 172.21.15.46:0/205379041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda68004d50 0x7fda680051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:48:16.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7ffff700 1 --2- 172.21.15.46:0/205379041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda68003a90 0x7fda68003ea0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fda7000c3c0 tx=0x7fda70007920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0284afb6a07d59e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:16.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7effd700 1 -- 172.21.15.46:0/205379041 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7000cc90 con 0x7fda68003a90 2024-01-26T23:48:16.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7effd700 1 -- 172.21.15.46:0/205379041 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7000ce10 con 0x7fda68003a90 2024-01-26T23:48:16.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.590+0000 7fda7effd700 1 -- 172.21.15.46:0/205379041 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda70007320 con 0x7fda68003a90 2024-01-26T23:48:16.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.591+0000 7fda865ae700 1 -- 172.21.15.46:0/205379041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda68003a90 msgr2=0x7fda68003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:16.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.591+0000 7fda865ae700 1 --2- 172.21.15.46:0/205379041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda68003a90 0x7fda68003ea0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fda7000c3c0 tx=0x7fda70007920 comp rx=0 tx=0).stop 2024-01-26T23:48:16.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.591+0000 7fda84b4b700 1 -- 172.21.15.46:0/205379041 reap_dead start 2024-01-26T23:48:16.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.591+0000 7fda865ae700 1 -- 172.21.15.46:0/205379041 shutdown_connections 2024-01-26T23:48:16.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.591+0000 7fda865ae700 1 -- 172.21.15.46:0/205379041 >> 172.21.15.46:0/205379041 conn(0x7fda80097820 msgr2=0x7fda80068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:16.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.591+0000 7fda865ae700 1 -- 172.21.15.46:0/205379041 shutdown_connections 2024-01-26T23:48:16.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.591+0000 7fda865ae700 1 -- 172.21.15.46:0/205379041 wait complete. 2024-01-26T23:48:16.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.592+0000 7fda865ae700 1 Processor -- start 2024-01-26T23:48:16.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.592+0000 7fda865ae700 1 -- start start 2024-01-26T23:48:16.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.593+0000 7fda865ae700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda68003a90 0x7fda80153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:16.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.593+0000 7fda865ae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda8016bb60 0x7fda8014aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:16.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.593+0000 7fda865ae700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda80155bc0 0x7fda80156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:16.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.593+0000 7fda865ae700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8016ae30 con 0x7fda8016bb60 2024-01-26T23:48:16.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.593+0000 7fda865ae700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8016b6d0 con 0x7fda68003a90 2024-01-26T23:48:16.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.593+0000 7fda865ae700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda80154540 con 0x7fda80155bc0 2024-01-26T23:48:16.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.593+0000 7fda7f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda8016bb60 0x7fda8014aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:16.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.593+0000 7fda7f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda8016bb60 0x7fda8014aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39032/0 (socket says 172.21.15.46:39032) 2024-01-26T23:48:16.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.593+0000 7fda7f7fe700 1 -- 172.21.15.46:0/1436420319 learned_addr learned my addr 172.21.15.46:0/1436420319 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:16.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.593+0000 7fda7ffff700 1 --2- 172.21.15.46:0/1436420319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda68003a90 0x7fda80153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:16.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.594+0000 7fda84b4b700 1 --2- 172.21.15.46:0/1436420319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda80155bc0 0x7fda80156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:16.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.594+0000 7fda7f7fe700 1 -- 172.21.15.46:0/1436420319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda80155bc0 msgr2=0x7fda80156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:16.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.594+0000 7fda7f7fe700 1 --2- 172.21.15.46:0/1436420319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda80155bc0 0x7fda80156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.594+0000 7fda7f7fe700 1 -- 172.21.15.46:0/1436420319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda68003a90 msgr2=0x7fda80153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:16.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.594+0000 7fda7f7fe700 1 --2- 172.21.15.46:0/1436420319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda68003a90 0x7fda80153bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.594+0000 7fda7f7fe700 1 -- 172.21.15.46:0/1436420319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8016b080 con 0x7fda8016bb60 2024-01-26T23:48:16.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.594+0000 7fda7ffff700 1 --2- 172.21.15.46:0/1436420319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda68003a90 0x7fda80153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:16.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.594+0000 7fda7f7fe700 1 --2- 172.21.15.46:0/1436420319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda8016bb60 0x7fda8014aea0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fda74000c00 tx=0x7fda74009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:16.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.594+0000 7fda67fff700 1 -- 172.21.15.46:0/1436420319 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74017070 con 0x7fda8016bb60 2024-01-26T23:48:16.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.594+0000 7fda67fff700 1 -- 172.21.15.46:0/1436420319 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda74006820 con 0x7fda8016bb60 2024-01-26T23:48:16.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.594+0000 7fda865ae700 1 -- 172.21.15.46:0/1436420319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8016b360 con 0x7fda8016bb60 2024-01-26T23:48:16.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.594+0000 7fda67fff700 1 -- 172.21.15.46:0/1436420319 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74006be0 con 0x7fda8016bb60 2024-01-26T23:48:16.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.594+0000 7fda865ae700 1 -- 172.21.15.46:0/1436420319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda80154110 con 0x7fda8016bb60 2024-01-26T23:48:16.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.595+0000 7fda865ae700 1 -- 172.21.15.46:0/1436420319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8016b840 con 0x7fda8016bb60 2024-01-26T23:48:16.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.596+0000 7fda67fff700 1 -- 172.21.15.46:0/1436420319 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fda74007020 con 0x7fda8016bb60 2024-01-26T23:48:16.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.596+0000 7fda67fff700 1 --2- 172.21.15.46:0/1436420319 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fda6c066f30 0x7fda6c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:16.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.596+0000 7fda67fff700 1 -- 172.21.15.46:0/1436420319 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fda74084370 con 0x7fda8016bb60 2024-01-26T23:48:16.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.596+0000 7fda7ffff700 1 --2- 172.21.15.46:0/1436420319 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fda6c066f30 0x7fda6c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:16.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.597+0000 7fda7ffff700 1 --2- 172.21.15.46:0/1436420319 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fda6c066f30 0x7fda6c0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fda70000ef0 tx=0x7fda70015000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:16.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.601+0000 7fda67fff700 1 -- 172.21.15.46:0/1436420319 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fda74054da0 con 0x7fda8016bb60 2024-01-26T23:48:16.779 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-01-26T23:48:16.777+0000 7f206e4d6700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-26T23:48:16.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.801+0000 7fda865ae700 1 -- 172.21.15.46:0/1436420319 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fda80154330 con 0x7fda6c066f30 2024-01-26T23:48:16.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.809+0000 7fda67fff700 1 -- 172.21.15.46:0/1436420319 <== mgr.7396 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289518 (secure 0 0 0) 0x7fda80154330 con 0x7fda6c066f30 2024-01-26T23:48:16.814 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:48:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.813+0000 7fda865ae700 1 -- 172.21.15.46:0/1436420319 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fda6c066f30 msgr2=0x7fda6c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.813+0000 7fda865ae700 1 --2- 172.21.15.46:0/1436420319 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fda6c066f30 0x7fda6c0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fda70000ef0 tx=0x7fda70015000 comp rx=0 tx=0).stop 2024-01-26T23:48:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.813+0000 7fda865ae700 1 -- 172.21.15.46:0/1436420319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda8016bb60 msgr2=0x7fda8014aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.813+0000 7fda865ae700 1 --2- 172.21.15.46:0/1436420319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda8016bb60 0x7fda8014aea0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fda74000c00 tx=0x7fda74009040 comp rx=0 tx=0).stop 2024-01-26T23:48:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.813+0000 7fda865ae700 1 -- 172.21.15.46:0/1436420319 shutdown_connections 2024-01-26T23:48:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.813+0000 7fda865ae700 1 --2- 172.21.15.46:0/1436420319 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fda6c066f30 0x7fda6c0693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.813+0000 7fda865ae700 1 --2- 172.21.15.46:0/1436420319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda80155bc0 0x7fda80156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.813+0000 7fda865ae700 1 --2- 172.21.15.46:0/1436420319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda68003a90 0x7fda80153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.813+0000 7fda865ae700 1 --2- 172.21.15.46:0/1436420319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda8016bb60 0x7fda8014aea0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:16.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.813+0000 7fda865ae700 1 -- 172.21.15.46:0/1436420319 >> 172.21.15.46:0/1436420319 conn(0x7fda80097820 msgr2=0x7fda800c9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:16.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.813+0000 7fda865ae700 1 -- 172.21.15.46:0/1436420319 shutdown_connections 2024-01-26T23:48:16.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:16.813+0000 7fda865ae700 1 -- 172.21.15.46:0/1436420319 wait complete. 2024-01-26T23:48:16.817 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:48:16.828 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-26T23:48:16.716700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":971,"num_read_kb":1902,"num_write":364,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8600,"ondisk_log_size":8600,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2445352,"kb_used_data":13800,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752529368,"statfs":{"total":773094113280,"available":770590072832,"internally_reserved":0,"allocated":14131200,"data_stored":7796268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001284"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693494+0000","last_change":"2024-01-26T23:48:11.690618+0000","last_active":"2024-01-26T23:48:11.693494+0000","last_peered":"2024-01-26T23:48:11.693494+0000","last_clean":"2024-01-26T23:48:11.693494+0000","last_became_active":"2024-01-26T23:48:11.690305+0000","last_became_peered":"2024-01-26T23:48:11.690305+0000","last_unstale":"2024-01-26T23:48:11.693494+0000","last_undegraded":"2024-01-26T23:48:11.693494+0000","last_fullsized":"2024-01-26T23:48:11.693494+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:17:24.039600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.700908+0000","last_change":"2024-01-26T23:48:11.695729+0000","last_active":"2024-01-26T23:48:11.700908+0000","last_peered":"2024-01-26T23:48:11.700908+0000","last_clean":"2024-01-26T23:48:11.700908+0000","last_became_active":"2024-01-26T23:48:11.694867+0000","last_became_peered":"2024-01-26T23:48:11.694867+0000","last_unstale":"2024-01-26T23:48:11.700908+0000","last_undegraded":"2024-01-26T23:48:11.700908+0000","last_fullsized":"2024-01-26T23:48:11.700908+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:22:27.593650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693576+0000","last_change":"2024-01-26T23:48:11.691344+0000","last_active":"2024-01-26T23:48:11.693576+0000","last_peered":"2024-01-26T23:48:11.693576+0000","last_clean":"2024-01-26T23:48:11.693576+0000","last_became_active":"2024-01-26T23:48:11.691069+0000","last_became_peered":"2024-01-26T23:48:11.691069+0000","last_unstale":"2024-01-26T23:48:11.693576+0000","last_undegraded":"2024-01-26T23:48:11.693576+0000","last_fullsized":"2024-01-26T23:48:11.693576+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:30.893321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698369+0000","last_change":"2024-01-26T23:48:11.694681+0000","last_active":"2024-01-26T23:48:11.698369+0000","last_peered":"2024-01-26T23:48:11.698369+0000","last_clean":"2024-01-26T23:48:11.698369+0000","last_became_active":"2024-01-26T23:48:11.693694+0000","last_became_peered":"2024-01-26T23:48:11.693694+0000","last_unstale":"2024-01-26T23:48:11.698369+0000","last_undegraded":"2024-01-26T23:48:11.698369+0000","last_fullsized":"2024-01-26T23:48:11.698369+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:10:51.561423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.689920+0000","last_change":"2024-01-26T23:48:11.688049+0000","last_active":"2024-01-26T23:48:11.689920+0000","last_peered":"2024-01-26T23:48:11.689920+0000","last_clean":"2024-01-26T23:48:11.689920+0000","last_became_active":"2024-01-26T23:48:11.687759+0000","last_became_peered":"2024-01-26T23:48:11.687759+0000","last_unstale":"2024-01-26T23:48:11.689920+0000","last_undegraded":"2024-01-26T23:48:11.689920+0000","last_fullsized":"2024-01-26T23:48:11.689920+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:08:43.039916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.700721+0000","last_change":"2024-01-26T23:48:11.694415+0000","last_active":"2024-01-26T23:48:11.700721+0000","last_peered":"2024-01-26T23:48:11.700721+0000","last_clean":"2024-01-26T23:48:11.700721+0000","last_became_active":"2024-01-26T23:48:11.694105+0000","last_became_peered":"2024-01-26T23:48:11.694105+0000","last_unstale":"2024-01-26T23:48:11.700721+0000","last_undegraded":"2024-01-26T23:48:11.700721+0000","last_fullsized":"2024-01-26T23:48:11.700721+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:24:22.377854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.707410+0000","last_change":"2024-01-26T23:48:11.702892+0000","last_active":"2024-01-26T23:48:11.707410+0000","last_peered":"2024-01-26T23:48:11.707410+0000","last_clean":"2024-01-26T23:48:11.707410+0000","last_became_active":"2024-01-26T23:48:11.702664+0000","last_became_peered":"2024-01-26T23:48:11.702664+0000","last_unstale":"2024-01-26T23:48:11.707410+0000","last_undegraded":"2024-01-26T23:48:11.707410+0000","last_fullsized":"2024-01-26T23:48:11.707410+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:25:47.188502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702212+0000","last_change":"2024-01-26T23:48:11.695520+0000","last_active":"2024-01-26T23:48:11.702212+0000","last_peered":"2024-01-26T23:48:11.702212+0000","last_clean":"2024-01-26T23:48:11.702212+0000","last_became_active":"2024-01-26T23:48:11.695205+0000","last_became_peered":"2024-01-26T23:48:11.695205+0000","last_unstale":"2024-01-26T23:48:11.702212+0000","last_undegraded":"2024-01-26T23:48:11.702212+0000","last_fullsized":"2024-01-26T23:48:11.702212+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:36:53.012187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.700678+0000","last_change":"2024-01-26T23:48:11.694658+0000","last_active":"2024-01-26T23:48:11.700678+0000","last_peered":"2024-01-26T23:48:11.700678+0000","last_clean":"2024-01-26T23:48:11.700678+0000","last_became_active":"2024-01-26T23:48:11.693934+0000","last_became_peered":"2024-01-26T23:48:11.693934+0000","last_unstale":"2024-01-26T23:48:11.700678+0000","last_undegraded":"2024-01-26T23:48:11.700678+0000","last_fullsized":"2024-01-26T23:48:11.700678+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:56:27.498369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.703088+0000","last_change":"2024-01-26T23:48:11.697158+0000","last_active":"2024-01-26T23:48:11.703088+0000","last_peered":"2024-01-26T23:48:11.703088+0000","last_clean":"2024-01-26T23:48:11.703088+0000","last_became_active":"2024-01-26T23:48:11.696633+0000","last_became_peered":"2024-01-26T23:48:11.696633+0000","last_unstale":"2024-01-26T23:48:11.703088+0000","last_undegraded":"2024-01-26T23:48:11.703088+0000","last_fullsized":"2024-01-26T23:48:11.703088+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:23:52.601066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702118+0000","last_change":"2024-01-26T23:48:11.697639+0000","last_active":"2024-01-26T23:48:11.702118+0000","last_peered":"2024-01-26T23:48:11.702118+0000","last_clean":"2024-01-26T23:48:11.702118+0000","last_became_active":"2024-01-26T23:48:11.697351+0000","last_became_peered":"2024-01-26T23:48:11.697351+0000","last_unstale":"2024-01-26T23:48:11.702118+0000","last_undegraded":"2024-01-26T23:48:11.702118+0000","last_fullsized":"2024-01-26T23:48:11.702118+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:45:48.834930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693709+0000","last_change":"2024-01-26T23:48:11.692041+0000","last_active":"2024-01-26T23:48:11.693709+0000","last_peered":"2024-01-26T23:48:11.693709+0000","last_clean":"2024-01-26T23:48:11.693709+0000","last_became_active":"2024-01-26T23:48:11.691748+0000","last_became_peered":"2024-01-26T23:48:11.691748+0000","last_unstale":"2024-01-26T23:48:11.693709+0000","last_undegraded":"2024-01-26T23:48:11.693709+0000","last_fullsized":"2024-01-26T23:48:11.693709+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:02:31.050232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.691654+0000","last_change":"2024-01-26T23:48:11.689915+0000","last_active":"2024-01-26T23:48:11.691654+0000","last_peered":"2024-01-26T23:48:11.691654+0000","last_clean":"2024-01-26T23:48:11.691654+0000","last_became_active":"2024-01-26T23:48:11.689529+0000","last_became_peered":"2024-01-26T23:48:11.689529+0000","last_unstale":"2024-01-26T23:48:11.691654+0000","last_undegraded":"2024-01-26T23:48:11.691654+0000","last_fullsized":"2024-01-26T23:48:11.691654+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:10.141932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.704794+0000","last_change":"2024-01-26T23:48:11.700345+0000","last_active":"2024-01-26T23:48:11.704794+0000","last_peered":"2024-01-26T23:48:11.704794+0000","last_clean":"2024-01-26T23:48:11.704794+0000","last_became_active":"2024-01-26T23:48:11.700057+0000","last_became_peered":"2024-01-26T23:48:11.700057+0000","last_unstale":"2024-01-26T23:48:11.704794+0000","last_undegraded":"2024-01-26T23:48:11.704794+0000","last_fullsized":"2024-01-26T23:48:11.704794+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:35:53.885389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.704742+0000","last_change":"2024-01-26T23:48:11.699805+0000","last_active":"2024-01-26T23:48:11.704742+0000","last_peered":"2024-01-26T23:48:11.704742+0000","last_clean":"2024-01-26T23:48:11.704742+0000","last_became_active":"2024-01-26T23:48:11.699526+0000","last_became_peered":"2024-01-26T23:48:11.699526+0000","last_unstale":"2024-01-26T23:48:11.704742+0000","last_undegraded":"2024-01-26T23:48:11.704742+0000","last_fullsized":"2024-01-26T23:48:11.704742+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:10.141932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.704822+0000","last_change":"2024-01-26T23:48:11.700542+0000","last_active":"2024-01-26T23:48:11.704822+0000","last_peered":"2024-01-26T23:48:11.704822+0000","last_clean":"2024-01-26T23:48:11.704822+0000","last_became_active":"2024-01-26T23:48:11.700277+0000","last_became_peered":"2024-01-26T23:48:11.700277+0000","last_unstale":"2024-01-26T23:48:11.704822+0000","last_undegraded":"2024-01-26T23:48:11.704822+0000","last_fullsized":"2024-01-26T23:48:11.704822+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:36:36.750288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693369+0000","last_change":"2024-01-26T23:48:11.686277+0000","last_active":"2024-01-26T23:48:11.693369+0000","last_peered":"2024-01-26T23:48:11.693369+0000","last_clean":"2024-01-26T23:48:11.693369+0000","last_became_active":"2024-01-26T23:48:11.686006+0000","last_became_peered":"2024-01-26T23:48:11.686006+0000","last_unstale":"2024-01-26T23:48:11.693369+0000","last_undegraded":"2024-01-26T23:48:11.693369+0000","last_fullsized":"2024-01-26T23:48:11.693369+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:22:19.028290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.700889+0000","last_change":"2024-01-26T23:48:11.694755+0000","last_active":"2024-01-26T23:48:11.700889+0000","last_peered":"2024-01-26T23:48:11.700889+0000","last_clean":"2024-01-26T23:48:11.700889+0000","last_became_active":"2024-01-26T23:48:11.694175+0000","last_became_peered":"2024-01-26T23:48:11.694175+0000","last_unstale":"2024-01-26T23:48:11.700889+0000","last_undegraded":"2024-01-26T23:48:11.700889+0000","last_fullsized":"2024-01-26T23:48:11.700889+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:55:06.345115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698855+0000","last_change":"2024-01-26T23:48:11.696828+0000","last_active":"2024-01-26T23:48:11.698855+0000","last_peered":"2024-01-26T23:48:11.698855+0000","last_clean":"2024-01-26T23:48:11.698855+0000","last_became_active":"2024-01-26T23:48:11.696291+0000","last_became_peered":"2024-01-26T23:48:11.696291+0000","last_unstale":"2024-01-26T23:48:11.698855+0000","last_undegraded":"2024-01-26T23:48:11.698855+0000","last_fullsized":"2024-01-26T23:48:11.698855+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:27:53.675503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.705011+0000","last_change":"2024-01-26T23:48:11.701106+0000","last_active":"2024-01-26T23:48:11.705011+0000","last_peered":"2024-01-26T23:48:11.705011+0000","last_clean":"2024-01-26T23:48:11.705011+0000","last_became_active":"2024-01-26T23:48:11.700818+0000","last_became_peered":"2024-01-26T23:48:11.700818+0000","last_unstale":"2024-01-26T23:48:11.705011+0000","last_undegraded":"2024-01-26T23:48:11.705011+0000","last_fullsized":"2024-01-26T23:48:11.705011+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:53:43.709494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.690830+0000","last_change":"2024-01-26T23:48:11.689380+0000","last_active":"2024-01-26T23:48:11.690830+0000","last_peered":"2024-01-26T23:48:11.690830+0000","last_clean":"2024-01-26T23:48:11.690830+0000","last_became_active":"2024-01-26T23:48:11.688997+0000","last_became_peered":"2024-01-26T23:48:11.688997+0000","last_unstale":"2024-01-26T23:48:11.690830+0000","last_undegraded":"2024-01-26T23:48:11.690830+0000","last_fullsized":"2024-01-26T23:48:11.690830+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:20.796536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.687203+0000","last_change":"2024-01-26T23:48:11.681763+0000","last_active":"2024-01-26T23:48:11.687203+0000","last_peered":"2024-01-26T23:48:11.687203+0000","last_clean":"2024-01-26T23:48:11.687203+0000","last_became_active":"2024-01-26T23:48:11.681378+0000","last_became_peered":"2024-01-26T23:48:11.681378+0000","last_unstale":"2024-01-26T23:48:11.687203+0000","last_undegraded":"2024-01-26T23:48:11.687203+0000","last_fullsized":"2024-01-26T23:48:11.687203+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:33:21.325282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693341+0000","last_change":"2024-01-26T23:48:11.685918+0000","last_active":"2024-01-26T23:48:11.693341+0000","last_peered":"2024-01-26T23:48:11.693341+0000","last_clean":"2024-01-26T23:48:11.693341+0000","last_became_active":"2024-01-26T23:48:11.685566+0000","last_became_peered":"2024-01-26T23:48:11.685566+0000","last_unstale":"2024-01-26T23:48:11.693341+0000","last_undegraded":"2024-01-26T23:48:11.693341+0000","last_fullsized":"2024-01-26T23:48:11.693341+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:06:45.718179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.684940+0000","last_change":"2024-01-26T23:48:11.682824+0000","last_active":"2024-01-26T23:48:11.684940+0000","last_peered":"2024-01-26T23:48:11.684940+0000","last_clean":"2024-01-26T23:48:11.684940+0000","last_became_active":"2024-01-26T23:48:11.682545+0000","last_became_peered":"2024-01-26T23:48:11.682545+0000","last_unstale":"2024-01-26T23:48:11.684940+0000","last_undegraded":"2024-01-26T23:48:11.684940+0000","last_fullsized":"2024-01-26T23:48:11.684940+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:50:50.668201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.701967+0000","last_change":"2024-01-26T23:48:11.696001+0000","last_active":"2024-01-26T23:48:11.701967+0000","last_peered":"2024-01-26T23:48:11.701967+0000","last_clean":"2024-01-26T23:48:11.701967+0000","last_became_active":"2024-01-26T23:48:11.695692+0000","last_became_peered":"2024-01-26T23:48:11.695692+0000","last_unstale":"2024-01-26T23:48:11.701967+0000","last_undegraded":"2024-01-26T23:48:11.701967+0000","last_fullsized":"2024-01-26T23:48:11.701967+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:25:59.436650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702281+0000","last_change":"2024-01-26T23:48:11.696416+0000","last_active":"2024-01-26T23:48:11.702281+0000","last_peered":"2024-01-26T23:48:11.702281+0000","last_clean":"2024-01-26T23:48:11.702281+0000","last_became_active":"2024-01-26T23:48:11.696137+0000","last_became_peered":"2024-01-26T23:48:11.696137+0000","last_unstale":"2024-01-26T23:48:11.702281+0000","last_undegraded":"2024-01-26T23:48:11.702281+0000","last_fullsized":"2024-01-26T23:48:11.702281+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:37:36.865287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.704679+0000","last_change":"2024-01-26T23:48:11.700455+0000","last_active":"2024-01-26T23:48:11.704679+0000","last_peered":"2024-01-26T23:48:11.704679+0000","last_clean":"2024-01-26T23:48:11.704679+0000","last_became_active":"2024-01-26T23:48:11.700192+0000","last_became_peered":"2024-01-26T23:48:11.700192+0000","last_unstale":"2024-01-26T23:48:11.704679+0000","last_undegraded":"2024-01-26T23:48:11.704679+0000","last_fullsized":"2024-01-26T23:48:11.704679+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:27.902905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702235+0000","last_change":"2024-01-26T23:48:11.698782+0000","last_active":"2024-01-26T23:48:11.702235+0000","last_peered":"2024-01-26T23:48:11.702235+0000","last_clean":"2024-01-26T23:48:11.702235+0000","last_became_active":"2024-01-26T23:48:11.698476+0000","last_became_peered":"2024-01-26T23:48:11.698476+0000","last_unstale":"2024-01-26T23:48:11.702235+0000","last_undegraded":"2024-01-26T23:48:11.702235+0000","last_fullsized":"2024-01-26T23:48:11.702235+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:27:13.343954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702333+0000","last_change":"2024-01-26T23:48:11.700735+0000","last_active":"2024-01-26T23:48:11.702333+0000","last_peered":"2024-01-26T23:48:11.702333+0000","last_clean":"2024-01-26T23:48:11.702333+0000","last_became_active":"2024-01-26T23:48:11.700416+0000","last_became_peered":"2024-01-26T23:48:11.700416+0000","last_unstale":"2024-01-26T23:48:11.702333+0000","last_undegraded":"2024-01-26T23:48:11.702333+0000","last_fullsized":"2024-01-26T23:48:11.702333+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:03:16.363445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699270+0000","last_change":"2024-01-26T23:48:11.693380+0000","last_active":"2024-01-26T23:48:11.699270+0000","last_peered":"2024-01-26T23:48:11.699270+0000","last_clean":"2024-01-26T23:48:11.699270+0000","last_became_active":"2024-01-26T23:48:11.692986+0000","last_became_peered":"2024-01-26T23:48:11.692986+0000","last_unstale":"2024-01-26T23:48:11.699270+0000","last_undegraded":"2024-01-26T23:48:11.699270+0000","last_fullsized":"2024-01-26T23:48:11.699270+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:37:57.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699950+0000","last_change":"2024-01-26T23:48:11.695328+0000","last_active":"2024-01-26T23:48:11.699950+0000","last_peered":"2024-01-26T23:48:11.699950+0000","last_clean":"2024-01-26T23:48:11.699950+0000","last_became_active":"2024-01-26T23:48:11.694749+0000","last_became_peered":"2024-01-26T23:48:11.694749+0000","last_unstale":"2024-01-26T23:48:11.699950+0000","last_undegraded":"2024-01-26T23:48:11.699950+0000","last_fullsized":"2024-01-26T23:48:11.699950+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:11:59.205203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699884+0000","last_change":"2024-01-26T23:48:11.694600+0000","last_active":"2024-01-26T23:48:11.699884+0000","last_peered":"2024-01-26T23:48:11.699884+0000","last_clean":"2024-01-26T23:48:11.699884+0000","last_became_active":"2024-01-26T23:48:11.694195+0000","last_became_peered":"2024-01-26T23:48:11.694195+0000","last_unstale":"2024-01-26T23:48:11.699884+0000","last_undegraded":"2024-01-26T23:48:11.699884+0000","last_fullsized":"2024-01-26T23:48:11.699884+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:54:52.459709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.697971+0000","last_change":"2024-01-26T23:48:11.695928+0000","last_active":"2024-01-26T23:48:11.697971+0000","last_peered":"2024-01-26T23:48:11.697971+0000","last_clean":"2024-01-26T23:48:11.697971+0000","last_became_active":"2024-01-26T23:48:11.695622+0000","last_became_peered":"2024-01-26T23:48:11.695622+0000","last_unstale":"2024-01-26T23:48:11.697971+0000","last_undegraded":"2024-01-26T23:48:11.697971+0000","last_fullsized":"2024-01-26T23:48:11.697971+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:37:48.746729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"129'4","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806763+0000","last_change":"2024-01-26T23:48:10.820996+0000","last_active":"2024-01-26T23:48:11.806763+0000","last_peered":"2024-01-26T23:48:11.806763+0000","last_clean":"2024-01-26T23:48:11.806763+0000","last_became_active":"2024-01-26T23:48:10.820659+0000","last_became_peered":"2024-01-26T23:48:10.820659+0000","last_unstale":"2024-01-26T23:48:11.806763+0000","last_undegraded":"2024-01-26T23:48:11.806763+0000","last_fullsized":"2024-01-26T23:48:11.806763+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:32:29.511141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702321+0000","last_change":"2024-01-26T23:48:11.699321+0000","last_active":"2024-01-26T23:48:11.702321+0000","last_peered":"2024-01-26T23:48:11.702321+0000","last_clean":"2024-01-26T23:48:11.702321+0000","last_became_active":"2024-01-26T23:48:11.699085+0000","last_became_peered":"2024-01-26T23:48:11.699085+0000","last_unstale":"2024-01-26T23:48:11.702321+0000","last_undegraded":"2024-01-26T23:48:11.702321+0000","last_fullsized":"2024-01-26T23:48:11.702321+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:49:57.681990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"129'1","reported_seq":11,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.805807+0000","last_change":"2024-01-26T23:48:10.816548+0000","last_active":"2024-01-26T23:48:11.805807+0000","last_peered":"2024-01-26T23:48:11.805807+0000","last_clean":"2024-01-26T23:48:11.805807+0000","last_became_active":"2024-01-26T23:48:10.816271+0000","last_became_peered":"2024-01-26T23:48:10.816271+0000","last_unstale":"2024-01-26T23:48:11.805807+0000","last_undegraded":"2024-01-26T23:48:11.805807+0000","last_fullsized":"2024-01-26T23:48:11.805807+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:10.976028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693908+0000","last_change":"2024-01-26T23:48:11.691251+0000","last_active":"2024-01-26T23:48:11.693908+0000","last_peered":"2024-01-26T23:48:11.693908+0000","last_clean":"2024-01-26T23:48:11.693908+0000","last_became_active":"2024-01-26T23:48:11.690973+0000","last_became_peered":"2024-01-26T23:48:11.690973+0000","last_unstale":"2024-01-26T23:48:11.693908+0000","last_undegraded":"2024-01-26T23:48:11.693908+0000","last_fullsized":"2024-01-26T23:48:11.693908+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:15:54.696377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.802933+0000","last_change":"2024-01-26T23:48:10.811743+0000","last_active":"2024-01-26T23:48:11.802933+0000","last_peered":"2024-01-26T23:48:11.802933+0000","last_clean":"2024-01-26T23:48:11.802933+0000","last_became_active":"2024-01-26T23:48:10.810670+0000","last_became_peered":"2024-01-26T23:48:10.810670+0000","last_unstale":"2024-01-26T23:48:11.802933+0000","last_undegraded":"2024-01-26T23:48:11.802933+0000","last_fullsized":"2024-01-26T23:48:11.802933+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:23:51.594366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699753+0000","last_change":"2024-01-26T23:48:11.696014+0000","last_active":"2024-01-26T23:48:11.699753+0000","last_peered":"2024-01-26T23:48:11.699753+0000","last_clean":"2024-01-26T23:48:11.699753+0000","last_became_active":"2024-01-26T23:48:11.695745+0000","last_became_peered":"2024-01-26T23:48:11.695745+0000","last_unstale":"2024-01-26T23:48:11.699753+0000","last_undegraded":"2024-01-26T23:48:11.699753+0000","last_fullsized":"2024-01-26T23:48:11.699753+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:26:08.221069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.803587+0000","last_change":"2024-01-26T23:48:10.812880+0000","last_active":"2024-01-26T23:48:11.803587+0000","last_peered":"2024-01-26T23:48:11.803587+0000","last_clean":"2024-01-26T23:48:11.803587+0000","last_became_active":"2024-01-26T23:48:10.812205+0000","last_became_peered":"2024-01-26T23:48:10.812205+0000","last_unstale":"2024-01-26T23:48:11.803587+0000","last_undegraded":"2024-01-26T23:48:11.803587+0000","last_fullsized":"2024-01-26T23:48:11.803587+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:26.896205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.691775+0000","last_change":"2024-01-26T23:48:11.687570+0000","last_active":"2024-01-26T23:48:11.691775+0000","last_peered":"2024-01-26T23:48:11.691775+0000","last_clean":"2024-01-26T23:48:11.691775+0000","last_became_active":"2024-01-26T23:48:11.686069+0000","last_became_peered":"2024-01-26T23:48:11.686069+0000","last_unstale":"2024-01-26T23:48:11.691775+0000","last_undegraded":"2024-01-26T23:48:11.691775+0000","last_fullsized":"2024-01-26T23:48:11.691775+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:36:36.750288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"129'1","reported_seq":11,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.805568+0000","last_change":"2024-01-26T23:48:10.815157+0000","last_active":"2024-01-26T23:48:11.805568+0000","last_peered":"2024-01-26T23:48:11.805568+0000","last_clean":"2024-01-26T23:48:11.805568+0000","last_became_active":"2024-01-26T23:48:10.814838+0000","last_became_peered":"2024-01-26T23:48:10.814838+0000","last_unstale":"2024-01-26T23:48:11.805568+0000","last_undegraded":"2024-01-26T23:48:11.805568+0000","last_fullsized":"2024-01-26T23:48:11.805568+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:49:44.669484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698877+0000","last_change":"2024-01-26T23:48:11.697027+0000","last_active":"2024-01-26T23:48:11.698877+0000","last_peered":"2024-01-26T23:48:11.698877+0000","last_clean":"2024-01-26T23:48:11.698877+0000","last_became_active":"2024-01-26T23:48:11.696702+0000","last_became_peered":"2024-01-26T23:48:11.696702+0000","last_unstale":"2024-01-26T23:48:11.698877+0000","last_undegraded":"2024-01-26T23:48:11.698877+0000","last_fullsized":"2024-01-26T23:48:11.698877+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:38:32.101523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"129'1","reported_seq":11,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806591+0000","last_change":"2024-01-26T23:48:10.813718+0000","last_active":"2024-01-26T23:48:11.806591+0000","last_peered":"2024-01-26T23:48:11.806591+0000","last_clean":"2024-01-26T23:48:11.806591+0000","last_became_active":"2024-01-26T23:48:10.813464+0000","last_became_peered":"2024-01-26T23:48:10.813464+0000","last_unstale":"2024-01-26T23:48:11.806591+0000","last_undegraded":"2024-01-26T23:48:11.806591+0000","last_fullsized":"2024-01-26T23:48:11.806591+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:13:40.216491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702230+0000","last_change":"2024-01-26T23:48:11.698344+0000","last_active":"2024-01-26T23:48:11.702230+0000","last_peered":"2024-01-26T23:48:11.702230+0000","last_clean":"2024-01-26T23:48:11.702230+0000","last_became_active":"2024-01-26T23:48:11.698034+0000","last_became_peered":"2024-01-26T23:48:11.698034+0000","last_unstale":"2024-01-26T23:48:11.702230+0000","last_undegraded":"2024-01-26T23:48:11.702230+0000","last_fullsized":"2024-01-26T23:48:11.702230+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:38:32.447049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685115+0000","last_change":"2024-01-26T23:48:10.811013+0000","last_active":"2024-01-26T23:48:11.685115+0000","last_peered":"2024-01-26T23:48:11.685115+0000","last_clean":"2024-01-26T23:48:11.685115+0000","last_became_active":"2024-01-26T23:48:10.810684+0000","last_became_peered":"2024-01-26T23:48:10.810684+0000","last_unstale":"2024-01-26T23:48:11.685115+0000","last_undegraded":"2024-01-26T23:48:11.685115+0000","last_fullsized":"2024-01-26T23:48:11.685115+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:29:34.900670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699671+0000","last_change":"2024-01-26T23:48:11.694858+0000","last_active":"2024-01-26T23:48:11.699671+0000","last_peered":"2024-01-26T23:48:11.699671+0000","last_clean":"2024-01-26T23:48:11.699671+0000","last_became_active":"2024-01-26T23:48:11.694407+0000","last_became_peered":"2024-01-26T23:48:11.694407+0000","last_unstale":"2024-01-26T23:48:11.699671+0000","last_undegraded":"2024-01-26T23:48:11.699671+0000","last_fullsized":"2024-01-26T23:48:11.699671+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:33:21.325282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.681769+0000","last_change":"2024-01-26T23:48:10.807380+0000","last_active":"2024-01-26T23:48:11.681769+0000","last_peered":"2024-01-26T23:48:11.681769+0000","last_clean":"2024-01-26T23:48:11.681769+0000","last_became_active":"2024-01-26T23:48:10.806968+0000","last_became_peered":"2024-01-26T23:48:10.806968+0000","last_unstale":"2024-01-26T23:48:11.681769+0000","last_undegraded":"2024-01-26T23:48:11.681769+0000","last_fullsized":"2024-01-26T23:48:11.681769+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:49:44.669484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.697301+0000","last_change":"2024-01-26T23:48:11.695119+0000","last_active":"2024-01-26T23:48:11.697301+0000","last_peered":"2024-01-26T23:48:11.697301+0000","last_clean":"2024-01-26T23:48:11.697301+0000","last_became_active":"2024-01-26T23:48:11.694615+0000","last_became_peered":"2024-01-26T23:48:11.694615+0000","last_unstale":"2024-01-26T23:48:11.697301+0000","last_undegraded":"2024-01-26T23:48:11.697301+0000","last_fullsized":"2024-01-26T23:48:11.697301+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:36:13.295894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685363+0000","last_change":"2024-01-26T23:48:10.811335+0000","last_active":"2024-01-26T23:48:11.685363+0000","last_peered":"2024-01-26T23:48:11.685363+0000","last_clean":"2024-01-26T23:48:11.685363+0000","last_became_active":"2024-01-26T23:48:10.810926+0000","last_became_peered":"2024-01-26T23:48:10.810926+0000","last_unstale":"2024-01-26T23:48:11.685363+0000","last_undegraded":"2024-01-26T23:48:11.685363+0000","last_fullsized":"2024-01-26T23:48:11.685363+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:22:26.586950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.703222+0000","last_change":"2024-01-26T23:48:11.697447+0000","last_active":"2024-01-26T23:48:11.703222+0000","last_peered":"2024-01-26T23:48:11.703222+0000","last_clean":"2024-01-26T23:48:11.703222+0000","last_became_active":"2024-01-26T23:48:11.697133+0000","last_became_peered":"2024-01-26T23:48:11.697133+0000","last_unstale":"2024-01-26T23:48:11.703222+0000","last_undegraded":"2024-01-26T23:48:11.703222+0000","last_fullsized":"2024-01-26T23:48:11.703222+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:28:53.642854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.677121+0000","last_change":"2024-01-26T23:48:10.810309+0000","last_active":"2024-01-26T23:48:11.677121+0000","last_peered":"2024-01-26T23:48:11.677121+0000","last_clean":"2024-01-26T23:48:11.677121+0000","last_became_active":"2024-01-26T23:48:10.810028+0000","last_became_peered":"2024-01-26T23:48:10.810028+0000","last_unstale":"2024-01-26T23:48:11.677121+0000","last_undegraded":"2024-01-26T23:48:11.677121+0000","last_fullsized":"2024-01-26T23:48:11.677121+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:36:07.346545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702237+0000","last_change":"2024-01-26T23:48:11.695812+0000","last_active":"2024-01-26T23:48:11.702237+0000","last_peered":"2024-01-26T23:48:11.702237+0000","last_clean":"2024-01-26T23:48:11.702237+0000","last_became_active":"2024-01-26T23:48:11.695535+0000","last_became_peered":"2024-01-26T23:48:11.695535+0000","last_unstale":"2024-01-26T23:48:11.702237+0000","last_undegraded":"2024-01-26T23:48:11.702237+0000","last_fullsized":"2024-01-26T23:48:11.702237+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:12:39.189557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"129'3","reported_seq":13,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806352+0000","last_change":"2024-01-26T23:48:10.813349+0000","last_active":"2024-01-26T23:48:11.806352+0000","last_peered":"2024-01-26T23:48:11.806352+0000","last_clean":"2024-01-26T23:48:11.806352+0000","last_became_active":"2024-01-26T23:48:10.813052+0000","last_became_peered":"2024-01-26T23:48:10.813052+0000","last_unstale":"2024-01-26T23:48:11.806352+0000","last_undegraded":"2024-01-26T23:48:11.806352+0000","last_fullsized":"2024-01-26T23:48:11.806352+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:50:15.445301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.703727+0000","last_change":"2024-01-26T23:48:11.703049+0000","last_active":"2024-01-26T23:48:11.703727+0000","last_peered":"2024-01-26T23:48:11.703727+0000","last_clean":"2024-01-26T23:48:11.703727+0000","last_became_active":"2024-01-26T23:48:11.702636+0000","last_became_peered":"2024-01-26T23:48:11.702636+0000","last_unstale":"2024-01-26T23:48:11.703727+0000","last_undegraded":"2024-01-26T23:48:11.703727+0000","last_fullsized":"2024-01-26T23:48:11.703727+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:50:12.326901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"129'6","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.807949+0000","last_change":"2024-01-26T23:48:10.814838+0000","last_active":"2024-01-26T23:48:11.807949+0000","last_peered":"2024-01-26T23:48:11.807949+0000","last_clean":"2024-01-26T23:48:11.807949+0000","last_became_active":"2024-01-26T23:48:10.814534+0000","last_became_peered":"2024-01-26T23:48:10.814534+0000","last_unstale":"2024-01-26T23:48:11.807949+0000","last_undegraded":"2024-01-26T23:48:11.807949+0000","last_fullsized":"2024-01-26T23:48:11.807949+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:51:57.293119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.688225+0000","last_change":"2024-01-26T23:48:11.683406+0000","last_active":"2024-01-26T23:48:11.688225+0000","last_peered":"2024-01-26T23:48:11.688225+0000","last_clean":"2024-01-26T23:48:11.688225+0000","last_became_active":"2024-01-26T23:48:11.683122+0000","last_became_peered":"2024-01-26T23:48:11.683122+0000","last_unstale":"2024-01-26T23:48:11.688225+0000","last_undegraded":"2024-01-26T23:48:11.688225+0000","last_fullsized":"2024-01-26T23:48:11.688225+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:59:48.214103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.804068+0000","last_change":"2024-01-26T23:48:10.812433+0000","last_active":"2024-01-26T23:48:11.804068+0000","last_peered":"2024-01-26T23:48:11.804068+0000","last_clean":"2024-01-26T23:48:11.804068+0000","last_became_active":"2024-01-26T23:48:10.812111+0000","last_became_peered":"2024-01-26T23:48:10.812111+0000","last_unstale":"2024-01-26T23:48:11.804068+0000","last_undegraded":"2024-01-26T23:48:11.804068+0000","last_fullsized":"2024-01-26T23:48:11.804068+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:49:44.669484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.697279+0000","last_change":"2024-01-26T23:48:11.694313+0000","last_active":"2024-01-26T23:48:11.697279+0000","last_peered":"2024-01-26T23:48:11.697279+0000","last_clean":"2024-01-26T23:48:11.697279+0000","last_became_active":"2024-01-26T23:48:11.693806+0000","last_became_peered":"2024-01-26T23:48:11.693806+0000","last_unstale":"2024-01-26T23:48:11.697279+0000","last_undegraded":"2024-01-26T23:48:11.697279+0000","last_fullsized":"2024-01-26T23:48:11.697279+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:07.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.682226+0000","last_change":"2024-01-26T23:48:10.819155+0000","last_active":"2024-01-26T23:48:11.682226+0000","last_peered":"2024-01-26T23:48:11.682226+0000","last_clean":"2024-01-26T23:48:11.682226+0000","last_became_active":"2024-01-26T23:48:10.818853+0000","last_became_peered":"2024-01-26T23:48:10.818853+0000","last_unstale":"2024-01-26T23:48:11.682226+0000","last_undegraded":"2024-01-26T23:48:11.682226+0000","last_fullsized":"2024-01-26T23:48:11.682226+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:50:15.445301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699954+0000","last_change":"2024-01-26T23:48:11.694802+0000","last_active":"2024-01-26T23:48:11.699954+0000","last_peered":"2024-01-26T23:48:11.699954+0000","last_clean":"2024-01-26T23:48:11.699954+0000","last_became_active":"2024-01-26T23:48:11.694277+0000","last_became_peered":"2024-01-26T23:48:11.694277+0000","last_unstale":"2024-01-26T23:48:11.699954+0000","last_undegraded":"2024-01-26T23:48:11.699954+0000","last_fullsized":"2024-01-26T23:48:11.699954+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:28:58.191038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.684638+0000","last_change":"2024-01-26T23:48:10.813933+0000","last_active":"2024-01-26T23:48:11.684638+0000","last_peered":"2024-01-26T23:48:11.684638+0000","last_clean":"2024-01-26T23:48:11.684638+0000","last_became_active":"2024-01-26T23:48:10.813646+0000","last_became_peered":"2024-01-26T23:48:10.813646+0000","last_unstale":"2024-01-26T23:48:11.684638+0000","last_undegraded":"2024-01-26T23:48:11.684638+0000","last_fullsized":"2024-01-26T23:48:11.684638+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:20:47.863263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693345+0000","last_change":"2024-01-26T23:48:11.683741+0000","last_active":"2024-01-26T23:48:11.693345+0000","last_peered":"2024-01-26T23:48:11.693345+0000","last_clean":"2024-01-26T23:48:11.693345+0000","last_became_active":"2024-01-26T23:48:11.683413+0000","last_became_peered":"2024-01-26T23:48:11.683413+0000","last_unstale":"2024-01-26T23:48:11.693345+0000","last_undegraded":"2024-01-26T23:48:11.693345+0000","last_fullsized":"2024-01-26T23:48:11.693345+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:53:44.050356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"129'8","reported_seq":18,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806654+0000","last_change":"2024-01-26T23:48:10.810870+0000","last_active":"2024-01-26T23:48:11.806654+0000","last_peered":"2024-01-26T23:48:11.806654+0000","last_clean":"2024-01-26T23:48:11.806654+0000","last_became_active":"2024-01-26T23:48:10.810516+0000","last_became_peered":"2024-01-26T23:48:10.810516+0000","last_unstale":"2024-01-26T23:48:11.806654+0000","last_undegraded":"2024-01-26T23:48:11.806654+0000","last_fullsized":"2024-01-26T23:48:11.806654+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:11:58.198504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699904+0000","last_change":"2024-01-26T23:48:11.694643+0000","last_active":"2024-01-26T23:48:11.699904+0000","last_peered":"2024-01-26T23:48:11.699904+0000","last_clean":"2024-01-26T23:48:11.699904+0000","last_became_active":"2024-01-26T23:48:11.694296+0000","last_became_peered":"2024-01-26T23:48:11.694296+0000","last_unstale":"2024-01-26T23:48:11.699904+0000","last_undegraded":"2024-01-26T23:48:11.699904+0000","last_fullsized":"2024-01-26T23:48:11.699904+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:53:21.191149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"129'1","reported_seq":11,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806724+0000","last_change":"2024-01-26T23:48:10.815595+0000","last_active":"2024-01-26T23:48:11.806724+0000","last_peered":"2024-01-26T23:48:11.806724+0000","last_clean":"2024-01-26T23:48:11.806724+0000","last_became_active":"2024-01-26T23:48:10.815314+0000","last_became_peered":"2024-01-26T23:48:10.815314+0000","last_unstale":"2024-01-26T23:48:11.806724+0000","last_undegraded":"2024-01-26T23:48:11.806724+0000","last_fullsized":"2024-01-26T23:48:11.806724+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:26.896205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.700740+0000","last_change":"2024-01-26T23:48:11.695066+0000","last_active":"2024-01-26T23:48:11.700740+0000","last_peered":"2024-01-26T23:48:11.700740+0000","last_clean":"2024-01-26T23:48:11.700740+0000","last_became_active":"2024-01-26T23:48:11.694666+0000","last_became_peered":"2024-01-26T23:48:11.694666+0000","last_unstale":"2024-01-26T23:48:11.700740+0000","last_undegraded":"2024-01-26T23:48:11.700740+0000","last_fullsized":"2024-01-26T23:48:11.700740+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:50:12.326901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"129'1","reported_seq":11,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806244+0000","last_change":"2024-01-26T23:48:10.814032+0000","last_active":"2024-01-26T23:48:11.806244+0000","last_peered":"2024-01-26T23:48:11.806244+0000","last_clean":"2024-01-26T23:48:11.806244+0000","last_became_active":"2024-01-26T23:48:10.813750+0000","last_became_peered":"2024-01-26T23:48:10.813750+0000","last_unstale":"2024-01-26T23:48:11.806244+0000","last_undegraded":"2024-01-26T23:48:11.806244+0000","last_fullsized":"2024-01-26T23:48:11.806244+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:45:19.808817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.703599+0000","last_change":"2024-01-26T23:48:11.701949+0000","last_active":"2024-01-26T23:48:11.703599+0000","last_peered":"2024-01-26T23:48:11.703599+0000","last_clean":"2024-01-26T23:48:11.703599+0000","last_became_active":"2024-01-26T23:48:11.701651+0000","last_became_peered":"2024-01-26T23:48:11.701651+0000","last_unstale":"2024-01-26T23:48:11.703599+0000","last_undegraded":"2024-01-26T23:48:11.703599+0000","last_fullsized":"2024-01-26T23:48:11.703599+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:10:51.561423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.681257+0000","last_change":"2024-01-26T23:48:10.815606+0000","last_active":"2024-01-26T23:48:11.681257+0000","last_peered":"2024-01-26T23:48:11.681257+0000","last_clean":"2024-01-26T23:48:11.681257+0000","last_became_active":"2024-01-26T23:48:10.815300+0000","last_became_peered":"2024-01-26T23:48:10.815300+0000","last_unstale":"2024-01-26T23:48:11.681257+0000","last_undegraded":"2024-01-26T23:48:11.681257+0000","last_fullsized":"2024-01-26T23:48:11.681257+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:46:44.771350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693545+0000","last_change":"2024-01-26T23:48:11.689724+0000","last_active":"2024-01-26T23:48:11.693545+0000","last_peered":"2024-01-26T23:48:11.693545+0000","last_clean":"2024-01-26T23:48:11.693545+0000","last_became_active":"2024-01-26T23:48:11.689438+0000","last_became_peered":"2024-01-26T23:48:11.689438+0000","last_unstale":"2024-01-26T23:48:11.693545+0000","last_undegraded":"2024-01-26T23:48:11.693545+0000","last_fullsized":"2024-01-26T23:48:11.693545+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:01.094001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.803125+0000","last_change":"2024-01-26T23:48:10.815064+0000","last_active":"2024-01-26T23:48:11.803125+0000","last_peered":"2024-01-26T23:48:11.803125+0000","last_clean":"2024-01-26T23:48:11.803125+0000","last_became_active":"2024-01-26T23:48:10.814778+0000","last_became_peered":"2024-01-26T23:48:10.814778+0000","last_unstale":"2024-01-26T23:48:11.803125+0000","last_undegraded":"2024-01-26T23:48:11.803125+0000","last_fullsized":"2024-01-26T23:48:11.803125+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:17.080283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.680810+0000","last_change":"2024-01-26T23:48:11.679952+0000","last_active":"2024-01-26T23:48:11.680810+0000","last_peered":"2024-01-26T23:48:11.680810+0000","last_clean":"2024-01-26T23:48:11.680810+0000","last_became_active":"2024-01-26T23:48:11.679509+0000","last_became_peered":"2024-01-26T23:48:11.679509+0000","last_unstale":"2024-01-26T23:48:11.680810+0000","last_undegraded":"2024-01-26T23:48:11.680810+0000","last_fullsized":"2024-01-26T23:48:11.680810+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:35:33.076540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.686302+0000","last_change":"2024-01-26T23:48:10.813851+0000","last_active":"2024-01-26T23:48:11.686302+0000","last_peered":"2024-01-26T23:48:11.686302+0000","last_clean":"2024-01-26T23:48:11.686302+0000","last_became_active":"2024-01-26T23:48:10.813510+0000","last_became_peered":"2024-01-26T23:48:10.813510+0000","last_unstale":"2024-01-26T23:48:11.686302+0000","last_undegraded":"2024-01-26T23:48:11.686302+0000","last_fullsized":"2024-01-26T23:48:11.686302+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:26:11.395397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698546+0000","last_change":"2024-01-26T23:48:11.695637+0000","last_active":"2024-01-26T23:48:11.698546+0000","last_peered":"2024-01-26T23:48:11.698546+0000","last_clean":"2024-01-26T23:48:11.698546+0000","last_became_active":"2024-01-26T23:48:11.695304+0000","last_became_peered":"2024-01-26T23:48:11.695304+0000","last_unstale":"2024-01-26T23:48:11.698546+0000","last_undegraded":"2024-01-26T23:48:11.698546+0000","last_fullsized":"2024-01-26T23:48:11.698546+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:50:50.668201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.690137+0000","last_change":"2024-01-26T23:48:10.815486+0000","last_active":"2024-01-26T23:48:11.690137+0000","last_peered":"2024-01-26T23:48:11.690137+0000","last_clean":"2024-01-26T23:48:11.690137+0000","last_became_active":"2024-01-26T23:48:10.815197+0000","last_became_peered":"2024-01-26T23:48:10.815197+0000","last_unstale":"2024-01-26T23:48:11.690137+0000","last_undegraded":"2024-01-26T23:48:11.690137+0000","last_fullsized":"2024-01-26T23:48:11.690137+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:57.119524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698608+0000","last_change":"2024-01-26T23:48:11.692941+0000","last_active":"2024-01-26T23:48:11.698608+0000","last_peered":"2024-01-26T23:48:11.698608+0000","last_clean":"2024-01-26T23:48:11.698608+0000","last_became_active":"2024-01-26T23:48:11.692605+0000","last_became_peered":"2024-01-26T23:48:11.692605+0000","last_unstale":"2024-01-26T23:48:11.698608+0000","last_undegraded":"2024-01-26T23:48:11.698608+0000","last_fullsized":"2024-01-26T23:48:11.698608+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:01:01.083122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685230+0000","last_change":"2024-01-26T23:48:10.816002+0000","last_active":"2024-01-26T23:48:11.685230+0000","last_peered":"2024-01-26T23:48:11.685230+0000","last_clean":"2024-01-26T23:48:11.685230+0000","last_became_active":"2024-01-26T23:48:10.815730+0000","last_became_peered":"2024-01-26T23:48:10.815730+0000","last_unstale":"2024-01-26T23:48:11.685230+0000","last_undegraded":"2024-01-26T23:48:11.685230+0000","last_fullsized":"2024-01-26T23:48:11.685230+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:59.034288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693414+0000","last_change":"2024-01-26T23:48:11.685991+0000","last_active":"2024-01-26T23:48:11.693414+0000","last_peered":"2024-01-26T23:48:11.693414+0000","last_clean":"2024-01-26T23:48:11.693414+0000","last_became_active":"2024-01-26T23:48:11.685598+0000","last_became_peered":"2024-01-26T23:48:11.685598+0000","last_unstale":"2024-01-26T23:48:11.693414+0000","last_undegraded":"2024-01-26T23:48:11.693414+0000","last_fullsized":"2024-01-26T23:48:11.693414+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:48:35.115441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.804084+0000","last_change":"2024-01-26T23:48:10.817696+0000","last_active":"2024-01-26T23:48:11.804084+0000","last_peered":"2024-01-26T23:48:11.804084+0000","last_clean":"2024-01-26T23:48:11.804084+0000","last_became_active":"2024-01-26T23:48:10.815112+0000","last_became_peered":"2024-01-26T23:48:10.815112+0000","last_unstale":"2024-01-26T23:48:11.804084+0000","last_undegraded":"2024-01-26T23:48:11.804084+0000","last_fullsized":"2024-01-26T23:48:11.804084+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:50:15.445301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693883+0000","last_change":"2024-01-26T23:48:11.691998+0000","last_active":"2024-01-26T23:48:11.693883+0000","last_peered":"2024-01-26T23:48:11.693883+0000","last_clean":"2024-01-26T23:48:11.693883+0000","last_became_active":"2024-01-26T23:48:11.691685+0000","last_became_peered":"2024-01-26T23:48:11.691685+0000","last_unstale":"2024-01-26T23:48:11.693883+0000","last_undegraded":"2024-01-26T23:48:11.693883+0000","last_fullsized":"2024-01-26T23:48:11.693883+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:35:42.319231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.677205+0000","last_change":"2024-01-26T23:48:10.806631+0000","last_active":"2024-01-26T23:48:11.677205+0000","last_peered":"2024-01-26T23:48:11.677205+0000","last_clean":"2024-01-26T23:48:11.677205+0000","last_became_active":"2024-01-26T23:48:10.806274+0000","last_became_peered":"2024-01-26T23:48:10.806274+0000","last_unstale":"2024-01-26T23:48:11.677205+0000","last_undegraded":"2024-01-26T23:48:11.677205+0000","last_fullsized":"2024-01-26T23:48:11.677205+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:59:28.765875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698753+0000","last_change":"2024-01-26T23:48:11.698307+0000","last_active":"2024-01-26T23:48:11.698753+0000","last_peered":"2024-01-26T23:48:11.698753+0000","last_clean":"2024-01-26T23:48:11.698753+0000","last_became_active":"2024-01-26T23:48:11.698019+0000","last_became_peered":"2024-01-26T23:48:11.698019+0000","last_unstale":"2024-01-26T23:48:11.698753+0000","last_undegraded":"2024-01-26T23:48:11.698753+0000","last_fullsized":"2024-01-26T23:48:11.698753+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:08:43.039916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.691955+0000","last_change":"2024-01-26T23:48:10.821818+0000","last_active":"2024-01-26T23:48:11.691955+0000","last_peered":"2024-01-26T23:48:11.691955+0000","last_clean":"2024-01-26T23:48:11.691955+0000","last_became_active":"2024-01-26T23:48:10.821530+0000","last_became_peered":"2024-01-26T23:48:10.821530+0000","last_unstale":"2024-01-26T23:48:11.691955+0000","last_undegraded":"2024-01-26T23:48:11.691955+0000","last_fullsized":"2024-01-26T23:48:11.691955+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:26:07.214369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.688038+0000","last_change":"2024-01-26T23:48:11.682394+0000","last_active":"2024-01-26T23:48:11.688038+0000","last_peered":"2024-01-26T23:48:11.688038+0000","last_clean":"2024-01-26T23:48:11.688038+0000","last_became_active":"2024-01-26T23:48:11.682136+0000","last_became_peered":"2024-01-26T23:48:11.682136+0000","last_unstale":"2024-01-26T23:48:11.688038+0000","last_undegraded":"2024-01-26T23:48:11.688038+0000","last_fullsized":"2024-01-26T23:48:11.688038+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:45:20.815517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.673973+0000","last_change":"2024-01-26T23:48:10.813068+0000","last_active":"2024-01-26T23:48:11.673973+0000","last_peered":"2024-01-26T23:48:11.673973+0000","last_clean":"2024-01-26T23:48:11.673973+0000","last_became_active":"2024-01-26T23:48:10.812610+0000","last_became_peered":"2024-01-26T23:48:10.812610+0000","last_unstale":"2024-01-26T23:48:11.673973+0000","last_undegraded":"2024-01-26T23:48:11.673973+0000","last_fullsized":"2024-01-26T23:48:11.673973+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:45:19.808817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698643+0000","last_change":"2024-01-26T23:48:11.692758+0000","last_active":"2024-01-26T23:48:11.698643+0000","last_peered":"2024-01-26T23:48:11.698643+0000","last_clean":"2024-01-26T23:48:11.698643+0000","last_became_active":"2024-01-26T23:48:11.692446+0000","last_became_peered":"2024-01-26T23:48:11.692446+0000","last_unstale":"2024-01-26T23:48:11.698643+0000","last_undegraded":"2024-01-26T23:48:11.698643+0000","last_fullsized":"2024-01-26T23:48:11.698643+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:54:10.694699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685024+0000","last_change":"2024-01-26T23:48:10.813057+0000","last_active":"2024-01-26T23:48:11.685024+0000","last_peered":"2024-01-26T23:48:11.685024+0000","last_clean":"2024-01-26T23:48:11.685024+0000","last_became_active":"2024-01-26T23:48:10.812778+0000","last_became_peered":"2024-01-26T23:48:10.812778+0000","last_unstale":"2024-01-26T23:48:11.685024+0000","last_undegraded":"2024-01-26T23:48:11.685024+0000","last_fullsized":"2024-01-26T23:48:11.685024+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:50:11.320201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702415+0000","last_change":"2024-01-26T23:48:11.696828+0000","last_active":"2024-01-26T23:48:11.702415+0000","last_peered":"2024-01-26T23:48:11.702415+0000","last_clean":"2024-01-26T23:48:11.702415+0000","last_became_active":"2024-01-26T23:48:11.695115+0000","last_became_peered":"2024-01-26T23:48:11.695115+0000","last_unstale":"2024-01-26T23:48:11.702415+0000","last_undegraded":"2024-01-26T23:48:11.702415+0000","last_fullsized":"2024-01-26T23:48:11.702415+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:25.597544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.687089+0000","last_change":"2024-01-26T23:48:10.815435+0000","last_active":"2024-01-26T23:48:11.687089+0000","last_peered":"2024-01-26T23:48:11.687089+0000","last_clean":"2024-01-26T23:48:11.687089+0000","last_became_active":"2024-01-26T23:48:10.815136+0000","last_became_peered":"2024-01-26T23:48:10.815136+0000","last_unstale":"2024-01-26T23:48:11.687089+0000","last_undegraded":"2024-01-26T23:48:11.687089+0000","last_fullsized":"2024-01-26T23:48:11.687089+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:36:28.278118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.704766+0000","last_change":"2024-01-26T23:48:11.699841+0000","last_active":"2024-01-26T23:48:11.704766+0000","last_peered":"2024-01-26T23:48:11.704766+0000","last_clean":"2024-01-26T23:48:11.704766+0000","last_became_active":"2024-01-26T23:48:11.699571+0000","last_became_peered":"2024-01-26T23:48:11.699571+0000","last_unstale":"2024-01-26T23:48:11.704766+0000","last_undegraded":"2024-01-26T23:48:11.704766+0000","last_fullsized":"2024-01-26T23:48:11.704766+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:33:21.325282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.690801+0000","last_change":"2024-01-26T23:48:10.821792+0000","last_active":"2024-01-26T23:48:11.690801+0000","last_peered":"2024-01-26T23:48:11.690801+0000","last_clean":"2024-01-26T23:48:11.690801+0000","last_became_active":"2024-01-26T23:48:10.821453+0000","last_became_peered":"2024-01-26T23:48:10.821453+0000","last_unstale":"2024-01-26T23:48:11.690801+0000","last_undegraded":"2024-01-26T23:48:11.690801+0000","last_fullsized":"2024-01-26T23:48:11.690801+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:59:47.207403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702140+0000","last_change":"2024-01-26T23:48:11.697093+0000","last_active":"2024-01-26T23:48:11.702140+0000","last_peered":"2024-01-26T23:48:11.702140+0000","last_clean":"2024-01-26T23:48:11.702140+0000","last_became_active":"2024-01-26T23:48:11.696468+0000","last_became_peered":"2024-01-26T23:48:11.696468+0000","last_unstale":"2024-01-26T23:48:11.702140+0000","last_undegraded":"2024-01-26T23:48:11.702140+0000","last_fullsized":"2024-01-26T23:48:11.702140+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:58.126224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685337+0000","last_change":"2024-01-26T23:48:10.817416+0000","last_active":"2024-01-26T23:48:11.685337+0000","last_peered":"2024-01-26T23:48:11.685337+0000","last_clean":"2024-01-26T23:48:11.685337+0000","last_became_active":"2024-01-26T23:48:10.817077+0000","last_became_peered":"2024-01-26T23:48:10.817077+0000","last_unstale":"2024-01-26T23:48:11.685337+0000","last_undegraded":"2024-01-26T23:48:11.685337+0000","last_fullsized":"2024-01-26T23:48:11.685337+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:05.190741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.695596+0000","last_change":"2024-01-26T23:48:11.692393+0000","last_active":"2024-01-26T23:48:11.695596+0000","last_peered":"2024-01-26T23:48:11.695596+0000","last_clean":"2024-01-26T23:48:11.695596+0000","last_became_active":"2024-01-26T23:48:11.690501+0000","last_became_peered":"2024-01-26T23:48:11.690501+0000","last_unstale":"2024-01-26T23:48:11.695596+0000","last_undegraded":"2024-01-26T23:48:11.695596+0000","last_fullsized":"2024-01-26T23:48:11.695596+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:51:58.299819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.804097+0000","last_change":"2024-01-26T23:48:10.814833+0000","last_active":"2024-01-26T23:48:11.804097+0000","last_peered":"2024-01-26T23:48:11.804097+0000","last_clean":"2024-01-26T23:48:11.804097+0000","last_became_active":"2024-01-26T23:48:10.814499+0000","last_became_peered":"2024-01-26T23:48:10.814499+0000","last_unstale":"2024-01-26T23:48:11.804097+0000","last_undegraded":"2024-01-26T23:48:11.804097+0000","last_fullsized":"2024-01-26T23:48:11.804097+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:27:02.987035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"120'8562","reported_seq":9017,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.687611+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:48:11.687611+0000","last_peered":"2024-01-26T23:48:11.687611+0000","last_clean":"2024-01-26T23:48:11.687611+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:48:11.687611+0000","last_undegraded":"2024-01-26T23:48:11.687611+0000","last_fullsized":"2024-01-26T23:48:11.687611+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8562,"log_dups_size":0,"ondisk_log_size":8562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":971,"num_read_kb":1902,"num_write":320,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":971,"num_read_kb":1902,"num_write":320,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8562,"ondisk_log_size":8562,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575070,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311736,"kb_used_data":2160,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060104,"statfs":{"total":96636764160,"available":96317546496,"internally_reserved":0,"allocated":2211840,"data_stored":1416893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.51000000000000001,"15min":0.56399999999999995},"min":{"1min":0.34100000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.625},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.52800000000000002,"15min":0.57899999999999996},"min":{"1min":0.379,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.83999999999999997},"last":0.52000000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.53500000000000003,"15min":0.57599999999999996},"min":{"1min":0.40999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.82899999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55800000000000005,"15min":0.59699999999999998},"min":{"1min":0.36199999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.84499999999999997,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.42599999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54600000000000004,"15min":0.57599999999999996},"min":{"1min":0.39600000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76800000000000002,"5min":0.80000000000000004,"15min":0.81699999999999995},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56000000000000005,"15min":0.59199999999999997},"min":{"1min":0.40799999999999997,"5min":0.30199999999999999,"15min":0.27400000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.88600000000000001},"last":0.69399999999999995}]},{"osd":3,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55300000000000005,"15min":0.58799999999999997},"min":{"1min":0.42399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82999999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.59799999999999998},"min":{"1min":0.442,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86399999999999999},"last":0.75600000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.42699999999999999,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.252,"15min":0.247},"max":{"1min":0.58399999999999996,"5min":0.66100000000000003,"15min":0.93899999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.44900000000000001,"15min":0.47399999999999998},"min":{"1min":0.316,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.66200000000000003,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.66200000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47199999999999998,"15min":0.48499999999999999},"min":{"1min":0.33500000000000002,"5min":0.27200000000000002,"15min":0.22800000000000001},"max":{"1min":0.78000000000000003,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.496,"15min":0.51900000000000002},"min":{"1min":0.377,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69499999999999995,"5min":0.74199999999999999,"15min":0.90200000000000002},"last":0.61299999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52500000000000002,"15min":0.54500000000000004},"min":{"1min":0.377,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68500000000000005,"5min":0.78700000000000003,"15min":0.95899999999999996},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52400000000000002,"15min":0.54000000000000004},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.85199999999999998},"last":0.70899999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575070,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302976,"kb_used_data":1592,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068864,"statfs":{"total":96636764160,"available":96326516736,"internally_reserved":0,"allocated":1630208,"data_stored":828477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.53300000000000003,"15min":0.56799999999999995},"min":{"1min":0.379,"5min":0.255,"15min":0.255},"max":{"1min":0.78700000000000003,"5min":1.423,"15min":1.423},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.53800000000000003,"15min":0.58399999999999996},"min":{"1min":0.45000000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.72999999999999998,"5min":0.79000000000000004,"15min":0.91100000000000003},"last":0.68200000000000005}]},{"osd":1,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.52400000000000002,"15min":0.56399999999999995},"min":{"1min":0.39800000000000002,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.76900000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55300000000000005,"15min":0.58999999999999997},"min":{"1min":0.432,"5min":0.312,"15min":0.312},"max":{"1min":0.749,"5min":1.498,"15min":1.498},"last":0.58399999999999996}]},{"osd":2,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53100000000000003,"15min":0.57399999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72699999999999998,"5min":0.74199999999999999,"15min":0.85799999999999998},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53500000000000003,"15min":0.57799999999999996},"min":{"1min":0.39400000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71799999999999997,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.56499999999999995}]},{"osd":3,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57199999999999995,"15min":0.59399999999999997},"min":{"1min":0.37,"5min":0.33800000000000002,"15min":0.29999999999999999},"max":{"1min":0.745,"5min":1.595,"15min":1.595},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.61199999999999999},"min":{"1min":0.39000000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.80300000000000005,"5min":1.369,"15min":1.369},"last":0.751}]},{"osd":4,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.45100000000000001,"15min":0.46000000000000002},"min":{"1min":0.32200000000000001,"5min":0.245,"15min":0.22},"max":{"1min":0.64400000000000002,"5min":1.556,"15min":1.556},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.47499999999999998,"15min":0.48299999999999998},"min":{"1min":0.32000000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65900000000000003,"5min":0.753,"15min":0.86199999999999999},"last":0.78000000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45900000000000002,"15min":0.47299999999999998},"min":{"1min":0.32600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.59699999999999998,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52000000000000002,"15min":0.51900000000000002},"min":{"1min":0.27200000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.72699999999999998,"5min":1.47,"15min":1.47},"last":0.69399999999999995}]},{"osd":7,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.505,"15min":0.51800000000000002},"min":{"1min":0.41299999999999998,"5min":0.26700000000000002,"15min":0.246},"max":{"1min":0.68700000000000006,"5min":0.82199999999999995,"15min":0.85899999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.374,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.67400000000000004,"5min":0.78500000000000003,"15min":0.80400000000000005},"last":0.73599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575070,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307056,"kb_used_data":1576,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064784,"statfs":{"total":96636764160,"available":96322338816,"internally_reserved":0,"allocated":1613824,"data_stored":826399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.41699999999999998,"15min":0.40699999999999997},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61799999999999999,"5min":0.65200000000000002,"15min":0.72599999999999998},"last":0.375},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.442,"15min":0.443},"min":{"1min":0.28999999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.63500000000000001,"5min":1.337,"15min":1.337},"last":0.432}]},{"osd":2,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40400000000000003,"15min":0.41099999999999998},"min":{"1min":0.28000000000000003,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.60199999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.71999999999999997,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.45000000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.51500000000000001},"min":{"1min":0.40000000000000002,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":0.77900000000000003,"5min":1.782,"15min":1.782},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48599999999999999,"15min":0.47999999999999998},"min":{"1min":0.38800000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.69899999999999995,"5min":1.379,"15min":1.379},"last":0.56699999999999995}]},{"osd":4,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.61199999999999999,"15min":0.64100000000000001},"min":{"1min":0.40799999999999997,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":0.73299999999999998,"5min":0.94099999999999995,"15min":1.0820000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63900000000000001,"15min":0.65700000000000003},"min":{"1min":0.443,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.872,"5min":0.89900000000000002,"15min":0.95699999999999996},"last":0.63100000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.65600000000000003,"15min":0.66900000000000004},"min":{"1min":0.501,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":0.68899999999999995,"5min":1.208,"15min":1.208},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.47099999999999997,"5min":0.45100000000000001,"15min":0.42499999999999999},"max":{"1min":0.77700000000000002,"5min":0.96599999999999997,"15min":1.018},"last":0.57799999999999996}]},{"osd":6,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.65900000000000003,"15min":0.67300000000000004},"min":{"1min":0.498,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.78400000000000003,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.73099999999999998,"5min":1.01,"15min":1.022},"last":0.55300000000000005}]},{"osd":7,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.68400000000000005,"15min":0.70599999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.875,"5min":0.98199999999999998,"15min":1.0069999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.70399999999999996,"15min":0.71799999999999997},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.753,"5min":1.296,"15min":1.296},"last":0.64800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575069,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307648,"kb_used_data":2168,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064192,"statfs":{"total":96636764160,"available":96321732608,"internally_reserved":0,"allocated":2220032,"data_stored":1418359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.41699999999999998,"15min":0.40600000000000003},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.50800000000000001,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.44,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.27400000000000002,"15min":0.27300000000000002},"max":{"1min":0.58899999999999997,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.53300000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40600000000000003,"15min":0.42399999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.67600000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45300000000000001,"15min":0.47199999999999998},"min":{"1min":0.34100000000000003,"5min":0.32500000000000001,"15min":0.28000000000000003},"max":{"1min":0.64300000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.39500000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47699999999999998,"15min":0.47199999999999998},"min":{"1min":0.35199999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.623,"5min":2.145,"15min":2.145},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48099999999999998,"15min":0.48799999999999999},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.63100000000000001,"5min":1.198,"15min":1.198},"last":0.55900000000000005}]},{"osd":4,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.621,"15min":0.64800000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.81399999999999995,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.59699999999999998,"15min":0.64300000000000002},"min":{"1min":0.38500000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.72099999999999997,"5min":0.88,"15min":0.98899999999999999},"last":0.78100000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.628,"15min":0.65400000000000003},"min":{"1min":0.45000000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78800000000000003,"5min":1.329,"15min":1.329},"last":0.623},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.66400000000000003,"15min":0.68100000000000005},"min":{"1min":0.47199999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93100000000000005,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.80400000000000005}]},{"osd":6,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.61099999999999999,"15min":0.66100000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80000000000000004,"5min":0.89800000000000002,"15min":1.032},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.65900000000000003,"15min":0.68999999999999995},"min":{"1min":0.48999999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82399999999999995,"5min":1.286,"15min":1.286},"last":0.747}]},{"osd":7,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":1.012,"15min":0.82699999999999996},"min":{"1min":0.498,"5min":0.435,"15min":0.435},"max":{"1min":0.76900000000000002,"5min":25.965,"15min":25.965},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":1.123,"15min":0.873},"min":{"1min":0.5,"5min":0.48799999999999999,"15min":0.433},"max":{"1min":0.89000000000000001,"5min":33.085000000000001,"15min":33.085000000000001},"last":0.69699999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575069,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307056,"kb_used_data":1576,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064784,"statfs":{"total":96636764160,"available":96322338816,"internally_reserved":0,"allocated":1613824,"data_stored":826399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.39100000000000001},"min":{"1min":0.27200000000000002,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.434}]},{"osd":1,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.42899999999999999,"15min":0.42799999999999999},"min":{"1min":0.27400000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.57999999999999996,"5min":0.61899999999999999,"15min":0.70499999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.50900000000000001,"15min":0.50600000000000001},"min":{"1min":0.379,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.51000000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46300000000000002,"15min":0.48599999999999999},"min":{"1min":0.32500000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.74099999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47899999999999998,"15min":0.47599999999999998},"min":{"1min":0.33500000000000002,"5min":0.32200000000000001,"15min":0.307},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.47799999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.60099999999999998,"15min":0.629},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.75800000000000001,"5min":1.087,"15min":1.087},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.628,"15min":0.64000000000000001},"min":{"1min":0.374,"5min":0.371,"15min":0.371},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.70599999999999996}]},{"osd":5,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63600000000000001,"15min":0.65400000000000003},"min":{"1min":0.44500000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.65000000000000002,"15min":0.67400000000000004},"min":{"1min":0.48199999999999998,"5min":0.39800000000000002,"15min":0.35199999999999998},"max":{"1min":0.80600000000000005,"5min":0.97699999999999998,"15min":1.085},"last":0.56000000000000005}]},{"osd":6,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65900000000000003,"15min":0.67800000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89600000000000002,"5min":1.022,"15min":1.1080000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.65700000000000003,"15min":0.67900000000000005},"min":{"1min":0.49099999999999999,"5min":0.42799999999999999,"15min":0.38800000000000001},"max":{"1min":0.88,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.58299999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68400000000000005,"15min":0.70199999999999996},"min":{"1min":0.53100000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.69799999999999995,"15min":0.72899999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.42099999999999999},"max":{"1min":0.84199999999999997,"5min":1.052,"15min":1.139},"last":0.61899999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575068,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307056,"kb_used_data":1576,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064784,"statfs":{"total":96636764160,"available":96322338816,"internally_reserved":0,"allocated":1613824,"data_stored":826919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.38200000000000001,"15min":0.39100000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":1.8540000000000001},"last":0.315},{"interface":"front","average":{"1min":0.441,"5min":0.42099999999999999,"15min":0.44},"min":{"1min":0.30599999999999999,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":1.2290000000000001},"last":0.29799999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44500000000000001,"15min":0.45400000000000001},"min":{"1min":0.33100000000000002,"5min":0.27900000000000003,"15min":0.253},"max":{"1min":0.63200000000000001,"5min":0.63800000000000001,"15min":1.633},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.48099999999999998,"15min":0.48499999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":0.65900000000000003,"5min":0.67800000000000005,"15min":1.0760000000000001},"last":0.36499999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.47399999999999998,"15min":0.47999999999999998},"min":{"1min":0.31900000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68600000000000005,"5min":0.70399999999999996,"15min":1.4059999999999999},"last":0.378},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.439,"15min":0.45900000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28299999999999997},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":1.1870000000000001},"last":0.35399999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.61699999999999999,"15min":0.65800000000000003},"min":{"1min":0.45400000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.86799999999999999,"5min":0.878,"15min":3.306},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.628,"15min":0.65700000000000003},"min":{"1min":0.50700000000000001,"5min":0.40100000000000002,"15min":0.379},"max":{"1min":0.83899999999999997,"5min":0.85199999999999998,"15min":1.008},"last":0.73599999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.62,"15min":0.65300000000000002},"min":{"1min":0.49199999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":1.593},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63500000000000001,"15min":0.66600000000000004},"min":{"1min":0.499,"5min":0.379,"15min":0.379},"max":{"1min":0.79500000000000004,"5min":0.92300000000000004,"15min":1.3819999999999999},"last":0.71099999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63400000000000001,"15min":0.66900000000000004},"min":{"1min":0.49099999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85599999999999998,"5min":0.88800000000000001,"15min":1.506},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.66000000000000003,"15min":0.68799999999999994},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.44400000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":1.427},"last":0.65300000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.63700000000000001,"15min":0.68799999999999994},"min":{"1min":0.442,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":3.238},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.69699999999999995,"15min":0.73299999999999998},"min":{"1min":0.55000000000000004,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":3.351},"last":0.54200000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575070,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302964,"kb_used_data":1580,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068876,"statfs":{"total":96636764160,"available":96326529024,"internally_reserved":0,"allocated":1617920,"data_stored":826457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53400000000000003,"15min":0.57099999999999995},"min":{"1min":0.34699999999999998,"5min":0.28999999999999998,"15min":0.28699999999999998},"max":{"1min":0.71199999999999997,"5min":1.21,"15min":1.21},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.35999999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.76400000000000001,"5min":5.367,"15min":5.367},"last":0.42799999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55700000000000005,"15min":0.58899999999999997},"min":{"1min":0.38800000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.88},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.53400000000000003,"15min":0.57699999999999996},"min":{"1min":0.35199999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.621,"5min":0.80600000000000005,"15min":0.81699999999999995},"last":0.52800000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52700000000000002,"15min":0.56699999999999995},"min":{"1min":0.33900000000000002,"5min":0.33300000000000002,"15min":0.32700000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.57199999999999995,"15min":0.59699999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80500000000000005,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.73199999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55100000000000005,"15min":0.58399999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.68500000000000005,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56200000000000006,"15min":0.58299999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.35699999999999998},"max":{"1min":0.94199999999999995,"5min":1.05,"15min":1.05},"last":0.499}]},{"osd":5,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.44400000000000001,"15min":0.45800000000000002},"min":{"1min":0.29199999999999998,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.68300000000000005,"5min":0.76200000000000001,"15min":0.77600000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.441,"5min":0.47499999999999998,"15min":0.47699999999999998},"min":{"1min":0.29099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72499999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.63900000000000001}]},{"osd":6,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45200000000000001,"15min":0.46700000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27200000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.78900000000000003,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.76900000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.51400000000000001,"15min":0.53400000000000003},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.63700000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.5,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.36899999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.58999999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575070,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298860,"kb_used_data":1572,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072980,"statfs":{"total":96636764160,"available":96330731520,"internally_reserved":0,"allocated":1609728,"data_stored":826365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.54500000000000004,"15min":0.56699999999999995},"min":{"1min":0.43099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.81799999999999995,"5min":0.92800000000000005,"15min":2.0289999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53000000000000003,"15min":0.56899999999999995},"min":{"1min":0.35799999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77100000000000002,"5min":0.77400000000000002,"15min":0.82199999999999995},"last":0.55700000000000005}]},{"osd":1,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57599999999999996,"15min":0.60799999999999998},"min":{"1min":0.33800000000000002,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":0.78000000000000003,"5min":0.85999999999999999,"15min":2.3319999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60599999999999998,"15min":0.627},"min":{"1min":0.47199999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80300000000000005,"5min":0.88500000000000001,"15min":1.1739999999999999},"last":0.59399999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.56100000000000005},"min":{"1min":0.438,"5min":0.34499999999999997,"15min":0.30099999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.83799999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55600000000000005,"15min":0.58699999999999997},"min":{"1min":0.38,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.80000000000000004,"5min":0.89700000000000002,"15min":2.0600000000000001},"last":0.498}]},{"osd":3,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56399999999999995,"15min":0.57699999999999996},"min":{"1min":0.40899999999999997,"5min":0.34999999999999998,"15min":0.314},"max":{"1min":0.68400000000000005,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56499999999999995,"15min":0.58699999999999997},"min":{"1min":0.39800000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.751,"5min":0.77900000000000003,"15min":2.5630000000000002},"last":0.57299999999999995}]},{"osd":4,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.42699999999999999,"15min":0.45300000000000001},"min":{"1min":0.30499999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64000000000000001,"5min":0.73099999999999998,"15min":2.3700000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.317,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.89200000000000002},"last":0.36799999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.47299999999999998},"min":{"1min":0.33100000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":2.5990000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.49299999999999999,"15min":0.51300000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70299999999999996,"5min":0.76200000000000001,"15min":1.99},"last":0.40699999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52400000000000002,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73499999999999999,"5min":0.76200000000000001,"15min":3.5270000000000001},"last":0.435},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54200000000000004,"15min":0.55800000000000005},"min":{"1min":0.33500000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.747,"5min":0.81499999999999995,"15min":0.82199999999999995},"last":0.33500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:48:16.833 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-26T23:48:17.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.001+0000 7f5744581700 1 Processor -- start 2024-01-26T23:48:17.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.001+0000 7f5744581700 1 -- start start 2024-01-26T23:48:17.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.002+0000 7f5744581700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f573c0b2160 0x7f573c15a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:17.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.002+0000 7f5744581700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f573c0b2530 0x7f573c0abe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:17.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.002+0000 7f5744581700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f573c15da10 0x7f573c15de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:17.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.002+0000 7f5744581700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573c0a9b90 con 0x7f573c0b2530 2024-01-26T23:48:17.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.002+0000 7f5744581700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573c0aa210 con 0x7f573c15da10 2024-01-26T23:48:17.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.002+0000 7f5744581700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c15d580 con 0x7f573c0b2160 2024-01-26T23:48:17.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.002+0000 7f5741b1c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f573c0b2530 0x7f573c0abe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:17.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.002+0000 7f5741b1c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f573c0b2530 0x7f573c0abe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39044/0 (socket says 172.21.15.46:39044) 2024-01-26T23:48:17.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.002+0000 7f5741b1c700 1 -- 172.21.15.46:0/1633108281 learned_addr learned my addr 172.21.15.46:0/1633108281 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:17.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.002+0000 7f574231d700 1 --2- 172.21.15.46:0/1633108281 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f573c0b2160 0x7f573c15a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:17.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.002+0000 7f5742b1e700 1 --2- 172.21.15.46:0/1633108281 >> v2:172.21.15.106:3300/0 conn(0x7f573c15da10 0x7f573c15de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:17.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.002+0000 7f5741b1c700 1 -- 172.21.15.46:0/1633108281 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f573c0b2160 msgr2=0x7f573c15a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:17.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.002+0000 7f5741b1c700 1 --2- 172.21.15.46:0/1633108281 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f573c0b2160 0x7f573c15a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.002+0000 7f5741b1c700 1 -- 172.21.15.46:0/1633108281 >> v2:172.21.15.106:3300/0 conn(0x7f573c15da10 msgr2=0x7f573c15de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:17.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.002+0000 7f5741b1c700 1 --2- 172.21.15.46:0/1633108281 >> v2:172.21.15.106:3300/0 conn(0x7f573c15da10 0x7f573c15de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.002+0000 7f5741b1c700 1 -- 172.21.15.46:0/1633108281 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c0ac390 con 0x7f573c0b2530 2024-01-26T23:48:17.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.003+0000 7f5741b1c700 1 --2- 172.21.15.46:0/1633108281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573c0b2530 0x7f573c0abe50 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f573c06e070 tx=0x7f572c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db806994a92935b7 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:17.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.003+0000 7f574131b700 1 -- 172.21.15.46:0/1633108281 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c00b640 con 0x7f573c0b2530 2024-01-26T23:48:17.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.003+0000 7f574131b700 1 -- 172.21.15.46:0/1633108281 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572c00b7a0 con 0x7f573c0b2530 2024-01-26T23:48:17.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.003+0000 7f574131b700 1 -- 172.21.15.46:0/1633108281 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c00bb40 con 0x7f573c0b2530 2024-01-26T23:48:17.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.003+0000 7f5744581700 1 -- 172.21.15.46:0/1633108281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573c0b2530 msgr2=0x7f573c0abe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:17.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.003+0000 7f5744581700 1 --2- 172.21.15.46:0/1633108281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573c0b2530 0x7f573c0abe50 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f573c06e070 tx=0x7f572c009250 comp rx=0 tx=0).stop 2024-01-26T23:48:17.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.003+0000 7f5744581700 1 -- 172.21.15.46:0/1633108281 shutdown_connections 2024-01-26T23:48:17.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.003+0000 7f5744581700 1 --2- 172.21.15.46:0/1633108281 >> v2:172.21.15.106:3300/0 conn(0x7f573c15da10 0x7f573c15de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.003+0000 7f5744581700 1 --2- 172.21.15.46:0/1633108281 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f573c0b2160 0x7f573c15a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.003+0000 7f5744581700 1 --2- 172.21.15.46:0/1633108281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573c0b2530 0x7f573c0abe50 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.003+0000 7f5744581700 1 -- 172.21.15.46:0/1633108281 >> 172.21.15.46:0/1633108281 conn(0x7f573c0ccbc0 msgr2=0x7f573c0ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:17.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.004+0000 7f5744581700 1 -- 172.21.15.46:0/1633108281 shutdown_connections 2024-01-26T23:48:17.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.004+0000 7f5744581700 1 -- 172.21.15.46:0/1633108281 wait complete. 2024-01-26T23:48:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.005+0000 7f5744581700 1 Processor -- start 2024-01-26T23:48:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.005+0000 7f5744581700 1 -- start start 2024-01-26T23:48:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.005+0000 7f5744581700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f573c0b2160 0x7f573c0ae390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.005+0000 7f5744581700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f573c0b2530 0x7f573c0ac5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.005+0000 7f5744581700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573c15da10 0x7f573c0acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.005+0000 7f5744581700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0adf00 con 0x7f573c15da10 2024-01-26T23:48:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.005+0000 7f5744581700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0ae8d0 con 0x7f573c0b2160 2024-01-26T23:48:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f5744581700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0ad040 con 0x7f573c0b2530 2024-01-26T23:48:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f5742b1e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573c15da10 0x7f573c0acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f5742b1e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573c15da10 0x7f573c0acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39054/0 (socket says 172.21.15.46:39054) 2024-01-26T23:48:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f5742b1e700 1 -- 172.21.15.46:0/667330397 learned_addr learned my addr 172.21.15.46:0/667330397 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f5741b1c700 1 --2- 172.21.15.46:0/667330397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f573c0b2530 0x7f573c0ac5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f574231d700 1 --2- 172.21.15.46:0/667330397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f573c0b2160 0x7f573c0ae390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f5742b1e700 1 -- 172.21.15.46:0/667330397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f573c0b2530 msgr2=0x7f573c0ac5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f5742b1e700 1 --2- 172.21.15.46:0/667330397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f573c0b2530 0x7f573c0ac5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f5742b1e700 1 -- 172.21.15.46:0/667330397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f573c0b2160 msgr2=0x7f573c0ae390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f5742b1e700 1 --2- 172.21.15.46:0/667330397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f573c0b2160 0x7f573c0ae390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f5742b1e700 1 -- 172.21.15.46:0/667330397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572c0094f0 con 0x7f573c15da10 2024-01-26T23:48:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f574231d700 1 --2- 172.21.15.46:0/667330397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f573c0b2160 0x7f573c0ae390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f5742b1e700 1 --2- 172.21.15.46:0/667330397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573c15da10 0x7f573c0acb00 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f5734009df0 tx=0x7f573400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f57327fc700 1 -- 172.21.15.46:0/667330397 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573400cb50 con 0x7f573c15da10 2024-01-26T23:48:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f57327fc700 1 -- 172.21.15.46:0/667330397 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573400f040 con 0x7f573c15da10 2024-01-26T23:48:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f5744581700 1 -- 172.21.15.46:0/667330397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573c0ad2f0 con 0x7f573c15da10 2024-01-26T23:48:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f57327fc700 1 -- 172.21.15.46:0/667330397 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5734014480 con 0x7f573c15da10 2024-01-26T23:48:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.006+0000 7f5744581700 1 -- 172.21.15.46:0/667330397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573c0ad930 con 0x7f573c15da10 2024-01-26T23:48:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.007+0000 7f5744581700 1 -- 172.21.15.46:0/667330397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573c0ae070 con 0x7f573c15da10 2024-01-26T23:48:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.008+0000 7f57327fc700 1 -- 172.21.15.46:0/667330397 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5734014830 con 0x7f573c15da10 2024-01-26T23:48:17.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.008+0000 7f57327fc700 1 --2- 172.21.15.46:0/667330397 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5720066f30 0x7f57200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:17.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.008+0000 7f57327fc700 1 -- 172.21.15.46:0/667330397 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f5734085110 con 0x7f573c15da10 2024-01-26T23:48:17.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.008+0000 7f574231d700 1 --2- 172.21.15.46:0/667330397 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5720066f30 0x7f57200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:17.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.009+0000 7f574231d700 1 --2- 172.21.15.46:0/667330397 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5720066f30 0x7f57200693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5738009d80 tx=0x7f5738008040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:17.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.013+0000 7f57327fc700 1 -- 172.21.15.46:0/667330397 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f57340520b0 con 0x7f573c15da10 2024-01-26T23:48:17.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.300+0000 7f5744581700 1 -- 172.21.15.46:0/667330397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f573c04c710 con 0x7f573c15da10 2024-01-26T23:48:17.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.301+0000 7f57327fc700 1 -- 172.21.15.46:0/667330397 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1089 (secure 0 0 0) 0x7f57340556d0 con 0x7f573c15da10 2024-01-26T23:48:17.302 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:48:17.302 INFO:teuthology.orchestra.run.smithi046.stdout:{"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1130,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":130,"num_osds":8,"num_up_osds":8,"osd_up_since":1706311777,"num_in_osds":8,"osd_in_since":1706311769,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":77},{"state_name":"active+clean","count":20}],"num_pgs":97,"num_pools":3,"num_objects":4,"data_bytes":590368,"bytes_used":2503634944,"bytes_avail":770590478336,"bytes_total":773094113280,"unknown_pgs_ratio":0.79381442070007324,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":62,"id":7,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":7,"rank":0,"name":"c","status":"up:active","gid":7794}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-26T23:47:52.647933+0000","services":{}},"progress_events":{}} 2024-01-26T23:48:17.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.303+0000 7f5744581700 1 -- 172.21.15.46:0/667330397 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5720066f30 msgr2=0x7f57200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:17.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.303+0000 7f5744581700 1 --2- 172.21.15.46:0/667330397 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5720066f30 0x7f57200693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5738009d80 tx=0x7f5738008040 comp rx=0 tx=0).stop 2024-01-26T23:48:17.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.303+0000 7f5744581700 1 -- 172.21.15.46:0/667330397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573c15da10 msgr2=0x7f573c0acb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:17.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.303+0000 7f5744581700 1 --2- 172.21.15.46:0/667330397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573c15da10 0x7f573c0acb00 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f5734009df0 tx=0x7f573400b040 comp rx=0 tx=0).stop 2024-01-26T23:48:17.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.303+0000 7f5744581700 1 -- 172.21.15.46:0/667330397 shutdown_connections 2024-01-26T23:48:17.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.303+0000 7f5744581700 1 --2- 172.21.15.46:0/667330397 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5720066f30 0x7f57200693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.303+0000 7f5744581700 1 --2- 172.21.15.46:0/667330397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573c15da10 0x7f573c0acb00 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.303+0000 7f5744581700 1 --2- 172.21.15.46:0/667330397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f573c0b2160 0x7f573c0ae390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.303+0000 7f5744581700 1 --2- 172.21.15.46:0/667330397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f573c0b2530 0x7f573c0ac5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.303+0000 7f5744581700 1 -- 172.21.15.46:0/667330397 >> 172.21.15.46:0/667330397 conn(0x7f573c0ccbc0 msgr2=0x7f573c15aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:17.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.303+0000 7f5744581700 1 -- 172.21.15.46:0/667330397 shutdown_connections 2024-01-26T23:48:17.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.303+0000 7f5744581700 1 -- 172.21.15.46:0/667330397 wait complete. 2024-01-26T23:48:17.318 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-26T23:48:17.319 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:48:17.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.449+0000 7f9d84b49700 1 Processor -- start 2024-01-26T23:48:17.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.449+0000 7f9d84b49700 1 -- start start 2024-01-26T23:48:17.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.450+0000 7f9d84b49700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d80155aa0 0x7f9d80155e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:17.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.451+0000 7f9d84b49700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8006a450 con 0x7f9d800cb960 2024-01-26T23:48:17.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.451+0000 7f9d84b49700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8006a730 con 0x7f9d80068ef0 2024-01-26T23:48:17.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.451+0000 7f9d84b49700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8006aa10 con 0x7f9d80155aa0 2024-01-26T23:48:17.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.451+0000 7f9d7e59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d80155aa0 0x7f9d80155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:17.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.451+0000 7f9d7ed9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9d800cb960 0x7f9d800c79c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40748/0 (socket says 172.21.15.46:40748) 2024-01-26T23:48:17.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.451+0000 7f9d7e59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d80155aa0 0x7f9d80155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39066/0 (socket says 172.21.15.46:39066) 2024-01-26T23:48:17.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.451+0000 7f9d7ed9d700 1 -- 172.21.15.46:0/3243980161 learned_addr learned my addr 172.21.15.46:0/3243980161 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:17.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.451+0000 7f9d7e59c700 1 -- 172.21.15.46:0/3243980161 >> v1:172.21.15.106:6789/0 conn(0x7f9d80068ef0 legacy=0x7f9d800cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:17.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.451+0000 7f9d7e59c700 1 -- 172.21.15.46:0/3243980161 >> v1:172.21.15.46:6789/0 conn(0x7f9d800cb960 legacy=0x7f9d800c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:17.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.451+0000 7f9d7d59a700 1 -- 172.21.15.46:0/3243980161 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248689849 0 0) 0x7f9d8006a450 con 0x7f9d800cb960 2024-01-26T23:48:17.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.451+0000 7f9d7e59c700 1 -- 172.21.15.46:0/3243980161 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8006acf0 con 0x7f9d80155aa0 2024-01-26T23:48:17.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.451+0000 7f9d7e59c700 1 --2- 172.21.15.46:0/3243980161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d80155aa0 0x7f9d80155e70 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f9d700098e0 tx=0x7f9d700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f81b3b2aa729ec10 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:17.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.451+0000 7f9d7d59a700 1 -- 172.21.15.46:0/3243980161 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d70009e80 con 0x7f9d80155aa0 2024-01-26T23:48:17.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.451+0000 7f9d7d59a700 1 -- 172.21.15.46:0/3243980161 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7000b860 con 0x7f9d80155aa0 2024-01-26T23:48:17.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.452+0000 7f9d7d59a700 1 -- 172.21.15.46:0/3243980161 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7000bbe0 con 0x7f9d80155aa0 2024-01-26T23:48:17.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.452+0000 7f9d84b49700 1 -- 172.21.15.46:0/3243980161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d80155aa0 msgr2=0x7f9d80155e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:17.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.452+0000 7f9d84b49700 1 --2- 172.21.15.46:0/3243980161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d80155aa0 0x7f9d80155e70 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f9d700098e0 tx=0x7f9d700091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.452+0000 7f9d84b49700 1 -- 172.21.15.46:0/3243980161 shutdown_connections 2024-01-26T23:48:17.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.452+0000 7f9d84b49700 1 --2- 172.21.15.46:0/3243980161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d80155aa0 0x7f9d80155e70 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.452+0000 7f9d84b49700 1 -- 172.21.15.46:0/3243980161 >> 172.21.15.46:0/3243980161 conn(0x7f9d800cd680 msgr2=0x7f9d800c9650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:17.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.452+0000 7f9d84b49700 1 -- 172.21.15.46:0/3243980161 shutdown_connections 2024-01-26T23:48:17.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.452+0000 7f9d84b49700 1 -- 172.21.15.46:0/3243980161 wait complete. 2024-01-26T23:48:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.453+0000 7f9d84b49700 1 Processor -- start 2024-01-26T23:48:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.453+0000 7f9d84b49700 1 -- start start 2024-01-26T23:48:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.454+0000 7f9d84b49700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d80068ef0 0x7f9d801436e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.454+0000 7f9d84b49700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d800cb960 0x7f9d80168b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.454+0000 7f9d84b49700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d80155aa0 0x7f9d800c5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.454+0000 7f9d84b49700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d80143c20 con 0x7f9d80068ef0 2024-01-26T23:48:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.454+0000 7f9d84b49700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d800c4c00 con 0x7f9d800cb960 2024-01-26T23:48:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.454+0000 7f9d84b49700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d800c3ff0 con 0x7f9d80155aa0 2024-01-26T23:48:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.454+0000 7f9d7e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d80068ef0 0x7f9d801436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.454+0000 7f9d7e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d80068ef0 0x7f9d801436e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39080/0 (socket says 172.21.15.46:39080) 2024-01-26T23:48:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.454+0000 7f9d7e59c700 1 -- 172.21.15.46:0/1555378330 learned_addr learned my addr 172.21.15.46:0/1555378330 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.454+0000 7f9d7dd9b700 1 --2- 172.21.15.46:0/1555378330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d800cb960 0x7f9d80168b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.455+0000 7f9d7ed9d700 1 --2- 172.21.15.46:0/1555378330 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d80155aa0 0x7f9d800c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.455+0000 7f9d7e59c700 1 -- 172.21.15.46:0/1555378330 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d80155aa0 msgr2=0x7f9d800c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.455+0000 7f9d7e59c700 1 --2- 172.21.15.46:0/1555378330 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d80155aa0 0x7f9d800c5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.455+0000 7f9d7e59c700 1 -- 172.21.15.46:0/1555378330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d800cb960 msgr2=0x7f9d80168b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.455+0000 7f9d7e59c700 1 --2- 172.21.15.46:0/1555378330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d800cb960 0x7f9d80168b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.455+0000 7f9d7e59c700 1 -- 172.21.15.46:0/1555378330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d70009420 con 0x7f9d80068ef0 2024-01-26T23:48:17.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.455+0000 7f9d7dd9b700 1 --2- 172.21.15.46:0/1555378330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d800cb960 0x7f9d80168b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:17.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.455+0000 7f9d7e59c700 1 --2- 172.21.15.46:0/1555378330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d80068ef0 0x7f9d801436e0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f9d7000b3c0 tx=0x7f9d70017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:17.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.455+0000 7f9d6e7fc700 1 -- 172.21.15.46:0/1555378330 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7001f070 con 0x7f9d80068ef0 2024-01-26T23:48:17.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.455+0000 7f9d6e7fc700 1 -- 172.21.15.46:0/1555378330 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7000b9d0 con 0x7f9d80068ef0 2024-01-26T23:48:17.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.455+0000 7f9d84b49700 1 -- 172.21.15.46:0/1555378330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d800c42d0 con 0x7f9d80068ef0 2024-01-26T23:48:17.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.455+0000 7f9d6e7fc700 1 -- 172.21.15.46:0/1555378330 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d70025650 con 0x7f9d80068ef0 2024-01-26T23:48:17.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.455+0000 7f9d84b49700 1 -- 172.21.15.46:0/1555378330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d800c4890 con 0x7f9d80068ef0 2024-01-26T23:48:17.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.456+0000 7f9d84b49700 1 -- 172.21.15.46:0/1555378330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8004c710 con 0x7f9d80068ef0 2024-01-26T23:48:17.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.457+0000 7f9d6e7fc700 1 -- 172.21.15.46:0/1555378330 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9d70023070 con 0x7f9d80068ef0 2024-01-26T23:48:17.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.457+0000 7f9d6e7fc700 1 --2- 172.21.15.46:0/1555378330 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9d60066f30 0x7f9d600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:17.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.457+0000 7f9d6e7fc700 1 -- 172.21.15.46:0/1555378330 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f9d70085cf0 con 0x7f9d80068ef0 2024-01-26T23:48:17.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.457+0000 7f9d7dd9b700 1 --2- 172.21.15.46:0/1555378330 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9d60066f30 0x7f9d600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.458+0000 7f9d7dd9b700 1 --2- 172.21.15.46:0/1555378330 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9d60066f30 0x7f9d600693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9d74007fd0 tx=0x7f9d74009040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:17.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.461+0000 7f9d6e7fc700 1 -- 172.21.15.46:0/1555378330 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9d7008a070 con 0x7f9d80068ef0 2024-01-26T23:48:17.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.662+0000 7f9d84b49700 1 -- 172.21.15.46:0/1555378330 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9d8006afd0 con 0x7f9d60066f30 2024-01-26T23:48:17.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.670+0000 7f9d6e7fc700 1 -- 172.21.15.46:0/1555378330 <== mgr.7396 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289518 (secure 0 0 0) 0x7f9d8006afd0 con 0x7f9d60066f30 2024-01-26T23:48:17.675 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:48:17.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.674+0000 7f9d84b49700 1 -- 172.21.15.46:0/1555378330 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9d60066f30 msgr2=0x7f9d600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:17.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.674+0000 7f9d84b49700 1 --2- 172.21.15.46:0/1555378330 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9d60066f30 0x7f9d600693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9d74007fd0 tx=0x7f9d74009040 comp rx=0 tx=0).stop 2024-01-26T23:48:17.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.674+0000 7f9d84b49700 1 -- 172.21.15.46:0/1555378330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d80068ef0 msgr2=0x7f9d801436e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.674+0000 7f9d84b49700 1 --2- 172.21.15.46:0/1555378330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d80068ef0 0x7f9d801436e0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f9d7000b3c0 tx=0x7f9d70017000 comp rx=0 tx=0).stop 2024-01-26T23:48:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.674+0000 7f9d84b49700 1 -- 172.21.15.46:0/1555378330 shutdown_connections 2024-01-26T23:48:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.674+0000 7f9d84b49700 1 --2- 172.21.15.46:0/1555378330 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9d60066f30 0x7f9d600693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.674+0000 7f9d84b49700 1 --2- 172.21.15.46:0/1555378330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d80068ef0 0x7f9d801436e0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.674+0000 7f9d84b49700 1 --2- 172.21.15.46:0/1555378330 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d80155aa0 0x7f9d800c5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.674+0000 7f9d84b49700 1 --2- 172.21.15.46:0/1555378330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d800cb960 0x7f9d80168b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.674+0000 7f9d84b49700 1 -- 172.21.15.46:0/1555378330 >> 172.21.15.46:0/1555378330 conn(0x7f9d800cd680 msgr2=0x7f9d800b1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.674+0000 7f9d84b49700 1 -- 172.21.15.46:0/1555378330 shutdown_connections 2024-01-26T23:48:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:17.674+0000 7f9d84b49700 1 -- 172.21.15.46:0/1555378330 wait complete. 2024-01-26T23:48:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:48:17.689 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-26T23:48:16.716700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":971,"num_read_kb":1902,"num_write":364,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8600,"ondisk_log_size":8600,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2445352,"kb_used_data":13800,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752529368,"statfs":{"total":773094113280,"available":770590072832,"internally_reserved":0,"allocated":14131200,"data_stored":7796268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001284"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693494+0000","last_change":"2024-01-26T23:48:11.690618+0000","last_active":"2024-01-26T23:48:11.693494+0000","last_peered":"2024-01-26T23:48:11.693494+0000","last_clean":"2024-01-26T23:48:11.693494+0000","last_became_active":"2024-01-26T23:48:11.690305+0000","last_became_peered":"2024-01-26T23:48:11.690305+0000","last_unstale":"2024-01-26T23:48:11.693494+0000","last_undegraded":"2024-01-26T23:48:11.693494+0000","last_fullsized":"2024-01-26T23:48:11.693494+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:17:24.039600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.700908+0000","last_change":"2024-01-26T23:48:11.695729+0000","last_active":"2024-01-26T23:48:11.700908+0000","last_peered":"2024-01-26T23:48:11.700908+0000","last_clean":"2024-01-26T23:48:11.700908+0000","last_became_active":"2024-01-26T23:48:11.694867+0000","last_became_peered":"2024-01-26T23:48:11.694867+0000","last_unstale":"2024-01-26T23:48:11.700908+0000","last_undegraded":"2024-01-26T23:48:11.700908+0000","last_fullsized":"2024-01-26T23:48:11.700908+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:22:27.593650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693576+0000","last_change":"2024-01-26T23:48:11.691344+0000","last_active":"2024-01-26T23:48:11.693576+0000","last_peered":"2024-01-26T23:48:11.693576+0000","last_clean":"2024-01-26T23:48:11.693576+0000","last_became_active":"2024-01-26T23:48:11.691069+0000","last_became_peered":"2024-01-26T23:48:11.691069+0000","last_unstale":"2024-01-26T23:48:11.693576+0000","last_undegraded":"2024-01-26T23:48:11.693576+0000","last_fullsized":"2024-01-26T23:48:11.693576+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:30.893321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698369+0000","last_change":"2024-01-26T23:48:11.694681+0000","last_active":"2024-01-26T23:48:11.698369+0000","last_peered":"2024-01-26T23:48:11.698369+0000","last_clean":"2024-01-26T23:48:11.698369+0000","last_became_active":"2024-01-26T23:48:11.693694+0000","last_became_peered":"2024-01-26T23:48:11.693694+0000","last_unstale":"2024-01-26T23:48:11.698369+0000","last_undegraded":"2024-01-26T23:48:11.698369+0000","last_fullsized":"2024-01-26T23:48:11.698369+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:10:51.561423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.689920+0000","last_change":"2024-01-26T23:48:11.688049+0000","last_active":"2024-01-26T23:48:11.689920+0000","last_peered":"2024-01-26T23:48:11.689920+0000","last_clean":"2024-01-26T23:48:11.689920+0000","last_became_active":"2024-01-26T23:48:11.687759+0000","last_became_peered":"2024-01-26T23:48:11.687759+0000","last_unstale":"2024-01-26T23:48:11.689920+0000","last_undegraded":"2024-01-26T23:48:11.689920+0000","last_fullsized":"2024-01-26T23:48:11.689920+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:08:43.039916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.700721+0000","last_change":"2024-01-26T23:48:11.694415+0000","last_active":"2024-01-26T23:48:11.700721+0000","last_peered":"2024-01-26T23:48:11.700721+0000","last_clean":"2024-01-26T23:48:11.700721+0000","last_became_active":"2024-01-26T23:48:11.694105+0000","last_became_peered":"2024-01-26T23:48:11.694105+0000","last_unstale":"2024-01-26T23:48:11.700721+0000","last_undegraded":"2024-01-26T23:48:11.700721+0000","last_fullsized":"2024-01-26T23:48:11.700721+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:24:22.377854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.707410+0000","last_change":"2024-01-26T23:48:11.702892+0000","last_active":"2024-01-26T23:48:11.707410+0000","last_peered":"2024-01-26T23:48:11.707410+0000","last_clean":"2024-01-26T23:48:11.707410+0000","last_became_active":"2024-01-26T23:48:11.702664+0000","last_became_peered":"2024-01-26T23:48:11.702664+0000","last_unstale":"2024-01-26T23:48:11.707410+0000","last_undegraded":"2024-01-26T23:48:11.707410+0000","last_fullsized":"2024-01-26T23:48:11.707410+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:25:47.188502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702212+0000","last_change":"2024-01-26T23:48:11.695520+0000","last_active":"2024-01-26T23:48:11.702212+0000","last_peered":"2024-01-26T23:48:11.702212+0000","last_clean":"2024-01-26T23:48:11.702212+0000","last_became_active":"2024-01-26T23:48:11.695205+0000","last_became_peered":"2024-01-26T23:48:11.695205+0000","last_unstale":"2024-01-26T23:48:11.702212+0000","last_undegraded":"2024-01-26T23:48:11.702212+0000","last_fullsized":"2024-01-26T23:48:11.702212+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:36:53.012187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.700678+0000","last_change":"2024-01-26T23:48:11.694658+0000","last_active":"2024-01-26T23:48:11.700678+0000","last_peered":"2024-01-26T23:48:11.700678+0000","last_clean":"2024-01-26T23:48:11.700678+0000","last_became_active":"2024-01-26T23:48:11.693934+0000","last_became_peered":"2024-01-26T23:48:11.693934+0000","last_unstale":"2024-01-26T23:48:11.700678+0000","last_undegraded":"2024-01-26T23:48:11.700678+0000","last_fullsized":"2024-01-26T23:48:11.700678+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:56:27.498369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.703088+0000","last_change":"2024-01-26T23:48:11.697158+0000","last_active":"2024-01-26T23:48:11.703088+0000","last_peered":"2024-01-26T23:48:11.703088+0000","last_clean":"2024-01-26T23:48:11.703088+0000","last_became_active":"2024-01-26T23:48:11.696633+0000","last_became_peered":"2024-01-26T23:48:11.696633+0000","last_unstale":"2024-01-26T23:48:11.703088+0000","last_undegraded":"2024-01-26T23:48:11.703088+0000","last_fullsized":"2024-01-26T23:48:11.703088+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:23:52.601066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702118+0000","last_change":"2024-01-26T23:48:11.697639+0000","last_active":"2024-01-26T23:48:11.702118+0000","last_peered":"2024-01-26T23:48:11.702118+0000","last_clean":"2024-01-26T23:48:11.702118+0000","last_became_active":"2024-01-26T23:48:11.697351+0000","last_became_peered":"2024-01-26T23:48:11.697351+0000","last_unstale":"2024-01-26T23:48:11.702118+0000","last_undegraded":"2024-01-26T23:48:11.702118+0000","last_fullsized":"2024-01-26T23:48:11.702118+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:45:48.834930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693709+0000","last_change":"2024-01-26T23:48:11.692041+0000","last_active":"2024-01-26T23:48:11.693709+0000","last_peered":"2024-01-26T23:48:11.693709+0000","last_clean":"2024-01-26T23:48:11.693709+0000","last_became_active":"2024-01-26T23:48:11.691748+0000","last_became_peered":"2024-01-26T23:48:11.691748+0000","last_unstale":"2024-01-26T23:48:11.693709+0000","last_undegraded":"2024-01-26T23:48:11.693709+0000","last_fullsized":"2024-01-26T23:48:11.693709+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:02:31.050232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.691654+0000","last_change":"2024-01-26T23:48:11.689915+0000","last_active":"2024-01-26T23:48:11.691654+0000","last_peered":"2024-01-26T23:48:11.691654+0000","last_clean":"2024-01-26T23:48:11.691654+0000","last_became_active":"2024-01-26T23:48:11.689529+0000","last_became_peered":"2024-01-26T23:48:11.689529+0000","last_unstale":"2024-01-26T23:48:11.691654+0000","last_undegraded":"2024-01-26T23:48:11.691654+0000","last_fullsized":"2024-01-26T23:48:11.691654+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:10.141932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.704794+0000","last_change":"2024-01-26T23:48:11.700345+0000","last_active":"2024-01-26T23:48:11.704794+0000","last_peered":"2024-01-26T23:48:11.704794+0000","last_clean":"2024-01-26T23:48:11.704794+0000","last_became_active":"2024-01-26T23:48:11.700057+0000","last_became_peered":"2024-01-26T23:48:11.700057+0000","last_unstale":"2024-01-26T23:48:11.704794+0000","last_undegraded":"2024-01-26T23:48:11.704794+0000","last_fullsized":"2024-01-26T23:48:11.704794+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:35:53.885389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.704742+0000","last_change":"2024-01-26T23:48:11.699805+0000","last_active":"2024-01-26T23:48:11.704742+0000","last_peered":"2024-01-26T23:48:11.704742+0000","last_clean":"2024-01-26T23:48:11.704742+0000","last_became_active":"2024-01-26T23:48:11.699526+0000","last_became_peered":"2024-01-26T23:48:11.699526+0000","last_unstale":"2024-01-26T23:48:11.704742+0000","last_undegraded":"2024-01-26T23:48:11.704742+0000","last_fullsized":"2024-01-26T23:48:11.704742+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:10.141932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.704822+0000","last_change":"2024-01-26T23:48:11.700542+0000","last_active":"2024-01-26T23:48:11.704822+0000","last_peered":"2024-01-26T23:48:11.704822+0000","last_clean":"2024-01-26T23:48:11.704822+0000","last_became_active":"2024-01-26T23:48:11.700277+0000","last_became_peered":"2024-01-26T23:48:11.700277+0000","last_unstale":"2024-01-26T23:48:11.704822+0000","last_undegraded":"2024-01-26T23:48:11.704822+0000","last_fullsized":"2024-01-26T23:48:11.704822+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:36:36.750288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693369+0000","last_change":"2024-01-26T23:48:11.686277+0000","last_active":"2024-01-26T23:48:11.693369+0000","last_peered":"2024-01-26T23:48:11.693369+0000","last_clean":"2024-01-26T23:48:11.693369+0000","last_became_active":"2024-01-26T23:48:11.686006+0000","last_became_peered":"2024-01-26T23:48:11.686006+0000","last_unstale":"2024-01-26T23:48:11.693369+0000","last_undegraded":"2024-01-26T23:48:11.693369+0000","last_fullsized":"2024-01-26T23:48:11.693369+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:22:19.028290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.700889+0000","last_change":"2024-01-26T23:48:11.694755+0000","last_active":"2024-01-26T23:48:11.700889+0000","last_peered":"2024-01-26T23:48:11.700889+0000","last_clean":"2024-01-26T23:48:11.700889+0000","last_became_active":"2024-01-26T23:48:11.694175+0000","last_became_peered":"2024-01-26T23:48:11.694175+0000","last_unstale":"2024-01-26T23:48:11.700889+0000","last_undegraded":"2024-01-26T23:48:11.700889+0000","last_fullsized":"2024-01-26T23:48:11.700889+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:55:06.345115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698855+0000","last_change":"2024-01-26T23:48:11.696828+0000","last_active":"2024-01-26T23:48:11.698855+0000","last_peered":"2024-01-26T23:48:11.698855+0000","last_clean":"2024-01-26T23:48:11.698855+0000","last_became_active":"2024-01-26T23:48:11.696291+0000","last_became_peered":"2024-01-26T23:48:11.696291+0000","last_unstale":"2024-01-26T23:48:11.698855+0000","last_undegraded":"2024-01-26T23:48:11.698855+0000","last_fullsized":"2024-01-26T23:48:11.698855+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:27:53.675503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.705011+0000","last_change":"2024-01-26T23:48:11.701106+0000","last_active":"2024-01-26T23:48:11.705011+0000","last_peered":"2024-01-26T23:48:11.705011+0000","last_clean":"2024-01-26T23:48:11.705011+0000","last_became_active":"2024-01-26T23:48:11.700818+0000","last_became_peered":"2024-01-26T23:48:11.700818+0000","last_unstale":"2024-01-26T23:48:11.705011+0000","last_undegraded":"2024-01-26T23:48:11.705011+0000","last_fullsized":"2024-01-26T23:48:11.705011+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:53:43.709494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.690830+0000","last_change":"2024-01-26T23:48:11.689380+0000","last_active":"2024-01-26T23:48:11.690830+0000","last_peered":"2024-01-26T23:48:11.690830+0000","last_clean":"2024-01-26T23:48:11.690830+0000","last_became_active":"2024-01-26T23:48:11.688997+0000","last_became_peered":"2024-01-26T23:48:11.688997+0000","last_unstale":"2024-01-26T23:48:11.690830+0000","last_undegraded":"2024-01-26T23:48:11.690830+0000","last_fullsized":"2024-01-26T23:48:11.690830+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:20.796536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.687203+0000","last_change":"2024-01-26T23:48:11.681763+0000","last_active":"2024-01-26T23:48:11.687203+0000","last_peered":"2024-01-26T23:48:11.687203+0000","last_clean":"2024-01-26T23:48:11.687203+0000","last_became_active":"2024-01-26T23:48:11.681378+0000","last_became_peered":"2024-01-26T23:48:11.681378+0000","last_unstale":"2024-01-26T23:48:11.687203+0000","last_undegraded":"2024-01-26T23:48:11.687203+0000","last_fullsized":"2024-01-26T23:48:11.687203+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:33:21.325282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693341+0000","last_change":"2024-01-26T23:48:11.685918+0000","last_active":"2024-01-26T23:48:11.693341+0000","last_peered":"2024-01-26T23:48:11.693341+0000","last_clean":"2024-01-26T23:48:11.693341+0000","last_became_active":"2024-01-26T23:48:11.685566+0000","last_became_peered":"2024-01-26T23:48:11.685566+0000","last_unstale":"2024-01-26T23:48:11.693341+0000","last_undegraded":"2024-01-26T23:48:11.693341+0000","last_fullsized":"2024-01-26T23:48:11.693341+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:06:45.718179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.684940+0000","last_change":"2024-01-26T23:48:11.682824+0000","last_active":"2024-01-26T23:48:11.684940+0000","last_peered":"2024-01-26T23:48:11.684940+0000","last_clean":"2024-01-26T23:48:11.684940+0000","last_became_active":"2024-01-26T23:48:11.682545+0000","last_became_peered":"2024-01-26T23:48:11.682545+0000","last_unstale":"2024-01-26T23:48:11.684940+0000","last_undegraded":"2024-01-26T23:48:11.684940+0000","last_fullsized":"2024-01-26T23:48:11.684940+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:50:50.668201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.701967+0000","last_change":"2024-01-26T23:48:11.696001+0000","last_active":"2024-01-26T23:48:11.701967+0000","last_peered":"2024-01-26T23:48:11.701967+0000","last_clean":"2024-01-26T23:48:11.701967+0000","last_became_active":"2024-01-26T23:48:11.695692+0000","last_became_peered":"2024-01-26T23:48:11.695692+0000","last_unstale":"2024-01-26T23:48:11.701967+0000","last_undegraded":"2024-01-26T23:48:11.701967+0000","last_fullsized":"2024-01-26T23:48:11.701967+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:25:59.436650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702281+0000","last_change":"2024-01-26T23:48:11.696416+0000","last_active":"2024-01-26T23:48:11.702281+0000","last_peered":"2024-01-26T23:48:11.702281+0000","last_clean":"2024-01-26T23:48:11.702281+0000","last_became_active":"2024-01-26T23:48:11.696137+0000","last_became_peered":"2024-01-26T23:48:11.696137+0000","last_unstale":"2024-01-26T23:48:11.702281+0000","last_undegraded":"2024-01-26T23:48:11.702281+0000","last_fullsized":"2024-01-26T23:48:11.702281+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:37:36.865287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.704679+0000","last_change":"2024-01-26T23:48:11.700455+0000","last_active":"2024-01-26T23:48:11.704679+0000","last_peered":"2024-01-26T23:48:11.704679+0000","last_clean":"2024-01-26T23:48:11.704679+0000","last_became_active":"2024-01-26T23:48:11.700192+0000","last_became_peered":"2024-01-26T23:48:11.700192+0000","last_unstale":"2024-01-26T23:48:11.704679+0000","last_undegraded":"2024-01-26T23:48:11.704679+0000","last_fullsized":"2024-01-26T23:48:11.704679+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:27.902905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702235+0000","last_change":"2024-01-26T23:48:11.698782+0000","last_active":"2024-01-26T23:48:11.702235+0000","last_peered":"2024-01-26T23:48:11.702235+0000","last_clean":"2024-01-26T23:48:11.702235+0000","last_became_active":"2024-01-26T23:48:11.698476+0000","last_became_peered":"2024-01-26T23:48:11.698476+0000","last_unstale":"2024-01-26T23:48:11.702235+0000","last_undegraded":"2024-01-26T23:48:11.702235+0000","last_fullsized":"2024-01-26T23:48:11.702235+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:27:13.343954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702333+0000","last_change":"2024-01-26T23:48:11.700735+0000","last_active":"2024-01-26T23:48:11.702333+0000","last_peered":"2024-01-26T23:48:11.702333+0000","last_clean":"2024-01-26T23:48:11.702333+0000","last_became_active":"2024-01-26T23:48:11.700416+0000","last_became_peered":"2024-01-26T23:48:11.700416+0000","last_unstale":"2024-01-26T23:48:11.702333+0000","last_undegraded":"2024-01-26T23:48:11.702333+0000","last_fullsized":"2024-01-26T23:48:11.702333+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:03:16.363445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699270+0000","last_change":"2024-01-26T23:48:11.693380+0000","last_active":"2024-01-26T23:48:11.699270+0000","last_peered":"2024-01-26T23:48:11.699270+0000","last_clean":"2024-01-26T23:48:11.699270+0000","last_became_active":"2024-01-26T23:48:11.692986+0000","last_became_peered":"2024-01-26T23:48:11.692986+0000","last_unstale":"2024-01-26T23:48:11.699270+0000","last_undegraded":"2024-01-26T23:48:11.699270+0000","last_fullsized":"2024-01-26T23:48:11.699270+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:37:57.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699950+0000","last_change":"2024-01-26T23:48:11.695328+0000","last_active":"2024-01-26T23:48:11.699950+0000","last_peered":"2024-01-26T23:48:11.699950+0000","last_clean":"2024-01-26T23:48:11.699950+0000","last_became_active":"2024-01-26T23:48:11.694749+0000","last_became_peered":"2024-01-26T23:48:11.694749+0000","last_unstale":"2024-01-26T23:48:11.699950+0000","last_undegraded":"2024-01-26T23:48:11.699950+0000","last_fullsized":"2024-01-26T23:48:11.699950+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:11:59.205203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699884+0000","last_change":"2024-01-26T23:48:11.694600+0000","last_active":"2024-01-26T23:48:11.699884+0000","last_peered":"2024-01-26T23:48:11.699884+0000","last_clean":"2024-01-26T23:48:11.699884+0000","last_became_active":"2024-01-26T23:48:11.694195+0000","last_became_peered":"2024-01-26T23:48:11.694195+0000","last_unstale":"2024-01-26T23:48:11.699884+0000","last_undegraded":"2024-01-26T23:48:11.699884+0000","last_fullsized":"2024-01-26T23:48:11.699884+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:54:52.459709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.697971+0000","last_change":"2024-01-26T23:48:11.695928+0000","last_active":"2024-01-26T23:48:11.697971+0000","last_peered":"2024-01-26T23:48:11.697971+0000","last_clean":"2024-01-26T23:48:11.697971+0000","last_became_active":"2024-01-26T23:48:11.695622+0000","last_became_peered":"2024-01-26T23:48:11.695622+0000","last_unstale":"2024-01-26T23:48:11.697971+0000","last_undegraded":"2024-01-26T23:48:11.697971+0000","last_fullsized":"2024-01-26T23:48:11.697971+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:37:48.746729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"129'4","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806763+0000","last_change":"2024-01-26T23:48:10.820996+0000","last_active":"2024-01-26T23:48:11.806763+0000","last_peered":"2024-01-26T23:48:11.806763+0000","last_clean":"2024-01-26T23:48:11.806763+0000","last_became_active":"2024-01-26T23:48:10.820659+0000","last_became_peered":"2024-01-26T23:48:10.820659+0000","last_unstale":"2024-01-26T23:48:11.806763+0000","last_undegraded":"2024-01-26T23:48:11.806763+0000","last_fullsized":"2024-01-26T23:48:11.806763+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:32:29.511141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702321+0000","last_change":"2024-01-26T23:48:11.699321+0000","last_active":"2024-01-26T23:48:11.702321+0000","last_peered":"2024-01-26T23:48:11.702321+0000","last_clean":"2024-01-26T23:48:11.702321+0000","last_became_active":"2024-01-26T23:48:11.699085+0000","last_became_peered":"2024-01-26T23:48:11.699085+0000","last_unstale":"2024-01-26T23:48:11.702321+0000","last_undegraded":"2024-01-26T23:48:11.702321+0000","last_fullsized":"2024-01-26T23:48:11.702321+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:49:57.681990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"129'1","reported_seq":11,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.805807+0000","last_change":"2024-01-26T23:48:10.816548+0000","last_active":"2024-01-26T23:48:11.805807+0000","last_peered":"2024-01-26T23:48:11.805807+0000","last_clean":"2024-01-26T23:48:11.805807+0000","last_became_active":"2024-01-26T23:48:10.816271+0000","last_became_peered":"2024-01-26T23:48:10.816271+0000","last_unstale":"2024-01-26T23:48:11.805807+0000","last_undegraded":"2024-01-26T23:48:11.805807+0000","last_fullsized":"2024-01-26T23:48:11.805807+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:10.976028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693908+0000","last_change":"2024-01-26T23:48:11.691251+0000","last_active":"2024-01-26T23:48:11.693908+0000","last_peered":"2024-01-26T23:48:11.693908+0000","last_clean":"2024-01-26T23:48:11.693908+0000","last_became_active":"2024-01-26T23:48:11.690973+0000","last_became_peered":"2024-01-26T23:48:11.690973+0000","last_unstale":"2024-01-26T23:48:11.693908+0000","last_undegraded":"2024-01-26T23:48:11.693908+0000","last_fullsized":"2024-01-26T23:48:11.693908+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:15:54.696377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.802933+0000","last_change":"2024-01-26T23:48:10.811743+0000","last_active":"2024-01-26T23:48:11.802933+0000","last_peered":"2024-01-26T23:48:11.802933+0000","last_clean":"2024-01-26T23:48:11.802933+0000","last_became_active":"2024-01-26T23:48:10.810670+0000","last_became_peered":"2024-01-26T23:48:10.810670+0000","last_unstale":"2024-01-26T23:48:11.802933+0000","last_undegraded":"2024-01-26T23:48:11.802933+0000","last_fullsized":"2024-01-26T23:48:11.802933+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:23:51.594366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699753+0000","last_change":"2024-01-26T23:48:11.696014+0000","last_active":"2024-01-26T23:48:11.699753+0000","last_peered":"2024-01-26T23:48:11.699753+0000","last_clean":"2024-01-26T23:48:11.699753+0000","last_became_active":"2024-01-26T23:48:11.695745+0000","last_became_peered":"2024-01-26T23:48:11.695745+0000","last_unstale":"2024-01-26T23:48:11.699753+0000","last_undegraded":"2024-01-26T23:48:11.699753+0000","last_fullsized":"2024-01-26T23:48:11.699753+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:26:08.221069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.803587+0000","last_change":"2024-01-26T23:48:10.812880+0000","last_active":"2024-01-26T23:48:11.803587+0000","last_peered":"2024-01-26T23:48:11.803587+0000","last_clean":"2024-01-26T23:48:11.803587+0000","last_became_active":"2024-01-26T23:48:10.812205+0000","last_became_peered":"2024-01-26T23:48:10.812205+0000","last_unstale":"2024-01-26T23:48:11.803587+0000","last_undegraded":"2024-01-26T23:48:11.803587+0000","last_fullsized":"2024-01-26T23:48:11.803587+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:26.896205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.691775+0000","last_change":"2024-01-26T23:48:11.687570+0000","last_active":"2024-01-26T23:48:11.691775+0000","last_peered":"2024-01-26T23:48:11.691775+0000","last_clean":"2024-01-26T23:48:11.691775+0000","last_became_active":"2024-01-26T23:48:11.686069+0000","last_became_peered":"2024-01-26T23:48:11.686069+0000","last_unstale":"2024-01-26T23:48:11.691775+0000","last_undegraded":"2024-01-26T23:48:11.691775+0000","last_fullsized":"2024-01-26T23:48:11.691775+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:36:36.750288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"129'1","reported_seq":11,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.805568+0000","last_change":"2024-01-26T23:48:10.815157+0000","last_active":"2024-01-26T23:48:11.805568+0000","last_peered":"2024-01-26T23:48:11.805568+0000","last_clean":"2024-01-26T23:48:11.805568+0000","last_became_active":"2024-01-26T23:48:10.814838+0000","last_became_peered":"2024-01-26T23:48:10.814838+0000","last_unstale":"2024-01-26T23:48:11.805568+0000","last_undegraded":"2024-01-26T23:48:11.805568+0000","last_fullsized":"2024-01-26T23:48:11.805568+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:49:44.669484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698877+0000","last_change":"2024-01-26T23:48:11.697027+0000","last_active":"2024-01-26T23:48:11.698877+0000","last_peered":"2024-01-26T23:48:11.698877+0000","last_clean":"2024-01-26T23:48:11.698877+0000","last_became_active":"2024-01-26T23:48:11.696702+0000","last_became_peered":"2024-01-26T23:48:11.696702+0000","last_unstale":"2024-01-26T23:48:11.698877+0000","last_undegraded":"2024-01-26T23:48:11.698877+0000","last_fullsized":"2024-01-26T23:48:11.698877+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:38:32.101523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"129'1","reported_seq":11,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806591+0000","last_change":"2024-01-26T23:48:10.813718+0000","last_active":"2024-01-26T23:48:11.806591+0000","last_peered":"2024-01-26T23:48:11.806591+0000","last_clean":"2024-01-26T23:48:11.806591+0000","last_became_active":"2024-01-26T23:48:10.813464+0000","last_became_peered":"2024-01-26T23:48:10.813464+0000","last_unstale":"2024-01-26T23:48:11.806591+0000","last_undegraded":"2024-01-26T23:48:11.806591+0000","last_fullsized":"2024-01-26T23:48:11.806591+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:13:40.216491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702230+0000","last_change":"2024-01-26T23:48:11.698344+0000","last_active":"2024-01-26T23:48:11.702230+0000","last_peered":"2024-01-26T23:48:11.702230+0000","last_clean":"2024-01-26T23:48:11.702230+0000","last_became_active":"2024-01-26T23:48:11.698034+0000","last_became_peered":"2024-01-26T23:48:11.698034+0000","last_unstale":"2024-01-26T23:48:11.702230+0000","last_undegraded":"2024-01-26T23:48:11.702230+0000","last_fullsized":"2024-01-26T23:48:11.702230+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:38:32.447049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685115+0000","last_change":"2024-01-26T23:48:10.811013+0000","last_active":"2024-01-26T23:48:11.685115+0000","last_peered":"2024-01-26T23:48:11.685115+0000","last_clean":"2024-01-26T23:48:11.685115+0000","last_became_active":"2024-01-26T23:48:10.810684+0000","last_became_peered":"2024-01-26T23:48:10.810684+0000","last_unstale":"2024-01-26T23:48:11.685115+0000","last_undegraded":"2024-01-26T23:48:11.685115+0000","last_fullsized":"2024-01-26T23:48:11.685115+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:29:34.900670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699671+0000","last_change":"2024-01-26T23:48:11.694858+0000","last_active":"2024-01-26T23:48:11.699671+0000","last_peered":"2024-01-26T23:48:11.699671+0000","last_clean":"2024-01-26T23:48:11.699671+0000","last_became_active":"2024-01-26T23:48:11.694407+0000","last_became_peered":"2024-01-26T23:48:11.694407+0000","last_unstale":"2024-01-26T23:48:11.699671+0000","last_undegraded":"2024-01-26T23:48:11.699671+0000","last_fullsized":"2024-01-26T23:48:11.699671+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:33:21.325282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.681769+0000","last_change":"2024-01-26T23:48:10.807380+0000","last_active":"2024-01-26T23:48:11.681769+0000","last_peered":"2024-01-26T23:48:11.681769+0000","last_clean":"2024-01-26T23:48:11.681769+0000","last_became_active":"2024-01-26T23:48:10.806968+0000","last_became_peered":"2024-01-26T23:48:10.806968+0000","last_unstale":"2024-01-26T23:48:11.681769+0000","last_undegraded":"2024-01-26T23:48:11.681769+0000","last_fullsized":"2024-01-26T23:48:11.681769+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:49:44.669484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.697301+0000","last_change":"2024-01-26T23:48:11.695119+0000","last_active":"2024-01-26T23:48:11.697301+0000","last_peered":"2024-01-26T23:48:11.697301+0000","last_clean":"2024-01-26T23:48:11.697301+0000","last_became_active":"2024-01-26T23:48:11.694615+0000","last_became_peered":"2024-01-26T23:48:11.694615+0000","last_unstale":"2024-01-26T23:48:11.697301+0000","last_undegraded":"2024-01-26T23:48:11.697301+0000","last_fullsized":"2024-01-26T23:48:11.697301+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:36:13.295894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685363+0000","last_change":"2024-01-26T23:48:10.811335+0000","last_active":"2024-01-26T23:48:11.685363+0000","last_peered":"2024-01-26T23:48:11.685363+0000","last_clean":"2024-01-26T23:48:11.685363+0000","last_became_active":"2024-01-26T23:48:10.810926+0000","last_became_peered":"2024-01-26T23:48:10.810926+0000","last_unstale":"2024-01-26T23:48:11.685363+0000","last_undegraded":"2024-01-26T23:48:11.685363+0000","last_fullsized":"2024-01-26T23:48:11.685363+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:22:26.586950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.703222+0000","last_change":"2024-01-26T23:48:11.697447+0000","last_active":"2024-01-26T23:48:11.703222+0000","last_peered":"2024-01-26T23:48:11.703222+0000","last_clean":"2024-01-26T23:48:11.703222+0000","last_became_active":"2024-01-26T23:48:11.697133+0000","last_became_peered":"2024-01-26T23:48:11.697133+0000","last_unstale":"2024-01-26T23:48:11.703222+0000","last_undegraded":"2024-01-26T23:48:11.703222+0000","last_fullsized":"2024-01-26T23:48:11.703222+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:28:53.642854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.677121+0000","last_change":"2024-01-26T23:48:10.810309+0000","last_active":"2024-01-26T23:48:11.677121+0000","last_peered":"2024-01-26T23:48:11.677121+0000","last_clean":"2024-01-26T23:48:11.677121+0000","last_became_active":"2024-01-26T23:48:10.810028+0000","last_became_peered":"2024-01-26T23:48:10.810028+0000","last_unstale":"2024-01-26T23:48:11.677121+0000","last_undegraded":"2024-01-26T23:48:11.677121+0000","last_fullsized":"2024-01-26T23:48:11.677121+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:36:07.346545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702237+0000","last_change":"2024-01-26T23:48:11.695812+0000","last_active":"2024-01-26T23:48:11.702237+0000","last_peered":"2024-01-26T23:48:11.702237+0000","last_clean":"2024-01-26T23:48:11.702237+0000","last_became_active":"2024-01-26T23:48:11.695535+0000","last_became_peered":"2024-01-26T23:48:11.695535+0000","last_unstale":"2024-01-26T23:48:11.702237+0000","last_undegraded":"2024-01-26T23:48:11.702237+0000","last_fullsized":"2024-01-26T23:48:11.702237+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:12:39.189557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"129'3","reported_seq":13,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806352+0000","last_change":"2024-01-26T23:48:10.813349+0000","last_active":"2024-01-26T23:48:11.806352+0000","last_peered":"2024-01-26T23:48:11.806352+0000","last_clean":"2024-01-26T23:48:11.806352+0000","last_became_active":"2024-01-26T23:48:10.813052+0000","last_became_peered":"2024-01-26T23:48:10.813052+0000","last_unstale":"2024-01-26T23:48:11.806352+0000","last_undegraded":"2024-01-26T23:48:11.806352+0000","last_fullsized":"2024-01-26T23:48:11.806352+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:50:15.445301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.703727+0000","last_change":"2024-01-26T23:48:11.703049+0000","last_active":"2024-01-26T23:48:11.703727+0000","last_peered":"2024-01-26T23:48:11.703727+0000","last_clean":"2024-01-26T23:48:11.703727+0000","last_became_active":"2024-01-26T23:48:11.702636+0000","last_became_peered":"2024-01-26T23:48:11.702636+0000","last_unstale":"2024-01-26T23:48:11.703727+0000","last_undegraded":"2024-01-26T23:48:11.703727+0000","last_fullsized":"2024-01-26T23:48:11.703727+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:50:12.326901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"129'6","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.807949+0000","last_change":"2024-01-26T23:48:10.814838+0000","last_active":"2024-01-26T23:48:11.807949+0000","last_peered":"2024-01-26T23:48:11.807949+0000","last_clean":"2024-01-26T23:48:11.807949+0000","last_became_active":"2024-01-26T23:48:10.814534+0000","last_became_peered":"2024-01-26T23:48:10.814534+0000","last_unstale":"2024-01-26T23:48:11.807949+0000","last_undegraded":"2024-01-26T23:48:11.807949+0000","last_fullsized":"2024-01-26T23:48:11.807949+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:51:57.293119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.688225+0000","last_change":"2024-01-26T23:48:11.683406+0000","last_active":"2024-01-26T23:48:11.688225+0000","last_peered":"2024-01-26T23:48:11.688225+0000","last_clean":"2024-01-26T23:48:11.688225+0000","last_became_active":"2024-01-26T23:48:11.683122+0000","last_became_peered":"2024-01-26T23:48:11.683122+0000","last_unstale":"2024-01-26T23:48:11.688225+0000","last_undegraded":"2024-01-26T23:48:11.688225+0000","last_fullsized":"2024-01-26T23:48:11.688225+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:59:48.214103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.804068+0000","last_change":"2024-01-26T23:48:10.812433+0000","last_active":"2024-01-26T23:48:11.804068+0000","last_peered":"2024-01-26T23:48:11.804068+0000","last_clean":"2024-01-26T23:48:11.804068+0000","last_became_active":"2024-01-26T23:48:10.812111+0000","last_became_peered":"2024-01-26T23:48:10.812111+0000","last_unstale":"2024-01-26T23:48:11.804068+0000","last_undegraded":"2024-01-26T23:48:11.804068+0000","last_fullsized":"2024-01-26T23:48:11.804068+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:49:44.669484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.697279+0000","last_change":"2024-01-26T23:48:11.694313+0000","last_active":"2024-01-26T23:48:11.697279+0000","last_peered":"2024-01-26T23:48:11.697279+0000","last_clean":"2024-01-26T23:48:11.697279+0000","last_became_active":"2024-01-26T23:48:11.693806+0000","last_became_peered":"2024-01-26T23:48:11.693806+0000","last_unstale":"2024-01-26T23:48:11.697279+0000","last_undegraded":"2024-01-26T23:48:11.697279+0000","last_fullsized":"2024-01-26T23:48:11.697279+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:07.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.682226+0000","last_change":"2024-01-26T23:48:10.819155+0000","last_active":"2024-01-26T23:48:11.682226+0000","last_peered":"2024-01-26T23:48:11.682226+0000","last_clean":"2024-01-26T23:48:11.682226+0000","last_became_active":"2024-01-26T23:48:10.818853+0000","last_became_peered":"2024-01-26T23:48:10.818853+0000","last_unstale":"2024-01-26T23:48:11.682226+0000","last_undegraded":"2024-01-26T23:48:11.682226+0000","last_fullsized":"2024-01-26T23:48:11.682226+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:50:15.445301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699954+0000","last_change":"2024-01-26T23:48:11.694802+0000","last_active":"2024-01-26T23:48:11.699954+0000","last_peered":"2024-01-26T23:48:11.699954+0000","last_clean":"2024-01-26T23:48:11.699954+0000","last_became_active":"2024-01-26T23:48:11.694277+0000","last_became_peered":"2024-01-26T23:48:11.694277+0000","last_unstale":"2024-01-26T23:48:11.699954+0000","last_undegraded":"2024-01-26T23:48:11.699954+0000","last_fullsized":"2024-01-26T23:48:11.699954+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:28:58.191038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.684638+0000","last_change":"2024-01-26T23:48:10.813933+0000","last_active":"2024-01-26T23:48:11.684638+0000","last_peered":"2024-01-26T23:48:11.684638+0000","last_clean":"2024-01-26T23:48:11.684638+0000","last_became_active":"2024-01-26T23:48:10.813646+0000","last_became_peered":"2024-01-26T23:48:10.813646+0000","last_unstale":"2024-01-26T23:48:11.684638+0000","last_undegraded":"2024-01-26T23:48:11.684638+0000","last_fullsized":"2024-01-26T23:48:11.684638+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:20:47.863263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693345+0000","last_change":"2024-01-26T23:48:11.683741+0000","last_active":"2024-01-26T23:48:11.693345+0000","last_peered":"2024-01-26T23:48:11.693345+0000","last_clean":"2024-01-26T23:48:11.693345+0000","last_became_active":"2024-01-26T23:48:11.683413+0000","last_became_peered":"2024-01-26T23:48:11.683413+0000","last_unstale":"2024-01-26T23:48:11.693345+0000","last_undegraded":"2024-01-26T23:48:11.693345+0000","last_fullsized":"2024-01-26T23:48:11.693345+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:53:44.050356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"129'8","reported_seq":18,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806654+0000","last_change":"2024-01-26T23:48:10.810870+0000","last_active":"2024-01-26T23:48:11.806654+0000","last_peered":"2024-01-26T23:48:11.806654+0000","last_clean":"2024-01-26T23:48:11.806654+0000","last_became_active":"2024-01-26T23:48:10.810516+0000","last_became_peered":"2024-01-26T23:48:10.810516+0000","last_unstale":"2024-01-26T23:48:11.806654+0000","last_undegraded":"2024-01-26T23:48:11.806654+0000","last_fullsized":"2024-01-26T23:48:11.806654+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:11:58.198504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699904+0000","last_change":"2024-01-26T23:48:11.694643+0000","last_active":"2024-01-26T23:48:11.699904+0000","last_peered":"2024-01-26T23:48:11.699904+0000","last_clean":"2024-01-26T23:48:11.699904+0000","last_became_active":"2024-01-26T23:48:11.694296+0000","last_became_peered":"2024-01-26T23:48:11.694296+0000","last_unstale":"2024-01-26T23:48:11.699904+0000","last_undegraded":"2024-01-26T23:48:11.699904+0000","last_fullsized":"2024-01-26T23:48:11.699904+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:53:21.191149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"129'1","reported_seq":11,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806724+0000","last_change":"2024-01-26T23:48:10.815595+0000","last_active":"2024-01-26T23:48:11.806724+0000","last_peered":"2024-01-26T23:48:11.806724+0000","last_clean":"2024-01-26T23:48:11.806724+0000","last_became_active":"2024-01-26T23:48:10.815314+0000","last_became_peered":"2024-01-26T23:48:10.815314+0000","last_unstale":"2024-01-26T23:48:11.806724+0000","last_undegraded":"2024-01-26T23:48:11.806724+0000","last_fullsized":"2024-01-26T23:48:11.806724+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:26.896205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.700740+0000","last_change":"2024-01-26T23:48:11.695066+0000","last_active":"2024-01-26T23:48:11.700740+0000","last_peered":"2024-01-26T23:48:11.700740+0000","last_clean":"2024-01-26T23:48:11.700740+0000","last_became_active":"2024-01-26T23:48:11.694666+0000","last_became_peered":"2024-01-26T23:48:11.694666+0000","last_unstale":"2024-01-26T23:48:11.700740+0000","last_undegraded":"2024-01-26T23:48:11.700740+0000","last_fullsized":"2024-01-26T23:48:11.700740+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:50:12.326901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"129'1","reported_seq":11,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806244+0000","last_change":"2024-01-26T23:48:10.814032+0000","last_active":"2024-01-26T23:48:11.806244+0000","last_peered":"2024-01-26T23:48:11.806244+0000","last_clean":"2024-01-26T23:48:11.806244+0000","last_became_active":"2024-01-26T23:48:10.813750+0000","last_became_peered":"2024-01-26T23:48:10.813750+0000","last_unstale":"2024-01-26T23:48:11.806244+0000","last_undegraded":"2024-01-26T23:48:11.806244+0000","last_fullsized":"2024-01-26T23:48:11.806244+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:45:19.808817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.703599+0000","last_change":"2024-01-26T23:48:11.701949+0000","last_active":"2024-01-26T23:48:11.703599+0000","last_peered":"2024-01-26T23:48:11.703599+0000","last_clean":"2024-01-26T23:48:11.703599+0000","last_became_active":"2024-01-26T23:48:11.701651+0000","last_became_peered":"2024-01-26T23:48:11.701651+0000","last_unstale":"2024-01-26T23:48:11.703599+0000","last_undegraded":"2024-01-26T23:48:11.703599+0000","last_fullsized":"2024-01-26T23:48:11.703599+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:10:51.561423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.681257+0000","last_change":"2024-01-26T23:48:10.815606+0000","last_active":"2024-01-26T23:48:11.681257+0000","last_peered":"2024-01-26T23:48:11.681257+0000","last_clean":"2024-01-26T23:48:11.681257+0000","last_became_active":"2024-01-26T23:48:10.815300+0000","last_became_peered":"2024-01-26T23:48:10.815300+0000","last_unstale":"2024-01-26T23:48:11.681257+0000","last_undegraded":"2024-01-26T23:48:11.681257+0000","last_fullsized":"2024-01-26T23:48:11.681257+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:46:44.771350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693545+0000","last_change":"2024-01-26T23:48:11.689724+0000","last_active":"2024-01-26T23:48:11.693545+0000","last_peered":"2024-01-26T23:48:11.693545+0000","last_clean":"2024-01-26T23:48:11.693545+0000","last_became_active":"2024-01-26T23:48:11.689438+0000","last_became_peered":"2024-01-26T23:48:11.689438+0000","last_unstale":"2024-01-26T23:48:11.693545+0000","last_undegraded":"2024-01-26T23:48:11.693545+0000","last_fullsized":"2024-01-26T23:48:11.693545+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:01.094001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.803125+0000","last_change":"2024-01-26T23:48:10.815064+0000","last_active":"2024-01-26T23:48:11.803125+0000","last_peered":"2024-01-26T23:48:11.803125+0000","last_clean":"2024-01-26T23:48:11.803125+0000","last_became_active":"2024-01-26T23:48:10.814778+0000","last_became_peered":"2024-01-26T23:48:10.814778+0000","last_unstale":"2024-01-26T23:48:11.803125+0000","last_undegraded":"2024-01-26T23:48:11.803125+0000","last_fullsized":"2024-01-26T23:48:11.803125+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:17.080283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.680810+0000","last_change":"2024-01-26T23:48:11.679952+0000","last_active":"2024-01-26T23:48:11.680810+0000","last_peered":"2024-01-26T23:48:11.680810+0000","last_clean":"2024-01-26T23:48:11.680810+0000","last_became_active":"2024-01-26T23:48:11.679509+0000","last_became_peered":"2024-01-26T23:48:11.679509+0000","last_unstale":"2024-01-26T23:48:11.680810+0000","last_undegraded":"2024-01-26T23:48:11.680810+0000","last_fullsized":"2024-01-26T23:48:11.680810+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:35:33.076540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.686302+0000","last_change":"2024-01-26T23:48:10.813851+0000","last_active":"2024-01-26T23:48:11.686302+0000","last_peered":"2024-01-26T23:48:11.686302+0000","last_clean":"2024-01-26T23:48:11.686302+0000","last_became_active":"2024-01-26T23:48:10.813510+0000","last_became_peered":"2024-01-26T23:48:10.813510+0000","last_unstale":"2024-01-26T23:48:11.686302+0000","last_undegraded":"2024-01-26T23:48:11.686302+0000","last_fullsized":"2024-01-26T23:48:11.686302+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:26:11.395397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698546+0000","last_change":"2024-01-26T23:48:11.695637+0000","last_active":"2024-01-26T23:48:11.698546+0000","last_peered":"2024-01-26T23:48:11.698546+0000","last_clean":"2024-01-26T23:48:11.698546+0000","last_became_active":"2024-01-26T23:48:11.695304+0000","last_became_peered":"2024-01-26T23:48:11.695304+0000","last_unstale":"2024-01-26T23:48:11.698546+0000","last_undegraded":"2024-01-26T23:48:11.698546+0000","last_fullsized":"2024-01-26T23:48:11.698546+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:50:50.668201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.690137+0000","last_change":"2024-01-26T23:48:10.815486+0000","last_active":"2024-01-26T23:48:11.690137+0000","last_peered":"2024-01-26T23:48:11.690137+0000","last_clean":"2024-01-26T23:48:11.690137+0000","last_became_active":"2024-01-26T23:48:10.815197+0000","last_became_peered":"2024-01-26T23:48:10.815197+0000","last_unstale":"2024-01-26T23:48:11.690137+0000","last_undegraded":"2024-01-26T23:48:11.690137+0000","last_fullsized":"2024-01-26T23:48:11.690137+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:57.119524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698608+0000","last_change":"2024-01-26T23:48:11.692941+0000","last_active":"2024-01-26T23:48:11.698608+0000","last_peered":"2024-01-26T23:48:11.698608+0000","last_clean":"2024-01-26T23:48:11.698608+0000","last_became_active":"2024-01-26T23:48:11.692605+0000","last_became_peered":"2024-01-26T23:48:11.692605+0000","last_unstale":"2024-01-26T23:48:11.698608+0000","last_undegraded":"2024-01-26T23:48:11.698608+0000","last_fullsized":"2024-01-26T23:48:11.698608+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:01:01.083122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685230+0000","last_change":"2024-01-26T23:48:10.816002+0000","last_active":"2024-01-26T23:48:11.685230+0000","last_peered":"2024-01-26T23:48:11.685230+0000","last_clean":"2024-01-26T23:48:11.685230+0000","last_became_active":"2024-01-26T23:48:10.815730+0000","last_became_peered":"2024-01-26T23:48:10.815730+0000","last_unstale":"2024-01-26T23:48:11.685230+0000","last_undegraded":"2024-01-26T23:48:11.685230+0000","last_fullsized":"2024-01-26T23:48:11.685230+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:59.034288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693414+0000","last_change":"2024-01-26T23:48:11.685991+0000","last_active":"2024-01-26T23:48:11.693414+0000","last_peered":"2024-01-26T23:48:11.693414+0000","last_clean":"2024-01-26T23:48:11.693414+0000","last_became_active":"2024-01-26T23:48:11.685598+0000","last_became_peered":"2024-01-26T23:48:11.685598+0000","last_unstale":"2024-01-26T23:48:11.693414+0000","last_undegraded":"2024-01-26T23:48:11.693414+0000","last_fullsized":"2024-01-26T23:48:11.693414+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:48:35.115441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.804084+0000","last_change":"2024-01-26T23:48:10.817696+0000","last_active":"2024-01-26T23:48:11.804084+0000","last_peered":"2024-01-26T23:48:11.804084+0000","last_clean":"2024-01-26T23:48:11.804084+0000","last_became_active":"2024-01-26T23:48:10.815112+0000","last_became_peered":"2024-01-26T23:48:10.815112+0000","last_unstale":"2024-01-26T23:48:11.804084+0000","last_undegraded":"2024-01-26T23:48:11.804084+0000","last_fullsized":"2024-01-26T23:48:11.804084+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:50:15.445301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693883+0000","last_change":"2024-01-26T23:48:11.691998+0000","last_active":"2024-01-26T23:48:11.693883+0000","last_peered":"2024-01-26T23:48:11.693883+0000","last_clean":"2024-01-26T23:48:11.693883+0000","last_became_active":"2024-01-26T23:48:11.691685+0000","last_became_peered":"2024-01-26T23:48:11.691685+0000","last_unstale":"2024-01-26T23:48:11.693883+0000","last_undegraded":"2024-01-26T23:48:11.693883+0000","last_fullsized":"2024-01-26T23:48:11.693883+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:35:42.319231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.677205+0000","last_change":"2024-01-26T23:48:10.806631+0000","last_active":"2024-01-26T23:48:11.677205+0000","last_peered":"2024-01-26T23:48:11.677205+0000","last_clean":"2024-01-26T23:48:11.677205+0000","last_became_active":"2024-01-26T23:48:10.806274+0000","last_became_peered":"2024-01-26T23:48:10.806274+0000","last_unstale":"2024-01-26T23:48:11.677205+0000","last_undegraded":"2024-01-26T23:48:11.677205+0000","last_fullsized":"2024-01-26T23:48:11.677205+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:59:28.765875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698753+0000","last_change":"2024-01-26T23:48:11.698307+0000","last_active":"2024-01-26T23:48:11.698753+0000","last_peered":"2024-01-26T23:48:11.698753+0000","last_clean":"2024-01-26T23:48:11.698753+0000","last_became_active":"2024-01-26T23:48:11.698019+0000","last_became_peered":"2024-01-26T23:48:11.698019+0000","last_unstale":"2024-01-26T23:48:11.698753+0000","last_undegraded":"2024-01-26T23:48:11.698753+0000","last_fullsized":"2024-01-26T23:48:11.698753+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:08:43.039916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.691955+0000","last_change":"2024-01-26T23:48:10.821818+0000","last_active":"2024-01-26T23:48:11.691955+0000","last_peered":"2024-01-26T23:48:11.691955+0000","last_clean":"2024-01-26T23:48:11.691955+0000","last_became_active":"2024-01-26T23:48:10.821530+0000","last_became_peered":"2024-01-26T23:48:10.821530+0000","last_unstale":"2024-01-26T23:48:11.691955+0000","last_undegraded":"2024-01-26T23:48:11.691955+0000","last_fullsized":"2024-01-26T23:48:11.691955+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:26:07.214369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.688038+0000","last_change":"2024-01-26T23:48:11.682394+0000","last_active":"2024-01-26T23:48:11.688038+0000","last_peered":"2024-01-26T23:48:11.688038+0000","last_clean":"2024-01-26T23:48:11.688038+0000","last_became_active":"2024-01-26T23:48:11.682136+0000","last_became_peered":"2024-01-26T23:48:11.682136+0000","last_unstale":"2024-01-26T23:48:11.688038+0000","last_undegraded":"2024-01-26T23:48:11.688038+0000","last_fullsized":"2024-01-26T23:48:11.688038+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:45:20.815517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.673973+0000","last_change":"2024-01-26T23:48:10.813068+0000","last_active":"2024-01-26T23:48:11.673973+0000","last_peered":"2024-01-26T23:48:11.673973+0000","last_clean":"2024-01-26T23:48:11.673973+0000","last_became_active":"2024-01-26T23:48:10.812610+0000","last_became_peered":"2024-01-26T23:48:10.812610+0000","last_unstale":"2024-01-26T23:48:11.673973+0000","last_undegraded":"2024-01-26T23:48:11.673973+0000","last_fullsized":"2024-01-26T23:48:11.673973+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:45:19.808817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698643+0000","last_change":"2024-01-26T23:48:11.692758+0000","last_active":"2024-01-26T23:48:11.698643+0000","last_peered":"2024-01-26T23:48:11.698643+0000","last_clean":"2024-01-26T23:48:11.698643+0000","last_became_active":"2024-01-26T23:48:11.692446+0000","last_became_peered":"2024-01-26T23:48:11.692446+0000","last_unstale":"2024-01-26T23:48:11.698643+0000","last_undegraded":"2024-01-26T23:48:11.698643+0000","last_fullsized":"2024-01-26T23:48:11.698643+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:54:10.694699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685024+0000","last_change":"2024-01-26T23:48:10.813057+0000","last_active":"2024-01-26T23:48:11.685024+0000","last_peered":"2024-01-26T23:48:11.685024+0000","last_clean":"2024-01-26T23:48:11.685024+0000","last_became_active":"2024-01-26T23:48:10.812778+0000","last_became_peered":"2024-01-26T23:48:10.812778+0000","last_unstale":"2024-01-26T23:48:11.685024+0000","last_undegraded":"2024-01-26T23:48:11.685024+0000","last_fullsized":"2024-01-26T23:48:11.685024+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:50:11.320201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702415+0000","last_change":"2024-01-26T23:48:11.696828+0000","last_active":"2024-01-26T23:48:11.702415+0000","last_peered":"2024-01-26T23:48:11.702415+0000","last_clean":"2024-01-26T23:48:11.702415+0000","last_became_active":"2024-01-26T23:48:11.695115+0000","last_became_peered":"2024-01-26T23:48:11.695115+0000","last_unstale":"2024-01-26T23:48:11.702415+0000","last_undegraded":"2024-01-26T23:48:11.702415+0000","last_fullsized":"2024-01-26T23:48:11.702415+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:25.597544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.687089+0000","last_change":"2024-01-26T23:48:10.815435+0000","last_active":"2024-01-26T23:48:11.687089+0000","last_peered":"2024-01-26T23:48:11.687089+0000","last_clean":"2024-01-26T23:48:11.687089+0000","last_became_active":"2024-01-26T23:48:10.815136+0000","last_became_peered":"2024-01-26T23:48:10.815136+0000","last_unstale":"2024-01-26T23:48:11.687089+0000","last_undegraded":"2024-01-26T23:48:11.687089+0000","last_fullsized":"2024-01-26T23:48:11.687089+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:36:28.278118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.704766+0000","last_change":"2024-01-26T23:48:11.699841+0000","last_active":"2024-01-26T23:48:11.704766+0000","last_peered":"2024-01-26T23:48:11.704766+0000","last_clean":"2024-01-26T23:48:11.704766+0000","last_became_active":"2024-01-26T23:48:11.699571+0000","last_became_peered":"2024-01-26T23:48:11.699571+0000","last_unstale":"2024-01-26T23:48:11.704766+0000","last_undegraded":"2024-01-26T23:48:11.704766+0000","last_fullsized":"2024-01-26T23:48:11.704766+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:33:21.325282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.690801+0000","last_change":"2024-01-26T23:48:10.821792+0000","last_active":"2024-01-26T23:48:11.690801+0000","last_peered":"2024-01-26T23:48:11.690801+0000","last_clean":"2024-01-26T23:48:11.690801+0000","last_became_active":"2024-01-26T23:48:10.821453+0000","last_became_peered":"2024-01-26T23:48:10.821453+0000","last_unstale":"2024-01-26T23:48:11.690801+0000","last_undegraded":"2024-01-26T23:48:11.690801+0000","last_fullsized":"2024-01-26T23:48:11.690801+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:59:47.207403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702140+0000","last_change":"2024-01-26T23:48:11.697093+0000","last_active":"2024-01-26T23:48:11.702140+0000","last_peered":"2024-01-26T23:48:11.702140+0000","last_clean":"2024-01-26T23:48:11.702140+0000","last_became_active":"2024-01-26T23:48:11.696468+0000","last_became_peered":"2024-01-26T23:48:11.696468+0000","last_unstale":"2024-01-26T23:48:11.702140+0000","last_undegraded":"2024-01-26T23:48:11.702140+0000","last_fullsized":"2024-01-26T23:48:11.702140+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:58.126224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685337+0000","last_change":"2024-01-26T23:48:10.817416+0000","last_active":"2024-01-26T23:48:11.685337+0000","last_peered":"2024-01-26T23:48:11.685337+0000","last_clean":"2024-01-26T23:48:11.685337+0000","last_became_active":"2024-01-26T23:48:10.817077+0000","last_became_peered":"2024-01-26T23:48:10.817077+0000","last_unstale":"2024-01-26T23:48:11.685337+0000","last_undegraded":"2024-01-26T23:48:11.685337+0000","last_fullsized":"2024-01-26T23:48:11.685337+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:05.190741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.695596+0000","last_change":"2024-01-26T23:48:11.692393+0000","last_active":"2024-01-26T23:48:11.695596+0000","last_peered":"2024-01-26T23:48:11.695596+0000","last_clean":"2024-01-26T23:48:11.695596+0000","last_became_active":"2024-01-26T23:48:11.690501+0000","last_became_peered":"2024-01-26T23:48:11.690501+0000","last_unstale":"2024-01-26T23:48:11.695596+0000","last_undegraded":"2024-01-26T23:48:11.695596+0000","last_fullsized":"2024-01-26T23:48:11.695596+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:51:58.299819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.804097+0000","last_change":"2024-01-26T23:48:10.814833+0000","last_active":"2024-01-26T23:48:11.804097+0000","last_peered":"2024-01-26T23:48:11.804097+0000","last_clean":"2024-01-26T23:48:11.804097+0000","last_became_active":"2024-01-26T23:48:10.814499+0000","last_became_peered":"2024-01-26T23:48:10.814499+0000","last_unstale":"2024-01-26T23:48:11.804097+0000","last_undegraded":"2024-01-26T23:48:11.804097+0000","last_fullsized":"2024-01-26T23:48:11.804097+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:27:02.987035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"120'8562","reported_seq":9017,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.687611+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:48:11.687611+0000","last_peered":"2024-01-26T23:48:11.687611+0000","last_clean":"2024-01-26T23:48:11.687611+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:48:11.687611+0000","last_undegraded":"2024-01-26T23:48:11.687611+0000","last_fullsized":"2024-01-26T23:48:11.687611+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8562,"log_dups_size":0,"ondisk_log_size":8562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":971,"num_read_kb":1902,"num_write":320,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":971,"num_read_kb":1902,"num_write":320,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8562,"ondisk_log_size":8562,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575070,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311736,"kb_used_data":2160,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060104,"statfs":{"total":96636764160,"available":96317546496,"internally_reserved":0,"allocated":2211840,"data_stored":1416893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.51000000000000001,"15min":0.56399999999999995},"min":{"1min":0.34100000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.625},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.52800000000000002,"15min":0.57899999999999996},"min":{"1min":0.379,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.83999999999999997},"last":0.52000000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.53500000000000003,"15min":0.57599999999999996},"min":{"1min":0.40999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.82899999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55800000000000005,"15min":0.59699999999999998},"min":{"1min":0.36199999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.84499999999999997,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.42599999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54600000000000004,"15min":0.57599999999999996},"min":{"1min":0.39600000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76800000000000002,"5min":0.80000000000000004,"15min":0.81699999999999995},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56000000000000005,"15min":0.59199999999999997},"min":{"1min":0.40799999999999997,"5min":0.30199999999999999,"15min":0.27400000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.88600000000000001},"last":0.69399999999999995}]},{"osd":3,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55300000000000005,"15min":0.58799999999999997},"min":{"1min":0.42399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82999999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.59799999999999998},"min":{"1min":0.442,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86399999999999999},"last":0.75600000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.42699999999999999,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.252,"15min":0.247},"max":{"1min":0.58399999999999996,"5min":0.66100000000000003,"15min":0.93899999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.44900000000000001,"15min":0.47399999999999998},"min":{"1min":0.316,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.66200000000000003,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.66200000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47199999999999998,"15min":0.48499999999999999},"min":{"1min":0.33500000000000002,"5min":0.27200000000000002,"15min":0.22800000000000001},"max":{"1min":0.78000000000000003,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.496,"15min":0.51900000000000002},"min":{"1min":0.377,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69499999999999995,"5min":0.74199999999999999,"15min":0.90200000000000002},"last":0.61299999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52500000000000002,"15min":0.54500000000000004},"min":{"1min":0.377,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68500000000000005,"5min":0.78700000000000003,"15min":0.95899999999999996},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52400000000000002,"15min":0.54000000000000004},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.85199999999999998},"last":0.70899999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575070,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302976,"kb_used_data":1592,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068864,"statfs":{"total":96636764160,"available":96326516736,"internally_reserved":0,"allocated":1630208,"data_stored":828477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.53300000000000003,"15min":0.56799999999999995},"min":{"1min":0.379,"5min":0.255,"15min":0.255},"max":{"1min":0.78700000000000003,"5min":1.423,"15min":1.423},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.53800000000000003,"15min":0.58399999999999996},"min":{"1min":0.45000000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.72999999999999998,"5min":0.79000000000000004,"15min":0.91100000000000003},"last":0.68200000000000005}]},{"osd":1,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.52400000000000002,"15min":0.56399999999999995},"min":{"1min":0.39800000000000002,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.76900000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55300000000000005,"15min":0.58999999999999997},"min":{"1min":0.432,"5min":0.312,"15min":0.312},"max":{"1min":0.749,"5min":1.498,"15min":1.498},"last":0.58399999999999996}]},{"osd":2,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53100000000000003,"15min":0.57399999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72699999999999998,"5min":0.74199999999999999,"15min":0.85799999999999998},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53500000000000003,"15min":0.57799999999999996},"min":{"1min":0.39400000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71799999999999997,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.56499999999999995}]},{"osd":3,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57199999999999995,"15min":0.59399999999999997},"min":{"1min":0.37,"5min":0.33800000000000002,"15min":0.29999999999999999},"max":{"1min":0.745,"5min":1.595,"15min":1.595},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.61199999999999999},"min":{"1min":0.39000000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.80300000000000005,"5min":1.369,"15min":1.369},"last":0.751}]},{"osd":4,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.45100000000000001,"15min":0.46000000000000002},"min":{"1min":0.32200000000000001,"5min":0.245,"15min":0.22},"max":{"1min":0.64400000000000002,"5min":1.556,"15min":1.556},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.47499999999999998,"15min":0.48299999999999998},"min":{"1min":0.32000000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65900000000000003,"5min":0.753,"15min":0.86199999999999999},"last":0.78000000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45900000000000002,"15min":0.47299999999999998},"min":{"1min":0.32600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.59699999999999998,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52000000000000002,"15min":0.51900000000000002},"min":{"1min":0.27200000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.72699999999999998,"5min":1.47,"15min":1.47},"last":0.69399999999999995}]},{"osd":7,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.505,"15min":0.51800000000000002},"min":{"1min":0.41299999999999998,"5min":0.26700000000000002,"15min":0.246},"max":{"1min":0.68700000000000006,"5min":0.82199999999999995,"15min":0.85899999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.374,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.67400000000000004,"5min":0.78500000000000003,"15min":0.80400000000000005},"last":0.73599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575070,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307056,"kb_used_data":1576,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064784,"statfs":{"total":96636764160,"available":96322338816,"internally_reserved":0,"allocated":1613824,"data_stored":826399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.41699999999999998,"15min":0.40699999999999997},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61799999999999999,"5min":0.65200000000000002,"15min":0.72599999999999998},"last":0.375},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.442,"15min":0.443},"min":{"1min":0.28999999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.63500000000000001,"5min":1.337,"15min":1.337},"last":0.432}]},{"osd":2,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40400000000000003,"15min":0.41099999999999998},"min":{"1min":0.28000000000000003,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.60199999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.71999999999999997,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.45000000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.51500000000000001},"min":{"1min":0.40000000000000002,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":0.77900000000000003,"5min":1.782,"15min":1.782},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48599999999999999,"15min":0.47999999999999998},"min":{"1min":0.38800000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.69899999999999995,"5min":1.379,"15min":1.379},"last":0.56699999999999995}]},{"osd":4,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.61199999999999999,"15min":0.64100000000000001},"min":{"1min":0.40799999999999997,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":0.73299999999999998,"5min":0.94099999999999995,"15min":1.0820000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63900000000000001,"15min":0.65700000000000003},"min":{"1min":0.443,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.872,"5min":0.89900000000000002,"15min":0.95699999999999996},"last":0.63100000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.65600000000000003,"15min":0.66900000000000004},"min":{"1min":0.501,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":0.68899999999999995,"5min":1.208,"15min":1.208},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.47099999999999997,"5min":0.45100000000000001,"15min":0.42499999999999999},"max":{"1min":0.77700000000000002,"5min":0.96599999999999997,"15min":1.018},"last":0.57799999999999996}]},{"osd":6,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.65900000000000003,"15min":0.67300000000000004},"min":{"1min":0.498,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.78400000000000003,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.73099999999999998,"5min":1.01,"15min":1.022},"last":0.55300000000000005}]},{"osd":7,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.68400000000000005,"15min":0.70599999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.875,"5min":0.98199999999999998,"15min":1.0069999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.70399999999999996,"15min":0.71799999999999997},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.753,"5min":1.296,"15min":1.296},"last":0.64800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575069,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307648,"kb_used_data":2168,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064192,"statfs":{"total":96636764160,"available":96321732608,"internally_reserved":0,"allocated":2220032,"data_stored":1418359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.41699999999999998,"15min":0.40600000000000003},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.50800000000000001,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.44,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.27400000000000002,"15min":0.27300000000000002},"max":{"1min":0.58899999999999997,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.53300000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40600000000000003,"15min":0.42399999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.67600000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45300000000000001,"15min":0.47199999999999998},"min":{"1min":0.34100000000000003,"5min":0.32500000000000001,"15min":0.28000000000000003},"max":{"1min":0.64300000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.39500000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47699999999999998,"15min":0.47199999999999998},"min":{"1min":0.35199999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.623,"5min":2.145,"15min":2.145},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48099999999999998,"15min":0.48799999999999999},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.63100000000000001,"5min":1.198,"15min":1.198},"last":0.55900000000000005}]},{"osd":4,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.621,"15min":0.64800000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.81399999999999995,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.59699999999999998,"15min":0.64300000000000002},"min":{"1min":0.38500000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.72099999999999997,"5min":0.88,"15min":0.98899999999999999},"last":0.78100000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.628,"15min":0.65400000000000003},"min":{"1min":0.45000000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78800000000000003,"5min":1.329,"15min":1.329},"last":0.623},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.66400000000000003,"15min":0.68100000000000005},"min":{"1min":0.47199999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93100000000000005,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.80400000000000005}]},{"osd":6,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.61099999999999999,"15min":0.66100000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80000000000000004,"5min":0.89800000000000002,"15min":1.032},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.65900000000000003,"15min":0.68999999999999995},"min":{"1min":0.48999999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82399999999999995,"5min":1.286,"15min":1.286},"last":0.747}]},{"osd":7,"last update":"Fri Jan 26 23:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":1.012,"15min":0.82699999999999996},"min":{"1min":0.498,"5min":0.435,"15min":0.435},"max":{"1min":0.76900000000000002,"5min":25.965,"15min":25.965},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":1.123,"15min":0.873},"min":{"1min":0.5,"5min":0.48799999999999999,"15min":0.433},"max":{"1min":0.89000000000000001,"5min":33.085000000000001,"15min":33.085000000000001},"last":0.69699999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575069,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307056,"kb_used_data":1576,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064784,"statfs":{"total":96636764160,"available":96322338816,"internally_reserved":0,"allocated":1613824,"data_stored":826399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.39100000000000001},"min":{"1min":0.27200000000000002,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.434}]},{"osd":1,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.42899999999999999,"15min":0.42799999999999999},"min":{"1min":0.27400000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.57999999999999996,"5min":0.61899999999999999,"15min":0.70499999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.50900000000000001,"15min":0.50600000000000001},"min":{"1min":0.379,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.51000000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46300000000000002,"15min":0.48599999999999999},"min":{"1min":0.32500000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.74099999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47899999999999998,"15min":0.47599999999999998},"min":{"1min":0.33500000000000002,"5min":0.32200000000000001,"15min":0.307},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.47799999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.60099999999999998,"15min":0.629},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.75800000000000001,"5min":1.087,"15min":1.087},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.628,"15min":0.64000000000000001},"min":{"1min":0.374,"5min":0.371,"15min":0.371},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.70599999999999996}]},{"osd":5,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63600000000000001,"15min":0.65400000000000003},"min":{"1min":0.44500000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.65000000000000002,"15min":0.67400000000000004},"min":{"1min":0.48199999999999998,"5min":0.39800000000000002,"15min":0.35199999999999998},"max":{"1min":0.80600000000000005,"5min":0.97699999999999998,"15min":1.085},"last":0.56000000000000005}]},{"osd":6,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65900000000000003,"15min":0.67800000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89600000000000002,"5min":1.022,"15min":1.1080000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.65700000000000003,"15min":0.67900000000000005},"min":{"1min":0.49099999999999999,"5min":0.42799999999999999,"15min":0.38800000000000001},"max":{"1min":0.88,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.58299999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68400000000000005,"15min":0.70199999999999996},"min":{"1min":0.53100000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.69799999999999995,"15min":0.72899999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.42099999999999999},"max":{"1min":0.84199999999999997,"5min":1.052,"15min":1.139},"last":0.61899999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575068,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307056,"kb_used_data":1576,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064784,"statfs":{"total":96636764160,"available":96322338816,"internally_reserved":0,"allocated":1613824,"data_stored":826919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.38200000000000001,"15min":0.39100000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":1.8540000000000001},"last":0.315},{"interface":"front","average":{"1min":0.441,"5min":0.42099999999999999,"15min":0.44},"min":{"1min":0.30599999999999999,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":1.2290000000000001},"last":0.29799999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44500000000000001,"15min":0.45400000000000001},"min":{"1min":0.33100000000000002,"5min":0.27900000000000003,"15min":0.253},"max":{"1min":0.63200000000000001,"5min":0.63800000000000001,"15min":1.633},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.48099999999999998,"15min":0.48499999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":0.65900000000000003,"5min":0.67800000000000005,"15min":1.0760000000000001},"last":0.36499999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.47399999999999998,"15min":0.47999999999999998},"min":{"1min":0.31900000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68600000000000005,"5min":0.70399999999999996,"15min":1.4059999999999999},"last":0.378},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.439,"15min":0.45900000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28299999999999997},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":1.1870000000000001},"last":0.35399999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.61699999999999999,"15min":0.65800000000000003},"min":{"1min":0.45400000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.86799999999999999,"5min":0.878,"15min":3.306},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.628,"15min":0.65700000000000003},"min":{"1min":0.50700000000000001,"5min":0.40100000000000002,"15min":0.379},"max":{"1min":0.83899999999999997,"5min":0.85199999999999998,"15min":1.008},"last":0.73599999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.62,"15min":0.65300000000000002},"min":{"1min":0.49199999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":1.593},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63500000000000001,"15min":0.66600000000000004},"min":{"1min":0.499,"5min":0.379,"15min":0.379},"max":{"1min":0.79500000000000004,"5min":0.92300000000000004,"15min":1.3819999999999999},"last":0.71099999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63400000000000001,"15min":0.66900000000000004},"min":{"1min":0.49099999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85599999999999998,"5min":0.88800000000000001,"15min":1.506},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.66000000000000003,"15min":0.68799999999999994},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.44400000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":1.427},"last":0.65300000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.63700000000000001,"15min":0.68799999999999994},"min":{"1min":0.442,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":3.238},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.69699999999999995,"15min":0.73299999999999998},"min":{"1min":0.55000000000000004,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":3.351},"last":0.54200000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575070,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302964,"kb_used_data":1580,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068876,"statfs":{"total":96636764160,"available":96326529024,"internally_reserved":0,"allocated":1617920,"data_stored":826457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53400000000000003,"15min":0.57099999999999995},"min":{"1min":0.34699999999999998,"5min":0.28999999999999998,"15min":0.28699999999999998},"max":{"1min":0.71199999999999997,"5min":1.21,"15min":1.21},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.35999999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.76400000000000001,"5min":5.367,"15min":5.367},"last":0.42799999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55700000000000005,"15min":0.58899999999999997},"min":{"1min":0.38800000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.88},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.53400000000000003,"15min":0.57699999999999996},"min":{"1min":0.35199999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.621,"5min":0.80600000000000005,"15min":0.81699999999999995},"last":0.52800000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52700000000000002,"15min":0.56699999999999995},"min":{"1min":0.33900000000000002,"5min":0.33300000000000002,"15min":0.32700000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.57199999999999995,"15min":0.59699999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80500000000000005,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.73199999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55100000000000005,"15min":0.58399999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.68500000000000005,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56200000000000006,"15min":0.58299999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.35699999999999998},"max":{"1min":0.94199999999999995,"5min":1.05,"15min":1.05},"last":0.499}]},{"osd":5,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.44400000000000001,"15min":0.45800000000000002},"min":{"1min":0.29199999999999998,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.68300000000000005,"5min":0.76200000000000001,"15min":0.77600000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.441,"5min":0.47499999999999998,"15min":0.47699999999999998},"min":{"1min":0.29099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72499999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.63900000000000001}]},{"osd":6,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45200000000000001,"15min":0.46700000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27200000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.78900000000000003,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.76900000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.51400000000000001,"15min":0.53400000000000003},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.63700000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.5,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.36899999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.58999999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575070,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298860,"kb_used_data":1572,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072980,"statfs":{"total":96636764160,"available":96330731520,"internally_reserved":0,"allocated":1609728,"data_stored":826365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.54500000000000004,"15min":0.56699999999999995},"min":{"1min":0.43099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.81799999999999995,"5min":0.92800000000000005,"15min":2.0289999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53000000000000003,"15min":0.56899999999999995},"min":{"1min":0.35799999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77100000000000002,"5min":0.77400000000000002,"15min":0.82199999999999995},"last":0.55700000000000005}]},{"osd":1,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57599999999999996,"15min":0.60799999999999998},"min":{"1min":0.33800000000000002,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":0.78000000000000003,"5min":0.85999999999999999,"15min":2.3319999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60599999999999998,"15min":0.627},"min":{"1min":0.47199999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80300000000000005,"5min":0.88500000000000001,"15min":1.1739999999999999},"last":0.59399999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.56100000000000005},"min":{"1min":0.438,"5min":0.34499999999999997,"15min":0.30099999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.83799999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55600000000000005,"15min":0.58699999999999997},"min":{"1min":0.38,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.80000000000000004,"5min":0.89700000000000002,"15min":2.0600000000000001},"last":0.498}]},{"osd":3,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56399999999999995,"15min":0.57699999999999996},"min":{"1min":0.40899999999999997,"5min":0.34999999999999998,"15min":0.314},"max":{"1min":0.68400000000000005,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56499999999999995,"15min":0.58699999999999997},"min":{"1min":0.39800000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.751,"5min":0.77900000000000003,"15min":2.5630000000000002},"last":0.57299999999999995}]},{"osd":4,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.42699999999999999,"15min":0.45300000000000001},"min":{"1min":0.30499999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64000000000000001,"5min":0.73099999999999998,"15min":2.3700000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.317,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.89200000000000002},"last":0.36799999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.47299999999999998},"min":{"1min":0.33100000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":2.5990000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.49299999999999999,"15min":0.51300000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70299999999999996,"5min":0.76200000000000001,"15min":1.99},"last":0.40699999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52400000000000002,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73499999999999999,"5min":0.76200000000000001,"15min":3.5270000000000001},"last":0.435},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54200000000000004,"15min":0.55800000000000005},"min":{"1min":0.33500000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.747,"5min":0.81499999999999995,"15min":0.82199999999999995},"last":0.33500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:48:20.695 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:48:20.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.835+0000 7eff722fc700 1 Processor -- start 2024-01-26T23:48:20.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.835+0000 7eff722fc700 1 -- start start 2024-01-26T23:48:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.835+0000 7eff722fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff6c157340 0x7eff6c15bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.835+0000 7eff722fc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7eff6c0b4180 0x7eff6c157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.836+0000 7eff722fc700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff6c158ae0 con 0x7eff6c0ae520 2024-01-26T23:48:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.836+0000 7eff722fc700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff6c15b650 con 0x7eff6c157340 2024-01-26T23:48:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.836+0000 7eff722fc700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff6c15c020 con 0x7eff6c0b4180 2024-01-26T23:48:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.836+0000 7eff6bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff6c157340 0x7eff6c15bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.836+0000 7eff6bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff6c157340 0x7eff6c15bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48500/0 (socket says 172.21.15.46:48500) 2024-01-26T23:48:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.836+0000 7eff6bfff700 1 -- 172.21.15.46:0/2074837202 learned_addr learned my addr 172.21.15.46:0/2074837202 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.836+0000 7eff70899700 1 --2- 172.21.15.46:0/2074837202 >> v2:172.21.15.106:3300/0 conn(0x7eff6c0b4180 0x7eff6c157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.836+0000 7eff6bfff700 1 -- 172.21.15.46:0/2074837202 >> v2:172.21.15.106:3300/0 conn(0x7eff6c0b4180 msgr2=0x7eff6c157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.836+0000 7eff6bfff700 1 --2- 172.21.15.46:0/2074837202 >> v2:172.21.15.106:3300/0 conn(0x7eff6c0b4180 0x7eff6c157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.836+0000 7eff6bfff700 1 -- 172.21.15.46:0/2074837202 >> v1:172.21.15.106:6789/0 conn(0x7eff6c0ae520 legacy=0x7eff6c0b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:20.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.836+0000 7eff6bfff700 1 -- 172.21.15.46:0/2074837202 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff6c158520 con 0x7eff6c157340 2024-01-26T23:48:20.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.836+0000 7eff70899700 1 --2- 172.21.15.46:0/2074837202 >> v2:172.21.15.106:3300/0 conn(0x7eff6c0b4180 0x7eff6c157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:20.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.836+0000 7eff6bfff700 1 --2- 172.21.15.46:0/2074837202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff6c157340 0x7eff6c15bae0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0caeb0 tx=0x7eff5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa7178107562d2f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:20.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.836+0000 7eff6affd700 1 -- 172.21.15.46:0/2074837202 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c00b860 con 0x7eff6c157340 2024-01-26T23:48:20.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.836+0000 7eff6affd700 1 -- 172.21.15.46:0/2074837202 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff5c00b9c0 con 0x7eff6c157340 2024-01-26T23:48:20.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.836+0000 7eff6affd700 1 -- 172.21.15.46:0/2074837202 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c00bd80 con 0x7eff6c157340 2024-01-26T23:48:20.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.837+0000 7eff722fc700 1 -- 172.21.15.46:0/2074837202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff6c157340 msgr2=0x7eff6c15bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:20.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.837+0000 7eff722fc700 1 --2- 172.21.15.46:0/2074837202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff6c157340 0x7eff6c15bae0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0caeb0 tx=0x7eff5c009250 comp rx=0 tx=0).stop 2024-01-26T23:48:20.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.837+0000 7eff722fc700 1 -- 172.21.15.46:0/2074837202 shutdown_connections 2024-01-26T23:48:20.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.837+0000 7eff722fc700 1 --2- 172.21.15.46:0/2074837202 >> v2:172.21.15.106:3300/0 conn(0x7eff6c0b4180 0x7eff6c157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:20.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.837+0000 7eff722fc700 1 --2- 172.21.15.46:0/2074837202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff6c157340 0x7eff6c15bae0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:20.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.837+0000 7eff722fc700 1 -- 172.21.15.46:0/2074837202 >> 172.21.15.46:0/2074837202 conn(0x7eff6c0ce530 msgr2=0x7eff6c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:20.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.837+0000 7eff722fc700 1 -- 172.21.15.46:0/2074837202 shutdown_connections 2024-01-26T23:48:20.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.837+0000 7eff722fc700 1 -- 172.21.15.46:0/2074837202 wait complete. 2024-01-26T23:48:20.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.838+0000 7eff722fc700 1 Processor -- start 2024-01-26T23:48:20.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.839+0000 7eff722fc700 1 -- start start 2024-01-26T23:48:20.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.839+0000 7eff722fc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff6c0ae520 0x7eff6c0c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:20.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.839+0000 7eff722fc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff6c0b4180 0x7eff6c0c4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.839+0000 7eff722fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff6c157340 0x7eff6c169eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.839+0000 7eff722fc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c0c24d0 con 0x7eff6c157340 2024-01-26T23:48:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.839+0000 7eff722fc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c068e10 con 0x7eff6c0b4180 2024-01-26T23:48:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.839+0000 7eff722fc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c068ae0 con 0x7eff6c0ae520 2024-01-26T23:48:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.839+0000 7eff70899700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff6c157340 0x7eff6c169eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.839+0000 7eff70899700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff6c157340 0x7eff6c169eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48504/0 (socket says 172.21.15.46:48504) 2024-01-26T23:48:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.839+0000 7eff70899700 1 -- 172.21.15.46:0/2641864424 learned_addr learned my addr 172.21.15.46:0/2641864424 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.840+0000 7eff6bfff700 1 --2- 172.21.15.46:0/2641864424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff6c0ae520 0x7eff6c0c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.839+0000 7eff6b7fe700 1 --2- 172.21.15.46:0/2641864424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff6c0b4180 0x7eff6c0c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.840+0000 7eff70899700 1 -- 172.21.15.46:0/2641864424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff6c0ae520 msgr2=0x7eff6c0c1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.840+0000 7eff70899700 1 --2- 172.21.15.46:0/2641864424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff6c0ae520 0x7eff6c0c1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.840+0000 7eff70899700 1 -- 172.21.15.46:0/2641864424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff6c0b4180 msgr2=0x7eff6c0c4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.840+0000 7eff70899700 1 --2- 172.21.15.46:0/2641864424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff6c0b4180 0x7eff6c0c4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.840+0000 7eff70899700 1 -- 172.21.15.46:0/2641864424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c0094f0 con 0x7eff6c157340 2024-01-26T23:48:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.840+0000 7eff70899700 1 --2- 172.21.15.46:0/2641864424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff6c157340 0x7eff6c169eb0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7eff6400d100 tx=0x7eff640099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.840+0000 7eff53fff700 1 -- 172.21.15.46:0/2641864424 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6400dc40 con 0x7eff6c157340 2024-01-26T23:48:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.840+0000 7eff53fff700 1 -- 172.21.15.46:0/2641864424 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6400f040 con 0x7eff6c157340 2024-01-26T23:48:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.840+0000 7eff53fff700 1 -- 172.21.15.46:0/2641864424 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff64015440 con 0x7eff6c157340 2024-01-26T23:48:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.840+0000 7eff722fc700 1 -- 172.21.15.46:0/2641864424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff6c0c26e0 con 0x7eff6c157340 2024-01-26T23:48:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.840+0000 7eff722fc700 1 -- 172.21.15.46:0/2641864424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6c0c2ca0 con 0x7eff6c157340 2024-01-26T23:48:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.841+0000 7eff722fc700 1 -- 172.21.15.46:0/2641864424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff6c0cacf0 con 0x7eff6c157340 2024-01-26T23:48:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.842+0000 7eff53fff700 1 -- 172.21.15.46:0/2641864424 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7eff6401e3b0 con 0x7eff6c157340 2024-01-26T23:48:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.842+0000 7eff53fff700 1 --2- 172.21.15.46:0/2641864424 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff58066f30 0x7eff580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.842+0000 7eff53fff700 1 -- 172.21.15.46:0/2641864424 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7eff64084c10 con 0x7eff6c157340 2024-01-26T23:48:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.842+0000 7eff6bfff700 1 --2- 172.21.15.46:0/2641864424 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff58066f30 0x7eff580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:20.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.843+0000 7eff6bfff700 1 --2- 172.21.15.46:0/2641864424 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff58066f30 0x7eff580693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0caeb0 tx=0x7eff5c017000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:20.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:20.847+0000 7eff53fff700 1 -- 172.21.15.46:0/2641864424 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7eff64051bb0 con 0x7eff6c157340 2024-01-26T23:48:21.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.049+0000 7eff722fc700 1 -- 172.21.15.46:0/2641864424 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7eff6c06e680 con 0x7eff58066f30 2024-01-26T23:48:21.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.058+0000 7eff53fff700 1 -- 172.21.15.46:0/2641864424 <== mgr.7396 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289608 (secure 0 0 0) 0x7eff6c06e680 con 0x7eff58066f30 2024-01-26T23:48:21.062 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:48:21.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.061+0000 7eff722fc700 1 -- 172.21.15.46:0/2641864424 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff58066f30 msgr2=0x7eff580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:21.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.061+0000 7eff722fc700 1 --2- 172.21.15.46:0/2641864424 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff58066f30 0x7eff580693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0caeb0 tx=0x7eff5c017000 comp rx=0 tx=0).stop 2024-01-26T23:48:21.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.061+0000 7eff722fc700 1 -- 172.21.15.46:0/2641864424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff6c157340 msgr2=0x7eff6c169eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:21.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.061+0000 7eff722fc700 1 --2- 172.21.15.46:0/2641864424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff6c157340 0x7eff6c169eb0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7eff6400d100 tx=0x7eff640099b0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.061+0000 7eff722fc700 1 -- 172.21.15.46:0/2641864424 shutdown_connections 2024-01-26T23:48:21.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.061+0000 7eff722fc700 1 --2- 172.21.15.46:0/2641864424 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff58066f30 0x7eff580693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.061+0000 7eff722fc700 1 --2- 172.21.15.46:0/2641864424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff6c157340 0x7eff6c169eb0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.061+0000 7eff722fc700 1 --2- 172.21.15.46:0/2641864424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff6c0ae520 0x7eff6c0c1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.061+0000 7eff722fc700 1 --2- 172.21.15.46:0/2641864424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff6c0b4180 0x7eff6c0c4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.061+0000 7eff722fc700 1 -- 172.21.15.46:0/2641864424 >> 172.21.15.46:0/2641864424 conn(0x7eff6c0ce530 msgr2=0x7eff6c0b0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:21.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.061+0000 7eff722fc700 1 -- 172.21.15.46:0/2641864424 shutdown_connections 2024-01-26T23:48:21.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.061+0000 7eff722fc700 1 -- 172.21.15.46:0/2641864424 wait complete. 2024-01-26T23:48:21.066 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:48:21.076 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-01-26T23:48:20.717839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":971,"num_read_kb":1902,"num_write":365,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8601,"ondisk_log_size":8601,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2445352,"kb_used_data":13800,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752529368,"statfs":{"total":773094113280,"available":770590072832,"internally_reserved":0,"allocated":14131200,"data_stored":7796268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.041033"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693494+0000","last_change":"2024-01-26T23:48:11.690618+0000","last_active":"2024-01-26T23:48:11.693494+0000","last_peered":"2024-01-26T23:48:11.693494+0000","last_clean":"2024-01-26T23:48:11.693494+0000","last_became_active":"2024-01-26T23:48:11.690305+0000","last_became_peered":"2024-01-26T23:48:11.690305+0000","last_unstale":"2024-01-26T23:48:11.693494+0000","last_undegraded":"2024-01-26T23:48:11.693494+0000","last_fullsized":"2024-01-26T23:48:11.693494+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:17:24.039600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.700908+0000","last_change":"2024-01-26T23:48:11.695729+0000","last_active":"2024-01-26T23:48:11.700908+0000","last_peered":"2024-01-26T23:48:11.700908+0000","last_clean":"2024-01-26T23:48:11.700908+0000","last_became_active":"2024-01-26T23:48:11.694867+0000","last_became_peered":"2024-01-26T23:48:11.694867+0000","last_unstale":"2024-01-26T23:48:11.700908+0000","last_undegraded":"2024-01-26T23:48:11.700908+0000","last_fullsized":"2024-01-26T23:48:11.700908+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:22:27.593650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693576+0000","last_change":"2024-01-26T23:48:11.691344+0000","last_active":"2024-01-26T23:48:11.693576+0000","last_peered":"2024-01-26T23:48:11.693576+0000","last_clean":"2024-01-26T23:48:11.693576+0000","last_became_active":"2024-01-26T23:48:11.691069+0000","last_became_peered":"2024-01-26T23:48:11.691069+0000","last_unstale":"2024-01-26T23:48:11.693576+0000","last_undegraded":"2024-01-26T23:48:11.693576+0000","last_fullsized":"2024-01-26T23:48:11.693576+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:30.893321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698369+0000","last_change":"2024-01-26T23:48:11.694681+0000","last_active":"2024-01-26T23:48:11.698369+0000","last_peered":"2024-01-26T23:48:11.698369+0000","last_clean":"2024-01-26T23:48:11.698369+0000","last_became_active":"2024-01-26T23:48:11.693694+0000","last_became_peered":"2024-01-26T23:48:11.693694+0000","last_unstale":"2024-01-26T23:48:11.698369+0000","last_undegraded":"2024-01-26T23:48:11.698369+0000","last_fullsized":"2024-01-26T23:48:11.698369+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:10:51.561423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.689920+0000","last_change":"2024-01-26T23:48:11.688049+0000","last_active":"2024-01-26T23:48:11.689920+0000","last_peered":"2024-01-26T23:48:11.689920+0000","last_clean":"2024-01-26T23:48:11.689920+0000","last_became_active":"2024-01-26T23:48:11.687759+0000","last_became_peered":"2024-01-26T23:48:11.687759+0000","last_unstale":"2024-01-26T23:48:11.689920+0000","last_undegraded":"2024-01-26T23:48:11.689920+0000","last_fullsized":"2024-01-26T23:48:11.689920+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:08:43.039916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.700721+0000","last_change":"2024-01-26T23:48:11.694415+0000","last_active":"2024-01-26T23:48:11.700721+0000","last_peered":"2024-01-26T23:48:11.700721+0000","last_clean":"2024-01-26T23:48:11.700721+0000","last_became_active":"2024-01-26T23:48:11.694105+0000","last_became_peered":"2024-01-26T23:48:11.694105+0000","last_unstale":"2024-01-26T23:48:11.700721+0000","last_undegraded":"2024-01-26T23:48:11.700721+0000","last_fullsized":"2024-01-26T23:48:11.700721+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:24:22.377854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.707410+0000","last_change":"2024-01-26T23:48:11.702892+0000","last_active":"2024-01-26T23:48:11.707410+0000","last_peered":"2024-01-26T23:48:11.707410+0000","last_clean":"2024-01-26T23:48:11.707410+0000","last_became_active":"2024-01-26T23:48:11.702664+0000","last_became_peered":"2024-01-26T23:48:11.702664+0000","last_unstale":"2024-01-26T23:48:11.707410+0000","last_undegraded":"2024-01-26T23:48:11.707410+0000","last_fullsized":"2024-01-26T23:48:11.707410+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:25:47.188502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702212+0000","last_change":"2024-01-26T23:48:11.695520+0000","last_active":"2024-01-26T23:48:11.702212+0000","last_peered":"2024-01-26T23:48:11.702212+0000","last_clean":"2024-01-26T23:48:11.702212+0000","last_became_active":"2024-01-26T23:48:11.695205+0000","last_became_peered":"2024-01-26T23:48:11.695205+0000","last_unstale":"2024-01-26T23:48:11.702212+0000","last_undegraded":"2024-01-26T23:48:11.702212+0000","last_fullsized":"2024-01-26T23:48:11.702212+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:36:53.012187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.700678+0000","last_change":"2024-01-26T23:48:11.694658+0000","last_active":"2024-01-26T23:48:11.700678+0000","last_peered":"2024-01-26T23:48:11.700678+0000","last_clean":"2024-01-26T23:48:11.700678+0000","last_became_active":"2024-01-26T23:48:11.693934+0000","last_became_peered":"2024-01-26T23:48:11.693934+0000","last_unstale":"2024-01-26T23:48:11.700678+0000","last_undegraded":"2024-01-26T23:48:11.700678+0000","last_fullsized":"2024-01-26T23:48:11.700678+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:56:27.498369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.703088+0000","last_change":"2024-01-26T23:48:11.697158+0000","last_active":"2024-01-26T23:48:11.703088+0000","last_peered":"2024-01-26T23:48:11.703088+0000","last_clean":"2024-01-26T23:48:11.703088+0000","last_became_active":"2024-01-26T23:48:11.696633+0000","last_became_peered":"2024-01-26T23:48:11.696633+0000","last_unstale":"2024-01-26T23:48:11.703088+0000","last_undegraded":"2024-01-26T23:48:11.703088+0000","last_fullsized":"2024-01-26T23:48:11.703088+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:23:52.601066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702118+0000","last_change":"2024-01-26T23:48:11.697639+0000","last_active":"2024-01-26T23:48:11.702118+0000","last_peered":"2024-01-26T23:48:11.702118+0000","last_clean":"2024-01-26T23:48:11.702118+0000","last_became_active":"2024-01-26T23:48:11.697351+0000","last_became_peered":"2024-01-26T23:48:11.697351+0000","last_unstale":"2024-01-26T23:48:11.702118+0000","last_undegraded":"2024-01-26T23:48:11.702118+0000","last_fullsized":"2024-01-26T23:48:11.702118+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:45:48.834930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693709+0000","last_change":"2024-01-26T23:48:11.692041+0000","last_active":"2024-01-26T23:48:11.693709+0000","last_peered":"2024-01-26T23:48:11.693709+0000","last_clean":"2024-01-26T23:48:11.693709+0000","last_became_active":"2024-01-26T23:48:11.691748+0000","last_became_peered":"2024-01-26T23:48:11.691748+0000","last_unstale":"2024-01-26T23:48:11.693709+0000","last_undegraded":"2024-01-26T23:48:11.693709+0000","last_fullsized":"2024-01-26T23:48:11.693709+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:02:31.050232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.691654+0000","last_change":"2024-01-26T23:48:11.689915+0000","last_active":"2024-01-26T23:48:11.691654+0000","last_peered":"2024-01-26T23:48:11.691654+0000","last_clean":"2024-01-26T23:48:11.691654+0000","last_became_active":"2024-01-26T23:48:11.689529+0000","last_became_peered":"2024-01-26T23:48:11.689529+0000","last_unstale":"2024-01-26T23:48:11.691654+0000","last_undegraded":"2024-01-26T23:48:11.691654+0000","last_fullsized":"2024-01-26T23:48:11.691654+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:10.141932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.704794+0000","last_change":"2024-01-26T23:48:11.700345+0000","last_active":"2024-01-26T23:48:11.704794+0000","last_peered":"2024-01-26T23:48:11.704794+0000","last_clean":"2024-01-26T23:48:11.704794+0000","last_became_active":"2024-01-26T23:48:11.700057+0000","last_became_peered":"2024-01-26T23:48:11.700057+0000","last_unstale":"2024-01-26T23:48:11.704794+0000","last_undegraded":"2024-01-26T23:48:11.704794+0000","last_fullsized":"2024-01-26T23:48:11.704794+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:35:53.885389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.704742+0000","last_change":"2024-01-26T23:48:11.699805+0000","last_active":"2024-01-26T23:48:11.704742+0000","last_peered":"2024-01-26T23:48:11.704742+0000","last_clean":"2024-01-26T23:48:11.704742+0000","last_became_active":"2024-01-26T23:48:11.699526+0000","last_became_peered":"2024-01-26T23:48:11.699526+0000","last_unstale":"2024-01-26T23:48:11.704742+0000","last_undegraded":"2024-01-26T23:48:11.704742+0000","last_fullsized":"2024-01-26T23:48:11.704742+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:10.141932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.704822+0000","last_change":"2024-01-26T23:48:11.700542+0000","last_active":"2024-01-26T23:48:11.704822+0000","last_peered":"2024-01-26T23:48:11.704822+0000","last_clean":"2024-01-26T23:48:11.704822+0000","last_became_active":"2024-01-26T23:48:11.700277+0000","last_became_peered":"2024-01-26T23:48:11.700277+0000","last_unstale":"2024-01-26T23:48:11.704822+0000","last_undegraded":"2024-01-26T23:48:11.704822+0000","last_fullsized":"2024-01-26T23:48:11.704822+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:36:36.750288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693369+0000","last_change":"2024-01-26T23:48:11.686277+0000","last_active":"2024-01-26T23:48:11.693369+0000","last_peered":"2024-01-26T23:48:11.693369+0000","last_clean":"2024-01-26T23:48:11.693369+0000","last_became_active":"2024-01-26T23:48:11.686006+0000","last_became_peered":"2024-01-26T23:48:11.686006+0000","last_unstale":"2024-01-26T23:48:11.693369+0000","last_undegraded":"2024-01-26T23:48:11.693369+0000","last_fullsized":"2024-01-26T23:48:11.693369+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:22:19.028290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.700889+0000","last_change":"2024-01-26T23:48:11.694755+0000","last_active":"2024-01-26T23:48:11.700889+0000","last_peered":"2024-01-26T23:48:11.700889+0000","last_clean":"2024-01-26T23:48:11.700889+0000","last_became_active":"2024-01-26T23:48:11.694175+0000","last_became_peered":"2024-01-26T23:48:11.694175+0000","last_unstale":"2024-01-26T23:48:11.700889+0000","last_undegraded":"2024-01-26T23:48:11.700889+0000","last_fullsized":"2024-01-26T23:48:11.700889+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:55:06.345115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698855+0000","last_change":"2024-01-26T23:48:11.696828+0000","last_active":"2024-01-26T23:48:11.698855+0000","last_peered":"2024-01-26T23:48:11.698855+0000","last_clean":"2024-01-26T23:48:11.698855+0000","last_became_active":"2024-01-26T23:48:11.696291+0000","last_became_peered":"2024-01-26T23:48:11.696291+0000","last_unstale":"2024-01-26T23:48:11.698855+0000","last_undegraded":"2024-01-26T23:48:11.698855+0000","last_fullsized":"2024-01-26T23:48:11.698855+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:27:53.675503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.705011+0000","last_change":"2024-01-26T23:48:11.701106+0000","last_active":"2024-01-26T23:48:11.705011+0000","last_peered":"2024-01-26T23:48:11.705011+0000","last_clean":"2024-01-26T23:48:11.705011+0000","last_became_active":"2024-01-26T23:48:11.700818+0000","last_became_peered":"2024-01-26T23:48:11.700818+0000","last_unstale":"2024-01-26T23:48:11.705011+0000","last_undegraded":"2024-01-26T23:48:11.705011+0000","last_fullsized":"2024-01-26T23:48:11.705011+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:53:43.709494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.690830+0000","last_change":"2024-01-26T23:48:11.689380+0000","last_active":"2024-01-26T23:48:11.690830+0000","last_peered":"2024-01-26T23:48:11.690830+0000","last_clean":"2024-01-26T23:48:11.690830+0000","last_became_active":"2024-01-26T23:48:11.688997+0000","last_became_peered":"2024-01-26T23:48:11.688997+0000","last_unstale":"2024-01-26T23:48:11.690830+0000","last_undegraded":"2024-01-26T23:48:11.690830+0000","last_fullsized":"2024-01-26T23:48:11.690830+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:20.796536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.687203+0000","last_change":"2024-01-26T23:48:11.681763+0000","last_active":"2024-01-26T23:48:11.687203+0000","last_peered":"2024-01-26T23:48:11.687203+0000","last_clean":"2024-01-26T23:48:11.687203+0000","last_became_active":"2024-01-26T23:48:11.681378+0000","last_became_peered":"2024-01-26T23:48:11.681378+0000","last_unstale":"2024-01-26T23:48:11.687203+0000","last_undegraded":"2024-01-26T23:48:11.687203+0000","last_fullsized":"2024-01-26T23:48:11.687203+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:33:21.325282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693341+0000","last_change":"2024-01-26T23:48:11.685918+0000","last_active":"2024-01-26T23:48:11.693341+0000","last_peered":"2024-01-26T23:48:11.693341+0000","last_clean":"2024-01-26T23:48:11.693341+0000","last_became_active":"2024-01-26T23:48:11.685566+0000","last_became_peered":"2024-01-26T23:48:11.685566+0000","last_unstale":"2024-01-26T23:48:11.693341+0000","last_undegraded":"2024-01-26T23:48:11.693341+0000","last_fullsized":"2024-01-26T23:48:11.693341+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:06:45.718179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.684940+0000","last_change":"2024-01-26T23:48:11.682824+0000","last_active":"2024-01-26T23:48:11.684940+0000","last_peered":"2024-01-26T23:48:11.684940+0000","last_clean":"2024-01-26T23:48:11.684940+0000","last_became_active":"2024-01-26T23:48:11.682545+0000","last_became_peered":"2024-01-26T23:48:11.682545+0000","last_unstale":"2024-01-26T23:48:11.684940+0000","last_undegraded":"2024-01-26T23:48:11.684940+0000","last_fullsized":"2024-01-26T23:48:11.684940+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:50:50.668201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.701967+0000","last_change":"2024-01-26T23:48:11.696001+0000","last_active":"2024-01-26T23:48:11.701967+0000","last_peered":"2024-01-26T23:48:11.701967+0000","last_clean":"2024-01-26T23:48:11.701967+0000","last_became_active":"2024-01-26T23:48:11.695692+0000","last_became_peered":"2024-01-26T23:48:11.695692+0000","last_unstale":"2024-01-26T23:48:11.701967+0000","last_undegraded":"2024-01-26T23:48:11.701967+0000","last_fullsized":"2024-01-26T23:48:11.701967+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:25:59.436650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702281+0000","last_change":"2024-01-26T23:48:11.696416+0000","last_active":"2024-01-26T23:48:11.702281+0000","last_peered":"2024-01-26T23:48:11.702281+0000","last_clean":"2024-01-26T23:48:11.702281+0000","last_became_active":"2024-01-26T23:48:11.696137+0000","last_became_peered":"2024-01-26T23:48:11.696137+0000","last_unstale":"2024-01-26T23:48:11.702281+0000","last_undegraded":"2024-01-26T23:48:11.702281+0000","last_fullsized":"2024-01-26T23:48:11.702281+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:37:36.865287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.704679+0000","last_change":"2024-01-26T23:48:11.700455+0000","last_active":"2024-01-26T23:48:11.704679+0000","last_peered":"2024-01-26T23:48:11.704679+0000","last_clean":"2024-01-26T23:48:11.704679+0000","last_became_active":"2024-01-26T23:48:11.700192+0000","last_became_peered":"2024-01-26T23:48:11.700192+0000","last_unstale":"2024-01-26T23:48:11.704679+0000","last_undegraded":"2024-01-26T23:48:11.704679+0000","last_fullsized":"2024-01-26T23:48:11.704679+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:27.902905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702235+0000","last_change":"2024-01-26T23:48:11.698782+0000","last_active":"2024-01-26T23:48:11.702235+0000","last_peered":"2024-01-26T23:48:11.702235+0000","last_clean":"2024-01-26T23:48:11.702235+0000","last_became_active":"2024-01-26T23:48:11.698476+0000","last_became_peered":"2024-01-26T23:48:11.698476+0000","last_unstale":"2024-01-26T23:48:11.702235+0000","last_undegraded":"2024-01-26T23:48:11.702235+0000","last_fullsized":"2024-01-26T23:48:11.702235+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:27:13.343954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702333+0000","last_change":"2024-01-26T23:48:11.700735+0000","last_active":"2024-01-26T23:48:11.702333+0000","last_peered":"2024-01-26T23:48:11.702333+0000","last_clean":"2024-01-26T23:48:11.702333+0000","last_became_active":"2024-01-26T23:48:11.700416+0000","last_became_peered":"2024-01-26T23:48:11.700416+0000","last_unstale":"2024-01-26T23:48:11.702333+0000","last_undegraded":"2024-01-26T23:48:11.702333+0000","last_fullsized":"2024-01-26T23:48:11.702333+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:03:16.363445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699270+0000","last_change":"2024-01-26T23:48:11.693380+0000","last_active":"2024-01-26T23:48:11.699270+0000","last_peered":"2024-01-26T23:48:11.699270+0000","last_clean":"2024-01-26T23:48:11.699270+0000","last_became_active":"2024-01-26T23:48:11.692986+0000","last_became_peered":"2024-01-26T23:48:11.692986+0000","last_unstale":"2024-01-26T23:48:11.699270+0000","last_undegraded":"2024-01-26T23:48:11.699270+0000","last_fullsized":"2024-01-26T23:48:11.699270+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:37:57.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699950+0000","last_change":"2024-01-26T23:48:11.695328+0000","last_active":"2024-01-26T23:48:11.699950+0000","last_peered":"2024-01-26T23:48:11.699950+0000","last_clean":"2024-01-26T23:48:11.699950+0000","last_became_active":"2024-01-26T23:48:11.694749+0000","last_became_peered":"2024-01-26T23:48:11.694749+0000","last_unstale":"2024-01-26T23:48:11.699950+0000","last_undegraded":"2024-01-26T23:48:11.699950+0000","last_fullsized":"2024-01-26T23:48:11.699950+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:11:59.205203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699884+0000","last_change":"2024-01-26T23:48:11.694600+0000","last_active":"2024-01-26T23:48:11.699884+0000","last_peered":"2024-01-26T23:48:11.699884+0000","last_clean":"2024-01-26T23:48:11.699884+0000","last_became_active":"2024-01-26T23:48:11.694195+0000","last_became_peered":"2024-01-26T23:48:11.694195+0000","last_unstale":"2024-01-26T23:48:11.699884+0000","last_undegraded":"2024-01-26T23:48:11.699884+0000","last_fullsized":"2024-01-26T23:48:11.699884+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:54:52.459709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.697971+0000","last_change":"2024-01-26T23:48:11.695928+0000","last_active":"2024-01-26T23:48:11.697971+0000","last_peered":"2024-01-26T23:48:11.697971+0000","last_clean":"2024-01-26T23:48:11.697971+0000","last_became_active":"2024-01-26T23:48:11.695622+0000","last_became_peered":"2024-01-26T23:48:11.695622+0000","last_unstale":"2024-01-26T23:48:11.697971+0000","last_undegraded":"2024-01-26T23:48:11.697971+0000","last_fullsized":"2024-01-26T23:48:11.697971+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:37:48.746729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"129'4","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806763+0000","last_change":"2024-01-26T23:48:10.820996+0000","last_active":"2024-01-26T23:48:11.806763+0000","last_peered":"2024-01-26T23:48:11.806763+0000","last_clean":"2024-01-26T23:48:11.806763+0000","last_became_active":"2024-01-26T23:48:10.820659+0000","last_became_peered":"2024-01-26T23:48:10.820659+0000","last_unstale":"2024-01-26T23:48:11.806763+0000","last_undegraded":"2024-01-26T23:48:11.806763+0000","last_fullsized":"2024-01-26T23:48:11.806763+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:32:29.511141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702321+0000","last_change":"2024-01-26T23:48:11.699321+0000","last_active":"2024-01-26T23:48:11.702321+0000","last_peered":"2024-01-26T23:48:11.702321+0000","last_clean":"2024-01-26T23:48:11.702321+0000","last_became_active":"2024-01-26T23:48:11.699085+0000","last_became_peered":"2024-01-26T23:48:11.699085+0000","last_unstale":"2024-01-26T23:48:11.702321+0000","last_undegraded":"2024-01-26T23:48:11.702321+0000","last_fullsized":"2024-01-26T23:48:11.702321+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:49:57.681990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"129'1","reported_seq":11,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.805807+0000","last_change":"2024-01-26T23:48:10.816548+0000","last_active":"2024-01-26T23:48:11.805807+0000","last_peered":"2024-01-26T23:48:11.805807+0000","last_clean":"2024-01-26T23:48:11.805807+0000","last_became_active":"2024-01-26T23:48:10.816271+0000","last_became_peered":"2024-01-26T23:48:10.816271+0000","last_unstale":"2024-01-26T23:48:11.805807+0000","last_undegraded":"2024-01-26T23:48:11.805807+0000","last_fullsized":"2024-01-26T23:48:11.805807+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:10.976028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693908+0000","last_change":"2024-01-26T23:48:11.691251+0000","last_active":"2024-01-26T23:48:11.693908+0000","last_peered":"2024-01-26T23:48:11.693908+0000","last_clean":"2024-01-26T23:48:11.693908+0000","last_became_active":"2024-01-26T23:48:11.690973+0000","last_became_peered":"2024-01-26T23:48:11.690973+0000","last_unstale":"2024-01-26T23:48:11.693908+0000","last_undegraded":"2024-01-26T23:48:11.693908+0000","last_fullsized":"2024-01-26T23:48:11.693908+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:15:54.696377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.802933+0000","last_change":"2024-01-26T23:48:10.811743+0000","last_active":"2024-01-26T23:48:11.802933+0000","last_peered":"2024-01-26T23:48:11.802933+0000","last_clean":"2024-01-26T23:48:11.802933+0000","last_became_active":"2024-01-26T23:48:10.810670+0000","last_became_peered":"2024-01-26T23:48:10.810670+0000","last_unstale":"2024-01-26T23:48:11.802933+0000","last_undegraded":"2024-01-26T23:48:11.802933+0000","last_fullsized":"2024-01-26T23:48:11.802933+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:23:51.594366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699753+0000","last_change":"2024-01-26T23:48:11.696014+0000","last_active":"2024-01-26T23:48:11.699753+0000","last_peered":"2024-01-26T23:48:11.699753+0000","last_clean":"2024-01-26T23:48:11.699753+0000","last_became_active":"2024-01-26T23:48:11.695745+0000","last_became_peered":"2024-01-26T23:48:11.695745+0000","last_unstale":"2024-01-26T23:48:11.699753+0000","last_undegraded":"2024-01-26T23:48:11.699753+0000","last_fullsized":"2024-01-26T23:48:11.699753+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:26:08.221069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.803587+0000","last_change":"2024-01-26T23:48:10.812880+0000","last_active":"2024-01-26T23:48:11.803587+0000","last_peered":"2024-01-26T23:48:11.803587+0000","last_clean":"2024-01-26T23:48:11.803587+0000","last_became_active":"2024-01-26T23:48:10.812205+0000","last_became_peered":"2024-01-26T23:48:10.812205+0000","last_unstale":"2024-01-26T23:48:11.803587+0000","last_undegraded":"2024-01-26T23:48:11.803587+0000","last_fullsized":"2024-01-26T23:48:11.803587+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:26.896205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.691775+0000","last_change":"2024-01-26T23:48:11.687570+0000","last_active":"2024-01-26T23:48:11.691775+0000","last_peered":"2024-01-26T23:48:11.691775+0000","last_clean":"2024-01-26T23:48:11.691775+0000","last_became_active":"2024-01-26T23:48:11.686069+0000","last_became_peered":"2024-01-26T23:48:11.686069+0000","last_unstale":"2024-01-26T23:48:11.691775+0000","last_undegraded":"2024-01-26T23:48:11.691775+0000","last_fullsized":"2024-01-26T23:48:11.691775+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:36:36.750288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"129'1","reported_seq":11,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.805568+0000","last_change":"2024-01-26T23:48:10.815157+0000","last_active":"2024-01-26T23:48:11.805568+0000","last_peered":"2024-01-26T23:48:11.805568+0000","last_clean":"2024-01-26T23:48:11.805568+0000","last_became_active":"2024-01-26T23:48:10.814838+0000","last_became_peered":"2024-01-26T23:48:10.814838+0000","last_unstale":"2024-01-26T23:48:11.805568+0000","last_undegraded":"2024-01-26T23:48:11.805568+0000","last_fullsized":"2024-01-26T23:48:11.805568+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:49:44.669484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698877+0000","last_change":"2024-01-26T23:48:11.697027+0000","last_active":"2024-01-26T23:48:11.698877+0000","last_peered":"2024-01-26T23:48:11.698877+0000","last_clean":"2024-01-26T23:48:11.698877+0000","last_became_active":"2024-01-26T23:48:11.696702+0000","last_became_peered":"2024-01-26T23:48:11.696702+0000","last_unstale":"2024-01-26T23:48:11.698877+0000","last_undegraded":"2024-01-26T23:48:11.698877+0000","last_fullsized":"2024-01-26T23:48:11.698877+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:38:32.101523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"129'1","reported_seq":11,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806591+0000","last_change":"2024-01-26T23:48:10.813718+0000","last_active":"2024-01-26T23:48:11.806591+0000","last_peered":"2024-01-26T23:48:11.806591+0000","last_clean":"2024-01-26T23:48:11.806591+0000","last_became_active":"2024-01-26T23:48:10.813464+0000","last_became_peered":"2024-01-26T23:48:10.813464+0000","last_unstale":"2024-01-26T23:48:11.806591+0000","last_undegraded":"2024-01-26T23:48:11.806591+0000","last_fullsized":"2024-01-26T23:48:11.806591+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:13:40.216491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702230+0000","last_change":"2024-01-26T23:48:11.698344+0000","last_active":"2024-01-26T23:48:11.702230+0000","last_peered":"2024-01-26T23:48:11.702230+0000","last_clean":"2024-01-26T23:48:11.702230+0000","last_became_active":"2024-01-26T23:48:11.698034+0000","last_became_peered":"2024-01-26T23:48:11.698034+0000","last_unstale":"2024-01-26T23:48:11.702230+0000","last_undegraded":"2024-01-26T23:48:11.702230+0000","last_fullsized":"2024-01-26T23:48:11.702230+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:38:32.447049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685115+0000","last_change":"2024-01-26T23:48:10.811013+0000","last_active":"2024-01-26T23:48:11.685115+0000","last_peered":"2024-01-26T23:48:11.685115+0000","last_clean":"2024-01-26T23:48:11.685115+0000","last_became_active":"2024-01-26T23:48:10.810684+0000","last_became_peered":"2024-01-26T23:48:10.810684+0000","last_unstale":"2024-01-26T23:48:11.685115+0000","last_undegraded":"2024-01-26T23:48:11.685115+0000","last_fullsized":"2024-01-26T23:48:11.685115+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:29:34.900670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699671+0000","last_change":"2024-01-26T23:48:11.694858+0000","last_active":"2024-01-26T23:48:11.699671+0000","last_peered":"2024-01-26T23:48:11.699671+0000","last_clean":"2024-01-26T23:48:11.699671+0000","last_became_active":"2024-01-26T23:48:11.694407+0000","last_became_peered":"2024-01-26T23:48:11.694407+0000","last_unstale":"2024-01-26T23:48:11.699671+0000","last_undegraded":"2024-01-26T23:48:11.699671+0000","last_fullsized":"2024-01-26T23:48:11.699671+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:33:21.325282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.681769+0000","last_change":"2024-01-26T23:48:10.807380+0000","last_active":"2024-01-26T23:48:11.681769+0000","last_peered":"2024-01-26T23:48:11.681769+0000","last_clean":"2024-01-26T23:48:11.681769+0000","last_became_active":"2024-01-26T23:48:10.806968+0000","last_became_peered":"2024-01-26T23:48:10.806968+0000","last_unstale":"2024-01-26T23:48:11.681769+0000","last_undegraded":"2024-01-26T23:48:11.681769+0000","last_fullsized":"2024-01-26T23:48:11.681769+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:49:44.669484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.697301+0000","last_change":"2024-01-26T23:48:11.695119+0000","last_active":"2024-01-26T23:48:11.697301+0000","last_peered":"2024-01-26T23:48:11.697301+0000","last_clean":"2024-01-26T23:48:11.697301+0000","last_became_active":"2024-01-26T23:48:11.694615+0000","last_became_peered":"2024-01-26T23:48:11.694615+0000","last_unstale":"2024-01-26T23:48:11.697301+0000","last_undegraded":"2024-01-26T23:48:11.697301+0000","last_fullsized":"2024-01-26T23:48:11.697301+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:36:13.295894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685363+0000","last_change":"2024-01-26T23:48:10.811335+0000","last_active":"2024-01-26T23:48:11.685363+0000","last_peered":"2024-01-26T23:48:11.685363+0000","last_clean":"2024-01-26T23:48:11.685363+0000","last_became_active":"2024-01-26T23:48:10.810926+0000","last_became_peered":"2024-01-26T23:48:10.810926+0000","last_unstale":"2024-01-26T23:48:11.685363+0000","last_undegraded":"2024-01-26T23:48:11.685363+0000","last_fullsized":"2024-01-26T23:48:11.685363+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:22:26.586950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.703222+0000","last_change":"2024-01-26T23:48:11.697447+0000","last_active":"2024-01-26T23:48:11.703222+0000","last_peered":"2024-01-26T23:48:11.703222+0000","last_clean":"2024-01-26T23:48:11.703222+0000","last_became_active":"2024-01-26T23:48:11.697133+0000","last_became_peered":"2024-01-26T23:48:11.697133+0000","last_unstale":"2024-01-26T23:48:11.703222+0000","last_undegraded":"2024-01-26T23:48:11.703222+0000","last_fullsized":"2024-01-26T23:48:11.703222+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:28:53.642854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.677121+0000","last_change":"2024-01-26T23:48:10.810309+0000","last_active":"2024-01-26T23:48:11.677121+0000","last_peered":"2024-01-26T23:48:11.677121+0000","last_clean":"2024-01-26T23:48:11.677121+0000","last_became_active":"2024-01-26T23:48:10.810028+0000","last_became_peered":"2024-01-26T23:48:10.810028+0000","last_unstale":"2024-01-26T23:48:11.677121+0000","last_undegraded":"2024-01-26T23:48:11.677121+0000","last_fullsized":"2024-01-26T23:48:11.677121+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:36:07.346545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702237+0000","last_change":"2024-01-26T23:48:11.695812+0000","last_active":"2024-01-26T23:48:11.702237+0000","last_peered":"2024-01-26T23:48:11.702237+0000","last_clean":"2024-01-26T23:48:11.702237+0000","last_became_active":"2024-01-26T23:48:11.695535+0000","last_became_peered":"2024-01-26T23:48:11.695535+0000","last_unstale":"2024-01-26T23:48:11.702237+0000","last_undegraded":"2024-01-26T23:48:11.702237+0000","last_fullsized":"2024-01-26T23:48:11.702237+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:12:39.189557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"129'3","reported_seq":13,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806352+0000","last_change":"2024-01-26T23:48:10.813349+0000","last_active":"2024-01-26T23:48:11.806352+0000","last_peered":"2024-01-26T23:48:11.806352+0000","last_clean":"2024-01-26T23:48:11.806352+0000","last_became_active":"2024-01-26T23:48:10.813052+0000","last_became_peered":"2024-01-26T23:48:10.813052+0000","last_unstale":"2024-01-26T23:48:11.806352+0000","last_undegraded":"2024-01-26T23:48:11.806352+0000","last_fullsized":"2024-01-26T23:48:11.806352+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:50:15.445301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.703727+0000","last_change":"2024-01-26T23:48:11.703049+0000","last_active":"2024-01-26T23:48:11.703727+0000","last_peered":"2024-01-26T23:48:11.703727+0000","last_clean":"2024-01-26T23:48:11.703727+0000","last_became_active":"2024-01-26T23:48:11.702636+0000","last_became_peered":"2024-01-26T23:48:11.702636+0000","last_unstale":"2024-01-26T23:48:11.703727+0000","last_undegraded":"2024-01-26T23:48:11.703727+0000","last_fullsized":"2024-01-26T23:48:11.703727+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:50:12.326901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"129'6","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.807949+0000","last_change":"2024-01-26T23:48:10.814838+0000","last_active":"2024-01-26T23:48:11.807949+0000","last_peered":"2024-01-26T23:48:11.807949+0000","last_clean":"2024-01-26T23:48:11.807949+0000","last_became_active":"2024-01-26T23:48:10.814534+0000","last_became_peered":"2024-01-26T23:48:10.814534+0000","last_unstale":"2024-01-26T23:48:11.807949+0000","last_undegraded":"2024-01-26T23:48:11.807949+0000","last_fullsized":"2024-01-26T23:48:11.807949+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:51:57.293119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.688225+0000","last_change":"2024-01-26T23:48:11.683406+0000","last_active":"2024-01-26T23:48:11.688225+0000","last_peered":"2024-01-26T23:48:11.688225+0000","last_clean":"2024-01-26T23:48:11.688225+0000","last_became_active":"2024-01-26T23:48:11.683122+0000","last_became_peered":"2024-01-26T23:48:11.683122+0000","last_unstale":"2024-01-26T23:48:11.688225+0000","last_undegraded":"2024-01-26T23:48:11.688225+0000","last_fullsized":"2024-01-26T23:48:11.688225+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:59:48.214103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.804068+0000","last_change":"2024-01-26T23:48:10.812433+0000","last_active":"2024-01-26T23:48:11.804068+0000","last_peered":"2024-01-26T23:48:11.804068+0000","last_clean":"2024-01-26T23:48:11.804068+0000","last_became_active":"2024-01-26T23:48:10.812111+0000","last_became_peered":"2024-01-26T23:48:10.812111+0000","last_unstale":"2024-01-26T23:48:11.804068+0000","last_undegraded":"2024-01-26T23:48:11.804068+0000","last_fullsized":"2024-01-26T23:48:11.804068+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:49:44.669484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.697279+0000","last_change":"2024-01-26T23:48:11.694313+0000","last_active":"2024-01-26T23:48:11.697279+0000","last_peered":"2024-01-26T23:48:11.697279+0000","last_clean":"2024-01-26T23:48:11.697279+0000","last_became_active":"2024-01-26T23:48:11.693806+0000","last_became_peered":"2024-01-26T23:48:11.693806+0000","last_unstale":"2024-01-26T23:48:11.697279+0000","last_undegraded":"2024-01-26T23:48:11.697279+0000","last_fullsized":"2024-01-26T23:48:11.697279+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:07.428513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.682226+0000","last_change":"2024-01-26T23:48:10.819155+0000","last_active":"2024-01-26T23:48:11.682226+0000","last_peered":"2024-01-26T23:48:11.682226+0000","last_clean":"2024-01-26T23:48:11.682226+0000","last_became_active":"2024-01-26T23:48:10.818853+0000","last_became_peered":"2024-01-26T23:48:10.818853+0000","last_unstale":"2024-01-26T23:48:11.682226+0000","last_undegraded":"2024-01-26T23:48:11.682226+0000","last_fullsized":"2024-01-26T23:48:11.682226+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:50:15.445301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699954+0000","last_change":"2024-01-26T23:48:11.694802+0000","last_active":"2024-01-26T23:48:11.699954+0000","last_peered":"2024-01-26T23:48:11.699954+0000","last_clean":"2024-01-26T23:48:11.699954+0000","last_became_active":"2024-01-26T23:48:11.694277+0000","last_became_peered":"2024-01-26T23:48:11.694277+0000","last_unstale":"2024-01-26T23:48:11.699954+0000","last_undegraded":"2024-01-26T23:48:11.699954+0000","last_fullsized":"2024-01-26T23:48:11.699954+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:28:58.191038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.684638+0000","last_change":"2024-01-26T23:48:10.813933+0000","last_active":"2024-01-26T23:48:11.684638+0000","last_peered":"2024-01-26T23:48:11.684638+0000","last_clean":"2024-01-26T23:48:11.684638+0000","last_became_active":"2024-01-26T23:48:10.813646+0000","last_became_peered":"2024-01-26T23:48:10.813646+0000","last_unstale":"2024-01-26T23:48:11.684638+0000","last_undegraded":"2024-01-26T23:48:11.684638+0000","last_fullsized":"2024-01-26T23:48:11.684638+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:20:47.863263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693345+0000","last_change":"2024-01-26T23:48:11.683741+0000","last_active":"2024-01-26T23:48:11.693345+0000","last_peered":"2024-01-26T23:48:11.693345+0000","last_clean":"2024-01-26T23:48:11.693345+0000","last_became_active":"2024-01-26T23:48:11.683413+0000","last_became_peered":"2024-01-26T23:48:11.683413+0000","last_unstale":"2024-01-26T23:48:11.693345+0000","last_undegraded":"2024-01-26T23:48:11.693345+0000","last_fullsized":"2024-01-26T23:48:11.693345+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:53:44.050356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"129'8","reported_seq":18,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806654+0000","last_change":"2024-01-26T23:48:10.810870+0000","last_active":"2024-01-26T23:48:11.806654+0000","last_peered":"2024-01-26T23:48:11.806654+0000","last_clean":"2024-01-26T23:48:11.806654+0000","last_became_active":"2024-01-26T23:48:10.810516+0000","last_became_peered":"2024-01-26T23:48:10.810516+0000","last_unstale":"2024-01-26T23:48:11.806654+0000","last_undegraded":"2024-01-26T23:48:11.806654+0000","last_fullsized":"2024-01-26T23:48:11.806654+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:11:58.198504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.699904+0000","last_change":"2024-01-26T23:48:11.694643+0000","last_active":"2024-01-26T23:48:11.699904+0000","last_peered":"2024-01-26T23:48:11.699904+0000","last_clean":"2024-01-26T23:48:11.699904+0000","last_became_active":"2024-01-26T23:48:11.694296+0000","last_became_peered":"2024-01-26T23:48:11.694296+0000","last_unstale":"2024-01-26T23:48:11.699904+0000","last_undegraded":"2024-01-26T23:48:11.699904+0000","last_fullsized":"2024-01-26T23:48:11.699904+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:53:21.191149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"129'1","reported_seq":11,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806724+0000","last_change":"2024-01-26T23:48:10.815595+0000","last_active":"2024-01-26T23:48:11.806724+0000","last_peered":"2024-01-26T23:48:11.806724+0000","last_clean":"2024-01-26T23:48:11.806724+0000","last_became_active":"2024-01-26T23:48:10.815314+0000","last_became_peered":"2024-01-26T23:48:10.815314+0000","last_unstale":"2024-01-26T23:48:11.806724+0000","last_undegraded":"2024-01-26T23:48:11.806724+0000","last_fullsized":"2024-01-26T23:48:11.806724+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:26.896205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.700740+0000","last_change":"2024-01-26T23:48:11.695066+0000","last_active":"2024-01-26T23:48:11.700740+0000","last_peered":"2024-01-26T23:48:11.700740+0000","last_clean":"2024-01-26T23:48:11.700740+0000","last_became_active":"2024-01-26T23:48:11.694666+0000","last_became_peered":"2024-01-26T23:48:11.694666+0000","last_unstale":"2024-01-26T23:48:11.700740+0000","last_undegraded":"2024-01-26T23:48:11.700740+0000","last_fullsized":"2024-01-26T23:48:11.700740+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:50:12.326901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"129'1","reported_seq":11,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.806244+0000","last_change":"2024-01-26T23:48:10.814032+0000","last_active":"2024-01-26T23:48:11.806244+0000","last_peered":"2024-01-26T23:48:11.806244+0000","last_clean":"2024-01-26T23:48:11.806244+0000","last_became_active":"2024-01-26T23:48:10.813750+0000","last_became_peered":"2024-01-26T23:48:10.813750+0000","last_unstale":"2024-01-26T23:48:11.806244+0000","last_undegraded":"2024-01-26T23:48:11.806244+0000","last_fullsized":"2024-01-26T23:48:11.806244+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:45:19.808817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.703599+0000","last_change":"2024-01-26T23:48:11.701949+0000","last_active":"2024-01-26T23:48:11.703599+0000","last_peered":"2024-01-26T23:48:11.703599+0000","last_clean":"2024-01-26T23:48:11.703599+0000","last_became_active":"2024-01-26T23:48:11.701651+0000","last_became_peered":"2024-01-26T23:48:11.701651+0000","last_unstale":"2024-01-26T23:48:11.703599+0000","last_undegraded":"2024-01-26T23:48:11.703599+0000","last_fullsized":"2024-01-26T23:48:11.703599+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:10:51.561423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.681257+0000","last_change":"2024-01-26T23:48:10.815606+0000","last_active":"2024-01-26T23:48:11.681257+0000","last_peered":"2024-01-26T23:48:11.681257+0000","last_clean":"2024-01-26T23:48:11.681257+0000","last_became_active":"2024-01-26T23:48:10.815300+0000","last_became_peered":"2024-01-26T23:48:10.815300+0000","last_unstale":"2024-01-26T23:48:11.681257+0000","last_undegraded":"2024-01-26T23:48:11.681257+0000","last_fullsized":"2024-01-26T23:48:11.681257+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:46:44.771350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693545+0000","last_change":"2024-01-26T23:48:11.689724+0000","last_active":"2024-01-26T23:48:11.693545+0000","last_peered":"2024-01-26T23:48:11.693545+0000","last_clean":"2024-01-26T23:48:11.693545+0000","last_became_active":"2024-01-26T23:48:11.689438+0000","last_became_peered":"2024-01-26T23:48:11.689438+0000","last_unstale":"2024-01-26T23:48:11.693545+0000","last_undegraded":"2024-01-26T23:48:11.693545+0000","last_fullsized":"2024-01-26T23:48:11.693545+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:01.094001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.803125+0000","last_change":"2024-01-26T23:48:10.815064+0000","last_active":"2024-01-26T23:48:11.803125+0000","last_peered":"2024-01-26T23:48:11.803125+0000","last_clean":"2024-01-26T23:48:11.803125+0000","last_became_active":"2024-01-26T23:48:10.814778+0000","last_became_peered":"2024-01-26T23:48:10.814778+0000","last_unstale":"2024-01-26T23:48:11.803125+0000","last_undegraded":"2024-01-26T23:48:11.803125+0000","last_fullsized":"2024-01-26T23:48:11.803125+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:17.080283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.680810+0000","last_change":"2024-01-26T23:48:11.679952+0000","last_active":"2024-01-26T23:48:11.680810+0000","last_peered":"2024-01-26T23:48:11.680810+0000","last_clean":"2024-01-26T23:48:11.680810+0000","last_became_active":"2024-01-26T23:48:11.679509+0000","last_became_peered":"2024-01-26T23:48:11.679509+0000","last_unstale":"2024-01-26T23:48:11.680810+0000","last_undegraded":"2024-01-26T23:48:11.680810+0000","last_fullsized":"2024-01-26T23:48:11.680810+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:35:33.076540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.686302+0000","last_change":"2024-01-26T23:48:10.813851+0000","last_active":"2024-01-26T23:48:11.686302+0000","last_peered":"2024-01-26T23:48:11.686302+0000","last_clean":"2024-01-26T23:48:11.686302+0000","last_became_active":"2024-01-26T23:48:10.813510+0000","last_became_peered":"2024-01-26T23:48:10.813510+0000","last_unstale":"2024-01-26T23:48:11.686302+0000","last_undegraded":"2024-01-26T23:48:11.686302+0000","last_fullsized":"2024-01-26T23:48:11.686302+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:26:11.395397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698546+0000","last_change":"2024-01-26T23:48:11.695637+0000","last_active":"2024-01-26T23:48:11.698546+0000","last_peered":"2024-01-26T23:48:11.698546+0000","last_clean":"2024-01-26T23:48:11.698546+0000","last_became_active":"2024-01-26T23:48:11.695304+0000","last_became_peered":"2024-01-26T23:48:11.695304+0000","last_unstale":"2024-01-26T23:48:11.698546+0000","last_undegraded":"2024-01-26T23:48:11.698546+0000","last_fullsized":"2024-01-26T23:48:11.698546+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:50:50.668201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.690137+0000","last_change":"2024-01-26T23:48:10.815486+0000","last_active":"2024-01-26T23:48:11.690137+0000","last_peered":"2024-01-26T23:48:11.690137+0000","last_clean":"2024-01-26T23:48:11.690137+0000","last_became_active":"2024-01-26T23:48:10.815197+0000","last_became_peered":"2024-01-26T23:48:10.815197+0000","last_unstale":"2024-01-26T23:48:11.690137+0000","last_undegraded":"2024-01-26T23:48:11.690137+0000","last_fullsized":"2024-01-26T23:48:11.690137+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:57.119524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698608+0000","last_change":"2024-01-26T23:48:11.692941+0000","last_active":"2024-01-26T23:48:11.698608+0000","last_peered":"2024-01-26T23:48:11.698608+0000","last_clean":"2024-01-26T23:48:11.698608+0000","last_became_active":"2024-01-26T23:48:11.692605+0000","last_became_peered":"2024-01-26T23:48:11.692605+0000","last_unstale":"2024-01-26T23:48:11.698608+0000","last_undegraded":"2024-01-26T23:48:11.698608+0000","last_fullsized":"2024-01-26T23:48:11.698608+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:01:01.083122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685230+0000","last_change":"2024-01-26T23:48:10.816002+0000","last_active":"2024-01-26T23:48:11.685230+0000","last_peered":"2024-01-26T23:48:11.685230+0000","last_clean":"2024-01-26T23:48:11.685230+0000","last_became_active":"2024-01-26T23:48:10.815730+0000","last_became_peered":"2024-01-26T23:48:10.815730+0000","last_unstale":"2024-01-26T23:48:11.685230+0000","last_undegraded":"2024-01-26T23:48:11.685230+0000","last_fullsized":"2024-01-26T23:48:11.685230+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:59.034288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693414+0000","last_change":"2024-01-26T23:48:11.685991+0000","last_active":"2024-01-26T23:48:11.693414+0000","last_peered":"2024-01-26T23:48:11.693414+0000","last_clean":"2024-01-26T23:48:11.693414+0000","last_became_active":"2024-01-26T23:48:11.685598+0000","last_became_peered":"2024-01-26T23:48:11.685598+0000","last_unstale":"2024-01-26T23:48:11.693414+0000","last_undegraded":"2024-01-26T23:48:11.693414+0000","last_fullsized":"2024-01-26T23:48:11.693414+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:48:35.115441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.804084+0000","last_change":"2024-01-26T23:48:10.817696+0000","last_active":"2024-01-26T23:48:11.804084+0000","last_peered":"2024-01-26T23:48:11.804084+0000","last_clean":"2024-01-26T23:48:11.804084+0000","last_became_active":"2024-01-26T23:48:10.815112+0000","last_became_peered":"2024-01-26T23:48:10.815112+0000","last_unstale":"2024-01-26T23:48:11.804084+0000","last_undegraded":"2024-01-26T23:48:11.804084+0000","last_fullsized":"2024-01-26T23:48:11.804084+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:50:15.445301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.693883+0000","last_change":"2024-01-26T23:48:11.691998+0000","last_active":"2024-01-26T23:48:11.693883+0000","last_peered":"2024-01-26T23:48:11.693883+0000","last_clean":"2024-01-26T23:48:11.693883+0000","last_became_active":"2024-01-26T23:48:11.691685+0000","last_became_peered":"2024-01-26T23:48:11.691685+0000","last_unstale":"2024-01-26T23:48:11.693883+0000","last_undegraded":"2024-01-26T23:48:11.693883+0000","last_fullsized":"2024-01-26T23:48:11.693883+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:35:42.319231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"130'1","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-26T23:48:12.675495+0000","last_change":"2024-01-26T23:48:10.806631+0000","last_active":"2024-01-26T23:48:12.675495+0000","last_peered":"2024-01-26T23:48:12.675495+0000","last_clean":"2024-01-26T23:48:12.675495+0000","last_became_active":"2024-01-26T23:48:10.806274+0000","last_became_peered":"2024-01-26T23:48:10.806274+0000","last_unstale":"2024-01-26T23:48:12.675495+0000","last_undegraded":"2024-01-26T23:48:12.675495+0000","last_fullsized":"2024-01-26T23:48:12.675495+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:59:28.765875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698753+0000","last_change":"2024-01-26T23:48:11.698307+0000","last_active":"2024-01-26T23:48:11.698753+0000","last_peered":"2024-01-26T23:48:11.698753+0000","last_clean":"2024-01-26T23:48:11.698753+0000","last_became_active":"2024-01-26T23:48:11.698019+0000","last_became_peered":"2024-01-26T23:48:11.698019+0000","last_unstale":"2024-01-26T23:48:11.698753+0000","last_undegraded":"2024-01-26T23:48:11.698753+0000","last_fullsized":"2024-01-26T23:48:11.698753+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:08:43.039916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.691955+0000","last_change":"2024-01-26T23:48:10.821818+0000","last_active":"2024-01-26T23:48:11.691955+0000","last_peered":"2024-01-26T23:48:11.691955+0000","last_clean":"2024-01-26T23:48:11.691955+0000","last_became_active":"2024-01-26T23:48:10.821530+0000","last_became_peered":"2024-01-26T23:48:10.821530+0000","last_unstale":"2024-01-26T23:48:11.691955+0000","last_undegraded":"2024-01-26T23:48:11.691955+0000","last_fullsized":"2024-01-26T23:48:11.691955+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:26:07.214369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.688038+0000","last_change":"2024-01-26T23:48:11.682394+0000","last_active":"2024-01-26T23:48:11.688038+0000","last_peered":"2024-01-26T23:48:11.688038+0000","last_clean":"2024-01-26T23:48:11.688038+0000","last_became_active":"2024-01-26T23:48:11.682136+0000","last_became_peered":"2024-01-26T23:48:11.682136+0000","last_unstale":"2024-01-26T23:48:11.688038+0000","last_undegraded":"2024-01-26T23:48:11.688038+0000","last_fullsized":"2024-01-26T23:48:11.688038+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:45:20.815517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.673973+0000","last_change":"2024-01-26T23:48:10.813068+0000","last_active":"2024-01-26T23:48:11.673973+0000","last_peered":"2024-01-26T23:48:11.673973+0000","last_clean":"2024-01-26T23:48:11.673973+0000","last_became_active":"2024-01-26T23:48:10.812610+0000","last_became_peered":"2024-01-26T23:48:10.812610+0000","last_unstale":"2024-01-26T23:48:11.673973+0000","last_undegraded":"2024-01-26T23:48:11.673973+0000","last_fullsized":"2024-01-26T23:48:11.673973+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:45:19.808817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.698643+0000","last_change":"2024-01-26T23:48:11.692758+0000","last_active":"2024-01-26T23:48:11.698643+0000","last_peered":"2024-01-26T23:48:11.698643+0000","last_clean":"2024-01-26T23:48:11.698643+0000","last_became_active":"2024-01-26T23:48:11.692446+0000","last_became_peered":"2024-01-26T23:48:11.692446+0000","last_unstale":"2024-01-26T23:48:11.698643+0000","last_undegraded":"2024-01-26T23:48:11.698643+0000","last_fullsized":"2024-01-26T23:48:11.698643+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:54:10.694699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685024+0000","last_change":"2024-01-26T23:48:10.813057+0000","last_active":"2024-01-26T23:48:11.685024+0000","last_peered":"2024-01-26T23:48:11.685024+0000","last_clean":"2024-01-26T23:48:11.685024+0000","last_became_active":"2024-01-26T23:48:10.812778+0000","last_became_peered":"2024-01-26T23:48:10.812778+0000","last_unstale":"2024-01-26T23:48:11.685024+0000","last_undegraded":"2024-01-26T23:48:11.685024+0000","last_fullsized":"2024-01-26T23:48:11.685024+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:50:11.320201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702415+0000","last_change":"2024-01-26T23:48:11.696828+0000","last_active":"2024-01-26T23:48:11.702415+0000","last_peered":"2024-01-26T23:48:11.702415+0000","last_clean":"2024-01-26T23:48:11.702415+0000","last_became_active":"2024-01-26T23:48:11.695115+0000","last_became_peered":"2024-01-26T23:48:11.695115+0000","last_unstale":"2024-01-26T23:48:11.702415+0000","last_undegraded":"2024-01-26T23:48:11.702415+0000","last_fullsized":"2024-01-26T23:48:11.702415+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:25.597544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.687089+0000","last_change":"2024-01-26T23:48:10.815435+0000","last_active":"2024-01-26T23:48:11.687089+0000","last_peered":"2024-01-26T23:48:11.687089+0000","last_clean":"2024-01-26T23:48:11.687089+0000","last_became_active":"2024-01-26T23:48:10.815136+0000","last_became_peered":"2024-01-26T23:48:10.815136+0000","last_unstale":"2024-01-26T23:48:11.687089+0000","last_undegraded":"2024-01-26T23:48:11.687089+0000","last_fullsized":"2024-01-26T23:48:11.687089+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:36:28.278118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.704766+0000","last_change":"2024-01-26T23:48:11.699841+0000","last_active":"2024-01-26T23:48:11.704766+0000","last_peered":"2024-01-26T23:48:11.704766+0000","last_clean":"2024-01-26T23:48:11.704766+0000","last_became_active":"2024-01-26T23:48:11.699571+0000","last_became_peered":"2024-01-26T23:48:11.699571+0000","last_unstale":"2024-01-26T23:48:11.704766+0000","last_undegraded":"2024-01-26T23:48:11.704766+0000","last_fullsized":"2024-01-26T23:48:11.704766+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:33:21.325282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.690801+0000","last_change":"2024-01-26T23:48:10.821792+0000","last_active":"2024-01-26T23:48:11.690801+0000","last_peered":"2024-01-26T23:48:11.690801+0000","last_clean":"2024-01-26T23:48:11.690801+0000","last_became_active":"2024-01-26T23:48:10.821453+0000","last_became_peered":"2024-01-26T23:48:10.821453+0000","last_unstale":"2024-01-26T23:48:11.690801+0000","last_undegraded":"2024-01-26T23:48:11.690801+0000","last_fullsized":"2024-01-26T23:48:11.690801+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:59:47.207403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.702140+0000","last_change":"2024-01-26T23:48:11.697093+0000","last_active":"2024-01-26T23:48:11.702140+0000","last_peered":"2024-01-26T23:48:11.702140+0000","last_clean":"2024-01-26T23:48:11.702140+0000","last_became_active":"2024-01-26T23:48:11.696468+0000","last_became_peered":"2024-01-26T23:48:11.696468+0000","last_unstale":"2024-01-26T23:48:11.702140+0000","last_undegraded":"2024-01-26T23:48:11.702140+0000","last_fullsized":"2024-01-26T23:48:11.702140+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:58.126224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":10,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.685337+0000","last_change":"2024-01-26T23:48:10.817416+0000","last_active":"2024-01-26T23:48:11.685337+0000","last_peered":"2024-01-26T23:48:11.685337+0000","last_clean":"2024-01-26T23:48:11.685337+0000","last_became_active":"2024-01-26T23:48:10.817077+0000","last_became_peered":"2024-01-26T23:48:10.817077+0000","last_unstale":"2024-01-26T23:48:11.685337+0000","last_undegraded":"2024-01-26T23:48:11.685337+0000","last_fullsized":"2024-01-26T23:48:11.685337+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:05.190741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.695596+0000","last_change":"2024-01-26T23:48:11.692393+0000","last_active":"2024-01-26T23:48:11.695596+0000","last_peered":"2024-01-26T23:48:11.695596+0000","last_clean":"2024-01-26T23:48:11.695596+0000","last_became_active":"2024-01-26T23:48:11.690501+0000","last_became_peered":"2024-01-26T23:48:11.690501+0000","last_unstale":"2024-01-26T23:48:11.695596+0000","last_undegraded":"2024-01-26T23:48:11.695596+0000","last_fullsized":"2024-01-26T23:48:11.695596+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:10.785466+0000","last_clean_scrub_stamp":"2024-01-26T23:48:10.785466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:51:58.299819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"129'2","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.804097+0000","last_change":"2024-01-26T23:48:10.814833+0000","last_active":"2024-01-26T23:48:11.804097+0000","last_peered":"2024-01-26T23:48:11.804097+0000","last_clean":"2024-01-26T23:48:11.804097+0000","last_became_active":"2024-01-26T23:48:10.814499+0000","last_became_peered":"2024-01-26T23:48:10.814499+0000","last_unstale":"2024-01-26T23:48:11.804097+0000","last_undegraded":"2024-01-26T23:48:11.804097+0000","last_fullsized":"2024-01-26T23:48:11.804097+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:48:09.778766+0000","last_clean_scrub_stamp":"2024-01-26T23:48:09.778766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:27:02.987035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"120'8562","reported_seq":9017,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-26T23:48:11.687611+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:48:11.687611+0000","last_peered":"2024-01-26T23:48:11.687611+0000","last_clean":"2024-01-26T23:48:11.687611+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:48:11.687611+0000","last_undegraded":"2024-01-26T23:48:11.687611+0000","last_fullsized":"2024-01-26T23:48:11.687611+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8562,"log_dups_size":0,"ondisk_log_size":8562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":971,"num_read_kb":1902,"num_write":320,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":971,"num_read_kb":1902,"num_write":320,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8562,"ondisk_log_size":8562,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575071,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311736,"kb_used_data":2160,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060104,"statfs":{"total":96636764160,"available":96317546496,"internally_reserved":0,"allocated":2211840,"data_stored":1416893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.51000000000000001,"15min":0.56399999999999995},"min":{"1min":0.34100000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.52800000000000002,"15min":0.57899999999999996},"min":{"1min":0.379,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.83999999999999997},"last":0.54900000000000004}]},{"osd":1,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.53500000000000003,"15min":0.57599999999999996},"min":{"1min":0.40999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.82899999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55800000000000005,"15min":0.59699999999999998},"min":{"1min":0.36199999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.84499999999999997,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.68700000000000006}]},{"osd":2,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54600000000000004,"15min":0.57599999999999996},"min":{"1min":0.39600000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76800000000000002,"5min":0.80000000000000004,"15min":0.81699999999999995},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56000000000000005,"15min":0.59199999999999997},"min":{"1min":0.40799999999999997,"5min":0.30199999999999999,"15min":0.27400000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.88600000000000001},"last":0.621}]},{"osd":3,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55300000000000005,"15min":0.58799999999999997},"min":{"1min":0.42399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82999999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.59799999999999998},"min":{"1min":0.442,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86399999999999999},"last":0.73799999999999999}]},{"osd":4,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.42699999999999999,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.252,"15min":0.247},"max":{"1min":0.58399999999999996,"5min":0.66100000000000003,"15min":0.93899999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.44900000000000001,"15min":0.47399999999999998},"min":{"1min":0.316,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.66200000000000003,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.57899999999999996}]},{"osd":5,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47199999999999998,"15min":0.48499999999999999},"min":{"1min":0.33500000000000002,"5min":0.27200000000000002,"15min":0.22800000000000001},"max":{"1min":0.78000000000000003,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.496,"15min":0.51900000000000002},"min":{"1min":0.377,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69499999999999995,"5min":0.74199999999999999,"15min":0.90200000000000002},"last":0.59899999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52500000000000002,"15min":0.54500000000000004},"min":{"1min":0.377,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68500000000000005,"5min":0.78700000000000003,"15min":0.95899999999999996},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52400000000000002,"15min":0.54000000000000004},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.85199999999999998},"last":0.67200000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575071,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302976,"kb_used_data":1592,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068864,"statfs":{"total":96636764160,"available":96326516736,"internally_reserved":0,"allocated":1630208,"data_stored":828477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.53300000000000003,"15min":0.56799999999999995},"min":{"1min":0.379,"5min":0.255,"15min":0.255},"max":{"1min":0.78700000000000003,"5min":1.423,"15min":1.423},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.53800000000000003,"15min":0.58399999999999996},"min":{"1min":0.45000000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.72999999999999998,"5min":0.79000000000000004,"15min":0.91100000000000003},"last":0.57699999999999996}]},{"osd":1,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.52400000000000002,"15min":0.56399999999999995},"min":{"1min":0.39800000000000002,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.76900000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55300000000000005,"15min":0.58999999999999997},"min":{"1min":0.432,"5min":0.312,"15min":0.312},"max":{"1min":0.749,"5min":1.498,"15min":1.498},"last":0.53300000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53100000000000003,"15min":0.57399999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72699999999999998,"5min":0.74199999999999999,"15min":0.85799999999999998},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53500000000000003,"15min":0.57799999999999996},"min":{"1min":0.39400000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71799999999999997,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.54700000000000004}]},{"osd":3,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57199999999999995,"15min":0.59399999999999997},"min":{"1min":0.37,"5min":0.33800000000000002,"15min":0.29999999999999999},"max":{"1min":0.745,"5min":1.595,"15min":1.595},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.61199999999999999},"min":{"1min":0.39000000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.80300000000000005,"5min":1.369,"15min":1.369},"last":0.60699999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.45100000000000001,"15min":0.46000000000000002},"min":{"1min":0.32200000000000001,"5min":0.245,"15min":0.22},"max":{"1min":0.64400000000000002,"5min":1.556,"15min":1.556},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.47499999999999998,"15min":0.48299999999999998},"min":{"1min":0.32000000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65900000000000003,"5min":0.753,"15min":0.86199999999999999},"last":0.51300000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45900000000000002,"15min":0.47299999999999998},"min":{"1min":0.32600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.59699999999999998,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52000000000000002,"15min":0.51900000000000002},"min":{"1min":0.27200000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.72699999999999998,"5min":1.47,"15min":1.47},"last":0.44400000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.505,"15min":0.51800000000000002},"min":{"1min":0.41299999999999998,"5min":0.26700000000000002,"15min":0.246},"max":{"1min":0.68700000000000006,"5min":0.82199999999999995,"15min":0.85899999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.374,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.67400000000000004,"5min":0.78500000000000003,"15min":0.80400000000000005},"last":0.42299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575071,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307056,"kb_used_data":1576,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064784,"statfs":{"total":96636764160,"available":96322338816,"internally_reserved":0,"allocated":1613824,"data_stored":826399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.41699999999999998,"15min":0.40699999999999997},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61799999999999999,"5min":0.65200000000000002,"15min":0.72599999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.442,"15min":0.443},"min":{"1min":0.28999999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.63500000000000001,"5min":1.337,"15min":1.337},"last":0.52600000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40400000000000003,"15min":0.41099999999999998},"min":{"1min":0.28000000000000003,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.60199999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.71999999999999997,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.67300000000000004}]},{"osd":3,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.51500000000000001},"min":{"1min":0.40000000000000002,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":0.77900000000000003,"5min":1.782,"15min":1.782},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48599999999999999,"15min":0.47999999999999998},"min":{"1min":0.38800000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.69899999999999995,"5min":1.379,"15min":1.379},"last":0.59999999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.61199999999999999,"15min":0.64100000000000001},"min":{"1min":0.40799999999999997,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":0.73299999999999998,"5min":0.94099999999999995,"15min":1.0820000000000001},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63900000000000001,"15min":0.65700000000000003},"min":{"1min":0.443,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.872,"5min":0.89900000000000002,"15min":0.95699999999999996},"last":0.88}]},{"osd":5,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.65600000000000003,"15min":0.66900000000000004},"min":{"1min":0.501,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":0.68899999999999995,"5min":1.208,"15min":1.208},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.47099999999999997,"5min":0.45100000000000001,"15min":0.42499999999999999},"max":{"1min":0.77700000000000002,"5min":0.96599999999999997,"15min":1.018},"last":0.96399999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.65900000000000003,"15min":0.67300000000000004},"min":{"1min":0.498,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.78400000000000003,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.73099999999999998,"5min":1.01,"15min":1.022},"last":0.93200000000000005}]},{"osd":7,"last update":"Fri Jan 26 23:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.68400000000000005,"15min":0.70599999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.875,"5min":0.98199999999999998,"15min":1.0069999999999999},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.70399999999999996,"15min":0.71799999999999997},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.753,"5min":1.296,"15min":1.296},"last":0.85999999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575070,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307648,"kb_used_data":2168,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064192,"statfs":{"total":96636764160,"available":96321732608,"internally_reserved":0,"allocated":2220032,"data_stored":1418359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.41899999999999998,"15min":0.40600000000000003},"min":{"1min":0.29499999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.49399999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.438,"15min":0.45300000000000001},"min":{"1min":0.33800000000000002,"5min":0.27400000000000002,"15min":0.27300000000000002},"max":{"1min":0.64500000000000002,"5min":1.363,"15min":1.4490000000000001},"last":0.42499999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40699999999999997,"15min":0.42299999999999999},"min":{"1min":0.29799999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.58299999999999996,"5min":0.67600000000000005,"15min":0.89600000000000002},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.46899999999999997,"15min":0.47099999999999997},"min":{"1min":0.372,"5min":0.32700000000000001,"15min":0.28000000000000003},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.38200000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.47299999999999998},"min":{"1min":0.36699999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.83099999999999996,"5min":2.145,"15min":2.145},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.35599999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":1.198},"last":0.441}]},{"osd":4,"last update":"Fri Jan 26 23:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.623,"15min":0.64300000000000002},"min":{"1min":0.47099999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75600000000000001,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.60499999999999998,"15min":0.64000000000000001},"min":{"1min":0.379,"5min":0.373,"15min":0.373},"max":{"1min":0.85999999999999999,"5min":0.88,"15min":0.94299999999999995},"last":0.55500000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63600000000000001,"15min":0.65100000000000002},"min":{"1min":0.50900000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.92500000000000004,"5min":1.329,"15min":1.329},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.67700000000000005,"15min":0.67500000000000004},"min":{"1min":0.50800000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.96299999999999997,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.67500000000000004}]},{"osd":6,"last update":"Fri Jan 26 23:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.61699999999999999,"15min":0.65500000000000003},"min":{"1min":0.501,"5min":0.376,"15min":0.376},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.032},"last":0.501},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.66900000000000004,"15min":0.68600000000000005},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.92400000000000004,"5min":1.286,"15min":1.286},"last":0.69199999999999995}]},{"osd":7,"last update":"Fri Jan 26 23:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.72099999999999997,"15min":0.82199999999999995},"min":{"1min":0.50600000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.86399999999999999,"5min":4.8710000000000004,"15min":25.965},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.751,"15min":0.86699999999999999},"min":{"1min":0.53800000000000003,"5min":0.48799999999999999,"15min":0.433},"max":{"1min":0.90200000000000002,"5min":4.9199999999999999,"15min":33.085000000000001},"last":0.70699999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575070,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307056,"kb_used_data":1576,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064784,"statfs":{"total":96636764160,"available":96322338816,"internally_reserved":0,"allocated":1613824,"data_stored":826399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.38800000000000001,"15min":0.39000000000000001},"min":{"1min":0.29399999999999998,"5min":0.26900000000000002,"15min":0.255},"max":{"1min":0.53800000000000003,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.43099999999999999,"15min":0.42699999999999999},"min":{"1min":0.26800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60499999999999998,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.34799999999999998}]},{"osd":1,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.435,"15min":0.42799999999999999},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.248},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52000000000000002,"15min":0.51000000000000001},"min":{"1min":0.379,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.044,"5min":1.171,"15min":1.171},"last":0.39600000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46800000000000003,"15min":0.48399999999999999},"min":{"1min":0.34000000000000002,"5min":0.32300000000000001,"15min":0.313},"max":{"1min":0.68700000000000006,"5min":0.72799999999999998,"15min":0.74099999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78200000000000003,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.41899999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.60799999999999998,"15min":0.63},"min":{"1min":0.53000000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.82299999999999995,"5min":1.087,"15min":1.087},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.46300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.95499999999999996,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.49299999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.64500000000000002,"15min":0.65600000000000003},"min":{"1min":0.495,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0740000000000001,"5min":1.24,"15min":1.24},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.66400000000000003,"15min":0.67600000000000005},"min":{"1min":0.53400000000000003,"5min":0.41799999999999998,"15min":0.35199999999999998},"max":{"1min":0.93000000000000005,"5min":0.97699999999999998,"15min":1.085},"last":0.66100000000000003}]},{"osd":6,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.67800000000000005},"min":{"1min":0.495,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.98599999999999999,"5min":1.022,"15min":1.1080000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66600000000000004,"15min":0.68100000000000005},"min":{"1min":0.51800000000000002,"5min":0.42799999999999999,"15min":0.38800000000000001},"max":{"1min":0.97499999999999998,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.57499999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.68999999999999995,"15min":0.70399999999999996},"min":{"1min":0.52800000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94999999999999996,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.70999999999999996,"15min":0.72699999999999998},"min":{"1min":0.53400000000000003,"5min":0.48099999999999998,"15min":0.42099999999999999},"max":{"1min":1.0189999999999999,"5min":1.052,"15min":1.139},"last":0.59199999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575069,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307056,"kb_used_data":1576,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064784,"statfs":{"total":96636764160,"available":96322338816,"internally_reserved":0,"allocated":1613824,"data_stored":826919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.38200000000000001,"15min":0.39100000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":1.8540000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.42099999999999999,"15min":0.44},"min":{"1min":0.30599999999999999,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":1.2290000000000001},"last":0.57899999999999996}]},{"osd":1,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44500000000000001,"15min":0.45400000000000001},"min":{"1min":0.33100000000000002,"5min":0.27900000000000003,"15min":0.253},"max":{"1min":0.63200000000000001,"5min":0.63800000000000001,"15min":1.633},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.48099999999999998,"15min":0.48499999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":0.65900000000000003,"5min":0.67800000000000005,"15min":1.0760000000000001},"last":0.52000000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.47399999999999998,"15min":0.47999999999999998},"min":{"1min":0.31900000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68600000000000005,"5min":0.70399999999999996,"15min":1.4059999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.439,"15min":0.45900000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28299999999999997},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":1.1870000000000001},"last":0.629}]},{"osd":4,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.61699999999999999,"15min":0.65800000000000003},"min":{"1min":0.45400000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.86799999999999999,"5min":0.878,"15min":3.306},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.628,"15min":0.65700000000000003},"min":{"1min":0.50700000000000001,"5min":0.40100000000000002,"15min":0.379},"max":{"1min":0.83899999999999997,"5min":0.85199999999999998,"15min":1.008},"last":0.75}]},{"osd":5,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.62,"15min":0.65300000000000002},"min":{"1min":0.49199999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":1.593},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63500000000000001,"15min":0.66600000000000004},"min":{"1min":0.499,"5min":0.379,"15min":0.379},"max":{"1min":0.79500000000000004,"5min":0.92300000000000004,"15min":1.3819999999999999},"last":0.77900000000000003}]},{"osd":6,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63400000000000001,"15min":0.66900000000000004},"min":{"1min":0.49099999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85599999999999998,"5min":0.88800000000000001,"15min":1.506},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.66000000000000003,"15min":0.68799999999999994},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.44400000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":1.427},"last":0.88500000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.63700000000000001,"15min":0.68799999999999994},"min":{"1min":0.442,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":3.238},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.69699999999999995,"15min":0.73299999999999998},"min":{"1min":0.55000000000000004,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":3.351},"last":0.90100000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575071,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302964,"kb_used_data":1580,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068876,"statfs":{"total":96636764160,"available":96326529024,"internally_reserved":0,"allocated":1617920,"data_stored":826457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53400000000000003,"15min":0.57099999999999995},"min":{"1min":0.34699999999999998,"5min":0.28999999999999998,"15min":0.28699999999999998},"max":{"1min":0.71199999999999997,"5min":1.21,"15min":1.21},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.35999999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.76400000000000001,"5min":5.367,"15min":5.367},"last":0.57199999999999995}]},{"osd":1,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55700000000000005,"15min":0.58899999999999997},"min":{"1min":0.38800000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.88},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.53400000000000003,"15min":0.57699999999999996},"min":{"1min":0.35199999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.621,"5min":0.80600000000000005,"15min":0.81699999999999995},"last":0.55100000000000005}]},{"osd":2,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52700000000000002,"15min":0.56699999999999995},"min":{"1min":0.33900000000000002,"5min":0.33300000000000002,"15min":0.32700000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.57199999999999995,"15min":0.59699999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80500000000000005,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.501}]},{"osd":3,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55100000000000005,"15min":0.58399999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.68500000000000005,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.622},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56200000000000006,"15min":0.58299999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.35699999999999998},"max":{"1min":0.94199999999999995,"5min":1.05,"15min":1.05},"last":0.63400000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.44400000000000001,"15min":0.45800000000000002},"min":{"1min":0.29199999999999998,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.68300000000000005,"5min":0.76200000000000001,"15min":0.77600000000000002},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.47499999999999998,"15min":0.47699999999999998},"min":{"1min":0.29099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72499999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.69699999999999995}]},{"osd":6,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45200000000000001,"15min":0.46700000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27200000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.78900000000000003,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.70699999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.51400000000000001,"15min":0.53400000000000003},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.63700000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.5,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.36899999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.68500000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575071,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298860,"kb_used_data":1572,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072980,"statfs":{"total":96636764160,"available":96330731520,"internally_reserved":0,"allocated":1609728,"data_stored":826365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.54500000000000004,"15min":0.56699999999999995},"min":{"1min":0.43099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.81799999999999995,"5min":0.92800000000000005,"15min":2.0289999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53000000000000003,"15min":0.56899999999999995},"min":{"1min":0.35799999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77100000000000002,"5min":0.77400000000000002,"15min":0.82199999999999995},"last":0.504}]},{"osd":1,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57599999999999996,"15min":0.60799999999999998},"min":{"1min":0.33800000000000002,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":0.78000000000000003,"5min":0.85999999999999999,"15min":2.3319999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60599999999999998,"15min":0.627},"min":{"1min":0.47199999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80300000000000005,"5min":0.88500000000000001,"15min":1.1739999999999999},"last":0.622}]},{"osd":2,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.56100000000000005},"min":{"1min":0.438,"5min":0.34499999999999997,"15min":0.30099999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.83799999999999997},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55600000000000005,"15min":0.58699999999999997},"min":{"1min":0.38,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.80000000000000004,"5min":0.89700000000000002,"15min":2.0600000000000001},"last":0.64200000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56399999999999995,"15min":0.57699999999999996},"min":{"1min":0.40899999999999997,"5min":0.34999999999999998,"15min":0.314},"max":{"1min":0.68400000000000005,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56499999999999995,"15min":0.58699999999999997},"min":{"1min":0.39800000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.751,"5min":0.77900000000000003,"15min":2.5630000000000002},"last":0.53400000000000003}]},{"osd":4,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.42699999999999999,"15min":0.45300000000000001},"min":{"1min":0.30499999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64000000000000001,"5min":0.73099999999999998,"15min":2.3700000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.317,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.89200000000000002},"last":0.65900000000000003}]},{"osd":6,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.47299999999999998},"min":{"1min":0.33100000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":2.5990000000000002},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.49299999999999999,"15min":0.51300000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70299999999999996,"5min":0.76200000000000001,"15min":1.99},"last":0.67700000000000005}]},{"osd":7,"last update":"Fri Jan 26 23:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52400000000000002,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73499999999999999,"5min":0.76200000000000001,"15min":3.5270000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54200000000000004,"15min":0.55800000000000005},"min":{"1min":0.33500000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.747,"5min":0.81499999999999995,"15min":0.82199999999999995},"last":0.69099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:48:21.113 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-26T23:48:21.113 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-26T23:48:21.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.256+0000 7f056f71d700 1 Processor -- start 2024-01-26T23:48:21.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.256+0000 7f056f71d700 1 -- start start 2024-01-26T23:48:21.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.257+0000 7f056f71d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0568159510 0x7f05681598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:21.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.257+0000 7f056f71d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f05680ca220 0x7f05680c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:21.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.257+0000 7f056f71d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0568069260 con 0x7f05680ca690 2024-01-26T23:48:21.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.257+0000 7f056f71d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f056815d670 con 0x7f05680ca220 2024-01-26T23:48:21.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.257+0000 7f056f71d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05680c9750 con 0x7f0568159510 2024-01-26T23:48:21.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.257+0000 7f056ccb8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f05680ca220 0x7f05680c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:21.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.258+0000 7f056ccb8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f05680ca220 0x7f05680c8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48512/0 (socket says 172.21.15.46:48512) 2024-01-26T23:48:21.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.258+0000 7f056ccb8700 1 -- 172.21.15.46:0/4231959185 learned_addr learned my addr 172.21.15.46:0/4231959185 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:21.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.258+0000 7f056d4b9700 1 --2- 172.21.15.46:0/4231959185 >> v2:172.21.15.106:3300/0 conn(0x7f0568159510 0x7f05681598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:21.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.258+0000 7f056ccb8700 1 -- 172.21.15.46:0/4231959185 >> v2:172.21.15.106:3300/0 conn(0x7f0568159510 msgr2=0x7f05681598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:21.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.258+0000 7f056ccb8700 1 --2- 172.21.15.46:0/4231959185 >> v2:172.21.15.106:3300/0 conn(0x7f0568159510 0x7f05681598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.258+0000 7f056ccb8700 1 -- 172.21.15.46:0/4231959185 >> v1:172.21.15.106:6789/0 conn(0x7f05680ca690 legacy=0x7f05680cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:21.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.258+0000 7f056ccb8700 1 -- 172.21.15.46:0/4231959185 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05680c9a30 con 0x7f05680ca220 2024-01-26T23:48:21.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.258+0000 7f056d4b9700 1 --2- 172.21.15.46:0/4231959185 >> v2:172.21.15.106:3300/0 conn(0x7f0568159510 0x7f05681598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:21.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.258+0000 7f056ccb8700 1 --2- 172.21.15.46:0/4231959185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680ca220 0x7f05680c8be0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f056806f760 tx=0x7f0564009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e0a7e3e8dade0f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:21.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.258+0000 7f055ffff700 1 -- 172.21.15.46:0/4231959185 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056400b860 con 0x7f05680ca220 2024-01-26T23:48:21.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.258+0000 7f055ffff700 1 -- 172.21.15.46:0/4231959185 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056400b9c0 con 0x7f05680ca220 2024-01-26T23:48:21.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.258+0000 7f055ffff700 1 -- 172.21.15.46:0/4231959185 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056400bd60 con 0x7f05680ca220 2024-01-26T23:48:21.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.259+0000 7f056f71d700 1 -- 172.21.15.46:0/4231959185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680ca220 msgr2=0x7f05680c8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:21.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.259+0000 7f056f71d700 1 --2- 172.21.15.46:0/4231959185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680ca220 0x7f05680c8be0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f056806f760 tx=0x7f0564009250 comp rx=0 tx=0).stop 2024-01-26T23:48:21.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.259+0000 7f056f71d700 1 -- 172.21.15.46:0/4231959185 shutdown_connections 2024-01-26T23:48:21.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.259+0000 7f056f71d700 1 --2- 172.21.15.46:0/4231959185 >> v2:172.21.15.106:3300/0 conn(0x7f0568159510 0x7f05681598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.259+0000 7f056f71d700 1 --2- 172.21.15.46:0/4231959185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680ca220 0x7f05680c8be0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.259+0000 7f056f71d700 1 -- 172.21.15.46:0/4231959185 >> 172.21.15.46:0/4231959185 conn(0x7f056806bd90 msgr2=0x7f056806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:21.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.259+0000 7f056f71d700 1 -- 172.21.15.46:0/4231959185 shutdown_connections 2024-01-26T23:48:21.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.259+0000 7f056f71d700 1 -- 172.21.15.46:0/4231959185 wait complete. 2024-01-26T23:48:21.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.260+0000 7f056f71d700 1 Processor -- start 2024-01-26T23:48:21.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.260+0000 7f056f71d700 1 -- start start 2024-01-26T23:48:21.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.261+0000 7f056f71d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680ca220 0x7f05680b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:21.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.261+0000 7f056f71d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0568159510 0x7f056814cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:21.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.261+0000 7f056f71d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05680b3500 0x7f05680caa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:21.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.261+0000 7f056f71d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0568150e60 con 0x7f05680ca220 2024-01-26T23:48:21.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.261+0000 7f056f71d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0568150530 con 0x7f0568159510 2024-01-26T23:48:21.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.261+0000 7f056f71d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0568150810 con 0x7f05680b3500 2024-01-26T23:48:21.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.261+0000 7f056d4b9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680ca220 0x7f05680b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:21.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.261+0000 7f056d4b9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680ca220 0x7f05680b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48520/0 (socket says 172.21.15.46:48520) 2024-01-26T23:48:21.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.261+0000 7f056d4b9700 1 -- 172.21.15.46:0/3120965163 learned_addr learned my addr 172.21.15.46:0/3120965163 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:21.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.262+0000 7f056ccb8700 1 --2- 172.21.15.46:0/3120965163 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0568159510 0x7f056814cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:21.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.262+0000 7f056dcba700 1 --2- 172.21.15.46:0/3120965163 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05680b3500 0x7f05680caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:21.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.262+0000 7f056d4b9700 1 -- 172.21.15.46:0/3120965163 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05680b3500 msgr2=0x7f05680caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.262+0000 7f056d4b9700 1 --2- 172.21.15.46:0/3120965163 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05680b3500 0x7f05680caa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.262+0000 7f056d4b9700 1 -- 172.21.15.46:0/3120965163 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0568159510 msgr2=0x7f056814cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.262+0000 7f056d4b9700 1 --2- 172.21.15.46:0/3120965163 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0568159510 0x7f056814cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.262+0000 7f056d4b9700 1 -- 172.21.15.46:0/3120965163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05640094f0 con 0x7f05680ca220 2024-01-26T23:48:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.262+0000 7f056d4b9700 1 --2- 172.21.15.46:0/3120965163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680ca220 0x7f05680b3a30 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f055800d860 tx=0x7f0558008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.262+0000 7f055d7fa700 1 -- 172.21.15.46:0/3120965163 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05580125f0 con 0x7f05680ca220 2024-01-26T23:48:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.262+0000 7f055d7fa700 1 -- 172.21.15.46:0/3120965163 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0558012750 con 0x7f05680ca220 2024-01-26T23:48:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.262+0000 7f056f71d700 1 -- 172.21.15.46:0/3120965163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0568150af0 con 0x7f05680ca220 2024-01-26T23:48:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.262+0000 7f055d7fa700 1 -- 172.21.15.46:0/3120965163 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0558012af0 con 0x7f05680ca220 2024-01-26T23:48:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.262+0000 7f056f71d700 1 -- 172.21.15.46:0/3120965163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f056814d430 con 0x7f05680ca220 2024-01-26T23:48:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.263+0000 7f056f71d700 1 -- 172.21.15.46:0/3120965163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056804c710 con 0x7f05680ca220 2024-01-26T23:48:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.264+0000 7f055d7fa700 1 -- 172.21.15.46:0/3120965163 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0558013480 con 0x7f05680ca220 2024-01-26T23:48:21.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.264+0000 7f055d7fa700 1 --2- 172.21.15.46:0/3120965163 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f054c066f30 0x7f054c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:21.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.264+0000 7f055d7fa700 1 -- 172.21.15.46:0/3120965163 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f05580849c0 con 0x7f05680ca220 2024-01-26T23:48:21.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.264+0000 7f056ccb8700 1 --2- 172.21.15.46:0/3120965163 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f054c066f30 0x7f054c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:21.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.265+0000 7f056ccb8700 1 --2- 172.21.15.46:0/3120965163 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f054c066f30 0x7f054c0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f056806f760 tx=0x7f0564017000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:21.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.269+0000 7f055d7fa700 1 -- 172.21.15.46:0/3120965163 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0558051960 con 0x7f05680ca220 2024-01-26T23:48:21.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.559+0000 7f056f71d700 1 -- 172.21.15.46:0/3120965163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f05680c9d10 con 0x7f05680ca220 2024-01-26T23:48:21.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.563+0000 7f055d7fa700 1 -- 172.21.15.46:0/3120965163 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0558054f80 con 0x7f05680ca220 2024-01-26T23:48:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.566+0000 7f056f71d700 1 -- 172.21.15.46:0/3120965163 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f054c066f30 msgr2=0x7f054c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.566+0000 7f056f71d700 1 --2- 172.21.15.46:0/3120965163 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f054c066f30 0x7f054c0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f056806f760 tx=0x7f0564017000 comp rx=0 tx=0).stop 2024-01-26T23:48:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.566+0000 7f056f71d700 1 -- 172.21.15.46:0/3120965163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680ca220 msgr2=0x7f05680b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.566+0000 7f056f71d700 1 --2- 172.21.15.46:0/3120965163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680ca220 0x7f05680b3a30 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f055800d860 tx=0x7f0558008040 comp rx=0 tx=0).stop 2024-01-26T23:48:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.566+0000 7f056f71d700 1 -- 172.21.15.46:0/3120965163 shutdown_connections 2024-01-26T23:48:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.566+0000 7f056f71d700 1 --2- 172.21.15.46:0/3120965163 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f054c066f30 0x7f054c0693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.566+0000 7f056f71d700 1 --2- 172.21.15.46:0/3120965163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680ca220 0x7f05680b3a30 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.566+0000 7f056f71d700 1 --2- 172.21.15.46:0/3120965163 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05680b3500 0x7f05680caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.566+0000 7f056f71d700 1 --2- 172.21.15.46:0/3120965163 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0568159510 0x7f056814cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.566+0000 7f056f71d700 1 -- 172.21.15.46:0/3120965163 >> 172.21.15.46:0/3120965163 conn(0x7f056806bd90 msgr2=0x7f05680c4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.566+0000 7f056f71d700 1 -- 172.21.15.46:0/3120965163 shutdown_connections 2024-01-26T23:48:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.566+0000 7f056f71d700 1 -- 172.21.15.46:0/3120965163 wait complete. 2024-01-26T23:48:21.569 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-01-26T23:48:21.581 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:48:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.712+0000 7fde7c264700 1 Processor -- start 2024-01-26T23:48:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.712+0000 7fde7c264700 1 -- start start 2024-01-26T23:48:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.713+0000 7fde7c264700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde741561d0 0x7fde740ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.713+0000 7fde7c264700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fde740b0280 0x7fde741565a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.713+0000 7fde7c264700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde740b0760 con 0x7fde7406eb70 2024-01-26T23:48:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.713+0000 7fde7c264700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde740b0a40 con 0x7fde740b0280 2024-01-26T23:48:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.713+0000 7fde7c264700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde740b0d20 con 0x7fde741561d0 2024-01-26T23:48:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.714+0000 7fde7a801700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fde7406eb70 0x7fde7406c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47262/0 (socket says 172.21.15.46:47262) 2024-01-26T23:48:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.714+0000 7fde7a000700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde741561d0 0x7fde740ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.714+0000 7fde7a801700 1 -- 172.21.15.46:0/4150914601 learned_addr learned my addr 172.21.15.46:0/4150914601 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.714+0000 7fde797ff700 1 --2- 172.21.15.46:0/4150914601 >> v2:172.21.15.106:3300/0 conn(0x7fde740b0280 0x7fde741565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.714+0000 7fde78ffe700 1 -- 172.21.15.46:0/4150914601 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403239947 0 0) 0x7fde740b0760 con 0x7fde7406eb70 2024-01-26T23:48:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.714+0000 7fde78ffe700 1 -- 172.21.15.46:0/4150914601 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde60003570 con 0x7fde7406eb70 2024-01-26T23:48:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.714+0000 7fde7a000700 1 -- 172.21.15.46:0/4150914601 >> v2:172.21.15.106:3300/0 conn(0x7fde740b0280 msgr2=0x7fde741565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.714+0000 7fde7a000700 1 --2- 172.21.15.46:0/4150914601 >> v2:172.21.15.106:3300/0 conn(0x7fde740b0280 0x7fde741565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.714+0000 7fde7a000700 1 -- 172.21.15.46:0/4150914601 >> v1:172.21.15.46:6789/0 conn(0x7fde7406eb70 legacy=0x7fde7406c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.714+0000 7fde7a000700 1 -- 172.21.15.46:0/4150914601 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde740b1000 con 0x7fde741561d0 2024-01-26T23:48:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.714+0000 7fde797ff700 1 --2- 172.21.15.46:0/4150914601 >> v2:172.21.15.106:3300/0 conn(0x7fde740b0280 0x7fde741565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:48:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.714+0000 7fde7a000700 1 --2- 172.21.15.46:0/4150914601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde741561d0 0x7fde740ca5f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fde700098e0 tx=0x7fde700091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28defea1ecc32e0d server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.714+0000 7fde78ffe700 1 -- 172.21.15.46:0/4150914601 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde70011630 con 0x7fde741561d0 2024-01-26T23:48:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.715+0000 7fde78ffe700 1 -- 172.21.15.46:0/4150914601 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde70011790 con 0x7fde741561d0 2024-01-26T23:48:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.715+0000 7fde78ffe700 1 -- 172.21.15.46:0/4150914601 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde70011b30 con 0x7fde741561d0 2024-01-26T23:48:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.715+0000 7fde7c264700 1 -- 172.21.15.46:0/4150914601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde741561d0 msgr2=0x7fde740ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.715+0000 7fde7c264700 1 --2- 172.21.15.46:0/4150914601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde741561d0 0x7fde740ca5f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fde700098e0 tx=0x7fde700091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.715+0000 7fde7c264700 1 -- 172.21.15.46:0/4150914601 shutdown_connections 2024-01-26T23:48:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.715+0000 7fde7c264700 1 --2- 172.21.15.46:0/4150914601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde741561d0 0x7fde740ca5f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.715+0000 7fde7c264700 1 --2- 172.21.15.46:0/4150914601 >> v2:172.21.15.106:3300/0 conn(0x7fde740b0280 0x7fde741565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.715+0000 7fde7c264700 1 -- 172.21.15.46:0/4150914601 >> 172.21.15.46:0/4150914601 conn(0x7fde7406b630 msgr2=0x7fde740aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.715+0000 7fde7c264700 1 -- 172.21.15.46:0/4150914601 shutdown_connections 2024-01-26T23:48:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.715+0000 7fde7c264700 1 -- 172.21.15.46:0/4150914601 wait complete. 2024-01-26T23:48:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.717+0000 7fde7c264700 1 Processor -- start 2024-01-26T23:48:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.717+0000 7fde7c264700 1 -- start start 2024-01-26T23:48:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.718+0000 7fde7c264700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde7406eb70 0x7fde740aee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.718+0000 7fde7c264700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde740b0280 0x7fde7414b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.718+0000 7fde7c264700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde741561d0 0x7fde740b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.718+0000 7fde7c264700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde740af370 con 0x7fde7406eb70 2024-01-26T23:48:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.718+0000 7fde7c264700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde74157de0 con 0x7fde741561d0 2024-01-26T23:48:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.718+0000 7fde7c264700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde740aeaf0 con 0x7fde740b0280 2024-01-26T23:48:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.718+0000 7fde7a000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde7406eb70 0x7fde740aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.718+0000 7fde7a000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde7406eb70 0x7fde740aee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48524/0 (socket says 172.21.15.46:48524) 2024-01-26T23:48:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.718+0000 7fde7a000700 1 -- 172.21.15.46:0/4126936652 learned_addr learned my addr 172.21.15.46:0/4126936652 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.718+0000 7fde7a801700 1 --2- 172.21.15.46:0/4126936652 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde741561d0 0x7fde740b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.718+0000 7fde797ff700 1 --2- 172.21.15.46:0/4126936652 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde740b0280 0x7fde7414b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.719+0000 7fde7a000700 1 -- 172.21.15.46:0/4126936652 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde740b0280 msgr2=0x7fde7414b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.719+0000 7fde7a000700 1 --2- 172.21.15.46:0/4126936652 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde740b0280 0x7fde7414b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.719+0000 7fde7a000700 1 -- 172.21.15.46:0/4126936652 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde741561d0 msgr2=0x7fde740b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.719+0000 7fde7a000700 1 --2- 172.21.15.46:0/4126936652 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde741561d0 0x7fde740b1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.719+0000 7fde7a000700 1 -- 172.21.15.46:0/4126936652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde64009420 con 0x7fde7406eb70 2024-01-26T23:48:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.719+0000 7fde7a801700 1 --2- 172.21.15.46:0/4126936652 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde741561d0 0x7fde740b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.719+0000 7fde7a000700 1 --2- 172.21.15.46:0/4126936652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde7406eb70 0x7fde740aee30 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fde70009ed0 tx=0x7fde70017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.719+0000 7fde69ffb700 1 -- 172.21.15.46:0/4126936652 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde70011b10 con 0x7fde7406eb70 2024-01-26T23:48:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.719+0000 7fde69ffb700 1 -- 172.21.15.46:0/4126936652 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde700253f0 con 0x7fde7406eb70 2024-01-26T23:48:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.719+0000 7fde69ffb700 1 -- 172.21.15.46:0/4126936652 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde700257b0 con 0x7fde7406eb70 2024-01-26T23:48:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.719+0000 7fde7c264700 1 -- 172.21.15.46:0/4126936652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde70009420 con 0x7fde7406eb70 2024-01-26T23:48:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.719+0000 7fde7c264700 1 -- 172.21.15.46:0/4126936652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde740680f0 con 0x7fde7406eb70 2024-01-26T23:48:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.720+0000 7fde7c264700 1 -- 172.21.15.46:0/4126936652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde74063930 con 0x7fde7406eb70 2024-01-26T23:48:21.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.721+0000 7fde69ffb700 1 -- 172.21.15.46:0/4126936652 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fde70025b60 con 0x7fde7406eb70 2024-01-26T23:48:21.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.721+0000 7fde69ffb700 1 --2- 172.21.15.46:0/4126936652 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fde58066f30 0x7fde580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:21.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.721+0000 7fde69ffb700 1 -- 172.21.15.46:0/4126936652 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fde70085f60 con 0x7fde7406eb70 2024-01-26T23:48:21.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.722+0000 7fde797ff700 1 --2- 172.21.15.46:0/4126936652 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fde58066f30 0x7fde580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:21.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.722+0000 7fde797ff700 1 --2- 172.21.15.46:0/4126936652 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fde58066f30 0x7fde580693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fde640093f0 tx=0x7fde64008040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.726+0000 7fde69ffb700 1 -- 172.21.15.46:0/4126936652 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fde70052f00 con 0x7fde7406eb70 2024-01-26T23:48:21.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.980+0000 7fde7c264700 1 -- 172.21.15.46:0/4126936652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde74157bb0 con 0x7fde7406eb70 2024-01-26T23:48:21.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.980+0000 7fde69ffb700 1 -- 172.21.15.46:0/4126936652 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4599 (secure 0 0 0) 0x7fde70015090 con 0x7fde7406eb70 2024-01-26T23:48:21.982 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:48:21.982 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":62,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6923,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/4123353304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4123353304},{"type":"v1","addr":"172.21.15.106:6834","nonce":4123353304}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/690662483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":690662483},{"type":"v1","addr":"172.21.15.106:6836","nonce":690662483}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7791,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3035969379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3035969379},{"type":"v1","addr":"172.21.15.46:6834","nonce":3035969379}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:48:11.234632+0000","modified":"2024-01-26T23:48:14.815720+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7794},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7794":{"gid":7794,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/783623731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":783623731},{"type":"v1","addr":"172.21.15.46:6835","nonce":783623731}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2024-01-26T23:48:21.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.983+0000 7fde7c264700 1 -- 172.21.15.46:0/4126936652 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fde58066f30 msgr2=0x7fde580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:21.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.983+0000 7fde7c264700 1 --2- 172.21.15.46:0/4126936652 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fde58066f30 0x7fde580693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fde640093f0 tx=0x7fde64008040 comp rx=0 tx=0).stop 2024-01-26T23:48:21.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.983+0000 7fde7c264700 1 -- 172.21.15.46:0/4126936652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde7406eb70 msgr2=0x7fde740aee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:21.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.983+0000 7fde7c264700 1 --2- 172.21.15.46:0/4126936652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde7406eb70 0x7fde740aee30 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fde70009ed0 tx=0x7fde70017000 comp rx=0 tx=0).stop 2024-01-26T23:48:21.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.983+0000 7fde7c264700 1 -- 172.21.15.46:0/4126936652 shutdown_connections 2024-01-26T23:48:21.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.983+0000 7fde7c264700 1 --2- 172.21.15.46:0/4126936652 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fde58066f30 0x7fde580693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.983+0000 7fde7c264700 1 --2- 172.21.15.46:0/4126936652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde7406eb70 0x7fde740aee30 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.983+0000 7fde7c264700 1 --2- 172.21.15.46:0/4126936652 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde741561d0 0x7fde740b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.983+0000 7fde7c264700 1 --2- 172.21.15.46:0/4126936652 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde740b0280 0x7fde7414b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:21.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.983+0000 7fde7c264700 1 -- 172.21.15.46:0/4126936652 >> 172.21.15.46:0/4126936652 conn(0x7fde7406b630 msgr2=0x7fde7406f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:21.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.983+0000 7fde7c264700 1 -- 172.21.15.46:0/4126936652 shutdown_connections 2024-01-26T23:48:21.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:21.983+0000 7fde7c264700 1 -- 172.21.15.46:0/4126936652 wait complete. 2024-01-26T23:48:21.986 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 62 2024-01-26T23:48:22.401 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-26T23:48:11.234632+0000', 'modified': '2024-01-26T23:48:14.815720+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7794}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7794': {'gid': 7794, 'name': 'c', 'rank': 0, 'incarnation': 58, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6835/783623731', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 783623731}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 783623731}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-26T23:48:22.401 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-26T23:48:22.401 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-26T23:48:22.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.543+0000 7f4c6aebd700 1 Processor -- start 2024-01-26T23:48:22.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.543+0000 7f4c6aebd700 1 -- start start 2024-01-26T23:48:22.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.544+0000 7f4c6aebd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c6406f4a0 0x7f4c641569a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.544+0000 7f4c6aebd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4c640c4700 0x7f4c640c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.544+0000 7f4c6aebd700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c640c9ee0 con 0x7f4c641565d0 2024-01-26T23:48:22.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.544+0000 7f4c6aebd700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c640cbe90 con 0x7f4c640c4700 2024-01-26T23:48:22.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.544+0000 7f4c6aebd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640cb560 con 0x7f4c6406f4a0 2024-01-26T23:48:22.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.544+0000 7f4c68c59700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4c641565d0 0x7f4c6406ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47268/0 (socket says 172.21.15.46:47268) 2024-01-26T23:48:22.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.544+0000 7f4c68c59700 1 -- 172.21.15.46:0/1697143488 learned_addr learned my addr 172.21.15.46:0/1697143488 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:22.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.545+0000 7f4c6945a700 1 --2- 172.21.15.46:0/1697143488 >> v2:172.21.15.106:3300/0 conn(0x7f4c640c4700 0x7f4c640c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.545+0000 7f4c63fff700 1 --2- 172.21.15.46:0/1697143488 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c6406f4a0 0x7f4c641569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.545+0000 7f4c637fe700 1 -- 172.21.15.46:0/1697143488 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663998059 0 0) 0x7f4c640c9ee0 con 0x7f4c641565d0 2024-01-26T23:48:22.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.545+0000 7f4c637fe700 1 -- 172.21.15.46:0/1697143488 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c50003570 con 0x7f4c641565d0 2024-01-26T23:48:22.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.545+0000 7f4c637fe700 1 -- 172.21.15.46:0/1697143488 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1014777478 0 0) 0x7f4c50003570 con 0x7f4c641565d0 2024-01-26T23:48:22.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.545+0000 7f4c6945a700 1 -- 172.21.15.46:0/1697143488 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c6406f4a0 msgr2=0x7f4c641569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:22.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.545+0000 7f4c6945a700 1 --2- 172.21.15.46:0/1697143488 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c6406f4a0 0x7f4c641569a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.545+0000 7f4c6945a700 1 -- 172.21.15.46:0/1697143488 >> v1:172.21.15.46:6789/0 conn(0x7f4c641565d0 legacy=0x7f4c6406ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:22.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.545+0000 7f4c6945a700 1 -- 172.21.15.46:0/1697143488 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c640cb840 con 0x7f4c640c4700 2024-01-26T23:48:22.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.545+0000 7f4c6945a700 1 --2- 172.21.15.46:0/1697143488 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c640c4700 0x7f4c640c9170 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c009e60 tx=0x7f4c5c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c7f589f072ab27f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:22.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.545+0000 7f4c637fe700 1 -- 172.21.15.46:0/1697143488 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c00e070 con 0x7f4c640c4700 2024-01-26T23:48:22.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.545+0000 7f4c637fe700 1 -- 172.21.15.46:0/1697143488 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5c00b7a0 con 0x7f4c640c4700 2024-01-26T23:48:22.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.545+0000 7f4c637fe700 1 -- 172.21.15.46:0/1697143488 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54001470 con 0x7f4c640c4700 2024-01-26T23:48:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.546+0000 7f4c6aebd700 1 -- 172.21.15.46:0/1697143488 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c640c4700 msgr2=0x7f4c640c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.546+0000 7f4c6aebd700 1 --2- 172.21.15.46:0/1697143488 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c640c4700 0x7f4c640c9170 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c009e60 tx=0x7f4c5c009780 comp rx=0 tx=0).stop 2024-01-26T23:48:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.546+0000 7f4c6aebd700 1 -- 172.21.15.46:0/1697143488 shutdown_connections 2024-01-26T23:48:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.546+0000 7f4c6aebd700 1 --2- 172.21.15.46:0/1697143488 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c640c4700 0x7f4c640c9170 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.546+0000 7f4c6aebd700 1 --2- 172.21.15.46:0/1697143488 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c6406f4a0 0x7f4c641569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.546+0000 7f4c6aebd700 1 -- 172.21.15.46:0/1697143488 >> 172.21.15.46:0/1697143488 conn(0x7f4c6406b820 msgr2=0x7f4c640cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.546+0000 7f4c6aebd700 1 -- 172.21.15.46:0/1697143488 shutdown_connections 2024-01-26T23:48:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.546+0000 7f4c6aebd700 1 -- 172.21.15.46:0/1697143488 wait complete. 2024-01-26T23:48:22.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.548+0000 7f4c6aebd700 1 Processor -- start 2024-01-26T23:48:22.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.548+0000 7f4c6aebd700 1 -- start start 2024-01-26T23:48:22.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.549+0000 7f4c6aebd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c6406f4a0 0x7f4c640b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.549+0000 7f4c6aebd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c640c4700 0x7f4c640b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.549+0000 7f4c6aebd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c641565d0 0x7f4c6414e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.549+0000 7f4c6aebd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640b0930 con 0x7f4c640c4700 2024-01-26T23:48:22.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.549+0000 7f4c6aebd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640b1400 con 0x7f4c641565d0 2024-01-26T23:48:22.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.549+0000 7f4c6aebd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640accc0 con 0x7f4c6406f4a0 2024-01-26T23:48:22.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.549+0000 7f4c63fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c640c4700 0x7f4c640b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.549+0000 7f4c63fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c640c4700 0x7f4c640b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48532/0 (socket says 172.21.15.46:48532) 2024-01-26T23:48:22.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.549+0000 7f4c63fff700 1 -- 172.21.15.46:0/2804264004 learned_addr learned my addr 172.21.15.46:0/2804264004 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:22.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.549+0000 7f4c6945a700 1 --2- 172.21.15.46:0/2804264004 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c641565d0 0x7f4c6414e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.549+0000 7f4c68c59700 1 --2- 172.21.15.46:0/2804264004 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c6406f4a0 0x7f4c640b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.549+0000 7f4c63fff700 1 -- 172.21.15.46:0/2804264004 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c6406f4a0 msgr2=0x7f4c640b21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:22.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.549+0000 7f4c63fff700 1 --2- 172.21.15.46:0/2804264004 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c6406f4a0 0x7f4c640b21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.549+0000 7f4c63fff700 1 -- 172.21.15.46:0/2804264004 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c641565d0 msgr2=0x7f4c6414e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:22.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.549+0000 7f4c63fff700 1 --2- 172.21.15.46:0/2804264004 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c641565d0 0x7f4c6414e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.549+0000 7f4c63fff700 1 -- 172.21.15.46:0/2804264004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c009a20 con 0x7f4c640c4700 2024-01-26T23:48:22.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.550+0000 7f4c63fff700 1 --2- 172.21.15.46:0/2804264004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c640c4700 0x7f4c640b03f0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c010ec0 tx=0x7f4c4c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:22.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.550+0000 7f4c60ff9700 1 -- 172.21.15.46:0/2804264004 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00ed40 con 0x7f4c640c4700 2024-01-26T23:48:22.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.550+0000 7f4c60ff9700 1 -- 172.21.15.46:0/2804264004 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c00eea0 con 0x7f4c640c4700 2024-01-26T23:48:22.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.550+0000 7f4c6aebd700 1 -- 172.21.15.46:0/2804264004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c640b0b60 con 0x7f4c640c4700 2024-01-26T23:48:22.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.550+0000 7f4c60ff9700 1 -- 172.21.15.46:0/2804264004 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c011660 con 0x7f4c640c4700 2024-01-26T23:48:22.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.550+0000 7f4c6aebd700 1 -- 172.21.15.46:0/2804264004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c640b1120 con 0x7f4c640c4700 2024-01-26T23:48:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.551+0000 7f4c60ff9700 1 -- 172.21.15.46:0/2804264004 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4c4c011a10 con 0x7f4c640c4700 2024-01-26T23:48:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.552+0000 7f4c60ff9700 1 --2- 172.21.15.46:0/2804264004 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4c44066f30 0x7f4c440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.552+0000 7f4c60ff9700 1 -- 172.21.15.46:0/2804264004 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f4c4c086ae0 con 0x7f4c640c4700 2024-01-26T23:48:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.552+0000 7f4c68c59700 1 --2- 172.21.15.46:0/2804264004 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4c44066f30 0x7f4c440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.553+0000 7f4c68c59700 1 --2- 172.21.15.46:0/2804264004 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4c44066f30 0x7f4c440693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4c540011e0 tx=0x7f4c54001a00 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:22.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.554+0000 7f4c741c0b80 1 Processor -- start 2024-01-26T23:48:22.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.554+0000 7f4c741c0b80 1 -- start start 2024-01-26T23:48:22.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c741c0b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e09740 0x557924e2af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c741c0b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924e2a360 0x557924e09b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c741c0b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924de1e40 0x557924de22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c741c0b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x557924ce59f0 con 0x557924e09740 2024-01-26T23:48:22.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c741c0b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x557924d37130 con 0x557924e2a360 2024-01-26T23:48:22.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c741c0b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x557924d34930 con 0x557924de1e40 2024-01-26T23:48:22.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c68c59700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e09740 0x557924e2af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c68c59700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e09740 0x557924e2af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48538/0 (socket says 172.21.15.46:48538) 2024-01-26T23:48:22.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c68c59700 1 -- 172.21.15.46:0/724698243 learned_addr learned my addr 172.21.15.46:0/724698243 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:22.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c63fff700 1 --2- 172.21.15.46:0/724698243 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924e2a360 0x557924e09b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c6945a700 1 --2- 172.21.15.46:0/724698243 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924de1e40 0x557924de22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c68c59700 1 -- 172.21.15.46:0/724698243 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924de1e40 msgr2=0x557924de22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:22.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c68c59700 1 --2- 172.21.15.46:0/724698243 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924de1e40 0x557924de22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c68c59700 1 -- 172.21.15.46:0/724698243 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924e2a360 msgr2=0x557924e09b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:22.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c68c59700 1 --2- 172.21.15.46:0/724698243 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924e2a360 0x557924e09b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c68c59700 1 -- 172.21.15.46:0/724698243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557924b6fb10 con 0x557924e09740 2024-01-26T23:48:22.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c68c59700 1 --2- 172.21.15.46:0/724698243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e09740 0x557924e2af30 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400fd00 tx=0x7f4c54014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6488b08b021484e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:22.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c59ffb700 1 -- 172.21.15.46:0/724698243 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54013a90 con 0x557924e09740 2024-01-26T23:48:22.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c59ffb700 1 -- 172.21.15.46:0/724698243 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c54013bf0 con 0x557924e09740 2024-01-26T23:48:22.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.555+0000 7f4c59ffb700 1 -- 172.21.15.46:0/724698243 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54021420 con 0x557924e09740 2024-01-26T23:48:22.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.556+0000 7f4c741c0b80 1 -- 172.21.15.46:0/724698243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e09740 msgr2=0x557924e2af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:22.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.556+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/724698243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e09740 0x557924e2af30 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400fd00 tx=0x7f4c54014000 comp rx=0 tx=0).stop 2024-01-26T23:48:22.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.556+0000 7f4c741c0b80 1 -- 172.21.15.46:0/724698243 shutdown_connections 2024-01-26T23:48:22.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.556+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/724698243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e09740 0x557924e2af30 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.556+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/724698243 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924de1e40 0x557924de22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.556+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/724698243 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924e2a360 0x557924e09b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.556+0000 7f4c741c0b80 1 -- 172.21.15.46:0/724698243 >> 172.21.15.46:0/724698243 conn(0x557924de1a70 msgr2=0x557924e2a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:22.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.556+0000 7f4c741c0b80 1 -- 172.21.15.46:0/724698243 shutdown_connections 2024-01-26T23:48:22.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.556+0000 7f4c741c0b80 1 -- 172.21.15.46:0/724698243 wait complete. 2024-01-26T23:48:22.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.557+0000 7f4c741c0b80 1 -- start start 2024-01-26T23:48:22.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.557+0000 7f4c741c0b80 1 Processor -- start 2024-01-26T23:48:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c741c0b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924de1e40 0x557924e0efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c741c0b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924e09740 0x557924e22210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c741c0b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e2a360 0x557924e08270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c741c0b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x557924d30720 con 0x557924e2a360 2024-01-26T23:48:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c741c0b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x557924cdc1b0 con 0x557924de1e40 2024-01-26T23:48:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c741c0b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x557924cdbe50 con 0x557924e09740 2024-01-26T23:48:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c741c0b80 10 client.0 _ll_register_callbacks cb 0x557924b3e7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:48:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c6945a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e2a360 0x557924e08270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c6945a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e2a360 0x557924e08270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48552/0 (socket says 172.21.15.46:48552) 2024-01-26T23:48:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c6945a700 1 -- 172.21.15.46:0/1756029538 learned_addr learned my addr 172.21.15.46:0/1756029538 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c63fff700 1 --2- 172.21.15.46:0/1756029538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924e09740 0x557924e22210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c68c59700 1 --2- 172.21.15.46:0/1756029538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924de1e40 0x557924e0efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c6945a700 1 -- 172.21.15.46:0/1756029538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924e09740 msgr2=0x557924e22210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:22.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c6945a700 1 --2- 172.21.15.46:0/1756029538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924e09740 0x557924e22210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c6945a700 1 -- 172.21.15.46:0/1756029538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924de1e40 msgr2=0x557924e0efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:22.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c6945a700 1 --2- 172.21.15.46:0/1756029538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924de1e40 0x557924e0efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c6945a700 1 -- 172.21.15.46:0/1756029538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c5400fa20 con 0x557924e2a360 2024-01-26T23:48:22.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.558+0000 7f4c68c59700 1 --2- 172.21.15.46:0/1756029538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924de1e40 0x557924e0efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:48:22.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.559+0000 7f4c6945a700 1 --2- 172.21.15.46:0/1756029538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e2a360 0x557924e08270 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00b610 tx=0x7f4c5c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b3aceff7729b8a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:22.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.559+0000 7f4c42ffd700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:48:22.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.559+0000 7f4c42ffd700 1 -- 172.21.15.46:0/1756029538 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c023070 con 0x557924e2a360 2024-01-26T23:48:22.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.559+0000 7f4c42ffd700 1 -- 172.21.15.46:0/1756029538 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f4c5c00bd50 con 0x557924e2a360 2024-01-26T23:48:22.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.559+0000 7f4c741c0b80 1 -- 172.21.15.46:0/1756029538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557924dd5190 con 0x557924e2a360 2024-01-26T23:48:22.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.559+0000 7f4c42ffd700 1 -- 172.21.15.46:0/1756029538 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c5c0032f0 con 0x557924e2a360 2024-01-26T23:48:22.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.559+0000 7f4c741c0b80 10 client.7965 fetch_fsmap learned FSMap version 62 2024-01-26T23:48:22.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.559+0000 7f4c741c0b80 1 -- 172.21.15.46:0/1756029538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) v3 -- 0x557924cfe290 con 0x557924e2a360 2024-01-26T23:48:22.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.559+0000 7f4c42ffd700 1 -- 172.21.15.46:0/1756029538 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557924dd5190 con 0x557924e2a360 2024-01-26T23:48:22.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.559+0000 7f4c42ffd700 1 -- 172.21.15.46:0/1756029538 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 62) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f4c5c00b050 con 0x557924e2a360 2024-01-26T23:48:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.559+0000 7f4c741c0b80 10 client.7965 fetch_fsmap finished waiting for FSMap version 62 2024-01-26T23:48:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.559+0000 7f4c741c0b80 10 client.7965 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-26T23:48:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.559+0000 7f4c741c0b80 20 client.7965 populate_metadata read hostname 'smithi046' 2024-01-26T23:48:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.559+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/1756029538 >> [v2:172.21.15.46:6833/783623731,v1:172.21.15.46:6835/783623731] conn(0x557924e26190 0x557924e26560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.559+0000 7f4c741c0b80 4 client.7965 mds_command: new command op to 7794 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-26T23:48:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.559+0000 7f4c741c0b80 1 -- 172.21.15.46:0/1756029538 --> [v2:172.21.15.46:6833/783623731,v1:172.21.15.46:6835/783623731] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557924cd2ee0 con 0x557924e26190 2024-01-26T23:48:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.560+0000 7f4c68c59700 1 --2- 172.21.15.46:0/1756029538 >> [v2:172.21.15.46:6833/783623731,v1:172.21.15.46:6835/783623731] conn(0x557924e26190 0x557924e26560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.560+0000 7f4c42ffd700 10 client.7965 ms_handle_connect on v2:172.21.15.46:6833/783623731 2024-01-26T23:48:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.560+0000 7f4c68c59700 1 --2- 172.21.15.46:0/1756029538 >> [v2:172.21.15.46:6833/783623731,v1:172.21.15.46:6835/783623731] conn(0x557924e26190 0x557924e26560 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c9b9ae4653f1924 server_cookie=7636ebcb468b0cd4 in_seq=0 out_seq=0 2024-01-26T23:48:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.561+0000 7f4c42ffd700 1 -- 172.21.15.46:0/1756029538 <== mds.0 v2:172.21.15.46:6833/783623731 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x557924cd2ee0 con 0x557924e26190 2024-01-26T23:48:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.561+0000 7f4c42ffd700 10 client.7965 handle_command_reply: tid=0 2024-01-26T23:48:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.561+0000 7f4c741c0b80 1 client.7965 shutdown 2024-01-26T23:48:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.561+0000 7f4c741c0b80 1 -- 172.21.15.46:0/1756029538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e2a360 msgr2=0x557924e08270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.561+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/1756029538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e2a360 0x557924e08270 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00b610 tx=0x7f4c5c018000 comp rx=0 tx=0).stop 2024-01-26T23:48:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.561+0000 7f4c741c0b80 1 -- 172.21.15.46:0/1756029538 shutdown_connections 2024-01-26T23:48:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.561+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/1756029538 >> [v2:172.21.15.46:6833/783623731,v1:172.21.15.46:6835/783623731] conn(0x557924e26190 0x557924e26560 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.561+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/1756029538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e2a360 0x557924e08270 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.561+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/1756029538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924de1e40 0x557924e0efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.561+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/1756029538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924e09740 0x557924e22210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.561+0000 7f4c42ffd700 0 client.7965 ms_handle_reset on v2:172.21.15.46:6833/783623731 2024-01-26T23:48:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.561+0000 7f4c741c0b80 1 -- 172.21.15.46:0/1756029538 >> 172.21.15.46:0/1756029538 conn(0x557924de1a70 msgr2=0x557924ddcfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.561+0000 7f4c741c0b80 1 -- 172.21.15.46:0/1756029538 shutdown_connections 2024-01-26T23:48:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.561+0000 7f4c741c0b80 1 -- 172.21.15.46:0/1756029538 wait complete. 2024-01-26T23:48:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.561+0000 7f4c741c0b80 20 client.7965 trim_cache size 0 max 16384 2024-01-26T23:48:22.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.579+0000 7f4c741c0b80 1 Processor -- start 2024-01-26T23:48:22.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.579+0000 7f4c741c0b80 1 -- start start 2024-01-26T23:48:22.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.580+0000 7f4c741c0b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924de1e40 0x557924e838f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.580+0000 7f4c741c0b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e09740 0x557924e23710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.580+0000 7f4c741c0b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924e2a360 0x557924ddd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.580+0000 7f4c741c0b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x557924c28e40 con 0x557924e09740 2024-01-26T23:48:22.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.580+0000 7f4c741c0b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x557924ddac50 con 0x557924de1e40 2024-01-26T23:48:22.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.580+0000 7f4c741c0b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x557924e28b40 con 0x557924e2a360 2024-01-26T23:48:22.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.580+0000 7f4c63fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e09740 0x557924e23710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.580+0000 7f4c63fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e09740 0x557924e23710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48554/0 (socket says 172.21.15.46:48554) 2024-01-26T23:48:22.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.580+0000 7f4c63fff700 1 -- 172.21.15.46:0/2244243185 learned_addr learned my addr 172.21.15.46:0/2244243185 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:22.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.580+0000 7f4c68c59700 1 --2- 172.21.15.46:0/2244243185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924de1e40 0x557924e838f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.581+0000 7f4c6945a700 1 --2- 172.21.15.46:0/2244243185 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924e2a360 0x557924ddd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.581+0000 7f4c63fff700 1 -- 172.21.15.46:0/2244243185 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924e2a360 msgr2=0x557924ddd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:22.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.581+0000 7f4c63fff700 1 --2- 172.21.15.46:0/2244243185 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924e2a360 0x557924ddd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.581+0000 7f4c63fff700 1 -- 172.21.15.46:0/2244243185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924de1e40 msgr2=0x557924e838f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:22.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.581+0000 7f4c63fff700 1 --2- 172.21.15.46:0/2244243185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924de1e40 0x557924e838f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.581+0000 7f4c63fff700 1 -- 172.21.15.46:0/2244243185 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557924e24d20 con 0x557924e09740 2024-01-26T23:48:22.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.581+0000 7f4c63fff700 1 --2- 172.21.15.46:0/2244243185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e09740 0x557924e23710 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00f010 tx=0x7f4c4c05a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d5d9257df0b674 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:22.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.581+0000 7f4c5a7fc700 1 -- 172.21.15.46:0/2244243185 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c057ec0 con 0x557924e09740 2024-01-26T23:48:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.581+0000 7f4c5a7fc700 1 -- 172.21.15.46:0/2244243185 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c057510 con 0x557924e09740 2024-01-26T23:48:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.581+0000 7f4c5a7fc700 1 -- 172.21.15.46:0/2244243185 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c057740 con 0x557924e09740 2024-01-26T23:48:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.582+0000 7f4c741c0b80 1 -- 172.21.15.46:0/2244243185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e09740 msgr2=0x557924e23710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.582+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/2244243185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e09740 0x557924e23710 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00f010 tx=0x7f4c4c05a040 comp rx=0 tx=0).stop 2024-01-26T23:48:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.582+0000 7f4c741c0b80 1 -- 172.21.15.46:0/2244243185 shutdown_connections 2024-01-26T23:48:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.582+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/2244243185 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924e2a360 0x557924ddd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.582+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/2244243185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924de1e40 0x557924e838f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.582+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/2244243185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924e09740 0x557924e23710 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.582+0000 7f4c741c0b80 1 -- 172.21.15.46:0/2244243185 >> 172.21.15.46:0/2244243185 conn(0x557924de1a70 msgr2=0x557924de2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.582+0000 7f4c741c0b80 1 -- 172.21.15.46:0/2244243185 shutdown_connections 2024-01-26T23:48:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.582+0000 7f4c741c0b80 1 -- 172.21.15.46:0/2244243185 wait complete. 2024-01-26T23:48:22.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.582+0000 7f4c741c0b80 1 -- start start 2024-01-26T23:48:22.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.583+0000 7f4c741c0b80 1 Processor -- start 2024-01-26T23:48:22.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.583+0000 7f4c741c0b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924de1e40 0x557924e2ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.583+0000 7f4c741c0b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924e09740 0x557924e901b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.583+0000 7f4c741c0b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924e2a360 0x557924e906f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.583+0000 7f4c741c0b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x557924e44500 con 0x557924de1e40 2024-01-26T23:48:22.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.583+0000 7f4c741c0b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x557924e44740 con 0x557924e2a360 2024-01-26T23:48:22.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.583+0000 7f4c741c0b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x557924e829a0 con 0x557924e09740 2024-01-26T23:48:22.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.583+0000 7f4c741c0b80 10 client.0 _ll_register_callbacks cb 0x557924b3e7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:48:22.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c68c59700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924de1e40 0x557924e2ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c68c59700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924de1e40 0x557924e2ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48566/0 (socket says 172.21.15.46:48566) 2024-01-26T23:48:22.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c68c59700 1 -- 172.21.15.46:0/804709195 learned_addr learned my addr 172.21.15.46:0/804709195 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:22.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c63fff700 1 --2- 172.21.15.46:0/804709195 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924e09740 0x557924e901b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c6945a700 1 --2- 172.21.15.46:0/804709195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924e2a360 0x557924e906f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c68c59700 1 -- 172.21.15.46:0/804709195 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924e09740 msgr2=0x557924e901b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:22.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c68c59700 1 --2- 172.21.15.46:0/804709195 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924e09740 0x557924e901b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c68c59700 1 -- 172.21.15.46:0/804709195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924e2a360 msgr2=0x557924e906f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:22.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c68c59700 1 --2- 172.21.15.46:0/804709195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924e2a360 0x557924e906f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c68c59700 1 -- 172.21.15.46:0/804709195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c640b1120 con 0x557924de1e40 2024-01-26T23:48:22.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c6945a700 1 --2- 172.21.15.46:0/804709195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924e2a360 0x557924e906f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:22.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c68c59700 1 --2- 172.21.15.46:0/804709195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924de1e40 0x557924e2ab10 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f4c54008000 tx=0x7f4c54030040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44ddd97dee6da5a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:22.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c42ffd700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:48:22.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c42ffd700 1 -- 172.21.15.46:0/804709195 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5400f840 con 0x557924de1e40 2024-01-26T23:48:22.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c42ffd700 1 -- 172.21.15.46:0/804709195 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f4c5401b050 con 0x557924de1e40 2024-01-26T23:48:22.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c741c0b80 1 -- 172.21.15.46:0/804709195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c6416a9e0 con 0x557924de1e40 2024-01-26T23:48:22.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c42ffd700 1 -- 172.21.15.46:0/804709195 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c54031130 con 0x557924de1e40 2024-01-26T23:48:22.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c741c0b80 10 client.7971 fetch_fsmap learned FSMap version 62 2024-01-26T23:48:22.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.584+0000 7f4c741c0b80 1 -- 172.21.15.46:0/804709195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) v3 -- 0x557924bc7c80 con 0x557924de1e40 2024-01-26T23:48:22.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.585+0000 7f4c42ffd700 1 -- 172.21.15.46:0/804709195 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c54041070 con 0x557924de1e40 2024-01-26T23:48:22.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.585+0000 7f4c42ffd700 1 -- 172.21.15.46:0/804709195 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 62) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f4c54015be0 con 0x557924de1e40 2024-01-26T23:48:22.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.585+0000 7f4c741c0b80 10 client.7971 fetch_fsmap finished waiting for FSMap version 62 2024-01-26T23:48:22.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.585+0000 7f4c741c0b80 10 client.7971 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-26T23:48:22.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.585+0000 7f4c741c0b80 20 client.7971 populate_metadata read hostname 'smithi046' 2024-01-26T23:48:22.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.585+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/804709195 >> [v2:172.21.15.46:6833/783623731,v1:172.21.15.46:6835/783623731] conn(0x557924e1cfb0 0x557924e1d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.585+0000 7f4c741c0b80 4 client.7971 mds_command: new command op to 7794 tid=0[{"prefix": "status"}] 2024-01-26T23:48:22.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.585+0000 7f4c741c0b80 1 -- 172.21.15.46:0/804709195 --> [v2:172.21.15.46:6833/783623731,v1:172.21.15.46:6835/783623731] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f4c54013bf0 con 0x557924e1cfb0 2024-01-26T23:48:22.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.585+0000 7f4c63fff700 1 --2- 172.21.15.46:0/804709195 >> [v2:172.21.15.46:6833/783623731,v1:172.21.15.46:6835/783623731] conn(0x557924e1cfb0 0x557924e1d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.585+0000 7f4c42ffd700 10 client.7971 ms_handle_connect on v2:172.21.15.46:6833/783623731 2024-01-26T23:48:22.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.585+0000 7f4c63fff700 1 --2- 172.21.15.46:0/804709195 >> [v2:172.21.15.46:6833/783623731,v1:172.21.15.46:6835/783623731] conn(0x557924e1cfb0 0x557924e1d380 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb7b1cee31683c89 server_cookie=4911ca013986c96 in_seq=0 out_seq=0 2024-01-26T23:48:22.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.585+0000 7f4c42ffd700 1 -- 172.21.15.46:0/804709195 <== mds.0 v2:172.21.15.46:6833/783623731 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x7f4c54013bf0 con 0x557924e1cfb0 2024-01-26T23:48:22.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.585+0000 7f4c42ffd700 10 client.7971 handle_command_reply: tid=0 2024-01-26T23:48:22.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.585+0000 7f4c741c0b80 1 client.7971 shutdown 2024-01-26T23:48:22.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.586+0000 7f4c741c0b80 1 -- 172.21.15.46:0/804709195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924de1e40 msgr2=0x557924e2ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:22.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.586+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/804709195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924de1e40 0x557924e2ab10 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f4c54008000 tx=0x7f4c54030040 comp rx=0 tx=0).stop 2024-01-26T23:48:22.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.586+0000 7f4c741c0b80 1 -- 172.21.15.46:0/804709195 shutdown_connections 2024-01-26T23:48:22.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.586+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/804709195 >> [v2:172.21.15.46:6833/783623731,v1:172.21.15.46:6835/783623731] conn(0x557924e1cfb0 0x557924e1d380 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.586+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/804709195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557924de1e40 0x557924e2ab10 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.586+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/804709195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557924e2a360 0x557924e906f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.586+0000 7f4c741c0b80 1 --2- 172.21.15.46:0/804709195 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x557924e09740 0x557924e901b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.586+0000 7f4c42ffd700 0 client.7971 ms_handle_reset on v2:172.21.15.46:6833/783623731 2024-01-26T23:48:22.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.586+0000 7f4c741c0b80 1 -- 172.21.15.46:0/804709195 >> 172.21.15.46:0/804709195 conn(0x557924de1a70 msgr2=0x557924ddd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:22.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.586+0000 7f4c741c0b80 1 -- 172.21.15.46:0/804709195 shutdown_connections 2024-01-26T23:48:22.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.586+0000 7f4c741c0b80 1 -- 172.21.15.46:0/804709195 wait complete. 2024-01-26T23:48:22.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.586+0000 7f4c741c0b80 20 client.7971 trim_cache size 0 max 16384 2024-01-26T23:48:22.593 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:48:22.593 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-26T23:48:22.593 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-26T23:48:22.593 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 7794, 2024-01-26T23:48:22.593 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-26T23:48:22.593 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-26T23:48:22.593 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-01-26T23:48:22.593 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 10.793220778, 2024-01-26T23:48:22.593 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 62, 2024-01-26T23:48:22.594 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 130, 2024-01-26T23:48:22.594 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 130, 2024-01-26T23:48:22.594 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 16.770143592 2024-01-26T23:48:22.594 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:48:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.586+0000 7f4c6aebd700 1 -- 172.21.15.46:0/2804264004 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4c44066f30 msgr2=0x7f4c440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.586+0000 7f4c6aebd700 1 --2- 172.21.15.46:0/2804264004 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4c44066f30 0x7f4c440693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4c540011e0 tx=0x7f4c54001a00 comp rx=0 tx=0).stop 2024-01-26T23:48:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.586+0000 7f4c6aebd700 1 -- 172.21.15.46:0/2804264004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c640c4700 msgr2=0x7f4c640b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.586+0000 7f4c6aebd700 1 --2- 172.21.15.46:0/2804264004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c640c4700 0x7f4c640b03f0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c010ec0 tx=0x7f4c4c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.587+0000 7f4c6aebd700 1 -- 172.21.15.46:0/2804264004 shutdown_connections 2024-01-26T23:48:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.587+0000 7f4c6aebd700 1 --2- 172.21.15.46:0/2804264004 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4c44066f30 0x7f4c440693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.587+0000 7f4c6aebd700 1 --2- 172.21.15.46:0/2804264004 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c641565d0 0x7f4c6414e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.587+0000 7f4c6aebd700 1 --2- 172.21.15.46:0/2804264004 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c6406f4a0 0x7f4c640b21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.587+0000 7f4c6aebd700 1 --2- 172.21.15.46:0/2804264004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c640c4700 0x7f4c640b03f0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.587+0000 7f4c6aebd700 1 -- 172.21.15.46:0/2804264004 >> 172.21.15.46:0/2804264004 conn(0x7f4c6406b820 msgr2=0x7f4c640c6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.587+0000 7f4c6aebd700 1 -- 172.21.15.46:0/2804264004 shutdown_connections 2024-01-26T23:48:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.587+0000 7f4c6aebd700 1 -- 172.21.15.46:0/2804264004 wait complete. 2024-01-26T23:48:22.601 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-26T23:48:22.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.731+0000 7f0c90426700 1 Processor -- start 2024-01-26T23:48:22.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.731+0000 7f0c90426700 1 -- start start 2024-01-26T23:48:22.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.732+0000 7f0c90426700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0c88159760 0x7f0c880b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.732+0000 7f0c90426700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c880b12c0 0x7f0c880a9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.732+0000 7f0c90426700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c8806f940 con 0x7f0c880b3060 2024-01-26T23:48:22.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.732+0000 7f0c90426700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c8806ed30 con 0x7f0c880b12c0 2024-01-26T23:48:22.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.732+0000 7f0c90426700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c8806f010 con 0x7f0c88159760 2024-01-26T23:48:22.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.732+0000 7f0c8e9c3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c880b12c0 0x7f0c880a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.732+0000 7f0c8e9c3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c880b12c0 0x7f0c880a9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48570/0 (socket says 172.21.15.46:48570) 2024-01-26T23:48:22.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.732+0000 7f0c8e9c3700 1 -- 172.21.15.46:0/2744415532 learned_addr learned my addr 172.21.15.46:0/2744415532 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:22.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.732+0000 7f0c8e1c2700 1 --2- 172.21.15.46:0/2744415532 >> v2:172.21.15.106:3300/0 conn(0x7f0c88159760 0x7f0c880b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.733+0000 7f0c8e9c3700 1 -- 172.21.15.46:0/2744415532 >> v2:172.21.15.106:3300/0 conn(0x7f0c88159760 msgr2=0x7f0c880b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:22.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.733+0000 7f0c8e9c3700 1 --2- 172.21.15.46:0/2744415532 >> v2:172.21.15.106:3300/0 conn(0x7f0c88159760 0x7f0c880b2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.733+0000 7f0c8e9c3700 1 -- 172.21.15.46:0/2744415532 >> v1:172.21.15.106:6789/0 conn(0x7f0c880b3060 legacy=0x7f0c880b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:22.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.733+0000 7f0c8e9c3700 1 -- 172.21.15.46:0/2744415532 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8806f2f0 con 0x7f0c880b12c0 2024-01-26T23:48:22.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.733+0000 7f0c8e9c3700 1 --2- 172.21.15.46:0/2744415532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c880b12c0 0x7f0c880a9840 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f0c88076af0 tx=0x7f0c80009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e32625b22e8c5d server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:22.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.733+0000 7f0c8d1c0700 1 -- 172.21.15.46:0/2744415532 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8000e070 con 0x7f0c880b12c0 2024-01-26T23:48:22.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.733+0000 7f0c8d1c0700 1 -- 172.21.15.46:0/2744415532 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8000ba20 con 0x7f0c880b12c0 2024-01-26T23:48:22.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.733+0000 7f0c8d1c0700 1 -- 172.21.15.46:0/2744415532 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c800092e0 con 0x7f0c880b12c0 2024-01-26T23:48:22.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.734+0000 7f0c90426700 1 -- 172.21.15.46:0/2744415532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c880b12c0 msgr2=0x7f0c880a9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:22.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.734+0000 7f0c90426700 1 --2- 172.21.15.46:0/2744415532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c880b12c0 0x7f0c880a9840 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f0c88076af0 tx=0x7f0c80009840 comp rx=0 tx=0).stop 2024-01-26T23:48:22.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.734+0000 7f0c90426700 1 -- 172.21.15.46:0/2744415532 shutdown_connections 2024-01-26T23:48:22.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.734+0000 7f0c90426700 1 --2- 172.21.15.46:0/2744415532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c880b12c0 0x7f0c880a9840 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.734+0000 7f0c90426700 1 --2- 172.21.15.46:0/2744415532 >> v2:172.21.15.106:3300/0 conn(0x7f0c88159760 0x7f0c880b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.734+0000 7f0c90426700 1 -- 172.21.15.46:0/2744415532 >> 172.21.15.46:0/2744415532 conn(0x7f0c88069510 msgr2=0x7f0c880aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:22.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.734+0000 7f0c90426700 1 -- 172.21.15.46:0/2744415532 shutdown_connections 2024-01-26T23:48:22.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.734+0000 7f0c90426700 1 -- 172.21.15.46:0/2744415532 wait complete. 2024-01-26T23:48:22.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.735+0000 7f0c90426700 1 Processor -- start 2024-01-26T23:48:22.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.735+0000 7f0c90426700 1 -- start start 2024-01-26T23:48:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.736+0000 7f0c90426700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c880b12c0 0x7f0c88140730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.736+0000 7f0c90426700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c880b3060 0x7f0c8815ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.736+0000 7f0c90426700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c88159760 0x7f0c8815f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.736+0000 7f0c90426700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c88140c70 con 0x7f0c880b12c0 2024-01-26T23:48:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.736+0000 7f0c90426700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8816b390 con 0x7f0c880b3060 2024-01-26T23:48:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.736+0000 7f0c90426700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8816a780 con 0x7f0c88159760 2024-01-26T23:48:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.736+0000 7f0c8e1c2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c880b12c0 0x7f0c88140730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.736+0000 7f0c8e1c2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c880b12c0 0x7f0c88140730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48586/0 (socket says 172.21.15.46:48586) 2024-01-26T23:48:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.736+0000 7f0c8e1c2700 1 -- 172.21.15.46:0/1680841644 learned_addr learned my addr 172.21.15.46:0/1680841644 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.736+0000 7f0c8e9c3700 1 --2- 172.21.15.46:0/1680841644 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c88159760 0x7f0c8815f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.736+0000 7f0c8d9c1700 1 --2- 172.21.15.46:0/1680841644 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c880b3060 0x7f0c8815ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.736+0000 7f0c8e1c2700 1 -- 172.21.15.46:0/1680841644 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c88159760 msgr2=0x7f0c8815f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.736+0000 7f0c8e1c2700 1 --2- 172.21.15.46:0/1680841644 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c88159760 0x7f0c8815f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.736+0000 7f0c8e1c2700 1 -- 172.21.15.46:0/1680841644 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c880b3060 msgr2=0x7f0c8815ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.736+0000 7f0c8e1c2700 1 --2- 172.21.15.46:0/1680841644 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c880b3060 0x7f0c8815ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.736+0000 7f0c8e1c2700 1 -- 172.21.15.46:0/1680841644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c80009ae0 con 0x7f0c880b12c0 2024-01-26T23:48:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.736+0000 7f0c8d9c1700 1 --2- 172.21.15.46:0/1680841644 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c880b3060 0x7f0c8815ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.737+0000 7f0c8e1c2700 1 --2- 172.21.15.46:0/1680841644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c880b12c0 0x7f0c88140730 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f0c8400e960 tx=0x7f0c840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.737+0000 7f0c7e7fc700 1 -- 172.21.15.46:0/1680841644 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c84013680 con 0x7f0c880b12c0 2024-01-26T23:48:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.737+0000 7f0c7e7fc700 1 -- 172.21.15.46:0/1680841644 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c840137e0 con 0x7f0c880b12c0 2024-01-26T23:48:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.737+0000 7f0c90426700 1 -- 172.21.15.46:0/1680841644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8816aa60 con 0x7f0c880b12c0 2024-01-26T23:48:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.737+0000 7f0c7e7fc700 1 -- 172.21.15.46:0/1680841644 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c84013b80 con 0x7f0c880b12c0 2024-01-26T23:48:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.737+0000 7f0c90426700 1 -- 172.21.15.46:0/1680841644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8816b020 con 0x7f0c880b12c0 2024-01-26T23:48:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.738+0000 7f0c90426700 1 -- 172.21.15.46:0/1680841644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c8804c710 con 0x7f0c880b12c0 2024-01-26T23:48:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.738+0000 7f0c7e7fc700 1 -- 172.21.15.46:0/1680841644 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0c8400c490 con 0x7f0c880b12c0 2024-01-26T23:48:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.739+0000 7f0c7e7fc700 1 --2- 172.21.15.46:0/1680841644 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0c6c066f30 0x7f0c6c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.739+0000 7f0c7e7fc700 1 -- 172.21.15.46:0/1680841644 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f0c84084970 con 0x7f0c880b12c0 2024-01-26T23:48:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.739+0000 7f0c8d9c1700 1 --2- 172.21.15.46:0/1680841644 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0c6c066f30 0x7f0c6c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.739+0000 7f0c8d9c1700 1 --2- 172.21.15.46:0/1680841644 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0c6c066f30 0x7f0c6c0693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0c78007fd0 tx=0x7f0c78009040 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:22.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:22.743+0000 7f0c7e7fc700 1 -- 172.21.15.46:0/1680841644 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0c84051910 con 0x7f0c880b12c0 2024-01-26T23:48:23.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:23.003+0000 7f0c90426700 1 -- 172.21.15.46:0/1680841644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0c880a9bc0 con 0x7f0c880b12c0 2024-01-26T23:48:23.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:23.004+0000 7f0c7e7fc700 1 -- 172.21.15.46:0/1680841644 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v62) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0c84054f30 con 0x7f0c880b12c0 2024-01-26T23:48:23.005 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-26T23:48:23.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:23.006+0000 7f0c90426700 1 -- 172.21.15.46:0/1680841644 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0c6c066f30 msgr2=0x7f0c6c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:23.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:23.006+0000 7f0c90426700 1 --2- 172.21.15.46:0/1680841644 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0c6c066f30 0x7f0c6c0693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0c78007fd0 tx=0x7f0c78009040 comp rx=0 tx=0).stop 2024-01-26T23:48:23.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:23.006+0000 7f0c90426700 1 -- 172.21.15.46:0/1680841644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c880b12c0 msgr2=0x7f0c88140730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:23.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:23.006+0000 7f0c90426700 1 --2- 172.21.15.46:0/1680841644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c880b12c0 0x7f0c88140730 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f0c8400e960 tx=0x7f0c840091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:23.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:23.006+0000 7f0c90426700 1 -- 172.21.15.46:0/1680841644 shutdown_connections 2024-01-26T23:48:23.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:23.006+0000 7f0c90426700 1 --2- 172.21.15.46:0/1680841644 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0c6c066f30 0x7f0c6c0693e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:23.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:23.006+0000 7f0c90426700 1 --2- 172.21.15.46:0/1680841644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c880b12c0 0x7f0c88140730 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:23.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:23.006+0000 7f0c90426700 1 --2- 172.21.15.46:0/1680841644 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c88159760 0x7f0c8815f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:23.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:23.006+0000 7f0c90426700 1 --2- 172.21.15.46:0/1680841644 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c880b3060 0x7f0c8815ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:23.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:23.006+0000 7f0c90426700 1 -- 172.21.15.46:0/1680841644 >> 172.21.15.46:0/1680841644 conn(0x7f0c88069510 msgr2=0x7f0c880aeb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:23.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:23.006+0000 7f0c90426700 1 -- 172.21.15.46:0/1680841644 shutdown_connections 2024-01-26T23:48:23.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:23.006+0000 7f0c90426700 1 -- 172.21.15.46:0/1680841644 wait complete. 2024-01-26T23:48:23.021 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-26T23:48:23.021 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-26T23:48:23.021 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-26T23:48:23.021 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-26T23:48:23.021 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-26T23:48:23.021 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-26T23:48:23.021 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-26T23:48:23.022 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-26T23:48:23.022 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:48:23.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:48:23.022 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-26T23:48:23.041 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-26T23:48:23.042 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-26T23:48:23.042 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-26T23:48:23.042 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:48:23.042 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-26T23:48:23.042 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:48:23.042 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:48:23.042 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:48:23.042 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-26T23:48:23.042 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:48:23.042 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:48:23.042 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-26T23:48:23.042 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-26T23:48:23.042 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:48:23.043 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-26T23:48:23.043 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-26T23:48:23.043 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 41227sec preferred_lft 41227sec 2024-01-26T23:48:23.043 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-26T23:48:23.043 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:48:23.043 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:48:23.043 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:48:23.043 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-26T23:48:23.043 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-26T23:48:23.043 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:48:23.043 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-26T23:48:23.043 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:48:23.043 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::cce6:a7ff:fe4f:38d/64 scope link 2024-01-26T23:48:23.044 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:48:23.044 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:48:23.044 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 76:ef:90:4c:9d:6f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-26T23:48:23.044 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::74ef:90ff:fe4c:9d6f/64 scope link 2024-01-26T23:48:23.044 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:48:23.044 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:48:23.044 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-26T23:48:23.044 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::2c73:afff:fef0:6f39/64 scope link 2024-01-26T23:48:23.044 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:48:23.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:48:23.045 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-26T23:48:23.112 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-26T23:48:23.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:48:23.114 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-26T23:48:23.178 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-26T23:48:23.178 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-26T23:48:23.178 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-26T23:48:23.178 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-26T23:48:23.178 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-26T23:48:23.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:48:23.178 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:48:23.179 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:48:23.179 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-26T23:48:23.179 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-26T23:48:23.179 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-26T23:48:23.179 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:48:23.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:48:23.302 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-26T23:48:23.323 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-26T23:48:23.323 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-26T23:48:23.324 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:48:23.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:48:23.324 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-01-26T23:48:23.399 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-26T23:48:23.399 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:48:23.399 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-26T23:48:23.461 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:48:23.462 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:48:23.462 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:48:23.524 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:48:23.599 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:48:23.600 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:48:23.677 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:48:23.678 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:48:23.678 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:23.678 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:48:23.741 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-26T23:48:23.742 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-26T23:48:23.784 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:48:23.819 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:48:23.819 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:48:23.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-01-26T23:48:23.886+0000 7fef08997540 -1 init, newargv = 0x55e8ff5f4d90 newargc=15 2024-01-26T23:48:23.888 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[82900]: starting ceph client 2024-01-26T23:48:23.891 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:48:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:48:23.893 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:23.893 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:48:23.893 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[82900]: starting fuse 2024-01-26T23:48:23.912 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-26T23:48:23.912 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-26T23:48:23.913 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-26T23:48:23.913 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-26T23:48:23.913 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-26T23:48:23.913 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-26T23:48:23.913 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-26T23:48:23.913 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:48:23.913 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-26T23:48:23.913 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-26T23:48:23.913 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-26T23:48:23.913 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-26T23:48:23.913 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:48:23.913 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-26T23:48:23.913 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-26T23:48:23.913 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-26T23:48:23.913 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-26T23:48:23.913 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:48:23.914 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-26T23:48:23.914 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-26T23:48:23.914 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-26T23:48:23.914 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-26T23:48:23.914 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-26T23:48:23.914 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-26T23:48:23.914 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-26T23:48:23.914 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-26T23:48:23.914 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:48:23.914 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-01-26T23:48:23.914 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-26T23:48:24.051 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.82900.asok 2024-01-26T23:48:24.232 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.82900.asok 2024-01-26T23:48:24.232 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:24.232 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.82900.asok status 2024-01-26T23:48:24.406 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:48:24.406 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-26T23:48:24.406 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-26T23:48:24.406 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-26T23:48:24.406 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-01-26T23:48:24.407 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-26T23:48:24.407 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-26T23:48:24.407 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "82900", 2024-01-26T23:48:24.407 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-26T23:48:24.407 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:24.407 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-26T23:48:24.407 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-26T23:48:24.407 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 7756, 2024-01-26T23:48:24.407 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-26T23:48:24.407 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-26T23:48:24.407 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-26T23:48:24.407 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 7756 2024-01-26T23:48:24.407 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:24.407 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:48:24.407 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:48:24.407 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:48:24.408 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2930540717 2024-01-26T23:48:24.408 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:24.408 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:24.408 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:48:24.408 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:48:24.408 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:48:24.408 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2930540717 2024-01-26T23:48:24.408 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:48:24.408 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.7756 192.168.0.1:0/2930540717", 2024-01-26T23:48:24.408 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/2930540717", 2024-01-26T23:48:24.408 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-26T23:48:24.408 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 62, 2024-01-26T23:48:24.408 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 130, 2024-01-26T23:48:24.409 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-26T23:48:24.409 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-26T23:48:24.409 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-26T23:48:24.409 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:48:24.418 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:24.418 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-26T23:48:24.419 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-26T23:48:24.438 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-26T23:48:24.438 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-26T23:48:24.438 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:24.438 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:48:24.511 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-26T23:48:24.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.693+0000 7fa9291e8700 1 Processor -- start 2024-01-26T23:48:24.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.693+0000 7fa9291e8700 1 -- start start 2024-01-26T23:48:24.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.694+0000 7fa9291e8700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa92415ce40 0x7fa9240d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:24.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.694+0000 7fa9291e8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa92415d210 0x7fa9240cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:24.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.694+0000 7fa9291e8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9240cff70 con 0x7fa9240d1390 2024-01-26T23:48:24.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.694+0000 7fa9291e8700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9240cf400 con 0x7fa92415d210 2024-01-26T23:48:24.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.694+0000 7fa9291e8700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9240cf660 con 0x7fa92415ce40 2024-01-26T23:48:24.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.694+0000 7fa92259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa92415d210 0x7fa9240cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:24.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.694+0000 7fa92359e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa9240d1390 0x7fa9240ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47284/0 (socket says 172.21.15.46:47284) 2024-01-26T23:48:24.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.694+0000 7fa92359e700 1 -- 172.21.15.46:0/914515372 learned_addr learned my addr 172.21.15.46:0/914515372 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:24.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.694+0000 7fa922d9d700 1 --2- 172.21.15.46:0/914515372 >> v2:172.21.15.106:3300/0 conn(0x7fa92415ce40 0x7fa9240d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:24.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.694+0000 7fa92259c700 1 -- 172.21.15.46:0/914515372 >> v2:172.21.15.106:3300/0 conn(0x7fa92415ce40 msgr2=0x7fa9240d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:24.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.694+0000 7fa92259c700 1 --2- 172.21.15.46:0/914515372 >> v2:172.21.15.106:3300/0 conn(0x7fa92415ce40 0x7fa9240d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:24.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.694+0000 7fa92259c700 1 -- 172.21.15.46:0/914515372 >> v1:172.21.15.46:6789/0 conn(0x7fa9240d1390 legacy=0x7fa9240ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:24.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.694+0000 7fa92259c700 1 -- 172.21.15.46:0/914515372 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9240d0500 con 0x7fa92415d210 2024-01-26T23:48:24.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.694+0000 7fa921d9b700 1 -- 172.21.15.46:0/914515372 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222810017 0 0) 0x7fa9240cff70 con 0x7fa9240d1390 2024-01-26T23:48:24.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.694+0000 7fa92259c700 1 --2- 172.21.15.46:0/914515372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa92415d210 0x7fa9240cbab0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fa924088e90 tx=0x7fa9180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee94587160276c2b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:24.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.695+0000 7fa921d9b700 1 -- 172.21.15.46:0/914515372 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918011630 con 0x7fa92415d210 2024-01-26T23:48:24.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.695+0000 7fa921d9b700 1 -- 172.21.15.46:0/914515372 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa918011790 con 0x7fa92415d210 2024-01-26T23:48:24.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.695+0000 7fa921d9b700 1 -- 172.21.15.46:0/914515372 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918011b30 con 0x7fa92415d210 2024-01-26T23:48:24.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.695+0000 7fa9291e8700 1 -- 172.21.15.46:0/914515372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa92415d210 msgr2=0x7fa9240cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:24.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.695+0000 7fa9291e8700 1 --2- 172.21.15.46:0/914515372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa92415d210 0x7fa9240cbab0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fa924088e90 tx=0x7fa9180091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:24.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.695+0000 7fa9291e8700 1 -- 172.21.15.46:0/914515372 shutdown_connections 2024-01-26T23:48:24.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.695+0000 7fa9291e8700 1 --2- 172.21.15.46:0/914515372 >> v2:172.21.15.106:3300/0 conn(0x7fa92415ce40 0x7fa9240d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:24.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.695+0000 7fa9291e8700 1 --2- 172.21.15.46:0/914515372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa92415d210 0x7fa9240cbab0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:24.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.695+0000 7fa9291e8700 1 -- 172.21.15.46:0/914515372 >> 172.21.15.46:0/914515372 conn(0x7fa92406f840 msgr2=0x7fa9240d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:24.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.695+0000 7fa9291e8700 1 -- 172.21.15.46:0/914515372 shutdown_connections 2024-01-26T23:48:24.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.695+0000 7fa9291e8700 1 -- 172.21.15.46:0/914515372 wait complete. 2024-01-26T23:48:24.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.696+0000 7fa9291e8700 1 Processor -- start 2024-01-26T23:48:24.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.697+0000 7fa9291e8700 1 -- start start 2024-01-26T23:48:24.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.697+0000 7fa9291e8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9240d1390 0x7fa92409bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:24.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.697+0000 7fa9291e8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa92415ce40 0x7fa9240c9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:24.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.697+0000 7fa9291e8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa92415d210 0x7fa9240c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:24.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.697+0000 7fa9291e8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9240c7750 con 0x7fa9240d1390 2024-01-26T23:48:24.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.697+0000 7fa9291e8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92409c5b0 con 0x7fa92415ce40 2024-01-26T23:48:24.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.697+0000 7fa9291e8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92409c890 con 0x7fa92415d210 2024-01-26T23:48:24.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.697+0000 7fa922d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9240d1390 0x7fa92409bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:24.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.697+0000 7fa922d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9240d1390 0x7fa92409bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48604/0 (socket says 172.21.15.46:48604) 2024-01-26T23:48:24.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.697+0000 7fa922d9d700 1 -- 172.21.15.46:0/2394012262 learned_addr learned my addr 172.21.15.46:0/2394012262 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:24.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.697+0000 7fa92259c700 1 --2- 172.21.15.46:0/2394012262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa92415ce40 0x7fa9240c9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:24.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.698+0000 7fa92359e700 1 --2- 172.21.15.46:0/2394012262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa92415d210 0x7fa9240c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:24.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.698+0000 7fa922d9d700 1 -- 172.21.15.46:0/2394012262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa92415d210 msgr2=0x7fa9240c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:24.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.698+0000 7fa922d9d700 1 --2- 172.21.15.46:0/2394012262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa92415d210 0x7fa9240c5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:24.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.698+0000 7fa922d9d700 1 -- 172.21.15.46:0/2394012262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa92415ce40 msgr2=0x7fa9240c9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:24.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.698+0000 7fa922d9d700 1 --2- 172.21.15.46:0/2394012262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa92415ce40 0x7fa9240c9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:24.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.698+0000 7fa922d9d700 1 -- 172.21.15.46:0/2394012262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa918009420 con 0x7fa9240d1390 2024-01-26T23:48:24.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.698+0000 7fa92359e700 1 --2- 172.21.15.46:0/2394012262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa92415d210 0x7fa9240c5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:48:24.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.698+0000 7fa922d9d700 1 --2- 172.21.15.46:0/2394012262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9240d1390 0x7fa92409bc30 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fa91400ec80 tx=0x7fa914009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:24.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.698+0000 7fa90affd700 1 -- 172.21.15.46:0/2394012262 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91400c850 con 0x7fa9240d1390 2024-01-26T23:48:24.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.698+0000 7fa90affd700 1 -- 172.21.15.46:0/2394012262 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91400c9b0 con 0x7fa9240d1390 2024-01-26T23:48:24.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.698+0000 7fa9291e8700 1 -- 172.21.15.46:0/2394012262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa92409cb70 con 0x7fa9240d1390 2024-01-26T23:48:24.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.698+0000 7fa90affd700 1 -- 172.21.15.46:0/2394012262 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91400cd30 con 0x7fa9240d1390 2024-01-26T23:48:24.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.698+0000 7fa9291e8700 1 -- 172.21.15.46:0/2394012262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa92409c170 con 0x7fa9240d1390 2024-01-26T23:48:24.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.699+0000 7fa9291e8700 1 -- 172.21.15.46:0/2394012262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9240c7d10 con 0x7fa9240d1390 2024-01-26T23:48:24.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.700+0000 7fa90affd700 1 -- 172.21.15.46:0/2394012262 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa914013680 con 0x7fa9240d1390 2024-01-26T23:48:24.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.700+0000 7fa90affd700 1 --2- 172.21.15.46:0/2394012262 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa910066f30 0x7fa9100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:24.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.700+0000 7fa90affd700 1 -- 172.21.15.46:0/2394012262 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fa914084e50 con 0x7fa9240d1390 2024-01-26T23:48:24.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.700+0000 7fa92259c700 1 --2- 172.21.15.46:0/2394012262 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa910066f30 0x7fa9100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:24.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.701+0000 7fa92259c700 1 --2- 172.21.15.46:0/2394012262 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa910066f30 0x7fa9100693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa924088e90 tx=0x7fa918017000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:24.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.704+0000 7fa90affd700 1 -- 172.21.15.46:0/2394012262 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa914051df0 con 0x7fa9240d1390 2024-01-26T23:48:24.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.940+0000 7fa9291e8700 1 -- 172.21.15.46:0/2394012262 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa92404c710 con 0x7fa910066f30 2024-01-26T23:48:24.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.942+0000 7fa90affd700 1 -- 172.21.15.46:0/2394012262 <== mgr.7396 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fa92404c710 con 0x7fa910066f30 2024-01-26T23:48:24.943 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-26T23:48:24.943 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:48:24.943 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephfs" 2024-01-26T23:48:24.943 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:48:24.943 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-26T23:48:24.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.944+0000 7fa9291e8700 1 -- 172.21.15.46:0/2394012262 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa910066f30 msgr2=0x7fa9100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:24.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.944+0000 7fa9291e8700 1 --2- 172.21.15.46:0/2394012262 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa910066f30 0x7fa9100693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa924088e90 tx=0x7fa918017000 comp rx=0 tx=0).stop 2024-01-26T23:48:24.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.944+0000 7fa9291e8700 1 -- 172.21.15.46:0/2394012262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9240d1390 msgr2=0x7fa92409bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:24.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.944+0000 7fa9291e8700 1 --2- 172.21.15.46:0/2394012262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9240d1390 0x7fa92409bc30 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fa91400ec80 tx=0x7fa914009120 comp rx=0 tx=0).stop 2024-01-26T23:48:24.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.944+0000 7fa9291e8700 1 -- 172.21.15.46:0/2394012262 shutdown_connections 2024-01-26T23:48:24.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.944+0000 7fa9291e8700 1 --2- 172.21.15.46:0/2394012262 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa910066f30 0x7fa9100693e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:24.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.944+0000 7fa9291e8700 1 --2- 172.21.15.46:0/2394012262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9240d1390 0x7fa92409bc30 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:24.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.944+0000 7fa9291e8700 1 --2- 172.21.15.46:0/2394012262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa92415d210 0x7fa9240c5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:24.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.944+0000 7fa9291e8700 1 --2- 172.21.15.46:0/2394012262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa92415ce40 0x7fa9240c9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:24.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.944+0000 7fa9291e8700 1 -- 172.21.15.46:0/2394012262 >> 172.21.15.46:0/2394012262 conn(0x7fa92406f840 msgr2=0x7fa92406a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:24.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.944+0000 7fa9291e8700 1 -- 172.21.15.46:0/2394012262 shutdown_connections 2024-01-26T23:48:24.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:24.944+0000 7fa9291e8700 1 -- 172.21.15.46:0/2394012262 wait complete. 2024-01-26T23:48:24.959 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-26T23:48:25.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.091+0000 7fbc47d8a700 1 Processor -- start 2024-01-26T23:48:25.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.091+0000 7fbc47d8a700 1 -- start start 2024-01-26T23:48:25.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.092+0000 7fbc47d8a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc400cc540 0x7fbc400cc910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:25.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.092+0000 7fbc47d8a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbc40068bb0 0x7fbc40069020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:25.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.092+0000 7fbc47d8a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4015e0f0 con 0x7fbc4015d560 2024-01-26T23:48:25.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.092+0000 7fbc47d8a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc400c8c80 con 0x7fbc40068bb0 2024-01-26T23:48:25.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.092+0000 7fbc47d8a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc40069920 con 0x7fbc400cc540 2024-01-26T23:48:25.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.092+0000 7fbc45325700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbc40068bb0 0x7fbc40069020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:25.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.092+0000 7fbc46327700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbc4015d560 0x7fbc4015d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47292/0 (socket says 172.21.15.46:47292) 2024-01-26T23:48:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.092+0000 7fbc46327700 1 -- 172.21.15.46:0/3747327426 learned_addr learned my addr 172.21.15.46:0/3747327426 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.092+0000 7fbc45b26700 1 --2- 172.21.15.46:0/3747327426 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc400cc540 0x7fbc400cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.092+0000 7fbc45325700 1 -- 172.21.15.46:0/3747327426 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc400cc540 msgr2=0x7fbc400cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.092+0000 7fbc45325700 1 --2- 172.21.15.46:0/3747327426 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc400cc540 0x7fbc400cc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.092+0000 7fbc45325700 1 -- 172.21.15.46:0/3747327426 >> v1:172.21.15.46:6789/0 conn(0x7fbc4015d560 legacy=0x7fbc4015d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.092+0000 7fbc44b24700 1 -- 172.21.15.46:0/3747327426 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992669171 0 0) 0x7fbc4015e0f0 con 0x7fbc4015d560 2024-01-26T23:48:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.093+0000 7fbc45325700 1 -- 172.21.15.46:0/3747327426 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc40069c00 con 0x7fbc40068bb0 2024-01-26T23:48:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.093+0000 7fbc45325700 1 --2- 172.21.15.46:0/3747327426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc40068bb0 0x7fbc40069020 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009910 tx=0x7fbc300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcff6e52174d8e87 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.093+0000 7fbc44b24700 1 -- 172.21.15.46:0/3747327426 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3000b820 con 0x7fbc40068bb0 2024-01-26T23:48:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.093+0000 7fbc44b24700 1 -- 172.21.15.46:0/3747327426 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3000b980 con 0x7fbc40068bb0 2024-01-26T23:48:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.093+0000 7fbc44b24700 1 -- 172.21.15.46:0/3747327426 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3000bd20 con 0x7fbc40068bb0 2024-01-26T23:48:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.093+0000 7fbc47d8a700 1 -- 172.21.15.46:0/3747327426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc40068bb0 msgr2=0x7fbc40069020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.093+0000 7fbc47d8a700 1 --2- 172.21.15.46:0/3747327426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc40068bb0 0x7fbc40069020 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009910 tx=0x7fbc300091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:25.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.093+0000 7fbc47d8a700 1 -- 172.21.15.46:0/3747327426 shutdown_connections 2024-01-26T23:48:25.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.093+0000 7fbc47d8a700 1 --2- 172.21.15.46:0/3747327426 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc400cc540 0x7fbc400cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:25.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.093+0000 7fbc47d8a700 1 --2- 172.21.15.46:0/3747327426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc40068bb0 0x7fbc40069020 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:25.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.093+0000 7fbc47d8a700 1 -- 172.21.15.46:0/3747327426 >> 172.21.15.46:0/3747327426 conn(0x7fbc4006bd90 msgr2=0x7fbc4006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:25.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.094+0000 7fbc47d8a700 1 -- 172.21.15.46:0/3747327426 shutdown_connections 2024-01-26T23:48:25.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.094+0000 7fbc47d8a700 1 -- 172.21.15.46:0/3747327426 wait complete. 2024-01-26T23:48:25.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.095+0000 7fbc47d8a700 1 Processor -- start 2024-01-26T23:48:25.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.095+0000 7fbc47d8a700 1 -- start start 2024-01-26T23:48:25.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc47d8a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc40068bb0 0x7fbc400b2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:25.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc47d8a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc400cc540 0x7fbc4016e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:25.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc47d8a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc4015d560 0x7fbc400c4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:25.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc47d8a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc400b17b0 con 0x7fbc40068bb0 2024-01-26T23:48:25.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc47d8a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc400b1a90 con 0x7fbc400cc540 2024-01-26T23:48:25.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc47d8a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc400b1d70 con 0x7fbc4015d560 2024-01-26T23:48:25.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc45b26700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc40068bb0 0x7fbc400b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:25.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc45b26700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc40068bb0 0x7fbc400b2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48628/0 (socket says 172.21.15.46:48628) 2024-01-26T23:48:25.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc45b26700 1 -- 172.21.15.46:0/3825432870 learned_addr learned my addr 172.21.15.46:0/3825432870 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:25.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc45325700 1 --2- 172.21.15.46:0/3825432870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc400cc540 0x7fbc4016e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:25.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc45b26700 1 -- 172.21.15.46:0/3825432870 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc4015d560 msgr2=0x7fbc400c4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:25.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc46327700 1 --2- 172.21.15.46:0/3825432870 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc4015d560 0x7fbc400c4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:25.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc45b26700 1 --2- 172.21.15.46:0/3825432870 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc4015d560 0x7fbc400c4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:25.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc45b26700 1 -- 172.21.15.46:0/3825432870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc400cc540 msgr2=0x7fbc4016e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:25.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc45b26700 1 --2- 172.21.15.46:0/3825432870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc400cc540 0x7fbc4016e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:25.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc45b26700 1 -- 172.21.15.46:0/3825432870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc30009420 con 0x7fbc40068bb0 2024-01-26T23:48:25.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc45325700 1 --2- 172.21.15.46:0/3825432870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc400cc540 0x7fbc4016e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:48:25.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc46327700 1 --2- 172.21.15.46:0/3825432870 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc4015d560 0x7fbc400c4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:48:25.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc45b26700 1 --2- 172.21.15.46:0/3825432870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc40068bb0 0x7fbc400b2850 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00d890 tx=0x7fbc3c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:25.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc35ffb700 1 -- 172.21.15.46:0/3825432870 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c00b600 con 0x7fbc40068bb0 2024-01-26T23:48:25.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc35ffb700 1 -- 172.21.15.46:0/3825432870 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3c00b760 con 0x7fbc40068bb0 2024-01-26T23:48:25.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.096+0000 7fbc47d8a700 1 -- 172.21.15.46:0/3825432870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc400b2050 con 0x7fbc40068bb0 2024-01-26T23:48:25.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.097+0000 7fbc35ffb700 1 -- 172.21.15.46:0/3825432870 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c00bae0 con 0x7fbc40068bb0 2024-01-26T23:48:25.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.097+0000 7fbc47d8a700 1 -- 172.21.15.46:0/3825432870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc400b1340 con 0x7fbc40068bb0 2024-01-26T23:48:25.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.098+0000 7fbc47d8a700 1 -- 172.21.15.46:0/3825432870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc400b1560 con 0x7fbc40068bb0 2024-01-26T23:48:25.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.098+0000 7fbc35ffb700 1 -- 172.21.15.46:0/3825432870 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 55) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbc3c01e460 con 0x7fbc40068bb0 2024-01-26T23:48:25.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.099+0000 7fbc35ffb700 1 --2- 172.21.15.46:0/3825432870 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbc24066f30 0x7fbc240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:25.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.099+0000 7fbc35ffb700 1 -- 172.21.15.46:0/3825432870 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fbc3c084f50 con 0x7fbc40068bb0 2024-01-26T23:48:25.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.099+0000 7fbc45325700 1 --2- 172.21.15.46:0/3825432870 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbc24066f30 0x7fbc240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:25.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.100+0000 7fbc45325700 1 --2- 172.21.15.46:0/3825432870 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbc24066f30 0x7fbc240693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbc401573c0 tx=0x7fbc30017000 comp rx=0 tx=0).ready entity=mgr.7396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:25.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.103+0000 7fbc35ffb700 1 -- 172.21.15.46:0/3825432870 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbc3c051e50 con 0x7fbc40068bb0 2024-01-26T23:48:25.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.370+0000 7fbc47d8a700 1 -- 172.21.15.46:0/3825432870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fbc4004c710 con 0x7fbc40068bb0 2024-01-26T23:48:25.556 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:48:25.559 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:48:25.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.793+0000 7fbc35ffb700 1 -- 172.21.15.46:0/3825432870 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 56) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fbc3c052220 con 0x7fbc40068bb0 2024-01-26T23:48:25.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.795+0000 7fbc35ffb700 1 -- 172.21.15.46:0/3825432870 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v56) v1 ==== 92+0+0 (secure 0 0 0) 0x7fbc3c051e50 con 0x7fbc40068bb0 2024-01-26T23:48:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.797+0000 7fbc47d8a700 1 -- 172.21.15.46:0/3825432870 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbc24066f30 msgr2=0x7fbc240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.797+0000 7fbc47d8a700 1 --2- 172.21.15.46:0/3825432870 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbc24066f30 0x7fbc240693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbc401573c0 tx=0x7fbc30017000 comp rx=0 tx=0).stop 2024-01-26T23:48:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.797+0000 7fbc47d8a700 1 -- 172.21.15.46:0/3825432870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc40068bb0 msgr2=0x7fbc400b2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:25.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.797+0000 7fbc47d8a700 1 --2- 172.21.15.46:0/3825432870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc40068bb0 0x7fbc400b2850 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00d890 tx=0x7fbc3c008040 comp rx=0 tx=0).stop 2024-01-26T23:48:25.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.797+0000 7fbc47d8a700 1 -- 172.21.15.46:0/3825432870 shutdown_connections 2024-01-26T23:48:25.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.797+0000 7fbc47d8a700 1 --2- 172.21.15.46:0/3825432870 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbc24066f30 0x7fbc240693e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:25.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.797+0000 7fbc47d8a700 1 --2- 172.21.15.46:0/3825432870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc40068bb0 0x7fbc400b2850 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:25.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.797+0000 7fbc47d8a700 1 --2- 172.21.15.46:0/3825432870 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc4015d560 0x7fbc400c4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:25.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.797+0000 7fbc47d8a700 1 --2- 172.21.15.46:0/3825432870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc400cc540 0x7fbc4016e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:25.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.797+0000 7fbc47d8a700 1 -- 172.21.15.46:0/3825432870 >> 172.21.15.46:0/3825432870 conn(0x7fbc4006bd90 msgr2=0x7fbc400c3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:25.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.798+0000 7fbc47d8a700 1 -- 172.21.15.46:0/3825432870 shutdown_connections 2024-01-26T23:48:25.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.798+0000 7fbc47d8a700 1 -- 172.21.15.46:0/3825432870 wait complete. 2024-01-26T23:48:25.814 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-26T23:48:25.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.952+0000 7f3c1bb16700 1 Processor -- start 2024-01-26T23:48:25.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.953+0000 7f3c1bb16700 1 -- start start 2024-01-26T23:48:25.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.954+0000 7f3c1bb16700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3c141599d0 0x7f3c14159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:25.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.954+0000 7f3c1bb16700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c140ab1c0 0x7f3c140b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:25.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.954+0000 7f3c1bb16700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c140ab620 con 0x7f3c1415d8c0 2024-01-26T23:48:25.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.954+0000 7f3c1bb16700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c140b1a60 con 0x7f3c140ab1c0 2024-01-26T23:48:25.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.954+0000 7f3c1bb16700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c140b0f40 con 0x7f3c141599d0 2024-01-26T23:48:25.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.954+0000 7f3c190b1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c140ab1c0 0x7f3c140b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:25.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.954+0000 7f3c1a0b3700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3c1415d8c0 0x7f3c140b20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47300/0 (socket says 172.21.15.46:47300) 2024-01-26T23:48:25.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.954+0000 7f3c190b1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c140ab1c0 0x7f3c140b0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48664/0 (socket says 172.21.15.46:48664) 2024-01-26T23:48:25.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.954+0000 7f3c1a0b3700 1 -- 172.21.15.46:0/3575782275 learned_addr learned my addr 172.21.15.46:0/3575782275 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:25.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.954+0000 7f3c198b2700 1 --2- 172.21.15.46:0/3575782275 >> v2:172.21.15.106:3300/0 conn(0x7f3c141599d0 0x7f3c14159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:25.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.954+0000 7f3c190b1700 1 -- 172.21.15.46:0/3575782275 >> v2:172.21.15.106:3300/0 conn(0x7f3c141599d0 msgr2=0x7f3c14159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:25.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.954+0000 7f3c190b1700 1 --2- 172.21.15.46:0/3575782275 >> v2:172.21.15.106:3300/0 conn(0x7f3c141599d0 0x7f3c14159da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:25.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.954+0000 7f3c190b1700 1 -- 172.21.15.46:0/3575782275 >> v1:172.21.15.46:6789/0 conn(0x7f3c1415d8c0 legacy=0x7f3c140b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:25.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.954+0000 7f3c190b1700 1 -- 172.21.15.46:0/3575782275 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c140b11c0 con 0x7f3c140ab1c0 2024-01-26T23:48:25.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.954+0000 7f3c188b0700 1 -- 172.21.15.46:0/3575782275 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092801782 0 0) 0x7f3c0c001110 con 0x7f3c1415d8c0 2024-01-26T23:48:25.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.955+0000 7f3c190b1700 1 --2- 172.21.15.46:0/3575782275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c140ab1c0 0x7f3c140b0a00 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f3c1408aa50 tx=0x7f3c080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f09901895807972d server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:25.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.955+0000 7f3c188b0700 1 -- 172.21.15.46:0/3575782275 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c08011630 con 0x7f3c140ab1c0 2024-01-26T23:48:25.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.955+0000 7f3c188b0700 1 -- 172.21.15.46:0/3575782275 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c08011790 con 0x7f3c140ab1c0 2024-01-26T23:48:25.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.955+0000 7f3c188b0700 1 -- 172.21.15.46:0/3575782275 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c08011b30 con 0x7f3c140ab1c0 2024-01-26T23:48:25.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.955+0000 7f3c1bb16700 1 -- 172.21.15.46:0/3575782275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c140ab1c0 msgr2=0x7f3c140b0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:25.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.955+0000 7f3c1bb16700 1 --2- 172.21.15.46:0/3575782275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c140ab1c0 0x7f3c140b0a00 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f3c1408aa50 tx=0x7f3c080091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:25.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.955+0000 7f3c1bb16700 1 -- 172.21.15.46:0/3575782275 shutdown_connections 2024-01-26T23:48:25.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.955+0000 7f3c1bb16700 1 --2- 172.21.15.46:0/3575782275 >> v2:172.21.15.106:3300/0 conn(0x7f3c141599d0 0x7f3c14159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:25.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.955+0000 7f3c1bb16700 1 --2- 172.21.15.46:0/3575782275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c140ab1c0 0x7f3c140b0a00 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:25.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.955+0000 7f3c1bb16700 1 -- 172.21.15.46:0/3575782275 >> 172.21.15.46:0/3575782275 conn(0x7f3c1406f840 msgr2=0x7f3c14087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:25.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.956+0000 7f3c1bb16700 1 -- 172.21.15.46:0/3575782275 shutdown_connections 2024-01-26T23:48:25.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.956+0000 7f3c1bb16700 1 -- 172.21.15.46:0/3575782275 wait complete. 2024-01-26T23:48:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.957+0000 7f3c1bb16700 1 Processor -- start 2024-01-26T23:48:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.957+0000 7f3c1bb16700 1 -- start start 2024-01-26T23:48:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.957+0000 7f3c1bb16700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c140ab1c0 0x7f3c1409b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c1bb16700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c141599d0 0x7f3c14158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c1bb16700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c1415d8c0 0x7f3c1415c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c1bb16700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1409a520 con 0x7f3c141599d0 2024-01-26T23:48:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c1bb16700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1409adc0 con 0x7f3c1415d8c0 2024-01-26T23:48:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c1bb16700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1409b790 con 0x7f3c140ab1c0 2024-01-26T23:48:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c190b1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c141599d0 0x7f3c14158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c190b1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c141599d0 0x7f3c14158e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48670/0 (socket says 172.21.15.46:48670) 2024-01-26T23:48:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c190b1700 1 -- 172.21.15.46:0/906644839 learned_addr learned my addr 172.21.15.46:0/906644839 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:25.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c198b2700 1 --2- 172.21.15.46:0/906644839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c140ab1c0 0x7f3c1409b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:25.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c1a0b3700 1 --2- 172.21.15.46:0/906644839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c1415d8c0 0x7f3c1415c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:25.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c198b2700 1 -- 172.21.15.46:0/906644839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c1415d8c0 msgr2=0x7f3c1415c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:25.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c198b2700 1 --2- 172.21.15.46:0/906644839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c1415d8c0 0x7f3c1415c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:25.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c198b2700 1 -- 172.21.15.46:0/906644839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c141599d0 msgr2=0x7f3c14158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:25.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c198b2700 1 --2- 172.21.15.46:0/906644839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c141599d0 0x7f3c14158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:25.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c198b2700 1 -- 172.21.15.46:0/906644839 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c08009420 con 0x7f3c140ab1c0 2024-01-26T23:48:25.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c190b1700 1 --2- 172.21.15.46:0/906644839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c141599d0 0x7f3c14158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:48:25.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c198b2700 1 --2- 172.21.15.46:0/906644839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c140ab1c0 0x7f3c1409b250 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3c04009f40 tx=0x7f3c0400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:25.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.958+0000 7f3c11ffb700 1 -- 172.21.15.46:0/906644839 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0400cb70 con 0x7f3c140ab1c0 2024-01-26T23:48:25.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.959+0000 7f3c11ffb700 1 -- 172.21.15.46:0/906644839 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0400f040 con 0x7f3c140ab1c0 2024-01-26T23:48:25.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.959+0000 7f3c1bb16700 1 -- 172.21.15.46:0/906644839 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c140cdd20 con 0x7f3c140ab1c0 2024-01-26T23:48:25.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.959+0000 7f3c11ffb700 1 -- 172.21.15.46:0/906644839 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04014480 con 0x7f3c140ab1c0 2024-01-26T23:48:25.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.959+0000 7f3c1bb16700 1 -- 172.21.15.46:0/906644839 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1409aa50 con 0x7f3c140ab1c0 2024-01-26T23:48:25.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.960+0000 7f3c1bb16700 1 -- 172.21.15.46:0/906644839 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1404c710 con 0x7f3c140ab1c0 2024-01-26T23:48:25.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.960+0000 7f3c11ffb700 1 -- 172.21.15.46:0/906644839 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 56) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f3c04014830 con 0x7f3c140ab1c0 2024-01-26T23:48:25.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.961+0000 7f3c11ffb700 1 --2- 172.21.15.46:0/906644839 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3c00066f80 0x7f3c00069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:25.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.961+0000 7f3c190b1700 1 -- 172.21.15.46:0/906644839 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3c00066f80 msgr2=0x7f3c00069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:48:25.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.961+0000 7f3c11ffb700 1 -- 172.21.15.46:0/906644839 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f3c04085690 con 0x7f3c140ab1c0 2024-01-26T23:48:25.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.961+0000 7f3c190b1700 1 --2- 172.21.15.46:0/906644839 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3c00066f80 0x7f3c00069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:48:25.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:25.965+0000 7f3c11ffb700 1 -- 172.21.15.46:0/906644839 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3c04052590 con 0x7f3c140ab1c0 2024-01-26T23:48:26.131 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:26.130+0000 7f92cdd571c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:48:26.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.162+0000 7f3c190b1700 1 -- 172.21.15.46:0/906644839 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3c00066f80 msgr2=0x7f3c00069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:48:26.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.162+0000 7f3c190b1700 1 --2- 172.21.15.46:0/906644839 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3c00066f80 0x7f3c00069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:48:26.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.170+0000 7f3c1bb16700 1 -- 172.21.15.46:0/906644839 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f3c140b14a0 con 0x7f3c140ab1c0 2024-01-26T23:48:26.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.175+0000 7f3c11ffb700 1 -- 172.21.15.46:0/906644839 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3c04055bb0 con 0x7f3c140ab1c0 2024-01-26T23:48:26.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.177+0000 7f3c1bb16700 1 -- 172.21.15.46:0/906644839 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3c00066f80 msgr2=0x7f3c00069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:48:26.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.177+0000 7f3c1bb16700 1 --2- 172.21.15.46:0/906644839 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3c00066f80 0x7f3c00069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.177+0000 7f3c1bb16700 1 -- 172.21.15.46:0/906644839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c140ab1c0 msgr2=0x7f3c1409b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.177+0000 7f3c1bb16700 1 --2- 172.21.15.46:0/906644839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c140ab1c0 0x7f3c1409b250 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3c04009f40 tx=0x7f3c0400b040 comp rx=0 tx=0).stop 2024-01-26T23:48:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.177+0000 7f3c1bb16700 1 -- 172.21.15.46:0/906644839 shutdown_connections 2024-01-26T23:48:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.177+0000 7f3c1bb16700 1 --2- 172.21.15.46:0/906644839 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3c00066f80 0x7f3c00069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.177+0000 7f3c1bb16700 1 --2- 172.21.15.46:0/906644839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c1415d8c0 0x7f3c1415c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.177+0000 7f3c1bb16700 1 --2- 172.21.15.46:0/906644839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c140ab1c0 0x7f3c1409b250 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.177+0000 7f3c1bb16700 1 --2- 172.21.15.46:0/906644839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c141599d0 0x7f3c14158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.177+0000 7f3c1bb16700 1 -- 172.21.15.46:0/906644839 >> 172.21.15.46:0/906644839 conn(0x7f3c1406f840 msgr2=0x7f3c14087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.177+0000 7f3c1bb16700 1 -- 172.21.15.46:0/906644839 shutdown_connections 2024-01-26T23:48:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.177+0000 7f3c1bb16700 1 -- 172.21.15.46:0/906644839 wait complete. 2024-01-26T23:48:26.180 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:26.179+0000 7f0b036411c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:48:26.192 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-26T23:48:26.208 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:26.208+0000 7f92cdd571c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:48:26.270 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:26.269+0000 7f0b036411c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:48:26.279 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:26.279+0000 7f92cdd571c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:48:26.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.328+0000 7f7677c4b700 1 Processor -- start 2024-01-26T23:48:26.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.328+0000 7f7677c4b700 1 -- start start 2024-01-26T23:48:26.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.330+0000 7f7677c4b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7670156940 0x7f767006c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:26.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.330+0000 7f7677c4b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76700c8b90 con 0x7f767006f9a0 2024-01-26T23:48:26.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.330+0000 7f7677c4b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76700c8e50 con 0x7f7670156570 2024-01-26T23:48:26.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.330+0000 7f7677c4b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f767006d0b0 con 0x7f7670156940 2024-01-26T23:48:26.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.330+0000 7f76759e7700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f7670156570 0x7f76700c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:51320/0 (socket says 172.21.15.46:51320) 2024-01-26T23:48:26.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.330+0000 7f76759e7700 1 -- 172.21.15.46:0/2025272574 learned_addr learned my addr 172.21.15.46:0/2025272574 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:26.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.330+0000 7f76751e6700 1 --2- 172.21.15.46:0/2025272574 >> v2:172.21.15.46:3300/0 conn(0x7f7670156940 0x7f767006c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:26.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.330+0000 7f76749e5700 1 -- 172.21.15.46:0/2025272574 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833717300 0 0) 0x7f76700c8e50 con 0x7f7670156570 2024-01-26T23:48:26.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.330+0000 7f76749e5700 1 -- 172.21.15.46:0/2025272574 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f765c003570 con 0x7f7670156570 2024-01-26T23:48:26.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.330+0000 7f76749e5700 1 -- 172.21.15.46:0/2025272574 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213935053 0 0) 0x7f76700c8b90 con 0x7f767006f9a0 2024-01-26T23:48:26.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.330+0000 7f76749e5700 1 -- 172.21.15.46:0/2025272574 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76700c8e50 con 0x7f767006f9a0 2024-01-26T23:48:26.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.330+0000 7f76749e5700 1 -- 172.21.15.46:0/2025272574 <== mon.1 v1:172.21.15.106:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1549773973 0 0) 0x7f765c003570 con 0x7f7670156570 2024-01-26T23:48:26.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.330+0000 7f76751e6700 1 -- 172.21.15.46:0/2025272574 >> v1:172.21.15.106:6789/0 conn(0x7f7670156570 legacy=0x7f76700c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:26.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.330+0000 7f76751e6700 1 -- 172.21.15.46:0/2025272574 >> v1:172.21.15.46:6789/0 conn(0x7f767006f9a0 legacy=0x7f767006eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:26.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.330+0000 7f76751e6700 1 -- 172.21.15.46:0/2025272574 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76700c4c40 con 0x7f7670156940 2024-01-26T23:48:26.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.330+0000 7f76751e6700 1 --2- 172.21.15.46:0/2025272574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7670156940 0x7f767006c110 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f7660009860 tx=0x7f76600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b95ed3b19bc012ca server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:26.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.331+0000 7f76749e5700 1 -- 172.21.15.46:0/2025272574 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c001470 con 0x7f7670156940 2024-01-26T23:48:26.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.331+0000 7f76749e5700 1 -- 172.21.15.46:0/2025272574 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7660011420 con 0x7f7670156940 2024-01-26T23:48:26.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.331+0000 7f76749e5700 1 -- 172.21.15.46:0/2025272574 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76600117a0 con 0x7f7670156940 2024-01-26T23:48:26.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.331+0000 7f7677c4b700 1 -- 172.21.15.46:0/2025272574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7670156940 msgr2=0x7f767006c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:26.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.331+0000 7f7677c4b700 1 --2- 172.21.15.46:0/2025272574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7670156940 0x7f767006c110 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f7660009860 tx=0x7f76600091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.331+0000 7f7677c4b700 1 -- 172.21.15.46:0/2025272574 shutdown_connections 2024-01-26T23:48:26.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.331+0000 7f7677c4b700 1 --2- 172.21.15.46:0/2025272574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7670156940 0x7f767006c110 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.331+0000 7f7677c4b700 1 -- 172.21.15.46:0/2025272574 >> 172.21.15.46:0/2025272574 conn(0x7f76700cdf50 msgr2=0x7f76700c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:26.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.332+0000 7f7677c4b700 1 -- 172.21.15.46:0/2025272574 shutdown_connections 2024-01-26T23:48:26.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.332+0000 7f7677c4b700 1 -- 172.21.15.46:0/2025272574 wait complete. 2024-01-26T23:48:26.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.333+0000 7f7677c4b700 1 Processor -- start 2024-01-26T23:48:26.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.333+0000 7f7677c4b700 1 -- start start 2024-01-26T23:48:26.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.334+0000 7f7677c4b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f767006f9a0 0x7f7670158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:26.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.334+0000 7f7677c4b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7670156570 0x7f7670151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:26.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.334+0000 7f7677c4b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7670156940 0x7f7670154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:26.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.334+0000 7f7677c4b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76700b0430 con 0x7f767006f9a0 2024-01-26T23:48:26.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.334+0000 7f7677c4b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7670159250 con 0x7f7670156940 2024-01-26T23:48:26.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.334+0000 7f7677c4b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7670159530 con 0x7f7670156570 2024-01-26T23:48:26.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.334+0000 7f76759e7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f767006f9a0 0x7f7670158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.334+0000 7f76759e7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f767006f9a0 0x7f7670158c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48700/0 (socket says 172.21.15.46:48700) 2024-01-26T23:48:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.334+0000 7f76759e7700 1 -- 172.21.15.46:0/2998259656 learned_addr learned my addr 172.21.15.46:0/2998259656 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.334+0000 7f76751e6700 1 --2- 172.21.15.46:0/2998259656 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7670156570 0x7f7670151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.334+0000 7f76761e8700 1 --2- 172.21.15.46:0/2998259656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7670156940 0x7f7670154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.334+0000 7f76759e7700 1 -- 172.21.15.46:0/2998259656 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7670156570 msgr2=0x7f7670151750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.334+0000 7f76759e7700 1 --2- 172.21.15.46:0/2998259656 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7670156570 0x7f7670151750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.334+0000 7f76759e7700 1 -- 172.21.15.46:0/2998259656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7670156940 msgr2=0x7f7670154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.334+0000 7f76759e7700 1 --2- 172.21.15.46:0/2998259656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7670156940 0x7f7670154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.334+0000 7f76759e7700 1 -- 172.21.15.46:0/2998259656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7660009420 con 0x7f767006f9a0 2024-01-26T23:48:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.334+0000 7f76759e7700 1 --2- 172.21.15.46:0/2998259656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f767006f9a0 0x7f7670158c00 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f766c00cb70 tx=0x7f766c001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.335+0000 7f7665ffb700 1 -- 172.21.15.46:0/2998259656 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c007150 con 0x7f767006f9a0 2024-01-26T23:48:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.335+0000 7f7665ffb700 1 -- 172.21.15.46:0/2998259656 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766c0072d0 con 0x7f767006f9a0 2024-01-26T23:48:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.335+0000 7f7677c4b700 1 -- 172.21.15.46:0/2998259656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7670159810 con 0x7f767006f9a0 2024-01-26T23:48:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.335+0000 7f7665ffb700 1 -- 172.21.15.46:0/2998259656 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c007690 con 0x7f767006f9a0 2024-01-26T23:48:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.335+0000 7f7677c4b700 1 -- 172.21.15.46:0/2998259656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7670158980 con 0x7f767006f9a0 2024-01-26T23:48:26.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.336+0000 7f7677c4b700 1 -- 172.21.15.46:0/2998259656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f767004c710 con 0x7f767006f9a0 2024-01-26T23:48:26.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.337+0000 7f7665ffb700 1 -- 172.21.15.46:0/2998259656 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 56) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f766c004a40 con 0x7f767006f9a0 2024-01-26T23:48:26.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.337+0000 7f7665ffb700 1 --2- 172.21.15.46:0/2998259656 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7654066f80 0x7f7654069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:26.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.337+0000 7f7665ffb700 1 -- 172.21.15.46:0/2998259656 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f766c084020 con 0x7f767006f9a0 2024-01-26T23:48:26.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.337+0000 7f76751e6700 1 -- 172.21.15.46:0/2998259656 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7654066f80 msgr2=0x7f7654069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:48:26.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.337+0000 7f76751e6700 1 --2- 172.21.15.46:0/2998259656 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7654066f80 0x7f7654069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:48:26.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.341+0000 7f7665ffb700 1 -- 172.21.15.46:0/2998259656 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f766c050fc0 con 0x7f767006f9a0 2024-01-26T23:48:26.344 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:26.344+0000 7f92cdd571c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:48:26.355 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:26.353+0000 7f0b036411c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:48:26.431 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:26.429+0000 7f0b036411c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:48:26.499 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:26.499+0000 7f92cdd571c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:48:26.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.538+0000 7f76751e6700 1 -- 172.21.15.46:0/2998259656 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7654066f80 msgr2=0x7f7654069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:48:26.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.538+0000 7f76751e6700 1 --2- 172.21.15.46:0/2998259656 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7654066f80 0x7f7654069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:48:26.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.549+0000 7f7677c4b700 1 -- 172.21.15.46:0/2998259656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f767006a530 con 0x7f767006f9a0 2024-01-26T23:48:26.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.557+0000 7f7665ffb700 1 -- 172.21.15.46:0/2998259656 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) v1 ==== 133+0+0 (secure 0 0 0) 0x7f766c0180c0 con 0x7f767006f9a0 2024-01-26T23:48:26.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.559+0000 7f7677c4b700 1 -- 172.21.15.46:0/2998259656 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7654066f80 msgr2=0x7f7654069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:48:26.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.559+0000 7f7677c4b700 1 --2- 172.21.15.46:0/2998259656 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7654066f80 0x7f7654069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.559+0000 7f7677c4b700 1 -- 172.21.15.46:0/2998259656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f767006f9a0 msgr2=0x7f7670158c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:26.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.559+0000 7f7677c4b700 1 --2- 172.21.15.46:0/2998259656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f767006f9a0 0x7f7670158c00 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f766c00cb70 tx=0x7f766c001ab0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.559+0000 7f7677c4b700 1 -- 172.21.15.46:0/2998259656 shutdown_connections 2024-01-26T23:48:26.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.559+0000 7f7677c4b700 1 --2- 172.21.15.46:0/2998259656 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7654066f80 0x7f7654069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.559+0000 7f7677c4b700 1 --2- 172.21.15.46:0/2998259656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f767006f9a0 0x7f7670158c00 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.559+0000 7f7677c4b700 1 --2- 172.21.15.46:0/2998259656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7670156940 0x7f7670154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.559+0000 7f7677c4b700 1 --2- 172.21.15.46:0/2998259656 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7670156570 0x7f7670151750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.559+0000 7f7677c4b700 1 -- 172.21.15.46:0/2998259656 >> 172.21.15.46:0/2998259656 conn(0x7f76700cdf50 msgr2=0x7f76700c4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:26.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.559+0000 7f7677c4b700 1 -- 172.21.15.46:0/2998259656 shutdown_connections 2024-01-26T23:48:26.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.559+0000 7f7677c4b700 1 -- 172.21.15.46:0/2998259656 wait complete. 2024-01-26T23:48:26.575 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:26.575 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-26T23:48:26.600 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-01-26T23:48:26.613 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:26.612+0000 7f0b036411c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:48:26.639 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:26.638+0000 7f92cdd571c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:48:26.706 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:26.705+0000 7f92cdd571c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:48:26.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.750+0000 7f7ea9439700 1 Processor -- start 2024-01-26T23:48:26.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.750+0000 7f7ea9439700 1 -- start start 2024-01-26T23:48:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.751+0000 7f7ea9439700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ea415dfd0 0x7f7ea40ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.751+0000 7f7ea9439700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea415a200 con 0x7f7ea4159da0 2024-01-26T23:48:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.751+0000 7f7ea9439700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea40b1a60 con 0x7f7ea41599d0 2024-01-26T23:48:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.751+0000 7f7ea9439700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea40b11c0 con 0x7f7ea415dfd0 2024-01-26T23:48:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.751+0000 7f7e9bfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7ea4159da0 0x7f7ea415d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47322/0 (socket says 172.21.15.46:47322) 2024-01-26T23:48:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.751+0000 7f7e9bfff700 1 -- 172.21.15.46:0/3100177847 learned_addr learned my addr 172.21.15.46:0/3100177847 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:48:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.752+0000 7f7ea37fe700 1 --2- 172.21.15.46:0/3100177847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ea415dfd0 0x7f7ea40ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.752+0000 7f7ea27fc700 1 -- 172.21.15.46:0/3100177847 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085816358 0 0) 0x7f7ea415a200 con 0x7f7ea4159da0 2024-01-26T23:48:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.752+0000 7f7ea27fc700 1 -- 172.21.15.46:0/3100177847 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e90003570 con 0x7f7ea4159da0 2024-01-26T23:48:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.752+0000 7f7ea27fc700 1 -- 172.21.15.46:0/3100177847 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841859580 0 0) 0x7f7ea40b1a60 con 0x7f7ea41599d0 2024-01-26T23:48:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.752+0000 7f7ea27fc700 1 -- 172.21.15.46:0/3100177847 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea415a200 con 0x7f7ea41599d0 2024-01-26T23:48:26.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.752+0000 7f7ea37fe700 1 -- 172.21.15.46:0/3100177847 >> v1:172.21.15.106:6789/0 conn(0x7f7ea41599d0 legacy=0x7f7ea40b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:26.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.752+0000 7f7ea37fe700 1 -- 172.21.15.46:0/3100177847 >> v1:172.21.15.46:6789/0 conn(0x7f7ea4159da0 legacy=0x7f7ea415d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:26.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.752+0000 7f7ea37fe700 1 -- 172.21.15.46:0/3100177847 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea40b14a0 con 0x7f7ea415dfd0 2024-01-26T23:48:26.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.752+0000 7f7ea37fe700 1 --2- 172.21.15.46:0/3100177847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ea415dfd0 0x7f7ea40ab1c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c009e90 tx=0x7f7e9c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17f8bd3836baf613 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:26.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.752+0000 7f7ea27fc700 1 -- 172.21.15.46:0/3100177847 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c00e070 con 0x7f7ea415dfd0 2024-01-26T23:48:26.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.752+0000 7f7ea27fc700 1 -- 172.21.15.46:0/3100177847 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9c00bb80 con 0x7f7ea415dfd0 2024-01-26T23:48:26.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.752+0000 7f7ea27fc700 1 -- 172.21.15.46:0/3100177847 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c012430 con 0x7f7ea415dfd0 2024-01-26T23:48:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.753+0000 7f7ea9439700 1 -- 172.21.15.46:0/3100177847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ea415dfd0 msgr2=0x7f7ea40ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:26.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.753+0000 7f7ea9439700 1 --2- 172.21.15.46:0/3100177847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ea415dfd0 0x7f7ea40ab1c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c009e90 tx=0x7f7e9c009780 comp rx=0 tx=0).stop 2024-01-26T23:48:26.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.753+0000 7f7ea9439700 1 -- 172.21.15.46:0/3100177847 shutdown_connections 2024-01-26T23:48:26.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.753+0000 7f7ea9439700 1 --2- 172.21.15.46:0/3100177847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ea415dfd0 0x7f7ea40ab1c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.753+0000 7f7ea9439700 1 -- 172.21.15.46:0/3100177847 >> 172.21.15.46:0/3100177847 conn(0x7f7ea406f840 msgr2=0x7f7ea4087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:26.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.753+0000 7f7ea9439700 1 -- 172.21.15.46:0/3100177847 shutdown_connections 2024-01-26T23:48:26.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.753+0000 7f7ea9439700 1 -- 172.21.15.46:0/3100177847 wait complete. 2024-01-26T23:48:26.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.755+0000 7f7ea9439700 1 Processor -- start 2024-01-26T23:48:26.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.755+0000 7f7ea9439700 1 -- start start 2024-01-26T23:48:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.756+0000 7f7ea9439700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ea41599d0 0x7f7ea416d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.756+0000 7f7ea9439700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ea4159da0 0x7f7ea409dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.756+0000 7f7ea9439700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ea415dfd0 0x7f7ea4169b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.756+0000 7f7ea9439700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea416a400 con 0x7f7ea4159da0 2024-01-26T23:48:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.756+0000 7f7ea9439700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea416a6e0 con 0x7f7ea41599d0 2024-01-26T23:48:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.756+0000 7f7ea9439700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea416a9c0 con 0x7f7ea415dfd0 2024-01-26T23:48:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.756+0000 7f7e9bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ea4159da0 0x7f7ea409dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.756+0000 7f7e9bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ea4159da0 0x7f7ea409dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48712/0 (socket says 172.21.15.46:48712) 2024-01-26T23:48:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.756+0000 7f7e9bfff700 1 -- 172.21.15.46:0/1310522507 learned_addr learned my addr 172.21.15.46:0/1310522507 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.756+0000 7f7ea37fe700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ea415dfd0 0x7f7ea4169b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.756+0000 7f7ea2ffd700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ea41599d0 0x7f7ea416d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.757+0000 7f7e9bfff700 1 -- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ea415dfd0 msgr2=0x7f7ea4169b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.757+0000 7f7e9bfff700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ea415dfd0 0x7f7ea4169b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.757+0000 7f7e9bfff700 1 -- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ea41599d0 msgr2=0x7f7ea416d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.757+0000 7f7e9bfff700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ea41599d0 0x7f7ea416d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.757+0000 7f7e9bfff700 1 -- 172.21.15.46:0/1310522507 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9c009a20 con 0x7f7ea4159da0 2024-01-26T23:48:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.757+0000 7f7e9bfff700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ea4159da0 0x7f7ea409dc20 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00c790 tx=0x7f7e8c007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.757+0000 7f7e9affd700 1 -- 172.21.15.46:0/1310522507 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c005b30 con 0x7f7ea4159da0 2024-01-26T23:48:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.757+0000 7f7e9affd700 1 -- 172.21.15.46:0/1310522507 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8c005c90 con 0x7f7ea4159da0 2024-01-26T23:48:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.757+0000 7f7ea9439700 1 -- 172.21.15.46:0/1310522507 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea416d1c0 con 0x7f7ea4159da0 2024-01-26T23:48:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.757+0000 7f7e9affd700 1 -- 172.21.15.46:0/1310522507 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c007300 con 0x7f7ea4159da0 2024-01-26T23:48:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.757+0000 7f7ea9439700 1 -- 172.21.15.46:0/1310522507 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea416df00 con 0x7f7ea4159da0 2024-01-26T23:48:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.758+0000 7f7ea9439700 1 -- 172.21.15.46:0/1310522507 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea404c710 con 0x7f7ea4159da0 2024-01-26T23:48:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.759+0000 7f7e9affd700 1 -- 172.21.15.46:0/1310522507 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 56) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f7e8c01b3b0 con 0x7f7ea4159da0 2024-01-26T23:48:26.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.759+0000 7f7e9affd700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 0x7f7e84069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:26.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.759+0000 7f7ea2ffd700 1 -- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 msgr2=0x7f7e84069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:48:26.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.759+0000 7f7e9affd700 1 -- 172.21.15.46:0/1310522507 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f7e8c083dc0 con 0x7f7ea4159da0 2024-01-26T23:48:26.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.759+0000 7f7ea2ffd700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 0x7f7e84069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:48:26.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.768+0000 7f7e9affd700 1 -- 172.21.15.46:0/1310522507 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7e8c050d60 con 0x7f7ea4159da0 2024-01-26T23:48:26.782 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:26.781+0000 7f0b036411c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:48:26.831 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:26.831+0000 7f92cdd571c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:48:26.862 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:26.861+0000 7f0b036411c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:48:26.905 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:26.904+0000 7f92cdd571c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:48:26.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.960+0000 7f7ea2ffd700 1 -- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 msgr2=0x7f7e84069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:48:26.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:26.960+0000 7f7ea2ffd700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 0x7f7e84069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:48:27.001 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:27.000+0000 7f0b036411c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:48:27.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:27.008+0000 7f7ea9439700 1 -- 172.21.15.46:0/1310522507 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ea40d0560 con 0x7f7e84066f80 2024-01-26T23:48:27.044 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:27.044+0000 7f92cdd571c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:48:27.084 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:27.083+0000 7f0b036411c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:48:27.171 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:27.170+0000 7f92cdd571c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:48:27.260 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:27.259+0000 7f0b036411c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:48:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:27.361+0000 7f7ea2ffd700 1 -- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 msgr2=0x7f7e84069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:48:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:27.361+0000 7f7ea2ffd700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 0x7f7e84069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-26T23:48:27.401 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:27.400+0000 7f0b036411c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:48:27.486 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:27.485+0000 7f92cdd571c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:48:27.697 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:27.696+0000 7f92cdd571c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:48:27.766 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:27.765+0000 7f0b036411c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:48:27.924 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:27.923+0000 7f92cdd571c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:48:27.994 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:27.993+0000 7f92cdd571c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:48:28.006 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:28.005+0000 7f0b036411c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:48:28.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:28.162+0000 7f7ea2ffd700 1 -- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 msgr2=0x7f7e84069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:48:28.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:28.162+0000 7f7ea2ffd700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 0x7f7e84069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-26T23:48:28.266 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:28.265+0000 7f0b036411c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:48:28.347 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:28.346+0000 7f0b036411c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:48:28.593 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:28.592+0000 7f92cdd571c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:48:28.667 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:28.666+0000 7f92cdd571c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:48:28.790 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:28.789+0000 7f92cdd571c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:48:29.041 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:29.039+0000 7f0b036411c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:48:29.125 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:29.124+0000 7f0b036411c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:48:29.267 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:29.265+0000 7f0b036411c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:48:29.291 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:29.290+0000 7f92cdd571c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:48:29.439 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:29.438+0000 7f92cdd571c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:48:29.589 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:29.588+0000 7f92cdd571c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:48:29.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:29.764+0000 7f7ea2ffd700 1 -- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 msgr2=0x7f7e84069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:48:29.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:29.764+0000 7f7ea2ffd700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 0x7f7e84069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-26T23:48:29.850 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:29.849+0000 7f0b036411c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:48:30.020 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:30.019+0000 7f0b036411c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:48:30.193 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:30.192+0000 7f0b036411c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:48:30.286 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:30.285+0000 7f92cdd571c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:48:30.506 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:30.505+0000 7f92cdd571c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:48:30.579 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:30.578+0000 7f92cdd571c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:48:30.734 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:30.733+0000 7f92cdd571c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:48:30.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:30.753+0000 7f7e9affd700 1 -- 172.21.15.46:0/1310522507 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 57) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f7e8c051130 con 0x7f7ea4159da0 2024-01-26T23:48:30.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:30.753+0000 7f7e9affd700 1 -- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 msgr2=0x7f7e84069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:48:30.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:30.753+0000 7f7e9affd700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 0x7f7e84069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:30.863 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:30.862+0000 7f9246639700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:48:30.863 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:30.862+0000 7f9246639700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:48:30.863 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:30.862+0000 7f9246639700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:48:30.863 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:30.862+0000 7f9246639700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:48:30.863 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:30.862+0000 7f9246639700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:48:30.864 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:30.863+0000 7f9244635700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:48:30.864 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:30.863+0000 7f9244635700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:48:30.864 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:30.863+0000 7f9244635700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:48:30.864 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:30.863+0000 7f9244635700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:48:30.864 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:48:30.863+0000 7f9244635700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:48:31.020 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:31.019+0000 7f0b036411c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:48:31.272 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:31.271+0000 7f0b036411c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:48:31.356 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:31.354+0000 7f0b036411c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:48:31.537 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:48:31.536+0000 7f0b036411c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:48:31.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.756+0000 7f7e9affd700 1 -- 172.21.15.46:0/1310522507 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 58) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f7e8c006030 con 0x7f7ea4159da0 2024-01-26T23:48:31.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.757+0000 7f7e9affd700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 0x7f7e84069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:31.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.757+0000 7f7e9affd700 1 -- 172.21.15.46:0/1310522507 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ea40d0560 con 0x7f7e84066f80 2024-01-26T23:48:31.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.762+0000 7f7ea2ffd700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 0x7f7e84069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:31.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.764+0000 7f7ea2ffd700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 0x7f7e84069430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7e94006870 tx=0x7f7e94009040 comp rx=0 tx=0).ready entity=mgr.7768 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:31.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.819+0000 7f7e9affd700 1 -- 172.21.15.46:0/1310522507 <== mgr.7768 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7ea40d0560 con 0x7f7e84066f80 2024-01-26T23:48:31.821 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path snap_test_dir1 2024-01-26T23:48:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.822+0000 7f7ea9439700 1 -- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 msgr2=0x7f7e84069430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.822+0000 7f7ea9439700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 0x7f7e84069430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7e94006870 tx=0x7f7e94009040 comp rx=0 tx=0).stop 2024-01-26T23:48:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.822+0000 7f7ea9439700 1 -- 172.21.15.46:0/1310522507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ea4159da0 msgr2=0x7f7ea409dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.822+0000 7f7ea9439700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ea4159da0 0x7f7ea409dc20 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00c790 tx=0x7f7e8c007ea0 comp rx=0 tx=0).stop 2024-01-26T23:48:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.822+0000 7f7ea9439700 1 -- 172.21.15.46:0/1310522507 shutdown_connections 2024-01-26T23:48:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.822+0000 7f7ea9439700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e84066f80 0x7f7e84069430 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.822+0000 7f7ea9439700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ea415dfd0 0x7f7ea4169b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.822+0000 7f7ea9439700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ea41599d0 0x7f7ea416d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.822+0000 7f7ea9439700 1 --2- 172.21.15.46:0/1310522507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ea4159da0 0x7f7ea409dc20 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:31.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.822+0000 7f7ea9439700 1 -- 172.21.15.46:0/1310522507 >> 172.21.15.46:0/1310522507 conn(0x7f7ea406f840 msgr2=0x7f7ea415c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:31.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.822+0000 7f7ea9439700 1 -- 172.21.15.46:0/1310522507 shutdown_connections 2024-01-26T23:48:31.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.822+0000 7f7ea9439700 1 -- 172.21.15.46:0/1310522507 wait complete. 2024-01-26T23:48:31.839 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-01-26T23:48:31.839 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-01-26-23_49_00 in ~29s... 2024-01-26T23:48:31.839 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-01-26T23:48:31.839 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-01-26T23:48:31.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.976+0000 7f0089dac700 1 Processor -- start 2024-01-26T23:48:31.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.976+0000 7f0089dac700 1 -- start start 2024-01-26T23:48:31.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.977+0000 7f0089dac700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0084067af0 0x7f0084067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:31.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.977+0000 7f0089dac700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f008406ebe0 0x7f00840ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:31.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.977+0000 7f0089dac700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f008406fdb0 con 0x7f00841597e0 2024-01-26T23:48:31.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.977+0000 7f0089dac700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00840ade10 con 0x7f008406ebe0 2024-01-26T23:48:31.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.977+0000 7f0089dac700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f008406f1a0 con 0x7f0084067af0 2024-01-26T23:48:31.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.977+0000 7f0082ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f008406ebe0 0x7f00840ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:31.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.977+0000 7f0082ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f008406ebe0 0x7f00840ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55486/0 (socket says 172.21.15.46:55486) 2024-01-26T23:48:31.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.977+0000 7f0082ffd700 1 -- 172.21.15.46:0/1251789372 learned_addr learned my addr 172.21.15.46:0/1251789372 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:31.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.977+0000 7f00837fe700 1 --2- 172.21.15.46:0/1251789372 >> v2:172.21.15.106:3300/0 conn(0x7f0084067af0 0x7f0084067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:31.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.977+0000 7f0082ffd700 1 -- 172.21.15.46:0/1251789372 >> v2:172.21.15.106:3300/0 conn(0x7f0084067af0 msgr2=0x7f0084067ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:48:31.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.977+0000 7f0082ffd700 1 --2- 172.21.15.46:0/1251789372 >> v2:172.21.15.106:3300/0 conn(0x7f0084067af0 0x7f0084067ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:31.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.977+0000 7f0082ffd700 1 -- 172.21.15.46:0/1251789372 >> v1:172.21.15.106:6789/0 conn(0x7f00841597e0 legacy=0x7f0084159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:31.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.977+0000 7f0082ffd700 1 -- 172.21.15.46:0/1251789372 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008406f480 con 0x7f008406ebe0 2024-01-26T23:48:31.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.977+0000 7f0082ffd700 1 --2- 172.21.15.46:0/1251789372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f008406ebe0 0x7f00840ad8d0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f0084064050 tx=0x7f006c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf8f4aa7b2a5721 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:31.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.978+0000 7f00827fc700 1 -- 172.21.15.46:0/1251789372 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c009e70 con 0x7f008406ebe0 2024-01-26T23:48:31.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.978+0000 7f00827fc700 1 -- 172.21.15.46:0/1251789372 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006c00b860 con 0x7f008406ebe0 2024-01-26T23:48:31.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.978+0000 7f00827fc700 1 -- 172.21.15.46:0/1251789372 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c00bc20 con 0x7f008406ebe0 2024-01-26T23:48:31.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.978+0000 7f0089dac700 1 -- 172.21.15.46:0/1251789372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f008406ebe0 msgr2=0x7f00840ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:31.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.978+0000 7f0089dac700 1 --2- 172.21.15.46:0/1251789372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f008406ebe0 0x7f00840ad8d0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f0084064050 tx=0x7f006c009250 comp rx=0 tx=0).stop 2024-01-26T23:48:31.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.978+0000 7f0089dac700 1 -- 172.21.15.46:0/1251789372 shutdown_connections 2024-01-26T23:48:31.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.978+0000 7f0089dac700 1 --2- 172.21.15.46:0/1251789372 >> v2:172.21.15.106:3300/0 conn(0x7f0084067af0 0x7f0084067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:31.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.978+0000 7f0089dac700 1 --2- 172.21.15.46:0/1251789372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f008406ebe0 0x7f00840ad8d0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:31.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.978+0000 7f0089dac700 1 -- 172.21.15.46:0/1251789372 >> 172.21.15.46:0/1251789372 conn(0x7f00840cdf80 msgr2=0x7f008406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:31.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.978+0000 7f0089dac700 1 -- 172.21.15.46:0/1251789372 shutdown_connections 2024-01-26T23:48:31.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.978+0000 7f0089dac700 1 -- 172.21.15.46:0/1251789372 wait complete. 2024-01-26T23:48:31.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.980+0000 7f0089dac700 1 Processor -- start 2024-01-26T23:48:31.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.980+0000 7f0089dac700 1 -- start start 2024-01-26T23:48:31.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f0089dac700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0084067af0 0x7f0084143830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:31.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f0089dac700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f008406ebe0 0x7f00840b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:31.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f0089dac700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00841597e0 0x7f00840b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:31.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f0089dac700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00841566a0 con 0x7f00841597e0 2024-01-26T23:48:31.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f0089dac700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0084152780 con 0x7f008406ebe0 2024-01-26T23:48:31.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f0089dac700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0084143d70 con 0x7f0084067af0 2024-01-26T23:48:31.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f0083fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00841597e0 0x7f00840b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:31.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f0083fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00841597e0 0x7f00840b2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55494/0 (socket says 172.21.15.46:55494) 2024-01-26T23:48:31.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f0083fff700 1 -- 172.21.15.46:0/513608654 learned_addr learned my addr 172.21.15.46:0/513608654 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:48:31.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f0082ffd700 1 --2- 172.21.15.46:0/513608654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f008406ebe0 0x7f00840b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:31.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f00837fe700 1 --2- 172.21.15.46:0/513608654 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0084067af0 0x7f0084143830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:31.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f0083fff700 1 -- 172.21.15.46:0/513608654 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0084067af0 msgr2=0x7f0084143830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:31.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f0083fff700 1 --2- 172.21.15.46:0/513608654 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0084067af0 0x7f0084143830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:31.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f0083fff700 1 -- 172.21.15.46:0/513608654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f008406ebe0 msgr2=0x7f00840b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:31.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f0083fff700 1 --2- 172.21.15.46:0/513608654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f008406ebe0 0x7f00840b25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:31.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f0083fff700 1 -- 172.21.15.46:0/513608654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c0094f0 con 0x7f00841597e0 2024-01-26T23:48:31.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f0083fff700 1 --2- 172.21.15.46:0/513608654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00841597e0 0x7f00840b2af0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f007c00c840 tx=0x7f007c001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:31.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f007b7fe700 1 -- 172.21.15.46:0/513608654 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007c012630 con 0x7f00841597e0 2024-01-26T23:48:31.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.981+0000 7f007b7fe700 1 -- 172.21.15.46:0/513608654 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f007c0127b0 con 0x7f00841597e0 2024-01-26T23:48:31.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.982+0000 7f0089dac700 1 -- 172.21.15.46:0/513608654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0084155d70 con 0x7f00841597e0 2024-01-26T23:48:31.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.982+0000 7f007b7fe700 1 -- 172.21.15.46:0/513608654 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007c012b70 con 0x7f00841597e0 2024-01-26T23:48:31.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.982+0000 7f0089dac700 1 -- 172.21.15.46:0/513608654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0084156330 con 0x7f00841597e0 2024-01-26T23:48:31.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.982+0000 7f0089dac700 1 -- 172.21.15.46:0/513608654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0084156810 con 0x7f00841597e0 2024-01-26T23:48:31.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.983+0000 7f007b7fe700 1 -- 172.21.15.46:0/513608654 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 58) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f007c01a3a0 con 0x7f00841597e0 2024-01-26T23:48:31.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.983+0000 7f007b7fe700 1 --2- 172.21.15.46:0/513608654 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0064066dd0 0x7f0064069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:48:31.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.984+0000 7f007b7fe700 1 -- 172.21.15.46:0/513608654 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f007c084740 con 0x7f00841597e0 2024-01-26T23:48:31.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.984+0000 7f00837fe700 1 --2- 172.21.15.46:0/513608654 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0064066dd0 0x7f0064069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:48:31.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.984+0000 7f00837fe700 1 --2- 172.21.15.46:0/513608654 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0064066dd0 0x7f0064069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0074009d10 tx=0x7f00740091a0 comp rx=0 tx=0).ready entity=mgr.7768 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:48:31.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:31.988+0000 7f007b7fe700 1 -- 172.21.15.46:0/513608654 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f007c051890 con 0x7f00841597e0 2024-01-26T23:48:32.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:32.222+0000 7f0089dac700 1 -- 172.21.15.46:0/513608654 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f008404c710 con 0x7f0064066dd0 2024-01-26T23:48:32.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:32.233+0000 7f007b7fe700 1 -- 172.21.15.46:0/513608654 <== mgr.7768 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f008404c710 con 0x7f0064066dd0 2024-01-26T23:48:32.234 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:48:32.234 INFO:teuthology.orchestra.run.smithi046.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-01-26T23:48:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:32.235+0000 7f0089dac700 1 -- 172.21.15.46:0/513608654 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0064066dd0 msgr2=0x7f0064069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:32.235+0000 7f0089dac700 1 --2- 172.21.15.46:0/513608654 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0064066dd0 0x7f0064069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0074009d10 tx=0x7f00740091a0 comp rx=0 tx=0).stop 2024-01-26T23:48:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:32.235+0000 7f0089dac700 1 -- 172.21.15.46:0/513608654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00841597e0 msgr2=0x7f00840b2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:48:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:32.235+0000 7f0089dac700 1 --2- 172.21.15.46:0/513608654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00841597e0 0x7f00840b2af0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f007c00c840 tx=0x7f007c001670 comp rx=0 tx=0).stop 2024-01-26T23:48:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:32.235+0000 7f0089dac700 1 -- 172.21.15.46:0/513608654 shutdown_connections 2024-01-26T23:48:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:32.235+0000 7f0089dac700 1 --2- 172.21.15.46:0/513608654 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0064066dd0 0x7f0064069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:32.235+0000 7f0089dac700 1 --2- 172.21.15.46:0/513608654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00841597e0 0x7f00840b2af0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:32.235+0000 7f0089dac700 1 --2- 172.21.15.46:0/513608654 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0084067af0 0x7f0084143830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:32.235+0000 7f0089dac700 1 --2- 172.21.15.46:0/513608654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f008406ebe0 0x7f00840b25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:48:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:32.235+0000 7f0089dac700 1 -- 172.21.15.46:0/513608654 >> 172.21.15.46:0/513608654 conn(0x7f00840cdf80 msgr2=0x7f00840cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:48:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:32.235+0000 7f0089dac700 1 -- 172.21.15.46:0/513608654 shutdown_connections 2024-01-26T23:48:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:48:32.235+0000 7f0089dac700 1 -- 172.21.15.46:0/513608654 wait complete. 2024-01-26T23:48:32.250 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-01-26T23:48:32.250 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{}]} 2024-01-26T23:48:32.250 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:32.251 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:32.274 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:32.274 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:32.274 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:33.275 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:33.275 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:33.302 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:33.302 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:33.302 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:34.303 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:34.304 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:34.331 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:34.331 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:34.331 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:35.332 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:35.333 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:35.360 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:35.360 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:35.360 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:36.361 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:36.361 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:36.388 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:36.388 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:36.388 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:37.389 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:37.389 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:37.416 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:37.416 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:37.416 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:38.417 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:38.417 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:38.445 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:38.445 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:38.445 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:39.446 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:39.446 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:39.474 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:39.474 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:39.474 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:40.475 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:40.475 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:40.503 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:40.504 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:40.504 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:41.505 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:41.505 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:41.532 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:41.532 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:41.532 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:42.533 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:42.533 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:42.561 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:42.561 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:42.561 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:43.562 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:43.562 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:43.590 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:43.590 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:43.590 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:44.590 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:44.591 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:44.618 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:44.618 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:44.618 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:45.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:45.619 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:45.646 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:45.647 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:45.647 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:46.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:46.647 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:46.675 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:46.675 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:46.675 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:47.676 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:47.676 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:47.703 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:47.703 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:47.703 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:48.704 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:48.704 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:48.731 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:48.731 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:48.732 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:49.732 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:49.732 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:49.760 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:49.760 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:49.760 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:50.761 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:50.762 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:50.789 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:50.789 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:50.789 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:51.790 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:51.790 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:51.818 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:51.818 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:51.818 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:52.819 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:52.819 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:52.847 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:52.847 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:52.847 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:53.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:53.849 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:53.876 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:53.877 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:53.877 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:54.878 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:54.878 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:54.906 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:54.906 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:54.906 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:55.626 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:48:55.629 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:48:55.908 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:55.908 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:55.935 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:55.935 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:55.936 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:56.936 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:56.936 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:56.964 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:56.965 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:56.965 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:57.966 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:57.966 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:57.996 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:57.996 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:57.996 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:48:58.997 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:48:58.997 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:48:59.024 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:48:59.024 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:48:59.024 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:49:00.025 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:00.025 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:49:00.052 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-26T23:49:00.052 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-26T23:49:00.052 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:49:00.832 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:00.831+0000 7f9236e1a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:00.832 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:00.831+0000 7f9236e1a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:00.832 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:00.831+0000 7f9236e1a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:00.832 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:00.831+0000 7f9236e1a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:00.832 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:00.831+0000 7f9236e1a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:01.053 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:01.054 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:49:01.081 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_49_00_UTC 2024-01-26T23:49:01.082 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-26-23_49_00_UTC'} 2024-01-26T23:49:01.082 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-01-26-23_49_00_UTC'} 2024-01-26T23:49:01.082 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-26T23:49:01.082 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-01-26-23_49_00_UTC'] 2024-01-26T23:49:01.082 DEBUG:tasks.cephfs.test_snap_schedules:exec=1706312911.83937, now = 1706312941.0826335, timo = 29 2024-01-26T23:49:02.083 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-01-26T23:49:02.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.224+0000 7f8269f5c700 1 Processor -- start 2024-01-26T23:49:02.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.225+0000 7f8269f5c700 1 -- start start 2024-01-26T23:49:02.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.225+0000 7f8269f5c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8264159510 0x7f82641598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:02.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.225+0000 7f8269f5c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82640ca220 0x7f82640c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:02.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.225+0000 7f8269f5c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8264069260 con 0x7f82640ca690 2024-01-26T23:49:02.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.225+0000 7f8269f5c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f826415d670 con 0x7f8264159510 2024-01-26T23:49:02.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.225+0000 7f8269f5c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82640c9750 con 0x7f82640ca220 2024-01-26T23:49:02.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.226+0000 7f82637fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8264159510 0x7f82641598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:02.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.226+0000 7f82637fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8264159510 0x7f82641598e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52136/0 (socket says 172.21.15.46:52136) 2024-01-26T23:49:02.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.226+0000 7f82637fe700 1 -- 172.21.15.46:0/3361302089 learned_addr learned my addr 172.21.15.46:0/3361302089 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:02.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.226+0000 7f8262ffd700 1 --2- 172.21.15.46:0/3361302089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82640ca220 0x7f82640c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:02.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.226+0000 7f82637fe700 1 -- 172.21.15.46:0/3361302089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82640ca220 msgr2=0x7f82640c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:02.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.226+0000 7f82637fe700 1 --2- 172.21.15.46:0/3361302089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82640ca220 0x7f82640c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:02.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.226+0000 7f82637fe700 1 -- 172.21.15.46:0/3361302089 >> v1:172.21.15.106:6789/0 conn(0x7f82640ca690 legacy=0x7f82640cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:02.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.226+0000 7f82637fe700 1 -- 172.21.15.46:0/3361302089 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82640c9a30 con 0x7f8264159510 2024-01-26T23:49:02.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.226+0000 7f82637fe700 1 --2- 172.21.15.46:0/3361302089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8264159510 0x7f82641598e0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f82540099e0 tx=0x7f8254009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db609fc95bb5be43 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.226+0000 7f82627fc700 1 -- 172.21.15.46:0/3361302089 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825400b820 con 0x7f8264159510 2024-01-26T23:49:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.226+0000 7f82627fc700 1 -- 172.21.15.46:0/3361302089 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825400b980 con 0x7f8264159510 2024-01-26T23:49:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.226+0000 7f82627fc700 1 -- 172.21.15.46:0/3361302089 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825400bd20 con 0x7f8264159510 2024-01-26T23:49:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.227+0000 7f8269f5c700 1 -- 172.21.15.46:0/3361302089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8264159510 msgr2=0x7f82641598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:02.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.227+0000 7f8269f5c700 1 --2- 172.21.15.46:0/3361302089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8264159510 0x7f82641598e0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f82540099e0 tx=0x7f8254009250 comp rx=0 tx=0).stop 2024-01-26T23:49:02.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.227+0000 7f8269f5c700 1 -- 172.21.15.46:0/3361302089 shutdown_connections 2024-01-26T23:49:02.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.227+0000 7f8269f5c700 1 --2- 172.21.15.46:0/3361302089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8264159510 0x7f82641598e0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:02.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.227+0000 7f8269f5c700 1 --2- 172.21.15.46:0/3361302089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82640ca220 0x7f82640c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:02.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.227+0000 7f8269f5c700 1 -- 172.21.15.46:0/3361302089 >> 172.21.15.46:0/3361302089 conn(0x7f826406bd90 msgr2=0x7f826406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:02.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.227+0000 7f8269f5c700 1 -- 172.21.15.46:0/3361302089 shutdown_connections 2024-01-26T23:49:02.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.227+0000 7f8269f5c700 1 -- 172.21.15.46:0/3361302089 wait complete. 2024-01-26T23:49:02.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.228+0000 7f8269f5c700 1 Processor -- start 2024-01-26T23:49:02.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.229+0000 7f8269f5c700 1 -- start start 2024-01-26T23:49:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.229+0000 7f8269f5c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82640ca220 0x7f8264156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.229+0000 7f8269f5c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8264159510 0x7f82641566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.229+0000 7f8269f5c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f826414d350 0x7f82640c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.229+0000 7f8269f5c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8264157dd0 con 0x7f82640ca220 2024-01-26T23:49:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.229+0000 7f8269f5c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82640b3840 con 0x7f8264159510 2024-01-26T23:49:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.229+0000 7f8269f5c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8264156d30 con 0x7f826414d350 2024-01-26T23:49:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.229+0000 7f82637fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82640ca220 0x7f8264156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.229+0000 7f82637fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82640ca220 0x7f8264156190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52150/0 (socket says 172.21.15.46:52150) 2024-01-26T23:49:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.229+0000 7f82637fe700 1 -- 172.21.15.46:0/4147353488 learned_addr learned my addr 172.21.15.46:0/4147353488 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.230+0000 7f8262ffd700 1 --2- 172.21.15.46:0/4147353488 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8264159510 0x7f82641566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.230+0000 7f8263fff700 1 --2- 172.21.15.46:0/4147353488 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f826414d350 0x7f82640c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.230+0000 7f82637fe700 1 -- 172.21.15.46:0/4147353488 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f826414d350 msgr2=0x7f82640c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.230+0000 7f82637fe700 1 --2- 172.21.15.46:0/4147353488 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f826414d350 0x7f82640c5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.230+0000 7f82637fe700 1 -- 172.21.15.46:0/4147353488 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8264159510 msgr2=0x7f82641566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.230+0000 7f82637fe700 1 --2- 172.21.15.46:0/4147353488 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8264159510 0x7f82641566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.230+0000 7f82637fe700 1 -- 172.21.15.46:0/4147353488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82540094f0 con 0x7f82640ca220 2024-01-26T23:49:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.230+0000 7f82637fe700 1 --2- 172.21.15.46:0/4147353488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82640ca220 0x7f8264156190 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f826406f5a0 tx=0x7f8254017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.230+0000 7f824b7fe700 1 -- 172.21.15.46:0/4147353488 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254011d10 con 0x7f82640ca220 2024-01-26T23:49:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.230+0000 7f824b7fe700 1 -- 172.21.15.46:0/4147353488 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8254011e90 con 0x7f82640ca220 2024-01-26T23:49:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.230+0000 7f8269f5c700 1 -- 172.21.15.46:0/4147353488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8264157010 con 0x7f82640ca220 2024-01-26T23:49:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.230+0000 7f824b7fe700 1 -- 172.21.15.46:0/4147353488 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82540266a0 con 0x7f82640ca220 2024-01-26T23:49:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.230+0000 7f8269f5c700 1 -- 172.21.15.46:0/4147353488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82641575d0 con 0x7f82640ca220 2024-01-26T23:49:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.231+0000 7f8269f5c700 1 -- 172.21.15.46:0/4147353488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f826404c710 con 0x7f82640ca220 2024-01-26T23:49:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.232+0000 7f824b7fe700 1 -- 172.21.15.46:0/4147353488 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 61) v1 ==== 84975+0+0 (secure 0 0 0) 0x7f825401a420 con 0x7f82640ca220 2024-01-26T23:49:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.232+0000 7f824b7fe700 1 --2- 172.21.15.46:0/4147353488 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f82500671c0 0x7f8250069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.232+0000 7f824b7fe700 1 -- 172.21.15.46:0/4147353488 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f82540868e0 con 0x7f82640ca220 2024-01-26T23:49:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.232+0000 7f8262ffd700 1 --2- 172.21.15.46:0/4147353488 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f82500671c0 0x7f8250069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:02.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.233+0000 7f8262ffd700 1 --2- 172.21.15.46:0/4147353488 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f82500671c0 0x7f8250069670 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8264158bd0 tx=0x7f8258008040 comp rx=0 tx=0).ready entity=mgr.7768 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:02.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.237+0000 7f824b7fe700 1 -- 172.21.15.46:0/4147353488 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8254053400 con 0x7f82640ca220 2024-01-26T23:49:02.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.469+0000 7f8269f5c700 1 -- 172.21.15.46:0/4147353488 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f82640b3db0 con 0x7f82500671c0 2024-01-26T23:49:02.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.533+0000 7f824b7fe700 1 -- 172.21.15.46:0/4147353488 <== mgr.7768 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f82580107b0 con 0x7f82500671c0 2024-01-26T23:49:02.533 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path snap_test_dir1 2024-01-26T23:49:02.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.535+0000 7f8269f5c700 1 -- 172.21.15.46:0/4147353488 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f82500671c0 msgr2=0x7f8250069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:02.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.535+0000 7f8269f5c700 1 --2- 172.21.15.46:0/4147353488 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f82500671c0 0x7f8250069670 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8264158bd0 tx=0x7f8258008040 comp rx=0 tx=0).stop 2024-01-26T23:49:02.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.535+0000 7f8269f5c700 1 -- 172.21.15.46:0/4147353488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82640ca220 msgr2=0x7f8264156190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:02.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.535+0000 7f8269f5c700 1 --2- 172.21.15.46:0/4147353488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82640ca220 0x7f8264156190 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f826406f5a0 tx=0x7f8254017000 comp rx=0 tx=0).stop 2024-01-26T23:49:02.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.535+0000 7f8269f5c700 1 -- 172.21.15.46:0/4147353488 shutdown_connections 2024-01-26T23:49:02.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.535+0000 7f8269f5c700 1 --2- 172.21.15.46:0/4147353488 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f82500671c0 0x7f8250069670 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:02.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.535+0000 7f8269f5c700 1 --2- 172.21.15.46:0/4147353488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82640ca220 0x7f8264156190 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:02.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.535+0000 7f8269f5c700 1 --2- 172.21.15.46:0/4147353488 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f826414d350 0x7f82640c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:02.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.535+0000 7f8269f5c700 1 --2- 172.21.15.46:0/4147353488 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8264159510 0x7f82641566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:02.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.535+0000 7f8269f5c700 1 -- 172.21.15.46:0/4147353488 >> 172.21.15.46:0/4147353488 conn(0x7f826406bd90 msgr2=0x7f82641580b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:02.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.535+0000 7f8269f5c700 1 -- 172.21.15.46:0/4147353488 shutdown_connections 2024-01-26T23:49:02.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.535+0000 7f8269f5c700 1 -- 172.21.15.46:0/4147353488 wait complete. 2024-01-26T23:49:02.550 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-01-26T23:49:02.550 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:02.550 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-26T23:49:02.574 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_49_00_UTC 2024-01-26T23:49:02.574 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-26-23_49_00_UTC 2024-01-26T23:49:02.574 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:02.574 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-26-23_49_00_UTC) 2024-01-26T23:49:02.644 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:02.644 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-01-26T23:49:02.710 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-26T23:49:02.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.895+0000 7f78d7439700 1 Processor -- start 2024-01-26T23:49:02.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.895+0000 7f78d7439700 1 -- start start 2024-01-26T23:49:02.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.896+0000 7f78d7439700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f78d0157200 0x7f78d0157680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:02.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.896+0000 7f78d7439700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d006c160 con 0x7f78d0067af0 2024-01-26T23:49:02.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.896+0000 7f78d7439700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d006ca00 con 0x7f78d0067ec0 2024-01-26T23:49:02.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.896+0000 7f78d7439700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d0068330 con 0x7f78d0157200 2024-01-26T23:49:02.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.896+0000 7f78d59d6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f78d0157200 0x7f78d0157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:02.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.896+0000 7f78d51d5700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f78d0067af0 0x7f78d006ce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42736/0 (socket says 172.21.15.46:42736) 2024-01-26T23:49:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.896+0000 7f78d59d6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f78d0157200 0x7f78d0157680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52162/0 (socket says 172.21.15.46:52162) 2024-01-26T23:49:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.896+0000 7f78d51d5700 1 -- 172.21.15.46:0/2748092200 learned_addr learned my addr 172.21.15.46:0/2748092200 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.897+0000 7f78d59d6700 1 -- 172.21.15.46:0/2748092200 >> v1:172.21.15.106:6789/0 conn(0x7f78d0067ec0 legacy=0x7f78d00c7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.897+0000 7f78d59d6700 1 -- 172.21.15.46:0/2748092200 >> v1:172.21.15.46:6789/0 conn(0x7f78d0067af0 legacy=0x7f78d006ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.897+0000 7f78d59d6700 1 -- 172.21.15.46:0/2748092200 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d00c84f0 con 0x7f78d0157200 2024-01-26T23:49:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.897+0000 7f78c3fff700 1 -- 172.21.15.46:0/2748092200 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143169164 0 0) 0x7f78d006c160 con 0x7f78d0067af0 2024-01-26T23:49:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.897+0000 7f78d59d6700 1 --2- 172.21.15.46:0/2748092200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78d0157200 0x7f78d0157680 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f78c8009e60 tx=0x7f78c8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c468ee7c7861ff72 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.897+0000 7f78c3fff700 1 -- 172.21.15.46:0/2748092200 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c800e070 con 0x7f78d0157200 2024-01-26T23:49:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.897+0000 7f78c3fff700 1 -- 172.21.15.46:0/2748092200 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c800bc30 con 0x7f78d0157200 2024-01-26T23:49:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.897+0000 7f78c3fff700 1 -- 172.21.15.46:0/2748092200 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c8012430 con 0x7f78d0157200 2024-01-26T23:49:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.897+0000 7f78d7439700 1 -- 172.21.15.46:0/2748092200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78d0157200 msgr2=0x7f78d0157680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.898+0000 7f78d7439700 1 --2- 172.21.15.46:0/2748092200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78d0157200 0x7f78d0157680 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f78c8009e60 tx=0x7f78c8009780 comp rx=0 tx=0).stop 2024-01-26T23:49:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.898+0000 7f78d7439700 1 -- 172.21.15.46:0/2748092200 shutdown_connections 2024-01-26T23:49:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.898+0000 7f78d7439700 1 --2- 172.21.15.46:0/2748092200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78d0157200 0x7f78d0157680 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:02.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.898+0000 7f78d7439700 1 -- 172.21.15.46:0/2748092200 >> 172.21.15.46:0/2748092200 conn(0x7f78d00ca9a0 msgr2=0x7f78d0068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:02.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.898+0000 7f78d7439700 1 -- 172.21.15.46:0/2748092200 shutdown_connections 2024-01-26T23:49:02.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.898+0000 7f78d7439700 1 -- 172.21.15.46:0/2748092200 wait complete. 2024-01-26T23:49:02.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.899+0000 7f78d7439700 1 Processor -- start 2024-01-26T23:49:02.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.899+0000 7f78d7439700 1 -- start start 2024-01-26T23:49:02.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.900+0000 7f78d7439700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78d0067af0 0x7f78d0154080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:02.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.900+0000 7f78d7439700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78d0067ec0 0x7f78d00c5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:02.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.900+0000 7f78d7439700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78d0157200 0x7f78d015d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:02.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.900+0000 7f78d7439700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d01545c0 con 0x7f78d0067af0 2024-01-26T23:49:02.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.900+0000 7f78d7439700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d00c5400 con 0x7f78d0067ec0 2024-01-26T23:49:02.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.900+0000 7f78d7439700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d00c4510 con 0x7f78d0157200 2024-01-26T23:49:02.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.900+0000 7f78d51d5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78d0067af0 0x7f78d0154080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:02.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.900+0000 7f78d51d5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78d0067af0 0x7f78d0154080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52174/0 (socket says 172.21.15.46:52174) 2024-01-26T23:49:02.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.900+0000 7f78d51d5700 1 -- 172.21.15.46:0/4070271644 learned_addr learned my addr 172.21.15.46:0/4070271644 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:02.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.900+0000 7f78d59d6700 1 --2- 172.21.15.46:0/4070271644 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78d0157200 0x7f78d015d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:02.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.900+0000 7f78d49d4700 1 --2- 172.21.15.46:0/4070271644 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78d0067ec0 0x7f78d00c5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:02.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.901+0000 7f78d51d5700 1 -- 172.21.15.46:0/4070271644 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78d0157200 msgr2=0x7f78d015d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:02.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.901+0000 7f78d51d5700 1 --2- 172.21.15.46:0/4070271644 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78d0157200 0x7f78d015d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:02.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.901+0000 7f78d51d5700 1 -- 172.21.15.46:0/4070271644 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78d0067ec0 msgr2=0x7f78d00c5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:02.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.901+0000 7f78d51d5700 1 --2- 172.21.15.46:0/4070271644 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78d0067ec0 0x7f78d00c5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:02.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.901+0000 7f78d51d5700 1 -- 172.21.15.46:0/4070271644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c8009a20 con 0x7f78d0067af0 2024-01-26T23:49:02.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.901+0000 7f78d51d5700 1 --2- 172.21.15.46:0/4070271644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78d0067af0 0x7f78d0154080 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f78c400c7e0 tx=0x7f78c4007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:02.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.901+0000 7f78c17fa700 1 -- 172.21.15.46:0/4070271644 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c4005b30 con 0x7f78d0067af0 2024-01-26T23:49:02.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.901+0000 7f78c17fa700 1 -- 172.21.15.46:0/4070271644 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c4005c90 con 0x7f78d0067af0 2024-01-26T23:49:02.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.901+0000 7f78d7439700 1 -- 172.21.15.46:0/4070271644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d00c47f0 con 0x7f78d0067af0 2024-01-26T23:49:02.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.901+0000 7f78c17fa700 1 -- 172.21.15.46:0/4070271644 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c4007300 con 0x7f78d0067af0 2024-01-26T23:49:02.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.901+0000 7f78d7439700 1 -- 172.21.15.46:0/4070271644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d00c4db0 con 0x7f78d0067af0 2024-01-26T23:49:02.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.902+0000 7f78d7439700 1 -- 172.21.15.46:0/4070271644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d004c710 con 0x7f78d0067af0 2024-01-26T23:49:02.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.903+0000 7f78c17fa700 1 -- 172.21.15.46:0/4070271644 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 61) v1 ==== 84975+0+0 (secure 0 0 0) 0x7f78c401b3b0 con 0x7f78d0067af0 2024-01-26T23:49:02.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.903+0000 7f78c17fa700 1 --2- 172.21.15.46:0/4070271644 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f78b40671c0 0x7f78b4069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:02.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.903+0000 7f78c17fa700 1 -- 172.21.15.46:0/4070271644 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f78c40842b0 con 0x7f78d0067af0 2024-01-26T23:49:02.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.903+0000 7f78d49d4700 1 --2- 172.21.15.46:0/4070271644 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f78b40671c0 0x7f78b4069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:02.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.904+0000 7f78d49d4700 1 --2- 172.21.15.46:0/4070271644 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f78b40671c0 0x7f78b4069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f78d00c78e0 tx=0x7f78cc000bc0 comp rx=0 tx=0).ready entity=mgr.7768 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:02.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:02.908+0000 7f78c17fa700 1 -- 172.21.15.46:0/4070271644 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f78c4050df0 con 0x7f78d0067af0 2024-01-26T23:49:03.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:03.178+0000 7f78d7439700 1 -- 172.21.15.46:0/4070271644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f78d006c3b0 con 0x7f78d0067af0 2024-01-26T23:49:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:03.853+0000 7f78c17fa700 1 -- 172.21.15.46:0/4070271644 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 62) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f78c40511c0 con 0x7f78d0067af0 2024-01-26T23:49:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:03.854+0000 7f78c17fa700 1 -- 172.21.15.46:0/4070271644 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v62) v1 ==== 93+0+0 (secure 0 0 0) 0x7f78c40076f0 con 0x7f78d0067af0 2024-01-26T23:49:03.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:03.857+0000 7f78d7439700 1 -- 172.21.15.46:0/4070271644 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f78b40671c0 msgr2=0x7f78b4069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:03.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:03.857+0000 7f78d7439700 1 --2- 172.21.15.46:0/4070271644 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f78b40671c0 0x7f78b4069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f78d00c78e0 tx=0x7f78cc000bc0 comp rx=0 tx=0).stop 2024-01-26T23:49:03.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:03.857+0000 7f78d7439700 1 -- 172.21.15.46:0/4070271644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78d0067af0 msgr2=0x7f78d0154080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:03.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:03.857+0000 7f78d7439700 1 --2- 172.21.15.46:0/4070271644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78d0067af0 0x7f78d0154080 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f78c400c7e0 tx=0x7f78c4007df0 comp rx=0 tx=0).stop 2024-01-26T23:49:03.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:03.857+0000 7f78d7439700 1 -- 172.21.15.46:0/4070271644 shutdown_connections 2024-01-26T23:49:03.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:03.857+0000 7f78d7439700 1 --2- 172.21.15.46:0/4070271644 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f78b40671c0 0x7f78b4069670 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:03.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:03.857+0000 7f78d7439700 1 --2- 172.21.15.46:0/4070271644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78d0067af0 0x7f78d0154080 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:03.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:03.857+0000 7f78d7439700 1 --2- 172.21.15.46:0/4070271644 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78d0157200 0x7f78d015d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:03.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:03.857+0000 7f78d7439700 1 --2- 172.21.15.46:0/4070271644 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78d0067ec0 0x7f78d00c5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:03.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:03.857+0000 7f78d7439700 1 -- 172.21.15.46:0/4070271644 >> 172.21.15.46:0/4070271644 conn(0x7f78d00ca9a0 msgr2=0x7f78d006eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:03.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:03.858+0000 7f78d7439700 1 -- 172.21.15.46:0/4070271644 shutdown_connections 2024-01-26T23:49:03.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:03.858+0000 7f78d7439700 1 -- 172.21.15.46:0/4070271644 wait complete. 2024-01-26T23:49:03.873 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-26T23:49:03.873 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:49:03.928 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:49:03.967 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-26T23:49:03.967 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-26T23:49:03.967 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-26T23:49:03.967 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-26T23:49:04.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[82900]: fuse finished with error 0 and tester_r 0 2024-01-26T23:49:04.008 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:49:04.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:49:04.008 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-26T23:49:04.179 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:49:04.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:49:04.180 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-26T23:49:04.193 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:04.193+0000 7f3d5745a1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:49:04.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:49:04.196 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-26T23:49:04.196 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:49:04.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:49:04.197 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-26T23:49:04.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:49:04.239 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-26T23:49:04.239 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:49:04.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:49:04.239 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-26T23:49:04.242 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:04.242+0000 7f30bcb891c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:49:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:49:04.256 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-26T23:49:04.256 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:49:04.272 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:04.273+0000 7f3d5745a1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:49:04.334 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:04.334+0000 7f30bcb891c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:49:04.344 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:04.344+0000 7f3d5745a1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:49:04.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.389+0000 7fd7923ff700 1 Processor -- start 2024-01-26T23:49:04.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.389+0000 7fd7923ff700 1 -- start start 2024-01-26T23:49:04.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.390+0000 7fd7923ff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd78c0c6520 0x7fd78c0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:04.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.390+0000 7fd7923ff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd78c0ced50 0x7fd78c0cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:04.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.390+0000 7fd7923ff700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd78c15a0b0 con 0x7fd78c0b2b40 2024-01-26T23:49:04.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.390+0000 7fd7923ff700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd78c0af070 con 0x7fd78c0c6520 2024-01-26T23:49:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.390+0000 7fd7923ff700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c099fc0 con 0x7fd78c0ced50 2024-01-26T23:49:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.390+0000 7fd78bfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd78c0b2b40 0x7fd78c0ae960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42754/0 (socket says 172.21.15.46:42754) 2024-01-26T23:49:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.390+0000 7fd78bfff700 1 -- 172.21.15.46:0/3035972424 learned_addr learned my addr 172.21.15.46:0/3035972424 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.390+0000 7fd78b7fe700 1 --2- 172.21.15.46:0/3035972424 >> v2:172.21.15.106:3300/0 conn(0x7fd78c0c6520 0x7fd78c0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.390+0000 7fd79099c700 1 --2- 172.21.15.46:0/3035972424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd78c0ced50 0x7fd78c0cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.391+0000 7fd78affd700 1 -- 172.21.15.46:0/3035972424 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128813778 0 0) 0x7fd78c15a0b0 con 0x7fd78c0b2b40 2024-01-26T23:49:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.391+0000 7fd78affd700 1 -- 172.21.15.46:0/3035972424 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd774003570 con 0x7fd78c0b2b40 2024-01-26T23:49:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.391+0000 7fd78b7fe700 1 -- 172.21.15.46:0/3035972424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd78c0ced50 msgr2=0x7fd78c0cf1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.391+0000 7fd78b7fe700 1 --2- 172.21.15.46:0/3035972424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd78c0ced50 0x7fd78c0cf1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.391+0000 7fd78b7fe700 1 -- 172.21.15.46:0/3035972424 >> v1:172.21.15.46:6789/0 conn(0x7fd78c0b2b40 legacy=0x7fd78c0ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.391+0000 7fd78b7fe700 1 -- 172.21.15.46:0/3035972424 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78c0ae4d0 con 0x7fd78c0c6520 2024-01-26T23:49:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.391+0000 7fd79099c700 1 --2- 172.21.15.46:0/3035972424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd78c0ced50 0x7fd78c0cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:04.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.391+0000 7fd78b7fe700 1 --2- 172.21.15.46:0/3035972424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd78c0c6520 0x7fd78c0ad1c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd7800098e0 tx=0x7fd7800091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0dcd4ac80ac718f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:04.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.391+0000 7fd78affd700 1 -- 172.21.15.46:0/3035972424 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd780011630 con 0x7fd78c0c6520 2024-01-26T23:49:04.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.391+0000 7fd78affd700 1 -- 172.21.15.46:0/3035972424 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd780011790 con 0x7fd78c0c6520 2024-01-26T23:49:04.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.391+0000 7fd78affd700 1 -- 172.21.15.46:0/3035972424 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd780011b30 con 0x7fd78c0c6520 2024-01-26T23:49:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.392+0000 7fd7923ff700 1 -- 172.21.15.46:0/3035972424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd78c0c6520 msgr2=0x7fd78c0ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.392+0000 7fd7923ff700 1 --2- 172.21.15.46:0/3035972424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd78c0c6520 0x7fd78c0ad1c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd7800098e0 tx=0x7fd7800091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.392+0000 7fd7923ff700 1 -- 172.21.15.46:0/3035972424 shutdown_connections 2024-01-26T23:49:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.392+0000 7fd7923ff700 1 --2- 172.21.15.46:0/3035972424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd78c0ced50 0x7fd78c0cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.392+0000 7fd7923ff700 1 --2- 172.21.15.46:0/3035972424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd78c0c6520 0x7fd78c0ad1c0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.392+0000 7fd7923ff700 1 -- 172.21.15.46:0/3035972424 >> 172.21.15.46:0/3035972424 conn(0x7fd78c099a30 msgr2=0x7fd78c06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.392+0000 7fd7923ff700 1 -- 172.21.15.46:0/3035972424 shutdown_connections 2024-01-26T23:49:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.392+0000 7fd7923ff700 1 -- 172.21.15.46:0/3035972424 wait complete. 2024-01-26T23:49:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.393+0000 7fd7923ff700 1 Processor -- start 2024-01-26T23:49:04.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.393+0000 7fd7923ff700 1 -- start start 2024-01-26T23:49:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.394+0000 7fd7923ff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd78c0b2b40 0x7fd78c0ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.394+0000 7fd7923ff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd78c0c6520 0x7fd78c1573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.394+0000 7fd7923ff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd78c0ced50 0x7fd78c0c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.394+0000 7fd7923ff700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c0c9be0 con 0x7fd78c0c6520 2024-01-26T23:49:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.394+0000 7fd7923ff700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c157920 con 0x7fd78c0ced50 2024-01-26T23:49:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.394+0000 7fd7923ff700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c0c8cf0 con 0x7fd78c0b2b40 2024-01-26T23:49:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.394+0000 7fd78b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd78c0c6520 0x7fd78c1573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.394+0000 7fd78b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd78c0c6520 0x7fd78c1573e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52196/0 (socket says 172.21.15.46:52196) 2024-01-26T23:49:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.394+0000 7fd78b7fe700 1 -- 172.21.15.46:0/2888236802 learned_addr learned my addr 172.21.15.46:0/2888236802 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.394+0000 7fd79099c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd78c0ced50 0x7fd78c0c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.394+0000 7fd78bfff700 1 --2- 172.21.15.46:0/2888236802 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd78c0b2b40 0x7fd78c0ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.395+0000 7fd79099c700 1 -- 172.21.15.46:0/2888236802 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd78c0b2b40 msgr2=0x7fd78c0ca070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:04.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.395+0000 7fd79099c700 1 --2- 172.21.15.46:0/2888236802 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd78c0b2b40 0x7fd78c0ca070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.395+0000 7fd79099c700 1 -- 172.21.15.46:0/2888236802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd78c0c6520 msgr2=0x7fd78c1573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.395+0000 7fd79099c700 1 --2- 172.21.15.46:0/2888236802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd78c0c6520 0x7fd78c1573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.395+0000 7fd79099c700 1 -- 172.21.15.46:0/2888236802 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd784009a20 con 0x7fd78c0ced50 2024-01-26T23:49:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.395+0000 7fd78b7fe700 1 --2- 172.21.15.46:0/2888236802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd78c0c6520 0x7fd78c1573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.395+0000 7fd79099c700 1 --2- 172.21.15.46:0/2888236802 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd78c0ced50 0x7fd78c0c81e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd784009e70 tx=0x7fd78400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.395+0000 7fd773fff700 1 -- 172.21.15.46:0/2888236802 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78400caa0 con 0x7fd78c0ced50 2024-01-26T23:49:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.395+0000 7fd773fff700 1 -- 172.21.15.46:0/2888236802 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78400f040 con 0x7fd78c0ced50 2024-01-26T23:49:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.395+0000 7fd773fff700 1 -- 172.21.15.46:0/2888236802 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78400ce90 con 0x7fd78c0ced50 2024-01-26T23:49:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.395+0000 7fd7923ff700 1 -- 172.21.15.46:0/2888236802 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd780009420 con 0x7fd78c0ced50 2024-01-26T23:49:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.395+0000 7fd7923ff700 1 -- 172.21.15.46:0/2888236802 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd78c0c92b0 con 0x7fd78c0ced50 2024-01-26T23:49:04.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.397+0000 7fd7923ff700 1 -- 172.21.15.46:0/2888236802 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd78c0c9d50 con 0x7fd78c0ced50 2024-01-26T23:49:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.397+0000 7fd773fff700 1 -- 172.21.15.46:0/2888236802 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 62) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fd784003020 con 0x7fd78c0ced50 2024-01-26T23:49:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.397+0000 7fd773fff700 1 --2- 172.21.15.46:0/2888236802 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd7780671f0 0x7fd7780696a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.397+0000 7fd78bfff700 1 -- 172.21.15.46:0/2888236802 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd7780671f0 msgr2=0x7fd7780696a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.397+0000 7fd773fff700 1 -- 172.21.15.46:0/2888236802 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fd7840855f0 con 0x7fd78c0ced50 2024-01-26T23:49:04.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.397+0000 7fd78bfff700 1 --2- 172.21.15.46:0/2888236802 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd7780671f0 0x7fd7780696a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:04.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.402+0000 7fd773fff700 1 -- 172.21.15.46:0/2888236802 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd784052050 con 0x7fd78c0ced50 2024-01-26T23:49:04.411 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:04.411+0000 7f3d5745a1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:49:04.418 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:04.418+0000 7f30bcb891c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:49:04.494 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:04.493+0000 7f30bcb891c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:49:04.567 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:04.566+0000 7f3d5745a1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:49:04.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.598+0000 7fd78bfff700 1 -- 172.21.15.46:0/2888236802 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd7780671f0 msgr2=0x7fd7780696a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:04.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.598+0000 7fd78bfff700 1 --2- 172.21.15.46:0/2888236802 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd7780671f0 0x7fd7780696a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:04.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.655+0000 7fd7923ff700 1 -- 172.21.15.46:0/2888236802 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd78c04c710 con 0x7fd78c0ced50 2024-01-26T23:49:04.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.656+0000 7fd773fff700 1 -- 172.21.15.46:0/2888236802 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4599 (secure 0 0 0) 0x7fd784055670 con 0x7fd78c0ced50 2024-01-26T23:49:04.656 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:04.656 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":62,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6923,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/4123353304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4123353304},{"type":"v1","addr":"172.21.15.106:6834","nonce":4123353304}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/690662483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":690662483},{"type":"v1","addr":"172.21.15.106:6836","nonce":690662483}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7791,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3035969379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3035969379},{"type":"v1","addr":"172.21.15.46:6834","nonce":3035969379}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:48:11.234632+0000","modified":"2024-01-26T23:48:14.815720+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7794},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7794":{"gid":7794,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/783623731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":783623731},{"type":"v1","addr":"172.21.15.46:6835","nonce":783623731}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2024-01-26T23:49:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.658+0000 7fd7923ff700 1 -- 172.21.15.46:0/2888236802 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd7780671f0 msgr2=0x7fd7780696a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.658+0000 7fd7923ff700 1 --2- 172.21.15.46:0/2888236802 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd7780671f0 0x7fd7780696a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.658+0000 7fd7923ff700 1 -- 172.21.15.46:0/2888236802 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd78c0ced50 msgr2=0x7fd78c0c81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.658+0000 7fd7923ff700 1 --2- 172.21.15.46:0/2888236802 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd78c0ced50 0x7fd78c0c81e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd784009e70 tx=0x7fd78400b040 comp rx=0 tx=0).stop 2024-01-26T23:49:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.658+0000 7fd7923ff700 1 -- 172.21.15.46:0/2888236802 shutdown_connections 2024-01-26T23:49:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.658+0000 7fd7923ff700 1 --2- 172.21.15.46:0/2888236802 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd7780671f0 0x7fd7780696a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.658+0000 7fd7923ff700 1 --2- 172.21.15.46:0/2888236802 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd78c0ced50 0x7fd78c0c81e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.658+0000 7fd7923ff700 1 --2- 172.21.15.46:0/2888236802 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd78c0b2b40 0x7fd78c0ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.658+0000 7fd7923ff700 1 --2- 172.21.15.46:0/2888236802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd78c0c6520 0x7fd78c1573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.658+0000 7fd7923ff700 1 -- 172.21.15.46:0/2888236802 >> 172.21.15.46:0/2888236802 conn(0x7fd78c099a30 msgr2=0x7fd78c159250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.658+0000 7fd7923ff700 1 -- 172.21.15.46:0/2888236802 shutdown_connections 2024-01-26T23:49:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.658+0000 7fd7923ff700 1 -- 172.21.15.46:0/2888236802 wait complete. 2024-01-26T23:49:04.660 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 62 2024-01-26T23:49:04.669 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:04.669+0000 7f30bcb891c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:49:04.673 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:49:04.703 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:04.702+0000 7f3d5745a1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:49:04.769 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:04.768+0000 7f3d5745a1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:49:04.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.805+0000 7faae7e5a700 1 Processor -- start 2024-01-26T23:49:04.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.806+0000 7faae7e5a700 1 -- start start 2024-01-26T23:49:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.806+0000 7faae7e5a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faae015ce40 0x7faae00d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.806+0000 7faae7e5a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7faae015d210 0x7faae00cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.806+0000 7faae7e5a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faae00d1390 0x7faae00cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.806+0000 7faae7e5a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faae00cff70 con 0x7faae00d1390 2024-01-26T23:49:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.806+0000 7faae7e5a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faae00cf400 con 0x7faae015d210 2024-01-26T23:49:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.807+0000 7faae7e5a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae00cf660 con 0x7faae015ce40 2024-01-26T23:49:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.807+0000 7faae63f7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faae00d1390 0x7faae00cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.807+0000 7faae63f7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faae00d1390 0x7faae00cbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52210/0 (socket says 172.21.15.46:52210) 2024-01-26T23:49:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.807+0000 7faae63f7700 1 -- 172.21.15.46:0/2743758603 learned_addr learned my addr 172.21.15.46:0/2743758603 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.807+0000 7faae53f5700 1 --2- 172.21.15.46:0/2743758603 >> v2:172.21.15.106:3300/0 conn(0x7faae015d210 0x7faae00cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.807+0000 7faae5bf6700 1 --2- 172.21.15.46:0/2743758603 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faae015ce40 0x7faae00d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.807+0000 7faae63f7700 1 -- 172.21.15.46:0/2743758603 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faae015ce40 msgr2=0x7faae00d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.807+0000 7faae63f7700 1 --2- 172.21.15.46:0/2743758603 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faae015ce40 0x7faae00d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.807+0000 7faae63f7700 1 -- 172.21.15.46:0/2743758603 >> v2:172.21.15.106:3300/0 conn(0x7faae015d210 msgr2=0x7faae00cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.807+0000 7faae63f7700 1 --2- 172.21.15.46:0/2743758603 >> v2:172.21.15.106:3300/0 conn(0x7faae015d210 0x7faae00cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.807+0000 7faae63f7700 1 -- 172.21.15.46:0/2743758603 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae00d0500 con 0x7faae00d1390 2024-01-26T23:49:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.807+0000 7faae63f7700 1 --2- 172.21.15.46:0/2743758603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae00d1390 0x7faae00cbff0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7faae0088cb0 tx=0x7faad8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2138d8acb14f8cbd server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.807+0000 7faae4bf4700 1 -- 172.21.15.46:0/2743758603 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad800e070 con 0x7faae00d1390 2024-01-26T23:49:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.808+0000 7faae4bf4700 1 -- 172.21.15.46:0/2743758603 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad800b970 con 0x7faae00d1390 2024-01-26T23:49:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.808+0000 7faae4bf4700 1 -- 172.21.15.46:0/2743758603 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad80092e0 con 0x7faae00d1390 2024-01-26T23:49:04.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.808+0000 7faae7e5a700 1 -- 172.21.15.46:0/2743758603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae00d1390 msgr2=0x7faae00cbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:04.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.808+0000 7faae7e5a700 1 --2- 172.21.15.46:0/2743758603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae00d1390 0x7faae00cbff0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7faae0088cb0 tx=0x7faad8009840 comp rx=0 tx=0).stop 2024-01-26T23:49:04.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.808+0000 7faae7e5a700 1 -- 172.21.15.46:0/2743758603 shutdown_connections 2024-01-26T23:49:04.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.808+0000 7faae7e5a700 1 --2- 172.21.15.46:0/2743758603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae00d1390 0x7faae00cbff0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.808+0000 7faae7e5a700 1 --2- 172.21.15.46:0/2743758603 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faae015ce40 0x7faae00d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.808+0000 7faae7e5a700 1 --2- 172.21.15.46:0/2743758603 >> v2:172.21.15.106:3300/0 conn(0x7faae015d210 0x7faae00cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.808+0000 7faae7e5a700 1 -- 172.21.15.46:0/2743758603 >> 172.21.15.46:0/2743758603 conn(0x7faae006f840 msgr2=0x7faae00d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:04.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.808+0000 7faae7e5a700 1 -- 172.21.15.46:0/2743758603 shutdown_connections 2024-01-26T23:49:04.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.808+0000 7faae7e5a700 1 -- 172.21.15.46:0/2743758603 wait complete. 2024-01-26T23:49:04.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.810+0000 7faae7e5a700 1 Processor -- start 2024-01-26T23:49:04.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.810+0000 7faae7e5a700 1 -- start start 2024-01-26T23:49:04.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.810+0000 7faae7e5a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faae00d1390 0x7faae014a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:04.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.810+0000 7faae7e5a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae015ce40 0x7faae00c6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:04.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.810+0000 7faae7e5a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faae015d210 0x7faae009cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:04.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.810+0000 7faae7e5a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae0147850 con 0x7faae015ce40 2024-01-26T23:49:04.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.810+0000 7faae7e5a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae009b290 con 0x7faae015d210 2024-01-26T23:49:04.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.810+0000 7faae7e5a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae014a780 con 0x7faae00d1390 2024-01-26T23:49:04.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.811+0000 7faae53f5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae015ce40 0x7faae00c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:04.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.811+0000 7faae53f5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae015ce40 0x7faae00c6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52226/0 (socket says 172.21.15.46:52226) 2024-01-26T23:49:04.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.811+0000 7faae53f5700 1 -- 172.21.15.46:0/3627434987 learned_addr learned my addr 172.21.15.46:0/3627434987 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:04.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.811+0000 7faae5bf6700 1 --2- 172.21.15.46:0/3627434987 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faae00d1390 0x7faae014a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:04.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.811+0000 7faae63f7700 1 --2- 172.21.15.46:0/3627434987 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faae015d210 0x7faae009cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:04.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.811+0000 7faae53f5700 1 -- 172.21.15.46:0/3627434987 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faae00d1390 msgr2=0x7faae014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:04.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.811+0000 7faae53f5700 1 --2- 172.21.15.46:0/3627434987 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faae00d1390 0x7faae014a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.811+0000 7faae53f5700 1 -- 172.21.15.46:0/3627434987 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faae015d210 msgr2=0x7faae009cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:04.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.811+0000 7faae53f5700 1 --2- 172.21.15.46:0/3627434987 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faae015d210 0x7faae009cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:04.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.811+0000 7faae53f5700 1 -- 172.21.15.46:0/3627434987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad8009ae0 con 0x7faae015ce40 2024-01-26T23:49:04.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.811+0000 7faae53f5700 1 --2- 172.21.15.46:0/3627434987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae015ce40 0x7faae00c6250 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7faad000e900 tx=0x7faad00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:04.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.811+0000 7faad5ffb700 1 -- 172.21.15.46:0/3627434987 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad0013680 con 0x7faae015ce40 2024-01-26T23:49:04.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.811+0000 7faad5ffb700 1 -- 172.21.15.46:0/3627434987 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad00137e0 con 0x7faae015ce40 2024-01-26T23:49:04.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.811+0000 7faae7e5a700 1 -- 172.21.15.46:0/3627434987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae009a680 con 0x7faae015ce40 2024-01-26T23:49:04.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.811+0000 7faad5ffb700 1 -- 172.21.15.46:0/3627434987 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad0013ba0 con 0x7faae015ce40 2024-01-26T23:49:04.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.812+0000 7faae7e5a700 1 -- 172.21.15.46:0/3627434987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae009ac40 con 0x7faae015ce40 2024-01-26T23:49:04.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.813+0000 7faae7e5a700 1 -- 172.21.15.46:0/3627434987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae009b400 con 0x7faae015ce40 2024-01-26T23:49:04.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.813+0000 7faad5ffb700 1 -- 172.21.15.46:0/3627434987 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 62) v1 ==== 84958+0+0 (secure 0 0 0) 0x7faad00123d0 con 0x7faae015ce40 2024-01-26T23:49:04.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.814+0000 7faad5ffb700 1 --2- 172.21.15.46:0/3627434987 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7faac4067170 0x7faac4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:04.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.814+0000 7faad5ffb700 1 -- 172.21.15.46:0/3627434987 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6675+0+0 (secure 0 0 0) 0x7faad0084a60 con 0x7faae015ce40 2024-01-26T23:49:04.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.814+0000 7faae5bf6700 1 -- 172.21.15.46:0/3627434987 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7faac4067170 msgr2=0x7faac4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:04.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.814+0000 7faae5bf6700 1 --2- 172.21.15.46:0/3627434987 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7faac4067170 0x7faac4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:04.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:04.818+0000 7faad5ffb700 1 -- 172.21.15.46:0/3627434987 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faad0051670 con 0x7faae015ce40 2024-01-26T23:49:04.835 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:04.835+0000 7f30bcb891c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:49:04.892 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:04.891+0000 7f3d5745a1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:49:04.912 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:04.912+0000 7f30bcb891c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:49:04.965 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:04.964+0000 7f3d5745a1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:49:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.015+0000 7faae5bf6700 1 -- 172.21.15.46:0/3627434987 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7faac4067170 msgr2=0x7faac4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.015+0000 7faae5bf6700 1 --2- 172.21.15.46:0/3627434987 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7faac4067170 0x7faac4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:05.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.025+0000 7faae7e5a700 1 -- 172.21.15.46:0/3627434987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faae004c710 con 0x7faae015ce40 2024-01-26T23:49:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.026+0000 7faad5ffb700 1 -- 172.21.15.46:0/3627434987 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+16535 (secure 0 0 0) 0x7faad0054c90 con 0x7faae015ce40 2024-01-26T23:49:05.027 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:05.027 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":131,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:48:30.748063+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:48:09.662590+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-01-26T23:48:10.200635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/1920429547":"2024-01-27T23:48:30.748025+0000","172.21.15.106:0/674271642":"2024-01-27T23:48:30.748025+0000","172.21.15.106:6837/43388":"2024-01-27T23:48:30.748025+0000","172.21.15.106:6835/43388":"2024-01-27T23:48:30.748025+0000","172.21.15.106:0/2229853363":"2024-01-27T23:48:30.748025+0000","172.21.15.106:0/3551046672":"2024-01-27T23:48:30.748025+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:49:05.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.029+0000 7faae7e5a700 1 -- 172.21.15.46:0/3627434987 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7faac4067170 msgr2=0x7faac4069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:05.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.029+0000 7faae7e5a700 1 --2- 172.21.15.46:0/3627434987 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7faac4067170 0x7faac4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.029+0000 7faae7e5a700 1 -- 172.21.15.46:0/3627434987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae015ce40 msgr2=0x7faae00c6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.029+0000 7faae7e5a700 1 --2- 172.21.15.46:0/3627434987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae015ce40 0x7faae00c6250 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7faad000e900 tx=0x7faad00091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.029+0000 7faae7e5a700 1 -- 172.21.15.46:0/3627434987 shutdown_connections 2024-01-26T23:49:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.029+0000 7faae7e5a700 1 --2- 172.21.15.46:0/3627434987 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7faac4067170 0x7faac4069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.029+0000 7faae7e5a700 1 --2- 172.21.15.46:0/3627434987 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faae015d210 0x7faae009cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.029+0000 7faae7e5a700 1 --2- 172.21.15.46:0/3627434987 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faae00d1390 0x7faae014a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.029+0000 7faae7e5a700 1 --2- 172.21.15.46:0/3627434987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae015ce40 0x7faae00c6250 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.029+0000 7faae7e5a700 1 -- 172.21.15.46:0/3627434987 >> 172.21.15.46:0/3627434987 conn(0x7faae006f840 msgr2=0x7faae00b5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.029+0000 7faae7e5a700 1 -- 172.21.15.46:0/3627434987 shutdown_connections 2024-01-26T23:49:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.029+0000 7faae7e5a700 1 -- 172.21.15.46:0/3627434987 wait complete. 2024-01-26T23:49:05.044 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-26T23:48:09.662590+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-01-26T23:48:10.200635+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-26T23:49:05.044 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:49:05.053 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:05.053+0000 7f30bcb891c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:49:05.103 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:05.103+0000 7f3d5745a1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:49:05.138 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:05.137+0000 7f30bcb891c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:49:05.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.180+0000 7fb032f33700 1 Processor -- start 2024-01-26T23:49:05.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.180+0000 7fb032f33700 1 -- start start 2024-01-26T23:49:05.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.181+0000 7fb032f33700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb02c159da0 0x7fb02c0ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:05.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.181+0000 7fb032f33700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb02c15d8c0 0x7fb02c15dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:05.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.181+0000 7fb032f33700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb02c15a200 con 0x7fb02c1599d0 2024-01-26T23:49:05.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.181+0000 7fb032f33700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02c15e260 con 0x7fb02c15d8c0 2024-01-26T23:49:05.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.181+0000 7fb032f33700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02c0b1a60 con 0x7fb02c159da0 2024-01-26T23:49:05.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.181+0000 7fb0314d0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb02c15d8c0 0x7fb02c15dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:05.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.182+0000 7fb0314d0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb02c15d8c0 0x7fb02c15dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52234/0 (socket says 172.21.15.46:52234) 2024-01-26T23:49:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.182+0000 7fb0314d0700 1 -- 172.21.15.46:0/2612621363 learned_addr learned my addr 172.21.15.46:0/2612621363 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.182+0000 7fb02bfff700 1 --2- 172.21.15.46:0/2612621363 >> v2:172.21.15.106:3300/0 conn(0x7fb02c159da0 0x7fb02c0ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.182+0000 7fb02bfff700 1 -- 172.21.15.46:0/2612621363 >> v2:172.21.15.46:3300/0 conn(0x7fb02c15d8c0 msgr2=0x7fb02c15dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.182+0000 7fb02bfff700 1 --2- 172.21.15.46:0/2612621363 >> v2:172.21.15.46:3300/0 conn(0x7fb02c15d8c0 0x7fb02c15dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.182+0000 7fb02bfff700 1 -- 172.21.15.46:0/2612621363 >> v1:172.21.15.106:6789/0 conn(0x7fb02c1599d0 legacy=0x7fb02c0b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.182+0000 7fb02b7fe700 1 -- 172.21.15.46:0/2612621363 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710905028 0 0) 0x7fb02c15a200 con 0x7fb02c1599d0 2024-01-26T23:49:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.182+0000 7fb02bfff700 1 -- 172.21.15.46:0/2612621363 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02c0b11c0 con 0x7fb02c159da0 2024-01-26T23:49:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.182+0000 7fb0314d0700 1 --2- 172.21.15.46:0/2612621363 >> v2:172.21.15.46:3300/0 conn(0x7fb02c15d8c0 0x7fb02c15dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.182+0000 7fb02bfff700 1 --2- 172.21.15.46:0/2612621363 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c159da0 0x7fb02c0ab1c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb02c15d7f0 tx=0x7fb0200091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c371512b7b4e120a server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.182+0000 7fb02b7fe700 1 -- 172.21.15.46:0/2612621363 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb020011630 con 0x7fb02c159da0 2024-01-26T23:49:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.182+0000 7fb02b7fe700 1 -- 172.21.15.46:0/2612621363 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb020011790 con 0x7fb02c159da0 2024-01-26T23:49:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.182+0000 7fb02b7fe700 1 -- 172.21.15.46:0/2612621363 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb020011b30 con 0x7fb02c159da0 2024-01-26T23:49:05.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.183+0000 7fb032f33700 1 -- 172.21.15.46:0/2612621363 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c159da0 msgr2=0x7fb02c0ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:05.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.183+0000 7fb032f33700 1 --2- 172.21.15.46:0/2612621363 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c159da0 0x7fb02c0ab1c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb02c15d7f0 tx=0x7fb0200091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.183+0000 7fb032f33700 1 -- 172.21.15.46:0/2612621363 shutdown_connections 2024-01-26T23:49:05.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.183+0000 7fb032f33700 1 --2- 172.21.15.46:0/2612621363 >> v2:172.21.15.46:3300/0 conn(0x7fb02c15d8c0 0x7fb02c15dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.183+0000 7fb032f33700 1 --2- 172.21.15.46:0/2612621363 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c159da0 0x7fb02c0ab1c0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.183+0000 7fb032f33700 1 -- 172.21.15.46:0/2612621363 >> 172.21.15.46:0/2612621363 conn(0x7fb02c06f840 msgr2=0x7fb02c087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:05.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.183+0000 7fb032f33700 1 -- 172.21.15.46:0/2612621363 shutdown_connections 2024-01-26T23:49:05.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.183+0000 7fb032f33700 1 -- 172.21.15.46:0/2612621363 wait complete. 2024-01-26T23:49:05.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.185+0000 7fb032f33700 1 Processor -- start 2024-01-26T23:49:05.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.186+0000 7fb032f33700 1 -- start start 2024-01-26T23:49:05.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.187+0000 7fb032f33700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c1599d0 0x7fb02c16af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.187+0000 7fb032f33700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c159da0 0x7fb02c0ac200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.187+0000 7fb032f33700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c15d8c0 0x7fb02c0ac740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.187+0000 7fb032f33700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c16b480 con 0x7fb02c1599d0 2024-01-26T23:49:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.187+0000 7fb032f33700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c09d4a0 con 0x7fb02c159da0 2024-01-26T23:49:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.187+0000 7fb032f33700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c09d780 con 0x7fb02c15d8c0 2024-01-26T23:49:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.187+0000 7fb030ccf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c1599d0 0x7fb02c16af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.187+0000 7fb030ccf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c1599d0 0x7fb02c16af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52242/0 (socket says 172.21.15.46:52242) 2024-01-26T23:49:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.187+0000 7fb030ccf700 1 -- 172.21.15.46:0/1941307324 learned_addr learned my addr 172.21.15.46:0/1941307324 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.187+0000 7fb02bfff700 1 --2- 172.21.15.46:0/1941307324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c159da0 0x7fb02c0ac200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.187+0000 7fb0314d0700 1 --2- 172.21.15.46:0/1941307324 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c15d8c0 0x7fb02c0ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.187+0000 7fb030ccf700 1 -- 172.21.15.46:0/1941307324 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c15d8c0 msgr2=0x7fb02c0ac740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.187+0000 7fb030ccf700 1 --2- 172.21.15.46:0/1941307324 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c15d8c0 0x7fb02c0ac740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.188+0000 7fb030ccf700 1 -- 172.21.15.46:0/1941307324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c159da0 msgr2=0x7fb02c0ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.188+0000 7fb030ccf700 1 --2- 172.21.15.46:0/1941307324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c159da0 0x7fb02c0ac200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.188+0000 7fb030ccf700 1 -- 172.21.15.46:0/1941307324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c0094f0 con 0x7fb02c1599d0 2024-01-26T23:49:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.188+0000 7fb02bfff700 1 --2- 172.21.15.46:0/1941307324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c159da0 0x7fb02c0ac200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.188+0000 7fb030ccf700 1 --2- 172.21.15.46:0/1941307324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c1599d0 0x7fb02c16af40 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fb02400ce00 tx=0x7fb024001bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.188+0000 7fb028ff9700 1 -- 172.21.15.46:0/1941307324 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02400aa60 con 0x7fb02c1599d0 2024-01-26T23:49:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.188+0000 7fb028ff9700 1 -- 172.21.15.46:0/1941307324 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb02400abe0 con 0x7fb02c1599d0 2024-01-26T23:49:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.188+0000 7fb028ff9700 1 -- 172.21.15.46:0/1941307324 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb024012420 con 0x7fb02c1599d0 2024-01-26T23:49:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.188+0000 7fb032f33700 1 -- 172.21.15.46:0/1941307324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb020009420 con 0x7fb02c1599d0 2024-01-26T23:49:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.188+0000 7fb032f33700 1 -- 172.21.15.46:0/1941307324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb02c09dd40 con 0x7fb02c1599d0 2024-01-26T23:49:05.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.189+0000 7fb032f33700 1 -- 172.21.15.46:0/1941307324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02c04c710 con 0x7fb02c1599d0 2024-01-26T23:49:05.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.190+0000 7fb028ff9700 1 -- 172.21.15.46:0/1941307324 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 62) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fb0240127d0 con 0x7fb02c1599d0 2024-01-26T23:49:05.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.191+0000 7fb028ff9700 1 --2- 172.21.15.46:0/1941307324 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb018067170 0x7fb018069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:05.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.191+0000 7fb02bfff700 1 -- 172.21.15.46:0/1941307324 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb018067170 msgr2=0x7fb018069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:05.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.191+0000 7fb028ff9700 1 -- 172.21.15.46:0/1941307324 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fb024085380 con 0x7fb02c1599d0 2024-01-26T23:49:05.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.191+0000 7fb02bfff700 1 --2- 172.21.15.46:0/1941307324 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb018067170 0x7fb018069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.195+0000 7fb028ff9700 1 -- 172.21.15.46:0/1941307324 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb024051f90 con 0x7fb02c1599d0 2024-01-26T23:49:05.232 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:05.231+0000 7f3d5745a1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:49:05.316 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:05.316+0000 7f30bcb891c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:49:05.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.392+0000 7fb02bfff700 1 -- 172.21.15.46:0/1941307324 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb018067170 msgr2=0x7fb018069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:05.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.392+0000 7fb02bfff700 1 --2- 172.21.15.46:0/1941307324 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb018067170 0x7fb018069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.444+0000 7fb032f33700 1 -- 172.21.15.46:0/1941307324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb02c0d0560 con 0x7fb02c1599d0 2024-01-26T23:49:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.445+0000 7fb028ff9700 1 -- 172.21.15.46:0/1941307324 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v131) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb0240555b0 con 0x7fb02c1599d0 2024-01-26T23:49:05.445 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:49:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.447+0000 7fb032f33700 1 -- 172.21.15.46:0/1941307324 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb018067170 msgr2=0x7fb018069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.447+0000 7fb032f33700 1 --2- 172.21.15.46:0/1941307324 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb018067170 0x7fb018069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.447+0000 7fb032f33700 1 -- 172.21.15.46:0/1941307324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c1599d0 msgr2=0x7fb02c16af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.447+0000 7fb032f33700 1 --2- 172.21.15.46:0/1941307324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c1599d0 0x7fb02c16af40 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fb02400ce00 tx=0x7fb024001bd0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.447+0000 7fb032f33700 1 -- 172.21.15.46:0/1941307324 shutdown_connections 2024-01-26T23:49:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.447+0000 7fb032f33700 1 --2- 172.21.15.46:0/1941307324 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb018067170 0x7fb018069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.447+0000 7fb032f33700 1 --2- 172.21.15.46:0/1941307324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c1599d0 0x7fb02c16af40 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.447+0000 7fb032f33700 1 --2- 172.21.15.46:0/1941307324 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c15d8c0 0x7fb02c0ac740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.447+0000 7fb032f33700 1 --2- 172.21.15.46:0/1941307324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c159da0 0x7fb02c0ac200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.447+0000 7fb032f33700 1 -- 172.21.15.46:0/1941307324 >> 172.21.15.46:0/1941307324 conn(0x7fb02c06f840 msgr2=0x7fb02c0cfd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:05.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.447+0000 7fb032f33700 1 -- 172.21.15.46:0/1941307324 shutdown_connections 2024-01-26T23:49:05.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.448+0000 7fb032f33700 1 -- 172.21.15.46:0/1941307324 wait complete. 2024-01-26T23:49:05.458 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:05.458+0000 7f30bcb891c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:49:05.461 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-26T23:49:05.547 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:05.546+0000 7f3d5745a1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:49:05.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.638+0000 7fa3eb7ca700 1 Processor -- start 2024-01-26T23:49:05.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.638+0000 7fa3eb7ca700 1 -- start start 2024-01-26T23:49:05.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.639+0000 7fa3eb7ca700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa3e4159cb0 0x7fa3e40cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:05.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.639+0000 7fa3eb7ca700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3e40cf050 0x7fa3e40af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:05.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.639+0000 7fa3eb7ca700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3e40cf4a0 0x7fa3e40af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:05.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.639+0000 7fa3eb7ca700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e40cc100 con 0x7fa3e40cf4a0 2024-01-26T23:49:05.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.639+0000 7fa3eb7ca700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e40cb850 con 0x7fa3e4159cb0 2024-01-26T23:49:05.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.639+0000 7fa3eb7ca700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e40cbaf0 con 0x7fa3e40cf050 2024-01-26T23:49:05.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.639+0000 7fa3e9d67700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3e40cf4a0 0x7fa3e40af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:05.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.639+0000 7fa3e9d67700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3e40cf4a0 0x7fa3e40af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52258/0 (socket says 172.21.15.46:52258) 2024-01-26T23:49:05.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.639+0000 7fa3e9d67700 1 -- 172.21.15.46:0/1266137576 learned_addr learned my addr 172.21.15.46:0/1266137576 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:05.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.639+0000 7fa3e9566700 1 --2- 172.21.15.46:0/1266137576 >> v2:172.21.15.106:3300/0 conn(0x7fa3e4159cb0 0x7fa3e40cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:05.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.639+0000 7fa3e8d65700 1 --2- 172.21.15.46:0/1266137576 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3e40cf050 0x7fa3e40af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:05.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.640+0000 7fa3e9d67700 1 -- 172.21.15.46:0/1266137576 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3e40cf050 msgr2=0x7fa3e40af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:05.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.640+0000 7fa3e9d67700 1 --2- 172.21.15.46:0/1266137576 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3e40cf050 0x7fa3e40af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.640+0000 7fa3e9d67700 1 -- 172.21.15.46:0/1266137576 >> v2:172.21.15.106:3300/0 conn(0x7fa3e4159cb0 msgr2=0x7fa3e40cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:05.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.640+0000 7fa3e9d67700 1 --2- 172.21.15.46:0/1266137576 >> v2:172.21.15.106:3300/0 conn(0x7fa3e4159cb0 0x7fa3e40cc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.640+0000 7fa3e9d67700 1 -- 172.21.15.46:0/1266137576 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e40cfc80 con 0x7fa3e40cf4a0 2024-01-26T23:49:05.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.640+0000 7fa3e9566700 1 --2- 172.21.15.46:0/1266137576 >> v2:172.21.15.106:3300/0 conn(0x7fa3e4159cb0 0x7fa3e40cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:05.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.640+0000 7fa3e8d65700 1 --2- 172.21.15.46:0/1266137576 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3e40cf050 0x7fa3e40af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:05.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.640+0000 7fa3e9d67700 1 --2- 172.21.15.46:0/1266137576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e40cf4a0 0x7fa3e40af970 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40b6880 tx=0x7fa3dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d16a4a7fecdf00d server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:05.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.640+0000 7fa3dbfff700 1 -- 172.21.15.46:0/1266137576 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc00b640 con 0x7fa3e40cf4a0 2024-01-26T23:49:05.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.640+0000 7fa3dbfff700 1 -- 172.21.15.46:0/1266137576 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3dc00b7a0 con 0x7fa3e40cf4a0 2024-01-26T23:49:05.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.640+0000 7fa3dbfff700 1 -- 172.21.15.46:0/1266137576 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc00bb60 con 0x7fa3e40cf4a0 2024-01-26T23:49:05.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.641+0000 7fa3eb7ca700 1 -- 172.21.15.46:0/1266137576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e40cf4a0 msgr2=0x7fa3e40af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:05.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.641+0000 7fa3eb7ca700 1 --2- 172.21.15.46:0/1266137576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e40cf4a0 0x7fa3e40af970 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40b6880 tx=0x7fa3dc009250 comp rx=0 tx=0).stop 2024-01-26T23:49:05.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.641+0000 7fa3eb7ca700 1 -- 172.21.15.46:0/1266137576 shutdown_connections 2024-01-26T23:49:05.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.641+0000 7fa3eb7ca700 1 --2- 172.21.15.46:0/1266137576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e40cf4a0 0x7fa3e40af970 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.641+0000 7fa3eb7ca700 1 --2- 172.21.15.46:0/1266137576 >> v2:172.21.15.106:3300/0 conn(0x7fa3e4159cb0 0x7fa3e40cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.641+0000 7fa3eb7ca700 1 --2- 172.21.15.46:0/1266137576 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3e40cf050 0x7fa3e40af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.641+0000 7fa3eb7ca700 1 -- 172.21.15.46:0/1266137576 >> 172.21.15.46:0/1266137576 conn(0x7fa3e4087f60 msgr2=0x7fa3e406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:05.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.641+0000 7fa3eb7ca700 1 -- 172.21.15.46:0/1266137576 shutdown_connections 2024-01-26T23:49:05.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.641+0000 7fa3eb7ca700 1 -- 172.21.15.46:0/1266137576 wait complete. 2024-01-26T23:49:05.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.642+0000 7fa3eb7ca700 1 Processor -- start 2024-01-26T23:49:05.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.642+0000 7fa3eb7ca700 1 -- start start 2024-01-26T23:49:05.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.643+0000 7fa3eb7ca700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3e40cf050 0x7fa3e41535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:05.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.643+0000 7fa3eb7ca700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3e40cf4a0 0x7fa3e40c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:05.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.643+0000 7fa3eb7ca700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e4159cb0 0x7fa3e40c8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:05.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.643+0000 7fa3eb7ca700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e4153b30 con 0x7fa3e4159cb0 2024-01-26T23:49:05.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.643+0000 7fa3eb7ca700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e40b08b0 con 0x7fa3e40cf050 2024-01-26T23:49:05.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.643+0000 7fa3eb7ca700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e40b63f0 con 0x7fa3e40cf4a0 2024-01-26T23:49:05.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.643+0000 7fa3e9d67700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e4159cb0 0x7fa3e40c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:05.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.643+0000 7fa3e9566700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3e40cf050 0x7fa3e41535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:05.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.643+0000 7fa3e9d67700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e4159cb0 0x7fa3e40c8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52272/0 (socket says 172.21.15.46:52272) 2024-01-26T23:49:05.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.643+0000 7fa3e9d67700 1 -- 172.21.15.46:0/2182902553 learned_addr learned my addr 172.21.15.46:0/2182902553 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:05.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.643+0000 7fa3e8d65700 1 --2- 172.21.15.46:0/2182902553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3e40cf4a0 0x7fa3e40c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:05.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.644+0000 7fa3e9d67700 1 -- 172.21.15.46:0/2182902553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3e40cf4a0 msgr2=0x7fa3e40c7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:05.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.644+0000 7fa3e9d67700 1 --2- 172.21.15.46:0/2182902553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3e40cf4a0 0x7fa3e40c7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.644+0000 7fa3e9d67700 1 -- 172.21.15.46:0/2182902553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3e40cf050 msgr2=0x7fa3e41535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:05.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.644+0000 7fa3e9d67700 1 --2- 172.21.15.46:0/2182902553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3e40cf050 0x7fa3e41535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.644+0000 7fa3e9d67700 1 -- 172.21.15.46:0/2182902553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d4009420 con 0x7fa3e4159cb0 2024-01-26T23:49:05.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.644+0000 7fa3e9566700 1 --2- 172.21.15.46:0/2182902553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3e40cf050 0x7fa3e41535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:05.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.644+0000 7fa3e9d67700 1 --2- 172.21.15.46:0/2182902553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e4159cb0 0x7fa3e40c8000 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc0094c0 tx=0x7fa3dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:05.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.644+0000 7fa3d97fa700 1 -- 172.21.15.46:0/2182902553 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc00bb20 con 0x7fa3e4159cb0 2024-01-26T23:49:05.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.644+0000 7fa3d97fa700 1 -- 172.21.15.46:0/2182902553 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3dc011420 con 0x7fa3e4159cb0 2024-01-26T23:49:05.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.644+0000 7fa3eb7ca700 1 -- 172.21.15.46:0/2182902553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e0009a20 con 0x7fa3e4159cb0 2024-01-26T23:49:05.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.644+0000 7fa3d97fa700 1 -- 172.21.15.46:0/2182902553 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc0117e0 con 0x7fa3e4159cb0 2024-01-26T23:49:05.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.644+0000 7fa3eb7ca700 1 -- 172.21.15.46:0/2182902553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3dc0094f0 con 0x7fa3e4159cb0 2024-01-26T23:49:05.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.645+0000 7fa3ceffd700 1 -- 172.21.15.46:0/2182902553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3bc0052b0 con 0x7fa3e4159cb0 2024-01-26T23:49:05.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.646+0000 7fa3d97fa700 1 -- 172.21.15.46:0/2182902553 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 62) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fa3dc011b50 con 0x7fa3e4159cb0 2024-01-26T23:49:05.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.646+0000 7fa3d97fa700 1 --2- 172.21.15.46:0/2182902553 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa3c8067170 0x7fa3c8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:05.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.646+0000 7fa3e9566700 1 -- 172.21.15.46:0/2182902553 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa3c8067170 msgr2=0x7fa3c8069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:05.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.646+0000 7fa3d97fa700 1 -- 172.21.15.46:0/2182902553 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fa3dc086530 con 0x7fa3e4159cb0 2024-01-26T23:49:05.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.646+0000 7fa3e9566700 1 --2- 172.21.15.46:0/2182902553 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa3c8067170 0x7fa3c8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.655+0000 7fa3d97fa700 1 -- 172.21.15.46:0/2182902553 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa3dc053140 con 0x7fa3e4159cb0 2024-01-26T23:49:05.759 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:05.758+0000 7f3d5745a1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:49:05.834 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:05.833+0000 7f30bcb891c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:49:05.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.847+0000 7fa3e9566700 1 -- 172.21.15.46:0/2182902553 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa3c8067170 msgr2=0x7fa3c8069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:05.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.847+0000 7fa3e9566700 1 --2- 172.21.15.46:0/2182902553 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa3c8067170 0x7fa3c8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:05.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.869+0000 7fa3ceffd700 1 -- 172.21.15.46:0/2182902553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa3bc005ae0 con 0x7fa3e4159cb0 2024-01-26T23:49:05.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.870+0000 7fa3d97fa700 1 -- 172.21.15.46:0/2182902553 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v132) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa3dc056760 con 0x7fa3e4159cb0 2024-01-26T23:49:05.870 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:49:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.872+0000 7fa3ceffd700 1 -- 172.21.15.46:0/2182902553 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa3c8067170 msgr2=0x7fa3c8069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.872+0000 7fa3ceffd700 1 --2- 172.21.15.46:0/2182902553 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa3c8067170 0x7fa3c8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.872+0000 7fa3ceffd700 1 -- 172.21.15.46:0/2182902553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e4159cb0 msgr2=0x7fa3e40c8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.872+0000 7fa3ceffd700 1 --2- 172.21.15.46:0/2182902553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e4159cb0 0x7fa3e40c8000 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc0094c0 tx=0x7fa3dc017000 comp rx=0 tx=0).stop 2024-01-26T23:49:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.873+0000 7fa3ceffd700 1 -- 172.21.15.46:0/2182902553 shutdown_connections 2024-01-26T23:49:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.873+0000 7fa3ceffd700 1 --2- 172.21.15.46:0/2182902553 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa3c8067170 0x7fa3c8069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.873+0000 7fa3ceffd700 1 --2- 172.21.15.46:0/2182902553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3e4159cb0 0x7fa3e40c8000 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.873+0000 7fa3ceffd700 1 --2- 172.21.15.46:0/2182902553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3e40cf050 0x7fa3e41535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.873+0000 7fa3ceffd700 1 --2- 172.21.15.46:0/2182902553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3e40cf4a0 0x7fa3e40c7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.873+0000 7fa3ceffd700 1 -- 172.21.15.46:0/2182902553 >> 172.21.15.46:0/2182902553 conn(0x7fa3e4087f60 msgr2=0x7fa3e41554c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.873+0000 7fa3ceffd700 1 -- 172.21.15.46:0/2182902553 shutdown_connections 2024-01-26T23:49:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:05.873+0000 7fa3ceffd700 1 -- 172.21.15.46:0/2182902553 wait complete. 2024-01-26T23:49:05.887 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-26T23:49:05.992 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:05.991+0000 7f3d5745a1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:49:06.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.028+0000 7f020b950700 1 Processor -- start 2024-01-26T23:49:06.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.028+0000 7f020b950700 1 -- start start 2024-01-26T23:49:06.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.029+0000 7f020b950700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02041599d0 0x7f0204159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.029+0000 7f020b950700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f02040ab1c0 0x7f02040b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.029+0000 7f020b950700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f020415d8c0 0x7f020415dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.029+0000 7f020b950700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f020415e260 con 0x7f020415d8c0 2024-01-26T23:49:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.029+0000 7f020b950700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02040b1a60 con 0x7f02040ab1c0 2024-01-26T23:49:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.029+0000 7f020b950700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02040ab5f0 con 0x7f02041599d0 2024-01-26T23:49:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.029+0000 7f0209eed700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f020415d8c0 0x7f020415dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.029+0000 7f0209eed700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f020415d8c0 0x7f020415dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52276/0 (socket says 172.21.15.46:52276) 2024-01-26T23:49:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.029+0000 7f02096ec700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02041599d0 0x7f0204159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.029+0000 7f0209eed700 1 -- 172.21.15.46:0/1711786606 learned_addr learned my addr 172.21.15.46:0/1711786606 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.029+0000 7f0208eeb700 1 --2- 172.21.15.46:0/1711786606 >> v2:172.21.15.106:3300/0 conn(0x7f02040ab1c0 0x7f02040b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.030+0000 7f0209eed700 1 -- 172.21.15.46:0/1711786606 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02041599d0 msgr2=0x7f0204159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.030+0000 7f0209eed700 1 --2- 172.21.15.46:0/1711786606 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02041599d0 0x7f0204159da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.030+0000 7f0209eed700 1 -- 172.21.15.46:0/1711786606 >> v2:172.21.15.106:3300/0 conn(0x7f02040ab1c0 msgr2=0x7f02040b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.030+0000 7f0209eed700 1 --2- 172.21.15.46:0/1711786606 >> v2:172.21.15.106:3300/0 conn(0x7f02040ab1c0 0x7f02040b0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.030+0000 7f0209eed700 1 -- 172.21.15.46:0/1711786606 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02040b0f40 con 0x7f020415d8c0 2024-01-26T23:49:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.030+0000 7f0209eed700 1 --2- 172.21.15.46:0/1711786606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f020415d8c0 0x7f020415dd20 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f0204067e70 tx=0x7f01fc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64f9485240128d85 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.030+0000 7f0203fff700 1 -- 172.21.15.46:0/1711786606 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc00e070 con 0x7f020415d8c0 2024-01-26T23:49:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.030+0000 7f0203fff700 1 -- 172.21.15.46:0/1711786606 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01fc00b970 con 0x7f020415d8c0 2024-01-26T23:49:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.030+0000 7f0203fff700 1 -- 172.21.15.46:0/1711786606 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc0092e0 con 0x7f020415d8c0 2024-01-26T23:49:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.031+0000 7f020b950700 1 -- 172.21.15.46:0/1711786606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f020415d8c0 msgr2=0x7f020415dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.031+0000 7f020b950700 1 --2- 172.21.15.46:0/1711786606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f020415d8c0 0x7f020415dd20 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f0204067e70 tx=0x7f01fc009840 comp rx=0 tx=0).stop 2024-01-26T23:49:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.031+0000 7f020b950700 1 -- 172.21.15.46:0/1711786606 shutdown_connections 2024-01-26T23:49:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.031+0000 7f020b950700 1 --2- 172.21.15.46:0/1711786606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f020415d8c0 0x7f020415dd20 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.031+0000 7f020b950700 1 --2- 172.21.15.46:0/1711786606 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02041599d0 0x7f0204159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.031+0000 7f020b950700 1 --2- 172.21.15.46:0/1711786606 >> v2:172.21.15.106:3300/0 conn(0x7f02040ab1c0 0x7f02040b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.031+0000 7f020b950700 1 -- 172.21.15.46:0/1711786606 >> 172.21.15.46:0/1711786606 conn(0x7f020406f840 msgr2=0x7f0204087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.031+0000 7f020b950700 1 -- 172.21.15.46:0/1711786606 shutdown_connections 2024-01-26T23:49:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.031+0000 7f020b950700 1 -- 172.21.15.46:0/1711786606 wait complete. 2024-01-26T23:49:06.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.033+0000 7f020b950700 1 Processor -- start 2024-01-26T23:49:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.033+0000 7f020b950700 1 -- start start 2024-01-26T23:49:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.034+0000 7f020b950700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02040ab1c0 0x7f020409da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.034+0000 7f020b950700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02041599d0 0x7f020414cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.034+0000 7f020b950700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f020415d8c0 0x7f02040ad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.034+0000 7f020b950700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020409dfb0 con 0x7f02040ab1c0 2024-01-26T23:49:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.034+0000 7f020b950700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020409d5e0 con 0x7f020415d8c0 2024-01-26T23:49:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.034+0000 7f020b950700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020409c9d0 con 0x7f02041599d0 2024-01-26T23:49:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.034+0000 7f02096ec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02040ab1c0 0x7f020409da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.034+0000 7f02096ec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02040ab1c0 0x7f020409da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52292/0 (socket says 172.21.15.46:52292) 2024-01-26T23:49:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.034+0000 7f02096ec700 1 -- 172.21.15.46:0/906539759 learned_addr learned my addr 172.21.15.46:0/906539759 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.034+0000 7f0208eeb700 1 --2- 172.21.15.46:0/906539759 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02041599d0 0x7f020414cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.034+0000 7f0209eed700 1 --2- 172.21.15.46:0/906539759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f020415d8c0 0x7f02040ad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.034+0000 7f02096ec700 1 -- 172.21.15.46:0/906539759 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02041599d0 msgr2=0x7f020414cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.034+0000 7f02096ec700 1 --2- 172.21.15.46:0/906539759 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02041599d0 0x7f020414cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.034+0000 7f02096ec700 1 -- 172.21.15.46:0/906539759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f020415d8c0 msgr2=0x7f02040ad0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.034+0000 7f02096ec700 1 --2- 172.21.15.46:0/906539759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f020415d8c0 0x7f02040ad0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.034+0000 7f02096ec700 1 -- 172.21.15.46:0/906539759 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01fc009ae0 con 0x7f02040ab1c0 2024-01-26T23:49:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.035+0000 7f02096ec700 1 --2- 172.21.15.46:0/906539759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02040ab1c0 0x7f020409da70 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f01f400ec80 tx=0x7f01f4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.035+0000 7f02017fa700 1 -- 172.21.15.46:0/906539759 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f400c8e0 con 0x7f02040ab1c0 2024-01-26T23:49:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.035+0000 7f02017fa700 1 -- 172.21.15.46:0/906539759 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f400ca40 con 0x7f02040ab1c0 2024-01-26T23:49:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.035+0000 7f020b950700 1 -- 172.21.15.46:0/906539759 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f020409ccb0 con 0x7f02040ab1c0 2024-01-26T23:49:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.035+0000 7f02017fa700 1 -- 172.21.15.46:0/906539759 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f400cdc0 con 0x7f02040ab1c0 2024-01-26T23:49:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.035+0000 7f020b950700 1 -- 172.21.15.46:0/906539759 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f020409d270 con 0x7f02040ab1c0 2024-01-26T23:49:06.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.036+0000 7f02017fa700 1 -- 172.21.15.46:0/906539759 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 62) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f01f4013680 con 0x7f02040ab1c0 2024-01-26T23:49:06.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.036+0000 7f020b950700 1 -- 172.21.15.46:0/906539759 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f020409d8c0 con 0x7f02040ab1c0 2024-01-26T23:49:06.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.037+0000 7f02017fa700 1 --2- 172.21.15.46:0/906539759 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f01f0067170 0x7f01f0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.037+0000 7f0208eeb700 1 -- 172.21.15.46:0/906539759 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f01f0067170 msgr2=0x7f01f0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:06.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.037+0000 7f02017fa700 1 -- 172.21.15.46:0/906539759 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6731+0+0 (secure 0 0 0) 0x7f01f4085270 con 0x7f02040ab1c0 2024-01-26T23:49:06.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.037+0000 7f0208eeb700 1 --2- 172.21.15.46:0/906539759 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f01f0067170 0x7f01f0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:06.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.042+0000 7f02017fa700 1 -- 172.21.15.46:0/906539759 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f01f4051e50 con 0x7f02040ab1c0 2024-01-26T23:49:06.063 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:06.062+0000 7f3d5745a1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:49:06.091 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:06.091+0000 7f30bcb891c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:49:06.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.238+0000 7f0208eeb700 1 -- 172.21.15.46:0/906539759 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f01f0067170 msgr2=0x7f01f0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:06.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.238+0000 7f0208eeb700 1 --2- 172.21.15.46:0/906539759 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f01f0067170 0x7f01f0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:06.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.248+0000 7f020b950700 1 -- 172.21.15.46:0/906539759 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f020404c710 con 0x7f02040ab1c0 2024-01-26T23:49:06.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.249+0000 7f02017fa700 1 -- 172.21.15.46:0/906539759 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v132) v1 ==== 100+0+11 (secure 0 0 0) 0x7f01f4055470 con 0x7f02040ab1c0 2024-01-26T23:49:06.249 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-26T23:49:06.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.251+0000 7f020b950700 1 -- 172.21.15.46:0/906539759 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f01f0067170 msgr2=0x7f01f0069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:06.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.251+0000 7f020b950700 1 --2- 172.21.15.46:0/906539759 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f01f0067170 0x7f01f0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.251+0000 7f020b950700 1 -- 172.21.15.46:0/906539759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02040ab1c0 msgr2=0x7f020409da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.251+0000 7f020b950700 1 --2- 172.21.15.46:0/906539759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02040ab1c0 0x7f020409da70 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f01f400ec80 tx=0x7f01f4009120 comp rx=0 tx=0).stop 2024-01-26T23:49:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.251+0000 7f020b950700 1 -- 172.21.15.46:0/906539759 shutdown_connections 2024-01-26T23:49:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.251+0000 7f020b950700 1 --2- 172.21.15.46:0/906539759 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f01f0067170 0x7f01f0069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.251+0000 7f020b950700 1 --2- 172.21.15.46:0/906539759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02040ab1c0 0x7f020409da70 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.251+0000 7f020b950700 1 --2- 172.21.15.46:0/906539759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f020415d8c0 0x7f02040ad0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.251+0000 7f020b950700 1 --2- 172.21.15.46:0/906539759 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02041599d0 0x7f020414cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.251+0000 7f020b950700 1 -- 172.21.15.46:0/906539759 >> 172.21.15.46:0/906539759 conn(0x7f020406f840 msgr2=0x7f0204087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.251+0000 7f020b950700 1 -- 172.21.15.46:0/906539759 shutdown_connections 2024-01-26T23:49:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.251+0000 7f020b950700 1 -- 172.21.15.46:0/906539759 wait complete. 2024-01-26T23:49:06.266 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:49:06.355 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:06.355+0000 7f30bcb891c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:49:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.398+0000 7f57710fa700 1 Processor -- start 2024-01-26T23:49:06.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.399+0000 7f57710fa700 1 -- start start 2024-01-26T23:49:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.400+0000 7f57710fa700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f576c159960 0x7f576c06bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.400+0000 7f57710fa700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f576c06c1f0 0x7f576c069370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.400+0000 7f57710fa700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576c06c600 con 0x7f576c159590 2024-01-26T23:49:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.400+0000 7f57710fa700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f576c15e350 con 0x7f576c159960 2024-01-26T23:49:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.400+0000 7f57710fa700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f576c0c97c0 con 0x7f576c06c1f0 2024-01-26T23:49:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.400+0000 7f576a59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f576c159960 0x7f576c06bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.400+0000 7f576a59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f576c159960 0x7f576c06bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52296/0 (socket says 172.21.15.46:52296) 2024-01-26T23:49:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.400+0000 7f576a59c700 1 -- 172.21.15.46:0/3356736453 learned_addr learned my addr 172.21.15.46:0/3356736453 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.400+0000 7f576b59e700 1 --2- 172.21.15.46:0/3356736453 >> v2:172.21.15.106:3300/0 conn(0x7f576c06c1f0 0x7f576c069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.401+0000 7f576a59c700 1 -- 172.21.15.46:0/3356736453 >> v2:172.21.15.106:3300/0 conn(0x7f576c06c1f0 msgr2=0x7f576c069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.401+0000 7f576a59c700 1 --2- 172.21.15.46:0/3356736453 >> v2:172.21.15.106:3300/0 conn(0x7f576c06c1f0 0x7f576c069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.401+0000 7f576a59c700 1 -- 172.21.15.46:0/3356736453 >> v1:172.21.15.106:6789/0 conn(0x7f576c159590 legacy=0x7f576c15d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.401+0000 7f576a59c700 1 -- 172.21.15.46:0/3356736453 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c15dac0 con 0x7f576c159960 2024-01-26T23:49:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.401+0000 7f576a59c700 1 --2- 172.21.15.46:0/3356736453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f576c159960 0x7f576c06bcb0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f576c0672b0 tx=0x7f5754009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e7760a06530b43 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.401+0000 7f5769d9b700 1 -- 172.21.15.46:0/3356736453 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5754009ea0 con 0x7f576c159960 2024-01-26T23:49:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.401+0000 7f5769d9b700 1 -- 172.21.15.46:0/3356736453 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575400b820 con 0x7f576c159960 2024-01-26T23:49:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.401+0000 7f5769d9b700 1 -- 172.21.15.46:0/3356736453 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575400bbe0 con 0x7f576c159960 2024-01-26T23:49:06.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.401+0000 7f57710fa700 1 -- 172.21.15.46:0/3356736453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f576c159960 msgr2=0x7f576c06bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.401+0000 7f57710fa700 1 --2- 172.21.15.46:0/3356736453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f576c159960 0x7f576c06bcb0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f576c0672b0 tx=0x7f5754009250 comp rx=0 tx=0).stop 2024-01-26T23:49:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.402+0000 7f57710fa700 1 -- 172.21.15.46:0/3356736453 shutdown_connections 2024-01-26T23:49:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.402+0000 7f57710fa700 1 --2- 172.21.15.46:0/3356736453 >> v2:172.21.15.106:3300/0 conn(0x7f576c06c1f0 0x7f576c069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.402+0000 7f57710fa700 1 --2- 172.21.15.46:0/3356736453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f576c159960 0x7f576c06bcb0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.402+0000 7f57710fa700 1 -- 172.21.15.46:0/3356736453 >> 172.21.15.46:0/3356736453 conn(0x7f576c0ce530 msgr2=0x7f576c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.402+0000 7f57710fa700 1 -- 172.21.15.46:0/3356736453 shutdown_connections 2024-01-26T23:49:06.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.402+0000 7f57710fa700 1 -- 172.21.15.46:0/3356736453 wait complete. 2024-01-26T23:49:06.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.403+0000 7f57710fa700 1 Processor -- start 2024-01-26T23:49:06.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.403+0000 7f57710fa700 1 -- start start 2024-01-26T23:49:06.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.404+0000 7f57710fa700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f576c06c1f0 0x7f576c155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.404+0000 7f57710fa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f576c159590 0x7f576c0c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.404+0000 7f57710fa700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f576c159960 0x7f576c0c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.404+0000 7f57710fa700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c0c4b20 con 0x7f576c159590 2024-01-26T23:49:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.404+0000 7f57710fa700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c0c3a80 con 0x7f576c06c1f0 2024-01-26T23:49:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.404+0000 7f57710fa700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576c0c3d60 con 0x7f576c159960 2024-01-26T23:49:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.404+0000 7f576a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f576c159590 0x7f576c0c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.404+0000 7f576a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f576c159590 0x7f576c0c76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52300/0 (socket says 172.21.15.46:52300) 2024-01-26T23:49:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.404+0000 7f576a59c700 1 -- 172.21.15.46:0/120609452 learned_addr learned my addr 172.21.15.46:0/120609452 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.404+0000 7f576ad9d700 1 --2- 172.21.15.46:0/120609452 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f576c06c1f0 0x7f576c155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.404+0000 7f576b59e700 1 --2- 172.21.15.46:0/120609452 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f576c159960 0x7f576c0c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.404+0000 7f576a59c700 1 -- 172.21.15.46:0/120609452 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f576c159960 msgr2=0x7f576c0c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.404+0000 7f576a59c700 1 --2- 172.21.15.46:0/120609452 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f576c159960 0x7f576c0c7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.404+0000 7f576a59c700 1 -- 172.21.15.46:0/120609452 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f576c06c1f0 msgr2=0x7f576c155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:06.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.404+0000 7f576a59c700 1 --2- 172.21.15.46:0/120609452 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f576c06c1f0 0x7f576c155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.404+0000 7f576a59c700 1 -- 172.21.15.46:0/120609452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57540094f0 con 0x7f576c159590 2024-01-26T23:49:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.405+0000 7f576a59c700 1 --2- 172.21.15.46:0/120609452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f576c159590 0x7f576c0c76c0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f576c06f390 tx=0x7f5754017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.405+0000 7f5762ffd700 1 -- 172.21.15.46:0/120609452 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575401f070 con 0x7f576c159590 2024-01-26T23:49:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.405+0000 7f5762ffd700 1 -- 172.21.15.46:0/120609452 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5754002f30 con 0x7f576c159590 2024-01-26T23:49:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.405+0000 7f57710fa700 1 -- 172.21.15.46:0/120609452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f576c0c4040 con 0x7f576c159590 2024-01-26T23:49:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.405+0000 7f5762ffd700 1 -- 172.21.15.46:0/120609452 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575400b990 con 0x7f576c159590 2024-01-26T23:49:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.405+0000 7f57710fa700 1 -- 172.21.15.46:0/120609452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f576c0c4600 con 0x7f576c159590 2024-01-26T23:49:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.406+0000 7f57710fa700 1 -- 172.21.15.46:0/120609452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576c0c47f0 con 0x7f576c159590 2024-01-26T23:49:06.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.406+0000 7f5762ffd700 1 -- 172.21.15.46:0/120609452 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 62) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f5754023070 con 0x7f576c159590 2024-01-26T23:49:06.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.407+0000 7f5762ffd700 1 --2- 172.21.15.46:0/120609452 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f574c067170 0x7f574c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.407+0000 7f5762ffd700 1 -- 172.21.15.46:0/120609452 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6731+0+0 (secure 0 0 0) 0x7f5754085ed0 con 0x7f576c159590 2024-01-26T23:49:06.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.407+0000 7f576ad9d700 1 -- 172.21.15.46:0/120609452 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f574c067170 msgr2=0x7f574c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:06.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.407+0000 7f576ad9d700 1 --2- 172.21.15.46:0/120609452 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f574c067170 0x7f574c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:06.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.411+0000 7f5762ffd700 1 -- 172.21.15.46:0/120609452 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5754052a20 con 0x7f576c159590 2024-01-26T23:49:06.457 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:06.456+0000 7f30bcb891c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:49:06.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.608+0000 7f576ad9d700 1 -- 172.21.15.46:0/120609452 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f574c067170 msgr2=0x7f574c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:06.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.608+0000 7f576ad9d700 1 --2- 172.21.15.46:0/120609452 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f574c067170 0x7f574c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:06.666 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:06.665+0000 7f3d5745a1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:49:06.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.700+0000 7f57710fa700 1 -- 172.21.15.46:0/120609452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f576c04c710 con 0x7f576c159590 2024-01-26T23:49:06.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.701+0000 7f5762ffd700 1 -- 172.21.15.46:0/120609452 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f5754015020 con 0x7f576c159590 2024-01-26T23:49:06.702 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:06.702 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":62,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6923,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/4123353304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4123353304},{"type":"v1","addr":"172.21.15.106:6834","nonce":4123353304}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/690662483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":690662483},{"type":"v1","addr":"172.21.15.106:6836","nonce":690662483}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7791,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3035969379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3035969379},{"type":"v1","addr":"172.21.15.46:6834","nonce":3035969379}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:48:11.234632+0000","modified":"2024-01-26T23:48:14.815720+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7794},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7794":{"gid":7794,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/783623731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":783623731},{"type":"v1","addr":"172.21.15.46:6835","nonce":783623731}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2024-01-26T23:49:06.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.703+0000 7f57710fa700 1 -- 172.21.15.46:0/120609452 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f574c067170 msgr2=0x7f574c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:06.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.703+0000 7f57710fa700 1 --2- 172.21.15.46:0/120609452 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f574c067170 0x7f574c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.703+0000 7f57710fa700 1 -- 172.21.15.46:0/120609452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f576c159590 msgr2=0x7f576c0c76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:06.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.703+0000 7f57710fa700 1 --2- 172.21.15.46:0/120609452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f576c159590 0x7f576c0c76c0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f576c06f390 tx=0x7f5754017000 comp rx=0 tx=0).stop 2024-01-26T23:49:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.703+0000 7f57710fa700 1 -- 172.21.15.46:0/120609452 shutdown_connections 2024-01-26T23:49:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.703+0000 7f57710fa700 1 --2- 172.21.15.46:0/120609452 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f574c067170 0x7f574c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.703+0000 7f57710fa700 1 --2- 172.21.15.46:0/120609452 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f576c159960 0x7f576c0c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.703+0000 7f57710fa700 1 --2- 172.21.15.46:0/120609452 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f576c06c1f0 0x7f576c155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.703+0000 7f57710fa700 1 --2- 172.21.15.46:0/120609452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f576c159590 0x7f576c0c76c0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.703+0000 7f57710fa700 1 -- 172.21.15.46:0/120609452 >> 172.21.15.46:0/120609452 conn(0x7f576c0ce530 msgr2=0x7f576c0b3990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.704+0000 7f57710fa700 1 -- 172.21.15.46:0/120609452 shutdown_connections 2024-01-26T23:49:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.704+0000 7f57710fa700 1 -- 172.21.15.46:0/120609452 wait complete. 2024-01-26T23:49:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 62 2024-01-26T23:49:06.718 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:49:06.741 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:06.740+0000 7f3d5745a1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:49:06.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.851+0000 7f3dd8754700 1 Processor -- start 2024-01-26T23:49:06.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.851+0000 7f3dd8754700 1 -- start start 2024-01-26T23:49:06.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.852+0000 7f3dd8754700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3dd015ce40 0x7f3dd00d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.852+0000 7f3dd8754700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3dd00cc1c0 0x7f3dd00ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.852+0000 7f3dd8754700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd00cff70 con 0x7f3dd015d210 2024-01-26T23:49:06.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.852+0000 7f3dd8754700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dd00cf400 con 0x7f3dd015ce40 2024-01-26T23:49:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.852+0000 7f3dd8754700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dd00cf660 con 0x7f3dd00cc1c0 2024-01-26T23:49:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.852+0000 7f3dd64f0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3dd015ce40 0x7f3dd00d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.852+0000 7f3dd64f0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3dd015ce40 0x7f3dd00d0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52310/0 (socket says 172.21.15.46:52310) 2024-01-26T23:49:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.852+0000 7f3dd64f0700 1 -- 172.21.15.46:0/819964962 learned_addr learned my addr 172.21.15.46:0/819964962 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.852+0000 7f3dd5cef700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3dd015d210 0x7f3dd00cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42762/0 (socket says 172.21.15.46:42762) 2024-01-26T23:49:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.852+0000 7f3dd6cf1700 1 --2- 172.21.15.46:0/819964962 >> v2:172.21.15.106:3300/0 conn(0x7f3dd00cc1c0 0x7f3dd00ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.852+0000 7f3dd64f0700 1 -- 172.21.15.46:0/819964962 >> v2:172.21.15.106:3300/0 conn(0x7f3dd00cc1c0 msgr2=0x7f3dd00ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.852+0000 7f3dd64f0700 1 --2- 172.21.15.46:0/819964962 >> v2:172.21.15.106:3300/0 conn(0x7f3dd00cc1c0 0x7f3dd00ade30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.852+0000 7f3dd64f0700 1 -- 172.21.15.46:0/819964962 >> v1:172.21.15.46:6789/0 conn(0x7f3dd015d210 legacy=0x7f3dd00cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.853+0000 7f3dd64f0700 1 -- 172.21.15.46:0/819964962 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd00d0500 con 0x7f3dd015ce40 2024-01-26T23:49:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.852+0000 7f3dd54ee700 1 -- 172.21.15.46:0/819964962 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622375398 0 0) 0x7f3dd00cff70 con 0x7f3dd015d210 2024-01-26T23:49:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.853+0000 7f3dd64f0700 1 --2- 172.21.15.46:0/819964962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dd015ce40 0x7f3dd00d0e50 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f3dd00d01c0 tx=0x7f3dcc009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d97449055f47d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.853+0000 7f3dd54ee700 1 -- 172.21.15.46:0/819964962 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc005c10 con 0x7f3dd015ce40 2024-01-26T23:49:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.853+0000 7f3dd54ee700 1 -- 172.21.15.46:0/819964962 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dcc005d70 con 0x7f3dd015ce40 2024-01-26T23:49:06.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.853+0000 7f3dd54ee700 1 -- 172.21.15.46:0/819964962 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc006260 con 0x7f3dd015ce40 2024-01-26T23:49:06.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.854+0000 7f3dd8754700 1 -- 172.21.15.46:0/819964962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dd015ce40 msgr2=0x7f3dd00d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:06.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.854+0000 7f3dd8754700 1 --2- 172.21.15.46:0/819964962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dd015ce40 0x7f3dd00d0e50 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f3dd00d01c0 tx=0x7f3dcc009220 comp rx=0 tx=0).stop 2024-01-26T23:49:06.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.854+0000 7f3dd8754700 1 -- 172.21.15.46:0/819964962 shutdown_connections 2024-01-26T23:49:06.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.854+0000 7f3dd8754700 1 --2- 172.21.15.46:0/819964962 >> v2:172.21.15.106:3300/0 conn(0x7f3dd00cc1c0 0x7f3dd00ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.854+0000 7f3dd8754700 1 --2- 172.21.15.46:0/819964962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dd015ce40 0x7f3dd00d0e50 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.854+0000 7f3dd8754700 1 -- 172.21.15.46:0/819964962 >> 172.21.15.46:0/819964962 conn(0x7f3dd006f840 msgr2=0x7f3dd00d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:06.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.854+0000 7f3dd8754700 1 -- 172.21.15.46:0/819964962 shutdown_connections 2024-01-26T23:49:06.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.854+0000 7f3dd8754700 1 -- 172.21.15.46:0/819964962 wait complete. 2024-01-26T23:49:06.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.855+0000 7f3dd8754700 1 Processor -- start 2024-01-26T23:49:06.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.855+0000 7f3dd8754700 1 -- start start 2024-01-26T23:49:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.856+0000 7f3dd8754700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dd00cc1c0 0x7f3dd00c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.856+0000 7f3dd8754700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3dd015ce40 0x7f3dd015b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.856+0000 7f3dd8754700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3dd015d210 0x7f3dd009d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.856+0000 7f3dd8754700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd00c8f10 con 0x7f3dd00cc1c0 2024-01-26T23:49:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.856+0000 7f3dd8754700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd015bc80 con 0x7f3dd015d210 2024-01-26T23:49:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.856+0000 7f3dd8754700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd00c69e0 con 0x7f3dd015ce40 2024-01-26T23:49:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.856+0000 7f3dd64f0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dd00cc1c0 0x7f3dd00c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.856+0000 7f3dd64f0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dd00cc1c0 0x7f3dd00c89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52314/0 (socket says 172.21.15.46:52314) 2024-01-26T23:49:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.856+0000 7f3dd64f0700 1 -- 172.21.15.46:0/3269804928 learned_addr learned my addr 172.21.15.46:0/3269804928 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.856+0000 7f3dd6cf1700 1 --2- 172.21.15.46:0/3269804928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3dd015d210 0x7f3dd009d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.856+0000 7f3dd5cef700 1 --2- 172.21.15.46:0/3269804928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3dd015ce40 0x7f3dd015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.856+0000 7f3dd64f0700 1 -- 172.21.15.46:0/3269804928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3dd015ce40 msgr2=0x7f3dd015b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:06.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.856+0000 7f3dd64f0700 1 --2- 172.21.15.46:0/3269804928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3dd015ce40 0x7f3dd015b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.856+0000 7f3dd64f0700 1 -- 172.21.15.46:0/3269804928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3dd015d210 msgr2=0x7f3dd009d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:06.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.856+0000 7f3dd64f0700 1 --2- 172.21.15.46:0/3269804928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3dd015d210 0x7f3dd009d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:06.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.856+0000 7f3dd64f0700 1 -- 172.21.15.46:0/3269804928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc009480 con 0x7f3dd00cc1c0 2024-01-26T23:49:06.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.857+0000 7f3dd64f0700 1 --2- 172.21.15.46:0/3269804928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dd00cc1c0 0x7f3dd00c89d0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0088ec0 tx=0x7f3dcc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:06.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.858+0000 7f3dc67fc700 1 -- 172.21.15.46:0/3269804928 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc00b780 con 0x7f3dd00cc1c0 2024-01-26T23:49:06.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.858+0000 7f3dd8754700 1 -- 172.21.15.46:0/3269804928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd009ac10 con 0x7f3dd00cc1c0 2024-01-26T23:49:06.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.858+0000 7f3dd8754700 1 -- 172.21.15.46:0/3269804928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd009b1d0 con 0x7f3dd00cc1c0 2024-01-26T23:49:06.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.858+0000 7f3dc67fc700 1 -- 172.21.15.46:0/3269804928 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dcc00baf0 con 0x7f3dd00cc1c0 2024-01-26T23:49:06.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.858+0000 7f3dc67fc700 1 -- 172.21.15.46:0/3269804928 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc00bc60 con 0x7f3dd00cc1c0 2024-01-26T23:49:06.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.859+0000 7f3dd8754700 1 -- 172.21.15.46:0/3269804928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db80052b0 con 0x7f3dd00cc1c0 2024-01-26T23:49:06.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.860+0000 7f3dc67fc700 1 -- 172.21.15.46:0/3269804928 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 62) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f3dcc026430 con 0x7f3dd00cc1c0 2024-01-26T23:49:06.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.860+0000 7f3dc67fc700 1 --2- 172.21.15.46:0/3269804928 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3db4067170 0x7f3db4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:06.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.861+0000 7f3dc67fc700 1 -- 172.21.15.46:0/3269804928 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6731+0+0 (secure 0 0 0) 0x7f3dcc0866e0 con 0x7f3dd00cc1c0 2024-01-26T23:49:06.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.861+0000 7f3dd5cef700 1 -- 172.21.15.46:0/3269804928 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3db4067170 msgr2=0x7f3db4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:06.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.861+0000 7f3dd5cef700 1 --2- 172.21.15.46:0/3269804928 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3db4067170 0x7f3db4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:06.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:06.865+0000 7f3dc67fc700 1 -- 172.21.15.46:0/3269804928 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3dcc0531f0 con 0x7f3dd00cc1c0 2024-01-26T23:49:06.866 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:06.866+0000 7f3d5745a1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:49:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.062+0000 7f3dd5cef700 1 -- 172.21.15.46:0/3269804928 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3db4067170 msgr2=0x7f3db4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.062+0000 7f3dd5cef700 1 --2- 172.21.15.46:0/3269804928 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3db4067170 0x7f3db4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.074+0000 7f3dd8754700 1 -- 172.21.15.46:0/3269804928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3db8005830 con 0x7f3dd00cc1c0 2024-01-26T23:49:07.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.074+0000 7f3dc67fc700 1 -- 172.21.15.46:0/3269804928 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+16647 (secure 0 0 0) 0x7f3dcc056810 con 0x7f3dd00cc1c0 2024-01-26T23:49:07.075 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:07.075 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":132,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:49:05.851604+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:48:09.662590+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-01-26T23:48:10.200635+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":132,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/1920429547":"2024-01-27T23:48:30.748025+0000","172.21.15.106:0/674271642":"2024-01-27T23:48:30.748025+0000","172.21.15.106:6837/43388":"2024-01-27T23:48:30.748025+0000","172.21.15.106:6835/43388":"2024-01-27T23:48:30.748025+0000","172.21.15.106:0/2229853363":"2024-01-27T23:48:30.748025+0000","172.21.15.106:0/3551046672":"2024-01-27T23:48:30.748025+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:49:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.077+0000 7f3dd8754700 1 -- 172.21.15.46:0/3269804928 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3db4067170 msgr2=0x7f3db4069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.077+0000 7f3dd8754700 1 --2- 172.21.15.46:0/3269804928 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3db4067170 0x7f3db4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.077+0000 7f3dd8754700 1 -- 172.21.15.46:0/3269804928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dd00cc1c0 msgr2=0x7f3dd00c89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.077+0000 7f3dd8754700 1 --2- 172.21.15.46:0/3269804928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dd00cc1c0 0x7f3dd00c89d0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0088ec0 tx=0x7f3dcc016000 comp rx=0 tx=0).stop 2024-01-26T23:49:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.077+0000 7f3dd8754700 1 -- 172.21.15.46:0/3269804928 shutdown_connections 2024-01-26T23:49:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.077+0000 7f3dd8754700 1 --2- 172.21.15.46:0/3269804928 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3db4067170 0x7f3db4069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.077+0000 7f3dd8754700 1 --2- 172.21.15.46:0/3269804928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dd00cc1c0 0x7f3dd00c89d0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.077+0000 7f3dd8754700 1 --2- 172.21.15.46:0/3269804928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3dd015d210 0x7f3dd009d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.077+0000 7f3dd8754700 1 --2- 172.21.15.46:0/3269804928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3dd015ce40 0x7f3dd015b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.077+0000 7f3dd8754700 1 -- 172.21.15.46:0/3269804928 >> 172.21.15.46:0/3269804928 conn(0x7f3dd006f840 msgr2=0x7f3dd00b4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.077+0000 7f3dd8754700 1 -- 172.21.15.46:0/3269804928 shutdown_connections 2024-01-26T23:49:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.077+0000 7f3dd8754700 1 -- 172.21.15.46:0/3269804928 wait complete. 2024-01-26T23:49:07.092 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-26T23:49:07.092 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:49:07.183 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:07.183+0000 7f30bcb891c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:49:07.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.225+0000 7fc205381700 1 Processor -- start 2024-01-26T23:49:07.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.225+0000 7fc205381700 1 -- start start 2024-01-26T23:49:07.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.226+0000 7fc205381700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2000b2b40 0x7fc2000ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:07.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.226+0000 7fc205381700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc2000aedd0 0x7fc2000cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:07.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.226+0000 7fc205381700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20015a0b0 con 0x7fc2000ae960 2024-01-26T23:49:07.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.226+0000 7fc205381700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc200099fc0 con 0x7fc2000aedd0 2024-01-26T23:49:07.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.226+0000 7fc205381700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2000ae4d0 con 0x7fc2000b2b40 2024-01-26T23:49:07.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.226+0000 7fc1ff7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc2000aedd0 0x7fc2000cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:07.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.226+0000 7fc1fe7fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc2000ae960 0x7fc2000ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42766/0 (socket says 172.21.15.46:42766) 2024-01-26T23:49:07.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.226+0000 7fc1ff7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc2000aedd0 0x7fc2000cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52324/0 (socket says 172.21.15.46:52324) 2024-01-26T23:49:07.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.226+0000 7fc1fe7fc700 1 -- 172.21.15.46:0/1450742927 learned_addr learned my addr 172.21.15.46:0/1450742927 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:07.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.226+0000 7fc1feffd700 1 --2- 172.21.15.46:0/1450742927 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2000b2b40 0x7fc2000ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:07.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.227+0000 7fc1ff7fe700 1 -- 172.21.15.46:0/1450742927 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2000b2b40 msgr2=0x7fc2000ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:07.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.227+0000 7fc1ff7fe700 1 --2- 172.21.15.46:0/1450742927 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2000b2b40 0x7fc2000ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.227+0000 7fc1ff7fe700 1 -- 172.21.15.46:0/1450742927 >> v1:172.21.15.46:6789/0 conn(0x7fc2000ae960 legacy=0x7fc2000ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:07.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.227+0000 7fc1ff7fe700 1 -- 172.21.15.46:0/1450742927 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2000ad8c0 con 0x7fc2000aedd0 2024-01-26T23:49:07.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.227+0000 7fc1fdffb700 1 -- 172.21.15.46:0/1450742927 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356947248 0 0) 0x7fc20015a0b0 con 0x7fc2000ae960 2024-01-26T23:49:07.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.227+0000 7fc1ff7fe700 1 --2- 172.21.15.46:0/1450742927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2000aedd0 0x7fc2000cb5e0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fc1f80098e0 tx=0x7fc1f80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea9f1a993cdb080 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:07.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.227+0000 7fc1fdffb700 1 -- 172.21.15.46:0/1450742927 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f800b640 con 0x7fc2000aedd0 2024-01-26T23:49:07.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.227+0000 7fc1fdffb700 1 -- 172.21.15.46:0/1450742927 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f800b7a0 con 0x7fc2000aedd0 2024-01-26T23:49:07.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.227+0000 7fc1fdffb700 1 -- 172.21.15.46:0/1450742927 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f800bb40 con 0x7fc2000aedd0 2024-01-26T23:49:07.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.228+0000 7fc205381700 1 -- 172.21.15.46:0/1450742927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2000aedd0 msgr2=0x7fc2000cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:07.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.228+0000 7fc205381700 1 --2- 172.21.15.46:0/1450742927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2000aedd0 0x7fc2000cb5e0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fc1f80098e0 tx=0x7fc1f80091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.228+0000 7fc205381700 1 -- 172.21.15.46:0/1450742927 shutdown_connections 2024-01-26T23:49:07.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.228+0000 7fc205381700 1 --2- 172.21.15.46:0/1450742927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2000aedd0 0x7fc2000cb5e0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.228+0000 7fc205381700 1 --2- 172.21.15.46:0/1450742927 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2000b2b40 0x7fc2000ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.228+0000 7fc205381700 1 -- 172.21.15.46:0/1450742927 >> 172.21.15.46:0/1450742927 conn(0x7fc200099a30 msgr2=0x7fc20006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:07.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.228+0000 7fc205381700 1 -- 172.21.15.46:0/1450742927 shutdown_connections 2024-01-26T23:49:07.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.228+0000 7fc205381700 1 -- 172.21.15.46:0/1450742927 wait complete. 2024-01-26T23:49:07.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.230+0000 7fc205381700 1 Processor -- start 2024-01-26T23:49:07.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.230+0000 7fc205381700 1 -- start start 2024-01-26T23:49:07.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.231+0000 7fc205381700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2000ae960 0x7fc2000b5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:07.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.232+0000 7fc205381700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2000b2b40 0x7fc2000b4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:07.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.232+0000 7fc205381700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc200144d30 0x7fc200145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:07.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.232+0000 7fc205381700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2000b63e0 con 0x7fc2000ae960 2024-01-26T23:49:07.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.232+0000 7fc205381700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2000b36b0 con 0x7fc200144d30 2024-01-26T23:49:07.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.232+0000 7fc205381700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2000b5b40 con 0x7fc2000b2b40 2024-01-26T23:49:07.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.232+0000 7fc1feffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2000ae960 0x7fc2000b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:07.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.232+0000 7fc1feffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2000ae960 0x7fc2000b5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52332/0 (socket says 172.21.15.46:52332) 2024-01-26T23:49:07.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.232+0000 7fc1fe7fc700 1 --2- 172.21.15.46:0/48093436 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2000b2b40 0x7fc2000b4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:07.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.232+0000 7fc1feffd700 1 -- 172.21.15.46:0/48093436 learned_addr learned my addr 172.21.15.46:0/48093436 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:07.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.232+0000 7fc1ff7fe700 1 --2- 172.21.15.46:0/48093436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc200144d30 0x7fc200145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:07.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.232+0000 7fc1fe7fc700 1 -- 172.21.15.46:0/48093436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc200144d30 msgr2=0x7fc200145710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:07.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.232+0000 7fc1fe7fc700 1 --2- 172.21.15.46:0/48093436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc200144d30 0x7fc200145710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.232+0000 7fc1fe7fc700 1 -- 172.21.15.46:0/48093436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2000ae960 msgr2=0x7fc2000b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:07.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.232+0000 7fc1fe7fc700 1 --2- 172.21.15.46:0/48093436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2000ae960 0x7fc2000b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.232+0000 7fc1fe7fc700 1 -- 172.21.15.46:0/48093436 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f8009420 con 0x7fc2000b2b40 2024-01-26T23:49:07.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.232+0000 7fc1fe7fc700 1 --2- 172.21.15.46:0/48093436 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2000b2b40 0x7fc2000b4d10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc1f400c520 tx=0x7fc1f4007f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:07.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.233+0000 7fc1e6ffd700 1 -- 172.21.15.46:0/48093436 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f400ce90 con 0x7fc2000b2b40 2024-01-26T23:49:07.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.233+0000 7fc1e6ffd700 1 -- 172.21.15.46:0/48093436 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f4005b30 con 0x7fc2000b2b40 2024-01-26T23:49:07.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.233+0000 7fc205381700 1 -- 172.21.15.46:0/48093436 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2000b5e20 con 0x7fc2000b2b40 2024-01-26T23:49:07.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.233+0000 7fc1e6ffd700 1 -- 172.21.15.46:0/48093436 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f40070d0 con 0x7fc2000b2b40 2024-01-26T23:49:07.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.233+0000 7fc205381700 1 -- 172.21.15.46:0/48093436 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc200145c50 con 0x7fc2000b2b40 2024-01-26T23:49:07.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.234+0000 7fc205381700 1 -- 172.21.15.46:0/48093436 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2000b3510 con 0x7fc2000b2b40 2024-01-26T23:49:07.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.235+0000 7fc1e6ffd700 1 -- 172.21.15.46:0/48093436 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 62) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fc1f40074c0 con 0x7fc2000b2b40 2024-01-26T23:49:07.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.235+0000 7fc1e6ffd700 1 --2- 172.21.15.46:0/48093436 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc1ec067170 0x7fc1ec069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:07.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.235+0000 7fc1feffd700 1 -- 172.21.15.46:0/48093436 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc1ec067170 msgr2=0x7fc1ec069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:07.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.235+0000 7fc1e6ffd700 1 -- 172.21.15.46:0/48093436 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6731+0+0 (secure 0 0 0) 0x7fc1f4083ff0 con 0x7fc2000b2b40 2024-01-26T23:49:07.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.235+0000 7fc1feffd700 1 --2- 172.21.15.46:0/48093436 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc1ec067170 0x7fc1ec069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:07.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.239+0000 7fc1e6ffd700 1 -- 172.21.15.46:0/48093436 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc2000b3510 con 0x7fc2000b2b40 2024-01-26T23:49:07.273 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:07.273+0000 7f30bcb891c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:49:07.370 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:07.369+0000 7f3d5745a1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:49:07.415 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:07.415+0000 7f30bcb891c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:49:07.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.436+0000 7fc1feffd700 1 -- 172.21.15.46:0/48093436 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc1ec067170 msgr2=0x7fc1ec069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:07.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.436+0000 7fc1feffd700 1 --2- 172.21.15.46:0/48093436 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc1ec067170 0x7fc1ec069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:07.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.499+0000 7fc205381700 1 -- 172.21.15.46:0/48093436 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2000b3510 con 0x7fc2000b2b40 2024-01-26T23:49:07.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.500+0000 7fc1e6ffd700 1 -- 172.21.15.46:0/48093436 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4599 (secure 0 0 0) 0x7fc2000b3510 con 0x7fc2000b2b40 2024-01-26T23:49:07.500 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:07.501 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":62,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6923,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/4123353304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4123353304},{"type":"v1","addr":"172.21.15.106:6834","nonce":4123353304}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/690662483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":690662483},{"type":"v1","addr":"172.21.15.106:6836","nonce":690662483}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7791,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3035969379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3035969379},{"type":"v1","addr":"172.21.15.46:6834","nonce":3035969379}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:48:11.234632+0000","modified":"2024-01-26T23:48:14.815720+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7794},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7794":{"gid":7794,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/783623731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":783623731},{"type":"v1","addr":"172.21.15.46:6835","nonce":783623731}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2024-01-26T23:49:07.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.502+0000 7fc205381700 1 -- 172.21.15.46:0/48093436 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc1ec067170 msgr2=0x7fc1ec069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:07.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.502+0000 7fc205381700 1 --2- 172.21.15.46:0/48093436 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc1ec067170 0x7fc1ec069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.502+0000 7fc205381700 1 -- 172.21.15.46:0/48093436 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2000b2b40 msgr2=0x7fc2000b4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:07.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.502+0000 7fc205381700 1 --2- 172.21.15.46:0/48093436 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2000b2b40 0x7fc2000b4d10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc1f400c520 tx=0x7fc1f4007f50 comp rx=0 tx=0).stop 2024-01-26T23:49:07.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.502+0000 7fc205381700 1 -- 172.21.15.46:0/48093436 shutdown_connections 2024-01-26T23:49:07.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.502+0000 7fc205381700 1 --2- 172.21.15.46:0/48093436 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc1ec067170 0x7fc1ec069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.503+0000 7fc205381700 1 --2- 172.21.15.46:0/48093436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2000ae960 0x7fc2000b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.503+0000 7fc205381700 1 --2- 172.21.15.46:0/48093436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc200144d30 0x7fc200145710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.503+0000 7fc205381700 1 --2- 172.21.15.46:0/48093436 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2000b2b40 0x7fc2000b4d10 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.503+0000 7fc205381700 1 -- 172.21.15.46:0/48093436 >> 172.21.15.46:0/48093436 conn(0x7fc200099a30 msgr2=0x7fc2001594e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:07.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.503+0000 7fc205381700 1 -- 172.21.15.46:0/48093436 shutdown_connections 2024-01-26T23:49:07.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.503+0000 7fc205381700 1 -- 172.21.15.46:0/48093436 wait complete. 2024-01-26T23:49:07.504 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 62 2024-01-26T23:49:07.517 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:49:07.519 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:07.518+0000 7f3d5745a1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:49:07.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.650+0000 7fb68f85e700 1 Processor -- start 2024-01-26T23:49:07.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.650+0000 7fb68f85e700 1 -- start start 2024-01-26T23:49:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.651+0000 7fb68f85e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6880c6520 0x7fb6880ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.651+0000 7fb68f85e700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68815a0b0 con 0x7fb6880b2b40 2024-01-26T23:49:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.651+0000 7fb68f85e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6880ae4d0 con 0x7fb6880ced50 2024-01-26T23:49:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.651+0000 7fb68f85e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6880af070 con 0x7fb6880c6520 2024-01-26T23:49:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.651+0000 7fb68d5fa700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb6880b2b40 0x7fb6880ae960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42782/0 (socket says 172.21.15.46:42782) 2024-01-26T23:49:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.651+0000 7fb68d5fa700 1 -- 172.21.15.46:0/2796182887 learned_addr learned my addr 172.21.15.46:0/2796182887 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:07.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.651+0000 7fb68cdf9700 1 --2- 172.21.15.46:0/2796182887 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6880c6520 0x7fb6880ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.651+0000 7fb67ffff700 1 -- 172.21.15.46:0/2796182887 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059731454 0 0) 0x7fb68815a0b0 con 0x7fb6880b2b40 2024-01-26T23:49:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.651+0000 7fb67ffff700 1 -- 172.21.15.46:0/2796182887 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb674003570 con 0x7fb6880b2b40 2024-01-26T23:49:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.651+0000 7fb67ffff700 1 -- 172.21.15.46:0/2796182887 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296622654 0 0) 0x7fb6880ae4d0 con 0x7fb6880ced50 2024-01-26T23:49:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.651+0000 7fb67ffff700 1 -- 172.21.15.46:0/2796182887 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb68815a0b0 con 0x7fb6880ced50 2024-01-26T23:49:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.651+0000 7fb68cdf9700 1 -- 172.21.15.46:0/2796182887 >> v1:172.21.15.106:6789/0 conn(0x7fb6880ced50 legacy=0x7fb6880cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.652+0000 7fb68cdf9700 1 -- 172.21.15.46:0/2796182887 >> v1:172.21.15.46:6789/0 conn(0x7fb6880b2b40 legacy=0x7fb6880ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.652+0000 7fb68cdf9700 1 -- 172.21.15.46:0/2796182887 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6880ad8c0 con 0x7fb6880c6520 2024-01-26T23:49:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.652+0000 7fb68cdf9700 1 --2- 172.21.15.46:0/2796182887 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6880c6520 0x7fb6880ad1c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb678009910 tx=0x7fb6780091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7407a8cd75c0adc server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.652+0000 7fb67ffff700 1 -- 172.21.15.46:0/2796182887 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678011630 con 0x7fb6880c6520 2024-01-26T23:49:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.652+0000 7fb67ffff700 1 -- 172.21.15.46:0/2796182887 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb678011790 con 0x7fb6880c6520 2024-01-26T23:49:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.652+0000 7fb67ffff700 1 -- 172.21.15.46:0/2796182887 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678011b30 con 0x7fb6880c6520 2024-01-26T23:49:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.653+0000 7fb68f85e700 1 -- 172.21.15.46:0/2796182887 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6880c6520 msgr2=0x7fb6880ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.653+0000 7fb68f85e700 1 --2- 172.21.15.46:0/2796182887 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6880c6520 0x7fb6880ad1c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb678009910 tx=0x7fb6780091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.653+0000 7fb68f85e700 1 -- 172.21.15.46:0/2796182887 shutdown_connections 2024-01-26T23:49:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.653+0000 7fb68f85e700 1 --2- 172.21.15.46:0/2796182887 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6880c6520 0x7fb6880ad1c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.653+0000 7fb68f85e700 1 -- 172.21.15.46:0/2796182887 >> 172.21.15.46:0/2796182887 conn(0x7fb688099a30 msgr2=0x7fb68806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.653+0000 7fb68f85e700 1 -- 172.21.15.46:0/2796182887 shutdown_connections 2024-01-26T23:49:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.653+0000 7fb68f85e700 1 -- 172.21.15.46:0/2796182887 wait complete. 2024-01-26T23:49:07.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.655+0000 7fb68f85e700 1 Processor -- start 2024-01-26T23:49:07.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.655+0000 7fb68f85e700 1 -- start start 2024-01-26T23:49:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.656+0000 7fb68f85e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6880b2b40 0x7fb6880b3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.656+0000 7fb68f85e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6880c6520 0x7fb6881516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.656+0000 7fb68f85e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6880ced50 0x7fb6880b0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.656+0000 7fb68f85e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68816d210 con 0x7fb6880ced50 2024-01-26T23:49:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.656+0000 7fb68f85e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68816d460 con 0x7fb6880b2b40 2024-01-26T23:49:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.656+0000 7fb68f85e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68816d740 con 0x7fb6880c6520 2024-01-26T23:49:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.656+0000 7fb68ddfb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6880ced50 0x7fb6880b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.656+0000 7fb68d5fa700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6880b2b40 0x7fb6880b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.656+0000 7fb68cdf9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6880c6520 0x7fb6881516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.656+0000 7fb68ddfb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6880ced50 0x7fb6880b0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52342/0 (socket says 172.21.15.46:52342) 2024-01-26T23:49:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.656+0000 7fb68ddfb700 1 -- 172.21.15.46:0/3604554238 learned_addr learned my addr 172.21.15.46:0/3604554238 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.656+0000 7fb68cdf9700 1 -- 172.21.15.46:0/3604554238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6880b2b40 msgr2=0x7fb6880b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.656+0000 7fb68cdf9700 1 --2- 172.21.15.46:0/3604554238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6880b2b40 0x7fb6880b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.656+0000 7fb68cdf9700 1 -- 172.21.15.46:0/3604554238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6880ced50 msgr2=0x7fb6880b0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.656+0000 7fb68cdf9700 1 --2- 172.21.15.46:0/3604554238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6880ced50 0x7fb6880b0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.656+0000 7fb68cdf9700 1 -- 172.21.15.46:0/3604554238 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb678009420 con 0x7fb6880c6520 2024-01-26T23:49:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.656+0000 7fb68ddfb700 1 --2- 172.21.15.46:0/3604554238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6880ced50 0x7fb6880b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.657+0000 7fb68cdf9700 1 --2- 172.21.15.46:0/3604554238 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6880c6520 0x7fb6881516d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb678016000 tx=0x7fb678017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.657+0000 7fb67d7fa700 1 -- 172.21.15.46:0/3604554238 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67801f070 con 0x7fb6880c6520 2024-01-26T23:49:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.657+0000 7fb67d7fa700 1 -- 172.21.15.46:0/3604554238 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb678002f70 con 0x7fb6880c6520 2024-01-26T23:49:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.657+0000 7fb68f85e700 1 -- 172.21.15.46:0/3604554238 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68816da20 con 0x7fb6880c6520 2024-01-26T23:49:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.657+0000 7fb67d7fa700 1 -- 172.21.15.46:0/3604554238 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678011900 con 0x7fb6880c6520 2024-01-26T23:49:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.657+0000 7fb68f85e700 1 -- 172.21.15.46:0/3604554238 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68816dfe0 con 0x7fb6880c6520 2024-01-26T23:49:07.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.658+0000 7fb68f85e700 1 -- 172.21.15.46:0/3604554238 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6880c7800 con 0x7fb6880c6520 2024-01-26T23:49:07.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.658+0000 7fb67d7fa700 1 -- 172.21.15.46:0/3604554238 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 62) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fb678023070 con 0x7fb6880c6520 2024-01-26T23:49:07.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.659+0000 7fb67d7fa700 1 --2- 172.21.15.46:0/3604554238 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb66c067170 0x7fb66c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:07.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.659+0000 7fb67d7fa700 1 -- 172.21.15.46:0/3604554238 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6731+0+0 (secure 0 0 0) 0x7fb678085de0 con 0x7fb6880c6520 2024-01-26T23:49:07.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.659+0000 7fb68d5fa700 1 -- 172.21.15.46:0/3604554238 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb66c067170 msgr2=0x7fb66c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:07.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.659+0000 7fb68d5fa700 1 --2- 172.21.15.46:0/3604554238 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb66c067170 0x7fb66c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:07.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.663+0000 7fb67d7fa700 1 -- 172.21.15.46:0/3604554238 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb678052920 con 0x7fb6880c6520 2024-01-26T23:49:07.671 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:07.670+0000 7f3d5745a1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:49:07.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.860+0000 7fb68d5fa700 1 -- 172.21.15.46:0/3604554238 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb66c067170 msgr2=0x7fb66c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:07.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.860+0000 7fb68d5fa700 1 --2- 172.21.15.46:0/3604554238 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb66c067170 0x7fb66c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:07.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.922+0000 7fb68f85e700 1 -- 172.21.15.46:0/3604554238 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb68804c710 con 0x7fb6880c6520 2024-01-26T23:49:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.923+0000 7fb67d7fa700 1 -- 172.21.15.46:0/3604554238 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4599 (secure 0 0 0) 0x7fb678055f40 con 0x7fb6880c6520 2024-01-26T23:49:07.923 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:07.923 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":62,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6923,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/4123353304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4123353304},{"type":"v1","addr":"172.21.15.106:6834","nonce":4123353304}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/690662483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":690662483},{"type":"v1","addr":"172.21.15.106:6836","nonce":690662483}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7791,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3035969379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3035969379},{"type":"v1","addr":"172.21.15.46:6834","nonce":3035969379}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:48:11.234632+0000","modified":"2024-01-26T23:48:14.815720+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7794},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7794":{"gid":7794,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/783623731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":783623731},{"type":"v1","addr":"172.21.15.46:6835","nonce":783623731}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2024-01-26T23:49:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.925+0000 7fb68f85e700 1 -- 172.21.15.46:0/3604554238 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb66c067170 msgr2=0x7fb66c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.925+0000 7fb68f85e700 1 --2- 172.21.15.46:0/3604554238 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb66c067170 0x7fb66c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.925+0000 7fb68f85e700 1 -- 172.21.15.46:0/3604554238 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6880c6520 msgr2=0x7fb6881516d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.925+0000 7fb68f85e700 1 --2- 172.21.15.46:0/3604554238 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6880c6520 0x7fb6881516d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb678016000 tx=0x7fb678017000 comp rx=0 tx=0).stop 2024-01-26T23:49:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.926+0000 7fb68f85e700 1 -- 172.21.15.46:0/3604554238 shutdown_connections 2024-01-26T23:49:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.926+0000 7fb68f85e700 1 --2- 172.21.15.46:0/3604554238 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb66c067170 0x7fb66c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.926+0000 7fb68f85e700 1 --2- 172.21.15.46:0/3604554238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6880ced50 0x7fb6880b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.926+0000 7fb68f85e700 1 --2- 172.21.15.46:0/3604554238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6880b2b40 0x7fb6880b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.926+0000 7fb68f85e700 1 --2- 172.21.15.46:0/3604554238 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6880c6520 0x7fb6881516d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.926+0000 7fb68f85e700 1 -- 172.21.15.46:0/3604554238 >> 172.21.15.46:0/3604554238 conn(0x7fb688099a30 msgr2=0x7fb688158f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.926+0000 7fb68f85e700 1 -- 172.21.15.46:0/3604554238 shutdown_connections 2024-01-26T23:49:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:07.926+0000 7fb68f85e700 1 -- 172.21.15.46:0/3604554238 wait complete. 2024-01-26T23:49:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 62 2024-01-26T23:49:07.940 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:49:08.014 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:08.013+0000 7f30bcb891c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:49:08.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.075+0000 7f0f9b150700 1 Processor -- start 2024-01-26T23:49:08.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.075+0000 7f0f9b150700 1 -- start start 2024-01-26T23:49:08.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.076+0000 7f0f9b150700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f941501e0 0x7f0f940af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:08.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.076+0000 7f0f9b150700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f94156f80 0x7f0f940afef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:08.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.076+0000 7f0f9b150700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f94156250 con 0x7f0f940b09a0 2024-01-26T23:49:08.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.076+0000 7f0f9b150700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f940b0800 con 0x7f0f94156f80 2024-01-26T23:49:08.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.076+0000 7f0f9b150700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f94156af0 con 0x7f0f941501e0 2024-01-26T23:49:08.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.076+0000 7f0f93fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f94156f80 0x7f0f940afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:08.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.076+0000 7f0f93fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f94156f80 0x7f0f940afef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52354/0 (socket says 172.21.15.46:52354) 2024-01-26T23:49:08.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.076+0000 7f0f93fff700 1 -- 172.21.15.46:0/1762236568 learned_addr learned my addr 172.21.15.46:0/1762236568 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:08.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.076+0000 7f0f98eec700 1 --2- 172.21.15.46:0/1762236568 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f941501e0 0x7f0f940af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:08.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.076+0000 7f0f93fff700 1 -- 172.21.15.46:0/1762236568 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f941501e0 msgr2=0x7f0f940af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:08.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.076+0000 7f0f93fff700 1 --2- 172.21.15.46:0/1762236568 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f941501e0 0x7f0f940af9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.076+0000 7f0f93fff700 1 -- 172.21.15.46:0/1762236568 >> v1:172.21.15.106:6789/0 conn(0x7f0f940b09a0 legacy=0x7f0f94159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:08.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.077+0000 7f0f93fff700 1 -- 172.21.15.46:0/1762236568 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f940691e0 con 0x7f0f94156f80 2024-01-26T23:49:08.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.076+0000 7f0f937fe700 1 -- 172.21.15.46:0/1762236568 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911107033 0 0) 0x7f0f94156250 con 0x7f0f940b09a0 2024-01-26T23:49:08.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.077+0000 7f0f93fff700 1 --2- 172.21.15.46:0/1762236568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f94156f80 0x7f0f940afef0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f0f8000ab50 tx=0x7f0f8000a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27189b7564aabbde server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:08.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.077+0000 7f0f937fe700 1 -- 172.21.15.46:0/1762236568 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8000ca40 con 0x7f0f94156f80 2024-01-26T23:49:08.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.077+0000 7f0f937fe700 1 -- 172.21.15.46:0/1762236568 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8000cba0 con 0x7f0f94156f80 2024-01-26T23:49:08.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.077+0000 7f0f937fe700 1 -- 172.21.15.46:0/1762236568 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f80018400 con 0x7f0f94156f80 2024-01-26T23:49:08.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.078+0000 7f0f9b150700 1 -- 172.21.15.46:0/1762236568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f94156f80 msgr2=0x7f0f940afef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:08.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.078+0000 7f0f9b150700 1 --2- 172.21.15.46:0/1762236568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f94156f80 0x7f0f940afef0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f0f8000ab50 tx=0x7f0f8000a3f0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.078+0000 7f0f9b150700 1 -- 172.21.15.46:0/1762236568 shutdown_connections 2024-01-26T23:49:08.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.078+0000 7f0f9b150700 1 --2- 172.21.15.46:0/1762236568 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f941501e0 0x7f0f940af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.078+0000 7f0f9b150700 1 --2- 172.21.15.46:0/1762236568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f94156f80 0x7f0f940afef0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.078+0000 7f0f9b150700 1 -- 172.21.15.46:0/1762236568 >> 172.21.15.46:0/1762236568 conn(0x7f0f9406e2c0 msgr2=0x7f0f940ae620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:08.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.078+0000 7f0f9b150700 1 -- 172.21.15.46:0/1762236568 shutdown_connections 2024-01-26T23:49:08.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.078+0000 7f0f9b150700 1 -- 172.21.15.46:0/1762236568 wait complete. 2024-01-26T23:49:08.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.079+0000 7f0f9b150700 1 Processor -- start 2024-01-26T23:49:08.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.079+0000 7f0f9b150700 1 -- start start 2024-01-26T23:49:08.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f9b150700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f940b09a0 0x7f0f9413bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:08.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f9b150700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f941501e0 0x7f0f940a8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:08.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f9b150700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f94156f80 0x7f0f940a8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:08.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f9b150700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f941687b0 con 0x7f0f940b09a0 2024-01-26T23:49:08.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f9b150700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f94154f70 con 0x7f0f94156f80 2024-01-26T23:49:08.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f9b150700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f94155250 con 0x7f0f941501e0 2024-01-26T23:49:08.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f98eec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f940b09a0 0x7f0f9413bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:08.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f98eec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f940b09a0 0x7f0f9413bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52364/0 (socket says 172.21.15.46:52364) 2024-01-26T23:49:08.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f98eec700 1 -- 172.21.15.46:0/3035317273 learned_addr learned my addr 172.21.15.46:0/3035317273 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:08.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f93fff700 1 --2- 172.21.15.46:0/3035317273 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f941501e0 0x7f0f940a8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:08.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f996ed700 1 --2- 172.21.15.46:0/3035317273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f94156f80 0x7f0f940a8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:08.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f98eec700 1 -- 172.21.15.46:0/3035317273 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f941501e0 msgr2=0x7f0f940a8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:08.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f98eec700 1 --2- 172.21.15.46:0/3035317273 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f941501e0 0x7f0f940a8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f98eec700 1 -- 172.21.15.46:0/3035317273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f94156f80 msgr2=0x7f0f940a8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:08.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f98eec700 1 --2- 172.21.15.46:0/3035317273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f94156f80 0x7f0f940a8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f98eec700 1 -- 172.21.15.46:0/3035317273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8000a690 con 0x7f0f940b09a0 2024-01-26T23:49:08.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f98eec700 1 --2- 172.21.15.46:0/3035317273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f940b09a0 0x7f0f9413bd30 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800ec80 tx=0x7f0f88009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:08.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.081+0000 7f0f90ff9700 1 -- 172.21.15.46:0/3035317273 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8800c850 con 0x7f0f940b09a0 2024-01-26T23:49:08.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.082+0000 7f0f90ff9700 1 -- 172.21.15.46:0/3035317273 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8800c9b0 con 0x7f0f940b09a0 2024-01-26T23:49:08.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.082+0000 7f0f90ff9700 1 -- 172.21.15.46:0/3035317273 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8800cd30 con 0x7f0f940b09a0 2024-01-26T23:49:08.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.082+0000 7f0f9b150700 1 -- 172.21.15.46:0/3035317273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f94155530 con 0x7f0f940b09a0 2024-01-26T23:49:08.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.082+0000 7f0f9b150700 1 -- 172.21.15.46:0/3035317273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9413c270 con 0x7f0f940b09a0 2024-01-26T23:49:08.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.083+0000 7f0f9b150700 1 -- 172.21.15.46:0/3035317273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f94168a90 con 0x7f0f940b09a0 2024-01-26T23:49:08.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.083+0000 7f0f90ff9700 1 -- 172.21.15.46:0/3035317273 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 62) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f0f88013680 con 0x7f0f940b09a0 2024-01-26T23:49:08.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.084+0000 7f0f90ff9700 1 --2- 172.21.15.46:0/3035317273 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0f78067170 0x7f0f78069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:08.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.084+0000 7f0f93fff700 1 -- 172.21.15.46:0/3035317273 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0f78067170 msgr2=0x7f0f78069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:08.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.084+0000 7f0f90ff9700 1 -- 172.21.15.46:0/3035317273 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6731+0+0 (secure 0 0 0) 0x7f0f88085270 con 0x7f0f940b09a0 2024-01-26T23:49:08.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.084+0000 7f0f93fff700 1 --2- 172.21.15.46:0/3035317273 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0f78067170 0x7f0f78069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:08.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.088+0000 7f0f90ff9700 1 -- 172.21.15.46:0/3035317273 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0f88051df0 con 0x7f0f940b09a0 2024-01-26T23:49:08.191 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:08.190+0000 7f30bcb891c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:49:08.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.285+0000 7f0f93fff700 1 -- 172.21.15.46:0/3035317273 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0f78067170 msgr2=0x7f0f78069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:08.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.285+0000 7f0f93fff700 1 --2- 172.21.15.46:0/3035317273 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0f78067170 0x7f0f78069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:08.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.295+0000 7f0f9b150700 1 -- 172.21.15.46:0/3035317273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f9404c710 con 0x7f0f940b09a0 2024-01-26T23:49:08.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.296+0000 7f0f90ff9700 1 -- 172.21.15.46:0/3035317273 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+16647 (secure 0 0 0) 0x7f0f88055410 con 0x7f0f940b09a0 2024-01-26T23:49:08.296 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:08.296 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":132,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:49:05.851604+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:48:09.662590+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-01-26T23:48:10.200635+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":132,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/1920429547":"2024-01-27T23:48:30.748025+0000","172.21.15.106:0/674271642":"2024-01-27T23:48:30.748025+0000","172.21.15.106:6837/43388":"2024-01-27T23:48:30.748025+0000","172.21.15.106:6835/43388":"2024-01-27T23:48:30.748025+0000","172.21.15.106:0/2229853363":"2024-01-27T23:48:30.748025+0000","172.21.15.106:0/3551046672":"2024-01-27T23:48:30.748025+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:49:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.298+0000 7f0f9b150700 1 -- 172.21.15.46:0/3035317273 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0f78067170 msgr2=0x7f0f78069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.298+0000 7f0f9b150700 1 --2- 172.21.15.46:0/3035317273 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0f78067170 0x7f0f78069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.298+0000 7f0f9b150700 1 -- 172.21.15.46:0/3035317273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f940b09a0 msgr2=0x7f0f9413bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.298+0000 7f0f9b150700 1 --2- 172.21.15.46:0/3035317273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f940b09a0 0x7f0f9413bd30 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800ec80 tx=0x7f0f88009120 comp rx=0 tx=0).stop 2024-01-26T23:49:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.298+0000 7f0f9b150700 1 -- 172.21.15.46:0/3035317273 shutdown_connections 2024-01-26T23:49:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.298+0000 7f0f9b150700 1 --2- 172.21.15.46:0/3035317273 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0f78067170 0x7f0f78069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.298+0000 7f0f9b150700 1 --2- 172.21.15.46:0/3035317273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f940b09a0 0x7f0f9413bd30 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.298+0000 7f0f9b150700 1 --2- 172.21.15.46:0/3035317273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f94156f80 0x7f0f940a8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.298+0000 7f0f9b150700 1 --2- 172.21.15.46:0/3035317273 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f941501e0 0x7f0f940a8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.298+0000 7f0f9b150700 1 -- 172.21.15.46:0/3035317273 >> 172.21.15.46:0/3035317273 conn(0x7f0f9406e2c0 msgr2=0x7f0f9406b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.298+0000 7f0f9b150700 1 -- 172.21.15.46:0/3035317273 shutdown_connections 2024-01-26T23:49:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.298+0000 7f0f9b150700 1 -- 172.21.15.46:0/3035317273 wait complete. 2024-01-26T23:49:08.313 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-26T23:49:08.379 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:08.378+0000 7f3d5745a1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:49:08.381 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:08.381+0000 7f30bcb891c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:49:08.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.489+0000 7f047923b700 1 Processor -- start 2024-01-26T23:49:08.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.489+0000 7f047923b700 1 -- start start 2024-01-26T23:49:08.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.489+0000 7f047923b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04740cc1c0 0x7f0474164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:08.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.490+0000 7f047923b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04740cff70 con 0x7f047415d210 2024-01-26T23:49:08.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.490+0000 7f047923b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04740cf400 con 0x7f047415ce40 2024-01-26T23:49:08.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.490+0000 7f047923b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04740cf660 con 0x7f04740cc1c0 2024-01-26T23:49:08.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.490+0000 7f047259c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f047415d210 0x7f04740cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42798/0 (socket says 172.21.15.46:42798) 2024-01-26T23:49:08.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.490+0000 7f047259c700 1 -- 172.21.15.46:0/1336464415 learned_addr learned my addr 172.21.15.46:0/1336464415 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:08.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.490+0000 7f047359e700 1 --2- 172.21.15.46:0/1336464415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04740cc1c0 0x7f0474164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:08.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.490+0000 7f0471d9b700 1 -- 172.21.15.46:0/1336464415 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646367405 0 0) 0x7f04740cff70 con 0x7f047415d210 2024-01-26T23:49:08.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.490+0000 7f0471d9b700 1 -- 172.21.15.46:0/1336464415 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f045c003570 con 0x7f047415d210 2024-01-26T23:49:08.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.490+0000 7f0471d9b700 1 -- 172.21.15.46:0/1336464415 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526917533 0 0) 0x7f04740cf400 con 0x7f047415ce40 2024-01-26T23:49:08.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.490+0000 7f0471d9b700 1 -- 172.21.15.46:0/1336464415 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04740cff70 con 0x7f047415ce40 2024-01-26T23:49:08.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.490+0000 7f047359e700 1 -- 172.21.15.46:0/1336464415 >> v1:172.21.15.106:6789/0 conn(0x7f047415ce40 legacy=0x7f04740d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:08.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.490+0000 7f047359e700 1 -- 172.21.15.46:0/1336464415 >> v1:172.21.15.46:6789/0 conn(0x7f047415d210 legacy=0x7f04740cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:08.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.490+0000 7f047359e700 1 -- 172.21.15.46:0/1336464415 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04740d0500 con 0x7f04740cc1c0 2024-01-26T23:49:08.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.491+0000 7f047359e700 1 --2- 172.21.15.46:0/1336464415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04740cc1c0 0x7f0474164700 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f046c009e90 tx=0x7f046c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d330e0e70f3aa0e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:08.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.491+0000 7f0471d9b700 1 -- 172.21.15.46:0/1336464415 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c00e070 con 0x7f04740cc1c0 2024-01-26T23:49:08.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.491+0000 7f0471d9b700 1 -- 172.21.15.46:0/1336464415 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046c00bb80 con 0x7f04740cc1c0 2024-01-26T23:49:08.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.491+0000 7f0471d9b700 1 -- 172.21.15.46:0/1336464415 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c012430 con 0x7f04740cc1c0 2024-01-26T23:49:08.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.491+0000 7f047923b700 1 -- 172.21.15.46:0/1336464415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04740cc1c0 msgr2=0x7f0474164700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:08.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.491+0000 7f047923b700 1 --2- 172.21.15.46:0/1336464415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04740cc1c0 0x7f0474164700 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f046c009e90 tx=0x7f046c009780 comp rx=0 tx=0).stop 2024-01-26T23:49:08.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.491+0000 7f047923b700 1 -- 172.21.15.46:0/1336464415 shutdown_connections 2024-01-26T23:49:08.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.491+0000 7f047923b700 1 --2- 172.21.15.46:0/1336464415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04740cc1c0 0x7f0474164700 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.491+0000 7f047923b700 1 -- 172.21.15.46:0/1336464415 >> 172.21.15.46:0/1336464415 conn(0x7f047406f840 msgr2=0x7f04740d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:08.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.491+0000 7f047923b700 1 -- 172.21.15.46:0/1336464415 shutdown_connections 2024-01-26T23:49:08.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.491+0000 7f047923b700 1 -- 172.21.15.46:0/1336464415 wait complete. 2024-01-26T23:49:08.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.493+0000 7f047923b700 1 Processor -- start 2024-01-26T23:49:08.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.493+0000 7f047923b700 1 -- start start 2024-01-26T23:49:08.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f047923b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04740cc1c0 0x7f047415e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:08.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f047923b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f047415ce40 0x7f047415e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f047923b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f047415d210 0x7f04740c5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f047923b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0474155840 con 0x7f047415d210 2024-01-26T23:49:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f047923b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04741560e0 con 0x7f047415ce40 2024-01-26T23:49:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f047923b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f047415ee40 con 0x7f04740cc1c0 2024-01-26T23:49:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f0472d9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04740cc1c0 0x7f047415e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f047359e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f047415d210 0x7f04740c5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f0472d9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04740cc1c0 0x7f047415e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:34282/0 (socket says 172.21.15.46:34282) 2024-01-26T23:49:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f0472d9d700 1 -- 172.21.15.46:0/686759024 learned_addr learned my addr 172.21.15.46:0/686759024 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f047259c700 1 --2- 172.21.15.46:0/686759024 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f047415ce40 0x7f047415e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f0472d9d700 1 -- 172.21.15.46:0/686759024 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f047415ce40 msgr2=0x7f047415e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f0472d9d700 1 --2- 172.21.15.46:0/686759024 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f047415ce40 0x7f047415e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f0472d9d700 1 -- 172.21.15.46:0/686759024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f047415d210 msgr2=0x7f04740c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f0472d9d700 1 --2- 172.21.15.46:0/686759024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f047415d210 0x7f04740c5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f0472d9d700 1 -- 172.21.15.46:0/686759024 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046c009a20 con 0x7f04740cc1c0 2024-01-26T23:49:08.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f047359e700 1 --2- 172.21.15.46:0/686759024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f047415d210 0x7f04740c5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:08.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f0472d9d700 1 --2- 172.21.15.46:0/686759024 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04740cc1c0 0x7f047415e0a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f046400c7c0 tx=0x7f0464007dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:08.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.494+0000 7f045affd700 1 -- 172.21.15.46:0/686759024 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0464001570 con 0x7f04740cc1c0 2024-01-26T23:49:08.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.495+0000 7f045affd700 1 -- 172.21.15.46:0/686759024 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04640016f0 con 0x7f04740cc1c0 2024-01-26T23:49:08.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.495+0000 7f047923b700 1 -- 172.21.15.46:0/686759024 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0474155a90 con 0x7f04740cc1c0 2024-01-26T23:49:08.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.495+0000 7f045affd700 1 -- 172.21.15.46:0/686759024 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0464001ab0 con 0x7f04740cc1c0 2024-01-26T23:49:08.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.495+0000 7f047923b700 1 -- 172.21.15.46:0/686759024 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f047415eb20 con 0x7f04740cc1c0 2024-01-26T23:49:08.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.496+0000 7f047923b700 1 -- 172.21.15.46:0/686759024 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04740cd760 con 0x7f04740cc1c0 2024-01-26T23:49:08.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.497+0000 7f045affd700 1 -- 172.21.15.46:0/686759024 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 62) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f046401b3b0 con 0x7f04740cc1c0 2024-01-26T23:49:08.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.497+0000 7f045affd700 1 --2- 172.21.15.46:0/686759024 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0460067170 0x7f0460069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:08.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.497+0000 7f047259c700 1 -- 172.21.15.46:0/686759024 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0460067170 msgr2=0x7f0460069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:08.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.497+0000 7f045affd700 1 -- 172.21.15.46:0/686759024 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6731+0+0 (secure 0 0 0) 0x7f0464084490 con 0x7f04740cc1c0 2024-01-26T23:49:08.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.497+0000 7f047259c700 1 --2- 172.21.15.46:0/686759024 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0460067170 0x7f0460069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:08.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.501+0000 7f045affd700 1 -- 172.21.15.46:0/686759024 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0464051070 con 0x7f04740cc1c0 2024-01-26T23:49:08.601 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:08.600+0000 7f3d5745a1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:49:08.673 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:08.673+0000 7f3d5745a1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:49:08.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.698+0000 7f047259c700 1 -- 172.21.15.46:0/686759024 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0460067170 msgr2=0x7f0460069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:08.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.698+0000 7f047259c700 1 --2- 172.21.15.46:0/686759024 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0460067170 0x7f0460069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:08.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.757+0000 7f047923b700 1 -- 172.21.15.46:0/686759024 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f047404c710 con 0x7f04740cc1c0 2024-01-26T23:49:08.758 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-26T23:49:08.758+0000 7f33ffe86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-26T23:49:08.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.763+0000 7f045affd700 1 -- 172.21.15.46:0/686759024 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v63) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0464054690 con 0x7f04740cc1c0 2024-01-26T23:49:08.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.765+0000 7f047923b700 1 -- 172.21.15.46:0/686759024 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0460067170 msgr2=0x7f0460069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:08.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.765+0000 7f047923b700 1 --2- 172.21.15.46:0/686759024 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0460067170 0x7f0460069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.765+0000 7f047923b700 1 -- 172.21.15.46:0/686759024 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04740cc1c0 msgr2=0x7f047415e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:08.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.765+0000 7f047923b700 1 --2- 172.21.15.46:0/686759024 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04740cc1c0 0x7f047415e0a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f046400c7c0 tx=0x7f0464007dc0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.766+0000 7f047923b700 1 -- 172.21.15.46:0/686759024 shutdown_connections 2024-01-26T23:49:08.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.766+0000 7f047923b700 1 --2- 172.21.15.46:0/686759024 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0460067170 0x7f0460069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.766+0000 7f047923b700 1 --2- 172.21.15.46:0/686759024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f047415d210 0x7f04740c5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.766+0000 7f047923b700 1 --2- 172.21.15.46:0/686759024 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04740cc1c0 0x7f047415e0a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.766+0000 7f047923b700 1 --2- 172.21.15.46:0/686759024 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f047415ce40 0x7f047415e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.766+0000 7f047923b700 1 -- 172.21.15.46:0/686759024 >> 172.21.15.46:0/686759024 conn(0x7f047406f840 msgr2=0x7f04740cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:08.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.766+0000 7f047923b700 1 -- 172.21.15.46:0/686759024 shutdown_connections 2024-01-26T23:49:08.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.766+0000 7f047923b700 1 -- 172.21.15.46:0/686759024 wait complete. 2024-01-26T23:49:08.767 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-26T23:49:08.781 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-26T23:49:08.803 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-26T23:49:08.835 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:08.835+0000 7f3d5745a1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:49:08.922 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:08.921+0000 7f3cd3543700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:08.923 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:08.921+0000 7f3cd3543700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:08.923 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:08.921+0000 7f3cd3543700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:08.923 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:08.921+0000 7f3cd3543700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:08.923 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:08.921+0000 7f3cd3543700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:08.923 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:08.921+0000 7f3cd5d48700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:08.923 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:08.921+0000 7f3cd5d48700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:08.923 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:08.921+0000 7f3cd5d48700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:08.923 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:08.921+0000 7f3cd5d48700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:08.923 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:08.921+0000 7f3cd5d48700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:08.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.929+0000 7f5902ef7700 1 Processor -- start 2024-01-26T23:49:08.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.929+0000 7f5902ef7700 1 -- start start 2024-01-26T23:49:08.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.930+0000 7f5902ef7700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f58fc157490 0x7f58fc0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.930+0000 7f5902ef7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58fc06ddb0 0x7f58fc0cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.930+0000 7f5902ef7700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58fc0cea80 con 0x7f58fc157860 2024-01-26T23:49:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.930+0000 7f5902ef7700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58fc0cdc60 con 0x7f58fc157490 2024-01-26T23:49:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.930+0000 7f5902ef7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc0b2ae0 con 0x7f58fc06ddb0 2024-01-26T23:49:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.930+0000 7f5900c93700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f58fc157490 0x7f58fc0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.930+0000 7f58fbfff700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f58fc157860 0x7f58fc06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:37598/0 (socket says 172.21.15.46:37598) 2024-01-26T23:49:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.930+0000 7f58fbfff700 1 -- 172.21.15.46:0/2620896480 learned_addr learned my addr 172.21.15.46:0/2620896480 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.930+0000 7f5901494700 1 --2- 172.21.15.46:0/2620896480 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58fc06ddb0 0x7f58fc0cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:08.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.931+0000 7f5900c93700 1 -- 172.21.15.46:0/2620896480 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58fc06ddb0 msgr2=0x7f58fc0cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:08.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.931+0000 7f5900c93700 1 --2- 172.21.15.46:0/2620896480 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58fc06ddb0 0x7f58fc0cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.931+0000 7f58fb7fe700 1 -- 172.21.15.46:0/2620896480 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667639668 0 0) 0x7f58fc0cea80 con 0x7f58fc157860 2024-01-26T23:49:08.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.931+0000 7f5900c93700 1 -- 172.21.15.46:0/2620896480 >> v1:172.21.15.106:6789/0 conn(0x7f58fc157860 legacy=0x7f58fc06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:08.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.931+0000 7f5900c93700 1 -- 172.21.15.46:0/2620896480 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58fc0cde70 con 0x7f58fc157490 2024-01-26T23:49:08.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.931+0000 7f5901494700 1 --2- 172.21.15.46:0/2620896480 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58fc06ddb0 0x7f58fc0cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:08.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.931+0000 7f5900c93700 1 --2- 172.21.15.46:0/2620896480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58fc157490 0x7f58fc0cc600 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f58f00098e0 tx=0x7f58f00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58e74ebb1027d4d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:08.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.931+0000 7f58fb7fe700 1 -- 172.21.15.46:0/2620896480 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f000b640 con 0x7f58fc157490 2024-01-26T23:49:08.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.931+0000 7f58fb7fe700 1 -- 172.21.15.46:0/2620896480 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f000b7a0 con 0x7f58fc157490 2024-01-26T23:49:08.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.931+0000 7f58fb7fe700 1 -- 172.21.15.46:0/2620896480 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f000bb40 con 0x7f58fc157490 2024-01-26T23:49:08.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.932+0000 7f5902ef7700 1 -- 172.21.15.46:0/2620896480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58fc157490 msgr2=0x7f58fc0cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:08.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.932+0000 7f5902ef7700 1 --2- 172.21.15.46:0/2620896480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58fc157490 0x7f58fc0cc600 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f58f00098e0 tx=0x7f58f00091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.932+0000 7f5902ef7700 1 -- 172.21.15.46:0/2620896480 shutdown_connections 2024-01-26T23:49:08.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.932+0000 7f5902ef7700 1 --2- 172.21.15.46:0/2620896480 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58fc06ddb0 0x7f58fc0cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.932+0000 7f5902ef7700 1 --2- 172.21.15.46:0/2620896480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58fc157490 0x7f58fc0cc600 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.932+0000 7f5902ef7700 1 -- 172.21.15.46:0/2620896480 >> 172.21.15.46:0/2620896480 conn(0x7f58fc067930 msgr2=0x7f58fc067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:08.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.932+0000 7f5902ef7700 1 -- 172.21.15.46:0/2620896480 shutdown_connections 2024-01-26T23:49:08.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.932+0000 7f5902ef7700 1 -- 172.21.15.46:0/2620896480 wait complete. 2024-01-26T23:49:08.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.933+0000 7f5902ef7700 1 Processor -- start 2024-01-26T23:49:08.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.933+0000 7f5902ef7700 1 -- start start 2024-01-26T23:49:08.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.934+0000 7f5902ef7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58fc06ddb0 0x7f58fc1678d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:08.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.934+0000 7f5902ef7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58fc157490 0x7f58fc1510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:08.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.934+0000 7f5902ef7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58fc157860 0x7f58fc0c3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:08.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.934+0000 7f5902ef7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc15a9e0 con 0x7f58fc06ddb0 2024-01-26T23:49:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.934+0000 7f5902ef7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc158be0 con 0x7f58fc157860 2024-01-26T23:49:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.934+0000 7f5902ef7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc167e10 con 0x7f58fc157490 2024-01-26T23:49:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.934+0000 7f5900c93700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58fc06ddb0 0x7f58fc1678d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.934+0000 7f5900c93700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58fc06ddb0 0x7f58fc1678d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52388/0 (socket says 172.21.15.46:52388) 2024-01-26T23:49:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.934+0000 7f5900c93700 1 -- 172.21.15.46:0/1404830638 learned_addr learned my addr 172.21.15.46:0/1404830638 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.935+0000 7f58fbfff700 1 --2- 172.21.15.46:0/1404830638 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58fc157490 0x7f58fc1510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.935+0000 7f5901494700 1 --2- 172.21.15.46:0/1404830638 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58fc157860 0x7f58fc0c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.935+0000 7f5900c93700 1 -- 172.21.15.46:0/1404830638 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58fc157490 msgr2=0x7f58fc1510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.935+0000 7f5900c93700 1 --2- 172.21.15.46:0/1404830638 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58fc157490 0x7f58fc1510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.935+0000 7f5900c93700 1 -- 172.21.15.46:0/1404830638 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58fc157860 msgr2=0x7f58fc0c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.935+0000 7f5900c93700 1 --2- 172.21.15.46:0/1404830638 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58fc157860 0x7f58fc0c3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.935+0000 7f5900c93700 1 -- 172.21.15.46:0/1404830638 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f4009a20 con 0x7f58fc06ddb0 2024-01-26T23:49:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.935+0000 7f5901494700 1 --2- 172.21.15.46:0/1404830638 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58fc157860 0x7f58fc0c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:08.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.935+0000 7f5900c93700 1 --2- 172.21.15.46:0/1404830638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58fc06ddb0 0x7f58fc1678d0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f58fc06d060 tx=0x7f58f0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:08.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.935+0000 7f58f8ff9700 1 -- 172.21.15.46:0/1404830638 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0000d10 con 0x7f58fc06ddb0 2024-01-26T23:49:08.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.935+0000 7f58f8ff9700 1 -- 172.21.15.46:0/1404830638 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f000bb20 con 0x7f58fc06ddb0 2024-01-26T23:49:08.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.935+0000 7f5902ef7700 1 -- 172.21.15.46:0/1404830638 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f0009420 con 0x7f58fc06ddb0 2024-01-26T23:49:08.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.935+0000 7f58f8ff9700 1 -- 172.21.15.46:0/1404830638 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0011650 con 0x7f58fc06ddb0 2024-01-26T23:49:08.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.935+0000 7f5902ef7700 1 -- 172.21.15.46:0/1404830638 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58fc158080 con 0x7f58fc06ddb0 2024-01-26T23:49:08.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.936+0000 7f5902ef7700 1 -- 172.21.15.46:0/1404830638 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58fc04c710 con 0x7f58fc06ddb0 2024-01-26T23:49:08.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.937+0000 7f58f8ff9700 1 -- 172.21.15.46:0/1404830638 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 63) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f58f00119c0 con 0x7f58fc06ddb0 2024-01-26T23:49:08.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.938+0000 7f58f8ff9700 1 -- 172.21.15.46:0/1404830638 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 7133+0+0 (secure 0 0 0) 0x7f58f0085880 con 0x7f58fc06ddb0 2024-01-26T23:49:08.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:08.942+0000 7f58f8ff9700 1 -- 172.21.15.46:0/1404830638 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f58f00529a0 con 0x7f58fc06ddb0 2024-01-26T23:49:09.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.197+0000 7f5902ef7700 1 -- 172.21.15.46:0/1404830638 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f58fc0b14d0 con 0x7f58fc06ddb0 2024-01-26T23:49:09.235 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:09.235+0000 7f30bcb891c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:49:09.488 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:09.488+0000 7f30bcb891c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:49:09.571 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:09.571+0000 7f30bcb891c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:49:09.749 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:09.748+0000 7f30bcb891c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:49:09.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.762+0000 7f58f8ff9700 1 -- 172.21.15.46:0/1404830638 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v64) v1 ==== 104+0+0 (secure 0 0 0) 0x7f58f0055fc0 con 0x7f58fc06ddb0 2024-01-26T23:49:09.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.765+0000 7f5902ef7700 1 -- 172.21.15.46:0/1404830638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58fc06ddb0 msgr2=0x7f58fc1678d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:09.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.765+0000 7f5902ef7700 1 --2- 172.21.15.46:0/1404830638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58fc06ddb0 0x7f58fc1678d0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f58fc06d060 tx=0x7f58f0017000 comp rx=0 tx=0).stop 2024-01-26T23:49:09.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.765+0000 7f5902ef7700 1 -- 172.21.15.46:0/1404830638 shutdown_connections 2024-01-26T23:49:09.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.765+0000 7f5902ef7700 1 --2- 172.21.15.46:0/1404830638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58fc06ddb0 0x7f58fc1678d0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:09.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.765+0000 7f5902ef7700 1 --2- 172.21.15.46:0/1404830638 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58fc157860 0x7f58fc0c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:09.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.765+0000 7f5902ef7700 1 --2- 172.21.15.46:0/1404830638 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58fc157490 0x7f58fc1510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:09.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.765+0000 7f5902ef7700 1 -- 172.21.15.46:0/1404830638 >> 172.21.15.46:0/1404830638 conn(0x7f58fc067930 msgr2=0x7f58fc154e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:09.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.765+0000 7f5902ef7700 1 -- 172.21.15.46:0/1404830638 shutdown_connections 2024-01-26T23:49:09.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.765+0000 7f5902ef7700 1 -- 172.21.15.46:0/1404830638 wait complete. 2024-01-26T23:49:09.779 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-26T23:49:09.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.962+0000 7f6ce5437700 1 Processor -- start 2024-01-26T23:49:09.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.963+0000 7f6ce5437700 1 -- start start 2024-01-26T23:49:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.963+0000 7f6ce5437700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6ce015ce40 0x7f6ce00d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.963+0000 7f6ce5437700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6ce015d210 0x7f6ce00cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.963+0000 7f6ce5437700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce00cff70 con 0x7f6ce00d1390 2024-01-26T23:49:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.963+0000 7f6ce5437700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce00cf400 con 0x7f6ce015ce40 2024-01-26T23:49:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.963+0000 7f6ce5437700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce00cf660 con 0x7f6ce015d210 2024-01-26T23:49:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.964+0000 7f6cdeffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6ce015ce40 0x7f6ce00d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.964+0000 7f6cdeffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6ce015ce40 0x7f6ce00d0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52398/0 (socket says 172.21.15.46:52398) 2024-01-26T23:49:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.964+0000 7f6cdeffd700 1 -- 172.21.15.46:0/802425784 learned_addr learned my addr 172.21.15.46:0/802425784 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.964+0000 7f6cde7fc700 1 --2- 172.21.15.46:0/802425784 >> v2:172.21.15.106:3300/0 conn(0x7f6ce015d210 0x7f6ce00cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.964+0000 7f6cdeffd700 1 -- 172.21.15.46:0/802425784 >> v2:172.21.15.106:3300/0 conn(0x7f6ce015d210 msgr2=0x7f6ce00cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.964+0000 7f6cdeffd700 1 --2- 172.21.15.46:0/802425784 >> v2:172.21.15.106:3300/0 conn(0x7f6ce015d210 0x7f6ce00cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.964+0000 7f6cdeffd700 1 -- 172.21.15.46:0/802425784 >> v1:172.21.15.106:6789/0 conn(0x7f6ce00d1390 legacy=0x7f6ce00ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.964+0000 7f6cdeffd700 1 -- 172.21.15.46:0/802425784 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce00d0500 con 0x7f6ce015ce40 2024-01-26T23:49:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.964+0000 7f6cdeffd700 1 --2- 172.21.15.46:0/802425784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce015ce40 0x7f6ce00d0e50 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00895f0 tx=0x7f6cd0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07b2193c5c9d71f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.964+0000 7f6cddffb700 1 -- 172.21.15.46:0/802425784 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd000b860 con 0x7f6ce015ce40 2024-01-26T23:49:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.964+0000 7f6cddffb700 1 -- 172.21.15.46:0/802425784 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd000b9c0 con 0x7f6ce015ce40 2024-01-26T23:49:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.964+0000 7f6cddffb700 1 -- 172.21.15.46:0/802425784 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd000bd60 con 0x7f6ce015ce40 2024-01-26T23:49:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.965+0000 7f6ce5437700 1 -- 172.21.15.46:0/802425784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce015ce40 msgr2=0x7f6ce00d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.965+0000 7f6ce5437700 1 --2- 172.21.15.46:0/802425784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce015ce40 0x7f6ce00d0e50 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00895f0 tx=0x7f6cd0009250 comp rx=0 tx=0).stop 2024-01-26T23:49:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.965+0000 7f6ce5437700 1 -- 172.21.15.46:0/802425784 shutdown_connections 2024-01-26T23:49:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.965+0000 7f6ce5437700 1 --2- 172.21.15.46:0/802425784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce015ce40 0x7f6ce00d0e50 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.965+0000 7f6ce5437700 1 --2- 172.21.15.46:0/802425784 >> v2:172.21.15.106:3300/0 conn(0x7f6ce015d210 0x7f6ce00cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.965+0000 7f6ce5437700 1 -- 172.21.15.46:0/802425784 >> 172.21.15.46:0/802425784 conn(0x7f6ce006f840 msgr2=0x7f6ce00d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.965+0000 7f6ce5437700 1 -- 172.21.15.46:0/802425784 shutdown_connections 2024-01-26T23:49:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.965+0000 7f6ce5437700 1 -- 172.21.15.46:0/802425784 wait complete. 2024-01-26T23:49:09.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.967+0000 7f6ce5437700 1 Processor -- start 2024-01-26T23:49:09.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.967+0000 7f6ce5437700 1 -- start start 2024-01-26T23:49:09.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.967+0000 7f6ce5437700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce00d1390 0x7f6ce009a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:09.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.967+0000 7f6ce5437700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ce015ce40 0x7f6ce015abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:09.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.967+0000 7f6ce5437700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ce015d210 0x7f6ce015d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:09.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.967+0000 7f6ce5437700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce009b710 con 0x7f6ce00d1390 2024-01-26T23:49:09.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.967+0000 7f6ce5437700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce0169fb0 con 0x7f6ce015d210 2024-01-26T23:49:09.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.967+0000 7f6ce5437700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce009d2d0 con 0x7f6ce015ce40 2024-01-26T23:49:09.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.967+0000 7f6cdeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce00d1390 0x7f6ce009a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:09.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.967+0000 7f6cdeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce00d1390 0x7f6ce009a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52408/0 (socket says 172.21.15.46:52408) 2024-01-26T23:49:09.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.967+0000 7f6cdeffd700 1 -- 172.21.15.46:0/3174942113 learned_addr learned my addr 172.21.15.46:0/3174942113 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:09.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.968+0000 7f6cde7fc700 1 --2- 172.21.15.46:0/3174942113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ce015ce40 0x7f6ce015abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:09.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.968+0000 7f6cdf7fe700 1 --2- 172.21.15.46:0/3174942113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ce015d210 0x7f6ce015d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:09.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.968+0000 7f6cdeffd700 1 -- 172.21.15.46:0/3174942113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ce015ce40 msgr2=0x7f6ce015abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:09.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.968+0000 7f6cdeffd700 1 --2- 172.21.15.46:0/3174942113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ce015ce40 0x7f6ce015abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:09.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.968+0000 7f6cdeffd700 1 -- 172.21.15.46:0/3174942113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ce015d210 msgr2=0x7f6ce015d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:09.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.968+0000 7f6cdeffd700 1 --2- 172.21.15.46:0/3174942113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ce015d210 0x7f6ce015d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:09.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.968+0000 7f6cdeffd700 1 -- 172.21.15.46:0/3174942113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd00094f0 con 0x7f6ce00d1390 2024-01-26T23:49:09.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.968+0000 7f6cdeffd700 1 --2- 172.21.15.46:0/3174942113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce00d1390 0x7f6ce009a770 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0088ec0 tx=0x7f6cd0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:09.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.968+0000 7f6cc6ffd700 1 -- 172.21.15.46:0/3174942113 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0011e00 con 0x7f6ce00d1390 2024-01-26T23:49:09.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.968+0000 7f6cc6ffd700 1 -- 172.21.15.46:0/3174942113 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd0024440 con 0x7f6ce00d1390 2024-01-26T23:49:09.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.968+0000 7f6ce5437700 1 -- 172.21.15.46:0/3174942113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce009ae70 con 0x7f6ce00d1390 2024-01-26T23:49:09.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.968+0000 7f6cc6ffd700 1 -- 172.21.15.46:0/3174942113 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0024800 con 0x7f6ce00d1390 2024-01-26T23:49:09.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.968+0000 7f6ce5437700 1 -- 172.21.15.46:0/3174942113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce009b430 con 0x7f6ce00d1390 2024-01-26T23:49:09.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.969+0000 7f6ce5437700 1 -- 172.21.15.46:0/3174942113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce0169d10 con 0x7f6ce00d1390 2024-01-26T23:49:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.970+0000 7f6cc6ffd700 1 -- 172.21.15.46:0/3174942113 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 64) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f6cd002d440 con 0x7f6ce00d1390 2024-01-26T23:49:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.970+0000 7f6cc6ffd700 1 --2- 172.21.15.46:0/3174942113 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6ccc066d80 0x7f6ccc069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.970+0000 7f6cc6ffd700 1 -- 172.21.15.46:0/3174942113 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 7133+0+0 (secure 0 0 0) 0x7f6cd0086200 con 0x7f6ce00d1390 2024-01-26T23:49:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.970+0000 7f6cde7fc700 1 --2- 172.21.15.46:0/3174942113 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6ccc066d80 0x7f6ccc069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.971+0000 7f6cde7fc700 1 --2- 172.21.15.46:0/3174942113 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6ccc066d80 0x7f6ccc069230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4006fd0 tx=0x7f6cd4008040 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:09.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:09.975+0000 7f6cc6ffd700 1 -- 172.21.15.46:0/3174942113 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6ce0169d10 con 0x7f6ce00d1390 2024-01-26T23:49:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:10.182+0000 7f6ce5437700 1 -- 172.21.15.46:0/3174942113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6ce0169d10 con 0x7f6ce00d1390 2024-01-26T23:49:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:10.889+0000 7f6cc6ffd700 1 -- 172.21.15.46:0/3174942113 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 65) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f6cd0053560 con 0x7f6ce00d1390 2024-01-26T23:49:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:10.889+0000 7f6cc6ffd700 1 -- 172.21.15.46:0/3174942113 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v135) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6cd004f290 con 0x7f6ce00d1390 2024-01-26T23:49:10.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:10.893+0000 7f6ce5437700 1 -- 172.21.15.46:0/3174942113 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6ccc066d80 msgr2=0x7f6ccc069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:10.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:10.893+0000 7f6ce5437700 1 --2- 172.21.15.46:0/3174942113 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6ccc066d80 0x7f6ccc069230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4006fd0 tx=0x7f6cd4008040 comp rx=0 tx=0).stop 2024-01-26T23:49:10.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:10.893+0000 7f6ce5437700 1 -- 172.21.15.46:0/3174942113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce00d1390 msgr2=0x7f6ce009a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:10.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:10.893+0000 7f6ce5437700 1 --2- 172.21.15.46:0/3174942113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce00d1390 0x7f6ce009a770 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0088ec0 tx=0x7f6cd0017000 comp rx=0 tx=0).stop 2024-01-26T23:49:10.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:10.894+0000 7f6ce5437700 1 -- 172.21.15.46:0/3174942113 shutdown_connections 2024-01-26T23:49:10.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:10.894+0000 7f6ce5437700 1 --2- 172.21.15.46:0/3174942113 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6ccc066d80 0x7f6ccc069230 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:10.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:10.894+0000 7f6ce5437700 1 --2- 172.21.15.46:0/3174942113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce00d1390 0x7f6ce009a770 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:10.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:10.894+0000 7f6ce5437700 1 --2- 172.21.15.46:0/3174942113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ce015d210 0x7f6ce015d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:10.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:10.894+0000 7f6ce5437700 1 --2- 172.21.15.46:0/3174942113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ce015ce40 0x7f6ce015abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:10.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:10.894+0000 7f6ce5437700 1 -- 172.21.15.46:0/3174942113 >> 172.21.15.46:0/3174942113 conn(0x7f6ce006f840 msgr2=0x7f6ce00b34f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:10.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:10.894+0000 7f6ce5437700 1 -- 172.21.15.46:0/3174942113 shutdown_connections 2024-01-26T23:49:10.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:10.894+0000 7f6ce5437700 1 -- 172.21.15.46:0/3174942113 wait complete. 2024-01-26T23:49:10.895 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-01-26T23:49:10.910 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-26T23:49:11.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.052+0000 7f475b699700 1 Processor -- start 2024-01-26T23:49:11.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.052+0000 7f475b699700 1 -- start start 2024-01-26T23:49:11.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.053+0000 7f475b699700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47540cd7a0 0x7f47540cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:11.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.053+0000 7f475b699700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f47540c66d0 0x7f47540cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:11.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.053+0000 7f475b699700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47540cccd0 con 0x7f4754159830 2024-01-26T23:49:11.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.053+0000 7f475b699700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47540cc100 con 0x7f47540c66d0 2024-01-26T23:49:11.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.053+0000 7f475b699700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47540cc3a0 con 0x7f47540cd7a0 2024-01-26T23:49:11.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.054+0000 7f4758c34700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47540cd7a0 0x7f47540cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:11.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.054+0000 7f4759c36700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f47540c66d0 0x7f47540cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:11.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.054+0000 7f4759435700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f4754159830 0x7f47540cd090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:46334/0 (socket says 172.21.15.46:46334) 2024-01-26T23:49:11.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.054+0000 7f4759435700 1 -- 172.21.15.46:0/3604820313 learned_addr learned my addr 172.21.15.46:0/3604820313 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:11.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.054+0000 7f4759c36700 1 -- 172.21.15.46:0/3604820313 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47540cd7a0 msgr2=0x7f47540cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:11.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.054+0000 7f4759c36700 1 --2- 172.21.15.46:0/3604820313 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47540cd7a0 0x7f47540cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:11.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.054+0000 7f4759c36700 1 -- 172.21.15.46:0/3604820313 >> v1:172.21.15.106:6789/0 conn(0x7f4754159830 legacy=0x7f47540cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:11.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.054+0000 7f4759c36700 1 -- 172.21.15.46:0/3604820313 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47540cc680 con 0x7f47540c66d0 2024-01-26T23:49:11.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.054+0000 7f4758c34700 1 --2- 172.21.15.46:0/3604820313 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47540cd7a0 0x7f47540cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:11.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.054+0000 7f474bfff700 1 -- 172.21.15.46:0/3604820313 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648409077 0 0) 0x7f47540cccd0 con 0x7f4754159830 2024-01-26T23:49:11.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.054+0000 7f4759c36700 1 --2- 172.21.15.46:0/3604820313 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47540c66d0 0x7f47540cf3c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f474c009e60 tx=0x7f474c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0849d10c2b890b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:11.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.054+0000 7f474bfff700 1 -- 172.21.15.46:0/3604820313 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c00e070 con 0x7f47540c66d0 2024-01-26T23:49:11.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.054+0000 7f474bfff700 1 -- 172.21.15.46:0/3604820313 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f474c00bb60 con 0x7f47540c66d0 2024-01-26T23:49:11.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.054+0000 7f474bfff700 1 -- 172.21.15.46:0/3604820313 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c012430 con 0x7f47540c66d0 2024-01-26T23:49:11.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.055+0000 7f475b699700 1 -- 172.21.15.46:0/3604820313 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47540c66d0 msgr2=0x7f47540cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:11.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.055+0000 7f475b699700 1 --2- 172.21.15.46:0/3604820313 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47540c66d0 0x7f47540cf3c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f474c009e60 tx=0x7f474c009780 comp rx=0 tx=0).stop 2024-01-26T23:49:11.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.055+0000 7f475b699700 1 -- 172.21.15.46:0/3604820313 shutdown_connections 2024-01-26T23:49:11.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.055+0000 7f475b699700 1 --2- 172.21.15.46:0/3604820313 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47540c66d0 0x7f47540cf3c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:11.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.055+0000 7f475b699700 1 --2- 172.21.15.46:0/3604820313 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47540cd7a0 0x7f47540cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:11.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.055+0000 7f475b699700 1 -- 172.21.15.46:0/3604820313 >> 172.21.15.46:0/3604820313 conn(0x7f4754099a30 msgr2=0x7f475406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:11.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.055+0000 7f475b699700 1 -- 172.21.15.46:0/3604820313 shutdown_connections 2024-01-26T23:49:11.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.055+0000 7f475b699700 1 -- 172.21.15.46:0/3604820313 wait complete. 2024-01-26T23:49:11.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.056+0000 7f475b699700 1 Processor -- start 2024-01-26T23:49:11.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.056+0000 7f475b699700 1 -- start start 2024-01-26T23:49:11.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.057+0000 7f475b699700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47540c66d0 0x7f475413fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:11.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.057+0000 7f475b699700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47540cd7a0 0x7f4754153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:11.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.057+0000 7f475b699700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4754159830 0x7f4754154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:11.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.057+0000 7f475b699700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4754153850 con 0x7f4754159830 2024-01-26T23:49:11.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.057+0000 7f475b699700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47540b3ef0 con 0x7f47540cd7a0 2024-01-26T23:49:11.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.057+0000 7f475b699700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47540b48e0 con 0x7f47540c66d0 2024-01-26T23:49:11.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.057+0000 7f4759c36700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4754159830 0x7f4754154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:11.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.057+0000 7f4759c36700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4754159830 0x7f4754154260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50682/0 (socket says 172.21.15.46:50682) 2024-01-26T23:49:11.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.057+0000 7f4759c36700 1 -- 172.21.15.46:0/2082802237 learned_addr learned my addr 172.21.15.46:0/2082802237 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:11.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.057+0000 7f4758c34700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47540cd7a0 0x7f4754153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:11.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.057+0000 7f4759435700 1 --2- 172.21.15.46:0/2082802237 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47540c66d0 0x7f475413fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:11.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.058+0000 7f4759c36700 1 -- 172.21.15.46:0/2082802237 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47540c66d0 msgr2=0x7f475413fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:11.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.058+0000 7f4759c36700 1 --2- 172.21.15.46:0/2082802237 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47540c66d0 0x7f475413fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:11.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.058+0000 7f4759c36700 1 -- 172.21.15.46:0/2082802237 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47540cd7a0 msgr2=0x7f4754153d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:11.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.058+0000 7f4759c36700 1 --2- 172.21.15.46:0/2082802237 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47540cd7a0 0x7f4754153d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:11.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.058+0000 7f4759c36700 1 -- 172.21.15.46:0/2082802237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4744009420 con 0x7f4754159830 2024-01-26T23:49:11.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.058+0000 7f4758c34700 1 --2- 172.21.15.46:0/2082802237 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47540cd7a0 0x7f4754153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:11.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.058+0000 7f4759c36700 1 --2- 172.21.15.46:0/2082802237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4754159830 0x7f4754154260 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f474c00b6c0 tx=0x7f474c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:11.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.058+0000 7f47497fa700 1 -- 172.21.15.46:0/2082802237 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c012e90 con 0x7f4754159830 2024-01-26T23:49:11.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.058+0000 7f47497fa700 1 -- 172.21.15.46:0/2082802237 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f474c027470 con 0x7f4754159830 2024-01-26T23:49:11.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.058+0000 7f475b699700 1 -- 172.21.15.46:0/2082802237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f474c009a20 con 0x7f4754159830 2024-01-26T23:49:11.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.058+0000 7f47497fa700 1 -- 172.21.15.46:0/2082802237 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c027830 con 0x7f4754159830 2024-01-26T23:49:11.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.058+0000 7f475b699700 1 -- 172.21.15.46:0/2082802237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f475415eb40 con 0x7f4754159830 2024-01-26T23:49:11.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.059+0000 7f475b699700 1 -- 172.21.15.46:0/2082802237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f475404c710 con 0x7f4754159830 2024-01-26T23:49:11.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.060+0000 7f47497fa700 1 -- 172.21.15.46:0/2082802237 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 65) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f474c024070 con 0x7f4754159830 2024-01-26T23:49:11.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.060+0000 7f47497fa700 1 --2- 172.21.15.46:0/2082802237 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4738066ea0 0x7f4738069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:11.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.060+0000 7f47497fa700 1 -- 172.21.15.46:0/2082802237 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6695+0+0 (secure 0 0 0) 0x7f474c086f00 con 0x7f4754159830 2024-01-26T23:49:11.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.061+0000 7f4759435700 1 --2- 172.21.15.46:0/2082802237 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4738066ea0 0x7f4738069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:11.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.061+0000 7f4759435700 1 --2- 172.21.15.46:0/2082802237 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4738066ea0 0x7f4738069350 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f47500019b0 tx=0x7f4750009040 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:11.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.065+0000 7f47497fa700 1 -- 172.21.15.46:0/2082802237 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f474c053e70 con 0x7f4754159830 2024-01-26T23:49:11.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.269+0000 7f475b699700 1 -- 172.21.15.46:0/2082802237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f47540cc960 con 0x7f4754159830 2024-01-26T23:49:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.689+0000 7f47497fa700 1 -- 172.21.15.46:0/2082802237 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v136) v1 ==== 169+0+0 (secure 0 0 0) 0x7f474c02c020 con 0x7f4754159830 2024-01-26T23:49:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.693+0000 7f475b699700 1 -- 172.21.15.46:0/2082802237 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4738066ea0 msgr2=0x7f4738069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.693+0000 7f475b699700 1 --2- 172.21.15.46:0/2082802237 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4738066ea0 0x7f4738069350 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f47500019b0 tx=0x7f4750009040 comp rx=0 tx=0).stop 2024-01-26T23:49:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.693+0000 7f475b699700 1 -- 172.21.15.46:0/2082802237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4754159830 msgr2=0x7f4754154260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.693+0000 7f475b699700 1 --2- 172.21.15.46:0/2082802237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4754159830 0x7f4754154260 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f474c00b6c0 tx=0x7f474c019000 comp rx=0 tx=0).stop 2024-01-26T23:49:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.693+0000 7f475b699700 1 -- 172.21.15.46:0/2082802237 shutdown_connections 2024-01-26T23:49:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.693+0000 7f475b699700 1 --2- 172.21.15.46:0/2082802237 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4738066ea0 0x7f4738069350 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.693+0000 7f475b699700 1 --2- 172.21.15.46:0/2082802237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4754159830 0x7f4754154260 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.693+0000 7f475b699700 1 --2- 172.21.15.46:0/2082802237 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47540c66d0 0x7f475413fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.693+0000 7f475b699700 1 --2- 172.21.15.46:0/2082802237 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47540cd7a0 0x7f4754153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.693+0000 7f475b699700 1 -- 172.21.15.46:0/2082802237 >> 172.21.15.46:0/2082802237 conn(0x7f4754099a30 msgr2=0x7f47541551d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.694+0000 7f475b699700 1 -- 172.21.15.46:0/2082802237 shutdown_connections 2024-01-26T23:49:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.694+0000 7f475b699700 1 -- 172.21.15.46:0/2082802237 wait complete. 2024-01-26T23:49:11.700 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-01-26T23:49:11.713 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-26T23:49:11.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.852+0000 7fd163151700 1 Processor -- start 2024-01-26T23:49:11.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.852+0000 7fd163151700 1 -- start start 2024-01-26T23:49:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.853+0000 7fd163151700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd15c153c40 0x7fd15c06d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.853+0000 7fd163151700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd15c154010 0x7fd15c06dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.853+0000 7fd163151700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd15c0b1e80 0x7fd15c06e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.853+0000 7fd163151700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd15c0b1d20 con 0x7fd15c0b1e80 2024-01-26T23:49:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.853+0000 7fd163151700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd15c15ad60 con 0x7fd15c153c40 2024-01-26T23:49:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.853+0000 7fd163151700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c15a150 con 0x7fd15c154010 2024-01-26T23:49:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.853+0000 7fd1616ee700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd15c0b1e80 0x7fd15c06e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.853+0000 7fd1616ee700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd15c0b1e80 0x7fd15c06e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50686/0 (socket says 172.21.15.46:50686) 2024-01-26T23:49:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.853+0000 7fd1616ee700 1 -- 172.21.15.46:0/682717188 learned_addr learned my addr 172.21.15.46:0/682717188 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.853+0000 7fd15bfff700 1 --2- 172.21.15.46:0/682717188 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd15c154010 0x7fd15c06dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.853+0000 7fd160eed700 1 --2- 172.21.15.46:0/682717188 >> v2:172.21.15.106:3300/0 conn(0x7fd15c153c40 0x7fd15c06d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.853+0000 7fd1616ee700 1 -- 172.21.15.46:0/682717188 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd15c154010 msgr2=0x7fd15c06dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.853+0000 7fd1616ee700 1 --2- 172.21.15.46:0/682717188 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd15c154010 0x7fd15c06dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.853+0000 7fd1616ee700 1 -- 172.21.15.46:0/682717188 >> v2:172.21.15.106:3300/0 conn(0x7fd15c153c40 msgr2=0x7fd15c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.853+0000 7fd1616ee700 1 --2- 172.21.15.46:0/682717188 >> v2:172.21.15.106:3300/0 conn(0x7fd15c153c40 0x7fd15c06d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.853+0000 7fd1616ee700 1 -- 172.21.15.46:0/682717188 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c15a430 con 0x7fd15c0b1e80 2024-01-26T23:49:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.854+0000 7fd1616ee700 1 --2- 172.21.15.46:0/682717188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd15c0b1e80 0x7fd15c06e120 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fd15c06f8e0 tx=0x7fd154009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caca339316da81fe server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.854+0000 7fd15b7fe700 1 -- 172.21.15.46:0/682717188 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15400e070 con 0x7fd15c0b1e80 2024-01-26T23:49:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.854+0000 7fd15b7fe700 1 -- 172.21.15.46:0/682717188 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15400ba50 con 0x7fd15c0b1e80 2024-01-26T23:49:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.854+0000 7fd15b7fe700 1 -- 172.21.15.46:0/682717188 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1540092e0 con 0x7fd15c0b1e80 2024-01-26T23:49:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.854+0000 7fd163151700 1 -- 172.21.15.46:0/682717188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd15c0b1e80 msgr2=0x7fd15c06e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.854+0000 7fd163151700 1 --2- 172.21.15.46:0/682717188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd15c0b1e80 0x7fd15c06e120 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fd15c06f8e0 tx=0x7fd154009840 comp rx=0 tx=0).stop 2024-01-26T23:49:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.854+0000 7fd163151700 1 -- 172.21.15.46:0/682717188 shutdown_connections 2024-01-26T23:49:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.854+0000 7fd163151700 1 --2- 172.21.15.46:0/682717188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd15c0b1e80 0x7fd15c06e120 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.854+0000 7fd163151700 1 --2- 172.21.15.46:0/682717188 >> v2:172.21.15.106:3300/0 conn(0x7fd15c153c40 0x7fd15c06d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.854+0000 7fd163151700 1 --2- 172.21.15.46:0/682717188 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd15c154010 0x7fd15c06dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.854+0000 7fd163151700 1 -- 172.21.15.46:0/682717188 >> 172.21.15.46:0/682717188 conn(0x7fd15c067930 msgr2=0x7fd15c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.855+0000 7fd163151700 1 -- 172.21.15.46:0/682717188 shutdown_connections 2024-01-26T23:49:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.855+0000 7fd163151700 1 -- 172.21.15.46:0/682717188 wait complete. 2024-01-26T23:49:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.856+0000 7fd163151700 1 Processor -- start 2024-01-26T23:49:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.856+0000 7fd163151700 1 -- start start 2024-01-26T23:49:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.857+0000 7fd163151700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd15c0b1e80 0x7fd15c150d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.857+0000 7fd163151700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd15c153c40 0x7fd15c15c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.857+0000 7fd163151700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd15c154010 0x7fd15c0af8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.857+0000 7fd163151700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c151290 con 0x7fd15c153c40 2024-01-26T23:49:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.857+0000 7fd163151700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c15d310 con 0x7fd15c154010 2024-01-26T23:49:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.857+0000 7fd163151700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c0cc680 con 0x7fd15c0b1e80 2024-01-26T23:49:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.858+0000 7fd15bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd15c153c40 0x7fd15c15c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.858+0000 7fd15bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd15c153c40 0x7fd15c15c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50698/0 (socket says 172.21.15.46:50698) 2024-01-26T23:49:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.858+0000 7fd15bfff700 1 -- 172.21.15.46:0/1399724113 learned_addr learned my addr 172.21.15.46:0/1399724113 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.858+0000 7fd160eed700 1 --2- 172.21.15.46:0/1399724113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd15c0b1e80 0x7fd15c150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.858+0000 7fd1616ee700 1 --2- 172.21.15.46:0/1399724113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd15c154010 0x7fd15c0af8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.858+0000 7fd15bfff700 1 -- 172.21.15.46:0/1399724113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd15c0b1e80 msgr2=0x7fd15c150d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.858+0000 7fd15bfff700 1 --2- 172.21.15.46:0/1399724113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd15c0b1e80 0x7fd15c150d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.858+0000 7fd15bfff700 1 -- 172.21.15.46:0/1399724113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd15c154010 msgr2=0x7fd15c0af8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.858+0000 7fd15bfff700 1 --2- 172.21.15.46:0/1399724113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd15c154010 0x7fd15c0af8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.858+0000 7fd15bfff700 1 -- 172.21.15.46:0/1399724113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd154009ae0 con 0x7fd15c153c40 2024-01-26T23:49:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.858+0000 7fd15bfff700 1 --2- 172.21.15.46:0/1399724113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd15c153c40 0x7fd15c15c650 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fd14800d860 tx=0x7fd148008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.858+0000 7fd158ff9700 1 -- 172.21.15.46:0/1399724113 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1480125f0 con 0x7fd15c153c40 2024-01-26T23:49:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.858+0000 7fd158ff9700 1 -- 172.21.15.46:0/1399724113 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd148012750 con 0x7fd15c153c40 2024-01-26T23:49:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.858+0000 7fd163151700 1 -- 172.21.15.46:0/1399724113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15c15d030 con 0x7fd15c153c40 2024-01-26T23:49:11.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.858+0000 7fd158ff9700 1 -- 172.21.15.46:0/1399724113 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148012b10 con 0x7fd15c153c40 2024-01-26T23:49:11.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.858+0000 7fd163151700 1 -- 172.21.15.46:0/1399724113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15c0cc310 con 0x7fd15c153c40 2024-01-26T23:49:11.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.859+0000 7fd163151700 1 -- 172.21.15.46:0/1399724113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15c0cc7f0 con 0x7fd15c153c40 2024-01-26T23:49:11.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.860+0000 7fd158ff9700 1 -- 172.21.15.46:0/1399724113 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 65) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fd14801e460 con 0x7fd15c153c40 2024-01-26T23:49:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.860+0000 7fd158ff9700 1 --2- 172.21.15.46:0/1399724113 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd140066ea0 0x7fd140069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.860+0000 7fd158ff9700 1 -- 172.21.15.46:0/1399724113 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6281+0+0 (secure 0 0 0) 0x7fd148084b80 con 0x7fd15c153c40 2024-01-26T23:49:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.861+0000 7fd160eed700 1 --2- 172.21.15.46:0/1399724113 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd140066ea0 0x7fd140069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.861+0000 7fd160eed700 1 --2- 172.21.15.46:0/1399724113 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd140066ea0 0x7fd140069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd150006fd0 tx=0x7fd150008040 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:11.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:11.865+0000 7fd158ff9700 1 -- 172.21.15.46:0/1399724113 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd148051c90 con 0x7fd15c153c40 2024-01-26T23:49:12.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.078+0000 7fd163151700 1 -- 172.21.15.46:0/1399724113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd15c04c710 con 0x7fd15c153c40 2024-01-26T23:49:12.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.082+0000 7fd158ff9700 1 -- 172.21.15.46:0/1399724113 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd1480552b0 con 0x7fd15c153c40 2024-01-26T23:49:12.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.084+0000 7fd163151700 1 -- 172.21.15.46:0/1399724113 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd140066ea0 msgr2=0x7fd140069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:12.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.084+0000 7fd163151700 1 --2- 172.21.15.46:0/1399724113 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd140066ea0 0x7fd140069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd150006fd0 tx=0x7fd150008040 comp rx=0 tx=0).stop 2024-01-26T23:49:12.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.084+0000 7fd163151700 1 -- 172.21.15.46:0/1399724113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd15c153c40 msgr2=0x7fd15c15c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:12.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.084+0000 7fd163151700 1 --2- 172.21.15.46:0/1399724113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd15c153c40 0x7fd15c15c650 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fd14800d860 tx=0x7fd148008040 comp rx=0 tx=0).stop 2024-01-26T23:49:12.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.085+0000 7fd163151700 1 -- 172.21.15.46:0/1399724113 shutdown_connections 2024-01-26T23:49:12.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.085+0000 7fd163151700 1 --2- 172.21.15.46:0/1399724113 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd140066ea0 0x7fd140069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.085+0000 7fd163151700 1 --2- 172.21.15.46:0/1399724113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd15c154010 0x7fd15c0af8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.085+0000 7fd163151700 1 --2- 172.21.15.46:0/1399724113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd15c0b1e80 0x7fd15c150d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.085+0000 7fd163151700 1 --2- 172.21.15.46:0/1399724113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd15c153c40 0x7fd15c15c650 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.085+0000 7fd163151700 1 -- 172.21.15.46:0/1399724113 >> 172.21.15.46:0/1399724113 conn(0x7fd15c067930 msgr2=0x7fd15c0adc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:12.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.085+0000 7fd163151700 1 -- 172.21.15.46:0/1399724113 shutdown_connections 2024-01-26T23:49:12.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.085+0000 7fd163151700 1 -- 172.21.15.46:0/1399724113 wait complete. 2024-01-26T23:49:12.098 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-26T23:49:12.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.232+0000 7f2e3279b700 1 Processor -- start 2024-01-26T23:49:12.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.232+0000 7f2e3279b700 1 -- start start 2024-01-26T23:49:12.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.233+0000 7f2e3279b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e2c0ca650 0x7f2e2c067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.233+0000 7f2e3279b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2c06fdb0 con 0x7f2e2c06ebe0 2024-01-26T23:49:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.233+0000 7f2e3279b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2c06f1a0 con 0x7f2e2c067af0 2024-01-26T23:49:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.233+0000 7f2e3279b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c06f480 con 0x7f2e2c0ca650 2024-01-26T23:49:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.233+0000 7f2e2b7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2e2c06ebe0 0x7f2e2c1597e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36818/0 (socket says 172.21.15.46:36818) 2024-01-26T23:49:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.233+0000 7f2e2b7fe700 1 -- 172.21.15.46:0/2462082787 learned_addr learned my addr 172.21.15.46:0/2462082787 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.233+0000 7f2e30d38700 1 --2- 172.21.15.46:0/2462082787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e2c0ca650 0x7f2e2c067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.234+0000 7f2e2affd700 1 -- 172.21.15.46:0/2462082787 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731734371 0 0) 0x7f2e2c06fdb0 con 0x7f2e2c06ebe0 2024-01-26T23:49:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.234+0000 7f2e2affd700 1 -- 172.21.15.46:0/2462082787 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e14003570 con 0x7f2e2c06ebe0 2024-01-26T23:49:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.234+0000 7f2e2affd700 1 -- 172.21.15.46:0/2462082787 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221065823 0 0) 0x7f2e2c06f1a0 con 0x7f2e2c067af0 2024-01-26T23:49:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.234+0000 7f2e2affd700 1 -- 172.21.15.46:0/2462082787 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e2c06fdb0 con 0x7f2e2c067af0 2024-01-26T23:49:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.234+0000 7f2e30d38700 1 -- 172.21.15.46:0/2462082787 >> v1:172.21.15.106:6789/0 conn(0x7f2e2c067af0 legacy=0x7f2e2c0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.234+0000 7f2e30d38700 1 -- 172.21.15.46:0/2462082787 >> v1:172.21.15.46:6789/0 conn(0x7f2e2c06ebe0 legacy=0x7f2e2c1597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.234+0000 7f2e30d38700 1 -- 172.21.15.46:0/2462082787 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c06f760 con 0x7f2e2c0ca650 2024-01-26T23:49:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.234+0000 7f2e30d38700 1 --2- 172.21.15.46:0/2462082787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e2c0ca650 0x7f2e2c067ec0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009e90 tx=0x7f2e24009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24f7155e9f5c3208 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.234+0000 7f2e2affd700 1 -- 172.21.15.46:0/2462082787 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2400e070 con 0x7f2e2c0ca650 2024-01-26T23:49:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.234+0000 7f2e2affd700 1 -- 172.21.15.46:0/2462082787 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2400bb80 con 0x7f2e2c0ca650 2024-01-26T23:49:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.234+0000 7f2e2affd700 1 -- 172.21.15.46:0/2462082787 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24012430 con 0x7f2e2c0ca650 2024-01-26T23:49:12.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.235+0000 7f2e3279b700 1 -- 172.21.15.46:0/2462082787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e2c0ca650 msgr2=0x7f2e2c067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.235+0000 7f2e3279b700 1 --2- 172.21.15.46:0/2462082787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e2c0ca650 0x7f2e2c067ec0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009e90 tx=0x7f2e24009780 comp rx=0 tx=0).stop 2024-01-26T23:49:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.235+0000 7f2e3279b700 1 -- 172.21.15.46:0/2462082787 shutdown_connections 2024-01-26T23:49:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.235+0000 7f2e3279b700 1 --2- 172.21.15.46:0/2462082787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e2c0ca650 0x7f2e2c067ec0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.235+0000 7f2e3279b700 1 -- 172.21.15.46:0/2462082787 >> 172.21.15.46:0/2462082787 conn(0x7f2e2c0cdf80 msgr2=0x7f2e2c06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.235+0000 7f2e3279b700 1 -- 172.21.15.46:0/2462082787 shutdown_connections 2024-01-26T23:49:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.235+0000 7f2e3279b700 1 -- 172.21.15.46:0/2462082787 wait complete. 2024-01-26T23:49:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.236+0000 7f2e3279b700 1 Processor -- start 2024-01-26T23:49:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.236+0000 7f2e3279b700 1 -- start start 2024-01-26T23:49:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.237+0000 7f2e3279b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e2c067af0 0x7f2e2c1492c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.237+0000 7f2e3279b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e2c06ebe0 0x7f2e2c15b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.237+0000 7f2e3279b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e2c0ca650 0x7f2e2c15bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.237+0000 7f2e3279b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0acff0 con 0x7f2e2c06ebe0 2024-01-26T23:49:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.237+0000 7f2e3279b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0ad2d0 con 0x7f2e2c067af0 2024-01-26T23:49:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.237+0000 7f2e3279b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0ad5b0 con 0x7f2e2c0ca650 2024-01-26T23:49:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.237+0000 7f2e2b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e2c06ebe0 0x7f2e2c15b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.237+0000 7f2e2b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e2c06ebe0 0x7f2e2c15b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50700/0 (socket says 172.21.15.46:50700) 2024-01-26T23:49:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.237+0000 7f2e2b7fe700 1 -- 172.21.15.46:0/17928127 learned_addr learned my addr 172.21.15.46:0/17928127 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.237+0000 7f2e30d38700 1 --2- 172.21.15.46:0/17928127 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e2c0ca650 0x7f2e2c15bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.237+0000 7f2e2bfff700 1 --2- 172.21.15.46:0/17928127 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e2c067af0 0x7f2e2c1492c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.238+0000 7f2e2b7fe700 1 -- 172.21.15.46:0/17928127 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e2c0ca650 msgr2=0x7f2e2c15bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.238+0000 7f2e2b7fe700 1 --2- 172.21.15.46:0/17928127 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e2c0ca650 0x7f2e2c15bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.238+0000 7f2e2b7fe700 1 -- 172.21.15.46:0/17928127 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e2c067af0 msgr2=0x7f2e2c1492c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.238+0000 7f2e2b7fe700 1 --2- 172.21.15.46:0/17928127 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e2c067af0 0x7f2e2c1492c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.238+0000 7f2e2b7fe700 1 -- 172.21.15.46:0/17928127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e24009a20 con 0x7f2e2c06ebe0 2024-01-26T23:49:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.238+0000 7f2e2bfff700 1 --2- 172.21.15.46:0/17928127 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e2c067af0 0x7f2e2c1492c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.238+0000 7f2e30d38700 1 --2- 172.21.15.46:0/17928127 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e2c0ca650 0x7f2e2c15bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.238+0000 7f2e2b7fe700 1 --2- 172.21.15.46:0/17928127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e2c06ebe0 0x7f2e2c15b700 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f2e2000c790 tx=0x7f2e20007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:12.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.238+0000 7f2e13fff700 1 -- 172.21.15.46:0/17928127 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e20005b30 con 0x7f2e2c06ebe0 2024-01-26T23:49:12.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.238+0000 7f2e13fff700 1 -- 172.21.15.46:0/17928127 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e20005c90 con 0x7f2e2c06ebe0 2024-01-26T23:49:12.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.238+0000 7f2e3279b700 1 -- 172.21.15.46:0/17928127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2c0ad890 con 0x7f2e2c06ebe0 2024-01-26T23:49:12.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.238+0000 7f2e13fff700 1 -- 172.21.15.46:0/17928127 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e20007300 con 0x7f2e2c06ebe0 2024-01-26T23:49:12.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.238+0000 7f2e3279b700 1 -- 172.21.15.46:0/17928127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e2c0ac3c0 con 0x7f2e2c06ebe0 2024-01-26T23:49:12.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.239+0000 7f2e3279b700 1 -- 172.21.15.46:0/17928127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e2c04c710 con 0x7f2e2c06ebe0 2024-01-26T23:49:12.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.240+0000 7f2e13fff700 1 -- 172.21.15.46:0/17928127 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 65) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f2e2001b3b0 con 0x7f2e2c06ebe0 2024-01-26T23:49:12.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.240+0000 7f2e13fff700 1 --2- 172.21.15.46:0/17928127 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2e18066ea0 0x7f2e18069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.240+0000 7f2e13fff700 1 -- 172.21.15.46:0/17928127 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6281+0+0 (secure 0 0 0) 0x7f2e20083c90 con 0x7f2e2c06ebe0 2024-01-26T23:49:12.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.240+0000 7f2e2bfff700 1 --2- 172.21.15.46:0/17928127 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2e18066ea0 0x7f2e18069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.241+0000 7f2e2bfff700 1 --2- 172.21.15.46:0/17928127 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2e18066ea0 0x7f2e18069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c000c00 tx=0x7f2e1c009040 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:12.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.245+0000 7f2e13fff700 1 -- 172.21.15.46:0/17928127 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2e20089020 con 0x7f2e2c06ebe0 2024-01-26T23:49:12.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.450+0000 7f2e3279b700 1 -- 172.21.15.46:0/17928127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f2e2c068690 con 0x7f2e2c06ebe0 2024-01-26T23:49:12.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.454+0000 7f2e13fff700 1 -- 172.21.15.46:0/17928127 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v44) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2e200170b0 con 0x7f2e2c06ebe0 2024-01-26T23:49:12.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.456+0000 7f2e3279b700 1 -- 172.21.15.46:0/17928127 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2e18066ea0 msgr2=0x7f2e18069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:12.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.457+0000 7f2e3279b700 1 --2- 172.21.15.46:0/17928127 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2e18066ea0 0x7f2e18069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c000c00 tx=0x7f2e1c009040 comp rx=0 tx=0).stop 2024-01-26T23:49:12.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.457+0000 7f2e3279b700 1 -- 172.21.15.46:0/17928127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e2c06ebe0 msgr2=0x7f2e2c15b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:12.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.457+0000 7f2e3279b700 1 --2- 172.21.15.46:0/17928127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e2c06ebe0 0x7f2e2c15b700 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f2e2000c790 tx=0x7f2e20007ea0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.457+0000 7f2e3279b700 1 -- 172.21.15.46:0/17928127 shutdown_connections 2024-01-26T23:49:12.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.457+0000 7f2e3279b700 1 --2- 172.21.15.46:0/17928127 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2e18066ea0 0x7f2e18069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.457+0000 7f2e3279b700 1 --2- 172.21.15.46:0/17928127 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e2c0ca650 0x7f2e2c15bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.457+0000 7f2e3279b700 1 --2- 172.21.15.46:0/17928127 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e2c067af0 0x7f2e2c1492c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.457+0000 7f2e3279b700 1 --2- 172.21.15.46:0/17928127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e2c06ebe0 0x7f2e2c15b700 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.457+0000 7f2e3279b700 1 -- 172.21.15.46:0/17928127 >> 172.21.15.46:0/17928127 conn(0x7f2e2c0cdf80 msgr2=0x7f2e2c0b1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:12.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.457+0000 7f2e3279b700 1 -- 172.21.15.46:0/17928127 shutdown_connections 2024-01-26T23:49:12.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.457+0000 7f2e3279b700 1 -- 172.21.15.46:0/17928127 wait complete. 2024-01-26T23:49:12.470 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-26T23:49:12.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.604+0000 7f0f42d85700 1 Processor -- start 2024-01-26T23:49:12.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.604+0000 7f0f42d85700 1 -- start start 2024-01-26T23:49:12.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f42d85700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0f3c067af0 0x7f0f3c067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f42d85700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f3c06ebe0 0x7f0f3c0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f42d85700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f3c1597e0 0x7f0f3c159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f42d85700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f3c06fdb0 con 0x7f0f3c1597e0 2024-01-26T23:49:12.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f42d85700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f3c0ade10 con 0x7f0f3c067af0 2024-01-26T23:49:12.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f42d85700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c06f1a0 con 0x7f0f3c06ebe0 2024-01-26T23:49:12.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f41322700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f3c1597e0 0x7f0f3c159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f41322700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f3c1597e0 0x7f0f3c159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50708/0 (socket says 172.21.15.46:50708) 2024-01-26T23:49:12.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f41322700 1 -- 172.21.15.46:0/1217953407 learned_addr learned my addr 172.21.15.46:0/1217953407 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f3bfff700 1 --2- 172.21.15.46:0/1217953407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f3c06ebe0 0x7f0f3c0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f40b21700 1 --2- 172.21.15.46:0/1217953407 >> v2:172.21.15.106:3300/0 conn(0x7f0f3c067af0 0x7f0f3c067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f41322700 1 -- 172.21.15.46:0/1217953407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f3c06ebe0 msgr2=0x7f0f3c0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f41322700 1 --2- 172.21.15.46:0/1217953407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f3c06ebe0 0x7f0f3c0ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f41322700 1 -- 172.21.15.46:0/1217953407 >> v2:172.21.15.106:3300/0 conn(0x7f0f3c067af0 msgr2=0x7f0f3c067ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f41322700 1 --2- 172.21.15.46:0/1217953407 >> v2:172.21.15.106:3300/0 conn(0x7f0f3c067af0 0x7f0f3c067ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f41322700 1 -- 172.21.15.46:0/1217953407 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c06f480 con 0x7f0f3c1597e0 2024-01-26T23:49:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f3bfff700 1 --2- 172.21.15.46:0/1217953407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f3c06ebe0 0x7f0f3c0ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:49:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.605+0000 7f0f40b21700 1 --2- 172.21.15.46:0/1217953407 >> v2:172.21.15.106:3300/0 conn(0x7f0f3c067af0 0x7f0f3c067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:49:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.606+0000 7f0f41322700 1 --2- 172.21.15.46:0/1217953407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f3c1597e0 0x7f0f3c159bf0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c064050 tx=0x7f0f34009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86c54e2b6c0b30dc server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.606+0000 7f0f3b7fe700 1 -- 172.21.15.46:0/1217953407 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3400e070 con 0x7f0f3c1597e0 2024-01-26T23:49:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.606+0000 7f0f3b7fe700 1 -- 172.21.15.46:0/1217953407 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f3400ba50 con 0x7f0f3c1597e0 2024-01-26T23:49:12.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.606+0000 7f0f3b7fe700 1 -- 172.21.15.46:0/1217953407 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f340092e0 con 0x7f0f3c1597e0 2024-01-26T23:49:12.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.606+0000 7f0f42d85700 1 -- 172.21.15.46:0/1217953407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f3c1597e0 msgr2=0x7f0f3c159bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:12.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.606+0000 7f0f42d85700 1 --2- 172.21.15.46:0/1217953407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f3c1597e0 0x7f0f3c159bf0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c064050 tx=0x7f0f34009840 comp rx=0 tx=0).stop 2024-01-26T23:49:12.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.606+0000 7f0f42d85700 1 -- 172.21.15.46:0/1217953407 shutdown_connections 2024-01-26T23:49:12.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.606+0000 7f0f42d85700 1 --2- 172.21.15.46:0/1217953407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f3c1597e0 0x7f0f3c159bf0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.606+0000 7f0f42d85700 1 --2- 172.21.15.46:0/1217953407 >> v2:172.21.15.106:3300/0 conn(0x7f0f3c067af0 0x7f0f3c067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.606+0000 7f0f42d85700 1 --2- 172.21.15.46:0/1217953407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f3c06ebe0 0x7f0f3c0ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.606+0000 7f0f42d85700 1 -- 172.21.15.46:0/1217953407 >> 172.21.15.46:0/1217953407 conn(0x7f0f3c0cdf80 msgr2=0x7f0f3c06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:12.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.607+0000 7f0f42d85700 1 -- 172.21.15.46:0/1217953407 shutdown_connections 2024-01-26T23:49:12.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.607+0000 7f0f42d85700 1 -- 172.21.15.46:0/1217953407 wait complete. 2024-01-26T23:49:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.608+0000 7f0f42d85700 1 Processor -- start 2024-01-26T23:49:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.608+0000 7f0f42d85700 1 -- start start 2024-01-26T23:49:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f42d85700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f3c067af0 0x7f0f3c16c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f42d85700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f3c06ebe0 0x7f0f3c0cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f42d85700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f3c1597e0 0x7f0f3c0cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f42d85700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c15a8d0 con 0x7f0f3c067af0 2024-01-26T23:49:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f42d85700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c159c50 con 0x7f0f3c1597e0 2024-01-26T23:49:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f42d85700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c159f30 con 0x7f0f3c06ebe0 2024-01-26T23:49:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f40b21700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f3c067af0 0x7f0f3c16c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f40b21700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f3c067af0 0x7f0f3c16c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50716/0 (socket says 172.21.15.46:50716) 2024-01-26T23:49:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f40b21700 1 -- 172.21.15.46:0/2679968275 learned_addr learned my addr 172.21.15.46:0/2679968275 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f3bfff700 1 --2- 172.21.15.46:0/2679968275 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f3c06ebe0 0x7f0f3c0cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f41322700 1 --2- 172.21.15.46:0/2679968275 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f3c1597e0 0x7f0f3c0cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f40b21700 1 -- 172.21.15.46:0/2679968275 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f3c06ebe0 msgr2=0x7f0f3c0cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f40b21700 1 --2- 172.21.15.46:0/2679968275 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f3c06ebe0 0x7f0f3c0cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f40b21700 1 -- 172.21.15.46:0/2679968275 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f3c1597e0 msgr2=0x7f0f3c0cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f40b21700 1 --2- 172.21.15.46:0/2679968275 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f3c1597e0 0x7f0f3c0cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f40b21700 1 -- 172.21.15.46:0/2679968275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f34009ae0 con 0x7f0f3c067af0 2024-01-26T23:49:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f3bfff700 1 --2- 172.21.15.46:0/2679968275 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f3c06ebe0 0x7f0f3c0cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f40b21700 1 --2- 172.21.15.46:0/2679968275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f3c067af0 0x7f0f3c16c570 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f0f3000c850 tx=0x7f0f30008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.609+0000 7f0f38ff9700 1 -- 172.21.15.46:0/2679968275 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f30011600 con 0x7f0f3c067af0 2024-01-26T23:49:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.610+0000 7f0f38ff9700 1 -- 172.21.15.46:0/2679968275 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f30011760 con 0x7f0f3c067af0 2024-01-26T23:49:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.610+0000 7f0f42d85700 1 -- 172.21.15.46:0/2679968275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f3c15a210 con 0x7f0f3c067af0 2024-01-26T23:49:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.610+0000 7f0f38ff9700 1 -- 172.21.15.46:0/2679968275 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f30011b20 con 0x7f0f3c067af0 2024-01-26T23:49:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.610+0000 7f0f42d85700 1 -- 172.21.15.46:0/2679968275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f3c16cab0 con 0x7f0f3c067af0 2024-01-26T23:49:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.611+0000 7f0f42d85700 1 -- 172.21.15.46:0/2679968275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f3c04c710 con 0x7f0f3c067af0 2024-01-26T23:49:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.611+0000 7f0f38ff9700 1 -- 172.21.15.46:0/2679968275 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 65) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f0f30018350 con 0x7f0f3c067af0 2024-01-26T23:49:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.612+0000 7f0f38ff9700 1 --2- 172.21.15.46:0/2679968275 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0f20066ea0 0x7f0f20069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.612+0000 7f0f38ff9700 1 -- 172.21.15.46:0/2679968275 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6281+0+0 (secure 0 0 0) 0x7f0f30084570 con 0x7f0f3c067af0 2024-01-26T23:49:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.612+0000 7f0f3bfff700 1 --2- 172.21.15.46:0/2679968275 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0f20066ea0 0x7f0f20069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.612+0000 7f0f3bfff700 1 --2- 172.21.15.46:0/2679968275 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0f20066ea0 0x7f0f20069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c15b810 tx=0x7f0f28008040 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:12.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.616+0000 7f0f38ff9700 1 -- 172.21.15.46:0/2679968275 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0f30051680 con 0x7f0f3c067af0 2024-01-26T23:49:12.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.820+0000 7f0f42d85700 1 -- 172.21.15.46:0/2679968275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f0f3c068690 con 0x7f0f3c067af0 2024-01-26T23:49:12.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.824+0000 7f0f38ff9700 1 -- 172.21.15.46:0/2679968275 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v45) v1 ==== 115+0+0 (secure 0 0 0) 0x7f0f30054ca0 con 0x7f0f3c067af0 2024-01-26T23:49:12.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.826+0000 7f0f42d85700 1 -- 172.21.15.46:0/2679968275 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0f20066ea0 msgr2=0x7f0f20069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:12.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.826+0000 7f0f42d85700 1 --2- 172.21.15.46:0/2679968275 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0f20066ea0 0x7f0f20069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c15b810 tx=0x7f0f28008040 comp rx=0 tx=0).stop 2024-01-26T23:49:12.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.826+0000 7f0f42d85700 1 -- 172.21.15.46:0/2679968275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f3c067af0 msgr2=0x7f0f3c16c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:12.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.826+0000 7f0f42d85700 1 --2- 172.21.15.46:0/2679968275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f3c067af0 0x7f0f3c16c570 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f0f3000c850 tx=0x7f0f30008040 comp rx=0 tx=0).stop 2024-01-26T23:49:12.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.826+0000 7f0f42d85700 1 -- 172.21.15.46:0/2679968275 shutdown_connections 2024-01-26T23:49:12.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.826+0000 7f0f42d85700 1 --2- 172.21.15.46:0/2679968275 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0f20066ea0 0x7f0f20069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.826+0000 7f0f42d85700 1 --2- 172.21.15.46:0/2679968275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f3c067af0 0x7f0f3c16c570 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.826+0000 7f0f42d85700 1 --2- 172.21.15.46:0/2679968275 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f3c1597e0 0x7f0f3c0cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.826+0000 7f0f42d85700 1 --2- 172.21.15.46:0/2679968275 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f3c06ebe0 0x7f0f3c0cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.826+0000 7f0f42d85700 1 -- 172.21.15.46:0/2679968275 >> 172.21.15.46:0/2679968275 conn(0x7f0f3c0cdf80 msgr2=0x7f0f3c153390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:12.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.826+0000 7f0f42d85700 1 -- 172.21.15.46:0/2679968275 shutdown_connections 2024-01-26T23:49:12.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.827+0000 7f0f42d85700 1 -- 172.21.15.46:0/2679968275 wait complete. 2024-01-26T23:49:12.840 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-01-26T23:49:12.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.972+0000 7f95fe245700 1 Processor -- start 2024-01-26T23:49:12.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.972+0000 7f95fe245700 1 -- start start 2024-01-26T23:49:12.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.973+0000 7f95fe245700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95f8159760 0x7f95f80b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.973+0000 7f95fe245700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95f8068040 0x7f95f80b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.973+0000 7f95fe245700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f815abc0 con 0x7f95f8159b30 2024-01-26T23:49:12.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.973+0000 7f95fe245700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95f80b2e90 con 0x7f95f8159760 2024-01-26T23:49:12.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.973+0000 7f95fe245700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f8159fb0 con 0x7f95f8068040 2024-01-26T23:49:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.973+0000 7f95f77fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95f8159760 0x7f95f80b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.973+0000 7f95f77fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95f8159760 0x7f95f80b0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50718/0 (socket says 172.21.15.46:50718) 2024-01-26T23:49:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.973+0000 7f95f77fe700 1 -- 172.21.15.46:0/4102871728 learned_addr learned my addr 172.21.15.46:0/4102871728 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.973+0000 7f95f7fff700 1 --2- 172.21.15.46:0/4102871728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95f8068040 0x7f95f80b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.974+0000 7f95f77fe700 1 -- 172.21.15.46:0/4102871728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95f8068040 msgr2=0x7f95f80b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.974+0000 7f95f77fe700 1 --2- 172.21.15.46:0/4102871728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95f8068040 0x7f95f80b11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.974+0000 7f95f77fe700 1 -- 172.21.15.46:0/4102871728 >> v1:172.21.15.106:6789/0 conn(0x7f95f8159b30 legacy=0x7f95f8067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.974+0000 7f95f77fe700 1 -- 172.21.15.46:0/4102871728 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f815a290 con 0x7f95f8159760 2024-01-26T23:49:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.974+0000 7f95f7fff700 1 --2- 172.21.15.46:0/4102871728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95f8068040 0x7f95f80b11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:49:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.974+0000 7f95f77fe700 1 --2- 172.21.15.46:0/4102871728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f8159760 0x7f95f80b0c70 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f95e00099b0 tx=0x7f95e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e102405731aa12c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.974+0000 7f95f67fc700 1 -- 172.21.15.46:0/4102871728 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e000b820 con 0x7f95f8159760 2024-01-26T23:49:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.974+0000 7f95f67fc700 1 -- 172.21.15.46:0/4102871728 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e000b980 con 0x7f95f8159760 2024-01-26T23:49:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.974+0000 7f95f67fc700 1 -- 172.21.15.46:0/4102871728 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e000bd40 con 0x7f95f8159760 2024-01-26T23:49:12.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.975+0000 7f95fe245700 1 -- 172.21.15.46:0/4102871728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f8159760 msgr2=0x7f95f80b0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:12.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.975+0000 7f95fe245700 1 --2- 172.21.15.46:0/4102871728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f8159760 0x7f95f80b0c70 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f95e00099b0 tx=0x7f95e0009250 comp rx=0 tx=0).stop 2024-01-26T23:49:12.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.975+0000 7f95fe245700 1 -- 172.21.15.46:0/4102871728 shutdown_connections 2024-01-26T23:49:12.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.975+0000 7f95fe245700 1 --2- 172.21.15.46:0/4102871728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95f8068040 0x7f95f80b11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.975+0000 7f95fe245700 1 --2- 172.21.15.46:0/4102871728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f8159760 0x7f95f80b0c70 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.975+0000 7f95fe245700 1 -- 172.21.15.46:0/4102871728 >> 172.21.15.46:0/4102871728 conn(0x7f95f806b630 msgr2=0x7f95f80aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:12.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.975+0000 7f95fe245700 1 -- 172.21.15.46:0/4102871728 shutdown_connections 2024-01-26T23:49:12.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.975+0000 7f95fe245700 1 -- 172.21.15.46:0/4102871728 wait complete. 2024-01-26T23:49:12.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.976+0000 7f95fe245700 1 Processor -- start 2024-01-26T23:49:12.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.976+0000 7f95fe245700 1 -- start start 2024-01-26T23:49:12.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.977+0000 7f95fe245700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95f8068040 0x7f95f814f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.977+0000 7f95fe245700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f8159760 0x7f95f813a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.977+0000 7f95fe245700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95f814cf40 0x7f95f815c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.977+0000 7f95fe245700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f80aada0 con 0x7f95f8159760 2024-01-26T23:49:12.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.977+0000 7f95fe245700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f814f7e0 con 0x7f95f814cf40 2024-01-26T23:49:12.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.977+0000 7f95fe245700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f80aa470 con 0x7f95f8068040 2024-01-26T23:49:12.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.977+0000 7f95f6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f8159760 0x7f95f813a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.977+0000 7f95f6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f8159760 0x7f95f813a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50730/0 (socket says 172.21.15.46:50730) 2024-01-26T23:49:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.977+0000 7f95f6ffd700 1 -- 172.21.15.46:0/3588650212 learned_addr learned my addr 172.21.15.46:0/3588650212 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.977+0000 7f95f7fff700 1 --2- 172.21.15.46:0/3588650212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95f814cf40 0x7f95f815c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.977+0000 7f95f77fe700 1 --2- 172.21.15.46:0/3588650212 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95f8068040 0x7f95f814f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.977+0000 7f95f6ffd700 1 -- 172.21.15.46:0/3588650212 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95f8068040 msgr2=0x7f95f814f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.977+0000 7f95f6ffd700 1 --2- 172.21.15.46:0/3588650212 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95f8068040 0x7f95f814f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.977+0000 7f95f6ffd700 1 -- 172.21.15.46:0/3588650212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95f814cf40 msgr2=0x7f95f815c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.977+0000 7f95f6ffd700 1 --2- 172.21.15.46:0/3588650212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95f814cf40 0x7f95f815c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.978+0000 7f95f6ffd700 1 -- 172.21.15.46:0/3588650212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e00094f0 con 0x7f95f8159760 2024-01-26T23:49:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.978+0000 7f95f6ffd700 1 --2- 172.21.15.46:0/3588650212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f8159760 0x7f95f813a5a0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f95e800b5e0 tx=0x7f95e8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.978+0000 7f95ef7fe700 1 -- 172.21.15.46:0/3588650212 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e800f510 con 0x7f95f8159760 2024-01-26T23:49:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.978+0000 7f95ef7fe700 1 -- 172.21.15.46:0/3588650212 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e800f670 con 0x7f95f8159760 2024-01-26T23:49:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.978+0000 7f95fe245700 1 -- 172.21.15.46:0/3588650212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f80aa750 con 0x7f95f8159760 2024-01-26T23:49:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.978+0000 7f95ef7fe700 1 -- 172.21.15.46:0/3588650212 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e800fa30 con 0x7f95f8159760 2024-01-26T23:49:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.978+0000 7f95fe245700 1 -- 172.21.15.46:0/3588650212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f813aae0 con 0x7f95f8159760 2024-01-26T23:49:12.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.979+0000 7f95fe245700 1 -- 172.21.15.46:0/3588650212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f804c710 con 0x7f95f8159760 2024-01-26T23:49:12.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.980+0000 7f95ef7fe700 1 -- 172.21.15.46:0/3588650212 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 65) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f95e801a400 con 0x7f95f8159760 2024-01-26T23:49:12.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.980+0000 7f95ef7fe700 1 --2- 172.21.15.46:0/3588650212 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f95d8066ea0 0x7f95d8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:12.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.980+0000 7f95ef7fe700 1 -- 172.21.15.46:0/3588650212 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6281+0+0 (secure 0 0 0) 0x7f95e80840f0 con 0x7f95f8159760 2024-01-26T23:49:12.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.980+0000 7f95f77fe700 1 --2- 172.21.15.46:0/3588650212 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f95d8066ea0 0x7f95d8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:12.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.981+0000 7f95f77fe700 1 --2- 172.21.15.46:0/3588650212 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f95d8066ea0 0x7f95d8069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f95e00099b0 tx=0x7f95e0016000 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:12.984+0000 7f95ef7fe700 1 -- 172.21.15.46:0/3588650212 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f95e8051280 con 0x7f95f8159760 2024-01-26T23:49:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:13.274+0000 7f95fe245700 1 -- 172.21.15.46:0/3588650212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f95f80aaee0 con 0x7f95f8159760 2024-01-26T23:49:13.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:13.830+0000 7f95ef7fe700 1 -- 172.21.15.46:0/3588650212 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f95e8051650 con 0x7f95f8159760 2024-01-26T23:49:13.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:13.889+0000 7f95ef7fe700 1 -- 172.21.15.46:0/3588650212 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v739) v1 ==== 148+0+0 (secure 0 0 0) 0x7f95e8051280 con 0x7f95f8159760 2024-01-26T23:49:13.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:13.891+0000 7f95fe245700 1 -- 172.21.15.46:0/3588650212 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f95d8066ea0 msgr2=0x7f95d8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:13.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:13.892+0000 7f95fe245700 1 --2- 172.21.15.46:0/3588650212 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f95d8066ea0 0x7f95d8069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f95e00099b0 tx=0x7f95e0016000 comp rx=0 tx=0).stop 2024-01-26T23:49:13.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:13.892+0000 7f95fe245700 1 -- 172.21.15.46:0/3588650212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f8159760 msgr2=0x7f95f813a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:13.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:13.892+0000 7f95fe245700 1 --2- 172.21.15.46:0/3588650212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f8159760 0x7f95f813a5a0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f95e800b5e0 tx=0x7f95e8009120 comp rx=0 tx=0).stop 2024-01-26T23:49:13.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:13.892+0000 7f95fe245700 1 -- 172.21.15.46:0/3588650212 shutdown_connections 2024-01-26T23:49:13.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:13.892+0000 7f95fe245700 1 --2- 172.21.15.46:0/3588650212 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f95d8066ea0 0x7f95d8069350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:13.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:13.892+0000 7f95fe245700 1 --2- 172.21.15.46:0/3588650212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95f814cf40 0x7f95f815c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:13.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:13.892+0000 7f95fe245700 1 --2- 172.21.15.46:0/3588650212 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95f8068040 0x7f95f814f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:13.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:13.892+0000 7f95fe245700 1 --2- 172.21.15.46:0/3588650212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f8159760 0x7f95f813a5a0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:13.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:13.892+0000 7f95fe245700 1 -- 172.21.15.46:0/3588650212 >> 172.21.15.46:0/3588650212 conn(0x7f95f806b630 msgr2=0x7f95f814bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:13.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:13.892+0000 7f95fe245700 1 -- 172.21.15.46:0/3588650212 shutdown_connections 2024-01-26T23:49:13.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:13.892+0000 7f95fe245700 1 -- 172.21.15.46:0/3588650212 wait complete. 2024-01-26T23:49:13.907 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2024-01-26T23:49:13.907 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-26T23:49:13.908 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-01-26T23:49:14.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.042+0000 7f96847e8700 1 Processor -- start 2024-01-26T23:49:14.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.042+0000 7f96847e8700 1 -- start start 2024-01-26T23:49:14.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.043+0000 7f96847e8700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f967c159610 0x7f967c0b2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:14.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.043+0000 7f96847e8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f967c0b2f90 0x7f967c067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:14.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.043+0000 7f96847e8700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967c0ce2b0 con 0x7f967c1599e0 2024-01-26T23:49:14.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.043+0000 7f96847e8700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f967c0afeb0 con 0x7f967c0b2f90 2024-01-26T23:49:14.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.043+0000 7f96847e8700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f967c0cd3c0 con 0x7f967c159610 2024-01-26T23:49:14.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.044+0000 7f9681d83700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f967c0b2f90 0x7f967c067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:14.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.044+0000 7f9681d83700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f967c0b2f90 0x7f967c067930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50738/0 (socket says 172.21.15.46:50738) 2024-01-26T23:49:14.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.044+0000 7f9681d83700 1 -- 172.21.15.46:0/1802852016 learned_addr learned my addr 172.21.15.46:0/1802852016 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:14.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.044+0000 7f9682584700 1 --2- 172.21.15.46:0/1802852016 >> v2:172.21.15.106:3300/0 conn(0x7f967c159610 0x7f967c0b2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:14.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.044+0000 7f9681d83700 1 -- 172.21.15.46:0/1802852016 >> v2:172.21.15.106:3300/0 conn(0x7f967c159610 msgr2=0x7f967c0b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:14.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.044+0000 7f9681d83700 1 --2- 172.21.15.46:0/1802852016 >> v2:172.21.15.106:3300/0 conn(0x7f967c159610 0x7f967c0b2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:14.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.044+0000 7f9681d83700 1 -- 172.21.15.46:0/1802852016 >> v1:172.21.15.106:6789/0 conn(0x7f967c1599e0 legacy=0x7f967c06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:14.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.044+0000 7f9681d83700 1 -- 172.21.15.46:0/1802852016 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c0cd6a0 con 0x7f967c0b2f90 2024-01-26T23:49:14.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.044+0000 7f9681d83700 1 --2- 172.21.15.46:0/1802852016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c0b2f90 0x7f967c067930 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f967c06bc80 tx=0x7f9678009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45c2f6126e47e701 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:14.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.044+0000 7f9681582700 1 -- 172.21.15.46:0/1802852016 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678009e70 con 0x7f967c0b2f90 2024-01-26T23:49:14.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.044+0000 7f9681582700 1 -- 172.21.15.46:0/1802852016 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967800b860 con 0x7f967c0b2f90 2024-01-26T23:49:14.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.044+0000 7f9681582700 1 -- 172.21.15.46:0/1802852016 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967800bc00 con 0x7f967c0b2f90 2024-01-26T23:49:14.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.045+0000 7f96847e8700 1 -- 172.21.15.46:0/1802852016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c0b2f90 msgr2=0x7f967c067930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:14.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.045+0000 7f96847e8700 1 --2- 172.21.15.46:0/1802852016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c0b2f90 0x7f967c067930 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f967c06bc80 tx=0x7f9678009250 comp rx=0 tx=0).stop 2024-01-26T23:49:14.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.045+0000 7f96847e8700 1 -- 172.21.15.46:0/1802852016 shutdown_connections 2024-01-26T23:49:14.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.045+0000 7f96847e8700 1 --2- 172.21.15.46:0/1802852016 >> v2:172.21.15.106:3300/0 conn(0x7f967c159610 0x7f967c0b2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:14.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.045+0000 7f96847e8700 1 --2- 172.21.15.46:0/1802852016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c0b2f90 0x7f967c067930 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:14.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.045+0000 7f96847e8700 1 -- 172.21.15.46:0/1802852016 >> 172.21.15.46:0/1802852016 conn(0x7f967c0694d0 msgr2=0x7f967c0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:14.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.045+0000 7f96847e8700 1 -- 172.21.15.46:0/1802852016 shutdown_connections 2024-01-26T23:49:14.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.045+0000 7f96847e8700 1 -- 172.21.15.46:0/1802852016 wait complete. 2024-01-26T23:49:14.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.046+0000 7f96847e8700 1 Processor -- start 2024-01-26T23:49:14.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.046+0000 7f96847e8700 1 -- start start 2024-01-26T23:49:14.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.047+0000 7f96847e8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f967c0b2f90 0x7f967c16b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:14.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.047+0000 7f96847e8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c159610 0x7f967c16a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:14.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.047+0000 7f96847e8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f967c1599e0 0x7f967c0ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:14.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.047+0000 7f96847e8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c16b6d0 con 0x7f967c159610 2024-01-26T23:49:14.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.047+0000 7f96847e8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c16b9b0 con 0x7f967c1599e0 2024-01-26T23:49:14.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.047+0000 7f96847e8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967c16bc90 con 0x7f967c0b2f90 2024-01-26T23:49:14.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.047+0000 7f9681d83700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c159610 0x7f967c16a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:14.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.047+0000 7f9681d83700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c159610 0x7f967c16a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50740/0 (socket says 172.21.15.46:50740) 2024-01-26T23:49:14.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.047+0000 7f9681d83700 1 -- 172.21.15.46:0/3953883958 learned_addr learned my addr 172.21.15.46:0/3953883958 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.047+0000 7f9682584700 1 --2- 172.21.15.46:0/3953883958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f967c0b2f90 0x7f967c16b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.047+0000 7f9682d85700 1 --2- 172.21.15.46:0/3953883958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f967c1599e0 0x7f967c0ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.048+0000 7f9681d83700 1 -- 172.21.15.46:0/3953883958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f967c0b2f90 msgr2=0x7f967c16b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.048+0000 7f9681d83700 1 --2- 172.21.15.46:0/3953883958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f967c0b2f90 0x7f967c16b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.048+0000 7f9681d83700 1 -- 172.21.15.46:0/3953883958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f967c1599e0 msgr2=0x7f967c0ab7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.048+0000 7f9681d83700 1 --2- 172.21.15.46:0/3953883958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f967c1599e0 0x7f967c0ab7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.048+0000 7f9681d83700 1 -- 172.21.15.46:0/3953883958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96780094f0 con 0x7f967c159610 2024-01-26T23:49:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.048+0000 7f9682d85700 1 --2- 172.21.15.46:0/3953883958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f967c1599e0 0x7f967c0ab7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.048+0000 7f9681d83700 1 --2- 172.21.15.46:0/3953883958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c159610 0x7f967c16a940 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f967c064970 tx=0x7f9678017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.048+0000 7f966e7fc700 1 -- 172.21.15.46:0/3953883958 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967800b9d0 con 0x7f967c159610 2024-01-26T23:49:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.048+0000 7f966e7fc700 1 -- 172.21.15.46:0/3953883958 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967800bb30 con 0x7f967c159610 2024-01-26T23:49:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.048+0000 7f966e7fc700 1 -- 172.21.15.46:0/3953883958 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678025670 con 0x7f967c159610 2024-01-26T23:49:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.048+0000 7f96847e8700 1 -- 172.21.15.46:0/3953883958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967c16bf70 con 0x7f967c159610 2024-01-26T23:49:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.048+0000 7f96847e8700 1 -- 172.21.15.46:0/3953883958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967c16ae80 con 0x7f967c159610 2024-01-26T23:49:14.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.049+0000 7f96847e8700 1 -- 172.21.15.46:0/3953883958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967c16c490 con 0x7f967c159610 2024-01-26T23:49:14.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.050+0000 7f966e7fc700 1 -- 172.21.15.46:0/3953883958 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9678023070 con 0x7f967c159610 2024-01-26T23:49:14.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.050+0000 7f966e7fc700 1 --2- 172.21.15.46:0/3953883958 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9660066f30 0x7f96600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:14.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.050+0000 7f966e7fc700 1 -- 172.21.15.46:0/3953883958 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6281+0+0 (secure 0 0 0) 0x7f9678085bb0 con 0x7f967c159610 2024-01-26T23:49:14.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.050+0000 7f9682584700 1 --2- 172.21.15.46:0/3953883958 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9660066f30 0x7f96600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:14.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.051+0000 7f9682584700 1 --2- 172.21.15.46:0/3953883958 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9660066f30 0x7f96600693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9670009db0 tx=0x7f96700091a0 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:14.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.054+0000 7f966e7fc700 1 -- 172.21.15.46:0/3953883958 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f967c16c490 con 0x7f967c159610 2024-01-26T23:49:14.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.341+0000 7f96847e8700 1 -- 172.21.15.46:0/3953883958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7f967c16c490 con 0x7f967c159610 2024-01-26T23:49:14.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.893+0000 7f966e7fc700 1 -- 172.21.15.46:0/3953883958 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v740) v1 ==== 163+0+0 (secure 0 0 0) 0x7f967c16c490 con 0x7f967c159610 2024-01-26T23:49:14.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.896+0000 7f96847e8700 1 -- 172.21.15.46:0/3953883958 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9660066f30 msgr2=0x7f96600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.896+0000 7f96847e8700 1 --2- 172.21.15.46:0/3953883958 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9660066f30 0x7f96600693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9670009db0 tx=0x7f96700091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.896+0000 7f96847e8700 1 -- 172.21.15.46:0/3953883958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c159610 msgr2=0x7f967c16a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.896+0000 7f96847e8700 1 --2- 172.21.15.46:0/3953883958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c159610 0x7f967c16a940 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f967c064970 tx=0x7f9678017000 comp rx=0 tx=0).stop 2024-01-26T23:49:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.896+0000 7f96847e8700 1 -- 172.21.15.46:0/3953883958 shutdown_connections 2024-01-26T23:49:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.896+0000 7f96847e8700 1 --2- 172.21.15.46:0/3953883958 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9660066f30 0x7f96600693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.896+0000 7f96847e8700 1 --2- 172.21.15.46:0/3953883958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f967c1599e0 0x7f967c0ab7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.896+0000 7f96847e8700 1 --2- 172.21.15.46:0/3953883958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f967c0b2f90 0x7f967c16b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.896+0000 7f96847e8700 1 --2- 172.21.15.46:0/3953883958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c159610 0x7f967c16a940 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.896+0000 7f96847e8700 1 -- 172.21.15.46:0/3953883958 >> 172.21.15.46:0/3953883958 conn(0x7f967c0694d0 msgr2=0x7f967c154ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.896+0000 7f96847e8700 1 -- 172.21.15.46:0/3953883958 shutdown_connections 2024-01-26T23:49:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:14.896+0000 7f96847e8700 1 -- 172.21.15.46:0/3953883958 wait complete. 2024-01-26T23:49:14.912 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-26T23:49:15.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.050+0000 7fd48ff6a700 1 Processor -- start 2024-01-26T23:49:15.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.050+0000 7fd48ff6a700 1 -- start start 2024-01-26T23:49:15.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.051+0000 7fd48ff6a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd4880c8da0 0x7fd4880c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:15.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.051+0000 7fd48ff6a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd4880caf30 0x7fd4880cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:15.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.051+0000 7fd48ff6a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4880ccc00 con 0x7fd488140af0 2024-01-26T23:49:15.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.051+0000 7fd48ff6a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4880cbbf0 con 0x7fd4880c8da0 2024-01-26T23:49:15.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.051+0000 7fd48ff6a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4880cc310 con 0x7fd4880caf30 2024-01-26T23:49:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.051+0000 7fd48dd06700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd4880c8da0 0x7fd4880c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.051+0000 7fd48d505700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd488140af0 0x7fd488140f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36832/0 (socket says 172.21.15.46:36832) 2024-01-26T23:49:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.051+0000 7fd48d505700 1 -- 172.21.15.46:0/3220686877 learned_addr learned my addr 172.21.15.46:0/3220686877 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.051+0000 7fd48e507700 1 --2- 172.21.15.46:0/3220686877 >> v2:172.21.15.106:3300/0 conn(0x7fd4880caf30 0x7fd4880cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.052+0000 7fd48dd06700 1 -- 172.21.15.46:0/3220686877 >> v2:172.21.15.106:3300/0 conn(0x7fd4880caf30 msgr2=0x7fd4880cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.052+0000 7fd48dd06700 1 --2- 172.21.15.46:0/3220686877 >> v2:172.21.15.106:3300/0 conn(0x7fd4880caf30 0x7fd4880cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.052+0000 7fd48dd06700 1 -- 172.21.15.46:0/3220686877 >> v1:172.21.15.46:6789/0 conn(0x7fd488140af0 legacy=0x7fd488140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.052+0000 7fd48dd06700 1 -- 172.21.15.46:0/3220686877 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4880cc5b0 con 0x7fd4880c8da0 2024-01-26T23:49:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.052+0000 7fd48cd04700 1 -- 172.21.15.46:0/3220686877 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262134105 0 0) 0x7fd4880ccc00 con 0x7fd488140af0 2024-01-26T23:49:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.052+0000 7fd48dd06700 1 --2- 172.21.15.46:0/3220686877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880c8da0 0x7fd4880c9170 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fd4881416d0 tx=0x7fd47c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4011a6b20d688dc5 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.052+0000 7fd48cd04700 1 -- 172.21.15.46:0/3220686877 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c011630 con 0x7fd4880c8da0 2024-01-26T23:49:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.052+0000 7fd48cd04700 1 -- 172.21.15.46:0/3220686877 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47c011790 con 0x7fd4880c8da0 2024-01-26T23:49:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.052+0000 7fd48cd04700 1 -- 172.21.15.46:0/3220686877 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c011b30 con 0x7fd4880c8da0 2024-01-26T23:49:15.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.053+0000 7fd48ff6a700 1 -- 172.21.15.46:0/3220686877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880c8da0 msgr2=0x7fd4880c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:15.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.053+0000 7fd48ff6a700 1 --2- 172.21.15.46:0/3220686877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880c8da0 0x7fd4880c9170 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fd4881416d0 tx=0x7fd47c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.053+0000 7fd48ff6a700 1 -- 172.21.15.46:0/3220686877 shutdown_connections 2024-01-26T23:49:15.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.053+0000 7fd48ff6a700 1 --2- 172.21.15.46:0/3220686877 >> v2:172.21.15.106:3300/0 conn(0x7fd4880caf30 0x7fd4880cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.053+0000 7fd48ff6a700 1 --2- 172.21.15.46:0/3220686877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880c8da0 0x7fd4880c9170 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.053+0000 7fd48ff6a700 1 -- 172.21.15.46:0/3220686877 >> 172.21.15.46:0/3220686877 conn(0x7fd48806bd90 msgr2=0x7fd48806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:15.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.053+0000 7fd48ff6a700 1 -- 172.21.15.46:0/3220686877 shutdown_connections 2024-01-26T23:49:15.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.053+0000 7fd48ff6a700 1 -- 172.21.15.46:0/3220686877 wait complete. 2024-01-26T23:49:15.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.054+0000 7fd48ff6a700 1 Processor -- start 2024-01-26T23:49:15.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.054+0000 7fd48ff6a700 1 -- start start 2024-01-26T23:49:15.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48ff6a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4880c8da0 0x7fd488156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:15.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48ff6a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880caf30 0x7fd4880b2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:15.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48ff6a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd488140af0 0x7fd4880b2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:15.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48ff6a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48813af60 con 0x7fd4880caf30 2024-01-26T23:49:15.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48ff6a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4880c5e50 con 0x7fd4880c8da0 2024-01-26T23:49:15.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48ff6a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4880c6130 con 0x7fd488140af0 2024-01-26T23:49:15.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48d505700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880caf30 0x7fd4880b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:15.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48d505700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880caf30 0x7fd4880b2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50754/0 (socket says 172.21.15.46:50754) 2024-01-26T23:49:15.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48d505700 1 -- 172.21.15.46:0/3975568661 learned_addr learned my addr 172.21.15.46:0/3975568661 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48dd06700 1 --2- 172.21.15.46:0/3975568661 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4880c8da0 0x7fd488156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48e507700 1 --2- 172.21.15.46:0/3975568661 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd488140af0 0x7fd4880b2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48d505700 1 -- 172.21.15.46:0/3975568661 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd488140af0 msgr2=0x7fd4880b2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48d505700 1 --2- 172.21.15.46:0/3975568661 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd488140af0 0x7fd4880b2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48d505700 1 -- 172.21.15.46:0/3975568661 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4880c8da0 msgr2=0x7fd488156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48d505700 1 --2- 172.21.15.46:0/3975568661 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4880c8da0 0x7fd488156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48d505700 1 -- 172.21.15.46:0/3975568661 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47c009420 con 0x7fd4880caf30 2024-01-26T23:49:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48dd06700 1 --2- 172.21.15.46:0/3975568661 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4880c8da0 0x7fd488156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.055+0000 7fd48d505700 1 --2- 172.21.15.46:0/3975568661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880caf30 0x7fd4880b2330 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fd48400c5b0 tx=0x7fd484001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.056+0000 7fd479ffb700 1 -- 172.21.15.46:0/3975568661 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd484005b30 con 0x7fd4880caf30 2024-01-26T23:49:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.056+0000 7fd479ffb700 1 -- 172.21.15.46:0/3975568661 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd484005c90 con 0x7fd4880caf30 2024-01-26T23:49:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.056+0000 7fd48ff6a700 1 -- 172.21.15.46:0/3975568661 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4880c6410 con 0x7fd4880caf30 2024-01-26T23:49:15.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.056+0000 7fd479ffb700 1 -- 172.21.15.46:0/3975568661 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd484007300 con 0x7fd4880caf30 2024-01-26T23:49:15.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.056+0000 7fd48ff6a700 1 -- 172.21.15.46:0/3975568661 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd488157090 con 0x7fd4880caf30 2024-01-26T23:49:15.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.057+0000 7fd48ff6a700 1 -- 172.21.15.46:0/3975568661 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd48804c710 con 0x7fd4880caf30 2024-01-26T23:49:15.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.057+0000 7fd479ffb700 1 -- 172.21.15.46:0/3975568661 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd48401b3b0 con 0x7fd4880caf30 2024-01-26T23:49:15.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.058+0000 7fd479ffb700 1 --2- 172.21.15.46:0/3975568661 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd46c066f30 0x7fd46c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:15.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.058+0000 7fd479ffb700 1 -- 172.21.15.46:0/3975568661 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6281+0+0 (secure 0 0 0) 0x7fd484083c70 con 0x7fd4880caf30 2024-01-26T23:49:15.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.058+0000 7fd48dd06700 1 --2- 172.21.15.46:0/3975568661 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd46c066f30 0x7fd46c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:15.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.058+0000 7fd48dd06700 1 --2- 172.21.15.46:0/3975568661 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd46c066f30 0x7fd46c0693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd47c0093f0 tx=0x7fd47c017000 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:15.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.062+0000 7fd479ffb700 1 -- 172.21.15.46:0/3975568661 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd484050ca0 con 0x7fd4880caf30 2024-01-26T23:49:15.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.268+0000 7fd48ff6a700 1 -- 172.21.15.46:0/3975568661 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd4880cc890 con 0x7fd4880caf30 2024-01-26T23:49:15.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.272+0000 7fd479ffb700 1 -- 172.21.15.46:0/3975568661 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd484017020 con 0x7fd4880caf30 2024-01-26T23:49:15.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.275+0000 7fd48ff6a700 1 -- 172.21.15.46:0/3975568661 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd46c066f30 msgr2=0x7fd46c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:15.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.275+0000 7fd48ff6a700 1 --2- 172.21.15.46:0/3975568661 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd46c066f30 0x7fd46c0693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd47c0093f0 tx=0x7fd47c017000 comp rx=0 tx=0).stop 2024-01-26T23:49:15.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.275+0000 7fd48ff6a700 1 -- 172.21.15.46:0/3975568661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880caf30 msgr2=0x7fd4880b2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:15.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.275+0000 7fd48ff6a700 1 --2- 172.21.15.46:0/3975568661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880caf30 0x7fd4880b2330 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fd48400c5b0 tx=0x7fd484001070 comp rx=0 tx=0).stop 2024-01-26T23:49:15.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.275+0000 7fd48ff6a700 1 -- 172.21.15.46:0/3975568661 shutdown_connections 2024-01-26T23:49:15.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.275+0000 7fd48ff6a700 1 --2- 172.21.15.46:0/3975568661 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd46c066f30 0x7fd46c0693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.275+0000 7fd48ff6a700 1 --2- 172.21.15.46:0/3975568661 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd488140af0 0x7fd4880b2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.275+0000 7fd48ff6a700 1 --2- 172.21.15.46:0/3975568661 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4880c8da0 0x7fd488156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.275+0000 7fd48ff6a700 1 --2- 172.21.15.46:0/3975568661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880caf30 0x7fd4880b2330 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.275+0000 7fd48ff6a700 1 -- 172.21.15.46:0/3975568661 >> 172.21.15.46:0/3975568661 conn(0x7fd48806bd90 msgr2=0x7fd48813dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:15.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.275+0000 7fd48ff6a700 1 -- 172.21.15.46:0/3975568661 shutdown_connections 2024-01-26T23:49:15.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.275+0000 7fd48ff6a700 1 -- 172.21.15.46:0/3975568661 wait complete. 2024-01-26T23:49:15.288 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-26T23:49:15.289 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:49:15.335 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:49:15.381 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-26T23:49:15.382 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-26T23:49:15.382 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:49:15.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.517+0000 7fa586eeb700 1 Processor -- start 2024-01-26T23:49:15.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.517+0000 7fa586eeb700 1 -- start start 2024-01-26T23:49:15.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.518+0000 7fa586eeb700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa580159590 0x7fa58015d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:15.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.518+0000 7fa586eeb700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa58015dd90 con 0x7fa58006c3c0 2024-01-26T23:49:15.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.518+0000 7fa586eeb700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa58015e070 con 0x7fa58015d8f0 2024-01-26T23:49:15.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.518+0000 7fa586eeb700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa58015e350 con 0x7fa580159590 2024-01-26T23:49:15.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.518+0000 7fa585488700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa58006c3c0 0x7fa580069370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36842/0 (socket says 172.21.15.46:36842) 2024-01-26T23:49:15.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.518+0000 7fa585488700 1 -- 172.21.15.46:0/140514954 learned_addr learned my addr 172.21.15.46:0/140514954 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa584c87700 1 --2- 172.21.15.46:0/140514954 >> v2:172.21.15.106:3300/0 conn(0x7fa580159590 0x7fa58015d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57f7fe700 1 -- 172.21.15.46:0/140514954 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809676113 0 0) 0x7fa58015dd90 con 0x7fa58006c3c0 2024-01-26T23:49:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57f7fe700 1 -- 172.21.15.46:0/140514954 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa56c003570 con 0x7fa58006c3c0 2024-01-26T23:49:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57f7fe700 1 -- 172.21.15.46:0/140514954 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670603439 0 0) 0x7fa58015e070 con 0x7fa58015d8f0 2024-01-26T23:49:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57f7fe700 1 -- 172.21.15.46:0/140514954 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa58015dd90 con 0x7fa58015d8f0 2024-01-26T23:49:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57f7fe700 1 -- 172.21.15.46:0/140514954 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1182642899 0 0) 0x7fa56c003570 con 0x7fa58006c3c0 2024-01-26T23:49:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57f7fe700 1 -- 172.21.15.46:0/140514954 >> v2:172.21.15.106:3300/0 conn(0x7fa580159590 msgr2=0x7fa58015d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57f7fe700 1 --2- 172.21.15.46:0/140514954 >> v2:172.21.15.106:3300/0 conn(0x7fa580159590 0x7fa58015d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57f7fe700 1 -- 172.21.15.46:0/140514954 >> v1:172.21.15.106:6789/0 conn(0x7fa58015d8f0 legacy=0x7fa58006bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57f7fe700 1 -- 172.21.15.46:0/140514954 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa580068b70 con 0x7fa58006c3c0 2024-01-26T23:49:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57f7fe700 1 -- 172.21.15.46:0/140514954 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fa5780017f0 con 0x7fa58006c3c0 2024-01-26T23:49:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa584c87700 1 --2- 172.21.15.46:0/140514954 >> v2:172.21.15.106:3300/0 conn(0x7fa580159590 0x7fa58015d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57f7fe700 1 -- 172.21.15.46:0/140514954 >> v1:172.21.15.46:6789/0 conn(0x7fa58006c3c0 legacy=0x7fa580069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57f7fe700 1 --2- 172.21.15.46:0/140514954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa56c003c40 0x7fa56c004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57f7fe700 1 --2- 172.21.15.46:0/140514954 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa56c004810 0x7fa56c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57f7fe700 1 --2- 172.21.15.46:0/140514954 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa56c005280 0x7fa56c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57f7fe700 1 -- 172.21.15.46:0/140514954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58015e350 con 0x7fa56c003c40 2024-01-26T23:49:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57f7fe700 1 -- 172.21.15.46:0/140514954 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c00c250 con 0x7fa56c004810 2024-01-26T23:49:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57f7fe700 1 -- 172.21.15.46:0/140514954 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c00c410 con 0x7fa56c005280 2024-01-26T23:49:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa584c87700 1 --2- 172.21.15.46:0/140514954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa56c003c40 0x7fa56c004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa57ffff700 1 --2- 172.21.15.46:0/140514954 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa56c004810 0x7fa56c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.519+0000 7fa585488700 1 --2- 172.21.15.46:0/140514954 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa56c005280 0x7fa56c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.520+0000 7fa584c87700 1 -- 172.21.15.46:0/140514954 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa56c005280 msgr2=0x7fa56c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.520+0000 7fa584c87700 1 --2- 172.21.15.46:0/140514954 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa56c005280 0x7fa56c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.520+0000 7fa584c87700 1 -- 172.21.15.46:0/140514954 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa56c004810 msgr2=0x7fa56c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:15.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.520+0000 7fa584c87700 1 --2- 172.21.15.46:0/140514954 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa56c004810 0x7fa56c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.520+0000 7fa584c87700 1 -- 172.21.15.46:0/140514954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c003570 con 0x7fa56c003c40 2024-01-26T23:49:15.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.520+0000 7fa585488700 1 -- 172.21.15.46:0/140514954 reap_dead start 2024-01-26T23:49:15.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.520+0000 7fa584c87700 1 --2- 172.21.15.46:0/140514954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa56c003c40 0x7fa56c004030 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fa570009df0 tx=0x7fa5700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e20c1961087fb30b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:15.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.520+0000 7fa57f7fe700 1 -- 172.21.15.46:0/140514954 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa570009740 con 0x7fa56c003c40 2024-01-26T23:49:15.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.520+0000 7fa57f7fe700 1 -- 172.21.15.46:0/140514954 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57000e920 con 0x7fa56c003c40 2024-01-26T23:49:15.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.520+0000 7fa57f7fe700 1 -- 172.21.15.46:0/140514954 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57000ece0 con 0x7fa56c003c40 2024-01-26T23:49:15.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.520+0000 7fa586eeb700 1 -- 172.21.15.46:0/140514954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa56c003c40 msgr2=0x7fa56c004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:15.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.520+0000 7fa586eeb700 1 --2- 172.21.15.46:0/140514954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa56c003c40 0x7fa56c004030 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fa570009df0 tx=0x7fa5700091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.520+0000 7fa586eeb700 1 -- 172.21.15.46:0/140514954 shutdown_connections 2024-01-26T23:49:15.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.520+0000 7fa586eeb700 1 --2- 172.21.15.46:0/140514954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa56c003c40 0x7fa56c004030 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.520+0000 7fa586eeb700 1 -- 172.21.15.46:0/140514954 >> 172.21.15.46:0/140514954 conn(0x7fa5800ce530 msgr2=0x7fa5800ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:15.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.521+0000 7fa586eeb700 1 -- 172.21.15.46:0/140514954 shutdown_connections 2024-01-26T23:49:15.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.521+0000 7fa586eeb700 1 -- 172.21.15.46:0/140514954 wait complete. 2024-01-26T23:49:15.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.522+0000 7fa586eeb700 1 Processor -- start 2024-01-26T23:49:15.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.522+0000 7fa586eeb700 1 -- start start 2024-01-26T23:49:15.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.523+0000 7fa586eeb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa56c003c40 0x7fa5801584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:15.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.523+0000 7fa586eeb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5800c5f50 0x7fa5800c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:15.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.523+0000 7fa586eeb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5800c63c0 0x7fa5800c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:15.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.523+0000 7fa586eeb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa580159e70 con 0x7fa5800c5f50 2024-01-26T23:49:15.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.523+0000 7fa586eeb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5800c3000 con 0x7fa5800c63c0 2024-01-26T23:49:15.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.523+0000 7fa586eeb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5800c6910 con 0x7fa56c003c40 2024-01-26T23:49:15.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.523+0000 7fa57ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5800c5f50 0x7fa5800c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:15.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.523+0000 7fa57ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5800c5f50 0x7fa5800c7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50766/0 (socket says 172.21.15.46:50766) 2024-01-26T23:49:15.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.523+0000 7fa57ffff700 1 -- 172.21.15.46:0/3294963733 learned_addr learned my addr 172.21.15.46:0/3294963733 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:15.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.523+0000 7fa584c87700 1 --2- 172.21.15.46:0/3294963733 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa56c003c40 0x7fa5801584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:15.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.523+0000 7fa585488700 1 --2- 172.21.15.46:0/3294963733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5800c63c0 0x7fa5800c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:15.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.523+0000 7fa57ffff700 1 -- 172.21.15.46:0/3294963733 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa56c003c40 msgr2=0x7fa5801584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:15.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.523+0000 7fa57ffff700 1 --2- 172.21.15.46:0/3294963733 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa56c003c40 0x7fa5801584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.523+0000 7fa57ffff700 1 -- 172.21.15.46:0/3294963733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5800c63c0 msgr2=0x7fa5800c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:15.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.523+0000 7fa57ffff700 1 --2- 172.21.15.46:0/3294963733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5800c63c0 0x7fa5800c7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.523+0000 7fa57ffff700 1 -- 172.21.15.46:0/3294963733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5800c32a0 con 0x7fa5800c5f50 2024-01-26T23:49:15.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.523+0000 7fa57ffff700 1 --2- 172.21.15.46:0/3294963733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5800c5f50 0x7fa5800c7880 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fa568002a80 tx=0x7fa568007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:15.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.524+0000 7fa57cff9700 1 -- 172.21.15.46:0/3294963733 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56800f870 con 0x7fa5800c5f50 2024-01-26T23:49:15.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.524+0000 7fa57cff9700 1 -- 172.21.15.46:0/3294963733 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56800f9f0 con 0x7fa5800c5f50 2024-01-26T23:49:15.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.524+0000 7fa586eeb700 1 -- 172.21.15.46:0/3294963733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5800c3580 con 0x7fa5800c5f50 2024-01-26T23:49:15.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.524+0000 7fa57cff9700 1 -- 172.21.15.46:0/3294963733 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56800fdb0 con 0x7fa5800c5f50 2024-01-26T23:49:15.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.524+0000 7fa586eeb700 1 -- 172.21.15.46:0/3294963733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5801589f0 con 0x7fa5800c5f50 2024-01-26T23:49:15.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.525+0000 7fa586eeb700 1 -- 172.21.15.46:0/3294963733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5800c3770 con 0x7fa5800c5f50 2024-01-26T23:49:15.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.525+0000 7fa57cff9700 1 -- 172.21.15.46:0/3294963733 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa568013600 con 0x7fa5800c5f50 2024-01-26T23:49:15.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.526+0000 7fa57cff9700 1 --2- 172.21.15.46:0/3294963733 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa560066f30 0x7fa5600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:15.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.526+0000 7fa57cff9700 1 -- 172.21.15.46:0/3294963733 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6281+0+0 (secure 0 0 0) 0x7fa568084dc0 con 0x7fa5800c5f50 2024-01-26T23:49:15.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.526+0000 7fa584c87700 1 --2- 172.21.15.46:0/3294963733 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa560066f30 0x7fa5600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:15.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.527+0000 7fa584c87700 1 --2- 172.21.15.46:0/3294963733 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa560066f30 0x7fa5600693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa5700091e0 tx=0x7fa57001a000 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:15.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.530+0000 7fa57cff9700 1 -- 172.21.15.46:0/3294963733 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa568051d50 con 0x7fa5800c5f50 2024-01-26T23:49:15.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.781+0000 7fa586eeb700 1 -- 172.21.15.46:0/3294963733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa58004c710 con 0x7fa5800c5f50 2024-01-26T23:49:15.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.782+0000 7fa57cff9700 1 -- 172.21.15.46:0/3294963733 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+3532 (secure 0 0 0) 0x7fa568017090 con 0x7fa5800c5f50 2024-01-26T23:49:15.782 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:15.782 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":64,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6923,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/4123353304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4123353304},{"type":"v1","addr":"172.21.15.106:6834","nonce":4123353304}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/690662483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":690662483},{"type":"v1","addr":"172.21.15.106:6836","nonce":690662483}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7791,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3035969379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3035969379},{"type":"v1","addr":"172.21.15.46:6834","nonce":3035969379}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3246057093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3246057093},{"type":"v1","addr":"172.21.15.46:6835","nonce":3246057093}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":64}],"filesystems":[]} 2024-01-26T23:49:15.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.784+0000 7fa586eeb700 1 -- 172.21.15.46:0/3294963733 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa560066f30 msgr2=0x7fa5600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:15.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.784+0000 7fa586eeb700 1 --2- 172.21.15.46:0/3294963733 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa560066f30 0x7fa5600693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa5700091e0 tx=0x7fa57001a000 comp rx=0 tx=0).stop 2024-01-26T23:49:15.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.784+0000 7fa586eeb700 1 -- 172.21.15.46:0/3294963733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5800c5f50 msgr2=0x7fa5800c7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:15.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.784+0000 7fa586eeb700 1 --2- 172.21.15.46:0/3294963733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5800c5f50 0x7fa5800c7880 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fa568002a80 tx=0x7fa568007d30 comp rx=0 tx=0).stop 2024-01-26T23:49:15.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.784+0000 7fa586eeb700 1 -- 172.21.15.46:0/3294963733 shutdown_connections 2024-01-26T23:49:15.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.784+0000 7fa586eeb700 1 --2- 172.21.15.46:0/3294963733 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa560066f30 0x7fa5600693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.784+0000 7fa586eeb700 1 --2- 172.21.15.46:0/3294963733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5800c63c0 0x7fa5800c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.784+0000 7fa586eeb700 1 --2- 172.21.15.46:0/3294963733 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa56c003c40 0x7fa5801584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.784+0000 7fa586eeb700 1 --2- 172.21.15.46:0/3294963733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5800c5f50 0x7fa5800c7880 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:15.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.784+0000 7fa586eeb700 1 -- 172.21.15.46:0/3294963733 >> 172.21.15.46:0/3294963733 conn(0x7fa5800ce530 msgr2=0x7fa5800cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:15.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.784+0000 7fa586eeb700 1 -- 172.21.15.46:0/3294963733 shutdown_connections 2024-01-26T23:49:15.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.784+0000 7fa586eeb700 1 -- 172.21.15.46:0/3294963733 wait complete. 2024-01-26T23:49:15.786 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 64 2024-01-26T23:49:15.798 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-26T23:49:15.799 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-26T23:49:15.799 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-26T23:49:15.799 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:49:15.799 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-26T23:49:15.799 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-26T23:49:15.799 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-26T23:49:15.799 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:49:15.799 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-26T23:49:15.800 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-26T23:49:15.800 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-26T23:49:15.800 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:49:15.800 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-26T23:49:15.800 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-26T23:49:15.800 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-26T23:49:15.800 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:49:15.808 INFO:tasks.ceph.mds.a:Stopped 2024-01-26T23:49:15.808 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-26T23:49:15.809 INFO:tasks.ceph.mds.c:Stopped 2024-01-26T23:49:15.809 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-26T23:49:15.811 INFO:tasks.ceph.mds.b:Stopped 2024-01-26T23:49:15.811 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-26T23:49:15.812 INFO:tasks.ceph.mds.d:Stopped 2024-01-26T23:49:15.812 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-26T23:49:15.851 INFO:tasks.ceph.mds.a:Started 2024-01-26T23:49:15.852 DEBUG:teuthology.parallel:result is None 2024-01-26T23:49:15.853 INFO:tasks.ceph.mds.c:Started 2024-01-26T23:49:15.853 DEBUG:teuthology.parallel:result is None 2024-01-26T23:49:15.854 INFO:tasks.ceph.mds.b:Started 2024-01-26T23:49:15.854 DEBUG:teuthology.parallel:result is None 2024-01-26T23:49:15.855 INFO:tasks.ceph.mds.d:Started 2024-01-26T23:49:15.855 DEBUG:teuthology.parallel:result is None 2024-01-26T23:49:15.855 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-26T23:49:15.939 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-26T23:49:15.940 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-26T23:49:15.941 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-01-26T23:49:15.941 INFO:tasks.ceph.mds.b.smithi106.stdout:starting mds.b at 2024-01-26T23:49:15.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.998+0000 7ff986ec4700 1 Processor -- start 2024-01-26T23:49:15.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.998+0000 7ff986ec4700 1 -- start start 2024-01-26T23:49:16.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.999+0000 7ff986ec4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9800c6520 0x7ff9800ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:16.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.999+0000 7ff986ec4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9800ced50 0x7ff9800cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:16.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.999+0000 7ff986ec4700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff98015a0b0 con 0x7ff9800b2b40 2024-01-26T23:49:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.999+0000 7ff986ec4700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9800af070 con 0x7ff9800ced50 2024-01-26T23:49:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.999+0000 7ff986ec4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980099fc0 con 0x7ff9800c6520 2024-01-26T23:49:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.999+0000 7ff985461700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9800ced50 0x7ff9800cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.999+0000 7ff985461700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9800ced50 0x7ff9800cf1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50792/0 (socket says 172.21.15.46:50792) 2024-01-26T23:49:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:15.999+0000 7ff985461700 1 -- 172.21.15.46:0/499337075 learned_addr learned my addr 172.21.15.46:0/499337075 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.000+0000 7ff97ffff700 1 --2- 172.21.15.46:0/499337075 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9800c6520 0x7ff9800ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.000+0000 7ff985461700 1 -- 172.21.15.46:0/499337075 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9800c6520 msgr2=0x7ff9800ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.000+0000 7ff985461700 1 --2- 172.21.15.46:0/499337075 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9800c6520 0x7ff9800ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.000+0000 7ff985461700 1 -- 172.21.15.46:0/499337075 >> v1:172.21.15.106:6789/0 conn(0x7ff9800b2b40 legacy=0x7ff9800ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.000+0000 7ff985461700 1 -- 172.21.15.46:0/499337075 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9800ae4d0 con 0x7ff9800ced50 2024-01-26T23:49:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.000+0000 7ff985461700 1 --2- 172.21.15.46:0/499337075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9800ced50 0x7ff9800cf1d0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7ff978009f20 tx=0x7ff978009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3545e1eba8cf8a84 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.000+0000 7ff97f7fe700 1 -- 172.21.15.46:0/499337075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97800e070 con 0x7ff9800ced50 2024-01-26T23:49:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.000+0000 7ff97f7fe700 1 -- 172.21.15.46:0/499337075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97800ba80 con 0x7ff9800ced50 2024-01-26T23:49:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.000+0000 7ff97f7fe700 1 -- 172.21.15.46:0/499337075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9780092e0 con 0x7ff9800ced50 2024-01-26T23:49:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.001+0000 7ff986ec4700 1 -- 172.21.15.46:0/499337075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9800ced50 msgr2=0x7ff9800cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.001+0000 7ff986ec4700 1 --2- 172.21.15.46:0/499337075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9800ced50 0x7ff9800cf1d0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7ff978009f20 tx=0x7ff978009840 comp rx=0 tx=0).stop 2024-01-26T23:49:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.001+0000 7ff986ec4700 1 -- 172.21.15.46:0/499337075 shutdown_connections 2024-01-26T23:49:16.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.001+0000 7ff986ec4700 1 --2- 172.21.15.46:0/499337075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9800ced50 0x7ff9800cf1d0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:16.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.001+0000 7ff986ec4700 1 --2- 172.21.15.46:0/499337075 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9800c6520 0x7ff9800ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:16.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.001+0000 7ff986ec4700 1 -- 172.21.15.46:0/499337075 >> 172.21.15.46:0/499337075 conn(0x7ff980099a30 msgr2=0x7ff98006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:16.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.001+0000 7ff986ec4700 1 -- 172.21.15.46:0/499337075 shutdown_connections 2024-01-26T23:49:16.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.001+0000 7ff986ec4700 1 -- 172.21.15.46:0/499337075 wait complete. 2024-01-26T23:49:16.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.002+0000 7ff986ec4700 1 Processor -- start 2024-01-26T23:49:16.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.002+0000 7ff986ec4700 1 -- start start 2024-01-26T23:49:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff986ec4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9800b2b40 0x7ff9800b5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff986ec4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9800c6520 0x7ff98015e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff986ec4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9800ced50 0x7ff98015a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff986ec4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800b3510 con 0x7ff9800ced50 2024-01-26T23:49:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff986ec4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800b4090 con 0x7ff9800c6520 2024-01-26T23:49:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff986ec4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9800b3760 con 0x7ff9800b2b40 2024-01-26T23:49:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff985461700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9800ced50 0x7ff98015a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff985461700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9800ced50 0x7ff98015a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50806/0 (socket says 172.21.15.46:50806) 2024-01-26T23:49:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff985461700 1 -- 172.21.15.46:0/225959602 learned_addr learned my addr 172.21.15.46:0/225959602 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff984c60700 1 --2- 172.21.15.46:0/225959602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9800b2b40 0x7ff9800b5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff97ffff700 1 --2- 172.21.15.46:0/225959602 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9800c6520 0x7ff98015e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff985461700 1 -- 172.21.15.46:0/225959602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9800b2b40 msgr2=0x7ff9800b5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff985461700 1 --2- 172.21.15.46:0/225959602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9800b2b40 0x7ff9800b5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff985461700 1 -- 172.21.15.46:0/225959602 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9800c6520 msgr2=0x7ff98015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff985461700 1 --2- 172.21.15.46:0/225959602 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9800c6520 0x7ff98015e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff985461700 1 -- 172.21.15.46:0/225959602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff978009ae0 con 0x7ff9800ced50 2024-01-26T23:49:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff984c60700 1 --2- 172.21.15.46:0/225959602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9800b2b40 0x7ff9800b5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.003+0000 7ff97ffff700 1 --2- 172.21.15.46:0/225959602 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9800c6520 0x7ff98015e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.004+0000 7ff985461700 1 --2- 172.21.15.46:0/225959602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9800ced50 0x7ff98015a240 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7ff97800b6c0 tx=0x7ff978018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:16.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.004+0000 7ff97cff9700 1 -- 172.21.15.46:0/225959602 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9780094e0 con 0x7ff9800ced50 2024-01-26T23:49:16.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.004+0000 7ff97cff9700 1 -- 172.21.15.46:0/225959602 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9780032f0 con 0x7ff9800ced50 2024-01-26T23:49:16.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.004+0000 7ff986ec4700 1 -- 172.21.15.46:0/225959602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9800b3a40 con 0x7ff9800ced50 2024-01-26T23:49:16.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.004+0000 7ff97cff9700 1 -- 172.21.15.46:0/225959602 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff978025690 con 0x7ff9800ced50 2024-01-26T23:49:16.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.004+0000 7ff986ec4700 1 -- 172.21.15.46:0/225959602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9800b6000 con 0x7ff9800ced50 2024-01-26T23:49:16.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.005+0000 7ff986ec4700 1 -- 172.21.15.46:0/225959602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9800b4200 con 0x7ff9800ced50 2024-01-26T23:49:16.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.005+0000 7ff97cff9700 1 -- 172.21.15.46:0/225959602 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff978003040 con 0x7ff9800ced50 2024-01-26T23:49:16.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.006+0000 7ff97cff9700 1 --2- 172.21.15.46:0/225959602 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff964066f30 0x7ff9640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:16.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.006+0000 7ff97cff9700 1 -- 172.21.15.46:0/225959602 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6281+0+0 (secure 0 0 0) 0x7ff978086410 con 0x7ff9800ced50 2024-01-26T23:49:16.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.006+0000 7ff984c60700 1 --2- 172.21.15.46:0/225959602 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff964066f30 0x7ff9640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:16.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.007+0000 7ff984c60700 1 --2- 172.21.15.46:0/225959602 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff964066f30 0x7ff9640693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff9740018f0 tx=0x7ff974009040 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:16.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.011+0000 7ff97cff9700 1 -- 172.21.15.46:0/225959602 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff978053440 con 0x7ff9800ced50 2024-01-26T23:49:16.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:16.230+0000 7ff986ec4700 1 -- 172.21.15.46:0/225959602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ff98004c710 con 0x7ff9800ced50 2024-01-26T23:49:17.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.718+0000 7ff97cff9700 1 -- 172.21.15.46:0/225959602 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v141) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff978056a60 con 0x7ff9800ced50 2024-01-26T23:49:17.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.721+0000 7ff986ec4700 1 -- 172.21.15.46:0/225959602 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff964066f30 msgr2=0x7ff9640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:17.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.721+0000 7ff986ec4700 1 --2- 172.21.15.46:0/225959602 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff964066f30 0x7ff9640693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff9740018f0 tx=0x7ff974009040 comp rx=0 tx=0).stop 2024-01-26T23:49:17.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.721+0000 7ff986ec4700 1 -- 172.21.15.46:0/225959602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9800ced50 msgr2=0x7ff98015a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:17.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.721+0000 7ff986ec4700 1 --2- 172.21.15.46:0/225959602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9800ced50 0x7ff98015a240 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7ff97800b6c0 tx=0x7ff978018000 comp rx=0 tx=0).stop 2024-01-26T23:49:17.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.721+0000 7ff986ec4700 1 -- 172.21.15.46:0/225959602 shutdown_connections 2024-01-26T23:49:17.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.721+0000 7ff986ec4700 1 --2- 172.21.15.46:0/225959602 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff964066f30 0x7ff9640693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:17.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.721+0000 7ff986ec4700 1 --2- 172.21.15.46:0/225959602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9800ced50 0x7ff98015a240 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:17.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.721+0000 7ff986ec4700 1 --2- 172.21.15.46:0/225959602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9800b2b40 0x7ff9800b5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:17.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.721+0000 7ff986ec4700 1 --2- 172.21.15.46:0/225959602 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9800c6520 0x7ff98015e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:17.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.721+0000 7ff986ec4700 1 -- 172.21.15.46:0/225959602 >> 172.21.15.46:0/225959602 conn(0x7ff980099a30 msgr2=0x7ff9800cd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:17.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.721+0000 7ff986ec4700 1 -- 172.21.15.46:0/225959602 shutdown_connections 2024-01-26T23:49:17.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.721+0000 7ff986ec4700 1 -- 172.21.15.46:0/225959602 wait complete. 2024-01-26T23:49:17.722 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-01-26T23:49:17.737 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-26T23:49:17.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.873+0000 7fac743de700 1 Processor -- start 2024-01-26T23:49:17.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.873+0000 7fac743de700 1 -- start start 2024-01-26T23:49:17.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.874+0000 7fac743de700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fac6c0b2b40 0x7fac6c0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:17.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.874+0000 7fac743de700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac6c0ae960 0x7fac6c0ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:17.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.874+0000 7fac743de700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6c15a0b0 con 0x7fac6c0aedd0 2024-01-26T23:49:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.874+0000 7fac743de700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac6c099fc0 con 0x7fac6c0b2b40 2024-01-26T23:49:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.874+0000 7fac743de700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c0ae4d0 con 0x7fac6c0ae960 2024-01-26T23:49:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.874+0000 7fac7217a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fac6c0b2b40 0x7fac6c0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.874+0000 7fac7297b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fac6c0aedd0 0x7fac6c0cb5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36874/0 (socket says 172.21.15.46:36874) 2024-01-26T23:49:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.874+0000 7fac7297b700 1 -- 172.21.15.46:0/1915809878 learned_addr learned my addr 172.21.15.46:0/1915809878 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.875+0000 7fac71979700 1 --2- 172.21.15.46:0/1915809878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac6c0ae960 0x7fac6c0ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.875+0000 7fac7217a700 1 -- 172.21.15.46:0/1915809878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac6c0ae960 msgr2=0x7fac6c0ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.875+0000 7fac7217a700 1 --2- 172.21.15.46:0/1915809878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac6c0ae960 0x7fac6c0ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.875+0000 7fac7217a700 1 -- 172.21.15.46:0/1915809878 >> v1:172.21.15.46:6789/0 conn(0x7fac6c0aedd0 legacy=0x7fac6c0cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.875+0000 7fac7217a700 1 -- 172.21.15.46:0/1915809878 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c0ad8c0 con 0x7fac6c0b2b40 2024-01-26T23:49:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.875+0000 7fac71178700 1 -- 172.21.15.46:0/1915809878 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24410030 0 0) 0x7fac6c15a0b0 con 0x7fac6c0aedd0 2024-01-26T23:49:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.875+0000 7fac7217a700 1 --2- 172.21.15.46:0/1915809878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac6c0b2b40 0x7fac6c0ad1c0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fac680098d0 tx=0x7fac68009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3de8a67ff2a9377 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:17.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.875+0000 7fac71178700 1 -- 172.21.15.46:0/1915809878 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac68009e70 con 0x7fac6c0b2b40 2024-01-26T23:49:17.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.875+0000 7fac71178700 1 -- 172.21.15.46:0/1915809878 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac68005c10 con 0x7fac6c0b2b40 2024-01-26T23:49:17.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.875+0000 7fac71178700 1 -- 172.21.15.46:0/1915809878 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6800b650 con 0x7fac6c0b2b40 2024-01-26T23:49:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.876+0000 7fac743de700 1 -- 172.21.15.46:0/1915809878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac6c0b2b40 msgr2=0x7fac6c0ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.876+0000 7fac743de700 1 --2- 172.21.15.46:0/1915809878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac6c0b2b40 0x7fac6c0ad1c0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fac680098d0 tx=0x7fac68009120 comp rx=0 tx=0).stop 2024-01-26T23:49:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.876+0000 7fac743de700 1 -- 172.21.15.46:0/1915809878 shutdown_connections 2024-01-26T23:49:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.876+0000 7fac743de700 1 --2- 172.21.15.46:0/1915809878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac6c0b2b40 0x7fac6c0ad1c0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.876+0000 7fac743de700 1 --2- 172.21.15.46:0/1915809878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac6c0ae960 0x7fac6c0ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.876+0000 7fac743de700 1 -- 172.21.15.46:0/1915809878 >> 172.21.15.46:0/1915809878 conn(0x7fac6c099a30 msgr2=0x7fac6c06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.876+0000 7fac743de700 1 -- 172.21.15.46:0/1915809878 shutdown_connections 2024-01-26T23:49:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.876+0000 7fac743de700 1 -- 172.21.15.46:0/1915809878 wait complete. 2024-01-26T23:49:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.877+0000 7fac743de700 1 Processor -- start 2024-01-26T23:49:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.877+0000 7fac743de700 1 -- start start 2024-01-26T23:49:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac743de700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac6c0ae960 0x7fac6c16ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac743de700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac6c0b2b40 0x7fac6c0b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac743de700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac6c169ea0 0x7fac6c16a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac743de700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c16a880 con 0x7fac6c0b2b40 2024-01-26T23:49:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac743de700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c1571c0 con 0x7fac6c169ea0 2024-01-26T23:49:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac743de700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c156890 con 0x7fac6c0ae960 2024-01-26T23:49:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac71979700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac6c0b2b40 0x7fac6c0b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac71979700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac6c0b2b40 0x7fac6c0b3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50824/0 (socket says 172.21.15.46:50824) 2024-01-26T23:49:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac71979700 1 -- 172.21.15.46:0/1233866975 learned_addr learned my addr 172.21.15.46:0/1233866975 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac7217a700 1 --2- 172.21.15.46:0/1233866975 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac6c0ae960 0x7fac6c16ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac7297b700 1 --2- 172.21.15.46:0/1233866975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac6c169ea0 0x7fac6c16a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac71979700 1 -- 172.21.15.46:0/1233866975 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac6c0ae960 msgr2=0x7fac6c16ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac71979700 1 --2- 172.21.15.46:0/1233866975 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac6c0ae960 0x7fac6c16ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac71979700 1 -- 172.21.15.46:0/1233866975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac6c169ea0 msgr2=0x7fac6c16a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac71979700 1 --2- 172.21.15.46:0/1233866975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac6c169ea0 0x7fac6c16a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac71979700 1 -- 172.21.15.46:0/1233866975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac680093c0 con 0x7fac6c0b2b40 2024-01-26T23:49:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac71979700 1 --2- 172.21.15.46:0/1233866975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac6c0b2b40 0x7fac6c0b3bc0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fac5c00d860 tx=0x7fac5c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.878+0000 7fac627fc700 1 -- 172.21.15.46:0/1233866975 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c009610 con 0x7fac6c0b2b40 2024-01-26T23:49:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.879+0000 7fac627fc700 1 -- 172.21.15.46:0/1233866975 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5c009770 con 0x7fac6c0b2b40 2024-01-26T23:49:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.879+0000 7fac627fc700 1 -- 172.21.15.46:0/1233866975 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c009b30 con 0x7fac6c0b2b40 2024-01-26T23:49:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.879+0000 7fac743de700 1 -- 172.21.15.46:0/1233866975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6c16aa70 con 0x7fac6c0b2b40 2024-01-26T23:49:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.879+0000 7fac743de700 1 -- 172.21.15.46:0/1233866975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6c156e50 con 0x7fac6c0b2b40 2024-01-26T23:49:17.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.880+0000 7fac743de700 1 -- 172.21.15.46:0/1233866975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6c157370 con 0x7fac6c0b2b40 2024-01-26T23:49:17.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.880+0000 7fac627fc700 1 -- 172.21.15.46:0/1233866975 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fac5c01e460 con 0x7fac6c0b2b40 2024-01-26T23:49:17.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.880+0000 7fac627fc700 1 --2- 172.21.15.46:0/1233866975 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fac50066f30 0x7fac500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:17.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.881+0000 7fac627fc700 1 -- 172.21.15.46:0/1233866975 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5565+0+0 (secure 0 0 0) 0x7fac5c084ac0 con 0x7fac6c0b2b40 2024-01-26T23:49:17.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.881+0000 7fac7217a700 1 --2- 172.21.15.46:0/1233866975 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fac50066f30 0x7fac500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:17.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.881+0000 7fac7217a700 1 --2- 172.21.15.46:0/1233866975 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fac50066f30 0x7fac500693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fac6c1588f0 tx=0x7fac68016000 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:17.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:17.885+0000 7fac627fc700 1 -- 172.21.15.46:0/1233866975 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fac6c157370 con 0x7fac6c0b2b40 2024-01-26T23:49:18.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.172+0000 7fac743de700 1 -- 172.21.15.46:0/1233866975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fac6c157330 con 0x7fac6c0b2b40 2024-01-26T23:49:18.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.173+0000 7fac627fc700 1 -- 172.21.15.46:0/1233866975 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fac5c084e60 con 0x7fac6c0b2b40 2024-01-26T23:49:18.173 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:18.173 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:49:18.173 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-01-26T23:49:18.173 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.173 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-01-26T23:49:18.173 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlM6dKKBAAPpeWeDZWrFgc/4zuxKCvHw==", 2024-01-26T23:49:18.173 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.174 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:49:18.174 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:49:18.174 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:49:18.174 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:49:18.174 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.174 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.174 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.174 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-01-26T23:49:18.174 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlmzHRNRAAflWYZ/p4AXeMy54tvLpugA==", 2024-01-26T23:49:18.174 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.174 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:49:18.174 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:49:18.174 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:49:18.175 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:49:18.175 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.175 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.175 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.175 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-01-26T23:49:18.175 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlMZxoMBAAG5kyfm+LLG86y/p2A8BfDQ==", 2024-01-26T23:49:18.175 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.175 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:49:18.175 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:49:18.175 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:49:18.175 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:49:18.175 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.175 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.175 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.176 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-01-26T23:49:18.176 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl5jx/ARAA3cwtLAOWQ/n2NG+xwEZ3QQ==", 2024-01-26T23:49:18.176 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.176 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:49:18.176 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:49:18.176 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:49:18.176 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:49:18.176 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.176 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.176 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.176 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-01-26T23:49:18.176 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4QLRlJ2OKJRAAWVhxEdQkUW20294w/KLL6g==", 2024-01-26T23:49:18.176 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.176 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:49:18.177 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:49:18.177 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:49:18.177 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.177 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.177 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.177 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-01-26T23:49:18.177 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7QLRlE2tAJRAAzNlTiCr6a2eWiabRy1SqSw==", 2024-01-26T23:49:18.177 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.177 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:49:18.177 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:49:18.177 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:49:18.177 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.177 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.177 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.177 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-01-26T23:49:18.178 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+QLRlyySaJBAA5072ba5HUNi2dYE3D02M/Q==", 2024-01-26T23:49:18.178 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.178 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:49:18.178 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:49:18.178 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:49:18.178 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.178 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.178 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.178 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-01-26T23:49:18.178 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBQLRl9+tAIxAApZk0btXIpQPWSb3BYUYsFQ==", 2024-01-26T23:49:18.178 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.178 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:49:18.178 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:49:18.178 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:49:18.179 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.179 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.179 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.179 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-01-26T23:49:18.179 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHQLRlurUtJxAAT9fwKrAWl62VOXwShg7NJg==", 2024-01-26T23:49:18.179 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.179 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:49:18.179 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:49:18.179 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:49:18.179 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.179 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.179 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.179 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-01-26T23:49:18.179 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKQLRl7aqoJRAAXu9fykWELsW3SwnahoIOLA==", 2024-01-26T23:49:18.179 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.180 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:49:18.180 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:49:18.180 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:49:18.180 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.180 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.180 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.180 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-01-26T23:49:18.180 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNQLRlWf7pJBAAopmFLIKb8s32YZs46VwbGg==", 2024-01-26T23:49:18.180 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.180 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:49:18.180 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:49:18.180 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:49:18.180 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.180 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.181 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.181 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-01-26T23:49:18.181 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQQLRlNGfhIxAAmCgmOB3QAGfSIYHBb5BUZw==", 2024-01-26T23:49:18.181 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.181 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:49:18.181 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:49:18.181 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:49:18.181 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.181 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.181 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.181 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-01-26T23:49:18.181 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRleO8wBhAA1018AcO3KcyOZxgB0Itxaw==", 2024-01-26T23:49:18.181 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.181 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:49:18.181 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-01-26T23:49:18.182 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-26T23:49:18.182 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.182 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.182 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.182 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-01-26T23:49:18.182 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlsiZEDRAAjXSamHe2bZcDqVsqXqIcGg==", 2024-01-26T23:49:18.182 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.182 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:49:18.182 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:49:18.182 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:49:18.182 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:49:18.182 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.182 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.182 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.182 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-01-26T23:49:18.183 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl6YT1CRAApXlN4j20xzwda0tDMy94Eg==", 2024-01-26T23:49:18.183 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.183 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:49:18.183 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:49:18.183 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:49:18.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:49:18.183 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.183 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.183 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.183 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-01-26T23:49:18.183 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlHvDQEhAAxMhgWJsZC5/6Af5zFV6XIA==", 2024-01-26T23:49:18.183 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.183 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:49:18.183 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:49:18.183 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:49:18.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:49:18.184 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.184 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.184 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.184 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-01-26T23:49:18.184 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlvMwDCBAAXnFyMduz7PfWdLWbjytIiQ==", 2024-01-26T23:49:18.184 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.184 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:49:18.184 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:49:18.184 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:49:18.184 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:49:18.184 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.184 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.184 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.184 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-01-26T23:49:18.185 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlhMNSJBAA38QU6DrCvFjO+u9zQ8bkEA==", 2024-01-26T23:49:18.185 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.185 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:49:18.185 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-26T23:49:18.185 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:49:18.185 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.185 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:18.185 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:18.185 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-01-26T23:49:18.185 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlI9EbIBAACTpzAO+8A/VFJZcT/WWBng==", 2024-01-26T23:49:18.185 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:49:18.185 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:49:18.185 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-26T23:49:18.185 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:49:18.185 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.186 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:18.186 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-01-26T23:49:18.186 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:49:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.175+0000 7fac743de700 1 -- 172.21.15.46:0/1233866975 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fac50066f30 msgr2=0x7fac500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.175+0000 7fac743de700 1 --2- 172.21.15.46:0/1233866975 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fac50066f30 0x7fac500693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fac6c1588f0 tx=0x7fac68016000 comp rx=0 tx=0).stop 2024-01-26T23:49:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.175+0000 7fac743de700 1 -- 172.21.15.46:0/1233866975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac6c0b2b40 msgr2=0x7fac6c0b3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.175+0000 7fac743de700 1 --2- 172.21.15.46:0/1233866975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac6c0b2b40 0x7fac6c0b3bc0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fac5c00d860 tx=0x7fac5c008040 comp rx=0 tx=0).stop 2024-01-26T23:49:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.175+0000 7fac743de700 1 -- 172.21.15.46:0/1233866975 shutdown_connections 2024-01-26T23:49:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.175+0000 7fac743de700 1 --2- 172.21.15.46:0/1233866975 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fac50066f30 0x7fac500693e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.175+0000 7fac743de700 1 --2- 172.21.15.46:0/1233866975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac6c169ea0 0x7fac6c16a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.175+0000 7fac743de700 1 --2- 172.21.15.46:0/1233866975 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac6c0ae960 0x7fac6c16ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.175+0000 7fac743de700 1 --2- 172.21.15.46:0/1233866975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac6c0b2b40 0x7fac6c0b3bc0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.175+0000 7fac743de700 1 -- 172.21.15.46:0/1233866975 >> 172.21.15.46:0/1233866975 conn(0x7fac6c099a30 msgr2=0x7fac6c0c6e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.175+0000 7fac743de700 1 -- 172.21.15.46:0/1233866975 shutdown_connections 2024-01-26T23:49:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.175+0000 7fac743de700 1 -- 172.21.15.46:0/1233866975 wait complete. 2024-01-26T23:49:18.189 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:49:18.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.320+0000 7fb89ee49700 1 Processor -- start 2024-01-26T23:49:18.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.320+0000 7fb89ee49700 1 -- start start 2024-01-26T23:49:18.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.321+0000 7fb89ee49700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb898159760 0x7fb8980b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:18.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.321+0000 7fb89ee49700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8980b12c0 0x7fb8980a9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:18.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.321+0000 7fb89ee49700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89806f940 con 0x7fb8980b3060 2024-01-26T23:49:18.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.321+0000 7fb89ee49700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb89806ed30 con 0x7fb898159760 2024-01-26T23:49:18.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.321+0000 7fb89ee49700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89806f010 con 0x7fb8980b12c0 2024-01-26T23:49:18.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.321+0000 7fb89cbe5700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb898159760 0x7fb8980b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:18.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.321+0000 7fb897fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb8980b3060 0x7fb8980b0bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36878/0 (socket says 172.21.15.46:36878) 2024-01-26T23:49:18.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.321+0000 7fb89cbe5700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb898159760 0x7fb8980b2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50838/0 (socket says 172.21.15.46:50838) 2024-01-26T23:49:18.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.321+0000 7fb897fff700 1 -- 172.21.15.46:0/1998443177 learned_addr learned my addr 172.21.15.46:0/1998443177 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:18.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.321+0000 7fb89d3e6700 1 --2- 172.21.15.46:0/1998443177 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8980b12c0 0x7fb8980a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:18.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.322+0000 7fb89cbe5700 1 -- 172.21.15.46:0/1998443177 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8980b12c0 msgr2=0x7fb8980a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:18.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.322+0000 7fb89cbe5700 1 --2- 172.21.15.46:0/1998443177 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8980b12c0 0x7fb8980a9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.322+0000 7fb89cbe5700 1 -- 172.21.15.46:0/1998443177 >> v1:172.21.15.46:6789/0 conn(0x7fb8980b3060 legacy=0x7fb8980b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:18.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.322+0000 7fb89cbe5700 1 -- 172.21.15.46:0/1998443177 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89806f2f0 con 0x7fb898159760 2024-01-26T23:49:18.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.322+0000 7fb8977fe700 1 -- 172.21.15.46:0/1998443177 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918643338 0 0) 0x7fb89806f940 con 0x7fb8980b3060 2024-01-26T23:49:18.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.322+0000 7fb89cbe5700 1 --2- 172.21.15.46:0/1998443177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb898159760 0x7fb8980b2b20 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fb8880098e0 tx=0x7fb8880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576f662447799456 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:18.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.322+0000 7fb8977fe700 1 -- 172.21.15.46:0/1998443177 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88800b640 con 0x7fb898159760 2024-01-26T23:49:18.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.322+0000 7fb8977fe700 1 -- 172.21.15.46:0/1998443177 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88800b7a0 con 0x7fb898159760 2024-01-26T23:49:18.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.322+0000 7fb8977fe700 1 -- 172.21.15.46:0/1998443177 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88800bb40 con 0x7fb898159760 2024-01-26T23:49:18.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.323+0000 7fb89ee49700 1 -- 172.21.15.46:0/1998443177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb898159760 msgr2=0x7fb8980b2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:18.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.323+0000 7fb89ee49700 1 --2- 172.21.15.46:0/1998443177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb898159760 0x7fb8980b2b20 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fb8880098e0 tx=0x7fb8880091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.323+0000 7fb89ee49700 1 -- 172.21.15.46:0/1998443177 shutdown_connections 2024-01-26T23:49:18.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.323+0000 7fb89ee49700 1 --2- 172.21.15.46:0/1998443177 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8980b12c0 0x7fb8980a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.323+0000 7fb89ee49700 1 --2- 172.21.15.46:0/1998443177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb898159760 0x7fb8980b2b20 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.323+0000 7fb89ee49700 1 -- 172.21.15.46:0/1998443177 >> 172.21.15.46:0/1998443177 conn(0x7fb898069510 msgr2=0x7fb8980aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:18.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.323+0000 7fb89ee49700 1 -- 172.21.15.46:0/1998443177 shutdown_connections 2024-01-26T23:49:18.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.323+0000 7fb89ee49700 1 -- 172.21.15.46:0/1998443177 wait complete. 2024-01-26T23:49:18.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.324+0000 7fb89ee49700 1 Processor -- start 2024-01-26T23:49:18.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.324+0000 7fb89ee49700 1 -- start start 2024-01-26T23:49:18.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.325+0000 7fb89ee49700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8980b12c0 0x7fb89814ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:18.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.325+0000 7fb89ee49700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8980b3060 0x7fb898151160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:18.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.325+0000 7fb89ee49700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb898159760 0x7fb8980adb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:18.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.325+0000 7fb89ee49700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89814fd10 con 0x7fb898159760 2024-01-26T23:49:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.325+0000 7fb89ee49700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89814f100 con 0x7fb8980b3060 2024-01-26T23:49:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.325+0000 7fb89ee49700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89814f3e0 con 0x7fb8980b12c0 2024-01-26T23:49:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.325+0000 7fb89d3e6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb898159760 0x7fb8980adb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.325+0000 7fb89d3e6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb898159760 0x7fb8980adb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50852/0 (socket says 172.21.15.46:50852) 2024-01-26T23:49:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.325+0000 7fb89d3e6700 1 -- 172.21.15.46:0/2864511853 learned_addr learned my addr 172.21.15.46:0/2864511853 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.325+0000 7fb89cbe5700 1 --2- 172.21.15.46:0/2864511853 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8980b12c0 0x7fb89814ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.325+0000 7fb897fff700 1 --2- 172.21.15.46:0/2864511853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8980b3060 0x7fb898151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.325+0000 7fb89d3e6700 1 -- 172.21.15.46:0/2864511853 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8980b12c0 msgr2=0x7fb89814ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.325+0000 7fb89d3e6700 1 --2- 172.21.15.46:0/2864511853 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8980b12c0 0x7fb89814ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.325+0000 7fb89d3e6700 1 -- 172.21.15.46:0/2864511853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8980b3060 msgr2=0x7fb898151160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.325+0000 7fb89d3e6700 1 --2- 172.21.15.46:0/2864511853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8980b3060 0x7fb898151160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.325+0000 7fb89d3e6700 1 -- 172.21.15.46:0/2864511853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb888009420 con 0x7fb898159760 2024-01-26T23:49:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.325+0000 7fb897fff700 1 --2- 172.21.15.46:0/2864511853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8980b3060 0x7fb898151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:49:18.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.326+0000 7fb89d3e6700 1 --2- 172.21.15.46:0/2864511853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb898159760 0x7fb8980adb70 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fb890009e90 tx=0x7fb89000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:18.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.326+0000 7fb894ff9700 1 -- 172.21.15.46:0/2864511853 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89000cc20 con 0x7fb898159760 2024-01-26T23:49:18.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.326+0000 7fb894ff9700 1 -- 172.21.15.46:0/2864511853 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89000f040 con 0x7fb898159760 2024-01-26T23:49:18.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.326+0000 7fb89ee49700 1 -- 172.21.15.46:0/2864511853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb89814f6c0 con 0x7fb898159760 2024-01-26T23:49:18.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.326+0000 7fb894ff9700 1 -- 172.21.15.46:0/2864511853 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb890014480 con 0x7fb898159760 2024-01-26T23:49:18.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.326+0000 7fb89ee49700 1 -- 172.21.15.46:0/2864511853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89814e310 con 0x7fb898159760 2024-01-26T23:49:18.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.327+0000 7fb89ee49700 1 -- 172.21.15.46:0/2864511853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8981500a0 con 0x7fb898159760 2024-01-26T23:49:18.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.327+0000 7fb894ff9700 1 -- 172.21.15.46:0/2864511853 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb890014830 con 0x7fb898159760 2024-01-26T23:49:18.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.328+0000 7fb894ff9700 1 --2- 172.21.15.46:0/2864511853 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb884066f30 0x7fb8840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:18.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.328+0000 7fb894ff9700 1 -- 172.21.15.46:0/2864511853 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5565+0+0 (secure 0 0 0) 0x7fb890084f30 con 0x7fb898159760 2024-01-26T23:49:18.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.328+0000 7fb89cbe5700 1 --2- 172.21.15.46:0/2864511853 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb884066f30 0x7fb8840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:18.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.329+0000 7fb89cbe5700 1 --2- 172.21.15.46:0/2864511853 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb884066f30 0x7fb8840693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb8880093f0 tx=0x7fb888017000 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:18.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.333+0000 7fb894ff9700 1 -- 172.21.15.46:0/2864511853 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb8981500a0 con 0x7fb898159760 2024-01-26T23:49:18.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.537+0000 7fb89ee49700 1 -- 172.21.15.46:0/2864511853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8981500a0 con 0x7fb898159760 2024-01-26T23:49:18.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.538+0000 7fb894ff9700 1 -- 172.21.15.46:0/2864511853 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+12943 (secure 0 0 0) 0x7fb8981500a0 con 0x7fb898159760 2024-01-26T23:49:18.538 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:18.538 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":141,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:49:17.712490+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:49:18.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.540+0000 7fb89ee49700 1 -- 172.21.15.46:0/2864511853 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb884066f30 msgr2=0x7fb8840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:18.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.540+0000 7fb89ee49700 1 --2- 172.21.15.46:0/2864511853 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb884066f30 0x7fb8840693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb8880093f0 tx=0x7fb888017000 comp rx=0 tx=0).stop 2024-01-26T23:49:18.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.540+0000 7fb89ee49700 1 -- 172.21.15.46:0/2864511853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb898159760 msgr2=0x7fb8980adb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:18.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.540+0000 7fb89ee49700 1 --2- 172.21.15.46:0/2864511853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb898159760 0x7fb8980adb70 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fb890009e90 tx=0x7fb89000b040 comp rx=0 tx=0).stop 2024-01-26T23:49:18.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.540+0000 7fb89ee49700 1 -- 172.21.15.46:0/2864511853 shutdown_connections 2024-01-26T23:49:18.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.540+0000 7fb89ee49700 1 --2- 172.21.15.46:0/2864511853 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb884066f30 0x7fb8840693e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.540+0000 7fb89ee49700 1 --2- 172.21.15.46:0/2864511853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb898159760 0x7fb8980adb70 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.540+0000 7fb89ee49700 1 --2- 172.21.15.46:0/2864511853 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb8980b12c0 0x7fb89814ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.540+0000 7fb89ee49700 1 --2- 172.21.15.46:0/2864511853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8980b3060 0x7fb898151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.540+0000 7fb89ee49700 1 -- 172.21.15.46:0/2864511853 >> 172.21.15.46:0/2864511853 conn(0x7fb898069510 msgr2=0x7fb8980b04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:18.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.540+0000 7fb89ee49700 1 -- 172.21.15.46:0/2864511853 shutdown_connections 2024-01-26T23:49:18.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.540+0000 7fb89ee49700 1 -- 172.21.15.46:0/2864511853 wait complete. 2024-01-26T23:49:18.554 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-26T23:49:18.554 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:49:18.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.688+0000 7f9741f5c700 1 Processor -- start 2024-01-26T23:49:18.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.688+0000 7f9741f5c700 1 -- start start 2024-01-26T23:49:18.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.689+0000 7f9741f5c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f973c1598e0 0x7f973c0c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:18.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.689+0000 7f9741f5c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f973c0c9120 0x7f973c0cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:18.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.689+0000 7f9741f5c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973c069260 con 0x7f973c159510 2024-01-26T23:49:18.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.689+0000 7f9741f5c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f973c0ca930 con 0x7f973c1598e0 2024-01-26T23:49:18.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.689+0000 7f9741f5c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f973c159d50 con 0x7f973c0c9120 2024-01-26T23:49:18.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.689+0000 7f973affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f973c1598e0 0x7f973c0c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:18.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.689+0000 7f973affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f973c1598e0 0x7f973c0c8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50858/0 (socket says 172.21.15.46:50858) 2024-01-26T23:49:18.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.689+0000 7f973affd700 1 -- 172.21.15.46:0/553014883 learned_addr learned my addr 172.21.15.46:0/553014883 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:18.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.689+0000 7f973bfff700 1 --2- 172.21.15.46:0/553014883 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f973c0c9120 0x7f973c0cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:18.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.689+0000 7f973affd700 1 -- 172.21.15.46:0/553014883 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f973c0c9120 msgr2=0x7f973c0cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:18.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.689+0000 7f973affd700 1 --2- 172.21.15.46:0/553014883 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f973c0c9120 0x7f973c0cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.689+0000 7f973affd700 1 -- 172.21.15.46:0/553014883 >> v1:172.21.15.106:6789/0 conn(0x7f973c159510 legacy=0x7f973c0ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:18.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.689+0000 7f973affd700 1 -- 172.21.15.46:0/553014883 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c15d670 con 0x7f973c1598e0 2024-01-26T23:49:18.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.690+0000 7f973affd700 1 --2- 172.21.15.46:0/553014883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f973c1598e0 0x7f973c0c8be0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f97300099b0 tx=0x7f9730009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ff1de4bc0b8fef server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:18.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.690+0000 7f973a7fc700 1 -- 172.21.15.46:0/553014883 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973000b860 con 0x7f973c1598e0 2024-01-26T23:49:18.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.690+0000 7f973a7fc700 1 -- 172.21.15.46:0/553014883 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973000b9c0 con 0x7f973c1598e0 2024-01-26T23:49:18.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.690+0000 7f973a7fc700 1 -- 172.21.15.46:0/553014883 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973000bd60 con 0x7f973c1598e0 2024-01-26T23:49:18.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.690+0000 7f9741f5c700 1 -- 172.21.15.46:0/553014883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f973c1598e0 msgr2=0x7f973c0c8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:18.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.690+0000 7f9741f5c700 1 --2- 172.21.15.46:0/553014883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f973c1598e0 0x7f973c0c8be0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f97300099b0 tx=0x7f9730009250 comp rx=0 tx=0).stop 2024-01-26T23:49:18.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.690+0000 7f9741f5c700 1 -- 172.21.15.46:0/553014883 shutdown_connections 2024-01-26T23:49:18.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.690+0000 7f9741f5c700 1 --2- 172.21.15.46:0/553014883 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f973c0c9120 0x7f973c0cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.690+0000 7f9741f5c700 1 --2- 172.21.15.46:0/553014883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f973c1598e0 0x7f973c0c8be0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.690+0000 7f9741f5c700 1 -- 172.21.15.46:0/553014883 >> 172.21.15.46:0/553014883 conn(0x7f973c06bd90 msgr2=0x7f973c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:18.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.690+0000 7f9741f5c700 1 -- 172.21.15.46:0/553014883 shutdown_connections 2024-01-26T23:49:18.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.690+0000 7f9741f5c700 1 -- 172.21.15.46:0/553014883 wait complete. 2024-01-26T23:49:18.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.692+0000 7f9741f5c700 1 Processor -- start 2024-01-26T23:49:18.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.692+0000 7f9741f5c700 1 -- start start 2024-01-26T23:49:18.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f9741f5c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f973c0c9120 0x7f973c14d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:18.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f9741f5c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f973c159510 0x7f973c15c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:18.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f9741f5c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f973c1598e0 0x7f973c0c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:18.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f9741f5c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c0b0e70 con 0x7f973c0c9120 2024-01-26T23:49:18.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f9741f5c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c0b27c0 con 0x7f973c1598e0 2024-01-26T23:49:18.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f9741f5c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f973c15b2d0 con 0x7f973c159510 2024-01-26T23:49:18.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f973b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f973c0c9120 0x7f973c14d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:18.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f973b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f973c0c9120 0x7f973c14d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50862/0 (socket says 172.21.15.46:50862) 2024-01-26T23:49:18.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f973b7fe700 1 -- 172.21.15.46:0/1151491610 learned_addr learned my addr 172.21.15.46:0/1151491610 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:18.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f973bfff700 1 --2- 172.21.15.46:0/1151491610 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f973c1598e0 0x7f973c0c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:18.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f973affd700 1 --2- 172.21.15.46:0/1151491610 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f973c159510 0x7f973c15c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:18.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f973b7fe700 1 -- 172.21.15.46:0/1151491610 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f973c159510 msgr2=0x7f973c15c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:18.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f973b7fe700 1 --2- 172.21.15.46:0/1151491610 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f973c159510 0x7f973c15c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f973b7fe700 1 -- 172.21.15.46:0/1151491610 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f973c1598e0 msgr2=0x7f973c0c3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:18.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f973b7fe700 1 --2- 172.21.15.46:0/1151491610 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f973c1598e0 0x7f973c0c3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f973b7fe700 1 -- 172.21.15.46:0/1151491610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97300094f0 con 0x7f973c0c9120 2024-01-26T23:49:18.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f973bfff700 1 --2- 172.21.15.46:0/1151491610 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f973c1598e0 0x7f973c0c3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:49:18.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f973b7fe700 1 --2- 172.21.15.46:0/1151491610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f973c0c9120 0x7f973c14d0b0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f972c00c590 tx=0x7f972c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:18.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f97237fe700 1 -- 172.21.15.46:0/1151491610 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c011590 con 0x7f973c0c9120 2024-01-26T23:49:18.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f97237fe700 1 -- 172.21.15.46:0/1151491610 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972c0116f0 con 0x7f973c0c9120 2024-01-26T23:49:18.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f9741f5c700 1 -- 172.21.15.46:0/1151491610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f973c15ad10 con 0x7f973c0c9120 2024-01-26T23:49:18.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.693+0000 7f97237fe700 1 -- 172.21.15.46:0/1151491610 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c011ab0 con 0x7f973c0c9120 2024-01-26T23:49:18.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.694+0000 7f9741f5c700 1 -- 172.21.15.46:0/1151491610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f973c0b2a60 con 0x7f973c0c9120 2024-01-26T23:49:18.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.695+0000 7f9741f5c700 1 -- 172.21.15.46:0/1151491610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f973c0b2620 con 0x7f973c0c9120 2024-01-26T23:49:18.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.695+0000 7f97237fe700 1 -- 172.21.15.46:0/1151491610 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f972c01b3b0 con 0x7f973c0c9120 2024-01-26T23:49:18.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.696+0000 7f97237fe700 1 --2- 172.21.15.46:0/1151491610 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9728066f30 0x7f97280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:18.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.696+0000 7f97237fe700 1 -- 172.21.15.46:0/1151491610 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5565+0+0 (secure 0 0 0) 0x7f972c0839f0 con 0x7f973c0c9120 2024-01-26T23:49:18.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.696+0000 7f973affd700 1 --2- 172.21.15.46:0/1151491610 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9728066f30 0x7f97280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:18.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.697+0000 7f973affd700 1 --2- 172.21.15.46:0/1151491610 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9728066f30 0x7f97280693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f97300099b0 tx=0x7f9730017000 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:18.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.700+0000 7f97237fe700 1 -- 172.21.15.46:0/1151491610 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f973c0b2620 con 0x7f973c0c9120 2024-01-26T23:49:18.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.904+0000 7f9741f5c700 1 -- 172.21.15.46:0/1151491610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f973c0b2620 con 0x7f973c0c9120 2024-01-26T23:49:18.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.904+0000 7f97237fe700 1 -- 172.21.15.46:0/1151491610 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v141) v1 ==== 93+0+10 (secure 0 0 0) 0x7f973c0b2620 con 0x7f973c0c9120 2024-01-26T23:49:18.905 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:49:18.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.907+0000 7f9741f5c700 1 -- 172.21.15.46:0/1151491610 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9728066f30 msgr2=0x7f97280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:18.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.907+0000 7f9741f5c700 1 --2- 172.21.15.46:0/1151491610 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9728066f30 0x7f97280693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f97300099b0 tx=0x7f9730017000 comp rx=0 tx=0).stop 2024-01-26T23:49:18.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.907+0000 7f9741f5c700 1 -- 172.21.15.46:0/1151491610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f973c0c9120 msgr2=0x7f973c14d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.907+0000 7f9741f5c700 1 --2- 172.21.15.46:0/1151491610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f973c0c9120 0x7f973c14d0b0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f972c00c590 tx=0x7f972c00c950 comp rx=0 tx=0).stop 2024-01-26T23:49:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.907+0000 7f9741f5c700 1 -- 172.21.15.46:0/1151491610 shutdown_connections 2024-01-26T23:49:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.907+0000 7f9741f5c700 1 --2- 172.21.15.46:0/1151491610 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9728066f30 0x7f97280693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.907+0000 7f9741f5c700 1 --2- 172.21.15.46:0/1151491610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f973c0c9120 0x7f973c14d0b0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.907+0000 7f9741f5c700 1 --2- 172.21.15.46:0/1151491610 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f973c1598e0 0x7f973c0c3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.907+0000 7f9741f5c700 1 --2- 172.21.15.46:0/1151491610 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f973c159510 0x7f973c15c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.907+0000 7f9741f5c700 1 -- 172.21.15.46:0/1151491610 >> 172.21.15.46:0/1151491610 conn(0x7f973c06bd90 msgr2=0x7f973c06c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.907+0000 7f9741f5c700 1 -- 172.21.15.46:0/1151491610 shutdown_connections 2024-01-26T23:49:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:18.907+0000 7f9741f5c700 1 -- 172.21.15.46:0/1151491610 wait complete. 2024-01-26T23:49:18.922 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-26T23:49:19.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.053+0000 7f89e8f6b700 1 Processor -- start 2024-01-26T23:49:19.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.053+0000 7f89e8f6b700 1 -- start start 2024-01-26T23:49:19.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.054+0000 7f89e8f6b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e415ce40 0x7f89e40d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:19.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.054+0000 7f89e8f6b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f89e415d210 0x7f89e40cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:19.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.054+0000 7f89e8f6b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e40cff70 con 0x7f89e40d1390 2024-01-26T23:49:19.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.054+0000 7f89e8f6b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e40cf400 con 0x7f89e415d210 2024-01-26T23:49:19.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.054+0000 7f89e8f6b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40cf660 con 0x7f89e415ce40 2024-01-26T23:49:19.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e2d9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f89e40d1390 0x7f89e40ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36894/0 (socket says 172.21.15.46:36894) 2024-01-26T23:49:19.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e2d9d700 1 -- 172.21.15.46:0/540616647 learned_addr learned my addr 172.21.15.46:0/540616647 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:19.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e259c700 1 --2- 172.21.15.46:0/540616647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e415ce40 0x7f89e40d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:19.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e1d9b700 1 --2- 172.21.15.46:0/540616647 >> v2:172.21.15.106:3300/0 conn(0x7f89e415d210 0x7f89e40cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:19.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e159a700 1 -- 172.21.15.46:0/540616647 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053460019 0 0) 0x7f89e40cff70 con 0x7f89e40d1390 2024-01-26T23:49:19.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e159a700 1 -- 172.21.15.46:0/540616647 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89cc003570 con 0x7f89e40d1390 2024-01-26T23:49:19.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e159a700 1 -- 172.21.15.46:0/540616647 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2631356556 0 0) 0x7f89cc003570 con 0x7f89e40d1390 2024-01-26T23:49:19.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e159a700 1 -- 172.21.15.46:0/540616647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e415ce40 msgr2=0x7f89e40d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e159a700 1 --2- 172.21.15.46:0/540616647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e415ce40 0x7f89e40d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e159a700 1 -- 172.21.15.46:0/540616647 >> v2:172.21.15.106:3300/0 conn(0x7f89e415d210 msgr2=0x7f89e40cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e159a700 1 --2- 172.21.15.46:0/540616647 >> v2:172.21.15.106:3300/0 conn(0x7f89e415d210 0x7f89e40cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e159a700 1 -- 172.21.15.46:0/540616647 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e40d0500 con 0x7f89e40d1390 2024-01-26T23:49:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e159a700 1 -- 172.21.15.46:0/540616647 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f89dc0017f0 con 0x7f89e40d1390 2024-01-26T23:49:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e159a700 1 -- 172.21.15.46:0/540616647 >> v1:172.21.15.46:6789/0 conn(0x7f89e40d1390 legacy=0x7f89e40ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e159a700 1 --2- 172.21.15.46:0/540616647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89cc003a60 0x7f89cc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e159a700 1 --2- 172.21.15.46:0/540616647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e415ce40 0x7f89cc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e159a700 1 --2- 172.21.15.46:0/540616647 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89cc004d50 0x7f89cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e159a700 1 -- 172.21.15.46:0/540616647 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40cf400 con 0x7f89cc003a60 2024-01-26T23:49:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e159a700 1 -- 172.21.15.46:0/540616647 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40cf660 con 0x7f89cc004d50 2024-01-26T23:49:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.055+0000 7f89e159a700 1 -- 172.21.15.46:0/540616647 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc00ba60 con 0x7f89e415ce40 2024-01-26T23:49:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.056+0000 7f89e259c700 1 --2- 172.21.15.46:0/540616647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89cc003a60 0x7f89cc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.056+0000 7f89e1d9b700 1 --2- 172.21.15.46:0/540616647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e415ce40 0x7f89cc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:19.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.056+0000 7f89e2d9d700 1 --2- 172.21.15.46:0/540616647 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89cc004d50 0x7f89cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:19.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.056+0000 7f89e259c700 1 -- 172.21.15.46:0/540616647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e415ce40 msgr2=0x7f89cc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:19.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.056+0000 7f89e259c700 1 --2- 172.21.15.46:0/540616647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e415ce40 0x7f89cc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.056+0000 7f89e259c700 1 -- 172.21.15.46:0/540616647 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89cc004d50 msgr2=0x7f89cc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:19.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.056+0000 7f89e259c700 1 --2- 172.21.15.46:0/540616647 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89cc004d50 0x7f89cc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.056+0000 7f89e259c700 1 -- 172.21.15.46:0/540616647 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc003570 con 0x7f89cc003a60 2024-01-26T23:49:19.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.056+0000 7f89e2d9d700 1 --2- 172.21.15.46:0/540616647 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89cc004d50 0x7f89cc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:49:19.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.056+0000 7f89e259c700 1 --2- 172.21.15.46:0/540616647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89cc003a60 0x7f89cc003e70 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009bf0 tx=0x7f89d4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f3a07f618f0532 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:19.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.056+0000 7f89e159a700 1 -- 172.21.15.46:0/540616647 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d4005a00 con 0x7f89cc003a60 2024-01-26T23:49:19.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.056+0000 7f89e159a700 1 -- 172.21.15.46:0/540616647 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d4005b60 con 0x7f89cc003a60 2024-01-26T23:49:19.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.056+0000 7f89e159a700 1 -- 172.21.15.46:0/540616647 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d400e6b0 con 0x7f89cc003a60 2024-01-26T23:49:19.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.057+0000 7f89e8f6b700 1 -- 172.21.15.46:0/540616647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89cc003a60 msgr2=0x7f89cc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:19.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.057+0000 7f89e8f6b700 1 --2- 172.21.15.46:0/540616647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89cc003a60 0x7f89cc003e70 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009bf0 tx=0x7f89d4009120 comp rx=0 tx=0).stop 2024-01-26T23:49:19.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.057+0000 7f89e2d9d700 1 -- 172.21.15.46:0/540616647 reap_dead start 2024-01-26T23:49:19.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.057+0000 7f89e8f6b700 1 -- 172.21.15.46:0/540616647 shutdown_connections 2024-01-26T23:49:19.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.057+0000 7f89e8f6b700 1 -- 172.21.15.46:0/540616647 >> 172.21.15.46:0/540616647 conn(0x7f89e406f840 msgr2=0x7f89e40d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:19.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.057+0000 7f89e8f6b700 1 -- 172.21.15.46:0/540616647 shutdown_connections 2024-01-26T23:49:19.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.057+0000 7f89e8f6b700 1 -- 172.21.15.46:0/540616647 wait complete. 2024-01-26T23:49:19.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.058+0000 7f89e8f6b700 1 Processor -- start 2024-01-26T23:49:19.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.058+0000 7f89e8f6b700 1 -- start start 2024-01-26T23:49:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.059+0000 7f89e8f6b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89cc003a60 0x7f89e416c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.059+0000 7f89e8f6b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e40b5750 0x7f89e40b5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.059+0000 7f89e8f6b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89e409b460 0x7f89e409b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.059+0000 7f89e8f6b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40b6100 con 0x7f89cc003a60 2024-01-26T23:49:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.059+0000 7f89e8f6b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e409cda0 con 0x7f89e409b460 2024-01-26T23:49:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.059+0000 7f89e8f6b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e409c190 con 0x7f89e40b5750 2024-01-26T23:49:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.059+0000 7f89e259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89cc003a60 0x7f89e416c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.059+0000 7f89e259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89cc003a60 0x7f89e416c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50868/0 (socket says 172.21.15.46:50868) 2024-01-26T23:49:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.059+0000 7f89e259c700 1 -- 172.21.15.46:0/949304256 learned_addr learned my addr 172.21.15.46:0/949304256 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.059+0000 7f89e2d9d700 1 --2- 172.21.15.46:0/949304256 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89e409b460 0x7f89e409b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.059+0000 7f89e1d9b700 1 --2- 172.21.15.46:0/949304256 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e40b5750 0x7f89e40b5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.059+0000 7f89e259c700 1 -- 172.21.15.46:0/949304256 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e40b5750 msgr2=0x7f89e40b5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.059+0000 7f89e259c700 1 --2- 172.21.15.46:0/949304256 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e40b5750 0x7f89e40b5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.060+0000 7f89e259c700 1 -- 172.21.15.46:0/949304256 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89e409b460 msgr2=0x7f89e409b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.060+0000 7f89e259c700 1 --2- 172.21.15.46:0/949304256 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89e409b460 0x7f89e409b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.060+0000 7f89e259c700 1 -- 172.21.15.46:0/949304256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e409c470 con 0x7f89cc003a60 2024-01-26T23:49:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.060+0000 7f89e259c700 1 --2- 172.21.15.46:0/949304256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89cc003a60 0x7f89e416c980 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f89cc004730 tx=0x7f89d4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.060+0000 7f89d27fc700 1 -- 172.21.15.46:0/949304256 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d400e6b0 con 0x7f89cc003a60 2024-01-26T23:49:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.060+0000 7f89d27fc700 1 -- 172.21.15.46:0/949304256 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d4006240 con 0x7f89cc003a60 2024-01-26T23:49:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.060+0000 7f89d27fc700 1 -- 172.21.15.46:0/949304256 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d4006620 con 0x7f89cc003a60 2024-01-26T23:49:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.060+0000 7f89e8f6b700 1 -- 172.21.15.46:0/949304256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e409c750 con 0x7f89cc003a60 2024-01-26T23:49:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.060+0000 7f89e8f6b700 1 -- 172.21.15.46:0/949304256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e409be10 con 0x7f89cc003a60 2024-01-26T23:49:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.061+0000 7f89e8f6b700 1 -- 172.21.15.46:0/949304256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e409cbf0 con 0x7f89cc003a60 2024-01-26T23:49:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.061+0000 7f89d27fc700 1 -- 172.21.15.46:0/949304256 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f89d4013470 con 0x7f89cc003a60 2024-01-26T23:49:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.062+0000 7f89d27fc700 1 --2- 172.21.15.46:0/949304256 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f89c4066f60 0x7f89c4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.062+0000 7f89d27fc700 1 -- 172.21.15.46:0/949304256 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5565+0+0 (secure 0 0 0) 0x7f89d4085bd0 con 0x7f89cc003a60 2024-01-26T23:49:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.062+0000 7f89e1d9b700 1 --2- 172.21.15.46:0/949304256 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f89c4066f60 0x7f89c4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.063+0000 7f89e1d9b700 1 --2- 172.21.15.46:0/949304256 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f89c4066f60 0x7f89c4069410 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f89e415cb00 tx=0x7f89d80091a0 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:19.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.067+0000 7f89d27fc700 1 -- 172.21.15.46:0/949304256 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f89d4052ed0 con 0x7f89cc003a60 2024-01-26T23:49:19.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.273+0000 7f89e8f6b700 1 -- 172.21.15.46:0/949304256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f89e40b6100 con 0x7f89cc003a60 2024-01-26T23:49:19.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.273+0000 7f89d27fc700 1 -- 172.21.15.46:0/949304256 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v141) v1 ==== 84+0+65 (secure 0 0 0) 0x7f89d40564f0 con 0x7f89cc003a60 2024-01-26T23:49:19.274 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:19.274 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-26T23:49:19.274 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:19.274 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-01-26T23:49:19.274 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-01-26T23:49:19.274 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:19.274 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-26T23:49:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.276+0000 7f89e8f6b700 1 -- 172.21.15.46:0/949304256 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f89c4066f60 msgr2=0x7f89c4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.276+0000 7f89e8f6b700 1 --2- 172.21.15.46:0/949304256 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f89c4066f60 0x7f89c4069410 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f89e415cb00 tx=0x7f89d80091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.276+0000 7f89e8f6b700 1 -- 172.21.15.46:0/949304256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89cc003a60 msgr2=0x7f89e416c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.276+0000 7f89e8f6b700 1 --2- 172.21.15.46:0/949304256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89cc003a60 0x7f89e416c980 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f89cc004730 tx=0x7f89d4019000 comp rx=0 tx=0).stop 2024-01-26T23:49:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.276+0000 7f89e8f6b700 1 -- 172.21.15.46:0/949304256 shutdown_connections 2024-01-26T23:49:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.276+0000 7f89e8f6b700 1 --2- 172.21.15.46:0/949304256 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f89c4066f60 0x7f89c4069410 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.276+0000 7f89e8f6b700 1 --2- 172.21.15.46:0/949304256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89cc003a60 0x7f89e416c980 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.276+0000 7f89e8f6b700 1 --2- 172.21.15.46:0/949304256 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89e409b460 0x7f89e409b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.276+0000 7f89e8f6b700 1 --2- 172.21.15.46:0/949304256 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f89e40b5750 0x7f89e40b5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.276+0000 7f89e8f6b700 1 -- 172.21.15.46:0/949304256 >> 172.21.15.46:0/949304256 conn(0x7f89e406f840 msgr2=0x7f89e40c6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.276+0000 7f89e8f6b700 1 -- 172.21.15.46:0/949304256 shutdown_connections 2024-01-26T23:49:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.276+0000 7f89e8f6b700 1 -- 172.21.15.46:0/949304256 wait complete. 2024-01-26T23:49:19.290 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-26T23:49:19.290 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-26T23:49:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.425+0000 7f9521b4d700 1 Processor -- start 2024-01-26T23:49:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.425+0000 7f9521b4d700 1 -- start start 2024-01-26T23:49:19.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.426+0000 7f9521b4d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f951c156b00 0x7f951c15ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:19.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.426+0000 7f9521b4d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f951c0c9d50 0x7f951c0b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:19.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.426+0000 7f9521b4d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f951c157000 con 0x7f951c0c9980 2024-01-26T23:49:19.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.426+0000 7f9521b4d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f951c06f3e0 con 0x7f951c156b00 2024-01-26T23:49:19.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.426+0000 7f9521b4d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f951c06f6c0 con 0x7f951c0c9d50 2024-01-26T23:49:19.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.426+0000 7f951affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f951c156b00 0x7f951c15ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:19.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.426+0000 7f951affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f951c156b00 0x7f951c15ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50882/0 (socket says 172.21.15.46:50882) 2024-01-26T23:49:19.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.426+0000 7f951affd700 1 -- 172.21.15.46:0/2166828518 learned_addr learned my addr 172.21.15.46:0/2166828518 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:19.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.426+0000 7f951bfff700 1 --2- 172.21.15.46:0/2166828518 >> v2:172.21.15.106:3300/0 conn(0x7f951c0c9d50 0x7f951c0b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:19.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.426+0000 7f951affd700 1 -- 172.21.15.46:0/2166828518 >> v2:172.21.15.106:3300/0 conn(0x7f951c0c9d50 msgr2=0x7f951c0b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:19.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.426+0000 7f951affd700 1 --2- 172.21.15.46:0/2166828518 >> v2:172.21.15.106:3300/0 conn(0x7f951c0c9d50 0x7f951c0b1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.426+0000 7f951affd700 1 -- 172.21.15.46:0/2166828518 >> v1:172.21.15.106:6789/0 conn(0x7f951c0c9980 legacy=0x7f951c06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:19.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.426+0000 7f951affd700 1 -- 172.21.15.46:0/2166828518 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951c06f9a0 con 0x7f951c156b00 2024-01-26T23:49:19.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.427+0000 7f951affd700 1 --2- 172.21.15.46:0/2166828518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f951c156b00 0x7f951c15ae10 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f951c0affb0 tx=0x7f9510009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e7b0c8ae704e40 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:19.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.427+0000 7f951a7fc700 1 -- 172.21.15.46:0/2166828518 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951000b860 con 0x7f951c156b00 2024-01-26T23:49:19.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.427+0000 7f951a7fc700 1 -- 172.21.15.46:0/2166828518 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951000b9c0 con 0x7f951c156b00 2024-01-26T23:49:19.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.427+0000 7f951a7fc700 1 -- 172.21.15.46:0/2166828518 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951000bd60 con 0x7f951c156b00 2024-01-26T23:49:19.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.427+0000 7f9521b4d700 1 -- 172.21.15.46:0/2166828518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f951c156b00 msgr2=0x7f951c15ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:19.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.427+0000 7f9521b4d700 1 --2- 172.21.15.46:0/2166828518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f951c156b00 0x7f951c15ae10 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f951c0affb0 tx=0x7f9510009250 comp rx=0 tx=0).stop 2024-01-26T23:49:19.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.427+0000 7f9521b4d700 1 -- 172.21.15.46:0/2166828518 shutdown_connections 2024-01-26T23:49:19.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.427+0000 7f9521b4d700 1 --2- 172.21.15.46:0/2166828518 >> v2:172.21.15.106:3300/0 conn(0x7f951c0c9d50 0x7f951c0b1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.427+0000 7f9521b4d700 1 --2- 172.21.15.46:0/2166828518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f951c156b00 0x7f951c15ae10 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.427+0000 7f9521b4d700 1 -- 172.21.15.46:0/2166828518 >> 172.21.15.46:0/2166828518 conn(0x7f951c069510 msgr2=0x7f951c0aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:19.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.427+0000 7f9521b4d700 1 -- 172.21.15.46:0/2166828518 shutdown_connections 2024-01-26T23:49:19.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.427+0000 7f9521b4d700 1 -- 172.21.15.46:0/2166828518 wait complete. 2024-01-26T23:49:19.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.429+0000 7f9521b4d700 1 Processor -- start 2024-01-26T23:49:19.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.429+0000 7f9521b4d700 1 -- start start 2024-01-26T23:49:19.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.429+0000 7f9521b4d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f951c0c9980 0x7f951c14b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:19.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.429+0000 7f9521b4d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f951c0c9d50 0x7f951c0a8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:19.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f9521b4d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f951c156b00 0x7f951c0af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:19.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f9521b4d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951c14b630 con 0x7f951c0c9d50 2024-01-26T23:49:19.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f9521b4d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951c0aed40 con 0x7f951c156b00 2024-01-26T23:49:19.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f9521b4d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951c0ae130 con 0x7f951c0c9980 2024-01-26T23:49:19.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f951affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f951c0c9d50 0x7f951c0a8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:19.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f951affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f951c0c9d50 0x7f951c0a8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50884/0 (socket says 172.21.15.46:50884) 2024-01-26T23:49:19.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f951affd700 1 -- 172.21.15.46:0/1946268403 learned_addr learned my addr 172.21.15.46:0/1946268403 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:19.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f951b7fe700 1 --2- 172.21.15.46:0/1946268403 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f951c0c9980 0x7f951c14b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:19.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f951bfff700 1 --2- 172.21.15.46:0/1946268403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f951c156b00 0x7f951c0af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:19.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f951affd700 1 -- 172.21.15.46:0/1946268403 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f951c0c9980 msgr2=0x7f951c14b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:19.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f951affd700 1 --2- 172.21.15.46:0/1946268403 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f951c0c9980 0x7f951c14b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f951affd700 1 -- 172.21.15.46:0/1946268403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f951c156b00 msgr2=0x7f951c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:19.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f951affd700 1 --2- 172.21.15.46:0/1946268403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f951c156b00 0x7f951c0af1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f951affd700 1 -- 172.21.15.46:0/1946268403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95100094f0 con 0x7f951c0c9d50 2024-01-26T23:49:19.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f951affd700 1 --2- 172.21.15.46:0/1946268403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f951c0c9d50 0x7f951c0a8b30 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f951c06d630 tx=0x7f9510017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:19.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f95037fe700 1 -- 172.21.15.46:0/1946268403 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9510011e60 con 0x7f951c0c9d50 2024-01-26T23:49:19.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f95037fe700 1 -- 172.21.15.46:0/1946268403 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951001a630 con 0x7f951c0c9d50 2024-01-26T23:49:19.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f9521b4d700 1 -- 172.21.15.46:0/1946268403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951c0ae410 con 0x7f951c0c9d50 2024-01-26T23:49:19.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f95037fe700 1 -- 172.21.15.46:0/1946268403 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951001a9f0 con 0x7f951c0c9d50 2024-01-26T23:49:19.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.430+0000 7f9521b4d700 1 -- 172.21.15.46:0/1946268403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951c0ae9d0 con 0x7f951c0c9d50 2024-01-26T23:49:19.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.431+0000 7f9521b4d700 1 -- 172.21.15.46:0/1946268403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f951c0aeed0 con 0x7f951c0c9d50 2024-01-26T23:49:19.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.432+0000 7f95037fe700 1 -- 172.21.15.46:0/1946268403 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f951002d440 con 0x7f951c0c9d50 2024-01-26T23:49:19.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.432+0000 7f95037fe700 1 --2- 172.21.15.46:0/1946268403 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9508066f30 0x7f95080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:19.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.432+0000 7f95037fe700 1 -- 172.21.15.46:0/1946268403 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5565+0+0 (secure 0 0 0) 0x7f9510085f80 con 0x7f951c0c9d50 2024-01-26T23:49:19.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.433+0000 7f951b7fe700 1 --2- 172.21.15.46:0/1946268403 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9508066f30 0x7f95080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:19.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.433+0000 7f951b7fe700 1 --2- 172.21.15.46:0/1946268403 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9508066f30 0x7f95080693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f950c003860 tx=0x7f950c009040 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:19.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.437+0000 7f95037fe700 1 -- 172.21.15.46:0/1946268403 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9510053260 con 0x7f951c0c9d50 2024-01-26T23:49:19.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.644+0000 7f9521b4d700 1 -- 172.21.15.46:0/1946268403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f951c04c710 con 0x7f951c0c9d50 2024-01-26T23:49:19.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.912+0000 7f95037fe700 1 -- 172.21.15.46:0/1946268403 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v142) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9510056880 con 0x7f951c0c9d50 2024-01-26T23:49:19.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.919+0000 7f9521b4d700 1 -- 172.21.15.46:0/1946268403 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9508066f30 msgr2=0x7f95080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:19.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.919+0000 7f9521b4d700 1 --2- 172.21.15.46:0/1946268403 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9508066f30 0x7f95080693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f950c003860 tx=0x7f950c009040 comp rx=0 tx=0).stop 2024-01-26T23:49:19.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.919+0000 7f9521b4d700 1 -- 172.21.15.46:0/1946268403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f951c0c9d50 msgr2=0x7f951c0a8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:19.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.919+0000 7f9521b4d700 1 --2- 172.21.15.46:0/1946268403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f951c0c9d50 0x7f951c0a8b30 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f951c06d630 tx=0x7f9510017000 comp rx=0 tx=0).stop 2024-01-26T23:49:19.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.920+0000 7f9521b4d700 1 -- 172.21.15.46:0/1946268403 shutdown_connections 2024-01-26T23:49:19.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.920+0000 7f9521b4d700 1 --2- 172.21.15.46:0/1946268403 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9508066f30 0x7f95080693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.920+0000 7f9521b4d700 1 --2- 172.21.15.46:0/1946268403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f951c156b00 0x7f951c0af1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.920+0000 7f9521b4d700 1 --2- 172.21.15.46:0/1946268403 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f951c0c9980 0x7f951c14b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.920+0000 7f9521b4d700 1 --2- 172.21.15.46:0/1946268403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f951c0c9d50 0x7f951c0a8b30 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:19.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.920+0000 7f9521b4d700 1 -- 172.21.15.46:0/1946268403 >> 172.21.15.46:0/1946268403 conn(0x7f951c069510 msgr2=0x7f951c06eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:19.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.921+0000 7f9521b4d700 1 -- 172.21.15.46:0/1946268403 shutdown_connections 2024-01-26T23:49:19.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:19.921+0000 7f9521b4d700 1 -- 172.21.15.46:0/1946268403 wait complete. 2024-01-26T23:49:19.923 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-01-26T23:49:19.937 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-26T23:49:20.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.081+0000 7f8361004700 1 Processor -- start 2024-01-26T23:49:20.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.081+0000 7f8361004700 1 -- start start 2024-01-26T23:49:20.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.082+0000 7f8361004700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f835c153c40 0x7f835c06d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:20.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.082+0000 7f8361004700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c15ad60 con 0x7f835c0b1e80 2024-01-26T23:49:20.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.082+0000 7f8361004700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c15a150 con 0x7f835c154010 2024-01-26T23:49:20.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.082+0000 7f8361004700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f835c0b1d20 con 0x7f835c153c40 2024-01-26T23:49:20.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.082+0000 7f835b59e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f835c0b1e80 0x7f835c0692e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36906/0 (socket says 172.21.15.46:36906) 2024-01-26T23:49:20.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.082+0000 7f835b59e700 1 -- 172.21.15.46:0/498550988 learned_addr learned my addr 172.21.15.46:0/498550988 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:20.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.082+0000 7f835ad9d700 1 --2- 172.21.15.46:0/498550988 >> v2:172.21.15.106:3300/0 conn(0x7f835c153c40 0x7f835c06d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:20.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.082+0000 7f8359d9b700 1 -- 172.21.15.46:0/498550988 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992277646 0 0) 0x7f835c15ad60 con 0x7f835c0b1e80 2024-01-26T23:49:20.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.082+0000 7f8359d9b700 1 -- 172.21.15.46:0/498550988 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8344003570 con 0x7f835c0b1e80 2024-01-26T23:49:20.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.082+0000 7f835ad9d700 1 -- 172.21.15.46:0/498550988 >> v1:172.21.15.106:6789/0 conn(0x7f835c154010 legacy=0x7f835c06dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:20.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.082+0000 7f835ad9d700 1 -- 172.21.15.46:0/498550988 >> v1:172.21.15.46:6789/0 conn(0x7f835c0b1e80 legacy=0x7f835c0692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.082+0000 7f8359d9b700 1 -- 172.21.15.46:0/498550988 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 811300402 0 0) 0x7f8344003570 con 0x7f835c0b1e80 2024-01-26T23:49:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.082+0000 7f835ad9d700 1 -- 172.21.15.46:0/498550988 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835c15a430 con 0x7f835c153c40 2024-01-26T23:49:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.083+0000 7f835ad9d700 1 --2- 172.21.15.46:0/498550988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f835c153c40 0x7f835c06d6a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f834c0098e0 tx=0x7f834c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f19f26e248fb588 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.083+0000 7f8359d9b700 1 -- 172.21.15.46:0/498550988 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83540017f0 con 0x7f835c153c40 2024-01-26T23:49:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.083+0000 7f8359d9b700 1 -- 172.21.15.46:0/498550988 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834c011420 con 0x7f835c153c40 2024-01-26T23:49:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.083+0000 7f8359d9b700 1 -- 172.21.15.46:0/498550988 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834c0117c0 con 0x7f835c153c40 2024-01-26T23:49:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.083+0000 7f8361004700 1 -- 172.21.15.46:0/498550988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f835c153c40 msgr2=0x7f835c06d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.083+0000 7f8361004700 1 --2- 172.21.15.46:0/498550988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f835c153c40 0x7f835c06d6a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f834c0098e0 tx=0x7f834c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.083+0000 7f8361004700 1 -- 172.21.15.46:0/498550988 shutdown_connections 2024-01-26T23:49:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.083+0000 7f8361004700 1 --2- 172.21.15.46:0/498550988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f835c153c40 0x7f835c06d6a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.083+0000 7f8361004700 1 -- 172.21.15.46:0/498550988 >> 172.21.15.46:0/498550988 conn(0x7f835c067930 msgr2=0x7f835c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.083+0000 7f8361004700 1 -- 172.21.15.46:0/498550988 shutdown_connections 2024-01-26T23:49:20.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.084+0000 7f8361004700 1 -- 172.21.15.46:0/498550988 wait complete. 2024-01-26T23:49:20.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.085+0000 7f8361004700 1 Processor -- start 2024-01-26T23:49:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.085+0000 7f8361004700 1 -- start start 2024-01-26T23:49:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f8361004700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f835c0b1e80 0x7f835c0cbb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f8361004700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f835c153c40 0x7f835c159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f8361004700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f835c154010 0x7f835c0b03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f8361004700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835c152590 con 0x7f835c154010 2024-01-26T23:49:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f8361004700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835c16a8e0 con 0x7f835c0b1e80 2024-01-26T23:49:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f8361004700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835c159dd0 con 0x7f835c153c40 2024-01-26T23:49:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f835b59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f835c154010 0x7f835c0b03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f835b59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f835c154010 0x7f835c0b03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50896/0 (socket says 172.21.15.46:50896) 2024-01-26T23:49:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f835b59e700 1 -- 172.21.15.46:0/349582049 learned_addr learned my addr 172.21.15.46:0/349582049 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f835ad9d700 1 --2- 172.21.15.46:0/349582049 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f835c0b1e80 0x7f835c0cbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f835a59c700 1 --2- 172.21.15.46:0/349582049 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f835c153c40 0x7f835c159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f835b59e700 1 -- 172.21.15.46:0/349582049 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f835c153c40 msgr2=0x7f835c159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:20.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f835b59e700 1 --2- 172.21.15.46:0/349582049 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f835c153c40 0x7f835c159890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:20.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f835b59e700 1 -- 172.21.15.46:0/349582049 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f835c0b1e80 msgr2=0x7f835c0cbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:20.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f835b59e700 1 --2- 172.21.15.46:0/349582049 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f835c0b1e80 0x7f835c0cbb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:20.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f835b59e700 1 -- 172.21.15.46:0/349582049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834c009420 con 0x7f835c154010 2024-01-26T23:49:20.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f835ad9d700 1 --2- 172.21.15.46:0/349582049 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f835c0b1e80 0x7f835c0cbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:20.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.086+0000 7f835b59e700 1 --2- 172.21.15.46:0/349582049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f835c154010 0x7f835c0b03d0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f835400cc10 tx=0x7f8354001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:20.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.087+0000 7f8342ffd700 1 -- 172.21.15.46:0/349582049 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8354004dc0 con 0x7f835c154010 2024-01-26T23:49:20.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.087+0000 7f8342ffd700 1 -- 172.21.15.46:0/349582049 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f835400a6a0 con 0x7f835c154010 2024-01-26T23:49:20.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.087+0000 7f8361004700 1 -- 172.21.15.46:0/349582049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f835c16a600 con 0x7f835c154010 2024-01-26T23:49:20.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.087+0000 7f8342ffd700 1 -- 172.21.15.46:0/349582049 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835400aa60 con 0x7f835c154010 2024-01-26T23:49:20.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.087+0000 7f8361004700 1 -- 172.21.15.46:0/349582049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f835c0cb620 con 0x7f835c154010 2024-01-26T23:49:20.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.088+0000 7f8361004700 1 -- 172.21.15.46:0/349582049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f835c04c710 con 0x7f835c154010 2024-01-26T23:49:20.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.088+0000 7f8342ffd700 1 -- 172.21.15.46:0/349582049 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8354007150 con 0x7f835c154010 2024-01-26T23:49:20.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.089+0000 7f8342ffd700 1 --2- 172.21.15.46:0/349582049 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8348066f30 0x7f83480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:20.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.089+0000 7f8342ffd700 1 -- 172.21.15.46:0/349582049 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5935+0+0 (secure 0 0 0) 0x7f8354084010 con 0x7f835c154010 2024-01-26T23:49:20.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.089+0000 7f835ad9d700 1 --2- 172.21.15.46:0/349582049 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8348066f30 0x7f83480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:20.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.090+0000 7f835ad9d700 1 --2- 172.21.15.46:0/349582049 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8348066f30 0x7f83480693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f834c0091e0 tx=0x7f834c016000 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:20.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.094+0000 7f8342ffd700 1 -- 172.21.15.46:0/349582049 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8354051220 con 0x7f835c154010 2024-01-26T23:49:20.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.297+0000 7f8361004700 1 -- 172.21.15.46:0/349582049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f835c0cc7c0 con 0x7f835c154010 2024-01-26T23:49:20.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.915+0000 7f8342ffd700 1 -- 172.21.15.46:0/349582049 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v143) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8354054840 con 0x7f835c154010 2024-01-26T23:49:20.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.921+0000 7f8361004700 1 -- 172.21.15.46:0/349582049 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8348066f30 msgr2=0x7f83480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:20.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.921+0000 7f8361004700 1 --2- 172.21.15.46:0/349582049 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8348066f30 0x7f83480693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f834c0091e0 tx=0x7f834c016000 comp rx=0 tx=0).stop 2024-01-26T23:49:20.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.921+0000 7f8361004700 1 -- 172.21.15.46:0/349582049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f835c154010 msgr2=0x7f835c0b03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:20.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.921+0000 7f8361004700 1 --2- 172.21.15.46:0/349582049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f835c154010 0x7f835c0b03d0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f835400cc10 tx=0x7f8354001640 comp rx=0 tx=0).stop 2024-01-26T23:49:20.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.923+0000 7f8361004700 1 -- 172.21.15.46:0/349582049 shutdown_connections 2024-01-26T23:49:20.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.923+0000 7f8361004700 1 --2- 172.21.15.46:0/349582049 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8348066f30 0x7f83480693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:20.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.923+0000 7f8361004700 1 --2- 172.21.15.46:0/349582049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f835c154010 0x7f835c0b03d0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:20.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.923+0000 7f8361004700 1 --2- 172.21.15.46:0/349582049 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f835c0b1e80 0x7f835c0cbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:20.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.923+0000 7f8361004700 1 --2- 172.21.15.46:0/349582049 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f835c153c40 0x7f835c159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:20.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.923+0000 7f8361004700 1 -- 172.21.15.46:0/349582049 >> 172.21.15.46:0/349582049 conn(0x7f835c067930 msgr2=0x7f835c15c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:20.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.929+0000 7f8361004700 1 -- 172.21.15.46:0/349582049 shutdown_connections 2024-01-26T23:49:20.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:20.929+0000 7f8361004700 1 -- 172.21.15.46:0/349582049 wait complete. 2024-01-26T23:49:20.935 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-01-26T23:49:20.952 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-26T23:49:21.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.089+0000 7f05fbf1b700 1 Processor -- start 2024-01-26T23:49:21.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.089+0000 7f05fbf1b700 1 -- start start 2024-01-26T23:49:21.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.090+0000 7f05fbf1b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f05f40cf050 0x7f05f40af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:21.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.090+0000 7f05fbf1b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05f40cf4a0 0x7f05f40af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.091+0000 7f05fbf1b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05f40cc100 con 0x7f05f4159cb0 2024-01-26T23:49:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.091+0000 7f05fbf1b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f40ccca0 con 0x7f05f40cf050 2024-01-26T23:49:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.091+0000 7f05fbf1b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f40cb850 con 0x7f05f40cf4a0 2024-01-26T23:49:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.091+0000 7f05f94b6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f05f40cf050 0x7f05f40af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.091+0000 7f05f94b6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f05f40cf050 0x7f05f40af430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40072/0 (socket says 172.21.15.46:40072) 2024-01-26T23:49:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.091+0000 7f05f94b6700 1 -- 172.21.15.46:0/4100676625 learned_addr learned my addr 172.21.15.46:0/4100676625 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.091+0000 7f05fa4b8700 1 --2- 172.21.15.46:0/4100676625 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05f40cf4a0 0x7f05f40af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.091+0000 7f05f94b6700 1 -- 172.21.15.46:0/4100676625 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05f40cf4a0 msgr2=0x7f05f40af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.091+0000 7f05f94b6700 1 --2- 172.21.15.46:0/4100676625 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05f40cf4a0 0x7f05f40af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.091+0000 7f05f94b6700 1 -- 172.21.15.46:0/4100676625 >> v1:172.21.15.106:6789/0 conn(0x7f05f4159cb0 legacy=0x7f05f40cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.091+0000 7f05f94b6700 1 -- 172.21.15.46:0/4100676625 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f40cbaf0 con 0x7f05f40cf050 2024-01-26T23:49:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.091+0000 7f05f94b6700 1 --2- 172.21.15.46:0/4100676625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05f40cf050 0x7f05f40af430 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f05e40099b0 tx=0x7f05e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d474e8aebce03a01 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:21.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.091+0000 7f05f8cb5700 1 -- 172.21.15.46:0/4100676625 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e400b860 con 0x7f05f40cf050 2024-01-26T23:49:21.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.092+0000 7f05f8cb5700 1 -- 172.21.15.46:0/4100676625 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e400b9c0 con 0x7f05f40cf050 2024-01-26T23:49:21.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.092+0000 7f05f8cb5700 1 -- 172.21.15.46:0/4100676625 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e400bd60 con 0x7f05f40cf050 2024-01-26T23:49:21.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.092+0000 7f05fbf1b700 1 -- 172.21.15.46:0/4100676625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05f40cf050 msgr2=0x7f05f40af430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:21.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.092+0000 7f05fbf1b700 1 --2- 172.21.15.46:0/4100676625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05f40cf050 0x7f05f40af430 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f05e40099b0 tx=0x7f05e4009250 comp rx=0 tx=0).stop 2024-01-26T23:49:21.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.092+0000 7f05fbf1b700 1 -- 172.21.15.46:0/4100676625 shutdown_connections 2024-01-26T23:49:21.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.092+0000 7f05fbf1b700 1 --2- 172.21.15.46:0/4100676625 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05f40cf4a0 0x7f05f40af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:21.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.092+0000 7f05fbf1b700 1 --2- 172.21.15.46:0/4100676625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05f40cf050 0x7f05f40af430 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:21.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.092+0000 7f05fbf1b700 1 -- 172.21.15.46:0/4100676625 >> 172.21.15.46:0/4100676625 conn(0x7f05f4087f60 msgr2=0x7f05f406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:21.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.092+0000 7f05fbf1b700 1 -- 172.21.15.46:0/4100676625 shutdown_connections 2024-01-26T23:49:21.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.092+0000 7f05fbf1b700 1 -- 172.21.15.46:0/4100676625 wait complete. 2024-01-26T23:49:21.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.093+0000 7f05fbf1b700 1 Processor -- start 2024-01-26T23:49:21.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.093+0000 7f05fbf1b700 1 -- start start 2024-01-26T23:49:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.094+0000 7f05fbf1b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05f40cf050 0x7f05f40aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.094+0000 7f05fbf1b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05f40cf4a0 0x7f05f4156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.094+0000 7f05fbf1b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05f4159cb0 0x7f05f4068fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.094+0000 7f05fbf1b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f4157200 con 0x7f05f4159cb0 2024-01-26T23:49:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.094+0000 7f05fbf1b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f415a490 con 0x7f05f40cf4a0 2024-01-26T23:49:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.094+0000 7f05fbf1b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f4157430 con 0x7f05f40cf050 2024-01-26T23:49:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.094+0000 7f05fa4b8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05f4159cb0 0x7f05f4068fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.094+0000 7f05fa4b8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05f4159cb0 0x7f05f4068fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40084/0 (socket says 172.21.15.46:40084) 2024-01-26T23:49:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.094+0000 7f05fa4b8700 1 -- 172.21.15.46:0/3063037260 learned_addr learned my addr 172.21.15.46:0/3063037260 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.094+0000 7f05f94b6700 1 --2- 172.21.15.46:0/3063037260 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05f40cf4a0 0x7f05f4156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.094+0000 7f05f9cb7700 1 --2- 172.21.15.46:0/3063037260 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05f40cf050 0x7f05f40aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.095+0000 7f05fa4b8700 1 -- 172.21.15.46:0/3063037260 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05f40cf050 msgr2=0x7f05f40aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.095+0000 7f05fa4b8700 1 --2- 172.21.15.46:0/3063037260 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05f40cf050 0x7f05f40aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:21.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.095+0000 7f05fa4b8700 1 -- 172.21.15.46:0/3063037260 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05f40cf4a0 msgr2=0x7f05f4156cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:21.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.095+0000 7f05fa4b8700 1 --2- 172.21.15.46:0/3063037260 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05f40cf4a0 0x7f05f4156cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:21.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.095+0000 7f05fa4b8700 1 -- 172.21.15.46:0/3063037260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e40094f0 con 0x7f05f4159cb0 2024-01-26T23:49:21.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.095+0000 7f05f9cb7700 1 --2- 172.21.15.46:0/3063037260 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05f40cf050 0x7f05f40aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:21.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.095+0000 7f05fa4b8700 1 --2- 172.21.15.46:0/3063037260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05f4159cb0 0x7f05f4068fc0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009fa0 tx=0x7f05ec009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:21.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.095+0000 7f05e9ffb700 1 -- 172.21.15.46:0/3063037260 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec00ec70 con 0x7f05f4159cb0 2024-01-26T23:49:21.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.095+0000 7f05e9ffb700 1 -- 172.21.15.46:0/3063037260 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ec00edd0 con 0x7f05f4159cb0 2024-01-26T23:49:21.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.095+0000 7f05fbf1b700 1 -- 172.21.15.46:0/3063037260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f40af710 con 0x7f05f4159cb0 2024-01-26T23:49:21.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.095+0000 7f05e9ffb700 1 -- 172.21.15.46:0/3063037260 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec014670 con 0x7f05f4159cb0 2024-01-26T23:49:21.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.095+0000 7f05fbf1b700 1 -- 172.21.15.46:0/3063037260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f40af430 con 0x7f05f4159cb0 2024-01-26T23:49:21.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.096+0000 7f05fbf1b700 1 -- 172.21.15.46:0/3063037260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f40d1940 con 0x7f05f4159cb0 2024-01-26T23:49:21.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.097+0000 7f05e9ffb700 1 -- 172.21.15.46:0/3063037260 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f05ec0149e0 con 0x7f05f4159cb0 2024-01-26T23:49:21.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.098+0000 7f05e9ffb700 1 --2- 172.21.15.46:0/3063037260 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f05d8066f30 0x7f05d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:21.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.098+0000 7f05e9ffb700 1 -- 172.21.15.46:0/3063037260 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f05ec084ed0 con 0x7f05f4159cb0 2024-01-26T23:49:21.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.098+0000 7f05f9cb7700 1 --2- 172.21.15.46:0/3063037260 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f05d8066f30 0x7f05d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:21.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.099+0000 7f05f9cb7700 1 --2- 172.21.15.46:0/3063037260 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f05d8066f30 0x7f05d80693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f05f0001850 tx=0x7f05f0009040 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:21.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.103+0000 7f05e9ffb700 1 -- 172.21.15.46:0/3063037260 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f05ec051ee0 con 0x7f05f4159cb0 2024-01-26T23:49:21.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.358+0000 7f05fbf1b700 1 -- 172.21.15.46:0/3063037260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f05f40ce410 con 0x7f05f4159cb0 2024-01-26T23:49:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.939+0000 7f05e9ffb700 1 -- 172.21.15.46:0/3063037260 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v66) v1 ==== 174+0+0 (secure 0 0 0) 0x7f05ec055500 con 0x7f05f4159cb0 2024-01-26T23:49:21.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.946+0000 7f05fbf1b700 1 -- 172.21.15.46:0/3063037260 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f05d8066f30 msgr2=0x7f05d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:21.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.946+0000 7f05fbf1b700 1 --2- 172.21.15.46:0/3063037260 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f05d8066f30 0x7f05d80693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f05f0001850 tx=0x7f05f0009040 comp rx=0 tx=0).stop 2024-01-26T23:49:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.946+0000 7f05fbf1b700 1 -- 172.21.15.46:0/3063037260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05f4159cb0 msgr2=0x7f05f4068fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.946+0000 7f05fbf1b700 1 --2- 172.21.15.46:0/3063037260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05f4159cb0 0x7f05f4068fc0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009fa0 tx=0x7f05ec009900 comp rx=0 tx=0).stop 2024-01-26T23:49:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.946+0000 7f05fbf1b700 1 -- 172.21.15.46:0/3063037260 shutdown_connections 2024-01-26T23:49:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.946+0000 7f05fbf1b700 1 --2- 172.21.15.46:0/3063037260 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f05d8066f30 0x7f05d80693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.946+0000 7f05fbf1b700 1 --2- 172.21.15.46:0/3063037260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05f4159cb0 0x7f05f4068fc0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.946+0000 7f05fbf1b700 1 --2- 172.21.15.46:0/3063037260 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05f40cf050 0x7f05f40aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.946+0000 7f05fbf1b700 1 --2- 172.21.15.46:0/3063037260 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05f40cf4a0 0x7f05f4156cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.946+0000 7f05fbf1b700 1 -- 172.21.15.46:0/3063037260 >> 172.21.15.46:0/3063037260 conn(0x7f05f4087f60 msgr2=0x7f05f40c6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.946+0000 7f05fbf1b700 1 -- 172.21.15.46:0/3063037260 shutdown_connections 2024-01-26T23:49:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:21.946+0000 7f05fbf1b700 1 -- 172.21.15.46:0/3063037260 wait complete. 2024-01-26T23:49:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 17 and data pool 18 2024-01-26T23:49:21.965 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:49:22.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.099+0000 7f4bdfc4b700 1 Processor -- start 2024-01-26T23:49:22.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.099+0000 7f4bdfc4b700 1 -- start start 2024-01-26T23:49:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.100+0000 7f4bdfc4b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4bd806c3c0 0x7f4bd81617a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.100+0000 7f4bdfc4b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd8162700 con 0x7f4bd80c1ab0 2024-01-26T23:49:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.100+0000 7f4bdfc4b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd80691c0 con 0x7f4bd80c16e0 2024-01-26T23:49:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.100+0000 7f4bdfc4b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd80b0e40 con 0x7f4bd806c3c0 2024-01-26T23:49:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.100+0000 7f4bde1e8700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4bd80c1ab0 0x7f4bd815d1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43406/0 (socket says 172.21.15.46:43406) 2024-01-26T23:49:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.100+0000 7f4bde1e8700 1 -- 172.21.15.46:0/3917453067 learned_addr learned my addr 172.21.15.46:0/3917453067 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.100+0000 7f4bdd1e6700 1 --2- 172.21.15.46:0/3917453067 >> v2:172.21.15.106:3300/0 conn(0x7f4bd806c3c0 0x7f4bd81617a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.101+0000 7f4bdc9e5700 1 -- 172.21.15.46:0/3917453067 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234393488 0 0) 0x7f4bd8162700 con 0x7f4bd80c1ab0 2024-01-26T23:49:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.101+0000 7f4bdc9e5700 1 -- 172.21.15.46:0/3917453067 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc4003570 con 0x7f4bd80c1ab0 2024-01-26T23:49:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.101+0000 7f4bdd1e6700 1 -- 172.21.15.46:0/3917453067 >> v1:172.21.15.106:6789/0 conn(0x7f4bd80c16e0 legacy=0x7f4bd806bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.101+0000 7f4bdc9e5700 1 -- 172.21.15.46:0/3917453067 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858910904 0 0) 0x7f4bd80691c0 con 0x7f4bd80c16e0 2024-01-26T23:49:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.101+0000 7f4bdd1e6700 1 -- 172.21.15.46:0/3917453067 >> v1:172.21.15.46:6789/0 conn(0x7f4bd80c1ab0 legacy=0x7f4bd815d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.101+0000 7f4bdd1e6700 1 -- 172.21.15.46:0/3917453067 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd8162420 con 0x7f4bd806c3c0 2024-01-26T23:49:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.101+0000 7f4bdd1e6700 1 --2- 172.21.15.46:0/3917453067 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4bd806c3c0 0x7f4bd81617a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4bc80098e0 tx=0x7f4bc80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12990be2b152abb3 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.101+0000 7f4bdc9e5700 1 -- 172.21.15.46:0/3917453067 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc800b640 con 0x7f4bd806c3c0 2024-01-26T23:49:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.101+0000 7f4bdc9e5700 1 -- 172.21.15.46:0/3917453067 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc800b7a0 con 0x7f4bd806c3c0 2024-01-26T23:49:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.101+0000 7f4bdc9e5700 1 -- 172.21.15.46:0/3917453067 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc800bb40 con 0x7f4bd806c3c0 2024-01-26T23:49:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.102+0000 7f4bdfc4b700 1 -- 172.21.15.46:0/3917453067 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4bd806c3c0 msgr2=0x7f4bd81617a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.102+0000 7f4bdfc4b700 1 --2- 172.21.15.46:0/3917453067 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4bd806c3c0 0x7f4bd81617a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4bc80098e0 tx=0x7f4bc80091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.102+0000 7f4bdfc4b700 1 -- 172.21.15.46:0/3917453067 shutdown_connections 2024-01-26T23:49:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.102+0000 7f4bdfc4b700 1 --2- 172.21.15.46:0/3917453067 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4bd806c3c0 0x7f4bd81617a0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.102+0000 7f4bdfc4b700 1 -- 172.21.15.46:0/3917453067 >> 172.21.15.46:0/3917453067 conn(0x7f4bd80ce530 msgr2=0x7f4bd80ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.102+0000 7f4bdfc4b700 1 -- 172.21.15.46:0/3917453067 shutdown_connections 2024-01-26T23:49:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.102+0000 7f4bdfc4b700 1 -- 172.21.15.46:0/3917453067 wait complete. 2024-01-26T23:49:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.103+0000 7f4bdfc4b700 1 Processor -- start 2024-01-26T23:49:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.103+0000 7f4bdfc4b700 1 -- start start 2024-01-26T23:49:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.104+0000 7f4bdfc4b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4bd806c3c0 0x7f4bd8161b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.104+0000 7f4bdfc4b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4bd80c16e0 0x7f4bd816df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.104+0000 7f4bdfc4b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bd80c1ab0 0x7f4bd80c40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.104+0000 7f4bdfc4b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd80b33d0 con 0x7f4bd80c1ab0 2024-01-26T23:49:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.104+0000 7f4bdfc4b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd80c6670 con 0x7f4bd806c3c0 2024-01-26T23:49:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.104+0000 7f4bdfc4b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd80c6950 con 0x7f4bd80c16e0 2024-01-26T23:49:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.104+0000 7f4bde1e8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bd80c1ab0 0x7f4bd80c40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.104+0000 7f4bde1e8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bd80c1ab0 0x7f4bd80c40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40088/0 (socket says 172.21.15.46:40088) 2024-01-26T23:49:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.104+0000 7f4bde1e8700 1 -- 172.21.15.46:0/1685282760 learned_addr learned my addr 172.21.15.46:0/1685282760 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.104+0000 7f4bdd9e7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4bd806c3c0 0x7f4bd8161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.104+0000 7f4bdd1e6700 1 --2- 172.21.15.46:0/1685282760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4bd80c16e0 0x7f4bd816df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.105+0000 7f4bde1e8700 1 -- 172.21.15.46:0/1685282760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4bd80c16e0 msgr2=0x7f4bd816df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.105+0000 7f4bde1e8700 1 --2- 172.21.15.46:0/1685282760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4bd80c16e0 0x7f4bd816df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.105+0000 7f4bde1e8700 1 -- 172.21.15.46:0/1685282760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4bd806c3c0 msgr2=0x7f4bd8161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.105+0000 7f4bde1e8700 1 --2- 172.21.15.46:0/1685282760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4bd806c3c0 0x7f4bd8161b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.105+0000 7f4bde1e8700 1 -- 172.21.15.46:0/1685282760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc8009420 con 0x7f4bd80c1ab0 2024-01-26T23:49:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.105+0000 7f4bdd9e7700 1 --2- 172.21.15.46:0/1685282760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4bd806c3c0 0x7f4bd8161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.105+0000 7f4bde1e8700 1 --2- 172.21.15.46:0/1685282760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bd80c1ab0 0x7f4bd80c40c0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0001460 tx=0x7f4bd0001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.105+0000 7f4bcdffb700 1 -- 172.21.15.46:0/1685282760 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd000a870 con 0x7f4bd80c1ab0 2024-01-26T23:49:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.105+0000 7f4bcdffb700 1 -- 172.21.15.46:0/1685282760 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd000a9d0 con 0x7f4bd80c1ab0 2024-01-26T23:49:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.105+0000 7f4bdfc4b700 1 -- 172.21.15.46:0/1685282760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd80c6c30 con 0x7f4bd80c1ab0 2024-01-26T23:49:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.105+0000 7f4bcdffb700 1 -- 172.21.15.46:0/1685282760 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd000ad50 con 0x7f4bd80c1ab0 2024-01-26T23:49:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.105+0000 7f4bdfc4b700 1 -- 172.21.15.46:0/1685282760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd81620d0 con 0x7f4bd80c1ab0 2024-01-26T23:49:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.106+0000 7f4bdfc4b700 1 -- 172.21.15.46:0/1685282760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd806bec0 con 0x7f4bd80c1ab0 2024-01-26T23:49:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.107+0000 7f4bcdffb700 1 -- 172.21.15.46:0/1685282760 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4bd0007300 con 0x7f4bd80c1ab0 2024-01-26T23:49:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.107+0000 7f4bcdffb700 1 --2- 172.21.15.46:0/1685282760 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4bbc066f30 0x7f4bbc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:22.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.107+0000 7f4bcdffb700 1 -- 172.21.15.46:0/1685282760 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f4bd0084c00 con 0x7f4bd80c1ab0 2024-01-26T23:49:22.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.108+0000 7f4bdd9e7700 1 --2- 172.21.15.46:0/1685282760 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4bbc066f30 0x7f4bbc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:22.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.108+0000 7f4bdd9e7700 1 --2- 172.21.15.46:0/1685282760 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4bbc066f30 0x7f4bbc0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4007fd0 tx=0x7f4bd4009040 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:22.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.112+0000 7f4bcdffb700 1 -- 172.21.15.46:0/1685282760 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4bd008b080 con 0x7f4bd80c1ab0 2024-01-26T23:49:22.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.316+0000 7f4bdfc4b700 1 -- 172.21.15.46:0/1685282760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bd8162690 con 0x7f4bd80c1ab0 2024-01-26T23:49:22.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.317+0000 7f4bcdffb700 1 -- 172.21.15.46:0/1685282760 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+16176 (secure 0 0 0) 0x7f4bd0051b10 con 0x7f4bd80c1ab0 2024-01-26T23:49:22.318 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:22.318 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":144,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:49:21.359665+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:49:19.677244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-01-26T23:49:20.330110+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:49:22.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.319+0000 7f4bdfc4b700 1 -- 172.21.15.46:0/1685282760 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4bbc066f30 msgr2=0x7f4bbc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:22.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.319+0000 7f4bdfc4b700 1 --2- 172.21.15.46:0/1685282760 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4bbc066f30 0x7f4bbc0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4007fd0 tx=0x7f4bd4009040 comp rx=0 tx=0).stop 2024-01-26T23:49:22.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.319+0000 7f4bdfc4b700 1 -- 172.21.15.46:0/1685282760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bd80c1ab0 msgr2=0x7f4bd80c40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:22.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.319+0000 7f4bdfc4b700 1 --2- 172.21.15.46:0/1685282760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bd80c1ab0 0x7f4bd80c40c0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0001460 tx=0x7f4bd0001a40 comp rx=0 tx=0).stop 2024-01-26T23:49:22.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.319+0000 7f4bdfc4b700 1 -- 172.21.15.46:0/1685282760 shutdown_connections 2024-01-26T23:49:22.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.319+0000 7f4bdfc4b700 1 --2- 172.21.15.46:0/1685282760 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4bbc066f30 0x7f4bbc0693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.320+0000 7f4bdfc4b700 1 --2- 172.21.15.46:0/1685282760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bd80c1ab0 0x7f4bd80c40c0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.320+0000 7f4bdfc4b700 1 --2- 172.21.15.46:0/1685282760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4bd806c3c0 0x7f4bd8161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.320+0000 7f4bdfc4b700 1 --2- 172.21.15.46:0/1685282760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4bd80c16e0 0x7f4bd816df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.320+0000 7f4bdfc4b700 1 -- 172.21.15.46:0/1685282760 >> 172.21.15.46:0/1685282760 conn(0x7f4bd80ce530 msgr2=0x7f4bd80ade70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:22.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.320+0000 7f4bdfc4b700 1 -- 172.21.15.46:0/1685282760 shutdown_connections 2024-01-26T23:49:22.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.320+0000 7f4bdfc4b700 1 -- 172.21.15.46:0/1685282760 wait complete. 2024-01-26T23:49:22.334 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:49:22.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.465+0000 7f7cc53af700 1 Processor -- start 2024-01-26T23:49:22.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.465+0000 7f7cc53af700 1 -- start start 2024-01-26T23:49:22.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.466+0000 7f7cc53af700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cc00b2b40 0x7f7cc00ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:22.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.466+0000 7f7cc53af700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7cc00aedd0 0x7f7cc00cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:22.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.466+0000 7f7cc53af700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc015a0b0 con 0x7f7cc00ae960 2024-01-26T23:49:22.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.466+0000 7f7cc53af700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cc0099fc0 con 0x7f7cc00aedd0 2024-01-26T23:49:22.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.466+0000 7f7cc53af700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc00ae4d0 con 0x7f7cc00b2b40 2024-01-26T23:49:22.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.466+0000 7f7cbf7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7cc00aedd0 0x7f7cc00cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:22.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.466+0000 7f7cbe7fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7cc00ae960 0x7f7cc00ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43408/0 (socket says 172.21.15.46:43408) 2024-01-26T23:49:22.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.466+0000 7f7cbe7fc700 1 -- 172.21.15.46:0/970363787 learned_addr learned my addr 172.21.15.46:0/970363787 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:22.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.466+0000 7f7cbeffd700 1 --2- 172.21.15.46:0/970363787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cc00b2b40 0x7f7cc00ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:22.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.466+0000 7f7cbf7fe700 1 -- 172.21.15.46:0/970363787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cc00b2b40 msgr2=0x7f7cc00ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:22.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.466+0000 7f7cbf7fe700 1 --2- 172.21.15.46:0/970363787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cc00b2b40 0x7f7cc00ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.466+0000 7f7cbf7fe700 1 -- 172.21.15.46:0/970363787 >> v1:172.21.15.46:6789/0 conn(0x7f7cc00ae960 legacy=0x7f7cc00ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:22.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.466+0000 7f7cbf7fe700 1 -- 172.21.15.46:0/970363787 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc00ad8c0 con 0x7f7cc00aedd0 2024-01-26T23:49:22.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.466+0000 7f7cbdffb700 1 -- 172.21.15.46:0/970363787 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793119651 0 0) 0x7f7cc015a0b0 con 0x7f7cc00ae960 2024-01-26T23:49:22.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.466+0000 7f7cbeffd700 1 --2- 172.21.15.46:0/970363787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cc00b2b40 0x7f7cc00ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:22.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.467+0000 7f7cbf7fe700 1 --2- 172.21.15.46:0/970363787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc00aedd0 0x7f7cc00cb5e0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009e60 tx=0x7f7cb8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=260284e8c6be0efe server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:22.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.467+0000 7f7cbdffb700 1 -- 172.21.15.46:0/970363787 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb800e070 con 0x7f7cc00aedd0 2024-01-26T23:49:22.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.467+0000 7f7cbdffb700 1 -- 172.21.15.46:0/970363787 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb800bc30 con 0x7f7cc00aedd0 2024-01-26T23:49:22.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.467+0000 7f7cbdffb700 1 -- 172.21.15.46:0/970363787 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8012430 con 0x7f7cc00aedd0 2024-01-26T23:49:22.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.467+0000 7f7cc53af700 1 -- 172.21.15.46:0/970363787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc00aedd0 msgr2=0x7f7cc00cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:22.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.467+0000 7f7cc53af700 1 --2- 172.21.15.46:0/970363787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc00aedd0 0x7f7cc00cb5e0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009e60 tx=0x7f7cb8009780 comp rx=0 tx=0).stop 2024-01-26T23:49:22.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.467+0000 7f7cc53af700 1 -- 172.21.15.46:0/970363787 shutdown_connections 2024-01-26T23:49:22.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.467+0000 7f7cc53af700 1 --2- 172.21.15.46:0/970363787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc00aedd0 0x7f7cc00cb5e0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.467+0000 7f7cc53af700 1 --2- 172.21.15.46:0/970363787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cc00b2b40 0x7f7cc00ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.467+0000 7f7cc53af700 1 -- 172.21.15.46:0/970363787 >> 172.21.15.46:0/970363787 conn(0x7f7cc0099a30 msgr2=0x7f7cc006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:22.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.468+0000 7f7cc53af700 1 -- 172.21.15.46:0/970363787 shutdown_connections 2024-01-26T23:49:22.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.468+0000 7f7cc53af700 1 -- 172.21.15.46:0/970363787 wait complete. 2024-01-26T23:49:22.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.469+0000 7f7cc53af700 1 Processor -- start 2024-01-26T23:49:22.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.469+0000 7f7cc53af700 1 -- start start 2024-01-26T23:49:22.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7cc53af700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cc00ae960 0x7f7cc0156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:22.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7cc53af700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cc00b2b40 0x7f7cc00afb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:22.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7cc53af700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc00b00b0 0x7f7cc00cca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:22.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7cc53af700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc00c7b70 con 0x7f7cc00b00b0 2024-01-26T23:49:22.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7cc53af700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc00b1290 con 0x7f7cc00b2b40 2024-01-26T23:49:22.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7cc53af700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc00b0680 con 0x7f7cc00ae960 2024-01-26T23:49:22.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7cbf7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc00b00b0 0x7f7cc00cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:22.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7cbf7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc00b00b0 0x7f7cc00cca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40106/0 (socket says 172.21.15.46:40106) 2024-01-26T23:49:22.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7cbf7fe700 1 -- 172.21.15.46:0/621107537 learned_addr learned my addr 172.21.15.46:0/621107537 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:22.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7cbeffd700 1 --2- 172.21.15.46:0/621107537 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cc00ae960 0x7f7cc0156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:22.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7cbe7fc700 1 --2- 172.21.15.46:0/621107537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cc00b2b40 0x7f7cc00afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:22.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7cbf7fe700 1 -- 172.21.15.46:0/621107537 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cc00ae960 msgr2=0x7f7cc0156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:22.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7cbf7fe700 1 --2- 172.21.15.46:0/621107537 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cc00ae960 0x7f7cc0156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7cbf7fe700 1 -- 172.21.15.46:0/621107537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cc00b2b40 msgr2=0x7f7cc00afb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:22.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7cbf7fe700 1 --2- 172.21.15.46:0/621107537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cc00b2b40 0x7f7cc00afb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7cbf7fe700 1 -- 172.21.15.46:0/621107537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb8009a20 con 0x7f7cc00b00b0 2024-01-26T23:49:22.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7cbf7fe700 1 --2- 172.21.15.46:0/621107537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc00b00b0 0x7f7cc00cca50 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0088e10 tx=0x7f7cb8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:22.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.470+0000 7f7ca6ffd700 1 -- 172.21.15.46:0/621107537 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8012e50 con 0x7f7cc00b00b0 2024-01-26T23:49:22.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.471+0000 7f7ca6ffd700 1 -- 172.21.15.46:0/621107537 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb801d630 con 0x7f7cc00b00b0 2024-01-26T23:49:22.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.471+0000 7f7cc53af700 1 -- 172.21.15.46:0/621107537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc00b0960 con 0x7f7cc00b00b0 2024-01-26T23:49:22.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.471+0000 7f7ca6ffd700 1 -- 172.21.15.46:0/621107537 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb801d9f0 con 0x7f7cc00b00b0 2024-01-26T23:49:22.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.471+0000 7f7cc53af700 1 -- 172.21.15.46:0/621107537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc00b0f20 con 0x7f7cc00b00b0 2024-01-26T23:49:22.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.472+0000 7f7cc53af700 1 -- 172.21.15.46:0/621107537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc00c7ce0 con 0x7f7cc00b00b0 2024-01-26T23:49:22.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.472+0000 7f7ca6ffd700 1 -- 172.21.15.46:0/621107537 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7cb8024070 con 0x7f7cc00b00b0 2024-01-26T23:49:22.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.472+0000 7f7ca6ffd700 1 --2- 172.21.15.46:0/621107537 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7cac066f30 0x7f7cac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:22.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.473+0000 7f7ca6ffd700 1 -- 172.21.15.46:0/621107537 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f7cb8086fd0 con 0x7f7cc00b00b0 2024-01-26T23:49:22.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.473+0000 7f7cbeffd700 1 --2- 172.21.15.46:0/621107537 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7cac066f30 0x7f7cac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:22.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.473+0000 7f7cbeffd700 1 --2- 172.21.15.46:0/621107537 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7cac066f30 0x7f7cac0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0006ea0 tx=0x7f7cb0008040 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:22.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.477+0000 7f7ca6ffd700 1 -- 172.21.15.46:0/621107537 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7cc00c7ce0 con 0x7f7cc00b00b0 2024-01-26T23:49:22.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.682+0000 7f7cc53af700 1 -- 172.21.15.46:0/621107537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cc00c7cb0 con 0x7f7cc00b00b0 2024-01-26T23:49:22.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.683+0000 7f7ca6ffd700 1 -- 172.21.15.46:0/621107537 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+16176 (secure 0 0 0) 0x7f7cb8053f70 con 0x7f7cc00b00b0 2024-01-26T23:49:22.684 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:22.684 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":144,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:49:21.359665+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:49:19.677244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-01-26T23:49:20.330110+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:49:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.685+0000 7f7cc53af700 1 -- 172.21.15.46:0/621107537 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7cac066f30 msgr2=0x7f7cac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.685+0000 7f7cc53af700 1 --2- 172.21.15.46:0/621107537 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7cac066f30 0x7f7cac0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0006ea0 tx=0x7f7cb0008040 comp rx=0 tx=0).stop 2024-01-26T23:49:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.685+0000 7f7cc53af700 1 -- 172.21.15.46:0/621107537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc00b00b0 msgr2=0x7f7cc00cca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.685+0000 7f7cc53af700 1 --2- 172.21.15.46:0/621107537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc00b00b0 0x7f7cc00cca50 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0088e10 tx=0x7f7cb8019000 comp rx=0 tx=0).stop 2024-01-26T23:49:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.685+0000 7f7cc53af700 1 -- 172.21.15.46:0/621107537 shutdown_connections 2024-01-26T23:49:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.685+0000 7f7cc53af700 1 --2- 172.21.15.46:0/621107537 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7cac066f30 0x7f7cac0693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.685+0000 7f7cc53af700 1 --2- 172.21.15.46:0/621107537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc00b00b0 0x7f7cc00cca50 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.685+0000 7f7cc53af700 1 --2- 172.21.15.46:0/621107537 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cc00ae960 0x7f7cc0156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.685+0000 7f7cc53af700 1 --2- 172.21.15.46:0/621107537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cc00b2b40 0x7f7cc00afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.685+0000 7f7cc53af700 1 -- 172.21.15.46:0/621107537 >> 172.21.15.46:0/621107537 conn(0x7f7cc0099a30 msgr2=0x7f7cc0069b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.685+0000 7f7cc53af700 1 -- 172.21.15.46:0/621107537 shutdown_connections 2024-01-26T23:49:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.685+0000 7f7cc53af700 1 -- 172.21.15.46:0/621107537 wait complete. 2024-01-26T23:49:22.702 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-26T23:49:22.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.836+0000 7f1b3670f700 1 Processor -- start 2024-01-26T23:49:22.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.836+0000 7f1b3670f700 1 -- start start 2024-01-26T23:49:22.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.837+0000 7f1b3670f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b301598e0 0x7f1b300c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:22.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.837+0000 7f1b3670f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1b300c9120 0x7f1b300cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.837+0000 7f1b3670f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b30069260 con 0x7f1b30159510 2024-01-26T23:49:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.837+0000 7f1b3670f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b300ca930 con 0x7f1b300c9120 2024-01-26T23:49:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.837+0000 7f1b3670f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b30159d50 con 0x7f1b301598e0 2024-01-26T23:49:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.837+0000 7f1b2ffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1b30159510 0x7f1b300ca220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43412/0 (socket says 172.21.15.46:43412) 2024-01-26T23:49:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.837+0000 7f1b2ffff700 1 -- 172.21.15.46:0/2108444602 learned_addr learned my addr 172.21.15.46:0/2108444602 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.838+0000 7f1b2f7fe700 1 --2- 172.21.15.46:0/2108444602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b301598e0 0x7f1b300c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.838+0000 7f1b34cac700 1 --2- 172.21.15.46:0/2108444602 >> v2:172.21.15.106:3300/0 conn(0x7f1b300c9120 0x7f1b300cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.838+0000 7f1b2effd700 1 -- 172.21.15.46:0/2108444602 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197192978 0 0) 0x7f1b30069260 con 0x7f1b30159510 2024-01-26T23:49:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.838+0000 7f1b2effd700 1 -- 172.21.15.46:0/2108444602 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b18003570 con 0x7f1b30159510 2024-01-26T23:49:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.838+0000 7f1b2f7fe700 1 -- 172.21.15.46:0/2108444602 >> v2:172.21.15.106:3300/0 conn(0x7f1b300c9120 msgr2=0x7f1b300cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.838+0000 7f1b2f7fe700 1 --2- 172.21.15.46:0/2108444602 >> v2:172.21.15.106:3300/0 conn(0x7f1b300c9120 0x7f1b300cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.838+0000 7f1b2effd700 1 -- 172.21.15.46:0/2108444602 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3108519253 0 0) 0x7f1b18003570 con 0x7f1b30159510 2024-01-26T23:49:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.838+0000 7f1b2f7fe700 1 -- 172.21.15.46:0/2108444602 >> v1:172.21.15.46:6789/0 conn(0x7f1b30159510 legacy=0x7f1b300ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.838+0000 7f1b2f7fe700 1 -- 172.21.15.46:0/2108444602 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3015d670 con 0x7f1b301598e0 2024-01-26T23:49:22.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.838+0000 7f1b34cac700 1 --2- 172.21.15.46:0/2108444602 >> v2:172.21.15.106:3300/0 conn(0x7f1b300c9120 0x7f1b300cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:22.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.838+0000 7f1b2f7fe700 1 --2- 172.21.15.46:0/2108444602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b301598e0 0x7f1b300c8be0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1b240098e0 tx=0x7f1b240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ea67326290b74ba server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:22.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.838+0000 7f1b2effd700 1 -- 172.21.15.46:0/2108444602 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b20001470 con 0x7f1b301598e0 2024-01-26T23:49:22.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.838+0000 7f1b2effd700 1 -- 172.21.15.46:0/2108444602 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b24011420 con 0x7f1b301598e0 2024-01-26T23:49:22.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.838+0000 7f1b2effd700 1 -- 172.21.15.46:0/2108444602 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b240117c0 con 0x7f1b301598e0 2024-01-26T23:49:22.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.839+0000 7f1b3670f700 1 -- 172.21.15.46:0/2108444602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b301598e0 msgr2=0x7f1b300c8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:22.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.839+0000 7f1b3670f700 1 --2- 172.21.15.46:0/2108444602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b301598e0 0x7f1b300c8be0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1b240098e0 tx=0x7f1b240091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.839+0000 7f1b3670f700 1 -- 172.21.15.46:0/2108444602 shutdown_connections 2024-01-26T23:49:22.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.839+0000 7f1b3670f700 1 --2- 172.21.15.46:0/2108444602 >> v2:172.21.15.106:3300/0 conn(0x7f1b300c9120 0x7f1b300cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.839+0000 7f1b3670f700 1 --2- 172.21.15.46:0/2108444602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b301598e0 0x7f1b300c8be0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.839+0000 7f1b3670f700 1 -- 172.21.15.46:0/2108444602 >> 172.21.15.46:0/2108444602 conn(0x7f1b3006bd90 msgr2=0x7f1b3006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:22.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.839+0000 7f1b3670f700 1 -- 172.21.15.46:0/2108444602 shutdown_connections 2024-01-26T23:49:22.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.839+0000 7f1b3670f700 1 -- 172.21.15.46:0/2108444602 wait complete. 2024-01-26T23:49:22.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.840+0000 7f1b3670f700 1 Processor -- start 2024-01-26T23:49:22.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.840+0000 7f1b3670f700 1 -- start start 2024-01-26T23:49:22.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.841+0000 7f1b3670f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b300c9120 0x7f1b30151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:22.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.841+0000 7f1b3670f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b30159510 0x7f1b300b3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:22.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.841+0000 7f1b3670f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b301598e0 0x7f1b3015b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:22.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.841+0000 7f1b3670f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b300b1ca0 con 0x7f1b30159510 2024-01-26T23:49:22.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.841+0000 7f1b3670f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b300b13e0 con 0x7f1b300c9120 2024-01-26T23:49:22.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.841+0000 7f1b3670f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b300b1650 con 0x7f1b301598e0 2024-01-26T23:49:22.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.841+0000 7f1b2f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b30159510 0x7f1b300b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:22.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.841+0000 7f1b2f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b30159510 0x7f1b300b3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40112/0 (socket says 172.21.15.46:40112) 2024-01-26T23:49:22.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.841+0000 7f1b2f7fe700 1 -- 172.21.15.46:0/1197807616 learned_addr learned my addr 172.21.15.46:0/1197807616 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:22.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.842+0000 7f1b2ffff700 1 --2- 172.21.15.46:0/1197807616 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b300c9120 0x7f1b30151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:22.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.842+0000 7f1b34cac700 1 --2- 172.21.15.46:0/1197807616 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b301598e0 0x7f1b3015b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:22.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.842+0000 7f1b2f7fe700 1 -- 172.21.15.46:0/1197807616 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b301598e0 msgr2=0x7f1b3015b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:22.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.842+0000 7f1b2f7fe700 1 --2- 172.21.15.46:0/1197807616 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b301598e0 0x7f1b3015b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.842+0000 7f1b2f7fe700 1 -- 172.21.15.46:0/1197807616 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b300c9120 msgr2=0x7f1b30151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:22.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.842+0000 7f1b2f7fe700 1 --2- 172.21.15.46:0/1197807616 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b300c9120 0x7f1b30151bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:22.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.842+0000 7f1b2f7fe700 1 -- 172.21.15.46:0/1197807616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b28009a20 con 0x7f1b30159510 2024-01-26T23:49:22.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.842+0000 7f1b2f7fe700 1 --2- 172.21.15.46:0/1197807616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b30159510 0x7f1b300b3630 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f1b240093f0 tx=0x7f1b24016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:22.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.842+0000 7f1b17fff700 1 -- 172.21.15.46:0/1197807616 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2401cc90 con 0x7f1b30159510 2024-01-26T23:49:22.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.842+0000 7f1b17fff700 1 -- 172.21.15.46:0/1197807616 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2401cdf0 con 0x7f1b30159510 2024-01-26T23:49:22.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.842+0000 7f1b17fff700 1 -- 172.21.15.46:0/1197807616 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b24025640 con 0x7f1b30159510 2024-01-26T23:49:22.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.842+0000 7f1b3670f700 1 -- 172.21.15.46:0/1197807616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b24009420 con 0x7f1b30159510 2024-01-26T23:49:22.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.842+0000 7f1b3670f700 1 -- 172.21.15.46:0/1197807616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b300b40b0 con 0x7f1b30159510 2024-01-26T23:49:22.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.843+0000 7f1b3670f700 1 -- 172.21.15.46:0/1197807616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b300b1e10 con 0x7f1b30159510 2024-01-26T23:49:22.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.844+0000 7f1b17fff700 1 -- 172.21.15.46:0/1197807616 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1b240193f0 con 0x7f1b30159510 2024-01-26T23:49:22.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.844+0000 7f1b17fff700 1 --2- 172.21.15.46:0/1197807616 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1b1c066f30 0x7f1b1c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:22.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.844+0000 7f1b17fff700 1 -- 172.21.15.46:0/1197807616 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f1b24085ba0 con 0x7f1b30159510 2024-01-26T23:49:22.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.844+0000 7f1b2ffff700 1 --2- 172.21.15.46:0/1197807616 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1b1c066f30 0x7f1b1c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:22.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.845+0000 7f1b2ffff700 1 --2- 172.21.15.46:0/1197807616 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1b1c066f30 0x7f1b1c0693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1b200011e0 tx=0x7f1b20001520 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:22.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:22.849+0000 7f1b17fff700 1 -- 172.21.15.46:0/1197807616 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1b24052b20 con 0x7f1b30159510 2024-01-26T23:49:23.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:23.105+0000 7f1b3670f700 1 -- 172.21.15.46:0/1197807616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1b3004c710 con 0x7f1b30159510 2024-01-26T23:49:23.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:23.957+0000 7f1b17fff700 1 -- 172.21.15.46:0/1197807616 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v69) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1b2402a0b0 con 0x7f1b30159510 2024-01-26T23:49:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:23.960+0000 7f1b3670f700 1 -- 172.21.15.46:0/1197807616 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1b1c066f30 msgr2=0x7f1b1c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:23.960+0000 7f1b3670f700 1 --2- 172.21.15.46:0/1197807616 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1b1c066f30 0x7f1b1c0693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1b200011e0 tx=0x7f1b20001520 comp rx=0 tx=0).stop 2024-01-26T23:49:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:23.960+0000 7f1b3670f700 1 -- 172.21.15.46:0/1197807616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b30159510 msgr2=0x7f1b300b3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:23.960+0000 7f1b3670f700 1 --2- 172.21.15.46:0/1197807616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b30159510 0x7f1b300b3630 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f1b240093f0 tx=0x7f1b24016000 comp rx=0 tx=0).stop 2024-01-26T23:49:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:23.960+0000 7f1b3670f700 1 -- 172.21.15.46:0/1197807616 shutdown_connections 2024-01-26T23:49:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:23.960+0000 7f1b3670f700 1 --2- 172.21.15.46:0/1197807616 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1b1c066f30 0x7f1b1c0693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:23.960+0000 7f1b3670f700 1 --2- 172.21.15.46:0/1197807616 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b301598e0 0x7f1b3015b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:23.960+0000 7f1b3670f700 1 --2- 172.21.15.46:0/1197807616 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b300c9120 0x7f1b30151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:23.960+0000 7f1b3670f700 1 --2- 172.21.15.46:0/1197807616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b30159510 0x7f1b300b3630 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:23.960+0000 7f1b3670f700 1 -- 172.21.15.46:0/1197807616 >> 172.21.15.46:0/1197807616 conn(0x7f1b3006bd90 msgr2=0x7f1b3006c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:23.960+0000 7f1b3670f700 1 -- 172.21.15.46:0/1197807616 shutdown_connections 2024-01-26T23:49:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:23.960+0000 7f1b3670f700 1 -- 172.21.15.46:0/1197807616 wait complete. 2024-01-26T23:49:23.976 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-26T23:49:24.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.114+0000 7f4f5a501700 1 Processor -- start 2024-01-26T23:49:24.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.114+0000 7f4f5a501700 1 -- start start 2024-01-26T23:49:24.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.115+0000 7f4f5a501700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f54157490 0x7f4f540cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:24.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.115+0000 7f4f5a501700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4f5406ddb0 0x7f4f540cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:24.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.115+0000 7f4f5a501700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f540cea80 con 0x7f4f54157860 2024-01-26T23:49:24.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.115+0000 7f4f5a501700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f540cdc60 con 0x7f4f5406ddb0 2024-01-26T23:49:24.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.115+0000 7f4f5a501700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f540b2ae0 con 0x7f4f54157490 2024-01-26T23:49:24.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.115+0000 7f4f58a9e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4f5406ddb0 0x7f4f540cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:24.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.115+0000 7f4f53fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f54157490 0x7f4f540cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:24.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.115+0000 7f4f58a9e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4f5406ddb0 0x7f4f540cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:51674/0 (socket says 172.21.15.46:51674) 2024-01-26T23:49:24.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.115+0000 7f4f58a9e700 1 -- 172.21.15.46:0/4251606264 learned_addr learned my addr 172.21.15.46:0/4251606264 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:24.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.115+0000 7f4f53fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f54157490 0x7f4f540cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:58708/0 (socket says 172.21.15.46:58708) 2024-01-26T23:49:24.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.115+0000 7f4f537fe700 1 --1- 172.21.15.46:0/4251606264 >> v1:172.21.15.106:6789/0 conn(0x7f4f54157860 0x7f4f5406d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:40694/0 (socket says 172.21.15.46:40694) 2024-01-26T23:49:24.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.116+0000 7f4f58a9e700 1 -- 172.21.15.46:0/4251606264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f54157490 msgr2=0x7f4f540cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:24.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.116+0000 7f4f58a9e700 1 --2- 172.21.15.46:0/4251606264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f54157490 0x7f4f540cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:24.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.116+0000 7f4f58a9e700 1 -- 172.21.15.46:0/4251606264 >> v1:172.21.15.106:6789/0 conn(0x7f4f54157860 legacy=0x7f4f5406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:24.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.116+0000 7f4f58a9e700 1 -- 172.21.15.46:0/4251606264 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f540cde70 con 0x7f4f5406ddb0 2024-01-26T23:49:24.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.116+0000 7f4f53fff700 1 --2- 172.21.15.46:0/4251606264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f54157490 0x7f4f540cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:24.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.116+0000 7f4f58a9e700 1 --2- 172.21.15.46:0/4251606264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f5406ddb0 0x7f4f540cd620 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c009f20 tx=0x7f4f4c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7f1e2d07300123c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:24.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.116+0000 7f4f52ffd700 1 -- 172.21.15.46:0/4251606264 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c00e070 con 0x7f4f5406ddb0 2024-01-26T23:49:24.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.116+0000 7f4f52ffd700 1 -- 172.21.15.46:0/4251606264 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4c00b9d0 con 0x7f4f5406ddb0 2024-01-26T23:49:24.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.116+0000 7f4f52ffd700 1 -- 172.21.15.46:0/4251606264 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c0092e0 con 0x7f4f5406ddb0 2024-01-26T23:49:24.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.117+0000 7f4f5a501700 1 -- 172.21.15.46:0/4251606264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f5406ddb0 msgr2=0x7f4f540cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:24.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.117+0000 7f4f5a501700 1 --2- 172.21.15.46:0/4251606264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f5406ddb0 0x7f4f540cd620 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c009f20 tx=0x7f4f4c009840 comp rx=0 tx=0).stop 2024-01-26T23:49:24.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.117+0000 7f4f5a501700 1 -- 172.21.15.46:0/4251606264 shutdown_connections 2024-01-26T23:49:24.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.117+0000 7f4f5a501700 1 --2- 172.21.15.46:0/4251606264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f5406ddb0 0x7f4f540cd620 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:24.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.117+0000 7f4f5a501700 1 --2- 172.21.15.46:0/4251606264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f54157490 0x7f4f540cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:24.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.117+0000 7f4f5a501700 1 -- 172.21.15.46:0/4251606264 >> 172.21.15.46:0/4251606264 conn(0x7f4f54067930 msgr2=0x7f4f54067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:24.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.117+0000 7f4f5a501700 1 -- 172.21.15.46:0/4251606264 shutdown_connections 2024-01-26T23:49:24.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.117+0000 7f4f5a501700 1 -- 172.21.15.46:0/4251606264 wait complete. 2024-01-26T23:49:24.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.118+0000 7f4f5a501700 1 Processor -- start 2024-01-26T23:49:24.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.119+0000 7f4f5a501700 1 -- start start 2024-01-26T23:49:24.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.120+0000 7f4f5a501700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f5406ddb0 0x7f4f54150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:24.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.120+0000 7f4f5a501700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f54157490 0x7f4f540c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:24.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.120+0000 7f4f5a501700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54157860 0x7f4f540c4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:24.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.120+0000 7f4f5a501700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f540ae360 con 0x7f4f54157860 2024-01-26T23:49:24.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.120+0000 7f4f5a501700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f540c40f0 con 0x7f4f54157490 2024-01-26T23:49:24.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.120+0000 7f4f5a501700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f540c43d0 con 0x7f4f5406ddb0 2024-01-26T23:49:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.120+0000 7f4f58a9e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54157860 0x7f4f540c4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.120+0000 7f4f58a9e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54157860 0x7f4f540c4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40126/0 (socket says 172.21.15.46:40126) 2024-01-26T23:49:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.120+0000 7f4f58a9e700 1 -- 172.21.15.46:0/3579649618 learned_addr learned my addr 172.21.15.46:0/3579649618 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.120+0000 7f4f53fff700 1 --2- 172.21.15.46:0/3579649618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f5406ddb0 0x7f4f54150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.120+0000 7f4f537fe700 1 --2- 172.21.15.46:0/3579649618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f54157490 0x7f4f540c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.120+0000 7f4f58a9e700 1 -- 172.21.15.46:0/3579649618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f5406ddb0 msgr2=0x7f4f54150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.120+0000 7f4f58a9e700 1 --2- 172.21.15.46:0/3579649618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f5406ddb0 0x7f4f54150fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.120+0000 7f4f58a9e700 1 -- 172.21.15.46:0/3579649618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f54157490 msgr2=0x7f4f540c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.120+0000 7f4f58a9e700 1 --2- 172.21.15.46:0/3579649618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f54157490 0x7f4f540c4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.120+0000 7f4f58a9e700 1 -- 172.21.15.46:0/3579649618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c009420 con 0x7f4f54157860 2024-01-26T23:49:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.120+0000 7f4f58a9e700 1 --2- 172.21.15.46:0/3579649618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54157860 0x7f4f540c4f90 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f4f5406d060 tx=0x7f4f4c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.121+0000 7f4f4bfff700 1 -- 172.21.15.46:0/3579649618 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c008b10 con 0x7f4f54157860 2024-01-26T23:49:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.121+0000 7f4f4bfff700 1 -- 172.21.15.46:0/3579649618 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4c008cb0 con 0x7f4f54157860 2024-01-26T23:49:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.121+0000 7f4f5a501700 1 -- 172.21.15.46:0/3579649618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4c009ae0 con 0x7f4f54157860 2024-01-26T23:49:24.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.121+0000 7f4f4bfff700 1 -- 172.21.15.46:0/3579649618 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c0266b0 con 0x7f4f54157860 2024-01-26T23:49:24.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.121+0000 7f4f5a501700 1 -- 172.21.15.46:0/3579649618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f540adff0 con 0x7f4f54157860 2024-01-26T23:49:24.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.122+0000 7f4f5a501700 1 -- 172.21.15.46:0/3579649618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f540ae510 con 0x7f4f54157860 2024-01-26T23:49:24.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.122+0000 7f4f4bfff700 1 -- 172.21.15.46:0/3579649618 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4f4c003040 con 0x7f4f54157860 2024-01-26T23:49:24.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.122+0000 7f4f4bfff700 1 --2- 172.21.15.46:0/3579649618 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f34066f30 0x7f4f340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:24.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.123+0000 7f4f4bfff700 1 -- 172.21.15.46:0/3579649618 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f4f4c0866c0 con 0x7f4f54157860 2024-01-26T23:49:24.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.123+0000 7f4f53fff700 1 --2- 172.21.15.46:0/3579649618 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f34066f30 0x7f4f340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:24.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.123+0000 7f4f53fff700 1 --2- 172.21.15.46:0/3579649618 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f34066f30 0x7f4f340693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c009f40 tx=0x7f4f3c0091a0 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:24.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.127+0000 7f4f4bfff700 1 -- 172.21.15.46:0/3579649618 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4f540ae510 con 0x7f4f54157860 2024-01-26T23:49:24.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.381+0000 7f4f5a501700 1 -- 172.21.15.46:0/3579649618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4f5404c710 con 0x7f4f54157860 2024-01-26T23:49:24.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.963+0000 7f4f4bfff700 1 -- 172.21.15.46:0/3579649618 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4f4c053660 con 0x7f4f54157860 2024-01-26T23:49:24.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.966+0000 7f4f5a501700 1 -- 172.21.15.46:0/3579649618 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f34066f30 msgr2=0x7f4f340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:24.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.966+0000 7f4f5a501700 1 --2- 172.21.15.46:0/3579649618 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f34066f30 0x7f4f340693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c009f40 tx=0x7f4f3c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:24.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.966+0000 7f4f5a501700 1 -- 172.21.15.46:0/3579649618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54157860 msgr2=0x7f4f540c4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:24.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.966+0000 7f4f5a501700 1 --2- 172.21.15.46:0/3579649618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54157860 0x7f4f540c4f90 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f4f5406d060 tx=0x7f4f4c018000 comp rx=0 tx=0).stop 2024-01-26T23:49:24.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.966+0000 7f4f5a501700 1 -- 172.21.15.46:0/3579649618 shutdown_connections 2024-01-26T23:49:24.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.966+0000 7f4f5a501700 1 --2- 172.21.15.46:0/3579649618 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f34066f30 0x7f4f340693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:24.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.966+0000 7f4f5a501700 1 --2- 172.21.15.46:0/3579649618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54157860 0x7f4f540c4f90 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:24.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.966+0000 7f4f5a501700 1 --2- 172.21.15.46:0/3579649618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f5406ddb0 0x7f4f54150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:24.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.966+0000 7f4f5a501700 1 --2- 172.21.15.46:0/3579649618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f54157490 0x7f4f540c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:24.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.966+0000 7f4f5a501700 1 -- 172.21.15.46:0/3579649618 >> 172.21.15.46:0/3579649618 conn(0x7f4f54067930 msgr2=0x7f4f54151ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:24.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.966+0000 7f4f5a501700 1 -- 172.21.15.46:0/3579649618 shutdown_connections 2024-01-26T23:49:24.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:24.966+0000 7f4f5a501700 1 -- 172.21.15.46:0/3579649618 wait complete. 2024-01-26T23:49:24.982 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:49:25.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.118+0000 7f9f1634f700 1 Processor -- start 2024-01-26T23:49:25.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.118+0000 7f9f1634f700 1 -- start start 2024-01-26T23:49:25.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.119+0000 7f9f1634f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f10159b30 0x7f9f100b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:25.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.119+0000 7f9f1634f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f1006f940 con 0x7f9f10159760 2024-01-26T23:49:25.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.119+0000 7f9f1634f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f100b3230 con 0x7f9f100b10f0 2024-01-26T23:49:25.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.119+0000 7f9f1634f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f100b1570 con 0x7f9f10159b30 2024-01-26T23:49:25.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.119+0000 7f9f0ffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9f10159760 0x7f9f100b2b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43424/0 (socket says 172.21.15.46:43424) 2024-01-26T23:49:25.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.119+0000 7f9f0ffff700 1 -- 172.21.15.46:0/3630199603 learned_addr learned my addr 172.21.15.46:0/3630199603 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:25.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.119+0000 7f9f0f7fe700 1 --2- 172.21.15.46:0/3630199603 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f10159b30 0x7f9f100b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:25.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.119+0000 7f9f0effd700 1 -- 172.21.15.46:0/3630199603 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235807807 0 0) 0x7f9f1006f940 con 0x7f9f10159760 2024-01-26T23:49:25.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.120+0000 7f9f0effd700 1 -- 172.21.15.46:0/3630199603 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ef8003570 con 0x7f9f10159760 2024-01-26T23:49:25.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.120+0000 7f9f0effd700 1 -- 172.21.15.46:0/3630199603 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227482683 0 0) 0x7f9f100b3230 con 0x7f9f100b10f0 2024-01-26T23:49:25.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.120+0000 7f9f0effd700 1 -- 172.21.15.46:0/3630199603 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f1006f940 con 0x7f9f100b10f0 2024-01-26T23:49:25.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.120+0000 7f9f0effd700 1 -- 172.21.15.46:0/3630199603 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3061519664 0 0) 0x7f9ef8003570 con 0x7f9f10159760 2024-01-26T23:49:25.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.120+0000 7f9f0f7fe700 1 -- 172.21.15.46:0/3630199603 >> v1:172.21.15.106:6789/0 conn(0x7f9f100b10f0 legacy=0x7f9f100a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.120+0000 7f9f0f7fe700 1 -- 172.21.15.46:0/3630199603 >> v1:172.21.15.46:6789/0 conn(0x7f9f10159760 legacy=0x7f9f100b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.120+0000 7f9f0f7fe700 1 -- 172.21.15.46:0/3630199603 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1006ed30 con 0x7f9f10159b30 2024-01-26T23:49:25.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.120+0000 7f9f0f7fe700 1 --2- 172.21.15.46:0/3630199603 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f10159b30 0x7f9f100b0bb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9f04009910 tx=0x7f9f040091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e4e2d5b23d1cfd server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:25.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.120+0000 7f9f0effd700 1 -- 172.21.15.46:0/3630199603 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f00001470 con 0x7f9f10159b30 2024-01-26T23:49:25.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.120+0000 7f9f0effd700 1 -- 172.21.15.46:0/3630199603 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f04011420 con 0x7f9f10159b30 2024-01-26T23:49:25.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.120+0000 7f9f0effd700 1 -- 172.21.15.46:0/3630199603 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f040117c0 con 0x7f9f10159b30 2024-01-26T23:49:25.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.121+0000 7f9f1634f700 1 -- 172.21.15.46:0/3630199603 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f10159b30 msgr2=0x7f9f100b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.121+0000 7f9f1634f700 1 --2- 172.21.15.46:0/3630199603 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f10159b30 0x7f9f100b0bb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9f04009910 tx=0x7f9f040091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.121+0000 7f9f1634f700 1 -- 172.21.15.46:0/3630199603 shutdown_connections 2024-01-26T23:49:25.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.121+0000 7f9f1634f700 1 --2- 172.21.15.46:0/3630199603 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f10159b30 0x7f9f100b0bb0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.121+0000 7f9f1634f700 1 -- 172.21.15.46:0/3630199603 >> 172.21.15.46:0/3630199603 conn(0x7f9f10069510 msgr2=0x7f9f100aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:25.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.121+0000 7f9f1634f700 1 -- 172.21.15.46:0/3630199603 shutdown_connections 2024-01-26T23:49:25.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.121+0000 7f9f1634f700 1 -- 172.21.15.46:0/3630199603 wait complete. 2024-01-26T23:49:25.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.123+0000 7f9f1634f700 1 Processor -- start 2024-01-26T23:49:25.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.123+0000 7f9f1634f700 1 -- start start 2024-01-26T23:49:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.124+0000 7f9f1634f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f100b10f0 0x7f9f10141b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.124+0000 7f9f1634f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f10159760 0x7f9f1015c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.124+0000 7f9f1634f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f10159b30 0x7f9f1016ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.124+0000 7f9f1634f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1015d000 con 0x7f9f100b10f0 2024-01-26T23:49:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.124+0000 7f9f1634f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1015d2e0 con 0x7f9f10159b30 2024-01-26T23:49:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.124+0000 7f9f1634f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1015d5c0 con 0x7f9f10159760 2024-01-26T23:49:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.124+0000 7f9f0ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f100b10f0 0x7f9f10141b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.124+0000 7f9f0ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f100b10f0 0x7f9f10141b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40138/0 (socket says 172.21.15.46:40138) 2024-01-26T23:49:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.124+0000 7f9f0ffff700 1 -- 172.21.15.46:0/741822813 learned_addr learned my addr 172.21.15.46:0/741822813 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.124+0000 7f9f0f7fe700 1 --2- 172.21.15.46:0/741822813 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f10159760 0x7f9f1015c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.124+0000 7f9f148ec700 1 --2- 172.21.15.46:0/741822813 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f10159b30 0x7f9f1016ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.124+0000 7f9f0ffff700 1 -- 172.21.15.46:0/741822813 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f10159760 msgr2=0x7f9f1015c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.124+0000 7f9f0ffff700 1 --2- 172.21.15.46:0/741822813 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f10159760 0x7f9f1015c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.124+0000 7f9f0ffff700 1 -- 172.21.15.46:0/741822813 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f10159b30 msgr2=0x7f9f1016ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.124+0000 7f9f0ffff700 1 --2- 172.21.15.46:0/741822813 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f10159b30 0x7f9f1016ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.124+0000 7f9f0ffff700 1 -- 172.21.15.46:0/741822813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f04009420 con 0x7f9f100b10f0 2024-01-26T23:49:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.125+0000 7f9f0ffff700 1 --2- 172.21.15.46:0/741822813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f100b10f0 0x7f9f10141b00 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000cb70 tx=0x7f9f00001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.125+0000 7f9ef7fff700 1 -- 172.21.15.46:0/741822813 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f00001d00 con 0x7f9f100b10f0 2024-01-26T23:49:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.125+0000 7f9ef7fff700 1 -- 172.21.15.46:0/741822813 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f00007360 con 0x7f9f100b10f0 2024-01-26T23:49:25.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.125+0000 7f9ef7fff700 1 -- 172.21.15.46:0/741822813 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f00004830 con 0x7f9f100b10f0 2024-01-26T23:49:25.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.125+0000 7f9f1634f700 1 -- 172.21.15.46:0/741822813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f1015d8a0 con 0x7f9f100b10f0 2024-01-26T23:49:25.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.125+0000 7f9f1634f700 1 -- 172.21.15.46:0/741822813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f100c2490 con 0x7f9f100b10f0 2024-01-26T23:49:25.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.126+0000 7f9f1634f700 1 -- 172.21.15.46:0/741822813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f1004c710 con 0x7f9f100b10f0 2024-01-26T23:49:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.126+0000 7f9ef7fff700 1 -- 172.21.15.46:0/741822813 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9f00004ba0 con 0x7f9f100b10f0 2024-01-26T23:49:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.127+0000 7f9ef7fff700 1 --2- 172.21.15.46:0/741822813 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9efc066f30 0x7f9efc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.127+0000 7f9ef7fff700 1 -- 172.21.15.46:0/741822813 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f9f00084110 con 0x7f9f100b10f0 2024-01-26T23:49:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.127+0000 7f9f0f7fe700 1 --2- 172.21.15.46:0/741822813 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9efc066f30 0x7f9efc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.128+0000 7f9f0f7fe700 1 --2- 172.21.15.46:0/741822813 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9efc066f30 0x7f9efc0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9f040030c0 tx=0x7f9f04016000 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:25.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.132+0000 7f9ef7fff700 1 -- 172.21.15.46:0/741822813 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9f000510b0 con 0x7f9f100b10f0 2024-01-26T23:49:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.383+0000 7f9f1634f700 1 -- 172.21.15.46:0/741822813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f1006e6e0 con 0x7f9f100b10f0 2024-01-26T23:49:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.384+0000 7f9ef7fff700 1 -- 172.21.15.46:0/741822813 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f9f000546d0 con 0x7f9f100b10f0 2024-01-26T23:49:25.384 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:25.384 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/406559819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":406559819},{"type":"v1","addr":"172.21.15.106:6836","nonce":406559819}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/3387155671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":3387155671},{"type":"v1","addr":"172.21.15.106:6834","nonce":3387155671}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8298,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/855914203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":855914203},{"type":"v1","addr":"172.21.15.46:6834","nonce":855914203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:49:21.360002+0000","modified":"2024-01-26T23:49:24.956671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/1052887453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1052887453},{"type":"v1","addr":"172.21.15.46:6835","nonce":1052887453}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2024-01-26T23:49:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.386+0000 7f9f1634f700 1 -- 172.21.15.46:0/741822813 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9efc066f30 msgr2=0x7f9efc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.386+0000 7f9f1634f700 1 --2- 172.21.15.46:0/741822813 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9efc066f30 0x7f9efc0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9f040030c0 tx=0x7f9f04016000 comp rx=0 tx=0).stop 2024-01-26T23:49:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.386+0000 7f9f1634f700 1 -- 172.21.15.46:0/741822813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f100b10f0 msgr2=0x7f9f10141b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.386+0000 7f9f1634f700 1 --2- 172.21.15.46:0/741822813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f100b10f0 0x7f9f10141b00 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000cb70 tx=0x7f9f00001ab0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.386+0000 7f9f1634f700 1 -- 172.21.15.46:0/741822813 shutdown_connections 2024-01-26T23:49:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.386+0000 7f9f1634f700 1 --2- 172.21.15.46:0/741822813 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9efc066f30 0x7f9efc0693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.386+0000 7f9f1634f700 1 --2- 172.21.15.46:0/741822813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f100b10f0 0x7f9f10141b00 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.386+0000 7f9f1634f700 1 --2- 172.21.15.46:0/741822813 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f10159b30 0x7f9f1016ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.386+0000 7f9f1634f700 1 --2- 172.21.15.46:0/741822813 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f10159760 0x7f9f1015c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.386+0000 7f9f1634f700 1 -- 172.21.15.46:0/741822813 >> 172.21.15.46:0/741822813 conn(0x7f9f10069510 msgr2=0x7f9f100b0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.387+0000 7f9f1634f700 1 -- 172.21.15.46:0/741822813 shutdown_connections 2024-01-26T23:49:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.387+0000 7f9f1634f700 1 -- 172.21.15.46:0/741822813 wait complete. 2024-01-26T23:49:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 70 2024-01-26T23:49:25.400 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:49:25.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.532+0000 7fdcddffb700 1 Processor -- start 2024-01-26T23:49:25.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.532+0000 7fdcddffb700 1 -- start start 2024-01-26T23:49:25.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.533+0000 7fdcddffb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd80ae520 0x7fdcd80b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:25.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.533+0000 7fdcddffb700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdcd80b3fb0 0x7fdcd8157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:25.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.533+0000 7fdcddffb700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd8158ae0 con 0x7fdcd8157340 2024-01-26T23:49:25.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.533+0000 7fdcddffb700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd80b4430 con 0x7fdcd80b3fb0 2024-01-26T23:49:25.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.533+0000 7fdcddffb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd815b650 con 0x7fdcd80ae520 2024-01-26T23:49:25.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.534+0000 7fdcd6ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd80ae520 0x7fdcd80b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:25.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.534+0000 7fdcd7fff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdcd80b3fb0 0x7fdcd8157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:25.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.534+0000 7fdcd6ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd80ae520 0x7fdcd80b3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:58722/0 (socket says 172.21.15.46:58722) 2024-01-26T23:49:25.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.534+0000 7fdcd6ffd700 1 -- 172.21.15.46:0/3019524589 learned_addr learned my addr 172.21.15.46:0/3019524589 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:25.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.534+0000 7fdcd6ffd700 1 -- 172.21.15.46:0/3019524589 >> v2:172.21.15.106:3300/0 conn(0x7fdcd80b3fb0 msgr2=0x7fdcd8157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:25.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.534+0000 7fdcd6ffd700 1 --2- 172.21.15.46:0/3019524589 >> v2:172.21.15.106:3300/0 conn(0x7fdcd80b3fb0 0x7fdcd8157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.534+0000 7fdcd6ffd700 1 -- 172.21.15.46:0/3019524589 >> v1:172.21.15.106:6789/0 conn(0x7fdcd8157340 legacy=0x7fdcd815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.534+0000 7fdcd6ffd700 1 -- 172.21.15.46:0/3019524589 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd8158520 con 0x7fdcd80ae520 2024-01-26T23:49:25.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.534+0000 7fdcd7fff700 1 --2- 172.21.15.46:0/3019524589 >> v2:172.21.15.106:3300/0 conn(0x7fdcd80b3fb0 0x7fdcd8157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:25.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.534+0000 7fdcd6ffd700 1 --2- 172.21.15.46:0/3019524589 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd80ae520 0x7fdcd80b3a70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0099b0 tx=0x7fdccc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79f21a5d237b3cdc server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:25.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.535+0000 7fdcd67fc700 1 -- 172.21.15.46:0/3019524589 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc009ea0 con 0x7fdcd80ae520 2024-01-26T23:49:25.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.535+0000 7fdcd67fc700 1 -- 172.21.15.46:0/3019524589 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdccc00b860 con 0x7fdcd80ae520 2024-01-26T23:49:25.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.535+0000 7fdcd67fc700 1 -- 172.21.15.46:0/3019524589 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc00bc20 con 0x7fdcd80ae520 2024-01-26T23:49:25.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.535+0000 7fdcddffb700 1 -- 172.21.15.46:0/3019524589 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd80ae520 msgr2=0x7fdcd80b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.535+0000 7fdcddffb700 1 --2- 172.21.15.46:0/3019524589 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd80ae520 0x7fdcd80b3a70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0099b0 tx=0x7fdccc009250 comp rx=0 tx=0).stop 2024-01-26T23:49:25.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.535+0000 7fdcddffb700 1 -- 172.21.15.46:0/3019524589 shutdown_connections 2024-01-26T23:49:25.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.535+0000 7fdcddffb700 1 --2- 172.21.15.46:0/3019524589 >> v2:172.21.15.106:3300/0 conn(0x7fdcd80b3fb0 0x7fdcd8157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.535+0000 7fdcddffb700 1 --2- 172.21.15.46:0/3019524589 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd80ae520 0x7fdcd80b3a70 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.535+0000 7fdcddffb700 1 -- 172.21.15.46:0/3019524589 >> 172.21.15.46:0/3019524589 conn(0x7fdcd80ce530 msgr2=0x7fdcd80ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:25.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.535+0000 7fdcddffb700 1 -- 172.21.15.46:0/3019524589 shutdown_connections 2024-01-26T23:49:25.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.535+0000 7fdcddffb700 1 -- 172.21.15.46:0/3019524589 wait complete. 2024-01-26T23:49:25.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.537+0000 7fdcddffb700 1 Processor -- start 2024-01-26T23:49:25.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.537+0000 7fdcddffb700 1 -- start start 2024-01-26T23:49:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.538+0000 7fdcddffb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd80ae520 0x7fdcd80afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.538+0000 7fdcddffb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd80b3fb0 0x7fdcd80b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.538+0000 7fdcddffb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcd8157340 0x7fdcd80cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.538+0000 7fdcddffb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd8167a30 con 0x7fdcd80b3fb0 2024-01-26T23:49:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.538+0000 7fdcddffb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd8168290 con 0x7fdcd8157340 2024-01-26T23:49:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.538+0000 7fdcddffb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd80c5d50 con 0x7fdcd80ae520 2024-01-26T23:49:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.538+0000 7fdcd6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd80b3fb0 0x7fdcd80b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.538+0000 7fdcd77fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd80ae520 0x7fdcd80afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.538+0000 7fdcd6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd80b3fb0 0x7fdcd80b0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40146/0 (socket says 172.21.15.46:40146) 2024-01-26T23:49:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.538+0000 7fdcd6ffd700 1 -- 172.21.15.46:0/3410498523 learned_addr learned my addr 172.21.15.46:0/3410498523 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.539+0000 7fdcd7fff700 1 --2- 172.21.15.46:0/3410498523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcd8157340 0x7fdcd80cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.539+0000 7fdcd6ffd700 1 -- 172.21.15.46:0/3410498523 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd80ae520 msgr2=0x7fdcd80afd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.539+0000 7fdcd6ffd700 1 --2- 172.21.15.46:0/3410498523 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd80ae520 0x7fdcd80afd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.539+0000 7fdcd6ffd700 1 -- 172.21.15.46:0/3410498523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcd8157340 msgr2=0x7fdcd80cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.539+0000 7fdcd6ffd700 1 --2- 172.21.15.46:0/3410498523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcd8157340 0x7fdcd80cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.539+0000 7fdcd6ffd700 1 -- 172.21.15.46:0/3410498523 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd0009a20 con 0x7fdcd80b3fb0 2024-01-26T23:49:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.539+0000 7fdcd77fe700 1 --2- 172.21.15.46:0/3410498523 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd80ae520 0x7fdcd80afd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.539+0000 7fdcd6ffd700 1 --2- 172.21.15.46:0/3410498523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd80b3fb0 0x7fdcd80b0270 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00b3c0 tx=0x7fdccc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.539+0000 7fdcbf7fe700 1 -- 172.21.15.46:0/3410498523 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc01f070 con 0x7fdcd80b3fb0 2024-01-26T23:49:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.539+0000 7fdcbf7fe700 1 -- 172.21.15.46:0/3410498523 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdccc00bbe0 con 0x7fdcd80b3fb0 2024-01-26T23:49:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.539+0000 7fdcddffb700 1 -- 172.21.15.46:0/3410498523 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdccc0094f0 con 0x7fdcd80b3fb0 2024-01-26T23:49:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.539+0000 7fdcbf7fe700 1 -- 172.21.15.46:0/3410498523 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc011650 con 0x7fdcd80b3fb0 2024-01-26T23:49:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.539+0000 7fdcddffb700 1 -- 172.21.15.46:0/3410498523 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd8167c40 con 0x7fdcd80b3fb0 2024-01-26T23:49:25.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.540+0000 7fdcddffb700 1 -- 172.21.15.46:0/3410498523 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd8168440 con 0x7fdcd80b3fb0 2024-01-26T23:49:25.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.541+0000 7fdcbf7fe700 1 -- 172.21.15.46:0/3410498523 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fdccc011a40 con 0x7fdcd80b3fb0 2024-01-26T23:49:25.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.541+0000 7fdcbf7fe700 1 --2- 172.21.15.46:0/3410498523 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fdcc4066f30 0x7fdcc40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:25.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.541+0000 7fdcbf7fe700 1 -- 172.21.15.46:0/3410498523 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fdccc085aa0 con 0x7fdcd80b3fb0 2024-01-26T23:49:25.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.541+0000 7fdcd77fe700 1 --2- 172.21.15.46:0/3410498523 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fdcc4066f30 0x7fdcc40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:25.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.542+0000 7fdcd77fe700 1 --2- 172.21.15.46:0/3410498523 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fdcc4066f30 0x7fdcc40693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8001dc0 tx=0x7fdcc8009040 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:25.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.546+0000 7fdcbf7fe700 1 -- 172.21.15.46:0/3410498523 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdcd8168440 con 0x7fdcd80b3fb0 2024-01-26T23:49:25.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.752+0000 7fdcddffb700 1 -- 172.21.15.46:0/3410498523 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcd8158890 con 0x7fdcd80b3fb0 2024-01-26T23:49:25.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.753+0000 7fdcbf7fe700 1 -- 172.21.15.46:0/3410498523 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+16163 (secure 0 0 0) 0x7fdccc028090 con 0x7fdcd80b3fb0 2024-01-26T23:49:25.753 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:25.754 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":145,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:49:22.688261+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:49:19.677244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-01-26T23:49:20.330110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:49:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.755+0000 7fdcddffb700 1 -- 172.21.15.46:0/3410498523 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fdcc4066f30 msgr2=0x7fdcc40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.755+0000 7fdcddffb700 1 --2- 172.21.15.46:0/3410498523 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fdcc4066f30 0x7fdcc40693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8001dc0 tx=0x7fdcc8009040 comp rx=0 tx=0).stop 2024-01-26T23:49:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.755+0000 7fdcddffb700 1 -- 172.21.15.46:0/3410498523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd80b3fb0 msgr2=0x7fdcd80b0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.755+0000 7fdcddffb700 1 --2- 172.21.15.46:0/3410498523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd80b3fb0 0x7fdcd80b0270 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00b3c0 tx=0x7fdccc017000 comp rx=0 tx=0).stop 2024-01-26T23:49:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.755+0000 7fdcddffb700 1 -- 172.21.15.46:0/3410498523 shutdown_connections 2024-01-26T23:49:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.755+0000 7fdcddffb700 1 --2- 172.21.15.46:0/3410498523 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fdcc4066f30 0x7fdcc40693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.755+0000 7fdcddffb700 1 --2- 172.21.15.46:0/3410498523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcd8157340 0x7fdcd80cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.755+0000 7fdcddffb700 1 --2- 172.21.15.46:0/3410498523 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd80ae520 0x7fdcd80afd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.755+0000 7fdcddffb700 1 --2- 172.21.15.46:0/3410498523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd80b3fb0 0x7fdcd80b0270 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.755+0000 7fdcddffb700 1 -- 172.21.15.46:0/3410498523 >> 172.21.15.46:0/3410498523 conn(0x7fdcd80ce530 msgr2=0x7fdcd81542f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:25.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.756+0000 7fdcddffb700 1 -- 172.21.15.46:0/3410498523 shutdown_connections 2024-01-26T23:49:25.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.756+0000 7fdcddffb700 1 -- 172.21.15.46:0/3410498523 wait complete. 2024-01-26T23:49:25.757 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:49:25.759 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:49:25.770 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-26T23:49:25.770 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:49:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.946+0000 7f84ce3fe700 1 Processor -- start 2024-01-26T23:49:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.946+0000 7f84ce3fe700 1 -- start start 2024-01-26T23:49:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.947+0000 7f84ce3fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f84c81575a0 0x7f84c8150af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.947+0000 7f84ce3fe700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c8068d20 con 0x7f84c806e2a0 2024-01-26T23:49:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.947+0000 7f84ce3fe700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c8068110 con 0x7f84c8150720 2024-01-26T23:49:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.947+0000 7f84ce3fe700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c80683f0 con 0x7f84c81575a0 2024-01-26T23:49:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.947+0000 7f84cc99b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f84c806e2a0 0x7f84c8157b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43434/0 (socket says 172.21.15.46:43434) 2024-01-26T23:49:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.947+0000 7f84cc99b700 1 -- 172.21.15.46:0/2903215157 learned_addr learned my addr 172.21.15.46:0/2903215157 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.947+0000 7f84bffff700 1 --2- 172.21.15.46:0/2903215157 >> v2:172.21.15.106:3300/0 conn(0x7f84c81575a0 0x7f84c8150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.947+0000 7f84c77fe700 1 -- 172.21.15.46:0/2903215157 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157798547 0 0) 0x7f84c8068d20 con 0x7f84c806e2a0 2024-01-26T23:49:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.947+0000 7f84c77fe700 1 -- 172.21.15.46:0/2903215157 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84b4003570 con 0x7f84c806e2a0 2024-01-26T23:49:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.947+0000 7f84c77fe700 1 -- 172.21.15.46:0/2903215157 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075841669 0 0) 0x7f84c8068110 con 0x7f84c8150720 2024-01-26T23:49:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.947+0000 7f84c77fe700 1 -- 172.21.15.46:0/2903215157 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84c8068d20 con 0x7f84c8150720 2024-01-26T23:49:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.947+0000 7f84c77fe700 1 -- 172.21.15.46:0/2903215157 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2101468539 0 0) 0x7f84b4003570 con 0x7f84c806e2a0 2024-01-26T23:49:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.948+0000 7f84bffff700 1 -- 172.21.15.46:0/2903215157 >> v1:172.21.15.106:6789/0 conn(0x7f84c8150720 legacy=0x7f84c8067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.948+0000 7f84bffff700 1 -- 172.21.15.46:0/2903215157 >> v1:172.21.15.46:6789/0 conn(0x7f84c806e2a0 legacy=0x7f84c8157b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.948+0000 7f84bffff700 1 -- 172.21.15.46:0/2903215157 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c80686d0 con 0x7f84c81575a0 2024-01-26T23:49:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.948+0000 7f84bffff700 1 --2- 172.21.15.46:0/2903215157 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c81575a0 0x7f84c8150af0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f84b00098e0 tx=0x7f84b00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccf8dc2a189c84e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.948+0000 7f84c77fe700 1 -- 172.21.15.46:0/2903215157 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c00017f0 con 0x7f84c81575a0 2024-01-26T23:49:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.948+0000 7f84c77fe700 1 -- 172.21.15.46:0/2903215157 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b0011420 con 0x7f84c81575a0 2024-01-26T23:49:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.948+0000 7f84c77fe700 1 -- 172.21.15.46:0/2903215157 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b00117c0 con 0x7f84c81575a0 2024-01-26T23:49:25.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.949+0000 7f84ce3fe700 1 -- 172.21.15.46:0/2903215157 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c81575a0 msgr2=0x7f84c8150af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.949+0000 7f84ce3fe700 1 --2- 172.21.15.46:0/2903215157 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c81575a0 0x7f84c8150af0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f84b00098e0 tx=0x7f84b00091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.949+0000 7f84ce3fe700 1 -- 172.21.15.46:0/2903215157 shutdown_connections 2024-01-26T23:49:25.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.949+0000 7f84ce3fe700 1 --2- 172.21.15.46:0/2903215157 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c81575a0 0x7f84c8150af0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.949+0000 7f84ce3fe700 1 -- 172.21.15.46:0/2903215157 >> 172.21.15.46:0/2903215157 conn(0x7f84c80ccbc0 msgr2=0x7f84c80ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.949+0000 7f84ce3fe700 1 -- 172.21.15.46:0/2903215157 shutdown_connections 2024-01-26T23:49:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.949+0000 7f84ce3fe700 1 -- 172.21.15.46:0/2903215157 wait complete. 2024-01-26T23:49:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.950+0000 7f84ce3fe700 1 Processor -- start 2024-01-26T23:49:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.950+0000 7f84ce3fe700 1 -- start start 2024-01-26T23:49:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.951+0000 7f84ce3fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c806e2a0 0x7f84c80ad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.951+0000 7f84ce3fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84c8150720 0x7f84c8154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.951+0000 7f84ce3fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84c81575a0 0x7f84c814d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.951+0000 7f84ce3fe700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c80add00 con 0x7f84c8150720 2024-01-26T23:49:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.951+0000 7f84ce3fe700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c8156310 con 0x7f84c806e2a0 2024-01-26T23:49:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.951+0000 7f84ce3fe700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c80c7430 con 0x7f84c81575a0 2024-01-26T23:49:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.951+0000 7f84bffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84c8150720 0x7f84c8154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.951+0000 7f84bffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84c8150720 0x7f84c8154fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40148/0 (socket says 172.21.15.46:40148) 2024-01-26T23:49:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.951+0000 7f84bffff700 1 -- 172.21.15.46:0/3854369003 learned_addr learned my addr 172.21.15.46:0/3854369003 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.951+0000 7f84bffff700 1 -- 172.21.15.46:0/3854369003 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84c81575a0 msgr2=0x7f84c814d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.951+0000 7f84cc99b700 1 --2- 172.21.15.46:0/3854369003 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84c81575a0 0x7f84c814d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:25.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.951+0000 7f84c7fff700 1 --2- 172.21.15.46:0/3854369003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c806e2a0 0x7f84c80ad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:25.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.951+0000 7f84bffff700 1 --2- 172.21.15.46:0/3854369003 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84c81575a0 0x7f84c814d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.951+0000 7f84bffff700 1 -- 172.21.15.46:0/3854369003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c806e2a0 msgr2=0x7f84c80ad7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:25.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.951+0000 7f84bffff700 1 --2- 172.21.15.46:0/3854369003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c806e2a0 0x7f84c80ad7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:25.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.951+0000 7f84bffff700 1 -- 172.21.15.46:0/3854369003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b0009420 con 0x7f84c8150720 2024-01-26T23:49:25.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.951+0000 7f84bffff700 1 --2- 172.21.15.46:0/3854369003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84c8150720 0x7f84c8154fa0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009ed0 tx=0x7f84b0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:25.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.952+0000 7f84c4ff9700 1 -- 172.21.15.46:0/3854369003 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b001cc80 con 0x7f84c8150720 2024-01-26T23:49:25.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.952+0000 7f84c4ff9700 1 -- 172.21.15.46:0/3854369003 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b001cde0 con 0x7f84c8150720 2024-01-26T23:49:25.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.952+0000 7f84ce3fe700 1 -- 172.21.15.46:0/3854369003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c81554e0 con 0x7f84c8150720 2024-01-26T23:49:25.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.952+0000 7f84c4ff9700 1 -- 172.21.15.46:0/3854369003 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b0025680 con 0x7f84c8150720 2024-01-26T23:49:25.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.952+0000 7f84ce3fe700 1 -- 172.21.15.46:0/3854369003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c816d940 con 0x7f84c8150720 2024-01-26T23:49:25.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.953+0000 7f84ce3fe700 1 -- 172.21.15.46:0/3854369003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c80c75e0 con 0x7f84c8150720 2024-01-26T23:49:25.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.953+0000 7f84c4ff9700 1 -- 172.21.15.46:0/3854369003 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f84b00193f0 con 0x7f84c8150720 2024-01-26T23:49:25.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.953+0000 7f84c4ff9700 1 --2- 172.21.15.46:0/3854369003 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f84a8066f30 0x7f84a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:25.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.954+0000 7f84c4ff9700 1 -- 172.21.15.46:0/3854369003 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f84b0085a00 con 0x7f84c8150720 2024-01-26T23:49:25.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.954+0000 7f84c7fff700 1 --2- 172.21.15.46:0/3854369003 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f84a8066f30 0x7f84a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:25.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.954+0000 7f84c7fff700 1 --2- 172.21.15.46:0/3854369003 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f84a8066f30 0x7f84a80693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f84b80027d0 tx=0x7f84b8009040 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:25.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:25.958+0000 7f84c4ff9700 1 -- 172.21.15.46:0/3854369003 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f84c80c75e0 con 0x7f84c8150720 2024-01-26T23:49:26.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.161+0000 7f84ce3fe700 1 -- 172.21.15.46:0/3854369003 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f84c80c75e0 con 0x7f84a8066f30 2024-01-26T23:49:26.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.170+0000 7f84c4ff9700 1 -- 172.21.15.46:0/3854369003 <== mgr.7148 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+274554 (secure 0 0 0) 0x7f84c80c75e0 con 0x7f84a8066f30 2024-01-26T23:49:26.174 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.173+0000 7f84ce3fe700 1 -- 172.21.15.46:0/3854369003 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f84a8066f30 msgr2=0x7f84a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.173+0000 7f84ce3fe700 1 --2- 172.21.15.46:0/3854369003 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f84a8066f30 0x7f84a80693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f84b80027d0 tx=0x7f84b8009040 comp rx=0 tx=0).stop 2024-01-26T23:49:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.173+0000 7f84ce3fe700 1 -- 172.21.15.46:0/3854369003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84c8150720 msgr2=0x7f84c8154fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.173+0000 7f84ce3fe700 1 --2- 172.21.15.46:0/3854369003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84c8150720 0x7f84c8154fa0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009ed0 tx=0x7f84b0016000 comp rx=0 tx=0).stop 2024-01-26T23:49:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.173+0000 7f84ce3fe700 1 -- 172.21.15.46:0/3854369003 shutdown_connections 2024-01-26T23:49:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.173+0000 7f84ce3fe700 1 --2- 172.21.15.46:0/3854369003 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f84a8066f30 0x7f84a80693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.173+0000 7f84ce3fe700 1 --2- 172.21.15.46:0/3854369003 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84c81575a0 0x7f84c814d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.173+0000 7f84ce3fe700 1 --2- 172.21.15.46:0/3854369003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84c806e2a0 0x7f84c80ad7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.173+0000 7f84ce3fe700 1 --2- 172.21.15.46:0/3854369003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84c8150720 0x7f84c8154fa0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.173+0000 7f84ce3fe700 1 -- 172.21.15.46:0/3854369003 >> 172.21.15.46:0/3854369003 conn(0x7f84c80ccbc0 msgr2=0x7f84c814f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.173+0000 7f84ce3fe700 1 -- 172.21.15.46:0/3854369003 shutdown_connections 2024-01-26T23:49:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.173+0000 7f84ce3fe700 1 -- 172.21.15.46:0/3854369003 wait complete. 2024-01-26T23:49:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:49:26.187 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-26T23:49:24.870286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":2314,"num_write":360,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8584,"ondisk_log_size":8584,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":2450492,"kb_used_data":14844,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752524228,"statfs":{"total":773094113280,"available":770584809472,"internally_reserved":0,"allocated":15200256,"data_stored":8390475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.959692"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"135'8584","reported_seq":9102,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-01-26T23:49:17.493871+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:49:17.493871+0000","last_peered":"2024-01-26T23:49:17.493871+0000","last_clean":"2024-01-26T23:49:17.493871+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:49:17.493871+0000","last_undegraded":"2024-01-26T23:49:17.493871+0000","last_fullsized":"2024-01-26T23:49:17.493871+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":2314,"num_write":360,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":2314,"num_write":360,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8584,"ondisk_log_size":8584,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575082,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311868,"kb_used_data":2292,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059972,"statfs":{"total":96636764160,"available":96317411328,"internally_reserved":0,"allocated":2347008,"data_stored":1492375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51500000000000001,"15min":0.56200000000000006},"min":{"1min":0.40699999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.74199999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52700000000000002,"15min":0.57599999999999996},"min":{"1min":0.35899999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67200000000000004,"5min":0.82499999999999996,"15min":0.83999999999999997},"last":0.629}]},{"osd":1,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54000000000000004,"15min":0.57399999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.77600000000000002,"5min":0.77800000000000002,"15min":0.82899999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55100000000000005,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.70399999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.56799999999999995}]},{"osd":2,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.57199999999999995},"min":{"1min":0.38700000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69199999999999995,"5min":0.80000000000000004,"15min":0.81699999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.38900000000000001,"5min":0.30199999999999999,"15min":0.27400000000000002},"max":{"1min":0.746,"5min":0.81999999999999995,"15min":0.88600000000000001},"last":0.60899999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54700000000000004,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.377,"15min":0.377},"max":{"1min":0.70399999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57499999999999996,"15min":0.59699999999999998},"min":{"1min":0.34399999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77900000000000003,"5min":0.86199999999999999,"15min":0.86399999999999999},"last":0.81399999999999995}]},{"osd":4,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.438,"15min":0.45200000000000001},"min":{"1min":0.34599999999999997,"5min":0.252,"15min":0.247},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.93899999999999995},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.45300000000000001,"15min":0.47399999999999998},"min":{"1min":0.32400000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67300000000000004,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.498}]},{"osd":5,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47599999999999998,"15min":0.48699999999999999},"min":{"1min":0.30499999999999999,"5min":0.27200000000000002,"15min":0.22800000000000001},"max":{"1min":0.749,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.751},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.504,"15min":0.52300000000000002},"min":{"1min":0.35799999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73399999999999999,"5min":0.74199999999999999,"15min":0.90200000000000002},"last":0.78900000000000003}]},{"osd":6,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53100000000000003,"15min":0.54800000000000004},"min":{"1min":0.41699999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76400000000000001,"5min":0.78700000000000003,"15min":0.95899999999999996},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.52700000000000002,"15min":0.54000000000000004},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75900000000000001,"5min":0.79500000000000004,"15min":0.85199999999999998},"last":0.71399999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303096,"kb_used_data":1712,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068744,"statfs":{"total":96636764160,"available":96326393856,"internally_reserved":0,"allocated":1753088,"data_stored":902007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52900000000000003,"15min":0.56499999999999995},"min":{"1min":0.40300000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.423,"15min":1.423},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.57899999999999996},"min":{"1min":0.36099999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.91100000000000003},"last":0.748}]},{"osd":1,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51700000000000002,"15min":0.55800000000000005},"min":{"1min":0.32200000000000001,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.629},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.58399999999999996},"min":{"1min":0.38200000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.66800000000000004,"5min":1.498,"15min":1.498},"last":0.76500000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52800000000000002,"15min":0.56999999999999995},"min":{"1min":0.38100000000000001,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.85799999999999998},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52600000000000002,"15min":0.57099999999999995},"min":{"1min":0.377,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69299999999999995,"5min":0.77000000000000002,"15min":0.82799999999999996},"last":0.53600000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57199999999999995,"15min":0.59099999999999997},"min":{"1min":0.34699999999999998,"5min":0.33800000000000002,"15min":0.29999999999999999},"max":{"1min":0.746,"5min":1.595,"15min":1.595},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56899999999999995,"15min":0.60599999999999998},"min":{"1min":0.45600000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.751,"5min":1.369,"15min":1.369},"last":0.57499999999999996}]},{"osd":4,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.443,"15min":0.46200000000000002},"min":{"1min":0.33800000000000002,"5min":0.245,"15min":0.22},"max":{"1min":0.67000000000000004,"5min":1.556,"15min":1.556},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.47399999999999998,"15min":0.48199999999999998},"min":{"1min":0.33200000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.86199999999999999},"last":0.46000000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45500000000000002,"15min":0.47099999999999997},"min":{"1min":0.32700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67900000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.51700000000000002,"15min":0.52000000000000002},"min":{"1min":0.30499999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.70999999999999996,"5min":1.47,"15min":1.47},"last":0.73499999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.5,"15min":0.51800000000000002},"min":{"1min":0.40100000000000002,"5min":0.26700000000000002,"15min":0.246},"max":{"1min":0.80200000000000005,"5min":0.81799999999999995,"15min":0.85899999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50900000000000001,"15min":0.52400000000000002},"min":{"1min":0.32300000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.80400000000000005},"last":0.71999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307192,"kb_used_data":1712,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064648,"statfs":{"total":96636764160,"available":96322199552,"internally_reserved":0,"allocated":1753088,"data_stored":902007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42099999999999999,"15min":0.40600000000000003},"min":{"1min":0.28999999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.64400000000000002,"5min":0.65200000000000002,"15min":0.72599999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.43099999999999999,"15min":0.44},"min":{"1min":0.30199999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.52600000000000002,"5min":1.0269999999999999,"15min":1.337},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40400000000000003,"15min":0.40799999999999997},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.23599999999999999},"max":{"1min":0.67700000000000005,"5min":0.68799999999999994,"15min":0.73299999999999998},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.47299999999999998,"15min":0.47399999999999998},"min":{"1min":0.35399999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.70199999999999996,"5min":0.74099999999999999,"15min":0.77300000000000002},"last":0.56100000000000005}]},{"osd":3,"last update":"Fri Jan 26 23:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51800000000000002,"15min":0.51400000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32100000000000001},"max":{"1min":0.78900000000000003,"5min":0.82199999999999995,"15min":1.782},"last":0.502},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.46899999999999997,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.312,"15min":0.26600000000000001},"max":{"1min":0.59999999999999998,"5min":0.81399999999999995,"15min":1.379},"last":0.38300000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.623,"15min":0.64100000000000001},"min":{"1min":0.42999999999999999,"5min":0.40799999999999997,"15min":0.39400000000000002},"max":{"1min":0.89200000000000002,"5min":0.94099999999999995,"15min":1.0820000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.65000000000000002,"15min":0.65600000000000003},"min":{"1min":0.45200000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.95699999999999996},"last":0.61499999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.65600000000000003,"15min":0.66800000000000004},"min":{"1min":0.54000000000000004,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":0.91100000000000003,"5min":0.96699999999999997,"15min":1.208},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.64600000000000002,"15min":0.65700000000000003},"min":{"1min":0.502,"5min":0.45100000000000001,"15min":0.42499999999999999},"max":{"1min":0.96399999999999997,"5min":0.96599999999999997,"15min":1.018},"last":0.57999999999999996}]},{"osd":6,"last update":"Fri Jan 26 23:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.67100000000000004},"min":{"1min":0.47899999999999998,"5min":0.47799999999999998,"15min":0.42599999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.2529999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.66600000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93200000000000005,"5min":1.01,"15min":1.022},"last":0.52800000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.68899999999999995,"15min":0.70599999999999996},"min":{"1min":0.49099999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.0069999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70299999999999996,"15min":0.71899999999999997},"min":{"1min":0.54600000000000004,"5min":0.496,"15min":0.496},"max":{"1min":0.98499999999999999,"5min":1.0049999999999999,"15min":1.296},"last":0.63600000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575081,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307760,"kb_used_data":2280,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064080,"statfs":{"total":96636764160,"available":96321617920,"internally_reserved":0,"allocated":2334720,"data_stored":1486058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.40600000000000003},"min":{"1min":0.25800000000000001,"5min":0.23899999999999999,"15min":0.23100000000000001},"max":{"1min":0.88500000000000001,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.42999999999999999,"15min":0.45100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65800000000000003,"5min":1.363,"15min":1.4490000000000001},"last":0.308}]},{"osd":2,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.41099999999999998,"15min":0.42299999999999999},"min":{"1min":0.28899999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.89600000000000002},"last":0.371},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.33900000000000002,"5min":0.32700000000000001,"15min":0.28000000000000003},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.41199999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.49199999999999999,"15min":0.47399999999999998},"min":{"1min":0.32800000000000001,"5min":0.32300000000000001,"15min":0.30599999999999999},"max":{"1min":0.85199999999999998,"5min":2.145,"15min":2.145},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36099999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":1.198},"last":0.501}]},{"osd":4,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60899999999999999,"15min":0.63800000000000001},"min":{"1min":0.41699999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78600000000000003,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.59699999999999998,"15min":0.63700000000000001},"min":{"1min":0.44400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.94299999999999995},"last":0.68200000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":1.329,"15min":1.329},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.66300000000000003,"15min":0.67000000000000004},"min":{"1min":0.46899999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82799999999999996,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.623}]},{"osd":6,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.61799999999999999,"15min":0.65100000000000002},"min":{"1min":0.47299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.89800000000000002,"5min":0.98699999999999999,"15min":1.032},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.65400000000000003,"15min":0.68000000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.41799999999999998},"max":{"1min":0.93100000000000005,"5min":1.286,"15min":1.286},"last":0.66000000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.70599999999999996,"15min":0.81699999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.435},"max":{"1min":0.81799999999999995,"5min":4.8710000000000004,"15min":25.965},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.73899999999999999,"15min":0.86099999999999999},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.433},"max":{"1min":0.88200000000000001,"5min":4.9199999999999999,"15min":33.085000000000001},"last":0.67200000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575081,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307192,"kb_used_data":1712,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064648,"statfs":{"total":96636764160,"available":96322199552,"internally_reserved":0,"allocated":1753088,"data_stored":902007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.38800000000000001,"15min":0.39000000000000001},"min":{"1min":0.29399999999999998,"5min":0.26900000000000002,"15min":0.255},"max":{"1min":0.53800000000000003,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.43099999999999999,"15min":0.42699999999999999},"min":{"1min":0.26800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60499999999999998,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.435,"15min":0.42799999999999999},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.248},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52000000000000002,"15min":0.51000000000000001},"min":{"1min":0.379,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.044,"5min":1.171,"15min":1.171},"last":0.63900000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46800000000000003,"15min":0.48399999999999999},"min":{"1min":0.34000000000000002,"5min":0.32300000000000001,"15min":0.313},"max":{"1min":0.68700000000000006,"5min":0.72799999999999998,"15min":0.74099999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78200000000000003,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.621}]},{"osd":4,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.60799999999999998,"15min":0.63},"min":{"1min":0.53000000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.82299999999999995,"5min":1.087,"15min":1.087},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.46300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.95499999999999996,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.73799999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.64500000000000002,"15min":0.65600000000000003},"min":{"1min":0.495,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0740000000000001,"5min":1.24,"15min":1.24},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.66400000000000003,"15min":0.67600000000000005},"min":{"1min":0.53400000000000003,"5min":0.41799999999999998,"15min":0.35199999999999998},"max":{"1min":0.93000000000000005,"5min":0.97699999999999998,"15min":1.085},"last":0.72999999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.67800000000000005},"min":{"1min":0.495,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.98599999999999999,"5min":1.022,"15min":1.1080000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66600000000000004,"15min":0.68100000000000005},"min":{"1min":0.51800000000000002,"5min":0.42799999999999999,"15min":0.38800000000000001},"max":{"1min":0.97499999999999998,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.83199999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.68999999999999995,"15min":0.70399999999999996},"min":{"1min":0.52800000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94999999999999996,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.70999999999999996,"15min":0.72699999999999998},"min":{"1min":0.53400000000000003,"5min":0.48099999999999998,"15min":0.42099999999999999},"max":{"1min":1.0189999999999999,"5min":1.052,"15min":1.139},"last":0.75}]}]},{"osd":3,"up_from":13,"seq":55834575080,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307192,"kb_used_data":1712,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064648,"statfs":{"total":96636764160,"available":96322199552,"internally_reserved":0,"allocated":1753088,"data_stored":902007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38400000000000001,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.55400000000000005,"5min":0.59699999999999998,"15min":1.8540000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.42899999999999999,"15min":0.438},"min":{"1min":0.29799999999999999,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":1.2290000000000001},"last":0.38700000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44500000000000001,"15min":0.45200000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":1.633},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.47999999999999998,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":1.0760000000000001},"last":0.48499999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48399999999999999,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65000000000000002,"5min":0.70399999999999996,"15min":1.4059999999999999},"last":0.5},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.44500000000000001,"15min":0.45800000000000002},"min":{"1min":0.34100000000000003,"5min":0.29499999999999998,"15min":0.28299999999999997},"max":{"1min":0.629,"5min":0.69399999999999995,"15min":1.1870000000000001},"last":0.44800000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.63,"15min":0.65800000000000003},"min":{"1min":0.48299999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":3.306},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.44800000000000001,"5min":0.40100000000000002,"15min":0.379},"max":{"1min":0.81499999999999995,"5min":0.85199999999999998,"15min":1.008},"last":0.60399999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.63600000000000001,"15min":0.65200000000000002},"min":{"1min":0.48599999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.85199999999999998,"5min":0.90400000000000003,"15min":1.593},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.47599999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.86299999999999999,"5min":0.92300000000000004,"15min":1.3819999999999999},"last":0.51800000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.66900000000000004},"min":{"1min":0.42299999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":1.506},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67200000000000004,"15min":0.68999999999999995},"min":{"1min":0.55600000000000005,"5min":0.46700000000000003,"15min":0.44400000000000001},"max":{"1min":0.88500000000000001,"5min":0.89000000000000001,"15min":1.427},"last":0.57899999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64400000000000002,"15min":0.68799999999999994},"min":{"1min":0.46700000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":3.238},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.71199999999999997,"15min":0.73399999999999999},"min":{"1min":0.54200000000000004,"5min":0.51900000000000002,"15min":0.48899999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":3.351},"last":0.67700000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303096,"kb_used_data":1712,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068744,"statfs":{"total":96636764160,"available":96326393856,"internally_reserved":0,"allocated":1753088,"data_stored":902007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53400000000000003,"15min":0.56799999999999995},"min":{"1min":0.34799999999999998,"5min":0.28999999999999998,"15min":0.28699999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":1.21},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58399999999999996,"15min":0.58299999999999996},"min":{"1min":0.377,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.81499999999999995,"5min":5.367,"15min":5.367},"last":0.432}]},{"osd":1,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56899999999999995,"15min":0.58699999999999997},"min":{"1min":0.378,"5min":0.311,"15min":0.311},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.54700000000000004,"15min":0.57399999999999995},"min":{"1min":0.40699999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79800000000000004,"5min":0.80600000000000005,"15min":0.81699999999999995},"last":0.61599999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.54600000000000004,"15min":0.56599999999999995},"min":{"1min":0.39300000000000002,"5min":0.33900000000000002,"15min":0.32700000000000001},"max":{"1min":0.747,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.57699999999999996,"15min":0.59599999999999997},"min":{"1min":0.43099999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":1.7050000000000001},"last":0.48599999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57299999999999995,"15min":0.58599999999999997},"min":{"1min":0.44900000000000001,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.35699999999999998},"max":{"1min":0.82499999999999996,"5min":0.94199999999999995,"15min":1.05},"last":0.65100000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.46500000000000002},"min":{"1min":0.31,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.48099999999999998,"15min":0.48399999999999999},"min":{"1min":0.40100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.56200000000000006}]},{"osd":6,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.46500000000000002,"15min":0.47399999999999998},"min":{"1min":0.34999999999999998,"5min":0.28599999999999998,"15min":0.27200000000000002},"max":{"1min":0.76600000000000001,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.501},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.42299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.57999999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.52600000000000002,"15min":0.54000000000000004},"min":{"1min":0.35599999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.56200000000000006,"15min":0.56899999999999995},"min":{"1min":0.436,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.88800000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.74199999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303096,"kb_used_data":1712,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068744,"statfs":{"total":96636764160,"available":96326393856,"internally_reserved":0,"allocated":1753088,"data_stored":902007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.56399999999999995},"min":{"1min":0.35199999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.71099999999999997,"5min":0.81799999999999995,"15min":2.0289999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52700000000000002,"15min":0.56699999999999995},"min":{"1min":0.41999999999999998,"5min":0.29999999999999999,"15min":0.28899999999999998},"max":{"1min":0.70799999999999996,"5min":0.77400000000000002,"15min":0.82199999999999995},"last":0.53300000000000003}]},{"osd":1,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56299999999999994,"15min":0.60399999999999998},"min":{"1min":0.443,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":0.754,"5min":0.79500000000000004,"15min":2.3319999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59899999999999998,"15min":0.626},"min":{"1min":0.45800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76400000000000001,"5min":0.80500000000000005,"15min":1.1739999999999999},"last":0.62}]},{"osd":2,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53700000000000003,"15min":0.55700000000000005},"min":{"1min":0.36799999999999999,"5min":0.34499999999999997,"15min":0.30099999999999999},"max":{"1min":0.68400000000000005,"5min":0.79400000000000004,"15min":0.83799999999999997},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55300000000000005,"15min":0.58699999999999997},"min":{"1min":0.41299999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72899999999999998,"5min":0.81100000000000005,"15min":2.0600000000000001},"last":0.63700000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55300000000000005,"15min":0.57499999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.314},"max":{"1min":0.69699999999999995,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55900000000000005,"15min":0.58699999999999997},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73999999999999999,"5min":0.77900000000000003,"15min":2.5630000000000002},"last":0.56399999999999995}]},{"osd":4,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.42899999999999999,"15min":0.45500000000000002},"min":{"1min":0.313,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.65100000000000002,"5min":0.67200000000000004,"15min":2.3700000000000001},"last":0.376},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45900000000000002,"15min":0.48099999999999998},"min":{"1min":0.32600000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.68200000000000005,"5min":0.82699999999999996,"15min":0.89200000000000002},"last":0.49299999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.47199999999999998},"min":{"1min":0.34300000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66900000000000004,"5min":0.67700000000000005,"15min":2.5990000000000002},"last":0.436},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.49299999999999999,"15min":0.51400000000000001},"min":{"1min":0.374,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71399999999999997,"5min":0.76200000000000001,"15min":1.99},"last":0.47099999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51400000000000001,"15min":0.53800000000000003},"min":{"1min":0.374,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73799999999999999,"5min":0.76200000000000001,"15min":3.5270000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54400000000000004,"15min":0.55900000000000005},"min":{"1min":0.43099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78900000000000003,"5min":0.81499999999999995,"15min":0.82199999999999995},"last":0.60499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:49:26.193 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:49:26.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.363+0000 7f5b9261b700 1 Processor -- start 2024-01-26T23:49:26.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.363+0000 7f5b9261b700 1 -- start start 2024-01-26T23:49:26.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.364+0000 7f5b9261b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5b8c1583a0 0x7f5b8c158770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:26.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.364+0000 7f5b9261b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b8c0b1cc0 0x7f5b8c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:26.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.364+0000 7f5b9261b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8c0b2660 con 0x7f5b8c06b4f0 2024-01-26T23:49:26.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.364+0000 7f5b9261b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b8c06a4e0 con 0x7f5b8c1583a0 2024-01-26T23:49:26.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.364+0000 7f5b9261b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c06b060 con 0x7f5b8c0b1cc0 2024-01-26T23:49:26.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.364+0000 7f5b8bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5b8c1583a0 0x7f5b8c158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:26.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.364+0000 7f5b8b7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5b8c06b4f0 0x7f5b8c0c8eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43450/0 (socket says 172.21.15.46:43450) 2024-01-26T23:49:26.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.364+0000 7f5b8b7fe700 1 -- 172.21.15.46:0/1058188549 learned_addr learned my addr 172.21.15.46:0/1058188549 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:26.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.364+0000 7f5b90bb8700 1 --2- 172.21.15.46:0/1058188549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b8c0b1cc0 0x7f5b8c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:26.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.364+0000 7f5b8bfff700 1 -- 172.21.15.46:0/1058188549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b8c0b1cc0 msgr2=0x7f5b8c0b2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:26.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.364+0000 7f5b8bfff700 1 --2- 172.21.15.46:0/1058188549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b8c0b1cc0 0x7f5b8c0b2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:26.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.365+0000 7f5b8bfff700 1 -- 172.21.15.46:0/1058188549 >> v1:172.21.15.46:6789/0 conn(0x7f5b8c06b4f0 legacy=0x7f5b8c0c8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:26.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.365+0000 7f5b8bfff700 1 -- 172.21.15.46:0/1058188549 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b8c06a730 con 0x7f5b8c1583a0 2024-01-26T23:49:26.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.365+0000 7f5b8affd700 1 -- 172.21.15.46:0/1058188549 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613926259 0 0) 0x7f5b80001110 con 0x7f5b8c06b4f0 2024-01-26T23:49:26.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.365+0000 7f5b90bb8700 1 --2- 172.21.15.46:0/1058188549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b8c0b1cc0 0x7f5b8c0b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:49:26.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.365+0000 7f5b8bfff700 1 --2- 172.21.15.46:0/1058188549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b8c1583a0 0x7f5b8c158770 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c0098d0 tx=0x7f5b7c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e13789168537dd28 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:26.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.365+0000 7f5b8affd700 1 -- 172.21.15.46:0/1058188549 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c009e70 con 0x7f5b8c1583a0 2024-01-26T23:49:26.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.365+0000 7f5b8affd700 1 -- 172.21.15.46:0/1058188549 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c005c10 con 0x7f5b8c1583a0 2024-01-26T23:49:26.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.365+0000 7f5b8affd700 1 -- 172.21.15.46:0/1058188549 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c00b650 con 0x7f5b8c1583a0 2024-01-26T23:49:26.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.365+0000 7f5b9261b700 1 -- 172.21.15.46:0/1058188549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b8c1583a0 msgr2=0x7f5b8c158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:26.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.365+0000 7f5b9261b700 1 --2- 172.21.15.46:0/1058188549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b8c1583a0 0x7f5b8c158770 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c0098d0 tx=0x7f5b7c009120 comp rx=0 tx=0).stop 2024-01-26T23:49:26.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.365+0000 7f5b9261b700 1 -- 172.21.15.46:0/1058188549 shutdown_connections 2024-01-26T23:49:26.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.365+0000 7f5b9261b700 1 --2- 172.21.15.46:0/1058188549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b8c0b1cc0 0x7f5b8c0b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:26.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.365+0000 7f5b9261b700 1 --2- 172.21.15.46:0/1058188549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b8c1583a0 0x7f5b8c158770 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:26.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.365+0000 7f5b9261b700 1 -- 172.21.15.46:0/1058188549 >> 172.21.15.46:0/1058188549 conn(0x7f5b8c06e2c0 msgr2=0x7f5b8c0ae620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:26.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.366+0000 7f5b9261b700 1 -- 172.21.15.46:0/1058188549 shutdown_connections 2024-01-26T23:49:26.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.366+0000 7f5b9261b700 1 -- 172.21.15.46:0/1058188549 wait complete. 2024-01-26T23:49:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.367+0000 7f5b9261b700 1 Processor -- start 2024-01-26T23:49:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.367+0000 7f5b9261b700 1 -- start start 2024-01-26T23:49:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.368+0000 7f5b9261b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b8c06b4f0 0x7f5b8c156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.368+0000 7f5b9261b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b8c0b1cc0 0x7f5b8c0cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.368+0000 7f5b9261b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b8c1583a0 0x7f5b8c0cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.368+0000 7f5b9261b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c1570c0 con 0x7f5b8c06b4f0 2024-01-26T23:49:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.368+0000 7f5b9261b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c14f6a0 con 0x7f5b8c1583a0 2024-01-26T23:49:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.368+0000 7f5b9261b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c13ef00 con 0x7f5b8c0b1cc0 2024-01-26T23:49:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.368+0000 7f5b8bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b8c06b4f0 0x7f5b8c156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.368+0000 7f5b8bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b8c06b4f0 0x7f5b8c156b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40170/0 (socket says 172.21.15.46:40170) 2024-01-26T23:49:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.368+0000 7f5b8bfff700 1 -- 172.21.15.46:0/2097249224 learned_addr learned my addr 172.21.15.46:0/2097249224 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.368+0000 7f5b8b7fe700 1 --2- 172.21.15.46:0/2097249224 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b8c0b1cc0 0x7f5b8c0cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.368+0000 7f5b90bb8700 1 --2- 172.21.15.46:0/2097249224 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b8c1583a0 0x7f5b8c0cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.368+0000 7f5b8bfff700 1 -- 172.21.15.46:0/2097249224 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b8c0b1cc0 msgr2=0x7f5b8c0cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.368+0000 7f5b8bfff700 1 --2- 172.21.15.46:0/2097249224 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b8c0b1cc0 0x7f5b8c0cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.368+0000 7f5b8bfff700 1 -- 172.21.15.46:0/2097249224 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b8c1583a0 msgr2=0x7f5b8c0cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.368+0000 7f5b8bfff700 1 --2- 172.21.15.46:0/2097249224 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b8c1583a0 0x7f5b8c0cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.368+0000 7f5b8bfff700 1 -- 172.21.15.46:0/2097249224 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c0093c0 con 0x7f5b8c06b4f0 2024-01-26T23:49:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.368+0000 7f5b8bfff700 1 --2- 172.21.15.46:0/2097249224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b8c06b4f0 0x7f5b8c156b80 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c00b3c0 tx=0x7f5b7c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.369+0000 7f5b73fff700 1 -- 172.21.15.46:0/2097249224 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c009e70 con 0x7f5b8c06b4f0 2024-01-26T23:49:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.369+0000 7f5b73fff700 1 -- 172.21.15.46:0/2097249224 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c01dcf0 con 0x7f5b8c06b4f0 2024-01-26T23:49:26.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.369+0000 7f5b73fff700 1 -- 172.21.15.46:0/2097249224 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c026680 con 0x7f5b8c06b4f0 2024-01-26T23:49:26.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.369+0000 7f5b9261b700 1 -- 172.21.15.46:0/2097249224 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b8c16a830 con 0x7f5b8c06b4f0 2024-01-26T23:49:26.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.369+0000 7f5b9261b700 1 -- 172.21.15.46:0/2097249224 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8c16b170 con 0x7f5b8c06b4f0 2024-01-26T23:49:26.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.370+0000 7f5b9261b700 1 -- 172.21.15.46:0/2097249224 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8c04c710 con 0x7f5b8c06b4f0 2024-01-26T23:49:26.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.370+0000 7f5b73fff700 1 -- 172.21.15.46:0/2097249224 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5b7c01a420 con 0x7f5b8c06b4f0 2024-01-26T23:49:26.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.371+0000 7f5b73fff700 1 --2- 172.21.15.46:0/2097249224 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5b78066f30 0x7f5b780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:26.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.371+0000 7f5b73fff700 1 -- 172.21.15.46:0/2097249224 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f5b7c0860a0 con 0x7f5b8c06b4f0 2024-01-26T23:49:26.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.371+0000 7f5b8b7fe700 1 --2- 172.21.15.46:0/2097249224 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5b78066f30 0x7f5b780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:26.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.372+0000 7f5b8b7fe700 1 --2- 172.21.15.46:0/2097249224 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5b78066f30 0x7f5b780693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c0b1c90 tx=0x7f5b80009040 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:26.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.375+0000 7f5b73fff700 1 -- 172.21.15.46:0/2097249224 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5b7c0530c0 con 0x7f5b8c06b4f0 2024-01-26T23:49:26.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.575+0000 7f5b9261b700 1 -- 172.21.15.46:0/2097249224 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5b8c1574b0 con 0x7f5b78066f30 2024-01-26T23:49:26.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.582+0000 7f5b73fff700 1 -- 172.21.15.46:0/2097249224 <== mgr.7148 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+274554 (secure 0 0 0) 0x7f5b8c1574b0 con 0x7f5b78066f30 2024-01-26T23:49:26.586 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.586+0000 7f5b9261b700 1 -- 172.21.15.46:0/2097249224 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5b78066f30 msgr2=0x7f5b780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.586+0000 7f5b9261b700 1 --2- 172.21.15.46:0/2097249224 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5b78066f30 0x7f5b780693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c0b1c90 tx=0x7f5b80009040 comp rx=0 tx=0).stop 2024-01-26T23:49:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.586+0000 7f5b9261b700 1 -- 172.21.15.46:0/2097249224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b8c06b4f0 msgr2=0x7f5b8c156b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.586+0000 7f5b9261b700 1 --2- 172.21.15.46:0/2097249224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b8c06b4f0 0x7f5b8c156b80 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c00b3c0 tx=0x7f5b7c016000 comp rx=0 tx=0).stop 2024-01-26T23:49:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.586+0000 7f5b9261b700 1 -- 172.21.15.46:0/2097249224 shutdown_connections 2024-01-26T23:49:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.586+0000 7f5b9261b700 1 --2- 172.21.15.46:0/2097249224 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5b78066f30 0x7f5b780693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.586+0000 7f5b9261b700 1 --2- 172.21.15.46:0/2097249224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b8c06b4f0 0x7f5b8c156b80 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.586+0000 7f5b9261b700 1 --2- 172.21.15.46:0/2097249224 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b8c1583a0 0x7f5b8c0cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.586+0000 7f5b9261b700 1 --2- 172.21.15.46:0/2097249224 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b8c0b1cc0 0x7f5b8c0cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.586+0000 7f5b9261b700 1 -- 172.21.15.46:0/2097249224 >> 172.21.15.46:0/2097249224 conn(0x7f5b8c06e2c0 msgr2=0x7f5b8c158d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.586+0000 7f5b9261b700 1 -- 172.21.15.46:0/2097249224 shutdown_connections 2024-01-26T23:49:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.586+0000 7f5b9261b700 1 -- 172.21.15.46:0/2097249224 wait complete. 2024-01-26T23:49:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:49:26.600 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-26T23:49:24.870286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":2314,"num_write":360,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8584,"ondisk_log_size":8584,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":2450492,"kb_used_data":14844,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752524228,"statfs":{"total":773094113280,"available":770584809472,"internally_reserved":0,"allocated":15200256,"data_stored":8390475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.959692"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:20.909706+0000","last_change":"2024-01-26T23:49:20.909706+0000","last_active":"2024-01-26T23:49:20.909706+0000","last_peered":"2024-01-26T23:49:20.909706+0000","last_clean":"2024-01-26T23:49:20.909706+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:20.909706+0000","last_undegraded":"2024-01-26T23:49:20.909706+0000","last_fullsized":"2024-01-26T23:49:20.909706+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-26T23:49:19.907877+0000","last_change":"2024-01-26T23:49:19.907877+0000","last_active":"2024-01-26T23:49:19.907877+0000","last_peered":"2024-01-26T23:49:19.907877+0000","last_clean":"2024-01-26T23:49:19.907877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-26T23:49:19.907877+0000","last_undegraded":"2024-01-26T23:49:19.907877+0000","last_fullsized":"2024-01-26T23:49:19.907877+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"135'8584","reported_seq":9102,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-01-26T23:49:17.493871+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:49:17.493871+0000","last_peered":"2024-01-26T23:49:17.493871+0000","last_clean":"2024-01-26T23:49:17.493871+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:49:17.493871+0000","last_undegraded":"2024-01-26T23:49:17.493871+0000","last_fullsized":"2024-01-26T23:49:17.493871+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":2314,"num_write":360,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":2314,"num_write":360,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8584,"ondisk_log_size":8584,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575082,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311868,"kb_used_data":2292,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059972,"statfs":{"total":96636764160,"available":96317411328,"internally_reserved":0,"allocated":2347008,"data_stored":1492375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51500000000000001,"15min":0.56200000000000006},"min":{"1min":0.40699999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.74199999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52700000000000002,"15min":0.57599999999999996},"min":{"1min":0.35899999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67200000000000004,"5min":0.82499999999999996,"15min":0.83999999999999997},"last":0.629}]},{"osd":1,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54000000000000004,"15min":0.57399999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.77600000000000002,"5min":0.77800000000000002,"15min":0.82899999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55100000000000005,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.70399999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.56799999999999995}]},{"osd":2,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.57199999999999995},"min":{"1min":0.38700000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69199999999999995,"5min":0.80000000000000004,"15min":0.81699999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.38900000000000001,"5min":0.30199999999999999,"15min":0.27400000000000002},"max":{"1min":0.746,"5min":0.81999999999999995,"15min":0.88600000000000001},"last":0.60899999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54700000000000004,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.377,"15min":0.377},"max":{"1min":0.70399999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57499999999999996,"15min":0.59699999999999998},"min":{"1min":0.34399999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77900000000000003,"5min":0.86199999999999999,"15min":0.86399999999999999},"last":0.81399999999999995}]},{"osd":4,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.438,"15min":0.45200000000000001},"min":{"1min":0.34599999999999997,"5min":0.252,"15min":0.247},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.93899999999999995},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.45300000000000001,"15min":0.47399999999999998},"min":{"1min":0.32400000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67300000000000004,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.498}]},{"osd":5,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47599999999999998,"15min":0.48699999999999999},"min":{"1min":0.30499999999999999,"5min":0.27200000000000002,"15min":0.22800000000000001},"max":{"1min":0.749,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.751},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.504,"15min":0.52300000000000002},"min":{"1min":0.35799999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73399999999999999,"5min":0.74199999999999999,"15min":0.90200000000000002},"last":0.78900000000000003}]},{"osd":6,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53100000000000003,"15min":0.54800000000000004},"min":{"1min":0.41699999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76400000000000001,"5min":0.78700000000000003,"15min":0.95899999999999996},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.52700000000000002,"15min":0.54000000000000004},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75900000000000001,"5min":0.79500000000000004,"15min":0.85199999999999998},"last":0.71399999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303096,"kb_used_data":1712,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068744,"statfs":{"total":96636764160,"available":96326393856,"internally_reserved":0,"allocated":1753088,"data_stored":902007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52900000000000003,"15min":0.56499999999999995},"min":{"1min":0.40300000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.423,"15min":1.423},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.57899999999999996},"min":{"1min":0.36099999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.91100000000000003},"last":0.748}]},{"osd":1,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51700000000000002,"15min":0.55800000000000005},"min":{"1min":0.32200000000000001,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.629},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.58399999999999996},"min":{"1min":0.38200000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.66800000000000004,"5min":1.498,"15min":1.498},"last":0.76500000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52800000000000002,"15min":0.56999999999999995},"min":{"1min":0.38100000000000001,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.85799999999999998},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52600000000000002,"15min":0.57099999999999995},"min":{"1min":0.377,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69299999999999995,"5min":0.77000000000000002,"15min":0.82799999999999996},"last":0.53600000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57199999999999995,"15min":0.59099999999999997},"min":{"1min":0.34699999999999998,"5min":0.33800000000000002,"15min":0.29999999999999999},"max":{"1min":0.746,"5min":1.595,"15min":1.595},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56899999999999995,"15min":0.60599999999999998},"min":{"1min":0.45600000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.751,"5min":1.369,"15min":1.369},"last":0.57499999999999996}]},{"osd":4,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.443,"15min":0.46200000000000002},"min":{"1min":0.33800000000000002,"5min":0.245,"15min":0.22},"max":{"1min":0.67000000000000004,"5min":1.556,"15min":1.556},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.47399999999999998,"15min":0.48199999999999998},"min":{"1min":0.33200000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.86199999999999999},"last":0.46000000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45500000000000002,"15min":0.47099999999999997},"min":{"1min":0.32700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67900000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.51700000000000002,"15min":0.52000000000000002},"min":{"1min":0.30499999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.70999999999999996,"5min":1.47,"15min":1.47},"last":0.73499999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.5,"15min":0.51800000000000002},"min":{"1min":0.40100000000000002,"5min":0.26700000000000002,"15min":0.246},"max":{"1min":0.80200000000000005,"5min":0.81799999999999995,"15min":0.85899999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50900000000000001,"15min":0.52400000000000002},"min":{"1min":0.32300000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.80400000000000005},"last":0.71999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307192,"kb_used_data":1712,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064648,"statfs":{"total":96636764160,"available":96322199552,"internally_reserved":0,"allocated":1753088,"data_stored":902007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42099999999999999,"15min":0.40600000000000003},"min":{"1min":0.28999999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.64400000000000002,"5min":0.65200000000000002,"15min":0.72599999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.43099999999999999,"15min":0.44},"min":{"1min":0.30199999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.52600000000000002,"5min":1.0269999999999999,"15min":1.337},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40400000000000003,"15min":0.40799999999999997},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.23599999999999999},"max":{"1min":0.67700000000000005,"5min":0.68799999999999994,"15min":0.73299999999999998},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.47299999999999998,"15min":0.47399999999999998},"min":{"1min":0.35399999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.70199999999999996,"5min":0.74099999999999999,"15min":0.77300000000000002},"last":0.56100000000000005}]},{"osd":3,"last update":"Fri Jan 26 23:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51800000000000002,"15min":0.51400000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32100000000000001},"max":{"1min":0.78900000000000003,"5min":0.82199999999999995,"15min":1.782},"last":0.502},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.46899999999999997,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.312,"15min":0.26600000000000001},"max":{"1min":0.59999999999999998,"5min":0.81399999999999995,"15min":1.379},"last":0.38300000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.623,"15min":0.64100000000000001},"min":{"1min":0.42999999999999999,"5min":0.40799999999999997,"15min":0.39400000000000002},"max":{"1min":0.89200000000000002,"5min":0.94099999999999995,"15min":1.0820000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.65000000000000002,"15min":0.65600000000000003},"min":{"1min":0.45200000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.95699999999999996},"last":0.61499999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.65600000000000003,"15min":0.66800000000000004},"min":{"1min":0.54000000000000004,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":0.91100000000000003,"5min":0.96699999999999997,"15min":1.208},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.64600000000000002,"15min":0.65700000000000003},"min":{"1min":0.502,"5min":0.45100000000000001,"15min":0.42499999999999999},"max":{"1min":0.96399999999999997,"5min":0.96599999999999997,"15min":1.018},"last":0.57999999999999996}]},{"osd":6,"last update":"Fri Jan 26 23:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.67100000000000004},"min":{"1min":0.47899999999999998,"5min":0.47799999999999998,"15min":0.42599999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.2529999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.66600000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93200000000000005,"5min":1.01,"15min":1.022},"last":0.52800000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.68899999999999995,"15min":0.70599999999999996},"min":{"1min":0.49099999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.0069999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70299999999999996,"15min":0.71899999999999997},"min":{"1min":0.54600000000000004,"5min":0.496,"15min":0.496},"max":{"1min":0.98499999999999999,"5min":1.0049999999999999,"15min":1.296},"last":0.63600000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575081,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307760,"kb_used_data":2280,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064080,"statfs":{"total":96636764160,"available":96321617920,"internally_reserved":0,"allocated":2334720,"data_stored":1486058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.40600000000000003},"min":{"1min":0.25800000000000001,"5min":0.23899999999999999,"15min":0.23100000000000001},"max":{"1min":0.88500000000000001,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.42999999999999999,"15min":0.45100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65800000000000003,"5min":1.363,"15min":1.4490000000000001},"last":0.308}]},{"osd":2,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.41099999999999998,"15min":0.42299999999999999},"min":{"1min":0.28899999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.89600000000000002},"last":0.371},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.33900000000000002,"5min":0.32700000000000001,"15min":0.28000000000000003},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.41199999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.49199999999999999,"15min":0.47399999999999998},"min":{"1min":0.32800000000000001,"5min":0.32300000000000001,"15min":0.30599999999999999},"max":{"1min":0.85199999999999998,"5min":2.145,"15min":2.145},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36099999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":1.198},"last":0.501}]},{"osd":4,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60899999999999999,"15min":0.63800000000000001},"min":{"1min":0.41699999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78600000000000003,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.59699999999999998,"15min":0.63700000000000001},"min":{"1min":0.44400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.94299999999999995},"last":0.68200000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":1.329,"15min":1.329},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.66300000000000003,"15min":0.67000000000000004},"min":{"1min":0.46899999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82799999999999996,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.623}]},{"osd":6,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.61799999999999999,"15min":0.65100000000000002},"min":{"1min":0.47299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.89800000000000002,"5min":0.98699999999999999,"15min":1.032},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.65400000000000003,"15min":0.68000000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.41799999999999998},"max":{"1min":0.93100000000000005,"5min":1.286,"15min":1.286},"last":0.66000000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.70599999999999996,"15min":0.81699999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.435},"max":{"1min":0.81799999999999995,"5min":4.8710000000000004,"15min":25.965},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.73899999999999999,"15min":0.86099999999999999},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.433},"max":{"1min":0.88200000000000001,"5min":4.9199999999999999,"15min":33.085000000000001},"last":0.67200000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575081,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307192,"kb_used_data":1712,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064648,"statfs":{"total":96636764160,"available":96322199552,"internally_reserved":0,"allocated":1753088,"data_stored":902007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.38800000000000001,"15min":0.39000000000000001},"min":{"1min":0.29399999999999998,"5min":0.26900000000000002,"15min":0.255},"max":{"1min":0.53800000000000003,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.43099999999999999,"15min":0.42699999999999999},"min":{"1min":0.26800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60499999999999998,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.435,"15min":0.42799999999999999},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.248},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52000000000000002,"15min":0.51000000000000001},"min":{"1min":0.379,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.044,"5min":1.171,"15min":1.171},"last":0.63900000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46800000000000003,"15min":0.48399999999999999},"min":{"1min":0.34000000000000002,"5min":0.32300000000000001,"15min":0.313},"max":{"1min":0.68700000000000006,"5min":0.72799999999999998,"15min":0.74099999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78200000000000003,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.621}]},{"osd":4,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.60799999999999998,"15min":0.63},"min":{"1min":0.53000000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.82299999999999995,"5min":1.087,"15min":1.087},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.46300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.95499999999999996,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.73799999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.64500000000000002,"15min":0.65600000000000003},"min":{"1min":0.495,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0740000000000001,"5min":1.24,"15min":1.24},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.66400000000000003,"15min":0.67600000000000005},"min":{"1min":0.53400000000000003,"5min":0.41799999999999998,"15min":0.35199999999999998},"max":{"1min":0.93000000000000005,"5min":0.97699999999999998,"15min":1.085},"last":0.72999999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.67800000000000005},"min":{"1min":0.495,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.98599999999999999,"5min":1.022,"15min":1.1080000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66600000000000004,"15min":0.68100000000000005},"min":{"1min":0.51800000000000002,"5min":0.42799999999999999,"15min":0.38800000000000001},"max":{"1min":0.97499999999999998,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.83199999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.68999999999999995,"15min":0.70399999999999996},"min":{"1min":0.52800000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94999999999999996,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.70999999999999996,"15min":0.72699999999999998},"min":{"1min":0.53400000000000003,"5min":0.48099999999999998,"15min":0.42099999999999999},"max":{"1min":1.0189999999999999,"5min":1.052,"15min":1.139},"last":0.75}]}]},{"osd":3,"up_from":13,"seq":55834575080,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307192,"kb_used_data":1712,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064648,"statfs":{"total":96636764160,"available":96322199552,"internally_reserved":0,"allocated":1753088,"data_stored":902007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38400000000000001,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.55400000000000005,"5min":0.59699999999999998,"15min":1.8540000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.42899999999999999,"15min":0.438},"min":{"1min":0.29799999999999999,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":1.2290000000000001},"last":0.38700000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44500000000000001,"15min":0.45200000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":1.633},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.47999999999999998,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":1.0760000000000001},"last":0.48499999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48399999999999999,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65000000000000002,"5min":0.70399999999999996,"15min":1.4059999999999999},"last":0.5},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.44500000000000001,"15min":0.45800000000000002},"min":{"1min":0.34100000000000003,"5min":0.29499999999999998,"15min":0.28299999999999997},"max":{"1min":0.629,"5min":0.69399999999999995,"15min":1.1870000000000001},"last":0.44800000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.63,"15min":0.65800000000000003},"min":{"1min":0.48299999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":3.306},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.44800000000000001,"5min":0.40100000000000002,"15min":0.379},"max":{"1min":0.81499999999999995,"5min":0.85199999999999998,"15min":1.008},"last":0.60399999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.63600000000000001,"15min":0.65200000000000002},"min":{"1min":0.48599999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.85199999999999998,"5min":0.90400000000000003,"15min":1.593},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.47599999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.86299999999999999,"5min":0.92300000000000004,"15min":1.3819999999999999},"last":0.51800000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.66900000000000004},"min":{"1min":0.42299999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":1.506},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67200000000000004,"15min":0.68999999999999995},"min":{"1min":0.55600000000000005,"5min":0.46700000000000003,"15min":0.44400000000000001},"max":{"1min":0.88500000000000001,"5min":0.89000000000000001,"15min":1.427},"last":0.57899999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64400000000000002,"15min":0.68799999999999994},"min":{"1min":0.46700000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":3.238},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.71199999999999997,"15min":0.73399999999999999},"min":{"1min":0.54200000000000004,"5min":0.51900000000000002,"15min":0.48899999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":3.351},"last":0.67700000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303096,"kb_used_data":1712,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068744,"statfs":{"total":96636764160,"available":96326393856,"internally_reserved":0,"allocated":1753088,"data_stored":902007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53400000000000003,"15min":0.56799999999999995},"min":{"1min":0.34799999999999998,"5min":0.28999999999999998,"15min":0.28699999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":1.21},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58399999999999996,"15min":0.58299999999999996},"min":{"1min":0.377,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.81499999999999995,"5min":5.367,"15min":5.367},"last":0.432}]},{"osd":1,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56899999999999995,"15min":0.58699999999999997},"min":{"1min":0.378,"5min":0.311,"15min":0.311},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.54700000000000004,"15min":0.57399999999999995},"min":{"1min":0.40699999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79800000000000004,"5min":0.80600000000000005,"15min":0.81699999999999995},"last":0.61599999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.54600000000000004,"15min":0.56599999999999995},"min":{"1min":0.39300000000000002,"5min":0.33900000000000002,"15min":0.32700000000000001},"max":{"1min":0.747,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.57699999999999996,"15min":0.59599999999999997},"min":{"1min":0.43099999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":1.7050000000000001},"last":0.48599999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57299999999999995,"15min":0.58599999999999997},"min":{"1min":0.44900000000000001,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.35699999999999998},"max":{"1min":0.82499999999999996,"5min":0.94199999999999995,"15min":1.05},"last":0.65100000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.46500000000000002},"min":{"1min":0.31,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.48099999999999998,"15min":0.48399999999999999},"min":{"1min":0.40100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.56200000000000006}]},{"osd":6,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.46500000000000002,"15min":0.47399999999999998},"min":{"1min":0.34999999999999998,"5min":0.28599999999999998,"15min":0.27200000000000002},"max":{"1min":0.76600000000000001,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.501},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.42299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.57999999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.52600000000000002,"15min":0.54000000000000004},"min":{"1min":0.35599999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.56200000000000006,"15min":0.56899999999999995},"min":{"1min":0.436,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.88800000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.74199999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303096,"kb_used_data":1712,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068744,"statfs":{"total":96636764160,"available":96326393856,"internally_reserved":0,"allocated":1753088,"data_stored":902007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.56399999999999995},"min":{"1min":0.35199999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.71099999999999997,"5min":0.81799999999999995,"15min":2.0289999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52700000000000002,"15min":0.56699999999999995},"min":{"1min":0.41999999999999998,"5min":0.29999999999999999,"15min":0.28899999999999998},"max":{"1min":0.70799999999999996,"5min":0.77400000000000002,"15min":0.82199999999999995},"last":0.53300000000000003}]},{"osd":1,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56299999999999994,"15min":0.60399999999999998},"min":{"1min":0.443,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":0.754,"5min":0.79500000000000004,"15min":2.3319999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59899999999999998,"15min":0.626},"min":{"1min":0.45800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76400000000000001,"5min":0.80500000000000005,"15min":1.1739999999999999},"last":0.62}]},{"osd":2,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53700000000000003,"15min":0.55700000000000005},"min":{"1min":0.36799999999999999,"5min":0.34499999999999997,"15min":0.30099999999999999},"max":{"1min":0.68400000000000005,"5min":0.79400000000000004,"15min":0.83799999999999997},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55300000000000005,"15min":0.58699999999999997},"min":{"1min":0.41299999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72899999999999998,"5min":0.81100000000000005,"15min":2.0600000000000001},"last":0.63700000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55300000000000005,"15min":0.57499999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.314},"max":{"1min":0.69699999999999995,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55900000000000005,"15min":0.58699999999999997},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73999999999999999,"5min":0.77900000000000003,"15min":2.5630000000000002},"last":0.56399999999999995}]},{"osd":4,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.42899999999999999,"15min":0.45500000000000002},"min":{"1min":0.313,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.65100000000000002,"5min":0.67200000000000004,"15min":2.3700000000000001},"last":0.376},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45900000000000002,"15min":0.48099999999999998},"min":{"1min":0.32600000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.68200000000000005,"5min":0.82699999999999996,"15min":0.89200000000000002},"last":0.49299999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.47199999999999998},"min":{"1min":0.34300000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66900000000000004,"5min":0.67700000000000005,"15min":2.5990000000000002},"last":0.436},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.49299999999999999,"15min":0.51400000000000001},"min":{"1min":0.374,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71399999999999997,"5min":0.76200000000000001,"15min":1.99},"last":0.47099999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51400000000000001,"15min":0.53800000000000003},"min":{"1min":0.374,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73799999999999999,"5min":0.76200000000000001,"15min":3.5270000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54400000000000004,"15min":0.55900000000000005},"min":{"1min":0.43099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78900000000000003,"5min":0.81499999999999995,"15min":0.82199999999999995},"last":0.60499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:49:26.604 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:49:26.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.776+0000 7f17df593700 1 Processor -- start 2024-01-26T23:49:26.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.776+0000 7f17df593700 1 -- start start 2024-01-26T23:49:26.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.777+0000 7f17df593700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f17d8159980 0x7f17d80af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:26.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.777+0000 7f17df593700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d8069100 con 0x7f17d81595b0 2024-01-26T23:49:26.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.777+0000 7f17df593700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d806bc90 con 0x7f17d806abf0 2024-01-26T23:49:26.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.777+0000 7f17df593700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d815def0 con 0x7f17d8159980 2024-01-26T23:49:26.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.777+0000 7f17dd32f700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f17d81595b0 0x7f17d806a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43458/0 (socket says 172.21.15.46:43458) 2024-01-26T23:49:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.777+0000 7f17dd32f700 1 -- 172.21.15.46:0/2822838807 learned_addr learned my addr 172.21.15.46:0/2822838807 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.777+0000 7f17ddb30700 1 --2- 172.21.15.46:0/2822838807 >> v2:172.21.15.106:3300/0 conn(0x7f17d8159980 0x7f17d80af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.777+0000 7f17cffff700 1 -- 172.21.15.46:0/2822838807 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729711615 0 0) 0x7f17d8069100 con 0x7f17d81595b0 2024-01-26T23:49:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.777+0000 7f17cffff700 1 -- 172.21.15.46:0/2822838807 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17c4003570 con 0x7f17d81595b0 2024-01-26T23:49:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.778+0000 7f17ddb30700 1 -- 172.21.15.46:0/2822838807 >> v1:172.21.15.106:6789/0 conn(0x7f17d806abf0 legacy=0x7f17d80a9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.778+0000 7f17ddb30700 1 -- 172.21.15.46:0/2822838807 >> v1:172.21.15.46:6789/0 conn(0x7f17d81595b0 legacy=0x7f17d806a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.778+0000 7f17cffff700 1 -- 172.21.15.46:0/2822838807 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941584376 0 0) 0x7f17d806bc90 con 0x7f17d806abf0 2024-01-26T23:49:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.778+0000 7f17ddb30700 1 -- 172.21.15.46:0/2822838807 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d815e190 con 0x7f17d8159980 2024-01-26T23:49:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.778+0000 7f17ddb30700 1 --2- 172.21.15.46:0/2822838807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17d8159980 0x7f17d80af190 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009e60 tx=0x7f17d0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5ef32377ddab8f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.778+0000 7f17cffff700 1 -- 172.21.15.46:0/2822838807 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d000e070 con 0x7f17d8159980 2024-01-26T23:49:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.778+0000 7f17cffff700 1 -- 172.21.15.46:0/2822838807 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d000bb60 con 0x7f17d8159980 2024-01-26T23:49:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.778+0000 7f17cffff700 1 -- 172.21.15.46:0/2822838807 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0012430 con 0x7f17d8159980 2024-01-26T23:49:26.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.778+0000 7f17df593700 1 -- 172.21.15.46:0/2822838807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17d8159980 msgr2=0x7f17d80af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:26.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.778+0000 7f17df593700 1 --2- 172.21.15.46:0/2822838807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17d8159980 0x7f17d80af190 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009e60 tx=0x7f17d0009780 comp rx=0 tx=0).stop 2024-01-26T23:49:26.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.779+0000 7f17df593700 1 -- 172.21.15.46:0/2822838807 shutdown_connections 2024-01-26T23:49:26.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.779+0000 7f17df593700 1 --2- 172.21.15.46:0/2822838807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17d8159980 0x7f17d80af190 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:26.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.779+0000 7f17df593700 1 -- 172.21.15.46:0/2822838807 >> 172.21.15.46:0/2822838807 conn(0x7f17d806e2c0 msgr2=0x7f17d80ae620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:26.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.779+0000 7f17df593700 1 -- 172.21.15.46:0/2822838807 shutdown_connections 2024-01-26T23:49:26.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.779+0000 7f17df593700 1 -- 172.21.15.46:0/2822838807 wait complete. 2024-01-26T23:49:26.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.780+0000 7f17df593700 1 Processor -- start 2024-01-26T23:49:26.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.780+0000 7f17df593700 1 -- start start 2024-01-26T23:49:26.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17df593700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17d806abf0 0x7f17d815ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:26.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17df593700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17d81595b0 0x7f17d81476c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:26.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17df593700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17d8159980 0x7f17d8067f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:26.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17df593700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d8159f10 con 0x7f17d81595b0 2024-01-26T23:49:26.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17df593700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d80b2720 con 0x7f17d8159980 2024-01-26T23:49:26.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17df593700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d815a160 con 0x7f17d806abf0 2024-01-26T23:49:26.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17dcb2e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17d81595b0 0x7f17d81476c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:26.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17dcb2e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17d81595b0 0x7f17d81476c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40182/0 (socket says 172.21.15.46:40182) 2024-01-26T23:49:26.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17dcb2e700 1 -- 172.21.15.46:0/3724133043 learned_addr learned my addr 172.21.15.46:0/3724133043 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:26.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17ddb30700 1 --2- 172.21.15.46:0/3724133043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17d8159980 0x7f17d8067f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:26.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17dd32f700 1 --2- 172.21.15.46:0/3724133043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17d806abf0 0x7f17d815ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:26.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17dcb2e700 1 -- 172.21.15.46:0/3724133043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17d806abf0 msgr2=0x7f17d815ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:26.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17dcb2e700 1 --2- 172.21.15.46:0/3724133043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17d806abf0 0x7f17d815ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:26.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17dcb2e700 1 -- 172.21.15.46:0/3724133043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17d8159980 msgr2=0x7f17d8067f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:26.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17dcb2e700 1 --2- 172.21.15.46:0/3724133043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17d8159980 0x7f17d8067f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:26.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17dcb2e700 1 -- 172.21.15.46:0/3724133043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d0009a20 con 0x7f17d81595b0 2024-01-26T23:49:26.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17dd32f700 1 --2- 172.21.15.46:0/3724133043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17d806abf0 0x7f17d815ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:49:26.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.781+0000 7f17ddb30700 1 --2- 172.21.15.46:0/3724133043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17d8159980 0x7f17d8067f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:26.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.782+0000 7f17dcb2e700 1 --2- 172.21.15.46:0/3724133043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17d81595b0 0x7f17d81476c0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f17c800c7c0 tx=0x7f17c8007d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:26.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.782+0000 7f17cd7fa700 1 -- 172.21.15.46:0/3724133043 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c8011590 con 0x7f17d81595b0 2024-01-26T23:49:26.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.782+0000 7f17df593700 1 -- 172.21.15.46:0/3724133043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d815a440 con 0x7f17d81595b0 2024-01-26T23:49:26.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.782+0000 7f17cd7fa700 1 -- 172.21.15.46:0/3724133043 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c80116f0 con 0x7f17d81595b0 2024-01-26T23:49:26.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.782+0000 7f17cd7fa700 1 -- 172.21.15.46:0/3724133043 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c8011ab0 con 0x7f17d81595b0 2024-01-26T23:49:26.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.782+0000 7f17df593700 1 -- 172.21.15.46:0/3724133043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d8147c00 con 0x7f17d81595b0 2024-01-26T23:49:26.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.783+0000 7f17df593700 1 -- 172.21.15.46:0/3724133043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d815a960 con 0x7f17d81595b0 2024-01-26T23:49:26.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.783+0000 7f17cd7fa700 1 -- 172.21.15.46:0/3724133043 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f17c801b3b0 con 0x7f17d81595b0 2024-01-26T23:49:26.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.784+0000 7f17cd7fa700 1 --2- 172.21.15.46:0/3724133043 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f17bc066f30 0x7f17bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:26.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.784+0000 7f17cd7fa700 1 -- 172.21.15.46:0/3724133043 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f17c8083e20 con 0x7f17d81595b0 2024-01-26T23:49:26.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.784+0000 7f17dd32f700 1 --2- 172.21.15.46:0/3724133043 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f17bc066f30 0x7f17bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:26.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.784+0000 7f17dd32f700 1 --2- 172.21.15.46:0/3724133043 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f17bc066f30 0x7f17bc0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f17d4000c00 tx=0x7f17d4009040 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:26.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.788+0000 7f17cd7fa700 1 -- 172.21.15.46:0/3724133043 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f17d815a960 con 0x7f17d81595b0 2024-01-26T23:49:26.907 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-01-26T23:49:26.906+0000 7fd7131bb700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-26T23:49:26.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.990+0000 7f17df593700 1 -- 172.21.15.46:0/3724133043 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f17d815a960 con 0x7f17bc066f30 2024-01-26T23:49:26.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:26.998+0000 7f17cd7fa700 1 -- 172.21.15.46:0/3724133043 <== mgr.7148 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289774 (secure 0 0 0) 0x7f17d815a960 con 0x7f17bc066f30 2024-01-26T23:49:27.004 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:27.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.002+0000 7f17df593700 1 -- 172.21.15.46:0/3724133043 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f17bc066f30 msgr2=0x7f17bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:27.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.002+0000 7f17df593700 1 --2- 172.21.15.46:0/3724133043 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f17bc066f30 0x7f17bc0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f17d4000c00 tx=0x7f17d4009040 comp rx=0 tx=0).stop 2024-01-26T23:49:27.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.002+0000 7f17df593700 1 -- 172.21.15.46:0/3724133043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17d81595b0 msgr2=0x7f17d81476c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:27.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.002+0000 7f17df593700 1 --2- 172.21.15.46:0/3724133043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17d81595b0 0x7f17d81476c0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f17c800c7c0 tx=0x7f17c8007d70 comp rx=0 tx=0).stop 2024-01-26T23:49:27.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.002+0000 7f17df593700 1 -- 172.21.15.46:0/3724133043 shutdown_connections 2024-01-26T23:49:27.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.002+0000 7f17df593700 1 --2- 172.21.15.46:0/3724133043 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f17bc066f30 0x7f17bc0693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.002+0000 7f17df593700 1 --2- 172.21.15.46:0/3724133043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17d8159980 0x7f17d8067f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.002+0000 7f17df593700 1 --2- 172.21.15.46:0/3724133043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17d806abf0 0x7f17d815ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.002+0000 7f17df593700 1 --2- 172.21.15.46:0/3724133043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17d81595b0 0x7f17d81476c0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.002+0000 7f17df593700 1 -- 172.21.15.46:0/3724133043 >> 172.21.15.46:0/3724133043 conn(0x7f17d806e2c0 msgr2=0x7f17d80a9840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:27.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.002+0000 7f17df593700 1 -- 172.21.15.46:0/3724133043 shutdown_connections 2024-01-26T23:49:27.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.002+0000 7f17df593700 1 -- 172.21.15.46:0/3724133043 wait complete. 2024-01-26T23:49:27.008 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:49:27.016 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-26T23:49:26.870738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1163,"num_read_kb":2314,"num_write":404,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8622,"ondisk_log_size":8622,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2450952,"kb_used_data":15304,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752523768,"statfs":{"total":773094113280,"available":770584338432,"internally_reserved":0,"allocated":15671296,"data_stored":8625140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001406"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720478+0000","last_change":"2024-01-26T23:49:21.716820+0000","last_active":"2024-01-26T23:49:21.720478+0000","last_peered":"2024-01-26T23:49:21.720478+0000","last_clean":"2024-01-26T23:49:21.720478+0000","last_became_active":"2024-01-26T23:49:21.716559+0000","last_became_peered":"2024-01-26T23:49:21.716559+0000","last_unstale":"2024-01-26T23:49:21.720478+0000","last_undegraded":"2024-01-26T23:49:21.720478+0000","last_fullsized":"2024-01-26T23:49:21.720478+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:31:49.800868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713577+0000","last_change":"2024-01-26T23:49:21.706608+0000","last_active":"2024-01-26T23:49:21.713577+0000","last_peered":"2024-01-26T23:49:21.713577+0000","last_clean":"2024-01-26T23:49:21.713577+0000","last_became_active":"2024-01-26T23:49:21.706095+0000","last_became_peered":"2024-01-26T23:49:21.706095+0000","last_unstale":"2024-01-26T23:49:21.713577+0000","last_undegraded":"2024-01-26T23:49:21.713577+0000","last_fullsized":"2024-01-26T23:49:21.713577+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:40:00.810209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713754+0000","last_change":"2024-01-26T23:49:21.710842+0000","last_active":"2024-01-26T23:49:21.713754+0000","last_peered":"2024-01-26T23:49:21.713754+0000","last_clean":"2024-01-26T23:49:21.713754+0000","last_became_active":"2024-01-26T23:49:21.710542+0000","last_became_peered":"2024-01-26T23:49:21.710542+0000","last_unstale":"2024-01-26T23:49:21.713754+0000","last_undegraded":"2024-01-26T23:49:21.713754+0000","last_fullsized":"2024-01-26T23:49:21.713754+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:46:28.563573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720399+0000","last_change":"2024-01-26T23:49:21.715413+0000","last_active":"2024-01-26T23:49:21.720399+0000","last_peered":"2024-01-26T23:49:21.720399+0000","last_clean":"2024-01-26T23:49:21.720399+0000","last_became_active":"2024-01-26T23:49:21.715122+0000","last_became_peered":"2024-01-26T23:49:21.715122+0000","last_unstale":"2024-01-26T23:49:21.720399+0000","last_undegraded":"2024-01-26T23:49:21.720399+0000","last_fullsized":"2024-01-26T23:49:21.720399+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:23:01.412861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714883+0000","last_change":"2024-01-26T23:49:21.711789+0000","last_active":"2024-01-26T23:49:21.714883+0000","last_peered":"2024-01-26T23:49:21.714883+0000","last_clean":"2024-01-26T23:49:21.714883+0000","last_became_active":"2024-01-26T23:49:21.711475+0000","last_became_peered":"2024-01-26T23:49:21.711475+0000","last_unstale":"2024-01-26T23:49:21.714883+0000","last_undegraded":"2024-01-26T23:49:21.714883+0000","last_fullsized":"2024-01-26T23:49:21.714883+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:22:39.451844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.709457+0000","last_change":"2024-01-26T23:49:21.704699+0000","last_active":"2024-01-26T23:49:21.709457+0000","last_peered":"2024-01-26T23:49:21.709457+0000","last_clean":"2024-01-26T23:49:21.709457+0000","last_became_active":"2024-01-26T23:49:21.704402+0000","last_became_peered":"2024-01-26T23:49:21.704402+0000","last_unstale":"2024-01-26T23:49:21.709457+0000","last_undegraded":"2024-01-26T23:49:21.709457+0000","last_fullsized":"2024-01-26T23:49:21.709457+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:38.865560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.710793+0000","last_change":"2024-01-26T23:49:21.705657+0000","last_active":"2024-01-26T23:49:21.710793+0000","last_peered":"2024-01-26T23:49:21.710793+0000","last_clean":"2024-01-26T23:49:21.710793+0000","last_became_active":"2024-01-26T23:49:21.705381+0000","last_became_peered":"2024-01-26T23:49:21.705381+0000","last_unstale":"2024-01-26T23:49:21.710793+0000","last_undegraded":"2024-01-26T23:49:21.710793+0000","last_fullsized":"2024-01-26T23:49:21.710793+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:53:27.231318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716206+0000","last_change":"2024-01-26T23:49:21.714982+0000","last_active":"2024-01-26T23:49:21.716206+0000","last_peered":"2024-01-26T23:49:21.716206+0000","last_clean":"2024-01-26T23:49:21.716206+0000","last_became_active":"2024-01-26T23:49:21.714655+0000","last_became_peered":"2024-01-26T23:49:21.714655+0000","last_unstale":"2024-01-26T23:49:21.716206+0000","last_undegraded":"2024-01-26T23:49:21.716206+0000","last_fullsized":"2024-01-26T23:49:21.716206+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:46:36.438311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720506+0000","last_change":"2024-01-26T23:49:21.716127+0000","last_active":"2024-01-26T23:49:21.720506+0000","last_peered":"2024-01-26T23:49:21.720506+0000","last_clean":"2024-01-26T23:49:21.720506+0000","last_became_active":"2024-01-26T23:49:21.715868+0000","last_became_peered":"2024-01-26T23:49:21.715868+0000","last_unstale":"2024-01-26T23:49:21.720506+0000","last_undegraded":"2024-01-26T23:49:21.720506+0000","last_fullsized":"2024-01-26T23:49:21.720506+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:27:10.909797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.705419+0000","last_change":"2024-01-26T23:49:21.701945+0000","last_active":"2024-01-26T23:49:21.705419+0000","last_peered":"2024-01-26T23:49:21.705419+0000","last_clean":"2024-01-26T23:49:21.705419+0000","last_became_active":"2024-01-26T23:49:21.701542+0000","last_became_peered":"2024-01-26T23:49:21.701542+0000","last_unstale":"2024-01-26T23:49:21.705419+0000","last_undegraded":"2024-01-26T23:49:21.705419+0000","last_fullsized":"2024-01-26T23:49:21.705419+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:51.588887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720594+0000","last_change":"2024-01-26T23:49:21.717788+0000","last_active":"2024-01-26T23:49:21.720594+0000","last_peered":"2024-01-26T23:49:21.720594+0000","last_clean":"2024-01-26T23:49:21.720594+0000","last_became_active":"2024-01-26T23:49:21.717188+0000","last_became_peered":"2024-01-26T23:49:21.717188+0000","last_unstale":"2024-01-26T23:49:21.720594+0000","last_undegraded":"2024-01-26T23:49:21.720594+0000","last_fullsized":"2024-01-26T23:49:21.720594+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:52:44.481461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716190+0000","last_change":"2024-01-26T23:49:21.713083+0000","last_active":"2024-01-26T23:49:21.716190+0000","last_peered":"2024-01-26T23:49:21.716190+0000","last_clean":"2024-01-26T23:49:21.716190+0000","last_became_active":"2024-01-26T23:49:21.712770+0000","last_became_peered":"2024-01-26T23:49:21.712770+0000","last_unstale":"2024-01-26T23:49:21.716190+0000","last_undegraded":"2024-01-26T23:49:21.716190+0000","last_fullsized":"2024-01-26T23:49:21.716190+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:55:44.107048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.701246+0000","last_change":"2024-01-26T23:49:21.698975+0000","last_active":"2024-01-26T23:49:21.701246+0000","last_peered":"2024-01-26T23:49:21.701246+0000","last_clean":"2024-01-26T23:49:21.701246+0000","last_became_active":"2024-01-26T23:49:21.698670+0000","last_became_peered":"2024-01-26T23:49:21.698670+0000","last_unstale":"2024-01-26T23:49:21.701246+0000","last_undegraded":"2024-01-26T23:49:21.701246+0000","last_fullsized":"2024-01-26T23:49:21.701246+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:46:36.438311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713198+0000","last_change":"2024-01-26T23:49:21.711141+0000","last_active":"2024-01-26T23:49:21.713198+0000","last_peered":"2024-01-26T23:49:21.713198+0000","last_clean":"2024-01-26T23:49:21.713198+0000","last_became_active":"2024-01-26T23:49:21.710788+0000","last_became_peered":"2024-01-26T23:49:21.710788+0000","last_unstale":"2024-01-26T23:49:21.713198+0000","last_undegraded":"2024-01-26T23:49:21.713198+0000","last_fullsized":"2024-01-26T23:49:21.713198+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:32:26.662337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715295+0000","last_change":"2024-01-26T23:49:21.712785+0000","last_active":"2024-01-26T23:49:21.715295+0000","last_peered":"2024-01-26T23:49:21.715295+0000","last_clean":"2024-01-26T23:49:21.715295+0000","last_became_active":"2024-01-26T23:49:21.711603+0000","last_became_peered":"2024-01-26T23:49:21.711603+0000","last_unstale":"2024-01-26T23:49:21.715295+0000","last_undegraded":"2024-01-26T23:49:21.715295+0000","last_fullsized":"2024-01-26T23:49:21.715295+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:42:38.767042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.710670+0000","last_change":"2024-01-26T23:49:21.704936+0000","last_active":"2024-01-26T23:49:21.710670+0000","last_peered":"2024-01-26T23:49:21.710670+0000","last_clean":"2024-01-26T23:49:21.710670+0000","last_became_active":"2024-01-26T23:49:21.704639+0000","last_became_peered":"2024-01-26T23:49:21.704639+0000","last_unstale":"2024-01-26T23:49:21.710670+0000","last_undegraded":"2024-01-26T23:49:21.710670+0000","last_fullsized":"2024-01-26T23:49:21.710670+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:26:52.470020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.707219+0000","last_change":"2024-01-26T23:49:21.702319+0000","last_active":"2024-01-26T23:49:21.707219+0000","last_peered":"2024-01-26T23:49:21.707219+0000","last_clean":"2024-01-26T23:49:21.707219+0000","last_became_active":"2024-01-26T23:49:21.701995+0000","last_became_peered":"2024-01-26T23:49:21.701995+0000","last_unstale":"2024-01-26T23:49:21.707219+0000","last_undegraded":"2024-01-26T23:49:21.707219+0000","last_fullsized":"2024-01-26T23:49:21.707219+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:44:38.742019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713556+0000","last_change":"2024-01-26T23:49:21.706418+0000","last_active":"2024-01-26T23:49:21.713556+0000","last_peered":"2024-01-26T23:49:21.713556+0000","last_clean":"2024-01-26T23:49:21.713556+0000","last_became_active":"2024-01-26T23:49:21.705710+0000","last_became_peered":"2024-01-26T23:49:21.705710+0000","last_unstale":"2024-01-26T23:49:21.713556+0000","last_undegraded":"2024-01-26T23:49:21.713556+0000","last_fullsized":"2024-01-26T23:49:21.713556+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:07:15.034585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715571+0000","last_change":"2024-01-26T23:49:21.708843+0000","last_active":"2024-01-26T23:49:21.715571+0000","last_peered":"2024-01-26T23:49:21.715571+0000","last_clean":"2024-01-26T23:49:21.715571+0000","last_became_active":"2024-01-26T23:49:21.708590+0000","last_became_peered":"2024-01-26T23:49:21.708590+0000","last_unstale":"2024-01-26T23:49:21.715571+0000","last_undegraded":"2024-01-26T23:49:21.715571+0000","last_fullsized":"2024-01-26T23:49:21.715571+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:47:59.288036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.712492+0000","last_change":"2024-01-26T23:49:21.709022+0000","last_active":"2024-01-26T23:49:21.712492+0000","last_peered":"2024-01-26T23:49:21.712492+0000","last_clean":"2024-01-26T23:49:21.712492+0000","last_became_active":"2024-01-26T23:49:21.708716+0000","last_became_peered":"2024-01-26T23:49:21.708716+0000","last_unstale":"2024-01-26T23:49:21.712492+0000","last_undegraded":"2024-01-26T23:49:21.712492+0000","last_fullsized":"2024-01-26T23:49:21.712492+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:18:32.453554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.717173+0000","last_change":"2024-01-26T23:49:21.712966+0000","last_active":"2024-01-26T23:49:21.717173+0000","last_peered":"2024-01-26T23:49:21.717173+0000","last_clean":"2024-01-26T23:49:21.717173+0000","last_became_active":"2024-01-26T23:49:21.712337+0000","last_became_peered":"2024-01-26T23:49:21.712337+0000","last_unstale":"2024-01-26T23:49:21.717173+0000","last_undegraded":"2024-01-26T23:49:21.717173+0000","last_fullsized":"2024-01-26T23:49:21.717173+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:36:49.497687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714994+0000","last_change":"2024-01-26T23:49:21.712389+0000","last_active":"2024-01-26T23:49:21.714994+0000","last_peered":"2024-01-26T23:49:21.714994+0000","last_clean":"2024-01-26T23:49:21.714994+0000","last_became_active":"2024-01-26T23:49:21.712138+0000","last_became_peered":"2024-01-26T23:49:21.712138+0000","last_unstale":"2024-01-26T23:49:21.714994+0000","last_undegraded":"2024-01-26T23:49:21.714994+0000","last_fullsized":"2024-01-26T23:49:21.714994+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:36:35.342280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.717216+0000","last_change":"2024-01-26T23:49:21.713016+0000","last_active":"2024-01-26T23:49:21.717216+0000","last_peered":"2024-01-26T23:49:21.717216+0000","last_clean":"2024-01-26T23:49:21.717216+0000","last_became_active":"2024-01-26T23:49:21.712716+0000","last_became_peered":"2024-01-26T23:49:21.712716+0000","last_unstale":"2024-01-26T23:49:21.717216+0000","last_undegraded":"2024-01-26T23:49:21.717216+0000","last_fullsized":"2024-01-26T23:49:21.717216+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:19:56.848512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716319+0000","last_change":"2024-01-26T23:49:21.712460+0000","last_active":"2024-01-26T23:49:21.716319+0000","last_peered":"2024-01-26T23:49:21.716319+0000","last_clean":"2024-01-26T23:49:21.716319+0000","last_became_active":"2024-01-26T23:49:21.712123+0000","last_became_peered":"2024-01-26T23:49:21.712123+0000","last_unstale":"2024-01-26T23:49:21.716319+0000","last_undegraded":"2024-01-26T23:49:21.716319+0000","last_fullsized":"2024-01-26T23:49:21.716319+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:57:09.862596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716309+0000","last_change":"2024-01-26T23:49:21.712418+0000","last_active":"2024-01-26T23:49:21.716309+0000","last_peered":"2024-01-26T23:49:21.716309+0000","last_clean":"2024-01-26T23:49:21.716309+0000","last_became_active":"2024-01-26T23:49:21.712099+0000","last_became_peered":"2024-01-26T23:49:21.712099+0000","last_unstale":"2024-01-26T23:49:21.716309+0000","last_undegraded":"2024-01-26T23:49:21.716309+0000","last_fullsized":"2024-01-26T23:49:21.716309+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:17.484964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.708922+0000","last_change":"2024-01-26T23:49:21.705150+0000","last_active":"2024-01-26T23:49:21.708922+0000","last_peered":"2024-01-26T23:49:21.708922+0000","last_clean":"2024-01-26T23:49:21.708922+0000","last_became_active":"2024-01-26T23:49:21.703248+0000","last_became_peered":"2024-01-26T23:49:21.703248+0000","last_unstale":"2024-01-26T23:49:21.708922+0000","last_undegraded":"2024-01-26T23:49:21.708922+0000","last_fullsized":"2024-01-26T23:49:21.708922+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:49:45.239681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713631+0000","last_change":"2024-01-26T23:49:21.708382+0000","last_active":"2024-01-26T23:49:21.713631+0000","last_peered":"2024-01-26T23:49:21.713631+0000","last_clean":"2024-01-26T23:49:21.713631+0000","last_became_active":"2024-01-26T23:49:21.708082+0000","last_became_peered":"2024-01-26T23:49:21.708082+0000","last_unstale":"2024-01-26T23:49:21.713631+0000","last_undegraded":"2024-01-26T23:49:21.713631+0000","last_fullsized":"2024-01-26T23:49:21.713631+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:44:38.742019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716169+0000","last_change":"2024-01-26T23:49:21.711944+0000","last_active":"2024-01-26T23:49:21.716169+0000","last_peered":"2024-01-26T23:49:21.716169+0000","last_clean":"2024-01-26T23:49:21.716169+0000","last_became_active":"2024-01-26T23:49:21.711661+0000","last_became_peered":"2024-01-26T23:49:21.711661+0000","last_unstale":"2024-01-26T23:49:21.716169+0000","last_undegraded":"2024-01-26T23:49:21.716169+0000","last_fullsized":"2024-01-26T23:49:21.716169+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:07.408591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714134+0000","last_change":"2024-01-26T23:49:21.711617+0000","last_active":"2024-01-26T23:49:21.714134+0000","last_peered":"2024-01-26T23:49:21.714134+0000","last_clean":"2024-01-26T23:49:21.714134+0000","last_became_active":"2024-01-26T23:49:21.711167+0000","last_became_peered":"2024-01-26T23:49:21.711167+0000","last_unstale":"2024-01-26T23:49:21.714134+0000","last_undegraded":"2024-01-26T23:49:21.714134+0000","last_fullsized":"2024-01-26T23:49:21.714134+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:07:55.842419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715903+0000","last_change":"2024-01-26T23:49:21.713998+0000","last_active":"2024-01-26T23:49:21.715903+0000","last_peered":"2024-01-26T23:49:21.715903+0000","last_clean":"2024-01-26T23:49:21.715903+0000","last_became_active":"2024-01-26T23:49:21.713699+0000","last_became_peered":"2024-01-26T23:49:21.713699+0000","last_unstale":"2024-01-26T23:49:21.715903+0000","last_undegraded":"2024-01-26T23:49:21.715903+0000","last_fullsized":"2024-01-26T23:49:21.715903+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:31:18.118068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.710887+0000","last_change":"2024-01-26T23:49:21.701621+0000","last_active":"2024-01-26T23:49:21.710887+0000","last_peered":"2024-01-26T23:49:21.710887+0000","last_clean":"2024-01-26T23:49:21.710887+0000","last_became_active":"2024-01-26T23:49:21.701321+0000","last_became_peered":"2024-01-26T23:49:21.701321+0000","last_unstale":"2024-01-26T23:49:21.710887+0000","last_undegraded":"2024-01-26T23:49:21.710887+0000","last_fullsized":"2024-01-26T23:49:21.710887+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:55:23.167302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716353+0000","last_change":"2024-01-26T23:49:21.712792+0000","last_active":"2024-01-26T23:49:21.716353+0000","last_peered":"2024-01-26T23:49:21.716353+0000","last_clean":"2024-01-26T23:49:21.716353+0000","last_became_active":"2024-01-26T23:49:21.711757+0000","last_became_peered":"2024-01-26T23:49:21.711757+0000","last_unstale":"2024-01-26T23:49:21.716353+0000","last_undegraded":"2024-01-26T23:49:21.716353+0000","last_fullsized":"2024-01-26T23:49:21.716353+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:03:11.827036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.708503+0000","last_change":"2024-01-26T23:49:21.706929+0000","last_active":"2024-01-26T23:49:21.708503+0000","last_peered":"2024-01-26T23:49:21.708503+0000","last_clean":"2024-01-26T23:49:21.708503+0000","last_became_active":"2024-01-26T23:49:21.706543+0000","last_became_peered":"2024-01-26T23:49:21.706543+0000","last_unstale":"2024-01-26T23:49:21.708503+0000","last_undegraded":"2024-01-26T23:49:21.708503+0000","last_fullsized":"2024-01-26T23:49:21.708503+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:34.441797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"144'4","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.947803+0000","last_change":"2024-01-26T23:49:20.933655+0000","last_active":"2024-01-26T23:49:21.947803+0000","last_peered":"2024-01-26T23:49:21.947803+0000","last_clean":"2024-01-26T23:49:21.947803+0000","last_became_active":"2024-01-26T23:49:20.933360+0000","last_became_peered":"2024-01-26T23:49:20.933360+0000","last_unstale":"2024-01-26T23:49:21.947803+0000","last_undegraded":"2024-01-26T23:49:21.947803+0000","last_fullsized":"2024-01-26T23:49:21.947803+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:23:00.411032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716434+0000","last_change":"2024-01-26T23:49:21.712338+0000","last_active":"2024-01-26T23:49:21.716434+0000","last_peered":"2024-01-26T23:49:21.716434+0000","last_clean":"2024-01-26T23:49:21.716434+0000","last_became_active":"2024-01-26T23:49:21.712030+0000","last_became_peered":"2024-01-26T23:49:21.712030+0000","last_unstale":"2024-01-26T23:49:21.716434+0000","last_undegraded":"2024-01-26T23:49:21.716434+0000","last_fullsized":"2024-01-26T23:49:21.716434+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:25:47.504184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.947564+0000","last_change":"2024-01-26T23:49:20.934413+0000","last_active":"2024-01-26T23:49:21.947564+0000","last_peered":"2024-01-26T23:49:21.947564+0000","last_clean":"2024-01-26T23:49:21.947564+0000","last_became_active":"2024-01-26T23:49:20.933769+0000","last_became_peered":"2024-01-26T23:49:20.933769+0000","last_unstale":"2024-01-26T23:49:21.947564+0000","last_undegraded":"2024-01-26T23:49:21.947564+0000","last_fullsized":"2024-01-26T23:49:21.947564+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:05:23.745399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716376+0000","last_change":"2024-01-26T23:49:21.712916+0000","last_active":"2024-01-26T23:49:21.716376+0000","last_peered":"2024-01-26T23:49:21.716376+0000","last_clean":"2024-01-26T23:49:21.716376+0000","last_became_active":"2024-01-26T23:49:21.712259+0000","last_became_peered":"2024-01-26T23:49:21.712259+0000","last_unstale":"2024-01-26T23:49:21.716376+0000","last_undegraded":"2024-01-26T23:49:21.716376+0000","last_fullsized":"2024-01-26T23:49:21.716376+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:05:24.747228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.947622+0000","last_change":"2024-01-26T23:49:20.933682+0000","last_active":"2024-01-26T23:49:21.947622+0000","last_peered":"2024-01-26T23:49:21.947622+0000","last_clean":"2024-01-26T23:49:21.947622+0000","last_became_active":"2024-01-26T23:49:20.933419+0000","last_became_peered":"2024-01-26T23:49:20.933419+0000","last_unstale":"2024-01-26T23:49:21.947622+0000","last_undegraded":"2024-01-26T23:49:21.947622+0000","last_fullsized":"2024-01-26T23:49:21.947622+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:31:23.721288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714845+0000","last_change":"2024-01-26T23:49:21.709566+0000","last_active":"2024-01-26T23:49:21.714845+0000","last_peered":"2024-01-26T23:49:21.714845+0000","last_clean":"2024-01-26T23:49:21.714845+0000","last_became_active":"2024-01-26T23:49:21.709308+0000","last_became_peered":"2024-01-26T23:49:21.709308+0000","last_unstale":"2024-01-26T23:49:21.714845+0000","last_undegraded":"2024-01-26T23:49:21.714845+0000","last_fullsized":"2024-01-26T23:49:21.714845+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:01:52.607618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.951313+0000","last_change":"2024-01-26T23:49:20.941242+0000","last_active":"2024-01-26T23:49:21.951313+0000","last_peered":"2024-01-26T23:49:21.951313+0000","last_clean":"2024-01-26T23:49:21.951313+0000","last_became_active":"2024-01-26T23:49:20.940914+0000","last_became_peered":"2024-01-26T23:49:20.940914+0000","last_unstale":"2024-01-26T23:49:21.951313+0000","last_undegraded":"2024-01-26T23:49:21.951313+0000","last_fullsized":"2024-01-26T23:49:21.951313+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:52:55.167098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715579+0000","last_change":"2024-01-26T23:49:21.713588+0000","last_active":"2024-01-26T23:49:21.715579+0000","last_peered":"2024-01-26T23:49:21.715579+0000","last_clean":"2024-01-26T23:49:21.715579+0000","last_became_active":"2024-01-26T23:49:21.713242+0000","last_became_peered":"2024-01-26T23:49:21.713242+0000","last_unstale":"2024-01-26T23:49:21.715579+0000","last_undegraded":"2024-01-26T23:49:21.715579+0000","last_fullsized":"2024-01-26T23:49:21.715579+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:27:10.909797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.950236+0000","last_change":"2024-01-26T23:49:20.935760+0000","last_active":"2024-01-26T23:49:21.950236+0000","last_peered":"2024-01-26T23:49:21.950236+0000","last_clean":"2024-01-26T23:49:21.950236+0000","last_became_active":"2024-01-26T23:49:20.935522+0000","last_became_peered":"2024-01-26T23:49:20.935522+0000","last_unstale":"2024-01-26T23:49:21.950236+0000","last_undegraded":"2024-01-26T23:49:21.950236+0000","last_fullsized":"2024-01-26T23:49:21.950236+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:46:35.436482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715276+0000","last_change":"2024-01-26T23:49:21.711623+0000","last_active":"2024-01-26T23:49:21.715276+0000","last_peered":"2024-01-26T23:49:21.715276+0000","last_clean":"2024-01-26T23:49:21.715276+0000","last_became_active":"2024-01-26T23:49:21.711237+0000","last_became_peered":"2024-01-26T23:49:21.711237+0000","last_unstale":"2024-01-26T23:49:21.715276+0000","last_undegraded":"2024-01-26T23:49:21.715276+0000","last_fullsized":"2024-01-26T23:49:21.715276+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:14:30.475748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.950458+0000","last_change":"2024-01-26T23:49:20.937373+0000","last_active":"2024-01-26T23:49:21.950458+0000","last_peered":"2024-01-26T23:49:21.950458+0000","last_clean":"2024-01-26T23:49:21.950458+0000","last_became_active":"2024-01-26T23:49:20.937065+0000","last_became_peered":"2024-01-26T23:49:20.937065+0000","last_unstale":"2024-01-26T23:49:21.950458+0000","last_undegraded":"2024-01-26T23:49:21.950458+0000","last_fullsized":"2024-01-26T23:49:21.950458+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:49:16.191682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.702584+0000","last_change":"2024-01-26T23:49:21.699753+0000","last_active":"2024-01-26T23:49:21.702584+0000","last_peered":"2024-01-26T23:49:21.702584+0000","last_clean":"2024-01-26T23:49:21.702584+0000","last_became_active":"2024-01-26T23:49:21.699464+0000","last_became_peered":"2024-01-26T23:49:21.699464+0000","last_unstale":"2024-01-26T23:49:21.702584+0000","last_undegraded":"2024-01-26T23:49:21.702584+0000","last_fullsized":"2024-01-26T23:49:21.702584+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:03.507801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.695979+0000","last_change":"2024-01-26T23:49:20.928944+0000","last_active":"2024-01-26T23:49:21.695979+0000","last_peered":"2024-01-26T23:49:21.695979+0000","last_clean":"2024-01-26T23:49:21.695979+0000","last_became_active":"2024-01-26T23:49:20.928649+0000","last_became_peered":"2024-01-26T23:49:20.928649+0000","last_unstale":"2024-01-26T23:49:21.695979+0000","last_undegraded":"2024-01-26T23:49:21.695979+0000","last_fullsized":"2024-01-26T23:49:21.695979+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:55:50.467126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.702548+0000","last_change":"2024-01-26T23:49:21.699714+0000","last_active":"2024-01-26T23:49:21.702548+0000","last_peered":"2024-01-26T23:49:21.702548+0000","last_clean":"2024-01-26T23:49:21.702548+0000","last_became_active":"2024-01-26T23:49:21.699419+0000","last_became_peered":"2024-01-26T23:49:21.699419+0000","last_unstale":"2024-01-26T23:49:21.702548+0000","last_undegraded":"2024-01-26T23:49:21.702548+0000","last_fullsized":"2024-01-26T23:49:21.702548+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:47:59.288036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.696916+0000","last_change":"2024-01-26T23:49:20.927169+0000","last_active":"2024-01-26T23:49:21.696916+0000","last_peered":"2024-01-26T23:49:21.696916+0000","last_clean":"2024-01-26T23:49:21.696916+0000","last_became_active":"2024-01-26T23:49:20.926865+0000","last_became_peered":"2024-01-26T23:49:20.926865+0000","last_unstale":"2024-01-26T23:49:21.696916+0000","last_undegraded":"2024-01-26T23:49:21.696916+0000","last_fullsized":"2024-01-26T23:49:21.696916+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:57:52.726572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715082+0000","last_change":"2024-01-26T23:49:21.708432+0000","last_active":"2024-01-26T23:49:21.715082+0000","last_peered":"2024-01-26T23:49:21.715082+0000","last_clean":"2024-01-26T23:49:21.715082+0000","last_became_active":"2024-01-26T23:49:21.708100+0000","last_became_peered":"2024-01-26T23:49:21.708100+0000","last_unstale":"2024-01-26T23:49:21.715082+0000","last_undegraded":"2024-01-26T23:49:21.715082+0000","last_fullsized":"2024-01-26T23:49:21.715082+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:31:18.118068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.707105+0000","last_change":"2024-01-26T23:49:20.928442+0000","last_active":"2024-01-26T23:49:21.707105+0000","last_peered":"2024-01-26T23:49:21.707105+0000","last_clean":"2024-01-26T23:49:21.707105+0000","last_became_active":"2024-01-26T23:49:20.928198+0000","last_became_peered":"2024-01-26T23:49:20.928198+0000","last_unstale":"2024-01-26T23:49:21.707105+0000","last_undegraded":"2024-01-26T23:49:21.707105+0000","last_fullsized":"2024-01-26T23:49:21.707105+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:33.439967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.710927+0000","last_change":"2024-01-26T23:49:21.702605+0000","last_active":"2024-01-26T23:49:21.710927+0000","last_peered":"2024-01-26T23:49:21.710927+0000","last_clean":"2024-01-26T23:49:21.710927+0000","last_became_active":"2024-01-26T23:49:21.702345+0000","last_became_peered":"2024-01-26T23:49:21.702345+0000","last_unstale":"2024-01-26T23:49:21.710927+0000","last_undegraded":"2024-01-26T23:49:21.710927+0000","last_fullsized":"2024-01-26T23:49:21.710927+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:00:37.977251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.696186+0000","last_change":"2024-01-26T23:49:20.933290+0000","last_active":"2024-01-26T23:49:21.696186+0000","last_peered":"2024-01-26T23:49:21.696186+0000","last_clean":"2024-01-26T23:49:21.696186+0000","last_became_active":"2024-01-26T23:49:20.932911+0000","last_became_peered":"2024-01-26T23:49:20.932911+0000","last_unstale":"2024-01-26T23:49:21.696186+0000","last_undegraded":"2024-01-26T23:49:21.696186+0000","last_fullsized":"2024-01-26T23:49:21.696186+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:06:47.740099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713595+0000","last_change":"2024-01-26T23:49:21.710063+0000","last_active":"2024-01-26T23:49:21.713595+0000","last_peered":"2024-01-26T23:49:21.713595+0000","last_clean":"2024-01-26T23:49:21.713595+0000","last_became_active":"2024-01-26T23:49:21.709757+0000","last_became_peered":"2024-01-26T23:49:21.709757+0000","last_unstale":"2024-01-26T23:49:21.713595+0000","last_undegraded":"2024-01-26T23:49:21.713595+0000","last_fullsized":"2024-01-26T23:49:21.713595+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:47:59.288036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"144'3","reported_seq":13,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.950273+0000","last_change":"2024-01-26T23:49:20.926313+0000","last_active":"2024-01-26T23:49:21.950273+0000","last_peered":"2024-01-26T23:49:21.950273+0000","last_clean":"2024-01-26T23:49:21.950273+0000","last_became_active":"2024-01-26T23:49:20.926067+0000","last_became_peered":"2024-01-26T23:49:20.926067+0000","last_unstale":"2024-01-26T23:49:21.950273+0000","last_undegraded":"2024-01-26T23:49:21.950273+0000","last_fullsized":"2024-01-26T23:49:21.950273+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:09:02.526081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714584+0000","last_change":"2024-01-26T23:49:21.709992+0000","last_active":"2024-01-26T23:49:21.714584+0000","last_peered":"2024-01-26T23:49:21.714584+0000","last_clean":"2024-01-26T23:49:21.714584+0000","last_became_active":"2024-01-26T23:49:21.709742+0000","last_became_peered":"2024-01-26T23:49:21.709742+0000","last_unstale":"2024-01-26T23:49:21.714584+0000","last_undegraded":"2024-01-26T23:49:21.714584+0000","last_fullsized":"2024-01-26T23:49:21.714584+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:39:24.864123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"144'6","reported_seq":16,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.955758+0000","last_change":"2024-01-26T23:49:20.932889+0000","last_active":"2024-01-26T23:49:21.955758+0000","last_peered":"2024-01-26T23:49:21.955758+0000","last_clean":"2024-01-26T23:49:21.955758+0000","last_became_active":"2024-01-26T23:49:20.932592+0000","last_became_peered":"2024-01-26T23:49:20.932592+0000","last_unstale":"2024-01-26T23:49:21.955758+0000","last_undegraded":"2024-01-26T23:49:21.955758+0000","last_fullsized":"2024-01-26T23:49:21.955758+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:16.483135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.718948+0000","last_change":"2024-01-26T23:49:21.714417+0000","last_active":"2024-01-26T23:49:21.718948+0000","last_peered":"2024-01-26T23:49:21.718948+0000","last_clean":"2024-01-26T23:49:21.718948+0000","last_became_active":"2024-01-26T23:49:21.714128+0000","last_became_peered":"2024-01-26T23:49:21.714128+0000","last_unstale":"2024-01-26T23:49:21.718948+0000","last_undegraded":"2024-01-26T23:49:21.718948+0000","last_fullsized":"2024-01-26T23:49:21.718948+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:46:36.438311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.952459+0000","last_change":"2024-01-26T23:49:20.934520+0000","last_active":"2024-01-26T23:49:21.952459+0000","last_peered":"2024-01-26T23:49:21.952459+0000","last_clean":"2024-01-26T23:49:21.952459+0000","last_became_active":"2024-01-26T23:49:20.933872+0000","last_became_peered":"2024-01-26T23:49:20.933872+0000","last_unstale":"2024-01-26T23:49:21.952459+0000","last_undegraded":"2024-01-26T23:49:21.952459+0000","last_fullsized":"2024-01-26T23:49:21.952459+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:39:31.988243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.708179+0000","last_change":"2024-01-26T23:49:21.705738+0000","last_active":"2024-01-26T23:49:21.708179+0000","last_peered":"2024-01-26T23:49:21.708179+0000","last_clean":"2024-01-26T23:49:21.708179+0000","last_became_active":"2024-01-26T23:49:21.705447+0000","last_became_peered":"2024-01-26T23:49:21.705447+0000","last_unstale":"2024-01-26T23:49:21.708179+0000","last_undegraded":"2024-01-26T23:49:21.708179+0000","last_fullsized":"2024-01-26T23:49:21.708179+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:43:20.056916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.697347+0000","last_change":"2024-01-26T23:49:20.925959+0000","last_active":"2024-01-26T23:49:21.697347+0000","last_peered":"2024-01-26T23:49:21.697347+0000","last_clean":"2024-01-26T23:49:21.697347+0000","last_became_active":"2024-01-26T23:49:20.925600+0000","last_became_peered":"2024-01-26T23:49:20.925600+0000","last_unstale":"2024-01-26T23:49:21.697347+0000","last_undegraded":"2024-01-26T23:49:21.697347+0000","last_fullsized":"2024-01-26T23:49:21.697347+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:55:22.165473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714932+0000","last_change":"2024-01-26T23:49:21.713422+0000","last_active":"2024-01-26T23:49:21.714932+0000","last_peered":"2024-01-26T23:49:21.714932+0000","last_clean":"2024-01-26T23:49:21.714932+0000","last_became_active":"2024-01-26T23:49:21.713129+0000","last_became_peered":"2024-01-26T23:49:21.713129+0000","last_unstale":"2024-01-26T23:49:21.714932+0000","last_undegraded":"2024-01-26T23:49:21.714932+0000","last_fullsized":"2024-01-26T23:49:21.714932+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:36:43.200780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.696779+0000","last_change":"2024-01-26T23:49:20.926498+0000","last_active":"2024-01-26T23:49:21.696779+0000","last_peered":"2024-01-26T23:49:21.696779+0000","last_clean":"2024-01-26T23:49:21.696779+0000","last_became_active":"2024-01-26T23:49:20.926183+0000","last_became_peered":"2024-01-26T23:49:20.926183+0000","last_unstale":"2024-01-26T23:49:21.696779+0000","last_undegraded":"2024-01-26T23:49:21.696779+0000","last_fullsized":"2024-01-26T23:49:21.696779+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:00:36.975421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714445+0000","last_change":"2024-01-26T23:49:21.708959+0000","last_active":"2024-01-26T23:49:21.714445+0000","last_peered":"2024-01-26T23:49:21.714445+0000","last_clean":"2024-01-26T23:49:21.714445+0000","last_became_active":"2024-01-26T23:49:21.708693+0000","last_became_peered":"2024-01-26T23:49:21.708693+0000","last_unstale":"2024-01-26T23:49:21.714445+0000","last_undegraded":"2024-01-26T23:49:21.714445+0000","last_fullsized":"2024-01-26T23:49:21.714445+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:39:32.990073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.948408+0000","last_change":"2024-01-26T23:49:20.934522+0000","last_active":"2024-01-26T23:49:21.948408+0000","last_peered":"2024-01-26T23:49:21.948408+0000","last_clean":"2024-01-26T23:49:21.948408+0000","last_became_active":"2024-01-26T23:49:20.934219+0000","last_became_peered":"2024-01-26T23:49:20.934219+0000","last_unstale":"2024-01-26T23:49:21.948408+0000","last_undegraded":"2024-01-26T23:49:21.948408+0000","last_fullsized":"2024-01-26T23:49:21.948408+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:31:23.721288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720614+0000","last_change":"2024-01-26T23:49:21.718008+0000","last_active":"2024-01-26T23:49:21.720614+0000","last_peered":"2024-01-26T23:49:21.720614+0000","last_clean":"2024-01-26T23:49:21.720614+0000","last_became_active":"2024-01-26T23:49:21.717522+0000","last_became_peered":"2024-01-26T23:49:21.717522+0000","last_unstale":"2024-01-26T23:49:21.720614+0000","last_undegraded":"2024-01-26T23:49:21.720614+0000","last_fullsized":"2024-01-26T23:49:21.720614+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:25:47.543272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.948077+0000","last_change":"2024-01-26T23:49:20.938034+0000","last_active":"2024-01-26T23:49:21.948077+0000","last_peered":"2024-01-26T23:49:21.948077+0000","last_clean":"2024-01-26T23:49:21.948077+0000","last_became_active":"2024-01-26T23:49:20.937700+0000","last_became_peered":"2024-01-26T23:49:20.937700+0000","last_unstale":"2024-01-26T23:49:21.948077+0000","last_undegraded":"2024-01-26T23:49:21.948077+0000","last_fullsized":"2024-01-26T23:49:21.948077+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:30:23.346922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715156+0000","last_change":"2024-01-26T23:49:21.711030+0000","last_active":"2024-01-26T23:49:21.715156+0000","last_peered":"2024-01-26T23:49:21.715156+0000","last_clean":"2024-01-26T23:49:21.715156+0000","last_became_active":"2024-01-26T23:49:21.710735+0000","last_became_peered":"2024-01-26T23:49:21.710735+0000","last_unstale":"2024-01-26T23:49:21.715156+0000","last_undegraded":"2024-01-26T23:49:21.715156+0000","last_fullsized":"2024-01-26T23:49:21.715156+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:22:26.000383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.950570+0000","last_change":"2024-01-26T23:49:20.932856+0000","last_active":"2024-01-26T23:49:21.950570+0000","last_peered":"2024-01-26T23:49:21.950570+0000","last_clean":"2024-01-26T23:49:21.950570+0000","last_became_active":"2024-01-26T23:49:20.932586+0000","last_became_peered":"2024-01-26T23:49:20.932586+0000","last_unstale":"2024-01-26T23:49:21.950570+0000","last_undegraded":"2024-01-26T23:49:21.950570+0000","last_fullsized":"2024-01-26T23:49:21.950570+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:41:32.548836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714568+0000","last_change":"2024-01-26T23:49:21.706806+0000","last_active":"2024-01-26T23:49:21.714568+0000","last_peered":"2024-01-26T23:49:21.714568+0000","last_clean":"2024-01-26T23:49:21.714568+0000","last_became_active":"2024-01-26T23:49:21.706309+0000","last_became_peered":"2024-01-26T23:49:21.706309+0000","last_unstale":"2024-01-26T23:49:21.714568+0000","last_undegraded":"2024-01-26T23:49:21.714568+0000","last_fullsized":"2024-01-26T23:49:21.714568+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:15.880089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.702136+0000","last_change":"2024-01-26T23:49:20.937835+0000","last_active":"2024-01-26T23:49:21.702136+0000","last_peered":"2024-01-26T23:49:21.702136+0000","last_clean":"2024-01-26T23:49:21.702136+0000","last_became_active":"2024-01-26T23:49:20.937594+0000","last_became_peered":"2024-01-26T23:49:20.937594+0000","last_unstale":"2024-01-26T23:49:21.702136+0000","last_undegraded":"2024-01-26T23:49:21.702136+0000","last_fullsized":"2024-01-26T23:49:21.702136+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:04:52.656671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714422+0000","last_change":"2024-01-26T23:49:21.709312+0000","last_active":"2024-01-26T23:49:21.714422+0000","last_peered":"2024-01-26T23:49:21.714422+0000","last_clean":"2024-01-26T23:49:21.714422+0000","last_became_active":"2024-01-26T23:49:21.709016+0000","last_became_peered":"2024-01-26T23:49:21.709016+0000","last_unstale":"2024-01-26T23:49:21.714422+0000","last_undegraded":"2024-01-26T23:49:21.714422+0000","last_fullsized":"2024-01-26T23:49:21.714422+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:15.415331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.947893+0000","last_change":"2024-01-26T23:49:20.930419+0000","last_active":"2024-01-26T23:49:21.947893+0000","last_peered":"2024-01-26T23:49:21.947893+0000","last_clean":"2024-01-26T23:49:21.947893+0000","last_became_active":"2024-01-26T23:49:20.930102+0000","last_became_peered":"2024-01-26T23:49:20.930102+0000","last_unstale":"2024-01-26T23:49:21.947893+0000","last_undegraded":"2024-01-26T23:49:21.947893+0000","last_fullsized":"2024-01-26T23:49:21.947893+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:01.796587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.709284+0000","last_change":"2024-01-26T23:49:21.703187+0000","last_active":"2024-01-26T23:49:21.709284+0000","last_peered":"2024-01-26T23:49:21.709284+0000","last_clean":"2024-01-26T23:49:21.709284+0000","last_became_active":"2024-01-26T23:49:21.702883+0000","last_became_peered":"2024-01-26T23:49:21.702883+0000","last_unstale":"2024-01-26T23:49:21.709284+0000","last_undegraded":"2024-01-26T23:49:21.709284+0000","last_fullsized":"2024-01-26T23:49:21.709284+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:23:30.759008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.701669+0000","last_change":"2024-01-26T23:49:20.931625+0000","last_active":"2024-01-26T23:49:21.701669+0000","last_peered":"2024-01-26T23:49:21.701669+0000","last_clean":"2024-01-26T23:49:21.701669+0000","last_became_active":"2024-01-26T23:49:20.930770+0000","last_became_peered":"2024-01-26T23:49:20.930770+0000","last_unstale":"2024-01-26T23:49:21.701669+0000","last_undegraded":"2024-01-26T23:49:21.701669+0000","last_fullsized":"2024-01-26T23:49:21.701669+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:05:23.745399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713219+0000","last_change":"2024-01-26T23:49:21.711158+0000","last_active":"2024-01-26T23:49:21.713219+0000","last_peered":"2024-01-26T23:49:21.713219+0000","last_clean":"2024-01-26T23:49:21.713219+0000","last_became_active":"2024-01-26T23:49:21.710828+0000","last_became_peered":"2024-01-26T23:49:21.710828+0000","last_unstale":"2024-01-26T23:49:21.713219+0000","last_undegraded":"2024-01-26T23:49:21.713219+0000","last_fullsized":"2024-01-26T23:49:21.713219+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:43:20.056916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.696741+0000","last_change":"2024-01-26T23:49:20.927120+0000","last_active":"2024-01-26T23:49:21.696741+0000","last_peered":"2024-01-26T23:49:21.696741+0000","last_clean":"2024-01-26T23:49:21.696741+0000","last_became_active":"2024-01-26T23:49:20.926800+0000","last_became_peered":"2024-01-26T23:49:20.926800+0000","last_unstale":"2024-01-26T23:49:21.696741+0000","last_undegraded":"2024-01-26T23:49:21.696741+0000","last_fullsized":"2024-01-26T23:49:21.696741+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:14.878260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714485+0000","last_change":"2024-01-26T23:49:21.709755+0000","last_active":"2024-01-26T23:49:21.714485+0000","last_peered":"2024-01-26T23:49:21.714485+0000","last_clean":"2024-01-26T23:49:21.714485+0000","last_became_active":"2024-01-26T23:49:21.709547+0000","last_became_peered":"2024-01-26T23:49:21.709547+0000","last_unstale":"2024-01-26T23:49:21.714485+0000","last_undegraded":"2024-01-26T23:49:21.714485+0000","last_fullsized":"2024-01-26T23:49:21.714485+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:46:51.230370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.701015+0000","last_change":"2024-01-26T23:49:20.933709+0000","last_active":"2024-01-26T23:49:21.701015+0000","last_peered":"2024-01-26T23:49:21.701015+0000","last_clean":"2024-01-26T23:49:21.701015+0000","last_became_active":"2024-01-26T23:49:20.933444+0000","last_became_peered":"2024-01-26T23:49:20.933444+0000","last_unstale":"2024-01-26T23:49:21.701015+0000","last_undegraded":"2024-01-26T23:49:21.701015+0000","last_fullsized":"2024-01-26T23:49:21.701015+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:41:45.760005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.711298+0000","last_change":"2024-01-26T23:49:21.704604+0000","last_active":"2024-01-26T23:49:21.711298+0000","last_peered":"2024-01-26T23:49:21.711298+0000","last_clean":"2024-01-26T23:49:21.711298+0000","last_became_active":"2024-01-26T23:49:21.704277+0000","last_became_peered":"2024-01-26T23:49:21.704277+0000","last_unstale":"2024-01-26T23:49:21.711298+0000","last_undegraded":"2024-01-26T23:49:21.711298+0000","last_fullsized":"2024-01-26T23:49:21.711298+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:01:33.387704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.951229+0000","last_change":"2024-01-26T23:49:20.926602+0000","last_active":"2024-01-26T23:49:21.951229+0000","last_peered":"2024-01-26T23:49:21.951229+0000","last_clean":"2024-01-26T23:49:21.951229+0000","last_became_active":"2024-01-26T23:49:20.926306+0000","last_became_peered":"2024-01-26T23:49:20.926306+0000","last_unstale":"2024-01-26T23:49:21.951229+0000","last_undegraded":"2024-01-26T23:49:21.951229+0000","last_fullsized":"2024-01-26T23:49:21.951229+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:58:14.912115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715403+0000","last_change":"2024-01-26T23:49:21.709775+0000","last_active":"2024-01-26T23:49:21.715403+0000","last_peered":"2024-01-26T23:49:21.715403+0000","last_clean":"2024-01-26T23:49:21.715403+0000","last_became_active":"2024-01-26T23:49:21.709484+0000","last_became_peered":"2024-01-26T23:49:21.709484+0000","last_unstale":"2024-01-26T23:49:21.715403+0000","last_undegraded":"2024-01-26T23:49:21.715403+0000","last_fullsized":"2024-01-26T23:49:21.715403+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:19:56.848512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.706846+0000","last_change":"2024-01-26T23:49:20.938849+0000","last_active":"2024-01-26T23:49:21.706846+0000","last_peered":"2024-01-26T23:49:21.706846+0000","last_clean":"2024-01-26T23:49:21.706846+0000","last_became_active":"2024-01-26T23:49:20.938634+0000","last_became_peered":"2024-01-26T23:49:20.938634+0000","last_unstale":"2024-01-26T23:49:21.706846+0000","last_undegraded":"2024-01-26T23:49:21.706846+0000","last_fullsized":"2024-01-26T23:49:21.706846+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:19:55.846683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720486+0000","last_change":"2024-01-26T23:49:21.715705+0000","last_active":"2024-01-26T23:49:21.720486+0000","last_peered":"2024-01-26T23:49:21.720486+0000","last_clean":"2024-01-26T23:49:21.720486+0000","last_became_active":"2024-01-26T23:49:21.715391+0000","last_became_peered":"2024-01-26T23:49:21.715391+0000","last_unstale":"2024-01-26T23:49:21.720486+0000","last_undegraded":"2024-01-26T23:49:21.720486+0000","last_fullsized":"2024-01-26T23:49:21.720486+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:46:51.230370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.706792+0000","last_change":"2024-01-26T23:49:20.931200+0000","last_active":"2024-01-26T23:49:21.706792+0000","last_peered":"2024-01-26T23:49:21.706792+0000","last_clean":"2024-01-26T23:49:21.706792+0000","last_became_active":"2024-01-26T23:49:20.930968+0000","last_became_peered":"2024-01-26T23:49:20.930968+0000","last_unstale":"2024-01-26T23:49:21.706792+0000","last_undegraded":"2024-01-26T23:49:21.706792+0000","last_fullsized":"2024-01-26T23:49:21.706792+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:01:51.605788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.711848+0000","last_change":"2024-01-26T23:49:21.705136+0000","last_active":"2024-01-26T23:49:21.711848+0000","last_peered":"2024-01-26T23:49:21.711848+0000","last_clean":"2024-01-26T23:49:21.711848+0000","last_became_active":"2024-01-26T23:49:21.704849+0000","last_became_peered":"2024-01-26T23:49:21.704849+0000","last_unstale":"2024-01-26T23:49:21.711848+0000","last_undegraded":"2024-01-26T23:49:21.711848+0000","last_fullsized":"2024-01-26T23:49:21.711848+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:54.174596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.696843+0000","last_change":"2024-01-26T23:49:20.935623+0000","last_active":"2024-01-26T23:49:21.696843+0000","last_peered":"2024-01-26T23:49:21.696843+0000","last_clean":"2024-01-26T23:49:21.696843+0000","last_became_active":"2024-01-26T23:49:20.935325+0000","last_became_peered":"2024-01-26T23:49:20.935325+0000","last_unstale":"2024-01-26T23:49:21.696843+0000","last_undegraded":"2024-01-26T23:49:21.696843+0000","last_fullsized":"2024-01-26T23:49:21.696843+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:29:03.326840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716154+0000","last_change":"2024-01-26T23:49:21.713771+0000","last_active":"2024-01-26T23:49:21.716154+0000","last_peered":"2024-01-26T23:49:21.716154+0000","last_clean":"2024-01-26T23:49:21.716154+0000","last_became_active":"2024-01-26T23:49:21.713471+0000","last_became_peered":"2024-01-26T23:49:21.713471+0000","last_unstale":"2024-01-26T23:49:21.716154+0000","last_undegraded":"2024-01-26T23:49:21.716154+0000","last_fullsized":"2024-01-26T23:49:21.716154+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:46:24.616619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.702410+0000","last_change":"2024-01-26T23:49:20.933716+0000","last_active":"2024-01-26T23:49:21.702410+0000","last_peered":"2024-01-26T23:49:21.702410+0000","last_clean":"2024-01-26T23:49:21.702410+0000","last_became_active":"2024-01-26T23:49:20.933447+0000","last_became_peered":"2024-01-26T23:49:20.933447+0000","last_unstale":"2024-01-26T23:49:21.702410+0000","last_undegraded":"2024-01-26T23:49:21.702410+0000","last_fullsized":"2024-01-26T23:49:21.702410+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:44:37.740189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.712186+0000","last_change":"2024-01-26T23:49:21.709903+0000","last_active":"2024-01-26T23:49:21.712186+0000","last_peered":"2024-01-26T23:49:21.712186+0000","last_clean":"2024-01-26T23:49:21.712186+0000","last_became_active":"2024-01-26T23:49:21.709608+0000","last_became_peered":"2024-01-26T23:49:21.709608+0000","last_unstale":"2024-01-26T23:49:21.712186+0000","last_undegraded":"2024-01-26T23:49:21.712186+0000","last_fullsized":"2024-01-26T23:49:21.712186+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:44:38.742019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.707040+0000","last_change":"2024-01-26T23:49:20.928339+0000","last_active":"2024-01-26T23:49:21.707040+0000","last_peered":"2024-01-26T23:49:21.707040+0000","last_clean":"2024-01-26T23:49:21.707040+0000","last_became_active":"2024-01-26T23:49:20.927842+0000","last_became_peered":"2024-01-26T23:49:20.927842+0000","last_unstale":"2024-01-26T23:49:21.707040+0000","last_undegraded":"2024-01-26T23:49:21.707040+0000","last_fullsized":"2024-01-26T23:49:21.707040+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:43:19.055087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713297+0000","last_change":"2024-01-26T23:49:21.709300+0000","last_active":"2024-01-26T23:49:21.713297+0000","last_peered":"2024-01-26T23:49:21.713297+0000","last_clean":"2024-01-26T23:49:21.713297+0000","last_became_active":"2024-01-26T23:49:21.709008+0000","last_became_peered":"2024-01-26T23:49:21.709008+0000","last_unstale":"2024-01-26T23:49:21.713297+0000","last_undegraded":"2024-01-26T23:49:21.713297+0000","last_fullsized":"2024-01-26T23:49:21.713297+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:34.441797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.694324+0000","last_change":"2024-01-26T23:49:20.935691+0000","last_active":"2024-01-26T23:49:21.694324+0000","last_peered":"2024-01-26T23:49:21.694324+0000","last_clean":"2024-01-26T23:49:21.694324+0000","last_became_active":"2024-01-26T23:49:20.935362+0000","last_became_peered":"2024-01-26T23:49:20.935362+0000","last_unstale":"2024-01-26T23:49:21.694324+0000","last_undegraded":"2024-01-26T23:49:21.694324+0000","last_fullsized":"2024-01-26T23:49:21.694324+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:56:22.957242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.708198+0000","last_change":"2024-01-26T23:49:21.705845+0000","last_active":"2024-01-26T23:49:21.708198+0000","last_peered":"2024-01-26T23:49:21.708198+0000","last_clean":"2024-01-26T23:49:21.708198+0000","last_became_active":"2024-01-26T23:49:21.705559+0000","last_became_peered":"2024-01-26T23:49:21.705559+0000","last_unstale":"2024-01-26T23:49:21.708198+0000","last_undegraded":"2024-01-26T23:49:21.708198+0000","last_fullsized":"2024-01-26T23:49:21.708198+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:00:34.207070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.701020+0000","last_change":"2024-01-26T23:49:20.931628+0000","last_active":"2024-01-26T23:49:21.701020+0000","last_peered":"2024-01-26T23:49:21.701020+0000","last_clean":"2024-01-26T23:49:21.701020+0000","last_became_active":"2024-01-26T23:49:20.931022+0000","last_became_peered":"2024-01-26T23:49:20.931022+0000","last_unstale":"2024-01-26T23:49:21.701020+0000","last_undegraded":"2024-01-26T23:49:21.701020+0000","last_fullsized":"2024-01-26T23:49:21.701020+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:28:33.436885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.717191+0000","last_change":"2024-01-26T23:49:21.713146+0000","last_active":"2024-01-26T23:49:21.717191+0000","last_peered":"2024-01-26T23:49:21.717191+0000","last_clean":"2024-01-26T23:49:21.717191+0000","last_became_active":"2024-01-26T23:49:21.712684+0000","last_became_peered":"2024-01-26T23:49:21.712684+0000","last_unstale":"2024-01-26T23:49:21.717191+0000","last_undegraded":"2024-01-26T23:49:21.717191+0000","last_fullsized":"2024-01-26T23:49:21.717191+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:51.588887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"144'8","reported_seq":18,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.950934+0000","last_change":"2024-01-26T23:49:20.933578+0000","last_active":"2024-01-26T23:49:21.950934+0000","last_peered":"2024-01-26T23:49:21.950934+0000","last_clean":"2024-01-26T23:49:21.950934+0000","last_became_active":"2024-01-26T23:49:20.933286+0000","last_became_peered":"2024-01-26T23:49:20.933286+0000","last_unstale":"2024-01-26T23:49:21.950934+0000","last_undegraded":"2024-01-26T23:49:21.950934+0000","last_fullsized":"2024-01-26T23:49:21.950934+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:52:55.167098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"135'8584","reported_seq":9106,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.701211+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:49:21.701211+0000","last_peered":"2024-01-26T23:49:21.701211+0000","last_clean":"2024-01-26T23:49:21.701211+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:49:21.701211+0000","last_undegraded":"2024-01-26T23:49:21.701211+0000","last_fullsized":"2024-01-26T23:49:21.701211+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":2314,"num_write":360,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":2314,"num_write":360,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8584,"ondisk_log_size":8584,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575083,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311932,"kb_used_data":2356,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059908,"statfs":{"total":96636764160,"available":96317345792,"internally_reserved":0,"allocated":2412544,"data_stored":1521090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51500000000000001,"15min":0.56200000000000006},"min":{"1min":0.40699999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.74199999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52700000000000002,"15min":0.57599999999999996},"min":{"1min":0.35899999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67200000000000004,"5min":0.82499999999999996,"15min":0.83999999999999997},"last":0.57799999999999996}]},{"osd":1,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54000000000000004,"15min":0.57399999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.77600000000000002,"5min":0.77800000000000002,"15min":0.82899999999999996},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55100000000000005,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.70399999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.71299999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.57199999999999995},"min":{"1min":0.38700000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69199999999999995,"5min":0.80000000000000004,"15min":0.81699999999999995},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.38900000000000001,"5min":0.30199999999999999,"15min":0.27400000000000002},"max":{"1min":0.746,"5min":0.81999999999999995,"15min":0.88600000000000001},"last":0.502}]},{"osd":3,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54700000000000004,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.377,"15min":0.377},"max":{"1min":0.70399999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57499999999999996,"15min":0.59699999999999998},"min":{"1min":0.34399999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77900000000000003,"5min":0.86199999999999999,"15min":0.86399999999999999},"last":0.69299999999999995}]},{"osd":4,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.438,"15min":0.45200000000000001},"min":{"1min":0.34599999999999997,"5min":0.252,"15min":0.247},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.93899999999999995},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.45300000000000001,"15min":0.47399999999999998},"min":{"1min":0.32400000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67300000000000004,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.47299999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47599999999999998,"15min":0.48699999999999999},"min":{"1min":0.30499999999999999,"5min":0.27200000000000002,"15min":0.22800000000000001},"max":{"1min":0.749,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.504,"15min":0.52300000000000002},"min":{"1min":0.35799999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73399999999999999,"5min":0.74199999999999999,"15min":0.90200000000000002},"last":0.67000000000000004}]},{"osd":6,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53100000000000003,"15min":0.54800000000000004},"min":{"1min":0.41699999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76400000000000001,"5min":0.78700000000000003,"15min":0.95899999999999996},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.52700000000000002,"15min":0.54000000000000004},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75900000000000001,"5min":0.79500000000000004,"15min":0.85199999999999998},"last":0.621}]}]},{"osd":6,"up_from":13,"seq":55834575083,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303148,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068692,"statfs":{"total":96636764160,"available":96326340608,"internally_reserved":0,"allocated":1806336,"data_stored":930528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52900000000000003,"15min":0.56499999999999995},"min":{"1min":0.40300000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.423,"15min":1.423},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.57899999999999996},"min":{"1min":0.36099999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.91100000000000003},"last":0.71999999999999997}]},{"osd":1,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51700000000000002,"15min":0.55800000000000005},"min":{"1min":0.32200000000000001,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.58399999999999996},"min":{"1min":0.38200000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.66800000000000004,"5min":1.498,"15min":1.498},"last":0.68100000000000005}]},{"osd":2,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52800000000000002,"15min":0.56999999999999995},"min":{"1min":0.38100000000000001,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.85799999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52600000000000002,"15min":0.57099999999999995},"min":{"1min":0.377,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69299999999999995,"5min":0.77000000000000002,"15min":0.82799999999999996},"last":0.69999999999999996}]},{"osd":3,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57199999999999995,"15min":0.59099999999999997},"min":{"1min":0.34699999999999998,"5min":0.33800000000000002,"15min":0.29999999999999999},"max":{"1min":0.746,"5min":1.595,"15min":1.595},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56899999999999995,"15min":0.60599999999999998},"min":{"1min":0.45600000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.751,"5min":1.369,"15min":1.369},"last":0.63700000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.443,"15min":0.46200000000000002},"min":{"1min":0.33800000000000002,"5min":0.245,"15min":0.22},"max":{"1min":0.67000000000000004,"5min":1.556,"15min":1.556},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.47399999999999998,"15min":0.48199999999999998},"min":{"1min":0.33200000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.86199999999999999},"last":0.40400000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45500000000000002,"15min":0.47099999999999997},"min":{"1min":0.32700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67900000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.433},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.51700000000000002,"15min":0.52000000000000002},"min":{"1min":0.30499999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.70999999999999996,"5min":1.47,"15min":1.47},"last":0.45800000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.5,"15min":0.51800000000000002},"min":{"1min":0.40100000000000002,"5min":0.26700000000000002,"15min":0.246},"max":{"1min":0.80200000000000005,"5min":0.81799999999999995,"15min":0.85899999999999999},"last":0.505},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50900000000000001,"15min":0.52400000000000002},"min":{"1min":0.32300000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.80400000000000005},"last":0.53000000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575083,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307244,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064596,"statfs":{"total":96636764160,"available":96322146304,"internally_reserved":0,"allocated":1806336,"data_stored":929996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41199999999999998,"15min":0.40600000000000003},"min":{"1min":0.28100000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.72599999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.41999999999999998,"15min":0.439},"min":{"1min":0.28100000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.61499999999999999,"5min":0.78600000000000003,"15min":1.337},"last":0.38300000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.39700000000000002,"15min":0.40899999999999997},"min":{"1min":0.28699999999999998,"5min":0.254,"15min":0.23599999999999999},"max":{"1min":0.66300000000000003,"5min":0.68799999999999994,"15min":0.73299999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.46300000000000002,"15min":0.47199999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59299999999999997,"5min":0.74099999999999999,"15min":0.77300000000000002},"last":0.41199999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51300000000000001,"15min":0.51400000000000001},"min":{"1min":0.38400000000000001,"5min":0.34399999999999997,"15min":0.33200000000000002},"max":{"1min":0.73499999999999999,"5min":0.78900000000000003,"15min":1.782},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.32900000000000001,"5min":0.32000000000000001,"15min":0.26600000000000001},"max":{"1min":0.70199999999999996,"5min":0.71499999999999997,"15min":1.379},"last":0.48999999999999999}]},{"osd":4,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60699999999999998,"15min":0.64200000000000002},"min":{"1min":0.45100000000000001,"5min":0.40799999999999997,"15min":0.40200000000000002},"max":{"1min":0.85199999999999998,"5min":0.89200000000000002,"15min":1.0820000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.64500000000000002,"15min":0.65700000000000003},"min":{"1min":0.49199999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.57899999999999996}]},{"osd":5,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.64000000000000001,"15min":0.66800000000000004},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45000000000000001},"max":{"1min":0.86499999999999999,"5min":0.91100000000000003,"15min":1.208},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.63400000000000001,"15min":0.65900000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.42499999999999999},"max":{"1min":0.85099999999999998,"5min":0.96399999999999997,"15min":1.018},"last":0.56499999999999995}]},{"osd":6,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.63800000000000001,"15min":0.67000000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.42599999999999999},"max":{"1min":0.97399999999999998,"5min":1.022,"15min":1.2529999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.65000000000000002,"15min":0.67700000000000005},"min":{"1min":0.47899999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92300000000000004,"5min":0.93200000000000005,"15min":1.022},"last":0.63400000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.68300000000000005,"15min":0.70699999999999996},"min":{"1min":0.5,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.747},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.71899999999999997},"min":{"1min":0.505,"5min":0.496,"15min":0.496},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.296},"last":0.69799999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575082,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307824,"kb_used_data":2344,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064016,"statfs":{"total":96636764160,"available":96321552384,"internally_reserved":0,"allocated":2400256,"data_stored":1520364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.40600000000000003},"min":{"1min":0.25800000000000001,"5min":0.23899999999999999,"15min":0.23100000000000001},"max":{"1min":0.88500000000000001,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.42999999999999999,"15min":0.45100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65800000000000003,"5min":1.363,"15min":1.4490000000000001},"last":0.308}]},{"osd":2,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.41099999999999998,"15min":0.42299999999999999},"min":{"1min":0.28899999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.89600000000000002},"last":0.371},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.33900000000000002,"5min":0.32700000000000001,"15min":0.28000000000000003},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.41199999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.49199999999999999,"15min":0.47399999999999998},"min":{"1min":0.32800000000000001,"5min":0.32300000000000001,"15min":0.30599999999999999},"max":{"1min":0.85199999999999998,"5min":2.145,"15min":2.145},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36099999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":1.198},"last":0.501}]},{"osd":4,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60899999999999999,"15min":0.63800000000000001},"min":{"1min":0.41699999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78600000000000003,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.59699999999999998,"15min":0.63700000000000001},"min":{"1min":0.44400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.94299999999999995},"last":0.68200000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":1.329,"15min":1.329},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.66300000000000003,"15min":0.67000000000000004},"min":{"1min":0.46899999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82799999999999996,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.623}]},{"osd":6,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.61799999999999999,"15min":0.65100000000000002},"min":{"1min":0.47299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.89800000000000002,"5min":0.98699999999999999,"15min":1.032},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.65400000000000003,"15min":0.68000000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.41799999999999998},"max":{"1min":0.93100000000000005,"5min":1.286,"15min":1.286},"last":0.66000000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.70599999999999996,"15min":0.81699999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.435},"max":{"1min":0.81799999999999995,"5min":4.8710000000000004,"15min":25.965},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.73899999999999999,"15min":0.86099999999999999},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.433},"max":{"1min":0.88200000000000001,"5min":4.9199999999999999,"15min":33.085000000000001},"last":0.67200000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575082,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307240,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064600,"statfs":{"total":96636764160,"available":96322150400,"internally_reserved":0,"allocated":1802240,"data_stored":929974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.379,"15min":0.39100000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.255},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.375},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.43099999999999999,"15min":0.42599999999999999},"min":{"1min":0.29699999999999999,"5min":0.255,"15min":0.255},"max":{"1min":0.59899999999999998,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.48799999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.435,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.26900000000000002,"15min":0.248},"max":{"1min":0.63500000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52200000000000002,"15min":0.51100000000000001},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.68300000000000005,"5min":1.171,"15min":1.171},"last":0.52200000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.46899999999999997,"15min":0.48399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.313},"max":{"1min":0.64300000000000002,"5min":0.72799999999999998,"15min":0.74099999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47699999999999998},"min":{"1min":0.32900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70599999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.55900000000000005}]},{"osd":4,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59599999999999997,"15min":0.628},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.087},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.625,"15min":0.64300000000000002},"min":{"1min":0.39100000000000001,"5min":0.371,"15min":0.371},"max":{"1min":1.006,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.83199999999999996}]},{"osd":5,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.63900000000000001,"15min":0.65500000000000003},"min":{"1min":0.42299999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94699999999999995,"5min":1.24,"15min":1.24},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.65200000000000002,"15min":0.67400000000000004},"min":{"1min":0.48799999999999999,"5min":0.41799999999999998,"15min":0.35199999999999998},"max":{"1min":0.874,"5min":0.97699999999999998,"15min":1.085},"last":0.74099999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65700000000000003,"15min":0.67700000000000005},"min":{"1min":0.41199999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88600000000000001,"5min":1.022,"15min":1.1080000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.65200000000000002,"15min":0.67800000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.38800000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.1299999999999999},"last":0.76700000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68400000000000005,"15min":0.70099999999999996},"min":{"1min":0.46400000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.88800000000000001,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.69699999999999995,"15min":0.72499999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.42099999999999999},"max":{"1min":0.96899999999999997,"5min":1.05,"15min":1.139},"last":0.79200000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575081,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307256,"kb_used_data":1776,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064584,"statfs":{"total":96636764160,"available":96322134016,"internally_reserved":0,"allocated":1818624,"data_stored":931544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38400000000000001,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.55400000000000005,"5min":0.59699999999999998,"15min":1.8540000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.42899999999999999,"15min":0.438},"min":{"1min":0.29799999999999999,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":1.2290000000000001},"last":0.30099999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44500000000000001,"15min":0.45200000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":1.633},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.47999999999999998,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":1.0760000000000001},"last":0.38200000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48399999999999999,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65000000000000002,"5min":0.70399999999999996,"15min":1.4059999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.44500000000000001,"15min":0.45800000000000002},"min":{"1min":0.34100000000000003,"5min":0.29499999999999998,"15min":0.28299999999999997},"max":{"1min":0.629,"5min":0.69399999999999995,"15min":1.1870000000000001},"last":0.36699999999999999}]},{"osd":4,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.63,"15min":0.65800000000000003},"min":{"1min":0.48299999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":3.306},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.44800000000000001,"5min":0.40100000000000002,"15min":0.379},"max":{"1min":0.81499999999999995,"5min":0.85199999999999998,"15min":1.008},"last":0.504}]},{"osd":5,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.63600000000000001,"15min":0.65200000000000002},"min":{"1min":0.48599999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.85199999999999998,"5min":0.90400000000000003,"15min":1.593},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.47599999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.86299999999999999,"5min":0.92300000000000004,"15min":1.3819999999999999},"last":0.47599999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.66900000000000004},"min":{"1min":0.42299999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":1.506},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67200000000000004,"15min":0.68999999999999995},"min":{"1min":0.55600000000000005,"5min":0.46700000000000003,"15min":0.44400000000000001},"max":{"1min":0.88500000000000001,"5min":0.89000000000000001,"15min":1.427},"last":0.57899999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64400000000000002,"15min":0.68799999999999994},"min":{"1min":0.46700000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":3.238},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.71199999999999997,"15min":0.73399999999999999},"min":{"1min":0.54200000000000004,"5min":0.51900000000000002,"15min":0.48899999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":3.351},"last":0.52300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575083,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303156,"kb_used_data":1772,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1814528,"data_stored":931510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53400000000000003,"15min":0.56799999999999995},"min":{"1min":0.34799999999999998,"5min":0.28999999999999998,"15min":0.28699999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":1.21},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58399999999999996,"15min":0.58299999999999996},"min":{"1min":0.377,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.81499999999999995,"5min":5.367,"15min":5.367},"last":0.57399999999999995}]},{"osd":1,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56899999999999995,"15min":0.58699999999999997},"min":{"1min":0.378,"5min":0.311,"15min":0.311},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.54700000000000004,"15min":0.57399999999999995},"min":{"1min":0.40699999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79800000000000004,"5min":0.80600000000000005,"15min":0.81699999999999995},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.54600000000000004,"15min":0.56599999999999995},"min":{"1min":0.39300000000000002,"5min":0.33900000000000002,"15min":0.32700000000000001},"max":{"1min":0.747,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.57699999999999996,"15min":0.59599999999999997},"min":{"1min":0.43099999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":1.7050000000000001},"last":0.47599999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57299999999999995,"15min":0.58599999999999997},"min":{"1min":0.44900000000000001,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.35699999999999998},"max":{"1min":0.82499999999999996,"5min":0.94199999999999995,"15min":1.05},"last":0.54400000000000004}]},{"osd":5,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.46500000000000002},"min":{"1min":0.31,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.48099999999999998,"15min":0.48399999999999999},"min":{"1min":0.40100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.495}]},{"osd":6,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.46500000000000002,"15min":0.47399999999999998},"min":{"1min":0.34999999999999998,"5min":0.28599999999999998,"15min":0.27200000000000002},"max":{"1min":0.76600000000000001,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.42299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.52600000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.52600000000000002,"15min":0.54000000000000004},"min":{"1min":0.35599999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.56200000000000006,"15min":0.56899999999999995},"min":{"1min":0.436,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.88800000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.58799999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575083,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1768,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1810432,"data_stored":930134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.56399999999999995},"min":{"1min":0.35199999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.71099999999999997,"5min":0.81799999999999995,"15min":2.0289999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52700000000000002,"15min":0.56699999999999995},"min":{"1min":0.41999999999999998,"5min":0.29999999999999999,"15min":0.28899999999999998},"max":{"1min":0.70799999999999996,"5min":0.77400000000000002,"15min":0.82199999999999995},"last":0.65100000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56299999999999994,"15min":0.60399999999999998},"min":{"1min":0.443,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":0.754,"5min":0.79500000000000004,"15min":2.3319999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59899999999999998,"15min":0.626},"min":{"1min":0.45800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76400000000000001,"5min":0.80500000000000005,"15min":1.1739999999999999},"last":0.83499999999999996}]},{"osd":2,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53700000000000003,"15min":0.55700000000000005},"min":{"1min":0.36799999999999999,"5min":0.34499999999999997,"15min":0.30099999999999999},"max":{"1min":0.68400000000000005,"5min":0.79400000000000004,"15min":0.83799999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55300000000000005,"15min":0.58699999999999997},"min":{"1min":0.41299999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72899999999999998,"5min":0.81100000000000005,"15min":2.0600000000000001},"last":0.74299999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55300000000000005,"15min":0.57499999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.314},"max":{"1min":0.69699999999999995,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55900000000000005,"15min":0.58699999999999997},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73999999999999999,"5min":0.77900000000000003,"15min":2.5630000000000002},"last":0.70999999999999996}]},{"osd":4,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.42899999999999999,"15min":0.45500000000000002},"min":{"1min":0.313,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.65100000000000002,"5min":0.67200000000000004,"15min":2.3700000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45900000000000002,"15min":0.48099999999999998},"min":{"1min":0.32600000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.68200000000000005,"5min":0.82699999999999996,"15min":0.89200000000000002},"last":0.46999999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.47199999999999998},"min":{"1min":0.34300000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66900000000000004,"5min":0.67700000000000005,"15min":2.5990000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.49299999999999999,"15min":0.51400000000000001},"min":{"1min":0.374,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71399999999999997,"5min":0.76200000000000001,"15min":1.99},"last":0.61399999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51400000000000001,"15min":0.53800000000000003},"min":{"1min":0.374,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73799999999999999,"5min":0.76200000000000001,"15min":3.5270000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54400000000000004,"15min":0.55900000000000005},"min":{"1min":0.43099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78900000000000003,"5min":0.81499999999999995,"15min":0.82199999999999995},"last":0.59099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:49:27.021 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-26T23:49:27.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.189+0000 7fbf3f486700 1 Processor -- start 2024-01-26T23:49:27.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.189+0000 7fbf3f486700 1 -- start start 2024-01-26T23:49:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.190+0000 7fbf3f486700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf38157380 0x7fbf3815aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.190+0000 7fbf3f486700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbf3815d600 0x7fbf380ad180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.190+0000 7fbf3f486700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf380cd320 con 0x7fbf38157750 2024-01-26T23:49:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.190+0000 7fbf3f486700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf3815dac0 con 0x7fbf3815d600 2024-01-26T23:49:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.190+0000 7fbf3f486700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3815dda0 con 0x7fbf38157380 2024-01-26T23:49:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.190+0000 7fbf3da23700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbf38157750 0x7fbf380abe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43462/0 (socket says 172.21.15.46:43462) 2024-01-26T23:49:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.190+0000 7fbf3da23700 1 -- 172.21.15.46:0/135495355 learned_addr learned my addr 172.21.15.46:0/135495355 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.190+0000 7fbf3ca21700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbf3815d600 0x7fbf380ad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.191+0000 7fbf3d222700 1 --2- 172.21.15.46:0/135495355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf38157380 0x7fbf3815aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.191+0000 7fbf2bfff700 1 -- 172.21.15.46:0/135495355 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462910167 0 0) 0x7fbf380cd320 con 0x7fbf38157750 2024-01-26T23:49:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.191+0000 7fbf2bfff700 1 -- 172.21.15.46:0/135495355 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf24003570 con 0x7fbf38157750 2024-01-26T23:49:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.191+0000 7fbf3ca21700 1 -- 172.21.15.46:0/135495355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf38157380 msgr2=0x7fbf3815aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.191+0000 7fbf3ca21700 1 --2- 172.21.15.46:0/135495355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf38157380 0x7fbf3815aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.191+0000 7fbf3ca21700 1 -- 172.21.15.46:0/135495355 >> v1:172.21.15.46:6789/0 conn(0x7fbf38157750 legacy=0x7fbf380abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.191+0000 7fbf3ca21700 1 -- 172.21.15.46:0/135495355 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3815e080 con 0x7fbf3815d600 2024-01-26T23:49:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.191+0000 7fbf3ca21700 1 --2- 172.21.15.46:0/135495355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf3815d600 0x7fbf380ad180 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0098e0 tx=0x7fbf2c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d3340c68af8a942 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.191+0000 7fbf2bfff700 1 -- 172.21.15.46:0/135495355 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c00b640 con 0x7fbf3815d600 2024-01-26T23:49:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.191+0000 7fbf2bfff700 1 -- 172.21.15.46:0/135495355 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2c00b7a0 con 0x7fbf3815d600 2024-01-26T23:49:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.191+0000 7fbf2bfff700 1 -- 172.21.15.46:0/135495355 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c00bb40 con 0x7fbf3815d600 2024-01-26T23:49:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.192+0000 7fbf3f486700 1 -- 172.21.15.46:0/135495355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf3815d600 msgr2=0x7fbf380ad180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.192+0000 7fbf3f486700 1 --2- 172.21.15.46:0/135495355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf3815d600 0x7fbf380ad180 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0098e0 tx=0x7fbf2c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.192+0000 7fbf3f486700 1 -- 172.21.15.46:0/135495355 shutdown_connections 2024-01-26T23:49:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.192+0000 7fbf3f486700 1 --2- 172.21.15.46:0/135495355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf38157380 0x7fbf3815aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.192+0000 7fbf3f486700 1 --2- 172.21.15.46:0/135495355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf3815d600 0x7fbf380ad180 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.192+0000 7fbf3f486700 1 -- 172.21.15.46:0/135495355 >> 172.21.15.46:0/135495355 conn(0x7fbf38068400 msgr2=0x7fbf380cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.192+0000 7fbf3f486700 1 -- 172.21.15.46:0/135495355 shutdown_connections 2024-01-26T23:49:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.192+0000 7fbf3f486700 1 -- 172.21.15.46:0/135495355 wait complete. 2024-01-26T23:49:27.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.193+0000 7fbf3f486700 1 Processor -- start 2024-01-26T23:49:27.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.193+0000 7fbf3f486700 1 -- start start 2024-01-26T23:49:27.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.194+0000 7fbf3f486700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf38157380 0x7fbf380aa890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:27.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.194+0000 7fbf3f486700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf38157750 0x7fbf380ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:27.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.194+0000 7fbf3f486700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf3815d600 0x7fbf380ab670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:27.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.194+0000 7fbf3f486700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3816a610 con 0x7fbf38157380 2024-01-26T23:49:27.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.194+0000 7fbf3f486700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380c1d10 con 0x7fbf3815d600 2024-01-26T23:49:27.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.194+0000 7fbf3f486700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380c1ff0 con 0x7fbf38157750 2024-01-26T23:49:27.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.194+0000 7fbf3d222700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf38157380 0x7fbf380aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:27.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.194+0000 7fbf3d222700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf38157380 0x7fbf380aa890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40210/0 (socket says 172.21.15.46:40210) 2024-01-26T23:49:27.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.194+0000 7fbf3d222700 1 -- 172.21.15.46:0/3372102319 learned_addr learned my addr 172.21.15.46:0/3372102319 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:27.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.194+0000 7fbf3ca21700 1 --2- 172.21.15.46:0/3372102319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf38157750 0x7fbf380ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:27.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.194+0000 7fbf3da23700 1 --2- 172.21.15.46:0/3372102319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf3815d600 0x7fbf380ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:27.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.194+0000 7fbf3d222700 1 -- 172.21.15.46:0/3372102319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf38157750 msgr2=0x7fbf380ab130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:27.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.194+0000 7fbf3d222700 1 --2- 172.21.15.46:0/3372102319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf38157750 0x7fbf380ab130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.195+0000 7fbf3d222700 1 -- 172.21.15.46:0/3372102319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf3815d600 msgr2=0x7fbf380ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:27.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.195+0000 7fbf3d222700 1 --2- 172.21.15.46:0/3372102319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf3815d600 0x7fbf380ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.195+0000 7fbf3d222700 1 -- 172.21.15.46:0/3372102319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2c009420 con 0x7fbf38157380 2024-01-26T23:49:27.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.195+0000 7fbf3da23700 1 --2- 172.21.15.46:0/3372102319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf3815d600 0x7fbf380ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:27.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.195+0000 7fbf3d222700 1 --2- 172.21.15.46:0/3372102319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf38157380 0x7fbf380aa890 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fbf3400d860 tx=0x7fbf34008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:27.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.195+0000 7fbf297fa700 1 -- 172.21.15.46:0/3372102319 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf340125f0 con 0x7fbf38157380 2024-01-26T23:49:27.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.195+0000 7fbf297fa700 1 -- 172.21.15.46:0/3372102319 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf34012750 con 0x7fbf38157380 2024-01-26T23:49:27.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.195+0000 7fbf3f486700 1 -- 172.21.15.46:0/3372102319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf380c22d0 con 0x7fbf38157380 2024-01-26T23:49:27.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.195+0000 7fbf297fa700 1 -- 172.21.15.46:0/3372102319 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf34012b10 con 0x7fbf38157380 2024-01-26T23:49:27.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.195+0000 7fbf3f486700 1 -- 172.21.15.46:0/3372102319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf380afcc0 con 0x7fbf38157380 2024-01-26T23:49:27.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.196+0000 7fbf3f486700 1 -- 172.21.15.46:0/3372102319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf380b0030 con 0x7fbf38157380 2024-01-26T23:49:27.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.197+0000 7fbf297fa700 1 -- 172.21.15.46:0/3372102319 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbf3401e460 con 0x7fbf38157380 2024-01-26T23:49:27.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.197+0000 7fbf297fa700 1 --2- 172.21.15.46:0/3372102319 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf1c066f30 0x7fbf1c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:27.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.197+0000 7fbf297fa700 1 -- 172.21.15.46:0/3372102319 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fbf34084d30 con 0x7fbf38157380 2024-01-26T23:49:27.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.197+0000 7fbf3ca21700 1 --2- 172.21.15.46:0/3372102319 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf1c066f30 0x7fbf1c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:27.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.198+0000 7fbf3ca21700 1 --2- 172.21.15.46:0/3372102319 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf1c066f30 0x7fbf1c0693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbf3815d9f0 tx=0x7fbf2c017000 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:27.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.201+0000 7fbf297fa700 1 -- 172.21.15.46:0/3372102319 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbf34051cd0 con 0x7fbf38157380 2024-01-26T23:49:27.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.494+0000 7fbf3f486700 1 -- 172.21.15.46:0/3372102319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fbf3804c710 con 0x7fbf38157380 2024-01-26T23:49:27.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.495+0000 7fbf297fa700 1 -- 172.21.15.46:0/3372102319 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1027 (secure 0 0 0) 0x7fbf340552f0 con 0x7fbf38157380 2024-01-26T23:49:27.495 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:27.495 INFO:teuthology.orchestra.run.smithi046.stdout:{"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1200,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":145,"num_osds":8,"num_up_osds":8,"osd_up_since":1706311777,"num_in_osds":8,"osd_in_since":1706311769,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":96},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":2509303808,"bytes_avail":770584809472,"bytes_total":773094113280,"unknown_pgs_ratio":0.98969072103500366},"fsmap":{"epoch":70,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"a","status":"up:active","gid":8301}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-26T23:47:52.647933+0000","services":{}},"progress_events":{}} 2024-01-26T23:49:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.497+0000 7fbf3f486700 1 -- 172.21.15.46:0/3372102319 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf1c066f30 msgr2=0x7fbf1c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.497+0000 7fbf3f486700 1 --2- 172.21.15.46:0/3372102319 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf1c066f30 0x7fbf1c0693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbf3815d9f0 tx=0x7fbf2c017000 comp rx=0 tx=0).stop 2024-01-26T23:49:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.497+0000 7fbf3f486700 1 -- 172.21.15.46:0/3372102319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf38157380 msgr2=0x7fbf380aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.497+0000 7fbf3f486700 1 --2- 172.21.15.46:0/3372102319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf38157380 0x7fbf380aa890 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fbf3400d860 tx=0x7fbf34008040 comp rx=0 tx=0).stop 2024-01-26T23:49:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.497+0000 7fbf3f486700 1 -- 172.21.15.46:0/3372102319 shutdown_connections 2024-01-26T23:49:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.497+0000 7fbf3f486700 1 --2- 172.21.15.46:0/3372102319 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf1c066f30 0x7fbf1c0693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.497+0000 7fbf3f486700 1 --2- 172.21.15.46:0/3372102319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf38157380 0x7fbf380aa890 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.497+0000 7fbf3f486700 1 --2- 172.21.15.46:0/3372102319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf3815d600 0x7fbf380ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.497+0000 7fbf3f486700 1 --2- 172.21.15.46:0/3372102319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf38157750 0x7fbf380ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.497+0000 7fbf3f486700 1 -- 172.21.15.46:0/3372102319 >> 172.21.15.46:0/3372102319 conn(0x7fbf38068400 msgr2=0x7fbf38154d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.497+0000 7fbf3f486700 1 -- 172.21.15.46:0/3372102319 shutdown_connections 2024-01-26T23:49:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.497+0000 7fbf3f486700 1 -- 172.21.15.46:0/3372102319 wait complete. 2024-01-26T23:49:27.511 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-26T23:49:27.511 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:49:27.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.643+0000 7fb38cd4c700 1 Processor -- start 2024-01-26T23:49:27.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.643+0000 7fb38cd4c700 1 -- start start 2024-01-26T23:49:27.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.644+0000 7fb38cd4c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb3880b1fc0 0x7fb3880b2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:27.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.644+0000 7fb38cd4c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3880cac60 con 0x7fb388068310 2024-01-26T23:49:27.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.644+0000 7fb38cd4c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb388068dc0 con 0x7fb388157690 2024-01-26T23:49:27.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.644+0000 7fb38cd4c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3880690a0 con 0x7fb3880b1fc0 2024-01-26T23:49:27.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.645+0000 7fb38659c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb3880b1fc0 0x7fb3880b2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:27.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.645+0000 7fb37dd9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb388068310 0x7fb388156f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43474/0 (socket says 172.21.15.46:43474) 2024-01-26T23:49:27.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.645+0000 7fb37dd9b700 1 -- 172.21.15.46:0/1984561267 learned_addr learned my addr 172.21.15.46:0/1984561267 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:27.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.645+0000 7fb38659c700 1 -- 172.21.15.46:0/1984561267 >> v1:172.21.15.106:6789/0 conn(0x7fb388157690 legacy=0x7fb3880cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:27.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.645+0000 7fb38659c700 1 -- 172.21.15.46:0/1984561267 >> v1:172.21.15.46:6789/0 conn(0x7fb388068310 legacy=0x7fb388156f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:27.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.645+0000 7fb38659c700 1 -- 172.21.15.46:0/1984561267 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3880caee0 con 0x7fb3880b1fc0 2024-01-26T23:49:27.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.645+0000 7fb385d9b700 1 -- 172.21.15.46:0/1984561267 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402898937 0 0) 0x7fb3880cac60 con 0x7fb388068310 2024-01-26T23:49:27.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.645+0000 7fb38659c700 1 --2- 172.21.15.46:0/1984561267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880b1fc0 0x7fb3880b2390 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fb3780098e0 tx=0x7fb3780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a3999174dfe0651 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:27.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.645+0000 7fb385d9b700 1 -- 172.21.15.46:0/1984561267 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb378009e80 con 0x7fb3880b1fc0 2024-01-26T23:49:27.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.645+0000 7fb385d9b700 1 -- 172.21.15.46:0/1984561267 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37800b860 con 0x7fb3880b1fc0 2024-01-26T23:49:27.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.645+0000 7fb385d9b700 1 -- 172.21.15.46:0/1984561267 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37800bbe0 con 0x7fb3880b1fc0 2024-01-26T23:49:27.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.646+0000 7fb38cd4c700 1 -- 172.21.15.46:0/1984561267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880b1fc0 msgr2=0x7fb3880b2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:27.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.646+0000 7fb38cd4c700 1 --2- 172.21.15.46:0/1984561267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880b1fc0 0x7fb3880b2390 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fb3780098e0 tx=0x7fb3780091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.646+0000 7fb38cd4c700 1 -- 172.21.15.46:0/1984561267 shutdown_connections 2024-01-26T23:49:27.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.646+0000 7fb38cd4c700 1 --2- 172.21.15.46:0/1984561267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880b1fc0 0x7fb3880b2390 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.646+0000 7fb38cd4c700 1 -- 172.21.15.46:0/1984561267 >> 172.21.15.46:0/1984561267 conn(0x7fb3880ccbc0 msgr2=0x7fb3880ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:27.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.646+0000 7fb38cd4c700 1 -- 172.21.15.46:0/1984561267 shutdown_connections 2024-01-26T23:49:27.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.646+0000 7fb38cd4c700 1 -- 172.21.15.46:0/1984561267 wait complete. 2024-01-26T23:49:27.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.647+0000 7fb38cd4c700 1 Processor -- start 2024-01-26T23:49:27.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.647+0000 7fb38cd4c700 1 -- start start 2024-01-26T23:49:27.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.648+0000 7fb38cd4c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb388068310 0x7fb3880c6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:27.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.648+0000 7fb38cd4c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880b1fc0 0x7fb3881444b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:27.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.648+0000 7fb38cd4c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb388157690 0x7fb38815c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:27.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.648+0000 7fb38cd4c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38815b8d0 con 0x7fb3880b1fc0 2024-01-26T23:49:27.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.648+0000 7fb38cd4c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3880c6d60 con 0x7fb388068310 2024-01-26T23:49:27.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.648+0000 7fb38cd4c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38815d000 con 0x7fb388157690 2024-01-26T23:49:27.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.648+0000 7fb37dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880b1fc0 0x7fb3881444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:27.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.648+0000 7fb37dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880b1fc0 0x7fb3881444b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40240/0 (socket says 172.21.15.46:40240) 2024-01-26T23:49:27.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.648+0000 7fb37dd9b700 1 -- 172.21.15.46:0/1510273257 learned_addr learned my addr 172.21.15.46:0/1510273257 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:27.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.648+0000 7fb38659c700 1 --2- 172.21.15.46:0/1510273257 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb388068310 0x7fb3880c6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:27.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.648+0000 7fb386d9d700 1 --2- 172.21.15.46:0/1510273257 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb388157690 0x7fb38815c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:27.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.649+0000 7fb37dd9b700 1 -- 172.21.15.46:0/1510273257 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb388157690 msgr2=0x7fb38815c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:27.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.649+0000 7fb37dd9b700 1 --2- 172.21.15.46:0/1510273257 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb388157690 0x7fb38815c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.649+0000 7fb37dd9b700 1 -- 172.21.15.46:0/1510273257 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb388068310 msgr2=0x7fb3880c6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:27.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.649+0000 7fb37dd9b700 1 --2- 172.21.15.46:0/1510273257 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb388068310 0x7fb3880c6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.649+0000 7fb37dd9b700 1 -- 172.21.15.46:0/1510273257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb378009420 con 0x7fb3880b1fc0 2024-01-26T23:49:27.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.649+0000 7fb37dd9b700 1 --2- 172.21.15.46:0/1510273257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880b1fc0 0x7fb3881444b0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fb37000c810 tx=0x7fb370007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:27.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.649+0000 7fb37effd700 1 -- 172.21.15.46:0/1510273257 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb370005c00 con 0x7fb3880b1fc0 2024-01-26T23:49:27.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.649+0000 7fb37effd700 1 -- 172.21.15.46:0/1510273257 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb370005d60 con 0x7fb3880b1fc0 2024-01-26T23:49:27.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.649+0000 7fb37effd700 1 -- 172.21.15.46:0/1510273257 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb370007320 con 0x7fb3880b1fc0 2024-01-26T23:49:27.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.649+0000 7fb38cd4c700 1 -- 172.21.15.46:0/1510273257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3880c2080 con 0x7fb3880b1fc0 2024-01-26T23:49:27.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.649+0000 7fb38cd4c700 1 -- 172.21.15.46:0/1510273257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3880c2640 con 0x7fb3880b1fc0 2024-01-26T23:49:27.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.650+0000 7fb38cd4c700 1 -- 172.21.15.46:0/1510273257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb38804c710 con 0x7fb3880b1fc0 2024-01-26T23:49:27.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.651+0000 7fb37effd700 1 -- 172.21.15.46:0/1510273257 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb37001b3b0 con 0x7fb3880b1fc0 2024-01-26T23:49:27.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.651+0000 7fb37effd700 1 --2- 172.21.15.46:0/1510273257 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb368066f30 0x7fb3680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:27.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.651+0000 7fb37effd700 1 -- 172.21.15.46:0/1510273257 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fb370083cf0 con 0x7fb3880b1fc0 2024-01-26T23:49:27.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.652+0000 7fb38659c700 1 --2- 172.21.15.46:0/1510273257 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb368066f30 0x7fb3680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:27.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.652+0000 7fb38659c700 1 --2- 172.21.15.46:0/1510273257 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb368066f30 0x7fb3680693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb3780093f0 tx=0x7fb378017000 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:27.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.656+0000 7fb37effd700 1 -- 172.21.15.46:0/1510273257 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb370050bf0 con 0x7fb3880b1fc0 2024-01-26T23:49:27.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.856+0000 7fb38cd4c700 1 -- 172.21.15.46:0/1510273257 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb3880cb1c0 con 0x7fb368066f30 2024-01-26T23:49:27.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.865+0000 7fb37effd700 1 -- 172.21.15.46:0/1510273257 <== mgr.7148 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289774 (secure 0 0 0) 0x7fb3880cb1c0 con 0x7fb368066f30 2024-01-26T23:49:27.869 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.868+0000 7fb38cd4c700 1 -- 172.21.15.46:0/1510273257 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb368066f30 msgr2=0x7fb3680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.868+0000 7fb38cd4c700 1 --2- 172.21.15.46:0/1510273257 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb368066f30 0x7fb3680693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb3780093f0 tx=0x7fb378017000 comp rx=0 tx=0).stop 2024-01-26T23:49:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.868+0000 7fb38cd4c700 1 -- 172.21.15.46:0/1510273257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880b1fc0 msgr2=0x7fb3881444b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.868+0000 7fb38cd4c700 1 --2- 172.21.15.46:0/1510273257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880b1fc0 0x7fb3881444b0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fb37000c810 tx=0x7fb370007d90 comp rx=0 tx=0).stop 2024-01-26T23:49:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.869+0000 7fb38cd4c700 1 -- 172.21.15.46:0/1510273257 shutdown_connections 2024-01-26T23:49:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.869+0000 7fb38cd4c700 1 --2- 172.21.15.46:0/1510273257 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb368066f30 0x7fb3680693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.869+0000 7fb38cd4c700 1 --2- 172.21.15.46:0/1510273257 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb388157690 0x7fb38815c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.869+0000 7fb38cd4c700 1 --2- 172.21.15.46:0/1510273257 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb388068310 0x7fb3880c6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.869+0000 7fb38cd4c700 1 --2- 172.21.15.46:0/1510273257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880b1fc0 0x7fb3881444b0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.869+0000 7fb38cd4c700 1 -- 172.21.15.46:0/1510273257 >> 172.21.15.46:0/1510273257 conn(0x7fb3880ccbc0 msgr2=0x7fb3880cb5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.869+0000 7fb38cd4c700 1 -- 172.21.15.46:0/1510273257 shutdown_connections 2024-01-26T23:49:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:27.869+0000 7fb38cd4c700 1 -- 172.21.15.46:0/1510273257 wait complete. 2024-01-26T23:49:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:49:27.882 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-26T23:49:26.870738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1163,"num_read_kb":2314,"num_write":404,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8622,"ondisk_log_size":8622,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2450952,"kb_used_data":15304,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752523768,"statfs":{"total":773094113280,"available":770584338432,"internally_reserved":0,"allocated":15671296,"data_stored":8625140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001406"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720478+0000","last_change":"2024-01-26T23:49:21.716820+0000","last_active":"2024-01-26T23:49:21.720478+0000","last_peered":"2024-01-26T23:49:21.720478+0000","last_clean":"2024-01-26T23:49:21.720478+0000","last_became_active":"2024-01-26T23:49:21.716559+0000","last_became_peered":"2024-01-26T23:49:21.716559+0000","last_unstale":"2024-01-26T23:49:21.720478+0000","last_undegraded":"2024-01-26T23:49:21.720478+0000","last_fullsized":"2024-01-26T23:49:21.720478+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:31:49.800868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713577+0000","last_change":"2024-01-26T23:49:21.706608+0000","last_active":"2024-01-26T23:49:21.713577+0000","last_peered":"2024-01-26T23:49:21.713577+0000","last_clean":"2024-01-26T23:49:21.713577+0000","last_became_active":"2024-01-26T23:49:21.706095+0000","last_became_peered":"2024-01-26T23:49:21.706095+0000","last_unstale":"2024-01-26T23:49:21.713577+0000","last_undegraded":"2024-01-26T23:49:21.713577+0000","last_fullsized":"2024-01-26T23:49:21.713577+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:40:00.810209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713754+0000","last_change":"2024-01-26T23:49:21.710842+0000","last_active":"2024-01-26T23:49:21.713754+0000","last_peered":"2024-01-26T23:49:21.713754+0000","last_clean":"2024-01-26T23:49:21.713754+0000","last_became_active":"2024-01-26T23:49:21.710542+0000","last_became_peered":"2024-01-26T23:49:21.710542+0000","last_unstale":"2024-01-26T23:49:21.713754+0000","last_undegraded":"2024-01-26T23:49:21.713754+0000","last_fullsized":"2024-01-26T23:49:21.713754+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:46:28.563573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720399+0000","last_change":"2024-01-26T23:49:21.715413+0000","last_active":"2024-01-26T23:49:21.720399+0000","last_peered":"2024-01-26T23:49:21.720399+0000","last_clean":"2024-01-26T23:49:21.720399+0000","last_became_active":"2024-01-26T23:49:21.715122+0000","last_became_peered":"2024-01-26T23:49:21.715122+0000","last_unstale":"2024-01-26T23:49:21.720399+0000","last_undegraded":"2024-01-26T23:49:21.720399+0000","last_fullsized":"2024-01-26T23:49:21.720399+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:23:01.412861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714883+0000","last_change":"2024-01-26T23:49:21.711789+0000","last_active":"2024-01-26T23:49:21.714883+0000","last_peered":"2024-01-26T23:49:21.714883+0000","last_clean":"2024-01-26T23:49:21.714883+0000","last_became_active":"2024-01-26T23:49:21.711475+0000","last_became_peered":"2024-01-26T23:49:21.711475+0000","last_unstale":"2024-01-26T23:49:21.714883+0000","last_undegraded":"2024-01-26T23:49:21.714883+0000","last_fullsized":"2024-01-26T23:49:21.714883+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:22:39.451844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.709457+0000","last_change":"2024-01-26T23:49:21.704699+0000","last_active":"2024-01-26T23:49:21.709457+0000","last_peered":"2024-01-26T23:49:21.709457+0000","last_clean":"2024-01-26T23:49:21.709457+0000","last_became_active":"2024-01-26T23:49:21.704402+0000","last_became_peered":"2024-01-26T23:49:21.704402+0000","last_unstale":"2024-01-26T23:49:21.709457+0000","last_undegraded":"2024-01-26T23:49:21.709457+0000","last_fullsized":"2024-01-26T23:49:21.709457+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:38.865560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.710793+0000","last_change":"2024-01-26T23:49:21.705657+0000","last_active":"2024-01-26T23:49:21.710793+0000","last_peered":"2024-01-26T23:49:21.710793+0000","last_clean":"2024-01-26T23:49:21.710793+0000","last_became_active":"2024-01-26T23:49:21.705381+0000","last_became_peered":"2024-01-26T23:49:21.705381+0000","last_unstale":"2024-01-26T23:49:21.710793+0000","last_undegraded":"2024-01-26T23:49:21.710793+0000","last_fullsized":"2024-01-26T23:49:21.710793+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:53:27.231318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716206+0000","last_change":"2024-01-26T23:49:21.714982+0000","last_active":"2024-01-26T23:49:21.716206+0000","last_peered":"2024-01-26T23:49:21.716206+0000","last_clean":"2024-01-26T23:49:21.716206+0000","last_became_active":"2024-01-26T23:49:21.714655+0000","last_became_peered":"2024-01-26T23:49:21.714655+0000","last_unstale":"2024-01-26T23:49:21.716206+0000","last_undegraded":"2024-01-26T23:49:21.716206+0000","last_fullsized":"2024-01-26T23:49:21.716206+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:46:36.438311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720506+0000","last_change":"2024-01-26T23:49:21.716127+0000","last_active":"2024-01-26T23:49:21.720506+0000","last_peered":"2024-01-26T23:49:21.720506+0000","last_clean":"2024-01-26T23:49:21.720506+0000","last_became_active":"2024-01-26T23:49:21.715868+0000","last_became_peered":"2024-01-26T23:49:21.715868+0000","last_unstale":"2024-01-26T23:49:21.720506+0000","last_undegraded":"2024-01-26T23:49:21.720506+0000","last_fullsized":"2024-01-26T23:49:21.720506+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:27:10.909797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.705419+0000","last_change":"2024-01-26T23:49:21.701945+0000","last_active":"2024-01-26T23:49:21.705419+0000","last_peered":"2024-01-26T23:49:21.705419+0000","last_clean":"2024-01-26T23:49:21.705419+0000","last_became_active":"2024-01-26T23:49:21.701542+0000","last_became_peered":"2024-01-26T23:49:21.701542+0000","last_unstale":"2024-01-26T23:49:21.705419+0000","last_undegraded":"2024-01-26T23:49:21.705419+0000","last_fullsized":"2024-01-26T23:49:21.705419+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:51.588887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720594+0000","last_change":"2024-01-26T23:49:21.717788+0000","last_active":"2024-01-26T23:49:21.720594+0000","last_peered":"2024-01-26T23:49:21.720594+0000","last_clean":"2024-01-26T23:49:21.720594+0000","last_became_active":"2024-01-26T23:49:21.717188+0000","last_became_peered":"2024-01-26T23:49:21.717188+0000","last_unstale":"2024-01-26T23:49:21.720594+0000","last_undegraded":"2024-01-26T23:49:21.720594+0000","last_fullsized":"2024-01-26T23:49:21.720594+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:52:44.481461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716190+0000","last_change":"2024-01-26T23:49:21.713083+0000","last_active":"2024-01-26T23:49:21.716190+0000","last_peered":"2024-01-26T23:49:21.716190+0000","last_clean":"2024-01-26T23:49:21.716190+0000","last_became_active":"2024-01-26T23:49:21.712770+0000","last_became_peered":"2024-01-26T23:49:21.712770+0000","last_unstale":"2024-01-26T23:49:21.716190+0000","last_undegraded":"2024-01-26T23:49:21.716190+0000","last_fullsized":"2024-01-26T23:49:21.716190+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:55:44.107048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.701246+0000","last_change":"2024-01-26T23:49:21.698975+0000","last_active":"2024-01-26T23:49:21.701246+0000","last_peered":"2024-01-26T23:49:21.701246+0000","last_clean":"2024-01-26T23:49:21.701246+0000","last_became_active":"2024-01-26T23:49:21.698670+0000","last_became_peered":"2024-01-26T23:49:21.698670+0000","last_unstale":"2024-01-26T23:49:21.701246+0000","last_undegraded":"2024-01-26T23:49:21.701246+0000","last_fullsized":"2024-01-26T23:49:21.701246+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:46:36.438311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713198+0000","last_change":"2024-01-26T23:49:21.711141+0000","last_active":"2024-01-26T23:49:21.713198+0000","last_peered":"2024-01-26T23:49:21.713198+0000","last_clean":"2024-01-26T23:49:21.713198+0000","last_became_active":"2024-01-26T23:49:21.710788+0000","last_became_peered":"2024-01-26T23:49:21.710788+0000","last_unstale":"2024-01-26T23:49:21.713198+0000","last_undegraded":"2024-01-26T23:49:21.713198+0000","last_fullsized":"2024-01-26T23:49:21.713198+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:32:26.662337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715295+0000","last_change":"2024-01-26T23:49:21.712785+0000","last_active":"2024-01-26T23:49:21.715295+0000","last_peered":"2024-01-26T23:49:21.715295+0000","last_clean":"2024-01-26T23:49:21.715295+0000","last_became_active":"2024-01-26T23:49:21.711603+0000","last_became_peered":"2024-01-26T23:49:21.711603+0000","last_unstale":"2024-01-26T23:49:21.715295+0000","last_undegraded":"2024-01-26T23:49:21.715295+0000","last_fullsized":"2024-01-26T23:49:21.715295+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:42:38.767042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.710670+0000","last_change":"2024-01-26T23:49:21.704936+0000","last_active":"2024-01-26T23:49:21.710670+0000","last_peered":"2024-01-26T23:49:21.710670+0000","last_clean":"2024-01-26T23:49:21.710670+0000","last_became_active":"2024-01-26T23:49:21.704639+0000","last_became_peered":"2024-01-26T23:49:21.704639+0000","last_unstale":"2024-01-26T23:49:21.710670+0000","last_undegraded":"2024-01-26T23:49:21.710670+0000","last_fullsized":"2024-01-26T23:49:21.710670+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:26:52.470020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.707219+0000","last_change":"2024-01-26T23:49:21.702319+0000","last_active":"2024-01-26T23:49:21.707219+0000","last_peered":"2024-01-26T23:49:21.707219+0000","last_clean":"2024-01-26T23:49:21.707219+0000","last_became_active":"2024-01-26T23:49:21.701995+0000","last_became_peered":"2024-01-26T23:49:21.701995+0000","last_unstale":"2024-01-26T23:49:21.707219+0000","last_undegraded":"2024-01-26T23:49:21.707219+0000","last_fullsized":"2024-01-26T23:49:21.707219+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:44:38.742019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713556+0000","last_change":"2024-01-26T23:49:21.706418+0000","last_active":"2024-01-26T23:49:21.713556+0000","last_peered":"2024-01-26T23:49:21.713556+0000","last_clean":"2024-01-26T23:49:21.713556+0000","last_became_active":"2024-01-26T23:49:21.705710+0000","last_became_peered":"2024-01-26T23:49:21.705710+0000","last_unstale":"2024-01-26T23:49:21.713556+0000","last_undegraded":"2024-01-26T23:49:21.713556+0000","last_fullsized":"2024-01-26T23:49:21.713556+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:07:15.034585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715571+0000","last_change":"2024-01-26T23:49:21.708843+0000","last_active":"2024-01-26T23:49:21.715571+0000","last_peered":"2024-01-26T23:49:21.715571+0000","last_clean":"2024-01-26T23:49:21.715571+0000","last_became_active":"2024-01-26T23:49:21.708590+0000","last_became_peered":"2024-01-26T23:49:21.708590+0000","last_unstale":"2024-01-26T23:49:21.715571+0000","last_undegraded":"2024-01-26T23:49:21.715571+0000","last_fullsized":"2024-01-26T23:49:21.715571+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:47:59.288036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.712492+0000","last_change":"2024-01-26T23:49:21.709022+0000","last_active":"2024-01-26T23:49:21.712492+0000","last_peered":"2024-01-26T23:49:21.712492+0000","last_clean":"2024-01-26T23:49:21.712492+0000","last_became_active":"2024-01-26T23:49:21.708716+0000","last_became_peered":"2024-01-26T23:49:21.708716+0000","last_unstale":"2024-01-26T23:49:21.712492+0000","last_undegraded":"2024-01-26T23:49:21.712492+0000","last_fullsized":"2024-01-26T23:49:21.712492+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:18:32.453554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.717173+0000","last_change":"2024-01-26T23:49:21.712966+0000","last_active":"2024-01-26T23:49:21.717173+0000","last_peered":"2024-01-26T23:49:21.717173+0000","last_clean":"2024-01-26T23:49:21.717173+0000","last_became_active":"2024-01-26T23:49:21.712337+0000","last_became_peered":"2024-01-26T23:49:21.712337+0000","last_unstale":"2024-01-26T23:49:21.717173+0000","last_undegraded":"2024-01-26T23:49:21.717173+0000","last_fullsized":"2024-01-26T23:49:21.717173+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:36:49.497687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714994+0000","last_change":"2024-01-26T23:49:21.712389+0000","last_active":"2024-01-26T23:49:21.714994+0000","last_peered":"2024-01-26T23:49:21.714994+0000","last_clean":"2024-01-26T23:49:21.714994+0000","last_became_active":"2024-01-26T23:49:21.712138+0000","last_became_peered":"2024-01-26T23:49:21.712138+0000","last_unstale":"2024-01-26T23:49:21.714994+0000","last_undegraded":"2024-01-26T23:49:21.714994+0000","last_fullsized":"2024-01-26T23:49:21.714994+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:36:35.342280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.717216+0000","last_change":"2024-01-26T23:49:21.713016+0000","last_active":"2024-01-26T23:49:21.717216+0000","last_peered":"2024-01-26T23:49:21.717216+0000","last_clean":"2024-01-26T23:49:21.717216+0000","last_became_active":"2024-01-26T23:49:21.712716+0000","last_became_peered":"2024-01-26T23:49:21.712716+0000","last_unstale":"2024-01-26T23:49:21.717216+0000","last_undegraded":"2024-01-26T23:49:21.717216+0000","last_fullsized":"2024-01-26T23:49:21.717216+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:19:56.848512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716319+0000","last_change":"2024-01-26T23:49:21.712460+0000","last_active":"2024-01-26T23:49:21.716319+0000","last_peered":"2024-01-26T23:49:21.716319+0000","last_clean":"2024-01-26T23:49:21.716319+0000","last_became_active":"2024-01-26T23:49:21.712123+0000","last_became_peered":"2024-01-26T23:49:21.712123+0000","last_unstale":"2024-01-26T23:49:21.716319+0000","last_undegraded":"2024-01-26T23:49:21.716319+0000","last_fullsized":"2024-01-26T23:49:21.716319+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:57:09.862596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716309+0000","last_change":"2024-01-26T23:49:21.712418+0000","last_active":"2024-01-26T23:49:21.716309+0000","last_peered":"2024-01-26T23:49:21.716309+0000","last_clean":"2024-01-26T23:49:21.716309+0000","last_became_active":"2024-01-26T23:49:21.712099+0000","last_became_peered":"2024-01-26T23:49:21.712099+0000","last_unstale":"2024-01-26T23:49:21.716309+0000","last_undegraded":"2024-01-26T23:49:21.716309+0000","last_fullsized":"2024-01-26T23:49:21.716309+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:17.484964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.708922+0000","last_change":"2024-01-26T23:49:21.705150+0000","last_active":"2024-01-26T23:49:21.708922+0000","last_peered":"2024-01-26T23:49:21.708922+0000","last_clean":"2024-01-26T23:49:21.708922+0000","last_became_active":"2024-01-26T23:49:21.703248+0000","last_became_peered":"2024-01-26T23:49:21.703248+0000","last_unstale":"2024-01-26T23:49:21.708922+0000","last_undegraded":"2024-01-26T23:49:21.708922+0000","last_fullsized":"2024-01-26T23:49:21.708922+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:49:45.239681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713631+0000","last_change":"2024-01-26T23:49:21.708382+0000","last_active":"2024-01-26T23:49:21.713631+0000","last_peered":"2024-01-26T23:49:21.713631+0000","last_clean":"2024-01-26T23:49:21.713631+0000","last_became_active":"2024-01-26T23:49:21.708082+0000","last_became_peered":"2024-01-26T23:49:21.708082+0000","last_unstale":"2024-01-26T23:49:21.713631+0000","last_undegraded":"2024-01-26T23:49:21.713631+0000","last_fullsized":"2024-01-26T23:49:21.713631+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:44:38.742019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716169+0000","last_change":"2024-01-26T23:49:21.711944+0000","last_active":"2024-01-26T23:49:21.716169+0000","last_peered":"2024-01-26T23:49:21.716169+0000","last_clean":"2024-01-26T23:49:21.716169+0000","last_became_active":"2024-01-26T23:49:21.711661+0000","last_became_peered":"2024-01-26T23:49:21.711661+0000","last_unstale":"2024-01-26T23:49:21.716169+0000","last_undegraded":"2024-01-26T23:49:21.716169+0000","last_fullsized":"2024-01-26T23:49:21.716169+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:07.408591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714134+0000","last_change":"2024-01-26T23:49:21.711617+0000","last_active":"2024-01-26T23:49:21.714134+0000","last_peered":"2024-01-26T23:49:21.714134+0000","last_clean":"2024-01-26T23:49:21.714134+0000","last_became_active":"2024-01-26T23:49:21.711167+0000","last_became_peered":"2024-01-26T23:49:21.711167+0000","last_unstale":"2024-01-26T23:49:21.714134+0000","last_undegraded":"2024-01-26T23:49:21.714134+0000","last_fullsized":"2024-01-26T23:49:21.714134+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:07:55.842419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715903+0000","last_change":"2024-01-26T23:49:21.713998+0000","last_active":"2024-01-26T23:49:21.715903+0000","last_peered":"2024-01-26T23:49:21.715903+0000","last_clean":"2024-01-26T23:49:21.715903+0000","last_became_active":"2024-01-26T23:49:21.713699+0000","last_became_peered":"2024-01-26T23:49:21.713699+0000","last_unstale":"2024-01-26T23:49:21.715903+0000","last_undegraded":"2024-01-26T23:49:21.715903+0000","last_fullsized":"2024-01-26T23:49:21.715903+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:31:18.118068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.710887+0000","last_change":"2024-01-26T23:49:21.701621+0000","last_active":"2024-01-26T23:49:21.710887+0000","last_peered":"2024-01-26T23:49:21.710887+0000","last_clean":"2024-01-26T23:49:21.710887+0000","last_became_active":"2024-01-26T23:49:21.701321+0000","last_became_peered":"2024-01-26T23:49:21.701321+0000","last_unstale":"2024-01-26T23:49:21.710887+0000","last_undegraded":"2024-01-26T23:49:21.710887+0000","last_fullsized":"2024-01-26T23:49:21.710887+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:55:23.167302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716353+0000","last_change":"2024-01-26T23:49:21.712792+0000","last_active":"2024-01-26T23:49:21.716353+0000","last_peered":"2024-01-26T23:49:21.716353+0000","last_clean":"2024-01-26T23:49:21.716353+0000","last_became_active":"2024-01-26T23:49:21.711757+0000","last_became_peered":"2024-01-26T23:49:21.711757+0000","last_unstale":"2024-01-26T23:49:21.716353+0000","last_undegraded":"2024-01-26T23:49:21.716353+0000","last_fullsized":"2024-01-26T23:49:21.716353+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:03:11.827036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.708503+0000","last_change":"2024-01-26T23:49:21.706929+0000","last_active":"2024-01-26T23:49:21.708503+0000","last_peered":"2024-01-26T23:49:21.708503+0000","last_clean":"2024-01-26T23:49:21.708503+0000","last_became_active":"2024-01-26T23:49:21.706543+0000","last_became_peered":"2024-01-26T23:49:21.706543+0000","last_unstale":"2024-01-26T23:49:21.708503+0000","last_undegraded":"2024-01-26T23:49:21.708503+0000","last_fullsized":"2024-01-26T23:49:21.708503+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:34.441797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"144'4","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.947803+0000","last_change":"2024-01-26T23:49:20.933655+0000","last_active":"2024-01-26T23:49:21.947803+0000","last_peered":"2024-01-26T23:49:21.947803+0000","last_clean":"2024-01-26T23:49:21.947803+0000","last_became_active":"2024-01-26T23:49:20.933360+0000","last_became_peered":"2024-01-26T23:49:20.933360+0000","last_unstale":"2024-01-26T23:49:21.947803+0000","last_undegraded":"2024-01-26T23:49:21.947803+0000","last_fullsized":"2024-01-26T23:49:21.947803+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:23:00.411032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716434+0000","last_change":"2024-01-26T23:49:21.712338+0000","last_active":"2024-01-26T23:49:21.716434+0000","last_peered":"2024-01-26T23:49:21.716434+0000","last_clean":"2024-01-26T23:49:21.716434+0000","last_became_active":"2024-01-26T23:49:21.712030+0000","last_became_peered":"2024-01-26T23:49:21.712030+0000","last_unstale":"2024-01-26T23:49:21.716434+0000","last_undegraded":"2024-01-26T23:49:21.716434+0000","last_fullsized":"2024-01-26T23:49:21.716434+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:25:47.504184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.947564+0000","last_change":"2024-01-26T23:49:20.934413+0000","last_active":"2024-01-26T23:49:21.947564+0000","last_peered":"2024-01-26T23:49:21.947564+0000","last_clean":"2024-01-26T23:49:21.947564+0000","last_became_active":"2024-01-26T23:49:20.933769+0000","last_became_peered":"2024-01-26T23:49:20.933769+0000","last_unstale":"2024-01-26T23:49:21.947564+0000","last_undegraded":"2024-01-26T23:49:21.947564+0000","last_fullsized":"2024-01-26T23:49:21.947564+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:05:23.745399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716376+0000","last_change":"2024-01-26T23:49:21.712916+0000","last_active":"2024-01-26T23:49:21.716376+0000","last_peered":"2024-01-26T23:49:21.716376+0000","last_clean":"2024-01-26T23:49:21.716376+0000","last_became_active":"2024-01-26T23:49:21.712259+0000","last_became_peered":"2024-01-26T23:49:21.712259+0000","last_unstale":"2024-01-26T23:49:21.716376+0000","last_undegraded":"2024-01-26T23:49:21.716376+0000","last_fullsized":"2024-01-26T23:49:21.716376+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:05:24.747228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.947622+0000","last_change":"2024-01-26T23:49:20.933682+0000","last_active":"2024-01-26T23:49:21.947622+0000","last_peered":"2024-01-26T23:49:21.947622+0000","last_clean":"2024-01-26T23:49:21.947622+0000","last_became_active":"2024-01-26T23:49:20.933419+0000","last_became_peered":"2024-01-26T23:49:20.933419+0000","last_unstale":"2024-01-26T23:49:21.947622+0000","last_undegraded":"2024-01-26T23:49:21.947622+0000","last_fullsized":"2024-01-26T23:49:21.947622+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:31:23.721288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714845+0000","last_change":"2024-01-26T23:49:21.709566+0000","last_active":"2024-01-26T23:49:21.714845+0000","last_peered":"2024-01-26T23:49:21.714845+0000","last_clean":"2024-01-26T23:49:21.714845+0000","last_became_active":"2024-01-26T23:49:21.709308+0000","last_became_peered":"2024-01-26T23:49:21.709308+0000","last_unstale":"2024-01-26T23:49:21.714845+0000","last_undegraded":"2024-01-26T23:49:21.714845+0000","last_fullsized":"2024-01-26T23:49:21.714845+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:01:52.607618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.951313+0000","last_change":"2024-01-26T23:49:20.941242+0000","last_active":"2024-01-26T23:49:21.951313+0000","last_peered":"2024-01-26T23:49:21.951313+0000","last_clean":"2024-01-26T23:49:21.951313+0000","last_became_active":"2024-01-26T23:49:20.940914+0000","last_became_peered":"2024-01-26T23:49:20.940914+0000","last_unstale":"2024-01-26T23:49:21.951313+0000","last_undegraded":"2024-01-26T23:49:21.951313+0000","last_fullsized":"2024-01-26T23:49:21.951313+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:52:55.167098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715579+0000","last_change":"2024-01-26T23:49:21.713588+0000","last_active":"2024-01-26T23:49:21.715579+0000","last_peered":"2024-01-26T23:49:21.715579+0000","last_clean":"2024-01-26T23:49:21.715579+0000","last_became_active":"2024-01-26T23:49:21.713242+0000","last_became_peered":"2024-01-26T23:49:21.713242+0000","last_unstale":"2024-01-26T23:49:21.715579+0000","last_undegraded":"2024-01-26T23:49:21.715579+0000","last_fullsized":"2024-01-26T23:49:21.715579+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:27:10.909797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.950236+0000","last_change":"2024-01-26T23:49:20.935760+0000","last_active":"2024-01-26T23:49:21.950236+0000","last_peered":"2024-01-26T23:49:21.950236+0000","last_clean":"2024-01-26T23:49:21.950236+0000","last_became_active":"2024-01-26T23:49:20.935522+0000","last_became_peered":"2024-01-26T23:49:20.935522+0000","last_unstale":"2024-01-26T23:49:21.950236+0000","last_undegraded":"2024-01-26T23:49:21.950236+0000","last_fullsized":"2024-01-26T23:49:21.950236+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:46:35.436482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715276+0000","last_change":"2024-01-26T23:49:21.711623+0000","last_active":"2024-01-26T23:49:21.715276+0000","last_peered":"2024-01-26T23:49:21.715276+0000","last_clean":"2024-01-26T23:49:21.715276+0000","last_became_active":"2024-01-26T23:49:21.711237+0000","last_became_peered":"2024-01-26T23:49:21.711237+0000","last_unstale":"2024-01-26T23:49:21.715276+0000","last_undegraded":"2024-01-26T23:49:21.715276+0000","last_fullsized":"2024-01-26T23:49:21.715276+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:14:30.475748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.950458+0000","last_change":"2024-01-26T23:49:20.937373+0000","last_active":"2024-01-26T23:49:21.950458+0000","last_peered":"2024-01-26T23:49:21.950458+0000","last_clean":"2024-01-26T23:49:21.950458+0000","last_became_active":"2024-01-26T23:49:20.937065+0000","last_became_peered":"2024-01-26T23:49:20.937065+0000","last_unstale":"2024-01-26T23:49:21.950458+0000","last_undegraded":"2024-01-26T23:49:21.950458+0000","last_fullsized":"2024-01-26T23:49:21.950458+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:49:16.191682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.702584+0000","last_change":"2024-01-26T23:49:21.699753+0000","last_active":"2024-01-26T23:49:21.702584+0000","last_peered":"2024-01-26T23:49:21.702584+0000","last_clean":"2024-01-26T23:49:21.702584+0000","last_became_active":"2024-01-26T23:49:21.699464+0000","last_became_peered":"2024-01-26T23:49:21.699464+0000","last_unstale":"2024-01-26T23:49:21.702584+0000","last_undegraded":"2024-01-26T23:49:21.702584+0000","last_fullsized":"2024-01-26T23:49:21.702584+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:03.507801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.695979+0000","last_change":"2024-01-26T23:49:20.928944+0000","last_active":"2024-01-26T23:49:21.695979+0000","last_peered":"2024-01-26T23:49:21.695979+0000","last_clean":"2024-01-26T23:49:21.695979+0000","last_became_active":"2024-01-26T23:49:20.928649+0000","last_became_peered":"2024-01-26T23:49:20.928649+0000","last_unstale":"2024-01-26T23:49:21.695979+0000","last_undegraded":"2024-01-26T23:49:21.695979+0000","last_fullsized":"2024-01-26T23:49:21.695979+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:55:50.467126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.702548+0000","last_change":"2024-01-26T23:49:21.699714+0000","last_active":"2024-01-26T23:49:21.702548+0000","last_peered":"2024-01-26T23:49:21.702548+0000","last_clean":"2024-01-26T23:49:21.702548+0000","last_became_active":"2024-01-26T23:49:21.699419+0000","last_became_peered":"2024-01-26T23:49:21.699419+0000","last_unstale":"2024-01-26T23:49:21.702548+0000","last_undegraded":"2024-01-26T23:49:21.702548+0000","last_fullsized":"2024-01-26T23:49:21.702548+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:47:59.288036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.696916+0000","last_change":"2024-01-26T23:49:20.927169+0000","last_active":"2024-01-26T23:49:21.696916+0000","last_peered":"2024-01-26T23:49:21.696916+0000","last_clean":"2024-01-26T23:49:21.696916+0000","last_became_active":"2024-01-26T23:49:20.926865+0000","last_became_peered":"2024-01-26T23:49:20.926865+0000","last_unstale":"2024-01-26T23:49:21.696916+0000","last_undegraded":"2024-01-26T23:49:21.696916+0000","last_fullsized":"2024-01-26T23:49:21.696916+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:57:52.726572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715082+0000","last_change":"2024-01-26T23:49:21.708432+0000","last_active":"2024-01-26T23:49:21.715082+0000","last_peered":"2024-01-26T23:49:21.715082+0000","last_clean":"2024-01-26T23:49:21.715082+0000","last_became_active":"2024-01-26T23:49:21.708100+0000","last_became_peered":"2024-01-26T23:49:21.708100+0000","last_unstale":"2024-01-26T23:49:21.715082+0000","last_undegraded":"2024-01-26T23:49:21.715082+0000","last_fullsized":"2024-01-26T23:49:21.715082+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:31:18.118068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.707105+0000","last_change":"2024-01-26T23:49:20.928442+0000","last_active":"2024-01-26T23:49:21.707105+0000","last_peered":"2024-01-26T23:49:21.707105+0000","last_clean":"2024-01-26T23:49:21.707105+0000","last_became_active":"2024-01-26T23:49:20.928198+0000","last_became_peered":"2024-01-26T23:49:20.928198+0000","last_unstale":"2024-01-26T23:49:21.707105+0000","last_undegraded":"2024-01-26T23:49:21.707105+0000","last_fullsized":"2024-01-26T23:49:21.707105+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:33.439967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.710927+0000","last_change":"2024-01-26T23:49:21.702605+0000","last_active":"2024-01-26T23:49:21.710927+0000","last_peered":"2024-01-26T23:49:21.710927+0000","last_clean":"2024-01-26T23:49:21.710927+0000","last_became_active":"2024-01-26T23:49:21.702345+0000","last_became_peered":"2024-01-26T23:49:21.702345+0000","last_unstale":"2024-01-26T23:49:21.710927+0000","last_undegraded":"2024-01-26T23:49:21.710927+0000","last_fullsized":"2024-01-26T23:49:21.710927+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:00:37.977251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.696186+0000","last_change":"2024-01-26T23:49:20.933290+0000","last_active":"2024-01-26T23:49:21.696186+0000","last_peered":"2024-01-26T23:49:21.696186+0000","last_clean":"2024-01-26T23:49:21.696186+0000","last_became_active":"2024-01-26T23:49:20.932911+0000","last_became_peered":"2024-01-26T23:49:20.932911+0000","last_unstale":"2024-01-26T23:49:21.696186+0000","last_undegraded":"2024-01-26T23:49:21.696186+0000","last_fullsized":"2024-01-26T23:49:21.696186+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:06:47.740099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713595+0000","last_change":"2024-01-26T23:49:21.710063+0000","last_active":"2024-01-26T23:49:21.713595+0000","last_peered":"2024-01-26T23:49:21.713595+0000","last_clean":"2024-01-26T23:49:21.713595+0000","last_became_active":"2024-01-26T23:49:21.709757+0000","last_became_peered":"2024-01-26T23:49:21.709757+0000","last_unstale":"2024-01-26T23:49:21.713595+0000","last_undegraded":"2024-01-26T23:49:21.713595+0000","last_fullsized":"2024-01-26T23:49:21.713595+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:47:59.288036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"144'3","reported_seq":13,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.950273+0000","last_change":"2024-01-26T23:49:20.926313+0000","last_active":"2024-01-26T23:49:21.950273+0000","last_peered":"2024-01-26T23:49:21.950273+0000","last_clean":"2024-01-26T23:49:21.950273+0000","last_became_active":"2024-01-26T23:49:20.926067+0000","last_became_peered":"2024-01-26T23:49:20.926067+0000","last_unstale":"2024-01-26T23:49:21.950273+0000","last_undegraded":"2024-01-26T23:49:21.950273+0000","last_fullsized":"2024-01-26T23:49:21.950273+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:09:02.526081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714584+0000","last_change":"2024-01-26T23:49:21.709992+0000","last_active":"2024-01-26T23:49:21.714584+0000","last_peered":"2024-01-26T23:49:21.714584+0000","last_clean":"2024-01-26T23:49:21.714584+0000","last_became_active":"2024-01-26T23:49:21.709742+0000","last_became_peered":"2024-01-26T23:49:21.709742+0000","last_unstale":"2024-01-26T23:49:21.714584+0000","last_undegraded":"2024-01-26T23:49:21.714584+0000","last_fullsized":"2024-01-26T23:49:21.714584+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:39:24.864123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"144'6","reported_seq":16,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.955758+0000","last_change":"2024-01-26T23:49:20.932889+0000","last_active":"2024-01-26T23:49:21.955758+0000","last_peered":"2024-01-26T23:49:21.955758+0000","last_clean":"2024-01-26T23:49:21.955758+0000","last_became_active":"2024-01-26T23:49:20.932592+0000","last_became_peered":"2024-01-26T23:49:20.932592+0000","last_unstale":"2024-01-26T23:49:21.955758+0000","last_undegraded":"2024-01-26T23:49:21.955758+0000","last_fullsized":"2024-01-26T23:49:21.955758+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:16.483135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.718948+0000","last_change":"2024-01-26T23:49:21.714417+0000","last_active":"2024-01-26T23:49:21.718948+0000","last_peered":"2024-01-26T23:49:21.718948+0000","last_clean":"2024-01-26T23:49:21.718948+0000","last_became_active":"2024-01-26T23:49:21.714128+0000","last_became_peered":"2024-01-26T23:49:21.714128+0000","last_unstale":"2024-01-26T23:49:21.718948+0000","last_undegraded":"2024-01-26T23:49:21.718948+0000","last_fullsized":"2024-01-26T23:49:21.718948+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:46:36.438311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.952459+0000","last_change":"2024-01-26T23:49:20.934520+0000","last_active":"2024-01-26T23:49:21.952459+0000","last_peered":"2024-01-26T23:49:21.952459+0000","last_clean":"2024-01-26T23:49:21.952459+0000","last_became_active":"2024-01-26T23:49:20.933872+0000","last_became_peered":"2024-01-26T23:49:20.933872+0000","last_unstale":"2024-01-26T23:49:21.952459+0000","last_undegraded":"2024-01-26T23:49:21.952459+0000","last_fullsized":"2024-01-26T23:49:21.952459+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:39:31.988243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.708179+0000","last_change":"2024-01-26T23:49:21.705738+0000","last_active":"2024-01-26T23:49:21.708179+0000","last_peered":"2024-01-26T23:49:21.708179+0000","last_clean":"2024-01-26T23:49:21.708179+0000","last_became_active":"2024-01-26T23:49:21.705447+0000","last_became_peered":"2024-01-26T23:49:21.705447+0000","last_unstale":"2024-01-26T23:49:21.708179+0000","last_undegraded":"2024-01-26T23:49:21.708179+0000","last_fullsized":"2024-01-26T23:49:21.708179+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:43:20.056916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.697347+0000","last_change":"2024-01-26T23:49:20.925959+0000","last_active":"2024-01-26T23:49:21.697347+0000","last_peered":"2024-01-26T23:49:21.697347+0000","last_clean":"2024-01-26T23:49:21.697347+0000","last_became_active":"2024-01-26T23:49:20.925600+0000","last_became_peered":"2024-01-26T23:49:20.925600+0000","last_unstale":"2024-01-26T23:49:21.697347+0000","last_undegraded":"2024-01-26T23:49:21.697347+0000","last_fullsized":"2024-01-26T23:49:21.697347+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:55:22.165473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714932+0000","last_change":"2024-01-26T23:49:21.713422+0000","last_active":"2024-01-26T23:49:21.714932+0000","last_peered":"2024-01-26T23:49:21.714932+0000","last_clean":"2024-01-26T23:49:21.714932+0000","last_became_active":"2024-01-26T23:49:21.713129+0000","last_became_peered":"2024-01-26T23:49:21.713129+0000","last_unstale":"2024-01-26T23:49:21.714932+0000","last_undegraded":"2024-01-26T23:49:21.714932+0000","last_fullsized":"2024-01-26T23:49:21.714932+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:36:43.200780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.696779+0000","last_change":"2024-01-26T23:49:20.926498+0000","last_active":"2024-01-26T23:49:21.696779+0000","last_peered":"2024-01-26T23:49:21.696779+0000","last_clean":"2024-01-26T23:49:21.696779+0000","last_became_active":"2024-01-26T23:49:20.926183+0000","last_became_peered":"2024-01-26T23:49:20.926183+0000","last_unstale":"2024-01-26T23:49:21.696779+0000","last_undegraded":"2024-01-26T23:49:21.696779+0000","last_fullsized":"2024-01-26T23:49:21.696779+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:00:36.975421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714445+0000","last_change":"2024-01-26T23:49:21.708959+0000","last_active":"2024-01-26T23:49:21.714445+0000","last_peered":"2024-01-26T23:49:21.714445+0000","last_clean":"2024-01-26T23:49:21.714445+0000","last_became_active":"2024-01-26T23:49:21.708693+0000","last_became_peered":"2024-01-26T23:49:21.708693+0000","last_unstale":"2024-01-26T23:49:21.714445+0000","last_undegraded":"2024-01-26T23:49:21.714445+0000","last_fullsized":"2024-01-26T23:49:21.714445+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:39:32.990073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.948408+0000","last_change":"2024-01-26T23:49:20.934522+0000","last_active":"2024-01-26T23:49:21.948408+0000","last_peered":"2024-01-26T23:49:21.948408+0000","last_clean":"2024-01-26T23:49:21.948408+0000","last_became_active":"2024-01-26T23:49:20.934219+0000","last_became_peered":"2024-01-26T23:49:20.934219+0000","last_unstale":"2024-01-26T23:49:21.948408+0000","last_undegraded":"2024-01-26T23:49:21.948408+0000","last_fullsized":"2024-01-26T23:49:21.948408+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:31:23.721288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720614+0000","last_change":"2024-01-26T23:49:21.718008+0000","last_active":"2024-01-26T23:49:21.720614+0000","last_peered":"2024-01-26T23:49:21.720614+0000","last_clean":"2024-01-26T23:49:21.720614+0000","last_became_active":"2024-01-26T23:49:21.717522+0000","last_became_peered":"2024-01-26T23:49:21.717522+0000","last_unstale":"2024-01-26T23:49:21.720614+0000","last_undegraded":"2024-01-26T23:49:21.720614+0000","last_fullsized":"2024-01-26T23:49:21.720614+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:25:47.543272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.948077+0000","last_change":"2024-01-26T23:49:20.938034+0000","last_active":"2024-01-26T23:49:21.948077+0000","last_peered":"2024-01-26T23:49:21.948077+0000","last_clean":"2024-01-26T23:49:21.948077+0000","last_became_active":"2024-01-26T23:49:20.937700+0000","last_became_peered":"2024-01-26T23:49:20.937700+0000","last_unstale":"2024-01-26T23:49:21.948077+0000","last_undegraded":"2024-01-26T23:49:21.948077+0000","last_fullsized":"2024-01-26T23:49:21.948077+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:30:23.346922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715156+0000","last_change":"2024-01-26T23:49:21.711030+0000","last_active":"2024-01-26T23:49:21.715156+0000","last_peered":"2024-01-26T23:49:21.715156+0000","last_clean":"2024-01-26T23:49:21.715156+0000","last_became_active":"2024-01-26T23:49:21.710735+0000","last_became_peered":"2024-01-26T23:49:21.710735+0000","last_unstale":"2024-01-26T23:49:21.715156+0000","last_undegraded":"2024-01-26T23:49:21.715156+0000","last_fullsized":"2024-01-26T23:49:21.715156+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:22:26.000383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.950570+0000","last_change":"2024-01-26T23:49:20.932856+0000","last_active":"2024-01-26T23:49:21.950570+0000","last_peered":"2024-01-26T23:49:21.950570+0000","last_clean":"2024-01-26T23:49:21.950570+0000","last_became_active":"2024-01-26T23:49:20.932586+0000","last_became_peered":"2024-01-26T23:49:20.932586+0000","last_unstale":"2024-01-26T23:49:21.950570+0000","last_undegraded":"2024-01-26T23:49:21.950570+0000","last_fullsized":"2024-01-26T23:49:21.950570+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:41:32.548836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714568+0000","last_change":"2024-01-26T23:49:21.706806+0000","last_active":"2024-01-26T23:49:21.714568+0000","last_peered":"2024-01-26T23:49:21.714568+0000","last_clean":"2024-01-26T23:49:21.714568+0000","last_became_active":"2024-01-26T23:49:21.706309+0000","last_became_peered":"2024-01-26T23:49:21.706309+0000","last_unstale":"2024-01-26T23:49:21.714568+0000","last_undegraded":"2024-01-26T23:49:21.714568+0000","last_fullsized":"2024-01-26T23:49:21.714568+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:15.880089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.702136+0000","last_change":"2024-01-26T23:49:20.937835+0000","last_active":"2024-01-26T23:49:21.702136+0000","last_peered":"2024-01-26T23:49:21.702136+0000","last_clean":"2024-01-26T23:49:21.702136+0000","last_became_active":"2024-01-26T23:49:20.937594+0000","last_became_peered":"2024-01-26T23:49:20.937594+0000","last_unstale":"2024-01-26T23:49:21.702136+0000","last_undegraded":"2024-01-26T23:49:21.702136+0000","last_fullsized":"2024-01-26T23:49:21.702136+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:04:52.656671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714422+0000","last_change":"2024-01-26T23:49:21.709312+0000","last_active":"2024-01-26T23:49:21.714422+0000","last_peered":"2024-01-26T23:49:21.714422+0000","last_clean":"2024-01-26T23:49:21.714422+0000","last_became_active":"2024-01-26T23:49:21.709016+0000","last_became_peered":"2024-01-26T23:49:21.709016+0000","last_unstale":"2024-01-26T23:49:21.714422+0000","last_undegraded":"2024-01-26T23:49:21.714422+0000","last_fullsized":"2024-01-26T23:49:21.714422+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:15.415331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.947893+0000","last_change":"2024-01-26T23:49:20.930419+0000","last_active":"2024-01-26T23:49:21.947893+0000","last_peered":"2024-01-26T23:49:21.947893+0000","last_clean":"2024-01-26T23:49:21.947893+0000","last_became_active":"2024-01-26T23:49:20.930102+0000","last_became_peered":"2024-01-26T23:49:20.930102+0000","last_unstale":"2024-01-26T23:49:21.947893+0000","last_undegraded":"2024-01-26T23:49:21.947893+0000","last_fullsized":"2024-01-26T23:49:21.947893+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:01.796587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.709284+0000","last_change":"2024-01-26T23:49:21.703187+0000","last_active":"2024-01-26T23:49:21.709284+0000","last_peered":"2024-01-26T23:49:21.709284+0000","last_clean":"2024-01-26T23:49:21.709284+0000","last_became_active":"2024-01-26T23:49:21.702883+0000","last_became_peered":"2024-01-26T23:49:21.702883+0000","last_unstale":"2024-01-26T23:49:21.709284+0000","last_undegraded":"2024-01-26T23:49:21.709284+0000","last_fullsized":"2024-01-26T23:49:21.709284+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:23:30.759008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.701669+0000","last_change":"2024-01-26T23:49:20.931625+0000","last_active":"2024-01-26T23:49:21.701669+0000","last_peered":"2024-01-26T23:49:21.701669+0000","last_clean":"2024-01-26T23:49:21.701669+0000","last_became_active":"2024-01-26T23:49:20.930770+0000","last_became_peered":"2024-01-26T23:49:20.930770+0000","last_unstale":"2024-01-26T23:49:21.701669+0000","last_undegraded":"2024-01-26T23:49:21.701669+0000","last_fullsized":"2024-01-26T23:49:21.701669+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:05:23.745399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713219+0000","last_change":"2024-01-26T23:49:21.711158+0000","last_active":"2024-01-26T23:49:21.713219+0000","last_peered":"2024-01-26T23:49:21.713219+0000","last_clean":"2024-01-26T23:49:21.713219+0000","last_became_active":"2024-01-26T23:49:21.710828+0000","last_became_peered":"2024-01-26T23:49:21.710828+0000","last_unstale":"2024-01-26T23:49:21.713219+0000","last_undegraded":"2024-01-26T23:49:21.713219+0000","last_fullsized":"2024-01-26T23:49:21.713219+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:43:20.056916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.696741+0000","last_change":"2024-01-26T23:49:20.927120+0000","last_active":"2024-01-26T23:49:21.696741+0000","last_peered":"2024-01-26T23:49:21.696741+0000","last_clean":"2024-01-26T23:49:21.696741+0000","last_became_active":"2024-01-26T23:49:20.926800+0000","last_became_peered":"2024-01-26T23:49:20.926800+0000","last_unstale":"2024-01-26T23:49:21.696741+0000","last_undegraded":"2024-01-26T23:49:21.696741+0000","last_fullsized":"2024-01-26T23:49:21.696741+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:14.878260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714485+0000","last_change":"2024-01-26T23:49:21.709755+0000","last_active":"2024-01-26T23:49:21.714485+0000","last_peered":"2024-01-26T23:49:21.714485+0000","last_clean":"2024-01-26T23:49:21.714485+0000","last_became_active":"2024-01-26T23:49:21.709547+0000","last_became_peered":"2024-01-26T23:49:21.709547+0000","last_unstale":"2024-01-26T23:49:21.714485+0000","last_undegraded":"2024-01-26T23:49:21.714485+0000","last_fullsized":"2024-01-26T23:49:21.714485+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:46:51.230370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.701015+0000","last_change":"2024-01-26T23:49:20.933709+0000","last_active":"2024-01-26T23:49:21.701015+0000","last_peered":"2024-01-26T23:49:21.701015+0000","last_clean":"2024-01-26T23:49:21.701015+0000","last_became_active":"2024-01-26T23:49:20.933444+0000","last_became_peered":"2024-01-26T23:49:20.933444+0000","last_unstale":"2024-01-26T23:49:21.701015+0000","last_undegraded":"2024-01-26T23:49:21.701015+0000","last_fullsized":"2024-01-26T23:49:21.701015+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:41:45.760005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.711298+0000","last_change":"2024-01-26T23:49:21.704604+0000","last_active":"2024-01-26T23:49:21.711298+0000","last_peered":"2024-01-26T23:49:21.711298+0000","last_clean":"2024-01-26T23:49:21.711298+0000","last_became_active":"2024-01-26T23:49:21.704277+0000","last_became_peered":"2024-01-26T23:49:21.704277+0000","last_unstale":"2024-01-26T23:49:21.711298+0000","last_undegraded":"2024-01-26T23:49:21.711298+0000","last_fullsized":"2024-01-26T23:49:21.711298+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:01:33.387704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.951229+0000","last_change":"2024-01-26T23:49:20.926602+0000","last_active":"2024-01-26T23:49:21.951229+0000","last_peered":"2024-01-26T23:49:21.951229+0000","last_clean":"2024-01-26T23:49:21.951229+0000","last_became_active":"2024-01-26T23:49:20.926306+0000","last_became_peered":"2024-01-26T23:49:20.926306+0000","last_unstale":"2024-01-26T23:49:21.951229+0000","last_undegraded":"2024-01-26T23:49:21.951229+0000","last_fullsized":"2024-01-26T23:49:21.951229+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:58:14.912115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715403+0000","last_change":"2024-01-26T23:49:21.709775+0000","last_active":"2024-01-26T23:49:21.715403+0000","last_peered":"2024-01-26T23:49:21.715403+0000","last_clean":"2024-01-26T23:49:21.715403+0000","last_became_active":"2024-01-26T23:49:21.709484+0000","last_became_peered":"2024-01-26T23:49:21.709484+0000","last_unstale":"2024-01-26T23:49:21.715403+0000","last_undegraded":"2024-01-26T23:49:21.715403+0000","last_fullsized":"2024-01-26T23:49:21.715403+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:19:56.848512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.706846+0000","last_change":"2024-01-26T23:49:20.938849+0000","last_active":"2024-01-26T23:49:21.706846+0000","last_peered":"2024-01-26T23:49:21.706846+0000","last_clean":"2024-01-26T23:49:21.706846+0000","last_became_active":"2024-01-26T23:49:20.938634+0000","last_became_peered":"2024-01-26T23:49:20.938634+0000","last_unstale":"2024-01-26T23:49:21.706846+0000","last_undegraded":"2024-01-26T23:49:21.706846+0000","last_fullsized":"2024-01-26T23:49:21.706846+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:19:55.846683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720486+0000","last_change":"2024-01-26T23:49:21.715705+0000","last_active":"2024-01-26T23:49:21.720486+0000","last_peered":"2024-01-26T23:49:21.720486+0000","last_clean":"2024-01-26T23:49:21.720486+0000","last_became_active":"2024-01-26T23:49:21.715391+0000","last_became_peered":"2024-01-26T23:49:21.715391+0000","last_unstale":"2024-01-26T23:49:21.720486+0000","last_undegraded":"2024-01-26T23:49:21.720486+0000","last_fullsized":"2024-01-26T23:49:21.720486+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:46:51.230370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.706792+0000","last_change":"2024-01-26T23:49:20.931200+0000","last_active":"2024-01-26T23:49:21.706792+0000","last_peered":"2024-01-26T23:49:21.706792+0000","last_clean":"2024-01-26T23:49:21.706792+0000","last_became_active":"2024-01-26T23:49:20.930968+0000","last_became_peered":"2024-01-26T23:49:20.930968+0000","last_unstale":"2024-01-26T23:49:21.706792+0000","last_undegraded":"2024-01-26T23:49:21.706792+0000","last_fullsized":"2024-01-26T23:49:21.706792+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:01:51.605788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.711848+0000","last_change":"2024-01-26T23:49:21.705136+0000","last_active":"2024-01-26T23:49:21.711848+0000","last_peered":"2024-01-26T23:49:21.711848+0000","last_clean":"2024-01-26T23:49:21.711848+0000","last_became_active":"2024-01-26T23:49:21.704849+0000","last_became_peered":"2024-01-26T23:49:21.704849+0000","last_unstale":"2024-01-26T23:49:21.711848+0000","last_undegraded":"2024-01-26T23:49:21.711848+0000","last_fullsized":"2024-01-26T23:49:21.711848+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:54.174596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.696843+0000","last_change":"2024-01-26T23:49:20.935623+0000","last_active":"2024-01-26T23:49:21.696843+0000","last_peered":"2024-01-26T23:49:21.696843+0000","last_clean":"2024-01-26T23:49:21.696843+0000","last_became_active":"2024-01-26T23:49:20.935325+0000","last_became_peered":"2024-01-26T23:49:20.935325+0000","last_unstale":"2024-01-26T23:49:21.696843+0000","last_undegraded":"2024-01-26T23:49:21.696843+0000","last_fullsized":"2024-01-26T23:49:21.696843+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:29:03.326840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716154+0000","last_change":"2024-01-26T23:49:21.713771+0000","last_active":"2024-01-26T23:49:21.716154+0000","last_peered":"2024-01-26T23:49:21.716154+0000","last_clean":"2024-01-26T23:49:21.716154+0000","last_became_active":"2024-01-26T23:49:21.713471+0000","last_became_peered":"2024-01-26T23:49:21.713471+0000","last_unstale":"2024-01-26T23:49:21.716154+0000","last_undegraded":"2024-01-26T23:49:21.716154+0000","last_fullsized":"2024-01-26T23:49:21.716154+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:46:24.616619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.702410+0000","last_change":"2024-01-26T23:49:20.933716+0000","last_active":"2024-01-26T23:49:21.702410+0000","last_peered":"2024-01-26T23:49:21.702410+0000","last_clean":"2024-01-26T23:49:21.702410+0000","last_became_active":"2024-01-26T23:49:20.933447+0000","last_became_peered":"2024-01-26T23:49:20.933447+0000","last_unstale":"2024-01-26T23:49:21.702410+0000","last_undegraded":"2024-01-26T23:49:21.702410+0000","last_fullsized":"2024-01-26T23:49:21.702410+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:44:37.740189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.712186+0000","last_change":"2024-01-26T23:49:21.709903+0000","last_active":"2024-01-26T23:49:21.712186+0000","last_peered":"2024-01-26T23:49:21.712186+0000","last_clean":"2024-01-26T23:49:21.712186+0000","last_became_active":"2024-01-26T23:49:21.709608+0000","last_became_peered":"2024-01-26T23:49:21.709608+0000","last_unstale":"2024-01-26T23:49:21.712186+0000","last_undegraded":"2024-01-26T23:49:21.712186+0000","last_fullsized":"2024-01-26T23:49:21.712186+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:44:38.742019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.707040+0000","last_change":"2024-01-26T23:49:20.928339+0000","last_active":"2024-01-26T23:49:21.707040+0000","last_peered":"2024-01-26T23:49:21.707040+0000","last_clean":"2024-01-26T23:49:21.707040+0000","last_became_active":"2024-01-26T23:49:20.927842+0000","last_became_peered":"2024-01-26T23:49:20.927842+0000","last_unstale":"2024-01-26T23:49:21.707040+0000","last_undegraded":"2024-01-26T23:49:21.707040+0000","last_fullsized":"2024-01-26T23:49:21.707040+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:43:19.055087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713297+0000","last_change":"2024-01-26T23:49:21.709300+0000","last_active":"2024-01-26T23:49:21.713297+0000","last_peered":"2024-01-26T23:49:21.713297+0000","last_clean":"2024-01-26T23:49:21.713297+0000","last_became_active":"2024-01-26T23:49:21.709008+0000","last_became_peered":"2024-01-26T23:49:21.709008+0000","last_unstale":"2024-01-26T23:49:21.713297+0000","last_undegraded":"2024-01-26T23:49:21.713297+0000","last_fullsized":"2024-01-26T23:49:21.713297+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:34.441797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.694324+0000","last_change":"2024-01-26T23:49:20.935691+0000","last_active":"2024-01-26T23:49:21.694324+0000","last_peered":"2024-01-26T23:49:21.694324+0000","last_clean":"2024-01-26T23:49:21.694324+0000","last_became_active":"2024-01-26T23:49:20.935362+0000","last_became_peered":"2024-01-26T23:49:20.935362+0000","last_unstale":"2024-01-26T23:49:21.694324+0000","last_undegraded":"2024-01-26T23:49:21.694324+0000","last_fullsized":"2024-01-26T23:49:21.694324+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:56:22.957242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.708198+0000","last_change":"2024-01-26T23:49:21.705845+0000","last_active":"2024-01-26T23:49:21.708198+0000","last_peered":"2024-01-26T23:49:21.708198+0000","last_clean":"2024-01-26T23:49:21.708198+0000","last_became_active":"2024-01-26T23:49:21.705559+0000","last_became_peered":"2024-01-26T23:49:21.705559+0000","last_unstale":"2024-01-26T23:49:21.708198+0000","last_undegraded":"2024-01-26T23:49:21.708198+0000","last_fullsized":"2024-01-26T23:49:21.708198+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:00:34.207070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.701020+0000","last_change":"2024-01-26T23:49:20.931628+0000","last_active":"2024-01-26T23:49:21.701020+0000","last_peered":"2024-01-26T23:49:21.701020+0000","last_clean":"2024-01-26T23:49:21.701020+0000","last_became_active":"2024-01-26T23:49:20.931022+0000","last_became_peered":"2024-01-26T23:49:20.931022+0000","last_unstale":"2024-01-26T23:49:21.701020+0000","last_undegraded":"2024-01-26T23:49:21.701020+0000","last_fullsized":"2024-01-26T23:49:21.701020+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:28:33.436885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.717191+0000","last_change":"2024-01-26T23:49:21.713146+0000","last_active":"2024-01-26T23:49:21.717191+0000","last_peered":"2024-01-26T23:49:21.717191+0000","last_clean":"2024-01-26T23:49:21.717191+0000","last_became_active":"2024-01-26T23:49:21.712684+0000","last_became_peered":"2024-01-26T23:49:21.712684+0000","last_unstale":"2024-01-26T23:49:21.717191+0000","last_undegraded":"2024-01-26T23:49:21.717191+0000","last_fullsized":"2024-01-26T23:49:21.717191+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:51.588887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"144'8","reported_seq":18,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.950934+0000","last_change":"2024-01-26T23:49:20.933578+0000","last_active":"2024-01-26T23:49:21.950934+0000","last_peered":"2024-01-26T23:49:21.950934+0000","last_clean":"2024-01-26T23:49:21.950934+0000","last_became_active":"2024-01-26T23:49:20.933286+0000","last_became_peered":"2024-01-26T23:49:20.933286+0000","last_unstale":"2024-01-26T23:49:21.950934+0000","last_undegraded":"2024-01-26T23:49:21.950934+0000","last_fullsized":"2024-01-26T23:49:21.950934+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:52:55.167098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"135'8584","reported_seq":9106,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.701211+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:49:21.701211+0000","last_peered":"2024-01-26T23:49:21.701211+0000","last_clean":"2024-01-26T23:49:21.701211+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:49:21.701211+0000","last_undegraded":"2024-01-26T23:49:21.701211+0000","last_fullsized":"2024-01-26T23:49:21.701211+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":2314,"num_write":360,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":2314,"num_write":360,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8584,"ondisk_log_size":8584,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575083,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311932,"kb_used_data":2356,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059908,"statfs":{"total":96636764160,"available":96317345792,"internally_reserved":0,"allocated":2412544,"data_stored":1521090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51500000000000001,"15min":0.56200000000000006},"min":{"1min":0.40699999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.74199999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52700000000000002,"15min":0.57599999999999996},"min":{"1min":0.35899999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67200000000000004,"5min":0.82499999999999996,"15min":0.83999999999999997},"last":0.57799999999999996}]},{"osd":1,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54000000000000004,"15min":0.57399999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.77600000000000002,"5min":0.77800000000000002,"15min":0.82899999999999996},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55100000000000005,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.70399999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.71299999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.57199999999999995},"min":{"1min":0.38700000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69199999999999995,"5min":0.80000000000000004,"15min":0.81699999999999995},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.38900000000000001,"5min":0.30199999999999999,"15min":0.27400000000000002},"max":{"1min":0.746,"5min":0.81999999999999995,"15min":0.88600000000000001},"last":0.502}]},{"osd":3,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54700000000000004,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.377,"15min":0.377},"max":{"1min":0.70399999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57499999999999996,"15min":0.59699999999999998},"min":{"1min":0.34399999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77900000000000003,"5min":0.86199999999999999,"15min":0.86399999999999999},"last":0.69299999999999995}]},{"osd":4,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.438,"15min":0.45200000000000001},"min":{"1min":0.34599999999999997,"5min":0.252,"15min":0.247},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.93899999999999995},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.45300000000000001,"15min":0.47399999999999998},"min":{"1min":0.32400000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67300000000000004,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.47299999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47599999999999998,"15min":0.48699999999999999},"min":{"1min":0.30499999999999999,"5min":0.27200000000000002,"15min":0.22800000000000001},"max":{"1min":0.749,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.504,"15min":0.52300000000000002},"min":{"1min":0.35799999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73399999999999999,"5min":0.74199999999999999,"15min":0.90200000000000002},"last":0.67000000000000004}]},{"osd":6,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53100000000000003,"15min":0.54800000000000004},"min":{"1min":0.41699999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76400000000000001,"5min":0.78700000000000003,"15min":0.95899999999999996},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.52700000000000002,"15min":0.54000000000000004},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75900000000000001,"5min":0.79500000000000004,"15min":0.85199999999999998},"last":0.621}]}]},{"osd":6,"up_from":13,"seq":55834575083,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303148,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068692,"statfs":{"total":96636764160,"available":96326340608,"internally_reserved":0,"allocated":1806336,"data_stored":930528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52900000000000003,"15min":0.56499999999999995},"min":{"1min":0.40300000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.423,"15min":1.423},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.57899999999999996},"min":{"1min":0.36099999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.91100000000000003},"last":0.71999999999999997}]},{"osd":1,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51700000000000002,"15min":0.55800000000000005},"min":{"1min":0.32200000000000001,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.58399999999999996},"min":{"1min":0.38200000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.66800000000000004,"5min":1.498,"15min":1.498},"last":0.68100000000000005}]},{"osd":2,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52800000000000002,"15min":0.56999999999999995},"min":{"1min":0.38100000000000001,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.85799999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52600000000000002,"15min":0.57099999999999995},"min":{"1min":0.377,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69299999999999995,"5min":0.77000000000000002,"15min":0.82799999999999996},"last":0.69999999999999996}]},{"osd":3,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57199999999999995,"15min":0.59099999999999997},"min":{"1min":0.34699999999999998,"5min":0.33800000000000002,"15min":0.29999999999999999},"max":{"1min":0.746,"5min":1.595,"15min":1.595},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56899999999999995,"15min":0.60599999999999998},"min":{"1min":0.45600000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.751,"5min":1.369,"15min":1.369},"last":0.63700000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.443,"15min":0.46200000000000002},"min":{"1min":0.33800000000000002,"5min":0.245,"15min":0.22},"max":{"1min":0.67000000000000004,"5min":1.556,"15min":1.556},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.47399999999999998,"15min":0.48199999999999998},"min":{"1min":0.33200000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.86199999999999999},"last":0.40400000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45500000000000002,"15min":0.47099999999999997},"min":{"1min":0.32700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67900000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.433},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.51700000000000002,"15min":0.52000000000000002},"min":{"1min":0.30499999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.70999999999999996,"5min":1.47,"15min":1.47},"last":0.45800000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.5,"15min":0.51800000000000002},"min":{"1min":0.40100000000000002,"5min":0.26700000000000002,"15min":0.246},"max":{"1min":0.80200000000000005,"5min":0.81799999999999995,"15min":0.85899999999999999},"last":0.505},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50900000000000001,"15min":0.52400000000000002},"min":{"1min":0.32300000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.80400000000000005},"last":0.53000000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575083,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307244,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064596,"statfs":{"total":96636764160,"available":96322146304,"internally_reserved":0,"allocated":1806336,"data_stored":929996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41199999999999998,"15min":0.40600000000000003},"min":{"1min":0.28100000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.72599999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.41999999999999998,"15min":0.439},"min":{"1min":0.28100000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.61499999999999999,"5min":0.78600000000000003,"15min":1.337},"last":0.38300000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.39700000000000002,"15min":0.40899999999999997},"min":{"1min":0.28699999999999998,"5min":0.254,"15min":0.23599999999999999},"max":{"1min":0.66300000000000003,"5min":0.68799999999999994,"15min":0.73299999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.46300000000000002,"15min":0.47199999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59299999999999997,"5min":0.74099999999999999,"15min":0.77300000000000002},"last":0.41199999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51300000000000001,"15min":0.51400000000000001},"min":{"1min":0.38400000000000001,"5min":0.34399999999999997,"15min":0.33200000000000002},"max":{"1min":0.73499999999999999,"5min":0.78900000000000003,"15min":1.782},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.32900000000000001,"5min":0.32000000000000001,"15min":0.26600000000000001},"max":{"1min":0.70199999999999996,"5min":0.71499999999999997,"15min":1.379},"last":0.48999999999999999}]},{"osd":4,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60699999999999998,"15min":0.64200000000000002},"min":{"1min":0.45100000000000001,"5min":0.40799999999999997,"15min":0.40200000000000002},"max":{"1min":0.85199999999999998,"5min":0.89200000000000002,"15min":1.0820000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.64500000000000002,"15min":0.65700000000000003},"min":{"1min":0.49199999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.57899999999999996}]},{"osd":5,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.64000000000000001,"15min":0.66800000000000004},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45000000000000001},"max":{"1min":0.86499999999999999,"5min":0.91100000000000003,"15min":1.208},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.63400000000000001,"15min":0.65900000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.42499999999999999},"max":{"1min":0.85099999999999998,"5min":0.96399999999999997,"15min":1.018},"last":0.56499999999999995}]},{"osd":6,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.63800000000000001,"15min":0.67000000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.42599999999999999},"max":{"1min":0.97399999999999998,"5min":1.022,"15min":1.2529999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.65000000000000002,"15min":0.67700000000000005},"min":{"1min":0.47899999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92300000000000004,"5min":0.93200000000000005,"15min":1.022},"last":0.63400000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.68300000000000005,"15min":0.70699999999999996},"min":{"1min":0.5,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.747},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.71899999999999997},"min":{"1min":0.505,"5min":0.496,"15min":0.496},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.296},"last":0.69799999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575082,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307824,"kb_used_data":2344,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064016,"statfs":{"total":96636764160,"available":96321552384,"internally_reserved":0,"allocated":2400256,"data_stored":1520364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.40600000000000003},"min":{"1min":0.25800000000000001,"5min":0.23899999999999999,"15min":0.23100000000000001},"max":{"1min":0.88500000000000001,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.42999999999999999,"15min":0.45100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65800000000000003,"5min":1.363,"15min":1.4490000000000001},"last":0.308}]},{"osd":2,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.41099999999999998,"15min":0.42299999999999999},"min":{"1min":0.28899999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.89600000000000002},"last":0.371},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.33900000000000002,"5min":0.32700000000000001,"15min":0.28000000000000003},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.41199999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.49199999999999999,"15min":0.47399999999999998},"min":{"1min":0.32800000000000001,"5min":0.32300000000000001,"15min":0.30599999999999999},"max":{"1min":0.85199999999999998,"5min":2.145,"15min":2.145},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36099999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":1.198},"last":0.501}]},{"osd":4,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60899999999999999,"15min":0.63800000000000001},"min":{"1min":0.41699999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78600000000000003,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.59699999999999998,"15min":0.63700000000000001},"min":{"1min":0.44400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.94299999999999995},"last":0.68200000000000005}]},{"osd":5,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":1.329,"15min":1.329},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.66300000000000003,"15min":0.67000000000000004},"min":{"1min":0.46899999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82799999999999996,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.623}]},{"osd":6,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.61799999999999999,"15min":0.65100000000000002},"min":{"1min":0.47299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.89800000000000002,"5min":0.98699999999999999,"15min":1.032},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.65400000000000003,"15min":0.68000000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.41799999999999998},"max":{"1min":0.93100000000000005,"5min":1.286,"15min":1.286},"last":0.66000000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.70599999999999996,"15min":0.81699999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.435},"max":{"1min":0.81799999999999995,"5min":4.8710000000000004,"15min":25.965},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.73899999999999999,"15min":0.86099999999999999},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.433},"max":{"1min":0.88200000000000001,"5min":4.9199999999999999,"15min":33.085000000000001},"last":0.67200000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575082,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307240,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064600,"statfs":{"total":96636764160,"available":96322150400,"internally_reserved":0,"allocated":1802240,"data_stored":929974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.379,"15min":0.39100000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.255},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.375},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.43099999999999999,"15min":0.42599999999999999},"min":{"1min":0.29699999999999999,"5min":0.255,"15min":0.255},"max":{"1min":0.59899999999999998,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.48799999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.435,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.26900000000000002,"15min":0.248},"max":{"1min":0.63500000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52200000000000002,"15min":0.51100000000000001},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.68300000000000005,"5min":1.171,"15min":1.171},"last":0.52200000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.46899999999999997,"15min":0.48399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.313},"max":{"1min":0.64300000000000002,"5min":0.72799999999999998,"15min":0.74099999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47699999999999998},"min":{"1min":0.32900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70599999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.55900000000000005}]},{"osd":4,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59599999999999997,"15min":0.628},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.087},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.625,"15min":0.64300000000000002},"min":{"1min":0.39100000000000001,"5min":0.371,"15min":0.371},"max":{"1min":1.006,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.83199999999999996}]},{"osd":5,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.63900000000000001,"15min":0.65500000000000003},"min":{"1min":0.42299999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94699999999999995,"5min":1.24,"15min":1.24},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.65200000000000002,"15min":0.67400000000000004},"min":{"1min":0.48799999999999999,"5min":0.41799999999999998,"15min":0.35199999999999998},"max":{"1min":0.874,"5min":0.97699999999999998,"15min":1.085},"last":0.74099999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65700000000000003,"15min":0.67700000000000005},"min":{"1min":0.41199999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88600000000000001,"5min":1.022,"15min":1.1080000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.65200000000000002,"15min":0.67800000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.38800000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.1299999999999999},"last":0.76700000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68400000000000005,"15min":0.70099999999999996},"min":{"1min":0.46400000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.88800000000000001,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.69699999999999995,"15min":0.72499999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.42099999999999999},"max":{"1min":0.96899999999999997,"5min":1.05,"15min":1.139},"last":0.79200000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575081,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307256,"kb_used_data":1776,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064584,"statfs":{"total":96636764160,"available":96322134016,"internally_reserved":0,"allocated":1818624,"data_stored":931544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38400000000000001,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.55400000000000005,"5min":0.59699999999999998,"15min":1.8540000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.42899999999999999,"15min":0.438},"min":{"1min":0.29799999999999999,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":1.2290000000000001},"last":0.30099999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44500000000000001,"15min":0.45200000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":1.633},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.47999999999999998,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":1.0760000000000001},"last":0.38200000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48399999999999999,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65000000000000002,"5min":0.70399999999999996,"15min":1.4059999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.44500000000000001,"15min":0.45800000000000002},"min":{"1min":0.34100000000000003,"5min":0.29499999999999998,"15min":0.28299999999999997},"max":{"1min":0.629,"5min":0.69399999999999995,"15min":1.1870000000000001},"last":0.36699999999999999}]},{"osd":4,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.63,"15min":0.65800000000000003},"min":{"1min":0.48299999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":3.306},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.44800000000000001,"5min":0.40100000000000002,"15min":0.379},"max":{"1min":0.81499999999999995,"5min":0.85199999999999998,"15min":1.008},"last":0.504}]},{"osd":5,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.63600000000000001,"15min":0.65200000000000002},"min":{"1min":0.48599999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.85199999999999998,"5min":0.90400000000000003,"15min":1.593},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.47599999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.86299999999999999,"5min":0.92300000000000004,"15min":1.3819999999999999},"last":0.47599999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.66900000000000004},"min":{"1min":0.42299999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":1.506},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67200000000000004,"15min":0.68999999999999995},"min":{"1min":0.55600000000000005,"5min":0.46700000000000003,"15min":0.44400000000000001},"max":{"1min":0.88500000000000001,"5min":0.89000000000000001,"15min":1.427},"last":0.57899999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64400000000000002,"15min":0.68799999999999994},"min":{"1min":0.46700000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":3.238},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.71199999999999997,"15min":0.73399999999999999},"min":{"1min":0.54200000000000004,"5min":0.51900000000000002,"15min":0.48899999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":3.351},"last":0.52300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575083,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303156,"kb_used_data":1772,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1814528,"data_stored":931510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53400000000000003,"15min":0.56799999999999995},"min":{"1min":0.34799999999999998,"5min":0.28999999999999998,"15min":0.28699999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":1.21},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58399999999999996,"15min":0.58299999999999996},"min":{"1min":0.377,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.81499999999999995,"5min":5.367,"15min":5.367},"last":0.57399999999999995}]},{"osd":1,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56899999999999995,"15min":0.58699999999999997},"min":{"1min":0.378,"5min":0.311,"15min":0.311},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.54700000000000004,"15min":0.57399999999999995},"min":{"1min":0.40699999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79800000000000004,"5min":0.80600000000000005,"15min":0.81699999999999995},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.54600000000000004,"15min":0.56599999999999995},"min":{"1min":0.39300000000000002,"5min":0.33900000000000002,"15min":0.32700000000000001},"max":{"1min":0.747,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.57699999999999996,"15min":0.59599999999999997},"min":{"1min":0.43099999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":1.7050000000000001},"last":0.47599999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57299999999999995,"15min":0.58599999999999997},"min":{"1min":0.44900000000000001,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.35699999999999998},"max":{"1min":0.82499999999999996,"5min":0.94199999999999995,"15min":1.05},"last":0.54400000000000004}]},{"osd":5,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.46500000000000002},"min":{"1min":0.31,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.48099999999999998,"15min":0.48399999999999999},"min":{"1min":0.40100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.495}]},{"osd":6,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.46500000000000002,"15min":0.47399999999999998},"min":{"1min":0.34999999999999998,"5min":0.28599999999999998,"15min":0.27200000000000002},"max":{"1min":0.76600000000000001,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.42299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.52600000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.52600000000000002,"15min":0.54000000000000004},"min":{"1min":0.35599999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.56200000000000006,"15min":0.56899999999999995},"min":{"1min":0.436,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.88800000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.58799999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575083,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1768,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1810432,"data_stored":930134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.56399999999999995},"min":{"1min":0.35199999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.71099999999999997,"5min":0.81799999999999995,"15min":2.0289999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52700000000000002,"15min":0.56699999999999995},"min":{"1min":0.41999999999999998,"5min":0.29999999999999999,"15min":0.28899999999999998},"max":{"1min":0.70799999999999996,"5min":0.77400000000000002,"15min":0.82199999999999995},"last":0.65100000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56299999999999994,"15min":0.60399999999999998},"min":{"1min":0.443,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":0.754,"5min":0.79500000000000004,"15min":2.3319999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59899999999999998,"15min":0.626},"min":{"1min":0.45800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76400000000000001,"5min":0.80500000000000005,"15min":1.1739999999999999},"last":0.83499999999999996}]},{"osd":2,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53700000000000003,"15min":0.55700000000000005},"min":{"1min":0.36799999999999999,"5min":0.34499999999999997,"15min":0.30099999999999999},"max":{"1min":0.68400000000000005,"5min":0.79400000000000004,"15min":0.83799999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55300000000000005,"15min":0.58699999999999997},"min":{"1min":0.41299999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72899999999999998,"5min":0.81100000000000005,"15min":2.0600000000000001},"last":0.74299999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55300000000000005,"15min":0.57499999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.314},"max":{"1min":0.69699999999999995,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55900000000000005,"15min":0.58699999999999997},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73999999999999999,"5min":0.77900000000000003,"15min":2.5630000000000002},"last":0.70999999999999996}]},{"osd":4,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.42899999999999999,"15min":0.45500000000000002},"min":{"1min":0.313,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.65100000000000002,"5min":0.67200000000000004,"15min":2.3700000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45900000000000002,"15min":0.48099999999999998},"min":{"1min":0.32600000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.68200000000000005,"5min":0.82699999999999996,"15min":0.89200000000000002},"last":0.46999999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.47199999999999998},"min":{"1min":0.34300000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66900000000000004,"5min":0.67700000000000005,"15min":2.5990000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.49299999999999999,"15min":0.51400000000000001},"min":{"1min":0.374,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71399999999999997,"5min":0.76200000000000001,"15min":1.99},"last":0.61399999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51400000000000001,"15min":0.53800000000000003},"min":{"1min":0.374,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73799999999999999,"5min":0.76200000000000001,"15min":3.5270000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54400000000000004,"15min":0.55900000000000005},"min":{"1min":0.43099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78900000000000003,"5min":0.81499999999999995,"15min":0.82199999999999995},"last":0.59099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:49:30.887 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:49:31.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.029+0000 7f8e0544f700 1 Processor -- start 2024-01-26T23:49:31.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.029+0000 7f8e0544f700 1 -- start start 2024-01-26T23:49:31.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.030+0000 7f8e0544f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8e000cb110 0x7f8e0006a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.030+0000 7f8e0544f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8e00159cf0 0x7f8e0006aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.030+0000 7f8e0544f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0006af60 con 0x7f8e00159920 2024-01-26T23:49:31.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.030+0000 7f8e0544f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e0006b1d0 con 0x7f8e000cb110 2024-01-26T23:49:31.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.030+0000 7f8e0544f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e0006b4b0 con 0x7f8e00159cf0 2024-01-26T23:49:31.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.030+0000 7f8dfe7fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8e000cb110 0x7f8e0006a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.030+0000 7f8dfeffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8e00159920 0x7f8e000caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39240/0 (socket says 172.21.15.46:39240) 2024-01-26T23:49:31.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.030+0000 7f8dfeffd700 1 -- 172.21.15.46:0/777519368 learned_addr learned my addr 172.21.15.46:0/777519368 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:31.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.030+0000 7f8dff7fe700 1 --2- 172.21.15.46:0/777519368 >> v2:172.21.15.106:3300/0 conn(0x7f8e00159cf0 0x7f8e0006aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.030+0000 7f8dfe7fc700 1 -- 172.21.15.46:0/777519368 >> v2:172.21.15.106:3300/0 conn(0x7f8e00159cf0 msgr2=0x7f8e0006aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:31.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.030+0000 7f8dfe7fc700 1 --2- 172.21.15.46:0/777519368 >> v2:172.21.15.106:3300/0 conn(0x7f8e00159cf0 0x7f8e0006aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.030+0000 7f8dfe7fc700 1 -- 172.21.15.46:0/777519368 >> v1:172.21.15.46:6789/0 conn(0x7f8e00159920 legacy=0x7f8e000caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:31.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.031+0000 7f8dfe7fc700 1 -- 172.21.15.46:0/777519368 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0006b790 con 0x7f8e000cb110 2024-01-26T23:49:31.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.031+0000 7f8dfdffb700 1 -- 172.21.15.46:0/777519368 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078461139 0 0) 0x7f8e0006af60 con 0x7f8e00159920 2024-01-26T23:49:31.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.031+0000 7f8dfe7fc700 1 --2- 172.21.15.46:0/777519368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e000cb110 0x7f8e0006a4e0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f8e00069420 tx=0x7f8df00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f7fbd18d8dc1c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:31.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.031+0000 7f8dfdffb700 1 -- 172.21.15.46:0/777519368 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df000b6b0 con 0x7f8e000cb110 2024-01-26T23:49:31.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.031+0000 7f8dfdffb700 1 -- 172.21.15.46:0/777519368 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df000b810 con 0x7f8e000cb110 2024-01-26T23:49:31.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.031+0000 7f8dfdffb700 1 -- 172.21.15.46:0/777519368 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df000bbb0 con 0x7f8e000cb110 2024-01-26T23:49:31.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.031+0000 7f8e0544f700 1 -- 172.21.15.46:0/777519368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e000cb110 msgr2=0x7f8e0006a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:31.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.031+0000 7f8e0544f700 1 --2- 172.21.15.46:0/777519368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e000cb110 0x7f8e0006a4e0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f8e00069420 tx=0x7f8df00091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.032+0000 7f8e0544f700 1 -- 172.21.15.46:0/777519368 shutdown_connections 2024-01-26T23:49:31.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.032+0000 7f8e0544f700 1 --2- 172.21.15.46:0/777519368 >> v2:172.21.15.106:3300/0 conn(0x7f8e00159cf0 0x7f8e0006aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.032+0000 7f8e0544f700 1 --2- 172.21.15.46:0/777519368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e000cb110 0x7f8e0006a4e0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.032+0000 7f8e0544f700 1 -- 172.21.15.46:0/777519368 >> 172.21.15.46:0/777519368 conn(0x7f8e000cd6f0 msgr2=0x7f8e000c9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:31.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.032+0000 7f8e0544f700 1 -- 172.21.15.46:0/777519368 shutdown_connections 2024-01-26T23:49:31.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.032+0000 7f8e0544f700 1 -- 172.21.15.46:0/777519368 wait complete. 2024-01-26T23:49:31.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.033+0000 7f8e0544f700 1 Processor -- start 2024-01-26T23:49:31.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.033+0000 7f8e0544f700 1 -- start start 2024-01-26T23:49:31.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.034+0000 7f8e0544f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8e000cb110 0x7f8e00154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.034+0000 7f8e0544f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e00159920 0x7f8e00169d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.034+0000 7f8e0544f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e00159cf0 0x7f8e001562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.034+0000 7f8e0544f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e00155130 con 0x7f8e00159cf0 2024-01-26T23:49:31.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.034+0000 7f8e0544f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e000afe80 con 0x7f8e000cb110 2024-01-26T23:49:31.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.034+0000 7f8e0544f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0016aca0 con 0x7f8e00159920 2024-01-26T23:49:31.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.034+0000 7f8dff7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e00159cf0 0x7f8e001562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.034+0000 7f8dff7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e00159cf0 0x7f8e001562b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59954/0 (socket says 172.21.15.46:59954) 2024-01-26T23:49:31.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.034+0000 7f8dff7fe700 1 -- 172.21.15.46:0/127362894 learned_addr learned my addr 172.21.15.46:0/127362894 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:31.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.034+0000 7f8dfeffd700 1 --2- 172.21.15.46:0/127362894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8e000cb110 0x7f8e00154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.034+0000 7f8dfe7fc700 1 --2- 172.21.15.46:0/127362894 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e00159920 0x7f8e00169d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.034+0000 7f8dff7fe700 1 -- 172.21.15.46:0/127362894 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e00159920 msgr2=0x7f8e00169d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:31.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.034+0000 7f8dff7fe700 1 --2- 172.21.15.46:0/127362894 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e00159920 0x7f8e00169d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.034+0000 7f8dff7fe700 1 -- 172.21.15.46:0/127362894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8e000cb110 msgr2=0x7f8e00154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:31.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.034+0000 7f8dff7fe700 1 --2- 172.21.15.46:0/127362894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8e000cb110 0x7f8e00154bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.034+0000 7f8dff7fe700 1 -- 172.21.15.46:0/127362894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df0009420 con 0x7f8e00159cf0 2024-01-26T23:49:31.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.035+0000 7f8dff7fe700 1 --2- 172.21.15.46:0/127362894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e00159cf0 0x7f8e001562b0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f8df8007f20 tx=0x7f8df8009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:31.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.035+0000 7f8df6ffd700 1 -- 172.21.15.46:0/127362894 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df800eb90 con 0x7f8e00159cf0 2024-01-26T23:49:31.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.035+0000 7f8df6ffd700 1 -- 172.21.15.46:0/127362894 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df800ed10 con 0x7f8e00159cf0 2024-01-26T23:49:31.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.035+0000 7f8e0544f700 1 -- 172.21.15.46:0/127362894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e000afb50 con 0x7f8e00159cf0 2024-01-26T23:49:31.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.035+0000 7f8df6ffd700 1 -- 172.21.15.46:0/127362894 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df8014670 con 0x7f8e00159cf0 2024-01-26T23:49:31.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.035+0000 7f8e0544f700 1 -- 172.21.15.46:0/127362894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e0016a9c0 con 0x7f8e00159cf0 2024-01-26T23:49:31.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.036+0000 7f8e0544f700 1 -- 172.21.15.46:0/127362894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e000afff0 con 0x7f8e00159cf0 2024-01-26T23:49:31.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.036+0000 7f8df6ffd700 1 -- 172.21.15.46:0/127362894 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8df80149e0 con 0x7f8e00159cf0 2024-01-26T23:49:31.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.037+0000 7f8df6ffd700 1 --2- 172.21.15.46:0/127362894 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8de0066f30 0x7f8de00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.037+0000 7f8df6ffd700 1 -- 172.21.15.46:0/127362894 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f8df8084e80 con 0x7f8e00159cf0 2024-01-26T23:49:31.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.037+0000 7f8dfeffd700 1 --2- 172.21.15.46:0/127362894 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8de0066f30 0x7f8de00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.037+0000 7f8dfeffd700 1 --2- 172.21.15.46:0/127362894 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8de0066f30 0x7f8de00693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8de8007fd0 tx=0x7f8de8007d30 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:31.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.041+0000 7f8df6ffd700 1 -- 172.21.15.46:0/127362894 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8df8051e20 con 0x7f8e00159cf0 2024-01-26T23:49:31.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.241+0000 7f8e0544f700 1 -- 172.21.15.46:0/127362894 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8e0004c710 con 0x7f8de0066f30 2024-01-26T23:49:31.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.250+0000 7f8df6ffd700 1 -- 172.21.15.46:0/127362894 <== mgr.7148 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289749 (secure 0 0 0) 0x7f8e0004c710 con 0x7f8de0066f30 2024-01-26T23:49:31.255 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.253+0000 7f8e0544f700 1 -- 172.21.15.46:0/127362894 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8de0066f30 msgr2=0x7f8de00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.253+0000 7f8e0544f700 1 --2- 172.21.15.46:0/127362894 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8de0066f30 0x7f8de00693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8de8007fd0 tx=0x7f8de8007d30 comp rx=0 tx=0).stop 2024-01-26T23:49:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.253+0000 7f8e0544f700 1 -- 172.21.15.46:0/127362894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e00159cf0 msgr2=0x7f8e001562b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.253+0000 7f8e0544f700 1 --2- 172.21.15.46:0/127362894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e00159cf0 0x7f8e001562b0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f8df8007f20 tx=0x7f8df8009860 comp rx=0 tx=0).stop 2024-01-26T23:49:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.253+0000 7f8e0544f700 1 -- 172.21.15.46:0/127362894 shutdown_connections 2024-01-26T23:49:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.253+0000 7f8e0544f700 1 --2- 172.21.15.46:0/127362894 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8de0066f30 0x7f8de00693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.253+0000 7f8e0544f700 1 --2- 172.21.15.46:0/127362894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e00159cf0 0x7f8e001562b0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.253+0000 7f8e0544f700 1 --2- 172.21.15.46:0/127362894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8e000cb110 0x7f8e00154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.253+0000 7f8e0544f700 1 --2- 172.21.15.46:0/127362894 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e00159920 0x7f8e00169d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.253+0000 7f8e0544f700 1 -- 172.21.15.46:0/127362894 >> 172.21.15.46:0/127362894 conn(0x7f8e000cd6f0 msgr2=0x7f8e000ca810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.253+0000 7f8e0544f700 1 -- 172.21.15.46:0/127362894 shutdown_connections 2024-01-26T23:49:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.253+0000 7f8e0544f700 1 -- 172.21.15.46:0/127362894 wait complete. 2024-01-26T23:49:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:49:31.268 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-01-26T23:49:30.871731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1163,"num_read_kb":2314,"num_write":405,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8623,"ondisk_log_size":8623,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2450952,"kb_used_data":15304,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752523768,"statfs":{"total":773094113280,"available":770584338432,"internally_reserved":0,"allocated":15671296,"data_stored":8625140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.179765"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720478+0000","last_change":"2024-01-26T23:49:21.716820+0000","last_active":"2024-01-26T23:49:21.720478+0000","last_peered":"2024-01-26T23:49:21.720478+0000","last_clean":"2024-01-26T23:49:21.720478+0000","last_became_active":"2024-01-26T23:49:21.716559+0000","last_became_peered":"2024-01-26T23:49:21.716559+0000","last_unstale":"2024-01-26T23:49:21.720478+0000","last_undegraded":"2024-01-26T23:49:21.720478+0000","last_fullsized":"2024-01-26T23:49:21.720478+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:31:49.800868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713577+0000","last_change":"2024-01-26T23:49:21.706608+0000","last_active":"2024-01-26T23:49:21.713577+0000","last_peered":"2024-01-26T23:49:21.713577+0000","last_clean":"2024-01-26T23:49:21.713577+0000","last_became_active":"2024-01-26T23:49:21.706095+0000","last_became_peered":"2024-01-26T23:49:21.706095+0000","last_unstale":"2024-01-26T23:49:21.713577+0000","last_undegraded":"2024-01-26T23:49:21.713577+0000","last_fullsized":"2024-01-26T23:49:21.713577+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:40:00.810209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713754+0000","last_change":"2024-01-26T23:49:21.710842+0000","last_active":"2024-01-26T23:49:21.713754+0000","last_peered":"2024-01-26T23:49:21.713754+0000","last_clean":"2024-01-26T23:49:21.713754+0000","last_became_active":"2024-01-26T23:49:21.710542+0000","last_became_peered":"2024-01-26T23:49:21.710542+0000","last_unstale":"2024-01-26T23:49:21.713754+0000","last_undegraded":"2024-01-26T23:49:21.713754+0000","last_fullsized":"2024-01-26T23:49:21.713754+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:46:28.563573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720399+0000","last_change":"2024-01-26T23:49:21.715413+0000","last_active":"2024-01-26T23:49:21.720399+0000","last_peered":"2024-01-26T23:49:21.720399+0000","last_clean":"2024-01-26T23:49:21.720399+0000","last_became_active":"2024-01-26T23:49:21.715122+0000","last_became_peered":"2024-01-26T23:49:21.715122+0000","last_unstale":"2024-01-26T23:49:21.720399+0000","last_undegraded":"2024-01-26T23:49:21.720399+0000","last_fullsized":"2024-01-26T23:49:21.720399+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:23:01.412861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714883+0000","last_change":"2024-01-26T23:49:21.711789+0000","last_active":"2024-01-26T23:49:21.714883+0000","last_peered":"2024-01-26T23:49:21.714883+0000","last_clean":"2024-01-26T23:49:21.714883+0000","last_became_active":"2024-01-26T23:49:21.711475+0000","last_became_peered":"2024-01-26T23:49:21.711475+0000","last_unstale":"2024-01-26T23:49:21.714883+0000","last_undegraded":"2024-01-26T23:49:21.714883+0000","last_fullsized":"2024-01-26T23:49:21.714883+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:22:39.451844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.709457+0000","last_change":"2024-01-26T23:49:21.704699+0000","last_active":"2024-01-26T23:49:21.709457+0000","last_peered":"2024-01-26T23:49:21.709457+0000","last_clean":"2024-01-26T23:49:21.709457+0000","last_became_active":"2024-01-26T23:49:21.704402+0000","last_became_peered":"2024-01-26T23:49:21.704402+0000","last_unstale":"2024-01-26T23:49:21.709457+0000","last_undegraded":"2024-01-26T23:49:21.709457+0000","last_fullsized":"2024-01-26T23:49:21.709457+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:38.865560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.710793+0000","last_change":"2024-01-26T23:49:21.705657+0000","last_active":"2024-01-26T23:49:21.710793+0000","last_peered":"2024-01-26T23:49:21.710793+0000","last_clean":"2024-01-26T23:49:21.710793+0000","last_became_active":"2024-01-26T23:49:21.705381+0000","last_became_peered":"2024-01-26T23:49:21.705381+0000","last_unstale":"2024-01-26T23:49:21.710793+0000","last_undegraded":"2024-01-26T23:49:21.710793+0000","last_fullsized":"2024-01-26T23:49:21.710793+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:53:27.231318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716206+0000","last_change":"2024-01-26T23:49:21.714982+0000","last_active":"2024-01-26T23:49:21.716206+0000","last_peered":"2024-01-26T23:49:21.716206+0000","last_clean":"2024-01-26T23:49:21.716206+0000","last_became_active":"2024-01-26T23:49:21.714655+0000","last_became_peered":"2024-01-26T23:49:21.714655+0000","last_unstale":"2024-01-26T23:49:21.716206+0000","last_undegraded":"2024-01-26T23:49:21.716206+0000","last_fullsized":"2024-01-26T23:49:21.716206+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:46:36.438311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720506+0000","last_change":"2024-01-26T23:49:21.716127+0000","last_active":"2024-01-26T23:49:21.720506+0000","last_peered":"2024-01-26T23:49:21.720506+0000","last_clean":"2024-01-26T23:49:21.720506+0000","last_became_active":"2024-01-26T23:49:21.715868+0000","last_became_peered":"2024-01-26T23:49:21.715868+0000","last_unstale":"2024-01-26T23:49:21.720506+0000","last_undegraded":"2024-01-26T23:49:21.720506+0000","last_fullsized":"2024-01-26T23:49:21.720506+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:27:10.909797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.705419+0000","last_change":"2024-01-26T23:49:21.701945+0000","last_active":"2024-01-26T23:49:21.705419+0000","last_peered":"2024-01-26T23:49:21.705419+0000","last_clean":"2024-01-26T23:49:21.705419+0000","last_became_active":"2024-01-26T23:49:21.701542+0000","last_became_peered":"2024-01-26T23:49:21.701542+0000","last_unstale":"2024-01-26T23:49:21.705419+0000","last_undegraded":"2024-01-26T23:49:21.705419+0000","last_fullsized":"2024-01-26T23:49:21.705419+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:51.588887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720594+0000","last_change":"2024-01-26T23:49:21.717788+0000","last_active":"2024-01-26T23:49:21.720594+0000","last_peered":"2024-01-26T23:49:21.720594+0000","last_clean":"2024-01-26T23:49:21.720594+0000","last_became_active":"2024-01-26T23:49:21.717188+0000","last_became_peered":"2024-01-26T23:49:21.717188+0000","last_unstale":"2024-01-26T23:49:21.720594+0000","last_undegraded":"2024-01-26T23:49:21.720594+0000","last_fullsized":"2024-01-26T23:49:21.720594+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:52:44.481461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716190+0000","last_change":"2024-01-26T23:49:21.713083+0000","last_active":"2024-01-26T23:49:21.716190+0000","last_peered":"2024-01-26T23:49:21.716190+0000","last_clean":"2024-01-26T23:49:21.716190+0000","last_became_active":"2024-01-26T23:49:21.712770+0000","last_became_peered":"2024-01-26T23:49:21.712770+0000","last_unstale":"2024-01-26T23:49:21.716190+0000","last_undegraded":"2024-01-26T23:49:21.716190+0000","last_fullsized":"2024-01-26T23:49:21.716190+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:55:44.107048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.701246+0000","last_change":"2024-01-26T23:49:21.698975+0000","last_active":"2024-01-26T23:49:21.701246+0000","last_peered":"2024-01-26T23:49:21.701246+0000","last_clean":"2024-01-26T23:49:21.701246+0000","last_became_active":"2024-01-26T23:49:21.698670+0000","last_became_peered":"2024-01-26T23:49:21.698670+0000","last_unstale":"2024-01-26T23:49:21.701246+0000","last_undegraded":"2024-01-26T23:49:21.701246+0000","last_fullsized":"2024-01-26T23:49:21.701246+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:46:36.438311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713198+0000","last_change":"2024-01-26T23:49:21.711141+0000","last_active":"2024-01-26T23:49:21.713198+0000","last_peered":"2024-01-26T23:49:21.713198+0000","last_clean":"2024-01-26T23:49:21.713198+0000","last_became_active":"2024-01-26T23:49:21.710788+0000","last_became_peered":"2024-01-26T23:49:21.710788+0000","last_unstale":"2024-01-26T23:49:21.713198+0000","last_undegraded":"2024-01-26T23:49:21.713198+0000","last_fullsized":"2024-01-26T23:49:21.713198+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:32:26.662337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715295+0000","last_change":"2024-01-26T23:49:21.712785+0000","last_active":"2024-01-26T23:49:21.715295+0000","last_peered":"2024-01-26T23:49:21.715295+0000","last_clean":"2024-01-26T23:49:21.715295+0000","last_became_active":"2024-01-26T23:49:21.711603+0000","last_became_peered":"2024-01-26T23:49:21.711603+0000","last_unstale":"2024-01-26T23:49:21.715295+0000","last_undegraded":"2024-01-26T23:49:21.715295+0000","last_fullsized":"2024-01-26T23:49:21.715295+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:42:38.767042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.710670+0000","last_change":"2024-01-26T23:49:21.704936+0000","last_active":"2024-01-26T23:49:21.710670+0000","last_peered":"2024-01-26T23:49:21.710670+0000","last_clean":"2024-01-26T23:49:21.710670+0000","last_became_active":"2024-01-26T23:49:21.704639+0000","last_became_peered":"2024-01-26T23:49:21.704639+0000","last_unstale":"2024-01-26T23:49:21.710670+0000","last_undegraded":"2024-01-26T23:49:21.710670+0000","last_fullsized":"2024-01-26T23:49:21.710670+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:26:52.470020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.707219+0000","last_change":"2024-01-26T23:49:21.702319+0000","last_active":"2024-01-26T23:49:21.707219+0000","last_peered":"2024-01-26T23:49:21.707219+0000","last_clean":"2024-01-26T23:49:21.707219+0000","last_became_active":"2024-01-26T23:49:21.701995+0000","last_became_peered":"2024-01-26T23:49:21.701995+0000","last_unstale":"2024-01-26T23:49:21.707219+0000","last_undegraded":"2024-01-26T23:49:21.707219+0000","last_fullsized":"2024-01-26T23:49:21.707219+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:44:38.742019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713556+0000","last_change":"2024-01-26T23:49:21.706418+0000","last_active":"2024-01-26T23:49:21.713556+0000","last_peered":"2024-01-26T23:49:21.713556+0000","last_clean":"2024-01-26T23:49:21.713556+0000","last_became_active":"2024-01-26T23:49:21.705710+0000","last_became_peered":"2024-01-26T23:49:21.705710+0000","last_unstale":"2024-01-26T23:49:21.713556+0000","last_undegraded":"2024-01-26T23:49:21.713556+0000","last_fullsized":"2024-01-26T23:49:21.713556+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:07:15.034585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715571+0000","last_change":"2024-01-26T23:49:21.708843+0000","last_active":"2024-01-26T23:49:21.715571+0000","last_peered":"2024-01-26T23:49:21.715571+0000","last_clean":"2024-01-26T23:49:21.715571+0000","last_became_active":"2024-01-26T23:49:21.708590+0000","last_became_peered":"2024-01-26T23:49:21.708590+0000","last_unstale":"2024-01-26T23:49:21.715571+0000","last_undegraded":"2024-01-26T23:49:21.715571+0000","last_fullsized":"2024-01-26T23:49:21.715571+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:47:59.288036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.712492+0000","last_change":"2024-01-26T23:49:21.709022+0000","last_active":"2024-01-26T23:49:21.712492+0000","last_peered":"2024-01-26T23:49:21.712492+0000","last_clean":"2024-01-26T23:49:21.712492+0000","last_became_active":"2024-01-26T23:49:21.708716+0000","last_became_peered":"2024-01-26T23:49:21.708716+0000","last_unstale":"2024-01-26T23:49:21.712492+0000","last_undegraded":"2024-01-26T23:49:21.712492+0000","last_fullsized":"2024-01-26T23:49:21.712492+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:18:32.453554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.717173+0000","last_change":"2024-01-26T23:49:21.712966+0000","last_active":"2024-01-26T23:49:21.717173+0000","last_peered":"2024-01-26T23:49:21.717173+0000","last_clean":"2024-01-26T23:49:21.717173+0000","last_became_active":"2024-01-26T23:49:21.712337+0000","last_became_peered":"2024-01-26T23:49:21.712337+0000","last_unstale":"2024-01-26T23:49:21.717173+0000","last_undegraded":"2024-01-26T23:49:21.717173+0000","last_fullsized":"2024-01-26T23:49:21.717173+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:36:49.497687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714994+0000","last_change":"2024-01-26T23:49:21.712389+0000","last_active":"2024-01-26T23:49:21.714994+0000","last_peered":"2024-01-26T23:49:21.714994+0000","last_clean":"2024-01-26T23:49:21.714994+0000","last_became_active":"2024-01-26T23:49:21.712138+0000","last_became_peered":"2024-01-26T23:49:21.712138+0000","last_unstale":"2024-01-26T23:49:21.714994+0000","last_undegraded":"2024-01-26T23:49:21.714994+0000","last_fullsized":"2024-01-26T23:49:21.714994+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:36:35.342280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.717216+0000","last_change":"2024-01-26T23:49:21.713016+0000","last_active":"2024-01-26T23:49:21.717216+0000","last_peered":"2024-01-26T23:49:21.717216+0000","last_clean":"2024-01-26T23:49:21.717216+0000","last_became_active":"2024-01-26T23:49:21.712716+0000","last_became_peered":"2024-01-26T23:49:21.712716+0000","last_unstale":"2024-01-26T23:49:21.717216+0000","last_undegraded":"2024-01-26T23:49:21.717216+0000","last_fullsized":"2024-01-26T23:49:21.717216+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:19:56.848512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716319+0000","last_change":"2024-01-26T23:49:21.712460+0000","last_active":"2024-01-26T23:49:21.716319+0000","last_peered":"2024-01-26T23:49:21.716319+0000","last_clean":"2024-01-26T23:49:21.716319+0000","last_became_active":"2024-01-26T23:49:21.712123+0000","last_became_peered":"2024-01-26T23:49:21.712123+0000","last_unstale":"2024-01-26T23:49:21.716319+0000","last_undegraded":"2024-01-26T23:49:21.716319+0000","last_fullsized":"2024-01-26T23:49:21.716319+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:57:09.862596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716309+0000","last_change":"2024-01-26T23:49:21.712418+0000","last_active":"2024-01-26T23:49:21.716309+0000","last_peered":"2024-01-26T23:49:21.716309+0000","last_clean":"2024-01-26T23:49:21.716309+0000","last_became_active":"2024-01-26T23:49:21.712099+0000","last_became_peered":"2024-01-26T23:49:21.712099+0000","last_unstale":"2024-01-26T23:49:21.716309+0000","last_undegraded":"2024-01-26T23:49:21.716309+0000","last_fullsized":"2024-01-26T23:49:21.716309+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:17.484964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.708922+0000","last_change":"2024-01-26T23:49:21.705150+0000","last_active":"2024-01-26T23:49:21.708922+0000","last_peered":"2024-01-26T23:49:21.708922+0000","last_clean":"2024-01-26T23:49:21.708922+0000","last_became_active":"2024-01-26T23:49:21.703248+0000","last_became_peered":"2024-01-26T23:49:21.703248+0000","last_unstale":"2024-01-26T23:49:21.708922+0000","last_undegraded":"2024-01-26T23:49:21.708922+0000","last_fullsized":"2024-01-26T23:49:21.708922+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:49:45.239681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713631+0000","last_change":"2024-01-26T23:49:21.708382+0000","last_active":"2024-01-26T23:49:21.713631+0000","last_peered":"2024-01-26T23:49:21.713631+0000","last_clean":"2024-01-26T23:49:21.713631+0000","last_became_active":"2024-01-26T23:49:21.708082+0000","last_became_peered":"2024-01-26T23:49:21.708082+0000","last_unstale":"2024-01-26T23:49:21.713631+0000","last_undegraded":"2024-01-26T23:49:21.713631+0000","last_fullsized":"2024-01-26T23:49:21.713631+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:44:38.742019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716169+0000","last_change":"2024-01-26T23:49:21.711944+0000","last_active":"2024-01-26T23:49:21.716169+0000","last_peered":"2024-01-26T23:49:21.716169+0000","last_clean":"2024-01-26T23:49:21.716169+0000","last_became_active":"2024-01-26T23:49:21.711661+0000","last_became_peered":"2024-01-26T23:49:21.711661+0000","last_unstale":"2024-01-26T23:49:21.716169+0000","last_undegraded":"2024-01-26T23:49:21.716169+0000","last_fullsized":"2024-01-26T23:49:21.716169+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:07.408591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714134+0000","last_change":"2024-01-26T23:49:21.711617+0000","last_active":"2024-01-26T23:49:21.714134+0000","last_peered":"2024-01-26T23:49:21.714134+0000","last_clean":"2024-01-26T23:49:21.714134+0000","last_became_active":"2024-01-26T23:49:21.711167+0000","last_became_peered":"2024-01-26T23:49:21.711167+0000","last_unstale":"2024-01-26T23:49:21.714134+0000","last_undegraded":"2024-01-26T23:49:21.714134+0000","last_fullsized":"2024-01-26T23:49:21.714134+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:07:55.842419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715903+0000","last_change":"2024-01-26T23:49:21.713998+0000","last_active":"2024-01-26T23:49:21.715903+0000","last_peered":"2024-01-26T23:49:21.715903+0000","last_clean":"2024-01-26T23:49:21.715903+0000","last_became_active":"2024-01-26T23:49:21.713699+0000","last_became_peered":"2024-01-26T23:49:21.713699+0000","last_unstale":"2024-01-26T23:49:21.715903+0000","last_undegraded":"2024-01-26T23:49:21.715903+0000","last_fullsized":"2024-01-26T23:49:21.715903+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:31:18.118068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.710887+0000","last_change":"2024-01-26T23:49:21.701621+0000","last_active":"2024-01-26T23:49:21.710887+0000","last_peered":"2024-01-26T23:49:21.710887+0000","last_clean":"2024-01-26T23:49:21.710887+0000","last_became_active":"2024-01-26T23:49:21.701321+0000","last_became_peered":"2024-01-26T23:49:21.701321+0000","last_unstale":"2024-01-26T23:49:21.710887+0000","last_undegraded":"2024-01-26T23:49:21.710887+0000","last_fullsized":"2024-01-26T23:49:21.710887+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:55:23.167302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716353+0000","last_change":"2024-01-26T23:49:21.712792+0000","last_active":"2024-01-26T23:49:21.716353+0000","last_peered":"2024-01-26T23:49:21.716353+0000","last_clean":"2024-01-26T23:49:21.716353+0000","last_became_active":"2024-01-26T23:49:21.711757+0000","last_became_peered":"2024-01-26T23:49:21.711757+0000","last_unstale":"2024-01-26T23:49:21.716353+0000","last_undegraded":"2024-01-26T23:49:21.716353+0000","last_fullsized":"2024-01-26T23:49:21.716353+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:03:11.827036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.708503+0000","last_change":"2024-01-26T23:49:21.706929+0000","last_active":"2024-01-26T23:49:21.708503+0000","last_peered":"2024-01-26T23:49:21.708503+0000","last_clean":"2024-01-26T23:49:21.708503+0000","last_became_active":"2024-01-26T23:49:21.706543+0000","last_became_peered":"2024-01-26T23:49:21.706543+0000","last_unstale":"2024-01-26T23:49:21.708503+0000","last_undegraded":"2024-01-26T23:49:21.708503+0000","last_fullsized":"2024-01-26T23:49:21.708503+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:34.441797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"144'4","reported_seq":14,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.947803+0000","last_change":"2024-01-26T23:49:20.933655+0000","last_active":"2024-01-26T23:49:21.947803+0000","last_peered":"2024-01-26T23:49:21.947803+0000","last_clean":"2024-01-26T23:49:21.947803+0000","last_became_active":"2024-01-26T23:49:20.933360+0000","last_became_peered":"2024-01-26T23:49:20.933360+0000","last_unstale":"2024-01-26T23:49:21.947803+0000","last_undegraded":"2024-01-26T23:49:21.947803+0000","last_fullsized":"2024-01-26T23:49:21.947803+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:23:00.411032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716434+0000","last_change":"2024-01-26T23:49:21.712338+0000","last_active":"2024-01-26T23:49:21.716434+0000","last_peered":"2024-01-26T23:49:21.716434+0000","last_clean":"2024-01-26T23:49:21.716434+0000","last_became_active":"2024-01-26T23:49:21.712030+0000","last_became_peered":"2024-01-26T23:49:21.712030+0000","last_unstale":"2024-01-26T23:49:21.716434+0000","last_undegraded":"2024-01-26T23:49:21.716434+0000","last_fullsized":"2024-01-26T23:49:21.716434+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:25:47.504184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.947564+0000","last_change":"2024-01-26T23:49:20.934413+0000","last_active":"2024-01-26T23:49:21.947564+0000","last_peered":"2024-01-26T23:49:21.947564+0000","last_clean":"2024-01-26T23:49:21.947564+0000","last_became_active":"2024-01-26T23:49:20.933769+0000","last_became_peered":"2024-01-26T23:49:20.933769+0000","last_unstale":"2024-01-26T23:49:21.947564+0000","last_undegraded":"2024-01-26T23:49:21.947564+0000","last_fullsized":"2024-01-26T23:49:21.947564+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:05:23.745399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716376+0000","last_change":"2024-01-26T23:49:21.712916+0000","last_active":"2024-01-26T23:49:21.716376+0000","last_peered":"2024-01-26T23:49:21.716376+0000","last_clean":"2024-01-26T23:49:21.716376+0000","last_became_active":"2024-01-26T23:49:21.712259+0000","last_became_peered":"2024-01-26T23:49:21.712259+0000","last_unstale":"2024-01-26T23:49:21.716376+0000","last_undegraded":"2024-01-26T23:49:21.716376+0000","last_fullsized":"2024-01-26T23:49:21.716376+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:05:24.747228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.947622+0000","last_change":"2024-01-26T23:49:20.933682+0000","last_active":"2024-01-26T23:49:21.947622+0000","last_peered":"2024-01-26T23:49:21.947622+0000","last_clean":"2024-01-26T23:49:21.947622+0000","last_became_active":"2024-01-26T23:49:20.933419+0000","last_became_peered":"2024-01-26T23:49:20.933419+0000","last_unstale":"2024-01-26T23:49:21.947622+0000","last_undegraded":"2024-01-26T23:49:21.947622+0000","last_fullsized":"2024-01-26T23:49:21.947622+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:31:23.721288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714845+0000","last_change":"2024-01-26T23:49:21.709566+0000","last_active":"2024-01-26T23:49:21.714845+0000","last_peered":"2024-01-26T23:49:21.714845+0000","last_clean":"2024-01-26T23:49:21.714845+0000","last_became_active":"2024-01-26T23:49:21.709308+0000","last_became_peered":"2024-01-26T23:49:21.709308+0000","last_unstale":"2024-01-26T23:49:21.714845+0000","last_undegraded":"2024-01-26T23:49:21.714845+0000","last_fullsized":"2024-01-26T23:49:21.714845+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:01:52.607618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.951313+0000","last_change":"2024-01-26T23:49:20.941242+0000","last_active":"2024-01-26T23:49:21.951313+0000","last_peered":"2024-01-26T23:49:21.951313+0000","last_clean":"2024-01-26T23:49:21.951313+0000","last_became_active":"2024-01-26T23:49:20.940914+0000","last_became_peered":"2024-01-26T23:49:20.940914+0000","last_unstale":"2024-01-26T23:49:21.951313+0000","last_undegraded":"2024-01-26T23:49:21.951313+0000","last_fullsized":"2024-01-26T23:49:21.951313+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:52:55.167098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715579+0000","last_change":"2024-01-26T23:49:21.713588+0000","last_active":"2024-01-26T23:49:21.715579+0000","last_peered":"2024-01-26T23:49:21.715579+0000","last_clean":"2024-01-26T23:49:21.715579+0000","last_became_active":"2024-01-26T23:49:21.713242+0000","last_became_peered":"2024-01-26T23:49:21.713242+0000","last_unstale":"2024-01-26T23:49:21.715579+0000","last_undegraded":"2024-01-26T23:49:21.715579+0000","last_fullsized":"2024-01-26T23:49:21.715579+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:27:10.909797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.950236+0000","last_change":"2024-01-26T23:49:20.935760+0000","last_active":"2024-01-26T23:49:21.950236+0000","last_peered":"2024-01-26T23:49:21.950236+0000","last_clean":"2024-01-26T23:49:21.950236+0000","last_became_active":"2024-01-26T23:49:20.935522+0000","last_became_peered":"2024-01-26T23:49:20.935522+0000","last_unstale":"2024-01-26T23:49:21.950236+0000","last_undegraded":"2024-01-26T23:49:21.950236+0000","last_fullsized":"2024-01-26T23:49:21.950236+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:46:35.436482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715276+0000","last_change":"2024-01-26T23:49:21.711623+0000","last_active":"2024-01-26T23:49:21.715276+0000","last_peered":"2024-01-26T23:49:21.715276+0000","last_clean":"2024-01-26T23:49:21.715276+0000","last_became_active":"2024-01-26T23:49:21.711237+0000","last_became_peered":"2024-01-26T23:49:21.711237+0000","last_unstale":"2024-01-26T23:49:21.715276+0000","last_undegraded":"2024-01-26T23:49:21.715276+0000","last_fullsized":"2024-01-26T23:49:21.715276+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:14:30.475748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.950458+0000","last_change":"2024-01-26T23:49:20.937373+0000","last_active":"2024-01-26T23:49:21.950458+0000","last_peered":"2024-01-26T23:49:21.950458+0000","last_clean":"2024-01-26T23:49:21.950458+0000","last_became_active":"2024-01-26T23:49:20.937065+0000","last_became_peered":"2024-01-26T23:49:20.937065+0000","last_unstale":"2024-01-26T23:49:21.950458+0000","last_undegraded":"2024-01-26T23:49:21.950458+0000","last_fullsized":"2024-01-26T23:49:21.950458+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:49:16.191682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.702584+0000","last_change":"2024-01-26T23:49:21.699753+0000","last_active":"2024-01-26T23:49:21.702584+0000","last_peered":"2024-01-26T23:49:21.702584+0000","last_clean":"2024-01-26T23:49:21.702584+0000","last_became_active":"2024-01-26T23:49:21.699464+0000","last_became_peered":"2024-01-26T23:49:21.699464+0000","last_unstale":"2024-01-26T23:49:21.702584+0000","last_undegraded":"2024-01-26T23:49:21.702584+0000","last_fullsized":"2024-01-26T23:49:21.702584+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:03.507801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.695979+0000","last_change":"2024-01-26T23:49:20.928944+0000","last_active":"2024-01-26T23:49:21.695979+0000","last_peered":"2024-01-26T23:49:21.695979+0000","last_clean":"2024-01-26T23:49:21.695979+0000","last_became_active":"2024-01-26T23:49:20.928649+0000","last_became_peered":"2024-01-26T23:49:20.928649+0000","last_unstale":"2024-01-26T23:49:21.695979+0000","last_undegraded":"2024-01-26T23:49:21.695979+0000","last_fullsized":"2024-01-26T23:49:21.695979+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:55:50.467126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.702548+0000","last_change":"2024-01-26T23:49:21.699714+0000","last_active":"2024-01-26T23:49:21.702548+0000","last_peered":"2024-01-26T23:49:21.702548+0000","last_clean":"2024-01-26T23:49:21.702548+0000","last_became_active":"2024-01-26T23:49:21.699419+0000","last_became_peered":"2024-01-26T23:49:21.699419+0000","last_unstale":"2024-01-26T23:49:21.702548+0000","last_undegraded":"2024-01-26T23:49:21.702548+0000","last_fullsized":"2024-01-26T23:49:21.702548+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:47:59.288036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.696916+0000","last_change":"2024-01-26T23:49:20.927169+0000","last_active":"2024-01-26T23:49:21.696916+0000","last_peered":"2024-01-26T23:49:21.696916+0000","last_clean":"2024-01-26T23:49:21.696916+0000","last_became_active":"2024-01-26T23:49:20.926865+0000","last_became_peered":"2024-01-26T23:49:20.926865+0000","last_unstale":"2024-01-26T23:49:21.696916+0000","last_undegraded":"2024-01-26T23:49:21.696916+0000","last_fullsized":"2024-01-26T23:49:21.696916+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:57:52.726572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715082+0000","last_change":"2024-01-26T23:49:21.708432+0000","last_active":"2024-01-26T23:49:21.715082+0000","last_peered":"2024-01-26T23:49:21.715082+0000","last_clean":"2024-01-26T23:49:21.715082+0000","last_became_active":"2024-01-26T23:49:21.708100+0000","last_became_peered":"2024-01-26T23:49:21.708100+0000","last_unstale":"2024-01-26T23:49:21.715082+0000","last_undegraded":"2024-01-26T23:49:21.715082+0000","last_fullsized":"2024-01-26T23:49:21.715082+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:31:18.118068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.707105+0000","last_change":"2024-01-26T23:49:20.928442+0000","last_active":"2024-01-26T23:49:21.707105+0000","last_peered":"2024-01-26T23:49:21.707105+0000","last_clean":"2024-01-26T23:49:21.707105+0000","last_became_active":"2024-01-26T23:49:20.928198+0000","last_became_peered":"2024-01-26T23:49:20.928198+0000","last_unstale":"2024-01-26T23:49:21.707105+0000","last_undegraded":"2024-01-26T23:49:21.707105+0000","last_fullsized":"2024-01-26T23:49:21.707105+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:33.439967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.710927+0000","last_change":"2024-01-26T23:49:21.702605+0000","last_active":"2024-01-26T23:49:21.710927+0000","last_peered":"2024-01-26T23:49:21.710927+0000","last_clean":"2024-01-26T23:49:21.710927+0000","last_became_active":"2024-01-26T23:49:21.702345+0000","last_became_peered":"2024-01-26T23:49:21.702345+0000","last_unstale":"2024-01-26T23:49:21.710927+0000","last_undegraded":"2024-01-26T23:49:21.710927+0000","last_fullsized":"2024-01-26T23:49:21.710927+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:00:37.977251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.696186+0000","last_change":"2024-01-26T23:49:20.933290+0000","last_active":"2024-01-26T23:49:21.696186+0000","last_peered":"2024-01-26T23:49:21.696186+0000","last_clean":"2024-01-26T23:49:21.696186+0000","last_became_active":"2024-01-26T23:49:20.932911+0000","last_became_peered":"2024-01-26T23:49:20.932911+0000","last_unstale":"2024-01-26T23:49:21.696186+0000","last_undegraded":"2024-01-26T23:49:21.696186+0000","last_fullsized":"2024-01-26T23:49:21.696186+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:06:47.740099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713595+0000","last_change":"2024-01-26T23:49:21.710063+0000","last_active":"2024-01-26T23:49:21.713595+0000","last_peered":"2024-01-26T23:49:21.713595+0000","last_clean":"2024-01-26T23:49:21.713595+0000","last_became_active":"2024-01-26T23:49:21.709757+0000","last_became_peered":"2024-01-26T23:49:21.709757+0000","last_unstale":"2024-01-26T23:49:21.713595+0000","last_undegraded":"2024-01-26T23:49:21.713595+0000","last_fullsized":"2024-01-26T23:49:21.713595+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:47:59.288036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"144'3","reported_seq":13,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.950273+0000","last_change":"2024-01-26T23:49:20.926313+0000","last_active":"2024-01-26T23:49:21.950273+0000","last_peered":"2024-01-26T23:49:21.950273+0000","last_clean":"2024-01-26T23:49:21.950273+0000","last_became_active":"2024-01-26T23:49:20.926067+0000","last_became_peered":"2024-01-26T23:49:20.926067+0000","last_unstale":"2024-01-26T23:49:21.950273+0000","last_undegraded":"2024-01-26T23:49:21.950273+0000","last_fullsized":"2024-01-26T23:49:21.950273+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:09:02.526081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714584+0000","last_change":"2024-01-26T23:49:21.709992+0000","last_active":"2024-01-26T23:49:21.714584+0000","last_peered":"2024-01-26T23:49:21.714584+0000","last_clean":"2024-01-26T23:49:21.714584+0000","last_became_active":"2024-01-26T23:49:21.709742+0000","last_became_peered":"2024-01-26T23:49:21.709742+0000","last_unstale":"2024-01-26T23:49:21.714584+0000","last_undegraded":"2024-01-26T23:49:21.714584+0000","last_fullsized":"2024-01-26T23:49:21.714584+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:39:24.864123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"144'6","reported_seq":16,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.955758+0000","last_change":"2024-01-26T23:49:20.932889+0000","last_active":"2024-01-26T23:49:21.955758+0000","last_peered":"2024-01-26T23:49:21.955758+0000","last_clean":"2024-01-26T23:49:21.955758+0000","last_became_active":"2024-01-26T23:49:20.932592+0000","last_became_peered":"2024-01-26T23:49:20.932592+0000","last_unstale":"2024-01-26T23:49:21.955758+0000","last_undegraded":"2024-01-26T23:49:21.955758+0000","last_fullsized":"2024-01-26T23:49:21.955758+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:16.483135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.718948+0000","last_change":"2024-01-26T23:49:21.714417+0000","last_active":"2024-01-26T23:49:21.718948+0000","last_peered":"2024-01-26T23:49:21.718948+0000","last_clean":"2024-01-26T23:49:21.718948+0000","last_became_active":"2024-01-26T23:49:21.714128+0000","last_became_peered":"2024-01-26T23:49:21.714128+0000","last_unstale":"2024-01-26T23:49:21.718948+0000","last_undegraded":"2024-01-26T23:49:21.718948+0000","last_fullsized":"2024-01-26T23:49:21.718948+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:46:36.438311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.952459+0000","last_change":"2024-01-26T23:49:20.934520+0000","last_active":"2024-01-26T23:49:21.952459+0000","last_peered":"2024-01-26T23:49:21.952459+0000","last_clean":"2024-01-26T23:49:21.952459+0000","last_became_active":"2024-01-26T23:49:20.933872+0000","last_became_peered":"2024-01-26T23:49:20.933872+0000","last_unstale":"2024-01-26T23:49:21.952459+0000","last_undegraded":"2024-01-26T23:49:21.952459+0000","last_fullsized":"2024-01-26T23:49:21.952459+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:39:31.988243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.708179+0000","last_change":"2024-01-26T23:49:21.705738+0000","last_active":"2024-01-26T23:49:21.708179+0000","last_peered":"2024-01-26T23:49:21.708179+0000","last_clean":"2024-01-26T23:49:21.708179+0000","last_became_active":"2024-01-26T23:49:21.705447+0000","last_became_peered":"2024-01-26T23:49:21.705447+0000","last_unstale":"2024-01-26T23:49:21.708179+0000","last_undegraded":"2024-01-26T23:49:21.708179+0000","last_fullsized":"2024-01-26T23:49:21.708179+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:43:20.056916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.697347+0000","last_change":"2024-01-26T23:49:20.925959+0000","last_active":"2024-01-26T23:49:21.697347+0000","last_peered":"2024-01-26T23:49:21.697347+0000","last_clean":"2024-01-26T23:49:21.697347+0000","last_became_active":"2024-01-26T23:49:20.925600+0000","last_became_peered":"2024-01-26T23:49:20.925600+0000","last_unstale":"2024-01-26T23:49:21.697347+0000","last_undegraded":"2024-01-26T23:49:21.697347+0000","last_fullsized":"2024-01-26T23:49:21.697347+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:55:22.165473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714932+0000","last_change":"2024-01-26T23:49:21.713422+0000","last_active":"2024-01-26T23:49:21.714932+0000","last_peered":"2024-01-26T23:49:21.714932+0000","last_clean":"2024-01-26T23:49:21.714932+0000","last_became_active":"2024-01-26T23:49:21.713129+0000","last_became_peered":"2024-01-26T23:49:21.713129+0000","last_unstale":"2024-01-26T23:49:21.714932+0000","last_undegraded":"2024-01-26T23:49:21.714932+0000","last_fullsized":"2024-01-26T23:49:21.714932+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:36:43.200780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.696779+0000","last_change":"2024-01-26T23:49:20.926498+0000","last_active":"2024-01-26T23:49:21.696779+0000","last_peered":"2024-01-26T23:49:21.696779+0000","last_clean":"2024-01-26T23:49:21.696779+0000","last_became_active":"2024-01-26T23:49:20.926183+0000","last_became_peered":"2024-01-26T23:49:20.926183+0000","last_unstale":"2024-01-26T23:49:21.696779+0000","last_undegraded":"2024-01-26T23:49:21.696779+0000","last_fullsized":"2024-01-26T23:49:21.696779+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:00:36.975421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714445+0000","last_change":"2024-01-26T23:49:21.708959+0000","last_active":"2024-01-26T23:49:21.714445+0000","last_peered":"2024-01-26T23:49:21.714445+0000","last_clean":"2024-01-26T23:49:21.714445+0000","last_became_active":"2024-01-26T23:49:21.708693+0000","last_became_peered":"2024-01-26T23:49:21.708693+0000","last_unstale":"2024-01-26T23:49:21.714445+0000","last_undegraded":"2024-01-26T23:49:21.714445+0000","last_fullsized":"2024-01-26T23:49:21.714445+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:39:32.990073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.948408+0000","last_change":"2024-01-26T23:49:20.934522+0000","last_active":"2024-01-26T23:49:21.948408+0000","last_peered":"2024-01-26T23:49:21.948408+0000","last_clean":"2024-01-26T23:49:21.948408+0000","last_became_active":"2024-01-26T23:49:20.934219+0000","last_became_peered":"2024-01-26T23:49:20.934219+0000","last_unstale":"2024-01-26T23:49:21.948408+0000","last_undegraded":"2024-01-26T23:49:21.948408+0000","last_fullsized":"2024-01-26T23:49:21.948408+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:31:23.721288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720614+0000","last_change":"2024-01-26T23:49:21.718008+0000","last_active":"2024-01-26T23:49:21.720614+0000","last_peered":"2024-01-26T23:49:21.720614+0000","last_clean":"2024-01-26T23:49:21.720614+0000","last_became_active":"2024-01-26T23:49:21.717522+0000","last_became_peered":"2024-01-26T23:49:21.717522+0000","last_unstale":"2024-01-26T23:49:21.720614+0000","last_undegraded":"2024-01-26T23:49:21.720614+0000","last_fullsized":"2024-01-26T23:49:21.720614+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:25:47.543272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.948077+0000","last_change":"2024-01-26T23:49:20.938034+0000","last_active":"2024-01-26T23:49:21.948077+0000","last_peered":"2024-01-26T23:49:21.948077+0000","last_clean":"2024-01-26T23:49:21.948077+0000","last_became_active":"2024-01-26T23:49:20.937700+0000","last_became_peered":"2024-01-26T23:49:20.937700+0000","last_unstale":"2024-01-26T23:49:21.948077+0000","last_undegraded":"2024-01-26T23:49:21.948077+0000","last_fullsized":"2024-01-26T23:49:21.948077+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:30:23.346922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715156+0000","last_change":"2024-01-26T23:49:21.711030+0000","last_active":"2024-01-26T23:49:21.715156+0000","last_peered":"2024-01-26T23:49:21.715156+0000","last_clean":"2024-01-26T23:49:21.715156+0000","last_became_active":"2024-01-26T23:49:21.710735+0000","last_became_peered":"2024-01-26T23:49:21.710735+0000","last_unstale":"2024-01-26T23:49:21.715156+0000","last_undegraded":"2024-01-26T23:49:21.715156+0000","last_fullsized":"2024-01-26T23:49:21.715156+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:22:26.000383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.950570+0000","last_change":"2024-01-26T23:49:20.932856+0000","last_active":"2024-01-26T23:49:21.950570+0000","last_peered":"2024-01-26T23:49:21.950570+0000","last_clean":"2024-01-26T23:49:21.950570+0000","last_became_active":"2024-01-26T23:49:20.932586+0000","last_became_peered":"2024-01-26T23:49:20.932586+0000","last_unstale":"2024-01-26T23:49:21.950570+0000","last_undegraded":"2024-01-26T23:49:21.950570+0000","last_fullsized":"2024-01-26T23:49:21.950570+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:41:32.548836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714568+0000","last_change":"2024-01-26T23:49:21.706806+0000","last_active":"2024-01-26T23:49:21.714568+0000","last_peered":"2024-01-26T23:49:21.714568+0000","last_clean":"2024-01-26T23:49:21.714568+0000","last_became_active":"2024-01-26T23:49:21.706309+0000","last_became_peered":"2024-01-26T23:49:21.706309+0000","last_unstale":"2024-01-26T23:49:21.714568+0000","last_undegraded":"2024-01-26T23:49:21.714568+0000","last_fullsized":"2024-01-26T23:49:21.714568+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:15.880089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.702136+0000","last_change":"2024-01-26T23:49:20.937835+0000","last_active":"2024-01-26T23:49:21.702136+0000","last_peered":"2024-01-26T23:49:21.702136+0000","last_clean":"2024-01-26T23:49:21.702136+0000","last_became_active":"2024-01-26T23:49:20.937594+0000","last_became_peered":"2024-01-26T23:49:20.937594+0000","last_unstale":"2024-01-26T23:49:21.702136+0000","last_undegraded":"2024-01-26T23:49:21.702136+0000","last_fullsized":"2024-01-26T23:49:21.702136+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:04:52.656671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714422+0000","last_change":"2024-01-26T23:49:21.709312+0000","last_active":"2024-01-26T23:49:21.714422+0000","last_peered":"2024-01-26T23:49:21.714422+0000","last_clean":"2024-01-26T23:49:21.714422+0000","last_became_active":"2024-01-26T23:49:21.709016+0000","last_became_peered":"2024-01-26T23:49:21.709016+0000","last_unstale":"2024-01-26T23:49:21.714422+0000","last_undegraded":"2024-01-26T23:49:21.714422+0000","last_fullsized":"2024-01-26T23:49:21.714422+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:15.415331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.947893+0000","last_change":"2024-01-26T23:49:20.930419+0000","last_active":"2024-01-26T23:49:21.947893+0000","last_peered":"2024-01-26T23:49:21.947893+0000","last_clean":"2024-01-26T23:49:21.947893+0000","last_became_active":"2024-01-26T23:49:20.930102+0000","last_became_peered":"2024-01-26T23:49:20.930102+0000","last_unstale":"2024-01-26T23:49:21.947893+0000","last_undegraded":"2024-01-26T23:49:21.947893+0000","last_fullsized":"2024-01-26T23:49:21.947893+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:01.796587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.709284+0000","last_change":"2024-01-26T23:49:21.703187+0000","last_active":"2024-01-26T23:49:21.709284+0000","last_peered":"2024-01-26T23:49:21.709284+0000","last_clean":"2024-01-26T23:49:21.709284+0000","last_became_active":"2024-01-26T23:49:21.702883+0000","last_became_peered":"2024-01-26T23:49:21.702883+0000","last_unstale":"2024-01-26T23:49:21.709284+0000","last_undegraded":"2024-01-26T23:49:21.709284+0000","last_fullsized":"2024-01-26T23:49:21.709284+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:23:30.759008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.701669+0000","last_change":"2024-01-26T23:49:20.931625+0000","last_active":"2024-01-26T23:49:21.701669+0000","last_peered":"2024-01-26T23:49:21.701669+0000","last_clean":"2024-01-26T23:49:21.701669+0000","last_became_active":"2024-01-26T23:49:20.930770+0000","last_became_peered":"2024-01-26T23:49:20.930770+0000","last_unstale":"2024-01-26T23:49:21.701669+0000","last_undegraded":"2024-01-26T23:49:21.701669+0000","last_fullsized":"2024-01-26T23:49:21.701669+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:05:23.745399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713219+0000","last_change":"2024-01-26T23:49:21.711158+0000","last_active":"2024-01-26T23:49:21.713219+0000","last_peered":"2024-01-26T23:49:21.713219+0000","last_clean":"2024-01-26T23:49:21.713219+0000","last_became_active":"2024-01-26T23:49:21.710828+0000","last_became_peered":"2024-01-26T23:49:21.710828+0000","last_unstale":"2024-01-26T23:49:21.713219+0000","last_undegraded":"2024-01-26T23:49:21.713219+0000","last_fullsized":"2024-01-26T23:49:21.713219+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:43:20.056916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.696741+0000","last_change":"2024-01-26T23:49:20.927120+0000","last_active":"2024-01-26T23:49:21.696741+0000","last_peered":"2024-01-26T23:49:21.696741+0000","last_clean":"2024-01-26T23:49:21.696741+0000","last_became_active":"2024-01-26T23:49:20.926800+0000","last_became_peered":"2024-01-26T23:49:20.926800+0000","last_unstale":"2024-01-26T23:49:21.696741+0000","last_undegraded":"2024-01-26T23:49:21.696741+0000","last_fullsized":"2024-01-26T23:49:21.696741+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:14.878260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.714485+0000","last_change":"2024-01-26T23:49:21.709755+0000","last_active":"2024-01-26T23:49:21.714485+0000","last_peered":"2024-01-26T23:49:21.714485+0000","last_clean":"2024-01-26T23:49:21.714485+0000","last_became_active":"2024-01-26T23:49:21.709547+0000","last_became_peered":"2024-01-26T23:49:21.709547+0000","last_unstale":"2024-01-26T23:49:21.714485+0000","last_undegraded":"2024-01-26T23:49:21.714485+0000","last_fullsized":"2024-01-26T23:49:21.714485+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:46:51.230370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.701015+0000","last_change":"2024-01-26T23:49:20.933709+0000","last_active":"2024-01-26T23:49:21.701015+0000","last_peered":"2024-01-26T23:49:21.701015+0000","last_clean":"2024-01-26T23:49:21.701015+0000","last_became_active":"2024-01-26T23:49:20.933444+0000","last_became_peered":"2024-01-26T23:49:20.933444+0000","last_unstale":"2024-01-26T23:49:21.701015+0000","last_undegraded":"2024-01-26T23:49:21.701015+0000","last_fullsized":"2024-01-26T23:49:21.701015+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:41:45.760005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.711298+0000","last_change":"2024-01-26T23:49:21.704604+0000","last_active":"2024-01-26T23:49:21.711298+0000","last_peered":"2024-01-26T23:49:21.711298+0000","last_clean":"2024-01-26T23:49:21.711298+0000","last_became_active":"2024-01-26T23:49:21.704277+0000","last_became_peered":"2024-01-26T23:49:21.704277+0000","last_unstale":"2024-01-26T23:49:21.711298+0000","last_undegraded":"2024-01-26T23:49:21.711298+0000","last_fullsized":"2024-01-26T23:49:21.711298+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:01:33.387704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.951229+0000","last_change":"2024-01-26T23:49:20.926602+0000","last_active":"2024-01-26T23:49:21.951229+0000","last_peered":"2024-01-26T23:49:21.951229+0000","last_clean":"2024-01-26T23:49:21.951229+0000","last_became_active":"2024-01-26T23:49:20.926306+0000","last_became_peered":"2024-01-26T23:49:20.926306+0000","last_unstale":"2024-01-26T23:49:21.951229+0000","last_undegraded":"2024-01-26T23:49:21.951229+0000","last_fullsized":"2024-01-26T23:49:21.951229+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:58:14.912115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.715403+0000","last_change":"2024-01-26T23:49:21.709775+0000","last_active":"2024-01-26T23:49:21.715403+0000","last_peered":"2024-01-26T23:49:21.715403+0000","last_clean":"2024-01-26T23:49:21.715403+0000","last_became_active":"2024-01-26T23:49:21.709484+0000","last_became_peered":"2024-01-26T23:49:21.709484+0000","last_unstale":"2024-01-26T23:49:21.715403+0000","last_undegraded":"2024-01-26T23:49:21.715403+0000","last_fullsized":"2024-01-26T23:49:21.715403+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:19:56.848512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"145'1","reported_seq":11,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-01-26T23:49:23.293810+0000","last_change":"2024-01-26T23:49:20.938849+0000","last_active":"2024-01-26T23:49:23.293810+0000","last_peered":"2024-01-26T23:49:23.293810+0000","last_clean":"2024-01-26T23:49:23.293810+0000","last_became_active":"2024-01-26T23:49:20.938634+0000","last_became_peered":"2024-01-26T23:49:20.938634+0000","last_unstale":"2024-01-26T23:49:23.293810+0000","last_undegraded":"2024-01-26T23:49:23.293810+0000","last_fullsized":"2024-01-26T23:49:23.293810+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:19:55.846683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.720486+0000","last_change":"2024-01-26T23:49:21.715705+0000","last_active":"2024-01-26T23:49:21.720486+0000","last_peered":"2024-01-26T23:49:21.720486+0000","last_clean":"2024-01-26T23:49:21.720486+0000","last_became_active":"2024-01-26T23:49:21.715391+0000","last_became_peered":"2024-01-26T23:49:21.715391+0000","last_unstale":"2024-01-26T23:49:21.720486+0000","last_undegraded":"2024-01-26T23:49:21.720486+0000","last_fullsized":"2024-01-26T23:49:21.720486+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:46:51.230370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.706792+0000","last_change":"2024-01-26T23:49:20.931200+0000","last_active":"2024-01-26T23:49:21.706792+0000","last_peered":"2024-01-26T23:49:21.706792+0000","last_clean":"2024-01-26T23:49:21.706792+0000","last_became_active":"2024-01-26T23:49:20.930968+0000","last_became_peered":"2024-01-26T23:49:20.930968+0000","last_unstale":"2024-01-26T23:49:21.706792+0000","last_undegraded":"2024-01-26T23:49:21.706792+0000","last_fullsized":"2024-01-26T23:49:21.706792+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:01:51.605788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.711848+0000","last_change":"2024-01-26T23:49:21.705136+0000","last_active":"2024-01-26T23:49:21.711848+0000","last_peered":"2024-01-26T23:49:21.711848+0000","last_clean":"2024-01-26T23:49:21.711848+0000","last_became_active":"2024-01-26T23:49:21.704849+0000","last_became_peered":"2024-01-26T23:49:21.704849+0000","last_unstale":"2024-01-26T23:49:21.711848+0000","last_undegraded":"2024-01-26T23:49:21.711848+0000","last_fullsized":"2024-01-26T23:49:21.711848+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:54.174596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.696843+0000","last_change":"2024-01-26T23:49:20.935623+0000","last_active":"2024-01-26T23:49:21.696843+0000","last_peered":"2024-01-26T23:49:21.696843+0000","last_clean":"2024-01-26T23:49:21.696843+0000","last_became_active":"2024-01-26T23:49:20.935325+0000","last_became_peered":"2024-01-26T23:49:20.935325+0000","last_unstale":"2024-01-26T23:49:21.696843+0000","last_undegraded":"2024-01-26T23:49:21.696843+0000","last_fullsized":"2024-01-26T23:49:21.696843+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:29:03.326840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.716154+0000","last_change":"2024-01-26T23:49:21.713771+0000","last_active":"2024-01-26T23:49:21.716154+0000","last_peered":"2024-01-26T23:49:21.716154+0000","last_clean":"2024-01-26T23:49:21.716154+0000","last_became_active":"2024-01-26T23:49:21.713471+0000","last_became_peered":"2024-01-26T23:49:21.713471+0000","last_unstale":"2024-01-26T23:49:21.716154+0000","last_undegraded":"2024-01-26T23:49:21.716154+0000","last_fullsized":"2024-01-26T23:49:21.716154+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:46:24.616619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.702410+0000","last_change":"2024-01-26T23:49:20.933716+0000","last_active":"2024-01-26T23:49:21.702410+0000","last_peered":"2024-01-26T23:49:21.702410+0000","last_clean":"2024-01-26T23:49:21.702410+0000","last_became_active":"2024-01-26T23:49:20.933447+0000","last_became_peered":"2024-01-26T23:49:20.933447+0000","last_unstale":"2024-01-26T23:49:21.702410+0000","last_undegraded":"2024-01-26T23:49:21.702410+0000","last_fullsized":"2024-01-26T23:49:21.702410+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:44:37.740189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.712186+0000","last_change":"2024-01-26T23:49:21.709903+0000","last_active":"2024-01-26T23:49:21.712186+0000","last_peered":"2024-01-26T23:49:21.712186+0000","last_clean":"2024-01-26T23:49:21.712186+0000","last_became_active":"2024-01-26T23:49:21.709608+0000","last_became_peered":"2024-01-26T23:49:21.709608+0000","last_unstale":"2024-01-26T23:49:21.712186+0000","last_undegraded":"2024-01-26T23:49:21.712186+0000","last_fullsized":"2024-01-26T23:49:21.712186+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:44:38.742019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.707040+0000","last_change":"2024-01-26T23:49:20.928339+0000","last_active":"2024-01-26T23:49:21.707040+0000","last_peered":"2024-01-26T23:49:21.707040+0000","last_clean":"2024-01-26T23:49:21.707040+0000","last_became_active":"2024-01-26T23:49:20.927842+0000","last_became_peered":"2024-01-26T23:49:20.927842+0000","last_unstale":"2024-01-26T23:49:21.707040+0000","last_undegraded":"2024-01-26T23:49:21.707040+0000","last_fullsized":"2024-01-26T23:49:21.707040+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:43:19.055087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.713297+0000","last_change":"2024-01-26T23:49:21.709300+0000","last_active":"2024-01-26T23:49:21.713297+0000","last_peered":"2024-01-26T23:49:21.713297+0000","last_clean":"2024-01-26T23:49:21.713297+0000","last_became_active":"2024-01-26T23:49:21.709008+0000","last_became_peered":"2024-01-26T23:49:21.709008+0000","last_unstale":"2024-01-26T23:49:21.713297+0000","last_undegraded":"2024-01-26T23:49:21.713297+0000","last_fullsized":"2024-01-26T23:49:21.713297+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:34.441797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.694324+0000","last_change":"2024-01-26T23:49:20.935691+0000","last_active":"2024-01-26T23:49:21.694324+0000","last_peered":"2024-01-26T23:49:21.694324+0000","last_clean":"2024-01-26T23:49:21.694324+0000","last_became_active":"2024-01-26T23:49:20.935362+0000","last_became_peered":"2024-01-26T23:49:20.935362+0000","last_unstale":"2024-01-26T23:49:21.694324+0000","last_undegraded":"2024-01-26T23:49:21.694324+0000","last_fullsized":"2024-01-26T23:49:21.694324+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:56:22.957242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.708198+0000","last_change":"2024-01-26T23:49:21.705845+0000","last_active":"2024-01-26T23:49:21.708198+0000","last_peered":"2024-01-26T23:49:21.708198+0000","last_clean":"2024-01-26T23:49:21.708198+0000","last_became_active":"2024-01-26T23:49:21.705559+0000","last_became_peered":"2024-01-26T23:49:21.705559+0000","last_unstale":"2024-01-26T23:49:21.708198+0000","last_undegraded":"2024-01-26T23:49:21.708198+0000","last_fullsized":"2024-01-26T23:49:21.708198+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:00:34.207070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.701020+0000","last_change":"2024-01-26T23:49:20.931628+0000","last_active":"2024-01-26T23:49:21.701020+0000","last_peered":"2024-01-26T23:49:21.701020+0000","last_clean":"2024-01-26T23:49:21.701020+0000","last_became_active":"2024-01-26T23:49:20.931022+0000","last_became_peered":"2024-01-26T23:49:20.931022+0000","last_unstale":"2024-01-26T23:49:21.701020+0000","last_undegraded":"2024-01-26T23:49:21.701020+0000","last_fullsized":"2024-01-26T23:49:21.701020+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:28:33.436885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.717191+0000","last_change":"2024-01-26T23:49:21.713146+0000","last_active":"2024-01-26T23:49:21.717191+0000","last_peered":"2024-01-26T23:49:21.717191+0000","last_clean":"2024-01-26T23:49:21.717191+0000","last_became_active":"2024-01-26T23:49:21.712684+0000","last_became_peered":"2024-01-26T23:49:21.712684+0000","last_unstale":"2024-01-26T23:49:21.717191+0000","last_undegraded":"2024-01-26T23:49:21.717191+0000","last_fullsized":"2024-01-26T23:49:21.717191+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:20.909706+0000","last_clean_scrub_stamp":"2024-01-26T23:49:20.909706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:51.588887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"144'8","reported_seq":18,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.950934+0000","last_change":"2024-01-26T23:49:20.933578+0000","last_active":"2024-01-26T23:49:21.950934+0000","last_peered":"2024-01-26T23:49:21.950934+0000","last_clean":"2024-01-26T23:49:21.950934+0000","last_became_active":"2024-01-26T23:49:20.933286+0000","last_became_peered":"2024-01-26T23:49:20.933286+0000","last_unstale":"2024-01-26T23:49:21.950934+0000","last_undegraded":"2024-01-26T23:49:21.950934+0000","last_fullsized":"2024-01-26T23:49:21.950934+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:49:19.907877+0000","last_clean_scrub_stamp":"2024-01-26T23:49:19.907877+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:52:55.167098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"135'8584","reported_seq":9106,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-26T23:49:21.701211+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:49:21.701211+0000","last_peered":"2024-01-26T23:49:21.701211+0000","last_clean":"2024-01-26T23:49:21.701211+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:49:21.701211+0000","last_undegraded":"2024-01-26T23:49:21.701211+0000","last_fullsized":"2024-01-26T23:49:21.701211+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":2314,"num_write":360,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":2314,"num_write":360,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8584,"ondisk_log_size":8584,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575084,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311932,"kb_used_data":2356,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059908,"statfs":{"total":96636764160,"available":96317345792,"internally_reserved":0,"allocated":2412544,"data_stored":1521090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51500000000000001,"15min":0.56200000000000006},"min":{"1min":0.40699999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.74199999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52700000000000002,"15min":0.57599999999999996},"min":{"1min":0.35899999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67200000000000004,"5min":0.82499999999999996,"15min":0.83999999999999997},"last":0.69199999999999995}]},{"osd":1,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54000000000000004,"15min":0.57399999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.77600000000000002,"5min":0.77800000000000002,"15min":0.82899999999999996},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55100000000000005,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.70399999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.57199999999999995},"min":{"1min":0.38700000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69199999999999995,"5min":0.80000000000000004,"15min":0.81699999999999995},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.38900000000000001,"5min":0.30199999999999999,"15min":0.27400000000000002},"max":{"1min":0.746,"5min":0.81999999999999995,"15min":0.88600000000000001},"last":0.55200000000000005}]},{"osd":3,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54700000000000004,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.377,"15min":0.377},"max":{"1min":0.70399999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57499999999999996,"15min":0.59699999999999998},"min":{"1min":0.34399999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77900000000000003,"5min":0.86199999999999999,"15min":0.86399999999999999},"last":0.60199999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.438,"15min":0.45200000000000001},"min":{"1min":0.34599999999999997,"5min":0.252,"15min":0.247},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.93899999999999995},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.45300000000000001,"15min":0.47399999999999998},"min":{"1min":0.32400000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67300000000000004,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.379}]},{"osd":5,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47599999999999998,"15min":0.48699999999999999},"min":{"1min":0.30499999999999999,"5min":0.27200000000000002,"15min":0.22800000000000001},"max":{"1min":0.749,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.504,"15min":0.52300000000000002},"min":{"1min":0.35799999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73399999999999999,"5min":0.74199999999999999,"15min":0.90200000000000002},"last":0.441}]},{"osd":6,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53100000000000003,"15min":0.54800000000000004},"min":{"1min":0.41699999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76400000000000001,"5min":0.78700000000000003,"15min":0.95899999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.52700000000000002,"15min":0.54000000000000004},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75900000000000001,"5min":0.79500000000000004,"15min":0.85199999999999998},"last":0.42499999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575084,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303148,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068692,"statfs":{"total":96636764160,"available":96326340608,"internally_reserved":0,"allocated":1806336,"data_stored":930528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52900000000000003,"15min":0.56499999999999995},"min":{"1min":0.40300000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.423,"15min":1.423},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.57899999999999996},"min":{"1min":0.36099999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.91100000000000003},"last":0.60799999999999998}]},{"osd":1,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51700000000000002,"15min":0.55800000000000005},"min":{"1min":0.32200000000000001,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.623},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.58399999999999996},"min":{"1min":0.38200000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.66800000000000004,"5min":1.498,"15min":1.498},"last":0.54100000000000004}]},{"osd":2,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52800000000000002,"15min":0.56999999999999995},"min":{"1min":0.38100000000000001,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.85799999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52600000000000002,"15min":0.57099999999999995},"min":{"1min":0.377,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69299999999999995,"5min":0.77000000000000002,"15min":0.82799999999999996},"last":0.56100000000000005}]},{"osd":3,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57199999999999995,"15min":0.59099999999999997},"min":{"1min":0.34699999999999998,"5min":0.33800000000000002,"15min":0.29999999999999999},"max":{"1min":0.746,"5min":1.595,"15min":1.595},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56899999999999995,"15min":0.60599999999999998},"min":{"1min":0.45600000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.751,"5min":1.369,"15min":1.369},"last":0.68000000000000005}]},{"osd":4,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.443,"15min":0.46200000000000002},"min":{"1min":0.33800000000000002,"5min":0.245,"15min":0.22},"max":{"1min":0.67000000000000004,"5min":1.556,"15min":1.556},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.47399999999999998,"15min":0.48199999999999998},"min":{"1min":0.33200000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.86199999999999999},"last":0.44700000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45500000000000002,"15min":0.47099999999999997},"min":{"1min":0.32700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67900000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.51700000000000002,"15min":0.52000000000000002},"min":{"1min":0.30499999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.70999999999999996,"5min":1.47,"15min":1.47},"last":0.47799999999999998}]},{"osd":7,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.5,"15min":0.51800000000000002},"min":{"1min":0.40100000000000002,"5min":0.26700000000000002,"15min":0.246},"max":{"1min":0.80200000000000005,"5min":0.81799999999999995,"15min":0.85899999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50900000000000001,"15min":0.52400000000000002},"min":{"1min":0.32300000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.80400000000000005},"last":0.52000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575084,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307244,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064596,"statfs":{"total":96636764160,"available":96322146304,"internally_reserved":0,"allocated":1806336,"data_stored":929996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41199999999999998,"15min":0.40600000000000003},"min":{"1min":0.28100000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.72599999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.433,"5min":0.41999999999999998,"15min":0.439},"min":{"1min":0.28100000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.61499999999999999,"5min":0.78600000000000003,"15min":1.337},"last":0.58699999999999997}]},{"osd":2,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.39700000000000002,"15min":0.40899999999999997},"min":{"1min":0.28699999999999998,"5min":0.254,"15min":0.23599999999999999},"max":{"1min":0.66300000000000003,"5min":0.68799999999999994,"15min":0.73299999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.46300000000000002,"15min":0.47199999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59299999999999997,"5min":0.74099999999999999,"15min":0.77300000000000002},"last":0.50700000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51300000000000001,"15min":0.51400000000000001},"min":{"1min":0.38400000000000001,"5min":0.34399999999999997,"15min":0.33200000000000002},"max":{"1min":0.73499999999999999,"5min":0.78900000000000003,"15min":1.782},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.32900000000000001,"5min":0.32000000000000001,"15min":0.26600000000000001},"max":{"1min":0.70199999999999996,"5min":0.71499999999999997,"15min":1.379},"last":0.63400000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60699999999999998,"15min":0.64200000000000002},"min":{"1min":0.45100000000000001,"5min":0.40799999999999997,"15min":0.40200000000000002},"max":{"1min":0.85199999999999998,"5min":0.89200000000000002,"15min":1.0820000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.64500000000000002,"15min":0.65700000000000003},"min":{"1min":0.49199999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.65200000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.64000000000000001,"15min":0.66800000000000004},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45000000000000001},"max":{"1min":0.86499999999999999,"5min":0.91100000000000003,"15min":1.208},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.63400000000000001,"15min":0.65900000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.42499999999999999},"max":{"1min":0.85099999999999998,"5min":0.96399999999999997,"15min":1.018},"last":0.76000000000000001}]},{"osd":6,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.63800000000000001,"15min":0.67000000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.42599999999999999},"max":{"1min":0.97399999999999998,"5min":1.022,"15min":1.2529999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.65000000000000002,"15min":0.67700000000000005},"min":{"1min":0.47899999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92300000000000004,"5min":0.93200000000000005,"15min":1.022},"last":0.78500000000000003}]},{"osd":7,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.68300000000000005,"15min":0.70699999999999996},"min":{"1min":0.5,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.71899999999999997},"min":{"1min":0.505,"5min":0.496,"15min":0.496},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.296},"last":0.97299999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575083,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307824,"kb_used_data":2344,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064016,"statfs":{"total":96636764160,"available":96321552384,"internally_reserved":0,"allocated":2400256,"data_stored":1520364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.40600000000000003},"min":{"1min":0.25800000000000001,"5min":0.23899999999999999,"15min":0.23100000000000001},"max":{"1min":0.88500000000000001,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.42999999999999999,"15min":0.45100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65800000000000003,"5min":1.363,"15min":1.4490000000000001},"last":0.22700000000000001}]},{"osd":2,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.41099999999999998,"15min":0.42299999999999999},"min":{"1min":0.28899999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.89600000000000002},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.33900000000000002,"5min":0.32700000000000001,"15min":0.28000000000000003},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.31900000000000001}]},{"osd":3,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.49199999999999999,"15min":0.47399999999999998},"min":{"1min":0.32800000000000001,"5min":0.32300000000000001,"15min":0.30599999999999999},"max":{"1min":0.85199999999999998,"5min":2.145,"15min":2.145},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.496,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36099999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":1.198},"last":0.378}]},{"osd":4,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60899999999999999,"15min":0.63800000000000001},"min":{"1min":0.41699999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78600000000000003,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.59699999999999998,"15min":0.63700000000000001},"min":{"1min":0.44400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.94299999999999995},"last":0.60999999999999999}]},{"osd":5,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":1.329,"15min":1.329},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.66300000000000003,"15min":0.67000000000000004},"min":{"1min":0.46899999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82799999999999996,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.67300000000000004}]},{"osd":6,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.61799999999999999,"15min":0.65100000000000002},"min":{"1min":0.47299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.89800000000000002,"5min":0.98699999999999999,"15min":1.032},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.65400000000000003,"15min":0.68000000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.41799999999999998},"max":{"1min":0.93100000000000005,"5min":1.286,"15min":1.286},"last":0.75900000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.70599999999999996,"15min":0.81699999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.435},"max":{"1min":0.81799999999999995,"5min":4.8710000000000004,"15min":25.965},"last":0.625},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.73899999999999999,"15min":0.86099999999999999},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.433},"max":{"1min":0.88200000000000001,"5min":4.9199999999999999,"15min":33.085000000000001},"last":0.73899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575083,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307240,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064600,"statfs":{"total":96636764160,"available":96322150400,"internally_reserved":0,"allocated":1802240,"data_stored":929974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.379,"15min":0.39100000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.255},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.314},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.43099999999999999,"15min":0.42599999999999999},"min":{"1min":0.29699999999999999,"5min":0.255,"15min":0.255},"max":{"1min":0.59899999999999998,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.38900000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.435,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.26900000000000002,"15min":0.248},"max":{"1min":0.63500000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52200000000000002,"15min":0.51100000000000001},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.68300000000000005,"5min":1.171,"15min":1.171},"last":0.41999999999999998}]},{"osd":3,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.46899999999999997,"15min":0.48399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.313},"max":{"1min":0.64300000000000002,"5min":0.72799999999999998,"15min":0.74099999999999999},"last":0.371},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47699999999999998},"min":{"1min":0.32900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70599999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.35699999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59599999999999997,"15min":0.628},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.087},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.625,"15min":0.64300000000000002},"min":{"1min":0.39100000000000001,"5min":0.371,"15min":0.371},"max":{"1min":1.006,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.51300000000000001}]},{"osd":5,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.63900000000000001,"15min":0.65500000000000003},"min":{"1min":0.42299999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94699999999999995,"5min":1.24,"15min":1.24},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.65200000000000002,"15min":0.67400000000000004},"min":{"1min":0.48799999999999999,"5min":0.41799999999999998,"15min":0.35199999999999998},"max":{"1min":0.874,"5min":0.97699999999999998,"15min":1.085},"last":0.46100000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65700000000000003,"15min":0.67700000000000005},"min":{"1min":0.41199999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88600000000000001,"5min":1.022,"15min":1.1080000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.65200000000000002,"15min":0.67800000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.38800000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.1299999999999999},"last":0.60599999999999998}]},{"osd":7,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68400000000000005,"15min":0.70099999999999996},"min":{"1min":0.46400000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.88800000000000001,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.69699999999999995,"15min":0.72499999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.42099999999999999},"max":{"1min":0.96899999999999997,"5min":1.05,"15min":1.139},"last":0.62}]}]},{"osd":3,"up_from":13,"seq":55834575082,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307256,"kb_used_data":1776,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064584,"statfs":{"total":96636764160,"available":96322134016,"internally_reserved":0,"allocated":1818624,"data_stored":931544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38400000000000001,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.55400000000000005,"5min":0.59699999999999998,"15min":1.8540000000000001},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.42899999999999999,"15min":0.438},"min":{"1min":0.29799999999999999,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":1.2290000000000001},"last":0.36599999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44500000000000001,"15min":0.45200000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":1.633},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.47999999999999998,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":1.0760000000000001},"last":0.43099999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48399999999999999,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65000000000000002,"5min":0.70399999999999996,"15min":1.4059999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.44500000000000001,"15min":0.45800000000000002},"min":{"1min":0.34100000000000003,"5min":0.29499999999999998,"15min":0.28299999999999997},"max":{"1min":0.629,"5min":0.69399999999999995,"15min":1.1870000000000001},"last":0.45800000000000002}]},{"osd":4,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.63,"15min":0.65800000000000003},"min":{"1min":0.48299999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":3.306},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.44800000000000001,"5min":0.40100000000000002,"15min":0.379},"max":{"1min":0.81499999999999995,"5min":0.85199999999999998,"15min":1.008},"last":0.59399999999999997}]},{"osd":5,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.63600000000000001,"15min":0.65200000000000002},"min":{"1min":0.48599999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.85199999999999998,"5min":0.90400000000000003,"15min":1.593},"last":0.62},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.47599999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.86299999999999999,"5min":0.92300000000000004,"15min":1.3819999999999999},"last":0.66500000000000004}]},{"osd":6,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.66900000000000004},"min":{"1min":0.42299999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":1.506},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67200000000000004,"15min":0.68999999999999995},"min":{"1min":0.55600000000000005,"5min":0.46700000000000003,"15min":0.44400000000000001},"max":{"1min":0.88500000000000001,"5min":0.89000000000000001,"15min":1.427},"last":0.70699999999999996}]},{"osd":7,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64400000000000002,"15min":0.68799999999999994},"min":{"1min":0.46700000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":3.238},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.71199999999999997,"15min":0.73399999999999999},"min":{"1min":0.54200000000000004,"5min":0.51900000000000002,"15min":0.48899999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":3.351},"last":0.69499999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575084,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303156,"kb_used_data":1772,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068684,"statfs":{"total":96636764160,"available":96326332416,"internally_reserved":0,"allocated":1814528,"data_stored":931510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52300000000000002,"15min":0.56200000000000006},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.72499999999999998,"5min":0.78900000000000003,"15min":1.21},"last":0.433},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.32700000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.751,"5min":5.367,"15min":5.367},"last":0.49299999999999999}]},{"osd":1,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.56899999999999995,"15min":0.58599999999999997},"min":{"1min":0.42699999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.79300000000000004,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54100000000000004,"15min":0.57199999999999995},"min":{"1min":0.41399999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73599999999999999,"5min":0.80600000000000005,"15min":0.81699999999999995},"last":0.56999999999999995}]},{"osd":2,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54500000000000004,"15min":0.56399999999999995},"min":{"1min":0.38500000000000001,"5min":0.33900000000000002,"15min":0.32700000000000001},"max":{"1min":0.751,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57099999999999995,"15min":0.59499999999999997},"min":{"1min":0.44500000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73399999999999999,"5min":0.86199999999999999,"15min":1.7050000000000001},"last":0.55900000000000005}]},{"osd":3,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56899999999999995,"15min":0.58499999999999996},"min":{"1min":0.45600000000000002,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.70499999999999996,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56599999999999995,"15min":0.58099999999999996},"min":{"1min":0.433,"5min":0.37,"15min":0.35699999999999998},"max":{"1min":0.78300000000000003,"5min":0.94199999999999995,"15min":1.05},"last":0.34399999999999997}]},{"osd":5,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45400000000000001,"15min":0.46500000000000002},"min":{"1min":0.32300000000000001,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.60499999999999998,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.378},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.47899999999999998,"15min":0.48599999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64300000000000002,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.35699999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.46200000000000002,"15min":0.47399999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27200000000000002},"max":{"1min":0.63500000000000001,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.38600000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72099999999999997,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.44600000000000001}]},{"osd":7,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52600000000000002,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.72499999999999998,"5min":0.879,"15min":0.879},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55800000000000005,"15min":0.57199999999999995},"min":{"1min":0.437,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.74199999999999999,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.46000000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575084,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303152,"kb_used_data":1768,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068688,"statfs":{"total":96636764160,"available":96326336512,"internally_reserved":0,"allocated":1810432,"data_stored":930134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.56399999999999995},"min":{"1min":0.35199999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.71099999999999997,"5min":0.81799999999999995,"15min":2.0289999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52700000000000002,"15min":0.56699999999999995},"min":{"1min":0.41999999999999998,"5min":0.29999999999999999,"15min":0.28899999999999998},"max":{"1min":0.70799999999999996,"5min":0.77400000000000002,"15min":0.82199999999999995},"last":0.51200000000000001}]},{"osd":1,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56299999999999994,"15min":0.60399999999999998},"min":{"1min":0.443,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":0.754,"5min":0.79500000000000004,"15min":2.3319999999999999},"last":0.501},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59899999999999998,"15min":0.626},"min":{"1min":0.45800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76400000000000001,"5min":0.80500000000000005,"15min":1.1739999999999999},"last":0.46000000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53700000000000003,"15min":0.55700000000000005},"min":{"1min":0.36799999999999999,"5min":0.34499999999999997,"15min":0.30099999999999999},"max":{"1min":0.68400000000000005,"5min":0.79400000000000004,"15min":0.83799999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55300000000000005,"15min":0.58699999999999997},"min":{"1min":0.41299999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72899999999999998,"5min":0.81100000000000005,"15min":2.0600000000000001},"last":0.55100000000000005}]},{"osd":3,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55300000000000005,"15min":0.57499999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.314},"max":{"1min":0.69699999999999995,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55900000000000005,"15min":0.58699999999999997},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73999999999999999,"5min":0.77900000000000003,"15min":2.5630000000000002},"last":0.48599999999999999}]},{"osd":4,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.42899999999999999,"15min":0.45500000000000002},"min":{"1min":0.313,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.65100000000000002,"5min":0.67200000000000004,"15min":2.3700000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45900000000000002,"15min":0.48099999999999998},"min":{"1min":0.32600000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.68200000000000005,"5min":0.82699999999999996,"15min":0.89200000000000002},"last":0.40799999999999997}]},{"osd":6,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.47199999999999998},"min":{"1min":0.34300000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66900000000000004,"5min":0.67700000000000005,"15min":2.5990000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.49299999999999999,"15min":0.51400000000000001},"min":{"1min":0.374,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71399999999999997,"5min":0.76200000000000001,"15min":1.99},"last":0.47099999999999997}]},{"osd":7,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51400000000000001,"15min":0.53800000000000003},"min":{"1min":0.374,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73799999999999999,"5min":0.76200000000000001,"15min":3.5270000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54400000000000004,"15min":0.55900000000000005},"min":{"1min":0.43099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78900000000000003,"5min":0.81499999999999995,"15min":0.82199999999999995},"last":0.52800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:49:31.273 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-26T23:49:31.273 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-26T23:49:31.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.448+0000 7f0890882700 1 Processor -- start 2024-01-26T23:49:31.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.448+0000 7f0890882700 1 -- start start 2024-01-26T23:49:31.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.449+0000 7f0890882700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f088c068b00 0x7f088c068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.449+0000 7f0890882700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f088c06a660 0x7f088c159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.449+0000 7f0890882700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f088c0a9a80 con 0x7f088c06add0 2024-01-26T23:49:31.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.449+0000 7f0890882700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f088c06aab0 con 0x7f088c068b00 2024-01-26T23:49:31.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.449+0000 7f0890882700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088c15a3e0 con 0x7f088c06a660 2024-01-26T23:49:31.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.449+0000 7f088a59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f088c068b00 0x7f088c068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.449+0000 7f088a59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f088c068b00 0x7f088c068ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59962/0 (socket says 172.21.15.46:59962) 2024-01-26T23:49:31.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.449+0000 7f088a59c700 1 -- 172.21.15.46:0/656301432 learned_addr learned my addr 172.21.15.46:0/656301432 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:31.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.449+0000 7f0889d9b700 1 --2- 172.21.15.46:0/656301432 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f088c06a660 0x7f088c159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.449+0000 7f088a59c700 1 -- 172.21.15.46:0/656301432 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f088c06a660 msgr2=0x7f088c159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:31.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.449+0000 7f088a59c700 1 --2- 172.21.15.46:0/656301432 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f088c06a660 0x7f088c159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.449+0000 7f088a59c700 1 -- 172.21.15.46:0/656301432 >> v1:172.21.15.106:6789/0 conn(0x7f088c06add0 legacy=0x7f088c15d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:31.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.449+0000 7f088a59c700 1 -- 172.21.15.46:0/656301432 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c15a680 con 0x7f088c068b00 2024-01-26T23:49:31.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.449+0000 7f0889d9b700 1 --2- 172.21.15.46:0/656301432 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f088c06a660 0x7f088c159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:49:31.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.450+0000 7f088a59c700 1 --2- 172.21.15.46:0/656301432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f088c068b00 0x7f088c068ed0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f0874009990 tx=0x7f0874009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb2a2603ebe27c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:31.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.450+0000 7f088959a700 1 -- 172.21.15.46:0/656301432 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0874005c10 con 0x7f088c068b00 2024-01-26T23:49:31.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.450+0000 7f088959a700 1 -- 172.21.15.46:0/656301432 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0874005d70 con 0x7f088c068b00 2024-01-26T23:49:31.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.450+0000 7f088959a700 1 -- 172.21.15.46:0/656301432 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087400b840 con 0x7f088c068b00 2024-01-26T23:49:31.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.450+0000 7f0890882700 1 -- 172.21.15.46:0/656301432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f088c068b00 msgr2=0x7f088c068ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:31.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.450+0000 7f0890882700 1 --2- 172.21.15.46:0/656301432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f088c068b00 0x7f088c068ed0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f0874009990 tx=0x7f0874009220 comp rx=0 tx=0).stop 2024-01-26T23:49:31.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.450+0000 7f0890882700 1 -- 172.21.15.46:0/656301432 shutdown_connections 2024-01-26T23:49:31.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.450+0000 7f0890882700 1 --2- 172.21.15.46:0/656301432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f088c068b00 0x7f088c068ed0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.450+0000 7f0890882700 1 --2- 172.21.15.46:0/656301432 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f088c06a660 0x7f088c159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.450+0000 7f0890882700 1 -- 172.21.15.46:0/656301432 >> 172.21.15.46:0/656301432 conn(0x7f088c0ca100 msgr2=0x7f088c06e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:31.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.450+0000 7f0890882700 1 -- 172.21.15.46:0/656301432 shutdown_connections 2024-01-26T23:49:31.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.450+0000 7f0890882700 1 -- 172.21.15.46:0/656301432 wait complete. 2024-01-26T23:49:31.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.452+0000 7f0890882700 1 Processor -- start 2024-01-26T23:49:31.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.452+0000 7f0890882700 1 -- start start 2024-01-26T23:49:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.452+0000 7f0890882700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f088c068b00 0x7f088c16a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.452+0000 7f0890882700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f088c06a660 0x7f088c169e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.452+0000 7f0890882700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f088c06add0 0x7f088c168d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.452+0000 7f0890882700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c16b820 con 0x7f088c06add0 2024-01-26T23:49:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.452+0000 7f0890882700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c16ac10 con 0x7f088c068b00 2024-01-26T23:49:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.452+0000 7f0890882700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088c16aef0 con 0x7f088c06a660 2024-01-26T23:49:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.453+0000 7f088ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f088c06add0 0x7f088c168d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.453+0000 7f088ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f088c06add0 0x7f088c168d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59970/0 (socket says 172.21.15.46:59970) 2024-01-26T23:49:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.453+0000 7f088ad9d700 1 -- 172.21.15.46:0/2636715588 learned_addr learned my addr 172.21.15.46:0/2636715588 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.453+0000 7f0889d9b700 1 --2- 172.21.15.46:0/2636715588 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f088c06a660 0x7f088c169e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.453+0000 7f088a59c700 1 --2- 172.21.15.46:0/2636715588 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f088c068b00 0x7f088c16a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.453+0000 7f088ad9d700 1 -- 172.21.15.46:0/2636715588 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f088c06a660 msgr2=0x7f088c169e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.453+0000 7f088ad9d700 1 --2- 172.21.15.46:0/2636715588 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f088c06a660 0x7f088c169e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.453+0000 7f088ad9d700 1 -- 172.21.15.46:0/2636715588 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f088c068b00 msgr2=0x7f088c16a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.453+0000 7f088ad9d700 1 --2- 172.21.15.46:0/2636715588 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f088c068b00 0x7f088c16a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.453+0000 7f088ad9d700 1 -- 172.21.15.46:0/2636715588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0874009480 con 0x7f088c06add0 2024-01-26T23:49:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.453+0000 7f088ad9d700 1 --2- 172.21.15.46:0/2636715588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f088c06add0 0x7f088c168d20 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f088400c930 tx=0x7f08840026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.453+0000 7f08827fc700 1 -- 172.21.15.46:0/2636715588 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0884012630 con 0x7f088c06add0 2024-01-26T23:49:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.453+0000 7f08827fc700 1 -- 172.21.15.46:0/2636715588 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08840127d0 con 0x7f088c06add0 2024-01-26T23:49:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.453+0000 7f0890882700 1 -- 172.21.15.46:0/2636715588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f088c16b1d0 con 0x7f088c06add0 2024-01-26T23:49:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.453+0000 7f08827fc700 1 -- 172.21.15.46:0/2636715588 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0884012b90 con 0x7f088c06add0 2024-01-26T23:49:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.453+0000 7f0890882700 1 -- 172.21.15.46:0/2636715588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f088c16a3c0 con 0x7f088c06add0 2024-01-26T23:49:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.454+0000 7f0890882700 1 -- 172.21.15.46:0/2636715588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f088c04c710 con 0x7f088c06add0 2024-01-26T23:49:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.455+0000 7f08827fc700 1 -- 172.21.15.46:0/2636715588 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f088401a3b0 con 0x7f088c06add0 2024-01-26T23:49:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.455+0000 7f08827fc700 1 --2- 172.21.15.46:0/2636715588 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f086c066f30 0x7f086c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.455+0000 7f08827fc700 1 -- 172.21.15.46:0/2636715588 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f0884084520 con 0x7f088c06add0 2024-01-26T23:49:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.455+0000 7f088a59c700 1 --2- 172.21.15.46:0/2636715588 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f086c066f30 0x7f086c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.456+0000 7f088a59c700 1 --2- 172.21.15.46:0/2636715588 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f086c066f30 0x7f086c0693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f08740030c0 tx=0x7f0874015000 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:31.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.460+0000 7f08827fc700 1 -- 172.21.15.46:0/2636715588 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f08840514c0 con 0x7f088c06add0 2024-01-26T23:49:31.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.749+0000 7f0890882700 1 -- 172.21.15.46:0/2636715588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f088c06edf0 con 0x7f088c06add0 2024-01-26T23:49:31.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.753+0000 7f08827fc700 1 -- 172.21.15.46:0/2636715588 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v23) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0884054ae0 con 0x7f088c06add0 2024-01-26T23:49:31.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.755+0000 7f0890882700 1 -- 172.21.15.46:0/2636715588 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f086c066f30 msgr2=0x7f086c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:31.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.755+0000 7f0890882700 1 --2- 172.21.15.46:0/2636715588 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f086c066f30 0x7f086c0693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f08740030c0 tx=0x7f0874015000 comp rx=0 tx=0).stop 2024-01-26T23:49:31.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.755+0000 7f0890882700 1 -- 172.21.15.46:0/2636715588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f088c06add0 msgr2=0x7f088c168d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:31.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.755+0000 7f0890882700 1 --2- 172.21.15.46:0/2636715588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f088c06add0 0x7f088c168d20 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f088400c930 tx=0x7f08840026e0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.755+0000 7f0890882700 1 -- 172.21.15.46:0/2636715588 shutdown_connections 2024-01-26T23:49:31.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.755+0000 7f0890882700 1 --2- 172.21.15.46:0/2636715588 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f086c066f30 0x7f086c0693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.755+0000 7f0890882700 1 --2- 172.21.15.46:0/2636715588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f088c06add0 0x7f088c168d20 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.755+0000 7f0890882700 1 --2- 172.21.15.46:0/2636715588 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f088c068b00 0x7f088c16a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.755+0000 7f0890882700 1 --2- 172.21.15.46:0/2636715588 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f088c06a660 0x7f088c169e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.755+0000 7f0890882700 1 -- 172.21.15.46:0/2636715588 >> 172.21.15.46:0/2636715588 conn(0x7f088c0ca100 msgr2=0x7f088c1570c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:31.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.755+0000 7f0890882700 1 -- 172.21.15.46:0/2636715588 shutdown_connections 2024-01-26T23:49:31.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.755+0000 7f0890882700 1 -- 172.21.15.46:0/2636715588 wait complete. 2024-01-26T23:49:31.758 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-01-26T23:49:31.770 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:49:31.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.901+0000 7ffbebcc9700 1 Processor -- start 2024-01-26T23:49:31.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.901+0000 7ffbebcc9700 1 -- start start 2024-01-26T23:49:31.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.902+0000 7ffbebcc9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffbe4157550 0x7ffbe406f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.902+0000 7ffbebcc9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ffbe4157920 0x7ffbe406fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.902+0000 7ffbebcc9700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe40cc550 con 0x7ffbe40b2520 2024-01-26T23:49:31.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.902+0000 7ffbebcc9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbe40ca020 con 0x7ffbe4157550 2024-01-26T23:49:31.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.902+0000 7ffbebcc9700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbe40cb6a0 con 0x7ffbe4157920 2024-01-26T23:49:31.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.902+0000 7ffbe9a65700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffbe4157550 0x7ffbe406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.902+0000 7ffbea266700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ffbe40b2520 0x7ffbe4067930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39256/0 (socket says 172.21.15.46:39256) 2024-01-26T23:49:31.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.902+0000 7ffbea266700 1 -- 172.21.15.46:0/1119940618 learned_addr learned my addr 172.21.15.46:0/1119940618 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:31.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.902+0000 7ffbe9264700 1 --2- 172.21.15.46:0/1119940618 >> v2:172.21.15.106:3300/0 conn(0x7ffbe4157920 0x7ffbe406fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.902+0000 7ffbe9a65700 1 -- 172.21.15.46:0/1119940618 >> v2:172.21.15.106:3300/0 conn(0x7ffbe4157920 msgr2=0x7ffbe406fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:31.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.902+0000 7ffbe9a65700 1 --2- 172.21.15.46:0/1119940618 >> v2:172.21.15.106:3300/0 conn(0x7ffbe4157920 0x7ffbe406fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.902+0000 7ffbe9a65700 1 -- 172.21.15.46:0/1119940618 >> v1:172.21.15.46:6789/0 conn(0x7ffbe40b2520 legacy=0x7ffbe4067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:31.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.902+0000 7ffbe9a65700 1 -- 172.21.15.46:0/1119940618 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe40cb940 con 0x7ffbe4157550 2024-01-26T23:49:31.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.902+0000 7ffbe8a63700 1 -- 172.21.15.46:0/1119940618 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147860734 0 0) 0x7ffbe40cc550 con 0x7ffbe40b2520 2024-01-26T23:49:31.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.902+0000 7ffbe9a65700 1 --2- 172.21.15.46:0/1119940618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbe4157550 0x7ffbe406f550 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7ffbe40cc4c0 tx=0x7ffbe00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a620e6e2df88f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:31.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.903+0000 7ffbe8a63700 1 -- 172.21.15.46:0/1119940618 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe0011630 con 0x7ffbe4157550 2024-01-26T23:49:31.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.903+0000 7ffbe8a63700 1 -- 172.21.15.46:0/1119940618 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe0011790 con 0x7ffbe4157550 2024-01-26T23:49:31.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.903+0000 7ffbe8a63700 1 -- 172.21.15.46:0/1119940618 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe0011b30 con 0x7ffbe4157550 2024-01-26T23:49:31.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.903+0000 7ffbebcc9700 1 -- 172.21.15.46:0/1119940618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbe4157550 msgr2=0x7ffbe406f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:31.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.903+0000 7ffbebcc9700 1 --2- 172.21.15.46:0/1119940618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbe4157550 0x7ffbe406f550 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7ffbe40cc4c0 tx=0x7ffbe00091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.903+0000 7ffbebcc9700 1 -- 172.21.15.46:0/1119940618 shutdown_connections 2024-01-26T23:49:31.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.903+0000 7ffbebcc9700 1 --2- 172.21.15.46:0/1119940618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbe4157550 0x7ffbe406f550 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.903+0000 7ffbebcc9700 1 --2- 172.21.15.46:0/1119940618 >> v2:172.21.15.106:3300/0 conn(0x7ffbe4157920 0x7ffbe406fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.903+0000 7ffbebcc9700 1 -- 172.21.15.46:0/1119940618 >> 172.21.15.46:0/1119940618 conn(0x7ffbe4069510 msgr2=0x7ffbe40aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:31.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.903+0000 7ffbebcc9700 1 -- 172.21.15.46:0/1119940618 shutdown_connections 2024-01-26T23:49:31.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.903+0000 7ffbebcc9700 1 -- 172.21.15.46:0/1119940618 wait complete. 2024-01-26T23:49:31.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.905+0000 7ffbebcc9700 1 Processor -- start 2024-01-26T23:49:31.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.905+0000 7ffbebcc9700 1 -- start start 2024-01-26T23:49:31.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.905+0000 7ffbebcc9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbe40b2520 0x7ffbe413c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.905+0000 7ffbebcc9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbe4157550 0x7ffbe4157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.905+0000 7ffbebcc9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbe4157920 0x7ffbe415b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.905+0000 7ffbebcc9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe413c5c0 con 0x7ffbe4157920 2024-01-26T23:49:31.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.905+0000 7ffbebcc9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe40c8ca0 con 0x7ffbe4157550 2024-01-26T23:49:31.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.905+0000 7ffbebcc9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe40c80d0 con 0x7ffbe40b2520 2024-01-26T23:49:31.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.906+0000 7ffbea266700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbe4157920 0x7ffbe415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.906+0000 7ffbea266700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbe4157920 0x7ffbe415b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59986/0 (socket says 172.21.15.46:59986) 2024-01-26T23:49:31.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.906+0000 7ffbea266700 1 -- 172.21.15.46:0/4277753048 learned_addr learned my addr 172.21.15.46:0/4277753048 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:31.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.906+0000 7ffbe9a65700 1 --2- 172.21.15.46:0/4277753048 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbe40b2520 0x7ffbe413c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.906+0000 7ffbe9264700 1 --2- 172.21.15.46:0/4277753048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbe4157550 0x7ffbe4157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.906+0000 7ffbea266700 1 -- 172.21.15.46:0/4277753048 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbe40b2520 msgr2=0x7ffbe413c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:31.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.906+0000 7ffbea266700 1 --2- 172.21.15.46:0/4277753048 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbe40b2520 0x7ffbe413c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.906+0000 7ffbea266700 1 -- 172.21.15.46:0/4277753048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbe4157550 msgr2=0x7ffbe4157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:31.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.906+0000 7ffbea266700 1 --2- 172.21.15.46:0/4277753048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbe4157550 0x7ffbe4157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:31.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.906+0000 7ffbea266700 1 -- 172.21.15.46:0/4277753048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe0009420 con 0x7ffbe4157920 2024-01-26T23:49:31.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.906+0000 7ffbea266700 1 --2- 172.21.15.46:0/4277753048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbe4157920 0x7ffbe415b430 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00ccc0 tx=0x7ffbdc001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:31.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.906+0000 7ffbd9ffb700 1 -- 172.21.15.46:0/4277753048 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc00a8a0 con 0x7ffbe4157920 2024-01-26T23:49:31.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.906+0000 7ffbd9ffb700 1 -- 172.21.15.46:0/4277753048 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbdc00aa00 con 0x7ffbe4157920 2024-01-26T23:49:31.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.906+0000 7ffbebcc9700 1 -- 172.21.15.46:0/4277753048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe40c8370 con 0x7ffbe4157920 2024-01-26T23:49:31.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.906+0000 7ffbd9ffb700 1 -- 172.21.15.46:0/4277753048 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc00adc0 con 0x7ffbe4157920 2024-01-26T23:49:31.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.907+0000 7ffbebcc9700 1 -- 172.21.15.46:0/4277753048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe40c8930 con 0x7ffbe4157920 2024-01-26T23:49:31.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.907+0000 7ffbebcc9700 1 -- 172.21.15.46:0/4277753048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe40c8e30 con 0x7ffbe4157920 2024-01-26T23:49:31.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.908+0000 7ffbd9ffb700 1 -- 172.21.15.46:0/4277753048 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ffbdc007300 con 0x7ffbe4157920 2024-01-26T23:49:31.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.908+0000 7ffbd9ffb700 1 --2- 172.21.15.46:0/4277753048 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ffbc8066f30 0x7ffbc80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:31.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.909+0000 7ffbd9ffb700 1 -- 172.21.15.46:0/4277753048 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7ffbdc084b80 con 0x7ffbe4157920 2024-01-26T23:49:31.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.909+0000 7ffbe9a65700 1 --2- 172.21.15.46:0/4277753048 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ffbc8066f30 0x7ffbc80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:31.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.909+0000 7ffbe9a65700 1 --2- 172.21.15.46:0/4277753048 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ffbc8066f30 0x7ffbc80693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffbe00093f0 tx=0x7ffbe0017000 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:31.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:31.913+0000 7ffbd9ffb700 1 -- 172.21.15.46:0/4277753048 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffbdc051b00 con 0x7ffbe4157920 2024-01-26T23:49:32.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.163+0000 7ffbebcc9700 1 -- 172.21.15.46:0/4277753048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbe404c710 con 0x7ffbe4157920 2024-01-26T23:49:32.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.164+0000 7ffbd9ffb700 1 -- 172.21.15.46:0/4277753048 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4599 (secure 0 0 0) 0x7ffbdc084e60 con 0x7ffbe4157920 2024-01-26T23:49:32.165 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:32.165 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/406559819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":406559819},{"type":"v1","addr":"172.21.15.106:6836","nonce":406559819}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/3387155671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":3387155671},{"type":"v1","addr":"172.21.15.106:6834","nonce":3387155671}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8298,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/855914203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":855914203},{"type":"v1","addr":"172.21.15.46:6834","nonce":855914203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:49:21.360002+0000","modified":"2024-01-26T23:49:24.956671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/1052887453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1052887453},{"type":"v1","addr":"172.21.15.46:6835","nonce":1052887453}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2024-01-26T23:49:32.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.166+0000 7ffbebcc9700 1 -- 172.21.15.46:0/4277753048 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ffbc8066f30 msgr2=0x7ffbc80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:32.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.166+0000 7ffbebcc9700 1 --2- 172.21.15.46:0/4277753048 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ffbc8066f30 0x7ffbc80693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffbe00093f0 tx=0x7ffbe0017000 comp rx=0 tx=0).stop 2024-01-26T23:49:32.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.166+0000 7ffbebcc9700 1 -- 172.21.15.46:0/4277753048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbe4157920 msgr2=0x7ffbe415b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:32.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.166+0000 7ffbebcc9700 1 --2- 172.21.15.46:0/4277753048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbe4157920 0x7ffbe415b430 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00ccc0 tx=0x7ffbdc001a80 comp rx=0 tx=0).stop 2024-01-26T23:49:32.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.167+0000 7ffbebcc9700 1 -- 172.21.15.46:0/4277753048 shutdown_connections 2024-01-26T23:49:32.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.167+0000 7ffbebcc9700 1 --2- 172.21.15.46:0/4277753048 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ffbc8066f30 0x7ffbc80693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.167+0000 7ffbebcc9700 1 --2- 172.21.15.46:0/4277753048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbe4157920 0x7ffbe415b430 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.167+0000 7ffbebcc9700 1 --2- 172.21.15.46:0/4277753048 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbe40b2520 0x7ffbe413c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.167+0000 7ffbebcc9700 1 --2- 172.21.15.46:0/4277753048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbe4157550 0x7ffbe4157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.167+0000 7ffbebcc9700 1 -- 172.21.15.46:0/4277753048 >> 172.21.15.46:0/4277753048 conn(0x7ffbe4069510 msgr2=0x7ffbe406f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:32.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.167+0000 7ffbebcc9700 1 -- 172.21.15.46:0/4277753048 shutdown_connections 2024-01-26T23:49:32.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.167+0000 7ffbebcc9700 1 -- 172.21.15.46:0/4277753048 wait complete. 2024-01-26T23:49:32.169 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 70 2024-01-26T23:49:32.584 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-26T23:49:21.360002+0000', 'modified': '2024-01-26T23:49:24.956671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8301}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8301': {'gid': 8301, 'name': 'a', 'rank': 0, 'incarnation': 66, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6835/1052887453', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 1052887453}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 1052887453}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-26T23:49:32.584 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-26T23:49:32.584 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-26T23:49:32.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.725+0000 7fd72d020700 1 Processor -- start 2024-01-26T23:49:32.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.726+0000 7fd72d020700 1 -- start start 2024-01-26T23:49:32.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.726+0000 7fd72d020700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd728159b30 0x7fd7280b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.726+0000 7fd72d020700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7280b10f0 0x7fd7280a9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.726+0000 7fd72d020700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd72806f940 con 0x7fd728159760 2024-01-26T23:49:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.726+0000 7fd72d020700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7280b1570 con 0x7fd728159b30 2024-01-26T23:49:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.726+0000 7fd72d020700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7280b3230 con 0x7fd7280b10f0 2024-01-26T23:49:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.727+0000 7fd72659c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd728159b30 0x7fd7280b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.727+0000 7fd72659c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd728159b30 0x7fd7280b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59996/0 (socket says 172.21.15.46:59996) 2024-01-26T23:49:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.727+0000 7fd72659c700 1 -- 172.21.15.46:0/3190644675 learned_addr learned my addr 172.21.15.46:0/3190644675 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.727+0000 7fd72759e700 1 --2- 172.21.15.46:0/3190644675 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7280b10f0 0x7fd7280a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.727+0000 7fd72659c700 1 -- 172.21.15.46:0/3190644675 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7280b10f0 msgr2=0x7fd7280a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.727+0000 7fd72659c700 1 --2- 172.21.15.46:0/3190644675 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7280b10f0 0x7fd7280a9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.727+0000 7fd72659c700 1 -- 172.21.15.46:0/3190644675 >> v1:172.21.15.106:6789/0 conn(0x7fd728159760 legacy=0x7fd7280b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.727+0000 7fd72659c700 1 -- 172.21.15.46:0/3190644675 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd72806ed30 con 0x7fd728159b30 2024-01-26T23:49:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.727+0000 7fd72659c700 1 --2- 172.21.15.46:0/3190644675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd728159b30 0x7fd7280b0bb0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fd7100099e0 tx=0x7fd710009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=110dbe35f64ef0c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.727+0000 7fd725d9b700 1 -- 172.21.15.46:0/3190644675 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71000b820 con 0x7fd728159b30 2024-01-26T23:49:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.727+0000 7fd725d9b700 1 -- 172.21.15.46:0/3190644675 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd71000b980 con 0x7fd728159b30 2024-01-26T23:49:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.727+0000 7fd725d9b700 1 -- 172.21.15.46:0/3190644675 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71000bd20 con 0x7fd728159b30 2024-01-26T23:49:32.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.728+0000 7fd72d020700 1 -- 172.21.15.46:0/3190644675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd728159b30 msgr2=0x7fd7280b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:32.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.728+0000 7fd72d020700 1 --2- 172.21.15.46:0/3190644675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd728159b30 0x7fd7280b0bb0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fd7100099e0 tx=0x7fd710009250 comp rx=0 tx=0).stop 2024-01-26T23:49:32.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.728+0000 7fd72d020700 1 -- 172.21.15.46:0/3190644675 shutdown_connections 2024-01-26T23:49:32.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.728+0000 7fd72d020700 1 --2- 172.21.15.46:0/3190644675 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7280b10f0 0x7fd7280a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.728+0000 7fd72d020700 1 --2- 172.21.15.46:0/3190644675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd728159b30 0x7fd7280b0bb0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.728+0000 7fd72d020700 1 -- 172.21.15.46:0/3190644675 >> 172.21.15.46:0/3190644675 conn(0x7fd728069510 msgr2=0x7fd7280aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:32.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.728+0000 7fd72d020700 1 -- 172.21.15.46:0/3190644675 shutdown_connections 2024-01-26T23:49:32.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.728+0000 7fd72d020700 1 -- 172.21.15.46:0/3190644675 wait complete. 2024-01-26T23:49:32.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.729+0000 7fd72d020700 1 Processor -- start 2024-01-26T23:49:32.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.729+0000 7fd72d020700 1 -- start start 2024-01-26T23:49:32.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.730+0000 7fd72d020700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7280b10f0 0x7fd72814ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.730+0000 7fd72d020700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd728159760 0x7fd72815f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.730+0000 7fd72d020700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd728159b30 0x7fd72815fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.730+0000 7fd72d020700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7281507a0 con 0x7fd7280b10f0 2024-01-26T23:49:32.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.730+0000 7fd72d020700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72816a2b0 con 0x7fd728159b30 2024-01-26T23:49:32.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.730+0000 7fd72d020700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd728169980 con 0x7fd728159760 2024-01-26T23:49:32.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.730+0000 7fd726d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7280b10f0 0x7fd72814ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.730+0000 7fd726d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7280b10f0 0x7fd72814ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60010/0 (socket says 172.21.15.46:60010) 2024-01-26T23:49:32.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.730+0000 7fd726d9d700 1 -- 172.21.15.46:0/2377669969 learned_addr learned my addr 172.21.15.46:0/2377669969 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:32.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.731+0000 7fd72759e700 1 --2- 172.21.15.46:0/2377669969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd728159b30 0x7fd72815fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.731+0000 7fd72659c700 1 --2- 172.21.15.46:0/2377669969 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd728159760 0x7fd72815f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.731+0000 7fd726d9d700 1 -- 172.21.15.46:0/2377669969 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd728159760 msgr2=0x7fd72815f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:32.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.731+0000 7fd726d9d700 1 --2- 172.21.15.46:0/2377669969 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd728159760 0x7fd72815f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.731+0000 7fd726d9d700 1 -- 172.21.15.46:0/2377669969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd728159b30 msgr2=0x7fd72815fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:32.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.731+0000 7fd726d9d700 1 --2- 172.21.15.46:0/2377669969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd728159b30 0x7fd72815fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.731+0000 7fd726d9d700 1 -- 172.21.15.46:0/2377669969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7100094f0 con 0x7fd7280b10f0 2024-01-26T23:49:32.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.731+0000 7fd726d9d700 1 --2- 172.21.15.46:0/2377669969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7280b10f0 0x7fd72814ba10 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fd71800c590 tx=0x7fd71800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:32.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.731+0000 7fd71effd700 1 -- 172.21.15.46:0/2377669969 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd718011590 con 0x7fd7280b10f0 2024-01-26T23:49:32.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.731+0000 7fd71effd700 1 -- 172.21.15.46:0/2377669969 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7180116f0 con 0x7fd7280b10f0 2024-01-26T23:49:32.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.731+0000 7fd72d020700 1 -- 172.21.15.46:0/2377669969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd728169c60 con 0x7fd7280b10f0 2024-01-26T23:49:32.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.731+0000 7fd71effd700 1 -- 172.21.15.46:0/2377669969 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd718011ab0 con 0x7fd7280b10f0 2024-01-26T23:49:32.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.731+0000 7fd72d020700 1 -- 172.21.15.46:0/2377669969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd72816a740 con 0x7fd7280b10f0 2024-01-26T23:49:32.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.733+0000 7fd71effd700 1 -- 172.21.15.46:0/2377669969 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd71801b3b0 con 0x7fd7280b10f0 2024-01-26T23:49:32.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.733+0000 7fd71effd700 1 --2- 172.21.15.46:0/2377669969 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd708066f30 0x7fd7080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.733+0000 7fd71effd700 1 -- 172.21.15.46:0/2377669969 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fd718083db0 con 0x7fd7280b10f0 2024-01-26T23:49:32.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.733+0000 7fd72659c700 1 --2- 172.21.15.46:0/2377669969 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd708066f30 0x7fd7080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.734+0000 7fd72659c700 1 --2- 172.21.15.46:0/2377669969 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd708066f30 0x7fd7080693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd7100099e0 tx=0x7fd710017000 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:32.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.735+0000 7fd736323b80 1 Processor -- start 2024-01-26T23:49:32.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.736+0000 7fd736323b80 1 -- start start 2024-01-26T23:49:32.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.736+0000 7fd736323b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1b7010 0x5562dc1b73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.736+0000 7fd736323b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1ff360 0x5562dc1e1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.736+0000 7fd736323b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1b65d0 0x5562dc1e1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.736+0000 7fd736323b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562dc0ba9f0 con 0x5562dc1ff360 2024-01-26T23:49:32.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.736+0000 7fd736323b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562dc10c130 con 0x5562dc1b7010 2024-01-26T23:49:32.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.736+0000 7fd736323b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562dc109930 con 0x5562dc1b65d0 2024-01-26T23:49:32.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.736+0000 7fd72659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1ff360 0x5562dc1e1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.736+0000 7fd72659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1ff360 0x5562dc1e1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60012/0 (socket says 172.21.15.46:60012) 2024-01-26T23:49:32.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.736+0000 7fd72659c700 1 -- 172.21.15.46:0/642632705 learned_addr learned my addr 172.21.15.46:0/642632705 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:32.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.736+0000 7fd726d9d700 1 --2- 172.21.15.46:0/642632705 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1b7010 0x5562dc1b73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.736+0000 7fd72759e700 1 --2- 172.21.15.46:0/642632705 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1b65d0 0x5562dc1e1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.737+0000 7fd72659c700 1 -- 172.21.15.46:0/642632705 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1b65d0 msgr2=0x5562dc1e1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:32.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.737+0000 7fd72659c700 1 --2- 172.21.15.46:0/642632705 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1b65d0 0x5562dc1e1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.737+0000 7fd72659c700 1 -- 172.21.15.46:0/642632705 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1b7010 msgr2=0x5562dc1b73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:32.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.737+0000 7fd72659c700 1 --2- 172.21.15.46:0/642632705 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1b7010 0x5562dc1b73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.737+0000 7fd72659c700 1 -- 172.21.15.46:0/642632705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562dbf44b10 con 0x5562dc1ff360 2024-01-26T23:49:32.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.737+0000 7fd726d9d700 1 --2- 172.21.15.46:0/642632705 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1b7010 0x5562dc1b73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:32.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.737+0000 7fd72659c700 1 --2- 172.21.15.46:0/642632705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1ff360 0x5562dc1e1840 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fd71000bc30 tx=0x7fd710021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b9eef32d097b14 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:32.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.737+0000 7fd707fff700 1 -- 172.21.15.46:0/642632705 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd710021070 con 0x5562dc1ff360 2024-01-26T23:49:32.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.737+0000 7fd707fff700 1 -- 172.21.15.46:0/642632705 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd710006b10 con 0x5562dc1ff360 2024-01-26T23:49:32.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.737+0000 7fd707fff700 1 -- 172.21.15.46:0/642632705 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd710011ac0 con 0x5562dc1ff360 2024-01-26T23:49:32.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.738+0000 7fd736323b80 1 -- 172.21.15.46:0/642632705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1ff360 msgr2=0x5562dc1e1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:32.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.738+0000 7fd736323b80 1 --2- 172.21.15.46:0/642632705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1ff360 0x5562dc1e1840 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fd71000bc30 tx=0x7fd710021000 comp rx=0 tx=0).stop 2024-01-26T23:49:32.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.738+0000 7fd736323b80 1 -- 172.21.15.46:0/642632705 shutdown_connections 2024-01-26T23:49:32.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.738+0000 7fd736323b80 1 --2- 172.21.15.46:0/642632705 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1b65d0 0x5562dc1e1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.738+0000 7fd736323b80 1 --2- 172.21.15.46:0/642632705 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1b7010 0x5562dc1b73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.738+0000 7fd736323b80 1 --2- 172.21.15.46:0/642632705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1ff360 0x5562dc1e1840 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.738+0000 7fd736323b80 1 -- 172.21.15.46:0/642632705 >> 172.21.15.46:0/642632705 conn(0x5562dc1b6a70 msgr2=0x5562dc1ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:32.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.738+0000 7fd736323b80 1 -- 172.21.15.46:0/642632705 shutdown_connections 2024-01-26T23:49:32.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.738+0000 7fd736323b80 1 -- 172.21.15.46:0/642632705 wait complete. 2024-01-26T23:49:32.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.739+0000 7fd736323b80 1 -- start start 2024-01-26T23:49:32.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.739+0000 7fd736323b80 1 Processor -- start 2024-01-26T23:49:32.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.740+0000 7fd736323b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1b65d0 0x5562dc1de690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.740+0000 7fd736323b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1b7010 0x5562dc1cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.740+0000 7fd736323b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1ff360 0x5562dc1e2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.740+0000 7fd736323b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562dc105720 con 0x5562dc1b65d0 2024-01-26T23:49:32.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.740+0000 7fd736323b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562dc0b11b0 con 0x5562dc1b7010 2024-01-26T23:49:32.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.740+0000 7fd736323b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562dc0b0e50 con 0x5562dc1ff360 2024-01-26T23:49:32.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.740+0000 7fd736323b80 10 client.0 _ll_register_callbacks cb 0x5562dbf137b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:49:32.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.740+0000 7fd726d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1b65d0 0x5562dc1de690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.740+0000 7fd726d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1b65d0 0x5562dc1de690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60020/0 (socket says 172.21.15.46:60020) 2024-01-26T23:49:32.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.740+0000 7fd726d9d700 1 -- 172.21.15.46:0/2870249626 learned_addr learned my addr 172.21.15.46:0/2870249626 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:32.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.740+0000 7fd72659c700 1 --2- 172.21.15.46:0/2870249626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1b7010 0x5562dc1cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.740+0000 7fd72759e700 1 --2- 172.21.15.46:0/2870249626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1ff360 0x5562dc1e2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.740+0000 7fd726d9d700 1 -- 172.21.15.46:0/2870249626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1ff360 msgr2=0x5562dc1e2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:32.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.740+0000 7fd726d9d700 1 --2- 172.21.15.46:0/2870249626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1ff360 0x5562dc1e2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd726d9d700 1 -- 172.21.15.46:0/2870249626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1b7010 msgr2=0x5562dc1cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:32.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd726d9d700 1 --2- 172.21.15.46:0/2870249626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1b7010 0x5562dc1cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd726d9d700 1 -- 172.21.15.46:0/2870249626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd72816a740 con 0x5562dc1b65d0 2024-01-26T23:49:32.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd72759e700 1 --2- 172.21.15.46:0/2870249626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1ff360 0x5562dc1e2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:32.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd726d9d700 1 --2- 172.21.15.46:0/2870249626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1b65d0 0x5562dc1de690 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fd718082090 tx=0x7fd718003040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e1da4925fc9f0fc server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:32.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd7057fa700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:49:32.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd7057fa700 1 -- 172.21.15.46:0/2870249626 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71805b9b0 con 0x5562dc1b65d0 2024-01-26T23:49:32.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd7057fa700 1 -- 172.21.15.46:0/2870249626 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fd718052db0 con 0x5562dc1b65d0 2024-01-26T23:49:32.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd736323b80 1 -- 172.21.15.46:0/2870249626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5562dc1aa190 con 0x5562dc1b65d0 2024-01-26T23:49:32.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd7057fa700 1 -- 172.21.15.46:0/2870249626 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd718054110 con 0x5562dc1b65d0 2024-01-26T23:49:32.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd736323b80 10 client.8466 fetch_fsmap learned FSMap version 70 2024-01-26T23:49:32.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd736323b80 1 -- 172.21.15.46:0/2870249626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x7fd72806ed30 con 0x5562dc1b65d0 2024-01-26T23:49:32.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd7057fa700 1 -- 172.21.15.46:0/2870249626 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd71805f020 con 0x5562dc1b65d0 2024-01-26T23:49:32.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd7057fa700 1 -- 172.21.15.46:0/2870249626 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 70) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fd71805b520 con 0x5562dc1b65d0 2024-01-26T23:49:32.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd736323b80 10 client.8466 fetch_fsmap finished waiting for FSMap version 70 2024-01-26T23:49:32.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd736323b80 10 client.8466 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-26T23:49:32.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd736323b80 20 client.8466 populate_metadata read hostname 'smithi046' 2024-01-26T23:49:32.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd736323b80 1 --2- 172.21.15.46:0/2870249626 >> [v2:172.21.15.46:6833/1052887453,v1:172.21.15.46:6835/1052887453] conn(0x5562dc1e3290 0x5562dc1fb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd736323b80 4 client.8466 mds_command: new command op to 8301 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-26T23:49:32.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.741+0000 7fd736323b80 1 -- 172.21.15.46:0/2870249626 --> [v2:172.21.15.46:6833/1052887453,v1:172.21.15.46:6835/1052887453] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5562dc0a7ee0 con 0x5562dc1e3290 2024-01-26T23:49:32.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.742+0000 7fd72659c700 1 --2- 172.21.15.46:0/2870249626 >> [v2:172.21.15.46:6833/1052887453,v1:172.21.15.46:6835/1052887453] conn(0x5562dc1e3290 0x5562dc1fb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.742+0000 7fd7057fa700 10 client.8466 ms_handle_connect on v2:172.21.15.46:6833/1052887453 2024-01-26T23:49:32.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.742+0000 7fd72659c700 1 --2- 172.21.15.46:0/2870249626 >> [v2:172.21.15.46:6833/1052887453,v1:172.21.15.46:6835/1052887453] conn(0x5562dc1e3290 0x5562dc1fb290 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a6691839e1dff7c server_cookie=3acfb805cb37035c in_seq=0 out_seq=0 2024-01-26T23:49:32.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.743+0000 7fd7057fa700 1 -- 172.21.15.46:0/2870249626 <== mds.0 v2:172.21.15.46:6833/1052887453 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x5562dc0a7ee0 con 0x5562dc1e3290 2024-01-26T23:49:32.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.743+0000 7fd7057fa700 10 client.8466 handle_command_reply: tid=0 2024-01-26T23:49:32.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.743+0000 7fd736323b80 1 client.8466 shutdown 2024-01-26T23:49:32.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.743+0000 7fd736323b80 1 -- 172.21.15.46:0/2870249626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1b65d0 msgr2=0x5562dc1de690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:32.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.743+0000 7fd736323b80 1 --2- 172.21.15.46:0/2870249626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1b65d0 0x5562dc1de690 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fd718082090 tx=0x7fd718003040 comp rx=0 tx=0).stop 2024-01-26T23:49:32.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.743+0000 7fd736323b80 1 -- 172.21.15.46:0/2870249626 shutdown_connections 2024-01-26T23:49:32.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.743+0000 7fd736323b80 1 --2- 172.21.15.46:0/2870249626 >> [v2:172.21.15.46:6833/1052887453,v1:172.21.15.46:6835/1052887453] conn(0x5562dc1e3290 0x5562dc1fb290 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.743+0000 7fd736323b80 1 --2- 172.21.15.46:0/2870249626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1b65d0 0x5562dc1de690 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.743+0000 7fd736323b80 1 --2- 172.21.15.46:0/2870249626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1ff360 0x5562dc1e2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.743+0000 7fd736323b80 1 --2- 172.21.15.46:0/2870249626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1b7010 0x5562dc1cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.743+0000 7fd7057fa700 0 client.8466 ms_handle_reset on v2:172.21.15.46:6833/1052887453 2024-01-26T23:49:32.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.743+0000 7fd736323b80 1 -- 172.21.15.46:0/2870249626 >> 172.21.15.46:0/2870249626 conn(0x5562dc1b6a70 msgr2=0x5562dc1b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:32.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.743+0000 7fd736323b80 1 -- 172.21.15.46:0/2870249626 shutdown_connections 2024-01-26T23:49:32.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.743+0000 7fd736323b80 1 -- 172.21.15.46:0/2870249626 wait complete. 2024-01-26T23:49:32.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.743+0000 7fd736323b80 20 client.8466 trim_cache size 0 max 16384 2024-01-26T23:49:32.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.761+0000 7fd736323b80 1 Processor -- start 2024-01-26T23:49:32.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.761+0000 7fd736323b80 1 -- start start 2024-01-26T23:49:32.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.762+0000 7fd736323b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1b65d0 0x5562dc1e0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.762+0000 7fd736323b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1b7010 0x5562dc1ff770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.762+0000 7fd736323b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1ff360 0x5562dc1ffcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.762+0000 7fd736323b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562dbffde40 con 0x5562dc1b7010 2024-01-26T23:49:32.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.762+0000 7fd736323b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562dc1fe140 con 0x5562dc1ff360 2024-01-26T23:49:32.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.762+0000 7fd736323b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562dc1b2de0 con 0x5562dc1b65d0 2024-01-26T23:49:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.762+0000 7fd72659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1b7010 0x5562dc1ff770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.762+0000 7fd72659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1b7010 0x5562dc1ff770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60036/0 (socket says 172.21.15.46:60036) 2024-01-26T23:49:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.762+0000 7fd72659c700 1 -- 172.21.15.46:0/2361853611 learned_addr learned my addr 172.21.15.46:0/2361853611 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.762+0000 7fd72759e700 1 --2- 172.21.15.46:0/2361853611 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1ff360 0x5562dc1ffcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.762+0000 7fd726d9d700 1 --2- 172.21.15.46:0/2361853611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1b65d0 0x5562dc1e0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.762+0000 7fd72659c700 1 -- 172.21.15.46:0/2361853611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1b65d0 msgr2=0x5562dc1e0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.762+0000 7fd72659c700 1 --2- 172.21.15.46:0/2361853611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1b65d0 0x5562dc1e0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.762+0000 7fd72659c700 1 -- 172.21.15.46:0/2361853611 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1ff360 msgr2=0x5562dc1ffcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.762+0000 7fd72659c700 1 --2- 172.21.15.46:0/2361853611 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1ff360 0x5562dc1ffcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.762+0000 7fd72659c700 1 -- 172.21.15.46:0/2361853611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562dc1b7e60 con 0x5562dc1b7010 2024-01-26T23:49:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.762+0000 7fd726d9d700 1 --2- 172.21.15.46:0/2361853611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1b65d0 0x5562dc1e0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:49:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.763+0000 7fd72659c700 1 --2- 172.21.15.46:0/2361853611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1b7010 0x5562dc1ff770 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fd710006d00 tx=0x7fd710014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cfd13d3624f6236 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.763+0000 7fd71cff9700 1 -- 172.21.15.46:0/2361853611 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd710038040 con 0x5562dc1b7010 2024-01-26T23:49:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.763+0000 7fd71cff9700 1 -- 172.21.15.46:0/2361853611 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7100119c0 con 0x5562dc1b7010 2024-01-26T23:49:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.763+0000 7fd71cff9700 1 -- 172.21.15.46:0/2361853611 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd710021040 con 0x5562dc1b7010 2024-01-26T23:49:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.763+0000 7fd736323b80 1 -- 172.21.15.46:0/2361853611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1b7010 msgr2=0x5562dc1ff770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.763+0000 7fd736323b80 1 --2- 172.21.15.46:0/2361853611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1b7010 0x5562dc1ff770 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fd710006d00 tx=0x7fd710014000 comp rx=0 tx=0).stop 2024-01-26T23:49:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.763+0000 7fd736323b80 1 -- 172.21.15.46:0/2361853611 shutdown_connections 2024-01-26T23:49:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.763+0000 7fd736323b80 1 --2- 172.21.15.46:0/2361853611 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1ff360 0x5562dc1ffcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.763+0000 7fd736323b80 1 --2- 172.21.15.46:0/2361853611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1b65d0 0x5562dc1e0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.763+0000 7fd736323b80 1 --2- 172.21.15.46:0/2361853611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1b7010 0x5562dc1ff770 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.763+0000 7fd736323b80 1 -- 172.21.15.46:0/2361853611 >> 172.21.15.46:0/2361853611 conn(0x5562dc1b6a70 msgr2=0x5562dc1de4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.764+0000 7fd736323b80 1 -- 172.21.15.46:0/2361853611 shutdown_connections 2024-01-26T23:49:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.764+0000 7fd736323b80 1 -- 172.21.15.46:0/2361853611 wait complete. 2024-01-26T23:49:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.764+0000 7fd736323b80 1 -- start start 2024-01-26T23:49:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.764+0000 7fd736323b80 1 Processor -- start 2024-01-26T23:49:32.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.766+0000 7fd736323b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1b65d0 0x5562dc20abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.766+0000 7fd736323b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1b7010 0x5562dc1ff730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.766+0000 7fd736323b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1ff360 0x5562dc1ffc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.766+0000 7fd736323b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562dc1f62d0 con 0x5562dc1ff360 2024-01-26T23:49:32.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.766+0000 7fd736323b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562dc1cec40 con 0x5562dc1b7010 2024-01-26T23:49:32.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.766+0000 7fd736323b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562dc260b40 con 0x5562dc1b65d0 2024-01-26T23:49:32.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.766+0000 7fd736323b80 10 client.0 _ll_register_callbacks cb 0x5562dbf137b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:49:32.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.766+0000 7fd72759e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1ff360 0x5562dc1ffc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.766+0000 7fd72759e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1ff360 0x5562dc1ffc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60042/0 (socket says 172.21.15.46:60042) 2024-01-26T23:49:32.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.766+0000 7fd72759e700 1 -- 172.21.15.46:0/822653405 learned_addr learned my addr 172.21.15.46:0/822653405 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:32.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.766+0000 7fd726d9d700 1 --2- 172.21.15.46:0/822653405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1b65d0 0x5562dc20abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.766+0000 7fd72659c700 1 --2- 172.21.15.46:0/822653405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1b7010 0x5562dc1ff730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.766+0000 7fd72759e700 1 -- 172.21.15.46:0/822653405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1b65d0 msgr2=0x5562dc20abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:32.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.766+0000 7fd72759e700 1 --2- 172.21.15.46:0/822653405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1b65d0 0x5562dc20abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.766+0000 7fd72759e700 1 -- 172.21.15.46:0/822653405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1b7010 msgr2=0x5562dc1ff730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:32.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.766+0000 7fd72759e700 1 --2- 172.21.15.46:0/822653405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1b7010 0x5562dc1ff730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.766+0000 7fd72759e700 1 -- 172.21.15.46:0/822653405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5562dbf44b10 con 0x5562dc1ff360 2024-01-26T23:49:32.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.767+0000 7fd72759e700 1 --2- 172.21.15.46:0/822653405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1ff360 0x5562dc1ffc70 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fd720016d20 tx=0x7fd72000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=146e4348544de311 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:32.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.767+0000 7fd7057fa700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:49:32.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.767+0000 7fd7057fa700 1 -- 172.21.15.46:0/822653405 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd720009d70 con 0x5562dc1ff360 2024-01-26T23:49:32.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.767+0000 7fd736323b80 1 -- 172.21.15.46:0/822653405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd728155630 con 0x5562dc1ff360 2024-01-26T23:49:32.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.767+0000 7fd7057fa700 1 -- 172.21.15.46:0/822653405 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fd720014ca0 con 0x5562dc1ff360 2024-01-26T23:49:32.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.767+0000 7fd7057fa700 1 -- 172.21.15.46:0/822653405 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd720014e60 con 0x5562dc1ff360 2024-01-26T23:49:32.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.767+0000 7fd736323b80 10 client.8472 fetch_fsmap learned FSMap version 70 2024-01-26T23:49:32.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.767+0000 7fd736323b80 1 -- 172.21.15.46:0/822653405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x5562dc0d3290 con 0x5562dc1ff360 2024-01-26T23:49:32.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.767+0000 7fd7057fa700 1 -- 172.21.15.46:0/822653405 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd728155630 con 0x5562dc1ff360 2024-01-26T23:49:32.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.767+0000 7fd7057fa700 1 -- 172.21.15.46:0/822653405 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 70) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fd720018630 con 0x5562dc1ff360 2024-01-26T23:49:32.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.767+0000 7fd736323b80 10 client.8472 fetch_fsmap finished waiting for FSMap version 70 2024-01-26T23:49:32.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.767+0000 7fd736323b80 10 client.8472 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-26T23:49:32.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.767+0000 7fd736323b80 20 client.8472 populate_metadata read hostname 'smithi046' 2024-01-26T23:49:32.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.767+0000 7fd736323b80 1 --2- 172.21.15.46:0/822653405 >> [v2:172.21.15.46:6833/1052887453,v1:172.21.15.46:6835/1052887453] conn(0x5562dc1b0c50 0x5562dc1b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.767+0000 7fd736323b80 4 client.8472 mds_command: new command op to 8301 tid=0[{"prefix": "status"}] 2024-01-26T23:49:32.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.767+0000 7fd736323b80 1 -- 172.21.15.46:0/822653405 --> [v2:172.21.15.46:6833/1052887453,v1:172.21.15.46:6835/1052887453] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fd710006b10 con 0x5562dc1b0c50 2024-01-26T23:49:32.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.767+0000 7fd726d9d700 1 --2- 172.21.15.46:0/822653405 >> [v2:172.21.15.46:6833/1052887453,v1:172.21.15.46:6835/1052887453] conn(0x5562dc1b0c50 0x5562dc1b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.768+0000 7fd7057fa700 10 client.8472 ms_handle_connect on v2:172.21.15.46:6833/1052887453 2024-01-26T23:49:32.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.768+0000 7fd726d9d700 1 --2- 172.21.15.46:0/822653405 >> [v2:172.21.15.46:6833/1052887453,v1:172.21.15.46:6835/1052887453] conn(0x5562dc1b0c50 0x5562dc1b4f80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47ed56ca3e05c4c4 server_cookie=ceb4a620b13e0b69 in_seq=0 out_seq=0 2024-01-26T23:49:32.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.768+0000 7fd7057fa700 1 -- 172.21.15.46:0/822653405 <== mds.0 v2:172.21.15.46:6833/1052887453 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x7fd710006b10 con 0x5562dc1b0c50 2024-01-26T23:49:32.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.768+0000 7fd7057fa700 10 client.8472 handle_command_reply: tid=0 2024-01-26T23:49:32.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.768+0000 7fd736323b80 1 client.8472 shutdown 2024-01-26T23:49:32.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.768+0000 7fd736323b80 1 -- 172.21.15.46:0/822653405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1ff360 msgr2=0x5562dc1ffc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:32.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.768+0000 7fd736323b80 1 --2- 172.21.15.46:0/822653405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1ff360 0x5562dc1ffc70 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fd720016d20 tx=0x7fd72000b040 comp rx=0 tx=0).stop 2024-01-26T23:49:32.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.768+0000 7fd736323b80 1 -- 172.21.15.46:0/822653405 shutdown_connections 2024-01-26T23:49:32.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.768+0000 7fd736323b80 1 --2- 172.21.15.46:0/822653405 >> [v2:172.21.15.46:6833/1052887453,v1:172.21.15.46:6835/1052887453] conn(0x5562dc1b0c50 0x5562dc1b4f80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.768+0000 7fd736323b80 1 --2- 172.21.15.46:0/822653405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5562dc1ff360 0x5562dc1ffc70 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.768+0000 7fd7057fa700 0 client.8472 ms_handle_reset on v2:172.21.15.46:6833/1052887453 2024-01-26T23:49:32.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.768+0000 7fd736323b80 1 --2- 172.21.15.46:0/822653405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5562dc1b65d0 0x5562dc20abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.768+0000 7fd736323b80 1 --2- 172.21.15.46:0/822653405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5562dc1b7010 0x5562dc1ff730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.768+0000 7fd736323b80 1 -- 172.21.15.46:0/822653405 >> 172.21.15.46:0/822653405 conn(0x5562dc1b6a70 msgr2=0x5562dc1fb100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:32.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.768+0000 7fd736323b80 1 -- 172.21.15.46:0/822653405 shutdown_connections 2024-01-26T23:49:32.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.768+0000 7fd736323b80 1 -- 172.21.15.46:0/822653405 wait complete. 2024-01-26T23:49:32.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.768+0000 7fd736323b80 20 client.8472 trim_cache size 0 max 16384 2024-01-26T23:49:32.775 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:49:32.775 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-26T23:49:32.775 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-26T23:49:32.775 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 8301, 2024-01-26T23:49:32.775 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-26T23:49:32.775 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-26T23:49:32.775 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-01-26T23:49:32.776 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 10.831981546, 2024-01-26T23:49:32.776 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 70, 2024-01-26T23:49:32.776 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 145, 2024-01-26T23:49:32.776 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 145, 2024-01-26T23:49:32.776 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 16.827634644 2024-01-26T23:49:32.776 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:49:32.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.769+0000 7fd72d020700 1 -- 172.21.15.46:0/2377669969 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd708066f30 msgr2=0x7fd7080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:32.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.769+0000 7fd72d020700 1 --2- 172.21.15.46:0/2377669969 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd708066f30 0x7fd7080693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd7100099e0 tx=0x7fd710017000 comp rx=0 tx=0).stop 2024-01-26T23:49:32.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.769+0000 7fd72d020700 1 -- 172.21.15.46:0/2377669969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7280b10f0 msgr2=0x7fd72814ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:32.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.769+0000 7fd72d020700 1 --2- 172.21.15.46:0/2377669969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7280b10f0 0x7fd72814ba10 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fd71800c590 tx=0x7fd71800c950 comp rx=0 tx=0).stop 2024-01-26T23:49:32.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.769+0000 7fd72d020700 1 -- 172.21.15.46:0/2377669969 shutdown_connections 2024-01-26T23:49:32.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.769+0000 7fd72d020700 1 --2- 172.21.15.46:0/2377669969 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd708066f30 0x7fd7080693e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.769+0000 7fd72d020700 1 --2- 172.21.15.46:0/2377669969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7280b10f0 0x7fd72814ba10 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.769+0000 7fd72d020700 1 --2- 172.21.15.46:0/2377669969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd728159b30 0x7fd72815fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.769+0000 7fd72d020700 1 --2- 172.21.15.46:0/2377669969 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd728159760 0x7fd72815f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.769+0000 7fd72d020700 1 -- 172.21.15.46:0/2377669969 >> 172.21.15.46:0/2377669969 conn(0x7fd728069510 msgr2=0x7fd728157520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:32.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.769+0000 7fd72d020700 1 -- 172.21.15.46:0/2377669969 shutdown_connections 2024-01-26T23:49:32.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.769+0000 7fd72d020700 1 -- 172.21.15.46:0/2377669969 wait complete. 2024-01-26T23:49:32.784 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-26T23:49:32.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.914+0000 7f781a041700 1 Processor -- start 2024-01-26T23:49:32.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.914+0000 7f781a041700 1 -- start start 2024-01-26T23:49:32.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.914+0000 7f781a041700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7814159760 0x7f78140b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.915+0000 7f781a041700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f78140b12c0 0x7f78140a9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.915+0000 7f781a041700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f781406f940 con 0x7f78140b3060 2024-01-26T23:49:32.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.915+0000 7f781a041700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f781406ed30 con 0x7f78140b12c0 2024-01-26T23:49:32.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.915+0000 7f781a041700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f781406f010 con 0x7f7814159760 2024-01-26T23:49:32.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.915+0000 7f7813fff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f78140b12c0 0x7f78140a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.915+0000 7f78137fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7814159760 0x7f78140b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.915+0000 7f7813fff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f78140b12c0 0x7f78140a9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:40806/0 (socket says 172.21.15.46:40806) 2024-01-26T23:49:32.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.915+0000 7f7813fff700 1 -- 172.21.15.46:0/2374090598 learned_addr learned my addr 172.21.15.46:0/2374090598 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:32.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.915+0000 7f7812ffd700 1 --1- 172.21.15.46:0/2374090598 >> v1:172.21.15.106:6789/0 conn(0x7f78140b3060 0x7f78140b0bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:44100/0 (socket says 172.21.15.46:44100) 2024-01-26T23:49:32.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.915+0000 7f7813fff700 1 -- 172.21.15.46:0/2374090598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7814159760 msgr2=0x7f78140b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:32.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.915+0000 7f7813fff700 1 --2- 172.21.15.46:0/2374090598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7814159760 0x7f78140b2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.915+0000 7f7813fff700 1 -- 172.21.15.46:0/2374090598 >> v1:172.21.15.106:6789/0 conn(0x7f78140b3060 legacy=0x7f78140b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.915+0000 7f7813fff700 1 -- 172.21.15.46:0/2374090598 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781406f2f0 con 0x7f78140b12c0 2024-01-26T23:49:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.915+0000 7f78137fe700 1 --2- 172.21.15.46:0/2374090598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7814159760 0x7f78140b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.915+0000 7f78127fc700 1 -- 172.21.15.46:0/2374090598 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691625253 0 0) 0x7f781406f940 con 0x7f78140b3060 2024-01-26T23:49:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.916+0000 7f7813fff700 1 --2- 172.21.15.46:0/2374090598 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78140b12c0 0x7f78140a9840 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f780c009f20 tx=0x7f780c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab0822c89a296415 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.916+0000 7f78127fc700 1 -- 172.21.15.46:0/2374090598 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780c00e070 con 0x7f78140b12c0 2024-01-26T23:49:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.916+0000 7f78127fc700 1 -- 172.21.15.46:0/2374090598 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780c00b9d0 con 0x7f78140b12c0 2024-01-26T23:49:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.916+0000 7f78127fc700 1 -- 172.21.15.46:0/2374090598 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780c0092e0 con 0x7f78140b12c0 2024-01-26T23:49:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.916+0000 7f781a041700 1 -- 172.21.15.46:0/2374090598 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78140b12c0 msgr2=0x7f78140a9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.916+0000 7f781a041700 1 --2- 172.21.15.46:0/2374090598 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78140b12c0 0x7f78140a9840 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f780c009f20 tx=0x7f780c009840 comp rx=0 tx=0).stop 2024-01-26T23:49:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.916+0000 7f781a041700 1 -- 172.21.15.46:0/2374090598 shutdown_connections 2024-01-26T23:49:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.916+0000 7f781a041700 1 --2- 172.21.15.46:0/2374090598 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78140b12c0 0x7f78140a9840 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.916+0000 7f781a041700 1 --2- 172.21.15.46:0/2374090598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7814159760 0x7f78140b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.916+0000 7f781a041700 1 -- 172.21.15.46:0/2374090598 >> 172.21.15.46:0/2374090598 conn(0x7f7814069510 msgr2=0x7f78140aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.916+0000 7f781a041700 1 -- 172.21.15.46:0/2374090598 shutdown_connections 2024-01-26T23:49:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.916+0000 7f781a041700 1 -- 172.21.15.46:0/2374090598 wait complete. 2024-01-26T23:49:32.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.918+0000 7f781a041700 1 Processor -- start 2024-01-26T23:49:32.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.918+0000 7f781a041700 1 -- start start 2024-01-26T23:49:32.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f781a041700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78140b12c0 0x7f781414df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f781a041700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78140b3060 0x7f7814152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f781a041700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7814159760 0x7f7814154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f781a041700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7814159f00 con 0x7f78140b3060 2024-01-26T23:49:32.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f781a041700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7814151390 con 0x7f78140b12c0 2024-01-26T23:49:32.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f781a041700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7814150780 con 0x7f7814159760 2024-01-26T23:49:32.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f7812ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78140b3060 0x7f7814152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f7812ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78140b3060 0x7f7814152510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60056/0 (socket says 172.21.15.46:60056) 2024-01-26T23:49:32.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f7812ffd700 1 -- 172.21.15.46:0/2289349379 learned_addr learned my addr 172.21.15.46:0/2289349379 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:32.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f7813fff700 1 --2- 172.21.15.46:0/2289349379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7814159760 0x7f7814154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f78137fe700 1 --2- 172.21.15.46:0/2289349379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78140b12c0 0x7f781414df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f7812ffd700 1 -- 172.21.15.46:0/2289349379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7814159760 msgr2=0x7f7814154a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:32.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f7812ffd700 1 --2- 172.21.15.46:0/2289349379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7814159760 0x7f7814154a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f7812ffd700 1 -- 172.21.15.46:0/2289349379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78140b12c0 msgr2=0x7f781414df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:32.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f7812ffd700 1 --2- 172.21.15.46:0/2289349379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78140b12c0 0x7f781414df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:32.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f7812ffd700 1 -- 172.21.15.46:0/2289349379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7804009420 con 0x7f78140b3060 2024-01-26T23:49:32.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f7812ffd700 1 --2- 172.21.15.46:0/2289349379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78140b3060 0x7f7814152510 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f78080011e0 tx=0x7f7808007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:32.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.919+0000 7f77fb7fe700 1 -- 172.21.15.46:0/2289349379 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7808007300 con 0x7f78140b3060 2024-01-26T23:49:32.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.920+0000 7f77fb7fe700 1 -- 172.21.15.46:0/2289349379 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7808007460 con 0x7f78140b3060 2024-01-26T23:49:32.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.920+0000 7f77fb7fe700 1 -- 172.21.15.46:0/2289349379 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7808004830 con 0x7f78140b3060 2024-01-26T23:49:32.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.920+0000 7f781a041700 1 -- 172.21.15.46:0/2289349379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f780c009ae0 con 0x7f78140b3060 2024-01-26T23:49:32.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.920+0000 7f781a041700 1 -- 172.21.15.46:0/2289349379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7814150d40 con 0x7f78140b3060 2024-01-26T23:49:32.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.921+0000 7f781a041700 1 -- 172.21.15.46:0/2289349379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f781404c710 con 0x7f78140b3060 2024-01-26T23:49:32.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.921+0000 7f77fb7fe700 1 -- 172.21.15.46:0/2289349379 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7808004ba0 con 0x7f78140b3060 2024-01-26T23:49:32.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.921+0000 7f77fb7fe700 1 --2- 172.21.15.46:0/2289349379 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7800066f30 0x7f78000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:32.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.922+0000 7f77fb7fe700 1 -- 172.21.15.46:0/2289349379 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f7808083d40 con 0x7f78140b3060 2024-01-26T23:49:32.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.922+0000 7f78137fe700 1 --2- 172.21.15.46:0/2289349379 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7800066f30 0x7f78000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:32.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.922+0000 7f78137fe700 1 --2- 172.21.15.46:0/2289349379 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7800066f30 0x7f78000693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f78040093f0 tx=0x7f7804008040 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:32.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:32.926+0000 7f77fb7fe700 1 -- 172.21.15.46:0/2289349379 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7808050d60 con 0x7f78140b3060 2024-01-26T23:49:33.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:33.181+0000 7f781a041700 1 -- 172.21.15.46:0/2289349379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7814068120 con 0x7f78140b3060 2024-01-26T23:49:33.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:33.181+0000 7f77fb7fe700 1 -- 172.21.15.46:0/2289349379 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v70) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7808018070 con 0x7f78140b3060 2024-01-26T23:49:33.182 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-26T23:49:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:33.183+0000 7f781a041700 1 -- 172.21.15.46:0/2289349379 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7800066f30 msgr2=0x7f78000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:33.183+0000 7f781a041700 1 --2- 172.21.15.46:0/2289349379 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7800066f30 0x7f78000693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f78040093f0 tx=0x7f7804008040 comp rx=0 tx=0).stop 2024-01-26T23:49:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:33.183+0000 7f781a041700 1 -- 172.21.15.46:0/2289349379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78140b3060 msgr2=0x7f7814152510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:33.183+0000 7f781a041700 1 --2- 172.21.15.46:0/2289349379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78140b3060 0x7f7814152510 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f78080011e0 tx=0x7f7808007e10 comp rx=0 tx=0).stop 2024-01-26T23:49:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:33.183+0000 7f781a041700 1 -- 172.21.15.46:0/2289349379 shutdown_connections 2024-01-26T23:49:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:33.183+0000 7f781a041700 1 --2- 172.21.15.46:0/2289349379 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7800066f30 0x7f78000693e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:33.183+0000 7f781a041700 1 --2- 172.21.15.46:0/2289349379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7814159760 0x7f7814154a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:33.183+0000 7f781a041700 1 --2- 172.21.15.46:0/2289349379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78140b12c0 0x7f781414df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:33.183+0000 7f781a041700 1 --2- 172.21.15.46:0/2289349379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78140b3060 0x7f7814152510 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:33.183+0000 7f781a041700 1 -- 172.21.15.46:0/2289349379 >> 172.21.15.46:0/2289349379 conn(0x7f7814069510 msgr2=0x7f78140b04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:33.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:33.183+0000 7f781a041700 1 -- 172.21.15.46:0/2289349379 shutdown_connections 2024-01-26T23:49:33.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:33.183+0000 7f781a041700 1 -- 172.21.15.46:0/2289349379 wait complete. 2024-01-26T23:49:33.198 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-26T23:49:33.198 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-26T23:49:33.198 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-26T23:49:33.198 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-26T23:49:33.198 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-26T23:49:33.198 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-26T23:49:33.198 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-26T23:49:33.198 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-26T23:49:33.198 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:49:33.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:49:33.199 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-26T23:49:33.218 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-26T23:49:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-26T23:49:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-26T23:49:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:49:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-26T23:49:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:49:33.218 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:49:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:49:33.218 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-26T23:49:33.218 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:49:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:49:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-26T23:49:33.219 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-26T23:49:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:49:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-26T23:49:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-26T23:49:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 41157sec preferred_lft 41157sec 2024-01-26T23:49:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-26T23:49:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:49:33.219 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:49:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:49:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-26T23:49:33.219 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-26T23:49:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:49:33.219 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-26T23:49:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:49:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::cce6:a7ff:fe4f:38d/64 scope link 2024-01-26T23:49:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:49:33.220 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:49:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 76:ef:90:4c:9d:6f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-26T23:49:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::74ef:90ff:fe4c:9d6f/64 scope link 2024-01-26T23:49:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:49:33.220 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:49:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-26T23:49:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::2c73:afff:fef0:6f39/64 scope link 2024-01-26T23:49:33.220 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:49:33.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:49:33.221 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-26T23:49:33.288 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-26T23:49:33.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:49:33.290 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-26T23:49:33.353 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-26T23:49:33.353 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-26T23:49:33.353 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-26T23:49:33.353 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-26T23:49:33.354 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-26T23:49:33.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:49:33.354 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:49:33.354 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:49:33.354 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-26T23:49:33.354 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-26T23:49:33.354 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-26T23:49:33.354 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:49:33.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:49:33.479 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-26T23:49:33.499 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-26T23:49:33.499 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-26T23:49:33.500 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:49:33.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:49:33.500 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-01-26T23:49:33.574 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-26T23:49:33.575 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:49:33.575 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-26T23:49:33.637 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:49:33.637 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:49:33.637 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:49:33.698 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:49:33.774 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:49:33.774 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:49:33.848 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:49:33.849 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:49:33.850 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:33.850 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:49:33.912 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-26T23:49:33.912 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-26T23:49:33.955 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:49:33.991 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:49:33.991 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:49:34.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-01-26T23:49:34.056+0000 7fdd0bf7c540 -1 init, newargv = 0x55ef593db820 newargc=15 2024-01-26T23:49:34.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[86941]: starting ceph client 2024-01-26T23:49:34.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[86941]: starting fuse 2024-01-26T23:49:34.064 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:49:34.065 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:49:34.066 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:34.066 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:49:34.126 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-26T23:49:34.126 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-26T23:49:34.126 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-26T23:49:34.126 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-26T23:49:34.126 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-26T23:49:34.126 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-26T23:49:34.126 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-26T23:49:34.127 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:49:34.127 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-26T23:49:34.127 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-26T23:49:34.127 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-26T23:49:34.127 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-26T23:49:34.127 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:49:34.127 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-26T23:49:34.127 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-26T23:49:34.127 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-26T23:49:34.127 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-26T23:49:34.127 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:49:34.127 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-26T23:49:34.127 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-26T23:49:34.127 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-26T23:49:34.127 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-26T23:49:34.128 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-26T23:49:34.128 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-26T23:49:34.128 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-26T23:49:34.128 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-26T23:49:34.128 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:49:34.128 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-01-26T23:49:34.128 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-26T23:49:34.265 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.86941.asok 2024-01-26T23:49:34.446 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.86941.asok 2024-01-26T23:49:34.447 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:34.447 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.86941.asok status 2024-01-26T23:49:34.621 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:49:34.621 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-26T23:49:34.621 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-26T23:49:34.621 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-26T23:49:34.621 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-01-26T23:49:34.621 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-26T23:49:34.622 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-26T23:49:34.622 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "86941", 2024-01-26T23:49:34.622 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-26T23:49:34.622 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:34.622 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-26T23:49:34.622 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-26T23:49:34.622 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 8481, 2024-01-26T23:49:34.622 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-26T23:49:34.622 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-26T23:49:34.622 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-26T23:49:34.622 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 8481 2024-01-26T23:49:34.622 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:34.622 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:49:34.622 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:49:34.622 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:49:34.623 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 4191799599 2024-01-26T23:49:34.623 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:34.623 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:34.623 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:49:34.623 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:49:34.623 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:49:34.623 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 4191799599 2024-01-26T23:49:34.623 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:49:34.623 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.8481 192.168.0.1:0/4191799599", 2024-01-26T23:49:34.623 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/4191799599", 2024-01-26T23:49:34.623 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-26T23:49:34.623 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 70, 2024-01-26T23:49:34.623 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 145, 2024-01-26T23:49:34.623 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-26T23:49:34.624 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-26T23:49:34.624 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-26T23:49:34.624 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:49:34.632 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:34.632 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-26T23:49:34.632 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-26T23:49:34.652 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-26T23:49:34.652 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-26T23:49:34.652 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:34.652 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:49:34.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-26T23:49:34.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.904+0000 7fe84c381700 1 Processor -- start 2024-01-26T23:49:34.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.904+0000 7fe84c381700 1 -- start start 2024-01-26T23:49:34.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.905+0000 7fe84c381700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8440cccd0 0x7fe8440c3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:34.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.905+0000 7fe84c381700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe8440c2240 0x7fe8440c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:34.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.905+0000 7fe84c381700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe844068220 con 0x7fe84415cd60 2024-01-26T23:49:34.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.905+0000 7fe84c381700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe844067930 con 0x7fe8440cccd0 2024-01-26T23:49:34.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.905+0000 7fe84c381700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8440c4210 con 0x7fe8440c2240 2024-01-26T23:49:34.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.905+0000 7fe84991c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8440cccd0 0x7fe8440c3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:34.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.905+0000 7fe84991c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8440cccd0 0x7fe8440c3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60068/0 (socket says 172.21.15.46:60068) 2024-01-26T23:49:34.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.905+0000 7fe84991c700 1 -- 172.21.15.46:0/3124516106 learned_addr learned my addr 172.21.15.46:0/3124516106 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:34.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.905+0000 7fe84a11d700 1 --1- 172.21.15.46:0/3124516106 >> v1:172.21.15.46:6789/0 conn(0x7fe84415cd60 0x7fe8440cc5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39264/0 (socket says 172.21.15.46:39264) 2024-01-26T23:49:34.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.905+0000 7fe84991c700 1 -- 172.21.15.46:0/3124516106 >> v2:172.21.15.106:3300/0 conn(0x7fe8440c2240 msgr2=0x7fe8440c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:34.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.905+0000 7fe84a91e700 1 --2- 172.21.15.46:0/3124516106 >> v2:172.21.15.106:3300/0 conn(0x7fe8440c2240 0x7fe8440c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:34.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.905+0000 7fe84991c700 1 --2- 172.21.15.46:0/3124516106 >> v2:172.21.15.106:3300/0 conn(0x7fe8440c2240 0x7fe8440c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:34.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.905+0000 7fe84911b700 1 -- 172.21.15.46:0/3124516106 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211841282 0 0) 0x7fe844068220 con 0x7fe84415cd60 2024-01-26T23:49:34.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.905+0000 7fe84991c700 1 -- 172.21.15.46:0/3124516106 >> v1:172.21.15.46:6789/0 conn(0x7fe84415cd60 legacy=0x7fe8440cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:34.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.905+0000 7fe84991c700 1 -- 172.21.15.46:0/3124516106 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe844067bd0 con 0x7fe8440cccd0 2024-01-26T23:49:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.905+0000 7fe84a91e700 1 --2- 172.21.15.46:0/3124516106 >> v2:172.21.15.106:3300/0 conn(0x7fe8440c2240 0x7fe8440c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:49:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.906+0000 7fe84991c700 1 --2- 172.21.15.46:0/3124516106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8440cccd0 0x7fe8440c3b30 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fe84406fca0 tx=0x7fe834009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6892b3a5cc5751d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.906+0000 7fe84911b700 1 -- 172.21.15.46:0/3124516106 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83400b860 con 0x7fe8440cccd0 2024-01-26T23:49:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.906+0000 7fe84911b700 1 -- 172.21.15.46:0/3124516106 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83400b9c0 con 0x7fe8440cccd0 2024-01-26T23:49:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.906+0000 7fe84911b700 1 -- 172.21.15.46:0/3124516106 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83400bd60 con 0x7fe8440cccd0 2024-01-26T23:49:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.906+0000 7fe84c381700 1 -- 172.21.15.46:0/3124516106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8440cccd0 msgr2=0x7fe8440c3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.906+0000 7fe84c381700 1 --2- 172.21.15.46:0/3124516106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8440cccd0 0x7fe8440c3b30 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fe84406fca0 tx=0x7fe834009250 comp rx=0 tx=0).stop 2024-01-26T23:49:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.906+0000 7fe84c381700 1 -- 172.21.15.46:0/3124516106 shutdown_connections 2024-01-26T23:49:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.906+0000 7fe84c381700 1 --2- 172.21.15.46:0/3124516106 >> v2:172.21.15.106:3300/0 conn(0x7fe8440c2240 0x7fe8440c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.906+0000 7fe84c381700 1 --2- 172.21.15.46:0/3124516106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8440cccd0 0x7fe8440c3b30 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.906+0000 7fe84c381700 1 -- 172.21.15.46:0/3124516106 >> 172.21.15.46:0/3124516106 conn(0x7fe844069220 msgr2=0x7fe8440cd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.906+0000 7fe84c381700 1 -- 172.21.15.46:0/3124516106 shutdown_connections 2024-01-26T23:49:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.906+0000 7fe84c381700 1 -- 172.21.15.46:0/3124516106 wait complete. 2024-01-26T23:49:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.908+0000 7fe84c381700 1 Processor -- start 2024-01-26T23:49:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.908+0000 7fe84c381700 1 -- start start 2024-01-26T23:49:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.909+0000 7fe84c381700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8440c2240 0x7fe844158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.909+0000 7fe84c381700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8440cccd0 0x7fe844097050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.909+0000 7fe84c381700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe84415cd60 0x7fe8440b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.909+0000 7fe84c381700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8441698b0 con 0x7fe8440c2240 2024-01-26T23:49:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.909+0000 7fe84c381700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe844169b20 con 0x7fe8440cccd0 2024-01-26T23:49:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.909+0000 7fe84c381700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe844098370 con 0x7fe84415cd60 2024-01-26T23:49:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.909+0000 7fe84a11d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8440c2240 0x7fe844158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.909+0000 7fe84a11d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8440c2240 0x7fe844158fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60078/0 (socket says 172.21.15.46:60078) 2024-01-26T23:49:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.909+0000 7fe84a11d700 1 -- 172.21.15.46:0/1526613116 learned_addr learned my addr 172.21.15.46:0/1526613116 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.909+0000 7fe84991c700 1 --2- 172.21.15.46:0/1526613116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8440cccd0 0x7fe844097050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.909+0000 7fe84a91e700 1 --2- 172.21.15.46:0/1526613116 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe84415cd60 0x7fe8440b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.910+0000 7fe84a11d700 1 -- 172.21.15.46:0/1526613116 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe84415cd60 msgr2=0x7fe8440b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.910+0000 7fe84a11d700 1 --2- 172.21.15.46:0/1526613116 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe84415cd60 0x7fe8440b1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.910+0000 7fe84a11d700 1 -- 172.21.15.46:0/1526613116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8440cccd0 msgr2=0x7fe844097050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.910+0000 7fe84a11d700 1 --2- 172.21.15.46:0/1526613116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8440cccd0 0x7fe844097050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.910+0000 7fe84a11d700 1 -- 172.21.15.46:0/1526613116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8340094f0 con 0x7fe8440c2240 2024-01-26T23:49:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.910+0000 7fe84991c700 1 --2- 172.21.15.46:0/1526613116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8440cccd0 0x7fe844097050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.910+0000 7fe84a11d700 1 --2- 172.21.15.46:0/1526613116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8440c2240 0x7fe844158fe0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fe84000c5b0 tx=0x7fe840007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.910+0000 7fe83a7fc700 1 -- 172.21.15.46:0/1526613116 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8400072e0 con 0x7fe8440c2240 2024-01-26T23:49:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.910+0000 7fe83a7fc700 1 -- 172.21.15.46:0/1526613116 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe840007460 con 0x7fe8440c2240 2024-01-26T23:49:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.910+0000 7fe84c381700 1 -- 172.21.15.46:0/1526613116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe844097ad0 con 0x7fe8440c2240 2024-01-26T23:49:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.910+0000 7fe83a7fc700 1 -- 172.21.15.46:0/1526613116 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8400057f0 con 0x7fe8440c2240 2024-01-26T23:49:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.910+0000 7fe84c381700 1 -- 172.21.15.46:0/1526613116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe844098110 con 0x7fe8440c2240 2024-01-26T23:49:34.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.911+0000 7fe84c381700 1 -- 172.21.15.46:0/1526613116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe844169dc0 con 0x7fe8440c2240 2024-01-26T23:49:34.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.912+0000 7fe83a7fc700 1 -- 172.21.15.46:0/1526613116 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe840005bc0 con 0x7fe8440c2240 2024-01-26T23:49:34.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.912+0000 7fe83a7fc700 1 --2- 172.21.15.46:0/1526613116 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe828066f30 0x7fe8280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:34.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.912+0000 7fe83a7fc700 1 -- 172.21.15.46:0/1526613116 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fe840083d20 con 0x7fe8440c2240 2024-01-26T23:49:34.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.912+0000 7fe84991c700 1 --2- 172.21.15.46:0/1526613116 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe828066f30 0x7fe8280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:34.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.913+0000 7fe84991c700 1 --2- 172.21.15.46:0/1526613116 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe828066f30 0x7fe8280693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe84406fca0 tx=0x7fe834017000 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:34.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:34.916+0000 7fe83a7fc700 1 -- 172.21.15.46:0/1526613116 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe840050cc0 con 0x7fe8440c2240 2024-01-26T23:49:35.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.150+0000 7fe84c381700 1 -- 172.21.15.46:0/1526613116 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe84404c710 con 0x7fe828066f30 2024-01-26T23:49:35.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.152+0000 7fe83a7fc700 1 -- 172.21.15.46:0/1526613116 <== mgr.7148 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe84404c710 con 0x7fe828066f30 2024-01-26T23:49:35.153 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-26T23:49:35.154 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:49:35.154 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephfs" 2024-01-26T23:49:35.154 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:49:35.154 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-26T23:49:35.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.154+0000 7fe84c381700 1 -- 172.21.15.46:0/1526613116 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe828066f30 msgr2=0x7fe8280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:35.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.154+0000 7fe84c381700 1 --2- 172.21.15.46:0/1526613116 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe828066f30 0x7fe8280693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe84406fca0 tx=0x7fe834017000 comp rx=0 tx=0).stop 2024-01-26T23:49:35.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.154+0000 7fe84c381700 1 -- 172.21.15.46:0/1526613116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8440c2240 msgr2=0x7fe844158fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:35.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.154+0000 7fe84c381700 1 --2- 172.21.15.46:0/1526613116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8440c2240 0x7fe844158fe0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fe84000c5b0 tx=0x7fe840007f40 comp rx=0 tx=0).stop 2024-01-26T23:49:35.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.154+0000 7fe84c381700 1 -- 172.21.15.46:0/1526613116 shutdown_connections 2024-01-26T23:49:35.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.154+0000 7fe84c381700 1 --2- 172.21.15.46:0/1526613116 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe828066f30 0x7fe8280693e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:35.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.154+0000 7fe84c381700 1 --2- 172.21.15.46:0/1526613116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8440c2240 0x7fe844158fe0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:35.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.154+0000 7fe84c381700 1 --2- 172.21.15.46:0/1526613116 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe84415cd60 0x7fe8440b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:35.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.154+0000 7fe84c381700 1 --2- 172.21.15.46:0/1526613116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8440cccd0 0x7fe844097050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:35.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.154+0000 7fe84c381700 1 -- 172.21.15.46:0/1526613116 >> 172.21.15.46:0/1526613116 conn(0x7fe844069220 msgr2=0x7fe84406c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:35.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.154+0000 7fe84c381700 1 -- 172.21.15.46:0/1526613116 shutdown_connections 2024-01-26T23:49:35.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.154+0000 7fe84c381700 1 -- 172.21.15.46:0/1526613116 wait complete. 2024-01-26T23:49:35.169 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-26T23:49:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.300+0000 7fcbd04a5700 1 Processor -- start 2024-01-26T23:49:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.300+0000 7fcbd04a5700 1 -- start start 2024-01-26T23:49:35.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.301+0000 7fcbd04a5700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcbc80ca650 0x7fcbc8067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:35.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.301+0000 7fcbd04a5700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc806fdb0 con 0x7fcbc8067af0 2024-01-26T23:49:35.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.301+0000 7fcbd04a5700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc806f1a0 con 0x7fcbc806ebe0 2024-01-26T23:49:35.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.301+0000 7fcbd04a5700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc806f480 con 0x7fcbc80ca650 2024-01-26T23:49:35.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.301+0000 7fcbcea42700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcbc80ca650 0x7fcbc8067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.301+0000 7fcbce241700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcbc8067af0 0x7fcbc80ad8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39280/0 (socket says 172.21.15.46:39280) 2024-01-26T23:49:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.301+0000 7fcbce241700 1 -- 172.21.15.46:0/2134558421 learned_addr learned my addr 172.21.15.46:0/2134558421 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.302+0000 7fcbcea42700 1 -- 172.21.15.46:0/2134558421 >> v1:172.21.15.106:6789/0 conn(0x7fcbc806ebe0 legacy=0x7fcbc81597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.302+0000 7fcbcea42700 1 -- 172.21.15.46:0/2134558421 >> v1:172.21.15.46:6789/0 conn(0x7fcbc8067af0 legacy=0x7fcbc80ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.302+0000 7fcbcea42700 1 -- 172.21.15.46:0/2134558421 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc806f760 con 0x7fcbc80ca650 2024-01-26T23:49:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.302+0000 7fcbcd23f700 1 -- 172.21.15.46:0/2134558421 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032358815 0 0) 0x7fcbc806fdb0 con 0x7fcbc8067af0 2024-01-26T23:49:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.302+0000 7fcbcea42700 1 --2- 172.21.15.46:0/2134558421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc80ca650 0x7fcbc8067ec0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009e60 tx=0x7fcbc0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=961f9a38110fa1b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.302+0000 7fcbcd23f700 1 -- 172.21.15.46:0/2134558421 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc000e070 con 0x7fcbc80ca650 2024-01-26T23:49:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.302+0000 7fcbcd23f700 1 -- 172.21.15.46:0/2134558421 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc000bc30 con 0x7fcbc80ca650 2024-01-26T23:49:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.302+0000 7fcbcd23f700 1 -- 172.21.15.46:0/2134558421 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0012430 con 0x7fcbc80ca650 2024-01-26T23:49:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.303+0000 7fcbd04a5700 1 -- 172.21.15.46:0/2134558421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc80ca650 msgr2=0x7fcbc8067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.303+0000 7fcbd04a5700 1 --2- 172.21.15.46:0/2134558421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc80ca650 0x7fcbc8067ec0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009e60 tx=0x7fcbc0009780 comp rx=0 tx=0).stop 2024-01-26T23:49:35.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.303+0000 7fcbd04a5700 1 -- 172.21.15.46:0/2134558421 shutdown_connections 2024-01-26T23:49:35.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.303+0000 7fcbd04a5700 1 --2- 172.21.15.46:0/2134558421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc80ca650 0x7fcbc8067ec0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:35.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.303+0000 7fcbd04a5700 1 -- 172.21.15.46:0/2134558421 >> 172.21.15.46:0/2134558421 conn(0x7fcbc80cdf80 msgr2=0x7fcbc806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:35.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.303+0000 7fcbd04a5700 1 -- 172.21.15.46:0/2134558421 shutdown_connections 2024-01-26T23:49:35.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.303+0000 7fcbd04a5700 1 -- 172.21.15.46:0/2134558421 wait complete. 2024-01-26T23:49:35.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.304+0000 7fcbd04a5700 1 Processor -- start 2024-01-26T23:49:35.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.304+0000 7fcbd04a5700 1 -- start start 2024-01-26T23:49:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbd04a5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbc8067af0 0x7fcbc8155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbd04a5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc806ebe0 0x7fcbc8158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbd04a5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbc80ca650 0x7fcbc815a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbd04a5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc81586f0 con 0x7fcbc806ebe0 2024-01-26T23:49:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbd04a5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc8157ae0 con 0x7fcbc8067af0 2024-01-26T23:49:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbd04a5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc8157dc0 con 0x7fcbc80ca650 2024-01-26T23:49:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbcda40700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc806ebe0 0x7fcbc8158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbcda40700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc806ebe0 0x7fcbc8158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60094/0 (socket says 172.21.15.46:60094) 2024-01-26T23:49:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbcda40700 1 -- 172.21.15.46:0/4102111348 learned_addr learned my addr 172.21.15.46:0/4102111348 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbce241700 1 --2- 172.21.15.46:0/4102111348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbc8067af0 0x7fcbc8155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbcea42700 1 --2- 172.21.15.46:0/4102111348 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbc80ca650 0x7fcbc815a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbcda40700 1 -- 172.21.15.46:0/4102111348 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbc80ca650 msgr2=0x7fcbc815a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbcda40700 1 --2- 172.21.15.46:0/4102111348 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbc80ca650 0x7fcbc815a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbcda40700 1 -- 172.21.15.46:0/4102111348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbc8067af0 msgr2=0x7fcbc8155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbcda40700 1 --2- 172.21.15.46:0/4102111348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbc8067af0 0x7fcbc8155130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbcda40700 1 -- 172.21.15.46:0/4102111348 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc0009a20 con 0x7fcbc806ebe0 2024-01-26T23:49:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbce241700 1 --2- 172.21.15.46:0/4102111348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbc8067af0 0x7fcbc8155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:49:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.305+0000 7fcbcda40700 1 --2- 172.21.15.46:0/4102111348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc806ebe0 0x7fcbc8158b80 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800c590 tx=0x7fcbb8007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:35.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.306+0000 7fcbbe7fc700 1 -- 172.21.15.46:0/4102111348 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8011590 con 0x7fcbc806ebe0 2024-01-26T23:49:35.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.306+0000 7fcbbe7fc700 1 -- 172.21.15.46:0/4102111348 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb8011730 con 0x7fcbc806ebe0 2024-01-26T23:49:35.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.306+0000 7fcbd04a5700 1 -- 172.21.15.46:0/4102111348 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc81580a0 con 0x7fcbc806ebe0 2024-01-26T23:49:35.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.306+0000 7fcbbe7fc700 1 -- 172.21.15.46:0/4102111348 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8011af0 con 0x7fcbc806ebe0 2024-01-26T23:49:35.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.306+0000 7fcbd04a5700 1 -- 172.21.15.46:0/4102111348 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc8155670 con 0x7fcbc806ebe0 2024-01-26T23:49:35.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.307+0000 7fcbd04a5700 1 -- 172.21.15.46:0/4102111348 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc804c710 con 0x7fcbc806ebe0 2024-01-26T23:49:35.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.307+0000 7fcbbe7fc700 1 -- 172.21.15.46:0/4102111348 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 66) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fcbb8012420 con 0x7fcbc806ebe0 2024-01-26T23:49:35.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.308+0000 7fcbbe7fc700 1 --2- 172.21.15.46:0/4102111348 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fcbac066f30 0x7fcbac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:35.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.308+0000 7fcbbe7fc700 1 -- 172.21.15.46:0/4102111348 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fcbb8083b40 con 0x7fcbc806ebe0 2024-01-26T23:49:35.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.308+0000 7fcbce241700 1 --2- 172.21.15.46:0/4102111348 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fcbac066f30 0x7fcbac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:35.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.309+0000 7fcbce241700 1 --2- 172.21.15.46:0/4102111348 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fcbac066f30 0x7fcbac0693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4001810 tx=0x7fcbc4009040 comp rx=0 tx=0).ready entity=mgr.7148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:35.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.313+0000 7fcbbe7fc700 1 -- 172.21.15.46:0/4102111348 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcbb804b5a0 con 0x7fcbc806ebe0 2024-01-26T23:49:35.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.577+0000 7fcbd04a5700 1 -- 172.21.15.46:0/4102111348 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fcbc80cbc40 con 0x7fcbc806ebe0 2024-01-26T23:49:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.962+0000 7fcbbe7fc700 1 -- 172.21.15.46:0/4102111348 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 67) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fcbb8050f60 con 0x7fcbc806ebe0 2024-01-26T23:49:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.964+0000 7fcbbe7fc700 1 -- 172.21.15.46:0/4102111348 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v67) v1 ==== 92+0+0 (secure 0 0 0) 0x7fcbb8050b90 con 0x7fcbc806ebe0 2024-01-26T23:49:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.967+0000 7fcbd04a5700 1 -- 172.21.15.46:0/4102111348 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fcbac066f30 msgr2=0x7fcbac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.967+0000 7fcbd04a5700 1 --2- 172.21.15.46:0/4102111348 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fcbac066f30 0x7fcbac0693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4001810 tx=0x7fcbc4009040 comp rx=0 tx=0).stop 2024-01-26T23:49:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.967+0000 7fcbd04a5700 1 -- 172.21.15.46:0/4102111348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc806ebe0 msgr2=0x7fcbc8158b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.967+0000 7fcbd04a5700 1 --2- 172.21.15.46:0/4102111348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc806ebe0 0x7fcbc8158b80 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800c590 tx=0x7fcbb8007f30 comp rx=0 tx=0).stop 2024-01-26T23:49:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.967+0000 7fcbd04a5700 1 -- 172.21.15.46:0/4102111348 shutdown_connections 2024-01-26T23:49:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.967+0000 7fcbd04a5700 1 --2- 172.21.15.46:0/4102111348 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fcbac066f30 0x7fcbac0693e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.967+0000 7fcbd04a5700 1 --2- 172.21.15.46:0/4102111348 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbc80ca650 0x7fcbc815a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.967+0000 7fcbd04a5700 1 --2- 172.21.15.46:0/4102111348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbc8067af0 0x7fcbc8155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.967+0000 7fcbd04a5700 1 --2- 172.21.15.46:0/4102111348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbc806ebe0 0x7fcbc8158b80 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.967+0000 7fcbd04a5700 1 -- 172.21.15.46:0/4102111348 >> 172.21.15.46:0/4102111348 conn(0x7fcbc80cdf80 msgr2=0x7fcbc80af330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.967+0000 7fcbd04a5700 1 -- 172.21.15.46:0/4102111348 shutdown_connections 2024-01-26T23:49:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:35.967+0000 7fcbd04a5700 1 -- 172.21.15.46:0/4102111348 wait complete. 2024-01-26T23:49:35.984 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-26T23:49:36.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.123+0000 7fbedf538700 1 Processor -- start 2024-01-26T23:49:36.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.123+0000 7fbedf538700 1 -- start start 2024-01-26T23:49:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.124+0000 7fbedf538700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbed80cf050 0x7fbed80af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.124+0000 7fbedf538700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed80cc100 con 0x7fbed8159cb0 2024-01-26T23:49:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.124+0000 7fbedf538700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed80cb850 con 0x7fbed80cf4a0 2024-01-26T23:49:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.124+0000 7fbedf538700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed80ccca0 con 0x7fbed80cf050 2024-01-26T23:49:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.124+0000 7fbedd2d4700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbed8159cb0 0x7fbed80cc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39302/0 (socket says 172.21.15.46:39302) 2024-01-26T23:49:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.124+0000 7fbedd2d4700 1 -- 172.21.15.46:0/3514183269 learned_addr learned my addr 172.21.15.46:0/3514183269 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.124+0000 7fbedcad3700 1 --2- 172.21.15.46:0/3514183269 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbed80cf050 0x7fbed80af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.125+0000 7fbecffff700 1 -- 172.21.15.46:0/3514183269 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484858879 0 0) 0x7fbed80cc100 con 0x7fbed8159cb0 2024-01-26T23:49:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.125+0000 7fbecffff700 1 -- 172.21.15.46:0/3514183269 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbec4003570 con 0x7fbed8159cb0 2024-01-26T23:49:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.125+0000 7fbecffff700 1 -- 172.21.15.46:0/3514183269 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911577108 0 0) 0x7fbed80cb850 con 0x7fbed80cf4a0 2024-01-26T23:49:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.125+0000 7fbecffff700 1 -- 172.21.15.46:0/3514183269 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed80cc100 con 0x7fbed80cf4a0 2024-01-26T23:49:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.125+0000 7fbedcad3700 1 -- 172.21.15.46:0/3514183269 >> v1:172.21.15.106:6789/0 conn(0x7fbed80cf4a0 legacy=0x7fbed80c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.125+0000 7fbedcad3700 1 -- 172.21.15.46:0/3514183269 >> v1:172.21.15.46:6789/0 conn(0x7fbed8159cb0 legacy=0x7fbed80cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.125+0000 7fbedcad3700 1 -- 172.21.15.46:0/3514183269 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed80cbaf0 con 0x7fbed80cf050 2024-01-26T23:49:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.125+0000 7fbedcad3700 1 --2- 172.21.15.46:0/3514183269 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbed80cf050 0x7fbed80af430 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009910 tx=0x7fbec80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ce4473c6a725557 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.125+0000 7fbecffff700 1 -- 172.21.15.46:0/3514183269 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec8011630 con 0x7fbed80cf050 2024-01-26T23:49:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.125+0000 7fbecffff700 1 -- 172.21.15.46:0/3514183269 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec8011790 con 0x7fbed80cf050 2024-01-26T23:49:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.125+0000 7fbecffff700 1 -- 172.21.15.46:0/3514183269 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec8011b30 con 0x7fbed80cf050 2024-01-26T23:49:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.126+0000 7fbedf538700 1 -- 172.21.15.46:0/3514183269 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbed80cf050 msgr2=0x7fbed80af430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.126+0000 7fbedf538700 1 --2- 172.21.15.46:0/3514183269 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbed80cf050 0x7fbed80af430 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009910 tx=0x7fbec80091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.126+0000 7fbedf538700 1 -- 172.21.15.46:0/3514183269 shutdown_connections 2024-01-26T23:49:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.126+0000 7fbedf538700 1 --2- 172.21.15.46:0/3514183269 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbed80cf050 0x7fbed80af430 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.126+0000 7fbedf538700 1 -- 172.21.15.46:0/3514183269 >> 172.21.15.46:0/3514183269 conn(0x7fbed8087f60 msgr2=0x7fbed806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.126+0000 7fbedf538700 1 -- 172.21.15.46:0/3514183269 shutdown_connections 2024-01-26T23:49:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.126+0000 7fbedf538700 1 -- 172.21.15.46:0/3514183269 wait complete. 2024-01-26T23:49:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.127+0000 7fbedf538700 1 Processor -- start 2024-01-26T23:49:36.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.127+0000 7fbedf538700 1 -- start start 2024-01-26T23:49:36.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.128+0000 7fbedf538700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbed80cf050 0x7fbed809c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:36.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.128+0000 7fbedf538700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbed80cf4a0 0x7fbed8068fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:36.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.128+0000 7fbedf538700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbed8159cb0 0x7fbed8069520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:36.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.128+0000 7fbedf538700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed8157820 con 0x7fbed80cf4a0 2024-01-26T23:49:36.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.128+0000 7fbedf538700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed816aa50 con 0x7fbed8159cb0 2024-01-26T23:49:36.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.128+0000 7fbedf538700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed8156ef0 con 0x7fbed80cf050 2024-01-26T23:49:36.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.129+0000 7fbedcad3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbed80cf4a0 0x7fbed8068fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:36.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.129+0000 7fbedd2d4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbed80cf050 0x7fbed809c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:36.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.129+0000 7fbedcad3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbed80cf4a0 0x7fbed8068fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60110/0 (socket says 172.21.15.46:60110) 2024-01-26T23:49:36.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.129+0000 7fbedd2d4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbed80cf050 0x7fbed809c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:47664/0 (socket says 172.21.15.46:47664) 2024-01-26T23:49:36.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.129+0000 7fbedcad3700 1 -- 172.21.15.46:0/435335380 learned_addr learned my addr 172.21.15.46:0/435335380 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:36.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.129+0000 7fbeddad5700 1 --2- 172.21.15.46:0/435335380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbed8159cb0 0x7fbed8069520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:36.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.129+0000 7fbedd2d4700 1 -- 172.21.15.46:0/435335380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbed8159cb0 msgr2=0x7fbed8069520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:36.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.129+0000 7fbedd2d4700 1 --2- 172.21.15.46:0/435335380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbed8159cb0 0x7fbed8069520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.129+0000 7fbedd2d4700 1 -- 172.21.15.46:0/435335380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbed80cf4a0 msgr2=0x7fbed8068fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:36.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.129+0000 7fbedd2d4700 1 --2- 172.21.15.46:0/435335380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbed80cf4a0 0x7fbed8068fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.129+0000 7fbedd2d4700 1 -- 172.21.15.46:0/435335380 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec8009420 con 0x7fbed80cf050 2024-01-26T23:49:36.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.129+0000 7fbedcad3700 1 --2- 172.21.15.46:0/435335380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbed80cf4a0 0x7fbed8068fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:36.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.129+0000 7fbedd2d4700 1 --2- 172.21.15.46:0/435335380 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbed80cf050 0x7fbed809c590 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fbed4001260 tx=0x7fbed4007c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:36.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.129+0000 7fbecd7fa700 1 -- 172.21.15.46:0/435335380 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed40072e0 con 0x7fbed80cf050 2024-01-26T23:49:36.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.129+0000 7fbecd7fa700 1 -- 172.21.15.46:0/435335380 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed4007440 con 0x7fbed80cf050 2024-01-26T23:49:36.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.129+0000 7fbedf538700 1 -- 172.21.15.46:0/435335380 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed81571d0 con 0x7fbed80cf050 2024-01-26T23:49:36.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.130+0000 7fbecd7fa700 1 -- 172.21.15.46:0/435335380 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed40057f0 con 0x7fbed80cf050 2024-01-26T23:49:36.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.130+0000 7fbedf538700 1 -- 172.21.15.46:0/435335380 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed8154f20 con 0x7fbed80cf050 2024-01-26T23:49:36.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.131+0000 7fbedf538700 1 -- 172.21.15.46:0/435335380 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed804c710 con 0x7fbed80cf050 2024-01-26T23:49:36.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.131+0000 7fbecd7fa700 1 -- 172.21.15.46:0/435335380 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 67) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fbed4005c20 con 0x7fbed80cf050 2024-01-26T23:49:36.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.131+0000 7fbecd7fa700 1 --2- 172.21.15.46:0/435335380 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbebc066f80 0x7fbebc069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:36.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.131+0000 7fbedcad3700 1 -- 172.21.15.46:0/435335380 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbebc066f80 msgr2=0x7fbebc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:36.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.131+0000 7fbecd7fa700 1 -- 172.21.15.46:0/435335380 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fbed4083d40 con 0x7fbed80cf050 2024-01-26T23:49:36.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.131+0000 7fbedcad3700 1 --2- 172.21.15.46:0/435335380 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbebc066f80 0x7fbebc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:36.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.136+0000 7fbecd7fa700 1 -- 172.21.15.46:0/435335380 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbed4050c50 con 0x7fbed80cf050 2024-01-26T23:49:36.302 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:36.301+0000 7f44f12601c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:49:36.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.332+0000 7fbedcad3700 1 -- 172.21.15.46:0/435335380 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbebc066f80 msgr2=0x7fbebc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:36.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.332+0000 7fbedcad3700 1 --2- 172.21.15.46:0/435335380 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbebc066f80 0x7fbebc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:36.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.345+0000 7fbedf538700 1 -- 172.21.15.46:0/435335380 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fbed80cfc80 con 0x7fbed80cf050 2024-01-26T23:49:36.347 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:36.346+0000 7f8a485811c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:49:36.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.350+0000 7fbecd7fa700 1 -- 172.21.15.46:0/435335380 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v47)=0 v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbed4017070 con 0x7fbed80cf050 2024-01-26T23:49:36.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.352+0000 7fbedf538700 1 -- 172.21.15.46:0/435335380 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbebc066f80 msgr2=0x7fbebc069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:36.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.352+0000 7fbedf538700 1 --2- 172.21.15.46:0/435335380 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbebc066f80 0x7fbebc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.352+0000 7fbedf538700 1 -- 172.21.15.46:0/435335380 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbed80cf050 msgr2=0x7fbed809c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:36.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.352+0000 7fbedf538700 1 --2- 172.21.15.46:0/435335380 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbed80cf050 0x7fbed809c590 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fbed4001260 tx=0x7fbed4007c00 comp rx=0 tx=0).stop 2024-01-26T23:49:36.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.352+0000 7fbedf538700 1 -- 172.21.15.46:0/435335380 shutdown_connections 2024-01-26T23:49:36.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.352+0000 7fbedf538700 1 --2- 172.21.15.46:0/435335380 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbebc066f80 0x7fbebc069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.352+0000 7fbedf538700 1 --2- 172.21.15.46:0/435335380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbed8159cb0 0x7fbed8069520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.352+0000 7fbedf538700 1 --2- 172.21.15.46:0/435335380 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbed80cf050 0x7fbed809c590 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.352+0000 7fbedf538700 1 --2- 172.21.15.46:0/435335380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbed80cf4a0 0x7fbed8068fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.352+0000 7fbedf538700 1 -- 172.21.15.46:0/435335380 >> 172.21.15.46:0/435335380 conn(0x7fbed8087f60 msgr2=0x7fbed80b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:36.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.352+0000 7fbedf538700 1 -- 172.21.15.46:0/435335380 shutdown_connections 2024-01-26T23:49:36.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.352+0000 7fbedf538700 1 -- 172.21.15.46:0/435335380 wait complete. 2024-01-26T23:49:36.368 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-26T23:49:36.379 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:36.379+0000 7f44f12601c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:49:36.439 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:36.437+0000 7f8a485811c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:49:36.450 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:36.449+0000 7f44f12601c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:49:36.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.503+0000 7f61d3a54700 1 Processor -- start 2024-01-26T23:49:36.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.504+0000 7f61d3a54700 1 -- start start 2024-01-26T23:49:36.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.505+0000 7f61d3a54700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f61cc0cb640 0x7f61cc0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:36.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.505+0000 7f61d3a54700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61cc0ccc00 con 0x7f61cc0c8da0 2024-01-26T23:49:36.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.505+0000 7f61d3a54700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61cc0cc310 con 0x7f61cc141200 2024-01-26T23:49:36.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.505+0000 7f61d3a54700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61cc0cbbf0 con 0x7f61cc0cb640 2024-01-26T23:49:36.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.505+0000 7f61d1ff1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f61cc0cb640 0x7f61cc0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:36.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.505+0000 7f61d17f0700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f61cc0c8da0 0x7f61cc140af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39318/0 (socket says 172.21.15.46:39318) 2024-01-26T23:49:36.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.505+0000 7f61d17f0700 1 -- 172.21.15.46:0/1033018174 learned_addr learned my addr 172.21.15.46:0/1033018174 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:36.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.505+0000 7f61d1ff1700 1 -- 172.21.15.46:0/1033018174 >> v1:172.21.15.106:6789/0 conn(0x7f61cc141200 legacy=0x7f61cc0caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:36.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.505+0000 7f61d1ff1700 1 -- 172.21.15.46:0/1033018174 >> v1:172.21.15.46:6789/0 conn(0x7f61cc0c8da0 legacy=0x7f61cc140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:36.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.505+0000 7f61d1ff1700 1 -- 172.21.15.46:0/1033018174 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61cc0cc5b0 con 0x7f61cc0cb640 2024-01-26T23:49:36.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.505+0000 7f61d1ff1700 1 --2- 172.21.15.46:0/1033018174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61cc0cb640 0x7f61cc0c9170 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f61c4009e60 tx=0x7f61c4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c575e3ce67bd9f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:36.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.506+0000 7f61c3fff700 1 -- 172.21.15.46:0/1033018174 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c400e070 con 0x7f61cc0cb640 2024-01-26T23:49:36.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.506+0000 7f61c3fff700 1 -- 172.21.15.46:0/1033018174 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c400bc30 con 0x7f61cc0cb640 2024-01-26T23:49:36.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.506+0000 7f61c3fff700 1 -- 172.21.15.46:0/1033018174 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c4012430 con 0x7f61cc0cb640 2024-01-26T23:49:36.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.506+0000 7f61d3a54700 1 -- 172.21.15.46:0/1033018174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61cc0cb640 msgr2=0x7f61cc0c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:36.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.506+0000 7f61d3a54700 1 --2- 172.21.15.46:0/1033018174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61cc0cb640 0x7f61cc0c9170 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f61c4009e60 tx=0x7f61c4009780 comp rx=0 tx=0).stop 2024-01-26T23:49:36.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.506+0000 7f61d3a54700 1 -- 172.21.15.46:0/1033018174 shutdown_connections 2024-01-26T23:49:36.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.506+0000 7f61d3a54700 1 --2- 172.21.15.46:0/1033018174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61cc0cb640 0x7f61cc0c9170 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.506+0000 7f61d3a54700 1 -- 172.21.15.46:0/1033018174 >> 172.21.15.46:0/1033018174 conn(0x7f61cc06bd90 msgr2=0x7f61cc06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:36.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.506+0000 7f61d3a54700 1 -- 172.21.15.46:0/1033018174 shutdown_connections 2024-01-26T23:49:36.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.506+0000 7f61d3a54700 1 -- 172.21.15.46:0/1033018174 wait complete. 2024-01-26T23:49:36.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.508+0000 7f61d3a54700 1 Processor -- start 2024-01-26T23:49:36.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.508+0000 7f61d3a54700 1 -- start start 2024-01-26T23:49:36.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.509+0000 7f61d3a54700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61cc0c8da0 0x7f61cc16b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:36.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.509+0000 7f61d3a54700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61cc0cb640 0x7f61cc156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:36.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.509+0000 7f61d3a54700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61cc141200 0x7f61cc0b1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:36.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.509+0000 7f61d3a54700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc0c7660 con 0x7f61cc0c8da0 2024-01-26T23:49:36.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.509+0000 7f61d3a54700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc16c000 con 0x7f61cc0cb640 2024-01-26T23:49:36.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.509+0000 7f61d3a54700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc0b29e0 con 0x7f61cc141200 2024-01-26T23:49:36.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.509+0000 7f61d17f0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61cc0c8da0 0x7f61cc16b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:36.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.509+0000 7f61d17f0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61cc0c8da0 0x7f61cc16b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60124/0 (socket says 172.21.15.46:60124) 2024-01-26T23:49:36.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.509+0000 7f61d17f0700 1 -- 172.21.15.46:0/3575480743 learned_addr learned my addr 172.21.15.46:0/3575480743 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:36.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.509+0000 7f61d1ff1700 1 --2- 172.21.15.46:0/3575480743 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61cc141200 0x7f61cc0b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:36.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.509+0000 7f61d0fef700 1 --2- 172.21.15.46:0/3575480743 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61cc0cb640 0x7f61cc156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:36.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.509+0000 7f61d17f0700 1 -- 172.21.15.46:0/3575480743 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61cc141200 msgr2=0x7f61cc0b1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:36.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.509+0000 7f61d17f0700 1 --2- 172.21.15.46:0/3575480743 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61cc141200 0x7f61cc0b1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.509+0000 7f61d17f0700 1 -- 172.21.15.46:0/3575480743 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61cc0cb640 msgr2=0x7f61cc156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:36.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.509+0000 7f61d17f0700 1 --2- 172.21.15.46:0/3575480743 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61cc0cb640 0x7f61cc156e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.509+0000 7f61d17f0700 1 -- 172.21.15.46:0/3575480743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c4009a20 con 0x7f61cc0c8da0 2024-01-26T23:49:36.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.509+0000 7f61d17f0700 1 --2- 172.21.15.46:0/3575480743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61cc0c8da0 0x7f61cc16b160 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f61c800c7c0 tx=0x7f61c8007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:36.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.510+0000 7f61c17fa700 1 -- 172.21.15.46:0/3575480743 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c8005b30 con 0x7f61cc0c8da0 2024-01-26T23:49:36.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.510+0000 7f61c17fa700 1 -- 172.21.15.46:0/3575480743 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c8005c90 con 0x7f61cc0c8da0 2024-01-26T23:49:36.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.510+0000 7f61c17fa700 1 -- 172.21.15.46:0/3575480743 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c8004a80 con 0x7f61cc0c8da0 2024-01-26T23:49:36.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.510+0000 7f61d3a54700 1 -- 172.21.15.46:0/3575480743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61cc16b760 con 0x7f61cc0c8da0 2024-01-26T23:49:36.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.510+0000 7f61d3a54700 1 -- 172.21.15.46:0/3575480743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61cc16bd20 con 0x7f61cc0c8da0 2024-01-26T23:49:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.511+0000 7f61d3a54700 1 -- 172.21.15.46:0/3575480743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61cc04c710 con 0x7f61cc0c8da0 2024-01-26T23:49:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.512+0000 7f61c17fa700 1 -- 172.21.15.46:0/3575480743 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 67) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f61c80070d0 con 0x7f61cc0c8da0 2024-01-26T23:49:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.512+0000 7f61c17fa700 1 --2- 172.21.15.46:0/3575480743 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f61b0066f80 0x7f61b0069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.512+0000 7f61d0fef700 1 -- 172.21.15.46:0/3575480743 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f61b0066f80 msgr2=0x7f61b0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.512+0000 7f61c17fa700 1 -- 172.21.15.46:0/3575480743 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f61c8083b50 con 0x7f61cc0c8da0 2024-01-26T23:49:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.512+0000 7f61d0fef700 1 --2- 172.21.15.46:0/3575480743 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f61b0066f80 0x7f61b0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:36.516 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:36.514+0000 7f44f12601c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:49:36.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.517+0000 7f61c17fa700 1 -- 172.21.15.46:0/3575480743 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f61c804b5a0 con 0x7f61cc0c8da0 2024-01-26T23:49:36.522 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:36.521+0000 7f8a485811c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:49:36.598 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:36.597+0000 7f8a485811c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:49:36.670 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:36.670+0000 7f44f12601c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:49:36.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.713+0000 7f61d0fef700 1 -- 172.21.15.46:0/3575480743 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f61b0066f80 msgr2=0x7f61b0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:36.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.713+0000 7f61d0fef700 1 --2- 172.21.15.46:0/3575480743 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f61b0066f80 0x7f61b0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:36.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.730+0000 7f61d3a54700 1 -- 172.21.15.46:0/3575480743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f61cc0cc890 con 0x7f61cc0c8da0 2024-01-26T23:49:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.734+0000 7f61c17fa700 1 -- 172.21.15.46:0/3575480743 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v48)=0 v48) v1 ==== 133+0+0 (secure 0 0 0) 0x7f61c8050af0 con 0x7f61cc0c8da0 2024-01-26T23:49:36.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.737+0000 7f61d3a54700 1 -- 172.21.15.46:0/3575480743 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f61b0066f80 msgr2=0x7f61b0069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:36.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.737+0000 7f61d3a54700 1 --2- 172.21.15.46:0/3575480743 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f61b0066f80 0x7f61b0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.737+0000 7f61d3a54700 1 -- 172.21.15.46:0/3575480743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61cc0c8da0 msgr2=0x7f61cc16b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:36.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.737+0000 7f61d3a54700 1 --2- 172.21.15.46:0/3575480743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61cc0c8da0 0x7f61cc16b160 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f61c800c7c0 tx=0x7f61c8007f70 comp rx=0 tx=0).stop 2024-01-26T23:49:36.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.737+0000 7f61d3a54700 1 -- 172.21.15.46:0/3575480743 shutdown_connections 2024-01-26T23:49:36.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.737+0000 7f61d3a54700 1 --2- 172.21.15.46:0/3575480743 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f61b0066f80 0x7f61b0069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.737+0000 7f61d3a54700 1 --2- 172.21.15.46:0/3575480743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61cc0c8da0 0x7f61cc16b160 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.737+0000 7f61d3a54700 1 --2- 172.21.15.46:0/3575480743 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61cc141200 0x7f61cc0b1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.737+0000 7f61d3a54700 1 --2- 172.21.15.46:0/3575480743 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61cc0cb640 0x7f61cc156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.737+0000 7f61d3a54700 1 -- 172.21.15.46:0/3575480743 >> 172.21.15.46:0/3575480743 conn(0x7f61cc06bd90 msgr2=0x7f61cc13e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:36.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.737+0000 7f61d3a54700 1 -- 172.21.15.46:0/3575480743 shutdown_connections 2024-01-26T23:49:36.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.737+0000 7f61d3a54700 1 -- 172.21.15.46:0/3575480743 wait complete. 2024-01-26T23:49:36.752 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:36.753 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/minutes) 2024-01-26T23:49:36.777 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/minutes 1m 2024-01-26T23:49:36.777 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:36.776+0000 7f8a485811c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:49:36.809 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:36.809+0000 7f44f12601c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:49:36.875 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:36.874+0000 7f44f12601c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:49:36.942 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:36.940+0000 7f8a485811c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:49:36.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.961+0000 7f9155e35700 1 Processor -- start 2024-01-26T23:49:36.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.961+0000 7f9155e35700 1 -- start start 2024-01-26T23:49:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.962+0000 7f9155e35700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f91500c93a0 0x7f91500c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.962+0000 7f9155e35700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9150067d50 0x7f9150159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.962+0000 7f9155e35700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91500681f0 0x7f9150159760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.962+0000 7f9155e35700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f915015e390 con 0x7f91500c93a0 2024-01-26T23:49:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.962+0000 7f9155e35700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9150159ca0 con 0x7f9150067d50 2024-01-26T23:49:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.962+0000 7f9155e35700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f915015d810 con 0x7f91500681f0 2024-01-26T23:49:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.962+0000 7f914f7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f91500c93a0 0x7f91500c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.963+0000 7f914f7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f91500c93a0 0x7f91500c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60140/0 (socket says 172.21.15.46:60140) 2024-01-26T23:49:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.963+0000 7f914f7fe700 1 -- 172.21.15.46:0/739057517 learned_addr learned my addr 172.21.15.46:0/739057517 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.963+0000 7f914ffff700 1 --2- 172.21.15.46:0/739057517 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91500681f0 0x7f9150159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.963+0000 7f914effd700 1 --2- 172.21.15.46:0/739057517 >> v2:172.21.15.106:3300/0 conn(0x7f9150067d50 0x7f9150159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.963+0000 7f914f7fe700 1 -- 172.21.15.46:0/739057517 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91500681f0 msgr2=0x7f9150159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.963+0000 7f914f7fe700 1 --2- 172.21.15.46:0/739057517 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91500681f0 0x7f9150159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.963+0000 7f914f7fe700 1 -- 172.21.15.46:0/739057517 >> v2:172.21.15.106:3300/0 conn(0x7f9150067d50 msgr2=0x7f9150159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.963+0000 7f914f7fe700 1 --2- 172.21.15.46:0/739057517 >> v2:172.21.15.106:3300/0 conn(0x7f9150067d50 0x7f9150159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.963+0000 7f914f7fe700 1 -- 172.21.15.46:0/739057517 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915015daf0 con 0x7f91500c93a0 2024-01-26T23:49:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.963+0000 7f914f7fe700 1 --2- 172.21.15.46:0/739057517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91500c93a0 0x7f91500c9770 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f9150064390 tx=0x7f9140009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ba348b28b914ac server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.963+0000 7f914e7fc700 1 -- 172.21.15.46:0/739057517 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914000b640 con 0x7f91500c93a0 2024-01-26T23:49:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.963+0000 7f914e7fc700 1 -- 172.21.15.46:0/739057517 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914000b7a0 con 0x7f91500c93a0 2024-01-26T23:49:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.963+0000 7f914e7fc700 1 -- 172.21.15.46:0/739057517 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914000bb40 con 0x7f91500c93a0 2024-01-26T23:49:36.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.964+0000 7f9155e35700 1 -- 172.21.15.46:0/739057517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91500c93a0 msgr2=0x7f91500c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:36.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.964+0000 7f9155e35700 1 --2- 172.21.15.46:0/739057517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91500c93a0 0x7f91500c9770 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f9150064390 tx=0x7f9140009250 comp rx=0 tx=0).stop 2024-01-26T23:49:36.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.964+0000 7f9155e35700 1 -- 172.21.15.46:0/739057517 shutdown_connections 2024-01-26T23:49:36.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.964+0000 7f9155e35700 1 --2- 172.21.15.46:0/739057517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91500c93a0 0x7f91500c9770 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.964+0000 7f9155e35700 1 --2- 172.21.15.46:0/739057517 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91500681f0 0x7f9150159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.964+0000 7f9155e35700 1 --2- 172.21.15.46:0/739057517 >> v2:172.21.15.106:3300/0 conn(0x7f9150067d50 0x7f9150159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.964+0000 7f9155e35700 1 -- 172.21.15.46:0/739057517 >> 172.21.15.46:0/739057517 conn(0x7f91500ca9a0 msgr2=0x7f9150068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:36.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.964+0000 7f9155e35700 1 -- 172.21.15.46:0/739057517 shutdown_connections 2024-01-26T23:49:36.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.964+0000 7f9155e35700 1 -- 172.21.15.46:0/739057517 wait complete. 2024-01-26T23:49:36.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.966+0000 7f9155e35700 1 Processor -- start 2024-01-26T23:49:36.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.966+0000 7f9155e35700 1 -- start start 2024-01-26T23:49:36.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f9155e35700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9150067d50 0x7f9150169e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:36.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f9155e35700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91500681f0 0x7f9150150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:36.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f9155e35700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91500c93a0 0x7f91500b2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:36.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f9155e35700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915016d280 con 0x7f91500c93a0 2024-01-26T23:49:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f9155e35700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915016a350 con 0x7f9150067d50 2024-01-26T23:49:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f9155e35700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9150150e70 con 0x7f91500681f0 2024-01-26T23:49:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f914ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91500c93a0 0x7f91500b2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f914f7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9150067d50 0x7f9150169e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f914ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91500c93a0 0x7f91500b2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60156/0 (socket says 172.21.15.46:60156) 2024-01-26T23:49:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f914ffff700 1 -- 172.21.15.46:0/1773568421 learned_addr learned my addr 172.21.15.46:0/1773568421 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f914f7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9150067d50 0x7f9150169e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:40904/0 (socket says 172.21.15.46:40904) 2024-01-26T23:49:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f914effd700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91500681f0 0x7f9150150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f914ffff700 1 -- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91500681f0 msgr2=0x7f9150150930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f914ffff700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91500681f0 0x7f9150150930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f914ffff700 1 -- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9150067d50 msgr2=0x7f9150169e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f914ffff700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9150067d50 0x7f9150169e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f914ffff700 1 -- 172.21.15.46:0/1773568421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91400094f0 con 0x7f91500c93a0 2024-01-26T23:49:36.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f914f7fe700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9150067d50 0x7f9150169e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.968+0000 7f914ffff700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91500c93a0 0x7f91500b2ad0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f9148009ec0 tx=0x7f914800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.969+0000 7f91377fe700 1 -- 172.21.15.46:0/1773568421 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914800cb70 con 0x7f91500c93a0 2024-01-26T23:49:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.969+0000 7f91377fe700 1 -- 172.21.15.46:0/1773568421 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914800f040 con 0x7f91500c93a0 2024-01-26T23:49:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.969+0000 7f9155e35700 1 -- 172.21.15.46:0/1773568421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91500b0c00 con 0x7f91500c93a0 2024-01-26T23:49:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.969+0000 7f91377fe700 1 -- 172.21.15.46:0/1773568421 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9148014480 con 0x7f91500c93a0 2024-01-26T23:49:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.969+0000 7f9155e35700 1 -- 172.21.15.46:0/1773568421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91500b0890 con 0x7f91500c93a0 2024-01-26T23:49:36.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.970+0000 7f9155e35700 1 -- 172.21.15.46:0/1773568421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91500b0d90 con 0x7f91500c93a0 2024-01-26T23:49:36.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.970+0000 7f91377fe700 1 -- 172.21.15.46:0/1773568421 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 67) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f91480147f0 con 0x7f91500c93a0 2024-01-26T23:49:36.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.971+0000 7f91377fe700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 0x7f913c069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:36.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.971+0000 7f914f7fe700 1 -- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 msgr2=0x7f913c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:36.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.971+0000 7f91377fe700 1 -- 172.21.15.46:0/1773568421 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f9148085400 con 0x7f91500c93a0 2024-01-26T23:49:36.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.971+0000 7f914f7fe700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 0x7f913c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:36.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:36.975+0000 7f91377fe700 1 -- 172.21.15.46:0/1773568421 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f91480523a0 con 0x7f91500c93a0 2024-01-26T23:49:36.998 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:36.997+0000 7f44f12601c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:49:37.038 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:37.037+0000 7f8a485811c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:49:37.070 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:37.070+0000 7f44f12601c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:49:37.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:37.172+0000 7f914f7fe700 1 -- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 msgr2=0x7f913c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:37.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:37.172+0000 7f914f7fe700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 0x7f913c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:37.177 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:37.176+0000 7f8a485811c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:49:37.209 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:37.208+0000 7f44f12601c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:49:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:37.223+0000 7f9155e35700 1 -- 172.21.15.46:0/1773568421 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f915004c710 con 0x7f913c066f80 2024-01-26T23:49:37.260 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:37.258+0000 7f8a485811c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:49:37.335 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:37.334+0000 7f44f12601c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:49:37.434 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:37.433+0000 7f8a485811c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:49:37.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:37.573+0000 7f914f7fe700 1 -- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 msgr2=0x7f913c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:37.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:37.573+0000 7f914f7fe700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 0x7f913c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-26T23:49:37.575 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:37.574+0000 7f8a485811c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:49:37.649 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:37.648+0000 7f44f12601c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:49:37.862 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:37.861+0000 7f44f12601c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:49:37.937 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:37.936+0000 7f8a485811c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:49:38.087 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:38.086+0000 7f44f12601c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:49:38.157 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:38.157+0000 7f44f12601c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:49:38.178 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:38.177+0000 7f8a485811c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:49:38.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:38.374+0000 7f914f7fe700 1 -- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 msgr2=0x7f913c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:38.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:38.374+0000 7f914f7fe700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 0x7f913c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-26T23:49:38.439 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:38.437+0000 7f8a485811c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:49:38.519 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:38.518+0000 7f8a485811c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:49:38.759 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:38.758+0000 7f44f12601c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:49:38.834 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:38.833+0000 7f44f12601c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:49:38.957 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:38.956+0000 7f44f12601c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:49:39.212 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:39.210+0000 7f8a485811c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:49:39.297 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:39.296+0000 7f8a485811c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:49:39.438 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:39.436+0000 7f8a485811c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:49:39.457 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:39.457+0000 7f44f12601c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:49:39.607 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:39.606+0000 7f44f12601c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:49:39.758 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:39.757+0000 7f44f12601c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:49:39.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:39.975+0000 7f914f7fe700 1 -- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 msgr2=0x7f913c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:39.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:39.975+0000 7f914f7fe700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 0x7f913c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-26T23:49:40.021 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:40.019+0000 7f8a485811c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:49:40.193 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:40.192+0000 7f8a485811c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:49:40.366 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:40.365+0000 7f8a485811c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:49:40.459 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:40.458+0000 7f44f12601c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:49:40.681 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:40.681+0000 7f44f12601c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:49:40.755 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:40.754+0000 7f44f12601c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:49:40.912 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:40.912+0000 7f44f12601c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:49:40.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:40.933+0000 7f91377fe700 1 -- 172.21.15.46:0/1773568421 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 68) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f9148052770 con 0x7f91500c93a0 2024-01-26T23:49:40.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:40.933+0000 7f91377fe700 1 -- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 msgr2=0x7f913c069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:40.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:40.933+0000 7f91377fe700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 0x7f913c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:41.031 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:41.030+0000 7f446ab44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:41.031 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:41.030+0000 7f446ab44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:41.031 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:41.030+0000 7f446ab44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:41.031 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:41.030+0000 7f446ab44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:41.031 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:41.030+0000 7f446ab44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:41.031 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:41.030+0000 7f4467b3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:41.031 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:41.030+0000 7f4467b3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:41.031 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:41.030+0000 7f4467b3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:41.031 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:41.030+0000 7f4467b3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:41.031 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:41.030+0000 7f4467b3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:41.185 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:41.184+0000 7f8a485811c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:49:41.433 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:41.432+0000 7f8a485811c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:49:41.516 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:41.515+0000 7f8a485811c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:49:41.695 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:41.694+0000 7f8a485811c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:49:41.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.938+0000 7f91377fe700 1 -- 172.21.15.46:0/1773568421 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 69) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f9148071b20 con 0x7f91500c93a0 2024-01-26T23:49:41.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.939+0000 7f91377fe700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 0x7f913c069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:41.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.939+0000 7f91377fe700 1 -- 172.21.15.46:0/1773568421 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f915004c710 con 0x7f913c066f80 2024-01-26T23:49:41.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.939+0000 7f914f7fe700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 0x7f913c069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:41.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.940+0000 7f914f7fe700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 0x7f913c069430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9150064390 tx=0x7f9140017000 comp rx=0 tx=0).ready entity=mgr.8245 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:41.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.989+0000 7f91377fe700 1 -- 172.21.15.46:0/1773568421 <== mgr.8245 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7f915004c710 con 0x7f913c066f80 2024-01-26T23:49:41.991 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path snap_test_dir1/minutes 2024-01-26T23:49:41.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.992+0000 7f9155e35700 1 -- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 msgr2=0x7f913c069430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:41.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.992+0000 7f9155e35700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 0x7f913c069430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9150064390 tx=0x7f9140017000 comp rx=0 tx=0).stop 2024-01-26T23:49:41.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.992+0000 7f9155e35700 1 -- 172.21.15.46:0/1773568421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91500c93a0 msgr2=0x7f91500b2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:41.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.992+0000 7f9155e35700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91500c93a0 0x7f91500b2ad0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f9148009ec0 tx=0x7f914800b040 comp rx=0 tx=0).stop 2024-01-26T23:49:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.992+0000 7f9155e35700 1 -- 172.21.15.46:0/1773568421 shutdown_connections 2024-01-26T23:49:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.992+0000 7f9155e35700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f913c066f80 0x7f913c069430 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.992+0000 7f9155e35700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91500c93a0 0x7f91500b2ad0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.992+0000 7f9155e35700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9150067d50 0x7f9150169e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.992+0000 7f9155e35700 1 --2- 172.21.15.46:0/1773568421 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91500681f0 0x7f9150150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.992+0000 7f9155e35700 1 -- 172.21.15.46:0/1773568421 >> 172.21.15.46:0/1773568421 conn(0x7f91500ca9a0 msgr2=0x7f9150158ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.992+0000 7f9155e35700 1 -- 172.21.15.46:0/1773568421 shutdown_connections 2024-01-26T23:49:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:41.992+0000 7f9155e35700 1 -- 172.21.15.46:0/1773568421 wait complete. 2024-01-26T23:49:42.009 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/minutes 2024-01-26T23:49:42.009 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:42.009 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/hourly) 2024-01-26T23:49:42.034 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/hourly 1h 2024-01-26T23:49:42.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.215+0000 7fc42934d700 1 Processor -- start 2024-01-26T23:49:42.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.215+0000 7fc42934d700 1 -- start start 2024-01-26T23:49:42.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.216+0000 7fc42934d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc424159da0 0x7fc4240ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:42.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.216+0000 7fc42934d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc42415d8c0 0x7fc42415dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:42.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.216+0000 7fc42934d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc42415e260 con 0x7fc4241599d0 2024-01-26T23:49:42.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.216+0000 7fc42934d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4240b1a60 con 0x7fc42415d8c0 2024-01-26T23:49:42.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.216+0000 7fc42934d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42415a1d0 con 0x7fc424159da0 2024-01-26T23:49:42.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.216+0000 7fc422ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc4241599d0 0x7fc4240b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37898/0 (socket says 172.21.15.46:37898) 2024-01-26T23:49:42.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.216+0000 7fc422ffd700 1 -- 172.21.15.46:0/563574823 learned_addr learned my addr 172.21.15.46:0/563574823 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:42.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.216+0000 7fc4237fe700 1 --2- 172.21.15.46:0/563574823 >> v2:172.21.15.106:3300/0 conn(0x7fc42415d8c0 0x7fc42415dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:42.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.216+0000 7fc4227fc700 1 --2- 172.21.15.46:0/563574823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc424159da0 0x7fc4240ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:42.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.216+0000 7fc421ffb700 1 -- 172.21.15.46:0/563574823 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865754589 0 0) 0x7fc414001290 con 0x7fc4241599d0 2024-01-26T23:49:42.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.216+0000 7fc421ffb700 1 -- 172.21.15.46:0/563574823 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc40c003570 con 0x7fc4241599d0 2024-01-26T23:49:42.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc421ffb700 1 -- 172.21.15.46:0/563574823 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1681716808 0 0) 0x7fc40c003570 con 0x7fc4241599d0 2024-01-26T23:49:42.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc421ffb700 1 -- 172.21.15.46:0/563574823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc424159da0 msgr2=0x7fc4240ab1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:42.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc421ffb700 1 --2- 172.21.15.46:0/563574823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc424159da0 0x7fc4240ab1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:42.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc421ffb700 1 -- 172.21.15.46:0/563574823 >> v2:172.21.15.106:3300/0 conn(0x7fc42415d8c0 msgr2=0x7fc42415dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:42.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc421ffb700 1 --2- 172.21.15.46:0/563574823 >> v2:172.21.15.106:3300/0 conn(0x7fc42415d8c0 0x7fc42415dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:42.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc421ffb700 1 -- 172.21.15.46:0/563574823 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4240b11c0 con 0x7fc4241599d0 2024-01-26T23:49:42.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc421ffb700 1 -- 172.21.15.46:0/563574823 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fc414001780 con 0x7fc4241599d0 2024-01-26T23:49:42.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc421ffb700 1 -- 172.21.15.46:0/563574823 >> v1:172.21.15.46:6789/0 conn(0x7fc4241599d0 legacy=0x7fc4240b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:42.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc421ffb700 1 --2- 172.21.15.46:0/563574823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc40c003a60 0x7fc40c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:42.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc421ffb700 1 --2- 172.21.15.46:0/563574823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc424159da0 0x7fc40c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:42.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc421ffb700 1 --2- 172.21.15.46:0/563574823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc40c004d50 0x7fc40c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:42.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc421ffb700 1 -- 172.21.15.46:0/563574823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4240b1a60 con 0x7fc40c003a60 2024-01-26T23:49:42.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc421ffb700 1 -- 172.21.15.46:0/563574823 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42415a1d0 con 0x7fc40c004d50 2024-01-26T23:49:42.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc421ffb700 1 -- 172.21.15.46:0/563574823 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c00ba60 con 0x7fc424159da0 2024-01-26T23:49:42.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc422ffd700 1 --2- 172.21.15.46:0/563574823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc40c003a60 0x7fc40c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:42.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc4227fc700 1 --2- 172.21.15.46:0/563574823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc424159da0 0x7fc40c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:42.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc4237fe700 1 --2- 172.21.15.46:0/563574823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc40c004d50 0x7fc40c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:42.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc422ffd700 1 -- 172.21.15.46:0/563574823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc424159da0 msgr2=0x7fc40c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:42.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc422ffd700 1 --2- 172.21.15.46:0/563574823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc424159da0 0x7fc40c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:42.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc422ffd700 1 -- 172.21.15.46:0/563574823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc40c004d50 msgr2=0x7fc40c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:42.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc422ffd700 1 --2- 172.21.15.46:0/563574823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc40c004d50 0x7fc40c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:42.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.217+0000 7fc422ffd700 1 -- 172.21.15.46:0/563574823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40c003570 con 0x7fc40c003a60 2024-01-26T23:49:42.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.218+0000 7fc422ffd700 1 --2- 172.21.15.46:0/563574823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc40c003a60 0x7fc40c003e70 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fc41400c5f0 tx=0x7fc414007ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb178ac940d659e9 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:42.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.218+0000 7fc421ffb700 1 -- 172.21.15.46:0/563574823 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41400cd90 con 0x7fc40c003a60 2024-01-26T23:49:42.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.218+0000 7fc421ffb700 1 -- 172.21.15.46:0/563574823 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4140070d0 con 0x7fc40c003a60 2024-01-26T23:49:42.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.218+0000 7fc421ffb700 1 -- 172.21.15.46:0/563574823 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc414007470 con 0x7fc40c003a60 2024-01-26T23:49:42.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.218+0000 7fc42934d700 1 -- 172.21.15.46:0/563574823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc40c003a60 msgr2=0x7fc40c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:42.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.218+0000 7fc42934d700 1 --2- 172.21.15.46:0/563574823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc40c003a60 0x7fc40c003e70 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fc41400c5f0 tx=0x7fc414007ab0 comp rx=0 tx=0).stop 2024-01-26T23:49:42.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.218+0000 7fc4237fe700 1 -- 172.21.15.46:0/563574823 reap_dead start 2024-01-26T23:49:42.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.218+0000 7fc42934d700 1 -- 172.21.15.46:0/563574823 shutdown_connections 2024-01-26T23:49:42.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.218+0000 7fc42934d700 1 -- 172.21.15.46:0/563574823 >> 172.21.15.46:0/563574823 conn(0x7fc42406f840 msgr2=0x7fc424087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:42.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.218+0000 7fc42934d700 1 -- 172.21.15.46:0/563574823 shutdown_connections 2024-01-26T23:49:42.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.218+0000 7fc42934d700 1 -- 172.21.15.46:0/563574823 wait complete. 2024-01-26T23:49:42.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.220+0000 7fc42934d700 1 Processor -- start 2024-01-26T23:49:42.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.220+0000 7fc42934d700 1 -- start start 2024-01-26T23:49:42.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.220+0000 7fc42934d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc424158820 0x7fc4240c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:42.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.220+0000 7fc42934d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc424158bf0 0x7fc42416a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:42.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc42934d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4240b3a00 0x7fc4240b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:42.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc42934d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42416af30 con 0x7fc424158bf0 2024-01-26T23:49:42.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc42934d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4240b43c0 con 0x7fc424158820 2024-01-26T23:49:42.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc42934d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42416b4e0 con 0x7fc4240b3a00 2024-01-26T23:49:42.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc4227fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc424158bf0 0x7fc42416a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:42.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc4227fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc424158bf0 0x7fc42416a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59170/0 (socket says 172.21.15.46:59170) 2024-01-26T23:49:42.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc4227fc700 1 -- 172.21.15.46:0/4061008914 learned_addr learned my addr 172.21.15.46:0/4061008914 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:42.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc422ffd700 1 --2- 172.21.15.46:0/4061008914 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc424158820 0x7fc4240c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:42.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc4237fe700 1 --2- 172.21.15.46:0/4061008914 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4240b3a00 0x7fc4240b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:42.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc4227fc700 1 -- 172.21.15.46:0/4061008914 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4240b3a00 msgr2=0x7fc4240b3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:42.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc4227fc700 1 --2- 172.21.15.46:0/4061008914 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4240b3a00 0x7fc4240b3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:42.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc4227fc700 1 -- 172.21.15.46:0/4061008914 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc424158820 msgr2=0x7fc4240c5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:42.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc4227fc700 1 --2- 172.21.15.46:0/4061008914 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc424158820 0x7fc4240c5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:42.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc4227fc700 1 -- 172.21.15.46:0/4061008914 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42416b7c0 con 0x7fc424158bf0 2024-01-26T23:49:42.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc4227fc700 1 --2- 172.21.15.46:0/4061008914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc424158bf0 0x7fc42416a9f0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fc418011d70 tx=0x7fc4180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:42.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc40affd700 1 -- 172.21.15.46:0/4061008914 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41800f990 con 0x7fc424158bf0 2024-01-26T23:49:42.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc40affd700 1 -- 172.21.15.46:0/4061008914 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41800faf0 con 0x7fc424158bf0 2024-01-26T23:49:42.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc42934d700 1 -- 172.21.15.46:0/4061008914 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc42416baa0 con 0x7fc424158bf0 2024-01-26T23:49:42.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc40affd700 1 -- 172.21.15.46:0/4061008914 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41800feb0 con 0x7fc424158bf0 2024-01-26T23:49:42.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.221+0000 7fc42934d700 1 -- 172.21.15.46:0/4061008914 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc42416c060 con 0x7fc424158bf0 2024-01-26T23:49:42.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.223+0000 7fc42934d700 1 -- 172.21.15.46:0/4061008914 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc42404c710 con 0x7fc424158bf0 2024-01-26T23:49:42.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.223+0000 7fc40affd700 1 -- 172.21.15.46:0/4061008914 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 69) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fc418014650 con 0x7fc424158bf0 2024-01-26T23:49:42.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.223+0000 7fc40affd700 1 --2- 172.21.15.46:0/4061008914 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc410066dd0 0x7fc410069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:42.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.223+0000 7fc40affd700 1 -- 172.21.15.46:0/4061008914 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fc418018030 con 0x7fc424158bf0 2024-01-26T23:49:42.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.224+0000 7fc422ffd700 1 --2- 172.21.15.46:0/4061008914 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc410066dd0 0x7fc410069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:42.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.224+0000 7fc422ffd700 1 --2- 172.21.15.46:0/4061008914 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc410066dd0 0x7fc410069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc4240686a0 tx=0x7fc414015000 comp rx=0 tx=0).ready entity=mgr.8245 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:42.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.228+0000 7fc40affd700 1 -- 172.21.15.46:0/4061008914 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc418055f40 con 0x7fc424158bf0 2024-01-26T23:49:42.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.460+0000 7fc42934d700 1 -- 172.21.15.46:0/4061008914 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/hourly", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4240b14a0 con 0x7fc410066dd0 2024-01-26T23:49:42.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.518+0000 7fc40affd700 1 -- 172.21.15.46:0/4061008914 <== mgr.8245 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7fc4240b14a0 con 0x7fc410066dd0 2024-01-26T23:49:42.519 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path snap_test_dir1/hourly 2024-01-26T23:49:42.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.521+0000 7fc42934d700 1 -- 172.21.15.46:0/4061008914 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc410066dd0 msgr2=0x7fc410069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:42.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.521+0000 7fc42934d700 1 --2- 172.21.15.46:0/4061008914 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc410066dd0 0x7fc410069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc4240686a0 tx=0x7fc414015000 comp rx=0 tx=0).stop 2024-01-26T23:49:42.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.521+0000 7fc42934d700 1 -- 172.21.15.46:0/4061008914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc424158bf0 msgr2=0x7fc42416a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.521+0000 7fc42934d700 1 --2- 172.21.15.46:0/4061008914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc424158bf0 0x7fc42416a9f0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fc418011d70 tx=0x7fc4180091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.521+0000 7fc42934d700 1 -- 172.21.15.46:0/4061008914 shutdown_connections 2024-01-26T23:49:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.521+0000 7fc42934d700 1 --2- 172.21.15.46:0/4061008914 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc410066dd0 0x7fc410069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.521+0000 7fc42934d700 1 --2- 172.21.15.46:0/4061008914 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc4240b3a00 0x7fc4240b3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.521+0000 7fc42934d700 1 --2- 172.21.15.46:0/4061008914 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc424158820 0x7fc4240c5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.521+0000 7fc42934d700 1 --2- 172.21.15.46:0/4061008914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc424158bf0 0x7fc42416a9f0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.521+0000 7fc42934d700 1 -- 172.21.15.46:0/4061008914 >> 172.21.15.46:0/4061008914 conn(0x7fc42406f840 msgr2=0x7fc424087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.521+0000 7fc42934d700 1 -- 172.21.15.46:0/4061008914 shutdown_connections 2024-01-26T23:49:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.521+0000 7fc42934d700 1 -- 172.21.15.46:0/4061008914 wait complete. 2024-01-26T23:49:42.538 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/hourly 2024-01-26T23:49:42.538 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:42.538 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/daily) 2024-01-26T23:49:42.561 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/daily 1d 2024-01-26T23:49:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.742+0000 7f05adde0700 1 Processor -- start 2024-01-26T23:49:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.742+0000 7f05adde0700 1 -- start start 2024-01-26T23:49:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.743+0000 7f05adde0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f05a80ca650 0x7f05a8067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.743+0000 7f05adde0700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a806fdb0 con 0x7f05a806ebe0 2024-01-26T23:49:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.743+0000 7f05adde0700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a806f1a0 con 0x7f05a8067af0 2024-01-26T23:49:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.743+0000 7f05adde0700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05a806f480 con 0x7f05a80ca650 2024-01-26T23:49:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.743+0000 7f05a7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f05a80ca650 0x7f05a8067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.743+0000 7f05a6ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f05a806ebe0 0x7f05a81597e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37902/0 (socket says 172.21.15.46:37902) 2024-01-26T23:49:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.743+0000 7f05a6ffd700 1 -- 172.21.15.46:0/2282771126 learned_addr learned my addr 172.21.15.46:0/2282771126 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.744+0000 7f05a67fc700 1 -- 172.21.15.46:0/2282771126 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911817840 0 0) 0x7f05a806f1a0 con 0x7f05a8067af0 2024-01-26T23:49:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.744+0000 7f05a7fff700 1 -- 172.21.15.46:0/2282771126 >> v1:172.21.15.106:6789/0 conn(0x7f05a8067af0 legacy=0x7f05a80ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.744+0000 7f05a7fff700 1 -- 172.21.15.46:0/2282771126 >> v1:172.21.15.46:6789/0 conn(0x7f05a806ebe0 legacy=0x7f05a81597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.744+0000 7f05a7fff700 1 -- 172.21.15.46:0/2282771126 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a806f760 con 0x7f05a80ca650 2024-01-26T23:49:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.744+0000 7f05a7fff700 1 --2- 172.21.15.46:0/2282771126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a80ca650 0x7f05a8067ec0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009e60 tx=0x7f05a0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6630fa7e9a94d6e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.744+0000 7f05a67fc700 1 -- 172.21.15.46:0/2282771126 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a000e070 con 0x7f05a80ca650 2024-01-26T23:49:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.744+0000 7f05a67fc700 1 -- 172.21.15.46:0/2282771126 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a000bb30 con 0x7f05a80ca650 2024-01-26T23:49:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.744+0000 7f05a67fc700 1 -- 172.21.15.46:0/2282771126 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a000bed0 con 0x7f05a80ca650 2024-01-26T23:49:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.745+0000 7f05adde0700 1 -- 172.21.15.46:0/2282771126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a80ca650 msgr2=0x7f05a8067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.745+0000 7f05adde0700 1 --2- 172.21.15.46:0/2282771126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a80ca650 0x7f05a8067ec0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009e60 tx=0x7f05a0009780 comp rx=0 tx=0).stop 2024-01-26T23:49:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.745+0000 7f05adde0700 1 -- 172.21.15.46:0/2282771126 shutdown_connections 2024-01-26T23:49:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.745+0000 7f05adde0700 1 --2- 172.21.15.46:0/2282771126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a80ca650 0x7f05a8067ec0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.745+0000 7f05adde0700 1 -- 172.21.15.46:0/2282771126 >> 172.21.15.46:0/2282771126 conn(0x7f05a80cdf80 msgr2=0x7f05a806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.745+0000 7f05adde0700 1 -- 172.21.15.46:0/2282771126 shutdown_connections 2024-01-26T23:49:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.745+0000 7f05adde0700 1 -- 172.21.15.46:0/2282771126 wait complete. 2024-01-26T23:49:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.746+0000 7f05adde0700 1 Processor -- start 2024-01-26T23:49:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.746+0000 7f05adde0700 1 -- start start 2024-01-26T23:49:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.747+0000 7f05adde0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a8067af0 0x7f05a80b0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.747+0000 7f05adde0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05a806ebe0 0x7f05a80b2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.747+0000 7f05adde0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05a80ca650 0x7f05a81597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.747+0000 7f05adde0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a8159d20 con 0x7f05a8067af0 2024-01-26T23:49:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.747+0000 7f05adde0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a80b15d0 con 0x7f05a806ebe0 2024-01-26T23:49:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.747+0000 7f05adde0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a80b18b0 con 0x7f05a80ca650 2024-01-26T23:49:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.747+0000 7f05a77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a8067af0 0x7f05a80b0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.747+0000 7f05a77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a8067af0 0x7f05a80b0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59176/0 (socket says 172.21.15.46:59176) 2024-01-26T23:49:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.747+0000 7f05a77fe700 1 -- 172.21.15.46:0/1390577587 learned_addr learned my addr 172.21.15.46:0/1390577587 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.747+0000 7f05a6ffd700 1 --2- 172.21.15.46:0/1390577587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05a806ebe0 0x7f05a80b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.747+0000 7f05a7fff700 1 --2- 172.21.15.46:0/1390577587 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05a80ca650 0x7f05a81597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.747+0000 7f05a77fe700 1 -- 172.21.15.46:0/1390577587 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05a80ca650 msgr2=0x7f05a81597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.747+0000 7f05a77fe700 1 --2- 172.21.15.46:0/1390577587 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05a80ca650 0x7f05a81597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.747+0000 7f05a77fe700 1 -- 172.21.15.46:0/1390577587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05a806ebe0 msgr2=0x7f05a80b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.747+0000 7f05a77fe700 1 --2- 172.21.15.46:0/1390577587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05a806ebe0 0x7f05a80b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.747+0000 7f05a77fe700 1 -- 172.21.15.46:0/1390577587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a0009a20 con 0x7f05a8067af0 2024-01-26T23:49:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.747+0000 7f05a7fff700 1 --2- 172.21.15.46:0/1390577587 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05a80ca650 0x7f05a81597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:49:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.748+0000 7f05a77fe700 1 --2- 172.21.15.46:0/1390577587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a8067af0 0x7f05a80b0ff0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f059800c530 tx=0x7f059800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.748+0000 7f059f7fe700 1 -- 172.21.15.46:0/1390577587 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0598011590 con 0x7f05a8067af0 2024-01-26T23:49:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.748+0000 7f059f7fe700 1 -- 172.21.15.46:0/1390577587 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05980116f0 con 0x7f05a8067af0 2024-01-26T23:49:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.748+0000 7f059f7fe700 1 -- 172.21.15.46:0/1390577587 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0598011ab0 con 0x7f05a8067af0 2024-01-26T23:49:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.748+0000 7f05adde0700 1 -- 172.21.15.46:0/1390577587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a80b1b90 con 0x7f05a8067af0 2024-01-26T23:49:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.748+0000 7f05adde0700 1 -- 172.21.15.46:0/1390577587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a80b2150 con 0x7f05a8067af0 2024-01-26T23:49:42.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.749+0000 7f05adde0700 1 -- 172.21.15.46:0/1390577587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a80b2370 con 0x7f05a8067af0 2024-01-26T23:49:42.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.749+0000 7f059f7fe700 1 -- 172.21.15.46:0/1390577587 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 69) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f059801b3b0 con 0x7f05a8067af0 2024-01-26T23:49:42.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.750+0000 7f059f7fe700 1 --2- 172.21.15.46:0/1390577587 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0588066dd0 0x7f0588069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:42.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.750+0000 7f059f7fe700 1 -- 172.21.15.46:0/1390577587 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f0598083be0 con 0x7f05a8067af0 2024-01-26T23:49:42.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.750+0000 7f05a6ffd700 1 --2- 172.21.15.46:0/1390577587 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0588066dd0 0x7f0588069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:42.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.751+0000 7f05a6ffd700 1 --2- 172.21.15.46:0/1390577587 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0588066dd0 0x7f0588069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f05a8171ac0 tx=0x7f0590009040 comp rx=0 tx=0).ready entity=mgr.8245 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:42.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.755+0000 7f059f7fe700 1 -- 172.21.15.46:0/1390577587 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0598050d30 con 0x7f05a8067af0 2024-01-26T23:49:42.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:42.988+0000 7f05adde0700 1 -- 172.21.15.46:0/1390577587 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/daily", "snap_schedule": "1d", "target": ["mon-mgr", ""]}) v1 -- 0x7f05a804c710 con 0x7f0588066dd0 2024-01-26T23:49:43.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.043+0000 7f059f7fe700 1 -- 172.21.15.46:0/1390577587 <== mgr.8245 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f05a804c710 con 0x7f0588066dd0 2024-01-26T23:49:43.044 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path snap_test_dir1/daily 2024-01-26T23:49:43.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.046+0000 7f05adde0700 1 -- 172.21.15.46:0/1390577587 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0588066dd0 msgr2=0x7f0588069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:43.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.046+0000 7f05adde0700 1 --2- 172.21.15.46:0/1390577587 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0588066dd0 0x7f0588069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f05a8171ac0 tx=0x7f0590009040 comp rx=0 tx=0).stop 2024-01-26T23:49:43.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.046+0000 7f05adde0700 1 -- 172.21.15.46:0/1390577587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a8067af0 msgr2=0x7f05a80b0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:43.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.046+0000 7f05adde0700 1 --2- 172.21.15.46:0/1390577587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a8067af0 0x7f05a80b0ff0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f059800c530 tx=0x7f059800c8f0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.046+0000 7f05adde0700 1 -- 172.21.15.46:0/1390577587 shutdown_connections 2024-01-26T23:49:43.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.046+0000 7f05adde0700 1 --2- 172.21.15.46:0/1390577587 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0588066dd0 0x7f0588069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.046+0000 7f05adde0700 1 --2- 172.21.15.46:0/1390577587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a8067af0 0x7f05a80b0ff0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.046+0000 7f05adde0700 1 --2- 172.21.15.46:0/1390577587 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05a80ca650 0x7f05a81597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.046+0000 7f05adde0700 1 --2- 172.21.15.46:0/1390577587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05a806ebe0 0x7f05a80b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.046+0000 7f05adde0700 1 -- 172.21.15.46:0/1390577587 >> 172.21.15.46:0/1390577587 conn(0x7f05a80cdf80 msgr2=0x7f05a806c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:43.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.046+0000 7f05adde0700 1 -- 172.21.15.46:0/1390577587 shutdown_connections 2024-01-26T23:49:43.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.046+0000 7f05adde0700 1 -- 172.21.15.46:0/1390577587 wait complete. 2024-01-26T23:49:43.061 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/daily 2024-01-26T23:49:43.062 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:43.062 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/weekly) 2024-01-26T23:49:43.083 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/weekly 1w 2024-01-26T23:49:43.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.265+0000 7fb9355ba700 1 Processor -- start 2024-01-26T23:49:43.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.265+0000 7fb9355ba700 1 -- start start 2024-01-26T23:49:43.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.266+0000 7fb9355ba700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb93015dfd0 0x7fb9300ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:43.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.266+0000 7fb9355ba700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb93015a200 con 0x7fb9301599d0 2024-01-26T23:49:43.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.266+0000 7fb9355ba700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9300b1a60 con 0x7fb930159da0 2024-01-26T23:49:43.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.266+0000 7fb9355ba700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9300b11c0 con 0x7fb93015dfd0 2024-01-26T23:49:43.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.266+0000 7fb92effd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb9301599d0 0x7fb9300b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37918/0 (socket says 172.21.15.46:37918) 2024-01-26T23:49:43.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.266+0000 7fb92effd700 1 -- 172.21.15.46:0/2256384611 learned_addr learned my addr 172.21.15.46:0/2256384611 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:43.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.266+0000 7fb92f7fe700 1 --2- 172.21.15.46:0/2256384611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb93015dfd0 0x7fb9300ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:43.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.266+0000 7fb92dffb700 1 -- 172.21.15.46:0/2256384611 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721672754 0 0) 0x7fb93015a200 con 0x7fb9301599d0 2024-01-26T23:49:43.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.266+0000 7fb92dffb700 1 -- 172.21.15.46:0/2256384611 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb918003570 con 0x7fb9301599d0 2024-01-26T23:49:43.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.266+0000 7fb92dffb700 1 -- 172.21.15.46:0/2256384611 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135817162 0 0) 0x7fb9300b1a60 con 0x7fb930159da0 2024-01-26T23:49:43.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.266+0000 7fb92dffb700 1 -- 172.21.15.46:0/2256384611 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb93015a200 con 0x7fb930159da0 2024-01-26T23:49:43.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.266+0000 7fb92f7fe700 1 -- 172.21.15.46:0/2256384611 >> v1:172.21.15.106:6789/0 conn(0x7fb930159da0 legacy=0x7fb93015d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:43.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.266+0000 7fb92f7fe700 1 -- 172.21.15.46:0/2256384611 >> v1:172.21.15.46:6789/0 conn(0x7fb9301599d0 legacy=0x7fb9300b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:43.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.266+0000 7fb92dffb700 1 -- 172.21.15.46:0/2256384611 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1138358462 0 0) 0x7fb918003570 con 0x7fb9301599d0 2024-01-26T23:49:43.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.266+0000 7fb92f7fe700 1 -- 172.21.15.46:0/2256384611 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9300b14a0 con 0x7fb93015dfd0 2024-01-26T23:49:43.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.267+0000 7fb92f7fe700 1 --2- 172.21.15.46:0/2256384611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb93015dfd0 0x7fb9300ab1c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb928009e90 tx=0x7fb928009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=602a168ad5b50e6b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:43.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.267+0000 7fb92dffb700 1 -- 172.21.15.46:0/2256384611 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92800e070 con 0x7fb93015dfd0 2024-01-26T23:49:43.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.267+0000 7fb92dffb700 1 -- 172.21.15.46:0/2256384611 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92800bb80 con 0x7fb93015dfd0 2024-01-26T23:49:43.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.267+0000 7fb92dffb700 1 -- 172.21.15.46:0/2256384611 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb928012430 con 0x7fb93015dfd0 2024-01-26T23:49:43.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.267+0000 7fb9355ba700 1 -- 172.21.15.46:0/2256384611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb93015dfd0 msgr2=0x7fb9300ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:43.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.267+0000 7fb9355ba700 1 --2- 172.21.15.46:0/2256384611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb93015dfd0 0x7fb9300ab1c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb928009e90 tx=0x7fb928009780 comp rx=0 tx=0).stop 2024-01-26T23:49:43.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.267+0000 7fb9355ba700 1 -- 172.21.15.46:0/2256384611 shutdown_connections 2024-01-26T23:49:43.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.267+0000 7fb9355ba700 1 --2- 172.21.15.46:0/2256384611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb93015dfd0 0x7fb9300ab1c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.267+0000 7fb9355ba700 1 -- 172.21.15.46:0/2256384611 >> 172.21.15.46:0/2256384611 conn(0x7fb93006f840 msgr2=0x7fb930087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:43.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.268+0000 7fb9355ba700 1 -- 172.21.15.46:0/2256384611 shutdown_connections 2024-01-26T23:49:43.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.268+0000 7fb9355ba700 1 -- 172.21.15.46:0/2256384611 wait complete. 2024-01-26T23:49:43.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.269+0000 7fb9355ba700 1 Processor -- start 2024-01-26T23:49:43.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.269+0000 7fb9355ba700 1 -- start start 2024-01-26T23:49:43.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.270+0000 7fb9355ba700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9301599d0 0x7fb93009c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:43.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.270+0000 7fb9355ba700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb930159da0 0x7fb93009a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:43.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.270+0000 7fb9355ba700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb93015dfd0 0x7fb93009ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:43.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.270+0000 7fb9355ba700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93016bf80 con 0x7fb930159da0 2024-01-26T23:49:43.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.270+0000 7fb9355ba700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93016b370 con 0x7fb93015dfd0 2024-01-26T23:49:43.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.270+0000 7fb9355ba700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb93016b650 con 0x7fb9301599d0 2024-01-26T23:49:43.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.270+0000 7fb92e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb930159da0 0x7fb93009a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:43.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.270+0000 7fb92e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb930159da0 0x7fb93009a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59182/0 (socket says 172.21.15.46:59182) 2024-01-26T23:49:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.270+0000 7fb92e7fc700 1 -- 172.21.15.46:0/1409801299 learned_addr learned my addr 172.21.15.46:0/1409801299 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.270+0000 7fb92effd700 1 --2- 172.21.15.46:0/1409801299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9301599d0 0x7fb93009c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.270+0000 7fb92f7fe700 1 --2- 172.21.15.46:0/1409801299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb93015dfd0 0x7fb93009ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.270+0000 7fb92e7fc700 1 -- 172.21.15.46:0/1409801299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9301599d0 msgr2=0x7fb93009c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.270+0000 7fb92e7fc700 1 --2- 172.21.15.46:0/1409801299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9301599d0 0x7fb93009c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.270+0000 7fb92e7fc700 1 -- 172.21.15.46:0/1409801299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb93015dfd0 msgr2=0x7fb93009ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.270+0000 7fb92e7fc700 1 --2- 172.21.15.46:0/1409801299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb93015dfd0 0x7fb93009ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.270+0000 7fb92e7fc700 1 -- 172.21.15.46:0/1409801299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb928009a20 con 0x7fb930159da0 2024-01-26T23:49:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.270+0000 7fb92e7fc700 1 --2- 172.21.15.46:0/1409801299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb930159da0 0x7fb93009a310 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb92400c790 tx=0x7fb924007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.271+0000 7fb916ffd700 1 -- 172.21.15.46:0/1409801299 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb924010510 con 0x7fb930159da0 2024-01-26T23:49:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.271+0000 7fb916ffd700 1 -- 172.21.15.46:0/1409801299 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb924010670 con 0x7fb930159da0 2024-01-26T23:49:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.271+0000 7fb9355ba700 1 -- 172.21.15.46:0/1409801299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb93016b930 con 0x7fb930159da0 2024-01-26T23:49:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.271+0000 7fb916ffd700 1 -- 172.21.15.46:0/1409801299 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb924010a10 con 0x7fb930159da0 2024-01-26T23:49:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.271+0000 7fb9355ba700 1 -- 172.21.15.46:0/1409801299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb93009a850 con 0x7fb930159da0 2024-01-26T23:49:43.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.272+0000 7fb9355ba700 1 -- 172.21.15.46:0/1409801299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb93016c0f0 con 0x7fb930159da0 2024-01-26T23:49:43.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.272+0000 7fb916ffd700 1 -- 172.21.15.46:0/1409801299 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 69) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fb92401b3b0 con 0x7fb930159da0 2024-01-26T23:49:43.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.272+0000 7fb916ffd700 1 --2- 172.21.15.46:0/1409801299 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb91c066dd0 0x7fb91c069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:43.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.273+0000 7fb916ffd700 1 -- 172.21.15.46:0/1409801299 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fb924083eb0 con 0x7fb930159da0 2024-01-26T23:49:43.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.273+0000 7fb92effd700 1 --2- 172.21.15.46:0/1409801299 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb91c066dd0 0x7fb91c069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:43.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.273+0000 7fb92effd700 1 --2- 172.21.15.46:0/1409801299 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb91c066dd0 0x7fb91c069280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb920007fd0 tx=0x7fb920009040 comp rx=0 tx=0).ready entity=mgr.8245 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:43.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.277+0000 7fb916ffd700 1 -- 172.21.15.46:0/1409801299 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb924051000 con 0x7fb930159da0 2024-01-26T23:49:43.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.512+0000 7fb9355ba700 1 -- 172.21.15.46:0/1409801299 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/weekly", "snap_schedule": "1w", "target": ["mon-mgr", ""]}) v1 -- 0x7fb93004c710 con 0x7fb91c066dd0 2024-01-26T23:49:43.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.565+0000 7fb916ffd700 1 -- 172.21.15.46:0/1409801299 <== mgr.8245 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7fb93004c710 con 0x7fb91c066dd0 2024-01-26T23:49:43.566 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path snap_test_dir1/weekly 2024-01-26T23:49:43.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.568+0000 7fb9355ba700 1 -- 172.21.15.46:0/1409801299 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb91c066dd0 msgr2=0x7fb91c069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:43.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.568+0000 7fb9355ba700 1 --2- 172.21.15.46:0/1409801299 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb91c066dd0 0x7fb91c069280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb920007fd0 tx=0x7fb920009040 comp rx=0 tx=0).stop 2024-01-26T23:49:43.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.568+0000 7fb9355ba700 1 -- 172.21.15.46:0/1409801299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb930159da0 msgr2=0x7fb93009a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:43.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.568+0000 7fb9355ba700 1 --2- 172.21.15.46:0/1409801299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb930159da0 0x7fb93009a310 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb92400c790 tx=0x7fb924007e10 comp rx=0 tx=0).stop 2024-01-26T23:49:43.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.568+0000 7fb9355ba700 1 -- 172.21.15.46:0/1409801299 shutdown_connections 2024-01-26T23:49:43.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.568+0000 7fb9355ba700 1 --2- 172.21.15.46:0/1409801299 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb91c066dd0 0x7fb91c069280 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.568+0000 7fb9355ba700 1 --2- 172.21.15.46:0/1409801299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb93015dfd0 0x7fb93009ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.568+0000 7fb9355ba700 1 --2- 172.21.15.46:0/1409801299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9301599d0 0x7fb93009c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.568+0000 7fb9355ba700 1 --2- 172.21.15.46:0/1409801299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb930159da0 0x7fb93009a310 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.568+0000 7fb9355ba700 1 -- 172.21.15.46:0/1409801299 >> 172.21.15.46:0/1409801299 conn(0x7fb93006f840 msgr2=0x7fb9300b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:43.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.568+0000 7fb9355ba700 1 -- 172.21.15.46:0/1409801299 shutdown_connections 2024-01-26T23:49:43.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.568+0000 7fb9355ba700 1 -- 172.21.15.46:0/1409801299 wait complete. 2024-01-26T23:49:43.584 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/weekly 2024-01-26T23:49:43.584 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:43.584 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/monthly) 2024-01-26T23:49:43.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/monthly 1M 2024-01-26T23:49:43.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.787+0000 7f78e7205700 1 Processor -- start 2024-01-26T23:49:43.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.787+0000 7f78e7205700 1 -- start start 2024-01-26T23:49:43.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.788+0000 7f78e7205700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78e006ebe0 0x7f78e0067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:43.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.788+0000 7f78e7205700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f78e01597e0 0x7f78e0159c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:43.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.788+0000 7f78e7205700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78e006fdb0 con 0x7f78e0067af0 2024-01-26T23:49:43.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.788+0000 7f78e7205700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78e006f1a0 con 0x7f78e01597e0 2024-01-26T23:49:43.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.788+0000 7f78e7205700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e006f480 con 0x7f78e006ebe0 2024-01-26T23:49:43.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.788+0000 7f78e4fa1700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f78e0067af0 0x7f78e00ad8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37928/0 (socket says 172.21.15.46:37928) 2024-01-26T23:49:43.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.788+0000 7f78e4fa1700 1 -- 172.21.15.46:0/4068853507 learned_addr learned my addr 172.21.15.46:0/4068853507 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:43.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.788+0000 7f78dffff700 1 --2- 172.21.15.46:0/4068853507 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78e006ebe0 0x7f78e0067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:43.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.788+0000 7f78e57a2700 1 --2- 172.21.15.46:0/4068853507 >> v2:172.21.15.106:3300/0 conn(0x7f78e01597e0 0x7f78e0159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:43.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.788+0000 7f78df7fe700 1 -- 172.21.15.46:0/4068853507 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352256420 0 0) 0x7f78e006fdb0 con 0x7f78e0067af0 2024-01-26T23:49:43.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.788+0000 7f78df7fe700 1 -- 172.21.15.46:0/4068853507 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78d0003570 con 0x7f78e0067af0 2024-01-26T23:49:43.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.789+0000 7f78dffff700 1 -- 172.21.15.46:0/4068853507 >> v2:172.21.15.106:3300/0 conn(0x7f78e01597e0 msgr2=0x7f78e0159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:43.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.789+0000 7f78dffff700 1 --2- 172.21.15.46:0/4068853507 >> v2:172.21.15.106:3300/0 conn(0x7f78e01597e0 0x7f78e0159c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.789+0000 7f78dffff700 1 -- 172.21.15.46:0/4068853507 >> v1:172.21.15.46:6789/0 conn(0x7f78e0067af0 legacy=0x7f78e00ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:43.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.789+0000 7f78dffff700 1 -- 172.21.15.46:0/4068853507 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e006f760 con 0x7f78e006ebe0 2024-01-26T23:49:43.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.789+0000 7f78dffff700 1 --2- 172.21.15.46:0/4068853507 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78e006ebe0 0x7f78e0067ec0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f78cc0098e0 tx=0x7f78cc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9b232497d418520 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:43.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.789+0000 7f78df7fe700 1 -- 172.21.15.46:0/4068853507 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78cc011630 con 0x7f78e006ebe0 2024-01-26T23:49:43.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.789+0000 7f78df7fe700 1 -- 172.21.15.46:0/4068853507 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78cc011790 con 0x7f78e006ebe0 2024-01-26T23:49:43.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.789+0000 7f78df7fe700 1 -- 172.21.15.46:0/4068853507 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78cc011b30 con 0x7f78e006ebe0 2024-01-26T23:49:43.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.790+0000 7f78e7205700 1 -- 172.21.15.46:0/4068853507 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78e006ebe0 msgr2=0x7f78e0067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:43.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.790+0000 7f78e7205700 1 --2- 172.21.15.46:0/4068853507 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78e006ebe0 0x7f78e0067ec0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f78cc0098e0 tx=0x7f78cc0091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.790+0000 7f78e7205700 1 -- 172.21.15.46:0/4068853507 shutdown_connections 2024-01-26T23:49:43.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.790+0000 7f78e7205700 1 --2- 172.21.15.46:0/4068853507 >> v2:172.21.15.106:3300/0 conn(0x7f78e01597e0 0x7f78e0159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.790+0000 7f78e7205700 1 --2- 172.21.15.46:0/4068853507 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78e006ebe0 0x7f78e0067ec0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.790+0000 7f78e7205700 1 -- 172.21.15.46:0/4068853507 >> 172.21.15.46:0/4068853507 conn(0x7f78e00cdf80 msgr2=0x7f78e006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:43.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.790+0000 7f78e7205700 1 -- 172.21.15.46:0/4068853507 shutdown_connections 2024-01-26T23:49:43.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.790+0000 7f78e7205700 1 -- 172.21.15.46:0/4068853507 wait complete. 2024-01-26T23:49:43.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.791+0000 7f78e7205700 1 Processor -- start 2024-01-26T23:49:43.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.791+0000 7f78e7205700 1 -- start start 2024-01-26T23:49:43.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.792+0000 7f78e7205700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78e0067af0 0x7f78e016ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:43.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.792+0000 7f78e7205700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78e006ebe0 0x7f78e0152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:43.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.792+0000 7f78e7205700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78e01597e0 0x7f78e015be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:43.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.792+0000 7f78e7205700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e015cd80 con 0x7f78e0067af0 2024-01-26T23:49:43.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.792+0000 7f78e7205700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e00ade70 con 0x7f78e01597e0 2024-01-26T23:49:43.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.792+0000 7f78e7205700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e015adf0 con 0x7f78e006ebe0 2024-01-26T23:49:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.792+0000 7f78e4fa1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78e0067af0 0x7f78e016ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.792+0000 7f78e4fa1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78e0067af0 0x7f78e016ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59190/0 (socket says 172.21.15.46:59190) 2024-01-26T23:49:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.792+0000 7f78e4fa1700 1 -- 172.21.15.46:0/3312818600 learned_addr learned my addr 172.21.15.46:0/3312818600 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.792+0000 7f78dffff700 1 --2- 172.21.15.46:0/3312818600 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78e006ebe0 0x7f78e0152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.792+0000 7f78e57a2700 1 --2- 172.21.15.46:0/3312818600 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78e01597e0 0x7f78e015be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.792+0000 7f78e4fa1700 1 -- 172.21.15.46:0/3312818600 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78e006ebe0 msgr2=0x7f78e0152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.792+0000 7f78e4fa1700 1 --2- 172.21.15.46:0/3312818600 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78e006ebe0 0x7f78e0152330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.792+0000 7f78e4fa1700 1 -- 172.21.15.46:0/3312818600 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78e01597e0 msgr2=0x7f78e015be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.792+0000 7f78e4fa1700 1 --2- 172.21.15.46:0/3312818600 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78e01597e0 0x7f78e015be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.792+0000 7f78e4fa1700 1 -- 172.21.15.46:0/3312818600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78cc009420 con 0x7f78e0067af0 2024-01-26T23:49:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.793+0000 7f78e4fa1700 1 --2- 172.21.15.46:0/3312818600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78e0067af0 0x7f78e016ecf0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f78d400c5c0 tx=0x7f78d4007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.793+0000 7f78dcff9700 1 -- 172.21.15.46:0/3312818600 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d4005b30 con 0x7f78e0067af0 2024-01-26T23:49:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.793+0000 7f78dcff9700 1 -- 172.21.15.46:0/3312818600 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d4005cb0 con 0x7f78e0067af0 2024-01-26T23:49:43.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.793+0000 7f78e7205700 1 -- 172.21.15.46:0/3312818600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e015a1e0 con 0x7f78e0067af0 2024-01-26T23:49:43.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.793+0000 7f78dcff9700 1 -- 172.21.15.46:0/3312818600 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d4007320 con 0x7f78e0067af0 2024-01-26T23:49:43.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.793+0000 7f78e7205700 1 -- 172.21.15.46:0/3312818600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e015a7a0 con 0x7f78e0067af0 2024-01-26T23:49:43.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.794+0000 7f78e7205700 1 -- 172.21.15.46:0/3312818600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78e004c710 con 0x7f78e0067af0 2024-01-26T23:49:43.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.794+0000 7f78dcff9700 1 -- 172.21.15.46:0/3312818600 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 69) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f78d401b3b0 con 0x7f78e0067af0 2024-01-26T23:49:43.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.795+0000 7f78dcff9700 1 --2- 172.21.15.46:0/3312818600 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f78c4066dd0 0x7f78c4069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:43.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.795+0000 7f78dcff9700 1 -- 172.21.15.46:0/3312818600 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f78d4083a40 con 0x7f78e0067af0 2024-01-26T23:49:43.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.795+0000 7f78dffff700 1 --2- 172.21.15.46:0/3312818600 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f78c4066dd0 0x7f78c4069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:43.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.796+0000 7f78dffff700 1 --2- 172.21.15.46:0/3312818600 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f78c4066dd0 0x7f78c4069280 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f78cc0098e0 tx=0x7f78cc017000 comp rx=0 tx=0).ready entity=mgr.8245 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:43.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.799+0000 7f78dcff9700 1 -- 172.21.15.46:0/3312818600 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f78d4050b90 con 0x7f78e0067af0 2024-01-26T23:49:43.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:43.944+0000 7f78dcff9700 1 -- 172.21.15.46:0/3312818600 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 70) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f78d4050f60 con 0x7f78e0067af0 2024-01-26T23:49:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.035+0000 7f78e7205700 1 -- 172.21.15.46:0/3312818600 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/monthly", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f78e015af30 con 0x7f78c4066dd0 2024-01-26T23:49:44.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.088+0000 7f78dcff9700 1 -- 172.21.15.46:0/3312818600 <== mgr.8245 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7f78cc000c50 con 0x7f78c4066dd0 2024-01-26T23:49:44.090 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path snap_test_dir1/monthly 2024-01-26T23:49:44.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.091+0000 7f78e7205700 1 -- 172.21.15.46:0/3312818600 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f78c4066dd0 msgr2=0x7f78c4069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:44.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.091+0000 7f78e7205700 1 --2- 172.21.15.46:0/3312818600 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f78c4066dd0 0x7f78c4069280 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f78cc0098e0 tx=0x7f78cc017000 comp rx=0 tx=0).stop 2024-01-26T23:49:44.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.091+0000 7f78e7205700 1 -- 172.21.15.46:0/3312818600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78e0067af0 msgr2=0x7f78e016ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:44.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.091+0000 7f78e7205700 1 --2- 172.21.15.46:0/3312818600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78e0067af0 0x7f78e016ecf0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f78d400c5c0 tx=0x7f78d4007e00 comp rx=0 tx=0).stop 2024-01-26T23:49:44.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.091+0000 7f78e7205700 1 -- 172.21.15.46:0/3312818600 shutdown_connections 2024-01-26T23:49:44.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.091+0000 7f78e7205700 1 --2- 172.21.15.46:0/3312818600 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f78c4066dd0 0x7f78c4069280 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:44.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.091+0000 7f78e7205700 1 --2- 172.21.15.46:0/3312818600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78e0067af0 0x7f78e016ecf0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:44.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.091+0000 7f78e7205700 1 --2- 172.21.15.46:0/3312818600 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78e01597e0 0x7f78e015be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:44.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.091+0000 7f78e7205700 1 --2- 172.21.15.46:0/3312818600 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f78e006ebe0 0x7f78e0152330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:44.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.091+0000 7f78e7205700 1 -- 172.21.15.46:0/3312818600 >> 172.21.15.46:0/3312818600 conn(0x7f78e00cdf80 msgr2=0x7f78e00cc840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:44.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.091+0000 7f78e7205700 1 -- 172.21.15.46:0/3312818600 shutdown_connections 2024-01-26T23:49:44.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.091+0000 7f78e7205700 1 -- 172.21.15.46:0/3312818600 wait complete. 2024-01-26T23:49:44.107 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/monthly 2024-01-26T23:49:44.107 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:44.107 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/yearly) 2024-01-26T23:49:44.129 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/yearly 1Y 2024-01-26T23:49:44.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.310+0000 7fde39aef700 1 Processor -- start 2024-01-26T23:49:44.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.310+0000 7fde39aef700 1 -- start start 2024-01-26T23:49:44.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.311+0000 7fde39aef700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde34141200 0x7fde340c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:44.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.311+0000 7fde39aef700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde340ccc00 con 0x7fde340caf30 2024-01-26T23:49:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.311+0000 7fde39aef700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde340cc310 con 0x7fde340c8da0 2024-01-26T23:49:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.311+0000 7fde39aef700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde340cbbf0 con 0x7fde34141200 2024-01-26T23:49:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.311+0000 7fde33fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fde340caf30 0x7fde340cb370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37944/0 (socket says 172.21.15.46:37944) 2024-01-26T23:49:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.311+0000 7fde33fff700 1 -- 172.21.15.46:0/2905241439 learned_addr learned my addr 172.21.15.46:0/2905241439 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.311+0000 7fde32ffd700 1 --2- 172.21.15.46:0/2905241439 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde34141200 0x7fde340c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.311+0000 7fde327fc700 1 -- 172.21.15.46:0/2905241439 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805089073 0 0) 0x7fde340ccc00 con 0x7fde340caf30 2024-01-26T23:49:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.312+0000 7fde327fc700 1 -- 172.21.15.46:0/2905241439 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde1c003570 con 0x7fde340caf30 2024-01-26T23:49:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.312+0000 7fde327fc700 1 -- 172.21.15.46:0/2905241439 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034740332 0 0) 0x7fde340cc310 con 0x7fde340c8da0 2024-01-26T23:49:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.312+0000 7fde327fc700 1 -- 172.21.15.46:0/2905241439 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde340ccc00 con 0x7fde340c8da0 2024-01-26T23:49:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.312+0000 7fde32ffd700 1 -- 172.21.15.46:0/2905241439 >> v1:172.21.15.106:6789/0 conn(0x7fde340c8da0 legacy=0x7fde34140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.312+0000 7fde327fc700 1 -- 172.21.15.46:0/2905241439 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1384258739 0 0) 0x7fde1c003570 con 0x7fde340caf30 2024-01-26T23:49:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.312+0000 7fde32ffd700 1 -- 172.21.15.46:0/2905241439 >> v1:172.21.15.46:6789/0 conn(0x7fde340caf30 legacy=0x7fde340cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.312+0000 7fde32ffd700 1 -- 172.21.15.46:0/2905241439 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde340cc5b0 con 0x7fde34141200 2024-01-26T23:49:44.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.312+0000 7fde32ffd700 1 --2- 172.21.15.46:0/2905241439 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde34141200 0x7fde340c9170 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fde28009910 tx=0x7fde280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e72aa03922533abb server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:44.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.312+0000 7fde327fc700 1 -- 172.21.15.46:0/2905241439 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2c0017f0 con 0x7fde34141200 2024-01-26T23:49:44.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.312+0000 7fde327fc700 1 -- 172.21.15.46:0/2905241439 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde28011420 con 0x7fde34141200 2024-01-26T23:49:44.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.312+0000 7fde327fc700 1 -- 172.21.15.46:0/2905241439 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde280117c0 con 0x7fde34141200 2024-01-26T23:49:44.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.313+0000 7fde39aef700 1 -- 172.21.15.46:0/2905241439 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde34141200 msgr2=0x7fde340c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:44.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.313+0000 7fde39aef700 1 --2- 172.21.15.46:0/2905241439 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde34141200 0x7fde340c9170 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fde28009910 tx=0x7fde280091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:44.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.313+0000 7fde39aef700 1 -- 172.21.15.46:0/2905241439 shutdown_connections 2024-01-26T23:49:44.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.313+0000 7fde39aef700 1 --2- 172.21.15.46:0/2905241439 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde34141200 0x7fde340c9170 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:44.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.313+0000 7fde39aef700 1 -- 172.21.15.46:0/2905241439 >> 172.21.15.46:0/2905241439 conn(0x7fde3406bd90 msgr2=0x7fde3406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:44.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.313+0000 7fde39aef700 1 -- 172.21.15.46:0/2905241439 shutdown_connections 2024-01-26T23:49:44.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.313+0000 7fde39aef700 1 -- 172.21.15.46:0/2905241439 wait complete. 2024-01-26T23:49:44.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.314+0000 7fde39aef700 1 Processor -- start 2024-01-26T23:49:44.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.314+0000 7fde39aef700 1 -- start start 2024-01-26T23:49:44.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.315+0000 7fde39aef700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde340c8da0 0x7fde340c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:44.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.315+0000 7fde39aef700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde340caf30 0x7fde34069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:44.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.315+0000 7fde39aef700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde34141200 0x7fde340b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:44.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.315+0000 7fde39aef700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde34069990 con 0x7fde340caf30 2024-01-26T23:49:44.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.315+0000 7fde39aef700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3416e460 con 0x7fde340c8da0 2024-01-26T23:49:44.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.315+0000 7fde39aef700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde3416d570 con 0x7fde34141200 2024-01-26T23:49:44.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.315+0000 7fde32ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde340caf30 0x7fde34069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:44.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.315+0000 7fde32ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde340caf30 0x7fde34069450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59192/0 (socket says 172.21.15.46:59192) 2024-01-26T23:49:44.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.315+0000 7fde32ffd700 1 -- 172.21.15.46:0/1874932395 learned_addr learned my addr 172.21.15.46:0/1874932395 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:44.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.315+0000 7fde337fe700 1 --2- 172.21.15.46:0/1874932395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde340c8da0 0x7fde340c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:44.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.315+0000 7fde33fff700 1 --2- 172.21.15.46:0/1874932395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde34141200 0x7fde340b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:44.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.315+0000 7fde32ffd700 1 -- 172.21.15.46:0/1874932395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde34141200 msgr2=0x7fde340b36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:44.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.315+0000 7fde32ffd700 1 --2- 172.21.15.46:0/1874932395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde34141200 0x7fde340b36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:44.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.315+0000 7fde32ffd700 1 -- 172.21.15.46:0/1874932395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde340c8da0 msgr2=0x7fde340c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:44.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.315+0000 7fde32ffd700 1 --2- 172.21.15.46:0/1874932395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde340c8da0 0x7fde340c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:44.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.316+0000 7fde32ffd700 1 -- 172.21.15.46:0/1874932395 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde28009420 con 0x7fde340caf30 2024-01-26T23:49:44.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.316+0000 7fde32ffd700 1 --2- 172.21.15.46:0/1874932395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde340caf30 0x7fde34069450 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fde3406f5a0 tx=0x7fde28016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:44.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.316+0000 7fde1b7fe700 1 -- 172.21.15.46:0/1874932395 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde28000d10 con 0x7fde340caf30 2024-01-26T23:49:44.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.316+0000 7fde1b7fe700 1 -- 172.21.15.46:0/1874932395 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde2801caf0 con 0x7fde340caf30 2024-01-26T23:49:44.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.316+0000 7fde39aef700 1 -- 172.21.15.46:0/1874932395 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde3416d850 con 0x7fde340caf30 2024-01-26T23:49:44.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.316+0000 7fde1b7fe700 1 -- 172.21.15.46:0/1874932395 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2801a450 con 0x7fde340caf30 2024-01-26T23:49:44.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.316+0000 7fde39aef700 1 -- 172.21.15.46:0/1874932395 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde3416de10 con 0x7fde340caf30 2024-01-26T23:49:44.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.317+0000 7fde39aef700 1 -- 172.21.15.46:0/1874932395 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde3404c710 con 0x7fde340caf30 2024-01-26T23:49:44.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.317+0000 7fde1b7fe700 1 -- 172.21.15.46:0/1874932395 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 70) v1 ==== 84862+0+0 (secure 0 0 0) 0x7fde2801a880 con 0x7fde340caf30 2024-01-26T23:49:44.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.318+0000 7fde1b7fe700 1 --2- 172.21.15.46:0/1874932395 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fde200670a0 0x7fde20069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:44.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.318+0000 7fde1b7fe700 1 -- 172.21.15.46:0/1874932395 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fde28086290 con 0x7fde340caf30 2024-01-26T23:49:44.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.318+0000 7fde337fe700 1 --2- 172.21.15.46:0/1874932395 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fde200670a0 0x7fde20069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:44.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.319+0000 7fde337fe700 1 --2- 172.21.15.46:0/1874932395 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fde200670a0 0x7fde20069550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fde341582b0 tx=0x7fde24007d50 comp rx=0 tx=0).ready entity=mgr.8245 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:44.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.323+0000 7fde1b7fe700 1 -- 172.21.15.46:0/1874932395 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fde280530a0 con 0x7fde340caf30 2024-01-26T23:49:44.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.555+0000 7fde39aef700 1 -- 172.21.15.46:0/1874932395 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/yearly", "snap_schedule": "1Y", "target": ["mon-mgr", ""]}) v1 -- 0x7fde340cc890 con 0x7fde200670a0 2024-01-26T23:49:44.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.611+0000 7fde1b7fe700 1 -- 172.21.15.46:0/1874932395 <== mgr.8245 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7fde340cc890 con 0x7fde200670a0 2024-01-26T23:49:44.613 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path snap_test_dir1/yearly 2024-01-26T23:49:44.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.614+0000 7fde39aef700 1 -- 172.21.15.46:0/1874932395 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fde200670a0 msgr2=0x7fde20069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:44.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.614+0000 7fde39aef700 1 --2- 172.21.15.46:0/1874932395 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fde200670a0 0x7fde20069550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fde341582b0 tx=0x7fde24007d50 comp rx=0 tx=0).stop 2024-01-26T23:49:44.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.614+0000 7fde39aef700 1 -- 172.21.15.46:0/1874932395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde340caf30 msgr2=0x7fde34069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:44.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.614+0000 7fde39aef700 1 --2- 172.21.15.46:0/1874932395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde340caf30 0x7fde34069450 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fde3406f5a0 tx=0x7fde28016000 comp rx=0 tx=0).stop 2024-01-26T23:49:44.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.614+0000 7fde39aef700 1 -- 172.21.15.46:0/1874932395 shutdown_connections 2024-01-26T23:49:44.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.614+0000 7fde39aef700 1 --2- 172.21.15.46:0/1874932395 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fde200670a0 0x7fde20069550 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:44.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.614+0000 7fde39aef700 1 --2- 172.21.15.46:0/1874932395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde34141200 0x7fde340b36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:44.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.614+0000 7fde39aef700 1 --2- 172.21.15.46:0/1874932395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde340c8da0 0x7fde340c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:44.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.614+0000 7fde39aef700 1 --2- 172.21.15.46:0/1874932395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde340caf30 0x7fde34069450 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:44.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.614+0000 7fde39aef700 1 -- 172.21.15.46:0/1874932395 >> 172.21.15.46:0/1874932395 conn(0x7fde3406bd90 msgr2=0x7fde3406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:44.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.614+0000 7fde39aef700 1 -- 172.21.15.46:0/1874932395 shutdown_connections 2024-01-26T23:49:44.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.614+0000 7fde39aef700 1 -- 172.21.15.46:0/1874932395 wait complete. 2024-01-26T23:49:44.630 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/yearly 2024-01-26T23:49:44.630 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:44.630 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/bad_period_spec) 2024-01-26T23:49:44.652 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1X 2024-01-26T23:49:44.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.832+0000 7f2b2471e700 1 Processor -- start 2024-01-26T23:49:44.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.832+0000 7f2b2471e700 1 -- start start 2024-01-26T23:49:44.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.833+0000 7f2b2471e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2b1c0c90c0 0x7f2b1c0af990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:44.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.833+0000 7f2b2471e700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1c0c87c0 con 0x7f2b1c0c8c50 2024-01-26T23:49:44.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.833+0000 7f2b2471e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1c0c7bb0 con 0x7f2b1c068940 2024-01-26T23:49:44.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.833+0000 7f2b2471e700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1c0c7e90 con 0x7f2b1c0c90c0 2024-01-26T23:49:44.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.834+0000 7f2b21cb9700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2b1c0c8c50 0x7f2b1c0c7160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37956/0 (socket says 172.21.15.46:37956) 2024-01-26T23:49:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.834+0000 7f2b21cb9700 1 -- 172.21.15.46:0/965744054 learned_addr learned my addr 172.21.15.46:0/965744054 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.834+0000 7f2b22cbb700 1 --2- 172.21.15.46:0/965744054 >> v2:172.21.15.106:3300/0 conn(0x7f2b1c0c90c0 0x7f2b1c0af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.834+0000 7f2b214b8700 1 -- 172.21.15.46:0/965744054 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748784534 0 0) 0x7f2b1c0c87c0 con 0x7f2b1c0c8c50 2024-01-26T23:49:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.834+0000 7f2b214b8700 1 -- 172.21.15.46:0/965744054 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b08003570 con 0x7f2b1c0c8c50 2024-01-26T23:49:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.834+0000 7f2b214b8700 1 -- 172.21.15.46:0/965744054 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956811715 0 0) 0x7f2b1c0c7bb0 con 0x7f2b1c068940 2024-01-26T23:49:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.834+0000 7f2b214b8700 1 -- 172.21.15.46:0/965744054 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b1c0c87c0 con 0x7f2b1c068940 2024-01-26T23:49:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.834+0000 7f2b22cbb700 1 -- 172.21.15.46:0/965744054 >> v1:172.21.15.106:6789/0 conn(0x7f2b1c068940 legacy=0x7f2b1c068d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.834+0000 7f2b22cbb700 1 -- 172.21.15.46:0/965744054 >> v1:172.21.15.46:6789/0 conn(0x7f2b1c0c8c50 legacy=0x7f2b1c0c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.834+0000 7f2b22cbb700 1 -- 172.21.15.46:0/965744054 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1c0c8170 con 0x7f2b1c0c90c0 2024-01-26T23:49:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.834+0000 7f2b22cbb700 1 --2- 172.21.15.46:0/965744054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b1c0c90c0 0x7f2b1c0af990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009e60 tx=0x7f2b14009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dd262e3476d6db1 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.834+0000 7f2b214b8700 1 -- 172.21.15.46:0/965744054 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1400e070 con 0x7f2b1c0c90c0 2024-01-26T23:49:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.835+0000 7f2b214b8700 1 -- 172.21.15.46:0/965744054 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1400bab0 con 0x7f2b1c0c90c0 2024-01-26T23:49:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.835+0000 7f2b214b8700 1 -- 172.21.15.46:0/965744054 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1400be70 con 0x7f2b1c0c90c0 2024-01-26T23:49:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.835+0000 7f2b2471e700 1 -- 172.21.15.46:0/965744054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b1c0c90c0 msgr2=0x7f2b1c0af990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.835+0000 7f2b2471e700 1 --2- 172.21.15.46:0/965744054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b1c0c90c0 0x7f2b1c0af990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009e60 tx=0x7f2b14009780 comp rx=0 tx=0).stop 2024-01-26T23:49:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.835+0000 7f2b2471e700 1 -- 172.21.15.46:0/965744054 shutdown_connections 2024-01-26T23:49:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.835+0000 7f2b2471e700 1 --2- 172.21.15.46:0/965744054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b1c0c90c0 0x7f2b1c0af990 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.835+0000 7f2b2471e700 1 -- 172.21.15.46:0/965744054 >> 172.21.15.46:0/965744054 conn(0x7f2b1c0ca100 msgr2=0x7f2b1c06e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.835+0000 7f2b2471e700 1 -- 172.21.15.46:0/965744054 shutdown_connections 2024-01-26T23:49:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.835+0000 7f2b2471e700 1 -- 172.21.15.46:0/965744054 wait complete. 2024-01-26T23:49:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.836+0000 7f2b2471e700 1 Processor -- start 2024-01-26T23:49:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.836+0000 7f2b2471e700 1 -- start start 2024-01-26T23:49:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.837+0000 7f2b2471e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b1c068940 0x7f2b1c14f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.837+0000 7f2b2471e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b1c0c8c50 0x7f2b1c0c2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.837+0000 7f2b2471e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b1c0c90c0 0x7f2b1c0c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.837+0000 7f2b2471e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c0c3040 con 0x7f2b1c0c90c0 2024-01-26T23:49:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.837+0000 7f2b2471e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c06b0c0 con 0x7f2b1c068940 2024-01-26T23:49:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.837+0000 7f2b2471e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c0c2d60 con 0x7f2b1c0c8c50 2024-01-26T23:49:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.837+0000 7f2b22cbb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b1c0c90c0 0x7f2b1c0c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.837+0000 7f2b22cbb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b1c0c90c0 0x7f2b1c0c4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59206/0 (socket says 172.21.15.46:59206) 2024-01-26T23:49:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.837+0000 7f2b22cbb700 1 -- 172.21.15.46:0/3662104319 learned_addr learned my addr 172.21.15.46:0/3662104319 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.838+0000 7f2b21cb9700 1 --2- 172.21.15.46:0/3662104319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b1c0c8c50 0x7f2b1c0c2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.838+0000 7f2b224ba700 1 --2- 172.21.15.46:0/3662104319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b1c068940 0x7f2b1c14f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.838+0000 7f2b22cbb700 1 -- 172.21.15.46:0/3662104319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b1c0c8c50 msgr2=0x7f2b1c0c2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.838+0000 7f2b22cbb700 1 --2- 172.21.15.46:0/3662104319 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b1c0c8c50 0x7f2b1c0c2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.838+0000 7f2b22cbb700 1 -- 172.21.15.46:0/3662104319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b1c068940 msgr2=0x7f2b1c14f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.838+0000 7f2b22cbb700 1 --2- 172.21.15.46:0/3662104319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b1c068940 0x7f2b1c14f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.838+0000 7f2b22cbb700 1 -- 172.21.15.46:0/3662104319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b14009a20 con 0x7f2b1c0c90c0 2024-01-26T23:49:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.838+0000 7f2b22cbb700 1 --2- 172.21.15.46:0/3662104319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b1c0c90c0 0x7f2b1c0c4880 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f2b1400b610 tx=0x7f2b14019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.838+0000 7f2b127fc700 1 -- 172.21.15.46:0/3662104319 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b14012de0 con 0x7f2b1c0c90c0 2024-01-26T23:49:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.838+0000 7f2b127fc700 1 -- 172.21.15.46:0/3662104319 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b14026480 con 0x7f2b1c0c90c0 2024-01-26T23:49:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.838+0000 7f2b2471e700 1 -- 172.21.15.46:0/3662104319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1c06aa70 con 0x7f2b1c0c90c0 2024-01-26T23:49:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.838+0000 7f2b127fc700 1 -- 172.21.15.46:0/3662104319 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b14026840 con 0x7f2b1c0c90c0 2024-01-26T23:49:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.838+0000 7f2b2471e700 1 -- 172.21.15.46:0/3662104319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1c14f550 con 0x7f2b1c0c90c0 2024-01-26T23:49:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.839+0000 7f2b2471e700 1 -- 172.21.15.46:0/3662104319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1c04c710 con 0x7f2b1c0c90c0 2024-01-26T23:49:44.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.840+0000 7f2b127fc700 1 -- 172.21.15.46:0/3662104319 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 70) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f2b1401c400 con 0x7f2b1c0c90c0 2024-01-26T23:49:44.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.840+0000 7f2b127fc700 1 --2- 172.21.15.46:0/3662104319 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2b000670a0 0x7f2b00069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:44.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.840+0000 7f2b127fc700 1 -- 172.21.15.46:0/3662104319 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f2b14086a00 con 0x7f2b1c0c90c0 2024-01-26T23:49:44.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.840+0000 7f2b224ba700 1 --2- 172.21.15.46:0/3662104319 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2b000670a0 0x7f2b00069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:44.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.841+0000 7f2b224ba700 1 --2- 172.21.15.46:0/3662104319 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2b000670a0 0x7f2b00069550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2b18003a40 tx=0x7f2b18007d40 comp rx=0 tx=0).ready entity=mgr.8245 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:44.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:44.845+0000 7f2b127fc700 1 -- 172.21.15.46:0/3662104319 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2b14053680 con 0x7f2b1c0c90c0 2024-01-26T23:49:45.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.081+0000 7f2b2471e700 1 -- 172.21.15.46:0/3662104319 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1X", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b1c06b200 con 0x7f2b000670a0 2024-01-26T23:49:45.083 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:45.084+0000 7f4471b52700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - multiplier should be one of h,d,w,M,Y 2024-01-26T23:49:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.083+0000 7f2b127fc700 1 -- 172.21.15.46:0/3662104319 <== mgr.8245 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - multiplier should be one of h,d,w,M,Y) v1 ==== 74+0+0 (secure 0 0 0) 0x7f2b1c06b200 con 0x7f2b000670a0 2024-01-26T23:49:45.086 INFO:teuthology.orchestra.run.smithi046.stderr:Error ENOENT: invalid schedule specified - multiplier should be one of h,d,w,M,Y 2024-01-26T23:49:45.089 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-26T23:49:45.090 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1MM 2024-01-26T23:49:45.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.224+0000 7f199fe2c700 1 Processor -- start 2024-01-26T23:49:45.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.224+0000 7f199fe2c700 1 -- start start 2024-01-26T23:49:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.225+0000 7f199fe2c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f19980c8da0 0x7f19980c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.225+0000 7f199fe2c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1998140af0 0x7f1998140f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.225+0000 7f199fe2c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19980caf30 0x7f19980cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.225+0000 7f199fe2c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19980cbbf0 con 0x7f1998140af0 2024-01-26T23:49:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.225+0000 7f199fe2c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19980ccc00 con 0x7f19980c8da0 2024-01-26T23:49:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.225+0000 7f199fe2c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19980cc310 con 0x7f19980caf30 2024-01-26T23:49:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.225+0000 7f199d3c7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1998140af0 0x7f1998140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.225+0000 7f199d3c7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1998140af0 0x7f1998140f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59218/0 (socket says 172.21.15.46:59218) 2024-01-26T23:49:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.225+0000 7f199d3c7700 1 -- 172.21.15.46:0/3024252193 learned_addr learned my addr 172.21.15.46:0/3024252193 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.225+0000 7f199e3c9700 1 --2- 172.21.15.46:0/3024252193 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19980caf30 0x7f19980cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.225+0000 7f199dbc8700 1 --2- 172.21.15.46:0/3024252193 >> v2:172.21.15.106:3300/0 conn(0x7f19980c8da0 0x7f19980c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.226+0000 7f199d3c7700 1 -- 172.21.15.46:0/3024252193 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19980caf30 msgr2=0x7f19980cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.226+0000 7f199d3c7700 1 --2- 172.21.15.46:0/3024252193 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19980caf30 0x7f19980cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.226+0000 7f199d3c7700 1 -- 172.21.15.46:0/3024252193 >> v2:172.21.15.106:3300/0 conn(0x7f19980c8da0 msgr2=0x7f19980c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.226+0000 7f199d3c7700 1 --2- 172.21.15.46:0/3024252193 >> v2:172.21.15.106:3300/0 conn(0x7f19980c8da0 0x7f19980c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.226+0000 7f199d3c7700 1 -- 172.21.15.46:0/3024252193 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19981414d0 con 0x7f1998140af0 2024-01-26T23:49:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.226+0000 7f199e3c9700 1 --2- 172.21.15.46:0/3024252193 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19980caf30 0x7f19980cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.226+0000 7f199d3c7700 1 --2- 172.21.15.46:0/3024252193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1998140af0 0x7f1998140f90 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f199806f100 tx=0x7f198c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a776b96a29358f9b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.226+0000 7f199cbc6700 1 -- 172.21.15.46:0/3024252193 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c00b860 con 0x7f1998140af0 2024-01-26T23:49:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.226+0000 7f199cbc6700 1 -- 172.21.15.46:0/3024252193 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198c00b9c0 con 0x7f1998140af0 2024-01-26T23:49:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.226+0000 7f199cbc6700 1 -- 172.21.15.46:0/3024252193 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c00bd60 con 0x7f1998140af0 2024-01-26T23:49:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.227+0000 7f199fe2c700 1 -- 172.21.15.46:0/3024252193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1998140af0 msgr2=0x7f1998140f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.227+0000 7f199fe2c700 1 --2- 172.21.15.46:0/3024252193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1998140af0 0x7f1998140f90 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f199806f100 tx=0x7f198c009250 comp rx=0 tx=0).stop 2024-01-26T23:49:45.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.227+0000 7f199fe2c700 1 -- 172.21.15.46:0/3024252193 shutdown_connections 2024-01-26T23:49:45.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.227+0000 7f199fe2c700 1 --2- 172.21.15.46:0/3024252193 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19980caf30 0x7f19980cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:45.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.227+0000 7f199fe2c700 1 --2- 172.21.15.46:0/3024252193 >> v2:172.21.15.106:3300/0 conn(0x7f19980c8da0 0x7f19980c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:45.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.227+0000 7f199fe2c700 1 --2- 172.21.15.46:0/3024252193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1998140af0 0x7f1998140f90 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:45.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.227+0000 7f199fe2c700 1 -- 172.21.15.46:0/3024252193 >> 172.21.15.46:0/3024252193 conn(0x7f199806bd90 msgr2=0x7f199806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:45.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.227+0000 7f199fe2c700 1 -- 172.21.15.46:0/3024252193 shutdown_connections 2024-01-26T23:49:45.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.227+0000 7f199fe2c700 1 -- 172.21.15.46:0/3024252193 wait complete. 2024-01-26T23:49:45.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.228+0000 7f199fe2c700 1 Processor -- start 2024-01-26T23:49:45.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.228+0000 7f199fe2c700 1 -- start start 2024-01-26T23:49:45.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.229+0000 7f199fe2c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19980c8da0 0x7f199816a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:45.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.229+0000 7f199fe2c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19980caf30 0x7f199813e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:45.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.229+0000 7f199fe2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1998140af0 0x7f199813ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:45.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.229+0000 7f199fe2c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1998159710 con 0x7f1998140af0 2024-01-26T23:49:45.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.229+0000 7f199fe2c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1998158a90 con 0x7f19980c8da0 2024-01-26T23:49:45.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.229+0000 7f199fe2c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1998158d70 con 0x7f19980caf30 2024-01-26T23:49:45.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.229+0000 7f199e3c9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1998140af0 0x7f199813ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:45.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.229+0000 7f199e3c9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1998140af0 0x7f199813ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59226/0 (socket says 172.21.15.46:59226) 2024-01-26T23:49:45.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.229+0000 7f199e3c9700 1 -- 172.21.15.46:0/2961685476 learned_addr learned my addr 172.21.15.46:0/2961685476 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:45.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.229+0000 7f199d3c7700 1 --2- 172.21.15.46:0/2961685476 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19980caf30 0x7f199813e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:45.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.229+0000 7f199dbc8700 1 --2- 172.21.15.46:0/2961685476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19980c8da0 0x7f199816a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:45.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.229+0000 7f199e3c9700 1 -- 172.21.15.46:0/2961685476 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19980caf30 msgr2=0x7f199813e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:45.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.230+0000 7f199e3c9700 1 --2- 172.21.15.46:0/2961685476 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19980caf30 0x7f199813e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:45.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.230+0000 7f199e3c9700 1 -- 172.21.15.46:0/2961685476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19980c8da0 msgr2=0x7f199816a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:45.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.230+0000 7f199e3c9700 1 --2- 172.21.15.46:0/2961685476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19980c8da0 0x7f199816a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:45.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.230+0000 7f199e3c9700 1 -- 172.21.15.46:0/2961685476 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198c0094f0 con 0x7f1998140af0 2024-01-26T23:49:45.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.230+0000 7f199e3c9700 1 --2- 172.21.15.46:0/2961685476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1998140af0 0x7f199813ee70 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f199000d180 tx=0x7f1990009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:45.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.230+0000 7f1989ffb700 1 -- 172.21.15.46:0/2961685476 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199000dcc0 con 0x7f1998140af0 2024-01-26T23:49:45.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.230+0000 7f1989ffb700 1 -- 172.21.15.46:0/2961685476 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199000f040 con 0x7f1998140af0 2024-01-26T23:49:45.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.230+0000 7f1989ffb700 1 -- 172.21.15.46:0/2961685476 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1990015690 con 0x7f1998140af0 2024-01-26T23:49:45.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.230+0000 7f199fe2c700 1 -- 172.21.15.46:0/2961685476 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1998159050 con 0x7f1998140af0 2024-01-26T23:49:45.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.230+0000 7f199fe2c700 1 -- 172.21.15.46:0/2961685476 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f199816aac0 con 0x7f1998140af0 2024-01-26T23:49:45.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.231+0000 7f199fe2c700 1 -- 172.21.15.46:0/2961685476 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f199804c710 con 0x7f1998140af0 2024-01-26T23:49:45.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.232+0000 7f1989ffb700 1 -- 172.21.15.46:0/2961685476 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 70) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f199001e3b0 con 0x7f1998140af0 2024-01-26T23:49:45.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.232+0000 7f1989ffb700 1 --2- 172.21.15.46:0/2961685476 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f197c0670a0 0x7f197c069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:45.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.232+0000 7f1989ffb700 1 -- 172.21.15.46:0/2961685476 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f19900850a0 con 0x7f1998140af0 2024-01-26T23:49:45.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.232+0000 7f199dbc8700 1 --2- 172.21.15.46:0/2961685476 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f197c0670a0 0x7f197c069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:45.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.233+0000 7f199dbc8700 1 --2- 172.21.15.46:0/2961685476 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f197c0670a0 0x7f197c069550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1994006fd0 tx=0x7f1994008040 comp rx=0 tx=0).ready entity=mgr.8245 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:45.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.236+0000 7f1989ffb700 1 -- 172.21.15.46:0/2961685476 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1990051d20 con 0x7f1998140af0 2024-01-26T23:49:45.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.467+0000 7f199fe2c700 1 -- 172.21.15.46:0/2961685476 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1MM", "target": ["mon-mgr", ""]}) v1 -- 0x7f19980cc5b0 con 0x7f197c0670a0 2024-01-26T23:49:45.469 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:45.469+0000 7f4471b52700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-01-26T23:49:45.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.469+0000 7f1989ffb700 1 -- 172.21.15.46:0/2961685476 <== mgr.8245 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f19980cc5b0 con 0x7f197c0670a0 2024-01-26T23:49:45.471 INFO:teuthology.orchestra.run.smithi046.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-01-26T23:49:45.475 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-26T23:49:45.475 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1 2024-01-26T23:49:45.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.609+0000 7f171480c700 1 Processor -- start 2024-01-26T23:49:45.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.609+0000 7f171480c700 1 -- start start 2024-01-26T23:49:45.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.610+0000 7f171480c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1710067ec0 0x7f17100c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:45.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.610+0000 7f171480c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17100c7fe0 0x7f1710157200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:45.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.610+0000 7f171480c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171006c160 con 0x7f1710067af0 2024-01-26T23:49:45.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.610+0000 7f171480c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1710068330 con 0x7f1710067ec0 2024-01-26T23:49:45.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.610+0000 7f171480c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171006ca00 con 0x7f17100c7fe0 2024-01-26T23:49:45.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.610+0000 7f170dd9b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1710067ec0 0x7f17100c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:45.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.610+0000 7f170ed9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17100c7fe0 0x7f1710157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.610+0000 7f170ed9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17100c7fe0 0x7f1710157200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:47988/0 (socket says 172.21.15.46:47988) 2024-01-26T23:49:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.610+0000 7f170dd9b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1710067ec0 0x7f17100c7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:57836/0 (socket says 172.21.15.46:57836) 2024-01-26T23:49:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.610+0000 7f170ed9d700 1 -- 172.21.15.46:0/2187591697 learned_addr learned my addr 172.21.15.46:0/2187591697 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.610+0000 7f170e59c700 1 --1- 172.21.15.46:0/2187591697 >> v1:172.21.15.106:6789/0 conn(0x7f1710067af0 0x7f171006ce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:44324/0 (socket says 172.21.15.46:44324) 2024-01-26T23:49:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.610+0000 7f170ed9d700 1 -- 172.21.15.46:0/2187591697 >> v2:172.21.15.106:3300/0 conn(0x7f1710067ec0 msgr2=0x7f17100c7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.611+0000 7f170ed9d700 1 --2- 172.21.15.46:0/2187591697 >> v2:172.21.15.106:3300/0 conn(0x7f1710067ec0 0x7f17100c7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.611+0000 7f170ed9d700 1 -- 172.21.15.46:0/2187591697 >> v1:172.21.15.106:6789/0 conn(0x7f1710067af0 legacy=0x7f171006ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.611+0000 7f170ed9d700 1 -- 172.21.15.46:0/2187591697 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17100c84f0 con 0x7f17100c7fe0 2024-01-26T23:49:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.611+0000 7f170ed9d700 1 --2- 172.21.15.46:0/2187591697 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17100c7fe0 0x7f1710157200 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1708009f20 tx=0x7f1708009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f22401cc6b62371 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.611+0000 7f170d59a700 1 -- 172.21.15.46:0/2187591697 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170800e070 con 0x7f17100c7fe0 2024-01-26T23:49:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.611+0000 7f170d59a700 1 -- 172.21.15.46:0/2187591697 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170800b9d0 con 0x7f17100c7fe0 2024-01-26T23:49:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.611+0000 7f170d59a700 1 -- 172.21.15.46:0/2187591697 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17080092e0 con 0x7f17100c7fe0 2024-01-26T23:49:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.611+0000 7f171480c700 1 -- 172.21.15.46:0/2187591697 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17100c7fe0 msgr2=0x7f1710157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.611+0000 7f171480c700 1 --2- 172.21.15.46:0/2187591697 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17100c7fe0 0x7f1710157200 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1708009f20 tx=0x7f1708009840 comp rx=0 tx=0).stop 2024-01-26T23:49:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.612+0000 7f171480c700 1 -- 172.21.15.46:0/2187591697 shutdown_connections 2024-01-26T23:49:45.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.612+0000 7f171480c700 1 --2- 172.21.15.46:0/2187591697 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17100c7fe0 0x7f1710157200 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:45.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.612+0000 7f171480c700 1 --2- 172.21.15.46:0/2187591697 >> v2:172.21.15.106:3300/0 conn(0x7f1710067ec0 0x7f17100c7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:45.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.612+0000 7f171480c700 1 -- 172.21.15.46:0/2187591697 >> 172.21.15.46:0/2187591697 conn(0x7f17100ca9a0 msgr2=0x7f1710068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:45.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.612+0000 7f171480c700 1 -- 172.21.15.46:0/2187591697 shutdown_connections 2024-01-26T23:49:45.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.612+0000 7f171480c700 1 -- 172.21.15.46:0/2187591697 wait complete. 2024-01-26T23:49:45.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.613+0000 7f171480c700 1 Processor -- start 2024-01-26T23:49:45.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.613+0000 7f171480c700 1 -- start start 2024-01-26T23:49:45.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.614+0000 7f171480c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1710067af0 0x7f171015e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:45.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.614+0000 7f171480c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1710067ec0 0x7f1710153280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:45.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.614+0000 7f171480c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17100c7fe0 0x7f17101587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:45.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.614+0000 7f171480c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17100c7e40 con 0x7f17100c7fe0 2024-01-26T23:49:45.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.614+0000 7f171480c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1710157990 con 0x7f1710067ec0 2024-01-26T23:49:45.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.614+0000 7f171480c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17100c7630 con 0x7f1710067af0 2024-01-26T23:49:45.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.614+0000 7f170ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17100c7fe0 0x7f17101587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.614+0000 7f170ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17100c7fe0 0x7f17101587a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59234/0 (socket says 172.21.15.46:59234) 2024-01-26T23:49:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.614+0000 7f170ed9d700 1 -- 172.21.15.46:0/3302288320 learned_addr learned my addr 172.21.15.46:0/3302288320 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.614+0000 7f170e59c700 1 --2- 172.21.15.46:0/3302288320 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1710067af0 0x7f171015e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.614+0000 7f170dd9b700 1 --2- 172.21.15.46:0/3302288320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1710067ec0 0x7f1710153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.614+0000 7f170ed9d700 1 -- 172.21.15.46:0/3302288320 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1710067af0 msgr2=0x7f171015e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.614+0000 7f170ed9d700 1 --2- 172.21.15.46:0/3302288320 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1710067af0 0x7f171015e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.614+0000 7f170ed9d700 1 -- 172.21.15.46:0/3302288320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1710067ec0 msgr2=0x7f1710153280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.614+0000 7f170ed9d700 1 --2- 172.21.15.46:0/3302288320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1710067ec0 0x7f1710153280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.614+0000 7f170ed9d700 1 -- 172.21.15.46:0/3302288320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1708009ae0 con 0x7f17100c7fe0 2024-01-26T23:49:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.614+0000 7f170ed9d700 1 --2- 172.21.15.46:0/3302288320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17100c7fe0 0x7f17101587a0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f171006cb40 tx=0x7f1708018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.615+0000 7f16fe7fc700 1 -- 172.21.15.46:0/3302288320 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1708007ac0 con 0x7f17100c7fe0 2024-01-26T23:49:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.615+0000 7f16fe7fc700 1 -- 172.21.15.46:0/3302288320 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1708007c20 con 0x7f17100c7fe0 2024-01-26T23:49:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.615+0000 7f171480c700 1 -- 172.21.15.46:0/3302288320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1710157ef0 con 0x7f17100c7fe0 2024-01-26T23:49:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.615+0000 7f16fe7fc700 1 -- 172.21.15.46:0/3302288320 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170801d420 con 0x7f17100c7fe0 2024-01-26T23:49:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.615+0000 7f171480c700 1 -- 172.21.15.46:0/3302288320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17100c7b60 con 0x7f17100c7fe0 2024-01-26T23:49:45.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.616+0000 7f171480c700 1 -- 172.21.15.46:0/3302288320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1710158080 con 0x7f17100c7fe0 2024-01-26T23:49:45.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.616+0000 7f16fe7fc700 1 -- 172.21.15.46:0/3302288320 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 70) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f1708003040 con 0x7f17100c7fe0 2024-01-26T23:49:45.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.617+0000 7f16fe7fc700 1 --2- 172.21.15.46:0/3302288320 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f16f00670a0 0x7f16f0069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:45.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.617+0000 7f16fe7fc700 1 -- 172.21.15.46:0/3302288320 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f17080867c0 con 0x7f17100c7fe0 2024-01-26T23:49:45.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.617+0000 7f170e59c700 1 --2- 172.21.15.46:0/3302288320 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f16f00670a0 0x7f16f0069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:45.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.617+0000 7f170e59c700 1 --2- 172.21.15.46:0/3302288320 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f16f00670a0 0x7f16f0069550 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1700004fd0 tx=0x7f1700007eb0 comp rx=0 tx=0).ready entity=mgr.8245 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:45.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.621+0000 7f16fe7fc700 1 -- 172.21.15.46:0/3302288320 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1708053440 con 0x7f17100c7fe0 2024-01-26T23:49:45.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.856+0000 7f171480c700 1 -- 172.21.15.46:0/3302288320 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f171004c710 con 0x7f16f00670a0 2024-01-26T23:49:45.859 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:45.858+0000 7f4471b52700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-01-26T23:49:45.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:45.858+0000 7f16fe7fc700 1 -- 172.21.15.46:0/3302288320 <== mgr.8245 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f171004c710 con 0x7f16f00670a0 2024-01-26T23:49:45.861 INFO:teuthology.orchestra.run.smithi046.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-01-26T23:49:45.864 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-26T23:49:45.865 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec M 2024-01-26T23:49:46.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.002+0000 7f574d5ef700 1 Processor -- start 2024-01-26T23:49:46.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.002+0000 7f574d5ef700 1 -- start start 2024-01-26T23:49:46.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.003+0000 7f574d5ef700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57480ab060 0x7f57480aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:46.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.003+0000 7f574d5ef700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5748159c10 0x7f57480af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:46.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.003+0000 7f574d5ef700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57480ad370 con 0x7f5748159840 2024-01-26T23:49:46.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.003+0000 7f574d5ef700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57480afa30 con 0x7f5748159c10 2024-01-26T23:49:46.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.003+0000 7f574d5ef700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57480ac760 con 0x7f57480ab060 2024-01-26T23:49:46.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.003+0000 7f57477fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5748159c10 0x7f57480af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:46.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.003+0000 7f57477fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5748159c10 0x7f57480af4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59244/0 (socket says 172.21.15.46:59244) 2024-01-26T23:49:46.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.003+0000 7f57477fe700 1 -- 172.21.15.46:0/305407721 learned_addr learned my addr 172.21.15.46:0/305407721 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:46.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.003+0000 7f57467fc700 1 --2- 172.21.15.46:0/305407721 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57480ab060 0x7f57480aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:46.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.004+0000 7f57477fe700 1 -- 172.21.15.46:0/305407721 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57480ab060 msgr2=0x7f57480aefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:46.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.004+0000 7f57477fe700 1 --2- 172.21.15.46:0/305407721 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57480ab060 0x7f57480aefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:46.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.004+0000 7f57477fe700 1 -- 172.21.15.46:0/305407721 >> v1:172.21.15.106:6789/0 conn(0x7f5748159840 legacy=0x7f57480abf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:46.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.004+0000 7f57477fe700 1 -- 172.21.15.46:0/305407721 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57480aca40 con 0x7f5748159c10 2024-01-26T23:49:46.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.004+0000 7f57477fe700 1 --2- 172.21.15.46:0/305407721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5748159c10 0x7f57480af4f0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f5740009f20 tx=0x7f5740009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f073fe10dd2682 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:46.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.004+0000 7f5745ffb700 1 -- 172.21.15.46:0/305407721 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574000e070 con 0x7f5748159c10 2024-01-26T23:49:46.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.004+0000 7f5745ffb700 1 -- 172.21.15.46:0/305407721 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574000ba40 con 0x7f5748159c10 2024-01-26T23:49:46.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.004+0000 7f5745ffb700 1 -- 172.21.15.46:0/305407721 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57400092e0 con 0x7f5748159c10 2024-01-26T23:49:46.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.004+0000 7f574d5ef700 1 -- 172.21.15.46:0/305407721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5748159c10 msgr2=0x7f57480af4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:46.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.004+0000 7f574d5ef700 1 --2- 172.21.15.46:0/305407721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5748159c10 0x7f57480af4f0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f5740009f20 tx=0x7f5740009840 comp rx=0 tx=0).stop 2024-01-26T23:49:46.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.004+0000 7f574d5ef700 1 -- 172.21.15.46:0/305407721 shutdown_connections 2024-01-26T23:49:46.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.005+0000 7f574d5ef700 1 --2- 172.21.15.46:0/305407721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5748159c10 0x7f57480af4f0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:46.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.005+0000 7f574d5ef700 1 --2- 172.21.15.46:0/305407721 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57480ab060 0x7f57480aefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:46.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.005+0000 7f574d5ef700 1 -- 172.21.15.46:0/305407721 >> 172.21.15.46:0/305407721 conn(0x7f5748069220 msgr2=0x7f57480cd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:46.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.005+0000 7f574d5ef700 1 -- 172.21.15.46:0/305407721 shutdown_connections 2024-01-26T23:49:46.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.005+0000 7f574d5ef700 1 -- 172.21.15.46:0/305407721 wait complete. 2024-01-26T23:49:46.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.006+0000 7f574d5ef700 1 Processor -- start 2024-01-26T23:49:46.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.006+0000 7f574d5ef700 1 -- start start 2024-01-26T23:49:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.007+0000 7f574d5ef700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57480ab060 0x7f574814e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.007+0000 7f574d5ef700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5748159840 0x7f574806b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.007+0000 7f574d5ef700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5748159c10 0x7f574815f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.007+0000 7f574d5ef700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57480b2ac0 con 0x7f57480ab060 2024-01-26T23:49:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.007+0000 7f574d5ef700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574806bd70 con 0x7f5748159840 2024-01-26T23:49:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.007+0000 7f574d5ef700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57480b1c10 con 0x7f5748159c10 2024-01-26T23:49:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.007+0000 7f5746ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57480ab060 0x7f574814e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.007+0000 7f5746ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57480ab060 0x7f574814e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59250/0 (socket says 172.21.15.46:59250) 2024-01-26T23:49:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.007+0000 7f5746ffd700 1 -- 172.21.15.46:0/1297480903 learned_addr learned my addr 172.21.15.46:0/1297480903 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.007+0000 7f57477fe700 1 --2- 172.21.15.46:0/1297480903 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5748159c10 0x7f574815f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.007+0000 7f57467fc700 1 --2- 172.21.15.46:0/1297480903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5748159840 0x7f574806b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.007+0000 7f5746ffd700 1 -- 172.21.15.46:0/1297480903 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5748159c10 msgr2=0x7f574815f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.007+0000 7f5746ffd700 1 --2- 172.21.15.46:0/1297480903 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5748159c10 0x7f574815f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.007+0000 7f5746ffd700 1 -- 172.21.15.46:0/1297480903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5748159840 msgr2=0x7f574806b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.007+0000 7f5746ffd700 1 --2- 172.21.15.46:0/1297480903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5748159840 0x7f574806b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.007+0000 7f5746ffd700 1 -- 172.21.15.46:0/1297480903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5740009ae0 con 0x7f57480ab060 2024-01-26T23:49:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.007+0000 7f5746ffd700 1 --2- 172.21.15.46:0/1297480903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57480ab060 0x7f574814e990 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f573800b5e0 tx=0x7f5738009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.008+0000 7f572effd700 1 -- 172.21.15.46:0/1297480903 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573800f510 con 0x7f57480ab060 2024-01-26T23:49:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.008+0000 7f572effd700 1 -- 172.21.15.46:0/1297480903 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573800f670 con 0x7f57480ab060 2024-01-26T23:49:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.008+0000 7f574d5ef700 1 -- 172.21.15.46:0/1297480903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57480b1eb0 con 0x7f57480ab060 2024-01-26T23:49:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.008+0000 7f572effd700 1 -- 172.21.15.46:0/1297480903 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573800fa30 con 0x7f57480ab060 2024-01-26T23:49:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.008+0000 7f574d5ef700 1 -- 172.21.15.46:0/1297480903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57480b2470 con 0x7f57480ab060 2024-01-26T23:49:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.009+0000 7f574d5ef700 1 -- 172.21.15.46:0/1297480903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574804c710 con 0x7f57480ab060 2024-01-26T23:49:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.009+0000 7f572effd700 1 -- 172.21.15.46:0/1297480903 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 71) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f573801a400 con 0x7f57480ab060 2024-01-26T23:49:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.010+0000 7f572effd700 1 --2- 172.21.15.46:0/1297480903 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5734067130 0x7f57340695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.010+0000 7f572effd700 1 -- 172.21.15.46:0/1297480903 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f5738084620 con 0x7f57480ab060 2024-01-26T23:49:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.010+0000 7f57467fc700 1 --2- 172.21.15.46:0/1297480903 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5734067130 0x7f57340695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.011+0000 7f57467fc700 1 --2- 172.21.15.46:0/1297480903 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5734067130 0x7f57340695e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5748156a30 tx=0x7f573c008040 comp rx=0 tx=0).ready entity=mgr.8245 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:46.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.014+0000 7f572effd700 1 -- 172.21.15.46:0/1297480903 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f57380512c0 con 0x7f57480ab060 2024-01-26T23:49:46.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.247+0000 7f574d5ef700 1 -- 172.21.15.46:0/1297480903 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "M", "target": ["mon-mgr", ""]}) v1 -- 0x7f57480b2c00 con 0x7f5734067130 2024-01-26T23:49:46.250 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:46.249+0000 7f4471b52700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-01-26T23:49:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.249+0000 7f572effd700 1 -- 172.21.15.46:0/1297480903 <== mgr.8245 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f573c00bdd0 con 0x7f5734067130 2024-01-26T23:49:46.252 INFO:teuthology.orchestra.run.smithi046.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-01-26T23:49:46.255 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-26T23:49:46.256 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec -1m 2024-01-26T23:49:46.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.391+0000 7fe4bcbc0700 1 Processor -- start 2024-01-26T23:49:46.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.391+0000 7fe4bcbc0700 1 -- start start 2024-01-26T23:49:46.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.392+0000 7fe4bcbc0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4b80af790 0x7fe4b8162520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:46.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.392+0000 7fe4bcbc0700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b8158f00 con 0x7fe4b8157750 2024-01-26T23:49:46.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.392+0000 7fe4bcbc0700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b806fe90 con 0x7fe4b806b3c0 2024-01-26T23:49:46.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.392+0000 7fe4bcbc0700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4b81582f0 con 0x7fe4b80af790 2024-01-26T23:49:46.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.392+0000 7fe4b6d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4b80af790 0x7fe4b8162520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:46.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.392+0000 7fe4b659c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe4b8157750 0x7fe4b8157b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37966/0 (socket says 172.21.15.46:37966) 2024-01-26T23:49:46.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.392+0000 7fe4b659c700 1 -- 172.21.15.46:0/474489534 learned_addr learned my addr 172.21.15.46:0/474489534 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:46.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.392+0000 7fe4b6d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4b80af790 0x7fe4b8162520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59262/0 (socket says 172.21.15.46:59262) 2024-01-26T23:49:46.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.393+0000 7fe4b6d9d700 1 -- 172.21.15.46:0/474489534 >> v1:172.21.15.106:6789/0 conn(0x7fe4b806b3c0 legacy=0x7fe4b80af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:46.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.393+0000 7fe4b6d9d700 1 -- 172.21.15.46:0/474489534 >> v1:172.21.15.46:6789/0 conn(0x7fe4b8157750 legacy=0x7fe4b8157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:46.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.393+0000 7fe4b6d9d700 1 -- 172.21.15.46:0/474489534 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b81585d0 con 0x7fe4b80af790 2024-01-26T23:49:46.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.393+0000 7fe4b6d9d700 1 --2- 172.21.15.46:0/474489534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4b80af790 0x7fe4b8162520 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0009e60 tx=0x7fe4b0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4614a14cd0ba3d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:46.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.393+0000 7fe4b5d9b700 1 -- 172.21.15.46:0/474489534 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b000e070 con 0x7fe4b80af790 2024-01-26T23:49:46.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.393+0000 7fe4b5d9b700 1 -- 172.21.15.46:0/474489534 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b000bc30 con 0x7fe4b80af790 2024-01-26T23:49:46.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.393+0000 7fe4b5d9b700 1 -- 172.21.15.46:0/474489534 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0012430 con 0x7fe4b80af790 2024-01-26T23:49:46.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.394+0000 7fe4bcbc0700 1 -- 172.21.15.46:0/474489534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4b80af790 msgr2=0x7fe4b8162520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:46.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.394+0000 7fe4bcbc0700 1 --2- 172.21.15.46:0/474489534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4b80af790 0x7fe4b8162520 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0009e60 tx=0x7fe4b0009780 comp rx=0 tx=0).stop 2024-01-26T23:49:46.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.394+0000 7fe4bcbc0700 1 -- 172.21.15.46:0/474489534 shutdown_connections 2024-01-26T23:49:46.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.394+0000 7fe4bcbc0700 1 --2- 172.21.15.46:0/474489534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4b80af790 0x7fe4b8162520 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:46.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.394+0000 7fe4bcbc0700 1 -- 172.21.15.46:0/474489534 >> 172.21.15.46:0/474489534 conn(0x7fe4b80cdf80 msgr2=0x7fe4b806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:46.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.394+0000 7fe4bcbc0700 1 -- 172.21.15.46:0/474489534 shutdown_connections 2024-01-26T23:49:46.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.394+0000 7fe4bcbc0700 1 -- 172.21.15.46:0/474489534 wait complete. 2024-01-26T23:49:46.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.395+0000 7fe4bcbc0700 1 Processor -- start 2024-01-26T23:49:46.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.395+0000 7fe4bcbc0700 1 -- start start 2024-01-26T23:49:46.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4bcbc0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4b806b3c0 0x7fe4b8150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:46.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4bcbc0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4b80af790 0x7fe4b80ae320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:46.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4bcbc0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4b8157750 0x7fe4b815d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:46.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4bcbc0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b816d7f0 con 0x7fe4b806b3c0 2024-01-26T23:49:46.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4bcbc0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b80a5430 con 0x7fe4b8157750 2024-01-26T23:49:46.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4bcbc0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b816dc80 con 0x7fe4b80af790 2024-01-26T23:49:46.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4b659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4b806b3c0 0x7fe4b8150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:46.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4b659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4b806b3c0 0x7fe4b8150890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59272/0 (socket says 172.21.15.46:59272) 2024-01-26T23:49:46.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4b659c700 1 -- 172.21.15.46:0/3040392908 learned_addr learned my addr 172.21.15.46:0/3040392908 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:46.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4affff700 1 --2- 172.21.15.46:0/3040392908 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4b80af790 0x7fe4b80ae320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:46.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4b6d9d700 1 --2- 172.21.15.46:0/3040392908 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4b8157750 0x7fe4b815d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4b659c700 1 -- 172.21.15.46:0/3040392908 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4b80af790 msgr2=0x7fe4b80ae320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4b659c700 1 --2- 172.21.15.46:0/3040392908 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4b80af790 0x7fe4b80ae320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4b659c700 1 -- 172.21.15.46:0/3040392908 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4b8157750 msgr2=0x7fe4b815d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4b659c700 1 --2- 172.21.15.46:0/3040392908 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4b8157750 0x7fe4b815d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4b659c700 1 -- 172.21.15.46:0/3040392908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b0009a20 con 0x7fe4b806b3c0 2024-01-26T23:49:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4b6d9d700 1 --2- 172.21.15.46:0/3040392908 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4b8157750 0x7fe4b815d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:49:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4affff700 1 --2- 172.21.15.46:0/3040392908 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4b80af790 0x7fe4b80ae320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:49:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4b659c700 1 --2- 172.21.15.46:0/3040392908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4b806b3c0 0x7fe4b8150890 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fe4a800c560 tx=0x7fe4a8001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4ae7fc700 1 -- 172.21.15.46:0/3040392908 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a8005b30 con 0x7fe4b806b3c0 2024-01-26T23:49:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4ae7fc700 1 -- 172.21.15.46:0/3040392908 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a8005cb0 con 0x7fe4b806b3c0 2024-01-26T23:49:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.396+0000 7fe4bcbc0700 1 -- 172.21.15.46:0/3040392908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b80aede0 con 0x7fe4b806b3c0 2024-01-26T23:49:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.397+0000 7fe4ae7fc700 1 -- 172.21.15.46:0/3040392908 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a8007320 con 0x7fe4b806b3c0 2024-01-26T23:49:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.397+0000 7fe4bcbc0700 1 -- 172.21.15.46:0/3040392908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b816d1a0 con 0x7fe4b806b3c0 2024-01-26T23:49:46.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.398+0000 7fe4bcbc0700 1 -- 172.21.15.46:0/3040392908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b816dad0 con 0x7fe4b806b3c0 2024-01-26T23:49:46.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.398+0000 7fe4ae7fc700 1 -- 172.21.15.46:0/3040392908 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 71) v1 ==== 84918+0+0 (secure 0 0 0) 0x7fe4a801b3b0 con 0x7fe4b806b3c0 2024-01-26T23:49:46.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.398+0000 7fe4ae7fc700 1 --2- 172.21.15.46:0/3040392908 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe498067130 0x7fe4980695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:46.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.399+0000 7fe4ae7fc700 1 -- 172.21.15.46:0/3040392908 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fe4a80841a0 con 0x7fe4b806b3c0 2024-01-26T23:49:46.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.399+0000 7fe4affff700 1 --2- 172.21.15.46:0/3040392908 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe498067130 0x7fe4980695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:46.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.399+0000 7fe4affff700 1 --2- 172.21.15.46:0/3040392908 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe498067130 0x7fe4980695e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe4b816cd80 tx=0x7fe4a0009040 comp rx=0 tx=0).ready entity=mgr.8245 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:46.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.403+0000 7fe4ae7fc700 1 -- 172.21.15.46:0/3040392908 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe4a8050dc0 con 0x7fe4b806b3c0 2024-01-26T23:49:46.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.640+0000 7fe4bcbc0700 1 -- 172.21.15.46:0/3040392908 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "-1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4b816d930 con 0x7fe498067130 2024-01-26T23:49:46.642 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:46.642+0000 7f4471b52700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-01-26T23:49:46.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.642+0000 7fe4ae7fc700 1 -- 172.21.15.46:0/3040392908 <== mgr.8245 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc.) v1 ==== 96+0+0 (secure 0 0 0) 0x7fe4a00014b0 con 0x7fe498067130 2024-01-26T23:49:46.644 INFO:teuthology.orchestra.run.smithi046.stderr:Error ENOENT: invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-01-26T23:49:46.648 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-26T23:49:46.649 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec '' 2024-01-26T23:49:46.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.783+0000 7f86678a9700 1 Processor -- start 2024-01-26T23:49:46.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.783+0000 7f86678a9700 1 -- start start 2024-01-26T23:49:46.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.784+0000 7f86678a9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f866015a520 0x7f8660156970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:46.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.784+0000 7f86678a9700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86600b0830 con 0x7f86600ae2f0 2024-01-26T23:49:46.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.784+0000 7f86678a9700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86600ad350 con 0x7f86601565a0 2024-01-26T23:49:46.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.784+0000 7f86678a9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86600ae130 con 0x7f866015a520 2024-01-26T23:49:46.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.784+0000 7f8665e46700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f86600ae2f0 0x7f86600b0ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37982/0 (socket says 172.21.15.46:37982) 2024-01-26T23:49:46.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.784+0000 7f8665e46700 1 -- 172.21.15.46:0/350194993 learned_addr learned my addr 172.21.15.46:0/350194993 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:46.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.784+0000 7f8664e44700 1 --2- 172.21.15.46:0/350194993 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f866015a520 0x7f8660156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:46.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.784+0000 7f8657fff700 1 -- 172.21.15.46:0/350194993 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288787570 0 0) 0x7f86600b0830 con 0x7f86600ae2f0 2024-01-26T23:49:46.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.784+0000 7f8657fff700 1 -- 172.21.15.46:0/350194993 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f864c003570 con 0x7f86600ae2f0 2024-01-26T23:49:46.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.784+0000 7f8657fff700 1 -- 172.21.15.46:0/350194993 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708621281 0 0) 0x7f86600ad350 con 0x7f86601565a0 2024-01-26T23:49:46.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.784+0000 7f8657fff700 1 -- 172.21.15.46:0/350194993 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86600b0830 con 0x7f86601565a0 2024-01-26T23:49:46.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.784+0000 7f8664e44700 1 -- 172.21.15.46:0/350194993 >> v1:172.21.15.106:6789/0 conn(0x7f86601565a0 legacy=0x7f86600acc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:46.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.784+0000 7f8664e44700 1 -- 172.21.15.46:0/350194993 >> v1:172.21.15.46:6789/0 conn(0x7f86600ae2f0 legacy=0x7f86600b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:46.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.784+0000 7f8664e44700 1 -- 172.21.15.46:0/350194993 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86600ad5b0 con 0x7f866015a520 2024-01-26T23:49:46.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.785+0000 7f8664e44700 1 --2- 172.21.15.46:0/350194993 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f866015a520 0x7f8660156970 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8650009910 tx=0x7f86500091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd227a9350b4beed server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:46.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.785+0000 7f8657fff700 1 -- 172.21.15.46:0/350194993 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8650011630 con 0x7f866015a520 2024-01-26T23:49:46.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.785+0000 7f8657fff700 1 -- 172.21.15.46:0/350194993 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8650011790 con 0x7f866015a520 2024-01-26T23:49:46.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.785+0000 7f8657fff700 1 -- 172.21.15.46:0/350194993 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8650011b30 con 0x7f866015a520 2024-01-26T23:49:46.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.785+0000 7f86678a9700 1 -- 172.21.15.46:0/350194993 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f866015a520 msgr2=0x7f8660156970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:46.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.785+0000 7f86678a9700 1 --2- 172.21.15.46:0/350194993 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f866015a520 0x7f8660156970 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8650009910 tx=0x7f86500091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:46.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.786+0000 7f86678a9700 1 -- 172.21.15.46:0/350194993 shutdown_connections 2024-01-26T23:49:46.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.786+0000 7f86678a9700 1 --2- 172.21.15.46:0/350194993 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f866015a520 0x7f8660156970 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:46.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.786+0000 7f86678a9700 1 -- 172.21.15.46:0/350194993 >> 172.21.15.46:0/350194993 conn(0x7f86600cdf80 msgr2=0x7f866006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:46.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.786+0000 7f86678a9700 1 -- 172.21.15.46:0/350194993 shutdown_connections 2024-01-26T23:49:46.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.786+0000 7f86678a9700 1 -- 172.21.15.46:0/350194993 wait complete. 2024-01-26T23:49:46.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.787+0000 7f86678a9700 1 Processor -- start 2024-01-26T23:49:46.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.787+0000 7f86678a9700 1 -- start start 2024-01-26T23:49:46.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f86678a9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86600ae2f0 0x7f86601696a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:46.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f86678a9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86601565a0 0x7f8660154e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:46.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f86678a9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f866015a520 0x7f8660155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:46.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f86678a9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8660159ee0 con 0x7f86600ae2f0 2024-01-26T23:49:46.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f86678a9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866016cff0 con 0x7f86601565a0 2024-01-26T23:49:46.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f86678a9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866016cca0 con 0x7f866015a520 2024-01-26T23:49:46.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f8665645700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86600ae2f0 0x7f86601696a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:46.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f8665645700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86600ae2f0 0x7f86601696a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59278/0 (socket says 172.21.15.46:59278) 2024-01-26T23:49:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f8665645700 1 -- 172.21.15.46:0/2958979563 learned_addr learned my addr 172.21.15.46:0/2958979563 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f8664e44700 1 --2- 172.21.15.46:0/2958979563 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86601565a0 0x7f8660154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f8665e46700 1 --2- 172.21.15.46:0/2958979563 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f866015a520 0x7f8660155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f8665645700 1 -- 172.21.15.46:0/2958979563 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f866015a520 msgr2=0x7f8660155390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f8665645700 1 --2- 172.21.15.46:0/2958979563 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f866015a520 0x7f8660155390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f8665645700 1 -- 172.21.15.46:0/2958979563 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86601565a0 msgr2=0x7f8660154e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f8665645700 1 --2- 172.21.15.46:0/2958979563 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86601565a0 0x7f8660154e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f8665645700 1 -- 172.21.15.46:0/2958979563 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8650009420 con 0x7f86600ae2f0 2024-01-26T23:49:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f8664e44700 1 --2- 172.21.15.46:0/2958979563 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f86601565a0 0x7f8660154e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:49:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.788+0000 7f8665645700 1 --2- 172.21.15.46:0/2958979563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86600ae2f0 0x7f86601696a0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f865c00c790 tx=0x7f865c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.789+0000 7f86557fa700 1 -- 172.21.15.46:0/2958979563 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c010510 con 0x7f86600ae2f0 2024-01-26T23:49:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.789+0000 7f86557fa700 1 -- 172.21.15.46:0/2958979563 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865c010670 con 0x7f86600ae2f0 2024-01-26T23:49:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.789+0000 7f86557fa700 1 -- 172.21.15.46:0/2958979563 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c010a10 con 0x7f86600ae2f0 2024-01-26T23:49:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.789+0000 7f86678a9700 1 -- 172.21.15.46:0/2958979563 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8660169ef0 con 0x7f86600ae2f0 2024-01-26T23:49:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.789+0000 7f86678a9700 1 -- 172.21.15.46:0/2958979563 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f866016a4b0 con 0x7f86600ae2f0 2024-01-26T23:49:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.790+0000 7f86678a9700 1 -- 172.21.15.46:0/2958979563 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f866016d2d0 con 0x7f86600ae2f0 2024-01-26T23:49:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.790+0000 7f86557fa700 1 -- 172.21.15.46:0/2958979563 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 71) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f865c01b400 con 0x7f86600ae2f0 2024-01-26T23:49:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.791+0000 7f86557fa700 1 --2- 172.21.15.46:0/2958979563 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8644067130 0x7f86440695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.791+0000 7f86557fa700 1 -- 172.21.15.46:0/2958979563 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f865c084a50 con 0x7f86600ae2f0 2024-01-26T23:49:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.791+0000 7f8664e44700 1 --2- 172.21.15.46:0/2958979563 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8644067130 0x7f86440695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.791+0000 7f8664e44700 1 --2- 172.21.15.46:0/2958979563 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8644067130 0x7f86440695e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f866015a8f0 tx=0x7f8650017000 comp rx=0 tx=0).ready entity=mgr.8245 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:46.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:46.795+0000 7f86557fa700 1 -- 172.21.15.46:0/2958979563 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f865c051670 con 0x7f86600ae2f0 2024-01-26T23:49:47.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.029+0000 7f86678a9700 1 -- 172.21.15.46:0/2958979563 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f866016d130 con 0x7f8644067130 2024-01-26T23:49:47.031 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:47.030+0000 7f4471b52700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-01-26T23:49:47.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.031+0000 7f86557fa700 1 -- 172.21.15.46:0/2958979563 <== mgr.8245 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8650000bc0 con 0x7f8644067130 2024-01-26T23:49:47.035 INFO:teuthology.orchestra.run.smithi046.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-01-26T23:49:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-26T23:49:47.038 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:47.038 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/minutes) 2024-01-26T23:49:47.061 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:47.061 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/hourly) 2024-01-26T23:49:47.124 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:47.125 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/daily) 2024-01-26T23:49:47.190 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:47.190 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/weekly) 2024-01-26T23:49:47.255 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:47.255 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/monthly) 2024-01-26T23:49:47.321 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:47.321 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/yearly) 2024-01-26T23:49:47.387 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:49:47.387 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/bad_period_spec) 2024-01-26T23:49:47.453 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-26T23:49:47.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.634+0000 7fe183cc2700 1 Processor -- start 2024-01-26T23:49:47.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.635+0000 7fe183cc2700 1 -- start start 2024-01-26T23:49:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.636+0000 7fe183cc2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe17c06ddb0 0x7fe17c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.636+0000 7fe183cc2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe17c0cea80 con 0x7fe17c157860 2024-01-26T23:49:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.636+0000 7fe183cc2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe17c0cde70 con 0x7fe17c157490 2024-01-26T23:49:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.636+0000 7fe183cc2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe17c0cdc60 con 0x7fe17c06ddb0 2024-01-26T23:49:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.636+0000 7fe18225f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe17c06ddb0 0x7fe17c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.636+0000 7fe18225f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe17c06ddb0 0x7fe17c0cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59284/0 (socket says 172.21.15.46:59284) 2024-01-26T23:49:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.636+0000 7fe18225f700 1 -- 172.21.15.46:0/711164366 learned_addr learned my addr 172.21.15.46:0/711164366 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.636+0000 7fe18125d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe17c157860 0x7fe17c0b1ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37984/0 (socket says 172.21.15.46:37984) 2024-01-26T23:49:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.636+0000 7fe18225f700 1 -- 172.21.15.46:0/711164366 >> v1:172.21.15.106:6789/0 conn(0x7fe17c157490 legacy=0x7fe17c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.636+0000 7fe18225f700 1 -- 172.21.15.46:0/711164366 >> v1:172.21.15.46:6789/0 conn(0x7fe17c157860 legacy=0x7fe17c0b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.636+0000 7fe18225f700 1 -- 172.21.15.46:0/711164366 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c0ce150 con 0x7fe17c06ddb0 2024-01-26T23:49:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.636+0000 7fe18225f700 1 --2- 172.21.15.46:0/711164366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe17c06ddb0 0x7fe17c0cc600 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fe174009f20 tx=0x7fe174009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14959f1c03aba8dc server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.636+0000 7fe180a5c700 1 -- 172.21.15.46:0/711164366 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17400e070 con 0x7fe17c06ddb0 2024-01-26T23:49:47.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.637+0000 7fe180a5c700 1 -- 172.21.15.46:0/711164366 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17400ba80 con 0x7fe17c06ddb0 2024-01-26T23:49:47.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.637+0000 7fe180a5c700 1 -- 172.21.15.46:0/711164366 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1740092e0 con 0x7fe17c06ddb0 2024-01-26T23:49:47.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.637+0000 7fe183cc2700 1 -- 172.21.15.46:0/711164366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe17c06ddb0 msgr2=0x7fe17c0cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:47.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.637+0000 7fe183cc2700 1 --2- 172.21.15.46:0/711164366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe17c06ddb0 0x7fe17c0cc600 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fe174009f20 tx=0x7fe174009840 comp rx=0 tx=0).stop 2024-01-26T23:49:47.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.637+0000 7fe183cc2700 1 -- 172.21.15.46:0/711164366 shutdown_connections 2024-01-26T23:49:47.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.637+0000 7fe183cc2700 1 --2- 172.21.15.46:0/711164366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe17c06ddb0 0x7fe17c0cc600 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:47.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.637+0000 7fe183cc2700 1 -- 172.21.15.46:0/711164366 >> 172.21.15.46:0/711164366 conn(0x7fe17c067930 msgr2=0x7fe17c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:47.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.637+0000 7fe183cc2700 1 -- 172.21.15.46:0/711164366 shutdown_connections 2024-01-26T23:49:47.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.637+0000 7fe183cc2700 1 -- 172.21.15.46:0/711164366 wait complete. 2024-01-26T23:49:47.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.638+0000 7fe183cc2700 1 Processor -- start 2024-01-26T23:49:47.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.638+0000 7fe183cc2700 1 -- start start 2024-01-26T23:49:47.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.639+0000 7fe183cc2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe17c06ddb0 0x7fe17c158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:47.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.639+0000 7fe183cc2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe17c157490 0x7fe17c14c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:47.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.639+0000 7fe183cc2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe17c157860 0x7fe17c168570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:47.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.639+0000 7fe183cc2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c159be0 con 0x7fe17c157490 2024-01-26T23:49:47.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.639+0000 7fe183cc2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c159620 con 0x7fe17c157860 2024-01-26T23:49:47.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.639+0000 7fe183cc2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c159900 con 0x7fe17c06ddb0 2024-01-26T23:49:47.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.639+0000 7fe18125d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe17c157490 0x7fe17c14c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:47.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.639+0000 7fe18125d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe17c157490 0x7fe17c14c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59290/0 (socket says 172.21.15.46:59290) 2024-01-26T23:49:47.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.639+0000 7fe18125d700 1 -- 172.21.15.46:0/2323979636 learned_addr learned my addr 172.21.15.46:0/2323979636 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:47.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.639+0000 7fe18225f700 1 --2- 172.21.15.46:0/2323979636 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe17c157860 0x7fe17c168570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:47.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.640+0000 7fe181a5e700 1 --2- 172.21.15.46:0/2323979636 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe17c06ddb0 0x7fe17c158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:47.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.640+0000 7fe18125d700 1 -- 172.21.15.46:0/2323979636 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe17c06ddb0 msgr2=0x7fe17c158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:47.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.640+0000 7fe18125d700 1 --2- 172.21.15.46:0/2323979636 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe17c06ddb0 0x7fe17c158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:47.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.640+0000 7fe18125d700 1 -- 172.21.15.46:0/2323979636 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe17c157860 msgr2=0x7fe17c168570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:47.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.640+0000 7fe18125d700 1 --2- 172.21.15.46:0/2323979636 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe17c157860 0x7fe17c168570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:47.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.640+0000 7fe18125d700 1 -- 172.21.15.46:0/2323979636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe174009ae0 con 0x7fe17c157490 2024-01-26T23:49:47.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.640+0000 7fe18125d700 1 --2- 172.21.15.46:0/2323979636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe17c157490 0x7fe17c14c090 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fe16c00c560 tx=0x7fe16c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:47.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.640+0000 7fe171ffb700 1 -- 172.21.15.46:0/2323979636 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c005a70 con 0x7fe17c157490 2024-01-26T23:49:47.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.640+0000 7fe171ffb700 1 -- 172.21.15.46:0/2323979636 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16c005bd0 con 0x7fe17c157490 2024-01-26T23:49:47.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.640+0000 7fe183cc2700 1 -- 172.21.15.46:0/2323979636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe17c167d70 con 0x7fe17c157490 2024-01-26T23:49:47.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.640+0000 7fe171ffb700 1 -- 172.21.15.46:0/2323979636 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c0070d0 con 0x7fe17c157490 2024-01-26T23:49:47.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.640+0000 7fe183cc2700 1 -- 172.21.15.46:0/2323979636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe17c159300 con 0x7fe17c157490 2024-01-26T23:49:47.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.642+0000 7fe183cc2700 1 -- 172.21.15.46:0/2323979636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe17c04c710 con 0x7fe17c157490 2024-01-26T23:49:47.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.642+0000 7fe171ffb700 1 -- 172.21.15.46:0/2323979636 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 71) v1 ==== 84918+0+0 (secure 0 0 0) 0x7fe16c007480 con 0x7fe17c157490 2024-01-26T23:49:47.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.642+0000 7fe171ffb700 1 --2- 172.21.15.46:0/2323979636 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe160067130 0x7fe1600695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:47.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.642+0000 7fe171ffb700 1 -- 172.21.15.46:0/2323979636 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fe16c083ff0 con 0x7fe17c157490 2024-01-26T23:49:47.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.642+0000 7fe181a5e700 1 --2- 172.21.15.46:0/2323979636 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe160067130 0x7fe1600695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:47.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.643+0000 7fe181a5e700 1 --2- 172.21.15.46:0/2323979636 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe160067130 0x7fe1600695e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe178003a80 tx=0x7fe178007ec0 comp rx=0 tx=0).ready entity=mgr.8245 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:47.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.647+0000 7fe171ffb700 1 -- 172.21.15.46:0/2323979636 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe16c050c10 con 0x7fe17c157490 2024-01-26T23:49:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:47.910+0000 7fe183cc2700 1 -- 172.21.15.46:0/2323979636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fe17c0cac50 con 0x7fe17c157490 2024-01-26T23:49:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:48.701+0000 7fe171ffb700 1 -- 172.21.15.46:0/2323979636 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 72) v1 ==== 84901+0+0 (secure 0 0 0) 0x7fe16c050fe0 con 0x7fe17c157490 2024-01-26T23:49:48.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:48.702+0000 7fe171ffb700 1 -- 172.21.15.46:0/2323979636 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v72) v1 ==== 93+0+0 (secure 0 0 0) 0x7fe16c0546a0 con 0x7fe17c157490 2024-01-26T23:49:48.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:48.705+0000 7fe183cc2700 1 -- 172.21.15.46:0/2323979636 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe160067130 msgr2=0x7fe1600695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:48.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:48.705+0000 7fe183cc2700 1 --2- 172.21.15.46:0/2323979636 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe160067130 0x7fe1600695e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe178003a80 tx=0x7fe178007ec0 comp rx=0 tx=0).stop 2024-01-26T23:49:48.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:48.705+0000 7fe183cc2700 1 -- 172.21.15.46:0/2323979636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe17c157490 msgr2=0x7fe17c14c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:48.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:48.705+0000 7fe183cc2700 1 --2- 172.21.15.46:0/2323979636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe17c157490 0x7fe17c14c090 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fe16c00c560 tx=0x7fe16c007f70 comp rx=0 tx=0).stop 2024-01-26T23:49:48.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:48.705+0000 7fe183cc2700 1 -- 172.21.15.46:0/2323979636 shutdown_connections 2024-01-26T23:49:48.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:48.705+0000 7fe183cc2700 1 --2- 172.21.15.46:0/2323979636 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe160067130 0x7fe1600695e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:48.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:48.705+0000 7fe183cc2700 1 --2- 172.21.15.46:0/2323979636 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe17c157860 0x7fe17c168570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:48.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:48.705+0000 7fe183cc2700 1 --2- 172.21.15.46:0/2323979636 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe17c06ddb0 0x7fe17c158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:48.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:48.705+0000 7fe183cc2700 1 --2- 172.21.15.46:0/2323979636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe17c157490 0x7fe17c14c090 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:48.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:48.705+0000 7fe183cc2700 1 -- 172.21.15.46:0/2323979636 >> 172.21.15.46:0/2323979636 conn(0x7fe17c067930 msgr2=0x7fe17c156650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:48.705+0000 7fe183cc2700 1 -- 172.21.15.46:0/2323979636 shutdown_connections 2024-01-26T23:49:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:48.705+0000 7fe183cc2700 1 -- 172.21.15.46:0/2323979636 wait complete. 2024-01-26T23:49:48.723 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-26T23:49:48.723 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:49:48.769 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:49:48.809 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-26T23:49:48.809 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-26T23:49:48.809 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-26T23:49:48.809 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-26T23:49:48.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[86941]: fuse finished with error 0 and tester_r 0 2024-01-26T23:49:48.853 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:49:48.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:49:48.854 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-26T23:49:49.023 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:49:49.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:49:49.023 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-26T23:49:49.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:49:49.039 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-26T23:49:49.040 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-26T23:49:49.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:49:49.040 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-26T23:49:49.046 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:49.045+0000 7fd054cba1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:49:49.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:49:49.083 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-26T23:49:49.083 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-26T23:49:49.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:49:49.084 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-26T23:49:49.096 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:49.095+0000 7ffb394e41c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:49:49.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-26T23:49:49.100 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-26T23:49:49.101 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:49:49.124 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:49.124+0000 7fd054cba1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:49:49.189 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:49.188+0000 7ffb394e41c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:49:49.194 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:49.194+0000 7fd054cba1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:49:49.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.233+0000 7f874ae01700 1 Processor -- start 2024-01-26T23:49:49.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.233+0000 7f874ae01700 1 -- start start 2024-01-26T23:49:49.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.234+0000 7f874ae01700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f87440b12c0 0x7f87441646a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:49.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.234+0000 7f874ae01700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f874406f940 con 0x7f8744159b30 2024-01-26T23:49:49.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.234+0000 7f874ae01700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87440b3230 con 0x7f8744159760 2024-01-26T23:49:49.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.234+0000 7f874ae01700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f874406ed30 con 0x7f87440b12c0 2024-01-26T23:49:49.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.234+0000 7f8743fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8744159b30 0x7f87440b0bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38010/0 (socket says 172.21.15.46:38010) 2024-01-26T23:49:49.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.234+0000 7f8743fff700 1 -- 172.21.15.46:0/1133894756 learned_addr learned my addr 172.21.15.46:0/1133894756 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:49.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.234+0000 7f874939e700 1 --2- 172.21.15.46:0/1133894756 >> v2:172.21.15.106:3300/0 conn(0x7f87440b12c0 0x7f87441646a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:49.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.234+0000 7f87437fe700 1 -- 172.21.15.46:0/1133894756 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160165157 0 0) 0x7f874406f940 con 0x7f8744159b30 2024-01-26T23:49:49.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.235+0000 7f87437fe700 1 -- 172.21.15.46:0/1133894756 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8734003570 con 0x7f8744159b30 2024-01-26T23:49:49.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.235+0000 7f874939e700 1 -- 172.21.15.46:0/1133894756 >> v1:172.21.15.106:6789/0 conn(0x7f8744159760 legacy=0x7f87440b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:49.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.235+0000 7f874939e700 1 -- 172.21.15.46:0/1133894756 >> v1:172.21.15.46:6789/0 conn(0x7f8744159b30 legacy=0x7f87440b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:49.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.235+0000 7f87437fe700 1 -- 172.21.15.46:0/1133894756 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119318384 0 0) 0x7f87440b3230 con 0x7f8744159760 2024-01-26T23:49:49.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.235+0000 7f874939e700 1 -- 172.21.15.46:0/1133894756 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874406f010 con 0x7f87440b12c0 2024-01-26T23:49:49.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.235+0000 7f874939e700 1 --2- 172.21.15.46:0/1133894756 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87440b12c0 0x7f87441646a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f873c009e90 tx=0x7f873c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=346e235903ed7bce server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:49.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.235+0000 7f87437fe700 1 -- 172.21.15.46:0/1133894756 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c00e070 con 0x7f87440b12c0 2024-01-26T23:49:49.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.235+0000 7f87437fe700 1 -- 172.21.15.46:0/1133894756 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873c00bc40 con 0x7f87440b12c0 2024-01-26T23:49:49.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.235+0000 7f87437fe700 1 -- 172.21.15.46:0/1133894756 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c012430 con 0x7f87440b12c0 2024-01-26T23:49:49.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.236+0000 7f874ae01700 1 -- 172.21.15.46:0/1133894756 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87440b12c0 msgr2=0x7f87441646a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:49.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.236+0000 7f874ae01700 1 --2- 172.21.15.46:0/1133894756 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87440b12c0 0x7f87441646a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f873c009e90 tx=0x7f873c009780 comp rx=0 tx=0).stop 2024-01-26T23:49:49.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.236+0000 7f874ae01700 1 -- 172.21.15.46:0/1133894756 shutdown_connections 2024-01-26T23:49:49.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.236+0000 7f874ae01700 1 --2- 172.21.15.46:0/1133894756 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87440b12c0 0x7f87441646a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.236+0000 7f874ae01700 1 -- 172.21.15.46:0/1133894756 >> 172.21.15.46:0/1133894756 conn(0x7f8744069510 msgr2=0x7f87440aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:49.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.236+0000 7f874ae01700 1 -- 172.21.15.46:0/1133894756 shutdown_connections 2024-01-26T23:49:49.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.236+0000 7f874ae01700 1 -- 172.21.15.46:0/1133894756 wait complete. 2024-01-26T23:49:49.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.237+0000 7f874ae01700 1 Processor -- start 2024-01-26T23:49:49.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.238+0000 7f874ae01700 1 -- start start 2024-01-26T23:49:49.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f874ae01700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87440b12c0 0x7f8744133c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:49.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f874ae01700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8744159760 0x7f8744131210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:49.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f874ae01700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8744159b30 0x7f8744132ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:49.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f874ae01700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8744132650 con 0x7f87440b12c0 2024-01-26T23:49:49.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f874ae01700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8744131a40 con 0x7f8744159b30 2024-01-26T23:49:49.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f874ae01700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8744131d20 con 0x7f8744159760 2024-01-26T23:49:49.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f8748b9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87440b12c0 0x7f8744133c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:49.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f8748b9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87440b12c0 0x7f8744133c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59314/0 (socket says 172.21.15.46:59314) 2024-01-26T23:49:49.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f8748b9d700 1 -- 172.21.15.46:0/4003261870 learned_addr learned my addr 172.21.15.46:0/4003261870 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:49.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f874939e700 1 --2- 172.21.15.46:0/4003261870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8744159b30 0x7f8744132ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:49.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f8743fff700 1 --2- 172.21.15.46:0/4003261870 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8744159760 0x7f8744131210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:49.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f8748b9d700 1 -- 172.21.15.46:0/4003261870 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8744159760 msgr2=0x7f8744131210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:49.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f8748b9d700 1 --2- 172.21.15.46:0/4003261870 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8744159760 0x7f8744131210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f8748b9d700 1 -- 172.21.15.46:0/4003261870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8744159b30 msgr2=0x7f8744132ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:49.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f8748b9d700 1 --2- 172.21.15.46:0/4003261870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8744159b30 0x7f8744132ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f8748b9d700 1 -- 172.21.15.46:0/4003261870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c009a20 con 0x7f87440b12c0 2024-01-26T23:49:49.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f874939e700 1 --2- 172.21.15.46:0/4003261870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8744159b30 0x7f8744132ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:49.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.239+0000 7f8748b9d700 1 --2- 172.21.15.46:0/4003261870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87440b12c0 0x7f8744133c00 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f873800c7e0 tx=0x7f8738007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:49.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.240+0000 7f8740ff9700 1 -- 172.21.15.46:0/4003261870 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87380105c0 con 0x7f87440b12c0 2024-01-26T23:49:49.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.240+0000 7f8740ff9700 1 -- 172.21.15.46:0/4003261870 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8738010740 con 0x7f87440b12c0 2024-01-26T23:49:49.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.240+0000 7f874ae01700 1 -- 172.21.15.46:0/4003261870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8744132000 con 0x7f87440b12c0 2024-01-26T23:49:49.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.240+0000 7f8740ff9700 1 -- 172.21.15.46:0/4003261870 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8738010b00 con 0x7f87440b12c0 2024-01-26T23:49:49.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.240+0000 7f874ae01700 1 -- 172.21.15.46:0/4003261870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8744134140 con 0x7f87440b12c0 2024-01-26T23:49:49.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.241+0000 7f874ae01700 1 -- 172.21.15.46:0/4003261870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874404c710 con 0x7f87440b12c0 2024-01-26T23:49:49.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.241+0000 7f8740ff9700 1 -- 172.21.15.46:0/4003261870 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 72) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f873801b3b0 con 0x7f87440b12c0 2024-01-26T23:49:49.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.242+0000 7f8740ff9700 1 --2- 172.21.15.46:0/4003261870 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f87280670e0 0x7f8728069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:49.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.242+0000 7f8743fff700 1 -- 172.21.15.46:0/4003261870 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f87280670e0 msgr2=0x7f8728069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:49.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.242+0000 7f8740ff9700 1 -- 172.21.15.46:0/4003261870 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f87380841d0 con 0x7f87440b12c0 2024-01-26T23:49:49.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.242+0000 7f8743fff700 1 --2- 172.21.15.46:0/4003261870 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f87280670e0 0x7f8728069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:49.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.246+0000 7f8740ff9700 1 -- 172.21.15.46:0/4003261870 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8738050df0 con 0x7f87440b12c0 2024-01-26T23:49:49.260 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:49.259+0000 7fd054cba1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:49:49.274 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:49.273+0000 7ffb394e41c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:49:49.350 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:49.348+0000 7ffb394e41c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:49:49.415 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:49.414+0000 7fd054cba1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:49:49.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.443+0000 7f8743fff700 1 -- 172.21.15.46:0/4003261870 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f87280670e0 msgr2=0x7f8728069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:49.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.443+0000 7f8743fff700 1 --2- 172.21.15.46:0/4003261870 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f87280670e0 0x7f8728069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:49.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.503+0000 7f874ae01700 1 -- 172.21.15.46:0/4003261870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f874406f5d0 con 0x7f87440b12c0 2024-01-26T23:49:49.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.504+0000 7f8740ff9700 1 -- 172.21.15.46:0/4003261870 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f8738017090 con 0x7f87440b12c0 2024-01-26T23:49:49.505 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:49.505 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/406559819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":406559819},{"type":"v1","addr":"172.21.15.106:6836","nonce":406559819}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/3387155671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":3387155671},{"type":"v1","addr":"172.21.15.106:6834","nonce":3387155671}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8298,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/855914203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":855914203},{"type":"v1","addr":"172.21.15.46:6834","nonce":855914203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:49:21.360002+0000","modified":"2024-01-26T23:49:24.956671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/1052887453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1052887453},{"type":"v1","addr":"172.21.15.46:6835","nonce":1052887453}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2024-01-26T23:49:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.506+0000 7f874ae01700 1 -- 172.21.15.46:0/4003261870 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f87280670e0 msgr2=0x7f8728069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.506+0000 7f874ae01700 1 --2- 172.21.15.46:0/4003261870 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f87280670e0 0x7f8728069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.506+0000 7f874ae01700 1 -- 172.21.15.46:0/4003261870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87440b12c0 msgr2=0x7f8744133c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.506+0000 7f874ae01700 1 --2- 172.21.15.46:0/4003261870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87440b12c0 0x7f8744133c00 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f873800c7e0 tx=0x7f8738007df0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.507+0000 7f874ae01700 1 -- 172.21.15.46:0/4003261870 shutdown_connections 2024-01-26T23:49:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.507+0000 7f874ae01700 1 --2- 172.21.15.46:0/4003261870 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f87280670e0 0x7f8728069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.507+0000 7f874ae01700 1 --2- 172.21.15.46:0/4003261870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87440b12c0 0x7f8744133c00 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.507+0000 7f874ae01700 1 --2- 172.21.15.46:0/4003261870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8744159b30 0x7f8744132ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.507+0000 7f874ae01700 1 --2- 172.21.15.46:0/4003261870 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8744159760 0x7f8744131210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.507+0000 7f874ae01700 1 -- 172.21.15.46:0/4003261870 >> 172.21.15.46:0/4003261870 conn(0x7f8744069510 msgr2=0x7f87440acb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.507+0000 7f874ae01700 1 -- 172.21.15.46:0/4003261870 shutdown_connections 2024-01-26T23:49:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.507+0000 7f874ae01700 1 -- 172.21.15.46:0/4003261870 wait complete. 2024-01-26T23:49:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 70 2024-01-26T23:49:49.528 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:49:49.536 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:49.535+0000 7ffb394e41c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:49:49.551 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:49.550+0000 7fd054cba1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:49:49.617 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:49.616+0000 7fd054cba1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:49:49.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.659+0000 7fc407f1a700 1 Processor -- start 2024-01-26T23:49:49.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.660+0000 7fc407f1a700 1 -- start start 2024-01-26T23:49:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.661+0000 7fc407f1a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc400159590 0x7fc40015d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.661+0000 7fc407f1a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc40015d8f0 0x7fc40006bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.661+0000 7fc407f1a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40015dd90 con 0x7fc400159960 2024-01-26T23:49:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.661+0000 7fc407f1a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc40015e350 con 0x7fc400159590 2024-01-26T23:49:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.661+0000 7fc407f1a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4000c97c0 con 0x7fc40015d8f0 2024-01-26T23:49:49.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.661+0000 7fc405cb6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc400159590 0x7fc40015d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:49.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.661+0000 7fc405cb6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc400159590 0x7fc40015d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59326/0 (socket says 172.21.15.46:59326) 2024-01-26T23:49:49.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.661+0000 7fc405cb6700 1 -- 172.21.15.46:0/900546379 learned_addr learned my addr 172.21.15.46:0/900546379 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:49.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.661+0000 7fc4054b5700 1 --2- 172.21.15.46:0/900546379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc40015d8f0 0x7fc40006bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:49.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.661+0000 7fc405cb6700 1 -- 172.21.15.46:0/900546379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc40015d8f0 msgr2=0x7fc40006bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:49.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.661+0000 7fc405cb6700 1 --2- 172.21.15.46:0/900546379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc40015d8f0 0x7fc40006bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.661+0000 7fc405cb6700 1 -- 172.21.15.46:0/900546379 >> v1:172.21.15.106:6789/0 conn(0x7fc400159960 legacy=0x7fc40006d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:49.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.661+0000 7fc405cb6700 1 -- 172.21.15.46:0/900546379 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40015e070 con 0x7fc400159590 2024-01-26T23:49:49.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.661+0000 7fc405cb6700 1 --2- 172.21.15.46:0/900546379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc400159590 0x7fc40015d3b0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc0099e0 tx=0x7fc3fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b78b9fd384b85f server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:49.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.662+0000 7fc404cb4700 1 -- 172.21.15.46:0/900546379 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc00b820 con 0x7fc400159590 2024-01-26T23:49:49.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.662+0000 7fc404cb4700 1 -- 172.21.15.46:0/900546379 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3fc00b980 con 0x7fc400159590 2024-01-26T23:49:49.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.662+0000 7fc404cb4700 1 -- 172.21.15.46:0/900546379 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc00bd40 con 0x7fc400159590 2024-01-26T23:49:49.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.662+0000 7fc407f1a700 1 -- 172.21.15.46:0/900546379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc400159590 msgr2=0x7fc40015d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:49.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.662+0000 7fc407f1a700 1 --2- 172.21.15.46:0/900546379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc400159590 0x7fc40015d3b0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc0099e0 tx=0x7fc3fc009250 comp rx=0 tx=0).stop 2024-01-26T23:49:49.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.662+0000 7fc407f1a700 1 -- 172.21.15.46:0/900546379 shutdown_connections 2024-01-26T23:49:49.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.662+0000 7fc407f1a700 1 --2- 172.21.15.46:0/900546379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc400159590 0x7fc40015d3b0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.662+0000 7fc407f1a700 1 --2- 172.21.15.46:0/900546379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc40015d8f0 0x7fc40006bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.662+0000 7fc407f1a700 1 -- 172.21.15.46:0/900546379 >> 172.21.15.46:0/900546379 conn(0x7fc4000ce530 msgr2=0x7fc4000ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:49.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.662+0000 7fc407f1a700 1 -- 172.21.15.46:0/900546379 shutdown_connections 2024-01-26T23:49:49.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.662+0000 7fc407f1a700 1 -- 172.21.15.46:0/900546379 wait complete. 2024-01-26T23:49:49.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.664+0000 7fc407f1a700 1 Processor -- start 2024-01-26T23:49:49.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.664+0000 7fc407f1a700 1 -- start start 2024-01-26T23:49:49.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.664+0000 7fc407f1a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc400159590 0x7fc4000c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:49.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.664+0000 7fc407f1a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc400159960 0x7fc4000c4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:49.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.664+0000 7fc407f1a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc40015d8f0 0x7fc4000c3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:49.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.664+0000 7fc407f1a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40015d2e0 con 0x7fc400159960 2024-01-26T23:49:49.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.664+0000 7fc407f1a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4001522d0 con 0x7fc400159590 2024-01-26T23:49:49.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.664+0000 7fc407f1a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40015d000 con 0x7fc40015d8f0 2024-01-26T23:49:49.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.665+0000 7fc4054b5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc400159960 0x7fc4000c4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:49.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.665+0000 7fc4054b5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc400159960 0x7fc4000c4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59330/0 (socket says 172.21.15.46:59330) 2024-01-26T23:49:49.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.665+0000 7fc4054b5700 1 -- 172.21.15.46:0/2314641400 learned_addr learned my addr 172.21.15.46:0/2314641400 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:49.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.665+0000 7fc405cb6700 1 --2- 172.21.15.46:0/2314641400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc400159590 0x7fc4000c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:49.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.665+0000 7fc4064b7700 1 --2- 172.21.15.46:0/2314641400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc40015d8f0 0x7fc4000c3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:49.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.665+0000 7fc4054b5700 1 -- 172.21.15.46:0/2314641400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc40015d8f0 msgr2=0x7fc4000c3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:49.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.665+0000 7fc4054b5700 1 --2- 172.21.15.46:0/2314641400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc40015d8f0 0x7fc4000c3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.665+0000 7fc4054b5700 1 -- 172.21.15.46:0/2314641400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc400159590 msgr2=0x7fc4000c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:49.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.665+0000 7fc4054b5700 1 --2- 172.21.15.46:0/2314641400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc400159590 0x7fc4000c61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.665+0000 7fc4054b5700 1 -- 172.21.15.46:0/2314641400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3fc0094f0 con 0x7fc400159960 2024-01-26T23:49:49.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.665+0000 7fc4054b5700 1 --2- 172.21.15.46:0/2314641400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc400159960 0x7fc4000c4780 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fc3f000e900 tx=0x7fc3f00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:49.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.665+0000 7fc3f5ffb700 1 -- 172.21.15.46:0/2314641400 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0013680 con 0x7fc400159960 2024-01-26T23:49:49.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.665+0000 7fc3f5ffb700 1 -- 172.21.15.46:0/2314641400 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f00137e0 con 0x7fc400159960 2024-01-26T23:49:49.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.665+0000 7fc3f5ffb700 1 -- 172.21.15.46:0/2314641400 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0013ba0 con 0x7fc400159960 2024-01-26T23:49:49.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.665+0000 7fc407f1a700 1 -- 172.21.15.46:0/2314641400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4000c56e0 con 0x7fc400159960 2024-01-26T23:49:49.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.666+0000 7fc407f1a700 1 -- 172.21.15.46:0/2314641400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4000c5ca0 con 0x7fc400159960 2024-01-26T23:49:49.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.667+0000 7fc407f1a700 1 -- 172.21.15.46:0/2314641400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4000c5540 con 0x7fc400159960 2024-01-26T23:49:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.667+0000 7fc3f5ffb700 1 -- 172.21.15.46:0/2314641400 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 72) v1 ==== 84901+0+0 (secure 0 0 0) 0x7fc3f00123d0 con 0x7fc400159960 2024-01-26T23:49:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.668+0000 7fc3f5ffb700 1 --2- 172.21.15.46:0/2314641400 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc3e40670e0 0x7fc3e4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.668+0000 7fc3f5ffb700 1 -- 172.21.15.46:0/2314641400 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fc3f0084cf0 con 0x7fc400159960 2024-01-26T23:49:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.668+0000 7fc405cb6700 1 -- 172.21.15.46:0/2314641400 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc3e40670e0 msgr2=0x7fc3e4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.668+0000 7fc405cb6700 1 --2- 172.21.15.46:0/2314641400 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc3e40670e0 0x7fc3e4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:49.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.672+0000 7fc3f5ffb700 1 -- 172.21.15.46:0/2314641400 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc4000c5540 con 0x7fc400159960 2024-01-26T23:49:49.696 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:49.695+0000 7ffb394e41c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:49:49.738 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:49.737+0000 7fd054cba1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:49:49.773 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:49.772+0000 7ffb394e41c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:49:49.809 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:49.809+0000 7fd054cba1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:49:49.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.869+0000 7fc405cb6700 1 -- 172.21.15.46:0/2314641400 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc3e40670e0 msgr2=0x7fc3e4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:49.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.869+0000 7fc405cb6700 1 --2- 172.21.15.46:0/2314641400 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc3e40670e0 0x7fc3e4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:49.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.878+0000 7fc407f1a700 1 -- 172.21.15.46:0/2314641400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4000c5540 con 0x7fc400159960 2024-01-26T23:49:49.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.879+0000 7fc3f5ffb700 1 -- 172.21.15.46:0/2314641400 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+16534 (secure 0 0 0) 0x7fc4000c5540 con 0x7fc400159960 2024-01-26T23:49:49.881 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:49.881 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":146,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:49:40.927013+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:49:19.677244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-01-26T23:49:20.330110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/4118930132":"2024-01-27T23:49:40.926975+0000","172.21.15.106:6837/43388":"2024-01-27T23:49:40.926975+0000","172.21.15.106:6835/43388":"2024-01-27T23:49:40.926975+0000","172.21.15.106:0/3470152548":"2024-01-27T23:49:40.926975+0000","172.21.15.106:0/128938556":"2024-01-27T23:49:40.926975+0000","172.21.15.106:0/860908837":"2024-01-27T23:49:40.926975+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:49:49.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.883+0000 7fc407f1a700 1 -- 172.21.15.46:0/2314641400 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc3e40670e0 msgr2=0x7fc3e4069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:49.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.883+0000 7fc407f1a700 1 --2- 172.21.15.46:0/2314641400 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc3e40670e0 0x7fc3e4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.883+0000 7fc407f1a700 1 -- 172.21.15.46:0/2314641400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc400159960 msgr2=0x7fc4000c4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:49.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.883+0000 7fc407f1a700 1 --2- 172.21.15.46:0/2314641400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc400159960 0x7fc4000c4780 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fc3f000e900 tx=0x7fc3f00091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.883+0000 7fc407f1a700 1 -- 172.21.15.46:0/2314641400 shutdown_connections 2024-01-26T23:49:49.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.883+0000 7fc407f1a700 1 --2- 172.21.15.46:0/2314641400 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc3e40670e0 0x7fc3e4069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.883+0000 7fc407f1a700 1 --2- 172.21.15.46:0/2314641400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc40015d8f0 0x7fc4000c3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.883+0000 7fc407f1a700 1 --2- 172.21.15.46:0/2314641400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc400159590 0x7fc4000c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.883+0000 7fc407f1a700 1 --2- 172.21.15.46:0/2314641400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc400159960 0x7fc4000c4780 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:49.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.883+0000 7fc407f1a700 1 -- 172.21.15.46:0/2314641400 >> 172.21.15.46:0/2314641400 conn(0x7fc4000ce530 msgr2=0x7fc400156480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:49.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.883+0000 7fc407f1a700 1 -- 172.21.15.46:0/2314641400 shutdown_connections 2024-01-26T23:49:49.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:49.883+0000 7fc407f1a700 1 -- 172.21.15.46:0/2314641400 wait complete. 2024-01-26T23:49:49.899 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-26T23:49:19.677244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-01-26T23:49:20.330110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '145', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-26T23:49:49.900 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:49:49.915 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:49.914+0000 7ffb394e41c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:49:49.947 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:49.947+0000 7fd054cba1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:49:50.002 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:50.001+0000 7ffb394e41c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:49:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.050+0000 7fbd57dd2700 1 Processor -- start 2024-01-26T23:49:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.050+0000 7fbd57dd2700 1 -- start start 2024-01-26T23:49:50.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.051+0000 7fbd57dd2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd500cc380 0x7fbd500cc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:50.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.051+0000 7fbd57dd2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd50069260 con 0x7fbd501598e0 2024-01-26T23:49:50.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.051+0000 7fbd57dd2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5015d670 con 0x7fbd50159510 2024-01-26T23:49:50.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.051+0000 7fbd57dd2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd500ca930 con 0x7fbd500cc380 2024-01-26T23:49:50.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.051+0000 7fbd5536d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbd501598e0 0x7fbd500c8be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38018/0 (socket says 172.21.15.46:38018) 2024-01-26T23:49:50.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.051+0000 7fbd5536d700 1 -- 172.21.15.46:0/676727531 learned_addr learned my addr 172.21.15.46:0/676727531 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:50.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.051+0000 7fbd5636f700 1 --2- 172.21.15.46:0/676727531 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd500cc380 0x7fbd500cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:50.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.052+0000 7fbd54b6c700 1 -- 172.21.15.46:0/676727531 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544919906 0 0) 0x7fbd50069260 con 0x7fbd501598e0 2024-01-26T23:49:50.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.052+0000 7fbd54b6c700 1 -- 172.21.15.46:0/676727531 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd3c003570 con 0x7fbd501598e0 2024-01-26T23:49:50.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.052+0000 7fbd54b6c700 1 -- 172.21.15.46:0/676727531 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747232005 0 0) 0x7fbd5015d670 con 0x7fbd50159510 2024-01-26T23:49:50.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.052+0000 7fbd54b6c700 1 -- 172.21.15.46:0/676727531 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd50069260 con 0x7fbd50159510 2024-01-26T23:49:50.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.052+0000 7fbd5636f700 1 -- 172.21.15.46:0/676727531 >> v1:172.21.15.106:6789/0 conn(0x7fbd50159510 legacy=0x7fbd500ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:50.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.052+0000 7fbd5636f700 1 -- 172.21.15.46:0/676727531 >> v1:172.21.15.46:6789/0 conn(0x7fbd501598e0 legacy=0x7fbd500c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:50.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.052+0000 7fbd5636f700 1 -- 172.21.15.46:0/676727531 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd500c9750 con 0x7fbd500cc380 2024-01-26T23:49:50.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.052+0000 7fbd5636f700 1 --2- 172.21.15.46:0/676727531 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd500cc380 0x7fbd500cc800 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbd48009e90 tx=0x7fbd48009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f41146b56434b49 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:50.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.052+0000 7fbd54b6c700 1 -- 172.21.15.46:0/676727531 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4800e070 con 0x7fbd500cc380 2024-01-26T23:49:50.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.052+0000 7fbd54b6c700 1 -- 172.21.15.46:0/676727531 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4800bb80 con 0x7fbd500cc380 2024-01-26T23:49:50.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.052+0000 7fbd54b6c700 1 -- 172.21.15.46:0/676727531 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd48012430 con 0x7fbd500cc380 2024-01-26T23:49:50.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.053+0000 7fbd57dd2700 1 -- 172.21.15.46:0/676727531 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd500cc380 msgr2=0x7fbd500cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:50.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.053+0000 7fbd57dd2700 1 --2- 172.21.15.46:0/676727531 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd500cc380 0x7fbd500cc800 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbd48009e90 tx=0x7fbd48009780 comp rx=0 tx=0).stop 2024-01-26T23:49:50.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.053+0000 7fbd57dd2700 1 -- 172.21.15.46:0/676727531 shutdown_connections 2024-01-26T23:49:50.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.053+0000 7fbd57dd2700 1 --2- 172.21.15.46:0/676727531 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd500cc380 0x7fbd500cc800 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.053+0000 7fbd57dd2700 1 -- 172.21.15.46:0/676727531 >> 172.21.15.46:0/676727531 conn(0x7fbd5006bd90 msgr2=0x7fbd5006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:50.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.053+0000 7fbd57dd2700 1 -- 172.21.15.46:0/676727531 shutdown_connections 2024-01-26T23:49:50.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.053+0000 7fbd57dd2700 1 -- 172.21.15.46:0/676727531 wait complete. 2024-01-26T23:49:50.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.054+0000 7fbd57dd2700 1 Processor -- start 2024-01-26T23:49:50.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.054+0000 7fbd57dd2700 1 -- start start 2024-01-26T23:49:50.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.055+0000 7fbd57dd2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd500cc380 0x7fbd50157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:50.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.055+0000 7fbd57dd2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd50159510 0x7fbd5015af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:50.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.055+0000 7fbd57dd2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd501598e0 0x7fbd5015b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:50.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.055+0000 7fbd57dd2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd50159d50 con 0x7fbd500cc380 2024-01-26T23:49:50.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.055+0000 7fbd57dd2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5015ab10 con 0x7fbd501598e0 2024-01-26T23:49:50.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.055+0000 7fbd57dd2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd50159e90 con 0x7fbd50159510 2024-01-26T23:49:50.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.055+0000 7fbd55b6e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd500cc380 0x7fbd50157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:50.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.055+0000 7fbd5536d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd50159510 0x7fbd5015af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:50.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.055+0000 7fbd55b6e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd500cc380 0x7fbd50157db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59340/0 (socket says 172.21.15.46:59340) 2024-01-26T23:49:50.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.055+0000 7fbd55b6e700 1 -- 172.21.15.46:0/690368164 learned_addr learned my addr 172.21.15.46:0/690368164 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:50.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.055+0000 7fbd5636f700 1 --2- 172.21.15.46:0/690368164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd501598e0 0x7fbd5015b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:50.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.056+0000 7fbd5536d700 1 -- 172.21.15.46:0/690368164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd501598e0 msgr2=0x7fbd5015b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:50.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.056+0000 7fbd5536d700 1 --2- 172.21.15.46:0/690368164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd501598e0 0x7fbd5015b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.056+0000 7fbd5536d700 1 -- 172.21.15.46:0/690368164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd500cc380 msgr2=0x7fbd50157db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:50.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.056+0000 7fbd5536d700 1 --2- 172.21.15.46:0/690368164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd500cc380 0x7fbd50157db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.056+0000 7fbd5536d700 1 -- 172.21.15.46:0/690368164 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd48009a20 con 0x7fbd50159510 2024-01-26T23:49:50.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.056+0000 7fbd5536d700 1 --2- 172.21.15.46:0/690368164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd50159510 0x7fbd5015af30 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000c560 tx=0x7fbd40007d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:50.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.056+0000 7fbd45ffb700 1 -- 172.21.15.46:0/690368164 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd40005b30 con 0x7fbd50159510 2024-01-26T23:49:50.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.056+0000 7fbd45ffb700 1 -- 172.21.15.46:0/690368164 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd40005cb0 con 0x7fbd50159510 2024-01-26T23:49:50.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.056+0000 7fbd57dd2700 1 -- 172.21.15.46:0/690368164 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd5015a170 con 0x7fbd50159510 2024-01-26T23:49:50.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.056+0000 7fbd45ffb700 1 -- 172.21.15.46:0/690368164 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd40007320 con 0x7fbd50159510 2024-01-26T23:49:50.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.056+0000 7fbd57dd2700 1 -- 172.21.15.46:0/690368164 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd5015a730 con 0x7fbd50159510 2024-01-26T23:49:50.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.058+0000 7fbd45ffb700 1 -- 172.21.15.46:0/690368164 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 72) v1 ==== 84901+0+0 (secure 0 0 0) 0x7fbd4001b3b0 con 0x7fbd50159510 2024-01-26T23:49:50.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.058+0000 7fbd57dd2700 1 -- 172.21.15.46:0/690368164 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5015ac50 con 0x7fbd50159510 2024-01-26T23:49:50.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.058+0000 7fbd45ffb700 1 --2- 172.21.15.46:0/690368164 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbd340670e0 0x7fbd34069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:50.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.058+0000 7fbd45ffb700 1 -- 172.21.15.46:0/690368164 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fbd40083f60 con 0x7fbd50159510 2024-01-26T23:49:50.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.058+0000 7fbd55b6e700 1 -- 172.21.15.46:0/690368164 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbd340670e0 msgr2=0x7fbd34069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:50.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.058+0000 7fbd55b6e700 1 --2- 172.21.15.46:0/690368164 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbd340670e0 0x7fbd34069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:50.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.062+0000 7fbd45ffb700 1 -- 172.21.15.46:0/690368164 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbd5015ac50 con 0x7fbd50159510 2024-01-26T23:49:50.074 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:50.074+0000 7fd054cba1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:49:50.178 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:50.177+0000 7ffb394e41c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:49:50.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.259+0000 7fbd55b6e700 1 -- 172.21.15.46:0/690368164 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbd340670e0 msgr2=0x7fbd34069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:50.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.259+0000 7fbd55b6e700 1 --2- 172.21.15.46:0/690368164 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbd340670e0 0x7fbd34069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:50.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.315+0000 7fbd57dd2700 1 -- 172.21.15.46:0/690368164 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbd5015ac50 con 0x7fbd50159510 2024-01-26T23:49:50.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.315+0000 7fbd45ffb700 1 -- 172.21.15.46:0/690368164 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v146) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbd5015ac50 con 0x7fbd50159510 2024-01-26T23:49:50.317 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:49:50.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.318+0000 7fbd57dd2700 1 -- 172.21.15.46:0/690368164 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbd340670e0 msgr2=0x7fbd34069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:50.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.318+0000 7fbd57dd2700 1 --2- 172.21.15.46:0/690368164 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbd340670e0 0x7fbd34069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.318+0000 7fbd57dd2700 1 -- 172.21.15.46:0/690368164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd50159510 msgr2=0x7fbd5015af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:50.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.318+0000 7fbd57dd2700 1 --2- 172.21.15.46:0/690368164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd50159510 0x7fbd5015af30 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000c560 tx=0x7fbd40007d70 comp rx=0 tx=0).stop 2024-01-26T23:49:50.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.318+0000 7fbd57dd2700 1 -- 172.21.15.46:0/690368164 shutdown_connections 2024-01-26T23:49:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.318+0000 7fbd57dd2700 1 --2- 172.21.15.46:0/690368164 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbd340670e0 0x7fbd34069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.318+0000 7fbd57dd2700 1 --2- 172.21.15.46:0/690368164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd500cc380 0x7fbd50157db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.318+0000 7fbd57dd2700 1 --2- 172.21.15.46:0/690368164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd501598e0 0x7fbd5015b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.318+0000 7fbd57dd2700 1 --2- 172.21.15.46:0/690368164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd50159510 0x7fbd5015af30 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.318+0000 7fbd57dd2700 1 -- 172.21.15.46:0/690368164 >> 172.21.15.46:0/690368164 conn(0x7fbd5006bd90 msgr2=0x7fbd50154e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.318+0000 7fbd57dd2700 1 -- 172.21.15.46:0/690368164 shutdown_connections 2024-01-26T23:49:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.318+0000 7fbd57dd2700 1 -- 172.21.15.46:0/690368164 wait complete. 2024-01-26T23:49:50.321 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:50.321+0000 7ffb394e41c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:49:50.338 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-26T23:49:50.387 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:50.386+0000 7fd054cba1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:49:50.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.473+0000 7f64e4944700 1 Processor -- start 2024-01-26T23:49:50.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.473+0000 7f64e4944700 1 -- start start 2024-01-26T23:49:50.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.474+0000 7f64e4944700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e00c8da0 0x7f64e00c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:50.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.474+0000 7f64e4944700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f64e0140af0 0x7f64e0140f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.474+0000 7f64e4944700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e00ccc00 con 0x7f64e00caf30 2024-01-26T23:49:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.474+0000 7f64e4944700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e00cbbf0 con 0x7f64e0140af0 2024-01-26T23:49:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.474+0000 7f64e4944700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e00cc310 con 0x7f64e00c8da0 2024-01-26T23:49:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.475+0000 7f64ddd9b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f64e0140af0 0x7f64e0140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.474+0000 7f64ded9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f64e00caf30 0x7f64e00cb370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58750/0 (socket says 172.21.15.46:58750) 2024-01-26T23:49:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.475+0000 7f64de59c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e00c8da0 0x7f64e00c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.475+0000 7f64ded9d700 1 -- 172.21.15.46:0/471299258 learned_addr learned my addr 172.21.15.46:0/471299258 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.475+0000 7f64ddd9b700 1 -- 172.21.15.46:0/471299258 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e00c8da0 msgr2=0x7f64e00c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.475+0000 7f64dd59a700 1 -- 172.21.15.46:0/471299258 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343530134 0 0) 0x7f64e00ccc00 con 0x7f64e00caf30 2024-01-26T23:49:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.475+0000 7f64ddd9b700 1 --2- 172.21.15.46:0/471299258 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e00c8da0 0x7f64e00c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.475+0000 7f64ddd9b700 1 -- 172.21.15.46:0/471299258 >> v1:172.21.15.46:6789/0 conn(0x7f64e00caf30 legacy=0x7f64e00cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.475+0000 7f64ddd9b700 1 -- 172.21.15.46:0/471299258 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e00cc5b0 con 0x7f64e0140af0 2024-01-26T23:49:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.475+0000 7f64ddd9b700 1 --2- 172.21.15.46:0/471299258 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e0140af0 0x7f64e0140f60 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f64d40098e0 tx=0x7f64d40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e81473df8d3b2dd server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.475+0000 7f64dd59a700 1 -- 172.21.15.46:0/471299258 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d400b640 con 0x7f64e0140af0 2024-01-26T23:49:50.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.475+0000 7f64dd59a700 1 -- 172.21.15.46:0/471299258 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d400b7a0 con 0x7f64e0140af0 2024-01-26T23:49:50.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.475+0000 7f64dd59a700 1 -- 172.21.15.46:0/471299258 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d400bb40 con 0x7f64e0140af0 2024-01-26T23:49:50.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.476+0000 7f64e4944700 1 -- 172.21.15.46:0/471299258 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e0140af0 msgr2=0x7f64e0140f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:50.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.476+0000 7f64e4944700 1 --2- 172.21.15.46:0/471299258 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e0140af0 0x7f64e0140f60 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f64d40098e0 tx=0x7f64d40091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.476+0000 7f64e4944700 1 -- 172.21.15.46:0/471299258 shutdown_connections 2024-01-26T23:49:50.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.476+0000 7f64e4944700 1 --2- 172.21.15.46:0/471299258 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e00c8da0 0x7f64e00c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.476+0000 7f64e4944700 1 --2- 172.21.15.46:0/471299258 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e0140af0 0x7f64e0140f60 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.476+0000 7f64e4944700 1 -- 172.21.15.46:0/471299258 >> 172.21.15.46:0/471299258 conn(0x7f64e006bd90 msgr2=0x7f64e006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:50.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.476+0000 7f64e4944700 1 -- 172.21.15.46:0/471299258 shutdown_connections 2024-01-26T23:49:50.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.476+0000 7f64e4944700 1 -- 172.21.15.46:0/471299258 wait complete. 2024-01-26T23:49:50.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.477+0000 7f64e4944700 1 Processor -- start 2024-01-26T23:49:50.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.477+0000 7f64e4944700 1 -- start start 2024-01-26T23:49:50.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.478+0000 7f64e4944700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e00c8da0 0x7f64e012bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:50.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.478+0000 7f64e4944700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64e00caf30 0x7f64e00c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:50.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.478+0000 7f64e4944700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e0140af0 0x7f64e015b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:50.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.478+0000 7f64e4944700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e00c5b90 con 0x7f64e00caf30 2024-01-26T23:49:50.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.478+0000 7f64e4944700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e00c5dc0 con 0x7f64e0140af0 2024-01-26T23:49:50.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.478+0000 7f64e4944700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e00c60a0 con 0x7f64e00c8da0 2024-01-26T23:49:50.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.478+0000 7f64ddd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64e00caf30 0x7f64e00c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:50.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.478+0000 7f64ddd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64e00caf30 0x7f64e00c5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54264/0 (socket says 172.21.15.46:54264) 2024-01-26T23:49:50.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.478+0000 7f64ddd9b700 1 -- 172.21.15.46:0/4060724496 learned_addr learned my addr 172.21.15.46:0/4060724496 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:50.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.478+0000 7f64de59c700 1 --2- 172.21.15.46:0/4060724496 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e00c8da0 0x7f64e012bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:50.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.478+0000 7f64ded9d700 1 --2- 172.21.15.46:0/4060724496 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e0140af0 0x7f64e015b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:50.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.479+0000 7f64de59c700 1 -- 172.21.15.46:0/4060724496 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e0140af0 msgr2=0x7f64e015b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:50.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.479+0000 7f64de59c700 1 --2- 172.21.15.46:0/4060724496 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e0140af0 0x7f64e015b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.479+0000 7f64de59c700 1 -- 172.21.15.46:0/4060724496 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64e00caf30 msgr2=0x7f64e00c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:50.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.479+0000 7f64de59c700 1 --2- 172.21.15.46:0/4060724496 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64e00caf30 0x7f64e00c5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.479+0000 7f64de59c700 1 -- 172.21.15.46:0/4060724496 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d4009420 con 0x7f64e00c8da0 2024-01-26T23:49:50.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.479+0000 7f64ddd9b700 1 --2- 172.21.15.46:0/4060724496 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64e00caf30 0x7f64e00c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:50.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.479+0000 7f64de59c700 1 --2- 172.21.15.46:0/4060724496 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e00c8da0 0x7f64e012bed0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f64d000ea50 tx=0x7f64d00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:50.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.479+0000 7f64ce7fc700 1 -- 172.21.15.46:0/4060724496 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d000ee90 con 0x7f64e00c8da0 2024-01-26T23:49:50.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.479+0000 7f64ce7fc700 1 -- 172.21.15.46:0/4060724496 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d000c7f0 con 0x7f64e00c8da0 2024-01-26T23:49:50.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.479+0000 7f64e4944700 1 -- 172.21.15.46:0/4060724496 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e00c6380 con 0x7f64e00c8da0 2024-01-26T23:49:50.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.479+0000 7f64ce7fc700 1 -- 172.21.15.46:0/4060724496 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d000cb90 con 0x7f64e00c8da0 2024-01-26T23:49:50.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.479+0000 7f64e4944700 1 -- 172.21.15.46:0/4060724496 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e00c7680 con 0x7f64e00c8da0 2024-01-26T23:49:50.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.480+0000 7f64e4944700 1 -- 172.21.15.46:0/4060724496 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e00c78a0 con 0x7f64e00c8da0 2024-01-26T23:49:50.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.481+0000 7f64ce7fc700 1 -- 172.21.15.46:0/4060724496 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 72) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f64d0013470 con 0x7f64e00c8da0 2024-01-26T23:49:50.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.481+0000 7f64ce7fc700 1 --2- 172.21.15.46:0/4060724496 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f64c00670e0 0x7f64c0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:50.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.481+0000 7f64ddd9b700 1 -- 172.21.15.46:0/4060724496 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f64c00670e0 msgr2=0x7f64c0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:50.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.481+0000 7f64ce7fc700 1 -- 172.21.15.46:0/4060724496 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f64d0084dc0 con 0x7f64e00c8da0 2024-01-26T23:49:50.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.481+0000 7f64ddd9b700 1 --2- 172.21.15.46:0/4060724496 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f64c00670e0 0x7f64c0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:50.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.486+0000 7f64ce7fc700 1 -- 172.21.15.46:0/4060724496 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f64d00519e0 con 0x7f64e00c8da0 2024-01-26T23:49:50.600 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:50.599+0000 7fd054cba1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:49:50.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.682+0000 7f64ddd9b700 1 -- 172.21.15.46:0/4060724496 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f64c00670e0 msgr2=0x7f64c0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:50.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.682+0000 7f64ddd9b700 1 --2- 172.21.15.46:0/4060724496 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f64c00670e0 0x7f64c0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:50.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.691+0000 7f64e4944700 1 -- 172.21.15.46:0/4060724496 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f64e004c710 con 0x7f64e00c8da0 2024-01-26T23:49:50.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.691+0000 7f64ce7fc700 1 -- 172.21.15.46:0/4060724496 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v146) v1 ==== 104+0+11 (secure 0 0 0) 0x7f64d0055000 con 0x7f64e00c8da0 2024-01-26T23:49:50.693 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-26T23:49:50.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.694+0000 7f64e4944700 1 -- 172.21.15.46:0/4060724496 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f64c00670e0 msgr2=0x7f64c0069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:50.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.694+0000 7f64e4944700 1 --2- 172.21.15.46:0/4060724496 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f64c00670e0 0x7f64c0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.694+0000 7f64e4944700 1 -- 172.21.15.46:0/4060724496 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e00c8da0 msgr2=0x7f64e012bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:50.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.694+0000 7f64e4944700 1 --2- 172.21.15.46:0/4060724496 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e00c8da0 0x7f64e012bed0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f64d000ea50 tx=0x7f64d00091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.694+0000 7f64e4944700 1 -- 172.21.15.46:0/4060724496 shutdown_connections 2024-01-26T23:49:50.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.694+0000 7f64e4944700 1 --2- 172.21.15.46:0/4060724496 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f64c00670e0 0x7f64c0069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.694+0000 7f64e4944700 1 --2- 172.21.15.46:0/4060724496 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e0140af0 0x7f64e015b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.694+0000 7f64e4944700 1 --2- 172.21.15.46:0/4060724496 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64e00c8da0 0x7f64e012bed0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.694+0000 7f64e4944700 1 --2- 172.21.15.46:0/4060724496 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64e00caf30 0x7f64e00c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.694+0000 7f64e4944700 1 -- 172.21.15.46:0/4060724496 >> 172.21.15.46:0/4060724496 conn(0x7f64e006bd90 msgr2=0x7f64e013fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:50.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.694+0000 7f64e4944700 1 -- 172.21.15.46:0/4060724496 shutdown_connections 2024-01-26T23:49:50.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.694+0000 7f64e4944700 1 -- 172.21.15.46:0/4060724496 wait complete. 2024-01-26T23:49:50.697 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:50.696+0000 7ffb394e41c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:49:50.709 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-26T23:49:50.825 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:50.824+0000 7fd054cba1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:49:50.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.845+0000 7fc41ecd8700 1 Processor -- start 2024-01-26T23:49:50.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.846+0000 7fc41ecd8700 1 -- start start 2024-01-26T23:49:50.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.846+0000 7fc41ecd8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc418159da0 0x7fc4180ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:50.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.846+0000 7fc41ecd8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41815a200 con 0x7fc4181599d0 2024-01-26T23:49:50.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.846+0000 7fc41ecd8700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4180b1a60 con 0x7fc41815d8c0 2024-01-26T23:49:50.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.846+0000 7fc41ecd8700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4180b11c0 con 0x7fc418159da0 2024-01-26T23:49:50.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.847+0000 7fc417fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc418159da0 0x7fc4180ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:50.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.847+0000 7fc41ca74700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc4181599d0 0x7fc4180b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58756/0 (socket says 172.21.15.46:58756) 2024-01-26T23:49:50.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.847+0000 7fc41ca74700 1 -- 172.21.15.46:0/2838386544 learned_addr learned my addr 172.21.15.46:0/2838386544 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:50.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.847+0000 7fc4177fe700 1 -- 172.21.15.46:0/2838386544 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136398212 0 0) 0x7fc41815a200 con 0x7fc4181599d0 2024-01-26T23:49:50.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.847+0000 7fc4177fe700 1 -- 172.21.15.46:0/2838386544 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc400003570 con 0x7fc4181599d0 2024-01-26T23:49:50.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.847+0000 7fc417fff700 1 -- 172.21.15.46:0/2838386544 >> v1:172.21.15.106:6789/0 conn(0x7fc41815d8c0 legacy=0x7fc4180b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:50.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.847+0000 7fc4177fe700 1 -- 172.21.15.46:0/2838386544 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153127176 0 0) 0x7fc4180b1a60 con 0x7fc41815d8c0 2024-01-26T23:49:50.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.847+0000 7fc417fff700 1 -- 172.21.15.46:0/2838386544 >> v1:172.21.15.46:6789/0 conn(0x7fc4181599d0 legacy=0x7fc4180b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:50.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.847+0000 7fc417fff700 1 -- 172.21.15.46:0/2838386544 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4180b14a0 con 0x7fc418159da0 2024-01-26T23:49:50.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.847+0000 7fc417fff700 1 --2- 172.21.15.46:0/2838386544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc418159da0 0x7fc4180ab1c0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fc40c0098e0 tx=0x7fc40c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155257b44caf0f33 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:50.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.847+0000 7fc4177fe700 1 -- 172.21.15.46:0/2838386544 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c00b640 con 0x7fc418159da0 2024-01-26T23:49:50.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.848+0000 7fc4177fe700 1 -- 172.21.15.46:0/2838386544 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40c00b7a0 con 0x7fc418159da0 2024-01-26T23:49:50.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.848+0000 7fc4177fe700 1 -- 172.21.15.46:0/2838386544 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c00bb40 con 0x7fc418159da0 2024-01-26T23:49:50.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.848+0000 7fc41ecd8700 1 -- 172.21.15.46:0/2838386544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc418159da0 msgr2=0x7fc4180ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:50.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.848+0000 7fc41ecd8700 1 --2- 172.21.15.46:0/2838386544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc418159da0 0x7fc4180ab1c0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fc40c0098e0 tx=0x7fc40c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.848+0000 7fc41ecd8700 1 -- 172.21.15.46:0/2838386544 shutdown_connections 2024-01-26T23:49:50.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.848+0000 7fc41ecd8700 1 --2- 172.21.15.46:0/2838386544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc418159da0 0x7fc4180ab1c0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.848+0000 7fc41ecd8700 1 -- 172.21.15.46:0/2838386544 >> 172.21.15.46:0/2838386544 conn(0x7fc41806f840 msgr2=0x7fc418087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:50.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.848+0000 7fc41ecd8700 1 -- 172.21.15.46:0/2838386544 shutdown_connections 2024-01-26T23:49:50.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.848+0000 7fc41ecd8700 1 -- 172.21.15.46:0/2838386544 wait complete. 2024-01-26T23:49:50.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.850+0000 7fc41ecd8700 1 Processor -- start 2024-01-26T23:49:50.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.850+0000 7fc41ecd8700 1 -- start start 2024-01-26T23:49:50.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.851+0000 7fc41ecd8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4181599d0 0x7fc41815bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:50.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.851+0000 7fc41ecd8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc418159da0 0x7fc41815fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:50.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.851+0000 7fc41ecd8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc41815d8c0 0x7fc41809d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:50.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.851+0000 7fc41ecd8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41815ced0 con 0x7fc4181599d0 2024-01-26T23:49:50.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.851+0000 7fc41ecd8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41815c040 con 0x7fc418159da0 2024-01-26T23:49:50.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.851+0000 7fc41ecd8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41815c2c0 con 0x7fc41815d8c0 2024-01-26T23:49:50.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.851+0000 7fc41ca74700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4181599d0 0x7fc41815bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:50.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.851+0000 7fc41ca74700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4181599d0 0x7fc41815bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54286/0 (socket says 172.21.15.46:54286) 2024-01-26T23:49:50.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.851+0000 7fc41ca74700 1 -- 172.21.15.46:0/3252777043 learned_addr learned my addr 172.21.15.46:0/3252777043 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.851+0000 7fc417fff700 1 --2- 172.21.15.46:0/3252777043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc418159da0 0x7fc41815fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.851+0000 7fc41d275700 1 --2- 172.21.15.46:0/3252777043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc41815d8c0 0x7fc41809d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.851+0000 7fc41ca74700 1 -- 172.21.15.46:0/3252777043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc41815d8c0 msgr2=0x7fc41809d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.851+0000 7fc41ca74700 1 --2- 172.21.15.46:0/3252777043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc41815d8c0 0x7fc41809d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.851+0000 7fc41ca74700 1 -- 172.21.15.46:0/3252777043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc418159da0 msgr2=0x7fc41815fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.851+0000 7fc41ca74700 1 --2- 172.21.15.46:0/3252777043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc418159da0 0x7fc41815fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.851+0000 7fc41ca74700 1 -- 172.21.15.46:0/3252777043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40c009420 con 0x7fc4181599d0 2024-01-26T23:49:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.851+0000 7fc41ca74700 1 --2- 172.21.15.46:0/3252777043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4181599d0 0x7fc41815bb00 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fc40800cb10 tx=0x7fc4080027e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.852+0000 7fc414ff9700 1 -- 172.21.15.46:0/3252777043 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40800ce20 con 0x7fc4181599d0 2024-01-26T23:49:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.852+0000 7fc414ff9700 1 -- 172.21.15.46:0/3252777043 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc408005d80 con 0x7fc4181599d0 2024-01-26T23:49:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.852+0000 7fc41ecd8700 1 -- 172.21.15.46:0/3252777043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc41815c5a0 con 0x7fc4181599d0 2024-01-26T23:49:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.852+0000 7fc414ff9700 1 -- 172.21.15.46:0/3252777043 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40800a820 con 0x7fc4181599d0 2024-01-26T23:49:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.852+0000 7fc41ecd8700 1 -- 172.21.15.46:0/3252777043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41815cb60 con 0x7fc4181599d0 2024-01-26T23:49:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.853+0000 7fc41ecd8700 1 -- 172.21.15.46:0/3252777043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41815d080 con 0x7fc4181599d0 2024-01-26T23:49:50.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.853+0000 7fc414ff9700 1 -- 172.21.15.46:0/3252777043 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 72) v1 ==== 84901+0+0 (secure 0 0 0) 0x7fc40800ac10 con 0x7fc4181599d0 2024-01-26T23:49:50.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.854+0000 7fc414ff9700 1 --2- 172.21.15.46:0/3252777043 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc4040670e0 0x7fc404069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:50.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.854+0000 7fc414ff9700 1 -- 172.21.15.46:0/3252777043 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fc408084930 con 0x7fc4181599d0 2024-01-26T23:49:50.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.854+0000 7fc417fff700 1 -- 172.21.15.46:0/3252777043 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc4040670e0 msgr2=0x7fc404069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:50.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.854+0000 7fc417fff700 1 --2- 172.21.15.46:0/3252777043 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc4040670e0 0x7fc404069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:50.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:50.858+0000 7fc414ff9700 1 -- 172.21.15.46:0/3252777043 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc4080515d0 con 0x7fc4181599d0 2024-01-26T23:49:50.897 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:50.896+0000 7fd054cba1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:49:50.949 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:50.948+0000 7ffb394e41c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:49:51.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.055+0000 7fc417fff700 1 -- 172.21.15.46:0/3252777043 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc4040670e0 msgr2=0x7fc404069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:51.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.055+0000 7fc417fff700 1 --2- 172.21.15.46:0/3252777043 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc4040670e0 0x7fc404069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:51.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.063+0000 7fc41ecd8700 1 -- 172.21.15.46:0/3252777043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc41804c710 con 0x7fc4181599d0 2024-01-26T23:49:51.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.064+0000 7fc414ff9700 1 -- 172.21.15.46:0/3252777043 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v146) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc408054bf0 con 0x7fc4181599d0 2024-01-26T23:49:51.065 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-26T23:49:51.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.066+0000 7fc41ecd8700 1 -- 172.21.15.46:0/3252777043 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc4040670e0 msgr2=0x7fc404069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:51.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.066+0000 7fc41ecd8700 1 --2- 172.21.15.46:0/3252777043 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc4040670e0 0x7fc404069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.066+0000 7fc41ecd8700 1 -- 172.21.15.46:0/3252777043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4181599d0 msgr2=0x7fc41815bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:51.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.066+0000 7fc41ecd8700 1 --2- 172.21.15.46:0/3252777043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4181599d0 0x7fc41815bb00 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fc40800cb10 tx=0x7fc4080027e0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.066+0000 7fc41ecd8700 1 -- 172.21.15.46:0/3252777043 shutdown_connections 2024-01-26T23:49:51.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.066+0000 7fc41ecd8700 1 --2- 172.21.15.46:0/3252777043 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc4040670e0 0x7fc404069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.066+0000 7fc41ecd8700 1 --2- 172.21.15.46:0/3252777043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4181599d0 0x7fc41815bb00 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.066+0000 7fc41ecd8700 1 --2- 172.21.15.46:0/3252777043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc41815d8c0 0x7fc41809d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.066+0000 7fc41ecd8700 1 --2- 172.21.15.46:0/3252777043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc418159da0 0x7fc41815fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.066+0000 7fc41ecd8700 1 -- 172.21.15.46:0/3252777043 >> 172.21.15.46:0/3252777043 conn(0x7fc41806f840 msgr2=0x7fc4180b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:51.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.066+0000 7fc41ecd8700 1 -- 172.21.15.46:0/3252777043 shutdown_connections 2024-01-26T23:49:51.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.066+0000 7fc41ecd8700 1 -- 172.21.15.46:0/3252777043 wait complete. 2024-01-26T23:49:51.083 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:49:51.214 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:51.213+0000 7ffb394e41c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:49:51.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.215+0000 7f80ae70c700 1 Processor -- start 2024-01-26T23:49:51.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.216+0000 7f80ae70c700 1 -- start start 2024-01-26T23:49:51.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.216+0000 7f80ae70c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f80a815ce40 0x7f80a80d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:51.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.216+0000 7f80ae70c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80a80cc1c0 0x7f80a80ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:51.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.216+0000 7f80ae70c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a80cff70 con 0x7f80a815d210 2024-01-26T23:49:51.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.216+0000 7f80ae70c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a80cf400 con 0x7f80a815ce40 2024-01-26T23:49:51.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.216+0000 7f80ae70c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a80cf660 con 0x7f80a80cc1c0 2024-01-26T23:49:51.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.217+0000 7f80a77fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f80a815d210 0x7f80a80cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58770/0 (socket says 172.21.15.46:58770) 2024-01-26T23:49:51.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.217+0000 7f80a77fe700 1 -- 172.21.15.46:0/4026476308 learned_addr learned my addr 172.21.15.46:0/4026476308 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:51.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.217+0000 7f80a7fff700 1 --2- 172.21.15.46:0/4026476308 >> v2:172.21.15.106:3300/0 conn(0x7f80a815ce40 0x7f80a80d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:51.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.217+0000 7f80acca9700 1 --2- 172.21.15.46:0/4026476308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80a80cc1c0 0x7f80a80ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:51.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.217+0000 7f80a6ffd700 1 -- 172.21.15.46:0/4026476308 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697924311 0 0) 0x7f80a80cff70 con 0x7f80a815d210 2024-01-26T23:49:51.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.217+0000 7f80a6ffd700 1 -- 172.21.15.46:0/4026476308 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8090003570 con 0x7f80a815d210 2024-01-26T23:49:51.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.217+0000 7f80acca9700 1 -- 172.21.15.46:0/4026476308 >> v2:172.21.15.106:3300/0 conn(0x7f80a815ce40 msgr2=0x7f80a80d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:51.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.217+0000 7f80acca9700 1 --2- 172.21.15.46:0/4026476308 >> v2:172.21.15.106:3300/0 conn(0x7f80a815ce40 0x7f80a80d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.217+0000 7f80acca9700 1 -- 172.21.15.46:0/4026476308 >> v1:172.21.15.46:6789/0 conn(0x7f80a815d210 legacy=0x7f80a80cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:51.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.217+0000 7f80acca9700 1 -- 172.21.15.46:0/4026476308 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a80d0500 con 0x7f80a80cc1c0 2024-01-26T23:49:51.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.217+0000 7f80a7fff700 1 --2- 172.21.15.46:0/4026476308 >> v2:172.21.15.106:3300/0 conn(0x7f80a815ce40 0x7f80a80d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:51.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.217+0000 7f80acca9700 1 --2- 172.21.15.46:0/4026476308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80a80cc1c0 0x7f80a80ade30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009e60 tx=0x7f80a0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55ba0755f9887f64 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:51.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.217+0000 7f80a6ffd700 1 -- 172.21.15.46:0/4026476308 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a000e070 con 0x7f80a80cc1c0 2024-01-26T23:49:51.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.218+0000 7f80a6ffd700 1 -- 172.21.15.46:0/4026476308 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a000bab0 con 0x7f80a80cc1c0 2024-01-26T23:49:51.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.218+0000 7f80a6ffd700 1 -- 172.21.15.46:0/4026476308 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a000be70 con 0x7f80a80cc1c0 2024-01-26T23:49:51.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.218+0000 7f80ae70c700 1 -- 172.21.15.46:0/4026476308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80a80cc1c0 msgr2=0x7f80a80ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:51.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.218+0000 7f80ae70c700 1 --2- 172.21.15.46:0/4026476308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80a80cc1c0 0x7f80a80ade30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009e60 tx=0x7f80a0009780 comp rx=0 tx=0).stop 2024-01-26T23:49:51.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.218+0000 7f80ae70c700 1 -- 172.21.15.46:0/4026476308 shutdown_connections 2024-01-26T23:49:51.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.218+0000 7f80ae70c700 1 --2- 172.21.15.46:0/4026476308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80a80cc1c0 0x7f80a80ade30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.218+0000 7f80ae70c700 1 --2- 172.21.15.46:0/4026476308 >> v2:172.21.15.106:3300/0 conn(0x7f80a815ce40 0x7f80a80d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.218+0000 7f80ae70c700 1 -- 172.21.15.46:0/4026476308 >> 172.21.15.46:0/4026476308 conn(0x7f80a806f840 msgr2=0x7f80a80d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:51.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.218+0000 7f80ae70c700 1 -- 172.21.15.46:0/4026476308 shutdown_connections 2024-01-26T23:49:51.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.218+0000 7f80ae70c700 1 -- 172.21.15.46:0/4026476308 wait complete. 2024-01-26T23:49:51.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.220+0000 7f80ae70c700 1 Processor -- start 2024-01-26T23:49:51.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.220+0000 7f80ae70c700 1 -- start start 2024-01-26T23:49:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.221+0000 7f80ae70c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80a80cc1c0 0x7f80a815fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.221+0000 7f80ae70c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a815ce40 0x7f80a809d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.221+0000 7f80ae70c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80a815d210 0x7f80a816a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.221+0000 7f80ae70c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a809dbd0 con 0x7f80a815ce40 2024-01-26T23:49:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.221+0000 7f80ae70c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a815f730 con 0x7f80a80cc1c0 2024-01-26T23:49:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.221+0000 7f80ae70c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a816adb0 con 0x7f80a815d210 2024-01-26T23:49:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.221+0000 7f80a77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a815ce40 0x7f80a809d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.221+0000 7f80acca9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80a815d210 0x7f80a816a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.221+0000 7f80a7fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80a80cc1c0 0x7f80a815fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.221+0000 7f80a77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a815ce40 0x7f80a809d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54290/0 (socket says 172.21.15.46:54290) 2024-01-26T23:49:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.221+0000 7f80a77fe700 1 -- 172.21.15.46:0/1611024792 learned_addr learned my addr 172.21.15.46:0/1611024792 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.221+0000 7f80acca9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80a815d210 0x7f80a816a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:37022/0 (socket says 172.21.15.46:37022) 2024-01-26T23:49:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.222+0000 7f80acca9700 1 -- 172.21.15.46:0/1611024792 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80a80cc1c0 msgr2=0x7f80a815fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.222+0000 7f80acca9700 1 --2- 172.21.15.46:0/1611024792 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80a80cc1c0 0x7f80a815fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.222+0000 7f80acca9700 1 -- 172.21.15.46:0/1611024792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a815ce40 msgr2=0x7f80a809d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.222+0000 7f80acca9700 1 --2- 172.21.15.46:0/1611024792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a815ce40 0x7f80a809d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.222+0000 7f80acca9700 1 -- 172.21.15.46:0/1611024792 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80980093c0 con 0x7f80a815d210 2024-01-26T23:49:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.222+0000 7f80a77fe700 1 --2- 172.21.15.46:0/1611024792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a815ce40 0x7f80a809d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.222+0000 7f80acca9700 1 --2- 172.21.15.46:0/1611024792 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80a815d210 0x7f80a816a870 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f80a0016000 tx=0x7f80a0019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.222+0000 7f808ffff700 1 -- 172.21.15.46:0/1611024792 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a0012e10 con 0x7f80a815d210 2024-01-26T23:49:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.222+0000 7f808ffff700 1 -- 172.21.15.46:0/1611024792 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a0027470 con 0x7f80a815d210 2024-01-26T23:49:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.222+0000 7f808ffff700 1 -- 172.21.15.46:0/1611024792 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a0027830 con 0x7f80a815d210 2024-01-26T23:49:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.222+0000 7f80ae70c700 1 -- 172.21.15.46:0/1611024792 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a0009a20 con 0x7f80a815d210 2024-01-26T23:49:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.222+0000 7f80ae70c700 1 -- 172.21.15.46:0/1611024792 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a815ee00 con 0x7f80a815d210 2024-01-26T23:49:51.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.223+0000 7f80ae70c700 1 -- 172.21.15.46:0/1611024792 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a815f8e0 con 0x7f80a815d210 2024-01-26T23:49:51.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.224+0000 7f808ffff700 1 -- 172.21.15.46:0/1611024792 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 72) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f80a0024070 con 0x7f80a815d210 2024-01-26T23:49:51.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.224+0000 7f808ffff700 1 --2- 172.21.15.46:0/1611024792 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f80940670e0 0x7f8094069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:51.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.224+0000 7f808ffff700 1 -- 172.21.15.46:0/1611024792 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f80a0087150 con 0x7f80a815d210 2024-01-26T23:49:51.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.224+0000 7f80a7fff700 1 -- 172.21.15.46:0/1611024792 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f80940670e0 msgr2=0x7f8094069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:51.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.224+0000 7f80a7fff700 1 --2- 172.21.15.46:0/1611024792 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f80940670e0 0x7f8094069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:51.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.228+0000 7f808ffff700 1 -- 172.21.15.46:0/1611024792 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f80a815f8e0 con 0x7f80a815d210 2024-01-26T23:49:51.298 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:51.297+0000 7ffb394e41c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:49:51.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.425+0000 7f80a7fff700 1 -- 172.21.15.46:0/1611024792 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f80940670e0 msgr2=0x7f8094069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:51.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.425+0000 7f80a7fff700 1 --2- 172.21.15.46:0/1611024792 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f80940670e0 0x7f8094069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:51.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.481+0000 7f80ae70c700 1 -- 172.21.15.46:0/1611024792 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80a815f8e0 con 0x7f80a815d210 2024-01-26T23:49:51.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.482+0000 7f808ffff700 1 -- 172.21.15.46:0/1611024792 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f80a815f8e0 con 0x7f80a815d210 2024-01-26T23:49:51.483 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:51.483 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/406559819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":406559819},{"type":"v1","addr":"172.21.15.106:6836","nonce":406559819}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/3387155671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":3387155671},{"type":"v1","addr":"172.21.15.106:6834","nonce":3387155671}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8298,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/855914203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":855914203},{"type":"v1","addr":"172.21.15.46:6834","nonce":855914203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:49:21.360002+0000","modified":"2024-01-26T23:49:24.956671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/1052887453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1052887453},{"type":"v1","addr":"172.21.15.46:6835","nonce":1052887453}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2024-01-26T23:49:51.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.484+0000 7f80ae70c700 1 -- 172.21.15.46:0/1611024792 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f80940670e0 msgr2=0x7f8094069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:51.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.484+0000 7f80ae70c700 1 --2- 172.21.15.46:0/1611024792 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f80940670e0 0x7f8094069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.484+0000 7f80ae70c700 1 -- 172.21.15.46:0/1611024792 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80a815d210 msgr2=0x7f80a816a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:51.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.484+0000 7f80ae70c700 1 --2- 172.21.15.46:0/1611024792 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80a815d210 0x7f80a816a870 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f80a0016000 tx=0x7f80a0019000 comp rx=0 tx=0).stop 2024-01-26T23:49:51.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.484+0000 7f80ae70c700 1 -- 172.21.15.46:0/1611024792 shutdown_connections 2024-01-26T23:49:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.484+0000 7f80ae70c700 1 --2- 172.21.15.46:0/1611024792 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f80940670e0 0x7f8094069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.484+0000 7f80ae70c700 1 --2- 172.21.15.46:0/1611024792 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80a815d210 0x7f80a816a870 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.484+0000 7f80ae70c700 1 --2- 172.21.15.46:0/1611024792 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80a80cc1c0 0x7f80a815fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.484+0000 7f80ae70c700 1 --2- 172.21.15.46:0/1611024792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a815ce40 0x7f80a809d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.484+0000 7f80ae70c700 1 -- 172.21.15.46:0/1611024792 >> 172.21.15.46:0/1611024792 conn(0x7f80a806f840 msgr2=0x7f80a80b5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.484+0000 7f80ae70c700 1 -- 172.21.15.46:0/1611024792 shutdown_connections 2024-01-26T23:49:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.484+0000 7f80ae70c700 1 -- 172.21.15.46:0/1611024792 wait complete. 2024-01-26T23:49:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 70 2024-01-26T23:49:51.500 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:49:51.500 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:51.499+0000 7fd054cba1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:49:51.574 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:51.573+0000 7fd054cba1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:49:51.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.632+0000 7ff73689a700 1 Processor -- start 2024-01-26T23:49:51.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.632+0000 7ff73689a700 1 -- start start 2024-01-26T23:49:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.633+0000 7ff73689a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff7300c6520 0x7ff7300ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.633+0000 7ff73689a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff7300ced50 0x7ff7300cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.633+0000 7ff73689a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff73015a0b0 con 0x7ff7300b2b40 2024-01-26T23:49:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.633+0000 7ff73689a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7300af070 con 0x7ff7300c6520 2024-01-26T23:49:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.633+0000 7ff73689a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff730099fc0 con 0x7ff7300ced50 2024-01-26T23:49:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.633+0000 7ff72ffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff7300b2b40 0x7ff7300ae960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58782/0 (socket says 172.21.15.46:58782) 2024-01-26T23:49:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.633+0000 7ff72ffff700 1 -- 172.21.15.46:0/70393135 learned_addr learned my addr 172.21.15.46:0/70393135 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:51.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.633+0000 7ff72f7fe700 1 --2- 172.21.15.46:0/70393135 >> v2:172.21.15.106:3300/0 conn(0x7ff7300c6520 0x7ff7300ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:51.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.633+0000 7ff734e37700 1 --2- 172.21.15.46:0/70393135 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff7300ced50 0x7ff7300cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:51.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.633+0000 7ff72effd700 1 -- 172.21.15.46:0/70393135 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199557604 0 0) 0x7ff73015a0b0 con 0x7ff7300b2b40 2024-01-26T23:49:51.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.633+0000 7ff72effd700 1 -- 172.21.15.46:0/70393135 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff718003570 con 0x7ff7300b2b40 2024-01-26T23:49:51.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.633+0000 7ff72f7fe700 1 -- 172.21.15.46:0/70393135 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff7300ced50 msgr2=0x7ff7300cf1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:51.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.633+0000 7ff72f7fe700 1 --2- 172.21.15.46:0/70393135 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff7300ced50 0x7ff7300cf1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.633+0000 7ff72effd700 1 -- 172.21.15.46:0/70393135 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1435681058 0 0) 0x7ff718003570 con 0x7ff7300b2b40 2024-01-26T23:49:51.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.633+0000 7ff72f7fe700 1 -- 172.21.15.46:0/70393135 >> v1:172.21.15.46:6789/0 conn(0x7ff7300b2b40 legacy=0x7ff7300ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:51.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.633+0000 7ff72f7fe700 1 -- 172.21.15.46:0/70393135 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7300ae4d0 con 0x7ff7300c6520 2024-01-26T23:49:51.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.633+0000 7ff734e37700 1 --2- 172.21.15.46:0/70393135 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff7300ced50 0x7ff7300cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:51.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.634+0000 7ff72f7fe700 1 --2- 172.21.15.46:0/70393135 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7300c6520 0x7ff7300ad1c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff7240098e0 tx=0x7ff7240091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d86703e4c4c8bda9 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:51.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.634+0000 7ff72effd700 1 -- 172.21.15.46:0/70393135 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7200015f0 con 0x7ff7300c6520 2024-01-26T23:49:51.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.634+0000 7ff72effd700 1 -- 172.21.15.46:0/70393135 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff724011420 con 0x7ff7300c6520 2024-01-26T23:49:51.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.634+0000 7ff72effd700 1 -- 172.21.15.46:0/70393135 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7240117c0 con 0x7ff7300c6520 2024-01-26T23:49:51.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.634+0000 7ff73689a700 1 -- 172.21.15.46:0/70393135 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7300c6520 msgr2=0x7ff7300ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:51.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.634+0000 7ff73689a700 1 --2- 172.21.15.46:0/70393135 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7300c6520 0x7ff7300ad1c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff7240098e0 tx=0x7ff7240091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.634+0000 7ff73689a700 1 -- 172.21.15.46:0/70393135 shutdown_connections 2024-01-26T23:49:51.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.634+0000 7ff73689a700 1 --2- 172.21.15.46:0/70393135 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff7300ced50 0x7ff7300cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.634+0000 7ff73689a700 1 --2- 172.21.15.46:0/70393135 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7300c6520 0x7ff7300ad1c0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.634+0000 7ff73689a700 1 -- 172.21.15.46:0/70393135 >> 172.21.15.46:0/70393135 conn(0x7ff730099a30 msgr2=0x7ff73006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:51.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.634+0000 7ff73689a700 1 -- 172.21.15.46:0/70393135 shutdown_connections 2024-01-26T23:49:51.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.635+0000 7ff73689a700 1 -- 172.21.15.46:0/70393135 wait complete. 2024-01-26T23:49:51.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.636+0000 7ff73689a700 1 Processor -- start 2024-01-26T23:49:51.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.636+0000 7ff73689a700 1 -- start start 2024-01-26T23:49:51.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.637+0000 7ff73689a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff7300b2b40 0x7ff7301571c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:51.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.637+0000 7ff73689a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7300c6520 0x7ff7300b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:51.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.637+0000 7ff73689a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7300ced50 0x7ff730069740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:51.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.637+0000 7ff73689a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7300b52d0 con 0x7ff7300c6520 2024-01-26T23:49:51.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.637+0000 7ff73689a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7300c7a70 con 0x7ff7300ced50 2024-01-26T23:49:51.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.637+0000 7ff73689a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff730157700 con 0x7ff7300b2b40 2024-01-26T23:49:51.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.637+0000 7ff72f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7300c6520 0x7ff7300b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:51.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.637+0000 7ff72f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7300c6520 0x7ff7300b3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54296/0 (socket says 172.21.15.46:54296) 2024-01-26T23:49:51.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.637+0000 7ff72f7fe700 1 -- 172.21.15.46:0/302275059 learned_addr learned my addr 172.21.15.46:0/302275059 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.637+0000 7ff72ffff700 1 --2- 172.21.15.46:0/302275059 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff7300b2b40 0x7ff7301571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.637+0000 7ff734e37700 1 --2- 172.21.15.46:0/302275059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7300ced50 0x7ff730069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.637+0000 7ff72f7fe700 1 -- 172.21.15.46:0/302275059 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff7300b2b40 msgr2=0x7ff7301571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.637+0000 7ff72f7fe700 1 --2- 172.21.15.46:0/302275059 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff7300b2b40 0x7ff7301571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.637+0000 7ff72f7fe700 1 -- 172.21.15.46:0/302275059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7300ced50 msgr2=0x7ff730069740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.637+0000 7ff72f7fe700 1 --2- 172.21.15.46:0/302275059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7300ced50 0x7ff730069740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.637+0000 7ff72f7fe700 1 -- 172.21.15.46:0/302275059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff728009a20 con 0x7ff7300c6520 2024-01-26T23:49:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.637+0000 7ff72f7fe700 1 --2- 172.21.15.46:0/302275059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7300c6520 0x7ff7300b3f20 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7ff7240030c0 tx=0x7ff724016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.638+0000 7ff717fff700 1 -- 172.21.15.46:0/302275059 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72401cbe0 con 0x7ff7300c6520 2024-01-26T23:49:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.638+0000 7ff717fff700 1 -- 172.21.15.46:0/302275059 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72401cd40 con 0x7ff7300c6520 2024-01-26T23:49:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.638+0000 7ff73689a700 1 -- 172.21.15.46:0/302275059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff724009420 con 0x7ff7300c6520 2024-01-26T23:49:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.638+0000 7ff717fff700 1 -- 172.21.15.46:0/302275059 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72401a6a0 con 0x7ff7300c6520 2024-01-26T23:49:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.638+0000 7ff73689a700 1 -- 172.21.15.46:0/302275059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7300b4d10 con 0x7ff7300c6520 2024-01-26T23:49:51.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.639+0000 7ff73689a700 1 -- 172.21.15.46:0/302275059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7300c7c00 con 0x7ff7300c6520 2024-01-26T23:49:51.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.639+0000 7ff717fff700 1 -- 172.21.15.46:0/302275059 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 72) v1 ==== 84901+0+0 (secure 0 0 0) 0x7ff72401aad0 con 0x7ff7300c6520 2024-01-26T23:49:51.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.640+0000 7ff717fff700 1 --2- 172.21.15.46:0/302275059 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff71c0670e0 0x7ff71c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.640+0000 7ff72ffff700 1 -- 172.21.15.46:0/302275059 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff71c0670e0 msgr2=0x7ff71c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.640+0000 7ff717fff700 1 -- 172.21.15.46:0/302275059 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7ff724086530 con 0x7ff7300c6520 2024-01-26T23:49:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.640+0000 7ff72ffff700 1 --2- 172.21.15.46:0/302275059 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff71c0670e0 0x7ff71c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:51.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.644+0000 7ff717fff700 1 -- 172.21.15.46:0/302275059 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff7300c7c00 con 0x7ff7300c6520 2024-01-26T23:49:51.698 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:51.698+0000 7fd054cba1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:49:51.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.841+0000 7ff72ffff700 1 -- 172.21.15.46:0/302275059 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff71c0670e0 msgr2=0x7ff71c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:51.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.841+0000 7ff72ffff700 1 --2- 172.21.15.46:0/302275059 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff71c0670e0 0x7ff71c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:51.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.853+0000 7ff73689a700 1 -- 172.21.15.46:0/302275059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7300c7bb0 con 0x7ff7300c6520 2024-01-26T23:49:51.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.855+0000 7ff717fff700 1 -- 172.21.15.46:0/302275059 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+16534 (secure 0 0 0) 0x7ff72405ae60 con 0x7ff7300c6520 2024-01-26T23:49:51.856 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:51.856 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":146,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:49:40.927013+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:49:19.677244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-01-26T23:49:20.330110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/4118930132":"2024-01-27T23:49:40.926975+0000","172.21.15.106:6837/43388":"2024-01-27T23:49:40.926975+0000","172.21.15.106:6835/43388":"2024-01-27T23:49:40.926975+0000","172.21.15.106:0/3470152548":"2024-01-27T23:49:40.926975+0000","172.21.15.106:0/128938556":"2024-01-27T23:49:40.926975+0000","172.21.15.106:0/860908837":"2024-01-27T23:49:40.926975+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:49:51.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.857+0000 7ff73689a700 1 -- 172.21.15.46:0/302275059 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff71c0670e0 msgr2=0x7ff71c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:51.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.857+0000 7ff73689a700 1 --2- 172.21.15.46:0/302275059 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff71c0670e0 0x7ff71c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.857+0000 7ff73689a700 1 -- 172.21.15.46:0/302275059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7300c6520 msgr2=0x7ff7300b3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.857+0000 7ff73689a700 1 --2- 172.21.15.46:0/302275059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7300c6520 0x7ff7300b3f20 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7ff7240030c0 tx=0x7ff724016000 comp rx=0 tx=0).stop 2024-01-26T23:49:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.857+0000 7ff73689a700 1 -- 172.21.15.46:0/302275059 shutdown_connections 2024-01-26T23:49:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.857+0000 7ff73689a700 1 --2- 172.21.15.46:0/302275059 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff71c0670e0 0x7ff71c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.857+0000 7ff73689a700 1 --2- 172.21.15.46:0/302275059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff7300ced50 0x7ff730069740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.857+0000 7ff73689a700 1 --2- 172.21.15.46:0/302275059 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff7300b2b40 0x7ff7301571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.857+0000 7ff73689a700 1 --2- 172.21.15.46:0/302275059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7300c6520 0x7ff7300b3f20 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.857+0000 7ff73689a700 1 -- 172.21.15.46:0/302275059 >> 172.21.15.46:0/302275059 conn(0x7ff730099a30 msgr2=0x7ff7300ce1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.857+0000 7ff73689a700 1 -- 172.21.15.46:0/302275059 shutdown_connections 2024-01-26T23:49:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:51.857+0000 7ff73689a700 1 -- 172.21.15.46:0/302275059 wait complete. 2024-01-26T23:49:51.873 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-26T23:49:51.873 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:49:52.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.007+0000 7fb273daa700 1 Processor -- start 2024-01-26T23:49:52.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.007+0000 7fb273daa700 1 -- start start 2024-01-26T23:49:52.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.008+0000 7fb273daa700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb26c0b2160 0x7fb26c15a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.008+0000 7fb273daa700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb26c0b2530 0x7fb26c0abe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.008+0000 7fb273daa700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26c0a9b90 con 0x7fb26c15da10 2024-01-26T23:49:52.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.008+0000 7fb273daa700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb26c0aa210 con 0x7fb26c0b2530 2024-01-26T23:49:52.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.008+0000 7fb273daa700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c15d580 con 0x7fb26c0b2160 2024-01-26T23:49:52.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.008+0000 7fb271345700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb26c0b2530 0x7fb26c0abe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:52.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.008+0000 7fb271b46700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb26c0b2160 0x7fb26c15a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:52.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.009+0000 7fb271b46700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb26c0b2160 0x7fb26c15a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:37046/0 (socket says 172.21.15.46:37046) 2024-01-26T23:49:52.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.009+0000 7fb271345700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb26c0b2530 0x7fb26c0abe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:50576/0 (socket says 172.21.15.46:50576) 2024-01-26T23:49:52.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.009+0000 7fb272347700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fb26c15da10 0x7fb26c06a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:51538/0 (socket says 172.21.15.46:51538) 2024-01-26T23:49:52.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.009+0000 7fb271b46700 1 -- 172.21.15.46:0/1334164228 learned_addr learned my addr 172.21.15.46:0/1334164228 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:52.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.009+0000 7fb271b46700 1 -- 172.21.15.46:0/1334164228 >> v2:172.21.15.106:3300/0 conn(0x7fb26c0b2530 msgr2=0x7fb26c0abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:52.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.009+0000 7fb271b46700 1 --2- 172.21.15.46:0/1334164228 >> v2:172.21.15.106:3300/0 conn(0x7fb26c0b2530 0x7fb26c0abe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.009+0000 7fb271b46700 1 -- 172.21.15.46:0/1334164228 >> v1:172.21.15.106:6789/0 conn(0x7fb26c15da10 legacy=0x7fb26c06a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:52.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.009+0000 7fb271b46700 1 -- 172.21.15.46:0/1334164228 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c0ac390 con 0x7fb26c0b2160 2024-01-26T23:49:52.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.009+0000 7fb271b46700 1 --2- 172.21.15.46:0/1334164228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb26c0b2160 0x7fb26c15a7a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb2680099b0 tx=0x7fb268009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0fe17bf0f9d5322 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:52.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.009+0000 7fb270b44700 1 -- 172.21.15.46:0/1334164228 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb268009ea0 con 0x7fb26c0b2160 2024-01-26T23:49:52.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.009+0000 7fb270b44700 1 -- 172.21.15.46:0/1334164228 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26800b860 con 0x7fb26c0b2160 2024-01-26T23:49:52.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.009+0000 7fb270b44700 1 -- 172.21.15.46:0/1334164228 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26800bc20 con 0x7fb26c0b2160 2024-01-26T23:49:52.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.010+0000 7fb273daa700 1 -- 172.21.15.46:0/1334164228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb26c0b2160 msgr2=0x7fb26c15a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:52.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.010+0000 7fb273daa700 1 --2- 172.21.15.46:0/1334164228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb26c0b2160 0x7fb26c15a7a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb2680099b0 tx=0x7fb268009250 comp rx=0 tx=0).stop 2024-01-26T23:49:52.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.010+0000 7fb273daa700 1 -- 172.21.15.46:0/1334164228 shutdown_connections 2024-01-26T23:49:52.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.010+0000 7fb273daa700 1 --2- 172.21.15.46:0/1334164228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb26c0b2160 0x7fb26c15a7a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.010+0000 7fb273daa700 1 --2- 172.21.15.46:0/1334164228 >> v2:172.21.15.106:3300/0 conn(0x7fb26c0b2530 0x7fb26c0abe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.010+0000 7fb273daa700 1 -- 172.21.15.46:0/1334164228 >> 172.21.15.46:0/1334164228 conn(0x7fb26c0ccbc0 msgr2=0x7fb26c0ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:52.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.010+0000 7fb273daa700 1 -- 172.21.15.46:0/1334164228 shutdown_connections 2024-01-26T23:49:52.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.010+0000 7fb273daa700 1 -- 172.21.15.46:0/1334164228 wait complete. 2024-01-26T23:49:52.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.011+0000 7fb273daa700 1 Processor -- start 2024-01-26T23:49:52.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.011+0000 7fb273daa700 1 -- start start 2024-01-26T23:49:52.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.012+0000 7fb273daa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb26c0b2160 0x7fb26c13fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.012+0000 7fb273daa700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb26c0b2530 0x7fb26c0b0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.012+0000 7fb273daa700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb26c15da10 0x7fb26c13f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.012+0000 7fb273daa700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c141170 con 0x7fb26c0b2160 2024-01-26T23:49:52.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.012+0000 7fb273daa700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c140560 con 0x7fb26c15da10 2024-01-26T23:49:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.012+0000 7fb273daa700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c140840 con 0x7fb26c0b2530 2024-01-26T23:49:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.013+0000 7fb271b46700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb26c0b2160 0x7fb26c13fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.013+0000 7fb271b46700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb26c0b2160 0x7fb26c13fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54308/0 (socket says 172.21.15.46:54308) 2024-01-26T23:49:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.013+0000 7fb271b46700 1 -- 172.21.15.46:0/3298875442 learned_addr learned my addr 172.21.15.46:0/3298875442 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.013+0000 7fb271345700 1 --2- 172.21.15.46:0/3298875442 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb26c0b2530 0x7fb26c0b0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.013+0000 7fb272347700 1 --2- 172.21.15.46:0/3298875442 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb26c15da10 0x7fb26c13f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.013+0000 7fb271b46700 1 -- 172.21.15.46:0/3298875442 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb26c0b2530 msgr2=0x7fb26c0b0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.013+0000 7fb271b46700 1 --2- 172.21.15.46:0/3298875442 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb26c0b2530 0x7fb26c0b0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.013+0000 7fb271b46700 1 -- 172.21.15.46:0/3298875442 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb26c15da10 msgr2=0x7fb26c13f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.013+0000 7fb271b46700 1 --2- 172.21.15.46:0/3298875442 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb26c15da10 0x7fb26c13f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.013+0000 7fb271b46700 1 -- 172.21.15.46:0/3298875442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2680094f0 con 0x7fb26c0b2160 2024-01-26T23:49:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.013+0000 7fb271b46700 1 --2- 172.21.15.46:0/3298875442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb26c0b2160 0x7fb26c13fc60 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fb268006fd0 tx=0x7fb268017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.013+0000 7fb261ffb700 1 -- 172.21.15.46:0/3298875442 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26801f070 con 0x7fb26c0b2160 2024-01-26T23:49:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.013+0000 7fb261ffb700 1 -- 172.21.15.46:0/3298875442 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26800b9d0 con 0x7fb26c0b2160 2024-01-26T23:49:52.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.013+0000 7fb273daa700 1 -- 172.21.15.46:0/3298875442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26c140b20 con 0x7fb26c0b2160 2024-01-26T23:49:52.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.013+0000 7fb261ffb700 1 -- 172.21.15.46:0/3298875442 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb268011420 con 0x7fb26c0b2160 2024-01-26T23:49:52.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.014+0000 7fb273daa700 1 -- 172.21.15.46:0/3298875442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb26c1401a0 con 0x7fb26c0b2160 2024-01-26T23:49:52.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.015+0000 7fb273daa700 1 -- 172.21.15.46:0/3298875442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb26c04c710 con 0x7fb26c0b2160 2024-01-26T23:49:52.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.015+0000 7fb261ffb700 1 -- 172.21.15.46:0/3298875442 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 72) v1 ==== 84901+0+0 (secure 0 0 0) 0x7fb268011790 con 0x7fb26c0b2160 2024-01-26T23:49:52.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.015+0000 7fb261ffb700 1 --2- 172.21.15.46:0/3298875442 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb2500670e0 0x7fb250069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.016+0000 7fb261ffb700 1 -- 172.21.15.46:0/3298875442 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fb268085fe0 con 0x7fb26c0b2160 2024-01-26T23:49:52.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.016+0000 7fb271345700 1 -- 172.21.15.46:0/3298875442 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb2500670e0 msgr2=0x7fb250069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:52.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.016+0000 7fb271345700 1 --2- 172.21.15.46:0/3298875442 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb2500670e0 0x7fb250069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:52.018 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:52.017+0000 7ffb394e41c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:49:52.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.020+0000 7fb261ffb700 1 -- 172.21.15.46:0/3298875442 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb268052c00 con 0x7fb26c0b2160 2024-01-26T23:49:52.107 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:52.106+0000 7ffb394e41c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:49:52.202 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:52.201+0000 7fd054cba1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:49:52.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.217+0000 7fb271345700 1 -- 172.21.15.46:0/3298875442 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb2500670e0 msgr2=0x7fb250069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:52.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.217+0000 7fb271345700 1 --2- 172.21.15.46:0/3298875442 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb2500670e0 0x7fb250069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:52.249 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:52.247+0000 7ffb394e41c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:49:52.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.275+0000 7fb273daa700 1 -- 172.21.15.46:0/3298875442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb26c0aa460 con 0x7fb26c0b2160 2024-01-26T23:49:52.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.275+0000 7fb261ffb700 1 -- 172.21.15.46:0/3298875442 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4599 (secure 0 0 0) 0x7fb268015090 con 0x7fb26c0b2160 2024-01-26T23:49:52.277 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:52.277 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/406559819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":406559819},{"type":"v1","addr":"172.21.15.106:6836","nonce":406559819}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/3387155671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":3387155671},{"type":"v1","addr":"172.21.15.106:6834","nonce":3387155671}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8298,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/855914203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":855914203},{"type":"v1","addr":"172.21.15.46:6834","nonce":855914203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:49:21.360002+0000","modified":"2024-01-26T23:49:24.956671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/1052887453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1052887453},{"type":"v1","addr":"172.21.15.46:6835","nonce":1052887453}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2024-01-26T23:49:52.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.278+0000 7fb273daa700 1 -- 172.21.15.46:0/3298875442 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb2500670e0 msgr2=0x7fb250069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:52.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.278+0000 7fb273daa700 1 --2- 172.21.15.46:0/3298875442 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb2500670e0 0x7fb250069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.278+0000 7fb273daa700 1 -- 172.21.15.46:0/3298875442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb26c0b2160 msgr2=0x7fb26c13fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:52.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.278+0000 7fb273daa700 1 --2- 172.21.15.46:0/3298875442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb26c0b2160 0x7fb26c13fc60 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fb268006fd0 tx=0x7fb268017000 comp rx=0 tx=0).stop 2024-01-26T23:49:52.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.278+0000 7fb273daa700 1 -- 172.21.15.46:0/3298875442 shutdown_connections 2024-01-26T23:49:52.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.278+0000 7fb273daa700 1 --2- 172.21.15.46:0/3298875442 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb2500670e0 0x7fb250069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.278+0000 7fb273daa700 1 --2- 172.21.15.46:0/3298875442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb26c0b2160 0x7fb26c13fc60 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.278+0000 7fb273daa700 1 --2- 172.21.15.46:0/3298875442 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb26c15da10 0x7fb26c13f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.278+0000 7fb273daa700 1 --2- 172.21.15.46:0/3298875442 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb26c0b2530 0x7fb26c0b0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.278+0000 7fb273daa700 1 -- 172.21.15.46:0/3298875442 >> 172.21.15.46:0/3298875442 conn(0x7fb26c0ccbc0 msgr2=0x7fb26c15b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:52.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.278+0000 7fb273daa700 1 -- 172.21.15.46:0/3298875442 shutdown_connections 2024-01-26T23:49:52.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.278+0000 7fb273daa700 1 -- 172.21.15.46:0/3298875442 wait complete. 2024-01-26T23:49:52.281 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 70 2024-01-26T23:49:52.293 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:49:52.351 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:52.350+0000 7fd054cba1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:49:52.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.425+0000 7f53c17e9700 1 Processor -- start 2024-01-26T23:49:52.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.425+0000 7f53c17e9700 1 -- start start 2024-01-26T23:49:52.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.426+0000 7f53c17e9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f53bc15a1d0 0x7f53bc0afc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.426+0000 7f53c17e9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53bc0b0160 0x7f53bc157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.426+0000 7f53c17e9700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53bc1509d0 con 0x7f53bc0af850 2024-01-26T23:49:52.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.426+0000 7f53c17e9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53bc157820 con 0x7f53bc0b0160 2024-01-26T23:49:52.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.426+0000 7f53c17e9700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53bc0b10d0 con 0x7f53bc15a1d0 2024-01-26T23:49:52.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.426+0000 7f53baffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f53bc0af850 0x7f53bc159ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58784/0 (socket says 172.21.15.46:58784) 2024-01-26T23:49:52.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.426+0000 7f53baffd700 1 -- 172.21.15.46:0/3355260179 learned_addr learned my addr 172.21.15.46:0/3355260179 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:52.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.426+0000 7f53bb7fe700 1 --2- 172.21.15.46:0/3355260179 >> v2:172.21.15.46:3300/0 conn(0x7f53bc0b0160 0x7f53bc157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:52.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.426+0000 7f53ba7fc700 1 --2- 172.21.15.46:0/3355260179 >> v2:172.21.15.106:3300/0 conn(0x7f53bc15a1d0 0x7f53bc0afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:52.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.427+0000 7f53b9ffb700 1 -- 172.21.15.46:0/3355260179 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376497043 0 0) 0x7f53ac001110 con 0x7f53bc0af850 2024-01-26T23:49:52.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.427+0000 7f53b9ffb700 1 -- 172.21.15.46:0/3355260179 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a4003570 con 0x7f53bc0af850 2024-01-26T23:49:52.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.427+0000 7f53bb7fe700 1 -- 172.21.15.46:0/3355260179 >> v2:172.21.15.106:3300/0 conn(0x7f53bc15a1d0 msgr2=0x7f53bc0afc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:52.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.427+0000 7f53bb7fe700 1 --2- 172.21.15.46:0/3355260179 >> v2:172.21.15.106:3300/0 conn(0x7f53bc15a1d0 0x7f53bc0afc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.427+0000 7f53bb7fe700 1 -- 172.21.15.46:0/3355260179 >> v1:172.21.15.46:6789/0 conn(0x7f53bc0af850 legacy=0x7f53bc159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:52.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.427+0000 7f53bb7fe700 1 -- 172.21.15.46:0/3355260179 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc0b0590 con 0x7f53bc0b0160 2024-01-26T23:49:52.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.427+0000 7f53bb7fe700 1 --2- 172.21.15.46:0/3355260179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc0b0160 0x7f53bc157cb0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f53bc06ee40 tx=0x7f53b400a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=384125c5192976f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:52.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.427+0000 7f53b9ffb700 1 -- 172.21.15.46:0/3355260179 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b400f070 con 0x7f53bc0b0160 2024-01-26T23:49:52.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.427+0000 7f53b9ffb700 1 -- 172.21.15.46:0/3355260179 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b400a2e0 con 0x7f53bc0b0160 2024-01-26T23:49:52.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.427+0000 7f53b9ffb700 1 -- 172.21.15.46:0/3355260179 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b400cbc0 con 0x7f53bc0b0160 2024-01-26T23:49:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.428+0000 7f53c17e9700 1 -- 172.21.15.46:0/3355260179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc0b0160 msgr2=0x7f53bc157cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.428+0000 7f53c17e9700 1 --2- 172.21.15.46:0/3355260179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc0b0160 0x7f53bc157cb0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f53bc06ee40 tx=0x7f53b400a890 comp rx=0 tx=0).stop 2024-01-26T23:49:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.428+0000 7f53c17e9700 1 -- 172.21.15.46:0/3355260179 shutdown_connections 2024-01-26T23:49:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.428+0000 7f53c17e9700 1 --2- 172.21.15.46:0/3355260179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc0b0160 0x7f53bc157cb0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.428+0000 7f53c17e9700 1 --2- 172.21.15.46:0/3355260179 >> v2:172.21.15.106:3300/0 conn(0x7f53bc15a1d0 0x7f53bc0afc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.428+0000 7f53c17e9700 1 -- 172.21.15.46:0/3355260179 >> 172.21.15.46:0/3355260179 conn(0x7f53bc068400 msgr2=0x7f53bc0cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.428+0000 7f53c17e9700 1 -- 172.21.15.46:0/3355260179 shutdown_connections 2024-01-26T23:49:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.428+0000 7f53c17e9700 1 -- 172.21.15.46:0/3355260179 wait complete. 2024-01-26T23:49:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.430+0000 7f53c17e9700 1 Processor -- start 2024-01-26T23:49:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.430+0000 7f53c17e9700 1 -- start start 2024-01-26T23:49:52.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.431+0000 7f53c17e9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc0af850 0x7f53bc167480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.431+0000 7f53c17e9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53bc0b0160 0x7f53bc0c1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.431+0000 7f53c17e9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53bc15a1d0 0x7f53bc16a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.431+0000 7f53c17e9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0c1880 con 0x7f53bc0af850 2024-01-26T23:49:52.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.431+0000 7f53c17e9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0c2350 con 0x7f53bc0b0160 2024-01-26T23:49:52.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.431+0000 7f53c17e9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc16a870 con 0x7f53bc15a1d0 2024-01-26T23:49:52.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.431+0000 7f53baffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc0af850 0x7f53bc167480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:52.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.431+0000 7f53baffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc0af850 0x7f53bc167480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54340/0 (socket says 172.21.15.46:54340) 2024-01-26T23:49:52.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.431+0000 7f53baffd700 1 -- 172.21.15.46:0/718642288 learned_addr learned my addr 172.21.15.46:0/718642288 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:52.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.431+0000 7f53ba7fc700 1 --2- 172.21.15.46:0/718642288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53bc0b0160 0x7f53bc0c1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:52.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.431+0000 7f53bb7fe700 1 --2- 172.21.15.46:0/718642288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53bc15a1d0 0x7f53bc16a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:52.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.431+0000 7f53baffd700 1 -- 172.21.15.46:0/718642288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53bc15a1d0 msgr2=0x7f53bc16a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:52.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.431+0000 7f53baffd700 1 --2- 172.21.15.46:0/718642288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53bc15a1d0 0x7f53bc16a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.431+0000 7f53baffd700 1 -- 172.21.15.46:0/718642288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53bc0b0160 msgr2=0x7f53bc0c1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:52.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.431+0000 7f53baffd700 1 --2- 172.21.15.46:0/718642288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53bc0b0160 0x7f53bc0c1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.431+0000 7f53baffd700 1 -- 172.21.15.46:0/718642288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b400ab30 con 0x7f53bc0af850 2024-01-26T23:49:52.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.432+0000 7f53baffd700 1 --2- 172.21.15.46:0/718642288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc0af850 0x7f53bc167480 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00c5c0 tx=0x7f53ac007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:52.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.432+0000 7f53a2ffd700 1 -- 172.21.15.46:0/718642288 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac005b30 con 0x7f53bc0af850 2024-01-26T23:49:52.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.432+0000 7f53a2ffd700 1 -- 172.21.15.46:0/718642288 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac005cb0 con 0x7f53bc0af850 2024-01-26T23:49:52.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.432+0000 7f53a2ffd700 1 -- 172.21.15.46:0/718642288 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac007320 con 0x7f53bc0af850 2024-01-26T23:49:52.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.432+0000 7f53c17e9700 1 -- 172.21.15.46:0/718642288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53bc0c1ab0 con 0x7f53bc0af850 2024-01-26T23:49:52.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.432+0000 7f53c17e9700 1 -- 172.21.15.46:0/718642288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53bc0c2070 con 0x7f53bc0af850 2024-01-26T23:49:52.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.433+0000 7f53c17e9700 1 -- 172.21.15.46:0/718642288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53bc04c710 con 0x7f53bc0af850 2024-01-26T23:49:52.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.434+0000 7f53a2ffd700 1 -- 172.21.15.46:0/718642288 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 72) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f53ac01b3b0 con 0x7f53bc0af850 2024-01-26T23:49:52.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.434+0000 7f53a2ffd700 1 --2- 172.21.15.46:0/718642288 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f53a80670e0 0x7f53a8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.435+0000 7f53a2ffd700 1 -- 172.21.15.46:0/718642288 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f53ac083ed0 con 0x7f53bc0af850 2024-01-26T23:49:52.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.435+0000 7f53ba7fc700 1 -- 172.21.15.46:0/718642288 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f53a80670e0 msgr2=0x7f53a8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:52.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.435+0000 7f53ba7fc700 1 --2- 172.21.15.46:0/718642288 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f53a80670e0 0x7f53a8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:52.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.443+0000 7f53a2ffd700 1 -- 172.21.15.46:0/718642288 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f53ac050af0 con 0x7f53bc0af850 2024-01-26T23:49:52.503 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:52.502+0000 7fd054cba1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:49:52.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.636+0000 7f53ba7fc700 1 -- 172.21.15.46:0/718642288 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f53a80670e0 msgr2=0x7f53a8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:52.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.636+0000 7f53ba7fc700 1 --2- 172.21.15.46:0/718642288 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f53a80670e0 0x7f53a8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:52.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.696+0000 7f53c17e9700 1 -- 172.21.15.46:0/718642288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53bc1509d0 con 0x7f53bc0af850 2024-01-26T23:49:52.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.697+0000 7f53a2ffd700 1 -- 172.21.15.46:0/718642288 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f53bc1509d0 con 0x7f53bc0af850 2024-01-26T23:49:52.698 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:52.699 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/406559819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":406559819},{"type":"v1","addr":"172.21.15.106:6836","nonce":406559819}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/3387155671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":3387155671},{"type":"v1","addr":"172.21.15.106:6834","nonce":3387155671}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8298,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/855914203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":855914203},{"type":"v1","addr":"172.21.15.46:6834","nonce":855914203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:49:21.360002+0000","modified":"2024-01-26T23:49:24.956671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/1052887453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1052887453},{"type":"v1","addr":"172.21.15.46:6835","nonce":1052887453}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2024-01-26T23:49:52.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.699+0000 7f53c17e9700 1 -- 172.21.15.46:0/718642288 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f53a80670e0 msgr2=0x7f53a8069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:52.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.699+0000 7f53c17e9700 1 --2- 172.21.15.46:0/718642288 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f53a80670e0 0x7f53a8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.699+0000 7f53c17e9700 1 -- 172.21.15.46:0/718642288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc0af850 msgr2=0x7f53bc167480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:52.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.699+0000 7f53c17e9700 1 --2- 172.21.15.46:0/718642288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc0af850 0x7f53bc167480 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00c5c0 tx=0x7f53ac007c70 comp rx=0 tx=0).stop 2024-01-26T23:49:52.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.699+0000 7f53c17e9700 1 -- 172.21.15.46:0/718642288 shutdown_connections 2024-01-26T23:49:52.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.699+0000 7f53c17e9700 1 --2- 172.21.15.46:0/718642288 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f53a80670e0 0x7f53a8069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.699+0000 7f53c17e9700 1 --2- 172.21.15.46:0/718642288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc0af850 0x7f53bc167480 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.700+0000 7f53c17e9700 1 --2- 172.21.15.46:0/718642288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53bc15a1d0 0x7f53bc16a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.700+0000 7f53c17e9700 1 --2- 172.21.15.46:0/718642288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53bc0b0160 0x7f53bc0c1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.700+0000 7f53c17e9700 1 -- 172.21.15.46:0/718642288 >> 172.21.15.46:0/718642288 conn(0x7f53bc068400 msgr2=0x7f53bc0cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:52.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.700+0000 7f53c17e9700 1 -- 172.21.15.46:0/718642288 shutdown_connections 2024-01-26T23:49:52.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.700+0000 7f53c17e9700 1 -- 172.21.15.46:0/718642288 wait complete. 2024-01-26T23:49:52.702 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 70 2024-01-26T23:49:52.715 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:49:52.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.847+0000 7f1d75d27700 1 Processor -- start 2024-01-26T23:49:52.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.847+0000 7f1d75d27700 1 -- start start 2024-01-26T23:49:52.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.848+0000 7f1d75d27700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d70154290 0x7f1d700b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.848+0000 7f1d75d27700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d700b2100 0x7f1d700b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.848+0000 7f1d75d27700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7015b140 con 0x7f1d70153ec0 2024-01-26T23:49:52.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.848+0000 7f1d75d27700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d700b2580 con 0x7f1d70154290 2024-01-26T23:49:52.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.848+0000 7f1d75d27700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7015a5c0 con 0x7f1d700b2100 2024-01-26T23:49:52.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.848+0000 7f1d6effd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d70154290 0x7f1d700b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:52.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.848+0000 7f1d6effd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d70154290 0x7f1d700b1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54350/0 (socket says 172.21.15.46:54350) 2024-01-26T23:49:52.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.848+0000 7f1d6effd700 1 -- 172.21.15.46:0/3064579540 learned_addr learned my addr 172.21.15.46:0/3064579540 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:52.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.848+0000 7f1d6ffff700 1 --2- 172.21.15.46:0/3064579540 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d700b2100 0x7f1d700b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:52.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.849+0000 7f1d6effd700 1 -- 172.21.15.46:0/3064579540 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d700b2100 msgr2=0x7f1d700b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:52.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.849+0000 7f1d6effd700 1 --2- 172.21.15.46:0/3064579540 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d700b2100 0x7f1d700b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.849+0000 7f1d6effd700 1 -- 172.21.15.46:0/3064579540 >> v1:172.21.15.106:6789/0 conn(0x7f1d70153ec0 legacy=0x7f1d70157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:52.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.849+0000 7f1d6effd700 1 -- 172.21.15.46:0/3064579540 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7015a8a0 con 0x7f1d70154290 2024-01-26T23:49:52.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.849+0000 7f1d6ffff700 1 --2- 172.21.15.46:0/3064579540 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d700b2100 0x7f1d700b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:52.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.849+0000 7f1d6effd700 1 --2- 172.21.15.46:0/3064579540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d70154290 0x7f1d700b1bc0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f1d640099b0 tx=0x7f1d64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e442da56b1111cc7 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:52.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.849+0000 7f1d6e7fc700 1 -- 172.21.15.46:0/3064579540 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6400b860 con 0x7f1d70154290 2024-01-26T23:49:52.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.849+0000 7f1d6e7fc700 1 -- 172.21.15.46:0/3064579540 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6400b9c0 con 0x7f1d70154290 2024-01-26T23:49:52.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.849+0000 7f1d6e7fc700 1 -- 172.21.15.46:0/3064579540 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6400bd60 con 0x7f1d70154290 2024-01-26T23:49:52.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.850+0000 7f1d75d27700 1 -- 172.21.15.46:0/3064579540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d70154290 msgr2=0x7f1d700b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:52.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.850+0000 7f1d75d27700 1 --2- 172.21.15.46:0/3064579540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d70154290 0x7f1d700b1bc0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f1d640099b0 tx=0x7f1d64009250 comp rx=0 tx=0).stop 2024-01-26T23:49:52.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.850+0000 7f1d75d27700 1 -- 172.21.15.46:0/3064579540 shutdown_connections 2024-01-26T23:49:52.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.850+0000 7f1d75d27700 1 --2- 172.21.15.46:0/3064579540 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d700b2100 0x7f1d700b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.850+0000 7f1d75d27700 1 --2- 172.21.15.46:0/3064579540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d70154290 0x7f1d700b1bc0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.850+0000 7f1d75d27700 1 -- 172.21.15.46:0/3064579540 >> 172.21.15.46:0/3064579540 conn(0x7f1d70069510 msgr2=0x7f1d700aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:52.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.850+0000 7f1d75d27700 1 -- 172.21.15.46:0/3064579540 shutdown_connections 2024-01-26T23:49:52.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.850+0000 7f1d75d27700 1 -- 172.21.15.46:0/3064579540 wait complete. 2024-01-26T23:49:52.855 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:52.851+0000 7ffb394e41c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:49:52.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.851+0000 7f1d75d27700 1 Processor -- start 2024-01-26T23:49:52.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.851+0000 7f1d75d27700 1 -- start start 2024-01-26T23:49:52.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.852+0000 7f1d75d27700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d700b2100 0x7f1d700cc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.852+0000 7f1d75d27700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d70153ec0 0x7f1d700cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.852+0000 7f1d75d27700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d70154290 0x7f1d70145e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.852+0000 7f1d75d27700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d700cc3e0 con 0x7f1d70153ec0 2024-01-26T23:49:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.852+0000 7f1d75d27700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d70167d20 con 0x7f1d70154290 2024-01-26T23:49:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.852+0000 7f1d75d27700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d70168000 con 0x7f1d700b2100 2024-01-26T23:49:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.852+0000 7f1d6effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d70153ec0 0x7f1d700cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.852+0000 7f1d6effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d70153ec0 0x7f1d700cbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54360/0 (socket says 172.21.15.46:54360) 2024-01-26T23:49:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.852+0000 7f1d6effd700 1 -- 172.21.15.46:0/912185479 learned_addr learned my addr 172.21.15.46:0/912185479 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.852+0000 7f1d6ffff700 1 --2- 172.21.15.46:0/912185479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d70154290 0x7f1d70145e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.852+0000 7f1d6f7fe700 1 --2- 172.21.15.46:0/912185479 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d700b2100 0x7f1d700cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.853+0000 7f1d6effd700 1 -- 172.21.15.46:0/912185479 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d700b2100 msgr2=0x7f1d700cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.853+0000 7f1d6effd700 1 --2- 172.21.15.46:0/912185479 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d700b2100 0x7f1d700cc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.853+0000 7f1d6effd700 1 -- 172.21.15.46:0/912185479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d70154290 msgr2=0x7f1d70145e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.853+0000 7f1d6effd700 1 --2- 172.21.15.46:0/912185479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d70154290 0x7f1d70145e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.853+0000 7f1d6effd700 1 -- 172.21.15.46:0/912185479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d640094f0 con 0x7f1d70153ec0 2024-01-26T23:49:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.853+0000 7f1d6effd700 1 --2- 172.21.15.46:0/912185479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d70153ec0 0x7f1d700cbea0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f1d640090b0 tx=0x7f1d64017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.853+0000 7f1d577fe700 1 -- 172.21.15.46:0/912185479 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d64011e00 con 0x7f1d70153ec0 2024-01-26T23:49:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.853+0000 7f1d577fe700 1 -- 172.21.15.46:0/912185479 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d64024440 con 0x7f1d70153ec0 2024-01-26T23:49:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.853+0000 7f1d75d27700 1 -- 172.21.15.46:0/912185479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d701682e0 con 0x7f1d70153ec0 2024-01-26T23:49:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.853+0000 7f1d577fe700 1 -- 172.21.15.46:0/912185479 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d64024800 con 0x7f1d70153ec0 2024-01-26T23:49:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.853+0000 7f1d75d27700 1 -- 172.21.15.46:0/912185479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d70167a30 con 0x7f1d70153ec0 2024-01-26T23:49:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.854+0000 7f1d75d27700 1 -- 172.21.15.46:0/912185479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7016b060 con 0x7f1d70153ec0 2024-01-26T23:49:52.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.855+0000 7f1d577fe700 1 -- 172.21.15.46:0/912185479 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 72) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f1d6401a420 con 0x7f1d70153ec0 2024-01-26T23:49:52.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.855+0000 7f1d577fe700 1 --2- 172.21.15.46:0/912185479 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1d5c0670e0 0x7f1d5c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:52.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.855+0000 7f1d577fe700 1 -- 172.21.15.46:0/912185479 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f1d64086b10 con 0x7f1d70153ec0 2024-01-26T23:49:52.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.855+0000 7f1d6f7fe700 1 -- 172.21.15.46:0/912185479 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1d5c0670e0 msgr2=0x7f1d5c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:52.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.855+0000 7f1d6f7fe700 1 --2- 172.21.15.46:0/912185479 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1d5c0670e0 0x7f1d5c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:52.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:52.864+0000 7f1d577fe700 1 -- 172.21.15.46:0/912185479 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1d7016b060 con 0x7f1d70153ec0 2024-01-26T23:49:53.031 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:53.030+0000 7ffb394e41c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:49:53.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.056+0000 7f1d6f7fe700 1 -- 172.21.15.46:0/912185479 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1d5c0670e0 msgr2=0x7f1d5c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:53.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.056+0000 7f1d6f7fe700 1 --2- 172.21.15.46:0/912185479 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1d5c0670e0 0x7f1d5c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.076+0000 7f1d75d27700 1 -- 172.21.15.46:0/912185479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d7016b060 con 0x7f1d70153ec0 2024-01-26T23:49:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.077+0000 7f1d577fe700 1 -- 172.21.15.46:0/912185479 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+16534 (secure 0 0 0) 0x7f1d7016b060 con 0x7f1d70153ec0 2024-01-26T23:49:53.079 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:49:53.079 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":146,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:49:40.927013+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:49:19.677244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-01-26T23:49:20.330110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/4118930132":"2024-01-27T23:49:40.926975+0000","172.21.15.106:6837/43388":"2024-01-27T23:49:40.926975+0000","172.21.15.106:6835/43388":"2024-01-27T23:49:40.926975+0000","172.21.15.106:0/3470152548":"2024-01-27T23:49:40.926975+0000","172.21.15.106:0/128938556":"2024-01-27T23:49:40.926975+0000","172.21.15.106:0/860908837":"2024-01-27T23:49:40.926975+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:49:53.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.080+0000 7f1d75d27700 1 -- 172.21.15.46:0/912185479 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1d5c0670e0 msgr2=0x7f1d5c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:53.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.080+0000 7f1d75d27700 1 --2- 172.21.15.46:0/912185479 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1d5c0670e0 0x7f1d5c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.080+0000 7f1d75d27700 1 -- 172.21.15.46:0/912185479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d70153ec0 msgr2=0x7f1d700cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.080+0000 7f1d75d27700 1 --2- 172.21.15.46:0/912185479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d70153ec0 0x7f1d700cbea0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f1d640090b0 tx=0x7f1d64017000 comp rx=0 tx=0).stop 2024-01-26T23:49:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.080+0000 7f1d75d27700 1 -- 172.21.15.46:0/912185479 shutdown_connections 2024-01-26T23:49:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.080+0000 7f1d75d27700 1 --2- 172.21.15.46:0/912185479 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1d5c0670e0 0x7f1d5c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.080+0000 7f1d75d27700 1 --2- 172.21.15.46:0/912185479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d70154290 0x7f1d70145e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.080+0000 7f1d75d27700 1 --2- 172.21.15.46:0/912185479 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d700b2100 0x7f1d700cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.080+0000 7f1d75d27700 1 --2- 172.21.15.46:0/912185479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d70153ec0 0x7f1d700cbea0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.080+0000 7f1d75d27700 1 -- 172.21.15.46:0/912185479 >> 172.21.15.46:0/912185479 conn(0x7f1d70069510 msgr2=0x7f1d70155f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.080+0000 7f1d75d27700 1 -- 172.21.15.46:0/912185479 shutdown_connections 2024-01-26T23:49:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.080+0000 7f1d75d27700 1 -- 172.21.15.46:0/912185479 wait complete. 2024-01-26T23:49:53.097 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-26T23:49:53.208 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:53.207+0000 7fd054cba1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:49:53.214 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:53.213+0000 7ffb394e41c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:49:53.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.232+0000 7efcf1a75700 1 Processor -- start 2024-01-26T23:49:53.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.232+0000 7efcf1a75700 1 -- start start 2024-01-26T23:49:53.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.233+0000 7efcf1a75700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7efcec15d210 0x7efcec0cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:53.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.233+0000 7efcf1a75700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcec0cc450 con 0x7efcec15ce40 2024-01-26T23:49:53.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.233+0000 7efcf1a75700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcec0cff70 con 0x7efcec0cbff0 2024-01-26T23:49:53.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.233+0000 7efcf1a75700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcec0cf400 con 0x7efcec15d210 2024-01-26T23:49:53.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.234+0000 7efceb7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efcec15ce40 0x7efcec0d0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58800/0 (socket says 172.21.15.46:58800) 2024-01-26T23:49:53.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.234+0000 7efceb7fe700 1 -- 172.21.15.46:0/1198360970 learned_addr learned my addr 172.21.15.46:0/1198360970 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:53.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.234+0000 7efceaffd700 1 --2- 172.21.15.46:0/1198360970 >> v2:172.21.15.106:3300/0 conn(0x7efcec15d210 0x7efcec0cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:53.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.234+0000 7efcea7fc700 1 -- 172.21.15.46:0/1198360970 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486447587 0 0) 0x7efcdc001290 con 0x7efcec15ce40 2024-01-26T23:49:53.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.234+0000 7efcea7fc700 1 -- 172.21.15.46:0/1198360970 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcd4003570 con 0x7efcec15ce40 2024-01-26T23:49:53.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.234+0000 7efceaffd700 1 -- 172.21.15.46:0/1198360970 >> v1:172.21.15.106:6789/0 conn(0x7efcec0cbff0 legacy=0x7efcec0ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:53.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.234+0000 7efceaffd700 1 -- 172.21.15.46:0/1198360970 >> v1:172.21.15.46:6789/0 conn(0x7efcec15ce40 legacy=0x7efcec0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:53.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.234+0000 7efceaffd700 1 -- 172.21.15.46:0/1198360970 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec0cf660 con 0x7efcec15d210 2024-01-26T23:49:53.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.234+0000 7efceaffd700 1 --2- 172.21.15.46:0/1198360970 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efcec15d210 0x7efcec0cbab0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7efce00098e0 tx=0x7efce00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcc00f3d975494d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:53.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.234+0000 7efcea7fc700 1 -- 172.21.15.46:0/1198360970 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce000b640 con 0x7efcec15d210 2024-01-26T23:49:53.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.234+0000 7efcea7fc700 1 -- 172.21.15.46:0/1198360970 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce000b7a0 con 0x7efcec15d210 2024-01-26T23:49:53.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.234+0000 7efcea7fc700 1 -- 172.21.15.46:0/1198360970 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce000bb40 con 0x7efcec15d210 2024-01-26T23:49:53.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.235+0000 7efcf1a75700 1 -- 172.21.15.46:0/1198360970 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efcec15d210 msgr2=0x7efcec0cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:53.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.235+0000 7efcf1a75700 1 --2- 172.21.15.46:0/1198360970 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efcec15d210 0x7efcec0cbab0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7efce00098e0 tx=0x7efce00091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.235+0000 7efcf1a75700 1 -- 172.21.15.46:0/1198360970 shutdown_connections 2024-01-26T23:49:53.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.235+0000 7efcf1a75700 1 --2- 172.21.15.46:0/1198360970 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efcec15d210 0x7efcec0cbab0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.235+0000 7efcf1a75700 1 -- 172.21.15.46:0/1198360970 >> 172.21.15.46:0/1198360970 conn(0x7efcec06f840 msgr2=0x7efcec0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:53.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.235+0000 7efcf1a75700 1 -- 172.21.15.46:0/1198360970 shutdown_connections 2024-01-26T23:49:53.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.235+0000 7efcf1a75700 1 -- 172.21.15.46:0/1198360970 wait complete. 2024-01-26T23:49:53.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.237+0000 7efcf1a75700 1 Processor -- start 2024-01-26T23:49:53.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.237+0000 7efcf1a75700 1 -- start start 2024-01-26T23:49:53.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.239+0000 7efcf1a75700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0cbff0 0x7efcec157ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:53.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.239+0000 7efcf1a75700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efcec15ce40 0x7efcec151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.239+0000 7efcf1a75700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efcec15d210 0x7efcec152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.239+0000 7efcf1a75700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec158320 con 0x7efcec0cbff0 2024-01-26T23:49:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.239+0000 7efcf1a75700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec158600 con 0x7efcec15ce40 2024-01-26T23:49:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.239+0000 7efcf1a75700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcec1588e0 con 0x7efcec15d210 2024-01-26T23:49:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.239+0000 7efceb7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0cbff0 0x7efcec157ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.239+0000 7efceb7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0cbff0 0x7efcec157ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54370/0 (socket says 172.21.15.46:54370) 2024-01-26T23:49:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.239+0000 7efceb7fe700 1 -- 172.21.15.46:0/2811422496 learned_addr learned my addr 172.21.15.46:0/2811422496 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.239+0000 7efceaffd700 1 --2- 172.21.15.46:0/2811422496 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efcec15ce40 0x7efcec151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.239+0000 7efcebfff700 1 --2- 172.21.15.46:0/2811422496 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efcec15d210 0x7efcec152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.239+0000 7efceb7fe700 1 -- 172.21.15.46:0/2811422496 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efcec15d210 msgr2=0x7efcec152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.239+0000 7efceb7fe700 1 --2- 172.21.15.46:0/2811422496 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efcec15d210 0x7efcec152440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.239+0000 7efceb7fe700 1 -- 172.21.15.46:0/2811422496 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efcec15ce40 msgr2=0x7efcec151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.239+0000 7efceb7fe700 1 --2- 172.21.15.46:0/2811422496 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efcec15ce40 0x7efcec151f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.239+0000 7efceb7fe700 1 -- 172.21.15.46:0/2811422496 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce0009420 con 0x7efcec0cbff0 2024-01-26T23:49:53.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.239+0000 7efceb7fe700 1 --2- 172.21.15.46:0/2811422496 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0cbff0 0x7efcec157ad0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00c7c0 tx=0x7efcdc007ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:53.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.240+0000 7efcd37fe700 1 -- 172.21.15.46:0/2811422496 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc0073b0 con 0x7efcec0cbff0 2024-01-26T23:49:53.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.240+0000 7efcd37fe700 1 -- 172.21.15.46:0/2811422496 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcdc007510 con 0x7efcec0cbff0 2024-01-26T23:49:53.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.240+0000 7efcf1a75700 1 -- 172.21.15.46:0/2811422496 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcec154040 con 0x7efcec0cbff0 2024-01-26T23:49:53.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.240+0000 7efcd37fe700 1 -- 172.21.15.46:0/2811422496 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc005a40 con 0x7efcec0cbff0 2024-01-26T23:49:53.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.240+0000 7efcf1a75700 1 -- 172.21.15.46:0/2811422496 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcec158010 con 0x7efcec0cbff0 2024-01-26T23:49:53.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.241+0000 7efcf1a75700 1 -- 172.21.15.46:0/2811422496 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcec154560 con 0x7efcec0cbff0 2024-01-26T23:49:53.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.241+0000 7efcd37fe700 1 -- 172.21.15.46:0/2811422496 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 72) v1 ==== 84901+0+0 (secure 0 0 0) 0x7efcdc01b3b0 con 0x7efcec0cbff0 2024-01-26T23:49:53.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.242+0000 7efcd37fe700 1 --2- 172.21.15.46:0/2811422496 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7efcd80670e0 0x7efcd8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:53.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.242+0000 7efceaffd700 1 -- 172.21.15.46:0/2811422496 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7efcd80670e0 msgr2=0x7efcd8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:53.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.242+0000 7efcd37fe700 1 -- 172.21.15.46:0/2811422496 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6675+0+0 (secure 0 0 0) 0x7efcdc084030 con 0x7efcec0cbff0 2024-01-26T23:49:53.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.242+0000 7efceaffd700 1 --2- 172.21.15.46:0/2811422496 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7efcd80670e0 0x7efcd8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:53.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.247+0000 7efcd37fe700 1 -- 172.21.15.46:0/2811422496 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efcec154560 con 0x7efcec0cbff0 2024-01-26T23:49:53.430 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:53.429+0000 7fd054cba1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:49:53.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.443+0000 7efceaffd700 1 -- 172.21.15.46:0/2811422496 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7efcd80670e0 msgr2=0x7efcd8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:53.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.443+0000 7efceaffd700 1 --2- 172.21.15.46:0/2811422496 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7efcd80670e0 0x7efcd8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:49:53.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.499+0000 7efcf1a75700 1 -- 172.21.15.46:0/2811422496 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7efcec0cc450 con 0x7efcec0cbff0 2024-01-26T23:49:53.501 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-26T23:49:53.500+0000 7f33ffe86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-26T23:49:53.503 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:53.503+0000 7fd054cba1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:49:53.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.504+0000 7efcd37fe700 1 -- 172.21.15.46:0/2811422496 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v71) v1 ==== 161+0+0 (secure 0 0 0) 0x7efcec0cc450 con 0x7efcec0cbff0 2024-01-26T23:49:53.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.507+0000 7efcf1a75700 1 -- 172.21.15.46:0/2811422496 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7efcd80670e0 msgr2=0x7efcd8069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:53.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.507+0000 7efcf1a75700 1 --2- 172.21.15.46:0/2811422496 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7efcd80670e0 0x7efcd8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.507+0000 7efcf1a75700 1 -- 172.21.15.46:0/2811422496 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0cbff0 msgr2=0x7efcec157ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:53.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.507+0000 7efcf1a75700 1 --2- 172.21.15.46:0/2811422496 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0cbff0 0x7efcec157ad0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00c7c0 tx=0x7efcdc007ae0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.507+0000 7efcf1a75700 1 -- 172.21.15.46:0/2811422496 shutdown_connections 2024-01-26T23:49:53.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.507+0000 7efcf1a75700 1 --2- 172.21.15.46:0/2811422496 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7efcd80670e0 0x7efcd8069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.507+0000 7efcf1a75700 1 --2- 172.21.15.46:0/2811422496 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0cbff0 0x7efcec157ad0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.507+0000 7efcf1a75700 1 --2- 172.21.15.46:0/2811422496 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efcec15d210 0x7efcec152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.507+0000 7efcf1a75700 1 --2- 172.21.15.46:0/2811422496 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efcec15ce40 0x7efcec151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.507+0000 7efcf1a75700 1 -- 172.21.15.46:0/2811422496 >> 172.21.15.46:0/2811422496 conn(0x7efcec06f840 msgr2=0x7efcec15adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:53.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.507+0000 7efcf1a75700 1 -- 172.21.15.46:0/2811422496 shutdown_connections 2024-01-26T23:49:53.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.507+0000 7efcf1a75700 1 -- 172.21.15.46:0/2811422496 wait complete. 2024-01-26T23:49:53.510 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-26T23:49:53.510 INFO:tasks.ceph.mds.a.smithi046.stderr: -8273> 2024-01-26T23:49:26.906+0000 7fd7131bb700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-26T23:49:53.523 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-26T23:49:53.542 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-26T23:49:53.662 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:53.662+0000 7fd054cba1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:49:53.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.663+0000 7effe950f700 1 Processor -- start 2024-01-26T23:49:53.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.663+0000 7effe950f700 1 -- start start 2024-01-26T23:49:53.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.664+0000 7effe950f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7effe40d0e60 0x7effe40adc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:53.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.664+0000 7effe950f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe40c3e50 con 0x7effe40cacf0 2024-01-26T23:49:53.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.664+0000 7effe950f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe40ad730 con 0x7effe40d1300 2024-01-26T23:49:53.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.664+0000 7effe950f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effe40cffb0 con 0x7effe40d0e60 2024-01-26T23:49:53.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.664+0000 7effe2ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7effe40cacf0 0x7effe40cb0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58804/0 (socket says 172.21.15.46:58804) 2024-01-26T23:49:53.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.664+0000 7effe2ffd700 1 -- 172.21.15.46:0/2764226661 learned_addr learned my addr 172.21.15.46:0/2764226661 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:53.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe27fc700 1 --2- 172.21.15.46:0/2764226661 >> v2:172.21.15.106:3300/0 conn(0x7effe40d0e60 0x7effe40adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:53.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe1ffb700 1 -- 172.21.15.46:0/2764226661 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947966639 0 0) 0x7effe40c3e50 con 0x7effe40cacf0 2024-01-26T23:49:53.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe1ffb700 1 -- 172.21.15.46:0/2764226661 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effcc003570 con 0x7effe40cacf0 2024-01-26T23:49:53.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe1ffb700 1 -- 172.21.15.46:0/2764226661 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961843577 0 0) 0x7effe40ad730 con 0x7effe40d1300 2024-01-26T23:49:53.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe1ffb700 1 -- 172.21.15.46:0/2764226661 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effe40c3e50 con 0x7effe40d1300 2024-01-26T23:49:53.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe1ffb700 1 -- 172.21.15.46:0/2764226661 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2760069796 0 0) 0x7effcc003570 con 0x7effe40cacf0 2024-01-26T23:49:53.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe1ffb700 1 -- 172.21.15.46:0/2764226661 >> v2:172.21.15.106:3300/0 conn(0x7effe40d0e60 msgr2=0x7effe40adc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:53.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe1ffb700 1 --2- 172.21.15.46:0/2764226661 >> v2:172.21.15.106:3300/0 conn(0x7effe40d0e60 0x7effe40adc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe1ffb700 1 -- 172.21.15.46:0/2764226661 >> v1:172.21.15.106:6789/0 conn(0x7effe40d1300 legacy=0x7effe40ae150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe1ffb700 1 -- 172.21.15.46:0/2764226661 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe40cf430 con 0x7effe40cacf0 2024-01-26T23:49:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe27fc700 1 --2- 172.21.15.46:0/2764226661 >> v2:172.21.15.106:3300/0 conn(0x7effe40d0e60 0x7effe40adc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe1ffb700 1 -- 172.21.15.46:0/2764226661 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7effd40015f0 con 0x7effe40cacf0 2024-01-26T23:49:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe1ffb700 1 -- 172.21.15.46:0/2764226661 >> v1:172.21.15.46:6789/0 conn(0x7effe40cacf0 legacy=0x7effe40cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe1ffb700 1 --2- 172.21.15.46:0/2764226661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effcc003bf0 0x7effcc003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe1ffb700 1 --2- 172.21.15.46:0/2764226661 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effcc004810 0x7effcc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe1ffb700 1 --2- 172.21.15.46:0/2764226661 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effcc005230 0x7effcc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe1ffb700 1 -- 172.21.15.46:0/2764226661 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe40cffb0 con 0x7effcc003bf0 2024-01-26T23:49:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe1ffb700 1 -- 172.21.15.46:0/2764226661 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc00c200 con 0x7effcc004810 2024-01-26T23:49:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe1ffb700 1 -- 172.21.15.46:0/2764226661 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effcc00c3c0 con 0x7effcc005230 2024-01-26T23:49:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe2ffd700 1 --2- 172.21.15.46:0/2764226661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effcc003bf0 0x7effcc003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.666+0000 7effe27fc700 1 --2- 172.21.15.46:0/2764226661 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effcc004810 0x7effcc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.665+0000 7effe37fe700 1 --2- 172.21.15.46:0/2764226661 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effcc005230 0x7effcc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.666+0000 7effe2ffd700 1 -- 172.21.15.46:0/2764226661 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effcc005230 msgr2=0x7effcc00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.666+0000 7effe2ffd700 1 --2- 172.21.15.46:0/2764226661 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effcc005230 0x7effcc00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.666+0000 7effe2ffd700 1 -- 172.21.15.46:0/2764226661 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effcc004810 msgr2=0x7effcc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.666+0000 7effe2ffd700 1 --2- 172.21.15.46:0/2764226661 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effcc004810 0x7effcc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.666+0000 7effe2ffd700 1 -- 172.21.15.46:0/2764226661 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effcc003570 con 0x7effcc003bf0 2024-01-26T23:49:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.666+0000 7effe37fe700 1 -- 172.21.15.46:0/2764226661 reap_dead start 2024-01-26T23:49:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.666+0000 7effe27fc700 1 --2- 172.21.15.46:0/2764226661 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effcc004810 0x7effcc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.666+0000 7effe2ffd700 1 --2- 172.21.15.46:0/2764226661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effcc003bf0 0x7effcc003fe0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7effd400c2a0 tx=0x7effd40078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ad2f50374a325 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.666+0000 7effe1ffb700 1 -- 172.21.15.46:0/2764226661 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd400cb70 con 0x7effcc003bf0 2024-01-26T23:49:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.666+0000 7effe1ffb700 1 -- 172.21.15.46:0/2764226661 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd400ccf0 con 0x7effcc003bf0 2024-01-26T23:49:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.666+0000 7effe1ffb700 1 -- 172.21.15.46:0/2764226661 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4007320 con 0x7effcc003bf0 2024-01-26T23:49:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.667+0000 7effe950f700 1 -- 172.21.15.46:0/2764226661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effcc003bf0 msgr2=0x7effcc003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.667+0000 7effe950f700 1 --2- 172.21.15.46:0/2764226661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effcc003bf0 0x7effcc003fe0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7effd400c2a0 tx=0x7effd40078a0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.667+0000 7effe950f700 1 -- 172.21.15.46:0/2764226661 shutdown_connections 2024-01-26T23:49:53.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.667+0000 7effe950f700 1 --2- 172.21.15.46:0/2764226661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effcc003bf0 0x7effcc003fe0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.667+0000 7effe950f700 1 -- 172.21.15.46:0/2764226661 >> 172.21.15.46:0/2764226661 conn(0x7effe406f840 msgr2=0x7effe40d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:53.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.667+0000 7effe950f700 1 -- 172.21.15.46:0/2764226661 shutdown_connections 2024-01-26T23:49:53.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.667+0000 7effe950f700 1 -- 172.21.15.46:0/2764226661 wait complete. 2024-01-26T23:49:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.668+0000 7effe950f700 1 Processor -- start 2024-01-26T23:49:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.668+0000 7effe950f700 1 -- start start 2024-01-26T23:49:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.669+0000 7effe950f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effcc003bf0 0x7effe40b5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.669+0000 7effe950f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effe40c6750 0x7effe4169fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.669+0000 7effe950f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effe416a4f0 0x7effe416a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.669+0000 7effe950f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe4155940 con 0x7effcc003bf0 2024-01-26T23:49:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.669+0000 7effe950f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe40c9780 con 0x7effe416a4f0 2024-01-26T23:49:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.669+0000 7effe950f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe416aea0 con 0x7effe40c6750 2024-01-26T23:49:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.669+0000 7effe2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effcc003bf0 0x7effe40b5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.669+0000 7effe27fc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effe40c6750 0x7effe4169fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.669+0000 7effe37fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effe416a4f0 0x7effe416a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.670+0000 7effe27fc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effe40c6750 0x7effe4169fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:37118/0 (socket says 172.21.15.46:37118) 2024-01-26T23:49:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.669+0000 7effe2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effcc003bf0 0x7effe40b5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54410/0 (socket says 172.21.15.46:54410) 2024-01-26T23:49:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.670+0000 7effe27fc700 1 -- 172.21.15.46:0/1073520438 learned_addr learned my addr 172.21.15.46:0/1073520438 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.670+0000 7effe27fc700 1 -- 172.21.15.46:0/1073520438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effe416a4f0 msgr2=0x7effe416a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.670+0000 7effe27fc700 1 --2- 172.21.15.46:0/1073520438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effe416a4f0 0x7effe416a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.670+0000 7effe27fc700 1 -- 172.21.15.46:0/1073520438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effcc003bf0 msgr2=0x7effe40b5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.670+0000 7effe27fc700 1 --2- 172.21.15.46:0/1073520438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effcc003bf0 0x7effe40b5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.670+0000 7effe27fc700 1 -- 172.21.15.46:0/1073520438 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe40b5f20 con 0x7effe40c6750 2024-01-26T23:49:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.670+0000 7effe37fe700 1 --2- 172.21.15.46:0/1073520438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effe416a4f0 0x7effe416a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.670+0000 7effe27fc700 1 --2- 172.21.15.46:0/1073520438 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effe40c6750 0x7effe4169fb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7effd8011890 tx=0x7effd8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.670+0000 7effcaffd700 1 -- 172.21.15.46:0/1073520438 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd8015670 con 0x7effe40c6750 2024-01-26T23:49:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.670+0000 7effcaffd700 1 -- 172.21.15.46:0/1073520438 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd80157d0 con 0x7effe40c6750 2024-01-26T23:49:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.670+0000 7effe950f700 1 -- 172.21.15.46:0/1073520438 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe40c9130 con 0x7effe40c6750 2024-01-26T23:49:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.670+0000 7effcaffd700 1 -- 172.21.15.46:0/1073520438 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd8015b90 con 0x7effe40c6750 2024-01-26T23:49:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.670+0000 7effe950f700 1 -- 172.21.15.46:0/1073520438 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe4155d50 con 0x7effe40c6750 2024-01-26T23:49:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.672+0000 7effe950f700 1 -- 172.21.15.46:0/1073520438 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe40c8f60 con 0x7effe40c6750 2024-01-26T23:49:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.672+0000 7effcaffd700 1 -- 172.21.15.46:0/1073520438 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 72) v1 ==== 84901+0+0 (secure 0 0 0) 0x7effd8013410 con 0x7effe40c6750 2024-01-26T23:49:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.672+0000 7effcaffd700 1 --2- 172.21.15.46:0/1073520438 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7effd00670e0 0x7effd0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.672+0000 7effcaffd700 1 -- 172.21.15.46:0/1073520438 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6761+0+0 (secure 0 0 0) 0x7effd8085390 con 0x7effe40c6750 2024-01-26T23:49:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.672+0000 7effe2ffd700 1 -- 172.21.15.46:0/1073520438 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7effd00670e0 msgr2=0x7effd0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:49:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.672+0000 7effe2ffd700 1 --2- 172.21.15.46:0/1073520438 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7effd00670e0 0x7effd0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:49:53.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.677+0000 7effcaffd700 1 -- 172.21.15.46:0/1073520438 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7effd8051f60 con 0x7effe40c6750 2024-01-26T23:49:53.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.689+0000 7effcaffd700 1 -- 172.21.15.46:0/1073520438 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mgrmap(e 73) v1 ==== 84453+0+0 (secure 0 0 0) 0x7effd8052330 con 0x7effe40c6750 2024-01-26T23:49:53.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.689+0000 7effcaffd700 1 -- 172.21.15.46:0/1073520438 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7effd00670e0 msgr2=0x7effd0069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:49:53.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.689+0000 7effcaffd700 1 --2- 172.21.15.46:0/1073520438 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7effd00670e0 0x7effd0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:53.747 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:53.746+0000 7fcfd35a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:53.747 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:53.746+0000 7fcfd35a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:53.748 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:53.746+0000 7fcfd35a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:53.748 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:53.746+0000 7fcfd35a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:53.748 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:53.746+0000 7fcfd35a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:53.748 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:53.747+0000 7fcfd05a2700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:53.748 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:53.747+0000 7fcfd05a2700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:53.748 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:53.747+0000 7fcfd05a2700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:53.748 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:53.747+0000 7fcfd05a2700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:53.748 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:49:53.747+0000 7fcfd05a2700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:49:53.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:53.940+0000 7effe950f700 1 -- 172.21.15.46:0/1073520438 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7effe404c710 con 0x7effe40c6750 2024-01-26T23:49:54.052 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:54.051+0000 7ffb394e41c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:49:54.305 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:54.304+0000 7ffb394e41c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:49:54.389 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:54.388+0000 7ffb394e41c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:49:54.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.507+0000 7effcaffd700 1 -- 172.21.15.46:0/1073520438 <== mon.2 v2:172.21.15.106:3301/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v72) v1 ==== 104+0+0 (secure 0 0 0) 0x7effd806f850 con 0x7effe40c6750 2024-01-26T23:49:54.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.510+0000 7effe950f700 1 -- 172.21.15.46:0/1073520438 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effe40c6750 msgr2=0x7effe4169fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:54.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.510+0000 7effe950f700 1 --2- 172.21.15.46:0/1073520438 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effe40c6750 0x7effe4169fb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7effd8011890 tx=0x7effd8008040 comp rx=0 tx=0).stop 2024-01-26T23:49:54.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.510+0000 7effe950f700 1 -- 172.21.15.46:0/1073520438 shutdown_connections 2024-01-26T23:49:54.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.510+0000 7effe950f700 1 --2- 172.21.15.46:0/1073520438 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7effd00670e0 0x7effd0069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.510+0000 7effe950f700 1 --2- 172.21.15.46:0/1073520438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effcc003bf0 0x7effe40b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.510+0000 7effe950f700 1 --2- 172.21.15.46:0/1073520438 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effe416a4f0 0x7effe416a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.510+0000 7effe950f700 1 --2- 172.21.15.46:0/1073520438 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effe40c6750 0x7effe4169fb0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.510+0000 7effe950f700 1 -- 172.21.15.46:0/1073520438 >> 172.21.15.46:0/1073520438 conn(0x7effe406f840 msgr2=0x7effe4069160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.510+0000 7effe950f700 1 -- 172.21.15.46:0/1073520438 shutdown_connections 2024-01-26T23:49:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.510+0000 7effe950f700 1 -- 172.21.15.46:0/1073520438 wait complete. 2024-01-26T23:49:54.526 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-26T23:49:54.571 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:49:54.570+0000 7ffb394e41c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:49:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.702+0000 7f16969f4700 1 Processor -- start 2024-01-26T23:49:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.702+0000 7f16969f4700 1 -- start start 2024-01-26T23:49:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.703+0000 7f16969f4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16900ca0e0 0x7f16900ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.703+0000 7f16969f4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f169006c0b0 0x7f16900afe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.703+0000 7f16969f4700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1690158010 con 0x7f169006c520 2024-01-26T23:49:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.703+0000 7f16969f4700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16900cada0 con 0x7f169006c0b0 2024-01-26T23:49:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.703+0000 7f16969f4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16901582f0 con 0x7f16900ca0e0 2024-01-26T23:49:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.703+0000 7f168f7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f169006c0b0 0x7f16900afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.703+0000 7f1694f91700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f169006c520 0x7f16900b3a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58814/0 (socket says 172.21.15.46:58814) 2024-01-26T23:49:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.703+0000 7f1694f91700 1 -- 172.21.15.46:0/2002776502 learned_addr learned my addr 172.21.15.46:0/2002776502 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.703+0000 7f168ffff700 1 --2- 172.21.15.46:0/2002776502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16900ca0e0 0x7f16900ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.704+0000 7f168f7fe700 1 -- 172.21.15.46:0/2002776502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16900ca0e0 msgr2=0x7f16900ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.704+0000 7f168f7fe700 1 --2- 172.21.15.46:0/2002776502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16900ca0e0 0x7f16900ca4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.704+0000 7f168f7fe700 1 -- 172.21.15.46:0/2002776502 >> v1:172.21.15.46:6789/0 conn(0x7f169006c520 legacy=0x7f16900b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.704+0000 7f168effd700 1 -- 172.21.15.46:0/2002776502 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092134216 0 0) 0x7f1690158010 con 0x7f169006c520 2024-01-26T23:49:54.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.704+0000 7f168f7fe700 1 -- 172.21.15.46:0/2002776502 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16901585d0 con 0x7f169006c0b0 2024-01-26T23:49:54.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.704+0000 7f168f7fe700 1 --2- 172.21.15.46:0/2002776502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f169006c0b0 0x7f16900afe40 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f16840098e0 tx=0x7f16840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d146047352ddb4 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:54.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.704+0000 7f168effd700 1 -- 172.21.15.46:0/2002776502 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168400b640 con 0x7f169006c0b0 2024-01-26T23:49:54.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.704+0000 7f168effd700 1 -- 172.21.15.46:0/2002776502 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168400b7a0 con 0x7f169006c0b0 2024-01-26T23:49:54.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.704+0000 7f168effd700 1 -- 172.21.15.46:0/2002776502 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168400bb40 con 0x7f169006c0b0 2024-01-26T23:49:54.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.705+0000 7f16969f4700 1 -- 172.21.15.46:0/2002776502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f169006c0b0 msgr2=0x7f16900afe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:54.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.705+0000 7f16969f4700 1 --2- 172.21.15.46:0/2002776502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f169006c0b0 0x7f16900afe40 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f16840098e0 tx=0x7f16840091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:54.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.705+0000 7f16969f4700 1 -- 172.21.15.46:0/2002776502 shutdown_connections 2024-01-26T23:49:54.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.705+0000 7f16969f4700 1 --2- 172.21.15.46:0/2002776502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16900ca0e0 0x7f16900ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:54.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.705+0000 7f16969f4700 1 --2- 172.21.15.46:0/2002776502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f169006c0b0 0x7f16900afe40 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:54.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.705+0000 7f16969f4700 1 -- 172.21.15.46:0/2002776502 >> 172.21.15.46:0/2002776502 conn(0x7f16900ce530 msgr2=0x7f16900ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:54.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.705+0000 7f16969f4700 1 -- 172.21.15.46:0/2002776502 shutdown_connections 2024-01-26T23:49:54.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.705+0000 7f16969f4700 1 -- 172.21.15.46:0/2002776502 wait complete. 2024-01-26T23:49:54.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.706+0000 7f16969f4700 1 Processor -- start 2024-01-26T23:49:54.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.706+0000 7f16969f4700 1 -- start start 2024-01-26T23:49:54.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.707+0000 7f16969f4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f169006c0b0 0x7f16900c6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:54.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.707+0000 7f16969f4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f169006c520 0x7f16900c6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:54.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.707+0000 7f16969f4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16900ca0e0 0x7f169016b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:54.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.707+0000 7f16969f4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16900c7900 con 0x7f16900ca0e0 2024-01-26T23:49:54.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.707+0000 7f16969f4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16900c6ae0 con 0x7f169006c520 2024-01-26T23:49:54.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.707+0000 7f16969f4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16900c6d80 con 0x7f169006c0b0 2024-01-26T23:49:54.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.707+0000 7f1694f91700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16900ca0e0 0x7f169016b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:54.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.707+0000 7f1694f91700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16900ca0e0 0x7f169016b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54426/0 (socket says 172.21.15.46:54426) 2024-01-26T23:49:54.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.707+0000 7f1694f91700 1 -- 172.21.15.46:0/3890243182 learned_addr learned my addr 172.21.15.46:0/3890243182 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:54.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.707+0000 7f168ffff700 1 --2- 172.21.15.46:0/3890243182 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f169006c0b0 0x7f16900c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:54.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.707+0000 7f168f7fe700 1 --2- 172.21.15.46:0/3890243182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f169006c520 0x7f16900c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:54.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.707+0000 7f1694f91700 1 -- 172.21.15.46:0/3890243182 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f169006c0b0 msgr2=0x7f16900c6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:54.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.707+0000 7f1694f91700 1 --2- 172.21.15.46:0/3890243182 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f169006c0b0 0x7f16900c6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:54.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.707+0000 7f1694f91700 1 -- 172.21.15.46:0/3890243182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f169006c520 msgr2=0x7f16900c6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:54.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.707+0000 7f1694f91700 1 --2- 172.21.15.46:0/3890243182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f169006c520 0x7f16900c6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:54.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.707+0000 7f1694f91700 1 -- 172.21.15.46:0/3890243182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1684009420 con 0x7f16900ca0e0 2024-01-26T23:49:54.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.707+0000 7f168ffff700 1 --2- 172.21.15.46:0/3890243182 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f169006c0b0 0x7f16900c6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:54.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.708+0000 7f1694f91700 1 --2- 172.21.15.46:0/3890243182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16900ca0e0 0x7f169016b0f0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f168800ccc0 tx=0x7f16880019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:54.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.708+0000 7f1677fff700 1 -- 172.21.15.46:0/3890243182 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168800ce70 con 0x7f16900ca0e0 2024-01-26T23:49:54.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.708+0000 7f1677fff700 1 -- 172.21.15.46:0/3890243182 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168800ab70 con 0x7f16900ca0e0 2024-01-26T23:49:54.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.708+0000 7f1677fff700 1 -- 172.21.15.46:0/3890243182 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16880070f0 con 0x7f16900ca0e0 2024-01-26T23:49:54.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.708+0000 7f16969f4700 1 -- 172.21.15.46:0/3890243182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16900c7340 con 0x7f16900ca0e0 2024-01-26T23:49:54.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.708+0000 7f16969f4700 1 -- 172.21.15.46:0/3890243182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1690152570 con 0x7f16900ca0e0 2024-01-26T23:49:54.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.709+0000 7f16969f4700 1 -- 172.21.15.46:0/3890243182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169004c710 con 0x7f16900ca0e0 2024-01-26T23:49:54.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.710+0000 7f1677fff700 1 -- 172.21.15.46:0/3890243182 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 74) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f16880074a0 con 0x7f16900ca0e0 2024-01-26T23:49:54.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.710+0000 7f1677fff700 1 --2- 172.21.15.46:0/3890243182 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f167c066d80 0x7f167c069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:54.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.710+0000 7f1677fff700 1 -- 172.21.15.46:0/3890243182 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 7062+0+0 (secure 0 0 0) 0x7f1688084b50 con 0x7f16900ca0e0 2024-01-26T23:49:54.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.710+0000 7f168ffff700 1 --2- 172.21.15.46:0/3890243182 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f167c066d80 0x7f167c069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:54.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.711+0000 7f168ffff700 1 --2- 172.21.15.46:0/3890243182 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f167c066d80 0x7f167c069230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1680009ce0 tx=0x7f1680008040 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:54.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.714+0000 7f1677fff700 1 -- 172.21.15.46:0/3890243182 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1688051ba0 con 0x7f16900ca0e0 2024-01-26T23:49:54.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:54.920+0000 7f16969f4700 1 -- 172.21.15.46:0/3890243182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f16901588b0 con 0x7f16900ca0e0 2024-01-26T23:49:55.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.695+0000 7f1677fff700 1 -- 172.21.15.46:0/3890243182 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v149) v1 ==== 181+0+0 (secure 0 0 0) 0x7f16880551c0 con 0x7f16900ca0e0 2024-01-26T23:49:55.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.699+0000 7f16969f4700 1 -- 172.21.15.46:0/3890243182 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f167c066d80 msgr2=0x7f167c069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:55.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.699+0000 7f16969f4700 1 --2- 172.21.15.46:0/3890243182 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f167c066d80 0x7f167c069230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1680009ce0 tx=0x7f1680008040 comp rx=0 tx=0).stop 2024-01-26T23:49:55.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.699+0000 7f16969f4700 1 -- 172.21.15.46:0/3890243182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16900ca0e0 msgr2=0x7f169016b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:55.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.699+0000 7f16969f4700 1 --2- 172.21.15.46:0/3890243182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16900ca0e0 0x7f169016b0f0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f168800ccc0 tx=0x7f16880019e0 comp rx=0 tx=0).stop 2024-01-26T23:49:55.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.699+0000 7f16969f4700 1 -- 172.21.15.46:0/3890243182 shutdown_connections 2024-01-26T23:49:55.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.699+0000 7f16969f4700 1 --2- 172.21.15.46:0/3890243182 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f167c066d80 0x7f167c069230 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:55.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.699+0000 7f16969f4700 1 --2- 172.21.15.46:0/3890243182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16900ca0e0 0x7f169016b0f0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:55.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.699+0000 7f16969f4700 1 --2- 172.21.15.46:0/3890243182 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f169006c0b0 0x7f16900c6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:55.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.699+0000 7f16969f4700 1 --2- 172.21.15.46:0/3890243182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f169006c520 0x7f16900c6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:55.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.699+0000 7f16969f4700 1 -- 172.21.15.46:0/3890243182 >> 172.21.15.46:0/3890243182 conn(0x7f16900ce530 msgr2=0x7f16900ad640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:55.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.699+0000 7f16969f4700 1 -- 172.21.15.46:0/3890243182 shutdown_connections 2024-01-26T23:49:55.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.699+0000 7f16969f4700 1 -- 172.21.15.46:0/3890243182 wait complete. 2024-01-26T23:49:55.701 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-01-26T23:49:55.716 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-26T23:49:55.831 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:49:55.833 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:49:55.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.903+0000 7f5a7dd75700 1 Processor -- start 2024-01-26T23:49:55.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.903+0000 7f5a7dd75700 1 -- start start 2024-01-26T23:49:55.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.903+0000 7f5a7dd75700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a78159750 0x7f5a78069650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.903+0000 7f5a7dd75700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a78069b90 0x7f5a7815d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.904+0000 7f5a7dd75700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a780691c0 con 0x7f5a78159b20 2024-01-26T23:49:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.904+0000 7f5a7dd75700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a7806a5e0 con 0x7f5a78159750 2024-01-26T23:49:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.904+0000 7f5a7dd75700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7815dfd0 con 0x7f5a78069b90 2024-01-26T23:49:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.904+0000 7f5a777fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a78159750 0x7f5a78069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.904+0000 7f5a76ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5a78159b20 0x7f5a780ade00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58818/0 (socket says 172.21.15.46:58818) 2024-01-26T23:49:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.904+0000 7f5a777fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a78159750 0x7f5a78069650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54436/0 (socket says 172.21.15.46:54436) 2024-01-26T23:49:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.904+0000 7f5a76ffd700 1 -- 172.21.15.46:0/4213704113 learned_addr learned my addr 172.21.15.46:0/4213704113 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.904+0000 7f5a77fff700 1 --2- 172.21.15.46:0/4213704113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a78069b90 0x7f5a7815d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.904+0000 7f5a777fe700 1 -- 172.21.15.46:0/4213704113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a78069b90 msgr2=0x7f5a7815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.904+0000 7f5a777fe700 1 --2- 172.21.15.46:0/4213704113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a78069b90 0x7f5a7815d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.904+0000 7f5a777fe700 1 -- 172.21.15.46:0/4213704113 >> v1:172.21.15.46:6789/0 conn(0x7f5a78159b20 legacy=0x7f5a780ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.904+0000 7f5a777fe700 1 -- 172.21.15.46:0/4213704113 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a78068b70 con 0x7f5a78159750 2024-01-26T23:49:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.904+0000 7f5a767fc700 1 -- 172.21.15.46:0/4213704113 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139317333 0 0) 0x7f5a780691c0 con 0x7f5a78159b20 2024-01-26T23:49:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.904+0000 7f5a777fe700 1 --2- 172.21.15.46:0/4213704113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a78159750 0x7f5a78069650 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f5a680098e0 tx=0x7f5a680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ae98028a4387c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.904+0000 7f5a767fc700 1 -- 172.21.15.46:0/4213704113 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6800b640 con 0x7f5a78159750 2024-01-26T23:49:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.904+0000 7f5a767fc700 1 -- 172.21.15.46:0/4213704113 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6800b7a0 con 0x7f5a78159750 2024-01-26T23:49:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.904+0000 7f5a767fc700 1 -- 172.21.15.46:0/4213704113 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6800bb40 con 0x7f5a78159750 2024-01-26T23:49:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.905+0000 7f5a7dd75700 1 -- 172.21.15.46:0/4213704113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a78159750 msgr2=0x7f5a78069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.905+0000 7f5a7dd75700 1 --2- 172.21.15.46:0/4213704113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a78159750 0x7f5a78069650 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f5a680098e0 tx=0x7f5a680091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.905+0000 7f5a7dd75700 1 -- 172.21.15.46:0/4213704113 shutdown_connections 2024-01-26T23:49:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.905+0000 7f5a7dd75700 1 --2- 172.21.15.46:0/4213704113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a78069b90 0x7f5a7815d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.905+0000 7f5a7dd75700 1 --2- 172.21.15.46:0/4213704113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a78159750 0x7f5a78069650 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.905+0000 7f5a7dd75700 1 -- 172.21.15.46:0/4213704113 >> 172.21.15.46:0/4213704113 conn(0x7f5a780ce530 msgr2=0x7f5a780ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.905+0000 7f5a7dd75700 1 -- 172.21.15.46:0/4213704113 shutdown_connections 2024-01-26T23:49:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.905+0000 7f5a7dd75700 1 -- 172.21.15.46:0/4213704113 wait complete. 2024-01-26T23:49:55.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.906+0000 7f5a7dd75700 1 Processor -- start 2024-01-26T23:49:55.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.907+0000 7f5a7dd75700 1 -- start start 2024-01-26T23:49:55.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.907+0000 7f5a7dd75700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a78069b90 0x7f5a7814a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:55.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.907+0000 7f5a7dd75700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a78159750 0x7f5a780afda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:55.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.907+0000 7f5a7dd75700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a78159b20 0x7f5a780b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:55.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.907+0000 7f5a7dd75700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7815ac10 con 0x7f5a78069b90 2024-01-26T23:49:55.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.907+0000 7f5a7dd75700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a780c5540 con 0x7f5a78159b20 2024-01-26T23:49:55.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.907+0000 7f5a7dd75700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a780c5820 con 0x7f5a78159750 2024-01-26T23:49:55.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.907+0000 7f5a777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a78069b90 0x7f5a7814a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:55.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.907+0000 7f5a777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a78069b90 0x7f5a7814a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54444/0 (socket says 172.21.15.46:54444) 2024-01-26T23:49:55.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.907+0000 7f5a777fe700 1 -- 172.21.15.46:0/4065313506 learned_addr learned my addr 172.21.15.46:0/4065313506 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:55.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.908+0000 7f5a76ffd700 1 --2- 172.21.15.46:0/4065313506 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a78159750 0x7f5a780afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:55.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.908+0000 7f5a77fff700 1 --2- 172.21.15.46:0/4065313506 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a78159b20 0x7f5a780b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:55.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.908+0000 7f5a777fe700 1 -- 172.21.15.46:0/4065313506 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a78159750 msgr2=0x7f5a780afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:55.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.908+0000 7f5a777fe700 1 --2- 172.21.15.46:0/4065313506 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a78159750 0x7f5a780afda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:55.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.908+0000 7f5a777fe700 1 -- 172.21.15.46:0/4065313506 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a78159b20 msgr2=0x7f5a780b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:55.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.908+0000 7f5a777fe700 1 --2- 172.21.15.46:0/4065313506 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a78159b20 0x7f5a780b02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:55.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.908+0000 7f5a777fe700 1 -- 172.21.15.46:0/4065313506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a68009420 con 0x7f5a78069b90 2024-01-26T23:49:55.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.908+0000 7f5a777fe700 1 --2- 172.21.15.46:0/4065313506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a78069b90 0x7f5a7814a7d0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f5a7806f390 tx=0x7f5a68017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:55.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.908+0000 7f5a5f7fe700 1 -- 172.21.15.46:0/4065313506 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6800bb20 con 0x7f5a78069b90 2024-01-26T23:49:55.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.908+0000 7f5a5f7fe700 1 -- 172.21.15.46:0/4065313506 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a68000c90 con 0x7f5a78069b90 2024-01-26T23:49:55.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.908+0000 7f5a5f7fe700 1 -- 172.21.15.46:0/4065313506 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a68011650 con 0x7f5a78069b90 2024-01-26T23:49:55.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.908+0000 7f5a7dd75700 1 -- 172.21.15.46:0/4065313506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a780aed30 con 0x7f5a78069b90 2024-01-26T23:49:55.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.908+0000 7f5a7dd75700 1 -- 172.21.15.46:0/4065313506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a780af2f0 con 0x7f5a78069b90 2024-01-26T23:49:55.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.909+0000 7f5a7dd75700 1 -- 172.21.15.46:0/4065313506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7804c710 con 0x7f5a78069b90 2024-01-26T23:49:55.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.910+0000 7f5a5f7fe700 1 -- 172.21.15.46:0/4065313506 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 74) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f5a68011a80 con 0x7f5a78069b90 2024-01-26T23:49:55.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.910+0000 7f5a5f7fe700 1 --2- 172.21.15.46:0/4065313506 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5a64066d80 0x7f5a64069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:55.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.910+0000 7f5a5f7fe700 1 -- 172.21.15.46:0/4065313506 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6624+0+0 (secure 0 0 0) 0x7f5a68085ac0 con 0x7f5a78069b90 2024-01-26T23:49:55.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.910+0000 7f5a76ffd700 1 --2- 172.21.15.46:0/4065313506 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5a64066d80 0x7f5a64069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:55.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.911+0000 7f5a76ffd700 1 --2- 172.21.15.46:0/4065313506 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5a64066d80 0x7f5a64069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5a780b3d60 tx=0x7f5a6c007f50 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:55.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:55.915+0000 7f5a5f7fe700 1 -- 172.21.15.46:0/4065313506 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a68052c50 con 0x7f5a78069b90 2024-01-26T23:49:56.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.123+0000 7f5a7dd75700 1 -- 172.21.15.46:0/4065313506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5a78068e50 con 0x7f5a78069b90 2024-01-26T23:49:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.695+0000 7f5a5f7fe700 1 -- 172.21.15.46:0/4065313506 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 75) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f5a68053020 con 0x7f5a78069b90 2024-01-26T23:49:56.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.702+0000 7f5a5f7fe700 1 -- 172.21.15.46:0/4065313506 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v150) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5a680566e0 con 0x7f5a78069b90 2024-01-26T23:49:56.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.707+0000 7f5a7dd75700 1 -- 172.21.15.46:0/4065313506 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5a64066d80 msgr2=0x7f5a64069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:56.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.707+0000 7f5a7dd75700 1 --2- 172.21.15.46:0/4065313506 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5a64066d80 0x7f5a64069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5a780b3d60 tx=0x7f5a6c007f50 comp rx=0 tx=0).stop 2024-01-26T23:49:56.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.707+0000 7f5a7dd75700 1 -- 172.21.15.46:0/4065313506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a78069b90 msgr2=0x7f5a7814a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:56.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.707+0000 7f5a7dd75700 1 --2- 172.21.15.46:0/4065313506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a78069b90 0x7f5a7814a7d0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f5a7806f390 tx=0x7f5a68017000 comp rx=0 tx=0).stop 2024-01-26T23:49:56.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.707+0000 7f5a7dd75700 1 -- 172.21.15.46:0/4065313506 shutdown_connections 2024-01-26T23:49:56.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.707+0000 7f5a7dd75700 1 --2- 172.21.15.46:0/4065313506 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5a64066d80 0x7f5a64069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:56.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.707+0000 7f5a7dd75700 1 --2- 172.21.15.46:0/4065313506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a78069b90 0x7f5a7814a7d0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:56.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.707+0000 7f5a7dd75700 1 --2- 172.21.15.46:0/4065313506 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a78159b20 0x7f5a780b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:56.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.707+0000 7f5a7dd75700 1 --2- 172.21.15.46:0/4065313506 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a78159750 0x7f5a780afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:56.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.707+0000 7f5a7dd75700 1 -- 172.21.15.46:0/4065313506 >> 172.21.15.46:0/4065313506 conn(0x7f5a780ce530 msgr2=0x7f5a78158790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:56.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.707+0000 7f5a7dd75700 1 -- 172.21.15.46:0/4065313506 shutdown_connections 2024-01-26T23:49:56.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.707+0000 7f5a7dd75700 1 -- 172.21.15.46:0/4065313506 wait complete. 2024-01-26T23:49:56.709 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-01-26T23:49:56.727 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-26T23:49:56.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.908+0000 7f45a3727700 1 Processor -- start 2024-01-26T23:49:56.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.909+0000 7f45a3727700 1 -- start start 2024-01-26T23:49:56.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.910+0000 7f45a3727700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f459c158400 0x7f459c069b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:56.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.910+0000 7f45a3727700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f459c0cd9d0 0x7f459c0cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:56.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.910+0000 7f45a3727700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c068d80 con 0x7f459c0b37e0 2024-01-26T23:49:56.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.910+0000 7f45a3727700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459c069060 con 0x7f459c0cd9d0 2024-01-26T23:49:56.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.910+0000 7f45a3727700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c069340 con 0x7f459c158400 2024-01-26T23:49:56.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.910+0000 7f45a0cc2700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f459c0b37e0 0x7f459c0b3c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58822/0 (socket says 172.21.15.46:58822) 2024-01-26T23:49:56.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.910+0000 7f45a0cc2700 1 -- 172.21.15.46:0/1759973742 learned_addr learned my addr 172.21.15.46:0/1759973742 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:56.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.910+0000 7f45a14c3700 1 --2- 172.21.15.46:0/1759973742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f459c158400 0x7f459c069b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:56.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.911+0000 7f45a1cc4700 1 --2- 172.21.15.46:0/1759973742 >> v2:172.21.15.106:3300/0 conn(0x7f459c0cd9d0 0x7f459c0cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:56.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.911+0000 7f4593fff700 1 -- 172.21.15.46:0/1759973742 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019040363 0 0) 0x7f459c068d80 con 0x7f459c0b37e0 2024-01-26T23:49:56.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.911+0000 7f4593fff700 1 -- 172.21.15.46:0/1759973742 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4588003570 con 0x7f459c0b37e0 2024-01-26T23:49:56.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.911+0000 7f45a14c3700 1 -- 172.21.15.46:0/1759973742 >> v2:172.21.15.106:3300/0 conn(0x7f459c0cd9d0 msgr2=0x7f459c0cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:56.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.911+0000 7f45a14c3700 1 --2- 172.21.15.46:0/1759973742 >> v2:172.21.15.106:3300/0 conn(0x7f459c0cd9d0 0x7f459c0cde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:56.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.911+0000 7f45a14c3700 1 -- 172.21.15.46:0/1759973742 >> v1:172.21.15.46:6789/0 conn(0x7f459c0b37e0 legacy=0x7f459c0b3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.911+0000 7f45a14c3700 1 -- 172.21.15.46:0/1759973742 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c069620 con 0x7f459c158400 2024-01-26T23:49:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.911+0000 7f45a1cc4700 1 --2- 172.21.15.46:0/1759973742 >> v2:172.21.15.106:3300/0 conn(0x7f459c0cd9d0 0x7f459c0cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.911+0000 7f45a14c3700 1 --2- 172.21.15.46:0/1759973742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f459c158400 0x7f459c069b40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f45980098e0 tx=0x7f45980091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=849d3e73cc6b7b75 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.911+0000 7f4593fff700 1 -- 172.21.15.46:0/1759973742 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4598011630 con 0x7f459c158400 2024-01-26T23:49:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.911+0000 7f4593fff700 1 -- 172.21.15.46:0/1759973742 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4598011790 con 0x7f459c158400 2024-01-26T23:49:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.911+0000 7f4593fff700 1 -- 172.21.15.46:0/1759973742 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4598011b30 con 0x7f459c158400 2024-01-26T23:49:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.912+0000 7f45a3727700 1 -- 172.21.15.46:0/1759973742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f459c158400 msgr2=0x7f459c069b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.912+0000 7f45a3727700 1 --2- 172.21.15.46:0/1759973742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f459c158400 0x7f459c069b40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f45980098e0 tx=0x7f45980091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.912+0000 7f45a3727700 1 -- 172.21.15.46:0/1759973742 shutdown_connections 2024-01-26T23:49:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.912+0000 7f45a3727700 1 --2- 172.21.15.46:0/1759973742 >> v2:172.21.15.106:3300/0 conn(0x7f459c0cd9d0 0x7f459c0cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.912+0000 7f45a3727700 1 --2- 172.21.15.46:0/1759973742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f459c158400 0x7f459c069b40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.912+0000 7f45a3727700 1 -- 172.21.15.46:0/1759973742 >> 172.21.15.46:0/1759973742 conn(0x7f459c067930 msgr2=0x7f459c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.912+0000 7f45a3727700 1 -- 172.21.15.46:0/1759973742 shutdown_connections 2024-01-26T23:49:56.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.912+0000 7f45a3727700 1 -- 172.21.15.46:0/1759973742 wait complete. 2024-01-26T23:49:56.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.913+0000 7f45a3727700 1 Processor -- start 2024-01-26T23:49:56.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.914+0000 7f45a3727700 1 -- start start 2024-01-26T23:49:56.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.914+0000 7f45a3727700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f459c0b37e0 0x7f459c154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:56.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.914+0000 7f45a3727700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f459c0cd9d0 0x7f459c150b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:56.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.914+0000 7f45a3727700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f459c158400 0x7f459c14cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.914+0000 7f45a3727700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c160340 con 0x7f459c0cd9d0 2024-01-26T23:49:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.915+0000 7f45a3727700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c1605e0 con 0x7f459c0b37e0 2024-01-26T23:49:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.915+0000 7f45a3727700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c154770 con 0x7f459c158400 2024-01-26T23:49:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.915+0000 7f45a0cc2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f459c0cd9d0 0x7f459c150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.915+0000 7f45a0cc2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f459c0cd9d0 0x7f459c150b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54450/0 (socket says 172.21.15.46:54450) 2024-01-26T23:49:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.915+0000 7f45a0cc2700 1 -- 172.21.15.46:0/143988232 learned_addr learned my addr 172.21.15.46:0/143988232 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:56.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.915+0000 7f45a1cc4700 1 --2- 172.21.15.46:0/143988232 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f459c158400 0x7f459c14cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:56.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.915+0000 7f45a14c3700 1 --2- 172.21.15.46:0/143988232 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f459c0b37e0 0x7f459c154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:56.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.915+0000 7f45a0cc2700 1 -- 172.21.15.46:0/143988232 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f459c158400 msgr2=0x7f459c14cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:56.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.915+0000 7f45a0cc2700 1 --2- 172.21.15.46:0/143988232 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f459c158400 0x7f459c14cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:56.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.915+0000 7f45a0cc2700 1 -- 172.21.15.46:0/143988232 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f459c0b37e0 msgr2=0x7f459c154160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:56.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.915+0000 7f45a0cc2700 1 --2- 172.21.15.46:0/143988232 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f459c0b37e0 0x7f459c154160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.915+0000 7f45a0cc2700 1 -- 172.21.15.46:0/143988232 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4594009a20 con 0x7f459c0cd9d0 2024-01-26T23:49:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.915+0000 7f45a0cc2700 1 --2- 172.21.15.46:0/143988232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f459c0cd9d0 0x7f459c150b50 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f458c00c7c0 tx=0x7f458c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.915+0000 7f45917fa700 1 -- 172.21.15.46:0/143988232 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c005b30 con 0x7f459c0cd9d0 2024-01-26T23:49:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.915+0000 7f45a3727700 1 -- 172.21.15.46:0/143988232 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4598009420 con 0x7f459c0cd9d0 2024-01-26T23:49:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.915+0000 7f45917fa700 1 -- 172.21.15.46:0/143988232 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458c005cb0 con 0x7f459c0cd9d0 2024-01-26T23:49:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.915+0000 7f45917fa700 1 -- 172.21.15.46:0/143988232 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c007320 con 0x7f459c0cd9d0 2024-01-26T23:49:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.916+0000 7f45a3727700 1 -- 172.21.15.46:0/143988232 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459c154d30 con 0x7f459c0cd9d0 2024-01-26T23:49:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.917+0000 7f45a3727700 1 -- 172.21.15.46:0/143988232 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f459c04c710 con 0x7f459c0cd9d0 2024-01-26T23:49:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.917+0000 7f45917fa700 1 -- 172.21.15.46:0/143988232 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 75) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f458c01b3b0 con 0x7f459c0cd9d0 2024-01-26T23:49:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.918+0000 7f45917fa700 1 --2- 172.21.15.46:0/143988232 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4580066ea0 0x7f4580069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.918+0000 7f45917fa700 1 -- 172.21.15.46:0/143988232 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f458c083ac0 con 0x7f459c0cd9d0 2024-01-26T23:49:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.918+0000 7f45a14c3700 1 --2- 172.21.15.46:0/143988232 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4580066ea0 0x7f4580069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.918+0000 7f45a14c3700 1 --2- 172.21.15.46:0/143988232 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4580066ea0 0x7f4580069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f459c06e950 tx=0x7f4598017000 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:56.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:56.922+0000 7f45917fa700 1 -- 172.21.15.46:0/143988232 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f458c050cd0 con 0x7f459c0cd9d0 2024-01-26T23:49:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.130+0000 7f45a3727700 1 -- 172.21.15.46:0/143988232 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f459c15dd00 con 0x7f459c0cd9d0 2024-01-26T23:49:57.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.135+0000 7f45917fa700 1 -- 172.21.15.46:0/143988232 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v49) v1 ==== 104+0+0 (secure 0 0 0) 0x7f458c0170a0 con 0x7f459c0cd9d0 2024-01-26T23:49:57.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.137+0000 7f45a3727700 1 -- 172.21.15.46:0/143988232 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4580066ea0 msgr2=0x7f4580069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:57.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.137+0000 7f45a3727700 1 --2- 172.21.15.46:0/143988232 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4580066ea0 0x7f4580069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f459c06e950 tx=0x7f4598017000 comp rx=0 tx=0).stop 2024-01-26T23:49:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.137+0000 7f45a3727700 1 -- 172.21.15.46:0/143988232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f459c0cd9d0 msgr2=0x7f459c150b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.137+0000 7f45a3727700 1 --2- 172.21.15.46:0/143988232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f459c0cd9d0 0x7f459c150b50 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f458c00c7c0 tx=0x7f458c007e00 comp rx=0 tx=0).stop 2024-01-26T23:49:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.137+0000 7f45a3727700 1 -- 172.21.15.46:0/143988232 shutdown_connections 2024-01-26T23:49:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.137+0000 7f45a3727700 1 --2- 172.21.15.46:0/143988232 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4580066ea0 0x7f4580069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.137+0000 7f45a3727700 1 --2- 172.21.15.46:0/143988232 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f459c158400 0x7f459c14cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.137+0000 7f45a3727700 1 --2- 172.21.15.46:0/143988232 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f459c0b37e0 0x7f459c154160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.137+0000 7f45a3727700 1 --2- 172.21.15.46:0/143988232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f459c0cd9d0 0x7f459c150b50 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.137+0000 7f45a3727700 1 -- 172.21.15.46:0/143988232 >> 172.21.15.46:0/143988232 conn(0x7f459c067930 msgr2=0x7f459c0cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.137+0000 7f45a3727700 1 -- 172.21.15.46:0/143988232 shutdown_connections 2024-01-26T23:49:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.137+0000 7f45a3727700 1 -- 172.21.15.46:0/143988232 wait complete. 2024-01-26T23:49:57.152 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-26T23:49:57.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.286+0000 7f73e6e5a700 1 Processor -- start 2024-01-26T23:49:57.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.287+0000 7f73e6e5a700 1 -- start start 2024-01-26T23:49:57.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.288+0000 7f73e6e5a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e0159cb0 0x7f73e00cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:57.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.288+0000 7f73e6e5a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f73e00cf050 0x7f73e00af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:57.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.288+0000 7f73e6e5a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e00cc100 con 0x7f73e00cf4c0 2024-01-26T23:49:57.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.288+0000 7f73e6e5a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e00cb850 con 0x7f73e00cf050 2024-01-26T23:49:57.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.288+0000 7f73e6e5a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00cbaf0 con 0x7f73e0159cb0 2024-01-26T23:49:57.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.288+0000 7f73e4bf6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e0159cb0 0x7f73e00cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.288+0000 7f73dffff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f73e00cf050 0x7f73e00af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.288+0000 7f73dffff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f73e00cf050 0x7f73e00af430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:50712/0 (socket says 172.21.15.46:50712) 2024-01-26T23:49:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.288+0000 7f73dffff700 1 -- 172.21.15.46:0/366735431 learned_addr learned my addr 172.21.15.46:0/366735431 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.288+0000 7f73e4bf6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e0159cb0 0x7f73e00cc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:37192/0 (socket says 172.21.15.46:37192) 2024-01-26T23:49:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.288+0000 7f73e53f7700 1 --1- 172.21.15.46:0/366735431 >> v1:172.21.15.106:6789/0 conn(0x7f73e00cf4c0 0x7f73e00c5530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:51592/0 (socket says 172.21.15.46:51592) 2024-01-26T23:49:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.289+0000 7f73dffff700 1 -- 172.21.15.46:0/366735431 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e0159cb0 msgr2=0x7f73e00cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.289+0000 7f73dffff700 1 --2- 172.21.15.46:0/366735431 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e0159cb0 0x7f73e00cc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.289+0000 7f73dffff700 1 -- 172.21.15.46:0/366735431 >> v1:172.21.15.106:6789/0 conn(0x7f73e00cf4c0 legacy=0x7f73e00c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.289+0000 7f73dffff700 1 -- 172.21.15.46:0/366735431 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e00cfc80 con 0x7f73e00cf050 2024-01-26T23:49:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.289+0000 7f73df7fe700 1 -- 172.21.15.46:0/366735431 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286249898 0 0) 0x7f73e00cc100 con 0x7f73e00cf4c0 2024-01-26T23:49:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.289+0000 7f73e4bf6700 1 --2- 172.21.15.46:0/366735431 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e0159cb0 0x7f73e00cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.289+0000 7f73dffff700 1 --2- 172.21.15.46:0/366735431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73e00cf050 0x7f73e00af430 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f73c80099b0 tx=0x7f73c8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b39022e2a986c259 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.289+0000 7f73df7fe700 1 -- 172.21.15.46:0/366735431 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c8009ea0 con 0x7f73e00cf050 2024-01-26T23:49:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.289+0000 7f73df7fe700 1 -- 172.21.15.46:0/366735431 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c800b860 con 0x7f73e00cf050 2024-01-26T23:49:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.289+0000 7f73df7fe700 1 -- 172.21.15.46:0/366735431 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c800bc00 con 0x7f73e00cf050 2024-01-26T23:49:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.290+0000 7f73e6e5a700 1 -- 172.21.15.46:0/366735431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73e00cf050 msgr2=0x7f73e00af430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.290+0000 7f73e6e5a700 1 --2- 172.21.15.46:0/366735431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73e00cf050 0x7f73e00af430 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f73c80099b0 tx=0x7f73c8009250 comp rx=0 tx=0).stop 2024-01-26T23:49:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.290+0000 7f73e6e5a700 1 -- 172.21.15.46:0/366735431 shutdown_connections 2024-01-26T23:49:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.290+0000 7f73e6e5a700 1 --2- 172.21.15.46:0/366735431 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e0159cb0 0x7f73e00cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.290+0000 7f73e6e5a700 1 --2- 172.21.15.46:0/366735431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73e00cf050 0x7f73e00af430 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.290+0000 7f73e6e5a700 1 -- 172.21.15.46:0/366735431 >> 172.21.15.46:0/366735431 conn(0x7f73e0087f60 msgr2=0x7f73e006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.290+0000 7f73e6e5a700 1 -- 172.21.15.46:0/366735431 shutdown_connections 2024-01-26T23:49:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.290+0000 7f73e6e5a700 1 -- 172.21.15.46:0/366735431 wait complete. 2024-01-26T23:49:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.291+0000 7f73e6e5a700 1 Processor -- start 2024-01-26T23:49:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.291+0000 7f73e6e5a700 1 -- start start 2024-01-26T23:49:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.292+0000 7f73e6e5a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73e00cf050 0x7f73e0155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.292+0000 7f73e6e5a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e00cf4c0 0x7f73e00ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.292+0000 7f73e6e5a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e0159cb0 0x7f73e0169b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.292+0000 7f73e6e5a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e016df70 con 0x7f73e00cf4c0 2024-01-26T23:49:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.292+0000 7f73e6e5a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00c9410 con 0x7f73e00cf050 2024-01-26T23:49:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.292+0000 7f73e6e5a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00c96f0 con 0x7f73e0159cb0 2024-01-26T23:49:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.292+0000 7f73dffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e00cf4c0 0x7f73e00ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.292+0000 7f73dffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e00cf4c0 0x7f73e00ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54452/0 (socket says 172.21.15.46:54452) 2024-01-26T23:49:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.292+0000 7f73dffff700 1 -- 172.21.15.46:0/454948971 learned_addr learned my addr 172.21.15.46:0/454948971 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.292+0000 7f73e4bf6700 1 --2- 172.21.15.46:0/454948971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73e00cf050 0x7f73e0155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.292+0000 7f73e53f7700 1 --2- 172.21.15.46:0/454948971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e0159cb0 0x7f73e0169b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.293+0000 7f73dffff700 1 -- 172.21.15.46:0/454948971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e0159cb0 msgr2=0x7f73e0169b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.293+0000 7f73dffff700 1 --2- 172.21.15.46:0/454948971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e0159cb0 0x7f73e0169b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.293+0000 7f73dffff700 1 -- 172.21.15.46:0/454948971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73e00cf050 msgr2=0x7f73e0155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.293+0000 7f73dffff700 1 --2- 172.21.15.46:0/454948971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73e00cf050 0x7f73e0155730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.293+0000 7f73dffff700 1 -- 172.21.15.46:0/454948971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d00093c0 con 0x7f73e00cf4c0 2024-01-26T23:49:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.293+0000 7f73dffff700 1 --2- 172.21.15.46:0/454948971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e00cf4c0 0x7f73e00ca1d0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f73e0088df0 tx=0x7f73c8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.293+0000 7f73dcff9700 1 -- 172.21.15.46:0/454948971 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c8000d10 con 0x7f73e00cf4c0 2024-01-26T23:49:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.293+0000 7f73dcff9700 1 -- 172.21.15.46:0/454948971 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c800bbe0 con 0x7f73e00cf4c0 2024-01-26T23:49:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.293+0000 7f73dcff9700 1 -- 172.21.15.46:0/454948971 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c8011690 con 0x7f73e00cf4c0 2024-01-26T23:49:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.293+0000 7f73e6e5a700 1 -- 172.21.15.46:0/454948971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c80094f0 con 0x7f73e00cf4c0 2024-01-26T23:49:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.293+0000 7f73e6e5a700 1 -- 172.21.15.46:0/454948971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e00c9cb0 con 0x7f73e00cf4c0 2024-01-26T23:49:57.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.294+0000 7f73e6e5a700 1 -- 172.21.15.46:0/454948971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e00c9ea0 con 0x7f73e00cf4c0 2024-01-26T23:49:57.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.295+0000 7f73dcff9700 1 -- 172.21.15.46:0/454948971 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 75) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f73c8011ac0 con 0x7f73e00cf4c0 2024-01-26T23:49:57.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.295+0000 7f73dcff9700 1 --2- 172.21.15.46:0/454948971 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f73c0066ea0 0x7f73c0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:57.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.295+0000 7f73dcff9700 1 -- 172.21.15.46:0/454948971 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f73c8085ea0 con 0x7f73e00cf4c0 2024-01-26T23:49:57.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.295+0000 7f73e4bf6700 1 --2- 172.21.15.46:0/454948971 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f73c0066ea0 0x7f73c0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:57.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.296+0000 7f73e4bf6700 1 --2- 172.21.15.46:0/454948971 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f73c0066ea0 0x7f73c0069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f73d0009e90 tx=0x7f73d0009120 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:57.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.300+0000 7f73dcff9700 1 -- 172.21.15.46:0/454948971 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f73c8052ff0 con 0x7f73e00cf4c0 2024-01-26T23:49:57.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.503+0000 7f73e6e5a700 1 -- 172.21.15.46:0/454948971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f73e004c710 con 0x7f73e00cf4c0 2024-01-26T23:49:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.507+0000 7f73dcff9700 1 -- 172.21.15.46:0/454948971 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v50) v1 ==== 120+0+0 (secure 0 0 0) 0x7f73c8056610 con 0x7f73e00cf4c0 2024-01-26T23:49:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.509+0000 7f73e6e5a700 1 -- 172.21.15.46:0/454948971 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f73c0066ea0 msgr2=0x7f73c0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.509+0000 7f73e6e5a700 1 --2- 172.21.15.46:0/454948971 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f73c0066ea0 0x7f73c0069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f73d0009e90 tx=0x7f73d0009120 comp rx=0 tx=0).stop 2024-01-26T23:49:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.509+0000 7f73e6e5a700 1 -- 172.21.15.46:0/454948971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e00cf4c0 msgr2=0x7f73e00ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.509+0000 7f73e6e5a700 1 --2- 172.21.15.46:0/454948971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e00cf4c0 0x7f73e00ca1d0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f73e0088df0 tx=0x7f73c8017000 comp rx=0 tx=0).stop 2024-01-26T23:49:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.509+0000 7f73e6e5a700 1 -- 172.21.15.46:0/454948971 shutdown_connections 2024-01-26T23:49:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.509+0000 7f73e6e5a700 1 --2- 172.21.15.46:0/454948971 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f73c0066ea0 0x7f73c0069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.509+0000 7f73e6e5a700 1 --2- 172.21.15.46:0/454948971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73e0159cb0 0x7f73e0169b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.509+0000 7f73e6e5a700 1 --2- 172.21.15.46:0/454948971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73e00cf050 0x7f73e0155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.509+0000 7f73e6e5a700 1 --2- 172.21.15.46:0/454948971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e00cf4c0 0x7f73e00ca1d0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.509+0000 7f73e6e5a700 1 -- 172.21.15.46:0/454948971 >> 172.21.15.46:0/454948971 conn(0x7f73e0087f60 msgr2=0x7f73e00c6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:57.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.509+0000 7f73e6e5a700 1 -- 172.21.15.46:0/454948971 shutdown_connections 2024-01-26T23:49:57.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.509+0000 7f73e6e5a700 1 -- 172.21.15.46:0/454948971 wait complete. 2024-01-26T23:49:57.524 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-26T23:49:57.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.656+0000 7fccc6556700 1 Processor -- start 2024-01-26T23:49:57.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.657+0000 7fccc6556700 1 -- start start 2024-01-26T23:49:57.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.657+0000 7fccc6556700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc00c8da0 0x7fccc00c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:57.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.657+0000 7fccc6556700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fccc00caf30 0x7fccc00cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:57.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.657+0000 7fccc6556700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc00ccc00 con 0x7fccc0140af0 2024-01-26T23:49:57.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.657+0000 7fccc6556700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc00cbbf0 con 0x7fccc00caf30 2024-01-26T23:49:57.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.657+0000 7fccc6556700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc00cc310 con 0x7fccc00c8da0 2024-01-26T23:49:57.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.658+0000 7fccc4af3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fccc00caf30 0x7fccc00cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:57.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.658+0000 7fccbf7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fccc0140af0 0x7fccc0140f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58828/0 (socket says 172.21.15.46:58828) 2024-01-26T23:49:57.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.658+0000 7fccbf7fe700 1 -- 172.21.15.46:0/668640660 learned_addr learned my addr 172.21.15.46:0/668640660 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:57.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.658+0000 7fccbffff700 1 --2- 172.21.15.46:0/668640660 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc00c8da0 0x7fccc00c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:57.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.658+0000 7fccc4af3700 1 -- 172.21.15.46:0/668640660 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc00c8da0 msgr2=0x7fccc00c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:57.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.658+0000 7fccc4af3700 1 --2- 172.21.15.46:0/668640660 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc00c8da0 0x7fccc00c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.658+0000 7fccc4af3700 1 -- 172.21.15.46:0/668640660 >> v1:172.21.15.46:6789/0 conn(0x7fccc0140af0 legacy=0x7fccc0140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:57.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.658+0000 7fccc4af3700 1 -- 172.21.15.46:0/668640660 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc00cc5b0 con 0x7fccc00caf30 2024-01-26T23:49:57.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.658+0000 7fccbeffd700 1 -- 172.21.15.46:0/668640660 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230388953 0 0) 0x7fccc00ccc00 con 0x7fccc0140af0 2024-01-26T23:49:57.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.658+0000 7fccc4af3700 1 --2- 172.21.15.46:0/668640660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc00caf30 0x7fccc00cb370 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009e60 tx=0x7fccb8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7a23f877b6b6ccf server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:57.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.658+0000 7fccbeffd700 1 -- 172.21.15.46:0/668640660 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb800e070 con 0x7fccc00caf30 2024-01-26T23:49:57.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.658+0000 7fccbeffd700 1 -- 172.21.15.46:0/668640660 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb800bb60 con 0x7fccc00caf30 2024-01-26T23:49:57.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.658+0000 7fccbeffd700 1 -- 172.21.15.46:0/668640660 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8012430 con 0x7fccc00caf30 2024-01-26T23:49:57.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.659+0000 7fccc6556700 1 -- 172.21.15.46:0/668640660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc00caf30 msgr2=0x7fccc00cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:57.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.659+0000 7fccc6556700 1 --2- 172.21.15.46:0/668640660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc00caf30 0x7fccc00cb370 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009e60 tx=0x7fccb8009780 comp rx=0 tx=0).stop 2024-01-26T23:49:57.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.659+0000 7fccc6556700 1 -- 172.21.15.46:0/668640660 shutdown_connections 2024-01-26T23:49:57.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.659+0000 7fccc6556700 1 --2- 172.21.15.46:0/668640660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc00caf30 0x7fccc00cb370 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.659+0000 7fccc6556700 1 --2- 172.21.15.46:0/668640660 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc00c8da0 0x7fccc00c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.659+0000 7fccc6556700 1 -- 172.21.15.46:0/668640660 >> 172.21.15.46:0/668640660 conn(0x7fccc006bd90 msgr2=0x7fccc006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:57.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.659+0000 7fccc6556700 1 -- 172.21.15.46:0/668640660 shutdown_connections 2024-01-26T23:49:57.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.659+0000 7fccc6556700 1 -- 172.21.15.46:0/668640660 wait complete. 2024-01-26T23:49:57.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.661+0000 7fccc6556700 1 Processor -- start 2024-01-26T23:49:57.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.661+0000 7fccc6556700 1 -- start start 2024-01-26T23:49:57.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccc6556700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccc00c8da0 0x7fccc00c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:57.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccc6556700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc00caf30 0x7fccc00c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:57.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccc6556700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc0140af0 0x7fccc00c8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:57.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccc6556700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc00c5ff0 con 0x7fccc00caf30 2024-01-26T23:49:57.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccc6556700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc00c6e10 con 0x7fccc00c8da0 2024-01-26T23:49:57.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccc6556700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc00c72a0 con 0x7fccc0140af0 2024-01-26T23:49:57.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccbf7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc00caf30 0x7fccc00c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:57.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccbf7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc00caf30 0x7fccc00c7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54468/0 (socket says 172.21.15.46:54468) 2024-01-26T23:49:57.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccbf7fe700 1 -- 172.21.15.46:0/3845946071 learned_addr learned my addr 172.21.15.46:0/3845946071 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:57.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccbffff700 1 --2- 172.21.15.46:0/3845946071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccc00c8da0 0x7fccc00c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:57.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccc4af3700 1 --2- 172.21.15.46:0/3845946071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc0140af0 0x7fccc00c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:57.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccbf7fe700 1 -- 172.21.15.46:0/3845946071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc0140af0 msgr2=0x7fccc00c8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:57.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccbf7fe700 1 --2- 172.21.15.46:0/3845946071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc0140af0 0x7fccc00c8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccbf7fe700 1 -- 172.21.15.46:0/3845946071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccc00c8da0 msgr2=0x7fccc00c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:57.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccbf7fe700 1 --2- 172.21.15.46:0/3845946071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccc00c8da0 0x7fccc00c5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccbf7fe700 1 -- 172.21.15.46:0/3845946071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb8009a20 con 0x7fccc00caf30 2024-01-26T23:49:57.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccc4af3700 1 --2- 172.21.15.46:0/3845946071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc0140af0 0x7fccc00c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:57.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.662+0000 7fccbffff700 1 --2- 172.21.15.46:0/3845946071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccc00c8da0 0x7fccc00c5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:57.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.663+0000 7fccbf7fe700 1 --2- 172.21.15.46:0/3845946071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc00caf30 0x7fccc00c7c00 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fccb400c5e0 tx=0x7fccb4001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:57.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.663+0000 7fcca7fff700 1 -- 172.21.15.46:0/3845946071 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4005b30 con 0x7fccc00caf30 2024-01-26T23:49:57.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.663+0000 7fcca7fff700 1 -- 172.21.15.46:0/3845946071 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb4005c90 con 0x7fccc00caf30 2024-01-26T23:49:57.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.663+0000 7fccc6556700 1 -- 172.21.15.46:0/3845946071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc00c6200 con 0x7fccc00caf30 2024-01-26T23:49:57.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.663+0000 7fcca7fff700 1 -- 172.21.15.46:0/3845946071 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb40072e0 con 0x7fccc00caf30 2024-01-26T23:49:57.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.663+0000 7fccc6556700 1 -- 172.21.15.46:0/3845946071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc00c67c0 con 0x7fccc00caf30 2024-01-26T23:49:57.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.664+0000 7fccc6556700 1 -- 172.21.15.46:0/3845946071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc004c710 con 0x7fccc00caf30 2024-01-26T23:49:57.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.664+0000 7fcca7fff700 1 -- 172.21.15.46:0/3845946071 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 75) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fccb401b3b0 con 0x7fccc00caf30 2024-01-26T23:49:57.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.665+0000 7fcca7fff700 1 --2- 172.21.15.46:0/3845946071 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fccac066ea0 0x7fccac069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:57.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.665+0000 7fcca7fff700 1 -- 172.21.15.46:0/3845946071 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6210+0+0 (secure 0 0 0) 0x7fccb4083a40 con 0x7fccc00caf30 2024-01-26T23:49:57.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.665+0000 7fccbffff700 1 --2- 172.21.15.46:0/3845946071 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fccac066ea0 0x7fccac069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:57.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.666+0000 7fccbffff700 1 --2- 172.21.15.46:0/3845946071 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fccac066ea0 0x7fccac069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fccc013cb90 tx=0x7fccb0008040 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:57.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.669+0000 7fcca7fff700 1 -- 172.21.15.46:0/3845946071 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fccb404b5a0 con 0x7fccc00caf30 2024-01-26T23:49:57.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.873+0000 7fccc6556700 1 -- 172.21.15.46:0/3845946071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fccc00c6f50 con 0x7fccc00caf30 2024-01-26T23:49:57.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.877+0000 7fcca7fff700 1 -- 172.21.15.46:0/3845946071 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v51) v1 ==== 115+0+0 (secure 0 0 0) 0x7fccb4017020 con 0x7fccc00caf30 2024-01-26T23:49:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.879+0000 7fccc6556700 1 -- 172.21.15.46:0/3845946071 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fccac066ea0 msgr2=0x7fccac069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.879+0000 7fccc6556700 1 --2- 172.21.15.46:0/3845946071 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fccac066ea0 0x7fccac069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fccc013cb90 tx=0x7fccb0008040 comp rx=0 tx=0).stop 2024-01-26T23:49:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.879+0000 7fccc6556700 1 -- 172.21.15.46:0/3845946071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc00caf30 msgr2=0x7fccc00c7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.879+0000 7fccc6556700 1 --2- 172.21.15.46:0/3845946071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc00caf30 0x7fccc00c7c00 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fccb400c5e0 tx=0x7fccb4001070 comp rx=0 tx=0).stop 2024-01-26T23:49:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.879+0000 7fccc6556700 1 -- 172.21.15.46:0/3845946071 shutdown_connections 2024-01-26T23:49:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.879+0000 7fccc6556700 1 --2- 172.21.15.46:0/3845946071 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fccac066ea0 0x7fccac069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.879+0000 7fccc6556700 1 --2- 172.21.15.46:0/3845946071 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc0140af0 0x7fccc00c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.879+0000 7fccc6556700 1 --2- 172.21.15.46:0/3845946071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccc00c8da0 0x7fccc00c5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.879+0000 7fccc6556700 1 --2- 172.21.15.46:0/3845946071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc00caf30 0x7fccc00c7c00 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:57.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.879+0000 7fccc6556700 1 -- 172.21.15.46:0/3845946071 >> 172.21.15.46:0/3845946071 conn(0x7fccc006bd90 msgr2=0x7fccc00cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:57.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.879+0000 7fccc6556700 1 -- 172.21.15.46:0/3845946071 shutdown_connections 2024-01-26T23:49:57.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:57.879+0000 7fccc6556700 1 -- 172.21.15.46:0/3845946071 wait complete. 2024-01-26T23:49:57.895 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-01-26T23:49:58.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.025+0000 7fbd557bc700 1 Processor -- start 2024-01-26T23:49:58.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.026+0000 7fbd557bc700 1 -- start start 2024-01-26T23:49:58.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.026+0000 7fbd557bc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd501596c0 0x7fbd5006bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.026+0000 7fbd557bc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd5015d460 0x7fbd5006c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.026+0000 7fbd557bc700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5015d8f0 con 0x7fbd50159a90 2024-01-26T23:49:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.026+0000 7fbd557bc700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd500c98c0 con 0x7fbd501596c0 2024-01-26T23:49:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.026+0000 7fbd557bc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5015dbd0 con 0x7fbd5015d460 2024-01-26T23:49:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.027+0000 7fbd4effd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd501596c0 0x7fbd5006bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.027+0000 7fbd4f7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbd50159a90 0x7fbd5006d310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58834/0 (socket says 172.21.15.46:58834) 2024-01-26T23:49:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.027+0000 7fbd4f7fe700 1 -- 172.21.15.46:0/1245004363 learned_addr learned my addr 172.21.15.46:0/1245004363 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.027+0000 7fbd4e7fc700 1 --2- 172.21.15.46:0/1245004363 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd5015d460 0x7fbd5006c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.027+0000 7fbd4effd700 1 -- 172.21.15.46:0/1245004363 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd5015d460 msgr2=0x7fbd5006c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.027+0000 7fbd4effd700 1 --2- 172.21.15.46:0/1245004363 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd5015d460 0x7fbd5006c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.027+0000 7fbd4effd700 1 -- 172.21.15.46:0/1245004363 >> v1:172.21.15.46:6789/0 conn(0x7fbd50159a90 legacy=0x7fbd5006d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.027+0000 7fbd4effd700 1 -- 172.21.15.46:0/1245004363 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5015deb0 con 0x7fbd501596c0 2024-01-26T23:49:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.027+0000 7fbd4dffb700 1 -- 172.21.15.46:0/1245004363 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960698465 0 0) 0x7fbd5015d8f0 con 0x7fbd50159a90 2024-01-26T23:49:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.027+0000 7fbd4effd700 1 --2- 172.21.15.46:0/1245004363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd501596c0 0x7fbd5006bcd0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fbd400098e0 tx=0x7fbd400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8923bd27ee227dc7 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.027+0000 7fbd4dffb700 1 -- 172.21.15.46:0/1245004363 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd40009e80 con 0x7fbd501596c0 2024-01-26T23:49:58.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.027+0000 7fbd4dffb700 1 -- 172.21.15.46:0/1245004363 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4000b860 con 0x7fbd501596c0 2024-01-26T23:49:58.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.027+0000 7fbd4dffb700 1 -- 172.21.15.46:0/1245004363 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4000bbe0 con 0x7fbd501596c0 2024-01-26T23:49:58.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.028+0000 7fbd557bc700 1 -- 172.21.15.46:0/1245004363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd501596c0 msgr2=0x7fbd5006bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.028+0000 7fbd557bc700 1 --2- 172.21.15.46:0/1245004363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd501596c0 0x7fbd5006bcd0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fbd400098e0 tx=0x7fbd400091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.028+0000 7fbd557bc700 1 -- 172.21.15.46:0/1245004363 shutdown_connections 2024-01-26T23:49:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.028+0000 7fbd557bc700 1 --2- 172.21.15.46:0/1245004363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd501596c0 0x7fbd5006bcd0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.028+0000 7fbd557bc700 1 --2- 172.21.15.46:0/1245004363 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd5015d460 0x7fbd5006c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.028+0000 7fbd557bc700 1 -- 172.21.15.46:0/1245004363 >> 172.21.15.46:0/1245004363 conn(0x7fbd500ce530 msgr2=0x7fbd500ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.028+0000 7fbd557bc700 1 -- 172.21.15.46:0/1245004363 shutdown_connections 2024-01-26T23:49:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.028+0000 7fbd557bc700 1 -- 172.21.15.46:0/1245004363 wait complete. 2024-01-26T23:49:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.030+0000 7fbd557bc700 1 Processor -- start 2024-01-26T23:49:58.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.030+0000 7fbd557bc700 1 -- start start 2024-01-26T23:49:58.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.031+0000 7fbd557bc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd501596c0 0x7fbd500c8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:58.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.031+0000 7fbd557bc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd50159a90 0x7fbd500b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.031+0000 7fbd557bc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd5015d460 0x7fbd500b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.031+0000 7fbd557bc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd500b0c90 con 0x7fbd5015d460 2024-01-26T23:49:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.031+0000 7fbd557bc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd500b02c0 con 0x7fbd501596c0 2024-01-26T23:49:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.031+0000 7fbd557bc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd500af6b0 con 0x7fbd50159a90 2024-01-26T23:49:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.031+0000 7fbd4f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd5015d460 0x7fbd500b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.031+0000 7fbd4f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd5015d460 0x7fbd500b2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54484/0 (socket says 172.21.15.46:54484) 2024-01-26T23:49:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.031+0000 7fbd4f7fe700 1 -- 172.21.15.46:0/607273777 learned_addr learned my addr 172.21.15.46:0/607273777 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.031+0000 7fbd4e7fc700 1 --2- 172.21.15.46:0/607273777 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd50159a90 0x7fbd500b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.031+0000 7fbd4effd700 1 --2- 172.21.15.46:0/607273777 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd501596c0 0x7fbd500c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.032+0000 7fbd4f7fe700 1 -- 172.21.15.46:0/607273777 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd50159a90 msgr2=0x7fbd500b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.032+0000 7fbd4f7fe700 1 --2- 172.21.15.46:0/607273777 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd50159a90 0x7fbd500b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.032+0000 7fbd4f7fe700 1 -- 172.21.15.46:0/607273777 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd501596c0 msgr2=0x7fbd500c8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.032+0000 7fbd4f7fe700 1 --2- 172.21.15.46:0/607273777 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd501596c0 0x7fbd500c8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.032+0000 7fbd4f7fe700 1 -- 172.21.15.46:0/607273777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd40009420 con 0x7fbd5015d460 2024-01-26T23:49:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.032+0000 7fbd4effd700 1 --2- 172.21.15.46:0/607273777 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd501596c0 0x7fbd500c8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:49:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.032+0000 7fbd4f7fe700 1 --2- 172.21.15.46:0/607273777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd5015d460 0x7fbd500b2af0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fbd4800cc90 tx=0x7fbd48001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.032+0000 7fbd36ffd700 1 -- 172.21.15.46:0/607273777 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4800a9f0 con 0x7fbd5015d460 2024-01-26T23:49:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.032+0000 7fbd36ffd700 1 -- 172.21.15.46:0/607273777 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4800ab50 con 0x7fbd5015d460 2024-01-26T23:49:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.032+0000 7fbd557bc700 1 -- 172.21.15.46:0/607273777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd500af990 con 0x7fbd5015d460 2024-01-26T23:49:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.032+0000 7fbd36ffd700 1 -- 172.21.15.46:0/607273777 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd480070f0 con 0x7fbd5015d460 2024-01-26T23:49:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.032+0000 7fbd557bc700 1 -- 172.21.15.46:0/607273777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd500affd0 con 0x7fbd5015d460 2024-01-26T23:49:58.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.033+0000 7fbd557bc700 1 -- 172.21.15.46:0/607273777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd500b0470 con 0x7fbd5015d460 2024-01-26T23:49:58.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.033+0000 7fbd36ffd700 1 -- 172.21.15.46:0/607273777 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 75) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fbd48007460 con 0x7fbd5015d460 2024-01-26T23:49:58.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.034+0000 7fbd36ffd700 1 --2- 172.21.15.46:0/607273777 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbd3c066ed0 0x7fbd3c069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:58.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.034+0000 7fbd36ffd700 1 -- 172.21.15.46:0/607273777 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6210+0+0 (secure 0 0 0) 0x7fbd480849e0 con 0x7fbd5015d460 2024-01-26T23:49:58.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.034+0000 7fbd4effd700 1 --2- 172.21.15.46:0/607273777 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbd3c066ed0 0x7fbd3c069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:58.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.035+0000 7fbd4effd700 1 --2- 172.21.15.46:0/607273777 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbd3c066ed0 0x7fbd3c069380 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbd400091e0 tx=0x7fbd40017000 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:58.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.038+0000 7fbd36ffd700 1 -- 172.21.15.46:0/607273777 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbd500b0470 con 0x7fbd5015d460 2024-01-26T23:49:58.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.332+0000 7fbd557bc700 1 -- 172.21.15.46:0/607273777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7fbd500b0430 con 0x7fbd5015d460 2024-01-26T23:49:58.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.516+0000 7fbd36ffd700 1 -- 172.21.15.46:0/607273777 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbd48051f80 con 0x7fbd5015d460 2024-01-26T23:49:58.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.704+0000 7fbd36ffd700 1 -- 172.21.15.46:0/607273777 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v777) v1 ==== 160+0+0 (secure 0 0 0) 0x7fbd48051bb0 con 0x7fbd5015d460 2024-01-26T23:49:58.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.707+0000 7fbd557bc700 1 -- 172.21.15.46:0/607273777 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbd3c066ed0 msgr2=0x7fbd3c069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:58.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.707+0000 7fbd557bc700 1 --2- 172.21.15.46:0/607273777 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbd3c066ed0 0x7fbd3c069380 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbd400091e0 tx=0x7fbd40017000 comp rx=0 tx=0).stop 2024-01-26T23:49:58.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.707+0000 7fbd557bc700 1 -- 172.21.15.46:0/607273777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd5015d460 msgr2=0x7fbd500b2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:58.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.707+0000 7fbd557bc700 1 --2- 172.21.15.46:0/607273777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd5015d460 0x7fbd500b2af0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fbd4800cc90 tx=0x7fbd48001a80 comp rx=0 tx=0).stop 2024-01-26T23:49:58.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.707+0000 7fbd557bc700 1 -- 172.21.15.46:0/607273777 shutdown_connections 2024-01-26T23:49:58.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.707+0000 7fbd557bc700 1 --2- 172.21.15.46:0/607273777 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbd3c066ed0 0x7fbd3c069380 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:58.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.707+0000 7fbd557bc700 1 --2- 172.21.15.46:0/607273777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd5015d460 0x7fbd500b2af0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:58.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.707+0000 7fbd557bc700 1 --2- 172.21.15.46:0/607273777 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd501596c0 0x7fbd500c8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:58.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.707+0000 7fbd557bc700 1 --2- 172.21.15.46:0/607273777 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd50159a90 0x7fbd500b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:58.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.707+0000 7fbd557bc700 1 -- 172.21.15.46:0/607273777 >> 172.21.15.46:0/607273777 conn(0x7fbd500ce530 msgr2=0x7fbd501588c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:58.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.707+0000 7fbd557bc700 1 -- 172.21.15.46:0/607273777 shutdown_connections 2024-01-26T23:49:58.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.707+0000 7fbd557bc700 1 -- 172.21.15.46:0/607273777 wait complete. 2024-01-26T23:49:58.724 INFO:tasks.cephfs_test_runner:test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) ... ok 2024-01-26T23:49:58.724 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2024-01-26T23:49:58.724 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-26T23:49:58.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-01-26T23:49:58.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.858+0000 7fc5fbe2f700 1 Processor -- start 2024-01-26T23:49:58.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.859+0000 7fc5fbe2f700 1 -- start start 2024-01-26T23:49:58.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.859+0000 7fc5fbe2f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5f4159760 0x7fc5f40b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:58.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.860+0000 7fc5fbe2f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc5f40b12c0 0x7fc5f40a9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:58.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.860+0000 7fc5fbe2f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f406f940 con 0x7fc5f40b3060 2024-01-26T23:49:58.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.860+0000 7fc5fbe2f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5f406ed30 con 0x7fc5f40b12c0 2024-01-26T23:49:58.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.860+0000 7fc5fbe2f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f406f010 con 0x7fc5f4159760 2024-01-26T23:49:58.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.860+0000 7fc5f9bcb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5f4159760 0x7fc5f40b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:58.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.860+0000 7fc5fa3cc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc5f40b12c0 0x7fc5f40a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:58.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.860+0000 7fc5f93ca700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fc5f40b3060 0x7fc5f40b0bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:51602/0 (socket says 172.21.15.46:51602) 2024-01-26T23:49:58.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.860+0000 7fc5f93ca700 1 -- 172.21.15.46:0/3961336535 learned_addr learned my addr 172.21.15.46:0/3961336535 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:49:58.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.860+0000 7fc5fa3cc700 1 -- 172.21.15.46:0/3961336535 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5f4159760 msgr2=0x7fc5f40b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:58.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.860+0000 7fc5fa3cc700 1 --2- 172.21.15.46:0/3961336535 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5f4159760 0x7fc5f40b2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:58.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.860+0000 7fc5fa3cc700 1 -- 172.21.15.46:0/3961336535 >> v1:172.21.15.106:6789/0 conn(0x7fc5f40b3060 legacy=0x7fc5f40b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:58.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.860+0000 7fc5fa3cc700 1 -- 172.21.15.46:0/3961336535 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f406f2f0 con 0x7fc5f40b12c0 2024-01-26T23:49:58.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.860+0000 7fc5f9bcb700 1 --2- 172.21.15.46:0/3961336535 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5f4159760 0x7fc5f40b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:49:58.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.860+0000 7fc5f8bc9700 1 -- 172.21.15.46:0/3961336535 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293541870 0 0) 0x7fc5f406f940 con 0x7fc5f40b3060 2024-01-26T23:49:58.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.861+0000 7fc5fa3cc700 1 --2- 172.21.15.46:0/3961336535 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5f40b12c0 0x7fc5f40a9840 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec009e60 tx=0x7fc5ec009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebbd9dcc2c92c4ea server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:58.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.861+0000 7fc5f8bc9700 1 -- 172.21.15.46:0/3961336535 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec00e070 con 0x7fc5f40b12c0 2024-01-26T23:49:58.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.861+0000 7fc5f8bc9700 1 -- 172.21.15.46:0/3961336535 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ec00bb60 con 0x7fc5f40b12c0 2024-01-26T23:49:58.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.861+0000 7fc5f8bc9700 1 -- 172.21.15.46:0/3961336535 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec012430 con 0x7fc5f40b12c0 2024-01-26T23:49:58.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.861+0000 7fc5fbe2f700 1 -- 172.21.15.46:0/3961336535 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5f40b12c0 msgr2=0x7fc5f40a9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:58.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.861+0000 7fc5fbe2f700 1 --2- 172.21.15.46:0/3961336535 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5f40b12c0 0x7fc5f40a9840 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec009e60 tx=0x7fc5ec009780 comp rx=0 tx=0).stop 2024-01-26T23:49:58.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.861+0000 7fc5fbe2f700 1 -- 172.21.15.46:0/3961336535 shutdown_connections 2024-01-26T23:49:58.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.861+0000 7fc5fbe2f700 1 --2- 172.21.15.46:0/3961336535 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5f40b12c0 0x7fc5f40a9840 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:58.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.861+0000 7fc5fbe2f700 1 --2- 172.21.15.46:0/3961336535 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5f4159760 0x7fc5f40b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:58.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.861+0000 7fc5fbe2f700 1 -- 172.21.15.46:0/3961336535 >> 172.21.15.46:0/3961336535 conn(0x7fc5f4069510 msgr2=0x7fc5f40aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:58.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.861+0000 7fc5fbe2f700 1 -- 172.21.15.46:0/3961336535 shutdown_connections 2024-01-26T23:49:58.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.861+0000 7fc5fbe2f700 1 -- 172.21.15.46:0/3961336535 wait complete. 2024-01-26T23:49:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.863+0000 7fc5fbe2f700 1 Processor -- start 2024-01-26T23:49:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.863+0000 7fc5fbe2f700 1 -- start start 2024-01-26T23:49:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.863+0000 7fc5fbe2f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5f40b12c0 0x7fc5f414df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.864+0000 7fc5fbe2f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5f40b3060 0x7fc5f4152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.864+0000 7fc5fbe2f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5f4159760 0x7fc5f4154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.864+0000 7fc5fbe2f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f4159f00 con 0x7fc5f4159760 2024-01-26T23:49:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.864+0000 7fc5fbe2f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f4151390 con 0x7fc5f40b12c0 2024-01-26T23:49:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.864+0000 7fc5fbe2f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f4150780 con 0x7fc5f40b3060 2024-01-26T23:49:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.864+0000 7fc5fa3cc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5f4159760 0x7fc5f4154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.864+0000 7fc5fa3cc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5f4159760 0x7fc5f4154a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54490/0 (socket says 172.21.15.46:54490) 2024-01-26T23:49:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.864+0000 7fc5fa3cc700 1 -- 172.21.15.46:0/1207732484 learned_addr learned my addr 172.21.15.46:0/1207732484 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.864+0000 7fc5f9bcb700 1 --2- 172.21.15.46:0/1207732484 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5f40b12c0 0x7fc5f414df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.864+0000 7fc5f93ca700 1 --2- 172.21.15.46:0/1207732484 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5f40b3060 0x7fc5f4152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.864+0000 7fc5fa3cc700 1 -- 172.21.15.46:0/1207732484 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5f40b3060 msgr2=0x7fc5f4152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.864+0000 7fc5fa3cc700 1 --2- 172.21.15.46:0/1207732484 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5f40b3060 0x7fc5f4152510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:58.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.864+0000 7fc5fa3cc700 1 -- 172.21.15.46:0/1207732484 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5f40b12c0 msgr2=0x7fc5f414df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:58.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.864+0000 7fc5fa3cc700 1 --2- 172.21.15.46:0/1207732484 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5f40b12c0 0x7fc5f414df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:58.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.864+0000 7fc5fa3cc700 1 -- 172.21.15.46:0/1207732484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e8009420 con 0x7fc5f4159760 2024-01-26T23:49:58.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.864+0000 7fc5fa3cc700 1 --2- 172.21.15.46:0/1207732484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5f4159760 0x7fc5f4154a30 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec013000 tx=0x7fc5ec019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:58.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.864+0000 7fc5e5ffb700 1 -- 172.21.15.46:0/1207732484 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec012ea0 con 0x7fc5f4159760 2024-01-26T23:49:58.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.865+0000 7fc5e5ffb700 1 -- 172.21.15.46:0/1207732484 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ec027470 con 0x7fc5f4159760 2024-01-26T23:49:58.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.865+0000 7fc5fbe2f700 1 -- 172.21.15.46:0/1207732484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ec009a20 con 0x7fc5f4159760 2024-01-26T23:49:58.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.865+0000 7fc5e5ffb700 1 -- 172.21.15.46:0/1207732484 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec027830 con 0x7fc5f4159760 2024-01-26T23:49:58.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.865+0000 7fc5fbe2f700 1 -- 172.21.15.46:0/1207732484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f4150d40 con 0x7fc5f4159760 2024-01-26T23:49:58.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.866+0000 7fc5fbe2f700 1 -- 172.21.15.46:0/1207732484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f404c710 con 0x7fc5f4159760 2024-01-26T23:49:58.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.866+0000 7fc5e5ffb700 1 -- 172.21.15.46:0/1207732484 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc5ec024070 con 0x7fc5f4159760 2024-01-26T23:49:58.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.867+0000 7fc5e5ffb700 1 --2- 172.21.15.46:0/1207732484 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc5d8066f30 0x7fc5d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:58.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.867+0000 7fc5e5ffb700 1 -- 172.21.15.46:0/1207732484 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6210+0+0 (secure 0 0 0) 0x7fc5ec086d70 con 0x7fc5f4159760 2024-01-26T23:49:58.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.867+0000 7fc5f9bcb700 1 --2- 172.21.15.46:0/1207732484 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc5d8066f30 0x7fc5d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:58.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.867+0000 7fc5f9bcb700 1 --2- 172.21.15.46:0/1207732484 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc5d8066f30 0x7fc5d80693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8009ea0 tx=0x7fc5e80091a0 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:58.871+0000 7fc5e5ffb700 1 -- 172.21.15.46:0/1207732484 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc5ec053d40 con 0x7fc5f4159760 2024-01-26T23:49:59.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.160+0000 7fc5fbe2f700 1 -- 172.21.15.46:0/1207732484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7fc5f4068120 con 0x7fc5f4159760 2024-01-26T23:49:59.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.709+0000 7fc5e5ffb700 1 -- 172.21.15.46:0/1207732484 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v778) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc5ec02c020 con 0x7fc5f4159760 2024-01-26T23:49:59.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.712+0000 7fc5fbe2f700 1 -- 172.21.15.46:0/1207732484 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc5d8066f30 msgr2=0x7fc5d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.712+0000 7fc5fbe2f700 1 --2- 172.21.15.46:0/1207732484 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc5d8066f30 0x7fc5d80693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8009ea0 tx=0x7fc5e80091a0 comp rx=0 tx=0).stop 2024-01-26T23:49:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.712+0000 7fc5fbe2f700 1 -- 172.21.15.46:0/1207732484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5f4159760 msgr2=0x7fc5f4154a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.712+0000 7fc5fbe2f700 1 --2- 172.21.15.46:0/1207732484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5f4159760 0x7fc5f4154a30 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec013000 tx=0x7fc5ec019000 comp rx=0 tx=0).stop 2024-01-26T23:49:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.712+0000 7fc5fbe2f700 1 -- 172.21.15.46:0/1207732484 shutdown_connections 2024-01-26T23:49:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.712+0000 7fc5fbe2f700 1 --2- 172.21.15.46:0/1207732484 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc5d8066f30 0x7fc5d80693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.712+0000 7fc5fbe2f700 1 --2- 172.21.15.46:0/1207732484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5f4159760 0x7fc5f4154a30 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.712+0000 7fc5fbe2f700 1 --2- 172.21.15.46:0/1207732484 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5f40b12c0 0x7fc5f414df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.712+0000 7fc5fbe2f700 1 --2- 172.21.15.46:0/1207732484 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5f40b3060 0x7fc5f4152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.712+0000 7fc5fbe2f700 1 -- 172.21.15.46:0/1207732484 >> 172.21.15.46:0/1207732484 conn(0x7fc5f4069510 msgr2=0x7fc5f40b04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.712+0000 7fc5fbe2f700 1 -- 172.21.15.46:0/1207732484 shutdown_connections 2024-01-26T23:49:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.712+0000 7fc5fbe2f700 1 -- 172.21.15.46:0/1207732484 wait complete. 2024-01-26T23:49:59.729 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-26T23:49:59.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.867+0000 7f97c6c37700 1 Processor -- start 2024-01-26T23:49:59.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.867+0000 7f97c6c37700 1 -- start start 2024-01-26T23:49:59.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97c6c37700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97c015cd40 0x7f97c00cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:59.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97c6c37700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97c015d110 0x7f97c00cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:59.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97c6c37700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f97c0067930 0x7f97c0067db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97c6c37700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c00c7ac0 con 0x7f97c015d110 2024-01-26T23:49:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97c6c37700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c00682f0 con 0x7f97c0067930 2024-01-26T23:49:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97c6c37700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c006caa0 con 0x7f97c015cd40 2024-01-26T23:49:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97bffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97c015d110 0x7f97c00cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97bffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97c015d110 0x7f97c00cb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54494/0 (socket says 172.21.15.46:54494) 2024-01-26T23:49:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97bffff700 1 -- 172.21.15.46:0/4043060429 learned_addr learned my addr 172.21.15.46:0/4043060429 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97c49d3700 1 --2- 172.21.15.46:0/4043060429 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97c015cd40 0x7f97c00cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97c51d4700 1 --2- 172.21.15.46:0/4043060429 >> v2:172.21.15.106:3300/0 conn(0x7f97c0067930 0x7f97c0067db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97bffff700 1 -- 172.21.15.46:0/4043060429 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97c015cd40 msgr2=0x7f97c00cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97bffff700 1 --2- 172.21.15.46:0/4043060429 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97c015cd40 0x7f97c00cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97bffff700 1 -- 172.21.15.46:0/4043060429 >> v2:172.21.15.106:3300/0 conn(0x7f97c0067930 msgr2=0x7f97c0067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:49:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97bffff700 1 --2- 172.21.15.46:0/4043060429 >> v2:172.21.15.106:3300/0 conn(0x7f97c0067930 0x7f97c0067db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97bffff700 1 -- 172.21.15.46:0/4043060429 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c00ca8b0 con 0x7f97c015d110 2024-01-26T23:49:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97c51d4700 1 --2- 172.21.15.46:0/4043060429 >> v2:172.21.15.106:3300/0 conn(0x7f97c0067930 0x7f97c0067db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:49:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97bffff700 1 --2- 172.21.15.46:0/4043060429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c015d110 0x7f97c00cb7a0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f97c00accd0 tx=0x7f97ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c2ec79a15b3df6 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.868+0000 7f97bf7fe700 1 -- 172.21.15.46:0/4043060429 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac00b860 con 0x7f97c015d110 2024-01-26T23:49:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.869+0000 7f97bf7fe700 1 -- 172.21.15.46:0/4043060429 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ac00b9c0 con 0x7f97c015d110 2024-01-26T23:49:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.869+0000 7f97bf7fe700 1 -- 172.21.15.46:0/4043060429 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac00bd60 con 0x7f97c015d110 2024-01-26T23:49:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.869+0000 7f97c6c37700 1 -- 172.21.15.46:0/4043060429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c015d110 msgr2=0x7f97c00cb7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.869+0000 7f97c6c37700 1 --2- 172.21.15.46:0/4043060429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c015d110 0x7f97c00cb7a0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f97c00accd0 tx=0x7f97ac009250 comp rx=0 tx=0).stop 2024-01-26T23:49:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.869+0000 7f97c6c37700 1 -- 172.21.15.46:0/4043060429 shutdown_connections 2024-01-26T23:49:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.869+0000 7f97c6c37700 1 --2- 172.21.15.46:0/4043060429 >> v2:172.21.15.106:3300/0 conn(0x7f97c0067930 0x7f97c0067db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.869+0000 7f97c6c37700 1 --2- 172.21.15.46:0/4043060429 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97c015cd40 0x7f97c00cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.869+0000 7f97c6c37700 1 --2- 172.21.15.46:0/4043060429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c015d110 0x7f97c00cb7a0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:59.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.869+0000 7f97c6c37700 1 -- 172.21.15.46:0/4043060429 >> 172.21.15.46:0/4043060429 conn(0x7f97c00cdf50 msgr2=0x7f97c00c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:49:59.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.869+0000 7f97c6c37700 1 -- 172.21.15.46:0/4043060429 shutdown_connections 2024-01-26T23:49:59.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.869+0000 7f97c6c37700 1 -- 172.21.15.46:0/4043060429 wait complete. 2024-01-26T23:49:59.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.870+0000 7f97c6c37700 1 Processor -- start 2024-01-26T23:49:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.871+0000 7f97c6c37700 1 -- start start 2024-01-26T23:49:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.871+0000 7f97c6c37700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c0067930 0x7f97c016c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.871+0000 7f97c6c37700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c015cd40 0x7f97c0156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.871+0000 7f97c6c37700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97c015d110 0x7f97c0156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.871+0000 7f97c6c37700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c01580b0 con 0x7f97c015cd40 2024-01-26T23:49:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.871+0000 7f97c6c37700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00b27f0 con 0x7f97c0067930 2024-01-26T23:49:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.871+0000 7f97c6c37700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c0157430 con 0x7f97c015d110 2024-01-26T23:49:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.872+0000 7f97bffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c015cd40 0x7f97c0156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.872+0000 7f97bffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c015cd40 0x7f97c0156850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54498/0 (socket says 172.21.15.46:54498) 2024-01-26T23:49:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.872+0000 7f97bffff700 1 -- 172.21.15.46:0/40478148 learned_addr learned my addr 172.21.15.46:0/40478148 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:49:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.872+0000 7f97c49d3700 1 --2- 172.21.15.46:0/40478148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c0067930 0x7f97c016c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.872+0000 7f97c51d4700 1 --2- 172.21.15.46:0/40478148 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97c015d110 0x7f97c0156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.872+0000 7f97bffff700 1 -- 172.21.15.46:0/40478148 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97c015d110 msgr2=0x7f97c0156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.872+0000 7f97bffff700 1 --2- 172.21.15.46:0/40478148 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97c015d110 0x7f97c0156d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.872+0000 7f97bffff700 1 -- 172.21.15.46:0/40478148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c0067930 msgr2=0x7f97c016c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:49:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.872+0000 7f97bffff700 1 --2- 172.21.15.46:0/40478148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c0067930 0x7f97c016c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:49:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.872+0000 7f97bffff700 1 -- 172.21.15.46:0/40478148 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ac0094f0 con 0x7f97c015cd40 2024-01-26T23:49:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.872+0000 7f97bffff700 1 --2- 172.21.15.46:0/40478148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c015cd40 0x7f97c0156850 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f97c006e800 tx=0x7f97ac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.872+0000 7f97bcff9700 1 -- 172.21.15.46:0/40478148 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac024440 con 0x7f97c015cd40 2024-01-26T23:49:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.872+0000 7f97bcff9700 1 -- 172.21.15.46:0/40478148 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ac0245a0 con 0x7f97c015cd40 2024-01-26T23:49:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.872+0000 7f97c6c37700 1 -- 172.21.15.46:0/40478148 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c0157710 con 0x7f97c015cd40 2024-01-26T23:49:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.872+0000 7f97bcff9700 1 -- 172.21.15.46:0/40478148 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac024960 con 0x7f97c015cd40 2024-01-26T23:49:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.872+0000 7f97c6c37700 1 -- 172.21.15.46:0/40478148 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c0157cd0 con 0x7f97c015cd40 2024-01-26T23:49:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.873+0000 7f97c6c37700 1 -- 172.21.15.46:0/40478148 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c0157e60 con 0x7f97c015cd40 2024-01-26T23:49:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.874+0000 7f97bcff9700 1 -- 172.21.15.46:0/40478148 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f97ac01a420 con 0x7f97c015cd40 2024-01-26T23:49:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.874+0000 7f97bcff9700 1 --2- 172.21.15.46:0/40478148 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f97a4066f30 0x7f97a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:49:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.874+0000 7f97bcff9700 1 -- 172.21.15.46:0/40478148 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f97ac086520 con 0x7f97c015cd40 2024-01-26T23:49:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.875+0000 7f97c49d3700 1 --2- 172.21.15.46:0/40478148 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f97a4066f30 0x7f97a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:49:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.875+0000 7f97c49d3700 1 --2- 172.21.15.46:0/40478148 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f97a4066f30 0x7f97a40693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f97b4006fd0 tx=0x7f97b4008040 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:49:59.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:49:59.879+0000 7f97bcff9700 1 -- 172.21.15.46:0/40478148 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f97ac053610 con 0x7f97c015cd40 2024-01-26T23:50:00.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.086+0000 7f97c6c37700 1 -- 172.21.15.46:0/40478148 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f97c004c710 con 0x7f97c015cd40 2024-01-26T23:50:00.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.090+0000 7f97bcff9700 1 -- 172.21.15.46:0/40478148 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7f97ac056c30 con 0x7f97c015cd40 2024-01-26T23:50:00.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.092+0000 7f97c6c37700 1 -- 172.21.15.46:0/40478148 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f97a4066f30 msgr2=0x7f97a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:00.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.092+0000 7f97c6c37700 1 --2- 172.21.15.46:0/40478148 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f97a4066f30 0x7f97a40693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f97b4006fd0 tx=0x7f97b4008040 comp rx=0 tx=0).stop 2024-01-26T23:50:00.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.092+0000 7f97c6c37700 1 -- 172.21.15.46:0/40478148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c015cd40 msgr2=0x7f97c0156850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:00.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.092+0000 7f97c6c37700 1 --2- 172.21.15.46:0/40478148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c015cd40 0x7f97c0156850 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f97c006e800 tx=0x7f97ac017000 comp rx=0 tx=0).stop 2024-01-26T23:50:00.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.092+0000 7f97c6c37700 1 -- 172.21.15.46:0/40478148 shutdown_connections 2024-01-26T23:50:00.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.092+0000 7f97c6c37700 1 --2- 172.21.15.46:0/40478148 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f97a4066f30 0x7f97a40693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.092+0000 7f97c6c37700 1 --2- 172.21.15.46:0/40478148 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97c015d110 0x7f97c0156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.092+0000 7f97c6c37700 1 --2- 172.21.15.46:0/40478148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c0067930 0x7f97c016c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.092+0000 7f97c6c37700 1 --2- 172.21.15.46:0/40478148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c015cd40 0x7f97c0156850 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.092+0000 7f97c6c37700 1 -- 172.21.15.46:0/40478148 >> 172.21.15.46:0/40478148 conn(0x7f97c00cdf50 msgr2=0x7f97c00c7c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:00.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.093+0000 7f97c6c37700 1 -- 172.21.15.46:0/40478148 shutdown_connections 2024-01-26T23:50:00.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.093+0000 7f97c6c37700 1 -- 172.21.15.46:0/40478148 wait complete. 2024-01-26T23:50:00.107 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-26T23:50:00.107 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:50:00.152 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-26T23:50:00.199 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-26T23:50:00.200 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-26T23:50:00.200 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:50:00.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.334+0000 7f6947efb700 1 Processor -- start 2024-01-26T23:50:00.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.335+0000 7f6947efb700 1 -- start start 2024-01-26T23:50:00.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.336+0000 7f6947efb700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f694015b9e0 0x7f69400b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:00.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.336+0000 7f6947efb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69400b00b0 0x7f6940160650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:00.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.336+0000 7f6947efb700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694015c050 con 0x7f694015ee60 2024-01-26T23:50:00.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.336+0000 7f6947efb700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69400b15b0 con 0x7f69400b00b0 2024-01-26T23:50:00.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.336+0000 7f6947efb700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69401601c0 con 0x7f694015b9e0 2024-01-26T23:50:00.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.336+0000 7f6946498700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69400b00b0 0x7f6940160650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:00.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.336+0000 7f6946498700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69400b00b0 0x7f6940160650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34440/0 (socket says 172.21.15.46:34440) 2024-01-26T23:50:00.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.336+0000 7f6946498700 1 -- 172.21.15.46:0/4063446570 learned_addr learned my addr 172.21.15.46:0/4063446570 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:00.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.336+0000 7f6945c97700 1 --2- 172.21.15.46:0/4063446570 >> v2:172.21.15.106:3300/0 conn(0x7f694015b9e0 0x7f69400b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:00.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.336+0000 7f6946498700 1 -- 172.21.15.46:0/4063446570 >> v2:172.21.15.106:3300/0 conn(0x7f694015b9e0 msgr2=0x7f69400b1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:00.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.336+0000 7f6946498700 1 --2- 172.21.15.46:0/4063446570 >> v2:172.21.15.106:3300/0 conn(0x7f694015b9e0 0x7f69400b1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.336+0000 7f6946498700 1 -- 172.21.15.46:0/4063446570 >> v1:172.21.15.106:6789/0 conn(0x7f694015ee60 legacy=0x7f69400af9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.337+0000 7f6946498700 1 -- 172.21.15.46:0/4063446570 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69400b2460 con 0x7f69400b00b0 2024-01-26T23:50:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.337+0000 7f6946498700 1 --2- 172.21.15.46:0/4063446570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69400b00b0 0x7f6940160650 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f6940068fc0 tx=0x7f6938009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221d51b7dd4b2bd5 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.337+0000 7f6944c95700 1 -- 172.21.15.46:0/4063446570 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693800e070 con 0x7f69400b00b0 2024-01-26T23:50:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.337+0000 7f6944c95700 1 -- 172.21.15.46:0/4063446570 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693800ba10 con 0x7f69400b00b0 2024-01-26T23:50:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.337+0000 7f6944c95700 1 -- 172.21.15.46:0/4063446570 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69380092e0 con 0x7f69400b00b0 2024-01-26T23:50:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.337+0000 7f6947efb700 1 -- 172.21.15.46:0/4063446570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69400b00b0 msgr2=0x7f6940160650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.337+0000 7f6947efb700 1 --2- 172.21.15.46:0/4063446570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69400b00b0 0x7f6940160650 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f6940068fc0 tx=0x7f6938009840 comp rx=0 tx=0).stop 2024-01-26T23:50:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.337+0000 7f6947efb700 1 -- 172.21.15.46:0/4063446570 shutdown_connections 2024-01-26T23:50:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.338+0000 7f6947efb700 1 --2- 172.21.15.46:0/4063446570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69400b00b0 0x7f6940160650 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.338+0000 7f6947efb700 1 --2- 172.21.15.46:0/4063446570 >> v2:172.21.15.106:3300/0 conn(0x7f694015b9e0 0x7f69400b1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.338+0000 7f6947efb700 1 -- 172.21.15.46:0/4063446570 >> 172.21.15.46:0/4063446570 conn(0x7f6940068400 msgr2=0x7f69400cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.338+0000 7f6947efb700 1 -- 172.21.15.46:0/4063446570 shutdown_connections 2024-01-26T23:50:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.338+0000 7f6947efb700 1 -- 172.21.15.46:0/4063446570 wait complete. 2024-01-26T23:50:00.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.339+0000 7f6947efb700 1 Processor -- start 2024-01-26T23:50:00.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.339+0000 7f6947efb700 1 -- start start 2024-01-26T23:50:00.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6947efb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69400b00b0 0x7f6940152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:00.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6947efb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f694015b9e0 0x7f69400ac840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:00.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6947efb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f694015ee60 0x7f694014f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:00.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6947efb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69400addb0 con 0x7f694015ee60 2024-01-26T23:50:00.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6947efb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69400ad7a0 con 0x7f69400b00b0 2024-01-26T23:50:00.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6947efb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6940153050 con 0x7f694015b9e0 2024-01-26T23:50:00.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6946498700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f694015ee60 0x7f694014f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:00.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6946498700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f694015ee60 0x7f694014f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34448/0 (socket says 172.21.15.46:34448) 2024-01-26T23:50:00.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6946498700 1 -- 172.21.15.46:0/3703832374 learned_addr learned my addr 172.21.15.46:0/3703832374 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:00.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6945c97700 1 --2- 172.21.15.46:0/3703832374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69400b00b0 0x7f6940152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:00.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6945496700 1 --2- 172.21.15.46:0/3703832374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f694015b9e0 0x7f69400ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:00.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6946498700 1 -- 172.21.15.46:0/3703832374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f694015b9e0 msgr2=0x7f69400ac840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:00.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6946498700 1 --2- 172.21.15.46:0/3703832374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f694015b9e0 0x7f69400ac840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6946498700 1 -- 172.21.15.46:0/3703832374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69400b00b0 msgr2=0x7f6940152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:00.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6946498700 1 --2- 172.21.15.46:0/3703832374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69400b00b0 0x7f6940152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6946498700 1 -- 172.21.15.46:0/3703832374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6938009ae0 con 0x7f694015ee60 2024-01-26T23:50:00.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6945496700 1 --2- 172.21.15.46:0/3703832374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f694015b9e0 0x7f69400ac840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:50:00.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6946498700 1 --2- 172.21.15.46:0/3703832374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f694015ee60 0x7f694014f7f0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f6938003000 tx=0x7f6938017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:00.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.340+0000 7f6931ffb700 1 -- 172.21.15.46:0/3703832374 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693800bee0 con 0x7f694015ee60 2024-01-26T23:50:00.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.341+0000 7f6931ffb700 1 -- 172.21.15.46:0/3703832374 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69380032f0 con 0x7f694015ee60 2024-01-26T23:50:00.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.341+0000 7f6947efb700 1 -- 172.21.15.46:0/3703832374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69400ada40 con 0x7f694015ee60 2024-01-26T23:50:00.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.341+0000 7f6931ffb700 1 -- 172.21.15.46:0/3703832374 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693801d420 con 0x7f694015ee60 2024-01-26T23:50:00.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.341+0000 7f6947efb700 1 -- 172.21.15.46:0/3703832374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6940153330 con 0x7f694015ee60 2024-01-26T23:50:00.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.342+0000 7f6947efb700 1 -- 172.21.15.46:0/3703832374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69400adf20 con 0x7f694015ee60 2024-01-26T23:50:00.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.342+0000 7f6931ffb700 1 -- 172.21.15.46:0/3703832374 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f693801d850 con 0x7f694015ee60 2024-01-26T23:50:00.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.343+0000 7f6931ffb700 1 --2- 172.21.15.46:0/3703832374 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6924066f30 0x7f69240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:00.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.343+0000 7f6931ffb700 1 -- 172.21.15.46:0/3703832374 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f6938085e80 con 0x7f694015ee60 2024-01-26T23:50:00.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.343+0000 7f6945c97700 1 --2- 172.21.15.46:0/3703832374 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6924066f30 0x7f69240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:00.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.344+0000 7f6945c97700 1 --2- 172.21.15.46:0/3703832374 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6924066f30 0x7f69240693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6934006fd0 tx=0x7f6934008040 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:00.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.347+0000 7f6931ffb700 1 -- 172.21.15.46:0/3703832374 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6938052ef0 con 0x7f694015ee60 2024-01-26T23:50:00.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.602+0000 7f6947efb700 1 -- 172.21.15.46:0/3703832374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f694004c710 con 0x7f694015ee60 2024-01-26T23:50:00.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.602+0000 7f6931ffb700 1 -- 172.21.15.46:0/3703832374 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+3529 (secure 0 0 0) 0x7f693802ade0 con 0x7f694015ee60 2024-01-26T23:50:00.603 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:50:00.603 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":72,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/406559819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":406559819},{"type":"v1","addr":"172.21.15.106:6836","nonce":406559819}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/3387155671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":3387155671},{"type":"v1","addr":"172.21.15.106:6834","nonce":3387155671}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8298,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/855914203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":855914203},{"type":"v1","addr":"172.21.15.46:6834","nonce":855914203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8476,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/4146802338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":4146802338},{"type":"v1","addr":"172.21.15.46:6835","nonce":4146802338}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[]} 2024-01-26T23:50:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.604+0000 7f6947efb700 1 -- 172.21.15.46:0/3703832374 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6924066f30 msgr2=0x7f69240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.604+0000 7f6947efb700 1 --2- 172.21.15.46:0/3703832374 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6924066f30 0x7f69240693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6934006fd0 tx=0x7f6934008040 comp rx=0 tx=0).stop 2024-01-26T23:50:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.604+0000 7f6947efb700 1 -- 172.21.15.46:0/3703832374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f694015ee60 msgr2=0x7f694014f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.604+0000 7f6947efb700 1 --2- 172.21.15.46:0/3703832374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f694015ee60 0x7f694014f7f0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f6938003000 tx=0x7f6938017000 comp rx=0 tx=0).stop 2024-01-26T23:50:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.604+0000 7f6947efb700 1 -- 172.21.15.46:0/3703832374 shutdown_connections 2024-01-26T23:50:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.604+0000 7f6947efb700 1 --2- 172.21.15.46:0/3703832374 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6924066f30 0x7f69240693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.604+0000 7f6947efb700 1 --2- 172.21.15.46:0/3703832374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f694015ee60 0x7f694014f7f0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.604+0000 7f6947efb700 1 --2- 172.21.15.46:0/3703832374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69400b00b0 0x7f6940152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.605+0000 7f6947efb700 1 --2- 172.21.15.46:0/3703832374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f694015b9e0 0x7f69400ac840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.605+0000 7f6947efb700 1 -- 172.21.15.46:0/3703832374 >> 172.21.15.46:0/3703832374 conn(0x7f6940068400 msgr2=0x7f6940162260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:00.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.605+0000 7f6947efb700 1 -- 172.21.15.46:0/3703832374 shutdown_connections 2024-01-26T23:50:00.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.605+0000 7f6947efb700 1 -- 172.21.15.46:0/3703832374 wait complete. 2024-01-26T23:50:00.607 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 72 2024-01-26T23:50:00.619 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-26T23:50:00.619 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-26T23:50:00.620 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-26T23:50:00.620 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:50:00.620 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-26T23:50:00.620 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-26T23:50:00.621 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-26T23:50:00.621 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:50:00.621 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-26T23:50:00.621 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-26T23:50:00.621 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-26T23:50:00.621 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:50:00.621 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-26T23:50:00.622 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-26T23:50:00.622 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-26T23:50:00.622 INFO:teuthology.orchestra.run:waiting for 300 2024-01-26T23:50:00.632 INFO:tasks.ceph.mds.c:Stopped 2024-01-26T23:50:00.632 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-26T23:50:00.633 INFO:tasks.ceph.mds.a:Stopped 2024-01-26T23:50:00.633 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-26T23:50:00.634 INFO:tasks.ceph.mds.d:Stopped 2024-01-26T23:50:00.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-26T23:50:00.635 INFO:tasks.ceph.mds.b:Stopped 2024-01-26T23:50:00.635 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-26T23:50:00.675 INFO:tasks.ceph.mds.c:Started 2024-01-26T23:50:00.675 DEBUG:teuthology.parallel:result is None 2024-01-26T23:50:00.675 INFO:tasks.ceph.mds.a:Started 2024-01-26T23:50:00.676 DEBUG:teuthology.parallel:result is None 2024-01-26T23:50:00.677 INFO:tasks.ceph.mds.d:Started 2024-01-26T23:50:00.678 DEBUG:teuthology.parallel:result is None 2024-01-26T23:50:00.678 INFO:tasks.ceph.mds.b:Started 2024-01-26T23:50:00.679 DEBUG:teuthology.parallel:result is None 2024-01-26T23:50:00.679 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-26T23:50:00.763 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-01-26T23:50:00.763 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-26T23:50:00.764 INFO:tasks.ceph.mds.b.smithi106.stdout:starting mds.b at 2024-01-26T23:50:00.764 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-26T23:50:00.764 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-01-26T23:50:00.763+0000 7f8b1409f700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-01-26T23:50:00.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.818+0000 7f10eb23c700 1 Processor -- start 2024-01-26T23:50:00.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.818+0000 7f10eb23c700 1 -- start start 2024-01-26T23:50:00.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.819+0000 7f10eb23c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10e415ce40 0x7f10e40d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:00.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.819+0000 7f10eb23c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f10e415d210 0x7f10e40cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:00.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.819+0000 7f10eb23c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e40cff70 con 0x7f10e40d1390 2024-01-26T23:50:00.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.819+0000 7f10eb23c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e40cf400 con 0x7f10e415d210 2024-01-26T23:50:00.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.819+0000 7f10eb23c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e40cf660 con 0x7f10e415ce40 2024-01-26T23:50:00.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.819+0000 7f10e8fd8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10e415ce40 0x7f10e40d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:00.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.819+0000 7f10e3fff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f10e415d210 0x7f10e40cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:00.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.819+0000 7f10e8fd8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10e415ce40 0x7f10e40d0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:47048/0 (socket says 172.21.15.46:47048) 2024-01-26T23:50:00.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.819+0000 7f10e8fd8700 1 -- 172.21.15.46:0/3424403876 learned_addr learned my addr 172.21.15.46:0/3424403876 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:00.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.819+0000 7f10e3fff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f10e415d210 0x7f10e40cbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:33824/0 (socket says 172.21.15.46:33824) 2024-01-26T23:50:00.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.820+0000 7f10e8fd8700 1 -- 172.21.15.46:0/3424403876 >> v2:172.21.15.106:3300/0 conn(0x7f10e415d210 msgr2=0x7f10e40cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:00.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.820+0000 7f10e8fd8700 1 --2- 172.21.15.46:0/3424403876 >> v2:172.21.15.106:3300/0 conn(0x7f10e415d210 0x7f10e40cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.820+0000 7f10e8fd8700 1 -- 172.21.15.46:0/3424403876 >> v1:172.21.15.106:6789/0 conn(0x7f10e40d1390 legacy=0x7f10e40ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.820+0000 7f10e8fd8700 1 -- 172.21.15.46:0/3424403876 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e40d0500 con 0x7f10e415ce40 2024-01-26T23:50:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.820+0000 7f10e3fff700 1 --2- 172.21.15.46:0/3424403876 >> v2:172.21.15.106:3300/0 conn(0x7f10e415d210 0x7f10e40cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:50:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.820+0000 7f10e37fe700 1 -- 172.21.15.46:0/3424403876 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199967911 0 0) 0x7f10e40cff70 con 0x7f10e40d1390 2024-01-26T23:50:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.820+0000 7f10e8fd8700 1 --2- 172.21.15.46:0/3424403876 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10e415ce40 0x7f10e40d0e50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009990 tx=0x7f10d4009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f9ff649df476e42 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.820+0000 7f10e37fe700 1 -- 172.21.15.46:0/3424403876 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d4009e80 con 0x7f10e415ce40 2024-01-26T23:50:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.820+0000 7f10e37fe700 1 -- 172.21.15.46:0/3424403876 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d4005c10 con 0x7f10e415ce40 2024-01-26T23:50:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.820+0000 7f10e37fe700 1 -- 172.21.15.46:0/3424403876 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d400b650 con 0x7f10e415ce40 2024-01-26T23:50:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.821+0000 7f10eb23c700 1 -- 172.21.15.46:0/3424403876 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10e415ce40 msgr2=0x7f10e40d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.821+0000 7f10eb23c700 1 --2- 172.21.15.46:0/3424403876 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10e415ce40 0x7f10e40d0e50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009990 tx=0x7f10d4009220 comp rx=0 tx=0).stop 2024-01-26T23:50:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.821+0000 7f10eb23c700 1 -- 172.21.15.46:0/3424403876 shutdown_connections 2024-01-26T23:50:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.821+0000 7f10eb23c700 1 --2- 172.21.15.46:0/3424403876 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10e415ce40 0x7f10e40d0e50 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.821+0000 7f10eb23c700 1 --2- 172.21.15.46:0/3424403876 >> v2:172.21.15.106:3300/0 conn(0x7f10e415d210 0x7f10e40cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.821+0000 7f10eb23c700 1 -- 172.21.15.46:0/3424403876 >> 172.21.15.46:0/3424403876 conn(0x7f10e406f840 msgr2=0x7f10e40d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.821+0000 7f10eb23c700 1 -- 172.21.15.46:0/3424403876 shutdown_connections 2024-01-26T23:50:00.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.821+0000 7f10eb23c700 1 -- 172.21.15.46:0/3424403876 wait complete. 2024-01-26T23:50:00.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.822+0000 7f10eb23c700 1 Processor -- start 2024-01-26T23:50:00.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.822+0000 7f10eb23c700 1 -- start start 2024-01-26T23:50:00.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.823+0000 7f10eb23c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10e40d1390 0x7f10e414ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:00.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.823+0000 7f10eb23c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10e415ce40 0x7f10e40c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.823+0000 7f10eb23c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e415d210 0x7f10e409c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.823+0000 7f10eb23c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e409bcb0 con 0x7f10e415d210 2024-01-26T23:50:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.823+0000 7f10eb23c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e414f120 con 0x7f10e40d1390 2024-01-26T23:50:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.823+0000 7f10eb23c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e409adc0 con 0x7f10e415ce40 2024-01-26T23:50:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.823+0000 7f10e97d9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e415d210 0x7f10e409c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.823+0000 7f10e97d9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e415d210 0x7f10e409c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34490/0 (socket says 172.21.15.46:34490) 2024-01-26T23:50:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.823+0000 7f10e97d9700 1 -- 172.21.15.46:0/3686844557 learned_addr learned my addr 172.21.15.46:0/3686844557 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.823+0000 7f10e3fff700 1 --2- 172.21.15.46:0/3686844557 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10e415ce40 0x7f10e40c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.823+0000 7f10e8fd8700 1 --2- 172.21.15.46:0/3686844557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10e40d1390 0x7f10e414ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.824+0000 7f10e97d9700 1 -- 172.21.15.46:0/3686844557 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10e415ce40 msgr2=0x7f10e40c64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.824+0000 7f10e97d9700 1 --2- 172.21.15.46:0/3686844557 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10e415ce40 0x7f10e40c64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.824+0000 7f10e97d9700 1 -- 172.21.15.46:0/3686844557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10e40d1390 msgr2=0x7f10e414ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:00.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.824+0000 7f10e97d9700 1 --2- 172.21.15.46:0/3686844557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10e40d1390 0x7f10e414ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:00.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.824+0000 7f10e97d9700 1 -- 172.21.15.46:0/3686844557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc009420 con 0x7f10e415d210 2024-01-26T23:50:00.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.824+0000 7f10e97d9700 1 --2- 172.21.15.46:0/3686844557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e415d210 0x7f10e409c140 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00c960 tx=0x7f10dc0016c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:00.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.824+0000 7f10e0ff9700 1 -- 172.21.15.46:0/3686844557 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc012630 con 0x7f10e415d210 2024-01-26T23:50:00.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.824+0000 7f10e0ff9700 1 -- 172.21.15.46:0/3686844557 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10dc012790 con 0x7f10e415d210 2024-01-26T23:50:00.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.824+0000 7f10e0ff9700 1 -- 172.21.15.46:0/3686844557 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc012b50 con 0x7f10e415d210 2024-01-26T23:50:00.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.824+0000 7f10eb23c700 1 -- 172.21.15.46:0/3686844557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d4009480 con 0x7f10e415d210 2024-01-26T23:50:00.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.824+0000 7f10eb23c700 1 -- 172.21.15.46:0/3686844557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e409b380 con 0x7f10e415d210 2024-01-26T23:50:00.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.825+0000 7f10eb23c700 1 -- 172.21.15.46:0/3686844557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e409be20 con 0x7f10e415d210 2024-01-26T23:50:00.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.826+0000 7f10e0ff9700 1 -- 172.21.15.46:0/3686844557 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f10dc01a3a0 con 0x7f10e415d210 2024-01-26T23:50:00.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.826+0000 7f10e0ff9700 1 --2- 172.21.15.46:0/3686844557 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f10c4066f30 0x7f10c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:00.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.826+0000 7f10e0ff9700 1 -- 172.21.15.46:0/3686844557 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6210+0+0 (secure 0 0 0) 0x7f10dc0848e0 con 0x7f10e415d210 2024-01-26T23:50:00.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.826+0000 7f10e8fd8700 1 --2- 172.21.15.46:0/3686844557 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f10c4066f30 0x7f10c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:00.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.827+0000 7f10e8fd8700 1 --2- 172.21.15.46:0/3686844557 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f10c4066f30 0x7f10c40693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009260 tx=0x7f10d4016000 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:00.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:00.831+0000 7f10e0ff9700 1 -- 172.21.15.46:0/3686844557 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f10dc051950 con 0x7f10e415d210 2024-01-26T23:50:01.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.053+0000 7f10eb23c700 1 -- 172.21.15.46:0/3686844557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f10e40cf920 con 0x7f10e415d210 2024-01-26T23:50:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.724+0000 7f10e0ff9700 1 -- 172.21.15.46:0/3686844557 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v153) v1 ==== 97+0+0 (secure 0 0 0) 0x7f10dc054f70 con 0x7f10e415d210 2024-01-26T23:50:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.727+0000 7f10eb23c700 1 -- 172.21.15.46:0/3686844557 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f10c4066f30 msgr2=0x7f10c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.727+0000 7f10eb23c700 1 --2- 172.21.15.46:0/3686844557 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f10c4066f30 0x7f10c40693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009260 tx=0x7f10d4016000 comp rx=0 tx=0).stop 2024-01-26T23:50:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.727+0000 7f10eb23c700 1 -- 172.21.15.46:0/3686844557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e415d210 msgr2=0x7f10e409c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.727+0000 7f10eb23c700 1 --2- 172.21.15.46:0/3686844557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e415d210 0x7f10e409c140 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00c960 tx=0x7f10dc0016c0 comp rx=0 tx=0).stop 2024-01-26T23:50:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.727+0000 7f10eb23c700 1 -- 172.21.15.46:0/3686844557 shutdown_connections 2024-01-26T23:50:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.727+0000 7f10eb23c700 1 --2- 172.21.15.46:0/3686844557 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f10c4066f30 0x7f10c40693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.727+0000 7f10eb23c700 1 --2- 172.21.15.46:0/3686844557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e415d210 0x7f10e409c140 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.727+0000 7f10eb23c700 1 --2- 172.21.15.46:0/3686844557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f10e40d1390 0x7f10e414ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.727+0000 7f10eb23c700 1 --2- 172.21.15.46:0/3686844557 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10e415ce40 0x7f10e40c64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.727+0000 7f10eb23c700 1 -- 172.21.15.46:0/3686844557 >> 172.21.15.46:0/3686844557 conn(0x7f10e406f840 msgr2=0x7f10e40cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.727+0000 7f10eb23c700 1 -- 172.21.15.46:0/3686844557 shutdown_connections 2024-01-26T23:50:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.727+0000 7f10eb23c700 1 -- 172.21.15.46:0/3686844557 wait complete. 2024-01-26T23:50:01.730 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-01-26T23:50:01.745 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-26T23:50:01.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.880+0000 7effa6d97700 1 Processor -- start 2024-01-26T23:50:01.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.880+0000 7effa6d97700 1 -- start start 2024-01-26T23:50:01.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.881+0000 7effa6d97700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa0159760 0x7effa00b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:01.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.881+0000 7effa6d97700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa006f940 con 0x7effa00b12c0 2024-01-26T23:50:01.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.881+0000 7effa6d97700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa006ed30 con 0x7effa00b3060 2024-01-26T23:50:01.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.881+0000 7effa6d97700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa006f010 con 0x7effa0159760 2024-01-26T23:50:01.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.881+0000 7effa5334700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7effa00b12c0 0x7effa00a9840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48900/0 (socket says 172.21.15.46:48900) 2024-01-26T23:50:01.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.881+0000 7effa5334700 1 -- 172.21.15.46:0/3528137067 learned_addr learned my addr 172.21.15.46:0/3528137067 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:01.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7effa4b33700 1 --2- 172.21.15.46:0/3528137067 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa0159760 0x7effa00b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:01.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9f7fe700 1 -- 172.21.15.46:0/3528137067 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102937817 0 0) 0x7effa006f940 con 0x7effa00b12c0 2024-01-26T23:50:01.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9f7fe700 1 -- 172.21.15.46:0/3528137067 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff90003570 con 0x7effa00b12c0 2024-01-26T23:50:01.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9f7fe700 1 -- 172.21.15.46:0/3528137067 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300446861 0 0) 0x7effa006ed30 con 0x7effa00b3060 2024-01-26T23:50:01.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9f7fe700 1 -- 172.21.15.46:0/3528137067 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effa006f940 con 0x7effa00b3060 2024-01-26T23:50:01.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9f7fe700 1 -- 172.21.15.46:0/3528137067 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 69523903 0 0) 0x7eff90003570 con 0x7effa00b12c0 2024-01-26T23:50:01.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9f7fe700 1 -- 172.21.15.46:0/3528137067 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa0159760 msgr2=0x7effa00b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:01.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9f7fe700 1 --2- 172.21.15.46:0/3528137067 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa0159760 0x7effa00b2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:01.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9f7fe700 1 -- 172.21.15.46:0/3528137067 >> v1:172.21.15.106:6789/0 conn(0x7effa00b3060 legacy=0x7effa00b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:01.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9f7fe700 1 -- 172.21.15.46:0/3528137067 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa006f2f0 con 0x7effa00b12c0 2024-01-26T23:50:01.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9f7fe700 1 -- 172.21.15.46:0/3528137067 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7eff980017f0 con 0x7effa00b12c0 2024-01-26T23:50:01.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9f7fe700 1 -- 172.21.15.46:0/3528137067 >> v1:172.21.15.46:6789/0 conn(0x7effa00b12c0 legacy=0x7effa00a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:01.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9f7fe700 1 --2- 172.21.15.46:0/3528137067 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff90003c20 0x7eff90004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9f7fe700 1 --2- 172.21.15.46:0/3528137067 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa0159760 0x7eff90004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9f7fe700 1 --2- 172.21.15.46:0/3528137067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff90004d50 0x7eff900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9f7fe700 1 -- 172.21.15.46:0/3528137067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa006f010 con 0x7eff90004d50 2024-01-26T23:50:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9f7fe700 1 -- 172.21.15.46:0/3528137067 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9000bbf0 con 0x7eff90003c20 2024-01-26T23:50:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9f7fe700 1 -- 172.21.15.46:0/3528137067 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9000bd60 con 0x7effa0159760 2024-01-26T23:50:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7effa5334700 1 --2- 172.21.15.46:0/3528137067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff90004d50 0x7eff900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9ffff700 1 --2- 172.21.15.46:0/3528137067 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa0159760 0x7eff90004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7effa4b33700 1 --2- 172.21.15.46:0/3528137067 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff90003c20 0x7eff90004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7effa5334700 1 -- 172.21.15.46:0/3528137067 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa0159760 msgr2=0x7eff90004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7effa5334700 1 --2- 172.21.15.46:0/3528137067 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa0159760 0x7eff90004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7effa5334700 1 -- 172.21.15.46:0/3528137067 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff90003c20 msgr2=0x7eff90004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7effa5334700 1 --2- 172.21.15.46:0/3528137067 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff90003c20 0x7eff90004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7effa5334700 1 -- 172.21.15.46:0/3528137067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff90003570 con 0x7eff90004d50 2024-01-26T23:50:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.882+0000 7eff9ffff700 1 --2- 172.21.15.46:0/3528137067 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effa0159760 0x7eff90004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:50:01.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.883+0000 7effa5334700 1 --2- 172.21.15.46:0/3528137067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff90004d50 0x7eff900051c0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7effa006eb40 tx=0x7eff980020c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e29dd2c0475eba server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:01.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.883+0000 7eff9f7fe700 1 -- 172.21.15.46:0/3528137067 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff980070d0 con 0x7eff90004d50 2024-01-26T23:50:01.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.883+0000 7eff9f7fe700 1 -- 172.21.15.46:0/3528137067 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff98007250 con 0x7eff90004d50 2024-01-26T23:50:01.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.883+0000 7eff9f7fe700 1 -- 172.21.15.46:0/3528137067 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff98007610 con 0x7eff90004d50 2024-01-26T23:50:01.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.883+0000 7effa6d97700 1 -- 172.21.15.46:0/3528137067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff90004d50 msgr2=0x7eff900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:01.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.883+0000 7effa6d97700 1 --2- 172.21.15.46:0/3528137067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff90004d50 0x7eff900051c0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7effa006eb40 tx=0x7eff980020c0 comp rx=0 tx=0).stop 2024-01-26T23:50:01.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.883+0000 7effa5334700 1 -- 172.21.15.46:0/3528137067 reap_dead start 2024-01-26T23:50:01.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.883+0000 7effa6d97700 1 -- 172.21.15.46:0/3528137067 shutdown_connections 2024-01-26T23:50:01.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.883+0000 7effa6d97700 1 -- 172.21.15.46:0/3528137067 >> 172.21.15.46:0/3528137067 conn(0x7effa0069510 msgr2=0x7effa00aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:01.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.883+0000 7effa6d97700 1 -- 172.21.15.46:0/3528137067 shutdown_connections 2024-01-26T23:50:01.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.884+0000 7effa6d97700 1 -- 172.21.15.46:0/3528137067 wait complete. 2024-01-26T23:50:01.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.885+0000 7effa6d97700 1 Processor -- start 2024-01-26T23:50:01.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.885+0000 7effa6d97700 1 -- start start 2024-01-26T23:50:01.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7effa6d97700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff90004d50 0x7effa016aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:01.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7effa6d97700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effa016d520 0x7effa00c4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:01.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7effa6d97700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effa0166f60 0x7effa01673c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:01.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7effa6d97700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa016dd50 con 0x7effa0166f60 2024-01-26T23:50:01.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7effa6d97700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa016e030 con 0x7effa016d520 2024-01-26T23:50:01.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7effa6d97700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa0167c00 con 0x7eff90004d50 2024-01-26T23:50:01.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7effa5334700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effa0166f60 0x7effa01673c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:01.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7effa5334700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effa0166f60 0x7effa01673c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34500/0 (socket says 172.21.15.46:34500) 2024-01-26T23:50:01.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7effa5334700 1 -- 172.21.15.46:0/1557231875 learned_addr learned my addr 172.21.15.46:0/1557231875 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:01.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7eff9ffff700 1 --2- 172.21.15.46:0/1557231875 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effa016d520 0x7effa00c4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:01.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7effa4b33700 1 --2- 172.21.15.46:0/1557231875 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff90004d50 0x7effa016aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:01.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7effa5334700 1 -- 172.21.15.46:0/1557231875 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff90004d50 msgr2=0x7effa016aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:01.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7effa5334700 1 --2- 172.21.15.46:0/1557231875 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff90004d50 0x7effa016aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:01.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7effa5334700 1 -- 172.21.15.46:0/1557231875 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effa016d520 msgr2=0x7effa00c4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:01.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7effa5334700 1 --2- 172.21.15.46:0/1557231875 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effa016d520 0x7effa00c4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:01.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7effa5334700 1 -- 172.21.15.46:0/1557231875 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa0167ee0 con 0x7effa0166f60 2024-01-26T23:50:01.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7effa5334700 1 --2- 172.21.15.46:0/1557231875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effa0166f60 0x7effa01673c0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7eff98016040 tx=0x7eff98012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:01.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7eff9cff9700 1 -- 172.21.15.46:0/1557231875 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff98007b60 con 0x7effa0166f60 2024-01-26T23:50:01.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7eff9cff9700 1 -- 172.21.15.46:0/1557231875 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff98012400 con 0x7effa0166f60 2024-01-26T23:50:01.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7eff9cff9700 1 -- 172.21.15.46:0/1557231875 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff980127a0 con 0x7effa0166f60 2024-01-26T23:50:01.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.886+0000 7effa6d97700 1 -- 172.21.15.46:0/1557231875 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa0167900 con 0x7effa0166f60 2024-01-26T23:50:01.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.887+0000 7effa6d97700 1 -- 172.21.15.46:0/1557231875 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa00c5330 con 0x7effa0166f60 2024-01-26T23:50:01.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.887+0000 7effa6d97700 1 -- 172.21.15.46:0/1557231875 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa00c5520 con 0x7effa0166f60 2024-01-26T23:50:01.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.888+0000 7eff9cff9700 1 -- 172.21.15.46:0/1557231875 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7eff9801a070 con 0x7effa0166f60 2024-01-26T23:50:01.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.888+0000 7eff9cff9700 1 --2- 172.21.15.46:0/1557231875 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff84066f30 0x7eff840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:01.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.889+0000 7eff9cff9700 1 -- 172.21.15.46:0/1557231875 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5565+0+0 (secure 0 0 0) 0x7eff98086220 con 0x7effa0166f60 2024-01-26T23:50:01.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.889+0000 7effa4b33700 1 --2- 172.21.15.46:0/1557231875 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff84066f30 0x7eff840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:01.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.889+0000 7effa4b33700 1 --2- 172.21.15.46:0/1557231875 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff84066f30 0x7eff840693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7eff94009d10 tx=0x7eff94008040 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:01.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:01.893+0000 7eff9cff9700 1 -- 172.21.15.46:0/1557231875 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7eff98053520 con 0x7effa0166f60 2024-01-26T23:50:02.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.180+0000 7effa6d97700 1 -- 172.21.15.46:0/1557231875 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7effa004c710 con 0x7effa0166f60 2024-01-26T23:50:02.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.180+0000 7eff9cff9700 1 -- 172.21.15.46:0/1557231875 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v23) v1 ==== 80+0+5304 (secure 0 0 0) 0x7eff98018090 con 0x7effa0166f60 2024-01-26T23:50:02.182 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:50:02.182 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:50:02.182 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-01-26T23:50:02.182 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.182 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-01-26T23:50:02.182 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlM6dKKBAAPpeWeDZWrFgc/4zuxKCvHw==", 2024-01-26T23:50:02.182 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.182 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:50:02.182 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:50:02.183 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:50:02.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:50:02.183 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.183 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.183 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.183 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-01-26T23:50:02.183 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlmzHRNRAAflWYZ/p4AXeMy54tvLpugA==", 2024-01-26T23:50:02.183 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.183 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:50:02.183 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:50:02.183 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:50:02.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:50:02.183 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.183 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.184 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.184 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-01-26T23:50:02.184 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlMZxoMBAAG5kyfm+LLG86y/p2A8BfDQ==", 2024-01-26T23:50:02.184 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.184 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:50:02.184 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:50:02.184 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:50:02.184 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:50:02.184 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.184 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.184 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.184 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-01-26T23:50:02.184 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl5jx/ARAA3cwtLAOWQ/n2NG+xwEZ3QQ==", 2024-01-26T23:50:02.185 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.185 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:50:02.185 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:50:02.185 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:50:02.185 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:50:02.185 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.185 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.185 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.185 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-01-26T23:50:02.185 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4QLRlJ2OKJRAAWVhxEdQkUW20294w/KLL6g==", 2024-01-26T23:50:02.185 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.185 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:50:02.185 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:50:02.185 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:50:02.185 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.186 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.186 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.186 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-01-26T23:50:02.186 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7QLRlE2tAJRAAzNlTiCr6a2eWiabRy1SqSw==", 2024-01-26T23:50:02.186 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.186 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:50:02.186 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:50:02.186 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:50:02.186 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.186 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.186 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.186 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-01-26T23:50:02.186 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+QLRlyySaJBAA5072ba5HUNi2dYE3D02M/Q==", 2024-01-26T23:50:02.186 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.187 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:50:02.187 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:50:02.187 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:50:02.187 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.187 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.187 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.187 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-01-26T23:50:02.187 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBQLRl9+tAIxAApZk0btXIpQPWSb3BYUYsFQ==", 2024-01-26T23:50:02.187 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.187 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:50:02.187 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:50:02.187 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:50:02.187 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.187 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.188 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.188 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-01-26T23:50:02.188 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHQLRlurUtJxAAT9fwKrAWl62VOXwShg7NJg==", 2024-01-26T23:50:02.188 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.188 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:50:02.188 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:50:02.188 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:50:02.188 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.188 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.188 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.188 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-01-26T23:50:02.188 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKQLRl7aqoJRAAXu9fykWELsW3SwnahoIOLA==", 2024-01-26T23:50:02.188 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.188 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:50:02.188 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:50:02.189 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:50:02.189 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.189 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.189 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.189 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-01-26T23:50:02.189 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNQLRlWf7pJBAAopmFLIKb8s32YZs46VwbGg==", 2024-01-26T23:50:02.189 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.189 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:50:02.189 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:50:02.189 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:50:02.189 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.189 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.189 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.189 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-01-26T23:50:02.189 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQQLRlNGfhIxAAmCgmOB3QAGfSIYHBb5BUZw==", 2024-01-26T23:50:02.190 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.190 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-26T23:50:02.190 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-26T23:50:02.190 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:50:02.190 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.190 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.190 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.190 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-01-26T23:50:02.190 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRleO8wBhAA1018AcO3KcyOZxgB0Itxaw==", 2024-01-26T23:50:02.190 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.190 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:50:02.190 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-01-26T23:50:02.190 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-26T23:50:02.190 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.190 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.191 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.191 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-01-26T23:50:02.191 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlsiZEDRAAjXSamHe2bZcDqVsqXqIcGg==", 2024-01-26T23:50:02.191 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.191 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:50:02.191 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:50:02.191 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:50:02.191 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:50:02.191 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.191 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.191 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.191 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-01-26T23:50:02.191 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl6YT1CRAApXlN4j20xzwda0tDMy94Eg==", 2024-01-26T23:50:02.191 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.191 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:50:02.192 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:50:02.192 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:50:02.192 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:50:02.192 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.192 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.192 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.192 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-01-26T23:50:02.192 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlHvDQEhAAxMhgWJsZC5/6Af5zFV6XIA==", 2024-01-26T23:50:02.192 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.192 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-26T23:50:02.192 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-26T23:50:02.192 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-26T23:50:02.192 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-26T23:50:02.192 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.193 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.193 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.193 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-01-26T23:50:02.193 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlvMwDCBAAXnFyMduz7PfWdLWbjytIiQ==", 2024-01-26T23:50:02.193 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.193 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:50:02.193 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-26T23:50:02.193 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-26T23:50:02.193 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:50:02.193 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.193 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.193 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.193 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-01-26T23:50:02.193 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlhMNSJBAA38QU6DrCvFjO+u9zQ8bkEA==", 2024-01-26T23:50:02.193 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.194 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:50:02.194 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-26T23:50:02.194 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:50:02.194 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.194 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:02.194 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:02.194 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-01-26T23:50:02.194 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlI9EbIBAACTpzAO+8A/VFJZcT/WWBng==", 2024-01-26T23:50:02.194 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-26T23:50:02.194 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-26T23:50:02.194 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-26T23:50:02.194 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-26T23:50:02.194 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.194 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:02.194 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-01-26T23:50:02.195 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:50:02.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.183+0000 7effa6d97700 1 -- 172.21.15.46:0/1557231875 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff84066f30 msgr2=0x7eff840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:02.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.183+0000 7effa6d97700 1 --2- 172.21.15.46:0/1557231875 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff84066f30 0x7eff840693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7eff94009d10 tx=0x7eff94008040 comp rx=0 tx=0).stop 2024-01-26T23:50:02.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.183+0000 7effa6d97700 1 -- 172.21.15.46:0/1557231875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effa0166f60 msgr2=0x7effa01673c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:02.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.183+0000 7effa6d97700 1 --2- 172.21.15.46:0/1557231875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effa0166f60 0x7effa01673c0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7eff98016040 tx=0x7eff98012000 comp rx=0 tx=0).stop 2024-01-26T23:50:02.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.183+0000 7effa6d97700 1 -- 172.21.15.46:0/1557231875 shutdown_connections 2024-01-26T23:50:02.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.183+0000 7effa6d97700 1 --2- 172.21.15.46:0/1557231875 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff84066f30 0x7eff840693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.183+0000 7effa6d97700 1 --2- 172.21.15.46:0/1557231875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effa0166f60 0x7effa01673c0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.183+0000 7effa6d97700 1 --2- 172.21.15.46:0/1557231875 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff90004d50 0x7effa016aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.183+0000 7effa6d97700 1 --2- 172.21.15.46:0/1557231875 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effa016d520 0x7effa00c4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.183+0000 7effa6d97700 1 -- 172.21.15.46:0/1557231875 >> 172.21.15.46:0/1557231875 conn(0x7effa0069510 msgr2=0x7effa015bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:02.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.183+0000 7effa6d97700 1 -- 172.21.15.46:0/1557231875 shutdown_connections 2024-01-26T23:50:02.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.183+0000 7effa6d97700 1 -- 172.21.15.46:0/1557231875 wait complete. 2024-01-26T23:50:02.198 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:50:02.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.328+0000 7fd2fdf75700 1 Processor -- start 2024-01-26T23:50:02.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.328+0000 7fd2fdf75700 1 -- start start 2024-01-26T23:50:02.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.329+0000 7fd2fdf75700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f8157340 0x7fd2f815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:02.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.329+0000 7fd2fdf75700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f8158ae0 con 0x7fd2f80ae520 2024-01-26T23:50:02.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.329+0000 7fd2fdf75700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f815b650 con 0x7fd2f80b4180 2024-01-26T23:50:02.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.329+0000 7fd2fdf75700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f815c020 con 0x7fd2f8157340 2024-01-26T23:50:02.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.329+0000 7fd2f6ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd2f80ae520 0x7fd2f80b3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48916/0 (socket says 172.21.15.46:48916) 2024-01-26T23:50:02.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.329+0000 7fd2f6ffd700 1 -- 172.21.15.46:0/2052814366 learned_addr learned my addr 172.21.15.46:0/2052814366 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:02.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.329+0000 7fd2f77fe700 1 --2- 172.21.15.46:0/2052814366 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f8157340 0x7fd2f815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:02.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.329+0000 7fd2f67fc700 1 -- 172.21.15.46:0/2052814366 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865798288 0 0) 0x7fd2f8158ae0 con 0x7fd2f80ae520 2024-01-26T23:50:02.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.329+0000 7fd2f67fc700 1 -- 172.21.15.46:0/2052814366 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e0003570 con 0x7fd2f80ae520 2024-01-26T23:50:02.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.329+0000 7fd2f67fc700 1 -- 172.21.15.46:0/2052814366 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372486148 0 0) 0x7fd2f815b650 con 0x7fd2f80b4180 2024-01-26T23:50:02.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.329+0000 7fd2f67fc700 1 -- 172.21.15.46:0/2052814366 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f8158ae0 con 0x7fd2f80b4180 2024-01-26T23:50:02.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.330+0000 7fd2f77fe700 1 -- 172.21.15.46:0/2052814366 >> v1:172.21.15.106:6789/0 conn(0x7fd2f80b4180 legacy=0x7fd2f8157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:02.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.330+0000 7fd2f77fe700 1 -- 172.21.15.46:0/2052814366 >> v1:172.21.15.46:6789/0 conn(0x7fd2f80ae520 legacy=0x7fd2f80b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:02.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.330+0000 7fd2f67fc700 1 -- 172.21.15.46:0/2052814366 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1641040495 0 0) 0x7fd2e0003570 con 0x7fd2f80ae520 2024-01-26T23:50:02.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.330+0000 7fd2f77fe700 1 -- 172.21.15.46:0/2052814366 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f8158520 con 0x7fd2f8157340 2024-01-26T23:50:02.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.330+0000 7fd2f77fe700 1 --2- 172.21.15.46:0/2052814366 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f8157340 0x7fd2f815bae0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009910 tx=0x7fd2ec0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6088c34be379c196 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:02.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.330+0000 7fd2f67fc700 1 -- 172.21.15.46:0/2052814366 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec011630 con 0x7fd2f8157340 2024-01-26T23:50:02.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.330+0000 7fd2f67fc700 1 -- 172.21.15.46:0/2052814366 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ec011790 con 0x7fd2f8157340 2024-01-26T23:50:02.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.330+0000 7fd2f67fc700 1 -- 172.21.15.46:0/2052814366 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec011b30 con 0x7fd2f8157340 2024-01-26T23:50:02.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.331+0000 7fd2fdf75700 1 -- 172.21.15.46:0/2052814366 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f8157340 msgr2=0x7fd2f815bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:02.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.331+0000 7fd2fdf75700 1 --2- 172.21.15.46:0/2052814366 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f8157340 0x7fd2f815bae0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009910 tx=0x7fd2ec0091a0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.331+0000 7fd2fdf75700 1 -- 172.21.15.46:0/2052814366 shutdown_connections 2024-01-26T23:50:02.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.331+0000 7fd2fdf75700 1 --2- 172.21.15.46:0/2052814366 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f8157340 0x7fd2f815bae0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.331+0000 7fd2fdf75700 1 -- 172.21.15.46:0/2052814366 >> 172.21.15.46:0/2052814366 conn(0x7fd2f80ce530 msgr2=0x7fd2f80ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:02.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.331+0000 7fd2fdf75700 1 -- 172.21.15.46:0/2052814366 shutdown_connections 2024-01-26T23:50:02.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.331+0000 7fd2fdf75700 1 -- 172.21.15.46:0/2052814366 wait complete. 2024-01-26T23:50:02.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.332+0000 7fd2fdf75700 1 Processor -- start 2024-01-26T23:50:02.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.332+0000 7fd2fdf75700 1 -- start start 2024-01-26T23:50:02.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.333+0000 7fd2fdf75700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f80ae520 0x7fd2f80c6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:02.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.333+0000 7fd2fdf75700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f80b4180 0x7fd2f81545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:02.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.333+0000 7fd2fdf75700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f8157340 0x7fd2f80c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:02.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.333+0000 7fd2fdf75700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80ca6f0 con 0x7fd2f8157340 2024-01-26T23:50:02.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.333+0000 7fd2fdf75700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80ca260 con 0x7fd2f80ae520 2024-01-26T23:50:02.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.333+0000 7fd2fdf75700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80c9650 con 0x7fd2f80b4180 2024-01-26T23:50:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.333+0000 7fd2f7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f8157340 0x7fd2f80c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.333+0000 7fd2f7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f8157340 0x7fd2f80c8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34516/0 (socket says 172.21.15.46:34516) 2024-01-26T23:50:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.333+0000 7fd2f7fff700 1 -- 172.21.15.46:0/1968839260 learned_addr learned my addr 172.21.15.46:0/1968839260 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.333+0000 7fd2f77fe700 1 --2- 172.21.15.46:0/1968839260 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f80ae520 0x7fd2f80c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.333+0000 7fd2f6ffd700 1 --2- 172.21.15.46:0/1968839260 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f80b4180 0x7fd2f81545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.333+0000 7fd2f7fff700 1 -- 172.21.15.46:0/1968839260 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f80b4180 msgr2=0x7fd2f81545d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.333+0000 7fd2f7fff700 1 --2- 172.21.15.46:0/1968839260 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f80b4180 0x7fd2f81545d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.333+0000 7fd2f7fff700 1 -- 172.21.15.46:0/1968839260 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f80ae520 msgr2=0x7fd2f80c6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.333+0000 7fd2f7fff700 1 --2- 172.21.15.46:0/1968839260 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f80ae520 0x7fd2f80c6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.333+0000 7fd2f7fff700 1 -- 172.21.15.46:0/1968839260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec009420 con 0x7fd2f8157340 2024-01-26T23:50:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.334+0000 7fd2f7fff700 1 --2- 172.21.15.46:0/1968839260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f8157340 0x7fd2f80c8ce0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fd2f000c910 tx=0x7fd2f00026f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.334+0000 7fd2df7fe700 1 -- 172.21.15.46:0/1968839260 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0012630 con 0x7fd2f8157340 2024-01-26T23:50:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.334+0000 7fd2df7fe700 1 -- 172.21.15.46:0/1968839260 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f0012790 con 0x7fd2f8157340 2024-01-26T23:50:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.334+0000 7fd2fdf75700 1 -- 172.21.15.46:0/1968839260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f80c9930 con 0x7fd2f8157340 2024-01-26T23:50:02.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.334+0000 7fd2df7fe700 1 -- 172.21.15.46:0/1968839260 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0012b50 con 0x7fd2f8157340 2024-01-26T23:50:02.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.334+0000 7fd2fdf75700 1 -- 172.21.15.46:0/1968839260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f80c9ef0 con 0x7fd2f8157340 2024-01-26T23:50:02.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.335+0000 7fd2fdf75700 1 -- 172.21.15.46:0/1968839260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f804c710 con 0x7fd2f8157340 2024-01-26T23:50:02.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.335+0000 7fd2df7fe700 1 -- 172.21.15.46:0/1968839260 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd2f001a3b0 con 0x7fd2f8157340 2024-01-26T23:50:02.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.336+0000 7fd2df7fe700 1 --2- 172.21.15.46:0/1968839260 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd2e4066f30 0x7fd2e40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:02.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.336+0000 7fd2df7fe700 1 -- 172.21.15.46:0/1968839260 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5565+0+0 (secure 0 0 0) 0x7fd2f0084570 con 0x7fd2f8157340 2024-01-26T23:50:02.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.336+0000 7fd2f77fe700 1 --2- 172.21.15.46:0/1968839260 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd2e4066f30 0x7fd2e40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:02.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.337+0000 7fd2f77fe700 1 --2- 172.21.15.46:0/1968839260 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd2e4066f30 0x7fd2e40693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009860 tx=0x7fd2ec017000 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:02.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.340+0000 7fd2df7fe700 1 -- 172.21.15.46:0/1968839260 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd2f0051870 con 0x7fd2f8157340 2024-01-26T23:50:02.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.547+0000 7fd2fdf75700 1 -- 172.21.15.46:0/1968839260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2f80ca3a0 con 0x7fd2f8157340 2024-01-26T23:50:02.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.547+0000 7fd2df7fe700 1 -- 172.21.15.46:0/1968839260 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+12943 (secure 0 0 0) 0x7fd2f0054e90 con 0x7fd2f8157340 2024-01-26T23:50:02.549 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:50:02.549 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":154,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:50:01.725930+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:50:02.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.549+0000 7fd2fdf75700 1 -- 172.21.15.46:0/1968839260 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd2e4066f30 msgr2=0x7fd2e40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:02.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.549+0000 7fd2fdf75700 1 --2- 172.21.15.46:0/1968839260 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd2e4066f30 0x7fd2e40693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009860 tx=0x7fd2ec017000 comp rx=0 tx=0).stop 2024-01-26T23:50:02.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.549+0000 7fd2fdf75700 1 -- 172.21.15.46:0/1968839260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f8157340 msgr2=0x7fd2f80c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:02.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.549+0000 7fd2fdf75700 1 --2- 172.21.15.46:0/1968839260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f8157340 0x7fd2f80c8ce0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fd2f000c910 tx=0x7fd2f00026f0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.550+0000 7fd2fdf75700 1 -- 172.21.15.46:0/1968839260 shutdown_connections 2024-01-26T23:50:02.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.550+0000 7fd2fdf75700 1 --2- 172.21.15.46:0/1968839260 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd2e4066f30 0x7fd2e40693e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.550+0000 7fd2fdf75700 1 --2- 172.21.15.46:0/1968839260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f8157340 0x7fd2f80c8ce0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.550+0000 7fd2fdf75700 1 --2- 172.21.15.46:0/1968839260 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f80ae520 0x7fd2f80c6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.550+0000 7fd2fdf75700 1 --2- 172.21.15.46:0/1968839260 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f80b4180 0x7fd2f81545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.550+0000 7fd2fdf75700 1 -- 172.21.15.46:0/1968839260 >> 172.21.15.46:0/1968839260 conn(0x7fd2f80ce530 msgr2=0x7fd2f80cc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:02.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.550+0000 7fd2fdf75700 1 -- 172.21.15.46:0/1968839260 shutdown_connections 2024-01-26T23:50:02.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.550+0000 7fd2fdf75700 1 -- 172.21.15.46:0/1968839260 wait complete. 2024-01-26T23:50:02.565 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-26T23:50:02.565 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-26T23:50:02.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.698+0000 7fe1989d5700 1 Processor -- start 2024-01-26T23:50:02.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.698+0000 7fe1989d5700 1 -- start start 2024-01-26T23:50:02.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.699+0000 7fe1989d5700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe1940c8da0 0x7fe1940c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.699+0000 7fe1989d5700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1940ccc00 con 0x7fe194140af0 2024-01-26T23:50:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.700+0000 7fe1989d5700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1940cc310 con 0x7fe1940caf30 2024-01-26T23:50:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.700+0000 7fe1989d5700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1940cbbf0 con 0x7fe1940c8da0 2024-01-26T23:50:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.700+0000 7fe19259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe1940c8da0 0x7fe1940c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.700+0000 7fe191d9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe194140af0 0x7fe194140f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48930/0 (socket says 172.21.15.46:48930) 2024-01-26T23:50:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.700+0000 7fe191d9b700 1 -- 172.21.15.46:0/101945712 learned_addr learned my addr 172.21.15.46:0/101945712 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.700+0000 7fe19259c700 1 -- 172.21.15.46:0/101945712 >> v1:172.21.15.106:6789/0 conn(0x7fe1940caf30 legacy=0x7fe1940cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.700+0000 7fe19259c700 1 -- 172.21.15.46:0/101945712 >> v1:172.21.15.46:6789/0 conn(0x7fe194140af0 legacy=0x7fe194140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.700+0000 7fe19259c700 1 -- 172.21.15.46:0/101945712 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1940cc5b0 con 0x7fe1940c8da0 2024-01-26T23:50:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.700+0000 7fe19259c700 1 --2- 172.21.15.46:0/101945712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1940c8da0 0x7fe1940c9170 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0098e0 tx=0x7fe18c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246bfaaec87e6a99 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.700+0000 7fe19159a700 1 -- 172.21.15.46:0/101945712 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c009e80 con 0x7fe1940c8da0 2024-01-26T23:50:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.700+0000 7fe19159a700 1 -- 172.21.15.46:0/101945712 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18c00b820 con 0x7fe1940c8da0 2024-01-26T23:50:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.700+0000 7fe19159a700 1 -- 172.21.15.46:0/101945712 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c00bba0 con 0x7fe1940c8da0 2024-01-26T23:50:02.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.701+0000 7fe1989d5700 1 -- 172.21.15.46:0/101945712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1940c8da0 msgr2=0x7fe1940c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:02.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.701+0000 7fe1989d5700 1 --2- 172.21.15.46:0/101945712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1940c8da0 0x7fe1940c9170 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0098e0 tx=0x7fe18c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.701+0000 7fe1989d5700 1 -- 172.21.15.46:0/101945712 shutdown_connections 2024-01-26T23:50:02.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.701+0000 7fe1989d5700 1 --2- 172.21.15.46:0/101945712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1940c8da0 0x7fe1940c9170 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.701+0000 7fe1989d5700 1 -- 172.21.15.46:0/101945712 >> 172.21.15.46:0/101945712 conn(0x7fe19406bd90 msgr2=0x7fe19406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:02.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.701+0000 7fe1989d5700 1 -- 172.21.15.46:0/101945712 shutdown_connections 2024-01-26T23:50:02.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.701+0000 7fe1989d5700 1 -- 172.21.15.46:0/101945712 wait complete. 2024-01-26T23:50:02.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.702+0000 7fe1989d5700 1 Processor -- start 2024-01-26T23:50:02.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.702+0000 7fe1989d5700 1 -- start start 2024-01-26T23:50:02.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.703+0000 7fe1989d5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe1940c8da0 0x7fe194159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:02.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.703+0000 7fe1989d5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1940caf30 0x7fe194158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:02.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.703+0000 7fe1989d5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe194140af0 0x7fe194156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:02.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.703+0000 7fe1989d5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe194159260 con 0x7fe1940caf30 2024-01-26T23:50:02.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.703+0000 7fe1989d5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe194159cc0 con 0x7fe194140af0 2024-01-26T23:50:02.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.703+0000 7fe1989d5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe194159fa0 con 0x7fe1940c8da0 2024-01-26T23:50:02.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.703+0000 7fe191d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1940caf30 0x7fe194158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:02.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.703+0000 7fe191d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1940caf30 0x7fe194158d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34526/0 (socket says 172.21.15.46:34526) 2024-01-26T23:50:02.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.703+0000 7fe191d9b700 1 -- 172.21.15.46:0/3617265368 learned_addr learned my addr 172.21.15.46:0/3617265368 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:02.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.703+0000 7fe192d9d700 1 --2- 172.21.15.46:0/3617265368 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe194140af0 0x7fe194156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:02.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.703+0000 7fe19259c700 1 --2- 172.21.15.46:0/3617265368 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe1940c8da0 0x7fe194159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:02.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.704+0000 7fe191d9b700 1 -- 172.21.15.46:0/3617265368 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe1940c8da0 msgr2=0x7fe194159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:02.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.704+0000 7fe191d9b700 1 --2- 172.21.15.46:0/3617265368 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe1940c8da0 0x7fe194159630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.704+0000 7fe191d9b700 1 -- 172.21.15.46:0/3617265368 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe194140af0 msgr2=0x7fe194156fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:02.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.704+0000 7fe191d9b700 1 --2- 172.21.15.46:0/3617265368 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe194140af0 0x7fe194156fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.704+0000 7fe191d9b700 1 -- 172.21.15.46:0/3617265368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18c009420 con 0x7fe1940caf30 2024-01-26T23:50:02.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.704+0000 7fe191d9b700 1 --2- 172.21.15.46:0/3617265368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1940caf30 0x7fe194158d20 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00c590 tx=0x7fe17c007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:02.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.704+0000 7fe18a7fc700 1 -- 172.21.15.46:0/3617265368 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c005b30 con 0x7fe1940caf30 2024-01-26T23:50:02.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.704+0000 7fe18a7fc700 1 -- 172.21.15.46:0/3617265368 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17c005cb0 con 0x7fe1940caf30 2024-01-26T23:50:02.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.704+0000 7fe1989d5700 1 -- 172.21.15.46:0/3617265368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19415a280 con 0x7fe1940caf30 2024-01-26T23:50:02.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.704+0000 7fe18a7fc700 1 -- 172.21.15.46:0/3617265368 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c007320 con 0x7fe1940caf30 2024-01-26T23:50:02.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.704+0000 7fe1989d5700 1 -- 172.21.15.46:0/3617265368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19415a840 con 0x7fe1940caf30 2024-01-26T23:50:02.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.705+0000 7fe1989d5700 1 -- 172.21.15.46:0/3617265368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19404c710 con 0x7fe1940caf30 2024-01-26T23:50:02.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.706+0000 7fe18a7fc700 1 -- 172.21.15.46:0/3617265368 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe17c01b3b0 con 0x7fe1940caf30 2024-01-26T23:50:02.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.706+0000 7fe18a7fc700 1 --2- 172.21.15.46:0/3617265368 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe174066f30 0x7fe1740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:02.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.706+0000 7fe18a7fc700 1 -- 172.21.15.46:0/3617265368 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5565+0+0 (secure 0 0 0) 0x7fe17c0837b0 con 0x7fe1940caf30 2024-01-26T23:50:02.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.706+0000 7fe19259c700 1 --2- 172.21.15.46:0/3617265368 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe174066f30 0x7fe1740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:02.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.707+0000 7fe19259c700 1 --2- 172.21.15.46:0/3617265368 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe174066f30 0x7fe1740693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0030c0 tx=0x7fe18c016000 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:02.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.710+0000 7fe18a7fc700 1 -- 172.21.15.46:0/3617265368 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe17c050ab0 con 0x7fe1940caf30 2024-01-26T23:50:02.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.914+0000 7fe1989d5700 1 -- 172.21.15.46:0/3617265368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe1940cc890 con 0x7fe1940caf30 2024-01-26T23:50:02.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.915+0000 7fe18a7fc700 1 -- 172.21.15.46:0/3617265368 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v154) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe17c0170a0 con 0x7fe1940caf30 2024-01-26T23:50:02.916 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-26T23:50:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.917+0000 7fe1989d5700 1 -- 172.21.15.46:0/3617265368 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe174066f30 msgr2=0x7fe1740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.917+0000 7fe1989d5700 1 --2- 172.21.15.46:0/3617265368 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe174066f30 0x7fe1740693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0030c0 tx=0x7fe18c016000 comp rx=0 tx=0).stop 2024-01-26T23:50:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.917+0000 7fe1989d5700 1 -- 172.21.15.46:0/3617265368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1940caf30 msgr2=0x7fe194158d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.917+0000 7fe1989d5700 1 --2- 172.21.15.46:0/3617265368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1940caf30 0x7fe194158d20 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00c590 tx=0x7fe17c007f90 comp rx=0 tx=0).stop 2024-01-26T23:50:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.917+0000 7fe1989d5700 1 -- 172.21.15.46:0/3617265368 shutdown_connections 2024-01-26T23:50:02.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.917+0000 7fe1989d5700 1 --2- 172.21.15.46:0/3617265368 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe174066f30 0x7fe1740693e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.917+0000 7fe1989d5700 1 --2- 172.21.15.46:0/3617265368 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe194140af0 0x7fe194156fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.917+0000 7fe1989d5700 1 --2- 172.21.15.46:0/3617265368 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe1940c8da0 0x7fe194159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.917+0000 7fe1989d5700 1 --2- 172.21.15.46:0/3617265368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1940caf30 0x7fe194158d20 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:02.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.917+0000 7fe1989d5700 1 -- 172.21.15.46:0/3617265368 >> 172.21.15.46:0/3617265368 conn(0x7fe19406bd90 msgr2=0x7fe19413dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:02.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.917+0000 7fe1989d5700 1 -- 172.21.15.46:0/3617265368 shutdown_connections 2024-01-26T23:50:02.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:02.917+0000 7fe1989d5700 1 -- 172.21.15.46:0/3617265368 wait complete. 2024-01-26T23:50:02.932 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-26T23:50:03.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.062+0000 7fba22bbc700 1 Processor -- start 2024-01-26T23:50:03.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.062+0000 7fba22bbc700 1 -- start start 2024-01-26T23:50:03.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.063+0000 7fba22bbc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c159ce0 0x7fba1c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.063+0000 7fba22bbc700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba1c15a120 con 0x7fba1c159910 2024-01-26T23:50:03.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.063+0000 7fba22bbc700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba1c0c7e90 con 0x7fba1c068040 2024-01-26T23:50:03.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.063+0000 7fba22bbc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c069340 con 0x7fba1c159ce0 2024-01-26T23:50:03.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.063+0000 7fba20958700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fba1c159910 0x7fba1c067930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48944/0 (socket says 172.21.15.46:48944) 2024-01-26T23:50:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.063+0000 7fba20958700 1 -- 172.21.15.46:0/2525766575 learned_addr learned my addr 172.21.15.46:0/2525766575 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.063+0000 7fba21159700 1 --2- 172.21.15.46:0/2525766575 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c159ce0 0x7fba1c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.063+0000 7fba1b7fe700 1 -- 172.21.15.46:0/2525766575 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652661202 0 0) 0x7fba1c15a120 con 0x7fba1c159910 2024-01-26T23:50:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.063+0000 7fba1b7fe700 1 -- 172.21.15.46:0/2525766575 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba0c003570 con 0x7fba1c159910 2024-01-26T23:50:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1b7fe700 1 -- 172.21.15.46:0/2525766575 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005728347 0 0) 0x7fba1c0c7e90 con 0x7fba1c068040 2024-01-26T23:50:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1b7fe700 1 -- 172.21.15.46:0/2525766575 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba1c15a120 con 0x7fba1c068040 2024-01-26T23:50:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1b7fe700 1 -- 172.21.15.46:0/2525766575 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 680257489 0 0) 0x7fba0c003570 con 0x7fba1c159910 2024-01-26T23:50:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1b7fe700 1 -- 172.21.15.46:0/2525766575 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c159ce0 msgr2=0x7fba1c068bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1b7fe700 1 --2- 172.21.15.46:0/2525766575 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c159ce0 0x7fba1c068bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1b7fe700 1 -- 172.21.15.46:0/2525766575 >> v1:172.21.15.106:6789/0 conn(0x7fba1c068040 legacy=0x7fba1c0c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1b7fe700 1 -- 172.21.15.46:0/2525766575 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c06b690 con 0x7fba1c159910 2024-01-26T23:50:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1b7fe700 1 -- 172.21.15.46:0/2525766575 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fba10001470 con 0x7fba1c159910 2024-01-26T23:50:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1b7fe700 1 -- 172.21.15.46:0/2525766575 >> v1:172.21.15.46:6789/0 conn(0x7fba1c159910 legacy=0x7fba1c067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1b7fe700 1 --2- 172.21.15.46:0/2525766575 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba0c003c20 0x7fba0c004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1b7fe700 1 --2- 172.21.15.46:0/2525766575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba0c004810 0x7fba0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1b7fe700 1 --2- 172.21.15.46:0/2525766575 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c159ce0 0x7fba0c005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1b7fe700 1 -- 172.21.15.46:0/2525766575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c069340 con 0x7fba0c004810 2024-01-26T23:50:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1b7fe700 1 -- 172.21.15.46:0/2525766575 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c00bbf0 con 0x7fba0c003c20 2024-01-26T23:50:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1b7fe700 1 -- 172.21.15.46:0/2525766575 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c00bdb0 con 0x7fba1c159ce0 2024-01-26T23:50:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1bfff700 1 --2- 172.21.15.46:0/2525766575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba0c004810 0x7fba0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba20958700 1 --2- 172.21.15.46:0/2525766575 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba0c003c20 0x7fba0c004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba21159700 1 --2- 172.21.15.46:0/2525766575 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c159ce0 0x7fba0c005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1bfff700 1 -- 172.21.15.46:0/2525766575 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c159ce0 msgr2=0x7fba0c005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1bfff700 1 --2- 172.21.15.46:0/2525766575 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c159ce0 0x7fba0c005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1bfff700 1 -- 172.21.15.46:0/2525766575 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba0c003c20 msgr2=0x7fba0c004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1bfff700 1 --2- 172.21.15.46:0/2525766575 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba0c003c20 0x7fba0c004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba1bfff700 1 -- 172.21.15.46:0/2525766575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0c003570 con 0x7fba0c004810 2024-01-26T23:50:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.064+0000 7fba20958700 1 --2- 172.21.15.46:0/2525766575 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba0c003c20 0x7fba0c004010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:50:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.065+0000 7fba1bfff700 1 --2- 172.21.15.46:0/2525766575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba0c004810 0x7fba0c004c80 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fba08001a20 tx=0x7fba08007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf5cb6f0c26d0625 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.065+0000 7fba1b7fe700 1 -- 172.21.15.46:0/2525766575 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0800c5d0 con 0x7fba0c004810 2024-01-26T23:50:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.065+0000 7fba1b7fe700 1 -- 172.21.15.46:0/2525766575 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0800c730 con 0x7fba0c004810 2024-01-26T23:50:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.065+0000 7fba1b7fe700 1 -- 172.21.15.46:0/2525766575 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0800cad0 con 0x7fba0c004810 2024-01-26T23:50:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.065+0000 7fba22bbc700 1 -- 172.21.15.46:0/2525766575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba0c004810 msgr2=0x7fba0c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.065+0000 7fba22bbc700 1 --2- 172.21.15.46:0/2525766575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba0c004810 0x7fba0c004c80 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fba08001a20 tx=0x7fba08007d30 comp rx=0 tx=0).stop 2024-01-26T23:50:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.065+0000 7fba21159700 1 -- 172.21.15.46:0/2525766575 reap_dead start 2024-01-26T23:50:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.065+0000 7fba22bbc700 1 -- 172.21.15.46:0/2525766575 shutdown_connections 2024-01-26T23:50:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.065+0000 7fba22bbc700 1 -- 172.21.15.46:0/2525766575 >> 172.21.15.46:0/2525766575 conn(0x7fba1c0cd6e0 msgr2=0x7fba1c0c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.065+0000 7fba22bbc700 1 -- 172.21.15.46:0/2525766575 shutdown_connections 2024-01-26T23:50:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.065+0000 7fba22bbc700 1 -- 172.21.15.46:0/2525766575 wait complete. 2024-01-26T23:50:03.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.067+0000 7fba22bbc700 1 Processor -- start 2024-01-26T23:50:03.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.067+0000 7fba22bbc700 1 -- start start 2024-01-26T23:50:03.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.068+0000 7fba22bbc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba0c004810 0x7fba1c0b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.068+0000 7fba22bbc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba1c152310 0x7fba1c15eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.068+0000 7fba22bbc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c15ae60 0x7fba1c154970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.068+0000 7fba22bbc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c15b990 con 0x7fba1c15ae60 2024-01-26T23:50:03.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.068+0000 7fba22bbc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c0b0a30 con 0x7fba1c152310 2024-01-26T23:50:03.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.068+0000 7fba22bbc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c15b3d0 con 0x7fba0c004810 2024-01-26T23:50:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.068+0000 7fba21159700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c15ae60 0x7fba1c154970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.068+0000 7fba21159700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c15ae60 0x7fba1c154970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34534/0 (socket says 172.21.15.46:34534) 2024-01-26T23:50:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.068+0000 7fba21159700 1 -- 172.21.15.46:0/2118429739 learned_addr learned my addr 172.21.15.46:0/2118429739 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.068+0000 7fba20958700 1 --2- 172.21.15.46:0/2118429739 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba0c004810 0x7fba1c0b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.068+0000 7fba1bfff700 1 --2- 172.21.15.46:0/2118429739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba1c152310 0x7fba1c15eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.069+0000 7fba21159700 1 -- 172.21.15.46:0/2118429739 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba0c004810 msgr2=0x7fba1c0b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.069+0000 7fba21159700 1 --2- 172.21.15.46:0/2118429739 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba0c004810 0x7fba1c0b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.069+0000 7fba21159700 1 -- 172.21.15.46:0/2118429739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba1c152310 msgr2=0x7fba1c15eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.069+0000 7fba21159700 1 --2- 172.21.15.46:0/2118429739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba1c152310 0x7fba1c15eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.069+0000 7fba21159700 1 -- 172.21.15.46:0/2118429739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c15b6b0 con 0x7fba1c15ae60 2024-01-26T23:50:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.069+0000 7fba21159700 1 --2- 172.21.15.46:0/2118429739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c15ae60 0x7fba1c154970 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fba1c0ca750 tx=0x7fba1400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.069+0000 7fba18ff9700 1 -- 172.21.15.46:0/2118429739 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1400e040 con 0x7fba1c15ae60 2024-01-26T23:50:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.069+0000 7fba18ff9700 1 -- 172.21.15.46:0/2118429739 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba1400f040 con 0x7fba1c15ae60 2024-01-26T23:50:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.069+0000 7fba22bbc700 1 -- 172.21.15.46:0/2118429739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba1c0b1320 con 0x7fba1c15ae60 2024-01-26T23:50:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.069+0000 7fba18ff9700 1 -- 172.21.15.46:0/2118429739 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1400fd70 con 0x7fba1c15ae60 2024-01-26T23:50:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.069+0000 7fba22bbc700 1 -- 172.21.15.46:0/2118429739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1c15f0e0 con 0x7fba1c15ae60 2024-01-26T23:50:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.070+0000 7fba22bbc700 1 -- 172.21.15.46:0/2118429739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba1c04c710 con 0x7fba1c15ae60 2024-01-26T23:50:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.071+0000 7fba18ff9700 1 -- 172.21.15.46:0/2118429739 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fba14015640 con 0x7fba1c15ae60 2024-01-26T23:50:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.071+0000 7fba18ff9700 1 --2- 172.21.15.46:0/2118429739 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fba00066f30 0x7fba000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.071+0000 7fba18ff9700 1 -- 172.21.15.46:0/2118429739 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5565+0+0 (secure 0 0 0) 0x7fba14084c50 con 0x7fba1c15ae60 2024-01-26T23:50:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.071+0000 7fba20958700 1 --2- 172.21.15.46:0/2118429739 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fba00066f30 0x7fba000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.072+0000 7fba20958700 1 --2- 172.21.15.46:0/2118429739 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fba00066f30 0x7fba000693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fba1c155850 tx=0x7fba10001d40 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:03.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.076+0000 7fba18ff9700 1 -- 172.21.15.46:0/2118429739 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fba14051f50 con 0x7fba1c15ae60 2024-01-26T23:50:03.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.285+0000 7fba22bbc700 1 -- 172.21.15.46:0/2118429739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fba1c06eb30 con 0x7fba1c15ae60 2024-01-26T23:50:03.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.286+0000 7fba18ff9700 1 -- 172.21.15.46:0/2118429739 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v154) v1 ==== 84+0+65 (secure 0 0 0) 0x7fba14055570 con 0x7fba1c15ae60 2024-01-26T23:50:03.287 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:50:03.287 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-26T23:50:03.287 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:03.288 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-01-26T23:50:03.288 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-01-26T23:50:03.288 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:03.288 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-26T23:50:03.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.288+0000 7fba22bbc700 1 -- 172.21.15.46:0/2118429739 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fba00066f30 msgr2=0x7fba000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:03.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.288+0000 7fba22bbc700 1 --2- 172.21.15.46:0/2118429739 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fba00066f30 0x7fba000693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fba1c155850 tx=0x7fba10001d40 comp rx=0 tx=0).stop 2024-01-26T23:50:03.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.288+0000 7fba22bbc700 1 -- 172.21.15.46:0/2118429739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c15ae60 msgr2=0x7fba1c154970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:03.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.288+0000 7fba22bbc700 1 --2- 172.21.15.46:0/2118429739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c15ae60 0x7fba1c154970 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fba1c0ca750 tx=0x7fba1400b040 comp rx=0 tx=0).stop 2024-01-26T23:50:03.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.288+0000 7fba22bbc700 1 -- 172.21.15.46:0/2118429739 shutdown_connections 2024-01-26T23:50:03.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.288+0000 7fba22bbc700 1 --2- 172.21.15.46:0/2118429739 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fba00066f30 0x7fba000693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.288+0000 7fba22bbc700 1 --2- 172.21.15.46:0/2118429739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c15ae60 0x7fba1c154970 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.288+0000 7fba22bbc700 1 --2- 172.21.15.46:0/2118429739 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba0c004810 0x7fba1c0b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.288+0000 7fba22bbc700 1 --2- 172.21.15.46:0/2118429739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba1c152310 0x7fba1c15eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.288+0000 7fba22bbc700 1 -- 172.21.15.46:0/2118429739 >> 172.21.15.46:0/2118429739 conn(0x7fba1c0cd6e0 msgr2=0x7fba1c067930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:03.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.288+0000 7fba22bbc700 1 -- 172.21.15.46:0/2118429739 shutdown_connections 2024-01-26T23:50:03.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.288+0000 7fba22bbc700 1 -- 172.21.15.46:0/2118429739 wait complete. 2024-01-26T23:50:03.303 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-26T23:50:03.303 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-26T23:50:03.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.438+0000 7f4abce2f700 1 Processor -- start 2024-01-26T23:50:03.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.438+0000 7f4abce2f700 1 -- start start 2024-01-26T23:50:03.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.439+0000 7f4abce2f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab80c9980 0x7f4ab815ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.439+0000 7f4abce2f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4ab8156b00 0x7f4ab806eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.439+0000 7f4abce2f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab8157000 con 0x7f4ab80c9d50 2024-01-26T23:50:03.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.439+0000 7f4abce2f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab806f3e0 con 0x7f4ab8156b00 2024-01-26T23:50:03.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.439+0000 7f4abce2f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab806f6c0 con 0x7f4ab80c9980 2024-01-26T23:50:03.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.440+0000 7f4ab5d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4ab8156b00 0x7f4ab806eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.440+0000 7f4ab5d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4ab8156b00 0x7f4ab806eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34546/0 (socket says 172.21.15.46:34546) 2024-01-26T23:50:03.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.440+0000 7f4ab5d9b700 1 -- 172.21.15.46:0/842600729 learned_addr learned my addr 172.21.15.46:0/842600729 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:03.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.440+0000 7f4ab659c700 1 --2- 172.21.15.46:0/842600729 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab80c9980 0x7f4ab815ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.440+0000 7f4ab5d9b700 1 -- 172.21.15.46:0/842600729 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab80c9980 msgr2=0x7f4ab815ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:03.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.440+0000 7f4ab5d9b700 1 --2- 172.21.15.46:0/842600729 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab80c9980 0x7f4ab815ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.440+0000 7f4ab5d9b700 1 -- 172.21.15.46:0/842600729 >> v1:172.21.15.106:6789/0 conn(0x7f4ab80c9d50 legacy=0x7f4ab80b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:03.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.440+0000 7f4ab5d9b700 1 -- 172.21.15.46:0/842600729 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab806f9a0 con 0x7f4ab8156b00 2024-01-26T23:50:03.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.440+0000 7f4ab5d9b700 1 --2- 172.21.15.46:0/842600729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab8156b00 0x7f4ab806eb90 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00099b0 tx=0x7f4aa0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1490ae0dfae3d0cb server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:03.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.440+0000 7f4ab559a700 1 -- 172.21.15.46:0/842600729 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa000b860 con 0x7f4ab8156b00 2024-01-26T23:50:03.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.440+0000 7f4ab559a700 1 -- 172.21.15.46:0/842600729 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa000b9c0 con 0x7f4ab8156b00 2024-01-26T23:50:03.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.440+0000 7f4ab559a700 1 -- 172.21.15.46:0/842600729 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa000bd60 con 0x7f4ab8156b00 2024-01-26T23:50:03.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.441+0000 7f4abce2f700 1 -- 172.21.15.46:0/842600729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab8156b00 msgr2=0x7f4ab806eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:03.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.441+0000 7f4abce2f700 1 --2- 172.21.15.46:0/842600729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab8156b00 0x7f4ab806eb90 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00099b0 tx=0x7f4aa0009250 comp rx=0 tx=0).stop 2024-01-26T23:50:03.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.441+0000 7f4abce2f700 1 -- 172.21.15.46:0/842600729 shutdown_connections 2024-01-26T23:50:03.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.441+0000 7f4abce2f700 1 --2- 172.21.15.46:0/842600729 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab80c9980 0x7f4ab815ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.441+0000 7f4abce2f700 1 --2- 172.21.15.46:0/842600729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab8156b00 0x7f4ab806eb90 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.441+0000 7f4abce2f700 1 -- 172.21.15.46:0/842600729 >> 172.21.15.46:0/842600729 conn(0x7f4ab8069510 msgr2=0x7f4ab80aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:03.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.441+0000 7f4abce2f700 1 -- 172.21.15.46:0/842600729 shutdown_connections 2024-01-26T23:50:03.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.441+0000 7f4abce2f700 1 -- 172.21.15.46:0/842600729 wait complete. 2024-01-26T23:50:03.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.442+0000 7f4abce2f700 1 Processor -- start 2024-01-26T23:50:03.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.442+0000 7f4abce2f700 1 -- start start 2024-01-26T23:50:03.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.443+0000 7f4abce2f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab80c9980 0x7f4ab80b05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.443+0000 7f4abce2f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab80c9d50 0x7f4ab816b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.443+0000 7f4abce2f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ab8156b00 0x7f4ab816b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.443+0000 7f4abce2f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab814c8a0 con 0x7f4ab80c9d50 2024-01-26T23:50:03.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.443+0000 7f4abce2f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab816bf40 con 0x7f4ab8156b00 2024-01-26T23:50:03.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.443+0000 7f4abce2f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab816c220 con 0x7f4ab80c9980 2024-01-26T23:50:03.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.443+0000 7f4ab5d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab80c9d50 0x7f4ab816b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.443+0000 7f4ab5d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab80c9d50 0x7f4ab816b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34548/0 (socket says 172.21.15.46:34548) 2024-01-26T23:50:03.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.443+0000 7f4ab5d9b700 1 -- 172.21.15.46:0/3520372736 learned_addr learned my addr 172.21.15.46:0/3520372736 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:03.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.443+0000 7f4ab659c700 1 --2- 172.21.15.46:0/3520372736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab80c9980 0x7f4ab80b05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.443+0000 7f4ab6d9d700 1 --2- 172.21.15.46:0/3520372736 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ab8156b00 0x7f4ab816b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.444+0000 7f4ab5d9b700 1 -- 172.21.15.46:0/3520372736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab80c9980 msgr2=0x7f4ab80b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:03.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.444+0000 7f4ab5d9b700 1 --2- 172.21.15.46:0/3520372736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab80c9980 0x7f4ab80b05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.444+0000 7f4ab5d9b700 1 -- 172.21.15.46:0/3520372736 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ab8156b00 msgr2=0x7f4ab816b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:03.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.444+0000 7f4ab5d9b700 1 --2- 172.21.15.46:0/3520372736 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ab8156b00 0x7f4ab816b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.444+0000 7f4ab5d9b700 1 -- 172.21.15.46:0/3520372736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa00094f0 con 0x7f4ab80c9d50 2024-01-26T23:50:03.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.444+0000 7f4ab5d9b700 1 --2- 172.21.15.46:0/3520372736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab80c9d50 0x7f4ab816b3d0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f4ab806d630 tx=0x7f4aa0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:03.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.444+0000 7f4aae7fc700 1 -- 172.21.15.46:0/3520372736 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0011e40 con 0x7f4ab80c9d50 2024-01-26T23:50:03.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.444+0000 7f4aae7fc700 1 -- 172.21.15.46:0/3520372736 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa0024440 con 0x7f4ab80c9d50 2024-01-26T23:50:03.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.444+0000 7f4abce2f700 1 -- 172.21.15.46:0/3520372736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab814c250 con 0x7f4ab80c9d50 2024-01-26T23:50:03.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.444+0000 7f4aae7fc700 1 -- 172.21.15.46:0/3520372736 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0024800 con 0x7f4ab80c9d50 2024-01-26T23:50:03.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.444+0000 7f4abce2f700 1 -- 172.21.15.46:0/3520372736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab8150540 con 0x7f4ab80c9d50 2024-01-26T23:50:03.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.445+0000 7f4abce2f700 1 -- 172.21.15.46:0/3520372736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab814ca10 con 0x7f4ab80c9d50 2024-01-26T23:50:03.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.446+0000 7f4aae7fc700 1 -- 172.21.15.46:0/3520372736 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4aa001a420 con 0x7f4ab80c9d50 2024-01-26T23:50:03.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.446+0000 7f4aae7fc700 1 --2- 172.21.15.46:0/3520372736 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4a98066f30 0x7f4a980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.446+0000 7f4aae7fc700 1 -- 172.21.15.46:0/3520372736 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5565+0+0 (secure 0 0 0) 0x7f4aa0086310 con 0x7f4ab80c9d50 2024-01-26T23:50:03.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.446+0000 7f4ab659c700 1 --2- 172.21.15.46:0/3520372736 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4a98066f30 0x7f4a980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.447+0000 7f4ab659c700 1 --2- 172.21.15.46:0/3520372736 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4a98066f30 0x7f4a980693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8006fd0 tx=0x7f4aa8008040 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:03.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.451+0000 7f4aae7fc700 1 -- 172.21.15.46:0/3520372736 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4aa0053610 con 0x7f4ab80c9d50 2024-01-26T23:50:03.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.656+0000 7f4abce2f700 1 -- 172.21.15.46:0/3520372736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4ab804c710 con 0x7f4ab80c9d50 2024-01-26T23:50:03.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.729+0000 7f4aae7fc700 1 -- 172.21.15.46:0/3520372736 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v155) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4aa0056c30 con 0x7f4ab80c9d50 2024-01-26T23:50:03.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.733+0000 7f4abce2f700 1 -- 172.21.15.46:0/3520372736 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4a98066f30 msgr2=0x7f4a980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.733+0000 7f4abce2f700 1 --2- 172.21.15.46:0/3520372736 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4a98066f30 0x7f4a980693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8006fd0 tx=0x7f4aa8008040 comp rx=0 tx=0).stop 2024-01-26T23:50:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.733+0000 7f4abce2f700 1 -- 172.21.15.46:0/3520372736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab80c9d50 msgr2=0x7f4ab816b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.733+0000 7f4abce2f700 1 --2- 172.21.15.46:0/3520372736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab80c9d50 0x7f4ab816b3d0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f4ab806d630 tx=0x7f4aa0017000 comp rx=0 tx=0).stop 2024-01-26T23:50:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.733+0000 7f4abce2f700 1 -- 172.21.15.46:0/3520372736 shutdown_connections 2024-01-26T23:50:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.733+0000 7f4abce2f700 1 --2- 172.21.15.46:0/3520372736 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4a98066f30 0x7f4a980693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.733+0000 7f4abce2f700 1 --2- 172.21.15.46:0/3520372736 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ab8156b00 0x7f4ab816b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.733+0000 7f4abce2f700 1 --2- 172.21.15.46:0/3520372736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4ab80c9980 0x7f4ab80b05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.733+0000 7f4abce2f700 1 --2- 172.21.15.46:0/3520372736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab80c9d50 0x7f4ab816b3d0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.733+0000 7f4abce2f700 1 -- 172.21.15.46:0/3520372736 >> 172.21.15.46:0/3520372736 conn(0x7f4ab8069510 msgr2=0x7f4ab80acdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.733+0000 7f4abce2f700 1 -- 172.21.15.46:0/3520372736 shutdown_connections 2024-01-26T23:50:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.733+0000 7f4abce2f700 1 -- 172.21.15.46:0/3520372736 wait complete. 2024-01-26T23:50:03.738 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-01-26T23:50:03.750 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-26T23:50:03.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.888+0000 7fb81c34a700 1 Processor -- start 2024-01-26T23:50:03.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.888+0000 7fb81c34a700 1 -- start start 2024-01-26T23:50:03.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.889+0000 7fb81c34a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb81406b4e0 0x7fb81406a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.889+0000 7fb81c34a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8140a9a80 con 0x7fb814068ed0 2024-01-26T23:50:03.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.889+0000 7fb81c34a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81415a680 con 0x7fb814068b00 2024-01-26T23:50:03.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.889+0000 7fb81c34a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb814069320 con 0x7fb81406b4e0 2024-01-26T23:50:03.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.889+0000 7fb8198e5700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb814068ed0 0x7fb81406add0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48960/0 (socket says 172.21.15.46:48960) 2024-01-26T23:50:03.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.889+0000 7fb8198e5700 1 -- 172.21.15.46:0/1571739469 learned_addr learned my addr 172.21.15.46:0/1571739469 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:03.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.890+0000 7fb81a8e7700 1 --2- 172.21.15.46:0/1571739469 >> v2:172.21.15.106:3300/0 conn(0x7fb81406b4e0 0x7fb81406a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.890+0000 7fb8190e4700 1 -- 172.21.15.46:0/1571739469 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682486373 0 0) 0x7fb8140a9a80 con 0x7fb814068ed0 2024-01-26T23:50:03.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.890+0000 7fb8190e4700 1 -- 172.21.15.46:0/1571739469 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb800003570 con 0x7fb814068ed0 2024-01-26T23:50:03.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.890+0000 7fb81a8e7700 1 -- 172.21.15.46:0/1571739469 >> v1:172.21.15.106:6789/0 conn(0x7fb814068b00 legacy=0x7fb814159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:03.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.890+0000 7fb81a8e7700 1 -- 172.21.15.46:0/1571739469 >> v1:172.21.15.46:6789/0 conn(0x7fb814068ed0 legacy=0x7fb81406add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:03.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.890+0000 7fb8190e4700 1 -- 172.21.15.46:0/1571739469 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12145505 0 0) 0x7fb81415a680 con 0x7fb814068b00 2024-01-26T23:50:03.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.890+0000 7fb81a8e7700 1 -- 172.21.15.46:0/1571739469 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81415a960 con 0x7fb81406b4e0 2024-01-26T23:50:03.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.890+0000 7fb81a8e7700 1 --2- 172.21.15.46:0/1571739469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb81406b4e0 0x7fb81406a660 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fb80c009e60 tx=0x7fb80c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cfb78060f1c64c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:03.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.890+0000 7fb8190e4700 1 -- 172.21.15.46:0/1571739469 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c00e070 con 0x7fb81406b4e0 2024-01-26T23:50:03.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.890+0000 7fb8190e4700 1 -- 172.21.15.46:0/1571739469 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80c00bb60 con 0x7fb81406b4e0 2024-01-26T23:50:03.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.891+0000 7fb8190e4700 1 -- 172.21.15.46:0/1571739469 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c012430 con 0x7fb81406b4e0 2024-01-26T23:50:03.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.891+0000 7fb81c34a700 1 -- 172.21.15.46:0/1571739469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb81406b4e0 msgr2=0x7fb81406a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:03.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.891+0000 7fb81c34a700 1 --2- 172.21.15.46:0/1571739469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb81406b4e0 0x7fb81406a660 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fb80c009e60 tx=0x7fb80c009780 comp rx=0 tx=0).stop 2024-01-26T23:50:03.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.891+0000 7fb81c34a700 1 -- 172.21.15.46:0/1571739469 shutdown_connections 2024-01-26T23:50:03.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.891+0000 7fb81c34a700 1 --2- 172.21.15.46:0/1571739469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb81406b4e0 0x7fb81406a660 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.891+0000 7fb81c34a700 1 -- 172.21.15.46:0/1571739469 >> 172.21.15.46:0/1571739469 conn(0x7fb8140ca100 msgr2=0x7fb81406e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:03.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.891+0000 7fb81c34a700 1 -- 172.21.15.46:0/1571739469 shutdown_connections 2024-01-26T23:50:03.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.891+0000 7fb81c34a700 1 -- 172.21.15.46:0/1571739469 wait complete. 2024-01-26T23:50:03.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.892+0000 7fb81c34a700 1 Processor -- start 2024-01-26T23:50:03.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.892+0000 7fb81c34a700 1 -- start start 2024-01-26T23:50:03.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.893+0000 7fb81c34a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb814068b00 0x7fb8141450e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.893+0000 7fb81c34a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb814068ed0 0x7fb8140b1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.893+0000 7fb81c34a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb81406b4e0 0x7fb81415d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.893+0000 7fb81c34a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8140b15f0 con 0x7fb814068b00 2024-01-26T23:50:03.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.893+0000 7fb81c34a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8140b09e0 con 0x7fb814068ed0 2024-01-26T23:50:03.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.893+0000 7fb81c34a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8140b0cc0 con 0x7fb81406b4e0 2024-01-26T23:50:03.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.893+0000 7fb81a0e6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb814068b00 0x7fb8141450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.893+0000 7fb81a0e6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb814068b00 0x7fb8141450e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34562/0 (socket says 172.21.15.46:34562) 2024-01-26T23:50:03.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.893+0000 7fb81a0e6700 1 -- 172.21.15.46:0/267984749 learned_addr learned my addr 172.21.15.46:0/267984749 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:03.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.893+0000 7fb8198e5700 1 --2- 172.21.15.46:0/267984749 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb814068ed0 0x7fb8140b1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.893+0000 7fb81a8e7700 1 --2- 172.21.15.46:0/267984749 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb81406b4e0 0x7fb81415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.894+0000 7fb81a0e6700 1 -- 172.21.15.46:0/267984749 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb81406b4e0 msgr2=0x7fb81415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:03.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.894+0000 7fb81a0e6700 1 --2- 172.21.15.46:0/267984749 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb81406b4e0 0x7fb81415d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.894+0000 7fb81a0e6700 1 -- 172.21.15.46:0/267984749 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb814068ed0 msgr2=0x7fb8140b1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:03.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.894+0000 7fb81a0e6700 1 --2- 172.21.15.46:0/267984749 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb814068ed0 0x7fb8140b1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:03.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.894+0000 7fb81a0e6700 1 -- 172.21.15.46:0/267984749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80c009a20 con 0x7fb814068b00 2024-01-26T23:50:03.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.894+0000 7fb81a0e6700 1 --2- 172.21.15.46:0/267984749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb814068b00 0x7fb8141450e0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fb810001f00 tx=0x7fb810001060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:03.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.894+0000 7fb80a7fc700 1 -- 172.21.15.46:0/267984749 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb810011590 con 0x7fb814068b00 2024-01-26T23:50:03.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.894+0000 7fb80a7fc700 1 -- 172.21.15.46:0/267984749 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8100116f0 con 0x7fb814068b00 2024-01-26T23:50:03.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.894+0000 7fb81c34a700 1 -- 172.21.15.46:0/267984749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8140b0fa0 con 0x7fb814068b00 2024-01-26T23:50:03.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.894+0000 7fb80a7fc700 1 -- 172.21.15.46:0/267984749 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb810011ab0 con 0x7fb814068b00 2024-01-26T23:50:03.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.894+0000 7fb81c34a700 1 -- 172.21.15.46:0/267984749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb814145620 con 0x7fb814068b00 2024-01-26T23:50:03.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.895+0000 7fb81c34a700 1 -- 172.21.15.46:0/267984749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81404c710 con 0x7fb814068b00 2024-01-26T23:50:03.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.896+0000 7fb80a7fc700 1 -- 172.21.15.46:0/267984749 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb81001b3b0 con 0x7fb814068b00 2024-01-26T23:50:03.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.896+0000 7fb80a7fc700 1 --2- 172.21.15.46:0/267984749 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb7f8066f30 0x7fb7f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:03.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.897+0000 7fb80a7fc700 1 -- 172.21.15.46:0/267984749 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5935+0+0 (secure 0 0 0) 0x7fb8100839d0 con 0x7fb814068b00 2024-01-26T23:50:03.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.897+0000 7fb8198e5700 1 --2- 172.21.15.46:0/267984749 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb7f8066f30 0x7fb7f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:03.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.897+0000 7fb8198e5700 1 --2- 172.21.15.46:0/267984749 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb7f8066f30 0x7fb7f80693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb804007fd0 tx=0x7fb804009040 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:03.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:03.901+0000 7fb80a7fc700 1 -- 172.21.15.46:0/267984749 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb810050c10 con 0x7fb814068b00 2024-01-26T23:50:04.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.105+0000 7fb81c34a700 1 -- 172.21.15.46:0/267984749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb8140c93e0 con 0x7fb814068b00 2024-01-26T23:50:04.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.737+0000 7fb80a7fc700 1 -- 172.21.15.46:0/267984749 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v156) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb810017070 con 0x7fb814068b00 2024-01-26T23:50:04.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.742+0000 7fb81c34a700 1 -- 172.21.15.46:0/267984749 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb7f8066f30 msgr2=0x7fb7f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:04.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.742+0000 7fb81c34a700 1 --2- 172.21.15.46:0/267984749 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb7f8066f30 0x7fb7f80693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb804007fd0 tx=0x7fb804009040 comp rx=0 tx=0).stop 2024-01-26T23:50:04.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.742+0000 7fb81c34a700 1 -- 172.21.15.46:0/267984749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb814068b00 msgr2=0x7fb8141450e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:04.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.742+0000 7fb81c34a700 1 --2- 172.21.15.46:0/267984749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb814068b00 0x7fb8141450e0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fb810001f00 tx=0x7fb810001060 comp rx=0 tx=0).stop 2024-01-26T23:50:04.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.743+0000 7fb81c34a700 1 -- 172.21.15.46:0/267984749 shutdown_connections 2024-01-26T23:50:04.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.743+0000 7fb81c34a700 1 --2- 172.21.15.46:0/267984749 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb7f8066f30 0x7fb7f80693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:04.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.743+0000 7fb81c34a700 1 --2- 172.21.15.46:0/267984749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb814068b00 0x7fb8141450e0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:04.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.743+0000 7fb81c34a700 1 --2- 172.21.15.46:0/267984749 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb81406b4e0 0x7fb81415d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:04.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.743+0000 7fb81c34a700 1 --2- 172.21.15.46:0/267984749 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb814068ed0 0x7fb8140b1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:04.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.743+0000 7fb81c34a700 1 -- 172.21.15.46:0/267984749 >> 172.21.15.46:0/267984749 conn(0x7fb8140ca100 msgr2=0x7fb814159ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:04.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.745+0000 7fb81c34a700 1 -- 172.21.15.46:0/267984749 shutdown_connections 2024-01-26T23:50:04.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.745+0000 7fb81c34a700 1 -- 172.21.15.46:0/267984749 wait complete. 2024-01-26T23:50:04.756 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-01-26T23:50:04.772 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-26T23:50:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.909+0000 7ffa6e94e700 1 Processor -- start 2024-01-26T23:50:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.909+0000 7ffa6e94e700 1 -- start start 2024-01-26T23:50:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.910+0000 7ffa6e94e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa68068940 0x7ffa68068d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.910+0000 7ffa6e94e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffa680c90c0 0x7ffa680af990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.910+0000 7ffa6e94e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa680c87c0 con 0x7ffa680c8c50 2024-01-26T23:50:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.910+0000 7ffa6e94e700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa68069250 con 0x7ffa680c90c0 2024-01-26T23:50:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.910+0000 7ffa6e94e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa680c7bb0 con 0x7ffa68068940 2024-01-26T23:50:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.910+0000 7ffa6ceeb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffa680c90c0 0x7ffa680af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.910+0000 7ffa6ceeb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffa680c90c0 0x7ffa680af990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34576/0 (socket says 172.21.15.46:34576) 2024-01-26T23:50:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.910+0000 7ffa6ceeb700 1 -- 172.21.15.46:0/3819002203 learned_addr learned my addr 172.21.15.46:0/3819002203 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.910+0000 7ffa67fff700 1 --2- 172.21.15.46:0/3819002203 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa68068940 0x7ffa68068d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.911+0000 7ffa6ceeb700 1 -- 172.21.15.46:0/3819002203 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa68068940 msgr2=0x7ffa68068d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.911+0000 7ffa6ceeb700 1 --2- 172.21.15.46:0/3819002203 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa68068940 0x7ffa68068d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.911+0000 7ffa6ceeb700 1 -- 172.21.15.46:0/3819002203 >> v1:172.21.15.106:6789/0 conn(0x7ffa680c8c50 legacy=0x7ffa680c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.911+0000 7ffa6ceeb700 1 -- 172.21.15.46:0/3819002203 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa680c7e90 con 0x7ffa680c90c0 2024-01-26T23:50:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.911+0000 7ffa66ffd700 1 -- 172.21.15.46:0/3819002203 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380741395 0 0) 0x7ffa680c87c0 con 0x7ffa680c8c50 2024-01-26T23:50:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.911+0000 7ffa6ceeb700 1 --2- 172.21.15.46:0/3819002203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa680c90c0 0x7ffa680af990 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009f20 tx=0x7ffa60009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2aebb1c10453529 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.911+0000 7ffa66ffd700 1 -- 172.21.15.46:0/3819002203 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6000e070 con 0x7ffa680c90c0 2024-01-26T23:50:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.911+0000 7ffa66ffd700 1 -- 172.21.15.46:0/3819002203 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6000b9d0 con 0x7ffa680c90c0 2024-01-26T23:50:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.911+0000 7ffa66ffd700 1 -- 172.21.15.46:0/3819002203 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa600092e0 con 0x7ffa680c90c0 2024-01-26T23:50:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.912+0000 7ffa6e94e700 1 -- 172.21.15.46:0/3819002203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa680c90c0 msgr2=0x7ffa680af990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:04.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.912+0000 7ffa6e94e700 1 --2- 172.21.15.46:0/3819002203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa680c90c0 0x7ffa680af990 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009f20 tx=0x7ffa60009840 comp rx=0 tx=0).stop 2024-01-26T23:50:04.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.912+0000 7ffa6e94e700 1 -- 172.21.15.46:0/3819002203 shutdown_connections 2024-01-26T23:50:04.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.912+0000 7ffa6e94e700 1 --2- 172.21.15.46:0/3819002203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa680c90c0 0x7ffa680af990 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:04.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.912+0000 7ffa6e94e700 1 --2- 172.21.15.46:0/3819002203 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa68068940 0x7ffa68068d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:04.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.912+0000 7ffa6e94e700 1 -- 172.21.15.46:0/3819002203 >> 172.21.15.46:0/3819002203 conn(0x7ffa680ca100 msgr2=0x7ffa6806e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:04.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.912+0000 7ffa6e94e700 1 -- 172.21.15.46:0/3819002203 shutdown_connections 2024-01-26T23:50:04.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.912+0000 7ffa6e94e700 1 -- 172.21.15.46:0/3819002203 wait complete. 2024-01-26T23:50:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.913+0000 7ffa6e94e700 1 Processor -- start 2024-01-26T23:50:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.913+0000 7ffa6e94e700 1 -- start start 2024-01-26T23:50:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.914+0000 7ffa6e94e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa68068940 0x7ffa680c2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.914+0000 7ffa6e94e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa680c8c50 0x7ffa6815b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.914+0000 7ffa6e94e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa680c90c0 0x7ffa680c48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.914+0000 7ffa6e94e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa680c3310 con 0x7ffa680c90c0 2024-01-26T23:50:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.914+0000 7ffa6e94e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa680c2940 con 0x7ffa68068940 2024-01-26T23:50:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.914+0000 7ffa6e94e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa680c1d30 con 0x7ffa680c8c50 2024-01-26T23:50:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.914+0000 7ffa6ceeb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa680c90c0 0x7ffa680c48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.914+0000 7ffa6ceeb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa680c90c0 0x7ffa680c48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34592/0 (socket says 172.21.15.46:34592) 2024-01-26T23:50:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.914+0000 7ffa6ceeb700 1 -- 172.21.15.46:0/1706263226 learned_addr learned my addr 172.21.15.46:0/1706263226 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.914+0000 7ffa67fff700 1 --2- 172.21.15.46:0/1706263226 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa68068940 0x7ffa680c2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.914+0000 7ffa677fe700 1 --2- 172.21.15.46:0/1706263226 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa680c8c50 0x7ffa6815b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.914+0000 7ffa6ceeb700 1 -- 172.21.15.46:0/1706263226 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa680c8c50 msgr2=0x7ffa6815b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.914+0000 7ffa6ceeb700 1 --2- 172.21.15.46:0/1706263226 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa680c8c50 0x7ffa6815b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.914+0000 7ffa6ceeb700 1 -- 172.21.15.46:0/1706263226 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa68068940 msgr2=0x7ffa680c2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.914+0000 7ffa6ceeb700 1 --2- 172.21.15.46:0/1706263226 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa68068940 0x7ffa680c2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.914+0000 7ffa6ceeb700 1 -- 172.21.15.46:0/1706263226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa60009ae0 con 0x7ffa680c90c0 2024-01-26T23:50:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.915+0000 7ffa6ceeb700 1 --2- 172.21.15.46:0/1706263226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa680c90c0 0x7ffa680c48f0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7ffa68069d70 tx=0x7ffa60018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.915+0000 7ffa4ffff700 1 -- 172.21.15.46:0/1706263226 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa60008c70 con 0x7ffa680c90c0 2024-01-26T23:50:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.915+0000 7ffa4ffff700 1 -- 172.21.15.46:0/1706263226 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa60008df0 con 0x7ffa680c90c0 2024-01-26T23:50:04.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.915+0000 7ffa6e94e700 1 -- 172.21.15.46:0/1706263226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa680c2010 con 0x7ffa680c90c0 2024-01-26T23:50:04.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.915+0000 7ffa4ffff700 1 -- 172.21.15.46:0/1706263226 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa600256b0 con 0x7ffa680c90c0 2024-01-26T23:50:04.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.915+0000 7ffa6e94e700 1 -- 172.21.15.46:0/1706263226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa680c25d0 con 0x7ffa680c90c0 2024-01-26T23:50:04.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.916+0000 7ffa6e94e700 1 -- 172.21.15.46:0/1706263226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa680c2af0 con 0x7ffa680c90c0 2024-01-26T23:50:04.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.916+0000 7ffa4ffff700 1 -- 172.21.15.46:0/1706263226 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ffa60003040 con 0x7ffa680c90c0 2024-01-26T23:50:04.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.917+0000 7ffa4ffff700 1 --2- 172.21.15.46:0/1706263226 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ffa54066f30 0x7ffa540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:04.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.917+0000 7ffa4ffff700 1 -- 172.21.15.46:0/1706263226 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6317+0+0 (secure 0 0 0) 0x7ffa600864a0 con 0x7ffa680c90c0 2024-01-26T23:50:04.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.917+0000 7ffa67fff700 1 --2- 172.21.15.46:0/1706263226 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ffa54066f30 0x7ffa540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:04.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.918+0000 7ffa67fff700 1 --2- 172.21.15.46:0/1706263226 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ffa54066f30 0x7ffa540693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffa58009d30 tx=0x7ffa58008040 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:04.922+0000 7ffa4ffff700 1 -- 172.21.15.46:0/1706263226 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffa680c2af0 con 0x7ffa680c90c0 2024-01-26T23:50:05.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.173+0000 7ffa6e94e700 1 -- 172.21.15.46:0/1706263226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ffa680c2ab0 con 0x7ffa680c90c0 2024-01-26T23:50:05.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.758+0000 7ffa4ffff700 1 -- 172.21.15.46:0/1706263226 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 19 and data pool 20 v75) v1 ==== 174+0+0 (secure 0 0 0) 0x7ffa60053410 con 0x7ffa680c90c0 2024-01-26T23:50:05.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.773+0000 7ffa6e94e700 1 -- 172.21.15.46:0/1706263226 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ffa54066f30 msgr2=0x7ffa540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:05.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.773+0000 7ffa6e94e700 1 --2- 172.21.15.46:0/1706263226 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ffa54066f30 0x7ffa540693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffa58009d30 tx=0x7ffa58008040 comp rx=0 tx=0).stop 2024-01-26T23:50:05.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.773+0000 7ffa6e94e700 1 -- 172.21.15.46:0/1706263226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa680c90c0 msgr2=0x7ffa680c48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:05.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.773+0000 7ffa6e94e700 1 --2- 172.21.15.46:0/1706263226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa680c90c0 0x7ffa680c48f0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7ffa68069d70 tx=0x7ffa60018000 comp rx=0 tx=0).stop 2024-01-26T23:50:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.773+0000 7ffa6e94e700 1 -- 172.21.15.46:0/1706263226 shutdown_connections 2024-01-26T23:50:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.773+0000 7ffa6e94e700 1 --2- 172.21.15.46:0/1706263226 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ffa54066f30 0x7ffa540693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.773+0000 7ffa6e94e700 1 --2- 172.21.15.46:0/1706263226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa680c90c0 0x7ffa680c48f0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.773+0000 7ffa6e94e700 1 --2- 172.21.15.46:0/1706263226 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa68068940 0x7ffa680c2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.773+0000 7ffa6e94e700 1 --2- 172.21.15.46:0/1706263226 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa680c8c50 0x7ffa6815b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.773+0000 7ffa6e94e700 1 -- 172.21.15.46:0/1706263226 >> 172.21.15.46:0/1706263226 conn(0x7ffa680ca100 msgr2=0x7ffa680b2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.774+0000 7ffa6e94e700 1 -- 172.21.15.46:0/1706263226 shutdown_connections 2024-01-26T23:50:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.774+0000 7ffa6e94e700 1 -- 172.21.15.46:0/1706263226 wait complete. 2024-01-26T23:50:05.776 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 19 and data pool 20 2024-01-26T23:50:05.790 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:50:05.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.924+0000 7f68323a2700 1 Processor -- start 2024-01-26T23:50:05.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.924+0000 7f68323a2700 1 -- start start 2024-01-26T23:50:05.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.925+0000 7f68323a2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f682c0ae520 0x7f682c0b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:05.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.925+0000 7f68323a2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f682c158ae0 con 0x7f682c0b3fb0 2024-01-26T23:50:05.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.925+0000 7f68323a2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f682c15b650 con 0x7f682c157340 2024-01-26T23:50:05.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.925+0000 7f68323a2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f682c0b4430 con 0x7f682c0ae520 2024-01-26T23:50:05.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.925+0000 7f683093f700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f682c0b3fb0 0x7f682c157710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48962/0 (socket says 172.21.15.46:48962) 2024-01-26T23:50:05.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.925+0000 7f683093f700 1 -- 172.21.15.46:0/2157670928 learned_addr learned my addr 172.21.15.46:0/2157670928 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:05.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.925+0000 7f682b7fe700 1 --2- 172.21.15.46:0/2157670928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f682c0ae520 0x7f682c0b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:05.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.925+0000 7f682affd700 1 -- 172.21.15.46:0/2157670928 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030617135 0 0) 0x7f682c158ae0 con 0x7f682c0b3fb0 2024-01-26T23:50:05.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.925+0000 7f682affd700 1 -- 172.21.15.46:0/2157670928 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6814003570 con 0x7f682c0b3fb0 2024-01-26T23:50:05.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f682affd700 1 -- 172.21.15.46:0/2157670928 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1487110428 0 0) 0x7f6814003570 con 0x7f682c0b3fb0 2024-01-26T23:50:05.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f682affd700 1 -- 172.21.15.46:0/2157670928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f682c0ae520 msgr2=0x7f682c0b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:05.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f682affd700 1 --2- 172.21.15.46:0/2157670928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f682c0ae520 0x7f682c0b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:05.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f682affd700 1 -- 172.21.15.46:0/2157670928 >> v1:172.21.15.106:6789/0 conn(0x7f682c157340 legacy=0x7f682c15bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:05.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f682affd700 1 -- 172.21.15.46:0/2157670928 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682c158520 con 0x7f682c0b3fb0 2024-01-26T23:50:05.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f682affd700 1 -- 172.21.15.46:0/2157670928 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f68240017f0 con 0x7f682c0b3fb0 2024-01-26T23:50:05.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f682affd700 1 -- 172.21.15.46:0/2157670928 >> v1:172.21.15.46:6789/0 conn(0x7f682c0b3fb0 legacy=0x7f682c157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:05.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f682affd700 1 --2- 172.21.15.46:0/2157670928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f682c0ae520 0x7f6814003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:05.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f682affd700 1 --2- 172.21.15.46:0/2157670928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6814003fd0 0x7f6814004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:05.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f682affd700 1 --2- 172.21.15.46:0/2157670928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6814004d50 0x7f68140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:05.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f682affd700 1 -- 172.21.15.46:0/2157670928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682c0b4430 con 0x7f6814004d50 2024-01-26T23:50:05.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f682affd700 1 -- 172.21.15.46:0/2157670928 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681400bbf0 con 0x7f6814003fd0 2024-01-26T23:50:05.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f682affd700 1 -- 172.21.15.46:0/2157670928 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681400bdb0 con 0x7f682c0ae520 2024-01-26T23:50:05.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f683093f700 1 --2- 172.21.15.46:0/2157670928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6814004d50 0x7f68140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:05.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f682b7fe700 1 --2- 172.21.15.46:0/2157670928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6814003fd0 0x7f6814004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:05.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f682bfff700 1 --2- 172.21.15.46:0/2157670928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f682c0ae520 0x7f6814003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:05.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f683093f700 1 -- 172.21.15.46:0/2157670928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f682c0ae520 msgr2=0x7f6814003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:05.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f683093f700 1 --2- 172.21.15.46:0/2157670928 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f682c0ae520 0x7f6814003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:05.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f683093f700 1 -- 172.21.15.46:0/2157670928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6814003fd0 msgr2=0x7f6814004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:05.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f683093f700 1 --2- 172.21.15.46:0/2157670928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6814003fd0 0x7f6814004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:05.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.926+0000 7f683093f700 1 -- 172.21.15.46:0/2157670928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6814003570 con 0x7f6814004d50 2024-01-26T23:50:05.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.927+0000 7f683093f700 1 --2- 172.21.15.46:0/2157670928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6814004d50 0x7f68140051c0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f682400a950 tx=0x7f68240021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4ee0cf9fd8d376 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:05.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.927+0000 7f682affd700 1 -- 172.21.15.46:0/2157670928 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68240070d0 con 0x7f6814004d50 2024-01-26T23:50:05.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.927+0000 7f682affd700 1 -- 172.21.15.46:0/2157670928 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6824007230 con 0x7f6814004d50 2024-01-26T23:50:05.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.927+0000 7f682affd700 1 -- 172.21.15.46:0/2157670928 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68240075f0 con 0x7f6814004d50 2024-01-26T23:50:05.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.927+0000 7f68323a2700 1 -- 172.21.15.46:0/2157670928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6814004d50 msgr2=0x7f68140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:05.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.927+0000 7f68323a2700 1 --2- 172.21.15.46:0/2157670928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6814004d50 0x7f68140051c0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f682400a950 tx=0x7f68240021f0 comp rx=0 tx=0).stop 2024-01-26T23:50:05.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.927+0000 7f683093f700 1 -- 172.21.15.46:0/2157670928 reap_dead start 2024-01-26T23:50:05.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.927+0000 7f68323a2700 1 -- 172.21.15.46:0/2157670928 shutdown_connections 2024-01-26T23:50:05.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.927+0000 7f68323a2700 1 -- 172.21.15.46:0/2157670928 >> 172.21.15.46:0/2157670928 conn(0x7f682c0ce530 msgr2=0x7f682c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:05.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.927+0000 7f68323a2700 1 -- 172.21.15.46:0/2157670928 shutdown_connections 2024-01-26T23:50:05.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.927+0000 7f68323a2700 1 -- 172.21.15.46:0/2157670928 wait complete. 2024-01-26T23:50:05.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.929+0000 7f68323a2700 1 Processor -- start 2024-01-26T23:50:05.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.929+0000 7f68323a2700 1 -- start start 2024-01-26T23:50:05.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f68323a2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f682c1551a0 0x7f682c0c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:05.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f68323a2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f682c167680 0x7f682c0c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:05.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f68323a2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f682c0ca3a0 0x7f682c0cd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:05.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f68323a2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682c1686d0 con 0x7f682c167680 2024-01-26T23:50:05.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f68323a2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682c0ca820 con 0x7f682c0ca3a0 2024-01-26T23:50:05.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f68323a2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f682c0c3180 con 0x7f682c1551a0 2024-01-26T23:50:05.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f682b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f682c167680 0x7f682c0c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:05.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f682b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f682c167680 0x7f682c0c9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34608/0 (socket says 172.21.15.46:34608) 2024-01-26T23:50:05.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f682b7fe700 1 -- 172.21.15.46:0/206953350 learned_addr learned my addr 172.21.15.46:0/206953350 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:05.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f683093f700 1 --2- 172.21.15.46:0/206953350 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f682c0ca3a0 0x7f682c0cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:05.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f682bfff700 1 --2- 172.21.15.46:0/206953350 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f682c1551a0 0x7f682c0c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:05.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f682b7fe700 1 -- 172.21.15.46:0/206953350 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f682c1551a0 msgr2=0x7f682c0c2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:05.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f682b7fe700 1 --2- 172.21.15.46:0/206953350 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f682c1551a0 0x7f682c0c2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:05.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f682b7fe700 1 -- 172.21.15.46:0/206953350 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f682c0ca3a0 msgr2=0x7f682c0cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:05.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f682b7fe700 1 --2- 172.21.15.46:0/206953350 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f682c0ca3a0 0x7f682c0cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:05.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f682b7fe700 1 -- 172.21.15.46:0/206953350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682c155570 con 0x7f682c167680 2024-01-26T23:50:05.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f683093f700 1 --2- 172.21.15.46:0/206953350 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f682c0ca3a0 0x7f682c0cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:50:05.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f682b7fe700 1 --2- 172.21.15.46:0/206953350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f682c167680 0x7f682c0c9e60 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f6820011b10 tx=0x7f68200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:05.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f6813fff700 1 -- 172.21.15.46:0/206953350 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682000f9f0 con 0x7f682c167680 2024-01-26T23:50:05.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f6813fff700 1 -- 172.21.15.46:0/206953350 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682000fb50 con 0x7f682c167680 2024-01-26T23:50:05.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.930+0000 7f68323a2700 1 -- 172.21.15.46:0/206953350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f682c0c65d0 con 0x7f682c167680 2024-01-26T23:50:05.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.931+0000 7f6813fff700 1 -- 172.21.15.46:0/206953350 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6820014440 con 0x7f682c167680 2024-01-26T23:50:05.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.931+0000 7f68323a2700 1 -- 172.21.15.46:0/206953350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f682c168110 con 0x7f682c167680 2024-01-26T23:50:05.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.932+0000 7f68323a2700 1 -- 172.21.15.46:0/206953350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f682c04c710 con 0x7f682c167680 2024-01-26T23:50:05.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.932+0000 7f6813fff700 1 -- 172.21.15.46:0/206953350 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f68200147f0 con 0x7f682c167680 2024-01-26T23:50:05.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.932+0000 7f6813fff700 1 --2- 172.21.15.46:0/206953350 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6818066f30 0x7f68180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:05.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.933+0000 7f6813fff700 1 -- 172.21.15.46:0/206953350 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f6820018030 con 0x7f682c167680 2024-01-26T23:50:05.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.933+0000 7f682bfff700 1 --2- 172.21.15.46:0/206953350 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6818066f30 0x7f68180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:05.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.933+0000 7f682bfff700 1 --2- 172.21.15.46:0/206953350 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6818066f30 0x7f68180693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f681c003c10 tx=0x7f681c009040 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:05.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:05.937+0000 7f6813fff700 1 -- 172.21.15.46:0/206953350 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f68200522b0 con 0x7f682c167680 2024-01-26T23:50:06.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.143+0000 7f68323a2700 1 -- 172.21.15.46:0/206953350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f682c158800 con 0x7f682c167680 2024-01-26T23:50:06.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.144+0000 7f6813fff700 1 -- 172.21.15.46:0/206953350 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) v1 ==== 74+0+16176 (secure 0 0 0) 0x7f68200558d0 con 0x7f682c167680 2024-01-26T23:50:06.145 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:50:06.146 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":157,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:50:05.175727+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:50:03.691049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-01-26T23:50:04.138930+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:50:06.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.146+0000 7f68323a2700 1 -- 172.21.15.46:0/206953350 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6818066f30 msgr2=0x7f68180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:06.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.146+0000 7f68323a2700 1 --2- 172.21.15.46:0/206953350 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6818066f30 0x7f68180693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f681c003c10 tx=0x7f681c009040 comp rx=0 tx=0).stop 2024-01-26T23:50:06.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.146+0000 7f68323a2700 1 -- 172.21.15.46:0/206953350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f682c167680 msgr2=0x7f682c0c9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:06.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.146+0000 7f68323a2700 1 --2- 172.21.15.46:0/206953350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f682c167680 0x7f682c0c9e60 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f6820011b10 tx=0x7f68200091a0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.146+0000 7f68323a2700 1 -- 172.21.15.46:0/206953350 shutdown_connections 2024-01-26T23:50:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.146+0000 7f68323a2700 1 --2- 172.21.15.46:0/206953350 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6818066f30 0x7f68180693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.146+0000 7f68323a2700 1 --2- 172.21.15.46:0/206953350 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f682c0ca3a0 0x7f682c0cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.146+0000 7f68323a2700 1 --2- 172.21.15.46:0/206953350 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f682c1551a0 0x7f682c0c2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.146+0000 7f68323a2700 1 --2- 172.21.15.46:0/206953350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f682c167680 0x7f682c0c9e60 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.146+0000 7f68323a2700 1 -- 172.21.15.46:0/206953350 >> 172.21.15.46:0/206953350 conn(0x7f682c0ce530 msgr2=0x7f682c159d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.146+0000 7f68323a2700 1 -- 172.21.15.46:0/206953350 shutdown_connections 2024-01-26T23:50:06.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.146+0000 7f68323a2700 1 -- 172.21.15.46:0/206953350 wait complete. 2024-01-26T23:50:06.161 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:50:06.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.292+0000 7fa272462700 1 Processor -- start 2024-01-26T23:50:06.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.293+0000 7fa272462700 1 -- start start 2024-01-26T23:50:06.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.293+0000 7fa272462700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa26c0c6520 0x7fa26c0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:06.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.293+0000 7fa272462700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa26c0ced50 0x7fa26c0cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:06.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.293+0000 7fa272462700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa26c15a0b0 con 0x7fa26c0b2b40 2024-01-26T23:50:06.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.294+0000 7fa272462700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa26c0af070 con 0x7fa26c0ced50 2024-01-26T23:50:06.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.294+0000 7fa272462700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c099fc0 con 0x7fa26c0c6520 2024-01-26T23:50:06.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.294+0000 7fa26bfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa26c0b2b40 0x7fa26c0ae960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48972/0 (socket says 172.21.15.46:48972) 2024-01-26T23:50:06.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.294+0000 7fa26bfff700 1 -- 172.21.15.46:0/3286653817 learned_addr learned my addr 172.21.15.46:0/3286653817 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:06.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.294+0000 7fa2709ff700 1 --2- 172.21.15.46:0/3286653817 >> v2:172.21.15.106:3300/0 conn(0x7fa26c0ced50 0x7fa26c0cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:06.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.294+0000 7fa26b7fe700 1 --2- 172.21.15.46:0/3286653817 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa26c0c6520 0x7fa26c0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:06.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.294+0000 7fa26affd700 1 -- 172.21.15.46:0/3286653817 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195698875 0 0) 0x7fa26c15a0b0 con 0x7fa26c0b2b40 2024-01-26T23:50:06.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.294+0000 7fa26affd700 1 -- 172.21.15.46:0/3286653817 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa258003570 con 0x7fa26c0b2b40 2024-01-26T23:50:06.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.294+0000 7fa26affd700 1 -- 172.21.15.46:0/3286653817 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2023492743 0 0) 0x7fa258003570 con 0x7fa26c0b2b40 2024-01-26T23:50:06.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.294+0000 7fa26b7fe700 1 -- 172.21.15.46:0/3286653817 >> v2:172.21.15.106:3300/0 conn(0x7fa26c0ced50 msgr2=0x7fa26c0cf1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:06.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.294+0000 7fa26b7fe700 1 --2- 172.21.15.46:0/3286653817 >> v2:172.21.15.106:3300/0 conn(0x7fa26c0ced50 0x7fa26c0cf1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.294+0000 7fa26b7fe700 1 -- 172.21.15.46:0/3286653817 >> v1:172.21.15.46:6789/0 conn(0x7fa26c0b2b40 legacy=0x7fa26c0ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:06.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.294+0000 7fa26b7fe700 1 -- 172.21.15.46:0/3286653817 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26c0ae4d0 con 0x7fa26c0c6520 2024-01-26T23:50:06.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.295+0000 7fa26b7fe700 1 --2- 172.21.15.46:0/3286653817 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa26c0c6520 0x7fa26c0ad1c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0098e0 tx=0x7fa25c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cb43960c575fa9b server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:06.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.295+0000 7fa26affd700 1 -- 172.21.15.46:0/3286653817 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2540015f0 con 0x7fa26c0c6520 2024-01-26T23:50:06.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.295+0000 7fa26affd700 1 -- 172.21.15.46:0/3286653817 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25c011420 con 0x7fa26c0c6520 2024-01-26T23:50:06.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.295+0000 7fa26affd700 1 -- 172.21.15.46:0/3286653817 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25c0117c0 con 0x7fa26c0c6520 2024-01-26T23:50:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.295+0000 7fa272462700 1 -- 172.21.15.46:0/3286653817 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa26c0c6520 msgr2=0x7fa26c0ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.295+0000 7fa272462700 1 --2- 172.21.15.46:0/3286653817 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa26c0c6520 0x7fa26c0ad1c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0098e0 tx=0x7fa25c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.295+0000 7fa272462700 1 -- 172.21.15.46:0/3286653817 shutdown_connections 2024-01-26T23:50:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.295+0000 7fa272462700 1 --2- 172.21.15.46:0/3286653817 >> v2:172.21.15.106:3300/0 conn(0x7fa26c0ced50 0x7fa26c0cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.295+0000 7fa272462700 1 --2- 172.21.15.46:0/3286653817 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa26c0c6520 0x7fa26c0ad1c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.296+0000 7fa272462700 1 -- 172.21.15.46:0/3286653817 >> 172.21.15.46:0/3286653817 conn(0x7fa26c099a30 msgr2=0x7fa26c06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:06.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.296+0000 7fa272462700 1 -- 172.21.15.46:0/3286653817 shutdown_connections 2024-01-26T23:50:06.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.296+0000 7fa272462700 1 -- 172.21.15.46:0/3286653817 wait complete. 2024-01-26T23:50:06.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.297+0000 7fa272462700 1 Processor -- start 2024-01-26T23:50:06.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.297+0000 7fa272462700 1 -- start start 2024-01-26T23:50:06.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa272462700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa26c0b2b40 0x7fa26c155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:06.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa272462700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa26c0c6520 0x7fa26c0c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:06.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa272462700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa26c0ced50 0x7fa26c0b58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:06.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa272462700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c0b4e40 con 0x7fa26c0b2b40 2024-01-26T23:50:06.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa272462700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c0b42c0 con 0x7fa26c0ced50 2024-01-26T23:50:06.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa272462700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c0b45a0 con 0x7fa26c0c6520 2024-01-26T23:50:06.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa26bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa26c0b2b40 0x7fa26c155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa26bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa26c0b2b40 0x7fa26c155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34610/0 (socket says 172.21.15.46:34610) 2024-01-26T23:50:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa26bfff700 1 -- 172.21.15.46:0/8033621 learned_addr learned my addr 172.21.15.46:0/8033621 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa2709ff700 1 --2- 172.21.15.46:0/8033621 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa26c0ced50 0x7fa26c0b58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa26b7fe700 1 --2- 172.21.15.46:0/8033621 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa26c0c6520 0x7fa26c0c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa26bfff700 1 -- 172.21.15.46:0/8033621 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa26c0c6520 msgr2=0x7fa26c0c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa26bfff700 1 --2- 172.21.15.46:0/8033621 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa26c0c6520 0x7fa26c0c9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa26bfff700 1 -- 172.21.15.46:0/8033621 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa26c0ced50 msgr2=0x7fa26c0b58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa26bfff700 1 --2- 172.21.15.46:0/8033621 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa26c0ced50 0x7fa26c0b58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa26bfff700 1 -- 172.21.15.46:0/8033621 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c009420 con 0x7fa26c0b2b40 2024-01-26T23:50:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa2709ff700 1 --2- 172.21.15.46:0/8033621 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa26c0ced50 0x7fa26c0b58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:50:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa26b7fe700 1 --2- 172.21.15.46:0/8033621 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa26c0c6520 0x7fa26c0c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:50:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.298+0000 7fa26bfff700 1 --2- 172.21.15.46:0/8033621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa26c0b2b40 0x7fa26c155f30 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fa25400cb90 tx=0x7fa254002ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.299+0000 7fa263fff700 1 -- 172.21.15.46:0/8033621 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254007300 con 0x7fa26c0b2b40 2024-01-26T23:50:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.299+0000 7fa263fff700 1 -- 172.21.15.46:0/8033621 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa254007460 con 0x7fa26c0b2b40 2024-01-26T23:50:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.299+0000 7fa272462700 1 -- 172.21.15.46:0/8033621 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa26c0b4880 con 0x7fa26c0b2b40 2024-01-26T23:50:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.299+0000 7fa263fff700 1 -- 172.21.15.46:0/8033621 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2540057f0 con 0x7fa26c0b2b40 2024-01-26T23:50:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.299+0000 7fa272462700 1 -- 172.21.15.46:0/8033621 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26c0b3f20 con 0x7fa26c0b2b40 2024-01-26T23:50:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.300+0000 7fa272462700 1 -- 172.21.15.46:0/8033621 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa26c04c710 con 0x7fa26c0b2b40 2024-01-26T23:50:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.300+0000 7fa263fff700 1 -- 172.21.15.46:0/8033621 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa254005c20 con 0x7fa26c0b2b40 2024-01-26T23:50:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.301+0000 7fa263fff700 1 --2- 172.21.15.46:0/8033621 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa24c066f30 0x7fa24c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.301+0000 7fa263fff700 1 -- 172.21.15.46:0/8033621 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fa2540846a0 con 0x7fa26c0b2b40 2024-01-26T23:50:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.301+0000 7fa26b7fe700 1 --2- 172.21.15.46:0/8033621 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa24c066f30 0x7fa24c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.301+0000 7fa26b7fe700 1 --2- 172.21.15.46:0/8033621 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa24c066f30 0x7fa24c0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0ca9a0 tx=0x7fa25c016000 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:06.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.305+0000 7fa263fff700 1 -- 172.21.15.46:0/8033621 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa2540516c0 con 0x7fa26c0b2b40 2024-01-26T23:50:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.508+0000 7fa272462700 1 -- 172.21.15.46:0/8033621 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa26c0682c0 con 0x7fa26c0b2b40 2024-01-26T23:50:06.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.509+0000 7fa263fff700 1 -- 172.21.15.46:0/8033621 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) v1 ==== 74+0+16176 (secure 0 0 0) 0x7fa254054ce0 con 0x7fa26c0b2b40 2024-01-26T23:50:06.510 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:50:06.510 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":157,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:50:05.175727+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:50:03.691049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-01-26T23:50:04.138930+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:50:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.511+0000 7fa272462700 1 -- 172.21.15.46:0/8033621 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa24c066f30 msgr2=0x7fa24c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.511+0000 7fa272462700 1 --2- 172.21.15.46:0/8033621 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa24c066f30 0x7fa24c0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0ca9a0 tx=0x7fa25c016000 comp rx=0 tx=0).stop 2024-01-26T23:50:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.511+0000 7fa272462700 1 -- 172.21.15.46:0/8033621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa26c0b2b40 msgr2=0x7fa26c155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.511+0000 7fa272462700 1 --2- 172.21.15.46:0/8033621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa26c0b2b40 0x7fa26c155f30 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fa25400cb90 tx=0x7fa254002ad0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.511+0000 7fa272462700 1 -- 172.21.15.46:0/8033621 shutdown_connections 2024-01-26T23:50:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.511+0000 7fa272462700 1 --2- 172.21.15.46:0/8033621 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa24c066f30 0x7fa24c0693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.511+0000 7fa272462700 1 --2- 172.21.15.46:0/8033621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa26c0b2b40 0x7fa26c155f30 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.511+0000 7fa272462700 1 --2- 172.21.15.46:0/8033621 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa26c0ced50 0x7fa26c0b58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.511+0000 7fa272462700 1 --2- 172.21.15.46:0/8033621 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa26c0c6520 0x7fa26c0c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.511+0000 7fa272462700 1 -- 172.21.15.46:0/8033621 >> 172.21.15.46:0/8033621 conn(0x7fa26c099a30 msgr2=0x7fa26c159350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.511+0000 7fa272462700 1 -- 172.21.15.46:0/8033621 shutdown_connections 2024-01-26T23:50:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.511+0000 7fa272462700 1 -- 172.21.15.46:0/8033621 wait complete. 2024-01-26T23:50:06.526 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-26T23:50:06.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.659+0000 7f883a895700 1 Processor -- start 2024-01-26T23:50:06.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.660+0000 7f883a895700 1 -- start start 2024-01-26T23:50:06.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.660+0000 7f883a895700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88341565a0 0x7f8834156970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:06.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.660+0000 7f883a895700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f883415a520 0x7f88340acc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:06.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.660+0000 7f883a895700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88340b0830 con 0x7f88340ae2f0 2024-01-26T23:50:06.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.660+0000 7f883a895700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88340ae130 con 0x7f883415a520 2024-01-26T23:50:06.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.660+0000 7f883a895700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88340ad5b0 con 0x7f88341565a0 2024-01-26T23:50:06.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.661+0000 7f88337fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f883415a520 0x7f88340acc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:06.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.661+0000 7f8838e32700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f88340ae2f0 0x7f883406ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48982/0 (socket says 172.21.15.46:48982) 2024-01-26T23:50:06.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.661+0000 7f8838e32700 1 -- 172.21.15.46:0/762229960 learned_addr learned my addr 172.21.15.46:0/762229960 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:06.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.661+0000 7f8833fff700 1 --2- 172.21.15.46:0/762229960 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88341565a0 0x7f8834156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:06.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.661+0000 7f88337fe700 1 -- 172.21.15.46:0/762229960 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88341565a0 msgr2=0x7f8834156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:06.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.661+0000 7f88337fe700 1 --2- 172.21.15.46:0/762229960 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88341565a0 0x7f8834156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.661+0000 7f88337fe700 1 -- 172.21.15.46:0/762229960 >> v1:172.21.15.46:6789/0 conn(0x7f88340ae2f0 legacy=0x7f883406ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:06.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.661+0000 7f88337fe700 1 -- 172.21.15.46:0/762229960 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88340ad890 con 0x7f883415a520 2024-01-26T23:50:06.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.661+0000 7f8832ffd700 1 -- 172.21.15.46:0/762229960 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723783144 0 0) 0x7f88340b0830 con 0x7f88340ae2f0 2024-01-26T23:50:06.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.661+0000 7f88337fe700 1 --2- 172.21.15.46:0/762229960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f883415a520 0x7f88340acc40 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f881c0098e0 tx=0x7f881c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f32f8d15a10ebc7 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:06.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.661+0000 7f8832ffd700 1 -- 172.21.15.46:0/762229960 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881c00b640 con 0x7f883415a520 2024-01-26T23:50:06.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.661+0000 7f8832ffd700 1 -- 172.21.15.46:0/762229960 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f881c00b7a0 con 0x7f883415a520 2024-01-26T23:50:06.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.661+0000 7f8832ffd700 1 -- 172.21.15.46:0/762229960 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881c00bb40 con 0x7f883415a520 2024-01-26T23:50:06.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.662+0000 7f883a895700 1 -- 172.21.15.46:0/762229960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f883415a520 msgr2=0x7f88340acc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:06.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.662+0000 7f883a895700 1 --2- 172.21.15.46:0/762229960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f883415a520 0x7f88340acc40 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f881c0098e0 tx=0x7f881c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.662+0000 7f883a895700 1 -- 172.21.15.46:0/762229960 shutdown_connections 2024-01-26T23:50:06.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.662+0000 7f883a895700 1 --2- 172.21.15.46:0/762229960 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88341565a0 0x7f8834156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.662+0000 7f883a895700 1 --2- 172.21.15.46:0/762229960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f883415a520 0x7f88340acc40 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.662+0000 7f883a895700 1 -- 172.21.15.46:0/762229960 >> 172.21.15.46:0/762229960 conn(0x7f88340cdf80 msgr2=0x7f883406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:06.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.662+0000 7f883a895700 1 -- 172.21.15.46:0/762229960 shutdown_connections 2024-01-26T23:50:06.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.662+0000 7f883a895700 1 -- 172.21.15.46:0/762229960 wait complete. 2024-01-26T23:50:06.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.663+0000 7f883a895700 1 Processor -- start 2024-01-26T23:50:06.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.663+0000 7f883a895700 1 -- start start 2024-01-26T23:50:06.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.664+0000 7f883a895700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88340ae2f0 0x7f883415bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:06.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.664+0000 7f883a895700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88341565a0 0x7f8834143240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:06.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.664+0000 7f883a895700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f883415a520 0x7f8834154e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:06.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.664+0000 7f883a895700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8834157cf0 con 0x7f88340ae2f0 2024-01-26T23:50:06.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.664+0000 7f883a895700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883415c590 con 0x7f883415a520 2024-01-26T23:50:06.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.664+0000 7f883a895700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f883415c870 con 0x7f88341565a0 2024-01-26T23:50:06.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.664+0000 7f8833fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88340ae2f0 0x7f883415bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:06.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.664+0000 7f8833fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88340ae2f0 0x7f883415bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34620/0 (socket says 172.21.15.46:34620) 2024-01-26T23:50:06.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.664+0000 7f8833fff700 1 -- 172.21.15.46:0/2053969474 learned_addr learned my addr 172.21.15.46:0/2053969474 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:06.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.664+0000 7f88337fe700 1 --2- 172.21.15.46:0/2053969474 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88341565a0 0x7f8834143240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:06.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.665+0000 7f8838e32700 1 --2- 172.21.15.46:0/2053969474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f883415a520 0x7f8834154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:06.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.665+0000 7f8833fff700 1 -- 172.21.15.46:0/2053969474 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88341565a0 msgr2=0x7f8834143240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:06.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.665+0000 7f8833fff700 1 --2- 172.21.15.46:0/2053969474 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88341565a0 0x7f8834143240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.665+0000 7f8833fff700 1 -- 172.21.15.46:0/2053969474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f883415a520 msgr2=0x7f8834154e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:06.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.665+0000 7f8833fff700 1 --2- 172.21.15.46:0/2053969474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f883415a520 0x7f8834154e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:06.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.665+0000 7f8833fff700 1 -- 172.21.15.46:0/2053969474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881c009420 con 0x7f88340ae2f0 2024-01-26T23:50:06.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.665+0000 7f88337fe700 1 --2- 172.21.15.46:0/2053969474 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88341565a0 0x7f8834143240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:50:06.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.665+0000 7f8833fff700 1 --2- 172.21.15.46:0/2053969474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88340ae2f0 0x7f883415bfb0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f882400ec80 tx=0x7f88240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:06.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.665+0000 7f882bfff700 1 -- 172.21.15.46:0/2053969474 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882400c9d0 con 0x7f88340ae2f0 2024-01-26T23:50:06.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.665+0000 7f882bfff700 1 -- 172.21.15.46:0/2053969474 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882400cb30 con 0x7f88340ae2f0 2024-01-26T23:50:06.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.665+0000 7f883a895700 1 -- 172.21.15.46:0/2053969474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8834157f40 con 0x7f88340ae2f0 2024-01-26T23:50:06.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.665+0000 7f882bfff700 1 -- 172.21.15.46:0/2053969474 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8824013470 con 0x7f88340ae2f0 2024-01-26T23:50:06.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.665+0000 7f883a895700 1 -- 172.21.15.46:0/2053969474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8834155390 con 0x7f88340ae2f0 2024-01-26T23:50:06.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.666+0000 7f883a895700 1 -- 172.21.15.46:0/2053969474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8834158590 con 0x7f88340ae2f0 2024-01-26T23:50:06.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.667+0000 7f882bfff700 1 -- 172.21.15.46:0/2053969474 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8824013820 con 0x7f88340ae2f0 2024-01-26T23:50:06.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.667+0000 7f882bfff700 1 --2- 172.21.15.46:0/2053969474 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8814066f30 0x7f88140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:06.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.667+0000 7f882bfff700 1 -- 172.21.15.46:0/2053969474 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f8824084bf0 con 0x7f88340ae2f0 2024-01-26T23:50:06.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.667+0000 7f88337fe700 1 --2- 172.21.15.46:0/2053969474 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8814066f30 0x7f88140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:06.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.668+0000 7f88337fe700 1 --2- 172.21.15.46:0/2053969474 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8814066f30 0x7f88140693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f883415a8f0 tx=0x7f881c017000 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.672+0000 7f882bfff700 1 -- 172.21.15.46:0/2053969474 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8824051b90 con 0x7f88340ae2f0 2024-01-26T23:50:06.724 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-01-26T23:50:06.723+0000 7f8b0e093700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-26T23:50:06.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:06.930+0000 7f883a895700 1 -- 172.21.15.46:0/2053969474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f883404c710 con 0x7f88340ae2f0 2024-01-26T23:50:07.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.758+0000 7f882bfff700 1 -- 172.21.15.46:0/2053969474 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v77) v1 ==== 118+0+0 (secure 0 0 0) 0x7f88240551b0 con 0x7f88340ae2f0 2024-01-26T23:50:07.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.761+0000 7f883a895700 1 -- 172.21.15.46:0/2053969474 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8814066f30 msgr2=0x7f88140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:07.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.761+0000 7f883a895700 1 --2- 172.21.15.46:0/2053969474 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8814066f30 0x7f88140693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f883415a8f0 tx=0x7f881c017000 comp rx=0 tx=0).stop 2024-01-26T23:50:07.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.761+0000 7f883a895700 1 -- 172.21.15.46:0/2053969474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88340ae2f0 msgr2=0x7f883415bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:07.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.761+0000 7f883a895700 1 --2- 172.21.15.46:0/2053969474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88340ae2f0 0x7f883415bfb0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f882400ec80 tx=0x7f88240091a0 comp rx=0 tx=0).stop 2024-01-26T23:50:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.761+0000 7f883a895700 1 -- 172.21.15.46:0/2053969474 shutdown_connections 2024-01-26T23:50:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.761+0000 7f883a895700 1 --2- 172.21.15.46:0/2053969474 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8814066f30 0x7f88140693e0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f883415a8f0 tx=0x7f881c017000 comp rx=0 tx=0).stop 2024-01-26T23:50:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.761+0000 7f883a895700 1 --2- 172.21.15.46:0/2053969474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88340ae2f0 0x7f883415bfb0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.761+0000 7f883a895700 1 --2- 172.21.15.46:0/2053969474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f883415a520 0x7f8834154e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.761+0000 7f883a895700 1 --2- 172.21.15.46:0/2053969474 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88341565a0 0x7f8834143240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.761+0000 7f883a895700 1 -- 172.21.15.46:0/2053969474 >> 172.21.15.46:0/2053969474 conn(0x7f88340cdf80 msgr2=0x7f883406f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.761+0000 7f883a895700 1 -- 172.21.15.46:0/2053969474 shutdown_connections 2024-01-26T23:50:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.761+0000 7f883a895700 1 -- 172.21.15.46:0/2053969474 wait complete. 2024-01-26T23:50:07.778 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-26T23:50:07.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.915+0000 7f8c4eccd700 1 Processor -- start 2024-01-26T23:50:07.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.915+0000 7f8c4eccd700 1 -- start start 2024-01-26T23:50:07.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.916+0000 7f8c4eccd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c4815cd40 0x7f8c480cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:07.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.916+0000 7f8c4eccd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c4815d110 0x7f8c480cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:07.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.916+0000 7f8c4eccd700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c4806caa0 con 0x7f8c48067930 2024-01-26T23:50:07.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.916+0000 7f8c4eccd700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c480c7ac0 con 0x7f8c4815cd40 2024-01-26T23:50:07.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.916+0000 7f8c4eccd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c480ca8b0 con 0x7f8c4815d110 2024-01-26T23:50:07.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.916+0000 7f8c4ca69700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c4815cd40 0x7f8c480cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:07.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.916+0000 7f8c4d26a700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8c48067930 0x7f8c4815d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48984/0 (socket says 172.21.15.46:48984) 2024-01-26T23:50:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.916+0000 7f8c4d26a700 1 -- 172.21.15.46:0/3813852437 learned_addr learned my addr 172.21.15.46:0/3813852437 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.916+0000 7f8c47fff700 1 --2- 172.21.15.46:0/3813852437 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c4815d110 0x7f8c480cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.916+0000 7f8c4ca69700 1 -- 172.21.15.46:0/3813852437 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c4815d110 msgr2=0x7f8c480cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.916+0000 7f8c4ca69700 1 --2- 172.21.15.46:0/3813852437 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c4815d110 0x7f8c480cb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.916+0000 7f8c4ca69700 1 -- 172.21.15.46:0/3813852437 >> v1:172.21.15.46:6789/0 conn(0x7f8c48067930 legacy=0x7f8c4815d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.916+0000 7f8c4ca69700 1 -- 172.21.15.46:0/3813852437 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4806c1b0 con 0x7f8c4815cd40 2024-01-26T23:50:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.916+0000 7f8c477fe700 1 -- 172.21.15.46:0/3813852437 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198631290 0 0) 0x7f8c4806caa0 con 0x7f8c48067930 2024-01-26T23:50:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.917+0000 7f8c4ca69700 1 --2- 172.21.15.46:0/3813852437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4815cd40 0x7f8c480cb260 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c0098e0 tx=0x7f8c3c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0eb6d923d22b894 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.917+0000 7f8c477fe700 1 -- 172.21.15.46:0/3813852437 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c009e80 con 0x7f8c4815cd40 2024-01-26T23:50:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.917+0000 7f8c477fe700 1 -- 172.21.15.46:0/3813852437 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c3c00b860 con 0x7f8c4815cd40 2024-01-26T23:50:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.917+0000 7f8c477fe700 1 -- 172.21.15.46:0/3813852437 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c00bbe0 con 0x7f8c4815cd40 2024-01-26T23:50:07.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.917+0000 7f8c4eccd700 1 -- 172.21.15.46:0/3813852437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4815cd40 msgr2=0x7f8c480cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:07.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.917+0000 7f8c4eccd700 1 --2- 172.21.15.46:0/3813852437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4815cd40 0x7f8c480cb260 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c0098e0 tx=0x7f8c3c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:50:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.917+0000 7f8c4eccd700 1 -- 172.21.15.46:0/3813852437 shutdown_connections 2024-01-26T23:50:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.917+0000 7f8c4eccd700 1 --2- 172.21.15.46:0/3813852437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4815cd40 0x7f8c480cb260 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.917+0000 7f8c4eccd700 1 --2- 172.21.15.46:0/3813852437 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c4815d110 0x7f8c480cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.917+0000 7f8c4eccd700 1 -- 172.21.15.46:0/3813852437 >> 172.21.15.46:0/3813852437 conn(0x7f8c480cdf50 msgr2=0x7f8c480c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.918+0000 7f8c4eccd700 1 -- 172.21.15.46:0/3813852437 shutdown_connections 2024-01-26T23:50:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.918+0000 7f8c4eccd700 1 -- 172.21.15.46:0/3813852437 wait complete. 2024-01-26T23:50:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.919+0000 7f8c4eccd700 1 Processor -- start 2024-01-26T23:50:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.919+0000 7f8c4eccd700 1 -- start start 2024-01-26T23:50:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c4eccd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c48067930 0x7f8c480b2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c4eccd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4815cd40 0x7f8c48169010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c4eccd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c4815d110 0x7f8c4806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c4eccd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c480b3140 con 0x7f8c4815cd40 2024-01-26T23:50:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c4eccd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c480ad7d0 con 0x7f8c48067930 2024-01-26T23:50:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c4eccd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c480acea0 con 0x7f8c4815d110 2024-01-26T23:50:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c47fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4815cd40 0x7f8c48169010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c47fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4815cd40 0x7f8c48169010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34652/0 (socket says 172.21.15.46:34652) 2024-01-26T23:50:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c47fff700 1 -- 172.21.15.46:0/954843757 learned_addr learned my addr 172.21.15.46:0/954843757 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c4ca69700 1 --2- 172.21.15.46:0/954843757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c48067930 0x7f8c480b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c4d26a700 1 --2- 172.21.15.46:0/954843757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c4815d110 0x7f8c4806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c47fff700 1 -- 172.21.15.46:0/954843757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c4815d110 msgr2=0x7f8c4806f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c47fff700 1 --2- 172.21.15.46:0/954843757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c4815d110 0x7f8c4806f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c47fff700 1 -- 172.21.15.46:0/954843757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c48067930 msgr2=0x7f8c480b2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c47fff700 1 --2- 172.21.15.46:0/954843757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c48067930 0x7f8c480b2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c47fff700 1 -- 172.21.15.46:0/954843757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c3c009420 con 0x7f8c4815cd40 2024-01-26T23:50:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c4d26a700 1 --2- 172.21.15.46:0/954843757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c4815d110 0x7f8c4806f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:50:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.920+0000 7f8c47fff700 1 --2- 172.21.15.46:0/954843757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4815cd40 0x7f8c48169010 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400d860 tx=0x7f8c34008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.921+0000 7f8c44ff9700 1 -- 172.21.15.46:0/954843757 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c340125f0 con 0x7f8c4815cd40 2024-01-26T23:50:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.921+0000 7f8c44ff9700 1 -- 172.21.15.46:0/954843757 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c34012750 con 0x7f8c4815cd40 2024-01-26T23:50:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.921+0000 7f8c4eccd700 1 -- 172.21.15.46:0/954843757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c480ad180 con 0x7f8c4815cd40 2024-01-26T23:50:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.921+0000 7f8c44ff9700 1 -- 172.21.15.46:0/954843757 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c34012b10 con 0x7f8c4815cd40 2024-01-26T23:50:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.921+0000 7f8c4eccd700 1 -- 172.21.15.46:0/954843757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c480b2e40 con 0x7f8c4815cd40 2024-01-26T23:50:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.922+0000 7f8c4eccd700 1 -- 172.21.15.46:0/954843757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c480ad960 con 0x7f8c4815cd40 2024-01-26T23:50:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.922+0000 7f8c44ff9700 1 -- 172.21.15.46:0/954843757 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8c3401e460 con 0x7f8c4815cd40 2024-01-26T23:50:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.922+0000 7f8c44ff9700 1 --2- 172.21.15.46:0/954843757 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c2c066f30 0x7f8c2c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.923+0000 7f8c44ff9700 1 -- 172.21.15.46:0/954843757 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f8c34084da0 con 0x7f8c4815cd40 2024-01-26T23:50:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.923+0000 7f8c4ca69700 1 --2- 172.21.15.46:0/954843757 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c2c066f30 0x7f8c2c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.923+0000 7f8c4ca69700 1 --2- 172.21.15.46:0/954843757 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c2c066f30 0x7f8c2c0693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c0091e0 tx=0x7f8c3c017000 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:07.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:07.927+0000 7f8c44ff9700 1 -- 172.21.15.46:0/954843757 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8c34051d40 con 0x7f8c4815cd40 2024-01-26T23:50:08.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.182+0000 7f8c4eccd700 1 -- 172.21.15.46:0/954843757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8c4804c710 con 0x7f8c4815cd40 2024-01-26T23:50:08.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.760+0000 7f8c44ff9700 1 -- 172.21.15.46:0/954843757 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v78) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8c34055360 con 0x7f8c4815cd40 2024-01-26T23:50:08.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.763+0000 7f8c4eccd700 1 -- 172.21.15.46:0/954843757 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c2c066f30 msgr2=0x7f8c2c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:08.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.763+0000 7f8c4eccd700 1 --2- 172.21.15.46:0/954843757 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c2c066f30 0x7f8c2c0693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c0091e0 tx=0x7f8c3c017000 comp rx=0 tx=0).stop 2024-01-26T23:50:08.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.763+0000 7f8c4eccd700 1 -- 172.21.15.46:0/954843757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4815cd40 msgr2=0x7f8c48169010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:08.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.763+0000 7f8c4eccd700 1 --2- 172.21.15.46:0/954843757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4815cd40 0x7f8c48169010 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400d860 tx=0x7f8c34008040 comp rx=0 tx=0).stop 2024-01-26T23:50:08.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.763+0000 7f8c4eccd700 1 -- 172.21.15.46:0/954843757 shutdown_connections 2024-01-26T23:50:08.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.763+0000 7f8c4eccd700 1 --2- 172.21.15.46:0/954843757 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c2c066f30 0x7f8c2c0693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:08.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.763+0000 7f8c4eccd700 1 --2- 172.21.15.46:0/954843757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c4815d110 0x7f8c4806f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:08.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.763+0000 7f8c4eccd700 1 --2- 172.21.15.46:0/954843757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c48067930 0x7f8c480b2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:08.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.763+0000 7f8c4eccd700 1 --2- 172.21.15.46:0/954843757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4815cd40 0x7f8c48169010 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:08.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.763+0000 7f8c4eccd700 1 -- 172.21.15.46:0/954843757 >> 172.21.15.46:0/954843757 conn(0x7f8c480cdf50 msgr2=0x7f8c4815d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:08.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.763+0000 7f8c4eccd700 1 -- 172.21.15.46:0/954843757 shutdown_connections 2024-01-26T23:50:08.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.763+0000 7f8c4eccd700 1 -- 172.21.15.46:0/954843757 wait complete. 2024-01-26T23:50:08.779 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:50:08.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.915+0000 7f533e9cd700 1 Processor -- start 2024-01-26T23:50:08.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.915+0000 7f533e9cd700 1 -- start start 2024-01-26T23:50:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.916+0000 7f533e9cd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53380b0a20 0x7f53380cae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.916+0000 7f533e9cd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f53380c4410 0x7f5338161300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.916+0000 7f533e9cd700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53380cb840 con 0x7f53380cb3d0 2024-01-26T23:50:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.916+0000 7f533e9cd700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5338157c00 con 0x7f53380c4410 2024-01-26T23:50:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.916+0000 7f533e9cd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53380b0df0 con 0x7f53380b0a20 2024-01-26T23:50:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.916+0000 7f53377fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f53380cb3d0 0x7f53380ccc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48992/0 (socket says 172.21.15.46:48992) 2024-01-26T23:50:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.916+0000 7f53377fe700 1 -- 172.21.15.46:0/2127225841 learned_addr learned my addr 172.21.15.46:0/2127225841 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f533cf6a700 1 --2- 172.21.15.46:0/2127225841 >> v2:172.21.15.106:3300/0 conn(0x7f53380c4410 0x7f5338161300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5337fff700 1 --2- 172.21.15.46:0/2127225841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53380b0a20 0x7f53380cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5336ffd700 1 -- 172.21.15.46:0/2127225841 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991706479 0 0) 0x7f53380cb840 con 0x7f53380cb3d0 2024-01-26T23:50:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5336ffd700 1 -- 172.21.15.46:0/2127225841 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5320003570 con 0x7f53380cb3d0 2024-01-26T23:50:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5336ffd700 1 -- 172.21.15.46:0/2127225841 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2479657135 0 0) 0x7f5320003570 con 0x7f53380cb3d0 2024-01-26T23:50:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5336ffd700 1 -- 172.21.15.46:0/2127225841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53380b0a20 msgr2=0x7f53380cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5336ffd700 1 --2- 172.21.15.46:0/2127225841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53380b0a20 0x7f53380cae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5336ffd700 1 -- 172.21.15.46:0/2127225841 >> v2:172.21.15.106:3300/0 conn(0x7f53380c4410 msgr2=0x7f5338161300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5336ffd700 1 --2- 172.21.15.46:0/2127225841 >> v2:172.21.15.106:3300/0 conn(0x7f53380c4410 0x7f5338161300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5336ffd700 1 -- 172.21.15.46:0/2127225841 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53380c9df0 con 0x7f53380cb3d0 2024-01-26T23:50:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5336ffd700 1 -- 172.21.15.46:0/2127225841 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f532c001470 con 0x7f53380cb3d0 2024-01-26T23:50:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5336ffd700 1 -- 172.21.15.46:0/2127225841 >> v1:172.21.15.46:6789/0 conn(0x7f53380cb3d0 legacy=0x7f53380ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5336ffd700 1 --2- 172.21.15.46:0/2127225841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53380b0a20 0x7f5320003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5336ffd700 1 --2- 172.21.15.46:0/2127225841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5320003fe0 0x7f5320004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5336ffd700 1 --2- 172.21.15.46:0/2127225841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5320004d50 0x7f53200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5336ffd700 1 -- 172.21.15.46:0/2127225841 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5338157c00 con 0x7f5320004d50 2024-01-26T23:50:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5336ffd700 1 -- 172.21.15.46:0/2127225841 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53380b0df0 con 0x7f5320003fe0 2024-01-26T23:50:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5336ffd700 1 -- 172.21.15.46:0/2127225841 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532000ba90 con 0x7f53380b0a20 2024-01-26T23:50:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f533cf6a700 1 --2- 172.21.15.46:0/2127225841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5320004d50 0x7f53200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f5337fff700 1 --2- 172.21.15.46:0/2127225841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53380b0a20 0x7f5320003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f53377fe700 1 --2- 172.21.15.46:0/2127225841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5320003fe0 0x7f5320004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f533cf6a700 1 -- 172.21.15.46:0/2127225841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53380b0a20 msgr2=0x7f5320003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f533cf6a700 1 --2- 172.21.15.46:0/2127225841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53380b0a20 0x7f5320003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f533cf6a700 1 -- 172.21.15.46:0/2127225841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5320003fe0 msgr2=0x7f5320004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f533cf6a700 1 --2- 172.21.15.46:0/2127225841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5320003fe0 0x7f5320004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.917+0000 7f533cf6a700 1 -- 172.21.15.46:0/2127225841 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5320003570 con 0x7f5320004d50 2024-01-26T23:50:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.918+0000 7f533cf6a700 1 --2- 172.21.15.46:0/2127225841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5320004d50 0x7f53200051c0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f533000e210 tx=0x7f533000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9067acb74b5c3336 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.918+0000 7f5336ffd700 1 -- 172.21.15.46:0/2127225841 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533000ea60 con 0x7f5320004d50 2024-01-26T23:50:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.918+0000 7f5336ffd700 1 -- 172.21.15.46:0/2127225841 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533000f040 con 0x7f5320004d50 2024-01-26T23:50:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.918+0000 7f5336ffd700 1 -- 172.21.15.46:0/2127225841 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533000ee10 con 0x7f5320004d50 2024-01-26T23:50:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.918+0000 7f533e9cd700 1 -- 172.21.15.46:0/2127225841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5320004d50 msgr2=0x7f53200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.918+0000 7f533e9cd700 1 --2- 172.21.15.46:0/2127225841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5320004d50 0x7f53200051c0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f533000e210 tx=0x7f533000b040 comp rx=0 tx=0).stop 2024-01-26T23:50:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.918+0000 7f533cf6a700 1 -- 172.21.15.46:0/2127225841 reap_dead start 2024-01-26T23:50:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.918+0000 7f533e9cd700 1 -- 172.21.15.46:0/2127225841 shutdown_connections 2024-01-26T23:50:08.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.918+0000 7f533e9cd700 1 -- 172.21.15.46:0/2127225841 >> 172.21.15.46:0/2127225841 conn(0x7f5338097820 msgr2=0x7f5338068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:08.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.918+0000 7f533e9cd700 1 -- 172.21.15.46:0/2127225841 shutdown_connections 2024-01-26T23:50:08.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.919+0000 7f533e9cd700 1 -- 172.21.15.46:0/2127225841 wait complete. 2024-01-26T23:50:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.920+0000 7f533e9cd700 1 Processor -- start 2024-01-26T23:50:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.920+0000 7f533e9cd700 1 -- start start 2024-01-26T23:50:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.921+0000 7f533e9cd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5338154b90 0x7f5338098190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.921+0000 7f533e9cd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5338154f60 0x7f5338099e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.921+0000 7f533e9cd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f533815c560 0x7f533809a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.921+0000 7f533e9cd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5338099930 con 0x7f533815c560 2024-01-26T23:50:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.921+0000 7f533e9cd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533809c150 con 0x7f5338154b90 2024-01-26T23:50:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.921+0000 7f533e9cd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5338098890 con 0x7f5338154f60 2024-01-26T23:50:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.921+0000 7f533cf6a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f533815c560 0x7f533809a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.921+0000 7f533cf6a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f533815c560 0x7f533809a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34668/0 (socket says 172.21.15.46:34668) 2024-01-26T23:50:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.921+0000 7f533cf6a700 1 -- 172.21.15.46:0/3562132018 learned_addr learned my addr 172.21.15.46:0/3562132018 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.921+0000 7f53377fe700 1 --2- 172.21.15.46:0/3562132018 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5338154f60 0x7f5338099e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.921+0000 7f5337fff700 1 --2- 172.21.15.46:0/3562132018 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5338154b90 0x7f5338098190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.921+0000 7f533cf6a700 1 -- 172.21.15.46:0/3562132018 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5338154f60 msgr2=0x7f5338099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.921+0000 7f533cf6a700 1 --2- 172.21.15.46:0/3562132018 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5338154f60 0x7f5338099e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.921+0000 7f533cf6a700 1 -- 172.21.15.46:0/3562132018 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5338154b90 msgr2=0x7f5338098190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.921+0000 7f533cf6a700 1 --2- 172.21.15.46:0/3562132018 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5338154b90 0x7f5338098190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.921+0000 7f533cf6a700 1 -- 172.21.15.46:0/3562132018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5338098b70 con 0x7f533815c560 2024-01-26T23:50:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.921+0000 7f533cf6a700 1 --2- 172.21.15.46:0/3562132018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f533815c560 0x7f533809a370 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f5330016000 tx=0x7f533001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.922+0000 7f531ffff700 1 -- 172.21.15.46:0/3562132018 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533000ee60 con 0x7f533815c560 2024-01-26T23:50:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.922+0000 7f531ffff700 1 -- 172.21.15.46:0/3562132018 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5330022070 con 0x7f533815c560 2024-01-26T23:50:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.922+0000 7f531ffff700 1 -- 172.21.15.46:0/3562132018 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533001e890 con 0x7f533815c560 2024-01-26T23:50:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.922+0000 7f533e9cd700 1 -- 172.21.15.46:0/3562132018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5338098e50 con 0x7f533815c560 2024-01-26T23:50:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.922+0000 7f533e9cd700 1 -- 172.21.15.46:0/3562132018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5338099410 con 0x7f533815c560 2024-01-26T23:50:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.923+0000 7f533e9cd700 1 -- 172.21.15.46:0/3562132018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5338099630 con 0x7f533815c560 2024-01-26T23:50:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.923+0000 7f531ffff700 1 -- 172.21.15.46:0/3562132018 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f533001ecc0 con 0x7f533815c560 2024-01-26T23:50:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.924+0000 7f531ffff700 1 --2- 172.21.15.46:0/3562132018 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5324066f30 0x7f53240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.924+0000 7f531ffff700 1 -- 172.21.15.46:0/3562132018 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f5330024070 con 0x7f533815c560 2024-01-26T23:50:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.924+0000 7f5337fff700 1 --2- 172.21.15.46:0/3562132018 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5324066f30 0x7f53240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:08.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.925+0000 7f5337fff700 1 --2- 172.21.15.46:0/3562132018 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5324066f30 0x7f53240693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5328006e50 tx=0x7f5328009120 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:08.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:08.928+0000 7f531ffff700 1 -- 172.21.15.46:0/3562132018 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5330053350 con 0x7f533815c560 2024-01-26T23:50:09.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.178+0000 7f533e9cd700 1 -- 172.21.15.46:0/3562132018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f533804c710 con 0x7f533815c560 2024-01-26T23:50:09.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.179+0000 7f531ffff700 1 -- 172.21.15.46:0/3562132018 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f5330056970 con 0x7f533815c560 2024-01-26T23:50:09.180 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:50:09.180 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8578,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1889696251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1889696251},{"type":"v1","addr":"172.21.15.106:6834","nonce":1889696251}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/631881132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":631881132},{"type":"v1","addr":"172.21.15.106:6836","nonce":631881132}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1045000694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1045000694},{"type":"v1","addr":"172.21.15.46:6834","nonce":1045000694}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:50:05.176064+0000","modified":"2024-01-26T23:50:08.757937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8871":{"gid":8871,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3419491709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3419491709},{"type":"v1","addr":"172.21.15.46:6835","nonce":3419491709}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2024-01-26T23:50:09.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.181+0000 7f533e9cd700 1 -- 172.21.15.46:0/3562132018 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5324066f30 msgr2=0x7f53240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:09.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.181+0000 7f533e9cd700 1 --2- 172.21.15.46:0/3562132018 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5324066f30 0x7f53240693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5328006e50 tx=0x7f5328009120 comp rx=0 tx=0).stop 2024-01-26T23:50:09.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.181+0000 7f533e9cd700 1 -- 172.21.15.46:0/3562132018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f533815c560 msgr2=0x7f533809a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:09.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.181+0000 7f533e9cd700 1 --2- 172.21.15.46:0/3562132018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f533815c560 0x7f533809a370 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f5330016000 tx=0x7f533001a000 comp rx=0 tx=0).stop 2024-01-26T23:50:09.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.181+0000 7f533e9cd700 1 -- 172.21.15.46:0/3562132018 shutdown_connections 2024-01-26T23:50:09.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.181+0000 7f533e9cd700 1 --2- 172.21.15.46:0/3562132018 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5324066f30 0x7f53240693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.181+0000 7f533e9cd700 1 --2- 172.21.15.46:0/3562132018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f533815c560 0x7f533809a370 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.181+0000 7f533e9cd700 1 --2- 172.21.15.46:0/3562132018 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5338154b90 0x7f5338098190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.181+0000 7f533e9cd700 1 --2- 172.21.15.46:0/3562132018 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5338154f60 0x7f5338099e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.181+0000 7f533e9cd700 1 -- 172.21.15.46:0/3562132018 >> 172.21.15.46:0/3562132018 conn(0x7f5338097820 msgr2=0x7f5338156a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:09.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.181+0000 7f533e9cd700 1 -- 172.21.15.46:0/3562132018 shutdown_connections 2024-01-26T23:50:09.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.181+0000 7f533e9cd700 1 -- 172.21.15.46:0/3562132018 wait complete. 2024-01-26T23:50:09.184 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 78 2024-01-26T23:50:09.196 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-26T23:50:09.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.326+0000 7f98266db700 1 Processor -- start 2024-01-26T23:50:09.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.326+0000 7f98266db700 1 -- start start 2024-01-26T23:50:09.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.326+0000 7f98266db700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f98200ae520 0x7f98200b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:09.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.326+0000 7f98266db700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9820158ae0 con 0x7f98200b3fb0 2024-01-26T23:50:09.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.326+0000 7f98266db700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982015b650 con 0x7f9820157340 2024-01-26T23:50:09.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.327+0000 7f98266db700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98200b43c0 con 0x7f98200ae520 2024-01-26T23:50:09.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.327+0000 7f9824c78700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f98200b3fb0 0x7f9820157710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49006/0 (socket says 172.21.15.46:49006) 2024-01-26T23:50:09.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.327+0000 7f9824c78700 1 -- 172.21.15.46:0/3283009708 learned_addr learned my addr 172.21.15.46:0/3283009708 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:09.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.327+0000 7f981f7fe700 1 --2- 172.21.15.46:0/3283009708 >> v2:172.21.15.106:3300/0 conn(0x7f98200ae520 0x7f98200b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:09.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.327+0000 7f981effd700 1 -- 172.21.15.46:0/3283009708 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883114758 0 0) 0x7f9820158ae0 con 0x7f98200b3fb0 2024-01-26T23:50:09.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.327+0000 7f981effd700 1 -- 172.21.15.46:0/3283009708 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9808003570 con 0x7f98200b3fb0 2024-01-26T23:50:09.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.327+0000 7f981f7fe700 1 -- 172.21.15.46:0/3283009708 >> v1:172.21.15.106:6789/0 conn(0x7f9820157340 legacy=0x7f982015bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:09.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.327+0000 7f981f7fe700 1 -- 172.21.15.46:0/3283009708 >> v1:172.21.15.46:6789/0 conn(0x7f98200b3fb0 legacy=0x7f9820157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:09.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.327+0000 7f981f7fe700 1 -- 172.21.15.46:0/3283009708 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9820158520 con 0x7f98200ae520 2024-01-26T23:50:09.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.327+0000 7f981f7fe700 1 --2- 172.21.15.46:0/3283009708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98200ae520 0x7f98200b3a70 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f98140098e0 tx=0x7f98140091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c65b401e08333ea0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:09.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.328+0000 7f981effd700 1 -- 172.21.15.46:0/3283009708 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981400b640 con 0x7f98200ae520 2024-01-26T23:50:09.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.328+0000 7f981effd700 1 -- 172.21.15.46:0/3283009708 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981400b7a0 con 0x7f98200ae520 2024-01-26T23:50:09.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.328+0000 7f981effd700 1 -- 172.21.15.46:0/3283009708 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981400bb40 con 0x7f98200ae520 2024-01-26T23:50:09.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.328+0000 7f98266db700 1 -- 172.21.15.46:0/3283009708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98200ae520 msgr2=0x7f98200b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:09.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.328+0000 7f98266db700 1 --2- 172.21.15.46:0/3283009708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98200ae520 0x7f98200b3a70 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f98140098e0 tx=0x7f98140091a0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.328+0000 7f98266db700 1 -- 172.21.15.46:0/3283009708 shutdown_connections 2024-01-26T23:50:09.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.328+0000 7f98266db700 1 --2- 172.21.15.46:0/3283009708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98200ae520 0x7f98200b3a70 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.328+0000 7f98266db700 1 -- 172.21.15.46:0/3283009708 >> 172.21.15.46:0/3283009708 conn(0x7f98200ce530 msgr2=0x7f98200ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:09.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.328+0000 7f98266db700 1 -- 172.21.15.46:0/3283009708 shutdown_connections 2024-01-26T23:50:09.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.328+0000 7f98266db700 1 -- 172.21.15.46:0/3283009708 wait complete. 2024-01-26T23:50:09.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.329+0000 7f98266db700 1 Processor -- start 2024-01-26T23:50:09.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.330+0000 7f98266db700 1 -- start start 2024-01-26T23:50:09.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.330+0000 7f98266db700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200ae520 0x7f9820068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:09.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.330+0000 7f98266db700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98200b3fb0 0x7f982016a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:09.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f98266db700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9820157340 0x7f9820154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:09.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f98266db700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982016a7e0 con 0x7f98200ae520 2024-01-26T23:50:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f98266db700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9820069020 con 0x7f98200b3fb0 2024-01-26T23:50:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f98266db700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9820069300 con 0x7f9820157340 2024-01-26T23:50:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f981ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200ae520 0x7f9820068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f981ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200ae520 0x7f9820068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34676/0 (socket says 172.21.15.46:34676) 2024-01-26T23:50:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f981ffff700 1 -- 172.21.15.46:0/9054981 learned_addr learned my addr 172.21.15.46:0/9054981 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f981f7fe700 1 --2- 172.21.15.46:0/9054981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98200b3fb0 0x7f982016a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f9824c78700 1 --2- 172.21.15.46:0/9054981 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9820157340 0x7f9820154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f981ffff700 1 -- 172.21.15.46:0/9054981 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9820157340 msgr2=0x7f9820154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f981ffff700 1 --2- 172.21.15.46:0/9054981 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9820157340 0x7f9820154150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f981ffff700 1 -- 172.21.15.46:0/9054981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98200b3fb0 msgr2=0x7f982016a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f981ffff700 1 --2- 172.21.15.46:0/9054981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98200b3fb0 0x7f982016a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f981ffff700 1 -- 172.21.15.46:0/9054981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9814009420 con 0x7f98200ae520 2024-01-26T23:50:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f981f7fe700 1 --2- 172.21.15.46:0/9054981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98200b3fb0 0x7f982016a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:50:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f981ffff700 1 --2- 172.21.15.46:0/9054981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200ae520 0x7f9820068a40 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f981000c590 tx=0x7f981000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f9807fff700 1 -- 172.21.15.46:0/9054981 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810011590 con 0x7f98200ae520 2024-01-26T23:50:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f9807fff700 1 -- 172.21.15.46:0/9054981 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98100116f0 con 0x7f98200ae520 2024-01-26T23:50:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f98266db700 1 -- 172.21.15.46:0/9054981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98200695e0 con 0x7f98200ae520 2024-01-26T23:50:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f9807fff700 1 -- 172.21.15.46:0/9054981 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810011ab0 con 0x7f98200ae520 2024-01-26T23:50:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.331+0000 7f98266db700 1 -- 172.21.15.46:0/9054981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9820069ba0 con 0x7f98200ae520 2024-01-26T23:50:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.332+0000 7f98266db700 1 -- 172.21.15.46:0/9054981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982004c710 con 0x7f98200ae520 2024-01-26T23:50:09.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.333+0000 7f9807fff700 1 -- 172.21.15.46:0/9054981 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f981001b3b0 con 0x7f98200ae520 2024-01-26T23:50:09.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.333+0000 7f9807fff700 1 --2- 172.21.15.46:0/9054981 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f980c066f30 0x7f980c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:09.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.333+0000 7f9807fff700 1 -- 172.21.15.46:0/9054981 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f9810083e40 con 0x7f98200ae520 2024-01-26T23:50:09.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.334+0000 7f981f7fe700 1 --2- 172.21.15.46:0/9054981 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f980c066f30 0x7f980c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:09.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.334+0000 7f981f7fe700 1 --2- 172.21.15.46:0/9054981 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f980c066f30 0x7f980c0693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f98140093f0 tx=0x7f9814017000 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:09.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.338+0000 7f9807fff700 1 -- 172.21.15.46:0/9054981 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9810050de0 con 0x7f98200ae520 2024-01-26T23:50:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.545+0000 7f98266db700 1 -- 172.21.15.46:0/9054981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9820158800 con 0x7f98200ae520 2024-01-26T23:50:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.546+0000 7f9807fff700 1 -- 172.21.15.46:0/9054981 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) v1 ==== 74+0+16163 (secure 0 0 0) 0x7f9810017070 con 0x7f98200ae520 2024-01-26T23:50:09.547 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:50:09.548 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":158,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-26T23:50:06.738359+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:50:03.691049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-01-26T23:50:04.138930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-26T23:50:09.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.549+0000 7f98266db700 1 -- 172.21.15.46:0/9054981 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f980c066f30 msgr2=0x7f980c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:09.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.549+0000 7f98266db700 1 --2- 172.21.15.46:0/9054981 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f980c066f30 0x7f980c0693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f98140093f0 tx=0x7f9814017000 comp rx=0 tx=0).stop 2024-01-26T23:50:09.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.549+0000 7f98266db700 1 -- 172.21.15.46:0/9054981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200ae520 msgr2=0x7f9820068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.549+0000 7f98266db700 1 --2- 172.21.15.46:0/9054981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200ae520 0x7f9820068a40 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f981000c590 tx=0x7f981000c950 comp rx=0 tx=0).stop 2024-01-26T23:50:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.549+0000 7f98266db700 1 -- 172.21.15.46:0/9054981 shutdown_connections 2024-01-26T23:50:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.549+0000 7f98266db700 1 --2- 172.21.15.46:0/9054981 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f980c066f30 0x7f980c0693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.549+0000 7f98266db700 1 --2- 172.21.15.46:0/9054981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200ae520 0x7f9820068a40 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.549+0000 7f98266db700 1 --2- 172.21.15.46:0/9054981 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9820157340 0x7f9820154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.549+0000 7f98266db700 1 --2- 172.21.15.46:0/9054981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98200b3fb0 0x7f982016a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.549+0000 7f98266db700 1 -- 172.21.15.46:0/9054981 >> 172.21.15.46:0/9054981 conn(0x7f98200ce530 msgr2=0x7f98200b3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.549+0000 7f98266db700 1 -- 172.21.15.46:0/9054981 shutdown_connections 2024-01-26T23:50:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.549+0000 7f98266db700 1 -- 172.21.15.46:0/9054981 wait complete. 2024-01-26T23:50:09.564 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-26T23:50:09.564 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:50:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.695+0000 7fed993e1700 1 Processor -- start 2024-01-26T23:50:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.695+0000 7fed993e1700 1 -- start start 2024-01-26T23:50:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.696+0000 7fed993e1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fed94159960 0x7fed9406bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.696+0000 7fed993e1700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9406c600 con 0x7fed94159590 2024-01-26T23:50:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.696+0000 7fed993e1700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9415dac0 con 0x7fed9406c1f0 2024-01-26T23:50:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.696+0000 7fed993e1700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed9415e350 con 0x7fed94159960 2024-01-26T23:50:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.696+0000 7fed927fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fed94159960 0x7fed9406bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.696+0000 7fed92ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fed94159590 0x7fed9415d3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49012/0 (socket says 172.21.15.46:49012) 2024-01-26T23:50:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.696+0000 7fed92ffd700 1 -- 172.21.15.46:0/4237942025 learned_addr learned my addr 172.21.15.46:0/4237942025 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:09.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.697+0000 7fed91ffb700 1 -- 172.21.15.46:0/4237942025 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677615701 0 0) 0x7fed9406c600 con 0x7fed94159590 2024-01-26T23:50:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.697+0000 7fed91ffb700 1 -- 172.21.15.46:0/4237942025 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed80003570 con 0x7fed94159590 2024-01-26T23:50:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.697+0000 7fed927fc700 1 -- 172.21.15.46:0/4237942025 >> v1:172.21.15.106:6789/0 conn(0x7fed9406c1f0 legacy=0x7fed94069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.697+0000 7fed91ffb700 1 -- 172.21.15.46:0/4237942025 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951101031 0 0) 0x7fed9415dac0 con 0x7fed9406c1f0 2024-01-26T23:50:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.697+0000 7fed927fc700 1 -- 172.21.15.46:0/4237942025 >> v1:172.21.15.46:6789/0 conn(0x7fed94159590 legacy=0x7fed9415d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.697+0000 7fed927fc700 1 -- 172.21.15.46:0/4237942025 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9415dd90 con 0x7fed94159960 2024-01-26T23:50:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.697+0000 7fed927fc700 1 --2- 172.21.15.46:0/4237942025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed94159960 0x7fed9406bcb0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0098e0 tx=0x7fed7c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a0410c819c3cfcc server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.697+0000 7fed91ffb700 1 -- 172.21.15.46:0/4237942025 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c00b640 con 0x7fed94159960 2024-01-26T23:50:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.697+0000 7fed91ffb700 1 -- 172.21.15.46:0/4237942025 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7c00b7a0 con 0x7fed94159960 2024-01-26T23:50:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.697+0000 7fed91ffb700 1 -- 172.21.15.46:0/4237942025 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c00bb40 con 0x7fed94159960 2024-01-26T23:50:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.698+0000 7fed993e1700 1 -- 172.21.15.46:0/4237942025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed94159960 msgr2=0x7fed9406bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.698+0000 7fed993e1700 1 --2- 172.21.15.46:0/4237942025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed94159960 0x7fed9406bcb0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0098e0 tx=0x7fed7c0091a0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.698+0000 7fed993e1700 1 -- 172.21.15.46:0/4237942025 shutdown_connections 2024-01-26T23:50:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.698+0000 7fed993e1700 1 --2- 172.21.15.46:0/4237942025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed94159960 0x7fed9406bcb0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.698+0000 7fed993e1700 1 -- 172.21.15.46:0/4237942025 >> 172.21.15.46:0/4237942025 conn(0x7fed940ce530 msgr2=0x7fed940ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.698+0000 7fed993e1700 1 -- 172.21.15.46:0/4237942025 shutdown_connections 2024-01-26T23:50:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.698+0000 7fed993e1700 1 -- 172.21.15.46:0/4237942025 wait complete. 2024-01-26T23:50:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.699+0000 7fed993e1700 1 Processor -- start 2024-01-26T23:50:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.699+0000 7fed993e1700 1 -- start start 2024-01-26T23:50:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed993e1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed9406c1f0 0x7fed940c3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed993e1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed94159590 0x7fed940c3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed993e1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed94159960 0x7fed940c3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed993e1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9415d1e0 con 0x7fed94159590 2024-01-26T23:50:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed993e1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9415cf00 con 0x7fed94159960 2024-01-26T23:50:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed993e1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed940b1ff0 con 0x7fed9406c1f0 2024-01-26T23:50:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed927fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed94159590 0x7fed940c3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed927fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed94159590 0x7fed940c3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34684/0 (socket says 172.21.15.46:34684) 2024-01-26T23:50:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed927fc700 1 -- 172.21.15.46:0/2050330000 learned_addr learned my addr 172.21.15.46:0/2050330000 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed92ffd700 1 --2- 172.21.15.46:0/2050330000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed9406c1f0 0x7fed940c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed937fe700 1 --2- 172.21.15.46:0/2050330000 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed94159960 0x7fed940c3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed927fc700 1 -- 172.21.15.46:0/2050330000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed9406c1f0 msgr2=0x7fed940c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed927fc700 1 --2- 172.21.15.46:0/2050330000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed9406c1f0 0x7fed940c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed927fc700 1 -- 172.21.15.46:0/2050330000 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed94159960 msgr2=0x7fed940c3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed927fc700 1 --2- 172.21.15.46:0/2050330000 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed94159960 0x7fed940c3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed927fc700 1 -- 172.21.15.46:0/2050330000 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7c009420 con 0x7fed94159590 2024-01-26T23:50:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed937fe700 1 --2- 172.21.15.46:0/2050330000 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed94159960 0x7fed940c3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:50:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed92ffd700 1 --2- 172.21.15.46:0/2050330000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed9406c1f0 0x7fed940c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:50:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.700+0000 7fed927fc700 1 --2- 172.21.15.46:0/2050330000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed94159590 0x7fed940c3650 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0093f0 tx=0x7fed7c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.701+0000 7fed8affd700 1 -- 172.21.15.46:0/2050330000 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c01f070 con 0x7fed94159590 2024-01-26T23:50:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.701+0000 7fed8affd700 1 -- 172.21.15.46:0/2050330000 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7c006af0 con 0x7fed94159590 2024-01-26T23:50:09.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.701+0000 7fed993e1700 1 -- 172.21.15.46:0/2050330000 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed940b22d0 con 0x7fed94159590 2024-01-26T23:50:09.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.701+0000 7fed8affd700 1 -- 172.21.15.46:0/2050330000 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c00b910 con 0x7fed94159590 2024-01-26T23:50:09.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.701+0000 7fed993e1700 1 -- 172.21.15.46:0/2050330000 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9415a220 con 0x7fed94159590 2024-01-26T23:50:09.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.702+0000 7fed993e1700 1 -- 172.21.15.46:0/2050330000 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed940b2770 con 0x7fed94159590 2024-01-26T23:50:09.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.702+0000 7fed8affd700 1 -- 172.21.15.46:0/2050330000 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fed7c023070 con 0x7fed94159590 2024-01-26T23:50:09.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.703+0000 7fed8affd700 1 --2- 172.21.15.46:0/2050330000 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fed74066f30 0x7fed740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:09.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.703+0000 7fed8affd700 1 -- 172.21.15.46:0/2050330000 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fed7c085ab0 con 0x7fed94159590 2024-01-26T23:50:09.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.703+0000 7fed92ffd700 1 --2- 172.21.15.46:0/2050330000 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fed74066f30 0x7fed740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:09.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.703+0000 7fed92ffd700 1 --2- 172.21.15.46:0/2050330000 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fed74066f30 0x7fed740693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fed840058d0 tx=0x7fed84009040 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:09.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.707+0000 7fed8affd700 1 -- 172.21.15.46:0/2050330000 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fed7c052a50 con 0x7fed94159590 2024-01-26T23:50:09.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.908+0000 7fed993e1700 1 -- 172.21.15.46:0/2050330000 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fed9404c710 con 0x7fed74066f30 2024-01-26T23:50:09.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.917+0000 7fed8affd700 1 -- 172.21.15.46:0/2050330000 <== mgr.8395 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289834 (secure 0 0 0) 0x7fed9404c710 con 0x7fed74066f30 2024-01-26T23:50:09.922 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:50:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.920+0000 7fed993e1700 1 -- 172.21.15.46:0/2050330000 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fed74066f30 msgr2=0x7fed740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.920+0000 7fed993e1700 1 --2- 172.21.15.46:0/2050330000 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fed74066f30 0x7fed740693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fed840058d0 tx=0x7fed84009040 comp rx=0 tx=0).stop 2024-01-26T23:50:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.920+0000 7fed993e1700 1 -- 172.21.15.46:0/2050330000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed94159590 msgr2=0x7fed940c3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.920+0000 7fed993e1700 1 --2- 172.21.15.46:0/2050330000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed94159590 0x7fed940c3650 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0093f0 tx=0x7fed7c017000 comp rx=0 tx=0).stop 2024-01-26T23:50:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.920+0000 7fed993e1700 1 -- 172.21.15.46:0/2050330000 shutdown_connections 2024-01-26T23:50:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.920+0000 7fed993e1700 1 --2- 172.21.15.46:0/2050330000 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fed74066f30 0x7fed740693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.920+0000 7fed993e1700 1 --2- 172.21.15.46:0/2050330000 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed94159960 0x7fed940c3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.920+0000 7fed993e1700 1 --2- 172.21.15.46:0/2050330000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed9406c1f0 0x7fed940c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.920+0000 7fed993e1700 1 --2- 172.21.15.46:0/2050330000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed94159590 0x7fed940c3650 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.920+0000 7fed993e1700 1 -- 172.21.15.46:0/2050330000 >> 172.21.15.46:0/2050330000 conn(0x7fed940ce530 msgr2=0x7fed94158920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.920+0000 7fed993e1700 1 -- 172.21.15.46:0/2050330000 shutdown_connections 2024-01-26T23:50:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:09.920+0000 7fed993e1700 1 -- 172.21.15.46:0/2050330000 wait complete. 2024-01-26T23:50:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:50:09.935 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-26T23:50:09.699670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1355,"num_read_kb":2726,"num_write":444,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8644,"ondisk_log_size":8644,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2452264,"kb_used_data":16616,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752522456,"statfs":{"total":773094113280,"available":770582994944,"internally_reserved":0,"allocated":17014784,"data_stored":9334684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.965340"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.766020+0000","last_change":"2024-01-26T23:50:05.760313+0000","last_active":"2024-01-26T23:50:05.766020+0000","last_peered":"2024-01-26T23:50:05.766020+0000","last_clean":"2024-01-26T23:50:05.766020+0000","last_became_active":"2024-01-26T23:50:05.760052+0000","last_became_peered":"2024-01-26T23:50:05.760052+0000","last_unstale":"2024-01-26T23:50:05.766020+0000","last_undegraded":"2024-01-26T23:50:05.766020+0000","last_fullsized":"2024-01-26T23:50:05.766020+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:58:56.234846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.755777+0000","last_change":"2024-01-26T23:50:05.754956+0000","last_active":"2024-01-26T23:50:05.755777+0000","last_peered":"2024-01-26T23:50:05.755777+0000","last_clean":"2024-01-26T23:50:05.755777+0000","last_became_active":"2024-01-26T23:50:05.754116+0000","last_became_peered":"2024-01-26T23:50:05.754116+0000","last_unstale":"2024-01-26T23:50:05.755777+0000","last_undegraded":"2024-01-26T23:50:05.755777+0000","last_fullsized":"2024-01-26T23:50:05.755777+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:58:34.814255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.769060+0000","last_change":"2024-01-26T23:50:05.763223+0000","last_active":"2024-01-26T23:50:05.769060+0000","last_peered":"2024-01-26T23:50:05.769060+0000","last_clean":"2024-01-26T23:50:05.769060+0000","last_became_active":"2024-01-26T23:50:05.762945+0000","last_became_peered":"2024-01-26T23:50:05.762945+0000","last_unstale":"2024-01-26T23:50:05.769060+0000","last_undegraded":"2024-01-26T23:50:05.769060+0000","last_fullsized":"2024-01-26T23:50:05.769060+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:27.332267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.756108+0000","last_change":"2024-01-26T23:50:05.751228+0000","last_active":"2024-01-26T23:50:05.756108+0000","last_peered":"2024-01-26T23:50:05.756108+0000","last_clean":"2024-01-26T23:50:05.756108+0000","last_became_active":"2024-01-26T23:50:05.750935+0000","last_became_peered":"2024-01-26T23:50:05.750935+0000","last_unstale":"2024-01-26T23:50:05.756108+0000","last_undegraded":"2024-01-26T23:50:05.756108+0000","last_fullsized":"2024-01-26T23:50:05.756108+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:55.611362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.761381+0000","last_change":"2024-01-26T23:50:05.756891+0000","last_active":"2024-01-26T23:50:05.761381+0000","last_peered":"2024-01-26T23:50:05.761381+0000","last_clean":"2024-01-26T23:50:05.761381+0000","last_became_active":"2024-01-26T23:50:05.756548+0000","last_became_peered":"2024-01-26T23:50:05.756548+0000","last_unstale":"2024-01-26T23:50:05.761381+0000","last_undegraded":"2024-01-26T23:50:05.761381+0000","last_fullsized":"2024-01-26T23:50:05.761381+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:27.332267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.763598+0000","last_change":"2024-01-26T23:50:05.756560+0000","last_active":"2024-01-26T23:50:05.763598+0000","last_peered":"2024-01-26T23:50:05.763598+0000","last_clean":"2024-01-26T23:50:05.763598+0000","last_became_active":"2024-01-26T23:50:05.756250+0000","last_became_peered":"2024-01-26T23:50:05.756250+0000","last_unstale":"2024-01-26T23:50:05.763598+0000","last_undegraded":"2024-01-26T23:50:05.763598+0000","last_fullsized":"2024-01-26T23:50:05.763598+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:21:26.665456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.773013+0000","last_change":"2024-01-26T23:50:05.764338+0000","last_active":"2024-01-26T23:50:05.773013+0000","last_peered":"2024-01-26T23:50:05.773013+0000","last_clean":"2024-01-26T23:50:05.773013+0000","last_became_active":"2024-01-26T23:50:05.763719+0000","last_became_peered":"2024-01-26T23:50:05.763719+0000","last_unstale":"2024-01-26T23:50:05.773013+0000","last_undegraded":"2024-01-26T23:50:05.773013+0000","last_fullsized":"2024-01-26T23:50:05.773013+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:34:45.453927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765775+0000","last_change":"2024-01-26T23:50:05.761592+0000","last_active":"2024-01-26T23:50:05.765775+0000","last_peered":"2024-01-26T23:50:05.765775+0000","last_clean":"2024-01-26T23:50:05.765775+0000","last_became_active":"2024-01-26T23:50:05.761388+0000","last_became_peered":"2024-01-26T23:50:05.761388+0000","last_unstale":"2024-01-26T23:50:05.765775+0000","last_undegraded":"2024-01-26T23:50:05.765775+0000","last_fullsized":"2024-01-26T23:50:05.765775+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:08:15.330869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.750472+0000","last_change":"2024-01-26T23:50:05.747564+0000","last_active":"2024-01-26T23:50:05.750472+0000","last_peered":"2024-01-26T23:50:05.750472+0000","last_clean":"2024-01-26T23:50:05.750472+0000","last_became_active":"2024-01-26T23:50:05.746771+0000","last_became_peered":"2024-01-26T23:50:05.746771+0000","last_unstale":"2024-01-26T23:50:05.750472+0000","last_undegraded":"2024-01-26T23:50:05.750472+0000","last_fullsized":"2024-01-26T23:50:05.750472+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:34:45.453927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.763687+0000","last_change":"2024-01-26T23:50:05.756949+0000","last_active":"2024-01-26T23:50:05.763687+0000","last_peered":"2024-01-26T23:50:05.763687+0000","last_clean":"2024-01-26T23:50:05.763687+0000","last_became_active":"2024-01-26T23:50:05.756661+0000","last_became_peered":"2024-01-26T23:50:05.756661+0000","last_unstale":"2024-01-26T23:50:05.763687+0000","last_undegraded":"2024-01-26T23:50:05.763687+0000","last_fullsized":"2024-01-26T23:50:05.763687+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:23:07.892082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765366+0000","last_change":"2024-01-26T23:50:05.760145+0000","last_active":"2024-01-26T23:50:05.765366+0000","last_peered":"2024-01-26T23:50:05.765366+0000","last_clean":"2024-01-26T23:50:05.765366+0000","last_became_active":"2024-01-26T23:50:05.758472+0000","last_became_peered":"2024-01-26T23:50:05.758472+0000","last_unstale":"2024-01-26T23:50:05.765366+0000","last_undegraded":"2024-01-26T23:50:05.765366+0000","last_fullsized":"2024-01-26T23:50:05.765366+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:12.031661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765918+0000","last_change":"2024-01-26T23:50:05.763183+0000","last_active":"2024-01-26T23:50:05.765918+0000","last_peered":"2024-01-26T23:50:05.765918+0000","last_clean":"2024-01-26T23:50:05.765918+0000","last_became_active":"2024-01-26T23:50:05.762802+0000","last_became_peered":"2024-01-26T23:50:05.762802+0000","last_unstale":"2024-01-26T23:50:05.765918+0000","last_undegraded":"2024-01-26T23:50:05.765918+0000","last_fullsized":"2024-01-26T23:50:05.765918+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:36:57.110071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.763905+0000","last_change":"2024-01-26T23:50:05.759112+0000","last_active":"2024-01-26T23:50:05.763905+0000","last_peered":"2024-01-26T23:50:05.763905+0000","last_clean":"2024-01-26T23:50:05.763905+0000","last_became_active":"2024-01-26T23:50:05.758822+0000","last_became_peered":"2024-01-26T23:50:05.758822+0000","last_unstale":"2024-01-26T23:50:05.763905+0000","last_undegraded":"2024-01-26T23:50:05.763905+0000","last_fullsized":"2024-01-26T23:50:05.763905+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:43:40.458165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765719+0000","last_change":"2024-01-26T23:50:05.759128+0000","last_active":"2024-01-26T23:50:05.765719+0000","last_peered":"2024-01-26T23:50:05.765719+0000","last_clean":"2024-01-26T23:50:05.765719+0000","last_became_active":"2024-01-26T23:50:05.758822+0000","last_became_peered":"2024-01-26T23:50:05.758822+0000","last_unstale":"2024-01-26T23:50:05.765719+0000","last_undegraded":"2024-01-26T23:50:05.765719+0000","last_fullsized":"2024-01-26T23:50:05.765719+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:50:26.914006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.769682+0000","last_change":"2024-01-26T23:50:05.766363+0000","last_active":"2024-01-26T23:50:05.769682+0000","last_peered":"2024-01-26T23:50:05.769682+0000","last_clean":"2024-01-26T23:50:05.769682+0000","last_became_active":"2024-01-26T23:50:05.766020+0000","last_became_peered":"2024-01-26T23:50:05.766020+0000","last_unstale":"2024-01-26T23:50:05.769682+0000","last_undegraded":"2024-01-26T23:50:05.769682+0000","last_fullsized":"2024-01-26T23:50:05.769682+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:42:57.962862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.764252+0000","last_change":"2024-01-26T23:50:05.751942+0000","last_active":"2024-01-26T23:50:05.764252+0000","last_peered":"2024-01-26T23:50:05.764252+0000","last_clean":"2024-01-26T23:50:05.764252+0000","last_became_active":"2024-01-26T23:50:05.751263+0000","last_became_peered":"2024-01-26T23:50:05.751263+0000","last_unstale":"2024-01-26T23:50:05.764252+0000","last_undegraded":"2024-01-26T23:50:05.764252+0000","last_fullsized":"2024-01-26T23:50:05.764252+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:13:16.831804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.766507+0000","last_change":"2024-01-26T23:50:05.752720+0000","last_active":"2024-01-26T23:50:05.766507+0000","last_peered":"2024-01-26T23:50:05.766507+0000","last_clean":"2024-01-26T23:50:05.766507+0000","last_became_active":"2024-01-26T23:50:05.752438+0000","last_became_peered":"2024-01-26T23:50:05.752438+0000","last_unstale":"2024-01-26T23:50:05.766507+0000","last_undegraded":"2024-01-26T23:50:05.766507+0000","last_fullsized":"2024-01-26T23:50:05.766507+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:04:10.945311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.761238+0000","last_change":"2024-01-26T23:50:05.755319+0000","last_active":"2024-01-26T23:50:05.761238+0000","last_peered":"2024-01-26T23:50:05.761238+0000","last_clean":"2024-01-26T23:50:05.761238+0000","last_became_active":"2024-01-26T23:50:05.755019+0000","last_became_peered":"2024-01-26T23:50:05.755019+0000","last_unstale":"2024-01-26T23:50:05.761238+0000","last_undegraded":"2024-01-26T23:50:05.761238+0000","last_fullsized":"2024-01-26T23:50:05.761238+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:34:45.453927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.773811+0000","last_change":"2024-01-26T23:50:05.764428+0000","last_active":"2024-01-26T23:50:05.773811+0000","last_peered":"2024-01-26T23:50:05.773811+0000","last_clean":"2024-01-26T23:50:05.773811+0000","last_became_active":"2024-01-26T23:50:05.763705+0000","last_became_peered":"2024-01-26T23:50:05.763705+0000","last_unstale":"2024-01-26T23:50:05.773811+0000","last_undegraded":"2024-01-26T23:50:05.773811+0000","last_fullsized":"2024-01-26T23:50:05.773811+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:27.332267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.761866+0000","last_change":"2024-01-26T23:50:05.757645+0000","last_active":"2024-01-26T23:50:05.761866+0000","last_peered":"2024-01-26T23:50:05.761866+0000","last_clean":"2024-01-26T23:50:05.761866+0000","last_became_active":"2024-01-26T23:50:05.757307+0000","last_became_peered":"2024-01-26T23:50:05.757307+0000","last_unstale":"2024-01-26T23:50:05.761866+0000","last_undegraded":"2024-01-26T23:50:05.761866+0000","last_fullsized":"2024-01-26T23:50:05.761866+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:13:16.831804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.755719+0000","last_change":"2024-01-26T23:50:05.751431+0000","last_active":"2024-01-26T23:50:05.755719+0000","last_peered":"2024-01-26T23:50:05.755719+0000","last_clean":"2024-01-26T23:50:05.755719+0000","last_became_active":"2024-01-26T23:50:05.751172+0000","last_became_peered":"2024-01-26T23:50:05.751172+0000","last_unstale":"2024-01-26T23:50:05.755719+0000","last_undegraded":"2024-01-26T23:50:05.755719+0000","last_fullsized":"2024-01-26T23:50:05.755719+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:40:49.040365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.762328+0000","last_change":"2024-01-26T23:50:05.756791+0000","last_active":"2024-01-26T23:50:05.762328+0000","last_peered":"2024-01-26T23:50:05.762328+0000","last_clean":"2024-01-26T23:50:05.762328+0000","last_became_active":"2024-01-26T23:50:05.756511+0000","last_became_peered":"2024-01-26T23:50:05.756511+0000","last_unstale":"2024-01-26T23:50:05.762328+0000","last_undegraded":"2024-01-26T23:50:05.762328+0000","last_fullsized":"2024-01-26T23:50:05.762328+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:57:56.258786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.764066+0000","last_change":"2024-01-26T23:50:05.758011+0000","last_active":"2024-01-26T23:50:05.764066+0000","last_peered":"2024-01-26T23:50:05.764066+0000","last_clean":"2024-01-26T23:50:05.764066+0000","last_became_active":"2024-01-26T23:50:05.757745+0000","last_became_peered":"2024-01-26T23:50:05.757745+0000","last_unstale":"2024-01-26T23:50:05.764066+0000","last_undegraded":"2024-01-26T23:50:05.764066+0000","last_fullsized":"2024-01-26T23:50:05.764066+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:19:26.227100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.750295+0000","last_change":"2024-01-26T23:50:05.748082+0000","last_active":"2024-01-26T23:50:05.750295+0000","last_peered":"2024-01-26T23:50:05.750295+0000","last_clean":"2024-01-26T23:50:05.750295+0000","last_became_active":"2024-01-26T23:50:05.747771+0000","last_became_peered":"2024-01-26T23:50:05.747771+0000","last_unstale":"2024-01-26T23:50:05.750295+0000","last_undegraded":"2024-01-26T23:50:05.750295+0000","last_fullsized":"2024-01-26T23:50:05.750295+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:27.332267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768843+0000","last_change":"2024-01-26T23:50:05.757097+0000","last_active":"2024-01-26T23:50:05.768843+0000","last_peered":"2024-01-26T23:50:05.768843+0000","last_clean":"2024-01-26T23:50:05.768843+0000","last_became_active":"2024-01-26T23:50:05.756761+0000","last_became_peered":"2024-01-26T23:50:05.756761+0000","last_unstale":"2024-01-26T23:50:05.768843+0000","last_undegraded":"2024-01-26T23:50:05.768843+0000","last_fullsized":"2024-01-26T23:50:05.768843+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:41:22.684297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768852+0000","last_change":"2024-01-26T23:50:05.764848+0000","last_active":"2024-01-26T23:50:05.768852+0000","last_peered":"2024-01-26T23:50:05.768852+0000","last_clean":"2024-01-26T23:50:05.768852+0000","last_became_active":"2024-01-26T23:50:05.764589+0000","last_became_peered":"2024-01-26T23:50:05.764589+0000","last_unstale":"2024-01-26T23:50:05.768852+0000","last_undegraded":"2024-01-26T23:50:05.768852+0000","last_fullsized":"2024-01-26T23:50:05.768852+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:34:45.453927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.773951+0000","last_change":"2024-01-26T23:50:05.768020+0000","last_active":"2024-01-26T23:50:05.773951+0000","last_peered":"2024-01-26T23:50:05.773951+0000","last_clean":"2024-01-26T23:50:05.773951+0000","last_became_active":"2024-01-26T23:50:05.766719+0000","last_became_peered":"2024-01-26T23:50:05.766719+0000","last_unstale":"2024-01-26T23:50:05.773951+0000","last_undegraded":"2024-01-26T23:50:05.773951+0000","last_fullsized":"2024-01-26T23:50:05.773951+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:59.234067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.761021+0000","last_change":"2024-01-26T23:50:05.752890+0000","last_active":"2024-01-26T23:50:05.761021+0000","last_peered":"2024-01-26T23:50:05.761021+0000","last_clean":"2024-01-26T23:50:05.761021+0000","last_became_active":"2024-01-26T23:50:05.752608+0000","last_became_peered":"2024-01-26T23:50:05.752608+0000","last_unstale":"2024-01-26T23:50:05.761021+0000","last_undegraded":"2024-01-26T23:50:05.761021+0000","last_fullsized":"2024-01-26T23:50:05.761021+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:28:16.609431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768851+0000","last_change":"2024-01-26T23:50:05.764753+0000","last_active":"2024-01-26T23:50:05.768851+0000","last_peered":"2024-01-26T23:50:05.768851+0000","last_clean":"2024-01-26T23:50:05.768851+0000","last_became_active":"2024-01-26T23:50:05.764494+0000","last_became_peered":"2024-01-26T23:50:05.764494+0000","last_unstale":"2024-01-26T23:50:05.768851+0000","last_undegraded":"2024-01-26T23:50:05.768851+0000","last_fullsized":"2024-01-26T23:50:05.768851+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:31.322920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.764031+0000","last_change":"2024-01-26T23:50:05.755009+0000","last_active":"2024-01-26T23:50:05.764031+0000","last_peered":"2024-01-26T23:50:05.764031+0000","last_clean":"2024-01-26T23:50:05.764031+0000","last_became_active":"2024-01-26T23:50:05.754763+0000","last_became_peered":"2024-01-26T23:50:05.754763+0000","last_unstale":"2024-01-26T23:50:05.764031+0000","last_undegraded":"2024-01-26T23:50:05.764031+0000","last_fullsized":"2024-01-26T23:50:05.764031+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:12:48.929340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768347+0000","last_change":"2024-01-26T23:50:05.765022+0000","last_active":"2024-01-26T23:50:05.768347+0000","last_peered":"2024-01-26T23:50:05.768347+0000","last_clean":"2024-01-26T23:50:05.768347+0000","last_became_active":"2024-01-26T23:50:05.764766+0000","last_became_peered":"2024-01-26T23:50:05.764766+0000","last_unstale":"2024-01-26T23:50:05.768347+0000","last_undegraded":"2024-01-26T23:50:05.768347+0000","last_fullsized":"2024-01-26T23:50:05.768347+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:21:26.665456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.760393+0000","last_change":"2024-01-26T23:50:05.753319+0000","last_active":"2024-01-26T23:50:05.760393+0000","last_peered":"2024-01-26T23:50:05.760393+0000","last_clean":"2024-01-26T23:50:05.760393+0000","last_became_active":"2024-01-26T23:50:05.753032+0000","last_became_peered":"2024-01-26T23:50:05.753032+0000","last_unstale":"2024-01-26T23:50:05.760393+0000","last_undegraded":"2024-01-26T23:50:05.760393+0000","last_fullsized":"2024-01-26T23:50:05.760393+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:09:57.835401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768567+0000","last_change":"2024-01-26T23:50:05.765445+0000","last_active":"2024-01-26T23:50:05.768567+0000","last_peered":"2024-01-26T23:50:05.768567+0000","last_clean":"2024-01-26T23:50:05.768567+0000","last_became_active":"2024-01-26T23:50:05.765130+0000","last_became_peered":"2024-01-26T23:50:05.765130+0000","last_unstale":"2024-01-26T23:50:05.768567+0000","last_undegraded":"2024-01-26T23:50:05.768567+0000","last_fullsized":"2024-01-26T23:50:05.768567+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:43:40.458165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"157'4","reported_seq":14,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.775671+0000","last_change":"2024-01-26T23:50:04.760218+0000","last_active":"2024-01-26T23:50:05.775671+0000","last_peered":"2024-01-26T23:50:05.775671+0000","last_clean":"2024-01-26T23:50:05.775671+0000","last_became_active":"2024-01-26T23:50:04.759915+0000","last_became_peered":"2024-01-26T23:50:04.759915+0000","last_unstale":"2024-01-26T23:50:05.775671+0000","last_undegraded":"2024-01-26T23:50:05.775671+0000","last_fullsized":"2024-01-26T23:50:05.775671+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:52.388343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.762445+0000","last_change":"2024-01-26T23:50:05.755855+0000","last_active":"2024-01-26T23:50:05.762445+0000","last_peered":"2024-01-26T23:50:05.762445+0000","last_clean":"2024-01-26T23:50:05.762445+0000","last_became_active":"2024-01-26T23:50:05.755573+0000","last_became_peered":"2024-01-26T23:50:05.755573+0000","last_unstale":"2024-01-26T23:50:05.762445+0000","last_undegraded":"2024-01-26T23:50:05.762445+0000","last_fullsized":"2024-01-26T23:50:05.762445+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:01:40.211350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"157'1","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.772369+0000","last_change":"2024-01-26T23:50:04.758971+0000","last_active":"2024-01-26T23:50:05.772369+0000","last_peered":"2024-01-26T23:50:05.772369+0000","last_clean":"2024-01-26T23:50:05.772369+0000","last_became_active":"2024-01-26T23:50:04.758003+0000","last_became_peered":"2024-01-26T23:50:04.758003+0000","last_unstale":"2024-01-26T23:50:05.772369+0000","last_undegraded":"2024-01-26T23:50:05.772369+0000","last_fullsized":"2024-01-26T23:50:05.772369+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:54:10.044522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.764131+0000","last_change":"2024-01-26T23:50:05.758246+0000","last_active":"2024-01-26T23:50:05.764131+0000","last_peered":"2024-01-26T23:50:05.764131+0000","last_clean":"2024-01-26T23:50:05.764131+0000","last_became_active":"2024-01-26T23:50:05.757889+0000","last_became_peered":"2024-01-26T23:50:05.757889+0000","last_unstale":"2024-01-26T23:50:05.764131+0000","last_undegraded":"2024-01-26T23:50:05.764131+0000","last_fullsized":"2024-01-26T23:50:05.764131+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:22:24.153814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"157'2","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.775316+0000","last_change":"2024-01-26T23:50:04.754647+0000","last_active":"2024-01-26T23:50:05.775316+0000","last_peered":"2024-01-26T23:50:05.775316+0000","last_clean":"2024-01-26T23:50:05.775316+0000","last_became_active":"2024-01-26T23:50:04.754246+0000","last_became_peered":"2024-01-26T23:50:04.754246+0000","last_unstale":"2024-01-26T23:50:05.775316+0000","last_undegraded":"2024-01-26T23:50:05.775316+0000","last_fullsized":"2024-01-26T23:50:05.775316+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:19:25.221567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.762401+0000","last_change":"2024-01-26T23:50:05.758309+0000","last_active":"2024-01-26T23:50:05.762401+0000","last_peered":"2024-01-26T23:50:05.762401+0000","last_clean":"2024-01-26T23:50:05.762401+0000","last_became_active":"2024-01-26T23:50:05.758049+0000","last_became_peered":"2024-01-26T23:50:05.758049+0000","last_unstale":"2024-01-26T23:50:05.762401+0000","last_undegraded":"2024-01-26T23:50:05.762401+0000","last_fullsized":"2024-01-26T23:50:05.762401+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:04:52.726605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"157'2","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.773414+0000","last_change":"2024-01-26T23:50:04.755194+0000","last_active":"2024-01-26T23:50:05.773414+0000","last_peered":"2024-01-26T23:50:05.773414+0000","last_clean":"2024-01-26T23:50:05.773414+0000","last_became_active":"2024-01-26T23:50:04.754928+0000","last_became_peered":"2024-01-26T23:50:04.754928+0000","last_unstale":"2024-01-26T23:50:05.773414+0000","last_undegraded":"2024-01-26T23:50:05.773414+0000","last_fullsized":"2024-01-26T23:50:05.773414+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:58:33.808722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765735+0000","last_change":"2024-01-26T23:50:05.762523+0000","last_active":"2024-01-26T23:50:05.765735+0000","last_peered":"2024-01-26T23:50:05.765735+0000","last_clean":"2024-01-26T23:50:05.765735+0000","last_became_active":"2024-01-26T23:50:05.762268+0000","last_became_peered":"2024-01-26T23:50:05.762268+0000","last_unstale":"2024-01-26T23:50:05.765735+0000","last_undegraded":"2024-01-26T23:50:05.765735+0000","last_fullsized":"2024-01-26T23:50:05.765735+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:13:16.831804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"157'1","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.774264+0000","last_change":"2024-01-26T23:50:04.764076+0000","last_active":"2024-01-26T23:50:05.774264+0000","last_peered":"2024-01-26T23:50:05.774264+0000","last_clean":"2024-01-26T23:50:05.774264+0000","last_became_active":"2024-01-26T23:50:04.763803+0000","last_became_peered":"2024-01-26T23:50:04.763803+0000","last_unstale":"2024-01-26T23:50:05.774264+0000","last_undegraded":"2024-01-26T23:50:05.774264+0000","last_fullsized":"2024-01-26T23:50:05.774264+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:41:21.678763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765714+0000","last_change":"2024-01-26T23:50:05.761793+0000","last_active":"2024-01-26T23:50:05.765714+0000","last_peered":"2024-01-26T23:50:05.765714+0000","last_clean":"2024-01-26T23:50:05.765714+0000","last_became_active":"2024-01-26T23:50:05.761077+0000","last_became_peered":"2024-01-26T23:50:05.761077+0000","last_unstale":"2024-01-26T23:50:05.765714+0000","last_undegraded":"2024-01-26T23:50:05.765714+0000","last_fullsized":"2024-01-26T23:50:05.765714+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:04:10.945311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"157'1","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.772637+0000","last_change":"2024-01-26T23:50:04.756494+0000","last_active":"2024-01-26T23:50:05.772637+0000","last_peered":"2024-01-26T23:50:05.772637+0000","last_clean":"2024-01-26T23:50:05.772637+0000","last_became_active":"2024-01-26T23:50:04.756129+0000","last_became_peered":"2024-01-26T23:50:04.756129+0000","last_unstale":"2024-01-26T23:50:05.772637+0000","last_undegraded":"2024-01-26T23:50:05.772637+0000","last_fullsized":"2024-01-26T23:50:05.772637+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:47:07.429822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768375+0000","last_change":"2024-01-26T23:50:05.760552+0000","last_active":"2024-01-26T23:50:05.768375+0000","last_peered":"2024-01-26T23:50:05.768375+0000","last_clean":"2024-01-26T23:50:05.768375+0000","last_became_active":"2024-01-26T23:50:05.758808+0000","last_became_peered":"2024-01-26T23:50:05.758808+0000","last_unstale":"2024-01-26T23:50:05.768375+0000","last_undegraded":"2024-01-26T23:50:05.768375+0000","last_fullsized":"2024-01-26T23:50:05.768375+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:12.635232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.740594+0000","last_change":"2024-01-26T23:50:04.758300+0000","last_active":"2024-01-26T23:50:05.740594+0000","last_peered":"2024-01-26T23:50:05.740594+0000","last_clean":"2024-01-26T23:50:05.740594+0000","last_became_active":"2024-01-26T23:50:04.757776+0000","last_became_peered":"2024-01-26T23:50:04.757776+0000","last_unstale":"2024-01-26T23:50:05.740594+0000","last_undegraded":"2024-01-26T23:50:05.740594+0000","last_fullsized":"2024-01-26T23:50:05.740594+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:11.026128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.773892+0000","last_change":"2024-01-26T23:50:05.773293+0000","last_active":"2024-01-26T23:50:05.773892+0000","last_peered":"2024-01-26T23:50:05.773892+0000","last_clean":"2024-01-26T23:50:05.773892+0000","last_became_active":"2024-01-26T23:50:05.773013+0000","last_became_peered":"2024-01-26T23:50:05.773013+0000","last_unstale":"2024-01-26T23:50:05.773892+0000","last_undegraded":"2024-01-26T23:50:05.773892+0000","last_fullsized":"2024-01-26T23:50:05.773892+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:16.574844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.752154+0000","last_change":"2024-01-26T23:50:04.762129+0000","last_active":"2024-01-26T23:50:05.752154+0000","last_peered":"2024-01-26T23:50:05.752154+0000","last_clean":"2024-01-26T23:50:05.752154+0000","last_became_active":"2024-01-26T23:50:04.761854+0000","last_became_peered":"2024-01-26T23:50:04.761854+0000","last_unstale":"2024-01-26T23:50:05.752154+0000","last_undegraded":"2024-01-26T23:50:05.752154+0000","last_fullsized":"2024-01-26T23:50:05.752154+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:22:23.148281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.762384+0000","last_change":"2024-01-26T23:50:05.758923+0000","last_active":"2024-01-26T23:50:05.762384+0000","last_peered":"2024-01-26T23:50:05.762384+0000","last_clean":"2024-01-26T23:50:05.762384+0000","last_became_active":"2024-01-26T23:50:05.758142+0000","last_became_peered":"2024-01-26T23:50:05.758142+0000","last_unstale":"2024-01-26T23:50:05.762384+0000","last_undegraded":"2024-01-26T23:50:05.762384+0000","last_fullsized":"2024-01-26T23:50:05.762384+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:55:29.374881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.745153+0000","last_change":"2024-01-26T23:50:04.758455+0000","last_active":"2024-01-26T23:50:05.745153+0000","last_peered":"2024-01-26T23:50:05.745153+0000","last_clean":"2024-01-26T23:50:05.745153+0000","last_became_active":"2024-01-26T23:50:04.758181+0000","last_became_peered":"2024-01-26T23:50:04.758181+0000","last_unstale":"2024-01-26T23:50:05.745153+0000","last_undegraded":"2024-01-26T23:50:05.745153+0000","last_fullsized":"2024-01-26T23:50:05.745153+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:45.611620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.763742+0000","last_change":"2024-01-26T23:50:05.753753+0000","last_active":"2024-01-26T23:50:05.763742+0000","last_peered":"2024-01-26T23:50:05.763742+0000","last_clean":"2024-01-26T23:50:05.763742+0000","last_became_active":"2024-01-26T23:50:05.751971+0000","last_became_peered":"2024-01-26T23:50:05.751971+0000","last_unstale":"2024-01-26T23:50:05.763742+0000","last_undegraded":"2024-01-26T23:50:05.763742+0000","last_fullsized":"2024-01-26T23:50:05.763742+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:45:58.463554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.750180+0000","last_change":"2024-01-26T23:50:04.757736+0000","last_active":"2024-01-26T23:50:05.750180+0000","last_peered":"2024-01-26T23:50:05.750180+0000","last_clean":"2024-01-26T23:50:05.750180+0000","last_became_active":"2024-01-26T23:50:04.757423+0000","last_became_peered":"2024-01-26T23:50:04.757423+0000","last_unstale":"2024-01-26T23:50:05.750180+0000","last_undegraded":"2024-01-26T23:50:05.750180+0000","last_fullsized":"2024-01-26T23:50:05.750180+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:09.475540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.766247+0000","last_change":"2024-01-26T23:50:05.756866+0000","last_active":"2024-01-26T23:50:05.766247+0000","last_peered":"2024-01-26T23:50:05.766247+0000","last_clean":"2024-01-26T23:50:05.766247+0000","last_became_active":"2024-01-26T23:50:05.756532+0000","last_became_peered":"2024-01-26T23:50:05.756532+0000","last_unstale":"2024-01-26T23:50:05.766247+0000","last_undegraded":"2024-01-26T23:50:05.766247+0000","last_fullsized":"2024-01-26T23:50:05.766247+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:59.234067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"157'3","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.774311+0000","last_change":"2024-01-26T23:50:04.757627+0000","last_active":"2024-01-26T23:50:05.774311+0000","last_peered":"2024-01-26T23:50:05.774311+0000","last_clean":"2024-01-26T23:50:05.774311+0000","last_became_active":"2024-01-26T23:50:04.757332+0000","last_became_peered":"2024-01-26T23:50:04.757332+0000","last_unstale":"2024-01-26T23:50:05.774311+0000","last_undegraded":"2024-01-26T23:50:05.774311+0000","last_fullsized":"2024-01-26T23:50:05.774311+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:50:00.006714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768869+0000","last_change":"2024-01-26T23:50:05.767003+0000","last_active":"2024-01-26T23:50:05.768869+0000","last_peered":"2024-01-26T23:50:05.768869+0000","last_clean":"2024-01-26T23:50:05.768869+0000","last_became_active":"2024-01-26T23:50:05.766724+0000","last_became_peered":"2024-01-26T23:50:05.766724+0000","last_unstale":"2024-01-26T23:50:05.768869+0000","last_undegraded":"2024-01-26T23:50:05.768869+0000","last_fullsized":"2024-01-26T23:50:05.768869+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:12.635232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"157'6","reported_seq":16,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.776719+0000","last_change":"2024-01-26T23:50:04.759904+0000","last_active":"2024-01-26T23:50:05.776719+0000","last_peered":"2024-01-26T23:50:05.776719+0000","last_clean":"2024-01-26T23:50:05.776719+0000","last_became_active":"2024-01-26T23:50:04.759625+0000","last_became_peered":"2024-01-26T23:50:04.759625+0000","last_unstale":"2024-01-26T23:50:05.776719+0000","last_undegraded":"2024-01-26T23:50:05.776719+0000","last_fullsized":"2024-01-26T23:50:05.776719+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:52.388343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765023+0000","last_change":"2024-01-26T23:50:05.760077+0000","last_active":"2024-01-26T23:50:05.765023+0000","last_peered":"2024-01-26T23:50:05.765023+0000","last_clean":"2024-01-26T23:50:05.765023+0000","last_became_active":"2024-01-26T23:50:05.759695+0000","last_became_peered":"2024-01-26T23:50:05.759695+0000","last_unstale":"2024-01-26T23:50:05.765023+0000","last_undegraded":"2024-01-26T23:50:05.765023+0000","last_fullsized":"2024-01-26T23:50:05.765023+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:19:17.243302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"157'2","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.775305+0000","last_change":"2024-01-26T23:50:04.755696+0000","last_active":"2024-01-26T23:50:05.775305+0000","last_peered":"2024-01-26T23:50:05.775305+0000","last_clean":"2024-01-26T23:50:05.775305+0000","last_became_active":"2024-01-26T23:50:04.755379+0000","last_became_peered":"2024-01-26T23:50:04.755379+0000","last_unstale":"2024-01-26T23:50:05.775305+0000","last_undegraded":"2024-01-26T23:50:05.775305+0000","last_fullsized":"2024-01-26T23:50:05.775305+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:47:11.376776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765677+0000","last_change":"2024-01-26T23:50:05.757271+0000","last_active":"2024-01-26T23:50:05.765677+0000","last_peered":"2024-01-26T23:50:05.765677+0000","last_clean":"2024-01-26T23:50:05.765677+0000","last_became_active":"2024-01-26T23:50:05.757007+0000","last_became_peered":"2024-01-26T23:50:05.757007+0000","last_unstale":"2024-01-26T23:50:05.765677+0000","last_undegraded":"2024-01-26T23:50:05.765677+0000","last_fullsized":"2024-01-26T23:50:05.765677+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:12:48.929340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.748074+0000","last_change":"2024-01-26T23:50:04.754865+0000","last_active":"2024-01-26T23:50:05.748074+0000","last_peered":"2024-01-26T23:50:05.748074+0000","last_clean":"2024-01-26T23:50:05.748074+0000","last_became_active":"2024-01-26T23:50:04.754600+0000","last_became_peered":"2024-01-26T23:50:04.754600+0000","last_unstale":"2024-01-26T23:50:05.748074+0000","last_undegraded":"2024-01-26T23:50:05.748074+0000","last_fullsized":"2024-01-26T23:50:05.748074+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:54.605829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.760928+0000","last_change":"2024-01-26T23:50:05.751464+0000","last_active":"2024-01-26T23:50:05.760928+0000","last_peered":"2024-01-26T23:50:05.760928+0000","last_clean":"2024-01-26T23:50:05.760928+0000","last_became_active":"2024-01-26T23:50:05.751120+0000","last_became_peered":"2024-01-26T23:50:05.751120+0000","last_unstale":"2024-01-26T23:50:05.760928+0000","last_undegraded":"2024-01-26T23:50:05.760928+0000","last_fullsized":"2024-01-26T23:50:05.760928+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:43:40.458165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"157'8","reported_seq":18,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.775360+0000","last_change":"2024-01-26T23:50:04.756977+0000","last_active":"2024-01-26T23:50:05.775360+0000","last_peered":"2024-01-26T23:50:05.775360+0000","last_clean":"2024-01-26T23:50:05.775360+0000","last_became_active":"2024-01-26T23:50:04.756685+0000","last_became_peered":"2024-01-26T23:50:04.756685+0000","last_unstale":"2024-01-26T23:50:05.775360+0000","last_undegraded":"2024-01-26T23:50:05.775360+0000","last_fullsized":"2024-01-26T23:50:05.775360+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:10:03.148195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768774+0000","last_change":"2024-01-26T23:50:05.763325+0000","last_active":"2024-01-26T23:50:05.768774+0000","last_peered":"2024-01-26T23:50:05.768774+0000","last_clean":"2024-01-26T23:50:05.768774+0000","last_became_active":"2024-01-26T23:50:05.763052+0000","last_became_peered":"2024-01-26T23:50:05.763052+0000","last_unstale":"2024-01-26T23:50:05.768774+0000","last_undegraded":"2024-01-26T23:50:05.768774+0000","last_fullsized":"2024-01-26T23:50:05.768774+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:21:30.381652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"157'1","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.772218+0000","last_change":"2024-01-26T23:50:04.747392+0000","last_active":"2024-01-26T23:50:05.772218+0000","last_peered":"2024-01-26T23:50:05.772218+0000","last_clean":"2024-01-26T23:50:05.772218+0000","last_became_active":"2024-01-26T23:50:04.747098+0000","last_became_peered":"2024-01-26T23:50:04.747098+0000","last_unstale":"2024-01-26T23:50:05.772218+0000","last_undegraded":"2024-01-26T23:50:05.772218+0000","last_fullsized":"2024-01-26T23:50:05.772218+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:51:42.125836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.773521+0000","last_change":"2024-01-26T23:50:05.770940+0000","last_active":"2024-01-26T23:50:05.773521+0000","last_peered":"2024-01-26T23:50:05.773521+0000","last_clean":"2024-01-26T23:50:05.773521+0000","last_became_active":"2024-01-26T23:50:05.770673+0000","last_became_peered":"2024-01-26T23:50:05.770673+0000","last_unstale":"2024-01-26T23:50:05.773521+0000","last_undegraded":"2024-01-26T23:50:05.773521+0000","last_fullsized":"2024-01-26T23:50:05.773521+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:12.635232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"157'2","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.774001+0000","last_change":"2024-01-26T23:50:04.758668+0000","last_active":"2024-01-26T23:50:05.774001+0000","last_peered":"2024-01-26T23:50:05.774001+0000","last_clean":"2024-01-26T23:50:05.774001+0000","last_became_active":"2024-01-26T23:50:04.758377+0000","last_became_peered":"2024-01-26T23:50:04.758377+0000","last_unstale":"2024-01-26T23:50:05.774001+0000","last_undegraded":"2024-01-26T23:50:05.774001+0000","last_fullsized":"2024-01-26T23:50:05.774001+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:19:25.221567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.773534+0000","last_change":"2024-01-26T23:50:05.768208+0000","last_active":"2024-01-26T23:50:05.773534+0000","last_peered":"2024-01-26T23:50:05.773534+0000","last_clean":"2024-01-26T23:50:05.773534+0000","last_became_active":"2024-01-26T23:50:05.767161+0000","last_became_peered":"2024-01-26T23:50:05.767161+0000","last_unstale":"2024-01-26T23:50:05.773534+0000","last_undegraded":"2024-01-26T23:50:05.773534+0000","last_fullsized":"2024-01-26T23:50:05.773534+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:31.322920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"157'1","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.774114+0000","last_change":"2024-01-26T23:50:04.758083+0000","last_active":"2024-01-26T23:50:05.774114+0000","last_peered":"2024-01-26T23:50:05.774114+0000","last_clean":"2024-01-26T23:50:05.774114+0000","last_became_active":"2024-01-26T23:50:04.757807+0000","last_became_peered":"2024-01-26T23:50:04.757807+0000","last_unstale":"2024-01-26T23:50:05.774114+0000","last_undegraded":"2024-01-26T23:50:05.774114+0000","last_fullsized":"2024-01-26T23:50:05.774114+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:10.357938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765524+0000","last_change":"2024-01-26T23:50:05.761833+0000","last_active":"2024-01-26T23:50:05.765524+0000","last_peered":"2024-01-26T23:50:05.765524+0000","last_clean":"2024-01-26T23:50:05.765524+0000","last_became_active":"2024-01-26T23:50:05.761121+0000","last_became_peered":"2024-01-26T23:50:05.761121+0000","last_unstale":"2024-01-26T23:50:05.765524+0000","last_undegraded":"2024-01-26T23:50:05.765524+0000","last_fullsized":"2024-01-26T23:50:05.765524+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:45:06.875004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.745077+0000","last_change":"2024-01-26T23:50:04.759073+0000","last_active":"2024-01-26T23:50:05.745077+0000","last_peered":"2024-01-26T23:50:05.745077+0000","last_clean":"2024-01-26T23:50:05.745077+0000","last_became_active":"2024-01-26T23:50:04.758779+0000","last_became_peered":"2024-01-26T23:50:04.758779+0000","last_unstale":"2024-01-26T23:50:05.745077+0000","last_undegraded":"2024-01-26T23:50:05.745077+0000","last_fullsized":"2024-01-26T23:50:05.745077+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:10:03.148195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.761219+0000","last_change":"2024-01-26T23:50:05.754281+0000","last_active":"2024-01-26T23:50:05.761219+0000","last_peered":"2024-01-26T23:50:05.761219+0000","last_clean":"2024-01-26T23:50:05.761219+0000","last_became_active":"2024-01-26T23:50:05.753970+0000","last_became_peered":"2024-01-26T23:50:05.753970+0000","last_unstale":"2024-01-26T23:50:05.761219+0000","last_undegraded":"2024-01-26T23:50:05.761219+0000","last_fullsized":"2024-01-26T23:50:05.761219+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:25:41.882518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"157'2","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.772208+0000","last_change":"2024-01-26T23:50:04.754231+0000","last_active":"2024-01-26T23:50:05.772208+0000","last_peered":"2024-01-26T23:50:05.772208+0000","last_clean":"2024-01-26T23:50:05.772208+0000","last_became_active":"2024-01-26T23:50:04.753393+0000","last_became_peered":"2024-01-26T23:50:04.753393+0000","last_unstale":"2024-01-26T23:50:05.772208+0000","last_undegraded":"2024-01-26T23:50:05.772208+0000","last_fullsized":"2024-01-26T23:50:05.772208+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:12:47.923807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765402+0000","last_change":"2024-01-26T23:50:05.758368+0000","last_active":"2024-01-26T23:50:05.765402+0000","last_peered":"2024-01-26T23:50:05.765402+0000","last_clean":"2024-01-26T23:50:05.765402+0000","last_became_active":"2024-01-26T23:50:05.758034+0000","last_became_peered":"2024-01-26T23:50:05.758034+0000","last_unstale":"2024-01-26T23:50:05.765402+0000","last_undegraded":"2024-01-26T23:50:05.765402+0000","last_fullsized":"2024-01-26T23:50:05.765402+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:20:06.258716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.748517+0000","last_change":"2024-01-26T23:50:04.754377+0000","last_active":"2024-01-26T23:50:05.748517+0000","last_peered":"2024-01-26T23:50:05.748517+0000","last_clean":"2024-01-26T23:50:05.748517+0000","last_became_active":"2024-01-26T23:50:04.753680+0000","last_became_peered":"2024-01-26T23:50:04.753680+0000","last_unstale":"2024-01-26T23:50:05.748517+0000","last_undegraded":"2024-01-26T23:50:05.748517+0000","last_fullsized":"2024-01-26T23:50:05.748517+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:15.214318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768350+0000","last_change":"2024-01-26T23:50:05.760353+0000","last_active":"2024-01-26T23:50:05.768350+0000","last_peered":"2024-01-26T23:50:05.768350+0000","last_clean":"2024-01-26T23:50:05.768350+0000","last_became_active":"2024-01-26T23:50:05.758430+0000","last_became_peered":"2024-01-26T23:50:05.758430+0000","last_unstale":"2024-01-26T23:50:05.768350+0000","last_undegraded":"2024-01-26T23:50:05.768350+0000","last_fullsized":"2024-01-26T23:50:05.768350+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:21:30.381652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.754999+0000","last_change":"2024-01-26T23:50:04.757707+0000","last_active":"2024-01-26T23:50:05.754999+0000","last_peered":"2024-01-26T23:50:05.754999+0000","last_clean":"2024-01-26T23:50:05.754999+0000","last_became_active":"2024-01-26T23:50:04.757423+0000","last_became_peered":"2024-01-26T23:50:04.757423+0000","last_unstale":"2024-01-26T23:50:05.754999+0000","last_undegraded":"2024-01-26T23:50:05.754999+0000","last_fullsized":"2024-01-26T23:50:05.754999+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:22:23.148281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.764202+0000","last_change":"2024-01-26T23:50:05.759105+0000","last_active":"2024-01-26T23:50:05.764202+0000","last_peered":"2024-01-26T23:50:05.764202+0000","last_clean":"2024-01-26T23:50:05.764202+0000","last_became_active":"2024-01-26T23:50:05.758813+0000","last_became_peered":"2024-01-26T23:50:05.758813+0000","last_unstale":"2024-01-26T23:50:05.764202+0000","last_undegraded":"2024-01-26T23:50:05.764202+0000","last_fullsized":"2024-01-26T23:50:05.764202+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:11.363471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.741299+0000","last_change":"2024-01-26T23:50:04.753988+0000","last_active":"2024-01-26T23:50:05.741299+0000","last_peered":"2024-01-26T23:50:05.741299+0000","last_clean":"2024-01-26T23:50:05.741299+0000","last_became_active":"2024-01-26T23:50:04.753703+0000","last_became_peered":"2024-01-26T23:50:04.753703+0000","last_unstale":"2024-01-26T23:50:05.741299+0000","last_undegraded":"2024-01-26T23:50:05.741299+0000","last_fullsized":"2024-01-26T23:50:05.741299+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:58:33.808722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.767110+0000","last_change":"2024-01-26T23:50:05.757932+0000","last_active":"2024-01-26T23:50:05.767110+0000","last_peered":"2024-01-26T23:50:05.767110+0000","last_clean":"2024-01-26T23:50:05.767110+0000","last_became_active":"2024-01-26T23:50:05.757334+0000","last_became_peered":"2024-01-26T23:50:05.757334+0000","last_unstale":"2024-01-26T23:50:05.767110+0000","last_undegraded":"2024-01-26T23:50:05.767110+0000","last_fullsized":"2024-01-26T23:50:05.767110+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:21:26.665456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"157'2","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.775403+0000","last_change":"2024-01-26T23:50:04.759684+0000","last_active":"2024-01-26T23:50:05.775403+0000","last_peered":"2024-01-26T23:50:05.775403+0000","last_clean":"2024-01-26T23:50:05.775403+0000","last_became_active":"2024-01-26T23:50:04.758634+0000","last_became_peered":"2024-01-26T23:50:04.758634+0000","last_unstale":"2024-01-26T23:50:05.775403+0000","last_undegraded":"2024-01-26T23:50:05.775403+0000","last_fullsized":"2024-01-26T23:50:05.775403+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:46.321004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.766709+0000","last_change":"2024-01-26T23:50:05.760420+0000","last_active":"2024-01-26T23:50:05.766709+0000","last_peered":"2024-01-26T23:50:05.766709+0000","last_clean":"2024-01-26T23:50:05.766709+0000","last_became_active":"2024-01-26T23:50:05.760141+0000","last_became_peered":"2024-01-26T23:50:05.760141+0000","last_unstale":"2024-01-26T23:50:05.766709+0000","last_undegraded":"2024-01-26T23:50:05.766709+0000","last_fullsized":"2024-01-26T23:50:05.766709+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:16.574844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.753651+0000","last_change":"2024-01-26T23:50:04.754921+0000","last_active":"2024-01-26T23:50:05.753651+0000","last_peered":"2024-01-26T23:50:05.753651+0000","last_clean":"2024-01-26T23:50:05.753651+0000","last_became_active":"2024-01-26T23:50:04.754650+0000","last_became_peered":"2024-01-26T23:50:04.754650+0000","last_unstale":"2024-01-26T23:50:05.753651+0000","last_undegraded":"2024-01-26T23:50:05.753651+0000","last_fullsized":"2024-01-26T23:50:05.753651+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:30.317387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.769745+0000","last_change":"2024-01-26T23:50:05.758521+0000","last_active":"2024-01-26T23:50:05.769745+0000","last_peered":"2024-01-26T23:50:05.769745+0000","last_clean":"2024-01-26T23:50:05.769745+0000","last_became_active":"2024-01-26T23:50:05.758266+0000","last_became_peered":"2024-01-26T23:50:05.758266+0000","last_unstale":"2024-01-26T23:50:05.769745+0000","last_undegraded":"2024-01-26T23:50:05.769745+0000","last_fullsized":"2024-01-26T23:50:05.769745+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:21:30.381652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.752131+0000","last_change":"2024-01-26T23:50:04.758950+0000","last_active":"2024-01-26T23:50:05.752131+0000","last_peered":"2024-01-26T23:50:05.752131+0000","last_clean":"2024-01-26T23:50:05.752131+0000","last_became_active":"2024-01-26T23:50:04.758691+0000","last_became_peered":"2024-01-26T23:50:04.758691+0000","last_unstale":"2024-01-26T23:50:05.752131+0000","last_undegraded":"2024-01-26T23:50:05.752131+0000","last_fullsized":"2024-01-26T23:50:05.752131+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:23:08.813586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765500+0000","last_change":"2024-01-26T23:50:05.761639+0000","last_active":"2024-01-26T23:50:05.765500+0000","last_peered":"2024-01-26T23:50:05.765500+0000","last_clean":"2024-01-26T23:50:05.765500+0000","last_became_active":"2024-01-26T23:50:05.760100+0000","last_became_peered":"2024-01-26T23:50:05.760100+0000","last_unstale":"2024-01-26T23:50:05.765500+0000","last_undegraded":"2024-01-26T23:50:05.765500+0000","last_fullsized":"2024-01-26T23:50:05.765500+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:13:16.808150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.740884+0000","last_change":"2024-01-26T23:50:04.748757+0000","last_active":"2024-01-26T23:50:05.740884+0000","last_peered":"2024-01-26T23:50:05.740884+0000","last_clean":"2024-01-26T23:50:05.740884+0000","last_became_active":"2024-01-26T23:50:04.748487+0000","last_became_peered":"2024-01-26T23:50:04.748487+0000","last_unstale":"2024-01-26T23:50:05.740884+0000","last_undegraded":"2024-01-26T23:50:05.740884+0000","last_fullsized":"2024-01-26T23:50:05.740884+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:54:10.044522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.763753+0000","last_change":"2024-01-26T23:50:05.757686+0000","last_active":"2024-01-26T23:50:05.763753+0000","last_peered":"2024-01-26T23:50:05.763753+0000","last_clean":"2024-01-26T23:50:05.763753+0000","last_became_active":"2024-01-26T23:50:05.757374+0000","last_became_peered":"2024-01-26T23:50:05.757374+0000","last_unstale":"2024-01-26T23:50:05.763753+0000","last_undegraded":"2024-01-26T23:50:05.763753+0000","last_fullsized":"2024-01-26T23:50:05.763753+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:25:41.882518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.744659+0000","last_change":"2024-01-26T23:50:04.759150+0000","last_active":"2024-01-26T23:50:05.744659+0000","last_peered":"2024-01-26T23:50:05.744659+0000","last_clean":"2024-01-26T23:50:05.744659+0000","last_became_active":"2024-01-26T23:50:04.758883+0000","last_became_peered":"2024-01-26T23:50:04.758883+0000","last_unstale":"2024-01-26T23:50:05.744659+0000","last_undegraded":"2024-01-26T23:50:05.744659+0000","last_fullsized":"2024-01-26T23:50:05.744659+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:49:45.035222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.763807+0000","last_change":"2024-01-26T23:50:05.758466+0000","last_active":"2024-01-26T23:50:05.763807+0000","last_peered":"2024-01-26T23:50:05.763807+0000","last_clean":"2024-01-26T23:50:05.763807+0000","last_became_active":"2024-01-26T23:50:05.758164+0000","last_became_peered":"2024-01-26T23:50:05.758164+0000","last_unstale":"2024-01-26T23:50:05.763807+0000","last_undegraded":"2024-01-26T23:50:05.763807+0000","last_fullsized":"2024-01-26T23:50:05.763807+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:08:15.330869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.749370+0000","last_change":"2024-01-26T23:50:04.754886+0000","last_active":"2024-01-26T23:50:05.749370+0000","last_peered":"2024-01-26T23:50:05.749370+0000","last_clean":"2024-01-26T23:50:05.749370+0000","last_became_active":"2024-01-26T23:50:04.754622+0000","last_became_peered":"2024-01-26T23:50:04.754622+0000","last_unstale":"2024-01-26T23:50:05.749370+0000","last_undegraded":"2024-01-26T23:50:05.749370+0000","last_fullsized":"2024-01-26T23:50:05.749370+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:10.357938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.763742+0000","last_change":"2024-01-26T23:50:05.758903+0000","last_active":"2024-01-26T23:50:05.763742+0000","last_peered":"2024-01-26T23:50:05.763742+0000","last_clean":"2024-01-26T23:50:05.763742+0000","last_became_active":"2024-01-26T23:50:05.758621+0000","last_became_peered":"2024-01-26T23:50:05.758621+0000","last_unstale":"2024-01-26T23:50:05.763742+0000","last_undegraded":"2024-01-26T23:50:05.763742+0000","last_fullsized":"2024-01-26T23:50:05.763742+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:21:30.381652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.740736+0000","last_change":"2024-01-26T23:50:04.747999+0000","last_active":"2024-01-26T23:50:05.740736+0000","last_peered":"2024-01-26T23:50:05.740736+0000","last_clean":"2024-01-26T23:50:05.740736+0000","last_became_active":"2024-01-26T23:50:04.747724+0000","last_became_peered":"2024-01-26T23:50:04.747724+0000","last_unstale":"2024-01-26T23:50:05.740736+0000","last_undegraded":"2024-01-26T23:50:05.740736+0000","last_fullsized":"2024-01-26T23:50:05.740736+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:50.221794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765610+0000","last_change":"2024-01-26T23:50:05.757871+0000","last_active":"2024-01-26T23:50:05.765610+0000","last_peered":"2024-01-26T23:50:05.765610+0000","last_clean":"2024-01-26T23:50:05.765610+0000","last_became_active":"2024-01-26T23:50:05.757619+0000","last_became_peered":"2024-01-26T23:50:05.757619+0000","last_unstale":"2024-01-26T23:50:05.765610+0000","last_undegraded":"2024-01-26T23:50:05.765610+0000","last_fullsized":"2024-01-26T23:50:05.765610+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:45:58.463554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.747306+0000","last_change":"2024-01-26T23:50:04.760619+0000","last_active":"2024-01-26T23:50:05.747306+0000","last_peered":"2024-01-26T23:50:05.747306+0000","last_clean":"2024-01-26T23:50:05.747306+0000","last_became_active":"2024-01-26T23:50:04.759526+0000","last_became_peered":"2024-01-26T23:50:04.759526+0000","last_unstale":"2024-01-26T23:50:05.747306+0000","last_undegraded":"2024-01-26T23:50:05.747306+0000","last_fullsized":"2024-01-26T23:50:05.747306+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:50.221794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.769081+0000","last_change":"2024-01-26T23:50:05.763347+0000","last_active":"2024-01-26T23:50:05.769081+0000","last_peered":"2024-01-26T23:50:05.769081+0000","last_clean":"2024-01-26T23:50:05.769081+0000","last_became_active":"2024-01-26T23:50:05.763075+0000","last_became_peered":"2024-01-26T23:50:05.763075+0000","last_unstale":"2024-01-26T23:50:05.769081+0000","last_undegraded":"2024-01-26T23:50:05.769081+0000","last_fullsized":"2024-01-26T23:50:05.769081+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:45:58.463554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.749257+0000","last_change":"2024-01-26T23:50:04.759360+0000","last_active":"2024-01-26T23:50:05.749257+0000","last_peered":"2024-01-26T23:50:05.749257+0000","last_clean":"2024-01-26T23:50:05.749257+0000","last_became_active":"2024-01-26T23:50:04.758570+0000","last_became_peered":"2024-01-26T23:50:04.758570+0000","last_unstale":"2024-01-26T23:50:05.749257+0000","last_undegraded":"2024-01-26T23:50:05.749257+0000","last_fullsized":"2024-01-26T23:50:05.749257+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:54.605829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"149'8606","reported_seq":9194,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.748691+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:50:05.748691+0000","last_peered":"2024-01-26T23:50:05.748691+0000","last_clean":"2024-01-26T23:50:05.748691+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:50:05.748691+0000","last_undegraded":"2024-01-26T23:50:05.748691+0000","last_fullsized":"2024-01-26T23:50:05.748691+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8606,"log_dups_size":0,"ondisk_log_size":8606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1355,"num_read_kb":2726,"num_write":400,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1355,"num_read_kb":2726,"num_write":400,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8606,"ondisk_log_size":8606,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575091,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312096,"kb_used_data":2520,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059744,"statfs":{"total":96636764160,"available":96317177856,"internally_reserved":0,"allocated":2580480,"data_stored":1610661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51500000000000001,"15min":0.56200000000000006},"min":{"1min":0.40699999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.74199999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52700000000000002,"15min":0.57599999999999996},"min":{"1min":0.35899999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67200000000000004,"5min":0.82499999999999996,"15min":0.83999999999999997},"last":0.56299999999999994}]},{"osd":1,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54000000000000004,"15min":0.57399999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.77600000000000002,"5min":0.77800000000000002,"15min":0.82899999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55100000000000005,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.70399999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.57199999999999995},"min":{"1min":0.38700000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69199999999999995,"5min":0.80000000000000004,"15min":0.81699999999999995},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.38900000000000001,"5min":0.30199999999999999,"15min":0.27400000000000002},"max":{"1min":0.746,"5min":0.81999999999999995,"15min":0.88600000000000001},"last":0.625}]},{"osd":3,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54700000000000004,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.377,"15min":0.377},"max":{"1min":0.70399999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57499999999999996,"15min":0.59699999999999998},"min":{"1min":0.34399999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77900000000000003,"5min":0.86199999999999999,"15min":0.86399999999999999},"last":0.66100000000000003}]},{"osd":4,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.438,"15min":0.45200000000000001},"min":{"1min":0.34599999999999997,"5min":0.252,"15min":0.247},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.93899999999999995},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.45300000000000001,"15min":0.47399999999999998},"min":{"1min":0.32400000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67300000000000004,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.51900000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47599999999999998,"15min":0.48699999999999999},"min":{"1min":0.30499999999999999,"5min":0.27200000000000002,"15min":0.22800000000000001},"max":{"1min":0.749,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.504,"15min":0.52300000000000002},"min":{"1min":0.35799999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73399999999999999,"5min":0.74199999999999999,"15min":0.90200000000000002},"last":0.60899999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53100000000000003,"15min":0.54800000000000004},"min":{"1min":0.41699999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76400000000000001,"5min":0.78700000000000003,"15min":0.95899999999999996},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.52700000000000002,"15min":0.54000000000000004},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75900000000000001,"5min":0.79500000000000004,"15min":0.85199999999999998},"last":0.64000000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575091,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303316,"kb_used_data":1932,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068524,"statfs":{"total":96636764160,"available":96326168576,"internally_reserved":0,"allocated":1978368,"data_stored":1019311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52900000000000003,"15min":0.56499999999999995},"min":{"1min":0.40300000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.423,"15min":1.423},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.57899999999999996},"min":{"1min":0.36099999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.91100000000000003},"last":0.57899999999999996}]},{"osd":1,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51700000000000002,"15min":0.55800000000000005},"min":{"1min":0.32200000000000001,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.58399999999999996},"min":{"1min":0.38200000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.66800000000000004,"5min":1.498,"15min":1.498},"last":0.53900000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52800000000000002,"15min":0.56999999999999995},"min":{"1min":0.38100000000000001,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.85799999999999998},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52600000000000002,"15min":0.57099999999999995},"min":{"1min":0.377,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69299999999999995,"5min":0.77000000000000002,"15min":0.82799999999999996},"last":0.70199999999999996}]},{"osd":3,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57199999999999995,"15min":0.59099999999999997},"min":{"1min":0.34699999999999998,"5min":0.33800000000000002,"15min":0.29999999999999999},"max":{"1min":0.746,"5min":1.595,"15min":1.595},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56899999999999995,"15min":0.60599999999999998},"min":{"1min":0.45600000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.751,"5min":1.369,"15min":1.369},"last":0.81100000000000005}]},{"osd":4,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.443,"15min":0.46200000000000002},"min":{"1min":0.33800000000000002,"5min":0.245,"15min":0.22},"max":{"1min":0.67000000000000004,"5min":1.556,"15min":1.556},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.47399999999999998,"15min":0.48199999999999998},"min":{"1min":0.33200000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.86199999999999999},"last":0.72899999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45500000000000002,"15min":0.47099999999999997},"min":{"1min":0.32700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67900000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.51700000000000002,"15min":0.52000000000000002},"min":{"1min":0.30499999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.70999999999999996,"5min":1.47,"15min":1.47},"last":0.622}]},{"osd":7,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.5,"15min":0.51800000000000002},"min":{"1min":0.40100000000000002,"5min":0.26700000000000002,"15min":0.246},"max":{"1min":0.80200000000000005,"5min":0.81799999999999995,"15min":0.85899999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50900000000000001,"15min":0.52400000000000002},"min":{"1min":0.32300000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.80400000000000005},"last":0.747}]}]},{"osd":1,"up_from":13,"seq":55834575091,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307408,"kb_used_data":1928,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064432,"statfs":{"total":96636764160,"available":96321978368,"internally_reserved":0,"allocated":1974272,"data_stored":1018737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41199999999999998,"15min":0.40600000000000003},"min":{"1min":0.28100000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.72599999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.41999999999999998,"15min":0.439},"min":{"1min":0.28100000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.61499999999999999,"5min":0.78600000000000003,"15min":1.337},"last":0.36499999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.39700000000000002,"15min":0.40899999999999997},"min":{"1min":0.28699999999999998,"5min":0.254,"15min":0.23599999999999999},"max":{"1min":0.66300000000000003,"5min":0.68799999999999994,"15min":0.73299999999999998},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.46300000000000002,"15min":0.47199999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59299999999999997,"5min":0.74099999999999999,"15min":0.77300000000000002},"last":0.52700000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51300000000000001,"15min":0.51400000000000001},"min":{"1min":0.38400000000000001,"5min":0.34399999999999997,"15min":0.33200000000000002},"max":{"1min":0.73499999999999999,"5min":0.78900000000000003,"15min":1.782},"last":0.436},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.32900000000000001,"5min":0.32000000000000001,"15min":0.26600000000000001},"max":{"1min":0.70199999999999996,"5min":0.71499999999999997,"15min":1.379},"last":0.44800000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60699999999999998,"15min":0.64200000000000002},"min":{"1min":0.45100000000000001,"5min":0.40799999999999997,"15min":0.40200000000000002},"max":{"1min":0.85199999999999998,"5min":0.89200000000000002,"15min":1.0820000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.64500000000000002,"15min":0.65700000000000003},"min":{"1min":0.49199999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.66700000000000004}]},{"osd":5,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.64000000000000001,"15min":0.66800000000000004},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45000000000000001},"max":{"1min":0.86499999999999999,"5min":0.91100000000000003,"15min":1.208},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.63400000000000001,"15min":0.65900000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.42499999999999999},"max":{"1min":0.85099999999999998,"5min":0.96399999999999997,"15min":1.018},"last":0.65800000000000003}]},{"osd":6,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.63800000000000001,"15min":0.67000000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.42599999999999999},"max":{"1min":0.97399999999999998,"5min":1.022,"15min":1.2529999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.65000000000000002,"15min":0.67700000000000005},"min":{"1min":0.47899999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92300000000000004,"5min":0.93200000000000005,"15min":1.022},"last":0.69599999999999995}]},{"osd":7,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.68300000000000005,"15min":0.70699999999999996},"min":{"1min":0.5,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.75},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.71899999999999997},"min":{"1min":0.505,"5min":0.496,"15min":0.496},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.296},"last":0.64000000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575090,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307992,"kb_used_data":2512,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063848,"statfs":{"total":96636764160,"available":96321380352,"internally_reserved":0,"allocated":2572288,"data_stored":1609091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.40600000000000003},"min":{"1min":0.25800000000000001,"5min":0.23899999999999999,"15min":0.23100000000000001},"max":{"1min":0.88500000000000001,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.42999999999999999,"15min":0.45100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65800000000000003,"5min":1.363,"15min":1.4490000000000001},"last":0.39900000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.41099999999999998,"15min":0.42299999999999999},"min":{"1min":0.28899999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.89600000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.33900000000000002,"5min":0.32700000000000001,"15min":0.28000000000000003},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.36599999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.49199999999999999,"15min":0.47399999999999998},"min":{"1min":0.32800000000000001,"5min":0.32300000000000001,"15min":0.30599999999999999},"max":{"1min":0.85199999999999998,"5min":2.145,"15min":2.145},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36099999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":1.198},"last":0.40799999999999997}]},{"osd":4,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60899999999999999,"15min":0.63800000000000001},"min":{"1min":0.41699999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78600000000000003,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.59699999999999998,"15min":0.63700000000000001},"min":{"1min":0.44400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.94299999999999995},"last":0.64900000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":1.329,"15min":1.329},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.627,"5min":0.66300000000000003,"15min":0.67000000000000004},"min":{"1min":0.46899999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82799999999999996,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.55800000000000005}]},{"osd":6,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.61799999999999999,"15min":0.65100000000000002},"min":{"1min":0.47299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.89800000000000002,"5min":0.98699999999999999,"15min":1.032},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.65400000000000003,"15min":0.68000000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.41799999999999998},"max":{"1min":0.93100000000000005,"5min":1.286,"15min":1.286},"last":0.76800000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.70599999999999996,"15min":0.81699999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.435},"max":{"1min":0.81799999999999995,"5min":4.8710000000000004,"15min":25.965},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.73899999999999999,"15min":0.86099999999999999},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.433},"max":{"1min":0.88200000000000001,"5min":4.9199999999999999,"15min":33.085000000000001},"last":0.67400000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575090,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307412,"kb_used_data":1932,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064428,"statfs":{"total":96636764160,"available":96321974272,"internally_reserved":0,"allocated":1978368,"data_stored":1019255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.379,"15min":0.39100000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.255},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.43099999999999999,"15min":0.42599999999999999},"min":{"1min":0.29699999999999999,"5min":0.255,"15min":0.255},"max":{"1min":0.59899999999999998,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.34000000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.435,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.26900000000000002,"15min":0.248},"max":{"1min":0.63500000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52200000000000002,"15min":0.51100000000000001},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.68300000000000005,"5min":1.171,"15min":1.171},"last":0.40400000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.46899999999999997,"15min":0.48399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.313},"max":{"1min":0.64300000000000002,"5min":0.72799999999999998,"15min":0.74099999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47699999999999998},"min":{"1min":0.32900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70599999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.375}]},{"osd":4,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59599999999999997,"15min":0.628},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.087},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.625,"15min":0.64300000000000002},"min":{"1min":0.39100000000000001,"5min":0.371,"15min":0.371},"max":{"1min":1.006,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.56699999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.63900000000000001,"15min":0.65500000000000003},"min":{"1min":0.42299999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94699999999999995,"5min":1.24,"15min":1.24},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.65200000000000002,"15min":0.67400000000000004},"min":{"1min":0.48799999999999999,"5min":0.41799999999999998,"15min":0.35199999999999998},"max":{"1min":0.874,"5min":0.97699999999999998,"15min":1.085},"last":0.627}]},{"osd":6,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65700000000000003,"15min":0.67700000000000005},"min":{"1min":0.41199999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88600000000000001,"5min":1.022,"15min":1.1080000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.65200000000000002,"15min":0.67800000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.38800000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.1299999999999999},"last":0.68899999999999995}]},{"osd":7,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68400000000000005,"15min":0.70099999999999996},"min":{"1min":0.46400000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.88800000000000001,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.69699999999999995,"15min":0.72499999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.42099999999999999},"max":{"1min":0.96899999999999997,"5min":1.05,"15min":1.139},"last":0.58899999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575089,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307420,"kb_used_data":1940,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064420,"statfs":{"total":96636764160,"available":96321966080,"internally_reserved":0,"allocated":1986560,"data_stored":1020273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38400000000000001,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.55400000000000005,"5min":0.59699999999999998,"15min":1.8540000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.42899999999999999,"15min":0.438},"min":{"1min":0.29799999999999999,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":1.2290000000000001},"last":0.379}]},{"osd":1,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44500000000000001,"15min":0.45200000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":1.633},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.47999999999999998,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":1.0760000000000001},"last":0.53700000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48399999999999999,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65000000000000002,"5min":0.70399999999999996,"15min":1.4059999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.44500000000000001,"15min":0.45800000000000002},"min":{"1min":0.34100000000000003,"5min":0.29499999999999998,"15min":0.28299999999999997},"max":{"1min":0.629,"5min":0.69399999999999995,"15min":1.1870000000000001},"last":0.41599999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.63,"15min":0.65800000000000003},"min":{"1min":0.48299999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":3.306},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.44800000000000001,"5min":0.40100000000000002,"15min":0.379},"max":{"1min":0.81499999999999995,"5min":0.85199999999999998,"15min":1.008},"last":0.67100000000000004}]},{"osd":5,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.63600000000000001,"15min":0.65200000000000002},"min":{"1min":0.48599999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.85199999999999998,"5min":0.90400000000000003,"15min":1.593},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.47599999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.86299999999999999,"5min":0.92300000000000004,"15min":1.3819999999999999},"last":0.57899999999999996}]},{"osd":6,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.66900000000000004},"min":{"1min":0.42299999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":1.506},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67200000000000004,"15min":0.68999999999999995},"min":{"1min":0.55600000000000005,"5min":0.46700000000000003,"15min":0.44400000000000001},"max":{"1min":0.88500000000000001,"5min":0.89000000000000001,"15min":1.427},"last":0.62}]},{"osd":7,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64400000000000002,"15min":0.68799999999999994},"min":{"1min":0.46700000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":3.238},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.71199999999999997,"15min":0.73399999999999999},"min":{"1min":0.54200000000000004,"5min":0.51900000000000002,"15min":0.48899999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":3.351},"last":0.69399999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575091,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303308,"kb_used_data":1924,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068532,"statfs":{"total":96636764160,"available":96326176768,"internally_reserved":0,"allocated":1970176,"data_stored":1018667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52300000000000002,"15min":0.56200000000000006},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.72499999999999998,"5min":0.78900000000000003,"15min":1.21},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.32700000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.751,"5min":5.367,"15min":5.367},"last":0.52700000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.56899999999999995,"15min":0.58599999999999997},"min":{"1min":0.42699999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.79300000000000004,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.627},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54100000000000004,"15min":0.57199999999999995},"min":{"1min":0.41399999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73599999999999999,"5min":0.80600000000000005,"15min":0.81699999999999995},"last":0.55400000000000005}]},{"osd":2,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54500000000000004,"15min":0.56399999999999995},"min":{"1min":0.38500000000000001,"5min":0.33900000000000002,"15min":0.32700000000000001},"max":{"1min":0.751,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57099999999999995,"15min":0.59499999999999997},"min":{"1min":0.44500000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73399999999999999,"5min":0.86199999999999999,"15min":1.7050000000000001},"last":0.67400000000000004}]},{"osd":3,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56899999999999995,"15min":0.58499999999999996},"min":{"1min":0.45600000000000002,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.70499999999999996,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56599999999999995,"15min":0.58099999999999996},"min":{"1min":0.433,"5min":0.37,"15min":0.35699999999999998},"max":{"1min":0.78300000000000003,"5min":0.94199999999999995,"15min":1.05},"last":0.65500000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45400000000000001,"15min":0.46500000000000002},"min":{"1min":0.32300000000000001,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.60499999999999998,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.47899999999999998,"15min":0.48599999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64300000000000002,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.40000000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.46200000000000002,"15min":0.47399999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27200000000000002},"max":{"1min":0.63500000000000001,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.436},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.38600000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72099999999999997,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.42199999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52600000000000002,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.72499999999999998,"5min":0.879,"15min":0.879},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55800000000000005,"15min":0.57199999999999995},"min":{"1min":0.437,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.74199999999999999,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.495}]}]},{"osd":5,"up_from":13,"seq":55834575091,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303312,"kb_used_data":1928,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068528,"statfs":{"total":96636764160,"available":96326172672,"internally_reserved":0,"allocated":1974272,"data_stored":1018689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.56399999999999995},"min":{"1min":0.35199999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.71099999999999997,"5min":0.81799999999999995,"15min":2.0289999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52700000000000002,"15min":0.56699999999999995},"min":{"1min":0.41999999999999998,"5min":0.29999999999999999,"15min":0.28899999999999998},"max":{"1min":0.70799999999999996,"5min":0.77400000000000002,"15min":0.82199999999999995},"last":0.373}]},{"osd":1,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56299999999999994,"15min":0.60399999999999998},"min":{"1min":0.443,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":0.754,"5min":0.79500000000000004,"15min":2.3319999999999999},"last":0.441},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59899999999999998,"15min":0.626},"min":{"1min":0.45800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76400000000000001,"5min":0.80500000000000005,"15min":1.1739999999999999},"last":0.53500000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53700000000000003,"15min":0.55700000000000005},"min":{"1min":0.36799999999999999,"5min":0.34499999999999997,"15min":0.30099999999999999},"max":{"1min":0.68400000000000005,"5min":0.79400000000000004,"15min":0.83799999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55300000000000005,"15min":0.58699999999999997},"min":{"1min":0.41299999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72899999999999998,"5min":0.81100000000000005,"15min":2.0600000000000001},"last":0.40500000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55300000000000005,"15min":0.57499999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.314},"max":{"1min":0.69699999999999995,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55900000000000005,"15min":0.58699999999999997},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73999999999999999,"5min":0.77900000000000003,"15min":2.5630000000000002},"last":0.50800000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.42899999999999999,"15min":0.45500000000000002},"min":{"1min":0.313,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.65100000000000002,"5min":0.67200000000000004,"15min":2.3700000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45900000000000002,"15min":0.48099999999999998},"min":{"1min":0.32600000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.68200000000000005,"5min":0.82699999999999996,"15min":0.89200000000000002},"last":0.41999999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.47199999999999998},"min":{"1min":0.34300000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66900000000000004,"5min":0.67700000000000005,"15min":2.5990000000000002},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.49299999999999999,"15min":0.51400000000000001},"min":{"1min":0.374,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71399999999999997,"5min":0.76200000000000001,"15min":1.99},"last":0.51900000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51400000000000001,"15min":0.53800000000000003},"min":{"1min":0.374,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73799999999999999,"5min":0.76200000000000001,"15min":3.5270000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54400000000000004,"15min":0.55900000000000005},"min":{"1min":0.43099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78900000000000003,"5min":0.81499999999999995,"15min":0.82199999999999995},"last":0.54800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:50:09.940 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-26T23:50:10.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.108+0000 7fa620c8f700 1 Processor -- start 2024-01-26T23:50:10.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.109+0000 7fa620c8f700 1 -- start start 2024-01-26T23:50:10.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.110+0000 7fa620c8f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa61c159590 0x7fa61c15d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.110+0000 7fa620c8f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa61c15d8f0 0x7fa61c06bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.110+0000 7fa620c8f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61c15dd90 con 0x7fa61c159960 2024-01-26T23:50:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.110+0000 7fa620c8f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61c15e350 con 0x7fa61c15d8f0 2024-01-26T23:50:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.110+0000 7fa620c8f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c0c97c0 con 0x7fa61c159590 2024-01-26T23:50:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.110+0000 7fa619d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa61c15d8f0 0x7fa61c06bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.110+0000 7fa619d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa61c15d8f0 0x7fa61c06bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34696/0 (socket says 172.21.15.46:34696) 2024-01-26T23:50:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.110+0000 7fa619d9b700 1 -- 172.21.15.46:0/3118922233 learned_addr learned my addr 172.21.15.46:0/3118922233 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.110+0000 7fa61a59c700 1 --2- 172.21.15.46:0/3118922233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa61c159590 0x7fa61c15d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.110+0000 7fa619d9b700 1 -- 172.21.15.46:0/3118922233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa61c159590 msgr2=0x7fa61c15d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.110+0000 7fa619d9b700 1 --2- 172.21.15.46:0/3118922233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa61c159590 0x7fa61c15d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.110+0000 7fa619d9b700 1 -- 172.21.15.46:0/3118922233 >> v1:172.21.15.106:6789/0 conn(0x7fa61c159960 legacy=0x7fa61c06d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.110+0000 7fa619d9b700 1 -- 172.21.15.46:0/3118922233 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61c15e070 con 0x7fa61c15d8f0 2024-01-26T23:50:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.110+0000 7fa61a59c700 1 --2- 172.21.15.46:0/3118922233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa61c159590 0x7fa61c15d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:50:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.110+0000 7fa619d9b700 1 --2- 172.21.15.46:0/3118922233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa61c15d8f0 0x7fa61c06bcb0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fa6100099e0 tx=0x7fa610009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=394449a5ff6dfa06 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:10.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.111+0000 7fa61959a700 1 -- 172.21.15.46:0/3118922233 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61000b820 con 0x7fa61c15d8f0 2024-01-26T23:50:10.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.111+0000 7fa61959a700 1 -- 172.21.15.46:0/3118922233 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61000b980 con 0x7fa61c15d8f0 2024-01-26T23:50:10.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.111+0000 7fa61959a700 1 -- 172.21.15.46:0/3118922233 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61000bd40 con 0x7fa61c15d8f0 2024-01-26T23:50:10.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.111+0000 7fa620c8f700 1 -- 172.21.15.46:0/3118922233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa61c15d8f0 msgr2=0x7fa61c06bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:10.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.111+0000 7fa620c8f700 1 --2- 172.21.15.46:0/3118922233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa61c15d8f0 0x7fa61c06bcb0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fa6100099e0 tx=0x7fa610009250 comp rx=0 tx=0).stop 2024-01-26T23:50:10.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.111+0000 7fa620c8f700 1 -- 172.21.15.46:0/3118922233 shutdown_connections 2024-01-26T23:50:10.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.111+0000 7fa620c8f700 1 --2- 172.21.15.46:0/3118922233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa61c159590 0x7fa61c15d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.111+0000 7fa620c8f700 1 --2- 172.21.15.46:0/3118922233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa61c15d8f0 0x7fa61c06bcb0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.111+0000 7fa620c8f700 1 -- 172.21.15.46:0/3118922233 >> 172.21.15.46:0/3118922233 conn(0x7fa61c0ce530 msgr2=0x7fa61c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:10.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.111+0000 7fa620c8f700 1 -- 172.21.15.46:0/3118922233 shutdown_connections 2024-01-26T23:50:10.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.111+0000 7fa620c8f700 1 -- 172.21.15.46:0/3118922233 wait complete. 2024-01-26T23:50:10.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.112+0000 7fa620c8f700 1 Processor -- start 2024-01-26T23:50:10.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.112+0000 7fa620c8f700 1 -- start start 2024-01-26T23:50:10.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.113+0000 7fa620c8f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa61c159590 0x7fa61c0b2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.113+0000 7fa620c8f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa61c159960 0x7fa61c0c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.113+0000 7fa620c8f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa61c15d8f0 0x7fa61c0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.113+0000 7fa620c8f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c0b21d0 con 0x7fa61c159960 2024-01-26T23:50:10.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.113+0000 7fa620c8f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c0b12e0 con 0x7fa61c15d8f0 2024-01-26T23:50:10.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.113+0000 7fa620c8f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c0b15c0 con 0x7fa61c159590 2024-01-26T23:50:10.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.113+0000 7fa619d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa61c159960 0x7fa61c0c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.113+0000 7fa619d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa61c159960 0x7fa61c0c6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34712/0 (socket says 172.21.15.46:34712) 2024-01-26T23:50:10.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.113+0000 7fa619d9b700 1 -- 172.21.15.46:0/3602473270 learned_addr learned my addr 172.21.15.46:0/3602473270 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:10.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.113+0000 7fa61a59c700 1 --2- 172.21.15.46:0/3602473270 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa61c159590 0x7fa61c0b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.113+0000 7fa61ad9d700 1 --2- 172.21.15.46:0/3602473270 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa61c15d8f0 0x7fa61c0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.114+0000 7fa619d9b700 1 -- 172.21.15.46:0/3602473270 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa61c159590 msgr2=0x7fa61c0b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:10.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.114+0000 7fa619d9b700 1 --2- 172.21.15.46:0/3602473270 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa61c159590 0x7fa61c0b2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.114+0000 7fa619d9b700 1 -- 172.21.15.46:0/3602473270 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa61c15d8f0 msgr2=0x7fa61c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:10.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.114+0000 7fa619d9b700 1 --2- 172.21.15.46:0/3602473270 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa61c15d8f0 0x7fa61c0b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.114+0000 7fa619d9b700 1 -- 172.21.15.46:0/3602473270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6100094f0 con 0x7fa61c159960 2024-01-26T23:50:10.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.114+0000 7fa619d9b700 1 --2- 172.21.15.46:0/3602473270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa61c159960 0x7fa61c0c6eb0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fa610009930 tx=0x7fa610017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:10.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.114+0000 7fa60a7fc700 1 -- 172.21.15.46:0/3602473270 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61000b820 con 0x7fa61c159960 2024-01-26T23:50:10.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.114+0000 7fa60a7fc700 1 -- 172.21.15.46:0/3602473270 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa610002f30 con 0x7fa61c159960 2024-01-26T23:50:10.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.114+0000 7fa620c8f700 1 -- 172.21.15.46:0/3602473270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa61c0b18a0 con 0x7fa61c159960 2024-01-26T23:50:10.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.114+0000 7fa60a7fc700 1 -- 172.21.15.46:0/3602473270 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa610011d50 con 0x7fa61c159960 2024-01-26T23:50:10.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.114+0000 7fa620c8f700 1 -- 172.21.15.46:0/3602473270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa61c0b1e60 con 0x7fa61c159960 2024-01-26T23:50:10.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.115+0000 7fa620c8f700 1 -- 172.21.15.46:0/3602473270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa61c04c710 con 0x7fa61c159960 2024-01-26T23:50:10.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.116+0000 7fa60a7fc700 1 -- 172.21.15.46:0/3602473270 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa610023070 con 0x7fa61c159960 2024-01-26T23:50:10.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.116+0000 7fa60a7fc700 1 --2- 172.21.15.46:0/3602473270 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa5fc066f30 0x7fa5fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.116+0000 7fa60a7fc700 1 -- 172.21.15.46:0/3602473270 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fa610085b10 con 0x7fa61c159960 2024-01-26T23:50:10.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.117+0000 7fa61a59c700 1 --2- 172.21.15.46:0/3602473270 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa5fc066f30 0x7fa5fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.117+0000 7fa61a59c700 1 --2- 172.21.15.46:0/3602473270 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa5fc066f30 0x7fa5fc0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa60c006f00 tx=0x7fa60c008040 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:10.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.121+0000 7fa60a7fc700 1 -- 172.21.15.46:0/3602473270 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa610052a10 con 0x7fa61c159960 2024-01-26T23:50:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.322+0000 7fa620c8f700 1 -- 172.21.15.46:0/3602473270 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa61c068b70 con 0x7fa5fc066f30 2024-01-26T23:50:10.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.330+0000 7fa60a7fc700 1 -- 172.21.15.46:0/3602473270 <== mgr.8395 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289834 (secure 0 0 0) 0x7fa61c068b70 con 0x7fa5fc066f30 2024-01-26T23:50:10.335 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:50:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.333+0000 7fa620c8f700 1 -- 172.21.15.46:0/3602473270 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa5fc066f30 msgr2=0x7fa5fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.333+0000 7fa620c8f700 1 --2- 172.21.15.46:0/3602473270 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa5fc066f30 0x7fa5fc0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa60c006f00 tx=0x7fa60c008040 comp rx=0 tx=0).stop 2024-01-26T23:50:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.333+0000 7fa620c8f700 1 -- 172.21.15.46:0/3602473270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa61c159960 msgr2=0x7fa61c0c6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.333+0000 7fa620c8f700 1 --2- 172.21.15.46:0/3602473270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa61c159960 0x7fa61c0c6eb0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fa610009930 tx=0x7fa610017000 comp rx=0 tx=0).stop 2024-01-26T23:50:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.334+0000 7fa620c8f700 1 -- 172.21.15.46:0/3602473270 shutdown_connections 2024-01-26T23:50:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.334+0000 7fa620c8f700 1 --2- 172.21.15.46:0/3602473270 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa5fc066f30 0x7fa5fc0693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.334+0000 7fa620c8f700 1 --2- 172.21.15.46:0/3602473270 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa61c15d8f0 0x7fa61c0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.334+0000 7fa620c8f700 1 --2- 172.21.15.46:0/3602473270 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa61c159590 0x7fa61c0b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.334+0000 7fa620c8f700 1 --2- 172.21.15.46:0/3602473270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa61c159960 0x7fa61c0c6eb0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.334+0000 7fa620c8f700 1 -- 172.21.15.46:0/3602473270 >> 172.21.15.46:0/3602473270 conn(0x7fa61c0ce530 msgr2=0x7fa61c06de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.334+0000 7fa620c8f700 1 -- 172.21.15.46:0/3602473270 shutdown_connections 2024-01-26T23:50:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.334+0000 7fa620c8f700 1 -- 172.21.15.46:0/3602473270 wait complete. 2024-01-26T23:50:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-26T23:50:10.348 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-26T23:50:09.699670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1355,"num_read_kb":2726,"num_write":444,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8644,"ondisk_log_size":8644,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2452264,"kb_used_data":16616,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752522456,"statfs":{"total":773094113280,"available":770582994944,"internally_reserved":0,"allocated":17014784,"data_stored":9334684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.965340"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.766020+0000","last_change":"2024-01-26T23:50:05.760313+0000","last_active":"2024-01-26T23:50:05.766020+0000","last_peered":"2024-01-26T23:50:05.766020+0000","last_clean":"2024-01-26T23:50:05.766020+0000","last_became_active":"2024-01-26T23:50:05.760052+0000","last_became_peered":"2024-01-26T23:50:05.760052+0000","last_unstale":"2024-01-26T23:50:05.766020+0000","last_undegraded":"2024-01-26T23:50:05.766020+0000","last_fullsized":"2024-01-26T23:50:05.766020+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:58:56.234846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.755777+0000","last_change":"2024-01-26T23:50:05.754956+0000","last_active":"2024-01-26T23:50:05.755777+0000","last_peered":"2024-01-26T23:50:05.755777+0000","last_clean":"2024-01-26T23:50:05.755777+0000","last_became_active":"2024-01-26T23:50:05.754116+0000","last_became_peered":"2024-01-26T23:50:05.754116+0000","last_unstale":"2024-01-26T23:50:05.755777+0000","last_undegraded":"2024-01-26T23:50:05.755777+0000","last_fullsized":"2024-01-26T23:50:05.755777+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:58:34.814255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.769060+0000","last_change":"2024-01-26T23:50:05.763223+0000","last_active":"2024-01-26T23:50:05.769060+0000","last_peered":"2024-01-26T23:50:05.769060+0000","last_clean":"2024-01-26T23:50:05.769060+0000","last_became_active":"2024-01-26T23:50:05.762945+0000","last_became_peered":"2024-01-26T23:50:05.762945+0000","last_unstale":"2024-01-26T23:50:05.769060+0000","last_undegraded":"2024-01-26T23:50:05.769060+0000","last_fullsized":"2024-01-26T23:50:05.769060+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:27.332267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.756108+0000","last_change":"2024-01-26T23:50:05.751228+0000","last_active":"2024-01-26T23:50:05.756108+0000","last_peered":"2024-01-26T23:50:05.756108+0000","last_clean":"2024-01-26T23:50:05.756108+0000","last_became_active":"2024-01-26T23:50:05.750935+0000","last_became_peered":"2024-01-26T23:50:05.750935+0000","last_unstale":"2024-01-26T23:50:05.756108+0000","last_undegraded":"2024-01-26T23:50:05.756108+0000","last_fullsized":"2024-01-26T23:50:05.756108+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:55.611362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.761381+0000","last_change":"2024-01-26T23:50:05.756891+0000","last_active":"2024-01-26T23:50:05.761381+0000","last_peered":"2024-01-26T23:50:05.761381+0000","last_clean":"2024-01-26T23:50:05.761381+0000","last_became_active":"2024-01-26T23:50:05.756548+0000","last_became_peered":"2024-01-26T23:50:05.756548+0000","last_unstale":"2024-01-26T23:50:05.761381+0000","last_undegraded":"2024-01-26T23:50:05.761381+0000","last_fullsized":"2024-01-26T23:50:05.761381+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:27.332267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.763598+0000","last_change":"2024-01-26T23:50:05.756560+0000","last_active":"2024-01-26T23:50:05.763598+0000","last_peered":"2024-01-26T23:50:05.763598+0000","last_clean":"2024-01-26T23:50:05.763598+0000","last_became_active":"2024-01-26T23:50:05.756250+0000","last_became_peered":"2024-01-26T23:50:05.756250+0000","last_unstale":"2024-01-26T23:50:05.763598+0000","last_undegraded":"2024-01-26T23:50:05.763598+0000","last_fullsized":"2024-01-26T23:50:05.763598+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:21:26.665456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.773013+0000","last_change":"2024-01-26T23:50:05.764338+0000","last_active":"2024-01-26T23:50:05.773013+0000","last_peered":"2024-01-26T23:50:05.773013+0000","last_clean":"2024-01-26T23:50:05.773013+0000","last_became_active":"2024-01-26T23:50:05.763719+0000","last_became_peered":"2024-01-26T23:50:05.763719+0000","last_unstale":"2024-01-26T23:50:05.773013+0000","last_undegraded":"2024-01-26T23:50:05.773013+0000","last_fullsized":"2024-01-26T23:50:05.773013+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:34:45.453927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765775+0000","last_change":"2024-01-26T23:50:05.761592+0000","last_active":"2024-01-26T23:50:05.765775+0000","last_peered":"2024-01-26T23:50:05.765775+0000","last_clean":"2024-01-26T23:50:05.765775+0000","last_became_active":"2024-01-26T23:50:05.761388+0000","last_became_peered":"2024-01-26T23:50:05.761388+0000","last_unstale":"2024-01-26T23:50:05.765775+0000","last_undegraded":"2024-01-26T23:50:05.765775+0000","last_fullsized":"2024-01-26T23:50:05.765775+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:08:15.330869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.750472+0000","last_change":"2024-01-26T23:50:05.747564+0000","last_active":"2024-01-26T23:50:05.750472+0000","last_peered":"2024-01-26T23:50:05.750472+0000","last_clean":"2024-01-26T23:50:05.750472+0000","last_became_active":"2024-01-26T23:50:05.746771+0000","last_became_peered":"2024-01-26T23:50:05.746771+0000","last_unstale":"2024-01-26T23:50:05.750472+0000","last_undegraded":"2024-01-26T23:50:05.750472+0000","last_fullsized":"2024-01-26T23:50:05.750472+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:34:45.453927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.763687+0000","last_change":"2024-01-26T23:50:05.756949+0000","last_active":"2024-01-26T23:50:05.763687+0000","last_peered":"2024-01-26T23:50:05.763687+0000","last_clean":"2024-01-26T23:50:05.763687+0000","last_became_active":"2024-01-26T23:50:05.756661+0000","last_became_peered":"2024-01-26T23:50:05.756661+0000","last_unstale":"2024-01-26T23:50:05.763687+0000","last_undegraded":"2024-01-26T23:50:05.763687+0000","last_fullsized":"2024-01-26T23:50:05.763687+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:23:07.892082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765366+0000","last_change":"2024-01-26T23:50:05.760145+0000","last_active":"2024-01-26T23:50:05.765366+0000","last_peered":"2024-01-26T23:50:05.765366+0000","last_clean":"2024-01-26T23:50:05.765366+0000","last_became_active":"2024-01-26T23:50:05.758472+0000","last_became_peered":"2024-01-26T23:50:05.758472+0000","last_unstale":"2024-01-26T23:50:05.765366+0000","last_undegraded":"2024-01-26T23:50:05.765366+0000","last_fullsized":"2024-01-26T23:50:05.765366+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:12.031661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765918+0000","last_change":"2024-01-26T23:50:05.763183+0000","last_active":"2024-01-26T23:50:05.765918+0000","last_peered":"2024-01-26T23:50:05.765918+0000","last_clean":"2024-01-26T23:50:05.765918+0000","last_became_active":"2024-01-26T23:50:05.762802+0000","last_became_peered":"2024-01-26T23:50:05.762802+0000","last_unstale":"2024-01-26T23:50:05.765918+0000","last_undegraded":"2024-01-26T23:50:05.765918+0000","last_fullsized":"2024-01-26T23:50:05.765918+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:36:57.110071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.763905+0000","last_change":"2024-01-26T23:50:05.759112+0000","last_active":"2024-01-26T23:50:05.763905+0000","last_peered":"2024-01-26T23:50:05.763905+0000","last_clean":"2024-01-26T23:50:05.763905+0000","last_became_active":"2024-01-26T23:50:05.758822+0000","last_became_peered":"2024-01-26T23:50:05.758822+0000","last_unstale":"2024-01-26T23:50:05.763905+0000","last_undegraded":"2024-01-26T23:50:05.763905+0000","last_fullsized":"2024-01-26T23:50:05.763905+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:43:40.458165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765719+0000","last_change":"2024-01-26T23:50:05.759128+0000","last_active":"2024-01-26T23:50:05.765719+0000","last_peered":"2024-01-26T23:50:05.765719+0000","last_clean":"2024-01-26T23:50:05.765719+0000","last_became_active":"2024-01-26T23:50:05.758822+0000","last_became_peered":"2024-01-26T23:50:05.758822+0000","last_unstale":"2024-01-26T23:50:05.765719+0000","last_undegraded":"2024-01-26T23:50:05.765719+0000","last_fullsized":"2024-01-26T23:50:05.765719+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:50:26.914006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.769682+0000","last_change":"2024-01-26T23:50:05.766363+0000","last_active":"2024-01-26T23:50:05.769682+0000","last_peered":"2024-01-26T23:50:05.769682+0000","last_clean":"2024-01-26T23:50:05.769682+0000","last_became_active":"2024-01-26T23:50:05.766020+0000","last_became_peered":"2024-01-26T23:50:05.766020+0000","last_unstale":"2024-01-26T23:50:05.769682+0000","last_undegraded":"2024-01-26T23:50:05.769682+0000","last_fullsized":"2024-01-26T23:50:05.769682+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:42:57.962862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.764252+0000","last_change":"2024-01-26T23:50:05.751942+0000","last_active":"2024-01-26T23:50:05.764252+0000","last_peered":"2024-01-26T23:50:05.764252+0000","last_clean":"2024-01-26T23:50:05.764252+0000","last_became_active":"2024-01-26T23:50:05.751263+0000","last_became_peered":"2024-01-26T23:50:05.751263+0000","last_unstale":"2024-01-26T23:50:05.764252+0000","last_undegraded":"2024-01-26T23:50:05.764252+0000","last_fullsized":"2024-01-26T23:50:05.764252+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:13:16.831804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.766507+0000","last_change":"2024-01-26T23:50:05.752720+0000","last_active":"2024-01-26T23:50:05.766507+0000","last_peered":"2024-01-26T23:50:05.766507+0000","last_clean":"2024-01-26T23:50:05.766507+0000","last_became_active":"2024-01-26T23:50:05.752438+0000","last_became_peered":"2024-01-26T23:50:05.752438+0000","last_unstale":"2024-01-26T23:50:05.766507+0000","last_undegraded":"2024-01-26T23:50:05.766507+0000","last_fullsized":"2024-01-26T23:50:05.766507+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:04:10.945311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.761238+0000","last_change":"2024-01-26T23:50:05.755319+0000","last_active":"2024-01-26T23:50:05.761238+0000","last_peered":"2024-01-26T23:50:05.761238+0000","last_clean":"2024-01-26T23:50:05.761238+0000","last_became_active":"2024-01-26T23:50:05.755019+0000","last_became_peered":"2024-01-26T23:50:05.755019+0000","last_unstale":"2024-01-26T23:50:05.761238+0000","last_undegraded":"2024-01-26T23:50:05.761238+0000","last_fullsized":"2024-01-26T23:50:05.761238+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:34:45.453927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.773811+0000","last_change":"2024-01-26T23:50:05.764428+0000","last_active":"2024-01-26T23:50:05.773811+0000","last_peered":"2024-01-26T23:50:05.773811+0000","last_clean":"2024-01-26T23:50:05.773811+0000","last_became_active":"2024-01-26T23:50:05.763705+0000","last_became_peered":"2024-01-26T23:50:05.763705+0000","last_unstale":"2024-01-26T23:50:05.773811+0000","last_undegraded":"2024-01-26T23:50:05.773811+0000","last_fullsized":"2024-01-26T23:50:05.773811+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:27.332267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.761866+0000","last_change":"2024-01-26T23:50:05.757645+0000","last_active":"2024-01-26T23:50:05.761866+0000","last_peered":"2024-01-26T23:50:05.761866+0000","last_clean":"2024-01-26T23:50:05.761866+0000","last_became_active":"2024-01-26T23:50:05.757307+0000","last_became_peered":"2024-01-26T23:50:05.757307+0000","last_unstale":"2024-01-26T23:50:05.761866+0000","last_undegraded":"2024-01-26T23:50:05.761866+0000","last_fullsized":"2024-01-26T23:50:05.761866+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:13:16.831804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.755719+0000","last_change":"2024-01-26T23:50:05.751431+0000","last_active":"2024-01-26T23:50:05.755719+0000","last_peered":"2024-01-26T23:50:05.755719+0000","last_clean":"2024-01-26T23:50:05.755719+0000","last_became_active":"2024-01-26T23:50:05.751172+0000","last_became_peered":"2024-01-26T23:50:05.751172+0000","last_unstale":"2024-01-26T23:50:05.755719+0000","last_undegraded":"2024-01-26T23:50:05.755719+0000","last_fullsized":"2024-01-26T23:50:05.755719+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:40:49.040365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.762328+0000","last_change":"2024-01-26T23:50:05.756791+0000","last_active":"2024-01-26T23:50:05.762328+0000","last_peered":"2024-01-26T23:50:05.762328+0000","last_clean":"2024-01-26T23:50:05.762328+0000","last_became_active":"2024-01-26T23:50:05.756511+0000","last_became_peered":"2024-01-26T23:50:05.756511+0000","last_unstale":"2024-01-26T23:50:05.762328+0000","last_undegraded":"2024-01-26T23:50:05.762328+0000","last_fullsized":"2024-01-26T23:50:05.762328+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:57:56.258786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.764066+0000","last_change":"2024-01-26T23:50:05.758011+0000","last_active":"2024-01-26T23:50:05.764066+0000","last_peered":"2024-01-26T23:50:05.764066+0000","last_clean":"2024-01-26T23:50:05.764066+0000","last_became_active":"2024-01-26T23:50:05.757745+0000","last_became_peered":"2024-01-26T23:50:05.757745+0000","last_unstale":"2024-01-26T23:50:05.764066+0000","last_undegraded":"2024-01-26T23:50:05.764066+0000","last_fullsized":"2024-01-26T23:50:05.764066+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:19:26.227100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.750295+0000","last_change":"2024-01-26T23:50:05.748082+0000","last_active":"2024-01-26T23:50:05.750295+0000","last_peered":"2024-01-26T23:50:05.750295+0000","last_clean":"2024-01-26T23:50:05.750295+0000","last_became_active":"2024-01-26T23:50:05.747771+0000","last_became_peered":"2024-01-26T23:50:05.747771+0000","last_unstale":"2024-01-26T23:50:05.750295+0000","last_undegraded":"2024-01-26T23:50:05.750295+0000","last_fullsized":"2024-01-26T23:50:05.750295+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:27.332267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768843+0000","last_change":"2024-01-26T23:50:05.757097+0000","last_active":"2024-01-26T23:50:05.768843+0000","last_peered":"2024-01-26T23:50:05.768843+0000","last_clean":"2024-01-26T23:50:05.768843+0000","last_became_active":"2024-01-26T23:50:05.756761+0000","last_became_peered":"2024-01-26T23:50:05.756761+0000","last_unstale":"2024-01-26T23:50:05.768843+0000","last_undegraded":"2024-01-26T23:50:05.768843+0000","last_fullsized":"2024-01-26T23:50:05.768843+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:41:22.684297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768852+0000","last_change":"2024-01-26T23:50:05.764848+0000","last_active":"2024-01-26T23:50:05.768852+0000","last_peered":"2024-01-26T23:50:05.768852+0000","last_clean":"2024-01-26T23:50:05.768852+0000","last_became_active":"2024-01-26T23:50:05.764589+0000","last_became_peered":"2024-01-26T23:50:05.764589+0000","last_unstale":"2024-01-26T23:50:05.768852+0000","last_undegraded":"2024-01-26T23:50:05.768852+0000","last_fullsized":"2024-01-26T23:50:05.768852+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:34:45.453927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.773951+0000","last_change":"2024-01-26T23:50:05.768020+0000","last_active":"2024-01-26T23:50:05.773951+0000","last_peered":"2024-01-26T23:50:05.773951+0000","last_clean":"2024-01-26T23:50:05.773951+0000","last_became_active":"2024-01-26T23:50:05.766719+0000","last_became_peered":"2024-01-26T23:50:05.766719+0000","last_unstale":"2024-01-26T23:50:05.773951+0000","last_undegraded":"2024-01-26T23:50:05.773951+0000","last_fullsized":"2024-01-26T23:50:05.773951+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:59.234067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.761021+0000","last_change":"2024-01-26T23:50:05.752890+0000","last_active":"2024-01-26T23:50:05.761021+0000","last_peered":"2024-01-26T23:50:05.761021+0000","last_clean":"2024-01-26T23:50:05.761021+0000","last_became_active":"2024-01-26T23:50:05.752608+0000","last_became_peered":"2024-01-26T23:50:05.752608+0000","last_unstale":"2024-01-26T23:50:05.761021+0000","last_undegraded":"2024-01-26T23:50:05.761021+0000","last_fullsized":"2024-01-26T23:50:05.761021+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:28:16.609431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768851+0000","last_change":"2024-01-26T23:50:05.764753+0000","last_active":"2024-01-26T23:50:05.768851+0000","last_peered":"2024-01-26T23:50:05.768851+0000","last_clean":"2024-01-26T23:50:05.768851+0000","last_became_active":"2024-01-26T23:50:05.764494+0000","last_became_peered":"2024-01-26T23:50:05.764494+0000","last_unstale":"2024-01-26T23:50:05.768851+0000","last_undegraded":"2024-01-26T23:50:05.768851+0000","last_fullsized":"2024-01-26T23:50:05.768851+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:31.322920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.764031+0000","last_change":"2024-01-26T23:50:05.755009+0000","last_active":"2024-01-26T23:50:05.764031+0000","last_peered":"2024-01-26T23:50:05.764031+0000","last_clean":"2024-01-26T23:50:05.764031+0000","last_became_active":"2024-01-26T23:50:05.754763+0000","last_became_peered":"2024-01-26T23:50:05.754763+0000","last_unstale":"2024-01-26T23:50:05.764031+0000","last_undegraded":"2024-01-26T23:50:05.764031+0000","last_fullsized":"2024-01-26T23:50:05.764031+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:12:48.929340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768347+0000","last_change":"2024-01-26T23:50:05.765022+0000","last_active":"2024-01-26T23:50:05.768347+0000","last_peered":"2024-01-26T23:50:05.768347+0000","last_clean":"2024-01-26T23:50:05.768347+0000","last_became_active":"2024-01-26T23:50:05.764766+0000","last_became_peered":"2024-01-26T23:50:05.764766+0000","last_unstale":"2024-01-26T23:50:05.768347+0000","last_undegraded":"2024-01-26T23:50:05.768347+0000","last_fullsized":"2024-01-26T23:50:05.768347+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:21:26.665456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.760393+0000","last_change":"2024-01-26T23:50:05.753319+0000","last_active":"2024-01-26T23:50:05.760393+0000","last_peered":"2024-01-26T23:50:05.760393+0000","last_clean":"2024-01-26T23:50:05.760393+0000","last_became_active":"2024-01-26T23:50:05.753032+0000","last_became_peered":"2024-01-26T23:50:05.753032+0000","last_unstale":"2024-01-26T23:50:05.760393+0000","last_undegraded":"2024-01-26T23:50:05.760393+0000","last_fullsized":"2024-01-26T23:50:05.760393+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:09:57.835401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768567+0000","last_change":"2024-01-26T23:50:05.765445+0000","last_active":"2024-01-26T23:50:05.768567+0000","last_peered":"2024-01-26T23:50:05.768567+0000","last_clean":"2024-01-26T23:50:05.768567+0000","last_became_active":"2024-01-26T23:50:05.765130+0000","last_became_peered":"2024-01-26T23:50:05.765130+0000","last_unstale":"2024-01-26T23:50:05.768567+0000","last_undegraded":"2024-01-26T23:50:05.768567+0000","last_fullsized":"2024-01-26T23:50:05.768567+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:43:40.458165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"157'4","reported_seq":14,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.775671+0000","last_change":"2024-01-26T23:50:04.760218+0000","last_active":"2024-01-26T23:50:05.775671+0000","last_peered":"2024-01-26T23:50:05.775671+0000","last_clean":"2024-01-26T23:50:05.775671+0000","last_became_active":"2024-01-26T23:50:04.759915+0000","last_became_peered":"2024-01-26T23:50:04.759915+0000","last_unstale":"2024-01-26T23:50:05.775671+0000","last_undegraded":"2024-01-26T23:50:05.775671+0000","last_fullsized":"2024-01-26T23:50:05.775671+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:52.388343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.762445+0000","last_change":"2024-01-26T23:50:05.755855+0000","last_active":"2024-01-26T23:50:05.762445+0000","last_peered":"2024-01-26T23:50:05.762445+0000","last_clean":"2024-01-26T23:50:05.762445+0000","last_became_active":"2024-01-26T23:50:05.755573+0000","last_became_peered":"2024-01-26T23:50:05.755573+0000","last_unstale":"2024-01-26T23:50:05.762445+0000","last_undegraded":"2024-01-26T23:50:05.762445+0000","last_fullsized":"2024-01-26T23:50:05.762445+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:01:40.211350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"157'1","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.772369+0000","last_change":"2024-01-26T23:50:04.758971+0000","last_active":"2024-01-26T23:50:05.772369+0000","last_peered":"2024-01-26T23:50:05.772369+0000","last_clean":"2024-01-26T23:50:05.772369+0000","last_became_active":"2024-01-26T23:50:04.758003+0000","last_became_peered":"2024-01-26T23:50:04.758003+0000","last_unstale":"2024-01-26T23:50:05.772369+0000","last_undegraded":"2024-01-26T23:50:05.772369+0000","last_fullsized":"2024-01-26T23:50:05.772369+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:54:10.044522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.764131+0000","last_change":"2024-01-26T23:50:05.758246+0000","last_active":"2024-01-26T23:50:05.764131+0000","last_peered":"2024-01-26T23:50:05.764131+0000","last_clean":"2024-01-26T23:50:05.764131+0000","last_became_active":"2024-01-26T23:50:05.757889+0000","last_became_peered":"2024-01-26T23:50:05.757889+0000","last_unstale":"2024-01-26T23:50:05.764131+0000","last_undegraded":"2024-01-26T23:50:05.764131+0000","last_fullsized":"2024-01-26T23:50:05.764131+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:22:24.153814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"157'2","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.775316+0000","last_change":"2024-01-26T23:50:04.754647+0000","last_active":"2024-01-26T23:50:05.775316+0000","last_peered":"2024-01-26T23:50:05.775316+0000","last_clean":"2024-01-26T23:50:05.775316+0000","last_became_active":"2024-01-26T23:50:04.754246+0000","last_became_peered":"2024-01-26T23:50:04.754246+0000","last_unstale":"2024-01-26T23:50:05.775316+0000","last_undegraded":"2024-01-26T23:50:05.775316+0000","last_fullsized":"2024-01-26T23:50:05.775316+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:19:25.221567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.762401+0000","last_change":"2024-01-26T23:50:05.758309+0000","last_active":"2024-01-26T23:50:05.762401+0000","last_peered":"2024-01-26T23:50:05.762401+0000","last_clean":"2024-01-26T23:50:05.762401+0000","last_became_active":"2024-01-26T23:50:05.758049+0000","last_became_peered":"2024-01-26T23:50:05.758049+0000","last_unstale":"2024-01-26T23:50:05.762401+0000","last_undegraded":"2024-01-26T23:50:05.762401+0000","last_fullsized":"2024-01-26T23:50:05.762401+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:04:52.726605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"157'2","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.773414+0000","last_change":"2024-01-26T23:50:04.755194+0000","last_active":"2024-01-26T23:50:05.773414+0000","last_peered":"2024-01-26T23:50:05.773414+0000","last_clean":"2024-01-26T23:50:05.773414+0000","last_became_active":"2024-01-26T23:50:04.754928+0000","last_became_peered":"2024-01-26T23:50:04.754928+0000","last_unstale":"2024-01-26T23:50:05.773414+0000","last_undegraded":"2024-01-26T23:50:05.773414+0000","last_fullsized":"2024-01-26T23:50:05.773414+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:58:33.808722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765735+0000","last_change":"2024-01-26T23:50:05.762523+0000","last_active":"2024-01-26T23:50:05.765735+0000","last_peered":"2024-01-26T23:50:05.765735+0000","last_clean":"2024-01-26T23:50:05.765735+0000","last_became_active":"2024-01-26T23:50:05.762268+0000","last_became_peered":"2024-01-26T23:50:05.762268+0000","last_unstale":"2024-01-26T23:50:05.765735+0000","last_undegraded":"2024-01-26T23:50:05.765735+0000","last_fullsized":"2024-01-26T23:50:05.765735+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:13:16.831804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"157'1","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.774264+0000","last_change":"2024-01-26T23:50:04.764076+0000","last_active":"2024-01-26T23:50:05.774264+0000","last_peered":"2024-01-26T23:50:05.774264+0000","last_clean":"2024-01-26T23:50:05.774264+0000","last_became_active":"2024-01-26T23:50:04.763803+0000","last_became_peered":"2024-01-26T23:50:04.763803+0000","last_unstale":"2024-01-26T23:50:05.774264+0000","last_undegraded":"2024-01-26T23:50:05.774264+0000","last_fullsized":"2024-01-26T23:50:05.774264+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:41:21.678763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765714+0000","last_change":"2024-01-26T23:50:05.761793+0000","last_active":"2024-01-26T23:50:05.765714+0000","last_peered":"2024-01-26T23:50:05.765714+0000","last_clean":"2024-01-26T23:50:05.765714+0000","last_became_active":"2024-01-26T23:50:05.761077+0000","last_became_peered":"2024-01-26T23:50:05.761077+0000","last_unstale":"2024-01-26T23:50:05.765714+0000","last_undegraded":"2024-01-26T23:50:05.765714+0000","last_fullsized":"2024-01-26T23:50:05.765714+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:04:10.945311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"157'1","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.772637+0000","last_change":"2024-01-26T23:50:04.756494+0000","last_active":"2024-01-26T23:50:05.772637+0000","last_peered":"2024-01-26T23:50:05.772637+0000","last_clean":"2024-01-26T23:50:05.772637+0000","last_became_active":"2024-01-26T23:50:04.756129+0000","last_became_peered":"2024-01-26T23:50:04.756129+0000","last_unstale":"2024-01-26T23:50:05.772637+0000","last_undegraded":"2024-01-26T23:50:05.772637+0000","last_fullsized":"2024-01-26T23:50:05.772637+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:47:07.429822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768375+0000","last_change":"2024-01-26T23:50:05.760552+0000","last_active":"2024-01-26T23:50:05.768375+0000","last_peered":"2024-01-26T23:50:05.768375+0000","last_clean":"2024-01-26T23:50:05.768375+0000","last_became_active":"2024-01-26T23:50:05.758808+0000","last_became_peered":"2024-01-26T23:50:05.758808+0000","last_unstale":"2024-01-26T23:50:05.768375+0000","last_undegraded":"2024-01-26T23:50:05.768375+0000","last_fullsized":"2024-01-26T23:50:05.768375+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:12.635232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.740594+0000","last_change":"2024-01-26T23:50:04.758300+0000","last_active":"2024-01-26T23:50:05.740594+0000","last_peered":"2024-01-26T23:50:05.740594+0000","last_clean":"2024-01-26T23:50:05.740594+0000","last_became_active":"2024-01-26T23:50:04.757776+0000","last_became_peered":"2024-01-26T23:50:04.757776+0000","last_unstale":"2024-01-26T23:50:05.740594+0000","last_undegraded":"2024-01-26T23:50:05.740594+0000","last_fullsized":"2024-01-26T23:50:05.740594+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:11.026128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.773892+0000","last_change":"2024-01-26T23:50:05.773293+0000","last_active":"2024-01-26T23:50:05.773892+0000","last_peered":"2024-01-26T23:50:05.773892+0000","last_clean":"2024-01-26T23:50:05.773892+0000","last_became_active":"2024-01-26T23:50:05.773013+0000","last_became_peered":"2024-01-26T23:50:05.773013+0000","last_unstale":"2024-01-26T23:50:05.773892+0000","last_undegraded":"2024-01-26T23:50:05.773892+0000","last_fullsized":"2024-01-26T23:50:05.773892+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:16.574844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.752154+0000","last_change":"2024-01-26T23:50:04.762129+0000","last_active":"2024-01-26T23:50:05.752154+0000","last_peered":"2024-01-26T23:50:05.752154+0000","last_clean":"2024-01-26T23:50:05.752154+0000","last_became_active":"2024-01-26T23:50:04.761854+0000","last_became_peered":"2024-01-26T23:50:04.761854+0000","last_unstale":"2024-01-26T23:50:05.752154+0000","last_undegraded":"2024-01-26T23:50:05.752154+0000","last_fullsized":"2024-01-26T23:50:05.752154+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:22:23.148281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.762384+0000","last_change":"2024-01-26T23:50:05.758923+0000","last_active":"2024-01-26T23:50:05.762384+0000","last_peered":"2024-01-26T23:50:05.762384+0000","last_clean":"2024-01-26T23:50:05.762384+0000","last_became_active":"2024-01-26T23:50:05.758142+0000","last_became_peered":"2024-01-26T23:50:05.758142+0000","last_unstale":"2024-01-26T23:50:05.762384+0000","last_undegraded":"2024-01-26T23:50:05.762384+0000","last_fullsized":"2024-01-26T23:50:05.762384+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:55:29.374881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.745153+0000","last_change":"2024-01-26T23:50:04.758455+0000","last_active":"2024-01-26T23:50:05.745153+0000","last_peered":"2024-01-26T23:50:05.745153+0000","last_clean":"2024-01-26T23:50:05.745153+0000","last_became_active":"2024-01-26T23:50:04.758181+0000","last_became_peered":"2024-01-26T23:50:04.758181+0000","last_unstale":"2024-01-26T23:50:05.745153+0000","last_undegraded":"2024-01-26T23:50:05.745153+0000","last_fullsized":"2024-01-26T23:50:05.745153+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:59:45.611620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.763742+0000","last_change":"2024-01-26T23:50:05.753753+0000","last_active":"2024-01-26T23:50:05.763742+0000","last_peered":"2024-01-26T23:50:05.763742+0000","last_clean":"2024-01-26T23:50:05.763742+0000","last_became_active":"2024-01-26T23:50:05.751971+0000","last_became_peered":"2024-01-26T23:50:05.751971+0000","last_unstale":"2024-01-26T23:50:05.763742+0000","last_undegraded":"2024-01-26T23:50:05.763742+0000","last_fullsized":"2024-01-26T23:50:05.763742+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:45:58.463554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.750180+0000","last_change":"2024-01-26T23:50:04.757736+0000","last_active":"2024-01-26T23:50:05.750180+0000","last_peered":"2024-01-26T23:50:05.750180+0000","last_clean":"2024-01-26T23:50:05.750180+0000","last_became_active":"2024-01-26T23:50:04.757423+0000","last_became_peered":"2024-01-26T23:50:04.757423+0000","last_unstale":"2024-01-26T23:50:05.750180+0000","last_undegraded":"2024-01-26T23:50:05.750180+0000","last_fullsized":"2024-01-26T23:50:05.750180+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:09.475540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.766247+0000","last_change":"2024-01-26T23:50:05.756866+0000","last_active":"2024-01-26T23:50:05.766247+0000","last_peered":"2024-01-26T23:50:05.766247+0000","last_clean":"2024-01-26T23:50:05.766247+0000","last_became_active":"2024-01-26T23:50:05.756532+0000","last_became_peered":"2024-01-26T23:50:05.756532+0000","last_unstale":"2024-01-26T23:50:05.766247+0000","last_undegraded":"2024-01-26T23:50:05.766247+0000","last_fullsized":"2024-01-26T23:50:05.766247+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:45:59.234067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"157'3","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.774311+0000","last_change":"2024-01-26T23:50:04.757627+0000","last_active":"2024-01-26T23:50:05.774311+0000","last_peered":"2024-01-26T23:50:05.774311+0000","last_clean":"2024-01-26T23:50:05.774311+0000","last_became_active":"2024-01-26T23:50:04.757332+0000","last_became_peered":"2024-01-26T23:50:04.757332+0000","last_unstale":"2024-01-26T23:50:05.774311+0000","last_undegraded":"2024-01-26T23:50:05.774311+0000","last_fullsized":"2024-01-26T23:50:05.774311+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:50:00.006714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768869+0000","last_change":"2024-01-26T23:50:05.767003+0000","last_active":"2024-01-26T23:50:05.768869+0000","last_peered":"2024-01-26T23:50:05.768869+0000","last_clean":"2024-01-26T23:50:05.768869+0000","last_became_active":"2024-01-26T23:50:05.766724+0000","last_became_peered":"2024-01-26T23:50:05.766724+0000","last_unstale":"2024-01-26T23:50:05.768869+0000","last_undegraded":"2024-01-26T23:50:05.768869+0000","last_fullsized":"2024-01-26T23:50:05.768869+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:12.635232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"157'6","reported_seq":16,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.776719+0000","last_change":"2024-01-26T23:50:04.759904+0000","last_active":"2024-01-26T23:50:05.776719+0000","last_peered":"2024-01-26T23:50:05.776719+0000","last_clean":"2024-01-26T23:50:05.776719+0000","last_became_active":"2024-01-26T23:50:04.759625+0000","last_became_peered":"2024-01-26T23:50:04.759625+0000","last_unstale":"2024-01-26T23:50:05.776719+0000","last_undegraded":"2024-01-26T23:50:05.776719+0000","last_fullsized":"2024-01-26T23:50:05.776719+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:52.388343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765023+0000","last_change":"2024-01-26T23:50:05.760077+0000","last_active":"2024-01-26T23:50:05.765023+0000","last_peered":"2024-01-26T23:50:05.765023+0000","last_clean":"2024-01-26T23:50:05.765023+0000","last_became_active":"2024-01-26T23:50:05.759695+0000","last_became_peered":"2024-01-26T23:50:05.759695+0000","last_unstale":"2024-01-26T23:50:05.765023+0000","last_undegraded":"2024-01-26T23:50:05.765023+0000","last_fullsized":"2024-01-26T23:50:05.765023+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:19:17.243302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"157'2","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.775305+0000","last_change":"2024-01-26T23:50:04.755696+0000","last_active":"2024-01-26T23:50:05.775305+0000","last_peered":"2024-01-26T23:50:05.775305+0000","last_clean":"2024-01-26T23:50:05.775305+0000","last_became_active":"2024-01-26T23:50:04.755379+0000","last_became_peered":"2024-01-26T23:50:04.755379+0000","last_unstale":"2024-01-26T23:50:05.775305+0000","last_undegraded":"2024-01-26T23:50:05.775305+0000","last_fullsized":"2024-01-26T23:50:05.775305+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:47:11.376776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765677+0000","last_change":"2024-01-26T23:50:05.757271+0000","last_active":"2024-01-26T23:50:05.765677+0000","last_peered":"2024-01-26T23:50:05.765677+0000","last_clean":"2024-01-26T23:50:05.765677+0000","last_became_active":"2024-01-26T23:50:05.757007+0000","last_became_peered":"2024-01-26T23:50:05.757007+0000","last_unstale":"2024-01-26T23:50:05.765677+0000","last_undegraded":"2024-01-26T23:50:05.765677+0000","last_fullsized":"2024-01-26T23:50:05.765677+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:12:48.929340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.748074+0000","last_change":"2024-01-26T23:50:04.754865+0000","last_active":"2024-01-26T23:50:05.748074+0000","last_peered":"2024-01-26T23:50:05.748074+0000","last_clean":"2024-01-26T23:50:05.748074+0000","last_became_active":"2024-01-26T23:50:04.754600+0000","last_became_peered":"2024-01-26T23:50:04.754600+0000","last_unstale":"2024-01-26T23:50:05.748074+0000","last_undegraded":"2024-01-26T23:50:05.748074+0000","last_fullsized":"2024-01-26T23:50:05.748074+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:54.605829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.760928+0000","last_change":"2024-01-26T23:50:05.751464+0000","last_active":"2024-01-26T23:50:05.760928+0000","last_peered":"2024-01-26T23:50:05.760928+0000","last_clean":"2024-01-26T23:50:05.760928+0000","last_became_active":"2024-01-26T23:50:05.751120+0000","last_became_peered":"2024-01-26T23:50:05.751120+0000","last_unstale":"2024-01-26T23:50:05.760928+0000","last_undegraded":"2024-01-26T23:50:05.760928+0000","last_fullsized":"2024-01-26T23:50:05.760928+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:43:40.458165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"157'8","reported_seq":18,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.775360+0000","last_change":"2024-01-26T23:50:04.756977+0000","last_active":"2024-01-26T23:50:05.775360+0000","last_peered":"2024-01-26T23:50:05.775360+0000","last_clean":"2024-01-26T23:50:05.775360+0000","last_became_active":"2024-01-26T23:50:04.756685+0000","last_became_peered":"2024-01-26T23:50:04.756685+0000","last_unstale":"2024-01-26T23:50:05.775360+0000","last_undegraded":"2024-01-26T23:50:05.775360+0000","last_fullsized":"2024-01-26T23:50:05.775360+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:10:03.148195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768774+0000","last_change":"2024-01-26T23:50:05.763325+0000","last_active":"2024-01-26T23:50:05.768774+0000","last_peered":"2024-01-26T23:50:05.768774+0000","last_clean":"2024-01-26T23:50:05.768774+0000","last_became_active":"2024-01-26T23:50:05.763052+0000","last_became_peered":"2024-01-26T23:50:05.763052+0000","last_unstale":"2024-01-26T23:50:05.768774+0000","last_undegraded":"2024-01-26T23:50:05.768774+0000","last_fullsized":"2024-01-26T23:50:05.768774+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:21:30.381652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"157'1","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.772218+0000","last_change":"2024-01-26T23:50:04.747392+0000","last_active":"2024-01-26T23:50:05.772218+0000","last_peered":"2024-01-26T23:50:05.772218+0000","last_clean":"2024-01-26T23:50:05.772218+0000","last_became_active":"2024-01-26T23:50:04.747098+0000","last_became_peered":"2024-01-26T23:50:04.747098+0000","last_unstale":"2024-01-26T23:50:05.772218+0000","last_undegraded":"2024-01-26T23:50:05.772218+0000","last_fullsized":"2024-01-26T23:50:05.772218+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-27T23:51:42.125836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.773521+0000","last_change":"2024-01-26T23:50:05.770940+0000","last_active":"2024-01-26T23:50:05.773521+0000","last_peered":"2024-01-26T23:50:05.773521+0000","last_clean":"2024-01-26T23:50:05.773521+0000","last_became_active":"2024-01-26T23:50:05.770673+0000","last_became_peered":"2024-01-26T23:50:05.770673+0000","last_unstale":"2024-01-26T23:50:05.773521+0000","last_undegraded":"2024-01-26T23:50:05.773521+0000","last_fullsized":"2024-01-26T23:50:05.773521+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:12.635232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"157'2","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.774001+0000","last_change":"2024-01-26T23:50:04.758668+0000","last_active":"2024-01-26T23:50:05.774001+0000","last_peered":"2024-01-26T23:50:05.774001+0000","last_clean":"2024-01-26T23:50:05.774001+0000","last_became_active":"2024-01-26T23:50:04.758377+0000","last_became_peered":"2024-01-26T23:50:04.758377+0000","last_unstale":"2024-01-26T23:50:05.774001+0000","last_undegraded":"2024-01-26T23:50:05.774001+0000","last_fullsized":"2024-01-26T23:50:05.774001+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:19:25.221567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.773534+0000","last_change":"2024-01-26T23:50:05.768208+0000","last_active":"2024-01-26T23:50:05.773534+0000","last_peered":"2024-01-26T23:50:05.773534+0000","last_clean":"2024-01-26T23:50:05.773534+0000","last_became_active":"2024-01-26T23:50:05.767161+0000","last_became_peered":"2024-01-26T23:50:05.767161+0000","last_unstale":"2024-01-26T23:50:05.773534+0000","last_undegraded":"2024-01-26T23:50:05.773534+0000","last_fullsized":"2024-01-26T23:50:05.773534+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:31.322920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"157'1","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.774114+0000","last_change":"2024-01-26T23:50:04.758083+0000","last_active":"2024-01-26T23:50:05.774114+0000","last_peered":"2024-01-26T23:50:05.774114+0000","last_clean":"2024-01-26T23:50:05.774114+0000","last_became_active":"2024-01-26T23:50:04.757807+0000","last_became_peered":"2024-01-26T23:50:04.757807+0000","last_unstale":"2024-01-26T23:50:05.774114+0000","last_undegraded":"2024-01-26T23:50:05.774114+0000","last_fullsized":"2024-01-26T23:50:05.774114+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:10.357938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765524+0000","last_change":"2024-01-26T23:50:05.761833+0000","last_active":"2024-01-26T23:50:05.765524+0000","last_peered":"2024-01-26T23:50:05.765524+0000","last_clean":"2024-01-26T23:50:05.765524+0000","last_became_active":"2024-01-26T23:50:05.761121+0000","last_became_peered":"2024-01-26T23:50:05.761121+0000","last_unstale":"2024-01-26T23:50:05.765524+0000","last_undegraded":"2024-01-26T23:50:05.765524+0000","last_fullsized":"2024-01-26T23:50:05.765524+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:45:06.875004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.745077+0000","last_change":"2024-01-26T23:50:04.759073+0000","last_active":"2024-01-26T23:50:05.745077+0000","last_peered":"2024-01-26T23:50:05.745077+0000","last_clean":"2024-01-26T23:50:05.745077+0000","last_became_active":"2024-01-26T23:50:04.758779+0000","last_became_peered":"2024-01-26T23:50:04.758779+0000","last_unstale":"2024-01-26T23:50:05.745077+0000","last_undegraded":"2024-01-26T23:50:05.745077+0000","last_fullsized":"2024-01-26T23:50:05.745077+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:10:03.148195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.761219+0000","last_change":"2024-01-26T23:50:05.754281+0000","last_active":"2024-01-26T23:50:05.761219+0000","last_peered":"2024-01-26T23:50:05.761219+0000","last_clean":"2024-01-26T23:50:05.761219+0000","last_became_active":"2024-01-26T23:50:05.753970+0000","last_became_peered":"2024-01-26T23:50:05.753970+0000","last_unstale":"2024-01-26T23:50:05.761219+0000","last_undegraded":"2024-01-26T23:50:05.761219+0000","last_fullsized":"2024-01-26T23:50:05.761219+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:25:41.882518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"157'2","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.772208+0000","last_change":"2024-01-26T23:50:04.754231+0000","last_active":"2024-01-26T23:50:05.772208+0000","last_peered":"2024-01-26T23:50:05.772208+0000","last_clean":"2024-01-26T23:50:05.772208+0000","last_became_active":"2024-01-26T23:50:04.753393+0000","last_became_peered":"2024-01-26T23:50:04.753393+0000","last_unstale":"2024-01-26T23:50:05.772208+0000","last_undegraded":"2024-01-26T23:50:05.772208+0000","last_fullsized":"2024-01-26T23:50:05.772208+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:12:47.923807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765402+0000","last_change":"2024-01-26T23:50:05.758368+0000","last_active":"2024-01-26T23:50:05.765402+0000","last_peered":"2024-01-26T23:50:05.765402+0000","last_clean":"2024-01-26T23:50:05.765402+0000","last_became_active":"2024-01-26T23:50:05.758034+0000","last_became_peered":"2024-01-26T23:50:05.758034+0000","last_unstale":"2024-01-26T23:50:05.765402+0000","last_undegraded":"2024-01-26T23:50:05.765402+0000","last_fullsized":"2024-01-26T23:50:05.765402+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:20:06.258716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.748517+0000","last_change":"2024-01-26T23:50:04.754377+0000","last_active":"2024-01-26T23:50:05.748517+0000","last_peered":"2024-01-26T23:50:05.748517+0000","last_clean":"2024-01-26T23:50:05.748517+0000","last_became_active":"2024-01-26T23:50:04.753680+0000","last_became_peered":"2024-01-26T23:50:04.753680+0000","last_unstale":"2024-01-26T23:50:05.748517+0000","last_undegraded":"2024-01-26T23:50:05.748517+0000","last_fullsized":"2024-01-26T23:50:05.748517+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:15.214318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.768350+0000","last_change":"2024-01-26T23:50:05.760353+0000","last_active":"2024-01-26T23:50:05.768350+0000","last_peered":"2024-01-26T23:50:05.768350+0000","last_clean":"2024-01-26T23:50:05.768350+0000","last_became_active":"2024-01-26T23:50:05.758430+0000","last_became_peered":"2024-01-26T23:50:05.758430+0000","last_unstale":"2024-01-26T23:50:05.768350+0000","last_undegraded":"2024-01-26T23:50:05.768350+0000","last_fullsized":"2024-01-26T23:50:05.768350+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:21:30.381652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.754999+0000","last_change":"2024-01-26T23:50:04.757707+0000","last_active":"2024-01-26T23:50:05.754999+0000","last_peered":"2024-01-26T23:50:05.754999+0000","last_clean":"2024-01-26T23:50:05.754999+0000","last_became_active":"2024-01-26T23:50:04.757423+0000","last_became_peered":"2024-01-26T23:50:04.757423+0000","last_unstale":"2024-01-26T23:50:05.754999+0000","last_undegraded":"2024-01-26T23:50:05.754999+0000","last_fullsized":"2024-01-26T23:50:05.754999+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:22:23.148281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.764202+0000","last_change":"2024-01-26T23:50:05.759105+0000","last_active":"2024-01-26T23:50:05.764202+0000","last_peered":"2024-01-26T23:50:05.764202+0000","last_clean":"2024-01-26T23:50:05.764202+0000","last_became_active":"2024-01-26T23:50:05.758813+0000","last_became_peered":"2024-01-26T23:50:05.758813+0000","last_unstale":"2024-01-26T23:50:05.764202+0000","last_undegraded":"2024-01-26T23:50:05.764202+0000","last_fullsized":"2024-01-26T23:50:05.764202+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:11.363471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.741299+0000","last_change":"2024-01-26T23:50:04.753988+0000","last_active":"2024-01-26T23:50:05.741299+0000","last_peered":"2024-01-26T23:50:05.741299+0000","last_clean":"2024-01-26T23:50:05.741299+0000","last_became_active":"2024-01-26T23:50:04.753703+0000","last_became_peered":"2024-01-26T23:50:04.753703+0000","last_unstale":"2024-01-26T23:50:05.741299+0000","last_undegraded":"2024-01-26T23:50:05.741299+0000","last_fullsized":"2024-01-26T23:50:05.741299+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:58:33.808722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.767110+0000","last_change":"2024-01-26T23:50:05.757932+0000","last_active":"2024-01-26T23:50:05.767110+0000","last_peered":"2024-01-26T23:50:05.767110+0000","last_clean":"2024-01-26T23:50:05.767110+0000","last_became_active":"2024-01-26T23:50:05.757334+0000","last_became_peered":"2024-01-26T23:50:05.757334+0000","last_unstale":"2024-01-26T23:50:05.767110+0000","last_undegraded":"2024-01-26T23:50:05.767110+0000","last_fullsized":"2024-01-26T23:50:05.767110+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:21:26.665456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"157'2","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.775403+0000","last_change":"2024-01-26T23:50:04.759684+0000","last_active":"2024-01-26T23:50:05.775403+0000","last_peered":"2024-01-26T23:50:05.775403+0000","last_clean":"2024-01-26T23:50:05.775403+0000","last_became_active":"2024-01-26T23:50:04.758634+0000","last_became_peered":"2024-01-26T23:50:04.758634+0000","last_unstale":"2024-01-26T23:50:05.775403+0000","last_undegraded":"2024-01-26T23:50:05.775403+0000","last_fullsized":"2024-01-26T23:50:05.775403+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:46.321004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.766709+0000","last_change":"2024-01-26T23:50:05.760420+0000","last_active":"2024-01-26T23:50:05.766709+0000","last_peered":"2024-01-26T23:50:05.766709+0000","last_clean":"2024-01-26T23:50:05.766709+0000","last_became_active":"2024-01-26T23:50:05.760141+0000","last_became_peered":"2024-01-26T23:50:05.760141+0000","last_unstale":"2024-01-26T23:50:05.766709+0000","last_undegraded":"2024-01-26T23:50:05.766709+0000","last_fullsized":"2024-01-26T23:50:05.766709+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:16.574844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.753651+0000","last_change":"2024-01-26T23:50:04.754921+0000","last_active":"2024-01-26T23:50:05.753651+0000","last_peered":"2024-01-26T23:50:05.753651+0000","last_clean":"2024-01-26T23:50:05.753651+0000","last_became_active":"2024-01-26T23:50:04.754650+0000","last_became_peered":"2024-01-26T23:50:04.754650+0000","last_unstale":"2024-01-26T23:50:05.753651+0000","last_undegraded":"2024-01-26T23:50:05.753651+0000","last_fullsized":"2024-01-26T23:50:05.753651+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:26:30.317387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.769745+0000","last_change":"2024-01-26T23:50:05.758521+0000","last_active":"2024-01-26T23:50:05.769745+0000","last_peered":"2024-01-26T23:50:05.769745+0000","last_clean":"2024-01-26T23:50:05.769745+0000","last_became_active":"2024-01-26T23:50:05.758266+0000","last_became_peered":"2024-01-26T23:50:05.758266+0000","last_unstale":"2024-01-26T23:50:05.769745+0000","last_undegraded":"2024-01-26T23:50:05.769745+0000","last_fullsized":"2024-01-26T23:50:05.769745+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:21:30.381652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.752131+0000","last_change":"2024-01-26T23:50:04.758950+0000","last_active":"2024-01-26T23:50:05.752131+0000","last_peered":"2024-01-26T23:50:05.752131+0000","last_clean":"2024-01-26T23:50:05.752131+0000","last_became_active":"2024-01-26T23:50:04.758691+0000","last_became_peered":"2024-01-26T23:50:04.758691+0000","last_unstale":"2024-01-26T23:50:05.752131+0000","last_undegraded":"2024-01-26T23:50:05.752131+0000","last_fullsized":"2024-01-26T23:50:05.752131+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:23:08.813586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765500+0000","last_change":"2024-01-26T23:50:05.761639+0000","last_active":"2024-01-26T23:50:05.765500+0000","last_peered":"2024-01-26T23:50:05.765500+0000","last_clean":"2024-01-26T23:50:05.765500+0000","last_became_active":"2024-01-26T23:50:05.760100+0000","last_became_peered":"2024-01-26T23:50:05.760100+0000","last_unstale":"2024-01-26T23:50:05.765500+0000","last_undegraded":"2024-01-26T23:50:05.765500+0000","last_fullsized":"2024-01-26T23:50:05.765500+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:13:16.808150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.740884+0000","last_change":"2024-01-26T23:50:04.748757+0000","last_active":"2024-01-26T23:50:05.740884+0000","last_peered":"2024-01-26T23:50:05.740884+0000","last_clean":"2024-01-26T23:50:05.740884+0000","last_became_active":"2024-01-26T23:50:04.748487+0000","last_became_peered":"2024-01-26T23:50:04.748487+0000","last_unstale":"2024-01-26T23:50:05.740884+0000","last_undegraded":"2024-01-26T23:50:05.740884+0000","last_fullsized":"2024-01-26T23:50:05.740884+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:54:10.044522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.763753+0000","last_change":"2024-01-26T23:50:05.757686+0000","last_active":"2024-01-26T23:50:05.763753+0000","last_peered":"2024-01-26T23:50:05.763753+0000","last_clean":"2024-01-26T23:50:05.763753+0000","last_became_active":"2024-01-26T23:50:05.757374+0000","last_became_peered":"2024-01-26T23:50:05.757374+0000","last_unstale":"2024-01-26T23:50:05.763753+0000","last_undegraded":"2024-01-26T23:50:05.763753+0000","last_fullsized":"2024-01-26T23:50:05.763753+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:25:41.882518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.744659+0000","last_change":"2024-01-26T23:50:04.759150+0000","last_active":"2024-01-26T23:50:05.744659+0000","last_peered":"2024-01-26T23:50:05.744659+0000","last_clean":"2024-01-26T23:50:05.744659+0000","last_became_active":"2024-01-26T23:50:04.758883+0000","last_became_peered":"2024-01-26T23:50:04.758883+0000","last_unstale":"2024-01-26T23:50:05.744659+0000","last_undegraded":"2024-01-26T23:50:05.744659+0000","last_fullsized":"2024-01-26T23:50:05.744659+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:49:45.035222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.763807+0000","last_change":"2024-01-26T23:50:05.758466+0000","last_active":"2024-01-26T23:50:05.763807+0000","last_peered":"2024-01-26T23:50:05.763807+0000","last_clean":"2024-01-26T23:50:05.763807+0000","last_became_active":"2024-01-26T23:50:05.758164+0000","last_became_peered":"2024-01-26T23:50:05.758164+0000","last_unstale":"2024-01-26T23:50:05.763807+0000","last_undegraded":"2024-01-26T23:50:05.763807+0000","last_fullsized":"2024-01-26T23:50:05.763807+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:08:15.330869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.749370+0000","last_change":"2024-01-26T23:50:04.754886+0000","last_active":"2024-01-26T23:50:05.749370+0000","last_peered":"2024-01-26T23:50:05.749370+0000","last_clean":"2024-01-26T23:50:05.749370+0000","last_became_active":"2024-01-26T23:50:04.754622+0000","last_became_peered":"2024-01-26T23:50:04.754622+0000","last_unstale":"2024-01-26T23:50:05.749370+0000","last_undegraded":"2024-01-26T23:50:05.749370+0000","last_fullsized":"2024-01-26T23:50:05.749370+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:10.357938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.763742+0000","last_change":"2024-01-26T23:50:05.758903+0000","last_active":"2024-01-26T23:50:05.763742+0000","last_peered":"2024-01-26T23:50:05.763742+0000","last_clean":"2024-01-26T23:50:05.763742+0000","last_became_active":"2024-01-26T23:50:05.758621+0000","last_became_peered":"2024-01-26T23:50:05.758621+0000","last_unstale":"2024-01-26T23:50:05.763742+0000","last_undegraded":"2024-01-26T23:50:05.763742+0000","last_fullsized":"2024-01-26T23:50:05.763742+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:21:30.381652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.740736+0000","last_change":"2024-01-26T23:50:04.747999+0000","last_active":"2024-01-26T23:50:05.740736+0000","last_peered":"2024-01-26T23:50:05.740736+0000","last_clean":"2024-01-26T23:50:05.740736+0000","last_became_active":"2024-01-26T23:50:04.747724+0000","last_became_peered":"2024-01-26T23:50:04.747724+0000","last_unstale":"2024-01-26T23:50:05.740736+0000","last_undegraded":"2024-01-26T23:50:05.740736+0000","last_fullsized":"2024-01-26T23:50:05.740736+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:50.221794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.765610+0000","last_change":"2024-01-26T23:50:05.757871+0000","last_active":"2024-01-26T23:50:05.765610+0000","last_peered":"2024-01-26T23:50:05.765610+0000","last_clean":"2024-01-26T23:50:05.765610+0000","last_became_active":"2024-01-26T23:50:05.757619+0000","last_became_peered":"2024-01-26T23:50:05.757619+0000","last_unstale":"2024-01-26T23:50:05.765610+0000","last_undegraded":"2024-01-26T23:50:05.765610+0000","last_fullsized":"2024-01-26T23:50:05.765610+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:45:58.463554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.747306+0000","last_change":"2024-01-26T23:50:04.760619+0000","last_active":"2024-01-26T23:50:05.747306+0000","last_peered":"2024-01-26T23:50:05.747306+0000","last_clean":"2024-01-26T23:50:05.747306+0000","last_became_active":"2024-01-26T23:50:04.759526+0000","last_became_peered":"2024-01-26T23:50:04.759526+0000","last_unstale":"2024-01-26T23:50:05.747306+0000","last_undegraded":"2024-01-26T23:50:05.747306+0000","last_fullsized":"2024-01-26T23:50:05.747306+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:50.221794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":9,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.769081+0000","last_change":"2024-01-26T23:50:05.763347+0000","last_active":"2024-01-26T23:50:05.769081+0000","last_peered":"2024-01-26T23:50:05.769081+0000","last_clean":"2024-01-26T23:50:05.769081+0000","last_became_active":"2024-01-26T23:50:05.763075+0000","last_became_peered":"2024-01-26T23:50:05.763075+0000","last_unstale":"2024-01-26T23:50:05.769081+0000","last_undegraded":"2024-01-26T23:50:05.769081+0000","last_fullsized":"2024-01-26T23:50:05.769081+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:04.728443+0000","last_clean_scrub_stamp":"2024-01-26T23:50:04.728443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:45:58.463554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":10,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.749257+0000","last_change":"2024-01-26T23:50:04.759360+0000","last_active":"2024-01-26T23:50:05.749257+0000","last_peered":"2024-01-26T23:50:05.749257+0000","last_clean":"2024-01-26T23:50:05.749257+0000","last_became_active":"2024-01-26T23:50:04.758570+0000","last_became_peered":"2024-01-26T23:50:04.758570+0000","last_unstale":"2024-01-26T23:50:05.749257+0000","last_undegraded":"2024-01-26T23:50:05.749257+0000","last_fullsized":"2024-01-26T23:50:05.749257+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:50:03.722909+0000","last_clean_scrub_stamp":"2024-01-26T23:50:03.722909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:54.605829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"149'8606","reported_seq":9194,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-01-26T23:50:05.748691+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-26T23:50:05.748691+0000","last_peered":"2024-01-26T23:50:05.748691+0000","last_clean":"2024-01-26T23:50:05.748691+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-26T23:50:05.748691+0000","last_undegraded":"2024-01-26T23:50:05.748691+0000","last_fullsized":"2024-01-26T23:50:05.748691+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8606,"log_dups_size":0,"ondisk_log_size":8606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1355,"num_read_kb":2726,"num_write":400,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1355,"num_read_kb":2726,"num_write":400,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8606,"ondisk_log_size":8606,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575091,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312096,"kb_used_data":2520,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059744,"statfs":{"total":96636764160,"available":96317177856,"internally_reserved":0,"allocated":2580480,"data_stored":1610661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51500000000000001,"15min":0.56200000000000006},"min":{"1min":0.40699999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.74199999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52700000000000002,"15min":0.57599999999999996},"min":{"1min":0.35899999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67200000000000004,"5min":0.82499999999999996,"15min":0.83999999999999997},"last":0.56299999999999994}]},{"osd":1,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54000000000000004,"15min":0.57399999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.77600000000000002,"5min":0.77800000000000002,"15min":0.82899999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55100000000000005,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.70399999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.57199999999999995},"min":{"1min":0.38700000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69199999999999995,"5min":0.80000000000000004,"15min":0.81699999999999995},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.38900000000000001,"5min":0.30199999999999999,"15min":0.27400000000000002},"max":{"1min":0.746,"5min":0.81999999999999995,"15min":0.88600000000000001},"last":0.625}]},{"osd":3,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54700000000000004,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.377,"15min":0.377},"max":{"1min":0.70399999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57499999999999996,"15min":0.59699999999999998},"min":{"1min":0.34399999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77900000000000003,"5min":0.86199999999999999,"15min":0.86399999999999999},"last":0.66100000000000003}]},{"osd":4,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.438,"15min":0.45200000000000001},"min":{"1min":0.34599999999999997,"5min":0.252,"15min":0.247},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.93899999999999995},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.45300000000000001,"15min":0.47399999999999998},"min":{"1min":0.32400000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67300000000000004,"5min":0.72799999999999998,"15min":0.73799999999999999},"last":0.51900000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47599999999999998,"15min":0.48699999999999999},"min":{"1min":0.30499999999999999,"5min":0.27200000000000002,"15min":0.22800000000000001},"max":{"1min":0.749,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.504,"15min":0.52300000000000002},"min":{"1min":0.35799999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73399999999999999,"5min":0.74199999999999999,"15min":0.90200000000000002},"last":0.60899999999999999}]},{"osd":6,"last update":"Fri Jan 26 23:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53100000000000003,"15min":0.54800000000000004},"min":{"1min":0.41699999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76400000000000001,"5min":0.78700000000000003,"15min":0.95899999999999996},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.52700000000000002,"15min":0.54000000000000004},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75900000000000001,"5min":0.79500000000000004,"15min":0.85199999999999998},"last":0.64000000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575091,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303316,"kb_used_data":1932,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068524,"statfs":{"total":96636764160,"available":96326168576,"internally_reserved":0,"allocated":1978368,"data_stored":1019311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52900000000000003,"15min":0.56499999999999995},"min":{"1min":0.40300000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.423,"15min":1.423},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.57899999999999996},"min":{"1min":0.36099999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.91100000000000003},"last":0.57899999999999996}]},{"osd":1,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51700000000000002,"15min":0.55800000000000005},"min":{"1min":0.32200000000000001,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.58399999999999996},"min":{"1min":0.38200000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.66800000000000004,"5min":1.498,"15min":1.498},"last":0.53900000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52800000000000002,"15min":0.56999999999999995},"min":{"1min":0.38100000000000001,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.85799999999999998},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52600000000000002,"15min":0.57099999999999995},"min":{"1min":0.377,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69299999999999995,"5min":0.77000000000000002,"15min":0.82799999999999996},"last":0.70199999999999996}]},{"osd":3,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57199999999999995,"15min":0.59099999999999997},"min":{"1min":0.34699999999999998,"5min":0.33800000000000002,"15min":0.29999999999999999},"max":{"1min":0.746,"5min":1.595,"15min":1.595},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56899999999999995,"15min":0.60599999999999998},"min":{"1min":0.45600000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.751,"5min":1.369,"15min":1.369},"last":0.81100000000000005}]},{"osd":4,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.443,"15min":0.46200000000000002},"min":{"1min":0.33800000000000002,"5min":0.245,"15min":0.22},"max":{"1min":0.67000000000000004,"5min":1.556,"15min":1.556},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.47399999999999998,"15min":0.48199999999999998},"min":{"1min":0.33200000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.86199999999999999},"last":0.72899999999999998}]},{"osd":5,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45500000000000002,"15min":0.47099999999999997},"min":{"1min":0.32700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67900000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.51700000000000002,"15min":0.52000000000000002},"min":{"1min":0.30499999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.70999999999999996,"5min":1.47,"15min":1.47},"last":0.622}]},{"osd":7,"last update":"Fri Jan 26 23:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.5,"15min":0.51800000000000002},"min":{"1min":0.40100000000000002,"5min":0.26700000000000002,"15min":0.246},"max":{"1min":0.80200000000000005,"5min":0.81799999999999995,"15min":0.85899999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50900000000000001,"15min":0.52400000000000002},"min":{"1min":0.32300000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.80400000000000005},"last":0.747}]}]},{"osd":1,"up_from":13,"seq":55834575091,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307408,"kb_used_data":1928,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064432,"statfs":{"total":96636764160,"available":96321978368,"internally_reserved":0,"allocated":1974272,"data_stored":1018737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41199999999999998,"15min":0.40600000000000003},"min":{"1min":0.28100000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.72599999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.41999999999999998,"15min":0.439},"min":{"1min":0.28100000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.61499999999999999,"5min":0.78600000000000003,"15min":1.337},"last":0.36499999999999999}]},{"osd":2,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.39700000000000002,"15min":0.40899999999999997},"min":{"1min":0.28699999999999998,"5min":0.254,"15min":0.23599999999999999},"max":{"1min":0.66300000000000003,"5min":0.68799999999999994,"15min":0.73299999999999998},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.46300000000000002,"15min":0.47199999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59299999999999997,"5min":0.74099999999999999,"15min":0.77300000000000002},"last":0.52700000000000002}]},{"osd":3,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51300000000000001,"15min":0.51400000000000001},"min":{"1min":0.38400000000000001,"5min":0.34399999999999997,"15min":0.33200000000000002},"max":{"1min":0.73499999999999999,"5min":0.78900000000000003,"15min":1.782},"last":0.436},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.32900000000000001,"5min":0.32000000000000001,"15min":0.26600000000000001},"max":{"1min":0.70199999999999996,"5min":0.71499999999999997,"15min":1.379},"last":0.44800000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60699999999999998,"15min":0.64200000000000002},"min":{"1min":0.45100000000000001,"5min":0.40799999999999997,"15min":0.40200000000000002},"max":{"1min":0.85199999999999998,"5min":0.89200000000000002,"15min":1.0820000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.64500000000000002,"15min":0.65700000000000003},"min":{"1min":0.49199999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.66700000000000004}]},{"osd":5,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.64000000000000001,"15min":0.66800000000000004},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45000000000000001},"max":{"1min":0.86499999999999999,"5min":0.91100000000000003,"15min":1.208},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.63400000000000001,"15min":0.65900000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.42499999999999999},"max":{"1min":0.85099999999999998,"5min":0.96399999999999997,"15min":1.018},"last":0.65800000000000003}]},{"osd":6,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.63800000000000001,"15min":0.67000000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.42599999999999999},"max":{"1min":0.97399999999999998,"5min":1.022,"15min":1.2529999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.65000000000000002,"15min":0.67700000000000005},"min":{"1min":0.47899999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92300000000000004,"5min":0.93200000000000005,"15min":1.022},"last":0.69599999999999995}]},{"osd":7,"last update":"Fri Jan 26 23:49:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.68300000000000005,"15min":0.70699999999999996},"min":{"1min":0.5,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.75},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.71899999999999997},"min":{"1min":0.505,"5min":0.496,"15min":0.496},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.296},"last":0.64000000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575090,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307992,"kb_used_data":2512,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063848,"statfs":{"total":96636764160,"available":96321380352,"internally_reserved":0,"allocated":2572288,"data_stored":1609091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.40600000000000003},"min":{"1min":0.25800000000000001,"5min":0.23899999999999999,"15min":0.23100000000000001},"max":{"1min":0.88500000000000001,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.42999999999999999,"15min":0.45100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65800000000000003,"5min":1.363,"15min":1.4490000000000001},"last":0.39900000000000002}]},{"osd":2,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.41099999999999998,"15min":0.42299999999999999},"min":{"1min":0.28899999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.89600000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.33900000000000002,"5min":0.32700000000000001,"15min":0.28000000000000003},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.36599999999999999}]},{"osd":3,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.49199999999999999,"15min":0.47399999999999998},"min":{"1min":0.32800000000000001,"5min":0.32300000000000001,"15min":0.30599999999999999},"max":{"1min":0.85199999999999998,"5min":2.145,"15min":2.145},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36099999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":1.198},"last":0.40799999999999997}]},{"osd":4,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60899999999999999,"15min":0.63800000000000001},"min":{"1min":0.41699999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78600000000000003,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.59699999999999998,"15min":0.63700000000000001},"min":{"1min":0.44400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.94299999999999995},"last":0.64900000000000002}]},{"osd":5,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":1.329,"15min":1.329},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.627,"5min":0.66300000000000003,"15min":0.67000000000000004},"min":{"1min":0.46899999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82799999999999996,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.55800000000000005}]},{"osd":6,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.61799999999999999,"15min":0.65100000000000002},"min":{"1min":0.47299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.89800000000000002,"5min":0.98699999999999999,"15min":1.032},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.65400000000000003,"15min":0.68000000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.41799999999999998},"max":{"1min":0.93100000000000005,"5min":1.286,"15min":1.286},"last":0.76800000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.70599999999999996,"15min":0.81699999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.435},"max":{"1min":0.81799999999999995,"5min":4.8710000000000004,"15min":25.965},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.73899999999999999,"15min":0.86099999999999999},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.433},"max":{"1min":0.88200000000000001,"5min":4.9199999999999999,"15min":33.085000000000001},"last":0.67400000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575090,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307412,"kb_used_data":1932,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064428,"statfs":{"total":96636764160,"available":96321974272,"internally_reserved":0,"allocated":1978368,"data_stored":1019255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.379,"15min":0.39100000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.255},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.43099999999999999,"15min":0.42599999999999999},"min":{"1min":0.29699999999999999,"5min":0.255,"15min":0.255},"max":{"1min":0.59899999999999998,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.34000000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.435,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.26900000000000002,"15min":0.248},"max":{"1min":0.63500000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52200000000000002,"15min":0.51100000000000001},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.68300000000000005,"5min":1.171,"15min":1.171},"last":0.40400000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.46899999999999997,"15min":0.48399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.313},"max":{"1min":0.64300000000000002,"5min":0.72799999999999998,"15min":0.74099999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47699999999999998},"min":{"1min":0.32900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70599999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.375}]},{"osd":4,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59599999999999997,"15min":0.628},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.087},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.625,"15min":0.64300000000000002},"min":{"1min":0.39100000000000001,"5min":0.371,"15min":0.371},"max":{"1min":1.006,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.56699999999999995}]},{"osd":5,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.63900000000000001,"15min":0.65500000000000003},"min":{"1min":0.42299999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94699999999999995,"5min":1.24,"15min":1.24},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.65200000000000002,"15min":0.67400000000000004},"min":{"1min":0.48799999999999999,"5min":0.41799999999999998,"15min":0.35199999999999998},"max":{"1min":0.874,"5min":0.97699999999999998,"15min":1.085},"last":0.627}]},{"osd":6,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65700000000000003,"15min":0.67700000000000005},"min":{"1min":0.41199999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88600000000000001,"5min":1.022,"15min":1.1080000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.65200000000000002,"15min":0.67800000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.38800000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.1299999999999999},"last":0.68899999999999995}]},{"osd":7,"last update":"Fri Jan 26 23:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68400000000000005,"15min":0.70099999999999996},"min":{"1min":0.46400000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.88800000000000001,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.69699999999999995,"15min":0.72499999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.42099999999999999},"max":{"1min":0.96899999999999997,"5min":1.05,"15min":1.139},"last":0.58899999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575089,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307420,"kb_used_data":1940,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064420,"statfs":{"total":96636764160,"available":96321966080,"internally_reserved":0,"allocated":1986560,"data_stored":1020273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38400000000000001,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.55400000000000005,"5min":0.59699999999999998,"15min":1.8540000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.42899999999999999,"15min":0.438},"min":{"1min":0.29799999999999999,"5min":0.25800000000000001,"15min":0.255},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":1.2290000000000001},"last":0.379}]},{"osd":1,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44500000000000001,"15min":0.45200000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":1.633},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.47999999999999998,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":1.0760000000000001},"last":0.53700000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48399999999999999,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65000000000000002,"5min":0.70399999999999996,"15min":1.4059999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.44500000000000001,"15min":0.45800000000000002},"min":{"1min":0.34100000000000003,"5min":0.29499999999999998,"15min":0.28299999999999997},"max":{"1min":0.629,"5min":0.69399999999999995,"15min":1.1870000000000001},"last":0.41599999999999998}]},{"osd":4,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.63,"15min":0.65800000000000003},"min":{"1min":0.48299999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":3.306},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.44800000000000001,"5min":0.40100000000000002,"15min":0.379},"max":{"1min":0.81499999999999995,"5min":0.85199999999999998,"15min":1.008},"last":0.67100000000000004}]},{"osd":5,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.63600000000000001,"15min":0.65200000000000002},"min":{"1min":0.48599999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.85199999999999998,"5min":0.90400000000000003,"15min":1.593},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.47599999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.86299999999999999,"5min":0.92300000000000004,"15min":1.3819999999999999},"last":0.57899999999999996}]},{"osd":6,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.66900000000000004},"min":{"1min":0.42299999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":1.506},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67200000000000004,"15min":0.68999999999999995},"min":{"1min":0.55600000000000005,"5min":0.46700000000000003,"15min":0.44400000000000001},"max":{"1min":0.88500000000000001,"5min":0.89000000000000001,"15min":1.427},"last":0.62}]},{"osd":7,"last update":"Fri Jan 26 23:49:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64400000000000002,"15min":0.68799999999999994},"min":{"1min":0.46700000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":3.238},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.71199999999999997,"15min":0.73399999999999999},"min":{"1min":0.54200000000000004,"5min":0.51900000000000002,"15min":0.48899999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":3.351},"last":0.69399999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575091,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303308,"kb_used_data":1924,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068532,"statfs":{"total":96636764160,"available":96326176768,"internally_reserved":0,"allocated":1970176,"data_stored":1018667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52300000000000002,"15min":0.56200000000000006},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.72499999999999998,"5min":0.78900000000000003,"15min":1.21},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.32700000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.751,"5min":5.367,"15min":5.367},"last":0.52700000000000002}]},{"osd":1,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.56899999999999995,"15min":0.58599999999999997},"min":{"1min":0.42699999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.79300000000000004,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.627},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54100000000000004,"15min":0.57199999999999995},"min":{"1min":0.41399999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73599999999999999,"5min":0.80600000000000005,"15min":0.81699999999999995},"last":0.55400000000000005}]},{"osd":2,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54500000000000004,"15min":0.56399999999999995},"min":{"1min":0.38500000000000001,"5min":0.33900000000000002,"15min":0.32700000000000001},"max":{"1min":0.751,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57099999999999995,"15min":0.59499999999999997},"min":{"1min":0.44500000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73399999999999999,"5min":0.86199999999999999,"15min":1.7050000000000001},"last":0.67400000000000004}]},{"osd":3,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56899999999999995,"15min":0.58499999999999996},"min":{"1min":0.45600000000000002,"5min":0.32200000000000001,"15min":0.318},"max":{"1min":0.70499999999999996,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56599999999999995,"15min":0.58099999999999996},"min":{"1min":0.433,"5min":0.37,"15min":0.35699999999999998},"max":{"1min":0.78300000000000003,"5min":0.94199999999999995,"15min":1.05},"last":0.65500000000000003}]},{"osd":5,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45400000000000001,"15min":0.46500000000000002},"min":{"1min":0.32300000000000001,"5min":0.25900000000000001,"15min":0.247},"max":{"1min":0.60499999999999998,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.47899999999999998,"15min":0.48599999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64300000000000002,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.40000000000000002}]},{"osd":6,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.46200000000000002,"15min":0.47399999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27200000000000002},"max":{"1min":0.63500000000000001,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.436},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.38600000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72099999999999997,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.42199999999999999}]},{"osd":7,"last update":"Fri Jan 26 23:49:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52600000000000002,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.72499999999999998,"5min":0.879,"15min":0.879},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55800000000000005,"15min":0.57199999999999995},"min":{"1min":0.437,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.74199999999999999,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.495}]}]},{"osd":5,"up_from":13,"seq":55834575091,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303312,"kb_used_data":1928,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068528,"statfs":{"total":96636764160,"available":96326172672,"internally_reserved":0,"allocated":1974272,"data_stored":1018689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.56399999999999995},"min":{"1min":0.35199999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.71099999999999997,"5min":0.81799999999999995,"15min":2.0289999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52700000000000002,"15min":0.56699999999999995},"min":{"1min":0.41999999999999998,"5min":0.29999999999999999,"15min":0.28899999999999998},"max":{"1min":0.70799999999999996,"5min":0.77400000000000002,"15min":0.82199999999999995},"last":0.373}]},{"osd":1,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56299999999999994,"15min":0.60399999999999998},"min":{"1min":0.443,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":0.754,"5min":0.79500000000000004,"15min":2.3319999999999999},"last":0.441},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59899999999999998,"15min":0.626},"min":{"1min":0.45800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76400000000000001,"5min":0.80500000000000005,"15min":1.1739999999999999},"last":0.53500000000000003}]},{"osd":2,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53700000000000003,"15min":0.55700000000000005},"min":{"1min":0.36799999999999999,"5min":0.34499999999999997,"15min":0.30099999999999999},"max":{"1min":0.68400000000000005,"5min":0.79400000000000004,"15min":0.83799999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55300000000000005,"15min":0.58699999999999997},"min":{"1min":0.41299999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72899999999999998,"5min":0.81100000000000005,"15min":2.0600000000000001},"last":0.40500000000000003}]},{"osd":3,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55300000000000005,"15min":0.57499999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.314},"max":{"1min":0.69699999999999995,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55900000000000005,"15min":0.58699999999999997},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73999999999999999,"5min":0.77900000000000003,"15min":2.5630000000000002},"last":0.50800000000000001}]},{"osd":4,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.42899999999999999,"15min":0.45500000000000002},"min":{"1min":0.313,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.65100000000000002,"5min":0.67200000000000004,"15min":2.3700000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45900000000000002,"15min":0.48099999999999998},"min":{"1min":0.32600000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.68200000000000005,"5min":0.82699999999999996,"15min":0.89200000000000002},"last":0.41999999999999998}]},{"osd":6,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.47199999999999998},"min":{"1min":0.34300000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66900000000000004,"5min":0.67700000000000005,"15min":2.5990000000000002},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.49299999999999999,"15min":0.51400000000000001},"min":{"1min":0.374,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71399999999999997,"5min":0.76200000000000001,"15min":1.99},"last":0.51900000000000002}]},{"osd":7,"last update":"Fri Jan 26 23:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51400000000000001,"15min":0.53800000000000003},"min":{"1min":0.374,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73799999999999999,"5min":0.76200000000000001,"15min":3.5270000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54400000000000004,"15min":0.55900000000000005},"min":{"1min":0.43099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78900000000000003,"5min":0.81499999999999995,"15min":0.82199999999999995},"last":0.54800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-26T23:50:10.353 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-26T23:50:10.353 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-26T23:50:10.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.530+0000 7f4f0a1a6700 1 Processor -- start 2024-01-26T23:50:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.530+0000 7f4f0a1a6700 1 -- start start 2024-01-26T23:50:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.531+0000 7f4f0a1a6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4f0415dfd0 0x7f4f040ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.531+0000 7f4f0a1a6700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f0415a200 con 0x7f4f04159da0 2024-01-26T23:50:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.531+0000 7f4f0a1a6700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f040b1a60 con 0x7f4f041599d0 2024-01-26T23:50:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.531+0000 7f4f0a1a6700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f040b11c0 con 0x7f4f0415dfd0 2024-01-26T23:50:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.531+0000 7f4f03fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4f0415dfd0 0x7f4f040ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.531+0000 7f4f02ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4f04159da0 0x7f4f0415d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37780/0 (socket says 172.21.15.46:37780) 2024-01-26T23:50:10.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.531+0000 7f4f02ffd700 1 -- 172.21.15.46:0/172080031 learned_addr learned my addr 172.21.15.46:0/172080031 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:10.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.532+0000 7f4f03fff700 1 -- 172.21.15.46:0/172080031 >> v1:172.21.15.106:6789/0 conn(0x7f4f041599d0 legacy=0x7f4f040b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:10.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.532+0000 7f4f03fff700 1 -- 172.21.15.46:0/172080031 >> v1:172.21.15.46:6789/0 conn(0x7f4f04159da0 legacy=0x7f4f0415d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:10.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.532+0000 7f4f03fff700 1 -- 172.21.15.46:0/172080031 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f040b14a0 con 0x7f4f0415dfd0 2024-01-26T23:50:10.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.532+0000 7f4f03fff700 1 --2- 172.21.15.46:0/172080031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f0415dfd0 0x7f4f040ab1c0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f4efc009e60 tx=0x7f4efc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899044c516e70898 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:10.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.532+0000 7f4f027fc700 1 -- 172.21.15.46:0/172080031 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc00e070 con 0x7f4f0415dfd0 2024-01-26T23:50:10.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.532+0000 7f4f027fc700 1 -- 172.21.15.46:0/172080031 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4efc00bbf0 con 0x7f4f0415dfd0 2024-01-26T23:50:10.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.532+0000 7f4f027fc700 1 -- 172.21.15.46:0/172080031 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc012430 con 0x7f4f0415dfd0 2024-01-26T23:50:10.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.533+0000 7f4f0a1a6700 1 -- 172.21.15.46:0/172080031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f0415dfd0 msgr2=0x7f4f040ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:10.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.533+0000 7f4f0a1a6700 1 --2- 172.21.15.46:0/172080031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f0415dfd0 0x7f4f040ab1c0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f4efc009e60 tx=0x7f4efc009780 comp rx=0 tx=0).stop 2024-01-26T23:50:10.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.533+0000 7f4f0a1a6700 1 -- 172.21.15.46:0/172080031 shutdown_connections 2024-01-26T23:50:10.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.533+0000 7f4f0a1a6700 1 --2- 172.21.15.46:0/172080031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f0415dfd0 0x7f4f040ab1c0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.533+0000 7f4f0a1a6700 1 -- 172.21.15.46:0/172080031 >> 172.21.15.46:0/172080031 conn(0x7f4f0406f840 msgr2=0x7f4f04087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:10.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.533+0000 7f4f0a1a6700 1 -- 172.21.15.46:0/172080031 shutdown_connections 2024-01-26T23:50:10.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.533+0000 7f4f0a1a6700 1 -- 172.21.15.46:0/172080031 wait complete. 2024-01-26T23:50:10.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.534+0000 7f4f0a1a6700 1 Processor -- start 2024-01-26T23:50:10.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.534+0000 7f4f0a1a6700 1 -- start start 2024-01-26T23:50:10.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.535+0000 7f4f0a1a6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f041599d0 0x7f4f0416a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.535+0000 7f4f0a1a6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f04159da0 0x7f4f041585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.535+0000 7f4f0a1a6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f0415dfd0 0x7f4f040cf090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.535+0000 7f4f0a1a6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0409a560 con 0x7f4f041599d0 2024-01-26T23:50:10.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.535+0000 7f4f0a1a6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f04167b30 con 0x7f4f04159da0 2024-01-26T23:50:10.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.535+0000 7f4f0a1a6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0416ae80 con 0x7f4f0415dfd0 2024-01-26T23:50:10.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.535+0000 7f4f037fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f041599d0 0x7f4f0416a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.535+0000 7f4f037fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f041599d0 0x7f4f0416a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36274/0 (socket says 172.21.15.46:36274) 2024-01-26T23:50:10.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.535+0000 7f4f037fe700 1 -- 172.21.15.46:0/1526931935 learned_addr learned my addr 172.21.15.46:0/1526931935 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:10.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.535+0000 7f4f02ffd700 1 --2- 172.21.15.46:0/1526931935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f04159da0 0x7f4f041585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.535+0000 7f4f03fff700 1 --2- 172.21.15.46:0/1526931935 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f0415dfd0 0x7f4f040cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.536+0000 7f4f037fe700 1 -- 172.21.15.46:0/1526931935 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f0415dfd0 msgr2=0x7f4f040cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:10.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.536+0000 7f4f037fe700 1 --2- 172.21.15.46:0/1526931935 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f0415dfd0 0x7f4f040cf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.536+0000 7f4f037fe700 1 -- 172.21.15.46:0/1526931935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f04159da0 msgr2=0x7f4f041585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:10.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.536+0000 7f4f037fe700 1 --2- 172.21.15.46:0/1526931935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f04159da0 0x7f4f041585d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.536+0000 7f4f037fe700 1 -- 172.21.15.46:0/1526931935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4efc009a20 con 0x7f4f041599d0 2024-01-26T23:50:10.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.536+0000 7f4f037fe700 1 --2- 172.21.15.46:0/1526931935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f041599d0 0x7f4f0416a940 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400c5b0 tx=0x7f4ef40020e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:10.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.536+0000 7f4efb7fe700 1 -- 172.21.15.46:0/1526931935 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4011590 con 0x7f4f041599d0 2024-01-26T23:50:10.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.536+0000 7f4efb7fe700 1 -- 172.21.15.46:0/1526931935 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef40116f0 con 0x7f4f041599d0 2024-01-26T23:50:10.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.536+0000 7f4f0a1a6700 1 -- 172.21.15.46:0/1526931935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0409bf80 con 0x7f4f041599d0 2024-01-26T23:50:10.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.536+0000 7f4efb7fe700 1 -- 172.21.15.46:0/1526931935 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4011a90 con 0x7f4f041599d0 2024-01-26T23:50:10.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.536+0000 7f4f0a1a6700 1 -- 172.21.15.46:0/1526931935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f040abff0 con 0x7f4f041599d0 2024-01-26T23:50:10.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.537+0000 7f4f0a1a6700 1 -- 172.21.15.46:0/1526931935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0404c710 con 0x7f4f041599d0 2024-01-26T23:50:10.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.538+0000 7f4efb7fe700 1 -- 172.21.15.46:0/1526931935 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4ef4012420 con 0x7f4f041599d0 2024-01-26T23:50:10.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.538+0000 7f4efb7fe700 1 --2- 172.21.15.46:0/1526931935 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4ee4066f30 0x7f4ee40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.538+0000 7f4efb7fe700 1 -- 172.21.15.46:0/1526931935 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f4ef4084300 con 0x7f4f041599d0 2024-01-26T23:50:10.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.538+0000 7f4f02ffd700 1 --2- 172.21.15.46:0/1526931935 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4ee4066f30 0x7f4ee40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.539+0000 7f4f02ffd700 1 --2- 172.21.15.46:0/1526931935 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4ee4066f30 0x7f4ee40693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4f040b0cb0 tx=0x7f4eec009040 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:10.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.542+0000 7f4efb7fe700 1 -- 172.21.15.46:0/1526931935 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4ef40512a0 con 0x7f4f041599d0 2024-01-26T23:50:10.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.829+0000 7f4f0a1a6700 1 -- 172.21.15.46:0/1526931935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f4f040d0560 con 0x7f4f041599d0 2024-01-26T23:50:10.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.833+0000 7f4efb7fe700 1 -- 172.21.15.46:0/1526931935 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v24) v1 ==== 190+0+0 (secure 0 0 0) 0x7f4ef40548c0 con 0x7f4f041599d0 2024-01-26T23:50:10.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.836+0000 7f4f0a1a6700 1 -- 172.21.15.46:0/1526931935 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4ee4066f30 msgr2=0x7f4ee40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:10.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.836+0000 7f4f0a1a6700 1 --2- 172.21.15.46:0/1526931935 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4ee4066f30 0x7f4ee40693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4f040b0cb0 tx=0x7f4eec009040 comp rx=0 tx=0).stop 2024-01-26T23:50:10.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.836+0000 7f4f0a1a6700 1 -- 172.21.15.46:0/1526931935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f041599d0 msgr2=0x7f4f0416a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.836+0000 7f4f0a1a6700 1 --2- 172.21.15.46:0/1526931935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f041599d0 0x7f4f0416a940 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400c5b0 tx=0x7f4ef40020e0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.836+0000 7f4f0a1a6700 1 -- 172.21.15.46:0/1526931935 shutdown_connections 2024-01-26T23:50:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.836+0000 7f4f0a1a6700 1 --2- 172.21.15.46:0/1526931935 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4ee4066f30 0x7f4ee40693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.836+0000 7f4f0a1a6700 1 --2- 172.21.15.46:0/1526931935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f041599d0 0x7f4f0416a940 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.836+0000 7f4f0a1a6700 1 --2- 172.21.15.46:0/1526931935 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f0415dfd0 0x7f4f040cf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.836+0000 7f4f0a1a6700 1 --2- 172.21.15.46:0/1526931935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f04159da0 0x7f4f041585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.836+0000 7f4f0a1a6700 1 -- 172.21.15.46:0/1526931935 >> 172.21.15.46:0/1526931935 conn(0x7f4f0406f840 msgr2=0x7f4f04069540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.836+0000 7f4f0a1a6700 1 -- 172.21.15.46:0/1526931935 shutdown_connections 2024-01-26T23:50:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.836+0000 7f4f0a1a6700 1 -- 172.21.15.46:0/1526931935 wait complete. 2024-01-26T23:50:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-01-26T23:50:10.851 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-26T23:50:10.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.981+0000 7fb065134700 1 Processor -- start 2024-01-26T23:50:10.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.981+0000 7fb065134700 1 -- start start 2024-01-26T23:50:10.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.982+0000 7fb065134700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb060159ce0 0x7fb060068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.982+0000 7fb065134700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0600c7e90 con 0x7fb060159910 2024-01-26T23:50:10.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.982+0000 7fb065134700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb06015a0e0 con 0x7fb060068040 2024-01-26T23:50:10.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.982+0000 7fb065134700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb060069340 con 0x7fb060159ce0 2024-01-26T23:50:10.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.982+0000 7fb05ed9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb060159910 0x7fb060067930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37786/0 (socket says 172.21.15.46:37786) 2024-01-26T23:50:10.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.982+0000 7fb05ed9d700 1 -- 172.21.15.46:0/4278200598 learned_addr learned my addr 172.21.15.46:0/4278200598 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:10.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.982+0000 7fb05f59e700 1 --2- 172.21.15.46:0/4278200598 >> v2:172.21.15.106:3300/0 conn(0x7fb060159ce0 0x7fb060068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.982+0000 7fb05dd9b700 1 -- 172.21.15.46:0/4278200598 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996835339 0 0) 0x7fb0600c7e90 con 0x7fb060159910 2024-01-26T23:50:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.982+0000 7fb05dd9b700 1 -- 172.21.15.46:0/4278200598 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb048003570 con 0x7fb060159910 2024-01-26T23:50:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05dd9b700 1 -- 172.21.15.46:0/4278200598 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878613857 0 0) 0x7fb06015a0e0 con 0x7fb060068040 2024-01-26T23:50:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05dd9b700 1 -- 172.21.15.46:0/4278200598 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0600c7e90 con 0x7fb060068040 2024-01-26T23:50:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05dd9b700 1 -- 172.21.15.46:0/4278200598 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2046043185 0 0) 0x7fb048003570 con 0x7fb060159910 2024-01-26T23:50:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05dd9b700 1 -- 172.21.15.46:0/4278200598 >> v2:172.21.15.106:3300/0 conn(0x7fb060159ce0 msgr2=0x7fb060068bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05dd9b700 1 --2- 172.21.15.46:0/4278200598 >> v2:172.21.15.106:3300/0 conn(0x7fb060159ce0 0x7fb060068bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05dd9b700 1 -- 172.21.15.46:0/4278200598 >> v1:172.21.15.106:6789/0 conn(0x7fb060068040 legacy=0x7fb0600c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05dd9b700 1 -- 172.21.15.46:0/4278200598 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06006b690 con 0x7fb060159910 2024-01-26T23:50:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05dd9b700 1 -- 172.21.15.46:0/4278200598 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fb050001470 con 0x7fb060159910 2024-01-26T23:50:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05f59e700 1 --2- 172.21.15.46:0/4278200598 >> v2:172.21.15.106:3300/0 conn(0x7fb060159ce0 0x7fb060068bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:50:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05dd9b700 1 -- 172.21.15.46:0/4278200598 >> v1:172.21.15.46:6789/0 conn(0x7fb060159910 legacy=0x7fb060067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05dd9b700 1 --2- 172.21.15.46:0/4278200598 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb048003bf0 0x7fb048003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05dd9b700 1 --2- 172.21.15.46:0/4278200598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb048004810 0x7fb048004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05dd9b700 1 --2- 172.21.15.46:0/4278200598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0480051c0 0x7fb04800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05dd9b700 1 -- 172.21.15.46:0/4278200598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb060069340 con 0x7fb048004810 2024-01-26T23:50:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05dd9b700 1 -- 172.21.15.46:0/4278200598 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04800c160 con 0x7fb048003bf0 2024-01-26T23:50:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05dd9b700 1 -- 172.21.15.46:0/4278200598 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb04800c320 con 0x7fb0480051c0 2024-01-26T23:50:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05e59c700 1 --2- 172.21.15.46:0/4278200598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb048004810 0x7fb048004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05ed9d700 1 --2- 172.21.15.46:0/4278200598 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb048003bf0 0x7fb048003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05f59e700 1 --2- 172.21.15.46:0/4278200598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0480051c0 0x7fb04800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05e59c700 1 -- 172.21.15.46:0/4278200598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0480051c0 msgr2=0x7fb04800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05e59c700 1 --2- 172.21.15.46:0/4278200598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0480051c0 0x7fb04800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05e59c700 1 -- 172.21.15.46:0/4278200598 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb048003bf0 msgr2=0x7fb048003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05e59c700 1 --2- 172.21.15.46:0/4278200598 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb048003bf0 0x7fb048003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05e59c700 1 -- 172.21.15.46:0/4278200598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb048003570 con 0x7fb048004810 2024-01-26T23:50:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.983+0000 7fb05f59e700 1 -- 172.21.15.46:0/4278200598 reap_dead start 2024-01-26T23:50:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.984+0000 7fb05e59c700 1 --2- 172.21.15.46:0/4278200598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb048004810 0x7fb048004c80 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fb060068440 tx=0x7fb054007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5496bd21965c2d8c server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.984+0000 7fb05dd9b700 1 -- 172.21.15.46:0/4278200598 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05400c590 con 0x7fb048004810 2024-01-26T23:50:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.984+0000 7fb05dd9b700 1 -- 172.21.15.46:0/4278200598 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05400c6f0 con 0x7fb048004810 2024-01-26T23:50:10.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.984+0000 7fb05dd9b700 1 -- 172.21.15.46:0/4278200598 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05400ca90 con 0x7fb048004810 2024-01-26T23:50:10.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.984+0000 7fb065134700 1 -- 172.21.15.46:0/4278200598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb048004810 msgr2=0x7fb048004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:10.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.984+0000 7fb065134700 1 --2- 172.21.15.46:0/4278200598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb048004810 0x7fb048004c80 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fb060068440 tx=0x7fb054007d30 comp rx=0 tx=0).stop 2024-01-26T23:50:10.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.984+0000 7fb065134700 1 -- 172.21.15.46:0/4278200598 shutdown_connections 2024-01-26T23:50:10.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.984+0000 7fb065134700 1 --2- 172.21.15.46:0/4278200598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb048004810 0x7fb048004c80 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.984+0000 7fb065134700 1 -- 172.21.15.46:0/4278200598 >> 172.21.15.46:0/4278200598 conn(0x7fb0600cd6e0 msgr2=0x7fb0600c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:10.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.984+0000 7fb065134700 1 -- 172.21.15.46:0/4278200598 shutdown_connections 2024-01-26T23:50:10.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.984+0000 7fb065134700 1 -- 172.21.15.46:0/4278200598 wait complete. 2024-01-26T23:50:10.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.986+0000 7fb065134700 1 Processor -- start 2024-01-26T23:50:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.986+0000 7fb065134700 1 -- start start 2024-01-26T23:50:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.986+0000 7fb065134700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb048004810 0x7fb060169210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.986+0000 7fb065134700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb06015d500 0x7fb06015d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.986+0000 7fb065134700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb06015c230 0x7fb06015c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb065134700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06015deb0 con 0x7fb06015c230 2024-01-26T23:50:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb065134700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0600b10f0 con 0x7fb048004810 2024-01-26T23:50:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb065134700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06015cbe0 con 0x7fb06015d500 2024-01-26T23:50:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb05f59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb06015c230 0x7fb06015c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb05f59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb06015c230 0x7fb06015c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36300/0 (socket says 172.21.15.46:36300) 2024-01-26T23:50:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb05f59e700 1 -- 172.21.15.46:0/4246727460 learned_addr learned my addr 172.21.15.46:0/4246727460 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb05ed9d700 1 --2- 172.21.15.46:0/4246727460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb048004810 0x7fb060169210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb05e59c700 1 --2- 172.21.15.46:0/4246727460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb06015d500 0x7fb06015d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb05f59e700 1 -- 172.21.15.46:0/4246727460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb06015d500 msgr2=0x7fb06015d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb05f59e700 1 --2- 172.21.15.46:0/4246727460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb06015d500 0x7fb06015d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb05f59e700 1 -- 172.21.15.46:0/4246727460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb048004810 msgr2=0x7fb060169210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb05f59e700 1 --2- 172.21.15.46:0/4246727460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb048004810 0x7fb060169210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb05f59e700 1 -- 172.21.15.46:0/4246727460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0600b04e0 con 0x7fb06015c230 2024-01-26T23:50:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb05f59e700 1 --2- 172.21.15.46:0/4246727460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb06015c230 0x7fb06015c6a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fb0580094d0 tx=0x7fb05800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb046ffd700 1 -- 172.21.15.46:0/4246727460 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05800e040 con 0x7fb06015c230 2024-01-26T23:50:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb046ffd700 1 -- 172.21.15.46:0/4246727460 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05800f040 con 0x7fb06015c230 2024-01-26T23:50:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb065134700 1 -- 172.21.15.46:0/4246727460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0600b07c0 con 0x7fb06015c230 2024-01-26T23:50:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb046ffd700 1 -- 172.21.15.46:0/4246727460 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05800fed0 con 0x7fb06015c230 2024-01-26T23:50:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.987+0000 7fb065134700 1 -- 172.21.15.46:0/4246727460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0600b0d80 con 0x7fb06015c230 2024-01-26T23:50:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.988+0000 7fb065134700 1 -- 172.21.15.46:0/4246727460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0600b12a0 con 0x7fb06015c230 2024-01-26T23:50:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.989+0000 7fb046ffd700 1 -- 172.21.15.46:0/4246727460 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb058015640 con 0x7fb06015c230 2024-01-26T23:50:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.989+0000 7fb046ffd700 1 --2- 172.21.15.46:0/4246727460 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb04c066f30 0x7fb04c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.990+0000 7fb046ffd700 1 -- 172.21.15.46:0/4246727460 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fb058085100 con 0x7fb06015c230 2024-01-26T23:50:10.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.990+0000 7fb05ed9d700 1 --2- 172.21.15.46:0/4246727460 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb04c066f30 0x7fb04c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:10.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.990+0000 7fb05ed9d700 1 --2- 172.21.15.46:0/4246727460 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb04c066f30 0x7fb04c0693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb050007ef0 tx=0x7fb050001df0 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:10.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:10.994+0000 7fb046ffd700 1 -- 172.21.15.46:0/4246727460 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb0600b12a0 con 0x7fb06015c230 2024-01-26T23:50:11.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.248+0000 7fb065134700 1 -- 172.21.15.46:0/4246727460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0600b12a0 con 0x7fb06015c230 2024-01-26T23:50:11.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.249+0000 7fb046ffd700 1 -- 172.21.15.46:0/4246727460 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4602 (secure 0 0 0) 0x7fb0600b12a0 con 0x7fb06015c230 2024-01-26T23:50:11.250 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:50:11.250 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8578,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1889696251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1889696251},{"type":"v1","addr":"172.21.15.106:6834","nonce":1889696251}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/631881132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":631881132},{"type":"v1","addr":"172.21.15.106:6836","nonce":631881132}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1045000694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1045000694},{"type":"v1","addr":"172.21.15.46:6834","nonce":1045000694}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:50:05.176064+0000","modified":"2024-01-26T23:50:08.757937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8871":{"gid":8871,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3419491709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3419491709},{"type":"v1","addr":"172.21.15.46:6835","nonce":3419491709}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2024-01-26T23:50:11.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.251+0000 7fb065134700 1 -- 172.21.15.46:0/4246727460 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb04c066f30 msgr2=0x7fb04c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:11.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.251+0000 7fb065134700 1 --2- 172.21.15.46:0/4246727460 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb04c066f30 0x7fb04c0693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb050007ef0 tx=0x7fb050001df0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.252+0000 7fb065134700 1 -- 172.21.15.46:0/4246727460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb06015c230 msgr2=0x7fb06015c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:11.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.252+0000 7fb065134700 1 --2- 172.21.15.46:0/4246727460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb06015c230 0x7fb06015c6a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fb0580094d0 tx=0x7fb05800b040 comp rx=0 tx=0).stop 2024-01-26T23:50:11.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.252+0000 7fb065134700 1 -- 172.21.15.46:0/4246727460 shutdown_connections 2024-01-26T23:50:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.252+0000 7fb065134700 1 --2- 172.21.15.46:0/4246727460 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb04c066f30 0x7fb04c0693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.252+0000 7fb065134700 1 --2- 172.21.15.46:0/4246727460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb06015c230 0x7fb06015c6a0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.252+0000 7fb065134700 1 --2- 172.21.15.46:0/4246727460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb048004810 0x7fb060169210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.252+0000 7fb065134700 1 --2- 172.21.15.46:0/4246727460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb06015d500 0x7fb06015d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.252+0000 7fb065134700 1 -- 172.21.15.46:0/4246727460 >> 172.21.15.46:0/4246727460 conn(0x7fb0600cd6e0 msgr2=0x7fb0600c8760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.252+0000 7fb065134700 1 -- 172.21.15.46:0/4246727460 shutdown_connections 2024-01-26T23:50:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.252+0000 7fb065134700 1 -- 172.21.15.46:0/4246727460 wait complete. 2024-01-26T23:50:11.254 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 78 2024-01-26T23:50:11.669 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-26T23:50:05.176064+0000', 'modified': '2024-01-26T23:50:08.757937+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8871': {'gid': 8871, 'name': 'c', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6835/3419491709', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 3419491709}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 3419491709}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-26T23:50:11.669 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-26T23:50:11.669 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-26T23:50:11.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.810+0000 7f970a7c9700 1 Processor -- start 2024-01-26T23:50:11.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.810+0000 7f970a7c9700 1 -- start start 2024-01-26T23:50:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.811+0000 7f970a7c9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9704068040 0x7f9704164430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.811+0000 7f970a7c9700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970415abc0 con 0x7f9704159760 2024-01-26T23:50:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.811+0000 7f970a7c9700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97040b2e90 con 0x7f9704159b30 2024-01-26T23:50:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.811+0000 7f970a7c9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9704159fb0 con 0x7f9704068040 2024-01-26T23:50:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.811+0000 7f9708d66700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9704068040 0x7f9704164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.811+0000 7f9703fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9704159760 0x7f97040b0c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37796/0 (socket says 172.21.15.46:37796) 2024-01-26T23:50:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.811+0000 7f9703fff700 1 -- 172.21.15.46:0/2532155647 learned_addr learned my addr 172.21.15.46:0/2532155647 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.811+0000 7f9708d66700 1 -- 172.21.15.46:0/2532155647 >> v1:172.21.15.106:6789/0 conn(0x7f9704159b30 legacy=0x7f9704067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.811+0000 7f9708d66700 1 -- 172.21.15.46:0/2532155647 >> v1:172.21.15.46:6789/0 conn(0x7f9704159760 legacy=0x7f97040b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.812+0000 7f9708d66700 1 -- 172.21.15.46:0/2532155647 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970415a290 con 0x7f9704068040 2024-01-26T23:50:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.811+0000 7f9702ffd700 1 -- 172.21.15.46:0/2532155647 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122193042 0 0) 0x7f970415abc0 con 0x7f9704159760 2024-01-26T23:50:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.812+0000 7f9708d66700 1 --2- 172.21.15.46:0/2532155647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9704068040 0x7f9704164430 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f96fc009e60 tx=0x7f96fc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b197d745890f8df2 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.812+0000 7f9702ffd700 1 -- 172.21.15.46:0/2532155647 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc00e070 con 0x7f9704068040 2024-01-26T23:50:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.812+0000 7f9702ffd700 1 -- 172.21.15.46:0/2532155647 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96fc00bc30 con 0x7f9704068040 2024-01-26T23:50:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.812+0000 7f9702ffd700 1 -- 172.21.15.46:0/2532155647 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc012430 con 0x7f9704068040 2024-01-26T23:50:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.812+0000 7f970a7c9700 1 -- 172.21.15.46:0/2532155647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9704068040 msgr2=0x7f9704164430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.812+0000 7f970a7c9700 1 --2- 172.21.15.46:0/2532155647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9704068040 0x7f9704164430 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f96fc009e60 tx=0x7f96fc009780 comp rx=0 tx=0).stop 2024-01-26T23:50:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.812+0000 7f970a7c9700 1 -- 172.21.15.46:0/2532155647 shutdown_connections 2024-01-26T23:50:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.812+0000 7f970a7c9700 1 --2- 172.21.15.46:0/2532155647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9704068040 0x7f9704164430 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.812+0000 7f970a7c9700 1 -- 172.21.15.46:0/2532155647 >> 172.21.15.46:0/2532155647 conn(0x7f970406b630 msgr2=0x7f97040aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.813+0000 7f970a7c9700 1 -- 172.21.15.46:0/2532155647 shutdown_connections 2024-01-26T23:50:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.813+0000 7f970a7c9700 1 -- 172.21.15.46:0/2532155647 wait complete. 2024-01-26T23:50:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.814+0000 7f970a7c9700 1 Processor -- start 2024-01-26T23:50:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.814+0000 7f970a7c9700 1 -- start start 2024-01-26T23:50:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.815+0000 7f970a7c9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9704068040 0x7f9704169eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.815+0000 7f970a7c9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9704159760 0x7f970415ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.815+0000 7f970a7c9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f970415af50 0x7f97040c2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.815+0000 7f970a7c9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970415c3b0 con 0x7f9704068040 2024-01-26T23:50:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.815+0000 7f970a7c9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970415f1d0 con 0x7f970415af50 2024-01-26T23:50:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.815+0000 7f970a7c9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970415b7a0 con 0x7f9704159760 2024-01-26T23:50:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.815+0000 7f9703fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9704068040 0x7f9704169eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.815+0000 7f9703fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9704068040 0x7f9704169eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36314/0 (socket says 172.21.15.46:36314) 2024-01-26T23:50:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.815+0000 7f9703fff700 1 -- 172.21.15.46:0/3580689469 learned_addr learned my addr 172.21.15.46:0/3580689469 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.815+0000 7f9708d66700 1 --2- 172.21.15.46:0/3580689469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f970415af50 0x7f97040c2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.815+0000 7f97037fe700 1 --2- 172.21.15.46:0/3580689469 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9704159760 0x7f970415ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.815+0000 7f9703fff700 1 -- 172.21.15.46:0/3580689469 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9704159760 msgr2=0x7f970415ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:11.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.815+0000 7f9703fff700 1 --2- 172.21.15.46:0/3580689469 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9704159760 0x7f970415ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.815+0000 7f9703fff700 1 -- 172.21.15.46:0/3580689469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f970415af50 msgr2=0x7f97040c2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:11.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.815+0000 7f9703fff700 1 --2- 172.21.15.46:0/3580689469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f970415af50 0x7f97040c2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.815+0000 7f9703fff700 1 -- 172.21.15.46:0/3580689469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96fc009a20 con 0x7f9704068040 2024-01-26T23:50:11.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.816+0000 7f9703fff700 1 --2- 172.21.15.46:0/3580689469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9704068040 0x7f9704169eb0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f96f400c7e0 tx=0x7f96f4007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:11.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.816+0000 7f96fbfff700 1 -- 172.21.15.46:0/3580689469 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4005b30 con 0x7f9704068040 2024-01-26T23:50:11.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.816+0000 7f96fbfff700 1 -- 172.21.15.46:0/3580689469 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f4005c90 con 0x7f9704068040 2024-01-26T23:50:11.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.816+0000 7f970a7c9700 1 -- 172.21.15.46:0/3580689469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f970415ba80 con 0x7f9704068040 2024-01-26T23:50:11.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.816+0000 7f96fbfff700 1 -- 172.21.15.46:0/3580689469 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4007300 con 0x7f9704068040 2024-01-26T23:50:11.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.816+0000 7f970a7c9700 1 -- 172.21.15.46:0/3580689469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f970415c040 con 0x7f9704068040 2024-01-26T23:50:11.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.817+0000 7f96fbfff700 1 -- 172.21.15.46:0/3580689469 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f96f401b3b0 con 0x7f9704068040 2024-01-26T23:50:11.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.818+0000 7f96fbfff700 1 --2- 172.21.15.46:0/3580689469 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f96e4066f30 0x7f96e40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.818+0000 7f96fbfff700 1 -- 172.21.15.46:0/3580689469 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f96f4083d40 con 0x7f9704068040 2024-01-26T23:50:11.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.818+0000 7f97037fe700 1 --2- 172.21.15.46:0/3580689469 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f96e4066f30 0x7f96e40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.819+0000 7f97037fe700 1 --2- 172.21.15.46:0/3580689469 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f96e4066f30 0x7f96e40693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f97040c2cb0 tx=0x7f96ec009040 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:11.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.820+0000 7f9713accb80 1 Processor -- start 2024-01-26T23:50:11.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.820+0000 7f9713accb80 1 -- start start 2024-01-26T23:50:11.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.821+0000 7f9713accb80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62e010 0x55a2be62e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.821+0000 7f9713accb80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be676360 0x55a2be658840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.821+0000 7f9713accb80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62d5d0 0x55a2be658d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.821+0000 7f9713accb80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2be5319f0 con 0x55a2be62e010 2024-01-26T23:50:11.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.821+0000 7f9713accb80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2be583130 con 0x55a2be676360 2024-01-26T23:50:11.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.821+0000 7f9713accb80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a2be580930 con 0x55a2be62d5d0 2024-01-26T23:50:11.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.821+0000 7f9703fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62e010 0x55a2be62e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.821+0000 7f9703fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62e010 0x55a2be62e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36324/0 (socket says 172.21.15.46:36324) 2024-01-26T23:50:11.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.821+0000 7f9703fff700 1 -- 172.21.15.46:0/2633112543 learned_addr learned my addr 172.21.15.46:0/2633112543 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:11.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.821+0000 7f97037fe700 1 --2- 172.21.15.46:0/2633112543 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be676360 0x55a2be658840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.821+0000 7f9708d66700 1 --2- 172.21.15.46:0/2633112543 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62d5d0 0x55a2be658d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.821+0000 7f9703fff700 1 -- 172.21.15.46:0/2633112543 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62d5d0 msgr2=0x55a2be658d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:11.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.821+0000 7f9703fff700 1 --2- 172.21.15.46:0/2633112543 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62d5d0 0x55a2be658d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.821+0000 7f9703fff700 1 -- 172.21.15.46:0/2633112543 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be676360 msgr2=0x55a2be658840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:11.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.821+0000 7f9703fff700 1 --2- 172.21.15.46:0/2633112543 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be676360 0x55a2be658840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.821+0000 7f9703fff700 1 -- 172.21.15.46:0/2633112543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2be3bbb10 con 0x55a2be62e010 2024-01-26T23:50:11.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.821+0000 7f97037fe700 1 --2- 172.21.15.46:0/2633112543 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be676360 0x55a2be658840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:50:11.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.822+0000 7f9703fff700 1 --2- 172.21.15.46:0/2633112543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62e010 0x55a2be62e3e0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f96f4081fa0 tx=0x7f96f4081ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f04763b3fcc07c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:11.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.822+0000 7f96f97fa700 1 -- 172.21.15.46:0/2633112543 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4054980 con 0x55a2be62e010 2024-01-26T23:50:11.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.822+0000 7f96f97fa700 1 -- 172.21.15.46:0/2633112543 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f4057bd0 con 0x55a2be62e010 2024-01-26T23:50:11.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.822+0000 7f96f97fa700 1 -- 172.21.15.46:0/2633112543 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f405b920 con 0x55a2be62e010 2024-01-26T23:50:11.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.822+0000 7f9713accb80 1 -- 172.21.15.46:0/2633112543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62e010 msgr2=0x55a2be62e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:11.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.822+0000 7f9713accb80 1 --2- 172.21.15.46:0/2633112543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62e010 0x55a2be62e3e0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f96f4081fa0 tx=0x7f96f4081ed0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.823+0000 7f9713accb80 1 -- 172.21.15.46:0/2633112543 shutdown_connections 2024-01-26T23:50:11.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.823+0000 7f9713accb80 1 --2- 172.21.15.46:0/2633112543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62e010 0x55a2be62e3e0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.823+0000 7f9713accb80 1 --2- 172.21.15.46:0/2633112543 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62d5d0 0x55a2be658d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.823+0000 7f9713accb80 1 --2- 172.21.15.46:0/2633112543 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be676360 0x55a2be658840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.823+0000 7f9713accb80 1 -- 172.21.15.46:0/2633112543 >> 172.21.15.46:0/2633112543 conn(0x55a2be62da70 msgr2=0x55a2be676820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:11.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.823+0000 7f9713accb80 1 -- 172.21.15.46:0/2633112543 shutdown_connections 2024-01-26T23:50:11.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.823+0000 7f9713accb80 1 -- 172.21.15.46:0/2633112543 wait complete. 2024-01-26T23:50:11.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.823+0000 7f9713accb80 1 -- start start 2024-01-26T23:50:11.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.823+0000 7f9713accb80 1 Processor -- start 2024-01-26T23:50:11.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.824+0000 7f9713accb80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62d5d0 0x55a2be655930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.824+0000 7f9713accb80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62e010 0x55a2be65a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.824+0000 7f9713accb80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be676360 0x55a2be6719e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.824+0000 7f9713accb80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2be57c720 con 0x55a2be62e010 2024-01-26T23:50:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.824+0000 7f9713accb80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2be5281b0 con 0x55a2be676360 2024-01-26T23:50:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.824+0000 7f9713accb80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a2be527e50 con 0x55a2be62d5d0 2024-01-26T23:50:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.824+0000 7f9713accb80 10 client.0 _ll_register_callbacks cb 0x55a2be38a7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:50:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.824+0000 7f97037fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62e010 0x55a2be65a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.824+0000 7f97037fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62e010 0x55a2be65a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36330/0 (socket says 172.21.15.46:36330) 2024-01-26T23:50:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.824+0000 7f97037fe700 1 -- 172.21.15.46:0/2496411767 learned_addr learned my addr 172.21.15.46:0/2496411767 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.824+0000 7f9708d66700 1 --2- 172.21.15.46:0/2496411767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be676360 0x55a2be6719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.824+0000 7f9703fff700 1 --2- 172.21.15.46:0/2496411767 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62d5d0 0x55a2be655930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.825+0000 7f97037fe700 1 -- 172.21.15.46:0/2496411767 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62d5d0 msgr2=0x55a2be655930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.825+0000 7f97037fe700 1 --2- 172.21.15.46:0/2496411767 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62d5d0 0x55a2be655930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.825+0000 7f97037fe700 1 -- 172.21.15.46:0/2496411767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be676360 msgr2=0x55a2be6719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.825+0000 7f97037fe700 1 --2- 172.21.15.46:0/2496411767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be676360 0x55a2be6719e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.825+0000 7f97037fe700 1 -- 172.21.15.46:0/2496411767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97040b2e90 con 0x55a2be62e010 2024-01-26T23:50:11.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.825+0000 7f97037fe700 1 --2- 172.21.15.46:0/2496411767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62e010 0x55a2be65a8c0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f96ec0147d0 tx=0x7f96ec015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c48b58681ea03b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:11.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.825+0000 7f96e27fc700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:50:11.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.825+0000 7f96e27fc700 1 -- 172.21.15.46:0/2496411767 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ec014e20 con 0x55a2be62e010 2024-01-26T23:50:11.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.825+0000 7f96e27fc700 1 -- 172.21.15.46:0/2496411767 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(158..158 src has 1..158) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f96ec019070 con 0x55a2be62e010 2024-01-26T23:50:11.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.825+0000 7f9713accb80 1 -- 172.21.15.46:0/2496411767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a2be621190 con 0x55a2be62e010 2024-01-26T23:50:11.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.825+0000 7f96e27fc700 1 -- 172.21.15.46:0/2496411767 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96ec00fe50 con 0x55a2be62e010 2024-01-26T23:50:11.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.825+0000 7f9713accb80 10 client.9006 fetch_fsmap learned FSMap version 78 2024-01-26T23:50:11.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.825+0000 7f9713accb80 1 -- 172.21.15.46:0/2496411767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x7f970415c040 con 0x55a2be62e010 2024-01-26T23:50:11.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.825+0000 7f96e27fc700 1 -- 172.21.15.46:0/2496411767 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a2be621190 con 0x55a2be62e010 2024-01-26T23:50:11.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.826+0000 7f96e27fc700 1 -- 172.21.15.46:0/2496411767 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 78) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f96ec029350 con 0x55a2be62e010 2024-01-26T23:50:11.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.826+0000 7f9713accb80 10 client.9006 fetch_fsmap finished waiting for FSMap version 78 2024-01-26T23:50:11.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.826+0000 7f9713accb80 10 client.9006 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-26T23:50:11.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.826+0000 7f9713accb80 20 client.9006 populate_metadata read hostname 'smithi046' 2024-01-26T23:50:11.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.826+0000 7f9713accb80 1 --2- 172.21.15.46:0/2496411767 >> [v2:172.21.15.46:6833/3419491709,v1:172.21.15.46:6835/3419491709] conn(0x55a2be682a30 0x55a2be682e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.826+0000 7f9713accb80 4 client.9006 mds_command: new command op to 8871 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-26T23:50:11.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.826+0000 7f9713accb80 1 -- 172.21.15.46:0/2496411767 --> [v2:172.21.15.46:6833/3419491709,v1:172.21.15.46:6835/3419491709] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a2be51eee0 con 0x55a2be682a30 2024-01-26T23:50:11.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.826+0000 7f9703fff700 1 --2- 172.21.15.46:0/2496411767 >> [v2:172.21.15.46:6833/3419491709,v1:172.21.15.46:6835/3419491709] conn(0x55a2be682a30 0x55a2be682e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.826+0000 7f96e27fc700 10 client.9006 ms_handle_connect on v2:172.21.15.46:6833/3419491709 2024-01-26T23:50:11.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.826+0000 7f9703fff700 1 --2- 172.21.15.46:0/2496411767 >> [v2:172.21.15.46:6833/3419491709,v1:172.21.15.46:6835/3419491709] conn(0x55a2be682a30 0x55a2be682e00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c6bcf82625c2f37 server_cookie=17b7dedff23366bb in_seq=0 out_seq=0 2024-01-26T23:50:11.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.827+0000 7f96e27fc700 1 -- 172.21.15.46:0/2496411767 <== mds.0 v2:172.21.15.46:6833/3419491709 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x55a2be51eee0 con 0x55a2be682a30 2024-01-26T23:50:11.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.827+0000 7f96e27fc700 10 client.9006 handle_command_reply: tid=0 2024-01-26T23:50:11.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.827+0000 7f9713accb80 1 client.9006 shutdown 2024-01-26T23:50:11.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.827+0000 7f9713accb80 1 -- 172.21.15.46:0/2496411767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62e010 msgr2=0x55a2be65a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:11.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.827+0000 7f9713accb80 1 --2- 172.21.15.46:0/2496411767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62e010 0x55a2be65a8c0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f96ec0147d0 tx=0x7f96ec015000 comp rx=0 tx=0).stop 2024-01-26T23:50:11.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.827+0000 7f9713accb80 1 -- 172.21.15.46:0/2496411767 shutdown_connections 2024-01-26T23:50:11.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.827+0000 7f9713accb80 1 --2- 172.21.15.46:0/2496411767 >> [v2:172.21.15.46:6833/3419491709,v1:172.21.15.46:6835/3419491709] conn(0x55a2be682a30 0x55a2be682e00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.827+0000 7f9713accb80 1 --2- 172.21.15.46:0/2496411767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be676360 0x55a2be6719e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.827+0000 7f9713accb80 1 --2- 172.21.15.46:0/2496411767 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62d5d0 0x55a2be655930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.827+0000 7f96e27fc700 0 client.9006 ms_handle_reset on v2:172.21.15.46:6833/3419491709 2024-01-26T23:50:11.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.827+0000 7f9713accb80 1 --2- 172.21.15.46:0/2496411767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62e010 0x55a2be65a8c0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.827+0000 7f9713accb80 1 -- 172.21.15.46:0/2496411767 >> 172.21.15.46:0/2496411767 conn(0x55a2be62da70 msgr2=0x55a2be627240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:11.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.827+0000 7f9713accb80 1 -- 172.21.15.46:0/2496411767 shutdown_connections 2024-01-26T23:50:11.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.827+0000 7f9713accb80 1 -- 172.21.15.46:0/2496411767 wait complete. 2024-01-26T23:50:11.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.827+0000 7f9713accb80 20 client.9006 trim_cache size 0 max 16384 2024-01-26T23:50:11.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.845+0000 7f9713accb80 1 Processor -- start 2024-01-26T23:50:11.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.845+0000 7f9713accb80 1 -- start start 2024-01-26T23:50:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.846+0000 7f9713accb80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62da70 0x55a2be654050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.846+0000 7f9713accb80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62d5d0 0x55a2be681ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.846+0000 7f9713accb80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be62e010 0x55a2be627240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.846+0000 7f9713accb80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2be474e40 con 0x55a2be62da70 2024-01-26T23:50:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.846+0000 7f9713accb80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2be628590 con 0x55a2be62e010 2024-01-26T23:50:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.846+0000 7f9713accb80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a2be65b2c0 con 0x55a2be62d5d0 2024-01-26T23:50:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.847+0000 7f9703fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62da70 0x55a2be654050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.847+0000 7f9703fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62da70 0x55a2be654050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36338/0 (socket says 172.21.15.46:36338) 2024-01-26T23:50:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.847+0000 7f9703fff700 1 -- 172.21.15.46:0/577010077 learned_addr learned my addr 172.21.15.46:0/577010077 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.847+0000 7f9708d66700 1 --2- 172.21.15.46:0/577010077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be62e010 0x55a2be627240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.847+0000 7f97037fe700 1 --2- 172.21.15.46:0/577010077 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62d5d0 0x55a2be681ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.847+0000 7f9703fff700 1 -- 172.21.15.46:0/577010077 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62d5d0 msgr2=0x55a2be681ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.847+0000 7f9703fff700 1 --2- 172.21.15.46:0/577010077 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62d5d0 0x55a2be681ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.847+0000 7f9703fff700 1 -- 172.21.15.46:0/577010077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be62e010 msgr2=0x55a2be627240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.847+0000 7f9703fff700 1 --2- 172.21.15.46:0/577010077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be62e010 0x55a2be627240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.847+0000 7f9703fff700 1 -- 172.21.15.46:0/577010077 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2be62c810 con 0x55a2be62da70 2024-01-26T23:50:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.847+0000 7f97037fe700 1 --2- 172.21.15.46:0/577010077 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62d5d0 0x55a2be681ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:50:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.847+0000 7f9703fff700 1 --2- 172.21.15.46:0/577010077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62da70 0x55a2be654050 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f96f4082050 tx=0x7f96f407e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa563279d8c1c12e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.847+0000 7f96f9ffb700 1 -- 172.21.15.46:0/577010077 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f405b920 con 0x55a2be62da70 2024-01-26T23:50:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.847+0000 7f96f9ffb700 1 -- 172.21.15.46:0/577010077 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f4054980 con 0x55a2be62da70 2024-01-26T23:50:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.847+0000 7f96f9ffb700 1 -- 172.21.15.46:0/577010077 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4005e00 con 0x55a2be62da70 2024-01-26T23:50:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.848+0000 7f9713accb80 1 -- 172.21.15.46:0/577010077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62da70 msgr2=0x55a2be654050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.848+0000 7f9713accb80 1 --2- 172.21.15.46:0/577010077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62da70 0x55a2be654050 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f96f4082050 tx=0x7f96f407e040 comp rx=0 tx=0).stop 2024-01-26T23:50:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.848+0000 7f9713accb80 1 -- 172.21.15.46:0/577010077 shutdown_connections 2024-01-26T23:50:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.848+0000 7f9713accb80 1 --2- 172.21.15.46:0/577010077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62da70 0x55a2be654050 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.848+0000 7f9713accb80 1 --2- 172.21.15.46:0/577010077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be62e010 0x55a2be627240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.848+0000 7f9713accb80 1 --2- 172.21.15.46:0/577010077 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62d5d0 0x55a2be681ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.848+0000 7f9713accb80 1 -- 172.21.15.46:0/577010077 >> 172.21.15.46:0/577010077 conn(0x55a2be6294b0 msgr2=0x55a2be65c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.848+0000 7f9713accb80 1 -- 172.21.15.46:0/577010077 shutdown_connections 2024-01-26T23:50:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.848+0000 7f9713accb80 1 -- 172.21.15.46:0/577010077 wait complete. 2024-01-26T23:50:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.849+0000 7f9713accb80 1 -- start start 2024-01-26T23:50:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.849+0000 7f9713accb80 1 Processor -- start 2024-01-26T23:50:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.850+0000 7f9713accb80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62d5d0 0x55a2be6597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.850+0000 7f9713accb80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62da70 0x55a2be65c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.850+0000 7f9713accb80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be62e010 0x55a2be627240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.850+0000 7f9713accb80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2be6d8120 con 0x55a2be62d5d0 2024-01-26T23:50:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.850+0000 7f9713accb80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2be6d9a50 con 0x55a2be62e010 2024-01-26T23:50:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.850+0000 7f9713accb80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a2be68f760 con 0x55a2be62da70 2024-01-26T23:50:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.850+0000 7f9713accb80 10 client.0 _ll_register_callbacks cb 0x55a2be38a7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-26T23:50:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.850+0000 7f9703fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62d5d0 0x55a2be6597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.850+0000 7f9703fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62d5d0 0x55a2be6597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36342/0 (socket says 172.21.15.46:36342) 2024-01-26T23:50:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.850+0000 7f9703fff700 1 -- 172.21.15.46:0/3404433129 learned_addr learned my addr 172.21.15.46:0/3404433129 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.850+0000 7f97037fe700 1 --2- 172.21.15.46:0/3404433129 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62da70 0x55a2be65c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.850+0000 7f9708d66700 1 --2- 172.21.15.46:0/3404433129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be62e010 0x55a2be627240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.851+0000 7f9703fff700 1 -- 172.21.15.46:0/3404433129 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62da70 msgr2=0x55a2be65c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.851+0000 7f9703fff700 1 --2- 172.21.15.46:0/3404433129 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62da70 0x55a2be65c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.851+0000 7f9703fff700 1 -- 172.21.15.46:0/3404433129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be62e010 msgr2=0x55a2be627240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.851+0000 7f9703fff700 1 --2- 172.21.15.46:0/3404433129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be62e010 0x55a2be627240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.851+0000 7f9703fff700 1 -- 172.21.15.46:0/3404433129 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a2be3bbb10 con 0x55a2be62d5d0 2024-01-26T23:50:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.851+0000 7f9703fff700 1 --2- 172.21.15.46:0/3404433129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62d5d0 0x55a2be6597b0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f96f4054680 tx=0x7f96f4081000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9696b8a62810f9a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.851+0000 7f96e27fc700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-26T23:50:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.851+0000 7f96e27fc700 1 -- 172.21.15.46:0/3404433129 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f405b920 con 0x55a2be62d5d0 2024-01-26T23:50:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.851+0000 7f96e27fc700 1 -- 172.21.15.46:0/3404433129 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(158..158 src has 1..158) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f96f4081070 con 0x55a2be62d5d0 2024-01-26T23:50:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.851+0000 7f9713accb80 1 -- 172.21.15.46:0/3404433129 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f970405e620 con 0x55a2be62d5d0 2024-01-26T23:50:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.851+0000 7f96e27fc700 1 -- 172.21.15.46:0/3404433129 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96f4005e00 con 0x55a2be62d5d0 2024-01-26T23:50:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.851+0000 7f9713accb80 10 client.9012 fetch_fsmap learned FSMap version 78 2024-01-26T23:50:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.851+0000 7f9713accb80 1 -- 172.21.15.46:0/3404433129 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x55a2be629e50 con 0x55a2be62d5d0 2024-01-26T23:50:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.851+0000 7f96e27fc700 1 -- 172.21.15.46:0/3404433129 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970405e620 con 0x55a2be62d5d0 2024-01-26T23:50:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.851+0000 7f96e27fc700 1 -- 172.21.15.46:0/3404433129 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 78) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f96f405b490 con 0x55a2be62d5d0 2024-01-26T23:50:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f9713accb80 10 client.9012 fetch_fsmap finished waiting for FSMap version 78 2024-01-26T23:50:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f9713accb80 10 client.9012 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-26T23:50:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f9713accb80 20 client.9012 populate_metadata read hostname 'smithi046' 2024-01-26T23:50:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f9713accb80 1 --2- 172.21.15.46:0/3404433129 >> [v2:172.21.15.46:6833/3419491709,v1:172.21.15.46:6835/3419491709] conn(0x55a2be676360 0x55a2be6279b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f9713accb80 4 client.9012 mds_command: new command op to 8871 tid=0[{"prefix": "status"}] 2024-01-26T23:50:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f9713accb80 1 -- 172.21.15.46:0/3404433129 --> [v2:172.21.15.46:6833/3419491709,v1:172.21.15.46:6835/3419491709] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f96f4057bd0 con 0x55a2be676360 2024-01-26T23:50:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f97037fe700 1 --2- 172.21.15.46:0/3404433129 >> [v2:172.21.15.46:6833/3419491709,v1:172.21.15.46:6835/3419491709] conn(0x55a2be676360 0x55a2be6279b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f96e27fc700 10 client.9012 ms_handle_connect on v2:172.21.15.46:6833/3419491709 2024-01-26T23:50:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f97037fe700 1 --2- 172.21.15.46:0/3404433129 >> [v2:172.21.15.46:6833/3419491709,v1:172.21.15.46:6835/3419491709] conn(0x55a2be676360 0x55a2be6279b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58503139105fc6a9 server_cookie=790232599a681ebd in_seq=0 out_seq=0 2024-01-26T23:50:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f96e27fc700 1 -- 172.21.15.46:0/3404433129 <== mds.0 v2:172.21.15.46:6833/3419491709 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x7f96f4057bd0 con 0x55a2be676360 2024-01-26T23:50:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f96e27fc700 10 client.9012 handle_command_reply: tid=0 2024-01-26T23:50:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f9713accb80 1 client.9012 shutdown 2024-01-26T23:50:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f9713accb80 1 -- 172.21.15.46:0/3404433129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62d5d0 msgr2=0x55a2be6597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f9713accb80 1 --2- 172.21.15.46:0/3404433129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62d5d0 0x55a2be6597b0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f96f4054680 tx=0x7f96f4081000 comp rx=0 tx=0).stop 2024-01-26T23:50:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f9713accb80 1 -- 172.21.15.46:0/3404433129 shutdown_connections 2024-01-26T23:50:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f9713accb80 1 --2- 172.21.15.46:0/3404433129 >> [v2:172.21.15.46:6833/3419491709,v1:172.21.15.46:6835/3419491709] conn(0x55a2be676360 0x55a2be6279b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f9713accb80 1 --2- 172.21.15.46:0/3404433129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2be62d5d0 0x55a2be6597b0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f9713accb80 1 --2- 172.21.15.46:0/3404433129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2be62e010 0x55a2be627240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f96e27fc700 0 client.9012 ms_handle_reset on v2:172.21.15.46:6833/3419491709 2024-01-26T23:50:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f9713accb80 1 --2- 172.21.15.46:0/3404433129 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55a2be62da70 0x55a2be65c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.852+0000 7f9713accb80 1 -- 172.21.15.46:0/3404433129 >> 172.21.15.46:0/3404433129 conn(0x55a2be6294b0 msgr2=0x55a2be654050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.853+0000 7f9713accb80 1 -- 172.21.15.46:0/3404433129 shutdown_connections 2024-01-26T23:50:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.853+0000 7f9713accb80 1 -- 172.21.15.46:0/3404433129 wait complete. 2024-01-26T23:50:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.853+0000 7f9713accb80 20 client.9012 trim_cache size 0 max 16384 2024-01-26T23:50:11.860 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:50:11.860 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-26T23:50:11.860 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-26T23:50:11.860 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 8871, 2024-01-26T23:50:11.860 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-26T23:50:11.860 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-26T23:50:11.860 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-01-26T23:50:11.860 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.1019474929999999, 2024-01-26T23:50:11.860 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 78, 2024-01-26T23:50:11.860 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 158, 2024-01-26T23:50:11.861 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 158, 2024-01-26T23:50:11.861 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 11.088369756000001 2024-01-26T23:50:11.861 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:50:11.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.853+0000 7f970a7c9700 1 -- 172.21.15.46:0/3580689469 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f96e4066f30 msgr2=0x7f96e40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.853+0000 7f970a7c9700 1 --2- 172.21.15.46:0/3580689469 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f96e4066f30 0x7f96e40693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f97040c2cb0 tx=0x7f96ec009040 comp rx=0 tx=0).stop 2024-01-26T23:50:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.853+0000 7f970a7c9700 1 -- 172.21.15.46:0/3580689469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9704068040 msgr2=0x7f9704169eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.853+0000 7f970a7c9700 1 --2- 172.21.15.46:0/3580689469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9704068040 0x7f9704169eb0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f96f400c7e0 tx=0x7f96f4007df0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.853+0000 7f970a7c9700 1 -- 172.21.15.46:0/3580689469 shutdown_connections 2024-01-26T23:50:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.853+0000 7f970a7c9700 1 --2- 172.21.15.46:0/3580689469 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f96e4066f30 0x7f96e40693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.853+0000 7f970a7c9700 1 --2- 172.21.15.46:0/3580689469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9704068040 0x7f9704169eb0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.853+0000 7f970a7c9700 1 --2- 172.21.15.46:0/3580689469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f970415af50 0x7f97040c2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.853+0000 7f970a7c9700 1 --2- 172.21.15.46:0/3580689469 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9704159760 0x7f970415ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.853+0000 7f970a7c9700 1 -- 172.21.15.46:0/3580689469 >> 172.21.15.46:0/3580689469 conn(0x7f970406b630 msgr2=0x7f97040af5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.853+0000 7f970a7c9700 1 -- 172.21.15.46:0/3580689469 shutdown_connections 2024-01-26T23:50:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.853+0000 7f970a7c9700 1 -- 172.21.15.46:0/3580689469 wait complete. 2024-01-26T23:50:11.868 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-26T23:50:12.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.999+0000 7f502a217700 1 Processor -- start 2024-01-26T23:50:12.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:11.999+0000 7f502a217700 1 -- start start 2024-01-26T23:50:12.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.000+0000 7f502a217700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5024159b30 0x7f50240b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:12.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.000+0000 7f502a217700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50240b10f0 0x7f50240a9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:12.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.000+0000 7f502a217700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f502406f940 con 0x7f5024159760 2024-01-26T23:50:12.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.000+0000 7f502a217700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50240b1570 con 0x7f5024159b30 2024-01-26T23:50:12.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.000+0000 7f502a217700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50240b3230 con 0x7f50240b10f0 2024-01-26T23:50:12.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.000+0000 7f5022ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5024159b30 0x7f50240b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:12.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.000+0000 7f5022ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5024159b30 0x7f50240b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36344/0 (socket says 172.21.15.46:36344) 2024-01-26T23:50:12.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.000+0000 7f5022ffd700 1 -- 172.21.15.46:0/1654936847 learned_addr learned my addr 172.21.15.46:0/1654936847 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:12.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.000+0000 7f5023fff700 1 --2- 172.21.15.46:0/1654936847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50240b10f0 0x7f50240a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:12.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.001+0000 7f5022ffd700 1 -- 172.21.15.46:0/1654936847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50240b10f0 msgr2=0x7f50240a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:12.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.001+0000 7f5022ffd700 1 --2- 172.21.15.46:0/1654936847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50240b10f0 0x7f50240a9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:12.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.001+0000 7f5022ffd700 1 -- 172.21.15.46:0/1654936847 >> v1:172.21.15.106:6789/0 conn(0x7f5024159760 legacy=0x7f50240b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:12.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.001+0000 7f5022ffd700 1 -- 172.21.15.46:0/1654936847 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f502406ed30 con 0x7f5024159b30 2024-01-26T23:50:12.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.001+0000 7f5022ffd700 1 --2- 172.21.15.46:0/1654936847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5024159b30 0x7f50240b0bb0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f50180099e0 tx=0x7f5018009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e8e829fa225a4e server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:12.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.001+0000 7f50227fc700 1 -- 172.21.15.46:0/1654936847 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501800b820 con 0x7f5024159b30 2024-01-26T23:50:12.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.001+0000 7f50227fc700 1 -- 172.21.15.46:0/1654936847 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501800b980 con 0x7f5024159b30 2024-01-26T23:50:12.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.001+0000 7f50227fc700 1 -- 172.21.15.46:0/1654936847 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501800bd20 con 0x7f5024159b30 2024-01-26T23:50:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.002+0000 7f502a217700 1 -- 172.21.15.46:0/1654936847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5024159b30 msgr2=0x7f50240b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.002+0000 7f502a217700 1 --2- 172.21.15.46:0/1654936847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5024159b30 0x7f50240b0bb0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f50180099e0 tx=0x7f5018009250 comp rx=0 tx=0).stop 2024-01-26T23:50:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.002+0000 7f502a217700 1 -- 172.21.15.46:0/1654936847 shutdown_connections 2024-01-26T23:50:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.002+0000 7f502a217700 1 --2- 172.21.15.46:0/1654936847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50240b10f0 0x7f50240a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.002+0000 7f502a217700 1 --2- 172.21.15.46:0/1654936847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5024159b30 0x7f50240b0bb0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.002+0000 7f502a217700 1 -- 172.21.15.46:0/1654936847 >> 172.21.15.46:0/1654936847 conn(0x7f5024069510 msgr2=0x7f50240aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.002+0000 7f502a217700 1 -- 172.21.15.46:0/1654936847 shutdown_connections 2024-01-26T23:50:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.002+0000 7f502a217700 1 -- 172.21.15.46:0/1654936847 wait complete. 2024-01-26T23:50:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.003+0000 7f502a217700 1 Processor -- start 2024-01-26T23:50:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.003+0000 7f502a217700 1 -- start start 2024-01-26T23:50:12.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f502a217700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50240b10f0 0x7f50241420e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:12.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f502a217700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5024159760 0x7f502414dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:12.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f502a217700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5024159b30 0x7f50240ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:12.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f502a217700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502414f240 con 0x7f5024159760 2024-01-26T23:50:12.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f502a217700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502414e630 con 0x7f5024159b30 2024-01-26T23:50:12.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f502a217700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f502414e910 con 0x7f50240b10f0 2024-01-26T23:50:12.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f5022ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5024159760 0x7f502414dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:12.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f5022ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5024159760 0x7f502414dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36356/0 (socket says 172.21.15.46:36356) 2024-01-26T23:50:12.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f5022ffd700 1 -- 172.21.15.46:0/4249133046 learned_addr learned my addr 172.21.15.46:0/4249133046 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:12.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f5023fff700 1 --2- 172.21.15.46:0/4249133046 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5024159b30 0x7f50240ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:12.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f50237fe700 1 --2- 172.21.15.46:0/4249133046 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50240b10f0 0x7f50241420e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:12.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f5022ffd700 1 -- 172.21.15.46:0/4249133046 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50240b10f0 msgr2=0x7f50241420e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:12.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f5022ffd700 1 --2- 172.21.15.46:0/4249133046 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50240b10f0 0x7f50241420e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:12.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f5022ffd700 1 -- 172.21.15.46:0/4249133046 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5024159b30 msgr2=0x7f50240ad6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:12.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f5022ffd700 1 --2- 172.21.15.46:0/4249133046 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5024159b30 0x7f50240ad6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:12.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f5022ffd700 1 -- 172.21.15.46:0/4249133046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50180094f0 con 0x7f5024159760 2024-01-26T23:50:12.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f5023fff700 1 --2- 172.21.15.46:0/4249133046 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5024159b30 0x7f50240ad6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:50:12.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f50237fe700 1 --2- 172.21.15.46:0/4249133046 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50240b10f0 0x7f50241420e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:50:12.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.004+0000 7f5022ffd700 1 --2- 172.21.15.46:0/4249133046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5024159760 0x7f502414dc50 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f5018009290 tx=0x7f5018017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:12.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.005+0000 7f500b7fe700 1 -- 172.21.15.46:0/4249133046 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501801a450 con 0x7f5024159760 2024-01-26T23:50:12.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.005+0000 7f500b7fe700 1 -- 172.21.15.46:0/4249133046 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501801a5b0 con 0x7f5024159760 2024-01-26T23:50:12.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.005+0000 7f502a217700 1 -- 172.21.15.46:0/4249133046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f502414ebf0 con 0x7f5024159760 2024-01-26T23:50:12.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.005+0000 7f500b7fe700 1 -- 172.21.15.46:0/4249133046 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501801a970 con 0x7f5024159760 2024-01-26T23:50:12.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.005+0000 7f502a217700 1 -- 172.21.15.46:0/4249133046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5024142620 con 0x7f5024159760 2024-01-26T23:50:12.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.006+0000 7f502a217700 1 -- 172.21.15.46:0/4249133046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50240ac7e0 con 0x7f5024159760 2024-01-26T23:50:12.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.006+0000 7f500b7fe700 1 -- 172.21.15.46:0/4249133046 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f501801ad20 con 0x7f5024159760 2024-01-26T23:50:12.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.007+0000 7f500b7fe700 1 --2- 172.21.15.46:0/4249133046 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5010066f30 0x7f50100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:12.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.007+0000 7f500b7fe700 1 -- 172.21.15.46:0/4249133046 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f50180862f0 con 0x7f5024159760 2024-01-26T23:50:12.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.007+0000 7f50237fe700 1 --2- 172.21.15.46:0/4249133046 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5010066f30 0x7f50100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:12.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.008+0000 7f50237fe700 1 --2- 172.21.15.46:0/4249133046 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5010066f30 0x7f50100693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5014007fd0 tx=0x7f5014009040 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:12.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.011+0000 7f500b7fe700 1 -- 172.21.15.46:0/4249133046 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5018053310 con 0x7f5024159760 2024-01-26T23:50:12.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.263+0000 7f502a217700 1 -- 172.21.15.46:0/4249133046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f502414f5d0 con 0x7f5024159760 2024-01-26T23:50:12.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.263+0000 7f500b7fe700 1 -- 172.21.15.46:0/4249133046 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v78) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5018056930 con 0x7f5024159760 2024-01-26T23:50:12.265 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-26T23:50:12.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.266+0000 7f502a217700 1 -- 172.21.15.46:0/4249133046 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5010066f30 msgr2=0x7f50100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:12.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.266+0000 7f502a217700 1 --2- 172.21.15.46:0/4249133046 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5010066f30 0x7f50100693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5014007fd0 tx=0x7f5014009040 comp rx=0 tx=0).stop 2024-01-26T23:50:12.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.266+0000 7f502a217700 1 -- 172.21.15.46:0/4249133046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5024159760 msgr2=0x7f502414dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:12.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.266+0000 7f502a217700 1 --2- 172.21.15.46:0/4249133046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5024159760 0x7f502414dc50 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f5018009290 tx=0x7f5018017000 comp rx=0 tx=0).stop 2024-01-26T23:50:12.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.266+0000 7f502a217700 1 -- 172.21.15.46:0/4249133046 shutdown_connections 2024-01-26T23:50:12.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.266+0000 7f502a217700 1 --2- 172.21.15.46:0/4249133046 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5010066f30 0x7f50100693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:12.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.266+0000 7f502a217700 1 --2- 172.21.15.46:0/4249133046 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5024159b30 0x7f50240ad6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:12.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.266+0000 7f502a217700 1 --2- 172.21.15.46:0/4249133046 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f50240b10f0 0x7f50241420e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:12.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.266+0000 7f502a217700 1 --2- 172.21.15.46:0/4249133046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5024159760 0x7f502414dc50 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:12.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.266+0000 7f502a217700 1 -- 172.21.15.46:0/4249133046 >> 172.21.15.46:0/4249133046 conn(0x7f5024069510 msgr2=0x7f50240ae7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:12.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.266+0000 7f502a217700 1 -- 172.21.15.46:0/4249133046 shutdown_connections 2024-01-26T23:50:12.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:12.266+0000 7f502a217700 1 -- 172.21.15.46:0/4249133046 wait complete. 2024-01-26T23:50:12.280 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-26T23:50:12.280 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-26T23:50:12.281 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-26T23:50:12.281 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-26T23:50:12.281 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-26T23:50:12.281 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-26T23:50:12.281 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-26T23:50:12.281 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-26T23:50:12.281 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:50:12.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:50:12.281 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-26T23:50:12.301 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-26T23:50:12.301 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-26T23:50:12.301 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-26T23:50:12.301 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:50:12.301 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-26T23:50:12.301 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:50:12.301 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:50:12.301 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:50:12.301 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-26T23:50:12.301 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:50:12.301 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:50:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-26T23:50:12.302 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-26T23:50:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:50:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-26T23:50:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-26T23:50:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 41118sec preferred_lft 41118sec 2024-01-26T23:50:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-26T23:50:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:50:12.302 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-26T23:50:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:50:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-26T23:50:12.302 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-26T23:50:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff 2024-01-26T23:50:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-26T23:50:12.303 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:50:12.303 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::cce6:a7ff:fe4f:38d/64 scope link 2024-01-26T23:50:12.303 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:50:12.303 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:50:12.303 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 76:ef:90:4c:9d:6f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-26T23:50:12.303 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::74ef:90ff:fe4c:9d6f/64 scope link 2024-01-26T23:50:12.303 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:50:12.303 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-26T23:50:12.303 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-26T23:50:12.303 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::2c73:afff:fef0:6f39/64 scope link 2024-01-26T23:50:12.303 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-26T23:50:12.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:50:12.304 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-26T23:50:12.372 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-26T23:50:12.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:50:12.373 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-26T23:50:12.437 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-26T23:50:12.438 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-26T23:50:12.438 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-26T23:50:12.438 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-26T23:50:12.438 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-26T23:50:12.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:50:12.438 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-26T23:50:12.438 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-26T23:50:12.438 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-26T23:50:12.438 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-26T23:50:12.439 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-26T23:50:12.439 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-26T23:50:12.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:50:12.561 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-26T23:50:12.582 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-26T23:50:12.582 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-26T23:50:12.582 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:50:12.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-26T23:50:12.582 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-01-26T23:50:12.657 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-26T23:50:12.657 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:50:12.657 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-26T23:50:12.720 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-26T23:50:12.720 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-26T23:50:12.720 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:50:12.781 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:50:12.858 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:50:12.858 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:50:12.932 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:50:12.934 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:50:12.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:50:12.934 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:50:12.996 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-26T23:50:12.996 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-26T23:50:13.038 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-26T23:50:13.073 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-26T23:50:13.073 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-26T23:50:13.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-01-26T23:50:13.139+0000 7fc240c47540 -1 init, newargv = 0x55ae89d8dd10 newargc=15 2024-01-26T23:50:13.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[90782]: starting ceph client 2024-01-26T23:50:13.144 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-26T23:50:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-26T23:50:13.146 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:50:13.147 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-26T23:50:13.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[90782]: starting fuse 2024-01-26T23:50:13.207 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-26T23:50:13.207 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-26T23:50:13.208 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-26T23:50:13.208 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-26T23:50:13.208 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-26T23:50:13.208 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-26T23:50:13.208 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-26T23:50:13.208 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:50:13.208 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-26T23:50:13.208 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-26T23:50:13.209 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-26T23:50:13.209 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-26T23:50:13.209 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:50:13.209 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-26T23:50:13.209 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-26T23:50:13.209 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-26T23:50:13.209 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-26T23:50:13.209 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:50:13.209 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-26T23:50:13.209 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-26T23:50:13.209 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-26T23:50:13.209 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-26T23:50:13.209 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-26T23:50:13.209 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-26T23:50:13.209 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-26T23:50:13.209 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-26T23:50:13.210 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-26T23:50:13.210 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-01-26T23:50:13.210 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-26T23:50:13.348 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.90782.asok 2024-01-26T23:50:13.529 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.90782.asok 2024-01-26T23:50:13.529 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:50:13.529 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.90782.asok status 2024-01-26T23:50:13.703 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-26T23:50:13.703 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-26T23:50:13.703 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-26T23:50:13.703 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-26T23:50:13.703 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-01-26T23:50:13.703 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-26T23:50:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-26T23:50:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "90782", 2024-01-26T23:50:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-26T23:50:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-26T23:50:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-26T23:50:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 9024, 2024-01-26T23:50:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-26T23:50:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-26T23:50:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-26T23:50:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 9024 2024-01-26T23:50:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:50:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:50:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:50:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 200820196 2024-01-26T23:50:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-26T23:50:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-26T23:50:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-26T23:50:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 200820196 2024-01-26T23:50:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-26T23:50:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.9024 192.168.0.1:0/200820196", 2024-01-26T23:50:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/200820196", 2024-01-26T23:50:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-26T23:50:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 78, 2024-01-26T23:50:13.706 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 158, 2024-01-26T23:50:13.706 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-26T23:50:13.706 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-26T23:50:13.706 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-26T23:50:13.706 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-26T23:50:13.714 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:50:13.714 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-26T23:50:13.714 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-26T23:50:13.733 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-26T23:50:13.734 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-26T23:50:13.734 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:50:13.734 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-26T23:50:13.807 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-26T23:50:13.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.986+0000 7f01612eb700 1 Processor -- start 2024-01-26T23:50:13.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.986+0000 7f01612eb700 1 -- start start 2024-01-26T23:50:13.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.986+0000 7f01612eb700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f015c0ad070 0x7f015c0ae610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:13.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.986+0000 7f01612eb700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015c0ae180 con 0x7f015c0cc1c0 2024-01-26T23:50:13.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.987+0000 7f01612eb700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015c156c60 con 0x7f015c0cbdf0 2024-01-26T23:50:13.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.987+0000 7f01612eb700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f015c0aeb50 con 0x7f015c0ad070 2024-01-26T23:50:13.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.987+0000 7f015a7fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f015c0cc1c0 0x7f015c0c98b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37808/0 (socket says 172.21.15.46:37808) 2024-01-26T23:50:13.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.987+0000 7f015a7fc700 1 -- 172.21.15.46:0/328786668 learned_addr learned my addr 172.21.15.46:0/328786668 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:13.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.987+0000 7f015b7fe700 1 --2- 172.21.15.46:0/328786668 >> v2:172.21.15.106:3300/0 conn(0x7f015c0ad070 0x7f015c0ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:13.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.987+0000 7f0159ffb700 1 -- 172.21.15.46:0/328786668 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463752306 0 0) 0x7f015c0ae180 con 0x7f015c0cc1c0 2024-01-26T23:50:13.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.987+0000 7f0159ffb700 1 -- 172.21.15.46:0/328786668 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0144003570 con 0x7f015c0cc1c0 2024-01-26T23:50:13.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.987+0000 7f015b7fe700 1 -- 172.21.15.46:0/328786668 >> v1:172.21.15.106:6789/0 conn(0x7f015c0cbdf0 legacy=0x7f015c0b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:13.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.987+0000 7f0159ffb700 1 -- 172.21.15.46:0/328786668 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2242779403 0 0) 0x7f0144003570 con 0x7f015c0cc1c0 2024-01-26T23:50:13.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.987+0000 7f015b7fe700 1 -- 172.21.15.46:0/328786668 >> v1:172.21.15.46:6789/0 conn(0x7f015c0cc1c0 legacy=0x7f015c0c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:13.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.987+0000 7f015b7fe700 1 -- 172.21.15.46:0/328786668 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c0ad570 con 0x7f015c0ad070 2024-01-26T23:50:13.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.987+0000 7f015b7fe700 1 --2- 172.21.15.46:0/328786668 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f015c0ad070 0x7f015c0ae610 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0154009e60 tx=0x7f0154009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff2b7a0e820a8b91 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:13.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.988+0000 7f0159ffb700 1 -- 172.21.15.46:0/328786668 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0150001470 con 0x7f015c0ad070 2024-01-26T23:50:13.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.988+0000 7f0159ffb700 1 -- 172.21.15.46:0/328786668 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015400b850 con 0x7f015c0ad070 2024-01-26T23:50:13.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.988+0000 7f0159ffb700 1 -- 172.21.15.46:0/328786668 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015400bbf0 con 0x7f015c0ad070 2024-01-26T23:50:13.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.988+0000 7f01612eb700 1 -- 172.21.15.46:0/328786668 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f015c0ad070 msgr2=0x7f015c0ae610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:13.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.988+0000 7f01612eb700 1 --2- 172.21.15.46:0/328786668 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f015c0ad070 0x7f015c0ae610 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0154009e60 tx=0x7f0154009780 comp rx=0 tx=0).stop 2024-01-26T23:50:13.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.988+0000 7f01612eb700 1 -- 172.21.15.46:0/328786668 shutdown_connections 2024-01-26T23:50:13.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.988+0000 7f01612eb700 1 --2- 172.21.15.46:0/328786668 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f015c0ad070 0x7f015c0ae610 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:13.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.988+0000 7f01612eb700 1 -- 172.21.15.46:0/328786668 >> 172.21.15.46:0/328786668 conn(0x7f015c068400 msgr2=0x7f015c0cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:13.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.988+0000 7f01612eb700 1 -- 172.21.15.46:0/328786668 shutdown_connections 2024-01-26T23:50:13.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.988+0000 7f01612eb700 1 -- 172.21.15.46:0/328786668 wait complete. 2024-01-26T23:50:13.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.990+0000 7f01612eb700 1 Processor -- start 2024-01-26T23:50:13.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.990+0000 7f01612eb700 1 -- start start 2024-01-26T23:50:13.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.990+0000 7f01612eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f015c0ad070 0x7f015c153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:13.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f01612eb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f015c0cbdf0 0x7f015c0a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:13.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f01612eb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f015c159b90 0x7f015c0a5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:13.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f01612eb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c0aa3c0 con 0x7f015c0ad070 2024-01-26T23:50:13.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f01612eb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c0ab220 con 0x7f015c0cbdf0 2024-01-26T23:50:13.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f01612eb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c0aa610 con 0x7f015c159b90 2024-01-26T23:50:13.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f015affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f015c0ad070 0x7f015c153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:13.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f015affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f015c0ad070 0x7f015c153620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36366/0 (socket says 172.21.15.46:36366) 2024-01-26T23:50:13.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f015affd700 1 -- 172.21.15.46:0/1399124684 learned_addr learned my addr 172.21.15.46:0/1399124684 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:13.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f015a7fc700 1 --2- 172.21.15.46:0/1399124684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f015c0cbdf0 0x7f015c0a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:13.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f015b7fe700 1 --2- 172.21.15.46:0/1399124684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f015c159b90 0x7f015c0a5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:13.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f015affd700 1 -- 172.21.15.46:0/1399124684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f015c159b90 msgr2=0x7f015c0a5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:13.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f015affd700 1 --2- 172.21.15.46:0/1399124684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f015c159b90 0x7f015c0a5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:13.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f015affd700 1 -- 172.21.15.46:0/1399124684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f015c0cbdf0 msgr2=0x7f015c0a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:13.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f015affd700 1 --2- 172.21.15.46:0/1399124684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f015c0cbdf0 0x7f015c0a5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:13.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f015affd700 1 -- 172.21.15.46:0/1399124684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0154009a20 con 0x7f015c0ad070 2024-01-26T23:50:13.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f015a7fc700 1 --2- 172.21.15.46:0/1399124684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f015c0cbdf0 0x7f015c0a5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-26T23:50:13.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f015affd700 1 --2- 172.21.15.46:0/1399124684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f015c0ad070 0x7f015c153620 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f014c00c5c0 tx=0x7f014c00c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:13.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f0142ffd700 1 -- 172.21.15.46:0/1399124684 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c011590 con 0x7f015c0ad070 2024-01-26T23:50:13.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f0142ffd700 1 -- 172.21.15.46:0/1399124684 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c011710 con 0x7f015c0ad070 2024-01-26T23:50:13.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f01612eb700 1 -- 172.21.15.46:0/1399124684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015c0aa8f0 con 0x7f015c0ad070 2024-01-26T23:50:13.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.991+0000 7f0142ffd700 1 -- 172.21.15.46:0/1399124684 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c011ad0 con 0x7f015c0ad070 2024-01-26T23:50:13.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.992+0000 7f01612eb700 1 -- 172.21.15.46:0/1399124684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015c0aaeb0 con 0x7f015c0ad070 2024-01-26T23:50:13.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.993+0000 7f01612eb700 1 -- 172.21.15.46:0/1399124684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015c04c710 con 0x7f015c0ad070 2024-01-26T23:50:13.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.993+0000 7f0142ffd700 1 -- 172.21.15.46:0/1399124684 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f014c01b3b0 con 0x7f015c0ad070 2024-01-26T23:50:13.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.994+0000 7f0142ffd700 1 --2- 172.21.15.46:0/1399124684 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0148066f30 0x7f01480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:13.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.994+0000 7f0142ffd700 1 -- 172.21.15.46:0/1399124684 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f014c083e30 con 0x7f015c0ad070 2024-01-26T23:50:13.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.994+0000 7f015a7fc700 1 --2- 172.21.15.46:0/1399124684 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0148066f30 0x7f01480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:13.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.994+0000 7f015a7fc700 1 --2- 172.21.15.46:0/1399124684 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0148066f30 0x7f01480693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0150001620 tx=0x7f0150000bc0 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:13.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:13.998+0000 7f0142ffd700 1 -- 172.21.15.46:0/1399124684 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f014c050dd0 con 0x7f015c0ad070 2024-01-26T23:50:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.232+0000 7f01612eb700 1 -- 172.21.15.46:0/1399124684 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f015c0ad850 con 0x7f0148066f30 2024-01-26T23:50:14.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.234+0000 7f0142ffd700 1 -- 172.21.15.46:0/1399124684 <== mgr.8395 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f015c0ad850 con 0x7f0148066f30 2024-01-26T23:50:14.236 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-26T23:50:14.236 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-26T23:50:14.236 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephfs" 2024-01-26T23:50:14.236 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-26T23:50:14.236 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-26T23:50:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.236+0000 7f01612eb700 1 -- 172.21.15.46:0/1399124684 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0148066f30 msgr2=0x7f01480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.236+0000 7f01612eb700 1 --2- 172.21.15.46:0/1399124684 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0148066f30 0x7f01480693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0150001620 tx=0x7f0150000bc0 comp rx=0 tx=0).stop 2024-01-26T23:50:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.236+0000 7f01612eb700 1 -- 172.21.15.46:0/1399124684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f015c0ad070 msgr2=0x7f015c153620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.236+0000 7f01612eb700 1 --2- 172.21.15.46:0/1399124684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f015c0ad070 0x7f015c153620 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f014c00c5c0 tx=0x7f014c00c980 comp rx=0 tx=0).stop 2024-01-26T23:50:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.236+0000 7f01612eb700 1 -- 172.21.15.46:0/1399124684 shutdown_connections 2024-01-26T23:50:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.236+0000 7f01612eb700 1 --2- 172.21.15.46:0/1399124684 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0148066f30 0x7f01480693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.236+0000 7f01612eb700 1 --2- 172.21.15.46:0/1399124684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f015c0ad070 0x7f015c153620 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.236+0000 7f01612eb700 1 --2- 172.21.15.46:0/1399124684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f015c159b90 0x7f015c0a5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.236+0000 7f01612eb700 1 --2- 172.21.15.46:0/1399124684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f015c0cbdf0 0x7f015c0a5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.236+0000 7f01612eb700 1 -- 172.21.15.46:0/1399124684 >> 172.21.15.46:0/1399124684 conn(0x7f015c068400 msgr2=0x7f015c0ab6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:14.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.236+0000 7f01612eb700 1 -- 172.21.15.46:0/1399124684 shutdown_connections 2024-01-26T23:50:14.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.237+0000 7f01612eb700 1 -- 172.21.15.46:0/1399124684 wait complete. 2024-01-26T23:50:14.251 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-26T23:50:14.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.384+0000 7f3e92b07700 1 Processor -- start 2024-01-26T23:50:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.384+0000 7f3e92b07700 1 -- start start 2024-01-26T23:50:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.385+0000 7f3e92b07700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3e8c0c8da0 0x7f3e8c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.385+0000 7f3e92b07700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3e8c0caf30 0x7f3e8c0cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.385+0000 7f3e92b07700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8c0ccc00 con 0x7f3e8c140af0 2024-01-26T23:50:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.385+0000 7f3e92b07700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e8c0cbbf0 con 0x7f3e8c0caf30 2024-01-26T23:50:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.385+0000 7f3e92b07700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e8c0cc310 con 0x7f3e8c0c8da0 2024-01-26T23:50:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.385+0000 7f3e910a4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3e8c0caf30 0x7f3e8c0cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.385+0000 7f3e910a4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3e8c0caf30 0x7f3e8c0cb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36382/0 (socket says 172.21.15.46:36382) 2024-01-26T23:50:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.385+0000 7f3e910a4700 1 -- 172.21.15.46:0/3738030358 learned_addr learned my addr 172.21.15.46:0/3738030358 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.385+0000 7f3e908a3700 1 --2- 172.21.15.46:0/3738030358 >> v2:172.21.15.106:3300/0 conn(0x7f3e8c0c8da0 0x7f3e8c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.386+0000 7f3e910a4700 1 -- 172.21.15.46:0/3738030358 >> v2:172.21.15.106:3300/0 conn(0x7f3e8c0c8da0 msgr2=0x7f3e8c0c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.386+0000 7f3e910a4700 1 --2- 172.21.15.46:0/3738030358 >> v2:172.21.15.106:3300/0 conn(0x7f3e8c0c8da0 0x7f3e8c0c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.386+0000 7f3e910a4700 1 -- 172.21.15.46:0/3738030358 >> v1:172.21.15.106:6789/0 conn(0x7f3e8c140af0 legacy=0x7f3e8c140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.386+0000 7f3e910a4700 1 -- 172.21.15.46:0/3738030358 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c0cc5b0 con 0x7f3e8c0caf30 2024-01-26T23:50:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.386+0000 7f3e910a4700 1 --2- 172.21.15.46:0/3738030358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e8c0caf30 0x7f3e8c0cb370 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c06f100 tx=0x7f3e84009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c9a2d4fe9c1ca96 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.386+0000 7f3e8b7fe700 1 -- 172.21.15.46:0/3738030358 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8400e070 con 0x7f3e8c0caf30 2024-01-26T23:50:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.386+0000 7f3e8b7fe700 1 -- 172.21.15.46:0/3738030358 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8400ba20 con 0x7f3e8c0caf30 2024-01-26T23:50:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.386+0000 7f3e8b7fe700 1 -- 172.21.15.46:0/3738030358 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e840092e0 con 0x7f3e8c0caf30 2024-01-26T23:50:14.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.387+0000 7f3e92b07700 1 -- 172.21.15.46:0/3738030358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e8c0caf30 msgr2=0x7f3e8c0cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:14.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.387+0000 7f3e92b07700 1 --2- 172.21.15.46:0/3738030358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e8c0caf30 0x7f3e8c0cb370 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c06f100 tx=0x7f3e84009840 comp rx=0 tx=0).stop 2024-01-26T23:50:14.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.387+0000 7f3e92b07700 1 -- 172.21.15.46:0/3738030358 shutdown_connections 2024-01-26T23:50:14.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.387+0000 7f3e92b07700 1 --2- 172.21.15.46:0/3738030358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e8c0caf30 0x7f3e8c0cb370 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:14.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.387+0000 7f3e92b07700 1 --2- 172.21.15.46:0/3738030358 >> v2:172.21.15.106:3300/0 conn(0x7f3e8c0c8da0 0x7f3e8c0c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:14.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.387+0000 7f3e92b07700 1 -- 172.21.15.46:0/3738030358 >> 172.21.15.46:0/3738030358 conn(0x7f3e8c06bd90 msgr2=0x7f3e8c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:14.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.387+0000 7f3e92b07700 1 -- 172.21.15.46:0/3738030358 shutdown_connections 2024-01-26T23:50:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.387+0000 7f3e92b07700 1 -- 172.21.15.46:0/3738030358 wait complete. 2024-01-26T23:50:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.388+0000 7f3e92b07700 1 Processor -- start 2024-01-26T23:50:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.388+0000 7f3e92b07700 1 -- start start 2024-01-26T23:50:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.389+0000 7f3e92b07700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e8c0c8da0 0x7f3e8c0c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.389+0000 7f3e92b07700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3e8c0caf30 0x7f3e8c1416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.389+0000 7f3e92b07700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3e8c140af0 0x7f3e8c13c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.389+0000 7f3e92b07700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c0b22c0 con 0x7f3e8c0c8da0 2024-01-26T23:50:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.389+0000 7f3e92b07700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c137820 con 0x7f3e8c140af0 2024-01-26T23:50:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.389+0000 7f3e92b07700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c0b3480 con 0x7f3e8c0caf30 2024-01-26T23:50:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.389+0000 7f3e908a3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e8c0c8da0 0x7f3e8c0c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.389+0000 7f3e908a3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e8c0c8da0 0x7f3e8c0c6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36388/0 (socket says 172.21.15.46:36388) 2024-01-26T23:50:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.389+0000 7f3e908a3700 1 -- 172.21.15.46:0/1641943307 learned_addr learned my addr 172.21.15.46:0/1641943307 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.389+0000 7f3e910a4700 1 --2- 172.21.15.46:0/1641943307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3e8c140af0 0x7f3e8c13c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.389+0000 7f3e908a3700 1 -- 172.21.15.46:0/1641943307 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3e8c0caf30 msgr2=0x7f3e8c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.389+0000 7f3e8bfff700 1 --2- 172.21.15.46:0/1641943307 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3e8c0caf30 0x7f3e8c1416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.389+0000 7f3e908a3700 1 --2- 172.21.15.46:0/1641943307 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3e8c0caf30 0x7f3e8c1416d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.389+0000 7f3e908a3700 1 -- 172.21.15.46:0/1641943307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3e8c140af0 msgr2=0x7f3e8c13c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.389+0000 7f3e908a3700 1 --2- 172.21.15.46:0/1641943307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3e8c140af0 0x7f3e8c13c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.389+0000 7f3e908a3700 1 -- 172.21.15.46:0/1641943307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e84009ae0 con 0x7f3e8c0c8da0 2024-01-26T23:50:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.389+0000 7f3e910a4700 1 --2- 172.21.15.46:0/1641943307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3e8c140af0 0x7f3e8c13c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-26T23:50:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.390+0000 7f3e908a3700 1 --2- 172.21.15.46:0/1641943307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e8c0c8da0 0x7f3e8c0c6d40 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f3e8000ec80 tx=0x7f3e800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.390+0000 7f3e88ff9700 1 -- 172.21.15.46:0/1641943307 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8000c9d0 con 0x7f3e8c0c8da0 2024-01-26T23:50:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.390+0000 7f3e88ff9700 1 -- 172.21.15.46:0/1641943307 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8000cb30 con 0x7f3e8c0c8da0 2024-01-26T23:50:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.390+0000 7f3e92b07700 1 -- 172.21.15.46:0/1641943307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e8c0c7280 con 0x7f3e8c0c8da0 2024-01-26T23:50:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.390+0000 7f3e88ff9700 1 -- 172.21.15.46:0/1641943307 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80013470 con 0x7f3e8c0c8da0 2024-01-26T23:50:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.390+0000 7f3e92b07700 1 -- 172.21.15.46:0/1641943307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8c0b2e30 con 0x7f3e8c0c8da0 2024-01-26T23:50:14.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.391+0000 7f3e92b07700 1 -- 172.21.15.46:0/1641943307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e8c0b2c90 con 0x7f3e8c0c8da0 2024-01-26T23:50:14.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.391+0000 7f3e88ff9700 1 -- 172.21.15.46:0/1641943307 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 76) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3e800137e0 con 0x7f3e8c0c8da0 2024-01-26T23:50:14.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.392+0000 7f3e88ff9700 1 --2- 172.21.15.46:0/1641943307 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3e70066f30 0x7f3e700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:14.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.392+0000 7f3e88ff9700 1 -- 172.21.15.46:0/1641943307 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f3e80084b70 con 0x7f3e8c0c8da0 2024-01-26T23:50:14.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.392+0000 7f3e8bfff700 1 --2- 172.21.15.46:0/1641943307 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3e70066f30 0x7f3e700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:14.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.393+0000 7f3e8bfff700 1 --2- 172.21.15.46:0/1641943307 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3e70066f30 0x7f3e700693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c13cc20 tx=0x7f3e78009040 comp rx=0 tx=0).ready entity=mgr.8395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:14.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.396+0000 7f3e88ff9700 1 -- 172.21.15.46:0/1641943307 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3e8c0b2c90 con 0x7f3e8c0c8da0 2024-01-26T23:50:14.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.660+0000 7f3e92b07700 1 -- 172.21.15.46:0/1641943307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f3e8c0b2c90 con 0x7f3e8c0c8da0 2024-01-26T23:50:14.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.856+0000 7f3e88ff9700 1 -- 172.21.15.46:0/1641943307 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 77) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f3e80051ee0 con 0x7f3e8c0c8da0 2024-01-26T23:50:14.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.858+0000 7f3e88ff9700 1 -- 172.21.15.46:0/1641943307 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v77) v1 ==== 92+0+0 (secure 0 0 0) 0x7f3e800555a0 con 0x7f3e8c0c8da0 2024-01-26T23:50:14.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.861+0000 7f3e92b07700 1 -- 172.21.15.46:0/1641943307 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3e70066f30 msgr2=0x7f3e700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:14.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.861+0000 7f3e92b07700 1 --2- 172.21.15.46:0/1641943307 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3e70066f30 0x7f3e700693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c13cc20 tx=0x7f3e78009040 comp rx=0 tx=0).stop 2024-01-26T23:50:14.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.861+0000 7f3e92b07700 1 -- 172.21.15.46:0/1641943307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e8c0c8da0 msgr2=0x7f3e8c0c6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:14.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.861+0000 7f3e92b07700 1 --2- 172.21.15.46:0/1641943307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e8c0c8da0 0x7f3e8c0c6d40 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f3e8000ec80 tx=0x7f3e800091a0 comp rx=0 tx=0).stop 2024-01-26T23:50:14.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.862+0000 7f3e92b07700 1 -- 172.21.15.46:0/1641943307 shutdown_connections 2024-01-26T23:50:14.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.862+0000 7f3e92b07700 1 --2- 172.21.15.46:0/1641943307 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3e70066f30 0x7f3e700693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:14.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.862+0000 7f3e92b07700 1 --2- 172.21.15.46:0/1641943307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e8c0c8da0 0x7f3e8c0c6d40 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:14.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.862+0000 7f3e92b07700 1 --2- 172.21.15.46:0/1641943307 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3e8c140af0 0x7f3e8c13c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:14.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.862+0000 7f3e92b07700 1 --2- 172.21.15.46:0/1641943307 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3e8c0caf30 0x7f3e8c1416d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:14.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.862+0000 7f3e92b07700 1 -- 172.21.15.46:0/1641943307 >> 172.21.15.46:0/1641943307 conn(0x7f3e8c06bd90 msgr2=0x7f3e8c0c3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:14.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.862+0000 7f3e92b07700 1 -- 172.21.15.46:0/1641943307 shutdown_connections 2024-01-26T23:50:14.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:14.862+0000 7f3e92b07700 1 -- 172.21.15.46:0/1641943307 wait complete. 2024-01-26T23:50:14.879 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-26T23:50:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.017+0000 7fd17362d700 1 Processor -- start 2024-01-26T23:50:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.017+0000 7fd17362d700 1 -- start start 2024-01-26T23:50:15.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.018+0000 7fd17362d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd16c159510 0x7fd16c1598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.019+0000 7fd17362d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd16c0ca220 0x7fd16c0c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.019+0000 7fd17362d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c069260 con 0x7fd16c0ca690 2024-01-26T23:50:15.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.019+0000 7fd17362d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd16c15d670 con 0x7fd16c159510 2024-01-26T23:50:15.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.019+0000 7fd17362d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c0c9750 con 0x7fd16c0ca220 2024-01-26T23:50:15.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.019+0000 7fd1713c9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd16c159510 0x7fd16c1598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:15.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.019+0000 7fd171bca700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd16c0ca690 0x7fd16c0cc380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37820/0 (socket says 172.21.15.46:37820) 2024-01-26T23:50:15.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.019+0000 7fd171bca700 1 -- 172.21.15.46:0/690626159 learned_addr learned my addr 172.21.15.46:0/690626159 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.019+0000 7fd170bc8700 1 --2- 172.21.15.46:0/690626159 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd16c0ca220 0x7fd16c0c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.019+0000 7fd1713c9700 1 -- 172.21.15.46:0/690626159 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd16c0ca220 msgr2=0x7fd16c0c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.019+0000 7fd1713c9700 1 --2- 172.21.15.46:0/690626159 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd16c0ca220 0x7fd16c0c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.019+0000 7fd1713c9700 1 -- 172.21.15.46:0/690626159 >> v1:172.21.15.46:6789/0 conn(0x7fd16c0ca690 legacy=0x7fd16c0cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.019+0000 7fd1713c9700 1 -- 172.21.15.46:0/690626159 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c0c9a30 con 0x7fd16c159510 2024-01-26T23:50:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.019+0000 7fd1713c9700 1 --2- 172.21.15.46:0/690626159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd16c159510 0x7fd16c1598e0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fd168009910 tx=0x7fd1680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a61a032b532d4de server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.019+0000 7fd163fff700 1 -- 172.21.15.46:0/690626159 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16800b640 con 0x7fd16c159510 2024-01-26T23:50:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.020+0000 7fd163fff700 1 -- 172.21.15.46:0/690626159 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16800b7a0 con 0x7fd16c159510 2024-01-26T23:50:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.020+0000 7fd163fff700 1 -- 172.21.15.46:0/690626159 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16800bb60 con 0x7fd16c159510 2024-01-26T23:50:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.020+0000 7fd17362d700 1 -- 172.21.15.46:0/690626159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd16c159510 msgr2=0x7fd16c1598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.020+0000 7fd17362d700 1 --2- 172.21.15.46:0/690626159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd16c159510 0x7fd16c1598e0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fd168009910 tx=0x7fd1680091a0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.020+0000 7fd17362d700 1 -- 172.21.15.46:0/690626159 shutdown_connections 2024-01-26T23:50:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.020+0000 7fd17362d700 1 --2- 172.21.15.46:0/690626159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd16c159510 0x7fd16c1598e0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.020+0000 7fd17362d700 1 --2- 172.21.15.46:0/690626159 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd16c0ca220 0x7fd16c0c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.020+0000 7fd17362d700 1 -- 172.21.15.46:0/690626159 >> 172.21.15.46:0/690626159 conn(0x7fd16c06bd90 msgr2=0x7fd16c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.020+0000 7fd17362d700 1 -- 172.21.15.46:0/690626159 shutdown_connections 2024-01-26T23:50:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.021+0000 7fd17362d700 1 -- 172.21.15.46:0/690626159 wait complete. 2024-01-26T23:50:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.022+0000 7fd17362d700 1 Processor -- start 2024-01-26T23:50:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.022+0000 7fd17362d700 1 -- start start 2024-01-26T23:50:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.023+0000 7fd17362d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd16c0ca220 0x7fd16c151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.023+0000 7fd17362d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd16c159510 0x7fd16c0b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.023+0000 7fd17362d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd16c15e520 0x7fd16c15e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.023+0000 7fd17362d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c0b38a0 con 0x7fd16c0ca220 2024-01-26T23:50:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.023+0000 7fd17362d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c15eee0 con 0x7fd16c159510 2024-01-26T23:50:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.023+0000 7fd17362d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c16dc80 con 0x7fd16c15e520 2024-01-26T23:50:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.023+0000 7fd1713c9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd16c0ca220 0x7fd16c151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.023+0000 7fd1713c9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd16c0ca220 0x7fd16c151d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36410/0 (socket says 172.21.15.46:36410) 2024-01-26T23:50:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.023+0000 7fd1713c9700 1 -- 172.21.15.46:0/4029329166 learned_addr learned my addr 172.21.15.46:0/4029329166 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.023+0000 7fd170bc8700 1 --2- 172.21.15.46:0/4029329166 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd16c159510 0x7fd16c0b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.023+0000 7fd171bca700 1 --2- 172.21.15.46:0/4029329166 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd16c15e520 0x7fd16c15e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:15.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.023+0000 7fd1713c9700 1 -- 172.21.15.46:0/4029329166 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd16c15e520 msgr2=0x7fd16c15e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:15.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.023+0000 7fd1713c9700 1 --2- 172.21.15.46:0/4029329166 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd16c15e520 0x7fd16c15e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.023+0000 7fd1713c9700 1 -- 172.21.15.46:0/4029329166 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd16c159510 msgr2=0x7fd16c0b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:15.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.023+0000 7fd1713c9700 1 --2- 172.21.15.46:0/4029329166 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd16c159510 0x7fd16c0b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.023+0000 7fd1713c9700 1 -- 172.21.15.46:0/4029329166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd168009420 con 0x7fd16c0ca220 2024-01-26T23:50:15.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.024+0000 7fd1713c9700 1 --2- 172.21.15.46:0/4029329166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd16c0ca220 0x7fd16c151d50 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fd16c06f5a0 tx=0x7fd168017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:15.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.024+0000 7fd1617fa700 1 -- 172.21.15.46:0/4029329166 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16801f070 con 0x7fd16c0ca220 2024-01-26T23:50:15.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.024+0000 7fd1617fa700 1 -- 172.21.15.46:0/4029329166 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16800b910 con 0x7fd16c0ca220 2024-01-26T23:50:15.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.024+0000 7fd17362d700 1 -- 172.21.15.46:0/4029329166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16c16d390 con 0x7fd16c0ca220 2024-01-26T23:50:15.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.024+0000 7fd1617fa700 1 -- 172.21.15.46:0/4029329166 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd168011420 con 0x7fd16c0ca220 2024-01-26T23:50:15.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.024+0000 7fd17362d700 1 -- 172.21.15.46:0/4029329166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16c16d910 con 0x7fd16c0ca220 2024-01-26T23:50:15.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.025+0000 7fd1617fa700 1 -- 172.21.15.46:0/4029329166 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 77) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fd168011790 con 0x7fd16c0ca220 2024-01-26T23:50:15.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.025+0000 7fd17362d700 1 -- 172.21.15.46:0/4029329166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16c068a40 con 0x7fd16c0ca220 2024-01-26T23:50:15.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.026+0000 7fd1617fa700 1 --2- 172.21.15.46:0/4029329166 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd150066f80 0x7fd150069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.026+0000 7fd1617fa700 1 -- 172.21.15.46:0/4029329166 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fd1680858b0 con 0x7fd16c0ca220 2024-01-26T23:50:15.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.026+0000 7fd170bc8700 1 -- 172.21.15.46:0/4029329166 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd150066f80 msgr2=0x7fd150069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:50:15.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.026+0000 7fd170bc8700 1 --2- 172.21.15.46:0/4029329166 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd150066f80 0x7fd150069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:50:15.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.030+0000 7fd1617fa700 1 -- 172.21.15.46:0/4029329166 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd168052850 con 0x7fd16c0ca220 2024-01-26T23:50:15.192 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:15.192+0000 7fb1bb8491c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:50:15.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.227+0000 7fd170bc8700 1 -- 172.21.15.46:0/4029329166 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd150066f80 msgr2=0x7fd150069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:50:15.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.227+0000 7fd170bc8700 1 --2- 172.21.15.46:0/4029329166 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd150066f80 0x7fd150069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:50:15.239 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:15.238+0000 7f2f43f9f1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-26T23:50:15.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.239+0000 7fd17362d700 1 -- 172.21.15.46:0/4029329166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fd16c04c710 con 0x7fd16c0ca220 2024-01-26T23:50:15.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.246+0000 7fd1617fa700 1 -- 172.21.15.46:0/4029329166 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd168055e70 con 0x7fd16c0ca220 2024-01-26T23:50:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.248+0000 7fd17362d700 1 -- 172.21.15.46:0/4029329166 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd150066f80 msgr2=0x7fd150069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:50:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.248+0000 7fd17362d700 1 --2- 172.21.15.46:0/4029329166 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd150066f80 0x7fd150069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.249+0000 7fd17362d700 1 -- 172.21.15.46:0/4029329166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd16c0ca220 msgr2=0x7fd16c151d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.249+0000 7fd17362d700 1 --2- 172.21.15.46:0/4029329166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd16c0ca220 0x7fd16c151d50 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fd16c06f5a0 tx=0x7fd168017000 comp rx=0 tx=0).stop 2024-01-26T23:50:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.249+0000 7fd17362d700 1 -- 172.21.15.46:0/4029329166 shutdown_connections 2024-01-26T23:50:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.249+0000 7fd17362d700 1 --2- 172.21.15.46:0/4029329166 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd150066f80 0x7fd150069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.249+0000 7fd17362d700 1 --2- 172.21.15.46:0/4029329166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd16c0ca220 0x7fd16c151d50 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.249+0000 7fd17362d700 1 --2- 172.21.15.46:0/4029329166 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd16c15e520 0x7fd16c15e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.249+0000 7fd17362d700 1 --2- 172.21.15.46:0/4029329166 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd16c159510 0x7fd16c0b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.249+0000 7fd17362d700 1 -- 172.21.15.46:0/4029329166 >> 172.21.15.46:0/4029329166 conn(0x7fd16c06bd90 msgr2=0x7fd16c0c8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:15.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.249+0000 7fd17362d700 1 -- 172.21.15.46:0/4029329166 shutdown_connections 2024-01-26T23:50:15.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.249+0000 7fd17362d700 1 -- 172.21.15.46:0/4029329166 wait complete. 2024-01-26T23:50:15.264 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-26T23:50:15.269 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:15.268+0000 7fb1bb8491c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:50:15.334 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:15.333+0000 7f2f43f9f1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-26T23:50:15.339 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:15.339+0000 7fb1bb8491c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:50:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.400+0000 7fe17300f700 1 Processor -- start 2024-01-26T23:50:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.400+0000 7fe17300f700 1 -- start start 2024-01-26T23:50:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.400+0000 7fe17300f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe16c068940 0x7fe16c068d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.400+0000 7fe17300f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe16c0c8c50 0x7fe16c0c7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.400+0000 7fe17300f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe16c0c76a0 0x7fe16c0af990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.400+0000 7fe17300f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe16c0c87c0 con 0x7fe16c068940 2024-01-26T23:50:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.400+0000 7fe17300f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe16c069250 con 0x7fe16c0c8c50 2024-01-26T23:50:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.400+0000 7fe17300f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c0c7bb0 con 0x7fe16c0c76a0 2024-01-26T23:50:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.401+0000 7fe170dab700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe16c068940 0x7fe16c068d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.401+0000 7fe170dab700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe16c068940 0x7fe16c068d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36412/0 (socket says 172.21.15.46:36412) 2024-01-26T23:50:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.401+0000 7fe170dab700 1 -- 172.21.15.46:0/924615745 learned_addr learned my addr 172.21.15.46:0/924615745 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.401+0000 7fe1715ac700 1 --2- 172.21.15.46:0/924615745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe16c0c76a0 0x7fe16c0af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.401+0000 7fe16bfff700 1 --2- 172.21.15.46:0/924615745 >> v2:172.21.15.106:3300/0 conn(0x7fe16c0c8c50 0x7fe16c0c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.401+0000 7fe170dab700 1 -- 172.21.15.46:0/924615745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe16c0c76a0 msgr2=0x7fe16c0af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.401+0000 7fe170dab700 1 --2- 172.21.15.46:0/924615745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe16c0c76a0 0x7fe16c0af990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.401+0000 7fe170dab700 1 -- 172.21.15.46:0/924615745 >> v2:172.21.15.106:3300/0 conn(0x7fe16c0c8c50 msgr2=0x7fe16c0c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.401+0000 7fe170dab700 1 --2- 172.21.15.46:0/924615745 >> v2:172.21.15.106:3300/0 conn(0x7fe16c0c8c50 0x7fe16c0c7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.401+0000 7fe170dab700 1 -- 172.21.15.46:0/924615745 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c0c7e90 con 0x7fe16c068940 2024-01-26T23:50:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.401+0000 7fe170dab700 1 --2- 172.21.15.46:0/924615745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c068940 0x7fe16c068d10 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0ccb20 tx=0x7fe160009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa917f39622d0647 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.401+0000 7fe16b7fe700 1 -- 172.21.15.46:0/924615745 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe160005c10 con 0x7fe16c068940 2024-01-26T23:50:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.401+0000 7fe16b7fe700 1 -- 172.21.15.46:0/924615745 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe160005d70 con 0x7fe16c068940 2024-01-26T23:50:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.401+0000 7fe16b7fe700 1 -- 172.21.15.46:0/924615745 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe160006260 con 0x7fe16c068940 2024-01-26T23:50:15.405 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:15.403+0000 7fb1bb8491c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:50:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.402+0000 7fe17300f700 1 -- 172.21.15.46:0/924615745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c068940 msgr2=0x7fe16c068d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.402+0000 7fe17300f700 1 --2- 172.21.15.46:0/924615745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c068940 0x7fe16c068d10 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0ccb20 tx=0x7fe160009220 comp rx=0 tx=0).stop 2024-01-26T23:50:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.402+0000 7fe17300f700 1 -- 172.21.15.46:0/924615745 shutdown_connections 2024-01-26T23:50:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.402+0000 7fe17300f700 1 --2- 172.21.15.46:0/924615745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c068940 0x7fe16c068d10 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.402+0000 7fe17300f700 1 --2- 172.21.15.46:0/924615745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe16c0c76a0 0x7fe16c0af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.402+0000 7fe17300f700 1 --2- 172.21.15.46:0/924615745 >> v2:172.21.15.106:3300/0 conn(0x7fe16c0c8c50 0x7fe16c0c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.402+0000 7fe17300f700 1 -- 172.21.15.46:0/924615745 >> 172.21.15.46:0/924615745 conn(0x7fe16c0ca100 msgr2=0x7fe16c06e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.402+0000 7fe17300f700 1 -- 172.21.15.46:0/924615745 shutdown_connections 2024-01-26T23:50:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.402+0000 7fe17300f700 1 -- 172.21.15.46:0/924615745 wait complete. 2024-01-26T23:50:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.403+0000 7fe17300f700 1 Processor -- start 2024-01-26T23:50:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.404+0000 7fe17300f700 1 -- start start 2024-01-26T23:50:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.404+0000 7fe17300f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c068940 0x7fe16c15aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.404+0000 7fe17300f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe16c0c76a0 0x7fe16c0c0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.404+0000 7fe17300f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe16c0c8c50 0x7fe16c0c1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.404+0000 7fe17300f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c168bf0 con 0x7fe16c068940 2024-01-26T23:50:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.404+0000 7fe17300f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c15af70 con 0x7fe16c0c76a0 2024-01-26T23:50:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.404+0000 7fe17300f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c0c5560 con 0x7fe16c0c8c50 2024-01-26T23:50:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.405+0000 7fe170dab700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c068940 0x7fe16c15aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.405+0000 7fe170dab700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c068940 0x7fe16c15aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36416/0 (socket says 172.21.15.46:36416) 2024-01-26T23:50:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.405+0000 7fe170dab700 1 -- 172.21.15.46:0/802501225 learned_addr learned my addr 172.21.15.46:0/802501225 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.405+0000 7fe16bfff700 1 --2- 172.21.15.46:0/802501225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe16c0c76a0 0x7fe16c0c0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.405+0000 7fe1715ac700 1 --2- 172.21.15.46:0/802501225 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe16c0c8c50 0x7fe16c0c1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.405+0000 7fe170dab700 1 -- 172.21.15.46:0/802501225 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe16c0c8c50 msgr2=0x7fe16c0c1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.405+0000 7fe170dab700 1 --2- 172.21.15.46:0/802501225 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe16c0c8c50 0x7fe16c0c1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.405+0000 7fe170dab700 1 -- 172.21.15.46:0/802501225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe16c0c76a0 msgr2=0x7fe16c0c0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.405+0000 7fe170dab700 1 --2- 172.21.15.46:0/802501225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe16c0c76a0 0x7fe16c0c0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.405+0000 7fe170dab700 1 -- 172.21.15.46:0/802501225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe160009480 con 0x7fe16c068940 2024-01-26T23:50:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.405+0000 7fe170dab700 1 --2- 172.21.15.46:0/802501225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c068940 0x7fe16c15aa30 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe160006a50 tx=0x7fe160016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.405+0000 7fe168ff9700 1 -- 172.21.15.46:0/802501225 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe160011850 con 0x7fe16c068940 2024-01-26T23:50:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.405+0000 7fe168ff9700 1 -- 172.21.15.46:0/802501225 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1600119b0 con 0x7fe16c068940 2024-01-26T23:50:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.405+0000 7fe17300f700 1 -- 172.21.15.46:0/802501225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16c0ac1b0 con 0x7fe16c068940 2024-01-26T23:50:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.405+0000 7fe168ff9700 1 -- 172.21.15.46:0/802501225 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe160011d50 con 0x7fe16c068940 2024-01-26T23:50:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.405+0000 7fe17300f700 1 -- 172.21.15.46:0/802501225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16c0a58e0 con 0x7fe16c068940 2024-01-26T23:50:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.406+0000 7fe17300f700 1 -- 172.21.15.46:0/802501225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe16c0a5440 con 0x7fe16c068940 2024-01-26T23:50:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.407+0000 7fe168ff9700 1 -- 172.21.15.46:0/802501225 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 77) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fe16001b640 con 0x7fe16c068940 2024-01-26T23:50:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.408+0000 7fe168ff9700 1 --2- 172.21.15.46:0/802501225 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe150066f80 0x7fe150069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.408+0000 7fe168ff9700 1 -- 172.21.15.46:0/802501225 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fe160085fb0 con 0x7fe16c068940 2024-01-26T23:50:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.408+0000 7fe16bfff700 1 -- 172.21.15.46:0/802501225 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe150066f80 msgr2=0x7fe150069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:50:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.408+0000 7fe16bfff700 1 --2- 172.21.15.46:0/802501225 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe150066f80 0x7fe150069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:50:15.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.412+0000 7fe168ff9700 1 -- 172.21.15.46:0/802501225 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe1600b75a0 con 0x7fe16c068940 2024-01-26T23:50:15.419 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:15.418+0000 7f2f43f9f1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-26T23:50:15.495 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:15.494+0000 7f2f43f9f1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-26T23:50:15.560 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:15.559+0000 7fb1bb8491c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:50:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.609+0000 7fe16bfff700 1 -- 172.21.15.46:0/802501225 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe150066f80 msgr2=0x7fe150069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:50:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.609+0000 7fe16bfff700 1 --2- 172.21.15.46:0/802501225 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe150066f80 0x7fe150069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:50:15.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.619+0000 7fe17300f700 1 -- 172.21.15.46:0/802501225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fe16c0c9470 con 0x7fe16c068940 2024-01-26T23:50:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.623+0000 7fe168ff9700 1 -- 172.21.15.46:0/802501225 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe160052f50 con 0x7fe16c068940 2024-01-26T23:50:15.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.626+0000 7fe17300f700 1 -- 172.21.15.46:0/802501225 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe150066f80 msgr2=0x7fe150069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:50:15.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.626+0000 7fe17300f700 1 --2- 172.21.15.46:0/802501225 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe150066f80 0x7fe150069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.626+0000 7fe17300f700 1 -- 172.21.15.46:0/802501225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c068940 msgr2=0x7fe16c15aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.626+0000 7fe17300f700 1 --2- 172.21.15.46:0/802501225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c068940 0x7fe16c15aa30 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe160006a50 tx=0x7fe160016000 comp rx=0 tx=0).stop 2024-01-26T23:50:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.626+0000 7fe17300f700 1 -- 172.21.15.46:0/802501225 shutdown_connections 2024-01-26T23:50:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.626+0000 7fe17300f700 1 --2- 172.21.15.46:0/802501225 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe150066f80 0x7fe150069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.626+0000 7fe17300f700 1 --2- 172.21.15.46:0/802501225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe16c068940 0x7fe16c15aa30 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.626+0000 7fe17300f700 1 --2- 172.21.15.46:0/802501225 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe16c0c8c50 0x7fe16c0c1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.626+0000 7fe17300f700 1 --2- 172.21.15.46:0/802501225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe16c0c76a0 0x7fe16c0c0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.626+0000 7fe17300f700 1 -- 172.21.15.46:0/802501225 >> 172.21.15.46:0/802501225 conn(0x7fe16c0ca100 msgr2=0x7fe16c0b0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.626+0000 7fe17300f700 1 -- 172.21.15.46:0/802501225 shutdown_connections 2024-01-26T23:50:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.626+0000 7fe17300f700 1 -- 172.21.15.46:0/802501225 wait complete. 2024-01-26T23:50:15.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-26T23:50:15.642 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-26T23:50:15.667 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1 1m 2024-01-26T23:50:15.674 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:15.673+0000 7f2f43f9f1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:50:15.696 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:15.695+0000 7fb1bb8491c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:50:15.762 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:15.761+0000 7fb1bb8491c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:50:15.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.809+0000 7f053631b700 1 Processor -- start 2024-01-26T23:50:15.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.809+0000 7f053631b700 1 -- start start 2024-01-26T23:50:15.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.810+0000 7f053631b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0530157750 0x7f0530157b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.810+0000 7f053631b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053006b3c0 0x7f05300af080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.810+0000 7f053631b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0530158f00 con 0x7f05300af5c0 2024-01-26T23:50:15.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.810+0000 7f053631b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05300afa20 con 0x7f0530157750 2024-01-26T23:50:15.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.810+0000 7f053631b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0530158060 con 0x7f053006b3c0 2024-01-26T23:50:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.810+0000 7f05348b8700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f05300af5c0 0x7f0530067930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37828/0 (socket says 172.21.15.46:37828) 2024-01-26T23:50:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.810+0000 7f05348b8700 1 -- 172.21.15.46:0/1594128831 learned_addr learned my addr 172.21.15.46:0/1594128831 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-26T23:50:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.810+0000 7f052ffff700 1 --2- 172.21.15.46:0/1594128831 >> v2:172.21.15.106:3300/0 conn(0x7f0530157750 0x7f0530157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.810+0000 7f052f7fe700 1 --2- 172.21.15.46:0/1594128831 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053006b3c0 0x7f05300af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.810+0000 7f052effd700 1 -- 172.21.15.46:0/1594128831 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873935320 0 0) 0x7f0530158f00 con 0x7f05300af5c0 2024-01-26T23:50:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.810+0000 7f052effd700 1 -- 172.21.15.46:0/1594128831 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f051c003570 con 0x7f05300af5c0 2024-01-26T23:50:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.811+0000 7f052ffff700 1 -- 172.21.15.46:0/1594128831 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053006b3c0 msgr2=0x7f05300af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.811+0000 7f052ffff700 1 --2- 172.21.15.46:0/1594128831 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053006b3c0 0x7f05300af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.811+0000 7f052ffff700 1 -- 172.21.15.46:0/1594128831 >> v1:172.21.15.46:6789/0 conn(0x7f05300af5c0 legacy=0x7f0530067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.811+0000 7f052ffff700 1 -- 172.21.15.46:0/1594128831 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053006fe90 con 0x7f0530157750 2024-01-26T23:50:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.811+0000 7f052f7fe700 1 --2- 172.21.15.46:0/1594128831 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053006b3c0 0x7f05300af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:50:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.811+0000 7f052ffff700 1 --2- 172.21.15.46:0/1594128831 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0530157750 0x7f0530157b20 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0530070080 tx=0x7f05200091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86e110bca8a1a1e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.811+0000 7f052effd700 1 -- 172.21.15.46:0/1594128831 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0520011630 con 0x7f0530157750 2024-01-26T23:50:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.811+0000 7f052effd700 1 -- 172.21.15.46:0/1594128831 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0520011790 con 0x7f0530157750 2024-01-26T23:50:15.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.811+0000 7f052effd700 1 -- 172.21.15.46:0/1594128831 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0520011b50 con 0x7f0530157750 2024-01-26T23:50:15.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.812+0000 7f053631b700 1 -- 172.21.15.46:0/1594128831 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0530157750 msgr2=0x7f0530157b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:15.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.812+0000 7f053631b700 1 --2- 172.21.15.46:0/1594128831 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0530157750 0x7f0530157b20 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0530070080 tx=0x7f05200091a0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.812+0000 7f053631b700 1 -- 172.21.15.46:0/1594128831 shutdown_connections 2024-01-26T23:50:15.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.812+0000 7f053631b700 1 --2- 172.21.15.46:0/1594128831 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0530157750 0x7f0530157b20 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.812+0000 7f053631b700 1 --2- 172.21.15.46:0/1594128831 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053006b3c0 0x7f05300af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.812+0000 7f053631b700 1 -- 172.21.15.46:0/1594128831 >> 172.21.15.46:0/1594128831 conn(0x7f05300cdf80 msgr2=0x7f053006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:15.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.812+0000 7f053631b700 1 -- 172.21.15.46:0/1594128831 shutdown_connections 2024-01-26T23:50:15.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.812+0000 7f053631b700 1 -- 172.21.15.46:0/1594128831 wait complete. 2024-01-26T23:50:15.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.813+0000 7f053631b700 1 Processor -- start 2024-01-26T23:50:15.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.813+0000 7f053631b700 1 -- start start 2024-01-26T23:50:15.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.814+0000 7f053631b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053006b3c0 0x7f05301697d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.814+0000 7f053631b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05300af5c0 0x7f0530153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.814+0000 7f053631b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0530157750 0x7f05300cc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.814+0000 7f053631b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05300aa7c0 con 0x7f05300af5c0 2024-01-26T23:50:15.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.814+0000 7f053631b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05300a9bb0 con 0x7f0530157750 2024-01-26T23:50:15.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.814+0000 7f053631b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05300a9e90 con 0x7f053006b3c0 2024-01-26T23:50:15.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.814+0000 7f052f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05300af5c0 0x7f0530153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:15.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.814+0000 7f052f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05300af5c0 0x7f0530153970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36418/0 (socket says 172.21.15.46:36418) 2024-01-26T23:50:15.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.814+0000 7f052f7fe700 1 -- 172.21.15.46:0/456429076 learned_addr learned my addr 172.21.15.46:0/456429076 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:15.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.814+0000 7f05348b8700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0530157750 0x7f05300cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:15.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.814+0000 7f052ffff700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053006b3c0 0x7f05301697d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:15.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.814+0000 7f052f7fe700 1 -- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053006b3c0 msgr2=0x7f05301697d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:15.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.814+0000 7f052f7fe700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053006b3c0 0x7f05301697d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.814+0000 7f052f7fe700 1 -- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0530157750 msgr2=0x7f05300cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:15.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.814+0000 7f052f7fe700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0530157750 0x7f05300cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:15.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.814+0000 7f052f7fe700 1 -- 172.21.15.46:0/456429076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0518009420 con 0x7f05300af5c0 2024-01-26T23:50:15.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.814+0000 7f05348b8700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0530157750 0x7f05300cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:50:15.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.815+0000 7f052f7fe700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05300af5c0 0x7f0530153970 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f051800ec80 tx=0x7f05180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:15.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.815+0000 7f0527fff700 1 -- 172.21.15.46:0/456429076 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051800c920 con 0x7f05300af5c0 2024-01-26T23:50:15.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.815+0000 7f0527fff700 1 -- 172.21.15.46:0/456429076 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051800ca80 con 0x7f05300af5c0 2024-01-26T23:50:15.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.815+0000 7f053631b700 1 -- 172.21.15.46:0/456429076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0520009420 con 0x7f05300af5c0 2024-01-26T23:50:15.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.815+0000 7f0527fff700 1 -- 172.21.15.46:0/456429076 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051800ce00 con 0x7f05300af5c0 2024-01-26T23:50:15.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.815+0000 7f053631b700 1 -- 172.21.15.46:0/456429076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05300aa450 con 0x7f05300af5c0 2024-01-26T23:50:15.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.816+0000 7f053631b700 1 -- 172.21.15.46:0/456429076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053004c710 con 0x7f05300af5c0 2024-01-26T23:50:15.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.816+0000 7f0527fff700 1 -- 172.21.15.46:0/456429076 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 77) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f0518013680 con 0x7f05300af5c0 2024-01-26T23:50:15.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.817+0000 7f0527fff700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 0x7f0510069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:15.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.817+0000 7f052ffff700 1 -- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 msgr2=0x7f0510069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:50:15.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.817+0000 7f0527fff700 1 -- 172.21.15.46:0/456429076 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f0518084f80 con 0x7f05300af5c0 2024-01-26T23:50:15.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.817+0000 7f052ffff700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 0x7f0510069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-26T23:50:15.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:15.826+0000 7f0527fff700 1 -- 172.21.15.46:0/456429076 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f05180b65a0 con 0x7f05300af5c0 2024-01-26T23:50:15.839 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:15.838+0000 7f2f43f9f1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-26T23:50:15.885 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:15.884+0000 7fb1bb8491c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:50:15.916 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:15.915+0000 7f2f43f9f1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-26T23:50:15.958 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:15.957+0000 7fb1bb8491c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:50:16.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:16.018+0000 7f052ffff700 1 -- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 msgr2=0x7f0510069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:50:16.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:16.018+0000 7f052ffff700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 0x7f0510069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-26T23:50:16.055 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:16.054+0000 7f2f43f9f1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-26T23:50:16.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:16.062+0000 7f053631b700 1 -- 172.21.15.46:0/456429076 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f05300aa900 con 0x7f0510066f80 2024-01-26T23:50:16.097 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:16.096+0000 7fb1bb8491c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:50:16.137 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:16.136+0000 7f2f43f9f1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-26T23:50:16.223 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:16.222+0000 7fb1bb8491c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:50:16.313 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:16.312+0000 7f2f43f9f1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-26T23:50:16.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:16.419+0000 7f052ffff700 1 -- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 msgr2=0x7f0510069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:50:16.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:16.419+0000 7f052ffff700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 0x7f0510069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-26T23:50:16.454 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:16.453+0000 7f2f43f9f1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-26T23:50:16.536 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:16.535+0000 7fb1bb8491c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:50:16.749 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:16.748+0000 7fb1bb8491c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:50:16.814 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:16.813+0000 7f2f43f9f1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-26T23:50:16.977 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:16.976+0000 7fb1bb8491c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:50:17.048 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:17.048+0000 7fb1bb8491c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:50:17.054 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:17.053+0000 7f2f43f9f1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-26T23:50:17.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:17.220+0000 7f052ffff700 1 -- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 msgr2=0x7f0510069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:50:17.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:17.220+0000 7f052ffff700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 0x7f0510069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-26T23:50:17.314 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:17.313+0000 7f2f43f9f1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-26T23:50:17.395 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:17.393+0000 7f2f43f9f1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-26T23:50:17.652 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:17.651+0000 7fb1bb8491c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:50:17.727 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:17.726+0000 7fb1bb8491c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:50:17.849 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:17.848+0000 7fb1bb8491c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:50:18.086 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:18.085+0000 7f2f43f9f1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-26T23:50:18.171 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:18.170+0000 7f2f43f9f1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-26T23:50:18.312 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:18.310+0000 7f2f43f9f1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-26T23:50:18.350 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:18.349+0000 7fb1bb8491c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:50:18.497 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:18.497+0000 7fb1bb8491c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:50:18.648 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:18.647+0000 7fb1bb8491c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:50:18.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:18.822+0000 7f052ffff700 1 -- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 msgr2=0x7f0510069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-26T23:50:18.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:18.822+0000 7f052ffff700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 0x7f0510069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-26T23:50:18.893 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:18.892+0000 7f2f43f9f1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-26T23:50:19.066 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:19.065+0000 7f2f43f9f1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-26T23:50:19.238 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:19.237+0000 7f2f43f9f1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-26T23:50:19.348 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:19.347+0000 7fb1bb8491c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:50:19.569 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:19.568+0000 7fb1bb8491c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:50:19.641 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:19.640+0000 7fb1bb8491c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:50:19.796 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:19.795+0000 7fb1bb8491c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:50:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:19.818+0000 7f0527fff700 1 -- 172.21.15.46:0/456429076 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 78) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f05180522f0 con 0x7f05300af5c0 2024-01-26T23:50:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:19.818+0000 7f0527fff700 1 -- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 msgr2=0x7f0510069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-26T23:50:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:19.818+0000 7f0527fff700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 0x7f0510069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:19.917 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:19.916+0000 7fb132127700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:50:19.917 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:19.916+0000 7fb132127700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:50:19.917 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:19.916+0000 7fb132127700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:50:19.917 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:19.916+0000 7fb132127700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:50:19.917 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:19.916+0000 7fb132127700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:50:19.917 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:19.916+0000 7fb13512d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:50:19.918 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:19.916+0000 7fb13512d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:50:19.918 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:19.916+0000 7fb13512d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:50:19.918 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:19.916+0000 7fb13512d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:50:19.918 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:50:19.916+0000 7fb13512d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:50:20.057 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:20.056+0000 7f2f43f9f1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-26T23:50:20.308 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:20.307+0000 7f2f43f9f1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-26T23:50:20.391 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:20.390+0000 7f2f43f9f1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-26T23:50:20.570 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-26T23:50:20.569+0000 7f2f43f9f1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-26T23:50:20.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.819+0000 7f0527fff700 1 -- 172.21.15.46:0/456429076 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 79) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f0518059090 con 0x7f05300af5c0 2024-01-26T23:50:20.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.819+0000 7f0527fff700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 0x7f05100693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:20.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.819+0000 7f0527fff700 1 -- 172.21.15.46:0/456429076 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f05300aa900 con 0x7f0510066f80 2024-01-26T23:50:20.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.821+0000 7f052ffff700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 0x7f05100693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:20.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.821+0000 7f052ffff700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 0x7f05100693b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0530070080 tx=0x7f0520016000 comp rx=0 tx=0).ready entity=mgr.8743 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:20.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.879+0000 7f0527fff700 1 -- 172.21.15.46:0/456429076 <== mgr.8743 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f05300aa900 con 0x7f0510066f80 2024-01-26T23:50:20.881 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path /snap_test_dir1 2024-01-26T23:50:20.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.882+0000 7f053631b700 1 -- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 msgr2=0x7f05100693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:20.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.883+0000 7f053631b700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 0x7f05100693b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0530070080 tx=0x7f0520016000 comp rx=0 tx=0).stop 2024-01-26T23:50:20.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.883+0000 7f053631b700 1 -- 172.21.15.46:0/456429076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05300af5c0 msgr2=0x7f0530153970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:20.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.883+0000 7f053631b700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05300af5c0 0x7f0530153970 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f051800ec80 tx=0x7f05180091a0 comp rx=0 tx=0).stop 2024-01-26T23:50:20.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.883+0000 7f053631b700 1 -- 172.21.15.46:0/456429076 shutdown_connections 2024-01-26T23:50:20.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.883+0000 7f053631b700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0510066f80 0x7f05100693b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:20.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.883+0000 7f053631b700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0530157750 0x7f05300cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:20.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.883+0000 7f053631b700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053006b3c0 0x7f05301697d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:20.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.883+0000 7f053631b700 1 --2- 172.21.15.46:0/456429076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05300af5c0 0x7f0530153970 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:20.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.883+0000 7f053631b700 1 -- 172.21.15.46:0/456429076 >> 172.21.15.46:0/456429076 conn(0x7f05300cdf80 msgr2=0x7f0530068080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:20.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.883+0000 7f053631b700 1 -- 172.21.15.46:0/456429076 shutdown_connections 2024-01-26T23:50:20.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:20.883+0000 7f053631b700 1 -- 172.21.15.46:0/456429076 wait complete. 2024-01-26T23:50:20.900 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2024-01-26T23:50:20.900 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add /snap_test_dir1 55n 2024-01-26T23:50:21.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.042+0000 7faff77c4700 1 Processor -- start 2024-01-26T23:50:21.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.043+0000 7faff77c4700 1 -- start start 2024-01-26T23:50:21.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.044+0000 7faff77c4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faff01566d0 0x7faff00c9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:21.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.044+0000 7faff77c4700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7faff00c9c90 0x7faff00ae220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.044+0000 7faff77c4700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff00ad180 con 0x7faff00ae760 2024-01-26T23:50:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.044+0000 7faff77c4700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff00aebc0 con 0x7faff01566d0 2024-01-26T23:50:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.044+0000 7faff77c4700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff00ad460 con 0x7faff00c9c90 2024-01-26T23:50:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.044+0000 7faff5560700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faff01566d0 0x7faff00c9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.044+0000 7faff5560700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faff01566d0 0x7faff00c9750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42666/0 (socket says 172.21.15.46:42666) 2024-01-26T23:50:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.044+0000 7faff5560700 1 -- 172.21.15.46:0/1009739864 learned_addr learned my addr 172.21.15.46:0/1009739864 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.044+0000 7faff4d5f700 1 --2- 172.21.15.46:0/1009739864 >> v2:172.21.15.106:3300/0 conn(0x7faff00c9c90 0x7faff00ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.044+0000 7faff5560700 1 -- 172.21.15.46:0/1009739864 >> v2:172.21.15.106:3300/0 conn(0x7faff00c9c90 msgr2=0x7faff00ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.044+0000 7faff5560700 1 --2- 172.21.15.46:0/1009739864 >> v2:172.21.15.106:3300/0 conn(0x7faff00c9c90 0x7faff00ae220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.044+0000 7faff5560700 1 -- 172.21.15.46:0/1009739864 >> v1:172.21.15.106:6789/0 conn(0x7faff00ae760 legacy=0x7faff00b0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.044+0000 7faff5560700 1 -- 172.21.15.46:0/1009739864 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff00ad740 con 0x7faff01566d0 2024-01-26T23:50:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.044+0000 7faff5560700 1 --2- 172.21.15.46:0/1009739864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff01566d0 0x7faff00c9750 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7faff0064a90 tx=0x7fafe4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d83c50a15ff009 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.045+0000 7fafe3fff700 1 -- 172.21.15.46:0/1009739864 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe4009e70 con 0x7faff01566d0 2024-01-26T23:50:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.045+0000 7fafe3fff700 1 -- 172.21.15.46:0/1009739864 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe400b860 con 0x7faff01566d0 2024-01-26T23:50:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.045+0000 7fafe3fff700 1 -- 172.21.15.46:0/1009739864 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe400bc20 con 0x7faff01566d0 2024-01-26T23:50:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.045+0000 7faff77c4700 1 -- 172.21.15.46:0/1009739864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff01566d0 msgr2=0x7faff00c9750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.045+0000 7faff77c4700 1 --2- 172.21.15.46:0/1009739864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff01566d0 0x7faff00c9750 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7faff0064a90 tx=0x7fafe4009250 comp rx=0 tx=0).stop 2024-01-26T23:50:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.045+0000 7faff77c4700 1 -- 172.21.15.46:0/1009739864 shutdown_connections 2024-01-26T23:50:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.045+0000 7faff77c4700 1 --2- 172.21.15.46:0/1009739864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff01566d0 0x7faff00c9750 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.045+0000 7faff77c4700 1 --2- 172.21.15.46:0/1009739864 >> v2:172.21.15.106:3300/0 conn(0x7faff00c9c90 0x7faff00ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.045+0000 7faff77c4700 1 -- 172.21.15.46:0/1009739864 >> 172.21.15.46:0/1009739864 conn(0x7faff0068400 msgr2=0x7faff0069c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.045+0000 7faff77c4700 1 -- 172.21.15.46:0/1009739864 shutdown_connections 2024-01-26T23:50:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.045+0000 7faff77c4700 1 -- 172.21.15.46:0/1009739864 wait complete. 2024-01-26T23:50:21.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.046+0000 7faff77c4700 1 Processor -- start 2024-01-26T23:50:21.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.046+0000 7faff77c4700 1 -- start start 2024-01-26T23:50:21.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.047+0000 7faff77c4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff00ae760 0x7faff015b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:21.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.047+0000 7faff77c4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faff00c9c90 0x7faff00c12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:21.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.047+0000 7faff77c4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faff01566d0 0x7faff00c1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:21.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.047+0000 7faff77c4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff0155d70 con 0x7faff00ae760 2024-01-26T23:50:21.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.047+0000 7faff77c4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff015bea0 con 0x7faff00c9c90 2024-01-26T23:50:21.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.047+0000 7faff77c4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff015c180 con 0x7faff01566d0 2024-01-26T23:50:21.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.048+0000 7faff5560700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff00ae760 0x7faff015b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:21.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.048+0000 7faff5560700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff00ae760 0x7faff015b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42672/0 (socket says 172.21.15.46:42672) 2024-01-26T23:50:21.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.048+0000 7faff5560700 1 -- 172.21.15.46:0/3205897651 learned_addr learned my addr 172.21.15.46:0/3205897651 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.048+0000 7faff4d5f700 1 --2- 172.21.15.46:0/3205897651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faff00c9c90 0x7faff00c12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.048+0000 7faff5d61700 1 --2- 172.21.15.46:0/3205897651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faff01566d0 0x7faff00c1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.048+0000 7faff5560700 1 -- 172.21.15.46:0/3205897651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faff01566d0 msgr2=0x7faff00c1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.048+0000 7faff5560700 1 --2- 172.21.15.46:0/3205897651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faff01566d0 0x7faff00c1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.048+0000 7faff5560700 1 -- 172.21.15.46:0/3205897651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faff00c9c90 msgr2=0x7faff00c12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.048+0000 7faff5560700 1 --2- 172.21.15.46:0/3205897651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faff00c9c90 0x7faff00c12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.048+0000 7faff5560700 1 -- 172.21.15.46:0/3205897651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe40094f0 con 0x7faff00ae760 2024-01-26T23:50:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.048+0000 7faff5560700 1 --2- 172.21.15.46:0/3205897651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff00ae760 0x7faff015b8c0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7faff0070310 tx=0x7fafe4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.048+0000 7fafe17fa700 1 -- 172.21.15.46:0/3205897651 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe401f070 con 0x7faff00ae760 2024-01-26T23:50:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.048+0000 7fafe17fa700 1 -- 172.21.15.46:0/3205897651 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe4002f30 con 0x7faff00ae760 2024-01-26T23:50:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.048+0000 7faff77c4700 1 -- 172.21.15.46:0/3205897651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff015c460 con 0x7faff00ae760 2024-01-26T23:50:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.048+0000 7fafe17fa700 1 -- 172.21.15.46:0/3205897651 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe400b9d0 con 0x7faff00ae760 2024-01-26T23:50:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.048+0000 7faff77c4700 1 -- 172.21.15.46:0/3205897651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff00a97d0 con 0x7faff00ae760 2024-01-26T23:50:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.049+0000 7faff77c4700 1 -- 172.21.15.46:0/3205897651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff004c710 con 0x7faff00ae760 2024-01-26T23:50:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.050+0000 7fafe17fa700 1 -- 172.21.15.46:0/3205897651 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 79) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fafe4023070 con 0x7faff00ae760 2024-01-26T23:50:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.050+0000 7fafe17fa700 1 --2- 172.21.15.46:0/3205897651 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fafd4066dd0 0x7fafd4069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.050+0000 7fafe17fa700 1 -- 172.21.15.46:0/3205897651 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fafe4085780 con 0x7faff00ae760 2024-01-26T23:50:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.050+0000 7faff4d5f700 1 --2- 172.21.15.46:0/3205897651 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fafd4066dd0 0x7fafd4069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.051+0000 7faff4d5f700 1 --2- 172.21.15.46:0/3205897651 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fafd4066dd0 0x7fafd4069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fafec009d10 tx=0x7fafec0091a0 comp rx=0 tx=0).ready entity=mgr.8743 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:21.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.055+0000 7fafe17fa700 1 -- 172.21.15.46:0/3205897651 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fafe4052910 con 0x7faff00ae760 2024-01-26T23:50:21.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.294+0000 7faff77c4700 1 -- 172.21.15.46:0/3205897651 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) v1 -- 0x7faff0068800 con 0x7fafd4066dd0 2024-01-26T23:50:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.347+0000 7fafe17fa700 1 -- 172.21.15.46:0/3205897651 <== mgr.8743 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+40 (secure 0 0 0) 0x7faff0068800 con 0x7fafd4066dd0 2024-01-26T23:50:21.349 INFO:teuthology.orchestra.run.smithi046.stdout:Retention added to path /snap_test_dir1 2024-01-26T23:50:21.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.350+0000 7faff77c4700 1 -- 172.21.15.46:0/3205897651 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fafd4066dd0 msgr2=0x7fafd4069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:21.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.350+0000 7faff77c4700 1 --2- 172.21.15.46:0/3205897651 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fafd4066dd0 0x7fafd4069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fafec009d10 tx=0x7fafec0091a0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.350+0000 7faff77c4700 1 -- 172.21.15.46:0/3205897651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff00ae760 msgr2=0x7faff015b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:21.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.350+0000 7faff77c4700 1 --2- 172.21.15.46:0/3205897651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff00ae760 0x7faff015b8c0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7faff0070310 tx=0x7fafe4017000 comp rx=0 tx=0).stop 2024-01-26T23:50:21.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.350+0000 7faff77c4700 1 -- 172.21.15.46:0/3205897651 shutdown_connections 2024-01-26T23:50:21.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.350+0000 7faff77c4700 1 --2- 172.21.15.46:0/3205897651 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fafd4066dd0 0x7fafd4069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.350+0000 7faff77c4700 1 --2- 172.21.15.46:0/3205897651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff00ae760 0x7faff015b8c0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.350+0000 7faff77c4700 1 --2- 172.21.15.46:0/3205897651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faff01566d0 0x7faff00c1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.350+0000 7faff77c4700 1 --2- 172.21.15.46:0/3205897651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faff00c9c90 0x7faff00c12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.350+0000 7faff77c4700 1 -- 172.21.15.46:0/3205897651 >> 172.21.15.46:0/3205897651 conn(0x7faff0068400 msgr2=0x7faff006ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:21.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.350+0000 7faff77c4700 1 -- 172.21.15.46:0/3205897651 shutdown_connections 2024-01-26T23:50:21.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.350+0000 7faff77c4700 1 -- 172.21.15.46:0/3205897651 wait complete. 2024-01-26T23:50:21.366 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2024-01-26T23:50:21.366 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-01-26T23:50:21.366 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json /snap_test_dir1 2024-01-26T23:50:21.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.500+0000 7f36c291d700 1 Processor -- start 2024-01-26T23:50:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.500+0000 7f36c291d700 1 -- start start 2024-01-26T23:50:21.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.501+0000 7f36c291d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc156570 0x7f36bc06c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.501+0000 7f36c291d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f36bc06f9a0 0x7f36bc06eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.501+0000 7f36c291d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc0c8b90 con 0x7f36bc156940 2024-01-26T23:50:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.501+0000 7f36c291d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36bc06d0b0 con 0x7f36bc06f9a0 2024-01-26T23:50:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.501+0000 7f36c291d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc0c8e50 con 0x7f36bc156570 2024-01-26T23:50:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.501+0000 7f36c0eba700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f36bc06f9a0 0x7f36bc06eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.501+0000 7f36bbfff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc156570 0x7f36bc06c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.502+0000 7f36c0eba700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f36bc06f9a0 0x7f36bc06eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:53608/0 (socket says 172.21.15.46:53608) 2024-01-26T23:50:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.502+0000 7f36bb7fe700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f36bc156940 0x7f36bc0c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:33610/0 (socket says 172.21.15.46:33610) 2024-01-26T23:50:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.502+0000 7f36c0eba700 1 -- 172.21.15.46:0/2499933579 learned_addr learned my addr 172.21.15.46:0/2499933579 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.502+0000 7f36bbfff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc156570 0x7f36bc06c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:44210/0 (socket says 172.21.15.46:44210) 2024-01-26T23:50:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.502+0000 7f36c0eba700 1 -- 172.21.15.46:0/2499933579 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc156570 msgr2=0x7f36bc06c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-26T23:50:21.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.502+0000 7f36c0eba700 1 --2- 172.21.15.46:0/2499933579 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc156570 0x7f36bc06c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.502+0000 7f36c0eba700 1 -- 172.21.15.46:0/2499933579 >> v1:172.21.15.106:6789/0 conn(0x7f36bc156940 legacy=0x7f36bc0c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:21.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.502+0000 7f36c0eba700 1 -- 172.21.15.46:0/2499933579 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36bc0c4c40 con 0x7f36bc06f9a0 2024-01-26T23:50:21.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.502+0000 7f36bbfff700 1 --2- 172.21.15.46:0/2499933579 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc156570 0x7f36bc06c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-26T23:50:21.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.502+0000 7f36c0eba700 1 --2- 172.21.15.46:0/2499933579 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36bc06f9a0 0x7f36bc06eb90 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f36b400d1d0 tx=0x7f36b400ba50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ce505dfcc361483 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:21.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.502+0000 7f36baffd700 1 -- 172.21.15.46:0/2499933579 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b4010070 con 0x7f36bc06f9a0 2024-01-26T23:50:21.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.502+0000 7f36baffd700 1 -- 172.21.15.46:0/2499933579 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b400de00 con 0x7f36bc06f9a0 2024-01-26T23:50:21.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.502+0000 7f36baffd700 1 -- 172.21.15.46:0/2499933579 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b400b320 con 0x7f36bc06f9a0 2024-01-26T23:50:21.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.503+0000 7f36c291d700 1 -- 172.21.15.46:0/2499933579 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36bc06f9a0 msgr2=0x7f36bc06eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:21.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.503+0000 7f36c291d700 1 --2- 172.21.15.46:0/2499933579 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36bc06f9a0 0x7f36bc06eb90 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f36b400d1d0 tx=0x7f36b400ba50 comp rx=0 tx=0).stop 2024-01-26T23:50:21.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.503+0000 7f36c291d700 1 -- 172.21.15.46:0/2499933579 shutdown_connections 2024-01-26T23:50:21.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.503+0000 7f36c291d700 1 --2- 172.21.15.46:0/2499933579 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36bc06f9a0 0x7f36bc06eb90 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.503+0000 7f36c291d700 1 --2- 172.21.15.46:0/2499933579 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc156570 0x7f36bc06c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.503+0000 7f36c291d700 1 -- 172.21.15.46:0/2499933579 >> 172.21.15.46:0/2499933579 conn(0x7f36bc0cdf50 msgr2=0x7f36bc0c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:21.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.503+0000 7f36c291d700 1 -- 172.21.15.46:0/2499933579 shutdown_connections 2024-01-26T23:50:21.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.503+0000 7f36c291d700 1 -- 172.21.15.46:0/2499933579 wait complete. 2024-01-26T23:50:21.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.504+0000 7f36c291d700 1 Processor -- start 2024-01-26T23:50:21.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.504+0000 7f36c291d700 1 -- start start 2024-01-26T23:50:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.505+0000 7f36c291d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc06f9a0 0x7f36bc14e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.505+0000 7f36c291d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36bc156570 0x7f36bc1595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.505+0000 7f36c291d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36bc156940 0x7f36bc0aa200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.505+0000 7f36c291d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc154ea0 con 0x7f36bc156940 2024-01-26T23:50:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.505+0000 7f36c291d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc154290 con 0x7f36bc156570 2024-01-26T23:50:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.505+0000 7f36c291d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc154570 con 0x7f36bc06f9a0 2024-01-26T23:50:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.505+0000 7f36c0eba700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36bc156940 0x7f36bc0aa200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.505+0000 7f36c0eba700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36bc156940 0x7f36bc0aa200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42686/0 (socket says 172.21.15.46:42686) 2024-01-26T23:50:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.505+0000 7f36c0eba700 1 -- 172.21.15.46:0/3015187879 learned_addr learned my addr 172.21.15.46:0/3015187879 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-26T23:50:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.505+0000 7f36bbfff700 1 --2- 172.21.15.46:0/3015187879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc06f9a0 0x7f36bc14e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:21.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.506+0000 7f36bb7fe700 1 --2- 172.21.15.46:0/3015187879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36bc156570 0x7f36bc1595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:21.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.506+0000 7f36c0eba700 1 -- 172.21.15.46:0/3015187879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc06f9a0 msgr2=0x7f36bc14e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:21.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.506+0000 7f36c0eba700 1 --2- 172.21.15.46:0/3015187879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc06f9a0 0x7f36bc14e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.506+0000 7f36c0eba700 1 -- 172.21.15.46:0/3015187879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36bc156570 msgr2=0x7f36bc1595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:21.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.506+0000 7f36c0eba700 1 --2- 172.21.15.46:0/3015187879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36bc156570 0x7f36bc1595b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.506+0000 7f36c0eba700 1 -- 172.21.15.46:0/3015187879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ac009420 con 0x7f36bc156940 2024-01-26T23:50:21.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.506+0000 7f36c0eba700 1 --2- 172.21.15.46:0/3015187879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36bc156940 0x7f36bc0aa200 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f36bc06e800 tx=0x7f36b401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:21.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.506+0000 7f36b3fff700 1 -- 172.21.15.46:0/3015187879 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b4009b00 con 0x7f36bc156940 2024-01-26T23:50:21.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.506+0000 7f36b3fff700 1 -- 172.21.15.46:0/3015187879 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b4009ca0 con 0x7f36bc156940 2024-01-26T23:50:21.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.506+0000 7f36c291d700 1 -- 172.21.15.46:0/3015187879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b400bcf0 con 0x7f36bc156940 2024-01-26T23:50:21.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.506+0000 7f36b3fff700 1 -- 172.21.15.46:0/3015187879 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b401e690 con 0x7f36bc156940 2024-01-26T23:50:21.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.506+0000 7f36c291d700 1 -- 172.21.15.46:0/3015187879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36bc154b30 con 0x7f36bc156940 2024-01-26T23:50:21.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.507+0000 7f36c291d700 1 -- 172.21.15.46:0/3015187879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36bc155010 con 0x7f36bc156940 2024-01-26T23:50:21.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.508+0000 7f36b3fff700 1 -- 172.21.15.46:0/3015187879 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 79) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f36b4003040 con 0x7f36bc156940 2024-01-26T23:50:21.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.508+0000 7f36b3fff700 1 --2- 172.21.15.46:0/3015187879 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f369c066dd0 0x7f369c069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-26T23:50:21.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.508+0000 7f36b3fff700 1 -- 172.21.15.46:0/3015187879 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f36b4085de0 con 0x7f36bc156940 2024-01-26T23:50:21.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.508+0000 7f36bbfff700 1 --2- 172.21.15.46:0/3015187879 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f369c066dd0 0x7f369c069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-26T23:50:21.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.509+0000 7f36bbfff700 1 --2- 172.21.15.46:0/3015187879 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f369c066dd0 0x7f369c069280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f36ac009f40 tx=0x7f36ac0091a0 comp rx=0 tx=0).ready entity=mgr.8743 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-26T23:50:21.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.513+0000 7f36b3fff700 1 -- 172.21.15.46:0/3015187879 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f36b4052f30 con 0x7f36bc156940 2024-01-26T23:50:21.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.748+0000 7f36c291d700 1 -- 172.21.15.46:0/3015187879 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f36bc04c710 con 0x7f369c066dd0 2024-01-26T23:50:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.760+0000 7f36b3fff700 1 -- 172.21.15.46:0/3015187879 <== mgr.8743 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+74 (secure 0 0 0) 0x7f36bc04c710 con 0x7f369c066dd0 2024-01-26T23:50:21.761 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-26T23:50:21.762 INFO:teuthology.orchestra.run.smithi046.stdout:{"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-01-26T23:50:21.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.763+0000 7f36c291d700 1 -- 172.21.15.46:0/3015187879 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f369c066dd0 msgr2=0x7f369c069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:21.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.763+0000 7f36c291d700 1 --2- 172.21.15.46:0/3015187879 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f369c066dd0 0x7f369c069280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f36ac009f40 tx=0x7f36ac0091a0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.763+0000 7f36c291d700 1 -- 172.21.15.46:0/3015187879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36bc156940 msgr2=0x7f36bc0aa200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-26T23:50:21.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.763+0000 7f36c291d700 1 --2- 172.21.15.46:0/3015187879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36bc156940 0x7f36bc0aa200 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f36bc06e800 tx=0x7f36b401a000 comp rx=0 tx=0).stop 2024-01-26T23:50:21.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.763+0000 7f36c291d700 1 -- 172.21.15.46:0/3015187879 shutdown_connections 2024-01-26T23:50:21.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.763+0000 7f36c291d700 1 --2- 172.21.15.46:0/3015187879 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f369c066dd0 0x7f369c069280 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.763+0000 7f36c291d700 1 --2- 172.21.15.46:0/3015187879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36bc156940 0x7f36bc0aa200 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.763+0000 7f36c291d700 1 --2- 172.21.15.46:0/3015187879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36bc06f9a0 0x7f36bc14e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.763+0000 7f36c291d700 1 --2- 172.21.15.46:0/3015187879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36bc156570 0x7f36bc1595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-26T23:50:21.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.763+0000 7f36c291d700 1 -- 172.21.15.46:0/3015187879 >> 172.21.15.46:0/3015187879 conn(0x7f36bc0cdf50 msgr2=0x7f36bc0c2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-26T23:50:21.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.763+0000 7f36c291d700 1 -- 172.21.15.46:0/3015187879 shutdown_connections 2024-01-26T23:50:21.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-26T23:50:21.763+0000 7f36c291d700 1 -- 172.21.15.46:0/3015187879 wait complete. 2024-01-26T23:50:21.778 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-01-26T23:50:21.778 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1m'], 'retention': [{'n': 55}]} 2024-01-26T23:50:25.896 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:50:25.899 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:50:56.026 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:50:56.030 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:51:00.358 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:51:00.357+0000 7fb12410b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:51:00.358 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:51:00.357+0000 7fb12410b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:51:00.358 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:51:00.357+0000 7fb12410b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:51:00.358 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:51:00.357+0000 7fb12410b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:51:00.359 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-26T23:51:00.357+0000 7fb12410b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-26T23:51:26.133 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:51:26.136 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:51:56.254 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:51:56.258 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:52:26.369 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:52:26.373 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:52:56.500 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:52:56.504 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:53:26.589 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:53:26.592 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:53:56.728 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:53:56.732 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:54:26.876 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:54:26.879 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:54:57.024 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:54:57.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:55:27.146 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:55:27.150 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:55:57.225 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:55:57.229 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:56:27.374 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:56:27.377 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:56:57.497 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:56:57.501 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:57:27.577 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:57:27.580 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:57:57.733 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:57:57.737 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:58:27.872 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:58:27.876 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:58:58.030 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:58:58.034 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:59:28.109 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:59:28.113 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:59:58.248 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-26T23:59:58.251 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:00:28.388 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:00:28.393 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:00:58.462 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:00:58.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:01:28.601 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:01:28.605 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:01:58.723 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:01:58.727 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:02:28.863 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:02:28.867 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:02:58.986 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:02:58.990 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:03:29.150 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:03:29.153 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:03:59.281 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:03:59.284 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:04:29.413 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:04:29.416 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:04:59.566 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:04:59.570 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:05:29.631 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:05:29.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:05:59.787 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:05:59.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:06:29.885 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:06:29.889 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:06:59.969 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:06:59.973 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:07:30.079 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:07:30.083 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:08:00.186 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:08:00.189 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:08:30.341 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:08:30.345 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:09:00.422 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:09:00.425 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:09:30.594 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:09:30.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:10:00.708 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:10:00.712 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:10:30.857 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:10:30.861 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:11:00.988 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:11:00.991 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:11:31.050 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:11:31.054 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:12:01.183 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:12:01.187 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:12:31.272 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:12:31.276 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:13:01.411 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:13:01.414 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:13:31.534 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:13:31.538 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:14:01.690 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:14:01.694 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:14:31.780 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:14:31.783 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:15:01.844 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:15:01.847 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:15:31.958 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:15:31.962 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:16:02.022 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:16:02.026 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:16:32.103 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:16:32.106 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:17:02.177 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:17:02.180 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:17:32.274 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:17:32.278 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:18:02.414 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:18:02.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:18:32.495 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:18:32.499 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:19:02.626 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:19:02.629 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:19:32.707 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:19:32.711 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:20:02.780 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:20:02.784 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:20:32.952 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:20:32.956 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:21:03.159 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:21:03.162 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:21:33.216 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:21:33.219 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:22:03.337 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:22:03.344 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:22:33.486 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:22:33.490 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:23:03.783 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:23:03.786 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:23:33.854 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:23:33.858 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:24:03.978 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:24:03.981 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:24:34.117 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:24:34.120 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:25:04.239 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:25:04.243 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:25:34.307 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:25:34.311 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:26:04.452 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:26:04.456 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:26:34.616 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:26:34.619 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:27:04.679 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:27:04.683 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:27:34.818 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:27:34.822 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:28:04.916 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:28:04.919 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:28:35.038 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:28:35.042 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:29:05.142 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:29:05.145 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:29:35.267 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:29:35.272 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:30:05.406 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:30:05.409 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:30:35.554 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:30:35.558 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:31:05.677 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:31:05.680 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:31:35.841 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:31:35.845 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:32:05.937 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:32:05.940 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:32:36.001 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:32:36.004 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:33:06.159 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:33:06.163 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:33:36.223 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:33:36.227 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:34:06.363 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:34:06.366 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:34:36.442 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:34:36.445 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:35:06.608 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:35:06.612 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:35:36.782 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:35:36.785 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:36:06.861 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:36:06.867 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:36:36.993 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:36:36.996 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:37:07.115 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:37:07.119 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:37:37.213 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:37:37.216 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:38:07.368 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:38:07.371 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:38:37.507 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:38:37.510 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:39:07.646 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:39:07.650 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:39:37.752 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:39:37.755 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:40:07.892 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:40:07.895 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:40:37.958 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:40:37.962 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:41:08.087 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:41:08.090 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:41:38.226 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:41:38.229 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:42:08.324 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:42:08.327 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:42:38.438 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:42:38.441 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:43:08.578 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:43:08.581 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:43:38.725 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:43:38.729 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:44:08.806 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:44:08.811 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:44:39.162 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:44:39.168 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:45:09.243 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:45:09.246 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:45:39.382 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:45:39.386 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:46:09.514 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:46:09.519 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:46:15.780 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:15.780 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T00:46:15.810 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_52_00_UTC 2024-01-27T00:46:15.810 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_53_00_UTC 2024-01-27T00:46:15.810 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_54_00_UTC 2024-01-27T00:46:15.810 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_55_00_UTC 2024-01-27T00:46:15.811 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_56_00_UTC 2024-01-27T00:46:15.811 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_57_00_UTC 2024-01-27T00:46:15.811 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_58_00_UTC 2024-01-27T00:46:15.811 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_59_00_UTC 2024-01-27T00:46:15.811 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_00_00_UTC 2024-01-27T00:46:15.811 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_01_00_UTC 2024-01-27T00:46:15.811 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_02_00_UTC 2024-01-27T00:46:15.811 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_03_00_UTC 2024-01-27T00:46:15.811 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_04_00_UTC 2024-01-27T00:46:15.811 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_05_00_UTC 2024-01-27T00:46:15.811 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_06_00_UTC 2024-01-27T00:46:15.811 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_07_00_UTC 2024-01-27T00:46:15.811 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_08_00_UTC 2024-01-27T00:46:15.811 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_09_00_UTC 2024-01-27T00:46:15.812 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_10_00_UTC 2024-01-27T00:46:15.812 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_11_00_UTC 2024-01-27T00:46:15.812 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_12_00_UTC 2024-01-27T00:46:15.812 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_13_00_UTC 2024-01-27T00:46:15.812 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_14_00_UTC 2024-01-27T00:46:15.812 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_15_00_UTC 2024-01-27T00:46:15.812 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_16_00_UTC 2024-01-27T00:46:15.812 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_17_00_UTC 2024-01-27T00:46:15.812 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_18_00_UTC 2024-01-27T00:46:15.812 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_19_00_UTC 2024-01-27T00:46:15.812 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_20_00_UTC 2024-01-27T00:46:15.812 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_21_00_UTC 2024-01-27T00:46:15.812 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_22_00_UTC 2024-01-27T00:46:15.813 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_23_00_UTC 2024-01-27T00:46:15.813 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_24_00_UTC 2024-01-27T00:46:15.813 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_25_00_UTC 2024-01-27T00:46:15.813 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_26_00_UTC 2024-01-27T00:46:15.813 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_27_00_UTC 2024-01-27T00:46:15.813 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_28_00_UTC 2024-01-27T00:46:15.813 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_29_00_UTC 2024-01-27T00:46:15.813 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_30_00_UTC 2024-01-27T00:46:15.813 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_31_00_UTC 2024-01-27T00:46:15.813 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_32_00_UTC 2024-01-27T00:46:15.813 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_33_00_UTC 2024-01-27T00:46:15.813 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_34_00_UTC 2024-01-27T00:46:15.813 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_35_00_UTC 2024-01-27T00:46:15.813 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_36_00_UTC 2024-01-27T00:46:15.813 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_37_00_UTC 2024-01-27T00:46:15.814 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_38_00_UTC 2024-01-27T00:46:15.814 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_39_00_UTC 2024-01-27T00:46:15.814 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_40_00_UTC 2024-01-27T00:46:15.814 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_41_00_UTC 2024-01-27T00:46:15.814 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_42_00_UTC 2024-01-27T00:46:15.814 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_43_00_UTC 2024-01-27T00:46:15.814 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_44_00_UTC 2024-01-27T00:46:15.814 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_45_00_UTC 2024-01-27T00:46:15.814 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_46_00_UTC 2024-01-27T00:46:15.815 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-26-23_52_00_UTC', 'scheduled-2024-01-26-23_53_00_UTC', 'scheduled-2024-01-26-23_54_00_UTC', 'scheduled-2024-01-26-23_55_00_UTC', 'scheduled-2024-01-26-23_56_00_UTC', 'scheduled-2024-01-26-23_57_00_UTC', 'scheduled-2024-01-26-23_58_00_UTC', 'scheduled-2024-01-26-23_59_00_UTC', 'scheduled-2024-01-27-00_00_00_UTC', 'scheduled-2024-01-27-00_01_00_UTC', 'scheduled-2024-01-27-00_02_00_UTC', 'scheduled-2024-01-27-00_03_00_UTC', 'scheduled-2024-01-27-00_04_00_UTC', 'scheduled-2024-01-27-00_05_00_UTC', 'scheduled-2024-01-27-00_06_00_UTC', 'scheduled-2024-01-27-00_07_00_UTC', 'scheduled-2024-01-27-00_08_00_UTC', 'scheduled-2024-01-27-00_09_00_UTC', 'scheduled-2024-01-27-00_10_00_UTC', 'scheduled-2024-01-27-00_11_00_UTC', 'scheduled-2024-01-27-00_12_00_UTC', 'scheduled-2024-01-27-00_13_00_UTC', 'scheduled-2024-01-27-00_14_00_UTC', 'scheduled-2024-01-27-00_15_00_UTC', 'scheduled-2024-01-27-00_16_00_UTC', 'scheduled-2024-01-27-00_17_00_UTC', 'scheduled-2024-01-27-00_18_00_UTC', 'scheduled-2024-01-27-00_19_00_UTC', 'scheduled-2024-01-27-00_20_00_UTC', 'scheduled-2024-01-27-00_21_00_UTC', 'scheduled-2024-01-27-00_22_00_UTC', 'scheduled-2024-01-27-00_23_00_UTC', 'scheduled-2024-01-27-00_24_00_UTC', 'scheduled-2024-01-27-00_25_00_UTC', 'scheduled-2024-01-27-00_26_00_UTC', 'scheduled-2024-01-27-00_27_00_UTC', 'scheduled-2024-01-27-00_28_00_UTC', 'scheduled-2024-01-27-00_29_00_UTC', 'scheduled-2024-01-27-00_30_00_UTC', 'scheduled-2024-01-27-00_31_00_UTC', 'scheduled-2024-01-27-00_32_00_UTC', 'scheduled-2024-01-27-00_33_00_UTC', 'scheduled-2024-01-27-00_34_00_UTC', 'scheduled-2024-01-27-00_35_00_UTC', 'scheduled-2024-01-27-00_36_00_UTC', 'scheduled-2024-01-27-00_37_00_UTC', 'scheduled-2024-01-27-00_38_00_UTC', 'scheduled-2024-01-27-00_39_00_UTC', 'scheduled-2024-01-27-00_40_00_UTC', 'scheduled-2024-01-27-00_41_00_UTC', 'scheduled-2024-01-27-00_42_00_UTC', 'scheduled-2024-01-27-00_43_00_UTC', 'scheduled-2024-01-27-00_44_00_UTC', 'scheduled-2024-01-27-00_45_00_UTC', 'scheduled-2024-01-27-00_46_00_UTC'] 2024-01-27T00:46:15.815 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1 2024-01-27T00:46:15.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:15.998+0000 7f36f3ec7700 1 Processor -- start 2024-01-27T00:46:15.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:15.998+0000 7f36f3ec7700 1 -- start start 2024-01-27T00:46:15.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:15.999+0000 7f36f3ec7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36ec15dfd0 0x7f36ec0ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:15.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:15.999+0000 7f36f3ec7700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ec15a200 con 0x7f36ec1599d0 2024-01-27T00:46:15.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:15.999+0000 7f36f3ec7700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ec0b1a60 con 0x7f36ec159da0 2024-01-27T00:46:15.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:15.999+0000 7f36f3ec7700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36ec0b11c0 con 0x7f36ec15dfd0 2024-01-27T00:46:15.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:15.999+0000 7f36f2464700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36ec15dfd0 0x7f36ec0ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:15.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:15.999+0000 7f36f1c63700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f36ec1599d0 0x7f36ec0b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37422/0 (socket says 172.21.15.46:37422) 2024-01-27T00:46:15.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:15.999+0000 7f36f1c63700 1 -- 172.21.15.46:0/321830828 learned_addr learned my addr 172.21.15.46:0/321830828 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:15.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:15.999+0000 7f36f2464700 1 -- 172.21.15.46:0/321830828 >> v1:172.21.15.106:6789/0 conn(0x7f36ec159da0 legacy=0x7f36ec15d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:16.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:15.999+0000 7f36f2464700 1 -- 172.21.15.46:0/321830828 >> v1:172.21.15.46:6789/0 conn(0x7f36ec1599d0 legacy=0x7f36ec0b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:16.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:15.999+0000 7f36f2464700 1 -- 172.21.15.46:0/321830828 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ec0b14a0 con 0x7f36ec15dfd0 2024-01-27T00:46:16.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:15.999+0000 7f36f0c61700 1 -- 172.21.15.46:0/321830828 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256249761 0 0) 0x7f36ec15a200 con 0x7f36ec1599d0 2024-01-27T00:46:16.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.000+0000 7f36f2464700 1 --2- 172.21.15.46:0/321830828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36ec15dfd0 0x7f36ec0ab1c0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f36e4009e60 tx=0x7f36e4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f236410246f6dccf server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:16.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.000+0000 7f36f0c61700 1 -- 172.21.15.46:0/321830828 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e400e070 con 0x7f36ec15dfd0 2024-01-27T00:46:16.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.000+0000 7f36f0c61700 1 -- 172.21.15.46:0/321830828 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e400bc30 con 0x7f36ec15dfd0 2024-01-27T00:46:16.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.000+0000 7f36f0c61700 1 -- 172.21.15.46:0/321830828 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e4012430 con 0x7f36ec15dfd0 2024-01-27T00:46:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.000+0000 7f36f3ec7700 1 -- 172.21.15.46:0/321830828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36ec15dfd0 msgr2=0x7f36ec0ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.000+0000 7f36f3ec7700 1 --2- 172.21.15.46:0/321830828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36ec15dfd0 0x7f36ec0ab1c0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f36e4009e60 tx=0x7f36e4009780 comp rx=0 tx=0).stop 2024-01-27T00:46:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.000+0000 7f36f3ec7700 1 -- 172.21.15.46:0/321830828 shutdown_connections 2024-01-27T00:46:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.000+0000 7f36f3ec7700 1 --2- 172.21.15.46:0/321830828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36ec15dfd0 0x7f36ec0ab1c0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.000+0000 7f36f3ec7700 1 -- 172.21.15.46:0/321830828 >> 172.21.15.46:0/321830828 conn(0x7f36ec06f840 msgr2=0x7f36ec087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.001+0000 7f36f3ec7700 1 -- 172.21.15.46:0/321830828 shutdown_connections 2024-01-27T00:46:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.001+0000 7f36f3ec7700 1 -- 172.21.15.46:0/321830828 wait complete. 2024-01-27T00:46:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.002+0000 7f36f3ec7700 1 Processor -- start 2024-01-27T00:46:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.002+0000 7f36f3ec7700 1 -- start start 2024-01-27T00:46:16.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.003+0000 7f36f3ec7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36ec1599d0 0x7f36ec0ab980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:16.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.003+0000 7f36f3ec7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36ec159da0 0x7f36ec157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:16.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.003+0000 7f36f3ec7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36ec15dfd0 0x7f36ec158530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:16.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.003+0000 7f36f3ec7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec09a310 con 0x7f36ec15dfd0 2024-01-27T00:46:16.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.003+0000 7f36f3ec7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec09ae50 con 0x7f36ec1599d0 2024-01-27T00:46:16.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.003+0000 7f36f3ec7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec09a520 con 0x7f36ec159da0 2024-01-27T00:46:16.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.003+0000 7f36f2464700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36ec15dfd0 0x7f36ec158530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:16.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.003+0000 7f36f2464700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36ec15dfd0 0x7f36ec158530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34748/0 (socket says 172.21.15.46:34748) 2024-01-27T00:46:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.003+0000 7f36f2464700 1 -- 172.21.15.46:0/4138589907 learned_addr learned my addr 172.21.15.46:0/4138589907 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.003+0000 7f36f1c63700 1 --2- 172.21.15.46:0/4138589907 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36ec1599d0 0x7f36ec0ab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.003+0000 7f36f1462700 1 --2- 172.21.15.46:0/4138589907 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36ec159da0 0x7f36ec157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.003+0000 7f36f2464700 1 -- 172.21.15.46:0/4138589907 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36ec159da0 msgr2=0x7f36ec157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.003+0000 7f36f2464700 1 --2- 172.21.15.46:0/4138589907 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36ec159da0 0x7f36ec157ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.003+0000 7f36f2464700 1 -- 172.21.15.46:0/4138589907 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36ec1599d0 msgr2=0x7f36ec0ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.003+0000 7f36f2464700 1 --2- 172.21.15.46:0/4138589907 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36ec1599d0 0x7f36ec0ab980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.003+0000 7f36f2464700 1 -- 172.21.15.46:0/4138589907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e4009a20 con 0x7f36ec15dfd0 2024-01-27T00:46:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.003+0000 7f36f2464700 1 --2- 172.21.15.46:0/4138589907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36ec15dfd0 0x7f36ec158530 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f36e4013000 tx=0x7f36e4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.004+0000 7f36e1ffb700 1 -- 172.21.15.46:0/4138589907 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e401b630 con 0x7f36ec15dfd0 2024-01-27T00:46:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.004+0000 7f36e1ffb700 1 -- 172.21.15.46:0/4138589907 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e401b790 con 0x7f36ec15dfd0 2024-01-27T00:46:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.004+0000 7f36f3ec7700 1 -- 172.21.15.46:0/4138589907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36ec09a800 con 0x7f36ec15dfd0 2024-01-27T00:46:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.004+0000 7f36e1ffb700 1 -- 172.21.15.46:0/4138589907 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e401bb30 con 0x7f36ec15dfd0 2024-01-27T00:46:16.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.004+0000 7f36f3ec7700 1 -- 172.21.15.46:0/4138589907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36ec0abec0 con 0x7f36ec15dfd0 2024-01-27T00:46:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.005+0000 7f36f3ec7700 1 -- 172.21.15.46:0/4138589907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36ec09afe0 con 0x7f36ec15dfd0 2024-01-27T00:46:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.005+0000 7f36e1ffb700 1 -- 172.21.15.46:0/4138589907 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 83) v1 ==== 84873+0+0 (secure 0 0 0) 0x7f36e401c400 con 0x7f36ec15dfd0 2024-01-27T00:46:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.006+0000 7f36e1ffb700 1 --2- 172.21.15.46:0/4138589907 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f36d00670a0 0x7f36d0069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:16.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.006+0000 7f36e1ffb700 1 -- 172.21.15.46:0/4138589907 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6763+0+0 (secure 0 0 0) 0x7f36e4086bb0 con 0x7f36ec15dfd0 2024-01-27T00:46:16.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.006+0000 7f36f1c63700 1 --2- 172.21.15.46:0/4138589907 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f36d00670a0 0x7f36d0069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:16.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.007+0000 7f36f1c63700 1 --2- 172.21.15.46:0/4138589907 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f36d00670a0 0x7f36d0069550 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f36e8001360 tx=0x7f36e8009040 comp rx=0 tx=0).ready entity=mgr.8743 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:16.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.010+0000 7f36e1ffb700 1 -- 172.21.15.46:0/4138589907 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f36e40537e0 con 0x7f36ec15dfd0 2024-01-27T00:46:16.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.244+0000 7f36f3ec7700 1 -- 172.21.15.46:0/4138589907 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f36ec09b1e0 con 0x7f36d00670a0 2024-01-27T00:46:16.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.253+0000 7f36e1ffb700 1 -- 172.21.15.46:0/4138589907 <== mgr.8743 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f36ec09b1e0 con 0x7f36d00670a0 2024-01-27T00:46:16.252 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:16.252 INFO:teuthology.orchestra.run.smithi046.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:50:20", "first": "2024-01-26T23:51:00", "last": "2024-01-27T00:46:00", "last_pruned": "2024-01-27T00:46:00", "created_count": 56, "pruned_count": 1, "active": true}] 2024-01-27T00:46:16.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.255+0000 7f36f3ec7700 1 -- 172.21.15.46:0/4138589907 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f36d00670a0 msgr2=0x7f36d0069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:16.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.255+0000 7f36f3ec7700 1 --2- 172.21.15.46:0/4138589907 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f36d00670a0 0x7f36d0069550 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f36e8001360 tx=0x7f36e8009040 comp rx=0 tx=0).stop 2024-01-27T00:46:16.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.255+0000 7f36f3ec7700 1 -- 172.21.15.46:0/4138589907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36ec15dfd0 msgr2=0x7f36ec158530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:16.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.255+0000 7f36f3ec7700 1 --2- 172.21.15.46:0/4138589907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36ec15dfd0 0x7f36ec158530 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f36e4013000 tx=0x7f36e4019000 comp rx=0 tx=0).stop 2024-01-27T00:46:16.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.255+0000 7f36f3ec7700 1 -- 172.21.15.46:0/4138589907 shutdown_connections 2024-01-27T00:46:16.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.255+0000 7f36f3ec7700 1 --2- 172.21.15.46:0/4138589907 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f36d00670a0 0x7f36d0069550 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:16.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.255+0000 7f36f3ec7700 1 --2- 172.21.15.46:0/4138589907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36ec15dfd0 0x7f36ec158530 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:16.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.255+0000 7f36f3ec7700 1 --2- 172.21.15.46:0/4138589907 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36ec1599d0 0x7f36ec0ab980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:16.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.255+0000 7f36f3ec7700 1 --2- 172.21.15.46:0/4138589907 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f36ec159da0 0x7f36ec157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:16.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.255+0000 7f36f3ec7700 1 -- 172.21.15.46:0/4138589907 >> 172.21.15.46:0/4138589907 conn(0x7f36ec06f840 msgr2=0x7f36ec15d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:16.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.255+0000 7f36f3ec7700 1 -- 172.21.15.46:0/4138589907 shutdown_connections 2024-01-27T00:46:16.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.255+0000 7f36f3ec7700 1 -- 172.21.15.46:0/4138589907 wait complete. 2024-01-27T00:46:16.268 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-01-26T00:00:00", "created": "2024-01-26T23:50:20", "first": "2024-01-26T23:51:00", "last": "2024-01-27T00:46:00", "last_pruned": "2024-01-27T00:46:00", "created_count": 56, "pruned_count": 1, "active": true}] 2024-01-27T00:46:16.268 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1m', 'retention': {'n': 55}, 'start': '2024-01-26T00:00:00', 'created': '2024-01-26T23:50:20', 'first': '2024-01-26T23:51:00', 'last': '2024-01-27T00:46:00', 'last_pruned': '2024-01-27T00:46:00', 'created_count': 56, 'pruned_count': 1, 'active': True} 2024-01-27T00:46:16.268 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2024-01-27T00:46:16.268 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2024-01-27T00:46:16.268 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1 2024-01-27T00:46:16.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.403+0000 7fe92becd700 1 Processor -- start 2024-01-27T00:46:16.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.403+0000 7fe92becd700 1 -- start start 2024-01-27T00:46:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.404+0000 7fe92becd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe924159cb0 0x7fe9240cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.404+0000 7fe92becd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe9240afb40 0x7fe9240c5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.404+0000 7fe92becd700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9240cc100 con 0x7fe9240cf050 2024-01-27T00:46:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.404+0000 7fe92becd700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9240cb850 con 0x7fe9240afb40 2024-01-27T00:46:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.404+0000 7fe92becd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9240cbaf0 con 0x7fe924159cb0 2024-01-27T00:46:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.404+0000 7fe929c69700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe924159cb0 0x7fe9240cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.404+0000 7fe92a46a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe9240afb40 0x7fe9240c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.404+0000 7fe92a46a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe9240afb40 0x7fe9240c5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:60380/0 (socket says 172.21.15.46:60380) 2024-01-27T00:46:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.404+0000 7fe929c69700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe924159cb0 0x7fe9240cc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:52034/0 (socket says 172.21.15.46:52034) 2024-01-27T00:46:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.404+0000 7fe929468700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fe9240cf050 0x7fe9240af430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:35386/0 (socket says 172.21.15.46:35386) 2024-01-27T00:46:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.404+0000 7fe92a46a700 1 -- 172.21.15.46:0/4118737296 learned_addr learned my addr 172.21.15.46:0/4118737296 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.405+0000 7fe92a46a700 1 -- 172.21.15.46:0/4118737296 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe924159cb0 msgr2=0x7fe9240cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.405+0000 7fe92a46a700 1 --2- 172.21.15.46:0/4118737296 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe924159cb0 0x7fe9240cc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.405+0000 7fe92a46a700 1 -- 172.21.15.46:0/4118737296 >> v1:172.21.15.106:6789/0 conn(0x7fe9240cf050 legacy=0x7fe9240af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.405+0000 7fe92a46a700 1 -- 172.21.15.46:0/4118737296 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9240cfc80 con 0x7fe9240afb40 2024-01-27T00:46:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.405+0000 7fe928c67700 1 -- 172.21.15.46:0/4118737296 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 352520423 0 0) 0x7fe9240cc100 con 0x7fe9240cf050 2024-01-27T00:46:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.405+0000 7fe929c69700 1 --2- 172.21.15.46:0/4118737296 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe924159cb0 0x7fe9240cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.405+0000 7fe92a46a700 1 --2- 172.21.15.46:0/4118737296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9240afb40 0x7fe9240c5530 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009f20 tx=0x7fe91c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=304c5c6e895ff848 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.405+0000 7fe928c67700 1 -- 172.21.15.46:0/4118737296 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c00e070 con 0x7fe9240afb40 2024-01-27T00:46:16.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.405+0000 7fe928c67700 1 -- 172.21.15.46:0/4118737296 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91c00b9d0 con 0x7fe9240afb40 2024-01-27T00:46:16.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.405+0000 7fe928c67700 1 -- 172.21.15.46:0/4118737296 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c0092e0 con 0x7fe9240afb40 2024-01-27T00:46:16.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.406+0000 7fe92becd700 1 -- 172.21.15.46:0/4118737296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9240afb40 msgr2=0x7fe9240c5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:16.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.406+0000 7fe92becd700 1 --2- 172.21.15.46:0/4118737296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9240afb40 0x7fe9240c5530 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009f20 tx=0x7fe91c009840 comp rx=0 tx=0).stop 2024-01-27T00:46:16.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.406+0000 7fe92becd700 1 -- 172.21.15.46:0/4118737296 shutdown_connections 2024-01-27T00:46:16.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.406+0000 7fe92becd700 1 --2- 172.21.15.46:0/4118737296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9240afb40 0x7fe9240c5530 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:16.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.406+0000 7fe92becd700 1 --2- 172.21.15.46:0/4118737296 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe924159cb0 0x7fe9240cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:16.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.406+0000 7fe92becd700 1 -- 172.21.15.46:0/4118737296 >> 172.21.15.46:0/4118737296 conn(0x7fe924087f60 msgr2=0x7fe92406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:16.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.406+0000 7fe92becd700 1 -- 172.21.15.46:0/4118737296 shutdown_connections 2024-01-27T00:46:16.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.406+0000 7fe92becd700 1 -- 172.21.15.46:0/4118737296 wait complete. 2024-01-27T00:46:16.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.407+0000 7fe92becd700 1 Processor -- start 2024-01-27T00:46:16.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.407+0000 7fe92becd700 1 -- start start 2024-01-27T00:46:16.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.408+0000 7fe92becd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9240afb40 0x7fe924156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:16.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.408+0000 7fe92becd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9240cf050 0x7fe9241566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:16.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.408+0000 7fe92becd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe924159cb0 0x7fe9241581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:16.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.408+0000 7fe92becd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9240ca4d0 con 0x7fe9240afb40 2024-01-27T00:46:16.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.408+0000 7fe92becd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9240b1150 con 0x7fe9240cf050 2024-01-27T00:46:16.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.408+0000 7fe92becd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9240b0820 con 0x7fe924159cb0 2024-01-27T00:46:16.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.408+0000 7fe929c69700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9240afb40 0x7fe924156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:16.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.408+0000 7fe929c69700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9240afb40 0x7fe924156190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34756/0 (socket says 172.21.15.46:34756) 2024-01-27T00:46:16.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.408+0000 7fe929c69700 1 -- 172.21.15.46:0/2677268097 learned_addr learned my addr 172.21.15.46:0/2677268097 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:16.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.408+0000 7fe929468700 1 --2- 172.21.15.46:0/2677268097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9240cf050 0x7fe9241566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:16.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.408+0000 7fe92a46a700 1 --2- 172.21.15.46:0/2677268097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe924159cb0 0x7fe9241581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:16.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.409+0000 7fe929c69700 1 -- 172.21.15.46:0/2677268097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe924159cb0 msgr2=0x7fe9241581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.409+0000 7fe929c69700 1 --2- 172.21.15.46:0/2677268097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe924159cb0 0x7fe9241581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.409+0000 7fe929c69700 1 -- 172.21.15.46:0/2677268097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9240cf050 msgr2=0x7fe9241566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.409+0000 7fe929c69700 1 --2- 172.21.15.46:0/2677268097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9240cf050 0x7fe9241566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.409+0000 7fe929c69700 1 -- 172.21.15.46:0/2677268097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9200093c0 con 0x7fe9240afb40 2024-01-27T00:46:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.409+0000 7fe929c69700 1 --2- 172.21.15.46:0/2677268097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9240afb40 0x7fe924156190 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fe92000ec80 tx=0x7fe920009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.409+0000 7fe919ffb700 1 -- 172.21.15.46:0/2677268097 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920005a00 con 0x7fe9240afb40 2024-01-27T00:46:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.409+0000 7fe919ffb700 1 -- 172.21.15.46:0/2677268097 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe920005b60 con 0x7fe9240afb40 2024-01-27T00:46:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.409+0000 7fe92becd700 1 -- 172.21.15.46:0/2677268097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe91c009ae0 con 0x7fe9240afb40 2024-01-27T00:46:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.409+0000 7fe919ffb700 1 -- 172.21.15.46:0/2677268097 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92000c850 con 0x7fe9240afb40 2024-01-27T00:46:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.409+0000 7fe92becd700 1 -- 172.21.15.46:0/2677268097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9240b0de0 con 0x7fe9240afb40 2024-01-27T00:46:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.410+0000 7fe92becd700 1 -- 172.21.15.46:0/2677268097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe92408a810 con 0x7fe9240afb40 2024-01-27T00:46:16.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.411+0000 7fe919ffb700 1 -- 172.21.15.46:0/2677268097 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 83) v1 ==== 84873+0+0 (secure 0 0 0) 0x7fe92000cbc0 con 0x7fe9240afb40 2024-01-27T00:46:16.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.411+0000 7fe919ffb700 1 --2- 172.21.15.46:0/2677268097 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe9080670a0 0x7fe908069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:16.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.411+0000 7fe919ffb700 1 -- 172.21.15.46:0/2677268097 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6763+0+0 (secure 0 0 0) 0x7fe920084f00 con 0x7fe9240afb40 2024-01-27T00:46:16.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.412+0000 7fe929468700 1 --2- 172.21.15.46:0/2677268097 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe9080670a0 0x7fe908069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:16.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.412+0000 7fe929468700 1 --2- 172.21.15.46:0/2677268097 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe9080670a0 0x7fe908069550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe92416d090 tx=0x7fe914007d50 comp rx=0 tx=0).ready entity=mgr.8743 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:16.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.416+0000 7fe919ffb700 1 -- 172.21.15.46:0/2677268097 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe920051c40 con 0x7fe9240afb40 2024-01-27T00:46:16.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.649+0000 7fe92becd700 1 -- 172.21.15.46:0/2677268097 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9240cf740 con 0x7fe9080670a0 2024-01-27T00:46:16.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.717+0000 7fe919ffb700 1 -- 172.21.15.46:0/2677268097 <== mgr.8743 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+42 (secure 0 0 0) 0x7fe9240cf740 con 0x7fe9080670a0 2024-01-27T00:46:16.717 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path /snap_test_dir1 2024-01-27T00:46:16.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.720+0000 7fe92becd700 1 -- 172.21.15.46:0/2677268097 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe9080670a0 msgr2=0x7fe908069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:16.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.720+0000 7fe92becd700 1 --2- 172.21.15.46:0/2677268097 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe9080670a0 0x7fe908069550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe92416d090 tx=0x7fe914007d50 comp rx=0 tx=0).stop 2024-01-27T00:46:16.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.720+0000 7fe92becd700 1 -- 172.21.15.46:0/2677268097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9240afb40 msgr2=0x7fe924156190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:16.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.720+0000 7fe92becd700 1 --2- 172.21.15.46:0/2677268097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9240afb40 0x7fe924156190 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fe92000ec80 tx=0x7fe920009120 comp rx=0 tx=0).stop 2024-01-27T00:46:16.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.720+0000 7fe92becd700 1 -- 172.21.15.46:0/2677268097 shutdown_connections 2024-01-27T00:46:16.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.720+0000 7fe92becd700 1 --2- 172.21.15.46:0/2677268097 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe9080670a0 0x7fe908069550 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:16.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.720+0000 7fe92becd700 1 --2- 172.21.15.46:0/2677268097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9240afb40 0x7fe924156190 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:16.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.720+0000 7fe92becd700 1 --2- 172.21.15.46:0/2677268097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe924159cb0 0x7fe9241581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:16.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.720+0000 7fe92becd700 1 --2- 172.21.15.46:0/2677268097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9240cf050 0x7fe9241566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:16.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.720+0000 7fe92becd700 1 -- 172.21.15.46:0/2677268097 >> 172.21.15.46:0/2677268097 conn(0x7fe924087f60 msgr2=0x7fe924155780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:16.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.720+0000 7fe92becd700 1 -- 172.21.15.46:0/2677268097 shutdown_connections 2024-01-27T00:46:16.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:16.720+0000 7fe92becd700 1 -- 172.21.15.46:0/2677268097 wait complete. 2024-01-27T00:46:16.734 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2024-01-27T00:46:16.734 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:16.734 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T00:46:16.761 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_52_00_UTC 2024-01-27T00:46:16.761 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_53_00_UTC 2024-01-27T00:46:16.761 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_54_00_UTC 2024-01-27T00:46:16.761 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_55_00_UTC 2024-01-27T00:46:16.761 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_56_00_UTC 2024-01-27T00:46:16.761 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_57_00_UTC 2024-01-27T00:46:16.762 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_58_00_UTC 2024-01-27T00:46:16.762 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-26-23_59_00_UTC 2024-01-27T00:46:16.762 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_00_00_UTC 2024-01-27T00:46:16.762 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_01_00_UTC 2024-01-27T00:46:16.762 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_02_00_UTC 2024-01-27T00:46:16.762 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_03_00_UTC 2024-01-27T00:46:16.762 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_04_00_UTC 2024-01-27T00:46:16.762 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_05_00_UTC 2024-01-27T00:46:16.762 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_06_00_UTC 2024-01-27T00:46:16.762 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_07_00_UTC 2024-01-27T00:46:16.762 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_08_00_UTC 2024-01-27T00:46:16.762 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_09_00_UTC 2024-01-27T00:46:16.762 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_10_00_UTC 2024-01-27T00:46:16.763 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_11_00_UTC 2024-01-27T00:46:16.763 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_12_00_UTC 2024-01-27T00:46:16.763 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_13_00_UTC 2024-01-27T00:46:16.763 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_14_00_UTC 2024-01-27T00:46:16.763 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_15_00_UTC 2024-01-27T00:46:16.763 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_16_00_UTC 2024-01-27T00:46:16.763 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_17_00_UTC 2024-01-27T00:46:16.763 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_18_00_UTC 2024-01-27T00:46:16.763 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_19_00_UTC 2024-01-27T00:46:16.763 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_20_00_UTC 2024-01-27T00:46:16.763 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_21_00_UTC 2024-01-27T00:46:16.763 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_22_00_UTC 2024-01-27T00:46:16.763 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_23_00_UTC 2024-01-27T00:46:16.763 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_24_00_UTC 2024-01-27T00:46:16.764 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_25_00_UTC 2024-01-27T00:46:16.764 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_26_00_UTC 2024-01-27T00:46:16.764 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_27_00_UTC 2024-01-27T00:46:16.764 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_28_00_UTC 2024-01-27T00:46:16.764 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_29_00_UTC 2024-01-27T00:46:16.764 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_30_00_UTC 2024-01-27T00:46:16.764 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_31_00_UTC 2024-01-27T00:46:16.764 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_32_00_UTC 2024-01-27T00:46:16.764 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_33_00_UTC 2024-01-27T00:46:16.764 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_34_00_UTC 2024-01-27T00:46:16.764 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_35_00_UTC 2024-01-27T00:46:16.764 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_36_00_UTC 2024-01-27T00:46:16.764 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_37_00_UTC 2024-01-27T00:46:16.765 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_38_00_UTC 2024-01-27T00:46:16.765 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_39_00_UTC 2024-01-27T00:46:16.765 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_40_00_UTC 2024-01-27T00:46:16.765 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_41_00_UTC 2024-01-27T00:46:16.765 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_42_00_UTC 2024-01-27T00:46:16.765 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_43_00_UTC 2024-01-27T00:46:16.765 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_44_00_UTC 2024-01-27T00:46:16.765 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_45_00_UTC 2024-01-27T00:46:16.765 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_46_00_UTC 2024-01-27T00:46:16.765 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-26-23_52_00_UTC 2024-01-27T00:46:16.765 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:16.766 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-26-23_52_00_UTC) 2024-01-27T00:46:16.832 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-26-23_53_00_UTC 2024-01-27T00:46:16.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:16.832 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-26-23_53_00_UTC) 2024-01-27T00:46:16.904 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-26-23_54_00_UTC 2024-01-27T00:46:16.904 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:16.905 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-26-23_54_00_UTC) 2024-01-27T00:46:16.976 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-26-23_55_00_UTC 2024-01-27T00:46:16.976 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:16.976 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-26-23_55_00_UTC) 2024-01-27T00:46:17.047 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-26-23_56_00_UTC 2024-01-27T00:46:17.048 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:17.048 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-26-23_56_00_UTC) 2024-01-27T00:46:17.119 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-26-23_57_00_UTC 2024-01-27T00:46:17.119 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:17.119 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-26-23_57_00_UTC) 2024-01-27T00:46:17.190 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-26-23_58_00_UTC 2024-01-27T00:46:17.191 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:17.191 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-26-23_58_00_UTC) 2024-01-27T00:46:17.262 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-26-23_59_00_UTC 2024-01-27T00:46:17.262 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:17.263 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-26-23_59_00_UTC) 2024-01-27T00:46:17.333 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_00_00_UTC 2024-01-27T00:46:17.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:17.334 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_00_00_UTC) 2024-01-27T00:46:17.405 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_01_00_UTC 2024-01-27T00:46:17.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:17.405 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_01_00_UTC) 2024-01-27T00:46:17.477 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_02_00_UTC 2024-01-27T00:46:17.477 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:17.477 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_02_00_UTC) 2024-01-27T00:46:17.548 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_03_00_UTC 2024-01-27T00:46:17.548 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:17.548 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_03_00_UTC) 2024-01-27T00:46:17.619 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_04_00_UTC 2024-01-27T00:46:17.620 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:17.620 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_04_00_UTC) 2024-01-27T00:46:17.692 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_05_00_UTC 2024-01-27T00:46:17.692 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:17.692 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_05_00_UTC) 2024-01-27T00:46:17.763 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_06_00_UTC 2024-01-27T00:46:17.763 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:17.763 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_06_00_UTC) 2024-01-27T00:46:17.834 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_07_00_UTC 2024-01-27T00:46:17.834 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:17.835 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_07_00_UTC) 2024-01-27T00:46:17.905 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_08_00_UTC 2024-01-27T00:46:17.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:17.905 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_08_00_UTC) 2024-01-27T00:46:17.974 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_09_00_UTC 2024-01-27T00:46:17.975 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:17.975 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_09_00_UTC) 2024-01-27T00:46:18.046 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_10_00_UTC 2024-01-27T00:46:18.047 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:18.047 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_10_00_UTC) 2024-01-27T00:46:18.119 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_11_00_UTC 2024-01-27T00:46:18.119 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:18.119 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_11_00_UTC) 2024-01-27T00:46:18.190 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_12_00_UTC 2024-01-27T00:46:18.191 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:18.191 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_12_00_UTC) 2024-01-27T00:46:18.262 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_13_00_UTC 2024-01-27T00:46:18.262 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:18.262 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_13_00_UTC) 2024-01-27T00:46:18.333 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_14_00_UTC 2024-01-27T00:46:18.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:18.333 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_14_00_UTC) 2024-01-27T00:46:18.406 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_15_00_UTC 2024-01-27T00:46:18.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:18.406 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_15_00_UTC) 2024-01-27T00:46:18.479 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_16_00_UTC 2024-01-27T00:46:18.479 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:18.479 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_16_00_UTC) 2024-01-27T00:46:18.551 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_17_00_UTC 2024-01-27T00:46:18.551 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:18.551 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_17_00_UTC) 2024-01-27T00:46:18.622 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_18_00_UTC 2024-01-27T00:46:18.623 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:18.623 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_18_00_UTC) 2024-01-27T00:46:18.695 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_19_00_UTC 2024-01-27T00:46:18.695 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:18.695 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_19_00_UTC) 2024-01-27T00:46:18.766 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_20_00_UTC 2024-01-27T00:46:18.766 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:18.766 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_20_00_UTC) 2024-01-27T00:46:18.838 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_21_00_UTC 2024-01-27T00:46:18.838 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:18.838 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_21_00_UTC) 2024-01-27T00:46:18.909 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_22_00_UTC 2024-01-27T00:46:18.909 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:18.909 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_22_00_UTC) 2024-01-27T00:46:18.981 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_23_00_UTC 2024-01-27T00:46:18.981 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:18.981 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_23_00_UTC) 2024-01-27T00:46:19.052 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_24_00_UTC 2024-01-27T00:46:19.052 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:19.052 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_24_00_UTC) 2024-01-27T00:46:19.123 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_25_00_UTC 2024-01-27T00:46:19.123 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:19.123 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_25_00_UTC) 2024-01-27T00:46:19.194 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_26_00_UTC 2024-01-27T00:46:19.194 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:19.194 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_26_00_UTC) 2024-01-27T00:46:19.265 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_27_00_UTC 2024-01-27T00:46:19.265 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:19.265 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_27_00_UTC) 2024-01-27T00:46:19.336 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_28_00_UTC 2024-01-27T00:46:19.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:19.337 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_28_00_UTC) 2024-01-27T00:46:19.408 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_29_00_UTC 2024-01-27T00:46:19.408 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:19.409 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_29_00_UTC) 2024-01-27T00:46:19.480 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_30_00_UTC 2024-01-27T00:46:19.480 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:19.481 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_30_00_UTC) 2024-01-27T00:46:19.552 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_31_00_UTC 2024-01-27T00:46:19.552 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:19.552 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_31_00_UTC) 2024-01-27T00:46:19.623 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_32_00_UTC 2024-01-27T00:46:19.623 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:19.624 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_32_00_UTC) 2024-01-27T00:46:19.695 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_33_00_UTC 2024-01-27T00:46:19.695 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:19.695 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_33_00_UTC) 2024-01-27T00:46:19.769 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_34_00_UTC 2024-01-27T00:46:19.769 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:19.769 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_34_00_UTC) 2024-01-27T00:46:19.840 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_35_00_UTC 2024-01-27T00:46:19.840 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:19.840 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_35_00_UTC) 2024-01-27T00:46:19.911 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_36_00_UTC 2024-01-27T00:46:19.911 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:19.911 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_36_00_UTC) 2024-01-27T00:46:19.982 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_37_00_UTC 2024-01-27T00:46:19.983 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:19.983 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_37_00_UTC) 2024-01-27T00:46:20.054 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_38_00_UTC 2024-01-27T00:46:20.054 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:20.054 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_38_00_UTC) 2024-01-27T00:46:20.125 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_39_00_UTC 2024-01-27T00:46:20.125 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:20.125 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_39_00_UTC) 2024-01-27T00:46:20.196 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_40_00_UTC 2024-01-27T00:46:20.196 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:20.197 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_40_00_UTC) 2024-01-27T00:46:20.267 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_41_00_UTC 2024-01-27T00:46:20.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:20.267 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_41_00_UTC) 2024-01-27T00:46:20.338 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_42_00_UTC 2024-01-27T00:46:20.338 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:20.338 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_42_00_UTC) 2024-01-27T00:46:20.408 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_43_00_UTC 2024-01-27T00:46:20.409 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:20.409 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_43_00_UTC) 2024-01-27T00:46:20.480 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_44_00_UTC 2024-01-27T00:46:20.480 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:20.481 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_44_00_UTC) 2024-01-27T00:46:20.551 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_45_00_UTC 2024-01-27T00:46:20.551 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:20.551 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_45_00_UTC) 2024-01-27T00:46:20.621 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-00_46_00_UTC 2024-01-27T00:46:20.621 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:20.621 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-00_46_00_UTC) 2024-01-27T00:46:20.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:20.691 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-01-27T00:46:20.758 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-27T00:46:20.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.943+0000 7f51d9f65700 1 Processor -- start 2024-01-27T00:46:20.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.943+0000 7f51d9f65700 1 -- start start 2024-01-27T00:46:20.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.944+0000 7f51d9f65700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51d40c8e80 0x7f51d4068940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:20.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.944+0000 7f51d9f65700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d415a720 con 0x7f51d40c8a30 2024-01-27T00:46:20.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.944+0000 7f51d9f65700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d406c870 con 0x7f51d40c8660 2024-01-27T00:46:20.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.944+0000 7f51d9f65700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d406c020 con 0x7f51d40c8e80 2024-01-27T00:46:20.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.944+0000 7f51d2ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f51d40c8a30 0x7f51d406b910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46070/0 (socket says 172.21.15.46:46070) 2024-01-27T00:46:20.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d2ffd700 1 -- 172.21.15.46:0/2482298833 learned_addr learned my addr 172.21.15.46:0/2482298833 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:20.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d3fff700 1 --2- 172.21.15.46:0/2482298833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51d40c8e80 0x7f51d4068940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:20.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d27fc700 1 -- 172.21.15.46:0/2482298833 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262639806 0 0) 0x7f51d415a720 con 0x7f51d40c8a30 2024-01-27T00:46:20.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d27fc700 1 -- 172.21.15.46:0/2482298833 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51c0003570 con 0x7f51d40c8a30 2024-01-27T00:46:20.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d27fc700 1 -- 172.21.15.46:0/2482298833 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629246449 0 0) 0x7f51d406c870 con 0x7f51d40c8660 2024-01-27T00:46:20.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d27fc700 1 -- 172.21.15.46:0/2482298833 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51d415a720 con 0x7f51d40c8660 2024-01-27T00:46:20.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d27fc700 1 -- 172.21.15.46:0/2482298833 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2070077982 0 0) 0x7f51c0003570 con 0x7f51d40c8a30 2024-01-27T00:46:20.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d27fc700 1 -- 172.21.15.46:0/2482298833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51d40c8e80 msgr2=0x7f51d4068940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:20.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d27fc700 1 --2- 172.21.15.46:0/2482298833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51d40c8e80 0x7f51d4068940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:20.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d27fc700 1 -- 172.21.15.46:0/2482298833 >> v1:172.21.15.106:6789/0 conn(0x7f51d40c8660 legacy=0x7f51d415abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d27fc700 1 -- 172.21.15.46:0/2482298833 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d406c2b0 con 0x7f51d40c8a30 2024-01-27T00:46:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d27fc700 1 -- 172.21.15.46:0/2482298833 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f51c4001470 con 0x7f51d40c8a30 2024-01-27T00:46:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d3fff700 1 --2- 172.21.15.46:0/2482298833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51d40c8e80 0x7f51d4068940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d27fc700 1 -- 172.21.15.46:0/2482298833 >> v1:172.21.15.46:6789/0 conn(0x7f51d40c8a30 legacy=0x7f51d406b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d27fc700 1 --2- 172.21.15.46:0/2482298833 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51c0003c20 0x7f51c0004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d27fc700 1 --2- 172.21.15.46:0/2482298833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51d40c8e80 0x7f51c0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d27fc700 1 --2- 172.21.15.46:0/2482298833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51c0004d50 0x7f51c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d27fc700 1 -- 172.21.15.46:0/2482298833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d406c020 con 0x7f51c0004d50 2024-01-27T00:46:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d27fc700 1 -- 172.21.15.46:0/2482298833 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c000bb10 con 0x7f51c0003c20 2024-01-27T00:46:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d27fc700 1 -- 172.21.15.46:0/2482298833 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c000dee0 con 0x7f51d40c8e80 2024-01-27T00:46:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d3fff700 1 --2- 172.21.15.46:0/2482298833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51c0004d50 0x7f51c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d2ffd700 1 --2- 172.21.15.46:0/2482298833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51d40c8e80 0x7f51c0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.945+0000 7f51d37fe700 1 --2- 172.21.15.46:0/2482298833 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51c0003c20 0x7f51c0004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.946+0000 7f51d3fff700 1 -- 172.21.15.46:0/2482298833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51d40c8e80 msgr2=0x7f51c0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.946+0000 7f51d3fff700 1 --2- 172.21.15.46:0/2482298833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51d40c8e80 0x7f51c0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.946+0000 7f51d3fff700 1 -- 172.21.15.46:0/2482298833 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51c0003c20 msgr2=0x7f51c0004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.946+0000 7f51d3fff700 1 --2- 172.21.15.46:0/2482298833 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51c0003c20 0x7f51c0004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.946+0000 7f51d3fff700 1 -- 172.21.15.46:0/2482298833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51cc009a20 con 0x7f51c0004d50 2024-01-27T00:46:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.946+0000 7f51d3fff700 1 --2- 172.21.15.46:0/2482298833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51c0004d50 0x7f51c00051c0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f51d4063db0 tx=0x7f51cc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d1019278388d828 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.946+0000 7f51d27fc700 1 -- 172.21.15.46:0/2482298833 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc00ebb0 con 0x7f51c0004d50 2024-01-27T00:46:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.946+0000 7f51d27fc700 1 -- 172.21.15.46:0/2482298833 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51cc00f040 con 0x7f51c0004d50 2024-01-27T00:46:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.946+0000 7f51d27fc700 1 -- 172.21.15.46:0/2482298833 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc014480 con 0x7f51c0004d50 2024-01-27T00:46:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.947+0000 7f51d9f65700 1 -- 172.21.15.46:0/2482298833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51c0004d50 msgr2=0x7f51c00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.947+0000 7f51d9f65700 1 --2- 172.21.15.46:0/2482298833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51c0004d50 0x7f51c00051c0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f51d4063db0 tx=0x7f51cc00b040 comp rx=0 tx=0).stop 2024-01-27T00:46:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.947+0000 7f51d3fff700 1 -- 172.21.15.46:0/2482298833 reap_dead start 2024-01-27T00:46:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.947+0000 7f51d9f65700 1 -- 172.21.15.46:0/2482298833 shutdown_connections 2024-01-27T00:46:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.947+0000 7f51d9f65700 1 -- 172.21.15.46:0/2482298833 >> 172.21.15.46:0/2482298833 conn(0x7f51d40ca100 msgr2=0x7f51d406e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:20.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.947+0000 7f51d9f65700 1 -- 172.21.15.46:0/2482298833 shutdown_connections 2024-01-27T00:46:20.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.947+0000 7f51d9f65700 1 -- 172.21.15.46:0/2482298833 wait complete. 2024-01-27T00:46:20.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.948+0000 7f51d9f65700 1 Processor -- start 2024-01-27T00:46:20.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.948+0000 7f51d9f65700 1 -- start start 2024-01-27T00:46:20.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d9f65700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51c0004d50 0x7f51d4155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:20.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d9f65700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51d40c26a0 0x7f51d40c2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:20.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d9f65700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51d40c54d0 0x7f51d40c5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:20.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d9f65700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d40c3050 con 0x7f51d40c54d0 2024-01-27T00:46:20.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d9f65700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d4159410 con 0x7f51c0004d50 2024-01-27T00:46:20.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d9f65700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d40c34e0 con 0x7f51d40c26a0 2024-01-27T00:46:20.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51d40c54d0 0x7f51d40c5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:20.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51d40c54d0 0x7f51d40c5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49240/0 (socket says 172.21.15.46:49240) 2024-01-27T00:46:20.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d3fff700 1 -- 172.21.15.46:0/2633897314 learned_addr learned my addr 172.21.15.46:0/2633897314 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:20.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d37fe700 1 --2- 172.21.15.46:0/2633897314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51c0004d50 0x7f51d4155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:20.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d2ffd700 1 --2- 172.21.15.46:0/2633897314 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51d40c26a0 0x7f51d40c2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:20.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d3fff700 1 -- 172.21.15.46:0/2633897314 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51d40c26a0 msgr2=0x7f51d40c2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:20.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d3fff700 1 --2- 172.21.15.46:0/2633897314 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51d40c26a0 0x7f51d40c2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:20.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d3fff700 1 -- 172.21.15.46:0/2633897314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51c0004d50 msgr2=0x7f51d4155380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:20.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d3fff700 1 --2- 172.21.15.46:0/2633897314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51c0004d50 0x7f51d4155380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:20.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d3fff700 1 -- 172.21.15.46:0/2633897314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d4158800 con 0x7f51d40c54d0 2024-01-27T00:46:20.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d37fe700 1 --2- 172.21.15.46:0/2633897314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51c0004d50 0x7f51d4155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:46:20.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d2ffd700 1 --2- 172.21.15.46:0/2633897314 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51d40c26a0 0x7f51d40c2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:46:20.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.949+0000 7f51d3fff700 1 --2- 172.21.15.46:0/2633897314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51d40c54d0 0x7f51d40c5920 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f51cc0094d0 tx=0x7f51cc01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:20.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.950+0000 7f51cb7fe700 1 -- 172.21.15.46:0/2633897314 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc01d630 con 0x7f51d40c54d0 2024-01-27T00:46:20.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.950+0000 7f51cb7fe700 1 -- 172.21.15.46:0/2633897314 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51cc01d7b0 con 0x7f51d40c54d0 2024-01-27T00:46:20.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.950+0000 7f51d9f65700 1 -- 172.21.15.46:0/2633897314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d4158ae0 con 0x7f51d40c54d0 2024-01-27T00:46:20.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.950+0000 7f51cb7fe700 1 -- 172.21.15.46:0/2633897314 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc01db70 con 0x7f51d40c54d0 2024-01-27T00:46:20.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.950+0000 7f51d9f65700 1 -- 172.21.15.46:0/2633897314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d41590a0 con 0x7f51d40c54d0 2024-01-27T00:46:20.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.951+0000 7f51d9f65700 1 -- 172.21.15.46:0/2633897314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d40c5340 con 0x7f51d40c54d0 2024-01-27T00:46:20.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.951+0000 7f51cb7fe700 1 -- 172.21.15.46:0/2633897314 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 83) v1 ==== 84873+0+0 (secure 0 0 0) 0x7f51cc0133e0 con 0x7f51d40c54d0 2024-01-27T00:46:20.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.952+0000 7f51cb7fe700 1 --2- 172.21.15.46:0/2633897314 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f51b40670a0 0x7f51b4069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:20.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.952+0000 7f51cb7fe700 1 -- 172.21.15.46:0/2633897314 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6763+0+0 (secure 0 0 0) 0x7f51cc085fe0 con 0x7f51d40c54d0 2024-01-27T00:46:20.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.952+0000 7f51d37fe700 1 --2- 172.21.15.46:0/2633897314 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f51b40670a0 0x7f51b4069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:20.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.952+0000 7f51d37fe700 1 --2- 172.21.15.46:0/2633897314 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f51b40670a0 0x7f51b4069550 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f51bc000fb0 tx=0x7f51bc009040 comp rx=0 tx=0).ready entity=mgr.8743 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:20.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:20.957+0000 7f51cb7fe700 1 -- 172.21.15.46:0/2633897314 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f51cc052b70 con 0x7f51d40c54d0 2024-01-27T00:46:21.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:21.225+0000 7f51d9f65700 1 -- 172.21.15.46:0/2633897314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f51d404c710 con 0x7f51d40c54d0 2024-01-27T00:46:21.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:21.488+0000 7f51cb7fe700 1 -- 172.21.15.46:0/2633897314 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 84) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f51cc052f40 con 0x7f51d40c54d0 2024-01-27T00:46:21.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:21.491+0000 7f51cb7fe700 1 -- 172.21.15.46:0/2633897314 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v84) v1 ==== 93+0+0 (secure 0 0 0) 0x7f51cc056600 con 0x7f51d40c54d0 2024-01-27T00:46:21.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:21.502+0000 7f51d9f65700 1 -- 172.21.15.46:0/2633897314 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f51b40670a0 msgr2=0x7f51b4069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:21.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:21.502+0000 7f51d9f65700 1 --2- 172.21.15.46:0/2633897314 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f51b40670a0 0x7f51b4069550 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f51bc000fb0 tx=0x7f51bc009040 comp rx=0 tx=0).stop 2024-01-27T00:46:21.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:21.502+0000 7f51d9f65700 1 -- 172.21.15.46:0/2633897314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51d40c54d0 msgr2=0x7f51d40c5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:21.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:21.502+0000 7f51d9f65700 1 --2- 172.21.15.46:0/2633897314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51d40c54d0 0x7f51d40c5920 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f51cc0094d0 tx=0x7f51cc01a000 comp rx=0 tx=0).stop 2024-01-27T00:46:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:21.502+0000 7f51d9f65700 1 -- 172.21.15.46:0/2633897314 shutdown_connections 2024-01-27T00:46:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:21.502+0000 7f51d9f65700 1 --2- 172.21.15.46:0/2633897314 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f51b40670a0 0x7f51b4069550 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:21.502+0000 7f51d9f65700 1 --2- 172.21.15.46:0/2633897314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51d40c54d0 0x7f51d40c5920 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:21.502+0000 7f51d9f65700 1 --2- 172.21.15.46:0/2633897314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51c0004d50 0x7f51d4155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:21.502+0000 7f51d9f65700 1 --2- 172.21.15.46:0/2633897314 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51d40c26a0 0x7f51d40c2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:21.502+0000 7f51d9f65700 1 -- 172.21.15.46:0/2633897314 >> 172.21.15.46:0/2633897314 conn(0x7f51d40ca100 msgr2=0x7f51d40cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:21.503+0000 7f51d9f65700 1 -- 172.21.15.46:0/2633897314 shutdown_connections 2024-01-27T00:46:21.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:21.503+0000 7f51d9f65700 1 -- 172.21.15.46:0/2633897314 wait complete. 2024-01-27T00:46:21.534 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-27T00:46:21.534 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T00:46:21.641 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T00:46:21.712 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-27T00:46:21.712 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-27T00:46:21.712 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-27T00:46:21.712 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-27T00:46:21.772 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[90782]: fuse finished with error 0 and tester_r 0 2024-01-27T00:46:21.775 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T00:46:21.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:46:21.775 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-27T00:46:21.927 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T00:46:21.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:46:21.927 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-27T00:46:21.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T00:46:21.947 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-27T00:46:21.947 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T00:46:21.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:46:21.948 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-27T00:46:21.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T00:46:21.994 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-27T00:46:21.994 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T00:46:21.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:46:21.994 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-27T00:46:22.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T00:46:22.011 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-27T00:46:22.012 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T00:46:22.014 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:22.014+0000 7f7a7a4bf1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T00:46:22.086 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:22.087+0000 7ff6f08dc1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T00:46:22.091 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:22.092+0000 7f7a7a4bf1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T00:46:22.165 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:22.166+0000 7f7a7a4bf1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T00:46:22.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.189+0000 7f4da6697700 1 Processor -- start 2024-01-27T00:46:22.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.189+0000 7f4da6697700 1 -- start start 2024-01-27T00:46:22.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.190+0000 7f4da6697700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4da00b0840 0x7f4da0156170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:22.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.190+0000 7f4da6697700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4da00afbd0 0x7f4da01566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:22.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.190+0000 7f4da6697700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da0157d50 con 0x7f4da006ed00 2024-01-27T00:46:22.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.190+0000 7f4da6697700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4da00af920 con 0x7f4da00afbd0 2024-01-27T00:46:22.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.190+0000 7f4da6697700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4da0157140 con 0x7f4da00b0840 2024-01-27T00:46:22.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.190+0000 7f4da4c34700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4da00afbd0 0x7f4da01566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:22.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.190+0000 7f4d9ffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4da006ed00 0x7f4da00b0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46094/0 (socket says 172.21.15.46:46094) 2024-01-27T00:46:22.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.190+0000 7f4d9ffff700 1 -- 172.21.15.46:0/369168625 learned_addr learned my addr 172.21.15.46:0/369168625 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.190+0000 7f4d9effd700 1 -- 172.21.15.46:0/369168625 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745070681 0 0) 0x7f4da0157d50 con 0x7f4da006ed00 2024-01-27T00:46:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.191+0000 7f4d9effd700 1 -- 172.21.15.46:0/369168625 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d88003570 con 0x7f4da006ed00 2024-01-27T00:46:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.191+0000 7f4da4c34700 1 -- 172.21.15.46:0/369168625 >> v2:172.21.15.106:3300/0 conn(0x7f4da00b0840 msgr2=0x7f4da0156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.191+0000 7f4da4c34700 1 --2- 172.21.15.46:0/369168625 >> v2:172.21.15.106:3300/0 conn(0x7f4da00b0840 0x7f4da0156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.191+0000 7f4da4c34700 1 -- 172.21.15.46:0/369168625 >> v1:172.21.15.46:6789/0 conn(0x7f4da006ed00 legacy=0x7f4da00b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.191+0000 7f4da4c34700 1 -- 172.21.15.46:0/369168625 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da0157420 con 0x7f4da00afbd0 2024-01-27T00:46:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.191+0000 7f4da4c34700 1 --2- 172.21.15.46:0/369168625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4da00afbd0 0x7f4da01566b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f4da00cd170 tx=0x7f4d90009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303aa9067573e426 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.192+0000 7f4d9effd700 1 -- 172.21.15.46:0/369168625 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9000e070 con 0x7f4da00afbd0 2024-01-27T00:46:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.192+0000 7f4d9effd700 1 -- 172.21.15.46:0/369168625 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9000bb60 con 0x7f4da00afbd0 2024-01-27T00:46:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.192+0000 7f4d9effd700 1 -- 172.21.15.46:0/369168625 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d90012430 con 0x7f4da00afbd0 2024-01-27T00:46:22.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.193+0000 7f4da6697700 1 -- 172.21.15.46:0/369168625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4da00afbd0 msgr2=0x7f4da01566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:22.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.193+0000 7f4da6697700 1 --2- 172.21.15.46:0/369168625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4da00afbd0 0x7f4da01566b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f4da00cd170 tx=0x7f4d90009780 comp rx=0 tx=0).stop 2024-01-27T00:46:22.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.193+0000 7f4da6697700 1 -- 172.21.15.46:0/369168625 shutdown_connections 2024-01-27T00:46:22.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.193+0000 7f4da6697700 1 --2- 172.21.15.46:0/369168625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4da00afbd0 0x7f4da01566b0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.193+0000 7f4da6697700 1 --2- 172.21.15.46:0/369168625 >> v2:172.21.15.106:3300/0 conn(0x7f4da00b0840 0x7f4da0156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.193+0000 7f4da6697700 1 -- 172.21.15.46:0/369168625 >> 172.21.15.46:0/369168625 conn(0x7f4da0068400 msgr2=0x7f4da00cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:22.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.193+0000 7f4da6697700 1 -- 172.21.15.46:0/369168625 shutdown_connections 2024-01-27T00:46:22.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.193+0000 7f4da6697700 1 -- 172.21.15.46:0/369168625 wait complete. 2024-01-27T00:46:22.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.195+0000 7f4da6697700 1 Processor -- start 2024-01-27T00:46:22.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.195+0000 7f4da6697700 1 -- start start 2024-01-27T00:46:22.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.196+0000 7f4da6697700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4da006ed00 0x7f4da0144890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:22.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.196+0000 7f4da6697700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4da00afbd0 0x7f4da013df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:22.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.196+0000 7f4da6697700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4da00b0840 0x7f4da0146550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:22.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.196+0000 7f4da6697700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da0144400 con 0x7f4da00b0840 2024-01-27T00:46:22.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.196+0000 7f4da6697700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da01437f0 con 0x7f4da00afbd0 2024-01-27T00:46:22.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.196+0000 7f4da6697700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da0143ad0 con 0x7f4da006ed00 2024-01-27T00:46:22.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.196+0000 7f4da4c34700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4da00b0840 0x7f4da0146550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:22.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.196+0000 7f4d9ffff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4da006ed00 0x7f4da0144890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:22.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.196+0000 7f4da4c34700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4da00b0840 0x7f4da0146550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49274/0 (socket says 172.21.15.46:49274) 2024-01-27T00:46:22.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.196+0000 7f4da4c34700 1 -- 172.21.15.46:0/3276552933 learned_addr learned my addr 172.21.15.46:0/3276552933 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:22.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.196+0000 7f4d9f7fe700 1 --2- 172.21.15.46:0/3276552933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4da00afbd0 0x7f4da013df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:22.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.197+0000 7f4d9f7fe700 1 -- 172.21.15.46:0/3276552933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4da006ed00 msgr2=0x7f4da0144890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:22.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.197+0000 7f4d9f7fe700 1 --2- 172.21.15.46:0/3276552933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4da006ed00 0x7f4da0144890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.197+0000 7f4d9f7fe700 1 -- 172.21.15.46:0/3276552933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4da00b0840 msgr2=0x7f4da0146550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:22.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.197+0000 7f4d9f7fe700 1 --2- 172.21.15.46:0/3276552933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4da00b0840 0x7f4da0146550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.197+0000 7f4d9f7fe700 1 -- 172.21.15.46:0/3276552933 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d90009a20 con 0x7f4da00afbd0 2024-01-27T00:46:22.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.197+0000 7f4d9f7fe700 1 --2- 172.21.15.46:0/3276552933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4da00afbd0 0x7f4da013df50 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4d9400b5b0 tx=0x7f4d940090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:22.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.197+0000 7f4d87fff700 1 -- 172.21.15.46:0/3276552933 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9400f510 con 0x7f4da00afbd0 2024-01-27T00:46:22.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.197+0000 7f4d87fff700 1 -- 172.21.15.46:0/3276552933 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9400f670 con 0x7f4da00afbd0 2024-01-27T00:46:22.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.197+0000 7f4d87fff700 1 -- 172.21.15.46:0/3276552933 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9400fa30 con 0x7f4da00afbd0 2024-01-27T00:46:22.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.197+0000 7f4da6697700 1 -- 172.21.15.46:0/3276552933 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da0143db0 con 0x7f4da00afbd0 2024-01-27T00:46:22.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.197+0000 7f4da6697700 1 -- 172.21.15.46:0/3276552933 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da0144dd0 con 0x7f4da00afbd0 2024-01-27T00:46:22.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.199+0000 7f4da6697700 1 -- 172.21.15.46:0/3276552933 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d800052b0 con 0x7f4da00afbd0 2024-01-27T00:46:22.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.201+0000 7f4d87fff700 1 -- 172.21.15.46:0/3276552933 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 84) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f4d94024430 con 0x7f4da00afbd0 2024-01-27T00:46:22.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.201+0000 7f4d87fff700 1 --2- 172.21.15.46:0/3276552933 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4d8c067120 0x7f4d8c0695d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:22.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.201+0000 7f4d9ffff700 1 -- 172.21.15.46:0/3276552933 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4d8c067120 msgr2=0x7f4d8c0695d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:22.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.201+0000 7f4d87fff700 1 -- 172.21.15.46:0/3276552933 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6763+0+0 (secure 0 0 0) 0x7f4d94084ee0 con 0x7f4da00afbd0 2024-01-27T00:46:22.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.201+0000 7f4d9ffff700 1 --2- 172.21.15.46:0/3276552933 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4d8c067120 0x7f4d8c0695d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:46:22.201 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:22.202+0000 7ff6f08dc1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T00:46:22.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.205+0000 7f4d87fff700 1 -- 172.21.15.46:0/3276552933 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4d94051d30 con 0x7f4da00afbd0 2024-01-27T00:46:22.243 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:22.243+0000 7f7a7a4bf1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T00:46:22.331 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:22.332+0000 7ff6f08dc1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T00:46:22.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.402+0000 7f4d9ffff700 1 -- 172.21.15.46:0/3276552933 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4d8c067120 msgr2=0x7f4d8c0695d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:22.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.402+0000 7f4d9ffff700 1 --2- 172.21.15.46:0/3276552933 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4d8c067120 0x7f4d8c0695d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:46:22.406 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:22.407+0000 7ff6f08dc1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T00:46:22.428 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:22.428+0000 7f7a7a4bf1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:46:22.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.536+0000 7f4da6697700 1 -- 172.21.15.46:0/3276552933 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d80005830 con 0x7f4da00afbd0 2024-01-27T00:46:22.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.536+0000 7f4d87fff700 1 -- 172.21.15.46:0/3276552933 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f4d940091f0 con 0x7f4da00afbd0 2024-01-27T00:46:22.537 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:22.537 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8578,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1889696251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1889696251},{"type":"v1","addr":"172.21.15.106:6834","nonce":1889696251}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/631881132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":631881132},{"type":"v1","addr":"172.21.15.106:6836","nonce":631881132}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1045000694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1045000694},{"type":"v1","addr":"172.21.15.46:6834","nonce":1045000694}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:50:05.176064+0000","modified":"2024-01-26T23:50:08.757937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8871":{"gid":8871,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3419491709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3419491709},{"type":"v1","addr":"172.21.15.46:6835","nonce":3419491709}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2024-01-27T00:46:22.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.539+0000 7f4da6697700 1 -- 172.21.15.46:0/3276552933 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4d8c067120 msgr2=0x7f4d8c0695d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:46:22.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.539+0000 7f4da6697700 1 --2- 172.21.15.46:0/3276552933 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4d8c067120 0x7f4d8c0695d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.539+0000 7f4da6697700 1 -- 172.21.15.46:0/3276552933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4da00afbd0 msgr2=0x7f4da013df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:22.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.539+0000 7f4da6697700 1 --2- 172.21.15.46:0/3276552933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4da00afbd0 0x7f4da013df50 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4d9400b5b0 tx=0x7f4d940090f0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.539+0000 7f4da6697700 1 -- 172.21.15.46:0/3276552933 shutdown_connections 2024-01-27T00:46:22.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.539+0000 7f4da6697700 1 --2- 172.21.15.46:0/3276552933 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4d8c067120 0x7f4d8c0695d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.539+0000 7f4da6697700 1 --2- 172.21.15.46:0/3276552933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4da00b0840 0x7f4da0146550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.539+0000 7f4da6697700 1 --2- 172.21.15.46:0/3276552933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4da006ed00 0x7f4da0144890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.539+0000 7f4da6697700 1 --2- 172.21.15.46:0/3276552933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4da00afbd0 0x7f4da013df50 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.539+0000 7f4da6697700 1 -- 172.21.15.46:0/3276552933 >> 172.21.15.46:0/3276552933 conn(0x7f4da0068400 msgr2=0x7f4da0147710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:22.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.539+0000 7f4da6697700 1 -- 172.21.15.46:0/3276552933 shutdown_connections 2024-01-27T00:46:22.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.539+0000 7f4da6697700 1 -- 172.21.15.46:0/3276552933 wait complete. 2024-01-27T00:46:22.540 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 78 2024-01-27T00:46:22.557 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T00:46:22.616 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:22.616+0000 7ff6f08dc1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:46:22.621 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:22.621+0000 7f7a7a4bf1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T00:46:22.690 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:22.690+0000 7f7a7a4bf1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T00:46:22.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.698+0000 7fe07ddff700 1 Processor -- start 2024-01-27T00:46:22.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.698+0000 7fe07ddff700 1 -- start start 2024-01-27T00:46:22.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.699+0000 7fe07ddff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe0780b2b40 0x7fe0780ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:22.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.699+0000 7fe07ddff700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07815a0b0 con 0x7fe0780ae960 2024-01-27T00:46:22.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.699+0000 7fe07ddff700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0780ae4d0 con 0x7fe0780aedd0 2024-01-27T00:46:22.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.699+0000 7fe07ddff700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe078099fc0 con 0x7fe0780b2b40 2024-01-27T00:46:22.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.700+0000 7fe06effd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe0780ae960 0x7fe0780ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46104/0 (socket says 172.21.15.46:46104) 2024-01-27T00:46:22.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.700+0000 7fe06effd700 1 -- 172.21.15.46:0/3350089085 learned_addr learned my addr 172.21.15.46:0/3350089085 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:22.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.700+0000 7fe0777fe700 1 --2- 172.21.15.46:0/3350089085 >> v2:172.21.15.106:3300/0 conn(0x7fe0780b2b40 0x7fe0780ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:22.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.700+0000 7fe076ffd700 1 -- 172.21.15.46:0/3350089085 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900686791 0 0) 0x7fe07815a0b0 con 0x7fe0780ae960 2024-01-27T00:46:22.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.700+0000 7fe076ffd700 1 -- 172.21.15.46:0/3350089085 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe064003570 con 0x7fe0780ae960 2024-01-27T00:46:22.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.700+0000 7fe076ffd700 1 -- 172.21.15.46:0/3350089085 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542026764 0 0) 0x7fe0780ae4d0 con 0x7fe0780aedd0 2024-01-27T00:46:22.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.700+0000 7fe076ffd700 1 -- 172.21.15.46:0/3350089085 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe07815a0b0 con 0x7fe0780aedd0 2024-01-27T00:46:22.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.700+0000 7fe0777fe700 1 -- 172.21.15.46:0/3350089085 >> v1:172.21.15.106:6789/0 conn(0x7fe0780aedd0 legacy=0x7fe0780cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:22.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.700+0000 7fe0777fe700 1 -- 172.21.15.46:0/3350089085 >> v1:172.21.15.46:6789/0 conn(0x7fe0780ae960 legacy=0x7fe0780ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:22.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.700+0000 7fe0777fe700 1 -- 172.21.15.46:0/3350089085 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0780ad8c0 con 0x7fe0780b2b40 2024-01-27T00:46:22.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.700+0000 7fe0777fe700 1 --2- 172.21.15.46:0/3350089085 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe0780b2b40 0x7fe0780ad1c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe0680098d0 tx=0x7fe068009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dacc80b764fe8c77 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:22.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.700+0000 7fe076ffd700 1 -- 172.21.15.46:0/3350089085 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe068005c10 con 0x7fe0780b2b40 2024-01-27T00:46:22.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.700+0000 7fe076ffd700 1 -- 172.21.15.46:0/3350089085 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe068005d70 con 0x7fe0780b2b40 2024-01-27T00:46:22.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.700+0000 7fe076ffd700 1 -- 172.21.15.46:0/3350089085 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe068006280 con 0x7fe0780b2b40 2024-01-27T00:46:22.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.701+0000 7fe07ddff700 1 -- 172.21.15.46:0/3350089085 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe0780b2b40 msgr2=0x7fe0780ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:22.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.701+0000 7fe07ddff700 1 --2- 172.21.15.46:0/3350089085 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe0780b2b40 0x7fe0780ad1c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe0680098d0 tx=0x7fe068009120 comp rx=0 tx=0).stop 2024-01-27T00:46:22.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.701+0000 7fe07ddff700 1 -- 172.21.15.46:0/3350089085 shutdown_connections 2024-01-27T00:46:22.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.701+0000 7fe07ddff700 1 --2- 172.21.15.46:0/3350089085 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe0780b2b40 0x7fe0780ad1c0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.701+0000 7fe07ddff700 1 -- 172.21.15.46:0/3350089085 >> 172.21.15.46:0/3350089085 conn(0x7fe078099a30 msgr2=0x7fe07806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:22.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.701+0000 7fe07ddff700 1 -- 172.21.15.46:0/3350089085 shutdown_connections 2024-01-27T00:46:22.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.701+0000 7fe07ddff700 1 -- 172.21.15.46:0/3350089085 wait complete. 2024-01-27T00:46:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.702+0000 7fe07ddff700 1 Processor -- start 2024-01-27T00:46:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.703+0000 7fe07ddff700 1 -- start start 2024-01-27T00:46:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.703+0000 7fe07ddff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0780ae960 0x7fe0780b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.703+0000 7fe07ddff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe0780b2b40 0x7fe07814f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.703+0000 7fe07ddff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe0780afc00 0x7fe0780b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.703+0000 7fe07ddff700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0780b09d0 con 0x7fe0780ae960 2024-01-27T00:46:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.703+0000 7fe07ddff700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0780c8fe0 con 0x7fe0780b2b40 2024-01-27T00:46:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.703+0000 7fe07ddff700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0780b06f0 con 0x7fe0780afc00 2024-01-27T00:46:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.704+0000 7fe0777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0780ae960 0x7fe0780b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.704+0000 7fe0777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0780ae960 0x7fe0780b3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49286/0 (socket says 172.21.15.46:49286) 2024-01-27T00:46:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.704+0000 7fe0777fe700 1 -- 172.21.15.46:0/2452062362 learned_addr learned my addr 172.21.15.46:0/2452062362 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.704+0000 7fe06effd700 1 --2- 172.21.15.46:0/2452062362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe0780b2b40 0x7fe07814f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.704+0000 7fe077fff700 1 --2- 172.21.15.46:0/2452062362 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe0780afc00 0x7fe0780b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:22.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.704+0000 7fe06effd700 1 -- 172.21.15.46:0/2452062362 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe0780afc00 msgr2=0x7fe0780b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:22.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.704+0000 7fe06effd700 1 --2- 172.21.15.46:0/2452062362 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe0780afc00 0x7fe0780b00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.704+0000 7fe06effd700 1 -- 172.21.15.46:0/2452062362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0780ae960 msgr2=0x7fe0780b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:22.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.704+0000 7fe06effd700 1 --2- 172.21.15.46:0/2452062362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0780ae960 0x7fe0780b3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.704+0000 7fe06effd700 1 -- 172.21.15.46:0/2452062362 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0680093c0 con 0x7fe0780b2b40 2024-01-27T00:46:22.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.704+0000 7fe0777fe700 1 --2- 172.21.15.46:0/2452062362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0780ae960 0x7fe0780b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:22.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.704+0000 7fe06effd700 1 --2- 172.21.15.46:0/2452062362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe0780b2b40 0x7fe07814f720 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fe06000c580 tx=0x7fe060007d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:22.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.704+0000 7fe06ffff700 1 -- 172.21.15.46:0/2452062362 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe060005b30 con 0x7fe0780b2b40 2024-01-27T00:46:22.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.704+0000 7fe06ffff700 1 -- 172.21.15.46:0/2452062362 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe060005c90 con 0x7fe0780b2b40 2024-01-27T00:46:22.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.705+0000 7fe07ddff700 1 -- 172.21.15.46:0/2452062362 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0780c9210 con 0x7fe0780b2b40 2024-01-27T00:46:22.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.705+0000 7fe06ffff700 1 -- 172.21.15.46:0/2452062362 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe060007320 con 0x7fe0780b2b40 2024-01-27T00:46:22.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.705+0000 7fe07ddff700 1 -- 172.21.15.46:0/2452062362 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0780c97d0 con 0x7fe0780b2b40 2024-01-27T00:46:22.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.706+0000 7fe07ddff700 1 -- 172.21.15.46:0/2452062362 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe07804c710 con 0x7fe0780b2b40 2024-01-27T00:46:22.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.706+0000 7fe06ffff700 1 -- 172.21.15.46:0/2452062362 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 84) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fe06001b3b0 con 0x7fe0780b2b40 2024-01-27T00:46:22.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.706+0000 7fe06ffff700 1 --2- 172.21.15.46:0/2452062362 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe058067050 0x7fe058069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:22.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.706+0000 7fe0777fe700 1 -- 172.21.15.46:0/2452062362 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe058067050 msgr2=0x7fe058069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:22.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.707+0000 7fe06ffff700 1 -- 172.21.15.46:0/2452062362 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6807+0+0 (secure 0 0 0) 0x7fe060083f10 con 0x7fe0780b2b40 2024-01-27T00:46:22.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.707+0000 7fe0777fe700 1 --2- 172.21.15.46:0/2452062362 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe058067050 0x7fe058069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:46:22.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.711+0000 7fe06ffff700 1 -- 172.21.15.46:0/2452062362 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe060050c20 con 0x7fe0780b2b40 2024-01-27T00:46:22.783 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:22.784+0000 7ff6f08dc1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T00:46:22.811 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:22.812+0000 7f7a7a4bf1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T00:46:22.859 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:22.860+0000 7ff6f08dc1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T00:46:22.886 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:22.886+0000 7f7a7a4bf1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T00:46:22.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.908+0000 7fe0777fe700 1 -- 172.21.15.46:0/2452062362 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe058067050 msgr2=0x7fe058069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:22.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.908+0000 7fe0777fe700 1 --2- 172.21.15.46:0/2452062362 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe058067050 0x7fe058069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:46:22.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.923+0000 7fe07ddff700 1 -- 172.21.15.46:0/2452062362 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0780682c0 con 0x7fe0780b2b40 2024-01-27T00:46:22.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.924+0000 7fe06ffff700 1 -- 172.21.15.46:0/2452062362 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+16769 (secure 0 0 0) 0x7fe060017020 con 0x7fe0780b2b40 2024-01-27T00:46:22.924 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:22.924 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":162,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T00:46:22.452559+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:50:03.691049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-01-26T23:50:04.138930+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":162,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/2632355058":"2024-01-27T23:50:19.809740+0000","172.21.15.106:0/156485008":"2024-01-27T23:50:19.809740+0000","172.21.15.106:6837/43388":"2024-01-27T23:50:19.809740+0000","172.21.15.106:6835/43388":"2024-01-27T23:50:19.809740+0000","172.21.15.106:0/2132373362":"2024-01-27T23:50:19.809740+0000","172.21.15.106:0/1443701093":"2024-01-27T23:50:19.809740+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[{"pool":20,"snaps":[{"begin":2,"length":1},{"begin":58,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T00:46:22.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.928+0000 7fe07ddff700 1 -- 172.21.15.46:0/2452062362 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe058067050 msgr2=0x7fe058069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:46:22.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.928+0000 7fe07ddff700 1 --2- 172.21.15.46:0/2452062362 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe058067050 0x7fe058069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.928+0000 7fe07ddff700 1 -- 172.21.15.46:0/2452062362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe0780b2b40 msgr2=0x7fe07814f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:22.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.928+0000 7fe07ddff700 1 --2- 172.21.15.46:0/2452062362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe0780b2b40 0x7fe07814f720 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fe06000c580 tx=0x7fe060007d70 comp rx=0 tx=0).stop 2024-01-27T00:46:22.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.928+0000 7fe07ddff700 1 -- 172.21.15.46:0/2452062362 shutdown_connections 2024-01-27T00:46:22.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.928+0000 7fe07ddff700 1 --2- 172.21.15.46:0/2452062362 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe058067050 0x7fe058069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.928+0000 7fe07ddff700 1 --2- 172.21.15.46:0/2452062362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0780ae960 0x7fe0780b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.928+0000 7fe07ddff700 1 --2- 172.21.15.46:0/2452062362 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe0780afc00 0x7fe0780b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.928+0000 7fe07ddff700 1 --2- 172.21.15.46:0/2452062362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe0780b2b40 0x7fe07814f720 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:22.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.928+0000 7fe07ddff700 1 -- 172.21.15.46:0/2452062362 >> 172.21.15.46:0/2452062362 conn(0x7fe078099a30 msgr2=0x7fe0780c5da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:22.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.928+0000 7fe07ddff700 1 -- 172.21.15.46:0/2452062362 shutdown_connections 2024-01-27T00:46:22.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:22.928+0000 7fe07ddff700 1 -- 172.21.15.46:0/2452062362 wait complete. 2024-01-27T00:46:22.941 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-26T23:50:03.691049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '157', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2024-01-26T23:50:04.138930+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '162', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 113, 'snap_epoch': 162, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-27T00:46:22.942 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-27T00:46:23.006 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:23.007+0000 7ff6f08dc1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T00:46:23.026 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:23.026+0000 7f7a7a4bf1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T00:46:23.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.079+0000 7f69ed8e8700 1 Processor -- start 2024-01-27T00:46:23.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.079+0000 7f69ed8e8700 1 -- start start 2024-01-27T00:46:23.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.080+0000 7f69ed8e8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69e81599d0 0x7f69e8159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:23.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.080+0000 7f69ed8e8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e80b1a60 con 0x7f69e815d8c0 2024-01-27T00:46:23.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.080+0000 7f69ed8e8700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e80ab5f0 con 0x7f69e80ab1c0 2024-01-27T00:46:23.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.080+0000 7f69ed8e8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e80b11c0 con 0x7f69e81599d0 2024-01-27T00:46:23.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.080+0000 7f69e77fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f69e815d8c0 0x7f69e80b3b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46116/0 (socket says 172.21.15.46:46116) 2024-01-27T00:46:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.080+0000 7f69e77fe700 1 -- 172.21.15.46:0/3808287636 learned_addr learned my addr 172.21.15.46:0/3808287636 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.080+0000 7f69e6ffd700 1 --2- 172.21.15.46:0/3808287636 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69e81599d0 0x7f69e8159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.080+0000 7f69e5ffb700 1 -- 172.21.15.46:0/3808287636 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942878928 0 0) 0x7f69e80b1a60 con 0x7f69e815d8c0 2024-01-27T00:46:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.081+0000 7f69e5ffb700 1 -- 172.21.15.46:0/3808287636 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69d4003570 con 0x7f69e815d8c0 2024-01-27T00:46:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.081+0000 7f69e5ffb700 1 -- 172.21.15.46:0/3808287636 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632434766 0 0) 0x7f69e80ab5f0 con 0x7f69e80ab1c0 2024-01-27T00:46:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.081+0000 7f69e5ffb700 1 -- 172.21.15.46:0/3808287636 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e80b1a60 con 0x7f69e80ab1c0 2024-01-27T00:46:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.081+0000 7f69e6ffd700 1 -- 172.21.15.46:0/3808287636 >> v1:172.21.15.106:6789/0 conn(0x7f69e80ab1c0 legacy=0x7f69e80b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.081+0000 7f69e6ffd700 1 -- 172.21.15.46:0/3808287636 >> v1:172.21.15.46:6789/0 conn(0x7f69e815d8c0 legacy=0x7f69e80b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.081+0000 7f69e6ffd700 1 -- 172.21.15.46:0/3808287636 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e80b14a0 con 0x7f69e81599d0 2024-01-27T00:46:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.081+0000 7f69e6ffd700 1 --2- 172.21.15.46:0/3808287636 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69e81599d0 0x7f69e8159da0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f69d8009900 tx=0x7f69d8009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be81b6c272072c53 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.081+0000 7f69e5ffb700 1 -- 172.21.15.46:0/3808287636 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d8005c10 con 0x7f69e81599d0 2024-01-27T00:46:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.081+0000 7f69e5ffb700 1 -- 172.21.15.46:0/3808287636 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d8005d70 con 0x7f69e81599d0 2024-01-27T00:46:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.081+0000 7f69e5ffb700 1 -- 172.21.15.46:0/3808287636 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d8006260 con 0x7f69e81599d0 2024-01-27T00:46:23.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.082+0000 7f69ed8e8700 1 -- 172.21.15.46:0/3808287636 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69e81599d0 msgr2=0x7f69e8159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:23.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.082+0000 7f69ed8e8700 1 --2- 172.21.15.46:0/3808287636 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69e81599d0 0x7f69e8159da0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f69d8009900 tx=0x7f69d8009120 comp rx=0 tx=0).stop 2024-01-27T00:46:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.082+0000 7f69ed8e8700 1 -- 172.21.15.46:0/3808287636 shutdown_connections 2024-01-27T00:46:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.082+0000 7f69ed8e8700 1 --2- 172.21.15.46:0/3808287636 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69e81599d0 0x7f69e8159da0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.082+0000 7f69ed8e8700 1 -- 172.21.15.46:0/3808287636 >> 172.21.15.46:0/3808287636 conn(0x7f69e806f840 msgr2=0x7f69e8087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.082+0000 7f69ed8e8700 1 -- 172.21.15.46:0/3808287636 shutdown_connections 2024-01-27T00:46:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.082+0000 7f69ed8e8700 1 -- 172.21.15.46:0/3808287636 wait complete. 2024-01-27T00:46:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.083+0000 7f69ed8e8700 1 Processor -- start 2024-01-27T00:46:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.083+0000 7f69ed8e8700 1 -- start start 2024-01-27T00:46:23.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.084+0000 7f69ed8e8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69e80ab1c0 0x7f69e8155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:23.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.084+0000 7f69ed8e8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69e81599d0 0x7f69e8153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:23.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.084+0000 7f69ed8e8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e815d8c0 0x7f69e815c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:23.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.084+0000 7f69ed8e8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e8156620 con 0x7f69e815d8c0 2024-01-27T00:46:23.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.084+0000 7f69ed8e8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e8155770 con 0x7f69e81599d0 2024-01-27T00:46:23.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.084+0000 7f69ed8e8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e8155a10 con 0x7f69e80ab1c0 2024-01-27T00:46:23.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.084+0000 7f69e77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e815d8c0 0x7f69e815c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:23.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.084+0000 7f69e77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e815d8c0 0x7f69e815c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49296/0 (socket says 172.21.15.46:49296) 2024-01-27T00:46:23.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.084+0000 7f69e77fe700 1 -- 172.21.15.46:0/2552757048 learned_addr learned my addr 172.21.15.46:0/2552757048 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.084+0000 7f69e6ffd700 1 --2- 172.21.15.46:0/2552757048 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69e80ab1c0 0x7f69e8155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.084+0000 7f69e67fc700 1 --2- 172.21.15.46:0/2552757048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69e81599d0 0x7f69e8153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.085+0000 7f69e77fe700 1 -- 172.21.15.46:0/2552757048 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69e80ab1c0 msgr2=0x7f69e8155230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.085+0000 7f69e77fe700 1 --2- 172.21.15.46:0/2552757048 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69e80ab1c0 0x7f69e8155230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.085+0000 7f69e77fe700 1 -- 172.21.15.46:0/2552757048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69e81599d0 msgr2=0x7f69e8153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.085+0000 7f69e77fe700 1 --2- 172.21.15.46:0/2552757048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69e81599d0 0x7f69e8153320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.085+0000 7f69e77fe700 1 -- 172.21.15.46:0/2552757048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d80093c0 con 0x7f69e815d8c0 2024-01-27T00:46:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.085+0000 7f69e77fe700 1 --2- 172.21.15.46:0/2552757048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e815d8c0 0x7f69e815c980 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f69e000cbc0 tx=0x7f69e0001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.085+0000 7f69deffd700 1 -- 172.21.15.46:0/2552757048 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e000a7e0 con 0x7f69e815d8c0 2024-01-27T00:46:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.085+0000 7f69deffd700 1 -- 172.21.15.46:0/2552757048 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e000a940 con 0x7f69e815d8c0 2024-01-27T00:46:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.085+0000 7f69ed8e8700 1 -- 172.21.15.46:0/2552757048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e8155cf0 con 0x7f69e815d8c0 2024-01-27T00:46:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.085+0000 7f69deffd700 1 -- 172.21.15.46:0/2552757048 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e000ad00 con 0x7f69e815d8c0 2024-01-27T00:46:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.085+0000 7f69ed8e8700 1 -- 172.21.15.46:0/2552757048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e81562b0 con 0x7f69e815d8c0 2024-01-27T00:46:23.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.086+0000 7f69ed8e8700 1 -- 172.21.15.46:0/2552757048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e81567d0 con 0x7f69e815d8c0 2024-01-27T00:46:23.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.087+0000 7f69deffd700 1 -- 172.21.15.46:0/2552757048 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 84) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f69e0007300 con 0x7f69e815d8c0 2024-01-27T00:46:23.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.087+0000 7f69deffd700 1 --2- 172.21.15.46:0/2552757048 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f69c8067050 0x7f69c8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:23.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.087+0000 7f69deffd700 1 -- 172.21.15.46:0/2552757048 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6807+0+0 (secure 0 0 0) 0x7f69e0084d90 con 0x7f69e815d8c0 2024-01-27T00:46:23.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.087+0000 7f69e6ffd700 1 -- 172.21.15.46:0/2552757048 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f69c8067050 msgr2=0x7f69c8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:23.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.087+0000 7f69e6ffd700 1 --2- 172.21.15.46:0/2552757048 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f69c8067050 0x7f69c8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:46:23.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.092+0000 7f69deffd700 1 -- 172.21.15.46:0/2552757048 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f69e81567d0 con 0x7f69e815d8c0 2024-01-27T00:46:23.092 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:23.093+0000 7ff6f08dc1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T00:46:23.153 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:23.153+0000 7f7a7a4bf1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T00:46:23.268 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:23.269+0000 7ff6f08dc1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T00:46:23.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.288+0000 7f69e6ffd700 1 -- 172.21.15.46:0/2552757048 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f69c8067050 msgr2=0x7f69c8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:23.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.288+0000 7f69e6ffd700 1 --2- 172.21.15.46:0/2552757048 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f69c8067050 0x7f69c8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:46:23.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.339+0000 7f69ed8e8700 1 -- 172.21.15.46:0/2552757048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f69e80b4d80 con 0x7f69e815d8c0 2024-01-27T00:46:23.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.340+0000 7f69deffd700 1 -- 172.21.15.46:0/2552757048 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v162) v1 ==== 93+0+10 (secure 0 0 0) 0x7f69e0051b00 con 0x7f69e815d8c0 2024-01-27T00:46:23.339 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-27T00:46:23.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.342+0000 7f69ed8e8700 1 -- 172.21.15.46:0/2552757048 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f69c8067050 msgr2=0x7f69c8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:46:23.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.342+0000 7f69ed8e8700 1 --2- 172.21.15.46:0/2552757048 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f69c8067050 0x7f69c8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.342+0000 7f69ed8e8700 1 -- 172.21.15.46:0/2552757048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e815d8c0 msgr2=0x7f69e815c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:23.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.342+0000 7f69ed8e8700 1 --2- 172.21.15.46:0/2552757048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e815d8c0 0x7f69e815c980 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f69e000cbc0 tx=0x7f69e0001a20 comp rx=0 tx=0).stop 2024-01-27T00:46:23.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.342+0000 7f69ed8e8700 1 -- 172.21.15.46:0/2552757048 shutdown_connections 2024-01-27T00:46:23.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.342+0000 7f69ed8e8700 1 --2- 172.21.15.46:0/2552757048 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f69c8067050 0x7f69c8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.342+0000 7f69ed8e8700 1 --2- 172.21.15.46:0/2552757048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e815d8c0 0x7f69e815c980 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.342+0000 7f69ed8e8700 1 --2- 172.21.15.46:0/2552757048 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69e80ab1c0 0x7f69e8155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.342+0000 7f69ed8e8700 1 --2- 172.21.15.46:0/2552757048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69e81599d0 0x7f69e8153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.342+0000 7f69ed8e8700 1 -- 172.21.15.46:0/2552757048 >> 172.21.15.46:0/2552757048 conn(0x7f69e806f840 msgr2=0x7f69e815a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:23.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.342+0000 7f69ed8e8700 1 -- 172.21.15.46:0/2552757048 shutdown_connections 2024-01-27T00:46:23.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.342+0000 7f69ed8e8700 1 -- 172.21.15.46:0/2552757048 wait complete. 2024-01-27T00:46:23.355 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-27T00:46:23.426 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:23.427+0000 7ff6f08dc1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T00:46:23.463 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:23.463+0000 7f7a7a4bf1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T00:46:23.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.505+0000 7f65393e0700 1 Processor -- start 2024-01-27T00:46:23.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.505+0000 7f65393e0700 1 -- start start 2024-01-27T00:46:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.507+0000 7f65393e0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f65340af7c0 0x7f65340cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.507+0000 7f65393e0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65340b30d0 0x7f65341579b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.507+0000 7f65393e0700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65340ab470 con 0x7f65341575e0 2024-01-27T00:46:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.507+0000 7f65393e0700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65340b2900 con 0x7f65340af7c0 2024-01-27T00:46:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.507+0000 7f65393e0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65340b2340 con 0x7f65340b30d0 2024-01-27T00:46:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.507+0000 7f65327fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f65340af7c0 0x7f65340cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.507+0000 7f65327fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f65340af7c0 0x7f65340cdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49308/0 (socket says 172.21.15.46:49308) 2024-01-27T00:46:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.507+0000 7f65327fc700 1 -- 172.21.15.46:0/2771233863 learned_addr learned my addr 172.21.15.46:0/2771233863 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.507+0000 7f65337fe700 1 --2- 172.21.15.46:0/2771233863 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65340b30d0 0x7f65341579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.507+0000 7f65327fc700 1 -- 172.21.15.46:0/2771233863 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65340b30d0 msgr2=0x7f65341579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:23.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.507+0000 7f65327fc700 1 --2- 172.21.15.46:0/2771233863 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65340b30d0 0x7f65341579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.507+0000 7f6531ffb700 1 -- 172.21.15.46:0/2771233863 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663138429 0 0) 0x7f65340ab470 con 0x7f65341575e0 2024-01-27T00:46:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.507+0000 7f65327fc700 1 -- 172.21.15.46:0/2771233863 >> v1:172.21.15.106:6789/0 conn(0x7f65341575e0 legacy=0x7f65340af0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.507+0000 7f65327fc700 1 -- 172.21.15.46:0/2771233863 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65340b2620 con 0x7f65340af7c0 2024-01-27T00:46:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.507+0000 7f65327fc700 1 --2- 172.21.15.46:0/2771233863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65340af7c0 0x7f65340cdbf0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f651c0099b0 tx=0x7f651c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ca6755a1655363 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.508+0000 7f6531ffb700 1 -- 172.21.15.46:0/2771233863 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651c009ea0 con 0x7f65340af7c0 2024-01-27T00:46:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.508+0000 7f6531ffb700 1 -- 172.21.15.46:0/2771233863 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f651c00b860 con 0x7f65340af7c0 2024-01-27T00:46:23.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.508+0000 7f6531ffb700 1 -- 172.21.15.46:0/2771233863 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651c00bc20 con 0x7f65340af7c0 2024-01-27T00:46:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.508+0000 7f65393e0700 1 -- 172.21.15.46:0/2771233863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65340af7c0 msgr2=0x7f65340cdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.508+0000 7f65393e0700 1 --2- 172.21.15.46:0/2771233863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65340af7c0 0x7f65340cdbf0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f651c0099b0 tx=0x7f651c009250 comp rx=0 tx=0).stop 2024-01-27T00:46:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.508+0000 7f65393e0700 1 -- 172.21.15.46:0/2771233863 shutdown_connections 2024-01-27T00:46:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.508+0000 7f65393e0700 1 --2- 172.21.15.46:0/2771233863 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65340b30d0 0x7f65341579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.508+0000 7f65393e0700 1 --2- 172.21.15.46:0/2771233863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65340af7c0 0x7f65340cdbf0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.508+0000 7f65393e0700 1 -- 172.21.15.46:0/2771233863 >> 172.21.15.46:0/2771233863 conn(0x7f6534069220 msgr2=0x7f65340cd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.508+0000 7f65393e0700 1 -- 172.21.15.46:0/2771233863 shutdown_connections 2024-01-27T00:46:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.508+0000 7f65393e0700 1 -- 172.21.15.46:0/2771233863 wait complete. 2024-01-27T00:46:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.510+0000 7f65393e0700 1 Processor -- start 2024-01-27T00:46:23.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.510+0000 7f65393e0700 1 -- start start 2024-01-27T00:46:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f65393e0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65340af7c0 0x7f6534150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f65393e0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65340b30d0 0x7f6534155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f65393e0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65341575e0 0x7f65341564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f65393e0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653416ab30 con 0x7f65340af7c0 2024-01-27T00:46:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f65393e0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653415aec0 con 0x7f65341575e0 2024-01-27T00:46:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f65393e0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f653414fde0 con 0x7f65340b30d0 2024-01-27T00:46:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f6532ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65340af7c0 0x7f6534150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:23.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f6532ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65340af7c0 0x7f6534150270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49310/0 (socket says 172.21.15.46:49310) 2024-01-27T00:46:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f6532ffd700 1 -- 172.21.15.46:0/899131925 learned_addr learned my addr 172.21.15.46:0/899131925 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f65327fc700 1 --2- 172.21.15.46:0/899131925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65340b30d0 0x7f6534155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f65337fe700 1 --2- 172.21.15.46:0/899131925 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65341575e0 0x7f65341564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f6532ffd700 1 -- 172.21.15.46:0/899131925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65340b30d0 msgr2=0x7f6534155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f6532ffd700 1 --2- 172.21.15.46:0/899131925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65340b30d0 0x7f6534155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f6532ffd700 1 -- 172.21.15.46:0/899131925 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65341575e0 msgr2=0x7f65341564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f6532ffd700 1 --2- 172.21.15.46:0/899131925 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65341575e0 0x7f65341564f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f6532ffd700 1 -- 172.21.15.46:0/899131925 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651c0094f0 con 0x7f65340af7c0 2024-01-27T00:46:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f65327fc700 1 --2- 172.21.15.46:0/899131925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65340b30d0 0x7f6534155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.511+0000 7f6532ffd700 1 --2- 172.21.15.46:0/899131925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65340af7c0 0x7f6534150270 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f652400cb60 tx=0x7f65240017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.512+0000 7f652affd700 1 -- 172.21.15.46:0/899131925 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65240020b0 con 0x7f65340af7c0 2024-01-27T00:46:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.512+0000 7f652affd700 1 -- 172.21.15.46:0/899131925 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f652400a7e0 con 0x7f65340af7c0 2024-01-27T00:46:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.512+0000 7f652affd700 1 -- 172.21.15.46:0/899131925 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652400aba0 con 0x7f65340af7c0 2024-01-27T00:46:23.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.512+0000 7f65393e0700 1 -- 172.21.15.46:0/899131925 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f653416a570 con 0x7f65340af7c0 2024-01-27T00:46:23.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.512+0000 7f65393e0700 1 -- 172.21.15.46:0/899131925 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f653414fa70 con 0x7f65340af7c0 2024-01-27T00:46:23.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.513+0000 7f65393e0700 1 -- 172.21.15.46:0/899131925 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f653404c710 con 0x7f65340af7c0 2024-01-27T00:46:23.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.513+0000 7f652affd700 1 -- 172.21.15.46:0/899131925 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 84) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f6524003030 con 0x7f65340af7c0 2024-01-27T00:46:23.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.514+0000 7f652affd700 1 --2- 172.21.15.46:0/899131925 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6514067050 0x7f6514069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:23.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.514+0000 7f652affd700 1 -- 172.21.15.46:0/899131925 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6807+0+0 (secure 0 0 0) 0x7f6524084b60 con 0x7f65340af7c0 2024-01-27T00:46:23.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.514+0000 7f65327fc700 1 -- 172.21.15.46:0/899131925 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6514067050 msgr2=0x7f6514069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:23.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.514+0000 7f65327fc700 1 --2- 172.21.15.46:0/899131925 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6514067050 0x7f6514069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:46:23.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.518+0000 7f652affd700 1 -- 172.21.15.46:0/899131925 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6524051870 con 0x7f65340af7c0 2024-01-27T00:46:23.677 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:23.677+0000 7f7a7a4bf1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T00:46:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.715+0000 7f65327fc700 1 -- 172.21.15.46:0/899131925 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6514067050 msgr2=0x7f6514069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.715+0000 7f65327fc700 1 --2- 172.21.15.46:0/899131925 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6514067050 0x7f6514069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:46:23.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.724+0000 7f65393e0700 1 -- 172.21.15.46:0/899131925 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f65340ab100 con 0x7f65340af7c0 2024-01-27T00:46:23.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.724+0000 7f652affd700 1 -- 172.21.15.46:0/899131925 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v162) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6524054e90 con 0x7f65340af7c0 2024-01-27T00:46:23.724 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-27T00:46:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.726+0000 7f65393e0700 1 -- 172.21.15.46:0/899131925 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6514067050 msgr2=0x7f6514069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:46:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.726+0000 7f65393e0700 1 --2- 172.21.15.46:0/899131925 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6514067050 0x7f6514069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.726+0000 7f65393e0700 1 -- 172.21.15.46:0/899131925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65340af7c0 msgr2=0x7f6534150270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.726+0000 7f65393e0700 1 --2- 172.21.15.46:0/899131925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65340af7c0 0x7f6534150270 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f652400cb60 tx=0x7f65240017a0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.726+0000 7f65393e0700 1 -- 172.21.15.46:0/899131925 shutdown_connections 2024-01-27T00:46:23.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.726+0000 7f65393e0700 1 --2- 172.21.15.46:0/899131925 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6514067050 0x7f6514069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.726+0000 7f65393e0700 1 --2- 172.21.15.46:0/899131925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65340af7c0 0x7f6534150270 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.726+0000 7f65393e0700 1 --2- 172.21.15.46:0/899131925 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65341575e0 0x7f65341564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.726+0000 7f65393e0700 1 --2- 172.21.15.46:0/899131925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65340b30d0 0x7f6534155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.726+0000 7f65393e0700 1 -- 172.21.15.46:0/899131925 >> 172.21.15.46:0/899131925 conn(0x7f6534069220 msgr2=0x7f65340681c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:23.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.727+0000 7f65393e0700 1 -- 172.21.15.46:0/899131925 shutdown_connections 2024-01-27T00:46:23.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.727+0000 7f65393e0700 1 -- 172.21.15.46:0/899131925 wait complete. 2024-01-27T00:46:23.740 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-27T00:46:23.797 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:23.798+0000 7ff6f08dc1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T00:46:23.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.877+0000 7f5a6feb1700 1 Processor -- start 2024-01-27T00:46:23.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.877+0000 7f5a6feb1700 1 -- start start 2024-01-27T00:46:23.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.878+0000 7f5a6feb1700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5a68159510 0x7f5a681598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:23.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.878+0000 7f5a6feb1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a680ca220 0x7f5a680c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:23.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.878+0000 7f5a6feb1700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a68069260 con 0x7f5a680ca690 2024-01-27T00:46:23.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.878+0000 7f5a6feb1700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a6815d670 con 0x7f5a680ca220 2024-01-27T00:46:23.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.878+0000 7f5a6feb1700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a680c9750 con 0x7f5a68159510 2024-01-27T00:46:23.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.878+0000 7f5a6d44c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a680ca220 0x7f5a680c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:23.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.878+0000 7f5a6d44c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a680ca220 0x7f5a680c8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49318/0 (socket says 172.21.15.46:49318) 2024-01-27T00:46:23.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.878+0000 7f5a6d44c700 1 -- 172.21.15.46:0/1470365238 learned_addr learned my addr 172.21.15.46:0/1470365238 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:23.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.878+0000 7f5a6dc4d700 1 --2- 172.21.15.46:0/1470365238 >> v2:172.21.15.106:3300/0 conn(0x7f5a68159510 0x7f5a681598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:23.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.879+0000 7f5a6d44c700 1 -- 172.21.15.46:0/1470365238 >> v2:172.21.15.106:3300/0 conn(0x7f5a68159510 msgr2=0x7f5a681598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:23.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.879+0000 7f5a6d44c700 1 --2- 172.21.15.46:0/1470365238 >> v2:172.21.15.106:3300/0 conn(0x7f5a68159510 0x7f5a681598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.879+0000 7f5a6d44c700 1 -- 172.21.15.46:0/1470365238 >> v1:172.21.15.106:6789/0 conn(0x7f5a680ca690 legacy=0x7f5a680cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:23.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.879+0000 7f5a6d44c700 1 -- 172.21.15.46:0/1470365238 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a680c9a30 con 0x7f5a680ca220 2024-01-27T00:46:23.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.879+0000 7f5a6d44c700 1 --2- 172.21.15.46:0/1470365238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a680ca220 0x7f5a680c8be0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f5a6806f760 tx=0x7f5a58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b844a8dd6644f2bb server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:23.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.879+0000 7f5a6cc4b700 1 -- 172.21.15.46:0/1470365238 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a58009e70 con 0x7f5a680ca220 2024-01-27T00:46:23.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.879+0000 7f5a6cc4b700 1 -- 172.21.15.46:0/1470365238 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a5800b860 con 0x7f5a680ca220 2024-01-27T00:46:23.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.879+0000 7f5a6cc4b700 1 -- 172.21.15.46:0/1470365238 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5800bc00 con 0x7f5a680ca220 2024-01-27T00:46:23.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.880+0000 7f5a6feb1700 1 -- 172.21.15.46:0/1470365238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a680ca220 msgr2=0x7f5a680c8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:23.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.880+0000 7f5a6feb1700 1 --2- 172.21.15.46:0/1470365238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a680ca220 0x7f5a680c8be0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f5a6806f760 tx=0x7f5a58009250 comp rx=0 tx=0).stop 2024-01-27T00:46:23.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.880+0000 7f5a6feb1700 1 -- 172.21.15.46:0/1470365238 shutdown_connections 2024-01-27T00:46:23.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.880+0000 7f5a6feb1700 1 --2- 172.21.15.46:0/1470365238 >> v2:172.21.15.106:3300/0 conn(0x7f5a68159510 0x7f5a681598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.880+0000 7f5a6feb1700 1 --2- 172.21.15.46:0/1470365238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a680ca220 0x7f5a680c8be0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.880+0000 7f5a6feb1700 1 -- 172.21.15.46:0/1470365238 >> 172.21.15.46:0/1470365238 conn(0x7f5a6806bd90 msgr2=0x7f5a6806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:23.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.880+0000 7f5a6feb1700 1 -- 172.21.15.46:0/1470365238 shutdown_connections 2024-01-27T00:46:23.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.880+0000 7f5a6feb1700 1 -- 172.21.15.46:0/1470365238 wait complete. 2024-01-27T00:46:23.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.881+0000 7f5a6feb1700 1 Processor -- start 2024-01-27T00:46:23.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.881+0000 7f5a6feb1700 1 -- start start 2024-01-27T00:46:23.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.882+0000 7f5a6feb1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a680ca220 0x7f5a6814d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:23.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.882+0000 7f5a6feb1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a68159510 0x7f5a680caa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:23.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.882+0000 7f5a6feb1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a680cafb0 0x7f5a6814f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:23.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.882+0000 7f5a6feb1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6814d5d0 con 0x7f5a680cafb0 2024-01-27T00:46:23.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.882+0000 7f5a6feb1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a680c6980 con 0x7f5a68159510 2024-01-27T00:46:23.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.882+0000 7f5a6feb1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a680cb400 con 0x7f5a680ca220 2024-01-27T00:46:23.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.882+0000 7f5a6e44e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a680cafb0 0x7f5a6814f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:23.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.882+0000 7f5a6e44e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a680cafb0 0x7f5a6814f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49324/0 (socket says 172.21.15.46:49324) 2024-01-27T00:46:23.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.882+0000 7f5a6e44e700 1 -- 172.21.15.46:0/1391057001 learned_addr learned my addr 172.21.15.46:0/1391057001 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:23.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.882+0000 7f5a6d44c700 1 --2- 172.21.15.46:0/1391057001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a68159510 0x7f5a680caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:23.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.882+0000 7f5a6dc4d700 1 --2- 172.21.15.46:0/1391057001 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a680ca220 0x7f5a6814d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:23.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.882+0000 7f5a6e44e700 1 -- 172.21.15.46:0/1391057001 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a680ca220 msgr2=0x7f5a6814d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:23.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.882+0000 7f5a6e44e700 1 --2- 172.21.15.46:0/1391057001 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a680ca220 0x7f5a6814d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.882+0000 7f5a6e44e700 1 -- 172.21.15.46:0/1391057001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a68159510 msgr2=0x7f5a680caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:23.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.882+0000 7f5a6e44e700 1 --2- 172.21.15.46:0/1391057001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a68159510 0x7f5a680caa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:23.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.882+0000 7f5a6e44e700 1 -- 172.21.15.46:0/1391057001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a580094f0 con 0x7f5a680cafb0 2024-01-27T00:46:23.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.883+0000 7f5a6d44c700 1 --2- 172.21.15.46:0/1391057001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a68159510 0x7f5a680caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:23.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.883+0000 7f5a6e44e700 1 --2- 172.21.15.46:0/1391057001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a680cafb0 0x7f5a6814f8d0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400c940 tx=0x7f5a640016b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:23.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.883+0000 7f5a5dffb700 1 -- 172.21.15.46:0/1391057001 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a64012630 con 0x7f5a680cafb0 2024-01-27T00:46:23.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.883+0000 7f5a5dffb700 1 -- 172.21.15.46:0/1391057001 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a640127b0 con 0x7f5a680cafb0 2024-01-27T00:46:23.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.883+0000 7f5a5dffb700 1 -- 172.21.15.46:0/1391057001 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a64012b70 con 0x7f5a680cafb0 2024-01-27T00:46:23.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.883+0000 7f5a6feb1700 1 -- 172.21.15.46:0/1391057001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a680c5db0 con 0x7f5a680cafb0 2024-01-27T00:46:23.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.883+0000 7f5a6feb1700 1 -- 172.21.15.46:0/1391057001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a680c6330 con 0x7f5a680cafb0 2024-01-27T00:46:23.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.884+0000 7f5a6feb1700 1 -- 172.21.15.46:0/1391057001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6804c710 con 0x7f5a680cafb0 2024-01-27T00:46:23.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.884+0000 7f5a5dffb700 1 -- 172.21.15.46:0/1391057001 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 84) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f5a6401a3a0 con 0x7f5a680cafb0 2024-01-27T00:46:23.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.885+0000 7f5a5dffb700 1 --2- 172.21.15.46:0/1391057001 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5a4c067050 0x7f5a4c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:23.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.885+0000 7f5a6dc4d700 1 -- 172.21.15.46:0/1391057001 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5a4c067050 msgr2=0x7f5a4c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:23.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.885+0000 7f5a5dffb700 1 -- 172.21.15.46:0/1391057001 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6807+0+0 (secure 0 0 0) 0x7f5a64084c00 con 0x7f5a680cafb0 2024-01-27T00:46:23.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.885+0000 7f5a6dc4d700 1 --2- 172.21.15.46:0/1391057001 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5a4c067050 0x7f5a4c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:46:23.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:23.890+0000 7f5a5dffb700 1 -- 172.21.15.46:0/1391057001 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a6408b080 con 0x7f5a680cafb0 2024-01-27T00:46:23.900 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:23.901+0000 7f7a7a4bf1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T00:46:23.970 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:23.970+0000 7f7a7a4bf1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T00:46:24.041 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:24.042+0000 7ff6f08dc1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T00:46:24.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.086+0000 7f5a6dc4d700 1 -- 172.21.15.46:0/1391057001 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5a4c067050 msgr2=0x7f5a4c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:24.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.086+0000 7f5a6dc4d700 1 --2- 172.21.15.46:0/1391057001 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5a4c067050 0x7f5a4c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:46:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.095+0000 7f5a6feb1700 1 -- 172.21.15.46:0/1391057001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5a680c9d10 con 0x7f5a680cafb0 2024-01-27T00:46:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.095+0000 7f5a5dffb700 1 -- 172.21.15.46:0/1391057001 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v162) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5a64051910 con 0x7f5a680cafb0 2024-01-27T00:46:24.094 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-27T00:46:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.098+0000 7f5a6feb1700 1 -- 172.21.15.46:0/1391057001 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5a4c067050 msgr2=0x7f5a4c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:46:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.098+0000 7f5a6feb1700 1 --2- 172.21.15.46:0/1391057001 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5a4c067050 0x7f5a4c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.098+0000 7f5a6feb1700 1 -- 172.21.15.46:0/1391057001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a680cafb0 msgr2=0x7f5a6814f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.098+0000 7f5a6feb1700 1 --2- 172.21.15.46:0/1391057001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a680cafb0 0x7f5a6814f8d0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400c940 tx=0x7f5a640016b0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.098+0000 7f5a6feb1700 1 -- 172.21.15.46:0/1391057001 shutdown_connections 2024-01-27T00:46:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.098+0000 7f5a6feb1700 1 --2- 172.21.15.46:0/1391057001 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5a4c067050 0x7f5a4c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.098+0000 7f5a6feb1700 1 --2- 172.21.15.46:0/1391057001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a680cafb0 0x7f5a6814f8d0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.098+0000 7f5a6feb1700 1 --2- 172.21.15.46:0/1391057001 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a680ca220 0x7f5a6814d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.098+0000 7f5a6feb1700 1 --2- 172.21.15.46:0/1391057001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a68159510 0x7f5a680caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.098+0000 7f5a6feb1700 1 -- 172.21.15.46:0/1391057001 >> 172.21.15.46:0/1391057001 conn(0x7f5a6806bd90 msgr2=0x7f5a68069d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.098+0000 7f5a6feb1700 1 -- 172.21.15.46:0/1391057001 shutdown_connections 2024-01-27T00:46:24.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.098+0000 7f5a6feb1700 1 -- 172.21.15.46:0/1391057001 wait complete. 2024-01-27T00:46:24.111 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T00:46:24.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.247+0000 7f689ea53700 1 Processor -- start 2024-01-27T00:46:24.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.247+0000 7f689ea53700 1 -- start start 2024-01-27T00:46:24.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.249+0000 7f689ea53700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f68980ae520 0x7f68980b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:24.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.249+0000 7f689ea53700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f68980b3fb0 0x7f6898157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:24.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.249+0000 7f689ea53700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6898158ae0 con 0x7f6898157340 2024-01-27T00:46:24.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.249+0000 7f689ea53700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68980b4430 con 0x7f68980b3fb0 2024-01-27T00:46:24.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.249+0000 7f689ea53700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f689815b650 con 0x7f68980ae520 2024-01-27T00:46:24.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.249+0000 7f68977fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f68980ae520 0x7f68980b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:24.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.249+0000 7f689cff0700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f68980b3fb0 0x7f6898157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:24.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.249+0000 7f6897fff700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f6898157340 0x7f689815bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:42136/0 (socket says 172.21.15.46:42136) 2024-01-27T00:46:24.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.249+0000 7f689cff0700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f68980b3fb0 0x7f6898157710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:39300/0 (socket says 172.21.15.46:39300) 2024-01-27T00:46:24.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.249+0000 7f68977fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f68980ae520 0x7f68980b3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:58458/0 (socket says 172.21.15.46:58458) 2024-01-27T00:46:24.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.249+0000 7f6897fff700 1 -- 172.21.15.46:0/4242400953 learned_addr learned my addr 172.21.15.46:0/4242400953 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:24.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.249+0000 7f689cff0700 1 -- 172.21.15.46:0/4242400953 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f68980ae520 msgr2=0x7f68980b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:24.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.249+0000 7f689cff0700 1 --2- 172.21.15.46:0/4242400953 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f68980ae520 0x7f68980b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.249+0000 7f689cff0700 1 -- 172.21.15.46:0/4242400953 >> v1:172.21.15.106:6789/0 conn(0x7f6898157340 legacy=0x7f689815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:24.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.249+0000 7f689cff0700 1 -- 172.21.15.46:0/4242400953 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6898158520 con 0x7f68980b3fb0 2024-01-27T00:46:24.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.249+0000 7f6896ffd700 1 -- 172.21.15.46:0/4242400953 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606148888 0 0) 0x7f6898158ae0 con 0x7f6898157340 2024-01-27T00:46:24.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.250+0000 7f689cff0700 1 --2- 172.21.15.46:0/4242400953 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f68980b3fb0 0x7f6898157710 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f6890009e60 tx=0x7f6890009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaa597d99095f432 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:24.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.250+0000 7f6896ffd700 1 -- 172.21.15.46:0/4242400953 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689000e070 con 0x7f68980b3fb0 2024-01-27T00:46:24.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.250+0000 7f6896ffd700 1 -- 172.21.15.46:0/4242400953 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689000bb60 con 0x7f68980b3fb0 2024-01-27T00:46:24.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.250+0000 7f6896ffd700 1 -- 172.21.15.46:0/4242400953 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6890012430 con 0x7f68980b3fb0 2024-01-27T00:46:24.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.250+0000 7f689ea53700 1 -- 172.21.15.46:0/4242400953 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f68980b3fb0 msgr2=0x7f6898157710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:24.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.250+0000 7f689ea53700 1 --2- 172.21.15.46:0/4242400953 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f68980b3fb0 0x7f6898157710 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f6890009e60 tx=0x7f6890009780 comp rx=0 tx=0).stop 2024-01-27T00:46:24.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.251+0000 7f689ea53700 1 -- 172.21.15.46:0/4242400953 shutdown_connections 2024-01-27T00:46:24.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.251+0000 7f689ea53700 1 --2- 172.21.15.46:0/4242400953 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f68980b3fb0 0x7f6898157710 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.251+0000 7f689ea53700 1 --2- 172.21.15.46:0/4242400953 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f68980ae520 0x7f68980b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.251+0000 7f689ea53700 1 -- 172.21.15.46:0/4242400953 >> 172.21.15.46:0/4242400953 conn(0x7f68980ce530 msgr2=0x7f68980ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:24.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.251+0000 7f689ea53700 1 -- 172.21.15.46:0/4242400953 shutdown_connections 2024-01-27T00:46:24.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.251+0000 7f689ea53700 1 -- 172.21.15.46:0/4242400953 wait complete. 2024-01-27T00:46:24.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.252+0000 7f689ea53700 1 Processor -- start 2024-01-27T00:46:24.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.252+0000 7f689ea53700 1 -- start start 2024-01-27T00:46:24.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f689ea53700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f68980ae520 0x7f68980b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:24.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f689ea53700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f68980b3fb0 0x7f68980ac430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:24.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f689ea53700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6898157340 0x7f6898069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:24.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f689ea53700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68980b1cb0 con 0x7f6898157340 2024-01-27T00:46:24.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f689ea53700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689816a970 con 0x7f68980b3fb0 2024-01-27T00:46:24.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f689ea53700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68980b1380 con 0x7f68980ae520 2024-01-27T00:46:24.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f689cff0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6898157340 0x7f6898069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:24.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f68977fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f68980b3fb0 0x7f68980ac430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:24.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f6897fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f68980ae520 0x7f68980b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:24.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f689cff0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6898157340 0x7f6898069a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49340/0 (socket says 172.21.15.46:49340) 2024-01-27T00:46:24.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f689cff0700 1 -- 172.21.15.46:0/4186862383 learned_addr learned my addr 172.21.15.46:0/4186862383 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:24.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f68977fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f68980b3fb0 0x7f68980ac430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:39306/0 (socket says 172.21.15.46:39306) 2024-01-27T00:46:24.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f6897fff700 1 -- 172.21.15.46:0/4186862383 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f68980b3fb0 msgr2=0x7f68980ac430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:24.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f6897fff700 1 --2- 172.21.15.46:0/4186862383 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f68980b3fb0 0x7f68980ac430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f6897fff700 1 -- 172.21.15.46:0/4186862383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6898157340 msgr2=0x7f6898069a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:24.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f6897fff700 1 --2- 172.21.15.46:0/4186862383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6898157340 0x7f6898069a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f6897fff700 1 -- 172.21.15.46:0/4186862383 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6890009a20 con 0x7f68980ae520 2024-01-27T00:46:24.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f689cff0700 1 --2- 172.21.15.46:0/4186862383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6898157340 0x7f6898069a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:24.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.253+0000 7f68977fe700 1 --2- 172.21.15.46:0/4186862383 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f68980b3fb0 0x7f68980ac430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:24.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.254+0000 7f6897fff700 1 --2- 172.21.15.46:0/4186862383 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f68980ae520 0x7f68980b2140 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f68880011e0 tx=0x7f6888007d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:24.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.254+0000 7f687ffff700 1 -- 172.21.15.46:0/4186862383 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6888005b30 con 0x7f68980ae520 2024-01-27T00:46:24.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.254+0000 7f687ffff700 1 -- 172.21.15.46:0/4186862383 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6888005c90 con 0x7f68980ae520 2024-01-27T00:46:24.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.254+0000 7f689ea53700 1 -- 172.21.15.46:0/4186862383 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68980b1660 con 0x7f68980ae520 2024-01-27T00:46:24.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.254+0000 7f687ffff700 1 -- 172.21.15.46:0/4186862383 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6888007300 con 0x7f68980ae520 2024-01-27T00:46:24.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.254+0000 7f689ea53700 1 -- 172.21.15.46:0/4186862383 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68980ac970 con 0x7f68980ae520 2024-01-27T00:46:24.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.255+0000 7f689ea53700 1 -- 172.21.15.46:0/4186862383 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68980b1f90 con 0x7f68980ae520 2024-01-27T00:46:24.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.255+0000 7f687ffff700 1 -- 172.21.15.46:0/4186862383 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 84) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f688801b3b0 con 0x7f68980ae520 2024-01-27T00:46:24.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.256+0000 7f687ffff700 1 --2- 172.21.15.46:0/4186862383 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6884067050 0x7f6884069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:24.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.256+0000 7f68977fe700 1 -- 172.21.15.46:0/4186862383 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6884067050 msgr2=0x7f6884069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:24.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.256+0000 7f687ffff700 1 -- 172.21.15.46:0/4186862383 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6807+0+0 (secure 0 0 0) 0x7f6888083f10 con 0x7f68980ae520 2024-01-27T00:46:24.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.256+0000 7f68977fe700 1 --2- 172.21.15.46:0/4186862383 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6884067050 0x7f6884069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:46:24.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.260+0000 7f687ffff700 1 -- 172.21.15.46:0/4186862383 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6888050c20 con 0x7f68980ae520 2024-01-27T00:46:24.320 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:24.321+0000 7ff6f08dc1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T00:46:24.402 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:24.402+0000 7ff6f08dc1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T00:46:24.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.457+0000 7f68977fe700 1 -- 172.21.15.46:0/4186862383 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6884067050 msgr2=0x7f6884069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:24.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.457+0000 7f68977fe700 1 --2- 172.21.15.46:0/4186862383 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6884067050 0x7f6884069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:46:24.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.531+0000 7f689ea53700 1 -- 172.21.15.46:0/4186862383 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68980b14c0 con 0x7f68980ae520 2024-01-27T00:46:24.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.532+0000 7f687ffff700 1 -- 172.21.15.46:0/4186862383 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f68980b14c0 con 0x7f68980ae520 2024-01-27T00:46:24.531 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:24.531 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8578,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1889696251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1889696251},{"type":"v1","addr":"172.21.15.106:6834","nonce":1889696251}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/631881132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":631881132},{"type":"v1","addr":"172.21.15.106:6836","nonce":631881132}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1045000694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1045000694},{"type":"v1","addr":"172.21.15.46:6834","nonce":1045000694}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:50:05.176064+0000","modified":"2024-01-26T23:50:08.757937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8871":{"gid":8871,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3419491709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3419491709},{"type":"v1","addr":"172.21.15.46:6835","nonce":3419491709}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2024-01-27T00:46:24.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.534+0000 7f689ea53700 1 -- 172.21.15.46:0/4186862383 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6884067050 msgr2=0x7f6884069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:46:24.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.534+0000 7f689ea53700 1 --2- 172.21.15.46:0/4186862383 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6884067050 0x7f6884069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.534+0000 7f689ea53700 1 -- 172.21.15.46:0/4186862383 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f68980ae520 msgr2=0x7f68980b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:24.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.534+0000 7f689ea53700 1 --2- 172.21.15.46:0/4186862383 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f68980ae520 0x7f68980b2140 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f68880011e0 tx=0x7f6888007d30 comp rx=0 tx=0).stop 2024-01-27T00:46:24.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.534+0000 7f689ea53700 1 -- 172.21.15.46:0/4186862383 shutdown_connections 2024-01-27T00:46:24.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.534+0000 7f689ea53700 1 --2- 172.21.15.46:0/4186862383 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6884067050 0x7f6884069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.534+0000 7f689ea53700 1 --2- 172.21.15.46:0/4186862383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6898157340 0x7f6898069a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.534+0000 7f689ea53700 1 --2- 172.21.15.46:0/4186862383 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f68980ae520 0x7f68980b2140 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.534+0000 7f689ea53700 1 --2- 172.21.15.46:0/4186862383 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f68980b3fb0 0x7f68980ac430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.534+0000 7f689ea53700 1 -- 172.21.15.46:0/4186862383 >> 172.21.15.46:0/4186862383 conn(0x7f68980ce530 msgr2=0x7f6898157710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:24.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.534+0000 7f689ea53700 1 -- 172.21.15.46:0/4186862383 shutdown_connections 2024-01-27T00:46:24.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.534+0000 7f689ea53700 1 -- 172.21.15.46:0/4186862383 wait complete. 2024-01-27T00:46:24.535 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 78 2024-01-27T00:46:24.547 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T00:46:24.585 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:24.585+0000 7f7a7a4bf1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T00:46:24.659 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:24.659+0000 7f7a7a4bf1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T00:46:24.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.680+0000 7f332e323700 1 Processor -- start 2024-01-27T00:46:24.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.680+0000 7f332e323700 1 -- start start 2024-01-27T00:46:24.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.681+0000 7f332e323700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f332806cbb0 0x7f33280ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:24.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.681+0000 7f332e323700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33280b0280 0x7f33281565a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:24.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.681+0000 7f332e323700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33280b0760 con 0x7f33281561d0 2024-01-27T00:46:24.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.681+0000 7f332e323700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33280b0a40 con 0x7f33280b0280 2024-01-27T00:46:24.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.681+0000 7f332e323700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33280b0d20 con 0x7f332806cbb0 2024-01-27T00:46:24.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.681+0000 7f332c8c0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33280b0280 0x7f33281565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:24.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.682+0000 7f332c8c0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33280b0280 0x7f33281565a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49350/0 (socket says 172.21.15.46:49350) 2024-01-27T00:46:24.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.682+0000 7f332c8c0700 1 -- 172.21.15.46:0/4189303867 learned_addr learned my addr 172.21.15.46:0/4189303867 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:24.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.682+0000 7f3327fff700 1 --1- 172.21.15.46:0/4189303867 >> v1:172.21.15.46:6789/0 conn(0x7f33281561d0 0x7f332806c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46118/0 (socket says 172.21.15.46:46118) 2024-01-27T00:46:24.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.682+0000 7f331ffff700 1 --2- 172.21.15.46:0/4189303867 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f332806cbb0 0x7f33280ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:24.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.682+0000 7f332c8c0700 1 -- 172.21.15.46:0/4189303867 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f332806cbb0 msgr2=0x7f33280ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:24.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.682+0000 7f332c8c0700 1 --2- 172.21.15.46:0/4189303867 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f332806cbb0 0x7f33280ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.682+0000 7f332c8c0700 1 -- 172.21.15.46:0/4189303867 >> v1:172.21.15.46:6789/0 conn(0x7f33281561d0 legacy=0x7f332806c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:24.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.682+0000 7f332c8c0700 1 -- 172.21.15.46:0/4189303867 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33280b1000 con 0x7f33280b0280 2024-01-27T00:46:24.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.682+0000 7f33277fe700 1 -- 172.21.15.46:0/4189303867 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29531155 0 0) 0x7f33280b0760 con 0x7f33281561d0 2024-01-27T00:46:24.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.682+0000 7f332c8c0700 1 --2- 172.21.15.46:0/4189303867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33280b0280 0x7f33281565a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f3320009f20 tx=0x7f3320009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1cada389251b793 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:24.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.682+0000 7f33277fe700 1 -- 172.21.15.46:0/4189303867 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332000e070 con 0x7f33280b0280 2024-01-27T00:46:24.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.682+0000 7f33277fe700 1 -- 172.21.15.46:0/4189303867 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f332000b9d0 con 0x7f33280b0280 2024-01-27T00:46:24.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.682+0000 7f33277fe700 1 -- 172.21.15.46:0/4189303867 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33200092e0 con 0x7f33280b0280 2024-01-27T00:46:24.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.683+0000 7f332e323700 1 -- 172.21.15.46:0/4189303867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33280b0280 msgr2=0x7f33281565a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:24.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.683+0000 7f332e323700 1 --2- 172.21.15.46:0/4189303867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33280b0280 0x7f33281565a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f3320009f20 tx=0x7f3320009840 comp rx=0 tx=0).stop 2024-01-27T00:46:24.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.683+0000 7f332e323700 1 -- 172.21.15.46:0/4189303867 shutdown_connections 2024-01-27T00:46:24.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.683+0000 7f332e323700 1 --2- 172.21.15.46:0/4189303867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33280b0280 0x7f33281565a0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.683+0000 7f332e323700 1 --2- 172.21.15.46:0/4189303867 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f332806cbb0 0x7f33280ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.683+0000 7f332e323700 1 -- 172.21.15.46:0/4189303867 >> 172.21.15.46:0/4189303867 conn(0x7f332806b630 msgr2=0x7f33280aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:24.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.683+0000 7f332e323700 1 -- 172.21.15.46:0/4189303867 shutdown_connections 2024-01-27T00:46:24.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.683+0000 7f332e323700 1 -- 172.21.15.46:0/4189303867 wait complete. 2024-01-27T00:46:24.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.685+0000 7f332e323700 1 Processor -- start 2024-01-27T00:46:24.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.685+0000 7f332e323700 1 -- start start 2024-01-27T00:46:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f332e323700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f332806cbb0 0x7f332813ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f332e323700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33280b0280 0x7f332815a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f332e323700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33281561d0 0x7f3328156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f332e323700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332815a120 con 0x7f33281561d0 2024-01-27T00:46:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f332e323700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332813efb0 con 0x7f332806cbb0 2024-01-27T00:46:24.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f332e323700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f332814d610 con 0x7f33280b0280 2024-01-27T00:46:24.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f332c8c0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33281561d0 0x7f3328156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:24.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f332c8c0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33281561d0 0x7f3328156b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49358/0 (socket says 172.21.15.46:49358) 2024-01-27T00:46:24.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f332c8c0700 1 -- 172.21.15.46:0/4049671155 learned_addr learned my addr 172.21.15.46:0/4049671155 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:24.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f3327fff700 1 --2- 172.21.15.46:0/4049671155 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f332806cbb0 0x7f332813ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:24.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f331ffff700 1 --2- 172.21.15.46:0/4049671155 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33280b0280 0x7f332815a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:24.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f332c8c0700 1 -- 172.21.15.46:0/4049671155 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33280b0280 msgr2=0x7f332815a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:24.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f332c8c0700 1 --2- 172.21.15.46:0/4049671155 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33280b0280 0x7f332815a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f332c8c0700 1 -- 172.21.15.46:0/4049671155 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f332806cbb0 msgr2=0x7f332813ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:24.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f332c8c0700 1 --2- 172.21.15.46:0/4049671155 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f332806cbb0 0x7f332813ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f332c8c0700 1 -- 172.21.15.46:0/4049671155 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3320009ae0 con 0x7f33281561d0 2024-01-27T00:46:24.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f331ffff700 1 --2- 172.21.15.46:0/4049671155 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33280b0280 0x7f332815a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:24.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.687+0000 7f332c8c0700 1 --2- 172.21.15.46:0/4049671155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33281561d0 0x7f3328156b70 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f332000b610 tx=0x7f3320018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:24.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.688+0000 7f3324ff9700 1 -- 172.21.15.46:0/4049671155 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3320007b30 con 0x7f33281561d0 2024-01-27T00:46:24.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.688+0000 7f3324ff9700 1 -- 172.21.15.46:0/4049671155 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3320007c90 con 0x7f33281561d0 2024-01-27T00:46:24.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.688+0000 7f332e323700 1 -- 172.21.15.46:0/4049671155 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3328159510 con 0x7f33281561d0 2024-01-27T00:46:24.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.688+0000 7f3324ff9700 1 -- 172.21.15.46:0/4049671155 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332001d670 con 0x7f33281561d0 2024-01-27T00:46:24.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.688+0000 7f332e323700 1 -- 172.21.15.46:0/4049671155 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3328159ad0 con 0x7f33281561d0 2024-01-27T00:46:24.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.689+0000 7f332e323700 1 -- 172.21.15.46:0/4049671155 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f332815a290 con 0x7f33281561d0 2024-01-27T00:46:24.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.689+0000 7f3324ff9700 1 -- 172.21.15.46:0/4049671155 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 84) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f3320003050 con 0x7f33281561d0 2024-01-27T00:46:24.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.690+0000 7f3324ff9700 1 --2- 172.21.15.46:0/4049671155 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3308067050 0x7f3308069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:24.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.690+0000 7f3327fff700 1 -- 172.21.15.46:0/4049671155 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3308067050 msgr2=0x7f3308069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:24.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.690+0000 7f3324ff9700 1 -- 172.21.15.46:0/4049671155 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6807+0+0 (secure 0 0 0) 0x7f33200867f0 con 0x7f33281561d0 2024-01-27T00:46:24.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.690+0000 7f3327fff700 1 --2- 172.21.15.46:0/4049671155 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3308067050 0x7f3308069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:46:24.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.694+0000 7f3324ff9700 1 -- 172.21.15.46:0/4049671155 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f33200533d0 con 0x7f33281561d0 2024-01-27T00:46:24.782 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:24.782+0000 7f7a7a4bf1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T00:46:24.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.891+0000 7f3327fff700 1 -- 172.21.15.46:0/4049671155 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3308067050 msgr2=0x7f3308069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:24.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.891+0000 7f3327fff700 1 --2- 172.21.15.46:0/4049671155 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3308067050 0x7f3308069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:46:24.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.901+0000 7f332e323700 1 -- 172.21.15.46:0/4049671155 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f332804c710 con 0x7f33281561d0 2024-01-27T00:46:24.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.902+0000 7f3324ff9700 1 -- 172.21.15.46:0/4049671155 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+16769 (secure 0 0 0) 0x7f33200569f0 con 0x7f33281561d0 2024-01-27T00:46:24.902 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:24.902 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":162,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T00:46:22.452559+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:50:03.691049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-01-26T23:50:04.138930+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":162,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/2632355058":"2024-01-27T23:50:19.809740+0000","172.21.15.106:0/156485008":"2024-01-27T23:50:19.809740+0000","172.21.15.106:6837/43388":"2024-01-27T23:50:19.809740+0000","172.21.15.106:6835/43388":"2024-01-27T23:50:19.809740+0000","172.21.15.106:0/2132373362":"2024-01-27T23:50:19.809740+0000","172.21.15.106:0/1443701093":"2024-01-27T23:50:19.809740+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[{"pool":20,"snaps":[{"begin":2,"length":1},{"begin":58,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T00:46:24.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.905+0000 7f332e323700 1 -- 172.21.15.46:0/4049671155 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3308067050 msgr2=0x7f3308069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:46:24.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.905+0000 7f332e323700 1 --2- 172.21.15.46:0/4049671155 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3308067050 0x7f3308069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.905+0000 7f332e323700 1 -- 172.21.15.46:0/4049671155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33281561d0 msgr2=0x7f3328156b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:24.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.905+0000 7f332e323700 1 --2- 172.21.15.46:0/4049671155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33281561d0 0x7f3328156b70 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f332000b610 tx=0x7f3320018000 comp rx=0 tx=0).stop 2024-01-27T00:46:24.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.905+0000 7f332e323700 1 -- 172.21.15.46:0/4049671155 shutdown_connections 2024-01-27T00:46:24.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.905+0000 7f332e323700 1 --2- 172.21.15.46:0/4049671155 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f3308067050 0x7f3308069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.905+0000 7f332e323700 1 --2- 172.21.15.46:0/4049671155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33281561d0 0x7f3328156b70 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.905+0000 7f332e323700 1 --2- 172.21.15.46:0/4049671155 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f332806cbb0 0x7f332813ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.905+0000 7f332e323700 1 --2- 172.21.15.46:0/4049671155 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33280b0280 0x7f332815a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:24.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.905+0000 7f332e323700 1 -- 172.21.15.46:0/4049671155 >> 172.21.15.46:0/4049671155 conn(0x7f332806b630 msgr2=0x7f3328067930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:24.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.905+0000 7f332e323700 1 -- 172.21.15.46:0/4049671155 shutdown_connections 2024-01-27T00:46:24.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:24.905+0000 7f332e323700 1 -- 172.21.15.46:0/4049671155 wait complete. 2024-01-27T00:46:24.920 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-27T00:46:24.920 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T00:46:25.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.056+0000 7f70ff7c2700 1 Processor -- start 2024-01-27T00:46:25.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.056+0000 7f70ff7c2700 1 -- start start 2024-01-27T00:46:25.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.057+0000 7f70ff7c2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f70f8068a40 0x7f70f806a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:25.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.057+0000 7f70ff7c2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f80690d0 con 0x7f70f806a6b0 2024-01-27T00:46:25.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.057+0000 7f70ff7c2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f80693b0 con 0x7f70f8159790 2024-01-27T00:46:25.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.057+0000 7f70ff7c2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70f8068ee0 con 0x7f70f8068a40 2024-01-27T00:46:25.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.057+0000 7f70fcd5d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f70f8068a40 0x7f70f806a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:25.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.057+0000 7f70fcd5d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f70f8068a40 0x7f70f806a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49362/0 (socket says 172.21.15.46:49362) 2024-01-27T00:46:25.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.057+0000 7f70fcd5d700 1 -- 172.21.15.46:0/4074334935 learned_addr learned my addr 172.21.15.46:0/4074334935 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:25.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.057+0000 7f70fdd5f700 1 --1- 172.21.15.46:0/4074334935 >> v1:172.21.15.46:6789/0 conn(0x7f70f806a6b0 0x7f70f80c3b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46128/0 (socket says 172.21.15.46:46128) 2024-01-27T00:46:25.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.058+0000 7f70fcd5d700 1 -- 172.21.15.46:0/4074334935 >> v1:172.21.15.106:6789/0 conn(0x7f70f8159790 legacy=0x7f70f80cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:25.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.058+0000 7f70f7fff700 1 -- 172.21.15.46:0/4074334935 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171565785 0 0) 0x7f70f80693b0 con 0x7f70f8159790 2024-01-27T00:46:25.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.058+0000 7f70fcd5d700 1 -- 172.21.15.46:0/4074334935 >> v1:172.21.15.46:6789/0 conn(0x7f70f806a6b0 legacy=0x7f70f80c3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:25.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.058+0000 7f70fcd5d700 1 -- 172.21.15.46:0/4074334935 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f8069690 con 0x7f70f8068a40 2024-01-27T00:46:25.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.058+0000 7f70fcd5d700 1 --2- 172.21.15.46:0/4074334935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70f8068a40 0x7f70f806a170 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f70ec0099b0 tx=0x7f70ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbf04788c8129692 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:25.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.058+0000 7f70f7fff700 1 -- 172.21.15.46:0/4074334935 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec00b760 con 0x7f70f8068a40 2024-01-27T00:46:25.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.058+0000 7f70f7fff700 1 -- 172.21.15.46:0/4074334935 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ec00b8c0 con 0x7f70f8068a40 2024-01-27T00:46:25.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.058+0000 7f70f7fff700 1 -- 172.21.15.46:0/4074334935 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec00bc80 con 0x7f70f8068a40 2024-01-27T00:46:25.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.059+0000 7f70ff7c2700 1 -- 172.21.15.46:0/4074334935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70f8068a40 msgr2=0x7f70f806a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:25.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.059+0000 7f70ff7c2700 1 --2- 172.21.15.46:0/4074334935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70f8068a40 0x7f70f806a170 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f70ec0099b0 tx=0x7f70ec009250 comp rx=0 tx=0).stop 2024-01-27T00:46:25.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.059+0000 7f70ff7c2700 1 -- 172.21.15.46:0/4074334935 shutdown_connections 2024-01-27T00:46:25.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.059+0000 7f70ff7c2700 1 --2- 172.21.15.46:0/4074334935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70f8068a40 0x7f70f806a170 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.059+0000 7f70ff7c2700 1 -- 172.21.15.46:0/4074334935 >> 172.21.15.46:0/4074334935 conn(0x7f70f8097820 msgr2=0x7f70f8068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:25.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.059+0000 7f70ff7c2700 1 -- 172.21.15.46:0/4074334935 shutdown_connections 2024-01-27T00:46:25.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.059+0000 7f70ff7c2700 1 -- 172.21.15.46:0/4074334935 wait complete. 2024-01-27T00:46:25.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.060+0000 7f70ff7c2700 1 Processor -- start 2024-01-27T00:46:25.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.060+0000 7f70ff7c2700 1 -- start start 2024-01-27T00:46:25.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.061+0000 7f70ff7c2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70f8068a40 0x7f70f80b09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.061+0000 7f70ff7c2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70f806a6b0 0x7f70f814b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.061+0000 7f70ff7c2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70f8159790 0x7f70f806bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.061+0000 7f70ff7c2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f80abc80 con 0x7f70f8068a40 2024-01-27T00:46:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.061+0000 7f70ff7c2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f80b0ee0 con 0x7f70f806a6b0 2024-01-27T00:46:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.061+0000 7f70ff7c2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f80ab0b0 con 0x7f70f8159790 2024-01-27T00:46:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.061+0000 7f70fd55e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70f8068a40 0x7f70f80b09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.061+0000 7f70fd55e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70f8068a40 0x7f70f80b09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49378/0 (socket says 172.21.15.46:49378) 2024-01-27T00:46:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.061+0000 7f70fd55e700 1 -- 172.21.15.46:0/3761103811 learned_addr learned my addr 172.21.15.46:0/3761103811 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.061+0000 7f70fdd5f700 1 --2- 172.21.15.46:0/3761103811 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70f8159790 0x7f70f806bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.061+0000 7f70fcd5d700 1 --2- 172.21.15.46:0/3761103811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70f806a6b0 0x7f70f814b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.062+0000 7f70fd55e700 1 -- 172.21.15.46:0/3761103811 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70f8159790 msgr2=0x7f70f806bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.062+0000 7f70fd55e700 1 --2- 172.21.15.46:0/3761103811 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70f8159790 0x7f70f806bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.062+0000 7f70fd55e700 1 -- 172.21.15.46:0/3761103811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70f806a6b0 msgr2=0x7f70f814b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:25.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.062+0000 7f70fd55e700 1 --2- 172.21.15.46:0/3761103811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70f806a6b0 0x7f70f814b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.062+0000 7f70fd55e700 1 -- 172.21.15.46:0/3761103811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec0094f0 con 0x7f70f8068a40 2024-01-27T00:46:25.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.062+0000 7f70fd55e700 1 --2- 172.21.15.46:0/3761103811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70f8068a40 0x7f70f80b09a0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f70e800ccc0 tx=0x7f70e8001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:25.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.062+0000 7f70f57fa700 1 -- 172.21.15.46:0/3761103811 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e800a810 con 0x7f70f8068a40 2024-01-27T00:46:25.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.062+0000 7f70f57fa700 1 -- 172.21.15.46:0/3761103811 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e800a970 con 0x7f70f8068a40 2024-01-27T00:46:25.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.062+0000 7f70ff7c2700 1 -- 172.21.15.46:0/3761103811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f80ab350 con 0x7f70f8068a40 2024-01-27T00:46:25.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.062+0000 7f70f57fa700 1 -- 172.21.15.46:0/3761103811 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e800ad30 con 0x7f70f8068a40 2024-01-27T00:46:25.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.062+0000 7f70ff7c2700 1 -- 172.21.15.46:0/3761103811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f80ab910 con 0x7f70f8068a40 2024-01-27T00:46:25.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.063+0000 7f70ff7c2700 1 -- 172.21.15.46:0/3761103811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f804c710 con 0x7f70f8068a40 2024-01-27T00:46:25.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.064+0000 7f70f57fa700 1 -- 172.21.15.46:0/3761103811 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 84) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f70e8012630 con 0x7f70f8068a40 2024-01-27T00:46:25.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.064+0000 7f70f57fa700 1 --2- 172.21.15.46:0/3761103811 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f70e4067050 0x7f70e4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:25.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.064+0000 7f70fcd5d700 1 -- 172.21.15.46:0/3761103811 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f70e4067050 msgr2=0x7f70e4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:25.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.064+0000 7f70f57fa700 1 -- 172.21.15.46:0/3761103811 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6807+0+0 (secure 0 0 0) 0x7f70e8085140 con 0x7f70f8068a40 2024-01-27T00:46:25.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.064+0000 7f70fcd5d700 1 --2- 172.21.15.46:0/3761103811 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f70e4067050 0x7f70e4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:46:25.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.069+0000 7f70f57fa700 1 -- 172.21.15.46:0/3761103811 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f70e8051e50 con 0x7f70f8068a40 2024-01-27T00:46:25.118 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:25.119+0000 7ff6f08dc1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T00:46:25.203 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:25.204+0000 7ff6f08dc1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T00:46:25.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.265+0000 7f70fcd5d700 1 -- 172.21.15.46:0/3761103811 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f70e4067050 msgr2=0x7f70e4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:25.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.265+0000 7f70fcd5d700 1 --2- 172.21.15.46:0/3761103811 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f70e4067050 0x7f70e4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:46:25.281 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:25.281+0000 7f7a7a4bf1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T00:46:25.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.321+0000 7f70ff7c2700 1 -- 172.21.15.46:0/3761103811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70f8069970 con 0x7f70f8068a40 2024-01-27T00:46:25.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.322+0000 7f70f57fa700 1 -- 172.21.15.46:0/3761103811 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f70e8055470 con 0x7f70f8068a40 2024-01-27T00:46:25.321 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:25.321 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8578,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1889696251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1889696251},{"type":"v1","addr":"172.21.15.106:6834","nonce":1889696251}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/631881132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":631881132},{"type":"v1","addr":"172.21.15.106:6836","nonce":631881132}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1045000694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1045000694},{"type":"v1","addr":"172.21.15.46:6834","nonce":1045000694}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:50:05.176064+0000","modified":"2024-01-26T23:50:08.757937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8871":{"gid":8871,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3419491709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3419491709},{"type":"v1","addr":"172.21.15.46:6835","nonce":3419491709}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2024-01-27T00:46:25.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.324+0000 7f70ff7c2700 1 -- 172.21.15.46:0/3761103811 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f70e4067050 msgr2=0x7f70e4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:46:25.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.324+0000 7f70ff7c2700 1 --2- 172.21.15.46:0/3761103811 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f70e4067050 0x7f70e4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.324+0000 7f70ff7c2700 1 -- 172.21.15.46:0/3761103811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70f8068a40 msgr2=0x7f70f80b09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:25.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.324+0000 7f70ff7c2700 1 --2- 172.21.15.46:0/3761103811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70f8068a40 0x7f70f80b09a0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f70e800ccc0 tx=0x7f70e8001ae0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.324+0000 7f70ff7c2700 1 -- 172.21.15.46:0/3761103811 shutdown_connections 2024-01-27T00:46:25.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.324+0000 7f70ff7c2700 1 --2- 172.21.15.46:0/3761103811 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f70e4067050 0x7f70e4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.324+0000 7f70ff7c2700 1 --2- 172.21.15.46:0/3761103811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70f8068a40 0x7f70f80b09a0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.324+0000 7f70ff7c2700 1 --2- 172.21.15.46:0/3761103811 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70f8159790 0x7f70f806bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.324+0000 7f70ff7c2700 1 --2- 172.21.15.46:0/3761103811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70f806a6b0 0x7f70f814b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.324+0000 7f70ff7c2700 1 -- 172.21.15.46:0/3761103811 >> 172.21.15.46:0/3761103811 conn(0x7f70f8097820 msgr2=0x7f70f80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:25.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.324+0000 7f70ff7c2700 1 -- 172.21.15.46:0/3761103811 shutdown_connections 2024-01-27T00:46:25.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.324+0000 7f70ff7c2700 1 -- 172.21.15.46:0/3761103811 wait complete. 2024-01-27T00:46:25.325 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 78 2024-01-27T00:46:25.338 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T00:46:25.346 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:25.347+0000 7ff6f08dc1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T00:46:25.427 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:25.428+0000 7f7a7a4bf1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T00:46:25.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.471+0000 7efc00c24700 1 Processor -- start 2024-01-27T00:46:25.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.471+0000 7efc00c24700 1 -- start start 2024-01-27T00:46:25.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.472+0000 7efc00c24700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efbfc15d210 0x7efbfc0cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:25.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.472+0000 7efc00c24700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efbfc0cbff0 0x7efbfc0ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:25.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.472+0000 7efc00c24700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbfc0cff70 con 0x7efbfc15ce40 2024-01-27T00:46:25.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.472+0000 7efc00c24700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbfc0cc490 con 0x7efbfc15d210 2024-01-27T00:46:25.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.472+0000 7efc00c24700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc0cf400 con 0x7efbfc0cbff0 2024-01-27T00:46:25.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.472+0000 7efbf9d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efbfc15d210 0x7efbfc0cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:25.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.472+0000 7efbf9d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efbfc15d210 0x7efbfc0cbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49382/0 (socket says 172.21.15.46:49382) 2024-01-27T00:46:25.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.472+0000 7efbf9d9b700 1 -- 172.21.15.46:0/469517139 learned_addr learned my addr 172.21.15.46:0/469517139 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:25.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.472+0000 7efbfa59c700 1 --1- 172.21.15.46:0/469517139 >> v1:172.21.15.106:6789/0 conn(0x7efbfc15ce40 0x7efbfc0d0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:42154/0 (socket says 172.21.15.46:42154) 2024-01-27T00:46:25.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.472+0000 7efbfad9d700 1 --2- 172.21.15.46:0/469517139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efbfc0cbff0 0x7efbfc0ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:25.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.472+0000 7efbf959a700 1 -- 172.21.15.46:0/469517139 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375315243 0 0) 0x7efbfc0cff70 con 0x7efbfc15ce40 2024-01-27T00:46:25.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.472+0000 7efbf959a700 1 -- 172.21.15.46:0/469517139 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbe4003570 con 0x7efbfc15ce40 2024-01-27T00:46:25.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.472+0000 7efbf9d9b700 1 -- 172.21.15.46:0/469517139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efbfc0cbff0 msgr2=0x7efbfc0ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:25.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.472+0000 7efbf9d9b700 1 --2- 172.21.15.46:0/469517139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efbfc0cbff0 0x7efbfc0ade30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.472+0000 7efbf9d9b700 1 -- 172.21.15.46:0/469517139 >> v1:172.21.15.106:6789/0 conn(0x7efbfc15ce40 legacy=0x7efbfc0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:25.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.473+0000 7efbf9d9b700 1 -- 172.21.15.46:0/469517139 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc0cf660 con 0x7efbfc15d210 2024-01-27T00:46:25.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.473+0000 7efbfad9d700 1 --2- 172.21.15.46:0/469517139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efbfc0cbff0 0x7efbfc0ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:25.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.473+0000 7efbf9d9b700 1 --2- 172.21.15.46:0/469517139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbfc15d210 0x7efbfc0cbab0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7efbf00099b0 tx=0x7efbf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b61faa78d9688a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:25.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.473+0000 7efbf959a700 1 -- 172.21.15.46:0/469517139 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf000b680 con 0x7efbfc15d210 2024-01-27T00:46:25.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.473+0000 7efbf959a700 1 -- 172.21.15.46:0/469517139 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf000b7e0 con 0x7efbfc15d210 2024-01-27T00:46:25.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.473+0000 7efbf959a700 1 -- 172.21.15.46:0/469517139 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf000bb80 con 0x7efbfc15d210 2024-01-27T00:46:25.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.474+0000 7efc00c24700 1 -- 172.21.15.46:0/469517139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbfc15d210 msgr2=0x7efbfc0cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:25.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.474+0000 7efc00c24700 1 --2- 172.21.15.46:0/469517139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbfc15d210 0x7efbfc0cbab0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7efbf00099b0 tx=0x7efbf0009250 comp rx=0 tx=0).stop 2024-01-27T00:46:25.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.474+0000 7efc00c24700 1 -- 172.21.15.46:0/469517139 shutdown_connections 2024-01-27T00:46:25.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.474+0000 7efc00c24700 1 --2- 172.21.15.46:0/469517139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efbfc0cbff0 0x7efbfc0ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.474+0000 7efc00c24700 1 --2- 172.21.15.46:0/469517139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbfc15d210 0x7efbfc0cbab0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.474+0000 7efc00c24700 1 -- 172.21.15.46:0/469517139 >> 172.21.15.46:0/469517139 conn(0x7efbfc06f840 msgr2=0x7efbfc0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:25.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.474+0000 7efc00c24700 1 -- 172.21.15.46:0/469517139 shutdown_connections 2024-01-27T00:46:25.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.474+0000 7efc00c24700 1 -- 172.21.15.46:0/469517139 wait complete. 2024-01-27T00:46:25.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.475+0000 7efc00c24700 1 Processor -- start 2024-01-27T00:46:25.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.475+0000 7efc00c24700 1 -- start start 2024-01-27T00:46:25.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.476+0000 7efc00c24700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efbfc0cbff0 0x7efbfc152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:25.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.476+0000 7efc00c24700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbfc15ce40 0x7efbfc156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:25.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.476+0000 7efc00c24700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efbfc15d210 0x7efbfc159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:25.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.476+0000 7efc00c24700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc1581a0 con 0x7efbfc15ce40 2024-01-27T00:46:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.476+0000 7efc00c24700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc158630 con 0x7efbfc15d210 2024-01-27T00:46:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.476+0000 7efc00c24700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc157590 con 0x7efbfc0cbff0 2024-01-27T00:46:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.476+0000 7efbf9d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbfc15ce40 0x7efbfc156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.476+0000 7efbf9d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbfc15ce40 0x7efbfc156c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49386/0 (socket says 172.21.15.46:49386) 2024-01-27T00:46:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.476+0000 7efbf9d9b700 1 -- 172.21.15.46:0/718504821 learned_addr learned my addr 172.21.15.46:0/718504821 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.476+0000 7efbfa59c700 1 --2- 172.21.15.46:0/718504821 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efbfc0cbff0 0x7efbfc152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.476+0000 7efbfad9d700 1 --2- 172.21.15.46:0/718504821 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efbfc15d210 0x7efbfc159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.476+0000 7efbf9d9b700 1 -- 172.21.15.46:0/718504821 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efbfc0cbff0 msgr2=0x7efbfc152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.476+0000 7efbf9d9b700 1 --2- 172.21.15.46:0/718504821 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efbfc0cbff0 0x7efbfc152ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.476+0000 7efbf9d9b700 1 -- 172.21.15.46:0/718504821 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efbfc15d210 msgr2=0x7efbfc159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.476+0000 7efbf9d9b700 1 --2- 172.21.15.46:0/718504821 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efbfc15d210 0x7efbfc159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.476+0000 7efbf9d9b700 1 -- 172.21.15.46:0/718504821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf4009a20 con 0x7efbfc15ce40 2024-01-27T00:46:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.476+0000 7efbfad9d700 1 --2- 172.21.15.46:0/718504821 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efbfc15d210 0x7efbfc159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.477+0000 7efbf9d9b700 1 --2- 172.21.15.46:0/718504821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbfc15ce40 0x7efbfc156c90 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7efbf0009290 tx=0x7efbf0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.477+0000 7efbea7fc700 1 -- 172.21.15.46:0/718504821 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0000d10 con 0x7efbfc15ce40 2024-01-27T00:46:25.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.477+0000 7efbea7fc700 1 -- 172.21.15.46:0/718504821 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf000bb60 con 0x7efbfc15ce40 2024-01-27T00:46:25.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.477+0000 7efc00c24700 1 -- 172.21.15.46:0/718504821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf00094f0 con 0x7efbfc15ce40 2024-01-27T00:46:25.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.477+0000 7efbea7fc700 1 -- 172.21.15.46:0/718504821 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0025670 con 0x7efbfc15ce40 2024-01-27T00:46:25.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.477+0000 7efc00c24700 1 -- 172.21.15.46:0/718504821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbfc157b50 con 0x7efbfc15ce40 2024-01-27T00:46:25.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.478+0000 7efc00c24700 1 -- 172.21.15.46:0/718504821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbfc04c710 con 0x7efbfc15ce40 2024-01-27T00:46:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.479+0000 7efbea7fc700 1 -- 172.21.15.46:0/718504821 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 84) v1 ==== 84856+0+0 (secure 0 0 0) 0x7efbf002e440 con 0x7efbfc15ce40 2024-01-27T00:46:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.479+0000 7efbea7fc700 1 --2- 172.21.15.46:0/718504821 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7efbdc067050 0x7efbdc069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.479+0000 7efbea7fc700 1 -- 172.21.15.46:0/718504821 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6807+0+0 (secure 0 0 0) 0x7efbf0086620 con 0x7efbfc15ce40 2024-01-27T00:46:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.479+0000 7efbfa59c700 1 -- 172.21.15.46:0/718504821 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7efbdc067050 msgr2=0x7efbdc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.479+0000 7efbfa59c700 1 --2- 172.21.15.46:0/718504821 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7efbdc067050 0x7efbdc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:46:25.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.484+0000 7efbea7fc700 1 -- 172.21.15.46:0/718504821 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efbf0053470 con 0x7efbfc15ce40 2024-01-27T00:46:25.578 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:25.578+0000 7f7a7a4bf1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T00:46:25.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.680+0000 7efbfa59c700 1 -- 172.21.15.46:0/718504821 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7efbdc067050 msgr2=0x7efbdc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:25.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.680+0000 7efbfa59c700 1 --2- 172.21.15.46:0/718504821 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7efbdc067050 0x7efbdc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:46:25.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.739+0000 7efc00c24700 1 -- 172.21.15.46:0/718504821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efbfc0d0500 con 0x7efbfc15ce40 2024-01-27T00:46:25.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.739+0000 7efbea7fc700 1 -- 172.21.15.46:0/718504821 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4602 (secure 0 0 0) 0x7efbf0056a90 con 0x7efbfc15ce40 2024-01-27T00:46:25.739 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:25.739 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8578,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1889696251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1889696251},{"type":"v1","addr":"172.21.15.106:6834","nonce":1889696251}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/631881132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":631881132},{"type":"v1","addr":"172.21.15.106:6836","nonce":631881132}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1045000694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1045000694},{"type":"v1","addr":"172.21.15.46:6834","nonce":1045000694}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-26T23:50:05.176064+0000","modified":"2024-01-26T23:50:08.757937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8871":{"gid":8871,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3419491709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3419491709},{"type":"v1","addr":"172.21.15.46:6835","nonce":3419491709}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2024-01-27T00:46:25.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.742+0000 7efc00c24700 1 -- 172.21.15.46:0/718504821 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7efbdc067050 msgr2=0x7efbdc069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:46:25.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.742+0000 7efc00c24700 1 --2- 172.21.15.46:0/718504821 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7efbdc067050 0x7efbdc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.742+0000 7efc00c24700 1 -- 172.21.15.46:0/718504821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbfc15ce40 msgr2=0x7efbfc156c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:25.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.742+0000 7efc00c24700 1 --2- 172.21.15.46:0/718504821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbfc15ce40 0x7efbfc156c90 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7efbf0009290 tx=0x7efbf0017000 comp rx=0 tx=0).stop 2024-01-27T00:46:25.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.742+0000 7efc00c24700 1 -- 172.21.15.46:0/718504821 shutdown_connections 2024-01-27T00:46:25.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.742+0000 7efc00c24700 1 --2- 172.21.15.46:0/718504821 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7efbdc067050 0x7efbdc069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.742+0000 7efc00c24700 1 --2- 172.21.15.46:0/718504821 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efbfc15d210 0x7efbfc159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.742+0000 7efc00c24700 1 --2- 172.21.15.46:0/718504821 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efbfc0cbff0 0x7efbfc152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.742+0000 7efc00c24700 1 --2- 172.21.15.46:0/718504821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbfc15ce40 0x7efbfc156c90 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.742+0000 7efc00c24700 1 -- 172.21.15.46:0/718504821 >> 172.21.15.46:0/718504821 conn(0x7efbfc06f840 msgr2=0x7efbfc0b45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:25.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.742+0000 7efc00c24700 1 -- 172.21.15.46:0/718504821 shutdown_connections 2024-01-27T00:46:25.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.742+0000 7efc00c24700 1 -- 172.21.15.46:0/718504821 wait complete. 2024-01-27T00:46:25.743 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 78 2024-01-27T00:46:25.755 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T00:46:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.942+0000 7f419a2ae700 1 Processor -- start 2024-01-27T00:46:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.942+0000 7f419a2ae700 1 -- start start 2024-01-27T00:46:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.943+0000 7f419a2ae700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f41940b2b40 0x7f41940ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.943+0000 7f419a2ae700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41940aedd0 0x7f41940cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.943+0000 7f419a2ae700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419415a0b0 con 0x7f41940ae960 2024-01-27T00:46:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.943+0000 7f419a2ae700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4194099fc0 con 0x7f41940b2b40 2024-01-27T00:46:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.943+0000 7f419a2ae700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41940ae4d0 con 0x7f41940aedd0 2024-01-27T00:46:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.943+0000 7f419884b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41940aedd0 0x7f41940cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.943+0000 7f4193fff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f41940b2b40 0x7f41940ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.943+0000 7f41937fe700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f41940ae960 0x7f41940ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:42164/0 (socket says 172.21.15.46:42164) 2024-01-27T00:46:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.943+0000 7f41937fe700 1 -- 172.21.15.46:0/1406414549 learned_addr learned my addr 172.21.15.46:0/1406414549 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.944+0000 7f419884b700 1 -- 172.21.15.46:0/1406414549 >> v2:172.21.15.106:3300/0 conn(0x7f41940b2b40 msgr2=0x7f41940ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.944+0000 7f419884b700 1 --2- 172.21.15.46:0/1406414549 >> v2:172.21.15.106:3300/0 conn(0x7f41940b2b40 0x7f41940ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.944+0000 7f419884b700 1 -- 172.21.15.46:0/1406414549 >> v1:172.21.15.106:6789/0 conn(0x7f41940ae960 legacy=0x7f41940ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.944+0000 7f419884b700 1 -- 172.21.15.46:0/1406414549 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41940ad8c0 con 0x7f41940aedd0 2024-01-27T00:46:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.944+0000 7f4192ffd700 1 -- 172.21.15.46:0/1406414549 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906039730 0 0) 0x7f419415a0b0 con 0x7f41940ae960 2024-01-27T00:46:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.944+0000 7f4193fff700 1 --2- 172.21.15.46:0/1406414549 >> v2:172.21.15.106:3300/0 conn(0x7f41940b2b40 0x7f41940ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.944+0000 7f419884b700 1 --2- 172.21.15.46:0/1406414549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41940aedd0 0x7f41940cb5e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f418c009e60 tx=0x7f418c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6681be7e85fd7d75 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.944+0000 7f4192ffd700 1 -- 172.21.15.46:0/1406414549 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c00e070 con 0x7f41940aedd0 2024-01-27T00:46:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.944+0000 7f4192ffd700 1 -- 172.21.15.46:0/1406414549 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418c00bb60 con 0x7f41940aedd0 2024-01-27T00:46:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.944+0000 7f4192ffd700 1 -- 172.21.15.46:0/1406414549 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c012430 con 0x7f41940aedd0 2024-01-27T00:46:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.945+0000 7f419a2ae700 1 -- 172.21.15.46:0/1406414549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41940aedd0 msgr2=0x7f41940cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.945+0000 7f419a2ae700 1 --2- 172.21.15.46:0/1406414549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41940aedd0 0x7f41940cb5e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f418c009e60 tx=0x7f418c009780 comp rx=0 tx=0).stop 2024-01-27T00:46:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.945+0000 7f419a2ae700 1 -- 172.21.15.46:0/1406414549 shutdown_connections 2024-01-27T00:46:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.945+0000 7f419a2ae700 1 --2- 172.21.15.46:0/1406414549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41940aedd0 0x7f41940cb5e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.945+0000 7f419a2ae700 1 --2- 172.21.15.46:0/1406414549 >> v2:172.21.15.106:3300/0 conn(0x7f41940b2b40 0x7f41940ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.945+0000 7f419a2ae700 1 -- 172.21.15.46:0/1406414549 >> 172.21.15.46:0/1406414549 conn(0x7f4194099a30 msgr2=0x7f419406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.945+0000 7f419a2ae700 1 -- 172.21.15.46:0/1406414549 shutdown_connections 2024-01-27T00:46:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.945+0000 7f419a2ae700 1 -- 172.21.15.46:0/1406414549 wait complete. 2024-01-27T00:46:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.946+0000 7f419a2ae700 1 Processor -- start 2024-01-27T00:46:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.946+0000 7f419a2ae700 1 -- start start 2024-01-27T00:46:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.947+0000 7f419a2ae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41940ae960 0x7f4194152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.947+0000 7f419a2ae700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41940b2b40 0x7f41940b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.947+0000 7f419a2ae700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41940b46e0 0x7f4194068910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.947+0000 7f419a2ae700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41940ca250 con 0x7f41940ae960 2024-01-27T00:46:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.947+0000 7f419a2ae700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41940c9360 con 0x7f41940b46e0 2024-01-27T00:46:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.947+0000 7f419a2ae700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41940c9640 con 0x7f41940b2b40 2024-01-27T00:46:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.947+0000 7f4193fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41940ae960 0x7f4194152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.947+0000 7f4193fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41940ae960 0x7f4194152540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49400/0 (socket says 172.21.15.46:49400) 2024-01-27T00:46:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.947+0000 7f4193fff700 1 -- 172.21.15.46:0/4135445138 learned_addr learned my addr 172.21.15.46:0/4135445138 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.947+0000 7f419884b700 1 --2- 172.21.15.46:0/4135445138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41940b46e0 0x7f4194068910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.947+0000 7f41937fe700 1 --2- 172.21.15.46:0/4135445138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41940b2b40 0x7f41940b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.948+0000 7f4193fff700 1 -- 172.21.15.46:0/4135445138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41940b2b40 msgr2=0x7f41940b41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.948+0000 7f4193fff700 1 --2- 172.21.15.46:0/4135445138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41940b2b40 0x7f41940b41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.948+0000 7f4193fff700 1 -- 172.21.15.46:0/4135445138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41940b46e0 msgr2=0x7f4194068910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.948+0000 7f4193fff700 1 --2- 172.21.15.46:0/4135445138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41940b46e0 0x7f4194068910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.948+0000 7f4193fff700 1 -- 172.21.15.46:0/4135445138 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41840093c0 con 0x7f41940ae960 2024-01-27T00:46:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.948+0000 7f4193fff700 1 --2- 172.21.15.46:0/4135445138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41940ae960 0x7f4194152540 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f418400ec80 tx=0x7f4184009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.948+0000 7f417bfff700 1 -- 172.21.15.46:0/4135445138 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4184005a00 con 0x7f41940ae960 2024-01-27T00:46:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.948+0000 7f419a2ae700 1 -- 172.21.15.46:0/4135445138 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418c009a20 con 0x7f41940ae960 2024-01-27T00:46:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.948+0000 7f417bfff700 1 -- 172.21.15.46:0/4135445138 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4184005b60 con 0x7f41940ae960 2024-01-27T00:46:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.948+0000 7f417bfff700 1 -- 172.21.15.46:0/4135445138 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418400c850 con 0x7f41940ae960 2024-01-27T00:46:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.948+0000 7f419a2ae700 1 -- 172.21.15.46:0/4135445138 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41940c9c00 con 0x7f41940ae960 2024-01-27T00:46:25.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.949+0000 7f419a2ae700 1 -- 172.21.15.46:0/4135445138 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f419404c710 con 0x7f41940ae960 2024-01-27T00:46:25.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.950+0000 7f417bfff700 1 -- 172.21.15.46:0/4135445138 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 84) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f418400cbc0 con 0x7f41940ae960 2024-01-27T00:46:25.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.950+0000 7f417bfff700 1 --2- 172.21.15.46:0/4135445138 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4180067050 0x7f4180069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:25.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.950+0000 7f41937fe700 1 -- 172.21.15.46:0/4135445138 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4180067050 msgr2=0x7f4180069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:25.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.950+0000 7f417bfff700 1 -- 172.21.15.46:0/4135445138 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6807+0+0 (secure 0 0 0) 0x7f4184084f30 con 0x7f41940ae960 2024-01-27T00:46:25.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.950+0000 7f41937fe700 1 --2- 172.21.15.46:0/4135445138 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4180067050 0x7f4180069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:46:25.953 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:25.954+0000 7ff6f08dc1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T00:46:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:25.954+0000 7f417bfff700 1 -- 172.21.15.46:0/4135445138 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4184051c40 con 0x7f41940ae960 2024-01-27T00:46:26.126 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:26.127+0000 7ff6f08dc1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T00:46:26.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.151+0000 7f41937fe700 1 -- 172.21.15.46:0/4135445138 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4180067050 msgr2=0x7f4180069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:26.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.151+0000 7f41937fe700 1 --2- 172.21.15.46:0/4135445138 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4180067050 0x7f4180069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:46:26.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.162+0000 7f419a2ae700 1 -- 172.21.15.46:0/4135445138 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41940cac20 con 0x7f41940ae960 2024-01-27T00:46:26.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.163+0000 7f417bfff700 1 -- 172.21.15.46:0/4135445138 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+16769 (secure 0 0 0) 0x7f4184020020 con 0x7f41940ae960 2024-01-27T00:46:26.163 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:26.163 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":162,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T00:46:22.452559+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-01-26T23:50:03.691049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-01-26T23:50:04.138930+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":162,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/2632355058":"2024-01-27T23:50:19.809740+0000","172.21.15.106:0/156485008":"2024-01-27T23:50:19.809740+0000","172.21.15.106:6837/43388":"2024-01-27T23:50:19.809740+0000","172.21.15.106:6835/43388":"2024-01-27T23:50:19.809740+0000","172.21.15.106:0/2132373362":"2024-01-27T23:50:19.809740+0000","172.21.15.106:0/1443701093":"2024-01-27T23:50:19.809740+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[{"pool":20,"snaps":[{"begin":2,"length":1},{"begin":58,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T00:46:26.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.165+0000 7f419a2ae700 1 -- 172.21.15.46:0/4135445138 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4180067050 msgr2=0x7f4180069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:46:26.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.165+0000 7f419a2ae700 1 --2- 172.21.15.46:0/4135445138 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4180067050 0x7f4180069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.165+0000 7f419a2ae700 1 -- 172.21.15.46:0/4135445138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41940ae960 msgr2=0x7f4194152540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:26.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.165+0000 7f419a2ae700 1 --2- 172.21.15.46:0/4135445138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41940ae960 0x7f4194152540 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f418400ec80 tx=0x7f4184009120 comp rx=0 tx=0).stop 2024-01-27T00:46:26.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.165+0000 7f419a2ae700 1 -- 172.21.15.46:0/4135445138 shutdown_connections 2024-01-27T00:46:26.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.165+0000 7f419a2ae700 1 --2- 172.21.15.46:0/4135445138 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4180067050 0x7f4180069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.165+0000 7f419a2ae700 1 --2- 172.21.15.46:0/4135445138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41940ae960 0x7f4194152540 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.165+0000 7f419a2ae700 1 --2- 172.21.15.46:0/4135445138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41940b46e0 0x7f4194068910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.165+0000 7f419a2ae700 1 --2- 172.21.15.46:0/4135445138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41940b2b40 0x7f41940b41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.165+0000 7f419a2ae700 1 -- 172.21.15.46:0/4135445138 >> 172.21.15.46:0/4135445138 conn(0x7f4194099a30 msgr2=0x7f41940c6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:26.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.166+0000 7f419a2ae700 1 -- 172.21.15.46:0/4135445138 shutdown_connections 2024-01-27T00:46:26.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.166+0000 7f419a2ae700 1 -- 172.21.15.46:0/4135445138 wait complete. 2024-01-27T00:46:26.179 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-27T00:46:26.273 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:26.274+0000 7f7a7a4bf1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T00:46:26.305 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:26.306+0000 7ff6f08dc1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T00:46:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.314+0000 7f9035af3700 1 Processor -- start 2024-01-27T00:46:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.314+0000 7f9035af3700 1 -- start start 2024-01-27T00:46:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.315+0000 7f9035af3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f903015ce40 0x7f90300d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.315+0000 7f9035af3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90300cc1c0 0x7f90300ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.315+0000 7f9035af3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90300cff70 con 0x7f903015d210 2024-01-27T00:46:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.315+0000 7f9035af3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90300cf400 con 0x7f903015ce40 2024-01-27T00:46:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.315+0000 7f9035af3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90300cf660 con 0x7f90300cc1c0 2024-01-27T00:46:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.315+0000 7f902f7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f903015ce40 0x7f90300d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.315+0000 7f902f7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f903015ce40 0x7f90300d0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49412/0 (socket says 172.21.15.46:49412) 2024-01-27T00:46:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.315+0000 7f902f7fe700 1 -- 172.21.15.46:0/2712805369 learned_addr learned my addr 172.21.15.46:0/2712805369 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.315+0000 7f902ffff700 1 --2- 172.21.15.46:0/2712805369 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90300cc1c0 0x7f90300ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.316+0000 7f902f7fe700 1 -- 172.21.15.46:0/2712805369 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90300cc1c0 msgr2=0x7f90300ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.316+0000 7f902f7fe700 1 --2- 172.21.15.46:0/2712805369 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90300cc1c0 0x7f90300ade30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.316+0000 7f902f7fe700 1 -- 172.21.15.46:0/2712805369 >> v1:172.21.15.106:6789/0 conn(0x7f903015d210 legacy=0x7f90300cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.316+0000 7f902e7fc700 1 -- 172.21.15.46:0/2712805369 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872311917 0 0) 0x7f90300cff70 con 0x7f903015d210 2024-01-27T00:46:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.316+0000 7f902f7fe700 1 -- 172.21.15.46:0/2712805369 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90300d0500 con 0x7f903015ce40 2024-01-27T00:46:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.316+0000 7f902f7fe700 1 --2- 172.21.15.46:0/2712805369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f903015ce40 0x7f90300d0e50 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f9020009f70 tx=0x7f9020009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56730ddb5d5ab339 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.316+0000 7f902e7fc700 1 -- 172.21.15.46:0/2712805369 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902000e070 con 0x7f903015ce40 2024-01-27T00:46:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.316+0000 7f902e7fc700 1 -- 172.21.15.46:0/2712805369 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902000ba20 con 0x7f903015ce40 2024-01-27T00:46:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.316+0000 7f902e7fc700 1 -- 172.21.15.46:0/2712805369 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90200092e0 con 0x7f903015ce40 2024-01-27T00:46:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.317+0000 7f9035af3700 1 -- 172.21.15.46:0/2712805369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f903015ce40 msgr2=0x7f90300d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.317+0000 7f9035af3700 1 --2- 172.21.15.46:0/2712805369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f903015ce40 0x7f90300d0e50 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f9020009f70 tx=0x7f9020009840 comp rx=0 tx=0).stop 2024-01-27T00:46:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.317+0000 7f9035af3700 1 -- 172.21.15.46:0/2712805369 shutdown_connections 2024-01-27T00:46:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.317+0000 7f9035af3700 1 --2- 172.21.15.46:0/2712805369 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90300cc1c0 0x7f90300ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.317+0000 7f9035af3700 1 --2- 172.21.15.46:0/2712805369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f903015ce40 0x7f90300d0e50 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.317+0000 7f9035af3700 1 -- 172.21.15.46:0/2712805369 >> 172.21.15.46:0/2712805369 conn(0x7f903006f840 msgr2=0x7f90300d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.317+0000 7f9035af3700 1 -- 172.21.15.46:0/2712805369 shutdown_connections 2024-01-27T00:46:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.317+0000 7f9035af3700 1 -- 172.21.15.46:0/2712805369 wait complete. 2024-01-27T00:46:26.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.319+0000 7f9035af3700 1 Processor -- start 2024-01-27T00:46:26.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.319+0000 7f9035af3700 1 -- start start 2024-01-27T00:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.320+0000 7f9035af3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90300cc1c0 0x7f90300c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.320+0000 7f9035af3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f903015ce40 0x7f903015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.320+0000 7f9035af3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f903015d210 0x7f903016cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.320+0000 7f9035af3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903015edf0 con 0x7f903015d210 2024-01-27T00:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.320+0000 7f9035af3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90300c8f10 con 0x7f903015ce40 2024-01-27T00:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.320+0000 7f9035af3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903009c240 con 0x7f90300cc1c0 2024-01-27T00:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.320+0000 7f902ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f903015d210 0x7f903016cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.320+0000 7f902f7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90300cc1c0 0x7f90300c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.320+0000 7f902effd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f903015ce40 0x7f903015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.320+0000 7f902ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f903015d210 0x7f903016cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49420/0 (socket says 172.21.15.46:49420) 2024-01-27T00:46:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.320+0000 7f902f7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90300cc1c0 0x7f90300c89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:58526/0 (socket says 172.21.15.46:58526) 2024-01-27T00:46:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.320+0000 7f902ffff700 1 -- 172.21.15.46:0/2876785821 learned_addr learned my addr 172.21.15.46:0/2876785821 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.321+0000 7f902f7fe700 1 -- 172.21.15.46:0/2876785821 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f903015ce40 msgr2=0x7f903015bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.321+0000 7f902f7fe700 1 --2- 172.21.15.46:0/2876785821 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f903015ce40 0x7f903015bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.321+0000 7f902f7fe700 1 -- 172.21.15.46:0/2876785821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f903015d210 msgr2=0x7f903016cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.321+0000 7f902f7fe700 1 --2- 172.21.15.46:0/2876785821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f903015d210 0x7f903016cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.321+0000 7f902f7fe700 1 -- 172.21.15.46:0/2876785821 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9020009ae0 con 0x7f90300cc1c0 2024-01-27T00:46:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.321+0000 7f902effd700 1 --2- 172.21.15.46:0/2876785821 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f903015ce40 0x7f903015bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.321+0000 7f902f7fe700 1 --2- 172.21.15.46:0/2876785821 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90300cc1c0 0x7f90300c89d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9020015000 tx=0x7f9020018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.321+0000 7f90277fe700 1 -- 172.21.15.46:0/2876785821 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9020008b50 con 0x7f90300cc1c0 2024-01-27T00:46:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.321+0000 7f90277fe700 1 -- 172.21.15.46:0/2876785821 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9020008cd0 con 0x7f90300cc1c0 2024-01-27T00:46:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.321+0000 7f90277fe700 1 -- 172.21.15.46:0/2876785821 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90200266b0 con 0x7f90300cc1c0 2024-01-27T00:46:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.321+0000 7f9035af3700 1 -- 172.21.15.46:0/2876785821 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903015e7a0 con 0x7f90300cc1c0 2024-01-27T00:46:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.321+0000 7f9035af3700 1 -- 172.21.15.46:0/2876785821 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903015c240 con 0x7f90300cc1c0 2024-01-27T00:46:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.322+0000 7f9035af3700 1 -- 172.21.15.46:0/2876785821 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903015c430 con 0x7f90300cc1c0 2024-01-27T00:46:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.323+0000 7f90277fe700 1 -- 172.21.15.46:0/2876785821 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 84) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f9020003040 con 0x7f90300cc1c0 2024-01-27T00:46:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.323+0000 7f90277fe700 1 --2- 172.21.15.46:0/2876785821 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9010067050 0x7f9010069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.323+0000 7f902effd700 1 -- 172.21.15.46:0/2876785821 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9010067050 msgr2=0x7f9010069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.323+0000 7f90277fe700 1 -- 172.21.15.46:0/2876785821 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6807+0+0 (secure 0 0 0) 0x7f90200869f0 con 0x7f90300cc1c0 2024-01-27T00:46:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.323+0000 7f902effd700 1 --2- 172.21.15.46:0/2876785821 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9010067050 0x7f9010069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:46:26.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.328+0000 7f90277fe700 1 -- 172.21.15.46:0/2876785821 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f903015c430 con 0x7f90300cc1c0 2024-01-27T00:46:26.492 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:26.491+0000 7f7a7a4bf1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:46:26.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.524+0000 7f902effd700 1 -- 172.21.15.46:0/2876785821 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9010067050 msgr2=0x7f9010069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:26.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.524+0000 7f902effd700 1 --2- 172.21.15.46:0/2876785821 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9010067050 0x7f9010069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:46:26.564 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:26.564+0000 7f7a7a4bf1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T00:46:26.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.584+0000 7f9035af3700 1 -- 172.21.15.46:0/2876785821 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f903015c430 con 0x7f90300cc1c0 2024-01-27T00:46:26.585 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-27T00:46:26.585+0000 7f33ffe86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-27T00:46:26.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.592+0000 7f90277fe700 1 -- 172.21.15.46:0/2876785821 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v79) v1 ==== 161+0+0 (secure 0 0 0) 0x7f903015c430 con 0x7f90300cc1c0 2024-01-27T00:46:26.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.595+0000 7f9035af3700 1 -- 172.21.15.46:0/2876785821 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9010067050 msgr2=0x7f9010069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:46:26.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.595+0000 7f9035af3700 1 --2- 172.21.15.46:0/2876785821 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9010067050 0x7f9010069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.595+0000 7f9035af3700 1 -- 172.21.15.46:0/2876785821 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90300cc1c0 msgr2=0x7f90300c89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:26.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.595+0000 7f9035af3700 1 --2- 172.21.15.46:0/2876785821 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90300cc1c0 0x7f90300c89d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9020015000 tx=0x7f9020018000 comp rx=0 tx=0).stop 2024-01-27T00:46:26.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.595+0000 7f9035af3700 1 -- 172.21.15.46:0/2876785821 shutdown_connections 2024-01-27T00:46:26.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.595+0000 7f9035af3700 1 --2- 172.21.15.46:0/2876785821 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9010067050 0x7f9010069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.595+0000 7f9035af3700 1 --2- 172.21.15.46:0/2876785821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f903015d210 0x7f903016cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.595+0000 7f9035af3700 1 --2- 172.21.15.46:0/2876785821 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90300cc1c0 0x7f90300c89d0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.595+0000 7f9035af3700 1 --2- 172.21.15.46:0/2876785821 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f903015ce40 0x7f903015bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.595+0000 7f9035af3700 1 -- 172.21.15.46:0/2876785821 >> 172.21.15.46:0/2876785821 conn(0x7f903006f840 msgr2=0x7f90300b47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:26.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.595+0000 7f9035af3700 1 -- 172.21.15.46:0/2876785821 shutdown_connections 2024-01-27T00:46:26.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.595+0000 7f9035af3700 1 -- 172.21.15.46:0/2876785821 wait complete. 2024-01-27T00:46:26.596 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-27T00:46:26.609 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-27T00:46:26.629 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-27T00:46:26.721 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:26.721+0000 7f7a7a4bf1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T00:46:26.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.752+0000 7fd4c94e2700 1 Processor -- start 2024-01-27T00:46:26.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.752+0000 7fd4c94e2700 1 -- start start 2024-01-27T00:46:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.753+0000 7fd4c94e2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd4c4159da0 0x7fd4c40ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.753+0000 7fd4c94e2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd4c415d8c0 0x7fd4c415dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.753+0000 7fd4c94e2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c415a200 con 0x7fd4c41599d0 2024-01-27T00:46:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.753+0000 7fd4c94e2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c415e260 con 0x7fd4c415d8c0 2024-01-27T00:46:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.753+0000 7fd4c94e2700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c40b1a60 con 0x7fd4c4159da0 2024-01-27T00:46:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.753+0000 7fd4c2ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd4c41599d0 0x7fd4c40b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46140/0 (socket says 172.21.15.46:46140) 2024-01-27T00:46:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.753+0000 7fd4c2ffd700 1 -- 172.21.15.46:0/3537684642 learned_addr learned my addr 172.21.15.46:0/3537684642 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.753+0000 7fd4c27fc700 1 --2- 172.21.15.46:0/3537684642 >> v2:172.21.15.106:3300/0 conn(0x7fd4c4159da0 0x7fd4c40ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.753+0000 7fd4c37fe700 1 --2- 172.21.15.46:0/3537684642 >> v2:172.21.15.46:3300/0 conn(0x7fd4c415d8c0 0x7fd4c415dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.754+0000 7fd4c1ffb700 1 -- 172.21.15.46:0/3537684642 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89128956 0 0) 0x7fd4c415a200 con 0x7fd4c41599d0 2024-01-27T00:46:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.754+0000 7fd4c1ffb700 1 -- 172.21.15.46:0/3537684642 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4ac003570 con 0x7fd4c41599d0 2024-01-27T00:46:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.754+0000 7fd4c27fc700 1 -- 172.21.15.46:0/3537684642 >> v2:172.21.15.46:3300/0 conn(0x7fd4c415d8c0 msgr2=0x7fd4c415dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.754+0000 7fd4c27fc700 1 --2- 172.21.15.46:0/3537684642 >> v2:172.21.15.46:3300/0 conn(0x7fd4c415d8c0 0x7fd4c415dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.754+0000 7fd4c27fc700 1 -- 172.21.15.46:0/3537684642 >> v1:172.21.15.46:6789/0 conn(0x7fd4c41599d0 legacy=0x7fd4c40b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:26.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.754+0000 7fd4c27fc700 1 -- 172.21.15.46:0/3537684642 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c40b11c0 con 0x7fd4c4159da0 2024-01-27T00:46:26.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.754+0000 7fd4c37fe700 1 --2- 172.21.15.46:0/3537684642 >> v2:172.21.15.46:3300/0 conn(0x7fd4c415d8c0 0x7fd4c415dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:26.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.754+0000 7fd4c27fc700 1 --2- 172.21.15.46:0/3537684642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4c4159da0 0x7fd4c40ab1c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd4c415d7f0 tx=0x7fd4b80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=697c31258b0e19c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:26.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.754+0000 7fd4c1ffb700 1 -- 172.21.15.46:0/3537684642 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8011630 con 0x7fd4c4159da0 2024-01-27T00:46:26.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.754+0000 7fd4c1ffb700 1 -- 172.21.15.46:0/3537684642 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b80117b0 con 0x7fd4c4159da0 2024-01-27T00:46:26.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.754+0000 7fd4c1ffb700 1 -- 172.21.15.46:0/3537684642 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8011b70 con 0x7fd4c4159da0 2024-01-27T00:46:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.755+0000 7fd4c94e2700 1 -- 172.21.15.46:0/3537684642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4c4159da0 msgr2=0x7fd4c40ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.755+0000 7fd4c94e2700 1 --2- 172.21.15.46:0/3537684642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4c4159da0 0x7fd4c40ab1c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd4c415d7f0 tx=0x7fd4b80091a0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.755+0000 7fd4c94e2700 1 -- 172.21.15.46:0/3537684642 shutdown_connections 2024-01-27T00:46:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.755+0000 7fd4c94e2700 1 --2- 172.21.15.46:0/3537684642 >> v2:172.21.15.46:3300/0 conn(0x7fd4c415d8c0 0x7fd4c415dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.755+0000 7fd4c94e2700 1 --2- 172.21.15.46:0/3537684642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4c4159da0 0x7fd4c40ab1c0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.755+0000 7fd4c94e2700 1 -- 172.21.15.46:0/3537684642 >> 172.21.15.46:0/3537684642 conn(0x7fd4c406f840 msgr2=0x7fd4c4087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.755+0000 7fd4c94e2700 1 -- 172.21.15.46:0/3537684642 shutdown_connections 2024-01-27T00:46:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.755+0000 7fd4c94e2700 1 -- 172.21.15.46:0/3537684642 wait complete. 2024-01-27T00:46:26.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.757+0000 7fd4c94e2700 1 Processor -- start 2024-01-27T00:46:26.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.757+0000 7fd4c94e2700 1 -- start start 2024-01-27T00:46:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.758+0000 7fd4c94e2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd4c41599d0 0x7fd4c40ab760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.758+0000 7fd4c94e2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4c4159da0 0x7fd4c416a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.758+0000 7fd4c94e2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4c415d8c0 0x7fd4c40cef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.758+0000 7fd4c94e2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c416ae00 con 0x7fd4c4159da0 2024-01-27T00:46:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.758+0000 7fd4c94e2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c40aca30 con 0x7fd4c415d8c0 2024-01-27T00:46:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.758+0000 7fd4c94e2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c40abe20 con 0x7fd4c41599d0 2024-01-27T00:46:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.759+0000 7fd4c2ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd4c41599d0 0x7fd4c40ab760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.759+0000 7fd4c37fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4c415d8c0 0x7fd4c40cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.759+0000 7fd4c27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4c4159da0 0x7fd4c416a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.759+0000 7fd4c2ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd4c41599d0 0x7fd4c40ab760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:58548/0 (socket says 172.21.15.46:58548) 2024-01-27T00:46:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.759+0000 7fd4c2ffd700 1 -- 172.21.15.46:0/3646401905 learned_addr learned my addr 172.21.15.46:0/3646401905 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.759+0000 7fd4c27fc700 1 -- 172.21.15.46:0/3646401905 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd4c41599d0 msgr2=0x7fd4c40ab760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.759+0000 7fd4c27fc700 1 --2- 172.21.15.46:0/3646401905 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd4c41599d0 0x7fd4c40ab760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.759+0000 7fd4c27fc700 1 -- 172.21.15.46:0/3646401905 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4c415d8c0 msgr2=0x7fd4c40cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.759+0000 7fd4c27fc700 1 --2- 172.21.15.46:0/3646401905 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4c415d8c0 0x7fd4c40cef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.759+0000 7fd4c27fc700 1 -- 172.21.15.46:0/3646401905 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4bc009420 con 0x7fd4c4159da0 2024-01-27T00:46:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.759+0000 7fd4c2ffd700 1 --2- 172.21.15.46:0/3646401905 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd4c41599d0 0x7fd4c40ab760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.759+0000 7fd4c37fe700 1 --2- 172.21.15.46:0/3646401905 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4c415d8c0 0x7fd4c40cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.759+0000 7fd4c27fc700 1 --2- 172.21.15.46:0/3646401905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4c4159da0 0x7fd4c416a8c0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8016000 tx=0x7fd4b8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.759+0000 7fd4aaffd700 1 -- 172.21.15.46:0/3646401905 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b801f070 con 0x7fd4c4159da0 2024-01-27T00:46:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.759+0000 7fd4aaffd700 1 -- 172.21.15.46:0/3646401905 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b8011920 con 0x7fd4c4159da0 2024-01-27T00:46:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.759+0000 7fd4c94e2700 1 -- 172.21.15.46:0/3646401905 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b8009420 con 0x7fd4c4159da0 2024-01-27T00:46:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.760+0000 7fd4aaffd700 1 -- 172.21.15.46:0/3646401905 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8023410 con 0x7fd4c4159da0 2024-01-27T00:46:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.760+0000 7fd4c94e2700 1 -- 172.21.15.46:0/3646401905 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c40ac3e0 con 0x7fd4c4159da0 2024-01-27T00:46:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.761+0000 7fd4c94e2700 1 -- 172.21.15.46:0/3646401905 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c40acbc0 con 0x7fd4c4159da0 2024-01-27T00:46:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.762+0000 7fd4aaffd700 1 -- 172.21.15.46:0/3646401905 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 85) v1 ==== 84453+0+0 (secure 0 0 0) 0x7fd4b8023800 con 0x7fd4c4159da0 2024-01-27T00:46:26.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.762+0000 7fd4aaffd700 1 -- 172.21.15.46:0/3646401905 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 7063+0+0 (secure 0 0 0) 0x7fd4b8085ab0 con 0x7fd4c4159da0 2024-01-27T00:46:26.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:26.766+0000 7fd4aaffd700 1 -- 172.21.15.46:0/3646401905 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd4b8052d00 con 0x7fd4c4159da0 2024-01-27T00:46:26.805 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:26.806+0000 7f79f956e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:26.806 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:26.806+0000 7f79f956e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:26.806 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:26.806+0000 7f79f956e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:26.806 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:26.806+0000 7f79f956e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:26.806 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:26.806+0000 7f79f956e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:26.806 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:26.806+0000 7f79f4d65700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:26.806 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:26.806+0000 7f79f4d65700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:26.806 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:26.806+0000 7f79f4d65700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:26.806 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:26.806+0000 7f79f4d65700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:26.806 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:26.806+0000 7f79f4d65700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:27.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.023+0000 7fd4c94e2700 1 -- 172.21.15.46:0/3646401905 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd4c40b14a0 con 0x7fd4c4159da0 2024-01-27T00:46:27.140 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:27.140+0000 7ff6f08dc1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T00:46:27.389 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:27.389+0000 7ff6f08dc1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:46:27.475 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:27.475+0000 7ff6f08dc1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T00:46:27.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.590+0000 7fd4aaffd700 1 -- 172.21.15.46:0/3646401905 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v80) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd4b8056320 con 0x7fd4c4159da0 2024-01-27T00:46:27.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.592+0000 7fd4c94e2700 1 -- 172.21.15.46:0/3646401905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4c4159da0 msgr2=0x7fd4c416a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:27.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.592+0000 7fd4c94e2700 1 --2- 172.21.15.46:0/3646401905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4c4159da0 0x7fd4c416a8c0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8016000 tx=0x7fd4b8017000 comp rx=0 tx=0).stop 2024-01-27T00:46:27.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.592+0000 7fd4c94e2700 1 -- 172.21.15.46:0/3646401905 shutdown_connections 2024-01-27T00:46:27.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.592+0000 7fd4c94e2700 1 --2- 172.21.15.46:0/3646401905 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4c415d8c0 0x7fd4c40cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:27.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.592+0000 7fd4c94e2700 1 --2- 172.21.15.46:0/3646401905 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd4c41599d0 0x7fd4c40ab760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:27.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.592+0000 7fd4c94e2700 1 --2- 172.21.15.46:0/3646401905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4c4159da0 0x7fd4c416a8c0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:27.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.592+0000 7fd4c94e2700 1 -- 172.21.15.46:0/3646401905 >> 172.21.15.46:0/3646401905 conn(0x7fd4c406f840 msgr2=0x7fd4c4068b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:27.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.592+0000 7fd4c94e2700 1 -- 172.21.15.46:0/3646401905 shutdown_connections 2024-01-27T00:46:27.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.592+0000 7fd4c94e2700 1 -- 172.21.15.46:0/3646401905 wait complete. 2024-01-27T00:46:27.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-27T00:46:27.655 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:27.656+0000 7ff6f08dc1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T00:46:27.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.786+0000 7f4786956700 1 Processor -- start 2024-01-27T00:46:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.786+0000 7f4786956700 1 -- start start 2024-01-27T00:46:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.787+0000 7f4786956700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4780159750 0x7f4780069650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.787+0000 7f4786956700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47800691c0 con 0x7f4780069b90 2024-01-27T00:46:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.787+0000 7f4786956700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478006a5e0 con 0x7f4780159b20 2024-01-27T00:46:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.787+0000 7f4786956700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f478015dfd0 con 0x7f4780159750 2024-01-27T00:46:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.787+0000 7f4784ef3700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4780069b90 0x7f478015d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46148/0 (socket says 172.21.15.46:46148) 2024-01-27T00:46:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.787+0000 7f4784ef3700 1 -- 172.21.15.46:0/3778748751 learned_addr learned my addr 172.21.15.46:0/3778748751 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.788+0000 7f477ffff700 1 --2- 172.21.15.46:0/3778748751 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4780159750 0x7f4780069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.788+0000 7f477effd700 1 -- 172.21.15.46:0/3778748751 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428997384 0 0) 0x7f47800691c0 con 0x7f4780069b90 2024-01-27T00:46:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.788+0000 7f477effd700 1 -- 172.21.15.46:0/3778748751 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4768003570 con 0x7f4780069b90 2024-01-27T00:46:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.788+0000 7f477ffff700 1 -- 172.21.15.46:0/3778748751 >> v1:172.21.15.106:6789/0 conn(0x7f4780159b20 legacy=0x7f47800ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.788+0000 7f477effd700 1 -- 172.21.15.46:0/3778748751 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569407012 0 0) 0x7f478006a5e0 con 0x7f4780159b20 2024-01-27T00:46:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.788+0000 7f477ffff700 1 -- 172.21.15.46:0/3778748751 >> v1:172.21.15.46:6789/0 conn(0x7f4780069b90 legacy=0x7f478015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.788+0000 7f477ffff700 1 -- 172.21.15.46:0/3778748751 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4780068b70 con 0x7f4780159750 2024-01-27T00:46:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.788+0000 7f477ffff700 1 --2- 172.21.15.46:0/3778748751 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4780159750 0x7f4780069650 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4770009910 tx=0x7f47700091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d5147fd1157f4b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.788+0000 7f477effd700 1 -- 172.21.15.46:0/3778748751 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477000b680 con 0x7f4780159750 2024-01-27T00:46:27.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.788+0000 7f477effd700 1 -- 172.21.15.46:0/3778748751 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477000b7e0 con 0x7f4780159750 2024-01-27T00:46:27.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.788+0000 7f477effd700 1 -- 172.21.15.46:0/3778748751 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477000bb60 con 0x7f4780159750 2024-01-27T00:46:27.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.789+0000 7f4786956700 1 -- 172.21.15.46:0/3778748751 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4780159750 msgr2=0x7f4780069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:27.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.789+0000 7f4786956700 1 --2- 172.21.15.46:0/3778748751 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4780159750 0x7f4780069650 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4770009910 tx=0x7f47700091a0 comp rx=0 tx=0).stop 2024-01-27T00:46:27.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.789+0000 7f4786956700 1 -- 172.21.15.46:0/3778748751 shutdown_connections 2024-01-27T00:46:27.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.789+0000 7f4786956700 1 --2- 172.21.15.46:0/3778748751 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4780159750 0x7f4780069650 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:27.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.789+0000 7f4786956700 1 -- 172.21.15.46:0/3778748751 >> 172.21.15.46:0/3778748751 conn(0x7f47800ce530 msgr2=0x7f47800ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:27.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.789+0000 7f4786956700 1 -- 172.21.15.46:0/3778748751 shutdown_connections 2024-01-27T00:46:27.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.789+0000 7f4786956700 1 -- 172.21.15.46:0/3778748751 wait complete. 2024-01-27T00:46:27.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.790+0000 7f4786956700 1 Processor -- start 2024-01-27T00:46:27.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.790+0000 7f4786956700 1 -- start start 2024-01-27T00:46:27.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.791+0000 7f4786956700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4780069b90 0x7f4780156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:27.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.791+0000 7f4786956700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4780159750 0x7f478015a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:27.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.791+0000 7f4786956700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4780159b20 0x7f47800b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:27.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.791+0000 7f4786956700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47800c5890 con 0x7f4780069b90 2024-01-27T00:46:27.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.791+0000 7f4786956700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47800c5a20 con 0x7f4780159750 2024-01-27T00:46:27.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.791+0000 7f4786956700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f478015b3c0 con 0x7f4780159b20 2024-01-27T00:46:27.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.791+0000 7f477ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4780069b90 0x7f4780156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:27.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.791+0000 7f477ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4780069b90 0x7f4780156930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49466/0 (socket says 172.21.15.46:49466) 2024-01-27T00:46:27.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.791+0000 7f477ffff700 1 -- 172.21.15.46:0/342211297 learned_addr learned my addr 172.21.15.46:0/342211297 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:27.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.791+0000 7f4784ef3700 1 --2- 172.21.15.46:0/342211297 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4780159b20 0x7f47800b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:27.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.791+0000 7f477f7fe700 1 --2- 172.21.15.46:0/342211297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4780159750 0x7f478015a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:27.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.792+0000 7f477ffff700 1 -- 172.21.15.46:0/342211297 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4780159b20 msgr2=0x7f47800b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:27.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.792+0000 7f477ffff700 1 --2- 172.21.15.46:0/342211297 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4780159b20 0x7f47800b35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:27.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.792+0000 7f477ffff700 1 -- 172.21.15.46:0/342211297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4780159750 msgr2=0x7f478015a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:27.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.792+0000 7f477ffff700 1 --2- 172.21.15.46:0/342211297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4780159750 0x7f478015a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:27.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.792+0000 7f477ffff700 1 -- 172.21.15.46:0/342211297 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4770009420 con 0x7f4780069b90 2024-01-27T00:46:27.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.792+0000 7f477ffff700 1 --2- 172.21.15.46:0/342211297 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4780069b90 0x7f4780156930 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f478006f390 tx=0x7f4770017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:27.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.792+0000 7f4767fff700 1 -- 172.21.15.46:0/342211297 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477001f070 con 0x7f4780069b90 2024-01-27T00:46:27.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.792+0000 7f4767fff700 1 -- 172.21.15.46:0/342211297 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4770000bc0 con 0x7f4780069b90 2024-01-27T00:46:27.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.792+0000 7f4786956700 1 -- 172.21.15.46:0/342211297 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f478015b6a0 con 0x7f4780069b90 2024-01-27T00:46:27.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.792+0000 7f4767fff700 1 -- 172.21.15.46:0/342211297 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4770025420 con 0x7f4780069b90 2024-01-27T00:46:27.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.792+0000 7f4786956700 1 -- 172.21.15.46:0/342211297 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478015bc60 con 0x7f4780069b90 2024-01-27T00:46:27.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.793+0000 7f4786956700 1 -- 172.21.15.46:0/342211297 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478004c710 con 0x7f4780069b90 2024-01-27T00:46:27.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.794+0000 7f4767fff700 1 -- 172.21.15.46:0/342211297 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 86) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f4770023070 con 0x7f4780069b90 2024-01-27T00:46:27.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.794+0000 7f4767fff700 1 --2- 172.21.15.46:0/342211297 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f476c066d80 0x7f476c069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:27.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.794+0000 7f4767fff700 1 -- 172.21.15.46:0/342211297 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f4770085a50 con 0x7f4780069b90 2024-01-27T00:46:27.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.794+0000 7f477f7fe700 1 --2- 172.21.15.46:0/342211297 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f476c066d80 0x7f476c069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:27.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.795+0000 7f477f7fe700 1 --2- 172.21.15.46:0/342211297 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f476c066d80 0x7f476c069230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f47800b0920 tx=0x7f4774007ec0 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:27.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:27.799+0000 7f4767fff700 1 -- 172.21.15.46:0/342211297 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4770052a20 con 0x7f4780069b90 2024-01-27T00:46:28.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.005+0000 7f4786956700 1 -- 172.21.15.46:0/342211297 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4780068e50 con 0x7f4780069b90 2024-01-27T00:46:28.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.753+0000 7f4767fff700 1 -- 172.21.15.46:0/342211297 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v165) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4770056040 con 0x7f4780069b90 2024-01-27T00:46:28.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.757+0000 7f4786956700 1 -- 172.21.15.46:0/342211297 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f476c066d80 msgr2=0x7f476c069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:28.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.757+0000 7f4786956700 1 --2- 172.21.15.46:0/342211297 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f476c066d80 0x7f476c069230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f47800b0920 tx=0x7f4774007ec0 comp rx=0 tx=0).stop 2024-01-27T00:46:28.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.757+0000 7f4786956700 1 -- 172.21.15.46:0/342211297 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4780069b90 msgr2=0x7f4780156930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:28.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.757+0000 7f4786956700 1 --2- 172.21.15.46:0/342211297 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4780069b90 0x7f4780156930 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f478006f390 tx=0x7f4770017000 comp rx=0 tx=0).stop 2024-01-27T00:46:28.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.757+0000 7f4786956700 1 -- 172.21.15.46:0/342211297 shutdown_connections 2024-01-27T00:46:28.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.757+0000 7f4786956700 1 --2- 172.21.15.46:0/342211297 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f476c066d80 0x7f476c069230 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:28.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.757+0000 7f4786956700 1 --2- 172.21.15.46:0/342211297 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4780069b90 0x7f4780156930 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:28.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.757+0000 7f4786956700 1 --2- 172.21.15.46:0/342211297 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4780159b20 0x7f47800b35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:28.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.757+0000 7f4786956700 1 --2- 172.21.15.46:0/342211297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4780159750 0x7f478015a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:28.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.757+0000 7f4786956700 1 -- 172.21.15.46:0/342211297 >> 172.21.15.46:0/342211297 conn(0x7f47800ce530 msgr2=0x7f47800ce900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:28.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.758+0000 7f4786956700 1 -- 172.21.15.46:0/342211297 shutdown_connections 2024-01-27T00:46:28.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.758+0000 7f4786956700 1 -- 172.21.15.46:0/342211297 wait complete. 2024-01-27T00:46:28.758 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-01-27T00:46:28.774 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-27T00:46:28.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.956+0000 7fcef5a49700 1 Processor -- start 2024-01-27T00:46:28.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.956+0000 7fcef5a49700 1 -- start start 2024-01-27T00:46:28.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.957+0000 7fcef5a49700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fcef015d210 0x7fcef00cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:28.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.957+0000 7fcef5a49700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcef00cbff0 0x7fcef00ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:28.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.957+0000 7fcef5a49700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef00cc450 con 0x7fcef015ce40 2024-01-27T00:46:28.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.957+0000 7fcef5a49700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcef00cff70 con 0x7fcef00cbff0 2024-01-27T00:46:28.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.957+0000 7fcef5a49700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcef00cf400 con 0x7fcef015d210 2024-01-27T00:46:28.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.957+0000 7fceef7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcef00cbff0 0x7fcef00ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:28.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.957+0000 7fceeeffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcef015ce40 0x7fcef00d0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46160/0 (socket says 172.21.15.46:46160) 2024-01-27T00:46:28.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.957+0000 7fceeeffd700 1 -- 172.21.15.46:0/3532176770 learned_addr learned my addr 172.21.15.46:0/3532176770 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:28.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.957+0000 7fceee7fc700 1 --2- 172.21.15.46:0/3532176770 >> v2:172.21.15.106:3300/0 conn(0x7fcef015d210 0x7fcef00cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:28.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.958+0000 7fceef7fe700 1 -- 172.21.15.46:0/3532176770 >> v2:172.21.15.106:3300/0 conn(0x7fcef015d210 msgr2=0x7fcef00cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:28.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.958+0000 7fceef7fe700 1 --2- 172.21.15.46:0/3532176770 >> v2:172.21.15.106:3300/0 conn(0x7fcef015d210 0x7fcef00cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:28.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.958+0000 7fceef7fe700 1 -- 172.21.15.46:0/3532176770 >> v1:172.21.15.46:6789/0 conn(0x7fcef015ce40 legacy=0x7fcef00d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:28.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.958+0000 7fceef7fe700 1 -- 172.21.15.46:0/3532176770 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef00cf660 con 0x7fcef00cbff0 2024-01-27T00:46:28.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.958+0000 7fceedffb700 1 -- 172.21.15.46:0/3532176770 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932782773 0 0) 0x7fcee0001110 con 0x7fcef015ce40 2024-01-27T00:46:28.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.958+0000 7fceef7fe700 1 --2- 172.21.15.46:0/3532176770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef00cbff0 0x7fcef00ade30 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fcef00895f0 tx=0x7fcee8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e280c68760191c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:28.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.958+0000 7fceedffb700 1 -- 172.21.15.46:0/3532176770 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee800e070 con 0x7fcef00cbff0 2024-01-27T00:46:28.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.958+0000 7fceedffb700 1 -- 172.21.15.46:0/3532176770 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee800bb60 con 0x7fcef00cbff0 2024-01-27T00:46:28.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.958+0000 7fceedffb700 1 -- 172.21.15.46:0/3532176770 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee8012430 con 0x7fcef00cbff0 2024-01-27T00:46:28.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.959+0000 7fcef5a49700 1 -- 172.21.15.46:0/3532176770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef00cbff0 msgr2=0x7fcef00ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:28.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.959+0000 7fcef5a49700 1 --2- 172.21.15.46:0/3532176770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef00cbff0 0x7fcef00ade30 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fcef00895f0 tx=0x7fcee8009780 comp rx=0 tx=0).stop 2024-01-27T00:46:28.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.959+0000 7fcef5a49700 1 -- 172.21.15.46:0/3532176770 shutdown_connections 2024-01-27T00:46:28.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.959+0000 7fcef5a49700 1 --2- 172.21.15.46:0/3532176770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef00cbff0 0x7fcef00ade30 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:28.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.959+0000 7fcef5a49700 1 --2- 172.21.15.46:0/3532176770 >> v2:172.21.15.106:3300/0 conn(0x7fcef015d210 0x7fcef00cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:28.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.959+0000 7fcef5a49700 1 -- 172.21.15.46:0/3532176770 >> 172.21.15.46:0/3532176770 conn(0x7fcef006f840 msgr2=0x7fcef00d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:28.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.959+0000 7fcef5a49700 1 -- 172.21.15.46:0/3532176770 shutdown_connections 2024-01-27T00:46:28.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.959+0000 7fcef5a49700 1 -- 172.21.15.46:0/3532176770 wait complete. 2024-01-27T00:46:28.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.960+0000 7fcef5a49700 1 Processor -- start 2024-01-27T00:46:28.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.960+0000 7fcef5a49700 1 -- start start 2024-01-27T00:46:28.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.961+0000 7fcef5a49700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef00cbff0 0x7fcef009aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:28.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.961+0000 7fcef5a49700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcef015ce40 0x7fcef00c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:28.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.961+0000 7fcef5a49700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcef015d210 0x7fcef009d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:28.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.961+0000 7fcef5a49700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef009d6e0 con 0x7fcef00cbff0 2024-01-27T00:46:28.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.961+0000 7fcef5a49700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef009d980 con 0x7fcef015ce40 2024-01-27T00:46:28.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.961+0000 7fcef5a49700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef00c4a30 con 0x7fcef015d210 2024-01-27T00:46:28.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.961+0000 7fceeeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef00cbff0 0x7fcef009aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:28.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.961+0000 7fceeeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef00cbff0 0x7fcef009aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49478/0 (socket says 172.21.15.46:49478) 2024-01-27T00:46:28.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.961+0000 7fceeeffd700 1 -- 172.21.15.46:0/646004308 learned_addr learned my addr 172.21.15.46:0/646004308 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:28.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.961+0000 7fceee7fc700 1 --2- 172.21.15.46:0/646004308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcef015ce40 0x7fcef00c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:28.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.961+0000 7fceef7fe700 1 --2- 172.21.15.46:0/646004308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcef015d210 0x7fcef009d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:28.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.961+0000 7fceeeffd700 1 -- 172.21.15.46:0/646004308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcef015d210 msgr2=0x7fcef009d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:28.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.961+0000 7fceeeffd700 1 --2- 172.21.15.46:0/646004308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcef015d210 0x7fcef009d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:28.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.961+0000 7fceeeffd700 1 -- 172.21.15.46:0/646004308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcef015ce40 msgr2=0x7fcef00c85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:28.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.961+0000 7fceeeffd700 1 --2- 172.21.15.46:0/646004308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcef015ce40 0x7fcef00c85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:28.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.961+0000 7fceeeffd700 1 -- 172.21.15.46:0/646004308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee8009a20 con 0x7fcef00cbff0 2024-01-27T00:46:28.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.961+0000 7fceef7fe700 1 --2- 172.21.15.46:0/646004308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcef015d210 0x7fcef009d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:46:28.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.962+0000 7fceeeffd700 1 --2- 172.21.15.46:0/646004308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef00cbff0 0x7fcef009aa70 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fcee000c810 tx=0x7fcee0007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:28.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.962+0000 7fcee6ffd700 1 -- 172.21.15.46:0/646004308 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee0005b30 con 0x7fcef00cbff0 2024-01-27T00:46:28.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.962+0000 7fcee6ffd700 1 -- 172.21.15.46:0/646004308 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee0005c90 con 0x7fcef00cbff0 2024-01-27T00:46:28.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.962+0000 7fcee6ffd700 1 -- 172.21.15.46:0/646004308 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee0007300 con 0x7fcef00cbff0 2024-01-27T00:46:28.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.962+0000 7fcef5a49700 1 -- 172.21.15.46:0/646004308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef00c4d10 con 0x7fcef00cbff0 2024-01-27T00:46:28.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.962+0000 7fcef5a49700 1 -- 172.21.15.46:0/646004308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef009dc60 con 0x7fcef00cbff0 2024-01-27T00:46:28.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.963+0000 7fcef5a49700 1 -- 172.21.15.46:0/646004308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcef00c5230 con 0x7fcef00cbff0 2024-01-27T00:46:28.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.963+0000 7fcee6ffd700 1 -- 172.21.15.46:0/646004308 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 86) v1 ==== 84580+0+0 (secure 0 0 0) 0x7fcee00076f0 con 0x7fcef00cbff0 2024-01-27T00:46:28.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.964+0000 7fcee6ffd700 1 --2- 172.21.15.46:0/646004308 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fced0066d80 0x7fced0069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:28.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.964+0000 7fcee6ffd700 1 -- 172.21.15.46:0/646004308 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6625+0+0 (secure 0 0 0) 0x7fcee0083f90 con 0x7fcef00cbff0 2024-01-27T00:46:28.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.964+0000 7fceee7fc700 1 --2- 172.21.15.46:0/646004308 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fced0066d80 0x7fced0069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:28.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.965+0000 7fceee7fc700 1 --2- 172.21.15.46:0/646004308 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fced0066d80 0x7fced0069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcef00cb350 tx=0x7fced8008040 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:28.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:28.968+0000 7fcee6ffd700 1 -- 172.21.15.46:0/646004308 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcef00c5230 con 0x7fcef00cbff0 2024-01-27T00:46:29.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.175+0000 7fcef5a49700 1 -- 172.21.15.46:0/646004308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcef00cc450 con 0x7fcef00cbff0 2024-01-27T00:46:29.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.754+0000 7fcee6ffd700 1 -- 172.21.15.46:0/646004308 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v166) v1 ==== 169+0+0 (secure 0 0 0) 0x7fcef00cc450 con 0x7fcef00cbff0 2024-01-27T00:46:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.763+0000 7fcef5a49700 1 -- 172.21.15.46:0/646004308 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fced0066d80 msgr2=0x7fced0069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.763+0000 7fcef5a49700 1 --2- 172.21.15.46:0/646004308 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fced0066d80 0x7fced0069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcef00cb350 tx=0x7fced8008040 comp rx=0 tx=0).stop 2024-01-27T00:46:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.763+0000 7fcef5a49700 1 -- 172.21.15.46:0/646004308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef00cbff0 msgr2=0x7fcef009aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.763+0000 7fcef5a49700 1 --2- 172.21.15.46:0/646004308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef00cbff0 0x7fcef009aa70 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fcee000c810 tx=0x7fcee0007c50 comp rx=0 tx=0).stop 2024-01-27T00:46:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.763+0000 7fcef5a49700 1 -- 172.21.15.46:0/646004308 shutdown_connections 2024-01-27T00:46:29.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.763+0000 7fcef5a49700 1 --2- 172.21.15.46:0/646004308 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fced0066d80 0x7fced0069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:29.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.763+0000 7fcef5a49700 1 --2- 172.21.15.46:0/646004308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef00cbff0 0x7fcef009aa70 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:29.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.763+0000 7fcef5a49700 1 --2- 172.21.15.46:0/646004308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcef015d210 0x7fcef009d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:29.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.763+0000 7fcef5a49700 1 --2- 172.21.15.46:0/646004308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcef015ce40 0x7fcef00c85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:29.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.763+0000 7fcef5a49700 1 -- 172.21.15.46:0/646004308 >> 172.21.15.46:0/646004308 conn(0x7fcef006f840 msgr2=0x7fcef00697a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:29.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.770+0000 7fcef5a49700 1 -- 172.21.15.46:0/646004308 shutdown_connections 2024-01-27T00:46:29.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.771+0000 7fcef5a49700 1 -- 172.21.15.46:0/646004308 wait complete. 2024-01-27T00:46:29.771 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-01-27T00:46:29.785 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-27T00:46:29.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.930+0000 7ff874d85700 1 Processor -- start 2024-01-27T00:46:29.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.930+0000 7ff874d85700 1 -- start start 2024-01-27T00:46:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.931+0000 7ff874d85700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff870157490 0x7ff8700cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.931+0000 7ff874d85700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff870157860 0x7ff8700cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.931+0000 7ff874d85700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8700cea80 con 0x7ff87006d6a0 2024-01-27T00:46:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.931+0000 7ff874d85700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8700cdc60 con 0x7ff870157490 2024-01-27T00:46:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.931+0000 7ff874d85700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8700b2ae0 con 0x7ff870157860 2024-01-27T00:46:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.932+0000 7ff86dd9b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff870157860 0x7ff8700cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.932+0000 7ff86e59c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff870157490 0x7ff8700cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.932+0000 7ff86ed9d700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7ff87006d6a0 0x7ff87006dab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:42186/0 (socket says 172.21.15.46:42186) 2024-01-27T00:46:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.932+0000 7ff86ed9d700 1 -- 172.21.15.46:0/1698118179 learned_addr learned my addr 172.21.15.46:0/1698118179 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.932+0000 7ff86d59a700 1 -- 172.21.15.46:0/1698118179 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441179993 0 0) 0x7ff8700cea80 con 0x7ff87006d6a0 2024-01-27T00:46:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.932+0000 7ff86d59a700 1 -- 172.21.15.46:0/1698118179 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff85c003570 con 0x7ff87006d6a0 2024-01-27T00:46:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.932+0000 7ff86dd9b700 1 -- 172.21.15.46:0/1698118179 >> v2:172.21.15.106:3300/0 conn(0x7ff870157490 msgr2=0x7ff8700cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.932+0000 7ff86dd9b700 1 --2- 172.21.15.46:0/1698118179 >> v2:172.21.15.106:3300/0 conn(0x7ff870157490 0x7ff8700cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.932+0000 7ff86dd9b700 1 -- 172.21.15.46:0/1698118179 >> v1:172.21.15.106:6789/0 conn(0x7ff87006d6a0 legacy=0x7ff87006dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.932+0000 7ff86dd9b700 1 -- 172.21.15.46:0/1698118179 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8700cde70 con 0x7ff870157860 2024-01-27T00:46:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.932+0000 7ff86e59c700 1 --2- 172.21.15.46:0/1698118179 >> v2:172.21.15.106:3300/0 conn(0x7ff870157490 0x7ff8700cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.932+0000 7ff86dd9b700 1 --2- 172.21.15.46:0/1698118179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff870157860 0x7ff8700cd620 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff8580098e0 tx=0x7ff8580091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82f8e44bb8c6f236 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.932+0000 7ff86d59a700 1 -- 172.21.15.46:0/1698118179 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff858011630 con 0x7ff870157860 2024-01-27T00:46:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.933+0000 7ff86d59a700 1 -- 172.21.15.46:0/1698118179 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff858011790 con 0x7ff870157860 2024-01-27T00:46:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.933+0000 7ff86d59a700 1 -- 172.21.15.46:0/1698118179 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff858011b30 con 0x7ff870157860 2024-01-27T00:46:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.933+0000 7ff874d85700 1 -- 172.21.15.46:0/1698118179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff870157860 msgr2=0x7ff8700cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.933+0000 7ff874d85700 1 --2- 172.21.15.46:0/1698118179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff870157860 0x7ff8700cd620 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff8580098e0 tx=0x7ff8580091a0 comp rx=0 tx=0).stop 2024-01-27T00:46:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.933+0000 7ff874d85700 1 -- 172.21.15.46:0/1698118179 shutdown_connections 2024-01-27T00:46:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.933+0000 7ff874d85700 1 --2- 172.21.15.46:0/1698118179 >> v2:172.21.15.106:3300/0 conn(0x7ff870157490 0x7ff8700cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.933+0000 7ff874d85700 1 --2- 172.21.15.46:0/1698118179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff870157860 0x7ff8700cd620 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.933+0000 7ff874d85700 1 -- 172.21.15.46:0/1698118179 >> 172.21.15.46:0/1698118179 conn(0x7ff870067930 msgr2=0x7ff870067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.933+0000 7ff874d85700 1 -- 172.21.15.46:0/1698118179 shutdown_connections 2024-01-27T00:46:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.933+0000 7ff874d85700 1 -- 172.21.15.46:0/1698118179 wait complete. 2024-01-27T00:46:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.934+0000 7ff874d85700 1 Processor -- start 2024-01-27T00:46:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.934+0000 7ff874d85700 1 -- start start 2024-01-27T00:46:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.935+0000 7ff874d85700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff87006d6a0 0x7ff870150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.935+0000 7ff874d85700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff870157490 0x7ff870169920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.935+0000 7ff874d85700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff870157860 0x7ff8700697c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.935+0000 7ff874d85700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8700c3350 con 0x7ff870157490 2024-01-27T00:46:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.935+0000 7ff874d85700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8700c41b0 con 0x7ff87006d6a0 2024-01-27T00:46:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.935+0000 7ff874d85700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8700c35a0 con 0x7ff870157860 2024-01-27T00:46:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.935+0000 7ff86dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff870157490 0x7ff870169920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.935+0000 7ff86e59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff87006d6a0 0x7ff870150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.935+0000 7ff86dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff870157490 0x7ff870169920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49480/0 (socket says 172.21.15.46:49480) 2024-01-27T00:46:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.935+0000 7ff86e59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff87006d6a0 0x7ff870150bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:39420/0 (socket says 172.21.15.46:39420) 2024-01-27T00:46:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.935+0000 7ff86dd9b700 1 -- 172.21.15.46:0/3363973327 learned_addr learned my addr 172.21.15.46:0/3363973327 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.935+0000 7ff86ed9d700 1 --2- 172.21.15.46:0/3363973327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff870157860 0x7ff8700697c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.936+0000 7ff86dd9b700 1 -- 172.21.15.46:0/3363973327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff870157860 msgr2=0x7ff8700697c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:29.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.936+0000 7ff86dd9b700 1 --2- 172.21.15.46:0/3363973327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff870157860 0x7ff8700697c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:29.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.936+0000 7ff86dd9b700 1 -- 172.21.15.46:0/3363973327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff87006d6a0 msgr2=0x7ff870150bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:29.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.936+0000 7ff86dd9b700 1 --2- 172.21.15.46:0/3363973327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff87006d6a0 0x7ff870150bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:29.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.936+0000 7ff86dd9b700 1 -- 172.21.15.46:0/3363973327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff860009420 con 0x7ff870157490 2024-01-27T00:46:29.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.936+0000 7ff86e59c700 1 --2- 172.21.15.46:0/3363973327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff87006d6a0 0x7ff870150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:29.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.936+0000 7ff86dd9b700 1 --2- 172.21.15.46:0/3363973327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff870157490 0x7ff870169920 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7ff8700adc40 tx=0x7ff858017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:29.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.936+0000 7ff8667fc700 1 -- 172.21.15.46:0/3363973327 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff858000c40 con 0x7ff870157490 2024-01-27T00:46:29.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.936+0000 7ff8667fc700 1 -- 172.21.15.46:0/3363973327 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff858011900 con 0x7ff870157490 2024-01-27T00:46:29.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.936+0000 7ff874d85700 1 -- 172.21.15.46:0/3363973327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff858009420 con 0x7ff870157490 2024-01-27T00:46:29.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.936+0000 7ff8667fc700 1 -- 172.21.15.46:0/3363973327 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff858025420 con 0x7ff870157490 2024-01-27T00:46:29.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.936+0000 7ff874d85700 1 -- 172.21.15.46:0/3363973327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8700c3930 con 0x7ff870157490 2024-01-27T00:46:29.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.937+0000 7ff874d85700 1 -- 172.21.15.46:0/3363973327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff87004c710 con 0x7ff870157490 2024-01-27T00:46:29.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.938+0000 7ff8667fc700 1 -- 172.21.15.46:0/3363973327 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 87) v1 ==== 84686+0+0 (secure 0 0 0) 0x7ff858023070 con 0x7ff870157490 2024-01-27T00:46:29.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.938+0000 7ff8667fc700 1 --2- 172.21.15.46:0/3363973327 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff850066ea0 0x7ff850069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:29.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.938+0000 7ff8667fc700 1 -- 172.21.15.46:0/3363973327 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6211+0+0 (secure 0 0 0) 0x7ff8580858d0 con 0x7ff870157490 2024-01-27T00:46:29.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.939+0000 7ff86e59c700 1 --2- 172.21.15.46:0/3363973327 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff850066ea0 0x7ff850069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:29.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.939+0000 7ff86e59c700 1 --2- 172.21.15.46:0/3363973327 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff850066ea0 0x7ff850069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff8701537e0 tx=0x7ff8600091a0 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:29.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:29.943+0000 7ff8667fc700 1 -- 172.21.15.46:0/3363973327 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff858052a20 con 0x7ff870157490 2024-01-27T00:46:30.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.148+0000 7ff874d85700 1 -- 172.21.15.46:0/3363973327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff8700c42f0 con 0x7ff870157490 2024-01-27T00:46:30.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.152+0000 7ff8667fc700 1 -- 172.21.15.46:0/3363973327 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v55) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff8580150a0 con 0x7ff870157490 2024-01-27T00:46:30.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.154+0000 7ff874d85700 1 -- 172.21.15.46:0/3363973327 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff850066ea0 msgr2=0x7ff850069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:30.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.154+0000 7ff874d85700 1 --2- 172.21.15.46:0/3363973327 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff850066ea0 0x7ff850069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff8701537e0 tx=0x7ff8600091a0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.154+0000 7ff874d85700 1 -- 172.21.15.46:0/3363973327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff870157490 msgr2=0x7ff870169920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:30.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.154+0000 7ff874d85700 1 --2- 172.21.15.46:0/3363973327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff870157490 0x7ff870169920 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7ff8700adc40 tx=0x7ff858017000 comp rx=0 tx=0).stop 2024-01-27T00:46:30.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.154+0000 7ff874d85700 1 -- 172.21.15.46:0/3363973327 shutdown_connections 2024-01-27T00:46:30.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.154+0000 7ff874d85700 1 --2- 172.21.15.46:0/3363973327 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7ff850066ea0 0x7ff850069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.154+0000 7ff874d85700 1 --2- 172.21.15.46:0/3363973327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff870157860 0x7ff8700697c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.154+0000 7ff874d85700 1 --2- 172.21.15.46:0/3363973327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff87006d6a0 0x7ff870150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.154+0000 7ff874d85700 1 --2- 172.21.15.46:0/3363973327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff870157490 0x7ff870169920 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.154+0000 7ff874d85700 1 -- 172.21.15.46:0/3363973327 >> 172.21.15.46:0/3363973327 conn(0x7ff870067930 msgr2=0x7ff8700b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:30.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.154+0000 7ff874d85700 1 -- 172.21.15.46:0/3363973327 shutdown_connections 2024-01-27T00:46:30.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.154+0000 7ff874d85700 1 -- 172.21.15.46:0/3363973327 wait complete. 2024-01-27T00:46:30.167 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-27T00:46:30.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.301+0000 7f61b3072700 1 Processor -- start 2024-01-27T00:46:30.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.301+0000 7f61b3072700 1 -- start start 2024-01-27T00:46:30.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.301+0000 7f61b3072700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac0c93a0 0x7f61ac0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:30.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.301+0000 7f61b3072700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f61ac067d50 0x7f61ac159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:30.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.301+0000 7f61b3072700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac15e390 con 0x7f61ac0681c0 2024-01-27T00:46:30.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.301+0000 7f61b3072700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61ac15d810 con 0x7f61ac067d50 2024-01-27T00:46:30.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.301+0000 7f61b3072700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac15daf0 con 0x7f61ac0c93a0 2024-01-27T00:46:30.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.302+0000 7f61b160f700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f61ac0681c0 0x7f61ac159760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46168/0 (socket says 172.21.15.46:46168) 2024-01-27T00:46:30.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.302+0000 7f61b160f700 1 -- 172.21.15.46:0/3771344959 learned_addr learned my addr 172.21.15.46:0/3771344959 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:30.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.302+0000 7f61b0e0e700 1 --2- 172.21.15.46:0/3771344959 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac0c93a0 0x7f61ac0c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:30.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.302+0000 7f61abfff700 1 --2- 172.21.15.46:0/3771344959 >> v2:172.21.15.106:3300/0 conn(0x7f61ac067d50 0x7f61ac159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:30.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.302+0000 7f61ab7fe700 1 -- 172.21.15.46:0/3771344959 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505812684 0 0) 0x7f61ac15e390 con 0x7f61ac0681c0 2024-01-27T00:46:30.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.302+0000 7f61ab7fe700 1 -- 172.21.15.46:0/3771344959 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f619c003570 con 0x7f61ac0681c0 2024-01-27T00:46:30.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.302+0000 7f61b0e0e700 1 -- 172.21.15.46:0/3771344959 >> v2:172.21.15.106:3300/0 conn(0x7f61ac067d50 msgr2=0x7f61ac159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:30.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.302+0000 7f61b0e0e700 1 --2- 172.21.15.46:0/3771344959 >> v2:172.21.15.106:3300/0 conn(0x7f61ac067d50 0x7f61ac159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.302+0000 7f61b0e0e700 1 -- 172.21.15.46:0/3771344959 >> v1:172.21.15.46:6789/0 conn(0x7f61ac0681c0 legacy=0x7f61ac159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:30.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.302+0000 7f61b0e0e700 1 -- 172.21.15.46:0/3771344959 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac15ddd0 con 0x7f61ac0c93a0 2024-01-27T00:46:30.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.302+0000 7f61abfff700 1 --2- 172.21.15.46:0/3771344959 >> v2:172.21.15.106:3300/0 conn(0x7f61ac067d50 0x7f61ac159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:30.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.302+0000 7f61b0e0e700 1 --2- 172.21.15.46:0/3771344959 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac0c93a0 0x7f61ac0c9770 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f61a00098e0 tx=0x7f61a00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6176977e5bcef0a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:30.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.303+0000 7f61ab7fe700 1 -- 172.21.15.46:0/3771344959 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a0011630 con 0x7f61ac0c93a0 2024-01-27T00:46:30.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.303+0000 7f61ab7fe700 1 -- 172.21.15.46:0/3771344959 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a0011790 con 0x7f61ac0c93a0 2024-01-27T00:46:30.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.303+0000 7f61ab7fe700 1 -- 172.21.15.46:0/3771344959 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a0011b10 con 0x7f61ac0c93a0 2024-01-27T00:46:30.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.303+0000 7f61b3072700 1 -- 172.21.15.46:0/3771344959 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac0c93a0 msgr2=0x7f61ac0c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:30.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.303+0000 7f61b3072700 1 --2- 172.21.15.46:0/3771344959 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac0c93a0 0x7f61ac0c9770 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f61a00098e0 tx=0x7f61a00091a0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.303+0000 7f61b3072700 1 -- 172.21.15.46:0/3771344959 shutdown_connections 2024-01-27T00:46:30.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.303+0000 7f61b3072700 1 --2- 172.21.15.46:0/3771344959 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac0c93a0 0x7f61ac0c9770 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.303+0000 7f61b3072700 1 --2- 172.21.15.46:0/3771344959 >> v2:172.21.15.106:3300/0 conn(0x7f61ac067d50 0x7f61ac159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.303+0000 7f61b3072700 1 -- 172.21.15.46:0/3771344959 >> 172.21.15.46:0/3771344959 conn(0x7f61ac0ca9a0 msgr2=0x7f61ac068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:30.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.303+0000 7f61b3072700 1 -- 172.21.15.46:0/3771344959 shutdown_connections 2024-01-27T00:46:30.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.303+0000 7f61b3072700 1 -- 172.21.15.46:0/3771344959 wait complete. 2024-01-27T00:46:30.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.304+0000 7f61b3072700 1 Processor -- start 2024-01-27T00:46:30.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.305+0000 7f61b3072700 1 -- start start 2024-01-27T00:46:30.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.305+0000 7f61b3072700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac067d50 0x7f61ac150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.305+0000 7f61b3072700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61ac0681c0 0x7f61ac169ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.305+0000 7f61b3072700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61ac0c93a0 0x7f61ac14a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.305+0000 7f61b3072700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac16d850 con 0x7f61ac0681c0 2024-01-27T00:46:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.305+0000 7f61b3072700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac16d520 con 0x7f61ac0c93a0 2024-01-27T00:46:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.305+0000 7f61b3072700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac16a750 con 0x7f61ac067d50 2024-01-27T00:46:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.306+0000 7f61abfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61ac0681c0 0x7f61ac169ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.306+0000 7f61abfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61ac0681c0 0x7f61ac169ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49494/0 (socket says 172.21.15.46:49494) 2024-01-27T00:46:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.306+0000 7f61abfff700 1 -- 172.21.15.46:0/2155368460 learned_addr learned my addr 172.21.15.46:0/2155368460 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.306+0000 7f61b160f700 1 --2- 172.21.15.46:0/2155368460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61ac0c93a0 0x7f61ac14a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.306+0000 7f61b0e0e700 1 --2- 172.21.15.46:0/2155368460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac067d50 0x7f61ac150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.306+0000 7f61abfff700 1 -- 172.21.15.46:0/2155368460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac067d50 msgr2=0x7f61ac150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.306+0000 7f61abfff700 1 --2- 172.21.15.46:0/2155368460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac067d50 0x7f61ac150d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.306+0000 7f61abfff700 1 -- 172.21.15.46:0/2155368460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61ac0c93a0 msgr2=0x7f61ac14a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.306+0000 7f61abfff700 1 --2- 172.21.15.46:0/2155368460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61ac0c93a0 0x7f61ac14a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.306+0000 7f61abfff700 1 -- 172.21.15.46:0/2155368460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6198009420 con 0x7f61ac0681c0 2024-01-27T00:46:30.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.306+0000 7f61abfff700 1 --2- 172.21.15.46:0/2155368460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61ac0681c0 0x7f61ac169ed0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f619800ec80 tx=0x7f61980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:30.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.306+0000 7f61a8ff9700 1 -- 172.21.15.46:0/2155368460 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619800c920 con 0x7f61ac0681c0 2024-01-27T00:46:30.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.306+0000 7f61a8ff9700 1 -- 172.21.15.46:0/2155368460 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619800ca80 con 0x7f61ac0681c0 2024-01-27T00:46:30.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.306+0000 7f61b3072700 1 -- 172.21.15.46:0/2155368460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a0009420 con 0x7f61ac0681c0 2024-01-27T00:46:30.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.306+0000 7f61a8ff9700 1 -- 172.21.15.46:0/2155368460 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619800ce00 con 0x7f61ac0681c0 2024-01-27T00:46:30.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.306+0000 7f61b3072700 1 -- 172.21.15.46:0/2155368460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ac16ad10 con 0x7f61ac0681c0 2024-01-27T00:46:30.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.307+0000 7f61b3072700 1 -- 172.21.15.46:0/2155368460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61ac16d9c0 con 0x7f61ac0681c0 2024-01-27T00:46:30.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.308+0000 7f61a8ff9700 1 -- 172.21.15.46:0/2155368460 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 87) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f6198013680 con 0x7f61ac0681c0 2024-01-27T00:46:30.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.308+0000 7f61a8ff9700 1 --2- 172.21.15.46:0/2155368460 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6190066ea0 0x7f6190069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:30.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.308+0000 7f61a8ff9700 1 -- 172.21.15.46:0/2155368460 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f6198084a90 con 0x7f61ac0681c0 2024-01-27T00:46:30.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.308+0000 7f61b0e0e700 1 --2- 172.21.15.46:0/2155368460 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6190066ea0 0x7f6190069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:30.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.309+0000 7f61b0e0e700 1 --2- 172.21.15.46:0/2155368460 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6190066ea0 0x7f6190069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f61a00093a0 tx=0x7f61a0017000 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:30.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.313+0000 7f61a8ff9700 1 -- 172.21.15.46:0/2155368460 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6198051b40 con 0x7f61ac0681c0 2024-01-27T00:46:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.519+0000 7f61b3072700 1 -- 172.21.15.46:0/2155368460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f61ac04c710 con 0x7f61ac0681c0 2024-01-27T00:46:30.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.522+0000 7f61a8ff9700 1 -- 172.21.15.46:0/2155368460 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v56) v1 ==== 120+0+0 (secure 0 0 0) 0x7f6198055160 con 0x7f61ac0681c0 2024-01-27T00:46:30.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.524+0000 7f61b3072700 1 -- 172.21.15.46:0/2155368460 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6190066ea0 msgr2=0x7f6190069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:30.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.524+0000 7f61b3072700 1 --2- 172.21.15.46:0/2155368460 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6190066ea0 0x7f6190069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f61a00093a0 tx=0x7f61a0017000 comp rx=0 tx=0).stop 2024-01-27T00:46:30.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.524+0000 7f61b3072700 1 -- 172.21.15.46:0/2155368460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61ac0681c0 msgr2=0x7f61ac169ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:30.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.524+0000 7f61b3072700 1 --2- 172.21.15.46:0/2155368460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61ac0681c0 0x7f61ac169ed0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f619800ec80 tx=0x7f61980091a0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.524+0000 7f61b3072700 1 -- 172.21.15.46:0/2155368460 shutdown_connections 2024-01-27T00:46:30.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.524+0000 7f61b3072700 1 --2- 172.21.15.46:0/2155368460 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6190066ea0 0x7f6190069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.524+0000 7f61b3072700 1 --2- 172.21.15.46:0/2155368460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61ac0c93a0 0x7f61ac14a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.524+0000 7f61b3072700 1 --2- 172.21.15.46:0/2155368460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac067d50 0x7f61ac150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.524+0000 7f61b3072700 1 --2- 172.21.15.46:0/2155368460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61ac0681c0 0x7f61ac169ed0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.524+0000 7f61b3072700 1 -- 172.21.15.46:0/2155368460 >> 172.21.15.46:0/2155368460 conn(0x7f61ac0ca9a0 msgr2=0x7f61ac0688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:30.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.524+0000 7f61b3072700 1 -- 172.21.15.46:0/2155368460 shutdown_connections 2024-01-27T00:46:30.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.524+0000 7f61b3072700 1 -- 172.21.15.46:0/2155368460 wait complete. 2024-01-27T00:46:30.537 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-27T00:46:30.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.671+0000 7f458a924700 1 Processor -- start 2024-01-27T00:46:30.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.671+0000 7f458a924700 1 -- start start 2024-01-27T00:46:30.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.672+0000 7f458a924700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4584068ed0 0x7f458406a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:30.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.672+0000 7f458a924700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45840a9a80 con 0x7f4584068b00 2024-01-27T00:46:30.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.672+0000 7f458a924700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f458415a680 con 0x7f458406add0 2024-01-27T00:46:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.672+0000 7f458a924700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4584069320 con 0x7f4584068ed0 2024-01-27T00:46:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.672+0000 7f4583fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4584068b00 0x7f4584159ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43358/0 (socket says 172.21.15.46:43358) 2024-01-27T00:46:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.673+0000 7f4583fff700 1 -- 172.21.15.46:0/3476744404 learned_addr learned my addr 172.21.15.46:0/3476744404 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.673+0000 7f45837fe700 1 --2- 172.21.15.46:0/3476744404 >> v2:172.21.15.106:3300/0 conn(0x7f4584068ed0 0x7f458406a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.673+0000 7f4582ffd700 1 -- 172.21.15.46:0/3476744404 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698676951 0 0) 0x7f45840a9a80 con 0x7f4584068b00 2024-01-27T00:46:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.673+0000 7f4582ffd700 1 -- 172.21.15.46:0/3476744404 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f456c003570 con 0x7f4584068b00 2024-01-27T00:46:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.673+0000 7f4582ffd700 1 -- 172.21.15.46:0/3476744404 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639520590 0 0) 0x7f458415a680 con 0x7f458406add0 2024-01-27T00:46:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.673+0000 7f4582ffd700 1 -- 172.21.15.46:0/3476744404 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45840a9a80 con 0x7f458406add0 2024-01-27T00:46:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.673+0000 7f45837fe700 1 -- 172.21.15.46:0/3476744404 >> v1:172.21.15.106:6789/0 conn(0x7f458406add0 legacy=0x7f45840a9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.673+0000 7f45837fe700 1 -- 172.21.15.46:0/3476744404 >> v1:172.21.15.46:6789/0 conn(0x7f4584068b00 legacy=0x7f4584159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.673+0000 7f45837fe700 1 -- 172.21.15.46:0/3476744404 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458415a960 con 0x7f4584068ed0 2024-01-27T00:46:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.673+0000 7f45837fe700 1 --2- 172.21.15.46:0/3476744404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4584068ed0 0x7f458406a660 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f45780098e0 tx=0x7f45780091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a72b14235dc4f72 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.673+0000 7f4582ffd700 1 -- 172.21.15.46:0/3476744404 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578011630 con 0x7f4584068ed0 2024-01-27T00:46:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.674+0000 7f4582ffd700 1 -- 172.21.15.46:0/3476744404 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4578011790 con 0x7f4584068ed0 2024-01-27T00:46:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.674+0000 7f4582ffd700 1 -- 172.21.15.46:0/3476744404 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578011b30 con 0x7f4584068ed0 2024-01-27T00:46:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.674+0000 7f458a924700 1 -- 172.21.15.46:0/3476744404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4584068ed0 msgr2=0x7f458406a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.674+0000 7f458a924700 1 --2- 172.21.15.46:0/3476744404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4584068ed0 0x7f458406a660 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f45780098e0 tx=0x7f45780091a0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.674+0000 7f458a924700 1 -- 172.21.15.46:0/3476744404 shutdown_connections 2024-01-27T00:46:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.674+0000 7f458a924700 1 --2- 172.21.15.46:0/3476744404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4584068ed0 0x7f458406a660 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.674+0000 7f458a924700 1 -- 172.21.15.46:0/3476744404 >> 172.21.15.46:0/3476744404 conn(0x7f45840ca100 msgr2=0x7f458406e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.674+0000 7f458a924700 1 -- 172.21.15.46:0/3476744404 shutdown_connections 2024-01-27T00:46:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.674+0000 7f458a924700 1 -- 172.21.15.46:0/3476744404 wait complete. 2024-01-27T00:46:30.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.675+0000 7f458a924700 1 Processor -- start 2024-01-27T00:46:30.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.676+0000 7f458a924700 1 -- start start 2024-01-27T00:46:30.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.676+0000 7f458a924700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4584068b00 0x7f458416a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:30.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.676+0000 7f458a924700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4584068ed0 0x7f45840b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:30.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.676+0000 7f458a924700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f458406add0 0x7f45840b1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:30.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.676+0000 7f458a924700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458416ada0 con 0x7f4584068ed0 2024-01-27T00:46:30.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.676+0000 7f458a924700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45840b2270 con 0x7f458406add0 2024-01-27T00:46:30.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.676+0000 7f458a924700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4584169f10 con 0x7f4584068b00 2024-01-27T00:46:30.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.676+0000 7f45837fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4584068ed0 0x7f45840b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:30.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.676+0000 7f45837fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4584068ed0 0x7f45840b0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43526/0 (socket says 172.21.15.46:43526) 2024-01-27T00:46:30.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.676+0000 7f45837fe700 1 -- 172.21.15.46:0/3187968721 learned_addr learned my addr 172.21.15.46:0/3187968721 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:30.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.677+0000 7f4583fff700 1 --2- 172.21.15.46:0/3187968721 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4584068b00 0x7f458416a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:30.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.677+0000 7f4588ec1700 1 --2- 172.21.15.46:0/3187968721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f458406add0 0x7f45840b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:30.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.677+0000 7f45837fe700 1 -- 172.21.15.46:0/3187968721 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4584068b00 msgr2=0x7f458416a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:30.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.677+0000 7f45837fe700 1 --2- 172.21.15.46:0/3187968721 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4584068b00 0x7f458416a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.677+0000 7f45837fe700 1 -- 172.21.15.46:0/3187968721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f458406add0 msgr2=0x7f45840b1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:30.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.677+0000 7f45837fe700 1 --2- 172.21.15.46:0/3187968721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f458406add0 0x7f45840b1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.677+0000 7f45837fe700 1 -- 172.21.15.46:0/3187968721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4578009420 con 0x7f4584068ed0 2024-01-27T00:46:30.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.677+0000 7f4588ec1700 1 --2- 172.21.15.46:0/3187968721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f458406add0 0x7f45840b1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:46:30.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.677+0000 7f45837fe700 1 --2- 172.21.15.46:0/3187968721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4584068ed0 0x7f45840b0af0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f4584069d70 tx=0x7f4578017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:30.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.677+0000 7f456bfff700 1 -- 172.21.15.46:0/3187968721 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457801f070 con 0x7f4584068ed0 2024-01-27T00:46:30.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.677+0000 7f456bfff700 1 -- 172.21.15.46:0/3187968721 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4578000bc0 con 0x7f4584068ed0 2024-01-27T00:46:30.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.677+0000 7f458a924700 1 -- 172.21.15.46:0/3187968721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45840b1660 con 0x7f4584068ed0 2024-01-27T00:46:30.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.677+0000 7f456bfff700 1 -- 172.21.15.46:0/3187968721 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578025420 con 0x7f4584068ed0 2024-01-27T00:46:30.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.677+0000 7f458a924700 1 -- 172.21.15.46:0/3187968721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45840b1c20 con 0x7f4584068ed0 2024-01-27T00:46:30.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.678+0000 7f458a924700 1 -- 172.21.15.46:0/3187968721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45840b23e0 con 0x7f4584068ed0 2024-01-27T00:46:30.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.679+0000 7f456bfff700 1 -- 172.21.15.46:0/3187968721 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 87) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f4578023070 con 0x7f4584068ed0 2024-01-27T00:46:30.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.679+0000 7f456bfff700 1 --2- 172.21.15.46:0/3187968721 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4570066ea0 0x7f4570069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:30.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.679+0000 7f456bfff700 1 -- 172.21.15.46:0/3187968721 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f4578085940 con 0x7f4584068ed0 2024-01-27T00:46:30.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.679+0000 7f4583fff700 1 --2- 172.21.15.46:0/3187968721 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4570066ea0 0x7f4570069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:30.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.680+0000 7f4583fff700 1 --2- 172.21.15.46:0/3187968721 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4570066ea0 0x7f4570069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4574001360 tx=0x7f4574009040 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:30.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.684+0000 7f456bfff700 1 -- 172.21.15.46:0/3187968721 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4578052a90 con 0x7f4584068ed0 2024-01-27T00:46:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.890+0000 7f458a924700 1 -- 172.21.15.46:0/3187968721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f458404c710 con 0x7f4584068ed0 2024-01-27T00:46:30.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.894+0000 7f456bfff700 1 -- 172.21.15.46:0/3187968721 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v57) v1 ==== 115+0+0 (secure 0 0 0) 0x7f45780560b0 con 0x7f4584068ed0 2024-01-27T00:46:30.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.897+0000 7f458a924700 1 -- 172.21.15.46:0/3187968721 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4570066ea0 msgr2=0x7f4570069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:30.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.897+0000 7f458a924700 1 --2- 172.21.15.46:0/3187968721 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4570066ea0 0x7f4570069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4574001360 tx=0x7f4574009040 comp rx=0 tx=0).stop 2024-01-27T00:46:30.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.897+0000 7f458a924700 1 -- 172.21.15.46:0/3187968721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4584068ed0 msgr2=0x7f45840b0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:30.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.897+0000 7f458a924700 1 --2- 172.21.15.46:0/3187968721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4584068ed0 0x7f45840b0af0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f4584069d70 tx=0x7f4578017000 comp rx=0 tx=0).stop 2024-01-27T00:46:30.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.897+0000 7f458a924700 1 -- 172.21.15.46:0/3187968721 shutdown_connections 2024-01-27T00:46:30.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.897+0000 7f458a924700 1 --2- 172.21.15.46:0/3187968721 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4570066ea0 0x7f4570069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.897+0000 7f458a924700 1 --2- 172.21.15.46:0/3187968721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f458406add0 0x7f45840b1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.897+0000 7f458a924700 1 --2- 172.21.15.46:0/3187968721 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4584068b00 0x7f458416a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.897+0000 7f458a924700 1 --2- 172.21.15.46:0/3187968721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4584068ed0 0x7f45840b0af0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:30.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.897+0000 7f458a924700 1 -- 172.21.15.46:0/3187968721 >> 172.21.15.46:0/3187968721 conn(0x7f45840ca100 msgr2=0x7f45840cb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:30.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.897+0000 7f458a924700 1 -- 172.21.15.46:0/3187968721 shutdown_connections 2024-01-27T00:46:30.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:30.897+0000 7f458a924700 1 -- 172.21.15.46:0/3187968721 wait complete. 2024-01-27T00:46:30.909 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-01-27T00:46:31.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.042+0000 7f0e8d073700 1 Processor -- start 2024-01-27T00:46:31.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.042+0000 7f0e8d073700 1 -- start start 2024-01-27T00:46:31.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.043+0000 7f0e8d073700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e88153ec0 0x7f0e88154290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:31.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.043+0000 7f0e8d073700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0e88157630 0x7f0e880b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:31.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.043+0000 7f0e8d073700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e880b1500 con 0x7f0e880b2100 2024-01-27T00:46:31.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.043+0000 7f0e8d073700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e880b2560 con 0x7f0e88153ec0 2024-01-27T00:46:31.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.043+0000 7f0e8d073700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8815b140 con 0x7f0e88157630 2024-01-27T00:46:31.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.043+0000 7f0e86d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e88153ec0 0x7f0e88154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.043+0000 7f0e86d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e88153ec0 0x7f0e88154290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43528/0 (socket says 172.21.15.46:43528) 2024-01-27T00:46:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.043+0000 7f0e86d9d700 1 -- 172.21.15.46:0/610647896 learned_addr learned my addr 172.21.15.46:0/610647896 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.043+0000 7f0e8659c700 1 --2- 172.21.15.46:0/610647896 >> v2:172.21.15.106:3300/0 conn(0x7f0e88157630 0x7f0e880b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.043+0000 7f0e86d9d700 1 -- 172.21.15.46:0/610647896 >> v2:172.21.15.106:3300/0 conn(0x7f0e88157630 msgr2=0x7f0e880b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.043+0000 7f0e86d9d700 1 --2- 172.21.15.46:0/610647896 >> v2:172.21.15.106:3300/0 conn(0x7f0e88157630 0x7f0e880b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.044+0000 7f0e86d9d700 1 -- 172.21.15.46:0/610647896 >> v1:172.21.15.106:6789/0 conn(0x7f0e880b2100 legacy=0x7f0e8806eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.044+0000 7f0e86d9d700 1 -- 172.21.15.46:0/610647896 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8815a5c0 con 0x7f0e88153ec0 2024-01-27T00:46:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.044+0000 7f0e86d9d700 1 --2- 172.21.15.46:0/610647896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e88153ec0 0x7f0e88154290 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f0e8806e2c0 tx=0x7f0e7000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=973a4b748b78d3d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.044+0000 7f0e85d9b700 1 -- 172.21.15.46:0/610647896 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7000dba0 con 0x7f0e88153ec0 2024-01-27T00:46:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.044+0000 7f0e85d9b700 1 -- 172.21.15.46:0/610647896 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7000dd00 con 0x7f0e88153ec0 2024-01-27T00:46:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.044+0000 7f0e85d9b700 1 -- 172.21.15.46:0/610647896 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e700196d0 con 0x7f0e88153ec0 2024-01-27T00:46:31.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.044+0000 7f0e8d073700 1 -- 172.21.15.46:0/610647896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e88153ec0 msgr2=0x7f0e88154290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:31.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.044+0000 7f0e8d073700 1 --2- 172.21.15.46:0/610647896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e88153ec0 0x7f0e88154290 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f0e8806e2c0 tx=0x7f0e7000b4f0 comp rx=0 tx=0).stop 2024-01-27T00:46:31.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.045+0000 7f0e8d073700 1 -- 172.21.15.46:0/610647896 shutdown_connections 2024-01-27T00:46:31.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.045+0000 7f0e8d073700 1 --2- 172.21.15.46:0/610647896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e88153ec0 0x7f0e88154290 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:31.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.045+0000 7f0e8d073700 1 --2- 172.21.15.46:0/610647896 >> v2:172.21.15.106:3300/0 conn(0x7f0e88157630 0x7f0e880b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:31.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.045+0000 7f0e8d073700 1 -- 172.21.15.46:0/610647896 >> 172.21.15.46:0/610647896 conn(0x7f0e88069510 msgr2=0x7f0e880aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:31.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.045+0000 7f0e8d073700 1 -- 172.21.15.46:0/610647896 shutdown_connections 2024-01-27T00:46:31.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.045+0000 7f0e8d073700 1 -- 172.21.15.46:0/610647896 wait complete. 2024-01-27T00:46:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.046+0000 7f0e8d073700 1 Processor -- start 2024-01-27T00:46:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.046+0000 7f0e8d073700 1 -- start start 2024-01-27T00:46:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.047+0000 7f0e8d073700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880b2100 0x7f0e881656f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.047+0000 7f0e8d073700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e88153ec0 0x7f0e881596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.047+0000 7f0e8d073700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e88157630 0x7f0e88159c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.047+0000 7f0e8d073700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e88150310 con 0x7f0e880b2100 2024-01-27T00:46:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.047+0000 7f0e8d073700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e881505f0 con 0x7f0e88157630 2024-01-27T00:46:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.047+0000 7f0e8d073700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e88165f80 con 0x7f0e88153ec0 2024-01-27T00:46:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.047+0000 7f0e86d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880b2100 0x7f0e881656f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.047+0000 7f0e86d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880b2100 0x7f0e881656f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43544/0 (socket says 172.21.15.46:43544) 2024-01-27T00:46:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.047+0000 7f0e86d9d700 1 -- 172.21.15.46:0/1852116217 learned_addr learned my addr 172.21.15.46:0/1852116217 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.047+0000 7f0e8759e700 1 --2- 172.21.15.46:0/1852116217 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e88157630 0x7f0e88159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.047+0000 7f0e8659c700 1 --2- 172.21.15.46:0/1852116217 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e88153ec0 0x7f0e881596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.047+0000 7f0e86d9d700 1 -- 172.21.15.46:0/1852116217 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e88153ec0 msgr2=0x7f0e881596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.047+0000 7f0e86d9d700 1 --2- 172.21.15.46:0/1852116217 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e88153ec0 0x7f0e881596c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.047+0000 7f0e86d9d700 1 -- 172.21.15.46:0/1852116217 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e88157630 msgr2=0x7f0e88159c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.047+0000 7f0e86d9d700 1 --2- 172.21.15.46:0/1852116217 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e88157630 0x7f0e88159c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.047+0000 7f0e86d9d700 1 -- 172.21.15.46:0/1852116217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7000b790 con 0x7f0e880b2100 2024-01-27T00:46:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.047+0000 7f0e8759e700 1 --2- 172.21.15.46:0/1852116217 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e88157630 0x7f0e88159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:46:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.048+0000 7f0e86d9d700 1 --2- 172.21.15.46:0/1852116217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880b2100 0x7f0e881656f0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000b760 tx=0x7f0e7001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.048+0000 7f0e7effd700 1 -- 172.21.15.46:0/1852116217 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e70010470 con 0x7f0e880b2100 2024-01-27T00:46:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.048+0000 7f0e7effd700 1 -- 172.21.15.46:0/1852116217 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e700105d0 con 0x7f0e880b2100 2024-01-27T00:46:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.048+0000 7f0e8d073700 1 -- 172.21.15.46:0/1852116217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e88166260 con 0x7f0e880b2100 2024-01-27T00:46:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.048+0000 7f0e7effd700 1 -- 172.21.15.46:0/1852116217 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e70010990 con 0x7f0e880b2100 2024-01-27T00:46:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.048+0000 7f0e8d073700 1 -- 172.21.15.46:0/1852116217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e88165c30 con 0x7f0e880b2100 2024-01-27T00:46:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.049+0000 7f0e8d073700 1 -- 172.21.15.46:0/1852116217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e8804c710 con 0x7f0e880b2100 2024-01-27T00:46:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.049+0000 7f0e7effd700 1 -- 172.21.15.46:0/1852116217 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 87) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f0e70022030 con 0x7f0e880b2100 2024-01-27T00:46:31.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.050+0000 7f0e7effd700 1 --2- 172.21.15.46:0/1852116217 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0e68066ea0 0x7f0e68069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:31.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.050+0000 7f0e7effd700 1 -- 172.21.15.46:0/1852116217 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f0e70014070 con 0x7f0e880b2100 2024-01-27T00:46:31.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.050+0000 7f0e8659c700 1 --2- 172.21.15.46:0/1852116217 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0e68066ea0 0x7f0e68069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:31.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.051+0000 7f0e8659c700 1 --2- 172.21.15.46:0/1852116217 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0e68066ea0 0x7f0e68069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0e780093f0 tx=0x7f0e78008040 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:31.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.055+0000 7f0e7effd700 1 -- 172.21.15.46:0/1852116217 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0e700558c0 con 0x7f0e880b2100 2024-01-27T00:46:31.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.343+0000 7f0e8d073700 1 -- 172.21.15.46:0/1852116217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f0e880ce060 con 0x7f0e880b2100 2024-01-27T00:46:31.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.500+0000 7f0e7effd700 1 -- 172.21.15.46:0/1852116217 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2489) v1 ==== 178+0+0 (secure 0 0 0) 0x7f0e70058ee0 con 0x7f0e880b2100 2024-01-27T00:46:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.502+0000 7f0e8d073700 1 -- 172.21.15.46:0/1852116217 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0e68066ea0 msgr2=0x7f0e68069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.502+0000 7f0e8d073700 1 --2- 172.21.15.46:0/1852116217 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0e68066ea0 0x7f0e68069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0e780093f0 tx=0x7f0e78008040 comp rx=0 tx=0).stop 2024-01-27T00:46:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.502+0000 7f0e8d073700 1 -- 172.21.15.46:0/1852116217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880b2100 msgr2=0x7f0e881656f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.502+0000 7f0e8d073700 1 --2- 172.21.15.46:0/1852116217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880b2100 0x7f0e881656f0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000b760 tx=0x7f0e7001b000 comp rx=0 tx=0).stop 2024-01-27T00:46:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.502+0000 7f0e8d073700 1 -- 172.21.15.46:0/1852116217 shutdown_connections 2024-01-27T00:46:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.502+0000 7f0e8d073700 1 --2- 172.21.15.46:0/1852116217 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0e68066ea0 0x7f0e68069350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.502+0000 7f0e8d073700 1 --2- 172.21.15.46:0/1852116217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e880b2100 0x7f0e881656f0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.502+0000 7f0e8d073700 1 --2- 172.21.15.46:0/1852116217 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e88157630 0x7f0e88159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.502+0000 7f0e8d073700 1 --2- 172.21.15.46:0/1852116217 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e88153ec0 0x7f0e881596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.502+0000 7f0e8d073700 1 -- 172.21.15.46:0/1852116217 >> 172.21.15.46:0/1852116217 conn(0x7f0e88069510 msgr2=0x7f0e88154800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.502+0000 7f0e8d073700 1 -- 172.21.15.46:0/1852116217 shutdown_connections 2024-01-27T00:46:31.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.502+0000 7f0e8d073700 1 -- 172.21.15.46:0/1852116217 wait complete. 2024-01-27T00:46:31.516 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2024-01-27T00:46:31.516 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2024-01-27T00:46:31.516 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-27T00:46:31.516 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-01-27T00:46:31.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.653+0000 7f5b06223700 1 Processor -- start 2024-01-27T00:46:31.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.653+0000 7f5b06223700 1 -- start start 2024-01-27T00:46:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.654+0000 7f5b06223700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b000693a0 0x7f5b00069770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.654+0000 7f5b06223700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5b000a8960 0x7f5b000a8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.654+0000 7f5b06223700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b000b2340 con 0x7f5b000b27d0 2024-01-27T00:46:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.654+0000 7f5b06223700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b000b1ae0 con 0x7f5b000a8960 2024-01-27T00:46:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.654+0000 7f5b06223700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b000b0c60 con 0x7f5b000693a0 2024-01-27T00:46:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.654+0000 7f5afffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5b000b27d0 0x7f5b000b2c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43374/0 (socket says 172.21.15.46:43374) 2024-01-27T00:46:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.654+0000 7f5afffff700 1 -- 172.21.15.46:0/605457955 learned_addr learned my addr 172.21.15.46:0/605457955 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.654+0000 7f5afeffd700 1 --2- 172.21.15.46:0/605457955 >> v2:172.21.15.106:3300/0 conn(0x7f5b000a8960 0x7f5b000a8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.654+0000 7f5aff7fe700 1 --2- 172.21.15.46:0/605457955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b000693a0 0x7f5b00069770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.654+0000 7f5afe7fc700 1 -- 172.21.15.46:0/605457955 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867080749 0 0) 0x7f5b000b2340 con 0x7f5b000b27d0 2024-01-27T00:46:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.654+0000 7f5afe7fc700 1 -- 172.21.15.46:0/605457955 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae8003570 con 0x7f5b000b27d0 2024-01-27T00:46:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afe7fc700 1 -- 172.21.15.46:0/605457955 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1917647496 0 0) 0x7f5ae8003570 con 0x7f5b000b27d0 2024-01-27T00:46:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afe7fc700 1 -- 172.21.15.46:0/605457955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b000693a0 msgr2=0x7f5b00069770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afe7fc700 1 --2- 172.21.15.46:0/605457955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b000693a0 0x7f5b00069770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afe7fc700 1 -- 172.21.15.46:0/605457955 >> v2:172.21.15.106:3300/0 conn(0x7f5b000a8960 msgr2=0x7f5b000a8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afe7fc700 1 --2- 172.21.15.46:0/605457955 >> v2:172.21.15.106:3300/0 conn(0x7f5b000a8960 0x7f5b000a8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afe7fc700 1 -- 172.21.15.46:0/605457955 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b000b0980 con 0x7f5b000b27d0 2024-01-27T00:46:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afe7fc700 1 -- 172.21.15.46:0/605457955 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f5af80017f0 con 0x7f5b000b27d0 2024-01-27T00:46:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afe7fc700 1 -- 172.21.15.46:0/605457955 >> v1:172.21.15.46:6789/0 conn(0x7f5b000b27d0 legacy=0x7f5b000b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afe7fc700 1 --2- 172.21.15.46:0/605457955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ae8003a60 0x7f5ae8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afe7fc700 1 --2- 172.21.15.46:0/605457955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b000693a0 0x7f5ae8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afe7fc700 1 --2- 172.21.15.46:0/605457955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ae8004d50 0x7f5ae80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afe7fc700 1 -- 172.21.15.46:0/605457955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b000b1ae0 con 0x7f5ae8004d50 2024-01-27T00:46:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afe7fc700 1 -- 172.21.15.46:0/605457955 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b000b0c60 con 0x7f5ae8003a60 2024-01-27T00:46:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afe7fc700 1 -- 172.21.15.46:0/605457955 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae800ba60 con 0x7f5b000693a0 2024-01-27T00:46:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5aff7fe700 1 --2- 172.21.15.46:0/605457955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ae8003a60 0x7f5ae8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afeffd700 1 --2- 172.21.15.46:0/605457955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b000693a0 0x7f5ae8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afffff700 1 --2- 172.21.15.46:0/605457955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ae8004d50 0x7f5ae80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afeffd700 1 -- 172.21.15.46:0/605457955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ae8003a60 msgr2=0x7f5ae8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afeffd700 1 --2- 172.21.15.46:0/605457955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ae8003a60 0x7f5ae8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afeffd700 1 -- 172.21.15.46:0/605457955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ae8004d50 msgr2=0x7f5ae80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afeffd700 1 --2- 172.21.15.46:0/605457955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ae8004d50 0x7f5ae80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afeffd700 1 -- 172.21.15.46:0/605457955 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae8003570 con 0x7f5b000693a0 2024-01-27T00:46:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5aff7fe700 1 --2- 172.21.15.46:0/605457955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ae8003a60 0x7f5ae8003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.655+0000 7f5afffff700 1 --2- 172.21.15.46:0/605457955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ae8004d50 0x7f5ae80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.656+0000 7f5afeffd700 1 --2- 172.21.15.46:0/605457955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b000693a0 0x7f5ae8004810 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5af4009bf0 tx=0x7f5af4009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=207f206a72564145 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.656+0000 7f5afe7fc700 1 -- 172.21.15.46:0/605457955 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af4005a00 con 0x7f5b000693a0 2024-01-27T00:46:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.656+0000 7f5afe7fc700 1 -- 172.21.15.46:0/605457955 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af4005b60 con 0x7f5b000693a0 2024-01-27T00:46:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.656+0000 7f5afe7fc700 1 -- 172.21.15.46:0/605457955 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af400e6b0 con 0x7f5b000693a0 2024-01-27T00:46:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.656+0000 7f5b06223700 1 -- 172.21.15.46:0/605457955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b000693a0 msgr2=0x7f5ae8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.656+0000 7f5b06223700 1 --2- 172.21.15.46:0/605457955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b000693a0 0x7f5ae8004810 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5af4009bf0 tx=0x7f5af4009120 comp rx=0 tx=0).stop 2024-01-27T00:46:31.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.656+0000 7f5afffff700 1 -- 172.21.15.46:0/605457955 reap_dead start 2024-01-27T00:46:31.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.656+0000 7f5b06223700 1 -- 172.21.15.46:0/605457955 shutdown_connections 2024-01-27T00:46:31.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.656+0000 7f5b06223700 1 -- 172.21.15.46:0/605457955 >> 172.21.15.46:0/605457955 conn(0x7f5b00087f60 msgr2=0x7f5b000ce5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:31.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.656+0000 7f5b06223700 1 -- 172.21.15.46:0/605457955 shutdown_connections 2024-01-27T00:46:31.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.657+0000 7f5b06223700 1 -- 172.21.15.46:0/605457955 wait complete. 2024-01-27T00:46:31.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.658+0000 7f5b06223700 1 Processor -- start 2024-01-27T00:46:31.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.658+0000 7f5b06223700 1 -- start start 2024-01-27T00:46:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5b06223700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b000693a0 0x7f5b00151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5b06223700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b0015f390 0x7f5b000cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5b06223700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b0006cf40 0x7f5b0006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5b06223700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0006d900 con 0x7f5b0015f390 2024-01-27T00:46:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5b06223700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b000cd4c0 con 0x7f5b000693a0 2024-01-27T00:46:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5b06223700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b000cc8d0 con 0x7f5b0006cf40 2024-01-27T00:46:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5afeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b0015f390 0x7f5b000cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5afeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b0015f390 0x7f5b000cb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43558/0 (socket says 172.21.15.46:43558) 2024-01-27T00:46:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5afeffd700 1 -- 172.21.15.46:0/4072713405 learned_addr learned my addr 172.21.15.46:0/4072713405 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5aff7fe700 1 --2- 172.21.15.46:0/4072713405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b000693a0 0x7f5b00151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5afffff700 1 --2- 172.21.15.46:0/4072713405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b0006cf40 0x7f5b0006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5afeffd700 1 -- 172.21.15.46:0/4072713405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b0006cf40 msgr2=0x7f5b0006d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5afeffd700 1 --2- 172.21.15.46:0/4072713405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b0006cf40 0x7f5b0006d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5afeffd700 1 -- 172.21.15.46:0/4072713405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b000693a0 msgr2=0x7f5b00151a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5afeffd700 1 --2- 172.21.15.46:0/4072713405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b000693a0 0x7f5b00151a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5afeffd700 1 -- 172.21.15.46:0/4072713405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b000ccb90 con 0x7f5b0015f390 2024-01-27T00:46:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5afeffd700 1 --2- 172.21.15.46:0/4072713405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b0015f390 0x7f5b000cb740 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f5af4000df0 tx=0x7f5af4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.659+0000 7f5ae77fe700 1 -- 172.21.15.46:0/4072713405 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af4006240 con 0x7f5b0015f390 2024-01-27T00:46:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.660+0000 7f5ae77fe700 1 -- 172.21.15.46:0/4072713405 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af40063a0 con 0x7f5b0015f390 2024-01-27T00:46:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.660+0000 7f5b06223700 1 -- 172.21.15.46:0/4072713405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b000cce70 con 0x7f5b0015f390 2024-01-27T00:46:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.660+0000 7f5ae77fe700 1 -- 172.21.15.46:0/4072713405 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af4006740 con 0x7f5b0015f390 2024-01-27T00:46:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.660+0000 7f5b06223700 1 -- 172.21.15.46:0/4072713405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b000cd950 con 0x7f5b0015f390 2024-01-27T00:46:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.661+0000 7f5b06223700 1 -- 172.21.15.46:0/4072713405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b000cd650 con 0x7f5b0015f390 2024-01-27T00:46:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.661+0000 7f5ae77fe700 1 -- 172.21.15.46:0/4072713405 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5af4013470 con 0x7f5b0015f390 2024-01-27T00:46:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.661+0000 7f5ae77fe700 1 --2- 172.21.15.46:0/4072713405 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5aec066f60 0x7f5aec069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.662+0000 7f5ae77fe700 1 -- 172.21.15.46:0/4072713405 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f5af4085f40 con 0x7f5b0015f390 2024-01-27T00:46:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.662+0000 7f5aff7fe700 1 --2- 172.21.15.46:0/4072713405 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5aec066f60 0x7f5aec069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:31.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.662+0000 7f5aff7fe700 1 --2- 172.21.15.46:0/4072713405 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5aec066f60 0x7f5aec069410 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5af0006f00 tx=0x7f5af00091a0 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:31.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.666+0000 7f5ae77fe700 1 -- 172.21.15.46:0/4072713405 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5af4053070 con 0x7f5b0015f390 2024-01-27T00:46:31.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:31.954+0000 7f5b06223700 1 -- 172.21.15.46:0/4072713405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f5b0006d900 con 0x7f5b0015f390 2024-01-27T00:46:32.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.501+0000 7f5ae77fe700 1 -- 172.21.15.46:0/4072713405 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2490) v1 ==== 177+0+0 (secure 0 0 0) 0x7f5af4056690 con 0x7f5b0015f390 2024-01-27T00:46:32.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.503+0000 7f5b06223700 1 -- 172.21.15.46:0/4072713405 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5aec066f60 msgr2=0x7f5aec069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:32.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.503+0000 7f5b06223700 1 --2- 172.21.15.46:0/4072713405 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5aec066f60 0x7f5aec069410 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5af0006f00 tx=0x7f5af00091a0 comp rx=0 tx=0).stop 2024-01-27T00:46:32.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.503+0000 7f5b06223700 1 -- 172.21.15.46:0/4072713405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b0015f390 msgr2=0x7f5b000cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:32.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.503+0000 7f5b06223700 1 --2- 172.21.15.46:0/4072713405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b0015f390 0x7f5b000cb740 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f5af4000df0 tx=0x7f5af4019000 comp rx=0 tx=0).stop 2024-01-27T00:46:32.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.503+0000 7f5b06223700 1 -- 172.21.15.46:0/4072713405 shutdown_connections 2024-01-27T00:46:32.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.503+0000 7f5b06223700 1 --2- 172.21.15.46:0/4072713405 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5aec066f60 0x7f5aec069410 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:32.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.503+0000 7f5b06223700 1 --2- 172.21.15.46:0/4072713405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b0006cf40 0x7f5b0006d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:32.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.503+0000 7f5b06223700 1 --2- 172.21.15.46:0/4072713405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b000693a0 0x7f5b00151a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:32.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.504+0000 7f5b06223700 1 --2- 172.21.15.46:0/4072713405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b0015f390 0x7f5b000cb740 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:32.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.504+0000 7f5b06223700 1 -- 172.21.15.46:0/4072713405 >> 172.21.15.46:0/4072713405 conn(0x7f5b00087f60 msgr2=0x7f5b000adf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:32.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.504+0000 7f5b06223700 1 -- 172.21.15.46:0/4072713405 shutdown_connections 2024-01-27T00:46:32.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.504+0000 7f5b06223700 1 -- 172.21.15.46:0/4072713405 wait complete. 2024-01-27T00:46:32.518 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-27T00:46:32.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.658+0000 7f4b647b3700 1 Processor -- start 2024-01-27T00:46:32.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.658+0000 7f4b647b3700 1 -- start start 2024-01-27T00:46:32.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.659+0000 7f4b647b3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b5c1565a0 0x7f4b5c156970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:32.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.659+0000 7f4b647b3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b5c15a520 0x7f4b5c0acc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.659+0000 7f4b647b3700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4b5c0ae2f0 0x7f4b5c0ae750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.659+0000 7f4b647b3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b5c0ae130 con 0x7f4b5c15a520 2024-01-27T00:46:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.659+0000 7f4b647b3700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b5c0b0830 con 0x7f4b5c0ae2f0 2024-01-27T00:46:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.659+0000 7f4b647b3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c0aec90 con 0x7f4b5c1565a0 2024-01-27T00:46:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.659+0000 7f4b61d4e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b5c15a520 0x7f4b5c0acc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.659+0000 7f4b61d4e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b5c15a520 0x7f4b5c0acc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43560/0 (socket says 172.21.15.46:43560) 2024-01-27T00:46:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.659+0000 7f4b61d4e700 1 -- 172.21.15.46:0/3984783330 learned_addr learned my addr 172.21.15.46:0/3984783330 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.659+0000 7f4b62d50700 1 --2- 172.21.15.46:0/3984783330 >> v2:172.21.15.106:3300/0 conn(0x7f4b5c0ae2f0 0x7f4b5c0ae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.659+0000 7f4b6254f700 1 --2- 172.21.15.46:0/3984783330 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b5c1565a0 0x7f4b5c156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.660+0000 7f4b61d4e700 1 -- 172.21.15.46:0/3984783330 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b5c1565a0 msgr2=0x7f4b5c156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.660+0000 7f4b61d4e700 1 --2- 172.21.15.46:0/3984783330 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b5c1565a0 0x7f4b5c156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.660+0000 7f4b61d4e700 1 -- 172.21.15.46:0/3984783330 >> v2:172.21.15.106:3300/0 conn(0x7f4b5c0ae2f0 msgr2=0x7f4b5c0ae750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.660+0000 7f4b61d4e700 1 --2- 172.21.15.46:0/3984783330 >> v2:172.21.15.106:3300/0 conn(0x7f4b5c0ae2f0 0x7f4b5c0ae750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:32.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.660+0000 7f4b61d4e700 1 -- 172.21.15.46:0/3984783330 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c0ad5b0 con 0x7f4b5c15a520 2024-01-27T00:46:32.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.660+0000 7f4b6254f700 1 --2- 172.21.15.46:0/3984783330 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b5c1565a0 0x7f4b5c156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:46:32.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.660+0000 7f4b61d4e700 1 --2- 172.21.15.46:0/3984783330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b5c15a520 0x7f4b5c0acc40 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c06b8d0 tx=0x7f4b58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c9eeb27b845e0fc server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:32.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.660+0000 7f4b6154d700 1 -- 172.21.15.46:0/3984783330 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b58009e70 con 0x7f4b5c15a520 2024-01-27T00:46:32.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.660+0000 7f4b6154d700 1 -- 172.21.15.46:0/3984783330 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5800b860 con 0x7f4b5c15a520 2024-01-27T00:46:32.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.660+0000 7f4b6154d700 1 -- 172.21.15.46:0/3984783330 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5800bc00 con 0x7f4b5c15a520 2024-01-27T00:46:32.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.660+0000 7f4b647b3700 1 -- 172.21.15.46:0/3984783330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b5c15a520 msgr2=0x7f4b5c0acc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:32.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.661+0000 7f4b647b3700 1 --2- 172.21.15.46:0/3984783330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b5c15a520 0x7f4b5c0acc40 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c06b8d0 tx=0x7f4b58009250 comp rx=0 tx=0).stop 2024-01-27T00:46:32.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.661+0000 7f4b647b3700 1 -- 172.21.15.46:0/3984783330 shutdown_connections 2024-01-27T00:46:32.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.661+0000 7f4b647b3700 1 --2- 172.21.15.46:0/3984783330 >> v2:172.21.15.106:3300/0 conn(0x7f4b5c0ae2f0 0x7f4b5c0ae750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:32.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.661+0000 7f4b647b3700 1 --2- 172.21.15.46:0/3984783330 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b5c1565a0 0x7f4b5c156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:32.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.661+0000 7f4b647b3700 1 --2- 172.21.15.46:0/3984783330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b5c15a520 0x7f4b5c0acc40 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:32.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.661+0000 7f4b647b3700 1 -- 172.21.15.46:0/3984783330 >> 172.21.15.46:0/3984783330 conn(0x7f4b5c0cdf80 msgr2=0x7f4b5c06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:32.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.661+0000 7f4b647b3700 1 -- 172.21.15.46:0/3984783330 shutdown_connections 2024-01-27T00:46:32.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.661+0000 7f4b647b3700 1 -- 172.21.15.46:0/3984783330 wait complete. 2024-01-27T00:46:32.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.662+0000 7f4b647b3700 1 Processor -- start 2024-01-27T00:46:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.662+0000 7f4b647b3700 1 -- start start 2024-01-27T00:46:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b647b3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b5c0ae2f0 0x7f4b5c137f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b647b3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b5c1565a0 0x7f4b5c0b2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b647b3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b5c15a520 0x7f4b5c0b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b647b3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c14b070 con 0x7f4b5c1565a0 2024-01-27T00:46:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b647b3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c14bed0 con 0x7f4b5c15a520 2024-01-27T00:46:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b647b3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c14b2c0 con 0x7f4b5c0ae2f0 2024-01-27T00:46:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b61d4e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b5c1565a0 0x7f4b5c0b2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b61d4e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b5c1565a0 0x7f4b5c0b2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43562/0 (socket says 172.21.15.46:43562) 2024-01-27T00:46:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b61d4e700 1 -- 172.21.15.46:0/599352598 learned_addr learned my addr 172.21.15.46:0/599352598 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b62d50700 1 --2- 172.21.15.46:0/599352598 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b5c15a520 0x7f4b5c0b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b6254f700 1 --2- 172.21.15.46:0/599352598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b5c0ae2f0 0x7f4b5c137f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b61d4e700 1 -- 172.21.15.46:0/599352598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b5c0ae2f0 msgr2=0x7f4b5c137f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b61d4e700 1 --2- 172.21.15.46:0/599352598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b5c0ae2f0 0x7f4b5c137f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b61d4e700 1 -- 172.21.15.46:0/599352598 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b5c15a520 msgr2=0x7f4b5c0b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b61d4e700 1 --2- 172.21.15.46:0/599352598 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b5c15a520 0x7f4b5c0b2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b61d4e700 1 -- 172.21.15.46:0/599352598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b580094f0 con 0x7f4b5c1565a0 2024-01-27T00:46:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b6254f700 1 --2- 172.21.15.46:0/599352598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b5c0ae2f0 0x7f4b5c137f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:46:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b61d4e700 1 --2- 172.21.15.46:0/599352598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b5c1565a0 0x7f4b5c0b2710 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f4b580090b0 tx=0x7f4b58017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b527fc700 1 -- 172.21.15.46:0/599352598 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5800bbe0 con 0x7f4b5c1565a0 2024-01-27T00:46:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b527fc700 1 -- 172.21.15.46:0/599352598 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b58011420 con 0x7f4b5c1565a0 2024-01-27T00:46:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b647b3700 1 -- 172.21.15.46:0/599352598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5c14b5a0 con 0x7f4b5c1565a0 2024-01-27T00:46:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.663+0000 7f4b527fc700 1 -- 172.21.15.46:0/599352598 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b58011820 con 0x7f4b5c1565a0 2024-01-27T00:46:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.664+0000 7f4b647b3700 1 -- 172.21.15.46:0/599352598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5c14bb60 con 0x7f4b5c1565a0 2024-01-27T00:46:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.665+0000 7f4b647b3700 1 -- 172.21.15.46:0/599352598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b5c14c040 con 0x7f4b5c1565a0 2024-01-27T00:46:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.665+0000 7f4b527fc700 1 -- 172.21.15.46:0/599352598 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4b58011c50 con 0x7f4b5c1565a0 2024-01-27T00:46:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.665+0000 7f4b527fc700 1 --2- 172.21.15.46:0/599352598 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4b40066f30 0x7f4b400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.666+0000 7f4b527fc700 1 -- 172.21.15.46:0/599352598 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f4b58086010 con 0x7f4b5c1565a0 2024-01-27T00:46:32.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.666+0000 7f4b6254f700 1 --2- 172.21.15.46:0/599352598 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4b40066f30 0x7f4b400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:32.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.666+0000 7f4b6254f700 1 --2- 172.21.15.46:0/599352598 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4b40066f30 0x7f4b400693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c006f00 tx=0x7f4b4c008040 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:32.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.670+0000 7f4b527fc700 1 -- 172.21.15.46:0/599352598 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4b58053080 con 0x7f4b5c1565a0 2024-01-27T00:46:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.875+0000 7f4b647b3700 1 -- 172.21.15.46:0/599352598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4b5c04c710 con 0x7f4b5c1565a0 2024-01-27T00:46:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.879+0000 7f4b527fc700 1 -- 172.21.15.46:0/599352598 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4b580566a0 con 0x7f4b5c1565a0 2024-01-27T00:46:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.881+0000 7f4b647b3700 1 -- 172.21.15.46:0/599352598 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4b40066f30 msgr2=0x7f4b400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.881+0000 7f4b647b3700 1 --2- 172.21.15.46:0/599352598 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4b40066f30 0x7f4b400693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c006f00 tx=0x7f4b4c008040 comp rx=0 tx=0).stop 2024-01-27T00:46:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.881+0000 7f4b647b3700 1 -- 172.21.15.46:0/599352598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b5c1565a0 msgr2=0x7f4b5c0b2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.881+0000 7f4b647b3700 1 --2- 172.21.15.46:0/599352598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b5c1565a0 0x7f4b5c0b2710 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f4b580090b0 tx=0x7f4b58017000 comp rx=0 tx=0).stop 2024-01-27T00:46:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.881+0000 7f4b647b3700 1 -- 172.21.15.46:0/599352598 shutdown_connections 2024-01-27T00:46:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.881+0000 7f4b647b3700 1 --2- 172.21.15.46:0/599352598 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4b40066f30 0x7f4b400693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:32.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.882+0000 7f4b647b3700 1 --2- 172.21.15.46:0/599352598 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b5c15a520 0x7f4b5c0b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:32.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.882+0000 7f4b647b3700 1 --2- 172.21.15.46:0/599352598 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b5c0ae2f0 0x7f4b5c137f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:32.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.882+0000 7f4b647b3700 1 --2- 172.21.15.46:0/599352598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b5c1565a0 0x7f4b5c0b2710 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:32.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.882+0000 7f4b647b3700 1 -- 172.21.15.46:0/599352598 >> 172.21.15.46:0/599352598 conn(0x7f4b5c0cdf80 msgr2=0x7f4b5c06f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:32.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.882+0000 7f4b647b3700 1 -- 172.21.15.46:0/599352598 shutdown_connections 2024-01-27T00:46:32.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:32.882+0000 7f4b647b3700 1 -- 172.21.15.46:0/599352598 wait complete. 2024-01-27T00:46:32.894 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-27T00:46:32.894 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T00:46:32.942 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T00:46:32.985 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-27T00:46:32.985 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-27T00:46:32.986 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T00:46:33.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.123+0000 7f93e900c700 1 Processor -- start 2024-01-27T00:46:33.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.123+0000 7f93e900c700 1 -- start start 2024-01-27T00:46:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.124+0000 7f93e900c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f93e4158f50 0x7f93e41593a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.124+0000 7f93e900c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f93e4157d80 0x7f93e41581d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.124+0000 7f93e900c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e406cae0 con 0x7f93e4155290 2024-01-27T00:46:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.124+0000 7f93e900c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93e41598e0 con 0x7f93e4157d80 2024-01-27T00:46:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.124+0000 7f93e900c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93e4158710 con 0x7f93e4158f50 2024-01-27T00:46:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.124+0000 7f93e359e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f93e4157d80 0x7f93e41581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.124+0000 7f93e359e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f93e4157d80 0x7f93e41581d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43564/0 (socket says 172.21.15.46:43564) 2024-01-27T00:46:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.124+0000 7f93e359e700 1 -- 172.21.15.46:0/1739239948 learned_addr learned my addr 172.21.15.46:0/1739239948 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.124+0000 7f93e259c700 1 --2- 172.21.15.46:0/1739239948 >> v2:172.21.15.106:3300/0 conn(0x7f93e4158f50 0x7f93e41593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.125+0000 7f93e359e700 1 -- 172.21.15.46:0/1739239948 >> v2:172.21.15.106:3300/0 conn(0x7f93e4158f50 msgr2=0x7f93e41593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:33.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.125+0000 7f93e359e700 1 --2- 172.21.15.46:0/1739239948 >> v2:172.21.15.106:3300/0 conn(0x7f93e4158f50 0x7f93e41593a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:33.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.125+0000 7f93e359e700 1 -- 172.21.15.46:0/1739239948 >> v1:172.21.15.106:6789/0 conn(0x7f93e4155290 legacy=0x7f93e406cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:33.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.125+0000 7f93e359e700 1 -- 172.21.15.46:0/1739239948 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e40ce010 con 0x7f93e4157d80 2024-01-27T00:46:33.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.125+0000 7f93e359e700 1 --2- 172.21.15.46:0/1739239948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e4157d80 0x7f93e41581d0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f93e40c9a00 tx=0x7f93dc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d6845383ba43be server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:33.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.125+0000 7f93e1d9b700 1 -- 172.21.15.46:0/1739239948 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc00e070 con 0x7f93e4157d80 2024-01-27T00:46:33.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.125+0000 7f93e1d9b700 1 -- 172.21.15.46:0/1739239948 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93dc00ba50 con 0x7f93e4157d80 2024-01-27T00:46:33.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.125+0000 7f93e1d9b700 1 -- 172.21.15.46:0/1739239948 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc0092e0 con 0x7f93e4157d80 2024-01-27T00:46:33.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.125+0000 7f93e900c700 1 -- 172.21.15.46:0/1739239948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e4157d80 msgr2=0x7f93e41581d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:33.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.125+0000 7f93e900c700 1 --2- 172.21.15.46:0/1739239948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e4157d80 0x7f93e41581d0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f93e40c9a00 tx=0x7f93dc009840 comp rx=0 tx=0).stop 2024-01-27T00:46:33.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.125+0000 7f93e900c700 1 -- 172.21.15.46:0/1739239948 shutdown_connections 2024-01-27T00:46:33.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.126+0000 7f93e900c700 1 --2- 172.21.15.46:0/1739239948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e4157d80 0x7f93e41581d0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:33.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.126+0000 7f93e900c700 1 --2- 172.21.15.46:0/1739239948 >> v2:172.21.15.106:3300/0 conn(0x7f93e4158f50 0x7f93e41593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:33.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.126+0000 7f93e900c700 1 -- 172.21.15.46:0/1739239948 >> 172.21.15.46:0/1739239948 conn(0x7f93e406b820 msgr2=0x7f93e40cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:33.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.126+0000 7f93e900c700 1 -- 172.21.15.46:0/1739239948 shutdown_connections 2024-01-27T00:46:33.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.126+0000 7f93e900c700 1 -- 172.21.15.46:0/1739239948 wait complete. 2024-01-27T00:46:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.127+0000 7f93e900c700 1 Processor -- start 2024-01-27T00:46:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.127+0000 7f93e900c700 1 -- start start 2024-01-27T00:46:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.128+0000 7f93e900c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e4155290 0x7f93e4150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.128+0000 7f93e900c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f93e4157d80 0x7f93e4152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.128+0000 7f93e900c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93e4158f50 0x7f93e40b1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.128+0000 7f93e900c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e40b03f0 con 0x7f93e4155290 2024-01-27T00:46:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.128+0000 7f93e900c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e40b0640 con 0x7f93e4158f50 2024-01-27T00:46:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.128+0000 7f93e900c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e40b0920 con 0x7f93e4157d80 2024-01-27T00:46:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.128+0000 7f93e2d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e4155290 0x7f93e4150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.128+0000 7f93e2d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e4155290 0x7f93e4150b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43578/0 (socket says 172.21.15.46:43578) 2024-01-27T00:46:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.128+0000 7f93e2d9d700 1 -- 172.21.15.46:0/2085967463 learned_addr learned my addr 172.21.15.46:0/2085967463 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.128+0000 7f93e359e700 1 --2- 172.21.15.46:0/2085967463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93e4158f50 0x7f93e40b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.128+0000 7f93e259c700 1 --2- 172.21.15.46:0/2085967463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f93e4157d80 0x7f93e4152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.128+0000 7f93e2d9d700 1 -- 172.21.15.46:0/2085967463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f93e4157d80 msgr2=0x7f93e4152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.128+0000 7f93e2d9d700 1 --2- 172.21.15.46:0/2085967463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f93e4157d80 0x7f93e4152580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.128+0000 7f93e2d9d700 1 -- 172.21.15.46:0/2085967463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93e4158f50 msgr2=0x7f93e40b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.128+0000 7f93e2d9d700 1 --2- 172.21.15.46:0/2085967463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93e4158f50 0x7f93e40b1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.128+0000 7f93e2d9d700 1 -- 172.21.15.46:0/2085967463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc009ae0 con 0x7f93e4155290 2024-01-27T00:46:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.128+0000 7f93e259c700 1 --2- 172.21.15.46:0/2085967463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f93e4157d80 0x7f93e4152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:46:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.129+0000 7f93e2d9d700 1 --2- 172.21.15.46:0/2085967463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e4155290 0x7f93e4150b20 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f93d400c590 tx=0x7f93d400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.129+0000 7f93daffd700 1 -- 172.21.15.46:0/2085967463 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d4011590 con 0x7f93e4155290 2024-01-27T00:46:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.129+0000 7f93daffd700 1 -- 172.21.15.46:0/2085967463 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d40116f0 con 0x7f93e4155290 2024-01-27T00:46:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.129+0000 7f93e900c700 1 -- 172.21.15.46:0/2085967463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e40b0c00 con 0x7f93e4155290 2024-01-27T00:46:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.129+0000 7f93daffd700 1 -- 172.21.15.46:0/2085967463 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d4011ab0 con 0x7f93e4155290 2024-01-27T00:46:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.129+0000 7f93e900c700 1 -- 172.21.15.46:0/2085967463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e40b30f0 con 0x7f93e4155290 2024-01-27T00:46:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.130+0000 7f93e900c700 1 -- 172.21.15.46:0/2085967463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e404c710 con 0x7f93e4155290 2024-01-27T00:46:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.130+0000 7f93daffd700 1 -- 172.21.15.46:0/2085967463 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f93d401b3b0 con 0x7f93e4155290 2024-01-27T00:46:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.131+0000 7f93daffd700 1 --2- 172.21.15.46:0/2085967463 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f93c4066f30 0x7f93c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.131+0000 7f93daffd700 1 -- 172.21.15.46:0/2085967463 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f93d4083ce0 con 0x7f93e4155290 2024-01-27T00:46:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.131+0000 7f93e259c700 1 --2- 172.21.15.46:0/2085967463 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f93c4066f30 0x7f93c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.132+0000 7f93e259c700 1 --2- 172.21.15.46:0/2085967463 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f93c4066f30 0x7f93c40693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f93e40b2f50 tx=0x7f93cc008040 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:33.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.136+0000 7f93daffd700 1 -- 172.21.15.46:0/2085967463 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f93d4050d50 con 0x7f93e4155290 2024-01-27T00:46:33.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.384+0000 7f93e900c700 1 -- 172.21.15.46:0/2085967463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93e4158960 con 0x7f93e4155290 2024-01-27T00:46:33.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.385+0000 7f93daffd700 1 -- 172.21.15.46:0/2085967463 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+3526 (secure 0 0 0) 0x7f93d4017070 con 0x7f93e4155290 2024-01-27T00:46:33.384 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:33.384 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":80,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8578,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1889696251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1889696251},{"type":"v1","addr":"172.21.15.106:6834","nonce":1889696251}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/631881132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":631881132},{"type":"v1","addr":"172.21.15.106:6836","nonce":631881132}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1045000694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1045000694},{"type":"v1","addr":"172.21.15.46:6834","nonce":1045000694}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74},{"gid":8941,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/53177280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":53177280},{"type":"v1","addr":"172.21.15.46:6835","nonce":53177280}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80}],"filesystems":[]} 2024-01-27T00:46:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.387+0000 7f93e900c700 1 -- 172.21.15.46:0/2085967463 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f93c4066f30 msgr2=0x7f93c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.387+0000 7f93e900c700 1 --2- 172.21.15.46:0/2085967463 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f93c4066f30 0x7f93c40693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f93e40b2f50 tx=0x7f93cc008040 comp rx=0 tx=0).stop 2024-01-27T00:46:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.387+0000 7f93e900c700 1 -- 172.21.15.46:0/2085967463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e4155290 msgr2=0x7f93e4150b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.387+0000 7f93e900c700 1 --2- 172.21.15.46:0/2085967463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e4155290 0x7f93e4150b20 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f93d400c590 tx=0x7f93d400c950 comp rx=0 tx=0).stop 2024-01-27T00:46:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.387+0000 7f93e900c700 1 -- 172.21.15.46:0/2085967463 shutdown_connections 2024-01-27T00:46:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.387+0000 7f93e900c700 1 --2- 172.21.15.46:0/2085967463 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f93c4066f30 0x7f93c40693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.387+0000 7f93e900c700 1 --2- 172.21.15.46:0/2085967463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e4155290 0x7f93e4150b20 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.387+0000 7f93e900c700 1 --2- 172.21.15.46:0/2085967463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93e4158f50 0x7f93e40b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.387+0000 7f93e900c700 1 --2- 172.21.15.46:0/2085967463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f93e4157d80 0x7f93e4152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.387+0000 7f93e900c700 1 -- 172.21.15.46:0/2085967463 >> 172.21.15.46:0/2085967463 conn(0x7f93e406b820 msgr2=0x7f93e415a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.387+0000 7f93e900c700 1 -- 172.21.15.46:0/2085967463 shutdown_connections 2024-01-27T00:46:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:33.387+0000 7f93e900c700 1 -- 172.21.15.46:0/2085967463 wait complete. 2024-01-27T00:46:33.388 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 80 2024-01-27T00:46:33.400 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-27T00:46:33.400 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-27T00:46:33.400 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-27T00:46:33.400 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T00:46:33.400 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-27T00:46:33.401 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-27T00:46:33.401 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-27T00:46:33.401 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T00:46:33.401 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-27T00:46:33.401 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-27T00:46:33.401 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-27T00:46:33.401 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T00:46:33.401 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-27T00:46:33.402 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-27T00:46:33.402 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-27T00:46:33.402 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T00:46:33.414 INFO:tasks.ceph.mds.c:Stopped 2024-01-27T00:46:33.414 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-27T00:46:33.415 INFO:tasks.ceph.mds.b:Stopped 2024-01-27T00:46:33.415 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-27T00:46:33.416 INFO:tasks.ceph.mds.a:Stopped 2024-01-27T00:46:33.416 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-27T00:46:33.457 INFO:tasks.ceph.mds.c:Started 2024-01-27T00:46:33.457 DEBUG:teuthology.parallel:result is None 2024-01-27T00:46:33.457 INFO:tasks.ceph.mds.a:Started 2024-01-27T00:46:33.458 DEBUG:teuthology.parallel:result is None 2024-01-27T00:46:33.458 INFO:tasks.ceph.mds.b:Started 2024-01-27T00:46:33.458 DEBUG:teuthology.parallel:result is None 2024-01-27T00:46:33.541 INFO:tasks.ceph.mds.b.smithi106.stdout:starting mds.b at 2024-01-27T00:46:33.543 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-27T00:46:33.543 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-27T00:46:38.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-01-27T00:46:39.415 INFO:tasks.ceph.mds.d:Stopped 2024-01-27T00:46:39.415 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-27T00:46:39.418 INFO:tasks.ceph.mds.d:Started 2024-01-27T00:46:39.418 DEBUG:teuthology.parallel:result is None 2024-01-27T00:46:39.419 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-27T00:46:39.502 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-01-27T00:46:39.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.563+0000 7f7c61ac9700 1 Processor -- start 2024-01-27T00:46:39.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.563+0000 7f7c61ac9700 1 -- start start 2024-01-27T00:46:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c61ac9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c5c0acf30 0x7f7c5c0c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c61ac9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c5c0ad300 0x7f7c5c0ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c61ac9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7c5c0ad750 0x7f7c5c0cd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c61ac9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c5c157be0 con 0x7f7c5c0acf30 2024-01-27T00:46:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c61ac9700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c5c0c4230 con 0x7f7c5c0ad750 2024-01-27T00:46:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c61ac9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0cbf10 con 0x7f7c5c0ad300 2024-01-27T00:46:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c5b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c5c0acf30 0x7f7c5c0c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c5b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c5c0acf30 0x7f7c5c0c3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43618/0 (socket says 172.21.15.46:43618) 2024-01-27T00:46:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c5b7fe700 1 -- 172.21.15.46:0/1601775429 learned_addr learned my addr 172.21.15.46:0/1601775429 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c5bfff700 1 --2- 172.21.15.46:0/1601775429 >> v2:172.21.15.106:3300/0 conn(0x7f7c5c0ad750 0x7f7c5c0cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c5affd700 1 --2- 172.21.15.46:0/1601775429 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c5c0ad300 0x7f7c5c0ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c5b7fe700 1 -- 172.21.15.46:0/1601775429 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c5c0ad300 msgr2=0x7f7c5c0ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c5b7fe700 1 --2- 172.21.15.46:0/1601775429 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c5c0ad300 0x7f7c5c0ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c5b7fe700 1 -- 172.21.15.46:0/1601775429 >> v2:172.21.15.106:3300/0 conn(0x7f7c5c0ad750 msgr2=0x7f7c5c0cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c5b7fe700 1 --2- 172.21.15.46:0/1601775429 >> v2:172.21.15.106:3300/0 conn(0x7f7c5c0ad750 0x7f7c5c0cd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c5b7fe700 1 -- 172.21.15.46:0/1601775429 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c0c4460 con 0x7f7c5c0acf30 2024-01-27T00:46:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c5affd700 1 --2- 172.21.15.46:0/1601775429 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c5c0ad300 0x7f7c5c0ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:46:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c5bfff700 1 --2- 172.21.15.46:0/1601775429 >> v2:172.21.15.106:3300/0 conn(0x7f7c5c0ad750 0x7f7c5c0cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:46:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.564+0000 7f7c5b7fe700 1 --2- 172.21.15.46:0/1601775429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c5c0acf30 0x7f7c5c0c3cf0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0add90 tx=0x7f7c4c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84cf3b80ef5c8a10 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.565+0000 7f7c5a7fc700 1 -- 172.21.15.46:0/1601775429 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c00b820 con 0x7f7c5c0acf30 2024-01-27T00:46:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.565+0000 7f7c5a7fc700 1 -- 172.21.15.46:0/1601775429 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4c00b980 con 0x7f7c5c0acf30 2024-01-27T00:46:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.565+0000 7f7c5a7fc700 1 -- 172.21.15.46:0/1601775429 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c00bd20 con 0x7f7c5c0acf30 2024-01-27T00:46:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.565+0000 7f7c61ac9700 1 -- 172.21.15.46:0/1601775429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c5c0acf30 msgr2=0x7f7c5c0c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:39.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.565+0000 7f7c61ac9700 1 --2- 172.21.15.46:0/1601775429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c5c0acf30 0x7f7c5c0c3cf0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0add90 tx=0x7f7c4c009250 comp rx=0 tx=0).stop 2024-01-27T00:46:39.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.565+0000 7f7c61ac9700 1 -- 172.21.15.46:0/1601775429 shutdown_connections 2024-01-27T00:46:39.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.565+0000 7f7c61ac9700 1 --2- 172.21.15.46:0/1601775429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c5c0acf30 0x7f7c5c0c3cf0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:39.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.565+0000 7f7c61ac9700 1 --2- 172.21.15.46:0/1601775429 >> v2:172.21.15.106:3300/0 conn(0x7f7c5c0ad750 0x7f7c5c0cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:39.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.565+0000 7f7c61ac9700 1 --2- 172.21.15.46:0/1601775429 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c5c0ad300 0x7f7c5c0ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:39.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.565+0000 7f7c61ac9700 1 -- 172.21.15.46:0/1601775429 >> 172.21.15.46:0/1601775429 conn(0x7f7c5c097820 msgr2=0x7f7c5c068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:39.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.565+0000 7f7c61ac9700 1 -- 172.21.15.46:0/1601775429 shutdown_connections 2024-01-27T00:46:39.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.565+0000 7f7c61ac9700 1 -- 172.21.15.46:0/1601775429 wait complete. 2024-01-27T00:46:39.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.567+0000 7f7c61ac9700 1 Processor -- start 2024-01-27T00:46:39.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.567+0000 7f7c61ac9700 1 -- start start 2024-01-27T00:46:39.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c61ac9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c5c0acf30 0x7f7c5c0b3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:39.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c61ac9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c5c0ad300 0x7f7c5c0c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:39.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c61ac9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c5c0ad750 0x7f7c5c0c5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:39.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c61ac9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0c9ed0 con 0x7f7c5c0ad300 2024-01-27T00:46:39.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c61ac9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0cad30 con 0x7f7c5c0acf30 2024-01-27T00:46:39.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c61ac9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0b3e80 con 0x7f7c5c0ad750 2024-01-27T00:46:39.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c5affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c5c0ad300 0x7f7c5c0c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c5affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c5c0ad300 0x7f7c5c0c55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43628/0 (socket says 172.21.15.46:43628) 2024-01-27T00:46:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c5affd700 1 -- 172.21.15.46:0/1245541891 learned_addr learned my addr 172.21.15.46:0/1245541891 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c5b7fe700 1 --2- 172.21.15.46:0/1245541891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c5c0acf30 0x7f7c5c0b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c5bfff700 1 --2- 172.21.15.46:0/1245541891 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c5c0ad750 0x7f7c5c0c5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c5affd700 1 -- 172.21.15.46:0/1245541891 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c5c0ad750 msgr2=0x7f7c5c0c5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c5affd700 1 --2- 172.21.15.46:0/1245541891 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c5c0ad750 0x7f7c5c0c5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c5affd700 1 -- 172.21.15.46:0/1245541891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c5c0acf30 msgr2=0x7f7c5c0b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c5affd700 1 --2- 172.21.15.46:0/1245541891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c5c0acf30 0x7f7c5c0b3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c5affd700 1 -- 172.21.15.46:0/1245541891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4c0094f0 con 0x7f7c5c0ad300 2024-01-27T00:46:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c5affd700 1 --2- 172.21.15.46:0/1245541891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c5c0ad300 0x7f7c5c0c55f0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f7c5000c8b0 tx=0x7f7c50008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c437fe700 1 -- 172.21.15.46:0/1245541891 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c50011600 con 0x7f7c5c0ad300 2024-01-27T00:46:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c437fe700 1 -- 172.21.15.46:0/1245541891 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c50011760 con 0x7f7c5c0ad300 2024-01-27T00:46:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.568+0000 7f7c61ac9700 1 -- 172.21.15.46:0/1245541891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c5c0ca120 con 0x7f7c5c0ad300 2024-01-27T00:46:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.569+0000 7f7c437fe700 1 -- 172.21.15.46:0/1245541891 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c50011b00 con 0x7f7c5c0ad300 2024-01-27T00:46:39.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.569+0000 7f7c61ac9700 1 -- 172.21.15.46:0/1245541891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c5c0ca6e0 con 0x7f7c5c0ad300 2024-01-27T00:46:39.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.570+0000 7f7c61ac9700 1 -- 172.21.15.46:0/1245541891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c5c04c710 con 0x7f7c5c0ad300 2024-01-27T00:46:39.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.570+0000 7f7c437fe700 1 -- 172.21.15.46:0/1245541891 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7c50018350 con 0x7f7c5c0ad300 2024-01-27T00:46:39.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.570+0000 7f7c437fe700 1 --2- 172.21.15.46:0/1245541891 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7c48066f30 0x7f7c480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:39.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.571+0000 7f7c437fe700 1 -- 172.21.15.46:0/1245541891 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f7c50084560 con 0x7f7c5c0ad300 2024-01-27T00:46:39.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.571+0000 7f7c5b7fe700 1 --2- 172.21.15.46:0/1245541891 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7c48066f30 0x7f7c480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:39.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.571+0000 7f7c5b7fe700 1 --2- 172.21.15.46:0/1245541891 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7c48066f30 0x7f7c480693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0add90 tx=0x7f7c4c017000 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:39.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.575+0000 7f7c437fe700 1 -- 172.21.15.46:0/1245541891 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c50055060 con 0x7f7c5c0ad300 2024-01-27T00:46:39.594 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:46:39.596 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:46:39.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:39.800+0000 7f7c61ac9700 1 -- 172.21.15.46:0/1245541891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f7c5c0c4740 con 0x7f7c5c0ad300 2024-01-27T00:46:41.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.523+0000 7f7c437fe700 1 -- 172.21.15.46:0/1245541891 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v171) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7c50054bf0 con 0x7f7c5c0ad300 2024-01-27T00:46:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.527+0000 7f7c61ac9700 1 -- 172.21.15.46:0/1245541891 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7c48066f30 msgr2=0x7f7c480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.527+0000 7f7c61ac9700 1 --2- 172.21.15.46:0/1245541891 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7c48066f30 0x7f7c480693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0add90 tx=0x7f7c4c017000 comp rx=0 tx=0).stop 2024-01-27T00:46:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.527+0000 7f7c61ac9700 1 -- 172.21.15.46:0/1245541891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c5c0ad300 msgr2=0x7f7c5c0c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.527+0000 7f7c61ac9700 1 --2- 172.21.15.46:0/1245541891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c5c0ad300 0x7f7c5c0c55f0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f7c5000c8b0 tx=0x7f7c50008040 comp rx=0 tx=0).stop 2024-01-27T00:46:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.527+0000 7f7c61ac9700 1 -- 172.21.15.46:0/1245541891 shutdown_connections 2024-01-27T00:46:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.527+0000 7f7c61ac9700 1 --2- 172.21.15.46:0/1245541891 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7c48066f30 0x7f7c480693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.527+0000 7f7c61ac9700 1 --2- 172.21.15.46:0/1245541891 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c5c0ad750 0x7f7c5c0c5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.527+0000 7f7c61ac9700 1 --2- 172.21.15.46:0/1245541891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c5c0acf30 0x7f7c5c0b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.527+0000 7f7c61ac9700 1 --2- 172.21.15.46:0/1245541891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c5c0ad300 0x7f7c5c0c55f0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.527+0000 7f7c61ac9700 1 -- 172.21.15.46:0/1245541891 >> 172.21.15.46:0/1245541891 conn(0x7f7c5c097820 msgr2=0x7f7c5c156db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.527+0000 7f7c61ac9700 1 -- 172.21.15.46:0/1245541891 shutdown_connections 2024-01-27T00:46:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.527+0000 7f7c61ac9700 1 -- 172.21.15.46:0/1245541891 wait complete. 2024-01-27T00:46:41.527 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-01-27T00:46:41.542 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-27T00:46:41.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.678+0000 7f9427d95700 1 Processor -- start 2024-01-27T00:46:41.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.679+0000 7f9427d95700 1 -- start start 2024-01-27T00:46:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.679+0000 7f9427d95700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9420157340 0x7f942015bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.679+0000 7f9427d95700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9420158ae0 con 0x7f94200b4180 2024-01-27T00:46:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.679+0000 7f9427d95700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f942015b650 con 0x7f94200ae520 2024-01-27T00:46:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.679+0000 7f9427d95700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f942015c020 con 0x7f9420157340 2024-01-27T00:46:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9426332700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f94200b4180 0x7f9420157710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41062/0 (socket says 172.21.15.46:41062) 2024-01-27T00:46:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9426332700 1 -- 172.21.15.46:0/3766145265 learned_addr learned my addr 172.21.15.46:0/3766145265 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9425b31700 1 --2- 172.21.15.46:0/3766145265 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9420157340 0x7f942015bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9424b2f700 1 -- 172.21.15.46:0/3766145265 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544293791 0 0) 0x7f9420158ae0 con 0x7f94200b4180 2024-01-27T00:46:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9424b2f700 1 -- 172.21.15.46:0/3766145265 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f940c003570 con 0x7f94200b4180 2024-01-27T00:46:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9424b2f700 1 -- 172.21.15.46:0/3766145265 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2519005253 0 0) 0x7f940c003570 con 0x7f94200b4180 2024-01-27T00:46:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9424b2f700 1 -- 172.21.15.46:0/3766145265 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9420157340 msgr2=0x7f942015bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:41.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9424b2f700 1 --2- 172.21.15.46:0/3766145265 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9420157340 0x7f942015bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9424b2f700 1 -- 172.21.15.46:0/3766145265 >> v1:172.21.15.106:6789/0 conn(0x7f94200ae520 legacy=0x7f94200b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9424b2f700 1 -- 172.21.15.46:0/3766145265 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9420158520 con 0x7f94200b4180 2024-01-27T00:46:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9424b2f700 1 -- 172.21.15.46:0/3766145265 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f94180017f0 con 0x7f94200b4180 2024-01-27T00:46:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9424b2f700 1 -- 172.21.15.46:0/3766145265 >> v1:172.21.15.46:6789/0 conn(0x7f94200b4180 legacy=0x7f9420157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9424b2f700 1 --2- 172.21.15.46:0/3766145265 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9420157340 0x7f940c003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9424b2f700 1 --2- 172.21.15.46:0/3766145265 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f940c003fd0 0x7f940c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9424b2f700 1 --2- 172.21.15.46:0/3766145265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f940c004d50 0x7f940c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9424b2f700 1 -- 172.21.15.46:0/3766145265 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942015c020 con 0x7f940c004d50 2024-01-27T00:46:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9424b2f700 1 -- 172.21.15.46:0/3766145265 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940c00bbf0 con 0x7f940c003fd0 2024-01-27T00:46:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.680+0000 7f9424b2f700 1 -- 172.21.15.46:0/3766145265 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940c00bdb0 con 0x7f9420157340 2024-01-27T00:46:41.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.681+0000 7f9426332700 1 --2- 172.21.15.46:0/3766145265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f940c004d50 0x7f940c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:41.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.681+0000 7f9425b31700 1 --2- 172.21.15.46:0/3766145265 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9420157340 0x7f940c003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:41.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.681+0000 7f9425330700 1 --2- 172.21.15.46:0/3766145265 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f940c003fd0 0x7f940c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:41.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.681+0000 7f9425b31700 1 -- 172.21.15.46:0/3766145265 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f940c003fd0 msgr2=0x7f940c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:41.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.681+0000 7f9425b31700 1 --2- 172.21.15.46:0/3766145265 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f940c003fd0 0x7f940c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:41.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.681+0000 7f9425b31700 1 -- 172.21.15.46:0/3766145265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f940c004d50 msgr2=0x7f940c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:41.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.681+0000 7f9425b31700 1 --2- 172.21.15.46:0/3766145265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f940c004d50 0x7f940c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:41.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.681+0000 7f9425b31700 1 -- 172.21.15.46:0/3766145265 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940c003570 con 0x7f9420157340 2024-01-27T00:46:41.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.681+0000 7f9426332700 1 --2- 172.21.15.46:0/3766145265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f940c004d50 0x7f940c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:41.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.681+0000 7f9425b31700 1 --2- 172.21.15.46:0/3766145265 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9420157340 0x7f940c003a90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f941c009c20 tx=0x7f941c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5729160cb82a66c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:41.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.681+0000 7f9424b2f700 1 -- 172.21.15.46:0/3766145265 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941c0095b0 con 0x7f9420157340 2024-01-27T00:46:41.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.681+0000 7f9424b2f700 1 -- 172.21.15.46:0/3766145265 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941c00e710 con 0x7f9420157340 2024-01-27T00:46:41.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.681+0000 7f9424b2f700 1 -- 172.21.15.46:0/3766145265 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941c00eab0 con 0x7f9420157340 2024-01-27T00:46:41.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.682+0000 7f9427d95700 1 -- 172.21.15.46:0/3766145265 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9420157340 msgr2=0x7f940c003a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:41.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.682+0000 7f9427d95700 1 --2- 172.21.15.46:0/3766145265 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9420157340 0x7f940c003a90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f941c009c20 tx=0x7f941c0091a0 comp rx=0 tx=0).stop 2024-01-27T00:46:41.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.682+0000 7f9426332700 1 -- 172.21.15.46:0/3766145265 reap_dead start 2024-01-27T00:46:41.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.682+0000 7f9427d95700 1 -- 172.21.15.46:0/3766145265 shutdown_connections 2024-01-27T00:46:41.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.682+0000 7f9427d95700 1 -- 172.21.15.46:0/3766145265 >> 172.21.15.46:0/3766145265 conn(0x7f94200ce530 msgr2=0x7f94200ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:41.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.682+0000 7f9427d95700 1 -- 172.21.15.46:0/3766145265 shutdown_connections 2024-01-27T00:46:41.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.682+0000 7f9427d95700 1 -- 172.21.15.46:0/3766145265 wait complete. 2024-01-27T00:46:41.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.683+0000 7f9427d95700 1 Processor -- start 2024-01-27T00:46:41.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.683+0000 7f9427d95700 1 -- start start 2024-01-27T00:46:41.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.684+0000 7f9427d95700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9420157340 0x7f94200b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:41.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.684+0000 7f9427d95700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9420151a00 0x7f94200c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:41.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.684+0000 7f9427d95700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9420155a90 0x7f9420155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:41.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.684+0000 7f9427d95700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9420156440 con 0x7f9420157340 2024-01-27T00:46:41.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.684+0000 7f9427d95700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94200c9220 con 0x7f9420155a90 2024-01-27T00:46:41.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.684+0000 7f9427d95700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94200c94c0 con 0x7f9420151a00 2024-01-27T00:46:41.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.684+0000 7f9425b31700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9420157340 0x7f94200b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:41.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.684+0000 7f9425b31700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9420157340 0x7f94200b3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59674/0 (socket says 172.21.15.46:59674) 2024-01-27T00:46:41.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.684+0000 7f9425b31700 1 -- 172.21.15.46:0/1149898913 learned_addr learned my addr 172.21.15.46:0/1149898913 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:41.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.684+0000 7f9425330700 1 --2- 172.21.15.46:0/1149898913 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9420151a00 0x7f94200c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:41.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.684+0000 7f9426332700 1 --2- 172.21.15.46:0/1149898913 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9420155a90 0x7f9420155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:41.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.684+0000 7f9425b31700 1 -- 172.21.15.46:0/1149898913 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9420151a00 msgr2=0x7f94200c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:41.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.684+0000 7f9425b31700 1 --2- 172.21.15.46:0/1149898913 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9420151a00 0x7f94200c8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:41.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.684+0000 7f9425b31700 1 -- 172.21.15.46:0/1149898913 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9420155a90 msgr2=0x7f9420155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:41.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.684+0000 7f9425b31700 1 --2- 172.21.15.46:0/1149898913 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9420155a90 0x7f9420155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:41.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.684+0000 7f9425b31700 1 -- 172.21.15.46:0/1149898913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94200c97a0 con 0x7f9420157340 2024-01-27T00:46:41.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.684+0000 7f9426332700 1 --2- 172.21.15.46:0/1149898913 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9420155a90 0x7f9420155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:46:41.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.685+0000 7f9425b31700 1 --2- 172.21.15.46:0/1149898913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9420157340 0x7f94200b3190 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f941c0091e0 tx=0x7f941c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:41.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.685+0000 7f9415ffb700 1 -- 172.21.15.46:0/1149898913 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941c023070 con 0x7f9420157340 2024-01-27T00:46:41.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.685+0000 7f9415ffb700 1 -- 172.21.15.46:0/1149898913 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941c00ea90 con 0x7f9420157340 2024-01-27T00:46:41.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.685+0000 7f9415ffb700 1 -- 172.21.15.46:0/1149898913 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941c01e240 con 0x7f9420157340 2024-01-27T00:46:41.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.685+0000 7f9427d95700 1 -- 172.21.15.46:0/1149898913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94200c9a80 con 0x7f9420157340 2024-01-27T00:46:41.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.685+0000 7f9427d95700 1 -- 172.21.15.46:0/1149898913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94200ca040 con 0x7f9420157340 2024-01-27T00:46:41.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.686+0000 7f9427d95700 1 -- 172.21.15.46:0/1149898913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94200ccbe0 con 0x7f9420157340 2024-01-27T00:46:41.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.687+0000 7f9415ffb700 1 -- 172.21.15.46:0/1149898913 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f941c013680 con 0x7f9420157340 2024-01-27T00:46:41.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.687+0000 7f9415ffb700 1 --2- 172.21.15.46:0/1149898913 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9404066f30 0x7f94040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:41.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.687+0000 7f9415ffb700 1 -- 172.21.15.46:0/1149898913 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f941c085c50 con 0x7f9420157340 2024-01-27T00:46:41.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.687+0000 7f9425330700 1 --2- 172.21.15.46:0/1149898913 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9404066f30 0x7f94040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:41.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.688+0000 7f9425330700 1 --2- 172.21.15.46:0/1149898913 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9404066f30 0x7f94040693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9410000f60 tx=0x7f9410009040 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:41.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.692+0000 7f9415ffb700 1 -- 172.21.15.46:0/1149898913 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f941c08b020 con 0x7f9420157340 2024-01-27T00:46:41.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.981+0000 7f9427d95700 1 -- 172.21.15.46:0/1149898913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f94200b3fc0 con 0x7f9420157340 2024-01-27T00:46:41.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.982+0000 7f9415ffb700 1 -- 172.21.15.46:0/1149898913 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v44) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f941c052f20 con 0x7f9420157340 2024-01-27T00:46:41.981 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:41.981 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T00:46:41.981 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-01-27T00:46:41.981 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.981 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-01-27T00:46:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlM6dKKBAAPpeWeDZWrFgc/4zuxKCvHw==", 2024-01-27T00:46:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T00:46:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T00:46:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T00:46:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:46:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-01-27T00:46:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlmzHRNRAAflWYZ/p4AXeMy54tvLpugA==", 2024-01-27T00:46:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T00:46:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T00:46:41.983 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T00:46:41.983 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:46:41.983 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.983 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.983 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.983 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-01-27T00:46:41.983 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlMZxoMBAAG5kyfm+LLG86y/p2A8BfDQ==", 2024-01-27T00:46:41.983 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.983 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T00:46:41.983 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T00:46:41.983 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T00:46:41.983 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:46:41.983 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.984 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.984 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.984 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-01-27T00:46:41.984 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl5jx/ARAA3cwtLAOWQ/n2NG+xwEZ3QQ==", 2024-01-27T00:46:41.984 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.984 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T00:46:41.984 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T00:46:41.984 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T00:46:41.984 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:46:41.984 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.984 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.984 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.984 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-01-27T00:46:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4QLRlJ2OKJRAAWVhxEdQkUW20294w/KLL6g==", 2024-01-27T00:46:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T00:46:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T00:46:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:46:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-01-27T00:46:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7QLRlE2tAJRAAzNlTiCr6a2eWiabRy1SqSw==", 2024-01-27T00:46:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T00:46:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T00:46:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:46:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-01-27T00:46:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+QLRlyySaJBAA5072ba5HUNi2dYE3D02M/Q==", 2024-01-27T00:46:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T00:46:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T00:46:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:46:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-01-27T00:46:41.987 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBQLRl9+tAIxAApZk0btXIpQPWSb3BYUYsFQ==", 2024-01-27T00:46:41.987 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.987 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T00:46:41.987 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T00:46:41.987 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:46:41.987 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.987 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.987 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.987 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-01-27T00:46:41.987 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHQLRlurUtJxAAT9fwKrAWl62VOXwShg7NJg==", 2024-01-27T00:46:41.987 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.987 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T00:46:41.987 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T00:46:41.987 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:46:41.988 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.988 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.988 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.988 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-01-27T00:46:41.988 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKQLRl7aqoJRAAXu9fykWELsW3SwnahoIOLA==", 2024-01-27T00:46:41.988 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.988 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T00:46:41.988 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T00:46:41.988 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:46:41.988 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.988 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.988 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.988 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-01-27T00:46:41.989 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNQLRlWf7pJBAAopmFLIKb8s32YZs46VwbGg==", 2024-01-27T00:46:41.989 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.989 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T00:46:41.989 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T00:46:41.989 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:46:41.989 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.989 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.989 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.989 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-01-27T00:46:41.989 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQQLRlNGfhIxAAmCgmOB3QAGfSIYHBb5BUZw==", 2024-01-27T00:46:41.989 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.989 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T00:46:41.989 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T00:46:41.989 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:46:41.990 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.990 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.990 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.990 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-01-27T00:46:41.990 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRleO8wBhAA1018AcO3KcyOZxgB0Itxaw==", 2024-01-27T00:46:41.990 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.990 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T00:46:41.990 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-01-27T00:46:41.990 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-27T00:46:41.990 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.990 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.990 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.990 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-01-27T00:46:41.991 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlsiZEDRAAjXSamHe2bZcDqVsqXqIcGg==", 2024-01-27T00:46:41.991 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.991 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T00:46:41.991 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T00:46:41.991 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T00:46:41.991 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T00:46:41.991 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.991 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.991 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.991 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-01-27T00:46:41.991 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl6YT1CRAApXlN4j20xzwda0tDMy94Eg==", 2024-01-27T00:46:41.991 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.991 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T00:46:41.991 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T00:46:41.992 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T00:46:41.992 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T00:46:41.992 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.992 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.992 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.992 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-01-27T00:46:41.992 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlHvDQEhAAxMhgWJsZC5/6Af5zFV6XIA==", 2024-01-27T00:46:41.992 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.992 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T00:46:41.992 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T00:46:41.992 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T00:46:41.992 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T00:46:41.992 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.993 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.993 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.993 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-01-27T00:46:41.993 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlvMwDCBAAXnFyMduz7PfWdLWbjytIiQ==", 2024-01-27T00:46:41.993 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.993 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T00:46:41.993 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T00:46:41.993 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T00:46:41.993 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:46:41.993 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.993 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.993 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.993 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-01-27T00:46:41.993 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlhMNSJBAA38QU6DrCvFjO+u9zQ8bkEA==", 2024-01-27T00:46:41.994 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.994 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T00:46:41.994 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-27T00:46:41.994 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:46:41.994 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.994 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:41.994 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:41.994 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-01-27T00:46:41.994 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlI9EbIBAACTpzAO+8A/VFJZcT/WWBng==", 2024-01-27T00:46:41.994 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:46:41.994 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T00:46:41.994 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-27T00:46:41.994 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:46:41.994 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.994 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:41.995 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-01-27T00:46:41.995 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T00:46:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.984+0000 7f9427d95700 1 -- 172.21.15.46:0/1149898913 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9404066f30 msgr2=0x7f94040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.984+0000 7f9427d95700 1 --2- 172.21.15.46:0/1149898913 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9404066f30 0x7f94040693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9410000f60 tx=0x7f9410009040 comp rx=0 tx=0).stop 2024-01-27T00:46:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.984+0000 7f9427d95700 1 -- 172.21.15.46:0/1149898913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9420157340 msgr2=0x7f94200b3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.984+0000 7f9427d95700 1 --2- 172.21.15.46:0/1149898913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9420157340 0x7f94200b3190 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f941c0091e0 tx=0x7f941c01a000 comp rx=0 tx=0).stop 2024-01-27T00:46:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.984+0000 7f9427d95700 1 -- 172.21.15.46:0/1149898913 shutdown_connections 2024-01-27T00:46:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.984+0000 7f9427d95700 1 --2- 172.21.15.46:0/1149898913 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9404066f30 0x7f94040693e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.984+0000 7f9427d95700 1 --2- 172.21.15.46:0/1149898913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9420157340 0x7f94200b3190 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.984+0000 7f9427d95700 1 --2- 172.21.15.46:0/1149898913 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9420155a90 0x7f9420155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.984+0000 7f9427d95700 1 --2- 172.21.15.46:0/1149898913 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9420151a00 0x7f94200c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.984+0000 7f9427d95700 1 -- 172.21.15.46:0/1149898913 >> 172.21.15.46:0/1149898913 conn(0x7f94200ce530 msgr2=0x7f942006dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.984+0000 7f9427d95700 1 -- 172.21.15.46:0/1149898913 shutdown_connections 2024-01-27T00:46:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:41.984+0000 7f9427d95700 1 -- 172.21.15.46:0/1149898913 wait complete. 2024-01-27T00:46:41.998 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T00:46:42.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.130+0000 7fee284c2700 1 Processor -- start 2024-01-27T00:46:42.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.130+0000 7fee284c2700 1 -- start start 2024-01-27T00:46:42.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.131+0000 7fee284c2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee200b2c30 0x7fee201623c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.131+0000 7fee284c2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee200cc550 con 0x7fee20157550 2024-01-27T00:46:42.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.131+0000 7fee284c2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee200cb6a0 con 0x7fee20157920 2024-01-27T00:46:42.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.131+0000 7fee284c2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee200ca020 con 0x7fee200b2c30 2024-01-27T00:46:42.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2625e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fee20157550 0x7fee2006f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41070/0 (socket says 172.21.15.46:41070) 2024-01-27T00:46:42.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2625e700 1 -- 172.21.15.46:0/3667344065 learned_addr learned my addr 172.21.15.46:0/3667344065 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee26a5f700 1 --2- 172.21.15.46:0/3667344065 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee200b2c30 0x7fee201623c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2525c700 1 -- 172.21.15.46:0/3667344065 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955850331 0 0) 0x7fee200cc550 con 0x7fee20157550 2024-01-27T00:46:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2525c700 1 -- 172.21.15.46:0/3667344065 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee0c003570 con 0x7fee20157550 2024-01-27T00:46:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2525c700 1 -- 172.21.15.46:0/3667344065 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3572612952 0 0) 0x7fee0c003570 con 0x7fee20157550 2024-01-27T00:46:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2525c700 1 -- 172.21.15.46:0/3667344065 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee200b2c30 msgr2=0x7fee201623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2525c700 1 --2- 172.21.15.46:0/3667344065 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee200b2c30 0x7fee201623c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2525c700 1 -- 172.21.15.46:0/3667344065 >> v1:172.21.15.106:6789/0 conn(0x7fee20157920 legacy=0x7fee200b2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2525c700 1 -- 172.21.15.46:0/3667344065 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee200cb940 con 0x7fee20157550 2024-01-27T00:46:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2525c700 1 -- 172.21.15.46:0/3667344065 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fee14001470 con 0x7fee20157550 2024-01-27T00:46:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2525c700 1 -- 172.21.15.46:0/3667344065 >> v1:172.21.15.46:6789/0 conn(0x7fee20157550 legacy=0x7fee2006f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2525c700 1 --2- 172.21.15.46:0/3667344065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee0c003a90 0x7fee0c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2525c700 1 --2- 172.21.15.46:0/3667344065 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee200b2c30 0x7fee0c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2525c700 1 --2- 172.21.15.46:0/3667344065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee0c004d50 0x7fee0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2525c700 1 -- 172.21.15.46:0/3667344065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee200ca020 con 0x7fee0c003a90 2024-01-27T00:46:42.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2525c700 1 -- 172.21.15.46:0/3667344065 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c00bc20 con 0x7fee0c004d50 2024-01-27T00:46:42.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2525c700 1 -- 172.21.15.46:0/3667344065 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c00bde0 con 0x7fee200b2c30 2024-01-27T00:46:42.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee2625e700 1 --2- 172.21.15.46:0/3667344065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee0c003a90 0x7fee0c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee25a5d700 1 --2- 172.21.15.46:0/3667344065 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee200b2c30 0x7fee0c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.132+0000 7fee26a5f700 1 --2- 172.21.15.46:0/3667344065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee0c004d50 0x7fee0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.133+0000 7fee2625e700 1 -- 172.21.15.46:0/3667344065 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee200b2c30 msgr2=0x7fee0c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:42.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.133+0000 7fee2625e700 1 --2- 172.21.15.46:0/3667344065 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee200b2c30 0x7fee0c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.133+0000 7fee2625e700 1 -- 172.21.15.46:0/3667344065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee0c004d50 msgr2=0x7fee0c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:42.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.133+0000 7fee2625e700 1 --2- 172.21.15.46:0/3667344065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee0c004d50 0x7fee0c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.133+0000 7fee2625e700 1 -- 172.21.15.46:0/3667344065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c003570 con 0x7fee0c003a90 2024-01-27T00:46:42.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.133+0000 7fee2625e700 1 --2- 172.21.15.46:0/3667344065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee0c003a90 0x7fee0c003ea0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fee1400c390 tx=0x7fee14001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a0efc85138e6002 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:42.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.133+0000 7fee2525c700 1 -- 172.21.15.46:0/3667344065 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1400cc60 con 0x7fee0c003a90 2024-01-27T00:46:42.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.133+0000 7fee2525c700 1 -- 172.21.15.46:0/3667344065 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1400cdc0 con 0x7fee0c003a90 2024-01-27T00:46:42.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.133+0000 7fee2525c700 1 -- 172.21.15.46:0/3667344065 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee14005b50 con 0x7fee0c003a90 2024-01-27T00:46:42.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.134+0000 7fee284c2700 1 -- 172.21.15.46:0/3667344065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee0c003a90 msgr2=0x7fee0c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:42.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.134+0000 7fee284c2700 1 --2- 172.21.15.46:0/3667344065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee0c003a90 0x7fee0c003ea0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fee1400c390 tx=0x7fee14001e50 comp rx=0 tx=0).stop 2024-01-27T00:46:42.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.134+0000 7fee26a5f700 1 -- 172.21.15.46:0/3667344065 reap_dead start 2024-01-27T00:46:42.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.134+0000 7fee284c2700 1 -- 172.21.15.46:0/3667344065 shutdown_connections 2024-01-27T00:46:42.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.134+0000 7fee284c2700 1 -- 172.21.15.46:0/3667344065 >> 172.21.15.46:0/3667344065 conn(0x7fee20069510 msgr2=0x7fee200aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:42.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.134+0000 7fee284c2700 1 -- 172.21.15.46:0/3667344065 shutdown_connections 2024-01-27T00:46:42.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.134+0000 7fee284c2700 1 -- 172.21.15.46:0/3667344065 wait complete. 2024-01-27T00:46:42.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.135+0000 7fee284c2700 1 Processor -- start 2024-01-27T00:46:42.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.135+0000 7fee284c2700 1 -- start start 2024-01-27T00:46:42.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee284c2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee0c003a90 0x7fee2014b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee284c2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee20150cc0 0x7fee2015c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee284c2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee201522d0 0x7fee20155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee284c2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2014b8c0 con 0x7fee201522d0 2024-01-27T00:46:42.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee284c2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee20151230 con 0x7fee0c003a90 2024-01-27T00:46:42.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee284c2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee20151510 con 0x7fee20150cc0 2024-01-27T00:46:42.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee26a5f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee201522d0 0x7fee20155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee26a5f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee201522d0 0x7fee20155130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59688/0 (socket says 172.21.15.46:59688) 2024-01-27T00:46:42.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee26a5f700 1 -- 172.21.15.46:0/1122509757 learned_addr learned my addr 172.21.15.46:0/1122509757 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee25a5d700 1 --2- 172.21.15.46:0/1122509757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee20150cc0 0x7fee2015c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee2625e700 1 --2- 172.21.15.46:0/1122509757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee0c003a90 0x7fee2014b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee26a5f700 1 -- 172.21.15.46:0/1122509757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee20150cc0 msgr2=0x7fee2015c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee26a5f700 1 --2- 172.21.15.46:0/1122509757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee20150cc0 0x7fee2015c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee26a5f700 1 -- 172.21.15.46:0/1122509757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee0c003a90 msgr2=0x7fee2014b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee26a5f700 1 --2- 172.21.15.46:0/1122509757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee0c003a90 0x7fee2014b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee26a5f700 1 -- 172.21.15.46:0/1122509757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee201517f0 con 0x7fee201522d0 2024-01-27T00:46:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee2625e700 1 --2- 172.21.15.46:0/1122509757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee0c003a90 0x7fee2014b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:46:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee26a5f700 1 --2- 172.21.15.46:0/1122509757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee201522d0 0x7fee20155130 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fee180094d0 tx=0x7fee1800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.136+0000 7fee127fc700 1 -- 172.21.15.46:0/1122509757 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1800e040 con 0x7fee201522d0 2024-01-27T00:46:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.137+0000 7fee127fc700 1 -- 172.21.15.46:0/1122509757 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1800f040 con 0x7fee201522d0 2024-01-27T00:46:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.137+0000 7fee284c2700 1 -- 172.21.15.46:0/1122509757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee20151ad0 con 0x7fee201522d0 2024-01-27T00:46:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.137+0000 7fee127fc700 1 -- 172.21.15.46:0/1122509757 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1800fde0 con 0x7fee201522d0 2024-01-27T00:46:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.137+0000 7fee284c2700 1 -- 172.21.15.46:0/1122509757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2015cd30 con 0x7fee201522d0 2024-01-27T00:46:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.138+0000 7fee284c2700 1 -- 172.21.15.46:0/1122509757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee200c9e30 con 0x7fee201522d0 2024-01-27T00:46:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.138+0000 7fee127fc700 1 -- 172.21.15.46:0/1122509757 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fee18015640 con 0x7fee201522d0 2024-01-27T00:46:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.138+0000 7fee127fc700 1 --2- 172.21.15.46:0/1122509757 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fee04066f30 0x7fee040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.138+0000 7fee127fc700 1 -- 172.21.15.46:0/1122509757 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fee18084d60 con 0x7fee201522d0 2024-01-27T00:46:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.139+0000 7fee2625e700 1 --2- 172.21.15.46:0/1122509757 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fee04066f30 0x7fee040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.139+0000 7fee2625e700 1 --2- 172.21.15.46:0/1122509757 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fee04066f30 0x7fee040693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fee14001ed0 tx=0x7fee14015000 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:42.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.143+0000 7fee127fc700 1 -- 172.21.15.46:0/1122509757 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fee18052030 con 0x7fee201522d0 2024-01-27T00:46:42.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.352+0000 7fee284c2700 1 -- 172.21.15.46:0/1122509757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee2004c710 con 0x7fee201522d0 2024-01-27T00:46:42.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.352+0000 7fee127fc700 1 -- 172.21.15.46:0/1122509757 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+13014 (secure 0 0 0) 0x7fee18055650 con 0x7fee201522d0 2024-01-27T00:46:42.352 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:42.352 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":171,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T00:46:41.519817+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T00:46:42.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.355+0000 7fee284c2700 1 -- 172.21.15.46:0/1122509757 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fee04066f30 msgr2=0x7fee040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:42.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.355+0000 7fee284c2700 1 --2- 172.21.15.46:0/1122509757 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fee04066f30 0x7fee040693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fee14001ed0 tx=0x7fee14015000 comp rx=0 tx=0).stop 2024-01-27T00:46:42.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.355+0000 7fee284c2700 1 -- 172.21.15.46:0/1122509757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee201522d0 msgr2=0x7fee20155130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:42.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.355+0000 7fee284c2700 1 --2- 172.21.15.46:0/1122509757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee201522d0 0x7fee20155130 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fee180094d0 tx=0x7fee1800b040 comp rx=0 tx=0).stop 2024-01-27T00:46:42.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.355+0000 7fee284c2700 1 -- 172.21.15.46:0/1122509757 shutdown_connections 2024-01-27T00:46:42.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.355+0000 7fee284c2700 1 --2- 172.21.15.46:0/1122509757 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fee04066f30 0x7fee040693e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.355+0000 7fee284c2700 1 --2- 172.21.15.46:0/1122509757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee201522d0 0x7fee20155130 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.355+0000 7fee284c2700 1 --2- 172.21.15.46:0/1122509757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee0c003a90 0x7fee2014b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.355+0000 7fee284c2700 1 --2- 172.21.15.46:0/1122509757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee20150cc0 0x7fee2015c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.355+0000 7fee284c2700 1 -- 172.21.15.46:0/1122509757 >> 172.21.15.46:0/1122509757 conn(0x7fee20069510 msgr2=0x7fee2006f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:42.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.355+0000 7fee284c2700 1 -- 172.21.15.46:0/1122509757 shutdown_connections 2024-01-27T00:46:42.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.355+0000 7fee284c2700 1 -- 172.21.15.46:0/1122509757 wait complete. 2024-01-27T00:46:42.368 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-27T00:46:42.368 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-27T00:46:42.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.505+0000 7f99636a4700 1 Processor -- start 2024-01-27T00:46:42.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.505+0000 7f99636a4700 1 -- start start 2024-01-27T00:46:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.506+0000 7f99636a4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f995c1597d0 0x7f995c06ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.506+0000 7f99636a4700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f995c15ad40 0x7f995c06f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.506+0000 7f99636a4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f995c159ba0 0x7f995c0cb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.506+0000 7f99636a4700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f995c0cbfb0 con 0x7f995c1597d0 2024-01-27T00:46:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.506+0000 7f99636a4700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f995c15a8b0 con 0x7f995c15ad40 2024-01-27T00:46:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.506+0000 7f99636a4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f995c0cbcd0 con 0x7f995c159ba0 2024-01-27T00:46:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.506+0000 7f9961440700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f995c1597d0 0x7f995c06ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.506+0000 7f9961440700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f995c1597d0 0x7f995c06ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59704/0 (socket says 172.21.15.46:59704) 2024-01-27T00:46:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.506+0000 7f9961440700 1 -- 172.21.15.46:0/262902320 learned_addr learned my addr 172.21.15.46:0/262902320 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.506+0000 7f9961c41700 1 --2- 172.21.15.46:0/262902320 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f995c159ba0 0x7f995c0cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.506+0000 7f9960c3f700 1 --2- 172.21.15.46:0/262902320 >> v2:172.21.15.106:3300/0 conn(0x7f995c15ad40 0x7f995c06f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.507+0000 7f9961440700 1 -- 172.21.15.46:0/262902320 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f995c159ba0 msgr2=0x7f995c0cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.507+0000 7f9961440700 1 --2- 172.21.15.46:0/262902320 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f995c159ba0 0x7f995c0cb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.507+0000 7f9961440700 1 -- 172.21.15.46:0/262902320 >> v2:172.21.15.106:3300/0 conn(0x7f995c15ad40 msgr2=0x7f995c06f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.507+0000 7f9961440700 1 --2- 172.21.15.46:0/262902320 >> v2:172.21.15.106:3300/0 conn(0x7f995c15ad40 0x7f995c06f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.507+0000 7f9961440700 1 -- 172.21.15.46:0/262902320 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f995c15a260 con 0x7f995c1597d0 2024-01-27T00:46:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.507+0000 7f9960c3f700 1 --2- 172.21.15.46:0/262902320 >> v2:172.21.15.106:3300/0 conn(0x7f995c15ad40 0x7f995c06f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:46:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.507+0000 7f9961c41700 1 --2- 172.21.15.46:0/262902320 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f995c159ba0 0x7f995c0cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:46:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.507+0000 7f9961440700 1 --2- 172.21.15.46:0/262902320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c1597d0 0x7f995c06ec50 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f995c06e640 tx=0x7f9958009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7f4c5074dcb57c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.507+0000 7f994ffff700 1 -- 172.21.15.46:0/262902320 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995800b860 con 0x7f995c1597d0 2024-01-27T00:46:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.507+0000 7f994ffff700 1 -- 172.21.15.46:0/262902320 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995800b9c0 con 0x7f995c1597d0 2024-01-27T00:46:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.507+0000 7f994ffff700 1 -- 172.21.15.46:0/262902320 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995800bd80 con 0x7f995c1597d0 2024-01-27T00:46:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.508+0000 7f99636a4700 1 -- 172.21.15.46:0/262902320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c1597d0 msgr2=0x7f995c06ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.508+0000 7f99636a4700 1 --2- 172.21.15.46:0/262902320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c1597d0 0x7f995c06ec50 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f995c06e640 tx=0x7f9958009250 comp rx=0 tx=0).stop 2024-01-27T00:46:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.508+0000 7f99636a4700 1 -- 172.21.15.46:0/262902320 shutdown_connections 2024-01-27T00:46:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.508+0000 7f99636a4700 1 --2- 172.21.15.46:0/262902320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c1597d0 0x7f995c06ec50 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.508+0000 7f99636a4700 1 --2- 172.21.15.46:0/262902320 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f995c159ba0 0x7f995c0cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.508+0000 7f99636a4700 1 --2- 172.21.15.46:0/262902320 >> v2:172.21.15.106:3300/0 conn(0x7f995c15ad40 0x7f995c06f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.508+0000 7f99636a4700 1 -- 172.21.15.46:0/262902320 >> 172.21.15.46:0/262902320 conn(0x7f995c0ca9a0 msgr2=0x7f995c068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.508+0000 7f99636a4700 1 -- 172.21.15.46:0/262902320 shutdown_connections 2024-01-27T00:46:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.508+0000 7f99636a4700 1 -- 172.21.15.46:0/262902320 wait complete. 2024-01-27T00:46:42.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.509+0000 7f99636a4700 1 Processor -- start 2024-01-27T00:46:42.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.509+0000 7f99636a4700 1 -- start start 2024-01-27T00:46:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.510+0000 7f99636a4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f995c1597d0 0x7f995c154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.510+0000 7f99636a4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c159ba0 0x7f995c167ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.510+0000 7f99636a4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f995c15ad40 0x7f995c0c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.510+0000 7f99636a4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995c157810 con 0x7f995c159ba0 2024-01-27T00:46:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.510+0000 7f99636a4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995c1683c0 con 0x7f995c15ad40 2024-01-27T00:46:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.510+0000 7f99636a4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f995c1686a0 con 0x7f995c1597d0 2024-01-27T00:46:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.510+0000 7f9960c3f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c159ba0 0x7f995c167ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.510+0000 7f9960c3f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c159ba0 0x7f995c167ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59712/0 (socket says 172.21.15.46:59712) 2024-01-27T00:46:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.510+0000 7f9960c3f700 1 -- 172.21.15.46:0/1354948837 learned_addr learned my addr 172.21.15.46:0/1354948837 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.510+0000 7f9961440700 1 --2- 172.21.15.46:0/1354948837 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f995c1597d0 0x7f995c154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.510+0000 7f9961c41700 1 --2- 172.21.15.46:0/1354948837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f995c15ad40 0x7f995c0c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.511+0000 7f9960c3f700 1 -- 172.21.15.46:0/1354948837 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f995c1597d0 msgr2=0x7f995c154bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.511+0000 7f9960c3f700 1 --2- 172.21.15.46:0/1354948837 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f995c1597d0 0x7f995c154bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.511+0000 7f9960c3f700 1 -- 172.21.15.46:0/1354948837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f995c15ad40 msgr2=0x7f995c0c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.511+0000 7f9960c3f700 1 --2- 172.21.15.46:0/1354948837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f995c15ad40 0x7f995c0c29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.511+0000 7f9960c3f700 1 -- 172.21.15.46:0/1354948837 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99580094f0 con 0x7f995c159ba0 2024-01-27T00:46:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.511+0000 7f9960c3f700 1 --2- 172.21.15.46:0/1354948837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c159ba0 0x7f995c167ba0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f995000d860 tx=0x7f9950008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.511+0000 7f994d7fa700 1 -- 172.21.15.46:0/1354948837 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9950009610 con 0x7f995c159ba0 2024-01-27T00:46:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.511+0000 7f994d7fa700 1 -- 172.21.15.46:0/1354948837 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9950009770 con 0x7f995c159ba0 2024-01-27T00:46:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.511+0000 7f99636a4700 1 -- 172.21.15.46:0/1354948837 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f995c168980 con 0x7f995c159ba0 2024-01-27T00:46:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.511+0000 7f994d7fa700 1 -- 172.21.15.46:0/1354948837 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9950009b10 con 0x7f995c159ba0 2024-01-27T00:46:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.511+0000 7f99636a4700 1 -- 172.21.15.46:0/1354948837 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f995c1680e0 con 0x7f995c159ba0 2024-01-27T00:46:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.512+0000 7f99636a4700 1 -- 172.21.15.46:0/1354948837 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f995c157cd0 con 0x7f995c159ba0 2024-01-27T00:46:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.512+0000 7f994d7fa700 1 -- 172.21.15.46:0/1354948837 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9950014430 con 0x7f995c159ba0 2024-01-27T00:46:42.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.513+0000 7f994d7fa700 1 --2- 172.21.15.46:0/1354948837 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9940066f30 0x7f99400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.513+0000 7f994d7fa700 1 -- 172.21.15.46:0/1354948837 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f9950084900 con 0x7f995c159ba0 2024-01-27T00:46:42.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.513+0000 7f9961440700 1 --2- 172.21.15.46:0/1354948837 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9940066f30 0x7f99400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.514+0000 7f9961440700 1 --2- 172.21.15.46:0/1354948837 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9940066f30 0x7f99400693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f995c06e640 tx=0x7f9958017000 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:42.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.518+0000 7f994d7fa700 1 -- 172.21.15.46:0/1354948837 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f995c157cd0 con 0x7f995c159ba0 2024-01-27T00:46:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.719+0000 7f99636a4700 1 -- 172.21.15.46:0/1354948837 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f995c157cd0 con 0x7f995c159ba0 2024-01-27T00:46:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.720+0000 7f994d7fa700 1 -- 172.21.15.46:0/1354948837 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v171) v1 ==== 93+0+10 (secure 0 0 0) 0x7f995c157cd0 con 0x7f995c159ba0 2024-01-27T00:46:42.719 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-27T00:46:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.722+0000 7f99636a4700 1 -- 172.21.15.46:0/1354948837 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9940066f30 msgr2=0x7f99400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.722+0000 7f99636a4700 1 --2- 172.21.15.46:0/1354948837 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9940066f30 0x7f99400693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f995c06e640 tx=0x7f9958017000 comp rx=0 tx=0).stop 2024-01-27T00:46:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.722+0000 7f99636a4700 1 -- 172.21.15.46:0/1354948837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c159ba0 msgr2=0x7f995c167ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.722+0000 7f99636a4700 1 --2- 172.21.15.46:0/1354948837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c159ba0 0x7f995c167ba0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f995000d860 tx=0x7f9950008040 comp rx=0 tx=0).stop 2024-01-27T00:46:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.722+0000 7f99636a4700 1 -- 172.21.15.46:0/1354948837 shutdown_connections 2024-01-27T00:46:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.722+0000 7f99636a4700 1 --2- 172.21.15.46:0/1354948837 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9940066f30 0x7f99400693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.722+0000 7f99636a4700 1 --2- 172.21.15.46:0/1354948837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f995c15ad40 0x7f995c0c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.722+0000 7f99636a4700 1 --2- 172.21.15.46:0/1354948837 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f995c1597d0 0x7f995c154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.722+0000 7f99636a4700 1 --2- 172.21.15.46:0/1354948837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f995c159ba0 0x7f995c167ba0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.722+0000 7f99636a4700 1 -- 172.21.15.46:0/1354948837 >> 172.21.15.46:0/1354948837 conn(0x7f995c0ca9a0 msgr2=0x7f995c0679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.722+0000 7f99636a4700 1 -- 172.21.15.46:0/1354948837 shutdown_connections 2024-01-27T00:46:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.722+0000 7f99636a4700 1 -- 172.21.15.46:0/1354948837 wait complete. 2024-01-27T00:46:42.736 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-27T00:46:42.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.868+0000 7fb56b93b700 1 Processor -- start 2024-01-27T00:46:42.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.869+0000 7fb56b93b700 1 -- start start 2024-01-27T00:46:42.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.870+0000 7fb56b93b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5640b4180 0x7fb564157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.870+0000 7fb56b93b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb564158ae0 con 0x7fb564157340 2024-01-27T00:46:42.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.870+0000 7fb56b93b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56415b650 con 0x7fb5640ae520 2024-01-27T00:46:42.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.870+0000 7fb56b93b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb564158520 con 0x7fb5640b4180 2024-01-27T00:46:42.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.870+0000 7fb5696d7700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb564157340 0x7fb56415bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41080/0 (socket says 172.21.15.46:41080) 2024-01-27T00:46:42.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.870+0000 7fb5696d7700 1 -- 172.21.15.46:0/3896147770 learned_addr learned my addr 172.21.15.46:0/3896147770 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:42.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.870+0000 7fb569ed8700 1 --2- 172.21.15.46:0/3896147770 >> v2:172.21.15.46:3300/0 conn(0x7fb5640b4180 0x7fb564157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.871+0000 7fb569ed8700 1 -- 172.21.15.46:0/3896147770 >> v1:172.21.15.106:6789/0 conn(0x7fb5640ae520 legacy=0x7fb5640b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:42.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.871+0000 7fb569ed8700 1 -- 172.21.15.46:0/3896147770 >> v1:172.21.15.46:6789/0 conn(0x7fb564157340 legacy=0x7fb56415bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:42.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.871+0000 7fb569ed8700 1 -- 172.21.15.46:0/3896147770 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb564158800 con 0x7fb5640b4180 2024-01-27T00:46:42.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.871+0000 7fb569ed8700 1 --2- 172.21.15.46:0/3896147770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640b4180 0x7fb564157710 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009e60 tx=0x7fb55c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da98d56f6571249 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:42.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.871+0000 7fb55bfff700 1 -- 172.21.15.46:0/3896147770 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c00e070 con 0x7fb5640b4180 2024-01-27T00:46:42.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.871+0000 7fb55bfff700 1 -- 172.21.15.46:0/3896147770 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55c00bc30 con 0x7fb5640b4180 2024-01-27T00:46:42.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.871+0000 7fb55bfff700 1 -- 172.21.15.46:0/3896147770 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c012430 con 0x7fb5640b4180 2024-01-27T00:46:42.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.871+0000 7fb56b93b700 1 -- 172.21.15.46:0/3896147770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640b4180 msgr2=0x7fb564157710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:42.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.871+0000 7fb56b93b700 1 --2- 172.21.15.46:0/3896147770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640b4180 0x7fb564157710 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009e60 tx=0x7fb55c009780 comp rx=0 tx=0).stop 2024-01-27T00:46:42.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.872+0000 7fb56b93b700 1 -- 172.21.15.46:0/3896147770 shutdown_connections 2024-01-27T00:46:42.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.872+0000 7fb56b93b700 1 --2- 172.21.15.46:0/3896147770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640b4180 0x7fb564157710 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.872+0000 7fb56b93b700 1 -- 172.21.15.46:0/3896147770 >> 172.21.15.46:0/3896147770 conn(0x7fb5640ce530 msgr2=0x7fb5640ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:42.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.872+0000 7fb56b93b700 1 -- 172.21.15.46:0/3896147770 shutdown_connections 2024-01-27T00:46:42.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.872+0000 7fb56b93b700 1 -- 172.21.15.46:0/3896147770 wait complete. 2024-01-27T00:46:42.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.873+0000 7fb56b93b700 1 Processor -- start 2024-01-27T00:46:42.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.873+0000 7fb56b93b700 1 -- start start 2024-01-27T00:46:42.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.874+0000 7fb56b93b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640ae520 0x7fb564069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.874+0000 7fb56b93b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5640b4180 0x7fb564154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.874+0000 7fb56b93b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb564157340 0x7fb564155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.874+0000 7fb56b93b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb564069990 con 0x7fb5640ae520 2024-01-27T00:46:42.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.874+0000 7fb56b93b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5640ca480 con 0x7fb5640b4180 2024-01-27T00:46:42.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.874+0000 7fb56b93b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5640ca760 con 0x7fb564157340 2024-01-27T00:46:42.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.874+0000 7fb5696d7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640ae520 0x7fb564069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.874+0000 7fb5696d7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640ae520 0x7fb564069450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59720/0 (socket says 172.21.15.46:59720) 2024-01-27T00:46:42.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.874+0000 7fb5696d7700 1 -- 172.21.15.46:0/1725490770 learned_addr learned my addr 172.21.15.46:0/1725490770 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:42.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.874+0000 7fb569ed8700 1 --2- 172.21.15.46:0/1725490770 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb564157340 0x7fb564155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.874+0000 7fb5696d7700 1 -- 172.21.15.46:0/1725490770 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb564157340 msgr2=0x7fb564155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:42.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.874+0000 7fb568ed6700 1 --2- 172.21.15.46:0/1725490770 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5640b4180 0x7fb564154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.874+0000 7fb5696d7700 1 --2- 172.21.15.46:0/1725490770 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb564157340 0x7fb564155be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.874+0000 7fb5696d7700 1 -- 172.21.15.46:0/1725490770 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5640b4180 msgr2=0x7fb564154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:42.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.874+0000 7fb5696d7700 1 --2- 172.21.15.46:0/1725490770 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5640b4180 0x7fb564154380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:42.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.874+0000 7fb5696d7700 1 -- 172.21.15.46:0/1725490770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55c009a20 con 0x7fb5640ae520 2024-01-27T00:46:42.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.874+0000 7fb568ed6700 1 --2- 172.21.15.46:0/1725490770 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5640b4180 0x7fb564154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:46:42.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.875+0000 7fb5696d7700 1 --2- 172.21.15.46:0/1725490770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640ae520 0x7fb564069450 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fb56000c7c0 tx=0x7fb560007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:42.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.875+0000 7fb5597fa700 1 -- 172.21.15.46:0/1725490770 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560005b30 con 0x7fb5640ae520 2024-01-27T00:46:42.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.875+0000 7fb5597fa700 1 -- 172.21.15.46:0/1725490770 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb560005cb0 con 0x7fb5640ae520 2024-01-27T00:46:42.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.875+0000 7fb56b93b700 1 -- 172.21.15.46:0/1725490770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5640caa40 con 0x7fb5640ae520 2024-01-27T00:46:42.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.875+0000 7fb5597fa700 1 -- 172.21.15.46:0/1725490770 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560004a80 con 0x7fb5640ae520 2024-01-27T00:46:42.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.875+0000 7fb56b93b700 1 -- 172.21.15.46:0/1725490770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5641548c0 con 0x7fb5640ae520 2024-01-27T00:46:42.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.876+0000 7fb56b93b700 1 -- 172.21.15.46:0/1725490770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb56404c710 con 0x7fb5640ae520 2024-01-27T00:46:42.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.877+0000 7fb5597fa700 1 -- 172.21.15.46:0/1725490770 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb5600070f0 con 0x7fb5640ae520 2024-01-27T00:46:42.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.877+0000 7fb5597fa700 1 --2- 172.21.15.46:0/1725490770 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb548066f30 0x7fb5480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:42.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.877+0000 7fb5597fa700 1 -- 172.21.15.46:0/1725490770 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fb560083950 con 0x7fb5640ae520 2024-01-27T00:46:42.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.877+0000 7fb568ed6700 1 --2- 172.21.15.46:0/1725490770 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb548066f30 0x7fb5480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:42.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.878+0000 7fb568ed6700 1 --2- 172.21.15.46:0/1725490770 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb548066f30 0x7fb5480693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb564157040 tx=0x7fb554009040 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:42.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:42.882+0000 7fb5597fa700 1 -- 172.21.15.46:0/1725490770 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb560050ca0 con 0x7fb5640ae520 2024-01-27T00:46:43.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.090+0000 7fb56b93b700 1 -- 172.21.15.46:0/1725490770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb56415b2e0 con 0x7fb5640ae520 2024-01-27T00:46:43.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.090+0000 7fb5597fa700 1 -- 172.21.15.46:0/1725490770 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v171) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb560018070 con 0x7fb5640ae520 2024-01-27T00:46:43.090 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:43.090 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-27T00:46:43.090 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:43.090 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-01-27T00:46:43.090 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-01-27T00:46:43.090 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:43.090 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-27T00:46:43.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.093+0000 7fb56b93b700 1 -- 172.21.15.46:0/1725490770 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb548066f30 msgr2=0x7fb5480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:43.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.093+0000 7fb56b93b700 1 --2- 172.21.15.46:0/1725490770 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb548066f30 0x7fb5480693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb564157040 tx=0x7fb554009040 comp rx=0 tx=0).stop 2024-01-27T00:46:43.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.093+0000 7fb56b93b700 1 -- 172.21.15.46:0/1725490770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640ae520 msgr2=0x7fb564069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:43.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.093+0000 7fb56b93b700 1 --2- 172.21.15.46:0/1725490770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640ae520 0x7fb564069450 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fb56000c7c0 tx=0x7fb560007f70 comp rx=0 tx=0).stop 2024-01-27T00:46:43.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.093+0000 7fb56b93b700 1 -- 172.21.15.46:0/1725490770 shutdown_connections 2024-01-27T00:46:43.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.093+0000 7fb56b93b700 1 --2- 172.21.15.46:0/1725490770 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb548066f30 0x7fb5480693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.093+0000 7fb56b93b700 1 --2- 172.21.15.46:0/1725490770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640ae520 0x7fb564069450 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.093+0000 7fb56b93b700 1 --2- 172.21.15.46:0/1725490770 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb564157340 0x7fb564155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.093+0000 7fb56b93b700 1 --2- 172.21.15.46:0/1725490770 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5640b4180 0x7fb564154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.093+0000 7fb56b93b700 1 -- 172.21.15.46:0/1725490770 >> 172.21.15.46:0/1725490770 conn(0x7fb5640ce530 msgr2=0x7fb56415bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:43.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.093+0000 7fb56b93b700 1 -- 172.21.15.46:0/1725490770 shutdown_connections 2024-01-27T00:46:43.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.093+0000 7fb56b93b700 1 -- 172.21.15.46:0/1725490770 wait complete. 2024-01-27T00:46:43.106 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-27T00:46:43.106 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-27T00:46:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.242+0000 7f41576b2700 1 Processor -- start 2024-01-27T00:46:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.242+0000 7f41576b2700 1 -- start start 2024-01-27T00:46:43.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.243+0000 7f41576b2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4150067af0 0x7f4150067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:43.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.243+0000 7f41576b2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f415006fc90 0x7f41500b1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:43.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.243+0000 7f41576b2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41500b2b90 0x7f4150161960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:43.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.243+0000 7f41576b2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f415006f800 con 0x7f4150067af0 2024-01-27T00:46:43.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.243+0000 7f41576b2700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f415006ec30 con 0x7f415006fc90 2024-01-27T00:46:43.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.243+0000 7f41576b2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415006eed0 con 0x7f41500b2b90 2024-01-27T00:46:43.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.244+0000 7f415544e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4150067af0 0x7f4150067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:43.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.244+0000 7f415544e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4150067af0 0x7f4150067ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59734/0 (socket says 172.21.15.46:59734) 2024-01-27T00:46:43.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.244+0000 7f415544e700 1 -- 172.21.15.46:0/2718505739 learned_addr learned my addr 172.21.15.46:0/2718505739 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:43.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.244+0000 7f4155c4f700 1 --2- 172.21.15.46:0/2718505739 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41500b2b90 0x7f4150161960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:43.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.244+0000 7f4154c4d700 1 --2- 172.21.15.46:0/2718505739 >> v2:172.21.15.106:3300/0 conn(0x7f415006fc90 0x7f41500b1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:43.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.244+0000 7f415544e700 1 -- 172.21.15.46:0/2718505739 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41500b2b90 msgr2=0x7f4150161960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:43.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.244+0000 7f415544e700 1 --2- 172.21.15.46:0/2718505739 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41500b2b90 0x7f4150161960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.244+0000 7f415544e700 1 -- 172.21.15.46:0/2718505739 >> v2:172.21.15.106:3300/0 conn(0x7f415006fc90 msgr2=0x7f41500b1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:43.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.244+0000 7f415544e700 1 --2- 172.21.15.46:0/2718505739 >> v2:172.21.15.106:3300/0 conn(0x7f415006fc90 0x7f41500b1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.244+0000 7f415544e700 1 -- 172.21.15.46:0/2718505739 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415006f1b0 con 0x7f4150067af0 2024-01-27T00:46:43.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.244+0000 7f415544e700 1 --2- 172.21.15.46:0/2718505739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4150067af0 0x7f4150067ec0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f4150068950 tx=0x7f4140009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a72d5dc68ca888 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:43.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.244+0000 7f4147fff700 1 -- 172.21.15.46:0/2718505739 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414000b640 con 0x7f4150067af0 2024-01-27T00:46:43.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.244+0000 7f4147fff700 1 -- 172.21.15.46:0/2718505739 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f414000b7a0 con 0x7f4150067af0 2024-01-27T00:46:43.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.244+0000 7f4147fff700 1 -- 172.21.15.46:0/2718505739 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414000bb60 con 0x7f4150067af0 2024-01-27T00:46:43.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.245+0000 7f41576b2700 1 -- 172.21.15.46:0/2718505739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4150067af0 msgr2=0x7f4150067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:43.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.245+0000 7f41576b2700 1 --2- 172.21.15.46:0/2718505739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4150067af0 0x7f4150067ec0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f4150068950 tx=0x7f4140009250 comp rx=0 tx=0).stop 2024-01-27T00:46:43.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.245+0000 7f41576b2700 1 -- 172.21.15.46:0/2718505739 shutdown_connections 2024-01-27T00:46:43.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.245+0000 7f41576b2700 1 --2- 172.21.15.46:0/2718505739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4150067af0 0x7f4150067ec0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.245+0000 7f41576b2700 1 --2- 172.21.15.46:0/2718505739 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41500b2b90 0x7f4150161960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.245+0000 7f41576b2700 1 --2- 172.21.15.46:0/2718505739 >> v2:172.21.15.106:3300/0 conn(0x7f415006fc90 0x7f41500b1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.245+0000 7f41576b2700 1 -- 172.21.15.46:0/2718505739 >> 172.21.15.46:0/2718505739 conn(0x7f4150069510 msgr2=0x7f41500aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:43.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.245+0000 7f41576b2700 1 -- 172.21.15.46:0/2718505739 shutdown_connections 2024-01-27T00:46:43.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.245+0000 7f41576b2700 1 -- 172.21.15.46:0/2718505739 wait complete. 2024-01-27T00:46:43.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.246+0000 7f41576b2700 1 Processor -- start 2024-01-27T00:46:43.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.246+0000 7f41576b2700 1 -- start start 2024-01-27T00:46:43.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.247+0000 7f41576b2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4150067af0 0x7f41501544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:43.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.247+0000 7f41576b2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f415006fc90 0x7f415015f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:43.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.247+0000 7f41576b2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41500b2b90 0x7f41500add20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:43.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.247+0000 7f41576b2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41500a5430 con 0x7f41500b2b90 2024-01-27T00:46:43.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.247+0000 7f41576b2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4150155be0 con 0x7f4150067af0 2024-01-27T00:46:43.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.247+0000 7f41576b2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4150155ec0 con 0x7f415006fc90 2024-01-27T00:46:43.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.247+0000 7f4155c4f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41500b2b90 0x7f41500add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:43.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.247+0000 7f4155c4f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41500b2b90 0x7f41500add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59748/0 (socket says 172.21.15.46:59748) 2024-01-27T00:46:43.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.247+0000 7f4155c4f700 1 -- 172.21.15.46:0/488912021 learned_addr learned my addr 172.21.15.46:0/488912021 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:43.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.247+0000 7f4154c4d700 1 --2- 172.21.15.46:0/488912021 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f415006fc90 0x7f415015f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:43.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.248+0000 7f415544e700 1 --2- 172.21.15.46:0/488912021 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4150067af0 0x7f41501544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:43.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.248+0000 7f4155c4f700 1 -- 172.21.15.46:0/488912021 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f415006fc90 msgr2=0x7f415015f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:43.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.248+0000 7f4155c4f700 1 --2- 172.21.15.46:0/488912021 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f415006fc90 0x7f415015f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.248+0000 7f4155c4f700 1 -- 172.21.15.46:0/488912021 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4150067af0 msgr2=0x7f41501544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:43.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.248+0000 7f4155c4f700 1 --2- 172.21.15.46:0/488912021 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4150067af0 0x7f41501544b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.248+0000 7f4155c4f700 1 -- 172.21.15.46:0/488912021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41400094f0 con 0x7f41500b2b90 2024-01-27T00:46:43.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.248+0000 7f4155c4f700 1 --2- 172.21.15.46:0/488912021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41500b2b90 0x7f41500add20 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f4148009e90 tx=0x7f414800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:43.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.248+0000 7f41457fa700 1 -- 172.21.15.46:0/488912021 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414800cb70 con 0x7f41500b2b90 2024-01-27T00:46:43.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.248+0000 7f41457fa700 1 -- 172.21.15.46:0/488912021 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f414800f040 con 0x7f41500b2b90 2024-01-27T00:46:43.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.248+0000 7f41576b2700 1 -- 172.21.15.46:0/488912021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41501561a0 con 0x7f41500b2b90 2024-01-27T00:46:43.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.248+0000 7f41457fa700 1 -- 172.21.15.46:0/488912021 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4148014480 con 0x7f41500b2b90 2024-01-27T00:46:43.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.248+0000 7f41576b2700 1 -- 172.21.15.46:0/488912021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41501549f0 con 0x7f41500b2b90 2024-01-27T00:46:43.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.249+0000 7f41576b2700 1 -- 172.21.15.46:0/488912021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f415004c710 con 0x7f41500b2b90 2024-01-27T00:46:43.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.250+0000 7f41457fa700 1 -- 172.21.15.46:0/488912021 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4148014830 con 0x7f41500b2b90 2024-01-27T00:46:43.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.250+0000 7f41457fa700 1 --2- 172.21.15.46:0/488912021 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4134066f30 0x7f41340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:43.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.250+0000 7f41457fa700 1 -- 172.21.15.46:0/488912021 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f4148085180 con 0x7f41500b2b90 2024-01-27T00:46:43.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.250+0000 7f415544e700 1 --2- 172.21.15.46:0/488912021 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4134066f30 0x7f41340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:43.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.251+0000 7f415544e700 1 --2- 172.21.15.46:0/488912021 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4134066f30 0x7f41340693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4150068950 tx=0x7f4140017000 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:43.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.255+0000 7f41457fa700 1 -- 172.21.15.46:0/488912021 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4148052450 con 0x7f41500b2b90 2024-01-27T00:46:43.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.459+0000 7f41576b2700 1 -- 172.21.15.46:0/488912021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f415006a310 con 0x7f41500b2b90 2024-01-27T00:46:43.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.533+0000 7f41457fa700 1 -- 172.21.15.46:0/488912021 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v172) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4148055a70 con 0x7f41500b2b90 2024-01-27T00:46:43.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.536+0000 7f41576b2700 1 -- 172.21.15.46:0/488912021 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4134066f30 msgr2=0x7f41340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:43.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.536+0000 7f41576b2700 1 --2- 172.21.15.46:0/488912021 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4134066f30 0x7f41340693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4150068950 tx=0x7f4140017000 comp rx=0 tx=0).stop 2024-01-27T00:46:43.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.536+0000 7f41576b2700 1 -- 172.21.15.46:0/488912021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41500b2b90 msgr2=0x7f41500add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:43.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.536+0000 7f41576b2700 1 --2- 172.21.15.46:0/488912021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41500b2b90 0x7f41500add20 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f4148009e90 tx=0x7f414800b040 comp rx=0 tx=0).stop 2024-01-27T00:46:43.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.537+0000 7f41576b2700 1 -- 172.21.15.46:0/488912021 shutdown_connections 2024-01-27T00:46:43.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.537+0000 7f41576b2700 1 --2- 172.21.15.46:0/488912021 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4134066f30 0x7f41340693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.537+0000 7f41576b2700 1 --2- 172.21.15.46:0/488912021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41500b2b90 0x7f41500add20 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.537+0000 7f41576b2700 1 --2- 172.21.15.46:0/488912021 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4150067af0 0x7f41501544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.537+0000 7f41576b2700 1 --2- 172.21.15.46:0/488912021 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f415006fc90 0x7f415015f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.537+0000 7f41576b2700 1 -- 172.21.15.46:0/488912021 >> 172.21.15.46:0/488912021 conn(0x7f4150069510 msgr2=0x7f41501583b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:43.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.538+0000 7f41576b2700 1 -- 172.21.15.46:0/488912021 shutdown_connections 2024-01-27T00:46:43.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.538+0000 7f41576b2700 1 -- 172.21.15.46:0/488912021 wait complete. 2024-01-27T00:46:43.540 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-01-27T00:46:43.553 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-27T00:46:43.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.693+0000 7fe7d2c29700 1 Processor -- start 2024-01-27T00:46:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.693+0000 7fe7d2c29700 1 -- start start 2024-01-27T00:46:43.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.694+0000 7fe7d2c29700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7cc0c9770 0x7fe7cc159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:43.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.694+0000 7fe7d2c29700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe7cc159760 0x7fe7cc0c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:43.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.694+0000 7fe7d2c29700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7cc15e390 con 0x7fe7cc0c93a0 2024-01-27T00:46:43.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.694+0000 7fe7d2c29700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7cc0c9be0 con 0x7fe7cc159760 2024-01-27T00:46:43.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.694+0000 7fe7d2c29700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7cc068460 con 0x7fe7cc0c9770 2024-01-27T00:46:43.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.694+0000 7fe7d11c6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe7cc159760 0x7fe7cc0c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:43.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.694+0000 7fe7d09c5700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe7cc0c93a0 0x7fe7cc067d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41094/0 (socket says 172.21.15.46:41094) 2024-01-27T00:46:43.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.694+0000 7fe7d09c5700 1 -- 172.21.15.46:0/4014998881 learned_addr learned my addr 172.21.15.46:0/4014998881 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:43.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.694+0000 7fe7cbfff700 1 --2- 172.21.15.46:0/4014998881 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7cc0c9770 0x7fe7cc159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:43.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.694+0000 7fe7d11c6700 1 -- 172.21.15.46:0/4014998881 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7cc0c9770 msgr2=0x7fe7cc159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:43.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.694+0000 7fe7d11c6700 1 --2- 172.21.15.46:0/4014998881 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7cc0c9770 0x7fe7cc159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.694+0000 7fe7d11c6700 1 -- 172.21.15.46:0/4014998881 >> v1:172.21.15.46:6789/0 conn(0x7fe7cc0c93a0 legacy=0x7fe7cc067d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:43.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.694+0000 7fe7d11c6700 1 -- 172.21.15.46:0/4014998881 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7cc15d810 con 0x7fe7cc159760 2024-01-27T00:46:43.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.694+0000 7fe7cb7fe700 1 -- 172.21.15.46:0/4014998881 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752955345 0 0) 0x7fe7cc15e390 con 0x7fe7cc0c93a0 2024-01-27T00:46:43.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.694+0000 7fe7cbfff700 1 --2- 172.21.15.46:0/4014998881 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7cc0c9770 0x7fe7cc159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:46:43.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.695+0000 7fe7d11c6700 1 --2- 172.21.15.46:0/4014998881 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc159760 0x7fe7cc0c7e10 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4009e60 tx=0x7fe7c4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c628698df5b588cc server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:43.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.695+0000 7fe7cb7fe700 1 -- 172.21.15.46:0/4014998881 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c400e070 con 0x7fe7cc159760 2024-01-27T00:46:43.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.695+0000 7fe7cb7fe700 1 -- 172.21.15.46:0/4014998881 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c400bc30 con 0x7fe7cc159760 2024-01-27T00:46:43.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.695+0000 7fe7cb7fe700 1 -- 172.21.15.46:0/4014998881 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4012430 con 0x7fe7cc159760 2024-01-27T00:46:43.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.695+0000 7fe7d2c29700 1 -- 172.21.15.46:0/4014998881 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc159760 msgr2=0x7fe7cc0c7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:43.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.695+0000 7fe7d2c29700 1 --2- 172.21.15.46:0/4014998881 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc159760 0x7fe7cc0c7e10 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4009e60 tx=0x7fe7c4009780 comp rx=0 tx=0).stop 2024-01-27T00:46:43.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.695+0000 7fe7d2c29700 1 -- 172.21.15.46:0/4014998881 shutdown_connections 2024-01-27T00:46:43.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.695+0000 7fe7d2c29700 1 --2- 172.21.15.46:0/4014998881 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc159760 0x7fe7cc0c7e10 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.695+0000 7fe7d2c29700 1 --2- 172.21.15.46:0/4014998881 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7cc0c9770 0x7fe7cc159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.695+0000 7fe7d2c29700 1 -- 172.21.15.46:0/4014998881 >> 172.21.15.46:0/4014998881 conn(0x7fe7cc0ca9a0 msgr2=0x7fe7cc068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:43.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.695+0000 7fe7d2c29700 1 -- 172.21.15.46:0/4014998881 shutdown_connections 2024-01-27T00:46:43.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.695+0000 7fe7d2c29700 1 -- 172.21.15.46:0/4014998881 wait complete. 2024-01-27T00:46:43.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.697+0000 7fe7d2c29700 1 Processor -- start 2024-01-27T00:46:43.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.697+0000 7fe7d2c29700 1 -- start start 2024-01-27T00:46:43.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.698+0000 7fe7d2c29700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7cc0c93a0 0x7fe7cc143010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:43.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.698+0000 7fe7d2c29700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7cc0c9770 0x7fe7cc0b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:43.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.698+0000 7fe7d2c29700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc159760 0x7fe7cc16a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:43.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7d2c29700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7cc143550 con 0x7fe7cc159760 2024-01-27T00:46:43.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7d2c29700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7cc0b3280 con 0x7fe7cc0c9770 2024-01-27T00:46:43.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7d2c29700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7cc0b28b0 con 0x7fe7cc0c93a0 2024-01-27T00:46:43.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7d11c6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc159760 0x7fe7cc16a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:43.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7d11c6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc159760 0x7fe7cc16a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59764/0 (socket says 172.21.15.46:59764) 2024-01-27T00:46:43.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7d11c6700 1 -- 172.21.15.46:0/2540217918 learned_addr learned my addr 172.21.15.46:0/2540217918 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:43.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7d09c5700 1 --2- 172.21.15.46:0/2540217918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7cc0c93a0 0x7fe7cc143010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:43.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7cbfff700 1 --2- 172.21.15.46:0/2540217918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7cc0c9770 0x7fe7cc0b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:43.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7d11c6700 1 -- 172.21.15.46:0/2540217918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7cc0c93a0 msgr2=0x7fe7cc143010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:43.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7d11c6700 1 --2- 172.21.15.46:0/2540217918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7cc0c93a0 0x7fe7cc143010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7d11c6700 1 -- 172.21.15.46:0/2540217918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7cc0c9770 msgr2=0x7fe7cc0b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:43.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7d11c6700 1 --2- 172.21.15.46:0/2540217918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7cc0c9770 0x7fe7cc0b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:43.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7d11c6700 1 -- 172.21.15.46:0/2540217918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c4009a20 con 0x7fe7cc159760 2024-01-27T00:46:43.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7d11c6700 1 --2- 172.21.15.46:0/2540217918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc159760 0x7fe7cc16a0d0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4012ad0 tx=0x7fe7c4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:43.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7c8ff9700 1 -- 172.21.15.46:0/2540217918 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c40263f0 con 0x7fe7cc159760 2024-01-27T00:46:43.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7c8ff9700 1 -- 172.21.15.46:0/2540217918 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c4026550 con 0x7fe7cc159760 2024-01-27T00:46:43.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7d2c29700 1 -- 172.21.15.46:0/2540217918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7cc16a610 con 0x7fe7cc159760 2024-01-27T00:46:43.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.699+0000 7fe7c8ff9700 1 -- 172.21.15.46:0/2540217918 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4026910 con 0x7fe7cc159760 2024-01-27T00:46:43.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.700+0000 7fe7d2c29700 1 -- 172.21.15.46:0/2540217918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7cc0b1f80 con 0x7fe7cc159760 2024-01-27T00:46:43.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.700+0000 7fe7d2c29700 1 -- 172.21.15.46:0/2540217918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7cc0b2a60 con 0x7fe7cc159760 2024-01-27T00:46:43.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.701+0000 7fe7c8ff9700 1 -- 172.21.15.46:0/2540217918 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe7c4026d00 con 0x7fe7cc159760 2024-01-27T00:46:43.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.701+0000 7fe7c8ff9700 1 --2- 172.21.15.46:0/2540217918 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe7b0066f30 0x7fe7b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:43.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.702+0000 7fe7c8ff9700 1 -- 172.21.15.46:0/2540217918 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fe7c4086700 con 0x7fe7cc159760 2024-01-27T00:46:43.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.702+0000 7fe7d09c5700 1 --2- 172.21.15.46:0/2540217918 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe7b0066f30 0x7fe7b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:43.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.702+0000 7fe7d09c5700 1 --2- 172.21.15.46:0/2540217918 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe7b0066f30 0x7fe7b00693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0007160 tx=0x7fe7c0007f50 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:43.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.706+0000 7fe7c8ff9700 1 -- 172.21.15.46:0/2540217918 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe7cc0b2a60 con 0x7fe7cc159760 2024-01-27T00:46:43.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:43.914+0000 7fe7d2c29700 1 -- 172.21.15.46:0/2540217918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe7cc0b2a20 con 0x7fe7cc159760 2024-01-27T00:46:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.541+0000 7fe7c8ff9700 1 -- 172.21.15.46:0/2540217918 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v173) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe7c4057250 con 0x7fe7cc159760 2024-01-27T00:46:44.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.549+0000 7fe7d2c29700 1 -- 172.21.15.46:0/2540217918 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe7b0066f30 msgr2=0x7fe7b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:44.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.549+0000 7fe7d2c29700 1 --2- 172.21.15.46:0/2540217918 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe7b0066f30 0x7fe7b00693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0007160 tx=0x7fe7c0007f50 comp rx=0 tx=0).stop 2024-01-27T00:46:44.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.549+0000 7fe7d2c29700 1 -- 172.21.15.46:0/2540217918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc159760 msgr2=0x7fe7cc16a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:44.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.549+0000 7fe7d2c29700 1 --2- 172.21.15.46:0/2540217918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc159760 0x7fe7cc16a0d0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4012ad0 tx=0x7fe7c4019000 comp rx=0 tx=0).stop 2024-01-27T00:46:44.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.549+0000 7fe7d2c29700 1 -- 172.21.15.46:0/2540217918 shutdown_connections 2024-01-27T00:46:44.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.549+0000 7fe7d2c29700 1 --2- 172.21.15.46:0/2540217918 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fe7b0066f30 0x7fe7b00693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:44.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.549+0000 7fe7d2c29700 1 --2- 172.21.15.46:0/2540217918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc159760 0x7fe7cc16a0d0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:44.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.549+0000 7fe7d2c29700 1 --2- 172.21.15.46:0/2540217918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe7cc0c93a0 0x7fe7cc143010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:44.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.549+0000 7fe7d2c29700 1 --2- 172.21.15.46:0/2540217918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7cc0c9770 0x7fe7cc0b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:44.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.549+0000 7fe7d2c29700 1 -- 172.21.15.46:0/2540217918 >> 172.21.15.46:0/2540217918 conn(0x7fe7cc0ca9a0 msgr2=0x7fe7cc06ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:44.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.550+0000 7fe7d2c29700 1 -- 172.21.15.46:0/2540217918 shutdown_connections 2024-01-27T00:46:44.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.552+0000 7fe7d2c29700 1 -- 172.21.15.46:0/2540217918 wait complete. 2024-01-27T00:46:44.561 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-01-27T00:46:44.573 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-27T00:46:44.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.711+0000 7f5494720700 1 Processor -- start 2024-01-27T00:46:44.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.711+0000 7f5494720700 1 -- start start 2024-01-27T00:46:44.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.712+0000 7f5494720700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f548c068b00 0x7f548c068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:44.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.712+0000 7f5494720700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f548c06b910 0x7f548c15aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:44.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.712+0000 7f5494720700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548c15a2b0 con 0x7f548c0ad380 2024-01-27T00:46:44.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.712+0000 7f5494720700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f548c15a620 con 0x7f548c068b00 2024-01-27T00:46:44.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.712+0000 7f5494720700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548c156b80 con 0x7f548c06b910 2024-01-27T00:46:44.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.713+0000 7f54924bc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f548c068b00 0x7f548c068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:44.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.713+0000 7f54924bc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f548c068b00 0x7f548c068ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59770/0 (socket says 172.21.15.46:59770) 2024-01-27T00:46:44.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.713+0000 7f54924bc700 1 -- 172.21.15.46:0/1995922294 learned_addr learned my addr 172.21.15.46:0/1995922294 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:44.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.713+0000 7f5491cbb700 1 --2- 172.21.15.46:0/1995922294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f548c06b910 0x7f548c15aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:44.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.713+0000 7f54924bc700 1 -- 172.21.15.46:0/1995922294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f548c06b910 msgr2=0x7f548c15aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:44.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.713+0000 7f54924bc700 1 --2- 172.21.15.46:0/1995922294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f548c06b910 0x7f548c15aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:44.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.713+0000 7f54924bc700 1 -- 172.21.15.46:0/1995922294 >> v1:172.21.15.106:6789/0 conn(0x7f548c0ad380 legacy=0x7f548c0ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:44.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.713+0000 7f54924bc700 1 -- 172.21.15.46:0/1995922294 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c06c140 con 0x7f548c068b00 2024-01-27T00:46:44.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.713+0000 7f54924bc700 1 --2- 172.21.15.46:0/1995922294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f548c068b00 0x7f548c068ed0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f54880099c0 tx=0x7f5488009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ca55479e21f1ed server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:44.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.713+0000 7f54914ba700 1 -- 172.21.15.46:0/1995922294 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5488005c10 con 0x7f548c068b00 2024-01-27T00:46:44.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.713+0000 7f54914ba700 1 -- 172.21.15.46:0/1995922294 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5488005d70 con 0x7f548c068b00 2024-01-27T00:46:44.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.713+0000 7f54914ba700 1 -- 172.21.15.46:0/1995922294 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548800b860 con 0x7f548c068b00 2024-01-27T00:46:44.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.714+0000 7f5494720700 1 -- 172.21.15.46:0/1995922294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f548c068b00 msgr2=0x7f548c068ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:44.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.714+0000 7f5494720700 1 --2- 172.21.15.46:0/1995922294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f548c068b00 0x7f548c068ed0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f54880099c0 tx=0x7f5488009220 comp rx=0 tx=0).stop 2024-01-27T00:46:44.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.714+0000 7f5494720700 1 -- 172.21.15.46:0/1995922294 shutdown_connections 2024-01-27T00:46:44.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.714+0000 7f5494720700 1 --2- 172.21.15.46:0/1995922294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f548c068b00 0x7f548c068ed0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:44.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.714+0000 7f5494720700 1 --2- 172.21.15.46:0/1995922294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f548c06b910 0x7f548c15aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:44.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.714+0000 7f5494720700 1 -- 172.21.15.46:0/1995922294 >> 172.21.15.46:0/1995922294 conn(0x7f548c0ca100 msgr2=0x7f548c06e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:44.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.714+0000 7f5494720700 1 -- 172.21.15.46:0/1995922294 shutdown_connections 2024-01-27T00:46:44.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.714+0000 7f5494720700 1 -- 172.21.15.46:0/1995922294 wait complete. 2024-01-27T00:46:44.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.715+0000 7f5494720700 1 Processor -- start 2024-01-27T00:46:44.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.715+0000 7f5494720700 1 -- start start 2024-01-27T00:46:44.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.716+0000 7f5494720700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f548c068b00 0x7f548c158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:44.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.716+0000 7f5494720700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f548c06b910 0x7f548c169be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:44.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.716+0000 7f5494720700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f548c0ad380 0x7f548c0c2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:44.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.716+0000 7f5494720700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c16a5a0 con 0x7f548c06b910 2024-01-27T00:46:44.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.716+0000 7f5494720700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c16a880 con 0x7f548c0ad380 2024-01-27T00:46:44.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.716+0000 7f5494720700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548c16ab60 con 0x7f548c068b00 2024-01-27T00:46:44.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.716+0000 7f5491cbb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f548c06b910 0x7f548c169be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:44.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.716+0000 7f5491cbb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f548c06b910 0x7f548c169be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59784/0 (socket says 172.21.15.46:59784) 2024-01-27T00:46:44.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.716+0000 7f5491cbb700 1 -- 172.21.15.46:0/3270422597 learned_addr learned my addr 172.21.15.46:0/3270422597 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:44.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.716+0000 7f54924bc700 1 --2- 172.21.15.46:0/3270422597 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f548c068b00 0x7f548c158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:44.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.716+0000 7f5492cbd700 1 --2- 172.21.15.46:0/3270422597 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f548c0ad380 0x7f548c0c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:44.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.717+0000 7f5491cbb700 1 -- 172.21.15.46:0/3270422597 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f548c068b00 msgr2=0x7f548c158f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:44.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.717+0000 7f5491cbb700 1 --2- 172.21.15.46:0/3270422597 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f548c068b00 0x7f548c158f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:44.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.717+0000 7f5491cbb700 1 -- 172.21.15.46:0/3270422597 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f548c0ad380 msgr2=0x7f548c0c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:44.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.717+0000 7f5491cbb700 1 --2- 172.21.15.46:0/3270422597 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f548c0ad380 0x7f548c0c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:44.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.717+0000 7f5491cbb700 1 -- 172.21.15.46:0/3270422597 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5488009480 con 0x7f548c06b910 2024-01-27T00:46:44.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.717+0000 7f5491cbb700 1 --2- 172.21.15.46:0/3270422597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f548c06b910 0x7f548c169be0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f548000d890 tx=0x7f5480008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:44.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.717+0000 7f547e7fc700 1 -- 172.21.15.46:0/3270422597 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54800125f0 con 0x7f548c06b910 2024-01-27T00:46:44.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.717+0000 7f547e7fc700 1 -- 172.21.15.46:0/3270422597 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5480012770 con 0x7f548c06b910 2024-01-27T00:46:44.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.717+0000 7f547e7fc700 1 -- 172.21.15.46:0/3270422597 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5480012b30 con 0x7f548c06b910 2024-01-27T00:46:44.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.717+0000 7f5494720700 1 -- 172.21.15.46:0/3270422597 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548c156e40 con 0x7f548c06b910 2024-01-27T00:46:44.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.717+0000 7f5494720700 1 -- 172.21.15.46:0/3270422597 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548c16a120 con 0x7f548c06b910 2024-01-27T00:46:44.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.718+0000 7f5494720700 1 -- 172.21.15.46:0/3270422597 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f548c16a340 con 0x7f548c06b910 2024-01-27T00:46:44.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.719+0000 7f547e7fc700 1 -- 172.21.15.46:0/3270422597 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f548001e460 con 0x7f548c06b910 2024-01-27T00:46:44.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.719+0000 7f547e7fc700 1 --2- 172.21.15.46:0/3270422597 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5470066fb0 0x7f5470069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:44.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.719+0000 7f547e7fc700 1 -- 172.21.15.46:0/3270422597 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6361+0+0 (secure 0 0 0) 0x7f5480085a10 con 0x7f548c06b910 2024-01-27T00:46:44.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.720+0000 7f54924bc700 1 --2- 172.21.15.46:0/3270422597 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5470066fb0 0x7f5470069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:44.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.720+0000 7f54924bc700 1 --2- 172.21.15.46:0/3270422597 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5470066fb0 0x7f5470069460 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5488009910 tx=0x7f5488016000 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:44.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.724+0000 7f547e7fc700 1 -- 172.21.15.46:0/3270422597 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f54800529d0 con 0x7f548c06b910 2024-01-27T00:46:44.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:44.977+0000 7f5494720700 1 -- 172.21.15.46:0/3270422597 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f548c16ad80 con 0x7f548c06b910 2024-01-27T00:46:45.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-01-27T00:46:45.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.551+0000 7f547e7fc700 1 -- 172.21.15.46:0/3270422597 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 21 and data pool 22 v84) v1 ==== 174+0+0 (secure 0 0 0) 0x7f5480055ff0 con 0x7f548c06b910 2024-01-27T00:46:45.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.561+0000 7f5494720700 1 -- 172.21.15.46:0/3270422597 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5470066fb0 msgr2=0x7f5470069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:45.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.561+0000 7f5494720700 1 --2- 172.21.15.46:0/3270422597 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5470066fb0 0x7f5470069460 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5488009910 tx=0x7f5488016000 comp rx=0 tx=0).stop 2024-01-27T00:46:45.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.561+0000 7f5494720700 1 -- 172.21.15.46:0/3270422597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f548c06b910 msgr2=0x7f548c169be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:45.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.561+0000 7f5494720700 1 --2- 172.21.15.46:0/3270422597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f548c06b910 0x7f548c169be0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f548000d890 tx=0x7f5480008040 comp rx=0 tx=0).stop 2024-01-27T00:46:45.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.561+0000 7f5494720700 1 -- 172.21.15.46:0/3270422597 shutdown_connections 2024-01-27T00:46:45.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.561+0000 7f5494720700 1 --2- 172.21.15.46:0/3270422597 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5470066fb0 0x7f5470069460 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:45.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.561+0000 7f5494720700 1 --2- 172.21.15.46:0/3270422597 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f548c0ad380 0x7f548c0c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:45.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.561+0000 7f5494720700 1 --2- 172.21.15.46:0/3270422597 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f548c068b00 0x7f548c158f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:45.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.561+0000 7f5494720700 1 --2- 172.21.15.46:0/3270422597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f548c06b910 0x7f548c169be0 secure :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f548000d890 tx=0x7f5480008040 comp rx=0 tx=0).stop 2024-01-27T00:46:45.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.561+0000 7f5494720700 1 -- 172.21.15.46:0/3270422597 >> 172.21.15.46:0/3270422597 conn(0x7f548c0ca100 msgr2=0x7f548c0ad800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:45.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.562+0000 7f5494720700 1 -- 172.21.15.46:0/3270422597 shutdown_connections 2024-01-27T00:46:45.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.563+0000 7f5494720700 1 -- 172.21.15.46:0/3270422597 wait complete. 2024-01-27T00:46:45.574 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 21 and data pool 22 2024-01-27T00:46:45.589 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T00:46:45.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.723+0000 7fc820c1c700 1 Processor -- start 2024-01-27T00:46:45.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.723+0000 7fc820c1c700 1 -- start start 2024-01-27T00:46:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.724+0000 7fc820c1c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc81c157340 0x7fc81c15bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.724+0000 7fc820c1c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc81c0ae520 0x7fc81c0b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.724+0000 7fc820c1c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81c158ae0 con 0x7fc81c0b3fb0 2024-01-27T00:46:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.724+0000 7fc820c1c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81c0b4430 con 0x7fc81c0ae520 2024-01-27T00:46:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.724+0000 7fc820c1c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c15b650 con 0x7fc81c157340 2024-01-27T00:46:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.724+0000 7fc819d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc81c0ae520 0x7fc81c0b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.724+0000 7fc81ad9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc81c0b3fb0 0x7fc81c157710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41096/0 (socket says 172.21.15.46:41096) 2024-01-27T00:46:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.724+0000 7fc81ad9d700 1 -- 172.21.15.46:0/1738799360 learned_addr learned my addr 172.21.15.46:0/1738799360 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.724+0000 7fc81a59c700 1 --2- 172.21.15.46:0/1738799360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc81c157340 0x7fc81c15bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.724+0000 7fc819d9b700 1 -- 172.21.15.46:0/1738799360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc81c157340 msgr2=0x7fc81c15bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.724+0000 7fc819d9b700 1 --2- 172.21.15.46:0/1738799360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc81c157340 0x7fc81c15bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.724+0000 7fc819d9b700 1 -- 172.21.15.46:0/1738799360 >> v1:172.21.15.46:6789/0 conn(0x7fc81c0b3fb0 legacy=0x7fc81c157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.724+0000 7fc819d9b700 1 -- 172.21.15.46:0/1738799360 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c15c020 con 0x7fc81c0ae520 2024-01-27T00:46:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.724+0000 7fc81959a700 1 -- 172.21.15.46:0/1738799360 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846541770 0 0) 0x7fc81c158ae0 con 0x7fc81c0b3fb0 2024-01-27T00:46:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.725+0000 7fc819d9b700 1 --2- 172.21.15.46:0/1738799360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c0ae520 0x7fc81c0b3a70 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fc8100098e0 tx=0x7fc8100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604f0a54c923e4f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.725+0000 7fc81959a700 1 -- 172.21.15.46:0/1738799360 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc810009e80 con 0x7fc81c0ae520 2024-01-27T00:46:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.725+0000 7fc81959a700 1 -- 172.21.15.46:0/1738799360 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81000b860 con 0x7fc81c0ae520 2024-01-27T00:46:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.725+0000 7fc81959a700 1 -- 172.21.15.46:0/1738799360 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81000bbe0 con 0x7fc81c0ae520 2024-01-27T00:46:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.725+0000 7fc820c1c700 1 -- 172.21.15.46:0/1738799360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c0ae520 msgr2=0x7fc81c0b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.725+0000 7fc820c1c700 1 --2- 172.21.15.46:0/1738799360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c0ae520 0x7fc81c0b3a70 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fc8100098e0 tx=0x7fc8100091a0 comp rx=0 tx=0).stop 2024-01-27T00:46:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.725+0000 7fc820c1c700 1 -- 172.21.15.46:0/1738799360 shutdown_connections 2024-01-27T00:46:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.725+0000 7fc820c1c700 1 --2- 172.21.15.46:0/1738799360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc81c157340 0x7fc81c15bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.725+0000 7fc820c1c700 1 --2- 172.21.15.46:0/1738799360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c0ae520 0x7fc81c0b3a70 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.725+0000 7fc820c1c700 1 -- 172.21.15.46:0/1738799360 >> 172.21.15.46:0/1738799360 conn(0x7fc81c0ce530 msgr2=0x7fc81c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.725+0000 7fc820c1c700 1 -- 172.21.15.46:0/1738799360 shutdown_connections 2024-01-27T00:46:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.726+0000 7fc820c1c700 1 -- 172.21.15.46:0/1738799360 wait complete. 2024-01-27T00:46:45.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.727+0000 7fc820c1c700 1 Processor -- start 2024-01-27T00:46:45.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.727+0000 7fc820c1c700 1 -- start start 2024-01-27T00:46:45.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.727+0000 7fc820c1c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc81c0ae520 0x7fc81c156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:45.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.727+0000 7fc820c1c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc81c0b3fb0 0x7fc81c0c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:45.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.727+0000 7fc820c1c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c157340 0x7fc81c0c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:45.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.727+0000 7fc820c1c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c06a590 con 0x7fc81c157340 2024-01-27T00:46:45.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.727+0000 7fc820c1c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c1699f0 con 0x7fc81c0b3fb0 2024-01-27T00:46:45.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.727+0000 7fc820c1c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c069a10 con 0x7fc81c0ae520 2024-01-27T00:46:45.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.728+0000 7fc81ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c157340 0x7fc81c0c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:45.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.728+0000 7fc81ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c157340 0x7fc81c0c6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59804/0 (socket says 172.21.15.46:59804) 2024-01-27T00:46:45.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.728+0000 7fc81ad9d700 1 -- 172.21.15.46:0/191693731 learned_addr learned my addr 172.21.15.46:0/191693731 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:45.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.728+0000 7fc81a59c700 1 --2- 172.21.15.46:0/191693731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc81c0ae520 0x7fc81c156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:45.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.728+0000 7fc819d9b700 1 --2- 172.21.15.46:0/191693731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc81c0b3fb0 0x7fc81c0c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:45.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.728+0000 7fc81ad9d700 1 -- 172.21.15.46:0/191693731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc81c0ae520 msgr2=0x7fc81c156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:45.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.728+0000 7fc81ad9d700 1 --2- 172.21.15.46:0/191693731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc81c0ae520 0x7fc81c156890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:45.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.728+0000 7fc81ad9d700 1 -- 172.21.15.46:0/191693731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc81c0b3fb0 msgr2=0x7fc81c0c5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:45.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.728+0000 7fc81ad9d700 1 --2- 172.21.15.46:0/191693731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc81c0b3fb0 0x7fc81c0c5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:45.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.728+0000 7fc81ad9d700 1 -- 172.21.15.46:0/191693731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc810009420 con 0x7fc81c157340 2024-01-27T00:46:45.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.728+0000 7fc819d9b700 1 --2- 172.21.15.46:0/191693731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc81c0b3fb0 0x7fc81c0c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:46:45.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.728+0000 7fc81ad9d700 1 --2- 172.21.15.46:0/191693731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c157340 0x7fc81c0c6e00 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fc81400cc90 tx=0x7fc814001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:45.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.728+0000 7fc80a7fc700 1 -- 172.21.15.46:0/191693731 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81400a960 con 0x7fc81c157340 2024-01-27T00:46:45.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.728+0000 7fc80a7fc700 1 -- 172.21.15.46:0/191693731 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81400aac0 con 0x7fc81c157340 2024-01-27T00:46:45.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.728+0000 7fc820c1c700 1 -- 172.21.15.46:0/191693731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81c069cf0 con 0x7fc81c157340 2024-01-27T00:46:45.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.728+0000 7fc80a7fc700 1 -- 172.21.15.46:0/191693731 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81400ae80 con 0x7fc81c157340 2024-01-27T00:46:45.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.728+0000 7fc820c1c700 1 -- 172.21.15.46:0/191693731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc81c06a2b0 con 0x7fc81c157340 2024-01-27T00:46:45.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.729+0000 7fc820c1c700 1 -- 172.21.15.46:0/191693731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc81c169850 con 0x7fc81c157340 2024-01-27T00:46:45.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.730+0000 7fc80a7fc700 1 -- 172.21.15.46:0/191693731 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc814007300 con 0x7fc81c157340 2024-01-27T00:46:45.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.730+0000 7fc80a7fc700 1 --2- 172.21.15.46:0/191693731 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc7fc066f30 0x7fc7fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:45.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.730+0000 7fc80a7fc700 1 -- 172.21.15.46:0/191693731 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fc814084c00 con 0x7fc81c157340 2024-01-27T00:46:45.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.730+0000 7fc81a59c700 1 --2- 172.21.15.46:0/191693731 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc7fc066f30 0x7fc7fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:45.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.731+0000 7fc81a59c700 1 --2- 172.21.15.46:0/191693731 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc7fc066f30 0x7fc7fc0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc80c006fd0 tx=0x7fc80c008040 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:45.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.735+0000 7fc80a7fc700 1 -- 172.21.15.46:0/191693731 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc81c169850 con 0x7fc81c157340 2024-01-27T00:46:45.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.940+0000 7fc820c1c700 1 -- 172.21.15.46:0/191693731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc81c169850 con 0x7fc81c157340 2024-01-27T00:46:45.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.941+0000 7fc80a7fc700 1 -- 172.21.15.46:0/191693731 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+16247 (secure 0 0 0) 0x7fc81c169850 con 0x7fc81c157340 2024-01-27T00:46:45.941 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:45.941 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":174,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T00:46:44.978203+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-01-27T00:46:43.494684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-01-27T00:46:43.949785+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T00:46:45.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.943+0000 7fc820c1c700 1 -- 172.21.15.46:0/191693731 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc7fc066f30 msgr2=0x7fc7fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:45.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.943+0000 7fc820c1c700 1 --2- 172.21.15.46:0/191693731 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc7fc066f30 0x7fc7fc0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc80c006fd0 tx=0x7fc80c008040 comp rx=0 tx=0).stop 2024-01-27T00:46:45.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.944+0000 7fc820c1c700 1 -- 172.21.15.46:0/191693731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c157340 msgr2=0x7fc81c0c6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:45.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.944+0000 7fc820c1c700 1 --2- 172.21.15.46:0/191693731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c157340 0x7fc81c0c6e00 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fc81400cc90 tx=0x7fc814001a80 comp rx=0 tx=0).stop 2024-01-27T00:46:45.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.944+0000 7fc820c1c700 1 -- 172.21.15.46:0/191693731 shutdown_connections 2024-01-27T00:46:45.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.944+0000 7fc820c1c700 1 --2- 172.21.15.46:0/191693731 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc7fc066f30 0x7fc7fc0693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:45.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.944+0000 7fc820c1c700 1 --2- 172.21.15.46:0/191693731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c157340 0x7fc81c0c6e00 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:45.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.944+0000 7fc820c1c700 1 --2- 172.21.15.46:0/191693731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc81c0ae520 0x7fc81c156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:45.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.944+0000 7fc820c1c700 1 --2- 172.21.15.46:0/191693731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc81c0b3fb0 0x7fc81c0c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:45.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.944+0000 7fc820c1c700 1 -- 172.21.15.46:0/191693731 >> 172.21.15.46:0/191693731 conn(0x7fc81c0ce530 msgr2=0x7fc81c15a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:45.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.944+0000 7fc820c1c700 1 -- 172.21.15.46:0/191693731 shutdown_connections 2024-01-27T00:46:45.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:45.944+0000 7fc820c1c700 1 -- 172.21.15.46:0/191693731 wait complete. 2024-01-27T00:46:45.958 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T00:46:46.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.090+0000 7fbfaecc2700 1 Processor -- start 2024-01-27T00:46:46.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.090+0000 7fbfaecc2700 1 -- start start 2024-01-27T00:46:46.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.091+0000 7fbfaecc2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbfa80ae520 0x7fbfa80b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:46.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.091+0000 7fbfaecc2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa8158ae0 con 0x7fbfa80b3fb0 2024-01-27T00:46:46.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.091+0000 7fbfaecc2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa815b650 con 0x7fbfa8157340 2024-01-27T00:46:46.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.091+0000 7fbfaecc2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa80b43c0 con 0x7fbfa80ae520 2024-01-27T00:46:46.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.091+0000 7fbfa7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbfa80ae520 0x7fbfa80b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:46.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.091+0000 7fbfad25f700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbfa80b3fb0 0x7fbfa8157710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41108/0 (socket says 172.21.15.46:41108) 2024-01-27T00:46:46.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.091+0000 7fbfad25f700 1 -- 172.21.15.46:0/3410762779 learned_addr learned my addr 172.21.15.46:0/3410762779 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:46.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.091+0000 7fbfa7fff700 1 -- 172.21.15.46:0/3410762779 >> v1:172.21.15.106:6789/0 conn(0x7fbfa8157340 legacy=0x7fbfa815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:46.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.091+0000 7fbfa7fff700 1 -- 172.21.15.46:0/3410762779 >> v1:172.21.15.46:6789/0 conn(0x7fbfa80b3fb0 legacy=0x7fbfa8157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:46.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.091+0000 7fbfa7fff700 1 -- 172.21.15.46:0/3410762779 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa8158520 con 0x7fbfa80ae520 2024-01-27T00:46:46.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.091+0000 7fbfa77fe700 1 -- 172.21.15.46:0/3410762779 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821333777 0 0) 0x7fbfa8158ae0 con 0x7fbfa80b3fb0 2024-01-27T00:46:46.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.092+0000 7fbfa7fff700 1 --2- 172.21.15.46:0/3410762779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa80ae520 0x7fbfa80b3a70 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fbf940098e0 tx=0x7fbf940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e9e6211c61bc6b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:46.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.092+0000 7fbfa77fe700 1 -- 172.21.15.46:0/3410762779 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94009e80 con 0x7fbfa80ae520 2024-01-27T00:46:46.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.092+0000 7fbfa77fe700 1 -- 172.21.15.46:0/3410762779 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9400b860 con 0x7fbfa80ae520 2024-01-27T00:46:46.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.092+0000 7fbfa77fe700 1 -- 172.21.15.46:0/3410762779 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9400bbe0 con 0x7fbfa80ae520 2024-01-27T00:46:46.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.092+0000 7fbfaecc2700 1 -- 172.21.15.46:0/3410762779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa80ae520 msgr2=0x7fbfa80b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:46.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.092+0000 7fbfaecc2700 1 --2- 172.21.15.46:0/3410762779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa80ae520 0x7fbfa80b3a70 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fbf940098e0 tx=0x7fbf940091a0 comp rx=0 tx=0).stop 2024-01-27T00:46:46.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.092+0000 7fbfaecc2700 1 -- 172.21.15.46:0/3410762779 shutdown_connections 2024-01-27T00:46:46.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.092+0000 7fbfaecc2700 1 --2- 172.21.15.46:0/3410762779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa80ae520 0x7fbfa80b3a70 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:46.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.092+0000 7fbfaecc2700 1 -- 172.21.15.46:0/3410762779 >> 172.21.15.46:0/3410762779 conn(0x7fbfa80ce530 msgr2=0x7fbfa80ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:46.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.092+0000 7fbfaecc2700 1 -- 172.21.15.46:0/3410762779 shutdown_connections 2024-01-27T00:46:46.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.092+0000 7fbfaecc2700 1 -- 172.21.15.46:0/3410762779 wait complete. 2024-01-27T00:46:46.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.094+0000 7fbfaecc2700 1 Processor -- start 2024-01-27T00:46:46.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.094+0000 7fbfaecc2700 1 -- start start 2024-01-27T00:46:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfaecc2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbfa80ae520 0x7fbfa80b1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfaecc2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa80b3fb0 0x7fbfa80b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfaecc2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbfa8157340 0x7fbfa8155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfaecc2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa80b2de0 con 0x7fbfa80b3fb0 2024-01-27T00:46:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfaecc2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa80b30c0 con 0x7fbfa8157340 2024-01-27T00:46:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfaecc2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa80b12e0 con 0x7fbfa80ae520 2024-01-27T00:46:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfa7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa80b3fb0 0x7fbfa80b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfa7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa80b3fb0 0x7fbfa80b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59820/0 (socket says 172.21.15.46:59820) 2024-01-27T00:46:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfa7fff700 1 -- 172.21.15.46:0/2722761569 learned_addr learned my addr 172.21.15.46:0/2722761569 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfad25f700 1 --2- 172.21.15.46:0/2722761569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbfa8157340 0x7fbfa8155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfaca5e700 1 --2- 172.21.15.46:0/2722761569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbfa80ae520 0x7fbfa80b1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfa7fff700 1 -- 172.21.15.46:0/2722761569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbfa80ae520 msgr2=0x7fbfa80b1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfa7fff700 1 --2- 172.21.15.46:0/2722761569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbfa80ae520 0x7fbfa80b1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfa7fff700 1 -- 172.21.15.46:0/2722761569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbfa8157340 msgr2=0x7fbfa8155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfa7fff700 1 --2- 172.21.15.46:0/2722761569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbfa8157340 0x7fbfa8155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfa7fff700 1 -- 172.21.15.46:0/2722761569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf94009420 con 0x7fbfa80b3fb0 2024-01-27T00:46:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfaca5e700 1 --2- 172.21.15.46:0/2722761569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbfa80ae520 0x7fbfa80b1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:46:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfa7fff700 1 --2- 172.21.15.46:0/2722761569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa80b3fb0 0x7fbfa80b2590 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fbfa806f390 tx=0x7fbf94017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.095+0000 7fbfa4ff9700 1 -- 172.21.15.46:0/2722761569 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9401f070 con 0x7fbfa80b3fb0 2024-01-27T00:46:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.096+0000 7fbfa4ff9700 1 -- 172.21.15.46:0/2722761569 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf94000c90 con 0x7fbfa80b3fb0 2024-01-27T00:46:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.096+0000 7fbfaecc2700 1 -- 172.21.15.46:0/2722761569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa80b15c0 con 0x7fbfa80b3fb0 2024-01-27T00:46:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.096+0000 7fbfa4ff9700 1 -- 172.21.15.46:0/2722761569 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94025420 con 0x7fbfa80b3fb0 2024-01-27T00:46:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.096+0000 7fbfaecc2700 1 -- 172.21.15.46:0/2722761569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa80b2300 con 0x7fbfa80b3fb0 2024-01-27T00:46:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.097+0000 7fbfaecc2700 1 -- 172.21.15.46:0/2722761569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa804c710 con 0x7fbfa80b3fb0 2024-01-27T00:46:46.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.097+0000 7fbfa4ff9700 1 -- 172.21.15.46:0/2722761569 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbf94023070 con 0x7fbfa80b3fb0 2024-01-27T00:46:46.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.098+0000 7fbfa4ff9700 1 --2- 172.21.15.46:0/2722761569 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf8c066f30 0x7fbf8c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:46.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.098+0000 7fbfa4ff9700 1 -- 172.21.15.46:0/2722761569 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fbf94085bf0 con 0x7fbfa80b3fb0 2024-01-27T00:46:46.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.098+0000 7fbfaca5e700 1 --2- 172.21.15.46:0/2722761569 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf8c066f30 0x7fbf8c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:46.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.098+0000 7fbfaca5e700 1 --2- 172.21.15.46:0/2722761569 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf8c066f30 0x7fbf8c0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c007fd0 tx=0x7fbf9c009040 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:46.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.102+0000 7fbfa4ff9700 1 -- 172.21.15.46:0/2722761569 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbf94052b70 con 0x7fbfa80b3fb0 2024-01-27T00:46:46.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.307+0000 7fbfaecc2700 1 -- 172.21.15.46:0/2722761569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfa8158800 con 0x7fbfa80b3fb0 2024-01-27T00:46:46.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.308+0000 7fbfa4ff9700 1 -- 172.21.15.46:0/2722761569 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+16247 (secure 0 0 0) 0x7fbf9402a090 con 0x7fbfa80b3fb0 2024-01-27T00:46:46.307 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:46.307 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":174,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T00:46:44.978203+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-01-27T00:46:43.494684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-01-27T00:46:43.949785+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T00:46:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.310+0000 7fbfaecc2700 1 -- 172.21.15.46:0/2722761569 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf8c066f30 msgr2=0x7fbf8c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.310+0000 7fbfaecc2700 1 --2- 172.21.15.46:0/2722761569 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf8c066f30 0x7fbf8c0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c007fd0 tx=0x7fbf9c009040 comp rx=0 tx=0).stop 2024-01-27T00:46:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.310+0000 7fbfaecc2700 1 -- 172.21.15.46:0/2722761569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa80b3fb0 msgr2=0x7fbfa80b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.310+0000 7fbfaecc2700 1 --2- 172.21.15.46:0/2722761569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa80b3fb0 0x7fbfa80b2590 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fbfa806f390 tx=0x7fbf94017000 comp rx=0 tx=0).stop 2024-01-27T00:46:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.310+0000 7fbfaecc2700 1 -- 172.21.15.46:0/2722761569 shutdown_connections 2024-01-27T00:46:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.310+0000 7fbfaecc2700 1 --2- 172.21.15.46:0/2722761569 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbf8c066f30 0x7fbf8c0693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.310+0000 7fbfaecc2700 1 --2- 172.21.15.46:0/2722761569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbfa8157340 0x7fbfa8155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.310+0000 7fbfaecc2700 1 --2- 172.21.15.46:0/2722761569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbfa80ae520 0x7fbfa80b1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.310+0000 7fbfaecc2700 1 --2- 172.21.15.46:0/2722761569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa80b3fb0 0x7fbfa80b2590 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.310+0000 7fbfaecc2700 1 -- 172.21.15.46:0/2722761569 >> 172.21.15.46:0/2722761569 conn(0x7fbfa80ce530 msgr2=0x7fbfa80ade00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:46.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.310+0000 7fbfaecc2700 1 -- 172.21.15.46:0/2722761569 shutdown_connections 2024-01-27T00:46:46.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.310+0000 7fbfaecc2700 1 -- 172.21.15.46:0/2722761569 wait complete. 2024-01-27T00:46:46.324 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-27T00:46:46.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.459+0000 7fd5c06b2700 1 Processor -- start 2024-01-27T00:46:46.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.459+0000 7fd5c06b2700 1 -- start start 2024-01-27T00:46:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.460+0000 7fd5c06b2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd5b80aef60 0x7fd5b80c8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.460+0000 7fd5c06b2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5b80c9190 0x7fd5b80c7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.460+0000 7fd5c06b2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b80c87c0 con 0x7fd5b80af330 2024-01-27T00:46:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.460+0000 7fd5c06b2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5b80c7bb0 con 0x7fd5b80aef60 2024-01-27T00:46:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.460+0000 7fd5c06b2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b80c7e90 con 0x7fd5b80c9190 2024-01-27T00:46:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.460+0000 7fd5be44e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd5b80aef60 0x7fd5b80c8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.460+0000 7fd5bec4f700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd5b80af330 0x7fd5b8068940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41110/0 (socket says 172.21.15.46:41110) 2024-01-27T00:46:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.461+0000 7fd5bec4f700 1 -- 172.21.15.46:0/3202899582 learned_addr learned my addr 172.21.15.46:0/3202899582 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.461+0000 7fd5bdc4d700 1 --2- 172.21.15.46:0/3202899582 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5b80c9190 0x7fd5b80c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.461+0000 7fd5be44e700 1 -- 172.21.15.46:0/3202899582 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5b80c9190 msgr2=0x7fd5b80c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.461+0000 7fd5be44e700 1 --2- 172.21.15.46:0/3202899582 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5b80c9190 0x7fd5b80c7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.461+0000 7fd5be44e700 1 -- 172.21.15.46:0/3202899582 >> v1:172.21.15.46:6789/0 conn(0x7fd5b80af330 legacy=0x7fd5b8068940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.461+0000 7fd5be44e700 1 -- 172.21.15.46:0/3202899582 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b80c8170 con 0x7fd5b80aef60 2024-01-27T00:46:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.461+0000 7fd5bd44c700 1 -- 172.21.15.46:0/3202899582 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4142213840 0 0) 0x7fd5b80c87c0 con 0x7fd5b80af330 2024-01-27T00:46:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.461+0000 7fd5be44e700 1 --2- 172.21.15.46:0/3202899582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5b80aef60 0x7fd5b80c8c50 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40098d0 tx=0x7fd5b4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b835b468be5277 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.461+0000 7fd5bd44c700 1 -- 172.21.15.46:0/3202899582 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4009e70 con 0x7fd5b80aef60 2024-01-27T00:46:46.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.461+0000 7fd5bd44c700 1 -- 172.21.15.46:0/3202899582 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b4005c10 con 0x7fd5b80aef60 2024-01-27T00:46:46.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.461+0000 7fd5bd44c700 1 -- 172.21.15.46:0/3202899582 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b400b650 con 0x7fd5b80aef60 2024-01-27T00:46:46.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.462+0000 7fd5c06b2700 1 -- 172.21.15.46:0/3202899582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5b80aef60 msgr2=0x7fd5b80c8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:46.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.462+0000 7fd5c06b2700 1 --2- 172.21.15.46:0/3202899582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5b80aef60 0x7fd5b80c8c50 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40098d0 tx=0x7fd5b4009120 comp rx=0 tx=0).stop 2024-01-27T00:46:46.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.462+0000 7fd5c06b2700 1 -- 172.21.15.46:0/3202899582 shutdown_connections 2024-01-27T00:46:46.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.462+0000 7fd5c06b2700 1 --2- 172.21.15.46:0/3202899582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5b80aef60 0x7fd5b80c8c50 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:46.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.462+0000 7fd5c06b2700 1 --2- 172.21.15.46:0/3202899582 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5b80c9190 0x7fd5b80c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:46.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.462+0000 7fd5c06b2700 1 -- 172.21.15.46:0/3202899582 >> 172.21.15.46:0/3202899582 conn(0x7fd5b80ca100 msgr2=0x7fd5b806e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:46.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.462+0000 7fd5c06b2700 1 -- 172.21.15.46:0/3202899582 shutdown_connections 2024-01-27T00:46:46.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.462+0000 7fd5c06b2700 1 -- 172.21.15.46:0/3202899582 wait complete. 2024-01-27T00:46:46.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.463+0000 7fd5c06b2700 1 Processor -- start 2024-01-27T00:46:46.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.463+0000 7fd5c06b2700 1 -- start start 2024-01-27T00:46:46.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.464+0000 7fd5c06b2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5b80aef60 0x7fd5b80c2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:46.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.464+0000 7fd5c06b2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5b80af330 0x7fd5b816a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:46.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.464+0000 7fd5c06b2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5b80c9190 0x7fd5b806ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:46.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.464+0000 7fd5c06b2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b806a7d0 con 0x7fd5b80c9190 2024-01-27T00:46:46.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.464+0000 7fd5c06b2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b816abc0 con 0x7fd5b80aef60 2024-01-27T00:46:46.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.464+0000 7fd5c06b2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b816aea0 con 0x7fd5b80af330 2024-01-27T00:46:46.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.464+0000 7fd5bec4f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5b80c9190 0x7fd5b806ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:46.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.464+0000 7fd5bec4f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5b80c9190 0x7fd5b806ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59834/0 (socket says 172.21.15.46:59834) 2024-01-27T00:46:46.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.464+0000 7fd5bec4f700 1 -- 172.21.15.46:0/700530631 learned_addr learned my addr 172.21.15.46:0/700530631 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:46.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.464+0000 7fd5bdc4d700 1 --2- 172.21.15.46:0/700530631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5b80af330 0x7fd5b816a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:46.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.464+0000 7fd5be44e700 1 --2- 172.21.15.46:0/700530631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5b80aef60 0x7fd5b80c2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:46.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.464+0000 7fd5bec4f700 1 -- 172.21.15.46:0/700530631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5b80af330 msgr2=0x7fd5b816a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:46.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.464+0000 7fd5bec4f700 1 --2- 172.21.15.46:0/700530631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5b80af330 0x7fd5b816a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:46.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.464+0000 7fd5bec4f700 1 -- 172.21.15.46:0/700530631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5b80aef60 msgr2=0x7fd5b80c2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:46.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.464+0000 7fd5bec4f700 1 --2- 172.21.15.46:0/700530631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5b80aef60 0x7fd5b80c2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:46.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.464+0000 7fd5bec4f700 1 -- 172.21.15.46:0/700530631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b40093c0 con 0x7fd5b80c9190 2024-01-27T00:46:46.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.464+0000 7fd5be44e700 1 --2- 172.21.15.46:0/700530631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5b80aef60 0x7fd5b80c2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:46:46.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.465+0000 7fd5bec4f700 1 --2- 172.21.15.46:0/700530631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5b80c9190 0x7fd5b806ac60 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fd5b000cc10 tx=0x7fd5b00019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:46.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.465+0000 7fd5ae7fc700 1 -- 172.21.15.46:0/700530631 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b000a990 con 0x7fd5b80c9190 2024-01-27T00:46:46.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.465+0000 7fd5ae7fc700 1 -- 172.21.15.46:0/700530631 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b000ab10 con 0x7fd5b80c9190 2024-01-27T00:46:46.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.465+0000 7fd5c06b2700 1 -- 172.21.15.46:0/700530631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b816b180 con 0x7fd5b80c9190 2024-01-27T00:46:46.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.465+0000 7fd5ae7fc700 1 -- 172.21.15.46:0/700530631 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b00070f0 con 0x7fd5b80c9190 2024-01-27T00:46:46.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.465+0000 7fd5c06b2700 1 -- 172.21.15.46:0/700530631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b80c2db0 con 0x7fd5b80c9190 2024-01-27T00:46:46.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.466+0000 7fd5c06b2700 1 -- 172.21.15.46:0/700530631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b806a940 con 0x7fd5b80c9190 2024-01-27T00:46:46.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.466+0000 7fd5ae7fc700 1 -- 172.21.15.46:0/700530631 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd5b00074e0 con 0x7fd5b80c9190 2024-01-27T00:46:46.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.467+0000 7fd5ae7fc700 1 --2- 172.21.15.46:0/700530631 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd59c066f30 0x7fd59c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:46.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.467+0000 7fd5ae7fc700 1 -- 172.21.15.46:0/700530631 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fd5b0084c30 con 0x7fd5b80c9190 2024-01-27T00:46:46.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.467+0000 7fd5be44e700 1 --2- 172.21.15.46:0/700530631 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd59c066f30 0x7fd59c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:46.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.468+0000 7fd5be44e700 1 --2- 172.21.15.46:0/700530631 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd59c066f30 0x7fd59c0693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4009fd0 tx=0x7fd5b4016000 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:46.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.471+0000 7fd5ae7fc700 1 -- 172.21.15.46:0/700530631 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd5b0051c30 con 0x7fd5b80c9190 2024-01-27T00:46:46.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:46.731+0000 7fd5c06b2700 1 -- 172.21.15.46:0/700530631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd5b804c710 con 0x7fd5b80c9190 2024-01-27T00:46:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.559+0000 7fd5ae7fc700 1 -- 172.21.15.46:0/700530631 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v87) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd5b0055250 con 0x7fd5b80c9190 2024-01-27T00:46:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.562+0000 7fd5c06b2700 1 -- 172.21.15.46:0/700530631 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd59c066f30 msgr2=0x7fd59c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.562+0000 7fd5c06b2700 1 --2- 172.21.15.46:0/700530631 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd59c066f30 0x7fd59c0693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4009fd0 tx=0x7fd5b4016000 comp rx=0 tx=0).stop 2024-01-27T00:46:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.562+0000 7fd5c06b2700 1 -- 172.21.15.46:0/700530631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5b80c9190 msgr2=0x7fd5b806ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.562+0000 7fd5c06b2700 1 --2- 172.21.15.46:0/700530631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5b80c9190 0x7fd5b806ac60 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fd5b000cc10 tx=0x7fd5b00019c0 comp rx=0 tx=0).stop 2024-01-27T00:46:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.562+0000 7fd5c06b2700 1 -- 172.21.15.46:0/700530631 shutdown_connections 2024-01-27T00:46:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.562+0000 7fd5c06b2700 1 --2- 172.21.15.46:0/700530631 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fd59c066f30 0x7fd59c0693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.562+0000 7fd5c06b2700 1 --2- 172.21.15.46:0/700530631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5b80c9190 0x7fd5b806ac60 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.562+0000 7fd5c06b2700 1 --2- 172.21.15.46:0/700530631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5b80aef60 0x7fd5b80c2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.562+0000 7fd5c06b2700 1 --2- 172.21.15.46:0/700530631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5b80af330 0x7fd5b816a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.562+0000 7fd5c06b2700 1 -- 172.21.15.46:0/700530631 >> 172.21.15.46:0/700530631 conn(0x7fd5b80ca100 msgr2=0x7fd5b80b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.562+0000 7fd5c06b2700 1 -- 172.21.15.46:0/700530631 shutdown_connections 2024-01-27T00:46:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.562+0000 7fd5c06b2700 1 -- 172.21.15.46:0/700530631 wait complete. 2024-01-27T00:46:47.577 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-27T00:46:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.716+0000 7f7494e3e700 1 Processor -- start 2024-01-27T00:46:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.716+0000 7f7494e3e700 1 -- start start 2024-01-27T00:46:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.718+0000 7f7494e3e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f74901597d0 0x7f749006ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.718+0000 7f7494e3e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f749015ad40 0x7f749006f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.718+0000 7f7494e3e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74900cbfb0 con 0x7f7490159ba0 2024-01-27T00:46:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.718+0000 7f7494e3e700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f749015a8b0 con 0x7f74901597d0 2024-01-27T00:46:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.718+0000 7f7494e3e700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74900cbcd0 con 0x7f749015ad40 2024-01-27T00:46:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.718+0000 7f748e59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f74901597d0 0x7f749006ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.718+0000 7f748e59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f74901597d0 0x7f749006ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59836/0 (socket says 172.21.15.46:59836) 2024-01-27T00:46:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.718+0000 7f748e59c700 1 -- 172.21.15.46:0/3207921249 learned_addr learned my addr 172.21.15.46:0/3207921249 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.718+0000 7f748dd9b700 1 --2- 172.21.15.46:0/3207921249 >> v2:172.21.15.106:3300/0 conn(0x7f749015ad40 0x7f749006f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.718+0000 7f748e59c700 1 -- 172.21.15.46:0/3207921249 >> v2:172.21.15.106:3300/0 conn(0x7f749015ad40 msgr2=0x7f749006f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.718+0000 7f748e59c700 1 --2- 172.21.15.46:0/3207921249 >> v2:172.21.15.106:3300/0 conn(0x7f749015ad40 0x7f749006f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.718+0000 7f748e59c700 1 -- 172.21.15.46:0/3207921249 >> v1:172.21.15.106:6789/0 conn(0x7f7490159ba0 legacy=0x7f74900cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.718+0000 7f748e59c700 1 -- 172.21.15.46:0/3207921249 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749015a260 con 0x7f74901597d0 2024-01-27T00:46:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.718+0000 7f748e59c700 1 --2- 172.21.15.46:0/3207921249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74901597d0 0x7f749006ec50 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f749006e640 tx=0x7f7480009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e17b09b923aaaae server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.719+0000 7f748d59a700 1 -- 172.21.15.46:0/3207921249 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748000b860 con 0x7f74901597d0 2024-01-27T00:46:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.719+0000 7f748d59a700 1 -- 172.21.15.46:0/3207921249 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748000b9c0 con 0x7f74901597d0 2024-01-27T00:46:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.719+0000 7f748d59a700 1 -- 172.21.15.46:0/3207921249 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748000bd80 con 0x7f74901597d0 2024-01-27T00:46:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.719+0000 7f7494e3e700 1 -- 172.21.15.46:0/3207921249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74901597d0 msgr2=0x7f749006ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.719+0000 7f7494e3e700 1 --2- 172.21.15.46:0/3207921249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74901597d0 0x7f749006ec50 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f749006e640 tx=0x7f7480009250 comp rx=0 tx=0).stop 2024-01-27T00:46:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.719+0000 7f7494e3e700 1 -- 172.21.15.46:0/3207921249 shutdown_connections 2024-01-27T00:46:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.719+0000 7f7494e3e700 1 --2- 172.21.15.46:0/3207921249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74901597d0 0x7f749006ec50 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.719+0000 7f7494e3e700 1 --2- 172.21.15.46:0/3207921249 >> v2:172.21.15.106:3300/0 conn(0x7f749015ad40 0x7f749006f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.719+0000 7f7494e3e700 1 -- 172.21.15.46:0/3207921249 >> 172.21.15.46:0/3207921249 conn(0x7f74900ca9a0 msgr2=0x7f7490068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.719+0000 7f7494e3e700 1 -- 172.21.15.46:0/3207921249 shutdown_connections 2024-01-27T00:46:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.719+0000 7f7494e3e700 1 -- 172.21.15.46:0/3207921249 wait complete. 2024-01-27T00:46:47.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.721+0000 7f7494e3e700 1 Processor -- start 2024-01-27T00:46:47.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.721+0000 7f7494e3e700 1 -- start start 2024-01-27T00:46:47.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.721+0000 7f7494e3e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74901597d0 0x7f7490166c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:47.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.721+0000 7f7494e3e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7490159ba0 0x7f74901513a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:47.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.721+0000 7f7494e3e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f749015ad40 0x7f74900c2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:47.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.721+0000 7f7494e3e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7490152420 con 0x7f749015ad40 2024-01-27T00:46:47.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.721+0000 7f7494e3e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749014ca70 con 0x7f74901597d0 2024-01-27T00:46:47.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.721+0000 7f7494e3e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74901671c0 con 0x7f7490159ba0 2024-01-27T00:46:47.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.722+0000 7f748ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f749015ad40 0x7f74900c2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.722+0000 7f748ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f749015ad40 0x7f74900c2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59842/0 (socket says 172.21.15.46:59842) 2024-01-27T00:46:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.722+0000 7f748ed9d700 1 -- 172.21.15.46:0/3420637762 learned_addr learned my addr 172.21.15.46:0/3420637762 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.722+0000 7f748dd9b700 1 --2- 172.21.15.46:0/3420637762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7490159ba0 0x7f74901513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.722+0000 7f748e59c700 1 --2- 172.21.15.46:0/3420637762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74901597d0 0x7f7490166c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.722+0000 7f748ed9d700 1 -- 172.21.15.46:0/3420637762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7490159ba0 msgr2=0x7f74901513a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.722+0000 7f748ed9d700 1 --2- 172.21.15.46:0/3420637762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7490159ba0 0x7f74901513a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.722+0000 7f748ed9d700 1 -- 172.21.15.46:0/3420637762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74901597d0 msgr2=0x7f7490166c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.722+0000 7f748ed9d700 1 --2- 172.21.15.46:0/3420637762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74901597d0 0x7f7490166c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.722+0000 7f748ed9d700 1 -- 172.21.15.46:0/3420637762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74800094f0 con 0x7f749015ad40 2024-01-27T00:46:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.722+0000 7f748ed9d700 1 --2- 172.21.15.46:0/3420637762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f749015ad40 0x7f74900c2200 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f748800c8e0 tx=0x7f7488001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.722+0000 7f74867fc700 1 -- 172.21.15.46:0/3420637762 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748800ce80 con 0x7f749015ad40 2024-01-27T00:46:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.722+0000 7f74867fc700 1 -- 172.21.15.46:0/3420637762 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748800a7e0 con 0x7f749015ad40 2024-01-27T00:46:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.722+0000 7f74867fc700 1 -- 172.21.15.46:0/3420637762 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748800ab80 con 0x7f749015ad40 2024-01-27T00:46:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.722+0000 7f7494e3e700 1 -- 172.21.15.46:0/3420637762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f749014cda0 con 0x7f749015ad40 2024-01-27T00:46:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.722+0000 7f7494e3e700 1 -- 172.21.15.46:0/3420637762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7490151e60 con 0x7f749015ad40 2024-01-27T00:46:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.723+0000 7f7494e3e700 1 -- 172.21.15.46:0/3420637762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f749004c710 con 0x7f749015ad40 2024-01-27T00:46:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.724+0000 7f74867fc700 1 -- 172.21.15.46:0/3420637762 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7488012420 con 0x7f749015ad40 2024-01-27T00:46:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.724+0000 7f74867fc700 1 --2- 172.21.15.46:0/3420637762 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7470066f30 0x7f74700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:47.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.724+0000 7f74867fc700 1 -- 172.21.15.46:0/3420637762 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f7488084b10 con 0x7f749015ad40 2024-01-27T00:46:47.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.724+0000 7f748e59c700 1 --2- 172.21.15.46:0/3420637762 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7470066f30 0x7f74700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:47.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.725+0000 7f748e59c700 1 --2- 172.21.15.46:0/3420637762 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7470066f30 0x7f74700693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f749006e640 tx=0x7f7480017000 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:47.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.729+0000 7f74867fc700 1 -- 172.21.15.46:0/3420637762 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7488051a90 con 0x7f749015ad40 2024-01-27T00:46:47.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:47.982+0000 7f7494e3e700 1 -- 172.21.15.46:0/3420637762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f749015a5b0 con 0x7f749015ad40 2024-01-27T00:46:48.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.563+0000 7f74867fc700 1 -- 172.21.15.46:0/3420637762 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v88) v1 ==== 122+0+0 (secure 0 0 0) 0x7f74880550b0 con 0x7f749015ad40 2024-01-27T00:46:48.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.566+0000 7f7494e3e700 1 -- 172.21.15.46:0/3420637762 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7470066f30 msgr2=0x7f74700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:48.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.566+0000 7f7494e3e700 1 --2- 172.21.15.46:0/3420637762 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7470066f30 0x7f74700693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f749006e640 tx=0x7f7480017000 comp rx=0 tx=0).stop 2024-01-27T00:46:48.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.566+0000 7f7494e3e700 1 -- 172.21.15.46:0/3420637762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f749015ad40 msgr2=0x7f74900c2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:48.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.566+0000 7f7494e3e700 1 --2- 172.21.15.46:0/3420637762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f749015ad40 0x7f74900c2200 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f748800c8e0 tx=0x7f7488001710 comp rx=0 tx=0).stop 2024-01-27T00:46:48.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.566+0000 7f7494e3e700 1 -- 172.21.15.46:0/3420637762 shutdown_connections 2024-01-27T00:46:48.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.566+0000 7f7494e3e700 1 --2- 172.21.15.46:0/3420637762 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7470066f30 0x7f74700693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:48.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.566+0000 7f7494e3e700 1 --2- 172.21.15.46:0/3420637762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f749015ad40 0x7f74900c2200 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:48.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.566+0000 7f7494e3e700 1 --2- 172.21.15.46:0/3420637762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74901597d0 0x7f7490166c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:48.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.566+0000 7f7494e3e700 1 --2- 172.21.15.46:0/3420637762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7490159ba0 0x7f74901513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:48.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.566+0000 7f7494e3e700 1 -- 172.21.15.46:0/3420637762 >> 172.21.15.46:0/3420637762 conn(0x7f74900ca9a0 msgr2=0x7f74900c91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:48.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.566+0000 7f7494e3e700 1 -- 172.21.15.46:0/3420637762 shutdown_connections 2024-01-27T00:46:48.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.566+0000 7f7494e3e700 1 -- 172.21.15.46:0/3420637762 wait complete. 2024-01-27T00:46:48.581 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T00:46:48.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.718+0000 7f54b4aee700 1 Processor -- start 2024-01-27T00:46:48.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.718+0000 7f54b4aee700 1 -- start start 2024-01-27T00:46:48.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.719+0000 7f54b4aee700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f54b01561d0 0x7f54b00ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:48.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.719+0000 7f54b4aee700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f54b006cbb0 0x7f54b01565a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:48.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.719+0000 7f54b4aee700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b00b0760 con 0x7f54b00b0280 2024-01-27T00:46:48.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.719+0000 7f54b4aee700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54b00b0a40 con 0x7f54b01561d0 2024-01-27T00:46:48.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.719+0000 7f54b4aee700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54b00b0d20 con 0x7f54b006cbb0 2024-01-27T00:46:48.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.719+0000 7f54ae59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f54b01561d0 0x7f54b00ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:48.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.719+0000 7f54ae59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f54b01561d0 0x7f54b00ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59846/0 (socket says 172.21.15.46:59846) 2024-01-27T00:46:48.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.719+0000 7f54ae59c700 1 -- 172.21.15.46:0/374061941 learned_addr learned my addr 172.21.15.46:0/374061941 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:48.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.720+0000 7f54aed9d700 1 --2- 172.21.15.46:0/374061941 >> v2:172.21.15.106:3300/0 conn(0x7f54b006cbb0 0x7f54b01565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:48.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.720+0000 7f54ae59c700 1 -- 172.21.15.46:0/374061941 >> v2:172.21.15.106:3300/0 conn(0x7f54b006cbb0 msgr2=0x7f54b01565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:48.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.720+0000 7f54ae59c700 1 --2- 172.21.15.46:0/374061941 >> v2:172.21.15.106:3300/0 conn(0x7f54b006cbb0 0x7f54b01565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:48.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.720+0000 7f54ae59c700 1 -- 172.21.15.46:0/374061941 >> v1:172.21.15.106:6789/0 conn(0x7f54b00b0280 legacy=0x7f54b006c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:48.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.720+0000 7f54ae59c700 1 -- 172.21.15.46:0/374061941 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b00b1000 con 0x7f54b01561d0 2024-01-27T00:46:48.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.720+0000 7f54ae59c700 1 --2- 172.21.15.46:0/374061941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b01561d0 0x7f54b00ca5f0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f54b0070010 tx=0x7f5498009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33c1c9ef20815f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:48.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.720+0000 7f54ad59a700 1 -- 172.21.15.46:0/374061941 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549800b860 con 0x7f54b01561d0 2024-01-27T00:46:48.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.720+0000 7f54ad59a700 1 -- 172.21.15.46:0/374061941 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549800b9c0 con 0x7f54b01561d0 2024-01-27T00:46:48.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.720+0000 7f54ad59a700 1 -- 172.21.15.46:0/374061941 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549800bd80 con 0x7f54b01561d0 2024-01-27T00:46:48.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.721+0000 7f54b4aee700 1 -- 172.21.15.46:0/374061941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b01561d0 msgr2=0x7f54b00ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:48.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.721+0000 7f54b4aee700 1 --2- 172.21.15.46:0/374061941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b01561d0 0x7f54b00ca5f0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f54b0070010 tx=0x7f5498009250 comp rx=0 tx=0).stop 2024-01-27T00:46:48.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.721+0000 7f54b4aee700 1 -- 172.21.15.46:0/374061941 shutdown_connections 2024-01-27T00:46:48.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.721+0000 7f54b4aee700 1 --2- 172.21.15.46:0/374061941 >> v2:172.21.15.106:3300/0 conn(0x7f54b006cbb0 0x7f54b01565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:48.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.721+0000 7f54b4aee700 1 --2- 172.21.15.46:0/374061941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b01561d0 0x7f54b00ca5f0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:48.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.721+0000 7f54b4aee700 1 -- 172.21.15.46:0/374061941 >> 172.21.15.46:0/374061941 conn(0x7f54b006b630 msgr2=0x7f54b00aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:48.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.721+0000 7f54b4aee700 1 -- 172.21.15.46:0/374061941 shutdown_connections 2024-01-27T00:46:48.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.721+0000 7f54b4aee700 1 -- 172.21.15.46:0/374061941 wait complete. 2024-01-27T00:46:48.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.722+0000 7f54b4aee700 1 Processor -- start 2024-01-27T00:46:48.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.722+0000 7f54b4aee700 1 -- start start 2024-01-27T00:46:48.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54b4aee700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f54b006cbb0 0x7f54b0157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:48.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54b4aee700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54b00b0280 0x7f54b00ada80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:48.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54b4aee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b01561d0 0x7f54b0150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:48.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54b4aee700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b0159620 con 0x7f54b01561d0 2024-01-27T00:46:48.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54b4aee700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b0158580 con 0x7f54b00b0280 2024-01-27T00:46:48.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54b4aee700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b0158860 con 0x7f54b006cbb0 2024-01-27T00:46:48.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54aed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b01561d0 0x7f54b0150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:48.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54aed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b01561d0 0x7f54b0150b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59852/0 (socket says 172.21.15.46:59852) 2024-01-27T00:46:48.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54aed9d700 1 -- 172.21.15.46:0/4067658923 learned_addr learned my addr 172.21.15.46:0/4067658923 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:48.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54ae59c700 1 --2- 172.21.15.46:0/4067658923 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f54b006cbb0 0x7f54b0157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:48.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54add9b700 1 --2- 172.21.15.46:0/4067658923 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54b00b0280 0x7f54b00ada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:48.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54aed9d700 1 -- 172.21.15.46:0/4067658923 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f54b006cbb0 msgr2=0x7f54b0157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:48.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54aed9d700 1 --2- 172.21.15.46:0/4067658923 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f54b006cbb0 0x7f54b0157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:48.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54aed9d700 1 -- 172.21.15.46:0/4067658923 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54b00b0280 msgr2=0x7f54b00ada80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:48.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54aed9d700 1 --2- 172.21.15.46:0/4067658923 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54b00b0280 0x7f54b00ada80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:48.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54aed9d700 1 -- 172.21.15.46:0/4067658923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54980094f0 con 0x7f54b01561d0 2024-01-27T00:46:48.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54ae59c700 1 --2- 172.21.15.46:0/4067658923 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f54b006cbb0 0x7f54b0157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:46:48.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.723+0000 7f54aed9d700 1 --2- 172.21.15.46:0/4067658923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b01561d0 0x7f54b0150b60 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f54a8007f30 tx=0x7f54a80098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:48.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.724+0000 7f54a67fc700 1 -- 172.21.15.46:0/4067658923 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a800ecf0 con 0x7f54b01561d0 2024-01-27T00:46:48.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.724+0000 7f54a67fc700 1 -- 172.21.15.46:0/4067658923 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a800ee50 con 0x7f54b01561d0 2024-01-27T00:46:48.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.724+0000 7f54b4aee700 1 -- 172.21.15.46:0/4067658923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b0158b40 con 0x7f54b01561d0 2024-01-27T00:46:48.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.724+0000 7f54a67fc700 1 -- 172.21.15.46:0/4067658923 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a8014670 con 0x7f54b01561d0 2024-01-27T00:46:48.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.724+0000 7f54b4aee700 1 -- 172.21.15.46:0/4067658923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b0159180 con 0x7f54b01561d0 2024-01-27T00:46:48.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.725+0000 7f54b4aee700 1 -- 172.21.15.46:0/4067658923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b00b29b0 con 0x7f54b01561d0 2024-01-27T00:46:48.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.725+0000 7f54a67fc700 1 -- 172.21.15.46:0/4067658923 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f54a80149e0 con 0x7f54b01561d0 2024-01-27T00:46:48.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.726+0000 7f54a67fc700 1 --2- 172.21.15.46:0/4067658923 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5490066f30 0x7f54900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:48.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.726+0000 7f54a67fc700 1 -- 172.21.15.46:0/4067658923 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f54a8084fa0 con 0x7f54b01561d0 2024-01-27T00:46:48.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.726+0000 7f54ae59c700 1 --2- 172.21.15.46:0/4067658923 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5490066f30 0x7f54900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:48.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.726+0000 7f54ae59c700 1 --2- 172.21.15.46:0/4067658923 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5490066f30 0x7f54900693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f54b0070010 tx=0x7f5498017000 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:48.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.730+0000 7f54a67fc700 1 -- 172.21.15.46:0/4067658923 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f54a8051f20 con 0x7f54b01561d0 2024-01-27T00:46:48.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.985+0000 7f54b4aee700 1 -- 172.21.15.46:0/4067658923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54b0158300 con 0x7f54b01561d0 2024-01-27T00:46:48.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.986+0000 7f54a67fc700 1 -- 172.21.15.46:0/4067658923 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4607 (secure 0 0 0) 0x7f54a8055540 con 0x7f54b01561d0 2024-01-27T00:46:48.985 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:48.985 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":88,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9061,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/3698053981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":3698053981},{"type":"v1","addr":"172.21.15.106:6833","nonce":3698053981}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":9064,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2056976879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2056976879},{"type":"v1","addr":"172.21.15.46:6834","nonce":2056976879}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":82},{"gid":9067,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1576406314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1576406314},{"type":"v1","addr":"172.21.15.46:6835","nonce":1576406314}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T00:46:44.978654+0000","modified":"2024-01-27T00:46:48.558026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9070":{"gid":9070,"name":"d","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6836/2857098625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2857098625},{"type":"v1","addr":"172.21.15.106:6836","nonce":2857098625}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2024-01-27T00:46:48.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.988+0000 7f54b4aee700 1 -- 172.21.15.46:0/4067658923 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5490066f30 msgr2=0x7f54900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:48.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.988+0000 7f54b4aee700 1 --2- 172.21.15.46:0/4067658923 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5490066f30 0x7f54900693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f54b0070010 tx=0x7f5498017000 comp rx=0 tx=0).stop 2024-01-27T00:46:48.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.988+0000 7f54b4aee700 1 -- 172.21.15.46:0/4067658923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b01561d0 msgr2=0x7f54b0150b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:48.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.988+0000 7f54b4aee700 1 --2- 172.21.15.46:0/4067658923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b01561d0 0x7f54b0150b60 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f54a8007f30 tx=0x7f54a80098e0 comp rx=0 tx=0).stop 2024-01-27T00:46:48.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.988+0000 7f54b4aee700 1 -- 172.21.15.46:0/4067658923 shutdown_connections 2024-01-27T00:46:48.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.988+0000 7f54b4aee700 1 --2- 172.21.15.46:0/4067658923 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f5490066f30 0x7f54900693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:48.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.988+0000 7f54b4aee700 1 --2- 172.21.15.46:0/4067658923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b01561d0 0x7f54b0150b60 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:48.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.988+0000 7f54b4aee700 1 --2- 172.21.15.46:0/4067658923 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f54b006cbb0 0x7f54b0157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:48.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.988+0000 7f54b4aee700 1 --2- 172.21.15.46:0/4067658923 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54b00b0280 0x7f54b00ada80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:48.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.988+0000 7f54b4aee700 1 -- 172.21.15.46:0/4067658923 >> 172.21.15.46:0/4067658923 conn(0x7f54b006b630 msgr2=0x7f54b00b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:48.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.988+0000 7f54b4aee700 1 -- 172.21.15.46:0/4067658923 shutdown_connections 2024-01-27T00:46:48.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:48.988+0000 7f54b4aee700 1 -- 172.21.15.46:0/4067658923 wait complete. 2024-01-27T00:46:48.989 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 88 2024-01-27T00:46:49.001 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T00:46:49.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.134+0000 7f6b2e5be700 1 Processor -- start 2024-01-27T00:46:49.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.134+0000 7f6b2e5be700 1 -- start start 2024-01-27T00:46:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.135+0000 7f6b2e5be700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b280cc6f0 0x7f6b28162170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.135+0000 7f6b2e5be700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b280cac60 con 0x7f6b280b1fc0 2024-01-27T00:46:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.135+0000 7f6b2e5be700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b28068dc0 con 0x7f6b280b2390 2024-01-27T00:46:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.135+0000 7f6b2e5be700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b280690a0 con 0x7f6b280cc6f0 2024-01-27T00:46:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.135+0000 7f6b27fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6b280b1fc0 0x7f6b28068310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41124/0 (socket says 172.21.15.46:41124) 2024-01-27T00:46:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.135+0000 7f6b27fff700 1 -- 172.21.15.46:0/2158623979 learned_addr learned my addr 172.21.15.46:0/2158623979 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.136+0000 7f6b2cb5b700 1 --2- 172.21.15.46:0/2158623979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b280cc6f0 0x7f6b28162170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.136+0000 7f6b26ffd700 1 -- 172.21.15.46:0/2158623979 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809252071 0 0) 0x7f6b280cac60 con 0x7f6b280b1fc0 2024-01-27T00:46:49.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.136+0000 7f6b26ffd700 1 -- 172.21.15.46:0/2158623979 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b10003570 con 0x7f6b280b1fc0 2024-01-27T00:46:49.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.136+0000 7f6b26ffd700 1 -- 172.21.15.46:0/2158623979 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126366667 0 0) 0x7f6b28068dc0 con 0x7f6b280b2390 2024-01-27T00:46:49.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.136+0000 7f6b26ffd700 1 -- 172.21.15.46:0/2158623979 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b280cac60 con 0x7f6b280b2390 2024-01-27T00:46:49.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.136+0000 7f6b2cb5b700 1 -- 172.21.15.46:0/2158623979 >> v1:172.21.15.106:6789/0 conn(0x7f6b280b2390 legacy=0x7f6b28156f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.136+0000 7f6b2cb5b700 1 -- 172.21.15.46:0/2158623979 >> v1:172.21.15.46:6789/0 conn(0x7f6b280b1fc0 legacy=0x7f6b28068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.136+0000 7f6b2cb5b700 1 -- 172.21.15.46:0/2158623979 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b280caee0 con 0x7f6b280cc6f0 2024-01-27T00:46:49.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.136+0000 7f6b2cb5b700 1 --2- 172.21.15.46:0/2158623979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b280cc6f0 0x7f6b28162170 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6b20009e90 tx=0x7f6b20009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9cd4765a0e355cc server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:49.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.136+0000 7f6b26ffd700 1 -- 172.21.15.46:0/2158623979 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2000e070 con 0x7f6b280cc6f0 2024-01-27T00:46:49.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.136+0000 7f6b26ffd700 1 -- 172.21.15.46:0/2158623979 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2000bb80 con 0x7f6b280cc6f0 2024-01-27T00:46:49.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.136+0000 7f6b26ffd700 1 -- 172.21.15.46:0/2158623979 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b20012430 con 0x7f6b280cc6f0 2024-01-27T00:46:49.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.137+0000 7f6b2e5be700 1 -- 172.21.15.46:0/2158623979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b280cc6f0 msgr2=0x7f6b28162170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.137+0000 7f6b2e5be700 1 --2- 172.21.15.46:0/2158623979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b280cc6f0 0x7f6b28162170 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6b20009e90 tx=0x7f6b20009780 comp rx=0 tx=0).stop 2024-01-27T00:46:49.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.137+0000 7f6b2e5be700 1 -- 172.21.15.46:0/2158623979 shutdown_connections 2024-01-27T00:46:49.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.137+0000 7f6b2e5be700 1 --2- 172.21.15.46:0/2158623979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b280cc6f0 0x7f6b28162170 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.137+0000 7f6b2e5be700 1 -- 172.21.15.46:0/2158623979 >> 172.21.15.46:0/2158623979 conn(0x7f6b280ccbc0 msgr2=0x7f6b280ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:49.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.137+0000 7f6b2e5be700 1 -- 172.21.15.46:0/2158623979 shutdown_connections 2024-01-27T00:46:49.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.137+0000 7f6b2e5be700 1 -- 172.21.15.46:0/2158623979 wait complete. 2024-01-27T00:46:49.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.139+0000 7f6b2e5be700 1 Processor -- start 2024-01-27T00:46:49.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.139+0000 7f6b2e5be700 1 -- start start 2024-01-27T00:46:49.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.140+0000 7f6b2e5be700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b280b1fc0 0x7f6b280c7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:49.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.140+0000 7f6b2e5be700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b280b2390 0x7f6b2814e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:49.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.140+0000 7f6b2e5be700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b280cc6f0 0x7f6b28157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:49.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.140+0000 7f6b2e5be700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b280b1ba0 con 0x7f6b280cc6f0 2024-01-27T00:46:49.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.140+0000 7f6b2e5be700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b28169260 con 0x7f6b280b2390 2024-01-27T00:46:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.140+0000 7f6b2e5be700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b280c7d40 con 0x7f6b280b1fc0 2024-01-27T00:46:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.140+0000 7f6b2cb5b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b280cc6f0 0x7f6b28157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.140+0000 7f6b2cb5b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b280cc6f0 0x7f6b28157dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59866/0 (socket says 172.21.15.46:59866) 2024-01-27T00:46:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.140+0000 7f6b2cb5b700 1 -- 172.21.15.46:0/1897094947 learned_addr learned my addr 172.21.15.46:0/1897094947 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.140+0000 7f6b27fff700 1 --2- 172.21.15.46:0/1897094947 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b280b1fc0 0x7f6b280c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.140+0000 7f6b277fe700 1 --2- 172.21.15.46:0/1897094947 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b280b2390 0x7f6b2814e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.140+0000 7f6b2cb5b700 1 -- 172.21.15.46:0/1897094947 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b280b1fc0 msgr2=0x7f6b280c7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.140+0000 7f6b2cb5b700 1 --2- 172.21.15.46:0/1897094947 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b280b1fc0 0x7f6b280c7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.140+0000 7f6b2cb5b700 1 -- 172.21.15.46:0/1897094947 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b280b2390 msgr2=0x7f6b2814e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.140+0000 7f6b2cb5b700 1 --2- 172.21.15.46:0/1897094947 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b280b2390 0x7f6b2814e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.140+0000 7f6b2cb5b700 1 -- 172.21.15.46:0/1897094947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b20009a20 con 0x7f6b280cc6f0 2024-01-27T00:46:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.140+0000 7f6b277fe700 1 --2- 172.21.15.46:0/1897094947 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b280b2390 0x7f6b2814e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:46:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.141+0000 7f6b2cb5b700 1 --2- 172.21.15.46:0/1897094947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b280cc6f0 0x7f6b28157dc0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f6b280cae10 tx=0x7f6b20019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.141+0000 7f6b0ffff700 1 -- 172.21.15.46:0/1897094947 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b20012e00 con 0x7f6b280cc6f0 2024-01-27T00:46:49.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.141+0000 7f6b0ffff700 1 -- 172.21.15.46:0/1897094947 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2001d630 con 0x7f6b280cc6f0 2024-01-27T00:46:49.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.141+0000 7f6b0ffff700 1 -- 172.21.15.46:0/1897094947 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2001d9f0 con 0x7f6b280cc6f0 2024-01-27T00:46:49.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.141+0000 7f6b2e5be700 1 -- 172.21.15.46:0/1897094947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b280c7f50 con 0x7f6b280cc6f0 2024-01-27T00:46:49.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.141+0000 7f6b2e5be700 1 -- 172.21.15.46:0/1897094947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b280c8510 con 0x7f6b280cc6f0 2024-01-27T00:46:49.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.142+0000 7f6b2e5be700 1 -- 172.21.15.46:0/1897094947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b280c8730 con 0x7f6b280cc6f0 2024-01-27T00:46:49.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.143+0000 7f6b0ffff700 1 -- 172.21.15.46:0/1897094947 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6b20024070 con 0x7f6b280cc6f0 2024-01-27T00:46:49.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.143+0000 7f6b0ffff700 1 --2- 172.21.15.46:0/1897094947 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b14066f30 0x7f6b140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:49.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.144+0000 7f6b0ffff700 1 -- 172.21.15.46:0/1897094947 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f6b20087340 con 0x7f6b280cc6f0 2024-01-27T00:46:49.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.144+0000 7f6b27fff700 1 --2- 172.21.15.46:0/1897094947 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b14066f30 0x7f6b140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:49.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.144+0000 7f6b27fff700 1 --2- 172.21.15.46:0/1897094947 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b14066f30 0x7f6b140693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6b180058d0 tx=0x7f6b18009040 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:49.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.147+0000 7f6b0ffff700 1 -- 172.21.15.46:0/1897094947 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6b20054350 con 0x7f6b280cc6f0 2024-01-27T00:46:49.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.355+0000 7f6b2e5be700 1 -- 172.21.15.46:0/1897094947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b280cb1c0 con 0x7f6b280cc6f0 2024-01-27T00:46:49.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.356+0000 7f6b0ffff700 1 -- 172.21.15.46:0/1897094947 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v175) v1 ==== 74+0+16234 (secure 0 0 0) 0x7f6b2002c090 con 0x7f6b280cc6f0 2024-01-27T00:46:49.356 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:49.356 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":175,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T00:46:46.547440+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-01-27T00:46:43.494684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-01-27T00:46:43.949785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T00:46:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.358+0000 7f6b2e5be700 1 -- 172.21.15.46:0/1897094947 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b14066f30 msgr2=0x7f6b140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.358+0000 7f6b2e5be700 1 --2- 172.21.15.46:0/1897094947 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b14066f30 0x7f6b140693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6b180058d0 tx=0x7f6b18009040 comp rx=0 tx=0).stop 2024-01-27T00:46:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.358+0000 7f6b2e5be700 1 -- 172.21.15.46:0/1897094947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b280cc6f0 msgr2=0x7f6b28157dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.358+0000 7f6b2e5be700 1 --2- 172.21.15.46:0/1897094947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b280cc6f0 0x7f6b28157dc0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f6b280cae10 tx=0x7f6b20019000 comp rx=0 tx=0).stop 2024-01-27T00:46:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.358+0000 7f6b2e5be700 1 -- 172.21.15.46:0/1897094947 shutdown_connections 2024-01-27T00:46:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.358+0000 7f6b2e5be700 1 --2- 172.21.15.46:0/1897094947 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f6b14066f30 0x7f6b140693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.358+0000 7f6b2e5be700 1 --2- 172.21.15.46:0/1897094947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b280cc6f0 0x7f6b28157dc0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.358+0000 7f6b2e5be700 1 --2- 172.21.15.46:0/1897094947 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b280b1fc0 0x7f6b280c7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.358+0000 7f6b2e5be700 1 --2- 172.21.15.46:0/1897094947 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b280b2390 0x7f6b2814e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.358+0000 7f6b2e5be700 1 -- 172.21.15.46:0/1897094947 >> 172.21.15.46:0/1897094947 conn(0x7f6b280ccbc0 msgr2=0x7f6b28153260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.358+0000 7f6b2e5be700 1 -- 172.21.15.46:0/1897094947 shutdown_connections 2024-01-27T00:46:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.358+0000 7f6b2e5be700 1 -- 172.21.15.46:0/1897094947 wait complete. 2024-01-27T00:46:49.372 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-27T00:46:49.372 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T00:46:49.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.504+0000 7f9095039700 1 Processor -- start 2024-01-27T00:46:49.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.504+0000 7f9095039700 1 -- start start 2024-01-27T00:46:49.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.505+0000 7f9095039700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90901561d0 0x7f90900ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:49.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.505+0000 7f9095039700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90900b0760 con 0x7f90900b0280 2024-01-27T00:46:49.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.505+0000 7f9095039700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90900b0a40 con 0x7f909006cbb0 2024-01-27T00:46:49.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.505+0000 7f9095039700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90900b0d20 con 0x7f90901561d0 2024-01-27T00:46:49.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.505+0000 7f908ed9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90901561d0 0x7f90900ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:49.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.505+0000 7f908e59c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f90900b0280 0x7f909006c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41132/0 (socket says 172.21.15.46:41132) 2024-01-27T00:46:49.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.505+0000 7f908e59c700 1 -- 172.21.15.46:0/243052100 learned_addr learned my addr 172.21.15.46:0/243052100 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.505+0000 7f908ed9d700 1 -- 172.21.15.46:0/243052100 >> v1:172.21.15.106:6789/0 conn(0x7f909006cbb0 legacy=0x7f90900b1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.505+0000 7f908ed9d700 1 -- 172.21.15.46:0/243052100 >> v1:172.21.15.46:6789/0 conn(0x7f90900b0280 legacy=0x7f909006c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.505+0000 7f908ed9d700 1 -- 172.21.15.46:0/243052100 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90900b1000 con 0x7f90901561d0 2024-01-27T00:46:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.505+0000 7f908dd9b700 1 -- 172.21.15.46:0/243052100 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374672568 0 0) 0x7f90900b0760 con 0x7f90900b0280 2024-01-27T00:46:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.506+0000 7f908ed9d700 1 --2- 172.21.15.46:0/243052100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90901561d0 0x7f90900ca5f0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f90800098e0 tx=0x7f90800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff431cf32170ab1 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.506+0000 7f908dd9b700 1 -- 172.21.15.46:0/243052100 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9080009e80 con 0x7f90901561d0 2024-01-27T00:46:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.506+0000 7f908dd9b700 1 -- 172.21.15.46:0/243052100 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f908000b860 con 0x7f90901561d0 2024-01-27T00:46:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.506+0000 7f908dd9b700 1 -- 172.21.15.46:0/243052100 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908000bbe0 con 0x7f90901561d0 2024-01-27T00:46:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.506+0000 7f9095039700 1 -- 172.21.15.46:0/243052100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90901561d0 msgr2=0x7f90900ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.506+0000 7f9095039700 1 --2- 172.21.15.46:0/243052100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90901561d0 0x7f90900ca5f0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f90800098e0 tx=0x7f90800091a0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.506+0000 7f9095039700 1 -- 172.21.15.46:0/243052100 shutdown_connections 2024-01-27T00:46:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.506+0000 7f9095039700 1 --2- 172.21.15.46:0/243052100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90901561d0 0x7f90900ca5f0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.506+0000 7f9095039700 1 -- 172.21.15.46:0/243052100 >> 172.21.15.46:0/243052100 conn(0x7f909006b630 msgr2=0x7f90900aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.506+0000 7f9095039700 1 -- 172.21.15.46:0/243052100 shutdown_connections 2024-01-27T00:46:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.507+0000 7f9095039700 1 -- 172.21.15.46:0/243052100 wait complete. 2024-01-27T00:46:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.508+0000 7f9095039700 1 Processor -- start 2024-01-27T00:46:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.508+0000 7f9095039700 1 -- start start 2024-01-27T00:46:49.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f9095039700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f909006cbb0 0x7f909014c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:49.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f9095039700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90900b0280 0x7f909014f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:49.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f9095039700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90901561d0 0x7f90900add80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:49.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f9095039700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909014c600 con 0x7f909006cbb0 2024-01-27T00:46:49.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f9095039700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90900ad8f0 con 0x7f90901561d0 2024-01-27T00:46:49.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f9095039700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909014e0e0 con 0x7f90900b0280 2024-01-27T00:46:49.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f908ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f909006cbb0 0x7f909014c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f908ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f909006cbb0 0x7f909014c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59894/0 (socket says 172.21.15.46:59894) 2024-01-27T00:46:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f908ed9d700 1 -- 172.21.15.46:0/3907279491 learned_addr learned my addr 172.21.15.46:0/3907279491 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f908e59c700 1 --2- 172.21.15.46:0/3907279491 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90900b0280 0x7f909014f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f908f59e700 1 --2- 172.21.15.46:0/3907279491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90901561d0 0x7f90900add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f908ed9d700 1 -- 172.21.15.46:0/3907279491 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90900b0280 msgr2=0x7f909014f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f908ed9d700 1 --2- 172.21.15.46:0/3907279491 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90900b0280 0x7f909014f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f908ed9d700 1 -- 172.21.15.46:0/3907279491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90901561d0 msgr2=0x7f90900add80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f908ed9d700 1 --2- 172.21.15.46:0/3907279491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90901561d0 0x7f90900add80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f908ed9d700 1 -- 172.21.15.46:0/3907279491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9080009420 con 0x7f909006cbb0 2024-01-27T00:46:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f908f59e700 1 --2- 172.21.15.46:0/3907279491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90901561d0 0x7f90900add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:46:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.509+0000 7f908ed9d700 1 --2- 172.21.15.46:0/3907279491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f909006cbb0 0x7f909014c0c0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f90900ce590 tx=0x7f9080017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.510+0000 7f9076ffd700 1 -- 172.21.15.46:0/3907279491 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908001f070 con 0x7f909006cbb0 2024-01-27T00:46:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.510+0000 7f9076ffd700 1 -- 172.21.15.46:0/3907279491 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9080000c90 con 0x7f909006cbb0 2024-01-27T00:46:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.510+0000 7f9095039700 1 -- 172.21.15.46:0/3907279491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90900acce0 con 0x7f909006cbb0 2024-01-27T00:46:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.510+0000 7f9076ffd700 1 -- 172.21.15.46:0/3907279491 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9080025420 con 0x7f909006cbb0 2024-01-27T00:46:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.510+0000 7f9095039700 1 -- 172.21.15.46:0/3907279491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90900ad2a0 con 0x7f909006cbb0 2024-01-27T00:46:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.511+0000 7f9095039700 1 -- 172.21.15.46:0/3907279491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9090064c40 con 0x7f909006cbb0 2024-01-27T00:46:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.511+0000 7f9076ffd700 1 -- 172.21.15.46:0/3907279491 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9080023070 con 0x7f909006cbb0 2024-01-27T00:46:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.511+0000 7f9076ffd700 1 --2- 172.21.15.46:0/3907279491 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f907c066f30 0x7f907c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.512+0000 7f9076ffd700 1 -- 172.21.15.46:0/3907279491 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f9080085c10 con 0x7f909006cbb0 2024-01-27T00:46:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.512+0000 7f908e59c700 1 --2- 172.21.15.46:0/3907279491 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f907c066f30 0x7f907c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.512+0000 7f908e59c700 1 --2- 172.21.15.46:0/3907279491 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f907c066f30 0x7f907c0693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f90900aecf0 tx=0x7f9084007d10 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.516+0000 7f9076ffd700 1 -- 172.21.15.46:0/3907279491 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9080052b90 con 0x7f909006cbb0 2024-01-27T00:46:49.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.714+0000 7f9095039700 1 -- 172.21.15.46:0/3907279491 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f909004c710 con 0x7f907c066f30 2024-01-27T00:46:49.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.724+0000 7f9076ffd700 1 -- 172.21.15.46:0/3907279491 <== mgr.8860 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290275 (secure 0 0 0) 0x7f909004c710 con 0x7f907c066f30 2024-01-27T00:46:49.727 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.727+0000 7f9095039700 1 -- 172.21.15.46:0/3907279491 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f907c066f30 msgr2=0x7f907c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.727+0000 7f9095039700 1 --2- 172.21.15.46:0/3907279491 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f907c066f30 0x7f907c0693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f90900aecf0 tx=0x7f9084007d10 comp rx=0 tx=0).stop 2024-01-27T00:46:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.727+0000 7f9095039700 1 -- 172.21.15.46:0/3907279491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f909006cbb0 msgr2=0x7f909014c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.727+0000 7f9095039700 1 --2- 172.21.15.46:0/3907279491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f909006cbb0 0x7f909014c0c0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f90900ce590 tx=0x7f9080017000 comp rx=0 tx=0).stop 2024-01-27T00:46:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.728+0000 7f9095039700 1 -- 172.21.15.46:0/3907279491 shutdown_connections 2024-01-27T00:46:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.728+0000 7f9095039700 1 --2- 172.21.15.46:0/3907279491 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f907c066f30 0x7f907c0693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.728+0000 7f9095039700 1 --2- 172.21.15.46:0/3907279491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f909006cbb0 0x7f909014c0c0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.728+0000 7f9095039700 1 --2- 172.21.15.46:0/3907279491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f90901561d0 0x7f90900add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.728+0000 7f9095039700 1 --2- 172.21.15.46:0/3907279491 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f90900b0280 0x7f909014f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.728+0000 7f9095039700 1 -- 172.21.15.46:0/3907279491 >> 172.21.15.46:0/3907279491 conn(0x7f909006b630 msgr2=0x7f9090152600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.728+0000 7f9095039700 1 -- 172.21.15.46:0/3907279491 shutdown_connections 2024-01-27T00:46:49.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.728+0000 7f9095039700 1 -- 172.21.15.46:0/3907279491 wait complete. 2024-01-27T00:46:49.731 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T00:46:49.741 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-01-27T00:46:48.760510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1592,"num_read_kb":3366,"num_write":534,"num_write_kb":3475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8711,"ondisk_log_size":8711,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2490824,"kb_used_data":18312,"kb_used_omap":0,"kb_used_meta":2472448,"kb_avail":752483896,"statfs":{"total":773094113280,"available":770543509504,"internally_reserved":0,"allocated":18751488,"data_stored":10280276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2531786752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.227445"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566300+0000","last_change":"2024-01-27T00:46:45.557055+0000","last_active":"2024-01-27T00:46:45.566300+0000","last_peered":"2024-01-27T00:46:45.566300+0000","last_clean":"2024-01-27T00:46:45.566300+0000","last_became_active":"2024-01-27T00:46:45.556790+0000","last_became_peered":"2024-01-27T00:46:45.556790+0000","last_unstale":"2024-01-27T00:46:45.566300+0000","last_undegraded":"2024-01-27T00:46:45.566300+0000","last_fullsized":"2024-01-27T00:46:45.566300+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:52:06.112690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574313+0000","last_change":"2024-01-27T00:46:45.568147+0000","last_active":"2024-01-27T00:46:45.574313+0000","last_peered":"2024-01-27T00:46:45.574313+0000","last_clean":"2024-01-27T00:46:45.574313+0000","last_became_active":"2024-01-27T00:46:45.567855+0000","last_became_peered":"2024-01-27T00:46:45.567855+0000","last_unstale":"2024-01-27T00:46:45.574313+0000","last_undegraded":"2024-01-27T00:46:45.574313+0000","last_fullsized":"2024-01-27T00:46:45.574313+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:26:59.092632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.565508+0000","last_change":"2024-01-27T00:46:45.562607+0000","last_active":"2024-01-27T00:46:45.565508+0000","last_peered":"2024-01-27T00:46:45.565508+0000","last_clean":"2024-01-27T00:46:45.565508+0000","last_became_active":"2024-01-27T00:46:45.561888+0000","last_became_peered":"2024-01-27T00:46:45.561888+0000","last_unstale":"2024-01-27T00:46:45.565508+0000","last_undegraded":"2024-01-27T00:46:45.565508+0000","last_fullsized":"2024-01-27T00:46:45.565508+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:35:54.188355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.567733+0000","last_change":"2024-01-27T00:46:45.560536+0000","last_active":"2024-01-27T00:46:45.567733+0000","last_peered":"2024-01-27T00:46:45.567733+0000","last_clean":"2024-01-27T00:46:45.567733+0000","last_became_active":"2024-01-27T00:46:45.560235+0000","last_became_peered":"2024-01-27T00:46:45.560235+0000","last_unstale":"2024-01-27T00:46:45.567733+0000","last_undegraded":"2024-01-27T00:46:45.567733+0000","last_fullsized":"2024-01-27T00:46:45.567733+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:41:40.453855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.571085+0000","last_change":"2024-01-27T00:46:45.568127+0000","last_active":"2024-01-27T00:46:45.571085+0000","last_peered":"2024-01-27T00:46:45.571085+0000","last_clean":"2024-01-27T00:46:45.571085+0000","last_became_active":"2024-01-27T00:46:45.567791+0000","last_became_peered":"2024-01-27T00:46:45.567791+0000","last_unstale":"2024-01-27T00:46:45.571085+0000","last_undegraded":"2024-01-27T00:46:45.571085+0000","last_fullsized":"2024-01-27T00:46:45.571085+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:35:43.360306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.570790+0000","last_change":"2024-01-27T00:46:45.568275+0000","last_active":"2024-01-27T00:46:45.570790+0000","last_peered":"2024-01-27T00:46:45.570790+0000","last_clean":"2024-01-27T00:46:45.570790+0000","last_became_active":"2024-01-27T00:46:45.567998+0000","last_became_peered":"2024-01-27T00:46:45.567998+0000","last_unstale":"2024-01-27T00:46:45.570790+0000","last_undegraded":"2024-01-27T00:46:45.570790+0000","last_fullsized":"2024-01-27T00:46:45.570790+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:26.817597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.567746+0000","last_change":"2024-01-27T00:46:45.559800+0000","last_active":"2024-01-27T00:46:45.567746+0000","last_peered":"2024-01-27T00:46:45.567746+0000","last_clean":"2024-01-27T00:46:45.567746+0000","last_became_active":"2024-01-27T00:46:45.559524+0000","last_became_peered":"2024-01-27T00:46:45.559524+0000","last_unstale":"2024-01-27T00:46:45.567746+0000","last_undegraded":"2024-01-27T00:46:45.567746+0000","last_fullsized":"2024-01-27T00:46:45.567746+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:04:20.968716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.552575+0000","last_change":"2024-01-27T00:46:45.548896+0000","last_active":"2024-01-27T00:46:45.552575+0000","last_peered":"2024-01-27T00:46:45.552575+0000","last_clean":"2024-01-27T00:46:45.552575+0000","last_became_active":"2024-01-27T00:46:45.548490+0000","last_became_peered":"2024-01-27T00:46:45.548490+0000","last_unstale":"2024-01-27T00:46:45.552575+0000","last_undegraded":"2024-01-27T00:46:45.552575+0000","last_fullsized":"2024-01-27T00:46:45.552575+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:00:27.609050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.569425+0000","last_change":"2024-01-27T00:46:45.565679+0000","last_active":"2024-01-27T00:46:45.569425+0000","last_peered":"2024-01-27T00:46:45.569425+0000","last_clean":"2024-01-27T00:46:45.569425+0000","last_became_active":"2024-01-27T00:46:45.565374+0000","last_became_peered":"2024-01-27T00:46:45.565374+0000","last_unstale":"2024-01-27T00:46:45.569425+0000","last_undegraded":"2024-01-27T00:46:45.569425+0000","last_fullsized":"2024-01-27T00:46:45.569425+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:00:18.369351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.569451+0000","last_change":"2024-01-27T00:46:45.557200+0000","last_active":"2024-01-27T00:46:45.569451+0000","last_peered":"2024-01-27T00:46:45.569451+0000","last_clean":"2024-01-27T00:46:45.569451+0000","last_became_active":"2024-01-27T00:46:45.556886+0000","last_became_peered":"2024-01-27T00:46:45.556886+0000","last_unstale":"2024-01-27T00:46:45.569451+0000","last_undegraded":"2024-01-27T00:46:45.569451+0000","last_fullsized":"2024-01-27T00:46:45.569451+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:56:30.192694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.558768+0000","last_change":"2024-01-27T00:46:45.557911+0000","last_active":"2024-01-27T00:46:45.558768+0000","last_peered":"2024-01-27T00:46:45.558768+0000","last_clean":"2024-01-27T00:46:45.558768+0000","last_became_active":"2024-01-27T00:46:45.557573+0000","last_became_peered":"2024-01-27T00:46:45.557573+0000","last_unstale":"2024-01-27T00:46:45.558768+0000","last_undegraded":"2024-01-27T00:46:45.558768+0000","last_fullsized":"2024-01-27T00:46:45.558768+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:59:17.572174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.567831+0000","last_change":"2024-01-27T00:46:45.565562+0000","last_active":"2024-01-27T00:46:45.567831+0000","last_peered":"2024-01-27T00:46:45.567831+0000","last_clean":"2024-01-27T00:46:45.567831+0000","last_became_active":"2024-01-27T00:46:45.565307+0000","last_became_peered":"2024-01-27T00:46:45.565307+0000","last_unstale":"2024-01-27T00:46:45.567831+0000","last_undegraded":"2024-01-27T00:46:45.567831+0000","last_fullsized":"2024-01-27T00:46:45.567831+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:39:52.381793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.564522+0000","last_change":"2024-01-27T00:46:45.559536+0000","last_active":"2024-01-27T00:46:45.564522+0000","last_peered":"2024-01-27T00:46:45.564522+0000","last_clean":"2024-01-27T00:46:45.564522+0000","last_became_active":"2024-01-27T00:46:45.559054+0000","last_became_peered":"2024-01-27T00:46:45.559054+0000","last_unstale":"2024-01-27T00:46:45.564522+0000","last_undegraded":"2024-01-27T00:46:45.564522+0000","last_fullsized":"2024-01-27T00:46:45.564522+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:33:10.523909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.567859+0000","last_change":"2024-01-27T00:46:45.560510+0000","last_active":"2024-01-27T00:46:45.567859+0000","last_peered":"2024-01-27T00:46:45.567859+0000","last_clean":"2024-01-27T00:46:45.567859+0000","last_became_active":"2024-01-27T00:46:45.560103+0000","last_became_peered":"2024-01-27T00:46:45.560103+0000","last_unstale":"2024-01-27T00:46:45.567859+0000","last_undegraded":"2024-01-27T00:46:45.567859+0000","last_fullsized":"2024-01-27T00:46:45.567859+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:42:33.340128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.573835+0000","last_change":"2024-01-27T00:46:45.563721+0000","last_active":"2024-01-27T00:46:45.573835+0000","last_peered":"2024-01-27T00:46:45.573835+0000","last_clean":"2024-01-27T00:46:45.573835+0000","last_became_active":"2024-01-27T00:46:45.563013+0000","last_became_peered":"2024-01-27T00:46:45.563013+0000","last_unstale":"2024-01-27T00:46:45.573835+0000","last_undegraded":"2024-01-27T00:46:45.573835+0000","last_fullsized":"2024-01-27T00:46:45.573835+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:25.389867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.564719+0000","last_change":"2024-01-27T00:46:45.562938+0000","last_active":"2024-01-27T00:46:45.564719+0000","last_peered":"2024-01-27T00:46:45.564719+0000","last_clean":"2024-01-27T00:46:45.564719+0000","last_became_active":"2024-01-27T00:46:45.562604+0000","last_became_peered":"2024-01-27T00:46:45.562604+0000","last_unstale":"2024-01-27T00:46:45.564719+0000","last_undegraded":"2024-01-27T00:46:45.564719+0000","last_fullsized":"2024-01-27T00:46:45.564719+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:22:25.580376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.568023+0000","last_change":"2024-01-27T00:46:45.561614+0000","last_active":"2024-01-27T00:46:45.568023+0000","last_peered":"2024-01-27T00:46:45.568023+0000","last_clean":"2024-01-27T00:46:45.568023+0000","last_became_active":"2024-01-27T00:46:45.561278+0000","last_became_peered":"2024-01-27T00:46:45.561278+0000","last_unstale":"2024-01-27T00:46:45.568023+0000","last_undegraded":"2024-01-27T00:46:45.568023+0000","last_fullsized":"2024-01-27T00:46:45.568023+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:58.834738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.561918+0000","last_change":"2024-01-27T00:46:45.561317+0000","last_active":"2024-01-27T00:46:45.561918+0000","last_peered":"2024-01-27T00:46:45.561918+0000","last_clean":"2024-01-27T00:46:45.561918+0000","last_became_active":"2024-01-27T00:46:45.561008+0000","last_became_peered":"2024-01-27T00:46:45.561008+0000","last_unstale":"2024-01-27T00:46:45.561918+0000","last_undegraded":"2024-01-27T00:46:45.561918+0000","last_fullsized":"2024-01-27T00:46:45.561918+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:47:36.930423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566596+0000","last_change":"2024-01-27T00:46:45.559098+0000","last_active":"2024-01-27T00:46:45.566596+0000","last_peered":"2024-01-27T00:46:45.566596+0000","last_clean":"2024-01-27T00:46:45.566596+0000","last_became_active":"2024-01-27T00:46:45.558697+0000","last_became_peered":"2024-01-27T00:46:45.558697+0000","last_unstale":"2024-01-27T00:46:45.566596+0000","last_undegraded":"2024-01-27T00:46:45.566596+0000","last_fullsized":"2024-01-27T00:46:45.566596+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:09:01.270842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574654+0000","last_change":"2024-01-27T00:46:45.569640+0000","last_active":"2024-01-27T00:46:45.574654+0000","last_peered":"2024-01-27T00:46:45.574654+0000","last_clean":"2024-01-27T00:46:45.574654+0000","last_became_active":"2024-01-27T00:46:45.569340+0000","last_became_peered":"2024-01-27T00:46:45.569340+0000","last_unstale":"2024-01-27T00:46:45.574654+0000","last_undegraded":"2024-01-27T00:46:45.574654+0000","last_fullsized":"2024-01-27T00:46:45.574654+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:59:49.044163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.570274+0000","last_change":"2024-01-27T00:46:45.565670+0000","last_active":"2024-01-27T00:46:45.570274+0000","last_peered":"2024-01-27T00:46:45.570274+0000","last_clean":"2024-01-27T00:46:45.570274+0000","last_became_active":"2024-01-27T00:46:45.565313+0000","last_became_peered":"2024-01-27T00:46:45.565313+0000","last_unstale":"2024-01-27T00:46:45.570274+0000","last_undegraded":"2024-01-27T00:46:45.570274+0000","last_fullsized":"2024-01-27T00:46:45.570274+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:13:12.945507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.573589+0000","last_change":"2024-01-27T00:46:45.569350+0000","last_active":"2024-01-27T00:46:45.573589+0000","last_peered":"2024-01-27T00:46:45.573589+0000","last_clean":"2024-01-27T00:46:45.573589+0000","last_became_active":"2024-01-27T00:46:45.569024+0000","last_became_peered":"2024-01-27T00:46:45.569024+0000","last_unstale":"2024-01-27T00:46:45.573589+0000","last_undegraded":"2024-01-27T00:46:45.573589+0000","last_fullsized":"2024-01-27T00:46:45.573589+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:44:56.459748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.559040+0000","last_change":"2024-01-27T00:46:45.556267+0000","last_active":"2024-01-27T00:46:45.559040+0000","last_peered":"2024-01-27T00:46:45.559040+0000","last_clean":"2024-01-27T00:46:45.559040+0000","last_became_active":"2024-01-27T00:46:45.555962+0000","last_became_peered":"2024-01-27T00:46:45.555962+0000","last_unstale":"2024-01-27T00:46:45.559040+0000","last_undegraded":"2024-01-27T00:46:45.559040+0000","last_fullsized":"2024-01-27T00:46:45.559040+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:35:54.188355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.557839+0000","last_change":"2024-01-27T00:46:45.555830+0000","last_active":"2024-01-27T00:46:45.557839+0000","last_peered":"2024-01-27T00:46:45.557839+0000","last_clean":"2024-01-27T00:46:45.557839+0000","last_became_active":"2024-01-27T00:46:45.555524+0000","last_became_peered":"2024-01-27T00:46:45.555524+0000","last_unstale":"2024-01-27T00:46:45.557839+0000","last_undegraded":"2024-01-27T00:46:45.557839+0000","last_fullsized":"2024-01-27T00:46:45.557839+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:59:50.650401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566322+0000","last_change":"2024-01-27T00:46:45.559100+0000","last_active":"2024-01-27T00:46:45.566322+0000","last_peered":"2024-01-27T00:46:45.566322+0000","last_clean":"2024-01-27T00:46:45.566322+0000","last_became_active":"2024-01-27T00:46:45.558851+0000","last_became_peered":"2024-01-27T00:46:45.558851+0000","last_unstale":"2024-01-27T00:46:45.566322+0000","last_undegraded":"2024-01-27T00:46:45.566322+0000","last_fullsized":"2024-01-27T00:46:45.566322+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:12:26.858732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566356+0000","last_change":"2024-01-27T00:46:45.558189+0000","last_active":"2024-01-27T00:46:45.566356+0000","last_peered":"2024-01-27T00:46:45.566356+0000","last_clean":"2024-01-27T00:46:45.566356+0000","last_became_active":"2024-01-27T00:46:45.557564+0000","last_became_peered":"2024-01-27T00:46:45.557564+0000","last_unstale":"2024-01-27T00:46:45.566356+0000","last_undegraded":"2024-01-27T00:46:45.566356+0000","last_fullsized":"2024-01-27T00:46:45.566356+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:23:24.863025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.573177+0000","last_change":"2024-01-27T00:46:45.570543+0000","last_active":"2024-01-27T00:46:45.573177+0000","last_peered":"2024-01-27T00:46:45.573177+0000","last_clean":"2024-01-27T00:46:45.573177+0000","last_became_active":"2024-01-27T00:46:45.570129+0000","last_became_peered":"2024-01-27T00:46:45.570129+0000","last_unstale":"2024-01-27T00:46:45.573177+0000","last_undegraded":"2024-01-27T00:46:45.573177+0000","last_fullsized":"2024-01-27T00:46:45.573177+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:14.376349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.554835+0000","last_change":"2024-01-27T00:46:45.553830+0000","last_active":"2024-01-27T00:46:45.554835+0000","last_peered":"2024-01-27T00:46:45.554835+0000","last_clean":"2024-01-27T00:46:45.554835+0000","last_became_active":"2024-01-27T00:46:45.553190+0000","last_became_peered":"2024-01-27T00:46:45.553190+0000","last_unstale":"2024-01-27T00:46:45.554835+0000","last_undegraded":"2024-01-27T00:46:45.554835+0000","last_fullsized":"2024-01-27T00:46:45.554835+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:58:19.948464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.571005+0000","last_change":"2024-01-27T00:46:45.562755+0000","last_active":"2024-01-27T00:46:45.571005+0000","last_peered":"2024-01-27T00:46:45.571005+0000","last_clean":"2024-01-27T00:46:45.571005+0000","last_became_active":"2024-01-27T00:46:45.562450+0000","last_became_peered":"2024-01-27T00:46:45.562450+0000","last_unstale":"2024-01-27T00:46:45.571005+0000","last_undegraded":"2024-01-27T00:46:45.571005+0000","last_fullsized":"2024-01-27T00:46:45.571005+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:35.210458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.563461+0000","last_change":"2024-01-27T00:46:45.562746+0000","last_active":"2024-01-27T00:46:45.563461+0000","last_peered":"2024-01-27T00:46:45.563461+0000","last_clean":"2024-01-27T00:46:45.563461+0000","last_became_active":"2024-01-27T00:46:45.562444+0000","last_became_peered":"2024-01-27T00:46:45.562444+0000","last_unstale":"2024-01-27T00:46:45.563461+0000","last_undegraded":"2024-01-27T00:46:45.563461+0000","last_fullsized":"2024-01-27T00:46:45.563461+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:09:10.316894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.564589+0000","last_change":"2024-01-27T00:46:45.560219+0000","last_active":"2024-01-27T00:46:45.564589+0000","last_peered":"2024-01-27T00:46:45.564589+0000","last_clean":"2024-01-27T00:46:45.564589+0000","last_became_active":"2024-01-27T00:46:45.559927+0000","last_became_peered":"2024-01-27T00:46:45.559927+0000","last_unstale":"2024-01-27T00:46:45.564589+0000","last_undegraded":"2024-01-27T00:46:45.564589+0000","last_fullsized":"2024-01-27T00:46:45.564589+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:07:59.691769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.551423+0000","last_change":"2024-01-27T00:46:45.548738+0000","last_active":"2024-01-27T00:46:45.551423+0000","last_peered":"2024-01-27T00:46:45.551423+0000","last_clean":"2024-01-27T00:46:45.551423+0000","last_became_active":"2024-01-27T00:46:45.548152+0000","last_became_peered":"2024-01-27T00:46:45.548152+0000","last_unstale":"2024-01-27T00:46:45.551423+0000","last_undegraded":"2024-01-27T00:46:45.551423+0000","last_fullsized":"2024-01-27T00:46:45.551423+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:20:59.598654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574345+0000","last_change":"2024-01-27T00:46:45.568632+0000","last_active":"2024-01-27T00:46:45.574345+0000","last_peered":"2024-01-27T00:46:45.574345+0000","last_clean":"2024-01-27T00:46:45.574345+0000","last_became_active":"2024-01-27T00:46:45.568319+0000","last_became_peered":"2024-01-27T00:46:45.568319+0000","last_unstale":"2024-01-27T00:46:45.574345+0000","last_undegraded":"2024-01-27T00:46:45.574345+0000","last_fullsized":"2024-01-27T00:46:45.574345+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:09:24.182020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"174'4","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.575252+0000","last_change":"2024-01-27T00:46:44.551082+0000","last_active":"2024-01-27T00:46:45.575252+0000","last_peered":"2024-01-27T00:46:45.575252+0000","last_clean":"2024-01-27T00:46:45.575252+0000","last_became_active":"2024-01-27T00:46:44.550800+0000","last_became_peered":"2024-01-27T00:46:44.550800+0000","last_unstale":"2024-01-27T00:46:45.575252+0000","last_undegraded":"2024-01-27T00:46:45.575252+0000","last_fullsized":"2024-01-27T00:46:45.575252+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:34.205561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574409+0000","last_change":"2024-01-27T00:46:45.565405+0000","last_active":"2024-01-27T00:46:45.574409+0000","last_peered":"2024-01-27T00:46:45.574409+0000","last_clean":"2024-01-27T00:46:45.574409+0000","last_became_active":"2024-01-27T00:46:45.562543+0000","last_became_peered":"2024-01-27T00:46:45.562543+0000","last_unstale":"2024-01-27T00:46:45.574409+0000","last_undegraded":"2024-01-27T00:46:45.574409+0000","last_fullsized":"2024-01-27T00:46:45.574409+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:34.525936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"174'1","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.576835+0000","last_change":"2024-01-27T00:46:44.564397+0000","last_active":"2024-01-27T00:46:45.576835+0000","last_peered":"2024-01-27T00:46:45.576835+0000","last_clean":"2024-01-27T00:46:45.576835+0000","last_became_active":"2024-01-27T00:46:44.564118+0000","last_became_peered":"2024-01-27T00:46:44.564118+0000","last_unstale":"2024-01-27T00:46:45.576835+0000","last_undegraded":"2024-01-27T00:46:45.576835+0000","last_fullsized":"2024-01-27T00:46:45.576835+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:24.384971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.571497+0000","last_change":"2024-01-27T00:46:45.563691+0000","last_active":"2024-01-27T00:46:45.571497+0000","last_peered":"2024-01-27T00:46:45.571497+0000","last_clean":"2024-01-27T00:46:45.571497+0000","last_became_active":"2024-01-27T00:46:45.562302+0000","last_became_peered":"2024-01-27T00:46:45.562302+0000","last_unstale":"2024-01-27T00:46:45.571497+0000","last_undegraded":"2024-01-27T00:46:45.571497+0000","last_fullsized":"2024-01-27T00:46:45.571497+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:43:42.530709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"174'2","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.576770+0000","last_change":"2024-01-27T00:46:44.561860+0000","last_active":"2024-01-27T00:46:45.576770+0000","last_peered":"2024-01-27T00:46:45.576770+0000","last_clean":"2024-01-27T00:46:45.576770+0000","last_became_active":"2024-01-27T00:46:44.561579+0000","last_became_peered":"2024-01-27T00:46:44.561579+0000","last_unstale":"2024-01-27T00:46:45.576770+0000","last_undegraded":"2024-01-27T00:46:45.576770+0000","last_fullsized":"2024-01-27T00:46:45.576770+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:33.521039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566453+0000","last_change":"2024-01-27T00:46:45.562542+0000","last_active":"2024-01-27T00:46:45.566453+0000","last_peered":"2024-01-27T00:46:45.566453+0000","last_clean":"2024-01-27T00:46:45.566453+0000","last_became_active":"2024-01-27T00:46:45.562264+0000","last_became_peered":"2024-01-27T00:46:45.562264+0000","last_unstale":"2024-01-27T00:46:45.566453+0000","last_undegraded":"2024-01-27T00:46:45.566453+0000","last_fullsized":"2024-01-27T00:46:45.566453+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:57:28.818719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"174'2","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.577261+0000","last_change":"2024-01-27T00:46:44.559897+0000","last_active":"2024-01-27T00:46:45.577261+0000","last_peered":"2024-01-27T00:46:45.577261+0000","last_clean":"2024-01-27T00:46:45.577261+0000","last_became_active":"2024-01-27T00:46:44.559359+0000","last_became_peered":"2024-01-27T00:46:44.559359+0000","last_unstale":"2024-01-27T00:46:45.577261+0000","last_undegraded":"2024-01-27T00:46:45.577261+0000","last_fullsized":"2024-01-27T00:46:45.577261+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:12:25.853835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.550893+0000","last_change":"2024-01-27T00:46:45.547644+0000","last_active":"2024-01-27T00:46:45.550893+0000","last_peered":"2024-01-27T00:46:45.550893+0000","last_clean":"2024-01-27T00:46:45.550893+0000","last_became_active":"2024-01-27T00:46:45.547338+0000","last_became_peered":"2024-01-27T00:46:45.547338+0000","last_unstale":"2024-01-27T00:46:45.550893+0000","last_undegraded":"2024-01-27T00:46:45.550893+0000","last_fullsized":"2024-01-27T00:46:45.550893+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:34:01.102364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"174'1","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.575799+0000","last_change":"2024-01-27T00:46:44.559909+0000","last_active":"2024-01-27T00:46:45.575799+0000","last_peered":"2024-01-27T00:46:45.575799+0000","last_clean":"2024-01-27T00:46:45.575799+0000","last_became_active":"2024-01-27T00:46:44.559628+0000","last_became_peered":"2024-01-27T00:46:44.559628+0000","last_unstale":"2024-01-27T00:46:45.575799+0000","last_undegraded":"2024-01-27T00:46:45.575799+0000","last_fullsized":"2024-01-27T00:46:45.575799+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:12:25.853835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.573630+0000","last_change":"2024-01-27T00:46:45.568628+0000","last_active":"2024-01-27T00:46:45.573630+0000","last_peered":"2024-01-27T00:46:45.573630+0000","last_clean":"2024-01-27T00:46:45.573630+0000","last_became_active":"2024-01-27T00:46:45.568296+0000","last_became_peered":"2024-01-27T00:46:45.568296+0000","last_unstale":"2024-01-27T00:46:45.573630+0000","last_undegraded":"2024-01-27T00:46:45.573630+0000","last_fullsized":"2024-01-27T00:46:45.573630+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:56:30.192694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"174'1","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.573858+0000","last_change":"2024-01-27T00:46:44.555019+0000","last_active":"2024-01-27T00:46:45.573858+0000","last_peered":"2024-01-27T00:46:45.573858+0000","last_clean":"2024-01-27T00:46:45.573858+0000","last_became_active":"2024-01-27T00:46:44.554324+0000","last_became_peered":"2024-01-27T00:46:44.554324+0000","last_unstale":"2024-01-27T00:46:45.573858+0000","last_undegraded":"2024-01-27T00:46:45.573858+0000","last_fullsized":"2024-01-27T00:46:45.573858+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:00:26.604154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574814+0000","last_change":"2024-01-27T00:46:45.572035+0000","last_active":"2024-01-27T00:46:45.574814+0000","last_peered":"2024-01-27T00:46:45.574814+0000","last_clean":"2024-01-27T00:46:45.574814+0000","last_became_active":"2024-01-27T00:46:45.571751+0000","last_became_peered":"2024-01-27T00:46:45.571751+0000","last_unstale":"2024-01-27T00:46:45.574814+0000","last_undegraded":"2024-01-27T00:46:45.574814+0000","last_fullsized":"2024-01-27T00:46:45.574814+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:17:32.090542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.549984+0000","last_change":"2024-01-27T00:46:44.554494+0000","last_active":"2024-01-27T00:46:45.549984+0000","last_peered":"2024-01-27T00:46:45.549984+0000","last_clean":"2024-01-27T00:46:45.549984+0000","last_became_active":"2024-01-27T00:46:44.554249+0000","last_became_peered":"2024-01-27T00:46:44.554249+0000","last_unstale":"2024-01-27T00:46:45.549984+0000","last_undegraded":"2024-01-27T00:46:45.549984+0000","last_fullsized":"2024-01-27T00:46:45.549984+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:13:07.172249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.567713+0000","last_change":"2024-01-27T00:46:45.559360+0000","last_active":"2024-01-27T00:46:45.567713+0000","last_peered":"2024-01-27T00:46:45.567713+0000","last_clean":"2024-01-27T00:46:45.567713+0000","last_became_active":"2024-01-27T00:46:45.557778+0000","last_became_peered":"2024-01-27T00:46:45.557778+0000","last_unstale":"2024-01-27T00:46:45.567713+0000","last_undegraded":"2024-01-27T00:46:45.567713+0000","last_fullsized":"2024-01-27T00:46:45.567713+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:33:23.078082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.548933+0000","last_change":"2024-01-27T00:46:44.558303+0000","last_active":"2024-01-27T00:46:45.548933+0000","last_peered":"2024-01-27T00:46:45.548933+0000","last_clean":"2024-01-27T00:46:45.548933+0000","last_became_active":"2024-01-27T00:46:44.558020+0000","last_became_peered":"2024-01-27T00:46:44.558020+0000","last_unstale":"2024-01-27T00:46:45.548933+0000","last_undegraded":"2024-01-27T00:46:45.548933+0000","last_fullsized":"2024-01-27T00:46:45.548933+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:46:04.221575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.562194+0000","last_change":"2024-01-27T00:46:45.558862+0000","last_active":"2024-01-27T00:46:45.562194+0000","last_peered":"2024-01-27T00:46:45.562194+0000","last_clean":"2024-01-27T00:46:45.562194+0000","last_became_active":"2024-01-27T00:46:45.558585+0000","last_became_peered":"2024-01-27T00:46:45.558585+0000","last_unstale":"2024-01-27T00:46:45.562194+0000","last_undegraded":"2024-01-27T00:46:45.562194+0000","last_fullsized":"2024-01-27T00:46:45.562194+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:13:52.377253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.552835+0000","last_change":"2024-01-27T00:46:44.553830+0000","last_active":"2024-01-27T00:46:45.552835+0000","last_peered":"2024-01-27T00:46:45.552835+0000","last_clean":"2024-01-27T00:46:45.552835+0000","last_became_active":"2024-01-27T00:46:44.553537+0000","last_became_peered":"2024-01-27T00:46:44.553537+0000","last_unstale":"2024-01-27T00:46:45.552835+0000","last_undegraded":"2024-01-27T00:46:45.552835+0000","last_fullsized":"2024-01-27T00:46:45.552835+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:00:26.604154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.558787+0000","last_change":"2024-01-27T00:46:45.558061+0000","last_active":"2024-01-27T00:46:45.558787+0000","last_peered":"2024-01-27T00:46:45.558787+0000","last_clean":"2024-01-27T00:46:45.558787+0000","last_became_active":"2024-01-27T00:46:45.557727+0000","last_became_peered":"2024-01-27T00:46:45.557727+0000","last_unstale":"2024-01-27T00:46:45.558787+0000","last_undegraded":"2024-01-27T00:46:45.558787+0000","last_fullsized":"2024-01-27T00:46:45.558787+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:48:54.022424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.552388+0000","last_change":"2024-01-27T00:46:44.562077+0000","last_active":"2024-01-27T00:46:45.552388+0000","last_peered":"2024-01-27T00:46:45.552388+0000","last_clean":"2024-01-27T00:46:45.552388+0000","last_became_active":"2024-01-27T00:46:44.561803+0000","last_became_peered":"2024-01-27T00:46:44.561803+0000","last_unstale":"2024-01-27T00:46:45.552388+0000","last_undegraded":"2024-01-27T00:46:45.552388+0000","last_fullsized":"2024-01-27T00:46:45.552388+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:09:09.311997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.561937+0000","last_change":"2024-01-27T00:46:45.561307+0000","last_active":"2024-01-27T00:46:45.561937+0000","last_peered":"2024-01-27T00:46:45.561937+0000","last_clean":"2024-01-27T00:46:45.561937+0000","last_became_active":"2024-01-27T00:46:45.560472+0000","last_became_peered":"2024-01-27T00:46:45.560472+0000","last_unstale":"2024-01-27T00:46:45.561937+0000","last_undegraded":"2024-01-27T00:46:45.561937+0000","last_fullsized":"2024-01-27T00:46:45.561937+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:36:33.493714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"174'3","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.576689+0000","last_change":"2024-01-27T00:46:44.566003+0000","last_active":"2024-01-27T00:46:45.576689+0000","last_peered":"2024-01-27T00:46:45.576689+0000","last_clean":"2024-01-27T00:46:45.576689+0000","last_became_active":"2024-01-27T00:46:44.565642+0000","last_became_peered":"2024-01-27T00:46:44.565642+0000","last_unstale":"2024-01-27T00:46:45.576689+0000","last_undegraded":"2024-01-27T00:46:45.576689+0000","last_fullsized":"2024-01-27T00:46:45.576689+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:19:13.127951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.570302+0000","last_change":"2024-01-27T00:46:45.560093+0000","last_active":"2024-01-27T00:46:45.570302+0000","last_peered":"2024-01-27T00:46:45.570302+0000","last_clean":"2024-01-27T00:46:45.570302+0000","last_became_active":"2024-01-27T00:46:45.559827+0000","last_became_peered":"2024-01-27T00:46:45.559827+0000","last_unstale":"2024-01-27T00:46:45.570302+0000","last_undegraded":"2024-01-27T00:46:45.570302+0000","last_fullsized":"2024-01-27T00:46:45.570302+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:09:24.182020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"174'2","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.575127+0000","last_change":"2024-01-27T00:46:44.554097+0000","last_active":"2024-01-27T00:46:45.575127+0000","last_peered":"2024-01-27T00:46:45.575127+0000","last_clean":"2024-01-27T00:46:45.575127+0000","last_became_active":"2024-01-27T00:46:44.553773+0000","last_became_peered":"2024-01-27T00:46:44.553773+0000","last_unstale":"2024-01-27T00:46:45.575127+0000","last_undegraded":"2024-01-27T00:46:45.575127+0000","last_fullsized":"2024-01-27T00:46:45.575127+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:20:58.593757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574328+0000","last_change":"2024-01-27T00:46:45.563853+0000","last_active":"2024-01-27T00:46:45.574328+0000","last_peered":"2024-01-27T00:46:45.574328+0000","last_clean":"2024-01-27T00:46:45.574328+0000","last_became_active":"2024-01-27T00:46:45.563296+0000","last_became_peered":"2024-01-27T00:46:45.563296+0000","last_unstale":"2024-01-27T00:46:45.574328+0000","last_undegraded":"2024-01-27T00:46:45.574328+0000","last_fullsized":"2024-01-27T00:46:45.574328+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:20:59.598654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.547835+0000","last_change":"2024-01-27T00:46:44.560443+0000","last_active":"2024-01-27T00:46:45.547835+0000","last_peered":"2024-01-27T00:46:45.547835+0000","last_clean":"2024-01-27T00:46:45.547835+0000","last_became_active":"2024-01-27T00:46:44.559737+0000","last_became_peered":"2024-01-27T00:46:44.559737+0000","last_unstale":"2024-01-27T00:46:45.547835+0000","last_undegraded":"2024-01-27T00:46:45.547835+0000","last_fullsized":"2024-01-27T00:46:45.547835+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:35:53.183459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.564490+0000","last_change":"2024-01-27T00:46:45.558068+0000","last_active":"2024-01-27T00:46:45.564490+0000","last_peered":"2024-01-27T00:46:45.564490+0000","last_clean":"2024-01-27T00:46:45.564490+0000","last_became_active":"2024-01-27T00:46:45.557671+0000","last_became_peered":"2024-01-27T00:46:45.557671+0000","last_unstale":"2024-01-27T00:46:45.564490+0000","last_undegraded":"2024-01-27T00:46:45.564490+0000","last_fullsized":"2024-01-27T00:46:45.564490+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:10:24.979198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.547330+0000","last_change":"2024-01-27T00:46:44.562797+0000","last_active":"2024-01-27T00:46:45.547330+0000","last_peered":"2024-01-27T00:46:45.547330+0000","last_clean":"2024-01-27T00:46:45.547330+0000","last_became_active":"2024-01-27T00:46:44.562519+0000","last_became_peered":"2024-01-27T00:46:44.562519+0000","last_unstale":"2024-01-27T00:46:45.547330+0000","last_undegraded":"2024-01-27T00:46:45.547330+0000","last_fullsized":"2024-01-27T00:46:45.547330+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:34.205561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574864+0000","last_change":"2024-01-27T00:46:45.570359+0000","last_active":"2024-01-27T00:46:45.574864+0000","last_peered":"2024-01-27T00:46:45.574864+0000","last_clean":"2024-01-27T00:46:45.574864+0000","last_became_active":"2024-01-27T00:46:45.568395+0000","last_became_peered":"2024-01-27T00:46:45.568395+0000","last_unstale":"2024-01-27T00:46:45.574864+0000","last_undegraded":"2024-01-27T00:46:45.574864+0000","last_fullsized":"2024-01-27T00:46:45.574864+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:35.210458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"174'8","reported_seq":18,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.576921+0000","last_change":"2024-01-27T00:46:44.560114+0000","last_active":"2024-01-27T00:46:45.576921+0000","last_peered":"2024-01-27T00:46:45.576921+0000","last_clean":"2024-01-27T00:46:45.576921+0000","last_became_active":"2024-01-27T00:46:44.559240+0000","last_became_peered":"2024-01-27T00:46:44.559240+0000","last_unstale":"2024-01-27T00:46:45.576921+0000","last_undegraded":"2024-01-27T00:46:45.576921+0000","last_fullsized":"2024-01-27T00:46:45.576921+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:59:49.645505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566628+0000","last_change":"2024-01-27T00:46:45.565567+0000","last_active":"2024-01-27T00:46:45.566628+0000","last_peered":"2024-01-27T00:46:45.566628+0000","last_clean":"2024-01-27T00:46:45.566628+0000","last_became_active":"2024-01-27T00:46:45.565320+0000","last_became_peered":"2024-01-27T00:46:45.565320+0000","last_unstale":"2024-01-27T00:46:45.566628+0000","last_undegraded":"2024-01-27T00:46:45.566628+0000","last_fullsized":"2024-01-27T00:46:45.566628+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:42:33.340128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"174'1","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.577148+0000","last_change":"2024-01-27T00:46:44.556531+0000","last_active":"2024-01-27T00:46:45.577148+0000","last_peered":"2024-01-27T00:46:45.577148+0000","last_clean":"2024-01-27T00:46:45.577148+0000","last_became_active":"2024-01-27T00:46:44.555951+0000","last_became_peered":"2024-01-27T00:46:44.555951+0000","last_unstale":"2024-01-27T00:46:45.577148+0000","last_undegraded":"2024-01-27T00:46:45.577148+0000","last_fullsized":"2024-01-27T00:46:45.577148+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:06:08.035902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566575+0000","last_change":"2024-01-27T00:46:45.559036+0000","last_active":"2024-01-27T00:46:45.566575+0000","last_peered":"2024-01-27T00:46:45.566575+0000","last_clean":"2024-01-27T00:46:45.566575+0000","last_became_active":"2024-01-27T00:46:45.558406+0000","last_became_peered":"2024-01-27T00:46:45.558406+0000","last_unstale":"2024-01-27T00:46:45.566575+0000","last_undegraded":"2024-01-27T00:46:45.566575+0000","last_fullsized":"2024-01-27T00:46:45.566575+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:39:52.381793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"174'2","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.577381+0000","last_change":"2024-01-27T00:46:44.563657+0000","last_active":"2024-01-27T00:46:45.577381+0000","last_peered":"2024-01-27T00:46:45.577381+0000","last_clean":"2024-01-27T00:46:45.577381+0000","last_became_active":"2024-01-27T00:46:44.563390+0000","last_became_peered":"2024-01-27T00:46:44.563390+0000","last_unstale":"2024-01-27T00:46:45.577381+0000","last_undegraded":"2024-01-27T00:46:45.577381+0000","last_fullsized":"2024-01-27T00:46:45.577381+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:35:53.183459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.575950+0000","last_change":"2024-01-27T00:46:45.570906+0000","last_active":"2024-01-27T00:46:45.575950+0000","last_peered":"2024-01-27T00:46:45.575950+0000","last_clean":"2024-01-27T00:46:45.575950+0000","last_became_active":"2024-01-27T00:46:45.570653+0000","last_became_peered":"2024-01-27T00:46:45.570653+0000","last_unstale":"2024-01-27T00:46:45.575950+0000","last_undegraded":"2024-01-27T00:46:45.575950+0000","last_fullsized":"2024-01-27T00:46:45.575950+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:42:57.456320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"174'1","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.577097+0000","last_change":"2024-01-27T00:46:44.559419+0000","last_active":"2024-01-27T00:46:45.577097+0000","last_peered":"2024-01-27T00:46:45.577097+0000","last_clean":"2024-01-27T00:46:45.577097+0000","last_became_active":"2024-01-27T00:46:44.559138+0000","last_became_peered":"2024-01-27T00:46:44.559138+0000","last_unstale":"2024-01-27T00:46:45.577097+0000","last_undegraded":"2024-01-27T00:46:45.577097+0000","last_fullsized":"2024-01-27T00:46:45.577097+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:42:56.451424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.571248+0000","last_change":"2024-01-27T00:46:45.567184+0000","last_active":"2024-01-27T00:46:45.571248+0000","last_peered":"2024-01-27T00:46:45.571248+0000","last_clean":"2024-01-27T00:46:45.571248+0000","last_became_active":"2024-01-27T00:46:45.566901+0000","last_became_peered":"2024-01-27T00:46:45.566901+0000","last_unstale":"2024-01-27T00:46:45.571248+0000","last_undegraded":"2024-01-27T00:46:45.571248+0000","last_fullsized":"2024-01-27T00:46:45.571248+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:17:24.137209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.548227+0000","last_change":"2024-01-27T00:46:44.559439+0000","last_active":"2024-01-27T00:46:45.548227+0000","last_peered":"2024-01-27T00:46:45.548227+0000","last_clean":"2024-01-27T00:46:45.548227+0000","last_became_active":"2024-01-27T00:46:44.558933+0000","last_became_peered":"2024-01-27T00:46:44.558933+0000","last_unstale":"2024-01-27T00:46:45.548227+0000","last_undegraded":"2024-01-27T00:46:45.548227+0000","last_fullsized":"2024-01-27T00:46:45.548227+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:20:58.593757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566429+0000","last_change":"2024-01-27T00:46:45.561881+0000","last_active":"2024-01-27T00:46:45.566429+0000","last_peered":"2024-01-27T00:46:45.566429+0000","last_clean":"2024-01-27T00:46:45.566429+0000","last_became_active":"2024-01-27T00:46:45.561282+0000","last_became_peered":"2024-01-27T00:46:45.561282+0000","last_unstale":"2024-01-27T00:46:45.566429+0000","last_undegraded":"2024-01-27T00:46:45.566429+0000","last_fullsized":"2024-01-27T00:46:45.566429+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:56:30.192694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"174'2","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.576728+0000","last_change":"2024-01-27T00:46:44.560948+0000","last_active":"2024-01-27T00:46:45.576728+0000","last_peered":"2024-01-27T00:46:45.576728+0000","last_clean":"2024-01-27T00:46:45.576728+0000","last_became_active":"2024-01-27T00:46:44.560614+0000","last_became_peered":"2024-01-27T00:46:44.560614+0000","last_unstale":"2024-01-27T00:46:45.576728+0000","last_undegraded":"2024-01-27T00:46:45.576728+0000","last_fullsized":"2024-01-27T00:46:45.576728+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:14:25.383144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.573815+0000","last_change":"2024-01-27T00:46:45.563574+0000","last_active":"2024-01-27T00:46:45.573815+0000","last_peered":"2024-01-27T00:46:45.573815+0000","last_clean":"2024-01-27T00:46:45.573815+0000","last_became_active":"2024-01-27T00:46:45.562151+0000","last_became_peered":"2024-01-27T00:46:45.562151+0000","last_unstale":"2024-01-27T00:46:45.573815+0000","last_undegraded":"2024-01-27T00:46:45.573815+0000","last_fullsized":"2024-01-27T00:46:45.573815+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:24:03.306804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.545921+0000","last_change":"2024-01-27T00:46:44.559051+0000","last_active":"2024-01-27T00:46:45.545921+0000","last_peered":"2024-01-27T00:46:45.545921+0000","last_clean":"2024-01-27T00:46:45.545921+0000","last_became_active":"2024-01-27T00:46:44.558570+0000","last_became_peered":"2024-01-27T00:46:44.558570+0000","last_unstale":"2024-01-27T00:46:45.545921+0000","last_undegraded":"2024-01-27T00:46:45.545921+0000","last_fullsized":"2024-01-27T00:46:45.545921+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:42:56.451424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.565788+0000","last_change":"2024-01-27T00:46:45.557551+0000","last_active":"2024-01-27T00:46:45.565788+0000","last_peered":"2024-01-27T00:46:45.565788+0000","last_clean":"2024-01-27T00:46:45.565788+0000","last_became_active":"2024-01-27T00:46:45.557156+0000","last_became_peered":"2024-01-27T00:46:45.557156+0000","last_unstale":"2024-01-27T00:46:45.565788+0000","last_undegraded":"2024-01-27T00:46:45.565788+0000","last_fullsized":"2024-01-27T00:46:45.565788+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:37:11.759228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.542781+0000","last_change":"2024-01-27T00:46:44.558348+0000","last_active":"2024-01-27T00:46:45.542781+0000","last_peered":"2024-01-27T00:46:45.542781+0000","last_clean":"2024-01-27T00:46:45.542781+0000","last_became_active":"2024-01-27T00:46:44.558037+0000","last_became_peered":"2024-01-27T00:46:44.558037+0000","last_unstale":"2024-01-27T00:46:45.542781+0000","last_undegraded":"2024-01-27T00:46:45.542781+0000","last_fullsized":"2024-01-27T00:46:45.542781+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:44.267221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574844+0000","last_change":"2024-01-27T00:46:45.570182+0000","last_active":"2024-01-27T00:46:45.574844+0000","last_peered":"2024-01-27T00:46:45.574844+0000","last_clean":"2024-01-27T00:46:45.574844+0000","last_became_active":"2024-01-27T00:46:45.568002+0000","last_became_peered":"2024-01-27T00:46:45.568002+0000","last_unstale":"2024-01-27T00:46:45.574844+0000","last_undegraded":"2024-01-27T00:46:45.574844+0000","last_fullsized":"2024-01-27T00:46:45.574844+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:55.735857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.545279+0000","last_change":"2024-01-27T00:46:44.555197+0000","last_active":"2024-01-27T00:46:45.545279+0000","last_peered":"2024-01-27T00:46:45.545279+0000","last_clean":"2024-01-27T00:46:45.545279+0000","last_became_active":"2024-01-27T00:46:44.554948+0000","last_became_peered":"2024-01-27T00:46:44.554948+0000","last_unstale":"2024-01-27T00:46:45.545279+0000","last_undegraded":"2024-01-27T00:46:45.545279+0000","last_fullsized":"2024-01-27T00:46:45.545279+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:55:52.338720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.572165+0000","last_change":"2024-01-27T00:46:45.568251+0000","last_active":"2024-01-27T00:46:45.572165+0000","last_peered":"2024-01-27T00:46:45.572165+0000","last_clean":"2024-01-27T00:46:45.572165+0000","last_became_active":"2024-01-27T00:46:45.567023+0000","last_became_peered":"2024-01-27T00:46:45.567023+0000","last_unstale":"2024-01-27T00:46:45.572165+0000","last_undegraded":"2024-01-27T00:46:45.572165+0000","last_fullsized":"2024-01-27T00:46:45.572165+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:57:28.818719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"174'2","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574142+0000","last_change":"2024-01-27T00:46:44.559295+0000","last_active":"2024-01-27T00:46:45.574142+0000","last_peered":"2024-01-27T00:46:45.574142+0000","last_clean":"2024-01-27T00:46:45.574142+0000","last_became_active":"2024-01-27T00:46:44.558998+0000","last_became_peered":"2024-01-27T00:46:44.558998+0000","last_unstale":"2024-01-27T00:46:45.574142+0000","last_undegraded":"2024-01-27T00:46:45.574142+0000","last_fullsized":"2024-01-27T00:46:45.574142+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:46:04.221575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.567807+0000","last_change":"2024-01-27T00:46:45.565473+0000","last_active":"2024-01-27T00:46:45.567807+0000","last_peered":"2024-01-27T00:46:45.567807+0000","last_clean":"2024-01-27T00:46:45.567807+0000","last_became_active":"2024-01-27T00:46:45.565226+0000","last_became_peered":"2024-01-27T00:46:45.565226+0000","last_unstale":"2024-01-27T00:46:45.567807+0000","last_undegraded":"2024-01-27T00:46:45.567807+0000","last_fullsized":"2024-01-27T00:46:45.567807+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:37:11.759228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.552294+0000","last_change":"2024-01-27T00:46:44.557760+0000","last_active":"2024-01-27T00:46:45.552294+0000","last_peered":"2024-01-27T00:46:45.552294+0000","last_clean":"2024-01-27T00:46:45.552294+0000","last_became_active":"2024-01-27T00:46:44.557025+0000","last_became_peered":"2024-01-27T00:46:44.557025+0000","last_unstale":"2024-01-27T00:46:45.552294+0000","last_undegraded":"2024-01-27T00:46:45.552294+0000","last_fullsized":"2024-01-27T00:46:45.552294+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:59:16.567278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566375+0000","last_change":"2024-01-27T00:46:45.558408+0000","last_active":"2024-01-27T00:46:45.566375+0000","last_peered":"2024-01-27T00:46:45.566375+0000","last_clean":"2024-01-27T00:46:45.566375+0000","last_became_active":"2024-01-27T00:46:45.557856+0000","last_became_peered":"2024-01-27T00:46:45.557856+0000","last_unstale":"2024-01-27T00:46:45.566375+0000","last_undegraded":"2024-01-27T00:46:45.566375+0000","last_fullsized":"2024-01-27T00:46:45.566375+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:13.453682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.549600+0000","last_change":"2024-01-27T00:46:44.559542+0000","last_active":"2024-01-27T00:46:45.549600+0000","last_peered":"2024-01-27T00:46:45.549600+0000","last_clean":"2024-01-27T00:46:45.549600+0000","last_became_active":"2024-01-27T00:46:44.559213+0000","last_became_peered":"2024-01-27T00:46:44.559213+0000","last_unstale":"2024-01-27T00:46:45.549600+0000","last_undegraded":"2024-01-27T00:46:45.549600+0000","last_fullsized":"2024-01-27T00:46:45.549600+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:55:52.338720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.575672+0000","last_change":"2024-01-27T00:46:45.571078+0000","last_active":"2024-01-27T00:46:45.575672+0000","last_peered":"2024-01-27T00:46:45.575672+0000","last_clean":"2024-01-27T00:46:45.575672+0000","last_became_active":"2024-01-27T00:46:45.570737+0000","last_became_peered":"2024-01-27T00:46:45.570737+0000","last_unstale":"2024-01-27T00:46:45.575672+0000","last_undegraded":"2024-01-27T00:46:45.575672+0000","last_fullsized":"2024-01-27T00:46:45.575672+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:18.466742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.552814+0000","last_change":"2024-01-27T00:46:44.558997+0000","last_active":"2024-01-27T00:46:45.552814+0000","last_peered":"2024-01-27T00:46:45.552814+0000","last_clean":"2024-01-27T00:46:45.552814+0000","last_became_active":"2024-01-27T00:46:44.558732+0000","last_became_peered":"2024-01-27T00:46:44.558732+0000","last_unstale":"2024-01-27T00:46:45.552814+0000","last_undegraded":"2024-01-27T00:46:45.552814+0000","last_fullsized":"2024-01-27T00:46:45.552814+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:35:53.183459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.555465+0000","last_change":"2024-01-27T00:46:45.553448+0000","last_active":"2024-01-27T00:46:45.555465+0000","last_peered":"2024-01-27T00:46:45.555465+0000","last_clean":"2024-01-27T00:46:45.555465+0000","last_became_active":"2024-01-27T00:46:45.553148+0000","last_became_peered":"2024-01-27T00:46:45.553148+0000","last_unstale":"2024-01-27T00:46:45.555465+0000","last_undegraded":"2024-01-27T00:46:45.555465+0000","last_fullsized":"2024-01-27T00:46:45.555465+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:37:09.425394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.549267+0000","last_change":"2024-01-27T00:46:44.558370+0000","last_active":"2024-01-27T00:46:45.549267+0000","last_peered":"2024-01-27T00:46:45.549267+0000","last_clean":"2024-01-27T00:46:45.549267+0000","last_became_active":"2024-01-27T00:46:44.558093+0000","last_became_peered":"2024-01-27T00:46:44.558093+0000","last_unstale":"2024-01-27T00:46:45.549267+0000","last_undegraded":"2024-01-27T00:46:45.549267+0000","last_fullsized":"2024-01-27T00:46:45.549267+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:48:53.017528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.567834+0000","last_change":"2024-01-27T00:46:45.560921+0000","last_active":"2024-01-27T00:46:45.567834+0000","last_peered":"2024-01-27T00:46:45.567834+0000","last_clean":"2024-01-27T00:46:45.567834+0000","last_became_active":"2024-01-27T00:46:45.560664+0000","last_became_peered":"2024-01-27T00:46:45.560664+0000","last_unstale":"2024-01-27T00:46:45.567834+0000","last_undegraded":"2024-01-27T00:46:45.567834+0000","last_fullsized":"2024-01-27T00:46:45.567834+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:09:01.270842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.543666+0000","last_change":"2024-01-27T00:46:44.558090+0000","last_active":"2024-01-27T00:46:45.543666+0000","last_peered":"2024-01-27T00:46:45.543666+0000","last_clean":"2024-01-27T00:46:45.543666+0000","last_became_active":"2024-01-27T00:46:44.557799+0000","last_became_peered":"2024-01-27T00:46:44.557799+0000","last_unstale":"2024-01-27T00:46:45.543666+0000","last_undegraded":"2024-01-27T00:46:45.543666+0000","last_fullsized":"2024-01-27T00:46:45.543666+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:12:01.551412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574376+0000","last_change":"2024-01-27T00:46:45.569203+0000","last_active":"2024-01-27T00:46:45.574376+0000","last_peered":"2024-01-27T00:46:45.574376+0000","last_clean":"2024-01-27T00:46:45.574376+0000","last_became_active":"2024-01-27T00:46:45.568932+0000","last_became_peered":"2024-01-27T00:46:45.568932+0000","last_unstale":"2024-01-27T00:46:45.574376+0000","last_undegraded":"2024-01-27T00:46:45.574376+0000","last_fullsized":"2024-01-27T00:46:45.574376+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:00:27.609050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.549433+0000","last_change":"2024-01-27T00:46:44.561365+0000","last_active":"2024-01-27T00:46:45.549433+0000","last_peered":"2024-01-27T00:46:45.549433+0000","last_clean":"2024-01-27T00:46:45.549433+0000","last_became_active":"2024-01-27T00:46:44.561061+0000","last_became_peered":"2024-01-27T00:46:44.561061+0000","last_unstale":"2024-01-27T00:46:45.549433+0000","last_undegraded":"2024-01-27T00:46:45.549433+0000","last_fullsized":"2024-01-27T00:46:45.549433+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:58:18.943567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.569800+0000","last_change":"2024-01-27T00:46:45.559506+0000","last_active":"2024-01-27T00:46:45.569800+0000","last_peered":"2024-01-27T00:46:45.569800+0000","last_clean":"2024-01-27T00:46:45.569800+0000","last_became_active":"2024-01-27T00:46:45.559227+0000","last_became_peered":"2024-01-27T00:46:45.559227+0000","last_unstale":"2024-01-27T00:46:45.569800+0000","last_undegraded":"2024-01-27T00:46:45.569800+0000","last_fullsized":"2024-01-27T00:46:45.569800+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:46:05.226471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.554257+0000","last_change":"2024-01-27T00:46:44.562886+0000","last_active":"2024-01-27T00:46:45.554257+0000","last_peered":"2024-01-27T00:46:45.554257+0000","last_clean":"2024-01-27T00:46:45.554257+0000","last_became_active":"2024-01-27T00:46:44.562596+0000","last_became_peered":"2024-01-27T00:46:44.562596+0000","last_unstale":"2024-01-27T00:46:45.554257+0000","last_undegraded":"2024-01-27T00:46:45.554257+0000","last_fullsized":"2024-01-27T00:46:45.554257+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:20:58.593757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.570630+0000","last_change":"2024-01-27T00:46:45.563329+0000","last_active":"2024-01-27T00:46:45.570630+0000","last_peered":"2024-01-27T00:46:45.570630+0000","last_clean":"2024-01-27T00:46:45.570630+0000","last_became_active":"2024-01-27T00:46:45.563031+0000","last_became_peered":"2024-01-27T00:46:45.563031+0000","last_unstale":"2024-01-27T00:46:45.570630+0000","last_undegraded":"2024-01-27T00:46:45.570630+0000","last_fullsized":"2024-01-27T00:46:45.570630+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:23:24.863025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"174'6","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.578394+0000","last_change":"2024-01-27T00:46:44.560987+0000","last_active":"2024-01-27T00:46:45.578394+0000","last_peered":"2024-01-27T00:46:45.578394+0000","last_clean":"2024-01-27T00:46:45.578394+0000","last_became_active":"2024-01-27T00:46:44.560651+0000","last_became_peered":"2024-01-27T00:46:44.560651+0000","last_unstale":"2024-01-27T00:46:45.578394+0000","last_undegraded":"2024-01-27T00:46:45.578394+0000","last_fullsized":"2024-01-27T00:46:45.578394+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:36:05.165696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"165'8673","reported_seq":9348,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.554345+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T00:46:45.554345+0000","last_peered":"2024-01-27T00:46:45.554345+0000","last_clean":"2024-01-27T00:46:45.554345+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T00:46:45.554345+0000","last_undegraded":"2024-01-27T00:46:45.554345+0000","last_fullsized":"2024-01-27T00:46:45.554345+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8673,"log_dups_size":0,"ondisk_log_size":8673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1592,"num_read_kb":3366,"num_write":490,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1592,"num_read_kb":3366,"num_write":490,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8673,"ondisk_log_size":8673,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575770,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316392,"kb_used_data":2720,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055448,"statfs":{"total":96636764160,"available":96312778752,"internally_reserved":0,"allocated":2785280,"data_stored":1727256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60799999999999998,"15min":0.60699999999999998},"min":{"1min":0.36199999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75800000000000001,"5min":0.83199999999999996,"15min":0.86899999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61299999999999999,"15min":0.61799999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.755,"5min":0.82099999999999995,"15min":0.90000000000000002},"last":0.76300000000000001}]},{"osd":1,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.34599999999999997},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.90400000000000003},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63500000000000001,"15min":0.63},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.38200000000000001},"max":{"1min":0.76700000000000002,"5min":0.85099999999999998,"15min":0.875},"last":0.79600000000000004}]},{"osd":2,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.374,"5min":0.36299999999999999,"15min":0.34000000000000002},"max":{"1min":0.76400000000000001,"5min":0.77700000000000002,"15min":0.89700000000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63800000000000001,"15min":0.63700000000000001},"min":{"1min":0.49399999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75600000000000001,"5min":0.86199999999999999,"15min":0.94199999999999995},"last":0.81100000000000005}]},{"osd":3,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.626,"15min":0.623},"min":{"1min":0.435,"5min":0.40999999999999998,"15min":0.39400000000000002},"max":{"1min":0.79200000000000004,"5min":0.86499999999999999,"15min":0.92600000000000005},"last":0.745},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.77700000000000002,"5min":0.83499999999999996,"15min":0.88400000000000001},"last":0.67600000000000005}]},{"osd":4,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49399999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.32900000000000001,"15min":0.29699999999999999},"max":{"1min":0.61699999999999999,"5min":0.86499999999999999,"15min":0.92000000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.34599999999999997},"max":{"1min":0.76600000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.51700000000000002}]},{"osd":5,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":0.67100000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.627},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":0.67700000000000005,"5min":0.89100000000000001,"15min":0.95699999999999996},"last":0.53700000000000003}]},{"osd":6,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59299999999999997,"15min":0.58299999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.36599999999999999},"max":{"1min":0.69299999999999995,"5min":0.91700000000000004,"15min":0.99399999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.36699999999999999},"max":{"1min":0.73599999999999999,"5min":0.82299999999999995,"15min":0.878},"last":0.56200000000000006}]}]},{"osd":6,"up_from":13,"seq":55834575770,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307620,"kb_used_data":2140,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064220,"statfs":{"total":96636764160,"available":96321761280,"internally_reserved":0,"allocated":2191360,"data_stored":1136936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.60599999999999998,"15min":0.61099999999999999},"min":{"1min":0.38700000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78200000000000003,"5min":0.82199999999999995,"15min":0.84899999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.40999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.43099999999999999}]},{"osd":1,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59999999999999998,"15min":0.59599999999999997},"min":{"1min":0.438,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78700000000000003,"5min":0.79800000000000004,"15min":0.80800000000000005},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61699999999999999,"15min":0.624},"min":{"1min":0.51200000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81999999999999995,"5min":0.87,"15min":0.87},"last":0.63800000000000001}]},{"osd":2,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42199999999999999,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.60599999999999998,"15min":0.60999999999999999},"min":{"1min":0.49099999999999999,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81799999999999995,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.623}]},{"osd":3,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.41999999999999998},"max":{"1min":0.83899999999999997,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.52000000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.79900000000000004,"5min":0.86099999999999999,"15min":0.879},"last":0.57199999999999995}]},{"osd":4,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.497,"15min":0.48899999999999999},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.313},"max":{"1min":0.69099999999999995,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53100000000000003,"15min":0.51700000000000002},"min":{"1min":0.39600000000000002,"5min":0.36699999999999999,"15min":0.32100000000000001},"max":{"1min":0.71399999999999997,"5min":0.79100000000000004,"15min":0.83299999999999996},"last":0.64800000000000002}]},{"osd":5,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52100000000000002,"15min":0.51000000000000001},"min":{"1min":0.39900000000000002,"5min":0.35099999999999998,"15min":0.32200000000000001},"max":{"1min":0.746,"5min":0.75900000000000001,"15min":0.79500000000000004},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56799999999999995,"15min":0.56000000000000005},"min":{"1min":0.42799999999999999,"5min":0.38300000000000001,"15min":0.34899999999999998},"max":{"1min":0.73799999999999999,"5min":0.86199999999999999,"15min":0.88800000000000001},"last":0.67500000000000004}]},{"osd":7,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.58499999999999996,"15min":0.56599999999999995},"min":{"1min":0.39200000000000002,"5min":0.38400000000000001,"15min":0.36799999999999999},"max":{"1min":0.77700000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.59299999999999997,"15min":0.57999999999999996},"min":{"1min":0.45100000000000001,"5min":0.40100000000000002,"15min":0.33600000000000002},"max":{"1min":0.77600000000000002,"5min":0.84599999999999997,"15min":0.85999999999999999},"last":0.68700000000000006}]}]},{"osd":1,"up_from":13,"seq":55834575770,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311716,"kb_used_data":2140,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060124,"statfs":{"total":96636764160,"available":96317566976,"internally_reserved":0,"allocated":2191360,"data_stored":1136936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.28699999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56299999999999994,"5min":0.70499999999999996,"15min":0.71399999999999997},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.44400000000000001,"15min":0.44700000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.66200000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.42999999999999999}]},{"osd":2,"last update":"Sat Jan 27 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.41399999999999998,"15min":0.42499999999999999},"min":{"1min":0.308,"5min":0.28699999999999998,"15min":0.26700000000000002},"max":{"1min":0.54000000000000004,"5min":0.59999999999999998,"15min":0.68899999999999995},"last":0.316},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48199999999999998,"15min":0.49399999999999999},"min":{"1min":0.31,"5min":0.29999999999999999,"15min":0.27900000000000003},"max":{"1min":0.70999999999999996,"5min":0.71799999999999997,"15min":0.76700000000000002},"last":0.46200000000000002}]},{"osd":3,"last update":"Sat Jan 27 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.499,"15min":0.52200000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.66200000000000003,"5min":0.69399999999999995,"15min":1.0469999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.49399999999999999,"15min":0.503},"min":{"1min":0.34300000000000003,"5min":0.32900000000000001,"15min":0.309},"max":{"1min":0.59299999999999997,"5min":0.72599999999999998,"15min":0.95599999999999996},"last":0.5}]},{"osd":4,"last update":"Sat Jan 27 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67200000000000004,"15min":0.68300000000000005},"min":{"1min":0.44700000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.02,"5min":1.02,"15min":1.3129999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68000000000000005,"15min":0.70199999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.097},"last":0.64200000000000002}]},{"osd":5,"last update":"Sat Jan 27 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.67200000000000004,"15min":0.69499999999999995},"min":{"1min":0.48099999999999998,"5min":0.47499999999999998,"15min":0.46300000000000002},"max":{"1min":0.90300000000000002,"5min":0.93100000000000005,"15min":1.091},"last":0.625},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69199999999999995,"15min":0.70499999999999996},"min":{"1min":0.49299999999999999,"5min":0.48199999999999998,"15min":0.36299999999999999},"max":{"1min":0.89200000000000002,"5min":0.96699999999999997,"15min":1.278},"last":0.66400000000000003}]},{"osd":6,"last update":"Sat Jan 27 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.68899999999999995,"15min":0.71499999999999997},"min":{"1min":0.48999999999999999,"5min":0.437,"15min":0.435},"max":{"1min":0.83999999999999997,"5min":0.96999999999999997,"15min":1.141},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.70599999999999996,"15min":0.72199999999999998},"min":{"1min":0.46700000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.96499999999999997,"5min":1.0229999999999999,"15min":1.1120000000000001},"last":0.51600000000000001}]},{"osd":7,"last update":"Sat Jan 27 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.72999999999999998,"15min":0.749},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.51100000000000001},"max":{"1min":0.97299999999999998,"5min":1.054,"15min":1.1819999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.74399999999999999,"15min":0.76200000000000001},"min":{"1min":0.54100000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.98599999999999999,"5min":0.98999999999999999,"15min":1.198},"last":0.68000000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575769,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312316,"kb_used_data":2740,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059524,"statfs":{"total":96636764160,"available":96316952576,"internally_reserved":0,"allocated":2805760,"data_stored":1729448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.41099999999999998,"15min":0.40999999999999998},"min":{"1min":0.316,"5min":0.25700000000000001,"15min":0.25},"max":{"1min":0.56699999999999995,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.45900000000000002,"15min":0.46300000000000002},"min":{"1min":0.28399999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.66100000000000003,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Jan 27 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.41399999999999998,"15min":0.42499999999999999},"min":{"1min":0.29999999999999999,"5min":0.28599999999999998,"15min":0.26700000000000002},"max":{"1min":0.60099999999999998,"5min":0.64300000000000002,"15min":0.69399999999999995},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.47799999999999998,"15min":0.48299999999999998},"min":{"1min":0.36499999999999999,"5min":0.31,"15min":0.307},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.72399999999999998},"last":0.25600000000000001}]},{"osd":3,"last update":"Sat Jan 27 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48699999999999999,"15min":0.48499999999999999},"min":{"1min":0.32300000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64400000000000002,"5min":0.755,"15min":0.755},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.49199999999999999,"15min":0.497},"min":{"1min":0.39000000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.69699999999999995},"last":0.33800000000000002}]},{"osd":4,"last update":"Sat Jan 27 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65800000000000003,"15min":0.68400000000000005},"min":{"1min":0.53300000000000003,"5min":0.44700000000000001,"15min":0.44400000000000001},"max":{"1min":0.91900000000000004,"5min":1.0349999999999999,"15min":1.125},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66400000000000003,"15min":0.68200000000000005},"min":{"1min":0.54200000000000004,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.94399999999999995,"5min":0.97599999999999998,"15min":1.0169999999999999},"last":0.58899999999999997}]},{"osd":5,"last update":"Sat Jan 27 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68400000000000005,"15min":0.70099999999999996},"min":{"1min":0.52500000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":1.054},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.71199999999999997,"15min":0.72099999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.47199999999999998},"max":{"1min":0.97799999999999998,"5min":1.056,"15min":1.056},"last":0.68200000000000005}]},{"osd":6,"last update":"Sat Jan 27 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.70499999999999996,"15min":0.71399999999999997},"min":{"1min":0.50700000000000001,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.89900000000000002,"5min":1.0449999999999999,"15min":1.087},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.72299999999999998,"15min":0.74199999999999999},"min":{"1min":0.48999999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.995,"5min":1.0349999999999999,"15min":1.052},"last":0.79200000000000004}]},{"osd":7,"last update":"Sat Jan 27 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.74199999999999999,"15min":0.75900000000000001},"min":{"1min":0.55100000000000005,"5min":0.505,"15min":0.505},"max":{"1min":0.96199999999999997,"5min":1.0880000000000001,"15min":1.1080000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.76600000000000001,"15min":0.78300000000000003},"min":{"1min":0.58499999999999996,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0249999999999999,"5min":1.071,"15min":1.071},"last":0.78100000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575769,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307624,"kb_used_data":2144,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064216,"statfs":{"total":96636764160,"available":96321757184,"internally_reserved":0,"allocated":2195456,"data_stored":1136978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38800000000000001,"15min":0.39800000000000002},"min":{"1min":0.28199999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.59299999999999997,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42999999999999999,"15min":0.44400000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.55000000000000004,"5min":0.66000000000000003,"15min":0.71899999999999997},"last":0.41399999999999998}]},{"osd":1,"last update":"Sat Jan 27 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.42299999999999999,"15min":0.434},"min":{"1min":0.28599999999999998,"5min":0.27700000000000002,"15min":0.27300000000000002},"max":{"1min":0.56799999999999995,"5min":0.627,"15min":0.68300000000000005},"last":0.255},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51100000000000001,"15min":0.51600000000000001},"min":{"1min":0.42599999999999999,"5min":0.38600000000000001,"15min":0.35399999999999998},"max":{"1min":0.64400000000000002,"5min":0.76400000000000001,"15min":0.80300000000000005},"last":0.46100000000000002}]},{"osd":3,"last update":"Sat Jan 27 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48099999999999998,"15min":0.49299999999999999},"min":{"1min":0.35199999999999998,"5min":0.34100000000000003,"15min":0.33100000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47799999999999998,"15min":0.48999999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.29499999999999998},"max":{"1min":0.72499999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.55400000000000005}]},{"osd":4,"last update":"Sat Jan 27 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65800000000000003,"15min":0.67000000000000004},"min":{"1min":0.48899999999999999,"5min":0.44900000000000001,"15min":0.41999999999999998},"max":{"1min":0.85899999999999999,"5min":0.93300000000000005,"15min":1.0660000000000001},"last":0.626},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.67100000000000004,"15min":0.68600000000000005},"min":{"1min":0.48299999999999998,"5min":0.433,"15min":0.433},"max":{"1min":0.79100000000000004,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.48599999999999999}]},{"osd":5,"last update":"Sat Jan 27 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.68000000000000005,"15min":0.69299999999999995},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.42699999999999999},"max":{"1min":0.77100000000000002,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.68999999999999995,"15min":0.70099999999999996},"min":{"1min":0.51300000000000001,"5min":0.46899999999999997,"15min":0.45900000000000002},"max":{"1min":0.89700000000000002,"5min":0.98799999999999999,"15min":1.044},"last":0.44}]},{"osd":6,"last update":"Sat Jan 27 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68700000000000006,"15min":0.70099999999999996},"min":{"1min":0.499,"5min":0.49099999999999999,"15min":0.47599999999999998},"max":{"1min":0.97399999999999998,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.497},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.70099999999999996,"15min":0.71399999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.93400000000000005,"5min":1.002,"15min":1.0720000000000001},"last":0.66400000000000003}]},{"osd":7,"last update":"Sat Jan 27 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.71799999999999997,"15min":0.73199999999999998},"min":{"1min":0.52700000000000002,"5min":0.50900000000000001,"15min":0.48999999999999999},"max":{"1min":0.93600000000000005,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.747,"15min":0.76300000000000001},"min":{"1min":0.56100000000000005,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0069999999999999,"5min":1.022,"15min":1.1299999999999999},"last":0.57899999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575768,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315816,"kb_used_data":2144,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94056024,"statfs":{"total":96636764160,"available":96313368576,"internally_reserved":0,"allocated":2195456,"data_stored":1137454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40100000000000002,"15min":0.40400000000000003},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.56299999999999994,"5min":0.63,"15min":0.752},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.35499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60199999999999998,"5min":0.71399999999999997,"15min":0.85699999999999998},"last":0.42799999999999999}]},{"osd":1,"last update":"Sat Jan 27 00:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45500000000000002,"15min":0.46700000000000003},"min":{"1min":0.32800000000000001,"5min":0.29699999999999999,"15min":0.27400000000000002},"max":{"1min":0.69499999999999995,"5min":0.69699999999999995,"15min":0.94599999999999995},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.498,"15min":0.5},"min":{"1min":0.38,"5min":0.33400000000000002,"15min":0.316},"max":{"1min":0.65700000000000003,"5min":0.745,"15min":0.745},"last":0.40500000000000003}]},{"osd":2,"last update":"Sat Jan 27 00:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49099999999999999,"15min":0.497},"min":{"1min":0.35899999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.621,"5min":0.67900000000000005,"15min":0.84499999999999997},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.46400000000000002,"15min":0.47399999999999998},"min":{"1min":0.32000000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.82299999999999995},"last":0.35399999999999998}]},{"osd":4,"last update":"Sat Jan 27 00:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69199999999999995},"min":{"1min":0.56699999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.88,"5min":0.89400000000000002,"15min":1},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.47699999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.61299999999999999}]},{"osd":5,"last update":"Sat Jan 27 00:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.69899999999999995,"15min":0.69199999999999995},"min":{"1min":0.51300000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.84999999999999998,"5min":0.97799999999999998,"15min":1.0429999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68999999999999995,"15min":0.69099999999999995},"min":{"1min":0.48499999999999999,"5min":0.44800000000000001,"15min":0.442},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":1.0289999999999999},"last":0.51800000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.68700000000000006,"15min":0.69399999999999995},"min":{"1min":0.60799999999999998,"5min":0.52000000000000002,"15min":0.48899999999999999},"max":{"1min":0.96599999999999997,"5min":0.97199999999999998,"15min":1.0820000000000001},"last":0.623},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.72199999999999998,"15min":0.72099999999999997},"min":{"1min":0.55000000000000004,"5min":0.499,"15min":0.44},"max":{"1min":0.94699999999999995,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.58499999999999996}]},{"osd":7,"last update":"Sat Jan 27 00:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71099999999999997,"15min":0.72099999999999997},"min":{"1min":0.54100000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.76500000000000001,"15min":0.76300000000000001},"min":{"1min":0.63100000000000001,"5min":0.48999999999999999,"15min":0.46300000000000002},"max":{"1min":1.012,"5min":1.0229999999999999,"15min":1.1240000000000001},"last":0.55800000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575770,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311724,"kb_used_data":2148,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060116,"statfs":{"total":96636764160,"available":96317558784,"internally_reserved":0,"allocated":2199552,"data_stored":1138402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61399999999999999,"15min":0.60599999999999998},"min":{"1min":0.40699999999999997,"5min":0.39300000000000002,"15min":0.32100000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.57799999999999996}]},{"osd":1,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.627,"15min":0.628},"min":{"1min":0.52900000000000003,"5min":0.42499999999999999,"15min":0.40999999999999998},"max":{"1min":0.75700000000000001,"5min":0.83199999999999996,"15min":0.86299999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61899999999999999,"15min":0.626},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84699999999999998},"last":0.61199999999999999}]},{"osd":2,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.46100000000000002,"5min":0.436,"15min":0.40899999999999997},"max":{"1min":0.753,"5min":0.81899999999999995,"15min":0.84499999999999997},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.5,"5min":0.44400000000000001,"15min":0.38100000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.63500000000000001}]},{"osd":3,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.626,"15min":0.627},"min":{"1min":0.45900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.61899999999999999,"15min":0.623},"min":{"1min":0.42599999999999999,"5min":0.41199999999999998,"15min":0.41099999999999998},"max":{"1min":0.80600000000000005,"5min":0.80900000000000005,"15min":0.876},"last":0.59799999999999998}]},{"osd":5,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48199999999999998,"15min":0.48499999999999999},"min":{"1min":0.35899999999999999,"5min":0.32100000000000001,"15min":0.28699999999999998},"max":{"1min":0.69699999999999995,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.33100000000000002,"5min":0.30499999999999999,"15min":0.30199999999999999},"max":{"1min":0.77300000000000002,"5min":0.876,"15min":0.876},"last":0.45400000000000001}]},{"osd":6,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49299999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.54100000000000004},"min":{"1min":0.39800000000000002,"5min":0.34300000000000003,"15min":0.32800000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.49199999999999999}]},{"osd":7,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41599999999999998,"5min":0.34999999999999998,"15min":0.33700000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.46000000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.64400000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575770,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307616,"kb_used_data":2136,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064224,"statfs":{"total":96636764160,"available":96321765376,"internally_reserved":0,"allocated":2187264,"data_stored":1136866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61699999999999999,"15min":0.61099999999999999},"min":{"1min":0.42499999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78300000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.628,"15min":0.624},"min":{"1min":0.54100000000000004,"5min":0.41299999999999998,"15min":0.38600000000000001},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.629}]},{"osd":1,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.66200000000000003,"15min":0.65200000000000002},"min":{"1min":0.60099999999999998,"5min":0.47599999999999998,"15min":0.40200000000000002},"max":{"1min":0.81100000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.67900000000000005,"15min":0.67500000000000004},"min":{"1min":0.53500000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.60499999999999998}]},{"osd":2,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.628,"15min":0.60799999999999998},"min":{"1min":0.53400000000000003,"5min":0.39700000000000002,"15min":0.39100000000000001},"max":{"1min":0.78400000000000003,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.623,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.51900000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85599999999999998,"5min":0.876,"15min":0.876},"last":0.38}]},{"osd":3,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.60699999999999998},"min":{"1min":0.45000000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78500000000000003,"5min":0.82799999999999996,"15min":0.83499999999999996},"last":0.499},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65000000000000002,"15min":0.63500000000000001},"min":{"1min":0.53800000000000003,"5min":0.436,"15min":0.42399999999999999},"max":{"1min":0.87,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.56999999999999995}]},{"osd":4,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48499999999999999,"15min":0.48099999999999998},"min":{"1min":0.36799999999999999,"5min":0.30399999999999999,"15min":0.30299999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.39400000000000002,"5min":0.33400000000000002,"15min":0.32700000000000001},"max":{"1min":0.72899999999999998,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55200000000000005}]},{"osd":6,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51400000000000001,"15min":0.504},"min":{"1min":0.40400000000000003,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.68700000000000006,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.41999999999999998,"5min":0.36799999999999999,"15min":0.36099999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.44900000000000001}]},{"osd":7,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57499999999999996,"15min":0.56699999999999995},"min":{"1min":0.38800000000000001,"5min":0.374,"15min":0.34699999999999998},"max":{"1min":0.83899999999999997,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.61199999999999999,"15min":0.60699999999999998},"min":{"1min":0.45400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.82599999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.61899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T00:46:49.746 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T00:46:49.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.916+0000 7f641450c700 1 Processor -- start 2024-01-27T00:46:49.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.916+0000 7f641450c700 1 -- start start 2024-01-27T00:46:49.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.917+0000 7f641450c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f640c157920 0x7f640c0b2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:49.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.918+0000 7f641450c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f640c0b2a60 0x7f640c0cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:49.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.918+0000 7f641450c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f640c0cc550 con 0x7f640c157550 2024-01-27T00:46:49.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.918+0000 7f641450c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f640c0ca020 con 0x7f640c0b2a60 2024-01-27T00:46:49.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.918+0000 7f641450c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640c0b2e90 con 0x7f640c157920 2024-01-27T00:46:49.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.918+0000 7f6412aa9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f640c0b2a60 0x7f640c0cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:49.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.918+0000 7f6411aa7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f640c157920 0x7f640c0b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:49.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.918+0000 7f6411aa7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f640c157920 0x7f640c0b2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:44530/0 (socket says 172.21.15.46:44530) 2024-01-27T00:46:49.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.918+0000 7f6412aa9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f640c0b2a60 0x7f640c0cc9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:54632/0 (socket says 172.21.15.46:54632) 2024-01-27T00:46:49.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.918+0000 7f6411aa7700 1 -- 172.21.15.46:0/2587846049 learned_addr learned my addr 172.21.15.46:0/2587846049 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:49.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.918+0000 7f6411aa7700 1 -- 172.21.15.46:0/2587846049 >> v2:172.21.15.106:3300/0 conn(0x7f640c0b2a60 msgr2=0x7f640c0cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:49.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.918+0000 7f6411aa7700 1 --2- 172.21.15.46:0/2587846049 >> v2:172.21.15.106:3300/0 conn(0x7f640c0b2a60 0x7f640c0cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.918+0000 7f6411aa7700 1 -- 172.21.15.46:0/2587846049 >> v1:172.21.15.106:6789/0 conn(0x7f640c157550 legacy=0x7f640c06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.918+0000 7f6411aa7700 1 -- 172.21.15.46:0/2587846049 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c0cb6a0 con 0x7f640c157920 2024-01-27T00:46:49.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.919+0000 7f6411aa7700 1 --2- 172.21.15.46:0/2587846049 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f640c157920 0x7f640c0b2520 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f64080099b0 tx=0x7f6408009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd1d84534541c9d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:49.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.919+0000 7f64112a6700 1 -- 172.21.15.46:0/2587846049 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6408009ea0 con 0x7f640c157920 2024-01-27T00:46:49.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.919+0000 7f64112a6700 1 -- 172.21.15.46:0/2587846049 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640800b860 con 0x7f640c157920 2024-01-27T00:46:49.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.919+0000 7f64112a6700 1 -- 172.21.15.46:0/2587846049 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640800bc20 con 0x7f640c157920 2024-01-27T00:46:49.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.919+0000 7f641450c700 1 -- 172.21.15.46:0/2587846049 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f640c157920 msgr2=0x7f640c0b2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.919+0000 7f641450c700 1 --2- 172.21.15.46:0/2587846049 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f640c157920 0x7f640c0b2520 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f64080099b0 tx=0x7f6408009250 comp rx=0 tx=0).stop 2024-01-27T00:46:49.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.920+0000 7f641450c700 1 -- 172.21.15.46:0/2587846049 shutdown_connections 2024-01-27T00:46:49.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.920+0000 7f641450c700 1 --2- 172.21.15.46:0/2587846049 >> v2:172.21.15.106:3300/0 conn(0x7f640c0b2a60 0x7f640c0cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.920+0000 7f641450c700 1 --2- 172.21.15.46:0/2587846049 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f640c157920 0x7f640c0b2520 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.920+0000 7f641450c700 1 -- 172.21.15.46:0/2587846049 >> 172.21.15.46:0/2587846049 conn(0x7f640c069510 msgr2=0x7f640c0aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:49.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.920+0000 7f641450c700 1 -- 172.21.15.46:0/2587846049 shutdown_connections 2024-01-27T00:46:49.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.920+0000 7f641450c700 1 -- 172.21.15.46:0/2587846049 wait complete. 2024-01-27T00:46:49.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.921+0000 7f641450c700 1 Processor -- start 2024-01-27T00:46:49.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.921+0000 7f641450c700 1 -- start start 2024-01-27T00:46:49.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f641450c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f640c0b2a60 0x7f640c152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:49.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f641450c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f640c157550 0x7f640c14fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:49.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f641450c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f640c157920 0x7f640c14aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:49.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f641450c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c1516f0 con 0x7f640c157920 2024-01-27T00:46:49.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f641450c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c1519d0 con 0x7f640c157550 2024-01-27T00:46:49.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f641450c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640c151cb0 con 0x7f640c0b2a60 2024-01-27T00:46:49.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f6412aa9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f640c157920 0x7f640c14aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:49.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f6412aa9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f640c157920 0x7f640c14aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59906/0 (socket says 172.21.15.46:59906) 2024-01-27T00:46:49.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f6412aa9700 1 -- 172.21.15.46:0/3249760143 learned_addr learned my addr 172.21.15.46:0/3249760143 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:49.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f6411aa7700 1 --2- 172.21.15.46:0/3249760143 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f640c157550 0x7f640c14fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:49.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f64122a8700 1 --2- 172.21.15.46:0/3249760143 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f640c0b2a60 0x7f640c152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:49.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f6412aa9700 1 -- 172.21.15.46:0/3249760143 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f640c0b2a60 msgr2=0x7f640c152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f6412aa9700 1 --2- 172.21.15.46:0/3249760143 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f640c0b2a60 0x7f640c152a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f6412aa9700 1 -- 172.21.15.46:0/3249760143 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f640c157550 msgr2=0x7f640c14fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:49.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f6412aa9700 1 --2- 172.21.15.46:0/3249760143 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f640c157550 0x7f640c14fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:49.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f6412aa9700 1 -- 172.21.15.46:0/3249760143 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64080094f0 con 0x7f640c157920 2024-01-27T00:46:49.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f6412aa9700 1 --2- 172.21.15.46:0/3249760143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f640c157920 0x7f640c14aef0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f6404009e90 tx=0x7f640400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:49.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f63fe7fc700 1 -- 172.21.15.46:0/3249760143 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640400cb70 con 0x7f640c157920 2024-01-27T00:46:49.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.922+0000 7f63fe7fc700 1 -- 172.21.15.46:0/3249760143 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640400f040 con 0x7f640c157920 2024-01-27T00:46:49.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.923+0000 7f641450c700 1 -- 172.21.15.46:0/3249760143 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f640c151f90 con 0x7f640c157920 2024-01-27T00:46:49.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.923+0000 7f63fe7fc700 1 -- 172.21.15.46:0/3249760143 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6404014480 con 0x7f640c157920 2024-01-27T00:46:49.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.923+0000 7f641450c700 1 -- 172.21.15.46:0/3249760143 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f640c152550 con 0x7f640c157920 2024-01-27T00:46:49.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.924+0000 7f641450c700 1 -- 172.21.15.46:0/3249760143 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f640c152770 con 0x7f640c157920 2024-01-27T00:46:49.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.924+0000 7f63fe7fc700 1 -- 172.21.15.46:0/3249760143 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6404014830 con 0x7f640c157920 2024-01-27T00:46:49.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.925+0000 7f63fe7fc700 1 --2- 172.21.15.46:0/3249760143 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f63f0066f30 0x7f63f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:49.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.925+0000 7f63fe7fc700 1 -- 172.21.15.46:0/3249760143 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f6404085290 con 0x7f640c157920 2024-01-27T00:46:49.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.925+0000 7f64122a8700 1 --2- 172.21.15.46:0/3249760143 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f63f0066f30 0x7f63f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:49.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.925+0000 7f64122a8700 1 --2- 172.21.15.46:0/3249760143 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f63f0066f30 0x7f63f00693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f64000039f0 tx=0x7f6400007ec0 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:49.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:49.929+0000 7f63fe7fc700 1 -- 172.21.15.46:0/3249760143 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6404052170 con 0x7f640c157920 2024-01-27T00:46:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.127+0000 7f641450c700 1 -- 172.21.15.46:0/3249760143 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f640c04c710 con 0x7f63f0066f30 2024-01-27T00:46:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.135+0000 7f63fe7fc700 1 -- 172.21.15.46:0/3249760143 <== mgr.8860 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290275 (secure 0 0 0) 0x7f640c04c710 con 0x7f63f0066f30 2024-01-27T00:46:50.138 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.139+0000 7f641450c700 1 -- 172.21.15.46:0/3249760143 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f63f0066f30 msgr2=0x7f63f00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.139+0000 7f641450c700 1 --2- 172.21.15.46:0/3249760143 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f63f0066f30 0x7f63f00693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f64000039f0 tx=0x7f6400007ec0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.139+0000 7f641450c700 1 -- 172.21.15.46:0/3249760143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f640c157920 msgr2=0x7f640c14aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.139+0000 7f641450c700 1 --2- 172.21.15.46:0/3249760143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f640c157920 0x7f640c14aef0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f6404009e90 tx=0x7f640400b040 comp rx=0 tx=0).stop 2024-01-27T00:46:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.139+0000 7f641450c700 1 -- 172.21.15.46:0/3249760143 shutdown_connections 2024-01-27T00:46:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.139+0000 7f641450c700 1 --2- 172.21.15.46:0/3249760143 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f63f0066f30 0x7f63f00693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.139+0000 7f641450c700 1 --2- 172.21.15.46:0/3249760143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f640c157920 0x7f640c14aef0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.139+0000 7f641450c700 1 --2- 172.21.15.46:0/3249760143 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f640c0b2a60 0x7f640c152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.139+0000 7f641450c700 1 --2- 172.21.15.46:0/3249760143 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f640c157550 0x7f640c14fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.139+0000 7f641450c700 1 -- 172.21.15.46:0/3249760143 >> 172.21.15.46:0/3249760143 conn(0x7f640c069510 msgr2=0x7f640c155250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.139+0000 7f641450c700 1 -- 172.21.15.46:0/3249760143 shutdown_connections 2024-01-27T00:46:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.139+0000 7f641450c700 1 -- 172.21.15.46:0/3249760143 wait complete. 2024-01-27T00:46:50.142 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T00:46:50.152 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-01-27T00:46:48.760510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1592,"num_read_kb":3366,"num_write":534,"num_write_kb":3475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8711,"ondisk_log_size":8711,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2490824,"kb_used_data":18312,"kb_used_omap":0,"kb_used_meta":2472448,"kb_avail":752483896,"statfs":{"total":773094113280,"available":770543509504,"internally_reserved":0,"allocated":18751488,"data_stored":10280276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2531786752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.227445"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566300+0000","last_change":"2024-01-27T00:46:45.557055+0000","last_active":"2024-01-27T00:46:45.566300+0000","last_peered":"2024-01-27T00:46:45.566300+0000","last_clean":"2024-01-27T00:46:45.566300+0000","last_became_active":"2024-01-27T00:46:45.556790+0000","last_became_peered":"2024-01-27T00:46:45.556790+0000","last_unstale":"2024-01-27T00:46:45.566300+0000","last_undegraded":"2024-01-27T00:46:45.566300+0000","last_fullsized":"2024-01-27T00:46:45.566300+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:52:06.112690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574313+0000","last_change":"2024-01-27T00:46:45.568147+0000","last_active":"2024-01-27T00:46:45.574313+0000","last_peered":"2024-01-27T00:46:45.574313+0000","last_clean":"2024-01-27T00:46:45.574313+0000","last_became_active":"2024-01-27T00:46:45.567855+0000","last_became_peered":"2024-01-27T00:46:45.567855+0000","last_unstale":"2024-01-27T00:46:45.574313+0000","last_undegraded":"2024-01-27T00:46:45.574313+0000","last_fullsized":"2024-01-27T00:46:45.574313+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:26:59.092632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.565508+0000","last_change":"2024-01-27T00:46:45.562607+0000","last_active":"2024-01-27T00:46:45.565508+0000","last_peered":"2024-01-27T00:46:45.565508+0000","last_clean":"2024-01-27T00:46:45.565508+0000","last_became_active":"2024-01-27T00:46:45.561888+0000","last_became_peered":"2024-01-27T00:46:45.561888+0000","last_unstale":"2024-01-27T00:46:45.565508+0000","last_undegraded":"2024-01-27T00:46:45.565508+0000","last_fullsized":"2024-01-27T00:46:45.565508+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:35:54.188355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.567733+0000","last_change":"2024-01-27T00:46:45.560536+0000","last_active":"2024-01-27T00:46:45.567733+0000","last_peered":"2024-01-27T00:46:45.567733+0000","last_clean":"2024-01-27T00:46:45.567733+0000","last_became_active":"2024-01-27T00:46:45.560235+0000","last_became_peered":"2024-01-27T00:46:45.560235+0000","last_unstale":"2024-01-27T00:46:45.567733+0000","last_undegraded":"2024-01-27T00:46:45.567733+0000","last_fullsized":"2024-01-27T00:46:45.567733+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:41:40.453855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.571085+0000","last_change":"2024-01-27T00:46:45.568127+0000","last_active":"2024-01-27T00:46:45.571085+0000","last_peered":"2024-01-27T00:46:45.571085+0000","last_clean":"2024-01-27T00:46:45.571085+0000","last_became_active":"2024-01-27T00:46:45.567791+0000","last_became_peered":"2024-01-27T00:46:45.567791+0000","last_unstale":"2024-01-27T00:46:45.571085+0000","last_undegraded":"2024-01-27T00:46:45.571085+0000","last_fullsized":"2024-01-27T00:46:45.571085+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:35:43.360306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.570790+0000","last_change":"2024-01-27T00:46:45.568275+0000","last_active":"2024-01-27T00:46:45.570790+0000","last_peered":"2024-01-27T00:46:45.570790+0000","last_clean":"2024-01-27T00:46:45.570790+0000","last_became_active":"2024-01-27T00:46:45.567998+0000","last_became_peered":"2024-01-27T00:46:45.567998+0000","last_unstale":"2024-01-27T00:46:45.570790+0000","last_undegraded":"2024-01-27T00:46:45.570790+0000","last_fullsized":"2024-01-27T00:46:45.570790+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:26.817597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.567746+0000","last_change":"2024-01-27T00:46:45.559800+0000","last_active":"2024-01-27T00:46:45.567746+0000","last_peered":"2024-01-27T00:46:45.567746+0000","last_clean":"2024-01-27T00:46:45.567746+0000","last_became_active":"2024-01-27T00:46:45.559524+0000","last_became_peered":"2024-01-27T00:46:45.559524+0000","last_unstale":"2024-01-27T00:46:45.567746+0000","last_undegraded":"2024-01-27T00:46:45.567746+0000","last_fullsized":"2024-01-27T00:46:45.567746+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:04:20.968716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.552575+0000","last_change":"2024-01-27T00:46:45.548896+0000","last_active":"2024-01-27T00:46:45.552575+0000","last_peered":"2024-01-27T00:46:45.552575+0000","last_clean":"2024-01-27T00:46:45.552575+0000","last_became_active":"2024-01-27T00:46:45.548490+0000","last_became_peered":"2024-01-27T00:46:45.548490+0000","last_unstale":"2024-01-27T00:46:45.552575+0000","last_undegraded":"2024-01-27T00:46:45.552575+0000","last_fullsized":"2024-01-27T00:46:45.552575+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:00:27.609050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.569425+0000","last_change":"2024-01-27T00:46:45.565679+0000","last_active":"2024-01-27T00:46:45.569425+0000","last_peered":"2024-01-27T00:46:45.569425+0000","last_clean":"2024-01-27T00:46:45.569425+0000","last_became_active":"2024-01-27T00:46:45.565374+0000","last_became_peered":"2024-01-27T00:46:45.565374+0000","last_unstale":"2024-01-27T00:46:45.569425+0000","last_undegraded":"2024-01-27T00:46:45.569425+0000","last_fullsized":"2024-01-27T00:46:45.569425+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:00:18.369351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.569451+0000","last_change":"2024-01-27T00:46:45.557200+0000","last_active":"2024-01-27T00:46:45.569451+0000","last_peered":"2024-01-27T00:46:45.569451+0000","last_clean":"2024-01-27T00:46:45.569451+0000","last_became_active":"2024-01-27T00:46:45.556886+0000","last_became_peered":"2024-01-27T00:46:45.556886+0000","last_unstale":"2024-01-27T00:46:45.569451+0000","last_undegraded":"2024-01-27T00:46:45.569451+0000","last_fullsized":"2024-01-27T00:46:45.569451+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:56:30.192694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.558768+0000","last_change":"2024-01-27T00:46:45.557911+0000","last_active":"2024-01-27T00:46:45.558768+0000","last_peered":"2024-01-27T00:46:45.558768+0000","last_clean":"2024-01-27T00:46:45.558768+0000","last_became_active":"2024-01-27T00:46:45.557573+0000","last_became_peered":"2024-01-27T00:46:45.557573+0000","last_unstale":"2024-01-27T00:46:45.558768+0000","last_undegraded":"2024-01-27T00:46:45.558768+0000","last_fullsized":"2024-01-27T00:46:45.558768+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:59:17.572174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.567831+0000","last_change":"2024-01-27T00:46:45.565562+0000","last_active":"2024-01-27T00:46:45.567831+0000","last_peered":"2024-01-27T00:46:45.567831+0000","last_clean":"2024-01-27T00:46:45.567831+0000","last_became_active":"2024-01-27T00:46:45.565307+0000","last_became_peered":"2024-01-27T00:46:45.565307+0000","last_unstale":"2024-01-27T00:46:45.567831+0000","last_undegraded":"2024-01-27T00:46:45.567831+0000","last_fullsized":"2024-01-27T00:46:45.567831+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:39:52.381793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.564522+0000","last_change":"2024-01-27T00:46:45.559536+0000","last_active":"2024-01-27T00:46:45.564522+0000","last_peered":"2024-01-27T00:46:45.564522+0000","last_clean":"2024-01-27T00:46:45.564522+0000","last_became_active":"2024-01-27T00:46:45.559054+0000","last_became_peered":"2024-01-27T00:46:45.559054+0000","last_unstale":"2024-01-27T00:46:45.564522+0000","last_undegraded":"2024-01-27T00:46:45.564522+0000","last_fullsized":"2024-01-27T00:46:45.564522+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:33:10.523909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.567859+0000","last_change":"2024-01-27T00:46:45.560510+0000","last_active":"2024-01-27T00:46:45.567859+0000","last_peered":"2024-01-27T00:46:45.567859+0000","last_clean":"2024-01-27T00:46:45.567859+0000","last_became_active":"2024-01-27T00:46:45.560103+0000","last_became_peered":"2024-01-27T00:46:45.560103+0000","last_unstale":"2024-01-27T00:46:45.567859+0000","last_undegraded":"2024-01-27T00:46:45.567859+0000","last_fullsized":"2024-01-27T00:46:45.567859+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:42:33.340128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.573835+0000","last_change":"2024-01-27T00:46:45.563721+0000","last_active":"2024-01-27T00:46:45.573835+0000","last_peered":"2024-01-27T00:46:45.573835+0000","last_clean":"2024-01-27T00:46:45.573835+0000","last_became_active":"2024-01-27T00:46:45.563013+0000","last_became_peered":"2024-01-27T00:46:45.563013+0000","last_unstale":"2024-01-27T00:46:45.573835+0000","last_undegraded":"2024-01-27T00:46:45.573835+0000","last_fullsized":"2024-01-27T00:46:45.573835+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:25.389867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.564719+0000","last_change":"2024-01-27T00:46:45.562938+0000","last_active":"2024-01-27T00:46:45.564719+0000","last_peered":"2024-01-27T00:46:45.564719+0000","last_clean":"2024-01-27T00:46:45.564719+0000","last_became_active":"2024-01-27T00:46:45.562604+0000","last_became_peered":"2024-01-27T00:46:45.562604+0000","last_unstale":"2024-01-27T00:46:45.564719+0000","last_undegraded":"2024-01-27T00:46:45.564719+0000","last_fullsized":"2024-01-27T00:46:45.564719+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:22:25.580376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.568023+0000","last_change":"2024-01-27T00:46:45.561614+0000","last_active":"2024-01-27T00:46:45.568023+0000","last_peered":"2024-01-27T00:46:45.568023+0000","last_clean":"2024-01-27T00:46:45.568023+0000","last_became_active":"2024-01-27T00:46:45.561278+0000","last_became_peered":"2024-01-27T00:46:45.561278+0000","last_unstale":"2024-01-27T00:46:45.568023+0000","last_undegraded":"2024-01-27T00:46:45.568023+0000","last_fullsized":"2024-01-27T00:46:45.568023+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:58.834738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.561918+0000","last_change":"2024-01-27T00:46:45.561317+0000","last_active":"2024-01-27T00:46:45.561918+0000","last_peered":"2024-01-27T00:46:45.561918+0000","last_clean":"2024-01-27T00:46:45.561918+0000","last_became_active":"2024-01-27T00:46:45.561008+0000","last_became_peered":"2024-01-27T00:46:45.561008+0000","last_unstale":"2024-01-27T00:46:45.561918+0000","last_undegraded":"2024-01-27T00:46:45.561918+0000","last_fullsized":"2024-01-27T00:46:45.561918+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:47:36.930423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566596+0000","last_change":"2024-01-27T00:46:45.559098+0000","last_active":"2024-01-27T00:46:45.566596+0000","last_peered":"2024-01-27T00:46:45.566596+0000","last_clean":"2024-01-27T00:46:45.566596+0000","last_became_active":"2024-01-27T00:46:45.558697+0000","last_became_peered":"2024-01-27T00:46:45.558697+0000","last_unstale":"2024-01-27T00:46:45.566596+0000","last_undegraded":"2024-01-27T00:46:45.566596+0000","last_fullsized":"2024-01-27T00:46:45.566596+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:09:01.270842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574654+0000","last_change":"2024-01-27T00:46:45.569640+0000","last_active":"2024-01-27T00:46:45.574654+0000","last_peered":"2024-01-27T00:46:45.574654+0000","last_clean":"2024-01-27T00:46:45.574654+0000","last_became_active":"2024-01-27T00:46:45.569340+0000","last_became_peered":"2024-01-27T00:46:45.569340+0000","last_unstale":"2024-01-27T00:46:45.574654+0000","last_undegraded":"2024-01-27T00:46:45.574654+0000","last_fullsized":"2024-01-27T00:46:45.574654+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:59:49.044163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.570274+0000","last_change":"2024-01-27T00:46:45.565670+0000","last_active":"2024-01-27T00:46:45.570274+0000","last_peered":"2024-01-27T00:46:45.570274+0000","last_clean":"2024-01-27T00:46:45.570274+0000","last_became_active":"2024-01-27T00:46:45.565313+0000","last_became_peered":"2024-01-27T00:46:45.565313+0000","last_unstale":"2024-01-27T00:46:45.570274+0000","last_undegraded":"2024-01-27T00:46:45.570274+0000","last_fullsized":"2024-01-27T00:46:45.570274+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:13:12.945507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.573589+0000","last_change":"2024-01-27T00:46:45.569350+0000","last_active":"2024-01-27T00:46:45.573589+0000","last_peered":"2024-01-27T00:46:45.573589+0000","last_clean":"2024-01-27T00:46:45.573589+0000","last_became_active":"2024-01-27T00:46:45.569024+0000","last_became_peered":"2024-01-27T00:46:45.569024+0000","last_unstale":"2024-01-27T00:46:45.573589+0000","last_undegraded":"2024-01-27T00:46:45.573589+0000","last_fullsized":"2024-01-27T00:46:45.573589+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:44:56.459748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.559040+0000","last_change":"2024-01-27T00:46:45.556267+0000","last_active":"2024-01-27T00:46:45.559040+0000","last_peered":"2024-01-27T00:46:45.559040+0000","last_clean":"2024-01-27T00:46:45.559040+0000","last_became_active":"2024-01-27T00:46:45.555962+0000","last_became_peered":"2024-01-27T00:46:45.555962+0000","last_unstale":"2024-01-27T00:46:45.559040+0000","last_undegraded":"2024-01-27T00:46:45.559040+0000","last_fullsized":"2024-01-27T00:46:45.559040+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:35:54.188355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.557839+0000","last_change":"2024-01-27T00:46:45.555830+0000","last_active":"2024-01-27T00:46:45.557839+0000","last_peered":"2024-01-27T00:46:45.557839+0000","last_clean":"2024-01-27T00:46:45.557839+0000","last_became_active":"2024-01-27T00:46:45.555524+0000","last_became_peered":"2024-01-27T00:46:45.555524+0000","last_unstale":"2024-01-27T00:46:45.557839+0000","last_undegraded":"2024-01-27T00:46:45.557839+0000","last_fullsized":"2024-01-27T00:46:45.557839+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:59:50.650401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566322+0000","last_change":"2024-01-27T00:46:45.559100+0000","last_active":"2024-01-27T00:46:45.566322+0000","last_peered":"2024-01-27T00:46:45.566322+0000","last_clean":"2024-01-27T00:46:45.566322+0000","last_became_active":"2024-01-27T00:46:45.558851+0000","last_became_peered":"2024-01-27T00:46:45.558851+0000","last_unstale":"2024-01-27T00:46:45.566322+0000","last_undegraded":"2024-01-27T00:46:45.566322+0000","last_fullsized":"2024-01-27T00:46:45.566322+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:12:26.858732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566356+0000","last_change":"2024-01-27T00:46:45.558189+0000","last_active":"2024-01-27T00:46:45.566356+0000","last_peered":"2024-01-27T00:46:45.566356+0000","last_clean":"2024-01-27T00:46:45.566356+0000","last_became_active":"2024-01-27T00:46:45.557564+0000","last_became_peered":"2024-01-27T00:46:45.557564+0000","last_unstale":"2024-01-27T00:46:45.566356+0000","last_undegraded":"2024-01-27T00:46:45.566356+0000","last_fullsized":"2024-01-27T00:46:45.566356+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:23:24.863025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.573177+0000","last_change":"2024-01-27T00:46:45.570543+0000","last_active":"2024-01-27T00:46:45.573177+0000","last_peered":"2024-01-27T00:46:45.573177+0000","last_clean":"2024-01-27T00:46:45.573177+0000","last_became_active":"2024-01-27T00:46:45.570129+0000","last_became_peered":"2024-01-27T00:46:45.570129+0000","last_unstale":"2024-01-27T00:46:45.573177+0000","last_undegraded":"2024-01-27T00:46:45.573177+0000","last_fullsized":"2024-01-27T00:46:45.573177+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:14.376349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.554835+0000","last_change":"2024-01-27T00:46:45.553830+0000","last_active":"2024-01-27T00:46:45.554835+0000","last_peered":"2024-01-27T00:46:45.554835+0000","last_clean":"2024-01-27T00:46:45.554835+0000","last_became_active":"2024-01-27T00:46:45.553190+0000","last_became_peered":"2024-01-27T00:46:45.553190+0000","last_unstale":"2024-01-27T00:46:45.554835+0000","last_undegraded":"2024-01-27T00:46:45.554835+0000","last_fullsized":"2024-01-27T00:46:45.554835+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:58:19.948464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.571005+0000","last_change":"2024-01-27T00:46:45.562755+0000","last_active":"2024-01-27T00:46:45.571005+0000","last_peered":"2024-01-27T00:46:45.571005+0000","last_clean":"2024-01-27T00:46:45.571005+0000","last_became_active":"2024-01-27T00:46:45.562450+0000","last_became_peered":"2024-01-27T00:46:45.562450+0000","last_unstale":"2024-01-27T00:46:45.571005+0000","last_undegraded":"2024-01-27T00:46:45.571005+0000","last_fullsized":"2024-01-27T00:46:45.571005+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:35.210458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.563461+0000","last_change":"2024-01-27T00:46:45.562746+0000","last_active":"2024-01-27T00:46:45.563461+0000","last_peered":"2024-01-27T00:46:45.563461+0000","last_clean":"2024-01-27T00:46:45.563461+0000","last_became_active":"2024-01-27T00:46:45.562444+0000","last_became_peered":"2024-01-27T00:46:45.562444+0000","last_unstale":"2024-01-27T00:46:45.563461+0000","last_undegraded":"2024-01-27T00:46:45.563461+0000","last_fullsized":"2024-01-27T00:46:45.563461+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:09:10.316894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.564589+0000","last_change":"2024-01-27T00:46:45.560219+0000","last_active":"2024-01-27T00:46:45.564589+0000","last_peered":"2024-01-27T00:46:45.564589+0000","last_clean":"2024-01-27T00:46:45.564589+0000","last_became_active":"2024-01-27T00:46:45.559927+0000","last_became_peered":"2024-01-27T00:46:45.559927+0000","last_unstale":"2024-01-27T00:46:45.564589+0000","last_undegraded":"2024-01-27T00:46:45.564589+0000","last_fullsized":"2024-01-27T00:46:45.564589+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:07:59.691769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.551423+0000","last_change":"2024-01-27T00:46:45.548738+0000","last_active":"2024-01-27T00:46:45.551423+0000","last_peered":"2024-01-27T00:46:45.551423+0000","last_clean":"2024-01-27T00:46:45.551423+0000","last_became_active":"2024-01-27T00:46:45.548152+0000","last_became_peered":"2024-01-27T00:46:45.548152+0000","last_unstale":"2024-01-27T00:46:45.551423+0000","last_undegraded":"2024-01-27T00:46:45.551423+0000","last_fullsized":"2024-01-27T00:46:45.551423+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:20:59.598654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574345+0000","last_change":"2024-01-27T00:46:45.568632+0000","last_active":"2024-01-27T00:46:45.574345+0000","last_peered":"2024-01-27T00:46:45.574345+0000","last_clean":"2024-01-27T00:46:45.574345+0000","last_became_active":"2024-01-27T00:46:45.568319+0000","last_became_peered":"2024-01-27T00:46:45.568319+0000","last_unstale":"2024-01-27T00:46:45.574345+0000","last_undegraded":"2024-01-27T00:46:45.574345+0000","last_fullsized":"2024-01-27T00:46:45.574345+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:09:24.182020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"174'4","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.575252+0000","last_change":"2024-01-27T00:46:44.551082+0000","last_active":"2024-01-27T00:46:45.575252+0000","last_peered":"2024-01-27T00:46:45.575252+0000","last_clean":"2024-01-27T00:46:45.575252+0000","last_became_active":"2024-01-27T00:46:44.550800+0000","last_became_peered":"2024-01-27T00:46:44.550800+0000","last_unstale":"2024-01-27T00:46:45.575252+0000","last_undegraded":"2024-01-27T00:46:45.575252+0000","last_fullsized":"2024-01-27T00:46:45.575252+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:34.205561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574409+0000","last_change":"2024-01-27T00:46:45.565405+0000","last_active":"2024-01-27T00:46:45.574409+0000","last_peered":"2024-01-27T00:46:45.574409+0000","last_clean":"2024-01-27T00:46:45.574409+0000","last_became_active":"2024-01-27T00:46:45.562543+0000","last_became_peered":"2024-01-27T00:46:45.562543+0000","last_unstale":"2024-01-27T00:46:45.574409+0000","last_undegraded":"2024-01-27T00:46:45.574409+0000","last_fullsized":"2024-01-27T00:46:45.574409+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:34.525936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"174'1","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.576835+0000","last_change":"2024-01-27T00:46:44.564397+0000","last_active":"2024-01-27T00:46:45.576835+0000","last_peered":"2024-01-27T00:46:45.576835+0000","last_clean":"2024-01-27T00:46:45.576835+0000","last_became_active":"2024-01-27T00:46:44.564118+0000","last_became_peered":"2024-01-27T00:46:44.564118+0000","last_unstale":"2024-01-27T00:46:45.576835+0000","last_undegraded":"2024-01-27T00:46:45.576835+0000","last_fullsized":"2024-01-27T00:46:45.576835+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:24.384971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.571497+0000","last_change":"2024-01-27T00:46:45.563691+0000","last_active":"2024-01-27T00:46:45.571497+0000","last_peered":"2024-01-27T00:46:45.571497+0000","last_clean":"2024-01-27T00:46:45.571497+0000","last_became_active":"2024-01-27T00:46:45.562302+0000","last_became_peered":"2024-01-27T00:46:45.562302+0000","last_unstale":"2024-01-27T00:46:45.571497+0000","last_undegraded":"2024-01-27T00:46:45.571497+0000","last_fullsized":"2024-01-27T00:46:45.571497+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:43:42.530709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"174'2","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.576770+0000","last_change":"2024-01-27T00:46:44.561860+0000","last_active":"2024-01-27T00:46:45.576770+0000","last_peered":"2024-01-27T00:46:45.576770+0000","last_clean":"2024-01-27T00:46:45.576770+0000","last_became_active":"2024-01-27T00:46:44.561579+0000","last_became_peered":"2024-01-27T00:46:44.561579+0000","last_unstale":"2024-01-27T00:46:45.576770+0000","last_undegraded":"2024-01-27T00:46:45.576770+0000","last_fullsized":"2024-01-27T00:46:45.576770+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:43:33.521039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566453+0000","last_change":"2024-01-27T00:46:45.562542+0000","last_active":"2024-01-27T00:46:45.566453+0000","last_peered":"2024-01-27T00:46:45.566453+0000","last_clean":"2024-01-27T00:46:45.566453+0000","last_became_active":"2024-01-27T00:46:45.562264+0000","last_became_peered":"2024-01-27T00:46:45.562264+0000","last_unstale":"2024-01-27T00:46:45.566453+0000","last_undegraded":"2024-01-27T00:46:45.566453+0000","last_fullsized":"2024-01-27T00:46:45.566453+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:57:28.818719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"174'2","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.577261+0000","last_change":"2024-01-27T00:46:44.559897+0000","last_active":"2024-01-27T00:46:45.577261+0000","last_peered":"2024-01-27T00:46:45.577261+0000","last_clean":"2024-01-27T00:46:45.577261+0000","last_became_active":"2024-01-27T00:46:44.559359+0000","last_became_peered":"2024-01-27T00:46:44.559359+0000","last_unstale":"2024-01-27T00:46:45.577261+0000","last_undegraded":"2024-01-27T00:46:45.577261+0000","last_fullsized":"2024-01-27T00:46:45.577261+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:12:25.853835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.550893+0000","last_change":"2024-01-27T00:46:45.547644+0000","last_active":"2024-01-27T00:46:45.550893+0000","last_peered":"2024-01-27T00:46:45.550893+0000","last_clean":"2024-01-27T00:46:45.550893+0000","last_became_active":"2024-01-27T00:46:45.547338+0000","last_became_peered":"2024-01-27T00:46:45.547338+0000","last_unstale":"2024-01-27T00:46:45.550893+0000","last_undegraded":"2024-01-27T00:46:45.550893+0000","last_fullsized":"2024-01-27T00:46:45.550893+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:34:01.102364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"174'1","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.575799+0000","last_change":"2024-01-27T00:46:44.559909+0000","last_active":"2024-01-27T00:46:45.575799+0000","last_peered":"2024-01-27T00:46:45.575799+0000","last_clean":"2024-01-27T00:46:45.575799+0000","last_became_active":"2024-01-27T00:46:44.559628+0000","last_became_peered":"2024-01-27T00:46:44.559628+0000","last_unstale":"2024-01-27T00:46:45.575799+0000","last_undegraded":"2024-01-27T00:46:45.575799+0000","last_fullsized":"2024-01-27T00:46:45.575799+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:12:25.853835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.573630+0000","last_change":"2024-01-27T00:46:45.568628+0000","last_active":"2024-01-27T00:46:45.573630+0000","last_peered":"2024-01-27T00:46:45.573630+0000","last_clean":"2024-01-27T00:46:45.573630+0000","last_became_active":"2024-01-27T00:46:45.568296+0000","last_became_peered":"2024-01-27T00:46:45.568296+0000","last_unstale":"2024-01-27T00:46:45.573630+0000","last_undegraded":"2024-01-27T00:46:45.573630+0000","last_fullsized":"2024-01-27T00:46:45.573630+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:56:30.192694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"174'1","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.573858+0000","last_change":"2024-01-27T00:46:44.555019+0000","last_active":"2024-01-27T00:46:45.573858+0000","last_peered":"2024-01-27T00:46:45.573858+0000","last_clean":"2024-01-27T00:46:45.573858+0000","last_became_active":"2024-01-27T00:46:44.554324+0000","last_became_peered":"2024-01-27T00:46:44.554324+0000","last_unstale":"2024-01-27T00:46:45.573858+0000","last_undegraded":"2024-01-27T00:46:45.573858+0000","last_fullsized":"2024-01-27T00:46:45.573858+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:00:26.604154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574814+0000","last_change":"2024-01-27T00:46:45.572035+0000","last_active":"2024-01-27T00:46:45.574814+0000","last_peered":"2024-01-27T00:46:45.574814+0000","last_clean":"2024-01-27T00:46:45.574814+0000","last_became_active":"2024-01-27T00:46:45.571751+0000","last_became_peered":"2024-01-27T00:46:45.571751+0000","last_unstale":"2024-01-27T00:46:45.574814+0000","last_undegraded":"2024-01-27T00:46:45.574814+0000","last_fullsized":"2024-01-27T00:46:45.574814+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:17:32.090542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.549984+0000","last_change":"2024-01-27T00:46:44.554494+0000","last_active":"2024-01-27T00:46:45.549984+0000","last_peered":"2024-01-27T00:46:45.549984+0000","last_clean":"2024-01-27T00:46:45.549984+0000","last_became_active":"2024-01-27T00:46:44.554249+0000","last_became_peered":"2024-01-27T00:46:44.554249+0000","last_unstale":"2024-01-27T00:46:45.549984+0000","last_undegraded":"2024-01-27T00:46:45.549984+0000","last_fullsized":"2024-01-27T00:46:45.549984+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:13:07.172249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.567713+0000","last_change":"2024-01-27T00:46:45.559360+0000","last_active":"2024-01-27T00:46:45.567713+0000","last_peered":"2024-01-27T00:46:45.567713+0000","last_clean":"2024-01-27T00:46:45.567713+0000","last_became_active":"2024-01-27T00:46:45.557778+0000","last_became_peered":"2024-01-27T00:46:45.557778+0000","last_unstale":"2024-01-27T00:46:45.567713+0000","last_undegraded":"2024-01-27T00:46:45.567713+0000","last_fullsized":"2024-01-27T00:46:45.567713+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:33:23.078082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.548933+0000","last_change":"2024-01-27T00:46:44.558303+0000","last_active":"2024-01-27T00:46:45.548933+0000","last_peered":"2024-01-27T00:46:45.548933+0000","last_clean":"2024-01-27T00:46:45.548933+0000","last_became_active":"2024-01-27T00:46:44.558020+0000","last_became_peered":"2024-01-27T00:46:44.558020+0000","last_unstale":"2024-01-27T00:46:45.548933+0000","last_undegraded":"2024-01-27T00:46:45.548933+0000","last_fullsized":"2024-01-27T00:46:45.548933+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:46:04.221575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.562194+0000","last_change":"2024-01-27T00:46:45.558862+0000","last_active":"2024-01-27T00:46:45.562194+0000","last_peered":"2024-01-27T00:46:45.562194+0000","last_clean":"2024-01-27T00:46:45.562194+0000","last_became_active":"2024-01-27T00:46:45.558585+0000","last_became_peered":"2024-01-27T00:46:45.558585+0000","last_unstale":"2024-01-27T00:46:45.562194+0000","last_undegraded":"2024-01-27T00:46:45.562194+0000","last_fullsized":"2024-01-27T00:46:45.562194+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:13:52.377253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.552835+0000","last_change":"2024-01-27T00:46:44.553830+0000","last_active":"2024-01-27T00:46:45.552835+0000","last_peered":"2024-01-27T00:46:45.552835+0000","last_clean":"2024-01-27T00:46:45.552835+0000","last_became_active":"2024-01-27T00:46:44.553537+0000","last_became_peered":"2024-01-27T00:46:44.553537+0000","last_unstale":"2024-01-27T00:46:45.552835+0000","last_undegraded":"2024-01-27T00:46:45.552835+0000","last_fullsized":"2024-01-27T00:46:45.552835+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:00:26.604154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.558787+0000","last_change":"2024-01-27T00:46:45.558061+0000","last_active":"2024-01-27T00:46:45.558787+0000","last_peered":"2024-01-27T00:46:45.558787+0000","last_clean":"2024-01-27T00:46:45.558787+0000","last_became_active":"2024-01-27T00:46:45.557727+0000","last_became_peered":"2024-01-27T00:46:45.557727+0000","last_unstale":"2024-01-27T00:46:45.558787+0000","last_undegraded":"2024-01-27T00:46:45.558787+0000","last_fullsized":"2024-01-27T00:46:45.558787+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:48:54.022424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.552388+0000","last_change":"2024-01-27T00:46:44.562077+0000","last_active":"2024-01-27T00:46:45.552388+0000","last_peered":"2024-01-27T00:46:45.552388+0000","last_clean":"2024-01-27T00:46:45.552388+0000","last_became_active":"2024-01-27T00:46:44.561803+0000","last_became_peered":"2024-01-27T00:46:44.561803+0000","last_unstale":"2024-01-27T00:46:45.552388+0000","last_undegraded":"2024-01-27T00:46:45.552388+0000","last_fullsized":"2024-01-27T00:46:45.552388+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:09:09.311997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.561937+0000","last_change":"2024-01-27T00:46:45.561307+0000","last_active":"2024-01-27T00:46:45.561937+0000","last_peered":"2024-01-27T00:46:45.561937+0000","last_clean":"2024-01-27T00:46:45.561937+0000","last_became_active":"2024-01-27T00:46:45.560472+0000","last_became_peered":"2024-01-27T00:46:45.560472+0000","last_unstale":"2024-01-27T00:46:45.561937+0000","last_undegraded":"2024-01-27T00:46:45.561937+0000","last_fullsized":"2024-01-27T00:46:45.561937+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:36:33.493714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"174'3","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.576689+0000","last_change":"2024-01-27T00:46:44.566003+0000","last_active":"2024-01-27T00:46:45.576689+0000","last_peered":"2024-01-27T00:46:45.576689+0000","last_clean":"2024-01-27T00:46:45.576689+0000","last_became_active":"2024-01-27T00:46:44.565642+0000","last_became_peered":"2024-01-27T00:46:44.565642+0000","last_unstale":"2024-01-27T00:46:45.576689+0000","last_undegraded":"2024-01-27T00:46:45.576689+0000","last_fullsized":"2024-01-27T00:46:45.576689+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:19:13.127951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.570302+0000","last_change":"2024-01-27T00:46:45.560093+0000","last_active":"2024-01-27T00:46:45.570302+0000","last_peered":"2024-01-27T00:46:45.570302+0000","last_clean":"2024-01-27T00:46:45.570302+0000","last_became_active":"2024-01-27T00:46:45.559827+0000","last_became_peered":"2024-01-27T00:46:45.559827+0000","last_unstale":"2024-01-27T00:46:45.570302+0000","last_undegraded":"2024-01-27T00:46:45.570302+0000","last_fullsized":"2024-01-27T00:46:45.570302+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:09:24.182020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"174'2","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.575127+0000","last_change":"2024-01-27T00:46:44.554097+0000","last_active":"2024-01-27T00:46:45.575127+0000","last_peered":"2024-01-27T00:46:45.575127+0000","last_clean":"2024-01-27T00:46:45.575127+0000","last_became_active":"2024-01-27T00:46:44.553773+0000","last_became_peered":"2024-01-27T00:46:44.553773+0000","last_unstale":"2024-01-27T00:46:45.575127+0000","last_undegraded":"2024-01-27T00:46:45.575127+0000","last_fullsized":"2024-01-27T00:46:45.575127+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:20:58.593757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574328+0000","last_change":"2024-01-27T00:46:45.563853+0000","last_active":"2024-01-27T00:46:45.574328+0000","last_peered":"2024-01-27T00:46:45.574328+0000","last_clean":"2024-01-27T00:46:45.574328+0000","last_became_active":"2024-01-27T00:46:45.563296+0000","last_became_peered":"2024-01-27T00:46:45.563296+0000","last_unstale":"2024-01-27T00:46:45.574328+0000","last_undegraded":"2024-01-27T00:46:45.574328+0000","last_fullsized":"2024-01-27T00:46:45.574328+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:20:59.598654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.547835+0000","last_change":"2024-01-27T00:46:44.560443+0000","last_active":"2024-01-27T00:46:45.547835+0000","last_peered":"2024-01-27T00:46:45.547835+0000","last_clean":"2024-01-27T00:46:45.547835+0000","last_became_active":"2024-01-27T00:46:44.559737+0000","last_became_peered":"2024-01-27T00:46:44.559737+0000","last_unstale":"2024-01-27T00:46:45.547835+0000","last_undegraded":"2024-01-27T00:46:45.547835+0000","last_fullsized":"2024-01-27T00:46:45.547835+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:35:53.183459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.564490+0000","last_change":"2024-01-27T00:46:45.558068+0000","last_active":"2024-01-27T00:46:45.564490+0000","last_peered":"2024-01-27T00:46:45.564490+0000","last_clean":"2024-01-27T00:46:45.564490+0000","last_became_active":"2024-01-27T00:46:45.557671+0000","last_became_peered":"2024-01-27T00:46:45.557671+0000","last_unstale":"2024-01-27T00:46:45.564490+0000","last_undegraded":"2024-01-27T00:46:45.564490+0000","last_fullsized":"2024-01-27T00:46:45.564490+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:10:24.979198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.547330+0000","last_change":"2024-01-27T00:46:44.562797+0000","last_active":"2024-01-27T00:46:45.547330+0000","last_peered":"2024-01-27T00:46:45.547330+0000","last_clean":"2024-01-27T00:46:45.547330+0000","last_became_active":"2024-01-27T00:46:44.562519+0000","last_became_peered":"2024-01-27T00:46:44.562519+0000","last_unstale":"2024-01-27T00:46:45.547330+0000","last_undegraded":"2024-01-27T00:46:45.547330+0000","last_fullsized":"2024-01-27T00:46:45.547330+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:34.205561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574864+0000","last_change":"2024-01-27T00:46:45.570359+0000","last_active":"2024-01-27T00:46:45.574864+0000","last_peered":"2024-01-27T00:46:45.574864+0000","last_clean":"2024-01-27T00:46:45.574864+0000","last_became_active":"2024-01-27T00:46:45.568395+0000","last_became_peered":"2024-01-27T00:46:45.568395+0000","last_unstale":"2024-01-27T00:46:45.574864+0000","last_undegraded":"2024-01-27T00:46:45.574864+0000","last_fullsized":"2024-01-27T00:46:45.574864+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:35.210458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"174'8","reported_seq":18,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.576921+0000","last_change":"2024-01-27T00:46:44.560114+0000","last_active":"2024-01-27T00:46:45.576921+0000","last_peered":"2024-01-27T00:46:45.576921+0000","last_clean":"2024-01-27T00:46:45.576921+0000","last_became_active":"2024-01-27T00:46:44.559240+0000","last_became_peered":"2024-01-27T00:46:44.559240+0000","last_unstale":"2024-01-27T00:46:45.576921+0000","last_undegraded":"2024-01-27T00:46:45.576921+0000","last_fullsized":"2024-01-27T00:46:45.576921+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:59:49.645505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566628+0000","last_change":"2024-01-27T00:46:45.565567+0000","last_active":"2024-01-27T00:46:45.566628+0000","last_peered":"2024-01-27T00:46:45.566628+0000","last_clean":"2024-01-27T00:46:45.566628+0000","last_became_active":"2024-01-27T00:46:45.565320+0000","last_became_peered":"2024-01-27T00:46:45.565320+0000","last_unstale":"2024-01-27T00:46:45.566628+0000","last_undegraded":"2024-01-27T00:46:45.566628+0000","last_fullsized":"2024-01-27T00:46:45.566628+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:42:33.340128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"174'1","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.577148+0000","last_change":"2024-01-27T00:46:44.556531+0000","last_active":"2024-01-27T00:46:45.577148+0000","last_peered":"2024-01-27T00:46:45.577148+0000","last_clean":"2024-01-27T00:46:45.577148+0000","last_became_active":"2024-01-27T00:46:44.555951+0000","last_became_peered":"2024-01-27T00:46:44.555951+0000","last_unstale":"2024-01-27T00:46:45.577148+0000","last_undegraded":"2024-01-27T00:46:45.577148+0000","last_fullsized":"2024-01-27T00:46:45.577148+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:06:08.035902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566575+0000","last_change":"2024-01-27T00:46:45.559036+0000","last_active":"2024-01-27T00:46:45.566575+0000","last_peered":"2024-01-27T00:46:45.566575+0000","last_clean":"2024-01-27T00:46:45.566575+0000","last_became_active":"2024-01-27T00:46:45.558406+0000","last_became_peered":"2024-01-27T00:46:45.558406+0000","last_unstale":"2024-01-27T00:46:45.566575+0000","last_undegraded":"2024-01-27T00:46:45.566575+0000","last_fullsized":"2024-01-27T00:46:45.566575+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:39:52.381793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"174'2","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.577381+0000","last_change":"2024-01-27T00:46:44.563657+0000","last_active":"2024-01-27T00:46:45.577381+0000","last_peered":"2024-01-27T00:46:45.577381+0000","last_clean":"2024-01-27T00:46:45.577381+0000","last_became_active":"2024-01-27T00:46:44.563390+0000","last_became_peered":"2024-01-27T00:46:44.563390+0000","last_unstale":"2024-01-27T00:46:45.577381+0000","last_undegraded":"2024-01-27T00:46:45.577381+0000","last_fullsized":"2024-01-27T00:46:45.577381+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:35:53.183459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.575950+0000","last_change":"2024-01-27T00:46:45.570906+0000","last_active":"2024-01-27T00:46:45.575950+0000","last_peered":"2024-01-27T00:46:45.575950+0000","last_clean":"2024-01-27T00:46:45.575950+0000","last_became_active":"2024-01-27T00:46:45.570653+0000","last_became_peered":"2024-01-27T00:46:45.570653+0000","last_unstale":"2024-01-27T00:46:45.575950+0000","last_undegraded":"2024-01-27T00:46:45.575950+0000","last_fullsized":"2024-01-27T00:46:45.575950+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:42:57.456320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"174'1","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.577097+0000","last_change":"2024-01-27T00:46:44.559419+0000","last_active":"2024-01-27T00:46:45.577097+0000","last_peered":"2024-01-27T00:46:45.577097+0000","last_clean":"2024-01-27T00:46:45.577097+0000","last_became_active":"2024-01-27T00:46:44.559138+0000","last_became_peered":"2024-01-27T00:46:44.559138+0000","last_unstale":"2024-01-27T00:46:45.577097+0000","last_undegraded":"2024-01-27T00:46:45.577097+0000","last_fullsized":"2024-01-27T00:46:45.577097+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:42:56.451424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.571248+0000","last_change":"2024-01-27T00:46:45.567184+0000","last_active":"2024-01-27T00:46:45.571248+0000","last_peered":"2024-01-27T00:46:45.571248+0000","last_clean":"2024-01-27T00:46:45.571248+0000","last_became_active":"2024-01-27T00:46:45.566901+0000","last_became_peered":"2024-01-27T00:46:45.566901+0000","last_unstale":"2024-01-27T00:46:45.571248+0000","last_undegraded":"2024-01-27T00:46:45.571248+0000","last_fullsized":"2024-01-27T00:46:45.571248+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:17:24.137209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.548227+0000","last_change":"2024-01-27T00:46:44.559439+0000","last_active":"2024-01-27T00:46:45.548227+0000","last_peered":"2024-01-27T00:46:45.548227+0000","last_clean":"2024-01-27T00:46:45.548227+0000","last_became_active":"2024-01-27T00:46:44.558933+0000","last_became_peered":"2024-01-27T00:46:44.558933+0000","last_unstale":"2024-01-27T00:46:45.548227+0000","last_undegraded":"2024-01-27T00:46:45.548227+0000","last_fullsized":"2024-01-27T00:46:45.548227+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:20:58.593757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566429+0000","last_change":"2024-01-27T00:46:45.561881+0000","last_active":"2024-01-27T00:46:45.566429+0000","last_peered":"2024-01-27T00:46:45.566429+0000","last_clean":"2024-01-27T00:46:45.566429+0000","last_became_active":"2024-01-27T00:46:45.561282+0000","last_became_peered":"2024-01-27T00:46:45.561282+0000","last_unstale":"2024-01-27T00:46:45.566429+0000","last_undegraded":"2024-01-27T00:46:45.566429+0000","last_fullsized":"2024-01-27T00:46:45.566429+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:56:30.192694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"174'2","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.576728+0000","last_change":"2024-01-27T00:46:44.560948+0000","last_active":"2024-01-27T00:46:45.576728+0000","last_peered":"2024-01-27T00:46:45.576728+0000","last_clean":"2024-01-27T00:46:45.576728+0000","last_became_active":"2024-01-27T00:46:44.560614+0000","last_became_peered":"2024-01-27T00:46:44.560614+0000","last_unstale":"2024-01-27T00:46:45.576728+0000","last_undegraded":"2024-01-27T00:46:45.576728+0000","last_fullsized":"2024-01-27T00:46:45.576728+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:14:25.383144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.573815+0000","last_change":"2024-01-27T00:46:45.563574+0000","last_active":"2024-01-27T00:46:45.573815+0000","last_peered":"2024-01-27T00:46:45.573815+0000","last_clean":"2024-01-27T00:46:45.573815+0000","last_became_active":"2024-01-27T00:46:45.562151+0000","last_became_peered":"2024-01-27T00:46:45.562151+0000","last_unstale":"2024-01-27T00:46:45.573815+0000","last_undegraded":"2024-01-27T00:46:45.573815+0000","last_fullsized":"2024-01-27T00:46:45.573815+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:24:03.306804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.545921+0000","last_change":"2024-01-27T00:46:44.559051+0000","last_active":"2024-01-27T00:46:45.545921+0000","last_peered":"2024-01-27T00:46:45.545921+0000","last_clean":"2024-01-27T00:46:45.545921+0000","last_became_active":"2024-01-27T00:46:44.558570+0000","last_became_peered":"2024-01-27T00:46:44.558570+0000","last_unstale":"2024-01-27T00:46:45.545921+0000","last_undegraded":"2024-01-27T00:46:45.545921+0000","last_fullsized":"2024-01-27T00:46:45.545921+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:42:56.451424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.565788+0000","last_change":"2024-01-27T00:46:45.557551+0000","last_active":"2024-01-27T00:46:45.565788+0000","last_peered":"2024-01-27T00:46:45.565788+0000","last_clean":"2024-01-27T00:46:45.565788+0000","last_became_active":"2024-01-27T00:46:45.557156+0000","last_became_peered":"2024-01-27T00:46:45.557156+0000","last_unstale":"2024-01-27T00:46:45.565788+0000","last_undegraded":"2024-01-27T00:46:45.565788+0000","last_fullsized":"2024-01-27T00:46:45.565788+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:37:11.759228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.542781+0000","last_change":"2024-01-27T00:46:44.558348+0000","last_active":"2024-01-27T00:46:45.542781+0000","last_peered":"2024-01-27T00:46:45.542781+0000","last_clean":"2024-01-27T00:46:45.542781+0000","last_became_active":"2024-01-27T00:46:44.558037+0000","last_became_peered":"2024-01-27T00:46:44.558037+0000","last_unstale":"2024-01-27T00:46:45.542781+0000","last_undegraded":"2024-01-27T00:46:45.542781+0000","last_fullsized":"2024-01-27T00:46:45.542781+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:44.267221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574844+0000","last_change":"2024-01-27T00:46:45.570182+0000","last_active":"2024-01-27T00:46:45.574844+0000","last_peered":"2024-01-27T00:46:45.574844+0000","last_clean":"2024-01-27T00:46:45.574844+0000","last_became_active":"2024-01-27T00:46:45.568002+0000","last_became_peered":"2024-01-27T00:46:45.568002+0000","last_unstale":"2024-01-27T00:46:45.574844+0000","last_undegraded":"2024-01-27T00:46:45.574844+0000","last_fullsized":"2024-01-27T00:46:45.574844+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:55.735857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.545279+0000","last_change":"2024-01-27T00:46:44.555197+0000","last_active":"2024-01-27T00:46:45.545279+0000","last_peered":"2024-01-27T00:46:45.545279+0000","last_clean":"2024-01-27T00:46:45.545279+0000","last_became_active":"2024-01-27T00:46:44.554948+0000","last_became_peered":"2024-01-27T00:46:44.554948+0000","last_unstale":"2024-01-27T00:46:45.545279+0000","last_undegraded":"2024-01-27T00:46:45.545279+0000","last_fullsized":"2024-01-27T00:46:45.545279+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:55:52.338720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.572165+0000","last_change":"2024-01-27T00:46:45.568251+0000","last_active":"2024-01-27T00:46:45.572165+0000","last_peered":"2024-01-27T00:46:45.572165+0000","last_clean":"2024-01-27T00:46:45.572165+0000","last_became_active":"2024-01-27T00:46:45.567023+0000","last_became_peered":"2024-01-27T00:46:45.567023+0000","last_unstale":"2024-01-27T00:46:45.572165+0000","last_undegraded":"2024-01-27T00:46:45.572165+0000","last_fullsized":"2024-01-27T00:46:45.572165+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:57:28.818719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"174'2","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574142+0000","last_change":"2024-01-27T00:46:44.559295+0000","last_active":"2024-01-27T00:46:45.574142+0000","last_peered":"2024-01-27T00:46:45.574142+0000","last_clean":"2024-01-27T00:46:45.574142+0000","last_became_active":"2024-01-27T00:46:44.558998+0000","last_became_peered":"2024-01-27T00:46:44.558998+0000","last_unstale":"2024-01-27T00:46:45.574142+0000","last_undegraded":"2024-01-27T00:46:45.574142+0000","last_fullsized":"2024-01-27T00:46:45.574142+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:46:04.221575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.567807+0000","last_change":"2024-01-27T00:46:45.565473+0000","last_active":"2024-01-27T00:46:45.567807+0000","last_peered":"2024-01-27T00:46:45.567807+0000","last_clean":"2024-01-27T00:46:45.567807+0000","last_became_active":"2024-01-27T00:46:45.565226+0000","last_became_peered":"2024-01-27T00:46:45.565226+0000","last_unstale":"2024-01-27T00:46:45.567807+0000","last_undegraded":"2024-01-27T00:46:45.567807+0000","last_fullsized":"2024-01-27T00:46:45.567807+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:37:11.759228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.552294+0000","last_change":"2024-01-27T00:46:44.557760+0000","last_active":"2024-01-27T00:46:45.552294+0000","last_peered":"2024-01-27T00:46:45.552294+0000","last_clean":"2024-01-27T00:46:45.552294+0000","last_became_active":"2024-01-27T00:46:44.557025+0000","last_became_peered":"2024-01-27T00:46:44.557025+0000","last_unstale":"2024-01-27T00:46:45.552294+0000","last_undegraded":"2024-01-27T00:46:45.552294+0000","last_fullsized":"2024-01-27T00:46:45.552294+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:59:16.567278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.566375+0000","last_change":"2024-01-27T00:46:45.558408+0000","last_active":"2024-01-27T00:46:45.566375+0000","last_peered":"2024-01-27T00:46:45.566375+0000","last_clean":"2024-01-27T00:46:45.566375+0000","last_became_active":"2024-01-27T00:46:45.557856+0000","last_became_peered":"2024-01-27T00:46:45.557856+0000","last_unstale":"2024-01-27T00:46:45.566375+0000","last_undegraded":"2024-01-27T00:46:45.566375+0000","last_fullsized":"2024-01-27T00:46:45.566375+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:13.453682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.549600+0000","last_change":"2024-01-27T00:46:44.559542+0000","last_active":"2024-01-27T00:46:45.549600+0000","last_peered":"2024-01-27T00:46:45.549600+0000","last_clean":"2024-01-27T00:46:45.549600+0000","last_became_active":"2024-01-27T00:46:44.559213+0000","last_became_peered":"2024-01-27T00:46:44.559213+0000","last_unstale":"2024-01-27T00:46:45.549600+0000","last_undegraded":"2024-01-27T00:46:45.549600+0000","last_fullsized":"2024-01-27T00:46:45.549600+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:55:52.338720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.575672+0000","last_change":"2024-01-27T00:46:45.571078+0000","last_active":"2024-01-27T00:46:45.575672+0000","last_peered":"2024-01-27T00:46:45.575672+0000","last_clean":"2024-01-27T00:46:45.575672+0000","last_became_active":"2024-01-27T00:46:45.570737+0000","last_became_peered":"2024-01-27T00:46:45.570737+0000","last_unstale":"2024-01-27T00:46:45.575672+0000","last_undegraded":"2024-01-27T00:46:45.575672+0000","last_fullsized":"2024-01-27T00:46:45.575672+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:18.466742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.552814+0000","last_change":"2024-01-27T00:46:44.558997+0000","last_active":"2024-01-27T00:46:45.552814+0000","last_peered":"2024-01-27T00:46:45.552814+0000","last_clean":"2024-01-27T00:46:45.552814+0000","last_became_active":"2024-01-27T00:46:44.558732+0000","last_became_peered":"2024-01-27T00:46:44.558732+0000","last_unstale":"2024-01-27T00:46:45.552814+0000","last_undegraded":"2024-01-27T00:46:45.552814+0000","last_fullsized":"2024-01-27T00:46:45.552814+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:35:53.183459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.555465+0000","last_change":"2024-01-27T00:46:45.553448+0000","last_active":"2024-01-27T00:46:45.555465+0000","last_peered":"2024-01-27T00:46:45.555465+0000","last_clean":"2024-01-27T00:46:45.555465+0000","last_became_active":"2024-01-27T00:46:45.553148+0000","last_became_peered":"2024-01-27T00:46:45.553148+0000","last_unstale":"2024-01-27T00:46:45.555465+0000","last_undegraded":"2024-01-27T00:46:45.555465+0000","last_fullsized":"2024-01-27T00:46:45.555465+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:37:09.425394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.549267+0000","last_change":"2024-01-27T00:46:44.558370+0000","last_active":"2024-01-27T00:46:45.549267+0000","last_peered":"2024-01-27T00:46:45.549267+0000","last_clean":"2024-01-27T00:46:45.549267+0000","last_became_active":"2024-01-27T00:46:44.558093+0000","last_became_peered":"2024-01-27T00:46:44.558093+0000","last_unstale":"2024-01-27T00:46:45.549267+0000","last_undegraded":"2024-01-27T00:46:45.549267+0000","last_fullsized":"2024-01-27T00:46:45.549267+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:48:53.017528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.567834+0000","last_change":"2024-01-27T00:46:45.560921+0000","last_active":"2024-01-27T00:46:45.567834+0000","last_peered":"2024-01-27T00:46:45.567834+0000","last_clean":"2024-01-27T00:46:45.567834+0000","last_became_active":"2024-01-27T00:46:45.560664+0000","last_became_peered":"2024-01-27T00:46:45.560664+0000","last_unstale":"2024-01-27T00:46:45.567834+0000","last_undegraded":"2024-01-27T00:46:45.567834+0000","last_fullsized":"2024-01-27T00:46:45.567834+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:09:01.270842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.543666+0000","last_change":"2024-01-27T00:46:44.558090+0000","last_active":"2024-01-27T00:46:45.543666+0000","last_peered":"2024-01-27T00:46:45.543666+0000","last_clean":"2024-01-27T00:46:45.543666+0000","last_became_active":"2024-01-27T00:46:44.557799+0000","last_became_peered":"2024-01-27T00:46:44.557799+0000","last_unstale":"2024-01-27T00:46:45.543666+0000","last_undegraded":"2024-01-27T00:46:45.543666+0000","last_fullsized":"2024-01-27T00:46:45.543666+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:12:01.551412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.574376+0000","last_change":"2024-01-27T00:46:45.569203+0000","last_active":"2024-01-27T00:46:45.574376+0000","last_peered":"2024-01-27T00:46:45.574376+0000","last_clean":"2024-01-27T00:46:45.574376+0000","last_became_active":"2024-01-27T00:46:45.568932+0000","last_became_peered":"2024-01-27T00:46:45.568932+0000","last_unstale":"2024-01-27T00:46:45.574376+0000","last_undegraded":"2024-01-27T00:46:45.574376+0000","last_fullsized":"2024-01-27T00:46:45.574376+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:00:27.609050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.549433+0000","last_change":"2024-01-27T00:46:44.561365+0000","last_active":"2024-01-27T00:46:45.549433+0000","last_peered":"2024-01-27T00:46:45.549433+0000","last_clean":"2024-01-27T00:46:45.549433+0000","last_became_active":"2024-01-27T00:46:44.561061+0000","last_became_peered":"2024-01-27T00:46:44.561061+0000","last_unstale":"2024-01-27T00:46:45.549433+0000","last_undegraded":"2024-01-27T00:46:45.549433+0000","last_fullsized":"2024-01-27T00:46:45.549433+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:58:18.943567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.569800+0000","last_change":"2024-01-27T00:46:45.559506+0000","last_active":"2024-01-27T00:46:45.569800+0000","last_peered":"2024-01-27T00:46:45.569800+0000","last_clean":"2024-01-27T00:46:45.569800+0000","last_became_active":"2024-01-27T00:46:45.559227+0000","last_became_peered":"2024-01-27T00:46:45.559227+0000","last_unstale":"2024-01-27T00:46:45.569800+0000","last_undegraded":"2024-01-27T00:46:45.569800+0000","last_fullsized":"2024-01-27T00:46:45.569800+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:46:05.226471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.554257+0000","last_change":"2024-01-27T00:46:44.562886+0000","last_active":"2024-01-27T00:46:45.554257+0000","last_peered":"2024-01-27T00:46:45.554257+0000","last_clean":"2024-01-27T00:46:45.554257+0000","last_became_active":"2024-01-27T00:46:44.562596+0000","last_became_peered":"2024-01-27T00:46:44.562596+0000","last_unstale":"2024-01-27T00:46:45.554257+0000","last_undegraded":"2024-01-27T00:46:45.554257+0000","last_fullsized":"2024-01-27T00:46:45.554257+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:20:58.593757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.570630+0000","last_change":"2024-01-27T00:46:45.563329+0000","last_active":"2024-01-27T00:46:45.570630+0000","last_peered":"2024-01-27T00:46:45.570630+0000","last_clean":"2024-01-27T00:46:45.570630+0000","last_became_active":"2024-01-27T00:46:45.563031+0000","last_became_peered":"2024-01-27T00:46:45.563031+0000","last_unstale":"2024-01-27T00:46:45.570630+0000","last_undegraded":"2024-01-27T00:46:45.570630+0000","last_fullsized":"2024-01-27T00:46:45.570630+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:44.531810+0000","last_clean_scrub_stamp":"2024-01-27T00:46:44.531810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:23:24.863025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"174'6","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.578394+0000","last_change":"2024-01-27T00:46:44.560987+0000","last_active":"2024-01-27T00:46:45.578394+0000","last_peered":"2024-01-27T00:46:45.578394+0000","last_clean":"2024-01-27T00:46:45.578394+0000","last_became_active":"2024-01-27T00:46:44.560651+0000","last_became_peered":"2024-01-27T00:46:44.560651+0000","last_unstale":"2024-01-27T00:46:45.578394+0000","last_undegraded":"2024-01-27T00:46:45.578394+0000","last_fullsized":"2024-01-27T00:46:45.578394+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:46:43.526914+0000","last_clean_scrub_stamp":"2024-01-27T00:46:43.526914+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:36:05.165696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"165'8673","reported_seq":9348,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-01-27T00:46:45.554345+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T00:46:45.554345+0000","last_peered":"2024-01-27T00:46:45.554345+0000","last_clean":"2024-01-27T00:46:45.554345+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T00:46:45.554345+0000","last_undegraded":"2024-01-27T00:46:45.554345+0000","last_fullsized":"2024-01-27T00:46:45.554345+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8673,"log_dups_size":0,"ondisk_log_size":8673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1592,"num_read_kb":3366,"num_write":490,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1592,"num_read_kb":3366,"num_write":490,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8673,"ondisk_log_size":8673,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575770,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316392,"kb_used_data":2720,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055448,"statfs":{"total":96636764160,"available":96312778752,"internally_reserved":0,"allocated":2785280,"data_stored":1727256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60799999999999998,"15min":0.60699999999999998},"min":{"1min":0.36199999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75800000000000001,"5min":0.83199999999999996,"15min":0.86899999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61299999999999999,"15min":0.61799999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.755,"5min":0.82099999999999995,"15min":0.90000000000000002},"last":0.76300000000000001}]},{"osd":1,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.34599999999999997},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.90400000000000003},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63500000000000001,"15min":0.63},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.38200000000000001},"max":{"1min":0.76700000000000002,"5min":0.85099999999999998,"15min":0.875},"last":0.79600000000000004}]},{"osd":2,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.374,"5min":0.36299999999999999,"15min":0.34000000000000002},"max":{"1min":0.76400000000000001,"5min":0.77700000000000002,"15min":0.89700000000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63800000000000001,"15min":0.63700000000000001},"min":{"1min":0.49399999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75600000000000001,"5min":0.86199999999999999,"15min":0.94199999999999995},"last":0.81100000000000005}]},{"osd":3,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.626,"15min":0.623},"min":{"1min":0.435,"5min":0.40999999999999998,"15min":0.39400000000000002},"max":{"1min":0.79200000000000004,"5min":0.86499999999999999,"15min":0.92600000000000005},"last":0.745},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.77700000000000002,"5min":0.83499999999999996,"15min":0.88400000000000001},"last":0.67600000000000005}]},{"osd":4,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49399999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.32900000000000001,"15min":0.29699999999999999},"max":{"1min":0.61699999999999999,"5min":0.86499999999999999,"15min":0.92000000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.34599999999999997},"max":{"1min":0.76600000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.51700000000000002}]},{"osd":5,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":0.67100000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.627},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":0.67700000000000005,"5min":0.89100000000000001,"15min":0.95699999999999996},"last":0.53700000000000003}]},{"osd":6,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59299999999999997,"15min":0.58299999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.36599999999999999},"max":{"1min":0.69299999999999995,"5min":0.91700000000000004,"15min":0.99399999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.36699999999999999},"max":{"1min":0.73599999999999999,"5min":0.82299999999999995,"15min":0.878},"last":0.56200000000000006}]}]},{"osd":6,"up_from":13,"seq":55834575770,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307620,"kb_used_data":2140,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064220,"statfs":{"total":96636764160,"available":96321761280,"internally_reserved":0,"allocated":2191360,"data_stored":1136936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.60599999999999998,"15min":0.61099999999999999},"min":{"1min":0.38700000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78200000000000003,"5min":0.82199999999999995,"15min":0.84899999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.40999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.43099999999999999}]},{"osd":1,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59999999999999998,"15min":0.59599999999999997},"min":{"1min":0.438,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78700000000000003,"5min":0.79800000000000004,"15min":0.80800000000000005},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61699999999999999,"15min":0.624},"min":{"1min":0.51200000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81999999999999995,"5min":0.87,"15min":0.87},"last":0.63800000000000001}]},{"osd":2,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42199999999999999,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.60599999999999998,"15min":0.60999999999999999},"min":{"1min":0.49099999999999999,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81799999999999995,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.623}]},{"osd":3,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.41999999999999998},"max":{"1min":0.83899999999999997,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.52000000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.79900000000000004,"5min":0.86099999999999999,"15min":0.879},"last":0.57199999999999995}]},{"osd":4,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.497,"15min":0.48899999999999999},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.313},"max":{"1min":0.69099999999999995,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53100000000000003,"15min":0.51700000000000002},"min":{"1min":0.39600000000000002,"5min":0.36699999999999999,"15min":0.32100000000000001},"max":{"1min":0.71399999999999997,"5min":0.79100000000000004,"15min":0.83299999999999996},"last":0.64800000000000002}]},{"osd":5,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52100000000000002,"15min":0.51000000000000001},"min":{"1min":0.39900000000000002,"5min":0.35099999999999998,"15min":0.32200000000000001},"max":{"1min":0.746,"5min":0.75900000000000001,"15min":0.79500000000000004},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56799999999999995,"15min":0.56000000000000005},"min":{"1min":0.42799999999999999,"5min":0.38300000000000001,"15min":0.34899999999999998},"max":{"1min":0.73799999999999999,"5min":0.86199999999999999,"15min":0.88800000000000001},"last":0.67500000000000004}]},{"osd":7,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.58499999999999996,"15min":0.56599999999999995},"min":{"1min":0.39200000000000002,"5min":0.38400000000000001,"15min":0.36799999999999999},"max":{"1min":0.77700000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.59299999999999997,"15min":0.57999999999999996},"min":{"1min":0.45100000000000001,"5min":0.40100000000000002,"15min":0.33600000000000002},"max":{"1min":0.77600000000000002,"5min":0.84599999999999997,"15min":0.85999999999999999},"last":0.68700000000000006}]}]},{"osd":1,"up_from":13,"seq":55834575770,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311716,"kb_used_data":2140,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060124,"statfs":{"total":96636764160,"available":96317566976,"internally_reserved":0,"allocated":2191360,"data_stored":1136936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.28699999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56299999999999994,"5min":0.70499999999999996,"15min":0.71399999999999997},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.44400000000000001,"15min":0.44700000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.66200000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.42999999999999999}]},{"osd":2,"last update":"Sat Jan 27 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.41399999999999998,"15min":0.42499999999999999},"min":{"1min":0.308,"5min":0.28699999999999998,"15min":0.26700000000000002},"max":{"1min":0.54000000000000004,"5min":0.59999999999999998,"15min":0.68899999999999995},"last":0.316},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48199999999999998,"15min":0.49399999999999999},"min":{"1min":0.31,"5min":0.29999999999999999,"15min":0.27900000000000003},"max":{"1min":0.70999999999999996,"5min":0.71799999999999997,"15min":0.76700000000000002},"last":0.46200000000000002}]},{"osd":3,"last update":"Sat Jan 27 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.499,"15min":0.52200000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.66200000000000003,"5min":0.69399999999999995,"15min":1.0469999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.49399999999999999,"15min":0.503},"min":{"1min":0.34300000000000003,"5min":0.32900000000000001,"15min":0.309},"max":{"1min":0.59299999999999997,"5min":0.72599999999999998,"15min":0.95599999999999996},"last":0.5}]},{"osd":4,"last update":"Sat Jan 27 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67200000000000004,"15min":0.68300000000000005},"min":{"1min":0.44700000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.02,"5min":1.02,"15min":1.3129999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68000000000000005,"15min":0.70199999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.097},"last":0.64200000000000002}]},{"osd":5,"last update":"Sat Jan 27 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.67200000000000004,"15min":0.69499999999999995},"min":{"1min":0.48099999999999998,"5min":0.47499999999999998,"15min":0.46300000000000002},"max":{"1min":0.90300000000000002,"5min":0.93100000000000005,"15min":1.091},"last":0.625},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69199999999999995,"15min":0.70499999999999996},"min":{"1min":0.49299999999999999,"5min":0.48199999999999998,"15min":0.36299999999999999},"max":{"1min":0.89200000000000002,"5min":0.96699999999999997,"15min":1.278},"last":0.66400000000000003}]},{"osd":6,"last update":"Sat Jan 27 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.68899999999999995,"15min":0.71499999999999997},"min":{"1min":0.48999999999999999,"5min":0.437,"15min":0.435},"max":{"1min":0.83999999999999997,"5min":0.96999999999999997,"15min":1.141},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.70599999999999996,"15min":0.72199999999999998},"min":{"1min":0.46700000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.96499999999999997,"5min":1.0229999999999999,"15min":1.1120000000000001},"last":0.51600000000000001}]},{"osd":7,"last update":"Sat Jan 27 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.72999999999999998,"15min":0.749},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.51100000000000001},"max":{"1min":0.97299999999999998,"5min":1.054,"15min":1.1819999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.74399999999999999,"15min":0.76200000000000001},"min":{"1min":0.54100000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.98599999999999999,"5min":0.98999999999999999,"15min":1.198},"last":0.68000000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575769,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312316,"kb_used_data":2740,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059524,"statfs":{"total":96636764160,"available":96316952576,"internally_reserved":0,"allocated":2805760,"data_stored":1729448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.41099999999999998,"15min":0.40999999999999998},"min":{"1min":0.316,"5min":0.25700000000000001,"15min":0.25},"max":{"1min":0.56699999999999995,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.45900000000000002,"15min":0.46300000000000002},"min":{"1min":0.28399999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.66100000000000003,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Jan 27 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.41399999999999998,"15min":0.42499999999999999},"min":{"1min":0.29999999999999999,"5min":0.28599999999999998,"15min":0.26700000000000002},"max":{"1min":0.60099999999999998,"5min":0.64300000000000002,"15min":0.69399999999999995},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.47799999999999998,"15min":0.48299999999999998},"min":{"1min":0.36499999999999999,"5min":0.31,"15min":0.307},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.72399999999999998},"last":0.25600000000000001}]},{"osd":3,"last update":"Sat Jan 27 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48699999999999999,"15min":0.48499999999999999},"min":{"1min":0.32300000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64400000000000002,"5min":0.755,"15min":0.755},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.49199999999999999,"15min":0.497},"min":{"1min":0.39000000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.69699999999999995},"last":0.33800000000000002}]},{"osd":4,"last update":"Sat Jan 27 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65800000000000003,"15min":0.68400000000000005},"min":{"1min":0.53300000000000003,"5min":0.44700000000000001,"15min":0.44400000000000001},"max":{"1min":0.91900000000000004,"5min":1.0349999999999999,"15min":1.125},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66400000000000003,"15min":0.68200000000000005},"min":{"1min":0.54200000000000004,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.94399999999999995,"5min":0.97599999999999998,"15min":1.0169999999999999},"last":0.58899999999999997}]},{"osd":5,"last update":"Sat Jan 27 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68400000000000005,"15min":0.70099999999999996},"min":{"1min":0.52500000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":1.054},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.71199999999999997,"15min":0.72099999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.47199999999999998},"max":{"1min":0.97799999999999998,"5min":1.056,"15min":1.056},"last":0.68200000000000005}]},{"osd":6,"last update":"Sat Jan 27 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.70499999999999996,"15min":0.71399999999999997},"min":{"1min":0.50700000000000001,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.89900000000000002,"5min":1.0449999999999999,"15min":1.087},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.72299999999999998,"15min":0.74199999999999999},"min":{"1min":0.48999999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.995,"5min":1.0349999999999999,"15min":1.052},"last":0.79200000000000004}]},{"osd":7,"last update":"Sat Jan 27 00:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.74199999999999999,"15min":0.75900000000000001},"min":{"1min":0.55100000000000005,"5min":0.505,"15min":0.505},"max":{"1min":0.96199999999999997,"5min":1.0880000000000001,"15min":1.1080000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.76600000000000001,"15min":0.78300000000000003},"min":{"1min":0.58499999999999996,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0249999999999999,"5min":1.071,"15min":1.071},"last":0.78100000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575769,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307624,"kb_used_data":2144,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064216,"statfs":{"total":96636764160,"available":96321757184,"internally_reserved":0,"allocated":2195456,"data_stored":1136978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38800000000000001,"15min":0.39800000000000002},"min":{"1min":0.28199999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.59299999999999997,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42999999999999999,"15min":0.44400000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.55000000000000004,"5min":0.66000000000000003,"15min":0.71899999999999997},"last":0.41399999999999998}]},{"osd":1,"last update":"Sat Jan 27 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.42299999999999999,"15min":0.434},"min":{"1min":0.28599999999999998,"5min":0.27700000000000002,"15min":0.27300000000000002},"max":{"1min":0.56799999999999995,"5min":0.627,"15min":0.68300000000000005},"last":0.255},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51100000000000001,"15min":0.51600000000000001},"min":{"1min":0.42599999999999999,"5min":0.38600000000000001,"15min":0.35399999999999998},"max":{"1min":0.64400000000000002,"5min":0.76400000000000001,"15min":0.80300000000000005},"last":0.46100000000000002}]},{"osd":3,"last update":"Sat Jan 27 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48099999999999998,"15min":0.49299999999999999},"min":{"1min":0.35199999999999998,"5min":0.34100000000000003,"15min":0.33100000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47799999999999998,"15min":0.48999999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.29499999999999998},"max":{"1min":0.72499999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.55400000000000005}]},{"osd":4,"last update":"Sat Jan 27 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65800000000000003,"15min":0.67000000000000004},"min":{"1min":0.48899999999999999,"5min":0.44900000000000001,"15min":0.41999999999999998},"max":{"1min":0.85899999999999999,"5min":0.93300000000000005,"15min":1.0660000000000001},"last":0.626},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.67100000000000004,"15min":0.68600000000000005},"min":{"1min":0.48299999999999998,"5min":0.433,"15min":0.433},"max":{"1min":0.79100000000000004,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.48599999999999999}]},{"osd":5,"last update":"Sat Jan 27 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.68000000000000005,"15min":0.69299999999999995},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.42699999999999999},"max":{"1min":0.77100000000000002,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.68999999999999995,"15min":0.70099999999999996},"min":{"1min":0.51300000000000001,"5min":0.46899999999999997,"15min":0.45900000000000002},"max":{"1min":0.89700000000000002,"5min":0.98799999999999999,"15min":1.044},"last":0.44}]},{"osd":6,"last update":"Sat Jan 27 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68700000000000006,"15min":0.70099999999999996},"min":{"1min":0.499,"5min":0.49099999999999999,"15min":0.47599999999999998},"max":{"1min":0.97399999999999998,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.497},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.70099999999999996,"15min":0.71399999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.93400000000000005,"5min":1.002,"15min":1.0720000000000001},"last":0.66400000000000003}]},{"osd":7,"last update":"Sat Jan 27 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.71799999999999997,"15min":0.73199999999999998},"min":{"1min":0.52700000000000002,"5min":0.50900000000000001,"15min":0.48999999999999999},"max":{"1min":0.93600000000000005,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.747,"15min":0.76300000000000001},"min":{"1min":0.56100000000000005,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0069999999999999,"5min":1.022,"15min":1.1299999999999999},"last":0.57899999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575768,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315816,"kb_used_data":2144,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94056024,"statfs":{"total":96636764160,"available":96313368576,"internally_reserved":0,"allocated":2195456,"data_stored":1137454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40100000000000002,"15min":0.40400000000000003},"min":{"1min":0.26800000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.56299999999999994,"5min":0.63,"15min":0.752},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.35499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60199999999999998,"5min":0.71399999999999997,"15min":0.85699999999999998},"last":0.42799999999999999}]},{"osd":1,"last update":"Sat Jan 27 00:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45500000000000002,"15min":0.46700000000000003},"min":{"1min":0.32800000000000001,"5min":0.29699999999999999,"15min":0.27400000000000002},"max":{"1min":0.69499999999999995,"5min":0.69699999999999995,"15min":0.94599999999999995},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.498,"15min":0.5},"min":{"1min":0.38,"5min":0.33400000000000002,"15min":0.316},"max":{"1min":0.65700000000000003,"5min":0.745,"15min":0.745},"last":0.40500000000000003}]},{"osd":2,"last update":"Sat Jan 27 00:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49099999999999999,"15min":0.497},"min":{"1min":0.35899999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.621,"5min":0.67900000000000005,"15min":0.84499999999999997},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.46400000000000002,"15min":0.47399999999999998},"min":{"1min":0.32000000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.82299999999999995},"last":0.35399999999999998}]},{"osd":4,"last update":"Sat Jan 27 00:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69199999999999995},"min":{"1min":0.56699999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.88,"5min":0.89400000000000002,"15min":1},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.47699999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.61299999999999999}]},{"osd":5,"last update":"Sat Jan 27 00:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.69899999999999995,"15min":0.69199999999999995},"min":{"1min":0.51300000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.84999999999999998,"5min":0.97799999999999998,"15min":1.0429999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68999999999999995,"15min":0.69099999999999995},"min":{"1min":0.48499999999999999,"5min":0.44800000000000001,"15min":0.442},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":1.0289999999999999},"last":0.51800000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.68700000000000006,"15min":0.69399999999999995},"min":{"1min":0.60799999999999998,"5min":0.52000000000000002,"15min":0.48899999999999999},"max":{"1min":0.96599999999999997,"5min":0.97199999999999998,"15min":1.0820000000000001},"last":0.623},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.72199999999999998,"15min":0.72099999999999997},"min":{"1min":0.55000000000000004,"5min":0.499,"15min":0.44},"max":{"1min":0.94699999999999995,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.58499999999999996}]},{"osd":7,"last update":"Sat Jan 27 00:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71099999999999997,"15min":0.72099999999999997},"min":{"1min":0.54100000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.76500000000000001,"15min":0.76300000000000001},"min":{"1min":0.63100000000000001,"5min":0.48999999999999999,"15min":0.46300000000000002},"max":{"1min":1.012,"5min":1.0229999999999999,"15min":1.1240000000000001},"last":0.55800000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575770,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311724,"kb_used_data":2148,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060116,"statfs":{"total":96636764160,"available":96317558784,"internally_reserved":0,"allocated":2199552,"data_stored":1138402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61399999999999999,"15min":0.60599999999999998},"min":{"1min":0.40699999999999997,"5min":0.39300000000000002,"15min":0.32100000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.57799999999999996}]},{"osd":1,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.627,"15min":0.628},"min":{"1min":0.52900000000000003,"5min":0.42499999999999999,"15min":0.40999999999999998},"max":{"1min":0.75700000000000001,"5min":0.83199999999999996,"15min":0.86299999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61899999999999999,"15min":0.626},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84699999999999998},"last":0.61199999999999999}]},{"osd":2,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.46100000000000002,"5min":0.436,"15min":0.40899999999999997},"max":{"1min":0.753,"5min":0.81899999999999995,"15min":0.84499999999999997},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.5,"5min":0.44400000000000001,"15min":0.38100000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.63500000000000001}]},{"osd":3,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.626,"15min":0.627},"min":{"1min":0.45900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.61899999999999999,"15min":0.623},"min":{"1min":0.42599999999999999,"5min":0.41199999999999998,"15min":0.41099999999999998},"max":{"1min":0.80600000000000005,"5min":0.80900000000000005,"15min":0.876},"last":0.59799999999999998}]},{"osd":5,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48199999999999998,"15min":0.48499999999999999},"min":{"1min":0.35899999999999999,"5min":0.32100000000000001,"15min":0.28699999999999998},"max":{"1min":0.69699999999999995,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.33100000000000002,"5min":0.30499999999999999,"15min":0.30199999999999999},"max":{"1min":0.77300000000000002,"5min":0.876,"15min":0.876},"last":0.45400000000000001}]},{"osd":6,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49299999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.54100000000000004},"min":{"1min":0.39800000000000002,"5min":0.34300000000000003,"15min":0.32800000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.49199999999999999}]},{"osd":7,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41599999999999998,"5min":0.34999999999999998,"15min":0.33700000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.46000000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.64400000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575770,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307616,"kb_used_data":2136,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064224,"statfs":{"total":96636764160,"available":96321765376,"internally_reserved":0,"allocated":2187264,"data_stored":1136866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61699999999999999,"15min":0.61099999999999999},"min":{"1min":0.42499999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78300000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.628,"15min":0.624},"min":{"1min":0.54100000000000004,"5min":0.41299999999999998,"15min":0.38600000000000001},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.629}]},{"osd":1,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.66200000000000003,"15min":0.65200000000000002},"min":{"1min":0.60099999999999998,"5min":0.47599999999999998,"15min":0.40200000000000002},"max":{"1min":0.81100000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.67900000000000005,"15min":0.67500000000000004},"min":{"1min":0.53500000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.60499999999999998}]},{"osd":2,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.628,"15min":0.60799999999999998},"min":{"1min":0.53400000000000003,"5min":0.39700000000000002,"15min":0.39100000000000001},"max":{"1min":0.78400000000000003,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.623,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.51900000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85599999999999998,"5min":0.876,"15min":0.876},"last":0.38}]},{"osd":3,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.60699999999999998},"min":{"1min":0.45000000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78500000000000003,"5min":0.82799999999999996,"15min":0.83499999999999996},"last":0.499},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65000000000000002,"15min":0.63500000000000001},"min":{"1min":0.53800000000000003,"5min":0.436,"15min":0.42399999999999999},"max":{"1min":0.87,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.56999999999999995}]},{"osd":4,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48499999999999999,"15min":0.48099999999999998},"min":{"1min":0.36799999999999999,"5min":0.30399999999999999,"15min":0.30299999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.39400000000000002,"5min":0.33400000000000002,"15min":0.32700000000000001},"max":{"1min":0.72899999999999998,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55200000000000005}]},{"osd":6,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51400000000000001,"15min":0.504},"min":{"1min":0.40400000000000003,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.68700000000000006,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.41999999999999998,"5min":0.36799999999999999,"15min":0.36099999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.44900000000000001}]},{"osd":7,"last update":"Sat Jan 27 00:45:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57499999999999996,"15min":0.56699999999999995},"min":{"1min":0.38800000000000001,"5min":0.374,"15min":0.34699999999999998},"max":{"1min":0.83899999999999997,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.61199999999999999,"15min":0.60699999999999998},"min":{"1min":0.45400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.82599999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.61899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T00:46:50.156 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-27T00:46:50.157 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-27T00:46:50.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.334+0000 7f2a9fe1f700 1 Processor -- start 2024-01-27T00:46:50.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.334+0000 7f2a9fe1f700 1 -- start start 2024-01-27T00:46:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.335+0000 7f2a9fe1f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a980c8da0 0x7f2a980c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.335+0000 7f2a9fe1f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2a98140af0 0x7f2a98140f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.335+0000 7f2a9fe1f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a980ccc00 con 0x7f2a980caf30 2024-01-27T00:46:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.335+0000 7f2a9fe1f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a980cbbf0 con 0x7f2a980c8da0 2024-01-27T00:46:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.335+0000 7f2a9fe1f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a980cc310 con 0x7f2a98140af0 2024-01-27T00:46:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.335+0000 7f2a9dbbb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a980c8da0 0x7f2a980c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.335+0000 7f2a9dbbb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a980c8da0 0x7f2a980c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41710/0 (socket says 172.21.15.46:41710) 2024-01-27T00:46:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.335+0000 7f2a9dbbb700 1 -- 172.21.15.46:0/2129861642 learned_addr learned my addr 172.21.15.46:0/2129861642 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.335+0000 7f2a9d3ba700 1 --2- 172.21.15.46:0/2129861642 >> v2:172.21.15.106:3300/0 conn(0x7f2a98140af0 0x7f2a98140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.336+0000 7f2a9dbbb700 1 -- 172.21.15.46:0/2129861642 >> v2:172.21.15.106:3300/0 conn(0x7f2a98140af0 msgr2=0x7f2a98140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.336+0000 7f2a9dbbb700 1 --2- 172.21.15.46:0/2129861642 >> v2:172.21.15.106:3300/0 conn(0x7f2a98140af0 0x7f2a98140f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.336+0000 7f2a9dbbb700 1 -- 172.21.15.46:0/2129861642 >> v1:172.21.15.106:6789/0 conn(0x7f2a980caf30 legacy=0x7f2a980cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.336+0000 7f2a9dbbb700 1 -- 172.21.15.46:0/2129861642 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a981414d0 con 0x7f2a980c8da0 2024-01-27T00:46:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.336+0000 7f2a9d3ba700 1 --2- 172.21.15.46:0/2129861642 >> v2:172.21.15.106:3300/0 conn(0x7f2a98140af0 0x7f2a98140f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:46:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.336+0000 7f2a9dbbb700 1 --2- 172.21.15.46:0/2129861642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a980c8da0 0x7f2a980c9170 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f2a980cc120 tx=0x7f2a94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c761edc43efd63a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.336+0000 7f2a9cbb9700 1 -- 172.21.15.46:0/2129861642 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9400b860 con 0x7f2a980c8da0 2024-01-27T00:46:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.336+0000 7f2a9cbb9700 1 -- 172.21.15.46:0/2129861642 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9400b9c0 con 0x7f2a980c8da0 2024-01-27T00:46:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.336+0000 7f2a9cbb9700 1 -- 172.21.15.46:0/2129861642 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9400bd60 con 0x7f2a980c8da0 2024-01-27T00:46:50.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.337+0000 7f2a9fe1f700 1 -- 172.21.15.46:0/2129861642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a980c8da0 msgr2=0x7f2a980c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:50.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.337+0000 7f2a9fe1f700 1 --2- 172.21.15.46:0/2129861642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a980c8da0 0x7f2a980c9170 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f2a980cc120 tx=0x7f2a94009250 comp rx=0 tx=0).stop 2024-01-27T00:46:50.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.337+0000 7f2a9fe1f700 1 -- 172.21.15.46:0/2129861642 shutdown_connections 2024-01-27T00:46:50.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.337+0000 7f2a9fe1f700 1 --2- 172.21.15.46:0/2129861642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a980c8da0 0x7f2a980c9170 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.337+0000 7f2a9fe1f700 1 --2- 172.21.15.46:0/2129861642 >> v2:172.21.15.106:3300/0 conn(0x7f2a98140af0 0x7f2a98140f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.337+0000 7f2a9fe1f700 1 -- 172.21.15.46:0/2129861642 >> 172.21.15.46:0/2129861642 conn(0x7f2a9806bd90 msgr2=0x7f2a9806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:50.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.337+0000 7f2a9fe1f700 1 -- 172.21.15.46:0/2129861642 shutdown_connections 2024-01-27T00:46:50.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.337+0000 7f2a9fe1f700 1 -- 172.21.15.46:0/2129861642 wait complete. 2024-01-27T00:46:50.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.338+0000 7f2a9fe1f700 1 Processor -- start 2024-01-27T00:46:50.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.338+0000 7f2a9fe1f700 1 -- start start 2024-01-27T00:46:50.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.339+0000 7f2a9fe1f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a980c8da0 0x7f2a98133250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:50.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.339+0000 7f2a9fe1f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a980caf30 0x7f2a980c7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:50.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.339+0000 7f2a9fe1f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a98140af0 0x7f2a980c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:50.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.339+0000 7f2a9fe1f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a98133790 con 0x7f2a98140af0 2024-01-27T00:46:50.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.339+0000 7f2a9fe1f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a98068a40 con 0x7f2a980c8da0 2024-01-27T00:46:50.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.339+0000 7f2a9fe1f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a98068d20 con 0x7f2a980caf30 2024-01-27T00:46:50.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.339+0000 7f2a9e3bc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a98140af0 0x7f2a980c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:50.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.339+0000 7f2a9e3bc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a98140af0 0x7f2a980c7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41712/0 (socket says 172.21.15.46:41712) 2024-01-27T00:46:50.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.339+0000 7f2a9e3bc700 1 -- 172.21.15.46:0/2077554179 learned_addr learned my addr 172.21.15.46:0/2077554179 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:50.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.339+0000 7f2a9d3ba700 1 --2- 172.21.15.46:0/2077554179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a980caf30 0x7f2a980c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:50.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.339+0000 7f2a9dbbb700 1 --2- 172.21.15.46:0/2077554179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a980c8da0 0x7f2a98133250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:50.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.339+0000 7f2a9e3bc700 1 -- 172.21.15.46:0/2077554179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a980caf30 msgr2=0x7f2a980c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:50.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.339+0000 7f2a9e3bc700 1 --2- 172.21.15.46:0/2077554179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a980caf30 0x7f2a980c7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.339+0000 7f2a9e3bc700 1 -- 172.21.15.46:0/2077554179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a980c8da0 msgr2=0x7f2a98133250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:50.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.339+0000 7f2a9e3bc700 1 --2- 172.21.15.46:0/2077554179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a980c8da0 0x7f2a98133250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.339+0000 7f2a9e3bc700 1 -- 172.21.15.46:0/2077554179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a940094f0 con 0x7f2a98140af0 2024-01-27T00:46:50.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.340+0000 7f2a9e3bc700 1 --2- 172.21.15.46:0/2077554179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a98140af0 0x7f2a980c7b70 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f2a9000c8e0 tx=0x7f2a90001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:50.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.340+0000 7f2a89ffb700 1 -- 172.21.15.46:0/2077554179 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90012630 con 0x7f2a98140af0 2024-01-27T00:46:50.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.340+0000 7f2a89ffb700 1 -- 172.21.15.46:0/2077554179 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a90012790 con 0x7f2a98140af0 2024-01-27T00:46:50.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.340+0000 7f2a9fe1f700 1 -- 172.21.15.46:0/2077554179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a98069000 con 0x7f2a98140af0 2024-01-27T00:46:50.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.340+0000 7f2a89ffb700 1 -- 172.21.15.46:0/2077554179 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90012b50 con 0x7f2a98140af0 2024-01-27T00:46:50.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.340+0000 7f2a9fe1f700 1 -- 172.21.15.46:0/2077554179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a980695c0 con 0x7f2a98140af0 2024-01-27T00:46:50.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.341+0000 7f2a9fe1f700 1 -- 172.21.15.46:0/2077554179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a980697e0 con 0x7f2a98140af0 2024-01-27T00:46:50.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.342+0000 7f2a89ffb700 1 -- 172.21.15.46:0/2077554179 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2a9001a3a0 con 0x7f2a98140af0 2024-01-27T00:46:50.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.342+0000 7f2a89ffb700 1 --2- 172.21.15.46:0/2077554179 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2a7c066f30 0x7f2a7c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:50.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.342+0000 7f2a89ffb700 1 -- 172.21.15.46:0/2077554179 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f2a90084960 con 0x7f2a98140af0 2024-01-27T00:46:50.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.342+0000 7f2a9dbbb700 1 --2- 172.21.15.46:0/2077554179 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2a7c066f30 0x7f2a7c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:50.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.343+0000 7f2a9dbbb700 1 --2- 172.21.15.46:0/2077554179 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2a7c066f30 0x7f2a7c0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2a980cc120 tx=0x7f2a94017000 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:50.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.346+0000 7f2a89ffb700 1 -- 172.21.15.46:0/2077554179 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2a90055370 con 0x7f2a98140af0 2024-01-27T00:46:50.528 INFO:tasks.ceph.mds.d.smithi106.stderr:2024-01-27T00:46:50.529+0000 7fe3ae9e9700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-27T00:46:50.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.639+0000 7f2a9fe1f700 1 -- 172.21.15.46:0/2077554179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2a9804c710 con 0x7f2a98140af0 2024-01-27T00:46:50.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.643+0000 7f2a89ffb700 1 -- 172.21.15.46:0/2077554179 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v45) v1 ==== 190+0+0 (secure 0 0 0) 0x7f2a90054f00 con 0x7f2a98140af0 2024-01-27T00:46:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.645+0000 7f2a9fe1f700 1 -- 172.21.15.46:0/2077554179 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2a7c066f30 msgr2=0x7f2a7c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.645+0000 7f2a9fe1f700 1 --2- 172.21.15.46:0/2077554179 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2a7c066f30 0x7f2a7c0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2a980cc120 tx=0x7f2a94017000 comp rx=0 tx=0).stop 2024-01-27T00:46:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.645+0000 7f2a9fe1f700 1 -- 172.21.15.46:0/2077554179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a98140af0 msgr2=0x7f2a980c7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.645+0000 7f2a9fe1f700 1 --2- 172.21.15.46:0/2077554179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a98140af0 0x7f2a980c7b70 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f2a9000c8e0 tx=0x7f2a90001710 comp rx=0 tx=0).stop 2024-01-27T00:46:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.645+0000 7f2a9fe1f700 1 -- 172.21.15.46:0/2077554179 shutdown_connections 2024-01-27T00:46:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.645+0000 7f2a9fe1f700 1 --2- 172.21.15.46:0/2077554179 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2a7c066f30 0x7f2a7c0693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.645+0000 7f2a9fe1f700 1 --2- 172.21.15.46:0/2077554179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a98140af0 0x7f2a980c7b70 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.645+0000 7f2a9fe1f700 1 --2- 172.21.15.46:0/2077554179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a980c8da0 0x7f2a98133250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.645+0000 7f2a9fe1f700 1 --2- 172.21.15.46:0/2077554179 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a980caf30 0x7f2a980c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.645+0000 7f2a9fe1f700 1 -- 172.21.15.46:0/2077554179 >> 172.21.15.46:0/2077554179 conn(0x7f2a9806bd90 msgr2=0x7f2a98156f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.645+0000 7f2a9fe1f700 1 -- 172.21.15.46:0/2077554179 shutdown_connections 2024-01-27T00:46:50.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.645+0000 7f2a9fe1f700 1 -- 172.21.15.46:0/2077554179 wait complete. 2024-01-27T00:46:50.646 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-01-27T00:46:50.658 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T00:46:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.791+0000 7fa442648700 1 Processor -- start 2024-01-27T00:46:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.791+0000 7fa442648700 1 -- start start 2024-01-27T00:46:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.792+0000 7fa442648700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa43c0b2530 0x7fa43c15a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.792+0000 7fa442648700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa43c15da10 0x7fa43c15de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.792+0000 7fa442648700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa43c0a9b90 con 0x7fa43c0b2160 2024-01-27T00:46:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.792+0000 7fa442648700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa43c0aa210 con 0x7fa43c15da10 2024-01-27T00:46:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.792+0000 7fa442648700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa43c15d580 con 0x7fa43c0b2530 2024-01-27T00:46:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.792+0000 7fa440be5700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa43c15da10 0x7fa43c15de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.792+0000 7fa43bfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa43c0b2160 0x7fa43c0abe50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60492/0 (socket says 172.21.15.46:60492) 2024-01-27T00:46:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.792+0000 7fa43bfff700 1 -- 172.21.15.46:0/2063755114 learned_addr learned my addr 172.21.15.46:0/2063755114 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.792+0000 7fa43b7fe700 1 --2- 172.21.15.46:0/2063755114 >> v2:172.21.15.106:3300/0 conn(0x7fa43c0b2530 0x7fa43c15a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.792+0000 7fa440be5700 1 -- 172.21.15.46:0/2063755114 >> v2:172.21.15.106:3300/0 conn(0x7fa43c0b2530 msgr2=0x7fa43c15a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.792+0000 7fa440be5700 1 --2- 172.21.15.46:0/2063755114 >> v2:172.21.15.106:3300/0 conn(0x7fa43c0b2530 0x7fa43c15a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.792+0000 7fa440be5700 1 -- 172.21.15.46:0/2063755114 >> v1:172.21.15.46:6789/0 conn(0x7fa43c0b2160 legacy=0x7fa43c0abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.792+0000 7fa440be5700 1 -- 172.21.15.46:0/2063755114 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa43c0aa460 con 0x7fa43c15da10 2024-01-27T00:46:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.792+0000 7fa43affd700 1 -- 172.21.15.46:0/2063755114 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894233783 0 0) 0x7fa43c0a9b90 con 0x7fa43c0b2160 2024-01-27T00:46:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.792+0000 7fa43b7fe700 1 --2- 172.21.15.46:0/2063755114 >> v2:172.21.15.106:3300/0 conn(0x7fa43c0b2530 0x7fa43c15a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:46:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.793+0000 7fa440be5700 1 --2- 172.21.15.46:0/2063755114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa43c15da10 0x7fa43c15de30 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fa43c063c70 tx=0x7fa434009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daee1c0aa1312d3d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.793+0000 7fa43affd700 1 -- 172.21.15.46:0/2063755114 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43400e070 con 0x7fa43c15da10 2024-01-27T00:46:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.793+0000 7fa43affd700 1 -- 172.21.15.46:0/2063755114 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43400bc30 con 0x7fa43c15da10 2024-01-27T00:46:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.793+0000 7fa43affd700 1 -- 172.21.15.46:0/2063755114 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434012430 con 0x7fa43c15da10 2024-01-27T00:46:50.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.793+0000 7fa442648700 1 -- 172.21.15.46:0/2063755114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa43c15da10 msgr2=0x7fa43c15de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:50.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.793+0000 7fa442648700 1 --2- 172.21.15.46:0/2063755114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa43c15da10 0x7fa43c15de30 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fa43c063c70 tx=0x7fa434009780 comp rx=0 tx=0).stop 2024-01-27T00:46:50.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.793+0000 7fa442648700 1 -- 172.21.15.46:0/2063755114 shutdown_connections 2024-01-27T00:46:50.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.793+0000 7fa442648700 1 --2- 172.21.15.46:0/2063755114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa43c15da10 0x7fa43c15de30 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.793+0000 7fa442648700 1 --2- 172.21.15.46:0/2063755114 >> v2:172.21.15.106:3300/0 conn(0x7fa43c0b2530 0x7fa43c15a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.794+0000 7fa442648700 1 -- 172.21.15.46:0/2063755114 >> 172.21.15.46:0/2063755114 conn(0x7fa43c0ccbc0 msgr2=0x7fa43c0ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:50.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.794+0000 7fa442648700 1 -- 172.21.15.46:0/2063755114 shutdown_connections 2024-01-27T00:46:50.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.794+0000 7fa442648700 1 -- 172.21.15.46:0/2063755114 wait complete. 2024-01-27T00:46:50.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.795+0000 7fa442648700 1 Processor -- start 2024-01-27T00:46:50.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.795+0000 7fa442648700 1 -- start start 2024-01-27T00:46:50.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa442648700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa43c0b2160 0x7fa43c158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:50.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa442648700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa43c0b2530 0x7fa43c1692d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:50.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa442648700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa43c15da10 0x7fa43c150720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:50.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa442648700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43c16a320 con 0x7fa43c0b2530 2024-01-27T00:46:50.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa442648700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43c0ad830 con 0x7fa43c15da10 2024-01-27T00:46:50.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa442648700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa43c169810 con 0x7fa43c0b2160 2024-01-27T00:46:50.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa43b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa43c0b2530 0x7fa43c1692d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:50.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa43b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa43c0b2530 0x7fa43c1692d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41716/0 (socket says 172.21.15.46:41716) 2024-01-27T00:46:50.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa43b7fe700 1 -- 172.21.15.46:0/2865616968 learned_addr learned my addr 172.21.15.46:0/2865616968 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:50.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa440be5700 1 --2- 172.21.15.46:0/2865616968 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa43c15da10 0x7fa43c150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:50.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa43bfff700 1 --2- 172.21.15.46:0/2865616968 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa43c0b2160 0x7fa43c158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:50.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa43b7fe700 1 -- 172.21.15.46:0/2865616968 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa43c0b2160 msgr2=0x7fa43c158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:50.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa43b7fe700 1 --2- 172.21.15.46:0/2865616968 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa43c0b2160 0x7fa43c158ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa43b7fe700 1 -- 172.21.15.46:0/2865616968 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa43c15da10 msgr2=0x7fa43c150720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:50.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa43b7fe700 1 --2- 172.21.15.46:0/2865616968 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa43c15da10 0x7fa43c150720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:50.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa43b7fe700 1 -- 172.21.15.46:0/2865616968 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa434009a20 con 0x7fa43c0b2530 2024-01-27T00:46:50.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa440be5700 1 --2- 172.21.15.46:0/2865616968 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa43c15da10 0x7fa43c150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:46:50.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.796+0000 7fa43b7fe700 1 --2- 172.21.15.46:0/2865616968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa43c0b2530 0x7fa43c1692d0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fa43000e900 tx=0x7fa430008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:50.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.797+0000 7fa423fff700 1 -- 172.21.15.46:0/2865616968 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43000eea0 con 0x7fa43c0b2530 2024-01-27T00:46:50.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.797+0000 7fa423fff700 1 -- 172.21.15.46:0/2865616968 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43000c5e0 con 0x7fa43c0b2530 2024-01-27T00:46:50.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.797+0000 7fa423fff700 1 -- 172.21.15.46:0/2865616968 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43000c980 con 0x7fa43c0b2530 2024-01-27T00:46:50.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.797+0000 7fa442648700 1 -- 172.21.15.46:0/2865616968 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa43c169a80 con 0x7fa43c0b2530 2024-01-27T00:46:50.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.797+0000 7fa442648700 1 -- 172.21.15.46:0/2865616968 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa43c16a040 con 0x7fa43c0b2530 2024-01-27T00:46:50.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.798+0000 7fa442648700 1 -- 172.21.15.46:0/2865616968 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa43c0ad9c0 con 0x7fa43c0b2530 2024-01-27T00:46:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.798+0000 7fa423fff700 1 -- 172.21.15.46:0/2865616968 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa430013410 con 0x7fa43c0b2530 2024-01-27T00:46:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.799+0000 7fa423fff700 1 --2- 172.21.15.46:0/2865616968 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa428066f30 0x7fa4280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.799+0000 7fa423fff700 1 -- 172.21.15.46:0/2865616968 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fa4300845b0 con 0x7fa43c0b2530 2024-01-27T00:46:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.799+0000 7fa43bfff700 1 --2- 172.21.15.46:0/2865616968 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa428066f30 0x7fa4280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.799+0000 7fa43bfff700 1 --2- 172.21.15.46:0/2865616968 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa428066f30 0x7fa4280693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa42c00c8f0 tx=0x7fa42c009040 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:50.803+0000 7fa423fff700 1 -- 172.21.15.46:0/2865616968 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa430051530 con 0x7fa43c0b2530 2024-01-27T00:46:51.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.055+0000 7fa442648700 1 -- 172.21.15.46:0/2865616968 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa43c04c710 con 0x7fa43c0b2530 2024-01-27T00:46:51.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.056+0000 7fa423fff700 1 -- 172.21.15.46:0/2865616968 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4607 (secure 0 0 0) 0x7fa430054b50 con 0x7fa43c0b2530 2024-01-27T00:46:51.056 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:46:51.056 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":88,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9061,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/3698053981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":3698053981},{"type":"v1","addr":"172.21.15.106:6833","nonce":3698053981}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":9064,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2056976879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2056976879},{"type":"v1","addr":"172.21.15.46:6834","nonce":2056976879}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":82},{"gid":9067,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1576406314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1576406314},{"type":"v1","addr":"172.21.15.46:6835","nonce":1576406314}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T00:46:44.978654+0000","modified":"2024-01-27T00:46:48.558026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9070":{"gid":9070,"name":"d","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6836/2857098625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2857098625},{"type":"v1","addr":"172.21.15.106:6836","nonce":2857098625}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2024-01-27T00:46:51.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.058+0000 7fa442648700 1 -- 172.21.15.46:0/2865616968 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa428066f30 msgr2=0x7fa4280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.058+0000 7fa442648700 1 --2- 172.21.15.46:0/2865616968 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa428066f30 0x7fa4280693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa42c00c8f0 tx=0x7fa42c009040 comp rx=0 tx=0).stop 2024-01-27T00:46:51.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.058+0000 7fa442648700 1 -- 172.21.15.46:0/2865616968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa43c0b2530 msgr2=0x7fa43c1692d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.058+0000 7fa442648700 1 --2- 172.21.15.46:0/2865616968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa43c0b2530 0x7fa43c1692d0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fa43000e900 tx=0x7fa430008040 comp rx=0 tx=0).stop 2024-01-27T00:46:51.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.059+0000 7fa442648700 1 -- 172.21.15.46:0/2865616968 shutdown_connections 2024-01-27T00:46:51.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.059+0000 7fa442648700 1 --2- 172.21.15.46:0/2865616968 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fa428066f30 0x7fa4280693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.059+0000 7fa442648700 1 --2- 172.21.15.46:0/2865616968 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa43c15da10 0x7fa43c150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.059+0000 7fa442648700 1 --2- 172.21.15.46:0/2865616968 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa43c0b2160 0x7fa43c158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.059+0000 7fa442648700 1 --2- 172.21.15.46:0/2865616968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa43c0b2530 0x7fa43c1692d0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.059+0000 7fa442648700 1 -- 172.21.15.46:0/2865616968 >> 172.21.15.46:0/2865616968 conn(0x7fa43c0ccbc0 msgr2=0x7fa43c0b06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:51.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.059+0000 7fa442648700 1 -- 172.21.15.46:0/2865616968 shutdown_connections 2024-01-27T00:46:51.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.059+0000 7fa442648700 1 -- 172.21.15.46:0/2865616968 wait complete. 2024-01-27T00:46:51.059 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 88 2024-01-27T00:46:51.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-27T00:46:44.978654+0000', 'modified': '2024-01-27T00:46:48.558026+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9070}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9070': {'gid': 9070, 'name': 'd', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.106:6836/2857098625', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6834', 'nonce': 2857098625}, {'type': 'v1', 'addr': '172.21.15.106:6836', 'nonce': 2857098625}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-27T00:46:51.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-27T00:46:51.475 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-27T00:46:51.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.620+0000 7fb950f6a700 1 Processor -- start 2024-01-27T00:46:51.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.620+0000 7fb950f6a700 1 -- start start 2024-01-27T00:46:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.621+0000 7fb950f6a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb94c15ce40 0x7fb94c0d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.621+0000 7fb950f6a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c0cff70 con 0x7fb94c0cc1c0 2024-01-27T00:46:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.621+0000 7fb950f6a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c0cf400 con 0x7fb94c15d210 2024-01-27T00:46:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.621+0000 7fb950f6a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb94c0cf660 con 0x7fb94c15ce40 2024-01-27T00:46:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.621+0000 7fb94ad9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb94c0cc1c0 0x7fb94c0ade30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60496/0 (socket says 172.21.15.46:60496) 2024-01-27T00:46:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.621+0000 7fb94ad9d700 1 -- 172.21.15.46:0/458139754 learned_addr learned my addr 172.21.15.46:0/458139754 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.622+0000 7fb94a59c700 1 --2- 172.21.15.46:0/458139754 >> v2:172.21.15.106:3300/0 conn(0x7fb94c15ce40 0x7fb94c0d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.622+0000 7fb94959a700 1 -- 172.21.15.46:0/458139754 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719466217 0 0) 0x7fb94c0cff70 con 0x7fb94c0cc1c0 2024-01-27T00:46:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.622+0000 7fb94959a700 1 -- 172.21.15.46:0/458139754 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb934003570 con 0x7fb94c0cc1c0 2024-01-27T00:46:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.622+0000 7fb94a59c700 1 -- 172.21.15.46:0/458139754 >> v1:172.21.15.106:6789/0 conn(0x7fb94c15d210 legacy=0x7fb94c0cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.622+0000 7fb94959a700 1 -- 172.21.15.46:0/458139754 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1309590261 0 0) 0x7fb934003570 con 0x7fb94c0cc1c0 2024-01-27T00:46:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.622+0000 7fb94a59c700 1 -- 172.21.15.46:0/458139754 >> v1:172.21.15.46:6789/0 conn(0x7fb94c0cc1c0 legacy=0x7fb94c0ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.622+0000 7fb94a59c700 1 -- 172.21.15.46:0/458139754 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c0d0500 con 0x7fb94c15ce40 2024-01-27T00:46:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.622+0000 7fb94a59c700 1 --2- 172.21.15.46:0/458139754 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb94c15ce40 0x7fb94c0d0e50 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0098d0 tx=0x7fb93c009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8189f0730824d3b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.622+0000 7fb94959a700 1 -- 172.21.15.46:0/458139754 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9440017f0 con 0x7fb94c15ce40 2024-01-27T00:46:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.623+0000 7fb94959a700 1 -- 172.21.15.46:0/458139754 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93c011420 con 0x7fb94c15ce40 2024-01-27T00:46:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.623+0000 7fb94959a700 1 -- 172.21.15.46:0/458139754 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c0117e0 con 0x7fb94c15ce40 2024-01-27T00:46:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.623+0000 7fb950f6a700 1 -- 172.21.15.46:0/458139754 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb94c15ce40 msgr2=0x7fb94c0d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.623+0000 7fb950f6a700 1 --2- 172.21.15.46:0/458139754 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb94c15ce40 0x7fb94c0d0e50 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0098d0 tx=0x7fb93c009120 comp rx=0 tx=0).stop 2024-01-27T00:46:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.623+0000 7fb950f6a700 1 -- 172.21.15.46:0/458139754 shutdown_connections 2024-01-27T00:46:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.623+0000 7fb950f6a700 1 --2- 172.21.15.46:0/458139754 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb94c15ce40 0x7fb94c0d0e50 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.623+0000 7fb950f6a700 1 -- 172.21.15.46:0/458139754 >> 172.21.15.46:0/458139754 conn(0x7fb94c06f840 msgr2=0x7fb94c0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.623+0000 7fb950f6a700 1 -- 172.21.15.46:0/458139754 shutdown_connections 2024-01-27T00:46:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.623+0000 7fb950f6a700 1 -- 172.21.15.46:0/458139754 wait complete. 2024-01-27T00:46:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.624+0000 7fb950f6a700 1 Processor -- start 2024-01-27T00:46:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.624+0000 7fb950f6a700 1 -- start start 2024-01-27T00:46:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.625+0000 7fb950f6a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb94c0cc1c0 0x7fb94c16d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.625+0000 7fb950f6a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb94c15ce40 0x7fb94c16b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.625+0000 7fb950f6a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c15d210 0x7fb94c157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.625+0000 7fb950f6a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c158440 con 0x7fb94c15d210 2024-01-27T00:46:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.625+0000 7fb950f6a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c0c96c0 con 0x7fb94c0cc1c0 2024-01-27T00:46:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.625+0000 7fb950f6a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c16d970 con 0x7fb94c15ce40 2024-01-27T00:46:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.625+0000 7fb94ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c15d210 0x7fb94c157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.625+0000 7fb94a59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb94c0cc1c0 0x7fb94c16d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.625+0000 7fb94ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c15d210 0x7fb94c157f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41724/0 (socket says 172.21.15.46:41724) 2024-01-27T00:46:51.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.625+0000 7fb94a59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb94c0cc1c0 0x7fb94c16d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:60288/0 (socket says 172.21.15.46:60288) 2024-01-27T00:46:51.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.625+0000 7fb94ad9d700 1 -- 172.21.15.46:0/1831908251 learned_addr learned my addr 172.21.15.46:0/1831908251 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:51.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.626+0000 7fb949d9b700 1 --2- 172.21.15.46:0/1831908251 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb94c15ce40 0x7fb94c16b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.626+0000 7fb94a59c700 1 -- 172.21.15.46:0/1831908251 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb94c15ce40 msgr2=0x7fb94c16b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.626+0000 7fb94a59c700 1 --2- 172.21.15.46:0/1831908251 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb94c15ce40 0x7fb94c16b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.626+0000 7fb94a59c700 1 -- 172.21.15.46:0/1831908251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c15d210 msgr2=0x7fb94c157f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.626+0000 7fb94a59c700 1 --2- 172.21.15.46:0/1831908251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c15d210 0x7fb94c157f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.626+0000 7fb94a59c700 1 -- 172.21.15.46:0/1831908251 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb93c0093c0 con 0x7fb94c0cc1c0 2024-01-27T00:46:51.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.626+0000 7fb94ad9d700 1 --2- 172.21.15.46:0/1831908251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c15d210 0x7fb94c157f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:51.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.626+0000 7fb94a59c700 1 --2- 172.21.15.46:0/1831908251 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb94c0cc1c0 0x7fb94c16d2d0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fb93c014000 tx=0x7fb93c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:51.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.626+0000 7fb93a7fc700 1 -- 172.21.15.46:0/1831908251 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c010af0 con 0x7fb94c0cc1c0 2024-01-27T00:46:51.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.626+0000 7fb93a7fc700 1 -- 172.21.15.46:0/1831908251 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93c010c50 con 0x7fb94c0cc1c0 2024-01-27T00:46:51.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.626+0000 7fb93a7fc700 1 -- 172.21.15.46:0/1831908251 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c01d430 con 0x7fb94c0cc1c0 2024-01-27T00:46:51.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.626+0000 7fb950f6a700 1 -- 172.21.15.46:0/1831908251 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94c16dc50 con 0x7fb94c0cc1c0 2024-01-27T00:46:51.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.626+0000 7fb950f6a700 1 -- 172.21.15.46:0/1831908251 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94c16a740 con 0x7fb94c0cc1c0 2024-01-27T00:46:51.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.628+0000 7fb93a7fc700 1 -- 172.21.15.46:0/1831908251 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb93c01d840 con 0x7fb94c0cc1c0 2024-01-27T00:46:51.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.628+0000 7fb93a7fc700 1 --2- 172.21.15.46:0/1831908251 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb92c066f30 0x7fb92c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.628+0000 7fb93a7fc700 1 -- 172.21.15.46:0/1831908251 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fb93c086470 con 0x7fb94c0cc1c0 2024-01-27T00:46:51.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.629+0000 7fb949d9b700 1 --2- 172.21.15.46:0/1831908251 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb92c066f30 0x7fb92c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.629+0000 7fb949d9b700 1 --2- 172.21.15.46:0/1831908251 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb92c066f30 0x7fb92c0693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb94c15d620 tx=0x7fb940007ec0 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:51.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.630+0000 7fb95a26db80 1 Processor -- start 2024-01-27T00:46:51.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.631+0000 7fb95a26db80 1 -- start start 2024-01-27T00:46:51.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.631+0000 7fb95a26db80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5b6010 0x561cef5b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.631+0000 7fb95a26db80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5fe360 0x561cef5e0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.631+0000 7fb95a26db80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b55d0 0x561cef5e0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.631+0000 7fb95a26db80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cef4b99f0 con 0x561cef5fe360 2024-01-27T00:46:51.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.631+0000 7fb95a26db80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cef50b130 con 0x561cef5b55d0 2024-01-27T00:46:51.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.631+0000 7fb95a26db80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x561cef508930 con 0x561cef5b6010 2024-01-27T00:46:51.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.631+0000 7fb949d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5fe360 0x561cef5e0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.632+0000 7fb949d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5fe360 0x561cef5e0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41732/0 (socket says 172.21.15.46:41732) 2024-01-27T00:46:51.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.632+0000 7fb94ad9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b55d0 0x561cef5e0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.632+0000 7fb949d9b700 1 -- 172.21.15.46:0/1706372391 learned_addr learned my addr 172.21.15.46:0/1706372391 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:51.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.632+0000 7fb94a59c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5b6010 0x561cef5b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.632+0000 7fb949d9b700 1 -- 172.21.15.46:0/1706372391 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5b6010 msgr2=0x561cef5b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:51.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.632+0000 7fb949d9b700 1 --2- 172.21.15.46:0/1706372391 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5b6010 0x561cef5b63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.632+0000 7fb949d9b700 1 -- 172.21.15.46:0/1706372391 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b55d0 msgr2=0x561cef5e0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:51.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.632+0000 7fb949d9b700 1 --2- 172.21.15.46:0/1706372391 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b55d0 0x561cef5e0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.632+0000 7fb949d9b700 1 -- 172.21.15.46:0/1706372391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561cef343b10 con 0x561cef5fe360 2024-01-27T00:46:51.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.632+0000 7fb94a59c700 1 --2- 172.21.15.46:0/1706372391 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5b6010 0x561cef5b63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:51.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.632+0000 7fb94ad9d700 1 --2- 172.21.15.46:0/1706372391 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b55d0 0x561cef5e0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:51.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.632+0000 7fb949d9b700 1 --2- 172.21.15.46:0/1706372391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5fe360 0x561cef5e0840 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fb940002c50 tx=0x7fb940015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe77876038d75a8d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:51.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.632+0000 7fb92b7fe700 1 -- 172.21.15.46:0/1706372391 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9400148e0 con 0x561cef5fe360 2024-01-27T00:46:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.632+0000 7fb92b7fe700 1 -- 172.21.15.46:0/1706372391 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb940014a40 con 0x561cef5fe360 2024-01-27T00:46:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.632+0000 7fb92b7fe700 1 -- 172.21.15.46:0/1706372391 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb940014e00 con 0x561cef5fe360 2024-01-27T00:46:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.633+0000 7fb95a26db80 1 -- 172.21.15.46:0/1706372391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5fe360 msgr2=0x561cef5e0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.633+0000 7fb95a26db80 1 --2- 172.21.15.46:0/1706372391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5fe360 0x561cef5e0840 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fb940002c50 tx=0x7fb940015000 comp rx=0 tx=0).stop 2024-01-27T00:46:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.633+0000 7fb95a26db80 1 -- 172.21.15.46:0/1706372391 shutdown_connections 2024-01-27T00:46:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.633+0000 7fb95a26db80 1 --2- 172.21.15.46:0/1706372391 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b55d0 0x561cef5e0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.633+0000 7fb95a26db80 1 --2- 172.21.15.46:0/1706372391 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5b6010 0x561cef5b63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.633+0000 7fb95a26db80 1 --2- 172.21.15.46:0/1706372391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5fe360 0x561cef5e0840 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.633+0000 7fb95a26db80 1 -- 172.21.15.46:0/1706372391 >> 172.21.15.46:0/1706372391 conn(0x561cef5b5a70 msgr2=0x561cef5fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.633+0000 7fb95a26db80 1 -- 172.21.15.46:0/1706372391 shutdown_connections 2024-01-27T00:46:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.633+0000 7fb95a26db80 1 -- 172.21.15.46:0/1706372391 wait complete. 2024-01-27T00:46:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.634+0000 7fb95a26db80 1 -- start start 2024-01-27T00:46:51.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.634+0000 7fb95a26db80 1 Processor -- start 2024-01-27T00:46:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb95a26db80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5b55d0 0x561cef5d99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb95a26db80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b6010 0x561cef5fce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb95a26db80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5fe360 0x561cef5f9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb95a26db80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cef504720 con 0x561cef5b55d0 2024-01-27T00:46:51.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb95a26db80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cef4b01b0 con 0x561cef5b6010 2024-01-27T00:46:51.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb95a26db80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x561cef4afe50 con 0x561cef5fe360 2024-01-27T00:46:51.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb94a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5b55d0 0x561cef5d99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb95a26db80 10 client.0 _ll_register_callbacks cb 0x561cef3127b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-27T00:46:51.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb94a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5b55d0 0x561cef5d99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41740/0 (socket says 172.21.15.46:41740) 2024-01-27T00:46:51.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb94a59c700 1 -- 172.21.15.46:0/4240433397 learned_addr learned my addr 172.21.15.46:0/4240433397 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:51.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb94ad9d700 1 --2- 172.21.15.46:0/4240433397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5fe360 0x561cef5f9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb949d9b700 1 --2- 172.21.15.46:0/4240433397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b6010 0x561cef5fce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb94a59c700 1 -- 172.21.15.46:0/4240433397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5fe360 msgr2=0x561cef5f9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:51.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb94a59c700 1 --2- 172.21.15.46:0/4240433397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5fe360 0x561cef5f9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb94a59c700 1 -- 172.21.15.46:0/4240433397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b6010 msgr2=0x561cef5fce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:51.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb94a59c700 1 --2- 172.21.15.46:0/4240433397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b6010 0x561cef5fce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb94a59c700 1 -- 172.21.15.46:0/4240433397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb944004cb0 con 0x561cef5b55d0 2024-01-27T00:46:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb94ad9d700 1 --2- 172.21.15.46:0/4240433397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5fe360 0x561cef5f9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:46:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb949d9b700 1 --2- 172.21.15.46:0/4240433397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b6010 0x561cef5fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:46:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb94a59c700 1 --2- 172.21.15.46:0/4240433397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5b55d0 0x561cef5d99f0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fb93c011950 tx=0x7fb93c01f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46734ae0dca5d541 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.635+0000 7fb928ff9700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-27T00:46:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.636+0000 7fb928ff9700 1 -- 172.21.15.46:0/4240433397 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c05a3a0 con 0x561cef5b55d0 2024-01-27T00:46:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.636+0000 7fb928ff9700 1 -- 172.21.15.46:0/4240433397 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(175..175 src has 1..175) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fb93c05e070 con 0x561cef5b55d0 2024-01-27T00:46:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.636+0000 7fb95a26db80 1 -- 172.21.15.46:0/4240433397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561cef5a9190 con 0x561cef5b55d0 2024-01-27T00:46:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.636+0000 7fb928ff9700 1 -- 172.21.15.46:0/4240433397 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb93c0842a0 con 0x561cef5b55d0 2024-01-27T00:46:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.636+0000 7fb95a26db80 10 client.9498 fetch_fsmap learned FSMap version 88 2024-01-27T00:46:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.636+0000 7fb95a26db80 1 -- 172.21.15.46:0/4240433397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) v3 -- 0x7fb94c16a740 con 0x561cef5b55d0 2024-01-27T00:46:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.636+0000 7fb928ff9700 1 -- 172.21.15.46:0/4240433397 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561cef5a9190 con 0x561cef5b55d0 2024-01-27T00:46:51.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.636+0000 7fb928ff9700 1 -- 172.21.15.46:0/4240433397 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 88) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fb93c018030 con 0x561cef5b55d0 2024-01-27T00:46:51.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.636+0000 7fb95a26db80 10 client.9498 fetch_fsmap finished waiting for FSMap version 88 2024-01-27T00:46:51.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.636+0000 7fb95a26db80 10 client.9498 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-27T00:46:51.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.636+0000 7fb95a26db80 20 client.9498 populate_metadata read hostname 'smithi046' 2024-01-27T00:46:51.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.636+0000 7fb95a26db80 1 --2- 172.21.15.46:0/4240433397 >> [v2:172.21.15.106:6834/2857098625,v1:172.21.15.106:6836/2857098625] conn(0x561cef5df6e0 0x561cef5dfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.636+0000 7fb95a26db80 4 client.9498 mds_command: new command op to 9070 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-27T00:46:51.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.636+0000 7fb95a26db80 1 -- 172.21.15.46:0/4240433397 --> [v2:172.21.15.106:6834/2857098625,v1:172.21.15.106:6836/2857098625] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561cef4a6ee0 con 0x561cef5df6e0 2024-01-27T00:46:51.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.636+0000 7fb949d9b700 1 --2- 172.21.15.46:0/4240433397 >> [v2:172.21.15.106:6834/2857098625,v1:172.21.15.106:6836/2857098625] conn(0x561cef5df6e0 0x561cef5dfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.637+0000 7fb928ff9700 10 client.9498 ms_handle_connect on v2:172.21.15.106:6834/2857098625 2024-01-27T00:46:51.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.637+0000 7fb949d9b700 1 --2- 172.21.15.46:0/4240433397 >> [v2:172.21.15.106:6834/2857098625,v1:172.21.15.106:6836/2857098625] conn(0x561cef5df6e0 0x561cef5dfab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c7b198519a5c47e server_cookie=2cc536c15c0789a8 in_seq=0 out_seq=0 2024-01-27T00:46:51.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.638+0000 7fb928ff9700 1 -- 172.21.15.46:0/4240433397 <== mds.0 v2:172.21.15.106:6834/2857098625 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x561cef4a6ee0 con 0x561cef5df6e0 2024-01-27T00:46:51.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.638+0000 7fb928ff9700 10 client.9498 handle_command_reply: tid=0 2024-01-27T00:46:51.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.638+0000 7fb95a26db80 1 client.9498 shutdown 2024-01-27T00:46:51.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.638+0000 7fb95a26db80 1 -- 172.21.15.46:0/4240433397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5b55d0 msgr2=0x561cef5d99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.638+0000 7fb95a26db80 1 --2- 172.21.15.46:0/4240433397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5b55d0 0x561cef5d99f0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fb93c011950 tx=0x7fb93c01f040 comp rx=0 tx=0).stop 2024-01-27T00:46:51.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.638+0000 7fb95a26db80 1 -- 172.21.15.46:0/4240433397 shutdown_connections 2024-01-27T00:46:51.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.638+0000 7fb95a26db80 1 --2- 172.21.15.46:0/4240433397 >> [v2:172.21.15.106:6834/2857098625,v1:172.21.15.106:6836/2857098625] conn(0x561cef5df6e0 0x561cef5dfab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.638+0000 7fb95a26db80 1 --2- 172.21.15.46:0/4240433397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5b55d0 0x561cef5d99f0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.638+0000 7fb95a26db80 1 --2- 172.21.15.46:0/4240433397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5fe360 0x561cef5f9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.638+0000 7fb95a26db80 1 --2- 172.21.15.46:0/4240433397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b6010 0x561cef5fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.638+0000 7fb928ff9700 0 client.9498 ms_handle_reset on v2:172.21.15.106:6834/2857098625 2024-01-27T00:46:51.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.638+0000 7fb95a26db80 1 -- 172.21.15.46:0/4240433397 >> 172.21.15.46:0/4240433397 conn(0x561cef5b5a70 msgr2=0x561cef5af240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:51.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.638+0000 7fb95a26db80 1 -- 172.21.15.46:0/4240433397 shutdown_connections 2024-01-27T00:46:51.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.638+0000 7fb95a26db80 1 -- 172.21.15.46:0/4240433397 wait complete. 2024-01-27T00:46:51.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.638+0000 7fb95a26db80 20 client.9498 trim_cache size 0 max 16384 2024-01-27T00:46:51.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.656+0000 7fb95a26db80 1 Processor -- start 2024-01-27T00:46:51.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.656+0000 7fb95a26db80 1 -- start start 2024-01-27T00:46:51.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.657+0000 7fb95a26db80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5b55d0 0x561cef5b0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.657+0000 7fb95a26db80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b6010 0x561cef5f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.657+0000 7fb95a26db80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5fe360 0x561cef60d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.657+0000 7fb95a26db80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cef3fce40 con 0x561cef5b55d0 2024-01-27T00:46:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.657+0000 7fb95a26db80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cef5d97a0 con 0x561cef5b6010 2024-01-27T00:46:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.657+0000 7fb95a26db80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x561cef5b1a00 con 0x561cef5fe360 2024-01-27T00:46:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.657+0000 7fb94a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5b55d0 0x561cef5b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.657+0000 7fb94a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5b55d0 0x561cef5b0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41752/0 (socket says 172.21.15.46:41752) 2024-01-27T00:46:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.657+0000 7fb94a59c700 1 -- 172.21.15.46:0/1927424545 learned_addr learned my addr 172.21.15.46:0/1927424545 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.657+0000 7fb94ad9d700 1 --2- 172.21.15.46:0/1927424545 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5fe360 0x561cef60d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.657+0000 7fb949d9b700 1 --2- 172.21.15.46:0/1927424545 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b6010 0x561cef5f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.658+0000 7fb94a59c700 1 -- 172.21.15.46:0/1927424545 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5fe360 msgr2=0x561cef60d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.658+0000 7fb94a59c700 1 --2- 172.21.15.46:0/1927424545 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5fe360 0x561cef60d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.658+0000 7fb94a59c700 1 -- 172.21.15.46:0/1927424545 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b6010 msgr2=0x561cef5f51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.658+0000 7fb94a59c700 1 --2- 172.21.15.46:0/1927424545 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b6010 0x561cef5f51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.658+0000 7fb94a59c700 1 -- 172.21.15.46:0/1927424545 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb940002800 con 0x561cef5b55d0 2024-01-27T00:46:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.658+0000 7fb94ad9d700 1 --2- 172.21.15.46:0/1927424545 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5fe360 0x561cef60d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:46:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.658+0000 7fb94a59c700 1 --2- 172.21.15.46:0/1927424545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5b55d0 0x561cef5b0690 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fb93c056dd0 tx=0x7fb93c06e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77649480fb4d6c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.658+0000 7fb92bfff700 1 -- 172.21.15.46:0/1927424545 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c0862a0 con 0x561cef5b55d0 2024-01-27T00:46:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.658+0000 7fb92bfff700 1 -- 172.21.15.46:0/1927424545 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93c072e90 con 0x561cef5b55d0 2024-01-27T00:46:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.658+0000 7fb92bfff700 1 -- 172.21.15.46:0/1927424545 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c0860c0 con 0x561cef5b55d0 2024-01-27T00:46:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.659+0000 7fb95a26db80 1 -- 172.21.15.46:0/1927424545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5b55d0 msgr2=0x561cef5b0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.659+0000 7fb95a26db80 1 --2- 172.21.15.46:0/1927424545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5b55d0 0x561cef5b0690 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fb93c056dd0 tx=0x7fb93c06e000 comp rx=0 tx=0).stop 2024-01-27T00:46:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.659+0000 7fb95a26db80 1 -- 172.21.15.46:0/1927424545 shutdown_connections 2024-01-27T00:46:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.659+0000 7fb95a26db80 1 --2- 172.21.15.46:0/1927424545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5b55d0 0x561cef5b0690 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.659+0000 7fb95a26db80 1 --2- 172.21.15.46:0/1927424545 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5fe360 0x561cef60d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.659+0000 7fb95a26db80 1 --2- 172.21.15.46:0/1927424545 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b6010 0x561cef5f51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.659+0000 7fb95a26db80 1 -- 172.21.15.46:0/1927424545 >> 172.21.15.46:0/1927424545 conn(0x561cef5b5a70 msgr2=0x561cef5fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:51.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.659+0000 7fb95a26db80 1 -- 172.21.15.46:0/1927424545 shutdown_connections 2024-01-27T00:46:51.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.659+0000 7fb95a26db80 1 -- 172.21.15.46:0/1927424545 wait complete. 2024-01-27T00:46:51.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.659+0000 7fb95a26db80 1 -- start start 2024-01-27T00:46:51.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.659+0000 7fb95a26db80 1 Processor -- start 2024-01-27T00:46:51.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.660+0000 7fb95a26db80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b55d0 0x561cef60c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.660+0000 7fb95a26db80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5b6010 0x561cef5b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.660+0000 7fb95a26db80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5fe360 0x561cef5b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.660+0000 7fb95a26db80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cef5b4a00 con 0x561cef5fe360 2024-01-27T00:46:51.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.660+0000 7fb95a26db80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cef5b4b70 con 0x561cef5b55d0 2024-01-27T00:46:51.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.660+0000 7fb95a26db80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x561cef5de860 con 0x561cef5b6010 2024-01-27T00:46:51.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.660+0000 7fb95a26db80 10 client.0 _ll_register_callbacks cb 0x7fb93c056f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-27T00:46:51.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.660+0000 7fb94ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5fe360 0x561cef5b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.661+0000 7fb94ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5fe360 0x561cef5b44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41758/0 (socket says 172.21.15.46:41758) 2024-01-27T00:46:51.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.661+0000 7fb94ad9d700 1 -- 172.21.15.46:0/1616767026 learned_addr learned my addr 172.21.15.46:0/1616767026 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:51.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.661+0000 7fb94a59c700 1 --2- 172.21.15.46:0/1616767026 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b55d0 0x561cef60c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.661+0000 7fb949d9b700 1 --2- 172.21.15.46:0/1616767026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5b6010 0x561cef5b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.661+0000 7fb94ad9d700 1 -- 172.21.15.46:0/1616767026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5b6010 msgr2=0x561cef5b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:51.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.661+0000 7fb94ad9d700 1 --2- 172.21.15.46:0/1616767026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5b6010 0x561cef5b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.661+0000 7fb94ad9d700 1 -- 172.21.15.46:0/1616767026 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b55d0 msgr2=0x561cef60c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:51.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.661+0000 7fb94ad9d700 1 --2- 172.21.15.46:0/1616767026 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b55d0 0x561cef60c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.661+0000 7fb94ad9d700 1 -- 172.21.15.46:0/1616767026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94c16a740 con 0x561cef5fe360 2024-01-27T00:46:51.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.661+0000 7fb94ad9d700 1 --2- 172.21.15.46:0/1616767026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5fe360 0x561cef5b44c0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0c9670 tx=0x7fb944009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290d3b5aa3332511 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:51.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.661+0000 7fb928ff9700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-27T00:46:51.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.661+0000 7fb928ff9700 1 -- 172.21.15.46:0/1616767026 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb944020430 con 0x561cef5fe360 2024-01-27T00:46:51.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.661+0000 7fb928ff9700 1 -- 172.21.15.46:0/1616767026 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(175..175 src has 1..175) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fb94401cc30 con 0x561cef5fe360 2024-01-27T00:46:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.661+0000 7fb95a26db80 1 -- 172.21.15.46:0/1616767026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb93c05a3a0 con 0x561cef5fe360 2024-01-27T00:46:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.661+0000 7fb928ff9700 1 -- 172.21.15.46:0/1616767026 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb94401cdf0 con 0x561cef5fe360 2024-01-27T00:46:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.661+0000 7fb95a26db80 10 client.9504 fetch_fsmap learned FSMap version 88 2024-01-27T00:46:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.661+0000 7fb95a26db80 1 -- 172.21.15.46:0/1616767026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) v3 -- 0x561cef4d2290 con 0x561cef5fe360 2024-01-27T00:46:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.662+0000 7fb928ff9700 1 -- 172.21.15.46:0/1616767026 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93c05a3a0 con 0x561cef5fe360 2024-01-27T00:46:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.662+0000 7fb928ff9700 1 -- 172.21.15.46:0/1616767026 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 88) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fb944012050 con 0x561cef5fe360 2024-01-27T00:46:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.662+0000 7fb95a26db80 10 client.9504 fetch_fsmap finished waiting for FSMap version 88 2024-01-27T00:46:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.662+0000 7fb95a26db80 10 client.9504 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-27T00:46:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.662+0000 7fb95a26db80 20 client.9504 populate_metadata read hostname 'smithi046' 2024-01-27T00:46:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.662+0000 7fb95a26db80 1 --2- 172.21.15.46:0/1616767026 >> [v2:172.21.15.106:6834/2857098625,v1:172.21.15.106:6836/2857098625] conn(0x561cef5d7cb0 0x561cef5d8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.662+0000 7fb95a26db80 4 client.9504 mds_command: new command op to 9070 tid=0[{"prefix": "status"}] 2024-01-27T00:46:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.662+0000 7fb95a26db80 1 -- 172.21.15.46:0/1616767026 --> [v2:172.21.15.106:6834/2857098625,v1:172.21.15.106:6836/2857098625] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fb940014a40 con 0x561cef5d7cb0 2024-01-27T00:46:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.662+0000 7fb94a59c700 1 --2- 172.21.15.46:0/1616767026 >> [v2:172.21.15.106:6834/2857098625,v1:172.21.15.106:6836/2857098625] conn(0x561cef5d7cb0 0x561cef5d8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.662+0000 7fb928ff9700 10 client.9504 ms_handle_connect on v2:172.21.15.106:6834/2857098625 2024-01-27T00:46:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.662+0000 7fb94a59c700 1 --2- 172.21.15.46:0/1616767026 >> [v2:172.21.15.106:6834/2857098625,v1:172.21.15.106:6836/2857098625] conn(0x561cef5d7cb0 0x561cef5d8080 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6d564138be7b0de server_cookie=2ad3c058bca3924f in_seq=0 out_seq=0 2024-01-27T00:46:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.663+0000 7fb928ff9700 1 -- 172.21.15.46:0/1616767026 <== mds.0 v2:172.21.15.106:6834/2857098625 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7fb940014a40 con 0x561cef5d7cb0 2024-01-27T00:46:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.663+0000 7fb928ff9700 10 client.9504 handle_command_reply: tid=0 2024-01-27T00:46:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.663+0000 7fb95a26db80 1 client.9504 shutdown 2024-01-27T00:46:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.663+0000 7fb95a26db80 1 -- 172.21.15.46:0/1616767026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5fe360 msgr2=0x561cef5b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.663+0000 7fb95a26db80 1 --2- 172.21.15.46:0/1616767026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5fe360 0x561cef5b44c0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0c9670 tx=0x7fb944009040 comp rx=0 tx=0).stop 2024-01-27T00:46:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.663+0000 7fb95a26db80 1 -- 172.21.15.46:0/1616767026 shutdown_connections 2024-01-27T00:46:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.663+0000 7fb95a26db80 1 --2- 172.21.15.46:0/1616767026 >> [v2:172.21.15.106:6834/2857098625,v1:172.21.15.106:6836/2857098625] conn(0x561cef5d7cb0 0x561cef5d8080 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.663+0000 7fb95a26db80 1 --2- 172.21.15.46:0/1616767026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cef5fe360 0x561cef5b44c0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.663+0000 7fb95a26db80 1 --2- 172.21.15.46:0/1616767026 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561cef5b55d0 0x561cef60c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.663+0000 7fb95a26db80 1 --2- 172.21.15.46:0/1616767026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561cef5b6010 0x561cef5b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.663+0000 7fb928ff9700 0 client.9504 ms_handle_reset on v2:172.21.15.106:6834/2857098625 2024-01-27T00:46:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.663+0000 7fb95a26db80 1 -- 172.21.15.46:0/1616767026 >> 172.21.15.46:0/1616767026 conn(0x561cef5b5a70 msgr2=0x561cef5fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.663+0000 7fb95a26db80 1 -- 172.21.15.46:0/1616767026 shutdown_connections 2024-01-27T00:46:51.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.663+0000 7fb95a26db80 1 -- 172.21.15.46:0/1616767026 wait complete. 2024-01-27T00:46:51.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.663+0000 7fb95a26db80 20 client.9504 trim_cache size 0 max 16384 2024-01-27T00:46:51.672 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T00:46:51.672 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-27T00:46:51.672 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-27T00:46:51.673 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 9070, 2024-01-27T00:46:51.673 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-27T00:46:51.673 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-27T00:46:51.673 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-01-27T00:46:51.673 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.1111250339999996, 2024-01-27T00:46:51.673 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 88, 2024-01-27T00:46:51.673 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 175, 2024-01-27T00:46:51.674 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 175, 2024-01-27T00:46:51.674 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 12.158032145 2024-01-27T00:46:51.674 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T00:46:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.664+0000 7fb950f6a700 1 -- 172.21.15.46:0/1831908251 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb92c066f30 msgr2=0x7fb92c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.664+0000 7fb950f6a700 1 --2- 172.21.15.46:0/1831908251 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb92c066f30 0x7fb92c0693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb94c15d620 tx=0x7fb940007ec0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.664+0000 7fb950f6a700 1 -- 172.21.15.46:0/1831908251 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb94c0cc1c0 msgr2=0x7fb94c16d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.664+0000 7fb950f6a700 1 --2- 172.21.15.46:0/1831908251 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb94c0cc1c0 0x7fb94c16d2d0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fb93c014000 tx=0x7fb93c017000 comp rx=0 tx=0).stop 2024-01-27T00:46:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.664+0000 7fb950f6a700 1 -- 172.21.15.46:0/1831908251 shutdown_connections 2024-01-27T00:46:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.664+0000 7fb950f6a700 1 --2- 172.21.15.46:0/1831908251 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fb92c066f30 0x7fb92c0693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.664+0000 7fb950f6a700 1 --2- 172.21.15.46:0/1831908251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c15d210 0x7fb94c157f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.664+0000 7fb950f6a700 1 --2- 172.21.15.46:0/1831908251 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb94c0cc1c0 0x7fb94c16d2d0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.664+0000 7fb950f6a700 1 --2- 172.21.15.46:0/1831908251 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb94c15ce40 0x7fb94c16b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.664+0000 7fb950f6a700 1 -- 172.21.15.46:0/1831908251 >> 172.21.15.46:0/1831908251 conn(0x7fb94c06f840 msgr2=0x7fb94c154e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.664+0000 7fb950f6a700 1 -- 172.21.15.46:0/1831908251 shutdown_connections 2024-01-27T00:46:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.664+0000 7fb950f6a700 1 -- 172.21.15.46:0/1831908251 wait complete. 2024-01-27T00:46:51.677 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-27T00:46:51.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.809+0000 7f2973bdb700 1 Processor -- start 2024-01-27T00:46:51.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.809+0000 7f2973bdb700 1 -- start start 2024-01-27T00:46:51.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.810+0000 7f2973bdb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f296c159590 0x7f296c15d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.810+0000 7f2973bdb700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296c15dd90 con 0x7f296c15d8f0 2024-01-27T00:46:51.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.810+0000 7f2973bdb700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296c15e070 con 0x7f296c06c3c0 2024-01-27T00:46:51.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.810+0000 7f2973bdb700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f296c15e350 con 0x7f296c159590 2024-01-27T00:46:51.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.810+0000 7f2971977700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f296c159590 0x7f296c15d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.810+0000 7f2971176700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f296c15d8f0 0x7f296c06bcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60510/0 (socket says 172.21.15.46:60510) 2024-01-27T00:46:51.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.810+0000 7f2971176700 1 -- 172.21.15.46:0/662289475 learned_addr learned my addr 172.21.15.46:0/662289475 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:51.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.811+0000 7f2970975700 1 -- 172.21.15.46:0/662289475 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151626162 0 0) 0x7f296c15dd90 con 0x7f296c15d8f0 2024-01-27T00:46:51.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.811+0000 7f2970975700 1 -- 172.21.15.46:0/662289475 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2958003570 con 0x7f296c15d8f0 2024-01-27T00:46:51.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.811+0000 7f2971977700 1 -- 172.21.15.46:0/662289475 >> v1:172.21.15.106:6789/0 conn(0x7f296c06c3c0 legacy=0x7f296c069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.811+0000 7f2971977700 1 -- 172.21.15.46:0/662289475 >> v1:172.21.15.46:6789/0 conn(0x7f296c15d8f0 legacy=0x7f296c06bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.811+0000 7f2971977700 1 -- 172.21.15.46:0/662289475 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296c068b70 con 0x7f296c159590 2024-01-27T00:46:51.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.811+0000 7f2971977700 1 --2- 172.21.15.46:0/662289475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f296c159590 0x7f296c15d3b0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f29680098e0 tx=0x7f29680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2063ed6e7f99712 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:51.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.811+0000 7f2970975700 1 -- 172.21.15.46:0/662289475 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296800b640 con 0x7f296c159590 2024-01-27T00:46:51.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.811+0000 7f2970975700 1 -- 172.21.15.46:0/662289475 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296800b7a0 con 0x7f296c159590 2024-01-27T00:46:51.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.811+0000 7f2970975700 1 -- 172.21.15.46:0/662289475 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296800bb40 con 0x7f296c159590 2024-01-27T00:46:51.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.812+0000 7f2973bdb700 1 -- 172.21.15.46:0/662289475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f296c159590 msgr2=0x7f296c15d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.812+0000 7f2973bdb700 1 --2- 172.21.15.46:0/662289475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f296c159590 0x7f296c15d3b0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f29680098e0 tx=0x7f29680091a0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.812+0000 7f2973bdb700 1 -- 172.21.15.46:0/662289475 shutdown_connections 2024-01-27T00:46:51.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.812+0000 7f2973bdb700 1 --2- 172.21.15.46:0/662289475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f296c159590 0x7f296c15d3b0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.812+0000 7f2973bdb700 1 -- 172.21.15.46:0/662289475 >> 172.21.15.46:0/662289475 conn(0x7f296c0ce530 msgr2=0x7f296c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:51.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.812+0000 7f2973bdb700 1 -- 172.21.15.46:0/662289475 shutdown_connections 2024-01-27T00:46:51.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.812+0000 7f2973bdb700 1 -- 172.21.15.46:0/662289475 wait complete. 2024-01-27T00:46:51.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.813+0000 7f2973bdb700 1 Processor -- start 2024-01-27T00:46:51.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.813+0000 7f2973bdb700 1 -- start start 2024-01-27T00:46:51.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.814+0000 7f2973bdb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f296c06c3c0 0x7f296c0af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.814+0000 7f2973bdb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f296c159590 0x7f296c0c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.814+0000 7f2973bdb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f296c15d8f0 0x7f296c157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.814+0000 7f2973bdb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c0c7140 con 0x7f296c06c3c0 2024-01-27T00:46:51.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.814+0000 7f2973bdb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c0c6e10 con 0x7f296c159590 2024-01-27T00:46:51.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.814+0000 7f2973bdb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296c0afa70 con 0x7f296c15d8f0 2024-01-27T00:46:51.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.814+0000 7f2971977700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f296c06c3c0 0x7f296c0af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.814+0000 7f2971977700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f296c06c3c0 0x7f296c0af490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41780/0 (socket says 172.21.15.46:41780) 2024-01-27T00:46:51.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.814+0000 7f2971977700 1 -- 172.21.15.46:0/3361412985 learned_addr learned my addr 172.21.15.46:0/3361412985 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:51.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.814+0000 7f2971176700 1 --2- 172.21.15.46:0/3361412985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f296c159590 0x7f296c0c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.814+0000 7f2972178700 1 --2- 172.21.15.46:0/3361412985 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f296c15d8f0 0x7f296c157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.815+0000 7f2971977700 1 -- 172.21.15.46:0/3361412985 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f296c15d8f0 msgr2=0x7f296c157d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.815+0000 7f2971977700 1 --2- 172.21.15.46:0/3361412985 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f296c15d8f0 0x7f296c157d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.815+0000 7f2971977700 1 -- 172.21.15.46:0/3361412985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f296c159590 msgr2=0x7f296c0c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:51.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.815+0000 7f2971977700 1 --2- 172.21.15.46:0/3361412985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f296c159590 0x7f296c0c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:51.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.815+0000 7f2971977700 1 -- 172.21.15.46:0/3361412985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2968009420 con 0x7f296c06c3c0 2024-01-27T00:46:51.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.815+0000 7f2972178700 1 --2- 172.21.15.46:0/3361412985 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f296c15d8f0 0x7f296c157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:46:51.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.815+0000 7f2971176700 1 --2- 172.21.15.46:0/3361412985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f296c159590 0x7f296c0c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:46:51.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.815+0000 7f2971977700 1 --2- 172.21.15.46:0/3361412985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f296c06c3c0 0x7f296c0af490 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f29680093f0 tx=0x7f2968017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:51.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.815+0000 7f295dffb700 1 -- 172.21.15.46:0/3361412985 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2968000c40 con 0x7f296c06c3c0 2024-01-27T00:46:51.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.815+0000 7f295dffb700 1 -- 172.21.15.46:0/3361412985 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296800b910 con 0x7f296c06c3c0 2024-01-27T00:46:51.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.815+0000 7f2973bdb700 1 -- 172.21.15.46:0/3361412985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f296c15c590 con 0x7f296c06c3c0 2024-01-27T00:46:51.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.815+0000 7f295dffb700 1 -- 172.21.15.46:0/3361412985 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2968025670 con 0x7f296c06c3c0 2024-01-27T00:46:51.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.815+0000 7f2973bdb700 1 -- 172.21.15.46:0/3361412985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296c15cb50 con 0x7f296c06c3c0 2024-01-27T00:46:51.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.816+0000 7f2973bdb700 1 -- 172.21.15.46:0/3361412985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f296c04c710 con 0x7f296c06c3c0 2024-01-27T00:46:51.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.817+0000 7f295dffb700 1 -- 172.21.15.46:0/3361412985 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2968023070 con 0x7f296c06c3c0 2024-01-27T00:46:51.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.817+0000 7f295dffb700 1 --2- 172.21.15.46:0/3361412985 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2950066f30 0x7f29500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:51.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.817+0000 7f295dffb700 1 -- 172.21.15.46:0/3361412985 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f2968085bf0 con 0x7f296c06c3c0 2024-01-27T00:46:51.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.817+0000 7f2971176700 1 --2- 172.21.15.46:0/3361412985 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2950066f30 0x7f29500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:51.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.818+0000 7f2971176700 1 --2- 172.21.15.46:0/3361412985 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2950066f30 0x7f29500693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f296c159fd0 tx=0x7f2960009040 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:51.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:51.822+0000 7f295dffb700 1 -- 172.21.15.46:0/3361412985 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2968052ad0 con 0x7f296c06c3c0 2024-01-27T00:46:52.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:52.077+0000 7f2973bdb700 1 -- 172.21.15.46:0/3361412985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f296c0b0a10 con 0x7f296c06c3c0 2024-01-27T00:46:52.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:52.077+0000 7f295dffb700 1 -- 172.21.15.46:0/3361412985 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v88) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2968015070 con 0x7f296c06c3c0 2024-01-27T00:46:52.077 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-27T00:46:52.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:52.079+0000 7f2973bdb700 1 -- 172.21.15.46:0/3361412985 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2950066f30 msgr2=0x7f29500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:52.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:52.079+0000 7f2973bdb700 1 --2- 172.21.15.46:0/3361412985 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2950066f30 0x7f29500693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f296c159fd0 tx=0x7f2960009040 comp rx=0 tx=0).stop 2024-01-27T00:46:52.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:52.079+0000 7f2973bdb700 1 -- 172.21.15.46:0/3361412985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f296c06c3c0 msgr2=0x7f296c0af490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:52.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:52.079+0000 7f2973bdb700 1 --2- 172.21.15.46:0/3361412985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f296c06c3c0 0x7f296c0af490 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f29680093f0 tx=0x7f2968017000 comp rx=0 tx=0).stop 2024-01-27T00:46:52.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:52.079+0000 7f2973bdb700 1 -- 172.21.15.46:0/3361412985 shutdown_connections 2024-01-27T00:46:52.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:52.079+0000 7f2973bdb700 1 --2- 172.21.15.46:0/3361412985 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f2950066f30 0x7f29500693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:52.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:52.079+0000 7f2973bdb700 1 --2- 172.21.15.46:0/3361412985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f296c06c3c0 0x7f296c0af490 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:52.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:52.079+0000 7f2973bdb700 1 --2- 172.21.15.46:0/3361412985 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f296c15d8f0 0x7f296c157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:52.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:52.079+0000 7f2973bdb700 1 --2- 172.21.15.46:0/3361412985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f296c159590 0x7f296c0c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:52.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:52.079+0000 7f2973bdb700 1 -- 172.21.15.46:0/3361412985 >> 172.21.15.46:0/3361412985 conn(0x7f296c0ce530 msgr2=0x7f296c158a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:52.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:52.080+0000 7f2973bdb700 1 -- 172.21.15.46:0/3361412985 shutdown_connections 2024-01-27T00:46:52.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:52.080+0000 7f2973bdb700 1 -- 172.21.15.46:0/3361412985 wait complete. 2024-01-27T00:46:52.092 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-27T00:46:52.092 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-27T00:46:52.092 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-27T00:46:52.092 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-27T00:46:52.093 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-27T00:46:52.093 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-27T00:46:52.093 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-27T00:46:52.093 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-27T00:46:52.093 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-27T00:46:52.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:46:52.094 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-27T00:46:52.115 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-27T00:46:52.115 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-27T00:46:52.115 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-27T00:46:52.115 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T00:46:52.115 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-27T00:46:52.115 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T00:46:52.115 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T00:46:52.115 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-27T00:46:52.115 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-27T00:46:52.115 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T00:46:52.115 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-27T00:46:52.115 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-27T00:46:52.115 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-27T00:46:52.115 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-27T00:46:52.116 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-27T00:46:52.116 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-27T00:46:52.116 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 37718sec preferred_lft 37718sec 2024-01-27T00:46:52.116 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-27T00:46:52.116 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T00:46:52.116 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T00:46:52.116 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-27T00:46:52.116 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-27T00:46:52.116 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-27T00:46:52.116 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff 2024-01-27T00:46:52.116 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-27T00:46:52.116 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T00:46:52.117 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::cce6:a7ff:fe4f:38d/64 scope link 2024-01-27T00:46:52.117 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T00:46:52.117 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-27T00:46:52.117 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 76:ef:90:4c:9d:6f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-27T00:46:52.117 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::74ef:90ff:fe4c:9d6f/64 scope link 2024-01-27T00:46:52.117 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T00:46:52.117 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-27T00:46:52.117 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-27T00:46:52.117 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::2c73:afff:fef0:6f39/64 scope link 2024-01-27T00:46:52.117 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T00:46:52.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:46:52.118 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-27T00:46:52.186 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-27T00:46:52.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:46:52.187 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-27T00:46:52.253 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-27T00:46:52.253 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-27T00:46:52.253 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-27T00:46:52.253 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-27T00:46:52.253 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-27T00:46:52.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:46:52.254 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-27T00:46:52.254 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-27T00:46:52.254 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-27T00:46:52.254 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-27T00:46:52.254 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-27T00:46:52.254 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-27T00:46:52.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:46:52.379 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-27T00:46:52.399 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-27T00:46:52.399 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-27T00:46:52.400 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-27T00:46:52.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:46:52.400 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-01-27T00:46:52.479 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-27T00:46:52.479 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-27T00:46:52.479 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-27T00:46:52.540 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-27T00:46:52.541 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-27T00:46:52.541 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-27T00:46:52.603 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-27T00:46:52.679 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-27T00:46:52.679 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-27T00:46:52.754 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-27T00:46:52.755 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-27T00:46:52.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:52.756 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-27T00:46:52.818 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-27T00:46:52.818 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-27T00:46:52.861 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-27T00:46:52.895 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-27T00:46:52.895 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-27T00:46:52.964 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-01-27T00:46:52.965+0000 7f6456784540 -1 init, newargv = 0x5583e960afe0 newargc=15 2024-01-27T00:46:52.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[98304]: starting ceph client 2024-01-27T00:46:52.966 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-27T00:46:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-27T00:46:52.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:52.969 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-27T00:46:52.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[98304]: starting fuse 2024-01-27T00:46:52.989 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-27T00:46:52.990 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-27T00:46:52.990 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-27T00:46:52.990 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-27T00:46:52.990 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-27T00:46:52.990 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-27T00:46:52.990 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-27T00:46:52.990 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T00:46:52.990 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-27T00:46:52.990 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-27T00:46:52.990 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-27T00:46:52.990 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-27T00:46:52.990 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T00:46:52.991 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-27T00:46:52.991 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-27T00:46:52.991 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-27T00:46:52.991 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-27T00:46:52.991 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T00:46:52.991 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-27T00:46:52.991 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-27T00:46:52.991 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-27T00:46:52.991 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-27T00:46:52.991 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-27T00:46:52.991 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-27T00:46:52.991 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-27T00:46:52.991 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-27T00:46:52.991 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T00:46:52.991 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-01-27T00:46:52.991 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-27T00:46:53.127 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.98304.asok 2024-01-27T00:46:53.309 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.98304.asok 2024-01-27T00:46:53.309 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:53.309 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.98304.asok status 2024-01-27T00:46:53.484 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T00:46:53.484 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-27T00:46:53.484 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-27T00:46:53.485 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-27T00:46:53.485 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-01-27T00:46:53.485 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-27T00:46:53.485 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-27T00:46:53.485 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "98304", 2024-01-27T00:46:53.485 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-27T00:46:53.486 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:53.486 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-27T00:46:53.486 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-27T00:46:53.486 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 9519, 2024-01-27T00:46:53.486 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-27T00:46:53.486 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-27T00:46:53.486 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-27T00:46:53.487 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 9519 2024-01-27T00:46:53.487 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:53.487 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-27T00:46:53.487 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-27T00:46:53.487 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-27T00:46:53.487 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3230097128 2024-01-27T00:46:53.487 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:53.488 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:53.488 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-27T00:46:53.488 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-27T00:46:53.488 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-27T00:46:53.488 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3230097128 2024-01-27T00:46:53.488 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:46:53.488 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.9519 192.168.0.1:0/3230097128", 2024-01-27T00:46:53.489 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/3230097128", 2024-01-27T00:46:53.489 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-27T00:46:53.489 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 88, 2024-01-27T00:46:53.489 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 175, 2024-01-27T00:46:53.489 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-27T00:46:53.489 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-27T00:46:53.489 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-27T00:46:53.490 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T00:46:53.496 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:53.496 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-27T00:46:53.496 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-27T00:46:53.516 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-27T00:46:53.516 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-27T00:46:53.516 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:53.517 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-27T00:46:53.592 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-27T00:46:53.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.774+0000 7f9dfd206700 1 Processor -- start 2024-01-27T00:46:53.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.774+0000 7f9dfd206700 1 -- start start 2024-01-27T00:46:53.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.775+0000 7f9dfd206700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9df815d260 0x7f9df80b3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:53.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.775+0000 7f9dfd206700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9df80cf4c0 0x7f9df80cc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:53.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.775+0000 7f9dfd206700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df80c6290 con 0x7f9df815ce90 2024-01-27T00:46:53.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.775+0000 7f9dfd206700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df80c5680 con 0x7f9df80cf4c0 2024-01-27T00:46:53.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.775+0000 7f9dfd206700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df80c5960 con 0x7f9df815d260 2024-01-27T00:46:53.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.775+0000 7f9df759e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9df80cf4c0 0x7f9df80cc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:53.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.775+0000 7f9df6d9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9df815ce90 0x7f9df80cedb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60514/0 (socket says 172.21.15.46:60514) 2024-01-27T00:46:53.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.775+0000 7f9df6d9d700 1 -- 172.21.15.46:0/1870252115 learned_addr learned my addr 172.21.15.46:0/1870252115 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:53.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.775+0000 7f9df659c700 1 --2- 172.21.15.46:0/1870252115 >> v2:172.21.15.106:3300/0 conn(0x7f9df815d260 0x7f9df80b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:53.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.775+0000 7f9df759e700 1 -- 172.21.15.46:0/1870252115 >> v2:172.21.15.106:3300/0 conn(0x7f9df815d260 msgr2=0x7f9df80b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:53.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.775+0000 7f9df759e700 1 --2- 172.21.15.46:0/1870252115 >> v2:172.21.15.106:3300/0 conn(0x7f9df815d260 0x7f9df80b3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:53.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.775+0000 7f9df759e700 1 -- 172.21.15.46:0/1870252115 >> v1:172.21.15.46:6789/0 conn(0x7f9df815ce90 legacy=0x7f9df80cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:53.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.775+0000 7f9df5d9b700 1 -- 172.21.15.46:0/1870252115 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111430792 0 0) 0x7f9df80c6290 con 0x7f9df815ce90 2024-01-27T00:46:53.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.775+0000 7f9df759e700 1 -- 172.21.15.46:0/1870252115 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df80c5c40 con 0x7f9df80cf4c0 2024-01-27T00:46:53.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.776+0000 7f9df759e700 1 --2- 172.21.15.46:0/1870252115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9df80cf4c0 0x7f9df80cc570 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f9df80b6790 tx=0x7f9df0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ae35cd81e572c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:53.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.776+0000 7f9df5d9b700 1 -- 172.21.15.46:0/1870252115 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df000e070 con 0x7f9df80cf4c0 2024-01-27T00:46:53.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.776+0000 7f9df5d9b700 1 -- 172.21.15.46:0/1870252115 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df000bc30 con 0x7f9df80cf4c0 2024-01-27T00:46:53.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.776+0000 7f9df5d9b700 1 -- 172.21.15.46:0/1870252115 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0012430 con 0x7f9df80cf4c0 2024-01-27T00:46:53.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.776+0000 7f9dfd206700 1 -- 172.21.15.46:0/1870252115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9df80cf4c0 msgr2=0x7f9df80cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:53.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.776+0000 7f9dfd206700 1 --2- 172.21.15.46:0/1870252115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9df80cf4c0 0x7f9df80cc570 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f9df80b6790 tx=0x7f9df0009780 comp rx=0 tx=0).stop 2024-01-27T00:46:53.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.776+0000 7f9dfd206700 1 -- 172.21.15.46:0/1870252115 shutdown_connections 2024-01-27T00:46:53.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.776+0000 7f9dfd206700 1 --2- 172.21.15.46:0/1870252115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9df80cf4c0 0x7f9df80cc570 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:53.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.776+0000 7f9dfd206700 1 --2- 172.21.15.46:0/1870252115 >> v2:172.21.15.106:3300/0 conn(0x7f9df815d260 0x7f9df80b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:53.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.776+0000 7f9dfd206700 1 -- 172.21.15.46:0/1870252115 >> 172.21.15.46:0/1870252115 conn(0x7f9df8099a70 msgr2=0x7f9df806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:53.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.776+0000 7f9dfd206700 1 -- 172.21.15.46:0/1870252115 shutdown_connections 2024-01-27T00:46:53.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.776+0000 7f9dfd206700 1 -- 172.21.15.46:0/1870252115 wait complete. 2024-01-27T00:46:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.778+0000 7f9dfd206700 1 Processor -- start 2024-01-27T00:46:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.778+0000 7f9dfd206700 1 -- start start 2024-01-27T00:46:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9dfd206700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9df80cf4c0 0x7f9df814c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9dfd206700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9df815ce90 0x7f9df815f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9dfd206700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9df815d260 0x7f9df815f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9dfd206700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df80cdb10 con 0x7f9df80cf4c0 2024-01-27T00:46:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9dfd206700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df80c85f0 con 0x7f9df815d260 2024-01-27T00:46:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9dfd206700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df80c79e0 con 0x7f9df815ce90 2024-01-27T00:46:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9df6d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9df80cf4c0 0x7f9df814c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9df6d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9df80cf4c0 0x7f9df814c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41796/0 (socket says 172.21.15.46:41796) 2024-01-27T00:46:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9df6d9d700 1 -- 172.21.15.46:0/921929106 learned_addr learned my addr 172.21.15.46:0/921929106 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9df759e700 1 --2- 172.21.15.46:0/921929106 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9df815d260 0x7f9df815f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9df659c700 1 --2- 172.21.15.46:0/921929106 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9df815ce90 0x7f9df815f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9df6d9d700 1 -- 172.21.15.46:0/921929106 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9df815ce90 msgr2=0x7f9df815f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9df6d9d700 1 --2- 172.21.15.46:0/921929106 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9df815ce90 0x7f9df815f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:53.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9df6d9d700 1 -- 172.21.15.46:0/921929106 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9df815d260 msgr2=0x7f9df815f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:53.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9df6d9d700 1 --2- 172.21.15.46:0/921929106 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9df815d260 0x7f9df815f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:53.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9df6d9d700 1 -- 172.21.15.46:0/921929106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df0009a20 con 0x7f9df80cf4c0 2024-01-27T00:46:53.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9df6d9d700 1 --2- 172.21.15.46:0/921929106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9df80cf4c0 0x7f9df814c730 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00c810 tx=0x7f9dec007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:53.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9ddeffd700 1 -- 172.21.15.46:0/921929106 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec007300 con 0x7f9df80cf4c0 2024-01-27T00:46:53.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9ddeffd700 1 -- 172.21.15.46:0/921929106 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dec007460 con 0x7f9df80cf4c0 2024-01-27T00:46:53.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9dfd206700 1 -- 172.21.15.46:0/921929106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df80c7cc0 con 0x7f9df80cf4c0 2024-01-27T00:46:53.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9ddeffd700 1 -- 172.21.15.46:0/921929106 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec0057f0 con 0x7f9df80cf4c0 2024-01-27T00:46:53.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.779+0000 7f9dfd206700 1 -- 172.21.15.46:0/921929106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df80c8280 con 0x7f9df80cf4c0 2024-01-27T00:46:53.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.780+0000 7f9dfd206700 1 -- 172.21.15.46:0/921929106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df804c710 con 0x7f9df80cf4c0 2024-01-27T00:46:53.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.781+0000 7f9ddeffd700 1 -- 172.21.15.46:0/921929106 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9dec005c00 con 0x7f9df80cf4c0 2024-01-27T00:46:53.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.781+0000 7f9ddeffd700 1 --2- 172.21.15.46:0/921929106 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9de4066f30 0x7f9de40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:53.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.781+0000 7f9ddeffd700 1 -- 172.21.15.46:0/921929106 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f9dec084300 con 0x7f9df80cf4c0 2024-01-27T00:46:53.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.781+0000 7f9df659c700 1 --2- 172.21.15.46:0/921929106 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9de4066f30 0x7f9de40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:53.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.782+0000 7f9df659c700 1 --2- 172.21.15.46:0/921929106 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9de4066f30 0x7f9de40693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9df8069b90 tx=0x7f9de8008040 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:53.786+0000 7f9ddeffd700 1 -- 172.21.15.46:0/921929106 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9dec051300 con 0x7f9df80cf4c0 2024-01-27T00:46:54.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.021+0000 7f9dfd206700 1 -- 172.21.15.46:0/921929106 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9df80c8730 con 0x7f9de4066f30 2024-01-27T00:46:54.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.023+0000 7f9ddeffd700 1 -- 172.21.15.46:0/921929106 <== mgr.8860 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f9de800b8b0 con 0x7f9de4066f30 2024-01-27T00:46:54.023 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-27T00:46:54.023 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:46:54.023 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephfs" 2024-01-27T00:46:54.023 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:46:54.024 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-27T00:46:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.025+0000 7f9dfd206700 1 -- 172.21.15.46:0/921929106 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9de4066f30 msgr2=0x7f9de40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.025+0000 7f9dfd206700 1 --2- 172.21.15.46:0/921929106 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9de4066f30 0x7f9de40693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9df8069b90 tx=0x7f9de8008040 comp rx=0 tx=0).stop 2024-01-27T00:46:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.025+0000 7f9dfd206700 1 -- 172.21.15.46:0/921929106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9df80cf4c0 msgr2=0x7f9df814c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.025+0000 7f9dfd206700 1 --2- 172.21.15.46:0/921929106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9df80cf4c0 0x7f9df814c730 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00c810 tx=0x7f9dec007c80 comp rx=0 tx=0).stop 2024-01-27T00:46:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.025+0000 7f9dfd206700 1 -- 172.21.15.46:0/921929106 shutdown_connections 2024-01-27T00:46:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.025+0000 7f9dfd206700 1 --2- 172.21.15.46:0/921929106 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f9de4066f30 0x7f9de40693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.025+0000 7f9dfd206700 1 --2- 172.21.15.46:0/921929106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9df80cf4c0 0x7f9df814c730 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.025+0000 7f9dfd206700 1 --2- 172.21.15.46:0/921929106 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9df815d260 0x7f9df815f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.025+0000 7f9dfd206700 1 --2- 172.21.15.46:0/921929106 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9df815ce90 0x7f9df815f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.025+0000 7f9dfd206700 1 -- 172.21.15.46:0/921929106 >> 172.21.15.46:0/921929106 conn(0x7f9df8099a70 msgr2=0x7f9df8158840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.025+0000 7f9dfd206700 1 -- 172.21.15.46:0/921929106 shutdown_connections 2024-01-27T00:46:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.025+0000 7f9dfd206700 1 -- 172.21.15.46:0/921929106 wait complete. 2024-01-27T00:46:54.039 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-27T00:46:54.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.172+0000 7f4b7845b700 1 Processor -- start 2024-01-27T00:46:54.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.172+0000 7f4b7845b700 1 -- start start 2024-01-27T00:46:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.173+0000 7f4b7845b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b700cf050 0x7f4b700af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.173+0000 7f4b7845b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b700cf4a0 0x7f4b700af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.173+0000 7f4b7845b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b700cc100 con 0x7f4b70159cb0 2024-01-27T00:46:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.173+0000 7f4b7845b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b700ccca0 con 0x7f4b700cf4a0 2024-01-27T00:46:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.173+0000 7f4b7845b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b700cb850 con 0x7f4b700cf050 2024-01-27T00:46:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.173+0000 7f4b769f8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b700cf4a0 0x7f4b700af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.173+0000 7f4b761f7700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4b70159cb0 0x7f4b700cc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60524/0 (socket says 172.21.15.46:60524) 2024-01-27T00:46:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.173+0000 7f4b761f7700 1 -- 172.21.15.46:0/3529563858 learned_addr learned my addr 172.21.15.46:0/3529563858 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:54.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.173+0000 7f4b759f6700 1 --2- 172.21.15.46:0/3529563858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b700cf050 0x7f4b700af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:54.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.173+0000 7f4b769f8700 1 -- 172.21.15.46:0/3529563858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b700cf050 msgr2=0x7f4b700af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:54.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.173+0000 7f4b769f8700 1 --2- 172.21.15.46:0/3529563858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b700cf050 0x7f4b700af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.173+0000 7f4b769f8700 1 -- 172.21.15.46:0/3529563858 >> v1:172.21.15.46:6789/0 conn(0x7f4b70159cb0 legacy=0x7f4b700cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:54.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.173+0000 7f4b769f8700 1 -- 172.21.15.46:0/3529563858 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b700cbaf0 con 0x7f4b700cf4a0 2024-01-27T00:46:54.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.173+0000 7f4b751f5700 1 -- 172.21.15.46:0/3529563858 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308610634 0 0) 0x7f4b700cc100 con 0x7f4b70159cb0 2024-01-27T00:46:54.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.174+0000 7f4b769f8700 1 --2- 172.21.15.46:0/3529563858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b700cf4a0 0x7f4b700af970 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009e60 tx=0x7f4b68009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dcdbe47c5d9f42a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:54.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.174+0000 7f4b751f5700 1 -- 172.21.15.46:0/3529563858 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6800e070 con 0x7f4b700cf4a0 2024-01-27T00:46:54.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.174+0000 7f4b751f5700 1 -- 172.21.15.46:0/3529563858 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6800bc30 con 0x7f4b700cf4a0 2024-01-27T00:46:54.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.174+0000 7f4b751f5700 1 -- 172.21.15.46:0/3529563858 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68012430 con 0x7f4b700cf4a0 2024-01-27T00:46:54.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.174+0000 7f4b7845b700 1 -- 172.21.15.46:0/3529563858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b700cf4a0 msgr2=0x7f4b700af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:54.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.174+0000 7f4b7845b700 1 --2- 172.21.15.46:0/3529563858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b700cf4a0 0x7f4b700af970 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009e60 tx=0x7f4b68009780 comp rx=0 tx=0).stop 2024-01-27T00:46:54.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.174+0000 7f4b7845b700 1 -- 172.21.15.46:0/3529563858 shutdown_connections 2024-01-27T00:46:54.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.174+0000 7f4b7845b700 1 --2- 172.21.15.46:0/3529563858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b700cf4a0 0x7f4b700af970 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.174+0000 7f4b7845b700 1 --2- 172.21.15.46:0/3529563858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b700cf050 0x7f4b700af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.174+0000 7f4b7845b700 1 -- 172.21.15.46:0/3529563858 >> 172.21.15.46:0/3529563858 conn(0x7f4b70087f60 msgr2=0x7f4b7006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:54.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.174+0000 7f4b7845b700 1 -- 172.21.15.46:0/3529563858 shutdown_connections 2024-01-27T00:46:54.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.174+0000 7f4b7845b700 1 -- 172.21.15.46:0/3529563858 wait complete. 2024-01-27T00:46:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.176+0000 7f4b7845b700 1 Processor -- start 2024-01-27T00:46:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.176+0000 7f4b7845b700 1 -- start start 2024-01-27T00:46:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.176+0000 7f4b7845b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b700cf050 0x7f4b7014a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.176+0000 7f4b7845b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b700cf4a0 0x7f4b7014bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.176+0000 7f4b7845b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70159cb0 0x7f4b70157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.176+0000 7f4b7845b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7014b930 con 0x7f4b70159cb0 2024-01-27T00:46:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.176+0000 7f4b7845b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7014ab30 con 0x7f4b700cf050 2024-01-27T00:46:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.176+0000 7f4b7845b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7014ad40 con 0x7f4b700cf4a0 2024-01-27T00:46:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.177+0000 7f4b769f8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70159cb0 0x7f4b70157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.177+0000 7f4b769f8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70159cb0 0x7f4b70157c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41820/0 (socket says 172.21.15.46:41820) 2024-01-27T00:46:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.177+0000 7f4b769f8700 1 -- 172.21.15.46:0/3861078273 learned_addr learned my addr 172.21.15.46:0/3861078273 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.177+0000 7f4b761f7700 1 --2- 172.21.15.46:0/3861078273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b700cf050 0x7f4b7014a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.177+0000 7f4b759f6700 1 --2- 172.21.15.46:0/3861078273 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b700cf4a0 0x7f4b7014bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.177+0000 7f4b769f8700 1 -- 172.21.15.46:0/3861078273 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b700cf4a0 msgr2=0x7f4b7014bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.177+0000 7f4b769f8700 1 --2- 172.21.15.46:0/3861078273 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b700cf4a0 0x7f4b7014bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.177+0000 7f4b769f8700 1 -- 172.21.15.46:0/3861078273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b700cf050 msgr2=0x7f4b7014a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.177+0000 7f4b769f8700 1 --2- 172.21.15.46:0/3861078273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b700cf050 0x7f4b7014a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.177+0000 7f4b769f8700 1 -- 172.21.15.46:0/3861078273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b68009a20 con 0x7f4b70159cb0 2024-01-27T00:46:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.177+0000 7f4b769f8700 1 --2- 172.21.15.46:0/3861078273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70159cb0 0x7f4b70157c00 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f4b68013000 tx=0x7f4b68019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.177+0000 7f4b667fc700 1 -- 172.21.15.46:0/3861078273 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68027470 con 0x7f4b70159cb0 2024-01-27T00:46:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.177+0000 7f4b667fc700 1 -- 172.21.15.46:0/3861078273 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b68027630 con 0x7f4b70159cb0 2024-01-27T00:46:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.177+0000 7f4b7845b700 1 -- 172.21.15.46:0/3861078273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b7014b000 con 0x7f4b70159cb0 2024-01-27T00:46:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.177+0000 7f4b667fc700 1 -- 172.21.15.46:0/3861078273 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b680279f0 con 0x7f4b70159cb0 2024-01-27T00:46:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.177+0000 7f4b7845b700 1 -- 172.21.15.46:0/3861078273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7014b5c0 con 0x7f4b70159cb0 2024-01-27T00:46:54.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.178+0000 7f4b7845b700 1 -- 172.21.15.46:0/3861078273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b7014baa0 con 0x7f4b70159cb0 2024-01-27T00:46:54.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.179+0000 7f4b667fc700 1 -- 172.21.15.46:0/3861078273 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 88) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4b68024070 con 0x7f4b70159cb0 2024-01-27T00:46:54.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.179+0000 7f4b667fc700 1 --2- 172.21.15.46:0/3861078273 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4b54066f30 0x7f4b540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:54.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.179+0000 7f4b667fc700 1 -- 172.21.15.46:0/3861078273 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f4b68086fd0 con 0x7f4b70159cb0 2024-01-27T00:46:54.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.179+0000 7f4b761f7700 1 --2- 172.21.15.46:0/3861078273 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4b54066f30 0x7f4b540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:54.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.180+0000 7f4b761f7700 1 --2- 172.21.15.46:0/3861078273 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4b54066f30 0x7f4b540693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c002970 tx=0x7f4b6c009040 comp rx=0 tx=0).ready entity=mgr.8860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:54.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.184+0000 7f4b667fc700 1 -- 172.21.15.46:0/3861078273 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4b68053f50 con 0x7f4b70159cb0 2024-01-27T00:46:54.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.446+0000 7f4b7845b700 1 -- 172.21.15.46:0/3861078273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f4b7004c710 con 0x7f4b70159cb0 2024-01-27T00:46:54.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.657+0000 7f4b667fc700 1 -- 172.21.15.46:0/3861078273 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 89) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f4b68003040 con 0x7f4b70159cb0 2024-01-27T00:46:54.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.659+0000 7f4b667fc700 1 -- 172.21.15.46:0/3861078273 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v89) v1 ==== 92+0+0 (secure 0 0 0) 0x7f4b680579e0 con 0x7f4b70159cb0 2024-01-27T00:46:54.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.662+0000 7f4b7845b700 1 -- 172.21.15.46:0/3861078273 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4b54066f30 msgr2=0x7f4b540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:54.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.662+0000 7f4b7845b700 1 --2- 172.21.15.46:0/3861078273 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4b54066f30 0x7f4b540693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c002970 tx=0x7f4b6c009040 comp rx=0 tx=0).stop 2024-01-27T00:46:54.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.662+0000 7f4b7845b700 1 -- 172.21.15.46:0/3861078273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70159cb0 msgr2=0x7f4b70157c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:54.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.662+0000 7f4b7845b700 1 --2- 172.21.15.46:0/3861078273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70159cb0 0x7f4b70157c00 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f4b68013000 tx=0x7f4b68019000 comp rx=0 tx=0).stop 2024-01-27T00:46:54.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.662+0000 7f4b7845b700 1 -- 172.21.15.46:0/3861078273 shutdown_connections 2024-01-27T00:46:54.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.662+0000 7f4b7845b700 1 --2- 172.21.15.46:0/3861078273 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4b54066f30 0x7f4b540693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.662+0000 7f4b7845b700 1 --2- 172.21.15.46:0/3861078273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70159cb0 0x7f4b70157c00 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.662+0000 7f4b7845b700 1 --2- 172.21.15.46:0/3861078273 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b700cf050 0x7f4b7014a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.662+0000 7f4b7845b700 1 --2- 172.21.15.46:0/3861078273 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b700cf4a0 0x7f4b7014bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.662+0000 7f4b7845b700 1 -- 172.21.15.46:0/3861078273 >> 172.21.15.46:0/3861078273 conn(0x7f4b70087f60 msgr2=0x7f4b70159040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:54.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.663+0000 7f4b7845b700 1 -- 172.21.15.46:0/3861078273 shutdown_connections 2024-01-27T00:46:54.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.663+0000 7f4b7845b700 1 -- 172.21.15.46:0/3861078273 wait complete. 2024-01-27T00:46:54.683 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-27T00:46:54.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.823+0000 7f168d09a700 1 Processor -- start 2024-01-27T00:46:54.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.823+0000 7f168d09a700 1 -- start start 2024-01-27T00:46:54.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.824+0000 7f168d09a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1688159cb0 0x7f16880cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:54.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.824+0000 7f168d09a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f16880afb40 0x7f16880c5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:54.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.824+0000 7f168d09a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16880cc100 con 0x7f16880cf050 2024-01-27T00:46:54.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.824+0000 7f168d09a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16880cb850 con 0x7f16880afb40 2024-01-27T00:46:54.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.824+0000 7f168d09a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16880cbaf0 con 0x7f1688159cb0 2024-01-27T00:46:54.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.824+0000 7f168759e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f16880afb40 0x7f16880c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:54.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.824+0000 7f168759e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f16880afb40 0x7f16880c5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41848/0 (socket says 172.21.15.46:41848) 2024-01-27T00:46:54.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.824+0000 7f168759e700 1 -- 172.21.15.46:0/90290831 learned_addr learned my addr 172.21.15.46:0/90290831 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:54.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.824+0000 7f1686d9d700 1 --2- 172.21.15.46:0/90290831 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1688159cb0 0x7f16880cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.824+0000 7f168759e700 1 -- 172.21.15.46:0/90290831 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1688159cb0 msgr2=0x7f16880cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.824+0000 7f168759e700 1 --2- 172.21.15.46:0/90290831 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1688159cb0 0x7f16880cc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.824+0000 7f168759e700 1 -- 172.21.15.46:0/90290831 >> v1:172.21.15.106:6789/0 conn(0x7f16880cf050 legacy=0x7f16880af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.824+0000 7f168759e700 1 -- 172.21.15.46:0/90290831 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16880cfc80 con 0x7f16880afb40 2024-01-27T00:46:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.824+0000 7f1685d9b700 1 -- 172.21.15.46:0/90290831 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576493313 0 0) 0x7f16880cc100 con 0x7f16880cf050 2024-01-27T00:46:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.825+0000 7f168759e700 1 --2- 172.21.15.46:0/90290831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16880afb40 0x7f16880c5530 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f1680009f20 tx=0x7f1680009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c451398a4fef99c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.825+0000 7f1685d9b700 1 -- 172.21.15.46:0/90290831 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168000e070 con 0x7f16880afb40 2024-01-27T00:46:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.825+0000 7f1685d9b700 1 -- 172.21.15.46:0/90290831 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168000b9d0 con 0x7f16880afb40 2024-01-27T00:46:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.825+0000 7f1685d9b700 1 -- 172.21.15.46:0/90290831 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16800092e0 con 0x7f16880afb40 2024-01-27T00:46:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.825+0000 7f168d09a700 1 -- 172.21.15.46:0/90290831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16880afb40 msgr2=0x7f16880c5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.825+0000 7f168d09a700 1 --2- 172.21.15.46:0/90290831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16880afb40 0x7f16880c5530 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f1680009f20 tx=0x7f1680009840 comp rx=0 tx=0).stop 2024-01-27T00:46:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.825+0000 7f168d09a700 1 -- 172.21.15.46:0/90290831 shutdown_connections 2024-01-27T00:46:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.825+0000 7f168d09a700 1 --2- 172.21.15.46:0/90290831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16880afb40 0x7f16880c5530 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.825+0000 7f168d09a700 1 --2- 172.21.15.46:0/90290831 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1688159cb0 0x7f16880cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.825+0000 7f168d09a700 1 -- 172.21.15.46:0/90290831 >> 172.21.15.46:0/90290831 conn(0x7f1688087f60 msgr2=0x7f168806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:54.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.825+0000 7f168d09a700 1 -- 172.21.15.46:0/90290831 shutdown_connections 2024-01-27T00:46:54.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.825+0000 7f168d09a700 1 -- 172.21.15.46:0/90290831 wait complete. 2024-01-27T00:46:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.827+0000 7f168d09a700 1 Processor -- start 2024-01-27T00:46:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.827+0000 7f168d09a700 1 -- start start 2024-01-27T00:46:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f168d09a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16880afb40 0x7f16880aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f168d09a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16880cf050 0x7f16880c9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f168d09a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1688159cb0 0x7f1688156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:54.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f168d09a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168814ad20 con 0x7f1688159cb0 2024-01-27T00:46:54.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f168d09a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16880c9850 con 0x7f16880cf050 2024-01-27T00:46:54.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f168d09a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1688152800 con 0x7f16880afb40 2024-01-27T00:46:54.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f168759e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1688159cb0 0x7f1688156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:54.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f168759e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1688159cb0 0x7f1688156ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41852/0 (socket says 172.21.15.46:41852) 2024-01-27T00:46:54.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f168759e700 1 -- 172.21.15.46:0/2282242149 learned_addr learned my addr 172.21.15.46:0/2282242149 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:54.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f1686d9d700 1 --2- 172.21.15.46:0/2282242149 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16880afb40 0x7f16880aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:54.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f168659c700 1 --2- 172.21.15.46:0/2282242149 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16880cf050 0x7f16880c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:54.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f168759e700 1 -- 172.21.15.46:0/2282242149 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16880afb40 msgr2=0x7f16880aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:54.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f168759e700 1 --2- 172.21.15.46:0/2282242149 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16880afb40 0x7f16880aff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f168759e700 1 -- 172.21.15.46:0/2282242149 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16880cf050 msgr2=0x7f16880c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:54.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f168759e700 1 --2- 172.21.15.46:0/2282242149 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16880cf050 0x7f16880c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:54.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f168759e700 1 -- 172.21.15.46:0/2282242149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1680009ae0 con 0x7f1688159cb0 2024-01-27T00:46:54.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f168659c700 1 --2- 172.21.15.46:0/2282242149 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16880cf050 0x7f16880c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:54.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.828+0000 7f168759e700 1 --2- 172.21.15.46:0/2282242149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1688159cb0 0x7f1688156ae0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f1688088df0 tx=0x7f1680018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:54.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.829+0000 7f166effd700 1 -- 172.21.15.46:0/2282242149 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1680008ad0 con 0x7f1688159cb0 2024-01-27T00:46:54.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.829+0000 7f166effd700 1 -- 172.21.15.46:0/2282242149 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1680008c50 con 0x7f1688159cb0 2024-01-27T00:46:54.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.829+0000 7f168d09a700 1 -- 172.21.15.46:0/2282242149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16880c9a70 con 0x7f1688159cb0 2024-01-27T00:46:54.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.829+0000 7f166effd700 1 -- 172.21.15.46:0/2282242149 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1680026460 con 0x7f1688159cb0 2024-01-27T00:46:54.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.829+0000 7f168d09a700 1 -- 172.21.15.46:0/2282242149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1688152490 con 0x7f1688159cb0 2024-01-27T00:46:54.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.830+0000 7f168d09a700 1 -- 172.21.15.46:0/2282242149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1688151ff0 con 0x7f1688159cb0 2024-01-27T00:46:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.832+0000 7f166effd700 1 -- 172.21.15.46:0/2282242149 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 89) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f1680003040 con 0x7f1688159cb0 2024-01-27T00:46:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.832+0000 7f166effd700 1 --2- 172.21.15.46:0/2282242149 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1674066f80 0x7f1674069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.832+0000 7f166effd700 1 -- 172.21.15.46:0/2282242149 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f1680086990 con 0x7f1688159cb0 2024-01-27T00:46:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.832+0000 7f1686d9d700 1 -- 172.21.15.46:0/2282242149 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1674066f80 msgr2=0x7f1674069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.832+0000 7f1686d9d700 1 --2- 172.21.15.46:0/2282242149 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1674066f80 0x7f1674069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:46:54.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:54.835+0000 7f166effd700 1 -- 172.21.15.46:0/2282242149 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1680053990 con 0x7f1688159cb0 2024-01-27T00:46:54.995 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:54.995+0000 7fb7e2b121c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T00:46:55.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.033+0000 7f1686d9d700 1 -- 172.21.15.46:0/2282242149 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1674066f80 msgr2=0x7f1674069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:55.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.033+0000 7f1686d9d700 1 --2- 172.21.15.46:0/2282242149 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1674066f80 0x7f1674069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:46:55.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.055+0000 7f168d09a700 1 -- 172.21.15.46:0/2282242149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f168804c710 con 0x7f1688159cb0 2024-01-27T00:46:55.056 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:55.056+0000 7f315df8a1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T00:46:55.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.059+0000 7f166effd700 1 -- 172.21.15.46:0/2282242149 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v59)=0 v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1680056fb0 con 0x7f1688159cb0 2024-01-27T00:46:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.061+0000 7f168d09a700 1 -- 172.21.15.46:0/2282242149 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1674066f80 msgr2=0x7f1674069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:46:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.061+0000 7f168d09a700 1 --2- 172.21.15.46:0/2282242149 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1674066f80 0x7f1674069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.061+0000 7f168d09a700 1 -- 172.21.15.46:0/2282242149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1688159cb0 msgr2=0x7f1688156ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.061+0000 7f168d09a700 1 --2- 172.21.15.46:0/2282242149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1688159cb0 0x7f1688156ae0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f1688088df0 tx=0x7f1680018000 comp rx=0 tx=0).stop 2024-01-27T00:46:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.061+0000 7f168d09a700 1 -- 172.21.15.46:0/2282242149 shutdown_connections 2024-01-27T00:46:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.061+0000 7f168d09a700 1 --2- 172.21.15.46:0/2282242149 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f1674066f80 0x7f1674069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.061+0000 7f168d09a700 1 --2- 172.21.15.46:0/2282242149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1688159cb0 0x7f1688156ae0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.061+0000 7f168d09a700 1 --2- 172.21.15.46:0/2282242149 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16880afb40 0x7f16880aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.061+0000 7f168d09a700 1 --2- 172.21.15.46:0/2282242149 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16880cf050 0x7f16880c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.061+0000 7f168d09a700 1 -- 172.21.15.46:0/2282242149 >> 172.21.15.46:0/2282242149 conn(0x7f1688087f60 msgr2=0x7f16880c6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.061+0000 7f168d09a700 1 -- 172.21.15.46:0/2282242149 shutdown_connections 2024-01-27T00:46:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.061+0000 7f168d09a700 1 -- 172.21.15.46:0/2282242149 wait complete. 2024-01-27T00:46:55.071 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:55.071+0000 7fb7e2b121c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T00:46:55.076 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-27T00:46:55.141 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:55.141+0000 7fb7e2b121c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T00:46:55.147 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:55.147+0000 7f315df8a1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T00:46:55.205 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:55.205+0000 7fb7e2b121c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T00:46:55.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.211+0000 7f7e73ea4700 1 Processor -- start 2024-01-27T00:46:55.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.212+0000 7f7e73ea4700 1 -- start start 2024-01-27T00:46:55.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.213+0000 7f7e73ea4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e6c141200 0x7f7e6c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:55.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.213+0000 7f7e73ea4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7e6c0caf30 0x7f7e6c0cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:55.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.213+0000 7f7e73ea4700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e6c0ccc00 con 0x7f7e6c0c8da0 2024-01-27T00:46:55.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.213+0000 7f7e73ea4700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e6c0cbbf0 con 0x7f7e6c0caf30 2024-01-27T00:46:55.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.213+0000 7f7e73ea4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c0cc310 con 0x7f7e6c141200 2024-01-27T00:46:55.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.213+0000 7f7e72441700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7e6c0caf30 0x7f7e6c0cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.213+0000 7f7e71c40700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7e6c0c8da0 0x7f7e6c140af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60536/0 (socket says 172.21.15.46:60536) 2024-01-27T00:46:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.213+0000 7f7e72441700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7e6c0caf30 0x7f7e6c0cb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41864/0 (socket says 172.21.15.46:41864) 2024-01-27T00:46:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.213+0000 7f7e71c40700 1 -- 172.21.15.46:0/2299729138 learned_addr learned my addr 172.21.15.46:0/2299729138 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.213+0000 7f7e7143f700 1 --2- 172.21.15.46:0/2299729138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e6c141200 0x7f7e6c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.213+0000 7f7e72441700 1 -- 172.21.15.46:0/2299729138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e6c141200 msgr2=0x7f7e6c0c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:46:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.213+0000 7f7e72441700 1 --2- 172.21.15.46:0/2299729138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e6c141200 0x7f7e6c0c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.213+0000 7f7e72441700 1 -- 172.21.15.46:0/2299729138 >> v1:172.21.15.46:6789/0 conn(0x7f7e6c0c8da0 legacy=0x7f7e6c140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.213+0000 7f7e72441700 1 -- 172.21.15.46:0/2299729138 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6c0cc5b0 con 0x7f7e6c0caf30 2024-01-27T00:46:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.213+0000 7f7e70c3e700 1 -- 172.21.15.46:0/2299729138 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091983792 0 0) 0x7f7e6c0ccc00 con 0x7f7e6c0c8da0 2024-01-27T00:46:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.213+0000 7f7e72441700 1 --2- 172.21.15.46:0/2299729138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c0caf30 0x7f7e6c0cb370 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f7e64009e60 tx=0x7f7e64009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32896458ea1143dd server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.214+0000 7f7e70c3e700 1 -- 172.21.15.46:0/2299729138 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400e070 con 0x7f7e6c0caf30 2024-01-27T00:46:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.214+0000 7f7e70c3e700 1 -- 172.21.15.46:0/2299729138 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6400bb60 con 0x7f7e6c0caf30 2024-01-27T00:46:55.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.214+0000 7f7e70c3e700 1 -- 172.21.15.46:0/2299729138 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64012430 con 0x7f7e6c0caf30 2024-01-27T00:46:55.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.214+0000 7f7e73ea4700 1 -- 172.21.15.46:0/2299729138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c0caf30 msgr2=0x7f7e6c0cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:55.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.214+0000 7f7e73ea4700 1 --2- 172.21.15.46:0/2299729138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c0caf30 0x7f7e6c0cb370 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f7e64009e60 tx=0x7f7e64009780 comp rx=0 tx=0).stop 2024-01-27T00:46:55.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.214+0000 7f7e73ea4700 1 -- 172.21.15.46:0/2299729138 shutdown_connections 2024-01-27T00:46:55.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.214+0000 7f7e73ea4700 1 --2- 172.21.15.46:0/2299729138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c0caf30 0x7f7e6c0cb370 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.214+0000 7f7e73ea4700 1 --2- 172.21.15.46:0/2299729138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e6c141200 0x7f7e6c0c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.214+0000 7f7e73ea4700 1 -- 172.21.15.46:0/2299729138 >> 172.21.15.46:0/2299729138 conn(0x7f7e6c06bd90 msgr2=0x7f7e6c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:55.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.214+0000 7f7e73ea4700 1 -- 172.21.15.46:0/2299729138 shutdown_connections 2024-01-27T00:46:55.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.215+0000 7f7e73ea4700 1 -- 172.21.15.46:0/2299729138 wait complete. 2024-01-27T00:46:55.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.216+0000 7f7e73ea4700 1 Processor -- start 2024-01-27T00:46:55.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.216+0000 7f7e73ea4700 1 -- start start 2024-01-27T00:46:55.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.216+0000 7f7e73ea4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e6c0c8da0 0x7f7e6c1570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:55.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.216+0000 7f7e73ea4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e6c0caf30 0x7f7e6c0c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:55.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.216+0000 7f7e73ea4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c141200 0x7f7e6c0a5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:55.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.216+0000 7f7e73ea4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c069760 con 0x7f7e6c141200 2024-01-27T00:46:55.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.216+0000 7f7e73ea4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c068b50 con 0x7f7e6c0c8da0 2024-01-27T00:46:55.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.216+0000 7f7e73ea4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c068e30 con 0x7f7e6c0caf30 2024-01-27T00:46:55.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.217+0000 7f7e72441700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c141200 0x7f7e6c0a5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:55.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.217+0000 7f7e72441700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c141200 0x7f7e6c0a5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41874/0 (socket says 172.21.15.46:41874) 2024-01-27T00:46:55.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.217+0000 7f7e72441700 1 -- 172.21.15.46:0/2127208788 learned_addr learned my addr 172.21.15.46:0/2127208788 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:55.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.217+0000 7f7e71c40700 1 --2- 172.21.15.46:0/2127208788 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e6c0c8da0 0x7f7e6c1570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:55.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.217+0000 7f7e7143f700 1 --2- 172.21.15.46:0/2127208788 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e6c0caf30 0x7f7e6c0c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:55.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.217+0000 7f7e72441700 1 -- 172.21.15.46:0/2127208788 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e6c0caf30 msgr2=0x7f7e6c0c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:55.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.217+0000 7f7e72441700 1 --2- 172.21.15.46:0/2127208788 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e6c0caf30 0x7f7e6c0c8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.217+0000 7f7e72441700 1 -- 172.21.15.46:0/2127208788 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e6c0c8da0 msgr2=0x7f7e6c1570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:55.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.217+0000 7f7e72441700 1 --2- 172.21.15.46:0/2127208788 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e6c0c8da0 0x7f7e6c1570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.217+0000 7f7e72441700 1 -- 172.21.15.46:0/2127208788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e64009a20 con 0x7f7e6c141200 2024-01-27T00:46:55.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.217+0000 7f7e72441700 1 --2- 172.21.15.46:0/2127208788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c141200 0x7f7e6c0a5f50 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400b6c0 tx=0x7f7e64019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:55.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.217+0000 7f7e61ffb700 1 -- 172.21.15.46:0/2127208788 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64025440 con 0x7f7e6c141200 2024-01-27T00:46:55.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.217+0000 7f7e61ffb700 1 -- 172.21.15.46:0/2127208788 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e640255a0 con 0x7f7e6c141200 2024-01-27T00:46:55.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.217+0000 7f7e73ea4700 1 -- 172.21.15.46:0/2127208788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e6c069110 con 0x7f7e6c141200 2024-01-27T00:46:55.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.217+0000 7f7e61ffb700 1 -- 172.21.15.46:0/2127208788 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64025960 con 0x7f7e6c141200 2024-01-27T00:46:55.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.218+0000 7f7e73ea4700 1 -- 172.21.15.46:0/2127208788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e6c157600 con 0x7f7e6c141200 2024-01-27T00:46:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.219+0000 7f7e73ea4700 1 -- 172.21.15.46:0/2127208788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e6c0695b0 con 0x7f7e6c141200 2024-01-27T00:46:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.219+0000 7f7e61ffb700 1 -- 172.21.15.46:0/2127208788 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 89) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f7e64003050 con 0x7f7e6c141200 2024-01-27T00:46:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.220+0000 7f7e61ffb700 1 --2- 172.21.15.46:0/2127208788 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e50066f80 0x7f7e50069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.220+0000 7f7e71c40700 1 -- 172.21.15.46:0/2127208788 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e50066f80 msgr2=0x7f7e50069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.220+0000 7f7e61ffb700 1 -- 172.21.15.46:0/2127208788 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f7e64086fd0 con 0x7f7e6c141200 2024-01-27T00:46:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.220+0000 7f7e71c40700 1 --2- 172.21.15.46:0/2127208788 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e50066f80 0x7f7e50069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:46:55.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.224+0000 7f7e61ffb700 1 -- 172.21.15.46:0/2127208788 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7e64053f30 con 0x7f7e6c141200 2024-01-27T00:46:55.232 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:55.232+0000 7f315df8a1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T00:46:55.308 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:55.307+0000 7f315df8a1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T00:46:55.360 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:55.361+0000 7fb7e2b121c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:46:55.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.421+0000 7f7e71c40700 1 -- 172.21.15.46:0/2127208788 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e50066f80 msgr2=0x7f7e50069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:55.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.421+0000 7f7e71c40700 1 --2- 172.21.15.46:0/2127208788 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e50066f80 0x7f7e50069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:46:55.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.431+0000 7f7e73ea4700 1 -- 172.21.15.46:0/2127208788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f7e6c04c710 con 0x7f7e6c141200 2024-01-27T00:46:55.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.435+0000 7f7e61ffb700 1 -- 172.21.15.46:0/2127208788 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v60)=0 v60) v1 ==== 133+0+0 (secure 0 0 0) 0x7f7e64057550 con 0x7f7e6c141200 2024-01-27T00:46:55.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.437+0000 7f7e73ea4700 1 -- 172.21.15.46:0/2127208788 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e50066f80 msgr2=0x7f7e50069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:46:55.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.437+0000 7f7e73ea4700 1 --2- 172.21.15.46:0/2127208788 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e50066f80 0x7f7e50069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.437+0000 7f7e73ea4700 1 -- 172.21.15.46:0/2127208788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c141200 msgr2=0x7f7e6c0a5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:55.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.437+0000 7f7e73ea4700 1 --2- 172.21.15.46:0/2127208788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c141200 0x7f7e6c0a5f50 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400b6c0 tx=0x7f7e64019000 comp rx=0 tx=0).stop 2024-01-27T00:46:55.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.438+0000 7f7e73ea4700 1 -- 172.21.15.46:0/2127208788 shutdown_connections 2024-01-27T00:46:55.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.438+0000 7f7e73ea4700 1 --2- 172.21.15.46:0/2127208788 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f7e50066f80 0x7f7e50069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.438+0000 7f7e73ea4700 1 --2- 172.21.15.46:0/2127208788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c141200 0x7f7e6c0a5f50 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.438+0000 7f7e73ea4700 1 --2- 172.21.15.46:0/2127208788 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e6c0c8da0 0x7f7e6c1570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.438+0000 7f7e73ea4700 1 --2- 172.21.15.46:0/2127208788 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e6c0caf30 0x7f7e6c0c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.438+0000 7f7e73ea4700 1 -- 172.21.15.46:0/2127208788 >> 172.21.15.46:0/2127208788 conn(0x7f7e6c06bd90 msgr2=0x7f7e6c13d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:55.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.438+0000 7f7e73ea4700 1 -- 172.21.15.46:0/2127208788 shutdown_connections 2024-01-27T00:46:55.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.438+0000 7f7e73ea4700 1 -- 172.21.15.46:0/2127208788 wait complete. 2024-01-27T00:46:55.452 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:46:55.452 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-27T00:46:55.475 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1h 2024-01-27T00:46:55.485 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:55.485+0000 7f315df8a1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:46:55.497 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:55.497+0000 7fb7e2b121c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T00:46:55.564 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:55.564+0000 7fb7e2b121c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T00:46:55.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.621+0000 7fc86adc2700 1 Processor -- start 2024-01-27T00:46:55.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.621+0000 7fc86adc2700 1 -- start start 2024-01-27T00:46:55.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.622+0000 7fc86adc2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc864153c40 0x7fc86406d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:55.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.622+0000 7fc86adc2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86415ad60 con 0x7fc864154010 2024-01-27T00:46:55.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.622+0000 7fc86adc2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86415a150 con 0x7fc8640b1e80 2024-01-27T00:46:55.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.622+0000 7fc86adc2700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8640b1d20 con 0x7fc864153c40 2024-01-27T00:46:55.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.622+0000 7fc863fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc864154010 0x7fc86406dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60552/0 (socket says 172.21.15.46:60552) 2024-01-27T00:46:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.622+0000 7fc863fff700 1 -- 172.21.15.46:0/3282568378 learned_addr learned my addr 172.21.15.46:0/3282568378 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:46:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.622+0000 7fc868b5e700 1 --2- 172.21.15.46:0/3282568378 >> v2:172.21.15.106:3300/0 conn(0x7fc864153c40 0x7fc86406d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.623+0000 7fc8637fe700 1 -- 172.21.15.46:0/3282568378 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995580367 0 0) 0x7fc86415ad60 con 0x7fc864154010 2024-01-27T00:46:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.623+0000 7fc8637fe700 1 -- 172.21.15.46:0/3282568378 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc854003570 con 0x7fc864154010 2024-01-27T00:46:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.623+0000 7fc8637fe700 1 -- 172.21.15.46:0/3282568378 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274597568 0 0) 0x7fc86415a150 con 0x7fc8640b1e80 2024-01-27T00:46:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.623+0000 7fc8637fe700 1 -- 172.21.15.46:0/3282568378 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc86415ad60 con 0x7fc8640b1e80 2024-01-27T00:46:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.623+0000 7fc868b5e700 1 -- 172.21.15.46:0/3282568378 >> v1:172.21.15.106:6789/0 conn(0x7fc8640b1e80 legacy=0x7fc8640692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.623+0000 7fc868b5e700 1 -- 172.21.15.46:0/3282568378 >> v1:172.21.15.46:6789/0 conn(0x7fc864154010 legacy=0x7fc86406dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.623+0000 7fc868b5e700 1 -- 172.21.15.46:0/3282568378 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86415a430 con 0x7fc864153c40 2024-01-27T00:46:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.623+0000 7fc868b5e700 1 --2- 172.21.15.46:0/3282568378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc864153c40 0x7fc86406d6a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fc8580098e0 tx=0x7fc8580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66f19a5ee3d6aca7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.623+0000 7fc8637fe700 1 -- 172.21.15.46:0/3282568378 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc858011630 con 0x7fc864153c40 2024-01-27T00:46:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.623+0000 7fc8637fe700 1 -- 172.21.15.46:0/3282568378 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc858011790 con 0x7fc864153c40 2024-01-27T00:46:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.623+0000 7fc8637fe700 1 -- 172.21.15.46:0/3282568378 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc858011b30 con 0x7fc864153c40 2024-01-27T00:46:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.624+0000 7fc86adc2700 1 -- 172.21.15.46:0/3282568378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc864153c40 msgr2=0x7fc86406d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.624+0000 7fc86adc2700 1 --2- 172.21.15.46:0/3282568378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc864153c40 0x7fc86406d6a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fc8580098e0 tx=0x7fc8580091a0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.624+0000 7fc86adc2700 1 -- 172.21.15.46:0/3282568378 shutdown_connections 2024-01-27T00:46:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.624+0000 7fc86adc2700 1 --2- 172.21.15.46:0/3282568378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc864153c40 0x7fc86406d6a0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.624+0000 7fc86adc2700 1 -- 172.21.15.46:0/3282568378 >> 172.21.15.46:0/3282568378 conn(0x7fc864067930 msgr2=0x7fc864067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:46:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.624+0000 7fc86adc2700 1 -- 172.21.15.46:0/3282568378 shutdown_connections 2024-01-27T00:46:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.624+0000 7fc86adc2700 1 -- 172.21.15.46:0/3282568378 wait complete. 2024-01-27T00:46:55.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.625+0000 7fc86adc2700 1 Processor -- start 2024-01-27T00:46:55.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.625+0000 7fc86adc2700 1 -- start start 2024-01-27T00:46:55.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.626+0000 7fc86adc2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8640b1e80 0x7fc8640ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:55.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.626+0000 7fc86adc2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc864153c40 0x7fc86414ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:55.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.626+0000 7fc86adc2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc864154010 0x7fc8640ce600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:55.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.626+0000 7fc86adc2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8640c8730 con 0x7fc8640b1e80 2024-01-27T00:46:55.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.626+0000 7fc86adc2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8640c8910 con 0x7fc864154010 2024-01-27T00:46:55.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.626+0000 7fc86adc2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8640c8bf0 con 0x7fc864153c40 2024-01-27T00:46:55.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.626+0000 7fc868b5e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8640b1e80 0x7fc8640ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:55.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.626+0000 7fc868b5e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8640b1e80 0x7fc8640ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41890/0 (socket says 172.21.15.46:41890) 2024-01-27T00:46:55.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.626+0000 7fc86935f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc864154010 0x7fc8640ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:55.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.626+0000 7fc868b5e700 1 -- 172.21.15.46:0/3905529207 learned_addr learned my addr 172.21.15.46:0/3905529207 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:46:55.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.626+0000 7fc863fff700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc864153c40 0x7fc86414ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:46:55.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.627+0000 7fc868b5e700 1 -- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc864153c40 msgr2=0x7fc86414ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:55.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.627+0000 7fc868b5e700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc864153c40 0x7fc86414ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.627+0000 7fc868b5e700 1 -- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc864154010 msgr2=0x7fc8640ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:46:55.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.627+0000 7fc868b5e700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc864154010 0x7fc8640ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:55.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.627+0000 7fc868b5e700 1 -- 172.21.15.46:0/3905529207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc858009420 con 0x7fc8640b1e80 2024-01-27T00:46:55.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.627+0000 7fc86935f700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc864154010 0x7fc8640ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:46:55.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.627+0000 7fc868b5e700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8640b1e80 0x7fc8640ccdb0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fc86406d060 tx=0x7fc858017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:46:55.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.627+0000 7fc860ff9700 1 -- 172.21.15.46:0/3905529207 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85801f070 con 0x7fc8640b1e80 2024-01-27T00:46:55.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.627+0000 7fc860ff9700 1 -- 172.21.15.46:0/3905529207 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc858011900 con 0x7fc8640b1e80 2024-01-27T00:46:55.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.627+0000 7fc86adc2700 1 -- 172.21.15.46:0/3905529207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8640c8ed0 con 0x7fc8640b1e80 2024-01-27T00:46:55.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.627+0000 7fc860ff9700 1 -- 172.21.15.46:0/3905529207 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc858023410 con 0x7fc8640b1e80 2024-01-27T00:46:55.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.627+0000 7fc86adc2700 1 -- 172.21.15.46:0/3905529207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc864150510 con 0x7fc8640b1e80 2024-01-27T00:46:55.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.628+0000 7fc86adc2700 1 -- 172.21.15.46:0/3905529207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc86404c710 con 0x7fc8640b1e80 2024-01-27T00:46:55.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.629+0000 7fc860ff9700 1 -- 172.21.15.46:0/3905529207 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 89) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fc85802c3b0 con 0x7fc8640b1e80 2024-01-27T00:46:55.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.629+0000 7fc860ff9700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 0x7fc848069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:46:55.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.629+0000 7fc860ff9700 1 -- 172.21.15.46:0/3905529207 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fc858085a90 con 0x7fc8640b1e80 2024-01-27T00:46:55.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.629+0000 7fc863fff700 1 -- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 msgr2=0x7fc848069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:55.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.629+0000 7fc863fff700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 0x7fc848069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:46:55.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.638+0000 7fc860ff9700 1 -- 172.21.15.46:0/3905529207 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc858052a10 con 0x7fc8640b1e80 2024-01-27T00:46:55.651 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:55.651+0000 7f315df8a1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T00:46:55.686 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:55.687+0000 7fb7e2b121c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T00:46:55.728 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:55.728+0000 7f315df8a1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T00:46:55.758 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:55.758+0000 7fb7e2b121c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T00:46:55.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.830+0000 7fc863fff700 1 -- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 msgr2=0x7fc848069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:55.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.830+0000 7fc863fff700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 0x7fc848069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:46:55.867 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:55.867+0000 7f315df8a1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T00:46:55.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:55.872+0000 7fc86adc2700 1 -- 172.21.15.46:0/3905529207 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7fc86415a710 con 0x7fc848066f80 2024-01-27T00:46:55.898 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:55.898+0000 7fb7e2b121c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T00:46:55.949 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:55.949+0000 7f315df8a1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T00:46:56.022 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:56.022+0000 7fb7e2b121c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T00:46:56.125 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:56.125+0000 7f315df8a1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T00:46:56.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:56.231+0000 7fc863fff700 1 -- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 msgr2=0x7fc848069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:56.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:56.231+0000 7fc863fff700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 0x7fc848069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-27T00:46:56.266 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:56.265+0000 7f315df8a1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T00:46:56.333 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:56.333+0000 7fb7e2b121c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T00:46:56.541 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:56.542+0000 7fb7e2b121c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T00:46:56.626 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:56.625+0000 7f315df8a1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T00:46:56.766 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:56.766+0000 7fb7e2b121c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T00:46:56.836 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:56.836+0000 7fb7e2b121c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T00:46:56.864 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:56.864+0000 7f315df8a1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T00:46:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:57.032+0000 7fc863fff700 1 -- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 msgr2=0x7fc848069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:57.032+0000 7fc863fff700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 0x7fc848069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-27T00:46:57.122 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:57.122+0000 7f315df8a1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T00:46:57.203 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:57.202+0000 7f315df8a1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T00:46:57.432 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:57.432+0000 7fb7e2b121c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T00:46:57.507 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:57.507+0000 7fb7e2b121c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T00:46:57.629 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:57.629+0000 7fb7e2b121c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T00:46:57.899 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:57.899+0000 7f315df8a1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T00:46:57.983 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:57.983+0000 7f315df8a1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T00:46:58.124 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:58.124+0000 7fb7e2b121c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T00:46:58.124 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:58.124+0000 7f315df8a1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T00:46:58.271 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:58.272+0000 7fb7e2b121c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T00:46:58.422 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:58.422+0000 7fb7e2b121c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T00:46:58.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:58.634+0000 7fc863fff700 1 -- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 msgr2=0x7fc848069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:46:58.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:58.634+0000 7fc863fff700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 0x7fc848069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-27T00:46:58.707 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:58.707+0000 7f315df8a1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T00:46:58.876 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:58.876+0000 7f315df8a1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T00:46:59.047 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:59.047+0000 7f315df8a1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T00:46:59.115 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:59.115+0000 7fb7e2b121c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T00:46:59.330 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:59.330+0000 7fb7e2b121c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:46:59.400 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:59.400+0000 7fb7e2b121c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T00:46:59.554 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:59.554+0000 7fb7e2b121c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T00:46:59.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:59.573+0000 7fc860ff9700 1 -- 172.21.15.46:0/3905529207 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 90) v1 ==== 84470+0+0 (secure 0 0 0) 0x7fc858052de0 con 0x7fc8640b1e80 2024-01-27T00:46:59.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:59.574+0000 7fc860ff9700 1 -- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 msgr2=0x7fc848069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:46:59.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:46:59.574+0000 7fc860ff9700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 0x7fc848069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:46:59.685 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:59.684+0000 7fb75bbb5700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:59.685 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:59.684+0000 7fb75bbb5700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:59.685 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:59.684+0000 7fb75bbb5700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:59.685 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:59.684+0000 7fb75bbb5700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:59.685 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:59.684+0000 7fb75bbb5700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:59.685 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:59.685+0000 7fb7593b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:59.685 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:59.685+0000 7fb7593b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:59.686 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:59.685+0000 7fb7593b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:59.686 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:59.685+0000 7fb7593b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:59.686 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:46:59.685+0000 7fb7593b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:46:59.865 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:46:59.865+0000 7f315df8a1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T00:47:00.113 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:00.112+0000 7f315df8a1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:47:00.195 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:00.195+0000 7f315df8a1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T00:47:00.371 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:00.371+0000 7f315df8a1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T00:47:00.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.578+0000 7fc860ff9700 1 -- 172.21.15.46:0/3905529207 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 91) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fc858072120 con 0x7fc8640b1e80 2024-01-27T00:47:00.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.578+0000 7fc860ff9700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 0x7fc848069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:00.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.578+0000 7fc860ff9700 1 -- 172.21.15.46:0/3905529207 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7fc86415a710 con 0x7fc848066f80 2024-01-27T00:47:00.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.579+0000 7fc863fff700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 0x7fc848069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:00.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.581+0000 7fc863fff700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 0x7fc848069430 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc8640ce3f0 tx=0x7fc850009040 comp rx=0 tx=0).ready entity=mgr.9208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:00.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.639+0000 7fc860ff9700 1 -- 172.21.15.46:0/3905529207 <== mgr.9208 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc86415a710 con 0x7fc848066f80 2024-01-27T00:47:00.640 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path snap_test_dir1 2024-01-27T00:47:00.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.642+0000 7fc86adc2700 1 -- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 msgr2=0x7fc848069430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:00.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.642+0000 7fc86adc2700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 0x7fc848069430 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc8640ce3f0 tx=0x7fc850009040 comp rx=0 tx=0).stop 2024-01-27T00:47:00.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.642+0000 7fc86adc2700 1 -- 172.21.15.46:0/3905529207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8640b1e80 msgr2=0x7fc8640ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:00.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.642+0000 7fc86adc2700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8640b1e80 0x7fc8640ccdb0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fc86406d060 tx=0x7fc858017000 comp rx=0 tx=0).stop 2024-01-27T00:47:00.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.643+0000 7fc86adc2700 1 -- 172.21.15.46:0/3905529207 shutdown_connections 2024-01-27T00:47:00.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.643+0000 7fc86adc2700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fc848066f80 0x7fc848069430 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:00.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.643+0000 7fc86adc2700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8640b1e80 0x7fc8640ccdb0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:00.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.643+0000 7fc86adc2700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc864154010 0x7fc8640ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:00.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.643+0000 7fc86adc2700 1 --2- 172.21.15.46:0/3905529207 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc864153c40 0x7fc86414ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:00.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.643+0000 7fc86adc2700 1 -- 172.21.15.46:0/3905529207 >> 172.21.15.46:0/3905529207 conn(0x7fc864067930 msgr2=0x7fc8640b07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:00.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.643+0000 7fc86adc2700 1 -- 172.21.15.46:0/3905529207 shutdown_connections 2024-01-27T00:47:00.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.643+0000 7fc86adc2700 1 -- 172.21.15.46:0/3905529207 wait complete. 2024-01-27T00:47:00.658 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-01-27T00:47:00.658 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-01-27T00:47:00.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.797+0000 7f92c202c700 1 Processor -- start 2024-01-27T00:47:00.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.797+0000 7f92c202c700 1 -- start start 2024-01-27T00:47:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.798+0000 7f92c202c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f92bc067af0 0x7f92bc067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.798+0000 7f92c202c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92bc1597e0 0x7f92bc159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.798+0000 7f92c202c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92bc06fdb0 con 0x7f92bc06ebe0 2024-01-27T00:47:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.798+0000 7f92c202c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92bc06f1a0 con 0x7f92bc067af0 2024-01-27T00:47:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.798+0000 7f92c202c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc06f480 con 0x7f92bc1597e0 2024-01-27T00:47:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.798+0000 7f92baffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f92bc06ebe0 0x7f92bc0ad8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37150/0 (socket says 172.21.15.46:37150) 2024-01-27T00:47:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.798+0000 7f92baffd700 1 -- 172.21.15.46:0/2302667847 learned_addr learned my addr 172.21.15.46:0/2302667847 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.798+0000 7f92bbfff700 1 --2- 172.21.15.46:0/2302667847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92bc1597e0 0x7f92bc159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.798+0000 7f92bb7fe700 1 --2- 172.21.15.46:0/2302667847 >> v2:172.21.15.106:3300/0 conn(0x7f92bc067af0 0x7f92bc067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.798+0000 7f92ba7fc700 1 -- 172.21.15.46:0/2302667847 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337369412 0 0) 0x7f92bc06fdb0 con 0x7f92bc06ebe0 2024-01-27T00:47:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.798+0000 7f92ba7fc700 1 -- 172.21.15.46:0/2302667847 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a4003570 con 0x7f92bc06ebe0 2024-01-27T00:47:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.799+0000 7f92bb7fe700 1 -- 172.21.15.46:0/2302667847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92bc1597e0 msgr2=0x7f92bc159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.799+0000 7f92bb7fe700 1 --2- 172.21.15.46:0/2302667847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92bc1597e0 0x7f92bc159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.799+0000 7f92bb7fe700 1 -- 172.21.15.46:0/2302667847 >> v1:172.21.15.46:6789/0 conn(0x7f92bc06ebe0 legacy=0x7f92bc0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.799+0000 7f92bb7fe700 1 -- 172.21.15.46:0/2302667847 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92bc06f760 con 0x7f92bc067af0 2024-01-27T00:47:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.799+0000 7f92bb7fe700 1 --2- 172.21.15.46:0/2302667847 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92bc067af0 0x7f92bc067ec0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0098e0 tx=0x7f92ac0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc5af6083e34ea7a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.799+0000 7f92ba7fc700 1 -- 172.21.15.46:0/2302667847 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac011630 con 0x7f92bc067af0 2024-01-27T00:47:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.799+0000 7f92ba7fc700 1 -- 172.21.15.46:0/2302667847 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ac011790 con 0x7f92bc067af0 2024-01-27T00:47:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.799+0000 7f92ba7fc700 1 -- 172.21.15.46:0/2302667847 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac011b30 con 0x7f92bc067af0 2024-01-27T00:47:00.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.800+0000 7f92c202c700 1 -- 172.21.15.46:0/2302667847 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92bc067af0 msgr2=0x7f92bc067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:00.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.800+0000 7f92c202c700 1 --2- 172.21.15.46:0/2302667847 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92bc067af0 0x7f92bc067ec0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0098e0 tx=0x7f92ac0091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:00.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.800+0000 7f92c202c700 1 -- 172.21.15.46:0/2302667847 shutdown_connections 2024-01-27T00:47:00.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.800+0000 7f92c202c700 1 --2- 172.21.15.46:0/2302667847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92bc1597e0 0x7f92bc159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:00.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.800+0000 7f92c202c700 1 --2- 172.21.15.46:0/2302667847 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92bc067af0 0x7f92bc067ec0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:00.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.800+0000 7f92c202c700 1 -- 172.21.15.46:0/2302667847 >> 172.21.15.46:0/2302667847 conn(0x7f92bc0cdf80 msgr2=0x7f92bc06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:00.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.800+0000 7f92c202c700 1 -- 172.21.15.46:0/2302667847 shutdown_connections 2024-01-27T00:47:00.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.800+0000 7f92c202c700 1 -- 172.21.15.46:0/2302667847 wait complete. 2024-01-27T00:47:00.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.801+0000 7f92c202c700 1 Processor -- start 2024-01-27T00:47:00.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.801+0000 7f92c202c700 1 -- start start 2024-01-27T00:47:00.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.802+0000 7f92c202c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc067af0 0x7f92bc157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:00.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.802+0000 7f92c202c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92bc06ebe0 0x7f92bc15b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:00.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.802+0000 7f92c202c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92bc1597e0 0x7f92bc16d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:00.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.802+0000 7f92c202c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc15b310 con 0x7f92bc067af0 2024-01-27T00:47:00.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.802+0000 7f92c202c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc15ad30 con 0x7f92bc1597e0 2024-01-27T00:47:00.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.802+0000 7f92c202c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc15a6c0 con 0x7f92bc06ebe0 2024-01-27T00:47:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.802+0000 7f92bb7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc067af0 0x7f92bc157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.802+0000 7f92bb7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc067af0 0x7f92bc157e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38798/0 (socket says 172.21.15.46:38798) 2024-01-27T00:47:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.802+0000 7f92bb7fe700 1 -- 172.21.15.46:0/1995802234 learned_addr learned my addr 172.21.15.46:0/1995802234 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.802+0000 7f92bbfff700 1 --2- 172.21.15.46:0/1995802234 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92bc1597e0 0x7f92bc16d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.802+0000 7f92baffd700 1 --2- 172.21.15.46:0/1995802234 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92bc06ebe0 0x7f92bc15b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.802+0000 7f92bb7fe700 1 -- 172.21.15.46:0/1995802234 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92bc06ebe0 msgr2=0x7f92bc15b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.802+0000 7f92bb7fe700 1 --2- 172.21.15.46:0/1995802234 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92bc06ebe0 0x7f92bc15b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.802+0000 7f92bb7fe700 1 -- 172.21.15.46:0/1995802234 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92bc1597e0 msgr2=0x7f92bc16d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.802+0000 7f92bb7fe700 1 --2- 172.21.15.46:0/1995802234 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92bc1597e0 0x7f92bc16d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.802+0000 7f92bb7fe700 1 -- 172.21.15.46:0/1995802234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ac009420 con 0x7f92bc067af0 2024-01-27T00:47:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.802+0000 7f92bbfff700 1 --2- 172.21.15.46:0/1995802234 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92bc1597e0 0x7f92bc16d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.803+0000 7f92bb7fe700 1 --2- 172.21.15.46:0/1995802234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc067af0 0x7f92bc157e90 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f92ac011f20 tx=0x7f92ac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.803+0000 7f92a37fe700 1 -- 172.21.15.46:0/1995802234 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac000d10 con 0x7f92bc067af0 2024-01-27T00:47:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.803+0000 7f92a37fe700 1 -- 172.21.15.46:0/1995802234 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ac011b10 con 0x7f92bc067af0 2024-01-27T00:47:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.803+0000 7f92c202c700 1 -- 172.21.15.46:0/1995802234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92bc16e150 con 0x7f92bc067af0 2024-01-27T00:47:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.803+0000 7f92a37fe700 1 -- 172.21.15.46:0/1995802234 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac025620 con 0x7f92bc067af0 2024-01-27T00:47:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.803+0000 7f92c202c700 1 -- 172.21.15.46:0/1995802234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92bc16e430 con 0x7f92bc067af0 2024-01-27T00:47:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.804+0000 7f92c202c700 1 -- 172.21.15.46:0/1995802234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92bc04c710 con 0x7f92bc067af0 2024-01-27T00:47:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.804+0000 7f92a37fe700 1 -- 172.21.15.46:0/1995802234 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 91) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f92ac025a10 con 0x7f92bc067af0 2024-01-27T00:47:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.805+0000 7f92a37fe700 1 --2- 172.21.15.46:0/1995802234 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f92a8066dd0 0x7f92a8069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.805+0000 7f92a37fe700 1 -- 172.21.15.46:0/1995802234 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f92ac085c80 con 0x7f92bc067af0 2024-01-27T00:47:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.805+0000 7f92baffd700 1 --2- 172.21.15.46:0/1995802234 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f92a8066dd0 0x7f92a8069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.805+0000 7f92baffd700 1 --2- 172.21.15.46:0/1995802234 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f92a8066dd0 0x7f92a8069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f92bc16d540 tx=0x7f92b0009040 comp rx=0 tx=0).ready entity=mgr.9208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:00.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:00.809+0000 7f92a37fe700 1 -- 172.21.15.46:0/1995802234 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f92ac052db0 con 0x7f92bc067af0 2024-01-27T00:47:01.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.040+0000 7f92c202c700 1 -- 172.21.15.46:0/1995802234 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f92bc1591f0 con 0x7f92a8066dd0 2024-01-27T00:47:01.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.095+0000 7f92a37fe700 1 -- 172.21.15.46:0/1995802234 <== mgr.9208 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f92bc1591f0 con 0x7f92a8066dd0 2024-01-27T00:47:01.095 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path snap_test_dir1 2024-01-27T00:47:01.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.098+0000 7f92c202c700 1 -- 172.21.15.46:0/1995802234 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f92a8066dd0 msgr2=0x7f92a8069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:01.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.098+0000 7f92c202c700 1 --2- 172.21.15.46:0/1995802234 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f92a8066dd0 0x7f92a8069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f92bc16d540 tx=0x7f92b0009040 comp rx=0 tx=0).stop 2024-01-27T00:47:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.098+0000 7f92c202c700 1 -- 172.21.15.46:0/1995802234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc067af0 msgr2=0x7f92bc157e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.098+0000 7f92c202c700 1 --2- 172.21.15.46:0/1995802234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc067af0 0x7f92bc157e90 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f92ac011f20 tx=0x7f92ac017000 comp rx=0 tx=0).stop 2024-01-27T00:47:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.098+0000 7f92c202c700 1 -- 172.21.15.46:0/1995802234 shutdown_connections 2024-01-27T00:47:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.098+0000 7f92c202c700 1 --2- 172.21.15.46:0/1995802234 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f92a8066dd0 0x7f92a8069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.098+0000 7f92c202c700 1 --2- 172.21.15.46:0/1995802234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc067af0 0x7f92bc157e90 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.098+0000 7f92c202c700 1 --2- 172.21.15.46:0/1995802234 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92bc1597e0 0x7f92bc16d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.098+0000 7f92c202c700 1 --2- 172.21.15.46:0/1995802234 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92bc06ebe0 0x7f92bc15b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.098+0000 7f92c202c700 1 -- 172.21.15.46:0/1995802234 >> 172.21.15.46:0/1995802234 conn(0x7f92bc0cdf80 msgr2=0x7f92bc0b0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.098+0000 7f92c202c700 1 -- 172.21.15.46:0/1995802234 shutdown_connections 2024-01-27T00:47:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.098+0000 7f92c202c700 1 -- 172.21.15.46:0/1995802234 wait complete. 2024-01-27T00:47:01.113 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-01-27T00:47:01.113 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-01-27T00:47:01.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.248+0000 7f49e229a700 1 Processor -- start 2024-01-27T00:47:01.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.248+0000 7f49e229a700 1 -- start start 2024-01-27T00:47:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.249+0000 7f49e229a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f49dc157490 0x7f49dc0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.249+0000 7f49e229a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f49dc157860 0x7f49dc0cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.249+0000 7f49e229a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49dc0cea80 con 0x7f49dc06d6a0 2024-01-27T00:47:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.249+0000 7f49e229a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49dc0cdc60 con 0x7f49dc157860 2024-01-27T00:47:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.249+0000 7f49e229a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49dc0b2ae0 con 0x7f49dc157490 2024-01-27T00:47:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.249+0000 7f49db7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f49dc157860 0x7f49dc0cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.250+0000 7f49db7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f49dc157860 0x7f49dc0cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38802/0 (socket says 172.21.15.46:38802) 2024-01-27T00:47:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.250+0000 7f49db7fe700 1 -- 172.21.15.46:0/878741259 learned_addr learned my addr 172.21.15.46:0/878741259 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.250+0000 7f49dbfff700 1 --2- 172.21.15.46:0/878741259 >> v2:172.21.15.106:3300/0 conn(0x7f49dc157490 0x7f49dc0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.250+0000 7f49db7fe700 1 -- 172.21.15.46:0/878741259 >> v2:172.21.15.106:3300/0 conn(0x7f49dc157490 msgr2=0x7f49dc0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.250+0000 7f49db7fe700 1 --2- 172.21.15.46:0/878741259 >> v2:172.21.15.106:3300/0 conn(0x7f49dc157490 0x7f49dc0cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.250+0000 7f49db7fe700 1 -- 172.21.15.46:0/878741259 >> v1:172.21.15.106:6789/0 conn(0x7f49dc06d6a0 legacy=0x7f49dc15b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.250+0000 7f49db7fe700 1 -- 172.21.15.46:0/878741259 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49dc0cde70 con 0x7f49dc157860 2024-01-27T00:47:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.250+0000 7f49db7fe700 1 --2- 172.21.15.46:0/878741259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc157860 0x7f49dc0cd620 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f49dc06f9d0 tx=0x7f49d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259630b538575d2c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:01.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.250+0000 7f49daffd700 1 -- 172.21.15.46:0/878741259 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0009e70 con 0x7f49dc157860 2024-01-27T00:47:01.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.250+0000 7f49daffd700 1 -- 172.21.15.46:0/878741259 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d000b860 con 0x7f49dc157860 2024-01-27T00:47:01.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.250+0000 7f49daffd700 1 -- 172.21.15.46:0/878741259 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d000bc00 con 0x7f49dc157860 2024-01-27T00:47:01.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.251+0000 7f49e229a700 1 -- 172.21.15.46:0/878741259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc157860 msgr2=0x7f49dc0cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:01.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.251+0000 7f49e229a700 1 --2- 172.21.15.46:0/878741259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc157860 0x7f49dc0cd620 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f49dc06f9d0 tx=0x7f49d0009250 comp rx=0 tx=0).stop 2024-01-27T00:47:01.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.251+0000 7f49e229a700 1 -- 172.21.15.46:0/878741259 shutdown_connections 2024-01-27T00:47:01.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.251+0000 7f49e229a700 1 --2- 172.21.15.46:0/878741259 >> v2:172.21.15.106:3300/0 conn(0x7f49dc157490 0x7f49dc0cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:01.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.251+0000 7f49e229a700 1 --2- 172.21.15.46:0/878741259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc157860 0x7f49dc0cd620 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:01.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.251+0000 7f49e229a700 1 -- 172.21.15.46:0/878741259 >> 172.21.15.46:0/878741259 conn(0x7f49dc067930 msgr2=0x7f49dc067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:01.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.251+0000 7f49e229a700 1 -- 172.21.15.46:0/878741259 shutdown_connections 2024-01-27T00:47:01.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.251+0000 7f49e229a700 1 -- 172.21.15.46:0/878741259 wait complete. 2024-01-27T00:47:01.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.253+0000 7f49e229a700 1 Processor -- start 2024-01-27T00:47:01.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.253+0000 7f49e229a700 1 -- start start 2024-01-27T00:47:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.254+0000 7f49e229a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49dc06d6a0 0x7f49dc14c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.254+0000 7f49e229a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49dc157490 0x7f49dc14d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.254+0000 7f49e229a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc157860 0x7f49dc14d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.254+0000 7f49e229a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc15aab0 con 0x7f49dc157860 2024-01-27T00:47:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.254+0000 7f49e229a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc159bf0 con 0x7f49dc06d6a0 2024-01-27T00:47:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.254+0000 7f49e229a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc159ea0 con 0x7f49dc157490 2024-01-27T00:47:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.254+0000 7f49e0837700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc157860 0x7f49dc14d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.254+0000 7f49e0837700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc157860 0x7f49dc14d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38812/0 (socket says 172.21.15.46:38812) 2024-01-27T00:47:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.254+0000 7f49e0837700 1 -- 172.21.15.46:0/3337811569 learned_addr learned my addr 172.21.15.46:0/3337811569 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.254+0000 7f49dbfff700 1 --2- 172.21.15.46:0/3337811569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49dc06d6a0 0x7f49dc14c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.254+0000 7f49db7fe700 1 --2- 172.21.15.46:0/3337811569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49dc157490 0x7f49dc14d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.254+0000 7f49e0837700 1 -- 172.21.15.46:0/3337811569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49dc157490 msgr2=0x7f49dc14d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.254+0000 7f49e0837700 1 --2- 172.21.15.46:0/3337811569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49dc157490 0x7f49dc14d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:01.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.254+0000 7f49e0837700 1 -- 172.21.15.46:0/3337811569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49dc06d6a0 msgr2=0x7f49dc14c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:01.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.254+0000 7f49e0837700 1 --2- 172.21.15.46:0/3337811569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49dc06d6a0 0x7f49dc14c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:01.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.254+0000 7f49e0837700 1 -- 172.21.15.46:0/3337811569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d00094f0 con 0x7f49dc157860 2024-01-27T00:47:01.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.254+0000 7f49e0837700 1 --2- 172.21.15.46:0/3337811569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc157860 0x7f49dc14d7a0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f49d400c840 tx=0x7f49d4001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:01.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.255+0000 7f49c3fff700 1 -- 172.21.15.46:0/3337811569 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d4012630 con 0x7f49dc157860 2024-01-27T00:47:01.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.255+0000 7f49c3fff700 1 -- 172.21.15.46:0/3337811569 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d40127b0 con 0x7f49dc157860 2024-01-27T00:47:01.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.255+0000 7f49c3fff700 1 -- 172.21.15.46:0/3337811569 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d4012b70 con 0x7f49dc157860 2024-01-27T00:47:01.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.255+0000 7f49e229a700 1 -- 172.21.15.46:0/3337811569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49dc15a180 con 0x7f49dc157860 2024-01-27T00:47:01.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.255+0000 7f49e229a700 1 -- 172.21.15.46:0/3337811569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49dc15a740 con 0x7f49dc157860 2024-01-27T00:47:01.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.256+0000 7f49e229a700 1 -- 172.21.15.46:0/3337811569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49dc04c710 con 0x7f49dc157860 2024-01-27T00:47:01.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.256+0000 7f49c3fff700 1 -- 172.21.15.46:0/3337811569 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 91) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f49d401a3a0 con 0x7f49dc157860 2024-01-27T00:47:01.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.257+0000 7f49c3fff700 1 --2- 172.21.15.46:0/3337811569 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f49c8066dd0 0x7f49c8069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:01.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.257+0000 7f49c3fff700 1 -- 172.21.15.46:0/3337811569 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f49d4084760 con 0x7f49dc157860 2024-01-27T00:47:01.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.257+0000 7f49dbfff700 1 --2- 172.21.15.46:0/3337811569 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f49c8066dd0 0x7f49c8069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:01.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.258+0000 7f49dbfff700 1 --2- 172.21.15.46:0/3337811569 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f49c8066dd0 0x7f49c8069280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f49cc009db0 tx=0x7f49cc0091a0 comp rx=0 tx=0).ready entity=mgr.9208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:01.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.261+0000 7f49c3fff700 1 -- 172.21.15.46:0/3337811569 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f49d4051890 con 0x7f49dc157860 2024-01-27T00:47:01.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.493+0000 7f49e229a700 1 -- 172.21.15.46:0/3337811569 --> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f49dc0b14d0 con 0x7f49c8066dd0 2024-01-27T00:47:01.500 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:01.501+0000 7fb7633c4700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-01-27T00:47:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.500+0000 7f49c3fff700 1 -- 172.21.15.46:0/3337811569 <== mgr.9208 v2:172.21.15.106:6835/43388 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f49dc0b14d0 con 0x7f49c8066dd0 2024-01-27T00:47:01.502 INFO:teuthology.orchestra.run.smithi046.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-01-27T00:47:01.505 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-27T00:47:01.506 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:47:01.506 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-01-27T00:47:01.530 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-27T00:47:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.712+0000 7f0a5a52f700 1 Processor -- start 2024-01-27T00:47:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.712+0000 7f0a5a52f700 1 -- start start 2024-01-27T00:47:01.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.713+0000 7f0a5a52f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a540c8da0 0x7f0a540c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:01.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.713+0000 7f0a5a52f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0a540caf30 0x7f0a540cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:01.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.713+0000 7f0a5a52f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a540ccc00 con 0x7f0a54140af0 2024-01-27T00:47:01.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.713+0000 7f0a5a52f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a540cbbf0 con 0x7f0a540c8da0 2024-01-27T00:47:01.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.713+0000 7f0a5a52f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a540cc310 con 0x7f0a540caf30 2024-01-27T00:47:01.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.713+0000 7f0a53fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a540c8da0 0x7f0a540c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.713+0000 7f0a53fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a540c8da0 0x7f0a540c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38820/0 (socket says 172.21.15.46:38820) 2024-01-27T00:47:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.713+0000 7f0a53fff700 1 -- 172.21.15.46:0/3453608160 learned_addr learned my addr 172.21.15.46:0/3453608160 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.713+0000 7f0a58acc700 1 --2- 172.21.15.46:0/3453608160 >> v2:172.21.15.106:3300/0 conn(0x7f0a540caf30 0x7f0a540cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.714+0000 7f0a53fff700 1 -- 172.21.15.46:0/3453608160 >> v2:172.21.15.106:3300/0 conn(0x7f0a540caf30 msgr2=0x7f0a540cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.714+0000 7f0a53fff700 1 --2- 172.21.15.46:0/3453608160 >> v2:172.21.15.106:3300/0 conn(0x7f0a540caf30 0x7f0a540cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.714+0000 7f0a53fff700 1 -- 172.21.15.46:0/3453608160 >> v1:172.21.15.106:6789/0 conn(0x7f0a54140af0 legacy=0x7f0a54140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.714+0000 7f0a53fff700 1 -- 172.21.15.46:0/3453608160 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a540cc5b0 con 0x7f0a540c8da0 2024-01-27T00:47:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.714+0000 7f0a53fff700 1 --2- 172.21.15.46:0/3453608160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a540c8da0 0x7f0a540c9170 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f0a541416d0 tx=0x7f0a44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93feb0c41c528eb server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.714+0000 7f0a52ffd700 1 -- 172.21.15.46:0/3453608160 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a44009e70 con 0x7f0a540c8da0 2024-01-27T00:47:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.714+0000 7f0a52ffd700 1 -- 172.21.15.46:0/3453608160 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4400b860 con 0x7f0a540c8da0 2024-01-27T00:47:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.714+0000 7f0a52ffd700 1 -- 172.21.15.46:0/3453608160 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4400bc00 con 0x7f0a540c8da0 2024-01-27T00:47:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.714+0000 7f0a5a52f700 1 -- 172.21.15.46:0/3453608160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a540c8da0 msgr2=0x7f0a540c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.714+0000 7f0a5a52f700 1 --2- 172.21.15.46:0/3453608160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a540c8da0 0x7f0a540c9170 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f0a541416d0 tx=0x7f0a44009250 comp rx=0 tx=0).stop 2024-01-27T00:47:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.714+0000 7f0a5a52f700 1 -- 172.21.15.46:0/3453608160 shutdown_connections 2024-01-27T00:47:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.714+0000 7f0a5a52f700 1 --2- 172.21.15.46:0/3453608160 >> v2:172.21.15.106:3300/0 conn(0x7f0a540caf30 0x7f0a540cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.715+0000 7f0a5a52f700 1 --2- 172.21.15.46:0/3453608160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a540c8da0 0x7f0a540c9170 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.715+0000 7f0a5a52f700 1 -- 172.21.15.46:0/3453608160 >> 172.21.15.46:0/3453608160 conn(0x7f0a5406bd90 msgr2=0x7f0a5406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.715+0000 7f0a5a52f700 1 -- 172.21.15.46:0/3453608160 shutdown_connections 2024-01-27T00:47:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.715+0000 7f0a5a52f700 1 -- 172.21.15.46:0/3453608160 wait complete. 2024-01-27T00:47:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.716+0000 7f0a5a52f700 1 Processor -- start 2024-01-27T00:47:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.716+0000 7f0a5a52f700 1 -- start start 2024-01-27T00:47:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a5a52f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a540c8da0 0x7f0a541571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a5a52f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a540caf30 0x7f0a540c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a5a52f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a54140af0 0x7f0a540c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a5a52f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a54158280 con 0x7f0a540c8da0 2024-01-27T00:47:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a5a52f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a541576f0 con 0x7f0a540caf30 2024-01-27T00:47:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a5a52f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a540c7810 con 0x7f0a54140af0 2024-01-27T00:47:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a53fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a540c8da0 0x7f0a541571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a53fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a540c8da0 0x7f0a541571b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38828/0 (socket says 172.21.15.46:38828) 2024-01-27T00:47:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a53fff700 1 -- 172.21.15.46:0/296601933 learned_addr learned my addr 172.21.15.46:0/296601933 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a537fe700 1 --2- 172.21.15.46:0/296601933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a540caf30 0x7f0a540c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a58acc700 1 --2- 172.21.15.46:0/296601933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a54140af0 0x7f0a540c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a53fff700 1 -- 172.21.15.46:0/296601933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a54140af0 msgr2=0x7f0a540c7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a53fff700 1 --2- 172.21.15.46:0/296601933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a54140af0 0x7f0a540c7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a53fff700 1 -- 172.21.15.46:0/296601933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a540caf30 msgr2=0x7f0a540c5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a53fff700 1 --2- 172.21.15.46:0/296601933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a540caf30 0x7f0a540c5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a53fff700 1 -- 172.21.15.46:0/296601933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a440094f0 con 0x7f0a540c8da0 2024-01-27T00:47:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a537fe700 1 --2- 172.21.15.46:0/296601933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a540caf30 0x7f0a540c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:47:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.717+0000 7f0a53fff700 1 --2- 172.21.15.46:0/296601933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a540c8da0 0x7f0a541571b0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f0a5406f5a0 tx=0x7f0a44017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.718+0000 7f0a3bfff700 1 -- 172.21.15.46:0/296601933 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a44011420 con 0x7f0a540c8da0 2024-01-27T00:47:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.718+0000 7f0a3bfff700 1 -- 172.21.15.46:0/296601933 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a440115a0 con 0x7f0a540c8da0 2024-01-27T00:47:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.718+0000 7f0a5a52f700 1 -- 172.21.15.46:0/296601933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a540c6f60 con 0x7f0a540c8da0 2024-01-27T00:47:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.718+0000 7f0a3bfff700 1 -- 172.21.15.46:0/296601933 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a44011960 con 0x7f0a540c8da0 2024-01-27T00:47:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.718+0000 7f0a5a52f700 1 -- 172.21.15.46:0/296601933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a540c71c0 con 0x7f0a540c8da0 2024-01-27T00:47:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.719+0000 7f0a5a52f700 1 -- 172.21.15.46:0/296601933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a540cdde0 con 0x7f0a540c8da0 2024-01-27T00:47:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.719+0000 7f0a3bfff700 1 -- 172.21.15.46:0/296601933 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 91) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f0a44011d10 con 0x7f0a540c8da0 2024-01-27T00:47:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.720+0000 7f0a3bfff700 1 --2- 172.21.15.46:0/296601933 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0a40066dd0 0x7f0a40069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.720+0000 7f0a3bfff700 1 -- 172.21.15.46:0/296601933 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f0a44085c90 con 0x7f0a540c8da0 2024-01-27T00:47:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.720+0000 7f0a537fe700 1 --2- 172.21.15.46:0/296601933 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0a40066dd0 0x7f0a40069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.720+0000 7f0a537fe700 1 --2- 172.21.15.46:0/296601933 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0a40066dd0 0x7f0a40069280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0a48007fd0 tx=0x7f0a48009040 comp rx=0 tx=0).ready entity=mgr.9208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.724+0000 7f0a3bfff700 1 -- 172.21.15.46:0/296601933 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a44052dc0 con 0x7f0a540c8da0 2024-01-27T00:47:01.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:01.992+0000 7f0a5a52f700 1 -- 172.21.15.46:0/296601933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f0a5404c710 con 0x7f0a540c8da0 2024-01-27T00:47:02.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:02.585+0000 7f0a3bfff700 1 -- 172.21.15.46:0/296601933 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 92) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f0a44053190 con 0x7f0a540c8da0 2024-01-27T00:47:02.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:02.587+0000 7f0a3bfff700 1 -- 172.21.15.46:0/296601933 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v92) v1 ==== 93+0+0 (secure 0 0 0) 0x7f0a44025ed0 con 0x7f0a540c8da0 2024-01-27T00:47:02.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:02.590+0000 7f0a5a52f700 1 -- 172.21.15.46:0/296601933 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0a40066dd0 msgr2=0x7f0a40069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:02.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:02.590+0000 7f0a5a52f700 1 --2- 172.21.15.46:0/296601933 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0a40066dd0 0x7f0a40069280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0a48007fd0 tx=0x7f0a48009040 comp rx=0 tx=0).stop 2024-01-27T00:47:02.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:02.590+0000 7f0a5a52f700 1 -- 172.21.15.46:0/296601933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a540c8da0 msgr2=0x7f0a541571b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:02.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:02.590+0000 7f0a5a52f700 1 --2- 172.21.15.46:0/296601933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a540c8da0 0x7f0a541571b0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f0a5406f5a0 tx=0x7f0a44017000 comp rx=0 tx=0).stop 2024-01-27T00:47:02.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:02.590+0000 7f0a5a52f700 1 -- 172.21.15.46:0/296601933 shutdown_connections 2024-01-27T00:47:02.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:02.590+0000 7f0a5a52f700 1 --2- 172.21.15.46:0/296601933 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0a40066dd0 0x7f0a40069280 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:02.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:02.590+0000 7f0a5a52f700 1 --2- 172.21.15.46:0/296601933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a540c8da0 0x7f0a541571b0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:02.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:02.590+0000 7f0a5a52f700 1 --2- 172.21.15.46:0/296601933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a54140af0 0x7f0a540c7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:02.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:02.590+0000 7f0a5a52f700 1 --2- 172.21.15.46:0/296601933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a540caf30 0x7f0a540c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:02.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:02.590+0000 7f0a5a52f700 1 -- 172.21.15.46:0/296601933 >> 172.21.15.46:0/296601933 conn(0x7f0a5406bd90 msgr2=0x7f0a540c42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:02.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:02.590+0000 7f0a5a52f700 1 -- 172.21.15.46:0/296601933 shutdown_connections 2024-01-27T00:47:02.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:02.590+0000 7f0a5a52f700 1 -- 172.21.15.46:0/296601933 wait complete. 2024-01-27T00:47:02.606 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-27T00:47:02.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T00:47:02.653 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T00:47:02.700 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-27T00:47:02.700 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-27T00:47:02.701 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-27T00:47:02.701 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-27T00:47:02.745 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[98304]: fuse finished with error 0 and tester_r 0 2024-01-27T00:47:02.747 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T00:47:02.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:47:02.747 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-27T00:47:02.767 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T00:47:02.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:47:02.768 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-27T00:47:02.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T00:47:02.788 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-27T00:47:02.788 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T00:47:02.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:47:02.788 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-27T00:47:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T00:47:02.826 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-27T00:47:02.826 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T00:47:02.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:47:02.826 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-27T00:47:02.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T00:47:02.846 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-27T00:47:02.846 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T00:47:02.936 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:02.936+0000 7f573af0c1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T00:47:02.971 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:02.971+0000 7fd1097bb1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T00:47:02.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:02.999+0000 7eff489e6700 1 Processor -- start 2024-01-27T00:47:02.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:02.999+0000 7eff489e6700 1 -- start start 2024-01-27T00:47:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.000+0000 7eff489e6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff440c6520 0x7eff440ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.000+0000 7eff489e6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff440ced50 0x7eff440cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.000+0000 7eff489e6700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff4415a0b0 con 0x7eff440b2b40 2024-01-27T00:47:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.000+0000 7eff489e6700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff440af070 con 0x7eff440ced50 2024-01-27T00:47:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.000+0000 7eff489e6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff44099fc0 con 0x7eff440c6520 2024-01-27T00:47:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.000+0000 7eff42d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff440ced50 0x7eff440cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.000+0000 7eff4259c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7eff440b2b40 0x7eff440ae960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37166/0 (socket says 172.21.15.46:37166) 2024-01-27T00:47:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.000+0000 7eff4259c700 1 -- 172.21.15.46:0/853231650 learned_addr learned my addr 172.21.15.46:0/853231650 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.000+0000 7eff41d9b700 1 --2- 172.21.15.46:0/853231650 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff440c6520 0x7eff440ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.001+0000 7eff4159a700 1 -- 172.21.15.46:0/853231650 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653890376 0 0) 0x7eff4415a0b0 con 0x7eff440b2b40 2024-01-27T00:47:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.001+0000 7eff4159a700 1 -- 172.21.15.46:0/853231650 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff2c003570 con 0x7eff440b2b40 2024-01-27T00:47:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.001+0000 7eff42d9d700 1 -- 172.21.15.46:0/853231650 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff440c6520 msgr2=0x7eff440ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.001+0000 7eff42d9d700 1 --2- 172.21.15.46:0/853231650 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff440c6520 0x7eff440ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.001+0000 7eff42d9d700 1 -- 172.21.15.46:0/853231650 >> v1:172.21.15.46:6789/0 conn(0x7eff440b2b40 legacy=0x7eff440ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.001+0000 7eff42d9d700 1 -- 172.21.15.46:0/853231650 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff440ae4d0 con 0x7eff440ced50 2024-01-27T00:47:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.001+0000 7eff41d9b700 1 --2- 172.21.15.46:0/853231650 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff440c6520 0x7eff440ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.001+0000 7eff42d9d700 1 --2- 172.21.15.46:0/853231650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440ced50 0x7eff440cf1d0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0098e0 tx=0x7eff3c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b338e93aedcf6115 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.001+0000 7eff4159a700 1 -- 172.21.15.46:0/853231650 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3c011630 con 0x7eff440ced50 2024-01-27T00:47:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.001+0000 7eff4159a700 1 -- 172.21.15.46:0/853231650 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3c011790 con 0x7eff440ced50 2024-01-27T00:47:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.001+0000 7eff4159a700 1 -- 172.21.15.46:0/853231650 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3c011b30 con 0x7eff440ced50 2024-01-27T00:47:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.002+0000 7eff489e6700 1 -- 172.21.15.46:0/853231650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440ced50 msgr2=0x7eff440cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.002+0000 7eff489e6700 1 --2- 172.21.15.46:0/853231650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440ced50 0x7eff440cf1d0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0098e0 tx=0x7eff3c0091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.002+0000 7eff489e6700 1 -- 172.21.15.46:0/853231650 shutdown_connections 2024-01-27T00:47:03.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.002+0000 7eff489e6700 1 --2- 172.21.15.46:0/853231650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440ced50 0x7eff440cf1d0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.002+0000 7eff489e6700 1 --2- 172.21.15.46:0/853231650 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff440c6520 0x7eff440ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.002+0000 7eff489e6700 1 -- 172.21.15.46:0/853231650 >> 172.21.15.46:0/853231650 conn(0x7eff44099a30 msgr2=0x7eff4406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:03.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.002+0000 7eff489e6700 1 -- 172.21.15.46:0/853231650 shutdown_connections 2024-01-27T00:47:03.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.002+0000 7eff489e6700 1 -- 172.21.15.46:0/853231650 wait complete. 2024-01-27T00:47:03.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.003+0000 7eff489e6700 1 Processor -- start 2024-01-27T00:47:03.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.003+0000 7eff489e6700 1 -- start start 2024-01-27T00:47:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.004+0000 7eff489e6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440b2b40 0x7eff440c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.004+0000 7eff489e6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff440c6520 0x7eff44169f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.004+0000 7eff489e6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff440ced50 0x7eff441565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.004+0000 7eff489e6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff440c8b50 con 0x7eff440b2b40 2024-01-27T00:47:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.004+0000 7eff489e6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff44169af0 con 0x7eff440c6520 2024-01-27T00:47:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.004+0000 7eff489e6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff440c82f0 con 0x7eff440ced50 2024-01-27T00:47:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.004+0000 7eff4259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440b2b40 0x7eff440c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.004+0000 7eff4259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440b2b40 0x7eff440c7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38864/0 (socket says 172.21.15.46:38864) 2024-01-27T00:47:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.004+0000 7eff4259c700 1 -- 172.21.15.46:0/1011829848 learned_addr learned my addr 172.21.15.46:0/1011829848 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.004+0000 7eff42d9d700 1 --2- 172.21.15.46:0/1011829848 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff440ced50 0x7eff441565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.004+0000 7eff41d9b700 1 --2- 172.21.15.46:0/1011829848 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff440c6520 0x7eff44169f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.004+0000 7eff4259c700 1 -- 172.21.15.46:0/1011829848 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff440ced50 msgr2=0x7eff441565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.004+0000 7eff4259c700 1 --2- 172.21.15.46:0/1011829848 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff440ced50 0x7eff441565d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.004+0000 7eff4259c700 1 -- 172.21.15.46:0/1011829848 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff440c6520 msgr2=0x7eff44169f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.004+0000 7eff4259c700 1 --2- 172.21.15.46:0/1011829848 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff440c6520 0x7eff44169f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.005+0000 7eff4259c700 1 -- 172.21.15.46:0/1011829848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff38009420 con 0x7eff440b2b40 2024-01-27T00:47:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.005+0000 7eff4259c700 1 --2- 172.21.15.46:0/1011829848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440b2b40 0x7eff440c7db0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7eff3400cb10 tx=0x7eff340027e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.005+0000 7eff327fc700 1 -- 172.21.15.46:0/1011829848 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3400ce20 con 0x7eff440b2b40 2024-01-27T00:47:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.005+0000 7eff327fc700 1 -- 172.21.15.46:0/1011829848 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff34005d80 con 0x7eff440b2b40 2024-01-27T00:47:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.005+0000 7eff489e6700 1 -- 172.21.15.46:0/1011829848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3c009420 con 0x7eff440b2b40 2024-01-27T00:47:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.005+0000 7eff327fc700 1 -- 172.21.15.46:0/1011829848 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3400a7e0 con 0x7eff440b2b40 2024-01-27T00:47:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.005+0000 7eff489e6700 1 -- 172.21.15.46:0/1011829848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff440c8870 con 0x7eff440b2b40 2024-01-27T00:47:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.006+0000 7eff489e6700 1 -- 172.21.15.46:0/1011829848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff4404c710 con 0x7eff440b2b40 2024-01-27T00:47:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.006+0000 7eff327fc700 1 -- 172.21.15.46:0/1011829848 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 92) v1 ==== 84845+0+0 (secure 0 0 0) 0x7eff3400ac10 con 0x7eff440b2b40 2024-01-27T00:47:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.007+0000 7eff327fc700 1 --2- 172.21.15.46:0/1011829848 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff24067050 0x7eff24069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.007+0000 7eff41d9b700 1 -- 172.21.15.46:0/1011829848 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff24067050 msgr2=0x7eff24069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.007+0000 7eff327fc700 1 -- 172.21.15.46:0/1011829848 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6719+0+0 (secure 0 0 0) 0x7eff340847a0 con 0x7eff440b2b40 2024-01-27T00:47:03.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.007+0000 7eff41d9b700 1 --2- 172.21.15.46:0/1011829848 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff24067050 0x7eff24069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:47:03.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.011+0000 7eff327fc700 1 -- 172.21.15.46:0/1011829848 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7eff34051590 con 0x7eff440b2b40 2024-01-27T00:47:03.013 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:03.013+0000 7f573af0c1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T00:47:03.062 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:03.062+0000 7fd1097bb1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T00:47:03.084 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:03.084+0000 7f573af0c1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T00:47:03.143 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:03.143+0000 7fd1097bb1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T00:47:03.148 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:03.148+0000 7f573af0c1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T00:47:03.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.208+0000 7eff41d9b700 1 -- 172.21.15.46:0/1011829848 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff24067050 msgr2=0x7eff24069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:03.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.208+0000 7eff41d9b700 1 --2- 172.21.15.46:0/1011829848 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff24067050 0x7eff24069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:47:03.217 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:03.217+0000 7fd1097bb1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T00:47:03.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.265+0000 7eff489e6700 1 -- 172.21.15.46:0/1011829848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff440682c0 con 0x7eff440b2b40 2024-01-27T00:47:03.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.265+0000 7eff327fc700 1 -- 172.21.15.46:0/1011829848 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4607 (secure 0 0 0) 0x7eff34054bb0 con 0x7eff440b2b40 2024-01-27T00:47:03.266 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:03.266 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":88,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9061,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/3698053981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":3698053981},{"type":"v1","addr":"172.21.15.106:6833","nonce":3698053981}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":9064,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2056976879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2056976879},{"type":"v1","addr":"172.21.15.46:6834","nonce":2056976879}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":82},{"gid":9067,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1576406314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1576406314},{"type":"v1","addr":"172.21.15.46:6835","nonce":1576406314}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T00:46:44.978654+0000","modified":"2024-01-27T00:46:48.558026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9070":{"gid":9070,"name":"d","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6836/2857098625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2857098625},{"type":"v1","addr":"172.21.15.106:6836","nonce":2857098625}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2024-01-27T00:47:03.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.268+0000 7eff489e6700 1 -- 172.21.15.46:0/1011829848 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff24067050 msgr2=0x7eff24069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:47:03.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.268+0000 7eff489e6700 1 --2- 172.21.15.46:0/1011829848 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff24067050 0x7eff24069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.268+0000 7eff489e6700 1 -- 172.21.15.46:0/1011829848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440b2b40 msgr2=0x7eff440c7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:03.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.268+0000 7eff489e6700 1 --2- 172.21.15.46:0/1011829848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440b2b40 0x7eff440c7db0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7eff3400cb10 tx=0x7eff340027e0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.268+0000 7eff489e6700 1 -- 172.21.15.46:0/1011829848 shutdown_connections 2024-01-27T00:47:03.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.268+0000 7eff489e6700 1 --2- 172.21.15.46:0/1011829848 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7eff24067050 0x7eff24069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.268+0000 7eff489e6700 1 --2- 172.21.15.46:0/1011829848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440b2b40 0x7eff440c7db0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.268+0000 7eff489e6700 1 --2- 172.21.15.46:0/1011829848 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff440ced50 0x7eff441565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.268+0000 7eff489e6700 1 --2- 172.21.15.46:0/1011829848 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff440c6520 0x7eff44169f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.268+0000 7eff489e6700 1 -- 172.21.15.46:0/1011829848 >> 172.21.15.46:0/1011829848 conn(0x7eff44099a30 msgr2=0x7eff441590b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:03.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.268+0000 7eff489e6700 1 -- 172.21.15.46:0/1011829848 shutdown_connections 2024-01-27T00:47:03.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.268+0000 7eff489e6700 1 -- 172.21.15.46:0/1011829848 wait complete. 2024-01-27T00:47:03.270 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 88 2024-01-27T00:47:03.283 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T00:47:03.302 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:03.302+0000 7f573af0c1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:47:03.403 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:03.402+0000 7fd1097bb1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:47:03.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.419+0000 7f4f41b61700 1 Processor -- start 2024-01-27T00:47:03.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.419+0000 7f4f41b61700 1 -- start start 2024-01-27T00:47:03.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.420+0000 7f4f41b61700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f3c15ce40 0x7f4f3c0d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:03.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.420+0000 7f4f41b61700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4f3c0cc1c0 0x7f4f3c0ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:03.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.420+0000 7f4f41b61700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3c0cff70 con 0x7f4f3c15d210 2024-01-27T00:47:03.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.420+0000 7f4f41b61700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f3c0cf400 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.420+0000 7f4f41b61700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c0cf660 con 0x7f4f3c15ce40 2024-01-27T00:47:03.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.420+0000 7f4f3b7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f3c15ce40 0x7f4f3c0d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:03.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.420+0000 7f4f3bfff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4f3c0cc1c0 0x7f4f3c0ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:03.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.420+0000 7f4f3affd700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f4f3c15d210 0x7f4f3c0cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:38124/0 (socket says 172.21.15.46:38124) 2024-01-27T00:47:03.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.420+0000 7f4f3b7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f3c15ce40 0x7f4f3c0d0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:33930/0 (socket says 172.21.15.46:33930) 2024-01-27T00:47:03.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.420+0000 7f4f3affd700 1 -- 172.21.15.46:0/1264288118 learned_addr learned my addr 172.21.15.46:0/1264288118 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:03.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.420+0000 7f4f3bfff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4f3c0cc1c0 0x7f4f3c0ade30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:33062/0 (socket says 172.21.15.46:33062) 2024-01-27T00:47:03.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.421+0000 7f4f3bfff700 1 -- 172.21.15.46:0/1264288118 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f3c15ce40 msgr2=0x7f4f3c0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:03.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.421+0000 7f4f3bfff700 1 --2- 172.21.15.46:0/1264288118 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f3c15ce40 0x7f4f3c0d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.421+0000 7f4f3bfff700 1 -- 172.21.15.46:0/1264288118 >> v1:172.21.15.106:6789/0 conn(0x7f4f3c15d210 legacy=0x7f4f3c0cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:03.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.421+0000 7f4f3bfff700 1 -- 172.21.15.46:0/1264288118 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c0d0500 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.421+0000 7f4f3a7fc700 1 -- 172.21.15.46:0/1264288118 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220841964 0 0) 0x7f4f3c0cff70 con 0x7f4f3c15d210 2024-01-27T00:47:03.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.421+0000 7f4f3b7fe700 1 --2- 172.21.15.46:0/1264288118 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f3c15ce40 0x7f4f3c0d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:03.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.421+0000 7f4f3bfff700 1 --2- 172.21.15.46:0/1264288118 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f3c0cc1c0 0x7f4f3c0ade30 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4f340098e0 tx=0x7f4f340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4bff759db140925 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:03.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.421+0000 7f4f3a7fc700 1 -- 172.21.15.46:0/1264288118 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3400b640 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.421+0000 7f4f3a7fc700 1 -- 172.21.15.46:0/1264288118 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3400b7a0 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.421+0000 7f4f3a7fc700 1 -- 172.21.15.46:0/1264288118 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3400bb40 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.422+0000 7f4f41b61700 1 -- 172.21.15.46:0/1264288118 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f3c0cc1c0 msgr2=0x7f4f3c0ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:03.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.422+0000 7f4f41b61700 1 --2- 172.21.15.46:0/1264288118 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f3c0cc1c0 0x7f4f3c0ade30 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4f340098e0 tx=0x7f4f340091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.422+0000 7f4f41b61700 1 -- 172.21.15.46:0/1264288118 shutdown_connections 2024-01-27T00:47:03.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.422+0000 7f4f41b61700 1 --2- 172.21.15.46:0/1264288118 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f3c0cc1c0 0x7f4f3c0ade30 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.422+0000 7f4f41b61700 1 --2- 172.21.15.46:0/1264288118 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f3c15ce40 0x7f4f3c0d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.422+0000 7f4f41b61700 1 -- 172.21.15.46:0/1264288118 >> 172.21.15.46:0/1264288118 conn(0x7f4f3c06f840 msgr2=0x7f4f3c0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:03.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.422+0000 7f4f41b61700 1 -- 172.21.15.46:0/1264288118 shutdown_connections 2024-01-27T00:47:03.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.422+0000 7f4f41b61700 1 -- 172.21.15.46:0/1264288118 wait complete. 2024-01-27T00:47:03.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.424+0000 7f4f41b61700 1 Processor -- start 2024-01-27T00:47:03.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.424+0000 7f4f41b61700 1 -- start start 2024-01-27T00:47:03.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.425+0000 7f4f41b61700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f3c0cc1c0 0x7f4f3c15e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:03.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.425+0000 7f4f41b61700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f3c15ce40 0x7f4f3c0c5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:03.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.425+0000 7f4f41b61700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f3c15d210 0x7f4f3c0c6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:03.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.425+0000 7f4f41b61700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c15fba0 con 0x7f4f3c15d210 2024-01-27T00:47:03.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.425+0000 7f4f41b61700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c15ef90 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.425+0000 7f4f41b61700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c15f270 con 0x7f4f3c15ce40 2024-01-27T00:47:03.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.425+0000 7f4f3b7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f3c0cc1c0 0x7f4f3c15e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:03.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.425+0000 7f4f3affd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f3c15ce40 0x7f4f3c0c5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:03.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.425+0000 7f4f3bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f3c15d210 0x7f4f3c0c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:03.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.425+0000 7f4f3b7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f3c0cc1c0 0x7f4f3c15e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:33066/0 (socket says 172.21.15.46:33066) 2024-01-27T00:47:03.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.425+0000 7f4f3affd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f3c15ce40 0x7f4f3c0c5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:33944/0 (socket says 172.21.15.46:33944) 2024-01-27T00:47:03.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.425+0000 7f4f3b7fe700 1 -- 172.21.15.46:0/990310563 learned_addr learned my addr 172.21.15.46:0/990310563 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:03.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.425+0000 7f4f3b7fe700 1 -- 172.21.15.46:0/990310563 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f3c15ce40 msgr2=0x7f4f3c0c5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:03.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.425+0000 7f4f3b7fe700 1 --2- 172.21.15.46:0/990310563 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f3c15ce40 0x7f4f3c0c5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.425+0000 7f4f3b7fe700 1 -- 172.21.15.46:0/990310563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f3c15d210 msgr2=0x7f4f3c0c6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:03.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.425+0000 7f4f3b7fe700 1 --2- 172.21.15.46:0/990310563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f3c15d210 0x7f4f3c0c6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.425+0000 7f4f3b7fe700 1 -- 172.21.15.46:0/990310563 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c009a20 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.426+0000 7f4f3b7fe700 1 --2- 172.21.15.46:0/990310563 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f3c0cc1c0 0x7f4f3c15e620 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00c1f0 tx=0x7f4f2c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:03.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.426+0000 7f4f237fe700 1 -- 172.21.15.46:0/990310563 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c00ccd0 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.426+0000 7f4f237fe700 1 -- 172.21.15.46:0/990310563 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2c00f040 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.426+0000 7f4f237fe700 1 -- 172.21.15.46:0/990310563 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c014480 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.426+0000 7f4f41b61700 1 -- 172.21.15.46:0/990310563 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f34009420 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.426+0000 7f4f41b61700 1 -- 172.21.15.46:0/990310563 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3c15f830 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.427+0000 7f4f237fe700 1 -- 172.21.15.46:0/990310563 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 92) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f4f2c0147f0 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.427+0000 7f4f41b61700 1 -- 172.21.15.46:0/990310563 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f3c04c710 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.428+0000 7f4f237fe700 1 --2- 172.21.15.46:0/990310563 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f28067050 0x7f4f28069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:03.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.428+0000 7f4f3affd700 1 -- 172.21.15.46:0/990310563 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f28067050 msgr2=0x7f4f28069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:03.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.428+0000 7f4f237fe700 1 -- 172.21.15.46:0/990310563 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f4f2c085ad0 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.428+0000 7f4f3affd700 1 --2- 172.21.15.46:0/990310563 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f28067050 0x7f4f28069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:47:03.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.432+0000 7f4f237fe700 1 -- 172.21.15.46:0/990310563 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4f2c052690 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.440 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:03.440+0000 7f573af0c1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T00:47:03.506 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:03.506+0000 7f573af0c1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T00:47:03.562 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:03.562+0000 7fd1097bb1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T00:47:03.628 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:03.629+0000 7f573af0c1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T00:47:03.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.629+0000 7f4f3affd700 1 -- 172.21.15.46:0/990310563 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f28067050 msgr2=0x7f4f28069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:03.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.629+0000 7f4f3affd700 1 --2- 172.21.15.46:0/990310563 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f28067050 0x7f4f28069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:47:03.640 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:03.640+0000 7fd1097bb1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T00:47:03.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.643+0000 7f4f41b61700 1 -- 172.21.15.46:0/990310563 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f3c0cf920 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.644+0000 7f4f237fe700 1 -- 172.21.15.46:0/990310563 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v176) v1 ==== 74+0+16607 (secure 0 0 0) 0x7f4f2c055cb0 con 0x7f4f3c0cc1c0 2024-01-27T00:47:03.645 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:03.645 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":176,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T00:46:59.569524+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-01-27T00:46:43.494684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-01-27T00:46:43.949785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/2302701803":"2024-01-28T00:46:59.569484+0000","172.21.15.106:0/3368647852":"2024-01-28T00:46:59.569484+0000","172.21.15.106:6837/43388":"2024-01-28T00:46:59.569484+0000","172.21.15.106:6835/43388":"2024-01-28T00:46:59.569484+0000","172.21.15.106:0/3900909328":"2024-01-28T00:46:59.569484+0000","172.21.15.106:0/4251291402":"2024-01-28T00:46:59.569484+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T00:47:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.647+0000 7f4f41b61700 1 -- 172.21.15.46:0/990310563 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f28067050 msgr2=0x7f4f28069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:47:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.647+0000 7f4f41b61700 1 --2- 172.21.15.46:0/990310563 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f28067050 0x7f4f28069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.647+0000 7f4f41b61700 1 -- 172.21.15.46:0/990310563 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f3c0cc1c0 msgr2=0x7f4f3c15e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.647+0000 7f4f41b61700 1 --2- 172.21.15.46:0/990310563 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f3c0cc1c0 0x7f4f3c15e620 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00c1f0 tx=0x7f4f2c00b040 comp rx=0 tx=0).stop 2024-01-27T00:47:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.647+0000 7f4f41b61700 1 -- 172.21.15.46:0/990310563 shutdown_connections 2024-01-27T00:47:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.647+0000 7f4f41b61700 1 --2- 172.21.15.46:0/990310563 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f28067050 0x7f4f28069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.647+0000 7f4f41b61700 1 --2- 172.21.15.46:0/990310563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f3c15d210 0x7f4f3c0c6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.647+0000 7f4f41b61700 1 --2- 172.21.15.46:0/990310563 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f3c0cc1c0 0x7f4f3c15e620 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.647+0000 7f4f41b61700 1 --2- 172.21.15.46:0/990310563 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f3c15ce40 0x7f4f3c0c5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.647+0000 7f4f41b61700 1 -- 172.21.15.46:0/990310563 >> 172.21.15.46:0/990310563 conn(0x7f4f3c06f840 msgr2=0x7f4f3c0b5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.647+0000 7f4f41b61700 1 -- 172.21.15.46:0/990310563 shutdown_connections 2024-01-27T00:47:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.647+0000 7f4f41b61700 1 -- 172.21.15.46:0/990310563 wait complete. 2024-01-27T00:47:03.661 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-27T00:46:43.494684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2024-01-27T00:46:43.949785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '175', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-27T00:47:03.662 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-27T00:47:03.702 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:03.702+0000 7f573af0c1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T00:47:03.784 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:03.783+0000 7fd1097bb1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T00:47:03.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.799+0000 7f6716209700 1 Processor -- start 2024-01-27T00:47:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.799+0000 7f6716209700 1 -- start start 2024-01-27T00:47:03.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.801+0000 7f6716209700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f67100cf050 0x7f67100af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:03.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.801+0000 7f6716209700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67100cc100 con 0x7f6710159cb0 2024-01-27T00:47:03.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.801+0000 7f6716209700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67100cb850 con 0x7f67100cf4a0 2024-01-27T00:47:03.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.801+0000 7f6716209700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67100ccca0 con 0x7f67100cf050 2024-01-27T00:47:03.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.801+0000 7f670f7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6710159cb0 0x7f67100cc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37170/0 (socket says 172.21.15.46:37170) 2024-01-27T00:47:03.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.801+0000 7f670f7fe700 1 -- 172.21.15.46:0/397079058 learned_addr learned my addr 172.21.15.46:0/397079058 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:03.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.801+0000 7f670effd700 1 --2- 172.21.15.46:0/397079058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f67100cf050 0x7f67100af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:03.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.801+0000 7f670e7fc700 1 -- 172.21.15.46:0/397079058 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677587349 0 0) 0x7f67100cc100 con 0x7f6710159cb0 2024-01-27T00:47:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.801+0000 7f670e7fc700 1 -- 172.21.15.46:0/397079058 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66f8003570 con 0x7f6710159cb0 2024-01-27T00:47:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.801+0000 7f670e7fc700 1 -- 172.21.15.46:0/397079058 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531113448 0 0) 0x7f67100cb850 con 0x7f67100cf4a0 2024-01-27T00:47:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.801+0000 7f670e7fc700 1 -- 172.21.15.46:0/397079058 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67100cc100 con 0x7f67100cf4a0 2024-01-27T00:47:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.801+0000 7f670effd700 1 -- 172.21.15.46:0/397079058 >> v1:172.21.15.106:6789/0 conn(0x7f67100cf4a0 legacy=0x7f67100c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.801+0000 7f670effd700 1 -- 172.21.15.46:0/397079058 >> v1:172.21.15.46:6789/0 conn(0x7f6710159cb0 legacy=0x7f67100cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.801+0000 7f670effd700 1 -- 172.21.15.46:0/397079058 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67100cbaf0 con 0x7f67100cf050 2024-01-27T00:47:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.802+0000 7f670effd700 1 --2- 172.21.15.46:0/397079058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f67100cf050 0x7f67100af430 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6704009910 tx=0x7f67040091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb939affc0fe9b99 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.802+0000 7f670e7fc700 1 -- 172.21.15.46:0/397079058 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704011630 con 0x7f67100cf050 2024-01-27T00:47:03.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.802+0000 7f670e7fc700 1 -- 172.21.15.46:0/397079058 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6704011790 con 0x7f67100cf050 2024-01-27T00:47:03.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.802+0000 7f670e7fc700 1 -- 172.21.15.46:0/397079058 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704011b30 con 0x7f67100cf050 2024-01-27T00:47:03.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.803+0000 7f6716209700 1 -- 172.21.15.46:0/397079058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f67100cf050 msgr2=0x7f67100af430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:03.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.803+0000 7f6716209700 1 --2- 172.21.15.46:0/397079058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f67100cf050 0x7f67100af430 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6704009910 tx=0x7f67040091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.803+0000 7f6716209700 1 -- 172.21.15.46:0/397079058 shutdown_connections 2024-01-27T00:47:03.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.803+0000 7f6716209700 1 --2- 172.21.15.46:0/397079058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f67100cf050 0x7f67100af430 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.803+0000 7f6716209700 1 -- 172.21.15.46:0/397079058 >> 172.21.15.46:0/397079058 conn(0x7f6710087f60 msgr2=0x7f671006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:03.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.803+0000 7f6716209700 1 -- 172.21.15.46:0/397079058 shutdown_connections 2024-01-27T00:47:03.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.803+0000 7f6716209700 1 -- 172.21.15.46:0/397079058 wait complete. 2024-01-27T00:47:03.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.805+0000 7f6716209700 1 Processor -- start 2024-01-27T00:47:03.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.805+0000 7f6716209700 1 -- start start 2024-01-27T00:47:03.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.806+0000 7f6716209700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67100cf050 0x7f671009c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:03.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.806+0000 7f6716209700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67100cf4a0 0x7f6710068fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:03.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.806+0000 7f6716209700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6710159cb0 0x7f6710069520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:03.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.806+0000 7f6716209700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6710157820 con 0x7f67100cf050 2024-01-27T00:47:03.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.806+0000 7f6716209700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671016aa50 con 0x7f67100cf4a0 2024-01-27T00:47:03.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.806+0000 7f6716209700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6710156ef0 con 0x7f6710159cb0 2024-01-27T00:47:03.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.807+0000 7f670f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67100cf050 0x7f671009c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:03.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.807+0000 7f670f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67100cf050 0x7f671009c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38870/0 (socket says 172.21.15.46:38870) 2024-01-27T00:47:03.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.807+0000 7f670f7fe700 1 -- 172.21.15.46:0/50161690 learned_addr learned my addr 172.21.15.46:0/50161690 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:03.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.807+0000 7f670effd700 1 --2- 172.21.15.46:0/50161690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67100cf4a0 0x7f6710068fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:03.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.807+0000 7f670f7fe700 1 -- 172.21.15.46:0/50161690 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6710159cb0 msgr2=0x7f6710069520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:03.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.807+0000 7f670f7fe700 1 --2- 172.21.15.46:0/50161690 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6710159cb0 0x7f6710069520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.807+0000 7f670f7fe700 1 -- 172.21.15.46:0/50161690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67100cf4a0 msgr2=0x7f6710068fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:03.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.807+0000 7f670f7fe700 1 --2- 172.21.15.46:0/50161690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67100cf4a0 0x7f6710068fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:03.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.807+0000 7f670f7fe700 1 -- 172.21.15.46:0/50161690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6704009420 con 0x7f67100cf050 2024-01-27T00:47:03.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.807+0000 7f670f7fe700 1 --2- 172.21.15.46:0/50161690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67100cf050 0x7f671009c590 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f670000c790 tx=0x7f6700007ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:03.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.807+0000 7f66f77fe700 1 -- 172.21.15.46:0/50161690 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67000072e0 con 0x7f67100cf050 2024-01-27T00:47:03.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.807+0000 7f66f77fe700 1 -- 172.21.15.46:0/50161690 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6700007440 con 0x7f67100cf050 2024-01-27T00:47:03.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.807+0000 7f6716209700 1 -- 172.21.15.46:0/50161690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67101571d0 con 0x7f67100cf050 2024-01-27T00:47:03.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.807+0000 7f66f77fe700 1 -- 172.21.15.46:0/50161690 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67000057f0 con 0x7f67100cf050 2024-01-27T00:47:03.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.807+0000 7f6716209700 1 -- 172.21.15.46:0/50161690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6710154f20 con 0x7f67100cf050 2024-01-27T00:47:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.809+0000 7f6716209700 1 -- 172.21.15.46:0/50161690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671004c710 con 0x7f67100cf050 2024-01-27T00:47:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.809+0000 7f66f77fe700 1 -- 172.21.15.46:0/50161690 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 92) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f6700005be0 con 0x7f67100cf050 2024-01-27T00:47:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.810+0000 7f66f77fe700 1 --2- 172.21.15.46:0/50161690 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f66fc067050 0x7f66fc069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.810+0000 7f670effd700 1 -- 172.21.15.46:0/50161690 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f66fc067050 msgr2=0x7f66fc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.810+0000 7f66f77fe700 1 -- 172.21.15.46:0/50161690 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f6700083f70 con 0x7f67100cf050 2024-01-27T00:47:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.810+0000 7f670effd700 1 --2- 172.21.15.46:0/50161690 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f66fc067050 0x7f66fc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:47:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:03.814+0000 7f66f77fe700 1 -- 172.21.15.46:0/50161690 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f670004b5a0 con 0x7f67100cf050 2024-01-27T00:47:03.840 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:03.840+0000 7f573af0c1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T00:47:03.872 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:03.872+0000 7fd1097bb1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T00:47:03.967 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:03.967+0000 7f573af0c1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T00:47:04.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.011+0000 7f670effd700 1 -- 172.21.15.46:0/50161690 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f66fc067050 msgr2=0x7f66fc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:04.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.011+0000 7f670effd700 1 --2- 172.21.15.46:0/50161690 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f66fc067050 0x7f66fc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:47:04.043 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:04.043+0000 7fd1097bb1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T00:47:04.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.066+0000 7f6716209700 1 -- 172.21.15.46:0/50161690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f67100cfc80 con 0x7f67100cf050 2024-01-27T00:47:04.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.066+0000 7f66f77fe700 1 -- 172.21.15.46:0/50161690 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v176) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6700017070 con 0x7f67100cf050 2024-01-27T00:47:04.066 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-27T00:47:04.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.068+0000 7f6716209700 1 -- 172.21.15.46:0/50161690 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f66fc067050 msgr2=0x7f66fc069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:47:04.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.068+0000 7f6716209700 1 --2- 172.21.15.46:0/50161690 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f66fc067050 0x7f66fc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.068+0000 7f6716209700 1 -- 172.21.15.46:0/50161690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67100cf050 msgr2=0x7f671009c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:04.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.068+0000 7f6716209700 1 --2- 172.21.15.46:0/50161690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67100cf050 0x7f671009c590 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f670000c790 tx=0x7f6700007ce0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.069+0000 7f6716209700 1 -- 172.21.15.46:0/50161690 shutdown_connections 2024-01-27T00:47:04.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.069+0000 7f6716209700 1 --2- 172.21.15.46:0/50161690 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f66fc067050 0x7f66fc069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.069+0000 7f6716209700 1 --2- 172.21.15.46:0/50161690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67100cf050 0x7f671009c590 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.069+0000 7f6716209700 1 --2- 172.21.15.46:0/50161690 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6710159cb0 0x7f6710069520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.069+0000 7f6716209700 1 --2- 172.21.15.46:0/50161690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67100cf4a0 0x7f6710068fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.069+0000 7f6716209700 1 -- 172.21.15.46:0/50161690 >> 172.21.15.46:0/50161690 conn(0x7f6710087f60 msgr2=0x7f67100b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:04.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.069+0000 7f6716209700 1 -- 172.21.15.46:0/50161690 shutdown_connections 2024-01-27T00:47:04.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.069+0000 7f6716209700 1 -- 172.21.15.46:0/50161690 wait complete. 2024-01-27T00:47:04.085 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-27T00:47:04.188 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:04.187+0000 7fd1097bb1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T00:47:04.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.221+0000 7f8c78978700 1 Processor -- start 2024-01-27T00:47:04.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.221+0000 7f8c78978700 1 -- start start 2024-01-27T00:47:04.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.222+0000 7f8c78978700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8c741599d0 0x7f8c74159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.222+0000 7f8c78978700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c740ab1c0 0x7f8c740b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.222+0000 7f8c78978700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c740ab620 con 0x7f8c7415d8c0 2024-01-27T00:47:04.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.222+0000 7f8c78978700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c740b1a60 con 0x7f8c740ab1c0 2024-01-27T00:47:04.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.222+0000 7f8c78978700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c740b0f40 con 0x7f8c741599d0 2024-01-27T00:47:04.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.222+0000 7f8c71d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c740ab1c0 0x7f8c740b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:04.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.222+0000 7f8c72d9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8c7415d8c0 0x7f8c740b20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37172/0 (socket says 172.21.15.46:37172) 2024-01-27T00:47:04.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.222+0000 7f8c72d9d700 1 -- 172.21.15.46:0/242610474 learned_addr learned my addr 172.21.15.46:0/242610474 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:04.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.222+0000 7f8c7259c700 1 --2- 172.21.15.46:0/242610474 >> v2:172.21.15.106:3300/0 conn(0x7f8c741599d0 0x7f8c74159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:04.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.222+0000 7f8c71d9b700 1 -- 172.21.15.46:0/242610474 >> v2:172.21.15.106:3300/0 conn(0x7f8c741599d0 msgr2=0x7f8c74159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:04.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.222+0000 7f8c71d9b700 1 --2- 172.21.15.46:0/242610474 >> v2:172.21.15.106:3300/0 conn(0x7f8c741599d0 0x7f8c74159da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.222+0000 7f8c71d9b700 1 -- 172.21.15.46:0/242610474 >> v1:172.21.15.46:6789/0 conn(0x7f8c7415d8c0 legacy=0x7f8c740b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:04.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.222+0000 7f8c71d9b700 1 -- 172.21.15.46:0/242610474 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c740b11c0 con 0x7f8c740ab1c0 2024-01-27T00:47:04.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.222+0000 7f8c71d9b700 1 --2- 172.21.15.46:0/242610474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c740ab1c0 0x7f8c740b0a00 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f8c7408aa50 tx=0x7f8c680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed63b85e262f2acf server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:04.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.223+0000 7f8c7159a700 1 -- 172.21.15.46:0/242610474 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c68011630 con 0x7f8c740ab1c0 2024-01-27T00:47:04.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.223+0000 7f8c7159a700 1 -- 172.21.15.46:0/242610474 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c68011790 con 0x7f8c740ab1c0 2024-01-27T00:47:04.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.223+0000 7f8c7159a700 1 -- 172.21.15.46:0/242610474 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c68011b30 con 0x7f8c740ab1c0 2024-01-27T00:47:04.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.223+0000 7f8c78978700 1 -- 172.21.15.46:0/242610474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c740ab1c0 msgr2=0x7f8c740b0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:04.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.223+0000 7f8c78978700 1 --2- 172.21.15.46:0/242610474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c740ab1c0 0x7f8c740b0a00 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f8c7408aa50 tx=0x7f8c680091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.223+0000 7f8c78978700 1 -- 172.21.15.46:0/242610474 shutdown_connections 2024-01-27T00:47:04.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.223+0000 7f8c78978700 1 --2- 172.21.15.46:0/242610474 >> v2:172.21.15.106:3300/0 conn(0x7f8c741599d0 0x7f8c74159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.223+0000 7f8c78978700 1 --2- 172.21.15.46:0/242610474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c740ab1c0 0x7f8c740b0a00 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.223+0000 7f8c78978700 1 -- 172.21.15.46:0/242610474 >> 172.21.15.46:0/242610474 conn(0x7f8c7406f840 msgr2=0x7f8c74087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:04.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.223+0000 7f8c78978700 1 -- 172.21.15.46:0/242610474 shutdown_connections 2024-01-27T00:47:04.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.224+0000 7f8c78978700 1 -- 172.21.15.46:0/242610474 wait complete. 2024-01-27T00:47:04.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.225+0000 7f8c78978700 1 Processor -- start 2024-01-27T00:47:04.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.225+0000 7f8c78978700 1 -- start start 2024-01-27T00:47:04.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c78978700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c740ab1c0 0x7f8c741523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c78978700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c741599d0 0x7f8c741550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c78978700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c7415d8c0 0x7f8c74155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c78978700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c740cd390 con 0x7f8c741599d0 2024-01-27T00:47:04.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c78978700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c741528e0 con 0x7f8c7415d8c0 2024-01-27T00:47:04.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c78978700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c74156460 con 0x7f8c740ab1c0 2024-01-27T00:47:04.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c71d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c741599d0 0x7f8c741550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:04.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c71d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c741599d0 0x7f8c741550e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38898/0 (socket says 172.21.15.46:38898) 2024-01-27T00:47:04.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c71d9b700 1 -- 172.21.15.46:0/1329409227 learned_addr learned my addr 172.21.15.46:0/1329409227 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:04.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c72d9d700 1 --2- 172.21.15.46:0/1329409227 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c7415d8c0 0x7f8c74155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:04.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c7259c700 1 --2- 172.21.15.46:0/1329409227 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c740ab1c0 0x7f8c741523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:04.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c71d9b700 1 -- 172.21.15.46:0/1329409227 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c740ab1c0 msgr2=0x7f8c741523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:04.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c71d9b700 1 --2- 172.21.15.46:0/1329409227 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c740ab1c0 0x7f8c741523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c71d9b700 1 -- 172.21.15.46:0/1329409227 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c7415d8c0 msgr2=0x7f8c74155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c71d9b700 1 --2- 172.21.15.46:0/1329409227 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c7415d8c0 0x7f8c74155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c71d9b700 1 -- 172.21.15.46:0/1329409227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c68009420 con 0x7f8c741599d0 2024-01-27T00:47:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c71d9b700 1 --2- 172.21.15.46:0/1329409227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c741599d0 0x7f8c741550e0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f8c74089010 tx=0x7f8c68017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c627fc700 1 -- 172.21.15.46:0/1329409227 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c68000d10 con 0x7f8c741599d0 2024-01-27T00:47:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c627fc700 1 -- 172.21.15.46:0/1329409227 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c68011b10 con 0x7f8c741599d0 2024-01-27T00:47:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c78978700 1 -- 172.21.15.46:0/1329409227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c74155ea0 con 0x7f8c741599d0 2024-01-27T00:47:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.226+0000 7f8c627fc700 1 -- 172.21.15.46:0/1329409227 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c68025640 con 0x7f8c741599d0 2024-01-27T00:47:04.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.227+0000 7f8c78978700 1 -- 172.21.15.46:0/1329409227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c740cd020 con 0x7f8c741599d0 2024-01-27T00:47:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.228+0000 7f8c78978700 1 -- 172.21.15.46:0/1329409227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c740cd500 con 0x7f8c741599d0 2024-01-27T00:47:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.228+0000 7f8c627fc700 1 -- 172.21.15.46:0/1329409227 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 92) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f8c68025a30 con 0x7f8c741599d0 2024-01-27T00:47:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.229+0000 7f8c627fc700 1 --2- 172.21.15.46:0/1329409227 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c54067050 0x7f8c54069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.229+0000 7f8c7259c700 1 -- 172.21.15.46:0/1329409227 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c54067050 msgr2=0x7f8c54069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.229+0000 7f8c627fc700 1 -- 172.21.15.46:0/1329409227 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f8c680862e0 con 0x7f8c741599d0 2024-01-27T00:47:04.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.229+0000 7f8c7259c700 1 --2- 172.21.15.46:0/1329409227 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c54067050 0x7f8c54069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:47:04.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.233+0000 7f8c627fc700 1 -- 172.21.15.46:0/1329409227 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8c68052fb0 con 0x7f8c741599d0 2024-01-27T00:47:04.281 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:04.281+0000 7f573af0c1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T00:47:04.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.430+0000 7f8c7259c700 1 -- 172.21.15.46:0/1329409227 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c54067050 msgr2=0x7f8c54069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:04.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.430+0000 7f8c7259c700 1 --2- 172.21.15.46:0/1329409227 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c54067050 0x7f8c54069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:47:04.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.441+0000 7f8c78978700 1 -- 172.21.15.46:0/1329409227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8c740b14a0 con 0x7f8c741599d0 2024-01-27T00:47:04.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.442+0000 7f8c627fc700 1 -- 172.21.15.46:0/1329409227 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v176) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8c680565d0 con 0x7f8c741599d0 2024-01-27T00:47:04.442 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-27T00:47:04.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.444+0000 7f8c78978700 1 -- 172.21.15.46:0/1329409227 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c54067050 msgr2=0x7f8c54069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:47:04.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.444+0000 7f8c78978700 1 --2- 172.21.15.46:0/1329409227 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c54067050 0x7f8c54069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.444+0000 7f8c78978700 1 -- 172.21.15.46:0/1329409227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c741599d0 msgr2=0x7f8c741550e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:04.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.444+0000 7f8c78978700 1 --2- 172.21.15.46:0/1329409227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c741599d0 0x7f8c741550e0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f8c74089010 tx=0x7f8c68017000 comp rx=0 tx=0).stop 2024-01-27T00:47:04.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.444+0000 7f8c78978700 1 -- 172.21.15.46:0/1329409227 shutdown_connections 2024-01-27T00:47:04.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.444+0000 7f8c78978700 1 --2- 172.21.15.46:0/1329409227 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c54067050 0x7f8c54069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.444+0000 7f8c78978700 1 --2- 172.21.15.46:0/1329409227 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c7415d8c0 0x7f8c74155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.444+0000 7f8c78978700 1 --2- 172.21.15.46:0/1329409227 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c740ab1c0 0x7f8c741523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.444+0000 7f8c78978700 1 --2- 172.21.15.46:0/1329409227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c741599d0 0x7f8c741550e0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.444+0000 7f8c78978700 1 -- 172.21.15.46:0/1329409227 >> 172.21.15.46:0/1329409227 conn(0x7f8c7406f840 msgr2=0x7f8c740693a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:04.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.445+0000 7f8c78978700 1 -- 172.21.15.46:0/1329409227 shutdown_connections 2024-01-27T00:47:04.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.445+0000 7f8c78978700 1 -- 172.21.15.46:0/1329409227 wait complete. 2024-01-27T00:47:04.460 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-27T00:47:04.490 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:04.490+0000 7f573af0c1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T00:47:04.557 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:04.556+0000 7fd1097bb1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T00:47:04.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.595+0000 7f8d41553700 1 Processor -- start 2024-01-27T00:47:04.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.596+0000 7f8d41553700 1 -- start start 2024-01-27T00:47:04.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.596+0000 7f8d41553700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8d3c159cb0 0x7f8d3c0cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.596+0000 7f8d41553700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d3c0cf050 0x7f8d3c0af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.596+0000 7f8d41553700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8d3c0cf4a0 0x7f8d3c0af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.596+0000 7f8d41553700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d3c0cc100 con 0x7f8d3c159cb0 2024-01-27T00:47:04.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.596+0000 7f8d41553700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d3c0cb850 con 0x7f8d3c0cf4a0 2024-01-27T00:47:04.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.596+0000 7f8d41553700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c0cbaf0 con 0x7f8d3c0cf050 2024-01-27T00:47:04.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.597+0000 7f8d3affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8d3c159cb0 0x7f8d3c0cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:04.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.597+0000 7f8d3affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8d3c159cb0 0x7f8d3c0cc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38914/0 (socket says 172.21.15.46:38914) 2024-01-27T00:47:04.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.597+0000 7f8d3affd700 1 -- 172.21.15.46:0/68345719 learned_addr learned my addr 172.21.15.46:0/68345719 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:04.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.597+0000 7f8d3b7fe700 1 --2- 172.21.15.46:0/68345719 >> v2:172.21.15.106:3300/0 conn(0x7f8d3c0cf4a0 0x7f8d3c0af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:04.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.597+0000 7f8d3a7fc700 1 --2- 172.21.15.46:0/68345719 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d3c0cf050 0x7f8d3c0af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:04.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.597+0000 7f8d3affd700 1 -- 172.21.15.46:0/68345719 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d3c0cf050 msgr2=0x7f8d3c0af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:04.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.597+0000 7f8d3affd700 1 --2- 172.21.15.46:0/68345719 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d3c0cf050 0x7f8d3c0af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.597+0000 7f8d3affd700 1 -- 172.21.15.46:0/68345719 >> v2:172.21.15.106:3300/0 conn(0x7f8d3c0cf4a0 msgr2=0x7f8d3c0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:04.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.597+0000 7f8d3affd700 1 --2- 172.21.15.46:0/68345719 >> v2:172.21.15.106:3300/0 conn(0x7f8d3c0cf4a0 0x7f8d3c0af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.597+0000 7f8d3affd700 1 -- 172.21.15.46:0/68345719 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c0cfc80 con 0x7f8d3c159cb0 2024-01-27T00:47:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.597+0000 7f8d3affd700 1 --2- 172.21.15.46:0/68345719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d3c159cb0 0x7f8d3c0cc590 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c0ce9e0 tx=0x7f8d2c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c56886a58d449c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.597+0000 7f8d39ffb700 1 -- 172.21.15.46:0/68345719 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c00e070 con 0x7f8d3c159cb0 2024-01-27T00:47:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.597+0000 7f8d39ffb700 1 -- 172.21.15.46:0/68345719 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2c00b970 con 0x7f8d3c159cb0 2024-01-27T00:47:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.598+0000 7f8d39ffb700 1 -- 172.21.15.46:0/68345719 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c0092e0 con 0x7f8d3c159cb0 2024-01-27T00:47:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.598+0000 7f8d41553700 1 -- 172.21.15.46:0/68345719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d3c159cb0 msgr2=0x7f8d3c0cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.598+0000 7f8d41553700 1 --2- 172.21.15.46:0/68345719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d3c159cb0 0x7f8d3c0cc590 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c0ce9e0 tx=0x7f8d2c009840 comp rx=0 tx=0).stop 2024-01-27T00:47:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.598+0000 7f8d41553700 1 -- 172.21.15.46:0/68345719 shutdown_connections 2024-01-27T00:47:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.598+0000 7f8d41553700 1 --2- 172.21.15.46:0/68345719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d3c159cb0 0x7f8d3c0cc590 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.598+0000 7f8d41553700 1 --2- 172.21.15.46:0/68345719 >> v2:172.21.15.106:3300/0 conn(0x7f8d3c0cf4a0 0x7f8d3c0af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.598+0000 7f8d41553700 1 --2- 172.21.15.46:0/68345719 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d3c0cf050 0x7f8d3c0af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.598+0000 7f8d41553700 1 -- 172.21.15.46:0/68345719 >> 172.21.15.46:0/68345719 conn(0x7f8d3c087f60 msgr2=0x7f8d3c06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.598+0000 7f8d41553700 1 -- 172.21.15.46:0/68345719 shutdown_connections 2024-01-27T00:47:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.598+0000 7f8d41553700 1 -- 172.21.15.46:0/68345719 wait complete. 2024-01-27T00:47:04.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.600+0000 7f8d41553700 1 Processor -- start 2024-01-27T00:47:04.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.600+0000 7f8d41553700 1 -- start start 2024-01-27T00:47:04.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.602+0000 7f8d41553700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d3c0cf050 0x7f8d3c0b0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.602+0000 7f8d41553700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d3c0cf4a0 0x7f8d3c16def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.602+0000 7f8d41553700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d3c159cb0 0x7f8d3c0b1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.602+0000 7f8d41553700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c0b2110 con 0x7f8d3c0cf050 2024-01-27T00:47:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.602+0000 7f8d41553700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c0b0a10 con 0x7f8d3c159cb0 2024-01-27T00:47:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.602+0000 7f8d41553700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c0b1e70 con 0x7f8d3c0cf4a0 2024-01-27T00:47:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.602+0000 7f8d3affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d3c0cf050 0x7f8d3c0b0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.602+0000 7f8d3affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d3c0cf050 0x7f8d3c0b0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38922/0 (socket says 172.21.15.46:38922) 2024-01-27T00:47:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.602+0000 7f8d3affd700 1 -- 172.21.15.46:0/579908380 learned_addr learned my addr 172.21.15.46:0/579908380 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.602+0000 7f8d3b7fe700 1 --2- 172.21.15.46:0/579908380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d3c159cb0 0x7f8d3c0b1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.602+0000 7f8d3a7fc700 1 --2- 172.21.15.46:0/579908380 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d3c0cf4a0 0x7f8d3c16def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.602+0000 7f8d3affd700 1 -- 172.21.15.46:0/579908380 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d3c0cf4a0 msgr2=0x7f8d3c16def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.602+0000 7f8d3affd700 1 --2- 172.21.15.46:0/579908380 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d3c0cf4a0 0x7f8d3c16def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.602+0000 7f8d3affd700 1 -- 172.21.15.46:0/579908380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d3c159cb0 msgr2=0x7f8d3c0b1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.602+0000 7f8d3affd700 1 --2- 172.21.15.46:0/579908380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d3c159cb0 0x7f8d3c0b1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.602+0000 7f8d3affd700 1 -- 172.21.15.46:0/579908380 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c009ae0 con 0x7f8d3c0cf050 2024-01-27T00:47:04.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.602+0000 7f8d3affd700 1 --2- 172.21.15.46:0/579908380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d3c0cf050 0x7f8d3c0b0ea0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c012000 tx=0x7f8d2c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:04.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.603+0000 7f8d22ffd700 1 -- 172.21.15.46:0/579908380 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c0087f0 con 0x7f8d3c0cf050 2024-01-27T00:47:04.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.603+0000 7f8d22ffd700 1 -- 172.21.15.46:0/579908380 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2c008970 con 0x7f8d3c0cf050 2024-01-27T00:47:04.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.603+0000 7f8d41553700 1 -- 172.21.15.46:0/579908380 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3c0b00e0 con 0x7f8d3c0cf050 2024-01-27T00:47:04.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.603+0000 7f8d22ffd700 1 -- 172.21.15.46:0/579908380 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c008d30 con 0x7f8d3c0cf050 2024-01-27T00:47:04.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.603+0000 7f8d41553700 1 -- 172.21.15.46:0/579908380 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3c0b06a0 con 0x7f8d3c0cf050 2024-01-27T00:47:04.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.604+0000 7f8d41553700 1 -- 172.21.15.46:0/579908380 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d3c0b0cf0 con 0x7f8d3c0cf050 2024-01-27T00:47:04.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.604+0000 7f8d22ffd700 1 -- 172.21.15.46:0/579908380 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 92) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f8d2c003040 con 0x7f8d3c0cf050 2024-01-27T00:47:04.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.605+0000 7f8d22ffd700 1 --2- 172.21.15.46:0/579908380 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8d28067050 0x7f8d28069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.605+0000 7f8d3a7fc700 1 -- 172.21.15.46:0/579908380 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8d28067050 msgr2=0x7f8d28069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:04.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.605+0000 7f8d22ffd700 1 -- 172.21.15.46:0/579908380 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f8d2c086170 con 0x7f8d3c0cf050 2024-01-27T00:47:04.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.605+0000 7f8d3a7fc700 1 --2- 172.21.15.46:0/579908380 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8d28067050 0x7f8d28069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:47:04.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.609+0000 7f8d22ffd700 1 -- 172.21.15.46:0/579908380 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8d2c052e80 con 0x7f8d3c0cf050 2024-01-27T00:47:04.716 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:04.716+0000 7f573af0c1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T00:47:04.787 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:04.787+0000 7f573af0c1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T00:47:04.802 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:04.802+0000 7fd1097bb1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T00:47:04.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.806+0000 7f8d3a7fc700 1 -- 172.21.15.46:0/579908380 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8d28067050 msgr2=0x7f8d28069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:04.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.806+0000 7f8d3a7fc700 1 --2- 172.21.15.46:0/579908380 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8d28067050 0x7f8d28069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:47:04.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.817+0000 7f8d41553700 1 -- 172.21.15.46:0/579908380 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8d3c04c710 con 0x7f8d3c0cf050 2024-01-27T00:47:04.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.817+0000 7f8d22ffd700 1 -- 172.21.15.46:0/579908380 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v176) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8d2c0564a0 con 0x7f8d3c0cf050 2024-01-27T00:47:04.817 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-27T00:47:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.819+0000 7f8d41553700 1 -- 172.21.15.46:0/579908380 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8d28067050 msgr2=0x7f8d28069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:47:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.819+0000 7f8d41553700 1 --2- 172.21.15.46:0/579908380 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8d28067050 0x7f8d28069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.819+0000 7f8d41553700 1 -- 172.21.15.46:0/579908380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d3c0cf050 msgr2=0x7f8d3c0b0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.819+0000 7f8d41553700 1 --2- 172.21.15.46:0/579908380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d3c0cf050 0x7f8d3c0b0ea0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c012000 tx=0x7f8d2c018000 comp rx=0 tx=0).stop 2024-01-27T00:47:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.819+0000 7f8d41553700 1 -- 172.21.15.46:0/579908380 shutdown_connections 2024-01-27T00:47:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.819+0000 7f8d41553700 1 --2- 172.21.15.46:0/579908380 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f8d28067050 0x7f8d28069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.819+0000 7f8d41553700 1 --2- 172.21.15.46:0/579908380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d3c0cf050 0x7f8d3c0b0ea0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.819+0000 7f8d41553700 1 --2- 172.21.15.46:0/579908380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d3c159cb0 0x7f8d3c0b1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.819+0000 7f8d41553700 1 --2- 172.21.15.46:0/579908380 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d3c0cf4a0 0x7f8d3c16def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.819+0000 7f8d41553700 1 -- 172.21.15.46:0/579908380 >> 172.21.15.46:0/579908380 conn(0x7f8d3c087f60 msgr2=0x7f8d3c0c6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:04.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.819+0000 7f8d41553700 1 -- 172.21.15.46:0/579908380 shutdown_connections 2024-01-27T00:47:04.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.819+0000 7f8d41553700 1 -- 172.21.15.46:0/579908380 wait complete. 2024-01-27T00:47:04.834 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T00:47:04.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.969+0000 7f4dcc411700 1 Processor -- start 2024-01-27T00:47:04.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.969+0000 7f4dcc411700 1 -- start start 2024-01-27T00:47:04.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.971+0000 7f4dcc411700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dc415ce40 0x7f4dc40d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.971+0000 7f4dcc411700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4dc415d210 0x7f4dc40cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.971+0000 7f4dcc411700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4dc40d1390 0x7f4dc40cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.971+0000 7f4dcc411700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc40cff70 con 0x7f4dc40d1390 2024-01-27T00:47:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.971+0000 7f4dcc411700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc40cf400 con 0x7f4dc415d210 2024-01-27T00:47:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.971+0000 7f4dcc411700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc40cf660 con 0x7f4dc415ce40 2024-01-27T00:47:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.971+0000 7f4dca9ae700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4dc40d1390 0x7f4dc40cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.971+0000 7f4dca9ae700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4dc40d1390 0x7f4dc40cbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38938/0 (socket says 172.21.15.46:38938) 2024-01-27T00:47:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.971+0000 7f4dca9ae700 1 -- 172.21.15.46:0/1710789787 learned_addr learned my addr 172.21.15.46:0/1710789787 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.972+0000 7f4dc99ac700 1 --2- 172.21.15.46:0/1710789787 >> v2:172.21.15.106:3300/0 conn(0x7f4dc415d210 0x7f4dc40cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.972+0000 7f4dca1ad700 1 --2- 172.21.15.46:0/1710789787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dc415ce40 0x7f4dc40d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.972+0000 7f4dca9ae700 1 -- 172.21.15.46:0/1710789787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dc415ce40 msgr2=0x7f4dc40d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.972+0000 7f4dca9ae700 1 --2- 172.21.15.46:0/1710789787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dc415ce40 0x7f4dc40d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.972+0000 7f4dca9ae700 1 -- 172.21.15.46:0/1710789787 >> v2:172.21.15.106:3300/0 conn(0x7f4dc415d210 msgr2=0x7f4dc40cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.972+0000 7f4dca9ae700 1 --2- 172.21.15.46:0/1710789787 >> v2:172.21.15.106:3300/0 conn(0x7f4dc415d210 0x7f4dc40cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.972+0000 7f4dca9ae700 1 -- 172.21.15.46:0/1710789787 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc40d0500 con 0x7f4dc40d1390 2024-01-27T00:47:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.972+0000 7f4dca9ae700 1 --2- 172.21.15.46:0/1710789787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc40d1390 0x7f4dc40cbff0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4088cb0 tx=0x7f4dbc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7366de6bff8f4dd6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.972+0000 7f4dc91ab700 1 -- 172.21.15.46:0/1710789787 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc00e070 con 0x7f4dc40d1390 2024-01-27T00:47:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.972+0000 7f4dc91ab700 1 -- 172.21.15.46:0/1710789787 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dbc00b970 con 0x7f4dc40d1390 2024-01-27T00:47:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.972+0000 7f4dc91ab700 1 -- 172.21.15.46:0/1710789787 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc0092e0 con 0x7f4dc40d1390 2024-01-27T00:47:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.973+0000 7f4dcc411700 1 -- 172.21.15.46:0/1710789787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc40d1390 msgr2=0x7f4dc40cbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.973+0000 7f4dcc411700 1 --2- 172.21.15.46:0/1710789787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc40d1390 0x7f4dc40cbff0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4088cb0 tx=0x7f4dbc009840 comp rx=0 tx=0).stop 2024-01-27T00:47:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.973+0000 7f4dcc411700 1 -- 172.21.15.46:0/1710789787 shutdown_connections 2024-01-27T00:47:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.973+0000 7f4dcc411700 1 --2- 172.21.15.46:0/1710789787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc40d1390 0x7f4dc40cbff0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.973+0000 7f4dcc411700 1 --2- 172.21.15.46:0/1710789787 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dc415ce40 0x7f4dc40d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.973+0000 7f4dcc411700 1 --2- 172.21.15.46:0/1710789787 >> v2:172.21.15.106:3300/0 conn(0x7f4dc415d210 0x7f4dc40cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.973+0000 7f4dcc411700 1 -- 172.21.15.46:0/1710789787 >> 172.21.15.46:0/1710789787 conn(0x7f4dc406f840 msgr2=0x7f4dc40d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.973+0000 7f4dcc411700 1 -- 172.21.15.46:0/1710789787 shutdown_connections 2024-01-27T00:47:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.973+0000 7f4dcc411700 1 -- 172.21.15.46:0/1710789787 wait complete. 2024-01-27T00:47:04.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.974+0000 7f4dcc411700 1 Processor -- start 2024-01-27T00:47:04.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.974+0000 7f4dcc411700 1 -- start start 2024-01-27T00:47:04.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.975+0000 7f4dcc411700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4dc40d1390 0x7f4dc414a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.975+0000 7f4dcc411700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dc415ce40 0x7f4dc40c6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.975+0000 7f4dcc411700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc415d210 0x7f4dc409cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.975+0000 7f4dcc411700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc4147850 con 0x7f4dc415d210 2024-01-27T00:47:04.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.975+0000 7f4dcc411700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc409b290 con 0x7f4dc40d1390 2024-01-27T00:47:04.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.975+0000 7f4dcc411700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc414a780 con 0x7f4dc415ce40 2024-01-27T00:47:04.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.975+0000 7f4dca9ae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc415d210 0x7f4dc409cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:04.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.975+0000 7f4dca1ad700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4dc40d1390 0x7f4dc414a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:04.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.975+0000 7f4dca9ae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc415d210 0x7f4dc409cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38944/0 (socket says 172.21.15.46:38944) 2024-01-27T00:47:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.975+0000 7f4dca9ae700 1 -- 172.21.15.46:0/1937993882 learned_addr learned my addr 172.21.15.46:0/1937993882 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.975+0000 7f4dca1ad700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4dc40d1390 0x7f4dc414a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:33128/0 (socket says 172.21.15.46:33128) 2024-01-27T00:47:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.975+0000 7f4dc99ac700 1 --2- 172.21.15.46:0/1937993882 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dc415ce40 0x7f4dc40c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.976+0000 7f4dca1ad700 1 -- 172.21.15.46:0/1937993882 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dc415ce40 msgr2=0x7f4dc40c6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.976+0000 7f4dca1ad700 1 --2- 172.21.15.46:0/1937993882 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dc415ce40 0x7f4dc40c6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.976+0000 7f4dca1ad700 1 -- 172.21.15.46:0/1937993882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc415d210 msgr2=0x7f4dc409cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.976+0000 7f4dca1ad700 1 --2- 172.21.15.46:0/1937993882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc415d210 0x7f4dc409cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.976+0000 7f4dca1ad700 1 -- 172.21.15.46:0/1937993882 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc009ae0 con 0x7f4dc40d1390 2024-01-27T00:47:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.976+0000 7f4dca1ad700 1 --2- 172.21.15.46:0/1937993882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4dc40d1390 0x7f4dc414a240 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f4db400ea50 tx=0x7f4db4009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.976+0000 7f4dc27fc700 1 -- 172.21.15.46:0/1937993882 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db400c640 con 0x7f4dc40d1390 2024-01-27T00:47:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.976+0000 7f4dc27fc700 1 -- 172.21.15.46:0/1937993882 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db400c7a0 con 0x7f4dc40d1390 2024-01-27T00:47:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.976+0000 7f4dc27fc700 1 -- 172.21.15.46:0/1937993882 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db400cb40 con 0x7f4dc40d1390 2024-01-27T00:47:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.976+0000 7f4dcc411700 1 -- 172.21.15.46:0/1937993882 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc409a680 con 0x7f4dc40d1390 2024-01-27T00:47:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.976+0000 7f4dcc411700 1 -- 172.21.15.46:0/1937993882 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc409ac40 con 0x7f4dc40d1390 2024-01-27T00:47:04.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.977+0000 7f4dcc411700 1 -- 172.21.15.46:0/1937993882 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc409b570 con 0x7f4dc40d1390 2024-01-27T00:47:04.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.978+0000 7f4dc27fc700 1 -- 172.21.15.46:0/1937993882 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 92) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f4db4013470 con 0x7f4dc40d1390 2024-01-27T00:47:04.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.978+0000 7f4dc27fc700 1 --2- 172.21.15.46:0/1937993882 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4db0067050 0x7f4db0069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:04.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.978+0000 7f4dc27fc700 1 -- 172.21.15.46:0/1937993882 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f4db4084fd0 con 0x7f4dc40d1390 2024-01-27T00:47:04.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.978+0000 7f4dc99ac700 1 -- 172.21.15.46:0/1937993882 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4db0067050 msgr2=0x7f4db0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:04.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.978+0000 7f4dc99ac700 1 --2- 172.21.15.46:0/1937993882 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4db0067050 0x7f4db0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:47:04.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:04.982+0000 7f4dc27fc700 1 -- 172.21.15.46:0/1937993882 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4db4051d40 con 0x7f4dc40d1390 2024-01-27T00:47:05.066 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:05.066+0000 7fd1097bb1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T00:47:05.147 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:05.147+0000 7fd1097bb1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T00:47:05.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.179+0000 7f4dc99ac700 1 -- 172.21.15.46:0/1937993882 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4db0067050 msgr2=0x7f4db0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:05.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.179+0000 7f4dc99ac700 1 --2- 172.21.15.46:0/1937993882 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4db0067050 0x7f4db0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:47:05.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.237+0000 7f4dcc411700 1 -- 172.21.15.46:0/1937993882 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dc404c710 con 0x7f4dc40d1390 2024-01-27T00:47:05.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.238+0000 7f4dc27fc700 1 -- 172.21.15.46:0/1937993882 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4607 (secure 0 0 0) 0x7f4db4017090 con 0x7f4dc40d1390 2024-01-27T00:47:05.238 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:05.238 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":88,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9061,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/3698053981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":3698053981},{"type":"v1","addr":"172.21.15.106:6833","nonce":3698053981}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":9064,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2056976879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2056976879},{"type":"v1","addr":"172.21.15.46:6834","nonce":2056976879}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":82},{"gid":9067,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1576406314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1576406314},{"type":"v1","addr":"172.21.15.46:6835","nonce":1576406314}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T00:46:44.978654+0000","modified":"2024-01-27T00:46:48.558026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9070":{"gid":9070,"name":"d","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6836/2857098625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2857098625},{"type":"v1","addr":"172.21.15.106:6836","nonce":2857098625}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2024-01-27T00:47:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.240+0000 7f4dcc411700 1 -- 172.21.15.46:0/1937993882 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4db0067050 msgr2=0x7f4db0069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:47:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.240+0000 7f4dcc411700 1 --2- 172.21.15.46:0/1937993882 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4db0067050 0x7f4db0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.240+0000 7f4dcc411700 1 -- 172.21.15.46:0/1937993882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4dc40d1390 msgr2=0x7f4dc414a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.240+0000 7f4dcc411700 1 --2- 172.21.15.46:0/1937993882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4dc40d1390 0x7f4dc414a240 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f4db400ea50 tx=0x7f4db4009120 comp rx=0 tx=0).stop 2024-01-27T00:47:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.240+0000 7f4dcc411700 1 -- 172.21.15.46:0/1937993882 shutdown_connections 2024-01-27T00:47:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.240+0000 7f4dcc411700 1 --2- 172.21.15.46:0/1937993882 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f4db0067050 0x7f4db0069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.240+0000 7f4dcc411700 1 --2- 172.21.15.46:0/1937993882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc415d210 0x7f4dc409cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.240+0000 7f4dcc411700 1 --2- 172.21.15.46:0/1937993882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4dc40d1390 0x7f4dc414a240 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.240+0000 7f4dcc411700 1 --2- 172.21.15.46:0/1937993882 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dc415ce40 0x7f4dc40c6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.240+0000 7f4dcc411700 1 -- 172.21.15.46:0/1937993882 >> 172.21.15.46:0/1937993882 conn(0x7f4dc406f840 msgr2=0x7f4dc40b5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.240+0000 7f4dcc411700 1 -- 172.21.15.46:0/1937993882 shutdown_connections 2024-01-27T00:47:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.240+0000 7f4dcc411700 1 -- 172.21.15.46:0/1937993882 wait complete. 2024-01-27T00:47:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 88 2024-01-27T00:47:05.255 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T00:47:05.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.387+0000 7fdd0b24a700 1 Processor -- start 2024-01-27T00:47:05.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.387+0000 7fdd0b24a700 1 -- start start 2024-01-27T00:47:05.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.388+0000 7fdd0b24a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdd0415ce40 0x7fdd040d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:05.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.388+0000 7fdd0b24a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd040cc1c0 0x7fdd040ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:05.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.388+0000 7fdd0b24a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd040cff70 con 0x7fdd0415d210 2024-01-27T00:47:05.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.388+0000 7fdd0b24a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd040cf400 con 0x7fdd0415ce40 2024-01-27T00:47:05.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.388+0000 7fdd0b24a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd040cf660 con 0x7fdd040cc1c0 2024-01-27T00:47:05.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.388+0000 7fdd08fe6700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdd0415ce40 0x7fdd040d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:05.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.388+0000 7fdd097e7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd040cc1c0 0x7fdd040ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:05.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.388+0000 7fdd03fff700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fdd0415d210 0x7fdd040cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:38144/0 (socket says 172.21.15.46:38144) 2024-01-27T00:47:05.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.388+0000 7fdd03fff700 1 -- 172.21.15.46:0/127978445 learned_addr learned my addr 172.21.15.46:0/127978445 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:05.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.389+0000 7fdd037fe700 1 -- 172.21.15.46:0/127978445 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906663174 0 0) 0x7fdd040cff70 con 0x7fdd0415d210 2024-01-27T00:47:05.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.389+0000 7fdd037fe700 1 -- 172.21.15.46:0/127978445 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcec003570 con 0x7fdd0415d210 2024-01-27T00:47:05.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.389+0000 7fdd08fe6700 1 -- 172.21.15.46:0/127978445 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd040cc1c0 msgr2=0x7fdd040ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:05.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.389+0000 7fdd08fe6700 1 --2- 172.21.15.46:0/127978445 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd040cc1c0 0x7fdd040ade30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.389+0000 7fdd08fe6700 1 -- 172.21.15.46:0/127978445 >> v1:172.21.15.106:6789/0 conn(0x7fdd0415d210 legacy=0x7fdd040cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:05.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.389+0000 7fdd08fe6700 1 -- 172.21.15.46:0/127978445 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd040d0500 con 0x7fdd0415ce40 2024-01-27T00:47:05.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.389+0000 7fdd097e7700 1 --2- 172.21.15.46:0/127978445 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd040cc1c0 0x7fdd040ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:05.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.389+0000 7fdd08fe6700 1 --2- 172.21.15.46:0/127978445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd0415ce40 0x7fdd040d0e50 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fdcf40098d0 tx=0x7fdcf4009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10d42144ea583fdc server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:05.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.389+0000 7fdd037fe700 1 -- 172.21.15.46:0/127978445 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4005c10 con 0x7fdd0415ce40 2024-01-27T00:47:05.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.389+0000 7fdd037fe700 1 -- 172.21.15.46:0/127978445 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf4005d70 con 0x7fdd0415ce40 2024-01-27T00:47:05.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.389+0000 7fdd037fe700 1 -- 172.21.15.46:0/127978445 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4006280 con 0x7fdd0415ce40 2024-01-27T00:47:05.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.390+0000 7fdd0b24a700 1 -- 172.21.15.46:0/127978445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd0415ce40 msgr2=0x7fdd040d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:05.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.390+0000 7fdd0b24a700 1 --2- 172.21.15.46:0/127978445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd0415ce40 0x7fdd040d0e50 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fdcf40098d0 tx=0x7fdcf4009120 comp rx=0 tx=0).stop 2024-01-27T00:47:05.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.390+0000 7fdd0b24a700 1 -- 172.21.15.46:0/127978445 shutdown_connections 2024-01-27T00:47:05.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.390+0000 7fdd0b24a700 1 --2- 172.21.15.46:0/127978445 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd040cc1c0 0x7fdd040ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.390+0000 7fdd0b24a700 1 --2- 172.21.15.46:0/127978445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd0415ce40 0x7fdd040d0e50 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.390+0000 7fdd0b24a700 1 -- 172.21.15.46:0/127978445 >> 172.21.15.46:0/127978445 conn(0x7fdd0406f840 msgr2=0x7fdd040d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:05.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.390+0000 7fdd0b24a700 1 -- 172.21.15.46:0/127978445 shutdown_connections 2024-01-27T00:47:05.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.390+0000 7fdd0b24a700 1 -- 172.21.15.46:0/127978445 wait complete. 2024-01-27T00:47:05.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.391+0000 7fdd0b24a700 1 Processor -- start 2024-01-27T00:47:05.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.392+0000 7fdd0b24a700 1 -- start start 2024-01-27T00:47:05.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd0b24a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd040cc1c0 0x7fdd0414ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:05.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd0b24a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd0415ce40 0x7fdd0409d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:05.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd0b24a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd0415d210 0x7fdd040c8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:05.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd0b24a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0409c790 con 0x7fdd040cc1c0 2024-01-27T00:47:05.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd0b24a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0409bb80 con 0x7fdd0415d210 2024-01-27T00:47:05.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd0b24a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0409be60 con 0x7fdd0415ce40 2024-01-27T00:47:05.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd08fe6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd040cc1c0 0x7fdd0414ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:05.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd08fe6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd040cc1c0 0x7fdd0414ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38958/0 (socket says 172.21.15.46:38958) 2024-01-27T00:47:05.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd08fe6700 1 -- 172.21.15.46:0/1362819770 learned_addr learned my addr 172.21.15.46:0/1362819770 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:05.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd03fff700 1 --2- 172.21.15.46:0/1362819770 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd0415ce40 0x7fdd0409d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:05.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd097e7700 1 --2- 172.21.15.46:0/1362819770 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd0415d210 0x7fdd040c8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:05.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd08fe6700 1 -- 172.21.15.46:0/1362819770 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd0415ce40 msgr2=0x7fdd0409d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:05.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd08fe6700 1 --2- 172.21.15.46:0/1362819770 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd0415ce40 0x7fdd0409d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd08fe6700 1 -- 172.21.15.46:0/1362819770 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd0415d210 msgr2=0x7fdd040c8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:05.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd08fe6700 1 --2- 172.21.15.46:0/1362819770 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd0415d210 0x7fdd040c8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd08fe6700 1 -- 172.21.15.46:0/1362819770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc009a20 con 0x7fdd040cc1c0 2024-01-27T00:47:05.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd03fff700 1 --2- 172.21.15.46:0/1362819770 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd0415ce40 0x7fdd0409d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:05.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.393+0000 7fdd08fe6700 1 --2- 172.21.15.46:0/1362819770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd040cc1c0 0x7fdd0414ea00 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009ec0 tx=0x7fdcf4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:05.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.394+0000 7fdd00ff9700 1 -- 172.21.15.46:0/1362819770 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf401d730 con 0x7fdd040cc1c0 2024-01-27T00:47:05.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.394+0000 7fdd00ff9700 1 -- 172.21.15.46:0/1362819770 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf401d890 con 0x7fdd040cc1c0 2024-01-27T00:47:05.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.394+0000 7fdd0b24a700 1 -- 172.21.15.46:0/1362819770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf40093c0 con 0x7fdd040cc1c0 2024-01-27T00:47:05.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.394+0000 7fdd00ff9700 1 -- 172.21.15.46:0/1362819770 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf401dc50 con 0x7fdd040cc1c0 2024-01-27T00:47:05.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.394+0000 7fdd0b24a700 1 -- 172.21.15.46:0/1362819770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0409c420 con 0x7fdd040cc1c0 2024-01-27T00:47:05.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.395+0000 7fdd0b24a700 1 -- 172.21.15.46:0/1362819770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd0404c710 con 0x7fdd040cc1c0 2024-01-27T00:47:05.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.395+0000 7fdd00ff9700 1 -- 172.21.15.46:0/1362819770 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 92) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fdcf401b480 con 0x7fdd040cc1c0 2024-01-27T00:47:05.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.396+0000 7fdd00ff9700 1 --2- 172.21.15.46:0/1362819770 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fdcf0067050 0x7fdcf0069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:05.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.396+0000 7fdd00ff9700 1 -- 172.21.15.46:0/1362819770 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fdcf4086630 con 0x7fdd040cc1c0 2024-01-27T00:47:05.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.396+0000 7fdd03fff700 1 -- 172.21.15.46:0/1362819770 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fdcf0067050 msgr2=0x7fdcf0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:05.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.396+0000 7fdd03fff700 1 --2- 172.21.15.46:0/1362819770 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fdcf0067050 0x7fdcf0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:47:05.399 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:05.396+0000 7f573af0c1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T00:47:05.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.404+0000 7fdd00ff9700 1 -- 172.21.15.46:0/1362819770 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdcf4053460 con 0x7fdd040cc1c0 2024-01-27T00:47:05.471 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:05.472+0000 7f573af0c1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T00:47:05.596 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:05.597+0000 7f573af0c1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T00:47:05.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.597+0000 7fdd03fff700 1 -- 172.21.15.46:0/1362819770 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fdcf0067050 msgr2=0x7fdcf0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:05.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.597+0000 7fdd03fff700 1 --2- 172.21.15.46:0/1362819770 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fdcf0067050 0x7fdcf0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:47:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.610+0000 7fdd0b24a700 1 -- 172.21.15.46:0/1362819770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd040cf920 con 0x7fdd040cc1c0 2024-01-27T00:47:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.612+0000 7fdd00ff9700 1 -- 172.21.15.46:0/1362819770 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v176) v1 ==== 74+0+16607 (secure 0 0 0) 0x7fdcf4056a80 con 0x7fdd040cc1c0 2024-01-27T00:47:05.612 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:05.612 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":176,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T00:46:59.569524+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-01-27T00:46:43.494684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-01-27T00:46:43.949785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/2302701803":"2024-01-28T00:46:59.569484+0000","172.21.15.106:0/3368647852":"2024-01-28T00:46:59.569484+0000","172.21.15.106:6837/43388":"2024-01-28T00:46:59.569484+0000","172.21.15.106:6835/43388":"2024-01-28T00:46:59.569484+0000","172.21.15.106:0/3900909328":"2024-01-28T00:46:59.569484+0000","172.21.15.106:0/4251291402":"2024-01-28T00:46:59.569484+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T00:47:05.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.614+0000 7fdd0b24a700 1 -- 172.21.15.46:0/1362819770 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fdcf0067050 msgr2=0x7fdcf0069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:47:05.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.614+0000 7fdd0b24a700 1 --2- 172.21.15.46:0/1362819770 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fdcf0067050 0x7fdcf0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.614+0000 7fdd0b24a700 1 -- 172.21.15.46:0/1362819770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd040cc1c0 msgr2=0x7fdd0414ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.614+0000 7fdd0b24a700 1 --2- 172.21.15.46:0/1362819770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd040cc1c0 0x7fdd0414ea00 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009ec0 tx=0x7fdcf4016000 comp rx=0 tx=0).stop 2024-01-27T00:47:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.614+0000 7fdd0b24a700 1 -- 172.21.15.46:0/1362819770 shutdown_connections 2024-01-27T00:47:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.614+0000 7fdd0b24a700 1 --2- 172.21.15.46:0/1362819770 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fdcf0067050 0x7fdcf0069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.614+0000 7fdd0b24a700 1 --2- 172.21.15.46:0/1362819770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd040cc1c0 0x7fdd0414ea00 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.614+0000 7fdd0b24a700 1 --2- 172.21.15.46:0/1362819770 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd0415d210 0x7fdd040c8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.614+0000 7fdd0b24a700 1 --2- 172.21.15.46:0/1362819770 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd0415ce40 0x7fdd0409d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.614+0000 7fdd0b24a700 1 -- 172.21.15.46:0/1362819770 >> 172.21.15.46:0/1362819770 conn(0x7fdd0406f840 msgr2=0x7fdd040cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.614+0000 7fdd0b24a700 1 -- 172.21.15.46:0/1362819770 shutdown_connections 2024-01-27T00:47:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.614+0000 7fdd0b24a700 1 -- 172.21.15.46:0/1362819770 wait complete. 2024-01-27T00:47:05.629 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-27T00:47:05.630 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T00:47:05.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.767+0000 7fabeeee7700 1 Processor -- start 2024-01-27T00:47:05.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.767+0000 7fabeeee7700 1 -- start start 2024-01-27T00:47:05.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.768+0000 7fabeeee7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fabe80c6520 0x7fabe80ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:05.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.768+0000 7fabeeee7700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe815a0b0 con 0x7fabe80b2b40 2024-01-27T00:47:05.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.768+0000 7fabeeee7700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe80ae4d0 con 0x7fabe80ced50 2024-01-27T00:47:05.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.768+0000 7fabeeee7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe80af070 con 0x7fabe80c6520 2024-01-27T00:47:05.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.768+0000 7fabecc83700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fabe80b2b40 0x7fabe80ae960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37186/0 (socket says 172.21.15.46:37186) 2024-01-27T00:47:05.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.768+0000 7fabecc83700 1 -- 172.21.15.46:0/1361536608 learned_addr learned my addr 172.21.15.46:0/1361536608 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:05.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.768+0000 7fabe7fff700 1 --2- 172.21.15.46:0/1361536608 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fabe80c6520 0x7fabe80ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:05.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.769+0000 7fabe77fe700 1 -- 172.21.15.46:0/1361536608 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248768858 0 0) 0x7fabe815a0b0 con 0x7fabe80b2b40 2024-01-27T00:47:05.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.769+0000 7fabe77fe700 1 -- 172.21.15.46:0/1361536608 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabd0003570 con 0x7fabe80b2b40 2024-01-27T00:47:05.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.769+0000 7fabe77fe700 1 -- 172.21.15.46:0/1361536608 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684267941 0 0) 0x7fabe80ae4d0 con 0x7fabe80ced50 2024-01-27T00:47:05.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.769+0000 7fabe77fe700 1 -- 172.21.15.46:0/1361536608 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe815a0b0 con 0x7fabe80ced50 2024-01-27T00:47:05.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.769+0000 7fabe7fff700 1 -- 172.21.15.46:0/1361536608 >> v1:172.21.15.106:6789/0 conn(0x7fabe80ced50 legacy=0x7fabe80cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:05.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.769+0000 7fabe7fff700 1 -- 172.21.15.46:0/1361536608 >> v1:172.21.15.46:6789/0 conn(0x7fabe80b2b40 legacy=0x7fabe80ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:05.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.769+0000 7fabe7fff700 1 -- 172.21.15.46:0/1361536608 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe80ad8c0 con 0x7fabe80c6520 2024-01-27T00:47:05.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.769+0000 7fabe7fff700 1 --2- 172.21.15.46:0/1361536608 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fabe80c6520 0x7fabe80ad1c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009910 tx=0x7fabdc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb58b72b8d63026f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:05.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.769+0000 7fabe77fe700 1 -- 172.21.15.46:0/1361536608 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabdc011630 con 0x7fabe80c6520 2024-01-27T00:47:05.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.769+0000 7fabe77fe700 1 -- 172.21.15.46:0/1361536608 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabdc011790 con 0x7fabe80c6520 2024-01-27T00:47:05.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.769+0000 7fabe77fe700 1 -- 172.21.15.46:0/1361536608 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabdc011b30 con 0x7fabe80c6520 2024-01-27T00:47:05.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.770+0000 7fabeeee7700 1 -- 172.21.15.46:0/1361536608 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fabe80c6520 msgr2=0x7fabe80ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:05.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.770+0000 7fabeeee7700 1 --2- 172.21.15.46:0/1361536608 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fabe80c6520 0x7fabe80ad1c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009910 tx=0x7fabdc0091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.770+0000 7fabeeee7700 1 -- 172.21.15.46:0/1361536608 shutdown_connections 2024-01-27T00:47:05.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.770+0000 7fabeeee7700 1 --2- 172.21.15.46:0/1361536608 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fabe80c6520 0x7fabe80ad1c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.770+0000 7fabeeee7700 1 -- 172.21.15.46:0/1361536608 >> 172.21.15.46:0/1361536608 conn(0x7fabe8099a30 msgr2=0x7fabe806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:05.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.770+0000 7fabeeee7700 1 -- 172.21.15.46:0/1361536608 shutdown_connections 2024-01-27T00:47:05.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.770+0000 7fabeeee7700 1 -- 172.21.15.46:0/1361536608 wait complete. 2024-01-27T00:47:05.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.772+0000 7fabeeee7700 1 Processor -- start 2024-01-27T00:47:05.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.772+0000 7fabeeee7700 1 -- start start 2024-01-27T00:47:05.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.773+0000 7fabeeee7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fabe80b2b40 0x7fabe80b3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:05.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.773+0000 7fabeeee7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe80c6520 0x7fabe81516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.773+0000 7fabeeee7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabe80ced50 0x7fabe80b0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.773+0000 7fabeeee7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe816d210 con 0x7fabe80c6520 2024-01-27T00:47:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.773+0000 7fabeeee7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe816d460 con 0x7fabe80ced50 2024-01-27T00:47:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.773+0000 7fabeeee7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe816d740 con 0x7fabe80b2b40 2024-01-27T00:47:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.773+0000 7fabe7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe80c6520 0x7fabe81516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.773+0000 7fabecc83700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fabe80b2b40 0x7fabe80b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.773+0000 7fabed484700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabe80ced50 0x7fabe80b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.773+0000 7fabe7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe80c6520 0x7fabe81516d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38968/0 (socket says 172.21.15.46:38968) 2024-01-27T00:47:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.773+0000 7fabe7fff700 1 -- 172.21.15.46:0/3891496855 learned_addr learned my addr 172.21.15.46:0/3891496855 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.774+0000 7fabed484700 1 -- 172.21.15.46:0/3891496855 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fabe80b2b40 msgr2=0x7fabe80b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.774+0000 7fabed484700 1 --2- 172.21.15.46:0/3891496855 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fabe80b2b40 0x7fabe80b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.774+0000 7fabed484700 1 -- 172.21.15.46:0/3891496855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe80c6520 msgr2=0x7fabe81516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.774+0000 7fabed484700 1 --2- 172.21.15.46:0/3891496855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe80c6520 0x7fabe81516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:05.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.774+0000 7fabed484700 1 -- 172.21.15.46:0/3891496855 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabdc009420 con 0x7fabe80ced50 2024-01-27T00:47:05.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.774+0000 7fabecc83700 1 --2- 172.21.15.46:0/3891496855 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fabe80b2b40 0x7fabe80b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:05.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.774+0000 7fabed484700 1 --2- 172.21.15.46:0/3891496855 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabe80ced50 0x7fabe80b0300 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fabe000c7c0 tx=0x7fabe0007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:05.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.774+0000 7fabe4ff9700 1 -- 172.21.15.46:0/3891496855 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0012630 con 0x7fabe80ced50 2024-01-27T00:47:05.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.774+0000 7fabe4ff9700 1 -- 172.21.15.46:0/3891496855 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe0012790 con 0x7fabe80ced50 2024-01-27T00:47:05.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.774+0000 7fabe4ff9700 1 -- 172.21.15.46:0/3891496855 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0012b50 con 0x7fabe80ced50 2024-01-27T00:47:05.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.774+0000 7fabeeee7700 1 -- 172.21.15.46:0/3891496855 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe816da20 con 0x7fabe80ced50 2024-01-27T00:47:05.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.774+0000 7fabeeee7700 1 -- 172.21.15.46:0/3891496855 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe816dfe0 con 0x7fabe80ced50 2024-01-27T00:47:05.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.775+0000 7fabeeee7700 1 -- 172.21.15.46:0/3891496855 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe80c7800 con 0x7fabe80ced50 2024-01-27T00:47:05.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.776+0000 7fabe4ff9700 1 -- 172.21.15.46:0/3891496855 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 92) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fabe001b400 con 0x7fabe80ced50 2024-01-27T00:47:05.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.776+0000 7fabe4ff9700 1 --2- 172.21.15.46:0/3891496855 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fabd4067050 0x7fabd4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:05.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.776+0000 7fabecc83700 1 -- 172.21.15.46:0/3891496855 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fabd4067050 msgr2=0x7fabd4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:05.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.776+0000 7fabe4ff9700 1 -- 172.21.15.46:0/3891496855 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fabe00849b0 con 0x7fabe80ced50 2024-01-27T00:47:05.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.776+0000 7fabecc83700 1 --2- 172.21.15.46:0/3891496855 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fabd4067050 0x7fabd4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:47:05.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.780+0000 7fabe4ff9700 1 -- 172.21.15.46:0/3891496855 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fabe0051720 con 0x7fabe80ced50 2024-01-27T00:47:05.862 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:05.862+0000 7fd1097bb1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T00:47:05.947 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:05.947+0000 7fd1097bb1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T00:47:05.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.977+0000 7fabecc83700 1 -- 172.21.15.46:0/3891496855 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fabd4067050 msgr2=0x7fabd4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:05.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:05.977+0000 7fabecc83700 1 --2- 172.21.15.46:0/3891496855 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fabd4067050 0x7fabd4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:47:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.032+0000 7fabeeee7700 1 -- 172.21.15.46:0/3891496855 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabe804c710 con 0x7fabe80ced50 2024-01-27T00:47:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.032+0000 7fabe4ff9700 1 -- 172.21.15.46:0/3891496855 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4607 (secure 0 0 0) 0x7fabe0054d40 con 0x7fabe80ced50 2024-01-27T00:47:06.033 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:06.033 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":88,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9061,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/3698053981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":3698053981},{"type":"v1","addr":"172.21.15.106:6833","nonce":3698053981}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":9064,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2056976879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2056976879},{"type":"v1","addr":"172.21.15.46:6834","nonce":2056976879}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":82},{"gid":9067,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1576406314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1576406314},{"type":"v1","addr":"172.21.15.46:6835","nonce":1576406314}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T00:46:44.978654+0000","modified":"2024-01-27T00:46:48.558026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9070":{"gid":9070,"name":"d","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6836/2857098625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2857098625},{"type":"v1","addr":"172.21.15.106:6836","nonce":2857098625}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2024-01-27T00:47:06.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.035+0000 7fabeeee7700 1 -- 172.21.15.46:0/3891496855 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fabd4067050 msgr2=0x7fabd4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:47:06.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.035+0000 7fabeeee7700 1 --2- 172.21.15.46:0/3891496855 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fabd4067050 0x7fabd4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.035+0000 7fabeeee7700 1 -- 172.21.15.46:0/3891496855 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabe80ced50 msgr2=0x7fabe80b0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.035+0000 7fabeeee7700 1 --2- 172.21.15.46:0/3891496855 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabe80ced50 0x7fabe80b0300 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fabe000c7c0 tx=0x7fabe0007f90 comp rx=0 tx=0).stop 2024-01-27T00:47:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.035+0000 7fabeeee7700 1 -- 172.21.15.46:0/3891496855 shutdown_connections 2024-01-27T00:47:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.035+0000 7fabeeee7700 1 --2- 172.21.15.46:0/3891496855 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fabd4067050 0x7fabd4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.035+0000 7fabeeee7700 1 --2- 172.21.15.46:0/3891496855 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabe80ced50 0x7fabe80b0300 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.035+0000 7fabeeee7700 1 --2- 172.21.15.46:0/3891496855 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fabe80b2b40 0x7fabe80b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.035+0000 7fabeeee7700 1 --2- 172.21.15.46:0/3891496855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe80c6520 0x7fabe81516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.035+0000 7fabeeee7700 1 -- 172.21.15.46:0/3891496855 >> 172.21.15.46:0/3891496855 conn(0x7fabe8099a30 msgr2=0x7fabe8158f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.035+0000 7fabeeee7700 1 -- 172.21.15.46:0/3891496855 shutdown_connections 2024-01-27T00:47:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.035+0000 7fabeeee7700 1 -- 172.21.15.46:0/3891496855 wait complete. 2024-01-27T00:47:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 88 2024-01-27T00:47:06.049 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T00:47:06.091 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:06.091+0000 7fd1097bb1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T00:47:06.098 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:06.099+0000 7f573af0c1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T00:47:06.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.181+0000 7f036b4fb700 1 Processor -- start 2024-01-27T00:47:06.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.181+0000 7f036b4fb700 1 -- start start 2024-01-27T00:47:06.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.183+0000 7f036b4fb700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f036415d260 0x7f03640b3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:06.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.183+0000 7f036b4fb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f03640cf4c0 0x7f03640cc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:06.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.183+0000 7f036b4fb700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03640c6290 con 0x7f036415ce90 2024-01-27T00:47:06.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.183+0000 7f036b4fb700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03640c5680 con 0x7f036415d260 2024-01-27T00:47:06.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.183+0000 7f036b4fb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03640c5960 con 0x7f03640cf4c0 2024-01-27T00:47:06.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.183+0000 7f0369297700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f036415ce90 0x7f03640cedb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37196/0 (socket says 172.21.15.46:37196) 2024-01-27T00:47:06.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.183+0000 7f0369297700 1 -- 172.21.15.46:0/629343599 learned_addr learned my addr 172.21.15.46:0/629343599 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:06.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.183+0000 7f0369a98700 1 --2- 172.21.15.46:0/629343599 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f03640cf4c0 0x7f03640cc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:06.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.183+0000 7f0368a96700 1 --2- 172.21.15.46:0/629343599 >> v2:172.21.15.106:3300/0 conn(0x7f036415d260 0x7f03640b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:06.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.183+0000 7f035bfff700 1 -- 172.21.15.46:0/629343599 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039078657 0 0) 0x7f03640c6290 con 0x7f036415ce90 2024-01-27T00:47:06.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.183+0000 7f035bfff700 1 -- 172.21.15.46:0/629343599 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0350003570 con 0x7f036415ce90 2024-01-27T00:47:06.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.184+0000 7f0369a98700 1 -- 172.21.15.46:0/629343599 >> v2:172.21.15.106:3300/0 conn(0x7f036415d260 msgr2=0x7f03640b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:06.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.184+0000 7f0369a98700 1 --2- 172.21.15.46:0/629343599 >> v2:172.21.15.106:3300/0 conn(0x7f036415d260 0x7f03640b3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.184+0000 7f0369a98700 1 -- 172.21.15.46:0/629343599 >> v1:172.21.15.46:6789/0 conn(0x7f036415ce90 legacy=0x7f03640cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:06.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.184+0000 7f0369a98700 1 -- 172.21.15.46:0/629343599 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03640c5c40 con 0x7f03640cf4c0 2024-01-27T00:47:06.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.184+0000 7f0368a96700 1 --2- 172.21.15.46:0/629343599 >> v2:172.21.15.106:3300/0 conn(0x7f036415d260 0x7f03640b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:06.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.184+0000 7f0369a98700 1 --2- 172.21.15.46:0/629343599 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f03640cf4c0 0x7f03640cc570 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f035c0098e0 tx=0x7f035c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c2c82b18bc05303 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:06.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.184+0000 7f035bfff700 1 -- 172.21.15.46:0/629343599 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c011630 con 0x7f03640cf4c0 2024-01-27T00:47:06.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.184+0000 7f035bfff700 1 -- 172.21.15.46:0/629343599 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035c011790 con 0x7f03640cf4c0 2024-01-27T00:47:06.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.184+0000 7f035bfff700 1 -- 172.21.15.46:0/629343599 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c011b30 con 0x7f03640cf4c0 2024-01-27T00:47:06.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.185+0000 7f036b4fb700 1 -- 172.21.15.46:0/629343599 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f03640cf4c0 msgr2=0x7f03640cc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:06.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.185+0000 7f036b4fb700 1 --2- 172.21.15.46:0/629343599 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f03640cf4c0 0x7f03640cc570 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f035c0098e0 tx=0x7f035c0091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.185+0000 7f036b4fb700 1 -- 172.21.15.46:0/629343599 shutdown_connections 2024-01-27T00:47:06.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.185+0000 7f036b4fb700 1 --2- 172.21.15.46:0/629343599 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f03640cf4c0 0x7f03640cc570 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.185+0000 7f036b4fb700 1 --2- 172.21.15.46:0/629343599 >> v2:172.21.15.106:3300/0 conn(0x7f036415d260 0x7f03640b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.185+0000 7f036b4fb700 1 -- 172.21.15.46:0/629343599 >> 172.21.15.46:0/629343599 conn(0x7f0364099a70 msgr2=0x7f036406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:06.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.185+0000 7f036b4fb700 1 -- 172.21.15.46:0/629343599 shutdown_connections 2024-01-27T00:47:06.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.185+0000 7f036b4fb700 1 -- 172.21.15.46:0/629343599 wait complete. 2024-01-27T00:47:06.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.186+0000 7f036b4fb700 1 Processor -- start 2024-01-27T00:47:06.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.186+0000 7f036b4fb700 1 -- start start 2024-01-27T00:47:06.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f036b4fb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03640cf4c0 0x7f03640b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:06.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f036b4fb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f036415ce90 0x7f036409dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:06.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f036b4fb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f036415d260 0x7f03640c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:06.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f036b4fb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03640b5480 con 0x7f036415ce90 2024-01-27T00:47:06.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f036b4fb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03640b5e50 con 0x7f03640cf4c0 2024-01-27T00:47:06.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f036b4fb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03640b4870 con 0x7f036415d260 2024-01-27T00:47:06.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f0368a96700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f036415ce90 0x7f036409dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:06.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f0368a96700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f036415ce90 0x7f036409dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38978/0 (socket says 172.21.15.46:38978) 2024-01-27T00:47:06.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f0368a96700 1 -- 172.21.15.46:0/248571628 learned_addr learned my addr 172.21.15.46:0/248571628 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:06.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f0369a98700 1 --2- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f036415d260 0x7f03640c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:06.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f0369297700 1 --2- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03640cf4c0 0x7f03640b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:06.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f0368a96700 1 -- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f036415d260 msgr2=0x7f03640c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f0368a96700 1 --2- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f036415d260 0x7f03640c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f0368a96700 1 -- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03640cf4c0 msgr2=0x7f03640b5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f0368a96700 1 --2- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03640cf4c0 0x7f03640b5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f0368a96700 1 -- 172.21.15.46:0/248571628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0354009420 con 0x7f036415ce90 2024-01-27T00:47:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f0369a98700 1 --2- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f036415d260 0x7f03640c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f0369297700 1 --2- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03640cf4c0 0x7f03640b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.188+0000 7f0368a96700 1 --2- 172.21.15.46:0/248571628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f036415ce90 0x7f036409dbc0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f035400ec50 tx=0x7f03540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.189+0000 7f03597fa700 1 -- 172.21.15.46:0/248571628 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035400c850 con 0x7f036415ce90 2024-01-27T00:47:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.189+0000 7f03597fa700 1 -- 172.21.15.46:0/248571628 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035400c9b0 con 0x7f036415ce90 2024-01-27T00:47:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.189+0000 7f03597fa700 1 -- 172.21.15.46:0/248571628 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035400cd50 con 0x7f036415ce90 2024-01-27T00:47:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.189+0000 7f036b4fb700 1 -- 172.21.15.46:0/248571628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f035c009420 con 0x7f036415ce90 2024-01-27T00:47:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.189+0000 7f036b4fb700 1 -- 172.21.15.46:0/248571628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03640b4e30 con 0x7f036415ce90 2024-01-27T00:47:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.190+0000 7f036b4fb700 1 -- 172.21.15.46:0/248571628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f036404c710 con 0x7f036415ce90 2024-01-27T00:47:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.190+0000 7f03597fa700 1 -- 172.21.15.46:0/248571628 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 92) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f0354013680 con 0x7f036415ce90 2024-01-27T00:47:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.191+0000 7f03597fa700 1 --2- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0348067050 0x7f0348069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:06.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.191+0000 7f03597fa700 1 -- 172.21.15.46:0/248571628 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f0354084d30 con 0x7f036415ce90 2024-01-27T00:47:06.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.191+0000 7f0369297700 1 -- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0348067050 msgr2=0x7f0348069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:06.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.191+0000 7f0369297700 1 --2- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0348067050 0x7f0348069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:47:06.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.195+0000 7f03597fa700 1 -- 172.21.15.46:0/248571628 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0354051aa0 con 0x7f036415ce90 2024-01-27T00:47:06.246 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:06.246+0000 7f573af0c1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T00:47:06.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.392+0000 7f0369297700 1 -- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0348067050 msgr2=0x7f0348069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:06.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.392+0000 7f0369297700 1 --2- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0348067050 0x7f0348069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:47:06.395 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:06.395+0000 7f573af0c1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T00:47:06.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.473+0000 7f036b4fb700 1 -- 172.21.15.46:0/248571628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03640b55c0 con 0x7f036415ce90 2024-01-27T00:47:06.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.474+0000 7f03597fa700 1 -- 172.21.15.46:0/248571628 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4607 (secure 0 0 0) 0x7f03540550c0 con 0x7f036415ce90 2024-01-27T00:47:06.474 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:06.474 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":88,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9061,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/3698053981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":3698053981},{"type":"v1","addr":"172.21.15.106:6833","nonce":3698053981}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":9064,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2056976879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2056976879},{"type":"v1","addr":"172.21.15.46:6834","nonce":2056976879}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":82},{"gid":9067,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1576406314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1576406314},{"type":"v1","addr":"172.21.15.46:6835","nonce":1576406314}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T00:46:44.978654+0000","modified":"2024-01-27T00:46:48.558026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9070":{"gid":9070,"name":"d","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6836/2857098625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":2857098625},{"type":"v1","addr":"172.21.15.106:6836","nonce":2857098625}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2024-01-27T00:47:06.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.476+0000 7f036b4fb700 1 -- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0348067050 msgr2=0x7f0348069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:47:06.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.476+0000 7f036b4fb700 1 --2- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0348067050 0x7f0348069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.476+0000 7f036b4fb700 1 -- 172.21.15.46:0/248571628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f036415ce90 msgr2=0x7f036409dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:06.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.476+0000 7f036b4fb700 1 --2- 172.21.15.46:0/248571628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f036415ce90 0x7f036409dbc0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f035400ec50 tx=0x7f03540091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.476+0000 7f036b4fb700 1 -- 172.21.15.46:0/248571628 shutdown_connections 2024-01-27T00:47:06.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.476+0000 7f036b4fb700 1 --2- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f0348067050 0x7f0348069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.476+0000 7f036b4fb700 1 --2- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f036415d260 0x7f03640c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.476+0000 7f036b4fb700 1 --2- 172.21.15.46:0/248571628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03640cf4c0 0x7f03640b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.476+0000 7f036b4fb700 1 --2- 172.21.15.46:0/248571628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f036415ce90 0x7f036409dbc0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.476+0000 7f036b4fb700 1 -- 172.21.15.46:0/248571628 >> 172.21.15.46:0/248571628 conn(0x7f0364099a70 msgr2=0x7f03640c6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:06.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.477+0000 7f036b4fb700 1 -- 172.21.15.46:0/248571628 shutdown_connections 2024-01-27T00:47:06.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.477+0000 7f036b4fb700 1 -- 172.21.15.46:0/248571628 wait complete. 2024-01-27T00:47:06.478 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 88 2024-01-27T00:47:06.491 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T00:47:06.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.668+0000 7fbafeeab700 1 Processor -- start 2024-01-27T00:47:06.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.669+0000 7fbafeeab700 1 -- start start 2024-01-27T00:47:06.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.669+0000 7fbafeeab700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbaf815ce40 0x7fbaf80d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:06.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.669+0000 7fbafeeab700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbaf80cc1c0 0x7fbaf80ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:06.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.669+0000 7fbafeeab700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf80cff70 con 0x7fbaf815d210 2024-01-27T00:47:06.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.669+0000 7fbafeeab700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf80cf400 con 0x7fbaf815ce40 2024-01-27T00:47:06.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.669+0000 7fbafeeab700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf80cf660 con 0x7fbaf80cc1c0 2024-01-27T00:47:06.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.670+0000 7fbaf7fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbaf815d210 0x7fbaf80cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37200/0 (socket says 172.21.15.46:37200) 2024-01-27T00:47:06.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.670+0000 7fbaf7fff700 1 -- 172.21.15.46:0/2096616493 learned_addr learned my addr 172.21.15.46:0/2096616493 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:06.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.670+0000 7fbafcc47700 1 --2- 172.21.15.46:0/2096616493 >> v2:172.21.15.106:3300/0 conn(0x7fbaf815ce40 0x7fbaf80d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:06.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.670+0000 7fbafd448700 1 --2- 172.21.15.46:0/2096616493 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbaf80cc1c0 0x7fbaf80ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:06.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.670+0000 7fbaf77fe700 1 -- 172.21.15.46:0/2096616493 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031714933 0 0) 0x7fbaf80cff70 con 0x7fbaf815d210 2024-01-27T00:47:06.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.670+0000 7fbaf77fe700 1 -- 172.21.15.46:0/2096616493 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae4003570 con 0x7fbaf815d210 2024-01-27T00:47:06.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.670+0000 7fbafd448700 1 -- 172.21.15.46:0/2096616493 >> v2:172.21.15.106:3300/0 conn(0x7fbaf815ce40 msgr2=0x7fbaf80d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:06.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.670+0000 7fbafd448700 1 --2- 172.21.15.46:0/2096616493 >> v2:172.21.15.106:3300/0 conn(0x7fbaf815ce40 0x7fbaf80d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.670+0000 7fbafd448700 1 -- 172.21.15.46:0/2096616493 >> v1:172.21.15.46:6789/0 conn(0x7fbaf815d210 legacy=0x7fbaf80cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.670+0000 7fbafd448700 1 -- 172.21.15.46:0/2096616493 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf80d0500 con 0x7fbaf80cc1c0 2024-01-27T00:47:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.670+0000 7fbafd448700 1 --2- 172.21.15.46:0/2096616493 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbaf80cc1c0 0x7fbaf80ade30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0009e60 tx=0x7fbaf0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c15e7e6b402575d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.670+0000 7fbaf77fe700 1 -- 172.21.15.46:0/2096616493 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf000e070 con 0x7fbaf80cc1c0 2024-01-27T00:47:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.671+0000 7fbaf77fe700 1 -- 172.21.15.46:0/2096616493 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf000bab0 con 0x7fbaf80cc1c0 2024-01-27T00:47:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.671+0000 7fbaf77fe700 1 -- 172.21.15.46:0/2096616493 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf000be70 con 0x7fbaf80cc1c0 2024-01-27T00:47:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.671+0000 7fbafeeab700 1 -- 172.21.15.46:0/2096616493 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbaf80cc1c0 msgr2=0x7fbaf80ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.671+0000 7fbafeeab700 1 --2- 172.21.15.46:0/2096616493 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbaf80cc1c0 0x7fbaf80ade30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0009e60 tx=0x7fbaf0009780 comp rx=0 tx=0).stop 2024-01-27T00:47:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.671+0000 7fbafeeab700 1 -- 172.21.15.46:0/2096616493 shutdown_connections 2024-01-27T00:47:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.671+0000 7fbafeeab700 1 --2- 172.21.15.46:0/2096616493 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbaf80cc1c0 0x7fbaf80ade30 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.671+0000 7fbafeeab700 1 --2- 172.21.15.46:0/2096616493 >> v2:172.21.15.106:3300/0 conn(0x7fbaf815ce40 0x7fbaf80d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.671+0000 7fbafeeab700 1 -- 172.21.15.46:0/2096616493 >> 172.21.15.46:0/2096616493 conn(0x7fbaf806f840 msgr2=0x7fbaf80d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.671+0000 7fbafeeab700 1 -- 172.21.15.46:0/2096616493 shutdown_connections 2024-01-27T00:47:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.671+0000 7fbafeeab700 1 -- 172.21.15.46:0/2096616493 wait complete. 2024-01-27T00:47:06.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.673+0000 7fbafeeab700 1 Processor -- start 2024-01-27T00:47:06.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.673+0000 7fbafeeab700 1 -- start start 2024-01-27T00:47:06.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.674+0000 7fbafeeab700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbaf80cc1c0 0x7fbaf815fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:06.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.674+0000 7fbafeeab700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbaf815ce40 0x7fbaf809d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:06.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.674+0000 7fbafeeab700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbaf815d210 0x7fbaf816a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:06.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.674+0000 7fbafeeab700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf809dbd0 con 0x7fbaf815d210 2024-01-27T00:47:06.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.674+0000 7fbafeeab700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf815f730 con 0x7fbaf80cc1c0 2024-01-27T00:47:06.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.674+0000 7fbafeeab700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf816adb0 con 0x7fbaf815ce40 2024-01-27T00:47:06.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.674+0000 7fbafd448700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbaf815d210 0x7fbaf816a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:06.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.674+0000 7fbafcc47700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbaf80cc1c0 0x7fbaf815fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:06.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.674+0000 7fbafd448700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbaf815d210 0x7fbaf816a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38992/0 (socket says 172.21.15.46:38992) 2024-01-27T00:47:06.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.674+0000 7fbafd448700 1 -- 172.21.15.46:0/3008682763 learned_addr learned my addr 172.21.15.46:0/3008682763 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:06.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.674+0000 7fbaf7fff700 1 --2- 172.21.15.46:0/3008682763 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbaf815ce40 0x7fbaf809d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:06.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.675+0000 7fbafd448700 1 -- 172.21.15.46:0/3008682763 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbaf815ce40 msgr2=0x7fbaf809d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:06.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.675+0000 7fbafd448700 1 --2- 172.21.15.46:0/3008682763 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbaf815ce40 0x7fbaf809d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.675+0000 7fbafd448700 1 -- 172.21.15.46:0/3008682763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbaf80cc1c0 msgr2=0x7fbaf815fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:06.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.675+0000 7fbafd448700 1 --2- 172.21.15.46:0/3008682763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbaf80cc1c0 0x7fbaf815fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.675+0000 7fbafd448700 1 -- 172.21.15.46:0/3008682763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf0009a20 con 0x7fbaf815d210 2024-01-27T00:47:06.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.675+0000 7fbafd448700 1 --2- 172.21.15.46:0/3008682763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbaf815d210 0x7fbaf816a870 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80cf7a0 tx=0x7fbaf0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:06.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.675+0000 7fbaf4ff9700 1 -- 172.21.15.46:0/3008682763 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0012dd0 con 0x7fbaf815d210 2024-01-27T00:47:06.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.675+0000 7fbaf4ff9700 1 -- 172.21.15.46:0/3008682763 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf0027470 con 0x7fbaf815d210 2024-01-27T00:47:06.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.675+0000 7fbafeeab700 1 -- 172.21.15.46:0/3008682763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf815eb20 con 0x7fbaf815d210 2024-01-27T00:47:06.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.675+0000 7fbaf4ff9700 1 -- 172.21.15.46:0/3008682763 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0027830 con 0x7fbaf815d210 2024-01-27T00:47:06.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.675+0000 7fbafeeab700 1 -- 172.21.15.46:0/3008682763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf815f0e0 con 0x7fbaf815d210 2024-01-27T00:47:06.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.676+0000 7fbafeeab700 1 -- 172.21.15.46:0/3008682763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf815f8e0 con 0x7fbaf815d210 2024-01-27T00:47:06.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.677+0000 7fbaf4ff9700 1 -- 172.21.15.46:0/3008682763 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 92) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fbaf0024070 con 0x7fbaf815d210 2024-01-27T00:47:06.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.677+0000 7fbaf4ff9700 1 --2- 172.21.15.46:0/3008682763 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbad8067050 0x7fbad8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:06.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.677+0000 7fbafcc47700 1 -- 172.21.15.46:0/3008682763 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbad8067050 msgr2=0x7fbad8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:06.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.677+0000 7fbaf4ff9700 1 -- 172.21.15.46:0/3008682763 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fbaf0087440 con 0x7fbaf815d210 2024-01-27T00:47:06.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.677+0000 7fbafcc47700 1 --2- 172.21.15.46:0/3008682763 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbad8067050 0x7fbad8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:47:06.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.682+0000 7fbaf4ff9700 1 -- 172.21.15.46:0/3008682763 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbaf815f8e0 con 0x7fbaf815d210 2024-01-27T00:47:06.733 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:06.733+0000 7fd1097bb1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T00:47:06.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.878+0000 7fbafcc47700 1 -- 172.21.15.46:0/3008682763 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbad8067050 msgr2=0x7fbad8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:06.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.878+0000 7fbafcc47700 1 --2- 172.21.15.46:0/3008682763 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbad8067050 0x7fbad8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:47:06.910 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:06.910+0000 7fd1097bb1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T00:47:06.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.921+0000 7fbafeeab700 1 -- 172.21.15.46:0/3008682763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbaf815f8e0 con 0x7fbaf815d210 2024-01-27T00:47:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.922+0000 7fbaf4ff9700 1 -- 172.21.15.46:0/3008682763 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v176) v1 ==== 74+0+16607 (secure 0 0 0) 0x7fbaf815f8e0 con 0x7fbaf815d210 2024-01-27T00:47:06.923 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:06.923 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":176,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T00:46:59.569524+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-01-27T00:46:43.494684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-01-27T00:46:43.949785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/2302701803":"2024-01-28T00:46:59.569484+0000","172.21.15.106:0/3368647852":"2024-01-28T00:46:59.569484+0000","172.21.15.106:6837/43388":"2024-01-28T00:46:59.569484+0000","172.21.15.106:6835/43388":"2024-01-28T00:46:59.569484+0000","172.21.15.106:0/3900909328":"2024-01-28T00:46:59.569484+0000","172.21.15.106:0/4251291402":"2024-01-28T00:46:59.569484+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T00:47:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.924+0000 7fbafeeab700 1 -- 172.21.15.46:0/3008682763 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbad8067050 msgr2=0x7fbad8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:47:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.924+0000 7fbafeeab700 1 --2- 172.21.15.46:0/3008682763 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbad8067050 0x7fbad8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.924+0000 7fbafeeab700 1 -- 172.21.15.46:0/3008682763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbaf815d210 msgr2=0x7fbaf816a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.924+0000 7fbafeeab700 1 --2- 172.21.15.46:0/3008682763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbaf815d210 0x7fbaf816a870 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80cf7a0 tx=0x7fbaf0019000 comp rx=0 tx=0).stop 2024-01-27T00:47:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.924+0000 7fbafeeab700 1 -- 172.21.15.46:0/3008682763 shutdown_connections 2024-01-27T00:47:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.924+0000 7fbafeeab700 1 --2- 172.21.15.46:0/3008682763 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7fbad8067050 0x7fbad8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.924+0000 7fbafeeab700 1 --2- 172.21.15.46:0/3008682763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbaf815d210 0x7fbaf816a870 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.924+0000 7fbafeeab700 1 --2- 172.21.15.46:0/3008682763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbaf80cc1c0 0x7fbaf815fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.924+0000 7fbafeeab700 1 --2- 172.21.15.46:0/3008682763 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbaf815ce40 0x7fbaf809d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:06.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.924+0000 7fbafeeab700 1 -- 172.21.15.46:0/3008682763 >> 172.21.15.46:0/3008682763 conn(0x7fbaf806f840 msgr2=0x7fbaf80b5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:06.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.924+0000 7fbafeeab700 1 -- 172.21.15.46:0/3008682763 shutdown_connections 2024-01-27T00:47:06.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:06.924+0000 7fbafeeab700 1 -- 172.21.15.46:0/3008682763 wait complete. 2024-01-27T00:47:06.940 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-27T00:47:07.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.073+0000 7f16072be700 1 Processor -- start 2024-01-27T00:47:07.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.073+0000 7f16072be700 1 -- start start 2024-01-27T00:47:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.074+0000 7f16072be700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16000c3cf0 0x7f16000ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.074+0000 7f16072be700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16000cd080 0x7f16000ab1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.074+0000 7f16072be700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16000cbf10 con 0x7f16000acf30 2024-01-27T00:47:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.074+0000 7f16072be700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16000ada10 con 0x7f16000c3cf0 2024-01-27T00:47:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.074+0000 7f16072be700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1600157be0 con 0x7f16000cd080 2024-01-27T00:47:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.075+0000 7f160505a700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f16000acf30 0x7f16000ad300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37204/0 (socket says 172.21.15.46:37204) 2024-01-27T00:47:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.075+0000 7f160505a700 1 -- 172.21.15.46:0/304261058 learned_addr learned my addr 172.21.15.46:0/304261058 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.075+0000 7f1604859700 1 --2- 172.21.15.46:0/304261058 >> v2:172.21.15.106:3300/0 conn(0x7f16000c3cf0 0x7f16000ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.075+0000 7f160585b700 1 --2- 172.21.15.46:0/304261058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16000cd080 0x7f16000ab1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.075+0000 7f15f7fff700 1 -- 172.21.15.46:0/304261058 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912390347 0 0) 0x7f16000cbf10 con 0x7f16000acf30 2024-01-27T00:47:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.075+0000 7f15f7fff700 1 -- 172.21.15.46:0/304261058 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15ec003570 con 0x7f16000acf30 2024-01-27T00:47:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.075+0000 7f160585b700 1 -- 172.21.15.46:0/304261058 >> v2:172.21.15.106:3300/0 conn(0x7f16000c3cf0 msgr2=0x7f16000ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.075+0000 7f160585b700 1 --2- 172.21.15.46:0/304261058 >> v2:172.21.15.106:3300/0 conn(0x7f16000c3cf0 0x7f16000ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.075+0000 7f160585b700 1 -- 172.21.15.46:0/304261058 >> v1:172.21.15.46:6789/0 conn(0x7f16000acf30 legacy=0x7f16000ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.075+0000 7f160585b700 1 -- 172.21.15.46:0/304261058 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16000cd480 con 0x7f16000cd080 2024-01-27T00:47:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.075+0000 7f1604859700 1 --2- 172.21.15.46:0/304261058 >> v2:172.21.15.106:3300/0 conn(0x7f16000c3cf0 0x7f16000ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.075+0000 7f160585b700 1 --2- 172.21.15.46:0/304261058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16000cd080 0x7f16000ab1b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f15f8009e60 tx=0x7f15f8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ad44954c2e5c9df server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.076+0000 7f15f7fff700 1 -- 172.21.15.46:0/304261058 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f800e070 con 0x7f16000cd080 2024-01-27T00:47:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.076+0000 7f15f7fff700 1 -- 172.21.15.46:0/304261058 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f800bab0 con 0x7f16000cd080 2024-01-27T00:47:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.076+0000 7f15f7fff700 1 -- 172.21.15.46:0/304261058 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f800be70 con 0x7f16000cd080 2024-01-27T00:47:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.076+0000 7f16072be700 1 -- 172.21.15.46:0/304261058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16000cd080 msgr2=0x7f16000ab1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.076+0000 7f16072be700 1 --2- 172.21.15.46:0/304261058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16000cd080 0x7f16000ab1b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f15f8009e60 tx=0x7f15f8009780 comp rx=0 tx=0).stop 2024-01-27T00:47:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.076+0000 7f16072be700 1 -- 172.21.15.46:0/304261058 shutdown_connections 2024-01-27T00:47:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.076+0000 7f16072be700 1 --2- 172.21.15.46:0/304261058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16000cd080 0x7f16000ab1b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.076+0000 7f16072be700 1 --2- 172.21.15.46:0/304261058 >> v2:172.21.15.106:3300/0 conn(0x7f16000c3cf0 0x7f16000ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.076+0000 7f16072be700 1 -- 172.21.15.46:0/304261058 >> 172.21.15.46:0/304261058 conn(0x7f1600097820 msgr2=0x7f1600068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:07.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.076+0000 7f16072be700 1 -- 172.21.15.46:0/304261058 shutdown_connections 2024-01-27T00:47:07.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.076+0000 7f16072be700 1 -- 172.21.15.46:0/304261058 wait complete. 2024-01-27T00:47:07.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.078+0000 7f16072be700 1 Processor -- start 2024-01-27T00:47:07.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.078+0000 7f16072be700 1 -- start start 2024-01-27T00:47:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f16072be700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16000acf30 0x7f1600152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f16072be700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16000c3cf0 0x7f1600155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f16072be700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16000cd080 0x7f16000b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f16072be700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1600155ab0 con 0x7f16000acf30 2024-01-27T00:47:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f16072be700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1600153430 con 0x7f16000cd080 2024-01-27T00:47:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f16072be700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1600154ee0 con 0x7f16000c3cf0 2024-01-27T00:47:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f160505a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16000acf30 0x7f1600152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f160505a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16000acf30 0x7f1600152ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38998/0 (socket says 172.21.15.46:38998) 2024-01-27T00:47:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f160505a700 1 -- 172.21.15.46:0/537474459 learned_addr learned my addr 172.21.15.46:0/537474459 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f1604859700 1 --2- 172.21.15.46:0/537474459 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16000c3cf0 0x7f1600155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f160585b700 1 --2- 172.21.15.46:0/537474459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16000cd080 0x7f16000b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f160505a700 1 -- 172.21.15.46:0/537474459 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16000c3cf0 msgr2=0x7f1600155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f160505a700 1 --2- 172.21.15.46:0/537474459 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16000c3cf0 0x7f1600155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f160505a700 1 -- 172.21.15.46:0/537474459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16000cd080 msgr2=0x7f16000b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f160505a700 1 --2- 172.21.15.46:0/537474459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16000cd080 0x7f16000b12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f160505a700 1 -- 172.21.15.46:0/537474459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f0009420 con 0x7f16000acf30 2024-01-27T00:47:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f160505a700 1 --2- 172.21.15.46:0/537474459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16000acf30 0x7f1600152ef0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f15fc00c7c0 tx=0x7f15fc007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.079+0000 7f15f57fa700 1 -- 172.21.15.46:0/537474459 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15fc005b30 con 0x7f16000acf30 2024-01-27T00:47:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.080+0000 7f15f57fa700 1 -- 172.21.15.46:0/537474459 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15fc005c90 con 0x7f16000acf30 2024-01-27T00:47:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.080+0000 7f16072be700 1 -- 172.21.15.46:0/537474459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f8009a20 con 0x7f16000acf30 2024-01-27T00:47:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.080+0000 7f15f57fa700 1 -- 172.21.15.46:0/537474459 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15fc007300 con 0x7f16000acf30 2024-01-27T00:47:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.080+0000 7f16072be700 1 -- 172.21.15.46:0/537474459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1600155460 con 0x7f16000acf30 2024-01-27T00:47:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.081+0000 7f16072be700 1 -- 172.21.15.46:0/537474459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f160004c710 con 0x7f16000acf30 2024-01-27T00:47:07.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.081+0000 7f15f57fa700 1 -- 172.21.15.46:0/537474459 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 92) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f15fc01b3b0 con 0x7f16000acf30 2024-01-27T00:47:07.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.081+0000 7f15f57fa700 1 --2- 172.21.15.46:0/537474459 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f15e4067050 0x7f15e4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:07.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.082+0000 7f15f57fa700 1 -- 172.21.15.46:0/537474459 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f15fc084040 con 0x7f16000acf30 2024-01-27T00:47:07.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.082+0000 7f1604859700 1 -- 172.21.15.46:0/537474459 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f15e4067050 msgr2=0x7f15e4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:07.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.082+0000 7f1604859700 1 --2- 172.21.15.46:0/537474459 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f15e4067050 0x7f15e4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:47:07.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.086+0000 7f15f57fa700 1 -- 172.21.15.46:0/537474459 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f15fc050db0 con 0x7f16000acf30 2024-01-27T00:47:07.091 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:07.091+0000 7fd1097bb1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T00:47:07.100 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:07.101+0000 7f573af0c1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T00:47:07.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.283+0000 7f1604859700 1 -- 172.21.15.46:0/537474459 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f15e4067050 msgr2=0x7f15e4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6835/43388 2024-01-27T00:47:07.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.283+0000 7f1604859700 1 --2- 172.21.15.46:0/537474459 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f15e4067050 0x7f15e4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:47:07.317 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:07.317+0000 7f573af0c1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:47:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.345+0000 7f16072be700 1 -- 172.21.15.46:0/537474459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f16000c4460 con 0x7f16000acf30 2024-01-27T00:47:07.346 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-27T00:47:07.346+0000 7f33ffe86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-27T00:47:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.350+0000 7f15f57fa700 1 -- 172.21.15.46:0/537474459 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v89) v1 ==== 161+0+0 (secure 0 0 0) 0x7f15fc017090 con 0x7f16000acf30 2024-01-27T00:47:07.351 INFO:tasks.ceph.mds.d.smithi106.stderr: -2341> 2024-01-27T00:46:50.529+0000 7fe3ae9e9700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-27T00:47:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.352+0000 7f16072be700 1 -- 172.21.15.46:0/537474459 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f15e4067050 msgr2=0x7f15e4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:47:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.352+0000 7f16072be700 1 --2- 172.21.15.46:0/537474459 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f15e4067050 0x7f15e4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.352+0000 7f16072be700 1 -- 172.21.15.46:0/537474459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16000acf30 msgr2=0x7f1600152ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.352+0000 7f16072be700 1 --2- 172.21.15.46:0/537474459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16000acf30 0x7f1600152ef0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f15fc00c7c0 tx=0x7f15fc007e00 comp rx=0 tx=0).stop 2024-01-27T00:47:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.352+0000 7f16072be700 1 -- 172.21.15.46:0/537474459 shutdown_connections 2024-01-27T00:47:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.352+0000 7f16072be700 1 --2- 172.21.15.46:0/537474459 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f15e4067050 0x7f15e4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.352+0000 7f16072be700 1 --2- 172.21.15.46:0/537474459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16000acf30 0x7f1600152ef0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.352+0000 7f16072be700 1 --2- 172.21.15.46:0/537474459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16000cd080 0x7f16000b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.352+0000 7f16072be700 1 --2- 172.21.15.46:0/537474459 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16000c3cf0 0x7f1600155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.352+0000 7f16072be700 1 -- 172.21.15.46:0/537474459 >> 172.21.15.46:0/537474459 conn(0x7f1600097820 msgr2=0x7f160006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.352+0000 7f16072be700 1 -- 172.21.15.46:0/537474459 shutdown_connections 2024-01-27T00:47:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.352+0000 7f16072be700 1 -- 172.21.15.46:0/537474459 wait complete. 2024-01-27T00:47:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-27T00:47:07.367 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-27T00:47:07.374 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-01-27T00:47:07.392 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:07.393+0000 7f573af0c1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T00:47:07.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.505+0000 7f44c7900700 1 Processor -- start 2024-01-27T00:47:07.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.505+0000 7f44c7900700 1 -- start start 2024-01-27T00:47:07.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.506+0000 7f44c7900700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f44c0159cb0 0x7f44c00cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:07.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.506+0000 7f44c7900700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f44c00afb40 0x7f44c00c5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:07.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.506+0000 7f44c7900700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c00cc100 con 0x7f44c00cf050 2024-01-27T00:47:07.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.506+0000 7f44c7900700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c00cb850 con 0x7f44c00afb40 2024-01-27T00:47:07.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.506+0000 7f44c7900700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c00cbaf0 con 0x7f44c0159cb0 2024-01-27T00:47:07.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.506+0000 7f44c5e9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f44c00afb40 0x7f44c00c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:07.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.506+0000 7f44c5e9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f44c00afb40 0x7f44c00c5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39012/0 (socket says 172.21.15.46:39012) 2024-01-27T00:47:07.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.506+0000 7f44c5e9d700 1 -- 172.21.15.46:0/411404904 learned_addr learned my addr 172.21.15.46:0/411404904 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:07.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.506+0000 7f44c569c700 1 --2- 172.21.15.46:0/411404904 >> v2:172.21.15.106:3300/0 conn(0x7f44c0159cb0 0x7f44c00cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:07.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.506+0000 7f44c4e9b700 1 --1- 172.21.15.46:0/411404904 >> v1:172.21.15.106:6789/0 conn(0x7f44c00cf050 0x7f44c00af430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:38166/0 (socket says 172.21.15.46:38166) 2024-01-27T00:47:07.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.507+0000 7f44c5e9d700 1 -- 172.21.15.46:0/411404904 >> v2:172.21.15.106:3300/0 conn(0x7f44c0159cb0 msgr2=0x7f44c00cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:07.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.507+0000 7f44c5e9d700 1 --2- 172.21.15.46:0/411404904 >> v2:172.21.15.106:3300/0 conn(0x7f44c0159cb0 0x7f44c00cc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:07.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.507+0000 7f44c5e9d700 1 -- 172.21.15.46:0/411404904 >> v1:172.21.15.106:6789/0 conn(0x7f44c00cf050 legacy=0x7f44c00af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:07.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.507+0000 7f44bffff700 1 -- 172.21.15.46:0/411404904 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116862608 0 0) 0x7f44c00cc100 con 0x7f44c00cf050 2024-01-27T00:47:07.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.507+0000 7f44c5e9d700 1 -- 172.21.15.46:0/411404904 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c00cfc80 con 0x7f44c00afb40 2024-01-27T00:47:07.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.507+0000 7f44c569c700 1 --2- 172.21.15.46:0/411404904 >> v2:172.21.15.106:3300/0 conn(0x7f44c0159cb0 0x7f44c00cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:07.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.507+0000 7f44c5e9d700 1 --2- 172.21.15.46:0/411404904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c00afb40 0x7f44c00c5530 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f44c00b6880 tx=0x7f44b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53abc34b949772bf server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:07.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.507+0000 7f44bffff700 1 -- 172.21.15.46:0/411404904 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b8009e70 con 0x7f44c00afb40 2024-01-27T00:47:07.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.507+0000 7f44bffff700 1 -- 172.21.15.46:0/411404904 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b800b860 con 0x7f44c00afb40 2024-01-27T00:47:07.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.507+0000 7f44bffff700 1 -- 172.21.15.46:0/411404904 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b800bc00 con 0x7f44c00afb40 2024-01-27T00:47:07.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.508+0000 7f44c7900700 1 -- 172.21.15.46:0/411404904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c00afb40 msgr2=0x7f44c00c5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:07.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.508+0000 7f44c7900700 1 --2- 172.21.15.46:0/411404904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c00afb40 0x7f44c00c5530 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f44c00b6880 tx=0x7f44b8009250 comp rx=0 tx=0).stop 2024-01-27T00:47:07.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.508+0000 7f44c7900700 1 -- 172.21.15.46:0/411404904 shutdown_connections 2024-01-27T00:47:07.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.508+0000 7f44c7900700 1 --2- 172.21.15.46:0/411404904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c00afb40 0x7f44c00c5530 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:07.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.508+0000 7f44c7900700 1 --2- 172.21.15.46:0/411404904 >> v2:172.21.15.106:3300/0 conn(0x7f44c0159cb0 0x7f44c00cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:07.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.508+0000 7f44c7900700 1 -- 172.21.15.46:0/411404904 >> 172.21.15.46:0/411404904 conn(0x7f44c0087f60 msgr2=0x7f44c006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:07.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.508+0000 7f44c7900700 1 -- 172.21.15.46:0/411404904 shutdown_connections 2024-01-27T00:47:07.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.508+0000 7f44c7900700 1 -- 172.21.15.46:0/411404904 wait complete. 2024-01-27T00:47:07.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.509+0000 7f44c7900700 1 Processor -- start 2024-01-27T00:47:07.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.510+0000 7f44c7900700 1 -- start start 2024-01-27T00:47:07.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.510+0000 7f44c7900700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44c00afb40 0x7f44c0169f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:07.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c7900700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c00cf050 0x7f44c00b2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:07.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c7900700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44c0159cb0 0x7f44c00aead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:07.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c7900700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c00b2e60 con 0x7f44c00cf050 2024-01-27T00:47:07.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c7900700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c016b370 con 0x7f44c0159cb0 2024-01-27T00:47:07.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c7900700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c016aa40 con 0x7f44c00afb40 2024-01-27T00:47:07.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c4e9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c00cf050 0x7f44c00b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:07.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c4e9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c00cf050 0x7f44c00b2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39024/0 (socket says 172.21.15.46:39024) 2024-01-27T00:47:07.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c4e9b700 1 -- 172.21.15.46:0/2660922973 learned_addr learned my addr 172.21.15.46:0/2660922973 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:07.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c569c700 1 --2- 172.21.15.46:0/2660922973 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44c00afb40 0x7f44c0169f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:07.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c5e9d700 1 --2- 172.21.15.46:0/2660922973 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44c0159cb0 0x7f44c00aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:07.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c4e9b700 1 -- 172.21.15.46:0/2660922973 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44c00afb40 msgr2=0x7f44c0169f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:07.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c4e9b700 1 --2- 172.21.15.46:0/2660922973 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44c00afb40 0x7f44c0169f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:07.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c4e9b700 1 -- 172.21.15.46:0/2660922973 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44c0159cb0 msgr2=0x7f44c00aead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:07.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c4e9b700 1 --2- 172.21.15.46:0/2660922973 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44c0159cb0 0x7f44c00aead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:07.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c4e9b700 1 -- 172.21.15.46:0/2660922973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b0009a20 con 0x7f44c00cf050 2024-01-27T00:47:07.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c569c700 1 --2- 172.21.15.46:0/2660922973 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44c00afb40 0x7f44c0169f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:07.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c5e9d700 1 --2- 172.21.15.46:0/2660922973 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44c0159cb0 0x7f44c00aead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:07.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44c4e9b700 1 --2- 172.21.15.46:0/2660922973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c00cf050 0x7f44c00b2920 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f44b40011e0 tx=0x7f44b4007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:07.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.511+0000 7f44bd7fa700 1 -- 172.21.15.46:0/2660922973 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b4005a70 con 0x7f44c00cf050 2024-01-27T00:47:07.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.512+0000 7f44bd7fa700 1 -- 172.21.15.46:0/2660922973 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b4005bd0 con 0x7f44c00cf050 2024-01-27T00:47:07.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.512+0000 7f44c7900700 1 -- 172.21.15.46:0/2660922973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b80094f0 con 0x7f44c00cf050 2024-01-27T00:47:07.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.512+0000 7f44bd7fa700 1 -- 172.21.15.46:0/2660922973 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b40070d0 con 0x7f44c00cf050 2024-01-27T00:47:07.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.512+0000 7f44c7900700 1 -- 172.21.15.46:0/2660922973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c016b000 con 0x7f44c00cf050 2024-01-27T00:47:07.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.513+0000 7f44c7900700 1 -- 172.21.15.46:0/2660922973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c00638f0 con 0x7f44c00cf050 2024-01-27T00:47:07.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.513+0000 7f44bd7fa700 1 -- 172.21.15.46:0/2660922973 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 92) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f44b4007480 con 0x7f44c00cf050 2024-01-27T00:47:07.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.514+0000 7f44bd7fa700 1 --2- 172.21.15.46:0/2660922973 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f44ac067050 0x7f44ac069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:07.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.514+0000 7f44bd7fa700 1 -- 172.21.15.46:0/2660922973 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f44b4083fa0 con 0x7f44c00cf050 2024-01-27T00:47:07.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.514+0000 7f44c569c700 -1 --2- 172.21.15.46:0/2660922973 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f44ac067050 0x7f44ac069500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] is using msgr V1 protocol 2024-01-27T00:47:07.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.514+0000 7f44c569c700 1 --2- 172.21.15.46:0/2660922973 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f44ac067050 0x7f44ac069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:47:07.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.518+0000 7f44bd7fa700 1 -- 172.21.15.46:0/2660922973 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f44b4050cb0 con 0x7f44c00cf050 2024-01-27T00:47:07.547 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:07.548+0000 7f573af0c1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T00:47:07.549 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:07.549+0000 7f5733c34700 -1 --2- 172.21.15.106:0/43388 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x55a2e3b70000 0x55a2e3b72000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] is using msgr V1 protocol 2024-01-27T00:47:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.556+0000 7f44bd7fa700 1 -- 172.21.15.46:0/2660922973 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 93) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f44b4051080 con 0x7f44c00cf050 2024-01-27T00:47:07.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.557+0000 7f44bd7fa700 1 -- 172.21.15.46:0/2660922973 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f44ac067050 msgr2=0x7f44ac069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:47:07.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.557+0000 7f44bd7fa700 1 --2- 172.21.15.46:0/2660922973 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f44ac067050 0x7f44ac069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:07.618 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:07.617+0000 7f56b6ff5700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:07.618 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:07.617+0000 7f56b6ff5700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:07.618 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:07.617+0000 7f56b6ff5700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:07.618 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:07.617+0000 7f56b6ff5700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:07.618 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:07.617+0000 7f56b6ff5700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:07.618 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:07.617+0000 7f56b9ffb700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:07.618 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:07.617+0000 7f56b9ffb700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:07.618 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:07.617+0000 7f56b9ffb700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:07.618 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:07.617+0000 7f56b9ffb700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:07.618 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:07.617+0000 7f56b9ffb700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:07.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:07.787+0000 7f44c7900700 1 -- 172.21.15.46:0/2660922973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f44c004c710 con 0x7f44c00cf050 2024-01-27T00:47:07.918 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:07.917+0000 7fd1097bb1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T00:47:08.168 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:08.167+0000 7fd1097bb1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:47:08.251 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:08.250+0000 7fd1097bb1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T00:47:08.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.350+0000 7f44bd7fa700 1 -- 172.21.15.46:0/2660922973 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v90) v1 ==== 104+0+0 (secure 0 0 0) 0x7f44b406eac0 con 0x7f44c00cf050 2024-01-27T00:47:08.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.353+0000 7f44c7900700 1 -- 172.21.15.46:0/2660922973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c00cf050 msgr2=0x7f44c00b2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:08.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.353+0000 7f44c7900700 1 --2- 172.21.15.46:0/2660922973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c00cf050 0x7f44c00b2920 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f44b40011e0 tx=0x7f44b4007e10 comp rx=0 tx=0).stop 2024-01-27T00:47:08.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.353+0000 7f44c7900700 1 -- 172.21.15.46:0/2660922973 shutdown_connections 2024-01-27T00:47:08.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.353+0000 7f44c7900700 1 --2- 172.21.15.46:0/2660922973 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x7f44ac067050 0x7f44ac069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:08.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.353+0000 7f44c7900700 1 --2- 172.21.15.46:0/2660922973 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44c0159cb0 0x7f44c00aead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:08.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.353+0000 7f44c7900700 1 --2- 172.21.15.46:0/2660922973 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44c00afb40 0x7f44c0169f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:08.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.353+0000 7f44c7900700 1 --2- 172.21.15.46:0/2660922973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c00cf050 0x7f44c00b2920 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:08.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.353+0000 7f44c7900700 1 -- 172.21.15.46:0/2660922973 >> 172.21.15.46:0/2660922973 conn(0x7f44c0087f60 msgr2=0x7f44c014e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:08.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.353+0000 7f44c7900700 1 -- 172.21.15.46:0/2660922973 shutdown_connections 2024-01-27T00:47:08.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.353+0000 7f44c7900700 1 -- 172.21.15.46:0/2660922973 wait complete. 2024-01-27T00:47:08.367 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-27T00:47:08.432 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:08.431+0000 7fd1097bb1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T00:47:08.445 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:08.444+0000 7fd101ce2700 -1 --2- 172.21.15.46:0/43402 >> [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] conn(0x560486cc4000 0x560486cc6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.106:6835/43388,v1:172.21.15.106:6837/43388] is using msgr V1 protocol 2024-01-27T00:47:08.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.504+0000 7fe5a59e9700 1 Processor -- start 2024-01-27T00:47:08.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.504+0000 7fe5a59e9700 1 -- start start 2024-01-27T00:47:08.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.504+0000 7fe5a59e9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5a0159590 0x7fe5a0159960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:08.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.504+0000 7fe5a59e9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5a00b12e0 0x7fe5a006bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:08.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.504+0000 7fe5a59e9700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a0068c50 con 0x7fe5a006a330 2024-01-27T00:47:08.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.504+0000 7fe5a59e9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a00c98c0 con 0x7fe5a0159590 2024-01-27T00:47:08.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.504+0000 7fe5a59e9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a015dc90 con 0x7fe5a00b12e0 2024-01-27T00:47:08.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.505+0000 7fe59effd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5a0159590 0x7fe5a0159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:08.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.505+0000 7fe59effd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5a0159590 0x7fe5a0159960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39030/0 (socket says 172.21.15.46:39030) 2024-01-27T00:47:08.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.505+0000 7fe59effd700 1 -- 172.21.15.46:0/1596858651 learned_addr learned my addr 172.21.15.46:0/1596858651 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:08.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.505+0000 7fe59f7fe700 1 --2- 172.21.15.46:0/1596858651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5a00b12e0 0x7fe5a006bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:08.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.505+0000 7fe59effd700 1 -- 172.21.15.46:0/1596858651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5a00b12e0 msgr2=0x7fe5a006bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:08.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.505+0000 7fe59effd700 1 --2- 172.21.15.46:0/1596858651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5a00b12e0 0x7fe5a006bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:08.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.505+0000 7fe59effd700 1 -- 172.21.15.46:0/1596858651 >> v1:172.21.15.106:6789/0 conn(0x7fe5a006a330 legacy=0x7fe5a015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:08.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.505+0000 7fe59effd700 1 -- 172.21.15.46:0/1596858651 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a015df70 con 0x7fe5a0159590 2024-01-27T00:47:08.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.505+0000 7fe59effd700 1 --2- 172.21.15.46:0/1596858651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5a0159590 0x7fe5a0159960 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fe5900099b0 tx=0x7fe590009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=741c56e210a882ac server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:08.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.505+0000 7fe59dffb700 1 -- 172.21.15.46:0/1596858651 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe590009ea0 con 0x7fe5a0159590 2024-01-27T00:47:08.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.505+0000 7fe59dffb700 1 -- 172.21.15.46:0/1596858651 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59000b860 con 0x7fe5a0159590 2024-01-27T00:47:08.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.505+0000 7fe59dffb700 1 -- 172.21.15.46:0/1596858651 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59000bc20 con 0x7fe5a0159590 2024-01-27T00:47:08.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.506+0000 7fe5a59e9700 1 -- 172.21.15.46:0/1596858651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5a0159590 msgr2=0x7fe5a0159960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:08.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.506+0000 7fe5a59e9700 1 --2- 172.21.15.46:0/1596858651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5a0159590 0x7fe5a0159960 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fe5900099b0 tx=0x7fe590009250 comp rx=0 tx=0).stop 2024-01-27T00:47:08.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.506+0000 7fe5a59e9700 1 -- 172.21.15.46:0/1596858651 shutdown_connections 2024-01-27T00:47:08.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.506+0000 7fe5a59e9700 1 --2- 172.21.15.46:0/1596858651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5a00b12e0 0x7fe5a006bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:08.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.506+0000 7fe5a59e9700 1 --2- 172.21.15.46:0/1596858651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5a0159590 0x7fe5a0159960 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:08.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.506+0000 7fe5a59e9700 1 -- 172.21.15.46:0/1596858651 >> 172.21.15.46:0/1596858651 conn(0x7fe5a00ce530 msgr2=0x7fe5a00ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:08.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.506+0000 7fe5a59e9700 1 -- 172.21.15.46:0/1596858651 shutdown_connections 2024-01-27T00:47:08.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.506+0000 7fe5a59e9700 1 -- 172.21.15.46:0/1596858651 wait complete. 2024-01-27T00:47:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.507+0000 7fe5a59e9700 1 Processor -- start 2024-01-27T00:47:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.507+0000 7fe5a59e9700 1 -- start start 2024-01-27T00:47:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.508+0000 7fe5a59e9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5a006a330 0x7fe5a016e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.508+0000 7fe5a59e9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5a00b12e0 0x7fe5a00c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.508+0000 7fe5a59e9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5a0159590 0x7fe5a015c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.508+0000 7fe5a59e9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a0159f30 con 0x7fe5a0159590 2024-01-27T00:47:08.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.508+0000 7fe5a59e9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a00c5fa0 con 0x7fe5a006a330 2024-01-27T00:47:08.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.508+0000 7fe5a59e9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a00c6280 con 0x7fe5a00b12e0 2024-01-27T00:47:08.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.508+0000 7fe59f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5a0159590 0x7fe5a015c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:08.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.508+0000 7fe59f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5a0159590 0x7fe5a015c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39038/0 (socket says 172.21.15.46:39038) 2024-01-27T00:47:08.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.508+0000 7fe59f7fe700 1 -- 172.21.15.46:0/4102352208 learned_addr learned my addr 172.21.15.46:0/4102352208 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:08.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.508+0000 7fe59effd700 1 --2- 172.21.15.46:0/4102352208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5a006a330 0x7fe5a016e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:08.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.508+0000 7fe59e7fc700 1 --2- 172.21.15.46:0/4102352208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5a00b12e0 0x7fe5a00c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:08.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.509+0000 7fe59f7fe700 1 -- 172.21.15.46:0/4102352208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5a00b12e0 msgr2=0x7fe5a00c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:08.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.509+0000 7fe59f7fe700 1 --2- 172.21.15.46:0/4102352208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5a00b12e0 0x7fe5a00c5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:08.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.509+0000 7fe59f7fe700 1 -- 172.21.15.46:0/4102352208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5a006a330 msgr2=0x7fe5a016e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:08.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.509+0000 7fe59f7fe700 1 --2- 172.21.15.46:0/4102352208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5a006a330 0x7fe5a016e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:08.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.509+0000 7fe59f7fe700 1 -- 172.21.15.46:0/4102352208 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5900094f0 con 0x7fe5a0159590 2024-01-27T00:47:08.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.509+0000 7fe59f7fe700 1 --2- 172.21.15.46:0/4102352208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5a0159590 0x7fe5a015c770 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fe598009df0 tx=0x7fe59800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:08.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.509+0000 7fe586ffd700 1 -- 172.21.15.46:0/4102352208 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59800caa0 con 0x7fe5a0159590 2024-01-27T00:47:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.509+0000 7fe586ffd700 1 -- 172.21.15.46:0/4102352208 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59800f040 con 0x7fe5a0159590 2024-01-27T00:47:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.509+0000 7fe5a59e9700 1 -- 172.21.15.46:0/4102352208 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a00c6560 con 0x7fe5a0159590 2024-01-27T00:47:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.509+0000 7fe586ffd700 1 -- 172.21.15.46:0/4102352208 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59800ce50 con 0x7fe5a0159590 2024-01-27T00:47:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.509+0000 7fe5a59e9700 1 -- 172.21.15.46:0/4102352208 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a00c5c90 con 0x7fe5a0159590 2024-01-27T00:47:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.510+0000 7fe5a59e9700 1 -- 172.21.15.46:0/4102352208 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a004c710 con 0x7fe5a0159590 2024-01-27T00:47:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.511+0000 7fe586ffd700 1 -- 172.21.15.46:0/4102352208 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 93) v1 ==== 84453+0+0 (secure 0 0 0) 0x7fe598014690 con 0x7fe5a0159590 2024-01-27T00:47:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.511+0000 7fe586ffd700 1 -- 172.21.15.46:0/4102352208 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 7063+0+0 (secure 0 0 0) 0x7fe598085030 con 0x7fe5a0159590 2024-01-27T00:47:08.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.516+0000 7fe586ffd700 1 -- 172.21.15.46:0/4102352208 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe598052110 con 0x7fe5a0159590 2024-01-27T00:47:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.559+0000 7fe586ffd700 1 -- 172.21.15.46:0/4102352208 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 94) v1 ==== 84580+0+0 (secure 0 0 0) 0x7fe598051bc0 con 0x7fe5a0159590 2024-01-27T00:47:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.560+0000 7fe586ffd700 1 --2- 172.21.15.46:0/4102352208 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fe58c06cfb0 0x7fe58c06f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.562+0000 7fe59effd700 1 --2- 172.21.15.46:0/4102352208 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fe58c06cfb0 0x7fe58c06f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:08.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.563+0000 7fe59effd700 1 --2- 172.21.15.46:0/4102352208 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fe58c06cfb0 0x7fe58c06f390 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe5900099b0 tx=0x7fe590017000 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:08.725+0000 7fe5a59e9700 1 -- 172.21.15.46:0/4102352208 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe5a015e250 con 0x7fe5a0159590 2024-01-27T00:47:09.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.635+0000 7fe586ffd700 1 -- 172.21.15.46:0/4102352208 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v179) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe59806ff20 con 0x7fe5a0159590 2024-01-27T00:47:09.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.639+0000 7fe5a59e9700 1 -- 172.21.15.46:0/4102352208 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fe58c06cfb0 msgr2=0x7fe58c06f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:09.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.639+0000 7fe5a59e9700 1 --2- 172.21.15.46:0/4102352208 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fe58c06cfb0 0x7fe58c06f390 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe5900099b0 tx=0x7fe590017000 comp rx=0 tx=0).stop 2024-01-27T00:47:09.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.639+0000 7fe5a59e9700 1 -- 172.21.15.46:0/4102352208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5a0159590 msgr2=0x7fe5a015c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:09.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.639+0000 7fe5a59e9700 1 --2- 172.21.15.46:0/4102352208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5a0159590 0x7fe5a015c770 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fe598009df0 tx=0x7fe59800b040 comp rx=0 tx=0).stop 2024-01-27T00:47:09.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.640+0000 7fe5a59e9700 1 -- 172.21.15.46:0/4102352208 shutdown_connections 2024-01-27T00:47:09.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.640+0000 7fe5a59e9700 1 --2- 172.21.15.46:0/4102352208 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fe58c06cfb0 0x7fe58c06f390 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:09.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.640+0000 7fe5a59e9700 1 --2- 172.21.15.46:0/4102352208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5a0159590 0x7fe5a015c770 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:09.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.640+0000 7fe5a59e9700 1 --2- 172.21.15.46:0/4102352208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5a006a330 0x7fe5a016e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:09.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.640+0000 7fe5a59e9700 1 --2- 172.21.15.46:0/4102352208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5a00b12e0 0x7fe5a00c5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:09.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.640+0000 7fe5a59e9700 1 -- 172.21.15.46:0/4102352208 >> 172.21.15.46:0/4102352208 conn(0x7fe5a00ce530 msgr2=0x7fe5a00cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:09.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.640+0000 7fe5a59e9700 1 -- 172.21.15.46:0/4102352208 shutdown_connections 2024-01-27T00:47:09.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.640+0000 7fe5a59e9700 1 -- 172.21.15.46:0/4102352208 wait complete. 2024-01-27T00:47:09.642 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-01-27T00:47:09.658 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-27T00:47:09.737 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:47:09.740 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:47:09.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.799+0000 7feaa4ebd700 1 Processor -- start 2024-01-27T00:47:09.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.799+0000 7feaa4ebd700 1 -- start start 2024-01-27T00:47:09.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.800+0000 7feaa4ebd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feaa01597e0 0x7feaa00abe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:09.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.800+0000 7feaa4ebd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feaa006ba70 0x7feaa0067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:09.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.800+0000 7feaa4ebd700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa0068a90 con 0x7feaa006aef0 2024-01-27T00:47:09.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.800+0000 7feaa4ebd700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaa00a9af0 con 0x7feaa006ba70 2024-01-27T00:47:09.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.800+0000 7feaa4ebd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaa0068d70 con 0x7feaa01597e0 2024-01-27T00:47:09.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.800+0000 7fea9ed9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feaa006ba70 0x7feaa0067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:09.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.800+0000 7fea9ed9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feaa006ba70 0x7feaa0067930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39052/0 (socket says 172.21.15.46:39052) 2024-01-27T00:47:09.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.800+0000 7fea9ed9d700 1 -- 172.21.15.46:0/157568693 learned_addr learned my addr 172.21.15.46:0/157568693 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:09.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.801+0000 7fea9e59c700 1 --2- 172.21.15.46:0/157568693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feaa01597e0 0x7feaa00abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:09.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.801+0000 7fea9ed9d700 1 -- 172.21.15.46:0/157568693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feaa01597e0 msgr2=0x7feaa00abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:09.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.801+0000 7fea9ed9d700 1 --2- 172.21.15.46:0/157568693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feaa01597e0 0x7feaa00abe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:09.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.801+0000 7fea9ed9d700 1 -- 172.21.15.46:0/157568693 >> v1:172.21.15.106:6789/0 conn(0x7feaa006aef0 legacy=0x7feaa006b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:09.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.801+0000 7fea9ed9d700 1 -- 172.21.15.46:0/157568693 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa0069200 con 0x7feaa006ba70 2024-01-27T00:47:09.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.801+0000 7fea9ed9d700 1 --2- 172.21.15.46:0/157568693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa006ba70 0x7feaa0067930 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fea98009f20 tx=0x7fea98009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3e088f8a5022b27 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:09.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.801+0000 7fea9d59a700 1 -- 172.21.15.46:0/157568693 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9800e070 con 0x7feaa006ba70 2024-01-27T00:47:09.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.801+0000 7fea9d59a700 1 -- 172.21.15.46:0/157568693 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea9800b9d0 con 0x7feaa006ba70 2024-01-27T00:47:09.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.801+0000 7fea9d59a700 1 -- 172.21.15.46:0/157568693 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea980092e0 con 0x7feaa006ba70 2024-01-27T00:47:09.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.802+0000 7feaa4ebd700 1 -- 172.21.15.46:0/157568693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa006ba70 msgr2=0x7feaa0067930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:09.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.802+0000 7feaa4ebd700 1 --2- 172.21.15.46:0/157568693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa006ba70 0x7feaa0067930 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fea98009f20 tx=0x7fea98009840 comp rx=0 tx=0).stop 2024-01-27T00:47:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.802+0000 7feaa4ebd700 1 -- 172.21.15.46:0/157568693 shutdown_connections 2024-01-27T00:47:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.802+0000 7feaa4ebd700 1 --2- 172.21.15.46:0/157568693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa006ba70 0x7feaa0067930 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.802+0000 7feaa4ebd700 1 --2- 172.21.15.46:0/157568693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feaa01597e0 0x7feaa00abe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.802+0000 7feaa4ebd700 1 -- 172.21.15.46:0/157568693 >> 172.21.15.46:0/157568693 conn(0x7feaa006e2c0 msgr2=0x7feaa00ae620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.802+0000 7feaa4ebd700 1 -- 172.21.15.46:0/157568693 shutdown_connections 2024-01-27T00:47:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.802+0000 7feaa4ebd700 1 -- 172.21.15.46:0/157568693 wait complete. 2024-01-27T00:47:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.803+0000 7feaa4ebd700 1 Processor -- start 2024-01-27T00:47:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.803+0000 7feaa4ebd700 1 -- start start 2024-01-27T00:47:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.804+0000 7feaa4ebd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feaa006aef0 0x7feaa01515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:09.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.804+0000 7feaa4ebd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa006ba70 0x7feaa0150150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:09.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.804+0000 7feaa4ebd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feaa01597e0 0x7feaa0153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:09.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.804+0000 7feaa4ebd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa0154440 con 0x7feaa006ba70 2024-01-27T00:47:09.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.804+0000 7feaa4ebd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa0150690 con 0x7feaa01597e0 2024-01-27T00:47:09.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.804+0000 7feaa4ebd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaa013ef20 con 0x7feaa006aef0 2024-01-27T00:47:09.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.804+0000 7fea9dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa006ba70 0x7feaa0150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:09.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.804+0000 7fea9dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa006ba70 0x7feaa0150150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39062/0 (socket says 172.21.15.46:39062) 2024-01-27T00:47:09.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.804+0000 7fea9dd9b700 1 -- 172.21.15.46:0/3239443318 learned_addr learned my addr 172.21.15.46:0/3239443318 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:09.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.804+0000 7fea9e59c700 1 --2- 172.21.15.46:0/3239443318 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feaa006aef0 0x7feaa01515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:09.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.804+0000 7fea9ed9d700 1 --2- 172.21.15.46:0/3239443318 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feaa01597e0 0x7feaa0153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:09.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.804+0000 7fea9dd9b700 1 -- 172.21.15.46:0/3239443318 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feaa006aef0 msgr2=0x7feaa01515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:09.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.804+0000 7fea9dd9b700 1 --2- 172.21.15.46:0/3239443318 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feaa006aef0 0x7feaa01515c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.804+0000 7fea9dd9b700 1 -- 172.21.15.46:0/3239443318 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feaa01597e0 msgr2=0x7feaa0153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.804+0000 7fea9dd9b700 1 --2- 172.21.15.46:0/3239443318 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feaa01597e0 0x7feaa0153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.804+0000 7fea9dd9b700 1 -- 172.21.15.46:0/3239443318 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea98009ae0 con 0x7feaa006ba70 2024-01-27T00:47:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.805+0000 7fea9dd9b700 1 --2- 172.21.15.46:0/3239443318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa006ba70 0x7feaa0150150 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fea8800c590 tx=0x7fea8800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.805+0000 7fea967fc700 1 -- 172.21.15.46:0/3239443318 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88011590 con 0x7feaa006ba70 2024-01-27T00:47:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.805+0000 7fea967fc700 1 -- 172.21.15.46:0/3239443318 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea880116f0 con 0x7feaa006ba70 2024-01-27T00:47:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.805+0000 7feaa4ebd700 1 -- 172.21.15.46:0/3239443318 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa015a7e0 con 0x7feaa006ba70 2024-01-27T00:47:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.805+0000 7fea967fc700 1 -- 172.21.15.46:0/3239443318 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88011ab0 con 0x7feaa006ba70 2024-01-27T00:47:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.805+0000 7feaa4ebd700 1 -- 172.21.15.46:0/3239443318 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa0154160 con 0x7feaa006ba70 2024-01-27T00:47:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.806+0000 7feaa4ebd700 1 -- 172.21.15.46:0/3239443318 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa015a970 con 0x7feaa006ba70 2024-01-27T00:47:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.806+0000 7fea967fc700 1 -- 172.21.15.46:0/3239443318 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 94) v1 ==== 84580+0+0 (secure 0 0 0) 0x7fea8801b3b0 con 0x7feaa006ba70 2024-01-27T00:47:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.807+0000 7fea967fc700 1 --2- 172.21.15.46:0/3239443318 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fea80066d80 0x7fea80069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.807+0000 7fea967fc700 1 -- 172.21.15.46:0/3239443318 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6625+0+0 (secure 0 0 0) 0x7fea88083b30 con 0x7feaa006ba70 2024-01-27T00:47:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.807+0000 7fea9e59c700 1 --2- 172.21.15.46:0/3239443318 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fea80066d80 0x7fea80069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.807+0000 7fea9e59c700 1 --2- 172.21.15.46:0/3239443318 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fea80066d80 0x7fea80069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fea90009c90 tx=0x7fea90008040 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:09.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:09.811+0000 7fea967fc700 1 -- 172.21.15.46:0/3239443318 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fea88050cb0 con 0x7feaa006ba70 2024-01-27T00:47:10.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.021+0000 7feaa4ebd700 1 -- 172.21.15.46:0/3239443318 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7feaa004c710 con 0x7feaa006ba70 2024-01-27T00:47:10.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.568+0000 7fea967fc700 1 -- 172.21.15.46:0/3239443318 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 95) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fea88051080 con 0x7feaa006ba70 2024-01-27T00:47:10.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.645+0000 7fea967fc700 1 -- 172.21.15.46:0/3239443318 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v180) v1 ==== 169+0+0 (secure 0 0 0) 0x7fea88054740 con 0x7feaa006ba70 2024-01-27T00:47:10.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.650+0000 7feaa4ebd700 1 -- 172.21.15.46:0/3239443318 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fea80066d80 msgr2=0x7fea80069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:10.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.650+0000 7feaa4ebd700 1 --2- 172.21.15.46:0/3239443318 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fea80066d80 0x7fea80069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fea90009c90 tx=0x7fea90008040 comp rx=0 tx=0).stop 2024-01-27T00:47:10.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.650+0000 7feaa4ebd700 1 -- 172.21.15.46:0/3239443318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa006ba70 msgr2=0x7feaa0150150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:10.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.650+0000 7feaa4ebd700 1 --2- 172.21.15.46:0/3239443318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa006ba70 0x7feaa0150150 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fea8800c590 tx=0x7fea8800c950 comp rx=0 tx=0).stop 2024-01-27T00:47:10.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.650+0000 7feaa4ebd700 1 -- 172.21.15.46:0/3239443318 shutdown_connections 2024-01-27T00:47:10.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.650+0000 7feaa4ebd700 1 --2- 172.21.15.46:0/3239443318 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fea80066d80 0x7fea80069230 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fea90009c90 tx=0x7fea90008040 comp rx=0 tx=0).stop 2024-01-27T00:47:10.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.650+0000 7feaa4ebd700 1 --2- 172.21.15.46:0/3239443318 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feaa01597e0 0x7feaa0153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:10.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.650+0000 7feaa4ebd700 1 --2- 172.21.15.46:0/3239443318 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feaa006aef0 0x7feaa01515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:10.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.650+0000 7feaa4ebd700 1 --2- 172.21.15.46:0/3239443318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa006ba70 0x7feaa0150150 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:10.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.650+0000 7feaa4ebd700 1 -- 172.21.15.46:0/3239443318 >> 172.21.15.46:0/3239443318 conn(0x7feaa006e2c0 msgr2=0x7feaa0156a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:10.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.651+0000 7feaa4ebd700 1 -- 172.21.15.46:0/3239443318 shutdown_connections 2024-01-27T00:47:10.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.652+0000 7feaa4ebd700 1 -- 172.21.15.46:0/3239443318 wait complete. 2024-01-27T00:47:10.660 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-01-27T00:47:10.673 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-27T00:47:10.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.854+0000 7f880092d700 1 Processor -- start 2024-01-27T00:47:10.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.854+0000 7f880092d700 1 -- start start 2024-01-27T00:47:10.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.855+0000 7f880092d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87fc159510 0x7f87fc1598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:10.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.855+0000 7f880092d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f87fc0ca690 0x7f87fc0cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:10.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.855+0000 7f880092d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc069260 con 0x7f87fc0ca220 2024-01-27T00:47:10.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.855+0000 7f880092d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87fc15d670 con 0x7f87fc0ca690 2024-01-27T00:47:10.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.855+0000 7f880092d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc0c9750 con 0x7f87fc159510 2024-01-27T00:47:10.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.855+0000 7f87fad9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f87fc0ca690 0x7f87fc0cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:10.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.855+0000 7f87fad9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f87fc0ca690 0x7f87fc0cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38764/0 (socket says 172.21.15.46:38764) 2024-01-27T00:47:10.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.855+0000 7f87fad9d700 1 -- 172.21.15.46:0/307064183 learned_addr learned my addr 172.21.15.46:0/307064183 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:10.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.855+0000 7f87fa59c700 1 --2- 172.21.15.46:0/307064183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87fc159510 0x7f87fc1598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:10.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.855+0000 7f87fad9d700 1 -- 172.21.15.46:0/307064183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87fc159510 msgr2=0x7f87fc1598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:10.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.855+0000 7f87fad9d700 1 --2- 172.21.15.46:0/307064183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87fc159510 0x7f87fc1598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:10.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.855+0000 7f87fad9d700 1 -- 172.21.15.46:0/307064183 >> v1:172.21.15.106:6789/0 conn(0x7f87fc0ca220 legacy=0x7f87fc0c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:10.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.856+0000 7f87fad9d700 1 -- 172.21.15.46:0/307064183 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc0c9a30 con 0x7f87fc0ca690 2024-01-27T00:47:10.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.856+0000 7f87fad9d700 1 --2- 172.21.15.46:0/307064183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc0ca690 0x7f87fc0cc380 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f87f4009f50 tx=0x7f87f4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19990640b46f6db3 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:10.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.856+0000 7f87f959a700 1 -- 172.21.15.46:0/307064183 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f400e070 con 0x7f87fc0ca690 2024-01-27T00:47:10.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.856+0000 7f87f959a700 1 -- 172.21.15.46:0/307064183 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f400ba70 con 0x7f87fc0ca690 2024-01-27T00:47:10.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.856+0000 7f87f959a700 1 -- 172.21.15.46:0/307064183 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f40092e0 con 0x7f87fc0ca690 2024-01-27T00:47:10.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.856+0000 7f880092d700 1 -- 172.21.15.46:0/307064183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc0ca690 msgr2=0x7f87fc0cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:10.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.856+0000 7f880092d700 1 --2- 172.21.15.46:0/307064183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc0ca690 0x7f87fc0cc380 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f87f4009f50 tx=0x7f87f4009840 comp rx=0 tx=0).stop 2024-01-27T00:47:10.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.857+0000 7f880092d700 1 -- 172.21.15.46:0/307064183 shutdown_connections 2024-01-27T00:47:10.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.857+0000 7f880092d700 1 --2- 172.21.15.46:0/307064183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc0ca690 0x7f87fc0cc380 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:10.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.857+0000 7f880092d700 1 --2- 172.21.15.46:0/307064183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87fc159510 0x7f87fc1598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:10.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.857+0000 7f880092d700 1 -- 172.21.15.46:0/307064183 >> 172.21.15.46:0/307064183 conn(0x7f87fc06bd90 msgr2=0x7f87fc06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:10.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.857+0000 7f880092d700 1 -- 172.21.15.46:0/307064183 shutdown_connections 2024-01-27T00:47:10.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.857+0000 7f880092d700 1 -- 172.21.15.46:0/307064183 wait complete. 2024-01-27T00:47:10.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.858+0000 7f880092d700 1 Processor -- start 2024-01-27T00:47:10.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.858+0000 7f880092d700 1 -- start start 2024-01-27T00:47:10.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.858+0000 7f880092d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87fc0ca220 0x7f87fc14c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:10.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f880092d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc159510 0x7f87fc0b2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:10.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f880092d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87fc1533f0 0x7f87fc153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:10.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f880092d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc153db0 con 0x7f87fc159510 2024-01-27T00:47:10.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f880092d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc15a540 con 0x7f87fc0ca220 2024-01-27T00:47:10.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f880092d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc0b3840 con 0x7f87fc1533f0 2024-01-27T00:47:10.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f87f9d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc159510 0x7f87fc0b2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:10.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f87f9d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc159510 0x7f87fc0b2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38768/0 (socket says 172.21.15.46:38768) 2024-01-27T00:47:10.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f87f9d9b700 1 -- 172.21.15.46:0/619616369 learned_addr learned my addr 172.21.15.46:0/619616369 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:10.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f87fad9d700 1 --2- 172.21.15.46:0/619616369 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87fc1533f0 0x7f87fc153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:10.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f87fa59c700 1 --2- 172.21.15.46:0/619616369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87fc0ca220 0x7f87fc14c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:10.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f87f9d9b700 1 -- 172.21.15.46:0/619616369 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87fc1533f0 msgr2=0x7f87fc153870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:10.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f87f9d9b700 1 --2- 172.21.15.46:0/619616369 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87fc1533f0 0x7f87fc153870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:10.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f87f9d9b700 1 -- 172.21.15.46:0/619616369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87fc0ca220 msgr2=0x7f87fc14c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:10.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f87f9d9b700 1 --2- 172.21.15.46:0/619616369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87fc0ca220 0x7f87fc14c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:10.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f87f9d9b700 1 -- 172.21.15.46:0/619616369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f4009ae0 con 0x7f87fc159510 2024-01-27T00:47:10.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f87fad9d700 1 --2- 172.21.15.46:0/619616369 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87fc1533f0 0x7f87fc153870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:47:10.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f87f9d9b700 1 --2- 172.21.15.46:0/619616369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc159510 0x7f87fc0b2ed0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00c590 tx=0x7f87ec00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:10.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f87ea7fc700 1 -- 172.21.15.46:0/619616369 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec011590 con 0x7f87fc159510 2024-01-27T00:47:10.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f87ea7fc700 1 -- 172.21.15.46:0/619616369 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ec0116f0 con 0x7f87fc159510 2024-01-27T00:47:10.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f880092d700 1 -- 172.21.15.46:0/619616369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87fc0b3b20 con 0x7f87fc159510 2024-01-27T00:47:10.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f87ea7fc700 1 -- 172.21.15.46:0/619616369 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec011ab0 con 0x7f87fc159510 2024-01-27T00:47:10.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.859+0000 7f880092d700 1 -- 172.21.15.46:0/619616369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87fc0b40e0 con 0x7f87fc159510 2024-01-27T00:47:10.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.860+0000 7f880092d700 1 -- 172.21.15.46:0/619616369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87fc15a6f0 con 0x7f87fc159510 2024-01-27T00:47:10.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.861+0000 7f87ea7fc700 1 -- 172.21.15.46:0/619616369 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 95) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f87ec01c440 con 0x7f87fc159510 2024-01-27T00:47:10.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.861+0000 7f87ea7fc700 1 --2- 172.21.15.46:0/619616369 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f87dc066ea0 0x7f87dc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:10.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.861+0000 7f87ea7fc700 1 -- 172.21.15.46:0/619616369 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f87ec083b60 con 0x7f87fc159510 2024-01-27T00:47:10.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.862+0000 7f87fa59c700 1 --2- 172.21.15.46:0/619616369 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f87dc066ea0 0x7f87dc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:10.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.862+0000 7f87fa59c700 1 --2- 172.21.15.46:0/619616369 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f87dc066ea0 0x7f87dc069350 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f87f0006fd0 tx=0x7f87f0008040 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:10.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:10.866+0000 7f87ea7fc700 1 -- 172.21.15.46:0/619616369 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f87fc15a6f0 con 0x7f87fc159510 2024-01-27T00:47:11.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.075+0000 7f880092d700 1 -- 172.21.15.46:0/619616369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f87fc06a9b0 con 0x7f87fc159510 2024-01-27T00:47:11.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.078+0000 7f87ea7fc700 1 -- 172.21.15.46:0/619616369 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v61) v1 ==== 104+0+0 (secure 0 0 0) 0x7f87ec017020 con 0x7f87fc159510 2024-01-27T00:47:11.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.081+0000 7f880092d700 1 -- 172.21.15.46:0/619616369 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f87dc066ea0 msgr2=0x7f87dc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.081+0000 7f880092d700 1 --2- 172.21.15.46:0/619616369 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f87dc066ea0 0x7f87dc069350 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f87f0006fd0 tx=0x7f87f0008040 comp rx=0 tx=0).stop 2024-01-27T00:47:11.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.081+0000 7f880092d700 1 -- 172.21.15.46:0/619616369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc159510 msgr2=0x7f87fc0b2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.081+0000 7f880092d700 1 --2- 172.21.15.46:0/619616369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc159510 0x7f87fc0b2ed0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00c590 tx=0x7f87ec00c950 comp rx=0 tx=0).stop 2024-01-27T00:47:11.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.081+0000 7f880092d700 1 -- 172.21.15.46:0/619616369 shutdown_connections 2024-01-27T00:47:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.081+0000 7f880092d700 1 --2- 172.21.15.46:0/619616369 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f87dc066ea0 0x7f87dc069350 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.081+0000 7f880092d700 1 --2- 172.21.15.46:0/619616369 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87fc1533f0 0x7f87fc153870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.081+0000 7f880092d700 1 --2- 172.21.15.46:0/619616369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87fc0ca220 0x7f87fc14c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.081+0000 7f880092d700 1 --2- 172.21.15.46:0/619616369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc159510 0x7f87fc0b2ed0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.081+0000 7f880092d700 1 -- 172.21.15.46:0/619616369 >> 172.21.15.46:0/619616369 conn(0x7f87fc06bd90 msgr2=0x7f87fc14d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.081+0000 7f880092d700 1 -- 172.21.15.46:0/619616369 shutdown_connections 2024-01-27T00:47:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.081+0000 7f880092d700 1 -- 172.21.15.46:0/619616369 wait complete. 2024-01-27T00:47:11.094 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-27T00:47:11.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.227+0000 7ff3c6865700 1 Processor -- start 2024-01-27T00:47:11.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.227+0000 7ff3c6865700 1 -- start start 2024-01-27T00:47:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.227+0000 7ff3c6865700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff3c0153c40 0x7ff3c006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.228+0000 7ff3c6865700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c015ad60 con 0x7ff3c0154010 2024-01-27T00:47:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.228+0000 7ff3c6865700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c015a150 con 0x7ff3c00b1e80 2024-01-27T00:47:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.228+0000 7ff3c6865700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c00b1d20 con 0x7ff3c0153c40 2024-01-27T00:47:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.228+0000 7ff3bf7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff3c0154010 0x7ff3c006dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40946/0 (socket says 172.21.15.46:40946) 2024-01-27T00:47:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.228+0000 7ff3bf7fe700 1 -- 172.21.15.46:0/1294798728 learned_addr learned my addr 172.21.15.46:0/1294798728 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.228+0000 7ff3bffff700 1 --2- 172.21.15.46:0/1294798728 >> v2:172.21.15.106:3300/0 conn(0x7ff3c0153c40 0x7ff3c006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.228+0000 7ff3beffd700 1 -- 172.21.15.46:0/1294798728 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867940288 0 0) 0x7ff3c015ad60 con 0x7ff3c0154010 2024-01-27T00:47:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.228+0000 7ff3beffd700 1 -- 172.21.15.46:0/1294798728 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a8003570 con 0x7ff3c0154010 2024-01-27T00:47:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.228+0000 7ff3beffd700 1 -- 172.21.15.46:0/1294798728 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017399772 0 0) 0x7ff3c015a150 con 0x7ff3c00b1e80 2024-01-27T00:47:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.228+0000 7ff3beffd700 1 -- 172.21.15.46:0/1294798728 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c015ad60 con 0x7ff3c00b1e80 2024-01-27T00:47:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.228+0000 7ff3bffff700 1 -- 172.21.15.46:0/1294798728 >> v1:172.21.15.106:6789/0 conn(0x7ff3c00b1e80 legacy=0x7ff3c00692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.228+0000 7ff3bffff700 1 -- 172.21.15.46:0/1294798728 >> v1:172.21.15.46:6789/0 conn(0x7ff3c0154010 legacy=0x7ff3c006dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.228+0000 7ff3bffff700 1 -- 172.21.15.46:0/1294798728 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c015a430 con 0x7ff3c0153c40 2024-01-27T00:47:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.229+0000 7ff3bffff700 1 --2- 172.21.15.46:0/1294798728 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3c0153c40 0x7ff3c006d6a0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff3b00098e0 tx=0x7ff3b00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e2023ad478dd1dc server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.229+0000 7ff3beffd700 1 -- 172.21.15.46:0/1294798728 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0011630 con 0x7ff3c0153c40 2024-01-27T00:47:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.229+0000 7ff3beffd700 1 -- 172.21.15.46:0/1294798728 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b0011790 con 0x7ff3c0153c40 2024-01-27T00:47:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.229+0000 7ff3beffd700 1 -- 172.21.15.46:0/1294798728 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0011b30 con 0x7ff3c0153c40 2024-01-27T00:47:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.229+0000 7ff3c6865700 1 -- 172.21.15.46:0/1294798728 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3c0153c40 msgr2=0x7ff3c006d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.229+0000 7ff3c6865700 1 --2- 172.21.15.46:0/1294798728 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3c0153c40 0x7ff3c006d6a0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff3b00098e0 tx=0x7ff3b00091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.229+0000 7ff3c6865700 1 -- 172.21.15.46:0/1294798728 shutdown_connections 2024-01-27T00:47:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.229+0000 7ff3c6865700 1 --2- 172.21.15.46:0/1294798728 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3c0153c40 0x7ff3c006d6a0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.229+0000 7ff3c6865700 1 -- 172.21.15.46:0/1294798728 >> 172.21.15.46:0/1294798728 conn(0x7ff3c0067930 msgr2=0x7ff3c0067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.229+0000 7ff3c6865700 1 -- 172.21.15.46:0/1294798728 shutdown_connections 2024-01-27T00:47:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.229+0000 7ff3c6865700 1 -- 172.21.15.46:0/1294798728 wait complete. 2024-01-27T00:47:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.231+0000 7ff3c6865700 1 Processor -- start 2024-01-27T00:47:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.231+0000 7ff3c6865700 1 -- start start 2024-01-27T00:47:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.232+0000 7ff3c6865700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3c00b1e80 0x7ff3c00ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.232+0000 7ff3c6865700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3c0153c40 0x7ff3c014ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.232+0000 7ff3c6865700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c0154010 0x7ff3c00ce600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.232+0000 7ff3c6865700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c00c8730 con 0x7ff3c0154010 2024-01-27T00:47:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.232+0000 7ff3c6865700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c00c8910 con 0x7ff3c0153c40 2024-01-27T00:47:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.232+0000 7ff3c6865700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c00c8bf0 con 0x7ff3c00b1e80 2024-01-27T00:47:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.232+0000 7ff3c4e02700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c0154010 0x7ff3c00ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.233+0000 7ff3c4e02700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c0154010 0x7ff3c00ce600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38776/0 (socket says 172.21.15.46:38776) 2024-01-27T00:47:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.233+0000 7ff3c4e02700 1 -- 172.21.15.46:0/3729106752 learned_addr learned my addr 172.21.15.46:0/3729106752 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.233+0000 7ff3bffff700 1 --2- 172.21.15.46:0/3729106752 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3c00b1e80 0x7ff3c00ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.233+0000 7ff3bf7fe700 1 --2- 172.21.15.46:0/3729106752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3c0153c40 0x7ff3c014ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.233+0000 7ff3c4e02700 1 -- 172.21.15.46:0/3729106752 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3c00b1e80 msgr2=0x7ff3c00ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.233+0000 7ff3c4e02700 1 --2- 172.21.15.46:0/3729106752 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3c00b1e80 0x7ff3c00ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.233+0000 7ff3c4e02700 1 -- 172.21.15.46:0/3729106752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3c0153c40 msgr2=0x7ff3c014ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.233+0000 7ff3c4e02700 1 --2- 172.21.15.46:0/3729106752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3c0153c40 0x7ff3c014ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.233+0000 7ff3c4e02700 1 -- 172.21.15.46:0/3729106752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b0009420 con 0x7ff3c0154010 2024-01-27T00:47:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.233+0000 7ff3c4e02700 1 --2- 172.21.15.46:0/3729106752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c0154010 0x7ff3c00ce600 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7ff3b800c860 tx=0x7ff3b8002700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.233+0000 7ff3a7fff700 1 -- 172.21.15.46:0/3729106752 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8002030 con 0x7ff3c0154010 2024-01-27T00:47:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.233+0000 7ff3a7fff700 1 -- 172.21.15.46:0/3729106752 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b80021b0 con 0x7ff3c0154010 2024-01-27T00:47:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.233+0000 7ff3c6865700 1 -- 172.21.15.46:0/3729106752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c00c8ed0 con 0x7ff3c0154010 2024-01-27T00:47:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.233+0000 7ff3a7fff700 1 -- 172.21.15.46:0/3729106752 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8012630 con 0x7ff3c0154010 2024-01-27T00:47:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.233+0000 7ff3c6865700 1 -- 172.21.15.46:0/3729106752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c0150510 con 0x7ff3c0154010 2024-01-27T00:47:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.234+0000 7ff3c6865700 1 -- 172.21.15.46:0/3729106752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c00cef30 con 0x7ff3c0154010 2024-01-27T00:47:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.235+0000 7ff3a7fff700 1 -- 172.21.15.46:0/3729106752 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 95) v1 ==== 84686+0+0 (secure 0 0 0) 0x7ff3b8012a20 con 0x7ff3c0154010 2024-01-27T00:47:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.235+0000 7ff3a7fff700 1 --2- 172.21.15.46:0/3729106752 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff3ac066ea0 0x7ff3ac069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.236+0000 7ff3a7fff700 1 -- 172.21.15.46:0/3729106752 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6211+0+0 (secure 0 0 0) 0x7ff3b8084650 con 0x7ff3c0154010 2024-01-27T00:47:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.236+0000 7ff3bffff700 1 --2- 172.21.15.46:0/3729106752 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff3ac066ea0 0x7ff3ac069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.236+0000 7ff3bffff700 1 --2- 172.21.15.46:0/3729106752 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff3ac066ea0 0x7ff3ac069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff3b00093f0 tx=0x7ff3b0017000 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:11.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.240+0000 7ff3a7fff700 1 -- 172.21.15.46:0/3729106752 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff3b8051700 con 0x7ff3c0154010 2024-01-27T00:47:11.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.445+0000 7ff3c6865700 1 -- 172.21.15.46:0/3729106752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7ff3c004c770 con 0x7ff3c0154010 2024-01-27T00:47:11.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.449+0000 7ff3a7fff700 1 -- 172.21.15.46:0/3729106752 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v62) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff3b8054d20 con 0x7ff3c0154010 2024-01-27T00:47:11.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.451+0000 7ff3c6865700 1 -- 172.21.15.46:0/3729106752 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff3ac066ea0 msgr2=0x7ff3ac069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.451+0000 7ff3c6865700 1 --2- 172.21.15.46:0/3729106752 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff3ac066ea0 0x7ff3ac069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff3b00093f0 tx=0x7ff3b0017000 comp rx=0 tx=0).stop 2024-01-27T00:47:11.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.451+0000 7ff3c6865700 1 -- 172.21.15.46:0/3729106752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c0154010 msgr2=0x7ff3c00ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.451+0000 7ff3c6865700 1 --2- 172.21.15.46:0/3729106752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c0154010 0x7ff3c00ce600 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7ff3b800c860 tx=0x7ff3b8002700 comp rx=0 tx=0).stop 2024-01-27T00:47:11.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.451+0000 7ff3c6865700 1 -- 172.21.15.46:0/3729106752 shutdown_connections 2024-01-27T00:47:11.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.451+0000 7ff3c6865700 1 --2- 172.21.15.46:0/3729106752 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff3ac066ea0 0x7ff3ac069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.451+0000 7ff3c6865700 1 --2- 172.21.15.46:0/3729106752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c0154010 0x7ff3c00ce600 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.451+0000 7ff3c6865700 1 --2- 172.21.15.46:0/3729106752 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3c00b1e80 0x7ff3c00ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.451+0000 7ff3c6865700 1 --2- 172.21.15.46:0/3729106752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3c0153c40 0x7ff3c014ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.451+0000 7ff3c6865700 1 -- 172.21.15.46:0/3729106752 >> 172.21.15.46:0/3729106752 conn(0x7ff3c0067930 msgr2=0x7ff3c00b07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:11.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.451+0000 7ff3c6865700 1 -- 172.21.15.46:0/3729106752 shutdown_connections 2024-01-27T00:47:11.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.451+0000 7ff3c6865700 1 -- 172.21.15.46:0/3729106752 wait complete. 2024-01-27T00:47:11.467 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-27T00:47:11.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.601+0000 7f8c8bb11700 1 Processor -- start 2024-01-27T00:47:11.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.601+0000 7f8c8bb11700 1 -- start start 2024-01-27T00:47:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.602+0000 7f8c8bb11700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8c84156940 0x7f8c8406c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.602+0000 7f8c8bb11700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c840c8b90 con 0x7f8c8406f9a0 2024-01-27T00:47:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.602+0000 7f8c8bb11700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c840c8e50 con 0x7f8c84156570 2024-01-27T00:47:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.602+0000 7f8c8bb11700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c8406d0b0 con 0x7f8c84156940 2024-01-27T00:47:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.602+0000 7f8c8a0ae700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8c8406f9a0 0x7f8c8406eb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40956/0 (socket says 172.21.15.46:40956) 2024-01-27T00:47:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.602+0000 7f8c8a0ae700 1 -- 172.21.15.46:0/242796472 learned_addr learned my addr 172.21.15.46:0/242796472 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.603+0000 7f8c890ac700 1 --2- 172.21.15.46:0/242796472 >> v2:172.21.15.106:3300/0 conn(0x7f8c84156940 0x7f8c8406c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.603+0000 7f8c888ab700 1 -- 172.21.15.46:0/242796472 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216041330 0 0) 0x7f8c840c8b90 con 0x7f8c8406f9a0 2024-01-27T00:47:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.603+0000 7f8c888ab700 1 -- 172.21.15.46:0/242796472 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c70003570 con 0x7f8c8406f9a0 2024-01-27T00:47:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.603+0000 7f8c890ac700 1 -- 172.21.15.46:0/242796472 >> v1:172.21.15.106:6789/0 conn(0x7f8c84156570 legacy=0x7f8c840c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.603+0000 7f8c888ab700 1 -- 172.21.15.46:0/242796472 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2245700149 0 0) 0x7f8c70003570 con 0x7f8c8406f9a0 2024-01-27T00:47:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.603+0000 7f8c890ac700 1 -- 172.21.15.46:0/242796472 >> v1:172.21.15.46:6789/0 conn(0x7f8c8406f9a0 legacy=0x7f8c8406eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.603+0000 7f8c890ac700 1 -- 172.21.15.46:0/242796472 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c840c4c40 con 0x7f8c84156940 2024-01-27T00:47:11.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.603+0000 7f8c890ac700 1 --2- 172.21.15.46:0/242796472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c84156940 0x7f8c8406c110 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f8c800098e0 tx=0x7f8c800091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6be3f56ad6ba362 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:11.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.603+0000 7f8c888ab700 1 -- 172.21.15.46:0/242796472 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c0017f0 con 0x7f8c84156940 2024-01-27T00:47:11.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.604+0000 7f8c888ab700 1 -- 172.21.15.46:0/242796472 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c80011420 con 0x7f8c84156940 2024-01-27T00:47:11.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.604+0000 7f8c888ab700 1 -- 172.21.15.46:0/242796472 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c800117c0 con 0x7f8c84156940 2024-01-27T00:47:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.604+0000 7f8c8bb11700 1 -- 172.21.15.46:0/242796472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c84156940 msgr2=0x7f8c8406c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.604+0000 7f8c8bb11700 1 --2- 172.21.15.46:0/242796472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c84156940 0x7f8c8406c110 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f8c800098e0 tx=0x7f8c800091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.604+0000 7f8c8bb11700 1 -- 172.21.15.46:0/242796472 shutdown_connections 2024-01-27T00:47:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.604+0000 7f8c8bb11700 1 --2- 172.21.15.46:0/242796472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c84156940 0x7f8c8406c110 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.604+0000 7f8c8bb11700 1 -- 172.21.15.46:0/242796472 >> 172.21.15.46:0/242796472 conn(0x7f8c840cdf50 msgr2=0x7f8c840c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.604+0000 7f8c8bb11700 1 -- 172.21.15.46:0/242796472 shutdown_connections 2024-01-27T00:47:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.604+0000 7f8c8bb11700 1 -- 172.21.15.46:0/242796472 wait complete. 2024-01-27T00:47:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.605+0000 7f8c8bb11700 1 Processor -- start 2024-01-27T00:47:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.605+0000 7f8c8bb11700 1 -- start start 2024-01-27T00:47:11.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.606+0000 7f8c8bb11700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c8406f9a0 0x7f8c841536b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.606+0000 7f8c8bb11700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c84156570 0x7f8c84153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.606+0000 7f8c8bb11700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c84156940 0x7f8c840b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.606+0000 7f8c8bb11700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c84151370 con 0x7f8c84156940 2024-01-27T00:47:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.606+0000 7f8c8bb11700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c84154fb0 con 0x7f8c84156570 2024-01-27T00:47:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.606+0000 7f8c8bb11700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c84154130 con 0x7f8c8406f9a0 2024-01-27T00:47:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.606+0000 7f8c8a0ae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c84156940 0x7f8c840b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.606+0000 7f8c8a0ae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c84156940 0x7f8c840b18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38782/0 (socket says 172.21.15.46:38782) 2024-01-27T00:47:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.606+0000 7f8c8a0ae700 1 -- 172.21.15.46:0/3645937707 learned_addr learned my addr 172.21.15.46:0/3645937707 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.606+0000 7f8c890ac700 1 --2- 172.21.15.46:0/3645937707 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c84156570 0x7f8c84153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.606+0000 7f8c898ad700 1 --2- 172.21.15.46:0/3645937707 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c8406f9a0 0x7f8c841536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.607+0000 7f8c8a0ae700 1 -- 172.21.15.46:0/3645937707 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c8406f9a0 msgr2=0x7f8c841536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.607+0000 7f8c8a0ae700 1 --2- 172.21.15.46:0/3645937707 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c8406f9a0 0x7f8c841536b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.607+0000 7f8c8a0ae700 1 -- 172.21.15.46:0/3645937707 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c84156570 msgr2=0x7f8c84153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.607+0000 7f8c8a0ae700 1 --2- 172.21.15.46:0/3645937707 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c84156570 0x7f8c84153bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.607+0000 7f8c8a0ae700 1 -- 172.21.15.46:0/3645937707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c80009420 con 0x7f8c84156940 2024-01-27T00:47:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.607+0000 7f8c890ac700 1 --2- 172.21.15.46:0/3645937707 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c84156570 0x7f8c84153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:11.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.607+0000 7f8c8a0ae700 1 --2- 172.21.15.46:0/3645937707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c84156940 0x7f8c840b18e0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c00cc10 tx=0x7f8c7c007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:11.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.607+0000 7f8c79ffb700 1 -- 172.21.15.46:0/3645937707 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c004dc0 con 0x7f8c84156940 2024-01-27T00:47:11.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.607+0000 7f8c79ffb700 1 -- 172.21.15.46:0/3645937707 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7c00a590 con 0x7f8c84156940 2024-01-27T00:47:11.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.607+0000 7f8c8bb11700 1 -- 172.21.15.46:0/3645937707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c84155440 con 0x7f8c84156940 2024-01-27T00:47:11.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.607+0000 7f8c79ffb700 1 -- 172.21.15.46:0/3645937707 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c00a950 con 0x7f8c84156940 2024-01-27T00:47:11.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.607+0000 7f8c8bb11700 1 -- 172.21.15.46:0/3645937707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c84154680 con 0x7f8c84156940 2024-01-27T00:47:11.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.609+0000 7f8c8bb11700 1 -- 172.21.15.46:0/3645937707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c84155160 con 0x7f8c84156940 2024-01-27T00:47:11.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.609+0000 7f8c79ffb700 1 -- 172.21.15.46:0/3645937707 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 95) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f8c7c00acc0 con 0x7f8c84156940 2024-01-27T00:47:11.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.609+0000 7f8c79ffb700 1 --2- 172.21.15.46:0/3645937707 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c68066ea0 0x7f8c68069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.609+0000 7f8c79ffb700 1 -- 172.21.15.46:0/3645937707 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f8c7c083b90 con 0x7f8c84156940 2024-01-27T00:47:11.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.610+0000 7f8c898ad700 1 --2- 172.21.15.46:0/3645937707 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c68066ea0 0x7f8c68069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.610+0000 7f8c898ad700 1 --2- 172.21.15.46:0/3645937707 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c68066ea0 0x7f8c68069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8c74003b80 tx=0x7f8c74007e40 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:11.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.614+0000 7f8c79ffb700 1 -- 172.21.15.46:0/3645937707 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8c84155160 con 0x7f8c84156940 2024-01-27T00:47:11.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.821+0000 7f8c8bb11700 1 -- 172.21.15.46:0/3645937707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f8c84155120 con 0x7f8c84156940 2024-01-27T00:47:11.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.825+0000 7f8c79ffb700 1 -- 172.21.15.46:0/3645937707 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v63) v1 ==== 115+0+0 (secure 0 0 0) 0x7f8c7c018020 con 0x7f8c84156940 2024-01-27T00:47:11.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.827+0000 7f8c8bb11700 1 -- 172.21.15.46:0/3645937707 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c68066ea0 msgr2=0x7f8c68069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.827+0000 7f8c8bb11700 1 --2- 172.21.15.46:0/3645937707 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c68066ea0 0x7f8c68069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8c74003b80 tx=0x7f8c74007e40 comp rx=0 tx=0).stop 2024-01-27T00:47:11.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.827+0000 7f8c8bb11700 1 -- 172.21.15.46:0/3645937707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c84156940 msgr2=0x7f8c840b18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.827+0000 7f8c8bb11700 1 --2- 172.21.15.46:0/3645937707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c84156940 0x7f8c840b18e0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c00cc10 tx=0x7f8c7c007c70 comp rx=0 tx=0).stop 2024-01-27T00:47:11.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.827+0000 7f8c8bb11700 1 -- 172.21.15.46:0/3645937707 shutdown_connections 2024-01-27T00:47:11.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.827+0000 7f8c8bb11700 1 --2- 172.21.15.46:0/3645937707 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f8c68066ea0 0x7f8c68069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.827+0000 7f8c8bb11700 1 --2- 172.21.15.46:0/3645937707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c84156940 0x7f8c840b18e0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.827+0000 7f8c8bb11700 1 --2- 172.21.15.46:0/3645937707 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c8406f9a0 0x7f8c841536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.827+0000 7f8c8bb11700 1 --2- 172.21.15.46:0/3645937707 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c84156570 0x7f8c84153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.827+0000 7f8c8bb11700 1 -- 172.21.15.46:0/3645937707 >> 172.21.15.46:0/3645937707 conn(0x7f8c840cdf50 msgr2=0x7f8c84152ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:11.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.827+0000 7f8c8bb11700 1 -- 172.21.15.46:0/3645937707 shutdown_connections 2024-01-27T00:47:11.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.827+0000 7f8c8bb11700 1 -- 172.21.15.46:0/3645937707 wait complete. 2024-01-27T00:47:11.841 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-01-27T00:47:11.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.972+0000 7efeb28c7700 1 Processor -- start 2024-01-27T00:47:11.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.972+0000 7efeb28c7700 1 -- start start 2024-01-27T00:47:11.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.973+0000 7efeb28c7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeac0b2b40 0x7efeac0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.973+0000 7efeb28c7700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeac15a0b0 con 0x7efeac0ae960 2024-01-27T00:47:11.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.973+0000 7efeb28c7700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeac0ae4d0 con 0x7efeac0aedd0 2024-01-27T00:47:11.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.973+0000 7efeb28c7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeac099fc0 con 0x7efeac0b2b40 2024-01-27T00:47:11.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.973+0000 7efeab7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efeac0ae960 0x7efeac0ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40964/0 (socket says 172.21.15.46:40964) 2024-01-27T00:47:11.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.973+0000 7efeab7fe700 1 -- 172.21.15.46:0/285073267 learned_addr learned my addr 172.21.15.46:0/285073267 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:11.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeabfff700 1 --2- 172.21.15.46:0/285073267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeac0b2b40 0x7efeac0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeaaffd700 1 -- 172.21.15.46:0/285073267 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632890178 0 0) 0x7efeac15a0b0 con 0x7efeac0ae960 2024-01-27T00:47:11.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeaaffd700 1 -- 172.21.15.46:0/285073267 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe94003570 con 0x7efeac0ae960 2024-01-27T00:47:11.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeaaffd700 1 -- 172.21.15.46:0/285073267 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127406539 0 0) 0x7efeac0ae4d0 con 0x7efeac0aedd0 2024-01-27T00:47:11.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeaaffd700 1 -- 172.21.15.46:0/285073267 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeac15a0b0 con 0x7efeac0aedd0 2024-01-27T00:47:11.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeaaffd700 1 -- 172.21.15.46:0/285073267 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 355846030 0 0) 0x7efe94003570 con 0x7efeac0ae960 2024-01-27T00:47:11.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeaaffd700 1 -- 172.21.15.46:0/285073267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeac0b2b40 msgr2=0x7efeac0ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:11.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeaaffd700 1 --2- 172.21.15.46:0/285073267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeac0b2b40 0x7efeac0ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeaaffd700 1 -- 172.21.15.46:0/285073267 >> v1:172.21.15.106:6789/0 conn(0x7efeac0aedd0 legacy=0x7efeac0cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeaaffd700 1 -- 172.21.15.46:0/285073267 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeac0ad8c0 con 0x7efeac0ae960 2024-01-27T00:47:11.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeaaffd700 1 -- 172.21.15.46:0/285073267 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7efea0001470 con 0x7efeac0ae960 2024-01-27T00:47:11.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeaaffd700 1 -- 172.21.15.46:0/285073267 >> v1:172.21.15.46:6789/0 conn(0x7efeac0ae960 legacy=0x7efeac0ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeaaffd700 1 --2- 172.21.15.46:0/285073267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeac0b2b40 0x7efe94003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeaaffd700 1 --2- 172.21.15.46:0/285073267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe94004180 0x7efe94004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeaaffd700 1 --2- 172.21.15.46:0/285073267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe94004d50 0x7efe940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeaaffd700 1 -- 172.21.15.46:0/285073267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9400bb70 con 0x7efe94004d50 2024-01-27T00:47:11.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeaaffd700 1 -- 172.21.15.46:0/285073267 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9400bce0 con 0x7efe94004180 2024-01-27T00:47:11.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeaaffd700 1 -- 172.21.15.46:0/285073267 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9400bea0 con 0x7efeac0b2b40 2024-01-27T00:47:11.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeb0e64700 1 --2- 172.21.15.46:0/285073267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe94004d50 0x7efe940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeabfff700 1 --2- 172.21.15.46:0/285073267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeac0b2b40 0x7efe94003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.974+0000 7efeab7fe700 1 --2- 172.21.15.46:0/285073267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe94004180 0x7efe94004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.975+0000 7efeb0e64700 1 -- 172.21.15.46:0/285073267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeac0b2b40 msgr2=0x7efe94003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:11.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.975+0000 7efeb0e64700 1 --2- 172.21.15.46:0/285073267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeac0b2b40 0x7efe94003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.975+0000 7efeb0e64700 1 -- 172.21.15.46:0/285073267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe94004180 msgr2=0x7efe94004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:11.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.975+0000 7efeb0e64700 1 --2- 172.21.15.46:0/285073267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe94004180 0x7efe94004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.975+0000 7efeb0e64700 1 -- 172.21.15.46:0/285073267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe94003570 con 0x7efe94004d50 2024-01-27T00:47:11.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.975+0000 7efeb0e64700 1 --2- 172.21.15.46:0/285073267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe94004d50 0x7efe940051c0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7efeac0c77d0 tx=0x7efea40017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48983dc9e3a18b74 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:11.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.975+0000 7efeaaffd700 1 -- 172.21.15.46:0/285073267 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea4001ab0 con 0x7efe94004d50 2024-01-27T00:47:11.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.975+0000 7efeaaffd700 1 -- 172.21.15.46:0/285073267 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea400c8b0 con 0x7efe94004d50 2024-01-27T00:47:11.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.975+0000 7efeaaffd700 1 -- 172.21.15.46:0/285073267 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea400cc50 con 0x7efe94004d50 2024-01-27T00:47:11.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.976+0000 7efeb28c7700 1 -- 172.21.15.46:0/285073267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe94004d50 msgr2=0x7efe940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.976+0000 7efeb28c7700 1 --2- 172.21.15.46:0/285073267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe94004d50 0x7efe940051c0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7efeac0c77d0 tx=0x7efea40017a0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.976+0000 7efeb0e64700 1 -- 172.21.15.46:0/285073267 reap_dead start 2024-01-27T00:47:11.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.976+0000 7efeb28c7700 1 -- 172.21.15.46:0/285073267 shutdown_connections 2024-01-27T00:47:11.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.976+0000 7efeb28c7700 1 -- 172.21.15.46:0/285073267 >> 172.21.15.46:0/285073267 conn(0x7efeac099a30 msgr2=0x7efeac06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:11.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.976+0000 7efeb28c7700 1 -- 172.21.15.46:0/285073267 shutdown_connections 2024-01-27T00:47:11.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.976+0000 7efeb28c7700 1 -- 172.21.15.46:0/285073267 wait complete. 2024-01-27T00:47:11.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.977+0000 7efeb28c7700 1 Processor -- start 2024-01-27T00:47:11.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.977+0000 7efeb28c7700 1 -- start start 2024-01-27T00:47:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeb28c7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeac1509f0 0x7efeac0b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeb28c7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efeac150dc0 0x7efeac1512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeb28c7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeac1517f0 0x7efeac0cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeb28c7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac0cd390 con 0x7efeac1517f0 2024-01-27T00:47:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeb28c7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac0b5920 con 0x7efeac150dc0 2024-01-27T00:47:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeb28c7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeac0b62f0 con 0x7efeac1509f0 2024-01-27T00:47:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeb0e64700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeac1517f0 0x7efeac0cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeb0e64700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeac1517f0 0x7efeac0cbfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38806/0 (socket says 172.21.15.46:38806) 2024-01-27T00:47:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeb0e64700 1 -- 172.21.15.46:0/4257813949 learned_addr learned my addr 172.21.15.46:0/4257813949 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeab7fe700 1 --2- 172.21.15.46:0/4257813949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efeac150dc0 0x7efeac1512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeabfff700 1 --2- 172.21.15.46:0/4257813949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeac1509f0 0x7efeac0b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeb0e64700 1 -- 172.21.15.46:0/4257813949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeac1509f0 msgr2=0x7efeac0b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeb0e64700 1 --2- 172.21.15.46:0/4257813949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeac1509f0 0x7efeac0b5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeb0e64700 1 -- 172.21.15.46:0/4257813949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efeac150dc0 msgr2=0x7efeac1512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeb0e64700 1 --2- 172.21.15.46:0/4257813949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efeac150dc0 0x7efeac1512b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeb0e64700 1 -- 172.21.15.46:0/4257813949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeac0b4d10 con 0x7efeac1517f0 2024-01-27T00:47:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeabfff700 1 --2- 172.21.15.46:0/4257813949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeac1509f0 0x7efeac0b5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:47:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeab7fe700 1 --2- 172.21.15.46:0/4257813949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efeac150dc0 0x7efeac1512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:47:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.978+0000 7efeb0e64700 1 --2- 172.21.15.46:0/4257813949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeac1517f0 0x7efeac0cbfd0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7efe9400bcb0 tx=0x7efea4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.979+0000 7efe93fff700 1 -- 172.21.15.46:0/4257813949 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea400c6d0 con 0x7efeac1517f0 2024-01-27T00:47:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.979+0000 7efe93fff700 1 -- 172.21.15.46:0/4257813949 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea4022070 con 0x7efeac1517f0 2024-01-27T00:47:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.979+0000 7efe93fff700 1 -- 172.21.15.46:0/4257813949 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea400ca20 con 0x7efeac1517f0 2024-01-27T00:47:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.979+0000 7efeb28c7700 1 -- 172.21.15.46:0/4257813949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeac0b4ff0 con 0x7efeac1517f0 2024-01-27T00:47:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.979+0000 7efeb28c7700 1 -- 172.21.15.46:0/4257813949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeac0b55b0 con 0x7efeac1517f0 2024-01-27T00:47:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.980+0000 7efeb28c7700 1 -- 172.21.15.46:0/4257813949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeac04c710 con 0x7efeac1517f0 2024-01-27T00:47:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.980+0000 7efe93fff700 1 -- 172.21.15.46:0/4257813949 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 95) v1 ==== 84686+0+0 (secure 0 0 0) 0x7efea4003030 con 0x7efeac1517f0 2024-01-27T00:47:11.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.981+0000 7efe93fff700 1 --2- 172.21.15.46:0/4257813949 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7efe98066ed0 0x7efe98069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:11.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.981+0000 7efe93fff700 1 -- 172.21.15.46:0/4257813949 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6211+0+0 (secure 0 0 0) 0x7efea4085ef0 con 0x7efeac1517f0 2024-01-27T00:47:11.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.981+0000 7efeabfff700 1 --2- 172.21.15.46:0/4257813949 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7efe98066ed0 0x7efe98069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:11.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.982+0000 7efeabfff700 1 --2- 172.21.15.46:0/4257813949 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7efe98066ed0 0x7efe98069380 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efe9c009390 tx=0x7efe9c008040 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:11.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:11.985+0000 7efe93fff700 1 -- 172.21.15.46:0/4257813949 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efea4053040 con 0x7efeac1517f0 2024-01-27T00:47:12.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.274+0000 7efeb28c7700 1 -- 172.21.15.46:0/4257813949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7efeac0b5a60 con 0x7efeac1517f0 2024-01-27T00:47:12.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.454+0000 7efe93fff700 1 -- 172.21.15.46:0/4257813949 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7efea4053410 con 0x7efeac1517f0 2024-01-27T00:47:12.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.656+0000 7efe93fff700 1 -- 172.21.15.46:0/4257813949 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2525) v1 ==== 174+0+0 (secure 0 0 0) 0x7efea406d560 con 0x7efeac1517f0 2024-01-27T00:47:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.659+0000 7efeb28c7700 1 -- 172.21.15.46:0/4257813949 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7efe98066ed0 msgr2=0x7efe98069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.659+0000 7efeb28c7700 1 --2- 172.21.15.46:0/4257813949 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7efe98066ed0 0x7efe98069380 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efe9c009390 tx=0x7efe9c008040 comp rx=0 tx=0).stop 2024-01-27T00:47:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.659+0000 7efeb28c7700 1 -- 172.21.15.46:0/4257813949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeac1517f0 msgr2=0x7efeac0cbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.659+0000 7efeb28c7700 1 --2- 172.21.15.46:0/4257813949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeac1517f0 0x7efeac0cbfd0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7efe9400bcb0 tx=0x7efea4016000 comp rx=0 tx=0).stop 2024-01-27T00:47:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.659+0000 7efeb28c7700 1 -- 172.21.15.46:0/4257813949 shutdown_connections 2024-01-27T00:47:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.659+0000 7efeb28c7700 1 --2- 172.21.15.46:0/4257813949 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7efe98066ed0 0x7efe98069380 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.659+0000 7efeb28c7700 1 --2- 172.21.15.46:0/4257813949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeac1517f0 0x7efeac0cbfd0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.659+0000 7efeb28c7700 1 --2- 172.21.15.46:0/4257813949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efeac1509f0 0x7efeac0b5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.659+0000 7efeb28c7700 1 --2- 172.21.15.46:0/4257813949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efeac150dc0 0x7efeac1512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.659+0000 7efeb28c7700 1 -- 172.21.15.46:0/4257813949 >> 172.21.15.46:0/4257813949 conn(0x7efeac099a30 msgr2=0x7efeac159930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.659+0000 7efeb28c7700 1 -- 172.21.15.46:0/4257813949 shutdown_connections 2024-01-27T00:47:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.659+0000 7efeb28c7700 1 -- 172.21.15.46:0/4257813949 wait complete. 2024-01-27T00:47:12.675 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2024-01-27T00:47:12.676 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2024-01-27T00:47:12.676 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-27T00:47:12.676 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-01-27T00:47:12.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.811+0000 7fb2abf38700 1 Processor -- start 2024-01-27T00:47:12.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.811+0000 7fb2abf38700 1 -- start start 2024-01-27T00:47:12.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.812+0000 7fb2abf38700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2a40acf30 0x7fb2a40c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.812+0000 7fb2abf38700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb2a40ad300 0x7fb2a40ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.812+0000 7fb2abf38700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a40c4230 con 0x7fb2a40ad780 2024-01-27T00:47:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.812+0000 7fb2abf38700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a4157be0 con 0x7fb2a40acf30 2024-01-27T00:47:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.812+0000 7fb2abf38700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a40cbf10 con 0x7fb2a40ad300 2024-01-27T00:47:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.813+0000 7fb2a9cd4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2a40acf30 0x7fb2a40c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.813+0000 7fb2a9cd4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2a40acf30 0x7fb2a40c3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38818/0 (socket says 172.21.15.46:38818) 2024-01-27T00:47:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.813+0000 7fb2a9cd4700 1 -- 172.21.15.46:0/2992894438 learned_addr learned my addr 172.21.15.46:0/2992894438 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.813+0000 7fb2a94d3700 1 --2- 172.21.15.46:0/2992894438 >> v2:172.21.15.106:3300/0 conn(0x7fb2a40ad300 0x7fb2a40ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.813+0000 7fb2a9cd4700 1 -- 172.21.15.46:0/2992894438 >> v2:172.21.15.106:3300/0 conn(0x7fb2a40ad300 msgr2=0x7fb2a40ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.813+0000 7fb2a9cd4700 1 --2- 172.21.15.46:0/2992894438 >> v2:172.21.15.106:3300/0 conn(0x7fb2a40ad300 0x7fb2a40ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.813+0000 7fb2a9cd4700 1 -- 172.21.15.46:0/2992894438 >> v1:172.21.15.46:6789/0 conn(0x7fb2a40ad780 legacy=0x7fb2a415b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.813+0000 7fb2a9cd4700 1 -- 172.21.15.46:0/2992894438 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a40c4460 con 0x7fb2a40acf30 2024-01-27T00:47:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.813+0000 7fb2a9cd4700 1 --2- 172.21.15.46:0/2992894438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40acf30 0x7fb2a40c3cf0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40b1060 tx=0x7fb2a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50c9ec24ae632a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.813+0000 7fb2a8cd2700 1 -- 172.21.15.46:0/2992894438 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0009e70 con 0x7fb2a40acf30 2024-01-27T00:47:12.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.813+0000 7fb2a8cd2700 1 -- 172.21.15.46:0/2992894438 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2a000b860 con 0x7fb2a40acf30 2024-01-27T00:47:12.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.813+0000 7fb2a8cd2700 1 -- 172.21.15.46:0/2992894438 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a000bc00 con 0x7fb2a40acf30 2024-01-27T00:47:12.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.814+0000 7fb2abf38700 1 -- 172.21.15.46:0/2992894438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40acf30 msgr2=0x7fb2a40c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:12.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.814+0000 7fb2abf38700 1 --2- 172.21.15.46:0/2992894438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40acf30 0x7fb2a40c3cf0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40b1060 tx=0x7fb2a0009250 comp rx=0 tx=0).stop 2024-01-27T00:47:12.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.814+0000 7fb2abf38700 1 -- 172.21.15.46:0/2992894438 shutdown_connections 2024-01-27T00:47:12.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.814+0000 7fb2abf38700 1 --2- 172.21.15.46:0/2992894438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40acf30 0x7fb2a40c3cf0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:12.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.814+0000 7fb2abf38700 1 --2- 172.21.15.46:0/2992894438 >> v2:172.21.15.106:3300/0 conn(0x7fb2a40ad300 0x7fb2a40ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:12.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.814+0000 7fb2abf38700 1 -- 172.21.15.46:0/2992894438 >> 172.21.15.46:0/2992894438 conn(0x7fb2a4097820 msgr2=0x7fb2a4068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:12.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.814+0000 7fb2abf38700 1 -- 172.21.15.46:0/2992894438 shutdown_connections 2024-01-27T00:47:12.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.814+0000 7fb2abf38700 1 -- 172.21.15.46:0/2992894438 wait complete. 2024-01-27T00:47:12.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.815+0000 7fb2abf38700 1 Processor -- start 2024-01-27T00:47:12.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.815+0000 7fb2abf38700 1 -- start start 2024-01-27T00:47:12.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.816+0000 7fb2abf38700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2a40acf30 0x7fb2a40aecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:12.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.816+0000 7fb2abf38700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40ad300 0x7fb2a40c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:12.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.816+0000 7fb2abf38700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2a40ad780 0x7fb2a41538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:12.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.816+0000 7fb2abf38700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a40ae4a0 con 0x7fb2a40ad300 2024-01-27T00:47:12.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.816+0000 7fb2abf38700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a40b2b70 con 0x7fb2a40ad780 2024-01-27T00:47:12.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.816+0000 7fb2abf38700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a40af230 con 0x7fb2a40acf30 2024-01-27T00:47:12.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.816+0000 7fb2a94d3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40ad300 0x7fb2a40c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:12.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.816+0000 7fb2a94d3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40ad300 0x7fb2a40c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38822/0 (socket says 172.21.15.46:38822) 2024-01-27T00:47:12.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.816+0000 7fb2a94d3700 1 -- 172.21.15.46:0/3311931728 learned_addr learned my addr 172.21.15.46:0/3311931728 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:12.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.816+0000 7fb2a9cd4700 1 --2- 172.21.15.46:0/3311931728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2a40acf30 0x7fb2a40aecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:12.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.816+0000 7fb2aa4d5700 1 --2- 172.21.15.46:0/3311931728 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2a40ad780 0x7fb2a41538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:12.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.817+0000 7fb2a94d3700 1 -- 172.21.15.46:0/3311931728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2a40acf30 msgr2=0x7fb2a40aecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:12.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.817+0000 7fb2a94d3700 1 --2- 172.21.15.46:0/3311931728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2a40acf30 0x7fb2a40aecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:12.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.817+0000 7fb2a94d3700 1 -- 172.21.15.46:0/3311931728 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2a40ad780 msgr2=0x7fb2a41538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:12.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.817+0000 7fb2a94d3700 1 --2- 172.21.15.46:0/3311931728 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2a40ad780 0x7fb2a41538a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:12.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.817+0000 7fb2a94d3700 1 -- 172.21.15.46:0/3311931728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a00094f0 con 0x7fb2a40ad300 2024-01-27T00:47:12.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.817+0000 7fb2a9cd4700 1 --2- 172.21.15.46:0/3311931728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2a40acf30 0x7fb2a40aecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:47:12.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.817+0000 7fb2a94d3700 1 --2- 172.21.15.46:0/3311931728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40ad300 0x7fb2a40c93b0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fb29400ec80 tx=0x7fb2940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:12.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.817+0000 7fb299ffb700 1 -- 172.21.15.46:0/3311931728 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29400c9d0 con 0x7fb2a40ad300 2024-01-27T00:47:12.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.817+0000 7fb299ffb700 1 -- 172.21.15.46:0/3311931728 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb29400cb30 con 0x7fb2a40ad300 2024-01-27T00:47:12.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.817+0000 7fb299ffb700 1 -- 172.21.15.46:0/3311931728 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb294013470 con 0x7fb2a40ad300 2024-01-27T00:47:12.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.817+0000 7fb2abf38700 1 -- 172.21.15.46:0/3311931728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a40c9d00 con 0x7fb2a40ad300 2024-01-27T00:47:12.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.817+0000 7fb2abf38700 1 -- 172.21.15.46:0/3311931728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a40ca2c0 con 0x7fb2a40ad300 2024-01-27T00:47:12.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.818+0000 7fb2abf38700 1 -- 172.21.15.46:0/3311931728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a40b2d00 con 0x7fb2a40ad300 2024-01-27T00:47:12.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.819+0000 7fb299ffb700 1 -- 172.21.15.46:0/3311931728 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb294013820 con 0x7fb2a40ad300 2024-01-27T00:47:12.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.819+0000 7fb299ffb700 1 --2- 172.21.15.46:0/3311931728 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fb288066f30 0x7fb2880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:12.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.819+0000 7fb299ffb700 1 -- 172.21.15.46:0/3311931728 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6211+0+0 (secure 0 0 0) 0x7fb294084ac0 con 0x7fb2a40ad300 2024-01-27T00:47:12.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.819+0000 7fb2a9cd4700 1 --2- 172.21.15.46:0/3311931728 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fb288066f30 0x7fb2880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:12.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.820+0000 7fb2a9cd4700 1 --2- 172.21.15.46:0/3311931728 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fb288066f30 0x7fb2880693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40b1060 tx=0x7fb2a0017000 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:12.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:12.824+0000 7fb299ffb700 1 -- 172.21.15.46:0/3311931728 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb294051b30 con 0x7fb2a40ad300 2024-01-27T00:47:13.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.112+0000 7fb2abf38700 1 -- 172.21.15.46:0/3311931728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7fb2a40b2960 con 0x7fb2a40ad300 2024-01-27T00:47:13.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.660+0000 7fb299ffb700 1 -- 172.21.15.46:0/3311931728 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2526) v1 ==== 168+0+0 (secure 0 0 0) 0x7fb294055150 con 0x7fb2a40ad300 2024-01-27T00:47:13.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.663+0000 7fb2abf38700 1 -- 172.21.15.46:0/3311931728 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fb288066f30 msgr2=0x7fb2880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:13.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.663+0000 7fb2abf38700 1 --2- 172.21.15.46:0/3311931728 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fb288066f30 0x7fb2880693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40b1060 tx=0x7fb2a0017000 comp rx=0 tx=0).stop 2024-01-27T00:47:13.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.663+0000 7fb2abf38700 1 -- 172.21.15.46:0/3311931728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40ad300 msgr2=0x7fb2a40c93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:13.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.663+0000 7fb2abf38700 1 --2- 172.21.15.46:0/3311931728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40ad300 0x7fb2a40c93b0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fb29400ec80 tx=0x7fb2940091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:13.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.663+0000 7fb2abf38700 1 -- 172.21.15.46:0/3311931728 shutdown_connections 2024-01-27T00:47:13.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.663+0000 7fb2abf38700 1 --2- 172.21.15.46:0/3311931728 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fb288066f30 0x7fb2880693e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:13.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.663+0000 7fb2abf38700 1 --2- 172.21.15.46:0/3311931728 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2a40ad780 0x7fb2a41538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:13.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.663+0000 7fb2abf38700 1 --2- 172.21.15.46:0/3311931728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2a40acf30 0x7fb2a40aecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:13.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.663+0000 7fb2abf38700 1 --2- 172.21.15.46:0/3311931728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40ad300 0x7fb2a40c93b0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:13.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.663+0000 7fb2abf38700 1 -- 172.21.15.46:0/3311931728 >> 172.21.15.46:0/3311931728 conn(0x7fb2a4097820 msgr2=0x7fb2a406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:13.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.663+0000 7fb2abf38700 1 -- 172.21.15.46:0/3311931728 shutdown_connections 2024-01-27T00:47:13.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.663+0000 7fb2abf38700 1 -- 172.21.15.46:0/3311931728 wait complete. 2024-01-27T00:47:13.678 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-27T00:47:13.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.817+0000 7f8e53792700 1 Processor -- start 2024-01-27T00:47:13.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.817+0000 7f8e53792700 1 -- start start 2024-01-27T00:47:13.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.818+0000 7f8e53792700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8e4c0c93a0 0x7f8e4c0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:13.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.818+0000 7f8e53792700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8e4c0681f0 0x7f8e4c159930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.818+0000 7f8e53792700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c15e390 con 0x7f8e4c067d50 2024-01-27T00:47:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.818+0000 7f8e53792700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e4c15d810 con 0x7f8e4c0681f0 2024-01-27T00:47:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.818+0000 7f8e53792700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e4c15daf0 con 0x7f8e4c0c93a0 2024-01-27T00:47:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.818+0000 7f8e51d2f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8e4c0681f0 0x7f8e4c159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.818+0000 7f8e51d2f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8e4c0681f0 0x7f8e4c159930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38830/0 (socket says 172.21.15.46:38830) 2024-01-27T00:47:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.818+0000 7f8e51d2f700 1 -- 172.21.15.46:0/418891480 learned_addr learned my addr 172.21.15.46:0/418891480 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.819+0000 7f8e5152e700 1 --2- 172.21.15.46:0/418891480 >> v2:172.21.15.106:3300/0 conn(0x7f8e4c0c93a0 0x7f8e4c0c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.819+0000 7f8e51d2f700 1 -- 172.21.15.46:0/418891480 >> v2:172.21.15.106:3300/0 conn(0x7f8e4c0c93a0 msgr2=0x7f8e4c0c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.819+0000 7f8e51d2f700 1 --2- 172.21.15.46:0/418891480 >> v2:172.21.15.106:3300/0 conn(0x7f8e4c0c93a0 0x7f8e4c0c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.819+0000 7f8e51d2f700 1 -- 172.21.15.46:0/418891480 >> v1:172.21.15.106:6789/0 conn(0x7f8e4c067d50 legacy=0x7f8e4c159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.819+0000 7f8e51d2f700 1 -- 172.21.15.46:0/418891480 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c15ddd0 con 0x7f8e4c0681f0 2024-01-27T00:47:13.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.819+0000 7f8e51d2f700 1 --2- 172.21.15.46:0/418891480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c0681f0 0x7f8e4c159930 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c06e380 tx=0x7f8e44009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c87a35527ca5622 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:13.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.819+0000 7f8e43fff700 1 -- 172.21.15.46:0/418891480 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4400e070 con 0x7f8e4c0681f0 2024-01-27T00:47:13.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.819+0000 7f8e43fff700 1 -- 172.21.15.46:0/418891480 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4400ba50 con 0x7f8e4c0681f0 2024-01-27T00:47:13.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.819+0000 7f8e43fff700 1 -- 172.21.15.46:0/418891480 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e440092e0 con 0x7f8e4c0681f0 2024-01-27T00:47:13.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.820+0000 7f8e53792700 1 -- 172.21.15.46:0/418891480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c0681f0 msgr2=0x7f8e4c159930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:13.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.820+0000 7f8e53792700 1 --2- 172.21.15.46:0/418891480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c0681f0 0x7f8e4c159930 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c06e380 tx=0x7f8e44009840 comp rx=0 tx=0).stop 2024-01-27T00:47:13.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.820+0000 7f8e53792700 1 -- 172.21.15.46:0/418891480 shutdown_connections 2024-01-27T00:47:13.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.820+0000 7f8e53792700 1 --2- 172.21.15.46:0/418891480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c0681f0 0x7f8e4c159930 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:13.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.820+0000 7f8e53792700 1 --2- 172.21.15.46:0/418891480 >> v2:172.21.15.106:3300/0 conn(0x7f8e4c0c93a0 0x7f8e4c0c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:13.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.820+0000 7f8e53792700 1 -- 172.21.15.46:0/418891480 >> 172.21.15.46:0/418891480 conn(0x7f8e4c0ca9a0 msgr2=0x7f8e4c068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:13.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.820+0000 7f8e53792700 1 -- 172.21.15.46:0/418891480 shutdown_connections 2024-01-27T00:47:13.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.820+0000 7f8e53792700 1 -- 172.21.15.46:0/418891480 wait complete. 2024-01-27T00:47:13.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.821+0000 7f8e53792700 1 Processor -- start 2024-01-27T00:47:13.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.821+0000 7f8e53792700 1 -- start start 2024-01-27T00:47:13.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.822+0000 7f8e53792700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8e4c067d50 0x7f8e4c152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:13.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.822+0000 7f8e53792700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c0681f0 0x7f8e4c169aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:13.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.822+0000 7f8e53792700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e4c0c93a0 0x7f8e4c15c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:13.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.822+0000 7f8e53792700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c15bf60 con 0x7f8e4c0681f0 2024-01-27T00:47:13.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.822+0000 7f8e53792700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c16a3d0 con 0x7f8e4c067d50 2024-01-27T00:47:13.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.822+0000 7f8e53792700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c16a6b0 con 0x7f8e4c0c93a0 2024-01-27T00:47:13.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.822+0000 7f8e50d2d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c0681f0 0x7f8e4c169aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:13.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.822+0000 7f8e50d2d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c0681f0 0x7f8e4c169aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38846/0 (socket says 172.21.15.46:38846) 2024-01-27T00:47:13.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.822+0000 7f8e50d2d700 1 -- 172.21.15.46:0/427228286 learned_addr learned my addr 172.21.15.46:0/427228286 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:13.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.822+0000 7f8e5152e700 1 --2- 172.21.15.46:0/427228286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8e4c067d50 0x7f8e4c152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:13.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.822+0000 7f8e51d2f700 1 --2- 172.21.15.46:0/427228286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e4c0c93a0 0x7f8e4c15c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:13.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.823+0000 7f8e50d2d700 1 -- 172.21.15.46:0/427228286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e4c0c93a0 msgr2=0x7f8e4c15c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:13.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.823+0000 7f8e50d2d700 1 --2- 172.21.15.46:0/427228286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e4c0c93a0 0x7f8e4c15c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.823+0000 7f8e50d2d700 1 -- 172.21.15.46:0/427228286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8e4c067d50 msgr2=0x7f8e4c152af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.823+0000 7f8e50d2d700 1 --2- 172.21.15.46:0/427228286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8e4c067d50 0x7f8e4c152af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.823+0000 7f8e50d2d700 1 -- 172.21.15.46:0/427228286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e44009ae0 con 0x7f8e4c0681f0 2024-01-27T00:47:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.823+0000 7f8e51d2f700 1 --2- 172.21.15.46:0/427228286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e4c0c93a0 0x7f8e4c15c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:47:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.823+0000 7f8e50d2d700 1 --2- 172.21.15.46:0/427228286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c0681f0 0x7f8e4c169aa0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800c4e0 tx=0x7f8e4800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.823+0000 7f8e417fa700 1 -- 172.21.15.46:0/427228286 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e48011590 con 0x7f8e4c0681f0 2024-01-27T00:47:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.823+0000 7f8e417fa700 1 -- 172.21.15.46:0/427228286 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e48011710 con 0x7f8e4c0681f0 2024-01-27T00:47:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.823+0000 7f8e53792700 1 -- 172.21.15.46:0/427228286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e4c16a990 con 0x7f8e4c0681f0 2024-01-27T00:47:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.823+0000 7f8e417fa700 1 -- 172.21.15.46:0/427228286 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e48011ad0 con 0x7f8e4c0681f0 2024-01-27T00:47:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.823+0000 7f8e53792700 1 -- 172.21.15.46:0/427228286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e4c153030 con 0x7f8e4c0681f0 2024-01-27T00:47:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.824+0000 7f8e53792700 1 -- 172.21.15.46:0/427228286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e4c15c110 con 0x7f8e4c0681f0 2024-01-27T00:47:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.824+0000 7f8e417fa700 1 -- 172.21.15.46:0/427228286 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8e4801b3b0 con 0x7f8e4c0681f0 2024-01-27T00:47:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.825+0000 7f8e417fa700 1 --2- 172.21.15.46:0/427228286 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f8e30066f30 0x7f8e300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.825+0000 7f8e417fa700 1 -- 172.21.15.46:0/427228286 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f8e48083c70 con 0x7f8e4c0681f0 2024-01-27T00:47:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.825+0000 7f8e5152e700 1 --2- 172.21.15.46:0/427228286 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f8e30066f30 0x7f8e300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.826+0000 7f8e5152e700 1 --2- 172.21.15.46:0/427228286 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f8e30066f30 0x7f8e300693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c006fd0 tx=0x7f8e3c008040 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:13.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:13.829+0000 7f8e417fa700 1 -- 172.21.15.46:0/427228286 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8e4c15c110 con 0x7f8e4c0681f0 2024-01-27T00:47:14.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.033+0000 7f8e53792700 1 -- 172.21.15.46:0/427228286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8e4c15c110 con 0x7f8e4c0681f0 2024-01-27T00:47:14.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.037+0000 7f8e417fa700 1 -- 172.21.15.46:0/427228286 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8e4c15c110 con 0x7f8e4c0681f0 2024-01-27T00:47:14.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.039+0000 7f8e53792700 1 -- 172.21.15.46:0/427228286 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f8e30066f30 msgr2=0x7f8e300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:14.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.039+0000 7f8e53792700 1 --2- 172.21.15.46:0/427228286 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f8e30066f30 0x7f8e300693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c006fd0 tx=0x7f8e3c008040 comp rx=0 tx=0).stop 2024-01-27T00:47:14.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.039+0000 7f8e53792700 1 -- 172.21.15.46:0/427228286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c0681f0 msgr2=0x7f8e4c169aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:14.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.039+0000 7f8e53792700 1 --2- 172.21.15.46:0/427228286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c0681f0 0x7f8e4c169aa0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800c4e0 tx=0x7f8e4800c8a0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.040+0000 7f8e53792700 1 -- 172.21.15.46:0/427228286 shutdown_connections 2024-01-27T00:47:14.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.040+0000 7f8e53792700 1 --2- 172.21.15.46:0/427228286 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f8e30066f30 0x7f8e300693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.040+0000 7f8e53792700 1 --2- 172.21.15.46:0/427228286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8e4c0c93a0 0x7f8e4c15c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.040+0000 7f8e53792700 1 --2- 172.21.15.46:0/427228286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8e4c067d50 0x7f8e4c152af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.040+0000 7f8e53792700 1 --2- 172.21.15.46:0/427228286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e4c0681f0 0x7f8e4c169aa0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.040+0000 7f8e53792700 1 -- 172.21.15.46:0/427228286 >> 172.21.15.46:0/427228286 conn(0x7f8e4c0ca9a0 msgr2=0x7f8e4c06eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:14.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.040+0000 7f8e53792700 1 -- 172.21.15.46:0/427228286 shutdown_connections 2024-01-27T00:47:14.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.040+0000 7f8e53792700 1 -- 172.21.15.46:0/427228286 wait complete. 2024-01-27T00:47:14.053 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-27T00:47:14.053 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T00:47:14.102 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T00:47:14.148 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-27T00:47:14.148 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-27T00:47:14.149 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T00:47:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.285+0000 7fa29e8db700 1 Processor -- start 2024-01-27T00:47:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.285+0000 7fa29e8db700 1 -- start start 2024-01-27T00:47:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.286+0000 7fa29e8db700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa2980ae520 0x7fa2980b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.286+0000 7fa29e8db700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2980b3fb0 0x7fa298157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.286+0000 7fa29e8db700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2980b4410 con 0x7fa298157340 2024-01-27T00:47:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.286+0000 7fa29e8db700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa298158ae0 con 0x7fa2980ae520 2024-01-27T00:47:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.286+0000 7fa29e8db700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29815b650 con 0x7fa2980b3fb0 2024-01-27T00:47:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.287+0000 7fa2977fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa2980ae520 0x7fa2980b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.287+0000 7fa29ce78700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2980b3fb0 0x7fa298157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.287+0000 7fa2977fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa2980ae520 0x7fa2980b3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:37036/0 (socket says 172.21.15.46:37036) 2024-01-27T00:47:14.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.287+0000 7fa29ce78700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2980b3fb0 0x7fa298157710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:38152/0 (socket says 172.21.15.46:38152) 2024-01-27T00:47:14.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.287+0000 7fa2977fe700 1 -- 172.21.15.46:0/1449288846 learned_addr learned my addr 172.21.15.46:0/1449288846 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:14.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.287+0000 7fa297fff700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fa298157340 0x7fa29815bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:45474/0 (socket says 172.21.15.46:45474) 2024-01-27T00:47:14.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.287+0000 7fa2977fe700 1 -- 172.21.15.46:0/1449288846 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2980b3fb0 msgr2=0x7fa298157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:14.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.287+0000 7fa2977fe700 1 --2- 172.21.15.46:0/1449288846 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2980b3fb0 0x7fa298157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.287+0000 7fa2977fe700 1 -- 172.21.15.46:0/1449288846 >> v1:172.21.15.106:6789/0 conn(0x7fa298157340 legacy=0x7fa29815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:14.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.287+0000 7fa2977fe700 1 -- 172.21.15.46:0/1449288846 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa298158520 con 0x7fa2980ae520 2024-01-27T00:47:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.287+0000 7fa296ffd700 1 -- 172.21.15.46:0/1449288846 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522515642 0 0) 0x7fa288001110 con 0x7fa298157340 2024-01-27T00:47:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.287+0000 7fa29ce78700 1 --2- 172.21.15.46:0/1449288846 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2980b3fb0 0x7fa298157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.287+0000 7fa2977fe700 1 --2- 172.21.15.46:0/1449288846 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2980ae520 0x7fa2980b3a70 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fa2800099b0 tx=0x7fa280009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b89f0978bfd704 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.288+0000 7fa296ffd700 1 -- 172.21.15.46:0/1449288846 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa280009ea0 con 0x7fa2980ae520 2024-01-27T00:47:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.288+0000 7fa296ffd700 1 -- 172.21.15.46:0/1449288846 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28000b860 con 0x7fa2980ae520 2024-01-27T00:47:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.288+0000 7fa296ffd700 1 -- 172.21.15.46:0/1449288846 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28000bc20 con 0x7fa2980ae520 2024-01-27T00:47:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.288+0000 7fa29e8db700 1 -- 172.21.15.46:0/1449288846 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2980ae520 msgr2=0x7fa2980b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.288+0000 7fa29e8db700 1 --2- 172.21.15.46:0/1449288846 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2980ae520 0x7fa2980b3a70 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fa2800099b0 tx=0x7fa280009250 comp rx=0 tx=0).stop 2024-01-27T00:47:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.288+0000 7fa29e8db700 1 -- 172.21.15.46:0/1449288846 shutdown_connections 2024-01-27T00:47:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.288+0000 7fa29e8db700 1 --2- 172.21.15.46:0/1449288846 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2980b3fb0 0x7fa298157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.288+0000 7fa29e8db700 1 --2- 172.21.15.46:0/1449288846 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2980ae520 0x7fa2980b3a70 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.288+0000 7fa29e8db700 1 -- 172.21.15.46:0/1449288846 >> 172.21.15.46:0/1449288846 conn(0x7fa2980ce530 msgr2=0x7fa2980ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:14.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.288+0000 7fa29e8db700 1 -- 172.21.15.46:0/1449288846 shutdown_connections 2024-01-27T00:47:14.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.288+0000 7fa29e8db700 1 -- 172.21.15.46:0/1449288846 wait complete. 2024-01-27T00:47:14.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.290+0000 7fa29e8db700 1 Processor -- start 2024-01-27T00:47:14.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.290+0000 7fa29e8db700 1 -- start start 2024-01-27T00:47:14.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa29e8db700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2980ae520 0x7fa298169940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:14.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa29e8db700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2980b3fb0 0x7fa2980696c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:14.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa29e8db700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa298157340 0x7fa2980c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:14.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa29e8db700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2980c4e50 con 0x7fa2980ae520 2024-01-27T00:47:14.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa29e8db700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa298069230 con 0x7fa298157340 2024-01-27T00:47:14.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa29e8db700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2980c4b70 con 0x7fa2980b3fb0 2024-01-27T00:47:14.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa297fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2980ae520 0x7fa298169940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa297fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2980ae520 0x7fa298169940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38852/0 (socket says 172.21.15.46:38852) 2024-01-27T00:47:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa297fff700 1 -- 172.21.15.46:0/1840443708 learned_addr learned my addr 172.21.15.46:0/1840443708 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa2977fe700 1 --2- 172.21.15.46:0/1840443708 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2980b3fb0 0x7fa2980696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa29ce78700 1 --2- 172.21.15.46:0/1840443708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa298157340 0x7fa2980c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa297fff700 1 -- 172.21.15.46:0/1840443708 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2980b3fb0 msgr2=0x7fa2980696c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa297fff700 1 --2- 172.21.15.46:0/1840443708 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2980b3fb0 0x7fa2980696c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa297fff700 1 -- 172.21.15.46:0/1840443708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa298157340 msgr2=0x7fa2980c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa297fff700 1 --2- 172.21.15.46:0/1840443708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa298157340 0x7fa2980c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa297fff700 1 -- 172.21.15.46:0/1840443708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa290009a20 con 0x7fa2980ae520 2024-01-27T00:47:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa2977fe700 1 --2- 172.21.15.46:0/1840443708 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2980b3fb0 0x7fa2980696c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:47:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa29ce78700 1 --2- 172.21.15.46:0/1840443708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa298157340 0x7fa2980c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:47:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa297fff700 1 --2- 172.21.15.46:0/1840443708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2980ae520 0x7fa298169940 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fa28800c810 tx=0x7fa288007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa28ffff700 1 -- 172.21.15.46:0/1840443708 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa288007300 con 0x7fa2980ae520 2024-01-27T00:47:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.291+0000 7fa28ffff700 1 -- 172.21.15.46:0/1840443708 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa288007480 con 0x7fa2980ae520 2024-01-27T00:47:14.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.292+0000 7fa28ffff700 1 -- 172.21.15.46:0/1840443708 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa288004830 con 0x7fa2980ae520 2024-01-27T00:47:14.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.292+0000 7fa29e8db700 1 -- 172.21.15.46:0/1840443708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2800094f0 con 0x7fa2980ae520 2024-01-27T00:47:14.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.292+0000 7fa29e8db700 1 -- 172.21.15.46:0/1840443708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa298068ec0 con 0x7fa2980ae520 2024-01-27T00:47:14.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.293+0000 7fa29e8db700 1 -- 172.21.15.46:0/1840443708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2980b0e20 con 0x7fa2980ae520 2024-01-27T00:47:14.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.293+0000 7fa28ffff700 1 -- 172.21.15.46:0/1840443708 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa288004be0 con 0x7fa2980ae520 2024-01-27T00:47:14.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.294+0000 7fa28ffff700 1 --2- 172.21.15.46:0/1840443708 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fa278066f30 0x7fa2780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:14.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.294+0000 7fa28ffff700 1 -- 172.21.15.46:0/1840443708 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6211+0+0 (secure 0 0 0) 0x7fa2880837c0 con 0x7fa2980ae520 2024-01-27T00:47:14.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.294+0000 7fa2977fe700 1 --2- 172.21.15.46:0/1840443708 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fa278066f30 0x7fa2780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:14.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.294+0000 7fa2977fe700 1 --2- 172.21.15.46:0/1840443708 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fa278066f30 0x7fa2780693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa2800099b0 tx=0x7fa280017000 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:14.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.298+0000 7fa28ffff700 1 -- 172.21.15.46:0/1840443708 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa288050830 con 0x7fa2980ae520 2024-01-27T00:47:14.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.550+0000 7fa29e8db700 1 -- 172.21.15.46:0/1840443708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2980b4410 con 0x7fa2980ae520 2024-01-27T00:47:14.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.551+0000 7fa28ffff700 1 -- 172.21.15.46:0/1840443708 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+3532 (secure 0 0 0) 0x7fa288018de0 con 0x7fa2980ae520 2024-01-27T00:47:14.551 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:14.551 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":90,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8345,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/201150167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":201150167},{"type":"v1","addr":"172.21.15.106:6835","nonce":201150167}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":9061,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/3698053981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":3698053981},{"type":"v1","addr":"172.21.15.106:6833","nonce":3698053981}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":9064,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2056976879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2056976879},{"type":"v1","addr":"172.21.15.46:6834","nonce":2056976879}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":82},{"gid":9067,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1576406314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1576406314},{"type":"v1","addr":"172.21.15.46:6835","nonce":1576406314}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":82}],"filesystems":[]} 2024-01-27T00:47:14.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.553+0000 7fa29e8db700 1 -- 172.21.15.46:0/1840443708 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fa278066f30 msgr2=0x7fa2780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:14.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.553+0000 7fa29e8db700 1 --2- 172.21.15.46:0/1840443708 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fa278066f30 0x7fa2780693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa2800099b0 tx=0x7fa280017000 comp rx=0 tx=0).stop 2024-01-27T00:47:14.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.553+0000 7fa29e8db700 1 -- 172.21.15.46:0/1840443708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2980ae520 msgr2=0x7fa298169940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:14.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.553+0000 7fa29e8db700 1 --2- 172.21.15.46:0/1840443708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2980ae520 0x7fa298169940 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fa28800c810 tx=0x7fa288007c50 comp rx=0 tx=0).stop 2024-01-27T00:47:14.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.553+0000 7fa29e8db700 1 -- 172.21.15.46:0/1840443708 shutdown_connections 2024-01-27T00:47:14.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.553+0000 7fa29e8db700 1 --2- 172.21.15.46:0/1840443708 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fa278066f30 0x7fa2780693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.553+0000 7fa29e8db700 1 --2- 172.21.15.46:0/1840443708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2980ae520 0x7fa298169940 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.553+0000 7fa29e8db700 1 --2- 172.21.15.46:0/1840443708 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa298157340 0x7fa2980c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.553+0000 7fa29e8db700 1 --2- 172.21.15.46:0/1840443708 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2980b3fb0 0x7fa2980696c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.553+0000 7fa29e8db700 1 -- 172.21.15.46:0/1840443708 >> 172.21.15.46:0/1840443708 conn(0x7fa2980ce530 msgr2=0x7fa2980ade00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:14.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.553+0000 7fa29e8db700 1 -- 172.21.15.46:0/1840443708 shutdown_connections 2024-01-27T00:47:14.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.553+0000 7fa29e8db700 1 -- 172.21.15.46:0/1840443708 wait complete. 2024-01-27T00:47:14.555 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 90 2024-01-27T00:47:14.567 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-27T00:47:14.567 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-27T00:47:14.568 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-27T00:47:14.568 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T00:47:14.568 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-27T00:47:14.568 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-27T00:47:14.568 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-27T00:47:14.568 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T00:47:14.569 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-27T00:47:14.569 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-27T00:47:14.569 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-27T00:47:14.569 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T00:47:14.569 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-27T00:47:14.569 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-27T00:47:14.569 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-27T00:47:14.569 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T00:47:14.577 INFO:tasks.ceph.mds.a:Stopped 2024-01-27T00:47:14.577 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-27T00:47:14.578 INFO:tasks.ceph.mds.c:Stopped 2024-01-27T00:47:14.578 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-27T00:47:14.580 INFO:tasks.ceph.mds.b:Stopped 2024-01-27T00:47:14.580 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-27T00:47:14.581 INFO:tasks.ceph.mds.d:Stopped 2024-01-27T00:47:14.581 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-27T00:47:14.621 INFO:tasks.ceph.mds.a:Started 2024-01-27T00:47:14.621 DEBUG:teuthology.parallel:result is None 2024-01-27T00:47:14.622 INFO:tasks.ceph.mds.c:Started 2024-01-27T00:47:14.622 DEBUG:teuthology.parallel:result is None 2024-01-27T00:47:14.623 INFO:tasks.ceph.mds.b:Started 2024-01-27T00:47:14.623 DEBUG:teuthology.parallel:result is None 2024-01-27T00:47:14.624 INFO:tasks.ceph.mds.d:Started 2024-01-27T00:47:14.624 DEBUG:teuthology.parallel:result is None 2024-01-27T00:47:14.624 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-27T00:47:14.707 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-01-27T00:47:14.708 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-27T00:47:14.708 INFO:tasks.ceph.mds.b.smithi106.stdout:starting mds.b at 2024-01-27T00:47:14.708 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-27T00:47:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.765+0000 7fc9941a8700 1 Processor -- start 2024-01-27T00:47:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.765+0000 7fc9941a8700 1 -- start start 2024-01-27T00:47:14.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.766+0000 7fc9941a8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc98c157340 0x7fc98c15bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:14.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.766+0000 7fc9941a8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc98c0ae520 0x7fc98c0b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:14.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.766+0000 7fc9941a8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98c158ae0 con 0x7fc98c0b3fb0 2024-01-27T00:47:14.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.766+0000 7fc9941a8700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc98c0b4430 con 0x7fc98c157340 2024-01-27T00:47:14.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.766+0000 7fc9941a8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c15b650 con 0x7fc98c0ae520 2024-01-27T00:47:14.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.766+0000 7fc991f44700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc98c157340 0x7fc98c15bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:14.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.766+0000 7fc992745700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc98c0b3fb0 0x7fc98c157710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40980/0 (socket says 172.21.15.46:40980) 2024-01-27T00:47:14.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.766+0000 7fc992745700 1 -- 172.21.15.46:0/1925380113 learned_addr learned my addr 172.21.15.46:0/1925380113 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:14.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.766+0000 7fc991f44700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc98c157340 0x7fc98c15bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38896/0 (socket says 172.21.15.46:38896) 2024-01-27T00:47:14.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.766+0000 7fc991743700 1 --2- 172.21.15.46:0/1925380113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc98c0ae520 0x7fc98c0b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:14.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.766+0000 7fc991f44700 1 -- 172.21.15.46:0/1925380113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc98c0ae520 msgr2=0x7fc98c0b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:14.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.766+0000 7fc991f44700 1 --2- 172.21.15.46:0/1925380113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc98c0ae520 0x7fc98c0b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.766+0000 7fc991f44700 1 -- 172.21.15.46:0/1925380113 >> v1:172.21.15.46:6789/0 conn(0x7fc98c0b3fb0 legacy=0x7fc98c157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:14.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.766+0000 7fc991f44700 1 -- 172.21.15.46:0/1925380113 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98c15c020 con 0x7fc98c157340 2024-01-27T00:47:14.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.766+0000 7fc990f42700 1 -- 172.21.15.46:0/1925380113 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981222272 0 0) 0x7fc98c158ae0 con 0x7fc98c0b3fb0 2024-01-27T00:47:14.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.766+0000 7fc991f44700 1 --2- 172.21.15.46:0/1925380113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c157340 0x7fc98c15bae0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0098e0 tx=0x7fc97c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c8fcedfb86f803c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:14.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.767+0000 7fc990f42700 1 -- 172.21.15.46:0/1925380113 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c009e80 con 0x7fc98c157340 2024-01-27T00:47:14.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.767+0000 7fc990f42700 1 -- 172.21.15.46:0/1925380113 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97c00b860 con 0x7fc98c157340 2024-01-27T00:47:14.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.767+0000 7fc990f42700 1 -- 172.21.15.46:0/1925380113 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c00bbe0 con 0x7fc98c157340 2024-01-27T00:47:14.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.767+0000 7fc9941a8700 1 -- 172.21.15.46:0/1925380113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c157340 msgr2=0x7fc98c15bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:14.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.767+0000 7fc9941a8700 1 --2- 172.21.15.46:0/1925380113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c157340 0x7fc98c15bae0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0098e0 tx=0x7fc97c0091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.767+0000 7fc9941a8700 1 -- 172.21.15.46:0/1925380113 shutdown_connections 2024-01-27T00:47:14.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.767+0000 7fc9941a8700 1 --2- 172.21.15.46:0/1925380113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c157340 0x7fc98c15bae0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.767+0000 7fc9941a8700 1 --2- 172.21.15.46:0/1925380113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc98c0ae520 0x7fc98c0b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.767+0000 7fc9941a8700 1 -- 172.21.15.46:0/1925380113 >> 172.21.15.46:0/1925380113 conn(0x7fc98c0ce530 msgr2=0x7fc98c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:14.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.767+0000 7fc9941a8700 1 -- 172.21.15.46:0/1925380113 shutdown_connections 2024-01-27T00:47:14.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.767+0000 7fc9941a8700 1 -- 172.21.15.46:0/1925380113 wait complete. 2024-01-27T00:47:14.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.768+0000 7fc9941a8700 1 Processor -- start 2024-01-27T00:47:14.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.769+0000 7fc9941a8700 1 -- start start 2024-01-27T00:47:14.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.769+0000 7fc9941a8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc98c0ae520 0x7fc98c16a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:14.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.769+0000 7fc9941a8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc98c0b3fb0 0x7fc98c156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:14.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.769+0000 7fc9941a8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c157340 0x7fc98c1598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:14.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.769+0000 7fc9941a8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c16ae70 con 0x7fc98c157340 2024-01-27T00:47:14.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.769+0000 7fc9941a8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c156f90 con 0x7fc98c0ae520 2024-01-27T00:47:14.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.769+0000 7fc9941a8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c16b060 con 0x7fc98c0b3fb0 2024-01-27T00:47:14.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.769+0000 7fc992745700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c157340 0x7fc98c1598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:14.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.769+0000 7fc992745700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c157340 0x7fc98c1598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38900/0 (socket says 172.21.15.46:38900) 2024-01-27T00:47:14.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.769+0000 7fc992745700 1 -- 172.21.15.46:0/209685792 learned_addr learned my addr 172.21.15.46:0/209685792 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:14.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.770+0000 7fc991743700 1 --2- 172.21.15.46:0/209685792 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc98c0b3fb0 0x7fc98c156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:14.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.770+0000 7fc991f44700 1 --2- 172.21.15.46:0/209685792 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc98c0ae520 0x7fc98c16a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:14.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.770+0000 7fc992745700 1 -- 172.21.15.46:0/209685792 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc98c0b3fb0 msgr2=0x7fc98c156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:14.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.770+0000 7fc992745700 1 --2- 172.21.15.46:0/209685792 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc98c0b3fb0 0x7fc98c156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.770+0000 7fc992745700 1 -- 172.21.15.46:0/209685792 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc98c0ae520 msgr2=0x7fc98c16a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:14.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.770+0000 7fc992745700 1 --2- 172.21.15.46:0/209685792 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc98c0ae520 0x7fc98c16a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:14.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.770+0000 7fc992745700 1 -- 172.21.15.46:0/209685792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97c009420 con 0x7fc98c157340 2024-01-27T00:47:14.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.770+0000 7fc992745700 1 --2- 172.21.15.46:0/209685792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c157340 0x7fc98c1598d0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fc98400ccc0 tx=0x7fc984001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:14.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.770+0000 7fc981ffb700 1 -- 172.21.15.46:0/209685792 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98400a8a0 con 0x7fc98c157340 2024-01-27T00:47:14.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.770+0000 7fc981ffb700 1 -- 172.21.15.46:0/209685792 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc98400aa00 con 0x7fc98c157340 2024-01-27T00:47:14.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.770+0000 7fc9941a8700 1 -- 172.21.15.46:0/209685792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc98c0ac830 con 0x7fc98c157340 2024-01-27T00:47:14.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.770+0000 7fc981ffb700 1 -- 172.21.15.46:0/209685792 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98400adc0 con 0x7fc98c157340 2024-01-27T00:47:14.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.770+0000 7fc9941a8700 1 -- 172.21.15.46:0/209685792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc98c0acdf0 con 0x7fc98c157340 2024-01-27T00:47:14.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.771+0000 7fc9941a8700 1 -- 172.21.15.46:0/209685792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc98c0b0f10 con 0x7fc98c157340 2024-01-27T00:47:14.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.772+0000 7fc981ffb700 1 -- 172.21.15.46:0/209685792 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc984007300 con 0x7fc98c157340 2024-01-27T00:47:14.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.772+0000 7fc981ffb700 1 --2- 172.21.15.46:0/209685792 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fc970066f30 0x7fc9700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:14.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.772+0000 7fc981ffb700 1 -- 172.21.15.46:0/209685792 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6211+0+0 (secure 0 0 0) 0x7fc984084ab0 con 0x7fc98c157340 2024-01-27T00:47:14.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.772+0000 7fc991f44700 1 --2- 172.21.15.46:0/209685792 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fc970066f30 0x7fc9700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:14.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.773+0000 7fc991f44700 1 --2- 172.21.15.46:0/209685792 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fc970066f30 0x7fc9700693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0093a0 tx=0x7fc97c017000 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:14.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.777+0000 7fc981ffb700 1 -- 172.21.15.46:0/209685792 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc984051b00 con 0x7fc98c157340 2024-01-27T00:47:14.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:14.997+0000 7fc9941a8700 1 -- 172.21.15.46:0/209685792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc98c04c710 con 0x7fc98c157340 2024-01-27T00:47:15.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.671+0000 7fc981ffb700 1 -- 172.21.15.46:0/209685792 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v182) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc984055120 con 0x7fc98c157340 2024-01-27T00:47:15.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.675+0000 7fc9941a8700 1 -- 172.21.15.46:0/209685792 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fc970066f30 msgr2=0x7fc9700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:15.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.675+0000 7fc9941a8700 1 --2- 172.21.15.46:0/209685792 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fc970066f30 0x7fc9700693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0093a0 tx=0x7fc97c017000 comp rx=0 tx=0).stop 2024-01-27T00:47:15.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.675+0000 7fc9941a8700 1 -- 172.21.15.46:0/209685792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c157340 msgr2=0x7fc98c1598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:15.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.675+0000 7fc9941a8700 1 --2- 172.21.15.46:0/209685792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c157340 0x7fc98c1598d0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fc98400ccc0 tx=0x7fc984001a80 comp rx=0 tx=0).stop 2024-01-27T00:47:15.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.675+0000 7fc9941a8700 1 -- 172.21.15.46:0/209685792 shutdown_connections 2024-01-27T00:47:15.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.675+0000 7fc9941a8700 1 --2- 172.21.15.46:0/209685792 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fc970066f30 0x7fc9700693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:15.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.675+0000 7fc9941a8700 1 --2- 172.21.15.46:0/209685792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c157340 0x7fc98c1598d0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:15.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.675+0000 7fc9941a8700 1 --2- 172.21.15.46:0/209685792 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc98c0ae520 0x7fc98c16a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:15.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.675+0000 7fc9941a8700 1 --2- 172.21.15.46:0/209685792 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc98c0b3fb0 0x7fc98c156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:15.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.675+0000 7fc9941a8700 1 -- 172.21.15.46:0/209685792 >> 172.21.15.46:0/209685792 conn(0x7fc98c0ce530 msgr2=0x7fc98c0addf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:15.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.675+0000 7fc9941a8700 1 -- 172.21.15.46:0/209685792 shutdown_connections 2024-01-27T00:47:15.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.675+0000 7fc9941a8700 1 -- 172.21.15.46:0/209685792 wait complete. 2024-01-27T00:47:15.677 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-01-27T00:47:15.693 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-27T00:47:15.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.828+0000 7f88c2b10700 1 Processor -- start 2024-01-27T00:47:15.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.828+0000 7f88c2b10700 1 -- start start 2024-01-27T00:47:15.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.829+0000 7f88c2b10700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88bc159590 0x7f88bc15d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:15.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.829+0000 7f88c2b10700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f88bc15d8f0 0x7f88bc06bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:15.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.829+0000 7f88c2b10700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88bc15dd90 con 0x7f88bc159960 2024-01-27T00:47:15.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.829+0000 7f88c2b10700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88bc15e350 con 0x7f88bc15d8f0 2024-01-27T00:47:15.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.829+0000 7f88c2b10700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc0c97c0 con 0x7f88bc159590 2024-01-27T00:47:15.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.829+0000 7f88bbfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f88bc15d8f0 0x7f88bc06bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:15.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.829+0000 7f88bbfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f88bc15d8f0 0x7f88bc06bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38908/0 (socket says 172.21.15.46:38908) 2024-01-27T00:47:15.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.829+0000 7f88bbfff700 1 -- 172.21.15.46:0/517273288 learned_addr learned my addr 172.21.15.46:0/517273288 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:15.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.829+0000 7f88c10ad700 1 --1- 172.21.15.46:0/517273288 >> v1:172.21.15.46:6789/0 conn(0x7f88bc159960 0x7f88bc06d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40982/0 (socket says 172.21.15.46:40982) 2024-01-27T00:47:15.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.829+0000 7f88c08ac700 1 --2- 172.21.15.46:0/517273288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88bc159590 0x7f88bc15d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:15.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.830+0000 7f88bbfff700 1 -- 172.21.15.46:0/517273288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88bc159590 msgr2=0x7f88bc15d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:15.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.830+0000 7f88bbfff700 1 --2- 172.21.15.46:0/517273288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88bc159590 0x7f88bc15d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:15.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.830+0000 7f88bbfff700 1 -- 172.21.15.46:0/517273288 >> v1:172.21.15.46:6789/0 conn(0x7f88bc159960 legacy=0x7f88bc06d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:15.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.830+0000 7f88bbfff700 1 -- 172.21.15.46:0/517273288 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88bc15e070 con 0x7f88bc15d8f0 2024-01-27T00:47:15.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.830+0000 7f88c08ac700 1 --2- 172.21.15.46:0/517273288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88bc159590 0x7f88bc15d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:47:15.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.830+0000 7f88bbfff700 1 --2- 172.21.15.46:0/517273288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc15d8f0 0x7f88bc06bcb0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f88a40099e0 tx=0x7f88a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d99adad7269362 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:15.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.830+0000 7f88bb7fe700 1 -- 172.21.15.46:0/517273288 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a400b820 con 0x7f88bc15d8f0 2024-01-27T00:47:15.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.830+0000 7f88bb7fe700 1 -- 172.21.15.46:0/517273288 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a400b980 con 0x7f88bc15d8f0 2024-01-27T00:47:15.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.830+0000 7f88bb7fe700 1 -- 172.21.15.46:0/517273288 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a400bd40 con 0x7f88bc15d8f0 2024-01-27T00:47:15.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.831+0000 7f88c2b10700 1 -- 172.21.15.46:0/517273288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc15d8f0 msgr2=0x7f88bc06bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:15.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.831+0000 7f88c2b10700 1 --2- 172.21.15.46:0/517273288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc15d8f0 0x7f88bc06bcb0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f88a40099e0 tx=0x7f88a4009250 comp rx=0 tx=0).stop 2024-01-27T00:47:15.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.831+0000 7f88c2b10700 1 -- 172.21.15.46:0/517273288 shutdown_connections 2024-01-27T00:47:15.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.831+0000 7f88c2b10700 1 --2- 172.21.15.46:0/517273288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88bc159590 0x7f88bc15d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:15.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.831+0000 7f88c2b10700 1 --2- 172.21.15.46:0/517273288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc15d8f0 0x7f88bc06bcb0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:15.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.831+0000 7f88c2b10700 1 -- 172.21.15.46:0/517273288 >> 172.21.15.46:0/517273288 conn(0x7f88bc0ce530 msgr2=0x7f88bc0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:15.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.831+0000 7f88c2b10700 1 -- 172.21.15.46:0/517273288 shutdown_connections 2024-01-27T00:47:15.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.831+0000 7f88c2b10700 1 -- 172.21.15.46:0/517273288 wait complete. 2024-01-27T00:47:15.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.832+0000 7f88c2b10700 1 Processor -- start 2024-01-27T00:47:15.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.832+0000 7f88c2b10700 1 -- start start 2024-01-27T00:47:15.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88c2b10700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88bc159590 0x7f88bc0b2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:15.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88c2b10700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc159960 0x7f88bc0c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:15.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88c2b10700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88bc15d8f0 0x7f88bc0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:15.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88c2b10700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc0b21d0 con 0x7f88bc159960 2024-01-27T00:47:15.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88c2b10700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc0b12e0 con 0x7f88bc15d8f0 2024-01-27T00:47:15.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88c2b10700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc0b15c0 con 0x7f88bc159590 2024-01-27T00:47:15.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88bbfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc159960 0x7f88bc0c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:15.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88bbfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc159960 0x7f88bc0c6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38916/0 (socket says 172.21.15.46:38916) 2024-01-27T00:47:15.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88bbfff700 1 -- 172.21.15.46:0/2807184205 learned_addr learned my addr 172.21.15.46:0/2807184205 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:15.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88c10ad700 1 --2- 172.21.15.46:0/2807184205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88bc15d8f0 0x7f88bc0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:15.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88c08ac700 1 --2- 172.21.15.46:0/2807184205 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88bc159590 0x7f88bc0b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:15.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88bbfff700 1 -- 172.21.15.46:0/2807184205 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88bc159590 msgr2=0x7f88bc0b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:15.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88bbfff700 1 --2- 172.21.15.46:0/2807184205 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88bc159590 0x7f88bc0b2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:15.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88bbfff700 1 -- 172.21.15.46:0/2807184205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88bc15d8f0 msgr2=0x7f88bc0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:15.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88bbfff700 1 --2- 172.21.15.46:0/2807184205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88bc15d8f0 0x7f88bc0b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:15.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88bbfff700 1 -- 172.21.15.46:0/2807184205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a40094f0 con 0x7f88bc159960 2024-01-27T00:47:15.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88c08ac700 1 --2- 172.21.15.46:0/2807184205 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88bc159590 0x7f88bc0b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:47:15.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.833+0000 7f88c10ad700 1 --2- 172.21.15.46:0/2807184205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88bc15d8f0 0x7f88bc0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:47:15.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.834+0000 7f88bbfff700 1 --2- 172.21.15.46:0/2807184205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc159960 0x7f88bc0c6eb0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f88a4009930 tx=0x7f88a4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:15.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.834+0000 7f88b8ff9700 1 -- 172.21.15.46:0/2807184205 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a400b820 con 0x7f88bc159960 2024-01-27T00:47:15.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.834+0000 7f88b8ff9700 1 -- 172.21.15.46:0/2807184205 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a4002f30 con 0x7f88bc159960 2024-01-27T00:47:15.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.834+0000 7f88b8ff9700 1 -- 172.21.15.46:0/2807184205 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a4025420 con 0x7f88bc159960 2024-01-27T00:47:15.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.834+0000 7f88c2b10700 1 -- 172.21.15.46:0/2807184205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88bc0b18a0 con 0x7f88bc159960 2024-01-27T00:47:15.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.834+0000 7f88c2b10700 1 -- 172.21.15.46:0/2807184205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88bc0b1e60 con 0x7f88bc159960 2024-01-27T00:47:15.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.835+0000 7f88c2b10700 1 -- 172.21.15.46:0/2807184205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88bc04c710 con 0x7f88bc159960 2024-01-27T00:47:15.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.835+0000 7f88b8ff9700 1 -- 172.21.15.46:0/2807184205 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f88a4023070 con 0x7f88bc159960 2024-01-27T00:47:15.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.836+0000 7f88b8ff9700 1 --2- 172.21.15.46:0/2807184205 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f889c066f30 0x7f889c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:15.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.836+0000 7f88b8ff9700 1 -- 172.21.15.46:0/2807184205 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f88a40858b0 con 0x7f88bc159960 2024-01-27T00:47:15.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.836+0000 7f88c08ac700 1 --2- 172.21.15.46:0/2807184205 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f889c066f30 0x7f889c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:15.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.837+0000 7f88c08ac700 1 --2- 172.21.15.46:0/2807184205 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f889c066f30 0x7f889c0693e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f88ac006f00 tx=0x7f88ac008040 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:15.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:15.841+0000 7f88b8ff9700 1 -- 172.21.15.46:0/2807184205 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f88a4052b80 con 0x7f88bc159960 2024-01-27T00:47:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.128+0000 7f88c2b10700 1 -- 172.21.15.46:0/2807184205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f88bc068b70 con 0x7f88bc159960 2024-01-27T00:47:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.128+0000 7f88b8ff9700 1 -- 172.21.15.46:0/2807184205 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v45) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f88a40561a0 con 0x7f88bc159960 2024-01-27T00:47:16.129 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:16.129 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T00:47:16.129 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-01-27T00:47:16.129 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.129 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-01-27T00:47:16.129 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlM6dKKBAAPpeWeDZWrFgc/4zuxKCvHw==", 2024-01-27T00:47:16.129 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.129 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T00:47:16.129 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T00:47:16.129 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T00:47:16.130 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:47:16.130 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.130 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.130 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.130 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-01-27T00:47:16.130 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlmzHRNRAAflWYZ/p4AXeMy54tvLpugA==", 2024-01-27T00:47:16.130 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.130 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T00:47:16.130 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T00:47:16.130 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T00:47:16.130 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:47:16.130 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.130 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.130 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.130 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-01-27T00:47:16.130 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlMZxoMBAAG5kyfm+LLG86y/p2A8BfDQ==", 2024-01-27T00:47:16.131 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.131 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T00:47:16.131 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T00:47:16.131 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T00:47:16.131 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:47:16.131 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.131 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.131 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.131 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-01-27T00:47:16.131 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl5jx/ARAA3cwtLAOWQ/n2NG+xwEZ3QQ==", 2024-01-27T00:47:16.131 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.131 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T00:47:16.131 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T00:47:16.131 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T00:47:16.131 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:47:16.132 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.132 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.132 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.132 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-01-27T00:47:16.132 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4QLRlJ2OKJRAAWVhxEdQkUW20294w/KLL6g==", 2024-01-27T00:47:16.132 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.132 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T00:47:16.132 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T00:47:16.132 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:47:16.132 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.132 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.132 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.132 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-01-27T00:47:16.132 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7QLRlE2tAJRAAzNlTiCr6a2eWiabRy1SqSw==", 2024-01-27T00:47:16.133 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.133 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T00:47:16.133 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T00:47:16.133 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:47:16.133 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.133 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.133 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.133 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-01-27T00:47:16.133 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+QLRlyySaJBAA5072ba5HUNi2dYE3D02M/Q==", 2024-01-27T00:47:16.133 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.133 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T00:47:16.133 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T00:47:16.133 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:47:16.133 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.133 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.134 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.134 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-01-27T00:47:16.134 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBQLRl9+tAIxAApZk0btXIpQPWSb3BYUYsFQ==", 2024-01-27T00:47:16.134 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.134 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T00:47:16.134 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T00:47:16.134 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:47:16.134 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.134 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.134 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.134 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-01-27T00:47:16.134 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHQLRlurUtJxAAT9fwKrAWl62VOXwShg7NJg==", 2024-01-27T00:47:16.134 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.134 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T00:47:16.134 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T00:47:16.135 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:47:16.135 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.135 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.135 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.135 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-01-27T00:47:16.135 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKQLRl7aqoJRAAXu9fykWELsW3SwnahoIOLA==", 2024-01-27T00:47:16.135 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.135 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T00:47:16.135 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T00:47:16.135 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:47:16.135 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.135 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.135 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.135 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-01-27T00:47:16.136 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNQLRlWf7pJBAAopmFLIKb8s32YZs46VwbGg==", 2024-01-27T00:47:16.136 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.136 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T00:47:16.136 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T00:47:16.136 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:47:16.136 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.136 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.136 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.136 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-01-27T00:47:16.136 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQQLRlNGfhIxAAmCgmOB3QAGfSIYHBb5BUZw==", 2024-01-27T00:47:16.136 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.136 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T00:47:16.136 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T00:47:16.136 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:47:16.137 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.137 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.137 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.137 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-01-27T00:47:16.137 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRleO8wBhAA1018AcO3KcyOZxgB0Itxaw==", 2024-01-27T00:47:16.137 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.137 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T00:47:16.137 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-01-27T00:47:16.137 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-27T00:47:16.137 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.137 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.137 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.137 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-01-27T00:47:16.137 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlsiZEDRAAjXSamHe2bZcDqVsqXqIcGg==", 2024-01-27T00:47:16.138 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.138 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T00:47:16.138 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T00:47:16.138 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T00:47:16.138 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T00:47:16.138 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.138 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.138 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.138 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-01-27T00:47:16.138 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl6YT1CRAApXlN4j20xzwda0tDMy94Eg==", 2024-01-27T00:47:16.138 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.138 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T00:47:16.138 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T00:47:16.138 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T00:47:16.139 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T00:47:16.139 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.139 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.139 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.139 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-01-27T00:47:16.139 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlHvDQEhAAxMhgWJsZC5/6Af5zFV6XIA==", 2024-01-27T00:47:16.139 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.139 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T00:47:16.139 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T00:47:16.139 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T00:47:16.139 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T00:47:16.139 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.139 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.140 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.140 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-01-27T00:47:16.140 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlvMwDCBAAXnFyMduz7PfWdLWbjytIiQ==", 2024-01-27T00:47:16.140 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.140 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T00:47:16.140 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T00:47:16.140 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T00:47:16.140 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:47:16.140 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.140 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.140 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.140 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-01-27T00:47:16.140 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlhMNSJBAA38QU6DrCvFjO+u9zQ8bkEA==", 2024-01-27T00:47:16.140 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.141 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T00:47:16.141 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-27T00:47:16.141 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:47:16.141 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.141 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:16.141 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:16.141 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-01-27T00:47:16.141 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlI9EbIBAACTpzAO+8A/VFJZcT/WWBng==", 2024-01-27T00:47:16.141 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T00:47:16.141 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T00:47:16.141 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-27T00:47:16.141 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T00:47:16.141 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.141 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:16.142 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-01-27T00:47:16.142 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T00:47:16.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.130+0000 7f88c2b10700 1 -- 172.21.15.46:0/2807184205 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f889c066f30 msgr2=0x7f889c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:16.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.130+0000 7f88c2b10700 1 --2- 172.21.15.46:0/2807184205 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f889c066f30 0x7f889c0693e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f88ac006f00 tx=0x7f88ac008040 comp rx=0 tx=0).stop 2024-01-27T00:47:16.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.130+0000 7f88c2b10700 1 -- 172.21.15.46:0/2807184205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc159960 msgr2=0x7f88bc0c6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:16.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.130+0000 7f88c2b10700 1 --2- 172.21.15.46:0/2807184205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc159960 0x7f88bc0c6eb0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f88a4009930 tx=0x7f88a4017000 comp rx=0 tx=0).stop 2024-01-27T00:47:16.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.130+0000 7f88c2b10700 1 -- 172.21.15.46:0/2807184205 shutdown_connections 2024-01-27T00:47:16.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.130+0000 7f88c2b10700 1 --2- 172.21.15.46:0/2807184205 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f889c066f30 0x7f889c0693e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.130+0000 7f88c2b10700 1 --2- 172.21.15.46:0/2807184205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88bc15d8f0 0x7f88bc0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.130+0000 7f88c2b10700 1 --2- 172.21.15.46:0/2807184205 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88bc159590 0x7f88bc0b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.130+0000 7f88c2b10700 1 --2- 172.21.15.46:0/2807184205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc159960 0x7f88bc0c6eb0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.130+0000 7f88c2b10700 1 -- 172.21.15.46:0/2807184205 >> 172.21.15.46:0/2807184205 conn(0x7f88bc0ce530 msgr2=0x7f88bc06de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:16.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.131+0000 7f88c2b10700 1 -- 172.21.15.46:0/2807184205 shutdown_connections 2024-01-27T00:47:16.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.131+0000 7f88c2b10700 1 -- 172.21.15.46:0/2807184205 wait complete. 2024-01-27T00:47:16.145 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T00:47:16.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.275+0000 7feded098700 1 Processor -- start 2024-01-27T00:47:16.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.276+0000 7feded098700 1 -- start start 2024-01-27T00:47:16.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.276+0000 7feded098700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fede815d210 0x7fede80cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:16.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.276+0000 7feded098700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fede80cbff0 0x7fede80ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:16.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.276+0000 7feded098700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede80cff70 con 0x7fede815ce40 2024-01-27T00:47:16.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.276+0000 7feded098700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede80cc490 con 0x7fede815d210 2024-01-27T00:47:16.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.276+0000 7feded098700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede80cf400 con 0x7fede80cbff0 2024-01-27T00:47:16.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.277+0000 7fede6d9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fede815ce40 0x7fede80d0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40988/0 (socket says 172.21.15.46:40988) 2024-01-27T00:47:16.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.277+0000 7fede6d9d700 1 -- 172.21.15.46:0/917049243 learned_addr learned my addr 172.21.15.46:0/917049243 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:16.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.277+0000 7fede759e700 1 --2- 172.21.15.46:0/917049243 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fede80cbff0 0x7fede80ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:16.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.277+0000 7fede659c700 1 --2- 172.21.15.46:0/917049243 >> v2:172.21.15.106:3300/0 conn(0x7fede815d210 0x7fede80cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:16.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.277+0000 7fede5d9b700 1 -- 172.21.15.46:0/917049243 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594217380 0 0) 0x7fede80cff70 con 0x7fede815ce40 2024-01-27T00:47:16.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.277+0000 7fede5d9b700 1 -- 172.21.15.46:0/917049243 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedd0003570 con 0x7fede815ce40 2024-01-27T00:47:16.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.277+0000 7fede5d9b700 1 -- 172.21.15.46:0/917049243 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 99033736 0 0) 0x7fedd0003570 con 0x7fede815ce40 2024-01-27T00:47:16.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.277+0000 7fede759e700 1 -- 172.21.15.46:0/917049243 >> v2:172.21.15.106:3300/0 conn(0x7fede815d210 msgr2=0x7fede80cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:16.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.277+0000 7fede759e700 1 --2- 172.21.15.46:0/917049243 >> v2:172.21.15.106:3300/0 conn(0x7fede815d210 0x7fede80cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.277+0000 7fede759e700 1 -- 172.21.15.46:0/917049243 >> v1:172.21.15.46:6789/0 conn(0x7fede815ce40 legacy=0x7fede80d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:16.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.277+0000 7fede759e700 1 -- 172.21.15.46:0/917049243 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede80cf660 con 0x7fede80cbff0 2024-01-27T00:47:16.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.277+0000 7fede659c700 1 --2- 172.21.15.46:0/917049243 >> v2:172.21.15.106:3300/0 conn(0x7fede815d210 0x7fede80cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:16.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.277+0000 7fede759e700 1 --2- 172.21.15.46:0/917049243 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fede80cbff0 0x7fede80ade30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fede0009e60 tx=0x7fede0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c827d7a5bee16423 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:16.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.278+0000 7fede5d9b700 1 -- 172.21.15.46:0/917049243 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede000e070 con 0x7fede80cbff0 2024-01-27T00:47:16.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.278+0000 7fede5d9b700 1 -- 172.21.15.46:0/917049243 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede000b7a0 con 0x7fede80cbff0 2024-01-27T00:47:16.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.278+0000 7fede5d9b700 1 -- 172.21.15.46:0/917049243 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd80015f0 con 0x7fede80cbff0 2024-01-27T00:47:16.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.278+0000 7feded098700 1 -- 172.21.15.46:0/917049243 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fede80cbff0 msgr2=0x7fede80ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:16.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.278+0000 7feded098700 1 --2- 172.21.15.46:0/917049243 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fede80cbff0 0x7fede80ade30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fede0009e60 tx=0x7fede0009780 comp rx=0 tx=0).stop 2024-01-27T00:47:16.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.278+0000 7feded098700 1 -- 172.21.15.46:0/917049243 shutdown_connections 2024-01-27T00:47:16.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.278+0000 7feded098700 1 --2- 172.21.15.46:0/917049243 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fede80cbff0 0x7fede80ade30 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.278+0000 7feded098700 1 --2- 172.21.15.46:0/917049243 >> v2:172.21.15.106:3300/0 conn(0x7fede815d210 0x7fede80cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.278+0000 7feded098700 1 -- 172.21.15.46:0/917049243 >> 172.21.15.46:0/917049243 conn(0x7fede806f840 msgr2=0x7fede80d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:16.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.278+0000 7feded098700 1 -- 172.21.15.46:0/917049243 shutdown_connections 2024-01-27T00:47:16.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.278+0000 7feded098700 1 -- 172.21.15.46:0/917049243 wait complete. 2024-01-27T00:47:16.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.280+0000 7feded098700 1 Processor -- start 2024-01-27T00:47:16.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.280+0000 7feded098700 1 -- start start 2024-01-27T00:47:16.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.281+0000 7feded098700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fede80cbff0 0x7fede80c5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:16.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.281+0000 7feded098700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede815ce40 0x7fede809c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.281+0000 7feded098700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fede815d210 0x7fede8169bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.281+0000 7feded098700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede80c3e80 con 0x7fede815ce40 2024-01-27T00:47:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.281+0000 7feded098700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede80c4160 con 0x7fede815d210 2024-01-27T00:47:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.281+0000 7feded098700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede80c4440 con 0x7fede80cbff0 2024-01-27T00:47:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.281+0000 7fede659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede815ce40 0x7fede809c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.281+0000 7fede659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede815ce40 0x7fede809c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38924/0 (socket says 172.21.15.46:38924) 2024-01-27T00:47:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.281+0000 7fede659c700 1 -- 172.21.15.46:0/316220860 learned_addr learned my addr 172.21.15.46:0/316220860 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.281+0000 7fede6d9d700 1 --2- 172.21.15.46:0/316220860 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fede80cbff0 0x7fede80c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.282+0000 7fede759e700 1 --2- 172.21.15.46:0/316220860 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fede815d210 0x7fede8169bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.282+0000 7fede659c700 1 -- 172.21.15.46:0/316220860 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fede80cbff0 msgr2=0x7fede80c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.282+0000 7fede659c700 1 --2- 172.21.15.46:0/316220860 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fede80cbff0 0x7fede80c5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.282+0000 7fede659c700 1 -- 172.21.15.46:0/316220860 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fede815d210 msgr2=0x7fede8169bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.282+0000 7fede659c700 1 --2- 172.21.15.46:0/316220860 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fede815d210 0x7fede8169bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.282+0000 7fede659c700 1 -- 172.21.15.46:0/316220860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc009420 con 0x7fede815ce40 2024-01-27T00:47:16.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.282+0000 7fede659c700 1 --2- 172.21.15.46:0/316220860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede815ce40 0x7fede809c450 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7feddc00ec50 tx=0x7feddc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:16.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.282+0000 7fedceffd700 1 -- 172.21.15.46:0/316220860 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc00c850 con 0x7fede815ce40 2024-01-27T00:47:16.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.282+0000 7fedceffd700 1 -- 172.21.15.46:0/316220860 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feddc00c9b0 con 0x7fede815ce40 2024-01-27T00:47:16.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.282+0000 7feded098700 1 -- 172.21.15.46:0/316220860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede0009a20 con 0x7fede815ce40 2024-01-27T00:47:16.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.282+0000 7fedceffd700 1 -- 172.21.15.46:0/316220860 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc00cd50 con 0x7fede815ce40 2024-01-27T00:47:16.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.282+0000 7feded098700 1 -- 172.21.15.46:0/316220860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede80c4a00 con 0x7fede815ce40 2024-01-27T00:47:16.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.283+0000 7feded098700 1 -- 172.21.15.46:0/316220860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede80b66b0 con 0x7fede815ce40 2024-01-27T00:47:16.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.284+0000 7fedceffd700 1 -- 172.21.15.46:0/316220860 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7feddc013680 con 0x7fede815ce40 2024-01-27T00:47:16.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.284+0000 7fedceffd700 1 --2- 172.21.15.46:0/316220860 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fedd4066f30 0x7fedd40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:16.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.284+0000 7fedceffd700 1 -- 172.21.15.46:0/316220860 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5609+0+0 (secure 0 0 0) 0x7feddc084810 con 0x7fede815ce40 2024-01-27T00:47:16.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.285+0000 7fede6d9d700 1 --2- 172.21.15.46:0/316220860 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fedd4066f30 0x7fedd40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:16.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.285+0000 7fede6d9d700 1 --2- 172.21.15.46:0/316220860 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fedd4066f30 0x7fedd40693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fedd8006d80 tx=0x7fedd80012c0 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:16.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.289+0000 7fedceffd700 1 -- 172.21.15.46:0/316220860 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feddc051ae0 con 0x7fede815ce40 2024-01-27T00:47:16.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.495+0000 7feded098700 1 -- 172.21.15.46:0/316220860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fede804c710 con 0x7fede815ce40 2024-01-27T00:47:16.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.496+0000 7fedceffd700 1 -- 172.21.15.46:0/316220860 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+13014 (secure 0 0 0) 0x7feddc017090 con 0x7fede815ce40 2024-01-27T00:47:16.496 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:16.496 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":184,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T00:47:15.677660+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T00:47:16.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.498+0000 7feded098700 1 -- 172.21.15.46:0/316220860 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fedd4066f30 msgr2=0x7fedd40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:16.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.498+0000 7feded098700 1 --2- 172.21.15.46:0/316220860 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fedd4066f30 0x7fedd40693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fedd8006d80 tx=0x7fedd80012c0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.498+0000 7feded098700 1 -- 172.21.15.46:0/316220860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede815ce40 msgr2=0x7fede809c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:16.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.498+0000 7feded098700 1 --2- 172.21.15.46:0/316220860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede815ce40 0x7fede809c450 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7feddc00ec50 tx=0x7feddc0091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.498+0000 7feded098700 1 -- 172.21.15.46:0/316220860 shutdown_connections 2024-01-27T00:47:16.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.498+0000 7feded098700 1 --2- 172.21.15.46:0/316220860 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fedd4066f30 0x7fedd40693e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.498+0000 7feded098700 1 --2- 172.21.15.46:0/316220860 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fede815d210 0x7fede8169bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.498+0000 7feded098700 1 --2- 172.21.15.46:0/316220860 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fede80cbff0 0x7fede80c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.498+0000 7feded098700 1 --2- 172.21.15.46:0/316220860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede815ce40 0x7fede809c450 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.498+0000 7feded098700 1 -- 172.21.15.46:0/316220860 >> 172.21.15.46:0/316220860 conn(0x7fede806f840 msgr2=0x7fede815b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:16.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.498+0000 7feded098700 1 -- 172.21.15.46:0/316220860 shutdown_connections 2024-01-27T00:47:16.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.498+0000 7feded098700 1 -- 172.21.15.46:0/316220860 wait complete. 2024-01-27T00:47:16.512 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-27T00:47:16.512 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-27T00:47:16.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.648+0000 7f4fa7bfd700 1 Processor -- start 2024-01-27T00:47:16.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.649+0000 7f4fa7bfd700 1 -- start start 2024-01-27T00:47:16.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.649+0000 7f4fa7bfd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4fa00c8da0 0x7f4fa00c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:16.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.649+0000 7f4fa7bfd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fa0140af0 0x7f4fa0140f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:16.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.650+0000 7f4fa7bfd700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa00ccc00 con 0x7f4fa00caf30 2024-01-27T00:47:16.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.650+0000 7f4fa7bfd700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fa00cbbf0 con 0x7f4fa00c8da0 2024-01-27T00:47:16.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.650+0000 7f4fa7bfd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa00cc310 con 0x7f4fa0140af0 2024-01-27T00:47:16.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.650+0000 7f4fa5999700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4fa00c8da0 0x7f4fa00c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:16.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.650+0000 7f4fa5999700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4fa00c8da0 0x7f4fa00c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38926/0 (socket says 172.21.15.46:38926) 2024-01-27T00:47:16.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.650+0000 7f4fa5999700 1 -- 172.21.15.46:0/876811637 learned_addr learned my addr 172.21.15.46:0/876811637 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:16.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.650+0000 7f4fa5198700 1 --2- 172.21.15.46:0/876811637 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fa0140af0 0x7f4fa0140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:16.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.650+0000 7f4fa5999700 1 -- 172.21.15.46:0/876811637 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fa0140af0 msgr2=0x7f4fa0140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:16.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.650+0000 7f4fa5999700 1 --2- 172.21.15.46:0/876811637 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fa0140af0 0x7f4fa0140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.650+0000 7f4fa5999700 1 -- 172.21.15.46:0/876811637 >> v1:172.21.15.106:6789/0 conn(0x7f4fa00caf30 legacy=0x7f4fa00cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:16.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.650+0000 7f4fa5999700 1 -- 172.21.15.46:0/876811637 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa00cc5b0 con 0x7f4fa00c8da0 2024-01-27T00:47:16.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.650+0000 7f4fa5999700 1 --2- 172.21.15.46:0/876811637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fa00c8da0 0x7f4fa00c9170 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0099b0 tx=0x7f4f9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98b7a6641f744c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:16.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.650+0000 7f4fa4997700 1 -- 172.21.15.46:0/876811637 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c009ea0 con 0x7f4fa00c8da0 2024-01-27T00:47:16.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.651+0000 7f4fa4997700 1 -- 172.21.15.46:0/876811637 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9c00b860 con 0x7f4fa00c8da0 2024-01-27T00:47:16.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.651+0000 7f4fa4997700 1 -- 172.21.15.46:0/876811637 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c00bc00 con 0x7f4fa00c8da0 2024-01-27T00:47:16.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.651+0000 7f4fa7bfd700 1 -- 172.21.15.46:0/876811637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fa00c8da0 msgr2=0x7f4fa00c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:16.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.651+0000 7f4fa7bfd700 1 --2- 172.21.15.46:0/876811637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fa00c8da0 0x7f4fa00c9170 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0099b0 tx=0x7f4f9c009250 comp rx=0 tx=0).stop 2024-01-27T00:47:16.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.651+0000 7f4fa7bfd700 1 -- 172.21.15.46:0/876811637 shutdown_connections 2024-01-27T00:47:16.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.651+0000 7f4fa7bfd700 1 --2- 172.21.15.46:0/876811637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fa00c8da0 0x7f4fa00c9170 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.651+0000 7f4fa7bfd700 1 --2- 172.21.15.46:0/876811637 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fa0140af0 0x7f4fa0140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.651+0000 7f4fa7bfd700 1 -- 172.21.15.46:0/876811637 >> 172.21.15.46:0/876811637 conn(0x7f4fa006bd90 msgr2=0x7f4fa006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:16.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.651+0000 7f4fa7bfd700 1 -- 172.21.15.46:0/876811637 shutdown_connections 2024-01-27T00:47:16.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.651+0000 7f4fa7bfd700 1 -- 172.21.15.46:0/876811637 wait complete. 2024-01-27T00:47:16.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.652+0000 7f4fa7bfd700 1 Processor -- start 2024-01-27T00:47:16.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.652+0000 7f4fa7bfd700 1 -- start start 2024-01-27T00:47:16.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.653+0000 7f4fa7bfd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fa00c8da0 0x7f4fa013cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:16.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.653+0000 7f4fa7bfd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fa00caf30 0x7f4fa0157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:16.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.653+0000 7f4fa7bfd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fa0140af0 0x7f4fa0156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:16.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.653+0000 7f4fa7bfd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa00c60f0 con 0x7f4fa0140af0 2024-01-27T00:47:16.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.653+0000 7f4fa7bfd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa00b3330 con 0x7f4fa00c8da0 2024-01-27T00:47:16.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.653+0000 7f4fa7bfd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa013d2d0 con 0x7f4fa00caf30 2024-01-27T00:47:16.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.653+0000 7f4fa619a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fa0140af0 0x7f4fa0156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:16.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.653+0000 7f4fa5999700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fa00c8da0 0x7f4fa013cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:16.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.653+0000 7f4fa619a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fa0140af0 0x7f4fa0156910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38932/0 (socket says 172.21.15.46:38932) 2024-01-27T00:47:16.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.653+0000 7f4fa619a700 1 -- 172.21.15.46:0/1506916996 learned_addr learned my addr 172.21.15.46:0/1506916996 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:16.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.653+0000 7f4fa5198700 1 --2- 172.21.15.46:0/1506916996 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fa00caf30 0x7f4fa0157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:16.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.654+0000 7f4fa5999700 1 -- 172.21.15.46:0/1506916996 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fa00caf30 msgr2=0x7f4fa0157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:16.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.654+0000 7f4fa5999700 1 --2- 172.21.15.46:0/1506916996 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fa00caf30 0x7f4fa0157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.654+0000 7f4fa5999700 1 -- 172.21.15.46:0/1506916996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fa0140af0 msgr2=0x7f4fa0156910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:16.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.654+0000 7f4fa5999700 1 --2- 172.21.15.46:0/1506916996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fa0140af0 0x7f4fa0156910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.654+0000 7f4fa5999700 1 -- 172.21.15.46:0/1506916996 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f9c0094f0 con 0x7f4fa00c8da0 2024-01-27T00:47:16.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.654+0000 7f4fa619a700 1 --2- 172.21.15.46:0/1506916996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fa0140af0 0x7f4fa0156910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:16.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.654+0000 7f4fa5999700 1 --2- 172.21.15.46:0/1506916996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fa00c8da0 0x7f4fa013cd90 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f4fa00ce550 tx=0x7f4f9c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:16.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.654+0000 7f4f95ffb700 1 -- 172.21.15.46:0/1506916996 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c000d10 con 0x7f4fa00c8da0 2024-01-27T00:47:16.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.654+0000 7f4f95ffb700 1 -- 172.21.15.46:0/1506916996 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9c00bbe0 con 0x7f4fa00c8da0 2024-01-27T00:47:16.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.654+0000 7f4f95ffb700 1 -- 172.21.15.46:0/1506916996 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c011670 con 0x7f4fa00c8da0 2024-01-27T00:47:16.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.654+0000 7f4fa7bfd700 1 -- 172.21.15.46:0/1506916996 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa00c5e10 con 0x7f4fa00c8da0 2024-01-27T00:47:16.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.655+0000 7f4fa7bfd700 1 -- 172.21.15.46:0/1506916996 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa00b2fc0 con 0x7f4fa00c8da0 2024-01-27T00:47:16.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.655+0000 7f4fa7bfd700 1 -- 172.21.15.46:0/1506916996 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa00b2b40 con 0x7f4fa00c8da0 2024-01-27T00:47:16.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.656+0000 7f4f95ffb700 1 -- 172.21.15.46:0/1506916996 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4f9c011aa0 con 0x7f4fa00c8da0 2024-01-27T00:47:16.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.657+0000 7f4f95ffb700 1 --2- 172.21.15.46:0/1506916996 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f84066f30 0x7f4f840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:16.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.657+0000 7f4f95ffb700 1 -- 172.21.15.46:0/1506916996 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f4f9c086080 con 0x7f4fa00c8da0 2024-01-27T00:47:16.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.657+0000 7f4fa5198700 1 --2- 172.21.15.46:0/1506916996 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f84066f30 0x7f4f840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:16.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.657+0000 7f4fa5198700 1 --2- 172.21.15.46:0/1506916996 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f84066f30 0x7f4f840693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4fa015aff0 tx=0x7f4f900091a0 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:16.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.661+0000 7f4f95ffb700 1 -- 172.21.15.46:0/1506916996 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4f9c053350 con 0x7f4fa00c8da0 2024-01-27T00:47:16.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.863+0000 7f4fa7bfd700 1 -- 172.21.15.46:0/1506916996 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4fa00b2b40 con 0x7f4fa00c8da0 2024-01-27T00:47:16.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.864+0000 7f4f95ffb700 1 -- 172.21.15.46:0/1506916996 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v184) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4fa00b2b40 con 0x7f4fa00c8da0 2024-01-27T00:47:16.865 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-27T00:47:16.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.866+0000 7f4fa7bfd700 1 -- 172.21.15.46:0/1506916996 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f84066f30 msgr2=0x7f4f840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:16.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.866+0000 7f4fa7bfd700 1 --2- 172.21.15.46:0/1506916996 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f84066f30 0x7f4f840693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4fa015aff0 tx=0x7f4f900091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.866+0000 7f4fa7bfd700 1 -- 172.21.15.46:0/1506916996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fa00c8da0 msgr2=0x7f4fa013cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:16.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.866+0000 7f4fa7bfd700 1 --2- 172.21.15.46:0/1506916996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fa00c8da0 0x7f4fa013cd90 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f4fa00ce550 tx=0x7f4f9c017000 comp rx=0 tx=0).stop 2024-01-27T00:47:16.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.866+0000 7f4fa7bfd700 1 -- 172.21.15.46:0/1506916996 shutdown_connections 2024-01-27T00:47:16.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.866+0000 7f4fa7bfd700 1 --2- 172.21.15.46:0/1506916996 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f4f84066f30 0x7f4f840693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.866+0000 7f4fa7bfd700 1 --2- 172.21.15.46:0/1506916996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fa0140af0 0x7f4fa0156910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.866+0000 7f4fa7bfd700 1 --2- 172.21.15.46:0/1506916996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fa00c8da0 0x7f4fa013cd90 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.866+0000 7f4fa7bfd700 1 --2- 172.21.15.46:0/1506916996 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fa00caf30 0x7f4fa0157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:16.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.866+0000 7f4fa7bfd700 1 -- 172.21.15.46:0/1506916996 >> 172.21.15.46:0/1506916996 conn(0x7f4fa006bd90 msgr2=0x7f4fa013fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:16.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.866+0000 7f4fa7bfd700 1 -- 172.21.15.46:0/1506916996 shutdown_connections 2024-01-27T00:47:16.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:16.867+0000 7f4fa7bfd700 1 -- 172.21.15.46:0/1506916996 wait complete. 2024-01-27T00:47:16.880 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-27T00:47:17.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.012+0000 7fee59adc700 1 Processor -- start 2024-01-27T00:47:17.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.012+0000 7fee59adc700 1 -- start start 2024-01-27T00:47:17.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.012+0000 7fee59adc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fee54159590 0x7fee5415d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:17.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.012+0000 7fee59adc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee5415d8f0 0x7fee5406bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:17.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.012+0000 7fee59adc700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5415dd90 con 0x7fee54159960 2024-01-27T00:47:17.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.012+0000 7fee59adc700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee5415e350 con 0x7fee54159590 2024-01-27T00:47:17.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.012+0000 7fee59adc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee540c97c0 con 0x7fee5415d8f0 2024-01-27T00:47:17.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.013+0000 7fee537fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fee54159590 0x7fee5415d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:17.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.013+0000 7fee52ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee5415d8f0 0x7fee5406bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:17.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.013+0000 7fee53fff700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fee54159960 0x7fee5406d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:45512/0 (socket says 172.21.15.46:45512) 2024-01-27T00:47:17.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.013+0000 7fee537fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fee54159590 0x7fee5415d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:37134/0 (socket says 172.21.15.46:37134) 2024-01-27T00:47:17.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.013+0000 7fee53fff700 1 -- 172.21.15.46:0/1199600829 learned_addr learned my addr 172.21.15.46:0/1199600829 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:17.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.013+0000 7fee527fc700 1 -- 172.21.15.46:0/1199600829 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861317434 0 0) 0x7fee5415dd90 con 0x7fee54159960 2024-01-27T00:47:17.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.013+0000 7fee527fc700 1 -- 172.21.15.46:0/1199600829 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee3c003570 con 0x7fee54159960 2024-01-27T00:47:17.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.013+0000 7fee537fe700 1 -- 172.21.15.46:0/1199600829 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee5415d8f0 msgr2=0x7fee5406bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:17.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.013+0000 7fee537fe700 1 --2- 172.21.15.46:0/1199600829 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee5415d8f0 0x7fee5406bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:17.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.013+0000 7fee537fe700 1 -- 172.21.15.46:0/1199600829 >> v1:172.21.15.106:6789/0 conn(0x7fee54159960 legacy=0x7fee5406d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:17.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.013+0000 7fee537fe700 1 -- 172.21.15.46:0/1199600829 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5415e070 con 0x7fee54159590 2024-01-27T00:47:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.013+0000 7fee52ffd700 1 --2- 172.21.15.46:0/1199600829 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee5415d8f0 0x7fee5406bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.014+0000 7fee537fe700 1 --2- 172.21.15.46:0/1199600829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee54159590 0x7fee5415d3b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fee44009910 tx=0x7fee440091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9da2ff75fa15677 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.014+0000 7fee527fc700 1 -- 172.21.15.46:0/1199600829 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee44011630 con 0x7fee54159590 2024-01-27T00:47:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.014+0000 7fee527fc700 1 -- 172.21.15.46:0/1199600829 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee44011790 con 0x7fee54159590 2024-01-27T00:47:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.014+0000 7fee527fc700 1 -- 172.21.15.46:0/1199600829 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee44011b30 con 0x7fee54159590 2024-01-27T00:47:17.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.014+0000 7fee59adc700 1 -- 172.21.15.46:0/1199600829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee54159590 msgr2=0x7fee5415d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.014+0000 7fee59adc700 1 --2- 172.21.15.46:0/1199600829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee54159590 0x7fee5415d3b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fee44009910 tx=0x7fee440091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.014+0000 7fee59adc700 1 -- 172.21.15.46:0/1199600829 shutdown_connections 2024-01-27T00:47:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.014+0000 7fee59adc700 1 --2- 172.21.15.46:0/1199600829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee54159590 0x7fee5415d3b0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.014+0000 7fee59adc700 1 --2- 172.21.15.46:0/1199600829 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee5415d8f0 0x7fee5406bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.014+0000 7fee59adc700 1 -- 172.21.15.46:0/1199600829 >> 172.21.15.46:0/1199600829 conn(0x7fee540ce530 msgr2=0x7fee540ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.014+0000 7fee59adc700 1 -- 172.21.15.46:0/1199600829 shutdown_connections 2024-01-27T00:47:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.014+0000 7fee59adc700 1 -- 172.21.15.46:0/1199600829 wait complete. 2024-01-27T00:47:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.016+0000 7fee59adc700 1 Processor -- start 2024-01-27T00:47:17.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.016+0000 7fee59adc700 1 -- start start 2024-01-27T00:47:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.017+0000 7fee59adc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee54159590 0x7fee54149dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.017+0000 7fee59adc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee54159960 0x7fee540c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.017+0000 7fee59adc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee5415d8f0 0x7fee540cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.017+0000 7fee59adc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee54147450 con 0x7fee54159960 2024-01-27T00:47:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.017+0000 7fee59adc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5415a3a0 con 0x7fee54159590 2024-01-27T00:47:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.017+0000 7fee59adc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee54159d90 con 0x7fee5415d8f0 2024-01-27T00:47:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.017+0000 7fee52ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee54159960 0x7fee540c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.017+0000 7fee52ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee54159960 0x7fee540c76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38936/0 (socket says 172.21.15.46:38936) 2024-01-27T00:47:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.017+0000 7fee52ffd700 1 -- 172.21.15.46:0/1724941332 learned_addr learned my addr 172.21.15.46:0/1724941332 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.018+0000 7fee53fff700 1 --2- 172.21.15.46:0/1724941332 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee5415d8f0 0x7fee540cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.018+0000 7fee537fe700 1 --2- 172.21.15.46:0/1724941332 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee54159590 0x7fee54149dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.018+0000 7fee52ffd700 1 -- 172.21.15.46:0/1724941332 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee5415d8f0 msgr2=0x7fee540cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:17.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.018+0000 7fee52ffd700 1 --2- 172.21.15.46:0/1724941332 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee5415d8f0 0x7fee540cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.018+0000 7fee52ffd700 1 -- 172.21.15.46:0/1724941332 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee54159590 msgr2=0x7fee54149dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.018+0000 7fee52ffd700 1 --2- 172.21.15.46:0/1724941332 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee54159590 0x7fee54149dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.018+0000 7fee52ffd700 1 -- 172.21.15.46:0/1724941332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee48009420 con 0x7fee54159960 2024-01-27T00:47:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.018+0000 7fee52ffd700 1 --2- 172.21.15.46:0/1724941332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee54159960 0x7fee540c76c0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fee4800ec80 tx=0x7fee480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.018+0000 7fee3b7fe700 1 -- 172.21.15.46:0/1724941332 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4800c850 con 0x7fee54159960 2024-01-27T00:47:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.018+0000 7fee3b7fe700 1 -- 172.21.15.46:0/1724941332 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4800c9b0 con 0x7fee54159960 2024-01-27T00:47:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.018+0000 7fee59adc700 1 -- 172.21.15.46:0/1724941332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee44009420 con 0x7fee54159960 2024-01-27T00:47:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.018+0000 7fee3b7fe700 1 -- 172.21.15.46:0/1724941332 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4800cd50 con 0x7fee54159960 2024-01-27T00:47:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.018+0000 7fee59adc700 1 -- 172.21.15.46:0/1724941332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee540c82d0 con 0x7fee54159960 2024-01-27T00:47:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.019+0000 7fee59adc700 1 -- 172.21.15.46:0/1724941332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5404c710 con 0x7fee54159960 2024-01-27T00:47:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.020+0000 7fee3b7fe700 1 -- 172.21.15.46:0/1724941332 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fee48013680 con 0x7fee54159960 2024-01-27T00:47:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.021+0000 7fee3b7fe700 1 --2- 172.21.15.46:0/1724941332 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fee40066f60 0x7fee40069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.021+0000 7fee3b7fe700 1 -- 172.21.15.46:0/1724941332 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fee480843e0 con 0x7fee54159960 2024-01-27T00:47:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.021+0000 7fee537fe700 1 --2- 172.21.15.46:0/1724941332 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fee40066f60 0x7fee40069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:17.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.022+0000 7fee537fe700 1 --2- 172.21.15.46:0/1724941332 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fee40066f60 0x7fee40069410 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fee44009860 tx=0x7fee44017000 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:17.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.025+0000 7fee3b7fe700 1 -- 172.21.15.46:0/1724941332 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fee480516b0 con 0x7fee54159960 2024-01-27T00:47:17.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.233+0000 7fee59adc700 1 -- 172.21.15.46:0/1724941332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fee54068b70 con 0x7fee54159960 2024-01-27T00:47:17.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.234+0000 7fee3b7fe700 1 -- 172.21.15.46:0/1724941332 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v184) v1 ==== 84+0+65 (secure 0 0 0) 0x7fee48054cd0 con 0x7fee54159960 2024-01-27T00:47:17.234 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:17.234 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-27T00:47:17.234 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:17.234 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-01-27T00:47:17.235 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-01-27T00:47:17.235 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:17.235 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-27T00:47:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.236+0000 7fee59adc700 1 -- 172.21.15.46:0/1724941332 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fee40066f60 msgr2=0x7fee40069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.236+0000 7fee59adc700 1 --2- 172.21.15.46:0/1724941332 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fee40066f60 0x7fee40069410 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fee44009860 tx=0x7fee44017000 comp rx=0 tx=0).stop 2024-01-27T00:47:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.236+0000 7fee59adc700 1 -- 172.21.15.46:0/1724941332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee54159960 msgr2=0x7fee540c76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.236+0000 7fee59adc700 1 --2- 172.21.15.46:0/1724941332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee54159960 0x7fee540c76c0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fee4800ec80 tx=0x7fee480091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.236+0000 7fee59adc700 1 -- 172.21.15.46:0/1724941332 shutdown_connections 2024-01-27T00:47:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.236+0000 7fee59adc700 1 --2- 172.21.15.46:0/1724941332 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fee40066f60 0x7fee40069410 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.236+0000 7fee59adc700 1 --2- 172.21.15.46:0/1724941332 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee5415d8f0 0x7fee540cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.236+0000 7fee59adc700 1 --2- 172.21.15.46:0/1724941332 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee54159590 0x7fee54149dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.236+0000 7fee59adc700 1 --2- 172.21.15.46:0/1724941332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee54159960 0x7fee540c76c0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.236+0000 7fee59adc700 1 -- 172.21.15.46:0/1724941332 >> 172.21.15.46:0/1724941332 conn(0x7fee540ce530 msgr2=0x7fee540cd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:17.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.236+0000 7fee59adc700 1 -- 172.21.15.46:0/1724941332 shutdown_connections 2024-01-27T00:47:17.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.236+0000 7fee59adc700 1 -- 172.21.15.46:0/1724941332 wait complete. 2024-01-27T00:47:17.250 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-27T00:47:17.250 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-27T00:47:17.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.385+0000 7fcccea37700 1 Processor -- start 2024-01-27T00:47:17.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.385+0000 7fcccea37700 1 -- start start 2024-01-27T00:47:17.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.386+0000 7fcccea37700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fccc80af570 0x7fccc80b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:17.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.386+0000 7fcccea37700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fccc8067930 0x7fccc8067d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:17.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.386+0000 7fcccea37700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc815ab40 con 0x7fccc8159630 2024-01-27T00:47:17.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.386+0000 7fcccea37700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc8068270 con 0x7fccc80af570 2024-01-27T00:47:17.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.386+0000 7fcccea37700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc815a210 con 0x7fccc8067930 2024-01-27T00:47:17.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.386+0000 7fccc77fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fccc80af570 0x7fccc80b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:17.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.386+0000 7fccc7fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fccc8159630 0x7fccc8159a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40990/0 (socket says 172.21.15.46:40990) 2024-01-27T00:47:17.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.386+0000 7fccc7fff700 1 -- 172.21.15.46:0/1517992060 learned_addr learned my addr 172.21.15.46:0/1517992060 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:17.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.387+0000 7fccccfd4700 1 --2- 172.21.15.46:0/1517992060 >> v2:172.21.15.106:3300/0 conn(0x7fccc8067930 0x7fccc8067d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:17.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.387+0000 7fccc77fe700 1 -- 172.21.15.46:0/1517992060 >> v2:172.21.15.106:3300/0 conn(0x7fccc8067930 msgr2=0x7fccc8067d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:17.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.387+0000 7fccc77fe700 1 --2- 172.21.15.46:0/1517992060 >> v2:172.21.15.106:3300/0 conn(0x7fccc8067930 0x7fccc8067d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:17.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.387+0000 7fccc77fe700 1 -- 172.21.15.46:0/1517992060 >> v1:172.21.15.46:6789/0 conn(0x7fccc8159630 legacy=0x7fccc8159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:17.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.387+0000 7fccc77fe700 1 -- 172.21.15.46:0/1517992060 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc815a4f0 con 0x7fccc80af570 2024-01-27T00:47:17.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.387+0000 7fccc77fe700 1 --2- 172.21.15.46:0/1517992060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80af570 0x7fccc80b2b90 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fccc806a890 tx=0x7fccbc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3fca8fc5185dae5 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:17.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.387+0000 7fccc6ffd700 1 -- 172.21.15.46:0/1517992060 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc00b6b0 con 0x7fccc80af570 2024-01-27T00:47:17.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.387+0000 7fccc6ffd700 1 -- 172.21.15.46:0/1517992060 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccbc00b810 con 0x7fccc80af570 2024-01-27T00:47:17.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.387+0000 7fccc6ffd700 1 -- 172.21.15.46:0/1517992060 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc00bbb0 con 0x7fccc80af570 2024-01-27T00:47:17.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.388+0000 7fcccea37700 1 -- 172.21.15.46:0/1517992060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80af570 msgr2=0x7fccc80b2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:17.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.388+0000 7fcccea37700 1 --2- 172.21.15.46:0/1517992060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80af570 0x7fccc80b2b90 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fccc806a890 tx=0x7fccbc0091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:17.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.388+0000 7fcccea37700 1 -- 172.21.15.46:0/1517992060 shutdown_connections 2024-01-27T00:47:17.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.388+0000 7fcccea37700 1 --2- 172.21.15.46:0/1517992060 >> v2:172.21.15.106:3300/0 conn(0x7fccc8067930 0x7fccc8067d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:17.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.388+0000 7fcccea37700 1 --2- 172.21.15.46:0/1517992060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80af570 0x7fccc80b2b90 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:17.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.388+0000 7fcccea37700 1 -- 172.21.15.46:0/1517992060 >> 172.21.15.46:0/1517992060 conn(0x7fccc8069510 msgr2=0x7fccc80aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:17.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.388+0000 7fcccea37700 1 -- 172.21.15.46:0/1517992060 shutdown_connections 2024-01-27T00:47:17.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.388+0000 7fcccea37700 1 -- 172.21.15.46:0/1517992060 wait complete. 2024-01-27T00:47:17.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.389+0000 7fcccea37700 1 Processor -- start 2024-01-27T00:47:17.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.389+0000 7fcccea37700 1 -- start start 2024-01-27T00:47:17.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.390+0000 7fcccea37700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc8067930 0x7fccc814bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:17.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.390+0000 7fcccea37700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80af570 0x7fccc80abed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:17.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.390+0000 7fcccea37700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccc8159630 0x7fccc80ac410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:17.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.390+0000 7fcccea37700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc81515c0 con 0x7fccc80af570 2024-01-27T00:47:17.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.390+0000 7fcccea37700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc814c530 con 0x7fccc8159630 2024-01-27T00:47:17.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.390+0000 7fcccea37700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc81509b0 con 0x7fccc8067930 2024-01-27T00:47:17.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.390+0000 7fccc77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80af570 0x7fccc80abed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:17.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.390+0000 7fccc77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80af570 0x7fccc80abed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38956/0 (socket says 172.21.15.46:38956) 2024-01-27T00:47:17.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.390+0000 7fccc77fe700 1 -- 172.21.15.46:0/588719054 learned_addr learned my addr 172.21.15.46:0/588719054 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:17.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.390+0000 7fccc7fff700 1 --2- 172.21.15.46:0/588719054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc8067930 0x7fccc814bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:17.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.390+0000 7fccccfd4700 1 --2- 172.21.15.46:0/588719054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccc8159630 0x7fccc80ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:17.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.391+0000 7fccc77fe700 1 -- 172.21.15.46:0/588719054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc8067930 msgr2=0x7fccc814bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:17.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.391+0000 7fccc77fe700 1 --2- 172.21.15.46:0/588719054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc8067930 0x7fccc814bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:17.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.391+0000 7fccc77fe700 1 -- 172.21.15.46:0/588719054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccc8159630 msgr2=0x7fccc80ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:17.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.391+0000 7fccc77fe700 1 --2- 172.21.15.46:0/588719054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccc8159630 0x7fccc80ac410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:17.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.391+0000 7fccc77fe700 1 -- 172.21.15.46:0/588719054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccbc009420 con 0x7fccc80af570 2024-01-27T00:47:17.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.391+0000 7fccc77fe700 1 --2- 172.21.15.46:0/588719054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80af570 0x7fccc80abed0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00b0e0 tx=0x7fccbc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:17.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.391+0000 7fccaffff700 1 -- 172.21.15.46:0/588719054 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc00bb90 con 0x7fccc80af570 2024-01-27T00:47:17.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.391+0000 7fccaffff700 1 -- 172.21.15.46:0/588719054 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccbc011420 con 0x7fccc80af570 2024-01-27T00:47:17.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.391+0000 7fcccea37700 1 -- 172.21.15.46:0/588719054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc8150c90 con 0x7fccc80af570 2024-01-27T00:47:17.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.391+0000 7fccaffff700 1 -- 172.21.15.46:0/588719054 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc0117c0 con 0x7fccc80af570 2024-01-27T00:47:17.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.391+0000 7fcccea37700 1 -- 172.21.15.46:0/588719054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc8151250 con 0x7fccc80af570 2024-01-27T00:47:17.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.392+0000 7fcccea37700 1 -- 172.21.15.46:0/588719054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc804c710 con 0x7fccc80af570 2024-01-27T00:47:17.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.392+0000 7fccaffff700 1 -- 172.21.15.46:0/588719054 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fccbc011bb0 con 0x7fccc80af570 2024-01-27T00:47:17.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.393+0000 7fccaffff700 1 --2- 172.21.15.46:0/588719054 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fccb4066f30 0x7fccb40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:17.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.393+0000 7fccaffff700 1 -- 172.21.15.46:0/588719054 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fccbc085810 con 0x7fccc80af570 2024-01-27T00:47:17.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.393+0000 7fccc7fff700 1 --2- 172.21.15.46:0/588719054 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fccb4066f30 0x7fccb40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:17.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.394+0000 7fccc7fff700 1 --2- 172.21.15.46:0/588719054 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fccb4066f30 0x7fccb40693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fccb8001950 tx=0x7fccb8009040 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:17.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.397+0000 7fccaffff700 1 -- 172.21.15.46:0/588719054 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fccbc052b60 con 0x7fccc80af570 2024-01-27T00:47:17.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:17.605+0000 7fcccea37700 1 -- 172.21.15.46:0/588719054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fccc806a310 con 0x7fccc80af570 2024-01-27T00:47:18.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.477+0000 7fccaffff700 1 -- 172.21.15.46:0/588719054 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v185) v1 ==== 138+0+0 (secure 0 0 0) 0x7fccbc011e60 con 0x7fccc80af570 2024-01-27T00:47:18.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.483+0000 7fcccea37700 1 -- 172.21.15.46:0/588719054 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fccb4066f30 msgr2=0x7fccb40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:18.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.483+0000 7fcccea37700 1 --2- 172.21.15.46:0/588719054 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fccb4066f30 0x7fccb40693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fccb8001950 tx=0x7fccb8009040 comp rx=0 tx=0).stop 2024-01-27T00:47:18.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.483+0000 7fcccea37700 1 -- 172.21.15.46:0/588719054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80af570 msgr2=0x7fccc80abed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:18.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.483+0000 7fcccea37700 1 --2- 172.21.15.46:0/588719054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80af570 0x7fccc80abed0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00b0e0 tx=0x7fccbc017000 comp rx=0 tx=0).stop 2024-01-27T00:47:18.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.484+0000 7fcccea37700 1 -- 172.21.15.46:0/588719054 shutdown_connections 2024-01-27T00:47:18.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.484+0000 7fcccea37700 1 --2- 172.21.15.46:0/588719054 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fccb4066f30 0x7fccb40693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:18.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.484+0000 7fcccea37700 1 --2- 172.21.15.46:0/588719054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccc8159630 0x7fccc80ac410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:18.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.484+0000 7fcccea37700 1 --2- 172.21.15.46:0/588719054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc8067930 0x7fccc814bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:18.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.484+0000 7fcccea37700 1 --2- 172.21.15.46:0/588719054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80af570 0x7fccc80abed0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:18.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.484+0000 7fcccea37700 1 -- 172.21.15.46:0/588719054 >> 172.21.15.46:0/588719054 conn(0x7fccc8069510 msgr2=0x7fccc80ae430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:18.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.484+0000 7fcccea37700 1 -- 172.21.15.46:0/588719054 shutdown_connections 2024-01-27T00:47:18.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.484+0000 7fcccea37700 1 -- 172.21.15.46:0/588719054 wait complete. 2024-01-27T00:47:18.485 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-01-27T00:47:18.500 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-27T00:47:18.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.643+0000 7f113a22a700 1 Processor -- start 2024-01-27T00:47:18.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.643+0000 7f113a22a700 1 -- start start 2024-01-27T00:47:18.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.644+0000 7f113a22a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11341597d0 0x7f113406ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:18.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.644+0000 7f113a22a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f113415ad40 0x7f113406f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:18.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.644+0000 7f113a22a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1134159ba0 0x7f11340cb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:18.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.644+0000 7f113a22a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11340cbfb0 con 0x7f1134159ba0 2024-01-27T00:47:18.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.644+0000 7f113a22a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f113415a8b0 con 0x7f113415ad40 2024-01-27T00:47:18.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.644+0000 7f113a22a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11340cbcd0 con 0x7f11341597d0 2024-01-27T00:47:18.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.644+0000 7f1133fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1134159ba0 0x7f11340cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:18.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.644+0000 7f1133fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1134159ba0 0x7f11340cb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38960/0 (socket says 172.21.15.46:38960) 2024-01-27T00:47:18.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.644+0000 7f1133fff700 1 -- 172.21.15.46:0/943533185 learned_addr learned my addr 172.21.15.46:0/943533185 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:18.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.644+0000 7f11337fe700 1 --2- 172.21.15.46:0/943533185 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11341597d0 0x7f113406ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:18.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.644+0000 7f1132ffd700 1 --2- 172.21.15.46:0/943533185 >> v2:172.21.15.106:3300/0 conn(0x7f113415ad40 0x7f113406f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:18.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.645+0000 7f1133fff700 1 -- 172.21.15.46:0/943533185 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11341597d0 msgr2=0x7f113406ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:18.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.645+0000 7f1133fff700 1 --2- 172.21.15.46:0/943533185 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11341597d0 0x7f113406ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:18.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.645+0000 7f1133fff700 1 -- 172.21.15.46:0/943533185 >> v2:172.21.15.106:3300/0 conn(0x7f113415ad40 msgr2=0x7f113406f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:18.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.645+0000 7f1133fff700 1 --2- 172.21.15.46:0/943533185 >> v2:172.21.15.106:3300/0 conn(0x7f113415ad40 0x7f113406f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:18.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.645+0000 7f1133fff700 1 -- 172.21.15.46:0/943533185 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113415a260 con 0x7f1134159ba0 2024-01-27T00:47:18.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.645+0000 7f1133fff700 1 --2- 172.21.15.46:0/943533185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134159ba0 0x7f11340cb030 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f113406a630 tx=0x7f112c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de0b637a033b47d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:18.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.645+0000 7f11327fc700 1 -- 172.21.15.46:0/943533185 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c00e070 con 0x7f1134159ba0 2024-01-27T00:47:18.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.645+0000 7f11327fc700 1 -- 172.21.15.46:0/943533185 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c00ba50 con 0x7f1134159ba0 2024-01-27T00:47:18.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.645+0000 7f11327fc700 1 -- 172.21.15.46:0/943533185 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c0092e0 con 0x7f1134159ba0 2024-01-27T00:47:18.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.645+0000 7f113a22a700 1 -- 172.21.15.46:0/943533185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134159ba0 msgr2=0x7f11340cb030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:18.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.645+0000 7f113a22a700 1 --2- 172.21.15.46:0/943533185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134159ba0 0x7f11340cb030 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f113406a630 tx=0x7f112c009840 comp rx=0 tx=0).stop 2024-01-27T00:47:18.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.646+0000 7f113a22a700 1 -- 172.21.15.46:0/943533185 shutdown_connections 2024-01-27T00:47:18.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.646+0000 7f113a22a700 1 --2- 172.21.15.46:0/943533185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134159ba0 0x7f11340cb030 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:18.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.646+0000 7f113a22a700 1 --2- 172.21.15.46:0/943533185 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11341597d0 0x7f113406ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:18.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.646+0000 7f113a22a700 1 --2- 172.21.15.46:0/943533185 >> v2:172.21.15.106:3300/0 conn(0x7f113415ad40 0x7f113406f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:18.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.646+0000 7f113a22a700 1 -- 172.21.15.46:0/943533185 >> 172.21.15.46:0/943533185 conn(0x7f11340ca9a0 msgr2=0x7f1134068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:18.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.646+0000 7f113a22a700 1 -- 172.21.15.46:0/943533185 shutdown_connections 2024-01-27T00:47:18.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.646+0000 7f113a22a700 1 -- 172.21.15.46:0/943533185 wait complete. 2024-01-27T00:47:18.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.647+0000 7f113a22a700 1 Processor -- start 2024-01-27T00:47:18.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.647+0000 7f113a22a700 1 -- start start 2024-01-27T00:47:18.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f113a22a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11341597d0 0x7f11340b0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:18.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f113a22a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134159ba0 0x7f113414c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:18.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f113a22a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f113415ad40 0x7f113415d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:18.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f113a22a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113414c6e0 con 0x7f1134159ba0 2024-01-27T00:47:18.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f113a22a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1134154b60 con 0x7f11341597d0 2024-01-27T00:47:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f113a22a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1134154e40 con 0x7f113415ad40 2024-01-27T00:47:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f1132ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134159ba0 0x7f113414c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f1132ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134159ba0 0x7f113414c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38976/0 (socket says 172.21.15.46:38976) 2024-01-27T00:47:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f1132ffd700 1 -- 172.21.15.46:0/925053716 learned_addr learned my addr 172.21.15.46:0/925053716 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f1133fff700 1 --2- 172.21.15.46:0/925053716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f113415ad40 0x7f113415d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f11337fe700 1 --2- 172.21.15.46:0/925053716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11341597d0 0x7f11340b0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f1132ffd700 1 -- 172.21.15.46:0/925053716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f113415ad40 msgr2=0x7f113415d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f1132ffd700 1 --2- 172.21.15.46:0/925053716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f113415ad40 0x7f113415d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f1132ffd700 1 -- 172.21.15.46:0/925053716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11341597d0 msgr2=0x7f11340b0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f1132ffd700 1 --2- 172.21.15.46:0/925053716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11341597d0 0x7f11340b0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f1132ffd700 1 -- 172.21.15.46:0/925053716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c009ae0 con 0x7f1134159ba0 2024-01-27T00:47:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f1133fff700 1 --2- 172.21.15.46:0/925053716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f113415ad40 0x7f113415d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:47:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.648+0000 7f11337fe700 1 --2- 172.21.15.46:0/925053716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11341597d0 0x7f11340b0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:47:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.649+0000 7f1132ffd700 1 --2- 172.21.15.46:0/925053716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134159ba0 0x7f113414c1a0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f111c00d860 tx=0x7f111c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.649+0000 7f112b7fe700 1 -- 172.21.15.46:0/925053716 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111c009610 con 0x7f1134159ba0 2024-01-27T00:47:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.649+0000 7f112b7fe700 1 -- 172.21.15.46:0/925053716 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f111c009770 con 0x7f1134159ba0 2024-01-27T00:47:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.649+0000 7f112b7fe700 1 -- 172.21.15.46:0/925053716 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111c009b30 con 0x7f1134159ba0 2024-01-27T00:47:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.649+0000 7f113a22a700 1 -- 172.21.15.46:0/925053716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1134155120 con 0x7f1134159ba0 2024-01-27T00:47:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.649+0000 7f113a22a700 1 -- 172.21.15.46:0/925053716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113415da50 con 0x7f1134159ba0 2024-01-27T00:47:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.650+0000 7f113a22a700 1 -- 172.21.15.46:0/925053716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11341555c0 con 0x7f1134159ba0 2024-01-27T00:47:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.650+0000 7f112b7fe700 1 -- 172.21.15.46:0/925053716 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f111c01e460 con 0x7f1134159ba0 2024-01-27T00:47:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.651+0000 7f112b7fe700 1 --2- 172.21.15.46:0/925053716 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f1114066f30 0x7f11140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.651+0000 7f112b7fe700 1 -- 172.21.15.46:0/925053716 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f111c084cb0 con 0x7f1134159ba0 2024-01-27T00:47:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.651+0000 7f11337fe700 1 --2- 172.21.15.46:0/925053716 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f1114066f30 0x7f11140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:18.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.652+0000 7f11337fe700 1 --2- 172.21.15.46:0/925053716 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f1114066f30 0x7f11140693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1124006fd0 tx=0x7f1124008040 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:18.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.656+0000 7f112b7fe700 1 -- 172.21.15.46:0/925053716 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f111c051e10 con 0x7f1134159ba0 2024-01-27T00:47:18.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:18.860+0000 7f113a22a700 1 -- 172.21.15.46:0/925053716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f113404c710 con 0x7f1134159ba0 2024-01-27T00:47:19.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.480+0000 7f112b7fe700 1 -- 172.21.15.46:0/925053716 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v186) v1 ==== 170+0+0 (secure 0 0 0) 0x7f111c055430 con 0x7f1134159ba0 2024-01-27T00:47:19.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.487+0000 7f113a22a700 1 -- 172.21.15.46:0/925053716 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f1114066f30 msgr2=0x7f11140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:19.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.487+0000 7f113a22a700 1 --2- 172.21.15.46:0/925053716 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f1114066f30 0x7f11140693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1124006fd0 tx=0x7f1124008040 comp rx=0 tx=0).stop 2024-01-27T00:47:19.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.487+0000 7f113a22a700 1 -- 172.21.15.46:0/925053716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134159ba0 msgr2=0x7f113414c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:19.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.487+0000 7f113a22a700 1 --2- 172.21.15.46:0/925053716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134159ba0 0x7f113414c1a0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f111c00d860 tx=0x7f111c008040 comp rx=0 tx=0).stop 2024-01-27T00:47:19.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.495+0000 7f113a22a700 1 -- 172.21.15.46:0/925053716 shutdown_connections 2024-01-27T00:47:19.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.495+0000 7f113a22a700 1 --2- 172.21.15.46:0/925053716 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f1114066f30 0x7f11140693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:19.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.495+0000 7f113a22a700 1 --2- 172.21.15.46:0/925053716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f113415ad40 0x7f113415d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:19.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.495+0000 7f113a22a700 1 --2- 172.21.15.46:0/925053716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11341597d0 0x7f11340b0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:19.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.495+0000 7f113a22a700 1 --2- 172.21.15.46:0/925053716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1134159ba0 0x7f113414c1a0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:19.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.495+0000 7f113a22a700 1 -- 172.21.15.46:0/925053716 >> 172.21.15.46:0/925053716 conn(0x7f11340ca9a0 msgr2=0x7f113406b630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:19.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.495+0000 7f113a22a700 1 -- 172.21.15.46:0/925053716 shutdown_connections 2024-01-27T00:47:19.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.498+0000 7f113a22a700 1 -- 172.21.15.46:0/925053716 wait complete. 2024-01-27T00:47:19.501 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-01-27T00:47:19.519 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-27T00:47:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.656+0000 7f7361780700 1 Processor -- start 2024-01-27T00:47:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.656+0000 7f7361780700 1 -- start start 2024-01-27T00:47:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.658+0000 7f7361780700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f735c0c8da0 0x7f735c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.658+0000 7f7361780700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f735c0caf30 0x7f735c0cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.658+0000 7f7361780700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f735c0ccc00 con 0x7f735c140af0 2024-01-27T00:47:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.658+0000 7f7361780700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f735c0cbbf0 con 0x7f735c0caf30 2024-01-27T00:47:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.658+0000 7f7361780700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c0cc310 con 0x7f735c0c8da0 2024-01-27T00:47:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.658+0000 7f735b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f735c0caf30 0x7f735c0cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.658+0000 7f735b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f735c0caf30 0x7f735c0cb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38988/0 (socket says 172.21.15.46:38988) 2024-01-27T00:47:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.658+0000 7f735b7fe700 1 -- 172.21.15.46:0/4274547199 learned_addr learned my addr 172.21.15.46:0/4274547199 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.658+0000 7f735affd700 1 --2- 172.21.15.46:0/4274547199 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f735c0c8da0 0x7f735c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.658+0000 7f735b7fe700 1 -- 172.21.15.46:0/4274547199 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f735c0c8da0 msgr2=0x7f735c0c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.658+0000 7f735b7fe700 1 --2- 172.21.15.46:0/4274547199 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f735c0c8da0 0x7f735c0c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.658+0000 7f735b7fe700 1 -- 172.21.15.46:0/4274547199 >> v1:172.21.15.106:6789/0 conn(0x7f735c140af0 legacy=0x7f735c140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.658+0000 7f735b7fe700 1 -- 172.21.15.46:0/4274547199 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c0cc5b0 con 0x7f735c0caf30 2024-01-27T00:47:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.659+0000 7f735b7fe700 1 --2- 172.21.15.46:0/4274547199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f735c0caf30 0x7f735c0cb370 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f7354009f20 tx=0x7f7354009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=132d3d11b153f0e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.659+0000 7f7359ffb700 1 -- 172.21.15.46:0/4274547199 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735400e070 con 0x7f735c0caf30 2024-01-27T00:47:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.659+0000 7f7359ffb700 1 -- 172.21.15.46:0/4274547199 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735400ba80 con 0x7f735c0caf30 2024-01-27T00:47:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.659+0000 7f7359ffb700 1 -- 172.21.15.46:0/4274547199 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73540092e0 con 0x7f735c0caf30 2024-01-27T00:47:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.659+0000 7f7361780700 1 -- 172.21.15.46:0/4274547199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f735c0caf30 msgr2=0x7f735c0cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.659+0000 7f7361780700 1 --2- 172.21.15.46:0/4274547199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f735c0caf30 0x7f735c0cb370 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f7354009f20 tx=0x7f7354009840 comp rx=0 tx=0).stop 2024-01-27T00:47:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.659+0000 7f7361780700 1 -- 172.21.15.46:0/4274547199 shutdown_connections 2024-01-27T00:47:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.659+0000 7f7361780700 1 --2- 172.21.15.46:0/4274547199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f735c0caf30 0x7f735c0cb370 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.659+0000 7f7361780700 1 --2- 172.21.15.46:0/4274547199 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f735c0c8da0 0x7f735c0c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.659+0000 7f7361780700 1 -- 172.21.15.46:0/4274547199 >> 172.21.15.46:0/4274547199 conn(0x7f735c06bd90 msgr2=0x7f735c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.659+0000 7f7361780700 1 -- 172.21.15.46:0/4274547199 shutdown_connections 2024-01-27T00:47:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.659+0000 7f7361780700 1 -- 172.21.15.46:0/4274547199 wait complete. 2024-01-27T00:47:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.661+0000 7f7361780700 1 Processor -- start 2024-01-27T00:47:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.661+0000 7f7361780700 1 -- start start 2024-01-27T00:47:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.661+0000 7f7361780700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f735c0c8da0 0x7f735c06a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.661+0000 7f7361780700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f735c0caf30 0x7f735c068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.661+0000 7f7361780700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f735c140af0 0x7f735c16a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.661+0000 7f7361780700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c16acc0 con 0x7f735c0c8da0 2024-01-27T00:47:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f7361780700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c069ce0 con 0x7f735c0caf30 2024-01-27T00:47:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f7361780700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c0690d0 con 0x7f735c140af0 2024-01-27T00:47:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f735affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f735c0c8da0 0x7f735c06a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f735affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f735c0c8da0 0x7f735c06a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38996/0 (socket says 172.21.15.46:38996) 2024-01-27T00:47:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f735affd700 1 -- 172.21.15.46:0/875416115 learned_addr learned my addr 172.21.15.46:0/875416115 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f735a7fc700 1 --2- 172.21.15.46:0/875416115 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f735c0caf30 0x7f735c068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f735b7fe700 1 --2- 172.21.15.46:0/875416115 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f735c140af0 0x7f735c16a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f735affd700 1 -- 172.21.15.46:0/875416115 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f735c140af0 msgr2=0x7f735c16a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f735affd700 1 --2- 172.21.15.46:0/875416115 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f735c140af0 0x7f735c16a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f735affd700 1 -- 172.21.15.46:0/875416115 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f735c0caf30 msgr2=0x7f735c068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f735affd700 1 --2- 172.21.15.46:0/875416115 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f735c0caf30 0x7f735c068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f735affd700 1 -- 172.21.15.46:0/875416115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7354009ae0 con 0x7f735c0c8da0 2024-01-27T00:47:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f735affd700 1 --2- 172.21.15.46:0/875416115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f735c0c8da0 0x7f735c06a170 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f734400d890 tx=0x7f7344008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f7352ffd700 1 -- 172.21.15.46:0/875416115 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73440125f0 con 0x7f735c0c8da0 2024-01-27T00:47:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f7352ffd700 1 -- 172.21.15.46:0/875416115 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7344012750 con 0x7f735c0c8da0 2024-01-27T00:47:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f7361780700 1 -- 172.21.15.46:0/875416115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f735c0693b0 con 0x7f735c0c8da0 2024-01-27T00:47:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f7352ffd700 1 -- 172.21.15.46:0/875416115 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7344012af0 con 0x7f735c0c8da0 2024-01-27T00:47:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.662+0000 7f7361780700 1 -- 172.21.15.46:0/875416115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f735c069970 con 0x7f735c0c8da0 2024-01-27T00:47:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.664+0000 7f7361780700 1 -- 172.21.15.46:0/875416115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f735c069fc0 con 0x7f735c0c8da0 2024-01-27T00:47:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.664+0000 7f7352ffd700 1 -- 172.21.15.46:0/875416115 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f734401e460 con 0x7f735c0c8da0 2024-01-27T00:47:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.665+0000 7f7352ffd700 1 --2- 172.21.15.46:0/875416115 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f733c066f30 0x7f733c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.665+0000 7f7352ffd700 1 -- 172.21.15.46:0/875416115 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6361+0+0 (secure 0 0 0) 0x7f7344084d80 con 0x7f735c0c8da0 2024-01-27T00:47:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.665+0000 7f735a7fc700 1 --2- 172.21.15.46:0/875416115 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f733c066f30 0x7f733c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.665+0000 7f735a7fc700 1 --2- 172.21.15.46:0/875416115 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f733c066f30 0x7f733c0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f735c0c7bf0 tx=0x7f734c007ec0 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:19.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.670+0000 7f7352ffd700 1 -- 172.21.15.46:0/875416115 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7344051cd0 con 0x7f735c0c8da0 2024-01-27T00:47:19.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:19.924+0000 7f7361780700 1 -- 172.21.15.46:0/875416115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f735c04c710 con 0x7f735c0c8da0 2024-01-27T00:47:20.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.509+0000 7f7352ffd700 1 -- 172.21.15.46:0/875416115 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 23 and data pool 24 v93) v1 ==== 174+0+0 (secure 0 0 0) 0x7f73440552f0 con 0x7f735c0c8da0 2024-01-27T00:47:20.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.523+0000 7f7361780700 1 -- 172.21.15.46:0/875416115 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f733c066f30 msgr2=0x7f733c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:20.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.523+0000 7f7361780700 1 --2- 172.21.15.46:0/875416115 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f733c066f30 0x7f733c0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f735c0c7bf0 tx=0x7f734c007ec0 comp rx=0 tx=0).stop 2024-01-27T00:47:20.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.523+0000 7f7361780700 1 -- 172.21.15.46:0/875416115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f735c0c8da0 msgr2=0x7f735c06a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:20.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.523+0000 7f7361780700 1 --2- 172.21.15.46:0/875416115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f735c0c8da0 0x7f735c06a170 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f734400d890 tx=0x7f7344008040 comp rx=0 tx=0).stop 2024-01-27T00:47:20.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.524+0000 7f7361780700 1 -- 172.21.15.46:0/875416115 shutdown_connections 2024-01-27T00:47:20.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.524+0000 7f7361780700 1 --2- 172.21.15.46:0/875416115 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f733c066f30 0x7f733c0693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:20.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.524+0000 7f7361780700 1 --2- 172.21.15.46:0/875416115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f735c0c8da0 0x7f735c06a170 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:20.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.524+0000 7f7361780700 1 --2- 172.21.15.46:0/875416115 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f735c140af0 0x7f735c16a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:20.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.524+0000 7f7361780700 1 --2- 172.21.15.46:0/875416115 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f735c0caf30 0x7f735c068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:20.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.524+0000 7f7361780700 1 -- 172.21.15.46:0/875416115 >> 172.21.15.46:0/875416115 conn(0x7f735c06bd90 msgr2=0x7f735c0b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:20.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.524+0000 7f7361780700 1 -- 172.21.15.46:0/875416115 shutdown_connections 2024-01-27T00:47:20.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.524+0000 7f7361780700 1 -- 172.21.15.46:0/875416115 wait complete. 2024-01-27T00:47:20.532 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 23 and data pool 24 2024-01-27T00:47:20.543 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T00:47:20.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.677+0000 7fc9bebc6700 1 Processor -- start 2024-01-27T00:47:20.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.677+0000 7fc9bebc6700 1 -- start start 2024-01-27T00:47:20.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.678+0000 7fc9bebc6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9b8157340 0x7fc9b815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:20.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.678+0000 7fc9bebc6700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc9b80ae520 0x7fc9b80b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:20.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.678+0000 7fc9bebc6700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b8158ae0 con 0x7fc9b80b3fb0 2024-01-27T00:47:20.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.678+0000 7fc9bebc6700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9b815b650 con 0x7fc9b8157340 2024-01-27T00:47:20.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.678+0000 7fc9bebc6700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9b815c020 con 0x7fc9b80ae520 2024-01-27T00:47:20.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.678+0000 7fc9bc962700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9b8157340 0x7fc9b815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:20.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.678+0000 7fc9bd163700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc9b80b3fb0 0x7fc9b8157710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45614/0 (socket says 172.21.15.46:45614) 2024-01-27T00:47:20.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.678+0000 7fc9bd163700 1 -- 172.21.15.46:0/2623616397 learned_addr learned my addr 172.21.15.46:0/2623616397 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:20.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.678+0000 7fc9b7fff700 1 --2- 172.21.15.46:0/2623616397 >> v2:172.21.15.106:3300/0 conn(0x7fc9b80ae520 0x7fc9b80b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:20.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.678+0000 7fc9bc962700 1 -- 172.21.15.46:0/2623616397 >> v2:172.21.15.106:3300/0 conn(0x7fc9b80ae520 msgr2=0x7fc9b80b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:20.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.678+0000 7fc9bc962700 1 --2- 172.21.15.46:0/2623616397 >> v2:172.21.15.106:3300/0 conn(0x7fc9b80ae520 0x7fc9b80b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:20.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.678+0000 7fc9bc962700 1 -- 172.21.15.46:0/2623616397 >> v1:172.21.15.46:6789/0 conn(0x7fc9b80b3fb0 legacy=0x7fc9b8157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:20.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.678+0000 7fc9b77fe700 1 -- 172.21.15.46:0/2623616397 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973755007 0 0) 0x7fc9b8158ae0 con 0x7fc9b80b3fb0 2024-01-27T00:47:20.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.678+0000 7fc9bc962700 1 -- 172.21.15.46:0/2623616397 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b80b43c0 con 0x7fc9b8157340 2024-01-27T00:47:20.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.679+0000 7fc9bc962700 1 --2- 172.21.15.46:0/2623616397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b8157340 0x7fc9b815bae0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fc9b806e840 tx=0x7fc9ac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6567083808feb872 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:20.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.679+0000 7fc9b77fe700 1 -- 172.21.15.46:0/2623616397 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac011630 con 0x7fc9b8157340 2024-01-27T00:47:20.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.679+0000 7fc9b77fe700 1 -- 172.21.15.46:0/2623616397 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ac011790 con 0x7fc9b8157340 2024-01-27T00:47:20.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.679+0000 7fc9b77fe700 1 -- 172.21.15.46:0/2623616397 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac011b30 con 0x7fc9b8157340 2024-01-27T00:47:20.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.679+0000 7fc9bebc6700 1 -- 172.21.15.46:0/2623616397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b8157340 msgr2=0x7fc9b815bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:20.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.679+0000 7fc9bebc6700 1 --2- 172.21.15.46:0/2623616397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b8157340 0x7fc9b815bae0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fc9b806e840 tx=0x7fc9ac0091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:20.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.679+0000 7fc9bebc6700 1 -- 172.21.15.46:0/2623616397 shutdown_connections 2024-01-27T00:47:20.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.679+0000 7fc9bebc6700 1 --2- 172.21.15.46:0/2623616397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b8157340 0x7fc9b815bae0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:20.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.679+0000 7fc9bebc6700 1 --2- 172.21.15.46:0/2623616397 >> v2:172.21.15.106:3300/0 conn(0x7fc9b80ae520 0x7fc9b80b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:20.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.679+0000 7fc9bebc6700 1 -- 172.21.15.46:0/2623616397 >> 172.21.15.46:0/2623616397 conn(0x7fc9b80ce530 msgr2=0x7fc9b80ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:20.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.680+0000 7fc9bebc6700 1 -- 172.21.15.46:0/2623616397 shutdown_connections 2024-01-27T00:47:20.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.680+0000 7fc9bebc6700 1 -- 172.21.15.46:0/2623616397 wait complete. 2024-01-27T00:47:20.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.681+0000 7fc9bebc6700 1 Processor -- start 2024-01-27T00:47:20.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.681+0000 7fc9bebc6700 1 -- start start 2024-01-27T00:47:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9bebc6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9b80ae520 0x7fc9b816cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9bebc6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9b80b3fb0 0x7fc9b8154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9bebc6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b8157340 0x7fc9b816d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9bebc6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b80c5740 con 0x7fc9b8157340 2024-01-27T00:47:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9bebc6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b8068b70 con 0x7fc9b80b3fb0 2024-01-27T00:47:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9bebc6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b81549b0 con 0x7fc9b80ae520 2024-01-27T00:47:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9bd163700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b8157340 0x7fc9b816d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:20.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9bd163700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b8157340 0x7fc9b816d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40498/0 (socket says 172.21.15.46:40498) 2024-01-27T00:47:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9bd163700 1 -- 172.21.15.46:0/1703866230 learned_addr learned my addr 172.21.15.46:0/1703866230 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9bc962700 1 --2- 172.21.15.46:0/1703866230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9b80ae520 0x7fc9b816cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9b7fff700 1 --2- 172.21.15.46:0/1703866230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9b80b3fb0 0x7fc9b8154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9bd163700 1 -- 172.21.15.46:0/1703866230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9b80ae520 msgr2=0x7fc9b816cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9bd163700 1 --2- 172.21.15.46:0/1703866230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9b80ae520 0x7fc9b816cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9bd163700 1 -- 172.21.15.46:0/1703866230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9b80b3fb0 msgr2=0x7fc9b8154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9bd163700 1 --2- 172.21.15.46:0/1703866230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9b80b3fb0 0x7fc9b8154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9bd163700 1 -- 172.21.15.46:0/1703866230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ac009420 con 0x7fc9b8157340 2024-01-27T00:47:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9bd163700 1 --2- 172.21.15.46:0/1703866230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b8157340 0x7fc9b816d270 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000cc90 tx=0x7fc9b0001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.682+0000 7fc9b4ff9700 1 -- 172.21.15.46:0/1703866230 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b000a8b0 con 0x7fc9b8157340 2024-01-27T00:47:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.683+0000 7fc9b4ff9700 1 -- 172.21.15.46:0/1703866230 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b000aa10 con 0x7fc9b8157340 2024-01-27T00:47:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.683+0000 7fc9bebc6700 1 -- 172.21.15.46:0/1703866230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b80b1eb0 con 0x7fc9b8157340 2024-01-27T00:47:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.683+0000 7fc9b4ff9700 1 -- 172.21.15.46:0/1703866230 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b000add0 con 0x7fc9b8157340 2024-01-27T00:47:20.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.683+0000 7fc9bebc6700 1 -- 172.21.15.46:0/1703866230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b80b2430 con 0x7fc9b8157340 2024-01-27T00:47:20.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.684+0000 7fc9bebc6700 1 -- 172.21.15.46:0/1703866230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b804c710 con 0x7fc9b8157340 2024-01-27T00:47:20.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.684+0000 7fc9b4ff9700 1 -- 172.21.15.46:0/1703866230 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc9b0007300 con 0x7fc9b8157340 2024-01-27T00:47:20.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.684+0000 7fc9b4ff9700 1 --2- 172.21.15.46:0/1703866230 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fc99c066f30 0x7fc99c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:20.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.685+0000 7fc9b4ff9700 1 -- 172.21.15.46:0/1703866230 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fc9b0084ba0 con 0x7fc9b8157340 2024-01-27T00:47:20.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.685+0000 7fc9bc962700 1 --2- 172.21.15.46:0/1703866230 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fc99c066f30 0x7fc99c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:20.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.685+0000 7fc9bc962700 1 --2- 172.21.15.46:0/1703866230 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fc99c066f30 0x7fc99c0693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac0093f0 tx=0x7fc9ac017000 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:20.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.689+0000 7fc9b4ff9700 1 -- 172.21.15.46:0/1703866230 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc9b0051b00 con 0x7fc9b8157340 2024-01-27T00:47:20.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.892+0000 7fc9bebc6700 1 -- 172.21.15.46:0/1703866230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9b8158520 con 0x7fc9b8157340 2024-01-27T00:47:20.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.893+0000 7fc9b4ff9700 1 -- 172.21.15.46:0/1703866230 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+16247 (secure 0 0 0) 0x7fc9b0055120 con 0x7fc9b8157340 2024-01-27T00:47:20.893 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:20.894 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":187,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T00:47:19.925953+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-01-27T00:47:17.642961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-01-27T00:47:18.897371+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T00:47:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.895+0000 7fc9bebc6700 1 -- 172.21.15.46:0/1703866230 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fc99c066f30 msgr2=0x7fc99c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.895+0000 7fc9bebc6700 1 --2- 172.21.15.46:0/1703866230 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fc99c066f30 0x7fc99c0693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac0093f0 tx=0x7fc9ac017000 comp rx=0 tx=0).stop 2024-01-27T00:47:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.895+0000 7fc9bebc6700 1 -- 172.21.15.46:0/1703866230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b8157340 msgr2=0x7fc9b816d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.895+0000 7fc9bebc6700 1 --2- 172.21.15.46:0/1703866230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b8157340 0x7fc9b816d270 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000cc90 tx=0x7fc9b0001a80 comp rx=0 tx=0).stop 2024-01-27T00:47:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.895+0000 7fc9bebc6700 1 -- 172.21.15.46:0/1703866230 shutdown_connections 2024-01-27T00:47:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.895+0000 7fc9bebc6700 1 --2- 172.21.15.46:0/1703866230 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fc99c066f30 0x7fc99c0693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.895+0000 7fc9bebc6700 1 --2- 172.21.15.46:0/1703866230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b8157340 0x7fc9b816d270 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.895+0000 7fc9bebc6700 1 --2- 172.21.15.46:0/1703866230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9b80ae520 0x7fc9b816cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.895+0000 7fc9bebc6700 1 --2- 172.21.15.46:0/1703866230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9b80b3fb0 0x7fc9b8154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:20.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.895+0000 7fc9bebc6700 1 -- 172.21.15.46:0/1703866230 >> 172.21.15.46:0/1703866230 conn(0x7fc9b80ce530 msgr2=0x7fc9b80ade00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:20.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.896+0000 7fc9bebc6700 1 -- 172.21.15.46:0/1703866230 shutdown_connections 2024-01-27T00:47:20.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:20.896+0000 7fc9bebc6700 1 -- 172.21.15.46:0/1703866230 wait complete. 2024-01-27T00:47:20.910 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T00:47:21.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.041+0000 7f4374258700 1 Processor -- start 2024-01-27T00:47:21.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.041+0000 7f4374258700 1 -- start start 2024-01-27T00:47:21.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.042+0000 7f4374258700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f436c15ce40 0x7f436c0d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:21.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.042+0000 7f4374258700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f436c15d210 0x7f436c0cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:21.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.042+0000 7f4374258700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436c0cff70 con 0x7f436c0d1390 2024-01-27T00:47:21.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.042+0000 7f4374258700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f436c0cf400 con 0x7f436c15d210 2024-01-27T00:47:21.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.042+0000 7f4374258700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f436c0cf660 con 0x7f436c15ce40 2024-01-27T00:47:21.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.042+0000 7f43727f5700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f436c0d1390 0x7f436c0ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45620/0 (socket says 172.21.15.46:45620) 2024-01-27T00:47:21.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.042+0000 7f43727f5700 1 -- 172.21.15.46:0/263053626 learned_addr learned my addr 172.21.15.46:0/263053626 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:21.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4371ff4700 1 --2- 172.21.15.46:0/263053626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f436c15ce40 0x7f436c0d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:21.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f43717f3700 1 --2- 172.21.15.46:0/263053626 >> v2:172.21.15.106:3300/0 conn(0x7f436c15d210 0x7f436c0cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:21.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4370ff2700 1 -- 172.21.15.46:0/263053626 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276552776 0 0) 0x7f436c0cff70 con 0x7f436c0d1390 2024-01-27T00:47:21.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4370ff2700 1 -- 172.21.15.46:0/263053626 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4358003570 con 0x7f436c0d1390 2024-01-27T00:47:21.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4370ff2700 1 -- 172.21.15.46:0/263053626 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2549389585 0 0) 0x7f4358003570 con 0x7f436c0d1390 2024-01-27T00:47:21.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4370ff2700 1 -- 172.21.15.46:0/263053626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f436c15ce40 msgr2=0x7f436c0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:21.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4370ff2700 1 --2- 172.21.15.46:0/263053626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f436c15ce40 0x7f436c0d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:21.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4370ff2700 1 -- 172.21.15.46:0/263053626 >> v2:172.21.15.106:3300/0 conn(0x7f436c15d210 msgr2=0x7f436c0cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:21.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4370ff2700 1 --2- 172.21.15.46:0/263053626 >> v2:172.21.15.106:3300/0 conn(0x7f436c15d210 0x7f436c0cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4370ff2700 1 -- 172.21.15.46:0/263053626 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c0d0500 con 0x7f436c0d1390 2024-01-27T00:47:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4370ff2700 1 -- 172.21.15.46:0/263053626 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f43640017f0 con 0x7f436c0d1390 2024-01-27T00:47:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4370ff2700 1 -- 172.21.15.46:0/263053626 >> v1:172.21.15.46:6789/0 conn(0x7f436c0d1390 legacy=0x7f436c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4370ff2700 1 --2- 172.21.15.46:0/263053626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4358003a60 0x7f4358003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4370ff2700 1 --2- 172.21.15.46:0/263053626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f436c15ce40 0x7f4358004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4370ff2700 1 --2- 172.21.15.46:0/263053626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4358004d50 0x7f43580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4370ff2700 1 -- 172.21.15.46:0/263053626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c0cf400 con 0x7f4358004d50 2024-01-27T00:47:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4370ff2700 1 -- 172.21.15.46:0/263053626 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c0cf660 con 0x7f4358003a60 2024-01-27T00:47:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4370ff2700 1 -- 172.21.15.46:0/263053626 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435800ba60 con 0x7f436c15ce40 2024-01-27T00:47:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f43717f3700 1 --2- 172.21.15.46:0/263053626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f436c15ce40 0x7f4358004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f43727f5700 1 --2- 172.21.15.46:0/263053626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4358004d50 0x7f43580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.043+0000 7f4371ff4700 1 --2- 172.21.15.46:0/263053626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4358003a60 0x7f4358003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.044+0000 7f43717f3700 1 -- 172.21.15.46:0/263053626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4358003a60 msgr2=0x7f4358003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:21.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.044+0000 7f43717f3700 1 --2- 172.21.15.46:0/263053626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4358003a60 0x7f4358003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.044+0000 7f43717f3700 1 -- 172.21.15.46:0/263053626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4358004d50 msgr2=0x7f43580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.044+0000 7f43717f3700 1 --2- 172.21.15.46:0/263053626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4358004d50 0x7f43580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.044+0000 7f43717f3700 1 -- 172.21.15.46:0/263053626 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4358003570 con 0x7f436c15ce40 2024-01-27T00:47:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.044+0000 7f43717f3700 1 --2- 172.21.15.46:0/263053626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f436c15ce40 0x7f4358004810 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f436c15d620 tx=0x7f435c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d70a235bd909b9a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.044+0000 7f4370ff2700 1 -- 172.21.15.46:0/263053626 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c00e6b0 con 0x7f436c15ce40 2024-01-27T00:47:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.044+0000 7f4370ff2700 1 -- 172.21.15.46:0/263053626 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435c00e810 con 0x7f436c15ce40 2024-01-27T00:47:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.044+0000 7f4370ff2700 1 -- 172.21.15.46:0/263053626 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c00ebd0 con 0x7f436c15ce40 2024-01-27T00:47:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.045+0000 7f4374258700 1 -- 172.21.15.46:0/263053626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f436c15ce40 msgr2=0x7f4358004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.045+0000 7f4374258700 1 --2- 172.21.15.46:0/263053626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f436c15ce40 0x7f4358004810 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f436c15d620 tx=0x7f435c0091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.045+0000 7f43727f5700 1 -- 172.21.15.46:0/263053626 reap_dead start 2024-01-27T00:47:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.045+0000 7f4374258700 1 -- 172.21.15.46:0/263053626 shutdown_connections 2024-01-27T00:47:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.045+0000 7f4374258700 1 -- 172.21.15.46:0/263053626 >> 172.21.15.46:0/263053626 conn(0x7f436c06f840 msgr2=0x7f436c0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.045+0000 7f4374258700 1 -- 172.21.15.46:0/263053626 shutdown_connections 2024-01-27T00:47:21.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.045+0000 7f4374258700 1 -- 172.21.15.46:0/263053626 wait complete. 2024-01-27T00:47:21.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.046+0000 7f4374258700 1 Processor -- start 2024-01-27T00:47:21.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.046+0000 7f4374258700 1 -- start start 2024-01-27T00:47:21.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.047+0000 7f4374258700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f436c15ce40 0x7f436c09aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:21.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.047+0000 7f4374258700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f436c09be40 0x7f436c149690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:21.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.047+0000 7f4374258700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f436c09c2b0 0x7f436c149bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:21.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.047+0000 7f4374258700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c14a110 con 0x7f436c09c2b0 2024-01-27T00:47:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.047+0000 7f4374258700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c09a1a0 con 0x7f436c15ce40 2024-01-27T00:47:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.047+0000 7f4374258700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f436c16a800 con 0x7f436c09be40 2024-01-27T00:47:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.047+0000 7f43727f5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f436c09c2b0 0x7f436c149bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.047+0000 7f43727f5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f436c09c2b0 0x7f436c149bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40522/0 (socket says 172.21.15.46:40522) 2024-01-27T00:47:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.047+0000 7f43727f5700 1 -- 172.21.15.46:0/680948012 learned_addr learned my addr 172.21.15.46:0/680948012 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.047+0000 7f4371ff4700 1 --2- 172.21.15.46:0/680948012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f436c15ce40 0x7f436c09aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.047+0000 7f43717f3700 1 --2- 172.21.15.46:0/680948012 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f436c09be40 0x7f436c149690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.047+0000 7f43727f5700 1 -- 172.21.15.46:0/680948012 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f436c09be40 msgr2=0x7f436c149690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.047+0000 7f43727f5700 1 --2- 172.21.15.46:0/680948012 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f436c09be40 0x7f436c149690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.047+0000 7f43727f5700 1 -- 172.21.15.46:0/680948012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f436c15ce40 msgr2=0x7f436c09aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.047+0000 7f43727f5700 1 --2- 172.21.15.46:0/680948012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f436c15ce40 0x7f436c09aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.047+0000 7f43727f5700 1 -- 172.21.15.46:0/680948012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c09a3f0 con 0x7f436c09c2b0 2024-01-27T00:47:21.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.048+0000 7f43727f5700 1 --2- 172.21.15.46:0/680948012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f436c09c2b0 0x7f436c149bd0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f4364000f80 tx=0x7f4364002210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.048+0000 7f4361ffb700 1 -- 172.21.15.46:0/680948012 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436400ae50 con 0x7f436c09c2b0 2024-01-27T00:47:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.048+0000 7f4361ffb700 1 -- 172.21.15.46:0/680948012 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43640133f0 con 0x7f436c09c2b0 2024-01-27T00:47:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.048+0000 7f4361ffb700 1 -- 172.21.15.46:0/680948012 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43640137d0 con 0x7f436c09c2b0 2024-01-27T00:47:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.048+0000 7f4374258700 1 -- 172.21.15.46:0/680948012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436c09a6d0 con 0x7f436c09c2b0 2024-01-27T00:47:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.048+0000 7f4374258700 1 -- 172.21.15.46:0/680948012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f436c09b410 con 0x7f436c09c2b0 2024-01-27T00:47:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.049+0000 7f4374258700 1 -- 172.21.15.46:0/680948012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f436c04c710 con 0x7f436c09c2b0 2024-01-27T00:47:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.049+0000 7f4361ffb700 1 -- 172.21.15.46:0/680948012 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4364013bc0 con 0x7f436c09c2b0 2024-01-27T00:47:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.050+0000 7f4361ffb700 1 --2- 172.21.15.46:0/680948012 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f4350066f60 0x7f4350069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.050+0000 7f4361ffb700 1 -- 172.21.15.46:0/680948012 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f43640853d0 con 0x7f436c09c2b0 2024-01-27T00:47:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.050+0000 7f4371ff4700 1 --2- 172.21.15.46:0/680948012 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f4350066f60 0x7f4350069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.050+0000 7f4371ff4700 1 --2- 172.21.15.46:0/680948012 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f4350066f60 0x7f4350069410 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4368009390 tx=0x7f4368008040 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:21.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.054+0000 7f4361ffb700 1 -- 172.21.15.46:0/680948012 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f43640522b0 con 0x7f436c09c2b0 2024-01-27T00:47:21.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.261+0000 7f4374258700 1 -- 172.21.15.46:0/680948012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f436c0cf920 con 0x7f436c09c2b0 2024-01-27T00:47:21.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.262+0000 7f4361ffb700 1 -- 172.21.15.46:0/680948012 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+16247 (secure 0 0 0) 0x7f43640558d0 con 0x7f436c09c2b0 2024-01-27T00:47:21.262 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:21.263 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":187,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T00:47:19.925953+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-01-27T00:47:17.642961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-01-27T00:47:18.897371+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T00:47:21.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.265+0000 7f4374258700 1 -- 172.21.15.46:0/680948012 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f4350066f60 msgr2=0x7f4350069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:21.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.265+0000 7f4374258700 1 --2- 172.21.15.46:0/680948012 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f4350066f60 0x7f4350069410 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4368009390 tx=0x7f4368008040 comp rx=0 tx=0).stop 2024-01-27T00:47:21.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.265+0000 7f4374258700 1 -- 172.21.15.46:0/680948012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f436c09c2b0 msgr2=0x7f436c149bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.265+0000 7f4374258700 1 --2- 172.21.15.46:0/680948012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f436c09c2b0 0x7f436c149bd0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f4364000f80 tx=0x7f4364002210 comp rx=0 tx=0).stop 2024-01-27T00:47:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.265+0000 7f4374258700 1 -- 172.21.15.46:0/680948012 shutdown_connections 2024-01-27T00:47:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.265+0000 7f4374258700 1 --2- 172.21.15.46:0/680948012 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f4350066f60 0x7f4350069410 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.265+0000 7f4374258700 1 --2- 172.21.15.46:0/680948012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f436c09c2b0 0x7f436c149bd0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.265+0000 7f4374258700 1 --2- 172.21.15.46:0/680948012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f436c15ce40 0x7f436c09aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.265+0000 7f4374258700 1 --2- 172.21.15.46:0/680948012 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f436c09be40 0x7f436c149690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.265+0000 7f4374258700 1 -- 172.21.15.46:0/680948012 >> 172.21.15.46:0/680948012 conn(0x7f436c06f840 msgr2=0x7f436c0697b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.265+0000 7f4374258700 1 -- 172.21.15.46:0/680948012 shutdown_connections 2024-01-27T00:47:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.265+0000 7f4374258700 1 -- 172.21.15.46:0/680948012 wait complete. 2024-01-27T00:47:21.279 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-27T00:47:21.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.413+0000 7fd9e0580700 1 Processor -- start 2024-01-27T00:47:21.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.413+0000 7fd9e0580700 1 -- start start 2024-01-27T00:47:21.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.414+0000 7fd9e0580700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd9d80c8da0 0x7fd9d80c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:21.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.414+0000 7fd9e0580700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9d8140af0 0x7fd9d8140f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:21.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.415+0000 7fd9e0580700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d80ccc00 con 0x7fd9d80caf30 2024-01-27T00:47:21.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.415+0000 7fd9e0580700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d80cbbf0 con 0x7fd9d80c8da0 2024-01-27T00:47:21.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.415+0000 7fd9e0580700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d80cc310 con 0x7fd9d8140af0 2024-01-27T00:47:21.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.415+0000 7fd9de31c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd9d80c8da0 0x7fd9d80c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:21.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.415+0000 7fd9de31c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd9d80c8da0 0x7fd9d80c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40526/0 (socket says 172.21.15.46:40526) 2024-01-27T00:47:21.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.415+0000 7fd9de31c700 1 -- 172.21.15.46:0/2668570120 learned_addr learned my addr 172.21.15.46:0/2668570120 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:21.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.415+0000 7fd9ddb1b700 1 --2- 172.21.15.46:0/2668570120 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9d8140af0 0x7fd9d8140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:21.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.415+0000 7fd9de31c700 1 -- 172.21.15.46:0/2668570120 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9d8140af0 msgr2=0x7fd9d8140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:21.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.415+0000 7fd9de31c700 1 --2- 172.21.15.46:0/2668570120 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9d8140af0 0x7fd9d8140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:21.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.415+0000 7fd9de31c700 1 -- 172.21.15.46:0/2668570120 >> v1:172.21.15.106:6789/0 conn(0x7fd9d80caf30 legacy=0x7fd9d80cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:21.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.415+0000 7fd9de31c700 1 -- 172.21.15.46:0/2668570120 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d80cc5b0 con 0x7fd9d80c8da0 2024-01-27T00:47:21.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.415+0000 7fd9de31c700 1 --2- 172.21.15.46:0/2668570120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9d80c8da0 0x7fd9d80c9170 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80099b0 tx=0x7fd9c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cee4bc13da5ffe7e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:21.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.415+0000 7fd9dd31a700 1 -- 172.21.15.46:0/2668570120 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8009ea0 con 0x7fd9d80c8da0 2024-01-27T00:47:21.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.416+0000 7fd9dd31a700 1 -- 172.21.15.46:0/2668570120 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c800b860 con 0x7fd9d80c8da0 2024-01-27T00:47:21.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.416+0000 7fd9dd31a700 1 -- 172.21.15.46:0/2668570120 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c800bc00 con 0x7fd9d80c8da0 2024-01-27T00:47:21.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.416+0000 7fd9e0580700 1 -- 172.21.15.46:0/2668570120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9d80c8da0 msgr2=0x7fd9d80c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:21.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.416+0000 7fd9e0580700 1 --2- 172.21.15.46:0/2668570120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9d80c8da0 0x7fd9d80c9170 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80099b0 tx=0x7fd9c8009250 comp rx=0 tx=0).stop 2024-01-27T00:47:21.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.416+0000 7fd9e0580700 1 -- 172.21.15.46:0/2668570120 shutdown_connections 2024-01-27T00:47:21.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.416+0000 7fd9e0580700 1 --2- 172.21.15.46:0/2668570120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9d80c8da0 0x7fd9d80c9170 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:21.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.416+0000 7fd9e0580700 1 --2- 172.21.15.46:0/2668570120 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9d8140af0 0x7fd9d8140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:21.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.416+0000 7fd9e0580700 1 -- 172.21.15.46:0/2668570120 >> 172.21.15.46:0/2668570120 conn(0x7fd9d806bd90 msgr2=0x7fd9d806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:21.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.416+0000 7fd9e0580700 1 -- 172.21.15.46:0/2668570120 shutdown_connections 2024-01-27T00:47:21.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.416+0000 7fd9e0580700 1 -- 172.21.15.46:0/2668570120 wait complete. 2024-01-27T00:47:21.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.417+0000 7fd9e0580700 1 Processor -- start 2024-01-27T00:47:21.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.417+0000 7fd9e0580700 1 -- start start 2024-01-27T00:47:21.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.418+0000 7fd9e0580700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9d80c8da0 0x7fd9d813cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:21.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.418+0000 7fd9e0580700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9d80caf30 0x7fd9d8157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:21.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.418+0000 7fd9e0580700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9d8140af0 0x7fd9d8156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:21.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.418+0000 7fd9e0580700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d80c60f0 con 0x7fd9d80c8da0 2024-01-27T00:47:21.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.418+0000 7fd9e0580700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d80b3330 con 0x7fd9d80caf30 2024-01-27T00:47:21.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.418+0000 7fd9e0580700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d813d2d0 con 0x7fd9d8140af0 2024-01-27T00:47:21.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.418+0000 7fd9de31c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9d80c8da0 0x7fd9d813cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:21.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.418+0000 7fd9de31c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9d80c8da0 0x7fd9d813cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40534/0 (socket says 172.21.15.46:40534) 2024-01-27T00:47:21.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.418+0000 7fd9de31c700 1 -- 172.21.15.46:0/1694378677 learned_addr learned my addr 172.21.15.46:0/1694378677 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:21.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.418+0000 7fd9deb1d700 1 --2- 172.21.15.46:0/1694378677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9d8140af0 0x7fd9d8156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:21.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.418+0000 7fd9ddb1b700 1 --2- 172.21.15.46:0/1694378677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9d80caf30 0x7fd9d8157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:21.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.419+0000 7fd9de31c700 1 -- 172.21.15.46:0/1694378677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9d8140af0 msgr2=0x7fd9d8156910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:21.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.419+0000 7fd9de31c700 1 --2- 172.21.15.46:0/1694378677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9d8140af0 0x7fd9d8156910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:21.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.419+0000 7fd9de31c700 1 -- 172.21.15.46:0/1694378677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9d80caf30 msgr2=0x7fd9d8157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:21.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.419+0000 7fd9de31c700 1 --2- 172.21.15.46:0/1694378677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9d80caf30 0x7fd9d8157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:21.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.419+0000 7fd9de31c700 1 -- 172.21.15.46:0/1694378677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c80094f0 con 0x7fd9d80c8da0 2024-01-27T00:47:21.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.419+0000 7fd9de31c700 1 --2- 172.21.15.46:0/1694378677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9d80c8da0 0x7fd9d813cd90 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fd9d806f5a0 tx=0x7fd9c8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:21.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.419+0000 7fd9ce7fc700 1 -- 172.21.15.46:0/1694378677 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8000d10 con 0x7fd9d80c8da0 2024-01-27T00:47:21.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.419+0000 7fd9ce7fc700 1 -- 172.21.15.46:0/1694378677 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c800bbe0 con 0x7fd9d80c8da0 2024-01-27T00:47:21.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.419+0000 7fd9e0580700 1 -- 172.21.15.46:0/1694378677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d80c5e10 con 0x7fd9d80c8da0 2024-01-27T00:47:21.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.419+0000 7fd9ce7fc700 1 -- 172.21.15.46:0/1694378677 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8011670 con 0x7fd9d80c8da0 2024-01-27T00:47:21.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.419+0000 7fd9e0580700 1 -- 172.21.15.46:0/1694378677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d80b2fc0 con 0x7fd9d80c8da0 2024-01-27T00:47:21.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.420+0000 7fd9e0580700 1 -- 172.21.15.46:0/1694378677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d80b2b40 con 0x7fd9d80c8da0 2024-01-27T00:47:21.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.421+0000 7fd9ce7fc700 1 -- 172.21.15.46:0/1694378677 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd9c8011aa0 con 0x7fd9d80c8da0 2024-01-27T00:47:21.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.421+0000 7fd9ce7fc700 1 --2- 172.21.15.46:0/1694378677 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fd9bc066f30 0x7fd9bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:21.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.421+0000 7fd9ce7fc700 1 -- 172.21.15.46:0/1694378677 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fd9c80860b0 con 0x7fd9d80c8da0 2024-01-27T00:47:21.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.421+0000 7fd9ddb1b700 1 --2- 172.21.15.46:0/1694378677 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fd9bc066f30 0x7fd9bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:21.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.422+0000 7fd9ddb1b700 1 --2- 172.21.15.46:0/1694378677 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fd9bc066f30 0x7fd9bc0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd9d815aff0 tx=0x7fd9d40091a0 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:21.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.425+0000 7fd9ce7fc700 1 -- 172.21.15.46:0/1694378677 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd9d80b2b40 con 0x7fd9d80c8da0 2024-01-27T00:47:21.672 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-01-27T00:47:21.672+0000 7f4721935700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-27T00:47:21.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:21.683+0000 7fd9e0580700 1 -- 172.21.15.46:0/1694378677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd9d80b3180 con 0x7fd9d80c8da0 2024-01-27T00:47:22.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.511+0000 7fd9ce7fc700 1 -- 172.21.15.46:0/1694378677 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v96) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd9c8053030 con 0x7fd9d80c8da0 2024-01-27T00:47:22.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.514+0000 7fd9e0580700 1 -- 172.21.15.46:0/1694378677 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fd9bc066f30 msgr2=0x7fd9bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:22.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.514+0000 7fd9e0580700 1 --2- 172.21.15.46:0/1694378677 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fd9bc066f30 0x7fd9bc0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd9d815aff0 tx=0x7fd9d40091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:22.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.514+0000 7fd9e0580700 1 -- 172.21.15.46:0/1694378677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9d80c8da0 msgr2=0x7fd9d813cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:22.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.514+0000 7fd9e0580700 1 --2- 172.21.15.46:0/1694378677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9d80c8da0 0x7fd9d813cd90 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fd9d806f5a0 tx=0x7fd9c8017000 comp rx=0 tx=0).stop 2024-01-27T00:47:22.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.514+0000 7fd9e0580700 1 -- 172.21.15.46:0/1694378677 shutdown_connections 2024-01-27T00:47:22.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.514+0000 7fd9e0580700 1 --2- 172.21.15.46:0/1694378677 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fd9bc066f30 0x7fd9bc0693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:22.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.514+0000 7fd9e0580700 1 --2- 172.21.15.46:0/1694378677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9d80c8da0 0x7fd9d813cd90 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:22.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.514+0000 7fd9e0580700 1 --2- 172.21.15.46:0/1694378677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9d8140af0 0x7fd9d8156910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:22.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.514+0000 7fd9e0580700 1 --2- 172.21.15.46:0/1694378677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9d80caf30 0x7fd9d8157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:22.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.514+0000 7fd9e0580700 1 -- 172.21.15.46:0/1694378677 >> 172.21.15.46:0/1694378677 conn(0x7fd9d806bd90 msgr2=0x7fd9d813fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:22.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.514+0000 7fd9e0580700 1 -- 172.21.15.46:0/1694378677 shutdown_connections 2024-01-27T00:47:22.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.514+0000 7fd9e0580700 1 -- 172.21.15.46:0/1694378677 wait complete. 2024-01-27T00:47:22.530 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-27T00:47:22.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.670+0000 7fe422afc700 1 Processor -- start 2024-01-27T00:47:22.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.670+0000 7fe422afc700 1 -- start start 2024-01-27T00:47:22.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.671+0000 7fe422afc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe41c067af0 0x7fe41c067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:22.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.671+0000 7fe422afc700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe41c06fdb0 con 0x7fe41c1597e0 2024-01-27T00:47:22.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.671+0000 7fe422afc700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe41c06f1a0 con 0x7fe41c06ebe0 2024-01-27T00:47:22.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.671+0000 7fe422afc700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe41c06f480 con 0x7fe41c067af0 2024-01-27T00:47:22.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.671+0000 7fe420898700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe41c067af0 0x7fe41c067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.671+0000 7fe421099700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe41c1597e0 0x7fe41c159bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45636/0 (socket says 172.21.15.46:45636) 2024-01-27T00:47:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.671+0000 7fe421099700 1 -- 172.21.15.46:0/1476453034 learned_addr learned my addr 172.21.15.46:0/1476453034 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.671+0000 7fe420898700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe41c067af0 0x7fe41c067ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40540/0 (socket says 172.21.15.46:40540) 2024-01-27T00:47:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.672+0000 7fe420898700 1 -- 172.21.15.46:0/1476453034 >> v1:172.21.15.106:6789/0 conn(0x7fe41c06ebe0 legacy=0x7fe41c0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.672+0000 7fe420898700 1 -- 172.21.15.46:0/1476453034 >> v1:172.21.15.46:6789/0 conn(0x7fe41c1597e0 legacy=0x7fe41c159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.672+0000 7fe420898700 1 -- 172.21.15.46:0/1476453034 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41c06f760 con 0x7fe41c067af0 2024-01-27T00:47:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.672+0000 7fe41b7fe700 1 -- 172.21.15.46:0/1476453034 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093026726 0 0) 0x7fe41c06fdb0 con 0x7fe41c1597e0 2024-01-27T00:47:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.672+0000 7fe420898700 1 --2- 172.21.15.46:0/1476453034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe41c067af0 0x7fe41c067ec0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fe4100098e0 tx=0x7fe4100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=353ae2dd051d64c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.672+0000 7fe41b7fe700 1 -- 172.21.15.46:0/1476453034 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe410009e80 con 0x7fe41c067af0 2024-01-27T00:47:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.672+0000 7fe41b7fe700 1 -- 172.21.15.46:0/1476453034 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41000b860 con 0x7fe41c067af0 2024-01-27T00:47:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.672+0000 7fe41b7fe700 1 -- 172.21.15.46:0/1476453034 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41000bbe0 con 0x7fe41c067af0 2024-01-27T00:47:22.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.672+0000 7fe422afc700 1 -- 172.21.15.46:0/1476453034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe41c067af0 msgr2=0x7fe41c067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:22.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.672+0000 7fe422afc700 1 --2- 172.21.15.46:0/1476453034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe41c067af0 0x7fe41c067ec0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fe4100098e0 tx=0x7fe4100091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:22.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.673+0000 7fe422afc700 1 -- 172.21.15.46:0/1476453034 shutdown_connections 2024-01-27T00:47:22.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.673+0000 7fe422afc700 1 --2- 172.21.15.46:0/1476453034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe41c067af0 0x7fe41c067ec0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:22.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.673+0000 7fe422afc700 1 -- 172.21.15.46:0/1476453034 >> 172.21.15.46:0/1476453034 conn(0x7fe41c0cdf80 msgr2=0x7fe41c06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:22.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.673+0000 7fe422afc700 1 -- 172.21.15.46:0/1476453034 shutdown_connections 2024-01-27T00:47:22.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.673+0000 7fe422afc700 1 -- 172.21.15.46:0/1476453034 wait complete. 2024-01-27T00:47:22.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.674+0000 7fe422afc700 1 Processor -- start 2024-01-27T00:47:22.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.674+0000 7fe422afc700 1 -- start start 2024-01-27T00:47:22.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.675+0000 7fe422afc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe41c067af0 0x7fe41c0aa370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:22.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.675+0000 7fe422afc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe41c06ebe0 0x7fe41c151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:22.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.675+0000 7fe422afc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe41c1597e0 0x7fe41c06c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:22.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.675+0000 7fe422afc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c16b220 con 0x7fe41c06ebe0 2024-01-27T00:47:22.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.675+0000 7fe422afc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c155c40 con 0x7fe41c067af0 2024-01-27T00:47:22.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.675+0000 7fe422afc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c1564e0 con 0x7fe41c1597e0 2024-01-27T00:47:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.675+0000 7fe41bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe41c06ebe0 0x7fe41c151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.675+0000 7fe41bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe41c06ebe0 0x7fe41c151d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40546/0 (socket says 172.21.15.46:40546) 2024-01-27T00:47:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.675+0000 7fe41bfff700 1 -- 172.21.15.46:0/4091710392 learned_addr learned my addr 172.21.15.46:0/4091710392 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.675+0000 7fe420898700 1 --2- 172.21.15.46:0/4091710392 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe41c067af0 0x7fe41c0aa370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.675+0000 7fe421099700 1 --2- 172.21.15.46:0/4091710392 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe41c1597e0 0x7fe41c06c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.675+0000 7fe41bfff700 1 -- 172.21.15.46:0/4091710392 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe41c1597e0 msgr2=0x7fe41c06c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.675+0000 7fe41bfff700 1 --2- 172.21.15.46:0/4091710392 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe41c1597e0 0x7fe41c06c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.675+0000 7fe41bfff700 1 -- 172.21.15.46:0/4091710392 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe41c067af0 msgr2=0x7fe41c0aa370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.675+0000 7fe41bfff700 1 --2- 172.21.15.46:0/4091710392 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe41c067af0 0x7fe41c0aa370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.675+0000 7fe41bfff700 1 -- 172.21.15.46:0/4091710392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe410009420 con 0x7fe41c06ebe0 2024-01-27T00:47:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.676+0000 7fe41bfff700 1 --2- 172.21.15.46:0/4091710392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe41c06ebe0 0x7fe41c151d70 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fe40800c4e0 tx=0x7fe40800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.676+0000 7fe418ff9700 1 -- 172.21.15.46:0/4091710392 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe408011590 con 0x7fe41c06ebe0 2024-01-27T00:47:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.676+0000 7fe418ff9700 1 -- 172.21.15.46:0/4091710392 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe408011710 con 0x7fe41c06ebe0 2024-01-27T00:47:22.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.676+0000 7fe422afc700 1 -- 172.21.15.46:0/4091710392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe41c0ab8d0 con 0x7fe41c06ebe0 2024-01-27T00:47:22.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.676+0000 7fe418ff9700 1 -- 172.21.15.46:0/4091710392 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe408011ad0 con 0x7fe41c06ebe0 2024-01-27T00:47:22.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.676+0000 7fe422afc700 1 -- 172.21.15.46:0/4091710392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe41c156170 con 0x7fe41c06ebe0 2024-01-27T00:47:22.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.677+0000 7fe422afc700 1 -- 172.21.15.46:0/4091710392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe41c156300 con 0x7fe41c06ebe0 2024-01-27T00:47:22.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.677+0000 7fe418ff9700 1 -- 172.21.15.46:0/4091710392 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe40801b3b0 con 0x7fe41c06ebe0 2024-01-27T00:47:22.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.678+0000 7fe418ff9700 1 --2- 172.21.15.46:0/4091710392 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fe400066f30 0x7fe4000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:22.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.678+0000 7fe418ff9700 1 -- 172.21.15.46:0/4091710392 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fe408083be0 con 0x7fe41c06ebe0 2024-01-27T00:47:22.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.678+0000 7fe420898700 1 --2- 172.21.15.46:0/4091710392 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fe400066f30 0x7fe4000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:22.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.679+0000 7fe420898700 1 --2- 172.21.15.46:0/4091710392 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fe400066f30 0x7fe4000693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe4100093f0 tx=0x7fe410017000 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:22.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.683+0000 7fe418ff9700 1 -- 172.21.15.46:0/4091710392 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe408050b60 con 0x7fe41c06ebe0 2024-01-27T00:47:22.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:22.935+0000 7fe422afc700 1 -- 172.21.15.46:0/4091710392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe41c04c710 con 0x7fe41c06ebe0 2024-01-27T00:47:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.519+0000 7fe418ff9700 1 -- 172.21.15.46:0/4091710392 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v97) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe408017070 con 0x7fe41c06ebe0 2024-01-27T00:47:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.521+0000 7fe422afc700 1 -- 172.21.15.46:0/4091710392 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fe400066f30 msgr2=0x7fe4000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.521+0000 7fe422afc700 1 --2- 172.21.15.46:0/4091710392 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fe400066f30 0x7fe4000693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe4100093f0 tx=0x7fe410017000 comp rx=0 tx=0).stop 2024-01-27T00:47:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.521+0000 7fe422afc700 1 -- 172.21.15.46:0/4091710392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe41c06ebe0 msgr2=0x7fe41c151d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.521+0000 7fe422afc700 1 --2- 172.21.15.46:0/4091710392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe41c06ebe0 0x7fe41c151d70 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fe40800c4e0 tx=0x7fe40800c7f0 comp rx=0 tx=0).stop 2024-01-27T00:47:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.521+0000 7fe422afc700 1 -- 172.21.15.46:0/4091710392 shutdown_connections 2024-01-27T00:47:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.521+0000 7fe422afc700 1 --2- 172.21.15.46:0/4091710392 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fe400066f30 0x7fe4000693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.521+0000 7fe422afc700 1 --2- 172.21.15.46:0/4091710392 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe41c1597e0 0x7fe41c06c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.521+0000 7fe422afc700 1 --2- 172.21.15.46:0/4091710392 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe41c067af0 0x7fe41c0aa370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.522+0000 7fe422afc700 1 --2- 172.21.15.46:0/4091710392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe41c06ebe0 0x7fe41c151d70 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.522+0000 7fe422afc700 1 -- 172.21.15.46:0/4091710392 >> 172.21.15.46:0/4091710392 conn(0x7fe41c0cdf80 msgr2=0x7fe41c0cb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.522+0000 7fe422afc700 1 -- 172.21.15.46:0/4091710392 shutdown_connections 2024-01-27T00:47:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.522+0000 7fe422afc700 1 -- 172.21.15.46:0/4091710392 wait complete. 2024-01-27T00:47:23.537 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T00:47:23.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.674+0000 7f08120ee700 1 Processor -- start 2024-01-27T00:47:23.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.674+0000 7f08120ee700 1 -- start start 2024-01-27T00:47:23.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.675+0000 7f08120ee700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f080c0ccc20 0x7f080c068340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:23.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.675+0000 7f08120ee700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f080c0b2590 0x7f080c0b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:23.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.675+0000 7f08120ee700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080c0b2f10 con 0x7f080c0ccff0 2024-01-27T00:47:23.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.675+0000 7f08120ee700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f080c158020 con 0x7f080c0b2590 2024-01-27T00:47:23.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.675+0000 7f08120ee700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080c158300 con 0x7f080c0ccc20 2024-01-27T00:47:23.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.675+0000 7f080affd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f080c0ccff0 0x7f080c06eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45644/0 (socket says 172.21.15.46:45644) 2024-01-27T00:47:23.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.675+0000 7f080affd700 1 -- 172.21.15.46:0/1377607279 learned_addr learned my addr 172.21.15.46:0/1377607279 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:23.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.675+0000 7f080bfff700 1 --2- 172.21.15.46:0/1377607279 >> v2:172.21.15.106:3300/0 conn(0x7f080c0b2590 0x7f080c0b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:23.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.675+0000 7f080b7fe700 1 --2- 172.21.15.46:0/1377607279 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f080c0ccc20 0x7f080c068340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:23.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.675+0000 7f080a7fc700 1 -- 172.21.15.46:0/1377607279 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836727597 0 0) 0x7f080c0b2f10 con 0x7f080c0ccff0 2024-01-27T00:47:23.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.675+0000 7f080a7fc700 1 -- 172.21.15.46:0/1377607279 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07f4003570 con 0x7f080c0ccff0 2024-01-27T00:47:23.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.675+0000 7f080bfff700 1 -- 172.21.15.46:0/1377607279 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f080c0ccc20 msgr2=0x7f080c068340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:23.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.675+0000 7f080bfff700 1 --2- 172.21.15.46:0/1377607279 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f080c0ccc20 0x7f080c068340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:23.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.675+0000 7f080bfff700 1 -- 172.21.15.46:0/1377607279 >> v1:172.21.15.46:6789/0 conn(0x7f080c0ccff0 legacy=0x7f080c06eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:23.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.675+0000 7f080bfff700 1 -- 172.21.15.46:0/1377607279 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c1585e0 con 0x7f080c0b2590 2024-01-27T00:47:23.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.675+0000 7f080b7fe700 1 --2- 172.21.15.46:0/1377607279 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f080c0ccc20 0x7f080c068340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:23.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.676+0000 7f080bfff700 1 --2- 172.21.15.46:0/1377607279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f080c0b2590 0x7f080c0b29d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f0804009e60 tx=0x7f0804009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b09e63411a95b625 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:23.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.676+0000 7f080a7fc700 1 -- 172.21.15.46:0/1377607279 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080400e070 con 0x7f080c0b2590 2024-01-27T00:47:23.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.676+0000 7f080a7fc700 1 -- 172.21.15.46:0/1377607279 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f080400bab0 con 0x7f080c0b2590 2024-01-27T00:47:23.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.676+0000 7f080a7fc700 1 -- 172.21.15.46:0/1377607279 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080400be30 con 0x7f080c0b2590 2024-01-27T00:47:23.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.676+0000 7f08120ee700 1 -- 172.21.15.46:0/1377607279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f080c0b2590 msgr2=0x7f080c0b29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:23.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.676+0000 7f08120ee700 1 --2- 172.21.15.46:0/1377607279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f080c0b2590 0x7f080c0b29d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f0804009e60 tx=0x7f0804009780 comp rx=0 tx=0).stop 2024-01-27T00:47:23.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.676+0000 7f08120ee700 1 -- 172.21.15.46:0/1377607279 shutdown_connections 2024-01-27T00:47:23.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.676+0000 7f08120ee700 1 --2- 172.21.15.46:0/1377607279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f080c0b2590 0x7f080c0b29d0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:23.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.676+0000 7f08120ee700 1 --2- 172.21.15.46:0/1377607279 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f080c0ccc20 0x7f080c068340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:23.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.676+0000 7f08120ee700 1 -- 172.21.15.46:0/1377607279 >> 172.21.15.46:0/1377607279 conn(0x7f080c06b630 msgr2=0x7f080c0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:23.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.677+0000 7f08120ee700 1 -- 172.21.15.46:0/1377607279 shutdown_connections 2024-01-27T00:47:23.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.677+0000 7f08120ee700 1 -- 172.21.15.46:0/1377607279 wait complete. 2024-01-27T00:47:23.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.678+0000 7f08120ee700 1 Processor -- start 2024-01-27T00:47:23.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.678+0000 7f08120ee700 1 -- start start 2024-01-27T00:47:23.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f08120ee700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f080c0b2590 0x7f080c0c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:23.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f08120ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f080c0ccc20 0x7f080c0c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:23.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f08120ee700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f080c0ccff0 0x7f080c167300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:23.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f08120ee700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c0c6770 con 0x7f080c0ccc20 2024-01-27T00:47:23.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f08120ee700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c0c5b60 con 0x7f080c0ccff0 2024-01-27T00:47:23.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f08120ee700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080c0c5e40 con 0x7f080c0b2590 2024-01-27T00:47:23.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f080affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f080c0ccc20 0x7f080c0c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:23.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f080affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f080c0ccc20 0x7f080c0c97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40556/0 (socket says 172.21.15.46:40556) 2024-01-27T00:47:23.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f080affd700 1 -- 172.21.15.46:0/753691198 learned_addr learned my addr 172.21.15.46:0/753691198 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:23.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f080b7fe700 1 --2- 172.21.15.46:0/753691198 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f080c0b2590 0x7f080c0c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:23.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f080bfff700 1 --2- 172.21.15.46:0/753691198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f080c0ccff0 0x7f080c167300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:23.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f080affd700 1 -- 172.21.15.46:0/753691198 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f080c0b2590 msgr2=0x7f080c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:23.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f080affd700 1 --2- 172.21.15.46:0/753691198 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f080c0b2590 0x7f080c0c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:23.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f080affd700 1 -- 172.21.15.46:0/753691198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f080c0ccff0 msgr2=0x7f080c167300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:23.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f080affd700 1 --2- 172.21.15.46:0/753691198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f080c0ccff0 0x7f080c167300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:23.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f080affd700 1 -- 172.21.15.46:0/753691198 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07fc009420 con 0x7f080c0ccc20 2024-01-27T00:47:23.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f080bfff700 1 --2- 172.21.15.46:0/753691198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f080c0ccff0 0x7f080c167300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:47:23.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.679+0000 7f080affd700 1 --2- 172.21.15.46:0/753691198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f080c0ccc20 0x7f080c0c97b0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f08000011e0 tx=0x7f0800007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:23.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.680+0000 7f07f37fe700 1 -- 172.21.15.46:0/753691198 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800005b30 con 0x7f080c0ccc20 2024-01-27T00:47:23.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.680+0000 7f07f37fe700 1 -- 172.21.15.46:0/753691198 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0800005c90 con 0x7f080c0ccc20 2024-01-27T00:47:23.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.680+0000 7f08120ee700 1 -- 172.21.15.46:0/753691198 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0804009a20 con 0x7f080c0ccc20 2024-01-27T00:47:23.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.680+0000 7f07f37fe700 1 -- 172.21.15.46:0/753691198 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800007320 con 0x7f080c0ccc20 2024-01-27T00:47:23.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.680+0000 7f08120ee700 1 -- 172.21.15.46:0/753691198 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080c0c6400 con 0x7f080c0ccc20 2024-01-27T00:47:23.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.681+0000 7f08120ee700 1 -- 172.21.15.46:0/753691198 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080c04c710 con 0x7f080c0ccc20 2024-01-27T00:47:23.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.681+0000 7f07f37fe700 1 -- 172.21.15.46:0/753691198 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0800007690 con 0x7f080c0ccc20 2024-01-27T00:47:23.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.682+0000 7f07f37fe700 1 --2- 172.21.15.46:0/753691198 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f07f8066f30 0x7f07f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:23.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.682+0000 7f07f37fe700 1 -- 172.21.15.46:0/753691198 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f0800083d20 con 0x7f080c0ccc20 2024-01-27T00:47:23.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.682+0000 7f080b7fe700 1 --2- 172.21.15.46:0/753691198 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f07f8066f30 0x7f07f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:23.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.682+0000 7f080b7fe700 1 --2- 172.21.15.46:0/753691198 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f07f8066f30 0x7f07f80693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f07fc0093f0 tx=0x7f07fc008040 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:23.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.686+0000 7f07f37fe700 1 -- 172.21.15.46:0/753691198 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0800050ca0 con 0x7f080c0ccc20 2024-01-27T00:47:23.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.937+0000 7f08120ee700 1 -- 172.21.15.46:0/753691198 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f080c0c68b0 con 0x7f080c0ccc20 2024-01-27T00:47:23.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.938+0000 7f07f37fe700 1 -- 172.21.15.46:0/753691198 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4601 (secure 0 0 0) 0x7f0800017020 con 0x7f080c0ccc20 2024-01-27T00:47:23.938 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:23.938 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":97,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8435,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/4207115513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":4207115513},{"type":"v1","addr":"172.21.15.106:6835","nonce":4207115513}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91},{"gid":8438,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/282430549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":282430549},{"type":"v1","addr":"172.21.15.46:6834","nonce":282430549}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9508,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/361546207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":361546207},{"type":"v1","addr":"172.21.15.106:6834","nonce":361546207}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T00:47:19.926407+0000","modified":"2024-01-27T00:47:23.511130+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9828":{"gid":9828,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2544525048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2544525048},{"type":"v1","addr":"172.21.15.46:6835","nonce":2544525048}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2024-01-27T00:47:23.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.940+0000 7f08120ee700 1 -- 172.21.15.46:0/753691198 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f07f8066f30 msgr2=0x7f07f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:23.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.940+0000 7f08120ee700 1 --2- 172.21.15.46:0/753691198 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f07f8066f30 0x7f07f80693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f07fc0093f0 tx=0x7f07fc008040 comp rx=0 tx=0).stop 2024-01-27T00:47:23.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.940+0000 7f08120ee700 1 -- 172.21.15.46:0/753691198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f080c0ccc20 msgr2=0x7f080c0c97b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:23.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.940+0000 7f08120ee700 1 --2- 172.21.15.46:0/753691198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f080c0ccc20 0x7f080c0c97b0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f08000011e0 tx=0x7f0800007df0 comp rx=0 tx=0).stop 2024-01-27T00:47:23.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.940+0000 7f08120ee700 1 -- 172.21.15.46:0/753691198 shutdown_connections 2024-01-27T00:47:23.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.940+0000 7f08120ee700 1 --2- 172.21.15.46:0/753691198 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f07f8066f30 0x7f07f80693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:23.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.940+0000 7f08120ee700 1 --2- 172.21.15.46:0/753691198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f080c0ccff0 0x7f080c167300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:23.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.940+0000 7f08120ee700 1 --2- 172.21.15.46:0/753691198 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f080c0b2590 0x7f080c0c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:23.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.940+0000 7f08120ee700 1 --2- 172.21.15.46:0/753691198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f080c0ccc20 0x7f080c0c97b0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:23.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.940+0000 7f08120ee700 1 -- 172.21.15.46:0/753691198 >> 172.21.15.46:0/753691198 conn(0x7f080c06b630 msgr2=0x7f080c0cb7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:23.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.940+0000 7f08120ee700 1 -- 172.21.15.46:0/753691198 shutdown_connections 2024-01-27T00:47:23.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:23.940+0000 7f08120ee700 1 -- 172.21.15.46:0/753691198 wait complete. 2024-01-27T00:47:23.942 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 97 2024-01-27T00:47:23.954 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T00:47:24.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.086+0000 7f84ee332700 1 Processor -- start 2024-01-27T00:47:24.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.086+0000 7f84ee332700 1 -- start start 2024-01-27T00:47:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.087+0000 7f84ee332700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f84e80ae520 0x7f84e80b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.087+0000 7f84ee332700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e8158ae0 con 0x7f84e8157340 2024-01-27T00:47:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.087+0000 7f84ee332700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e815b650 con 0x7f84e80b3fb0 2024-01-27T00:47:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.087+0000 7f84ee332700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e80b43c0 con 0x7f84e80ae520 2024-01-27T00:47:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.087+0000 7f84e7fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f84e8157340 0x7f84e815bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45660/0 (socket says 172.21.15.46:45660) 2024-01-27T00:47:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.087+0000 7f84e7fff700 1 -- 172.21.15.46:0/2537711419 learned_addr learned my addr 172.21.15.46:0/2537711419 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.087+0000 7f84e77fe700 1 --2- 172.21.15.46:0/2537711419 >> v2:172.21.15.106:3300/0 conn(0x7f84e80ae520 0x7f84e80b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.088+0000 7f84e6ffd700 1 -- 172.21.15.46:0/2537711419 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594511505 0 0) 0x7f84e8158ae0 con 0x7f84e8157340 2024-01-27T00:47:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.088+0000 7f84e6ffd700 1 -- 172.21.15.46:0/2537711419 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84d0003570 con 0x7f84e8157340 2024-01-27T00:47:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.088+0000 7f84e77fe700 1 -- 172.21.15.46:0/2537711419 >> v1:172.21.15.106:6789/0 conn(0x7f84e80b3fb0 legacy=0x7f84e8157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.088+0000 7f84e77fe700 1 -- 172.21.15.46:0/2537711419 >> v1:172.21.15.46:6789/0 conn(0x7f84e8157340 legacy=0x7f84e815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.088+0000 7f84e77fe700 1 -- 172.21.15.46:0/2537711419 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e8158520 con 0x7f84e80ae520 2024-01-27T00:47:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.088+0000 7f84e77fe700 1 --2- 172.21.15.46:0/2537711419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84e80ae520 0x7f84e80b3a70 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f84dc0098e0 tx=0x7f84dc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24c0fdbce1a176fc server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.088+0000 7f84e6ffd700 1 -- 172.21.15.46:0/2537711419 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc00b640 con 0x7f84e80ae520 2024-01-27T00:47:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.088+0000 7f84e6ffd700 1 -- 172.21.15.46:0/2537711419 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84dc00b7a0 con 0x7f84e80ae520 2024-01-27T00:47:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.088+0000 7f84e6ffd700 1 -- 172.21.15.46:0/2537711419 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc00bb40 con 0x7f84e80ae520 2024-01-27T00:47:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.089+0000 7f84ee332700 1 -- 172.21.15.46:0/2537711419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84e80ae520 msgr2=0x7f84e80b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.089+0000 7f84ee332700 1 --2- 172.21.15.46:0/2537711419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84e80ae520 0x7f84e80b3a70 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f84dc0098e0 tx=0x7f84dc0091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.089+0000 7f84ee332700 1 -- 172.21.15.46:0/2537711419 shutdown_connections 2024-01-27T00:47:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.089+0000 7f84ee332700 1 --2- 172.21.15.46:0/2537711419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84e80ae520 0x7f84e80b3a70 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.089+0000 7f84ee332700 1 -- 172.21.15.46:0/2537711419 >> 172.21.15.46:0/2537711419 conn(0x7f84e80ce530 msgr2=0x7f84e80ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:24.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.089+0000 7f84ee332700 1 -- 172.21.15.46:0/2537711419 shutdown_connections 2024-01-27T00:47:24.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.089+0000 7f84ee332700 1 -- 172.21.15.46:0/2537711419 wait complete. 2024-01-27T00:47:24.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.090+0000 7f84ee332700 1 Processor -- start 2024-01-27T00:47:24.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.091+0000 7f84ee332700 1 -- start start 2024-01-27T00:47:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84ee332700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84e80ae520 0x7f84e81479d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84ee332700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84e80b3fb0 0x7f84e80c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84ee332700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84e8157340 0x7f84e8167b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84ee332700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e8147f10 con 0x7f84e80b3fb0 2024-01-27T00:47:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84ee332700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e80c83c0 con 0x7f84e8157340 2024-01-27T00:47:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84ee332700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e80cb180 con 0x7f84e80ae520 2024-01-27T00:47:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84e77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84e80b3fb0 0x7f84e80c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84e77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84e80b3fb0 0x7f84e80c7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40568/0 (socket says 172.21.15.46:40568) 2024-01-27T00:47:24.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84e77fe700 1 -- 172.21.15.46:0/2891946197 learned_addr learned my addr 172.21.15.46:0/2891946197 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:24.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84e7fff700 1 --2- 172.21.15.46:0/2891946197 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84e80ae520 0x7f84e81479d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:24.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84ec8cf700 1 --2- 172.21.15.46:0/2891946197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84e8157340 0x7f84e8167b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:24.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84e77fe700 1 -- 172.21.15.46:0/2891946197 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84e80ae520 msgr2=0x7f84e81479d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:24.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84e77fe700 1 --2- 172.21.15.46:0/2891946197 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84e80ae520 0x7f84e81479d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84e77fe700 1 -- 172.21.15.46:0/2891946197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84e8157340 msgr2=0x7f84e8167b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:24.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84e77fe700 1 --2- 172.21.15.46:0/2891946197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84e8157340 0x7f84e8167b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84e77fe700 1 -- 172.21.15.46:0/2891946197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc009420 con 0x7f84e80b3fb0 2024-01-27T00:47:24.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84e7fff700 1 --2- 172.21.15.46:0/2891946197 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84e80ae520 0x7f84e81479d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:47:24.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.092+0000 7f84e77fe700 1 --2- 172.21.15.46:0/2891946197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84e80b3fb0 0x7f84e80c7e80 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f84dc0093f0 tx=0x7f84dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:24.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.093+0000 7f84cffff700 1 -- 172.21.15.46:0/2891946197 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc000c40 con 0x7f84e80b3fb0 2024-01-27T00:47:24.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.093+0000 7f84ee332700 1 -- 172.21.15.46:0/2891946197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e80cb460 con 0x7f84e80b3fb0 2024-01-27T00:47:24.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.093+0000 7f84cffff700 1 -- 172.21.15.46:0/2891946197 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84dc00b910 con 0x7f84e80b3fb0 2024-01-27T00:47:24.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.093+0000 7f84cffff700 1 -- 172.21.15.46:0/2891946197 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc025420 con 0x7f84e80b3fb0 2024-01-27T00:47:24.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.093+0000 7f84ee332700 1 -- 172.21.15.46:0/2891946197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e80cba20 con 0x7f84e80b3fb0 2024-01-27T00:47:24.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.094+0000 7f84ee332700 1 -- 172.21.15.46:0/2891946197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e80cbdf0 con 0x7f84e80b3fb0 2024-01-27T00:47:24.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.094+0000 7f84cffff700 1 -- 172.21.15.46:0/2891946197 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f84dc023070 con 0x7f84e80b3fb0 2024-01-27T00:47:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.095+0000 7f84cffff700 1 --2- 172.21.15.46:0/2891946197 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f84d4066f30 0x7f84d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.095+0000 7f84cffff700 1 -- 172.21.15.46:0/2891946197 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f84dc085c50 con 0x7f84e80b3fb0 2024-01-27T00:47:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.095+0000 7f84e7fff700 1 --2- 172.21.15.46:0/2891946197 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f84d4066f30 0x7f84d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.095+0000 7f84e7fff700 1 --2- 172.21.15.46:0/2891946197 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f84d4066f30 0x7f84d40693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f84d8007fd0 tx=0x7f84d8009040 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:24.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.099+0000 7f84cffff700 1 -- 172.21.15.46:0/2891946197 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f84dc052b30 con 0x7f84e80b3fb0 2024-01-27T00:47:24.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.305+0000 7f84ee332700 1 -- 172.21.15.46:0/2891946197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84e804c710 con 0x7f84e80b3fb0 2024-01-27T00:47:24.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.306+0000 7f84cffff700 1 -- 172.21.15.46:0/2891946197 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+16234 (secure 0 0 0) 0x7f84dc015090 con 0x7f84e80b3fb0 2024-01-27T00:47:24.307 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:24.307 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":188,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T00:47:21.490683+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-01-27T00:47:17.642961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-01-27T00:47:18.897371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T00:47:24.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.308+0000 7f84ee332700 1 -- 172.21.15.46:0/2891946197 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f84d4066f30 msgr2=0x7f84d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:24.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.308+0000 7f84ee332700 1 --2- 172.21.15.46:0/2891946197 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f84d4066f30 0x7f84d40693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f84d8007fd0 tx=0x7f84d8009040 comp rx=0 tx=0).stop 2024-01-27T00:47:24.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.308+0000 7f84ee332700 1 -- 172.21.15.46:0/2891946197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84e80b3fb0 msgr2=0x7f84e80c7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:24.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.308+0000 7f84ee332700 1 --2- 172.21.15.46:0/2891946197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84e80b3fb0 0x7f84e80c7e80 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f84dc0093f0 tx=0x7f84dc017000 comp rx=0 tx=0).stop 2024-01-27T00:47:24.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.308+0000 7f84ee332700 1 -- 172.21.15.46:0/2891946197 shutdown_connections 2024-01-27T00:47:24.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.308+0000 7f84ee332700 1 --2- 172.21.15.46:0/2891946197 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f84d4066f30 0x7f84d40693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.308+0000 7f84ee332700 1 --2- 172.21.15.46:0/2891946197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84e8157340 0x7f84e8167b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.308+0000 7f84ee332700 1 --2- 172.21.15.46:0/2891946197 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84e80ae520 0x7f84e81479d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.308+0000 7f84ee332700 1 --2- 172.21.15.46:0/2891946197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84e80b3fb0 0x7f84e80c7e80 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.309+0000 7f84ee332700 1 -- 172.21.15.46:0/2891946197 >> 172.21.15.46:0/2891946197 conn(0x7f84e80ce530 msgr2=0x7f84e8156980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:24.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.309+0000 7f84ee332700 1 -- 172.21.15.46:0/2891946197 shutdown_connections 2024-01-27T00:47:24.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.309+0000 7f84ee332700 1 -- 172.21.15.46:0/2891946197 wait complete. 2024-01-27T00:47:24.323 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-27T00:47:24.323 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T00:47:24.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.454+0000 7fd2ad622700 1 Processor -- start 2024-01-27T00:47:24.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.455+0000 7fd2ad622700 1 -- start start 2024-01-27T00:47:24.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.455+0000 7fd2ad622700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd2a815d460 0x7fd2a8069030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:24.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.455+0000 7fd2ad622700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2a806c3e0 0x7fd2a8069570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:24.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.455+0000 7fd2ad622700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a815d8f0 con 0x7fd2a81596c0 2024-01-27T00:47:24.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.455+0000 7fd2ad622700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a80c98c0 con 0x7fd2a815d460 2024-01-27T00:47:24.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.455+0000 7fd2ad622700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a815dbd0 con 0x7fd2a806c3e0 2024-01-27T00:47:24.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.456+0000 7fd2a67fc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd2a815d460 0x7fd2a8069030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:24.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.456+0000 7fd2a77fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2a806c3e0 0x7fd2a8069570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:24.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.456+0000 7fd2a67fc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd2a815d460 0x7fd2a8069030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:60976/0 (socket says 172.21.15.46:60976) 2024-01-27T00:47:24.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.456+0000 7fd2a6ffd700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fd2a81596c0 0x7fd2a806bcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:47750/0 (socket says 172.21.15.46:47750) 2024-01-27T00:47:24.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.456+0000 7fd2a67fc700 1 -- 172.21.15.46:0/2142302188 learned_addr learned my addr 172.21.15.46:0/2142302188 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:24.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.456+0000 7fd2a67fc700 1 -- 172.21.15.46:0/2142302188 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2a806c3e0 msgr2=0x7fd2a8069570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:24.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.456+0000 7fd2a67fc700 1 --2- 172.21.15.46:0/2142302188 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2a806c3e0 0x7fd2a8069570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.456+0000 7fd2a67fc700 1 -- 172.21.15.46:0/2142302188 >> v1:172.21.15.106:6789/0 conn(0x7fd2a81596c0 legacy=0x7fd2a806bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:24.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.456+0000 7fd2a67fc700 1 -- 172.21.15.46:0/2142302188 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a815deb0 con 0x7fd2a815d460 2024-01-27T00:47:24.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.456+0000 7fd2a67fc700 1 --2- 172.21.15.46:0/2142302188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2a815d460 0x7fd2a8069030 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fd29c0099b0 tx=0x7fd29c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91ca40c6a39eadf7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:24.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.457+0000 7fd2a5ffb700 1 -- 172.21.15.46:0/2142302188 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c009ea0 con 0x7fd2a815d460 2024-01-27T00:47:24.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.457+0000 7fd2a5ffb700 1 -- 172.21.15.46:0/2142302188 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c00b860 con 0x7fd2a815d460 2024-01-27T00:47:24.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.457+0000 7fd2a5ffb700 1 -- 172.21.15.46:0/2142302188 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c00bc00 con 0x7fd2a815d460 2024-01-27T00:47:24.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.457+0000 7fd2ad622700 1 -- 172.21.15.46:0/2142302188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2a815d460 msgr2=0x7fd2a8069030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:24.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.457+0000 7fd2ad622700 1 --2- 172.21.15.46:0/2142302188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2a815d460 0x7fd2a8069030 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fd29c0099b0 tx=0x7fd29c009250 comp rx=0 tx=0).stop 2024-01-27T00:47:24.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.457+0000 7fd2ad622700 1 -- 172.21.15.46:0/2142302188 shutdown_connections 2024-01-27T00:47:24.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.457+0000 7fd2ad622700 1 --2- 172.21.15.46:0/2142302188 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2a806c3e0 0x7fd2a8069570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.457+0000 7fd2ad622700 1 --2- 172.21.15.46:0/2142302188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2a815d460 0x7fd2a8069030 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.457+0000 7fd2ad622700 1 -- 172.21.15.46:0/2142302188 >> 172.21.15.46:0/2142302188 conn(0x7fd2a80ce530 msgr2=0x7fd2a80ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:24.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.457+0000 7fd2ad622700 1 -- 172.21.15.46:0/2142302188 shutdown_connections 2024-01-27T00:47:24.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.457+0000 7fd2ad622700 1 -- 172.21.15.46:0/2142302188 wait complete. 2024-01-27T00:47:24.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.459+0000 7fd2ad622700 1 Processor -- start 2024-01-27T00:47:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.459+0000 7fd2ad622700 1 -- start start 2024-01-27T00:47:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.459+0000 7fd2ad622700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2a806c3e0 0x7fd2a815a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.459+0000 7fd2ad622700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2a81596c0 0x7fd2a8169ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.459+0000 7fd2ad622700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a815d460 0x7fd2a816ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.459+0000 7fd2ad622700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a815c000 con 0x7fd2a815d460 2024-01-27T00:47:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.459+0000 7fd2ad622700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a815a710 con 0x7fd2a81596c0 2024-01-27T00:47:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.459+0000 7fd2ad622700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a815bcd0 con 0x7fd2a806c3e0 2024-01-27T00:47:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.460+0000 7fd2a77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a815d460 0x7fd2a816ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.460+0000 7fd2a77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a815d460 0x7fd2a816ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40578/0 (socket says 172.21.15.46:40578) 2024-01-27T00:47:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.460+0000 7fd2a77fe700 1 -- 172.21.15.46:0/911370877 learned_addr learned my addr 172.21.15.46:0/911370877 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.460+0000 7fd2a6ffd700 1 --2- 172.21.15.46:0/911370877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2a806c3e0 0x7fd2a815a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.460+0000 7fd2a67fc700 1 --2- 172.21.15.46:0/911370877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2a81596c0 0x7fd2a8169ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.460+0000 7fd2a67fc700 1 -- 172.21.15.46:0/911370877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2a806c3e0 msgr2=0x7fd2a815a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.460+0000 7fd2a67fc700 1 --2- 172.21.15.46:0/911370877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2a806c3e0 0x7fd2a815a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.460+0000 7fd2a67fc700 1 -- 172.21.15.46:0/911370877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a815d460 msgr2=0x7fd2a816ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.460+0000 7fd2a67fc700 1 --2- 172.21.15.46:0/911370877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a815d460 0x7fd2a816ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.460+0000 7fd2a67fc700 1 -- 172.21.15.46:0/911370877 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29c0094f0 con 0x7fd2a81596c0 2024-01-27T00:47:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.460+0000 7fd2a77fe700 1 --2- 172.21.15.46:0/911370877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a815d460 0x7fd2a816ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.460+0000 7fd2a67fc700 1 --2- 172.21.15.46:0/911370877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2a81596c0 0x7fd2a8169ad0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fd2a80cf480 tx=0x7fd29c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.460+0000 7fd28effd700 1 -- 172.21.15.46:0/911370877 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c000d10 con 0x7fd2a81596c0 2024-01-27T00:47:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.460+0000 7fd28effd700 1 -- 172.21.15.46:0/911370877 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c00bbe0 con 0x7fd2a81596c0 2024-01-27T00:47:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.460+0000 7fd2ad622700 1 -- 172.21.15.46:0/911370877 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a816a4a0 con 0x7fd2a81596c0 2024-01-27T00:47:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.460+0000 7fd28effd700 1 -- 172.21.15.46:0/911370877 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c011650 con 0x7fd2a81596c0 2024-01-27T00:47:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.461+0000 7fd2ad622700 1 -- 172.21.15.46:0/911370877 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a816aa60 con 0x7fd2a81596c0 2024-01-27T00:47:24.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.461+0000 7fd2ad622700 1 -- 172.21.15.46:0/911370877 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a815bb30 con 0x7fd2a81596c0 2024-01-27T00:47:24.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.462+0000 7fd28effd700 1 -- 172.21.15.46:0/911370877 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd29c011a80 con 0x7fd2a81596c0 2024-01-27T00:47:24.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.462+0000 7fd28effd700 1 --2- 172.21.15.46:0/911370877 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fd294066f30 0x7fd2940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:24.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.463+0000 7fd28effd700 1 -- 172.21.15.46:0/911370877 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fd29c085be0 con 0x7fd2a81596c0 2024-01-27T00:47:24.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.463+0000 7fd2a6ffd700 1 --2- 172.21.15.46:0/911370877 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fd294066f30 0x7fd2940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:24.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.463+0000 7fd2a6ffd700 1 --2- 172.21.15.46:0/911370877 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fd294066f30 0x7fd2940693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd298004fd0 tx=0x7fd298007eb0 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:24.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.467+0000 7fd28effd700 1 -- 172.21.15.46:0/911370877 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd2a815bb30 con 0x7fd2a81596c0 2024-01-27T00:47:24.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.668+0000 7fd2ad622700 1 -- 172.21.15.46:0/911370877 --> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd2a815c1b0 con 0x7fd294066f30 2024-01-27T00:47:24.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.677+0000 7fd28effd700 1 -- 172.21.15.46:0/911370877 <== mgr.9313 v2:172.21.15.106:6836/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284817 (secure 0 0 0) 0x7fd2a815c1b0 con 0x7fd294066f30 2024-01-27T00:47:24.681 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:24.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.681+0000 7fd2ad622700 1 -- 172.21.15.46:0/911370877 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fd294066f30 msgr2=0x7fd2940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:24.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.681+0000 7fd2ad622700 1 --2- 172.21.15.46:0/911370877 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fd294066f30 0x7fd2940693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd298004fd0 tx=0x7fd298007eb0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.681+0000 7fd2ad622700 1 -- 172.21.15.46:0/911370877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2a81596c0 msgr2=0x7fd2a8169ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:24.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.681+0000 7fd2ad622700 1 --2- 172.21.15.46:0/911370877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2a81596c0 0x7fd2a8169ad0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fd2a80cf480 tx=0x7fd29c017000 comp rx=0 tx=0).stop 2024-01-27T00:47:24.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.681+0000 7fd2ad622700 1 -- 172.21.15.46:0/911370877 shutdown_connections 2024-01-27T00:47:24.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.681+0000 7fd2ad622700 1 --2- 172.21.15.46:0/911370877 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fd294066f30 0x7fd2940693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.681+0000 7fd2ad622700 1 --2- 172.21.15.46:0/911370877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a815d460 0x7fd2a816ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.681+0000 7fd2ad622700 1 --2- 172.21.15.46:0/911370877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2a806c3e0 0x7fd2a815a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.681+0000 7fd2ad622700 1 --2- 172.21.15.46:0/911370877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2a81596c0 0x7fd2a8169ad0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.681+0000 7fd2ad622700 1 -- 172.21.15.46:0/911370877 >> 172.21.15.46:0/911370877 conn(0x7fd2a80ce530 msgr2=0x7fd2a8157bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:24.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.681+0000 7fd2ad622700 1 -- 172.21.15.46:0/911370877 shutdown_connections 2024-01-27T00:47:24.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.681+0000 7fd2ad622700 1 -- 172.21.15.46:0/911370877 wait complete. 2024-01-27T00:47:24.685 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T00:47:24.695 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-27T00:47:23.570969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592030,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1784,"num_read_kb":3778,"num_write":563,"num_write_kb":3473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8724,"ondisk_log_size":8724,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2500112,"kb_used_data":19408,"kb_used_omap":0,"kb_used_meta":2480640,"kb_avail":752474608,"statfs":{"total":773094113280,"available":770533998592,"internally_reserved":0,"allocated":19873792,"data_stored":10873836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2540175360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1662,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.095847"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.523334+0000","last_change":"2024-01-27T00:47:20.509897+0000","last_active":"2024-01-27T00:47:20.523334+0000","last_peered":"2024-01-27T00:47:20.523334+0000","last_clean":"2024-01-27T00:47:20.523334+0000","last_became_active":"2024-01-27T00:47:20.509610+0000","last_became_peered":"2024-01-27T00:47:20.509610+0000","last_unstale":"2024-01-27T00:47:20.523334+0000","last_undegraded":"2024-01-27T00:47:20.523334+0000","last_fullsized":"2024-01-27T00:47:20.523334+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:42:01.971654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527541+0000","last_change":"2024-01-27T00:47:20.514165+0000","last_active":"2024-01-27T00:47:20.527541+0000","last_peered":"2024-01-27T00:47:20.527541+0000","last_clean":"2024-01-27T00:47:20.527541+0000","last_became_active":"2024-01-27T00:47:20.513848+0000","last_became_peered":"2024-01-27T00:47:20.513848+0000","last_unstale":"2024-01-27T00:47:20.527541+0000","last_undegraded":"2024-01-27T00:47:20.527541+0000","last_fullsized":"2024-01-27T00:47:20.527541+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:43:04.739251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524282+0000","last_change":"2024-01-27T00:47:20.511666+0000","last_active":"2024-01-27T00:47:20.524282+0000","last_peered":"2024-01-27T00:47:20.524282+0000","last_clean":"2024-01-27T00:47:20.524282+0000","last_became_active":"2024-01-27T00:47:20.509826+0000","last_became_peered":"2024-01-27T00:47:20.509826+0000","last_unstale":"2024-01-27T00:47:20.524282+0000","last_undegraded":"2024-01-27T00:47:20.524282+0000","last_fullsized":"2024-01-27T00:47:20.524282+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:53:47.125547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524159+0000","last_change":"2024-01-27T00:47:20.511731+0000","last_active":"2024-01-27T00:47:20.524159+0000","last_peered":"2024-01-27T00:47:20.524159+0000","last_clean":"2024-01-27T00:47:20.524159+0000","last_became_active":"2024-01-27T00:47:20.511436+0000","last_became_peered":"2024-01-27T00:47:20.511436+0000","last_unstale":"2024-01-27T00:47:20.524159+0000","last_undegraded":"2024-01-27T00:47:20.524159+0000","last_fullsized":"2024-01-27T00:47:20.524159+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:40.342669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524015+0000","last_change":"2024-01-27T00:47:20.510790+0000","last_active":"2024-01-27T00:47:20.524015+0000","last_peered":"2024-01-27T00:47:20.524015+0000","last_clean":"2024-01-27T00:47:20.524015+0000","last_became_active":"2024-01-27T00:47:20.510447+0000","last_became_peered":"2024-01-27T00:47:20.510447+0000","last_unstale":"2024-01-27T00:47:20.524015+0000","last_undegraded":"2024-01-27T00:47:20.524015+0000","last_fullsized":"2024-01-27T00:47:20.524015+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:56:44.894090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531204+0000","last_change":"2024-01-27T00:47:20.520667+0000","last_active":"2024-01-27T00:47:20.531204+0000","last_peered":"2024-01-27T00:47:20.531204+0000","last_clean":"2024-01-27T00:47:20.531204+0000","last_became_active":"2024-01-27T00:47:20.519891+0000","last_became_peered":"2024-01-27T00:47:20.519891+0000","last_unstale":"2024-01-27T00:47:20.531204+0000","last_undegraded":"2024-01-27T00:47:20.531204+0000","last_fullsized":"2024-01-27T00:47:20.531204+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:49:22.885968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522694+0000","last_change":"2024-01-27T00:47:20.513550+0000","last_active":"2024-01-27T00:47:20.522694+0000","last_peered":"2024-01-27T00:47:20.522694+0000","last_clean":"2024-01-27T00:47:20.522694+0000","last_became_active":"2024-01-27T00:47:20.512667+0000","last_became_peered":"2024-01-27T00:47:20.512667+0000","last_unstale":"2024-01-27T00:47:20.522694+0000","last_undegraded":"2024-01-27T00:47:20.522694+0000","last_fullsized":"2024-01-27T00:47:20.522694+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:09:43.978584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.504493+0000","last_change":"2024-01-27T00:47:20.500113+0000","last_active":"2024-01-27T00:47:20.504493+0000","last_peered":"2024-01-27T00:47:20.504493+0000","last_clean":"2024-01-27T00:47:20.504493+0000","last_became_active":"2024-01-27T00:47:20.499814+0000","last_became_peered":"2024-01-27T00:47:20.499814+0000","last_unstale":"2024-01-27T00:47:20.504493+0000","last_undegraded":"2024-01-27T00:47:20.504493+0000","last_fullsized":"2024-01-27T00:47:20.504493+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:05:38.788368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.532911+0000","last_change":"2024-01-27T00:47:20.518490+0000","last_active":"2024-01-27T00:47:20.532911+0000","last_peered":"2024-01-27T00:47:20.532911+0000","last_clean":"2024-01-27T00:47:20.532911+0000","last_became_active":"2024-01-27T00:47:20.518138+0000","last_became_peered":"2024-01-27T00:47:20.518138+0000","last_unstale":"2024-01-27T00:47:20.532911+0000","last_undegraded":"2024-01-27T00:47:20.532911+0000","last_fullsized":"2024-01-27T00:47:20.532911+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:28.557943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531230+0000","last_change":"2024-01-27T00:47:20.520931+0000","last_active":"2024-01-27T00:47:20.531230+0000","last_peered":"2024-01-27T00:47:20.531230+0000","last_clean":"2024-01-27T00:47:20.531230+0000","last_became_active":"2024-01-27T00:47:20.520322+0000","last_became_peered":"2024-01-27T00:47:20.520322+0000","last_unstale":"2024-01-27T00:47:20.531230+0000","last_undegraded":"2024-01-27T00:47:20.531230+0000","last_fullsized":"2024-01-27T00:47:20.531230+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:04:43.917528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.530444+0000","last_change":"2024-01-27T00:47:20.522803+0000","last_active":"2024-01-27T00:47:20.530444+0000","last_peered":"2024-01-27T00:47:20.530444+0000","last_clean":"2024-01-27T00:47:20.530444+0000","last_became_active":"2024-01-27T00:47:20.522463+0000","last_became_peered":"2024-01-27T00:47:20.522463+0000","last_unstale":"2024-01-27T00:47:20.530444+0000","last_undegraded":"2024-01-27T00:47:20.530444+0000","last_fullsized":"2024-01-27T00:47:20.530444+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:04:18.616847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.504894+0000","last_change":"2024-01-27T00:47:20.503158+0000","last_active":"2024-01-27T00:47:20.504894+0000","last_peered":"2024-01-27T00:47:20.504894+0000","last_clean":"2024-01-27T00:47:20.504894+0000","last_became_active":"2024-01-27T00:47:20.502875+0000","last_became_peered":"2024-01-27T00:47:20.502875+0000","last_unstale":"2024-01-27T00:47:20.504894+0000","last_undegraded":"2024-01-27T00:47:20.504894+0000","last_fullsized":"2024-01-27T00:47:20.504894+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:33:13.706183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.513540+0000","last_change":"2024-01-27T00:47:20.509928+0000","last_active":"2024-01-27T00:47:20.513540+0000","last_peered":"2024-01-27T00:47:20.513540+0000","last_clean":"2024-01-27T00:47:20.513540+0000","last_became_active":"2024-01-27T00:47:20.509630+0000","last_became_peered":"2024-01-27T00:47:20.509630+0000","last_unstale":"2024-01-27T00:47:20.513540+0000","last_undegraded":"2024-01-27T00:47:20.513540+0000","last_fullsized":"2024-01-27T00:47:20.513540+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:02:52.058658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531770+0000","last_change":"2024-01-27T00:47:20.527150+0000","last_active":"2024-01-27T00:47:20.531770+0000","last_peered":"2024-01-27T00:47:20.531770+0000","last_clean":"2024-01-27T00:47:20.531770+0000","last_became_active":"2024-01-27T00:47:20.523827+0000","last_became_peered":"2024-01-27T00:47:20.523827+0000","last_unstale":"2024-01-27T00:47:20.531770+0000","last_undegraded":"2024-01-27T00:47:20.531770+0000","last_fullsized":"2024-01-27T00:47:20.531770+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:52:04.785901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.480155+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:27:44.775426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.510752+0000","last_change":"2024-01-27T00:47:20.509849+0000","last_active":"2024-01-27T00:47:20.510752+0000","last_peered":"2024-01-27T00:47:20.510752+0000","last_clean":"2024-01-27T00:47:20.510752+0000","last_became_active":"2024-01-27T00:47:20.509553+0000","last_became_peered":"2024-01-27T00:47:20.509553+0000","last_unstale":"2024-01-27T00:47:20.510752+0000","last_undegraded":"2024-01-27T00:47:20.510752+0000","last_fullsized":"2024-01-27T00:47:20.510752+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:44:26.489354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.528860+0000","last_change":"2024-01-27T00:47:19.498621+0000","last_active":"2024-01-27T00:47:20.528860+0000","last_peered":"2024-01-27T00:47:20.528860+0000","last_clean":"2024-01-27T00:47:20.528860+0000","last_became_active":"2024-01-27T00:47:19.498357+0000","last_became_peered":"2024-01-27T00:47:19.498357+0000","last_unstale":"2024-01-27T00:47:20.528860+0000","last_undegraded":"2024-01-27T00:47:20.528860+0000","last_fullsized":"2024-01-27T00:47:20.528860+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:20:56.150512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522707+0000","last_change":"2024-01-27T00:47:20.513840+0000","last_active":"2024-01-27T00:47:20.522707+0000","last_peered":"2024-01-27T00:47:20.522707+0000","last_clean":"2024-01-27T00:47:20.522707+0000","last_became_active":"2024-01-27T00:47:20.512745+0000","last_became_peered":"2024-01-27T00:47:20.512745+0000","last_unstale":"2024-01-27T00:47:20.522707+0000","last_undegraded":"2024-01-27T00:47:20.522707+0000","last_fullsized":"2024-01-27T00:47:20.522707+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:11:59.061116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.482562+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:43:05.614968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1c","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.533308+0000","last_change":"2024-01-27T00:47:19.505152+0000","last_active":"2024-01-27T00:47:20.533308+0000","last_peered":"2024-01-27T00:47:20.533308+0000","last_clean":"2024-01-27T00:47:20.533308+0000","last_became_active":"2024-01-27T00:47:19.504128+0000","last_became_peered":"2024-01-27T00:47:19.504128+0000","last_unstale":"2024-01-27T00:47:20.533308+0000","last_undegraded":"2024-01-27T00:47:20.533308+0000","last_fullsized":"2024-01-27T00:47:20.533308+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:52.404716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524071+0000","last_change":"2024-01-27T00:47:20.510375+0000","last_active":"2024-01-27T00:47:20.524071+0000","last_peered":"2024-01-27T00:47:20.524071+0000","last_clean":"2024-01-27T00:47:20.524071+0000","last_became_active":"2024-01-27T00:47:20.510069+0000","last_became_peered":"2024-01-27T00:47:20.510069+0000","last_unstale":"2024-01-27T00:47:20.524071+0000","last_undegraded":"2024-01-27T00:47:20.524071+0000","last_fullsized":"2024-01-27T00:47:20.524071+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:44.855002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.528716+0000","last_change":"2024-01-27T00:47:19.488294+0000","last_active":"2024-01-27T00:47:20.528716+0000","last_peered":"2024-01-27T00:47:20.528716+0000","last_clean":"2024-01-27T00:47:20.528716+0000","last_became_active":"2024-01-27T00:47:19.487984+0000","last_became_peered":"2024-01-27T00:47:19.487984+0000","last_unstale":"2024-01-27T00:47:20.528716+0000","last_undegraded":"2024-01-27T00:47:20.528716+0000","last_fullsized":"2024-01-27T00:47:20.528716+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:08:02.191122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.513175+0000","last_change":"2024-01-27T00:47:20.508823+0000","last_active":"2024-01-27T00:47:20.513175+0000","last_peered":"2024-01-27T00:47:20.513175+0000","last_clean":"2024-01-27T00:47:20.513175+0000","last_became_active":"2024-01-27T00:47:20.507955+0000","last_became_peered":"2024-01-27T00:47:20.507955+0000","last_unstale":"2024-01-27T00:47:20.513175+0000","last_undegraded":"2024-01-27T00:47:20.513175+0000","last_fullsized":"2024-01-27T00:47:20.513175+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:44.632841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531724+0000","last_change":"2024-01-27T00:47:19.493726+0000","last_active":"2024-01-27T00:47:20.531724+0000","last_peered":"2024-01-27T00:47:20.531724+0000","last_clean":"2024-01-27T00:47:20.531724+0000","last_became_active":"2024-01-27T00:47:19.493237+0000","last_became_peered":"2024-01-27T00:47:19.493237+0000","last_unstale":"2024-01-27T00:47:20.531724+0000","last_undegraded":"2024-01-27T00:47:20.531724+0000","last_fullsized":"2024-01-27T00:47:20.531724+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:33:12.701672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.532639+0000","last_change":"2024-01-27T00:47:20.516445+0000","last_active":"2024-01-27T00:47:20.532639+0000","last_peered":"2024-01-27T00:47:20.532639+0000","last_clean":"2024-01-27T00:47:20.532639+0000","last_became_active":"2024-01-27T00:47:20.515582+0000","last_became_peered":"2024-01-27T00:47:20.515582+0000","last_unstale":"2024-01-27T00:47:20.532639+0000","last_undegraded":"2024-01-27T00:47:20.532639+0000","last_fullsized":"2024-01-27T00:47:20.532639+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:07:16.136646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.507907+0000","last_change":"2024-01-27T00:47:19.505534+0000","last_active":"2024-01-27T00:47:20.507907+0000","last_peered":"2024-01-27T00:47:20.507907+0000","last_clean":"2024-01-27T00:47:20.507907+0000","last_became_active":"2024-01-27T00:47:19.504749+0000","last_became_peered":"2024-01-27T00:47:19.504749+0000","last_unstale":"2024-01-27T00:47:20.507907+0000","last_undegraded":"2024-01-27T00:47:20.507907+0000","last_fullsized":"2024-01-27T00:47:20.507907+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:48.314323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.492777+0000","last_change":"2024-01-27T00:47:19.488903+0000","last_active":"2024-01-27T00:47:20.492777+0000","last_peered":"2024-01-27T00:47:20.492777+0000","last_clean":"2024-01-27T00:47:20.492777+0000","last_became_active":"2024-01-27T00:47:19.488649+0000","last_became_peered":"2024-01-27T00:47:19.488649+0000","last_unstale":"2024-01-27T00:47:20.492777+0000","last_undegraded":"2024-01-27T00:47:20.492777+0000","last_fullsized":"2024-01-27T00:47:20.492777+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:36:32.698812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499880+0000","last_change":"2024-01-27T00:47:19.505276+0000","last_active":"2024-01-27T00:47:20.499880+0000","last_peered":"2024-01-27T00:47:20.499880+0000","last_clean":"2024-01-27T00:47:20.499880+0000","last_became_active":"2024-01-27T00:47:19.504824+0000","last_became_peered":"2024-01-27T00:47:19.504824+0000","last_unstale":"2024-01-27T00:47:20.499880+0000","last_undegraded":"2024-01-27T00:47:20.499880+0000","last_fullsized":"2024-01-27T00:47:20.499880+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:53:04.664266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.510043+0000","last_change":"2024-01-27T00:47:20.507243+0000","last_active":"2024-01-27T00:47:20.510043+0000","last_peered":"2024-01-27T00:47:20.510043+0000","last_clean":"2024-01-27T00:47:20.510043+0000","last_became_active":"2024-01-27T00:47:20.506953+0000","last_became_peered":"2024-01-27T00:47:20.506953+0000","last_unstale":"2024-01-27T00:47:20.510043+0000","last_undegraded":"2024-01-27T00:47:20.510043+0000","last_fullsized":"2024-01-27T00:47:20.510043+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:23:47.466789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.479792+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:44:20.701222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.530619+0000","last_change":"2024-01-27T00:47:20.516228+0000","last_active":"2024-01-27T00:47:20.530619+0000","last_peered":"2024-01-27T00:47:20.530619+0000","last_clean":"2024-01-27T00:47:20.530619+0000","last_became_active":"2024-01-27T00:47:20.515906+0000","last_became_peered":"2024-01-27T00:47:20.515906+0000","last_unstale":"2024-01-27T00:47:20.530619+0000","last_undegraded":"2024-01-27T00:47:20.530619+0000","last_fullsized":"2024-01-27T00:47:20.530619+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:12:20.214602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"187'6","reported_seq":16,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.534091+0000","last_change":"2024-01-27T00:47:19.502338+0000","last_active":"2024-01-27T00:47:20.534091+0000","last_peered":"2024-01-27T00:47:20.534091+0000","last_clean":"2024-01-27T00:47:20.534091+0000","last_became_active":"2024-01-27T00:47:19.502072+0000","last_became_peered":"2024-01-27T00:47:19.502072+0000","last_unstale":"2024-01-27T00:47:20.534091+0000","last_undegraded":"2024-01-27T00:47:20.534091+0000","last_fullsized":"2024-01-27T00:47:20.534091+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:34:16.405629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.13","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527814+0000","last_change":"2024-01-27T00:47:19.491092+0000","last_active":"2024-01-27T00:47:20.527814+0000","last_peered":"2024-01-27T00:47:20.527814+0000","last_clean":"2024-01-27T00:47:20.527814+0000","last_became_active":"2024-01-27T00:47:19.490799+0000","last_became_peered":"2024-01-27T00:47:19.490799+0000","last_unstale":"2024-01-27T00:47:20.527814+0000","last_undegraded":"2024-01-27T00:47:20.527814+0000","last_fullsized":"2024-01-27T00:47:20.527814+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:23:46.462278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527448+0000","last_change":"2024-01-27T00:47:20.515730+0000","last_active":"2024-01-27T00:47:20.527448+0000","last_peered":"2024-01-27T00:47:20.527448+0000","last_clean":"2024-01-27T00:47:20.527448+0000","last_became_active":"2024-01-27T00:47:20.515422+0000","last_became_peered":"2024-01-27T00:47:20.515422+0000","last_unstale":"2024-01-27T00:47:20.527448+0000","last_undegraded":"2024-01-27T00:47:20.527448+0000","last_fullsized":"2024-01-27T00:47:20.527448+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:33.654223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.498821+0000","last_change":"2024-01-27T00:47:19.499915+0000","last_active":"2024-01-27T00:47:20.498821+0000","last_peered":"2024-01-27T00:47:20.498821+0000","last_clean":"2024-01-27T00:47:20.498821+0000","last_became_active":"2024-01-27T00:47:19.499418+0000","last_became_peered":"2024-01-27T00:47:19.499418+0000","last_unstale":"2024-01-27T00:47:20.498821+0000","last_undegraded":"2024-01-27T00:47:20.498821+0000","last_fullsized":"2024-01-27T00:47:20.498821+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:46.642397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.482438+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:43.628330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.514859+0000","last_change":"2024-01-27T00:47:20.513843+0000","last_active":"2024-01-27T00:47:20.514859+0000","last_peered":"2024-01-27T00:47:20.514859+0000","last_clean":"2024-01-27T00:47:20.514859+0000","last_became_active":"2024-01-27T00:47:20.513188+0000","last_became_peered":"2024-01-27T00:47:20.513188+0000","last_unstale":"2024-01-27T00:47:20.514859+0000","last_undegraded":"2024-01-27T00:47:20.514859+0000","last_fullsized":"2024-01-27T00:47:20.514859+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:15:01.330526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"187'8","reported_seq":18,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.529029+0000","last_change":"2024-01-27T00:47:19.491898+0000","last_active":"2024-01-27T00:47:20.529029+0000","last_peered":"2024-01-27T00:47:20.529029+0000","last_clean":"2024-01-27T00:47:20.529029+0000","last_became_active":"2024-01-27T00:47:19.491597+0000","last_became_peered":"2024-01-27T00:47:19.491597+0000","last_unstale":"2024-01-27T00:47:20.529029+0000","last_undegraded":"2024-01-27T00:47:20.529029+0000","last_fullsized":"2024-01-27T00:47:20.529029+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:47:33.870316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524260+0000","last_change":"2024-01-27T00:47:20.511230+0000","last_active":"2024-01-27T00:47:20.524260+0000","last_peered":"2024-01-27T00:47:20.524260+0000","last_clean":"2024-01-27T00:47:20.524260+0000","last_became_active":"2024-01-27T00:47:20.510893+0000","last_became_peered":"2024-01-27T00:47:20.510893+0000","last_unstale":"2024-01-27T00:47:20.524260+0000","last_undegraded":"2024-01-27T00:47:20.524260+0000","last_fullsized":"2024-01-27T00:47:20.524260+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:51:40.896423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531660+0000","last_change":"2024-01-27T00:47:19.504875+0000","last_active":"2024-01-27T00:47:20.531660+0000","last_peered":"2024-01-27T00:47:20.531660+0000","last_clean":"2024-01-27T00:47:20.531660+0000","last_became_active":"2024-01-27T00:47:19.504575+0000","last_became_peered":"2024-01-27T00:47:19.504575+0000","last_unstale":"2024-01-27T00:47:20.531660+0000","last_undegraded":"2024-01-27T00:47:20.531660+0000","last_fullsized":"2024-01-27T00:47:20.531660+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:43:03.734740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.e","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.533275+0000","last_change":"2024-01-27T00:47:19.509498+0000","last_active":"2024-01-27T00:47:20.533275+0000","last_peered":"2024-01-27T00:47:20.533275+0000","last_clean":"2024-01-27T00:47:20.533275+0000","last_became_active":"2024-01-27T00:47:19.509151+0000","last_became_peered":"2024-01-27T00:47:19.509151+0000","last_unstale":"2024-01-27T00:47:20.533275+0000","last_undegraded":"2024-01-27T00:47:20.533275+0000","last_fullsized":"2024-01-27T00:47:20.533275+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:54:21.075786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.d","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.529256+0000","last_change":"2024-01-27T00:47:19.498982+0000","last_active":"2024-01-27T00:47:20.529256+0000","last_peered":"2024-01-27T00:47:20.529256+0000","last_clean":"2024-01-27T00:47:20.529256+0000","last_became_active":"2024-01-27T00:47:19.498718+0000","last_became_peered":"2024-01-27T00:47:19.498718+0000","last_unstale":"2024-01-27T00:47:20.529256+0000","last_undegraded":"2024-01-27T00:47:20.529256+0000","last_fullsized":"2024-01-27T00:47:20.529256+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:04:42.913017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.493212+0000","last_change":"2024-01-27T00:47:19.493973+0000","last_active":"2024-01-27T00:47:20.493212+0000","last_peered":"2024-01-27T00:47:20.493212+0000","last_clean":"2024-01-27T00:47:20.493212+0000","last_became_active":"2024-01-27T00:47:19.493685+0000","last_became_peered":"2024-01-27T00:47:19.493685+0000","last_unstale":"2024-01-27T00:47:20.493212+0000","last_undegraded":"2024-01-27T00:47:20.493212+0000","last_fullsized":"2024-01-27T00:47:20.493212+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:08:00.978209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531595+0000","last_change":"2024-01-27T00:47:20.525394+0000","last_active":"2024-01-27T00:47:20.531595+0000","last_peered":"2024-01-27T00:47:20.531595+0000","last_clean":"2024-01-27T00:47:20.531595+0000","last_became_active":"2024-01-27T00:47:20.521446+0000","last_became_peered":"2024-01-27T00:47:20.521446+0000","last_unstale":"2024-01-27T00:47:20.531595+0000","last_undegraded":"2024-01-27T00:47:20.531595+0000","last_fullsized":"2024-01-27T00:47:20.531595+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:31.735208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.529633+0000","last_change":"2024-01-27T00:47:19.495677+0000","last_active":"2024-01-27T00:47:20.529633+0000","last_peered":"2024-01-27T00:47:20.529633+0000","last_clean":"2024-01-27T00:47:20.529633+0000","last_became_active":"2024-01-27T00:47:19.494329+0000","last_became_peered":"2024-01-27T00:47:19.494329+0000","last_unstale":"2024-01-27T00:47:20.529633+0000","last_undegraded":"2024-01-27T00:47:20.529633+0000","last_fullsized":"2024-01-27T00:47:20.529633+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:19:25.040151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522962+0000","last_change":"2024-01-27T00:47:20.512100+0000","last_active":"2024-01-27T00:47:20.522962+0000","last_peered":"2024-01-27T00:47:20.522962+0000","last_clean":"2024-01-27T00:47:20.522962+0000","last_became_active":"2024-01-27T00:47:20.511807+0000","last_became_peered":"2024-01-27T00:47:20.511807+0000","last_unstale":"2024-01-27T00:47:20.522962+0000","last_undegraded":"2024-01-27T00:47:20.522962+0000","last_fullsized":"2024-01-27T00:47:20.522962+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:41:19.485849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.482501+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:10:10.727612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522508+0000","last_change":"2024-01-27T00:47:20.512072+0000","last_active":"2024-01-27T00:47:20.522508+0000","last_peered":"2024-01-27T00:47:20.522508+0000","last_clean":"2024-01-27T00:47:20.522508+0000","last_became_active":"2024-01-27T00:47:20.511716+0000","last_became_peered":"2024-01-27T00:47:20.511716+0000","last_unstale":"2024-01-27T00:47:20.522508+0000","last_undegraded":"2024-01-27T00:47:20.522508+0000","last_fullsized":"2024-01-27T00:47:20.522508+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:18:15.805971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.508031+0000","last_change":"2024-01-27T00:47:19.505685+0000","last_active":"2024-01-27T00:47:20.508031+0000","last_peered":"2024-01-27T00:47:20.508031+0000","last_clean":"2024-01-27T00:47:20.508031+0000","last_became_active":"2024-01-27T00:47:19.505201+0000","last_became_peered":"2024-01-27T00:47:19.505201+0000","last_unstale":"2024-01-27T00:47:20.508031+0000","last_undegraded":"2024-01-27T00:47:20.508031+0000","last_fullsized":"2024-01-27T00:47:20.508031+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:10:58.917172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.481168+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:38:22.205934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.7","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527493+0000","last_change":"2024-01-27T00:47:19.498236+0000","last_active":"2024-01-27T00:47:20.527493+0000","last_peered":"2024-01-27T00:47:20.527493+0000","last_clean":"2024-01-27T00:47:20.527493+0000","last_became_active":"2024-01-27T00:47:19.497955+0000","last_became_peered":"2024-01-27T00:47:19.497955+0000","last_unstale":"2024-01-27T00:47:20.527493+0000","last_undegraded":"2024-01-27T00:47:20.527493+0000","last_fullsized":"2024-01-27T00:47:20.527493+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:52:03.781390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524123+0000","last_change":"2024-01-27T00:47:20.510553+0000","last_active":"2024-01-27T00:47:20.524123+0000","last_peered":"2024-01-27T00:47:20.524123+0000","last_clean":"2024-01-27T00:47:20.524123+0000","last_became_active":"2024-01-27T00:47:20.510231+0000","last_became_peered":"2024-01-27T00:47:20.510231+0000","last_unstale":"2024-01-27T00:47:20.524123+0000","last_undegraded":"2024-01-27T00:47:20.524123+0000","last_fullsized":"2024-01-27T00:47:20.524123+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:15.349063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499847+0000","last_change":"2024-01-27T00:47:19.505767+0000","last_active":"2024-01-27T00:47:20.499847+0000","last_peered":"2024-01-27T00:47:20.499847+0000","last_clean":"2024-01-27T00:47:20.499847+0000","last_became_active":"2024-01-27T00:47:19.505313+0000","last_became_peered":"2024-01-27T00:47:19.505313+0000","last_unstale":"2024-01-27T00:47:20.499847+0000","last_undegraded":"2024-01-27T00:47:20.499847+0000","last_fullsized":"2024-01-27T00:47:20.499847+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:48:00.955590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.530928+0000","last_change":"2024-01-27T00:47:20.516018+0000","last_active":"2024-01-27T00:47:20.530928+0000","last_peered":"2024-01-27T00:47:20.530928+0000","last_clean":"2024-01-27T00:47:20.530928+0000","last_became_active":"2024-01-27T00:47:20.515703+0000","last_became_peered":"2024-01-27T00:47:20.515703+0000","last_unstale":"2024-01-27T00:47:20.530928+0000","last_undegraded":"2024-01-27T00:47:20.530928+0000","last_fullsized":"2024-01-27T00:47:20.530928+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:12:19.305838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499639+0000","last_change":"2024-01-27T00:47:19.499255+0000","last_active":"2024-01-27T00:47:20.499639+0000","last_peered":"2024-01-27T00:47:20.499639+0000","last_clean":"2024-01-27T00:47:20.499639+0000","last_became_active":"2024-01-27T00:47:19.498933+0000","last_became_peered":"2024-01-27T00:47:19.498933+0000","last_unstale":"2024-01-27T00:47:20.499639+0000","last_undegraded":"2024-01-27T00:47:20.499639+0000","last_fullsized":"2024-01-27T00:47:20.499639+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:45.555078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524255+0000","last_change":"2024-01-27T00:47:20.511077+0000","last_active":"2024-01-27T00:47:20.524255+0000","last_peered":"2024-01-27T00:47:20.524255+0000","last_clean":"2024-01-27T00:47:20.524255+0000","last_became_active":"2024-01-27T00:47:20.510732+0000","last_became_peered":"2024-01-27T00:47:20.510732+0000","last_unstale":"2024-01-27T00:47:20.524255+0000","last_undegraded":"2024-01-27T00:47:20.524255+0000","last_fullsized":"2024-01-27T00:47:20.524255+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:05.094009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.508098+0000","last_change":"2024-01-27T00:47:19.504494+0000","last_active":"2024-01-27T00:47:20.508098+0000","last_peered":"2024-01-27T00:47:20.508098+0000","last_clean":"2024-01-27T00:47:20.508098+0000","last_became_active":"2024-01-27T00:47:19.504090+0000","last_became_peered":"2024-01-27T00:47:19.504090+0000","last_unstale":"2024-01-27T00:47:20.508098+0000","last_undegraded":"2024-01-27T00:47:20.508098+0000","last_fullsized":"2024-01-27T00:47:20.508098+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:22:59.518350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.479789+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:07.269026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.498924+0000","last_change":"2024-01-27T00:47:19.503541+0000","last_active":"2024-01-27T00:47:20.498924+0000","last_peered":"2024-01-27T00:47:20.498924+0000","last_clean":"2024-01-27T00:47:20.498924+0000","last_became_active":"2024-01-27T00:47:19.503073+0000","last_became_peered":"2024-01-27T00:47:19.503073+0000","last_unstale":"2024-01-27T00:47:20.498924+0000","last_undegraded":"2024-01-27T00:47:20.498924+0000","last_fullsized":"2024-01-27T00:47:20.498924+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:14.344552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531950+0000","last_change":"2024-01-27T00:47:20.527327+0000","last_active":"2024-01-27T00:47:20.531950+0000","last_peered":"2024-01-27T00:47:20.531950+0000","last_clean":"2024-01-27T00:47:20.531950+0000","last_became_active":"2024-01-27T00:47:20.520248+0000","last_became_peered":"2024-01-27T00:47:20.520248+0000","last_unstale":"2024-01-27T00:47:20.531950+0000","last_undegraded":"2024-01-27T00:47:20.531950+0000","last_fullsized":"2024-01-27T00:47:20.531950+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:19:26.044662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.482920+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:20:46.023087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.532846+0000","last_change":"2024-01-27T00:47:20.516514+0000","last_active":"2024-01-27T00:47:20.532846+0000","last_peered":"2024-01-27T00:47:20.532846+0000","last_clean":"2024-01-27T00:47:20.532846+0000","last_became_active":"2024-01-27T00:47:20.516160+0000","last_became_peered":"2024-01-27T00:47:20.516160+0000","last_unstale":"2024-01-27T00:47:20.532846+0000","last_undegraded":"2024-01-27T00:47:20.532846+0000","last_fullsized":"2024-01-27T00:47:20.532846+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:36:41.113078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.497052+0000","last_change":"2024-01-27T00:47:19.493487+0000","last_active":"2024-01-27T00:47:20.497052+0000","last_peered":"2024-01-27T00:47:20.497052+0000","last_clean":"2024-01-27T00:47:20.497052+0000","last_became_active":"2024-01-27T00:47:19.492721+0000","last_became_peered":"2024-01-27T00:47:19.492721+0000","last_unstale":"2024-01-27T00:47:20.497052+0000","last_undegraded":"2024-01-27T00:47:20.497052+0000","last_fullsized":"2024-01-27T00:47:20.497052+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:30.730697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499725+0000","last_change":"2024-01-27T00:47:19.501570+0000","last_active":"2024-01-27T00:47:20.499725+0000","last_peered":"2024-01-27T00:47:20.499725+0000","last_clean":"2024-01-27T00:47:20.499725+0000","last_became_active":"2024-01-27T00:47:19.501187+0000","last_became_peered":"2024-01-27T00:47:19.501187+0000","last_unstale":"2024-01-27T00:47:20.499725+0000","last_undegraded":"2024-01-27T00:47:20.499725+0000","last_fullsized":"2024-01-27T00:47:20.499725+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:54:58.914033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"179'8695","reported_seq":9433,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-01-27T00:47:17.003489+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T00:47:17.003489+0000","last_peered":"2024-01-27T00:47:17.003489+0000","last_clean":"2024-01-27T00:47:17.003489+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T00:47:17.003489+0000","last_undegraded":"2024-01-27T00:47:17.003489+0000","last_fullsized":"2024-01-27T00:47:17.003489+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8695,"log_dups_size":0,"ondisk_log_size":8695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1784,"num_read_kb":3778,"num_write":530,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":1662,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1784,"num_read_kb":3778,"num_write":530,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8695,"ondisk_log_size":8695,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316504,"kb_used_data":2832,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055336,"statfs":{"total":96636764160,"available":96312664064,"internally_reserved":0,"allocated":2899968,"data_stored":1787469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60799999999999998,"15min":0.60699999999999998},"min":{"1min":0.36199999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75800000000000001,"5min":0.83199999999999996,"15min":0.86899999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61299999999999999,"15min":0.61799999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.755,"5min":0.82099999999999995,"15min":0.90000000000000002},"last":0.61299999999999999}]},{"osd":1,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.34599999999999997},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.90400000000000003},"last":0.627},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63500000000000001,"15min":0.63},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.38200000000000001},"max":{"1min":0.76700000000000002,"5min":0.85099999999999998,"15min":0.875},"last":0.54900000000000004}]},{"osd":2,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.374,"5min":0.36299999999999999,"15min":0.34000000000000002},"max":{"1min":0.76400000000000001,"5min":0.77700000000000002,"15min":0.89700000000000002},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63800000000000001,"15min":0.63700000000000001},"min":{"1min":0.49399999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75600000000000001,"5min":0.86199999999999999,"15min":0.94199999999999995},"last":0.57499999999999996}]},{"osd":3,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.626,"15min":0.623},"min":{"1min":0.435,"5min":0.40999999999999998,"15min":0.39400000000000002},"max":{"1min":0.79200000000000004,"5min":0.86499999999999999,"15min":0.92600000000000005},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.77700000000000002,"5min":0.83499999999999996,"15min":0.88400000000000001},"last":0.67100000000000004}]},{"osd":4,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49399999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.32900000000000001,"15min":0.29699999999999999},"max":{"1min":0.61699999999999999,"5min":0.86499999999999999,"15min":0.92000000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.34599999999999997},"max":{"1min":0.76600000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.42499999999999999}]},{"osd":5,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":0.67100000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":0.67700000000000005,"5min":0.89100000000000001,"15min":0.95699999999999996},"last":0.45800000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59299999999999997,"15min":0.58299999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.36599999999999999},"max":{"1min":0.69299999999999995,"5min":0.91700000000000004,"15min":0.99399999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.36699999999999999},"max":{"1min":0.73599999999999999,"5min":0.82299999999999995,"15min":0.878},"last":0.49199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575775,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307732,"kb_used_data":2252,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064108,"statfs":{"total":96636764160,"available":96321646592,"internally_reserved":0,"allocated":2306048,"data_stored":1197101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58999999999999997,"15min":0.60399999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81000000000000005,"5min":0.82199999999999995,"15min":0.84899999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.38900000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.78500000000000003,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.55700000000000005}]},{"osd":1,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.748,"5min":0.79800000000000004,"15min":0.80800000000000005},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81499999999999995,"5min":0.87,"15min":0.87},"last":0.49299999999999999}]},{"osd":2,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60199999999999998,"15min":0.60899999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83699999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59899999999999998,"15min":0.60599999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74199999999999999,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.50600000000000001}]},{"osd":3,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86299999999999999,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64600000000000002,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.63400000000000001}]},{"osd":4,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.496,"15min":0.49099999999999999},"min":{"1min":0.34300000000000003,"5min":0.33200000000000002,"15min":0.313},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32100000000000001},"max":{"1min":0.77700000000000002,"5min":0.79100000000000004,"15min":0.83199999999999996},"last":0.39300000000000002}]},{"osd":5,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51900000000000002,"15min":0.51000000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.32200000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.375},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56499999999999995,"15min":0.56000000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.88800000000000001},"last":0.40999999999999998}]},{"osd":7,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57399999999999995,"15min":0.56499999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.58399999999999996,"15min":0.58099999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.33600000000000002},"max":{"1min":0.84099999999999997,"5min":0.84599999999999997,"15min":0.85999999999999999},"last":0.434}]}]},{"osd":1,"up_from":13,"seq":55834575776,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311888,"kb_used_data":2312,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059952,"statfs":{"total":96636764160,"available":96317390848,"internally_reserved":0,"allocated":2367488,"data_stored":1226772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.41799999999999998},"min":{"1min":0.311,"5min":0.254,"15min":0.23499999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.434,"5min":0.441,"15min":0.44500000000000001},"min":{"1min":0.30199999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.60599999999999998,"5min":0.66200000000000003,"15min":0.70199999999999996},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.41899999999999998,"15min":0.42499999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55100000000000005,"5min":0.59999999999999998,"15min":0.68899999999999995},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48499999999999999,"15min":0.49399999999999999},"min":{"1min":0.33500000000000002,"5min":0.31,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.76700000000000002},"last":0.48499999999999999}]},{"osd":3,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.503,"15min":0.52100000000000002},"min":{"1min":0.39400000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.63900000000000001,"5min":0.69399999999999995,"15min":1.0469999999999999},"last":0.621},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.49099999999999999,"15min":0.5},"min":{"1min":0.318,"5min":0.318,"15min":0.309},"max":{"1min":0.69599999999999995,"5min":0.72599999999999998,"15min":0.95599999999999996},"last":0.56899999999999995}]},{"osd":4,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.66100000000000003,"15min":0.67900000000000005},"min":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86299999999999999,"5min":1.02,"15min":1.3129999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67700000000000005,"15min":0.69999999999999996},"min":{"1min":0.505,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.93500000000000005,"5min":1.0069999999999999,"15min":1.097},"last":0.68500000000000005}]},{"osd":5,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66700000000000004,"15min":0.69199999999999995},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.46300000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":1.091},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.36299999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.278},"last":0.64600000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.68500000000000005,"15min":0.71099999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.435},"max":{"1min":0.88100000000000001,"5min":0.96999999999999997,"15min":1.141},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70799999999999996,"15min":0.72099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.91400000000000003,"5min":1.0229999999999999,"15min":1.1120000000000001},"last":0.77900000000000003}]},{"osd":7,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.73299999999999998,"15min":0.749},"min":{"1min":0.52600000000000002,"5min":0.52500000000000002,"15min":0.51100000000000001},"max":{"1min":1.0129999999999999,"5min":1.054,"15min":1.1819999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.73499999999999999,"15min":0.75800000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.51200000000000001},"max":{"1min":0.98299999999999998,"5min":0.98999999999999999,"15min":1.198},"last":0.71199999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575775,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312464,"kb_used_data":2888,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059376,"statfs":{"total":96636764160,"available":96316801024,"internally_reserved":0,"allocated":2957312,"data_stored":1815708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.28299999999999997,"5min":0.25800000000000001,"15min":0.25},"max":{"1min":0.55500000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.34499999999999997,"5min":0.27500000000000002,"15min":0.26300000000000001},"max":{"1min":0.61299999999999999,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.56999999999999995}]},{"osd":2,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.42099999999999999,"15min":0.42599999999999999},"min":{"1min":0.312,"5min":0.28599999999999998,"15min":0.26700000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.69399999999999995},"last":0.621},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48099999999999998,"15min":0.48299999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.629}]},{"osd":3,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48999999999999999,"15min":0.48699999999999999},"min":{"1min":0.32900000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.65600000000000003,"5min":0.755,"15min":0.755},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.496,"15min":0.498},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.42199999999999999}]},{"osd":4,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65200000000000002,"15min":0.68100000000000005},"min":{"1min":0.45100000000000001,"5min":0.44700000000000001,"15min":0.44400000000000001},"max":{"1min":0.77500000000000002,"5min":0.97099999999999997,"15min":1.125},"last":0.496},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.66000000000000003,"15min":0.67800000000000005},"min":{"1min":0.46200000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.96199999999999997,"5min":0.97599999999999998,"15min":1.0169999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68100000000000005,"15min":0.69399999999999995},"min":{"1min":0.47899999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.84099999999999997,"5min":0.94199999999999995,"15min":1.054},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.70799999999999996,"15min":0.71899999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.98199999999999998,"5min":1.056,"15min":1.056},"last":0.46200000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69999999999999996,"15min":0.71199999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.84399999999999997,"5min":1.0269999999999999,"15min":1.087},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.72299999999999998,"15min":0.73899999999999999},"min":{"1min":0.52500000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.94599999999999995,"5min":1.0349999999999999,"15min":1.052},"last":0.88400000000000001}]},{"osd":7,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.747,"15min":0.75700000000000001},"min":{"1min":0.60599999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.016,"5min":1.0880000000000001,"15min":1.1080000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.53700000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.034,"5min":1.071,"15min":1.071},"last":0.55400000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575775,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311876,"kb_used_data":2300,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059964,"statfs":{"total":96636764160,"available":96317403136,"internally_reserved":0,"allocated":2355200,"data_stored":1225236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38100000000000001,"15min":0.39800000000000002},"min":{"1min":0.29499999999999998,"5min":0.25900000000000001,"15min":0.252},"max":{"1min":0.48699999999999999,"5min":0.60499999999999998,"15min":0.65800000000000003},"last":0.375},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":0.438},"min":{"1min":0.29499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.71899999999999997},"last":0.35599999999999998}]},{"osd":1,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.41999999999999998,"15min":0.432},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.60799999999999998,"5min":0.627,"15min":0.68300000000000005},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.5,"15min":0.51400000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66100000000000003,"5min":0.76400000000000001,"15min":0.80300000000000005},"last":0.40400000000000003}]},{"osd":3,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47799999999999998,"15min":0.49199999999999999},"min":{"1min":0.34999999999999998,"5min":0.34100000000000003,"15min":0.33100000000000002},"max":{"1min":0.65200000000000002,"5min":0.745,"15min":0.745},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47099999999999997,"15min":0.48699999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.29499999999999998},"max":{"1min":0.68000000000000005,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.44900000000000001}]},{"osd":4,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.65000000000000002,"15min":0.66700000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41999999999999998},"max":{"1min":0.77000000000000002,"5min":0.85899999999999999,"15min":1.0660000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66700000000000004,"15min":0.68200000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92400000000000004,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.83999999999999997}]},{"osd":5,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66800000000000004,"15min":0.68999999999999995},"min":{"1min":0.49099999999999999,"5min":0.46500000000000002,"15min":0.42899999999999999},"max":{"1min":0.78400000000000003,"5min":0.96499999999999997,"15min":1.0649999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68200000000000005,"15min":0.69799999999999995},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.90600000000000003,"5min":0.98799999999999999,"15min":1.044},"last":0.77600000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.68100000000000005,"15min":0.70099999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.86299999999999999,"5min":1.0569999999999999,"15min":1.1140000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.69399999999999995,"15min":0.70999999999999996},"min":{"1min":0.47999999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89400000000000002,"5min":1.002,"15min":1.0720000000000001},"last":0.79000000000000004}]},{"osd":7,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.71499999999999997,"15min":0.72799999999999998},"min":{"1min":0.55600000000000005,"5min":0.50900000000000001,"15min":0.48999999999999999},"max":{"1min":0.97499999999999998,"5min":1.0860000000000001,"15min":1.1479999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.73899999999999999,"15min":0.76200000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.98899999999999999,"5min":1.022,"15min":1.1299999999999999},"last":0.82799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575774,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315992,"kb_used_data":2320,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055848,"statfs":{"total":96636764160,"available":96313188352,"internally_reserved":0,"allocated":2375680,"data_stored":1227348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.39600000000000002,"15min":0.40100000000000002},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.50600000000000001,"5min":0.63,"15min":0.752},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.443,"5min":0.45700000000000002,"15min":0.45300000000000001},"min":{"1min":0.32900000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.57899999999999996,"5min":0.68799999999999994,"15min":0.85699999999999998},"last":0.69599999999999995}]},{"osd":1,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.46600000000000003},"min":{"1min":0.307,"5min":0.29699999999999999,"15min":0.27400000000000002},"max":{"1min":0.61499999999999999,"5min":0.69699999999999995,"15min":0.94599999999999995},"last":0.627},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.498,"15min":0.498},"min":{"1min":0.36299999999999999,"5min":0.34399999999999997,"15min":0.316},"max":{"1min":0.69299999999999995,"5min":0.745,"15min":0.745},"last":0.72999999999999998}]},{"osd":2,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.35699999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64900000000000002,"5min":0.67900000000000005,"15min":0.84499999999999997},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.46000000000000002,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.66600000000000004,"5min":0.72699999999999998,"15min":0.82299999999999995},"last":0.80400000000000005}]},{"osd":4,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.69299999999999995,"15min":0.68999999999999995},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.88400000000000001,"5min":0.89300000000000002,"15min":1},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.48199999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.98699999999999999,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.89700000000000002}]},{"osd":5,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.69299999999999995,"15min":0.68799999999999994},"min":{"1min":0.505,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.88900000000000001,"5min":0.94499999999999995,"15min":1.0429999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68400000000000005,"15min":0.68700000000000006},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90400000000000003,"5min":0.99099999999999999,"15min":1.0289999999999999},"last":1}]},{"osd":6,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.68300000000000005,"15min":0.69099999999999995},"min":{"1min":0.53000000000000003,"5min":0.52000000000000002,"15min":0.48899999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.0820000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.71799999999999997,"15min":0.71599999999999997},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.44},"max":{"1min":0.91600000000000004,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.95199999999999996}]},{"osd":7,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.69999999999999996,"15min":0.71399999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.96399999999999997,"5min":1.081,"15min":1.081},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.55800000000000005,"5min":0.48999999999999999,"15min":0.46300000000000002},"max":{"1min":0.91700000000000004,"5min":1.0229999999999999,"15min":1.1240000000000001},"last":0.98999999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575775,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311828,"kb_used_data":2252,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060012,"statfs":{"total":96636764160,"available":96317452288,"internally_reserved":0,"allocated":2306048,"data_stored":1197101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61399999999999999,"15min":0.60599999999999998},"min":{"1min":0.40699999999999997,"5min":0.39300000000000002,"15min":0.32100000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59299999999999997}]},{"osd":1,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.627,"15min":0.628},"min":{"1min":0.52900000000000003,"5min":0.42499999999999999,"15min":0.40999999999999998},"max":{"1min":0.75700000000000001,"5min":0.83199999999999996,"15min":0.86299999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61899999999999999,"15min":0.626},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84699999999999998},"last":0.56399999999999995}]},{"osd":2,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.46100000000000002,"5min":0.436,"15min":0.40899999999999997},"max":{"1min":0.753,"5min":0.81899999999999995,"15min":0.84499999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.5,"5min":0.44400000000000001,"15min":0.38100000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.41799999999999998}]},{"osd":3,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.626,"15min":0.627},"min":{"1min":0.45900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.61899999999999999,"15min":0.623},"min":{"1min":0.42599999999999999,"5min":0.41199999999999998,"15min":0.41099999999999998},"max":{"1min":0.80600000000000005,"5min":0.80900000000000005,"15min":0.876},"last":0.76800000000000002}]},{"osd":5,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48199999999999998,"15min":0.48499999999999999},"min":{"1min":0.35899999999999999,"5min":0.32100000000000001,"15min":0.28699999999999998},"max":{"1min":0.69699999999999995,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.33100000000000002,"5min":0.30499999999999999,"15min":0.30199999999999999},"max":{"1min":0.77300000000000002,"5min":0.876,"15min":0.876},"last":0.65800000000000003}]},{"osd":6,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49299999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.54100000000000004},"min":{"1min":0.39800000000000002,"5min":0.34300000000000003,"15min":0.32800000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.67500000000000004}]},{"osd":7,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41599999999999998,"5min":0.34999999999999998,"15min":0.33700000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.625},{"interface":"front","average":{"1min":0.627,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.46000000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.80400000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575775,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311828,"kb_used_data":2252,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060012,"statfs":{"total":96636764160,"available":96317452288,"internally_reserved":0,"allocated":2306048,"data_stored":1197101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61399999999999999,"15min":0.60799999999999998},"min":{"1min":0.379,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76200000000000001,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.61599999999999999,"15min":0.623},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38600000000000001},"max":{"1min":0.76600000000000001,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.46800000000000003}]},{"osd":1,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65900000000000003,"15min":0.64900000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66400000000000003,"15min":0.67100000000000004},"min":{"1min":0.44600000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80300000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.60499999999999998}]},{"osd":2,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.61399999999999999,"15min":0.60299999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.71999999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.79100000000000004,"5min":0.876,"15min":0.876},"last":0.41399999999999998}]},{"osd":3,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60399999999999998,"15min":0.60299999999999998},"min":{"1min":0.38200000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.70099999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.64200000000000002,"15min":0.63400000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.79600000000000004,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.64900000000000002}]},{"osd":4,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.47499999999999998,"15min":0.47599999999999998},"min":{"1min":0.307,"5min":0.30399999999999999,"15min":0.30299999999999999},"max":{"1min":0.67600000000000005,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.82799999999999996},"last":0.71699999999999997}]},{"osd":6,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50700000000000001,"15min":0.504},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.31900000000000001},"max":{"1min":0.67400000000000004,"5min":0.71399999999999997,"15min":0.79900000000000004},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.379,"5min":0.36799999999999999,"15min":0.36099999999999999},"max":{"1min":0.70099999999999996,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.63500000000000001}]},{"osd":7,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34699999999999998},"max":{"1min":0.747,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.45800000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.69999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T00:47:24.701 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T00:47:24.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.869+0000 7fae18632700 1 Processor -- start 2024-01-27T00:47:24.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.870+0000 7fae18632700 1 -- start start 2024-01-27T00:47:24.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.871+0000 7fae18632700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae100c6520 0x7fae100ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:24.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.871+0000 7fae18632700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fae100ced50 0x7fae100cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:24.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.871+0000 7fae18632700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1015a0b0 con 0x7fae100b2b40 2024-01-27T00:47:24.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.871+0000 7fae18632700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae100af070 con 0x7fae100ced50 2024-01-27T00:47:24.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.871+0000 7fae18632700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae10099fc0 con 0x7fae100c6520 2024-01-27T00:47:24.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.871+0000 7fae16bcf700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fae100ced50 0x7fae100cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:24.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.871+0000 7fae163ce700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fae100b2b40 0x7fae100ae960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:47758/0 (socket says 172.21.15.46:47758) 2024-01-27T00:47:24.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.871+0000 7fae163ce700 1 -- 172.21.15.46:0/3503931165 learned_addr learned my addr 172.21.15.46:0/3503931165 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:24.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.871+0000 7fae15bcd700 1 --2- 172.21.15.46:0/3503931165 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae100c6520 0x7fae100ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.871+0000 7fae16bcf700 1 -- 172.21.15.46:0/3503931165 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae100c6520 msgr2=0x7fae100ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.871+0000 7fae16bcf700 1 --2- 172.21.15.46:0/3503931165 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae100c6520 0x7fae100ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.871+0000 7fae16bcf700 1 -- 172.21.15.46:0/3503931165 >> v1:172.21.15.106:6789/0 conn(0x7fae100b2b40 legacy=0x7fae100ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.871+0000 7fae16bcf700 1 -- 172.21.15.46:0/3503931165 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae100ae4d0 con 0x7fae100ced50 2024-01-27T00:47:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.871+0000 7fae15bcd700 1 --2- 172.21.15.46:0/3503931165 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae100c6520 0x7fae100ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.871+0000 7fae153cc700 1 -- 172.21.15.46:0/3503931165 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381998485 0 0) 0x7fae1015a0b0 con 0x7fae100b2b40 2024-01-27T00:47:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.872+0000 7fae16bcf700 1 --2- 172.21.15.46:0/3503931165 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae100ced50 0x7fae100cf1d0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fae08009e60 tx=0x7fae08009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e862926d9d93803 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.872+0000 7fae153cc700 1 -- 172.21.15.46:0/3503931165 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0800e070 con 0x7fae100ced50 2024-01-27T00:47:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.872+0000 7fae153cc700 1 -- 172.21.15.46:0/3503931165 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0800bb60 con 0x7fae100ced50 2024-01-27T00:47:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.872+0000 7fae153cc700 1 -- 172.21.15.46:0/3503931165 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae08012430 con 0x7fae100ced50 2024-01-27T00:47:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.872+0000 7fae18632700 1 -- 172.21.15.46:0/3503931165 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae100ced50 msgr2=0x7fae100cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.872+0000 7fae18632700 1 --2- 172.21.15.46:0/3503931165 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae100ced50 0x7fae100cf1d0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fae08009e60 tx=0x7fae08009780 comp rx=0 tx=0).stop 2024-01-27T00:47:24.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.872+0000 7fae18632700 1 -- 172.21.15.46:0/3503931165 shutdown_connections 2024-01-27T00:47:24.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.872+0000 7fae18632700 1 --2- 172.21.15.46:0/3503931165 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae100ced50 0x7fae100cf1d0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.872+0000 7fae18632700 1 --2- 172.21.15.46:0/3503931165 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae100c6520 0x7fae100ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.872+0000 7fae18632700 1 -- 172.21.15.46:0/3503931165 >> 172.21.15.46:0/3503931165 conn(0x7fae10099a30 msgr2=0x7fae1006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:24.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.872+0000 7fae18632700 1 -- 172.21.15.46:0/3503931165 shutdown_connections 2024-01-27T00:47:24.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.872+0000 7fae18632700 1 -- 172.21.15.46:0/3503931165 wait complete. 2024-01-27T00:47:24.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.874+0000 7fae18632700 1 Processor -- start 2024-01-27T00:47:24.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.874+0000 7fae18632700 1 -- start start 2024-01-27T00:47:24.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.874+0000 7fae18632700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae100b2b40 0x7fae101517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.874+0000 7fae18632700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae100c6520 0x7fae100b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.874+0000 7fae18632700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae100ced50 0x7fae100b55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.874+0000 7fae18632700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae10154cf0 con 0x7fae100ced50 2024-01-27T00:47:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.874+0000 7fae18632700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae10151d10 con 0x7fae100c6520 2024-01-27T00:47:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.874+0000 7fae18632700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae100c7820 con 0x7fae100b2b40 2024-01-27T00:47:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.875+0000 7fae16bcf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae100ced50 0x7fae100b55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.875+0000 7fae16bcf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae100ced50 0x7fae100b55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40590/0 (socket says 172.21.15.46:40590) 2024-01-27T00:47:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.875+0000 7fae16bcf700 1 -- 172.21.15.46:0/719789037 learned_addr learned my addr 172.21.15.46:0/719789037 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.875+0000 7fae15bcd700 1 --2- 172.21.15.46:0/719789037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae100c6520 0x7fae100b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.875+0000 7fae163ce700 1 --2- 172.21.15.46:0/719789037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae100b2b40 0x7fae101517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.875+0000 7fae16bcf700 1 -- 172.21.15.46:0/719789037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae100b2b40 msgr2=0x7fae101517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.875+0000 7fae16bcf700 1 --2- 172.21.15.46:0/719789037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae100b2b40 0x7fae101517d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.875+0000 7fae16bcf700 1 -- 172.21.15.46:0/719789037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae100c6520 msgr2=0x7fae100b5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.875+0000 7fae16bcf700 1 --2- 172.21.15.46:0/719789037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae100c6520 0x7fae100b5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.875+0000 7fae16bcf700 1 -- 172.21.15.46:0/719789037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae00009420 con 0x7fae100ced50 2024-01-27T00:47:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.875+0000 7fae16bcf700 1 --2- 172.21.15.46:0/719789037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae100ced50 0x7fae100b55d0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fae0800b6c0 tx=0x7fae08019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.875+0000 7fae067fc700 1 -- 172.21.15.46:0/719789037 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae08012e90 con 0x7fae100ced50 2024-01-27T00:47:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.875+0000 7fae067fc700 1 -- 172.21.15.46:0/719789037 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae08009420 con 0x7fae100ced50 2024-01-27T00:47:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.875+0000 7fae18632700 1 -- 172.21.15.46:0/719789037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae08009a20 con 0x7fae100ced50 2024-01-27T00:47:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.875+0000 7fae067fc700 1 -- 172.21.15.46:0/719789037 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae080276c0 con 0x7fae100ced50 2024-01-27T00:47:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.875+0000 7fae18632700 1 -- 172.21.15.46:0/719789037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae100c83a0 con 0x7fae100ced50 2024-01-27T00:47:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.877+0000 7fae18632700 1 -- 172.21.15.46:0/719789037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae100c8530 con 0x7fae100ced50 2024-01-27T00:47:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.877+0000 7fae067fc700 1 -- 172.21.15.46:0/719789037 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fae08024070 con 0x7fae100ced50 2024-01-27T00:47:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.877+0000 7fae067fc700 1 --2- 172.21.15.46:0/719789037 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fadf4066f30 0x7fadf40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.877+0000 7fae067fc700 1 -- 172.21.15.46:0/719789037 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fae08086ee0 con 0x7fae100ced50 2024-01-27T00:47:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.878+0000 7fae163ce700 1 --2- 172.21.15.46:0/719789037 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fadf4066f30 0x7fadf40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:24.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.878+0000 7fae163ce700 1 --2- 172.21.15.46:0/719789037 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fadf4066f30 0x7fadf40693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fae1009e490 tx=0x7fae0c009040 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:24.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:24.882+0000 7fae067fc700 1 -- 172.21.15.46:0/719789037 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fae08053dc0 con 0x7fae100ced50 2024-01-27T00:47:25.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.082+0000 7fae18632700 1 -- 172.21.15.46:0/719789037 --> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fae1004c710 con 0x7fadf4066f30 2024-01-27T00:47:25.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.090+0000 7fae067fc700 1 -- 172.21.15.46:0/719789037 <== mgr.9313 v2:172.21.15.106:6836/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284817 (secure 0 0 0) 0x7fae1004c710 con 0x7fadf4066f30 2024-01-27T00:47:25.094 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.093+0000 7fae18632700 1 -- 172.21.15.46:0/719789037 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fadf4066f30 msgr2=0x7fadf40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.093+0000 7fae18632700 1 --2- 172.21.15.46:0/719789037 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fadf4066f30 0x7fadf40693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fae1009e490 tx=0x7fae0c009040 comp rx=0 tx=0).stop 2024-01-27T00:47:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.093+0000 7fae18632700 1 -- 172.21.15.46:0/719789037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae100ced50 msgr2=0x7fae100b55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.093+0000 7fae18632700 1 --2- 172.21.15.46:0/719789037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae100ced50 0x7fae100b55d0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fae0800b6c0 tx=0x7fae08019000 comp rx=0 tx=0).stop 2024-01-27T00:47:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.094+0000 7fae18632700 1 -- 172.21.15.46:0/719789037 shutdown_connections 2024-01-27T00:47:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.094+0000 7fae18632700 1 --2- 172.21.15.46:0/719789037 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fadf4066f30 0x7fadf40693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.094+0000 7fae18632700 1 --2- 172.21.15.46:0/719789037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae100ced50 0x7fae100b55d0 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.094+0000 7fae18632700 1 --2- 172.21.15.46:0/719789037 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fae100b2b40 0x7fae101517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.094+0000 7fae18632700 1 --2- 172.21.15.46:0/719789037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae100c6520 0x7fae100b5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.094+0000 7fae18632700 1 -- 172.21.15.46:0/719789037 >> 172.21.15.46:0/719789037 conn(0x7fae10099a30 msgr2=0x7fae10068ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:25.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.094+0000 7fae18632700 1 -- 172.21.15.46:0/719789037 shutdown_connections 2024-01-27T00:47:25.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.094+0000 7fae18632700 1 -- 172.21.15.46:0/719789037 wait complete. 2024-01-27T00:47:25.097 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T00:47:25.107 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-27T00:47:23.570969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592030,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1784,"num_read_kb":3778,"num_write":563,"num_write_kb":3473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8724,"ondisk_log_size":8724,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2500112,"kb_used_data":19408,"kb_used_omap":0,"kb_used_meta":2480640,"kb_avail":752474608,"statfs":{"total":773094113280,"available":770533998592,"internally_reserved":0,"allocated":19873792,"data_stored":10873836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2540175360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1662,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.095847"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.523334+0000","last_change":"2024-01-27T00:47:20.509897+0000","last_active":"2024-01-27T00:47:20.523334+0000","last_peered":"2024-01-27T00:47:20.523334+0000","last_clean":"2024-01-27T00:47:20.523334+0000","last_became_active":"2024-01-27T00:47:20.509610+0000","last_became_peered":"2024-01-27T00:47:20.509610+0000","last_unstale":"2024-01-27T00:47:20.523334+0000","last_undegraded":"2024-01-27T00:47:20.523334+0000","last_fullsized":"2024-01-27T00:47:20.523334+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:42:01.971654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527541+0000","last_change":"2024-01-27T00:47:20.514165+0000","last_active":"2024-01-27T00:47:20.527541+0000","last_peered":"2024-01-27T00:47:20.527541+0000","last_clean":"2024-01-27T00:47:20.527541+0000","last_became_active":"2024-01-27T00:47:20.513848+0000","last_became_peered":"2024-01-27T00:47:20.513848+0000","last_unstale":"2024-01-27T00:47:20.527541+0000","last_undegraded":"2024-01-27T00:47:20.527541+0000","last_fullsized":"2024-01-27T00:47:20.527541+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:43:04.739251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524282+0000","last_change":"2024-01-27T00:47:20.511666+0000","last_active":"2024-01-27T00:47:20.524282+0000","last_peered":"2024-01-27T00:47:20.524282+0000","last_clean":"2024-01-27T00:47:20.524282+0000","last_became_active":"2024-01-27T00:47:20.509826+0000","last_became_peered":"2024-01-27T00:47:20.509826+0000","last_unstale":"2024-01-27T00:47:20.524282+0000","last_undegraded":"2024-01-27T00:47:20.524282+0000","last_fullsized":"2024-01-27T00:47:20.524282+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:53:47.125547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524159+0000","last_change":"2024-01-27T00:47:20.511731+0000","last_active":"2024-01-27T00:47:20.524159+0000","last_peered":"2024-01-27T00:47:20.524159+0000","last_clean":"2024-01-27T00:47:20.524159+0000","last_became_active":"2024-01-27T00:47:20.511436+0000","last_became_peered":"2024-01-27T00:47:20.511436+0000","last_unstale":"2024-01-27T00:47:20.524159+0000","last_undegraded":"2024-01-27T00:47:20.524159+0000","last_fullsized":"2024-01-27T00:47:20.524159+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:40.342669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524015+0000","last_change":"2024-01-27T00:47:20.510790+0000","last_active":"2024-01-27T00:47:20.524015+0000","last_peered":"2024-01-27T00:47:20.524015+0000","last_clean":"2024-01-27T00:47:20.524015+0000","last_became_active":"2024-01-27T00:47:20.510447+0000","last_became_peered":"2024-01-27T00:47:20.510447+0000","last_unstale":"2024-01-27T00:47:20.524015+0000","last_undegraded":"2024-01-27T00:47:20.524015+0000","last_fullsized":"2024-01-27T00:47:20.524015+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:56:44.894090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531204+0000","last_change":"2024-01-27T00:47:20.520667+0000","last_active":"2024-01-27T00:47:20.531204+0000","last_peered":"2024-01-27T00:47:20.531204+0000","last_clean":"2024-01-27T00:47:20.531204+0000","last_became_active":"2024-01-27T00:47:20.519891+0000","last_became_peered":"2024-01-27T00:47:20.519891+0000","last_unstale":"2024-01-27T00:47:20.531204+0000","last_undegraded":"2024-01-27T00:47:20.531204+0000","last_fullsized":"2024-01-27T00:47:20.531204+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:49:22.885968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522694+0000","last_change":"2024-01-27T00:47:20.513550+0000","last_active":"2024-01-27T00:47:20.522694+0000","last_peered":"2024-01-27T00:47:20.522694+0000","last_clean":"2024-01-27T00:47:20.522694+0000","last_became_active":"2024-01-27T00:47:20.512667+0000","last_became_peered":"2024-01-27T00:47:20.512667+0000","last_unstale":"2024-01-27T00:47:20.522694+0000","last_undegraded":"2024-01-27T00:47:20.522694+0000","last_fullsized":"2024-01-27T00:47:20.522694+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:09:43.978584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.504493+0000","last_change":"2024-01-27T00:47:20.500113+0000","last_active":"2024-01-27T00:47:20.504493+0000","last_peered":"2024-01-27T00:47:20.504493+0000","last_clean":"2024-01-27T00:47:20.504493+0000","last_became_active":"2024-01-27T00:47:20.499814+0000","last_became_peered":"2024-01-27T00:47:20.499814+0000","last_unstale":"2024-01-27T00:47:20.504493+0000","last_undegraded":"2024-01-27T00:47:20.504493+0000","last_fullsized":"2024-01-27T00:47:20.504493+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:05:38.788368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.532911+0000","last_change":"2024-01-27T00:47:20.518490+0000","last_active":"2024-01-27T00:47:20.532911+0000","last_peered":"2024-01-27T00:47:20.532911+0000","last_clean":"2024-01-27T00:47:20.532911+0000","last_became_active":"2024-01-27T00:47:20.518138+0000","last_became_peered":"2024-01-27T00:47:20.518138+0000","last_unstale":"2024-01-27T00:47:20.532911+0000","last_undegraded":"2024-01-27T00:47:20.532911+0000","last_fullsized":"2024-01-27T00:47:20.532911+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:28.557943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531230+0000","last_change":"2024-01-27T00:47:20.520931+0000","last_active":"2024-01-27T00:47:20.531230+0000","last_peered":"2024-01-27T00:47:20.531230+0000","last_clean":"2024-01-27T00:47:20.531230+0000","last_became_active":"2024-01-27T00:47:20.520322+0000","last_became_peered":"2024-01-27T00:47:20.520322+0000","last_unstale":"2024-01-27T00:47:20.531230+0000","last_undegraded":"2024-01-27T00:47:20.531230+0000","last_fullsized":"2024-01-27T00:47:20.531230+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:04:43.917528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.530444+0000","last_change":"2024-01-27T00:47:20.522803+0000","last_active":"2024-01-27T00:47:20.530444+0000","last_peered":"2024-01-27T00:47:20.530444+0000","last_clean":"2024-01-27T00:47:20.530444+0000","last_became_active":"2024-01-27T00:47:20.522463+0000","last_became_peered":"2024-01-27T00:47:20.522463+0000","last_unstale":"2024-01-27T00:47:20.530444+0000","last_undegraded":"2024-01-27T00:47:20.530444+0000","last_fullsized":"2024-01-27T00:47:20.530444+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:04:18.616847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.504894+0000","last_change":"2024-01-27T00:47:20.503158+0000","last_active":"2024-01-27T00:47:20.504894+0000","last_peered":"2024-01-27T00:47:20.504894+0000","last_clean":"2024-01-27T00:47:20.504894+0000","last_became_active":"2024-01-27T00:47:20.502875+0000","last_became_peered":"2024-01-27T00:47:20.502875+0000","last_unstale":"2024-01-27T00:47:20.504894+0000","last_undegraded":"2024-01-27T00:47:20.504894+0000","last_fullsized":"2024-01-27T00:47:20.504894+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:33:13.706183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.513540+0000","last_change":"2024-01-27T00:47:20.509928+0000","last_active":"2024-01-27T00:47:20.513540+0000","last_peered":"2024-01-27T00:47:20.513540+0000","last_clean":"2024-01-27T00:47:20.513540+0000","last_became_active":"2024-01-27T00:47:20.509630+0000","last_became_peered":"2024-01-27T00:47:20.509630+0000","last_unstale":"2024-01-27T00:47:20.513540+0000","last_undegraded":"2024-01-27T00:47:20.513540+0000","last_fullsized":"2024-01-27T00:47:20.513540+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:02:52.058658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531770+0000","last_change":"2024-01-27T00:47:20.527150+0000","last_active":"2024-01-27T00:47:20.531770+0000","last_peered":"2024-01-27T00:47:20.531770+0000","last_clean":"2024-01-27T00:47:20.531770+0000","last_became_active":"2024-01-27T00:47:20.523827+0000","last_became_peered":"2024-01-27T00:47:20.523827+0000","last_unstale":"2024-01-27T00:47:20.531770+0000","last_undegraded":"2024-01-27T00:47:20.531770+0000","last_fullsized":"2024-01-27T00:47:20.531770+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:52:04.785901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.480155+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:27:44.775426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.510752+0000","last_change":"2024-01-27T00:47:20.509849+0000","last_active":"2024-01-27T00:47:20.510752+0000","last_peered":"2024-01-27T00:47:20.510752+0000","last_clean":"2024-01-27T00:47:20.510752+0000","last_became_active":"2024-01-27T00:47:20.509553+0000","last_became_peered":"2024-01-27T00:47:20.509553+0000","last_unstale":"2024-01-27T00:47:20.510752+0000","last_undegraded":"2024-01-27T00:47:20.510752+0000","last_fullsized":"2024-01-27T00:47:20.510752+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:44:26.489354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.528860+0000","last_change":"2024-01-27T00:47:19.498621+0000","last_active":"2024-01-27T00:47:20.528860+0000","last_peered":"2024-01-27T00:47:20.528860+0000","last_clean":"2024-01-27T00:47:20.528860+0000","last_became_active":"2024-01-27T00:47:19.498357+0000","last_became_peered":"2024-01-27T00:47:19.498357+0000","last_unstale":"2024-01-27T00:47:20.528860+0000","last_undegraded":"2024-01-27T00:47:20.528860+0000","last_fullsized":"2024-01-27T00:47:20.528860+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:20:56.150512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522707+0000","last_change":"2024-01-27T00:47:20.513840+0000","last_active":"2024-01-27T00:47:20.522707+0000","last_peered":"2024-01-27T00:47:20.522707+0000","last_clean":"2024-01-27T00:47:20.522707+0000","last_became_active":"2024-01-27T00:47:20.512745+0000","last_became_peered":"2024-01-27T00:47:20.512745+0000","last_unstale":"2024-01-27T00:47:20.522707+0000","last_undegraded":"2024-01-27T00:47:20.522707+0000","last_fullsized":"2024-01-27T00:47:20.522707+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:11:59.061116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.482562+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:43:05.614968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1c","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.533308+0000","last_change":"2024-01-27T00:47:19.505152+0000","last_active":"2024-01-27T00:47:20.533308+0000","last_peered":"2024-01-27T00:47:20.533308+0000","last_clean":"2024-01-27T00:47:20.533308+0000","last_became_active":"2024-01-27T00:47:19.504128+0000","last_became_peered":"2024-01-27T00:47:19.504128+0000","last_unstale":"2024-01-27T00:47:20.533308+0000","last_undegraded":"2024-01-27T00:47:20.533308+0000","last_fullsized":"2024-01-27T00:47:20.533308+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:52.404716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524071+0000","last_change":"2024-01-27T00:47:20.510375+0000","last_active":"2024-01-27T00:47:20.524071+0000","last_peered":"2024-01-27T00:47:20.524071+0000","last_clean":"2024-01-27T00:47:20.524071+0000","last_became_active":"2024-01-27T00:47:20.510069+0000","last_became_peered":"2024-01-27T00:47:20.510069+0000","last_unstale":"2024-01-27T00:47:20.524071+0000","last_undegraded":"2024-01-27T00:47:20.524071+0000","last_fullsized":"2024-01-27T00:47:20.524071+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:44.855002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.528716+0000","last_change":"2024-01-27T00:47:19.488294+0000","last_active":"2024-01-27T00:47:20.528716+0000","last_peered":"2024-01-27T00:47:20.528716+0000","last_clean":"2024-01-27T00:47:20.528716+0000","last_became_active":"2024-01-27T00:47:19.487984+0000","last_became_peered":"2024-01-27T00:47:19.487984+0000","last_unstale":"2024-01-27T00:47:20.528716+0000","last_undegraded":"2024-01-27T00:47:20.528716+0000","last_fullsized":"2024-01-27T00:47:20.528716+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:08:02.191122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.513175+0000","last_change":"2024-01-27T00:47:20.508823+0000","last_active":"2024-01-27T00:47:20.513175+0000","last_peered":"2024-01-27T00:47:20.513175+0000","last_clean":"2024-01-27T00:47:20.513175+0000","last_became_active":"2024-01-27T00:47:20.507955+0000","last_became_peered":"2024-01-27T00:47:20.507955+0000","last_unstale":"2024-01-27T00:47:20.513175+0000","last_undegraded":"2024-01-27T00:47:20.513175+0000","last_fullsized":"2024-01-27T00:47:20.513175+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:44.632841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531724+0000","last_change":"2024-01-27T00:47:19.493726+0000","last_active":"2024-01-27T00:47:20.531724+0000","last_peered":"2024-01-27T00:47:20.531724+0000","last_clean":"2024-01-27T00:47:20.531724+0000","last_became_active":"2024-01-27T00:47:19.493237+0000","last_became_peered":"2024-01-27T00:47:19.493237+0000","last_unstale":"2024-01-27T00:47:20.531724+0000","last_undegraded":"2024-01-27T00:47:20.531724+0000","last_fullsized":"2024-01-27T00:47:20.531724+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:33:12.701672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.532639+0000","last_change":"2024-01-27T00:47:20.516445+0000","last_active":"2024-01-27T00:47:20.532639+0000","last_peered":"2024-01-27T00:47:20.532639+0000","last_clean":"2024-01-27T00:47:20.532639+0000","last_became_active":"2024-01-27T00:47:20.515582+0000","last_became_peered":"2024-01-27T00:47:20.515582+0000","last_unstale":"2024-01-27T00:47:20.532639+0000","last_undegraded":"2024-01-27T00:47:20.532639+0000","last_fullsized":"2024-01-27T00:47:20.532639+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:07:16.136646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.507907+0000","last_change":"2024-01-27T00:47:19.505534+0000","last_active":"2024-01-27T00:47:20.507907+0000","last_peered":"2024-01-27T00:47:20.507907+0000","last_clean":"2024-01-27T00:47:20.507907+0000","last_became_active":"2024-01-27T00:47:19.504749+0000","last_became_peered":"2024-01-27T00:47:19.504749+0000","last_unstale":"2024-01-27T00:47:20.507907+0000","last_undegraded":"2024-01-27T00:47:20.507907+0000","last_fullsized":"2024-01-27T00:47:20.507907+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:48.314323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.492777+0000","last_change":"2024-01-27T00:47:19.488903+0000","last_active":"2024-01-27T00:47:20.492777+0000","last_peered":"2024-01-27T00:47:20.492777+0000","last_clean":"2024-01-27T00:47:20.492777+0000","last_became_active":"2024-01-27T00:47:19.488649+0000","last_became_peered":"2024-01-27T00:47:19.488649+0000","last_unstale":"2024-01-27T00:47:20.492777+0000","last_undegraded":"2024-01-27T00:47:20.492777+0000","last_fullsized":"2024-01-27T00:47:20.492777+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:36:32.698812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499880+0000","last_change":"2024-01-27T00:47:19.505276+0000","last_active":"2024-01-27T00:47:20.499880+0000","last_peered":"2024-01-27T00:47:20.499880+0000","last_clean":"2024-01-27T00:47:20.499880+0000","last_became_active":"2024-01-27T00:47:19.504824+0000","last_became_peered":"2024-01-27T00:47:19.504824+0000","last_unstale":"2024-01-27T00:47:20.499880+0000","last_undegraded":"2024-01-27T00:47:20.499880+0000","last_fullsized":"2024-01-27T00:47:20.499880+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:53:04.664266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.510043+0000","last_change":"2024-01-27T00:47:20.507243+0000","last_active":"2024-01-27T00:47:20.510043+0000","last_peered":"2024-01-27T00:47:20.510043+0000","last_clean":"2024-01-27T00:47:20.510043+0000","last_became_active":"2024-01-27T00:47:20.506953+0000","last_became_peered":"2024-01-27T00:47:20.506953+0000","last_unstale":"2024-01-27T00:47:20.510043+0000","last_undegraded":"2024-01-27T00:47:20.510043+0000","last_fullsized":"2024-01-27T00:47:20.510043+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:23:47.466789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.479792+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:44:20.701222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.530619+0000","last_change":"2024-01-27T00:47:20.516228+0000","last_active":"2024-01-27T00:47:20.530619+0000","last_peered":"2024-01-27T00:47:20.530619+0000","last_clean":"2024-01-27T00:47:20.530619+0000","last_became_active":"2024-01-27T00:47:20.515906+0000","last_became_peered":"2024-01-27T00:47:20.515906+0000","last_unstale":"2024-01-27T00:47:20.530619+0000","last_undegraded":"2024-01-27T00:47:20.530619+0000","last_fullsized":"2024-01-27T00:47:20.530619+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:12:20.214602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"187'6","reported_seq":16,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.534091+0000","last_change":"2024-01-27T00:47:19.502338+0000","last_active":"2024-01-27T00:47:20.534091+0000","last_peered":"2024-01-27T00:47:20.534091+0000","last_clean":"2024-01-27T00:47:20.534091+0000","last_became_active":"2024-01-27T00:47:19.502072+0000","last_became_peered":"2024-01-27T00:47:19.502072+0000","last_unstale":"2024-01-27T00:47:20.534091+0000","last_undegraded":"2024-01-27T00:47:20.534091+0000","last_fullsized":"2024-01-27T00:47:20.534091+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:34:16.405629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.13","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527814+0000","last_change":"2024-01-27T00:47:19.491092+0000","last_active":"2024-01-27T00:47:20.527814+0000","last_peered":"2024-01-27T00:47:20.527814+0000","last_clean":"2024-01-27T00:47:20.527814+0000","last_became_active":"2024-01-27T00:47:19.490799+0000","last_became_peered":"2024-01-27T00:47:19.490799+0000","last_unstale":"2024-01-27T00:47:20.527814+0000","last_undegraded":"2024-01-27T00:47:20.527814+0000","last_fullsized":"2024-01-27T00:47:20.527814+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:23:46.462278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527448+0000","last_change":"2024-01-27T00:47:20.515730+0000","last_active":"2024-01-27T00:47:20.527448+0000","last_peered":"2024-01-27T00:47:20.527448+0000","last_clean":"2024-01-27T00:47:20.527448+0000","last_became_active":"2024-01-27T00:47:20.515422+0000","last_became_peered":"2024-01-27T00:47:20.515422+0000","last_unstale":"2024-01-27T00:47:20.527448+0000","last_undegraded":"2024-01-27T00:47:20.527448+0000","last_fullsized":"2024-01-27T00:47:20.527448+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:33.654223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.498821+0000","last_change":"2024-01-27T00:47:19.499915+0000","last_active":"2024-01-27T00:47:20.498821+0000","last_peered":"2024-01-27T00:47:20.498821+0000","last_clean":"2024-01-27T00:47:20.498821+0000","last_became_active":"2024-01-27T00:47:19.499418+0000","last_became_peered":"2024-01-27T00:47:19.499418+0000","last_unstale":"2024-01-27T00:47:20.498821+0000","last_undegraded":"2024-01-27T00:47:20.498821+0000","last_fullsized":"2024-01-27T00:47:20.498821+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:46.642397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.482438+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:43.628330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.514859+0000","last_change":"2024-01-27T00:47:20.513843+0000","last_active":"2024-01-27T00:47:20.514859+0000","last_peered":"2024-01-27T00:47:20.514859+0000","last_clean":"2024-01-27T00:47:20.514859+0000","last_became_active":"2024-01-27T00:47:20.513188+0000","last_became_peered":"2024-01-27T00:47:20.513188+0000","last_unstale":"2024-01-27T00:47:20.514859+0000","last_undegraded":"2024-01-27T00:47:20.514859+0000","last_fullsized":"2024-01-27T00:47:20.514859+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:15:01.330526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"187'8","reported_seq":18,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.529029+0000","last_change":"2024-01-27T00:47:19.491898+0000","last_active":"2024-01-27T00:47:20.529029+0000","last_peered":"2024-01-27T00:47:20.529029+0000","last_clean":"2024-01-27T00:47:20.529029+0000","last_became_active":"2024-01-27T00:47:19.491597+0000","last_became_peered":"2024-01-27T00:47:19.491597+0000","last_unstale":"2024-01-27T00:47:20.529029+0000","last_undegraded":"2024-01-27T00:47:20.529029+0000","last_fullsized":"2024-01-27T00:47:20.529029+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:47:33.870316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524260+0000","last_change":"2024-01-27T00:47:20.511230+0000","last_active":"2024-01-27T00:47:20.524260+0000","last_peered":"2024-01-27T00:47:20.524260+0000","last_clean":"2024-01-27T00:47:20.524260+0000","last_became_active":"2024-01-27T00:47:20.510893+0000","last_became_peered":"2024-01-27T00:47:20.510893+0000","last_unstale":"2024-01-27T00:47:20.524260+0000","last_undegraded":"2024-01-27T00:47:20.524260+0000","last_fullsized":"2024-01-27T00:47:20.524260+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:51:40.896423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531660+0000","last_change":"2024-01-27T00:47:19.504875+0000","last_active":"2024-01-27T00:47:20.531660+0000","last_peered":"2024-01-27T00:47:20.531660+0000","last_clean":"2024-01-27T00:47:20.531660+0000","last_became_active":"2024-01-27T00:47:19.504575+0000","last_became_peered":"2024-01-27T00:47:19.504575+0000","last_unstale":"2024-01-27T00:47:20.531660+0000","last_undegraded":"2024-01-27T00:47:20.531660+0000","last_fullsized":"2024-01-27T00:47:20.531660+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:43:03.734740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.e","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.533275+0000","last_change":"2024-01-27T00:47:19.509498+0000","last_active":"2024-01-27T00:47:20.533275+0000","last_peered":"2024-01-27T00:47:20.533275+0000","last_clean":"2024-01-27T00:47:20.533275+0000","last_became_active":"2024-01-27T00:47:19.509151+0000","last_became_peered":"2024-01-27T00:47:19.509151+0000","last_unstale":"2024-01-27T00:47:20.533275+0000","last_undegraded":"2024-01-27T00:47:20.533275+0000","last_fullsized":"2024-01-27T00:47:20.533275+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:54:21.075786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.d","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.529256+0000","last_change":"2024-01-27T00:47:19.498982+0000","last_active":"2024-01-27T00:47:20.529256+0000","last_peered":"2024-01-27T00:47:20.529256+0000","last_clean":"2024-01-27T00:47:20.529256+0000","last_became_active":"2024-01-27T00:47:19.498718+0000","last_became_peered":"2024-01-27T00:47:19.498718+0000","last_unstale":"2024-01-27T00:47:20.529256+0000","last_undegraded":"2024-01-27T00:47:20.529256+0000","last_fullsized":"2024-01-27T00:47:20.529256+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:04:42.913017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.493212+0000","last_change":"2024-01-27T00:47:19.493973+0000","last_active":"2024-01-27T00:47:20.493212+0000","last_peered":"2024-01-27T00:47:20.493212+0000","last_clean":"2024-01-27T00:47:20.493212+0000","last_became_active":"2024-01-27T00:47:19.493685+0000","last_became_peered":"2024-01-27T00:47:19.493685+0000","last_unstale":"2024-01-27T00:47:20.493212+0000","last_undegraded":"2024-01-27T00:47:20.493212+0000","last_fullsized":"2024-01-27T00:47:20.493212+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:08:00.978209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531595+0000","last_change":"2024-01-27T00:47:20.525394+0000","last_active":"2024-01-27T00:47:20.531595+0000","last_peered":"2024-01-27T00:47:20.531595+0000","last_clean":"2024-01-27T00:47:20.531595+0000","last_became_active":"2024-01-27T00:47:20.521446+0000","last_became_peered":"2024-01-27T00:47:20.521446+0000","last_unstale":"2024-01-27T00:47:20.531595+0000","last_undegraded":"2024-01-27T00:47:20.531595+0000","last_fullsized":"2024-01-27T00:47:20.531595+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:31.735208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.529633+0000","last_change":"2024-01-27T00:47:19.495677+0000","last_active":"2024-01-27T00:47:20.529633+0000","last_peered":"2024-01-27T00:47:20.529633+0000","last_clean":"2024-01-27T00:47:20.529633+0000","last_became_active":"2024-01-27T00:47:19.494329+0000","last_became_peered":"2024-01-27T00:47:19.494329+0000","last_unstale":"2024-01-27T00:47:20.529633+0000","last_undegraded":"2024-01-27T00:47:20.529633+0000","last_fullsized":"2024-01-27T00:47:20.529633+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:19:25.040151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522962+0000","last_change":"2024-01-27T00:47:20.512100+0000","last_active":"2024-01-27T00:47:20.522962+0000","last_peered":"2024-01-27T00:47:20.522962+0000","last_clean":"2024-01-27T00:47:20.522962+0000","last_became_active":"2024-01-27T00:47:20.511807+0000","last_became_peered":"2024-01-27T00:47:20.511807+0000","last_unstale":"2024-01-27T00:47:20.522962+0000","last_undegraded":"2024-01-27T00:47:20.522962+0000","last_fullsized":"2024-01-27T00:47:20.522962+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:41:19.485849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.482501+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:10:10.727612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522508+0000","last_change":"2024-01-27T00:47:20.512072+0000","last_active":"2024-01-27T00:47:20.522508+0000","last_peered":"2024-01-27T00:47:20.522508+0000","last_clean":"2024-01-27T00:47:20.522508+0000","last_became_active":"2024-01-27T00:47:20.511716+0000","last_became_peered":"2024-01-27T00:47:20.511716+0000","last_unstale":"2024-01-27T00:47:20.522508+0000","last_undegraded":"2024-01-27T00:47:20.522508+0000","last_fullsized":"2024-01-27T00:47:20.522508+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:18:15.805971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.508031+0000","last_change":"2024-01-27T00:47:19.505685+0000","last_active":"2024-01-27T00:47:20.508031+0000","last_peered":"2024-01-27T00:47:20.508031+0000","last_clean":"2024-01-27T00:47:20.508031+0000","last_became_active":"2024-01-27T00:47:19.505201+0000","last_became_peered":"2024-01-27T00:47:19.505201+0000","last_unstale":"2024-01-27T00:47:20.508031+0000","last_undegraded":"2024-01-27T00:47:20.508031+0000","last_fullsized":"2024-01-27T00:47:20.508031+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:10:58.917172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.481168+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:38:22.205934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.7","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527493+0000","last_change":"2024-01-27T00:47:19.498236+0000","last_active":"2024-01-27T00:47:20.527493+0000","last_peered":"2024-01-27T00:47:20.527493+0000","last_clean":"2024-01-27T00:47:20.527493+0000","last_became_active":"2024-01-27T00:47:19.497955+0000","last_became_peered":"2024-01-27T00:47:19.497955+0000","last_unstale":"2024-01-27T00:47:20.527493+0000","last_undegraded":"2024-01-27T00:47:20.527493+0000","last_fullsized":"2024-01-27T00:47:20.527493+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:52:03.781390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524123+0000","last_change":"2024-01-27T00:47:20.510553+0000","last_active":"2024-01-27T00:47:20.524123+0000","last_peered":"2024-01-27T00:47:20.524123+0000","last_clean":"2024-01-27T00:47:20.524123+0000","last_became_active":"2024-01-27T00:47:20.510231+0000","last_became_peered":"2024-01-27T00:47:20.510231+0000","last_unstale":"2024-01-27T00:47:20.524123+0000","last_undegraded":"2024-01-27T00:47:20.524123+0000","last_fullsized":"2024-01-27T00:47:20.524123+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:15.349063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499847+0000","last_change":"2024-01-27T00:47:19.505767+0000","last_active":"2024-01-27T00:47:20.499847+0000","last_peered":"2024-01-27T00:47:20.499847+0000","last_clean":"2024-01-27T00:47:20.499847+0000","last_became_active":"2024-01-27T00:47:19.505313+0000","last_became_peered":"2024-01-27T00:47:19.505313+0000","last_unstale":"2024-01-27T00:47:20.499847+0000","last_undegraded":"2024-01-27T00:47:20.499847+0000","last_fullsized":"2024-01-27T00:47:20.499847+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:48:00.955590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.530928+0000","last_change":"2024-01-27T00:47:20.516018+0000","last_active":"2024-01-27T00:47:20.530928+0000","last_peered":"2024-01-27T00:47:20.530928+0000","last_clean":"2024-01-27T00:47:20.530928+0000","last_became_active":"2024-01-27T00:47:20.515703+0000","last_became_peered":"2024-01-27T00:47:20.515703+0000","last_unstale":"2024-01-27T00:47:20.530928+0000","last_undegraded":"2024-01-27T00:47:20.530928+0000","last_fullsized":"2024-01-27T00:47:20.530928+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:12:19.305838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499639+0000","last_change":"2024-01-27T00:47:19.499255+0000","last_active":"2024-01-27T00:47:20.499639+0000","last_peered":"2024-01-27T00:47:20.499639+0000","last_clean":"2024-01-27T00:47:20.499639+0000","last_became_active":"2024-01-27T00:47:19.498933+0000","last_became_peered":"2024-01-27T00:47:19.498933+0000","last_unstale":"2024-01-27T00:47:20.499639+0000","last_undegraded":"2024-01-27T00:47:20.499639+0000","last_fullsized":"2024-01-27T00:47:20.499639+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:45.555078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524255+0000","last_change":"2024-01-27T00:47:20.511077+0000","last_active":"2024-01-27T00:47:20.524255+0000","last_peered":"2024-01-27T00:47:20.524255+0000","last_clean":"2024-01-27T00:47:20.524255+0000","last_became_active":"2024-01-27T00:47:20.510732+0000","last_became_peered":"2024-01-27T00:47:20.510732+0000","last_unstale":"2024-01-27T00:47:20.524255+0000","last_undegraded":"2024-01-27T00:47:20.524255+0000","last_fullsized":"2024-01-27T00:47:20.524255+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:05.094009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.508098+0000","last_change":"2024-01-27T00:47:19.504494+0000","last_active":"2024-01-27T00:47:20.508098+0000","last_peered":"2024-01-27T00:47:20.508098+0000","last_clean":"2024-01-27T00:47:20.508098+0000","last_became_active":"2024-01-27T00:47:19.504090+0000","last_became_peered":"2024-01-27T00:47:19.504090+0000","last_unstale":"2024-01-27T00:47:20.508098+0000","last_undegraded":"2024-01-27T00:47:20.508098+0000","last_fullsized":"2024-01-27T00:47:20.508098+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:22:59.518350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.479789+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:07.269026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.498924+0000","last_change":"2024-01-27T00:47:19.503541+0000","last_active":"2024-01-27T00:47:20.498924+0000","last_peered":"2024-01-27T00:47:20.498924+0000","last_clean":"2024-01-27T00:47:20.498924+0000","last_became_active":"2024-01-27T00:47:19.503073+0000","last_became_peered":"2024-01-27T00:47:19.503073+0000","last_unstale":"2024-01-27T00:47:20.498924+0000","last_undegraded":"2024-01-27T00:47:20.498924+0000","last_fullsized":"2024-01-27T00:47:20.498924+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:14.344552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531950+0000","last_change":"2024-01-27T00:47:20.527327+0000","last_active":"2024-01-27T00:47:20.531950+0000","last_peered":"2024-01-27T00:47:20.531950+0000","last_clean":"2024-01-27T00:47:20.531950+0000","last_became_active":"2024-01-27T00:47:20.520248+0000","last_became_peered":"2024-01-27T00:47:20.520248+0000","last_unstale":"2024-01-27T00:47:20.531950+0000","last_undegraded":"2024-01-27T00:47:20.531950+0000","last_fullsized":"2024-01-27T00:47:20.531950+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:19:26.044662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.482920+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:20:46.023087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.532846+0000","last_change":"2024-01-27T00:47:20.516514+0000","last_active":"2024-01-27T00:47:20.532846+0000","last_peered":"2024-01-27T00:47:20.532846+0000","last_clean":"2024-01-27T00:47:20.532846+0000","last_became_active":"2024-01-27T00:47:20.516160+0000","last_became_peered":"2024-01-27T00:47:20.516160+0000","last_unstale":"2024-01-27T00:47:20.532846+0000","last_undegraded":"2024-01-27T00:47:20.532846+0000","last_fullsized":"2024-01-27T00:47:20.532846+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:36:41.113078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.497052+0000","last_change":"2024-01-27T00:47:19.493487+0000","last_active":"2024-01-27T00:47:20.497052+0000","last_peered":"2024-01-27T00:47:20.497052+0000","last_clean":"2024-01-27T00:47:20.497052+0000","last_became_active":"2024-01-27T00:47:19.492721+0000","last_became_peered":"2024-01-27T00:47:19.492721+0000","last_unstale":"2024-01-27T00:47:20.497052+0000","last_undegraded":"2024-01-27T00:47:20.497052+0000","last_fullsized":"2024-01-27T00:47:20.497052+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:30.730697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499725+0000","last_change":"2024-01-27T00:47:19.501570+0000","last_active":"2024-01-27T00:47:20.499725+0000","last_peered":"2024-01-27T00:47:20.499725+0000","last_clean":"2024-01-27T00:47:20.499725+0000","last_became_active":"2024-01-27T00:47:19.501187+0000","last_became_peered":"2024-01-27T00:47:19.501187+0000","last_unstale":"2024-01-27T00:47:20.499725+0000","last_undegraded":"2024-01-27T00:47:20.499725+0000","last_fullsized":"2024-01-27T00:47:20.499725+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:54:58.914033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"179'8695","reported_seq":9433,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-01-27T00:47:17.003489+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T00:47:17.003489+0000","last_peered":"2024-01-27T00:47:17.003489+0000","last_clean":"2024-01-27T00:47:17.003489+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T00:47:17.003489+0000","last_undegraded":"2024-01-27T00:47:17.003489+0000","last_fullsized":"2024-01-27T00:47:17.003489+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8695,"log_dups_size":0,"ondisk_log_size":8695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1784,"num_read_kb":3778,"num_write":530,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":1662,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1784,"num_read_kb":3778,"num_write":530,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8695,"ondisk_log_size":8695,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316504,"kb_used_data":2832,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055336,"statfs":{"total":96636764160,"available":96312664064,"internally_reserved":0,"allocated":2899968,"data_stored":1787469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60799999999999998,"15min":0.60699999999999998},"min":{"1min":0.36199999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75800000000000001,"5min":0.83199999999999996,"15min":0.86899999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61299999999999999,"15min":0.61799999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.755,"5min":0.82099999999999995,"15min":0.90000000000000002},"last":0.61299999999999999}]},{"osd":1,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.34599999999999997},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.90400000000000003},"last":0.627},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63500000000000001,"15min":0.63},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.38200000000000001},"max":{"1min":0.76700000000000002,"5min":0.85099999999999998,"15min":0.875},"last":0.54900000000000004}]},{"osd":2,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.374,"5min":0.36299999999999999,"15min":0.34000000000000002},"max":{"1min":0.76400000000000001,"5min":0.77700000000000002,"15min":0.89700000000000002},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63800000000000001,"15min":0.63700000000000001},"min":{"1min":0.49399999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75600000000000001,"5min":0.86199999999999999,"15min":0.94199999999999995},"last":0.57499999999999996}]},{"osd":3,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.626,"15min":0.623},"min":{"1min":0.435,"5min":0.40999999999999998,"15min":0.39400000000000002},"max":{"1min":0.79200000000000004,"5min":0.86499999999999999,"15min":0.92600000000000005},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.77700000000000002,"5min":0.83499999999999996,"15min":0.88400000000000001},"last":0.67100000000000004}]},{"osd":4,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49399999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.32900000000000001,"15min":0.29699999999999999},"max":{"1min":0.61699999999999999,"5min":0.86499999999999999,"15min":0.92000000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.34599999999999997},"max":{"1min":0.76600000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.42499999999999999}]},{"osd":5,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":0.67100000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":0.67700000000000005,"5min":0.89100000000000001,"15min":0.95699999999999996},"last":0.45800000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59299999999999997,"15min":0.58299999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.36599999999999999},"max":{"1min":0.69299999999999995,"5min":0.91700000000000004,"15min":0.99399999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.36699999999999999},"max":{"1min":0.73599999999999999,"5min":0.82299999999999995,"15min":0.878},"last":0.49199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575775,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307732,"kb_used_data":2252,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064108,"statfs":{"total":96636764160,"available":96321646592,"internally_reserved":0,"allocated":2306048,"data_stored":1197101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58999999999999997,"15min":0.60399999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81000000000000005,"5min":0.82199999999999995,"15min":0.84899999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.38900000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.78500000000000003,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.55700000000000005}]},{"osd":1,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.748,"5min":0.79800000000000004,"15min":0.80800000000000005},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81499999999999995,"5min":0.87,"15min":0.87},"last":0.49299999999999999}]},{"osd":2,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60199999999999998,"15min":0.60899999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83699999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59899999999999998,"15min":0.60599999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74199999999999999,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.50600000000000001}]},{"osd":3,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86299999999999999,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64600000000000002,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.63400000000000001}]},{"osd":4,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.496,"15min":0.49099999999999999},"min":{"1min":0.34300000000000003,"5min":0.33200000000000002,"15min":0.313},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32100000000000001},"max":{"1min":0.77700000000000002,"5min":0.79100000000000004,"15min":0.83199999999999996},"last":0.39300000000000002}]},{"osd":5,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51900000000000002,"15min":0.51000000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.32200000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.375},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56499999999999995,"15min":0.56000000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.88800000000000001},"last":0.40999999999999998}]},{"osd":7,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57399999999999995,"15min":0.56499999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.58399999999999996,"15min":0.58099999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.33600000000000002},"max":{"1min":0.84099999999999997,"5min":0.84599999999999997,"15min":0.85999999999999999},"last":0.434}]}]},{"osd":1,"up_from":13,"seq":55834575776,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311888,"kb_used_data":2312,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059952,"statfs":{"total":96636764160,"available":96317390848,"internally_reserved":0,"allocated":2367488,"data_stored":1226772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.41799999999999998},"min":{"1min":0.311,"5min":0.254,"15min":0.23499999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.434,"5min":0.441,"15min":0.44500000000000001},"min":{"1min":0.30199999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.60599999999999998,"5min":0.66200000000000003,"15min":0.70199999999999996},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.41899999999999998,"15min":0.42499999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55100000000000005,"5min":0.59999999999999998,"15min":0.68899999999999995},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48499999999999999,"15min":0.49399999999999999},"min":{"1min":0.33500000000000002,"5min":0.31,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.76700000000000002},"last":0.48499999999999999}]},{"osd":3,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.503,"15min":0.52100000000000002},"min":{"1min":0.39400000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.63900000000000001,"5min":0.69399999999999995,"15min":1.0469999999999999},"last":0.621},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.49099999999999999,"15min":0.5},"min":{"1min":0.318,"5min":0.318,"15min":0.309},"max":{"1min":0.69599999999999995,"5min":0.72599999999999998,"15min":0.95599999999999996},"last":0.56899999999999995}]},{"osd":4,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.66100000000000003,"15min":0.67900000000000005},"min":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86299999999999999,"5min":1.02,"15min":1.3129999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67700000000000005,"15min":0.69999999999999996},"min":{"1min":0.505,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.93500000000000005,"5min":1.0069999999999999,"15min":1.097},"last":0.68500000000000005}]},{"osd":5,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66700000000000004,"15min":0.69199999999999995},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.46300000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":1.091},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.36299999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.278},"last":0.64600000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.68500000000000005,"15min":0.71099999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.435},"max":{"1min":0.88100000000000001,"5min":0.96999999999999997,"15min":1.141},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70799999999999996,"15min":0.72099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.91400000000000003,"5min":1.0229999999999999,"15min":1.1120000000000001},"last":0.77900000000000003}]},{"osd":7,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.73299999999999998,"15min":0.749},"min":{"1min":0.52600000000000002,"5min":0.52500000000000002,"15min":0.51100000000000001},"max":{"1min":1.0129999999999999,"5min":1.054,"15min":1.1819999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.73499999999999999,"15min":0.75800000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.51200000000000001},"max":{"1min":0.98299999999999998,"5min":0.98999999999999999,"15min":1.198},"last":0.71199999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575775,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312464,"kb_used_data":2888,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059376,"statfs":{"total":96636764160,"available":96316801024,"internally_reserved":0,"allocated":2957312,"data_stored":1815708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.28299999999999997,"5min":0.25800000000000001,"15min":0.25},"max":{"1min":0.55500000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.34499999999999997,"5min":0.27500000000000002,"15min":0.26300000000000001},"max":{"1min":0.61299999999999999,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.56999999999999995}]},{"osd":2,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.42099999999999999,"15min":0.42599999999999999},"min":{"1min":0.312,"5min":0.28599999999999998,"15min":0.26700000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.69399999999999995},"last":0.621},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48099999999999998,"15min":0.48299999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.629}]},{"osd":3,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48999999999999999,"15min":0.48699999999999999},"min":{"1min":0.32900000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.65600000000000003,"5min":0.755,"15min":0.755},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.496,"15min":0.498},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.42199999999999999}]},{"osd":4,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65200000000000002,"15min":0.68100000000000005},"min":{"1min":0.45100000000000001,"5min":0.44700000000000001,"15min":0.44400000000000001},"max":{"1min":0.77500000000000002,"5min":0.97099999999999997,"15min":1.125},"last":0.496},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.66000000000000003,"15min":0.67800000000000005},"min":{"1min":0.46200000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.96199999999999997,"5min":0.97599999999999998,"15min":1.0169999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68100000000000005,"15min":0.69399999999999995},"min":{"1min":0.47899999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.84099999999999997,"5min":0.94199999999999995,"15min":1.054},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.70799999999999996,"15min":0.71899999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.98199999999999998,"5min":1.056,"15min":1.056},"last":0.46200000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69999999999999996,"15min":0.71199999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.84399999999999997,"5min":1.0269999999999999,"15min":1.087},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.72299999999999998,"15min":0.73899999999999999},"min":{"1min":0.52500000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.94599999999999995,"5min":1.0349999999999999,"15min":1.052},"last":0.88400000000000001}]},{"osd":7,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.747,"15min":0.75700000000000001},"min":{"1min":0.60599999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.016,"5min":1.0880000000000001,"15min":1.1080000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.53700000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.034,"5min":1.071,"15min":1.071},"last":0.55400000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575775,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311876,"kb_used_data":2300,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059964,"statfs":{"total":96636764160,"available":96317403136,"internally_reserved":0,"allocated":2355200,"data_stored":1225236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38100000000000001,"15min":0.39800000000000002},"min":{"1min":0.29499999999999998,"5min":0.25900000000000001,"15min":0.252},"max":{"1min":0.48699999999999999,"5min":0.60499999999999998,"15min":0.65800000000000003},"last":0.375},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":0.438},"min":{"1min":0.29499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.71899999999999997},"last":0.35599999999999998}]},{"osd":1,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.41999999999999998,"15min":0.432},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.60799999999999998,"5min":0.627,"15min":0.68300000000000005},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.5,"15min":0.51400000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66100000000000003,"5min":0.76400000000000001,"15min":0.80300000000000005},"last":0.40400000000000003}]},{"osd":3,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47799999999999998,"15min":0.49199999999999999},"min":{"1min":0.34999999999999998,"5min":0.34100000000000003,"15min":0.33100000000000002},"max":{"1min":0.65200000000000002,"5min":0.745,"15min":0.745},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47099999999999997,"15min":0.48699999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.29499999999999998},"max":{"1min":0.68000000000000005,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.44900000000000001}]},{"osd":4,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.65000000000000002,"15min":0.66700000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41999999999999998},"max":{"1min":0.77000000000000002,"5min":0.85899999999999999,"15min":1.0660000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66700000000000004,"15min":0.68200000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92400000000000004,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.83999999999999997}]},{"osd":5,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66800000000000004,"15min":0.68999999999999995},"min":{"1min":0.49099999999999999,"5min":0.46500000000000002,"15min":0.42899999999999999},"max":{"1min":0.78400000000000003,"5min":0.96499999999999997,"15min":1.0649999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68200000000000005,"15min":0.69799999999999995},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.90600000000000003,"5min":0.98799999999999999,"15min":1.044},"last":0.77600000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.68100000000000005,"15min":0.70099999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.86299999999999999,"5min":1.0569999999999999,"15min":1.1140000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.69399999999999995,"15min":0.70999999999999996},"min":{"1min":0.47999999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89400000000000002,"5min":1.002,"15min":1.0720000000000001},"last":0.79000000000000004}]},{"osd":7,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.71499999999999997,"15min":0.72799999999999998},"min":{"1min":0.55600000000000005,"5min":0.50900000000000001,"15min":0.48999999999999999},"max":{"1min":0.97499999999999998,"5min":1.0860000000000001,"15min":1.1479999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.73899999999999999,"15min":0.76200000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.98899999999999999,"5min":1.022,"15min":1.1299999999999999},"last":0.82799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575774,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315992,"kb_used_data":2320,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055848,"statfs":{"total":96636764160,"available":96313188352,"internally_reserved":0,"allocated":2375680,"data_stored":1227348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.39600000000000002,"15min":0.40100000000000002},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.50600000000000001,"5min":0.63,"15min":0.752},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.443,"5min":0.45700000000000002,"15min":0.45300000000000001},"min":{"1min":0.32900000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.57899999999999996,"5min":0.68799999999999994,"15min":0.85699999999999998},"last":0.69599999999999995}]},{"osd":1,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.46600000000000003},"min":{"1min":0.307,"5min":0.29699999999999999,"15min":0.27400000000000002},"max":{"1min":0.61499999999999999,"5min":0.69699999999999995,"15min":0.94599999999999995},"last":0.627},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.498,"15min":0.498},"min":{"1min":0.36299999999999999,"5min":0.34399999999999997,"15min":0.316},"max":{"1min":0.69299999999999995,"5min":0.745,"15min":0.745},"last":0.72999999999999998}]},{"osd":2,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.35699999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64900000000000002,"5min":0.67900000000000005,"15min":0.84499999999999997},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.46000000000000002,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.66600000000000004,"5min":0.72699999999999998,"15min":0.82299999999999995},"last":0.80400000000000005}]},{"osd":4,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.69299999999999995,"15min":0.68999999999999995},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.88400000000000001,"5min":0.89300000000000002,"15min":1},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.48199999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.98699999999999999,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.89700000000000002}]},{"osd":5,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.69299999999999995,"15min":0.68799999999999994},"min":{"1min":0.505,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.88900000000000001,"5min":0.94499999999999995,"15min":1.0429999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68400000000000005,"15min":0.68700000000000006},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90400000000000003,"5min":0.99099999999999999,"15min":1.0289999999999999},"last":1}]},{"osd":6,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.68300000000000005,"15min":0.69099999999999995},"min":{"1min":0.53000000000000003,"5min":0.52000000000000002,"15min":0.48899999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.0820000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.71799999999999997,"15min":0.71599999999999997},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.44},"max":{"1min":0.91600000000000004,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.95199999999999996}]},{"osd":7,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.69999999999999996,"15min":0.71399999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.96399999999999997,"5min":1.081,"15min":1.081},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.55800000000000005,"5min":0.48999999999999999,"15min":0.46300000000000002},"max":{"1min":0.91700000000000004,"5min":1.0229999999999999,"15min":1.1240000000000001},"last":0.98999999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575775,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311828,"kb_used_data":2252,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060012,"statfs":{"total":96636764160,"available":96317452288,"internally_reserved":0,"allocated":2306048,"data_stored":1197101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61399999999999999,"15min":0.60599999999999998},"min":{"1min":0.40699999999999997,"5min":0.39300000000000002,"15min":0.32100000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59299999999999997}]},{"osd":1,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.627,"15min":0.628},"min":{"1min":0.52900000000000003,"5min":0.42499999999999999,"15min":0.40999999999999998},"max":{"1min":0.75700000000000001,"5min":0.83199999999999996,"15min":0.86299999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61899999999999999,"15min":0.626},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84699999999999998},"last":0.56399999999999995}]},{"osd":2,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.46100000000000002,"5min":0.436,"15min":0.40899999999999997},"max":{"1min":0.753,"5min":0.81899999999999995,"15min":0.84499999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.5,"5min":0.44400000000000001,"15min":0.38100000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.41799999999999998}]},{"osd":3,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.626,"15min":0.627},"min":{"1min":0.45900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.61899999999999999,"15min":0.623},"min":{"1min":0.42599999999999999,"5min":0.41199999999999998,"15min":0.41099999999999998},"max":{"1min":0.80600000000000005,"5min":0.80900000000000005,"15min":0.876},"last":0.76800000000000002}]},{"osd":5,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48199999999999998,"15min":0.48499999999999999},"min":{"1min":0.35899999999999999,"5min":0.32100000000000001,"15min":0.28699999999999998},"max":{"1min":0.69699999999999995,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.33100000000000002,"5min":0.30499999999999999,"15min":0.30199999999999999},"max":{"1min":0.77300000000000002,"5min":0.876,"15min":0.876},"last":0.65800000000000003}]},{"osd":6,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49299999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.54100000000000004},"min":{"1min":0.39800000000000002,"5min":0.34300000000000003,"15min":0.32800000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.67500000000000004}]},{"osd":7,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41599999999999998,"5min":0.34999999999999998,"15min":0.33700000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.625},{"interface":"front","average":{"1min":0.627,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.46000000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.80400000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575775,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311828,"kb_used_data":2252,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060012,"statfs":{"total":96636764160,"available":96317452288,"internally_reserved":0,"allocated":2306048,"data_stored":1197101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61399999999999999,"15min":0.60799999999999998},"min":{"1min":0.379,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76200000000000001,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.61599999999999999,"15min":0.623},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38600000000000001},"max":{"1min":0.76600000000000001,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.46800000000000003}]},{"osd":1,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65900000000000003,"15min":0.64900000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66400000000000003,"15min":0.67100000000000004},"min":{"1min":0.44600000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80300000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.60499999999999998}]},{"osd":2,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.61399999999999999,"15min":0.60299999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.71999999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.79100000000000004,"5min":0.876,"15min":0.876},"last":0.41399999999999998}]},{"osd":3,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60399999999999998,"15min":0.60299999999999998},"min":{"1min":0.38200000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.70099999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.64200000000000002,"15min":0.63400000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.79600000000000004,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.64900000000000002}]},{"osd":4,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.47499999999999998,"15min":0.47599999999999998},"min":{"1min":0.307,"5min":0.30399999999999999,"15min":0.30299999999999999},"max":{"1min":0.67600000000000005,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.82799999999999996},"last":0.71699999999999997}]},{"osd":6,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50700000000000001,"15min":0.504},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.31900000000000001},"max":{"1min":0.67400000000000004,"5min":0.71399999999999997,"15min":0.79900000000000004},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.379,"5min":0.36799999999999999,"15min":0.36099999999999999},"max":{"1min":0.70099999999999996,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.63500000000000001}]},{"osd":7,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34699999999999998},"max":{"1min":0.747,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.45800000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.69999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T00:47:25.112 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T00:47:25.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.282+0000 7f357e612700 1 Processor -- start 2024-01-27T00:47:25.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.282+0000 7f357e612700 1 -- start start 2024-01-27T00:47:25.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.283+0000 7f357e612700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3578158f50 0x7f35781593a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:25.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.283+0000 7f357e612700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3578157d80 0x7f35781581d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:25.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.283+0000 7f357e612700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f357806cae0 con 0x7f3578155290 2024-01-27T00:47:25.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.283+0000 7f357e612700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35781598e0 con 0x7f3578158f50 2024-01-27T00:47:25.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.283+0000 7f357e612700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3578158710 con 0x7f3578157d80 2024-01-27T00:47:25.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.283+0000 7f3577fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3578155290 0x7f357806cf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45664/0 (socket says 172.21.15.46:45664) 2024-01-27T00:47:25.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.283+0000 7f3577fff700 1 -- 172.21.15.46:0/2251282567 learned_addr learned my addr 172.21.15.46:0/2251282567 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:25.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.283+0000 7f35777fe700 1 --2- 172.21.15.46:0/2251282567 >> v2:172.21.15.106:3300/0 conn(0x7f3578158f50 0x7f35781593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:25.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.283+0000 7f357cbaf700 1 --2- 172.21.15.46:0/2251282567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3578157d80 0x7f35781581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:25.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.283+0000 7f3576ffd700 1 -- 172.21.15.46:0/2251282567 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304856673 0 0) 0x7f357806cae0 con 0x7f3578155290 2024-01-27T00:47:25.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.283+0000 7f3576ffd700 1 -- 172.21.15.46:0/2251282567 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3564003570 con 0x7f3578155290 2024-01-27T00:47:25.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.284+0000 7f3576ffd700 1 -- 172.21.15.46:0/2251282567 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2765726894 0 0) 0x7f3564003570 con 0x7f3578155290 2024-01-27T00:47:25.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.284+0000 7f35777fe700 1 -- 172.21.15.46:0/2251282567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3578157d80 msgr2=0x7f35781581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:25.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.284+0000 7f35777fe700 1 --2- 172.21.15.46:0/2251282567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3578157d80 0x7f35781581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.284+0000 7f35777fe700 1 -- 172.21.15.46:0/2251282567 >> v1:172.21.15.46:6789/0 conn(0x7f3578155290 legacy=0x7f357806cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:25.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.284+0000 7f35777fe700 1 -- 172.21.15.46:0/2251282567 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35780ce010 con 0x7f3578158f50 2024-01-27T00:47:25.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.284+0000 7f357cbaf700 1 --2- 172.21.15.46:0/2251282567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3578157d80 0x7f35781581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:25.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.284+0000 7f35777fe700 1 --2- 172.21.15.46:0/2251282567 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3578158f50 0x7f35781593a0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f35600098e0 tx=0x7f35600091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a6f0b8cf702da86 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:25.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.284+0000 7f3576ffd700 1 -- 172.21.15.46:0/2251282567 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3568001470 con 0x7f3578158f50 2024-01-27T00:47:25.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.284+0000 7f3576ffd700 1 -- 172.21.15.46:0/2251282567 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3560011420 con 0x7f3578158f50 2024-01-27T00:47:25.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.284+0000 7f3576ffd700 1 -- 172.21.15.46:0/2251282567 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35600117a0 con 0x7f3578158f50 2024-01-27T00:47:25.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.285+0000 7f357e612700 1 -- 172.21.15.46:0/2251282567 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3578158f50 msgr2=0x7f35781593a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:25.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.285+0000 7f357e612700 1 --2- 172.21.15.46:0/2251282567 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3578158f50 0x7f35781593a0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f35600098e0 tx=0x7f35600091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.285+0000 7f357e612700 1 -- 172.21.15.46:0/2251282567 shutdown_connections 2024-01-27T00:47:25.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.285+0000 7f357e612700 1 --2- 172.21.15.46:0/2251282567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3578157d80 0x7f35781581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.285+0000 7f357e612700 1 --2- 172.21.15.46:0/2251282567 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3578158f50 0x7f35781593a0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.285+0000 7f357e612700 1 -- 172.21.15.46:0/2251282567 >> 172.21.15.46:0/2251282567 conn(0x7f357806b820 msgr2=0x7f35780cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:25.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.285+0000 7f357e612700 1 -- 172.21.15.46:0/2251282567 shutdown_connections 2024-01-27T00:47:25.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.285+0000 7f357e612700 1 -- 172.21.15.46:0/2251282567 wait complete. 2024-01-27T00:47:25.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.286+0000 7f357e612700 1 Processor -- start 2024-01-27T00:47:25.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.286+0000 7f357e612700 1 -- start start 2024-01-27T00:47:25.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.287+0000 7f357e612700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3578155290 0x7f35781688f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:25.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.287+0000 7f357e612700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3578157d80 0x7f35780c7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:25.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.287+0000 7f357e612700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3578158f50 0x7f3578152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:25.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.287+0000 7f357e612700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35780c75d0 con 0x7f3578155290 2024-01-27T00:47:25.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.287+0000 7f357e612700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3578152980 con 0x7f3578158f50 2024-01-27T00:47:25.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.287+0000 7f357e612700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3578151c00 con 0x7f3578157d80 2024-01-27T00:47:25.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.287+0000 7f3577fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3578155290 0x7f35781688f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:25.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.287+0000 7f3577fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3578155290 0x7f35781688f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40594/0 (socket says 172.21.15.46:40594) 2024-01-27T00:47:25.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.287+0000 7f3577fff700 1 -- 172.21.15.46:0/3842347089 learned_addr learned my addr 172.21.15.46:0/3842347089 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:25.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.287+0000 7f35777fe700 1 --2- 172.21.15.46:0/3842347089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3578157d80 0x7f35780c7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:25.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.287+0000 7f357cbaf700 1 --2- 172.21.15.46:0/3842347089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3578158f50 0x7f3578152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:25.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.287+0000 7f3577fff700 1 -- 172.21.15.46:0/3842347089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3578157d80 msgr2=0x7f35780c7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:25.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.287+0000 7f3577fff700 1 --2- 172.21.15.46:0/3842347089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3578157d80 0x7f35780c7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.287+0000 7f3577fff700 1 -- 172.21.15.46:0/3842347089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3578158f50 msgr2=0x7f3578152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:25.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.287+0000 7f3577fff700 1 --2- 172.21.15.46:0/3842347089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3578158f50 0x7f3578152440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.287+0000 7f3577fff700 1 -- 172.21.15.46:0/3842347089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3570009a20 con 0x7f3578155290 2024-01-27T00:47:25.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.288+0000 7f3577fff700 1 --2- 172.21.15.46:0/3842347089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3578155290 0x7f35781688f0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f3568001660 tx=0x7f3568001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:25.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.288+0000 7f356ffff700 1 -- 172.21.15.46:0/3842347089 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3568001d10 con 0x7f3578155290 2024-01-27T00:47:25.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.288+0000 7f356ffff700 1 -- 172.21.15.46:0/3842347089 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3568007360 con 0x7f3578155290 2024-01-27T00:47:25.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.288+0000 7f357e612700 1 -- 172.21.15.46:0/3842347089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3560009420 con 0x7f3578155290 2024-01-27T00:47:25.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.288+0000 7f356ffff700 1 -- 172.21.15.46:0/3842347089 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3568004830 con 0x7f3578155290 2024-01-27T00:47:25.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.288+0000 7f357e612700 1 -- 172.21.15.46:0/3842347089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35781680f0 con 0x7f3578155290 2024-01-27T00:47:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.289+0000 7f357e612700 1 -- 172.21.15.46:0/3842347089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f357804c710 con 0x7f3578155290 2024-01-27T00:47:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.289+0000 7f356ffff700 1 -- 172.21.15.46:0/3842347089 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3568004c20 con 0x7f3578155290 2024-01-27T00:47:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.290+0000 7f356ffff700 1 --2- 172.21.15.46:0/3842347089 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3558066f30 0x7f35580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.290+0000 7f356ffff700 1 -- 172.21.15.46:0/3842347089 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f3568083f50 con 0x7f3578155290 2024-01-27T00:47:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.290+0000 7f35777fe700 1 --2- 172.21.15.46:0/3842347089 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3558066f30 0x7f35580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.291+0000 7f35777fe700 1 --2- 172.21.15.46:0/3842347089 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3558066f30 0x7f35580693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f35600091e0 tx=0x7f3560016000 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:25.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.294+0000 7f356ffff700 1 -- 172.21.15.46:0/3842347089 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3568051010 con 0x7f3578155290 2024-01-27T00:47:25.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.493+0000 7f357e612700 1 -- 172.21.15.46:0/3842347089 --> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3578158960 con 0x7f3558066f30 2024-01-27T00:47:25.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.502+0000 7f356ffff700 1 -- 172.21.15.46:0/3842347089 <== mgr.9313 v2:172.21.15.106:6836/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284817 (secure 0 0 0) 0x7f3578158960 con 0x7f3558066f30 2024-01-27T00:47:25.507 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.505+0000 7f357e612700 1 -- 172.21.15.46:0/3842347089 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3558066f30 msgr2=0x7f35580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.505+0000 7f357e612700 1 --2- 172.21.15.46:0/3842347089 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3558066f30 0x7f35580693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f35600091e0 tx=0x7f3560016000 comp rx=0 tx=0).stop 2024-01-27T00:47:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.505+0000 7f357e612700 1 -- 172.21.15.46:0/3842347089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3578155290 msgr2=0x7f35781688f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.505+0000 7f357e612700 1 --2- 172.21.15.46:0/3842347089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3578155290 0x7f35781688f0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f3568001660 tx=0x7f3568001a80 comp rx=0 tx=0).stop 2024-01-27T00:47:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.505+0000 7f357e612700 1 -- 172.21.15.46:0/3842347089 shutdown_connections 2024-01-27T00:47:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.505+0000 7f357e612700 1 --2- 172.21.15.46:0/3842347089 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3558066f30 0x7f35580693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.505+0000 7f357e612700 1 --2- 172.21.15.46:0/3842347089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3578155290 0x7f35781688f0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.505+0000 7f357e612700 1 --2- 172.21.15.46:0/3842347089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3578158f50 0x7f3578152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.505+0000 7f357e612700 1 --2- 172.21.15.46:0/3842347089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3578157d80 0x7f35780c7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.505+0000 7f357e612700 1 -- 172.21.15.46:0/3842347089 >> 172.21.15.46:0/3842347089 conn(0x7f357806b820 msgr2=0x7f357806f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.505+0000 7f357e612700 1 -- 172.21.15.46:0/3842347089 shutdown_connections 2024-01-27T00:47:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.505+0000 7f357e612700 1 -- 172.21.15.46:0/3842347089 wait complete. 2024-01-27T00:47:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T00:47:25.519 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-27T00:47:23.570969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592030,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1784,"num_read_kb":3778,"num_write":563,"num_write_kb":3473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8724,"ondisk_log_size":8724,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2500112,"kb_used_data":19408,"kb_used_omap":0,"kb_used_meta":2480640,"kb_avail":752474608,"statfs":{"total":773094113280,"available":770533998592,"internally_reserved":0,"allocated":19873792,"data_stored":10873836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2540175360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1662,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.095847"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.523334+0000","last_change":"2024-01-27T00:47:20.509897+0000","last_active":"2024-01-27T00:47:20.523334+0000","last_peered":"2024-01-27T00:47:20.523334+0000","last_clean":"2024-01-27T00:47:20.523334+0000","last_became_active":"2024-01-27T00:47:20.509610+0000","last_became_peered":"2024-01-27T00:47:20.509610+0000","last_unstale":"2024-01-27T00:47:20.523334+0000","last_undegraded":"2024-01-27T00:47:20.523334+0000","last_fullsized":"2024-01-27T00:47:20.523334+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:42:01.971654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527541+0000","last_change":"2024-01-27T00:47:20.514165+0000","last_active":"2024-01-27T00:47:20.527541+0000","last_peered":"2024-01-27T00:47:20.527541+0000","last_clean":"2024-01-27T00:47:20.527541+0000","last_became_active":"2024-01-27T00:47:20.513848+0000","last_became_peered":"2024-01-27T00:47:20.513848+0000","last_unstale":"2024-01-27T00:47:20.527541+0000","last_undegraded":"2024-01-27T00:47:20.527541+0000","last_fullsized":"2024-01-27T00:47:20.527541+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:43:04.739251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524282+0000","last_change":"2024-01-27T00:47:20.511666+0000","last_active":"2024-01-27T00:47:20.524282+0000","last_peered":"2024-01-27T00:47:20.524282+0000","last_clean":"2024-01-27T00:47:20.524282+0000","last_became_active":"2024-01-27T00:47:20.509826+0000","last_became_peered":"2024-01-27T00:47:20.509826+0000","last_unstale":"2024-01-27T00:47:20.524282+0000","last_undegraded":"2024-01-27T00:47:20.524282+0000","last_fullsized":"2024-01-27T00:47:20.524282+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:53:47.125547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524159+0000","last_change":"2024-01-27T00:47:20.511731+0000","last_active":"2024-01-27T00:47:20.524159+0000","last_peered":"2024-01-27T00:47:20.524159+0000","last_clean":"2024-01-27T00:47:20.524159+0000","last_became_active":"2024-01-27T00:47:20.511436+0000","last_became_peered":"2024-01-27T00:47:20.511436+0000","last_unstale":"2024-01-27T00:47:20.524159+0000","last_undegraded":"2024-01-27T00:47:20.524159+0000","last_fullsized":"2024-01-27T00:47:20.524159+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:40.342669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524015+0000","last_change":"2024-01-27T00:47:20.510790+0000","last_active":"2024-01-27T00:47:20.524015+0000","last_peered":"2024-01-27T00:47:20.524015+0000","last_clean":"2024-01-27T00:47:20.524015+0000","last_became_active":"2024-01-27T00:47:20.510447+0000","last_became_peered":"2024-01-27T00:47:20.510447+0000","last_unstale":"2024-01-27T00:47:20.524015+0000","last_undegraded":"2024-01-27T00:47:20.524015+0000","last_fullsized":"2024-01-27T00:47:20.524015+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:56:44.894090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531204+0000","last_change":"2024-01-27T00:47:20.520667+0000","last_active":"2024-01-27T00:47:20.531204+0000","last_peered":"2024-01-27T00:47:20.531204+0000","last_clean":"2024-01-27T00:47:20.531204+0000","last_became_active":"2024-01-27T00:47:20.519891+0000","last_became_peered":"2024-01-27T00:47:20.519891+0000","last_unstale":"2024-01-27T00:47:20.531204+0000","last_undegraded":"2024-01-27T00:47:20.531204+0000","last_fullsized":"2024-01-27T00:47:20.531204+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:49:22.885968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522694+0000","last_change":"2024-01-27T00:47:20.513550+0000","last_active":"2024-01-27T00:47:20.522694+0000","last_peered":"2024-01-27T00:47:20.522694+0000","last_clean":"2024-01-27T00:47:20.522694+0000","last_became_active":"2024-01-27T00:47:20.512667+0000","last_became_peered":"2024-01-27T00:47:20.512667+0000","last_unstale":"2024-01-27T00:47:20.522694+0000","last_undegraded":"2024-01-27T00:47:20.522694+0000","last_fullsized":"2024-01-27T00:47:20.522694+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:09:43.978584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.504493+0000","last_change":"2024-01-27T00:47:20.500113+0000","last_active":"2024-01-27T00:47:20.504493+0000","last_peered":"2024-01-27T00:47:20.504493+0000","last_clean":"2024-01-27T00:47:20.504493+0000","last_became_active":"2024-01-27T00:47:20.499814+0000","last_became_peered":"2024-01-27T00:47:20.499814+0000","last_unstale":"2024-01-27T00:47:20.504493+0000","last_undegraded":"2024-01-27T00:47:20.504493+0000","last_fullsized":"2024-01-27T00:47:20.504493+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:05:38.788368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.532911+0000","last_change":"2024-01-27T00:47:20.518490+0000","last_active":"2024-01-27T00:47:20.532911+0000","last_peered":"2024-01-27T00:47:20.532911+0000","last_clean":"2024-01-27T00:47:20.532911+0000","last_became_active":"2024-01-27T00:47:20.518138+0000","last_became_peered":"2024-01-27T00:47:20.518138+0000","last_unstale":"2024-01-27T00:47:20.532911+0000","last_undegraded":"2024-01-27T00:47:20.532911+0000","last_fullsized":"2024-01-27T00:47:20.532911+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:28.557943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531230+0000","last_change":"2024-01-27T00:47:20.520931+0000","last_active":"2024-01-27T00:47:20.531230+0000","last_peered":"2024-01-27T00:47:20.531230+0000","last_clean":"2024-01-27T00:47:20.531230+0000","last_became_active":"2024-01-27T00:47:20.520322+0000","last_became_peered":"2024-01-27T00:47:20.520322+0000","last_unstale":"2024-01-27T00:47:20.531230+0000","last_undegraded":"2024-01-27T00:47:20.531230+0000","last_fullsized":"2024-01-27T00:47:20.531230+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:04:43.917528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.530444+0000","last_change":"2024-01-27T00:47:20.522803+0000","last_active":"2024-01-27T00:47:20.530444+0000","last_peered":"2024-01-27T00:47:20.530444+0000","last_clean":"2024-01-27T00:47:20.530444+0000","last_became_active":"2024-01-27T00:47:20.522463+0000","last_became_peered":"2024-01-27T00:47:20.522463+0000","last_unstale":"2024-01-27T00:47:20.530444+0000","last_undegraded":"2024-01-27T00:47:20.530444+0000","last_fullsized":"2024-01-27T00:47:20.530444+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:04:18.616847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.504894+0000","last_change":"2024-01-27T00:47:20.503158+0000","last_active":"2024-01-27T00:47:20.504894+0000","last_peered":"2024-01-27T00:47:20.504894+0000","last_clean":"2024-01-27T00:47:20.504894+0000","last_became_active":"2024-01-27T00:47:20.502875+0000","last_became_peered":"2024-01-27T00:47:20.502875+0000","last_unstale":"2024-01-27T00:47:20.504894+0000","last_undegraded":"2024-01-27T00:47:20.504894+0000","last_fullsized":"2024-01-27T00:47:20.504894+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:33:13.706183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.513540+0000","last_change":"2024-01-27T00:47:20.509928+0000","last_active":"2024-01-27T00:47:20.513540+0000","last_peered":"2024-01-27T00:47:20.513540+0000","last_clean":"2024-01-27T00:47:20.513540+0000","last_became_active":"2024-01-27T00:47:20.509630+0000","last_became_peered":"2024-01-27T00:47:20.509630+0000","last_unstale":"2024-01-27T00:47:20.513540+0000","last_undegraded":"2024-01-27T00:47:20.513540+0000","last_fullsized":"2024-01-27T00:47:20.513540+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:02:52.058658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531770+0000","last_change":"2024-01-27T00:47:20.527150+0000","last_active":"2024-01-27T00:47:20.531770+0000","last_peered":"2024-01-27T00:47:20.531770+0000","last_clean":"2024-01-27T00:47:20.531770+0000","last_became_active":"2024-01-27T00:47:20.523827+0000","last_became_peered":"2024-01-27T00:47:20.523827+0000","last_unstale":"2024-01-27T00:47:20.531770+0000","last_undegraded":"2024-01-27T00:47:20.531770+0000","last_fullsized":"2024-01-27T00:47:20.531770+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:52:04.785901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.480155+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:27:44.775426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.510752+0000","last_change":"2024-01-27T00:47:20.509849+0000","last_active":"2024-01-27T00:47:20.510752+0000","last_peered":"2024-01-27T00:47:20.510752+0000","last_clean":"2024-01-27T00:47:20.510752+0000","last_became_active":"2024-01-27T00:47:20.509553+0000","last_became_peered":"2024-01-27T00:47:20.509553+0000","last_unstale":"2024-01-27T00:47:20.510752+0000","last_undegraded":"2024-01-27T00:47:20.510752+0000","last_fullsized":"2024-01-27T00:47:20.510752+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:44:26.489354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.528860+0000","last_change":"2024-01-27T00:47:19.498621+0000","last_active":"2024-01-27T00:47:20.528860+0000","last_peered":"2024-01-27T00:47:20.528860+0000","last_clean":"2024-01-27T00:47:20.528860+0000","last_became_active":"2024-01-27T00:47:19.498357+0000","last_became_peered":"2024-01-27T00:47:19.498357+0000","last_unstale":"2024-01-27T00:47:20.528860+0000","last_undegraded":"2024-01-27T00:47:20.528860+0000","last_fullsized":"2024-01-27T00:47:20.528860+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:20:56.150512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522707+0000","last_change":"2024-01-27T00:47:20.513840+0000","last_active":"2024-01-27T00:47:20.522707+0000","last_peered":"2024-01-27T00:47:20.522707+0000","last_clean":"2024-01-27T00:47:20.522707+0000","last_became_active":"2024-01-27T00:47:20.512745+0000","last_became_peered":"2024-01-27T00:47:20.512745+0000","last_unstale":"2024-01-27T00:47:20.522707+0000","last_undegraded":"2024-01-27T00:47:20.522707+0000","last_fullsized":"2024-01-27T00:47:20.522707+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:11:59.061116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.482562+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:43:05.614968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1c","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.533308+0000","last_change":"2024-01-27T00:47:19.505152+0000","last_active":"2024-01-27T00:47:20.533308+0000","last_peered":"2024-01-27T00:47:20.533308+0000","last_clean":"2024-01-27T00:47:20.533308+0000","last_became_active":"2024-01-27T00:47:19.504128+0000","last_became_peered":"2024-01-27T00:47:19.504128+0000","last_unstale":"2024-01-27T00:47:20.533308+0000","last_undegraded":"2024-01-27T00:47:20.533308+0000","last_fullsized":"2024-01-27T00:47:20.533308+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:52.404716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524071+0000","last_change":"2024-01-27T00:47:20.510375+0000","last_active":"2024-01-27T00:47:20.524071+0000","last_peered":"2024-01-27T00:47:20.524071+0000","last_clean":"2024-01-27T00:47:20.524071+0000","last_became_active":"2024-01-27T00:47:20.510069+0000","last_became_peered":"2024-01-27T00:47:20.510069+0000","last_unstale":"2024-01-27T00:47:20.524071+0000","last_undegraded":"2024-01-27T00:47:20.524071+0000","last_fullsized":"2024-01-27T00:47:20.524071+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:44.855002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.528716+0000","last_change":"2024-01-27T00:47:19.488294+0000","last_active":"2024-01-27T00:47:20.528716+0000","last_peered":"2024-01-27T00:47:20.528716+0000","last_clean":"2024-01-27T00:47:20.528716+0000","last_became_active":"2024-01-27T00:47:19.487984+0000","last_became_peered":"2024-01-27T00:47:19.487984+0000","last_unstale":"2024-01-27T00:47:20.528716+0000","last_undegraded":"2024-01-27T00:47:20.528716+0000","last_fullsized":"2024-01-27T00:47:20.528716+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:08:02.191122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.513175+0000","last_change":"2024-01-27T00:47:20.508823+0000","last_active":"2024-01-27T00:47:20.513175+0000","last_peered":"2024-01-27T00:47:20.513175+0000","last_clean":"2024-01-27T00:47:20.513175+0000","last_became_active":"2024-01-27T00:47:20.507955+0000","last_became_peered":"2024-01-27T00:47:20.507955+0000","last_unstale":"2024-01-27T00:47:20.513175+0000","last_undegraded":"2024-01-27T00:47:20.513175+0000","last_fullsized":"2024-01-27T00:47:20.513175+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:44.632841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531724+0000","last_change":"2024-01-27T00:47:19.493726+0000","last_active":"2024-01-27T00:47:20.531724+0000","last_peered":"2024-01-27T00:47:20.531724+0000","last_clean":"2024-01-27T00:47:20.531724+0000","last_became_active":"2024-01-27T00:47:19.493237+0000","last_became_peered":"2024-01-27T00:47:19.493237+0000","last_unstale":"2024-01-27T00:47:20.531724+0000","last_undegraded":"2024-01-27T00:47:20.531724+0000","last_fullsized":"2024-01-27T00:47:20.531724+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:33:12.701672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.532639+0000","last_change":"2024-01-27T00:47:20.516445+0000","last_active":"2024-01-27T00:47:20.532639+0000","last_peered":"2024-01-27T00:47:20.532639+0000","last_clean":"2024-01-27T00:47:20.532639+0000","last_became_active":"2024-01-27T00:47:20.515582+0000","last_became_peered":"2024-01-27T00:47:20.515582+0000","last_unstale":"2024-01-27T00:47:20.532639+0000","last_undegraded":"2024-01-27T00:47:20.532639+0000","last_fullsized":"2024-01-27T00:47:20.532639+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:07:16.136646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.507907+0000","last_change":"2024-01-27T00:47:19.505534+0000","last_active":"2024-01-27T00:47:20.507907+0000","last_peered":"2024-01-27T00:47:20.507907+0000","last_clean":"2024-01-27T00:47:20.507907+0000","last_became_active":"2024-01-27T00:47:19.504749+0000","last_became_peered":"2024-01-27T00:47:19.504749+0000","last_unstale":"2024-01-27T00:47:20.507907+0000","last_undegraded":"2024-01-27T00:47:20.507907+0000","last_fullsized":"2024-01-27T00:47:20.507907+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:48.314323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.492777+0000","last_change":"2024-01-27T00:47:19.488903+0000","last_active":"2024-01-27T00:47:20.492777+0000","last_peered":"2024-01-27T00:47:20.492777+0000","last_clean":"2024-01-27T00:47:20.492777+0000","last_became_active":"2024-01-27T00:47:19.488649+0000","last_became_peered":"2024-01-27T00:47:19.488649+0000","last_unstale":"2024-01-27T00:47:20.492777+0000","last_undegraded":"2024-01-27T00:47:20.492777+0000","last_fullsized":"2024-01-27T00:47:20.492777+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:36:32.698812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499880+0000","last_change":"2024-01-27T00:47:19.505276+0000","last_active":"2024-01-27T00:47:20.499880+0000","last_peered":"2024-01-27T00:47:20.499880+0000","last_clean":"2024-01-27T00:47:20.499880+0000","last_became_active":"2024-01-27T00:47:19.504824+0000","last_became_peered":"2024-01-27T00:47:19.504824+0000","last_unstale":"2024-01-27T00:47:20.499880+0000","last_undegraded":"2024-01-27T00:47:20.499880+0000","last_fullsized":"2024-01-27T00:47:20.499880+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:53:04.664266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.510043+0000","last_change":"2024-01-27T00:47:20.507243+0000","last_active":"2024-01-27T00:47:20.510043+0000","last_peered":"2024-01-27T00:47:20.510043+0000","last_clean":"2024-01-27T00:47:20.510043+0000","last_became_active":"2024-01-27T00:47:20.506953+0000","last_became_peered":"2024-01-27T00:47:20.506953+0000","last_unstale":"2024-01-27T00:47:20.510043+0000","last_undegraded":"2024-01-27T00:47:20.510043+0000","last_fullsized":"2024-01-27T00:47:20.510043+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:23:47.466789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.479792+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:44:20.701222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.530619+0000","last_change":"2024-01-27T00:47:20.516228+0000","last_active":"2024-01-27T00:47:20.530619+0000","last_peered":"2024-01-27T00:47:20.530619+0000","last_clean":"2024-01-27T00:47:20.530619+0000","last_became_active":"2024-01-27T00:47:20.515906+0000","last_became_peered":"2024-01-27T00:47:20.515906+0000","last_unstale":"2024-01-27T00:47:20.530619+0000","last_undegraded":"2024-01-27T00:47:20.530619+0000","last_fullsized":"2024-01-27T00:47:20.530619+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:12:20.214602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"187'6","reported_seq":16,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.534091+0000","last_change":"2024-01-27T00:47:19.502338+0000","last_active":"2024-01-27T00:47:20.534091+0000","last_peered":"2024-01-27T00:47:20.534091+0000","last_clean":"2024-01-27T00:47:20.534091+0000","last_became_active":"2024-01-27T00:47:19.502072+0000","last_became_peered":"2024-01-27T00:47:19.502072+0000","last_unstale":"2024-01-27T00:47:20.534091+0000","last_undegraded":"2024-01-27T00:47:20.534091+0000","last_fullsized":"2024-01-27T00:47:20.534091+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:34:16.405629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.13","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527814+0000","last_change":"2024-01-27T00:47:19.491092+0000","last_active":"2024-01-27T00:47:20.527814+0000","last_peered":"2024-01-27T00:47:20.527814+0000","last_clean":"2024-01-27T00:47:20.527814+0000","last_became_active":"2024-01-27T00:47:19.490799+0000","last_became_peered":"2024-01-27T00:47:19.490799+0000","last_unstale":"2024-01-27T00:47:20.527814+0000","last_undegraded":"2024-01-27T00:47:20.527814+0000","last_fullsized":"2024-01-27T00:47:20.527814+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:23:46.462278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527448+0000","last_change":"2024-01-27T00:47:20.515730+0000","last_active":"2024-01-27T00:47:20.527448+0000","last_peered":"2024-01-27T00:47:20.527448+0000","last_clean":"2024-01-27T00:47:20.527448+0000","last_became_active":"2024-01-27T00:47:20.515422+0000","last_became_peered":"2024-01-27T00:47:20.515422+0000","last_unstale":"2024-01-27T00:47:20.527448+0000","last_undegraded":"2024-01-27T00:47:20.527448+0000","last_fullsized":"2024-01-27T00:47:20.527448+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:33.654223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.498821+0000","last_change":"2024-01-27T00:47:19.499915+0000","last_active":"2024-01-27T00:47:20.498821+0000","last_peered":"2024-01-27T00:47:20.498821+0000","last_clean":"2024-01-27T00:47:20.498821+0000","last_became_active":"2024-01-27T00:47:19.499418+0000","last_became_peered":"2024-01-27T00:47:19.499418+0000","last_unstale":"2024-01-27T00:47:20.498821+0000","last_undegraded":"2024-01-27T00:47:20.498821+0000","last_fullsized":"2024-01-27T00:47:20.498821+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:46.642397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.482438+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:43.628330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.514859+0000","last_change":"2024-01-27T00:47:20.513843+0000","last_active":"2024-01-27T00:47:20.514859+0000","last_peered":"2024-01-27T00:47:20.514859+0000","last_clean":"2024-01-27T00:47:20.514859+0000","last_became_active":"2024-01-27T00:47:20.513188+0000","last_became_peered":"2024-01-27T00:47:20.513188+0000","last_unstale":"2024-01-27T00:47:20.514859+0000","last_undegraded":"2024-01-27T00:47:20.514859+0000","last_fullsized":"2024-01-27T00:47:20.514859+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:15:01.330526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"187'8","reported_seq":18,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.529029+0000","last_change":"2024-01-27T00:47:19.491898+0000","last_active":"2024-01-27T00:47:20.529029+0000","last_peered":"2024-01-27T00:47:20.529029+0000","last_clean":"2024-01-27T00:47:20.529029+0000","last_became_active":"2024-01-27T00:47:19.491597+0000","last_became_peered":"2024-01-27T00:47:19.491597+0000","last_unstale":"2024-01-27T00:47:20.529029+0000","last_undegraded":"2024-01-27T00:47:20.529029+0000","last_fullsized":"2024-01-27T00:47:20.529029+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:47:33.870316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524260+0000","last_change":"2024-01-27T00:47:20.511230+0000","last_active":"2024-01-27T00:47:20.524260+0000","last_peered":"2024-01-27T00:47:20.524260+0000","last_clean":"2024-01-27T00:47:20.524260+0000","last_became_active":"2024-01-27T00:47:20.510893+0000","last_became_peered":"2024-01-27T00:47:20.510893+0000","last_unstale":"2024-01-27T00:47:20.524260+0000","last_undegraded":"2024-01-27T00:47:20.524260+0000","last_fullsized":"2024-01-27T00:47:20.524260+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:51:40.896423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531660+0000","last_change":"2024-01-27T00:47:19.504875+0000","last_active":"2024-01-27T00:47:20.531660+0000","last_peered":"2024-01-27T00:47:20.531660+0000","last_clean":"2024-01-27T00:47:20.531660+0000","last_became_active":"2024-01-27T00:47:19.504575+0000","last_became_peered":"2024-01-27T00:47:19.504575+0000","last_unstale":"2024-01-27T00:47:20.531660+0000","last_undegraded":"2024-01-27T00:47:20.531660+0000","last_fullsized":"2024-01-27T00:47:20.531660+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:43:03.734740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.e","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.533275+0000","last_change":"2024-01-27T00:47:19.509498+0000","last_active":"2024-01-27T00:47:20.533275+0000","last_peered":"2024-01-27T00:47:20.533275+0000","last_clean":"2024-01-27T00:47:20.533275+0000","last_became_active":"2024-01-27T00:47:19.509151+0000","last_became_peered":"2024-01-27T00:47:19.509151+0000","last_unstale":"2024-01-27T00:47:20.533275+0000","last_undegraded":"2024-01-27T00:47:20.533275+0000","last_fullsized":"2024-01-27T00:47:20.533275+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:54:21.075786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.d","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.529256+0000","last_change":"2024-01-27T00:47:19.498982+0000","last_active":"2024-01-27T00:47:20.529256+0000","last_peered":"2024-01-27T00:47:20.529256+0000","last_clean":"2024-01-27T00:47:20.529256+0000","last_became_active":"2024-01-27T00:47:19.498718+0000","last_became_peered":"2024-01-27T00:47:19.498718+0000","last_unstale":"2024-01-27T00:47:20.529256+0000","last_undegraded":"2024-01-27T00:47:20.529256+0000","last_fullsized":"2024-01-27T00:47:20.529256+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:04:42.913017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.493212+0000","last_change":"2024-01-27T00:47:19.493973+0000","last_active":"2024-01-27T00:47:20.493212+0000","last_peered":"2024-01-27T00:47:20.493212+0000","last_clean":"2024-01-27T00:47:20.493212+0000","last_became_active":"2024-01-27T00:47:19.493685+0000","last_became_peered":"2024-01-27T00:47:19.493685+0000","last_unstale":"2024-01-27T00:47:20.493212+0000","last_undegraded":"2024-01-27T00:47:20.493212+0000","last_fullsized":"2024-01-27T00:47:20.493212+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:08:00.978209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531595+0000","last_change":"2024-01-27T00:47:20.525394+0000","last_active":"2024-01-27T00:47:20.531595+0000","last_peered":"2024-01-27T00:47:20.531595+0000","last_clean":"2024-01-27T00:47:20.531595+0000","last_became_active":"2024-01-27T00:47:20.521446+0000","last_became_peered":"2024-01-27T00:47:20.521446+0000","last_unstale":"2024-01-27T00:47:20.531595+0000","last_undegraded":"2024-01-27T00:47:20.531595+0000","last_fullsized":"2024-01-27T00:47:20.531595+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:31.735208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.529633+0000","last_change":"2024-01-27T00:47:19.495677+0000","last_active":"2024-01-27T00:47:20.529633+0000","last_peered":"2024-01-27T00:47:20.529633+0000","last_clean":"2024-01-27T00:47:20.529633+0000","last_became_active":"2024-01-27T00:47:19.494329+0000","last_became_peered":"2024-01-27T00:47:19.494329+0000","last_unstale":"2024-01-27T00:47:20.529633+0000","last_undegraded":"2024-01-27T00:47:20.529633+0000","last_fullsized":"2024-01-27T00:47:20.529633+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:19:25.040151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522962+0000","last_change":"2024-01-27T00:47:20.512100+0000","last_active":"2024-01-27T00:47:20.522962+0000","last_peered":"2024-01-27T00:47:20.522962+0000","last_clean":"2024-01-27T00:47:20.522962+0000","last_became_active":"2024-01-27T00:47:20.511807+0000","last_became_peered":"2024-01-27T00:47:20.511807+0000","last_unstale":"2024-01-27T00:47:20.522962+0000","last_undegraded":"2024-01-27T00:47:20.522962+0000","last_fullsized":"2024-01-27T00:47:20.522962+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:41:19.485849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.482501+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:10:10.727612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522508+0000","last_change":"2024-01-27T00:47:20.512072+0000","last_active":"2024-01-27T00:47:20.522508+0000","last_peered":"2024-01-27T00:47:20.522508+0000","last_clean":"2024-01-27T00:47:20.522508+0000","last_became_active":"2024-01-27T00:47:20.511716+0000","last_became_peered":"2024-01-27T00:47:20.511716+0000","last_unstale":"2024-01-27T00:47:20.522508+0000","last_undegraded":"2024-01-27T00:47:20.522508+0000","last_fullsized":"2024-01-27T00:47:20.522508+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:18:15.805971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.508031+0000","last_change":"2024-01-27T00:47:19.505685+0000","last_active":"2024-01-27T00:47:20.508031+0000","last_peered":"2024-01-27T00:47:20.508031+0000","last_clean":"2024-01-27T00:47:20.508031+0000","last_became_active":"2024-01-27T00:47:19.505201+0000","last_became_peered":"2024-01-27T00:47:19.505201+0000","last_unstale":"2024-01-27T00:47:20.508031+0000","last_undegraded":"2024-01-27T00:47:20.508031+0000","last_fullsized":"2024-01-27T00:47:20.508031+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:10:58.917172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.481168+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:38:22.205934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.7","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527493+0000","last_change":"2024-01-27T00:47:19.498236+0000","last_active":"2024-01-27T00:47:20.527493+0000","last_peered":"2024-01-27T00:47:20.527493+0000","last_clean":"2024-01-27T00:47:20.527493+0000","last_became_active":"2024-01-27T00:47:19.497955+0000","last_became_peered":"2024-01-27T00:47:19.497955+0000","last_unstale":"2024-01-27T00:47:20.527493+0000","last_undegraded":"2024-01-27T00:47:20.527493+0000","last_fullsized":"2024-01-27T00:47:20.527493+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:52:03.781390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524123+0000","last_change":"2024-01-27T00:47:20.510553+0000","last_active":"2024-01-27T00:47:20.524123+0000","last_peered":"2024-01-27T00:47:20.524123+0000","last_clean":"2024-01-27T00:47:20.524123+0000","last_became_active":"2024-01-27T00:47:20.510231+0000","last_became_peered":"2024-01-27T00:47:20.510231+0000","last_unstale":"2024-01-27T00:47:20.524123+0000","last_undegraded":"2024-01-27T00:47:20.524123+0000","last_fullsized":"2024-01-27T00:47:20.524123+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:15.349063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499847+0000","last_change":"2024-01-27T00:47:19.505767+0000","last_active":"2024-01-27T00:47:20.499847+0000","last_peered":"2024-01-27T00:47:20.499847+0000","last_clean":"2024-01-27T00:47:20.499847+0000","last_became_active":"2024-01-27T00:47:19.505313+0000","last_became_peered":"2024-01-27T00:47:19.505313+0000","last_unstale":"2024-01-27T00:47:20.499847+0000","last_undegraded":"2024-01-27T00:47:20.499847+0000","last_fullsized":"2024-01-27T00:47:20.499847+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:48:00.955590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.530928+0000","last_change":"2024-01-27T00:47:20.516018+0000","last_active":"2024-01-27T00:47:20.530928+0000","last_peered":"2024-01-27T00:47:20.530928+0000","last_clean":"2024-01-27T00:47:20.530928+0000","last_became_active":"2024-01-27T00:47:20.515703+0000","last_became_peered":"2024-01-27T00:47:20.515703+0000","last_unstale":"2024-01-27T00:47:20.530928+0000","last_undegraded":"2024-01-27T00:47:20.530928+0000","last_fullsized":"2024-01-27T00:47:20.530928+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:12:19.305838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499639+0000","last_change":"2024-01-27T00:47:19.499255+0000","last_active":"2024-01-27T00:47:20.499639+0000","last_peered":"2024-01-27T00:47:20.499639+0000","last_clean":"2024-01-27T00:47:20.499639+0000","last_became_active":"2024-01-27T00:47:19.498933+0000","last_became_peered":"2024-01-27T00:47:19.498933+0000","last_unstale":"2024-01-27T00:47:20.499639+0000","last_undegraded":"2024-01-27T00:47:20.499639+0000","last_fullsized":"2024-01-27T00:47:20.499639+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:45.555078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524255+0000","last_change":"2024-01-27T00:47:20.511077+0000","last_active":"2024-01-27T00:47:20.524255+0000","last_peered":"2024-01-27T00:47:20.524255+0000","last_clean":"2024-01-27T00:47:20.524255+0000","last_became_active":"2024-01-27T00:47:20.510732+0000","last_became_peered":"2024-01-27T00:47:20.510732+0000","last_unstale":"2024-01-27T00:47:20.524255+0000","last_undegraded":"2024-01-27T00:47:20.524255+0000","last_fullsized":"2024-01-27T00:47:20.524255+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:05.094009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.508098+0000","last_change":"2024-01-27T00:47:19.504494+0000","last_active":"2024-01-27T00:47:20.508098+0000","last_peered":"2024-01-27T00:47:20.508098+0000","last_clean":"2024-01-27T00:47:20.508098+0000","last_became_active":"2024-01-27T00:47:19.504090+0000","last_became_peered":"2024-01-27T00:47:19.504090+0000","last_unstale":"2024-01-27T00:47:20.508098+0000","last_undegraded":"2024-01-27T00:47:20.508098+0000","last_fullsized":"2024-01-27T00:47:20.508098+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:22:59.518350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.479789+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:07.269026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.498924+0000","last_change":"2024-01-27T00:47:19.503541+0000","last_active":"2024-01-27T00:47:20.498924+0000","last_peered":"2024-01-27T00:47:20.498924+0000","last_clean":"2024-01-27T00:47:20.498924+0000","last_became_active":"2024-01-27T00:47:19.503073+0000","last_became_peered":"2024-01-27T00:47:19.503073+0000","last_unstale":"2024-01-27T00:47:20.498924+0000","last_undegraded":"2024-01-27T00:47:20.498924+0000","last_fullsized":"2024-01-27T00:47:20.498924+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:14.344552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531950+0000","last_change":"2024-01-27T00:47:20.527327+0000","last_active":"2024-01-27T00:47:20.531950+0000","last_peered":"2024-01-27T00:47:20.531950+0000","last_clean":"2024-01-27T00:47:20.531950+0000","last_became_active":"2024-01-27T00:47:20.520248+0000","last_became_peered":"2024-01-27T00:47:20.520248+0000","last_unstale":"2024-01-27T00:47:20.531950+0000","last_undegraded":"2024-01-27T00:47:20.531950+0000","last_fullsized":"2024-01-27T00:47:20.531950+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:19:26.044662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T00:47:18.469785+0000","last_change":"2024-01-27T00:47:18.482920+0000","last_active":"2024-01-27T00:47:18.469785+0000","last_peered":"2024-01-27T00:47:18.469785+0000","last_clean":"2024-01-27T00:47:18.469785+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:18.469785+0000","last_undegraded":"2024-01-27T00:47:18.469785+0000","last_fullsized":"2024-01-27T00:47:18.469785+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:20:46.023087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.532846+0000","last_change":"2024-01-27T00:47:20.516514+0000","last_active":"2024-01-27T00:47:20.532846+0000","last_peered":"2024-01-27T00:47:20.532846+0000","last_clean":"2024-01-27T00:47:20.532846+0000","last_became_active":"2024-01-27T00:47:20.516160+0000","last_became_peered":"2024-01-27T00:47:20.516160+0000","last_unstale":"2024-01-27T00:47:20.532846+0000","last_undegraded":"2024-01-27T00:47:20.532846+0000","last_fullsized":"2024-01-27T00:47:20.532846+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:36:41.113078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.497052+0000","last_change":"2024-01-27T00:47:19.493487+0000","last_active":"2024-01-27T00:47:20.497052+0000","last_peered":"2024-01-27T00:47:20.497052+0000","last_clean":"2024-01-27T00:47:20.497052+0000","last_became_active":"2024-01-27T00:47:19.492721+0000","last_became_peered":"2024-01-27T00:47:19.492721+0000","last_unstale":"2024-01-27T00:47:20.497052+0000","last_undegraded":"2024-01-27T00:47:20.497052+0000","last_fullsized":"2024-01-27T00:47:20.497052+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:30.730697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T00:47:19.474295+0000","last_change":"2024-01-27T00:47:19.474295+0000","last_active":"2024-01-27T00:47:19.474295+0000","last_peered":"2024-01-27T00:47:19.474295+0000","last_clean":"2024-01-27T00:47:19.474295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T00:47:19.474295+0000","last_undegraded":"2024-01-27T00:47:19.474295+0000","last_fullsized":"2024-01-27T00:47:19.474295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499725+0000","last_change":"2024-01-27T00:47:19.501570+0000","last_active":"2024-01-27T00:47:20.499725+0000","last_peered":"2024-01-27T00:47:20.499725+0000","last_clean":"2024-01-27T00:47:20.499725+0000","last_became_active":"2024-01-27T00:47:19.501187+0000","last_became_peered":"2024-01-27T00:47:19.501187+0000","last_unstale":"2024-01-27T00:47:20.499725+0000","last_undegraded":"2024-01-27T00:47:20.499725+0000","last_fullsized":"2024-01-27T00:47:20.499725+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:54:58.914033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"179'8695","reported_seq":9433,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-01-27T00:47:17.003489+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T00:47:17.003489+0000","last_peered":"2024-01-27T00:47:17.003489+0000","last_clean":"2024-01-27T00:47:17.003489+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T00:47:17.003489+0000","last_undegraded":"2024-01-27T00:47:17.003489+0000","last_fullsized":"2024-01-27T00:47:17.003489+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8695,"log_dups_size":0,"ondisk_log_size":8695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1784,"num_read_kb":3778,"num_write":530,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":1662,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1784,"num_read_kb":3778,"num_write":530,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8695,"ondisk_log_size":8695,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316504,"kb_used_data":2832,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055336,"statfs":{"total":96636764160,"available":96312664064,"internally_reserved":0,"allocated":2899968,"data_stored":1787469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60799999999999998,"15min":0.60699999999999998},"min":{"1min":0.36199999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75800000000000001,"5min":0.83199999999999996,"15min":0.86899999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61299999999999999,"15min":0.61799999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.755,"5min":0.82099999999999995,"15min":0.90000000000000002},"last":0.61299999999999999}]},{"osd":1,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.34599999999999997},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.90400000000000003},"last":0.627},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63500000000000001,"15min":0.63},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.38200000000000001},"max":{"1min":0.76700000000000002,"5min":0.85099999999999998,"15min":0.875},"last":0.54900000000000004}]},{"osd":2,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.374,"5min":0.36299999999999999,"15min":0.34000000000000002},"max":{"1min":0.76400000000000001,"5min":0.77700000000000002,"15min":0.89700000000000002},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63800000000000001,"15min":0.63700000000000001},"min":{"1min":0.49399999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75600000000000001,"5min":0.86199999999999999,"15min":0.94199999999999995},"last":0.57499999999999996}]},{"osd":3,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.626,"15min":0.623},"min":{"1min":0.435,"5min":0.40999999999999998,"15min":0.39400000000000002},"max":{"1min":0.79200000000000004,"5min":0.86499999999999999,"15min":0.92600000000000005},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.77700000000000002,"5min":0.83499999999999996,"15min":0.88400000000000001},"last":0.67100000000000004}]},{"osd":4,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49399999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.32900000000000001,"15min":0.29699999999999999},"max":{"1min":0.61699999999999999,"5min":0.86499999999999999,"15min":0.92000000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.34599999999999997},"max":{"1min":0.76600000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.42499999999999999}]},{"osd":5,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":0.67100000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":0.67700000000000005,"5min":0.89100000000000001,"15min":0.95699999999999996},"last":0.45800000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59299999999999997,"15min":0.58299999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.36599999999999999},"max":{"1min":0.69299999999999995,"5min":0.91700000000000004,"15min":0.99399999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.36699999999999999},"max":{"1min":0.73599999999999999,"5min":0.82299999999999995,"15min":0.878},"last":0.49199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575775,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307732,"kb_used_data":2252,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064108,"statfs":{"total":96636764160,"available":96321646592,"internally_reserved":0,"allocated":2306048,"data_stored":1197101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58999999999999997,"15min":0.60399999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81000000000000005,"5min":0.82199999999999995,"15min":0.84899999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.38900000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.78500000000000003,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.55700000000000005}]},{"osd":1,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.748,"5min":0.79800000000000004,"15min":0.80800000000000005},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81499999999999995,"5min":0.87,"15min":0.87},"last":0.49299999999999999}]},{"osd":2,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60199999999999998,"15min":0.60899999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83699999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59899999999999998,"15min":0.60599999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74199999999999999,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.50600000000000001}]},{"osd":3,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86299999999999999,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64600000000000002,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.63400000000000001}]},{"osd":4,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.496,"15min":0.49099999999999999},"min":{"1min":0.34300000000000003,"5min":0.33200000000000002,"15min":0.313},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32100000000000001},"max":{"1min":0.77700000000000002,"5min":0.79100000000000004,"15min":0.83199999999999996},"last":0.39300000000000002}]},{"osd":5,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51900000000000002,"15min":0.51000000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.32200000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.375},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56499999999999995,"15min":0.56000000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.88800000000000001},"last":0.40999999999999998}]},{"osd":7,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57399999999999995,"15min":0.56499999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.58399999999999996,"15min":0.58099999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.33600000000000002},"max":{"1min":0.84099999999999997,"5min":0.84599999999999997,"15min":0.85999999999999999},"last":0.434}]}]},{"osd":1,"up_from":13,"seq":55834575776,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311888,"kb_used_data":2312,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059952,"statfs":{"total":96636764160,"available":96317390848,"internally_reserved":0,"allocated":2367488,"data_stored":1226772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.41799999999999998},"min":{"1min":0.311,"5min":0.254,"15min":0.23499999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.434,"5min":0.441,"15min":0.44500000000000001},"min":{"1min":0.30199999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.60599999999999998,"5min":0.66200000000000003,"15min":0.70199999999999996},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.41899999999999998,"15min":0.42499999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55100000000000005,"5min":0.59999999999999998,"15min":0.68899999999999995},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48499999999999999,"15min":0.49399999999999999},"min":{"1min":0.33500000000000002,"5min":0.31,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.76700000000000002},"last":0.48499999999999999}]},{"osd":3,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.503,"15min":0.52100000000000002},"min":{"1min":0.39400000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.63900000000000001,"5min":0.69399999999999995,"15min":1.0469999999999999},"last":0.621},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.49099999999999999,"15min":0.5},"min":{"1min":0.318,"5min":0.318,"15min":0.309},"max":{"1min":0.69599999999999995,"5min":0.72599999999999998,"15min":0.95599999999999996},"last":0.56899999999999995}]},{"osd":4,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.66100000000000003,"15min":0.67900000000000005},"min":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86299999999999999,"5min":1.02,"15min":1.3129999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67700000000000005,"15min":0.69999999999999996},"min":{"1min":0.505,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.93500000000000005,"5min":1.0069999999999999,"15min":1.097},"last":0.68500000000000005}]},{"osd":5,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66700000000000004,"15min":0.69199999999999995},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.46300000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":1.091},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.36299999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.278},"last":0.64600000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.68500000000000005,"15min":0.71099999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.435},"max":{"1min":0.88100000000000001,"5min":0.96999999999999997,"15min":1.141},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70799999999999996,"15min":0.72099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.91400000000000003,"5min":1.0229999999999999,"15min":1.1120000000000001},"last":0.77900000000000003}]},{"osd":7,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.73299999999999998,"15min":0.749},"min":{"1min":0.52600000000000002,"5min":0.52500000000000002,"15min":0.51100000000000001},"max":{"1min":1.0129999999999999,"5min":1.054,"15min":1.1819999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.73499999999999999,"15min":0.75800000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.51200000000000001},"max":{"1min":0.98299999999999998,"5min":0.98999999999999999,"15min":1.198},"last":0.71199999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575775,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312464,"kb_used_data":2888,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059376,"statfs":{"total":96636764160,"available":96316801024,"internally_reserved":0,"allocated":2957312,"data_stored":1815708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.28299999999999997,"5min":0.25800000000000001,"15min":0.25},"max":{"1min":0.55500000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.34499999999999997,"5min":0.27500000000000002,"15min":0.26300000000000001},"max":{"1min":0.61299999999999999,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.56999999999999995}]},{"osd":2,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.42099999999999999,"15min":0.42599999999999999},"min":{"1min":0.312,"5min":0.28599999999999998,"15min":0.26700000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.69399999999999995},"last":0.621},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48099999999999998,"15min":0.48299999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.629}]},{"osd":3,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48999999999999999,"15min":0.48699999999999999},"min":{"1min":0.32900000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.65600000000000003,"5min":0.755,"15min":0.755},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.496,"15min":0.498},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.42199999999999999}]},{"osd":4,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65200000000000002,"15min":0.68100000000000005},"min":{"1min":0.45100000000000001,"5min":0.44700000000000001,"15min":0.44400000000000001},"max":{"1min":0.77500000000000002,"5min":0.97099999999999997,"15min":1.125},"last":0.496},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.66000000000000003,"15min":0.67800000000000005},"min":{"1min":0.46200000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.96199999999999997,"5min":0.97599999999999998,"15min":1.0169999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68100000000000005,"15min":0.69399999999999995},"min":{"1min":0.47899999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.84099999999999997,"5min":0.94199999999999995,"15min":1.054},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.70799999999999996,"15min":0.71899999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.98199999999999998,"5min":1.056,"15min":1.056},"last":0.46200000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69999999999999996,"15min":0.71199999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.84399999999999997,"5min":1.0269999999999999,"15min":1.087},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.72299999999999998,"15min":0.73899999999999999},"min":{"1min":0.52500000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.94599999999999995,"5min":1.0349999999999999,"15min":1.052},"last":0.88400000000000001}]},{"osd":7,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.747,"15min":0.75700000000000001},"min":{"1min":0.60599999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.016,"5min":1.0880000000000001,"15min":1.1080000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.53700000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.034,"5min":1.071,"15min":1.071},"last":0.55400000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575775,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311876,"kb_used_data":2300,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059964,"statfs":{"total":96636764160,"available":96317403136,"internally_reserved":0,"allocated":2355200,"data_stored":1225236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38100000000000001,"15min":0.39800000000000002},"min":{"1min":0.29499999999999998,"5min":0.25900000000000001,"15min":0.252},"max":{"1min":0.48699999999999999,"5min":0.60499999999999998,"15min":0.65800000000000003},"last":0.375},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":0.438},"min":{"1min":0.29499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.71899999999999997},"last":0.35599999999999998}]},{"osd":1,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.41999999999999998,"15min":0.432},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.60799999999999998,"5min":0.627,"15min":0.68300000000000005},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.5,"15min":0.51400000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66100000000000003,"5min":0.76400000000000001,"15min":0.80300000000000005},"last":0.40400000000000003}]},{"osd":3,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47799999999999998,"15min":0.49199999999999999},"min":{"1min":0.34999999999999998,"5min":0.34100000000000003,"15min":0.33100000000000002},"max":{"1min":0.65200000000000002,"5min":0.745,"15min":0.745},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47099999999999997,"15min":0.48699999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.29499999999999998},"max":{"1min":0.68000000000000005,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.44900000000000001}]},{"osd":4,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.65000000000000002,"15min":0.66700000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41999999999999998},"max":{"1min":0.77000000000000002,"5min":0.85899999999999999,"15min":1.0660000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66700000000000004,"15min":0.68200000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92400000000000004,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.83999999999999997}]},{"osd":5,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66800000000000004,"15min":0.68999999999999995},"min":{"1min":0.49099999999999999,"5min":0.46500000000000002,"15min":0.42899999999999999},"max":{"1min":0.78400000000000003,"5min":0.96499999999999997,"15min":1.0649999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68200000000000005,"15min":0.69799999999999995},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.90600000000000003,"5min":0.98799999999999999,"15min":1.044},"last":0.77600000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.68100000000000005,"15min":0.70099999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.86299999999999999,"5min":1.0569999999999999,"15min":1.1140000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.69399999999999995,"15min":0.70999999999999996},"min":{"1min":0.47999999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89400000000000002,"5min":1.002,"15min":1.0720000000000001},"last":0.79000000000000004}]},{"osd":7,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.71499999999999997,"15min":0.72799999999999998},"min":{"1min":0.55600000000000005,"5min":0.50900000000000001,"15min":0.48999999999999999},"max":{"1min":0.97499999999999998,"5min":1.0860000000000001,"15min":1.1479999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.73899999999999999,"15min":0.76200000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.98899999999999999,"5min":1.022,"15min":1.1299999999999999},"last":0.82799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575774,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315992,"kb_used_data":2320,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055848,"statfs":{"total":96636764160,"available":96313188352,"internally_reserved":0,"allocated":2375680,"data_stored":1227348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.39600000000000002,"15min":0.40100000000000002},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.50600000000000001,"5min":0.63,"15min":0.752},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.443,"5min":0.45700000000000002,"15min":0.45300000000000001},"min":{"1min":0.32900000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.57899999999999996,"5min":0.68799999999999994,"15min":0.85699999999999998},"last":0.69599999999999995}]},{"osd":1,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.46600000000000003},"min":{"1min":0.307,"5min":0.29699999999999999,"15min":0.27400000000000002},"max":{"1min":0.61499999999999999,"5min":0.69699999999999995,"15min":0.94599999999999995},"last":0.627},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.498,"15min":0.498},"min":{"1min":0.36299999999999999,"5min":0.34399999999999997,"15min":0.316},"max":{"1min":0.69299999999999995,"5min":0.745,"15min":0.745},"last":0.72999999999999998}]},{"osd":2,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.35699999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64900000000000002,"5min":0.67900000000000005,"15min":0.84499999999999997},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.46000000000000002,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.66600000000000004,"5min":0.72699999999999998,"15min":0.82299999999999995},"last":0.80400000000000005}]},{"osd":4,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.69299999999999995,"15min":0.68999999999999995},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.88400000000000001,"5min":0.89300000000000002,"15min":1},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.48199999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.98699999999999999,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.89700000000000002}]},{"osd":5,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.69299999999999995,"15min":0.68799999999999994},"min":{"1min":0.505,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.88900000000000001,"5min":0.94499999999999995,"15min":1.0429999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68400000000000005,"15min":0.68700000000000006},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90400000000000003,"5min":0.99099999999999999,"15min":1.0289999999999999},"last":1}]},{"osd":6,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.68300000000000005,"15min":0.69099999999999995},"min":{"1min":0.53000000000000003,"5min":0.52000000000000002,"15min":0.48899999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.0820000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.71799999999999997,"15min":0.71599999999999997},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.44},"max":{"1min":0.91600000000000004,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.95199999999999996}]},{"osd":7,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.69999999999999996,"15min":0.71399999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.96399999999999997,"5min":1.081,"15min":1.081},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.55800000000000005,"5min":0.48999999999999999,"15min":0.46300000000000002},"max":{"1min":0.91700000000000004,"5min":1.0229999999999999,"15min":1.1240000000000001},"last":0.98999999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575775,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311828,"kb_used_data":2252,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060012,"statfs":{"total":96636764160,"available":96317452288,"internally_reserved":0,"allocated":2306048,"data_stored":1197101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61399999999999999,"15min":0.60599999999999998},"min":{"1min":0.40699999999999997,"5min":0.39300000000000002,"15min":0.32100000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59299999999999997}]},{"osd":1,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.627,"15min":0.628},"min":{"1min":0.52900000000000003,"5min":0.42499999999999999,"15min":0.40999999999999998},"max":{"1min":0.75700000000000001,"5min":0.83199999999999996,"15min":0.86299999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61899999999999999,"15min":0.626},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84699999999999998},"last":0.56399999999999995}]},{"osd":2,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.46100000000000002,"5min":0.436,"15min":0.40899999999999997},"max":{"1min":0.753,"5min":0.81899999999999995,"15min":0.84499999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.5,"5min":0.44400000000000001,"15min":0.38100000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.41799999999999998}]},{"osd":3,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.626,"15min":0.627},"min":{"1min":0.45900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.61899999999999999,"15min":0.623},"min":{"1min":0.42599999999999999,"5min":0.41199999999999998,"15min":0.41099999999999998},"max":{"1min":0.80600000000000005,"5min":0.80900000000000005,"15min":0.876},"last":0.76800000000000002}]},{"osd":5,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48199999999999998,"15min":0.48499999999999999},"min":{"1min":0.35899999999999999,"5min":0.32100000000000001,"15min":0.28699999999999998},"max":{"1min":0.69699999999999995,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.33100000000000002,"5min":0.30499999999999999,"15min":0.30199999999999999},"max":{"1min":0.77300000000000002,"5min":0.876,"15min":0.876},"last":0.65800000000000003}]},{"osd":6,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49299999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.54100000000000004},"min":{"1min":0.39800000000000002,"5min":0.34300000000000003,"15min":0.32800000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.67500000000000004}]},{"osd":7,"last update":"Sat Jan 27 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41599999999999998,"5min":0.34999999999999998,"15min":0.33700000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.625},{"interface":"front","average":{"1min":0.627,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.46000000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.80400000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575775,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311828,"kb_used_data":2252,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060012,"statfs":{"total":96636764160,"available":96317452288,"internally_reserved":0,"allocated":2306048,"data_stored":1197101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61399999999999999,"15min":0.60799999999999998},"min":{"1min":0.379,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76200000000000001,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.61599999999999999,"15min":0.623},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38600000000000001},"max":{"1min":0.76600000000000001,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.46800000000000003}]},{"osd":1,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65900000000000003,"15min":0.64900000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66400000000000003,"15min":0.67100000000000004},"min":{"1min":0.44600000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80300000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.60499999999999998}]},{"osd":2,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.61399999999999999,"15min":0.60299999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.71999999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.79100000000000004,"5min":0.876,"15min":0.876},"last":0.41399999999999998}]},{"osd":3,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60399999999999998,"15min":0.60299999999999998},"min":{"1min":0.38200000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.70099999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.64200000000000002,"15min":0.63400000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.79600000000000004,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.64900000000000002}]},{"osd":4,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.47499999999999998,"15min":0.47599999999999998},"min":{"1min":0.307,"5min":0.30399999999999999,"15min":0.30299999999999999},"max":{"1min":0.67600000000000005,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.82799999999999996},"last":0.71699999999999997}]},{"osd":6,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50700000000000001,"15min":0.504},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.31900000000000001},"max":{"1min":0.67400000000000004,"5min":0.71399999999999997,"15min":0.79900000000000004},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.379,"5min":0.36799999999999999,"15min":0.36099999999999999},"max":{"1min":0.70099999999999996,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.63500000000000001}]},{"osd":7,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34699999999999998},"max":{"1min":0.747,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.45800000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.69999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T00:47:25.524 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.3f is not active+clean 2024-01-27T00:47:25.524 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.524 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.3e is not active+clean 2024-01-27T00:47:25.524 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.524 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.3d is not active+clean 2024-01-27T00:47:25.524 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.525 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.3c is not active+clean 2024-01-27T00:47:25.525 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.525 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.3a is not active+clean 2024-01-27T00:47:25.525 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.525 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.35 is not active+clean 2024-01-27T00:47:25.525 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.525 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.34 is not active+clean 2024-01-27T00:47:25.525 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.525 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.31 is not active+clean 2024-01-27T00:47:25.525 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.525 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.30 is not active+clean 2024-01-27T00:47:25.526 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.526 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.2e is not active+clean 2024-01-27T00:47:25.526 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.526 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.2d is not active+clean 2024-01-27T00:47:25.526 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.526 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.2a is not active+clean 2024-01-27T00:47:25.526 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.526 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.28 is not active+clean 2024-01-27T00:47:25.526 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.526 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.25 is not active+clean 2024-01-27T00:47:25.526 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.527 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.23 is not active+clean 2024-01-27T00:47:25.527 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.527 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.22 is not active+clean 2024-01-27T00:47:25.527 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.527 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.21 is not active+clean 2024-01-27T00:47:25.527 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.527 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.20 is not active+clean 2024-01-27T00:47:25.527 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.527 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.10 is not active+clean 2024-01-27T00:47:25.527 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.528 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.1f is not active+clean 2024-01-27T00:47:25.528 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T00:47:18.469785+0000', 'last_change': '2024-01-27T00:47:18.480155+0000', 'last_active': '2024-01-27T00:47:18.469785+0000', 'last_peered': '2024-01-27T00:47:18.469785+0000', 'last_clean': '2024-01-27T00:47:18.469785+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:18.469785+0000', 'last_undegraded': '2024-01-27T00:47:18.469785+0000', 'last_fullsized': '2024-01-27T00:47:18.469785+0000', 'mapping_epoch': 185, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 185, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T12:27:44.775426+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-27T00:47:25.528 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.1d is not active+clean 2024-01-27T00:47:25.528 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T00:47:18.469785+0000', 'last_change': '2024-01-27T00:47:18.482562+0000', 'last_active': '2024-01-27T00:47:18.469785+0000', 'last_peered': '2024-01-27T00:47:18.469785+0000', 'last_clean': '2024-01-27T00:47:18.469785+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:18.469785+0000', 'last_undegraded': '2024-01-27T00:47:18.469785+0000', 'last_fullsized': '2024-01-27T00:47:18.469785+0000', 'mapping_epoch': 185, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 185, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T07:43:05.614968+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-27T00:47:25.528 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.13 is not active+clean 2024-01-27T00:47:25.528 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.528 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.17 is not active+clean 2024-01-27T00:47:25.528 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.528 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.18 is not active+clean 2024-01-27T00:47:25.528 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.15 is not active+clean 2024-01-27T00:47:25.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T00:47:18.469785+0000', 'last_change': '2024-01-27T00:47:18.479792+0000', 'last_active': '2024-01-27T00:47:18.469785+0000', 'last_peered': '2024-01-27T00:47:18.469785+0000', 'last_clean': '2024-01-27T00:47:18.469785+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:18.469785+0000', 'last_undegraded': '2024-01-27T00:47:18.469785+0000', 'last_fullsized': '2024-01-27T00:47:18.469785+0000', 'mapping_epoch': 185, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 185, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T02:44:20.701222+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-27T00:47:25.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.1c is not active+clean 2024-01-27T00:47:25.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.1e is not active+clean 2024-01-27T00:47:25.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.11 is not active+clean 2024-01-27T00:47:25.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T00:47:18.469785+0000', 'last_change': '2024-01-27T00:47:18.482438+0000', 'last_active': '2024-01-27T00:47:18.469785+0000', 'last_peered': '2024-01-27T00:47:18.469785+0000', 'last_clean': '2024-01-27T00:47:18.469785+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:18.469785+0000', 'last_undegraded': '2024-01-27T00:47:18.469785+0000', 'last_fullsized': '2024-01-27T00:47:18.469785+0000', 'mapping_epoch': 185, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 185, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:04:43.628330+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-27T00:47:25.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.1 is not active+clean 2024-01-27T00:47:25.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.2 is not active+clean 2024-01-27T00:47:25.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.3 is not active+clean 2024-01-27T00:47:25.530 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.a is not active+clean 2024-01-27T00:47:25.530 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T00:47:18.469785+0000', 'last_change': '2024-01-27T00:47:18.482501+0000', 'last_active': '2024-01-27T00:47:18.469785+0000', 'last_peered': '2024-01-27T00:47:18.469785+0000', 'last_clean': '2024-01-27T00:47:18.469785+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:18.469785+0000', 'last_undegraded': '2024-01-27T00:47:18.469785+0000', 'last_fullsized': '2024-01-27T00:47:18.469785+0000', 'mapping_epoch': 185, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 185, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T12:10:10.727612+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-27T00:47:25.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.7 is not active+clean 2024-01-27T00:47:25.530 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.8 is not active+clean 2024-01-27T00:47:25.530 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T00:47:18.469785+0000', 'last_change': '2024-01-27T00:47:18.481168+0000', 'last_active': '2024-01-27T00:47:18.469785+0000', 'last_peered': '2024-01-27T00:47:18.469785+0000', 'last_clean': '2024-01-27T00:47:18.469785+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:18.469785+0000', 'last_undegraded': '2024-01-27T00:47:18.469785+0000', 'last_fullsized': '2024-01-27T00:47:18.469785+0000', 'mapping_epoch': 185, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 185, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T01:38:22.205934+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-27T00:47:25.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.8 is not active+clean 2024-01-27T00:47:25.530 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.531 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.c is not active+clean 2024-01-27T00:47:25.531 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.531 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.3 is not active+clean 2024-01-27T00:47:25.531 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T00:47:18.469785+0000', 'last_change': '2024-01-27T00:47:18.479789+0000', 'last_active': '2024-01-27T00:47:18.469785+0000', 'last_peered': '2024-01-27T00:47:18.469785+0000', 'last_clean': '2024-01-27T00:47:18.469785+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:18.469785+0000', 'last_undegraded': '2024-01-27T00:47:18.469785+0000', 'last_fullsized': '2024-01-27T00:47:18.469785+0000', 'mapping_epoch': 185, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 185, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T07:11:07.269026+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-27T00:47:25.531 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.d is not active+clean 2024-01-27T00:47:25.531 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.531 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.1 is not active+clean 2024-01-27T00:47:25.531 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T00:47:18.469785+0000', 'last_change': '2024-01-27T00:47:18.482920+0000', 'last_active': '2024-01-27T00:47:18.469785+0000', 'last_peered': '2024-01-27T00:47:18.469785+0000', 'last_clean': '2024-01-27T00:47:18.469785+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:18.469785+0000', 'last_undegraded': '2024-01-27T00:47:18.469785+0000', 'last_fullsized': '2024-01-27T00:47:18.469785+0000', 'mapping_epoch': 185, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 185, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:18.469785+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T08:20:46.023087+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-27T00:47:25.531 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.19 is not active+clean 2024-01-27T00:47:25.531 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T00:47:19.474295+0000', 'last_change': '2024-01-27T00:47:19.474295+0000', 'last_active': '2024-01-27T00:47:19.474295+0000', 'last_peered': '2024-01-27T00:47:19.474295+0000', 'last_clean': '2024-01-27T00:47:19.474295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T00:47:19.474295+0000', 'last_undegraded': '2024-01-27T00:47:19.474295+0000', 'last_fullsized': '2024-01-27T00:47:19.474295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'last_clean_scrub_stamp': '2024-01-27T00:47:19.474295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T00:47:25.532 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-27T00:47:25.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.694+0000 7f65c4095700 1 Processor -- start 2024-01-27T00:47:25.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.694+0000 7f65c4095700 1 -- start start 2024-01-27T00:47:25.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.695+0000 7f65c4095700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f65bc157920 0x7f65bc0b2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:25.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.695+0000 7f65c4095700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65bc0cc550 con 0x7f65bc0b2a60 2024-01-27T00:47:25.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.695+0000 7f65c4095700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65bc0cb6a0 con 0x7f65bc157550 2024-01-27T00:47:25.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.695+0000 7f65c4095700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65bc0ca020 con 0x7f65bc157920 2024-01-27T00:47:25.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.695+0000 7f65c1630700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f65bc157920 0x7f65bc0b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:25.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.695+0000 7f65c2632700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f65bc0b2a60 0x7f65bc0cc9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45670/0 (socket says 172.21.15.46:45670) 2024-01-27T00:47:25.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.695+0000 7f65c1630700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f65bc157920 0x7f65bc0b2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40602/0 (socket says 172.21.15.46:40602) 2024-01-27T00:47:25.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.695+0000 7f65c2632700 1 -- 172.21.15.46:0/1184999807 learned_addr learned my addr 172.21.15.46:0/1184999807 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:25.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.696+0000 7f65c1630700 1 -- 172.21.15.46:0/1184999807 >> v1:172.21.15.106:6789/0 conn(0x7f65bc157550 legacy=0x7f65bc06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:25.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.696+0000 7f65c1630700 1 -- 172.21.15.46:0/1184999807 >> v1:172.21.15.46:6789/0 conn(0x7f65bc0b2a60 legacy=0x7f65bc0cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:25.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.696+0000 7f65c0e2f700 1 -- 172.21.15.46:0/1184999807 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790384371 0 0) 0x7f65bc0cc550 con 0x7f65bc0b2a60 2024-01-27T00:47:25.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.696+0000 7f65c1630700 1 -- 172.21.15.46:0/1184999807 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65bc0cb940 con 0x7f65bc157920 2024-01-27T00:47:25.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.696+0000 7f65c1630700 1 --2- 172.21.15.46:0/1184999807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65bc157920 0x7f65bc0b2520 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f65b80098e0 tx=0x7f65b80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea85f60657b4aad server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:25.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.696+0000 7f65c0e2f700 1 -- 172.21.15.46:0/1184999807 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b8009e80 con 0x7f65bc157920 2024-01-27T00:47:25.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.696+0000 7f65c0e2f700 1 -- 172.21.15.46:0/1184999807 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b800b860 con 0x7f65bc157920 2024-01-27T00:47:25.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.696+0000 7f65c0e2f700 1 -- 172.21.15.46:0/1184999807 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b800bbe0 con 0x7f65bc157920 2024-01-27T00:47:25.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.697+0000 7f65c4095700 1 -- 172.21.15.46:0/1184999807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65bc157920 msgr2=0x7f65bc0b2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:25.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.697+0000 7f65c4095700 1 --2- 172.21.15.46:0/1184999807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65bc157920 0x7f65bc0b2520 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f65b80098e0 tx=0x7f65b80091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.697+0000 7f65c4095700 1 -- 172.21.15.46:0/1184999807 shutdown_connections 2024-01-27T00:47:25.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.697+0000 7f65c4095700 1 --2- 172.21.15.46:0/1184999807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65bc157920 0x7f65bc0b2520 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.697+0000 7f65c4095700 1 -- 172.21.15.46:0/1184999807 >> 172.21.15.46:0/1184999807 conn(0x7f65bc069510 msgr2=0x7f65bc0aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:25.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.697+0000 7f65c4095700 1 -- 172.21.15.46:0/1184999807 shutdown_connections 2024-01-27T00:47:25.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.697+0000 7f65c4095700 1 -- 172.21.15.46:0/1184999807 wait complete. 2024-01-27T00:47:25.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.698+0000 7f65c4095700 1 Processor -- start 2024-01-27T00:47:25.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.698+0000 7f65c4095700 1 -- start start 2024-01-27T00:47:25.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.699+0000 7f65c4095700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65bc0b2a60 0x7f65bc16a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:25.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.699+0000 7f65c4095700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65bc157550 0x7f65bc15a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:25.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.699+0000 7f65c4095700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65bc157920 0x7f65bc15ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:25.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.699+0000 7f65c4095700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc16ac10 con 0x7f65bc157550 2024-01-27T00:47:25.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.699+0000 7f65c4095700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc15c8f0 con 0x7f65bc0b2a60 2024-01-27T00:47:25.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.699+0000 7f65c4095700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc0adbf0 con 0x7f65bc157920 2024-01-27T00:47:25.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.699+0000 7f65c1630700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65bc157550 0x7f65bc15a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:25.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.699+0000 7f65c1630700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65bc157550 0x7f65bc15a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40616/0 (socket says 172.21.15.46:40616) 2024-01-27T00:47:25.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.699+0000 7f65c1630700 1 -- 172.21.15.46:0/1112042519 learned_addr learned my addr 172.21.15.46:0/1112042519 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:25.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.699+0000 7f65c2632700 1 --2- 172.21.15.46:0/1112042519 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65bc157920 0x7f65bc15ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:25.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.699+0000 7f65c1e31700 1 --2- 172.21.15.46:0/1112042519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65bc0b2a60 0x7f65bc16a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:25.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.699+0000 7f65c1630700 1 -- 172.21.15.46:0/1112042519 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65bc157920 msgr2=0x7f65bc15ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:25.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.699+0000 7f65c1630700 1 --2- 172.21.15.46:0/1112042519 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65bc157920 0x7f65bc15ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.699+0000 7f65c1630700 1 -- 172.21.15.46:0/1112042519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65bc0b2a60 msgr2=0x7f65bc16a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:25.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.699+0000 7f65c1630700 1 --2- 172.21.15.46:0/1112042519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65bc0b2a60 0x7f65bc16a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:25.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.699+0000 7f65c1630700 1 -- 172.21.15.46:0/1112042519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b8009420 con 0x7f65bc157550 2024-01-27T00:47:25.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.699+0000 7f65c1630700 1 --2- 172.21.15.46:0/1112042519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65bc157550 0x7f65bc15a8c0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f65b8013000 tx=0x7f65b8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:25.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.700+0000 7f65b1ffb700 1 -- 172.21.15.46:0/1112042519 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b801f070 con 0x7f65bc157550 2024-01-27T00:47:25.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.700+0000 7f65b1ffb700 1 -- 172.21.15.46:0/1112042519 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b8000c90 con 0x7f65bc157550 2024-01-27T00:47:25.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.700+0000 7f65c4095700 1 -- 172.21.15.46:0/1112042519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65bc0aded0 con 0x7f65bc157550 2024-01-27T00:47:25.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.700+0000 7f65b1ffb700 1 -- 172.21.15.46:0/1112042519 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b8025420 con 0x7f65bc157550 2024-01-27T00:47:25.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.700+0000 7f65c4095700 1 -- 172.21.15.46:0/1112042519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65bc0ae490 con 0x7f65bc157550 2024-01-27T00:47:25.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.701+0000 7f65c4095700 1 -- 172.21.15.46:0/1112042519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65bc15cab0 con 0x7f65bc157550 2024-01-27T00:47:25.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.702+0000 7f65b1ffb700 1 -- 172.21.15.46:0/1112042519 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f65b8023070 con 0x7f65bc157550 2024-01-27T00:47:25.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.702+0000 7f65b1ffb700 1 --2- 172.21.15.46:0/1112042519 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f65a0066f30 0x7f65a00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:25.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.702+0000 7f65b1ffb700 1 -- 172.21.15.46:0/1112042519 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f65b8085d60 con 0x7f65bc157550 2024-01-27T00:47:25.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.702+0000 7f65c1e31700 1 --2- 172.21.15.46:0/1112042519 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f65a0066f30 0x7f65a00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:25.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.703+0000 7f65c1e31700 1 --2- 172.21.15.46:0/1112042519 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f65a0066f30 0x7f65a00693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f65ac003860 tx=0x7f65ac009040 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:25.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:25.706+0000 7f65b1ffb700 1 -- 172.21.15.46:0/1112042519 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f65b8052c40 con 0x7f65bc157550 2024-01-27T00:47:26.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.001+0000 7f65c4095700 1 -- 172.21.15.46:0/1112042519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f65bc04c710 con 0x7f65bc157550 2024-01-27T00:47:26.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.002+0000 7f65b1ffb700 1 -- 172.21.15.46:0/1112042519 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1020 (secure 0 0 0) 0x7f65b8015080 con 0x7f65bc157550 2024-01-27T00:47:26.003 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:26.003 INFO:teuthology.orchestra.run.smithi046.stdout:{"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":4679,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":188,"num_osds":8,"num_up_osds":8,"osd_up_since":1706311777,"num_in_osds":8,"osd_in_since":1706311769,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":23,"data_bytes":592674,"bytes_used":2543558656,"bytes_avail":770550554624,"bytes_total":773094113280,"write_bytes_sec":2184,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":97,"id":11,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":11,"rank":0,"name":"c","status":"up:active","gid":9828}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-01-27T00:36:22.537969+0000","services":{}},"progress_events":{}} 2024-01-27T00:47:26.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.004+0000 7f65c4095700 1 -- 172.21.15.46:0/1112042519 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f65a0066f30 msgr2=0x7f65a00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:26.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.004+0000 7f65c4095700 1 --2- 172.21.15.46:0/1112042519 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f65a0066f30 0x7f65a00693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f65ac003860 tx=0x7f65ac009040 comp rx=0 tx=0).stop 2024-01-27T00:47:26.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.004+0000 7f65c4095700 1 -- 172.21.15.46:0/1112042519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65bc157550 msgr2=0x7f65bc15a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:26.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.004+0000 7f65c4095700 1 --2- 172.21.15.46:0/1112042519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65bc157550 0x7f65bc15a8c0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f65b8013000 tx=0x7f65b8017000 comp rx=0 tx=0).stop 2024-01-27T00:47:26.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.004+0000 7f65c4095700 1 -- 172.21.15.46:0/1112042519 shutdown_connections 2024-01-27T00:47:26.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.004+0000 7f65c4095700 1 --2- 172.21.15.46:0/1112042519 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f65a0066f30 0x7f65a00693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:26.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.004+0000 7f65c4095700 1 --2- 172.21.15.46:0/1112042519 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65bc157920 0x7f65bc15ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:26.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.004+0000 7f65c4095700 1 --2- 172.21.15.46:0/1112042519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65bc0b2a60 0x7f65bc16a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:26.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.004+0000 7f65c4095700 1 --2- 172.21.15.46:0/1112042519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65bc157550 0x7f65bc15a8c0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:26.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.004+0000 7f65c4095700 1 -- 172.21.15.46:0/1112042519 >> 172.21.15.46:0/1112042519 conn(0x7f65bc069510 msgr2=0x7f65bc0cdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:26.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.004+0000 7f65c4095700 1 -- 172.21.15.46:0/1112042519 shutdown_connections 2024-01-27T00:47:26.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.004+0000 7f65c4095700 1 -- 172.21.15.46:0/1112042519 wait complete. 2024-01-27T00:47:26.018 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-27T00:47:26.019 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T00:47:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.151+0000 7f3e8a925700 1 Processor -- start 2024-01-27T00:47:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.152+0000 7f3e8a925700 1 -- start start 2024-01-27T00:47:26.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.152+0000 7f3e8a925700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3e840b2b40 0x7f3e840ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:26.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.152+0000 7f3e8a925700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3e840ae960 0x7f3e840ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:26.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.152+0000 7f3e8a925700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8415a0b0 con 0x7f3e840aedd0 2024-01-27T00:47:26.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.152+0000 7f3e8a925700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e84099fc0 con 0x7f3e840ae960 2024-01-27T00:47:26.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.152+0000 7f3e8a925700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e840ae4d0 con 0x7f3e840b2b40 2024-01-27T00:47:26.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.153+0000 7f3e88ec2700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3e840aedd0 0x7f3e840cb5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45678/0 (socket says 172.21.15.46:45678) 2024-01-27T00:47:26.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.153+0000 7f3e88ec2700 1 -- 172.21.15.46:0/1746971532 learned_addr learned my addr 172.21.15.46:0/1746971532 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:26.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.153+0000 7f3e83fff700 1 --2- 172.21.15.46:0/1746971532 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3e840b2b40 0x7f3e840ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:26.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.153+0000 7f3e837fe700 1 --2- 172.21.15.46:0/1746971532 >> v2:172.21.15.106:3300/0 conn(0x7f3e840ae960 0x7f3e840ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:26.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.153+0000 7f3e82ffd700 1 -- 172.21.15.46:0/1746971532 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761525518 0 0) 0x7f3e8415a0b0 con 0x7f3e840aedd0 2024-01-27T00:47:26.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.153+0000 7f3e82ffd700 1 -- 172.21.15.46:0/1746971532 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e6c003570 con 0x7f3e840aedd0 2024-01-27T00:47:26.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.153+0000 7f3e83fff700 1 -- 172.21.15.46:0/1746971532 >> v2:172.21.15.106:3300/0 conn(0x7f3e840ae960 msgr2=0x7f3e840ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:26.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.153+0000 7f3e83fff700 1 --2- 172.21.15.46:0/1746971532 >> v2:172.21.15.106:3300/0 conn(0x7f3e840ae960 0x7f3e840ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:26.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.153+0000 7f3e83fff700 1 -- 172.21.15.46:0/1746971532 >> v1:172.21.15.46:6789/0 conn(0x7f3e840aedd0 legacy=0x7f3e840cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:26.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.153+0000 7f3e83fff700 1 -- 172.21.15.46:0/1746971532 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e840ad8c0 con 0x7f3e840b2b40 2024-01-27T00:47:26.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.153+0000 7f3e83fff700 1 --2- 172.21.15.46:0/1746971532 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3e840b2b40 0x7f3e840ad1c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3e740098d0 tx=0x7f3e74009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6547a6a4f0fe4ad4 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:26.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.153+0000 7f3e82ffd700 1 -- 172.21.15.46:0/1746971532 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e74005c10 con 0x7f3e840b2b40 2024-01-27T00:47:26.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.154+0000 7f3e82ffd700 1 -- 172.21.15.46:0/1746971532 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e74005d70 con 0x7f3e840b2b40 2024-01-27T00:47:26.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.154+0000 7f3e82ffd700 1 -- 172.21.15.46:0/1746971532 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e74006280 con 0x7f3e840b2b40 2024-01-27T00:47:26.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.154+0000 7f3e8a925700 1 -- 172.21.15.46:0/1746971532 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3e840b2b40 msgr2=0x7f3e840ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:26.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.154+0000 7f3e8a925700 1 --2- 172.21.15.46:0/1746971532 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3e840b2b40 0x7f3e840ad1c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3e740098d0 tx=0x7f3e74009120 comp rx=0 tx=0).stop 2024-01-27T00:47:26.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.154+0000 7f3e8a925700 1 -- 172.21.15.46:0/1746971532 shutdown_connections 2024-01-27T00:47:26.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.154+0000 7f3e8a925700 1 --2- 172.21.15.46:0/1746971532 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3e840b2b40 0x7f3e840ad1c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:26.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.154+0000 7f3e8a925700 1 --2- 172.21.15.46:0/1746971532 >> v2:172.21.15.106:3300/0 conn(0x7f3e840ae960 0x7f3e840ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:26.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.154+0000 7f3e8a925700 1 -- 172.21.15.46:0/1746971532 >> 172.21.15.46:0/1746971532 conn(0x7f3e84099a30 msgr2=0x7f3e8406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:26.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.154+0000 7f3e8a925700 1 -- 172.21.15.46:0/1746971532 shutdown_connections 2024-01-27T00:47:26.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.154+0000 7f3e8a925700 1 -- 172.21.15.46:0/1746971532 wait complete. 2024-01-27T00:47:26.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.155+0000 7f3e8a925700 1 Processor -- start 2024-01-27T00:47:26.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.156+0000 7f3e8a925700 1 -- start start 2024-01-27T00:47:26.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.156+0000 7f3e8a925700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3e840ae960 0x7f3e84150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:26.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.156+0000 7f3e8a925700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e840b2b40 0x7f3e84154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:26.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.156+0000 7f3e8a925700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3e840b11d0 0x7f3e8416dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:26.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.156+0000 7f3e8a925700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e84150db0 con 0x7f3e840b2b40 2024-01-27T00:47:26.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.156+0000 7f3e8a925700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e840b2270 con 0x7f3e840ae960 2024-01-27T00:47:26.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.156+0000 7f3e8a925700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e84154aa0 con 0x7f3e840b11d0 2024-01-27T00:47:26.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.156+0000 7f3e837fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e840b2b40 0x7f3e84154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:26.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.156+0000 7f3e837fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e840b2b40 0x7f3e84154560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40620/0 (socket says 172.21.15.46:40620) 2024-01-27T00:47:26.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.157+0000 7f3e837fe700 1 -- 172.21.15.46:0/2079474918 learned_addr learned my addr 172.21.15.46:0/2079474918 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:26.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.157+0000 7f3e88ec2700 1 --2- 172.21.15.46:0/2079474918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3e840b11d0 0x7f3e8416dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:26.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.157+0000 7f3e83fff700 1 --2- 172.21.15.46:0/2079474918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3e840ae960 0x7f3e84150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:26.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.157+0000 7f3e837fe700 1 -- 172.21.15.46:0/2079474918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3e840b11d0 msgr2=0x7f3e8416dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:26.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.157+0000 7f3e837fe700 1 --2- 172.21.15.46:0/2079474918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3e840b11d0 0x7f3e8416dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:26.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.157+0000 7f3e837fe700 1 -- 172.21.15.46:0/2079474918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3e840ae960 msgr2=0x7f3e84150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:26.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.157+0000 7f3e837fe700 1 --2- 172.21.15.46:0/2079474918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3e840ae960 0x7f3e84150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:26.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.157+0000 7f3e837fe700 1 -- 172.21.15.46:0/2079474918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e740093c0 con 0x7f3e840b2b40 2024-01-27T00:47:26.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.157+0000 7f3e837fe700 1 --2- 172.21.15.46:0/2079474918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e840b2b40 0x7f3e84154560 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f3e7800e960 tx=0x7f3e780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:26.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.157+0000 7f3e6bfff700 1 -- 172.21.15.46:0/2079474918 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7800c850 con 0x7f3e840b2b40 2024-01-27T00:47:26.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.157+0000 7f3e6bfff700 1 -- 172.21.15.46:0/2079474918 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7800c9b0 con 0x7f3e840b2b40 2024-01-27T00:47:26.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.157+0000 7f3e8a925700 1 -- 172.21.15.46:0/2079474918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e8416e000 con 0x7f3e840b2b40 2024-01-27T00:47:26.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.157+0000 7f3e6bfff700 1 -- 172.21.15.46:0/2079474918 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7800cd50 con 0x7f3e840b2b40 2024-01-27T00:47:26.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.157+0000 7f3e8a925700 1 -- 172.21.15.46:0/2079474918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e840b1660 con 0x7f3e840b2b40 2024-01-27T00:47:26.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.158+0000 7f3e8a925700 1 -- 172.21.15.46:0/2079474918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e840afb30 con 0x7f3e840b2b40 2024-01-27T00:47:26.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.159+0000 7f3e6bfff700 1 -- 172.21.15.46:0/2079474918 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3e78013680 con 0x7f3e840b2b40 2024-01-27T00:47:26.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.159+0000 7f3e6bfff700 1 --2- 172.21.15.46:0/2079474918 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3e70066f30 0x7f3e700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:26.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.159+0000 7f3e6bfff700 1 -- 172.21.15.46:0/2079474918 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f3e78084aa0 con 0x7f3e840b2b40 2024-01-27T00:47:26.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.160+0000 7f3e83fff700 1 --2- 172.21.15.46:0/2079474918 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3e70066f30 0x7f3e700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:26.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.160+0000 7f3e83fff700 1 --2- 172.21.15.46:0/2079474918 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3e70066f30 0x7f3e700693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3e740098d0 tx=0x7f3e74016000 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:26.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.164+0000 7f3e6bfff700 1 -- 172.21.15.46:0/2079474918 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3e840afb30 con 0x7f3e840b2b40 2024-01-27T00:47:26.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.368+0000 7f3e8a925700 1 -- 172.21.15.46:0/2079474918 --> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3e840afb30 con 0x7f3e70066f30 2024-01-27T00:47:26.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.377+0000 7f3e6bfff700 1 -- 172.21.15.46:0/2079474918 <== mgr.9313 v2:172.21.15.106:6836/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290035 (secure 0 0 0) 0x7f3e840afb30 con 0x7f3e70066f30 2024-01-27T00:47:26.380 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:26.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.380+0000 7f3e8a925700 1 -- 172.21.15.46:0/2079474918 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3e70066f30 msgr2=0x7f3e700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:26.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.380+0000 7f3e8a925700 1 --2- 172.21.15.46:0/2079474918 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3e70066f30 0x7f3e700693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3e740098d0 tx=0x7f3e74016000 comp rx=0 tx=0).stop 2024-01-27T00:47:26.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.380+0000 7f3e8a925700 1 -- 172.21.15.46:0/2079474918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e840b2b40 msgr2=0x7f3e84154560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:26.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.380+0000 7f3e8a925700 1 --2- 172.21.15.46:0/2079474918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e840b2b40 0x7f3e84154560 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f3e7800e960 tx=0x7f3e780091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:26.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.380+0000 7f3e8a925700 1 -- 172.21.15.46:0/2079474918 shutdown_connections 2024-01-27T00:47:26.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.380+0000 7f3e8a925700 1 --2- 172.21.15.46:0/2079474918 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3e70066f30 0x7f3e700693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:26.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.380+0000 7f3e8a925700 1 --2- 172.21.15.46:0/2079474918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3e840b11d0 0x7f3e8416dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:26.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.380+0000 7f3e8a925700 1 --2- 172.21.15.46:0/2079474918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3e840ae960 0x7f3e84150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:26.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.380+0000 7f3e8a925700 1 --2- 172.21.15.46:0/2079474918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e840b2b40 0x7f3e84154560 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:26.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.380+0000 7f3e8a925700 1 -- 172.21.15.46:0/2079474918 >> 172.21.15.46:0/2079474918 conn(0x7f3e84099a30 msgr2=0x7f3e840c6a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:26.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.380+0000 7f3e8a925700 1 -- 172.21.15.46:0/2079474918 shutdown_connections 2024-01-27T00:47:26.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:26.380+0000 7f3e8a925700 1 -- 172.21.15.46:0/2079474918 wait complete. 2024-01-27T00:47:26.384 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T00:47:26.394 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-27T00:47:25.571465+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1784,"num_read_kb":3778,"num_write":574,"num_write_kb":3475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8733,"ondisk_log_size":8733,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":186,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2483944,"kb_used_data":19624,"kb_used_omap":0,"kb_used_meta":2464256,"kb_avail":752490776,"statfs":{"total":773094113280,"available":770550554624,"internally_reserved":0,"allocated":20094976,"data_stored":10987236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2523398144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.093524"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512073+0000","last_change":"2024-01-27T00:47:20.509860+0000","last_active":"2024-01-27T00:47:20.512073+0000","last_peered":"2024-01-27T00:47:20.512073+0000","last_clean":"2024-01-27T00:47:20.512073+0000","last_became_active":"2024-01-27T00:47:20.509586+0000","last_became_peered":"2024-01-27T00:47:20.509586+0000","last_unstale":"2024-01-27T00:47:20.512073+0000","last_undegraded":"2024-01-27T00:47:20.512073+0000","last_fullsized":"2024-01-27T00:47:20.512073+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:38:23.210445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.518716+0000","last_change":"2024-01-27T00:47:20.514877+0000","last_active":"2024-01-27T00:47:20.518716+0000","last_peered":"2024-01-27T00:47:20.518716+0000","last_clean":"2024-01-27T00:47:20.518716+0000","last_became_active":"2024-01-27T00:47:20.514587+0000","last_became_peered":"2024-01-27T00:47:20.514587+0000","last_unstale":"2024-01-27T00:47:20.518716+0000","last_undegraded":"2024-01-27T00:47:20.518716+0000","last_fullsized":"2024-01-27T00:47:20.518716+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:44.855002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.518231+0000","last_change":"2024-01-27T00:47:20.514155+0000","last_active":"2024-01-27T00:47:20.518231+0000","last_peered":"2024-01-27T00:47:20.518231+0000","last_clean":"2024-01-27T00:47:20.518231+0000","last_became_active":"2024-01-27T00:47:20.513885+0000","last_became_peered":"2024-01-27T00:47:20.513885+0000","last_unstale":"2024-01-27T00:47:20.518231+0000","last_undegraded":"2024-01-27T00:47:20.518231+0000","last_fullsized":"2024-01-27T00:47:20.518231+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:49:42.939991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.511955+0000","last_change":"2024-01-27T00:47:20.509503+0000","last_active":"2024-01-27T00:47:20.511955+0000","last_peered":"2024-01-27T00:47:20.511955+0000","last_clean":"2024-01-27T00:47:20.511955+0000","last_became_active":"2024-01-27T00:47:20.509094+0000","last_became_peered":"2024-01-27T00:47:20.509094+0000","last_unstale":"2024-01-27T00:47:20.511955+0000","last_undegraded":"2024-01-27T00:47:20.511955+0000","last_fullsized":"2024-01-27T00:47:20.511955+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:53:55.794828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.523334+0000","last_change":"2024-01-27T00:47:20.509897+0000","last_active":"2024-01-27T00:47:20.523334+0000","last_peered":"2024-01-27T00:47:20.523334+0000","last_clean":"2024-01-27T00:47:20.523334+0000","last_became_active":"2024-01-27T00:47:20.509610+0000","last_became_peered":"2024-01-27T00:47:20.509610+0000","last_unstale":"2024-01-27T00:47:20.523334+0000","last_undegraded":"2024-01-27T00:47:20.523334+0000","last_fullsized":"2024-01-27T00:47:20.523334+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:42:01.971654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.518008+0000","last_change":"2024-01-27T00:47:20.514363+0000","last_active":"2024-01-27T00:47:20.518008+0000","last_peered":"2024-01-27T00:47:20.518008+0000","last_clean":"2024-01-27T00:47:20.518008+0000","last_became_active":"2024-01-27T00:47:20.514085+0000","last_became_peered":"2024-01-27T00:47:20.514085+0000","last_unstale":"2024-01-27T00:47:20.518008+0000","last_undegraded":"2024-01-27T00:47:20.518008+0000","last_fullsized":"2024-01-27T00:47:20.518008+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:33.654223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527541+0000","last_change":"2024-01-27T00:47:20.514165+0000","last_active":"2024-01-27T00:47:20.527541+0000","last_peered":"2024-01-27T00:47:20.527541+0000","last_clean":"2024-01-27T00:47:20.527541+0000","last_became_active":"2024-01-27T00:47:20.513848+0000","last_became_peered":"2024-01-27T00:47:20.513848+0000","last_unstale":"2024-01-27T00:47:20.527541+0000","last_undegraded":"2024-01-27T00:47:20.527541+0000","last_fullsized":"2024-01-27T00:47:20.527541+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:43:04.739251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524282+0000","last_change":"2024-01-27T00:47:20.511666+0000","last_active":"2024-01-27T00:47:20.524282+0000","last_peered":"2024-01-27T00:47:20.524282+0000","last_clean":"2024-01-27T00:47:20.524282+0000","last_became_active":"2024-01-27T00:47:20.509826+0000","last_became_peered":"2024-01-27T00:47:20.509826+0000","last_unstale":"2024-01-27T00:47:20.524282+0000","last_undegraded":"2024-01-27T00:47:20.524282+0000","last_fullsized":"2024-01-27T00:47:20.524282+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:53:47.125547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524159+0000","last_change":"2024-01-27T00:47:20.511731+0000","last_active":"2024-01-27T00:47:20.524159+0000","last_peered":"2024-01-27T00:47:20.524159+0000","last_clean":"2024-01-27T00:47:20.524159+0000","last_became_active":"2024-01-27T00:47:20.511436+0000","last_became_peered":"2024-01-27T00:47:20.511436+0000","last_unstale":"2024-01-27T00:47:20.524159+0000","last_undegraded":"2024-01-27T00:47:20.524159+0000","last_fullsized":"2024-01-27T00:47:20.524159+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:40.342669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524015+0000","last_change":"2024-01-27T00:47:20.510790+0000","last_active":"2024-01-27T00:47:20.524015+0000","last_peered":"2024-01-27T00:47:20.524015+0000","last_clean":"2024-01-27T00:47:20.524015+0000","last_became_active":"2024-01-27T00:47:20.510447+0000","last_became_peered":"2024-01-27T00:47:20.510447+0000","last_unstale":"2024-01-27T00:47:20.524015+0000","last_undegraded":"2024-01-27T00:47:20.524015+0000","last_fullsized":"2024-01-27T00:47:20.524015+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:56:44.894090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512593+0000","last_change":"2024-01-27T00:47:20.508709+0000","last_active":"2024-01-27T00:47:20.512593+0000","last_peered":"2024-01-27T00:47:20.512593+0000","last_clean":"2024-01-27T00:47:20.512593+0000","last_became_active":"2024-01-27T00:47:20.508433+0000","last_became_peered":"2024-01-27T00:47:20.508433+0000","last_unstale":"2024-01-27T00:47:20.512593+0000","last_undegraded":"2024-01-27T00:47:20.512593+0000","last_fullsized":"2024-01-27T00:47:20.512593+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:33:01.452427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.517995+0000","last_change":"2024-01-27T00:47:20.513994+0000","last_active":"2024-01-27T00:47:20.517995+0000","last_peered":"2024-01-27T00:47:20.517995+0000","last_clean":"2024-01-27T00:47:20.517995+0000","last_became_active":"2024-01-27T00:47:20.513691+0000","last_became_peered":"2024-01-27T00:47:20.513691+0000","last_unstale":"2024-01-27T00:47:20.517995+0000","last_undegraded":"2024-01-27T00:47:20.517995+0000","last_fullsized":"2024-01-27T00:47:20.517995+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:53:47.125547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531204+0000","last_change":"2024-01-27T00:47:20.520667+0000","last_active":"2024-01-27T00:47:20.531204+0000","last_peered":"2024-01-27T00:47:20.531204+0000","last_clean":"2024-01-27T00:47:20.531204+0000","last_became_active":"2024-01-27T00:47:20.519891+0000","last_became_peered":"2024-01-27T00:47:20.519891+0000","last_unstale":"2024-01-27T00:47:20.531204+0000","last_undegraded":"2024-01-27T00:47:20.531204+0000","last_fullsized":"2024-01-27T00:47:20.531204+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:49:22.885968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522694+0000","last_change":"2024-01-27T00:47:20.513550+0000","last_active":"2024-01-27T00:47:20.522694+0000","last_peered":"2024-01-27T00:47:20.522694+0000","last_clean":"2024-01-27T00:47:20.522694+0000","last_became_active":"2024-01-27T00:47:20.512667+0000","last_became_peered":"2024-01-27T00:47:20.512667+0000","last_unstale":"2024-01-27T00:47:20.522694+0000","last_undegraded":"2024-01-27T00:47:20.522694+0000","last_fullsized":"2024-01-27T00:47:20.522694+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:09:43.978584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512205+0000","last_change":"2024-01-27T00:47:20.506077+0000","last_active":"2024-01-27T00:47:20.512205+0000","last_peered":"2024-01-27T00:47:20.512205+0000","last_clean":"2024-01-27T00:47:20.512205+0000","last_became_active":"2024-01-27T00:47:20.505787+0000","last_became_peered":"2024-01-27T00:47:20.505787+0000","last_unstale":"2024-01-27T00:47:20.512205+0000","last_undegraded":"2024-01-27T00:47:20.512205+0000","last_fullsized":"2024-01-27T00:47:20.512205+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:44:21.535352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.511915+0000","last_change":"2024-01-27T00:47:20.509025+0000","last_active":"2024-01-27T00:47:20.511915+0000","last_peered":"2024-01-27T00:47:20.511915+0000","last_clean":"2024-01-27T00:47:20.511915+0000","last_became_active":"2024-01-27T00:47:20.508726+0000","last_became_peered":"2024-01-27T00:47:20.508726+0000","last_unstale":"2024-01-27T00:47:20.511915+0000","last_undegraded":"2024-01-27T00:47:20.511915+0000","last_fullsized":"2024-01-27T00:47:20.511915+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:55:06.853231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.504493+0000","last_change":"2024-01-27T00:47:20.500113+0000","last_active":"2024-01-27T00:47:20.504493+0000","last_peered":"2024-01-27T00:47:20.504493+0000","last_clean":"2024-01-27T00:47:20.504493+0000","last_became_active":"2024-01-27T00:47:20.499814+0000","last_became_peered":"2024-01-27T00:47:20.499814+0000","last_unstale":"2024-01-27T00:47:20.504493+0000","last_undegraded":"2024-01-27T00:47:20.504493+0000","last_fullsized":"2024-01-27T00:47:20.504493+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:05:38.788368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527940+0000","last_change":"2024-01-27T00:47:20.524935+0000","last_active":"2024-01-27T00:47:20.527940+0000","last_peered":"2024-01-27T00:47:20.527940+0000","last_clean":"2024-01-27T00:47:20.527940+0000","last_became_active":"2024-01-27T00:47:20.524672+0000","last_became_peered":"2024-01-27T00:47:20.524672+0000","last_unstale":"2024-01-27T00:47:20.527940+0000","last_undegraded":"2024-01-27T00:47:20.527940+0000","last_fullsized":"2024-01-27T00:47:20.527940+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:28.323536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512193+0000","last_change":"2024-01-27T00:47:20.506031+0000","last_active":"2024-01-27T00:47:20.512193+0000","last_peered":"2024-01-27T00:47:20.512193+0000","last_clean":"2024-01-27T00:47:20.512193+0000","last_became_active":"2024-01-27T00:47:20.505758+0000","last_became_peered":"2024-01-27T00:47:20.505758+0000","last_unstale":"2024-01-27T00:47:20.512193+0000","last_undegraded":"2024-01-27T00:47:20.512193+0000","last_fullsized":"2024-01-27T00:47:20.512193+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:18:50.559323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.532911+0000","last_change":"2024-01-27T00:47:20.518490+0000","last_active":"2024-01-27T00:47:20.532911+0000","last_peered":"2024-01-27T00:47:20.532911+0000","last_clean":"2024-01-27T00:47:20.532911+0000","last_became_active":"2024-01-27T00:47:20.518138+0000","last_became_peered":"2024-01-27T00:47:20.518138+0000","last_unstale":"2024-01-27T00:47:20.532911+0000","last_undegraded":"2024-01-27T00:47:20.532911+0000","last_fullsized":"2024-01-27T00:47:20.532911+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:28.557943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531230+0000","last_change":"2024-01-27T00:47:20.520931+0000","last_active":"2024-01-27T00:47:20.531230+0000","last_peered":"2024-01-27T00:47:20.531230+0000","last_clean":"2024-01-27T00:47:20.531230+0000","last_became_active":"2024-01-27T00:47:20.520322+0000","last_became_peered":"2024-01-27T00:47:20.520322+0000","last_unstale":"2024-01-27T00:47:20.531230+0000","last_undegraded":"2024-01-27T00:47:20.531230+0000","last_fullsized":"2024-01-27T00:47:20.531230+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:04:43.917528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.528575+0000","last_change":"2024-01-27T00:47:20.523687+0000","last_active":"2024-01-27T00:47:20.528575+0000","last_peered":"2024-01-27T00:47:20.528575+0000","last_clean":"2024-01-27T00:47:20.528575+0000","last_became_active":"2024-01-27T00:47:20.523382+0000","last_became_peered":"2024-01-27T00:47:20.523382+0000","last_unstale":"2024-01-27T00:47:20.528575+0000","last_undegraded":"2024-01-27T00:47:20.528575+0000","last_fullsized":"2024-01-27T00:47:20.528575+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:05.094009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.530444+0000","last_change":"2024-01-27T00:47:20.522803+0000","last_active":"2024-01-27T00:47:20.530444+0000","last_peered":"2024-01-27T00:47:20.530444+0000","last_clean":"2024-01-27T00:47:20.530444+0000","last_became_active":"2024-01-27T00:47:20.522463+0000","last_became_peered":"2024-01-27T00:47:20.522463+0000","last_unstale":"2024-01-27T00:47:20.530444+0000","last_undegraded":"2024-01-27T00:47:20.530444+0000","last_fullsized":"2024-01-27T00:47:20.530444+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:04:18.616847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.511248+0000","last_change":"2024-01-27T00:47:20.508251+0000","last_active":"2024-01-27T00:47:20.511248+0000","last_peered":"2024-01-27T00:47:20.511248+0000","last_clean":"2024-01-27T00:47:20.511248+0000","last_became_active":"2024-01-27T00:47:20.507933+0000","last_became_peered":"2024-01-27T00:47:20.507933+0000","last_unstale":"2024-01-27T00:47:20.511248+0000","last_undegraded":"2024-01-27T00:47:20.511248+0000","last_fullsized":"2024-01-27T00:47:20.511248+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:22.403262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.504894+0000","last_change":"2024-01-27T00:47:20.503158+0000","last_active":"2024-01-27T00:47:20.504894+0000","last_peered":"2024-01-27T00:47:20.504894+0000","last_clean":"2024-01-27T00:47:20.504894+0000","last_became_active":"2024-01-27T00:47:20.502875+0000","last_became_peered":"2024-01-27T00:47:20.502875+0000","last_unstale":"2024-01-27T00:47:20.504894+0000","last_undegraded":"2024-01-27T00:47:20.504894+0000","last_fullsized":"2024-01-27T00:47:20.504894+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:33:13.706183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.513540+0000","last_change":"2024-01-27T00:47:20.509928+0000","last_active":"2024-01-27T00:47:20.513540+0000","last_peered":"2024-01-27T00:47:20.513540+0000","last_clean":"2024-01-27T00:47:20.513540+0000","last_became_active":"2024-01-27T00:47:20.509630+0000","last_became_peered":"2024-01-27T00:47:20.509630+0000","last_unstale":"2024-01-27T00:47:20.513540+0000","last_undegraded":"2024-01-27T00:47:20.513540+0000","last_fullsized":"2024-01-27T00:47:20.513540+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:02:52.058658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.528592+0000","last_change":"2024-01-27T00:47:20.524118+0000","last_active":"2024-01-27T00:47:20.528592+0000","last_peered":"2024-01-27T00:47:20.528592+0000","last_clean":"2024-01-27T00:47:20.528592+0000","last_became_active":"2024-01-27T00:47:20.523779+0000","last_became_peered":"2024-01-27T00:47:20.523779+0000","last_unstale":"2024-01-27T00:47:20.528592+0000","last_undegraded":"2024-01-27T00:47:20.528592+0000","last_fullsized":"2024-01-27T00:47:20.528592+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:43:04.739251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531770+0000","last_change":"2024-01-27T00:47:20.527150+0000","last_active":"2024-01-27T00:47:20.531770+0000","last_peered":"2024-01-27T00:47:20.531770+0000","last_clean":"2024-01-27T00:47:20.531770+0000","last_became_active":"2024-01-27T00:47:20.523827+0000","last_became_peered":"2024-01-27T00:47:20.523827+0000","last_unstale":"2024-01-27T00:47:20.531770+0000","last_undegraded":"2024-01-27T00:47:20.531770+0000","last_fullsized":"2024-01-27T00:47:20.531770+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:52:04.785901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.517901+0000","last_change":"2024-01-27T00:47:20.513610+0000","last_active":"2024-01-27T00:47:20.517901+0000","last_peered":"2024-01-27T00:47:20.517901+0000","last_clean":"2024-01-27T00:47:20.517901+0000","last_became_active":"2024-01-27T00:47:20.513327+0000","last_became_peered":"2024-01-27T00:47:20.513327+0000","last_unstale":"2024-01-27T00:47:20.517901+0000","last_undegraded":"2024-01-27T00:47:20.517901+0000","last_fullsized":"2024-01-27T00:47:20.517901+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:15.349063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.523328+0000","last_change":"2024-01-27T00:47:20.522714+0000","last_active":"2024-01-27T00:47:20.523328+0000","last_peered":"2024-01-27T00:47:20.523328+0000","last_clean":"2024-01-27T00:47:20.523328+0000","last_became_active":"2024-01-27T00:47:20.522460+0000","last_became_peered":"2024-01-27T00:47:20.522460+0000","last_unstale":"2024-01-27T00:47:20.523328+0000","last_undegraded":"2024-01-27T00:47:20.523328+0000","last_fullsized":"2024-01-27T00:47:20.523328+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:40.342669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512795+0000","last_change":"2024-01-27T00:47:20.510711+0000","last_active":"2024-01-27T00:47:20.512795+0000","last_peered":"2024-01-27T00:47:20.512795+0000","last_clean":"2024-01-27T00:47:20.512795+0000","last_became_active":"2024-01-27T00:47:20.510442+0000","last_became_peered":"2024-01-27T00:47:20.510442+0000","last_unstale":"2024-01-27T00:47:20.512795+0000","last_undegraded":"2024-01-27T00:47:20.512795+0000","last_fullsized":"2024-01-27T00:47:20.512795+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:54:22.080297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.519209+0000","last_change":"2024-01-27T00:47:20.517165+0000","last_active":"2024-01-27T00:47:20.519209+0000","last_peered":"2024-01-27T00:47:20.519209+0000","last_clean":"2024-01-27T00:47:20.519209+0000","last_became_active":"2024-01-27T00:47:20.516890+0000","last_became_peered":"2024-01-27T00:47:20.516890+0000","last_unstale":"2024-01-27T00:47:20.519209+0000","last_undegraded":"2024-01-27T00:47:20.519209+0000","last_fullsized":"2024-01-27T00:47:20.519209+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:14:43.680318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.517519+0000","last_change":"2024-01-27T00:47:20.510127+0000","last_active":"2024-01-27T00:47:20.517519+0000","last_peered":"2024-01-27T00:47:20.517519+0000","last_clean":"2024-01-27T00:47:20.517519+0000","last_became_active":"2024-01-27T00:47:20.509834+0000","last_became_peered":"2024-01-27T00:47:20.509834+0000","last_unstale":"2024-01-27T00:47:20.517519+0000","last_undegraded":"2024-01-27T00:47:20.517519+0000","last_fullsized":"2024-01-27T00:47:20.517519+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:34:17.410140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"187'4","reported_seq":14,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.532184+0000","last_change":"2024-01-27T00:47:19.505622+0000","last_active":"2024-01-27T00:47:20.532184+0000","last_peered":"2024-01-27T00:47:20.532184+0000","last_clean":"2024-01-27T00:47:20.532184+0000","last_became_active":"2024-01-27T00:47:19.505359+0000","last_became_peered":"2024-01-27T00:47:19.505359+0000","last_unstale":"2024-01-27T00:47:20.532184+0000","last_undegraded":"2024-01-27T00:47:20.532184+0000","last_fullsized":"2024-01-27T00:47:20.532184+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:30:46.693321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.510752+0000","last_change":"2024-01-27T00:47:20.509849+0000","last_active":"2024-01-27T00:47:20.510752+0000","last_peered":"2024-01-27T00:47:20.510752+0000","last_clean":"2024-01-27T00:47:20.510752+0000","last_became_active":"2024-01-27T00:47:20.509553+0000","last_became_peered":"2024-01-27T00:47:20.509553+0000","last_unstale":"2024-01-27T00:47:20.510752+0000","last_undegraded":"2024-01-27T00:47:20.510752+0000","last_fullsized":"2024-01-27T00:47:20.510752+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:44:26.489354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.528860+0000","last_change":"2024-01-27T00:47:19.498621+0000","last_active":"2024-01-27T00:47:20.528860+0000","last_peered":"2024-01-27T00:47:20.528860+0000","last_clean":"2024-01-27T00:47:20.528860+0000","last_became_active":"2024-01-27T00:47:19.498357+0000","last_became_peered":"2024-01-27T00:47:19.498357+0000","last_unstale":"2024-01-27T00:47:20.528860+0000","last_undegraded":"2024-01-27T00:47:20.528860+0000","last_fullsized":"2024-01-27T00:47:20.528860+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:20:56.150512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522707+0000","last_change":"2024-01-27T00:47:20.513840+0000","last_active":"2024-01-27T00:47:20.522707+0000","last_peered":"2024-01-27T00:47:20.522707+0000","last_clean":"2024-01-27T00:47:20.522707+0000","last_became_active":"2024-01-27T00:47:20.512745+0000","last_became_peered":"2024-01-27T00:47:20.512745+0000","last_unstale":"2024-01-27T00:47:20.522707+0000","last_undegraded":"2024-01-27T00:47:20.522707+0000","last_fullsized":"2024-01-27T00:47:20.522707+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:11:59.061116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.533311+0000","last_change":"2024-01-27T00:47:19.509540+0000","last_active":"2024-01-27T00:47:20.533311+0000","last_peered":"2024-01-27T00:47:20.533311+0000","last_clean":"2024-01-27T00:47:20.533311+0000","last_became_active":"2024-01-27T00:47:19.509301+0000","last_became_peered":"2024-01-27T00:47:19.509301+0000","last_unstale":"2024-01-27T00:47:20.533311+0000","last_undegraded":"2024-01-27T00:47:20.533311+0000","last_fullsized":"2024-01-27T00:47:20.533311+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:12:19.210091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522758+0000","last_change":"2024-01-27T00:47:20.521914+0000","last_active":"2024-01-27T00:47:20.522758+0000","last_peered":"2024-01-27T00:47:20.522758+0000","last_clean":"2024-01-27T00:47:20.522758+0000","last_became_active":"2024-01-27T00:47:20.521531+0000","last_became_peered":"2024-01-27T00:47:20.521531+0000","last_unstale":"2024-01-27T00:47:20.522758+0000","last_undegraded":"2024-01-27T00:47:20.522758+0000","last_fullsized":"2024-01-27T00:47:20.522758+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:46.559589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.533308+0000","last_change":"2024-01-27T00:47:19.505152+0000","last_active":"2024-01-27T00:47:20.533308+0000","last_peered":"2024-01-27T00:47:20.533308+0000","last_clean":"2024-01-27T00:47:20.533308+0000","last_became_active":"2024-01-27T00:47:19.504128+0000","last_became_peered":"2024-01-27T00:47:19.504128+0000","last_unstale":"2024-01-27T00:47:20.533308+0000","last_undegraded":"2024-01-27T00:47:20.533308+0000","last_fullsized":"2024-01-27T00:47:20.533308+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:52.404716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524071+0000","last_change":"2024-01-27T00:47:20.510375+0000","last_active":"2024-01-27T00:47:20.524071+0000","last_peered":"2024-01-27T00:47:20.524071+0000","last_clean":"2024-01-27T00:47:20.524071+0000","last_became_active":"2024-01-27T00:47:20.510069+0000","last_became_peered":"2024-01-27T00:47:20.510069+0000","last_unstale":"2024-01-27T00:47:20.524071+0000","last_undegraded":"2024-01-27T00:47:20.524071+0000","last_fullsized":"2024-01-27T00:47:20.524071+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:44.855002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.528716+0000","last_change":"2024-01-27T00:47:19.488294+0000","last_active":"2024-01-27T00:47:20.528716+0000","last_peered":"2024-01-27T00:47:20.528716+0000","last_clean":"2024-01-27T00:47:20.528716+0000","last_became_active":"2024-01-27T00:47:19.487984+0000","last_became_peered":"2024-01-27T00:47:19.487984+0000","last_unstale":"2024-01-27T00:47:20.528716+0000","last_undegraded":"2024-01-27T00:47:20.528716+0000","last_fullsized":"2024-01-27T00:47:20.528716+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:08:02.191122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.513175+0000","last_change":"2024-01-27T00:47:20.508823+0000","last_active":"2024-01-27T00:47:20.513175+0000","last_peered":"2024-01-27T00:47:20.513175+0000","last_clean":"2024-01-27T00:47:20.513175+0000","last_became_active":"2024-01-27T00:47:20.507955+0000","last_became_peered":"2024-01-27T00:47:20.507955+0000","last_unstale":"2024-01-27T00:47:20.513175+0000","last_undegraded":"2024-01-27T00:47:20.513175+0000","last_fullsized":"2024-01-27T00:47:20.513175+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:44.632841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531724+0000","last_change":"2024-01-27T00:47:19.493726+0000","last_active":"2024-01-27T00:47:20.531724+0000","last_peered":"2024-01-27T00:47:20.531724+0000","last_clean":"2024-01-27T00:47:20.531724+0000","last_became_active":"2024-01-27T00:47:19.493237+0000","last_became_peered":"2024-01-27T00:47:19.493237+0000","last_unstale":"2024-01-27T00:47:20.531724+0000","last_undegraded":"2024-01-27T00:47:20.531724+0000","last_fullsized":"2024-01-27T00:47:20.531724+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:33:12.701672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.532639+0000","last_change":"2024-01-27T00:47:20.516445+0000","last_active":"2024-01-27T00:47:20.532639+0000","last_peered":"2024-01-27T00:47:20.532639+0000","last_clean":"2024-01-27T00:47:20.532639+0000","last_became_active":"2024-01-27T00:47:20.515582+0000","last_became_peered":"2024-01-27T00:47:20.515582+0000","last_unstale":"2024-01-27T00:47:20.532639+0000","last_undegraded":"2024-01-27T00:47:20.532639+0000","last_fullsized":"2024-01-27T00:47:20.532639+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:07:16.136646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.507907+0000","last_change":"2024-01-27T00:47:19.505534+0000","last_active":"2024-01-27T00:47:20.507907+0000","last_peered":"2024-01-27T00:47:20.507907+0000","last_clean":"2024-01-27T00:47:20.507907+0000","last_became_active":"2024-01-27T00:47:19.504749+0000","last_became_peered":"2024-01-27T00:47:19.504749+0000","last_unstale":"2024-01-27T00:47:20.507907+0000","last_undegraded":"2024-01-27T00:47:20.507907+0000","last_fullsized":"2024-01-27T00:47:20.507907+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:48.314323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512091+0000","last_change":"2024-01-27T00:47:20.509887+0000","last_active":"2024-01-27T00:47:20.512091+0000","last_peered":"2024-01-27T00:47:20.512091+0000","last_clean":"2024-01-27T00:47:20.512091+0000","last_became_active":"2024-01-27T00:47:20.509560+0000","last_became_peered":"2024-01-27T00:47:20.509560+0000","last_unstale":"2024-01-27T00:47:20.512091+0000","last_undegraded":"2024-01-27T00:47:20.512091+0000","last_fullsized":"2024-01-27T00:47:20.512091+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:14:57.244737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.492777+0000","last_change":"2024-01-27T00:47:19.488903+0000","last_active":"2024-01-27T00:47:20.492777+0000","last_peered":"2024-01-27T00:47:20.492777+0000","last_clean":"2024-01-27T00:47:20.492777+0000","last_became_active":"2024-01-27T00:47:19.488649+0000","last_became_peered":"2024-01-27T00:47:19.488649+0000","last_unstale":"2024-01-27T00:47:20.492777+0000","last_undegraded":"2024-01-27T00:47:20.492777+0000","last_fullsized":"2024-01-27T00:47:20.492777+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:36:32.698812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.517497+0000","last_change":"2024-01-27T00:47:20.509808+0000","last_active":"2024-01-27T00:47:20.517497+0000","last_peered":"2024-01-27T00:47:20.517497+0000","last_clean":"2024-01-27T00:47:20.517497+0000","last_became_active":"2024-01-27T00:47:20.509527+0000","last_became_peered":"2024-01-27T00:47:20.509527+0000","last_unstale":"2024-01-27T00:47:20.517497+0000","last_undegraded":"2024-01-27T00:47:20.517497+0000","last_fullsized":"2024-01-27T00:47:20.517497+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:53:05.668777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499880+0000","last_change":"2024-01-27T00:47:19.505276+0000","last_active":"2024-01-27T00:47:20.499880+0000","last_peered":"2024-01-27T00:47:20.499880+0000","last_clean":"2024-01-27T00:47:20.499880+0000","last_became_active":"2024-01-27T00:47:19.504824+0000","last_became_peered":"2024-01-27T00:47:19.504824+0000","last_unstale":"2024-01-27T00:47:20.499880+0000","last_undegraded":"2024-01-27T00:47:20.499880+0000","last_fullsized":"2024-01-27T00:47:20.499880+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:53:04.664266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.510043+0000","last_change":"2024-01-27T00:47:20.507243+0000","last_active":"2024-01-27T00:47:20.510043+0000","last_peered":"2024-01-27T00:47:20.510043+0000","last_clean":"2024-01-27T00:47:20.510043+0000","last_became_active":"2024-01-27T00:47:20.506953+0000","last_became_peered":"2024-01-27T00:47:20.506953+0000","last_unstale":"2024-01-27T00:47:20.510043+0000","last_undegraded":"2024-01-27T00:47:20.510043+0000","last_fullsized":"2024-01-27T00:47:20.510043+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:23:47.466789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"187'3","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.528490+0000","last_change":"2024-01-27T00:47:19.493169+0000","last_active":"2024-01-27T00:47:20.528490+0000","last_peered":"2024-01-27T00:47:20.528490+0000","last_clean":"2024-01-27T00:47:20.528490+0000","last_became_active":"2024-01-27T00:47:19.492797+0000","last_became_peered":"2024-01-27T00:47:19.492797+0000","last_unstale":"2024-01-27T00:47:20.528490+0000","last_undegraded":"2024-01-27T00:47:20.528490+0000","last_fullsized":"2024-01-27T00:47:20.528490+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:48:59.935385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.530619+0000","last_change":"2024-01-27T00:47:20.516228+0000","last_active":"2024-01-27T00:47:20.530619+0000","last_peered":"2024-01-27T00:47:20.530619+0000","last_clean":"2024-01-27T00:47:20.530619+0000","last_became_active":"2024-01-27T00:47:20.515906+0000","last_became_peered":"2024-01-27T00:47:20.515906+0000","last_unstale":"2024-01-27T00:47:20.530619+0000","last_undegraded":"2024-01-27T00:47:20.530619+0000","last_fullsized":"2024-01-27T00:47:20.530619+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:12:20.214602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"187'6","reported_seq":16,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.534091+0000","last_change":"2024-01-27T00:47:19.502338+0000","last_active":"2024-01-27T00:47:20.534091+0000","last_peered":"2024-01-27T00:47:20.534091+0000","last_clean":"2024-01-27T00:47:20.534091+0000","last_became_active":"2024-01-27T00:47:19.502072+0000","last_became_peered":"2024-01-27T00:47:19.502072+0000","last_unstale":"2024-01-27T00:47:20.534091+0000","last_undegraded":"2024-01-27T00:47:20.534091+0000","last_fullsized":"2024-01-27T00:47:20.534091+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:34:16.405629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.519133+0000","last_change":"2024-01-27T00:47:20.516500+0000","last_active":"2024-01-27T00:47:20.519133+0000","last_peered":"2024-01-27T00:47:20.519133+0000","last_clean":"2024-01-27T00:47:20.519133+0000","last_became_active":"2024-01-27T00:47:20.516247+0000","last_became_peered":"2024-01-27T00:47:20.516247+0000","last_unstale":"2024-01-27T00:47:20.519133+0000","last_undegraded":"2024-01-27T00:47:20.519133+0000","last_fullsized":"2024-01-27T00:47:20.519133+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:44:28.491993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527814+0000","last_change":"2024-01-27T00:47:19.491092+0000","last_active":"2024-01-27T00:47:20.527814+0000","last_peered":"2024-01-27T00:47:20.527814+0000","last_clean":"2024-01-27T00:47:20.527814+0000","last_became_active":"2024-01-27T00:47:19.490799+0000","last_became_peered":"2024-01-27T00:47:19.490799+0000","last_unstale":"2024-01-27T00:47:20.527814+0000","last_undegraded":"2024-01-27T00:47:20.527814+0000","last_fullsized":"2024-01-27T00:47:20.527814+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:23:46.462278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527448+0000","last_change":"2024-01-27T00:47:20.515730+0000","last_active":"2024-01-27T00:47:20.527448+0000","last_peered":"2024-01-27T00:47:20.527448+0000","last_clean":"2024-01-27T00:47:20.527448+0000","last_became_active":"2024-01-27T00:47:20.515422+0000","last_became_peered":"2024-01-27T00:47:20.515422+0000","last_unstale":"2024-01-27T00:47:20.527448+0000","last_undegraded":"2024-01-27T00:47:20.527448+0000","last_fullsized":"2024-01-27T00:47:20.527448+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:33.654223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.498821+0000","last_change":"2024-01-27T00:47:19.499915+0000","last_active":"2024-01-27T00:47:20.498821+0000","last_peered":"2024-01-27T00:47:20.498821+0000","last_clean":"2024-01-27T00:47:20.498821+0000","last_became_active":"2024-01-27T00:47:19.499418+0000","last_became_peered":"2024-01-27T00:47:19.499418+0000","last_unstale":"2024-01-27T00:47:20.498821+0000","last_undegraded":"2024-01-27T00:47:20.498821+0000","last_fullsized":"2024-01-27T00:47:20.498821+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:46.642397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512934+0000","last_change":"2024-01-27T00:47:20.509483+0000","last_active":"2024-01-27T00:47:20.512934+0000","last_peered":"2024-01-27T00:47:20.512934+0000","last_clean":"2024-01-27T00:47:20.512934+0000","last_became_active":"2024-01-27T00:47:20.508998+0000","last_became_peered":"2024-01-27T00:47:20.508998+0000","last_unstale":"2024-01-27T00:47:20.512934+0000","last_undegraded":"2024-01-27T00:47:20.512934+0000","last_fullsized":"2024-01-27T00:47:20.512934+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:20.403842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.496693+0000","last_change":"2024-01-27T00:47:19.489957+0000","last_active":"2024-01-27T00:47:20.496693+0000","last_peered":"2024-01-27T00:47:20.496693+0000","last_clean":"2024-01-27T00:47:20.496693+0000","last_became_active":"2024-01-27T00:47:19.489686+0000","last_became_peered":"2024-01-27T00:47:19.489686+0000","last_unstale":"2024-01-27T00:47:20.496693+0000","last_undegraded":"2024-01-27T00:47:20.496693+0000","last_fullsized":"2024-01-27T00:47:20.496693+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:36:40.108567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.514859+0000","last_change":"2024-01-27T00:47:20.513843+0000","last_active":"2024-01-27T00:47:20.514859+0000","last_peered":"2024-01-27T00:47:20.514859+0000","last_clean":"2024-01-27T00:47:20.514859+0000","last_became_active":"2024-01-27T00:47:20.513188+0000","last_became_peered":"2024-01-27T00:47:20.513188+0000","last_unstale":"2024-01-27T00:47:20.514859+0000","last_undegraded":"2024-01-27T00:47:20.514859+0000","last_fullsized":"2024-01-27T00:47:20.514859+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:15:01.330526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"187'8","reported_seq":18,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.529029+0000","last_change":"2024-01-27T00:47:19.491898+0000","last_active":"2024-01-27T00:47:20.529029+0000","last_peered":"2024-01-27T00:47:20.529029+0000","last_clean":"2024-01-27T00:47:20.529029+0000","last_became_active":"2024-01-27T00:47:19.491597+0000","last_became_peered":"2024-01-27T00:47:19.491597+0000","last_unstale":"2024-01-27T00:47:20.529029+0000","last_undegraded":"2024-01-27T00:47:20.529029+0000","last_fullsized":"2024-01-27T00:47:20.529029+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:47:33.870316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524260+0000","last_change":"2024-01-27T00:47:20.511230+0000","last_active":"2024-01-27T00:47:20.524260+0000","last_peered":"2024-01-27T00:47:20.524260+0000","last_clean":"2024-01-27T00:47:20.524260+0000","last_became_active":"2024-01-27T00:47:20.510893+0000","last_became_peered":"2024-01-27T00:47:20.510893+0000","last_unstale":"2024-01-27T00:47:20.524260+0000","last_undegraded":"2024-01-27T00:47:20.524260+0000","last_fullsized":"2024-01-27T00:47:20.524260+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:51:40.896423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531660+0000","last_change":"2024-01-27T00:47:19.504875+0000","last_active":"2024-01-27T00:47:20.531660+0000","last_peered":"2024-01-27T00:47:20.531660+0000","last_clean":"2024-01-27T00:47:20.531660+0000","last_became_active":"2024-01-27T00:47:19.504575+0000","last_became_peered":"2024-01-27T00:47:19.504575+0000","last_unstale":"2024-01-27T00:47:20.531660+0000","last_undegraded":"2024-01-27T00:47:20.531660+0000","last_fullsized":"2024-01-27T00:47:20.531660+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:43:03.734740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.515064+0000","last_change":"2024-01-27T00:47:20.510860+0000","last_active":"2024-01-27T00:47:20.515064+0000","last_peered":"2024-01-27T00:47:20.515064+0000","last_clean":"2024-01-27T00:47:20.515064+0000","last_became_active":"2024-01-27T00:47:20.510567+0000","last_became_peered":"2024-01-27T00:47:20.510567+0000","last_unstale":"2024-01-27T00:47:20.515064+0000","last_undegraded":"2024-01-27T00:47:20.515064+0000","last_fullsized":"2024-01-27T00:47:20.515064+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:33:31.812775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.533275+0000","last_change":"2024-01-27T00:47:19.509498+0000","last_active":"2024-01-27T00:47:20.533275+0000","last_peered":"2024-01-27T00:47:20.533275+0000","last_clean":"2024-01-27T00:47:20.533275+0000","last_became_active":"2024-01-27T00:47:19.509151+0000","last_became_peered":"2024-01-27T00:47:19.509151+0000","last_unstale":"2024-01-27T00:47:20.533275+0000","last_undegraded":"2024-01-27T00:47:20.533275+0000","last_fullsized":"2024-01-27T00:47:20.533275+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:54:21.075786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512563+0000","last_change":"2024-01-27T00:47:20.510494+0000","last_active":"2024-01-27T00:47:20.512563+0000","last_peered":"2024-01-27T00:47:20.512563+0000","last_clean":"2024-01-27T00:47:20.512563+0000","last_became_active":"2024-01-27T00:47:20.510159+0000","last_became_peered":"2024-01-27T00:47:20.510159+0000","last_unstale":"2024-01-27T00:47:20.512563+0000","last_undegraded":"2024-01-27T00:47:20.512563+0000","last_fullsized":"2024-01-27T00:47:20.512563+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:08:01.982720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.529256+0000","last_change":"2024-01-27T00:47:19.498982+0000","last_active":"2024-01-27T00:47:20.529256+0000","last_peered":"2024-01-27T00:47:20.529256+0000","last_clean":"2024-01-27T00:47:20.529256+0000","last_became_active":"2024-01-27T00:47:19.498718+0000","last_became_peered":"2024-01-27T00:47:19.498718+0000","last_unstale":"2024-01-27T00:47:20.529256+0000","last_undegraded":"2024-01-27T00:47:20.529256+0000","last_fullsized":"2024-01-27T00:47:20.529256+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:04:42.913017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.517959+0000","last_change":"2024-01-27T00:47:20.514875+0000","last_active":"2024-01-27T00:47:20.517959+0000","last_peered":"2024-01-27T00:47:20.517959+0000","last_clean":"2024-01-27T00:47:20.517959+0000","last_became_active":"2024-01-27T00:47:20.514574+0000","last_became_peered":"2024-01-27T00:47:20.514574+0000","last_unstale":"2024-01-27T00:47:20.517959+0000","last_undegraded":"2024-01-27T00:47:20.517959+0000","last_fullsized":"2024-01-27T00:47:20.517959+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:47:34.874827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.493212+0000","last_change":"2024-01-27T00:47:19.493973+0000","last_active":"2024-01-27T00:47:20.493212+0000","last_peered":"2024-01-27T00:47:20.493212+0000","last_clean":"2024-01-27T00:47:20.493212+0000","last_became_active":"2024-01-27T00:47:19.493685+0000","last_became_peered":"2024-01-27T00:47:19.493685+0000","last_unstale":"2024-01-27T00:47:20.493212+0000","last_undegraded":"2024-01-27T00:47:20.493212+0000","last_fullsized":"2024-01-27T00:47:20.493212+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:08:00.978209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531595+0000","last_change":"2024-01-27T00:47:20.525394+0000","last_active":"2024-01-27T00:47:20.531595+0000","last_peered":"2024-01-27T00:47:20.531595+0000","last_clean":"2024-01-27T00:47:20.531595+0000","last_became_active":"2024-01-27T00:47:20.521446+0000","last_became_peered":"2024-01-27T00:47:20.521446+0000","last_unstale":"2024-01-27T00:47:20.531595+0000","last_undegraded":"2024-01-27T00:47:20.531595+0000","last_fullsized":"2024-01-27T00:47:20.531595+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:31.735208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.529633+0000","last_change":"2024-01-27T00:47:19.495677+0000","last_active":"2024-01-27T00:47:20.529633+0000","last_peered":"2024-01-27T00:47:20.529633+0000","last_clean":"2024-01-27T00:47:20.529633+0000","last_became_active":"2024-01-27T00:47:19.494329+0000","last_became_peered":"2024-01-27T00:47:19.494329+0000","last_unstale":"2024-01-27T00:47:20.529633+0000","last_undegraded":"2024-01-27T00:47:20.529633+0000","last_fullsized":"2024-01-27T00:47:20.529633+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:19:25.040151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522962+0000","last_change":"2024-01-27T00:47:20.512100+0000","last_active":"2024-01-27T00:47:20.522962+0000","last_peered":"2024-01-27T00:47:20.522962+0000","last_clean":"2024-01-27T00:47:20.522962+0000","last_became_active":"2024-01-27T00:47:20.511807+0000","last_became_peered":"2024-01-27T00:47:20.511807+0000","last_unstale":"2024-01-27T00:47:20.522962+0000","last_undegraded":"2024-01-27T00:47:20.522962+0000","last_fullsized":"2024-01-27T00:47:20.522962+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:41:19.485849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.498403+0000","last_change":"2024-01-27T00:47:19.495458+0000","last_active":"2024-01-27T00:47:20.498403+0000","last_peered":"2024-01-27T00:47:20.498403+0000","last_clean":"2024-01-27T00:47:20.498403+0000","last_became_active":"2024-01-27T00:47:19.495193+0000","last_became_peered":"2024-01-27T00:47:19.495193+0000","last_unstale":"2024-01-27T00:47:20.498403+0000","last_undegraded":"2024-01-27T00:47:20.498403+0000","last_fullsized":"2024-01-27T00:47:20.498403+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:12:18.301327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522508+0000","last_change":"2024-01-27T00:47:20.512072+0000","last_active":"2024-01-27T00:47:20.522508+0000","last_peered":"2024-01-27T00:47:20.522508+0000","last_clean":"2024-01-27T00:47:20.522508+0000","last_became_active":"2024-01-27T00:47:20.511716+0000","last_became_peered":"2024-01-27T00:47:20.511716+0000","last_unstale":"2024-01-27T00:47:20.522508+0000","last_undegraded":"2024-01-27T00:47:20.522508+0000","last_fullsized":"2024-01-27T00:47:20.522508+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:18:15.805971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.508031+0000","last_change":"2024-01-27T00:47:19.505685+0000","last_active":"2024-01-27T00:47:20.508031+0000","last_peered":"2024-01-27T00:47:20.508031+0000","last_clean":"2024-01-27T00:47:20.508031+0000","last_became_active":"2024-01-27T00:47:19.505201+0000","last_became_peered":"2024-01-27T00:47:19.505201+0000","last_unstale":"2024-01-27T00:47:20.508031+0000","last_undegraded":"2024-01-27T00:47:20.508031+0000","last_fullsized":"2024-01-27T00:47:20.508031+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:10:58.917172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.521282+0000","last_change":"2024-01-27T00:47:20.518813+0000","last_active":"2024-01-27T00:47:20.521282+0000","last_peered":"2024-01-27T00:47:20.521282+0000","last_clean":"2024-01-27T00:47:20.521282+0000","last_became_active":"2024-01-27T00:47:20.518485+0000","last_became_peered":"2024-01-27T00:47:20.518485+0000","last_unstale":"2024-01-27T00:47:20.521282+0000","last_undegraded":"2024-01-27T00:47:20.521282+0000","last_fullsized":"2024-01-27T00:47:20.521282+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:54:59.918544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.502196+0000","last_change":"2024-01-27T00:47:19.490229+0000","last_active":"2024-01-27T00:47:20.502196+0000","last_peered":"2024-01-27T00:47:20.502196+0000","last_clean":"2024-01-27T00:47:20.502196+0000","last_became_active":"2024-01-27T00:47:19.489969+0000","last_became_peered":"2024-01-27T00:47:19.489969+0000","last_unstale":"2024-01-27T00:47:20.502196+0000","last_undegraded":"2024-01-27T00:47:20.502196+0000","last_fullsized":"2024-01-27T00:47:20.502196+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:14:56.240226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512461+0000","last_change":"2024-01-27T00:47:20.504909+0000","last_active":"2024-01-27T00:47:20.512461+0000","last_peered":"2024-01-27T00:47:20.512461+0000","last_clean":"2024-01-27T00:47:20.512461+0000","last_became_active":"2024-01-27T00:47:20.504607+0000","last_became_peered":"2024-01-27T00:47:20.504607+0000","last_unstale":"2024-01-27T00:47:20.512461+0000","last_undegraded":"2024-01-27T00:47:20.512461+0000","last_fullsized":"2024-01-27T00:47:20.512461+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:36:33.703323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527493+0000","last_change":"2024-01-27T00:47:19.498236+0000","last_active":"2024-01-27T00:47:20.527493+0000","last_peered":"2024-01-27T00:47:20.527493+0000","last_clean":"2024-01-27T00:47:20.527493+0000","last_became_active":"2024-01-27T00:47:19.497955+0000","last_became_peered":"2024-01-27T00:47:19.497955+0000","last_unstale":"2024-01-27T00:47:20.527493+0000","last_undegraded":"2024-01-27T00:47:20.527493+0000","last_fullsized":"2024-01-27T00:47:20.527493+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:52:03.781390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524123+0000","last_change":"2024-01-27T00:47:20.510553+0000","last_active":"2024-01-27T00:47:20.524123+0000","last_peered":"2024-01-27T00:47:20.524123+0000","last_clean":"2024-01-27T00:47:20.524123+0000","last_became_active":"2024-01-27T00:47:20.510231+0000","last_became_peered":"2024-01-27T00:47:20.510231+0000","last_unstale":"2024-01-27T00:47:20.524123+0000","last_undegraded":"2024-01-27T00:47:20.524123+0000","last_fullsized":"2024-01-27T00:47:20.524123+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:15.349063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499847+0000","last_change":"2024-01-27T00:47:19.505767+0000","last_active":"2024-01-27T00:47:20.499847+0000","last_peered":"2024-01-27T00:47:20.499847+0000","last_clean":"2024-01-27T00:47:20.499847+0000","last_became_active":"2024-01-27T00:47:19.505313+0000","last_became_peered":"2024-01-27T00:47:19.505313+0000","last_unstale":"2024-01-27T00:47:20.499847+0000","last_undegraded":"2024-01-27T00:47:20.499847+0000","last_fullsized":"2024-01-27T00:47:20.499847+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:48:00.955590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.530928+0000","last_change":"2024-01-27T00:47:20.516018+0000","last_active":"2024-01-27T00:47:20.530928+0000","last_peered":"2024-01-27T00:47:20.530928+0000","last_clean":"2024-01-27T00:47:20.530928+0000","last_became_active":"2024-01-27T00:47:20.515703+0000","last_became_peered":"2024-01-27T00:47:20.515703+0000","last_unstale":"2024-01-27T00:47:20.530928+0000","last_undegraded":"2024-01-27T00:47:20.530928+0000","last_fullsized":"2024-01-27T00:47:20.530928+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:12:19.305838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499639+0000","last_change":"2024-01-27T00:47:19.499255+0000","last_active":"2024-01-27T00:47:20.499639+0000","last_peered":"2024-01-27T00:47:20.499639+0000","last_clean":"2024-01-27T00:47:20.499639+0000","last_became_active":"2024-01-27T00:47:19.498933+0000","last_became_peered":"2024-01-27T00:47:19.498933+0000","last_unstale":"2024-01-27T00:47:20.499639+0000","last_undegraded":"2024-01-27T00:47:20.499639+0000","last_fullsized":"2024-01-27T00:47:20.499639+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:45.555078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524255+0000","last_change":"2024-01-27T00:47:20.511077+0000","last_active":"2024-01-27T00:47:20.524255+0000","last_peered":"2024-01-27T00:47:20.524255+0000","last_clean":"2024-01-27T00:47:20.524255+0000","last_became_active":"2024-01-27T00:47:20.510732+0000","last_became_peered":"2024-01-27T00:47:20.510732+0000","last_unstale":"2024-01-27T00:47:20.524255+0000","last_undegraded":"2024-01-27T00:47:20.524255+0000","last_fullsized":"2024-01-27T00:47:20.524255+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:05.094009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.508098+0000","last_change":"2024-01-27T00:47:19.504494+0000","last_active":"2024-01-27T00:47:20.508098+0000","last_peered":"2024-01-27T00:47:20.508098+0000","last_clean":"2024-01-27T00:47:20.508098+0000","last_became_active":"2024-01-27T00:47:19.504090+0000","last_became_peered":"2024-01-27T00:47:19.504090+0000","last_unstale":"2024-01-27T00:47:20.508098+0000","last_undegraded":"2024-01-27T00:47:20.508098+0000","last_fullsized":"2024-01-27T00:47:20.508098+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:22:59.518350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.518283+0000","last_change":"2024-01-27T00:47:20.514544+0000","last_active":"2024-01-27T00:47:20.518283+0000","last_peered":"2024-01-27T00:47:20.518283+0000","last_clean":"2024-01-27T00:47:20.518283+0000","last_became_active":"2024-01-27T00:47:20.514270+0000","last_became_peered":"2024-01-27T00:47:20.514270+0000","last_unstale":"2024-01-27T00:47:20.518283+0000","last_undegraded":"2024-01-27T00:47:20.518283+0000","last_fullsized":"2024-01-27T00:47:20.518283+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:32:14.595404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.507239+0000","last_change":"2024-01-27T00:47:19.501914+0000","last_active":"2024-01-27T00:47:20.507239+0000","last_peered":"2024-01-27T00:47:20.507239+0000","last_clean":"2024-01-27T00:47:20.507239+0000","last_became_active":"2024-01-27T00:47:19.501663+0000","last_became_peered":"2024-01-27T00:47:19.501663+0000","last_unstale":"2024-01-27T00:47:20.507239+0000","last_undegraded":"2024-01-27T00:47:20.507239+0000","last_fullsized":"2024-01-27T00:47:20.507239+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:18:49.554812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522022+0000","last_change":"2024-01-27T00:47:20.519790+0000","last_active":"2024-01-27T00:47:20.522022+0000","last_peered":"2024-01-27T00:47:20.522022+0000","last_clean":"2024-01-27T00:47:20.522022+0000","last_became_active":"2024-01-27T00:47:20.519499+0000","last_became_peered":"2024-01-27T00:47:20.519499+0000","last_unstale":"2024-01-27T00:47:20.522022+0000","last_undegraded":"2024-01-27T00:47:20.522022+0000","last_fullsized":"2024-01-27T00:47:20.522022+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:48:01.960101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.498924+0000","last_change":"2024-01-27T00:47:19.503541+0000","last_active":"2024-01-27T00:47:20.498924+0000","last_peered":"2024-01-27T00:47:20.498924+0000","last_clean":"2024-01-27T00:47:20.498924+0000","last_became_active":"2024-01-27T00:47:19.503073+0000","last_became_peered":"2024-01-27T00:47:19.503073+0000","last_unstale":"2024-01-27T00:47:20.498924+0000","last_undegraded":"2024-01-27T00:47:20.498924+0000","last_fullsized":"2024-01-27T00:47:20.498924+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:14.344552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531950+0000","last_change":"2024-01-27T00:47:20.527327+0000","last_active":"2024-01-27T00:47:20.531950+0000","last_peered":"2024-01-27T00:47:20.531950+0000","last_clean":"2024-01-27T00:47:20.531950+0000","last_became_active":"2024-01-27T00:47:20.520248+0000","last_became_peered":"2024-01-27T00:47:20.520248+0000","last_unstale":"2024-01-27T00:47:20.531950+0000","last_undegraded":"2024-01-27T00:47:20.531950+0000","last_fullsized":"2024-01-27T00:47:20.531950+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:19:26.044662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.496693+0000","last_change":"2024-01-27T00:47:19.503480+0000","last_active":"2024-01-27T00:47:20.496693+0000","last_peered":"2024-01-27T00:47:20.496693+0000","last_clean":"2024-01-27T00:47:20.496693+0000","last_became_active":"2024-01-27T00:47:19.503158+0000","last_became_peered":"2024-01-27T00:47:19.503158+0000","last_unstale":"2024-01-27T00:47:20.496693+0000","last_undegraded":"2024-01-27T00:47:20.496693+0000","last_fullsized":"2024-01-27T00:47:20.496693+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:15:00.326015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.532846+0000","last_change":"2024-01-27T00:47:20.516514+0000","last_active":"2024-01-27T00:47:20.532846+0000","last_peered":"2024-01-27T00:47:20.532846+0000","last_clean":"2024-01-27T00:47:20.532846+0000","last_became_active":"2024-01-27T00:47:20.516160+0000","last_became_peered":"2024-01-27T00:47:20.516160+0000","last_unstale":"2024-01-27T00:47:20.532846+0000","last_undegraded":"2024-01-27T00:47:20.532846+0000","last_fullsized":"2024-01-27T00:47:20.532846+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:36:41.113078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.497052+0000","last_change":"2024-01-27T00:47:19.493487+0000","last_active":"2024-01-27T00:47:20.497052+0000","last_peered":"2024-01-27T00:47:20.497052+0000","last_clean":"2024-01-27T00:47:20.497052+0000","last_became_active":"2024-01-27T00:47:19.492721+0000","last_became_peered":"2024-01-27T00:47:19.492721+0000","last_unstale":"2024-01-27T00:47:20.497052+0000","last_undegraded":"2024-01-27T00:47:20.497052+0000","last_fullsized":"2024-01-27T00:47:20.497052+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:30.730697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.519011+0000","last_change":"2024-01-27T00:47:20.516157+0000","last_active":"2024-01-27T00:47:20.519011+0000","last_peered":"2024-01-27T00:47:20.519011+0000","last_clean":"2024-01-27T00:47:20.519011+0000","last_became_active":"2024-01-27T00:47:20.515871+0000","last_became_peered":"2024-01-27T00:47:20.515871+0000","last_unstale":"2024-01-27T00:47:20.519011+0000","last_undegraded":"2024-01-27T00:47:20.519011+0000","last_fullsized":"2024-01-27T00:47:20.519011+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:05:38.788368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499725+0000","last_change":"2024-01-27T00:47:19.501570+0000","last_active":"2024-01-27T00:47:20.499725+0000","last_peered":"2024-01-27T00:47:20.499725+0000","last_clean":"2024-01-27T00:47:20.499725+0000","last_became_active":"2024-01-27T00:47:19.501187+0000","last_became_peered":"2024-01-27T00:47:19.501187+0000","last_unstale":"2024-01-27T00:47:20.499725+0000","last_undegraded":"2024-01-27T00:47:20.499725+0000","last_fullsized":"2024-01-27T00:47:20.499725+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:54:58.914033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"179'8695","reported_seq":9436,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.501981+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T00:47:20.501981+0000","last_peered":"2024-01-27T00:47:20.501981+0000","last_clean":"2024-01-27T00:47:20.501981+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T00:47:20.501981+0000","last_undegraded":"2024-01-27T00:47:20.501981+0000","last_fullsized":"2024-01-27T00:47:20.501981+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8695,"log_dups_size":0,"ondisk_log_size":8695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1784,"num_read_kb":3778,"num_write":530,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1784,"num_read_kb":3778,"num_write":530,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8695,"ondisk_log_size":8695,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575776,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300168,"kb_used_data":2880,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071672,"statfs":{"total":96636764160,"available":96329392128,"internally_reserved":0,"allocated":2949120,"data_stored":1815604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60799999999999998,"15min":0.60699999999999998},"min":{"1min":0.36199999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75800000000000001,"5min":0.83199999999999996,"15min":0.86899999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61299999999999999,"15min":0.61799999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.755,"5min":0.82099999999999995,"15min":0.90000000000000002},"last":0.48899999999999999}]},{"osd":1,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.34599999999999997},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.90400000000000003},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63500000000000001,"15min":0.63},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.38200000000000001},"max":{"1min":0.76700000000000002,"5min":0.85099999999999998,"15min":0.875},"last":0.58299999999999996}]},{"osd":2,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.374,"5min":0.36299999999999999,"15min":0.34000000000000002},"max":{"1min":0.76400000000000001,"5min":0.77700000000000002,"15min":0.89700000000000002},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63800000000000001,"15min":0.63700000000000001},"min":{"1min":0.49399999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75600000000000001,"5min":0.86199999999999999,"15min":0.94199999999999995},"last":0.55800000000000005}]},{"osd":3,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.626,"15min":0.623},"min":{"1min":0.435,"5min":0.40999999999999998,"15min":0.39400000000000002},"max":{"1min":0.79200000000000004,"5min":0.86499999999999999,"15min":0.92600000000000005},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.77700000000000002,"5min":0.83499999999999996,"15min":0.88400000000000001},"last":0.59699999999999998}]},{"osd":4,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49399999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.32900000000000001,"15min":0.29699999999999999},"max":{"1min":0.61699999999999999,"5min":0.86499999999999999,"15min":0.92000000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.34599999999999997},"max":{"1min":0.76600000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.46700000000000003}]},{"osd":5,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":0.67100000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":0.67700000000000005,"5min":0.89100000000000001,"15min":0.95699999999999996},"last":0.63400000000000001}]},{"osd":6,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59299999999999997,"15min":0.58299999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.36599999999999999},"max":{"1min":0.69299999999999995,"5min":0.91700000000000004,"15min":0.99399999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.36699999999999999},"max":{"1min":0.73599999999999999,"5min":0.82299999999999995,"15min":0.878},"last":0.57299999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575776,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307780,"kb_used_data":2300,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064060,"statfs":{"total":96636764160,"available":96321597440,"internally_reserved":0,"allocated":2355200,"data_stored":1225236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58999999999999997,"15min":0.60399999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81000000000000005,"5min":0.82199999999999995,"15min":0.84899999999999998},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.38900000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.78500000000000003,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.73199999999999998}]},{"osd":1,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.748,"5min":0.79800000000000004,"15min":0.80800000000000005},"last":0.751},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81499999999999995,"5min":0.87,"15min":0.87},"last":0.79100000000000004}]},{"osd":2,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60199999999999998,"15min":0.60899999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83699999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59899999999999998,"15min":0.60599999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74199999999999999,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.69899999999999995}]},{"osd":3,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86299999999999999,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64600000000000002,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.80700000000000005}]},{"osd":4,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.496,"15min":0.49099999999999999},"min":{"1min":0.34300000000000003,"5min":0.33200000000000002,"15min":0.313},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32100000000000001},"max":{"1min":0.77700000000000002,"5min":0.79100000000000004,"15min":0.83199999999999996},"last":0.54400000000000004}]},{"osd":5,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51900000000000002,"15min":0.51000000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.32200000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56499999999999995,"15min":0.56000000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.88800000000000001},"last":0.59999999999999998}]},{"osd":7,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57399999999999995,"15min":0.56499999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.58399999999999996,"15min":0.58099999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.33600000000000002},"max":{"1min":0.84099999999999997,"5min":0.84599999999999997,"15min":0.85999999999999999},"last":0.621}]}]},{"osd":1,"up_from":13,"seq":55834575776,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311888,"kb_used_data":2312,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059952,"statfs":{"total":96636764160,"available":96317390848,"internally_reserved":0,"allocated":2367488,"data_stored":1226772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.41799999999999998},"min":{"1min":0.311,"5min":0.254,"15min":0.23499999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.434,"5min":0.441,"15min":0.44500000000000001},"min":{"1min":0.30199999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.60599999999999998,"5min":0.66200000000000003,"15min":0.70199999999999996},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.41899999999999998,"15min":0.42499999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55100000000000005,"5min":0.59999999999999998,"15min":0.68899999999999995},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48499999999999999,"15min":0.49399999999999999},"min":{"1min":0.33500000000000002,"5min":0.31,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.76700000000000002},"last":0.48499999999999999}]},{"osd":3,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.503,"15min":0.52100000000000002},"min":{"1min":0.39400000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.63900000000000001,"5min":0.69399999999999995,"15min":1.0469999999999999},"last":0.621},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.49099999999999999,"15min":0.5},"min":{"1min":0.318,"5min":0.318,"15min":0.309},"max":{"1min":0.69599999999999995,"5min":0.72599999999999998,"15min":0.95599999999999996},"last":0.56899999999999995}]},{"osd":4,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.66100000000000003,"15min":0.67900000000000005},"min":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86299999999999999,"5min":1.02,"15min":1.3129999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67700000000000005,"15min":0.69999999999999996},"min":{"1min":0.505,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.93500000000000005,"5min":1.0069999999999999,"15min":1.097},"last":0.68500000000000005}]},{"osd":5,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66700000000000004,"15min":0.69199999999999995},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.46300000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":1.091},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.36299999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.278},"last":0.64600000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.68500000000000005,"15min":0.71099999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.435},"max":{"1min":0.88100000000000001,"5min":0.96999999999999997,"15min":1.141},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70799999999999996,"15min":0.72099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.91400000000000003,"5min":1.0229999999999999,"15min":1.1120000000000001},"last":0.77900000000000003}]},{"osd":7,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.73299999999999998,"15min":0.749},"min":{"1min":0.52600000000000002,"5min":0.52500000000000002,"15min":0.51100000000000001},"max":{"1min":1.0129999999999999,"5min":1.054,"15min":1.1819999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.73499999999999999,"15min":0.75800000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.51200000000000001},"max":{"1min":0.98299999999999998,"5min":0.98999999999999999,"15min":1.198},"last":0.71199999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575775,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312464,"kb_used_data":2888,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059376,"statfs":{"total":96636764160,"available":96316801024,"internally_reserved":0,"allocated":2957312,"data_stored":1815708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.28299999999999997,"5min":0.25800000000000001,"15min":0.25},"max":{"1min":0.55500000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.34499999999999997,"5min":0.27500000000000002,"15min":0.26300000000000001},"max":{"1min":0.61299999999999999,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.56999999999999995}]},{"osd":2,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.42099999999999999,"15min":0.42599999999999999},"min":{"1min":0.312,"5min":0.28599999999999998,"15min":0.26700000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.69399999999999995},"last":0.621},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48099999999999998,"15min":0.48299999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.629}]},{"osd":3,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48999999999999999,"15min":0.48699999999999999},"min":{"1min":0.32900000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.65600000000000003,"5min":0.755,"15min":0.755},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.496,"15min":0.498},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.42199999999999999}]},{"osd":4,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65200000000000002,"15min":0.68100000000000005},"min":{"1min":0.45100000000000001,"5min":0.44700000000000001,"15min":0.44400000000000001},"max":{"1min":0.77500000000000002,"5min":0.97099999999999997,"15min":1.125},"last":0.496},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.66000000000000003,"15min":0.67800000000000005},"min":{"1min":0.46200000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.96199999999999997,"5min":0.97599999999999998,"15min":1.0169999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68100000000000005,"15min":0.69399999999999995},"min":{"1min":0.47899999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.84099999999999997,"5min":0.94199999999999995,"15min":1.054},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.70799999999999996,"15min":0.71899999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.98199999999999998,"5min":1.056,"15min":1.056},"last":0.46200000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69999999999999996,"15min":0.71199999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.84399999999999997,"5min":1.0269999999999999,"15min":1.087},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.72299999999999998,"15min":0.73899999999999999},"min":{"1min":0.52500000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.94599999999999995,"5min":1.0349999999999999,"15min":1.052},"last":0.88400000000000001}]},{"osd":7,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.747,"15min":0.75700000000000001},"min":{"1min":0.60599999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.016,"5min":1.0880000000000001,"15min":1.1080000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.53700000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.034,"5min":1.071,"15min":1.071},"last":0.55400000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575775,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311876,"kb_used_data":2300,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059964,"statfs":{"total":96636764160,"available":96317403136,"internally_reserved":0,"allocated":2355200,"data_stored":1225236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38100000000000001,"15min":0.39800000000000002},"min":{"1min":0.29499999999999998,"5min":0.25900000000000001,"15min":0.252},"max":{"1min":0.48699999999999999,"5min":0.60499999999999998,"15min":0.65800000000000003},"last":0.375},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":0.438},"min":{"1min":0.29499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.71899999999999997},"last":0.35599999999999998}]},{"osd":1,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.41999999999999998,"15min":0.432},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.60799999999999998,"5min":0.627,"15min":0.68300000000000005},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.5,"15min":0.51400000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66100000000000003,"5min":0.76400000000000001,"15min":0.80300000000000005},"last":0.40400000000000003}]},{"osd":3,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47799999999999998,"15min":0.49199999999999999},"min":{"1min":0.34999999999999998,"5min":0.34100000000000003,"15min":0.33100000000000002},"max":{"1min":0.65200000000000002,"5min":0.745,"15min":0.745},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47099999999999997,"15min":0.48699999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.29499999999999998},"max":{"1min":0.68000000000000005,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.44900000000000001}]},{"osd":4,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.65000000000000002,"15min":0.66700000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41999999999999998},"max":{"1min":0.77000000000000002,"5min":0.85899999999999999,"15min":1.0660000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66700000000000004,"15min":0.68200000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92400000000000004,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.83999999999999997}]},{"osd":5,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66800000000000004,"15min":0.68999999999999995},"min":{"1min":0.49099999999999999,"5min":0.46500000000000002,"15min":0.42899999999999999},"max":{"1min":0.78400000000000003,"5min":0.96499999999999997,"15min":1.0649999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68200000000000005,"15min":0.69799999999999995},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.90600000000000003,"5min":0.98799999999999999,"15min":1.044},"last":0.77600000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.68100000000000005,"15min":0.70099999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.86299999999999999,"5min":1.0569999999999999,"15min":1.1140000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.69399999999999995,"15min":0.70999999999999996},"min":{"1min":0.47999999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89400000000000002,"5min":1.002,"15min":1.0720000000000001},"last":0.79000000000000004}]},{"osd":7,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.71499999999999997,"15min":0.72799999999999998},"min":{"1min":0.55600000000000005,"5min":0.50900000000000001,"15min":0.48999999999999999},"max":{"1min":0.97499999999999998,"5min":1.0860000000000001,"15min":1.1479999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.73899999999999999,"15min":0.76200000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.98899999999999999,"5min":1.022,"15min":1.1299999999999999},"last":0.82799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575774,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315992,"kb_used_data":2320,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055848,"statfs":{"total":96636764160,"available":96313188352,"internally_reserved":0,"allocated":2375680,"data_stored":1227348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.39600000000000002,"15min":0.40100000000000002},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.50600000000000001,"5min":0.63,"15min":0.752},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.443,"5min":0.45700000000000002,"15min":0.45300000000000001},"min":{"1min":0.32900000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.57899999999999996,"5min":0.68799999999999994,"15min":0.85699999999999998},"last":0.69599999999999995}]},{"osd":1,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.46600000000000003},"min":{"1min":0.307,"5min":0.29699999999999999,"15min":0.27400000000000002},"max":{"1min":0.61499999999999999,"5min":0.69699999999999995,"15min":0.94599999999999995},"last":0.627},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.498,"15min":0.498},"min":{"1min":0.36299999999999999,"5min":0.34399999999999997,"15min":0.316},"max":{"1min":0.69299999999999995,"5min":0.745,"15min":0.745},"last":0.72999999999999998}]},{"osd":2,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.35699999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64900000000000002,"5min":0.67900000000000005,"15min":0.84499999999999997},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.46000000000000002,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.66600000000000004,"5min":0.72699999999999998,"15min":0.82299999999999995},"last":0.80400000000000005}]},{"osd":4,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.69299999999999995,"15min":0.68999999999999995},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.88400000000000001,"5min":0.89300000000000002,"15min":1},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.48199999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.98699999999999999,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.89700000000000002}]},{"osd":5,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.69299999999999995,"15min":0.68799999999999994},"min":{"1min":0.505,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.88900000000000001,"5min":0.94499999999999995,"15min":1.0429999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68400000000000005,"15min":0.68700000000000006},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90400000000000003,"5min":0.99099999999999999,"15min":1.0289999999999999},"last":1}]},{"osd":6,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.68300000000000005,"15min":0.69099999999999995},"min":{"1min":0.53000000000000003,"5min":0.52000000000000002,"15min":0.48899999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.0820000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.71799999999999997,"15min":0.71599999999999997},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.44},"max":{"1min":0.91600000000000004,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.95199999999999996}]},{"osd":7,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.69999999999999996,"15min":0.71399999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.96399999999999997,"5min":1.081,"15min":1.081},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.55800000000000005,"5min":0.48999999999999999,"15min":0.46300000000000002},"max":{"1min":0.91700000000000004,"5min":1.0229999999999999,"15min":1.1240000000000001},"last":0.98999999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575776,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311892,"kb_used_data":2316,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059948,"statfs":{"total":96636764160,"available":96317386752,"internally_reserved":0,"allocated":2371584,"data_stored":1225972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.32100000000000001},"max":{"1min":0.73799999999999999,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.59699999999999998,"15min":0.61499999999999999},"min":{"1min":0.42199999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81200000000000006,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.58999999999999997}]},{"osd":1,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61699999999999999,"15min":0.624},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61299999999999999,"15min":0.624},"min":{"1min":0.42399999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":0.56899999999999995}]},{"osd":2,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.58399999999999996,"15min":0.59999999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73799999999999999,"5min":0.81899999999999995,"15min":0.84499999999999997},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.62,"15min":0.63100000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.38100000000000001},"max":{"1min":0.81599999999999995,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.55400000000000005}]},{"osd":3,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.63400000000000001,"15min":0.628},"min":{"1min":0.43099999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.3860000000000001,"5min":2.3860000000000001,"15min":2.3860000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60699999999999998,"15min":0.62},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80200000000000005,"5min":0.80600000000000005,"15min":0.876},"last":0.45400000000000001}]},{"osd":5,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.48499999999999999,"15min":0.48599999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.28699999999999998},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.49199999999999999,"15min":0.49399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80200000000000005,"5min":0.876,"15min":0.876},"last":0.46500000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48899999999999999,"15min":0.49299999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.82399999999999995,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54000000000000004,"15min":0.53900000000000003},"min":{"1min":0.35099999999999998,"5min":0.34300000000000003,"15min":0.32800000000000001},"max":{"1min":0.746,"5min":1.028,"15min":1.028},"last":0.436}]},{"osd":7,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54700000000000004,"15min":0.55600000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33700000000000002},"max":{"1min":0.84299999999999997,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.49199999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575776,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311884,"kb_used_data":2308,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059956,"statfs":{"total":96636764160,"available":96317394944,"internally_reserved":0,"allocated":2363392,"data_stored":1225360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61399999999999999,"15min":0.60799999999999998},"min":{"1min":0.379,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76200000000000001,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.61599999999999999,"15min":0.623},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38600000000000001},"max":{"1min":0.76600000000000001,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.84999999999999998}]},{"osd":1,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65900000000000003,"15min":0.64900000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66400000000000003,"15min":0.67100000000000004},"min":{"1min":0.44600000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80300000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":1.056}]},{"osd":2,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.61399999999999999,"15min":0.60299999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.71999999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.79100000000000004,"5min":0.876,"15min":0.876},"last":0.754}]},{"osd":3,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60399999999999998,"15min":0.60299999999999998},"min":{"1min":0.38200000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.70099999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":1.008},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.64200000000000002,"15min":0.63400000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.79600000000000004,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.80800000000000005}]},{"osd":4,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.47499999999999998,"15min":0.47599999999999998},"min":{"1min":0.307,"5min":0.30399999999999999,"15min":0.30299999999999999},"max":{"1min":0.67600000000000005,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.5},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.82799999999999996},"last":0.59699999999999998}]},{"osd":6,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50700000000000001,"15min":0.504},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.31900000000000001},"max":{"1min":0.67400000000000004,"5min":0.71399999999999997,"15min":0.79900000000000004},"last":0.624},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.379,"5min":0.36799999999999999,"15min":0.36099999999999999},"max":{"1min":0.70099999999999996,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.65800000000000003}]},{"osd":7,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34699999999999998},"max":{"1min":0.747,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.45800000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.92200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T00:47:29.400 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T00:47:29.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.540+0000 7f3ad94af700 1 Processor -- start 2024-01-27T00:47:29.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.540+0000 7f3ad94af700 1 -- start start 2024-01-27T00:47:29.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.541+0000 7f3ad94af700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ad40c6520 0x7f3ad40ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:29.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.541+0000 7f3ad94af700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ad40ced50 0x7f3ad40cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:29.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.541+0000 7f3ad94af700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad415a0b0 con 0x7f3ad40b2b40 2024-01-27T00:47:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.541+0000 7f3ad94af700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ad40af070 con 0x7f3ad40ced50 2024-01-27T00:47:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.541+0000 7f3ad94af700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad4099fc0 con 0x7f3ad40c6520 2024-01-27T00:47:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.541+0000 7f3ad37fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ad40ced50 0x7f3ad40cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.541+0000 7f3ad2ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3ad40b2b40 0x7f3ad40ae960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45680/0 (socket says 172.21.15.46:45680) 2024-01-27T00:47:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.541+0000 7f3ad2ffd700 1 -- 172.21.15.46:0/3221891508 learned_addr learned my addr 172.21.15.46:0/3221891508 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.541+0000 7f3ad27fc700 1 --2- 172.21.15.46:0/3221891508 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ad40c6520 0x7f3ad40ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.542+0000 7f3ad37fe700 1 -- 172.21.15.46:0/3221891508 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ad40c6520 msgr2=0x7f3ad40ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.542+0000 7f3ad37fe700 1 --2- 172.21.15.46:0/3221891508 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ad40c6520 0x7f3ad40ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.542+0000 7f3ad37fe700 1 -- 172.21.15.46:0/3221891508 >> v1:172.21.15.46:6789/0 conn(0x7f3ad40b2b40 legacy=0x7f3ad40ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.542+0000 7f3ad37fe700 1 -- 172.21.15.46:0/3221891508 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad40ae4d0 con 0x7f3ad40ced50 2024-01-27T00:47:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.542+0000 7f3ad1ffb700 1 -- 172.21.15.46:0/3221891508 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156606550 0 0) 0x7f3ad415a0b0 con 0x7f3ad40b2b40 2024-01-27T00:47:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.542+0000 7f3ad27fc700 1 --2- 172.21.15.46:0/3221891508 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ad40c6520 0x7f3ad40ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:47:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.542+0000 7f3ad37fe700 1 --2- 172.21.15.46:0/3221891508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad40ced50 0x7f3ad40cf1d0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f3acc009e60 tx=0x7f3acc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=144a2a6a512518c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.542+0000 7f3ad1ffb700 1 -- 172.21.15.46:0/3221891508 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3acc00e070 con 0x7f3ad40ced50 2024-01-27T00:47:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.542+0000 7f3ad1ffb700 1 -- 172.21.15.46:0/3221891508 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3acc00bc30 con 0x7f3ad40ced50 2024-01-27T00:47:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.542+0000 7f3ad1ffb700 1 -- 172.21.15.46:0/3221891508 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3acc012430 con 0x7f3ad40ced50 2024-01-27T00:47:29.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.543+0000 7f3ad94af700 1 -- 172.21.15.46:0/3221891508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad40ced50 msgr2=0x7f3ad40cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:29.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.543+0000 7f3ad94af700 1 --2- 172.21.15.46:0/3221891508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad40ced50 0x7f3ad40cf1d0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f3acc009e60 tx=0x7f3acc009780 comp rx=0 tx=0).stop 2024-01-27T00:47:29.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.543+0000 7f3ad94af700 1 -- 172.21.15.46:0/3221891508 shutdown_connections 2024-01-27T00:47:29.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.543+0000 7f3ad94af700 1 --2- 172.21.15.46:0/3221891508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad40ced50 0x7f3ad40cf1d0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:29.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.543+0000 7f3ad94af700 1 --2- 172.21.15.46:0/3221891508 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ad40c6520 0x7f3ad40ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:29.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.543+0000 7f3ad94af700 1 -- 172.21.15.46:0/3221891508 >> 172.21.15.46:0/3221891508 conn(0x7f3ad4099a30 msgr2=0x7f3ad406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:29.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.543+0000 7f3ad94af700 1 -- 172.21.15.46:0/3221891508 shutdown_connections 2024-01-27T00:47:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.543+0000 7f3ad94af700 1 -- 172.21.15.46:0/3221891508 wait complete. 2024-01-27T00:47:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.544+0000 7f3ad94af700 1 Processor -- start 2024-01-27T00:47:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.544+0000 7f3ad94af700 1 -- start start 2024-01-27T00:47:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.545+0000 7f3ad94af700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad40b2b40 0x7f3ad4153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.545+0000 7f3ad94af700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ad40c6520 0x7f3ad40b2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.545+0000 7f3ad94af700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ad40ced50 0x7f3ad40cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.545+0000 7f3ad94af700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad4157f20 con 0x7f3ad40b2b40 2024-01-27T00:47:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.545+0000 7f3ad94af700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad40c8730 con 0x7f3ad40ced50 2024-01-27T00:47:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.545+0000 7f3ad94af700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad40c8a10 con 0x7f3ad40c6520 2024-01-27T00:47:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.545+0000 7f3ad2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad40b2b40 0x7f3ad4153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.545+0000 7f3ad2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad40b2b40 0x7f3ad4153860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40626/0 (socket says 172.21.15.46:40626) 2024-01-27T00:47:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.545+0000 7f3ad2ffd700 1 -- 172.21.15.46:0/1386633722 learned_addr learned my addr 172.21.15.46:0/1386633722 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.545+0000 7f3ad27fc700 1 --2- 172.21.15.46:0/1386633722 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ad40c6520 0x7f3ad40b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.545+0000 7f3ad37fe700 1 --2- 172.21.15.46:0/1386633722 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ad40ced50 0x7f3ad40cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.545+0000 7f3ad2ffd700 1 -- 172.21.15.46:0/1386633722 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ad40c6520 msgr2=0x7f3ad40b2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.545+0000 7f3ad2ffd700 1 --2- 172.21.15.46:0/1386633722 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ad40c6520 0x7f3ad40b2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.545+0000 7f3ad2ffd700 1 -- 172.21.15.46:0/1386633722 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ad40ced50 msgr2=0x7f3ad40cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.545+0000 7f3ad2ffd700 1 --2- 172.21.15.46:0/1386633722 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ad40ced50 0x7f3ad40cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.545+0000 7f3ad2ffd700 1 -- 172.21.15.46:0/1386633722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc009a20 con 0x7f3ad40b2b40 2024-01-27T00:47:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.546+0000 7f3ad2ffd700 1 --2- 172.21.15.46:0/1386633722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad40b2b40 0x7f3ad4153860 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400c7e0 tx=0x7f3ac4007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.546+0000 7f3abaffd700 1 -- 172.21.15.46:0/1386633722 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac40072e0 con 0x7f3ad40b2b40 2024-01-27T00:47:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.546+0000 7f3abaffd700 1 -- 172.21.15.46:0/1386633722 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac4007460 con 0x7f3ad40b2b40 2024-01-27T00:47:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.546+0000 7f3ad94af700 1 -- 172.21.15.46:0/1386633722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad4157bb0 con 0x7f3ad40b2b40 2024-01-27T00:47:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.546+0000 7f3abaffd700 1 -- 172.21.15.46:0/1386633722 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac40057f0 con 0x7f3ad40b2b40 2024-01-27T00:47:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.546+0000 7f3ad94af700 1 -- 172.21.15.46:0/1386633722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad40af1b0 con 0x7f3ad40b2b40 2024-01-27T00:47:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.547+0000 7f3ad94af700 1 -- 172.21.15.46:0/1386633722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad4089ea0 con 0x7f3ad40b2b40 2024-01-27T00:47:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.547+0000 7f3abaffd700 1 -- 172.21.15.46:0/1386633722 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3ac4005b60 con 0x7f3ad40b2b40 2024-01-27T00:47:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.548+0000 7f3abaffd700 1 --2- 172.21.15.46:0/1386633722 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3ac0066f30 0x7f3ac00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.548+0000 7f3abaffd700 1 -- 172.21.15.46:0/1386633722 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f3ac4001e40 con 0x7f3ad40b2b40 2024-01-27T00:47:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.548+0000 7f3ad27fc700 1 --2- 172.21.15.46:0/1386633722 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3ac0066f30 0x7f3ac00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.549+0000 7f3ad27fc700 1 --2- 172.21.15.46:0/1386633722 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3ac0066f30 0x7f3ac00693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3ad40b31f0 tx=0x7f3ac8008040 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:29.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.552+0000 7f3abaffd700 1 -- 172.21.15.46:0/1386633722 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3ac404b5a0 con 0x7f3ad40b2b40 2024-01-27T00:47:29.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.753+0000 7f3ad94af700 1 -- 172.21.15.46:0/1386633722 --> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3ad4157a30 con 0x7f3ac0066f30 2024-01-27T00:47:29.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.761+0000 7f3abaffd700 1 -- 172.21.15.46:0/1386633722 <== mgr.9313 v2:172.21.15.106:6836/43388 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289999 (secure 0 0 0) 0x7f3ac8011700 con 0x7f3ac0066f30 2024-01-27T00:47:29.765 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.764+0000 7f3ad94af700 1 -- 172.21.15.46:0/1386633722 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3ac0066f30 msgr2=0x7f3ac00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.764+0000 7f3ad94af700 1 --2- 172.21.15.46:0/1386633722 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3ac0066f30 0x7f3ac00693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3ad40b31f0 tx=0x7f3ac8008040 comp rx=0 tx=0).stop 2024-01-27T00:47:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.764+0000 7f3ad94af700 1 -- 172.21.15.46:0/1386633722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad40b2b40 msgr2=0x7f3ad4153860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.764+0000 7f3ad94af700 1 --2- 172.21.15.46:0/1386633722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad40b2b40 0x7f3ad4153860 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400c7e0 tx=0x7f3ac4007c80 comp rx=0 tx=0).stop 2024-01-27T00:47:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.765+0000 7f3ad94af700 1 -- 172.21.15.46:0/1386633722 shutdown_connections 2024-01-27T00:47:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.765+0000 7f3ad94af700 1 --2- 172.21.15.46:0/1386633722 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3ac0066f30 0x7f3ac00693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:29.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.765+0000 7f3ad94af700 1 --2- 172.21.15.46:0/1386633722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ad40b2b40 0x7f3ad4153860 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:29.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.765+0000 7f3ad94af700 1 --2- 172.21.15.46:0/1386633722 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ad40ced50 0x7f3ad40cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:29.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.765+0000 7f3ad94af700 1 --2- 172.21.15.46:0/1386633722 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ad40c6520 0x7f3ad40b2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:29.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.765+0000 7f3ad94af700 1 -- 172.21.15.46:0/1386633722 >> 172.21.15.46:0/1386633722 conn(0x7f3ad4099a30 msgr2=0x7f3ad41556c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:29.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.765+0000 7f3ad94af700 1 -- 172.21.15.46:0/1386633722 shutdown_connections 2024-01-27T00:47:29.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.765+0000 7f3ad94af700 1 -- 172.21.15.46:0/1386633722 wait complete. 2024-01-27T00:47:29.768 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T00:47:29.778 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-01-27T00:47:29.572483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1784,"num_read_kb":3778,"num_write":575,"num_write_kb":3475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8734,"ondisk_log_size":8734,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2483944,"kb_used_data":19624,"kb_used_omap":0,"kb_used_meta":2464256,"kb_avail":752490776,"statfs":{"total":773094113280,"available":770550554624,"internally_reserved":0,"allocated":20094976,"data_stored":10987236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2523398144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.077772"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512073+0000","last_change":"2024-01-27T00:47:20.509860+0000","last_active":"2024-01-27T00:47:20.512073+0000","last_peered":"2024-01-27T00:47:20.512073+0000","last_clean":"2024-01-27T00:47:20.512073+0000","last_became_active":"2024-01-27T00:47:20.509586+0000","last_became_peered":"2024-01-27T00:47:20.509586+0000","last_unstale":"2024-01-27T00:47:20.512073+0000","last_undegraded":"2024-01-27T00:47:20.512073+0000","last_fullsized":"2024-01-27T00:47:20.512073+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:38:23.210445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.518716+0000","last_change":"2024-01-27T00:47:20.514877+0000","last_active":"2024-01-27T00:47:20.518716+0000","last_peered":"2024-01-27T00:47:20.518716+0000","last_clean":"2024-01-27T00:47:20.518716+0000","last_became_active":"2024-01-27T00:47:20.514587+0000","last_became_peered":"2024-01-27T00:47:20.514587+0000","last_unstale":"2024-01-27T00:47:20.518716+0000","last_undegraded":"2024-01-27T00:47:20.518716+0000","last_fullsized":"2024-01-27T00:47:20.518716+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:44.855002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.518231+0000","last_change":"2024-01-27T00:47:20.514155+0000","last_active":"2024-01-27T00:47:20.518231+0000","last_peered":"2024-01-27T00:47:20.518231+0000","last_clean":"2024-01-27T00:47:20.518231+0000","last_became_active":"2024-01-27T00:47:20.513885+0000","last_became_peered":"2024-01-27T00:47:20.513885+0000","last_unstale":"2024-01-27T00:47:20.518231+0000","last_undegraded":"2024-01-27T00:47:20.518231+0000","last_fullsized":"2024-01-27T00:47:20.518231+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:49:42.939991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.511955+0000","last_change":"2024-01-27T00:47:20.509503+0000","last_active":"2024-01-27T00:47:20.511955+0000","last_peered":"2024-01-27T00:47:20.511955+0000","last_clean":"2024-01-27T00:47:20.511955+0000","last_became_active":"2024-01-27T00:47:20.509094+0000","last_became_peered":"2024-01-27T00:47:20.509094+0000","last_unstale":"2024-01-27T00:47:20.511955+0000","last_undegraded":"2024-01-27T00:47:20.511955+0000","last_fullsized":"2024-01-27T00:47:20.511955+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:53:55.794828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.523334+0000","last_change":"2024-01-27T00:47:20.509897+0000","last_active":"2024-01-27T00:47:20.523334+0000","last_peered":"2024-01-27T00:47:20.523334+0000","last_clean":"2024-01-27T00:47:20.523334+0000","last_became_active":"2024-01-27T00:47:20.509610+0000","last_became_peered":"2024-01-27T00:47:20.509610+0000","last_unstale":"2024-01-27T00:47:20.523334+0000","last_undegraded":"2024-01-27T00:47:20.523334+0000","last_fullsized":"2024-01-27T00:47:20.523334+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:42:01.971654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.518008+0000","last_change":"2024-01-27T00:47:20.514363+0000","last_active":"2024-01-27T00:47:20.518008+0000","last_peered":"2024-01-27T00:47:20.518008+0000","last_clean":"2024-01-27T00:47:20.518008+0000","last_became_active":"2024-01-27T00:47:20.514085+0000","last_became_peered":"2024-01-27T00:47:20.514085+0000","last_unstale":"2024-01-27T00:47:20.518008+0000","last_undegraded":"2024-01-27T00:47:20.518008+0000","last_fullsized":"2024-01-27T00:47:20.518008+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:33.654223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527541+0000","last_change":"2024-01-27T00:47:20.514165+0000","last_active":"2024-01-27T00:47:20.527541+0000","last_peered":"2024-01-27T00:47:20.527541+0000","last_clean":"2024-01-27T00:47:20.527541+0000","last_became_active":"2024-01-27T00:47:20.513848+0000","last_became_peered":"2024-01-27T00:47:20.513848+0000","last_unstale":"2024-01-27T00:47:20.527541+0000","last_undegraded":"2024-01-27T00:47:20.527541+0000","last_fullsized":"2024-01-27T00:47:20.527541+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:43:04.739251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524282+0000","last_change":"2024-01-27T00:47:20.511666+0000","last_active":"2024-01-27T00:47:20.524282+0000","last_peered":"2024-01-27T00:47:20.524282+0000","last_clean":"2024-01-27T00:47:20.524282+0000","last_became_active":"2024-01-27T00:47:20.509826+0000","last_became_peered":"2024-01-27T00:47:20.509826+0000","last_unstale":"2024-01-27T00:47:20.524282+0000","last_undegraded":"2024-01-27T00:47:20.524282+0000","last_fullsized":"2024-01-27T00:47:20.524282+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:53:47.125547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524159+0000","last_change":"2024-01-27T00:47:20.511731+0000","last_active":"2024-01-27T00:47:20.524159+0000","last_peered":"2024-01-27T00:47:20.524159+0000","last_clean":"2024-01-27T00:47:20.524159+0000","last_became_active":"2024-01-27T00:47:20.511436+0000","last_became_peered":"2024-01-27T00:47:20.511436+0000","last_unstale":"2024-01-27T00:47:20.524159+0000","last_undegraded":"2024-01-27T00:47:20.524159+0000","last_fullsized":"2024-01-27T00:47:20.524159+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:40.342669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524015+0000","last_change":"2024-01-27T00:47:20.510790+0000","last_active":"2024-01-27T00:47:20.524015+0000","last_peered":"2024-01-27T00:47:20.524015+0000","last_clean":"2024-01-27T00:47:20.524015+0000","last_became_active":"2024-01-27T00:47:20.510447+0000","last_became_peered":"2024-01-27T00:47:20.510447+0000","last_unstale":"2024-01-27T00:47:20.524015+0000","last_undegraded":"2024-01-27T00:47:20.524015+0000","last_fullsized":"2024-01-27T00:47:20.524015+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:56:44.894090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512593+0000","last_change":"2024-01-27T00:47:20.508709+0000","last_active":"2024-01-27T00:47:20.512593+0000","last_peered":"2024-01-27T00:47:20.512593+0000","last_clean":"2024-01-27T00:47:20.512593+0000","last_became_active":"2024-01-27T00:47:20.508433+0000","last_became_peered":"2024-01-27T00:47:20.508433+0000","last_unstale":"2024-01-27T00:47:20.512593+0000","last_undegraded":"2024-01-27T00:47:20.512593+0000","last_fullsized":"2024-01-27T00:47:20.512593+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:33:01.452427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.517995+0000","last_change":"2024-01-27T00:47:20.513994+0000","last_active":"2024-01-27T00:47:20.517995+0000","last_peered":"2024-01-27T00:47:20.517995+0000","last_clean":"2024-01-27T00:47:20.517995+0000","last_became_active":"2024-01-27T00:47:20.513691+0000","last_became_peered":"2024-01-27T00:47:20.513691+0000","last_unstale":"2024-01-27T00:47:20.517995+0000","last_undegraded":"2024-01-27T00:47:20.517995+0000","last_fullsized":"2024-01-27T00:47:20.517995+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:53:47.125547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531204+0000","last_change":"2024-01-27T00:47:20.520667+0000","last_active":"2024-01-27T00:47:20.531204+0000","last_peered":"2024-01-27T00:47:20.531204+0000","last_clean":"2024-01-27T00:47:20.531204+0000","last_became_active":"2024-01-27T00:47:20.519891+0000","last_became_peered":"2024-01-27T00:47:20.519891+0000","last_unstale":"2024-01-27T00:47:20.531204+0000","last_undegraded":"2024-01-27T00:47:20.531204+0000","last_fullsized":"2024-01-27T00:47:20.531204+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:49:22.885968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522694+0000","last_change":"2024-01-27T00:47:20.513550+0000","last_active":"2024-01-27T00:47:20.522694+0000","last_peered":"2024-01-27T00:47:20.522694+0000","last_clean":"2024-01-27T00:47:20.522694+0000","last_became_active":"2024-01-27T00:47:20.512667+0000","last_became_peered":"2024-01-27T00:47:20.512667+0000","last_unstale":"2024-01-27T00:47:20.522694+0000","last_undegraded":"2024-01-27T00:47:20.522694+0000","last_fullsized":"2024-01-27T00:47:20.522694+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:09:43.978584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512205+0000","last_change":"2024-01-27T00:47:20.506077+0000","last_active":"2024-01-27T00:47:20.512205+0000","last_peered":"2024-01-27T00:47:20.512205+0000","last_clean":"2024-01-27T00:47:20.512205+0000","last_became_active":"2024-01-27T00:47:20.505787+0000","last_became_peered":"2024-01-27T00:47:20.505787+0000","last_unstale":"2024-01-27T00:47:20.512205+0000","last_undegraded":"2024-01-27T00:47:20.512205+0000","last_fullsized":"2024-01-27T00:47:20.512205+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:44:21.535352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.511915+0000","last_change":"2024-01-27T00:47:20.509025+0000","last_active":"2024-01-27T00:47:20.511915+0000","last_peered":"2024-01-27T00:47:20.511915+0000","last_clean":"2024-01-27T00:47:20.511915+0000","last_became_active":"2024-01-27T00:47:20.508726+0000","last_became_peered":"2024-01-27T00:47:20.508726+0000","last_unstale":"2024-01-27T00:47:20.511915+0000","last_undegraded":"2024-01-27T00:47:20.511915+0000","last_fullsized":"2024-01-27T00:47:20.511915+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:55:06.853231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.504493+0000","last_change":"2024-01-27T00:47:20.500113+0000","last_active":"2024-01-27T00:47:20.504493+0000","last_peered":"2024-01-27T00:47:20.504493+0000","last_clean":"2024-01-27T00:47:20.504493+0000","last_became_active":"2024-01-27T00:47:20.499814+0000","last_became_peered":"2024-01-27T00:47:20.499814+0000","last_unstale":"2024-01-27T00:47:20.504493+0000","last_undegraded":"2024-01-27T00:47:20.504493+0000","last_fullsized":"2024-01-27T00:47:20.504493+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:05:38.788368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527940+0000","last_change":"2024-01-27T00:47:20.524935+0000","last_active":"2024-01-27T00:47:20.527940+0000","last_peered":"2024-01-27T00:47:20.527940+0000","last_clean":"2024-01-27T00:47:20.527940+0000","last_became_active":"2024-01-27T00:47:20.524672+0000","last_became_peered":"2024-01-27T00:47:20.524672+0000","last_unstale":"2024-01-27T00:47:20.527940+0000","last_undegraded":"2024-01-27T00:47:20.527940+0000","last_fullsized":"2024-01-27T00:47:20.527940+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:28.323536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512193+0000","last_change":"2024-01-27T00:47:20.506031+0000","last_active":"2024-01-27T00:47:20.512193+0000","last_peered":"2024-01-27T00:47:20.512193+0000","last_clean":"2024-01-27T00:47:20.512193+0000","last_became_active":"2024-01-27T00:47:20.505758+0000","last_became_peered":"2024-01-27T00:47:20.505758+0000","last_unstale":"2024-01-27T00:47:20.512193+0000","last_undegraded":"2024-01-27T00:47:20.512193+0000","last_fullsized":"2024-01-27T00:47:20.512193+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:18:50.559323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.532911+0000","last_change":"2024-01-27T00:47:20.518490+0000","last_active":"2024-01-27T00:47:20.532911+0000","last_peered":"2024-01-27T00:47:20.532911+0000","last_clean":"2024-01-27T00:47:20.532911+0000","last_became_active":"2024-01-27T00:47:20.518138+0000","last_became_peered":"2024-01-27T00:47:20.518138+0000","last_unstale":"2024-01-27T00:47:20.532911+0000","last_undegraded":"2024-01-27T00:47:20.532911+0000","last_fullsized":"2024-01-27T00:47:20.532911+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:28.557943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531230+0000","last_change":"2024-01-27T00:47:20.520931+0000","last_active":"2024-01-27T00:47:20.531230+0000","last_peered":"2024-01-27T00:47:20.531230+0000","last_clean":"2024-01-27T00:47:20.531230+0000","last_became_active":"2024-01-27T00:47:20.520322+0000","last_became_peered":"2024-01-27T00:47:20.520322+0000","last_unstale":"2024-01-27T00:47:20.531230+0000","last_undegraded":"2024-01-27T00:47:20.531230+0000","last_fullsized":"2024-01-27T00:47:20.531230+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:04:43.917528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.528575+0000","last_change":"2024-01-27T00:47:20.523687+0000","last_active":"2024-01-27T00:47:20.528575+0000","last_peered":"2024-01-27T00:47:20.528575+0000","last_clean":"2024-01-27T00:47:20.528575+0000","last_became_active":"2024-01-27T00:47:20.523382+0000","last_became_peered":"2024-01-27T00:47:20.523382+0000","last_unstale":"2024-01-27T00:47:20.528575+0000","last_undegraded":"2024-01-27T00:47:20.528575+0000","last_fullsized":"2024-01-27T00:47:20.528575+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:05.094009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.530444+0000","last_change":"2024-01-27T00:47:20.522803+0000","last_active":"2024-01-27T00:47:20.530444+0000","last_peered":"2024-01-27T00:47:20.530444+0000","last_clean":"2024-01-27T00:47:20.530444+0000","last_became_active":"2024-01-27T00:47:20.522463+0000","last_became_peered":"2024-01-27T00:47:20.522463+0000","last_unstale":"2024-01-27T00:47:20.530444+0000","last_undegraded":"2024-01-27T00:47:20.530444+0000","last_fullsized":"2024-01-27T00:47:20.530444+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:04:18.616847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.511248+0000","last_change":"2024-01-27T00:47:20.508251+0000","last_active":"2024-01-27T00:47:20.511248+0000","last_peered":"2024-01-27T00:47:20.511248+0000","last_clean":"2024-01-27T00:47:20.511248+0000","last_became_active":"2024-01-27T00:47:20.507933+0000","last_became_peered":"2024-01-27T00:47:20.507933+0000","last_unstale":"2024-01-27T00:47:20.511248+0000","last_undegraded":"2024-01-27T00:47:20.511248+0000","last_fullsized":"2024-01-27T00:47:20.511248+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:22.403262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.504894+0000","last_change":"2024-01-27T00:47:20.503158+0000","last_active":"2024-01-27T00:47:20.504894+0000","last_peered":"2024-01-27T00:47:20.504894+0000","last_clean":"2024-01-27T00:47:20.504894+0000","last_became_active":"2024-01-27T00:47:20.502875+0000","last_became_peered":"2024-01-27T00:47:20.502875+0000","last_unstale":"2024-01-27T00:47:20.504894+0000","last_undegraded":"2024-01-27T00:47:20.504894+0000","last_fullsized":"2024-01-27T00:47:20.504894+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:33:13.706183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.513540+0000","last_change":"2024-01-27T00:47:20.509928+0000","last_active":"2024-01-27T00:47:20.513540+0000","last_peered":"2024-01-27T00:47:20.513540+0000","last_clean":"2024-01-27T00:47:20.513540+0000","last_became_active":"2024-01-27T00:47:20.509630+0000","last_became_peered":"2024-01-27T00:47:20.509630+0000","last_unstale":"2024-01-27T00:47:20.513540+0000","last_undegraded":"2024-01-27T00:47:20.513540+0000","last_fullsized":"2024-01-27T00:47:20.513540+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:02:52.058658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.528592+0000","last_change":"2024-01-27T00:47:20.524118+0000","last_active":"2024-01-27T00:47:20.528592+0000","last_peered":"2024-01-27T00:47:20.528592+0000","last_clean":"2024-01-27T00:47:20.528592+0000","last_became_active":"2024-01-27T00:47:20.523779+0000","last_became_peered":"2024-01-27T00:47:20.523779+0000","last_unstale":"2024-01-27T00:47:20.528592+0000","last_undegraded":"2024-01-27T00:47:20.528592+0000","last_fullsized":"2024-01-27T00:47:20.528592+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:43:04.739251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531770+0000","last_change":"2024-01-27T00:47:20.527150+0000","last_active":"2024-01-27T00:47:20.531770+0000","last_peered":"2024-01-27T00:47:20.531770+0000","last_clean":"2024-01-27T00:47:20.531770+0000","last_became_active":"2024-01-27T00:47:20.523827+0000","last_became_peered":"2024-01-27T00:47:20.523827+0000","last_unstale":"2024-01-27T00:47:20.531770+0000","last_undegraded":"2024-01-27T00:47:20.531770+0000","last_fullsized":"2024-01-27T00:47:20.531770+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:52:04.785901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.517901+0000","last_change":"2024-01-27T00:47:20.513610+0000","last_active":"2024-01-27T00:47:20.517901+0000","last_peered":"2024-01-27T00:47:20.517901+0000","last_clean":"2024-01-27T00:47:20.517901+0000","last_became_active":"2024-01-27T00:47:20.513327+0000","last_became_peered":"2024-01-27T00:47:20.513327+0000","last_unstale":"2024-01-27T00:47:20.517901+0000","last_undegraded":"2024-01-27T00:47:20.517901+0000","last_fullsized":"2024-01-27T00:47:20.517901+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:15.349063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.523328+0000","last_change":"2024-01-27T00:47:20.522714+0000","last_active":"2024-01-27T00:47:20.523328+0000","last_peered":"2024-01-27T00:47:20.523328+0000","last_clean":"2024-01-27T00:47:20.523328+0000","last_became_active":"2024-01-27T00:47:20.522460+0000","last_became_peered":"2024-01-27T00:47:20.522460+0000","last_unstale":"2024-01-27T00:47:20.523328+0000","last_undegraded":"2024-01-27T00:47:20.523328+0000","last_fullsized":"2024-01-27T00:47:20.523328+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:40.342669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512795+0000","last_change":"2024-01-27T00:47:20.510711+0000","last_active":"2024-01-27T00:47:20.512795+0000","last_peered":"2024-01-27T00:47:20.512795+0000","last_clean":"2024-01-27T00:47:20.512795+0000","last_became_active":"2024-01-27T00:47:20.510442+0000","last_became_peered":"2024-01-27T00:47:20.510442+0000","last_unstale":"2024-01-27T00:47:20.512795+0000","last_undegraded":"2024-01-27T00:47:20.512795+0000","last_fullsized":"2024-01-27T00:47:20.512795+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:54:22.080297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.519209+0000","last_change":"2024-01-27T00:47:20.517165+0000","last_active":"2024-01-27T00:47:20.519209+0000","last_peered":"2024-01-27T00:47:20.519209+0000","last_clean":"2024-01-27T00:47:20.519209+0000","last_became_active":"2024-01-27T00:47:20.516890+0000","last_became_peered":"2024-01-27T00:47:20.516890+0000","last_unstale":"2024-01-27T00:47:20.519209+0000","last_undegraded":"2024-01-27T00:47:20.519209+0000","last_fullsized":"2024-01-27T00:47:20.519209+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:14:43.680318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.517519+0000","last_change":"2024-01-27T00:47:20.510127+0000","last_active":"2024-01-27T00:47:20.517519+0000","last_peered":"2024-01-27T00:47:20.517519+0000","last_clean":"2024-01-27T00:47:20.517519+0000","last_became_active":"2024-01-27T00:47:20.509834+0000","last_became_peered":"2024-01-27T00:47:20.509834+0000","last_unstale":"2024-01-27T00:47:20.517519+0000","last_undegraded":"2024-01-27T00:47:20.517519+0000","last_fullsized":"2024-01-27T00:47:20.517519+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:34:17.410140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"187'4","reported_seq":14,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.532184+0000","last_change":"2024-01-27T00:47:19.505622+0000","last_active":"2024-01-27T00:47:20.532184+0000","last_peered":"2024-01-27T00:47:20.532184+0000","last_clean":"2024-01-27T00:47:20.532184+0000","last_became_active":"2024-01-27T00:47:19.505359+0000","last_became_peered":"2024-01-27T00:47:19.505359+0000","last_unstale":"2024-01-27T00:47:20.532184+0000","last_undegraded":"2024-01-27T00:47:20.532184+0000","last_fullsized":"2024-01-27T00:47:20.532184+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:30:46.693321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.510752+0000","last_change":"2024-01-27T00:47:20.509849+0000","last_active":"2024-01-27T00:47:20.510752+0000","last_peered":"2024-01-27T00:47:20.510752+0000","last_clean":"2024-01-27T00:47:20.510752+0000","last_became_active":"2024-01-27T00:47:20.509553+0000","last_became_peered":"2024-01-27T00:47:20.509553+0000","last_unstale":"2024-01-27T00:47:20.510752+0000","last_undegraded":"2024-01-27T00:47:20.510752+0000","last_fullsized":"2024-01-27T00:47:20.510752+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:44:26.489354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.528860+0000","last_change":"2024-01-27T00:47:19.498621+0000","last_active":"2024-01-27T00:47:20.528860+0000","last_peered":"2024-01-27T00:47:20.528860+0000","last_clean":"2024-01-27T00:47:20.528860+0000","last_became_active":"2024-01-27T00:47:19.498357+0000","last_became_peered":"2024-01-27T00:47:19.498357+0000","last_unstale":"2024-01-27T00:47:20.528860+0000","last_undegraded":"2024-01-27T00:47:20.528860+0000","last_fullsized":"2024-01-27T00:47:20.528860+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:20:56.150512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522707+0000","last_change":"2024-01-27T00:47:20.513840+0000","last_active":"2024-01-27T00:47:20.522707+0000","last_peered":"2024-01-27T00:47:20.522707+0000","last_clean":"2024-01-27T00:47:20.522707+0000","last_became_active":"2024-01-27T00:47:20.512745+0000","last_became_peered":"2024-01-27T00:47:20.512745+0000","last_unstale":"2024-01-27T00:47:20.522707+0000","last_undegraded":"2024-01-27T00:47:20.522707+0000","last_fullsized":"2024-01-27T00:47:20.522707+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:11:59.061116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.533311+0000","last_change":"2024-01-27T00:47:19.509540+0000","last_active":"2024-01-27T00:47:20.533311+0000","last_peered":"2024-01-27T00:47:20.533311+0000","last_clean":"2024-01-27T00:47:20.533311+0000","last_became_active":"2024-01-27T00:47:19.509301+0000","last_became_peered":"2024-01-27T00:47:19.509301+0000","last_unstale":"2024-01-27T00:47:20.533311+0000","last_undegraded":"2024-01-27T00:47:20.533311+0000","last_fullsized":"2024-01-27T00:47:20.533311+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:12:19.210091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522758+0000","last_change":"2024-01-27T00:47:20.521914+0000","last_active":"2024-01-27T00:47:20.522758+0000","last_peered":"2024-01-27T00:47:20.522758+0000","last_clean":"2024-01-27T00:47:20.522758+0000","last_became_active":"2024-01-27T00:47:20.521531+0000","last_became_peered":"2024-01-27T00:47:20.521531+0000","last_unstale":"2024-01-27T00:47:20.522758+0000","last_undegraded":"2024-01-27T00:47:20.522758+0000","last_fullsized":"2024-01-27T00:47:20.522758+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:46.559589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.533308+0000","last_change":"2024-01-27T00:47:19.505152+0000","last_active":"2024-01-27T00:47:20.533308+0000","last_peered":"2024-01-27T00:47:20.533308+0000","last_clean":"2024-01-27T00:47:20.533308+0000","last_became_active":"2024-01-27T00:47:19.504128+0000","last_became_peered":"2024-01-27T00:47:19.504128+0000","last_unstale":"2024-01-27T00:47:20.533308+0000","last_undegraded":"2024-01-27T00:47:20.533308+0000","last_fullsized":"2024-01-27T00:47:20.533308+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:52.404716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524071+0000","last_change":"2024-01-27T00:47:20.510375+0000","last_active":"2024-01-27T00:47:20.524071+0000","last_peered":"2024-01-27T00:47:20.524071+0000","last_clean":"2024-01-27T00:47:20.524071+0000","last_became_active":"2024-01-27T00:47:20.510069+0000","last_became_peered":"2024-01-27T00:47:20.510069+0000","last_unstale":"2024-01-27T00:47:20.524071+0000","last_undegraded":"2024-01-27T00:47:20.524071+0000","last_fullsized":"2024-01-27T00:47:20.524071+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:47:44.855002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.528716+0000","last_change":"2024-01-27T00:47:19.488294+0000","last_active":"2024-01-27T00:47:20.528716+0000","last_peered":"2024-01-27T00:47:20.528716+0000","last_clean":"2024-01-27T00:47:20.528716+0000","last_became_active":"2024-01-27T00:47:19.487984+0000","last_became_peered":"2024-01-27T00:47:19.487984+0000","last_unstale":"2024-01-27T00:47:20.528716+0000","last_undegraded":"2024-01-27T00:47:20.528716+0000","last_fullsized":"2024-01-27T00:47:20.528716+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:08:02.191122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.513175+0000","last_change":"2024-01-27T00:47:20.508823+0000","last_active":"2024-01-27T00:47:20.513175+0000","last_peered":"2024-01-27T00:47:20.513175+0000","last_clean":"2024-01-27T00:47:20.513175+0000","last_became_active":"2024-01-27T00:47:20.507955+0000","last_became_peered":"2024-01-27T00:47:20.507955+0000","last_unstale":"2024-01-27T00:47:20.513175+0000","last_undegraded":"2024-01-27T00:47:20.513175+0000","last_fullsized":"2024-01-27T00:47:20.513175+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:44.632841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531724+0000","last_change":"2024-01-27T00:47:19.493726+0000","last_active":"2024-01-27T00:47:20.531724+0000","last_peered":"2024-01-27T00:47:20.531724+0000","last_clean":"2024-01-27T00:47:20.531724+0000","last_became_active":"2024-01-27T00:47:19.493237+0000","last_became_peered":"2024-01-27T00:47:19.493237+0000","last_unstale":"2024-01-27T00:47:20.531724+0000","last_undegraded":"2024-01-27T00:47:20.531724+0000","last_fullsized":"2024-01-27T00:47:20.531724+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:33:12.701672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.532639+0000","last_change":"2024-01-27T00:47:20.516445+0000","last_active":"2024-01-27T00:47:20.532639+0000","last_peered":"2024-01-27T00:47:20.532639+0000","last_clean":"2024-01-27T00:47:20.532639+0000","last_became_active":"2024-01-27T00:47:20.515582+0000","last_became_peered":"2024-01-27T00:47:20.515582+0000","last_unstale":"2024-01-27T00:47:20.532639+0000","last_undegraded":"2024-01-27T00:47:20.532639+0000","last_fullsized":"2024-01-27T00:47:20.532639+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:07:16.136646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.507907+0000","last_change":"2024-01-27T00:47:19.505534+0000","last_active":"2024-01-27T00:47:20.507907+0000","last_peered":"2024-01-27T00:47:20.507907+0000","last_clean":"2024-01-27T00:47:20.507907+0000","last_became_active":"2024-01-27T00:47:19.504749+0000","last_became_peered":"2024-01-27T00:47:19.504749+0000","last_unstale":"2024-01-27T00:47:20.507907+0000","last_undegraded":"2024-01-27T00:47:20.507907+0000","last_fullsized":"2024-01-27T00:47:20.507907+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:48.314323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512091+0000","last_change":"2024-01-27T00:47:20.509887+0000","last_active":"2024-01-27T00:47:20.512091+0000","last_peered":"2024-01-27T00:47:20.512091+0000","last_clean":"2024-01-27T00:47:20.512091+0000","last_became_active":"2024-01-27T00:47:20.509560+0000","last_became_peered":"2024-01-27T00:47:20.509560+0000","last_unstale":"2024-01-27T00:47:20.512091+0000","last_undegraded":"2024-01-27T00:47:20.512091+0000","last_fullsized":"2024-01-27T00:47:20.512091+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:14:57.244737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.492777+0000","last_change":"2024-01-27T00:47:19.488903+0000","last_active":"2024-01-27T00:47:20.492777+0000","last_peered":"2024-01-27T00:47:20.492777+0000","last_clean":"2024-01-27T00:47:20.492777+0000","last_became_active":"2024-01-27T00:47:19.488649+0000","last_became_peered":"2024-01-27T00:47:19.488649+0000","last_unstale":"2024-01-27T00:47:20.492777+0000","last_undegraded":"2024-01-27T00:47:20.492777+0000","last_fullsized":"2024-01-27T00:47:20.492777+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:36:32.698812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.517497+0000","last_change":"2024-01-27T00:47:20.509808+0000","last_active":"2024-01-27T00:47:20.517497+0000","last_peered":"2024-01-27T00:47:20.517497+0000","last_clean":"2024-01-27T00:47:20.517497+0000","last_became_active":"2024-01-27T00:47:20.509527+0000","last_became_peered":"2024-01-27T00:47:20.509527+0000","last_unstale":"2024-01-27T00:47:20.517497+0000","last_undegraded":"2024-01-27T00:47:20.517497+0000","last_fullsized":"2024-01-27T00:47:20.517497+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:53:05.668777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499880+0000","last_change":"2024-01-27T00:47:19.505276+0000","last_active":"2024-01-27T00:47:20.499880+0000","last_peered":"2024-01-27T00:47:20.499880+0000","last_clean":"2024-01-27T00:47:20.499880+0000","last_became_active":"2024-01-27T00:47:19.504824+0000","last_became_peered":"2024-01-27T00:47:19.504824+0000","last_unstale":"2024-01-27T00:47:20.499880+0000","last_undegraded":"2024-01-27T00:47:20.499880+0000","last_fullsized":"2024-01-27T00:47:20.499880+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T00:53:04.664266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.510043+0000","last_change":"2024-01-27T00:47:20.507243+0000","last_active":"2024-01-27T00:47:20.510043+0000","last_peered":"2024-01-27T00:47:20.510043+0000","last_clean":"2024-01-27T00:47:20.510043+0000","last_became_active":"2024-01-27T00:47:20.506953+0000","last_became_peered":"2024-01-27T00:47:20.506953+0000","last_unstale":"2024-01-27T00:47:20.510043+0000","last_undegraded":"2024-01-27T00:47:20.510043+0000","last_fullsized":"2024-01-27T00:47:20.510043+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:23:47.466789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"187'3","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.528490+0000","last_change":"2024-01-27T00:47:19.493169+0000","last_active":"2024-01-27T00:47:20.528490+0000","last_peered":"2024-01-27T00:47:20.528490+0000","last_clean":"2024-01-27T00:47:20.528490+0000","last_became_active":"2024-01-27T00:47:19.492797+0000","last_became_peered":"2024-01-27T00:47:19.492797+0000","last_unstale":"2024-01-27T00:47:20.528490+0000","last_undegraded":"2024-01-27T00:47:20.528490+0000","last_fullsized":"2024-01-27T00:47:20.528490+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:48:59.935385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.530619+0000","last_change":"2024-01-27T00:47:20.516228+0000","last_active":"2024-01-27T00:47:20.530619+0000","last_peered":"2024-01-27T00:47:20.530619+0000","last_clean":"2024-01-27T00:47:20.530619+0000","last_became_active":"2024-01-27T00:47:20.515906+0000","last_became_peered":"2024-01-27T00:47:20.515906+0000","last_unstale":"2024-01-27T00:47:20.530619+0000","last_undegraded":"2024-01-27T00:47:20.530619+0000","last_fullsized":"2024-01-27T00:47:20.530619+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:12:20.214602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"187'6","reported_seq":16,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.534091+0000","last_change":"2024-01-27T00:47:19.502338+0000","last_active":"2024-01-27T00:47:20.534091+0000","last_peered":"2024-01-27T00:47:20.534091+0000","last_clean":"2024-01-27T00:47:20.534091+0000","last_became_active":"2024-01-27T00:47:19.502072+0000","last_became_peered":"2024-01-27T00:47:19.502072+0000","last_unstale":"2024-01-27T00:47:20.534091+0000","last_undegraded":"2024-01-27T00:47:20.534091+0000","last_fullsized":"2024-01-27T00:47:20.534091+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:34:16.405629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.519133+0000","last_change":"2024-01-27T00:47:20.516500+0000","last_active":"2024-01-27T00:47:20.519133+0000","last_peered":"2024-01-27T00:47:20.519133+0000","last_clean":"2024-01-27T00:47:20.519133+0000","last_became_active":"2024-01-27T00:47:20.516247+0000","last_became_peered":"2024-01-27T00:47:20.516247+0000","last_unstale":"2024-01-27T00:47:20.519133+0000","last_undegraded":"2024-01-27T00:47:20.519133+0000","last_fullsized":"2024-01-27T00:47:20.519133+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:44:28.491993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527814+0000","last_change":"2024-01-27T00:47:19.491092+0000","last_active":"2024-01-27T00:47:20.527814+0000","last_peered":"2024-01-27T00:47:20.527814+0000","last_clean":"2024-01-27T00:47:20.527814+0000","last_became_active":"2024-01-27T00:47:19.490799+0000","last_became_peered":"2024-01-27T00:47:19.490799+0000","last_unstale":"2024-01-27T00:47:20.527814+0000","last_undegraded":"2024-01-27T00:47:20.527814+0000","last_fullsized":"2024-01-27T00:47:20.527814+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:23:46.462278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527448+0000","last_change":"2024-01-27T00:47:20.515730+0000","last_active":"2024-01-27T00:47:20.527448+0000","last_peered":"2024-01-27T00:47:20.527448+0000","last_clean":"2024-01-27T00:47:20.527448+0000","last_became_active":"2024-01-27T00:47:20.515422+0000","last_became_peered":"2024-01-27T00:47:20.515422+0000","last_unstale":"2024-01-27T00:47:20.527448+0000","last_undegraded":"2024-01-27T00:47:20.527448+0000","last_fullsized":"2024-01-27T00:47:20.527448+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:33.654223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.498821+0000","last_change":"2024-01-27T00:47:19.499915+0000","last_active":"2024-01-27T00:47:20.498821+0000","last_peered":"2024-01-27T00:47:20.498821+0000","last_clean":"2024-01-27T00:47:20.498821+0000","last_became_active":"2024-01-27T00:47:19.499418+0000","last_became_peered":"2024-01-27T00:47:19.499418+0000","last_unstale":"2024-01-27T00:47:20.498821+0000","last_undegraded":"2024-01-27T00:47:20.498821+0000","last_fullsized":"2024-01-27T00:47:20.498821+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:46.642397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512934+0000","last_change":"2024-01-27T00:47:20.509483+0000","last_active":"2024-01-27T00:47:20.512934+0000","last_peered":"2024-01-27T00:47:20.512934+0000","last_clean":"2024-01-27T00:47:20.512934+0000","last_became_active":"2024-01-27T00:47:20.508998+0000","last_became_peered":"2024-01-27T00:47:20.508998+0000","last_unstale":"2024-01-27T00:47:20.512934+0000","last_undegraded":"2024-01-27T00:47:20.512934+0000","last_fullsized":"2024-01-27T00:47:20.512934+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:57:20.403842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.496693+0000","last_change":"2024-01-27T00:47:19.489957+0000","last_active":"2024-01-27T00:47:20.496693+0000","last_peered":"2024-01-27T00:47:20.496693+0000","last_clean":"2024-01-27T00:47:20.496693+0000","last_became_active":"2024-01-27T00:47:19.489686+0000","last_became_peered":"2024-01-27T00:47:19.489686+0000","last_unstale":"2024-01-27T00:47:20.496693+0000","last_undegraded":"2024-01-27T00:47:20.496693+0000","last_fullsized":"2024-01-27T00:47:20.496693+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:36:40.108567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.514859+0000","last_change":"2024-01-27T00:47:20.513843+0000","last_active":"2024-01-27T00:47:20.514859+0000","last_peered":"2024-01-27T00:47:20.514859+0000","last_clean":"2024-01-27T00:47:20.514859+0000","last_became_active":"2024-01-27T00:47:20.513188+0000","last_became_peered":"2024-01-27T00:47:20.513188+0000","last_unstale":"2024-01-27T00:47:20.514859+0000","last_undegraded":"2024-01-27T00:47:20.514859+0000","last_fullsized":"2024-01-27T00:47:20.514859+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:15:01.330526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"187'8","reported_seq":18,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.529029+0000","last_change":"2024-01-27T00:47:19.491898+0000","last_active":"2024-01-27T00:47:20.529029+0000","last_peered":"2024-01-27T00:47:20.529029+0000","last_clean":"2024-01-27T00:47:20.529029+0000","last_became_active":"2024-01-27T00:47:19.491597+0000","last_became_peered":"2024-01-27T00:47:19.491597+0000","last_unstale":"2024-01-27T00:47:20.529029+0000","last_undegraded":"2024-01-27T00:47:20.529029+0000","last_fullsized":"2024-01-27T00:47:20.529029+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:47:33.870316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524260+0000","last_change":"2024-01-27T00:47:20.511230+0000","last_active":"2024-01-27T00:47:20.524260+0000","last_peered":"2024-01-27T00:47:20.524260+0000","last_clean":"2024-01-27T00:47:20.524260+0000","last_became_active":"2024-01-27T00:47:20.510893+0000","last_became_peered":"2024-01-27T00:47:20.510893+0000","last_unstale":"2024-01-27T00:47:20.524260+0000","last_undegraded":"2024-01-27T00:47:20.524260+0000","last_fullsized":"2024-01-27T00:47:20.524260+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:51:40.896423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531660+0000","last_change":"2024-01-27T00:47:19.504875+0000","last_active":"2024-01-27T00:47:20.531660+0000","last_peered":"2024-01-27T00:47:20.531660+0000","last_clean":"2024-01-27T00:47:20.531660+0000","last_became_active":"2024-01-27T00:47:19.504575+0000","last_became_peered":"2024-01-27T00:47:19.504575+0000","last_unstale":"2024-01-27T00:47:20.531660+0000","last_undegraded":"2024-01-27T00:47:20.531660+0000","last_fullsized":"2024-01-27T00:47:20.531660+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:43:03.734740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.515064+0000","last_change":"2024-01-27T00:47:20.510860+0000","last_active":"2024-01-27T00:47:20.515064+0000","last_peered":"2024-01-27T00:47:20.515064+0000","last_clean":"2024-01-27T00:47:20.515064+0000","last_became_active":"2024-01-27T00:47:20.510567+0000","last_became_peered":"2024-01-27T00:47:20.510567+0000","last_unstale":"2024-01-27T00:47:20.515064+0000","last_undegraded":"2024-01-27T00:47:20.515064+0000","last_fullsized":"2024-01-27T00:47:20.515064+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:33:31.812775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.533275+0000","last_change":"2024-01-27T00:47:19.509498+0000","last_active":"2024-01-27T00:47:20.533275+0000","last_peered":"2024-01-27T00:47:20.533275+0000","last_clean":"2024-01-27T00:47:20.533275+0000","last_became_active":"2024-01-27T00:47:19.509151+0000","last_became_peered":"2024-01-27T00:47:19.509151+0000","last_unstale":"2024-01-27T00:47:20.533275+0000","last_undegraded":"2024-01-27T00:47:20.533275+0000","last_fullsized":"2024-01-27T00:47:20.533275+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:54:21.075786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512563+0000","last_change":"2024-01-27T00:47:20.510494+0000","last_active":"2024-01-27T00:47:20.512563+0000","last_peered":"2024-01-27T00:47:20.512563+0000","last_clean":"2024-01-27T00:47:20.512563+0000","last_became_active":"2024-01-27T00:47:20.510159+0000","last_became_peered":"2024-01-27T00:47:20.510159+0000","last_unstale":"2024-01-27T00:47:20.512563+0000","last_undegraded":"2024-01-27T00:47:20.512563+0000","last_fullsized":"2024-01-27T00:47:20.512563+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:08:01.982720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"187'1","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.529256+0000","last_change":"2024-01-27T00:47:19.498982+0000","last_active":"2024-01-27T00:47:20.529256+0000","last_peered":"2024-01-27T00:47:20.529256+0000","last_clean":"2024-01-27T00:47:20.529256+0000","last_became_active":"2024-01-27T00:47:19.498718+0000","last_became_peered":"2024-01-27T00:47:19.498718+0000","last_unstale":"2024-01-27T00:47:20.529256+0000","last_undegraded":"2024-01-27T00:47:20.529256+0000","last_fullsized":"2024-01-27T00:47:20.529256+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:04:42.913017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.517959+0000","last_change":"2024-01-27T00:47:20.514875+0000","last_active":"2024-01-27T00:47:20.517959+0000","last_peered":"2024-01-27T00:47:20.517959+0000","last_clean":"2024-01-27T00:47:20.517959+0000","last_became_active":"2024-01-27T00:47:20.514574+0000","last_became_peered":"2024-01-27T00:47:20.514574+0000","last_unstale":"2024-01-27T00:47:20.517959+0000","last_undegraded":"2024-01-27T00:47:20.517959+0000","last_fullsized":"2024-01-27T00:47:20.517959+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:47:34.874827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.493212+0000","last_change":"2024-01-27T00:47:19.493973+0000","last_active":"2024-01-27T00:47:20.493212+0000","last_peered":"2024-01-27T00:47:20.493212+0000","last_clean":"2024-01-27T00:47:20.493212+0000","last_became_active":"2024-01-27T00:47:19.493685+0000","last_became_peered":"2024-01-27T00:47:19.493685+0000","last_unstale":"2024-01-27T00:47:20.493212+0000","last_undegraded":"2024-01-27T00:47:20.493212+0000","last_fullsized":"2024-01-27T00:47:20.493212+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:08:00.978209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531595+0000","last_change":"2024-01-27T00:47:20.525394+0000","last_active":"2024-01-27T00:47:20.531595+0000","last_peered":"2024-01-27T00:47:20.531595+0000","last_clean":"2024-01-27T00:47:20.531595+0000","last_became_active":"2024-01-27T00:47:20.521446+0000","last_became_peered":"2024-01-27T00:47:20.521446+0000","last_unstale":"2024-01-27T00:47:20.531595+0000","last_undegraded":"2024-01-27T00:47:20.531595+0000","last_fullsized":"2024-01-27T00:47:20.531595+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:31.735208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.529633+0000","last_change":"2024-01-27T00:47:19.495677+0000","last_active":"2024-01-27T00:47:20.529633+0000","last_peered":"2024-01-27T00:47:20.529633+0000","last_clean":"2024-01-27T00:47:20.529633+0000","last_became_active":"2024-01-27T00:47:19.494329+0000","last_became_peered":"2024-01-27T00:47:19.494329+0000","last_unstale":"2024-01-27T00:47:20.529633+0000","last_undegraded":"2024-01-27T00:47:20.529633+0000","last_fullsized":"2024-01-27T00:47:20.529633+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:19:25.040151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522962+0000","last_change":"2024-01-27T00:47:20.512100+0000","last_active":"2024-01-27T00:47:20.522962+0000","last_peered":"2024-01-27T00:47:20.522962+0000","last_clean":"2024-01-27T00:47:20.522962+0000","last_became_active":"2024-01-27T00:47:20.511807+0000","last_became_peered":"2024-01-27T00:47:20.511807+0000","last_unstale":"2024-01-27T00:47:20.522962+0000","last_undegraded":"2024-01-27T00:47:20.522962+0000","last_fullsized":"2024-01-27T00:47:20.522962+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:41:19.485849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.498403+0000","last_change":"2024-01-27T00:47:19.495458+0000","last_active":"2024-01-27T00:47:20.498403+0000","last_peered":"2024-01-27T00:47:20.498403+0000","last_clean":"2024-01-27T00:47:20.498403+0000","last_became_active":"2024-01-27T00:47:19.495193+0000","last_became_peered":"2024-01-27T00:47:19.495193+0000","last_unstale":"2024-01-27T00:47:20.498403+0000","last_undegraded":"2024-01-27T00:47:20.498403+0000","last_fullsized":"2024-01-27T00:47:20.498403+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:12:18.301327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522508+0000","last_change":"2024-01-27T00:47:20.512072+0000","last_active":"2024-01-27T00:47:20.522508+0000","last_peered":"2024-01-27T00:47:20.522508+0000","last_clean":"2024-01-27T00:47:20.522508+0000","last_became_active":"2024-01-27T00:47:20.511716+0000","last_became_peered":"2024-01-27T00:47:20.511716+0000","last_unstale":"2024-01-27T00:47:20.522508+0000","last_undegraded":"2024-01-27T00:47:20.522508+0000","last_fullsized":"2024-01-27T00:47:20.522508+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:18:15.805971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.508031+0000","last_change":"2024-01-27T00:47:19.505685+0000","last_active":"2024-01-27T00:47:20.508031+0000","last_peered":"2024-01-27T00:47:20.508031+0000","last_clean":"2024-01-27T00:47:20.508031+0000","last_became_active":"2024-01-27T00:47:19.505201+0000","last_became_peered":"2024-01-27T00:47:19.505201+0000","last_unstale":"2024-01-27T00:47:20.508031+0000","last_undegraded":"2024-01-27T00:47:20.508031+0000","last_fullsized":"2024-01-27T00:47:20.508031+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:10:58.917172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.521282+0000","last_change":"2024-01-27T00:47:20.518813+0000","last_active":"2024-01-27T00:47:20.521282+0000","last_peered":"2024-01-27T00:47:20.521282+0000","last_clean":"2024-01-27T00:47:20.521282+0000","last_became_active":"2024-01-27T00:47:20.518485+0000","last_became_peered":"2024-01-27T00:47:20.518485+0000","last_unstale":"2024-01-27T00:47:20.521282+0000","last_undegraded":"2024-01-27T00:47:20.521282+0000","last_fullsized":"2024-01-27T00:47:20.521282+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:54:59.918544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.502196+0000","last_change":"2024-01-27T00:47:19.490229+0000","last_active":"2024-01-27T00:47:20.502196+0000","last_peered":"2024-01-27T00:47:20.502196+0000","last_clean":"2024-01-27T00:47:20.502196+0000","last_became_active":"2024-01-27T00:47:19.489969+0000","last_became_peered":"2024-01-27T00:47:19.489969+0000","last_unstale":"2024-01-27T00:47:20.502196+0000","last_undegraded":"2024-01-27T00:47:20.502196+0000","last_fullsized":"2024-01-27T00:47:20.502196+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:14:56.240226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.512461+0000","last_change":"2024-01-27T00:47:20.504909+0000","last_active":"2024-01-27T00:47:20.512461+0000","last_peered":"2024-01-27T00:47:20.512461+0000","last_clean":"2024-01-27T00:47:20.512461+0000","last_became_active":"2024-01-27T00:47:20.504607+0000","last_became_peered":"2024-01-27T00:47:20.504607+0000","last_unstale":"2024-01-27T00:47:20.512461+0000","last_undegraded":"2024-01-27T00:47:20.512461+0000","last_fullsized":"2024-01-27T00:47:20.512461+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:36:33.703323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"187'2","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.527493+0000","last_change":"2024-01-27T00:47:19.498236+0000","last_active":"2024-01-27T00:47:20.527493+0000","last_peered":"2024-01-27T00:47:20.527493+0000","last_clean":"2024-01-27T00:47:20.527493+0000","last_became_active":"2024-01-27T00:47:19.497955+0000","last_became_peered":"2024-01-27T00:47:19.497955+0000","last_unstale":"2024-01-27T00:47:20.527493+0000","last_undegraded":"2024-01-27T00:47:20.527493+0000","last_fullsized":"2024-01-27T00:47:20.527493+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:52:03.781390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524123+0000","last_change":"2024-01-27T00:47:20.510553+0000","last_active":"2024-01-27T00:47:20.524123+0000","last_peered":"2024-01-27T00:47:20.524123+0000","last_clean":"2024-01-27T00:47:20.524123+0000","last_became_active":"2024-01-27T00:47:20.510231+0000","last_became_peered":"2024-01-27T00:47:20.510231+0000","last_unstale":"2024-01-27T00:47:20.524123+0000","last_undegraded":"2024-01-27T00:47:20.524123+0000","last_fullsized":"2024-01-27T00:47:20.524123+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:15.349063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"188'1","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-01-27T00:47:21.495943+0000","last_change":"2024-01-27T00:47:19.505767+0000","last_active":"2024-01-27T00:47:21.495943+0000","last_peered":"2024-01-27T00:47:21.495943+0000","last_clean":"2024-01-27T00:47:21.495943+0000","last_became_active":"2024-01-27T00:47:19.505313+0000","last_became_peered":"2024-01-27T00:47:19.505313+0000","last_unstale":"2024-01-27T00:47:21.495943+0000","last_undegraded":"2024-01-27T00:47:21.495943+0000","last_fullsized":"2024-01-27T00:47:21.495943+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:48:00.955590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.530928+0000","last_change":"2024-01-27T00:47:20.516018+0000","last_active":"2024-01-27T00:47:20.530928+0000","last_peered":"2024-01-27T00:47:20.530928+0000","last_clean":"2024-01-27T00:47:20.530928+0000","last_became_active":"2024-01-27T00:47:20.515703+0000","last_became_peered":"2024-01-27T00:47:20.515703+0000","last_unstale":"2024-01-27T00:47:20.530928+0000","last_undegraded":"2024-01-27T00:47:20.530928+0000","last_fullsized":"2024-01-27T00:47:20.530928+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:12:19.305838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499639+0000","last_change":"2024-01-27T00:47:19.499255+0000","last_active":"2024-01-27T00:47:20.499639+0000","last_peered":"2024-01-27T00:47:20.499639+0000","last_clean":"2024-01-27T00:47:20.499639+0000","last_became_active":"2024-01-27T00:47:19.498933+0000","last_became_peered":"2024-01-27T00:47:19.498933+0000","last_unstale":"2024-01-27T00:47:20.499639+0000","last_undegraded":"2024-01-27T00:47:20.499639+0000","last_fullsized":"2024-01-27T00:47:20.499639+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:47:45.555078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.524255+0000","last_change":"2024-01-27T00:47:20.511077+0000","last_active":"2024-01-27T00:47:20.524255+0000","last_peered":"2024-01-27T00:47:20.524255+0000","last_clean":"2024-01-27T00:47:20.524255+0000","last_became_active":"2024-01-27T00:47:20.510732+0000","last_became_peered":"2024-01-27T00:47:20.510732+0000","last_unstale":"2024-01-27T00:47:20.524255+0000","last_undegraded":"2024-01-27T00:47:20.524255+0000","last_fullsized":"2024-01-27T00:47:20.524255+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:05.094009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.508098+0000","last_change":"2024-01-27T00:47:19.504494+0000","last_active":"2024-01-27T00:47:20.508098+0000","last_peered":"2024-01-27T00:47:20.508098+0000","last_clean":"2024-01-27T00:47:20.508098+0000","last_became_active":"2024-01-27T00:47:19.504090+0000","last_became_peered":"2024-01-27T00:47:19.504090+0000","last_unstale":"2024-01-27T00:47:20.508098+0000","last_undegraded":"2024-01-27T00:47:20.508098+0000","last_fullsized":"2024-01-27T00:47:20.508098+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:22:59.518350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.518283+0000","last_change":"2024-01-27T00:47:20.514544+0000","last_active":"2024-01-27T00:47:20.518283+0000","last_peered":"2024-01-27T00:47:20.518283+0000","last_clean":"2024-01-27T00:47:20.518283+0000","last_became_active":"2024-01-27T00:47:20.514270+0000","last_became_peered":"2024-01-27T00:47:20.514270+0000","last_unstale":"2024-01-27T00:47:20.518283+0000","last_undegraded":"2024-01-27T00:47:20.518283+0000","last_fullsized":"2024-01-27T00:47:20.518283+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:32:14.595404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.507239+0000","last_change":"2024-01-27T00:47:19.501914+0000","last_active":"2024-01-27T00:47:20.507239+0000","last_peered":"2024-01-27T00:47:20.507239+0000","last_clean":"2024-01-27T00:47:20.507239+0000","last_became_active":"2024-01-27T00:47:19.501663+0000","last_became_peered":"2024-01-27T00:47:19.501663+0000","last_unstale":"2024-01-27T00:47:20.507239+0000","last_undegraded":"2024-01-27T00:47:20.507239+0000","last_fullsized":"2024-01-27T00:47:20.507239+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:18:49.554812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.522022+0000","last_change":"2024-01-27T00:47:20.519790+0000","last_active":"2024-01-27T00:47:20.522022+0000","last_peered":"2024-01-27T00:47:20.522022+0000","last_clean":"2024-01-27T00:47:20.522022+0000","last_became_active":"2024-01-27T00:47:20.519499+0000","last_became_peered":"2024-01-27T00:47:20.519499+0000","last_unstale":"2024-01-27T00:47:20.522022+0000","last_undegraded":"2024-01-27T00:47:20.522022+0000","last_fullsized":"2024-01-27T00:47:20.522022+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:48:01.960101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.498924+0000","last_change":"2024-01-27T00:47:19.503541+0000","last_active":"2024-01-27T00:47:20.498924+0000","last_peered":"2024-01-27T00:47:20.498924+0000","last_clean":"2024-01-27T00:47:20.498924+0000","last_became_active":"2024-01-27T00:47:19.503073+0000","last_became_peered":"2024-01-27T00:47:19.503073+0000","last_unstale":"2024-01-27T00:47:20.498924+0000","last_undegraded":"2024-01-27T00:47:20.498924+0000","last_fullsized":"2024-01-27T00:47:20.498924+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:37:14.344552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.531950+0000","last_change":"2024-01-27T00:47:20.527327+0000","last_active":"2024-01-27T00:47:20.531950+0000","last_peered":"2024-01-27T00:47:20.531950+0000","last_clean":"2024-01-27T00:47:20.531950+0000","last_became_active":"2024-01-27T00:47:20.520248+0000","last_became_peered":"2024-01-27T00:47:20.520248+0000","last_unstale":"2024-01-27T00:47:20.531950+0000","last_undegraded":"2024-01-27T00:47:20.531950+0000","last_fullsized":"2024-01-27T00:47:20.531950+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:19:26.044662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.496693+0000","last_change":"2024-01-27T00:47:19.503480+0000","last_active":"2024-01-27T00:47:20.496693+0000","last_peered":"2024-01-27T00:47:20.496693+0000","last_clean":"2024-01-27T00:47:20.496693+0000","last_became_active":"2024-01-27T00:47:19.503158+0000","last_became_peered":"2024-01-27T00:47:19.503158+0000","last_unstale":"2024-01-27T00:47:20.496693+0000","last_undegraded":"2024-01-27T00:47:20.496693+0000","last_fullsized":"2024-01-27T00:47:20.496693+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:15:00.326015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.532846+0000","last_change":"2024-01-27T00:47:20.516514+0000","last_active":"2024-01-27T00:47:20.532846+0000","last_peered":"2024-01-27T00:47:20.532846+0000","last_clean":"2024-01-27T00:47:20.532846+0000","last_became_active":"2024-01-27T00:47:20.516160+0000","last_became_peered":"2024-01-27T00:47:20.516160+0000","last_unstale":"2024-01-27T00:47:20.532846+0000","last_undegraded":"2024-01-27T00:47:20.532846+0000","last_fullsized":"2024-01-27T00:47:20.532846+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:36:41.113078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.497052+0000","last_change":"2024-01-27T00:47:19.493487+0000","last_active":"2024-01-27T00:47:20.497052+0000","last_peered":"2024-01-27T00:47:20.497052+0000","last_clean":"2024-01-27T00:47:20.497052+0000","last_became_active":"2024-01-27T00:47:19.492721+0000","last_became_peered":"2024-01-27T00:47:19.492721+0000","last_unstale":"2024-01-27T00:47:20.497052+0000","last_undegraded":"2024-01-27T00:47:20.497052+0000","last_fullsized":"2024-01-27T00:47:20.497052+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:11:30.730697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.519011+0000","last_change":"2024-01-27T00:47:20.516157+0000","last_active":"2024-01-27T00:47:20.519011+0000","last_peered":"2024-01-27T00:47:20.519011+0000","last_clean":"2024-01-27T00:47:20.519011+0000","last_became_active":"2024-01-27T00:47:20.515871+0000","last_became_peered":"2024-01-27T00:47:20.515871+0000","last_unstale":"2024-01-27T00:47:20.519011+0000","last_undegraded":"2024-01-27T00:47:20.519011+0000","last_fullsized":"2024-01-27T00:47:20.519011+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:19.474295+0000","last_clean_scrub_stamp":"2024-01-27T00:47:19.474295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:05:38.788368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":10,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.499725+0000","last_change":"2024-01-27T00:47:19.501570+0000","last_active":"2024-01-27T00:47:20.499725+0000","last_peered":"2024-01-27T00:47:20.499725+0000","last_clean":"2024-01-27T00:47:20.499725+0000","last_became_active":"2024-01-27T00:47:19.501187+0000","last_became_peered":"2024-01-27T00:47:19.501187+0000","last_unstale":"2024-01-27T00:47:20.499725+0000","last_undegraded":"2024-01-27T00:47:20.499725+0000","last_fullsized":"2024-01-27T00:47:20.499725+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T00:47:18.469785+0000","last_clean_scrub_stamp":"2024-01-27T00:47:18.469785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:54:58.914033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"179'8695","reported_seq":9436,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-01-27T00:47:20.501981+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T00:47:20.501981+0000","last_peered":"2024-01-27T00:47:20.501981+0000","last_clean":"2024-01-27T00:47:20.501981+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T00:47:20.501981+0000","last_undegraded":"2024-01-27T00:47:20.501981+0000","last_fullsized":"2024-01-27T00:47:20.501981+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8695,"log_dups_size":0,"ondisk_log_size":8695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1784,"num_read_kb":3778,"num_write":530,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1784,"num_read_kb":3778,"num_write":530,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8695,"ondisk_log_size":8695,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575777,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300168,"kb_used_data":2880,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071672,"statfs":{"total":96636764160,"available":96329392128,"internally_reserved":0,"allocated":2949120,"data_stored":1815604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60799999999999998,"15min":0.60699999999999998},"min":{"1min":0.36199999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75800000000000001,"5min":0.83199999999999996,"15min":0.86899999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61299999999999999,"15min":0.61799999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.755,"5min":0.82099999999999995,"15min":0.90000000000000002},"last":0.66500000000000004}]},{"osd":1,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.34599999999999997},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.90400000000000003},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63500000000000001,"15min":0.63},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.38200000000000001},"max":{"1min":0.76700000000000002,"5min":0.85099999999999998,"15min":0.875},"last":0.56999999999999995}]},{"osd":2,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.374,"5min":0.36299999999999999,"15min":0.34000000000000002},"max":{"1min":0.76400000000000001,"5min":0.77700000000000002,"15min":0.89700000000000002},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63800000000000001,"15min":0.63700000000000001},"min":{"1min":0.49399999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75600000000000001,"5min":0.86199999999999999,"15min":0.94199999999999995},"last":0.57999999999999996}]},{"osd":3,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.626,"15min":0.623},"min":{"1min":0.435,"5min":0.40999999999999998,"15min":0.39400000000000002},"max":{"1min":0.79200000000000004,"5min":0.86499999999999999,"15min":0.92600000000000005},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.77700000000000002,"5min":0.83499999999999996,"15min":0.88400000000000001},"last":0.60799999999999998}]},{"osd":4,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49399999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.32900000000000001,"15min":0.29699999999999999},"max":{"1min":0.61699999999999999,"5min":0.86499999999999999,"15min":0.92000000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.34599999999999997},"max":{"1min":0.76600000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.39600000000000002}]},{"osd":5,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":0.67100000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":0.67700000000000005,"5min":0.89100000000000001,"15min":0.95699999999999996},"last":0.59599999999999997}]},{"osd":6,"last update":"Sat Jan 27 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59299999999999997,"15min":0.58299999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.36599999999999999},"max":{"1min":0.69299999999999995,"5min":0.91700000000000004,"15min":0.99399999999999999},"last":0.623},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.36699999999999999},"max":{"1min":0.73599999999999999,"5min":0.82299999999999995,"15min":0.878},"last":0.55600000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575777,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307780,"kb_used_data":2300,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064060,"statfs":{"total":96636764160,"available":96321597440,"internally_reserved":0,"allocated":2355200,"data_stored":1225236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58999999999999997,"15min":0.60399999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81000000000000005,"5min":0.82199999999999995,"15min":0.84899999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.38900000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.78500000000000003,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.497}]},{"osd":1,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.748,"5min":0.79800000000000004,"15min":0.80800000000000005},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81499999999999995,"5min":0.87,"15min":0.87},"last":0.42899999999999999}]},{"osd":2,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60199999999999998,"15min":0.60899999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83699999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.44},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59899999999999998,"15min":0.60599999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74199999999999999,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.50700000000000001}]},{"osd":3,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86299999999999999,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64600000000000002,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.52100000000000002}]},{"osd":4,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.496,"15min":0.49099999999999999},"min":{"1min":0.34300000000000003,"5min":0.33200000000000002,"15min":0.313},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32100000000000001},"max":{"1min":0.77700000000000002,"5min":0.79100000000000004,"15min":0.83199999999999996},"last":0.36899999999999999}]},{"osd":5,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51900000000000002,"15min":0.51000000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.32200000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56499999999999995,"15min":0.56000000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.88800000000000001},"last":0.32900000000000001}]},{"osd":7,"last update":"Sat Jan 27 00:46:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57399999999999995,"15min":0.56499999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.58399999999999996,"15min":0.58099999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.33600000000000002},"max":{"1min":0.84099999999999997,"5min":0.84599999999999997,"15min":0.85999999999999999},"last":0.46100000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575777,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311888,"kb_used_data":2312,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059952,"statfs":{"total":96636764160,"available":96317390848,"internally_reserved":0,"allocated":2367488,"data_stored":1226772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.41799999999999998},"min":{"1min":0.311,"5min":0.254,"15min":0.23499999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.434,"5min":0.441,"15min":0.44500000000000001},"min":{"1min":0.30199999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.60599999999999998,"5min":0.66200000000000003,"15min":0.70199999999999996},"last":0.47199999999999998}]},{"osd":2,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.41899999999999998,"15min":0.42499999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55100000000000005,"5min":0.59999999999999998,"15min":0.68899999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48499999999999999,"15min":0.49399999999999999},"min":{"1min":0.33500000000000002,"5min":0.31,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.76700000000000002},"last":0.61199999999999999}]},{"osd":3,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.503,"15min":0.52100000000000002},"min":{"1min":0.39400000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.63900000000000001,"5min":0.69399999999999995,"15min":1.0469999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.49099999999999999,"15min":0.5},"min":{"1min":0.318,"5min":0.318,"15min":0.309},"max":{"1min":0.69599999999999995,"5min":0.72599999999999998,"15min":0.95599999999999996},"last":0.55800000000000005}]},{"osd":4,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.66100000000000003,"15min":0.67900000000000005},"min":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86299999999999999,"5min":1.02,"15min":1.3129999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67700000000000005,"15min":0.69999999999999996},"min":{"1min":0.505,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.93500000000000005,"5min":1.0069999999999999,"15min":1.097},"last":0.752}]},{"osd":5,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66700000000000004,"15min":0.69199999999999995},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.46300000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":1.091},"last":0.874},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.36299999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.278},"last":0.76700000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.68500000000000005,"15min":0.71099999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.435},"max":{"1min":0.88100000000000001,"5min":0.96999999999999997,"15min":1.141},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70799999999999996,"15min":0.72099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.91400000000000003,"5min":1.0229999999999999,"15min":1.1120000000000001},"last":0.71799999999999997}]},{"osd":7,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.73299999999999998,"15min":0.749},"min":{"1min":0.52600000000000002,"5min":0.52500000000000002,"15min":0.51100000000000001},"max":{"1min":1.0129999999999999,"5min":1.054,"15min":1.1819999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.73499999999999999,"15min":0.75800000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.51200000000000001},"max":{"1min":0.98299999999999998,"5min":0.98999999999999999,"15min":1.198},"last":0.83699999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575776,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312464,"kb_used_data":2888,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059376,"statfs":{"total":96636764160,"available":96316801024,"internally_reserved":0,"allocated":2957312,"data_stored":1815708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.28299999999999997,"5min":0.25800000000000001,"15min":0.25},"max":{"1min":0.55500000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.34499999999999997,"5min":0.27500000000000002,"15min":0.26300000000000001},"max":{"1min":0.61299999999999999,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.38100000000000001}]},{"osd":2,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.42099999999999999,"15min":0.42599999999999999},"min":{"1min":0.312,"5min":0.28599999999999998,"15min":0.26700000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.69399999999999995},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48099999999999998,"15min":0.48299999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.438}]},{"osd":3,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48999999999999999,"15min":0.48699999999999999},"min":{"1min":0.32900000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.65600000000000003,"5min":0.755,"15min":0.755},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.496,"15min":0.498},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.40000000000000002}]},{"osd":4,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65200000000000002,"15min":0.68100000000000005},"min":{"1min":0.45100000000000001,"5min":0.44700000000000001,"15min":0.44400000000000001},"max":{"1min":0.77500000000000002,"5min":0.97099999999999997,"15min":1.125},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.66000000000000003,"15min":0.67800000000000005},"min":{"1min":0.46200000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.96199999999999997,"5min":0.97599999999999998,"15min":1.0169999999999999},"last":0.56599999999999995}]},{"osd":5,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68100000000000005,"15min":0.69399999999999995},"min":{"1min":0.47899999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.84099999999999997,"5min":0.94199999999999995,"15min":1.054},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.70799999999999996,"15min":0.71899999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.98199999999999998,"5min":1.056,"15min":1.056},"last":0.68000000000000005}]},{"osd":6,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69999999999999996,"15min":0.71199999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.84399999999999997,"5min":1.0269999999999999,"15min":1.087},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.72299999999999998,"15min":0.73899999999999999},"min":{"1min":0.52500000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.94599999999999995,"5min":1.0349999999999999,"15min":1.052},"last":0.78000000000000003}]},{"osd":7,"last update":"Sat Jan 27 00:46:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.747,"15min":0.75700000000000001},"min":{"1min":0.60599999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.016,"5min":1.0880000000000001,"15min":1.1080000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.53700000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.034,"5min":1.071,"15min":1.071},"last":0.72299999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575776,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311876,"kb_used_data":2300,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059964,"statfs":{"total":96636764160,"available":96317403136,"internally_reserved":0,"allocated":2355200,"data_stored":1225236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38100000000000001,"15min":0.39800000000000002},"min":{"1min":0.29499999999999998,"5min":0.25900000000000001,"15min":0.252},"max":{"1min":0.48699999999999999,"5min":0.60499999999999998,"15min":0.65800000000000003},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41999999999999998,"15min":0.438},"min":{"1min":0.29499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.71899999999999997},"last":0.53700000000000003}]},{"osd":1,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.41999999999999998,"15min":0.432},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.60799999999999998,"5min":0.627,"15min":0.68300000000000005},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.5,"15min":0.51400000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66100000000000003,"5min":0.76400000000000001,"15min":0.80300000000000005},"last":0.59099999999999997}]},{"osd":3,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47799999999999998,"15min":0.49199999999999999},"min":{"1min":0.34999999999999998,"5min":0.34100000000000003,"15min":0.33100000000000002},"max":{"1min":0.65200000000000002,"5min":0.745,"15min":0.745},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47099999999999997,"15min":0.48699999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.29499999999999998},"max":{"1min":0.68000000000000005,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.56200000000000006}]},{"osd":4,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.65000000000000002,"15min":0.66700000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41999999999999998},"max":{"1min":0.77000000000000002,"5min":0.85899999999999999,"15min":1.0660000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66700000000000004,"15min":0.68200000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92400000000000004,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66800000000000004,"15min":0.68999999999999995},"min":{"1min":0.49099999999999999,"5min":0.46500000000000002,"15min":0.42899999999999999},"max":{"1min":0.78400000000000003,"5min":0.96499999999999997,"15min":1.0649999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68200000000000005,"15min":0.69799999999999995},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.90600000000000003,"5min":0.98799999999999999,"15min":1.044},"last":0.69299999999999995}]},{"osd":6,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.68100000000000005,"15min":0.70099999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.86299999999999999,"5min":1.0569999999999999,"15min":1.1140000000000001},"last":0.755},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.69399999999999995,"15min":0.70999999999999996},"min":{"1min":0.47999999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89400000000000002,"5min":1.002,"15min":1.0720000000000001},"last":0.63500000000000001}]},{"osd":7,"last update":"Sat Jan 27 00:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.71499999999999997,"15min":0.72799999999999998},"min":{"1min":0.55600000000000005,"5min":0.50900000000000001,"15min":0.48999999999999999},"max":{"1min":0.97499999999999998,"5min":1.0860000000000001,"15min":1.1479999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.73899999999999999,"15min":0.76200000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.98899999999999999,"5min":1.022,"15min":1.1299999999999999},"last":0.72699999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575775,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315992,"kb_used_data":2320,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055848,"statfs":{"total":96636764160,"available":96313188352,"internally_reserved":0,"allocated":2375680,"data_stored":1227348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.39600000000000002,"15min":0.40100000000000002},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.50600000000000001,"5min":0.63,"15min":0.752},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.443,"5min":0.45700000000000002,"15min":0.45300000000000001},"min":{"1min":0.32900000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.57899999999999996,"5min":0.68799999999999994,"15min":0.85699999999999998},"last":0.33900000000000002}]},{"osd":1,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.46600000000000003},"min":{"1min":0.307,"5min":0.29699999999999999,"15min":0.27400000000000002},"max":{"1min":0.61499999999999999,"5min":0.69699999999999995,"15min":0.94599999999999995},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.498,"15min":0.498},"min":{"1min":0.36299999999999999,"5min":0.34399999999999997,"15min":0.316},"max":{"1min":0.69299999999999995,"5min":0.745,"15min":0.745},"last":0.47599999999999998}]},{"osd":2,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.35699999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64900000000000002,"5min":0.67900000000000005,"15min":0.84499999999999997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.438,"5min":0.46000000000000002,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.66600000000000004,"5min":0.72699999999999998,"15min":0.82299999999999995},"last":0.40000000000000002}]},{"osd":4,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.69299999999999995,"15min":0.68999999999999995},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.88400000000000001,"5min":0.89300000000000002,"15min":1},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.48199999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.98699999999999999,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.71999999999999997}]},{"osd":5,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.69299999999999995,"15min":0.68799999999999994},"min":{"1min":0.505,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.88900000000000001,"5min":0.94499999999999995,"15min":1.0429999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68400000000000005,"15min":0.68700000000000006},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90400000000000003,"5min":0.99099999999999999,"15min":1.0289999999999999},"last":0.65300000000000002}]},{"osd":6,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.68300000000000005,"15min":0.69099999999999995},"min":{"1min":0.53000000000000003,"5min":0.52000000000000002,"15min":0.48899999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.0820000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.71799999999999997,"15min":0.71599999999999997},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.44},"max":{"1min":0.91600000000000004,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.77500000000000002}]},{"osd":7,"last update":"Sat Jan 27 00:47:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.69999999999999996,"15min":0.71399999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.96399999999999997,"5min":1.081,"15min":1.081},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.55800000000000005,"5min":0.48999999999999999,"15min":0.46300000000000002},"max":{"1min":0.91700000000000004,"5min":1.0229999999999999,"15min":1.1240000000000001},"last":0.79900000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575777,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311892,"kb_used_data":2316,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059948,"statfs":{"total":96636764160,"available":96317386752,"internally_reserved":0,"allocated":2371584,"data_stored":1225972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.32100000000000001},"max":{"1min":0.73799999999999999,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.59699999999999998,"15min":0.61499999999999999},"min":{"1min":0.42199999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81200000000000006,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59699999999999998}]},{"osd":1,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61699999999999999,"15min":0.624},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61299999999999999,"15min":0.624},"min":{"1min":0.42399999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":0.73399999999999999}]},{"osd":2,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.58399999999999996,"15min":0.59999999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73799999999999999,"5min":0.81899999999999995,"15min":0.84499999999999997},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.62,"15min":0.63100000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.38100000000000001},"max":{"1min":0.81599999999999995,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.72099999999999997}]},{"osd":3,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.63400000000000001,"15min":0.628},"min":{"1min":0.43099999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.3860000000000001,"5min":2.3860000000000001,"15min":2.3860000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60699999999999998,"15min":0.62},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80200000000000005,"5min":0.80600000000000005,"15min":0.876},"last":0.70999999999999996}]},{"osd":5,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.48499999999999999,"15min":0.48599999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.28699999999999998},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.49199999999999999,"15min":0.49399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80200000000000005,"5min":0.876,"15min":0.876},"last":0.48199999999999998}]},{"osd":6,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48899999999999999,"15min":0.49299999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.82399999999999995,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.442},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54000000000000004,"15min":0.53900000000000003},"min":{"1min":0.35099999999999998,"5min":0.34300000000000003,"15min":0.32800000000000001},"max":{"1min":0.746,"5min":1.028,"15min":1.028},"last":0.46200000000000002}]},{"osd":7,"last update":"Sat Jan 27 00:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54700000000000004,"15min":0.55600000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33700000000000002},"max":{"1min":0.84299999999999997,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53700000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575777,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311884,"kb_used_data":2308,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059956,"statfs":{"total":96636764160,"available":96317394944,"internally_reserved":0,"allocated":2363392,"data_stored":1225360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61399999999999999,"15min":0.60799999999999998},"min":{"1min":0.379,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76200000000000001,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.501},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.61599999999999999,"15min":0.623},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38600000000000001},"max":{"1min":0.76600000000000001,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.46700000000000003}]},{"osd":1,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65900000000000003,"15min":0.64900000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.44},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66400000000000003,"15min":0.67100000000000004},"min":{"1min":0.44600000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80300000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.60799999999999998}]},{"osd":2,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.61399999999999999,"15min":0.60299999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.71999999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.79100000000000004,"5min":0.876,"15min":0.876},"last":0.63500000000000001}]},{"osd":3,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60399999999999998,"15min":0.60299999999999998},"min":{"1min":0.38200000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.70099999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.64200000000000002,"15min":0.63400000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.79600000000000004,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.66000000000000003}]},{"osd":4,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.47499999999999998,"15min":0.47599999999999998},"min":{"1min":0.307,"5min":0.30399999999999999,"15min":0.30299999999999999},"max":{"1min":0.67600000000000005,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.82799999999999996},"last":0.68100000000000005}]},{"osd":6,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50700000000000001,"15min":0.504},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.31900000000000001},"max":{"1min":0.67400000000000004,"5min":0.71399999999999997,"15min":0.79900000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.379,"5min":0.36799999999999999,"15min":0.36099999999999999},"max":{"1min":0.70099999999999996,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.73799999999999999}]},{"osd":7,"last update":"Sat Jan 27 00:46:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34699999999999998},"max":{"1min":0.747,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.45800000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.71299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T00:47:29.783 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-27T00:47:29.783 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-27T00:47:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.961+0000 7f2055f76700 1 Processor -- start 2024-01-27T00:47:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.961+0000 7f2055f76700 1 -- start start 2024-01-27T00:47:29.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.961+0000 7f2055f76700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2050157860 0x7f20500cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.961+0000 7f2055f76700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f205006ddb0 0x7f20500cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.961+0000 7f2055f76700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20500cea80 con 0x7f2050157490 2024-01-27T00:47:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.961+0000 7f2055f76700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20500cdc60 con 0x7f205006ddb0 2024-01-27T00:47:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.961+0000 7f2055f76700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20500b2ae0 con 0x7f2050157860 2024-01-27T00:47:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.962+0000 7f204ffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f205006ddb0 0x7f20500cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.962+0000 7f204ffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f205006ddb0 0x7f20500cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40630/0 (socket says 172.21.15.46:40630) 2024-01-27T00:47:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.962+0000 7f204ffff700 1 -- 172.21.15.46:0/2861368757 learned_addr learned my addr 172.21.15.46:0/2861368757 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.962+0000 7f204f7fe700 1 --1- 172.21.15.46:0/2861368757 >> v1:172.21.15.46:6789/0 conn(0x7f2050157490 0x7f205006d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45684/0 (socket says 172.21.15.46:45684) 2024-01-27T00:47:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.962+0000 7f204effd700 1 --2- 172.21.15.46:0/2861368757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2050157860 0x7f20500cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.962+0000 7f204ffff700 1 -- 172.21.15.46:0/2861368757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2050157860 msgr2=0x7f20500cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.962+0000 7f204ffff700 1 --2- 172.21.15.46:0/2861368757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2050157860 0x7f20500cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.962+0000 7f204ffff700 1 -- 172.21.15.46:0/2861368757 >> v1:172.21.15.46:6789/0 conn(0x7f2050157490 legacy=0x7f205006d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.962+0000 7f204ffff700 1 -- 172.21.15.46:0/2861368757 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20500cde70 con 0x7f205006ddb0 2024-01-27T00:47:29.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.962+0000 7f204ffff700 1 --2- 172.21.15.46:0/2861368757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f205006ddb0 0x7f20500cd620 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f2048009f20 tx=0x7f2048009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=800a1c822f865ac server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:29.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.962+0000 7f204e7fc700 1 -- 172.21.15.46:0/2861368757 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204800e070 con 0x7f205006ddb0 2024-01-27T00:47:29.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.962+0000 7f204e7fc700 1 -- 172.21.15.46:0/2861368757 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204800ba80 con 0x7f205006ddb0 2024-01-27T00:47:29.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.962+0000 7f204e7fc700 1 -- 172.21.15.46:0/2861368757 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20480092e0 con 0x7f205006ddb0 2024-01-27T00:47:29.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.963+0000 7f2055f76700 1 -- 172.21.15.46:0/2861368757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f205006ddb0 msgr2=0x7f20500cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:29.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.963+0000 7f2055f76700 1 --2- 172.21.15.46:0/2861368757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f205006ddb0 0x7f20500cd620 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f2048009f20 tx=0x7f2048009840 comp rx=0 tx=0).stop 2024-01-27T00:47:29.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.963+0000 7f2055f76700 1 -- 172.21.15.46:0/2861368757 shutdown_connections 2024-01-27T00:47:29.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.963+0000 7f2055f76700 1 --2- 172.21.15.46:0/2861368757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f205006ddb0 0x7f20500cd620 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:29.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.963+0000 7f2055f76700 1 --2- 172.21.15.46:0/2861368757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2050157860 0x7f20500cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:29.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.963+0000 7f2055f76700 1 -- 172.21.15.46:0/2861368757 >> 172.21.15.46:0/2861368757 conn(0x7f2050067930 msgr2=0x7f2050067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:29.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.963+0000 7f2055f76700 1 -- 172.21.15.46:0/2861368757 shutdown_connections 2024-01-27T00:47:29.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.963+0000 7f2055f76700 1 -- 172.21.15.46:0/2861368757 wait complete. 2024-01-27T00:47:29.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.964+0000 7f2055f76700 1 Processor -- start 2024-01-27T00:47:29.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.965+0000 7f2055f76700 1 -- start start 2024-01-27T00:47:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.965+0000 7f2055f76700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f205006ddb0 0x7f2050158360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.965+0000 7f2055f76700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2050157490 0x7f20500b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.965+0000 7f2055f76700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2050157860 0x7f205014bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.965+0000 7f2055f76700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20500ae360 con 0x7f2050157860 2024-01-27T00:47:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.965+0000 7f2055f76700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205015a140 con 0x7f205006ddb0 2024-01-27T00:47:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.965+0000 7f2055f76700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205015a420 con 0x7f2050157490 2024-01-27T00:47:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.965+0000 7f204ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2050157860 0x7f205014bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.965+0000 7f204ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2050157860 0x7f205014bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40638/0 (socket says 172.21.15.46:40638) 2024-01-27T00:47:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.965+0000 7f204ffff700 1 -- 172.21.15.46:0/2663155016 learned_addr learned my addr 172.21.15.46:0/2663155016 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.966+0000 7f204f7fe700 1 --2- 172.21.15.46:0/2663155016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f205006ddb0 0x7f2050158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.966+0000 7f204effd700 1 --2- 172.21.15.46:0/2663155016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2050157490 0x7f20500b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.966+0000 7f204ffff700 1 -- 172.21.15.46:0/2663155016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2050157490 msgr2=0x7f20500b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.966+0000 7f204ffff700 1 --2- 172.21.15.46:0/2663155016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2050157490 0x7f20500b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.966+0000 7f204ffff700 1 -- 172.21.15.46:0/2663155016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f205006ddb0 msgr2=0x7f2050158360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:29.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.966+0000 7f204ffff700 1 --2- 172.21.15.46:0/2663155016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f205006ddb0 0x7f2050158360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:29.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.966+0000 7f204ffff700 1 -- 172.21.15.46:0/2663155016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2048009ae0 con 0x7f2050157860 2024-01-27T00:47:29.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.966+0000 7f204ffff700 1 --2- 172.21.15.46:0/2663155016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2050157860 0x7f205014bdf0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f205006d060 tx=0x7f2048018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:29.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.966+0000 7f20377fe700 1 -- 172.21.15.46:0/2663155016 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20480032f0 con 0x7f2050157860 2024-01-27T00:47:29.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.966+0000 7f20377fe700 1 -- 172.21.15.46:0/2663155016 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2048008d20 con 0x7f2050157860 2024-01-27T00:47:29.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.966+0000 7f2055f76700 1 -- 172.21.15.46:0/2663155016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20500adff0 con 0x7f2050157860 2024-01-27T00:47:29.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.966+0000 7f20377fe700 1 -- 172.21.15.46:0/2663155016 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2048026690 con 0x7f2050157860 2024-01-27T00:47:29.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.966+0000 7f2055f76700 1 -- 172.21.15.46:0/2663155016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2050159da0 con 0x7f2050157860 2024-01-27T00:47:29.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.967+0000 7f2055f76700 1 -- 172.21.15.46:0/2663155016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20500ae510 con 0x7f2050157860 2024-01-27T00:47:29.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.968+0000 7f20377fe700 1 -- 172.21.15.46:0/2663155016 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2048003040 con 0x7f2050157860 2024-01-27T00:47:29.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.968+0000 7f20377fe700 1 --2- 172.21.15.46:0/2663155016 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f203c066f30 0x7f203c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:29.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.968+0000 7f20377fe700 1 -- 172.21.15.46:0/2663155016 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f20480867e0 con 0x7f2050157860 2024-01-27T00:47:29.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.968+0000 7f204f7fe700 1 --2- 172.21.15.46:0/2663155016 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f203c066f30 0x7f203c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:29.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.969+0000 7f204f7fe700 1 --2- 172.21.15.46:0/2663155016 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f203c066f30 0x7f203c0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f20500afec0 tx=0x7f2040007eb0 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:29.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:29.973+0000 7f20377fe700 1 -- 172.21.15.46:0/2663155016 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f20480536c0 con 0x7f2050157860 2024-01-27T00:47:30.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.260+0000 7f2055f76700 1 -- 172.21.15.46:0/2663155016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f205004c710 con 0x7f2050157860 2024-01-27T00:47:30.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.264+0000 7f20377fe700 1 -- 172.21.15.46:0/2663155016 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v46) v1 ==== 190+0+0 (secure 0 0 0) 0x7f2048056ce0 con 0x7f2050157860 2024-01-27T00:47:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.266+0000 7f2055f76700 1 -- 172.21.15.46:0/2663155016 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f203c066f30 msgr2=0x7f203c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.266+0000 7f2055f76700 1 --2- 172.21.15.46:0/2663155016 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f203c066f30 0x7f203c0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f20500afec0 tx=0x7f2040007eb0 comp rx=0 tx=0).stop 2024-01-27T00:47:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.266+0000 7f2055f76700 1 -- 172.21.15.46:0/2663155016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2050157860 msgr2=0x7f205014bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.266+0000 7f2055f76700 1 --2- 172.21.15.46:0/2663155016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2050157860 0x7f205014bdf0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f205006d060 tx=0x7f2048018000 comp rx=0 tx=0).stop 2024-01-27T00:47:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.266+0000 7f2055f76700 1 -- 172.21.15.46:0/2663155016 shutdown_connections 2024-01-27T00:47:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.266+0000 7f2055f76700 1 --2- 172.21.15.46:0/2663155016 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f203c066f30 0x7f203c0693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.266+0000 7f2055f76700 1 --2- 172.21.15.46:0/2663155016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2050157860 0x7f205014bdf0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.266+0000 7f2055f76700 1 --2- 172.21.15.46:0/2663155016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f205006ddb0 0x7f2050158360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.266+0000 7f2055f76700 1 --2- 172.21.15.46:0/2663155016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2050157490 0x7f20500b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.266+0000 7f2055f76700 1 -- 172.21.15.46:0/2663155016 >> 172.21.15.46:0/2663155016 conn(0x7f2050067930 msgr2=0x7f2050156400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:30.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.266+0000 7f2055f76700 1 -- 172.21.15.46:0/2663155016 shutdown_connections 2024-01-27T00:47:30.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.266+0000 7f2055f76700 1 -- 172.21.15.46:0/2663155016 wait complete. 2024-01-27T00:47:30.268 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-01-27T00:47:30.280 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T00:47:30.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.412+0000 7f51968e6700 1 Processor -- start 2024-01-27T00:47:30.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.412+0000 7f51968e6700 1 -- start start 2024-01-27T00:47:30.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.412+0000 7f51968e6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5190159760 0x7f51900b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:30.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.412+0000 7f51968e6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f51900b3060 0x7f51900b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:30.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.413+0000 7f51968e6700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519006f940 con 0x7f5190159b30 2024-01-27T00:47:30.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.413+0000 7f51968e6700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f519006ed30 con 0x7f51900b3060 2024-01-27T00:47:30.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.413+0000 7f51968e6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f519006f010 con 0x7f5190159760 2024-01-27T00:47:30.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.413+0000 7f518f7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f51900b3060 0x7f51900b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.413+0000 7f518f7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f51900b3060 0x7f51900b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47752/0 (socket says 172.21.15.46:47752) 2024-01-27T00:47:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.413+0000 7f518f7fe700 1 -- 172.21.15.46:0/378229463 learned_addr learned my addr 172.21.15.46:0/378229463 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.413+0000 7f518ffff700 1 --2- 172.21.15.46:0/378229463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5190159760 0x7f51900b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.413+0000 7f518f7fe700 1 -- 172.21.15.46:0/378229463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5190159760 msgr2=0x7f51900b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.413+0000 7f518f7fe700 1 --2- 172.21.15.46:0/378229463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5190159760 0x7f51900b2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.413+0000 7f518f7fe700 1 -- 172.21.15.46:0/378229463 >> v1:172.21.15.106:6789/0 conn(0x7f5190159b30 legacy=0x7f519015d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.413+0000 7f518f7fe700 1 -- 172.21.15.46:0/378229463 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519006f2f0 con 0x7f51900b3060 2024-01-27T00:47:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.413+0000 7f518f7fe700 1 --2- 172.21.15.46:0/378229463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51900b3060 0x7f51900b0bb0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f51780099b0 tx=0x7f5178009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5a3f21d465fccc server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.413+0000 7f518effd700 1 -- 172.21.15.46:0/378229463 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5178009ea0 con 0x7f51900b3060 2024-01-27T00:47:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.413+0000 7f518effd700 1 -- 172.21.15.46:0/378229463 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f517800b860 con 0x7f51900b3060 2024-01-27T00:47:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.413+0000 7f518effd700 1 -- 172.21.15.46:0/378229463 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517800bc20 con 0x7f51900b3060 2024-01-27T00:47:30.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.414+0000 7f51968e6700 1 -- 172.21.15.46:0/378229463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51900b3060 msgr2=0x7f51900b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:30.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.414+0000 7f51968e6700 1 --2- 172.21.15.46:0/378229463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51900b3060 0x7f51900b0bb0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f51780099b0 tx=0x7f5178009250 comp rx=0 tx=0).stop 2024-01-27T00:47:30.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.414+0000 7f51968e6700 1 -- 172.21.15.46:0/378229463 shutdown_connections 2024-01-27T00:47:30.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.414+0000 7f51968e6700 1 --2- 172.21.15.46:0/378229463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5190159760 0x7f51900b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:30.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.414+0000 7f51968e6700 1 --2- 172.21.15.46:0/378229463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51900b3060 0x7f51900b0bb0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:30.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.414+0000 7f51968e6700 1 -- 172.21.15.46:0/378229463 >> 172.21.15.46:0/378229463 conn(0x7f5190069510 msgr2=0x7f51900aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:30.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.414+0000 7f51968e6700 1 -- 172.21.15.46:0/378229463 shutdown_connections 2024-01-27T00:47:30.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.414+0000 7f51968e6700 1 -- 172.21.15.46:0/378229463 wait complete. 2024-01-27T00:47:30.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.416+0000 7f51968e6700 1 Processor -- start 2024-01-27T00:47:30.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.416+0000 7f51968e6700 1 -- start start 2024-01-27T00:47:30.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.417+0000 7f51968e6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51900b3060 0x7f519015b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:30.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.417+0000 7f51968e6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5190159760 0x7f519015deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:30.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.417+0000 7f51968e6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5190159b30 0x7f51900c2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:30.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.417+0000 7f51968e6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519015eb70 con 0x7f51900b3060 2024-01-27T00:47:30.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.417+0000 7f51968e6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5190169dc0 con 0x7f5190159760 2024-01-27T00:47:30.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.417+0000 7f51968e6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f519015e5b0 con 0x7f5190159b30 2024-01-27T00:47:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.417+0000 7f518ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51900b3060 0x7f519015b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.417+0000 7f518ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51900b3060 0x7f519015b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47762/0 (socket says 172.21.15.46:47762) 2024-01-27T00:47:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.417+0000 7f518ffff700 1 -- 172.21.15.46:0/1498334085 learned_addr learned my addr 172.21.15.46:0/1498334085 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.417+0000 7f518f7fe700 1 --2- 172.21.15.46:0/1498334085 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5190159760 0x7f519015deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.417+0000 7f5194e83700 1 --2- 172.21.15.46:0/1498334085 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5190159b30 0x7f51900c2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.418+0000 7f518ffff700 1 -- 172.21.15.46:0/1498334085 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5190159b30 msgr2=0x7f51900c2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.418+0000 7f518ffff700 1 --2- 172.21.15.46:0/1498334085 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5190159b30 0x7f51900c2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.418+0000 7f518ffff700 1 -- 172.21.15.46:0/1498334085 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5190159760 msgr2=0x7f519015deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.418+0000 7f518ffff700 1 --2- 172.21.15.46:0/1498334085 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5190159760 0x7f519015deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.418+0000 7f518ffff700 1 -- 172.21.15.46:0/1498334085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51780094f0 con 0x7f51900b3060 2024-01-27T00:47:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.418+0000 7f518f7fe700 1 --2- 172.21.15.46:0/1498334085 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5190159760 0x7f519015deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:47:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.418+0000 7f518ffff700 1 --2- 172.21.15.46:0/1498334085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51900b3060 0x7f519015b810 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f518000ec80 tx=0x7f51800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.418+0000 7f5187fff700 1 -- 172.21.15.46:0/1498334085 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518000c9d0 con 0x7f51900b3060 2024-01-27T00:47:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.418+0000 7f5187fff700 1 -- 172.21.15.46:0/1498334085 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518000cb30 con 0x7f51900b3060 2024-01-27T00:47:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.418+0000 7f51968e6700 1 -- 172.21.15.46:0/1498334085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f519015e890 con 0x7f51900b3060 2024-01-27T00:47:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.418+0000 7f5187fff700 1 -- 172.21.15.46:0/1498334085 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5180013470 con 0x7f51900b3060 2024-01-27T00:47:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.418+0000 7f51968e6700 1 -- 172.21.15.46:0/1498334085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519016a2b0 con 0x7f51900b3060 2024-01-27T00:47:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.419+0000 7f51968e6700 1 -- 172.21.15.46:0/1498334085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f519004c710 con 0x7f51900b3060 2024-01-27T00:47:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.419+0000 7f5187fff700 1 -- 172.21.15.46:0/1498334085 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5180013820 con 0x7f51900b3060 2024-01-27T00:47:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.420+0000 7f5187fff700 1 --2- 172.21.15.46:0/1498334085 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f5170066f30 0x7f51700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:30.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.420+0000 7f5187fff700 1 -- 172.21.15.46:0/1498334085 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f5180084bb0 con 0x7f51900b3060 2024-01-27T00:47:30.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.420+0000 7f518f7fe700 1 --2- 172.21.15.46:0/1498334085 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f5170066f30 0x7f51700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:30.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.421+0000 7f518f7fe700 1 --2- 172.21.15.46:0/1498334085 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f5170066f30 0x7f51700693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f51780099b0 tx=0x7f5178017000 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:30.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.424+0000 7f5187fff700 1 -- 172.21.15.46:0/1498334085 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5180051b30 con 0x7f51900b3060 2024-01-27T00:47:30.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.677+0000 7f51968e6700 1 -- 172.21.15.46:0/1498334085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51900ad530 con 0x7f51900b3060 2024-01-27T00:47:30.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.677+0000 7f5187fff700 1 -- 172.21.15.46:0/1498334085 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4601 (secure 0 0 0) 0x7f5180055150 con 0x7f51900b3060 2024-01-27T00:47:30.679 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:47:30.679 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":97,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8435,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/4207115513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":4207115513},{"type":"v1","addr":"172.21.15.106:6835","nonce":4207115513}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91},{"gid":8438,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/282430549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":282430549},{"type":"v1","addr":"172.21.15.46:6834","nonce":282430549}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9508,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/361546207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":361546207},{"type":"v1","addr":"172.21.15.106:6834","nonce":361546207}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T00:47:19.926407+0000","modified":"2024-01-27T00:47:23.511130+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9828":{"gid":9828,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2544525048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2544525048},{"type":"v1","addr":"172.21.15.46:6835","nonce":2544525048}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2024-01-27T00:47:30.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.679+0000 7f51968e6700 1 -- 172.21.15.46:0/1498334085 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f5170066f30 msgr2=0x7f51700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:30.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.679+0000 7f51968e6700 1 --2- 172.21.15.46:0/1498334085 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f5170066f30 0x7f51700693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f51780099b0 tx=0x7f5178017000 comp rx=0 tx=0).stop 2024-01-27T00:47:30.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.679+0000 7f51968e6700 1 -- 172.21.15.46:0/1498334085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51900b3060 msgr2=0x7f519015b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:30.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.679+0000 7f51968e6700 1 --2- 172.21.15.46:0/1498334085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51900b3060 0x7f519015b810 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f518000ec80 tx=0x7f51800091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:30.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.680+0000 7f51968e6700 1 -- 172.21.15.46:0/1498334085 shutdown_connections 2024-01-27T00:47:30.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.680+0000 7f51968e6700 1 --2- 172.21.15.46:0/1498334085 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f5170066f30 0x7f51700693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:30.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.680+0000 7f51968e6700 1 --2- 172.21.15.46:0/1498334085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51900b3060 0x7f519015b810 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:30.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.680+0000 7f51968e6700 1 --2- 172.21.15.46:0/1498334085 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5190159b30 0x7f51900c2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:30.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.680+0000 7f51968e6700 1 --2- 172.21.15.46:0/1498334085 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5190159760 0x7f519015deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:30.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.680+0000 7f51968e6700 1 -- 172.21.15.46:0/1498334085 >> 172.21.15.46:0/1498334085 conn(0x7f5190069510 msgr2=0x7f51900ada50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:30.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.680+0000 7f51968e6700 1 -- 172.21.15.46:0/1498334085 shutdown_connections 2024-01-27T00:47:30.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:30.680+0000 7f51968e6700 1 -- 172.21.15.46:0/1498334085 wait complete. 2024-01-27T00:47:30.681 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 97 2024-01-27T00:47:31.097 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-27T00:47:19.926407+0000', 'modified': '2024-01-27T00:47:23.511130+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9828}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9828': {'gid': 9828, 'name': 'c', 'rank': 0, 'incarnation': 93, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6835/2544525048', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 2544525048}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 2544525048}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-27T00:47:31.097 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-27T00:47:31.097 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-27T00:47:31.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.238+0000 7f0a69171700 1 Processor -- start 2024-01-27T00:47:31.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.238+0000 7f0a69171700 1 -- start start 2024-01-27T00:47:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.239+0000 7f0a69171700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0a640b2160 0x7f0a6415a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.239+0000 7f0a69171700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a640b2530 0x7f0a640abe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.239+0000 7f0a69171700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a6415da10 0x7f0a6415de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.239+0000 7f0a69171700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a640a9b90 con 0x7f0a640b2530 2024-01-27T00:47:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.239+0000 7f0a69171700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a640aa210 con 0x7f0a640b2160 2024-01-27T00:47:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.239+0000 7f0a69171700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6415d580 con 0x7f0a6415da10 2024-01-27T00:47:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.239+0000 7f0a6259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a640b2530 0x7f0a640abe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.239+0000 7f0a6259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a640b2530 0x7f0a640abe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47768/0 (socket says 172.21.15.46:47768) 2024-01-27T00:47:31.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.239+0000 7f0a6259c700 1 -- 172.21.15.46:0/3824704373 learned_addr learned my addr 172.21.15.46:0/3824704373 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:31.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.239+0000 7f0a62d9d700 1 --2- 172.21.15.46:0/3824704373 >> v2:172.21.15.106:3300/0 conn(0x7f0a640b2160 0x7f0a6415a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.239+0000 7f0a6359e700 1 --2- 172.21.15.46:0/3824704373 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a6415da10 0x7f0a6415de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.239+0000 7f0a6259c700 1 -- 172.21.15.46:0/3824704373 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a6415da10 msgr2=0x7f0a6415de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:31.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.239+0000 7f0a6259c700 1 --2- 172.21.15.46:0/3824704373 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a6415da10 0x7f0a6415de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.239+0000 7f0a6259c700 1 -- 172.21.15.46:0/3824704373 >> v2:172.21.15.106:3300/0 conn(0x7f0a640b2160 msgr2=0x7f0a6415a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:31.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.239+0000 7f0a6259c700 1 --2- 172.21.15.46:0/3824704373 >> v2:172.21.15.106:3300/0 conn(0x7f0a640b2160 0x7f0a6415a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.240+0000 7f0a6259c700 1 -- 172.21.15.46:0/3824704373 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a640ac390 con 0x7f0a640b2530 2024-01-27T00:47:31.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.240+0000 7f0a62d9d700 1 --2- 172.21.15.46:0/3824704373 >> v2:172.21.15.106:3300/0 conn(0x7f0a640b2160 0x7f0a6415a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:47:31.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.240+0000 7f0a6259c700 1 --2- 172.21.15.46:0/3824704373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a640b2530 0x7f0a640abe50 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f0a64063ca0 tx=0x7f0a54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79db350fe5864bdd server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:31.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.240+0000 7f0a61d9b700 1 -- 172.21.15.46:0/3824704373 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5400b860 con 0x7f0a640b2530 2024-01-27T00:47:31.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.240+0000 7f0a61d9b700 1 -- 172.21.15.46:0/3824704373 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5400b9c0 con 0x7f0a640b2530 2024-01-27T00:47:31.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.240+0000 7f0a61d9b700 1 -- 172.21.15.46:0/3824704373 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5400bd80 con 0x7f0a640b2530 2024-01-27T00:47:31.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.240+0000 7f0a69171700 1 -- 172.21.15.46:0/3824704373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a640b2530 msgr2=0x7f0a640abe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:31.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.240+0000 7f0a69171700 1 --2- 172.21.15.46:0/3824704373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a640b2530 0x7f0a640abe50 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f0a64063ca0 tx=0x7f0a54009250 comp rx=0 tx=0).stop 2024-01-27T00:47:31.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.240+0000 7f0a69171700 1 -- 172.21.15.46:0/3824704373 shutdown_connections 2024-01-27T00:47:31.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.241+0000 7f0a69171700 1 --2- 172.21.15.46:0/3824704373 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a6415da10 0x7f0a6415de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.241+0000 7f0a69171700 1 --2- 172.21.15.46:0/3824704373 >> v2:172.21.15.106:3300/0 conn(0x7f0a640b2160 0x7f0a6415a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.241+0000 7f0a69171700 1 --2- 172.21.15.46:0/3824704373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a640b2530 0x7f0a640abe50 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.241+0000 7f0a69171700 1 -- 172.21.15.46:0/3824704373 >> 172.21.15.46:0/3824704373 conn(0x7f0a640ccbc0 msgr2=0x7f0a640ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:31.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.241+0000 7f0a69171700 1 -- 172.21.15.46:0/3824704373 shutdown_connections 2024-01-27T00:47:31.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.241+0000 7f0a69171700 1 -- 172.21.15.46:0/3824704373 wait complete. 2024-01-27T00:47:31.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.242+0000 7f0a69171700 1 Processor -- start 2024-01-27T00:47:31.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.242+0000 7f0a69171700 1 -- start start 2024-01-27T00:47:31.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.243+0000 7f0a69171700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a640b2160 0x7f0a641425b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.243+0000 7f0a69171700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a640b2530 0x7f0a640ad470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.243+0000 7f0a69171700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a6415da10 0x7f0a640ad9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.243+0000 7f0a69171700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a640ae1d0 con 0x7f0a640b2160 2024-01-27T00:47:31.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.243+0000 7f0a69171700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a640ae4b0 con 0x7f0a640b2530 2024-01-27T00:47:31.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.243+0000 7f0a69171700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a640ae790 con 0x7f0a6415da10 2024-01-27T00:47:31.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.243+0000 7f0a62d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a640b2160 0x7f0a641425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.243+0000 7f0a62d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a640b2160 0x7f0a641425b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47772/0 (socket says 172.21.15.46:47772) 2024-01-27T00:47:31.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.243+0000 7f0a62d9d700 1 -- 172.21.15.46:0/1849011790 learned_addr learned my addr 172.21.15.46:0/1849011790 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:31.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.243+0000 7f0a6359e700 1 --2- 172.21.15.46:0/1849011790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a6415da10 0x7f0a640ad9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.243+0000 7f0a6259c700 1 --2- 172.21.15.46:0/1849011790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a640b2530 0x7f0a640ad470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.243+0000 7f0a62d9d700 1 -- 172.21.15.46:0/1849011790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a6415da10 msgr2=0x7f0a640ad9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:31.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.243+0000 7f0a62d9d700 1 --2- 172.21.15.46:0/1849011790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a6415da10 0x7f0a640ad9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.243+0000 7f0a62d9d700 1 -- 172.21.15.46:0/1849011790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a640b2530 msgr2=0x7f0a640ad470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:31.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.243+0000 7f0a62d9d700 1 --2- 172.21.15.46:0/1849011790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a640b2530 0x7f0a640ad470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.243+0000 7f0a62d9d700 1 -- 172.21.15.46:0/1849011790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a540094f0 con 0x7f0a640b2160 2024-01-27T00:47:31.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.243+0000 7f0a62d9d700 1 --2- 172.21.15.46:0/1849011790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a640b2160 0x7f0a641425b0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f0a5800e8d0 tx=0x7f0a58008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:31.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.244+0000 7f0a4affd700 1 -- 172.21.15.46:0/1849011790 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a580093b0 con 0x7f0a640b2160 2024-01-27T00:47:31.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.244+0000 7f0a4affd700 1 -- 172.21.15.46:0/1849011790 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a58013410 con 0x7f0a640b2160 2024-01-27T00:47:31.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.244+0000 7f0a69171700 1 -- 172.21.15.46:0/1849011790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a640aea70 con 0x7f0a640b2160 2024-01-27T00:47:31.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.244+0000 7f0a4affd700 1 -- 172.21.15.46:0/1849011790 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a580137b0 con 0x7f0a640b2160 2024-01-27T00:47:31.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.244+0000 7f0a69171700 1 -- 172.21.15.46:0/1849011790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a640adef0 con 0x7f0a640b2160 2024-01-27T00:47:31.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.245+0000 7f0a4affd700 1 -- 172.21.15.46:0/1849011790 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0a58013be0 con 0x7f0a640b2160 2024-01-27T00:47:31.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.246+0000 7f0a4affd700 1 --2- 172.21.15.46:0/1849011790 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f0a50066f30 0x7f0a500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.246+0000 7f0a4affd700 1 -- 172.21.15.46:0/1849011790 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f0a58084800 con 0x7f0a640b2160 2024-01-27T00:47:31.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.246+0000 7f0a6259c700 1 --2- 172.21.15.46:0/1849011790 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f0a50066f30 0x7f0a500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.246+0000 7f0a6259c700 1 --2- 172.21.15.46:0/1849011790 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f0a50066f30 0x7f0a500693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0a64063ca0 tx=0x7f0a54017000 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:31.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.248+0000 7f0a72474b80 1 Processor -- start 2024-01-27T00:47:31.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.248+0000 7f0a72474b80 1 -- start start 2024-01-27T00:47:31.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.249+0000 7f0a72474b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f251f010 0x5614f251f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.249+0000 7f0a72474b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f2567360 0x5614f2549840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.249+0000 7f0a72474b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251e5d0 0x5614f2549d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.249+0000 7f0a72474b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f24229f0 con 0x5614f251e5d0 2024-01-27T00:47:31.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.249+0000 7f0a72474b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f2474130 con 0x5614f2567360 2024-01-27T00:47:31.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.249+0000 7f0a72474b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5614f2471930 con 0x5614f251f010 2024-01-27T00:47:31.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.249+0000 7f0a6359e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251e5d0 0x5614f2549d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.249+0000 7f0a6359e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251e5d0 0x5614f2549d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47784/0 (socket says 172.21.15.46:47784) 2024-01-27T00:47:31.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.249+0000 7f0a6359e700 1 -- 172.21.15.46:0/948124673 learned_addr learned my addr 172.21.15.46:0/948124673 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:31.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.249+0000 7f0a6259c700 1 --2- 172.21.15.46:0/948124673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f2567360 0x5614f2549840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.249+0000 7f0a62d9d700 1 --2- 172.21.15.46:0/948124673 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f251f010 0x5614f251f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.249+0000 7f0a6359e700 1 -- 172.21.15.46:0/948124673 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f251f010 msgr2=0x5614f251f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:31.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.249+0000 7f0a6359e700 1 --2- 172.21.15.46:0/948124673 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f251f010 0x5614f251f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.249+0000 7f0a6359e700 1 -- 172.21.15.46:0/948124673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f2567360 msgr2=0x5614f2549840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:31.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.249+0000 7f0a6359e700 1 --2- 172.21.15.46:0/948124673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f2567360 0x5614f2549840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.249+0000 7f0a6359e700 1 -- 172.21.15.46:0/948124673 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5614f22acb10 con 0x5614f251e5d0 2024-01-27T00:47:31.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.250+0000 7f0a6359e700 1 --2- 172.21.15.46:0/948124673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251e5d0 0x5614f2549d80 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009fc0 tx=0x7f0a5c0097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e80ac47a9668ff server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:31.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.250+0000 7f0a43fff700 1 -- 172.21.15.46:0/948124673 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c007d90 con 0x5614f251e5d0 2024-01-27T00:47:31.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.250+0000 7f0a43fff700 1 -- 172.21.15.46:0/948124673 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c015420 con 0x5614f251e5d0 2024-01-27T00:47:31.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.250+0000 7f0a43fff700 1 -- 172.21.15.46:0/948124673 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c0157c0 con 0x5614f251e5d0 2024-01-27T00:47:31.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.250+0000 7f0a72474b80 1 -- 172.21.15.46:0/948124673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251e5d0 msgr2=0x5614f2549d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:31.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.250+0000 7f0a72474b80 1 --2- 172.21.15.46:0/948124673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251e5d0 0x5614f2549d80 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009fc0 tx=0x7f0a5c0097b0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.250+0000 7f0a72474b80 1 -- 172.21.15.46:0/948124673 shutdown_connections 2024-01-27T00:47:31.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.250+0000 7f0a72474b80 1 --2- 172.21.15.46:0/948124673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251e5d0 0x5614f2549d80 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.250+0000 7f0a72474b80 1 --2- 172.21.15.46:0/948124673 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f251f010 0x5614f251f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.250+0000 7f0a72474b80 1 --2- 172.21.15.46:0/948124673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f2567360 0x5614f2549840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.250+0000 7f0a72474b80 1 -- 172.21.15.46:0/948124673 >> 172.21.15.46:0/948124673 conn(0x5614f251ea70 msgr2=0x5614f2567820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:31.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.251+0000 7f0a72474b80 1 -- 172.21.15.46:0/948124673 shutdown_connections 2024-01-27T00:47:31.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.251+0000 7f0a72474b80 1 -- 172.21.15.46:0/948124673 wait complete. 2024-01-27T00:47:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.251+0000 7f0a72474b80 1 -- start start 2024-01-27T00:47:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.251+0000 7f0a72474b80 1 Processor -- start 2024-01-27T00:47:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.252+0000 7f0a72474b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f251e5d0 0x5614f2574050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.252+0000 7f0a72474b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251f010 0x5614f2561590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.252+0000 7f0a72474b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f2567360 0x5614f254b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.252+0000 7f0a72474b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f246d720 con 0x5614f251f010 2024-01-27T00:47:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.252+0000 7f0a72474b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f24191b0 con 0x5614f2567360 2024-01-27T00:47:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.252+0000 7f0a72474b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5614f2418e50 con 0x5614f251e5d0 2024-01-27T00:47:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.252+0000 7f0a72474b80 10 client.0 _ll_register_callbacks cb 0x5614f227b7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-27T00:47:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.252+0000 7f0a6259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251f010 0x5614f2561590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.253+0000 7f0a6259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251f010 0x5614f2561590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47786/0 (socket says 172.21.15.46:47786) 2024-01-27T00:47:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.253+0000 7f0a6259c700 1 -- 172.21.15.46:0/1003244881 learned_addr learned my addr 172.21.15.46:0/1003244881 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.253+0000 7f0a62d9d700 1 --2- 172.21.15.46:0/1003244881 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f251e5d0 0x5614f2574050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.253+0000 7f0a6359e700 1 --2- 172.21.15.46:0/1003244881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f2567360 0x5614f254b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.253+0000 7f0a6259c700 1 -- 172.21.15.46:0/1003244881 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f251e5d0 msgr2=0x5614f2574050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.253+0000 7f0a6259c700 1 --2- 172.21.15.46:0/1003244881 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f251e5d0 0x5614f2574050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.253+0000 7f0a6259c700 1 -- 172.21.15.46:0/1003244881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f2567360 msgr2=0x5614f254b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.253+0000 7f0a6259c700 1 --2- 172.21.15.46:0/1003244881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f2567360 0x5614f254b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.253+0000 7f0a6259c700 1 -- 172.21.15.46:0/1003244881 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a5c011040 con 0x5614f251f010 2024-01-27T00:47:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.253+0000 7f0a6259c700 1 --2- 172.21.15.46:0/1003244881 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251f010 0x5614f2561590 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400b3c0 tx=0x7f0a54020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2e5e38fda0322d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.253+0000 7f0a417fa700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-27T00:47:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.253+0000 7f0a417fa700 1 -- 172.21.15.46:0/1003244881 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5402b020 con 0x5614f251f010 2024-01-27T00:47:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.253+0000 7f0a417fa700 1 -- 172.21.15.46:0/1003244881 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f0a54026440 con 0x5614f251f010 2024-01-27T00:47:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.253+0000 7f0a72474b80 1 -- 172.21.15.46:0/1003244881 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5614f2512190 con 0x5614f251f010 2024-01-27T00:47:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.253+0000 7f0a417fa700 1 -- 172.21.15.46:0/1003244881 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a5402e070 con 0x5614f251f010 2024-01-27T00:47:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.253+0000 7f0a72474b80 10 client.9987 fetch_fsmap learned FSMap version 97 2024-01-27T00:47:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.253+0000 7f0a72474b80 1 -- 172.21.15.46:0/1003244881 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) v3 -- 0x5614f243b290 con 0x5614f251f010 2024-01-27T00:47:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.254+0000 7f0a417fa700 1 -- 172.21.15.46:0/1003244881 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5614f2512190 con 0x5614f251f010 2024-01-27T00:47:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.254+0000 7f0a417fa700 1 -- 172.21.15.46:0/1003244881 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 97) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f0a54026810 con 0x5614f251f010 2024-01-27T00:47:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.254+0000 7f0a72474b80 10 client.9987 fetch_fsmap finished waiting for FSMap version 97 2024-01-27T00:47:31.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.254+0000 7f0a72474b80 10 client.9987 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-27T00:47:31.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.254+0000 7f0a72474b80 20 client.9987 populate_metadata read hostname 'smithi046' 2024-01-27T00:47:31.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.254+0000 7f0a72474b80 1 --2- 172.21.15.46:0/1003244881 >> [v2:172.21.15.46:6833/2544525048,v1:172.21.15.46:6835/2544525048] conn(0x5614f251d550 0x5614f251d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.254+0000 7f0a72474b80 4 client.9987 mds_command: new command op to 9828 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-27T00:47:31.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.254+0000 7f0a72474b80 1 -- 172.21.15.46:0/1003244881 --> [v2:172.21.15.46:6833/2544525048,v1:172.21.15.46:6835/2544525048] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5614f240fee0 con 0x5614f251d550 2024-01-27T00:47:31.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.254+0000 7f0a62d9d700 1 --2- 172.21.15.46:0/1003244881 >> [v2:172.21.15.46:6833/2544525048,v1:172.21.15.46:6835/2544525048] conn(0x5614f251d550 0x5614f251d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.254+0000 7f0a417fa700 10 client.9987 ms_handle_connect on v2:172.21.15.46:6833/2544525048 2024-01-27T00:47:31.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.254+0000 7f0a62d9d700 1 --2- 172.21.15.46:0/1003244881 >> [v2:172.21.15.46:6833/2544525048,v1:172.21.15.46:6835/2544525048] conn(0x5614f251d550 0x5614f251d920 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba2bdf7144087777 server_cookie=37c66a34a7b48ed5 in_seq=0 out_seq=0 2024-01-27T00:47:31.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.255+0000 7f0a417fa700 1 -- 172.21.15.46:0/1003244881 <== mds.0 v2:172.21.15.46:6833/2544525048 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x5614f240fee0 con 0x5614f251d550 2024-01-27T00:47:31.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.255+0000 7f0a417fa700 10 client.9987 handle_command_reply: tid=0 2024-01-27T00:47:31.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.255+0000 7f0a72474b80 1 client.9987 shutdown 2024-01-27T00:47:31.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.255+0000 7f0a72474b80 1 -- 172.21.15.46:0/1003244881 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251f010 msgr2=0x5614f2561590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:31.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.255+0000 7f0a72474b80 1 --2- 172.21.15.46:0/1003244881 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251f010 0x5614f2561590 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400b3c0 tx=0x7f0a54020000 comp rx=0 tx=0).stop 2024-01-27T00:47:31.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.255+0000 7f0a72474b80 1 -- 172.21.15.46:0/1003244881 shutdown_connections 2024-01-27T00:47:31.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.255+0000 7f0a72474b80 1 --2- 172.21.15.46:0/1003244881 >> [v2:172.21.15.46:6833/2544525048,v1:172.21.15.46:6835/2544525048] conn(0x5614f251d550 0x5614f251d920 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.255+0000 7f0a72474b80 1 --2- 172.21.15.46:0/1003244881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f2567360 0x5614f254b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.255+0000 7f0a72474b80 1 --2- 172.21.15.46:0/1003244881 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f251e5d0 0x5614f2574050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.255+0000 7f0a72474b80 1 --2- 172.21.15.46:0/1003244881 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251f010 0x5614f2561590 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.255+0000 7f0a417fa700 0 client.9987 ms_handle_reset on v2:172.21.15.46:6833/2544525048 2024-01-27T00:47:31.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.255+0000 7f0a72474b80 1 -- 172.21.15.46:0/1003244881 >> 172.21.15.46:0/1003244881 conn(0x5614f251ea70 msgr2=0x5614f2519d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:31.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.256+0000 7f0a72474b80 1 -- 172.21.15.46:0/1003244881 shutdown_connections 2024-01-27T00:47:31.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.256+0000 7f0a72474b80 1 -- 172.21.15.46:0/1003244881 wait complete. 2024-01-27T00:47:31.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.256+0000 7f0a72474b80 20 client.9987 trim_cache size 0 max 16384 2024-01-27T00:47:31.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.274+0000 7f0a72474b80 1 Processor -- start 2024-01-27T00:47:31.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.274+0000 7f0a72474b80 1 -- start start 2024-01-27T00:47:31.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.274+0000 7f0a72474b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251e5d0 0x5614f255e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.274+0000 7f0a72474b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f251f010 0x5614f255eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.274+0000 7f0a72474b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f2567360 0x5614f2570350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.274+0000 7f0a72474b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f2365e40 con 0x5614f251e5d0 2024-01-27T00:47:31.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.274+0000 7f0a72474b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f2518d60 con 0x5614f2567360 2024-01-27T00:47:31.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.274+0000 7f0a72474b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5614f25619d0 con 0x5614f251f010 2024-01-27T00:47:31.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.275+0000 7f0a62d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251e5d0 0x5614f255e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.275+0000 7f0a62d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251e5d0 0x5614f255e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47800/0 (socket says 172.21.15.46:47800) 2024-01-27T00:47:31.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.275+0000 7f0a62d9d700 1 -- 172.21.15.46:0/1606478558 learned_addr learned my addr 172.21.15.46:0/1606478558 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:31.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.275+0000 7f0a6259c700 1 --2- 172.21.15.46:0/1606478558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f251f010 0x5614f255eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.275+0000 7f0a6359e700 1 --2- 172.21.15.46:0/1606478558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f2567360 0x5614f2570350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.275+0000 7f0a62d9d700 1 -- 172.21.15.46:0/1606478558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f251f010 msgr2=0x5614f255eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:31.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.275+0000 7f0a62d9d700 1 --2- 172.21.15.46:0/1606478558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f251f010 0x5614f255eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.275+0000 7f0a62d9d700 1 -- 172.21.15.46:0/1606478558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f2567360 msgr2=0x5614f2570350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:31.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.275+0000 7f0a62d9d700 1 --2- 172.21.15.46:0/1606478558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f2567360 0x5614f2570350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.275+0000 7f0a62d9d700 1 -- 172.21.15.46:0/1606478558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6405f3f0 con 0x5614f251e5d0 2024-01-27T00:47:31.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.275+0000 7f0a6359e700 1 --2- 172.21.15.46:0/1606478558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f2567360 0x5614f2570350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:47:31.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.275+0000 7f0a62d9d700 1 --2- 172.21.15.46:0/1606478558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251e5d0 0x5614f255e980 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f0a58055140 tx=0x7f0a58084040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1533f16bb5e868 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:31.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.275+0000 7f0a48ff9700 1 -- 172.21.15.46:0/1606478558 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a58054da0 con 0x5614f251e5d0 2024-01-27T00:47:31.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.275+0000 7f0a48ff9700 1 -- 172.21.15.46:0/1606478558 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a58013e90 con 0x5614f251e5d0 2024-01-27T00:47:31.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.275+0000 7f0a48ff9700 1 -- 172.21.15.46:0/1606478558 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a58053580 con 0x5614f251e5d0 2024-01-27T00:47:31.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.276+0000 7f0a72474b80 1 -- 172.21.15.46:0/1606478558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251e5d0 msgr2=0x5614f255e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:31.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.276+0000 7f0a72474b80 1 --2- 172.21.15.46:0/1606478558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251e5d0 0x5614f255e980 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f0a58055140 tx=0x7f0a58084040 comp rx=0 tx=0).stop 2024-01-27T00:47:31.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.276+0000 7f0a72474b80 1 -- 172.21.15.46:0/1606478558 shutdown_connections 2024-01-27T00:47:31.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.276+0000 7f0a72474b80 1 --2- 172.21.15.46:0/1606478558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251e5d0 0x5614f255e980 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.276+0000 7f0a72474b80 1 --2- 172.21.15.46:0/1606478558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f2567360 0x5614f2570350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.276+0000 7f0a72474b80 1 --2- 172.21.15.46:0/1606478558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f251f010 0x5614f255eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.276+0000 7f0a72474b80 1 -- 172.21.15.46:0/1606478558 >> 172.21.15.46:0/1606478558 conn(0x5614f251ea70 msgr2=0x5614f255d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:31.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.276+0000 7f0a72474b80 1 -- 172.21.15.46:0/1606478558 shutdown_connections 2024-01-27T00:47:31.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.276+0000 7f0a72474b80 1 -- 172.21.15.46:0/1606478558 wait complete. 2024-01-27T00:47:31.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.277+0000 7f0a72474b80 1 -- start start 2024-01-27T00:47:31.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.277+0000 7f0a72474b80 1 Processor -- start 2024-01-27T00:47:31.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a72474b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f251e5d0 0x5614f251a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a72474b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251f010 0x5614f25486b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a72474b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f2567360 0x5614f2548bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a72474b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f2580ea0 con 0x5614f251f010 2024-01-27T00:47:31.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a72474b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f2580850 con 0x5614f251e5d0 2024-01-27T00:47:31.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a72474b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5614f2580b30 con 0x5614f2567360 2024-01-27T00:47:31.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a72474b80 10 client.0 _ll_register_callbacks cb 0x5614f227b7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-27T00:47:31.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a6259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251f010 0x5614f25486b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a6259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251f010 0x5614f25486b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47808/0 (socket says 172.21.15.46:47808) 2024-01-27T00:47:31.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a6259c700 1 -- 172.21.15.46:0/1848874235 learned_addr learned my addr 172.21.15.46:0/1848874235 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:31.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a62d9d700 1 --2- 172.21.15.46:0/1848874235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f251e5d0 0x5614f251a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a6359e700 1 --2- 172.21.15.46:0/1848874235 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f2567360 0x5614f2548bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a6259c700 1 -- 172.21.15.46:0/1848874235 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f2567360 msgr2=0x5614f2548bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:31.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a6259c700 1 --2- 172.21.15.46:0/1848874235 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f2567360 0x5614f2548bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a6259c700 1 -- 172.21.15.46:0/1848874235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f251e5d0 msgr2=0x5614f251a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:31.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a6259c700 1 --2- 172.21.15.46:0/1848874235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f251e5d0 0x5614f251a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a6259c700 1 -- 172.21.15.46:0/1848874235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a640adef0 con 0x5614f251f010 2024-01-27T00:47:31.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a6259c700 1 --2- 172.21.15.46:0/1848874235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251f010 0x5614f25486b0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400b3c0 tx=0x7f0a5400b050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d24ede59269738f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:31.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a417fa700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-27T00:47:31.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a417fa700 1 -- 172.21.15.46:0/1848874235 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5402e040 con 0x5614f251f010 2024-01-27T00:47:31.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a417fa700 1 -- 172.21.15.46:0/1848874235 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f0a54026440 con 0x5614f251f010 2024-01-27T00:47:31.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.278+0000 7f0a72474b80 1 -- 172.21.15.46:0/1848874235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0a6405ec40 con 0x5614f251f010 2024-01-27T00:47:31.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.279+0000 7f0a417fa700 1 -- 172.21.15.46:0/1848874235 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a5402b040 con 0x5614f251f010 2024-01-27T00:47:31.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.279+0000 7f0a72474b80 10 client.9993 fetch_fsmap learned FSMap version 97 2024-01-27T00:47:31.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.279+0000 7f0a72474b80 1 -- 172.21.15.46:0/1848874235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) v3 -- 0x5614f2304c80 con 0x5614f251f010 2024-01-27T00:47:31.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.279+0000 7f0a417fa700 1 -- 172.21.15.46:0/1848874235 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6405ec40 con 0x5614f251f010 2024-01-27T00:47:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.279+0000 7f0a417fa700 1 -- 172.21.15.46:0/1848874235 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 97) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f0a54026c30 con 0x5614f251f010 2024-01-27T00:47:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.279+0000 7f0a72474b80 10 client.9993 fetch_fsmap finished waiting for FSMap version 97 2024-01-27T00:47:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.279+0000 7f0a72474b80 10 client.9993 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-27T00:47:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.279+0000 7f0a72474b80 20 client.9993 populate_metadata read hostname 'smithi046' 2024-01-27T00:47:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.279+0000 7f0a72474b80 1 --2- 172.21.15.46:0/1848874235 >> [v2:172.21.15.46:6833/2544525048,v1:172.21.15.46:6835/2544525048] conn(0x5614f25c8620 0x5614f25c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.279+0000 7f0a72474b80 4 client.9993 mds_command: new command op to 9828 tid=0[{"prefix": "status"}] 2024-01-27T00:47:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.279+0000 7f0a72474b80 1 -- 172.21.15.46:0/1848874235 --> [v2:172.21.15.46:6833/2544525048,v1:172.21.15.46:6835/2544525048] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f0a5c015420 con 0x5614f25c8620 2024-01-27T00:47:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.279+0000 7f0a62d9d700 1 --2- 172.21.15.46:0/1848874235 >> [v2:172.21.15.46:6833/2544525048,v1:172.21.15.46:6835/2544525048] conn(0x5614f25c8620 0x5614f25c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.279+0000 7f0a417fa700 10 client.9993 ms_handle_connect on v2:172.21.15.46:6833/2544525048 2024-01-27T00:47:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.279+0000 7f0a62d9d700 1 --2- 172.21.15.46:0/1848874235 >> [v2:172.21.15.46:6833/2544525048,v1:172.21.15.46:6835/2544525048] conn(0x5614f25c8620 0x5614f25c89f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b81bc6756ebd779 server_cookie=ad4b28668d68e013 in_seq=0 out_seq=0 2024-01-27T00:47:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.280+0000 7f0a417fa700 1 -- 172.21.15.46:0/1848874235 <== mds.0 v2:172.21.15.46:6833/2544525048 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x7f0a5c015420 con 0x5614f25c8620 2024-01-27T00:47:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.280+0000 7f0a417fa700 10 client.9993 handle_command_reply: tid=0 2024-01-27T00:47:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.280+0000 7f0a72474b80 1 client.9993 shutdown 2024-01-27T00:47:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.280+0000 7f0a72474b80 1 -- 172.21.15.46:0/1848874235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251f010 msgr2=0x5614f25486b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.280+0000 7f0a72474b80 1 --2- 172.21.15.46:0/1848874235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251f010 0x5614f25486b0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400b3c0 tx=0x7f0a5400b050 comp rx=0 tx=0).stop 2024-01-27T00:47:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.280+0000 7f0a72474b80 1 -- 172.21.15.46:0/1848874235 shutdown_connections 2024-01-27T00:47:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.280+0000 7f0a72474b80 1 --2- 172.21.15.46:0/1848874235 >> [v2:172.21.15.46:6833/2544525048,v1:172.21.15.46:6835/2544525048] conn(0x5614f25c8620 0x5614f25c89f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.280+0000 7f0a72474b80 1 --2- 172.21.15.46:0/1848874235 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5614f2567360 0x5614f2548bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.280+0000 7f0a72474b80 1 --2- 172.21.15.46:0/1848874235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5614f251e5d0 0x5614f251a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.280+0000 7f0a72474b80 1 --2- 172.21.15.46:0/1848874235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614f251f010 0x5614f25486b0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.280+0000 7f0a417fa700 0 client.9993 ms_handle_reset on v2:172.21.15.46:6833/2544525048 2024-01-27T00:47:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.280+0000 7f0a72474b80 1 -- 172.21.15.46:0/1848874235 >> 172.21.15.46:0/1848874235 conn(0x5614f251ea70 msgr2=0x5614f2519070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.280+0000 7f0a72474b80 1 -- 172.21.15.46:0/1848874235 shutdown_connections 2024-01-27T00:47:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.280+0000 7f0a72474b80 1 -- 172.21.15.46:0/1848874235 wait complete. 2024-01-27T00:47:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.280+0000 7f0a72474b80 20 client.9993 trim_cache size 0 max 16384 2024-01-27T00:47:31.287 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T00:47:31.287 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-27T00:47:31.287 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-27T00:47:31.287 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 9828, 2024-01-27T00:47:31.288 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-27T00:47:31.288 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-27T00:47:31.288 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-01-27T00:47:31.288 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 10.776937088, 2024-01-27T00:47:31.288 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 97, 2024-01-27T00:47:31.288 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 188, 2024-01-27T00:47:31.288 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 188, 2024-01-27T00:47:31.288 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 16.569046088 2024-01-27T00:47:31.288 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T00:47:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.281+0000 7f0a69171700 1 -- 172.21.15.46:0/1849011790 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f0a50066f30 msgr2=0x7f0a500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.281+0000 7f0a69171700 1 --2- 172.21.15.46:0/1849011790 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f0a50066f30 0x7f0a500693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0a64063ca0 tx=0x7f0a54017000 comp rx=0 tx=0).stop 2024-01-27T00:47:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.281+0000 7f0a69171700 1 -- 172.21.15.46:0/1849011790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a640b2160 msgr2=0x7f0a641425b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.281+0000 7f0a69171700 1 --2- 172.21.15.46:0/1849011790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a640b2160 0x7f0a641425b0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f0a5800e8d0 tx=0x7f0a58008040 comp rx=0 tx=0).stop 2024-01-27T00:47:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.281+0000 7f0a69171700 1 -- 172.21.15.46:0/1849011790 shutdown_connections 2024-01-27T00:47:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.281+0000 7f0a69171700 1 --2- 172.21.15.46:0/1849011790 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f0a50066f30 0x7f0a500693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.281+0000 7f0a69171700 1 --2- 172.21.15.46:0/1849011790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a640b2160 0x7f0a641425b0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.281+0000 7f0a69171700 1 --2- 172.21.15.46:0/1849011790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a6415da10 0x7f0a640ad9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.281+0000 7f0a69171700 1 --2- 172.21.15.46:0/1849011790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a640b2530 0x7f0a640ad470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.281+0000 7f0a69171700 1 -- 172.21.15.46:0/1849011790 >> 172.21.15.46:0/1849011790 conn(0x7f0a640ccbc0 msgr2=0x7f0a640b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.281+0000 7f0a69171700 1 -- 172.21.15.46:0/1849011790 shutdown_connections 2024-01-27T00:47:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.281+0000 7f0a69171700 1 -- 172.21.15.46:0/1849011790 wait complete. 2024-01-27T00:47:31.294 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-27T00:47:31.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.426+0000 7ff69d8de700 1 Processor -- start 2024-01-27T00:47:31.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.426+0000 7ff69d8de700 1 -- start start 2024-01-27T00:47:31.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.427+0000 7ff69d8de700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff698159920 0x7ff6980caa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.427+0000 7ff69d8de700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff69806abf0 0x7ff6980caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.427+0000 7ff69d8de700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff69806b1d0 con 0x7ff698159cf0 2024-01-27T00:47:31.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.427+0000 7ff69d8de700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff69806b4b0 con 0x7ff69806abf0 2024-01-27T00:47:31.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.427+0000 7ff69d8de700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69806b790 con 0x7ff698159920 2024-01-27T00:47:31.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.427+0000 7ff6977fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff69806abf0 0x7ff6980caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.427+0000 7ff6967fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff698159cf0 0x7ff69806a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57386/0 (socket says 172.21.15.46:57386) 2024-01-27T00:47:31.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.427+0000 7ff6967fc700 1 -- 172.21.15.46:0/3711163790 learned_addr learned my addr 172.21.15.46:0/3711163790 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:31.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.427+0000 7ff696ffd700 1 --2- 172.21.15.46:0/3711163790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff698159920 0x7ff6980caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.427+0000 7ff6977fe700 1 -- 172.21.15.46:0/3711163790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff698159920 msgr2=0x7ff6980caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:31.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.427+0000 7ff6977fe700 1 --2- 172.21.15.46:0/3711163790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff698159920 0x7ff6980caa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.427+0000 7ff6977fe700 1 -- 172.21.15.46:0/3711163790 >> v1:172.21.15.46:6789/0 conn(0x7ff698159cf0 legacy=0x7ff69806a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:31.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.427+0000 7ff695ffb700 1 -- 172.21.15.46:0/3711163790 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772475477 0 0) 0x7ff69806b1d0 con 0x7ff698159cf0 2024-01-27T00:47:31.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.427+0000 7ff6977fe700 1 -- 172.21.15.46:0/3711163790 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69806ba70 con 0x7ff69806abf0 2024-01-27T00:47:31.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.427+0000 7ff6977fe700 1 --2- 172.21.15.46:0/3711163790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69806abf0 0x7ff6980caf40 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7ff690009e60 tx=0x7ff690009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41c76fec91eda9ef server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:31.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.428+0000 7ff695ffb700 1 -- 172.21.15.46:0/3711163790 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69000e070 con 0x7ff69806abf0 2024-01-27T00:47:31.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.428+0000 7ff695ffb700 1 -- 172.21.15.46:0/3711163790 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff69000bc30 con 0x7ff69806abf0 2024-01-27T00:47:31.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.428+0000 7ff695ffb700 1 -- 172.21.15.46:0/3711163790 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff690012430 con 0x7ff69806abf0 2024-01-27T00:47:31.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.428+0000 7ff69d8de700 1 -- 172.21.15.46:0/3711163790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69806abf0 msgr2=0x7ff6980caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:31.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.428+0000 7ff69d8de700 1 --2- 172.21.15.46:0/3711163790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69806abf0 0x7ff6980caf40 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7ff690009e60 tx=0x7ff690009780 comp rx=0 tx=0).stop 2024-01-27T00:47:31.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.428+0000 7ff69d8de700 1 -- 172.21.15.46:0/3711163790 shutdown_connections 2024-01-27T00:47:31.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.428+0000 7ff69d8de700 1 --2- 172.21.15.46:0/3711163790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69806abf0 0x7ff6980caf40 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.428+0000 7ff69d8de700 1 --2- 172.21.15.46:0/3711163790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff698159920 0x7ff6980caa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.428+0000 7ff69d8de700 1 -- 172.21.15.46:0/3711163790 >> 172.21.15.46:0/3711163790 conn(0x7ff6980cd6f0 msgr2=0x7ff6980c9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:31.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.428+0000 7ff69d8de700 1 -- 172.21.15.46:0/3711163790 shutdown_connections 2024-01-27T00:47:31.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.428+0000 7ff69d8de700 1 -- 172.21.15.46:0/3711163790 wait complete. 2024-01-27T00:47:31.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.430+0000 7ff69d8de700 1 Processor -- start 2024-01-27T00:47:31.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.430+0000 7ff69d8de700 1 -- start start 2024-01-27T00:47:31.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.430+0000 7ff69d8de700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69806abf0 0x7ff698158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.430+0000 7ff69d8de700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff698159920 0x7ff6980b2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.430+0000 7ff69d8de700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff698159cf0 0x7ff6980ad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.430+0000 7ff69d8de700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69816a440 con 0x7ff69806abf0 2024-01-27T00:47:31.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.430+0000 7ff69d8de700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6980b0110 con 0x7ff698159cf0 2024-01-27T00:47:31.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.430+0000 7ff69d8de700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6980aca10 con 0x7ff698159920 2024-01-27T00:47:31.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.431+0000 7ff696ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69806abf0 0x7ff698158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.431+0000 7ff696ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69806abf0 0x7ff698158320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47830/0 (socket says 172.21.15.46:47830) 2024-01-27T00:47:31.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.431+0000 7ff696ffd700 1 -- 172.21.15.46:0/3614249606 learned_addr learned my addr 172.21.15.46:0/3614249606 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:31.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.431+0000 7ff6967fc700 1 --2- 172.21.15.46:0/3614249606 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff698159920 0x7ff6980b2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.431+0000 7ff6977fe700 1 --2- 172.21.15.46:0/3614249606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff698159cf0 0x7ff6980ad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.431+0000 7ff696ffd700 1 -- 172.21.15.46:0/3614249606 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff698159920 msgr2=0x7ff6980b2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:31.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.431+0000 7ff696ffd700 1 --2- 172.21.15.46:0/3614249606 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff698159920 0x7ff6980b2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.431+0000 7ff696ffd700 1 -- 172.21.15.46:0/3614249606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff698159cf0 msgr2=0x7ff6980ad9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:31.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.431+0000 7ff696ffd700 1 --2- 172.21.15.46:0/3614249606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff698159cf0 0x7ff6980ad9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.431+0000 7ff696ffd700 1 -- 172.21.15.46:0/3614249606 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff690009a20 con 0x7ff69806abf0 2024-01-27T00:47:31.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.431+0000 7ff6977fe700 1 --2- 172.21.15.46:0/3614249606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff698159cf0 0x7ff6980ad9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:47:31.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.431+0000 7ff696ffd700 1 --2- 172.21.15.46:0/3614249606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69806abf0 0x7ff698158320 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7ff68800d890 tx=0x7ff688008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:31.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.431+0000 7ff67effd700 1 -- 172.21.15.46:0/3614249606 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68800b600 con 0x7ff69806abf0 2024-01-27T00:47:31.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.431+0000 7ff67effd700 1 -- 172.21.15.46:0/3614249606 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68800b760 con 0x7ff69806abf0 2024-01-27T00:47:31.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.431+0000 7ff69d8de700 1 -- 172.21.15.46:0/3614249606 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6980accf0 con 0x7ff69806abf0 2024-01-27T00:47:31.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.431+0000 7ff67effd700 1 -- 172.21.15.46:0/3614249606 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68800bae0 con 0x7ff69806abf0 2024-01-27T00:47:31.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.431+0000 7ff69d8de700 1 -- 172.21.15.46:0/3614249606 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6980afda0 con 0x7ff69806abf0 2024-01-27T00:47:31.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.432+0000 7ff69d8de700 1 -- 172.21.15.46:0/3614249606 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69804c710 con 0x7ff69806abf0 2024-01-27T00:47:31.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.433+0000 7ff67effd700 1 -- 172.21.15.46:0/3614249606 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff68801e460 con 0x7ff69806abf0 2024-01-27T00:47:31.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.433+0000 7ff67effd700 1 --2- 172.21.15.46:0/3614249606 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff684066f30 0x7ff6840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:31.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.433+0000 7ff67effd700 1 -- 172.21.15.46:0/3614249606 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7ff688084eb0 con 0x7ff69806abf0 2024-01-27T00:47:31.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.433+0000 7ff6967fc700 1 --2- 172.21.15.46:0/3614249606 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff684066f30 0x7ff6840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:31.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.434+0000 7ff6967fc700 1 --2- 172.21.15.46:0/3614249606 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff684066f30 0x7ff6840693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff6980cbe20 tx=0x7ff68c009040 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:31.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.438+0000 7ff67effd700 1 -- 172.21.15.46:0/3614249606 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff688051e30 con 0x7ff69806abf0 2024-01-27T00:47:31.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.691+0000 7ff69d8de700 1 -- 172.21.15.46:0/3614249606 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff6980a9c20 con 0x7ff69806abf0 2024-01-27T00:47:31.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.691+0000 7ff67effd700 1 -- 172.21.15.46:0/3614249606 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v97) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff688055450 con 0x7ff69806abf0 2024-01-27T00:47:31.692 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-27T00:47:31.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.694+0000 7ff69d8de700 1 -- 172.21.15.46:0/3614249606 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff684066f30 msgr2=0x7ff6840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:31.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.694+0000 7ff69d8de700 1 --2- 172.21.15.46:0/3614249606 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff684066f30 0x7ff6840693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff6980cbe20 tx=0x7ff68c009040 comp rx=0 tx=0).stop 2024-01-27T00:47:31.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.694+0000 7ff69d8de700 1 -- 172.21.15.46:0/3614249606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69806abf0 msgr2=0x7ff698158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:31.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.694+0000 7ff69d8de700 1 --2- 172.21.15.46:0/3614249606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69806abf0 0x7ff698158320 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7ff68800d890 tx=0x7ff688008040 comp rx=0 tx=0).stop 2024-01-27T00:47:31.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.694+0000 7ff69d8de700 1 -- 172.21.15.46:0/3614249606 shutdown_connections 2024-01-27T00:47:31.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.694+0000 7ff69d8de700 1 --2- 172.21.15.46:0/3614249606 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff684066f30 0x7ff6840693e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.694+0000 7ff69d8de700 1 --2- 172.21.15.46:0/3614249606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69806abf0 0x7ff698158320 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.694+0000 7ff69d8de700 1 --2- 172.21.15.46:0/3614249606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff698159cf0 0x7ff6980ad9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.694+0000 7ff69d8de700 1 --2- 172.21.15.46:0/3614249606 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff698159920 0x7ff6980b2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:31.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.694+0000 7ff69d8de700 1 -- 172.21.15.46:0/3614249606 >> 172.21.15.46:0/3614249606 conn(0x7ff6980cd6f0 msgr2=0x7ff6980c1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:31.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.694+0000 7ff69d8de700 1 -- 172.21.15.46:0/3614249606 shutdown_connections 2024-01-27T00:47:31.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:31.694+0000 7ff69d8de700 1 -- 172.21.15.46:0/3614249606 wait complete. 2024-01-27T00:47:31.707 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-27T00:47:31.708 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-27T00:47:31.708 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-27T00:47:31.708 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-27T00:47:31.708 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-27T00:47:31.708 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-27T00:47:31.708 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-27T00:47:31.708 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-27T00:47:31.708 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-27T00:47:31.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:47:31.708 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-27T00:47:31.728 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-27T00:47:31.728 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-27T00:47:31.728 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-27T00:47:31.728 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T00:47:31.728 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-27T00:47:31.728 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T00:47:31.728 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T00:47:31.728 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-27T00:47:31.728 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-27T00:47:31.728 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T00:47:31.728 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-27T00:47:31.729 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-27T00:47:31.729 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-27T00:47:31.729 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-27T00:47:31.729 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-27T00:47:31.729 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-27T00:47:31.729 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 37679sec preferred_lft 37679sec 2024-01-27T00:47:31.729 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-27T00:47:31.729 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T00:47:31.729 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T00:47:31.729 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-27T00:47:31.729 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-27T00:47:31.729 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-27T00:47:31.729 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff 2024-01-27T00:47:31.729 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-27T00:47:31.730 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T00:47:31.730 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::cce6:a7ff:fe4f:38d/64 scope link 2024-01-27T00:47:31.730 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T00:47:31.730 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-27T00:47:31.730 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 76:ef:90:4c:9d:6f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-27T00:47:31.730 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::74ef:90ff:fe4c:9d6f/64 scope link 2024-01-27T00:47:31.730 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T00:47:31.730 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-27T00:47:31.730 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-27T00:47:31.730 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::2c73:afff:fef0:6f39/64 scope link 2024-01-27T00:47:31.730 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T00:47:31.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:47:31.731 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-27T00:47:31.798 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-27T00:47:31.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:47:31.800 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-27T00:47:31.863 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-27T00:47:31.863 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-27T00:47:31.864 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-27T00:47:31.864 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-27T00:47:31.864 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-27T00:47:31.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:47:31.864 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-27T00:47:31.864 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-27T00:47:31.864 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-27T00:47:31.864 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-27T00:47:31.864 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-27T00:47:31.864 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-27T00:47:31.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:47:31.988 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-27T00:47:32.009 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-27T00:47:32.009 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-27T00:47:32.009 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-27T00:47:32.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T00:47:32.010 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-01-27T00:47:32.084 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-27T00:47:32.084 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-27T00:47:32.084 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-27T00:47:32.146 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-27T00:47:32.146 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-27T00:47:32.147 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-27T00:47:32.207 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-27T00:47:32.283 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-27T00:47:32.283 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-27T00:47:32.357 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-27T00:47:32.358 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-27T00:47:32.359 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:47:32.359 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-27T00:47:32.421 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-27T00:47:32.421 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-27T00:47:32.464 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-27T00:47:32.499 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-27T00:47:32.499 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-27T00:47:32.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-01-27T00:47:32.567+0000 7f6dd88b9540 -1 init, newargv = 0x559df824d560 newargc=15 2024-01-27T00:47:32.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[101665]: starting ceph client 2024-01-27T00:47:32.571 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-27T00:47:32.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[101665]: starting fuse 2024-01-27T00:47:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-27T00:47:32.573 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:47:32.573 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-27T00:47:32.634 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-27T00:47:32.634 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-27T00:47:32.634 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-27T00:47:32.634 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-27T00:47:32.634 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-27T00:47:32.634 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-27T00:47:32.634 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-27T00:47:32.635 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T00:47:32.635 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-27T00:47:32.635 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-27T00:47:32.635 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-27T00:47:32.635 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-27T00:47:32.635 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T00:47:32.635 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-27T00:47:32.635 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-27T00:47:32.635 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-27T00:47:32.635 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-27T00:47:32.635 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T00:47:32.635 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-27T00:47:32.635 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-27T00:47:32.635 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-27T00:47:32.635 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-27T00:47:32.636 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-27T00:47:32.636 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-27T00:47:32.636 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-27T00:47:32.636 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-27T00:47:32.636 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T00:47:32.636 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-01-27T00:47:32.636 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-27T00:47:32.773 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.101665.asok 2024-01-27T00:47:32.955 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.101665.asok 2024-01-27T00:47:32.955 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:47:32.955 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.101665.asok status 2024-01-27T00:47:33.129 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T00:47:33.130 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-27T00:47:33.130 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-27T00:47:33.130 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-27T00:47:33.130 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-01-27T00:47:33.130 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-27T00:47:33.130 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-27T00:47:33.130 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "101665", 2024-01-27T00:47:33.130 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-27T00:47:33.130 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:33.130 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-27T00:47:33.130 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-27T00:47:33.130 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 10008, 2024-01-27T00:47:33.131 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-27T00:47:33.131 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-27T00:47:33.131 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-27T00:47:33.131 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 10008 2024-01-27T00:47:33.131 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:33.131 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-27T00:47:33.131 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-27T00:47:33.131 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-27T00:47:33.131 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1328145655 2024-01-27T00:47:33.131 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:33.131 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:33.131 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-27T00:47:33.131 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-27T00:47:33.131 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-27T00:47:33.131 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1328145655 2024-01-27T00:47:33.132 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T00:47:33.132 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.10008 192.168.0.1:0/1328145655", 2024-01-27T00:47:33.132 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/1328145655", 2024-01-27T00:47:33.132 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-27T00:47:33.132 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 97, 2024-01-27T00:47:33.132 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 188, 2024-01-27T00:47:33.132 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-27T00:47:33.132 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-27T00:47:33.132 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-27T00:47:33.132 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T00:47:33.141 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:47:33.141 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-27T00:47:33.142 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-27T00:47:33.161 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-27T00:47:33.161 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-27T00:47:33.161 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:47:33.161 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-27T00:47:33.234 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-27T00:47:33.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.416+0000 7fcfaeb6b700 1 Processor -- start 2024-01-27T00:47:33.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.416+0000 7fcfaeb6b700 1 -- start start 2024-01-27T00:47:33.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.417+0000 7fcfaeb6b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fcfa80c8dc0 0x7fcfa80c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:33.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.417+0000 7fcfaeb6b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcfa8068620 0x7fcfa80cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:33.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.417+0000 7fcfaeb6b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfa80c2240 0x7fcfa80ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:33.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.417+0000 7fcfaeb6b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa8067930 con 0x7fcfa8068620 2024-01-27T00:47:33.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.417+0000 7fcfaeb6b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa8068190 con 0x7fcfa80c8dc0 2024-01-27T00:47:33.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.417+0000 7fcfaeb6b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa80c40c0 con 0x7fcfa80c2240 2024-01-27T00:47:33.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.417+0000 7fcfa7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcfa8068620 0x7fcfa80cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:33.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.417+0000 7fcfa7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcfa8068620 0x7fcfa80cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47834/0 (socket says 172.21.15.46:47834) 2024-01-27T00:47:33.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.417+0000 7fcfa7fff700 1 -- 172.21.15.46:0/908503692 learned_addr learned my addr 172.21.15.46:0/908503692 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:33.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.417+0000 7fcfad108700 1 --2- 172.21.15.46:0/908503692 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfa80c2240 0x7fcfa80ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:33.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.417+0000 7fcfac907700 1 --2- 172.21.15.46:0/908503692 >> v2:172.21.15.106:3300/0 conn(0x7fcfa80c8dc0 0x7fcfa80c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:33.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.417+0000 7fcfa7fff700 1 -- 172.21.15.46:0/908503692 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfa80c2240 msgr2=0x7fcfa80ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:33.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.417+0000 7fcfa7fff700 1 --2- 172.21.15.46:0/908503692 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfa80c2240 0x7fcfa80ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:33.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.417+0000 7fcfa7fff700 1 -- 172.21.15.46:0/908503692 >> v2:172.21.15.106:3300/0 conn(0x7fcfa80c8dc0 msgr2=0x7fcfa80c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:33.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.417+0000 7fcfa7fff700 1 --2- 172.21.15.46:0/908503692 >> v2:172.21.15.106:3300/0 conn(0x7fcfa80c8dc0 0x7fcfa80c3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:33.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.417+0000 7fcfa7fff700 1 -- 172.21.15.46:0/908503692 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa8067b40 con 0x7fcfa8068620 2024-01-27T00:47:33.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.417+0000 7fcfa7fff700 1 --2- 172.21.15.46:0/908503692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfa8068620 0x7fcfa80cc5c0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fcfa806db80 tx=0x7fcf9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa4b1f08cad48f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:33.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.418+0000 7fcfa77fe700 1 -- 172.21.15.46:0/908503692 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c009e70 con 0x7fcfa8068620 2024-01-27T00:47:33.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.418+0000 7fcfa77fe700 1 -- 172.21.15.46:0/908503692 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9c00b860 con 0x7fcfa8068620 2024-01-27T00:47:33.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.418+0000 7fcfa77fe700 1 -- 172.21.15.46:0/908503692 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c00bc00 con 0x7fcfa8068620 2024-01-27T00:47:33.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.418+0000 7fcfaeb6b700 1 -- 172.21.15.46:0/908503692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfa8068620 msgr2=0x7fcfa80cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:33.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.418+0000 7fcfaeb6b700 1 --2- 172.21.15.46:0/908503692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfa8068620 0x7fcfa80cc5c0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fcfa806db80 tx=0x7fcf9c009250 comp rx=0 tx=0).stop 2024-01-27T00:47:33.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.418+0000 7fcfaeb6b700 1 -- 172.21.15.46:0/908503692 shutdown_connections 2024-01-27T00:47:33.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.418+0000 7fcfaeb6b700 1 --2- 172.21.15.46:0/908503692 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfa80c2240 0x7fcfa80ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:33.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.418+0000 7fcfaeb6b700 1 --2- 172.21.15.46:0/908503692 >> v2:172.21.15.106:3300/0 conn(0x7fcfa80c8dc0 0x7fcfa80c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:33.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.418+0000 7fcfaeb6b700 1 --2- 172.21.15.46:0/908503692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfa8068620 0x7fcfa80cc5c0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:33.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.418+0000 7fcfaeb6b700 1 -- 172.21.15.46:0/908503692 >> 172.21.15.46:0/908503692 conn(0x7fcfa8069220 msgr2=0x7fcfa80cd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:33.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.418+0000 7fcfaeb6b700 1 -- 172.21.15.46:0/908503692 shutdown_connections 2024-01-27T00:47:33.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.418+0000 7fcfaeb6b700 1 -- 172.21.15.46:0/908503692 wait complete. 2024-01-27T00:47:33.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.420+0000 7fcfaeb6b700 1 Processor -- start 2024-01-27T00:47:33.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.420+0000 7fcfaeb6b700 1 -- start start 2024-01-27T00:47:33.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.421+0000 7fcfaeb6b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfa8068620 0x7fcfa81587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:33.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.421+0000 7fcfaeb6b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfa80c2240 0x7fcfa809a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:33.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.421+0000 7fcfaeb6b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfa80c8dc0 0x7fcfa80986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:33.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.421+0000 7fcfaeb6b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa80c6940 con 0x7fcfa8068620 2024-01-27T00:47:33.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.421+0000 7fcfaeb6b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa8158d30 con 0x7fcfa80c8dc0 2024-01-27T00:47:33.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.421+0000 7fcfaeb6b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa809b1b0 con 0x7fcfa80c2240 2024-01-27T00:47:33.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.421+0000 7fcfac907700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfa8068620 0x7fcfa81587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:33.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.421+0000 7fcfac907700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfa8068620 0x7fcfa81587f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47850/0 (socket says 172.21.15.46:47850) 2024-01-27T00:47:33.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.421+0000 7fcfac907700 1 -- 172.21.15.46:0/3225005219 learned_addr learned my addr 172.21.15.46:0/3225005219 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:33.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.421+0000 7fcfad108700 1 --2- 172.21.15.46:0/3225005219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfa80c8dc0 0x7fcfa80986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:33.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.421+0000 7fcfa7fff700 1 --2- 172.21.15.46:0/3225005219 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfa80c2240 0x7fcfa809a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:33.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.422+0000 7fcfac907700 1 -- 172.21.15.46:0/3225005219 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfa80c2240 msgr2=0x7fcfa809a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:33.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.422+0000 7fcfac907700 1 --2- 172.21.15.46:0/3225005219 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfa80c2240 0x7fcfa809a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:33.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.422+0000 7fcfac907700 1 -- 172.21.15.46:0/3225005219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfa80c8dc0 msgr2=0x7fcfa80986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:33.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.422+0000 7fcfac907700 1 --2- 172.21.15.46:0/3225005219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfa80c8dc0 0x7fcfa80986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:33.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.422+0000 7fcfac907700 1 -- 172.21.15.46:0/3225005219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c0094f0 con 0x7fcfa8068620 2024-01-27T00:47:33.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.422+0000 7fcfac907700 1 --2- 172.21.15.46:0/3225005219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfa8068620 0x7fcfa81587f0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800ea30 tx=0x7fcf98009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:33.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.422+0000 7fcfa4ff9700 1 -- 172.21.15.46:0/3225005219 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9800ee70 con 0x7fcfa8068620 2024-01-27T00:47:33.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.422+0000 7fcfa4ff9700 1 -- 172.21.15.46:0/3225005219 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9800c850 con 0x7fcfa8068620 2024-01-27T00:47:33.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.422+0000 7fcfaeb6b700 1 -- 172.21.15.46:0/3225005219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa809aed0 con 0x7fcfa8068620 2024-01-27T00:47:33.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.422+0000 7fcfa4ff9700 1 -- 172.21.15.46:0/3225005219 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9800cbd0 con 0x7fcfa8068620 2024-01-27T00:47:33.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.422+0000 7fcfaeb6b700 1 -- 172.21.15.46:0/3225005219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa80c65d0 con 0x7fcfa8068620 2024-01-27T00:47:33.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.423+0000 7fcfaeb6b700 1 -- 172.21.15.46:0/3225005219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa804c710 con 0x7fcfa8068620 2024-01-27T00:47:33.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.423+0000 7fcfa4ff9700 1 -- 172.21.15.46:0/3225005219 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fcf98013470 con 0x7fcfa8068620 2024-01-27T00:47:33.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.424+0000 7fcfa4ff9700 1 --2- 172.21.15.46:0/3225005219 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fcf94066f30 0x7fcf940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:33.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.424+0000 7fcfa4ff9700 1 -- 172.21.15.46:0/3225005219 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fcf98084d60 con 0x7fcfa8068620 2024-01-27T00:47:33.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.424+0000 7fcfa7fff700 1 --2- 172.21.15.46:0/3225005219 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fcf94066f30 0x7fcf940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:33.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.425+0000 7fcfa7fff700 1 --2- 172.21.15.46:0/3225005219 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fcf94066f30 0x7fcf940693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcfa806db80 tx=0x7fcf9c017000 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:33.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.428+0000 7fcfa4ff9700 1 -- 172.21.15.46:0/3225005219 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcf98051ce0 con 0x7fcfa8068620 2024-01-27T00:47:33.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.662+0000 7fcfaeb6b700 1 -- 172.21.15.46:0/3225005219 --> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfa80994c0 con 0x7fcf94066f30 2024-01-27T00:47:33.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.664+0000 7fcfa4ff9700 1 -- 172.21.15.46:0/3225005219 <== mgr.9313 v2:172.21.15.106:6836/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fcfa80994c0 con 0x7fcf94066f30 2024-01-27T00:47:33.664 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-27T00:47:33.664 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T00:47:33.664 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephfs" 2024-01-27T00:47:33.664 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T00:47:33.664 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-27T00:47:33.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.666+0000 7fcfaeb6b700 1 -- 172.21.15.46:0/3225005219 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fcf94066f30 msgr2=0x7fcf940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:33.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.666+0000 7fcfaeb6b700 1 --2- 172.21.15.46:0/3225005219 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fcf94066f30 0x7fcf940693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcfa806db80 tx=0x7fcf9c017000 comp rx=0 tx=0).stop 2024-01-27T00:47:33.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.666+0000 7fcfaeb6b700 1 -- 172.21.15.46:0/3225005219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfa8068620 msgr2=0x7fcfa81587f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:33.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.666+0000 7fcfaeb6b700 1 --2- 172.21.15.46:0/3225005219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfa8068620 0x7fcfa81587f0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800ea30 tx=0x7fcf98009120 comp rx=0 tx=0).stop 2024-01-27T00:47:33.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.666+0000 7fcfaeb6b700 1 -- 172.21.15.46:0/3225005219 shutdown_connections 2024-01-27T00:47:33.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.666+0000 7fcfaeb6b700 1 --2- 172.21.15.46:0/3225005219 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fcf94066f30 0x7fcf940693e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:33.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.666+0000 7fcfaeb6b700 1 --2- 172.21.15.46:0/3225005219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfa8068620 0x7fcfa81587f0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:33.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.666+0000 7fcfaeb6b700 1 --2- 172.21.15.46:0/3225005219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfa80c8dc0 0x7fcfa80986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:33.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.666+0000 7fcfaeb6b700 1 --2- 172.21.15.46:0/3225005219 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfa80c2240 0x7fcfa809a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:33.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.666+0000 7fcfaeb6b700 1 -- 172.21.15.46:0/3225005219 >> 172.21.15.46:0/3225005219 conn(0x7fcfa8069220 msgr2=0x7fcfa806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:33.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.666+0000 7fcfaeb6b700 1 -- 172.21.15.46:0/3225005219 shutdown_connections 2024-01-27T00:47:33.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.666+0000 7fcfaeb6b700 1 -- 172.21.15.46:0/3225005219 wait complete. 2024-01-27T00:47:33.680 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-27T00:47:33.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.813+0000 7f6510270700 1 Processor -- start 2024-01-27T00:47:33.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.813+0000 7f6510270700 1 -- start start 2024-01-27T00:47:33.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.814+0000 7f6510270700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65080cb640 0x7f65080c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:33.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.814+0000 7f6510270700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65080ccc00 con 0x7f6508141200 2024-01-27T00:47:33.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.814+0000 7f6510270700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65080cc310 con 0x7f65080c8da0 2024-01-27T00:47:33.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.814+0000 7f6510270700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65080cbbf0 con 0x7f65080cb640 2024-01-27T00:47:33.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.814+0000 7f650d80b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6508141200 0x7f65080caf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57398/0 (socket says 172.21.15.46:57398) 2024-01-27T00:47:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.814+0000 7f650d80b700 1 -- 172.21.15.46:0/40731299 learned_addr learned my addr 172.21.15.46:0/40731299 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.814+0000 7f650e80d700 1 --2- 172.21.15.46:0/40731299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65080cb640 0x7f65080c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.814+0000 7f650d00a700 1 -- 172.21.15.46:0/40731299 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211966578 0 0) 0x7f65080ccc00 con 0x7f6508141200 2024-01-27T00:47:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.814+0000 7f650d00a700 1 -- 172.21.15.46:0/40731299 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64f4003570 con 0x7f6508141200 2024-01-27T00:47:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.814+0000 7f650d00a700 1 -- 172.21.15.46:0/40731299 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603554228 0 0) 0x7f65080cc310 con 0x7f65080c8da0 2024-01-27T00:47:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.814+0000 7f650d00a700 1 -- 172.21.15.46:0/40731299 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65080ccc00 con 0x7f65080c8da0 2024-01-27T00:47:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.814+0000 7f650e80d700 1 -- 172.21.15.46:0/40731299 >> v1:172.21.15.106:6789/0 conn(0x7f65080c8da0 legacy=0x7f6508140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.814+0000 7f650e80d700 1 -- 172.21.15.46:0/40731299 >> v1:172.21.15.46:6789/0 conn(0x7f6508141200 legacy=0x7f65080caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.814+0000 7f650e80d700 1 -- 172.21.15.46:0/40731299 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65080cc5b0 con 0x7f65080cb640 2024-01-27T00:47:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.815+0000 7f650e80d700 1 --2- 172.21.15.46:0/40731299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65080cb640 0x7f65080c9170 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6500009e90 tx=0x7f6500009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ed937431212ebe7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.815+0000 7f650d00a700 1 -- 172.21.15.46:0/40731299 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650000e070 con 0x7f65080cb640 2024-01-27T00:47:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.815+0000 7f650d00a700 1 -- 172.21.15.46:0/40731299 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650000bb80 con 0x7f65080cb640 2024-01-27T00:47:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.815+0000 7f650d00a700 1 -- 172.21.15.46:0/40731299 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6500012430 con 0x7f65080cb640 2024-01-27T00:47:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.815+0000 7f6510270700 1 -- 172.21.15.46:0/40731299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65080cb640 msgr2=0x7f65080c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.815+0000 7f6510270700 1 --2- 172.21.15.46:0/40731299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65080cb640 0x7f65080c9170 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6500009e90 tx=0x7f6500009780 comp rx=0 tx=0).stop 2024-01-27T00:47:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.816+0000 7f6510270700 1 -- 172.21.15.46:0/40731299 shutdown_connections 2024-01-27T00:47:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.816+0000 7f6510270700 1 --2- 172.21.15.46:0/40731299 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65080cb640 0x7f65080c9170 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.816+0000 7f6510270700 1 -- 172.21.15.46:0/40731299 >> 172.21.15.46:0/40731299 conn(0x7f650806bd90 msgr2=0x7f650806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.816+0000 7f6510270700 1 -- 172.21.15.46:0/40731299 shutdown_connections 2024-01-27T00:47:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.816+0000 7f6510270700 1 -- 172.21.15.46:0/40731299 wait complete. 2024-01-27T00:47:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.817+0000 7f6510270700 1 Processor -- start 2024-01-27T00:47:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.817+0000 7f6510270700 1 -- start start 2024-01-27T00:47:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f6510270700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65080c8da0 0x7f650813c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f6510270700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65080cb640 0x7f65080b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f6510270700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6508141200 0x7f65080c6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f6510270700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65080b1340 con 0x7f6508141200 2024-01-27T00:47:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f6510270700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65080b1620 con 0x7f65080cb640 2024-01-27T00:47:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f6510270700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65080b1900 con 0x7f65080c8da0 2024-01-27T00:47:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f650e80d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6508141200 0x7f65080c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f650e80d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6508141200 0x7f65080c6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47860/0 (socket says 172.21.15.46:47860) 2024-01-27T00:47:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f650e80d700 1 -- 172.21.15.46:0/3181385456 learned_addr learned my addr 172.21.15.46:0/3181385456 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f650d80b700 1 --2- 172.21.15.46:0/3181385456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65080cb640 0x7f65080b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f650e00c700 1 --2- 172.21.15.46:0/3181385456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65080c8da0 0x7f650813c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f650e80d700 1 -- 172.21.15.46:0/3181385456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65080c8da0 msgr2=0x7f650813c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:33.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f650e80d700 1 --2- 172.21.15.46:0/3181385456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65080c8da0 0x7f650813c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:33.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f650e80d700 1 -- 172.21.15.46:0/3181385456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65080cb640 msgr2=0x7f65080b26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:33.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f650e80d700 1 --2- 172.21.15.46:0/3181385456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65080cb640 0x7f65080b26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:33.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f650e80d700 1 -- 172.21.15.46:0/3181385456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6500009a20 con 0x7f6508141200 2024-01-27T00:47:33.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f650e00c700 1 --2- 172.21.15.46:0/3181385456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65080c8da0 0x7f650813c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:47:33.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f650e80d700 1 --2- 172.21.15.46:0/3181385456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6508141200 0x7f65080c6060 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f650000b610 tx=0x7f6500019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:33.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f64fe7fc700 1 -- 172.21.15.46:0/3181385456 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6500012e40 con 0x7f6508141200 2024-01-27T00:47:33.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f64fe7fc700 1 -- 172.21.15.46:0/3181385456 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650001d630 con 0x7f6508141200 2024-01-27T00:47:33.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.818+0000 7f64fe7fc700 1 -- 172.21.15.46:0/3181385456 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650001d9f0 con 0x7f6508141200 2024-01-27T00:47:33.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.819+0000 7f6510270700 1 -- 172.21.15.46:0/3181385456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65080b1be0 con 0x7f6508141200 2024-01-27T00:47:33.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.819+0000 7f6510270700 1 -- 172.21.15.46:0/3181385456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65080b21a0 con 0x7f6508141200 2024-01-27T00:47:33.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.819+0000 7f6510270700 1 -- 172.21.15.46:0/3181385456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f650804c710 con 0x7f6508141200 2024-01-27T00:47:33.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.820+0000 7f64fe7fc700 1 -- 172.21.15.46:0/3181385456 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 96) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6500024070 con 0x7f6508141200 2024-01-27T00:47:33.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.820+0000 7f64fe7fc700 1 --2- 172.21.15.46:0/3181385456 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f64ec066f30 0x7f64ec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:33.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.820+0000 7f64fe7fc700 1 -- 172.21.15.46:0/3181385456 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f6500087090 con 0x7f6508141200 2024-01-27T00:47:33.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.821+0000 7f650e00c700 1 --2- 172.21.15.46:0/3181385456 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f64ec066f30 0x7f64ec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:33.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.821+0000 7f650e00c700 1 --2- 172.21.15.46:0/3181385456 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f64ec066f30 0x7f64ec0693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f65080c4760 tx=0x7f6504009040 comp rx=0 tx=0).ready entity=mgr.9313 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:33.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:33.825+0000 7f64fe7fc700 1 -- 172.21.15.46:0/3181385456 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6500054010 con 0x7f6508141200 2024-01-27T00:47:34.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.088+0000 7f6510270700 1 -- 172.21.15.46:0/3181385456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f65080cc890 con 0x7f6508141200 2024-01-27T00:47:34.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.693+0000 7f64fe7fc700 1 -- 172.21.15.46:0/3181385456 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 97) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f6500003040 con 0x7f6508141200 2024-01-27T00:47:34.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.695+0000 7f64fe7fc700 1 -- 172.21.15.46:0/3181385456 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v97) v1 ==== 92+0+0 (secure 0 0 0) 0x7f6500057aa0 con 0x7f6508141200 2024-01-27T00:47:34.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.698+0000 7f6510270700 1 -- 172.21.15.46:0/3181385456 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f64ec066f30 msgr2=0x7f64ec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:34.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.698+0000 7f6510270700 1 --2- 172.21.15.46:0/3181385456 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f64ec066f30 0x7f64ec0693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f65080c4760 tx=0x7f6504009040 comp rx=0 tx=0).stop 2024-01-27T00:47:34.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.698+0000 7f6510270700 1 -- 172.21.15.46:0/3181385456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6508141200 msgr2=0x7f65080c6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:34.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.698+0000 7f6510270700 1 --2- 172.21.15.46:0/3181385456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6508141200 0x7f65080c6060 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f650000b610 tx=0x7f6500019000 comp rx=0 tx=0).stop 2024-01-27T00:47:34.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.698+0000 7f6510270700 1 -- 172.21.15.46:0/3181385456 shutdown_connections 2024-01-27T00:47:34.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.698+0000 7f6510270700 1 --2- 172.21.15.46:0/3181385456 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f64ec066f30 0x7f64ec0693e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:34.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.698+0000 7f6510270700 1 --2- 172.21.15.46:0/3181385456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6508141200 0x7f65080c6060 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:34.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.698+0000 7f6510270700 1 --2- 172.21.15.46:0/3181385456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f65080c8da0 0x7f650813c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:34.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.698+0000 7f6510270700 1 --2- 172.21.15.46:0/3181385456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f65080cb640 0x7f65080b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:34.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.698+0000 7f6510270700 1 -- 172.21.15.46:0/3181385456 >> 172.21.15.46:0/3181385456 conn(0x7f650806bd90 msgr2=0x7f650813e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:34.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.700+0000 7f6510270700 1 -- 172.21.15.46:0/3181385456 shutdown_connections 2024-01-27T00:47:34.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.700+0000 7f6510270700 1 -- 172.21.15.46:0/3181385456 wait complete. 2024-01-27T00:47:34.716 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-27T00:47:34.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.856+0000 7f87b572b700 1 Processor -- start 2024-01-27T00:47:34.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.856+0000 7f87b572b700 1 -- start start 2024-01-27T00:47:34.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.857+0000 7f87b572b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f87b00abfc0 0x7f87b006a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:34.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.857+0000 7f87b572b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87b0159990 0x7f87b006aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:34.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.857+0000 7f87b572b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b00aa4e0 con 0x7f87b01595c0 2024-01-27T00:47:34.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.857+0000 7f87b572b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87b00aa1b0 con 0x7f87b00abfc0 2024-01-27T00:47:34.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.857+0000 7f87b572b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b0068570 con 0x7f87b0159990 2024-01-27T00:47:34.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.857+0000 7f87aeffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f87b01595c0 0x7f87b006b180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57404/0 (socket says 172.21.15.46:57404) 2024-01-27T00:47:34.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.857+0000 7f87aeffd700 1 -- 172.21.15.46:0/96852833 learned_addr learned my addr 172.21.15.46:0/96852833 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.857+0000 7f87ae7fc700 1 --2- 172.21.15.46:0/96852833 >> v2:172.21.15.106:3300/0 conn(0x7f87b00abfc0 0x7f87b006a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.857+0000 7f87af7fe700 1 --2- 172.21.15.46:0/96852833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87b0159990 0x7f87b006aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.858+0000 7f87adffb700 1 -- 172.21.15.46:0/96852833 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954614818 0 0) 0x7f87b00aa4e0 con 0x7f87b01595c0 2024-01-27T00:47:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.858+0000 7f87adffb700 1 -- 172.21.15.46:0/96852833 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8798003570 con 0x7f87b01595c0 2024-01-27T00:47:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.858+0000 7f87af7fe700 1 -- 172.21.15.46:0/96852833 >> v2:172.21.15.106:3300/0 conn(0x7f87b00abfc0 msgr2=0x7f87b006a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.858+0000 7f87af7fe700 1 --2- 172.21.15.46:0/96852833 >> v2:172.21.15.106:3300/0 conn(0x7f87b00abfc0 0x7f87b006a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.858+0000 7f87af7fe700 1 -- 172.21.15.46:0/96852833 >> v1:172.21.15.46:6789/0 conn(0x7f87b01595c0 legacy=0x7f87b006b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.858+0000 7f87af7fe700 1 -- 172.21.15.46:0/96852833 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b0068850 con 0x7f87b0159990 2024-01-27T00:47:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.858+0000 7f87ae7fc700 1 --2- 172.21.15.46:0/96852833 >> v2:172.21.15.106:3300/0 conn(0x7f87b00abfc0 0x7f87b006a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.858+0000 7f87af7fe700 1 --2- 172.21.15.46:0/96852833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87b0159990 0x7f87b006aa40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f87a8009e60 tx=0x7f87a8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd5aa5691edb9c6b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.858+0000 7f87adffb700 1 -- 172.21.15.46:0/96852833 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a800e070 con 0x7f87b0159990 2024-01-27T00:47:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.858+0000 7f87adffb700 1 -- 172.21.15.46:0/96852833 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a800bab0 con 0x7f87b0159990 2024-01-27T00:47:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.858+0000 7f87adffb700 1 -- 172.21.15.46:0/96852833 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a800be50 con 0x7f87b0159990 2024-01-27T00:47:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.859+0000 7f87b572b700 1 -- 172.21.15.46:0/96852833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87b0159990 msgr2=0x7f87b006aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:34.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.859+0000 7f87b572b700 1 --2- 172.21.15.46:0/96852833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87b0159990 0x7f87b006aa40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f87a8009e60 tx=0x7f87a8009780 comp rx=0 tx=0).stop 2024-01-27T00:47:34.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.859+0000 7f87b572b700 1 -- 172.21.15.46:0/96852833 shutdown_connections 2024-01-27T00:47:34.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.859+0000 7f87b572b700 1 --2- 172.21.15.46:0/96852833 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87b0159990 0x7f87b006aa40 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:34.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.859+0000 7f87b572b700 1 --2- 172.21.15.46:0/96852833 >> v2:172.21.15.106:3300/0 conn(0x7f87b00abfc0 0x7f87b006a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:34.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.859+0000 7f87b572b700 1 -- 172.21.15.46:0/96852833 >> 172.21.15.46:0/96852833 conn(0x7f87b00ccc20 msgr2=0x7f87b00cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:34.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.859+0000 7f87b572b700 1 -- 172.21.15.46:0/96852833 shutdown_connections 2024-01-27T00:47:34.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.859+0000 7f87b572b700 1 -- 172.21.15.46:0/96852833 wait complete. 2024-01-27T00:47:34.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.860+0000 7f87b572b700 1 Processor -- start 2024-01-27T00:47:34.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.861+0000 7f87b572b700 1 -- start start 2024-01-27T00:47:34.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.861+0000 7f87b572b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87b00abfc0 0x7f87b00c23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:34.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.862+0000 7f87b572b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87b01595c0 0x7f87b016ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:34.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.862+0000 7f87b572b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87b0159990 0x7f87b016bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.862+0000 7f87b572b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b00c2c90 con 0x7f87b01595c0 2024-01-27T00:47:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.862+0000 7f87b572b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b00c2f70 con 0x7f87b00abfc0 2024-01-27T00:47:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.862+0000 7f87b572b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b00c3250 con 0x7f87b0159990 2024-01-27T00:47:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.862+0000 7f87ae7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87b01595c0 0x7f87b016ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.862+0000 7f87ae7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87b01595c0 0x7f87b016ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47882/0 (socket says 172.21.15.46:47882) 2024-01-27T00:47:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.862+0000 7f87ae7fc700 1 -- 172.21.15.46:0/1026598208 learned_addr learned my addr 172.21.15.46:0/1026598208 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.862+0000 7f87aeffd700 1 --2- 172.21.15.46:0/1026598208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87b00abfc0 0x7f87b00c23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.862+0000 7f87af7fe700 1 --2- 172.21.15.46:0/1026598208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87b0159990 0x7f87b016bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.862+0000 7f87aeffd700 1 -- 172.21.15.46:0/1026598208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87b0159990 msgr2=0x7f87b016bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.862+0000 7f87aeffd700 1 --2- 172.21.15.46:0/1026598208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87b0159990 0x7f87b016bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.862+0000 7f87aeffd700 1 -- 172.21.15.46:0/1026598208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87b01595c0 msgr2=0x7f87b016ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.862+0000 7f87aeffd700 1 --2- 172.21.15.46:0/1026598208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87b01595c0 0x7f87b016ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.862+0000 7f87aeffd700 1 -- 172.21.15.46:0/1026598208 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a4009420 con 0x7f87b00abfc0 2024-01-27T00:47:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.862+0000 7f87ae7fc700 1 --2- 172.21.15.46:0/1026598208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87b01595c0 0x7f87b016ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.862+0000 7f87aeffd700 1 --2- 172.21.15.46:0/1026598208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87b00abfc0 0x7f87b00c23d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f87a000c580 tx=0x7f87a0007d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:34.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.863+0000 7f8796ffd700 1 -- 172.21.15.46:0/1026598208 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a0005b30 con 0x7f87b00abfc0 2024-01-27T00:47:34.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.863+0000 7f8796ffd700 1 -- 172.21.15.46:0/1026598208 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a0005cb0 con 0x7f87b00abfc0 2024-01-27T00:47:34.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.863+0000 7f8796ffd700 1 -- 172.21.15.46:0/1026598208 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a0007320 con 0x7f87b00abfc0 2024-01-27T00:47:34.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.863+0000 7f87b572b700 1 -- 172.21.15.46:0/1026598208 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a8009a20 con 0x7f87b00abfc0 2024-01-27T00:47:34.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.863+0000 7f87b572b700 1 -- 172.21.15.46:0/1026598208 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b00c35e0 con 0x7f87b00abfc0 2024-01-27T00:47:34.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.864+0000 7f87b572b700 1 -- 172.21.15.46:0/1026598208 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b004c710 con 0x7f87b00abfc0 2024-01-27T00:47:34.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.865+0000 7f8796ffd700 1 -- 172.21.15.46:0/1026598208 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 97) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f87a001b3b0 con 0x7f87b00abfc0 2024-01-27T00:47:34.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.865+0000 7f8796ffd700 1 --2- 172.21.15.46:0/1026598208 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f879c066f80 0x7f879c069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:34.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.865+0000 7f8796ffd700 1 -- 172.21.15.46:0/1026598208 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f87a0083cb0 con 0x7f87b00abfc0 2024-01-27T00:47:34.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.865+0000 7f87ae7fc700 1 -- 172.21.15.46:0/1026598208 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f879c066f80 msgr2=0x7f879c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/43388 2024-01-27T00:47:34.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.865+0000 7f87ae7fc700 1 --2- 172.21.15.46:0/1026598208 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f879c066f80 0x7f879c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:47:34.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:34.869+0000 7f8796ffd700 1 -- 172.21.15.46:0/1026598208 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f87a0050c30 con 0x7f87b00abfc0 2024-01-27T00:47:35.026 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:35.026+0000 7fdc443891c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T00:47:35.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.066+0000 7f87ae7fc700 1 -- 172.21.15.46:0/1026598208 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f879c066f80 msgr2=0x7f879c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/43388 2024-01-27T00:47:35.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.066+0000 7f87ae7fc700 1 --2- 172.21.15.46:0/1026598208 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f879c066f80 0x7f879c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:47:35.075 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:35.075+0000 7f6342be71c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T00:47:35.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.077+0000 7f87b572b700 1 -- 172.21.15.46:0/1026598208 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f87b0068b30 con 0x7f87b00abfc0 2024-01-27T00:47:35.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.084+0000 7f8796ffd700 1 -- 172.21.15.46:0/1026598208 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7f87a0017020 con 0x7f87b00abfc0 2024-01-27T00:47:35.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.086+0000 7f87b572b700 1 -- 172.21.15.46:0/1026598208 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f879c066f80 msgr2=0x7f879c069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:47:35.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.086+0000 7f87b572b700 1 --2- 172.21.15.46:0/1026598208 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f879c066f80 0x7f879c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.086+0000 7f87b572b700 1 -- 172.21.15.46:0/1026598208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87b00abfc0 msgr2=0x7f87b00c23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:35.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.086+0000 7f87b572b700 1 --2- 172.21.15.46:0/1026598208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87b00abfc0 0x7f87b00c23d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f87a000c580 tx=0x7f87a0007d50 comp rx=0 tx=0).stop 2024-01-27T00:47:35.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.087+0000 7f87b572b700 1 -- 172.21.15.46:0/1026598208 shutdown_connections 2024-01-27T00:47:35.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.087+0000 7f87b572b700 1 --2- 172.21.15.46:0/1026598208 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f879c066f80 0x7f879c069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.087+0000 7f87b572b700 1 --2- 172.21.15.46:0/1026598208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87b0159990 0x7f87b016bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.087+0000 7f87b572b700 1 --2- 172.21.15.46:0/1026598208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87b00abfc0 0x7f87b00c23d0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.087+0000 7f87b572b700 1 --2- 172.21.15.46:0/1026598208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87b01595c0 0x7f87b016ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.087+0000 7f87b572b700 1 -- 172.21.15.46:0/1026598208 >> 172.21.15.46:0/1026598208 conn(0x7f87b00ccc20 msgr2=0x7f87b00b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:35.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.087+0000 7f87b572b700 1 -- 172.21.15.46:0/1026598208 shutdown_connections 2024-01-27T00:47:35.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.087+0000 7f87b572b700 1 -- 172.21.15.46:0/1026598208 wait complete. 2024-01-27T00:47:35.101 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-27T00:47:35.102 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:35.102+0000 7fdc443891c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T00:47:35.166 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:35.166+0000 7f6342be71c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T00:47:35.172 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:35.172+0000 7fdc443891c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T00:47:35.236 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:35.237+0000 7fdc443891c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T00:47:35.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.241+0000 7f9126325700 1 Processor -- start 2024-01-27T00:47:35.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.241+0000 7f9126325700 1 -- start start 2024-01-27T00:47:35.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.242+0000 7f9126325700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f91201599d0 0x7f9120159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.242+0000 7f9126325700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91200ab1c0 0x7f91200b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.242+0000 7f9126325700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91200b1a60 con 0x7f912015d8c0 2024-01-27T00:47:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.242+0000 7f9126325700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91200ab5f0 con 0x7f91201599d0 2024-01-27T00:47:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.242+0000 7f9126325700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91200b0f40 con 0x7f91200ab1c0 2024-01-27T00:47:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.242+0000 7f911f7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91200ab1c0 0x7f91200b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.242+0000 7f911ffff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f91201599d0 0x7f9120159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.242+0000 7f91248c2700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f912015d8c0 0x7f91200b20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:55212/0 (socket says 172.21.15.46:55212) 2024-01-27T00:47:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.242+0000 7f91248c2700 1 -- 172.21.15.46:0/3175738253 learned_addr learned my addr 172.21.15.46:0/3175738253 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.243+0000 7f911ffff700 1 -- 172.21.15.46:0/3175738253 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91200ab1c0 msgr2=0x7f91200b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.243+0000 7f911ffff700 1 --2- 172.21.15.46:0/3175738253 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91200ab1c0 0x7f91200b0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.243+0000 7f911ffff700 1 -- 172.21.15.46:0/3175738253 >> v1:172.21.15.106:6789/0 conn(0x7f912015d8c0 legacy=0x7f91200b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.243+0000 7f911ffff700 1 -- 172.21.15.46:0/3175738253 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91200b11c0 con 0x7f91201599d0 2024-01-27T00:47:35.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.243+0000 7f911effd700 1 -- 172.21.15.46:0/3175738253 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927523267 0 0) 0x7f91200b1a60 con 0x7f912015d8c0 2024-01-27T00:47:35.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.243+0000 7f911f7fe700 1 --2- 172.21.15.46:0/3175738253 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91200ab1c0 0x7f91200b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:35.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.243+0000 7f911ffff700 1 --2- 172.21.15.46:0/3175738253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91201599d0 0x7f9120159da0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f91100098d0 tx=0x7f9110009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4ef494e9fb21928 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:35.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.243+0000 7f911effd700 1 -- 172.21.15.46:0/3175738253 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9110005c10 con 0x7f91201599d0 2024-01-27T00:47:35.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.243+0000 7f911effd700 1 -- 172.21.15.46:0/3175738253 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9110005d70 con 0x7f91201599d0 2024-01-27T00:47:35.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.243+0000 7f911effd700 1 -- 172.21.15.46:0/3175738253 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9110006280 con 0x7f91201599d0 2024-01-27T00:47:35.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.244+0000 7f9126325700 1 -- 172.21.15.46:0/3175738253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91201599d0 msgr2=0x7f9120159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:35.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.244+0000 7f9126325700 1 --2- 172.21.15.46:0/3175738253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91201599d0 0x7f9120159da0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f91100098d0 tx=0x7f9110009120 comp rx=0 tx=0).stop 2024-01-27T00:47:35.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.244+0000 7f9126325700 1 -- 172.21.15.46:0/3175738253 shutdown_connections 2024-01-27T00:47:35.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.244+0000 7f9126325700 1 --2- 172.21.15.46:0/3175738253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91201599d0 0x7f9120159da0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.244+0000 7f9126325700 1 --2- 172.21.15.46:0/3175738253 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91200ab1c0 0x7f91200b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.244+0000 7f9126325700 1 -- 172.21.15.46:0/3175738253 >> 172.21.15.46:0/3175738253 conn(0x7f912006f840 msgr2=0x7f9120087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:35.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.244+0000 7f9126325700 1 -- 172.21.15.46:0/3175738253 shutdown_connections 2024-01-27T00:47:35.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.244+0000 7f9126325700 1 -- 172.21.15.46:0/3175738253 wait complete. 2024-01-27T00:47:35.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.246+0000 7f9126325700 1 Processor -- start 2024-01-27T00:47:35.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.246+0000 7f9126325700 1 -- start start 2024-01-27T00:47:35.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f9126325700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91200ab1c0 0x7f912009c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:35.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f9126325700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91201599d0 0x7f9120149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:35.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f9126325700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f912015d8c0 0x7f91201586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:35.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f9126325700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912009d400 con 0x7f912015d8c0 2024-01-27T00:47:35.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f9126325700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9120157c40 con 0x7f91200ab1c0 2024-01-27T00:47:35.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f9126325700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91200ab980 con 0x7f91201599d0 2024-01-27T00:47:35.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f91248c2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f912015d8c0 0x7f91201586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:35.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f911ffff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91200ab1c0 0x7f912009c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:35.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f911f7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91201599d0 0x7f9120149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:35.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f91248c2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f912015d8c0 0x7f91201586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47886/0 (socket says 172.21.15.46:47886) 2024-01-27T00:47:35.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f91248c2700 1 -- 172.21.15.46:0/3384261769 learned_addr learned my addr 172.21.15.46:0/3384261769 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:35.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f911ffff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91200ab1c0 0x7f912009c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:51900/0 (socket says 172.21.15.46:51900) 2024-01-27T00:47:35.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f911f7fe700 1 -- 172.21.15.46:0/3384261769 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91200ab1c0 msgr2=0x7f912009c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:35.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f911f7fe700 1 --2- 172.21.15.46:0/3384261769 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91200ab1c0 0x7f912009c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f911f7fe700 1 -- 172.21.15.46:0/3384261769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f912015d8c0 msgr2=0x7f91201586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:35.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f911f7fe700 1 --2- 172.21.15.46:0/3384261769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f912015d8c0 0x7f91201586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f911f7fe700 1 -- 172.21.15.46:0/3384261769 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9114009420 con 0x7f91201599d0 2024-01-27T00:47:35.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.248+0000 7f911f7fe700 1 --2- 172.21.15.46:0/3384261769 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91201599d0 0x7f9120149b90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f911400ecb0 tx=0x7f91140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:35.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.249+0000 7f9107fff700 1 -- 172.21.15.46:0/3384261769 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114009700 con 0x7f91201599d0 2024-01-27T00:47:35.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.249+0000 7f9107fff700 1 -- 172.21.15.46:0/3384261769 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911400c850 con 0x7f91201599d0 2024-01-27T00:47:35.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.249+0000 7f9126325700 1 -- 172.21.15.46:0/3384261769 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91100093c0 con 0x7f91201599d0 2024-01-27T00:47:35.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.249+0000 7f9107fff700 1 -- 172.21.15.46:0/3384261769 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911400cbf0 con 0x7f91201599d0 2024-01-27T00:47:35.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.249+0000 7f9126325700 1 -- 172.21.15.46:0/3384261769 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f912009ce40 con 0x7f91201599d0 2024-01-27T00:47:35.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.250+0000 7f9126325700 1 -- 172.21.15.46:0/3384261769 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912004c710 con 0x7f91201599d0 2024-01-27T00:47:35.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.250+0000 7f9107fff700 1 -- 172.21.15.46:0/3384261769 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 97) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f9114013470 con 0x7f91201599d0 2024-01-27T00:47:35.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.251+0000 7f9107fff700 1 --2- 172.21.15.46:0/3384261769 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f910c066f80 0x7f910c069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:35.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.251+0000 7f911ffff700 1 -- 172.21.15.46:0/3384261769 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f910c066f80 msgr2=0x7f910c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/43388 2024-01-27T00:47:35.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.251+0000 7f9107fff700 1 -- 172.21.15.46:0/3384261769 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f9114084890 con 0x7f91201599d0 2024-01-27T00:47:35.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.251+0000 7f911ffff700 1 --2- 172.21.15.46:0/3384261769 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f910c066f80 0x7f910c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:47:35.253 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:35.251+0000 7f6342be71c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T00:47:35.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.255+0000 7f9107fff700 1 -- 172.21.15.46:0/3384261769 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9114051770 con 0x7f91201599d0 2024-01-27T00:47:35.328 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:35.327+0000 7f6342be71c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T00:47:35.392 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:35.393+0000 7fdc443891c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:47:35.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.452+0000 7f911ffff700 1 -- 172.21.15.46:0/3384261769 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f910c066f80 msgr2=0x7f910c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/43388 2024-01-27T00:47:35.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.452+0000 7f911ffff700 1 --2- 172.21.15.46:0/3384261769 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f910c066f80 0x7f910c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:47:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.462+0000 7f9126325700 1 -- 172.21.15.46:0/3384261769 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f91200b14a0 con 0x7f91201599d0 2024-01-27T00:47:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.469+0000 7f9107fff700 1 -- 172.21.15.46:0/3384261769 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9114054d90 con 0x7f91201599d0 2024-01-27T00:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.471+0000 7f9126325700 1 -- 172.21.15.46:0/3384261769 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f910c066f80 msgr2=0x7f910c069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.471+0000 7f9126325700 1 --2- 172.21.15.46:0/3384261769 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f910c066f80 0x7f910c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.471+0000 7f9126325700 1 -- 172.21.15.46:0/3384261769 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91201599d0 msgr2=0x7f9120149b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.471+0000 7f9126325700 1 --2- 172.21.15.46:0/3384261769 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91201599d0 0x7f9120149b90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f911400ecb0 tx=0x7f91140091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.471+0000 7f9126325700 1 -- 172.21.15.46:0/3384261769 shutdown_connections 2024-01-27T00:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.471+0000 7f9126325700 1 --2- 172.21.15.46:0/3384261769 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f910c066f80 0x7f910c069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.471+0000 7f9126325700 1 --2- 172.21.15.46:0/3384261769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f912015d8c0 0x7f91201586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.471+0000 7f9126325700 1 --2- 172.21.15.46:0/3384261769 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91200ab1c0 0x7f912009c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.471+0000 7f9126325700 1 --2- 172.21.15.46:0/3384261769 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91201599d0 0x7f9120149b90 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.471+0000 7f9126325700 1 -- 172.21.15.46:0/3384261769 >> 172.21.15.46:0/3384261769 conn(0x7f912006f840 msgr2=0x7f91200b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.471+0000 7f9126325700 1 -- 172.21.15.46:0/3384261769 shutdown_connections 2024-01-27T00:47:35.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.471+0000 7f9126325700 1 -- 172.21.15.46:0/3384261769 wait complete. 2024-01-27T00:47:35.486 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:47:35.486 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-27T00:47:35.509 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:35.509+0000 7f6342be71c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:47:35.510 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:47:35.510 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2024-01-27T00:47:35.531 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:35.531+0000 7fdc443891c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T00:47:35.572 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/test_restart 1m 2024-01-27T00:47:35.597 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:35.597+0000 7fdc443891c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T00:47:35.668 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:35.668+0000 7f6342be71c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T00:47:35.719 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:35.719+0000 7fdc443891c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T00:47:35.748 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:35.748+0000 7f6342be71c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T00:47:35.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.748+0000 7f3c1094e700 1 Processor -- start 2024-01-27T00:47:35.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.749+0000 7f3c1094e700 1 -- start start 2024-01-27T00:47:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.750+0000 7f3c1094e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c0c068b00 0x7f3c0c068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.750+0000 7f3c1094e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c0c06a660 0x7f3c0c159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.750+0000 7f3c1094e700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0c0a9a80 con 0x7f3c0c06add0 2024-01-27T00:47:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.750+0000 7f3c1094e700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c0c15a3e0 con 0x7f3c0c06a660 2024-01-27T00:47:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.750+0000 7f3c1094e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c15a680 con 0x7f3c0c068b00 2024-01-27T00:47:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.750+0000 7f3c09d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c0c06a660 0x7f3c0c159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.750+0000 7f3c0ad9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3c0c06add0 0x7f3c0c15d4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57408/0 (socket says 172.21.15.46:57408) 2024-01-27T00:47:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.750+0000 7f3c0ad9d700 1 -- 172.21.15.46:0/3316119668 learned_addr learned my addr 172.21.15.46:0/3316119668 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:47:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.750+0000 7f3c0a59c700 1 --2- 172.21.15.46:0/3316119668 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c0c068b00 0x7f3c0c068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.750+0000 7f3c09d9b700 1 -- 172.21.15.46:0/3316119668 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c0c068b00 msgr2=0x7f3c0c068ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.750+0000 7f3c09d9b700 1 --2- 172.21.15.46:0/3316119668 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c0c068b00 0x7f3c0c068ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.750+0000 7f3c09d9b700 1 -- 172.21.15.46:0/3316119668 >> v1:172.21.15.46:6789/0 conn(0x7f3c0c06add0 legacy=0x7f3c0c15d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.750+0000 7f3c0959a700 1 -- 172.21.15.46:0/3316119668 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045044797 0 0) 0x7f3c0c0a9a80 con 0x7f3c0c06add0 2024-01-27T00:47:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.750+0000 7f3c09d9b700 1 -- 172.21.15.46:0/3316119668 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c06aad0 con 0x7f3c0c06a660 2024-01-27T00:47:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.750+0000 7f3c0a59c700 1 --2- 172.21.15.46:0/3316119668 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c0c068b00 0x7f3c0c068ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.751+0000 7f3c09d9b700 1 --2- 172.21.15.46:0/3316119668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c0c06a660 0x7f3c0c159ea0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f3bf40098e0 tx=0x7f3bf40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01fbcbd86c438b7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.751+0000 7f3c0959a700 1 -- 172.21.15.46:0/3316119668 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf400b640 con 0x7f3c0c06a660 2024-01-27T00:47:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.751+0000 7f3c0959a700 1 -- 172.21.15.46:0/3316119668 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf400b7a0 con 0x7f3c0c06a660 2024-01-27T00:47:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.751+0000 7f3c0959a700 1 -- 172.21.15.46:0/3316119668 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf400bb40 con 0x7f3c0c06a660 2024-01-27T00:47:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.752+0000 7f3c1094e700 1 -- 172.21.15.46:0/3316119668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c0c06a660 msgr2=0x7f3c0c159ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.752+0000 7f3c1094e700 1 --2- 172.21.15.46:0/3316119668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c0c06a660 0x7f3c0c159ea0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f3bf40098e0 tx=0x7f3bf40091a0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.752+0000 7f3c1094e700 1 -- 172.21.15.46:0/3316119668 shutdown_connections 2024-01-27T00:47:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.752+0000 7f3c1094e700 1 --2- 172.21.15.46:0/3316119668 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c0c068b00 0x7f3c0c068ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.752+0000 7f3c1094e700 1 --2- 172.21.15.46:0/3316119668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c0c06a660 0x7f3c0c159ea0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.752+0000 7f3c1094e700 1 -- 172.21.15.46:0/3316119668 >> 172.21.15.46:0/3316119668 conn(0x7f3c0c0ca100 msgr2=0x7f3c0c06e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:35.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.752+0000 7f3c1094e700 1 -- 172.21.15.46:0/3316119668 shutdown_connections 2024-01-27T00:47:35.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.752+0000 7f3c1094e700 1 -- 172.21.15.46:0/3316119668 wait complete. 2024-01-27T00:47:35.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.754+0000 7f3c1094e700 1 Processor -- start 2024-01-27T00:47:35.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.754+0000 7f3c1094e700 1 -- start start 2024-01-27T00:47:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.755+0000 7f3c1094e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c0c068b00 0x7f3c0c0c32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.755+0000 7f3c1094e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c0c06a660 0x7f3c0c0c3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.755+0000 7f3c1094e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c0c06add0 0x7f3c0c0c68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.755+0000 7f3c1094e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c0b22e0 con 0x7f3c0c06a660 2024-01-27T00:47:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.755+0000 7f3c1094e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c0b16d0 con 0x7f3c0c068b00 2024-01-27T00:47:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.755+0000 7f3c1094e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c0b19b0 con 0x7f3c0c06add0 2024-01-27T00:47:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.755+0000 7f3c09d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c0c06a660 0x7f3c0c0c3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.755+0000 7f3c0a59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c0c068b00 0x7f3c0c0c32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.755+0000 7f3c09d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c0c06a660 0x7f3c0c0c3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47892/0 (socket says 172.21.15.46:47892) 2024-01-27T00:47:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.755+0000 7f3c09d9b700 1 -- 172.21.15.46:0/1857075303 learned_addr learned my addr 172.21.15.46:0/1857075303 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.755+0000 7f3c0ad9d700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c0c06add0 0x7f3c0c0c68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:35.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.756+0000 7f3c09d9b700 1 -- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c0c06add0 msgr2=0x7f3c0c0c68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.756+0000 7f3c09d9b700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c0c06add0 0x7f3c0c0c68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.756+0000 7f3c09d9b700 1 -- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c0c068b00 msgr2=0x7f3c0c0c32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.756+0000 7f3c09d9b700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c0c068b00 0x7f3c0c0c32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.756+0000 7f3c09d9b700 1 -- 172.21.15.46:0/1857075303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc0093c0 con 0x7f3c0c06a660 2024-01-27T00:47:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.756+0000 7f3c09d9b700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c0c06a660 0x7f3c0c0c3ee0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c069d70 tx=0x7f3bf4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.756+0000 7f3c027fc700 1 -- 172.21.15.46:0/1857075303 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf400bb20 con 0x7f3c0c06a660 2024-01-27T00:47:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.756+0000 7f3c027fc700 1 -- 172.21.15.46:0/1857075303 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf4000c90 con 0x7f3c0c06a660 2024-01-27T00:47:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.756+0000 7f3c1094e700 1 -- 172.21.15.46:0/1857075303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf4009420 con 0x7f3c0c06a660 2024-01-27T00:47:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.756+0000 7f3c027fc700 1 -- 172.21.15.46:0/1857075303 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4011630 con 0x7f3c0c06a660 2024-01-27T00:47:35.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.756+0000 7f3c1094e700 1 -- 172.21.15.46:0/1857075303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c0c0b1f70 con 0x7f3c0c06a660 2024-01-27T00:47:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.757+0000 7f3c1094e700 1 -- 172.21.15.46:0/1857075303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0c04c710 con 0x7f3c0c06a660 2024-01-27T00:47:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.758+0000 7f3c027fc700 1 -- 172.21.15.46:0/1857075303 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 97) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f3bf40119a0 con 0x7f3c0c06a660 2024-01-27T00:47:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.758+0000 7f3c027fc700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 0x7f3bec069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.758+0000 7f3c027fc700 1 -- 172.21.15.46:0/1857075303 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f3bf4085c30 con 0x7f3c0c06a660 2024-01-27T00:47:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.758+0000 7f3c0a59c700 1 -- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 msgr2=0x7f3bec069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/43388 2024-01-27T00:47:35.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.759+0000 7f3c0a59c700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 0x7f3bec069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T00:47:35.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.763+0000 7f3c027fc700 1 -- 172.21.15.46:0/1857075303 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3bf4052b10 con 0x7f3c0c06a660 2024-01-27T00:47:35.791 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:35.791+0000 7fdc443891c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T00:47:35.890 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:35.890+0000 7f6342be71c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T00:47:35.930 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:35.930+0000 7fdc443891c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T00:47:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.960+0000 7f3c0a59c700 1 -- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 msgr2=0x7f3bec069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/43388 2024-01-27T00:47:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:35.960+0000 7f3c0a59c700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 0x7f3bec069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T00:47:35.973 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:35.973+0000 7f6342be71c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T00:47:36.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:36.001+0000 7f3c1094e700 1 -- 172.21.15.46:0/1857075303 --> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c0c06edf0 con 0x7f3bec066f80 2024-01-27T00:47:36.055 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:36.055+0000 7fdc443891c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T00:47:36.149 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:36.148+0000 7f6342be71c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T00:47:36.289 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:36.289+0000 7f6342be71c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T00:47:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:36.361+0000 7f3c0a59c700 1 -- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 msgr2=0x7f3bec069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/43388 2024-01-27T00:47:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:36.361+0000 7f3c0a59c700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 0x7f3bec069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-27T00:47:36.367 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:36.368+0000 7fdc443891c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T00:47:36.579 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:36.579+0000 7fdc443891c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T00:47:36.651 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:36.651+0000 7f6342be71c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T00:47:36.803 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:36.803+0000 7fdc443891c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T00:47:36.873 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:36.873+0000 7fdc443891c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T00:47:36.890 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:36.890+0000 7f6342be71c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T00:47:37.150 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:37.150+0000 7f6342be71c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T00:47:37.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:37.162+0000 7f3c0a59c700 1 -- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 msgr2=0x7f3bec069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/43388 2024-01-27T00:47:37.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:37.162+0000 7f3c0a59c700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 0x7f3bec069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-27T00:47:37.231 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:37.230+0000 7f6342be71c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T00:47:37.468 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:37.468+0000 7fdc443891c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T00:47:37.543 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:37.543+0000 7fdc443891c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T00:47:37.664 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:37.664+0000 7fdc443891c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T00:47:37.927 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:37.926+0000 7f6342be71c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T00:47:38.011 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:38.011+0000 7f6342be71c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T00:47:38.152 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:38.152+0000 7f6342be71c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T00:47:38.160 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:38.161+0000 7fdc443891c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T00:47:38.309 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:38.309+0000 7fdc443891c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T00:47:38.456 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:38.457+0000 7fdc443891c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T00:47:38.735 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:38.735+0000 7f6342be71c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T00:47:38.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:38.763+0000 7f3c0a59c700 1 -- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 msgr2=0x7f3bec069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/43388 2024-01-27T00:47:38.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:38.763+0000 7f3c0a59c700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 0x7f3bec069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-27T00:47:38.905 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:38.905+0000 7f6342be71c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T00:47:39.077 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:39.077+0000 7f6342be71c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T00:47:39.153 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:39.153+0000 7fdc443891c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T00:47:39.369 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:39.369+0000 7fdc443891c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:47:39.441 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:39.441+0000 7fdc443891c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T00:47:39.594 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:39.594+0000 7fdc443891c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T00:47:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:39.616+0000 7f3c027fc700 1 -- 172.21.15.46:0/1857075303 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 98) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f3bf4083e50 con 0x7f3c0c06a660 2024-01-27T00:47:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:39.616+0000 7f3c027fc700 1 -- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 msgr2=0x7f3bec069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T00:47:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:39.616+0000 7f3c027fc700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 0x7f3bec069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:39.720 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:39.720+0000 7fdbbac67700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:39.720 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:39.720+0000 7fdbbac67700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:39.720 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:39.720+0000 7fdbbac67700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:39.720 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:39.720+0000 7fdbbac67700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:39.720 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:39.720+0000 7fdbbac67700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:39.720 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:39.720+0000 7fdbbdc6d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:39.720 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:39.720+0000 7fdbbdc6d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:39.720 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:39.720+0000 7fdbbdc6d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:39.721 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:39.720+0000 7fdbbdc6d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:39.721 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:47:39.720+0000 7fdbbdc6d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:47:39.865 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:47:39.867 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:47:39.905 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:39.905+0000 7f6342be71c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T00:47:40.155 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:40.154+0000 7f6342be71c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:47:40.237 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:40.237+0000 7f6342be71c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T00:47:40.414 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T00:47:40.414+0000 7f6342be71c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T00:47:40.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.618+0000 7f3c027fc700 1 -- 172.21.15.46:0/1857075303 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 99) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f3bf4052ee0 con 0x7f3c0c06a660 2024-01-27T00:47:40.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.618+0000 7f3c027fc700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 0x7f3bec0693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:40.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.618+0000 7f3c027fc700 1 -- 172.21.15.46:0/1857075303 --> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c0c06edf0 con 0x7f3bec066f80 2024-01-27T00:47:40.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.619+0000 7f3c0a59c700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 0x7f3bec0693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:40.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.621+0000 7f3c0a59c700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 0x7f3bec0693b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc0097c0 tx=0x7f3bfc009120 comp rx=0 tx=0).ready entity=mgr.8588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:40.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.683+0000 7f3c027fc700 1 -- 172.21.15.46:0/1857075303 <== mgr.8588 v2:172.21.15.106:6836/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f3c0c06edf0 con 0x7f3bec066f80 2024-01-27T00:47:40.684 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path /snap_test_dir1/test_restart 2024-01-27T00:47:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.686+0000 7f3c1094e700 1 -- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 msgr2=0x7f3bec0693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.686+0000 7f3c1094e700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 0x7f3bec0693b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc0097c0 tx=0x7f3bfc009120 comp rx=0 tx=0).stop 2024-01-27T00:47:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.686+0000 7f3c1094e700 1 -- 172.21.15.46:0/1857075303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c0c06a660 msgr2=0x7f3c0c0c3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.686+0000 7f3c1094e700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c0c06a660 0x7f3c0c0c3ee0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c069d70 tx=0x7f3bf4017000 comp rx=0 tx=0).stop 2024-01-27T00:47:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.686+0000 7f3c1094e700 1 -- 172.21.15.46:0/1857075303 shutdown_connections 2024-01-27T00:47:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.686+0000 7f3c1094e700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f3bec066f80 0x7f3bec0693b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.686+0000 7f3c1094e700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c0c06add0 0x7f3c0c0c68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.686+0000 7f3c1094e700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c0c068b00 0x7f3c0c0c32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.686+0000 7f3c1094e700 1 --2- 172.21.15.46:0/1857075303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c0c06a660 0x7f3c0c0c3ee0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.686+0000 7f3c1094e700 1 -- 172.21.15.46:0/1857075303 >> 172.21.15.46:0/1857075303 conn(0x7f3c0c0ca100 msgr2=0x7f3c0c0aaea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.686+0000 7f3c1094e700 1 -- 172.21.15.46:0/1857075303 shutdown_connections 2024-01-27T00:47:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.686+0000 7f3c1094e700 1 -- 172.21.15.46:0/1857075303 wait complete. 2024-01-27T00:47:40.702 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2024-01-27T00:47:40.702 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/test_restart 1m 2024-01-27T00:47:40.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.843+0000 7fdd794b5700 1 Processor -- start 2024-01-27T00:47:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.843+0000 7fdd794b5700 1 -- start start 2024-01-27T00:47:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.844+0000 7fdd794b5700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdd741598e0 0x7fdd740c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.844+0000 7fdd794b5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd740c9120 0x7fdd740cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.844+0000 7fdd794b5700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd74069260 con 0x7fdd74159510 2024-01-27T00:47:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.844+0000 7fdd794b5700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd740ca930 con 0x7fdd741598e0 2024-01-27T00:47:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.844+0000 7fdd794b5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd74159d50 con 0x7fdd740c9120 2024-01-27T00:47:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.844+0000 7fdd737fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd740c9120 0x7fdd740cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.844+0000 7fdd6bfff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdd741598e0 0x7fdd740c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.844+0000 7fdd737fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd740c9120 0x7fdd740cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:46866/0 (socket says 172.21.15.46:46866) 2024-01-27T00:47:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.844+0000 7fdd737fe700 1 -- 172.21.15.46:0/2696369679 learned_addr learned my addr 172.21.15.46:0/2696369679 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.844+0000 7fdd72ffd700 1 --1- 172.21.15.46:0/2696369679 >> v1:172.21.15.106:6789/0 conn(0x7fdd74159510 0x7fdd740ca220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:39230/0 (socket says 172.21.15.46:39230) 2024-01-27T00:47:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.845+0000 7fdd737fe700 1 -- 172.21.15.46:0/2696369679 >> v2:172.21.15.106:3300/0 conn(0x7fdd741598e0 msgr2=0x7fdd740c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:47:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.845+0000 7fdd737fe700 1 --2- 172.21.15.46:0/2696369679 >> v2:172.21.15.106:3300/0 conn(0x7fdd741598e0 0x7fdd740c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.845+0000 7fdd737fe700 1 -- 172.21.15.46:0/2696369679 >> v1:172.21.15.106:6789/0 conn(0x7fdd74159510 legacy=0x7fdd740ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.845+0000 7fdd737fe700 1 -- 172.21.15.46:0/2696369679 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7415d670 con 0x7fdd740c9120 2024-01-27T00:47:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.845+0000 7fdd6bfff700 1 --2- 172.21.15.46:0/2696369679 >> v2:172.21.15.106:3300/0 conn(0x7fdd741598e0 0x7fdd740c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.845+0000 7fdd737fe700 1 --2- 172.21.15.46:0/2696369679 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd740c9120 0x7fdd740cc380 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c009f20 tx=0x7fdd6c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c98a786f1daafc4e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.845+0000 7fdd727fc700 1 -- 172.21.15.46:0/2696369679 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c00e070 con 0x7fdd740c9120 2024-01-27T00:47:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.845+0000 7fdd727fc700 1 -- 172.21.15.46:0/2696369679 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6c00b9d0 con 0x7fdd740c9120 2024-01-27T00:47:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.845+0000 7fdd727fc700 1 -- 172.21.15.46:0/2696369679 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c0092e0 con 0x7fdd740c9120 2024-01-27T00:47:40.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.846+0000 7fdd794b5700 1 -- 172.21.15.46:0/2696369679 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd740c9120 msgr2=0x7fdd740cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.846+0000 7fdd794b5700 1 --2- 172.21.15.46:0/2696369679 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd740c9120 0x7fdd740cc380 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c009f20 tx=0x7fdd6c009840 comp rx=0 tx=0).stop 2024-01-27T00:47:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.846+0000 7fdd794b5700 1 -- 172.21.15.46:0/2696369679 shutdown_connections 2024-01-27T00:47:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.846+0000 7fdd794b5700 1 --2- 172.21.15.46:0/2696369679 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd740c9120 0x7fdd740cc380 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.846+0000 7fdd794b5700 1 --2- 172.21.15.46:0/2696369679 >> v2:172.21.15.106:3300/0 conn(0x7fdd741598e0 0x7fdd740c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.846+0000 7fdd794b5700 1 -- 172.21.15.46:0/2696369679 >> 172.21.15.46:0/2696369679 conn(0x7fdd7406bd90 msgr2=0x7fdd7406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.846+0000 7fdd794b5700 1 -- 172.21.15.46:0/2696369679 shutdown_connections 2024-01-27T00:47:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.846+0000 7fdd794b5700 1 -- 172.21.15.46:0/2696369679 wait complete. 2024-01-27T00:47:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.847+0000 7fdd794b5700 1 Processor -- start 2024-01-27T00:47:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.847+0000 7fdd794b5700 1 -- start start 2024-01-27T00:47:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.848+0000 7fdd794b5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd740c9120 0x7fdd7415c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.848+0000 7fdd794b5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd74159510 0x7fdd7415a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.848+0000 7fdd794b5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd741598e0 0x7fdd74153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.848+0000 7fdd794b5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7415a0c0 con 0x7fdd740c9120 2024-01-27T00:47:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.848+0000 7fdd794b5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd740c6ba0 con 0x7fdd74159510 2024-01-27T00:47:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.848+0000 7fdd794b5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd740c6300 con 0x7fdd741598e0 2024-01-27T00:47:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.848+0000 7fdd72ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd740c9120 0x7fdd7415c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.848+0000 7fdd72ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd740c9120 0x7fdd7415c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49338/0 (socket says 172.21.15.46:49338) 2024-01-27T00:47:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.848+0000 7fdd72ffd700 1 -- 172.21.15.46:0/681071118 learned_addr learned my addr 172.21.15.46:0/681071118 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:47:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.848+0000 7fdd6bfff700 1 --2- 172.21.15.46:0/681071118 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd74159510 0x7fdd7415a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.848+0000 7fdd737fe700 1 --2- 172.21.15.46:0/681071118 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd741598e0 0x7fdd74153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.848+0000 7fdd737fe700 1 -- 172.21.15.46:0/681071118 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd74159510 msgr2=0x7fdd7415a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.848+0000 7fdd737fe700 1 --2- 172.21.15.46:0/681071118 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd74159510 0x7fdd7415a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.848+0000 7fdd737fe700 1 -- 172.21.15.46:0/681071118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd740c9120 msgr2=0x7fdd7415c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.848+0000 7fdd737fe700 1 --2- 172.21.15.46:0/681071118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd740c9120 0x7fdd7415c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.848+0000 7fdd737fe700 1 -- 172.21.15.46:0/681071118 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c009420 con 0x7fdd741598e0 2024-01-27T00:47:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.848+0000 7fdd72ffd700 1 --2- 172.21.15.46:0/681071118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd740c9120 0x7fdd7415c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T00:47:40.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.849+0000 7fdd737fe700 1 --2- 172.21.15.46:0/681071118 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd741598e0 0x7fdd74153480 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c015000 tx=0x7fdd6c018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:40.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.849+0000 7fdd6affd700 1 -- 172.21.15.46:0/681071118 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c007b00 con 0x7fdd741598e0 2024-01-27T00:47:40.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.849+0000 7fdd6affd700 1 -- 172.21.15.46:0/681071118 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6c007c80 con 0x7fdd741598e0 2024-01-27T00:47:40.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.849+0000 7fdd6affd700 1 -- 172.21.15.46:0/681071118 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c01c660 con 0x7fdd741598e0 2024-01-27T00:47:40.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.849+0000 7fdd794b5700 1 -- 172.21.15.46:0/681071118 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6c009ae0 con 0x7fdd741598e0 2024-01-27T00:47:40.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.849+0000 7fdd794b5700 1 -- 172.21.15.46:0/681071118 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd740c68c0 con 0x7fdd741598e0 2024-01-27T00:47:40.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.850+0000 7fdd794b5700 1 -- 172.21.15.46:0/681071118 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd74159e90 con 0x7fdd741598e0 2024-01-27T00:47:40.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.850+0000 7fdd6affd700 1 -- 172.21.15.46:0/681071118 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 99) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fdd6c003050 con 0x7fdd741598e0 2024-01-27T00:47:40.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.851+0000 7fdd6affd700 1 --2- 172.21.15.46:0/681071118 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fdd54066e00 0x7fdd540692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:47:40.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.851+0000 7fdd6affd700 1 -- 172.21.15.46:0/681071118 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fdd6c085e70 con 0x7fdd741598e0 2024-01-27T00:47:40.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.851+0000 7fdd72ffd700 1 --2- 172.21.15.46:0/681071118 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fdd54066e00 0x7fdd540692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:47:40.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.851+0000 7fdd72ffd700 1 --2- 172.21.15.46:0/681071118 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fdd54066e00 0x7fdd540692b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdd64001690 tx=0x7fdd64007f70 comp rx=0 tx=0).ready entity=mgr.8588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:47:40.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:40.855+0000 7fdd6affd700 1 -- 172.21.15.46:0/681071118 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdd74159e90 con 0x7fdd741598e0 2024-01-27T00:47:41.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:41.090+0000 7fdd794b5700 1 -- 172.21.15.46:0/681071118 --> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd740c8030 con 0x7fdd54066e00 2024-01-27T00:47:41.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:41.140+0000 7fdd6affd700 1 -- 172.21.15.46:0/681071118 <== mgr.8588 v2:172.21.15.106:6836/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7fdd740c8030 con 0x7fdd54066e00 2024-01-27T00:47:41.141 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule activated for path /snap_test_dir1/test_restart 2024-01-27T00:47:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:41.143+0000 7fdd794b5700 1 -- 172.21.15.46:0/681071118 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fdd54066e00 msgr2=0x7fdd540692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:41.143+0000 7fdd794b5700 1 --2- 172.21.15.46:0/681071118 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fdd54066e00 0x7fdd540692b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdd64001690 tx=0x7fdd64007f70 comp rx=0 tx=0).stop 2024-01-27T00:47:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:41.143+0000 7fdd794b5700 1 -- 172.21.15.46:0/681071118 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd741598e0 msgr2=0x7fdd74153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:47:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:41.143+0000 7fdd794b5700 1 --2- 172.21.15.46:0/681071118 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd741598e0 0x7fdd74153480 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c015000 tx=0x7fdd6c018000 comp rx=0 tx=0).stop 2024-01-27T00:47:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:41.143+0000 7fdd794b5700 1 -- 172.21.15.46:0/681071118 shutdown_connections 2024-01-27T00:47:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:41.143+0000 7fdd794b5700 1 --2- 172.21.15.46:0/681071118 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7fdd54066e00 0x7fdd540692b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:41.143+0000 7fdd794b5700 1 --2- 172.21.15.46:0/681071118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd740c9120 0x7fdd7415c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:41.143+0000 7fdd794b5700 1 --2- 172.21.15.46:0/681071118 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd741598e0 0x7fdd74153480 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:41.143+0000 7fdd794b5700 1 --2- 172.21.15.46:0/681071118 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd74159510 0x7fdd7415a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:47:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:41.143+0000 7fdd794b5700 1 -- 172.21.15.46:0/681071118 >> 172.21.15.46:0/681071118 conn(0x7fdd7406bd90 msgr2=0x7fdd740cea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:47:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:41.143+0000 7fdd794b5700 1 -- 172.21.15.46:0/681071118 shutdown_connections 2024-01-27T00:47:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:47:41.143+0000 7fdd794b5700 1 -- 172.21.15.46:0/681071118 wait complete. 2024-01-27T00:47:41.158 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2024-01-27T00:48:00.152 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:48:00.152+0000 7fdbacc0b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:48:00.152 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:48:00.152+0000 7fdbacc0b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:48:00.153 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:48:00.152+0000 7fdbacc0b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:48:00.153 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:48:00.152+0000 7fdbacc0b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:48:00.153 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:48:00.152+0000 7fdbacc0b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:48:09.942 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:48:09.946 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:48:40.078 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:48:40.082 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:49:10.166 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:49:10.169 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:49:40.298 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:49:40.301 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:50:10.428 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:50:10.432 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:50:40.543 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:50:40.547 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:51:10.666 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:51:10.670 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:51:40.738 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:51:40.741 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:52:10.869 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:52:10.873 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:52:41.017 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:52:41.021 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:53:11.122 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:53:11.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:53:41.186 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:53:41.190 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:54:11.319 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:54:11.322 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:54:41.474 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:54:41.478 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:55:11.581 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:55:11.587 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:55:41.723 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:55:41.731 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:56:11.793 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:56:11.797 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:56:41.876 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:56:41.879 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:57:12.022 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:57:12.025 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:57:42.153 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:57:42.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:58:12.242 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:58:12.247 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:58:16.159 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T00:58:16.160 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-01-27T00:58:16.191 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_48_00_UTC 2024-01-27T00:58:16.191 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_49_00_UTC 2024-01-27T00:58:16.192 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_50_00_UTC 2024-01-27T00:58:16.192 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_51_00_UTC 2024-01-27T00:58:16.192 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_52_00_UTC 2024-01-27T00:58:16.192 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_53_00_UTC 2024-01-27T00:58:16.192 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_54_00_UTC 2024-01-27T00:58:16.192 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_55_00_UTC 2024-01-27T00:58:16.192 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_56_00_UTC 2024-01-27T00:58:16.192 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_57_00_UTC 2024-01-27T00:58:16.194 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_58_00_UTC 2024-01-27T00:58:16.194 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-27-00_48_00_UTC', 'scheduled-2024-01-27-00_49_00_UTC', 'scheduled-2024-01-27-00_50_00_UTC', 'scheduled-2024-01-27-00_51_00_UTC', 'scheduled-2024-01-27-00_52_00_UTC', 'scheduled-2024-01-27-00_53_00_UTC', 'scheduled-2024-01-27-00_54_00_UTC', 'scheduled-2024-01-27-00_55_00_UTC', 'scheduled-2024-01-27-00_56_00_UTC', 'scheduled-2024-01-27-00_57_00_UTC', 'scheduled-2024-01-27-00_58_00_UTC'] 2024-01-27T00:58:16.195 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-01-27T00:58:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.385+0000 7ff9fa787700 1 Processor -- start 2024-01-27T00:58:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.385+0000 7ff9fa787700 1 -- start start 2024-01-27T00:58:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.386+0000 7ff9fa787700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9f4067af0 0x7ff9f4067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:58:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.386+0000 7ff9fa787700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff9f406ebe0 0x7ff9f40ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:58:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.386+0000 7ff9fa787700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f406fdb0 con 0x7ff9f41597e0 2024-01-27T00:58:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.386+0000 7ff9fa787700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f40ade10 con 0x7ff9f4067af0 2024-01-27T00:58:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.386+0000 7ff9fa787700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f406f1a0 con 0x7ff9f406ebe0 2024-01-27T00:58:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.386+0000 7ff9f3fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9f4067af0 0x7ff9f4067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:58:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.386+0000 7ff9f8d24700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff9f41597e0 0x7ff9f4159c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48104/0 (socket says 172.21.15.46:48104) 2024-01-27T00:58:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.386+0000 7ff9f8d24700 1 -- 172.21.15.46:0/1300734010 learned_addr learned my addr 172.21.15.46:0/1300734010 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:58:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.386+0000 7ff9f37fe700 1 --2- 172.21.15.46:0/1300734010 >> v2:172.21.15.106:3300/0 conn(0x7ff9f406ebe0 0x7ff9f40ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:58:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.387+0000 7ff9f3fff700 1 -- 172.21.15.46:0/1300734010 >> v2:172.21.15.106:3300/0 conn(0x7ff9f406ebe0 msgr2=0x7ff9f40ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T00:58:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.387+0000 7ff9f3fff700 1 --2- 172.21.15.46:0/1300734010 >> v2:172.21.15.106:3300/0 conn(0x7ff9f406ebe0 0x7ff9f40ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:58:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.387+0000 7ff9f3fff700 1 -- 172.21.15.46:0/1300734010 >> v1:172.21.15.46:6789/0 conn(0x7ff9f41597e0 legacy=0x7ff9f4159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:58:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.387+0000 7ff9f3fff700 1 -- 172.21.15.46:0/1300734010 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f406f480 con 0x7ff9f4067af0 2024-01-27T00:58:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.387+0000 7ff9f2ffd700 1 -- 172.21.15.46:0/1300734010 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312052451 0 0) 0x7ff9f406fdb0 con 0x7ff9f41597e0 2024-01-27T00:58:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.387+0000 7ff9f37fe700 1 --2- 172.21.15.46:0/1300734010 >> v2:172.21.15.106:3300/0 conn(0x7ff9f406ebe0 0x7ff9f40ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T00:58:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.387+0000 7ff9f3fff700 1 --2- 172.21.15.46:0/1300734010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f4067af0 0x7ff9f4067ec0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7ff9f40ce5c0 tx=0x7ff9e40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fccf32b69adc235b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:58:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.387+0000 7ff9f2ffd700 1 -- 172.21.15.46:0/1300734010 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e400b6b0 con 0x7ff9f4067af0 2024-01-27T00:58:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.387+0000 7ff9f2ffd700 1 -- 172.21.15.46:0/1300734010 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e400b810 con 0x7ff9f4067af0 2024-01-27T00:58:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.387+0000 7ff9f2ffd700 1 -- 172.21.15.46:0/1300734010 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e400bbb0 con 0x7ff9f4067af0 2024-01-27T00:58:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.387+0000 7ff9fa787700 1 -- 172.21.15.46:0/1300734010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f4067af0 msgr2=0x7ff9f4067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:58:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.387+0000 7ff9fa787700 1 --2- 172.21.15.46:0/1300734010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f4067af0 0x7ff9f4067ec0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7ff9f40ce5c0 tx=0x7ff9e40091a0 comp rx=0 tx=0).stop 2024-01-27T00:58:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.388+0000 7ff9fa787700 1 -- 172.21.15.46:0/1300734010 shutdown_connections 2024-01-27T00:58:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.388+0000 7ff9fa787700 1 --2- 172.21.15.46:0/1300734010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f4067af0 0x7ff9f4067ec0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:58:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.388+0000 7ff9fa787700 1 --2- 172.21.15.46:0/1300734010 >> v2:172.21.15.106:3300/0 conn(0x7ff9f406ebe0 0x7ff9f40ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:58:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.388+0000 7ff9fa787700 1 -- 172.21.15.46:0/1300734010 >> 172.21.15.46:0/1300734010 conn(0x7ff9f40cdf80 msgr2=0x7ff9f406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:58:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.388+0000 7ff9fa787700 1 -- 172.21.15.46:0/1300734010 shutdown_connections 2024-01-27T00:58:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.388+0000 7ff9fa787700 1 -- 172.21.15.46:0/1300734010 wait complete. 2024-01-27T00:58:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.389+0000 7ff9fa787700 1 Processor -- start 2024-01-27T00:58:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.389+0000 7ff9fa787700 1 -- start start 2024-01-27T00:58:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9fa787700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f4067af0 0x7ff9f415b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:58:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9fa787700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9f406ebe0 0x7ff9f41713c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:58:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9fa787700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9f41597e0 0x7ff9f416e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:58:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9fa787700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f4156c10 con 0x7ff9f4067af0 2024-01-27T00:58:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9fa787700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f4156e20 con 0x7ff9f406ebe0 2024-01-27T00:58:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9fa787700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f4157100 con 0x7ff9f41597e0 2024-01-27T00:58:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9f3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f4067af0 0x7ff9f415b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:58:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9f3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f4067af0 0x7ff9f415b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35090/0 (socket says 172.21.15.46:35090) 2024-01-27T00:58:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9f3fff700 1 -- 172.21.15.46:0/4210103583 learned_addr learned my addr 172.21.15.46:0/4210103583 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:58:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9f37fe700 1 --2- 172.21.15.46:0/4210103583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9f406ebe0 0x7ff9f41713c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:58:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9f8d24700 1 --2- 172.21.15.46:0/4210103583 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9f41597e0 0x7ff9f416e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:58:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9f3fff700 1 -- 172.21.15.46:0/4210103583 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9f41597e0 msgr2=0x7ff9f416e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:58:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9f3fff700 1 --2- 172.21.15.46:0/4210103583 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9f41597e0 0x7ff9f416e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:58:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9f3fff700 1 -- 172.21.15.46:0/4210103583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9f406ebe0 msgr2=0x7ff9f41713c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:58:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9f3fff700 1 --2- 172.21.15.46:0/4210103583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9f406ebe0 0x7ff9f41713c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:58:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9f3fff700 1 -- 172.21.15.46:0/4210103583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e4009420 con 0x7ff9f4067af0 2024-01-27T00:58:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9f37fe700 1 --2- 172.21.15.46:0/4210103583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9f406ebe0 0x7ff9f41713c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:58:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.390+0000 7ff9f8d24700 1 --2- 172.21.15.46:0/4210103583 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9f41597e0 0x7ff9f416e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:58:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.391+0000 7ff9f3fff700 1 --2- 172.21.15.46:0/4210103583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f4067af0 0x7ff9f415b480 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4064440 tx=0x7ff9e4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:58:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.391+0000 7ff9ebfff700 1 -- 172.21.15.46:0/4210103583 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4000cc0 con 0x7ff9f4067af0 2024-01-27T00:58:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.391+0000 7ff9ebfff700 1 -- 172.21.15.46:0/4210103583 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e400b980 con 0x7ff9f4067af0 2024-01-27T00:58:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.391+0000 7ff9fa787700 1 -- 172.21.15.46:0/4210103583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f41573e0 con 0x7ff9f4067af0 2024-01-27T00:58:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.391+0000 7ff9ebfff700 1 -- 172.21.15.46:0/4210103583 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4026430 con 0x7ff9f4067af0 2024-01-27T00:58:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.391+0000 7ff9fa787700 1 -- 172.21.15.46:0/4210103583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f415b9c0 con 0x7ff9f4067af0 2024-01-27T00:58:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.392+0000 7ff9fa787700 1 -- 172.21.15.46:0/4210103583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f404c710 con 0x7ff9f4067af0 2024-01-27T00:58:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.392+0000 7ff9ebfff700 1 -- 172.21.15.46:0/4210103583 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 103) v1 ==== 84873+0+0 (secure 0 0 0) 0x7ff9e4023030 con 0x7ff9f4067af0 2024-01-27T00:58:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.393+0000 7ff9ebfff700 1 --2- 172.21.15.46:0/4210103583 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff9d40670a0 0x7ff9d4069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:58:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.393+0000 7ff9ebfff700 1 -- 172.21.15.46:0/4210103583 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6719+0+0 (secure 0 0 0) 0x7ff9e40862e0 con 0x7ff9f4067af0 2024-01-27T00:58:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.393+0000 7ff9f37fe700 1 --2- 172.21.15.46:0/4210103583 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff9d40670a0 0x7ff9d4069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:58:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.394+0000 7ff9f37fe700 1 --2- 172.21.15.46:0/4210103583 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff9d40670a0 0x7ff9d4069550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc006f00 tx=0x7ff9dc008040 comp rx=0 tx=0).ready entity=mgr.8588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:58:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.398+0000 7ff9ebfff700 1 -- 172.21.15.46:0/4210103583 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff9e4053050 con 0x7ff9f4067af0 2024-01-27T00:58:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.631+0000 7ff9fa787700 1 -- 172.21.15.46:0/4210103583 --> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff9f4068690 con 0x7ff9d40670a0 2024-01-27T00:58:16.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.639+0000 7ff9ebfff700 1 -- 172.21.15.46:0/4210103583 <== mgr.8588 v2:172.21.15.106:6836/43388 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7ff9f4068690 con 0x7ff9d40670a0 2024-01-27T00:58:16.639 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:58:16.639 INFO:teuthology.orchestra.run.smithi046.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-01-27T00:00:00", "created": "2024-01-27T00:47:40", "first": "2024-01-27T00:48:00", "last": "2024-01-27T00:58:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-01-27T00:58:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.641+0000 7ff9fa787700 1 -- 172.21.15.46:0/4210103583 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff9d40670a0 msgr2=0x7ff9d4069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:58:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.641+0000 7ff9fa787700 1 --2- 172.21.15.46:0/4210103583 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff9d40670a0 0x7ff9d4069550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc006f00 tx=0x7ff9dc008040 comp rx=0 tx=0).stop 2024-01-27T00:58:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.641+0000 7ff9fa787700 1 -- 172.21.15.46:0/4210103583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f4067af0 msgr2=0x7ff9f415b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:58:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.641+0000 7ff9fa787700 1 --2- 172.21.15.46:0/4210103583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f4067af0 0x7ff9f415b480 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4064440 tx=0x7ff9e4017000 comp rx=0 tx=0).stop 2024-01-27T00:58:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.641+0000 7ff9fa787700 1 -- 172.21.15.46:0/4210103583 shutdown_connections 2024-01-27T00:58:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.641+0000 7ff9fa787700 1 --2- 172.21.15.46:0/4210103583 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7ff9d40670a0 0x7ff9d4069550 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:58:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.641+0000 7ff9fa787700 1 --2- 172.21.15.46:0/4210103583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f4067af0 0x7ff9f415b480 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:58:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.641+0000 7ff9fa787700 1 --2- 172.21.15.46:0/4210103583 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9f41597e0 0x7ff9f416e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:58:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.641+0000 7ff9fa787700 1 --2- 172.21.15.46:0/4210103583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9f406ebe0 0x7ff9f41713c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:58:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.641+0000 7ff9fa787700 1 -- 172.21.15.46:0/4210103583 >> 172.21.15.46:0/4210103583 conn(0x7ff9f40cdf80 msgr2=0x7ff9f40cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:58:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.641+0000 7ff9fa787700 1 -- 172.21.15.46:0/4210103583 shutdown_connections 2024-01-27T00:58:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.641+0000 7ff9fa787700 1 -- 172.21.15.46:0/4210103583 wait complete. 2024-01-27T00:58:16.653 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-01-27T00:00:00", "created": "2024-01-27T00:47:40", "first": "2024-01-27T00:48:00", "last": "2024-01-27T00:58:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-01-27T00:58:16.653 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-01-27T00:00:00', 'created': '2024-01-27T00:47:40', 'first': '2024-01-27T00:48:00', 'last': '2024-01-27T00:58:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2024-01-27T00:58:16.653 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-27T00:58:16.653 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-27T00:58:16.654 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-27T00:58:16.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.787+0000 7f9589363700 1 Processor -- start 2024-01-27T00:58:16.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.787+0000 7f9589363700 1 -- start start 2024-01-27T00:58:16.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.787+0000 7f9589363700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f958406c3e0 0x7f9584069740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:58:16.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.787+0000 7f9589363700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f958415d8f0 con 0x7f95841596c0 2024-01-27T00:58:16.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.787+0000 7f9589363700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f958415dbd0 con 0x7f958415d460 2024-01-27T00:58:16.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.788+0000 7f9589363700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95840c98c0 con 0x7f958406c3e0 2024-01-27T00:58:16.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.788+0000 7f9582ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f95841596c0 0x7f958406bcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48108/0 (socket says 172.21.15.46:48108) 2024-01-27T00:58:16.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.788+0000 7f9582ffd700 1 -- 172.21.15.46:0/4140535543 learned_addr learned my addr 172.21.15.46:0/4140535543 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T00:58:16.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.788+0000 7f95837fe700 1 --2- 172.21.15.46:0/4140535543 >> v2:172.21.15.106:3300/0 conn(0x7f958406c3e0 0x7f9584069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:58:16.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.788+0000 7f9581ffb700 1 -- 172.21.15.46:0/4140535543 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038798088 0 0) 0x7f958415d8f0 con 0x7f95841596c0 2024-01-27T00:58:16.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.788+0000 7f9581ffb700 1 -- 172.21.15.46:0/4140535543 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f956c003570 con 0x7f95841596c0 2024-01-27T00:58:16.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.788+0000 7f9581ffb700 1 -- 172.21.15.46:0/4140535543 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291289021 0 0) 0x7f958415dbd0 con 0x7f958415d460 2024-01-27T00:58:16.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.788+0000 7f9581ffb700 1 -- 172.21.15.46:0/4140535543 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f958415d8f0 con 0x7f958415d460 2024-01-27T00:58:16.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.788+0000 7f95837fe700 1 -- 172.21.15.46:0/4140535543 >> v1:172.21.15.106:6789/0 conn(0x7f958415d460 legacy=0x7f9584069030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:58:16.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.788+0000 7f95837fe700 1 -- 172.21.15.46:0/4140535543 >> v1:172.21.15.46:6789/0 conn(0x7f95841596c0 legacy=0x7f958406bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:58:16.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.788+0000 7f9581ffb700 1 -- 172.21.15.46:0/4140535543 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3443392080 0 0) 0x7f956c003570 con 0x7f95841596c0 2024-01-27T00:58:16.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.788+0000 7f95837fe700 1 -- 172.21.15.46:0/4140535543 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958415deb0 con 0x7f958406c3e0 2024-01-27T00:58:16.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.788+0000 7f95837fe700 1 --2- 172.21.15.46:0/4140535543 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f958406c3e0 0x7f9584069740 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f957c009e60 tx=0x7f957c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63c0082499aeb69d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:58:16.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.789+0000 7f9581ffb700 1 -- 172.21.15.46:0/4140535543 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c00e070 con 0x7f958406c3e0 2024-01-27T00:58:16.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.789+0000 7f9581ffb700 1 -- 172.21.15.46:0/4140535543 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957c00b7a0 con 0x7f958406c3e0 2024-01-27T00:58:16.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.789+0000 7f9581ffb700 1 -- 172.21.15.46:0/4140535543 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9574001470 con 0x7f958406c3e0 2024-01-27T00:58:16.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.789+0000 7f9589363700 1 -- 172.21.15.46:0/4140535543 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f958406c3e0 msgr2=0x7f9584069740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:58:16.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.789+0000 7f9589363700 1 --2- 172.21.15.46:0/4140535543 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f958406c3e0 0x7f9584069740 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f957c009e60 tx=0x7f957c009780 comp rx=0 tx=0).stop 2024-01-27T00:58:16.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.789+0000 7f9589363700 1 -- 172.21.15.46:0/4140535543 shutdown_connections 2024-01-27T00:58:16.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.789+0000 7f9589363700 1 --2- 172.21.15.46:0/4140535543 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f958406c3e0 0x7f9584069740 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:58:16.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.789+0000 7f9589363700 1 -- 172.21.15.46:0/4140535543 >> 172.21.15.46:0/4140535543 conn(0x7f95840ce530 msgr2=0x7f95840ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:58:16.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.789+0000 7f9589363700 1 -- 172.21.15.46:0/4140535543 shutdown_connections 2024-01-27T00:58:16.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.789+0000 7f9589363700 1 -- 172.21.15.46:0/4140535543 wait complete. 2024-01-27T00:58:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.791+0000 7f9589363700 1 Processor -- start 2024-01-27T00:58:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.791+0000 7f9589363700 1 -- start start 2024-01-27T00:58:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f9589363700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f958406c3e0 0x7f9584169140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:58:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f9589363700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95841596c0 0x7f958416a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:58:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f9589363700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f958415d460 0x7f958415be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:58:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f9589363700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958415ae90 con 0x7f958415d460 2024-01-27T00:58:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f9589363700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958415a280 con 0x7f958406c3e0 2024-01-27T00:58:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f9589363700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f958415a560 con 0x7f95841596c0 2024-01-27T00:58:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f95837fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f958415d460 0x7f958415be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:58:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f95837fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f958415d460 0x7f958415be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35104/0 (socket says 172.21.15.46:35104) 2024-01-27T00:58:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f95837fe700 1 -- 172.21.15.46:0/2701103121 learned_addr learned my addr 172.21.15.46:0/2701103121 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T00:58:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f9582ffd700 1 --2- 172.21.15.46:0/2701103121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f958406c3e0 0x7f9584169140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:58:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f95827fc700 1 --2- 172.21.15.46:0/2701103121 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95841596c0 0x7f958416a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:58:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f95837fe700 1 -- 172.21.15.46:0/2701103121 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95841596c0 msgr2=0x7f958416a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:58:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f95837fe700 1 --2- 172.21.15.46:0/2701103121 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95841596c0 0x7f958416a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:58:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f95837fe700 1 -- 172.21.15.46:0/2701103121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f958406c3e0 msgr2=0x7f9584169140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:58:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f95837fe700 1 --2- 172.21.15.46:0/2701103121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f958406c3e0 0x7f9584169140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:58:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f95837fe700 1 -- 172.21.15.46:0/2701103121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957c009a20 con 0x7f958415d460 2024-01-27T00:58:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f95827fc700 1 --2- 172.21.15.46:0/2701103121 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95841596c0 0x7f958416a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T00:58:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f95837fe700 1 --2- 172.21.15.46:0/2701103121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f958415d460 0x7f958415be30 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f958406f790 tx=0x7f957c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:58:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.792+0000 7f956affd700 1 -- 172.21.15.46:0/2701103121 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c012cd0 con 0x7f958415d460 2024-01-27T00:58:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.793+0000 7f956affd700 1 -- 172.21.15.46:0/2701103121 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957c012e30 con 0x7f958415d460 2024-01-27T00:58:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.793+0000 7f956affd700 1 -- 172.21.15.46:0/2701103121 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c01e660 con 0x7f958415d460 2024-01-27T00:58:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.793+0000 7f9589363700 1 -- 172.21.15.46:0/2701103121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958415a840 con 0x7f958415d460 2024-01-27T00:58:16.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.793+0000 7f9589363700 1 -- 172.21.15.46:0/2701103121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9584169680 con 0x7f958415d460 2024-01-27T00:58:16.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.794+0000 7f9589363700 1 -- 172.21.15.46:0/2701103121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f958415b000 con 0x7f958415d460 2024-01-27T00:58:16.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.794+0000 7f956affd700 1 -- 172.21.15.46:0/2701103121 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 103) v1 ==== 84873+0+0 (secure 0 0 0) 0x7f957c01e9d0 con 0x7f958415d460 2024-01-27T00:58:16.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.795+0000 7f956affd700 1 --2- 172.21.15.46:0/2701103121 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f95700670a0 0x7f9570069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T00:58:16.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.795+0000 7f956affd700 1 -- 172.21.15.46:0/2701103121 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f957c086e70 con 0x7f958415d460 2024-01-27T00:58:16.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.795+0000 7f9582ffd700 1 --2- 172.21.15.46:0/2701103121 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f95700670a0 0x7f9570069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T00:58:16.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.795+0000 7f9582ffd700 1 --2- 172.21.15.46:0/2701103121 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f95700670a0 0x7f9570069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f95740011e0 tx=0x7f95740015e0 comp rx=0 tx=0).ready entity=mgr.8588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T00:58:16.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:16.799+0000 7f956affd700 1 -- 172.21.15.46:0/2701103121 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f957c053ad0 con 0x7f958415d460 2024-01-27T00:58:17.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:17.061+0000 7f9589363700 1 -- 172.21.15.46:0/2701103121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f958404c710 con 0x7f958415d460 2024-01-27T00:58:17.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:17.065+0000 7f956affd700 1 -- 172.21.15.46:0/2701103121 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v103) v1 ==== 74+0+163915 (secure 0 0 0) 0x7f957c016090 con 0x7f958415d460 2024-01-27T00:58:17.064 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T00:58:17.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:17.068+0000 7f9589363700 1 -- 172.21.15.46:0/2701103121 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f95700670a0 msgr2=0x7f9570069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:58:17.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:17.068+0000 7f9589363700 1 --2- 172.21.15.46:0/2701103121 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f95700670a0 0x7f9570069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f95740011e0 tx=0x7f95740015e0 comp rx=0 tx=0).stop 2024-01-27T00:58:17.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:17.068+0000 7f9589363700 1 -- 172.21.15.46:0/2701103121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f958415d460 msgr2=0x7f958415be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T00:58:17.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:17.068+0000 7f9589363700 1 --2- 172.21.15.46:0/2701103121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f958415d460 0x7f958415be30 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f958406f790 tx=0x7f957c017000 comp rx=0 tx=0).stop 2024-01-27T00:58:17.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:17.068+0000 7f9589363700 1 -- 172.21.15.46:0/2701103121 shutdown_connections 2024-01-27T00:58:17.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:17.068+0000 7f9589363700 1 --2- 172.21.15.46:0/2701103121 >> [v2:172.21.15.106:6836/43388,v1:172.21.15.106:6837/43388] conn(0x7f95700670a0 0x7f9570069550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:58:17.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:17.068+0000 7f9589363700 1 --2- 172.21.15.46:0/2701103121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f958415d460 0x7f958415be30 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:58:17.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:17.068+0000 7f9589363700 1 --2- 172.21.15.46:0/2701103121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f958406c3e0 0x7f9584169140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:58:17.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:17.068+0000 7f9589363700 1 --2- 172.21.15.46:0/2701103121 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f95841596c0 0x7f958416a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T00:58:17.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:17.068+0000 7f9589363700 1 -- 172.21.15.46:0/2701103121 >> 172.21.15.46:0/2701103121 conn(0x7f95840ce530 msgr2=0x7f95840cd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T00:58:17.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:17.068+0000 7f9589363700 1 -- 172.21.15.46:0/2701103121 shutdown_connections 2024-01-27T00:58:17.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T00:58:17.068+0000 7f9589363700 1 -- 172.21.15.46:0/2701103121 wait complete. 2024-01-27T00:58:17.081 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":103,"active_gid":8588,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":43388},{"type":"v1","addr":"172.21.15.106:6837","nonce":43388}]},"active_addr":"172.21.15.106:6837/43388","active_change":"2024-01-27T00:47:39.609902+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":9691,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":189,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":1872294075}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":1578971181}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":1532248390}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":683705968}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":211490639}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":273126119}]}]} 2024-01-27T00:58:17.086 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: x 2024-01-27T00:58:17.086 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-27T00:58:17.086 INFO:tasks.ceph.mgr.x:Stopping old one... 2024-01-27T00:58:17.086 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-27T00:58:17.087 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T00:58:17.117 INFO:tasks.ceph.mgr.x:Stopped 2024-01-27T00:58:17.117 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-27T00:58:17.160 INFO:tasks.ceph.mgr.x:Started 2024-01-27T00:58:17.532 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:17.532+0000 7fcce61971c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T00:58:17.609 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:17.610+0000 7fcce61971c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T00:58:17.681 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:17.681+0000 7fcce61971c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T00:58:17.747 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:17.747+0000 7fcce61971c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T00:58:17.904 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:17.903+0000 7fcce61971c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:58:18.038 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:18.038+0000 7fcce61971c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T00:58:18.104 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:18.104+0000 7fcce61971c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T00:58:18.223 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:18.223+0000 7fcce61971c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T00:58:18.294 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:18.294+0000 7fcce61971c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T00:58:18.433 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:18.433+0000 7fcce61971c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T00:58:18.559 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:18.560+0000 7fcce61971c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T00:58:18.873 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:18.873+0000 7fcce61971c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T00:58:19.082 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:19.082+0000 7fcce61971c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T00:58:19.305 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:19.305+0000 7fcce61971c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T00:58:19.375 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:19.375+0000 7fcce61971c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T00:58:19.973 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:19.974+0000 7fcce61971c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T00:58:20.047 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:20.047+0000 7fcce61971c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T00:58:20.169 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:20.169+0000 7fcce61971c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T00:58:20.664 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:20.664+0000 7fcce61971c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T00:58:20.812 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:20.812+0000 7fcce61971c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T00:58:20.960 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:20.960+0000 7fcce61971c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T00:58:21.653 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:21.653+0000 7fcce61971c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T00:58:21.871 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:21.872+0000 7fcce61971c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T00:58:21.945 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:21.945+0000 7fcce61971c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T00:58:22.103 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:22.103+0000 7fcce61971c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T00:58:22.271 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:22.271+0000 7fcc5c234700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:58:22.271 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:22.271+0000 7fcc5c234700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:58:22.271 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:22.271+0000 7fcc5c234700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:58:22.271 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:22.271+0000 7fcc5c234700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:58:22.271 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:22.271+0000 7fcc5c234700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:58:22.271 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:22.271+0000 7fcc5f23a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:58:22.271 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:22.271+0000 7fcc5f23a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:58:22.271 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:22.271+0000 7fcc5f23a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:58:22.271 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:22.271+0000 7fcc5f23a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:58:22.271 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:58:22.271+0000 7fcc5f23a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:58:42.407 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:58:42.411 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:59:00.270 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:59:00.269+0000 7fcc6728a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:59:00.270 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:59:00.270+0000 7fcc6728a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:59:00.270 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:59:00.270+0000 7fcc6728a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:59:00.270 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:59:00.270+0000 7fcc6728a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:59:00.270 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T00:59:00.270+0000 7fcc6728a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T00:59:12.572 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:59:12.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:59:42.735 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T00:59:42.739 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:00:12.849 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:00:12.853 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:00:43.014 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:00:43.017 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:01:13.162 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:01:13.165 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:01:43.268 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:01:43.271 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:02:13.423 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:02:13.429 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:02:43.580 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:02:43.583 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:03:13.677 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:03:13.680 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:03:17.161 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/test_restart 1m 2024-01-27T01:03:17.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.307+0000 7effa2dc7700 1 Processor -- start 2024-01-27T01:03:17.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.307+0000 7effa2dc7700 1 -- start start 2024-01-27T01:03:17.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.308+0000 7effa2dc7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff9c1599d0 0x7eff9c159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:17.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.308+0000 7effa2dc7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff9c15d8c0 0x7eff9c15dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:17.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.308+0000 7effa2dc7700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff9c0ab620 con 0x7eff9c0ab1c0 2024-01-27T01:03:17.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.308+0000 7effa2dc7700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff9c0b1a60 con 0x7eff9c1599d0 2024-01-27T01:03:17.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.308+0000 7effa2dc7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c15e230 con 0x7eff9c15d8c0 2024-01-27T01:03:17.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.308+0000 7effa0b63700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff9c1599d0 0x7eff9c159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:17.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.308+0000 7eff9bfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7eff9c0ab1c0 0x7eff9c0b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50822/0 (socket says 172.21.15.46:50822) 2024-01-27T01:03:17.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.308+0000 7eff9bfff700 1 -- 172.21.15.46:0/2378979854 learned_addr learned my addr 172.21.15.46:0/2378979854 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:17.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.308+0000 7effa0b63700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff9c1599d0 0x7eff9c159da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57040/0 (socket says 172.21.15.46:57040) 2024-01-27T01:03:17.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.308+0000 7effa1364700 1 --2- 172.21.15.46:0/2378979854 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff9c15d8c0 0x7eff9c15dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:17.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.308+0000 7effa0b63700 1 -- 172.21.15.46:0/2378979854 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff9c15d8c0 msgr2=0x7eff9c15dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:17.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.308+0000 7effa0b63700 1 --2- 172.21.15.46:0/2378979854 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff9c15d8c0 0x7eff9c15dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:17.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.308+0000 7effa0b63700 1 -- 172.21.15.46:0/2378979854 >> v1:172.21.15.46:6789/0 conn(0x7eff9c0ab1c0 legacy=0x7eff9c0b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:17.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.308+0000 7eff9b7fe700 1 -- 172.21.15.46:0/2378979854 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250323906 0 0) 0x7eff88001110 con 0x7eff9c0ab1c0 2024-01-27T01:03:17.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.308+0000 7effa0b63700 1 -- 172.21.15.46:0/2378979854 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9c0b11c0 con 0x7eff9c1599d0 2024-01-27T01:03:17.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.309+0000 7effa0b63700 1 --2- 172.21.15.46:0/2378979854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9c1599d0 0x7eff9c159da0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7eff900098d0 tx=0x7eff90009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e2f2c89dd9727f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:17.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.309+0000 7eff9b7fe700 1 -- 172.21.15.46:0/2378979854 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90005c10 con 0x7eff9c1599d0 2024-01-27T01:03:17.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.309+0000 7eff9b7fe700 1 -- 172.21.15.46:0/2378979854 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff90005d70 con 0x7eff9c1599d0 2024-01-27T01:03:17.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.309+0000 7eff9b7fe700 1 -- 172.21.15.46:0/2378979854 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90006280 con 0x7eff9c1599d0 2024-01-27T01:03:17.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.309+0000 7effa2dc7700 1 -- 172.21.15.46:0/2378979854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9c1599d0 msgr2=0x7eff9c159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:17.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.309+0000 7effa2dc7700 1 --2- 172.21.15.46:0/2378979854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9c1599d0 0x7eff9c159da0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7eff900098d0 tx=0x7eff90009120 comp rx=0 tx=0).stop 2024-01-27T01:03:17.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.309+0000 7effa2dc7700 1 -- 172.21.15.46:0/2378979854 shutdown_connections 2024-01-27T01:03:17.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.309+0000 7effa2dc7700 1 --2- 172.21.15.46:0/2378979854 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff9c15d8c0 0x7eff9c15dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:17.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.309+0000 7effa2dc7700 1 --2- 172.21.15.46:0/2378979854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9c1599d0 0x7eff9c159da0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:17.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.309+0000 7effa2dc7700 1 -- 172.21.15.46:0/2378979854 >> 172.21.15.46:0/2378979854 conn(0x7eff9c06f840 msgr2=0x7eff9c087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:17.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.309+0000 7effa2dc7700 1 -- 172.21.15.46:0/2378979854 shutdown_connections 2024-01-27T01:03:17.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.309+0000 7effa2dc7700 1 -- 172.21.15.46:0/2378979854 wait complete. 2024-01-27T01:03:17.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.311+0000 7effa2dc7700 1 Processor -- start 2024-01-27T01:03:17.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.311+0000 7effa2dc7700 1 -- start start 2024-01-27T01:03:17.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7effa2dc7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff9c0ab1c0 0x7eff9c14a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:17.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7effa2dc7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff9c1599d0 0x7eff9c148a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:17.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7effa2dc7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9c15d8c0 0x7eff9c148f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:17.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7effa2dc7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c14a390 con 0x7eff9c15d8c0 2024-01-27T01:03:17.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7effa2dc7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c14ad60 con 0x7eff9c1599d0 2024-01-27T01:03:17.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7effa2dc7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c147b70 con 0x7eff9c0ab1c0 2024-01-27T01:03:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7effa1364700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9c15d8c0 0x7eff9c148f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7effa1364700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9c15d8c0 0x7eff9c148f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57042/0 (socket says 172.21.15.46:57042) 2024-01-27T01:03:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7effa1364700 1 -- 172.21.15.46:0/3791392182 learned_addr learned my addr 172.21.15.46:0/3791392182 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7eff9bfff700 1 --2- 172.21.15.46:0/3791392182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff9c1599d0 0x7eff9c148a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7effa0b63700 1 --2- 172.21.15.46:0/3791392182 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff9c0ab1c0 0x7eff9c14a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7effa1364700 1 -- 172.21.15.46:0/3791392182 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff9c0ab1c0 msgr2=0x7eff9c14a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7effa1364700 1 --2- 172.21.15.46:0/3791392182 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff9c0ab1c0 0x7eff9c14a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7effa1364700 1 -- 172.21.15.46:0/3791392182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff9c1599d0 msgr2=0x7eff9c148a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7effa1364700 1 --2- 172.21.15.46:0/3791392182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff9c1599d0 0x7eff9c148a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7effa1364700 1 -- 172.21.15.46:0/3791392182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff900093c0 con 0x7eff9c15d8c0 2024-01-27T01:03:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7effa0b63700 1 --2- 172.21.15.46:0/3791392182 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff9c0ab1c0 0x7eff9c14a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:03:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.312+0000 7effa1364700 1 --2- 172.21.15.46:0/3791392182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9c15d8c0 0x7eff9c148f90 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7eff94009df0 tx=0x7eff9400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.313+0000 7eff98ff9700 1 -- 172.21.15.46:0/3791392182 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9400cb50 con 0x7eff9c15d8c0 2024-01-27T01:03:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.313+0000 7eff98ff9700 1 -- 172.21.15.46:0/3791392182 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9400f040 con 0x7eff9c15d8c0 2024-01-27T01:03:17.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.313+0000 7effa2dc7700 1 -- 172.21.15.46:0/3791392182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff9c1494d0 con 0x7eff9c15d8c0 2024-01-27T01:03:17.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.313+0000 7eff98ff9700 1 -- 172.21.15.46:0/3791392182 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff94014480 con 0x7eff9c15d8c0 2024-01-27T01:03:17.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.313+0000 7effa2dc7700 1 -- 172.21.15.46:0/3791392182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff9c149a60 con 0x7eff9c15d8c0 2024-01-27T01:03:17.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.314+0000 7effa2dc7700 1 -- 172.21.15.46:0/3791392182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff9c04c710 con 0x7eff9c15d8c0 2024-01-27T01:03:17.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.314+0000 7eff98ff9700 1 -- 172.21.15.46:0/3791392182 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 109) v1 ==== 84873+0+0 (secure 0 0 0) 0x7eff94014830 con 0x7eff9c15d8c0 2024-01-27T01:03:17.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.315+0000 7eff98ff9700 1 --2- 172.21.15.46:0/3791392182 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7eff800670a0 0x7eff80069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:17.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.315+0000 7eff98ff9700 1 -- 172.21.15.46:0/3791392182 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6977+0+0 (secure 0 0 0) 0x7eff94085630 con 0x7eff9c15d8c0 2024-01-27T01:03:17.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.315+0000 7effa0b63700 1 --2- 172.21.15.46:0/3791392182 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7eff800670a0 0x7eff80069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:17.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.315+0000 7effa0b63700 1 --2- 172.21.15.46:0/3791392182 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7eff800670a0 0x7eff80069550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7eff90009160 tx=0x7eff90016000 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:17.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.319+0000 7eff98ff9700 1 -- 172.21.15.46:0/3791392182 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7eff94052180 con 0x7eff9c15d8c0 2024-01-27T01:03:17.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.554+0000 7effa2dc7700 1 -- 172.21.15.46:0/3791392182 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7eff9c14a4d0 con 0x7eff800670a0 2024-01-27T01:03:17.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.617+0000 7eff98ff9700 1 -- 172.21.15.46:0/3791392182 <== mgr.8666 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7eff9c14a4d0 con 0x7eff800670a0 2024-01-27T01:03:17.616 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2024-01-27T01:03:17.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.619+0000 7effa2dc7700 1 -- 172.21.15.46:0/3791392182 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7eff800670a0 msgr2=0x7eff80069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:17.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.619+0000 7effa2dc7700 1 --2- 172.21.15.46:0/3791392182 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7eff800670a0 0x7eff80069550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7eff90009160 tx=0x7eff90016000 comp rx=0 tx=0).stop 2024-01-27T01:03:17.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.619+0000 7effa2dc7700 1 -- 172.21.15.46:0/3791392182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9c15d8c0 msgr2=0x7eff9c148f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:17.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.619+0000 7effa2dc7700 1 --2- 172.21.15.46:0/3791392182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9c15d8c0 0x7eff9c148f90 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7eff94009df0 tx=0x7eff9400b040 comp rx=0 tx=0).stop 2024-01-27T01:03:17.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.620+0000 7effa2dc7700 1 -- 172.21.15.46:0/3791392182 shutdown_connections 2024-01-27T01:03:17.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.620+0000 7effa2dc7700 1 --2- 172.21.15.46:0/3791392182 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7eff800670a0 0x7eff80069550 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:17.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.620+0000 7effa2dc7700 1 --2- 172.21.15.46:0/3791392182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff9c15d8c0 0x7eff9c148f90 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:17.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.620+0000 7effa2dc7700 1 --2- 172.21.15.46:0/3791392182 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7eff9c0ab1c0 0x7eff9c14a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:17.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.620+0000 7effa2dc7700 1 --2- 172.21.15.46:0/3791392182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff9c1599d0 0x7eff9c148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:17.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.620+0000 7effa2dc7700 1 -- 172.21.15.46:0/3791392182 >> 172.21.15.46:0/3791392182 conn(0x7eff9c06f840 msgr2=0x7eff9c0b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:17.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.620+0000 7effa2dc7700 1 -- 172.21.15.46:0/3791392182 shutdown_connections 2024-01-27T01:03:17.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.620+0000 7effa2dc7700 1 -- 172.21.15.46:0/3791392182 wait complete. 2024-01-27T01:03:17.633 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2024-01-27T01:03:17.633 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:17.633 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-01-27T01:03:17.660 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_48_00_UTC 2024-01-27T01:03:17.660 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_49_00_UTC 2024-01-27T01:03:17.660 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_50_00_UTC 2024-01-27T01:03:17.660 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_51_00_UTC 2024-01-27T01:03:17.660 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_52_00_UTC 2024-01-27T01:03:17.660 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_53_00_UTC 2024-01-27T01:03:17.660 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_54_00_UTC 2024-01-27T01:03:17.660 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_55_00_UTC 2024-01-27T01:03:17.661 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_56_00_UTC 2024-01-27T01:03:17.661 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_57_00_UTC 2024-01-27T01:03:17.661 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_58_00_UTC 2024-01-27T01:03:17.661 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_59_00_UTC 2024-01-27T01:03:17.661 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_00_00_UTC 2024-01-27T01:03:17.661 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_01_00_UTC 2024-01-27T01:03:17.661 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_02_00_UTC 2024-01-27T01:03:17.661 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_03_00_UTC 2024-01-27T01:03:17.662 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-27-00_48_00_UTC', 'scheduled-2024-01-27-00_49_00_UTC', 'scheduled-2024-01-27-00_50_00_UTC', 'scheduled-2024-01-27-00_51_00_UTC', 'scheduled-2024-01-27-00_52_00_UTC', 'scheduled-2024-01-27-00_53_00_UTC', 'scheduled-2024-01-27-00_54_00_UTC', 'scheduled-2024-01-27-00_55_00_UTC', 'scheduled-2024-01-27-00_56_00_UTC', 'scheduled-2024-01-27-00_57_00_UTC', 'scheduled-2024-01-27-00_58_00_UTC', 'scheduled-2024-01-27-00_59_00_UTC', 'scheduled-2024-01-27-01_00_00_UTC', 'scheduled-2024-01-27-01_01_00_UTC', 'scheduled-2024-01-27-01_02_00_UTC', 'scheduled-2024-01-27-01_03_00_UTC'] 2024-01-27T01:03:17.662 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-01-27T01:03:17.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.840+0000 7f7e20659700 1 Processor -- start 2024-01-27T01:03:17.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.840+0000 7f7e20659700 1 -- start start 2024-01-27T01:03:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.841+0000 7f7e20659700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7e18159cb0 0x7f7e180cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.841+0000 7f7e20659700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e180cf050 0x7f7e180af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.841+0000 7f7e20659700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e180cc100 con 0x7f7e180cf4a0 2024-01-27T01:03:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.841+0000 7f7e20659700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e180cb850 con 0x7f7e18159cb0 2024-01-27T01:03:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.841+0000 7f7e20659700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e180cbaf0 con 0x7f7e180cf050 2024-01-27T01:03:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.842+0000 7f7e1e3f5700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7e18159cb0 0x7f7e180cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.842+0000 7f7e1dbf4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e180cf050 0x7f7e180af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.842+0000 7f7e1e3f5700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7e18159cb0 0x7f7e180cc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:40630/0 (socket says 172.21.15.46:40630) 2024-01-27T01:03:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.842+0000 7f7e1dbf4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e180cf050 0x7f7e180af430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:56096/0 (socket says 172.21.15.46:56096) 2024-01-27T01:03:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.842+0000 7f7e1e3f5700 1 -- 172.21.15.46:0/2844480732 learned_addr learned my addr 172.21.15.46:0/2844480732 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:17.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.842+0000 7f7e1dbf4700 1 -- 172.21.15.46:0/2844480732 >> v2:172.21.15.106:3300/0 conn(0x7f7e18159cb0 msgr2=0x7f7e180cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:17.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.842+0000 7f7e1dbf4700 1 --2- 172.21.15.46:0/2844480732 >> v2:172.21.15.106:3300/0 conn(0x7f7e18159cb0 0x7f7e180cc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:17.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.842+0000 7f7e1dbf4700 1 -- 172.21.15.46:0/2844480732 >> v1:172.21.15.106:6789/0 conn(0x7f7e180cf4a0 legacy=0x7f7e180c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:17.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.842+0000 7f7e1dbf4700 1 -- 172.21.15.46:0/2844480732 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e180cfc80 con 0x7f7e180cf050 2024-01-27T01:03:17.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.842+0000 7f7e1d3f3700 1 -- 172.21.15.46:0/2844480732 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050769590 0 0) 0x7f7e180cc100 con 0x7f7e180cf4a0 2024-01-27T01:03:17.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.842+0000 7f7e1e3f5700 1 --2- 172.21.15.46:0/2844480732 >> v2:172.21.15.106:3300/0 conn(0x7f7e18159cb0 0x7f7e180cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:17.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.843+0000 7f7e1dbf4700 1 --2- 172.21.15.46:0/2844480732 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e180cf050 0x7f7e180af430 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0098e0 tx=0x7f7e0c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=992b4f6170263f57 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:17.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.843+0000 7f7e1d3f3700 1 -- 172.21.15.46:0/2844480732 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c00b640 con 0x7f7e180cf050 2024-01-27T01:03:17.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.843+0000 7f7e1d3f3700 1 -- 172.21.15.46:0/2844480732 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0c00b7a0 con 0x7f7e180cf050 2024-01-27T01:03:17.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.843+0000 7f7e1d3f3700 1 -- 172.21.15.46:0/2844480732 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c00bb40 con 0x7f7e180cf050 2024-01-27T01:03:17.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.843+0000 7f7e20659700 1 -- 172.21.15.46:0/2844480732 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e180cf050 msgr2=0x7f7e180af430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:17.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.843+0000 7f7e20659700 1 --2- 172.21.15.46:0/2844480732 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e180cf050 0x7f7e180af430 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0098e0 tx=0x7f7e0c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:17.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.843+0000 7f7e20659700 1 -- 172.21.15.46:0/2844480732 shutdown_connections 2024-01-27T01:03:17.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.843+0000 7f7e20659700 1 --2- 172.21.15.46:0/2844480732 >> v2:172.21.15.106:3300/0 conn(0x7f7e18159cb0 0x7f7e180cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:17.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.843+0000 7f7e20659700 1 --2- 172.21.15.46:0/2844480732 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e180cf050 0x7f7e180af430 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:17.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.843+0000 7f7e20659700 1 -- 172.21.15.46:0/2844480732 >> 172.21.15.46:0/2844480732 conn(0x7f7e18087f60 msgr2=0x7f7e1806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:17.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.843+0000 7f7e20659700 1 -- 172.21.15.46:0/2844480732 shutdown_connections 2024-01-27T01:03:17.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.843+0000 7f7e20659700 1 -- 172.21.15.46:0/2844480732 wait complete. 2024-01-27T01:03:17.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.845+0000 7f7e20659700 1 Processor -- start 2024-01-27T01:03:17.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.845+0000 7f7e20659700 1 -- start start 2024-01-27T01:03:17.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.845+0000 7f7e20659700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e180cf050 0x7f7e181555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:17.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.845+0000 7f7e20659700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e180cf4a0 0x7f7e1816d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:17.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.845+0000 7f7e20659700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e18159cb0 0x7f7e1809b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:17.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.845+0000 7f7e20659700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e18155b00 con 0x7f7e180cf4a0 2024-01-27T01:03:17.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.845+0000 7f7e20659700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1816d630 con 0x7f7e180cf050 2024-01-27T01:03:17.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.845+0000 7f7e20659700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e18169e80 con 0x7f7e18159cb0 2024-01-27T01:03:17.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.846+0000 7f7e1dbf4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e180cf4a0 0x7f7e1816d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:17.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.846+0000 7f7e1e3f5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e180cf050 0x7f7e181555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:17.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.846+0000 7f7e1dbf4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e180cf4a0 0x7f7e1816d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57058/0 (socket says 172.21.15.46:57058) 2024-01-27T01:03:17.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.846+0000 7f7e1dbf4700 1 -- 172.21.15.46:0/1388013459 learned_addr learned my addr 172.21.15.46:0/1388013459 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:17.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.846+0000 7f7e1ebf6700 1 --2- 172.21.15.46:0/1388013459 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e18159cb0 0x7f7e1809b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:17.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.846+0000 7f7e1dbf4700 1 -- 172.21.15.46:0/1388013459 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e18159cb0 msgr2=0x7f7e1809b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:17.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.846+0000 7f7e1dbf4700 1 --2- 172.21.15.46:0/1388013459 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e18159cb0 0x7f7e1809b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:17.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.846+0000 7f7e1dbf4700 1 -- 172.21.15.46:0/1388013459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e180cf050 msgr2=0x7f7e181555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:17.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.846+0000 7f7e1dbf4700 1 --2- 172.21.15.46:0/1388013459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e180cf050 0x7f7e181555c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:17.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.846+0000 7f7e1dbf4700 1 -- 172.21.15.46:0/1388013459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e14009480 con 0x7f7e180cf4a0 2024-01-27T01:03:17.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.846+0000 7f7e1dbf4700 1 --2- 172.21.15.46:0/1388013459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e180cf4a0 0x7f7e1816d0f0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0093f0 tx=0x7f7e0c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:17.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.846+0000 7f7e0a7fc700 1 -- 172.21.15.46:0/1388013459 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c01f070 con 0x7f7e180cf4a0 2024-01-27T01:03:17.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.846+0000 7f7e0a7fc700 1 -- 172.21.15.46:0/1388013459 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0c00b910 con 0x7f7e180cf4a0 2024-01-27T01:03:17.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.846+0000 7f7e20659700 1 -- 172.21.15.46:0/1388013459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0c009420 con 0x7f7e180cf4a0 2024-01-27T01:03:17.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.846+0000 7f7e0a7fc700 1 -- 172.21.15.46:0/1388013459 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c011420 con 0x7f7e180cf4a0 2024-01-27T01:03:17.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.846+0000 7f7e20659700 1 -- 172.21.15.46:0/1388013459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1816db40 con 0x7f7e180cf4a0 2024-01-27T01:03:17.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.847+0000 7f7e20659700 1 -- 172.21.15.46:0/1388013459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1816a030 con 0x7f7e180cf4a0 2024-01-27T01:03:17.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.848+0000 7f7e0a7fc700 1 -- 172.21.15.46:0/1388013459 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 109) v1 ==== 84873+0+0 (secure 0 0 0) 0x7f7e0c0117d0 con 0x7f7e180cf4a0 2024-01-27T01:03:17.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.848+0000 7f7e0a7fc700 1 --2- 172.21.15.46:0/1388013459 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7dfc0670a0 0x7f7dfc069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:17.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.848+0000 7f7e0a7fc700 1 -- 172.21.15.46:0/1388013459 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f7e0c085ec0 con 0x7f7e180cf4a0 2024-01-27T01:03:17.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.849+0000 7f7e1e3f5700 1 --2- 172.21.15.46:0/1388013459 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7dfc0670a0 0x7f7dfc069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:17.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.849+0000 7f7e1e3f5700 1 --2- 172.21.15.46:0/1388013459 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7dfc0670a0 0x7f7dfc069550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009e90 tx=0x7f7e14009220 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:17.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:17.853+0000 7f7e0a7fc700 1 -- 172.21.15.46:0/1388013459 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7e1816a030 con 0x7f7e180cf4a0 2024-01-27T01:03:18.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.085+0000 7f7e20659700 1 -- 172.21.15.46:0/1388013459 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7e18169ff0 con 0x7f7dfc0670a0 2024-01-27T01:03:18.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.093+0000 7f7e0a7fc700 1 -- 172.21.15.46:0/1388013459 <== mgr.8666 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+361 (secure 0 0 0) 0x7f7e18169ff0 con 0x7f7dfc0670a0 2024-01-27T01:03:18.092 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:18.092 INFO:teuthology.orchestra.run.smithi046.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-01-27T00:00:00", "created": "2024-01-27T00:47:40", "first": "2024-01-27T00:48:00", "last": "2024-01-27T01:03:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-01-27T01:03:18.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.095+0000 7f7e20659700 1 -- 172.21.15.46:0/1388013459 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7dfc0670a0 msgr2=0x7f7dfc069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:18.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.095+0000 7f7e20659700 1 --2- 172.21.15.46:0/1388013459 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7dfc0670a0 0x7f7dfc069550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009e90 tx=0x7f7e14009220 comp rx=0 tx=0).stop 2024-01-27T01:03:18.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.095+0000 7f7e20659700 1 -- 172.21.15.46:0/1388013459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e180cf4a0 msgr2=0x7f7e1816d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:18.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.095+0000 7f7e20659700 1 --2- 172.21.15.46:0/1388013459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e180cf4a0 0x7f7e1816d0f0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0093f0 tx=0x7f7e0c017000 comp rx=0 tx=0).stop 2024-01-27T01:03:18.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.095+0000 7f7e20659700 1 -- 172.21.15.46:0/1388013459 shutdown_connections 2024-01-27T01:03:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.095+0000 7f7e20659700 1 --2- 172.21.15.46:0/1388013459 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7dfc0670a0 0x7f7dfc069550 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.095+0000 7f7e20659700 1 --2- 172.21.15.46:0/1388013459 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7e18159cb0 0x7f7e1809b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.095+0000 7f7e20659700 1 --2- 172.21.15.46:0/1388013459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7e180cf050 0x7f7e181555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.095+0000 7f7e20659700 1 --2- 172.21.15.46:0/1388013459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e180cf4a0 0x7f7e1816d0f0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.095+0000 7f7e20659700 1 -- 172.21.15.46:0/1388013459 >> 172.21.15.46:0/1388013459 conn(0x7f7e18087f60 msgr2=0x7f7e18068ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.095+0000 7f7e20659700 1 -- 172.21.15.46:0/1388013459 shutdown_connections 2024-01-27T01:03:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.095+0000 7f7e20659700 1 -- 172.21.15.46:0/1388013459 wait complete. 2024-01-27T01:03:18.107 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-01-27T00:00:00", "created": "2024-01-27T00:47:40", "first": "2024-01-27T00:48:00", "last": "2024-01-27T01:03:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-01-27T01:03:18.108 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-01-27T00:00:00', 'created': '2024-01-27T00:47:40', 'first': '2024-01-27T00:48:00', 'last': '2024-01-27T01:03:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2024-01-27T01:03:18.108 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2024-01-27T01:03:18.108 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2024-01-27T01:03:18.108 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/test_restart 1m 2024-01-27T01:03:18.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.244+0000 7f132505b700 1 Processor -- start 2024-01-27T01:03:18.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.244+0000 7f132505b700 1 -- start start 2024-01-27T01:03:18.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.245+0000 7f132505b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f132015a520 0x7f1320156970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:18.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.245+0000 7f132505b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13200b0830 con 0x7f13200ae2f0 2024-01-27T01:03:18.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.245+0000 7f132505b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13200ad350 con 0x7f13201565a0 2024-01-27T01:03:18.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.245+0000 7f132505b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13200ae130 con 0x7f132015a520 2024-01-27T01:03:18.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.245+0000 7f131f59e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f13200ae2f0 0x7f13200ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50832/0 (socket says 172.21.15.46:50832) 2024-01-27T01:03:18.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.245+0000 7f1317fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f132015a520 0x7f1320156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:18.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.245+0000 7f131f59e700 1 -- 172.21.15.46:0/3328776260 learned_addr learned my addr 172.21.15.46:0/3328776260 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:18.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.246+0000 7f1317fff700 1 -- 172.21.15.46:0/3328776260 >> v1:172.21.15.106:6789/0 conn(0x7f13201565a0 legacy=0x7f13200acc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:18.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.246+0000 7f1317fff700 1 -- 172.21.15.46:0/3328776260 >> v1:172.21.15.46:6789/0 conn(0x7f13200ae2f0 legacy=0x7f13200ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:18.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.246+0000 7f1317fff700 1 -- 172.21.15.46:0/3328776260 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13200ad5b0 con 0x7f132015a520 2024-01-27T01:03:18.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.246+0000 7f131e59c700 1 -- 172.21.15.46:0/3328776260 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934410200 0 0) 0x7f13200b0830 con 0x7f13200ae2f0 2024-01-27T01:03:18.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.246+0000 7f1317fff700 1 --2- 172.21.15.46:0/3328776260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f132015a520 0x7f1320156970 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f13080098e0 tx=0x7f13080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89446429ad0888d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:18.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.246+0000 7f131e59c700 1 -- 172.21.15.46:0/3328776260 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1308009e80 con 0x7f132015a520 2024-01-27T01:03:18.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.246+0000 7f131e59c700 1 -- 172.21.15.46:0/3328776260 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130800b860 con 0x7f132015a520 2024-01-27T01:03:18.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.246+0000 7f131e59c700 1 -- 172.21.15.46:0/3328776260 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130800bbe0 con 0x7f132015a520 2024-01-27T01:03:18.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.247+0000 7f132505b700 1 -- 172.21.15.46:0/3328776260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f132015a520 msgr2=0x7f1320156970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:18.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.247+0000 7f132505b700 1 --2- 172.21.15.46:0/3328776260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f132015a520 0x7f1320156970 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f13080098e0 tx=0x7f13080091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:18.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.247+0000 7f132505b700 1 -- 172.21.15.46:0/3328776260 shutdown_connections 2024-01-27T01:03:18.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.247+0000 7f132505b700 1 --2- 172.21.15.46:0/3328776260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f132015a520 0x7f1320156970 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:18.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.247+0000 7f132505b700 1 -- 172.21.15.46:0/3328776260 >> 172.21.15.46:0/3328776260 conn(0x7f13200cdf80 msgr2=0x7f132006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:18.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.247+0000 7f132505b700 1 -- 172.21.15.46:0/3328776260 shutdown_connections 2024-01-27T01:03:18.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.247+0000 7f132505b700 1 -- 172.21.15.46:0/3328776260 wait complete. 2024-01-27T01:03:18.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.248+0000 7f132505b700 1 Processor -- start 2024-01-27T01:03:18.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.248+0000 7f132505b700 1 -- start start 2024-01-27T01:03:18.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.249+0000 7f132505b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13200ae2f0 0x7f13200b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:18.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.249+0000 7f132505b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13201565a0 0x7f13201555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:18.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.249+0000 7f132505b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f132015a520 0x7f132016e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.249+0000 7f132505b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1320159f90 con 0x7f13200ae2f0 2024-01-27T01:03:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.249+0000 7f132505b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13201508c0 con 0x7f13201565a0 2024-01-27T01:03:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.249+0000 7f132505b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1320159660 con 0x7f132015a520 2024-01-27T01:03:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.249+0000 7f131ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13200ae2f0 0x7f13200b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.249+0000 7f131ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13200ae2f0 0x7f13200b15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57066/0 (socket says 172.21.15.46:57066) 2024-01-27T01:03:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.249+0000 7f131ed9d700 1 -- 172.21.15.46:0/2926574705 learned_addr learned my addr 172.21.15.46:0/2926574705 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.249+0000 7f131f59e700 1 --2- 172.21.15.46:0/2926574705 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f132015a520 0x7f132016e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.249+0000 7f1317fff700 1 --2- 172.21.15.46:0/2926574705 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13201565a0 0x7f13201555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.249+0000 7f131ed9d700 1 -- 172.21.15.46:0/2926574705 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f132015a520 msgr2=0x7f132016e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.249+0000 7f131ed9d700 1 --2- 172.21.15.46:0/2926574705 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f132015a520 0x7f132016e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.249+0000 7f131ed9d700 1 -- 172.21.15.46:0/2926574705 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13201565a0 msgr2=0x7f13201555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.249+0000 7f131ed9d700 1 --2- 172.21.15.46:0/2926574705 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13201565a0 0x7f13201555c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.249+0000 7f131ed9d700 1 -- 172.21.15.46:0/2926574705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1308009420 con 0x7f13200ae2f0 2024-01-27T01:03:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.249+0000 7f131ed9d700 1 --2- 172.21.15.46:0/2926574705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13200ae2f0 0x7f13200b15c0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f131000c590 tx=0x7f131000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:18.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.250+0000 7f1316ffd700 1 -- 172.21.15.46:0/2926574705 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1310011590 con 0x7f13200ae2f0 2024-01-27T01:03:18.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.250+0000 7f1316ffd700 1 -- 172.21.15.46:0/2926574705 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13100116f0 con 0x7f13200ae2f0 2024-01-27T01:03:18.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.250+0000 7f1316ffd700 1 -- 172.21.15.46:0/2926574705 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1310011ab0 con 0x7f13200ae2f0 2024-01-27T01:03:18.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.250+0000 7f132505b700 1 -- 172.21.15.46:0/2926574705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1320159940 con 0x7f13200ae2f0 2024-01-27T01:03:18.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.250+0000 7f132505b700 1 -- 172.21.15.46:0/2926574705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13200b1b00 con 0x7f13200ae2f0 2024-01-27T01:03:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.251+0000 7f132505b700 1 -- 172.21.15.46:0/2926574705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f132016df30 con 0x7f13200ae2f0 2024-01-27T01:03:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.251+0000 7f1316ffd700 1 -- 172.21.15.46:0/2926574705 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 109) v1 ==== 84873+0+0 (secure 0 0 0) 0x7f131001b3b0 con 0x7f13200ae2f0 2024-01-27T01:03:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.252+0000 7f1316ffd700 1 --2- 172.21.15.46:0/2926574705 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f13000670a0 0x7f1300069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.252+0000 7f1316ffd700 1 -- 172.21.15.46:0/2926574705 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f13100841c0 con 0x7f13200ae2f0 2024-01-27T01:03:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.252+0000 7f1317fff700 1 --2- 172.21.15.46:0/2926574705 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f13000670a0 0x7f1300069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.253+0000 7f1317fff700 1 --2- 172.21.15.46:0/2926574705 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f13000670a0 0x7f1300069550 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f13080093f0 tx=0x7f1308017000 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:18.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.256+0000 7f1316ffd700 1 -- 172.21.15.46:0/2926574705 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1310050e20 con 0x7f13200ae2f0 2024-01-27T01:03:18.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.488+0000 7f132505b700 1 -- 172.21.15.46:0/2926574705 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f132004c710 con 0x7f13000670a0 2024-01-27T01:03:18.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.552+0000 7f1316ffd700 1 -- 172.21.15.46:0/2926574705 <== mgr.8666 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f132004c710 con 0x7f13000670a0 2024-01-27T01:03:18.551 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path /snap_test_dir1/test_restart 2024-01-27T01:03:18.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.555+0000 7f132505b700 1 -- 172.21.15.46:0/2926574705 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f13000670a0 msgr2=0x7f1300069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:18.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.555+0000 7f132505b700 1 --2- 172.21.15.46:0/2926574705 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f13000670a0 0x7f1300069550 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f13080093f0 tx=0x7f1308017000 comp rx=0 tx=0).stop 2024-01-27T01:03:18.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.555+0000 7f132505b700 1 -- 172.21.15.46:0/2926574705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13200ae2f0 msgr2=0x7f13200b15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:18.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.555+0000 7f132505b700 1 --2- 172.21.15.46:0/2926574705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13200ae2f0 0x7f13200b15c0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f131000c590 tx=0x7f131000c950 comp rx=0 tx=0).stop 2024-01-27T01:03:18.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.555+0000 7f132505b700 1 -- 172.21.15.46:0/2926574705 shutdown_connections 2024-01-27T01:03:18.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.555+0000 7f132505b700 1 --2- 172.21.15.46:0/2926574705 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f13000670a0 0x7f1300069550 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:18.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.555+0000 7f132505b700 1 --2- 172.21.15.46:0/2926574705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13200ae2f0 0x7f13200b15c0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:18.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.555+0000 7f132505b700 1 --2- 172.21.15.46:0/2926574705 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f132015a520 0x7f132016e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:18.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.555+0000 7f132505b700 1 --2- 172.21.15.46:0/2926574705 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f13201565a0 0x7f13201555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:18.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.555+0000 7f132505b700 1 -- 172.21.15.46:0/2926574705 >> 172.21.15.46:0/2926574705 conn(0x7f13200cdf80 msgr2=0x7f1320151db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:18.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.555+0000 7f132505b700 1 -- 172.21.15.46:0/2926574705 shutdown_connections 2024-01-27T01:03:18.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:18.555+0000 7f132505b700 1 -- 172.21.15.46:0/2926574705 wait complete. 2024-01-27T01:03:18.568 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2024-01-27T01:03:18.568 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:18.569 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-01-27T01:03:18.594 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_48_00_UTC 2024-01-27T01:03:18.594 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_49_00_UTC 2024-01-27T01:03:18.595 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_50_00_UTC 2024-01-27T01:03:18.595 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_51_00_UTC 2024-01-27T01:03:18.595 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_52_00_UTC 2024-01-27T01:03:18.595 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_53_00_UTC 2024-01-27T01:03:18.595 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_54_00_UTC 2024-01-27T01:03:18.595 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_55_00_UTC 2024-01-27T01:03:18.595 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_56_00_UTC 2024-01-27T01:03:18.595 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_57_00_UTC 2024-01-27T01:03:18.595 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_58_00_UTC 2024-01-27T01:03:18.595 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-00_59_00_UTC 2024-01-27T01:03:18.595 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_00_00_UTC 2024-01-27T01:03:18.595 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_01_00_UTC 2024-01-27T01:03:18.595 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_02_00_UTC 2024-01-27T01:03:18.595 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_03_00_UTC 2024-01-27T01:03:18.596 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_48_00_UTC 2024-01-27T01:03:18.596 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:18.596 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_48_00_UTC) 2024-01-27T01:03:18.665 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_49_00_UTC 2024-01-27T01:03:18.665 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:18.665 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_49_00_UTC) 2024-01-27T01:03:18.736 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_50_00_UTC 2024-01-27T01:03:18.736 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:18.736 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_50_00_UTC) 2024-01-27T01:03:18.806 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_51_00_UTC 2024-01-27T01:03:18.806 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:18.806 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_51_00_UTC) 2024-01-27T01:03:18.877 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_52_00_UTC 2024-01-27T01:03:18.877 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:18.877 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_52_00_UTC) 2024-01-27T01:03:18.947 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_53_00_UTC 2024-01-27T01:03:18.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:18.948 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_53_00_UTC) 2024-01-27T01:03:19.018 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_54_00_UTC 2024-01-27T01:03:19.018 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:19.018 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_54_00_UTC) 2024-01-27T01:03:19.087 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_55_00_UTC 2024-01-27T01:03:19.088 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:19.088 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_55_00_UTC) 2024-01-27T01:03:19.158 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_56_00_UTC 2024-01-27T01:03:19.159 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:19.159 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_56_00_UTC) 2024-01-27T01:03:19.229 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_57_00_UTC 2024-01-27T01:03:19.229 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:19.230 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_57_00_UTC) 2024-01-27T01:03:19.300 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_58_00_UTC 2024-01-27T01:03:19.300 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:19.300 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_58_00_UTC) 2024-01-27T01:03:19.370 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_59_00_UTC 2024-01-27T01:03:19.370 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:19.370 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-00_59_00_UTC) 2024-01-27T01:03:19.440 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-01_00_00_UTC 2024-01-27T01:03:19.440 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:19.440 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-01_00_00_UTC) 2024-01-27T01:03:19.510 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-01_01_00_UTC 2024-01-27T01:03:19.510 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:19.510 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-01_01_00_UTC) 2024-01-27T01:03:19.582 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-01_02_00_UTC 2024-01-27T01:03:19.582 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:19.582 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-01_02_00_UTC) 2024-01-27T01:03:19.654 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-01_03_00_UTC 2024-01-27T01:03:19.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:19.655 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-27-01_03_00_UTC) 2024-01-27T01:03:19.726 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:19.726 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2024-01-27T01:03:19.794 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-27T01:03:19.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.978+0000 7f1b80895700 1 Processor -- start 2024-01-27T01:03:19.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.978+0000 7f1b80895700 1 -- start start 2024-01-27T01:03:19.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.979+0000 7f1b80895700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b7c156940 0x7f1b7c06c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:19.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.979+0000 7f1b80895700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1b7c06f9a0 0x7f1b7c06eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:19.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.979+0000 7f1b80895700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7c0c8b90 con 0x7f1b7c156570 2024-01-27T01:03:19.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.979+0000 7f1b80895700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b7c06d0b0 con 0x7f1b7c06f9a0 2024-01-27T01:03:19.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.979+0000 7f1b80895700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c0c8e50 con 0x7f1b7c156940 2024-01-27T01:03:19.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.980+0000 7f1b7ad9d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1b7c06f9a0 0x7f1b7c06eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:19.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.980+0000 7f1b79d9b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b7c156940 0x7f1b7c06c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:19.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.980+0000 7f1b7ad9d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1b7c06f9a0 0x7f1b7c06eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:40672/0 (socket says 172.21.15.46:40672) 2024-01-27T01:03:19.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.980+0000 7f1b7ad9d700 1 -- 172.21.15.46:0/475433697 learned_addr learned my addr 172.21.15.46:0/475433697 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:19.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.980+0000 7f1b7ad9d700 1 -- 172.21.15.46:0/475433697 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b7c156940 msgr2=0x7f1b7c06c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:19.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.980+0000 7f1b7ad9d700 1 --2- 172.21.15.46:0/475433697 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b7c156940 0x7f1b7c06c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:19.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.980+0000 7f1b7ad9d700 1 -- 172.21.15.46:0/475433697 >> v1:172.21.15.106:6789/0 conn(0x7f1b7c156570 legacy=0x7f1b7c0c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:19.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.980+0000 7f1b7ad9d700 1 -- 172.21.15.46:0/475433697 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c0c4c40 con 0x7f1b7c06f9a0 2024-01-27T01:03:19.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.980+0000 7f1b7959a700 1 -- 172.21.15.46:0/475433697 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634900691 0 0) 0x7f1b7c0c8b90 con 0x7f1b7c156570 2024-01-27T01:03:19.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.980+0000 7f1b79d9b700 1 --2- 172.21.15.46:0/475433697 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b7c156940 0x7f1b7c06c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:19.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.980+0000 7f1b7ad9d700 1 --2- 172.21.15.46:0/475433697 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b7c06f9a0 0x7f1b7c06eb90 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009f50 tx=0x7f1b74009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc79bcbb46c74f38 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:19.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.981+0000 7f1b7959a700 1 -- 172.21.15.46:0/475433697 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7400e070 con 0x7f1b7c06f9a0 2024-01-27T01:03:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.981+0000 7f1b7959a700 1 -- 172.21.15.46:0/475433697 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7400b9c0 con 0x7f1b7c06f9a0 2024-01-27T01:03:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.981+0000 7f1b7959a700 1 -- 172.21.15.46:0/475433697 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b740092e0 con 0x7f1b7c06f9a0 2024-01-27T01:03:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.981+0000 7f1b80895700 1 -- 172.21.15.46:0/475433697 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b7c06f9a0 msgr2=0x7f1b7c06eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.981+0000 7f1b80895700 1 --2- 172.21.15.46:0/475433697 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b7c06f9a0 0x7f1b7c06eb90 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009f50 tx=0x7f1b74009840 comp rx=0 tx=0).stop 2024-01-27T01:03:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.981+0000 7f1b80895700 1 -- 172.21.15.46:0/475433697 shutdown_connections 2024-01-27T01:03:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.981+0000 7f1b80895700 1 --2- 172.21.15.46:0/475433697 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b7c06f9a0 0x7f1b7c06eb90 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.981+0000 7f1b80895700 1 --2- 172.21.15.46:0/475433697 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b7c156940 0x7f1b7c06c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.981+0000 7f1b80895700 1 -- 172.21.15.46:0/475433697 >> 172.21.15.46:0/475433697 conn(0x7f1b7c0cdf50 msgr2=0x7f1b7c0c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.981+0000 7f1b80895700 1 -- 172.21.15.46:0/475433697 shutdown_connections 2024-01-27T01:03:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.981+0000 7f1b80895700 1 -- 172.21.15.46:0/475433697 wait complete. 2024-01-27T01:03:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.983+0000 7f1b80895700 1 Processor -- start 2024-01-27T01:03:19.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.983+0000 7f1b80895700 1 -- start start 2024-01-27T01:03:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.983+0000 7f1b80895700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b7c06f9a0 0x7f1b7c1572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.983+0000 7f1b80895700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b7c156570 0x7f1b7c0c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.983+0000 7f1b80895700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b7c156940 0x7f1b7c0c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.984+0000 7f1b80895700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c0b0e10 con 0x7f1b7c156570 2024-01-27T01:03:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.984+0000 7f1b80895700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c0b0220 con 0x7f1b7c156940 2024-01-27T01:03:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.984+0000 7f1b80895700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c0b04e0 con 0x7f1b7c06f9a0 2024-01-27T01:03:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.984+0000 7f1b79d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b7c156570 0x7f1b7c0c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.984+0000 7f1b79d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b7c156570 0x7f1b7c0c6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57078/0 (socket says 172.21.15.46:57078) 2024-01-27T01:03:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.984+0000 7f1b79d9b700 1 -- 172.21.15.46:0/2251541009 learned_addr learned my addr 172.21.15.46:0/2251541009 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:19.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.984+0000 7f1b7ad9d700 1 --2- 172.21.15.46:0/2251541009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b7c156940 0x7f1b7c0c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.984+0000 7f1b7a59c700 1 --2- 172.21.15.46:0/2251541009 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b7c06f9a0 0x7f1b7c1572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.984+0000 7f1b79d9b700 1 -- 172.21.15.46:0/2251541009 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b7c06f9a0 msgr2=0x7f1b7c1572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.984+0000 7f1b79d9b700 1 --2- 172.21.15.46:0/2251541009 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b7c06f9a0 0x7f1b7c1572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.984+0000 7f1b79d9b700 1 -- 172.21.15.46:0/2251541009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b7c156940 msgr2=0x7f1b7c0c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.984+0000 7f1b79d9b700 1 --2- 172.21.15.46:0/2251541009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b7c156940 0x7f1b7c0c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.984+0000 7f1b79d9b700 1 -- 172.21.15.46:0/2251541009 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7000b6d0 con 0x7f1b7c156570 2024-01-27T01:03:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.984+0000 7f1b7ad9d700 1 --2- 172.21.15.46:0/2251541009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b7c156940 0x7f1b7c0c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.984+0000 7f1b79d9b700 1 --2- 172.21.15.46:0/2251541009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b7c156570 0x7f1b7c0c6100 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000ede0 tx=0x7f1b7000b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.985+0000 7f1b6a7fc700 1 -- 172.21.15.46:0/2251541009 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7000eee0 con 0x7f1b7c156570 2024-01-27T01:03:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.985+0000 7f1b6a7fc700 1 -- 172.21.15.46:0/2251541009 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7000dd30 con 0x7f1b7c156570 2024-01-27T01:03:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.985+0000 7f1b80895700 1 -- 172.21.15.46:0/2251541009 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b74009ae0 con 0x7f1b7c156570 2024-01-27T01:03:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.985+0000 7f1b6a7fc700 1 -- 172.21.15.46:0/2251541009 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7000dea0 con 0x7f1b7c156570 2024-01-27T01:03:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.985+0000 7f1b80895700 1 -- 172.21.15.46:0/2251541009 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7c0b0aa0 con 0x7f1b7c156570 2024-01-27T01:03:19.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.986+0000 7f1b80895700 1 -- 172.21.15.46:0/2251541009 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b7c0b0f80 con 0x7f1b7c156570 2024-01-27T01:03:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.986+0000 7f1b6a7fc700 1 -- 172.21.15.46:0/2251541009 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 109) v1 ==== 84873+0+0 (secure 0 0 0) 0x7f1b700108a0 con 0x7f1b7c156570 2024-01-27T01:03:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.987+0000 7f1b6a7fc700 1 --2- 172.21.15.46:0/2251541009 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b5c0670a0 0x7f1b5c069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.987+0000 7f1b6a7fc700 1 -- 172.21.15.46:0/2251541009 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f1b70085010 con 0x7f1b7c156570 2024-01-27T01:03:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.987+0000 7f1b7a59c700 1 --2- 172.21.15.46:0/2251541009 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b5c0670a0 0x7f1b5c069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:19.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.988+0000 7f1b7a59c700 1 --2- 172.21.15.46:0/2251541009 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b5c0670a0 0x7f1b5c069550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0cc2e0 tx=0x7f1b6c009040 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:19.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:19.992+0000 7f1b6a7fc700 1 -- 172.21.15.46:0/2251541009 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1b70051b60 con 0x7f1b7c156570 2024-01-27T01:03:20.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:20.264+0000 7f1b80895700 1 -- 172.21.15.46:0/2251541009 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f1b7c04c710 con 0x7f1b7c156570 2024-01-27T01:03:20.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:20.692+0000 7f1b6a7fc700 1 -- 172.21.15.46:0/2251541009 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 110) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f1b70083020 con 0x7f1b7c156570 2024-01-27T01:03:20.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:20.693+0000 7f1b6a7fc700 1 -- 172.21.15.46:0/2251541009 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v110) v1 ==== 93+0+0 (secure 0 0 0) 0x7f1b70051b60 con 0x7f1b7c156570 2024-01-27T01:03:20.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:20.698+0000 7f1b80895700 1 -- 172.21.15.46:0/2251541009 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b5c0670a0 msgr2=0x7f1b5c069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:20.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:20.698+0000 7f1b80895700 1 --2- 172.21.15.46:0/2251541009 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b5c0670a0 0x7f1b5c069550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0cc2e0 tx=0x7f1b6c009040 comp rx=0 tx=0).stop 2024-01-27T01:03:20.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:20.698+0000 7f1b80895700 1 -- 172.21.15.46:0/2251541009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b7c156570 msgr2=0x7f1b7c0c6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:20.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:20.698+0000 7f1b80895700 1 --2- 172.21.15.46:0/2251541009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b7c156570 0x7f1b7c0c6100 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000ede0 tx=0x7f1b7000b430 comp rx=0 tx=0).stop 2024-01-27T01:03:20.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:20.698+0000 7f1b80895700 1 -- 172.21.15.46:0/2251541009 shutdown_connections 2024-01-27T01:03:20.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:20.698+0000 7f1b80895700 1 --2- 172.21.15.46:0/2251541009 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b5c0670a0 0x7f1b5c069550 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:20.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:20.698+0000 7f1b80895700 1 --2- 172.21.15.46:0/2251541009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b7c156940 0x7f1b7c0c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:20.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:20.698+0000 7f1b80895700 1 --2- 172.21.15.46:0/2251541009 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b7c06f9a0 0x7f1b7c1572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:20.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:20.698+0000 7f1b80895700 1 --2- 172.21.15.46:0/2251541009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b7c156570 0x7f1b7c0c6100 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:20.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:20.698+0000 7f1b80895700 1 -- 172.21.15.46:0/2251541009 >> 172.21.15.46:0/2251541009 conn(0x7f1b7c0cdf50 msgr2=0x7f1b7c0c5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:20.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:20.699+0000 7f1b80895700 1 -- 172.21.15.46:0/2251541009 shutdown_connections 2024-01-27T01:03:20.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:20.699+0000 7f1b80895700 1 -- 172.21.15.46:0/2251541009 wait complete. 2024-01-27T01:03:20.732 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-27T01:03:20.732 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:03:20.793 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:03:20.833 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-27T01:03:20.833 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-27T01:03:20.834 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-27T01:03:20.834 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-27T01:03:20.871 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[101665]: fuse finished with error 0 and tester_r 0 2024-01-27T01:03:20.873 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T01:03:20.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:03:20.873 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-27T01:03:21.051 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T01:03:21.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:03:21.052 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-27T01:03:21.053 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:21.052+0000 7fd80323e1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:03:21.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:03:21.070 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-27T01:03:21.070 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T01:03:21.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:03:21.071 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-27T01:03:21.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:03:21.114 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-27T01:03:21.115 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T01:03:21.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:03:21.115 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-27T01:03:21.120 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:21.121+0000 7fa5b9ca21c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:03:21.132 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-27T01:03:21.132 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:21.132+0000 7fd80323e1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:03:21.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:03:21.134 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:03:21.203 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:21.203+0000 7fd80323e1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:03:21.210 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:21.211+0000 7fa5b9ca21c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:03:21.269 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:21.269+0000 7fd80323e1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:03:21.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.288+0000 7fbb3d426700 1 Processor -- start 2024-01-27T01:03:21.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.289+0000 7fbb3d426700 1 -- start start 2024-01-27T01:03:21.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.291+0000 7fbb3d426700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb38161780 0x7fbb3815dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:21.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.291+0000 7fbb3d426700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbb3806b4f0 0x7fbb380afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:21.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.291+0000 7fbb3d426700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb38161bf0 con 0x7fbb3815d860 2024-01-27T01:03:21.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.291+0000 7fbb3d426700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb380cc260 con 0x7fbb3806b4f0 2024-01-27T01:03:21.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.291+0000 7fbb3d426700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb38162770 con 0x7fbb38161780 2024-01-27T01:03:21.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.291+0000 7fbb367fc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb38161780 0x7fbb3815dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:21.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.291+0000 7fbb36ffd700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fbb3815d860 0x7fbb3806a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:43568/0 (socket says 172.21.15.46:43568) 2024-01-27T01:03:21.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.291+0000 7fbb36ffd700 1 -- 172.21.15.46:0/1796535674 learned_addr learned my addr 172.21.15.46:0/1796535674 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:21.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.291+0000 7fbb377fe700 1 --2- 172.21.15.46:0/1796535674 >> v2:172.21.15.46:3300/0 conn(0x7fbb3806b4f0 0x7fbb380afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:21.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.291+0000 7fbb367fc700 1 -- 172.21.15.46:0/1796535674 >> v2:172.21.15.46:3300/0 conn(0x7fbb3806b4f0 msgr2=0x7fbb380afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:21.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.291+0000 7fbb367fc700 1 --2- 172.21.15.46:0/1796535674 >> v2:172.21.15.46:3300/0 conn(0x7fbb3806b4f0 0x7fbb380afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.291+0000 7fbb367fc700 1 -- 172.21.15.46:0/1796535674 >> v1:172.21.15.106:6789/0 conn(0x7fbb3815d860 legacy=0x7fbb3806a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:21.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.292+0000 7fbb367fc700 1 -- 172.21.15.46:0/1796535674 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb38161ed0 con 0x7fbb38161780 2024-01-27T01:03:21.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.292+0000 7fbb35ffb700 1 -- 172.21.15.46:0/1796535674 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404934749 0 0) 0x7fbb38161bf0 con 0x7fbb3815d860 2024-01-27T01:03:21.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.292+0000 7fbb367fc700 1 --2- 172.21.15.46:0/1796535674 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb38161780 0x7fbb3815dc30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbb280098e0 tx=0x7fbb280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11274f2492dce14c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:21.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.292+0000 7fbb35ffb700 1 -- 172.21.15.46:0/1796535674 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2800b640 con 0x7fbb38161780 2024-01-27T01:03:21.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.292+0000 7fbb35ffb700 1 -- 172.21.15.46:0/1796535674 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2800b7a0 con 0x7fbb38161780 2024-01-27T01:03:21.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.292+0000 7fbb35ffb700 1 -- 172.21.15.46:0/1796535674 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2800bb40 con 0x7fbb38161780 2024-01-27T01:03:21.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.294+0000 7fbb3d426700 1 -- 172.21.15.46:0/1796535674 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb38161780 msgr2=0x7fbb3815dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:21.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.294+0000 7fbb3d426700 1 --2- 172.21.15.46:0/1796535674 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb38161780 0x7fbb3815dc30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbb280098e0 tx=0x7fbb280091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.294+0000 7fbb3d426700 1 -- 172.21.15.46:0/1796535674 shutdown_connections 2024-01-27T01:03:21.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.294+0000 7fbb3d426700 1 --2- 172.21.15.46:0/1796535674 >> v2:172.21.15.46:3300/0 conn(0x7fbb3806b4f0 0x7fbb380afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.294+0000 7fbb3d426700 1 --2- 172.21.15.46:0/1796535674 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb38161780 0x7fbb3815dc30 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.294+0000 7fbb3d426700 1 -- 172.21.15.46:0/1796535674 >> 172.21.15.46:0/1796535674 conn(0x7fbb380ccbc0 msgr2=0x7fbb380ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:21.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.294+0000 7fbb3d426700 1 -- 172.21.15.46:0/1796535674 shutdown_connections 2024-01-27T01:03:21.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.294+0000 7fbb3d426700 1 -- 172.21.15.46:0/1796535674 wait complete. 2024-01-27T01:03:21.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.296+0000 7fbb3d426700 1 Processor -- start 2024-01-27T01:03:21.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.296+0000 7fbb3d426700 1 -- start start 2024-01-27T01:03:21.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.297+0000 7fbb3d426700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb3815d860 0x7fbb380ae0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:21.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.297+0000 7fbb3d426700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb38161780 0x7fbb3815fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:21.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.297+0000 7fbb36ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb3815d860 0x7fbb380ae0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:21.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.297+0000 7fbb3d426700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb38160430 0x7fbb3815b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:21.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.297+0000 7fbb3d426700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb380ae5e0 con 0x7fbb38161780 2024-01-27T01:03:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.297+0000 7fbb3d426700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb380ae880 con 0x7fbb3815d860 2024-01-27T01:03:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.297+0000 7fbb3d426700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb380aeb60 con 0x7fbb38160430 2024-01-27T01:03:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.297+0000 7fbb36ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb3815d860 0x7fbb380ae0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:45324/0 (socket says 172.21.15.46:45324) 2024-01-27T01:03:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.297+0000 7fbb36ffd700 1 -- 172.21.15.46:0/593630056 learned_addr learned my addr 172.21.15.46:0/593630056 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.297+0000 7fbb367fc700 1 --2- 172.21.15.46:0/593630056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb38161780 0x7fbb3815fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.298+0000 7fbb36ffd700 1 -- 172.21.15.46:0/593630056 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb38160430 msgr2=0x7fbb3815b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.298+0000 7fbb377fe700 1 --2- 172.21.15.46:0/593630056 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb38160430 0x7fbb3815b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.298+0000 7fbb36ffd700 1 --2- 172.21.15.46:0/593630056 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb38160430 0x7fbb3815b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.298+0000 7fbb36ffd700 1 -- 172.21.15.46:0/593630056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb38161780 msgr2=0x7fbb3815fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.298+0000 7fbb36ffd700 1 --2- 172.21.15.46:0/593630056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb38161780 0x7fbb3815fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.298+0000 7fbb36ffd700 1 -- 172.21.15.46:0/593630056 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb28009420 con 0x7fbb3815d860 2024-01-27T01:03:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.298+0000 7fbb367fc700 1 --2- 172.21.15.46:0/593630056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb38161780 0x7fbb3815fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:03:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.298+0000 7fbb36ffd700 1 --2- 172.21.15.46:0/593630056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb3815d860 0x7fbb380ae0a0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c00c7e0 tx=0x7fbb2c007df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.298+0000 7fbb1effd700 1 -- 172.21.15.46:0/593630056 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c005b30 con 0x7fbb3815d860 2024-01-27T01:03:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.298+0000 7fbb1effd700 1 -- 172.21.15.46:0/593630056 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2c005c90 con 0x7fbb3815d860 2024-01-27T01:03:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.298+0000 7fbb3d426700 1 -- 172.21.15.46:0/593630056 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb3815f410 con 0x7fbb3815d860 2024-01-27T01:03:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.298+0000 7fbb1effd700 1 -- 172.21.15.46:0/593630056 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c007300 con 0x7fbb3815d860 2024-01-27T01:03:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.298+0000 7fbb3d426700 1 -- 172.21.15.46:0/593630056 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb3815f9d0 con 0x7fbb3815d860 2024-01-27T01:03:21.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.299+0000 7fbb3d426700 1 -- 172.21.15.46:0/593630056 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb3804c710 con 0x7fbb3815d860 2024-01-27T01:03:21.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.299+0000 7fbb1effd700 1 -- 172.21.15.46:0/593630056 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 110) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fbb2c01b400 con 0x7fbb3815d860 2024-01-27T01:03:21.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.300+0000 7fbb1effd700 1 --2- 172.21.15.46:0/593630056 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb24067050 0x7fbb24069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:21.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.300+0000 7fbb1effd700 1 -- 172.21.15.46:0/593630056 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 7065+0+0 (secure 0 0 0) 0x7fbb2c084790 con 0x7fbb3815d860 2024-01-27T01:03:21.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.300+0000 7fbb367fc700 1 -- 172.21.15.46:0/593630056 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb24067050 msgr2=0x7fbb24069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:21.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.300+0000 7fbb367fc700 1 --2- 172.21.15.46:0/593630056 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb24067050 0x7fbb24069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:03:21.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.304+0000 7fbb1effd700 1 -- 172.21.15.46:0/593630056 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbb2c051380 con 0x7fbb3815d860 2024-01-27T01:03:21.316 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:21.317+0000 7fa5b9ca21c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:03:21.392 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:21.393+0000 7fa5b9ca21c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:03:21.453 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:21.442+0000 7fd80323e1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:03:21.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.501+0000 7fbb367fc700 1 -- 172.21.15.46:0/593630056 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb24067050 msgr2=0x7fbb24069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:21.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.501+0000 7fbb367fc700 1 --2- 172.21.15.46:0/593630056 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb24067050 0x7fbb24069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:03:21.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.589+0000 7fbb3d426700 1 -- 172.21.15.46:0/593630056 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb380b0ba0 con 0x7fbb3815d860 2024-01-27T01:03:21.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.590+0000 7fbb1effd700 1 -- 172.21.15.46:0/593630056 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4601 (secure 0 0 0) 0x7fbb2c0549a0 con 0x7fbb3815d860 2024-01-27T01:03:21.589 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:21.589 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":97,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8435,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/4207115513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":4207115513},{"type":"v1","addr":"172.21.15.106:6835","nonce":4207115513}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91},{"gid":8438,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/282430549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":282430549},{"type":"v1","addr":"172.21.15.46:6834","nonce":282430549}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9508,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/361546207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":361546207},{"type":"v1","addr":"172.21.15.106:6834","nonce":361546207}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T00:47:19.926407+0000","modified":"2024-01-27T00:47:23.511130+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9828":{"gid":9828,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2544525048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2544525048},{"type":"v1","addr":"172.21.15.46:6835","nonce":2544525048}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2024-01-27T01:03:21.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.592+0000 7fbb3d426700 1 -- 172.21.15.46:0/593630056 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb24067050 msgr2=0x7fbb24069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:03:21.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.592+0000 7fbb3d426700 1 --2- 172.21.15.46:0/593630056 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb24067050 0x7fbb24069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.592+0000 7fbb3d426700 1 -- 172.21.15.46:0/593630056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb3815d860 msgr2=0x7fbb380ae0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:21.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.592+0000 7fbb3d426700 1 --2- 172.21.15.46:0/593630056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb3815d860 0x7fbb380ae0a0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c00c7e0 tx=0x7fbb2c007df0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.592+0000 7fbb3d426700 1 -- 172.21.15.46:0/593630056 shutdown_connections 2024-01-27T01:03:21.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.592+0000 7fbb3d426700 1 --2- 172.21.15.46:0/593630056 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb24067050 0x7fbb24069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.592+0000 7fbb3d426700 1 --2- 172.21.15.46:0/593630056 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb38160430 0x7fbb3815b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.592+0000 7fbb3d426700 1 --2- 172.21.15.46:0/593630056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb3815d860 0x7fbb380ae0a0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.592+0000 7fbb3d426700 1 --2- 172.21.15.46:0/593630056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb38161780 0x7fbb3815fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.592+0000 7fbb3d426700 1 -- 172.21.15.46:0/593630056 >> 172.21.15.46:0/593630056 conn(0x7fbb380ccbc0 msgr2=0x7fbb3806a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:21.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.592+0000 7fbb3d426700 1 -- 172.21.15.46:0/593630056 shutdown_connections 2024-01-27T01:03:21.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.592+0000 7fbb3d426700 1 -- 172.21.15.46:0/593630056 wait complete. 2024-01-27T01:03:21.593 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 97 2024-01-27T01:03:21.595 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:21.596+0000 7fa5b9ca21c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:03:21.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:03:21.608 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:21.608+0000 7fd80323e1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:03:21.674 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:21.674+0000 7fd80323e1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:03:21.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.743+0000 7fbc77981700 1 Processor -- start 2024-01-27T01:03:21.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.744+0000 7fbc77981700 1 -- start start 2024-01-27T01:03:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.745+0000 7fbc77981700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbc700b2b40 0x7fbc700ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.745+0000 7fbc77981700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc700aedd0 0x7fbc700cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.745+0000 7fbc77981700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7015a0b0 con 0x7fbc700ae960 2024-01-27T01:03:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.745+0000 7fbc77981700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc70099fc0 con 0x7fbc700b2b40 2024-01-27T01:03:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.745+0000 7fbc77981700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc700ae4d0 con 0x7fbc700aedd0 2024-01-27T01:03:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.745+0000 7fbc7571d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbc700b2b40 0x7fbc700ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.745+0000 7fbc7571d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbc700b2b40 0x7fbc700ad1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47096/0 (socket says 172.21.15.46:47096) 2024-01-27T01:03:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.745+0000 7fbc7571d700 1 -- 172.21.15.46:0/499816995 learned_addr learned my addr 172.21.15.46:0/499816995 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.745+0000 7fbc75f1e700 1 --2- 172.21.15.46:0/499816995 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc700aedd0 0x7fbc700cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.745+0000 7fbc7571d700 1 -- 172.21.15.46:0/499816995 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc700aedd0 msgr2=0x7fbc700cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.745+0000 7fbc6ffff700 1 -- 172.21.15.46:0/499816995 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925134655 0 0) 0x7fbc7015a0b0 con 0x7fbc700ae960 2024-01-27T01:03:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.745+0000 7fbc7571d700 1 --2- 172.21.15.46:0/499816995 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc700aedd0 0x7fbc700cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.745+0000 7fbc7571d700 1 -- 172.21.15.46:0/499816995 >> v1:172.21.15.106:6789/0 conn(0x7fbc700ae960 legacy=0x7fbc700ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.745+0000 7fbc7571d700 1 -- 172.21.15.46:0/499816995 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc700ad8c0 con 0x7fbc700b2b40 2024-01-27T01:03:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.745+0000 7fbc7571d700 1 --2- 172.21.15.46:0/499816995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc700b2b40 0x7fbc700ad1c0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fbc60009990 tx=0x7fbc60009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce557b5162d4833 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.746+0000 7fbc6ffff700 1 -- 172.21.15.46:0/499816995 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc60009e80 con 0x7fbc700b2b40 2024-01-27T01:03:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.746+0000 7fbc6ffff700 1 -- 172.21.15.46:0/499816995 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc60005c10 con 0x7fbc700b2b40 2024-01-27T01:03:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.746+0000 7fbc6ffff700 1 -- 172.21.15.46:0/499816995 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6000b650 con 0x7fbc700b2b40 2024-01-27T01:03:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.746+0000 7fbc77981700 1 -- 172.21.15.46:0/499816995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc700b2b40 msgr2=0x7fbc700ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.746+0000 7fbc77981700 1 --2- 172.21.15.46:0/499816995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc700b2b40 0x7fbc700ad1c0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fbc60009990 tx=0x7fbc60009220 comp rx=0 tx=0).stop 2024-01-27T01:03:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.746+0000 7fbc77981700 1 -- 172.21.15.46:0/499816995 shutdown_connections 2024-01-27T01:03:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.746+0000 7fbc77981700 1 --2- 172.21.15.46:0/499816995 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc700aedd0 0x7fbc700cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.746+0000 7fbc77981700 1 --2- 172.21.15.46:0/499816995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc700b2b40 0x7fbc700ad1c0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.746+0000 7fbc77981700 1 -- 172.21.15.46:0/499816995 >> 172.21.15.46:0/499816995 conn(0x7fbc70099a30 msgr2=0x7fbc7006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.746+0000 7fbc77981700 1 -- 172.21.15.46:0/499816995 shutdown_connections 2024-01-27T01:03:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.746+0000 7fbc77981700 1 -- 172.21.15.46:0/499816995 wait complete. 2024-01-27T01:03:21.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.748+0000 7fbc77981700 1 Processor -- start 2024-01-27T01:03:21.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.748+0000 7fbc77981700 1 -- start start 2024-01-27T01:03:21.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc77981700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc700ae960 0x7fbc701573c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:21.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc77981700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc700b2b40 0x7fbc700b5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:21.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc77981700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc700b3f20 0x7fbc700b1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:21.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc77981700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7015ce00 con 0x7fbc700b2b40 2024-01-27T01:03:21.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc77981700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc700b4420 con 0x7fbc700b3f20 2024-01-27T01:03:21.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc77981700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc700b4700 con 0x7fbc700ae960 2024-01-27T01:03:21.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc74f1c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc700b2b40 0x7fbc700b5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc74f1c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc700b2b40 0x7fbc700b5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47110/0 (socket says 172.21.15.46:47110) 2024-01-27T01:03:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc74f1c700 1 -- 172.21.15.46:0/1600048014 learned_addr learned my addr 172.21.15.46:0/1600048014 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc7571d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc700ae960 0x7fbc701573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc75f1e700 1 --2- 172.21.15.46:0/1600048014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc700b3f20 0x7fbc700b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc74f1c700 1 -- 172.21.15.46:0/1600048014 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc700ae960 msgr2=0x7fbc701573c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc74f1c700 1 --2- 172.21.15.46:0/1600048014 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc700ae960 0x7fbc701573c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc74f1c700 1 -- 172.21.15.46:0/1600048014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc700b3f20 msgr2=0x7fbc700b1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc74f1c700 1 --2- 172.21.15.46:0/1600048014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc700b3f20 0x7fbc700b1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc74f1c700 1 -- 172.21.15.46:0/1600048014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc60009480 con 0x7fbc700b2b40 2024-01-27T01:03:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc74f1c700 1 --2- 172.21.15.46:0/1600048014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc700b2b40 0x7fbc700b5ce0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400c530 tx=0x7fbc6400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.749+0000 7fbc6d7fa700 1 -- 172.21.15.46:0/1600048014 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc64011590 con 0x7fbc700b2b40 2024-01-27T01:03:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.750+0000 7fbc6d7fa700 1 -- 172.21.15.46:0/1600048014 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc64011710 con 0x7fbc700b2b40 2024-01-27T01:03:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.750+0000 7fbc77981700 1 -- 172.21.15.46:0/1600048014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc700b49e0 con 0x7fbc700b2b40 2024-01-27T01:03:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.750+0000 7fbc6d7fa700 1 -- 172.21.15.46:0/1600048014 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc64011ad0 con 0x7fbc700b2b40 2024-01-27T01:03:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.750+0000 7fbc77981700 1 -- 172.21.15.46:0/1600048014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc70157900 con 0x7fbc700b2b40 2024-01-27T01:03:21.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.751+0000 7fbc77981700 1 -- 172.21.15.46:0/1600048014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc7015d000 con 0x7fbc700b2b40 2024-01-27T01:03:21.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.751+0000 7fbc6d7fa700 1 -- 172.21.15.46:0/1600048014 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 110) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fbc6401b3b0 con 0x7fbc700b2b40 2024-01-27T01:03:21.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.752+0000 7fbc6d7fa700 1 --2- 172.21.15.46:0/1600048014 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbc5c067050 0x7fbc5c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:21.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.752+0000 7fbc7571d700 1 -- 172.21.15.46:0/1600048014 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbc5c067050 msgr2=0x7fbc5c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:21.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.752+0000 7fbc6d7fa700 1 -- 172.21.15.46:0/1600048014 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 7049+0+0 (secure 0 0 0) 0x7fbc64084080 con 0x7fbc700b2b40 2024-01-27T01:03:21.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.752+0000 7fbc7571d700 1 --2- 172.21.15.46:0/1600048014 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbc5c067050 0x7fbc5c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:03:21.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.756+0000 7fbc6d7fa700 1 -- 172.21.15.46:0/1600048014 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbc7015d000 con 0x7fbc700b2b40 2024-01-27T01:03:21.765 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:21.766+0000 7fa5b9ca21c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:03:21.797 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:21.798+0000 7fd80323e1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:03:21.842 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:21.843+0000 7fa5b9ca21c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:03:21.873 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:21.873+0000 7fd80323e1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:03:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.953+0000 7fbc7571d700 1 -- 172.21.15.46:0/1600048014 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbc5c067050 msgr2=0x7fbc5c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.953+0000 7fbc7571d700 1 --2- 172.21.15.46:0/1600048014 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbc5c067050 0x7fbc5c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:03:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.972+0000 7fbc77981700 1 -- 172.21.15.46:0/1600048014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc7004c710 con 0x7fbc700b2b40 2024-01-27T01:03:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.973+0000 7fbc6d7fa700 1 -- 172.21.15.46:0/1600048014 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) v1 ==== 74+0+17118 (secure 0 0 0) 0x7fbc64017020 con 0x7fbc700b2b40 2024-01-27T01:03:21.972 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:21.972 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":192,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:03:21.686956+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-01-27T00:47:17.642961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-01-27T00:47:18.897371+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":192,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/273126119":"2024-01-28T00:58:22.118992+0000","172.21.15.106:0/683705968":"2024-01-28T00:58:22.118992+0000","172.21.15.106:0/211490639":"2024-01-28T00:58:22.118992+0000","172.21.15.106:0/1013704950":"2024-01-28T00:47:39.609727+0000","172.21.15.106:0/243674269":"2024-01-28T00:47:39.609727+0000","172.21.15.106:0/1532248390":"2024-01-28T00:58:22.118992+0000","172.21.15.106:6837/43388":"2024-01-28T00:47:39.609727+0000","172.21.15.106:6836/43388":"2024-01-28T00:47:39.609727+0000","172.21.15.106:0/1578971181":"2024-01-28T00:58:22.118992+0000","172.21.15.106:0/2043045162":"2024-01-28T00:47:39.609727+0000","172.21.15.106:0/1872294075":"2024-01-28T00:58:22.118992+0000","172.21.15.106:0/14745894":"2024-01-28T00:47:39.609727+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:03:21.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.975+0000 7fbc77981700 1 -- 172.21.15.46:0/1600048014 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbc5c067050 msgr2=0x7fbc5c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:03:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.975+0000 7fbc77981700 1 --2- 172.21.15.46:0/1600048014 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbc5c067050 0x7fbc5c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.975+0000 7fbc77981700 1 -- 172.21.15.46:0/1600048014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc700b2b40 msgr2=0x7fbc700b5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.975+0000 7fbc77981700 1 --2- 172.21.15.46:0/1600048014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc700b2b40 0x7fbc700b5ce0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400c530 tx=0x7fbc6400c840 comp rx=0 tx=0).stop 2024-01-27T01:03:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.975+0000 7fbc77981700 1 -- 172.21.15.46:0/1600048014 shutdown_connections 2024-01-27T01:03:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.975+0000 7fbc77981700 1 --2- 172.21.15.46:0/1600048014 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbc5c067050 0x7fbc5c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.975+0000 7fbc77981700 1 --2- 172.21.15.46:0/1600048014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc700b3f20 0x7fbc700b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.975+0000 7fbc77981700 1 --2- 172.21.15.46:0/1600048014 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc700ae960 0x7fbc701573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.975+0000 7fbc77981700 1 --2- 172.21.15.46:0/1600048014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc700b2b40 0x7fbc700b5ce0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.975+0000 7fbc77981700 1 -- 172.21.15.46:0/1600048014 >> 172.21.15.46:0/1600048014 conn(0x7fbc70099a30 msgr2=0x7fbc70159460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.975+0000 7fbc77981700 1 -- 172.21.15.46:0/1600048014 shutdown_connections 2024-01-27T01:03:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:21.975+0000 7fbc77981700 1 -- 172.21.15.46:0/1600048014 wait complete. 2024-01-27T01:03:21.988 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:21.989+0000 7fa5b9ca21c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:03:21.989 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-27T00:47:17.642961+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '187', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2024-01-27T00:47:18.897371+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '192', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 192, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-27T01:03:21.989 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-27T01:03:22.017 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:22.017+0000 7fd80323e1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:03:22.077 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:22.078+0000 7fa5b9ca21c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:03:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.127+0000 7fbeb7940700 1 Processor -- start 2024-01-27T01:03:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.127+0000 7fbeb7940700 1 -- start start 2024-01-27T01:03:22.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.128+0000 7fbeb7940700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbeb00cc540 0x7fbeb00cc910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.128+0000 7fbeb7940700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbeb0068bb0 0x7fbeb0069020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.128+0000 7fbeb7940700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb015e0f0 con 0x7fbeb015d560 2024-01-27T01:03:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.128+0000 7fbeb7940700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeb00c8c80 con 0x7fbeb00cc540 2024-01-27T01:03:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.128+0000 7fbeb7940700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb0069920 con 0x7fbeb0068bb0 2024-01-27T01:03:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.128+0000 7fbeb5edd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbeb015d560 0x7fbeb015d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45730/0 (socket says 172.21.15.46:45730) 2024-01-27T01:03:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.128+0000 7fbeb5edd700 1 -- 172.21.15.46:0/2730350562 learned_addr learned my addr 172.21.15.46:0/2730350562 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.128+0000 7fbeb4edb700 1 --2- 172.21.15.46:0/2730350562 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbeb0068bb0 0x7fbeb0069020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.128+0000 7fbeb56dc700 1 --2- 172.21.15.46:0/2730350562 >> v2:172.21.15.106:3300/0 conn(0x7fbeb00cc540 0x7fbeb00cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.129+0000 7fbea7fff700 1 -- 172.21.15.46:0/2730350562 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99310877 0 0) 0x7fbeb015e0f0 con 0x7fbeb015d560 2024-01-27T01:03:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.129+0000 7fbea7fff700 1 -- 172.21.15.46:0/2730350562 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe9c003570 con 0x7fbeb015d560 2024-01-27T01:03:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.129+0000 7fbeb4edb700 1 -- 172.21.15.46:0/2730350562 >> v2:172.21.15.106:3300/0 conn(0x7fbeb00cc540 msgr2=0x7fbeb00cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.129+0000 7fbeb4edb700 1 --2- 172.21.15.46:0/2730350562 >> v2:172.21.15.106:3300/0 conn(0x7fbeb00cc540 0x7fbeb00cc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.129+0000 7fbeb4edb700 1 -- 172.21.15.46:0/2730350562 >> v1:172.21.15.46:6789/0 conn(0x7fbeb015d560 legacy=0x7fbeb015d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.129+0000 7fbeb4edb700 1 -- 172.21.15.46:0/2730350562 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb0069c00 con 0x7fbeb0068bb0 2024-01-27T01:03:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.129+0000 7fbeb56dc700 1 --2- 172.21.15.46:0/2730350562 >> v2:172.21.15.106:3300/0 conn(0x7fbeb00cc540 0x7fbeb00cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.129+0000 7fbeb4edb700 1 --2- 172.21.15.46:0/2730350562 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbeb0068bb0 0x7fbeb0069020 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fbea0009910 tx=0x7fbea00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=191bd90b821dfb3b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:22.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.129+0000 7fbea7fff700 1 -- 172.21.15.46:0/2730350562 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea0011630 con 0x7fbeb0068bb0 2024-01-27T01:03:22.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.129+0000 7fbea7fff700 1 -- 172.21.15.46:0/2730350562 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea0011790 con 0x7fbeb0068bb0 2024-01-27T01:03:22.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.129+0000 7fbea7fff700 1 -- 172.21.15.46:0/2730350562 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea0011b30 con 0x7fbeb0068bb0 2024-01-27T01:03:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.130+0000 7fbeb7940700 1 -- 172.21.15.46:0/2730350562 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbeb0068bb0 msgr2=0x7fbeb0069020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.130+0000 7fbeb7940700 1 --2- 172.21.15.46:0/2730350562 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbeb0068bb0 0x7fbeb0069020 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fbea0009910 tx=0x7fbea00091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.130+0000 7fbeb7940700 1 -- 172.21.15.46:0/2730350562 shutdown_connections 2024-01-27T01:03:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.130+0000 7fbeb7940700 1 --2- 172.21.15.46:0/2730350562 >> v2:172.21.15.106:3300/0 conn(0x7fbeb00cc540 0x7fbeb00cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.130+0000 7fbeb7940700 1 --2- 172.21.15.46:0/2730350562 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbeb0068bb0 0x7fbeb0069020 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.130+0000 7fbeb7940700 1 -- 172.21.15.46:0/2730350562 >> 172.21.15.46:0/2730350562 conn(0x7fbeb006bd90 msgr2=0x7fbeb006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.130+0000 7fbeb7940700 1 -- 172.21.15.46:0/2730350562 shutdown_connections 2024-01-27T01:03:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.130+0000 7fbeb7940700 1 -- 172.21.15.46:0/2730350562 wait complete. 2024-01-27T01:03:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.131+0000 7fbeb7940700 1 Processor -- start 2024-01-27T01:03:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.132+0000 7fbeb7940700 1 -- start start 2024-01-27T01:03:22.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.132+0000 7fbeb7940700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbeb0068bb0 0x7fbeb0170dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.132+0000 7fbeb7940700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb00cc540 0x7fbeb016df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.132+0000 7fbeb7940700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbeb015d560 0x7fbeb016e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.132+0000 7fbeb7940700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb016f000 con 0x7fbeb00cc540 2024-01-27T01:03:22.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.132+0000 7fbeb7940700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb0171300 con 0x7fbeb0068bb0 2024-01-27T01:03:22.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.132+0000 7fbeb7940700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb00b21e0 con 0x7fbeb015d560 2024-01-27T01:03:22.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.133+0000 7fbeb4edb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb00cc540 0x7fbeb016df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:22.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.133+0000 7fbeb4edb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb00cc540 0x7fbeb016df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47122/0 (socket says 172.21.15.46:47122) 2024-01-27T01:03:22.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.133+0000 7fbeb4edb700 1 -- 172.21.15.46:0/3911353192 learned_addr learned my addr 172.21.15.46:0/3911353192 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:22.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.133+0000 7fbeb5edd700 1 --2- 172.21.15.46:0/3911353192 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbeb015d560 0x7fbeb016e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:22.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.133+0000 7fbeb56dc700 1 --2- 172.21.15.46:0/3911353192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbeb0068bb0 0x7fbeb0170dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:22.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.133+0000 7fbeb4edb700 1 -- 172.21.15.46:0/3911353192 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbeb015d560 msgr2=0x7fbeb016e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:22.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.133+0000 7fbeb4edb700 1 --2- 172.21.15.46:0/3911353192 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbeb015d560 0x7fbeb016e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.133+0000 7fbeb4edb700 1 -- 172.21.15.46:0/3911353192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbeb0068bb0 msgr2=0x7fbeb0170dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:22.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.133+0000 7fbeb4edb700 1 --2- 172.21.15.46:0/3911353192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbeb0068bb0 0x7fbeb0170dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.133+0000 7fbeb4edb700 1 -- 172.21.15.46:0/3911353192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac009420 con 0x7fbeb00cc540 2024-01-27T01:03:22.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.133+0000 7fbeb4edb700 1 --2- 172.21.15.46:0/3911353192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb00cc540 0x7fbeb016df80 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fbea0009f00 tx=0x7fbea0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:22.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.133+0000 7fbea57fa700 1 -- 172.21.15.46:0/3911353192 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea0011900 con 0x7fbeb00cc540 2024-01-27T01:03:22.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.133+0000 7fbea57fa700 1 -- 172.21.15.46:0/3911353192 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea0011a60 con 0x7fbeb00cc540 2024-01-27T01:03:22.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.133+0000 7fbeb7940700 1 -- 172.21.15.46:0/3911353192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea0009420 con 0x7fbeb00cc540 2024-01-27T01:03:22.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.133+0000 7fbea57fa700 1 -- 172.21.15.46:0/3911353192 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea001a680 con 0x7fbeb00cc540 2024-01-27T01:03:22.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.133+0000 7fbeb7940700 1 -- 172.21.15.46:0/3911353192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb00b27a0 con 0x7fbeb00cc540 2024-01-27T01:03:22.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.135+0000 7fbeb7940700 1 -- 172.21.15.46:0/3911353192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb016f140 con 0x7fbeb00cc540 2024-01-27T01:03:22.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.136+0000 7fbea57fa700 1 -- 172.21.15.46:0/3911353192 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 110) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fbea001aa30 con 0x7fbeb00cc540 2024-01-27T01:03:22.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.136+0000 7fbea57fa700 1 --2- 172.21.15.46:0/3911353192 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbe94067050 0x7fbe94069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.136+0000 7fbeb56dc700 1 -- 172.21.15.46:0/3911353192 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbe94067050 msgr2=0x7fbe94069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:22.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.136+0000 7fbea57fa700 1 -- 172.21.15.46:0/3911353192 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 7049+0+0 (secure 0 0 0) 0x7fbea00866b0 con 0x7fbeb00cc540 2024-01-27T01:03:22.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.136+0000 7fbeb56dc700 1 --2- 172.21.15.46:0/3911353192 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbe94067050 0x7fbe94069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:03:22.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.141+0000 7fbea57fa700 1 -- 172.21.15.46:0/3911353192 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbea00532d0 con 0x7fbeb00cc540 2024-01-27T01:03:22.143 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:22.143+0000 7fd80323e1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:03:22.270 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:22.271+0000 7fa5b9ca21c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:03:22.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.337+0000 7fbeb56dc700 1 -- 172.21.15.46:0/3911353192 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbe94067050 msgr2=0x7fbe94069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:22.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.337+0000 7fbeb56dc700 1 --2- 172.21.15.46:0/3911353192 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbe94067050 0x7fbe94069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:03:22.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.392+0000 7fbeb7940700 1 -- 172.21.15.46:0/3911353192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbeb004c710 con 0x7fbeb00cc540 2024-01-27T01:03:22.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.393+0000 7fbea57fa700 1 -- 172.21.15.46:0/3911353192 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v192) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbea00568f0 con 0x7fbeb00cc540 2024-01-27T01:03:22.392 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-27T01:03:22.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.395+0000 7fbeb7940700 1 -- 172.21.15.46:0/3911353192 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbe94067050 msgr2=0x7fbe94069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:03:22.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.395+0000 7fbeb7940700 1 --2- 172.21.15.46:0/3911353192 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbe94067050 0x7fbe94069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.395+0000 7fbeb7940700 1 -- 172.21.15.46:0/3911353192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb00cc540 msgr2=0x7fbeb016df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:22.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.395+0000 7fbeb7940700 1 --2- 172.21.15.46:0/3911353192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb00cc540 0x7fbeb016df80 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fbea0009f00 tx=0x7fbea0017000 comp rx=0 tx=0).stop 2024-01-27T01:03:22.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.395+0000 7fbeb7940700 1 -- 172.21.15.46:0/3911353192 shutdown_connections 2024-01-27T01:03:22.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.395+0000 7fbeb7940700 1 --2- 172.21.15.46:0/3911353192 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbe94067050 0x7fbe94069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.395+0000 7fbeb7940700 1 --2- 172.21.15.46:0/3911353192 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbeb015d560 0x7fbeb016e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.395+0000 7fbeb7940700 1 --2- 172.21.15.46:0/3911353192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbeb0068bb0 0x7fbeb0170dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.395+0000 7fbeb7940700 1 --2- 172.21.15.46:0/3911353192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb00cc540 0x7fbeb016df80 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.395+0000 7fbeb7940700 1 -- 172.21.15.46:0/3911353192 >> 172.21.15.46:0/3911353192 conn(0x7fbeb006bd90 msgr2=0x7fbeb015c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:22.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.395+0000 7fbeb7940700 1 -- 172.21.15.46:0/3911353192 shutdown_connections 2024-01-27T01:03:22.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.395+0000 7fbeb7940700 1 -- 172.21.15.46:0/3911353192 wait complete. 2024-01-27T01:03:22.408 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-27T01:03:22.413 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:22.414+0000 7fa5b9ca21c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:03:22.455 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:22.455+0000 7fd80323e1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:03:22.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.547+0000 7f9b59a8b700 1 Processor -- start 2024-01-27T01:03:22.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.548+0000 7f9b59a8b700 1 -- start start 2024-01-27T01:03:22.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.548+0000 7f9b59a8b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9b54067af0 0x7f9b54067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.548+0000 7f9b59a8b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9b541597e0 0x7f9b54159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.548+0000 7f9b59a8b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b5406fdb0 con 0x7f9b5406ebe0 2024-01-27T01:03:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.548+0000 7f9b59a8b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b5406f1a0 con 0x7f9b54067af0 2024-01-27T01:03:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.548+0000 7f9b59a8b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b5406f480 con 0x7f9b541597e0 2024-01-27T01:03:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.549+0000 7f9b537fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9b54067af0 0x7f9b54067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.549+0000 7f9b537fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9b54067af0 0x7f9b54067ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47132/0 (socket says 172.21.15.46:47132) 2024-01-27T01:03:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.549+0000 7f9b537fe700 1 -- 172.21.15.46:0/2180450532 learned_addr learned my addr 172.21.15.46:0/2180450532 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.549+0000 7f9b53fff700 1 --2- 172.21.15.46:0/2180450532 >> v2:172.21.15.106:3300/0 conn(0x7f9b541597e0 0x7f9b54159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.549+0000 7f9b537fe700 1 -- 172.21.15.46:0/2180450532 >> v2:172.21.15.106:3300/0 conn(0x7f9b541597e0 msgr2=0x7f9b54159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.549+0000 7f9b537fe700 1 --2- 172.21.15.46:0/2180450532 >> v2:172.21.15.106:3300/0 conn(0x7f9b541597e0 0x7f9b54159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.549+0000 7f9b537fe700 1 -- 172.21.15.46:0/2180450532 >> v1:172.21.15.46:6789/0 conn(0x7f9b5406ebe0 legacy=0x7f9b540ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:22.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.549+0000 7f9b537fe700 1 -- 172.21.15.46:0/2180450532 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5406f760 con 0x7f9b54067af0 2024-01-27T01:03:22.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.549+0000 7f9b537fe700 1 --2- 172.21.15.46:0/2180450532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b54067af0 0x7f9b54067ec0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f9b5415eb20 tx=0x7f9b44009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ee34f6e7f66cf7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:22.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.549+0000 7f9b527fc700 1 -- 172.21.15.46:0/2180450532 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4400e070 con 0x7f9b54067af0 2024-01-27T01:03:22.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.549+0000 7f9b527fc700 1 -- 172.21.15.46:0/2180450532 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b4400ba20 con 0x7f9b54067af0 2024-01-27T01:03:22.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.549+0000 7f9b527fc700 1 -- 172.21.15.46:0/2180450532 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b440092e0 con 0x7f9b54067af0 2024-01-27T01:03:22.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.550+0000 7f9b59a8b700 1 -- 172.21.15.46:0/2180450532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b54067af0 msgr2=0x7f9b54067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:22.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.550+0000 7f9b59a8b700 1 --2- 172.21.15.46:0/2180450532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b54067af0 0x7f9b54067ec0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f9b5415eb20 tx=0x7f9b44009840 comp rx=0 tx=0).stop 2024-01-27T01:03:22.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.550+0000 7f9b59a8b700 1 -- 172.21.15.46:0/2180450532 shutdown_connections 2024-01-27T01:03:22.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.550+0000 7f9b59a8b700 1 --2- 172.21.15.46:0/2180450532 >> v2:172.21.15.106:3300/0 conn(0x7f9b541597e0 0x7f9b54159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.550+0000 7f9b59a8b700 1 --2- 172.21.15.46:0/2180450532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b54067af0 0x7f9b54067ec0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.550+0000 7f9b59a8b700 1 -- 172.21.15.46:0/2180450532 >> 172.21.15.46:0/2180450532 conn(0x7f9b540cdf80 msgr2=0x7f9b5406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:22.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.550+0000 7f9b59a8b700 1 -- 172.21.15.46:0/2180450532 shutdown_connections 2024-01-27T01:03:22.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.550+0000 7f9b59a8b700 1 -- 172.21.15.46:0/2180450532 wait complete. 2024-01-27T01:03:22.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.551+0000 7f9b59a8b700 1 Processor -- start 2024-01-27T01:03:22.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.552+0000 7f9b59a8b700 1 -- start start 2024-01-27T01:03:22.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.552+0000 7f9b59a8b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b54067af0 0x7f9b54158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.552+0000 7f9b59a8b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b5406ebe0 0x7f9b54144020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.552+0000 7f9b59a8b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b541597e0 0x7f9b5415bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.552+0000 7f9b59a8b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5415af60 con 0x7f9b541597e0 2024-01-27T01:03:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.552+0000 7f9b59a8b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5415b240 con 0x7f9b54067af0 2024-01-27T01:03:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.552+0000 7f9b59a8b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5415b520 con 0x7f9b5406ebe0 2024-01-27T01:03:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.552+0000 7f9b53fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b541597e0 0x7f9b5415bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.552+0000 7f9b53fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b541597e0 0x7f9b5415bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47148/0 (socket says 172.21.15.46:47148) 2024-01-27T01:03:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.552+0000 7f9b53fff700 1 -- 172.21.15.46:0/3541700840 learned_addr learned my addr 172.21.15.46:0/3541700840 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.553+0000 7f9b52ffd700 1 --2- 172.21.15.46:0/3541700840 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b5406ebe0 0x7f9b54144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.553+0000 7f9b537fe700 1 --2- 172.21.15.46:0/3541700840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b54067af0 0x7f9b54158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.553+0000 7f9b53fff700 1 -- 172.21.15.46:0/3541700840 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b5406ebe0 msgr2=0x7f9b54144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.553+0000 7f9b53fff700 1 --2- 172.21.15.46:0/3541700840 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b5406ebe0 0x7f9b54144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.553+0000 7f9b53fff700 1 -- 172.21.15.46:0/3541700840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b54067af0 msgr2=0x7f9b54158c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:22.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.553+0000 7f9b53fff700 1 --2- 172.21.15.46:0/3541700840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b54067af0 0x7f9b54158c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.553+0000 7f9b53fff700 1 -- 172.21.15.46:0/3541700840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b44009ae0 con 0x7f9b541597e0 2024-01-27T01:03:22.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.553+0000 7f9b53fff700 1 --2- 172.21.15.46:0/3541700840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b541597e0 0x7f9b5415bed0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00ea50 tx=0x7f9b4c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:22.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.553+0000 7f9b4b7fe700 1 -- 172.21.15.46:0/3541700840 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c00ee90 con 0x7f9b541597e0 2024-01-27T01:03:22.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.553+0000 7f9b4b7fe700 1 -- 172.21.15.46:0/3541700840 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b4c00c850 con 0x7f9b541597e0 2024-01-27T01:03:22.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.553+0000 7f9b4b7fe700 1 -- 172.21.15.46:0/3541700840 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c00cc10 con 0x7f9b541597e0 2024-01-27T01:03:22.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.553+0000 7f9b59a8b700 1 -- 172.21.15.46:0/3541700840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b5415b800 con 0x7f9b541597e0 2024-01-27T01:03:22.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.553+0000 7f9b59a8b700 1 -- 172.21.15.46:0/3541700840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b54144560 con 0x7f9b541597e0 2024-01-27T01:03:22.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.554+0000 7f9b59a8b700 1 -- 172.21.15.46:0/3541700840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b5415b380 con 0x7f9b541597e0 2024-01-27T01:03:22.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.555+0000 7f9b4b7fe700 1 -- 172.21.15.46:0/3541700840 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 110) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f9b4c013470 con 0x7f9b541597e0 2024-01-27T01:03:22.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.555+0000 7f9b4b7fe700 1 --2- 172.21.15.46:0/3541700840 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b34067050 0x7f9b34069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.555+0000 7f9b537fe700 1 -- 172.21.15.46:0/3541700840 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b34067050 msgr2=0x7f9b34069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:22.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.555+0000 7f9b4b7fe700 1 -- 172.21.15.46:0/3541700840 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 7049+0+0 (secure 0 0 0) 0x7f9b4c084e30 con 0x7f9b541597e0 2024-01-27T01:03:22.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.555+0000 7f9b537fe700 1 --2- 172.21.15.46:0/3541700840 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b34067050 0x7f9b34069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:03:22.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.560+0000 7f9b4b7fe700 1 -- 172.21.15.46:0/3541700840 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9b5415b380 con 0x7f9b541597e0 2024-01-27T01:03:22.666 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:22.666+0000 7fd80323e1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:03:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.756+0000 7f9b537fe700 1 -- 172.21.15.46:0/3541700840 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b34067050 msgr2=0x7f9b34069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.756+0000 7f9b537fe700 1 --2- 172.21.15.46:0/3541700840 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b34067050 0x7f9b34069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:03:22.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.766+0000 7f9b59a8b700 1 -- 172.21.15.46:0/3541700840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9b5415b380 con 0x7f9b541597e0 2024-01-27T01:03:22.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.766+0000 7f9b4b7fe700 1 -- 172.21.15.46:0/3541700840 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v192) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9b5415b380 con 0x7f9b541597e0 2024-01-27T01:03:22.765 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-27T01:03:22.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.768+0000 7f9b59a8b700 1 -- 172.21.15.46:0/3541700840 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b34067050 msgr2=0x7f9b34069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:03:22.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.768+0000 7f9b59a8b700 1 --2- 172.21.15.46:0/3541700840 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b34067050 0x7f9b34069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.768+0000 7f9b59a8b700 1 -- 172.21.15.46:0/3541700840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b541597e0 msgr2=0x7f9b5415bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:22.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.768+0000 7f9b59a8b700 1 --2- 172.21.15.46:0/3541700840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b541597e0 0x7f9b5415bed0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00ea50 tx=0x7f9b4c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.768+0000 7f9b59a8b700 1 -- 172.21.15.46:0/3541700840 shutdown_connections 2024-01-27T01:03:22.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.768+0000 7f9b59a8b700 1 --2- 172.21.15.46:0/3541700840 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b34067050 0x7f9b34069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.768+0000 7f9b59a8b700 1 --2- 172.21.15.46:0/3541700840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b541597e0 0x7f9b5415bed0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.768+0000 7f9b59a8b700 1 --2- 172.21.15.46:0/3541700840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b54067af0 0x7f9b54158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.768+0000 7f9b59a8b700 1 --2- 172.21.15.46:0/3541700840 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b5406ebe0 0x7f9b54144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.768+0000 7f9b59a8b700 1 -- 172.21.15.46:0/3541700840 >> 172.21.15.46:0/3541700840 conn(0x7f9b540cdf80 msgr2=0x7f9b540b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:22.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.769+0000 7f9b59a8b700 1 -- 172.21.15.46:0/3541700840 shutdown_connections 2024-01-27T01:03:22.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.769+0000 7f9b59a8b700 1 -- 172.21.15.46:0/3541700840 wait complete. 2024-01-27T01:03:22.782 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-27T01:03:22.786 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:22.787+0000 7fa5b9ca21c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:03:22.893 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:22.893+0000 7fd80323e1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:03:22.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.925+0000 7ff0cf0a8700 1 Processor -- start 2024-01-27T01:03:22.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.925+0000 7ff0cf0a8700 1 -- start start 2024-01-27T01:03:22.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.926+0000 7ff0cf0a8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff0c806ebe0 0x7ff0c8067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.926+0000 7ff0cf0a8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0c81597e0 0x7ff0c8159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.926+0000 7ff0cf0a8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c806fdb0 con 0x7ff0c8067af0 2024-01-27T01:03:22.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.926+0000 7ff0cf0a8700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c806f1a0 con 0x7ff0c806ebe0 2024-01-27T01:03:22.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.926+0000 7ff0cf0a8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c806f480 con 0x7ff0c81597e0 2024-01-27T01:03:22.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.926+0000 7ff0c7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff0c806ebe0 0x7ff0c8067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:22.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.927+0000 7ff0c7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff0c806ebe0 0x7ff0c8067ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47156/0 (socket says 172.21.15.46:47156) 2024-01-27T01:03:22.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.927+0000 7ff0c7fff700 1 -- 172.21.15.46:0/390619174 learned_addr learned my addr 172.21.15.46:0/390619174 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:22.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.927+0000 7ff0cd645700 1 --2- 172.21.15.46:0/390619174 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0c81597e0 0x7ff0c8159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:22.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.927+0000 7ff0c7fff700 1 -- 172.21.15.46:0/390619174 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0c81597e0 msgr2=0x7ff0c8159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:22.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.927+0000 7ff0c7fff700 1 --2- 172.21.15.46:0/390619174 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0c81597e0 0x7ff0c8159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.927+0000 7ff0c7fff700 1 -- 172.21.15.46:0/390619174 >> v1:172.21.15.46:6789/0 conn(0x7ff0c8067af0 legacy=0x7ff0c80ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:22.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.927+0000 7ff0c7fff700 1 -- 172.21.15.46:0/390619174 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c806f760 con 0x7ff0c806ebe0 2024-01-27T01:03:22.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.927+0000 7ff0c7fff700 1 --2- 172.21.15.46:0/390619174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c806ebe0 0x7ff0c8067ec0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7ff0b40099b0 tx=0x7ff0b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f7e0514f745d93 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:22.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.927+0000 7ff0c77fe700 1 -- 172.21.15.46:0/390619174 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b4009ea0 con 0x7ff0c806ebe0 2024-01-27T01:03:22.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.927+0000 7ff0c77fe700 1 -- 172.21.15.46:0/390619174 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b400b860 con 0x7ff0c806ebe0 2024-01-27T01:03:22.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.927+0000 7ff0c77fe700 1 -- 172.21.15.46:0/390619174 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b400bc20 con 0x7ff0c806ebe0 2024-01-27T01:03:22.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.928+0000 7ff0cf0a8700 1 -- 172.21.15.46:0/390619174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c806ebe0 msgr2=0x7ff0c8067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:22.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.928+0000 7ff0cf0a8700 1 --2- 172.21.15.46:0/390619174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c806ebe0 0x7ff0c8067ec0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7ff0b40099b0 tx=0x7ff0b4009250 comp rx=0 tx=0).stop 2024-01-27T01:03:22.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.928+0000 7ff0cf0a8700 1 -- 172.21.15.46:0/390619174 shutdown_connections 2024-01-27T01:03:22.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.928+0000 7ff0cf0a8700 1 --2- 172.21.15.46:0/390619174 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0c81597e0 0x7ff0c8159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.928+0000 7ff0cf0a8700 1 --2- 172.21.15.46:0/390619174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c806ebe0 0x7ff0c8067ec0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.928+0000 7ff0cf0a8700 1 -- 172.21.15.46:0/390619174 >> 172.21.15.46:0/390619174 conn(0x7ff0c80cdf80 msgr2=0x7ff0c806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:22.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.928+0000 7ff0cf0a8700 1 -- 172.21.15.46:0/390619174 shutdown_connections 2024-01-27T01:03:22.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.928+0000 7ff0cf0a8700 1 -- 172.21.15.46:0/390619174 wait complete. 2024-01-27T01:03:22.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.929+0000 7ff0cf0a8700 1 Processor -- start 2024-01-27T01:03:22.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.930+0000 7ff0cf0a8700 1 -- start start 2024-01-27T01:03:22.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.930+0000 7ff0cf0a8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0c8067af0 0x7ff0c80abfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.930+0000 7ff0cf0a8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0c806ebe0 0x7ff0c80aaba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.930+0000 7ff0cf0a8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c81597e0 0x7ff0c816a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.930+0000 7ff0cf0a8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c8156dc0 con 0x7ff0c81597e0 2024-01-27T01:03:22.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.930+0000 7ff0cf0a8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c81561b0 con 0x7ff0c806ebe0 2024-01-27T01:03:22.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.930+0000 7ff0cf0a8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c8156490 con 0x7ff0c8067af0 2024-01-27T01:03:22.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.931+0000 7ff0cd645700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c81597e0 0x7ff0c816a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:22.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.931+0000 7ff0cd645700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c81597e0 0x7ff0c816a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47168/0 (socket says 172.21.15.46:47168) 2024-01-27T01:03:22.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.931+0000 7ff0cce44700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0c8067af0 0x7ff0c80abfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:22.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.931+0000 7ff0cd645700 1 -- 172.21.15.46:0/675635091 learned_addr learned my addr 172.21.15.46:0/675635091 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:22.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.931+0000 7ff0c7fff700 1 --2- 172.21.15.46:0/675635091 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0c806ebe0 0x7ff0c80aaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:22.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.931+0000 7ff0cce44700 1 -- 172.21.15.46:0/675635091 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0c806ebe0 msgr2=0x7ff0c80aaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:22.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.931+0000 7ff0cce44700 1 --2- 172.21.15.46:0/675635091 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0c806ebe0 0x7ff0c80aaba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.931+0000 7ff0cce44700 1 -- 172.21.15.46:0/675635091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c81597e0 msgr2=0x7ff0c816a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:22.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.931+0000 7ff0cce44700 1 --2- 172.21.15.46:0/675635091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c81597e0 0x7ff0c816a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:22.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.931+0000 7ff0cce44700 1 -- 172.21.15.46:0/675635091 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b40094f0 con 0x7ff0c8067af0 2024-01-27T01:03:22.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.931+0000 7ff0cd645700 1 --2- 172.21.15.46:0/675635091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c81597e0 0x7ff0c816a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:22.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.931+0000 7ff0cce44700 1 --2- 172.21.15.46:0/675635091 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0c8067af0 0x7ff0c80abfd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc00c490 tx=0x7ff0bc007ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:22.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.931+0000 7ff0c4ff9700 1 -- 172.21.15.46:0/675635091 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc00ce40 con 0x7ff0c8067af0 2024-01-27T01:03:22.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.931+0000 7ff0c4ff9700 1 -- 172.21.15.46:0/675635091 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0bc011590 con 0x7ff0c8067af0 2024-01-27T01:03:22.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.931+0000 7ff0cf0a8700 1 -- 172.21.15.46:0/675635091 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c8156770 con 0x7ff0c8067af0 2024-01-27T01:03:22.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.931+0000 7ff0c4ff9700 1 -- 172.21.15.46:0/675635091 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc011950 con 0x7ff0c8067af0 2024-01-27T01:03:22.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.931+0000 7ff0cf0a8700 1 -- 172.21.15.46:0/675635091 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c80a9dd0 con 0x7ff0c8067af0 2024-01-27T01:03:22.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.932+0000 7ff0cf0a8700 1 -- 172.21.15.46:0/675635091 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c80c9460 con 0x7ff0c8067af0 2024-01-27T01:03:22.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.933+0000 7ff0c4ff9700 1 -- 172.21.15.46:0/675635091 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 110) v1 ==== 84856+0+0 (secure 0 0 0) 0x7ff0bc011cc0 con 0x7ff0c8067af0 2024-01-27T01:03:22.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.933+0000 7ff0c4ff9700 1 --2- 172.21.15.46:0/675635091 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff0ac067050 0x7ff0ac069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:22.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.933+0000 7ff0c7fff700 1 -- 172.21.15.46:0/675635091 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff0ac067050 msgr2=0x7ff0ac069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:22.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.933+0000 7ff0c4ff9700 1 -- 172.21.15.46:0/675635091 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 7049+0+0 (secure 0 0 0) 0x7ff0bc083ee0 con 0x7ff0c8067af0 2024-01-27T01:03:22.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.933+0000 7ff0c7fff700 1 --2- 172.21.15.46:0/675635091 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff0ac067050 0x7ff0ac069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:03:22.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:22.938+0000 7ff0c4ff9700 1 -- 172.21.15.46:0/675635091 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff0bc050b00 con 0x7ff0c8067af0 2024-01-27T01:03:22.964 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:22.964+0000 7fd80323e1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:03:23.036 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:23.037+0000 7fa5b9ca21c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:03:23.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.134+0000 7ff0c7fff700 1 -- 172.21.15.46:0/675635091 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff0ac067050 msgr2=0x7ff0ac069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:23.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.134+0000 7ff0c7fff700 1 --2- 172.21.15.46:0/675635091 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff0ac067050 0x7ff0ac069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:03:23.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.143+0000 7ff0cf0a8700 1 -- 172.21.15.46:0/675635091 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff0c804c710 con 0x7ff0c8067af0 2024-01-27T01:03:23.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.143+0000 7ff0c4ff9700 1 -- 172.21.15.46:0/675635091 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v192) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff0bc017070 con 0x7ff0c8067af0 2024-01-27T01:03:23.142 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-27T01:03:23.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.146+0000 7ff0cf0a8700 1 -- 172.21.15.46:0/675635091 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff0ac067050 msgr2=0x7ff0ac069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:03:23.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.146+0000 7ff0cf0a8700 1 --2- 172.21.15.46:0/675635091 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff0ac067050 0x7ff0ac069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.146+0000 7ff0cf0a8700 1 -- 172.21.15.46:0/675635091 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0c8067af0 msgr2=0x7ff0c80abfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:23.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.146+0000 7ff0cf0a8700 1 --2- 172.21.15.46:0/675635091 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0c8067af0 0x7ff0c80abfd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc00c490 tx=0x7ff0bc007ec0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.146+0000 7ff0cf0a8700 1 -- 172.21.15.46:0/675635091 shutdown_connections 2024-01-27T01:03:23.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.146+0000 7ff0cf0a8700 1 --2- 172.21.15.46:0/675635091 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff0ac067050 0x7ff0ac069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.146+0000 7ff0cf0a8700 1 --2- 172.21.15.46:0/675635091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c81597e0 0x7ff0c816a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.146+0000 7ff0cf0a8700 1 --2- 172.21.15.46:0/675635091 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0c8067af0 0x7ff0c80abfd0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.146+0000 7ff0cf0a8700 1 --2- 172.21.15.46:0/675635091 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0c806ebe0 0x7ff0c80aaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.146+0000 7ff0cf0a8700 1 -- 172.21.15.46:0/675635091 >> 172.21.15.46:0/675635091 conn(0x7ff0c80cdf80 msgr2=0x7ff0c80af3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:23.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.146+0000 7ff0cf0a8700 1 -- 172.21.15.46:0/675635091 shutdown_connections 2024-01-27T01:03:23.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.146+0000 7ff0cf0a8700 1 -- 172.21.15.46:0/675635091 wait complete. 2024-01-27T01:03:23.159 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:03:23.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.296+0000 7f52d6f3b700 1 Processor -- start 2024-01-27T01:03:23.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.296+0000 7f52d6f3b700 1 -- start start 2024-01-27T01:03:23.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.297+0000 7f52d6f3b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f52d01561d0 0x7f52d00ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:23.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.297+0000 7f52d6f3b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52d006cbb0 0x7f52d01565a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:23.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.297+0000 7f52d6f3b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d00b0760 con 0x7f52d00b0280 2024-01-27T01:03:23.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.297+0000 7f52d6f3b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d00b0a40 con 0x7f52d01561d0 2024-01-27T01:03:23.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.297+0000 7f52d6f3b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00b0d20 con 0x7f52d006cbb0 2024-01-27T01:03:23.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.297+0000 7f52d4cd7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f52d01561d0 0x7f52d00ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:23.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.297+0000 7f52d4cd7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f52d01561d0 0x7f52d00ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47174/0 (socket says 172.21.15.46:47174) 2024-01-27T01:03:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.298+0000 7f52d4cd7700 1 -- 172.21.15.46:0/3515821339 learned_addr learned my addr 172.21.15.46:0/3515821339 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.298+0000 7f52d54d8700 1 --2- 172.21.15.46:0/3515821339 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52d006cbb0 0x7f52d01565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.298+0000 7f52d4cd7700 1 -- 172.21.15.46:0/3515821339 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52d006cbb0 msgr2=0x7f52d01565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.298+0000 7f52d4cd7700 1 --2- 172.21.15.46:0/3515821339 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52d006cbb0 0x7f52d01565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.298+0000 7f52d4cd7700 1 -- 172.21.15.46:0/3515821339 >> v1:172.21.15.46:6789/0 conn(0x7f52d00b0280 legacy=0x7f52d006c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.298+0000 7f52d4cd7700 1 -- 172.21.15.46:0/3515821339 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d00b1000 con 0x7f52d01561d0 2024-01-27T01:03:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.298+0000 7f52d4cd7700 1 --2- 172.21.15.46:0/3515821339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d01561d0 0x7f52d00ca5f0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f52c400bc50 tx=0x7f52c400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39ba111ae68a8151 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.298+0000 7f52cf7fe700 1 -- 172.21.15.46:0/3515821339 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c400dbf0 con 0x7f52d01561d0 2024-01-27T01:03:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.298+0000 7f52cf7fe700 1 -- 172.21.15.46:0/3515821339 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c400dd50 con 0x7f52d01561d0 2024-01-27T01:03:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.298+0000 7f52cf7fe700 1 -- 172.21.15.46:0/3515821339 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c40196d0 con 0x7f52d01561d0 2024-01-27T01:03:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.299+0000 7f52d6f3b700 1 -- 172.21.15.46:0/3515821339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d01561d0 msgr2=0x7f52d00ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.299+0000 7f52d6f3b700 1 --2- 172.21.15.46:0/3515821339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d01561d0 0x7f52d00ca5f0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f52c400bc50 tx=0x7f52c400b4f0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.299+0000 7f52d6f3b700 1 -- 172.21.15.46:0/3515821339 shutdown_connections 2024-01-27T01:03:23.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.299+0000 7f52d6f3b700 1 --2- 172.21.15.46:0/3515821339 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52d006cbb0 0x7f52d01565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.299+0000 7f52d6f3b700 1 --2- 172.21.15.46:0/3515821339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d01561d0 0x7f52d00ca5f0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.299+0000 7f52d6f3b700 1 -- 172.21.15.46:0/3515821339 >> 172.21.15.46:0/3515821339 conn(0x7f52d006b630 msgr2=0x7f52d00aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:23.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.299+0000 7f52d6f3b700 1 -- 172.21.15.46:0/3515821339 shutdown_connections 2024-01-27T01:03:23.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.299+0000 7f52d6f3b700 1 -- 172.21.15.46:0/3515821339 wait complete. 2024-01-27T01:03:23.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.301+0000 7f52d6f3b700 1 Processor -- start 2024-01-27T01:03:23.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.301+0000 7f52d6f3b700 1 -- start start 2024-01-27T01:03:23.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.302+0000 7f52d6f3b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52d006cbb0 0x7f52d014b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:23.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.302+0000 7f52d6f3b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d00b0280 0x7f52d00ca780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:23.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.302+0000 7f52d6f3b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52d01561d0 0x7f52d014a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:23.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.302+0000 7f52d6f3b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00cacc0 con 0x7f52d00b0280 2024-01-27T01:03:23.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.302+0000 7f52d6f3b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00cbb00 con 0x7f52d01561d0 2024-01-27T01:03:23.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.302+0000 7f52d6f3b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00caef0 con 0x7f52d006cbb0 2024-01-27T01:03:23.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.302+0000 7f52cffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d00b0280 0x7f52d00ca780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:23.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.302+0000 7f52d4cd7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52d006cbb0 0x7f52d014b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:23.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.302+0000 7f52cffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d00b0280 0x7f52d00ca780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47188/0 (socket says 172.21.15.46:47188) 2024-01-27T01:03:23.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.302+0000 7f52cffff700 1 -- 172.21.15.46:0/3367263538 learned_addr learned my addr 172.21.15.46:0/3367263538 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:23.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.302+0000 7f52d54d8700 1 --2- 172.21.15.46:0/3367263538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52d01561d0 0x7f52d014a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:23.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.302+0000 7f52d4cd7700 1 -- 172.21.15.46:0/3367263538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52d01561d0 msgr2=0x7f52d014a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:23.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.302+0000 7f52d4cd7700 1 --2- 172.21.15.46:0/3367263538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52d01561d0 0x7f52d014a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.302+0000 7f52d4cd7700 1 -- 172.21.15.46:0/3367263538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d00b0280 msgr2=0x7f52d00ca780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:23.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.302+0000 7f52d4cd7700 1 --2- 172.21.15.46:0/3367263538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d00b0280 0x7f52d00ca780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.302+0000 7f52d4cd7700 1 -- 172.21.15.46:0/3367263538 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c400b790 con 0x7f52d006cbb0 2024-01-27T01:03:23.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.302+0000 7f52cffff700 1 --2- 172.21.15.46:0/3367263538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d00b0280 0x7f52d00ca780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:23.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.303+0000 7f52d4cd7700 1 --2- 172.21.15.46:0/3367263538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52d006cbb0 0x7f52d014b700 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f52c4015000 tx=0x7f52c401b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:23.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.303+0000 7f52ccff9700 1 -- 172.21.15.46:0/3367263538 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c400dbf0 con 0x7f52d006cbb0 2024-01-27T01:03:23.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.303+0000 7f52ccff9700 1 -- 172.21.15.46:0/3367263538 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c400dec0 con 0x7f52d006cbb0 2024-01-27T01:03:23.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.303+0000 7f52d6f3b700 1 -- 172.21.15.46:0/3367263538 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d00cb1d0 con 0x7f52d006cbb0 2024-01-27T01:03:23.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.303+0000 7f52ccff9700 1 -- 172.21.15.46:0/3367263538 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c40106c0 con 0x7f52d006cbb0 2024-01-27T01:03:23.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.303+0000 7f52d6f3b700 1 -- 172.21.15.46:0/3367263538 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d00cb790 con 0x7f52d006cbb0 2024-01-27T01:03:23.304 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:23.304+0000 7fa5b9ca21c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:03:23.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.304+0000 7f52d6f3b700 1 -- 172.21.15.46:0/3367263538 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d004c710 con 0x7f52d006cbb0 2024-01-27T01:03:23.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.304+0000 7f52ccff9700 1 -- 172.21.15.46:0/3367263538 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 110) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f52c4022030 con 0x7f52d006cbb0 2024-01-27T01:03:23.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.305+0000 7f52ccff9700 1 --2- 172.21.15.46:0/3367263538 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f52b4067050 0x7f52b4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:23.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.305+0000 7f52cffff700 1 -- 172.21.15.46:0/3367263538 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f52b4067050 msgr2=0x7f52b4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:23.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.305+0000 7f52ccff9700 1 -- 172.21.15.46:0/3367263538 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 7049+0+0 (secure 0 0 0) 0x7f52c4014070 con 0x7f52d006cbb0 2024-01-27T01:03:23.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.305+0000 7f52cffff700 1 --2- 172.21.15.46:0/3367263538 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f52b4067050 0x7f52b4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:03:23.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.309+0000 7f52ccff9700 1 -- 172.21.15.46:0/3367263538 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f52c40550e0 con 0x7f52d006cbb0 2024-01-27T01:03:23.385 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:23.386+0000 7fa5b9ca21c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:03:23.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.506+0000 7f52cffff700 1 -- 172.21.15.46:0/3367263538 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f52b4067050 msgr2=0x7f52b4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:23.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.506+0000 7f52cffff700 1 --2- 172.21.15.46:0/3367263538 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f52b4067050 0x7f52b4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:03:23.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.560+0000 7f52d6f3b700 1 -- 172.21.15.46:0/3367263538 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52d00b12e0 con 0x7f52d006cbb0 2024-01-27T01:03:23.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.561+0000 7f52ccff9700 1 -- 172.21.15.46:0/3367263538 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4601 (secure 0 0 0) 0x7f52c4058700 con 0x7f52d006cbb0 2024-01-27T01:03:23.560 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:23.560 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":97,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8435,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/4207115513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":4207115513},{"type":"v1","addr":"172.21.15.106:6835","nonce":4207115513}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91},{"gid":8438,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/282430549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":282430549},{"type":"v1","addr":"172.21.15.46:6834","nonce":282430549}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9508,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/361546207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":361546207},{"type":"v1","addr":"172.21.15.106:6834","nonce":361546207}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T00:47:19.926407+0000","modified":"2024-01-27T00:47:23.511130+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9828":{"gid":9828,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2544525048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2544525048},{"type":"v1","addr":"172.21.15.46:6835","nonce":2544525048}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2024-01-27T01:03:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.563+0000 7f52d6f3b700 1 -- 172.21.15.46:0/3367263538 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f52b4067050 msgr2=0x7f52b4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:03:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.563+0000 7f52d6f3b700 1 --2- 172.21.15.46:0/3367263538 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f52b4067050 0x7f52b4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.563+0000 7f52d6f3b700 1 -- 172.21.15.46:0/3367263538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52d006cbb0 msgr2=0x7f52d014b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.563+0000 7f52d6f3b700 1 --2- 172.21.15.46:0/3367263538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52d006cbb0 0x7f52d014b700 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f52c4015000 tx=0x7f52c401b000 comp rx=0 tx=0).stop 2024-01-27T01:03:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.563+0000 7f52d6f3b700 1 -- 172.21.15.46:0/3367263538 shutdown_connections 2024-01-27T01:03:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.563+0000 7f52d6f3b700 1 --2- 172.21.15.46:0/3367263538 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f52b4067050 0x7f52b4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.563+0000 7f52d6f3b700 1 --2- 172.21.15.46:0/3367263538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52d01561d0 0x7f52d014a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.563+0000 7f52d6f3b700 1 --2- 172.21.15.46:0/3367263538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52d006cbb0 0x7f52d014b700 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.563+0000 7f52d6f3b700 1 --2- 172.21.15.46:0/3367263538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d00b0280 0x7f52d00ca780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.563+0000 7f52d6f3b700 1 -- 172.21.15.46:0/3367263538 >> 172.21.15.46:0/3367263538 conn(0x7f52d006b630 msgr2=0x7f52d01504e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.563+0000 7f52d6f3b700 1 -- 172.21.15.46:0/3367263538 shutdown_connections 2024-01-27T01:03:23.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.564+0000 7f52d6f3b700 1 -- 172.21.15.46:0/3367263538 wait complete. 2024-01-27T01:03:23.564 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:23.564+0000 7fd80323e1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:03:23.564 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 97 2024-01-27T01:03:23.577 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:03:23.637 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:23.638+0000 7fd80323e1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:03:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.711+0000 7f7bf8d2d700 1 Processor -- start 2024-01-27T01:03:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.711+0000 7f7bf8d2d700 1 -- start start 2024-01-27T01:03:23.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.712+0000 7f7bf8d2d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7bf40acf30 0x7f7bf40c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:23.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.712+0000 7f7bf8d2d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bf40ad300 0x7f7bf40ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:23.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.712+0000 7f7bf8d2d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf40c4230 con 0x7f7bf40ad750 2024-01-27T01:03:23.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.712+0000 7f7bf8d2d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf4157be0 con 0x7f7bf40acf30 2024-01-27T01:03:23.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.712+0000 7f7bf8d2d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40cbf10 con 0x7f7bf40ad300 2024-01-27T01:03:23.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.712+0000 7f7bf2d9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7bf40ad750 0x7f7bf415b270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45762/0 (socket says 172.21.15.46:45762) 2024-01-27T01:03:23.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.712+0000 7f7bf2d9d700 1 -- 172.21.15.46:0/716407136 learned_addr learned my addr 172.21.15.46:0/716407136 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:23.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.712+0000 7f7bf1d9b700 1 --2- 172.21.15.46:0/716407136 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bf40ad300 0x7f7bf40ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:23.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.712+0000 7f7bf259c700 1 --2- 172.21.15.46:0/716407136 >> v2:172.21.15.106:3300/0 conn(0x7f7bf40acf30 0x7f7bf40c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:23.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.712+0000 7f7bf159a700 1 -- 172.21.15.46:0/716407136 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122380250 0 0) 0x7f7bf40c4230 con 0x7f7bf40ad750 2024-01-27T01:03:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.712+0000 7f7bf159a700 1 -- 172.21.15.46:0/716407136 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bdc003570 con 0x7f7bf40ad750 2024-01-27T01:03:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.712+0000 7f7bf259c700 1 -- 172.21.15.46:0/716407136 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bf40ad300 msgr2=0x7f7bf40ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.713+0000 7f7bf259c700 1 --2- 172.21.15.46:0/716407136 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bf40ad300 0x7f7bf40ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.713+0000 7f7bf259c700 1 -- 172.21.15.46:0/716407136 >> v1:172.21.15.46:6789/0 conn(0x7f7bf40ad750 legacy=0x7f7bf415b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.713+0000 7f7bf259c700 1 -- 172.21.15.46:0/716407136 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf40c4460 con 0x7f7bf40acf30 2024-01-27T01:03:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.713+0000 7f7bf1d9b700 1 --2- 172.21.15.46:0/716407136 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bf40ad300 0x7f7bf40ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.713+0000 7f7bf259c700 1 --2- 172.21.15.46:0/716407136 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bf40acf30 0x7f7bf40c3cf0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f7be40098e0 tx=0x7f7be40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ce38148a4edd484 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.713+0000 7f7bf159a700 1 -- 172.21.15.46:0/716407136 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be4011630 con 0x7f7bf40acf30 2024-01-27T01:03:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.713+0000 7f7bf159a700 1 -- 172.21.15.46:0/716407136 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be4011790 con 0x7f7bf40acf30 2024-01-27T01:03:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.713+0000 7f7bf159a700 1 -- 172.21.15.46:0/716407136 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be4011b30 con 0x7f7bf40acf30 2024-01-27T01:03:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.714+0000 7f7bf8d2d700 1 -- 172.21.15.46:0/716407136 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bf40acf30 msgr2=0x7f7bf40c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.714+0000 7f7bf8d2d700 1 --2- 172.21.15.46:0/716407136 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bf40acf30 0x7f7bf40c3cf0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f7be40098e0 tx=0x7f7be40091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.714+0000 7f7bf8d2d700 1 -- 172.21.15.46:0/716407136 shutdown_connections 2024-01-27T01:03:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.714+0000 7f7bf8d2d700 1 --2- 172.21.15.46:0/716407136 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bf40acf30 0x7f7bf40c3cf0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.714+0000 7f7bf8d2d700 1 --2- 172.21.15.46:0/716407136 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bf40ad300 0x7f7bf40ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.714+0000 7f7bf8d2d700 1 -- 172.21.15.46:0/716407136 >> 172.21.15.46:0/716407136 conn(0x7f7bf4097820 msgr2=0x7f7bf4068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.714+0000 7f7bf8d2d700 1 -- 172.21.15.46:0/716407136 shutdown_connections 2024-01-27T01:03:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.714+0000 7f7bf8d2d700 1 -- 172.21.15.46:0/716407136 wait complete. 2024-01-27T01:03:23.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.715+0000 7f7bf8d2d700 1 Processor -- start 2024-01-27T01:03:23.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.715+0000 7f7bf8d2d700 1 -- start start 2024-01-27T01:03:23.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf8d2d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf40acf30 0x7f7bf40b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:23.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf8d2d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bf40ad300 0x7f7bf4152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:23.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf8d2d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bf40ad750 0x7f7bf40cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:23.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf8d2d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf4153b60 con 0x7f7bf40acf30 2024-01-27T01:03:23.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf8d2d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40c9440 con 0x7f7bf40ad300 2024-01-27T01:03:23.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf8d2d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40b2d40 con 0x7f7bf40ad750 2024-01-27T01:03:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf40acf30 0x7f7bf40b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf40acf30 0x7f7bf40b12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47192/0 (socket says 172.21.15.46:47192) 2024-01-27T01:03:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf259c700 1 -- 172.21.15.46:0/2103918823 learned_addr learned my addr 172.21.15.46:0/2103918823 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf1d9b700 1 --2- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bf40ad300 0x7f7bf4152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf2d9d700 1 --2- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bf40ad750 0x7f7bf40cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf2d9d700 1 -- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bf40ad300 msgr2=0x7f7bf4152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf2d9d700 1 --2- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bf40ad300 0x7f7bf4152f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf2d9d700 1 -- 172.21.15.46:0/2103918823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf40acf30 msgr2=0x7f7bf40b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf2d9d700 1 --2- 172.21.15.46:0/2103918823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf40acf30 0x7f7bf40b12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf2d9d700 1 -- 172.21.15.46:0/2103918823 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be8009420 con 0x7f7bf40ad750 2024-01-27T01:03:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf1d9b700 1 --2- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bf40ad300 0x7f7bf4152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf259c700 1 --2- 172.21.15.46:0/2103918823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf40acf30 0x7f7bf40b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.716+0000 7f7bf2d9d700 1 --2- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bf40ad750 0x7f7bf40cb1f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40b1080 tx=0x7f7bec001980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.717+0000 7f7be27fc700 1 -- 172.21.15.46:0/2103918823 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec00a7e0 con 0x7f7bf40ad750 2024-01-27T01:03:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.717+0000 7f7be27fc700 1 -- 172.21.15.46:0/2103918823 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bec00a940 con 0x7f7bf40ad750 2024-01-27T01:03:23.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.717+0000 7f7bf8d2d700 1 -- 172.21.15.46:0/2103918823 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be4009420 con 0x7f7bf40ad750 2024-01-27T01:03:23.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.717+0000 7f7be27fc700 1 -- 172.21.15.46:0/2103918823 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec00acc0 con 0x7f7bf40ad750 2024-01-27T01:03:23.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.717+0000 7f7bf8d2d700 1 -- 172.21.15.46:0/2103918823 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf40b3360 con 0x7f7bf40ad750 2024-01-27T01:03:23.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.718+0000 7f7bf8d2d700 1 -- 172.21.15.46:0/2103918823 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf40acbb0 con 0x7f7bf40ad750 2024-01-27T01:03:23.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.718+0000 7f7be27fc700 1 -- 172.21.15.46:0/2103918823 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 110) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f7bec007300 con 0x7f7bf40ad750 2024-01-27T01:03:23.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.718+0000 7f7be27fc700 1 --2- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bd4067050 0x7f7bd4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:23.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.719+0000 7f7bf259c700 1 -- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bd4067050 msgr2=0x7f7bd4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:23.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.719+0000 7f7be27fc700 1 -- 172.21.15.46:0/2103918823 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 7049+0+0 (secure 0 0 0) 0x7f7bec0847a0 con 0x7f7bf40ad750 2024-01-27T01:03:23.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.719+0000 7f7bf259c700 1 --2- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bd4067050 0x7f7bd4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:03:23.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.723+0000 7f7be27fc700 1 -- 172.21.15.46:0/2103918823 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7bec04b5a0 con 0x7f7bf40ad750 2024-01-27T01:03:23.763 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:23.763+0000 7fd80323e1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:03:23.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.920+0000 7f7bf259c700 1 -- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bd4067050 msgr2=0x7f7bd4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:23.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.920+0000 7f7bf259c700 1 --2- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bd4067050 0x7f7bd4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:03:23.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.930+0000 7f7bf8d2d700 1 -- 172.21.15.46:0/2103918823 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7bf4088a90 con 0x7f7bf40ad750 2024-01-27T01:03:23.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.931+0000 7f7be27fc700 1 -- 172.21.15.46:0/2103918823 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) v1 ==== 74+0+17118 (secure 0 0 0) 0x7f7bec054e50 con 0x7f7bf40ad750 2024-01-27T01:03:23.930 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:23.930 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":192,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:03:21.686956+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-01-27T00:47:17.642961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-01-27T00:47:18.897371+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":192,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/273126119":"2024-01-28T00:58:22.118992+0000","172.21.15.106:0/683705968":"2024-01-28T00:58:22.118992+0000","172.21.15.106:0/211490639":"2024-01-28T00:58:22.118992+0000","172.21.15.106:0/1013704950":"2024-01-28T00:47:39.609727+0000","172.21.15.106:0/243674269":"2024-01-28T00:47:39.609727+0000","172.21.15.106:0/1532248390":"2024-01-28T00:58:22.118992+0000","172.21.15.106:6837/43388":"2024-01-28T00:47:39.609727+0000","172.21.15.106:6836/43388":"2024-01-28T00:47:39.609727+0000","172.21.15.106:0/1578971181":"2024-01-28T00:58:22.118992+0000","172.21.15.106:0/2043045162":"2024-01-28T00:47:39.609727+0000","172.21.15.106:0/1872294075":"2024-01-28T00:58:22.118992+0000","172.21.15.106:0/14745894":"2024-01-28T00:47:39.609727+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:03:23.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.933+0000 7f7bf8d2d700 1 -- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bd4067050 msgr2=0x7f7bd4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:03:23.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.933+0000 7f7bf8d2d700 1 --2- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bd4067050 0x7f7bd4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.933+0000 7f7bf8d2d700 1 -- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bf40ad750 msgr2=0x7f7bf40cb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:23.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.933+0000 7f7bf8d2d700 1 --2- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bf40ad750 0x7f7bf40cb1f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40b1080 tx=0x7f7bec001980 comp rx=0 tx=0).stop 2024-01-27T01:03:23.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.933+0000 7f7bf8d2d700 1 -- 172.21.15.46:0/2103918823 shutdown_connections 2024-01-27T01:03:23.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.933+0000 7f7bf8d2d700 1 --2- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bd4067050 0x7f7bd4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.933+0000 7f7bf8d2d700 1 --2- 172.21.15.46:0/2103918823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf40acf30 0x7f7bf40b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.933+0000 7f7bf8d2d700 1 --2- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bf40ad750 0x7f7bf40cb1f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.933+0000 7f7bf8d2d700 1 --2- 172.21.15.46:0/2103918823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bf40ad300 0x7f7bf4152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:23.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.933+0000 7f7bf8d2d700 1 -- 172.21.15.46:0/2103918823 >> 172.21.15.46:0/2103918823 conn(0x7f7bf4097820 msgr2=0x7f7bf40ced60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:23.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.933+0000 7f7bf8d2d700 1 -- 172.21.15.46:0/2103918823 shutdown_connections 2024-01-27T01:03:23.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:23.933+0000 7f7bf8d2d700 1 -- 172.21.15.46:0/2103918823 wait complete. 2024-01-27T01:03:23.947 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-27T01:03:23.947 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:03:24.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.081+0000 7fde8e232700 1 Processor -- start 2024-01-27T01:03:24.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.081+0000 7fde8e232700 1 -- start start 2024-01-27T01:03:24.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.082+0000 7fde8e232700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde880b2390 0x7fde88156f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.082+0000 7fde8e232700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde881574c0 0x7fde880cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.082+0000 7fde8e232700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde880cac60 con 0x7fde880b1fc0 2024-01-27T01:03:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.082+0000 7fde8e232700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde88068dc0 con 0x7fde881574c0 2024-01-27T01:03:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.082+0000 7fde8e232700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde880690a0 con 0x7fde880b2390 2024-01-27T01:03:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.083+0000 7fde87fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde881574c0 0x7fde880cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.083+0000 7fde87fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde881574c0 0x7fde880cb6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47204/0 (socket says 172.21.15.46:47204) 2024-01-27T01:03:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.083+0000 7fde87fff700 1 -- 172.21.15.46:0/3068022563 learned_addr learned my addr 172.21.15.46:0/3068022563 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.083+0000 7fde86ffd700 1 --2- 172.21.15.46:0/3068022563 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde880b2390 0x7fde88156f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.083+0000 7fde867fc700 1 -- 172.21.15.46:0/3068022563 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2390790559 0 0) 0x7fde880cac60 con 0x7fde880b1fc0 2024-01-27T01:03:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.083+0000 7fde867fc700 1 -- 172.21.15.46:0/3068022563 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde70003570 con 0x7fde880b1fc0 2024-01-27T01:03:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.083+0000 7fde87fff700 1 -- 172.21.15.46:0/3068022563 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde880b2390 msgr2=0x7fde88156f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.083+0000 7fde87fff700 1 --2- 172.21.15.46:0/3068022563 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde880b2390 0x7fde88156f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.083+0000 7fde87fff700 1 -- 172.21.15.46:0/3068022563 >> v1:172.21.15.106:6789/0 conn(0x7fde880b1fc0 legacy=0x7fde88068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.083+0000 7fde87fff700 1 -- 172.21.15.46:0/3068022563 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde880caee0 con 0x7fde881574c0 2024-01-27T01:03:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.083+0000 7fde86ffd700 1 --2- 172.21.15.46:0/3068022563 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde880b2390 0x7fde88156f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.083+0000 7fde87fff700 1 --2- 172.21.15.46:0/3068022563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde881574c0 0x7fde880cb6e0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fde800099b0 tx=0x7fde80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3846cec980499fc5 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.083+0000 7fde867fc700 1 -- 172.21.15.46:0/3068022563 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8000b680 con 0x7fde881574c0 2024-01-27T01:03:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.083+0000 7fde867fc700 1 -- 172.21.15.46:0/3068022563 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8000b7e0 con 0x7fde881574c0 2024-01-27T01:03:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.083+0000 7fde867fc700 1 -- 172.21.15.46:0/3068022563 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8000bb80 con 0x7fde881574c0 2024-01-27T01:03:24.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.084+0000 7fde8e232700 1 -- 172.21.15.46:0/3068022563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde881574c0 msgr2=0x7fde880cb6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:24.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.084+0000 7fde8e232700 1 --2- 172.21.15.46:0/3068022563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde881574c0 0x7fde880cb6e0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fde800099b0 tx=0x7fde80009250 comp rx=0 tx=0).stop 2024-01-27T01:03:24.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.084+0000 7fde8e232700 1 -- 172.21.15.46:0/3068022563 shutdown_connections 2024-01-27T01:03:24.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.084+0000 7fde8e232700 1 --2- 172.21.15.46:0/3068022563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde881574c0 0x7fde880cb6e0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.084+0000 7fde8e232700 1 --2- 172.21.15.46:0/3068022563 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde880b2390 0x7fde88156f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.084+0000 7fde8e232700 1 -- 172.21.15.46:0/3068022563 >> 172.21.15.46:0/3068022563 conn(0x7fde880ccbc0 msgr2=0x7fde880ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:24.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.084+0000 7fde8e232700 1 -- 172.21.15.46:0/3068022563 shutdown_connections 2024-01-27T01:03:24.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.084+0000 7fde8e232700 1 -- 172.21.15.46:0/3068022563 wait complete. 2024-01-27T01:03:24.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.086+0000 7fde8e232700 1 Processor -- start 2024-01-27T01:03:24.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.086+0000 7fde8e232700 1 -- start start 2024-01-27T01:03:24.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.086+0000 7fde8e232700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde880b1fc0 0x7fde880b0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:24.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.086+0000 7fde8e232700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde880b2390 0x7fde8813e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:24.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.086+0000 7fde8e232700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde881574c0 0x7fde8813ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:24.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.086+0000 7fde8e232700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8813f1d0 con 0x7fde880b2390 2024-01-27T01:03:24.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.086+0000 7fde8e232700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8815eb90 con 0x7fde880b1fc0 2024-01-27T01:03:24.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.086+0000 7fde8e232700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde8815e8b0 con 0x7fde881574c0 2024-01-27T01:03:24.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.087+0000 7fde86ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde880b2390 0x7fde8813e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:24.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.087+0000 7fde86ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde880b2390 0x7fde8813e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47210/0 (socket says 172.21.15.46:47210) 2024-01-27T01:03:24.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.087+0000 7fde86ffd700 1 -- 172.21.15.46:0/4140368601 learned_addr learned my addr 172.21.15.46:0/4140368601 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:24.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.087+0000 7fde87fff700 1 --2- 172.21.15.46:0/4140368601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde881574c0 0x7fde8813ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:24.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.087+0000 7fde877fe700 1 --2- 172.21.15.46:0/4140368601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde880b1fc0 0x7fde880b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:24.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.087+0000 7fde87fff700 1 -- 172.21.15.46:0/4140368601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde880b1fc0 msgr2=0x7fde880b0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.087+0000 7fde87fff700 1 --2- 172.21.15.46:0/4140368601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde880b1fc0 0x7fde880b0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.087+0000 7fde87fff700 1 -- 172.21.15.46:0/4140368601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde880b2390 msgr2=0x7fde8813e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.087+0000 7fde87fff700 1 --2- 172.21.15.46:0/4140368601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde880b2390 0x7fde8813e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.087+0000 7fde87fff700 1 -- 172.21.15.46:0/4140368601 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c009420 con 0x7fde881574c0 2024-01-27T01:03:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.087+0000 7fde86ffd700 1 --2- 172.21.15.46:0/4140368601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde880b2390 0x7fde8813e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.087+0000 7fde87fff700 1 --2- 172.21.15.46:0/4140368601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde881574c0 0x7fde8813ec90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fde880b2910 tx=0x7fde80017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.087+0000 7fde6f7fe700 1 -- 172.21.15.46:0/4140368601 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde80002f90 con 0x7fde881574c0 2024-01-27T01:03:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.087+0000 7fde6f7fe700 1 -- 172.21.15.46:0/4140368601 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8000bb60 con 0x7fde881574c0 2024-01-27T01:03:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.087+0000 7fde6f7fe700 1 -- 172.21.15.46:0/4140368601 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8001a680 con 0x7fde881574c0 2024-01-27T01:03:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.087+0000 7fde8e232700 1 -- 172.21.15.46:0/4140368601 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde800094f0 con 0x7fde881574c0 2024-01-27T01:03:24.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.088+0000 7fde8e232700 1 -- 172.21.15.46:0/4140368601 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8813f850 con 0x7fde881574c0 2024-01-27T01:03:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.088+0000 7fde8e232700 1 -- 172.21.15.46:0/4140368601 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8804c710 con 0x7fde881574c0 2024-01-27T01:03:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.089+0000 7fde6f7fe700 1 -- 172.21.15.46:0/4140368601 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 110) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fde8001aab0 con 0x7fde881574c0 2024-01-27T01:03:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.089+0000 7fde6f7fe700 1 --2- 172.21.15.46:0/4140368601 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fde74067050 0x7fde74069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.089+0000 7fde877fe700 1 -- 172.21.15.46:0/4140368601 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fde74067050 msgr2=0x7fde74069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.089+0000 7fde6f7fe700 1 -- 172.21.15.46:0/4140368601 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 7049+0+0 (secure 0 0 0) 0x7fde800868f0 con 0x7fde881574c0 2024-01-27T01:03:24.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.089+0000 7fde877fe700 1 --2- 172.21.15.46:0/4140368601 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fde74067050 0x7fde74069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:03:24.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.093+0000 7fde6f7fe700 1 -- 172.21.15.46:0/4140368601 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fde80053470 con 0x7fde881574c0 2024-01-27T01:03:24.098 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:24.099+0000 7fa5b9ca21c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:03:24.183 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:24.184+0000 7fa5b9ca21c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:03:24.264 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:24.265+0000 7fd80323e1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:03:24.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.290+0000 7fde877fe700 1 -- 172.21.15.46:0/4140368601 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fde74067050 msgr2=0x7fde74069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:24.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.290+0000 7fde877fe700 1 --2- 172.21.15.46:0/4140368601 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fde74067050 0x7fde74069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:03:24.325 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:24.326+0000 7fa5b9ca21c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:03:24.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.346+0000 7fde8e232700 1 -- 172.21.15.46:0/4140368601 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde880cb1c0 con 0x7fde881574c0 2024-01-27T01:03:24.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.347+0000 7fde6f7fe700 1 -- 172.21.15.46:0/4140368601 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4601 (secure 0 0 0) 0x7fde80056a90 con 0x7fde881574c0 2024-01-27T01:03:24.346 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:24.346 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":97,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8435,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/4207115513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":4207115513},{"type":"v1","addr":"172.21.15.106:6835","nonce":4207115513}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91},{"gid":8438,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/282430549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":282430549},{"type":"v1","addr":"172.21.15.46:6834","nonce":282430549}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9508,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/361546207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":361546207},{"type":"v1","addr":"172.21.15.106:6834","nonce":361546207}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T00:47:19.926407+0000","modified":"2024-01-27T00:47:23.511130+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9828":{"gid":9828,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2544525048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2544525048},{"type":"v1","addr":"172.21.15.46:6835","nonce":2544525048}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2024-01-27T01:03:24.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.349+0000 7fde8e232700 1 -- 172.21.15.46:0/4140368601 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fde74067050 msgr2=0x7fde74069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:03:24.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.349+0000 7fde8e232700 1 --2- 172.21.15.46:0/4140368601 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fde74067050 0x7fde74069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.349+0000 7fde8e232700 1 -- 172.21.15.46:0/4140368601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde881574c0 msgr2=0x7fde8813ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:24.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.349+0000 7fde8e232700 1 --2- 172.21.15.46:0/4140368601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde881574c0 0x7fde8813ec90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fde880b2910 tx=0x7fde80017000 comp rx=0 tx=0).stop 2024-01-27T01:03:24.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.349+0000 7fde8e232700 1 -- 172.21.15.46:0/4140368601 shutdown_connections 2024-01-27T01:03:24.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.349+0000 7fde8e232700 1 --2- 172.21.15.46:0/4140368601 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fde74067050 0x7fde74069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.349+0000 7fde8e232700 1 --2- 172.21.15.46:0/4140368601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde881574c0 0x7fde8813ec90 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.349+0000 7fde8e232700 1 --2- 172.21.15.46:0/4140368601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde880b1fc0 0x7fde880b0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.349+0000 7fde8e232700 1 --2- 172.21.15.46:0/4140368601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde880b2390 0x7fde8813e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.349+0000 7fde8e232700 1 -- 172.21.15.46:0/4140368601 >> 172.21.15.46:0/4140368601 conn(0x7fde880ccbc0 msgr2=0x7fde88154cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:24.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.349+0000 7fde8e232700 1 -- 172.21.15.46:0/4140368601 shutdown_connections 2024-01-27T01:03:24.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.349+0000 7fde8e232700 1 -- 172.21.15.46:0/4140368601 wait complete. 2024-01-27T01:03:24.350 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 97 2024-01-27T01:03:24.362 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:03:24.413 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:24.414+0000 7fd80323e1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:03:24.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.496+0000 7f5a70a34700 1 Processor -- start 2024-01-27T01:03:24.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.496+0000 7f5a70a34700 1 -- start start 2024-01-27T01:03:24.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.497+0000 7f5a70a34700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a6c0cb4c0 0x7f5a6c159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:24.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.497+0000 7f5a70a34700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c0c97f0 con 0x7f5a6c1596e0 2024-01-27T01:03:24.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.497+0000 7f5a70a34700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c0c8c20 con 0x7f5a6c0c4720 2024-01-27T01:03:24.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.497+0000 7f5a70a34700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a6c0ce0d0 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.497+0000 7f5a69d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a6c0cb4c0 0x7f5a6c159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:24.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.497+0000 7f5a69d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a6c0cb4c0 0x7f5a6c159ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47212/0 (socket says 172.21.15.46:47212) 2024-01-27T01:03:24.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.497+0000 7f5a69d9b700 1 -- 172.21.15.46:0/2995367783 learned_addr learned my addr 172.21.15.46:0/2995367783 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:24.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.497+0000 7f5a69d9b700 1 -- 172.21.15.46:0/2995367783 >> v1:172.21.15.106:6789/0 conn(0x7f5a6c0c4720 legacy=0x7f5a6c0cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:24.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.497+0000 7f5a69d9b700 1 -- 172.21.15.46:0/2995367783 >> v1:172.21.15.46:6789/0 conn(0x7f5a6c1596e0 legacy=0x7f5a6c0cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:24.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.497+0000 7f5a69d9b700 1 -- 172.21.15.46:0/2995367783 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c0c8ec0 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.498+0000 7f5a69d9b700 1 --2- 172.21.15.46:0/2995367783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c0cb4c0 0x7f5a6c159ab0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f5a600099b0 tx=0x7f5a60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd5b378f0a0a175 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:24.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.498+0000 7f5a6959a700 1 -- 172.21.15.46:0/2995367783 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6000b860 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.498+0000 7f5a6959a700 1 -- 172.21.15.46:0/2995367783 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6000b9c0 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.498+0000 7f5a6959a700 1 -- 172.21.15.46:0/2995367783 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6000bd60 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.498+0000 7f5a70a34700 1 -- 172.21.15.46:0/2995367783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c0cb4c0 msgr2=0x7f5a6c159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.498+0000 7f5a70a34700 1 --2- 172.21.15.46:0/2995367783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c0cb4c0 0x7f5a6c159ab0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f5a600099b0 tx=0x7f5a60009250 comp rx=0 tx=0).stop 2024-01-27T01:03:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.499+0000 7f5a70a34700 1 -- 172.21.15.46:0/2995367783 shutdown_connections 2024-01-27T01:03:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.499+0000 7f5a70a34700 1 --2- 172.21.15.46:0/2995367783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c0cb4c0 0x7f5a6c159ab0 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.499+0000 7f5a70a34700 1 -- 172.21.15.46:0/2995367783 >> 172.21.15.46:0/2995367783 conn(0x7f5a6c097820 msgr2=0x7f5a6c068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.499+0000 7f5a70a34700 1 -- 172.21.15.46:0/2995367783 shutdown_connections 2024-01-27T01:03:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.499+0000 7f5a70a34700 1 -- 172.21.15.46:0/2995367783 wait complete. 2024-01-27T01:03:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.500+0000 7f5a70a34700 1 Processor -- start 2024-01-27T01:03:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.500+0000 7f5a70a34700 1 -- start start 2024-01-27T01:03:24.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.501+0000 7f5a70a34700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a6c0c4720 0x7f5a6c16d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:24.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.501+0000 7f5a70a34700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c0cb4c0 0x7f5a6c16d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:24.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.501+0000 7f5a70a34700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a6c1596e0 0x7f5a6c147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:24.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.501+0000 7f5a70a34700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c157390 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.501+0000 7f5a70a34700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c157820 con 0x7f5a6c1596e0 2024-01-27T01:03:24.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.501+0000 7f5a70a34700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c156780 con 0x7f5a6c0c4720 2024-01-27T01:03:24.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.501+0000 7f5a69d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c0cb4c0 0x7f5a6c16d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:24.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.501+0000 7f5a69d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c0cb4c0 0x7f5a6c16d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47222/0 (socket says 172.21.15.46:47222) 2024-01-27T01:03:24.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.501+0000 7f5a69d9b700 1 -- 172.21.15.46:0/499731124 learned_addr learned my addr 172.21.15.46:0/499731124 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:24.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.501+0000 7f5a6a59c700 1 --2- 172.21.15.46:0/499731124 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a6c0c4720 0x7f5a6c16d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:24.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.501+0000 7f5a6ad9d700 1 --2- 172.21.15.46:0/499731124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a6c1596e0 0x7f5a6c147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:24.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.501+0000 7f5a69d9b700 1 -- 172.21.15.46:0/499731124 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a6c0c4720 msgr2=0x7f5a6c16d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:24.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.501+0000 7f5a69d9b700 1 --2- 172.21.15.46:0/499731124 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a6c0c4720 0x7f5a6c16d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.501+0000 7f5a69d9b700 1 -- 172.21.15.46:0/499731124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a6c1596e0 msgr2=0x7f5a6c147ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:24.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.501+0000 7f5a69d9b700 1 --2- 172.21.15.46:0/499731124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a6c1596e0 0x7f5a6c147ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.501+0000 7f5a69d9b700 1 -- 172.21.15.46:0/499731124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a600094f0 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.501+0000 7f5a6a59c700 1 --2- 172.21.15.46:0/499731124 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a6c0c4720 0x7f5a6c16d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:24.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.502+0000 7f5a69d9b700 1 --2- 172.21.15.46:0/499731124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c0cb4c0 0x7f5a6c16d950 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c06ef50 tx=0x7f5a60017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:24.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.502+0000 7f5a5a7fc700 1 -- 172.21.15.46:0/499731124 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a60024440 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.502+0000 7f5a5a7fc700 1 -- 172.21.15.46:0/499731124 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a600245a0 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.502+0000 7f5a5a7fc700 1 -- 172.21.15.46:0/499731124 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a60024940 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.502+0000 7f5a70a34700 1 -- 172.21.15.46:0/499731124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6c156a60 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.502+0000 7f5a70a34700 1 -- 172.21.15.46:0/499731124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6c157020 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.503+0000 7f5a70a34700 1 -- 172.21.15.46:0/499731124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6c157500 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.503+0000 7f5a5a7fc700 1 -- 172.21.15.46:0/499731124 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 110) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f5a6001a420 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.504+0000 7f5a5a7fc700 1 --2- 172.21.15.46:0/499731124 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5a4c067050 0x7f5a4c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:24.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.504+0000 7f5a6a59c700 1 -- 172.21.15.46:0/499731124 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5a4c067050 msgr2=0x7f5a4c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:24.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.504+0000 7f5a5a7fc700 1 -- 172.21.15.46:0/499731124 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 7049+0+0 (secure 0 0 0) 0x7f5a60086f80 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.504+0000 7f5a6a59c700 1 --2- 172.21.15.46:0/499731124 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5a4c067050 0x7f5a4c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:03:24.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.508+0000 7f5a5a7fc700 1 -- 172.21.15.46:0/499731124 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a60053db0 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.568 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:24.568+0000 7fd80323e1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:03:24.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.705+0000 7f5a6a59c700 1 -- 172.21.15.46:0/499731124 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5a4c067050 msgr2=0x7f5a4c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:24.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.705+0000 7f5a6a59c700 1 --2- 172.21.15.46:0/499731124 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5a4c067050 0x7f5a4c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:03:24.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.763+0000 7f5a70a34700 1 -- 172.21.15.46:0/499731124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a6c04c710 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.764+0000 7f5a5a7fc700 1 -- 172.21.15.46:0/499731124 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4601 (secure 0 0 0) 0x7f5a60015090 con 0x7f5a6c0cb4c0 2024-01-27T01:03:24.763 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:24.763 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":97,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8435,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/4207115513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":4207115513},{"type":"v1","addr":"172.21.15.106:6835","nonce":4207115513}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91},{"gid":8438,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/282430549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":282430549},{"type":"v1","addr":"172.21.15.46:6834","nonce":282430549}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9508,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/361546207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":361546207},{"type":"v1","addr":"172.21.15.106:6834","nonce":361546207}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T00:47:19.926407+0000","modified":"2024-01-27T00:47:23.511130+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9828":{"gid":9828,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2544525048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2544525048},{"type":"v1","addr":"172.21.15.46:6835","nonce":2544525048}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2024-01-27T01:03:24.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.766+0000 7f5a70a34700 1 -- 172.21.15.46:0/499731124 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5a4c067050 msgr2=0x7f5a4c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:03:24.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.766+0000 7f5a70a34700 1 --2- 172.21.15.46:0/499731124 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5a4c067050 0x7f5a4c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.766+0000 7f5a70a34700 1 -- 172.21.15.46:0/499731124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c0cb4c0 msgr2=0x7f5a6c16d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:24.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.766+0000 7f5a70a34700 1 --2- 172.21.15.46:0/499731124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c0cb4c0 0x7f5a6c16d950 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c06ef50 tx=0x7f5a60017000 comp rx=0 tx=0).stop 2024-01-27T01:03:24.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.766+0000 7f5a70a34700 1 -- 172.21.15.46:0/499731124 shutdown_connections 2024-01-27T01:03:24.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.766+0000 7f5a70a34700 1 --2- 172.21.15.46:0/499731124 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5a4c067050 0x7f5a4c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.766+0000 7f5a70a34700 1 --2- 172.21.15.46:0/499731124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a6c1596e0 0x7f5a6c147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.766+0000 7f5a70a34700 1 --2- 172.21.15.46:0/499731124 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5a6c0c4720 0x7f5a6c16d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.766+0000 7f5a70a34700 1 --2- 172.21.15.46:0/499731124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c0cb4c0 0x7f5a6c16d950 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.766+0000 7f5a70a34700 1 -- 172.21.15.46:0/499731124 >> 172.21.15.46:0/499731124 conn(0x7f5a6c097820 msgr2=0x7f5a6c1583f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:24.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.766+0000 7f5a70a34700 1 -- 172.21.15.46:0/499731124 shutdown_connections 2024-01-27T01:03:24.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.766+0000 7f5a70a34700 1 -- 172.21.15.46:0/499731124 wait complete. 2024-01-27T01:03:24.767 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 97 2024-01-27T01:03:24.779 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:03:24.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.913+0000 7f1ebe1cf700 1 Processor -- start 2024-01-27T01:03:24.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.913+0000 7f1ebe1cf700 1 -- start start 2024-01-27T01:03:24.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.915+0000 7f1ebe1cf700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1eb80cc1c0 0x7f1eb8164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:24.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.915+0000 7f1ebe1cf700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb80cff70 con 0x7f1eb815ce40 2024-01-27T01:03:24.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.915+0000 7f1ebe1cf700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb80cf400 con 0x7f1eb815d210 2024-01-27T01:03:24.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.915+0000 7f1ebe1cf700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb80cf660 con 0x7f1eb80cc1c0 2024-01-27T01:03:24.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.915+0000 7f1eb77fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1eb815ce40 0x7f1eb80d0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45780/0 (socket says 172.21.15.46:45780) 2024-01-27T01:03:24.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.915+0000 7f1eb77fe700 1 -- 172.21.15.46:0/4015236945 learned_addr learned my addr 172.21.15.46:0/4015236945 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:24.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.915+0000 7f1eb7fff700 1 --2- 172.21.15.46:0/4015236945 >> v2:172.21.15.46:3300/0 conn(0x7f1eb80cc1c0 0x7f1eb8164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:24.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.915+0000 7f1eb7fff700 1 -- 172.21.15.46:0/4015236945 >> v1:172.21.15.106:6789/0 conn(0x7f1eb815d210 legacy=0x7f1eb80cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:24.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.915+0000 7f1eb7fff700 1 -- 172.21.15.46:0/4015236945 >> v1:172.21.15.46:6789/0 conn(0x7f1eb815ce40 legacy=0x7f1eb80d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:24.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.915+0000 7f1eb7fff700 1 -- 172.21.15.46:0/4015236945 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb80d0500 con 0x7f1eb80cc1c0 2024-01-27T01:03:24.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.916+0000 7f1eb7fff700 1 --2- 172.21.15.46:0/4015236945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb80cc1c0 0x7f1eb8164700 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00098e0 tx=0x7f1eb00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46af7108b58f550f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:24.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.916+0000 7f1eb67fc700 1 -- 172.21.15.46:0/4015236945 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0009e80 con 0x7f1eb80cc1c0 2024-01-27T01:03:24.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.916+0000 7f1eb67fc700 1 -- 172.21.15.46:0/4015236945 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb000b860 con 0x7f1eb80cc1c0 2024-01-27T01:03:24.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.916+0000 7f1eb67fc700 1 -- 172.21.15.46:0/4015236945 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb000bbe0 con 0x7f1eb80cc1c0 2024-01-27T01:03:24.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.917+0000 7f1ebe1cf700 1 -- 172.21.15.46:0/4015236945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb80cc1c0 msgr2=0x7f1eb8164700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:24.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.917+0000 7f1ebe1cf700 1 --2- 172.21.15.46:0/4015236945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb80cc1c0 0x7f1eb8164700 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00098e0 tx=0x7f1eb00091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.917+0000 7f1ebe1cf700 1 -- 172.21.15.46:0/4015236945 shutdown_connections 2024-01-27T01:03:24.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.917+0000 7f1ebe1cf700 1 --2- 172.21.15.46:0/4015236945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb80cc1c0 0x7f1eb8164700 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.917+0000 7f1ebe1cf700 1 -- 172.21.15.46:0/4015236945 >> 172.21.15.46:0/4015236945 conn(0x7f1eb806f840 msgr2=0x7f1eb80d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:24.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.917+0000 7f1ebe1cf700 1 -- 172.21.15.46:0/4015236945 shutdown_connections 2024-01-27T01:03:24.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.917+0000 7f1ebe1cf700 1 -- 172.21.15.46:0/4015236945 wait complete. 2024-01-27T01:03:24.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.919+0000 7f1ebe1cf700 1 Processor -- start 2024-01-27T01:03:24.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.919+0000 7f1ebe1cf700 1 -- start start 2024-01-27T01:03:24.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.920+0000 7f1ebe1cf700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb80cc1c0 0x7f1eb815f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:24.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.920+0000 7f1ebe1cf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb815ce40 0x7f1eb80c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:24.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.920+0000 7f1ebe1cf700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1eb815d210 0x7f1eb8154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:24.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.920+0000 7f1ebe1cf700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb815fbf0 con 0x7f1eb815ce40 2024-01-27T01:03:24.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.920+0000 7f1ebe1cf700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb815fed0 con 0x7f1eb815d210 2024-01-27T01:03:24.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.920+0000 7f1ebe1cf700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb80c41c0 con 0x7f1eb80cc1c0 2024-01-27T01:03:24.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.920+0000 7f1eb6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb815ce40 0x7f1eb80c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:24.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.920+0000 7f1eb6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb815ce40 0x7f1eb80c7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47254/0 (socket says 172.21.15.46:47254) 2024-01-27T01:03:24.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.920+0000 7f1eb6ffd700 1 -- 172.21.15.46:0/535389027 learned_addr learned my addr 172.21.15.46:0/535389027 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:24.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.921+0000 7f1eb77fe700 1 --2- 172.21.15.46:0/535389027 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb80cc1c0 0x7f1eb815f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:24.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.921+0000 7f1eb7fff700 1 --2- 172.21.15.46:0/535389027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1eb815d210 0x7f1eb8154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:24.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.921+0000 7f1eb6ffd700 1 -- 172.21.15.46:0/535389027 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb80cc1c0 msgr2=0x7f1eb815f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:24.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.921+0000 7f1eb6ffd700 1 --2- 172.21.15.46:0/535389027 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb80cc1c0 0x7f1eb815f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.921+0000 7f1eb6ffd700 1 -- 172.21.15.46:0/535389027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1eb815d210 msgr2=0x7f1eb8154100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:24.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.921+0000 7f1eb6ffd700 1 --2- 172.21.15.46:0/535389027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1eb815d210 0x7f1eb8154100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:24.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.921+0000 7f1eb6ffd700 1 -- 172.21.15.46:0/535389027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb0009420 con 0x7f1eb815ce40 2024-01-27T01:03:24.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.921+0000 7f1eb6ffd700 1 --2- 172.21.15.46:0/535389027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb815ce40 0x7f1eb80c7750 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f1eac00c590 tx=0x7f1eac007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:24.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.921+0000 7f1e9f7fe700 1 -- 172.21.15.46:0/535389027 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac011590 con 0x7f1eb815ce40 2024-01-27T01:03:24.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.921+0000 7f1e9f7fe700 1 -- 172.21.15.46:0/535389027 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac011710 con 0x7f1eb815ce40 2024-01-27T01:03:24.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.921+0000 7f1ebe1cf700 1 -- 172.21.15.46:0/535389027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb80c44a0 con 0x7f1eb815ce40 2024-01-27T01:03:24.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.921+0000 7f1e9f7fe700 1 -- 172.21.15.46:0/535389027 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac011ad0 con 0x7f1eb815ce40 2024-01-27T01:03:24.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.921+0000 7f1ebe1cf700 1 -- 172.21.15.46:0/535389027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb8155120 con 0x7f1eb815ce40 2024-01-27T01:03:24.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.922+0000 7f1ebe1cf700 1 -- 172.21.15.46:0/535389027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb804c710 con 0x7f1eb815ce40 2024-01-27T01:03:24.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.923+0000 7f1e9f7fe700 1 -- 172.21.15.46:0/535389027 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 110) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f1eac01b3b0 con 0x7f1eb815ce40 2024-01-27T01:03:24.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.923+0000 7f1e9f7fe700 1 --2- 172.21.15.46:0/535389027 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ea4067050 0x7f1ea4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:24.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.924+0000 7f1e9f7fe700 1 -- 172.21.15.46:0/535389027 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 7049+0+0 (secure 0 0 0) 0x7f1eac083eb0 con 0x7f1eb815ce40 2024-01-27T01:03:24.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.924+0000 7f1eb77fe700 1 -- 172.21.15.46:0/535389027 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ea4067050 msgr2=0x7f1ea4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:24.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.924+0000 7f1eb77fe700 1 --2- 172.21.15.46:0/535389027 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ea4067050 0x7f1ea4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:03:24.926 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:24.928+0000 7fa5b9ca21c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:03:24.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:24.928+0000 7f1e9f7fe700 1 -- 172.21.15.46:0/535389027 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1eac050b90 con 0x7f1eb815ce40 2024-01-27T01:03:25.100 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:25.101+0000 7fa5b9ca21c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:03:25.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.125+0000 7f1eb77fe700 1 -- 172.21.15.46:0/535389027 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ea4067050 msgr2=0x7f1ea4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:25.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.125+0000 7f1eb77fe700 1 --2- 172.21.15.46:0/535389027 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ea4067050 0x7f1ea4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:03:25.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.137+0000 7f1ebe1cf700 1 -- 172.21.15.46:0/535389027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1eb80cf920 con 0x7f1eb815ce40 2024-01-27T01:03:25.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.138+0000 7f1e9f7fe700 1 -- 172.21.15.46:0/535389027 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) v1 ==== 74+0+17118 (secure 0 0 0) 0x7f1eac017070 con 0x7f1eb815ce40 2024-01-27T01:03:25.137 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:25.138 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":192,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:03:21.686956+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-01-27T00:47:17.642961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-01-27T00:47:18.897371+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":192,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/273126119":"2024-01-28T00:58:22.118992+0000","172.21.15.106:0/683705968":"2024-01-28T00:58:22.118992+0000","172.21.15.106:0/211490639":"2024-01-28T00:58:22.118992+0000","172.21.15.106:0/1013704950":"2024-01-28T00:47:39.609727+0000","172.21.15.106:0/243674269":"2024-01-28T00:47:39.609727+0000","172.21.15.106:0/1532248390":"2024-01-28T00:58:22.118992+0000","172.21.15.106:6837/43388":"2024-01-28T00:47:39.609727+0000","172.21.15.106:6836/43388":"2024-01-28T00:47:39.609727+0000","172.21.15.106:0/1578971181":"2024-01-28T00:58:22.118992+0000","172.21.15.106:0/2043045162":"2024-01-28T00:47:39.609727+0000","172.21.15.106:0/1872294075":"2024-01-28T00:58:22.118992+0000","172.21.15.106:0/14745894":"2024-01-28T00:47:39.609727+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:03:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.140+0000 7f1ebe1cf700 1 -- 172.21.15.46:0/535389027 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ea4067050 msgr2=0x7f1ea4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:03:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.140+0000 7f1ebe1cf700 1 --2- 172.21.15.46:0/535389027 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ea4067050 0x7f1ea4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.141+0000 7f1ebe1cf700 1 -- 172.21.15.46:0/535389027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb815ce40 msgr2=0x7f1eb80c7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.141+0000 7f1ebe1cf700 1 --2- 172.21.15.46:0/535389027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb815ce40 0x7f1eb80c7750 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f1eac00c590 tx=0x7f1eac007f50 comp rx=0 tx=0).stop 2024-01-27T01:03:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.141+0000 7f1ebe1cf700 1 -- 172.21.15.46:0/535389027 shutdown_connections 2024-01-27T01:03:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.141+0000 7f1ebe1cf700 1 --2- 172.21.15.46:0/535389027 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ea4067050 0x7f1ea4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.141+0000 7f1ebe1cf700 1 --2- 172.21.15.46:0/535389027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1eb815d210 0x7f1eb8154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.141+0000 7f1ebe1cf700 1 --2- 172.21.15.46:0/535389027 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb80cc1c0 0x7f1eb815f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.141+0000 7f1ebe1cf700 1 --2- 172.21.15.46:0/535389027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb815ce40 0x7f1eb80c7750 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.141+0000 7f1ebe1cf700 1 -- 172.21.15.46:0/535389027 >> 172.21.15.46:0/535389027 conn(0x7f1eb806f840 msgr2=0x7f1eb80cba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:25.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.141+0000 7f1ebe1cf700 1 -- 172.21.15.46:0/535389027 shutdown_connections 2024-01-27T01:03:25.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.141+0000 7f1ebe1cf700 1 -- 172.21.15.46:0/535389027 wait complete. 2024-01-27T01:03:25.154 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-27T01:03:25.270 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:25.270+0000 7fd80323e1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:03:25.282 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:25.283+0000 7fa5b9ca21c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:03:25.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.290+0000 7f2db197b700 1 Processor -- start 2024-01-27T01:03:25.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.290+0000 7f2db197b700 1 -- start start 2024-01-27T01:03:25.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.291+0000 7f2db197b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2dac157340 0x7f2dac15bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.291+0000 7f2db197b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dac158ae0 con 0x7f2dac0b4180 2024-01-27T01:03:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.291+0000 7f2db197b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dac15b650 con 0x7f2dac0ae520 2024-01-27T01:03:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.291+0000 7f2db197b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dac15c020 con 0x7f2dac157340 2024-01-27T01:03:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.291+0000 7f2dab7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2dac0b4180 0x7f2dac157710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45782/0 (socket says 172.21.15.46:45782) 2024-01-27T01:03:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.292+0000 7f2dab7fe700 1 -- 172.21.15.46:0/218671873 learned_addr learned my addr 172.21.15.46:0/218671873 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.292+0000 7f2daaffd700 1 --2- 172.21.15.46:0/218671873 >> v2:172.21.15.106:3300/0 conn(0x7f2dac157340 0x7f2dac15bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.292+0000 7f2da9ffb700 1 -- 172.21.15.46:0/218671873 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877924410 0 0) 0x7f2dac158ae0 con 0x7f2dac0b4180 2024-01-27T01:03:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.292+0000 7f2da9ffb700 1 -- 172.21.15.46:0/218671873 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d94003570 con 0x7f2dac0b4180 2024-01-27T01:03:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.292+0000 7f2da9ffb700 1 -- 172.21.15.46:0/218671873 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810290214 0 0) 0x7f2dac15b650 con 0x7f2dac0ae520 2024-01-27T01:03:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.292+0000 7f2da9ffb700 1 -- 172.21.15.46:0/218671873 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dac158ae0 con 0x7f2dac0ae520 2024-01-27T01:03:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.292+0000 7f2daaffd700 1 -- 172.21.15.46:0/218671873 >> v1:172.21.15.106:6789/0 conn(0x7f2dac0ae520 legacy=0x7f2dac0b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.292+0000 7f2daaffd700 1 -- 172.21.15.46:0/218671873 >> v1:172.21.15.46:6789/0 conn(0x7f2dac0b4180 legacy=0x7f2dac157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.292+0000 7f2daaffd700 1 -- 172.21.15.46:0/218671873 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac158520 con 0x7f2dac157340 2024-01-27T01:03:25.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.292+0000 7f2daaffd700 1 --2- 172.21.15.46:0/218671873 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2dac157340 0x7f2dac15bae0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c009e60 tx=0x7f2d9c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ec3cb5af5aab74a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:25.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.292+0000 7f2da9ffb700 1 -- 172.21.15.46:0/218671873 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c00e070 con 0x7f2dac157340 2024-01-27T01:03:25.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.292+0000 7f2da9ffb700 1 -- 172.21.15.46:0/218671873 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9c00bab0 con 0x7f2dac157340 2024-01-27T01:03:25.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.292+0000 7f2da9ffb700 1 -- 172.21.15.46:0/218671873 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c00be50 con 0x7f2dac157340 2024-01-27T01:03:25.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.293+0000 7f2db197b700 1 -- 172.21.15.46:0/218671873 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2dac157340 msgr2=0x7f2dac15bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:25.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.293+0000 7f2db197b700 1 --2- 172.21.15.46:0/218671873 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2dac157340 0x7f2dac15bae0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c009e60 tx=0x7f2d9c009780 comp rx=0 tx=0).stop 2024-01-27T01:03:25.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.293+0000 7f2db197b700 1 -- 172.21.15.46:0/218671873 shutdown_connections 2024-01-27T01:03:25.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.293+0000 7f2db197b700 1 --2- 172.21.15.46:0/218671873 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2dac157340 0x7f2dac15bae0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.293+0000 7f2db197b700 1 -- 172.21.15.46:0/218671873 >> 172.21.15.46:0/218671873 conn(0x7f2dac0ce530 msgr2=0x7f2dac0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:25.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.293+0000 7f2db197b700 1 -- 172.21.15.46:0/218671873 shutdown_connections 2024-01-27T01:03:25.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.293+0000 7f2db197b700 1 -- 172.21.15.46:0/218671873 wait complete. 2024-01-27T01:03:25.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.295+0000 7f2db197b700 1 Processor -- start 2024-01-27T01:03:25.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.295+0000 7f2db197b700 1 -- start start 2024-01-27T01:03:25.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.295+0000 7f2db197b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2dac0ae520 0x7f2dac154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:25.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.295+0000 7f2db197b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2dac0b4180 0x7f2dac06db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:25.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.295+0000 7f2db197b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dac157340 0x7f2dac06e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:25.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.295+0000 7f2db197b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac0b2d80 con 0x7f2dac157340 2024-01-27T01:03:25.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.295+0000 7f2db197b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac06a9b0 con 0x7f2dac0b4180 2024-01-27T01:03:25.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.295+0000 7f2db197b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac0b2450 con 0x7f2dac0ae520 2024-01-27T01:03:25.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.295+0000 7f2dab7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dac157340 0x7f2dac06e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:25.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.296+0000 7f2daa7fc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2dac0b4180 0x7f2dac06db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:25.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.296+0000 7f2daaffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2dac0ae520 0x7f2dac154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:25.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.296+0000 7f2dab7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dac157340 0x7f2dac06e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47262/0 (socket says 172.21.15.46:47262) 2024-01-27T01:03:25.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.296+0000 7f2dab7fe700 1 -- 172.21.15.46:0/3215004573 learned_addr learned my addr 172.21.15.46:0/3215004573 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:25.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.296+0000 7f2dab7fe700 1 -- 172.21.15.46:0/3215004573 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2dac0ae520 msgr2=0x7f2dac154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:25.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.296+0000 7f2dab7fe700 1 --2- 172.21.15.46:0/3215004573 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2dac0ae520 0x7f2dac154aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.296+0000 7f2dab7fe700 1 -- 172.21.15.46:0/3215004573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2dac0b4180 msgr2=0x7f2dac06db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:25.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.296+0000 7f2dab7fe700 1 --2- 172.21.15.46:0/3215004573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2dac0b4180 0x7f2dac06db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.296+0000 7f2dab7fe700 1 -- 172.21.15.46:0/3215004573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9c009a20 con 0x7f2dac157340 2024-01-27T01:03:25.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.296+0000 7f2daa7fc700 1 --2- 172.21.15.46:0/3215004573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2dac0b4180 0x7f2dac06db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:25.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.296+0000 7f2dab7fe700 1 --2- 172.21.15.46:0/3215004573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dac157340 0x7f2dac06e070 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f2da400c7c0 tx=0x7f2da4007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:25.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.296+0000 7f2d92ffd700 1 -- 172.21.15.46:0/3215004573 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4005c00 con 0x7f2dac157340 2024-01-27T01:03:25.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.296+0000 7f2d92ffd700 1 -- 172.21.15.46:0/3215004573 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da4005d60 con 0x7f2dac157340 2024-01-27T01:03:25.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.296+0000 7f2db197b700 1 -- 172.21.15.46:0/3215004573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dac0b2730 con 0x7f2dac157340 2024-01-27T01:03:25.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.296+0000 7f2d92ffd700 1 -- 172.21.15.46:0/3215004573 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4007320 con 0x7f2dac157340 2024-01-27T01:03:25.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.296+0000 7f2db197b700 1 -- 172.21.15.46:0/3215004573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dac0b3210 con 0x7f2dac157340 2024-01-27T01:03:25.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.297+0000 7f2db197b700 1 -- 172.21.15.46:0/3215004573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dac04c710 con 0x7f2dac157340 2024-01-27T01:03:25.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.299+0000 7f2d92ffd700 1 -- 172.21.15.46:0/3215004573 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 110) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f2da401b3b0 con 0x7f2dac157340 2024-01-27T01:03:25.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.299+0000 7f2d92ffd700 1 --2- 172.21.15.46:0/3215004573 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2d98067050 0x7f2d98069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:25.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.299+0000 7f2d92ffd700 1 -- 172.21.15.46:0/3215004573 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 7049+0+0 (secure 0 0 0) 0x7f2da40840e0 con 0x7f2dac157340 2024-01-27T01:03:25.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.299+0000 7f2daaffd700 1 -- 172.21.15.46:0/3215004573 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2d98067050 msgr2=0x7f2d98069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:25.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.299+0000 7f2daaffd700 1 --2- 172.21.15.46:0/3215004573 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2d98067050 0x7f2d98069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:03:25.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.308+0000 7f2d92ffd700 1 -- 172.21.15.46:0/3215004573 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2da4050d00 con 0x7f2dac157340 2024-01-27T01:03:25.488 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:25.489+0000 7fd80323e1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:03:25.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.500+0000 7f2daaffd700 1 -- 172.21.15.46:0/3215004573 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2d98067050 msgr2=0x7f2d98069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:25.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.500+0000 7f2daaffd700 1 --2- 172.21.15.46:0/3215004573 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2d98067050 0x7f2d98069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:03:25.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.558+0000 7f2db197b700 1 -- 172.21.15.46:0/3215004573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f2dac158800 con 0x7f2dac157340 2024-01-27T01:03:25.558 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-27T01:03:25.559+0000 7f33ffe86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-27T01:03:25.562 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:25.562+0000 7fd80323e1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:03:25.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.564+0000 7f2d92ffd700 1 -- 172.21.15.46:0/3215004573 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v98) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2da4017090 con 0x7f2dac157340 2024-01-27T01:03:25.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.566+0000 7f2db197b700 1 -- 172.21.15.46:0/3215004573 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2d98067050 msgr2=0x7f2d98069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:03:25.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.566+0000 7f2db197b700 1 --2- 172.21.15.46:0/3215004573 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2d98067050 0x7f2d98069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.566+0000 7f2db197b700 1 -- 172.21.15.46:0/3215004573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dac157340 msgr2=0x7f2dac06e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:25.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.566+0000 7f2db197b700 1 --2- 172.21.15.46:0/3215004573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dac157340 0x7f2dac06e070 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f2da400c7c0 tx=0x7f2da4007db0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.566+0000 7f2db197b700 1 -- 172.21.15.46:0/3215004573 shutdown_connections 2024-01-27T01:03:25.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.566+0000 7f2db197b700 1 --2- 172.21.15.46:0/3215004573 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2d98067050 0x7f2d98069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.566+0000 7f2db197b700 1 --2- 172.21.15.46:0/3215004573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dac157340 0x7f2dac06e070 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.566+0000 7f2db197b700 1 --2- 172.21.15.46:0/3215004573 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2dac0ae520 0x7f2dac154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.566+0000 7f2db197b700 1 --2- 172.21.15.46:0/3215004573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2dac0b4180 0x7f2dac06db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.566+0000 7f2db197b700 1 -- 172.21.15.46:0/3215004573 >> 172.21.15.46:0/3215004573 conn(0x7f2dac0ce530 msgr2=0x7f2dac155960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:25.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.567+0000 7f2db197b700 1 -- 172.21.15.46:0/3215004573 shutdown_connections 2024-01-27T01:03:25.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.567+0000 7f2db197b700 1 -- 172.21.15.46:0/3215004573 wait complete. 2024-01-27T01:03:25.567 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-27T01:03:25.580 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-27T01:03:25.601 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-27T01:03:25.720 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:25.720+0000 7fd80323e1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:03:25.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.722+0000 7f3fcf629700 1 Processor -- start 2024-01-27T01:03:25.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.722+0000 7f3fcf629700 1 -- start start 2024-01-27T01:03:25.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fcf629700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fc8159750 0x7f3fc815df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:25.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fcf629700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3fc8069cd0 0x7f3fc80b1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:25.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fcf629700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3fc80b3a10 0x7f3fc80b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:25.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fcf629700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fc80b2510 con 0x7f3fc80b3a10 2024-01-27T01:03:25.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fcf629700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fc8069840 con 0x7f3fc8069cd0 2024-01-27T01:03:25.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fcf629700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc806a140 con 0x7f3fc8159750 2024-01-27T01:03:25.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fcdbc6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3fc80b3a10 0x7f3fc80b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:25.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fccbc4700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3fc8069cd0 0x7f3fc80b1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:25.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fcd3c5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fc8159750 0x7f3fc815df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:25.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fcdbc6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3fc80b3a10 0x7f3fc80b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47300/0 (socket says 172.21.15.46:47300) 2024-01-27T01:03:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fccbc4700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3fc8069cd0 0x7f3fc80b1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:45474/0 (socket says 172.21.15.46:45474) 2024-01-27T01:03:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fcd3c5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fc8159750 0x7f3fc815df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:54092/0 (socket says 172.21.15.46:54092) 2024-01-27T01:03:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fcdbc6700 1 -- 172.21.15.46:0/3991610097 learned_addr learned my addr 172.21.15.46:0/3991610097 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fccbc4700 1 -- 172.21.15.46:0/3991610097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fc8159750 msgr2=0x7f3fc815df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fccbc4700 1 --2- 172.21.15.46:0/3991610097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fc8159750 0x7f3fc815df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fccbc4700 1 -- 172.21.15.46:0/3991610097 >> v2:172.21.15.46:3300/0 conn(0x7f3fc80b3a10 msgr2=0x7f3fc80b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fccbc4700 1 --2- 172.21.15.46:0/3991610097 >> v2:172.21.15.46:3300/0 conn(0x7f3fc80b3a10 0x7f3fc80b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.723+0000 7f3fccbc4700 1 -- 172.21.15.46:0/3991610097 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc80b1f90 con 0x7f3fc8069cd0 2024-01-27T01:03:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.724+0000 7f3fcdbc6700 1 --2- 172.21.15.46:0/3991610097 >> v2:172.21.15.46:3300/0 conn(0x7f3fc80b3a10 0x7f3fc80b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.724+0000 7f3fcd3c5700 1 --2- 172.21.15.46:0/3991610097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fc8159750 0x7f3fc815df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.724+0000 7f3fccbc4700 1 --2- 172.21.15.46:0/3991610097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fc8069cd0 0x7f3fc80b1a50 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f3fc80cf6c0 tx=0x7f3fb80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7ecccdd574b5a61 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.724+0000 7f3fbffff700 1 -- 172.21.15.46:0/3991610097 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8011630 con 0x7f3fc8069cd0 2024-01-27T01:03:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.724+0000 7f3fbffff700 1 -- 172.21.15.46:0/3991610097 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb80117b0 con 0x7f3fc8069cd0 2024-01-27T01:03:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.724+0000 7f3fbffff700 1 -- 172.21.15.46:0/3991610097 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8011b70 con 0x7f3fc8069cd0 2024-01-27T01:03:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.725+0000 7f3fcf629700 1 -- 172.21.15.46:0/3991610097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fc8069cd0 msgr2=0x7f3fc80b1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.725+0000 7f3fcf629700 1 --2- 172.21.15.46:0/3991610097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fc8069cd0 0x7f3fc80b1a50 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f3fc80cf6c0 tx=0x7f3fb80091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.725+0000 7f3fcf629700 1 -- 172.21.15.46:0/3991610097 shutdown_connections 2024-01-27T01:03:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.725+0000 7f3fcf629700 1 --2- 172.21.15.46:0/3991610097 >> v2:172.21.15.46:3300/0 conn(0x7f3fc80b3a10 0x7f3fc80b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.725+0000 7f3fcf629700 1 --2- 172.21.15.46:0/3991610097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fc8159750 0x7f3fc815df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.725+0000 7f3fcf629700 1 --2- 172.21.15.46:0/3991610097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fc8069cd0 0x7f3fc80b1a50 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.725+0000 7f3fcf629700 1 -- 172.21.15.46:0/3991610097 >> 172.21.15.46:0/3991610097 conn(0x7f3fc8067930 msgr2=0x7f3fc8067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.725+0000 7f3fcf629700 1 -- 172.21.15.46:0/3991610097 shutdown_connections 2024-01-27T01:03:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.725+0000 7f3fcf629700 1 -- 172.21.15.46:0/3991610097 wait complete. 2024-01-27T01:03:25.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.726+0000 7f3fcf629700 1 Processor -- start 2024-01-27T01:03:25.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.726+0000 7f3fcf629700 1 -- start start 2024-01-27T01:03:25.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.727+0000 7f3fcf629700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fc8069cd0 0x7f3fc8145fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:25.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.727+0000 7f3fcf629700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fc80b3a10 0x7f3fc80aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:25.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.727+0000 7f3fcf629700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fc8159750 0x7f3fc80af190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:25.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.727+0000 7f3fcf629700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc8159e70 con 0x7f3fc8159750 2024-01-27T01:03:25.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.727+0000 7f3fcf629700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc8159b20 con 0x7f3fc80b3a10 2024-01-27T01:03:25.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.727+0000 7f3fcf629700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc80ce730 con 0x7f3fc8069cd0 2024-01-27T01:03:25.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.727+0000 7f3fcdbc6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fc8159750 0x7f3fc80af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:25.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.727+0000 7f3fcdbc6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fc8159750 0x7f3fc80af190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47312/0 (socket says 172.21.15.46:47312) 2024-01-27T01:03:25.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.727+0000 7f3fcdbc6700 1 -- 172.21.15.46:0/710704735 learned_addr learned my addr 172.21.15.46:0/710704735 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:25.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.727+0000 7f3fccbc4700 1 --2- 172.21.15.46:0/710704735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fc80b3a10 0x7f3fc80aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:25.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.728+0000 7f3fcd3c5700 1 --2- 172.21.15.46:0/710704735 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fc8069cd0 0x7f3fc8145fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:25.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.728+0000 7f3fcdbc6700 1 -- 172.21.15.46:0/710704735 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fc8069cd0 msgr2=0x7f3fc8145fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:25.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.728+0000 7f3fcdbc6700 1 --2- 172.21.15.46:0/710704735 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fc8069cd0 0x7f3fc8145fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.728+0000 7f3fcdbc6700 1 -- 172.21.15.46:0/710704735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fc80b3a10 msgr2=0x7f3fc80aec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:25.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.728+0000 7f3fcdbc6700 1 --2- 172.21.15.46:0/710704735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fc80b3a10 0x7f3fc80aec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.728+0000 7f3fcdbc6700 1 -- 172.21.15.46:0/710704735 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc0009ae0 con 0x7f3fc8159750 2024-01-27T01:03:25.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.728+0000 7f3fcdbc6700 1 --2- 172.21.15.46:0/710704735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fc8159750 0x7f3fc80af190 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0009f30 tx=0x7f3fc000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:25.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.728+0000 7f3fbd7fa700 1 -- 172.21.15.46:0/710704735 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc0018070 con 0x7f3fc8159750 2024-01-27T01:03:25.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.728+0000 7f3fbd7fa700 1 -- 172.21.15.46:0/710704735 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc000f040 con 0x7f3fc8159750 2024-01-27T01:03:25.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.728+0000 7f3fcf629700 1 -- 172.21.15.46:0/710704735 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc4009420 con 0x7f3fc8159750 2024-01-27T01:03:25.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.728+0000 7f3fbd7fa700 1 -- 172.21.15.46:0/710704735 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc000cb70 con 0x7f3fc8159750 2024-01-27T01:03:25.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.728+0000 7f3fcf629700 1 -- 172.21.15.46:0/710704735 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb8009420 con 0x7f3fc8159750 2024-01-27T01:03:25.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.729+0000 7f3fcf629700 1 -- 172.21.15.46:0/710704735 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc8071290 con 0x7f3fc8159750 2024-01-27T01:03:25.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.730+0000 7f3fbd7fa700 1 -- 172.21.15.46:0/710704735 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 110) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f3fc0003020 con 0x7f3fc8159750 2024-01-27T01:03:25.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.730+0000 7f3fbd7fa700 1 --2- 172.21.15.46:0/710704735 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fac067050 0x7f3fac069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:25.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.731+0000 7f3fbd7fa700 1 -- 172.21.15.46:0/710704735 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 7091+0+0 (secure 0 0 0) 0x7f3fc00850d0 con 0x7f3fc8159750 2024-01-27T01:03:25.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.731+0000 7f3fcd3c5700 1 -- 172.21.15.46:0/710704735 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fac067050 msgr2=0x7f3fac069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:25.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.731+0000 7f3fcd3c5700 1 --2- 172.21.15.46:0/710704735 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fac067050 0x7f3fac069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:03:25.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.736+0000 7f3fbd7fa700 1 -- 172.21.15.46:0/710704735 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3fc0051bb0 con 0x7f3fc8159750 2024-01-27T01:03:25.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.742+0000 7f3fbd7fa700 1 -- 172.21.15.46:0/710704735 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 111) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f3fc0083070 con 0x7f3fc8159750 2024-01-27T01:03:25.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.742+0000 7f3fbd7fa700 1 -- 172.21.15.46:0/710704735 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fac067050 msgr2=0x7f3fac069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:03:25.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.742+0000 7f3fbd7fa700 1 --2- 172.21.15.46:0/710704735 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fac067050 0x7f3fac069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:25.800 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:25.800+0000 7fd77eb26700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:25.800 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:25.800+0000 7fd77eb26700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:25.800 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:25.800+0000 7fd77eb26700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:25.800 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:25.800+0000 7fd77eb26700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:25.800 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:25.800+0000 7fd77eb26700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:25.801 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:25.801+0000 7fd78232d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:25.801 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:25.801+0000 7fd78232d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:25.801 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:25.801+0000 7fd78232d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:25.801 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:25.801+0000 7fd78232d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:25.801 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:25.801+0000 7fd78232d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:25.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:25.996+0000 7f3fcf629700 1 -- 172.21.15.46:0/710704735 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3fc804c710 con 0x7f3fc8159750 2024-01-27T01:03:26.121 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:26.122+0000 7fa5b9ca21c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:03:26.375 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:26.376+0000 7fa5b9ca21c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:03:26.460 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:26.460+0000 7fa5b9ca21c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:03:26.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.565+0000 7f3fbd7fa700 1 -- 172.21.15.46:0/710704735 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v99) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3fc006ef30 con 0x7f3fc8159750 2024-01-27T01:03:26.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.567+0000 7f3fcf629700 1 -- 172.21.15.46:0/710704735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fc8159750 msgr2=0x7f3fc80af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:26.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.567+0000 7f3fcf629700 1 --2- 172.21.15.46:0/710704735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fc8159750 0x7f3fc80af190 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0009f30 tx=0x7f3fc000b040 comp rx=0 tx=0).stop 2024-01-27T01:03:26.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.567+0000 7f3fcf629700 1 -- 172.21.15.46:0/710704735 shutdown_connections 2024-01-27T01:03:26.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.567+0000 7f3fcf629700 1 --2- 172.21.15.46:0/710704735 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fac067050 0x7f3fac069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:26.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.567+0000 7f3fcf629700 1 --2- 172.21.15.46:0/710704735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fc8159750 0x7f3fc80af190 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:26.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.567+0000 7f3fcf629700 1 --2- 172.21.15.46:0/710704735 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fc8069cd0 0x7f3fc8145fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:26.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.567+0000 7f3fcf629700 1 --2- 172.21.15.46:0/710704735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fc80b3a10 0x7f3fc80aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:26.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.568+0000 7f3fcf629700 1 -- 172.21.15.46:0/710704735 >> 172.21.15.46:0/710704735 conn(0x7f3fc8067930 msgr2=0x7f3fc8158a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:26.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.568+0000 7f3fcf629700 1 -- 172.21.15.46:0/710704735 shutdown_connections 2024-01-27T01:03:26.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.568+0000 7f3fcf629700 1 -- 172.21.15.46:0/710704735 wait complete. 2024-01-27T01:03:26.580 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-27T01:03:26.639 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:26.640+0000 7fa5b9ca21c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:03:26.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.765+0000 7f2eff6db700 1 Processor -- start 2024-01-27T01:03:26.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.765+0000 7f2eff6db700 1 -- start start 2024-01-27T01:03:26.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.766+0000 7f2eff6db700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2ef8159b20 0x7f2ef80ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:26.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.766+0000 7f2eff6db700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2ef80ae340 0x7f2ef815d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:26.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.766+0000 7f2eff6db700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef80691c0 con 0x7f2ef8159750 2024-01-27T01:03:26.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.766+0000 7f2eff6db700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef806a5e0 con 0x7f2ef8159b20 2024-01-27T01:03:26.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.766+0000 7f2eff6db700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef815dfd0 con 0x7f2ef80ae340 2024-01-27T01:03:26.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.766+0000 7f2efcc76700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2ef8159b20 0x7f2ef80ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:26.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.766+0000 7f2efd477700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2ef8159750 0x7f2ef8069650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45798/0 (socket says 172.21.15.46:45798) 2024-01-27T01:03:26.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.766+0000 7f2efd477700 1 -- 172.21.15.46:0/836646999 learned_addr learned my addr 172.21.15.46:0/836646999 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:26.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.766+0000 7f2efdc78700 1 --2- 172.21.15.46:0/836646999 >> v2:172.21.15.106:3300/0 conn(0x7f2ef80ae340 0x7f2ef815d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:26.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.766+0000 7f2efcc76700 1 -- 172.21.15.46:0/836646999 >> v2:172.21.15.106:3300/0 conn(0x7f2ef80ae340 msgr2=0x7f2ef815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:26.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.766+0000 7f2efcc76700 1 --2- 172.21.15.46:0/836646999 >> v2:172.21.15.106:3300/0 conn(0x7f2ef80ae340 0x7f2ef815d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:26.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.766+0000 7f2efcc76700 1 -- 172.21.15.46:0/836646999 >> v1:172.21.15.46:6789/0 conn(0x7f2ef8159750 legacy=0x7f2ef8069650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:26.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.766+0000 7f2efcc76700 1 -- 172.21.15.46:0/836646999 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef8068b70 con 0x7f2ef8159b20 2024-01-27T01:03:26.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.766+0000 7f2eeffff700 1 -- 172.21.15.46:0/836646999 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658477609 0 0) 0x7f2ef80691c0 con 0x7f2ef8159750 2024-01-27T01:03:26.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.766+0000 7f2efcc76700 1 --2- 172.21.15.46:0/836646999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef8159b20 0x7f2ef80ade00 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f2ef80c9f20 tx=0x7f2ee80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1484f68df89b0ac server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:26.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.767+0000 7f2eeffff700 1 -- 172.21.15.46:0/836646999 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee800b6b0 con 0x7f2ef8159b20 2024-01-27T01:03:26.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.767+0000 7f2eeffff700 1 -- 172.21.15.46:0/836646999 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee800b810 con 0x7f2ef8159b20 2024-01-27T01:03:26.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.767+0000 7f2eeffff700 1 -- 172.21.15.46:0/836646999 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee800bbb0 con 0x7f2ef8159b20 2024-01-27T01:03:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.767+0000 7f2eff6db700 1 -- 172.21.15.46:0/836646999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef8159b20 msgr2=0x7f2ef80ade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.767+0000 7f2eff6db700 1 --2- 172.21.15.46:0/836646999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef8159b20 0x7f2ef80ade00 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f2ef80c9f20 tx=0x7f2ee80091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.767+0000 7f2eff6db700 1 -- 172.21.15.46:0/836646999 shutdown_connections 2024-01-27T01:03:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.767+0000 7f2eff6db700 1 --2- 172.21.15.46:0/836646999 >> v2:172.21.15.106:3300/0 conn(0x7f2ef80ae340 0x7f2ef815d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.767+0000 7f2eff6db700 1 --2- 172.21.15.46:0/836646999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef8159b20 0x7f2ef80ade00 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.767+0000 7f2eff6db700 1 -- 172.21.15.46:0/836646999 >> 172.21.15.46:0/836646999 conn(0x7f2ef80ce530 msgr2=0x7f2ef80ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.767+0000 7f2eff6db700 1 -- 172.21.15.46:0/836646999 shutdown_connections 2024-01-27T01:03:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.767+0000 7f2eff6db700 1 -- 172.21.15.46:0/836646999 wait complete. 2024-01-27T01:03:26.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.768+0000 7f2eff6db700 1 Processor -- start 2024-01-27T01:03:26.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.769+0000 7f2eff6db700 1 -- start start 2024-01-27T01:03:26.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.769+0000 7f2eff6db700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ef80ae340 0x7f2ef80b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:26.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.769+0000 7f2eff6db700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef8159750 0x7f2ef81573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:26.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.769+0000 7f2eff6db700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ef8159b20 0x7f2ef816d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:26.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.769+0000 7f2eff6db700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef80b2b10 con 0x7f2ef8159750 2024-01-27T01:03:26.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.769+0000 7f2eff6db700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef816e670 con 0x7f2ef80ae340 2024-01-27T01:03:26.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.769+0000 7f2eff6db700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef80b2d40 con 0x7f2ef8159b20 2024-01-27T01:03:26.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.770+0000 7f2efcc76700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef8159750 0x7f2ef81573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:26.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.770+0000 7f2efcc76700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef8159750 0x7f2ef81573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47340/0 (socket says 172.21.15.46:47340) 2024-01-27T01:03:26.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.770+0000 7f2efcc76700 1 -- 172.21.15.46:0/3635021651 learned_addr learned my addr 172.21.15.46:0/3635021651 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:26.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.770+0000 7f2efd477700 1 --2- 172.21.15.46:0/3635021651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ef80ae340 0x7f2ef80b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:26.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.770+0000 7f2efdc78700 1 --2- 172.21.15.46:0/3635021651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ef8159b20 0x7f2ef816d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:26.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.770+0000 7f2efcc76700 1 -- 172.21.15.46:0/3635021651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ef8159b20 msgr2=0x7f2ef816d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:26.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.770+0000 7f2efcc76700 1 --2- 172.21.15.46:0/3635021651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ef8159b20 0x7f2ef816d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:26.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.770+0000 7f2efcc76700 1 -- 172.21.15.46:0/3635021651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ef80ae340 msgr2=0x7f2ef80b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:26.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.770+0000 7f2efcc76700 1 --2- 172.21.15.46:0/3635021651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ef80ae340 0x7f2ef80b3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:26.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.770+0000 7f2efcc76700 1 -- 172.21.15.46:0/3635021651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee8009420 con 0x7f2ef8159750 2024-01-27T01:03:26.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.770+0000 7f2efcc76700 1 --2- 172.21.15.46:0/3635021651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef8159750 0x7f2ef81573b0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8013000 tx=0x7f2ee8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:26.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.770+0000 7f2eed7fa700 1 -- 172.21.15.46:0/3635021651 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee800bb90 con 0x7f2ef8159750 2024-01-27T01:03:26.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.770+0000 7f2eed7fa700 1 -- 172.21.15.46:0/3635021651 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee8011420 con 0x7f2ef8159750 2024-01-27T01:03:26.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.770+0000 7f2eff6db700 1 -- 172.21.15.46:0/3635021651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef80b3020 con 0x7f2ef8159750 2024-01-27T01:03:26.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.770+0000 7f2eed7fa700 1 -- 172.21.15.46:0/3635021651 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee80117c0 con 0x7f2ef8159750 2024-01-27T01:03:26.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.771+0000 7f2eff6db700 1 -- 172.21.15.46:0/3635021651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef80b33b0 con 0x7f2ef8159750 2024-01-27T01:03:26.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.771+0000 7f2eff6db700 1 -- 172.21.15.46:0/3635021651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef80b3820 con 0x7f2ef8159750 2024-01-27T01:03:26.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.772+0000 7f2eed7fa700 1 -- 172.21.15.46:0/3635021651 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 112) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f2ee8011bb0 con 0x7f2ef8159750 2024-01-27T01:03:26.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.772+0000 7f2eed7fa700 1 --2- 172.21.15.46:0/3635021651 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2edc066d80 0x7f2edc069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:26.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.772+0000 7f2eed7fa700 1 -- 172.21.15.46:0/3635021651 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 7435+0+0 (secure 0 0 0) 0x7f2ee8086000 con 0x7f2ef8159750 2024-01-27T01:03:26.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.772+0000 7f2efd477700 1 --2- 172.21.15.46:0/3635021651 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2edc066d80 0x7f2edc069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:26.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.773+0000 7f2efd477700 1 --2- 172.21.15.46:0/3635021651 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2edc066d80 0x7f2edc069230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4007fd0 tx=0x7f2ef4007d30 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:26.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.777+0000 7f2eed7fa700 1 -- 172.21.15.46:0/3635021651 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2ef80b3820 con 0x7f2ef8159750 2024-01-27T01:03:26.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:26.979+0000 7f2eff6db700 1 -- 172.21.15.46:0/3635021651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2ef80b3820 con 0x7f2ef8159750 2024-01-27T01:03:27.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.751+0000 7f2eed7fa700 1 -- 172.21.15.46:0/3635021651 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v195) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2ef80b3820 con 0x7f2ef8159750 2024-01-27T01:03:27.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.755+0000 7f2eff6db700 1 -- 172.21.15.46:0/3635021651 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2edc066d80 msgr2=0x7f2edc069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:27.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.755+0000 7f2eff6db700 1 --2- 172.21.15.46:0/3635021651 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2edc066d80 0x7f2edc069230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4007fd0 tx=0x7f2ef4007d30 comp rx=0 tx=0).stop 2024-01-27T01:03:27.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.755+0000 7f2eff6db700 1 -- 172.21.15.46:0/3635021651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef8159750 msgr2=0x7f2ef81573b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:27.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.755+0000 7f2eff6db700 1 --2- 172.21.15.46:0/3635021651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef8159750 0x7f2ef81573b0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8013000 tx=0x7f2ee8017000 comp rx=0 tx=0).stop 2024-01-27T01:03:27.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.755+0000 7f2eff6db700 1 -- 172.21.15.46:0/3635021651 shutdown_connections 2024-01-27T01:03:27.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.755+0000 7f2eff6db700 1 --2- 172.21.15.46:0/3635021651 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2edc066d80 0x7f2edc069230 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:27.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.755+0000 7f2eff6db700 1 --2- 172.21.15.46:0/3635021651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ef8159b20 0x7f2ef816d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:27.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.755+0000 7f2eff6db700 1 --2- 172.21.15.46:0/3635021651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ef80ae340 0x7f2ef80b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:27.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.755+0000 7f2eff6db700 1 --2- 172.21.15.46:0/3635021651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ef8159750 0x7f2ef81573b0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:27.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.755+0000 7f2eff6db700 1 -- 172.21.15.46:0/3635021651 >> 172.21.15.46:0/3635021651 conn(0x7f2ef80ce530 msgr2=0x7f2ef806d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:27.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.756+0000 7f2eff6db700 1 -- 172.21.15.46:0/3635021651 shutdown_connections 2024-01-27T01:03:27.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.756+0000 7f2eff6db700 1 -- 172.21.15.46:0/3635021651 wait complete. 2024-01-27T01:03:27.756 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-01-27T01:03:27.771 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-27T01:03:27.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.953+0000 7f9888c31700 1 Processor -- start 2024-01-27T01:03:27.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.953+0000 7f9888c31700 1 -- start start 2024-01-27T01:03:27.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.954+0000 7f9888c31700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f988415ce40 0x7f98840d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:27.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.954+0000 7f9888c31700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f988415d210 0x7f98840cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:27.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.954+0000 7f9888c31700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f98840d1390 0x7f98840cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:27.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.954+0000 7f9888c31700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98840cff70 con 0x7f988415ce40 2024-01-27T01:03:27.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.954+0000 7f9888c31700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98840cf400 con 0x7f98840d1390 2024-01-27T01:03:27.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.954+0000 7f9888c31700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98840cf660 con 0x7f988415d210 2024-01-27T01:03:27.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.955+0000 7f988259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f988415ce40 0x7f98840d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:27.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.955+0000 7f988259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f988415ce40 0x7f98840d0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47344/0 (socket says 172.21.15.46:47344) 2024-01-27T01:03:27.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.955+0000 7f988259c700 1 -- 172.21.15.46:0/3423029555 learned_addr learned my addr 172.21.15.46:0/3423029555 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:27.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.955+0000 7f9881d9b700 1 --2- 172.21.15.46:0/3423029555 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f988415d210 0x7f98840cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.955+0000 7f9882d9d700 1 --2- 172.21.15.46:0/3423029555 >> v2:172.21.15.106:3300/0 conn(0x7f98840d1390 0x7f98840cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.955+0000 7f988259c700 1 -- 172.21.15.46:0/3423029555 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f988415d210 msgr2=0x7f98840cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.955+0000 7f988259c700 1 --2- 172.21.15.46:0/3423029555 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f988415d210 0x7f98840cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.955+0000 7f988259c700 1 -- 172.21.15.46:0/3423029555 >> v2:172.21.15.106:3300/0 conn(0x7f98840d1390 msgr2=0x7f98840cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.955+0000 7f988259c700 1 --2- 172.21.15.46:0/3423029555 >> v2:172.21.15.106:3300/0 conn(0x7f98840d1390 0x7f98840cbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.955+0000 7f988259c700 1 -- 172.21.15.46:0/3423029555 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98840d0500 con 0x7f988415ce40 2024-01-27T01:03:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.955+0000 7f9882d9d700 1 --2- 172.21.15.46:0/3423029555 >> v2:172.21.15.106:3300/0 conn(0x7f98840d1390 0x7f98840cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:03:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.955+0000 7f988259c700 1 --2- 172.21.15.46:0/3423029555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f988415ce40 0x7f98840d0e50 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f9884089b70 tx=0x7f9874009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=741e9765b852b4d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.955+0000 7f988159a700 1 -- 172.21.15.46:0/3423029555 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987400b860 con 0x7f988415ce40 2024-01-27T01:03:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.955+0000 7f988159a700 1 -- 172.21.15.46:0/3423029555 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987400b9c0 con 0x7f988415ce40 2024-01-27T01:03:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.955+0000 7f988159a700 1 -- 172.21.15.46:0/3423029555 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987400bd60 con 0x7f988415ce40 2024-01-27T01:03:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.956+0000 7f9888c31700 1 -- 172.21.15.46:0/3423029555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f988415ce40 msgr2=0x7f98840d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.956+0000 7f9888c31700 1 --2- 172.21.15.46:0/3423029555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f988415ce40 0x7f98840d0e50 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f9884089b70 tx=0x7f9874009250 comp rx=0 tx=0).stop 2024-01-27T01:03:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.956+0000 7f9888c31700 1 -- 172.21.15.46:0/3423029555 shutdown_connections 2024-01-27T01:03:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.956+0000 7f9888c31700 1 --2- 172.21.15.46:0/3423029555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f988415ce40 0x7f98840d0e50 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:27.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.956+0000 7f9888c31700 1 --2- 172.21.15.46:0/3423029555 >> v2:172.21.15.106:3300/0 conn(0x7f98840d1390 0x7f98840cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:27.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.956+0000 7f9888c31700 1 --2- 172.21.15.46:0/3423029555 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f988415d210 0x7f98840cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:27.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.956+0000 7f9888c31700 1 -- 172.21.15.46:0/3423029555 >> 172.21.15.46:0/3423029555 conn(0x7f988406f840 msgr2=0x7f98840d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:27.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.956+0000 7f9888c31700 1 -- 172.21.15.46:0/3423029555 shutdown_connections 2024-01-27T01:03:27.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.956+0000 7f9888c31700 1 -- 172.21.15.46:0/3423029555 wait complete. 2024-01-27T01:03:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.957+0000 7f9888c31700 1 Processor -- start 2024-01-27T01:03:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.957+0000 7f9888c31700 1 -- start start 2024-01-27T01:03:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.958+0000 7f9888c31700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98840d1390 0x7f988416e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.958+0000 7f9888c31700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f988415ce40 0x7f988415ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.958+0000 7f9888c31700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f988415d210 0x7f98841554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.958+0000 7f9888c31700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988416e760 con 0x7f988415ce40 2024-01-27T01:03:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.958+0000 7f9888c31700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98840c4270 con 0x7f988415d210 2024-01-27T01:03:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.958+0000 7f9888c31700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98840c4550 con 0x7f98840d1390 2024-01-27T01:03:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.958+0000 7f9881d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f988415ce40 0x7f988415ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.958+0000 7f9881d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f988415ce40 0x7f988415ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47356/0 (socket says 172.21.15.46:47356) 2024-01-27T01:03:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.958+0000 7f9881d9b700 1 -- 172.21.15.46:0/3149259028 learned_addr learned my addr 172.21.15.46:0/3149259028 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.958+0000 7f988259c700 1 --2- 172.21.15.46:0/3149259028 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98840d1390 0x7f988416e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.959+0000 7f9882d9d700 1 --2- 172.21.15.46:0/3149259028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f988415d210 0x7f98841554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.959+0000 7f9881d9b700 1 -- 172.21.15.46:0/3149259028 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98840d1390 msgr2=0x7f988416e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.959+0000 7f9881d9b700 1 --2- 172.21.15.46:0/3149259028 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98840d1390 0x7f988416e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.959+0000 7f9881d9b700 1 -- 172.21.15.46:0/3149259028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f988415d210 msgr2=0x7f98841554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.959+0000 7f9881d9b700 1 --2- 172.21.15.46:0/3149259028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f988415d210 0x7f98841554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.959+0000 7f9881d9b700 1 -- 172.21.15.46:0/3149259028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98740094f0 con 0x7f988415ce40 2024-01-27T01:03:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.959+0000 7f988259c700 1 --2- 172.21.15.46:0/3149259028 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98840d1390 0x7f988416e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:03:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.959+0000 7f9881d9b700 1 --2- 172.21.15.46:0/3149259028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f988415ce40 0x7f988415ef40 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f987800d860 tx=0x7f9878008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.959+0000 7f98727fc700 1 -- 172.21.15.46:0/3149259028 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9878012630 con 0x7f988415ce40 2024-01-27T01:03:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.959+0000 7f98727fc700 1 -- 172.21.15.46:0/3149259028 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9878012790 con 0x7f988415ce40 2024-01-27T01:03:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.959+0000 7f9888c31700 1 -- 172.21.15.46:0/3149259028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98840c4830 con 0x7f988415ce40 2024-01-27T01:03:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.959+0000 7f98727fc700 1 -- 172.21.15.46:0/3149259028 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9878012b50 con 0x7f988415ce40 2024-01-27T01:03:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.959+0000 7f9888c31700 1 -- 172.21.15.46:0/3149259028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98840c4df0 con 0x7f988415ce40 2024-01-27T01:03:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.960+0000 7f9888c31700 1 -- 172.21.15.46:0/3149259028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98840b66b0 con 0x7f988415ce40 2024-01-27T01:03:27.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.961+0000 7f98727fc700 1 -- 172.21.15.46:0/3149259028 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 112) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f987801d400 con 0x7f988415ce40 2024-01-27T01:03:27.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.961+0000 7f98727fc700 1 --2- 172.21.15.46:0/3149259028 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9864066d80 0x7f9864069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:27.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.961+0000 7f98727fc700 1 -- 172.21.15.46:0/3149259028 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6997+0+0 (secure 0 0 0) 0x7f9878084b00 con 0x7f988415ce40 2024-01-27T01:03:27.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.961+0000 7f988259c700 1 --2- 172.21.15.46:0/3149259028 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9864066d80 0x7f9864069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:27.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.962+0000 7f988259c700 1 --2- 172.21.15.46:0/3149259028 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9864066d80 0x7f9864069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9884089b70 tx=0x7f9874017000 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:27.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:27.966+0000 7f98727fc700 1 -- 172.21.15.46:0/3149259028 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9878051b10 con 0x7f988415ce40 2024-01-27T01:03:28.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.176+0000 7f9888c31700 1 -- 172.21.15.46:0/3149259028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f988404c760 con 0x7f988415ce40 2024-01-27T01:03:28.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.750+0000 7f98727fc700 1 -- 172.21.15.46:0/3149259028 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 113) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f9878051ee0 con 0x7f988415ce40 2024-01-27T01:03:28.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.766+0000 7f98727fc700 1 -- 172.21.15.46:0/3149259028 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v196) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9878051b10 con 0x7f988415ce40 2024-01-27T01:03:28.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.770+0000 7f9888c31700 1 -- 172.21.15.46:0/3149259028 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9864066d80 msgr2=0x7f9864069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:28.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.770+0000 7f9888c31700 1 --2- 172.21.15.46:0/3149259028 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9864066d80 0x7f9864069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9884089b70 tx=0x7f9874017000 comp rx=0 tx=0).stop 2024-01-27T01:03:28.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.770+0000 7f9888c31700 1 -- 172.21.15.46:0/3149259028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f988415ce40 msgr2=0x7f988415ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:28.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.770+0000 7f9888c31700 1 --2- 172.21.15.46:0/3149259028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f988415ce40 0x7f988415ef40 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f987800d860 tx=0x7f9878008040 comp rx=0 tx=0).stop 2024-01-27T01:03:28.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.770+0000 7f9888c31700 1 -- 172.21.15.46:0/3149259028 shutdown_connections 2024-01-27T01:03:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.770+0000 7f9888c31700 1 --2- 172.21.15.46:0/3149259028 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9864066d80 0x7f9864069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.770+0000 7f9888c31700 1 --2- 172.21.15.46:0/3149259028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f988415d210 0x7f98841554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.770+0000 7f9888c31700 1 --2- 172.21.15.46:0/3149259028 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98840d1390 0x7f988416e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.770+0000 7f9888c31700 1 --2- 172.21.15.46:0/3149259028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f988415ce40 0x7f988415ef40 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.770+0000 7f9888c31700 1 -- 172.21.15.46:0/3149259028 >> 172.21.15.46:0/3149259028 conn(0x7f988406f840 msgr2=0x7f98840cd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.770+0000 7f9888c31700 1 -- 172.21.15.46:0/3149259028 shutdown_connections 2024-01-27T01:03:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.772+0000 7f9888c31700 1 -- 172.21.15.46:0/3149259028 wait complete. 2024-01-27T01:03:28.774 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-01-27T01:03:28.789 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-27T01:03:28.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.928+0000 7f5cde460700 1 Processor -- start 2024-01-27T01:03:28.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.928+0000 7f5cde460700 1 -- start start 2024-01-27T01:03:28.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.929+0000 7f5cde460700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cd8159cb0 0x7f5cd80cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:28.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.929+0000 7f5cde460700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5cd80afb40 0x7f5cd80c5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:28.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.929+0000 7f5cde460700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cd80cc100 con 0x7f5cd80cf050 2024-01-27T01:03:28.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.929+0000 7f5cde460700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cd80cb850 con 0x7f5cd80afb40 2024-01-27T01:03:28.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.929+0000 7f5cde460700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd80cbaf0 con 0x7f5cd8159cb0 2024-01-27T01:03:28.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.929+0000 7f5cd7fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cd8159cb0 0x7f5cd80cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:28.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.929+0000 7f5cdc9fd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5cd80afb40 0x7f5cd80c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:28.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.929+0000 7f5cd7fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cd8159cb0 0x7f5cd80cc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:54160/0 (socket says 172.21.15.46:54160) 2024-01-27T01:03:28.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.929+0000 7f5cd7fff700 1 -- 172.21.15.46:0/182405464 learned_addr learned my addr 172.21.15.46:0/182405464 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:28.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.930+0000 7f5cd7fff700 1 -- 172.21.15.46:0/182405464 >> v2:172.21.15.106:3300/0 conn(0x7f5cd80afb40 msgr2=0x7f5cd80c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:28.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.930+0000 7f5cd7fff700 1 --2- 172.21.15.46:0/182405464 >> v2:172.21.15.106:3300/0 conn(0x7f5cd80afb40 0x7f5cd80c5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:28.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.930+0000 7f5cd7fff700 1 -- 172.21.15.46:0/182405464 >> v1:172.21.15.106:6789/0 conn(0x7f5cd80cf050 legacy=0x7f5cd80af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:28.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.930+0000 7f5cd7fff700 1 -- 172.21.15.46:0/182405464 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd80cfc80 con 0x7f5cd8159cb0 2024-01-27T01:03:28.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.930+0000 7f5cd7fff700 1 --2- 172.21.15.46:0/182405464 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cd8159cb0 0x7f5cd80cc590 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8009990 tx=0x7f5cc8009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75afdb1f01f7f4ec server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:28.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.930+0000 7f5cd6ffd700 1 -- 172.21.15.46:0/182405464 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc8009e80 con 0x7f5cd8159cb0 2024-01-27T01:03:28.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.930+0000 7f5cd6ffd700 1 -- 172.21.15.46:0/182405464 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc8005c10 con 0x7f5cd8159cb0 2024-01-27T01:03:28.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.930+0000 7f5cd6ffd700 1 -- 172.21.15.46:0/182405464 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc800b650 con 0x7f5cd8159cb0 2024-01-27T01:03:28.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.931+0000 7f5cde460700 1 -- 172.21.15.46:0/182405464 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cd8159cb0 msgr2=0x7f5cd80cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:28.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.931+0000 7f5cde460700 1 --2- 172.21.15.46:0/182405464 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cd8159cb0 0x7f5cd80cc590 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8009990 tx=0x7f5cc8009220 comp rx=0 tx=0).stop 2024-01-27T01:03:28.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.931+0000 7f5cde460700 1 -- 172.21.15.46:0/182405464 shutdown_connections 2024-01-27T01:03:28.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.931+0000 7f5cde460700 1 --2- 172.21.15.46:0/182405464 >> v2:172.21.15.106:3300/0 conn(0x7f5cd80afb40 0x7f5cd80c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:28.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.931+0000 7f5cde460700 1 --2- 172.21.15.46:0/182405464 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cd8159cb0 0x7f5cd80cc590 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:28.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.931+0000 7f5cde460700 1 -- 172.21.15.46:0/182405464 >> 172.21.15.46:0/182405464 conn(0x7f5cd8087f60 msgr2=0x7f5cd806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:28.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.931+0000 7f5cde460700 1 -- 172.21.15.46:0/182405464 shutdown_connections 2024-01-27T01:03:28.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.931+0000 7f5cde460700 1 -- 172.21.15.46:0/182405464 wait complete. 2024-01-27T01:03:28.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.932+0000 7f5cde460700 1 Processor -- start 2024-01-27T01:03:28.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.933+0000 7f5cde460700 1 -- start start 2024-01-27T01:03:28.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.934+0000 7f5cde460700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cd80afb40 0x7f5cd80b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:28.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.934+0000 7f5cde460700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cd80cf050 0x7f5cd8169f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:28.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.934+0000 7f5cde460700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cd8159cb0 0x7f5cd816a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:28.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.934+0000 7f5cde460700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd80b46d0 con 0x7f5cd8159cb0 2024-01-27T01:03:28.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.934+0000 7f5cde460700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd80b3550 con 0x7f5cd80afb40 2024-01-27T01:03:28.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.934+0000 7f5cde460700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd80b44a0 con 0x7f5cd80cf050 2024-01-27T01:03:28.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.935+0000 7f5cdc9fd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cd8159cb0 0x7f5cd816a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:28.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.935+0000 7f5cdc9fd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cd8159cb0 0x7f5cd816a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47360/0 (socket says 172.21.15.46:47360) 2024-01-27T01:03:28.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.935+0000 7f5cdc9fd700 1 -- 172.21.15.46:0/3377960318 learned_addr learned my addr 172.21.15.46:0/3377960318 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:28.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.935+0000 7f5cd77fe700 1 --2- 172.21.15.46:0/3377960318 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cd80cf050 0x7f5cd8169f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:28.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.935+0000 7f5cd7fff700 1 --2- 172.21.15.46:0/3377960318 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cd80afb40 0x7f5cd80b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:28.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.935+0000 7f5cd77fe700 1 -- 172.21.15.46:0/3377960318 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cd80afb40 msgr2=0x7f5cd80b3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:28.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.935+0000 7f5cd77fe700 1 --2- 172.21.15.46:0/3377960318 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cd80afb40 0x7f5cd80b3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:28.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.935+0000 7f5cd77fe700 1 -- 172.21.15.46:0/3377960318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cd8159cb0 msgr2=0x7f5cd816a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:28.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.935+0000 7f5cd77fe700 1 --2- 172.21.15.46:0/3377960318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cd8159cb0 0x7f5cd816a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:28.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.935+0000 7f5cd77fe700 1 -- 172.21.15.46:0/3377960318 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc8009480 con 0x7f5cd80cf050 2024-01-27T01:03:28.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.935+0000 7f5cd77fe700 1 --2- 172.21.15.46:0/3377960318 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cd80cf050 0x7f5cd8169f50 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00c5c0 tx=0x7f5ccc007ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:28.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.935+0000 7f5cbffff700 1 -- 172.21.15.46:0/3377960318 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc011590 con 0x7f5cd80cf050 2024-01-27T01:03:28.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.936+0000 7f5cbffff700 1 -- 172.21.15.46:0/3377960318 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ccc011710 con 0x7f5cd80cf050 2024-01-27T01:03:28.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.936+0000 7f5cbffff700 1 -- 172.21.15.46:0/3377960318 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc011ad0 con 0x7f5cd80cf050 2024-01-27T01:03:28.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.936+0000 7f5cde460700 1 -- 172.21.15.46:0/3377960318 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd80c8850 con 0x7f5cd80cf050 2024-01-27T01:03:28.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.936+0000 7f5cde460700 1 -- 172.21.15.46:0/3377960318 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd80b3a40 con 0x7f5cd80cf050 2024-01-27T01:03:28.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.937+0000 7f5cde460700 1 -- 172.21.15.46:0/3377960318 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cd80b3c60 con 0x7f5cd80cf050 2024-01-27T01:03:28.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.938+0000 7f5cbffff700 1 -- 172.21.15.46:0/3377960318 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 113) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f5ccc01b3b0 con 0x7f5cd80cf050 2024-01-27T01:03:28.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.938+0000 7f5cbffff700 1 --2- 172.21.15.46:0/3377960318 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5cc4066ea0 0x7f5cc4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:28.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.938+0000 7f5cbffff700 1 -- 172.21.15.46:0/3377960318 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6583+0+0 (secure 0 0 0) 0x7f5ccc083b70 con 0x7f5cd80cf050 2024-01-27T01:03:28.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.938+0000 7f5cd7fff700 1 --2- 172.21.15.46:0/3377960318 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5cc4066ea0 0x7f5cc4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:28.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.939+0000 7f5cd7fff700 1 --2- 172.21.15.46:0/3377960318 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5cc4066ea0 0x7f5cc4069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5cd80cf570 tx=0x7f5cc8016000 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:28.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:28.943+0000 7f5cbffff700 1 -- 172.21.15.46:0/3377960318 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5ccc050b50 con 0x7f5cd80cf050 2024-01-27T01:03:29.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.154+0000 7f5cde460700 1 -- 172.21.15.46:0/3377960318 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f5cd80c89e0 con 0x7f5cd80cf050 2024-01-27T01:03:29.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.160+0000 7f5cbffff700 1 -- 172.21.15.46:0/3377960318 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v67) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5ccc017020 con 0x7f5cd80cf050 2024-01-27T01:03:29.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.163+0000 7f5cde460700 1 -- 172.21.15.46:0/3377960318 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5cc4066ea0 msgr2=0x7f5cc4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:29.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.163+0000 7f5cde460700 1 --2- 172.21.15.46:0/3377960318 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5cc4066ea0 0x7f5cc4069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5cd80cf570 tx=0x7f5cc8016000 comp rx=0 tx=0).stop 2024-01-27T01:03:29.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.163+0000 7f5cde460700 1 -- 172.21.15.46:0/3377960318 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cd80cf050 msgr2=0x7f5cd8169f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:29.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.163+0000 7f5cde460700 1 --2- 172.21.15.46:0/3377960318 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cd80cf050 0x7f5cd8169f50 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00c5c0 tx=0x7f5ccc007ec0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.163+0000 7f5cde460700 1 -- 172.21.15.46:0/3377960318 shutdown_connections 2024-01-27T01:03:29.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.163+0000 7f5cde460700 1 --2- 172.21.15.46:0/3377960318 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5cc4066ea0 0x7f5cc4069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.163+0000 7f5cde460700 1 --2- 172.21.15.46:0/3377960318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cd8159cb0 0x7f5cd816a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.163+0000 7f5cde460700 1 --2- 172.21.15.46:0/3377960318 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cd80afb40 0x7f5cd80b3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.163+0000 7f5cde460700 1 --2- 172.21.15.46:0/3377960318 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cd80cf050 0x7f5cd8169f50 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.163+0000 7f5cde460700 1 -- 172.21.15.46:0/3377960318 >> 172.21.15.46:0/3377960318 conn(0x7f5cd8087f60 msgr2=0x7f5cd80cc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:29.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.163+0000 7f5cde460700 1 -- 172.21.15.46:0/3377960318 shutdown_connections 2024-01-27T01:03:29.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.163+0000 7f5cde460700 1 -- 172.21.15.46:0/3377960318 wait complete. 2024-01-27T01:03:29.175 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-27T01:03:29.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.310+0000 7ff5487da700 1 Processor -- start 2024-01-27T01:03:29.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.310+0000 7ff5487da700 1 -- start start 2024-01-27T01:03:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.311+0000 7ff5487da700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff5400c93a0 0x7ff5400c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.311+0000 7ff5487da700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5400681f0 0x7ff540159930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.311+0000 7ff5487da700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff54015e390 con 0x7ff540067d50 2024-01-27T01:03:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.311+0000 7ff5487da700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff54015d810 con 0x7ff5400c93a0 2024-01-27T01:03:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.311+0000 7ff5487da700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54015daf0 con 0x7ff5400681f0 2024-01-27T01:03:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.311+0000 7ff546576700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff5400c93a0 0x7ff5400c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.311+0000 7ff545d75700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff540067d50 0x7ff540159220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45814/0 (socket says 172.21.15.46:45814) 2024-01-27T01:03:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.311+0000 7ff545d75700 1 -- 172.21.15.46:0/750120822 learned_addr learned my addr 172.21.15.46:0/750120822 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.311+0000 7ff546d77700 1 --2- 172.21.15.46:0/750120822 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5400681f0 0x7ff540159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.311+0000 7ff546576700 1 -- 172.21.15.46:0/750120822 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5400681f0 msgr2=0x7ff540159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.311+0000 7ff546576700 1 --2- 172.21.15.46:0/750120822 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5400681f0 0x7ff540159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.311+0000 7ff546576700 1 -- 172.21.15.46:0/750120822 >> v1:172.21.15.46:6789/0 conn(0x7ff540067d50 legacy=0x7ff540159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.311+0000 7ff545574700 1 -- 172.21.15.46:0/750120822 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756771544 0 0) 0x7ff54015e390 con 0x7ff540067d50 2024-01-27T01:03:29.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.312+0000 7ff546576700 1 -- 172.21.15.46:0/750120822 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54015ddd0 con 0x7ff5400c93a0 2024-01-27T01:03:29.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.312+0000 7ff546576700 1 --2- 172.21.15.46:0/750120822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5400c93a0 0x7ff5400c9770 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0098e0 tx=0x7ff53c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a165f6cad4c5a8c7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:29.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.312+0000 7ff545574700 1 -- 172.21.15.46:0/750120822 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c00b640 con 0x7ff5400c93a0 2024-01-27T01:03:29.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.312+0000 7ff545574700 1 -- 172.21.15.46:0/750120822 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53c00b7a0 con 0x7ff5400c93a0 2024-01-27T01:03:29.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.312+0000 7ff545574700 1 -- 172.21.15.46:0/750120822 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c00bb40 con 0x7ff5400c93a0 2024-01-27T01:03:29.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.312+0000 7ff5487da700 1 -- 172.21.15.46:0/750120822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5400c93a0 msgr2=0x7ff5400c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:29.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.312+0000 7ff5487da700 1 --2- 172.21.15.46:0/750120822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5400c93a0 0x7ff5400c9770 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0098e0 tx=0x7ff53c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.313+0000 7ff5487da700 1 -- 172.21.15.46:0/750120822 shutdown_connections 2024-01-27T01:03:29.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.313+0000 7ff5487da700 1 --2- 172.21.15.46:0/750120822 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5400681f0 0x7ff540159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.313+0000 7ff5487da700 1 --2- 172.21.15.46:0/750120822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5400c93a0 0x7ff5400c9770 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.313+0000 7ff5487da700 1 -- 172.21.15.46:0/750120822 >> 172.21.15.46:0/750120822 conn(0x7ff5400ca9a0 msgr2=0x7ff540068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:29.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.313+0000 7ff5487da700 1 -- 172.21.15.46:0/750120822 shutdown_connections 2024-01-27T01:03:29.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.313+0000 7ff5487da700 1 -- 172.21.15.46:0/750120822 wait complete. 2024-01-27T01:03:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.314+0000 7ff5487da700 1 Processor -- start 2024-01-27T01:03:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.314+0000 7ff5487da700 1 -- start start 2024-01-27T01:03:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff5487da700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff540067d50 0x7ff540154c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff5487da700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5400681f0 0x7ff54016a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff5487da700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5400c93a0 0x7ff540144f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff5487da700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff540156070 con 0x7ff5400c93a0 2024-01-27T01:03:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff5487da700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54016a8f0 con 0x7ff5400681f0 2024-01-27T01:03:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff5487da700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff540145440 con 0x7ff540067d50 2024-01-27T01:03:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff546d77700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5400c93a0 0x7ff540144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff546d77700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5400c93a0 0x7ff540144f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47376/0 (socket says 172.21.15.46:47376) 2024-01-27T01:03:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff546d77700 1 -- 172.21.15.46:0/4040066418 learned_addr learned my addr 172.21.15.46:0/4040066418 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff546576700 1 --2- 172.21.15.46:0/4040066418 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff540067d50 0x7ff540154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff545d75700 1 --2- 172.21.15.46:0/4040066418 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5400681f0 0x7ff54016a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff546d77700 1 -- 172.21.15.46:0/4040066418 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff540067d50 msgr2=0x7ff540154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff546d77700 1 --2- 172.21.15.46:0/4040066418 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff540067d50 0x7ff540154c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff546d77700 1 -- 172.21.15.46:0/4040066418 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5400681f0 msgr2=0x7ff54016a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff546d77700 1 --2- 172.21.15.46:0/4040066418 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5400681f0 0x7ff54016a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff546d77700 1 -- 172.21.15.46:0/4040066418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c009420 con 0x7ff5400c93a0 2024-01-27T01:03:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff546d77700 1 --2- 172.21.15.46:0/4040066418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5400c93a0 0x7ff540144f00 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7ff538009ec0 tx=0x7ff53800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff5367fc700 1 -- 172.21.15.46:0/4040066418 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53800cc20 con 0x7ff5400c93a0 2024-01-27T01:03:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff5367fc700 1 -- 172.21.15.46:0/4040066418 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53800f040 con 0x7ff5400c93a0 2024-01-27T01:03:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.315+0000 7ff5487da700 1 -- 172.21.15.46:0/4040066418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff540155740 con 0x7ff5400c93a0 2024-01-27T01:03:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.316+0000 7ff5367fc700 1 -- 172.21.15.46:0/4040066418 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff538014480 con 0x7ff5400c93a0 2024-01-27T01:03:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.316+0000 7ff5487da700 1 -- 172.21.15.46:0/4040066418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff540155d00 con 0x7ff5400c93a0 2024-01-27T01:03:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.316+0000 7ff5487da700 1 -- 172.21.15.46:0/4040066418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff540156220 con 0x7ff5400c93a0 2024-01-27T01:03:29.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.317+0000 7ff5367fc700 1 -- 172.21.15.46:0/4040066418 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 113) v1 ==== 84686+0+0 (secure 0 0 0) 0x7ff538014830 con 0x7ff5400c93a0 2024-01-27T01:03:29.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.317+0000 7ff5367fc700 1 --2- 172.21.15.46:0/4040066418 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff524066ea0 0x7ff524069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:29.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.318+0000 7ff5367fc700 1 -- 172.21.15.46:0/4040066418 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6583+0+0 (secure 0 0 0) 0x7ff538085240 con 0x7ff5400c93a0 2024-01-27T01:03:29.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.318+0000 7ff546576700 1 --2- 172.21.15.46:0/4040066418 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff524066ea0 0x7ff524069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:29.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.318+0000 7ff546576700 1 --2- 172.21.15.46:0/4040066418 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff524066ea0 0x7ff524069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff53c00a010 tx=0x7ff53c017000 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:29.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.322+0000 7ff5367fc700 1 -- 172.21.15.46:0/4040066418 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff540156220 con 0x7ff5400c93a0 2024-01-27T01:03:29.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.526+0000 7ff5487da700 1 -- 172.21.15.46:0/4040066418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7ff540156220 con 0x7ff5400c93a0 2024-01-27T01:03:29.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.530+0000 7ff5367fc700 1 -- 172.21.15.46:0/4040066418 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v68) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff540156220 con 0x7ff5400c93a0 2024-01-27T01:03:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.532+0000 7ff5487da700 1 -- 172.21.15.46:0/4040066418 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff524066ea0 msgr2=0x7ff524069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.532+0000 7ff5487da700 1 --2- 172.21.15.46:0/4040066418 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff524066ea0 0x7ff524069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff53c00a010 tx=0x7ff53c017000 comp rx=0 tx=0).stop 2024-01-27T01:03:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.532+0000 7ff5487da700 1 -- 172.21.15.46:0/4040066418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5400c93a0 msgr2=0x7ff540144f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.532+0000 7ff5487da700 1 --2- 172.21.15.46:0/4040066418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5400c93a0 0x7ff540144f00 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7ff538009ec0 tx=0x7ff53800b040 comp rx=0 tx=0).stop 2024-01-27T01:03:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.532+0000 7ff5487da700 1 -- 172.21.15.46:0/4040066418 shutdown_connections 2024-01-27T01:03:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.532+0000 7ff5487da700 1 --2- 172.21.15.46:0/4040066418 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff524066ea0 0x7ff524069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.532+0000 7ff5487da700 1 --2- 172.21.15.46:0/4040066418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5400c93a0 0x7ff540144f00 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.532+0000 7ff5487da700 1 --2- 172.21.15.46:0/4040066418 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff540067d50 0x7ff540154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.532+0000 7ff5487da700 1 --2- 172.21.15.46:0/4040066418 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5400681f0 0x7ff54016a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.532+0000 7ff5487da700 1 -- 172.21.15.46:0/4040066418 >> 172.21.15.46:0/4040066418 conn(0x7ff5400ca9a0 msgr2=0x7ff540070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.532+0000 7ff5487da700 1 -- 172.21.15.46:0/4040066418 shutdown_connections 2024-01-27T01:03:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.532+0000 7ff5487da700 1 -- 172.21.15.46:0/4040066418 wait complete. 2024-01-27T01:03:29.545 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-27T01:03:29.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.681+0000 7fd798600700 1 Processor -- start 2024-01-27T01:03:29.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.681+0000 7fd798600700 1 -- start start 2024-01-27T01:03:29.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.682+0000 7fd798600700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd79006b3c0 0x7fd7900af080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:29.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.682+0000 7fd798600700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd790158f00 con 0x7fd7900af5c0 2024-01-27T01:03:29.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.682+0000 7fd798600700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7900afa20 con 0x7fd790157750 2024-01-27T01:03:29.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.682+0000 7fd798600700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79006fe90 con 0x7fd79006b3c0 2024-01-27T01:03:29.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.682+0000 7fd796b9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd7900af5c0 0x7fd7900b1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45820/0 (socket says 172.21.15.46:45820) 2024-01-27T01:03:29.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.682+0000 7fd796b9d700 1 -- 172.21.15.46:0/316573866 learned_addr learned my addr 172.21.15.46:0/316573866 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:29.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.682+0000 7fd795b9b700 1 --2- 172.21.15.46:0/316573866 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd79006b3c0 0x7fd7900af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:29.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.682+0000 7fd79539a700 1 -- 172.21.15.46:0/316573866 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660580200 0 0) 0x7fd790158f00 con 0x7fd7900af5c0 2024-01-27T01:03:29.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.682+0000 7fd79539a700 1 -- 172.21.15.46:0/316573866 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd77c003570 con 0x7fd7900af5c0 2024-01-27T01:03:29.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd79539a700 1 -- 172.21.15.46:0/316573866 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3990837453 0 0) 0x7fd77c003570 con 0x7fd7900af5c0 2024-01-27T01:03:29.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd79539a700 1 -- 172.21.15.46:0/316573866 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd79006b3c0 msgr2=0x7fd7900af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:29.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd79539a700 1 --2- 172.21.15.46:0/316573866 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd79006b3c0 0x7fd7900af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd79539a700 1 -- 172.21.15.46:0/316573866 >> v1:172.21.15.106:6789/0 conn(0x7fd790157750 legacy=0x7fd790157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:29.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd79539a700 1 -- 172.21.15.46:0/316573866 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7901582f0 con 0x7fd7900af5c0 2024-01-27T01:03:29.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd79539a700 1 -- 172.21.15.46:0/316573866 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fd7880017f0 con 0x7fd7900af5c0 2024-01-27T01:03:29.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd79539a700 1 -- 172.21.15.46:0/316573866 >> v1:172.21.15.46:6789/0 conn(0x7fd7900af5c0 legacy=0x7fd7900b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:29.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd79539a700 1 --2- 172.21.15.46:0/316573866 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd79006b3c0 0x7fd77c003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:29.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd79539a700 1 --2- 172.21.15.46:0/316573866 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd77c003fd0 0x7fd77c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:29.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd79539a700 1 --2- 172.21.15.46:0/316573866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd77c004d50 0x7fd77c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd79539a700 1 -- 172.21.15.46:0/316573866 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79006fe90 con 0x7fd77c004d50 2024-01-27T01:03:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd79539a700 1 -- 172.21.15.46:0/316573866 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c00bbf0 con 0x7fd77c003fd0 2024-01-27T01:03:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd79539a700 1 -- 172.21.15.46:0/316573866 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c00bdb0 con 0x7fd79006b3c0 2024-01-27T01:03:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd796b9d700 1 --2- 172.21.15.46:0/316573866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd77c004d50 0x7fd77c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd79639c700 1 --2- 172.21.15.46:0/316573866 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd79006b3c0 0x7fd77c003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd795b9b700 1 --2- 172.21.15.46:0/316573866 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd77c003fd0 0x7fd77c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd796b9d700 1 -- 172.21.15.46:0/316573866 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd79006b3c0 msgr2=0x7fd77c003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd796b9d700 1 --2- 172.21.15.46:0/316573866 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd79006b3c0 0x7fd77c003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd796b9d700 1 -- 172.21.15.46:0/316573866 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd77c003fd0 msgr2=0x7fd77c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd796b9d700 1 --2- 172.21.15.46:0/316573866 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd77c003fd0 0x7fd77c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd796b9d700 1 -- 172.21.15.46:0/316573866 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77c003570 con 0x7fd77c004d50 2024-01-27T01:03:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.683+0000 7fd79639c700 1 --2- 172.21.15.46:0/316573866 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd79006b3c0 0x7fd77c003a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.684+0000 7fd796b9d700 1 --2- 172.21.15.46:0/316573866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd77c004d50 0x7fd77c0051c0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fd78800a950 tx=0x7fd7880021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f018a72a04bf43c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.684+0000 7fd79539a700 1 -- 172.21.15.46:0/316573866 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7880070d0 con 0x7fd77c004d50 2024-01-27T01:03:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.684+0000 7fd79539a700 1 -- 172.21.15.46:0/316573866 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd788007230 con 0x7fd77c004d50 2024-01-27T01:03:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.684+0000 7fd79539a700 1 -- 172.21.15.46:0/316573866 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7880075f0 con 0x7fd77c004d50 2024-01-27T01:03:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.684+0000 7fd798600700 1 -- 172.21.15.46:0/316573866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd77c004d50 msgr2=0x7fd77c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.684+0000 7fd798600700 1 --2- 172.21.15.46:0/316573866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd77c004d50 0x7fd77c0051c0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fd78800a950 tx=0x7fd7880021f0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.684+0000 7fd796b9d700 1 -- 172.21.15.46:0/316573866 reap_dead start 2024-01-27T01:03:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.684+0000 7fd798600700 1 -- 172.21.15.46:0/316573866 shutdown_connections 2024-01-27T01:03:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.684+0000 7fd798600700 1 -- 172.21.15.46:0/316573866 >> 172.21.15.46:0/316573866 conn(0x7fd7900cdf80 msgr2=0x7fd79006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.684+0000 7fd798600700 1 -- 172.21.15.46:0/316573866 shutdown_connections 2024-01-27T01:03:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.684+0000 7fd798600700 1 -- 172.21.15.46:0/316573866 wait complete. 2024-01-27T01:03:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.686+0000 7fd798600700 1 Processor -- start 2024-01-27T01:03:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.686+0000 7fd798600700 1 -- start start 2024-01-27T01:03:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.686+0000 7fd798600700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd77c004d50 0x7fd790151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.686+0000 7fd798600700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd79015bfe0 0x7fd79015c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.686+0000 7fd798600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd790152e30 0x7fd7901532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.686+0000 7fd798600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7901537f0 con 0x7fd790152e30 2024-01-27T01:03:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.686+0000 7fd798600700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd790152190 con 0x7fd79015bfe0 2024-01-27T01:03:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.686+0000 7fd798600700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79015c990 con 0x7fd77c004d50 2024-01-27T01:03:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.687+0000 7fd796b9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd790152e30 0x7fd7901532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.687+0000 7fd796b9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd790152e30 0x7fd7901532b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47394/0 (socket says 172.21.15.46:47394) 2024-01-27T01:03:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.687+0000 7fd796b9d700 1 -- 172.21.15.46:0/3206070706 learned_addr learned my addr 172.21.15.46:0/3206070706 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.687+0000 7fd795b9b700 1 --2- 172.21.15.46:0/3206070706 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd79015bfe0 0x7fd79015c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.687+0000 7fd79639c700 1 --2- 172.21.15.46:0/3206070706 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd77c004d50 0x7fd790151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.687+0000 7fd796b9d700 1 -- 172.21.15.46:0/3206070706 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd77c004d50 msgr2=0x7fd790151680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.687+0000 7fd796b9d700 1 --2- 172.21.15.46:0/3206070706 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd77c004d50 0x7fd790151680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.687+0000 7fd796b9d700 1 -- 172.21.15.46:0/3206070706 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd79015bfe0 msgr2=0x7fd79015c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.687+0000 7fd796b9d700 1 --2- 172.21.15.46:0/3206070706 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd79015bfe0 0x7fd79015c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.687+0000 7fd796b9d700 1 -- 172.21.15.46:0/3206070706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79015b7a0 con 0x7fd790152e30 2024-01-27T01:03:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.687+0000 7fd796b9d700 1 --2- 172.21.15.46:0/3206070706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd790152e30 0x7fd7901532b0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fd788000f80 tx=0x7fd788015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.687+0000 7fd7867fc700 1 -- 172.21.15.46:0/3206070706 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd788023070 con 0x7fd790152e30 2024-01-27T01:03:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.687+0000 7fd7867fc700 1 -- 172.21.15.46:0/3206070706 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78801e450 con 0x7fd790152e30 2024-01-27T01:03:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.687+0000 7fd7867fc700 1 -- 172.21.15.46:0/3206070706 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78801e7f0 con 0x7fd790152e30 2024-01-27T01:03:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.687+0000 7fd798600700 1 -- 172.21.15.46:0/3206070706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd79015aeb0 con 0x7fd790152e30 2024-01-27T01:03:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.687+0000 7fd798600700 1 -- 172.21.15.46:0/3206070706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd79015b150 con 0x7fd790152e30 2024-01-27T01:03:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.689+0000 7fd798600700 1 -- 172.21.15.46:0/3206070706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd79015b930 con 0x7fd790152e30 2024-01-27T01:03:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.689+0000 7fd7867fc700 1 -- 172.21.15.46:0/3206070706 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 113) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fd78801ec20 con 0x7fd790152e30 2024-01-27T01:03:29.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.689+0000 7fd7867fc700 1 --2- 172.21.15.46:0/3206070706 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd774066ea0 0x7fd774069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:29.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.689+0000 7fd7867fc700 1 -- 172.21.15.46:0/3206070706 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6583+0+0 (secure 0 0 0) 0x7fd7880866f0 con 0x7fd790152e30 2024-01-27T01:03:29.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.690+0000 7fd79639c700 1 --2- 172.21.15.46:0/3206070706 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd774066ea0 0x7fd774069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:29.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.690+0000 7fd79639c700 1 --2- 172.21.15.46:0/3206070706 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd774066ea0 0x7fd774069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd78c001270 tx=0x7fd78c009040 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:29.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.694+0000 7fd7867fc700 1 -- 172.21.15.46:0/3206070706 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd7880536d0 con 0x7fd790152e30 2024-01-27T01:03:29.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.904+0000 7fd798600700 1 -- 172.21.15.46:0/3206070706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fd79004c710 con 0x7fd790152e30 2024-01-27T01:03:29.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.907+0000 7fd7867fc700 1 -- 172.21.15.46:0/3206070706 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v69) v1 ==== 115+0+0 (secure 0 0 0) 0x7fd788056cf0 con 0x7fd790152e30 2024-01-27T01:03:29.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.909+0000 7fd798600700 1 -- 172.21.15.46:0/3206070706 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd774066ea0 msgr2=0x7fd774069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:29.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.909+0000 7fd798600700 1 --2- 172.21.15.46:0/3206070706 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd774066ea0 0x7fd774069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd78c001270 tx=0x7fd78c009040 comp rx=0 tx=0).stop 2024-01-27T01:03:29.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.909+0000 7fd798600700 1 -- 172.21.15.46:0/3206070706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd790152e30 msgr2=0x7fd7901532b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:29.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.909+0000 7fd798600700 1 --2- 172.21.15.46:0/3206070706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd790152e30 0x7fd7901532b0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fd788000f80 tx=0x7fd788015000 comp rx=0 tx=0).stop 2024-01-27T01:03:29.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.909+0000 7fd798600700 1 -- 172.21.15.46:0/3206070706 shutdown_connections 2024-01-27T01:03:29.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.910+0000 7fd798600700 1 --2- 172.21.15.46:0/3206070706 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd774066ea0 0x7fd774069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.910+0000 7fd798600700 1 --2- 172.21.15.46:0/3206070706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd790152e30 0x7fd7901532b0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.910+0000 7fd798600700 1 --2- 172.21.15.46:0/3206070706 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd77c004d50 0x7fd790151680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.910+0000 7fd798600700 1 --2- 172.21.15.46:0/3206070706 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd79015bfe0 0x7fd79015c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:29.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.910+0000 7fd798600700 1 -- 172.21.15.46:0/3206070706 >> 172.21.15.46:0/3206070706 conn(0x7fd7900cdf80 msgr2=0x7fd790153f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:29.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.910+0000 7fd798600700 1 -- 172.21.15.46:0/3206070706 shutdown_connections 2024-01-27T01:03:29.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:29.910+0000 7fd798600700 1 -- 172.21.15.46:0/3206070706 wait complete. 2024-01-27T01:03:29.922 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-01-27T01:03:30.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.055+0000 7fbff8821700 1 Processor -- start 2024-01-27T01:03:30.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.055+0000 7fbff8821700 1 -- start start 2024-01-27T01:03:30.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.056+0000 7fbff8821700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbff40acf30 0x7fbff40c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:30.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.056+0000 7fbff8821700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbff40ad780 0x7fbff40ab1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:30.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.056+0000 7fbff8821700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff40c4230 con 0x7fbff40ad300 2024-01-27T01:03:30.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.056+0000 7fbff8821700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff4157be0 con 0x7fbff40acf30 2024-01-27T01:03:30.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.056+0000 7fbff8821700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40cbf10 con 0x7fbff40ad780 2024-01-27T01:03:30.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.056+0000 7fbff259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbff40acf30 0x7fbff40c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:30.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.056+0000 7fbff1d9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbff40ad300 0x7fbff40ccb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45824/0 (socket says 172.21.15.46:45824) 2024-01-27T01:03:30.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.056+0000 7fbff1d9b700 1 -- 172.21.15.46:0/3947568803 learned_addr learned my addr 172.21.15.46:0/3947568803 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:30.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.056+0000 7fbff2d9d700 1 --2- 172.21.15.46:0/3947568803 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbff40ad780 0x7fbff40ab1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:30.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.056+0000 7fbff259c700 1 -- 172.21.15.46:0/3947568803 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbff40ad780 msgr2=0x7fbff40ab1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:30.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.056+0000 7fbff259c700 1 --2- 172.21.15.46:0/3947568803 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbff40ad780 0x7fbff40ab1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:30.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.056+0000 7fbff259c700 1 -- 172.21.15.46:0/3947568803 >> v1:172.21.15.46:6789/0 conn(0x7fbff40ad300 legacy=0x7fbff40ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:30.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.056+0000 7fbff259c700 1 -- 172.21.15.46:0/3947568803 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff40c4460 con 0x7fbff40acf30 2024-01-27T01:03:30.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.056+0000 7fbff2d9d700 1 --2- 172.21.15.46:0/3947568803 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbff40ad780 0x7fbff40ab1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:03:30.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.057+0000 7fbff259c700 1 --2- 172.21.15.46:0/3947568803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbff40acf30 0x7fbff40c3cf0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40098e0 tx=0x7fbfe40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b57c506413153041 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:30.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.057+0000 7fbff159a700 1 -- 172.21.15.46:0/3947568803 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4009e80 con 0x7fbff40acf30 2024-01-27T01:03:30.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.057+0000 7fbff159a700 1 -- 172.21.15.46:0/3947568803 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe400b860 con 0x7fbff40acf30 2024-01-27T01:03:30.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.057+0000 7fbff159a700 1 -- 172.21.15.46:0/3947568803 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe400bbe0 con 0x7fbff40acf30 2024-01-27T01:03:30.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.057+0000 7fbff8821700 1 -- 172.21.15.46:0/3947568803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbff40acf30 msgr2=0x7fbff40c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:30.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.057+0000 7fbff8821700 1 --2- 172.21.15.46:0/3947568803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbff40acf30 0x7fbff40c3cf0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40098e0 tx=0x7fbfe40091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:30.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.058+0000 7fbff8821700 1 -- 172.21.15.46:0/3947568803 shutdown_connections 2024-01-27T01:03:30.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.058+0000 7fbff8821700 1 --2- 172.21.15.46:0/3947568803 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbff40ad780 0x7fbff40ab1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:30.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.058+0000 7fbff8821700 1 --2- 172.21.15.46:0/3947568803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbff40acf30 0x7fbff40c3cf0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:30.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.058+0000 7fbff8821700 1 -- 172.21.15.46:0/3947568803 >> 172.21.15.46:0/3947568803 conn(0x7fbff4097820 msgr2=0x7fbff4068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:30.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.058+0000 7fbff8821700 1 -- 172.21.15.46:0/3947568803 shutdown_connections 2024-01-27T01:03:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.058+0000 7fbff8821700 1 -- 172.21.15.46:0/3947568803 wait complete. 2024-01-27T01:03:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.059+0000 7fbff8821700 1 Processor -- start 2024-01-27T01:03:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.059+0000 7fbff8821700 1 -- start start 2024-01-27T01:03:30.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff8821700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbff40acf30 0x7fbff4148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:30.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff8821700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbff40ad300 0x7fbff40b1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:30.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff8821700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbff40ad780 0x7fbff40c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:30.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff8821700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40c6650 con 0x7fbff40ad780 2024-01-27T01:03:30.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff8821700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40c6090 con 0x7fbff40ad300 2024-01-27T01:03:30.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff8821700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40c6370 con 0x7fbff40acf30 2024-01-27T01:03:30.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff2d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbff40ad780 0x7fbff40c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:30.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff2d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbff40ad780 0x7fbff40c55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47412/0 (socket says 172.21.15.46:47412) 2024-01-27T01:03:30.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff2d9d700 1 -- 172.21.15.46:0/2074937586 learned_addr learned my addr 172.21.15.46:0/2074937586 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:30.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff259c700 1 --2- 172.21.15.46:0/2074937586 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbff40acf30 0x7fbff4148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:30.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff1d9b700 1 --2- 172.21.15.46:0/2074937586 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbff40ad300 0x7fbff40b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:30.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff2d9d700 1 -- 172.21.15.46:0/2074937586 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbff40acf30 msgr2=0x7fbff4148ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:30.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff2d9d700 1 --2- 172.21.15.46:0/2074937586 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbff40acf30 0x7fbff4148ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:30.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff2d9d700 1 -- 172.21.15.46:0/2074937586 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbff40ad300 msgr2=0x7fbff40b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:30.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff2d9d700 1 --2- 172.21.15.46:0/2074937586 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbff40ad300 0x7fbff40b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:30.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff2d9d700 1 -- 172.21.15.46:0/2074937586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe4009420 con 0x7fbff40ad780 2024-01-27T01:03:30.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff259c700 1 --2- 172.21.15.46:0/2074937586 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbff40acf30 0x7fbff4148ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:03:30.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff2d9d700 1 --2- 172.21.15.46:0/2074937586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbff40ad780 0x7fbff40c55f0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00cef0 tx=0x7fbfec007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:30.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbfe27fc700 1 -- 172.21.15.46:0/2074937586 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec00ae00 con 0x7fbff40ad780 2024-01-27T01:03:30.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbfe27fc700 1 -- 172.21.15.46:0/2074937586 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfec016420 con 0x7fbff40ad780 2024-01-27T01:03:30.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbff8821700 1 -- 172.21.15.46:0/2074937586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff40ade60 con 0x7fbff40ad780 2024-01-27T01:03:30.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.060+0000 7fbfe27fc700 1 -- 172.21.15.46:0/2074937586 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec0167e0 con 0x7fbff40ad780 2024-01-27T01:03:30.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.061+0000 7fbff8821700 1 -- 172.21.15.46:0/2074937586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff40c7430 con 0x7fbff40ad780 2024-01-27T01:03:30.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.062+0000 7fbff8821700 1 -- 172.21.15.46:0/2074937586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff404c710 con 0x7fbff40ad780 2024-01-27T01:03:30.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.062+0000 7fbfe27fc700 1 -- 172.21.15.46:0/2074937586 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 113) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fbfec016bd0 con 0x7fbff40ad780 2024-01-27T01:03:30.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.063+0000 7fbfe27fc700 1 --2- 172.21.15.46:0/2074937586 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbfd4066ea0 0x7fbfd4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.063+0000 7fbfe27fc700 1 -- 172.21.15.46:0/2074937586 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6583+0+0 (secure 0 0 0) 0x7fbfec012070 con 0x7fbff40ad780 2024-01-27T01:03:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.063+0000 7fbff259c700 1 --2- 172.21.15.46:0/2074937586 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbfd4066ea0 0x7fbfd4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.064+0000 7fbff259c700 1 --2- 172.21.15.46:0/2074937586 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbfd4066ea0 0x7fbfd4069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40091e0 tx=0x7fbfe4017000 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:30.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.067+0000 7fbfe27fc700 1 -- 172.21.15.46:0/2074937586 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbfec04e5a0 con 0x7fbff40ad780 2024-01-27T01:03:30.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.358+0000 7fbff8821700 1 -- 172.21.15.46:0/2074937586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7fbff40c4740 con 0x7fbff40ad780 2024-01-27T01:03:30.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.536+0000 7fbfe27fc700 1 -- 172.21.15.46:0/2074937586 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbfec059d50 con 0x7fbff40ad780 2024-01-27T01:03:30.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.767+0000 7fbfe27fc700 1 -- 172.21.15.46:0/2074937586 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v3034) v1 ==== 165+0+0 (secure 0 0 0) 0x7fbfec0719c0 con 0x7fbff40ad780 2024-01-27T01:03:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.770+0000 7fbff8821700 1 -- 172.21.15.46:0/2074937586 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbfd4066ea0 msgr2=0x7fbfd4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.770+0000 7fbff8821700 1 --2- 172.21.15.46:0/2074937586 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbfd4066ea0 0x7fbfd4069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40091e0 tx=0x7fbfe4017000 comp rx=0 tx=0).stop 2024-01-27T01:03:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.770+0000 7fbff8821700 1 -- 172.21.15.46:0/2074937586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbff40ad780 msgr2=0x7fbff40c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.770+0000 7fbff8821700 1 --2- 172.21.15.46:0/2074937586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbff40ad780 0x7fbff40c55f0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00cef0 tx=0x7fbfec007860 comp rx=0 tx=0).stop 2024-01-27T01:03:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.770+0000 7fbff8821700 1 -- 172.21.15.46:0/2074937586 shutdown_connections 2024-01-27T01:03:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.771+0000 7fbff8821700 1 --2- 172.21.15.46:0/2074937586 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbfd4066ea0 0x7fbfd4069350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.771+0000 7fbff8821700 1 --2- 172.21.15.46:0/2074937586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbff40ad780 0x7fbff40c55f0 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.771+0000 7fbff8821700 1 --2- 172.21.15.46:0/2074937586 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbff40acf30 0x7fbff4148ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.771+0000 7fbff8821700 1 --2- 172.21.15.46:0/2074937586 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbff40ad300 0x7fbff40b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.771+0000 7fbff8821700 1 -- 172.21.15.46:0/2074937586 >> 172.21.15.46:0/2074937586 conn(0x7fbff4097820 msgr2=0x7fbff40cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.771+0000 7fbff8821700 1 -- 172.21.15.46:0/2074937586 shutdown_connections 2024-01-27T01:03:30.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.771+0000 7fbff8821700 1 -- 172.21.15.46:0/2074937586 wait complete. 2024-01-27T01:03:30.785 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2024-01-27T01:03:30.785 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2024-01-27T01:03:30.785 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-27T01:03:30.786 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-01-27T01:03:30.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.923+0000 7fcc5b25b700 1 Processor -- start 2024-01-27T01:03:30.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.923+0000 7fcc5b25b700 1 -- start start 2024-01-27T01:03:30.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.924+0000 7fcc5b25b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc540cc1c0 0x7fcc54164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.924+0000 7fcc5b25b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc540cff70 con 0x7fcc5415d210 2024-01-27T01:03:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.924+0000 7fcc5b25b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc540cf400 con 0x7fcc5415ce40 2024-01-27T01:03:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.924+0000 7fcc5b25b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc540cf660 con 0x7fcc540cc1c0 2024-01-27T01:03:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.924+0000 7fcc53fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcc5415d210 0x7fcc540cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38796/0 (socket says 172.21.15.46:38796) 2024-01-27T01:03:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.924+0000 7fcc53fff700 1 -- 172.21.15.46:0/766784359 learned_addr learned my addr 172.21.15.46:0/766784359 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc597f8700 1 --2- 172.21.15.46:0/766784359 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc540cc1c0 0x7fcc54164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc537fe700 1 -- 172.21.15.46:0/766784359 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495463525 0 0) 0x7fcc540cff70 con 0x7fcc5415d210 2024-01-27T01:03:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc537fe700 1 -- 172.21.15.46:0/766784359 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc40003570 con 0x7fcc5415d210 2024-01-27T01:03:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc537fe700 1 -- 172.21.15.46:0/766784359 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729504162 0 0) 0x7fcc540cf400 con 0x7fcc5415ce40 2024-01-27T01:03:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc537fe700 1 -- 172.21.15.46:0/766784359 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc540cff70 con 0x7fcc5415ce40 2024-01-27T01:03:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc537fe700 1 -- 172.21.15.46:0/766784359 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 165647335 0 0) 0x7fcc40003570 con 0x7fcc5415d210 2024-01-27T01:03:30.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc537fe700 1 -- 172.21.15.46:0/766784359 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc540cc1c0 msgr2=0x7fcc54164700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc537fe700 1 --2- 172.21.15.46:0/766784359 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc540cc1c0 0x7fcc54164700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc537fe700 1 -- 172.21.15.46:0/766784359 >> v1:172.21.15.106:6789/0 conn(0x7fcc5415ce40 legacy=0x7fcc540d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc537fe700 1 -- 172.21.15.46:0/766784359 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc540d0500 con 0x7fcc5415d210 2024-01-27T01:03:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc537fe700 1 -- 172.21.15.46:0/766784359 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fcc3c001470 con 0x7fcc5415d210 2024-01-27T01:03:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc537fe700 1 -- 172.21.15.46:0/766784359 >> v1:172.21.15.46:6789/0 conn(0x7fcc5415d210 legacy=0x7fcc540cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc537fe700 1 --2- 172.21.15.46:0/766784359 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc540cc1c0 0x7fcc40003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc537fe700 1 --2- 172.21.15.46:0/766784359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc400041d0 0x7fcc40004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc537fe700 1 --2- 172.21.15.46:0/766784359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc40004d50 0x7fcc400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc537fe700 1 -- 172.21.15.46:0/766784359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc540cf660 con 0x7fcc40004d50 2024-01-27T01:03:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc537fe700 1 -- 172.21.15.46:0/766784359 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4000bbf0 con 0x7fcc400041d0 2024-01-27T01:03:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc537fe700 1 -- 172.21.15.46:0/766784359 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4000bdb0 con 0x7fcc540cc1c0 2024-01-27T01:03:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc597f8700 1 --2- 172.21.15.46:0/766784359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc40004d50 0x7fcc400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:30.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc53fff700 1 --2- 172.21.15.46:0/766784359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc400041d0 0x7fcc40004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc58ff7700 1 --2- 172.21.15.46:0/766784359 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc540cc1c0 0x7fcc40003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc597f8700 1 -- 172.21.15.46:0/766784359 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc540cc1c0 msgr2=0x7fcc40003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.925+0000 7fcc597f8700 1 --2- 172.21.15.46:0/766784359 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc540cc1c0 0x7fcc40003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.926+0000 7fcc597f8700 1 -- 172.21.15.46:0/766784359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc400041d0 msgr2=0x7fcc40004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.926+0000 7fcc597f8700 1 --2- 172.21.15.46:0/766784359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc400041d0 0x7fcc40004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.926+0000 7fcc597f8700 1 -- 172.21.15.46:0/766784359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc40003570 con 0x7fcc40004d50 2024-01-27T01:03:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.926+0000 7fcc597f8700 1 --2- 172.21.15.46:0/766784359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc40004d50 0x7fcc400051c0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fcc540d01e0 tx=0x7fcc4c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08f88d29c849a5b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.926+0000 7fcc537fe700 1 -- 172.21.15.46:0/766784359 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c00ea60 con 0x7fcc40004d50 2024-01-27T01:03:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.926+0000 7fcc537fe700 1 -- 172.21.15.46:0/766784359 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4c00f040 con 0x7fcc40004d50 2024-01-27T01:03:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.926+0000 7fcc537fe700 1 -- 172.21.15.46:0/766784359 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c00ee10 con 0x7fcc40004d50 2024-01-27T01:03:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.926+0000 7fcc5b25b700 1 -- 172.21.15.46:0/766784359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc40004d50 msgr2=0x7fcc400051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.926+0000 7fcc5b25b700 1 --2- 172.21.15.46:0/766784359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc40004d50 0x7fcc400051c0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fcc540d01e0 tx=0x7fcc4c00b040 comp rx=0 tx=0).stop 2024-01-27T01:03:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.926+0000 7fcc597f8700 1 -- 172.21.15.46:0/766784359 reap_dead start 2024-01-27T01:03:30.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.927+0000 7fcc5b25b700 1 -- 172.21.15.46:0/766784359 shutdown_connections 2024-01-27T01:03:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.927+0000 7fcc5b25b700 1 -- 172.21.15.46:0/766784359 >> 172.21.15.46:0/766784359 conn(0x7fcc5406f840 msgr2=0x7fcc540d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.927+0000 7fcc5b25b700 1 -- 172.21.15.46:0/766784359 shutdown_connections 2024-01-27T01:03:30.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.927+0000 7fcc5b25b700 1 -- 172.21.15.46:0/766784359 wait complete. 2024-01-27T01:03:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.928+0000 7fcc5b25b700 1 Processor -- start 2024-01-27T01:03:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.929+0000 7fcc5b25b700 1 -- start start 2024-01-27T01:03:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.929+0000 7fcc5b25b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc40004d50 0x7fcc54157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.929+0000 7fcc5b25b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc540c6960 0x7fcc5415a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.929+0000 7fcc5b25b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc540c6dd0 0x7fcc5415abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.929+0000 7fcc5b25b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc54156990 con 0x7fcc40004d50 2024-01-27T01:03:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.929+0000 7fcc5b25b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc54156c70 con 0x7fcc540c6960 2024-01-27T01:03:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.929+0000 7fcc5b25b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc54156f50 con 0x7fcc540c6dd0 2024-01-27T01:03:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.930+0000 7fcc58ff7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc40004d50 0x7fcc54157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.930+0000 7fcc58ff7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc40004d50 0x7fcc54157d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36374/0 (socket says 172.21.15.46:36374) 2024-01-27T01:03:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.930+0000 7fcc58ff7700 1 -- 172.21.15.46:0/3942477468 learned_addr learned my addr 172.21.15.46:0/3942477468 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:30.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.930+0000 7fcc53fff700 1 --2- 172.21.15.46:0/3942477468 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc540c6960 0x7fcc5415a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.930+0000 7fcc597f8700 1 --2- 172.21.15.46:0/3942477468 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc540c6dd0 0x7fcc5415abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.930+0000 7fcc58ff7700 1 -- 172.21.15.46:0/3942477468 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc540c6dd0 msgr2=0x7fcc5415abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.930+0000 7fcc58ff7700 1 --2- 172.21.15.46:0/3942477468 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc540c6dd0 0x7fcc5415abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.930+0000 7fcc58ff7700 1 -- 172.21.15.46:0/3942477468 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc540c6960 msgr2=0x7fcc5415a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.930+0000 7fcc58ff7700 1 --2- 172.21.15.46:0/3942477468 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc540c6960 0x7fcc5415a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.930+0000 7fcc58ff7700 1 -- 172.21.15.46:0/3942477468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc54157230 con 0x7fcc40004d50 2024-01-27T01:03:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.930+0000 7fcc53fff700 1 --2- 172.21.15.46:0/3942477468 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc540c6960 0x7fcc5415a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:03:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.930+0000 7fcc58ff7700 1 --2- 172.21.15.46:0/3942477468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc40004d50 0x7fcc54157d10 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fcc44000c00 tx=0x7fcc44007d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.930+0000 7fcc50ff9700 1 -- 172.21.15.46:0/3942477468 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc44013600 con 0x7fcc40004d50 2024-01-27T01:03:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.930+0000 7fcc50ff9700 1 -- 172.21.15.46:0/3942477468 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc44013760 con 0x7fcc40004d50 2024-01-27T01:03:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.930+0000 7fcc50ff9700 1 -- 172.21.15.46:0/3942477468 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc44013b20 con 0x7fcc40004d50 2024-01-27T01:03:30.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.930+0000 7fcc5b25b700 1 -- 172.21.15.46:0/3942477468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc54157510 con 0x7fcc40004d50 2024-01-27T01:03:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.930+0000 7fcc5b25b700 1 -- 172.21.15.46:0/3942477468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc54158250 con 0x7fcc40004d50 2024-01-27T01:03:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.931+0000 7fcc5b25b700 1 -- 172.21.15.46:0/3942477468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5404c710 con 0x7fcc40004d50 2024-01-27T01:03:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.932+0000 7fcc50ff9700 1 -- 172.21.15.46:0/3942477468 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fcc4401c400 con 0x7fcc40004d50 2024-01-27T01:03:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.932+0000 7fcc50ff9700 1 --2- 172.21.15.46:0/3942477468 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcc34066f30 0x7fcc340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.933+0000 7fcc50ff9700 1 -- 172.21.15.46:0/3942477468 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6583+0+0 (secure 0 0 0) 0x7fcc44084c20 con 0x7fcc40004d50 2024-01-27T01:03:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.933+0000 7fcc53fff700 1 --2- 172.21.15.46:0/3942477468 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcc34066f30 0x7fcc340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:30.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.933+0000 7fcc53fff700 1 --2- 172.21.15.46:0/3942477468 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcc34066f30 0x7fcc340693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c001bd0 tx=0x7fcc3c009040 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:30.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:30.937+0000 7fcc50ff9700 1 -- 172.21.15.46:0/3942477468 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc44051b20 con 0x7fcc40004d50 2024-01-27T01:03:31.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.229+0000 7fcc5b25b700 1 -- 172.21.15.46:0/3942477468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7fcc540cf920 con 0x7fcc40004d50 2024-01-27T01:03:31.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.773+0000 7fcc50ff9700 1 -- 172.21.15.46:0/3942477468 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3035) v1 ==== 166+0+0 (secure 0 0 0) 0x7fcc44017020 con 0x7fcc40004d50 2024-01-27T01:03:31.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.776+0000 7fcc5b25b700 1 -- 172.21.15.46:0/3942477468 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcc34066f30 msgr2=0x7fcc340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:31.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.776+0000 7fcc5b25b700 1 --2- 172.21.15.46:0/3942477468 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcc34066f30 0x7fcc340693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c001bd0 tx=0x7fcc3c009040 comp rx=0 tx=0).stop 2024-01-27T01:03:31.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.776+0000 7fcc5b25b700 1 -- 172.21.15.46:0/3942477468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc40004d50 msgr2=0x7fcc54157d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:31.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.776+0000 7fcc5b25b700 1 --2- 172.21.15.46:0/3942477468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc40004d50 0x7fcc54157d10 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fcc44000c00 tx=0x7fcc44007d80 comp rx=0 tx=0).stop 2024-01-27T01:03:31.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.776+0000 7fcc5b25b700 1 -- 172.21.15.46:0/3942477468 shutdown_connections 2024-01-27T01:03:31.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.776+0000 7fcc5b25b700 1 --2- 172.21.15.46:0/3942477468 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcc34066f30 0x7fcc340693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:31.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.776+0000 7fcc5b25b700 1 --2- 172.21.15.46:0/3942477468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc40004d50 0x7fcc54157d10 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:31.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.776+0000 7fcc5b25b700 1 --2- 172.21.15.46:0/3942477468 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc540c6dd0 0x7fcc5415abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:31.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.776+0000 7fcc5b25b700 1 --2- 172.21.15.46:0/3942477468 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc540c6960 0x7fcc5415a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:31.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.776+0000 7fcc5b25b700 1 -- 172.21.15.46:0/3942477468 >> 172.21.15.46:0/3942477468 conn(0x7fcc5406f840 msgr2=0x7fcc540b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:31.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.776+0000 7fcc5b25b700 1 -- 172.21.15.46:0/3942477468 shutdown_connections 2024-01-27T01:03:31.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.776+0000 7fcc5b25b700 1 -- 172.21.15.46:0/3942477468 wait complete. 2024-01-27T01:03:31.791 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-27T01:03:31.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.930+0000 7fca51f62700 1 Processor -- start 2024-01-27T01:03:31.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.930+0000 7fca51f62700 1 -- start start 2024-01-27T01:03:31.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.931+0000 7fca51f62700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fca4c157860 0x7fca4c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:31.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.931+0000 7fca51f62700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fca4c06ddb0 0x7fca4c0cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:31.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.931+0000 7fca51f62700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4c0cea80 con 0x7fca4c157490 2024-01-27T01:03:31.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.931+0000 7fca51f62700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca4c0cdc60 con 0x7fca4c157860 2024-01-27T01:03:31.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.931+0000 7fca51f62700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca4c0b2ae0 con 0x7fca4c06ddb0 2024-01-27T01:03:31.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.931+0000 7fca4affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fca4c157860 0x7fca4c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:31.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.931+0000 7fca4affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fca4c157860 0x7fca4c0cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36386/0 (socket says 172.21.15.46:36386) 2024-01-27T01:03:31.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.931+0000 7fca4affd700 1 -- 172.21.15.46:0/3465778081 learned_addr learned my addr 172.21.15.46:0/3465778081 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:31.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.931+0000 7fca4bfff700 1 --2- 172.21.15.46:0/3465778081 >> v2:172.21.15.106:3300/0 conn(0x7fca4c06ddb0 0x7fca4c0cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:31.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.931+0000 7fca4affd700 1 -- 172.21.15.46:0/3465778081 >> v2:172.21.15.106:3300/0 conn(0x7fca4c06ddb0 msgr2=0x7fca4c0cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:31.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.931+0000 7fca4affd700 1 --2- 172.21.15.46:0/3465778081 >> v2:172.21.15.106:3300/0 conn(0x7fca4c06ddb0 0x7fca4c0cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:31.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.931+0000 7fca4affd700 1 -- 172.21.15.46:0/3465778081 >> v1:172.21.15.106:6789/0 conn(0x7fca4c157490 legacy=0x7fca4c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:31.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.931+0000 7fca4affd700 1 -- 172.21.15.46:0/3465778081 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca4c0cde70 con 0x7fca4c157860 2024-01-27T01:03:31.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.931+0000 7fca4bfff700 1 --2- 172.21.15.46:0/3465778081 >> v2:172.21.15.106:3300/0 conn(0x7fca4c06ddb0 0x7fca4c0cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:03:31.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.932+0000 7fca4affd700 1 --2- 172.21.15.46:0/3465778081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca4c157860 0x7fca4c0cc600 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fca4c068680 tx=0x7fca34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d00bb46820b6c3f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:31.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.932+0000 7fca4a7fc700 1 -- 172.21.15.46:0/3465778081 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3400b820 con 0x7fca4c157860 2024-01-27T01:03:31.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.932+0000 7fca4a7fc700 1 -- 172.21.15.46:0/3465778081 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3400b980 con 0x7fca4c157860 2024-01-27T01:03:31.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.932+0000 7fca4a7fc700 1 -- 172.21.15.46:0/3465778081 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3400bd20 con 0x7fca4c157860 2024-01-27T01:03:31.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.932+0000 7fca51f62700 1 -- 172.21.15.46:0/3465778081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca4c157860 msgr2=0x7fca4c0cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:31.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.932+0000 7fca51f62700 1 --2- 172.21.15.46:0/3465778081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca4c157860 0x7fca4c0cc600 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fca4c068680 tx=0x7fca34009250 comp rx=0 tx=0).stop 2024-01-27T01:03:31.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.932+0000 7fca51f62700 1 -- 172.21.15.46:0/3465778081 shutdown_connections 2024-01-27T01:03:31.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.932+0000 7fca51f62700 1 --2- 172.21.15.46:0/3465778081 >> v2:172.21.15.106:3300/0 conn(0x7fca4c06ddb0 0x7fca4c0cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:31.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.932+0000 7fca51f62700 1 --2- 172.21.15.46:0/3465778081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca4c157860 0x7fca4c0cc600 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:31.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.932+0000 7fca51f62700 1 -- 172.21.15.46:0/3465778081 >> 172.21.15.46:0/3465778081 conn(0x7fca4c067930 msgr2=0x7fca4c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:31.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.933+0000 7fca51f62700 1 -- 172.21.15.46:0/3465778081 shutdown_connections 2024-01-27T01:03:31.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.933+0000 7fca51f62700 1 -- 172.21.15.46:0/3465778081 wait complete. 2024-01-27T01:03:31.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.934+0000 7fca51f62700 1 Processor -- start 2024-01-27T01:03:31.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.934+0000 7fca51f62700 1 -- start start 2024-01-27T01:03:31.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.935+0000 7fca51f62700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca4c06ddb0 0x7fca4c15a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:31.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.935+0000 7fca51f62700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca4c157490 0x7fca4c14b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:31.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.935+0000 7fca51f62700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fca4c157860 0x7fca4c151760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:31.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.935+0000 7fca51f62700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4c15af20 con 0x7fca4c06ddb0 2024-01-27T01:03:31.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.935+0000 7fca51f62700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4c16acc0 con 0x7fca4c157490 2024-01-27T01:03:31.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.935+0000 7fca51f62700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca4c16afa0 con 0x7fca4c157860 2024-01-27T01:03:31.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.935+0000 7fca4b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca4c06ddb0 0x7fca4c15a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:31.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.935+0000 7fca4b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca4c06ddb0 0x7fca4c15a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36402/0 (socket says 172.21.15.46:36402) 2024-01-27T01:03:31.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.935+0000 7fca4b7fe700 1 -- 172.21.15.46:0/561033782 learned_addr learned my addr 172.21.15.46:0/561033782 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:31.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.935+0000 7fca4affd700 1 --2- 172.21.15.46:0/561033782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca4c157490 0x7fca4c14b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:31.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.935+0000 7fca4bfff700 1 --2- 172.21.15.46:0/561033782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fca4c157860 0x7fca4c151760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:31.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.935+0000 7fca4b7fe700 1 -- 172.21.15.46:0/561033782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fca4c157860 msgr2=0x7fca4c151760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:31.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.935+0000 7fca4b7fe700 1 --2- 172.21.15.46:0/561033782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fca4c157860 0x7fca4c151760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:31.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.935+0000 7fca4b7fe700 1 -- 172.21.15.46:0/561033782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca4c157490 msgr2=0x7fca4c14b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:31.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.935+0000 7fca4b7fe700 1 --2- 172.21.15.46:0/561033782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca4c157490 0x7fca4c14b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:31.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.935+0000 7fca4b7fe700 1 -- 172.21.15.46:0/561033782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca340094f0 con 0x7fca4c06ddb0 2024-01-27T01:03:31.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.936+0000 7fca4b7fe700 1 --2- 172.21.15.46:0/561033782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca4c06ddb0 0x7fca4c15a9e0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fca3c00b600 tx=0x7fca3c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:31.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.936+0000 7fca437fe700 1 -- 172.21.15.46:0/561033782 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c00f510 con 0x7fca4c06ddb0 2024-01-27T01:03:31.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.936+0000 7fca437fe700 1 -- 172.21.15.46:0/561033782 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3c00f670 con 0x7fca4c06ddb0 2024-01-27T01:03:31.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.936+0000 7fca51f62700 1 -- 172.21.15.46:0/561033782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca4c16b280 con 0x7fca4c06ddb0 2024-01-27T01:03:31.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.936+0000 7fca437fe700 1 -- 172.21.15.46:0/561033782 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c00fa30 con 0x7fca4c06ddb0 2024-01-27T01:03:31.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.936+0000 7fca51f62700 1 -- 172.21.15.46:0/561033782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca4c16b840 con 0x7fca4c06ddb0 2024-01-27T01:03:31.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.937+0000 7fca51f62700 1 -- 172.21.15.46:0/561033782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca4c04c710 con 0x7fca4c06ddb0 2024-01-27T01:03:31.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.937+0000 7fca437fe700 1 -- 172.21.15.46:0/561033782 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fca3c01a400 con 0x7fca4c06ddb0 2024-01-27T01:03:31.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.938+0000 7fca437fe700 1 --2- 172.21.15.46:0/561033782 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fca2c066f30 0x7fca2c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:31.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.938+0000 7fca437fe700 1 -- 172.21.15.46:0/561033782 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6583+0+0 (secure 0 0 0) 0x7fca3c084470 con 0x7fca4c06ddb0 2024-01-27T01:03:31.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.938+0000 7fca4affd700 1 --2- 172.21.15.46:0/561033782 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fca2c066f30 0x7fca2c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:31.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.939+0000 7fca4affd700 1 --2- 172.21.15.46:0/561033782 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fca2c066f30 0x7fca2c0693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fca4c068680 tx=0x7fca34016000 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:31.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:31.942+0000 7fca437fe700 1 -- 172.21.15.46:0/561033782 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fca3c051370 con 0x7fca4c06ddb0 2024-01-27T01:03:32.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.149+0000 7fca51f62700 1 -- 172.21.15.46:0/561033782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fca4c0b14d0 con 0x7fca4c06ddb0 2024-01-27T01:03:32.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.153+0000 7fca437fe700 1 -- 172.21.15.46:0/561033782 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v70)=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7fca3c054990 con 0x7fca4c06ddb0 2024-01-27T01:03:32.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.155+0000 7fca51f62700 1 -- 172.21.15.46:0/561033782 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fca2c066f30 msgr2=0x7fca2c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:32.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.155+0000 7fca51f62700 1 --2- 172.21.15.46:0/561033782 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fca2c066f30 0x7fca2c0693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fca4c068680 tx=0x7fca34016000 comp rx=0 tx=0).stop 2024-01-27T01:03:32.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.155+0000 7fca51f62700 1 -- 172.21.15.46:0/561033782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca4c06ddb0 msgr2=0x7fca4c15a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:32.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.155+0000 7fca51f62700 1 --2- 172.21.15.46:0/561033782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca4c06ddb0 0x7fca4c15a9e0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fca3c00b600 tx=0x7fca3c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:32.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.155+0000 7fca51f62700 1 -- 172.21.15.46:0/561033782 shutdown_connections 2024-01-27T01:03:32.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.155+0000 7fca51f62700 1 --2- 172.21.15.46:0/561033782 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fca2c066f30 0x7fca2c0693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:32.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.155+0000 7fca51f62700 1 --2- 172.21.15.46:0/561033782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca4c06ddb0 0x7fca4c15a9e0 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:32.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.155+0000 7fca51f62700 1 --2- 172.21.15.46:0/561033782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fca4c157860 0x7fca4c151760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:32.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.155+0000 7fca51f62700 1 --2- 172.21.15.46:0/561033782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca4c157490 0x7fca4c14b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:32.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.155+0000 7fca51f62700 1 -- 172.21.15.46:0/561033782 >> 172.21.15.46:0/561033782 conn(0x7fca4c067930 msgr2=0x7fca4c155860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:32.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.155+0000 7fca51f62700 1 -- 172.21.15.46:0/561033782 shutdown_connections 2024-01-27T01:03:32.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.155+0000 7fca51f62700 1 -- 172.21.15.46:0/561033782 wait complete. 2024-01-27T01:03:32.168 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-27T01:03:32.168 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:03:32.213 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:03:32.262 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-27T01:03:32.262 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-27T01:03:32.263 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:03:32.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.400+0000 7fa8d92f9700 1 Processor -- start 2024-01-27T01:03:32.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.400+0000 7fa8d92f9700 1 -- start start 2024-01-27T01:03:32.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.401+0000 7fa8d92f9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa8d40c6520 0x7fa8d40ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:32.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.401+0000 7fa8d92f9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8d40ced50 0x7fa8d40cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:32.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.401+0000 7fa8d92f9700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d415a0b0 con 0x7fa8d40b2b40 2024-01-27T01:03:32.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.401+0000 7fa8d92f9700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d40af070 con 0x7fa8d40c6520 2024-01-27T01:03:32.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.401+0000 7fa8d92f9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d4099fc0 con 0x7fa8d40ced50 2024-01-27T01:03:32.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.401+0000 7fa8d2ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa8d40b2b40 0x7fa8d40ae960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38804/0 (socket says 172.21.15.46:38804) 2024-01-27T01:03:32.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.401+0000 7fa8d2ffd700 1 -- 172.21.15.46:0/3580483456 learned_addr learned my addr 172.21.15.46:0/3580483456 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:32.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d27fc700 1 --2- 172.21.15.46:0/3580483456 >> v2:172.21.15.106:3300/0 conn(0x7fa8d40c6520 0x7fa8d40ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:32.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d1ffb700 1 -- 172.21.15.46:0/3580483456 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490013344 0 0) 0x7fa8d415a0b0 con 0x7fa8d40b2b40 2024-01-27T01:03:32.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d37fe700 1 --2- 172.21.15.46:0/3580483456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8d40ced50 0x7fa8d40cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:32.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d1ffb700 1 -- 172.21.15.46:0/3580483456 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8bc003570 con 0x7fa8d40b2b40 2024-01-27T01:03:32.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d1ffb700 1 -- 172.21.15.46:0/3580483456 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4159516582 0 0) 0x7fa8bc003570 con 0x7fa8d40b2b40 2024-01-27T01:03:32.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d1ffb700 1 -- 172.21.15.46:0/3580483456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8d40ced50 msgr2=0x7fa8d40cf1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:32.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d1ffb700 1 --2- 172.21.15.46:0/3580483456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8d40ced50 0x7fa8d40cf1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:32.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d1ffb700 1 -- 172.21.15.46:0/3580483456 >> v2:172.21.15.106:3300/0 conn(0x7fa8d40c6520 msgr2=0x7fa8d40ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:32.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d1ffb700 1 --2- 172.21.15.46:0/3580483456 >> v2:172.21.15.106:3300/0 conn(0x7fa8d40c6520 0x7fa8d40ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:32.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d1ffb700 1 -- 172.21.15.46:0/3580483456 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d40ae4d0 con 0x7fa8d40b2b40 2024-01-27T01:03:32.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d1ffb700 1 -- 172.21.15.46:0/3580483456 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fa8c40015f0 con 0x7fa8d40b2b40 2024-01-27T01:03:32.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d1ffb700 1 -- 172.21.15.46:0/3580483456 >> v1:172.21.15.46:6789/0 conn(0x7fa8d40b2b40 legacy=0x7fa8d40ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:32.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d1ffb700 1 --2- 172.21.15.46:0/3580483456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8bc003ae0 0x7fa8bc003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:32.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d1ffb700 1 --2- 172.21.15.46:0/3580483456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8d40ced50 0x7fa8bc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:32.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d1ffb700 1 --2- 172.21.15.46:0/3580483456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8bc004d50 0x7fa8bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:32.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d1ffb700 1 -- 172.21.15.46:0/3580483456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d40af070 con 0x7fa8bc004d50 2024-01-27T01:03:32.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d1ffb700 1 -- 172.21.15.46:0/3580483456 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc00bb10 con 0x7fa8bc003ae0 2024-01-27T01:03:32.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d1ffb700 1 -- 172.21.15.46:0/3580483456 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc00bc50 con 0x7fa8d40ced50 2024-01-27T01:03:32.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d37fe700 1 --2- 172.21.15.46:0/3580483456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8bc004d50 0x7fa8bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d2ffd700 1 --2- 172.21.15.46:0/3580483456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8bc003ae0 0x7fa8bc003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.402+0000 7fa8d27fc700 1 --2- 172.21.15.46:0/3580483456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8d40ced50 0x7fa8bc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.403+0000 7fa8d37fe700 1 -- 172.21.15.46:0/3580483456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8d40ced50 msgr2=0x7fa8bc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.403+0000 7fa8d37fe700 1 --2- 172.21.15.46:0/3580483456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8d40ced50 0x7fa8bc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.403+0000 7fa8d37fe700 1 -- 172.21.15.46:0/3580483456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8bc003ae0 msgr2=0x7fa8bc003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.403+0000 7fa8d37fe700 1 --2- 172.21.15.46:0/3580483456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8bc003ae0 0x7fa8bc003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.403+0000 7fa8d37fe700 1 -- 172.21.15.46:0/3580483456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8bc003570 con 0x7fa8bc004d50 2024-01-27T01:03:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.403+0000 7fa8d2ffd700 1 --2- 172.21.15.46:0/3580483456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8bc003ae0 0x7fa8bc003ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.403+0000 7fa8d37fe700 1 --2- 172.21.15.46:0/3580483456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8bc004d50 0x7fa8bc0051c0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00e240 tx=0x7fa8cc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f13a48e28e28c28c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.403+0000 7fa8d1ffb700 1 -- 172.21.15.46:0/3580483456 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc00eaf0 con 0x7fa8bc004d50 2024-01-27T01:03:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.403+0000 7fa8d1ffb700 1 -- 172.21.15.46:0/3580483456 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8cc00f040 con 0x7fa8bc004d50 2024-01-27T01:03:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.403+0000 7fa8d1ffb700 1 -- 172.21.15.46:0/3580483456 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc00eea0 con 0x7fa8bc004d50 2024-01-27T01:03:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.403+0000 7fa8d92f9700 1 -- 172.21.15.46:0/3580483456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8bc004d50 msgr2=0x7fa8bc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.403+0000 7fa8d92f9700 1 --2- 172.21.15.46:0/3580483456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8bc004d50 0x7fa8bc0051c0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00e240 tx=0x7fa8cc00b040 comp rx=0 tx=0).stop 2024-01-27T01:03:32.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.404+0000 7fa8d37fe700 1 -- 172.21.15.46:0/3580483456 reap_dead start 2024-01-27T01:03:32.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.404+0000 7fa8d92f9700 1 -- 172.21.15.46:0/3580483456 shutdown_connections 2024-01-27T01:03:32.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.404+0000 7fa8d92f9700 1 -- 172.21.15.46:0/3580483456 >> 172.21.15.46:0/3580483456 conn(0x7fa8d4099a30 msgr2=0x7fa8d406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:32.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.404+0000 7fa8d92f9700 1 -- 172.21.15.46:0/3580483456 shutdown_connections 2024-01-27T01:03:32.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.404+0000 7fa8d92f9700 1 -- 172.21.15.46:0/3580483456 wait complete. 2024-01-27T01:03:32.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.405+0000 7fa8d92f9700 1 Processor -- start 2024-01-27T01:03:32.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.405+0000 7fa8d92f9700 1 -- start start 2024-01-27T01:03:32.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.406+0000 7fa8d92f9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8d4157590 0x7fa8d40c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:32.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.406+0000 7fa8d92f9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8d4157960 0x7fa8d40c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:32.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.406+0000 7fa8d92f9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8d40cbde0 0x7fa8d40cc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:32.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.406+0000 7fa8d92f9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d409de50 con 0x7fa8d4157960 2024-01-27T01:03:32.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.406+0000 7fa8d92f9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d40ca270 con 0x7fa8d4157590 2024-01-27T01:03:32.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.406+0000 7fa8d92f9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d40cc7a0 con 0x7fa8d40cbde0 2024-01-27T01:03:32.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.406+0000 7fa8d27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8d4157960 0x7fa8d40c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:32.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.406+0000 7fa8d27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8d4157960 0x7fa8d40c9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36424/0 (socket says 172.21.15.46:36424) 2024-01-27T01:03:32.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.406+0000 7fa8d27fc700 1 -- 172.21.15.46:0/205361336 learned_addr learned my addr 172.21.15.46:0/205361336 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:32.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.406+0000 7fa8d2ffd700 1 --2- 172.21.15.46:0/205361336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8d4157590 0x7fa8d40c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:32.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.406+0000 7fa8d37fe700 1 --2- 172.21.15.46:0/205361336 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8d40cbde0 0x7fa8d40cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:32.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.406+0000 7fa8d27fc700 1 -- 172.21.15.46:0/205361336 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8d40cbde0 msgr2=0x7fa8d40cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:32.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.406+0000 7fa8d27fc700 1 --2- 172.21.15.46:0/205361336 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8d40cbde0 0x7fa8d40cc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:32.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.406+0000 7fa8d27fc700 1 -- 172.21.15.46:0/205361336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8d4157590 msgr2=0x7fa8d40c7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:32.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.406+0000 7fa8d27fc700 1 --2- 172.21.15.46:0/205361336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8d4157590 0x7fa8d40c7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:32.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.406+0000 7fa8d27fc700 1 -- 172.21.15.46:0/205361336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d40af1b0 con 0x7fa8d4157960 2024-01-27T01:03:32.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.406+0000 7fa8d37fe700 1 --2- 172.21.15.46:0/205361336 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8d40cbde0 0x7fa8d40cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:03:32.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.407+0000 7fa8d27fc700 1 --2- 172.21.15.46:0/205361336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8d4157960 0x7fa8d40c9d30 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8011b40 tx=0x7fa8c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:32.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.407+0000 7fa8baffd700 1 -- 172.21.15.46:0/205361336 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c800f940 con 0x7fa8d4157960 2024-01-27T01:03:32.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.407+0000 7fa8baffd700 1 -- 172.21.15.46:0/205361336 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c800faa0 con 0x7fa8d4157960 2024-01-27T01:03:32.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.407+0000 7fa8baffd700 1 -- 172.21.15.46:0/205361336 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c800fe20 con 0x7fa8d4157960 2024-01-27T01:03:32.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.407+0000 7fa8d92f9700 1 -- 172.21.15.46:0/205361336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d40c7e00 con 0x7fa8d4157960 2024-01-27T01:03:32.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.407+0000 7fa8d92f9700 1 -- 172.21.15.46:0/205361336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d40c83c0 con 0x7fa8d4157960 2024-01-27T01:03:32.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.408+0000 7fa8d92f9700 1 -- 172.21.15.46:0/205361336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d406a2d0 con 0x7fa8d4157960 2024-01-27T01:03:32.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.408+0000 7fa8baffd700 1 -- 172.21.15.46:0/205361336 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa8c8014650 con 0x7fa8d4157960 2024-01-27T01:03:32.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.409+0000 7fa8baffd700 1 --2- 172.21.15.46:0/205361336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa8c0066f30 0x7fa8c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:32.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.409+0000 7fa8baffd700 1 -- 172.21.15.46:0/205361336 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6583+0+0 (secure 0 0 0) 0x7fa8c8018030 con 0x7fa8d4157960 2024-01-27T01:03:32.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.409+0000 7fa8d2ffd700 1 --2- 172.21.15.46:0/205361336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa8c0066f30 0x7fa8c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:32.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.410+0000 7fa8d2ffd700 1 --2- 172.21.15.46:0/205361336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa8c0066f30 0x7fa8c00693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4007fd0 tx=0x7fa8c4009040 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:32.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.413+0000 7fa8baffd700 1 -- 172.21.15.46:0/205361336 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa8c80520f0 con 0x7fa8d4157960 2024-01-27T01:03:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.665+0000 7fa8d92f9700 1 -- 172.21.15.46:0/205361336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8d404c710 con 0x7fa8d4157960 2024-01-27T01:03:32.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.666+0000 7fa8baffd700 1 -- 172.21.15.46:0/205361336 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3530 (secure 0 0 0) 0x7fa8c8022de0 con 0x7fa8d4157960 2024-01-27T01:03:32.665 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:32.666 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":99,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8435,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/4207115513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":4207115513},{"type":"v1","addr":"172.21.15.106:6835","nonce":4207115513}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91},{"gid":8438,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/282430549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":282430549},{"type":"v1","addr":"172.21.15.46:6834","nonce":282430549}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9508,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/361546207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":361546207},{"type":"v1","addr":"172.21.15.106:6834","nonce":361546207}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":10269,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/4203399392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":4203399392},{"type":"v1","addr":"172.21.15.46:6835","nonce":4203399392}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":99}],"filesystems":[]} 2024-01-27T01:03:32.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.668+0000 7fa8d92f9700 1 -- 172.21.15.46:0/205361336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa8c0066f30 msgr2=0x7fa8c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:32.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.668+0000 7fa8d92f9700 1 --2- 172.21.15.46:0/205361336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa8c0066f30 0x7fa8c00693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4007fd0 tx=0x7fa8c4009040 comp rx=0 tx=0).stop 2024-01-27T01:03:32.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.668+0000 7fa8d92f9700 1 -- 172.21.15.46:0/205361336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8d4157960 msgr2=0x7fa8d40c9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:32.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.668+0000 7fa8d92f9700 1 --2- 172.21.15.46:0/205361336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8d4157960 0x7fa8d40c9d30 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8011b40 tx=0x7fa8c80091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:32.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.668+0000 7fa8d92f9700 1 -- 172.21.15.46:0/205361336 shutdown_connections 2024-01-27T01:03:32.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.668+0000 7fa8d92f9700 1 --2- 172.21.15.46:0/205361336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa8c0066f30 0x7fa8c00693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:32.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.668+0000 7fa8d92f9700 1 --2- 172.21.15.46:0/205361336 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa8d40cbde0 0x7fa8d40cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:32.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.668+0000 7fa8d92f9700 1 --2- 172.21.15.46:0/205361336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa8d4157590 0x7fa8d40c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:32.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.668+0000 7fa8d92f9700 1 --2- 172.21.15.46:0/205361336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8d4157960 0x7fa8d40c9d30 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:32.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.668+0000 7fa8d92f9700 1 -- 172.21.15.46:0/205361336 >> 172.21.15.46:0/205361336 conn(0x7fa8d4099a30 msgr2=0x7fa8d406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:32.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.668+0000 7fa8d92f9700 1 -- 172.21.15.46:0/205361336 shutdown_connections 2024-01-27T01:03:32.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:32.668+0000 7fa8d92f9700 1 -- 172.21.15.46:0/205361336 wait complete. 2024-01-27T01:03:32.669 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 99 2024-01-27T01:03:32.682 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-27T01:03:32.682 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-27T01:03:32.682 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-27T01:03:32.682 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:03:32.682 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-27T01:03:32.683 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-27T01:03:32.683 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-27T01:03:32.683 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:03:32.683 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-27T01:03:32.683 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-27T01:03:32.683 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-27T01:03:32.683 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:03:32.684 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-27T01:03:32.684 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-27T01:03:32.684 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-27T01:03:32.684 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:03:32.693 INFO:tasks.ceph.mds.a:Stopped 2024-01-27T01:03:32.693 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-27T01:03:32.694 INFO:tasks.ceph.mds.c:Stopped 2024-01-27T01:03:32.694 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-27T01:03:32.695 INFO:tasks.ceph.mds.d:Stopped 2024-01-27T01:03:32.696 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-27T01:03:32.735 INFO:tasks.ceph.mds.a:Started 2024-01-27T01:03:32.736 DEBUG:teuthology.parallel:result is None 2024-01-27T01:03:32.737 INFO:tasks.ceph.mds.c:Started 2024-01-27T01:03:32.737 DEBUG:teuthology.parallel:result is None 2024-01-27T01:03:32.738 INFO:tasks.ceph.mds.d:Started 2024-01-27T01:03:32.738 DEBUG:teuthology.parallel:result is None 2024-01-27T01:03:32.818 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-01-27T01:03:32.821 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-27T01:03:32.821 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-27T01:03:38.695 INFO:tasks.ceph.mds.b:Stopped 2024-01-27T01:03:38.695 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-27T01:03:38.698 INFO:tasks.ceph.mds.b:Started 2024-01-27T01:03:38.699 DEBUG:teuthology.parallel:result is None 2024-01-27T01:03:38.699 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-27T01:03:38.784 INFO:tasks.ceph.mds.b.smithi106.stdout:starting mds.b at 2024-01-27T01:03:38.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.842+0000 7faf456d5700 1 Processor -- start 2024-01-27T01:03:38.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.842+0000 7faf456d5700 1 -- start start 2024-01-27T01:03:38.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.843+0000 7faf456d5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf40159d30 0x7faf400cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:38.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.843+0000 7faf456d5700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4006d760 con 0x7faf40159960 2024-01-27T01:03:38.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.843+0000 7faf456d5700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4006f820 con 0x7faf40068040 2024-01-27T01:03:38.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.843+0000 7faf456d5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf400c9f00 con 0x7faf40159d30 2024-01-27T01:03:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.844+0000 7faf3effd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faf40159960 0x7faf40067930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38838/0 (socket says 172.21.15.46:38838) 2024-01-27T01:03:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.844+0000 7faf3effd700 1 -- 172.21.15.46:0/1627748247 learned_addr learned my addr 172.21.15.46:0/1627748247 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.844+0000 7faf3f7fe700 1 --2- 172.21.15.46:0/1627748247 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf40159d30 0x7faf400cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.844+0000 7faf3dffb700 1 -- 172.21.15.46:0/1627748247 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160798 0 0) 0x7faf4006d760 con 0x7faf40159960 2024-01-27T01:03:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.844+0000 7faf3dffb700 1 -- 172.21.15.46:0/1627748247 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf2c003570 con 0x7faf40159960 2024-01-27T01:03:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.844+0000 7faf3dffb700 1 -- 172.21.15.46:0/1627748247 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053818502 0 0) 0x7faf4006f820 con 0x7faf40068040 2024-01-27T01:03:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.844+0000 7faf3dffb700 1 -- 172.21.15.46:0/1627748247 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf4006d760 con 0x7faf40068040 2024-01-27T01:03:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.844+0000 7faf3f7fe700 1 -- 172.21.15.46:0/1627748247 >> v1:172.21.15.106:6789/0 conn(0x7faf40068040 legacy=0x7faf400c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.844+0000 7faf3f7fe700 1 -- 172.21.15.46:0/1627748247 >> v1:172.21.15.46:6789/0 conn(0x7faf40159960 legacy=0x7faf40067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.844+0000 7faf3f7fe700 1 -- 172.21.15.46:0/1627748247 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4006fcb0 con 0x7faf40159d30 2024-01-27T01:03:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.844+0000 7faf3f7fe700 1 --2- 172.21.15.46:0/1627748247 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf40159d30 0x7faf400cbc30 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7faf38009e90 tx=0x7faf38009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c026c66962dd239 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.844+0000 7faf3dffb700 1 -- 172.21.15.46:0/1627748247 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3800e070 con 0x7faf40159d30 2024-01-27T01:03:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.844+0000 7faf3dffb700 1 -- 172.21.15.46:0/1627748247 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3800bb80 con 0x7faf40159d30 2024-01-27T01:03:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.844+0000 7faf3dffb700 1 -- 172.21.15.46:0/1627748247 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf38012430 con 0x7faf40159d30 2024-01-27T01:03:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.845+0000 7faf456d5700 1 -- 172.21.15.46:0/1627748247 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf40159d30 msgr2=0x7faf400cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.845+0000 7faf456d5700 1 --2- 172.21.15.46:0/1627748247 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf40159d30 0x7faf400cbc30 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7faf38009e90 tx=0x7faf38009780 comp rx=0 tx=0).stop 2024-01-27T01:03:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.845+0000 7faf456d5700 1 -- 172.21.15.46:0/1627748247 shutdown_connections 2024-01-27T01:03:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.845+0000 7faf456d5700 1 --2- 172.21.15.46:0/1627748247 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf40159d30 0x7faf400cbc30 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.845+0000 7faf456d5700 1 -- 172.21.15.46:0/1627748247 >> 172.21.15.46:0/1627748247 conn(0x7faf4006b850 msgr2=0x7faf400cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.845+0000 7faf456d5700 1 -- 172.21.15.46:0/1627748247 shutdown_connections 2024-01-27T01:03:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.845+0000 7faf456d5700 1 -- 172.21.15.46:0/1627748247 wait complete. 2024-01-27T01:03:38.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-01-27T01:03:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.846+0000 7faf456d5700 1 Processor -- start 2024-01-27T01:03:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.847+0000 7faf456d5700 1 -- start start 2024-01-27T01:03:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.847+0000 7faf456d5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf40068040 0x7faf400b2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.847+0000 7faf456d5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf40159960 0x7faf40158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.847+0000 7faf456d5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf40159d30 0x7faf400b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.847+0000 7faf456d5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf400b1180 con 0x7faf40159d30 2024-01-27T01:03:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.847+0000 7faf456d5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf400b1460 con 0x7faf40159960 2024-01-27T01:03:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.847+0000 7faf456d5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf400b1740 con 0x7faf40068040 2024-01-27T01:03:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.847+0000 7faf3f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf40159d30 0x7faf400b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.847+0000 7faf3f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf40159d30 0x7faf400b0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36456/0 (socket says 172.21.15.46:36456) 2024-01-27T01:03:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.847+0000 7faf3f7fe700 1 -- 172.21.15.46:0/200153197 learned_addr learned my addr 172.21.15.46:0/200153197 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.847+0000 7faf3effd700 1 --2- 172.21.15.46:0/200153197 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf40068040 0x7faf400b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.847+0000 7faf3e7fc700 1 --2- 172.21.15.46:0/200153197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf40159960 0x7faf40158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.848+0000 7faf3f7fe700 1 -- 172.21.15.46:0/200153197 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf40068040 msgr2=0x7faf400b2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.848+0000 7faf3f7fe700 1 --2- 172.21.15.46:0/200153197 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf40068040 0x7faf400b2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.848+0000 7faf3f7fe700 1 -- 172.21.15.46:0/200153197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf40159960 msgr2=0x7faf40158880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.848+0000 7faf3f7fe700 1 --2- 172.21.15.46:0/200153197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf40159960 0x7faf40158880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.848+0000 7faf3f7fe700 1 -- 172.21.15.46:0/200153197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf38009a20 con 0x7faf40159d30 2024-01-27T01:03:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.848+0000 7faf3f7fe700 1 --2- 172.21.15.46:0/200153197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf40159d30 0x7faf400b0750 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7faf4015e810 tx=0x7faf38019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.848+0000 7faf36ffd700 1 -- 172.21.15.46:0/200153197 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3801b420 con 0x7faf40159d30 2024-01-27T01:03:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.848+0000 7faf36ffd700 1 -- 172.21.15.46:0/200153197 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3801b5a0 con 0x7faf40159d30 2024-01-27T01:03:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.848+0000 7faf456d5700 1 -- 172.21.15.46:0/200153197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf400b1a20 con 0x7faf40159d30 2024-01-27T01:03:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.848+0000 7faf36ffd700 1 -- 172.21.15.46:0/200153197 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3801b960 con 0x7faf40159d30 2024-01-27T01:03:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.848+0000 7faf456d5700 1 -- 172.21.15.46:0/200153197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf400b2760 con 0x7faf40159d30 2024-01-27T01:03:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.849+0000 7faf456d5700 1 -- 172.21.15.46:0/200153197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf4004c710 con 0x7faf40159d30 2024-01-27T01:03:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.850+0000 7faf36ffd700 1 -- 172.21.15.46:0/200153197 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7faf3801bd50 con 0x7faf40159d30 2024-01-27T01:03:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.850+0000 7faf36ffd700 1 --2- 172.21.15.46:0/200153197 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faf20066f30 0x7faf200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.850+0000 7faf36ffd700 1 -- 172.21.15.46:0/200153197 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6583+0+0 (secure 0 0 0) 0x7faf38086680 con 0x7faf40159d30 2024-01-27T01:03:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.850+0000 7faf3effd700 1 --2- 172.21.15.46:0/200153197 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faf20066f30 0x7faf200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.851+0000 7faf3effd700 1 --2- 172.21.15.46:0/200153197 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faf20066f30 0x7faf200693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7faf30007fd0 tx=0x7faf30009040 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:38.855+0000 7faf36ffd700 1 -- 172.21.15.46:0/200153197 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faf38053580 con 0x7faf40159d30 2024-01-27T01:03:39.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:39.076+0000 7faf456d5700 1 -- 172.21.15.46:0/200153197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7faf400affa0 con 0x7faf40159d30 2024-01-27T01:03:40.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.682+0000 7faf36ffd700 1 -- 172.21.15.46:0/200153197 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v201) v1 ==== 97+0+0 (secure 0 0 0) 0x7faf38056ba0 con 0x7faf40159d30 2024-01-27T01:03:40.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.685+0000 7faf456d5700 1 -- 172.21.15.46:0/200153197 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faf20066f30 msgr2=0x7faf200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:40.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.685+0000 7faf456d5700 1 --2- 172.21.15.46:0/200153197 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faf20066f30 0x7faf200693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7faf30007fd0 tx=0x7faf30009040 comp rx=0 tx=0).stop 2024-01-27T01:03:40.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.685+0000 7faf456d5700 1 -- 172.21.15.46:0/200153197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf40159d30 msgr2=0x7faf400b0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:40.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.685+0000 7faf456d5700 1 --2- 172.21.15.46:0/200153197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf40159d30 0x7faf400b0750 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7faf4015e810 tx=0x7faf38019000 comp rx=0 tx=0).stop 2024-01-27T01:03:40.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.685+0000 7faf456d5700 1 -- 172.21.15.46:0/200153197 shutdown_connections 2024-01-27T01:03:40.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.685+0000 7faf456d5700 1 --2- 172.21.15.46:0/200153197 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faf20066f30 0x7faf200693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:40.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.685+0000 7faf456d5700 1 --2- 172.21.15.46:0/200153197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf40159d30 0x7faf400b0750 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:40.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.685+0000 7faf456d5700 1 --2- 172.21.15.46:0/200153197 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf40068040 0x7faf400b2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:40.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.685+0000 7faf456d5700 1 --2- 172.21.15.46:0/200153197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf40159960 0x7faf40158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:40.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.685+0000 7faf456d5700 1 -- 172.21.15.46:0/200153197 >> 172.21.15.46:0/200153197 conn(0x7faf4006b850 msgr2=0x7faf400c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:40.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.685+0000 7faf456d5700 1 -- 172.21.15.46:0/200153197 shutdown_connections 2024-01-27T01:03:40.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.685+0000 7faf456d5700 1 -- 172.21.15.46:0/200153197 wait complete. 2024-01-27T01:03:40.686 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-01-27T01:03:40.701 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-27T01:03:40.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.838+0000 7fc3e0ae0700 1 Processor -- start 2024-01-27T01:03:40.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.838+0000 7fc3e0ae0700 1 -- start start 2024-01-27T01:03:40.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.839+0000 7fc3e0ae0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3dc159760 0x7fc3dc0b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:40.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.840+0000 7fc3e0ae0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3dc0b12c0 0x7fc3dc0a9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:40.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.840+0000 7fc3e0ae0700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3dc06f940 con 0x7fc3dc0b3060 2024-01-27T01:03:40.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.840+0000 7fc3e0ae0700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3dc06ed30 con 0x7fc3dc0b12c0 2024-01-27T01:03:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.840+0000 7fc3e0ae0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc06f010 con 0x7fc3dc159760 2024-01-27T01:03:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.840+0000 7fc3dad9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3dc0b12c0 0x7fc3dc0a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.840+0000 7fc3dad9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3dc0b12c0 0x7fc3dc0a9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38864/0 (socket says 172.21.15.46:38864) 2024-01-27T01:03:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.840+0000 7fc3dad9d700 1 -- 172.21.15.46:0/3456564388 learned_addr learned my addr 172.21.15.46:0/3456564388 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.840+0000 7fc3da59c700 1 --2- 172.21.15.46:0/3456564388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3dc159760 0x7fc3dc0b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.840+0000 7fc3dad9d700 1 -- 172.21.15.46:0/3456564388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3dc159760 msgr2=0x7fc3dc0b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.840+0000 7fc3dad9d700 1 --2- 172.21.15.46:0/3456564388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3dc159760 0x7fc3dc0b2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.840+0000 7fc3dad9d700 1 -- 172.21.15.46:0/3456564388 >> v1:172.21.15.46:6789/0 conn(0x7fc3dc0b3060 legacy=0x7fc3dc0b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.840+0000 7fc3dad9d700 1 -- 172.21.15.46:0/3456564388 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc06f2f0 con 0x7fc3dc0b12c0 2024-01-27T01:03:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.840+0000 7fc3dad9d700 1 --2- 172.21.15.46:0/3456564388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b12c0 0x7fc3dc0a9840 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009f20 tx=0x7fc3d4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880fc952e898021a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.840+0000 7fc3d959a700 1 -- 172.21.15.46:0/3456564388 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d400e070 con 0x7fc3dc0b12c0 2024-01-27T01:03:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.841+0000 7fc3d959a700 1 -- 172.21.15.46:0/3456564388 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d400b9d0 con 0x7fc3dc0b12c0 2024-01-27T01:03:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.841+0000 7fc3d959a700 1 -- 172.21.15.46:0/3456564388 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d40092e0 con 0x7fc3dc0b12c0 2024-01-27T01:03:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.841+0000 7fc3e0ae0700 1 -- 172.21.15.46:0/3456564388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b12c0 msgr2=0x7fc3dc0a9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.841+0000 7fc3e0ae0700 1 --2- 172.21.15.46:0/3456564388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b12c0 0x7fc3dc0a9840 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009f20 tx=0x7fc3d4009840 comp rx=0 tx=0).stop 2024-01-27T01:03:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.841+0000 7fc3e0ae0700 1 -- 172.21.15.46:0/3456564388 shutdown_connections 2024-01-27T01:03:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.841+0000 7fc3e0ae0700 1 --2- 172.21.15.46:0/3456564388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b12c0 0x7fc3dc0a9840 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.841+0000 7fc3e0ae0700 1 --2- 172.21.15.46:0/3456564388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3dc159760 0x7fc3dc0b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.841+0000 7fc3e0ae0700 1 -- 172.21.15.46:0/3456564388 >> 172.21.15.46:0/3456564388 conn(0x7fc3dc069510 msgr2=0x7fc3dc0aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.841+0000 7fc3e0ae0700 1 -- 172.21.15.46:0/3456564388 shutdown_connections 2024-01-27T01:03:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.841+0000 7fc3e0ae0700 1 -- 172.21.15.46:0/3456564388 wait complete. 2024-01-27T01:03:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.842+0000 7fc3e0ae0700 1 Processor -- start 2024-01-27T01:03:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.842+0000 7fc3e0ae0700 1 -- start start 2024-01-27T01:03:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.843+0000 7fc3e0ae0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3dc0b12c0 0x7fc3dc155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.843+0000 7fc3e0ae0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b3060 0x7fc3dc16cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.843+0000 7fc3e0ae0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3dc159760 0x7fc3dc16d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.843+0000 7fc3e0ae0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc15b490 con 0x7fc3dc0b3060 2024-01-27T01:03:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.843+0000 7fc3e0ae0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc150eb0 con 0x7fc3dc159760 2024-01-27T01:03:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.843+0000 7fc3e0ae0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc151190 con 0x7fc3dc0b12c0 2024-01-27T01:03:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.843+0000 7fc3d9d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b3060 0x7fc3dc16cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.843+0000 7fc3d9d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b3060 0x7fc3dc16cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38872/0 (socket says 172.21.15.46:38872) 2024-01-27T01:03:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.843+0000 7fc3d9d9b700 1 -- 172.21.15.46:0/2582724355 learned_addr learned my addr 172.21.15.46:0/2582724355 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.844+0000 7fc3da59c700 1 --2- 172.21.15.46:0/2582724355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3dc0b12c0 0x7fc3dc155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.844+0000 7fc3dad9d700 1 --2- 172.21.15.46:0/2582724355 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3dc159760 0x7fc3dc16d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.844+0000 7fc3d9d9b700 1 -- 172.21.15.46:0/2582724355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3dc0b12c0 msgr2=0x7fc3dc155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.844+0000 7fc3d9d9b700 1 --2- 172.21.15.46:0/2582724355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3dc0b12c0 0x7fc3dc155270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.844+0000 7fc3d9d9b700 1 -- 172.21.15.46:0/2582724355 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3dc159760 msgr2=0x7fc3dc16d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.844+0000 7fc3d9d9b700 1 --2- 172.21.15.46:0/2582724355 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3dc159760 0x7fc3dc16d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.844+0000 7fc3d9d9b700 1 -- 172.21.15.46:0/2582724355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d4009ae0 con 0x7fc3dc0b3060 2024-01-27T01:03:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.844+0000 7fc3dad9d700 1 --2- 172.21.15.46:0/2582724355 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3dc159760 0x7fc3dc16d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:03:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.844+0000 7fc3d9d9b700 1 --2- 172.21.15.46:0/2582724355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b3060 0x7fc3dc16cad0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fc3d000c590 tx=0x7fc3d000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:40.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.844+0000 7fc3ca7fc700 1 -- 172.21.15.46:0/2582724355 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0011590 con 0x7fc3dc0b3060 2024-01-27T01:03:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.844+0000 7fc3ca7fc700 1 -- 172.21.15.46:0/2582724355 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d00116f0 con 0x7fc3dc0b3060 2024-01-27T01:03:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.844+0000 7fc3ca7fc700 1 -- 172.21.15.46:0/2582724355 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0011ab0 con 0x7fc3dc0b3060 2024-01-27T01:03:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.844+0000 7fc3e0ae0700 1 -- 172.21.15.46:0/2582724355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3dc151470 con 0x7fc3dc0b3060 2024-01-27T01:03:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.844+0000 7fc3e0ae0700 1 -- 172.21.15.46:0/2582724355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3dc14ffa0 con 0x7fc3dc0b3060 2024-01-27T01:03:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.845+0000 7fc3e0ae0700 1 -- 172.21.15.46:0/2582724355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3dc151660 con 0x7fc3dc0b3060 2024-01-27T01:03:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.846+0000 7fc3ca7fc700 1 -- 172.21.15.46:0/2582724355 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc3d001b3b0 con 0x7fc3dc0b3060 2024-01-27T01:03:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.846+0000 7fc3ca7fc700 1 --2- 172.21.15.46:0/2582724355 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc3bc066f30 0x7fc3bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.846+0000 7fc3ca7fc700 1 -- 172.21.15.46:0/2582724355 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5637+0+0 (secure 0 0 0) 0x7fc3d0083aa0 con 0x7fc3dc0b3060 2024-01-27T01:03:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.846+0000 7fc3da59c700 1 --2- 172.21.15.46:0/2582724355 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc3bc066f30 0x7fc3bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.847+0000 7fc3da59c700 1 --2- 172.21.15.46:0/2582724355 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc3bc066f30 0x7fc3bc0693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc009ca0 tx=0x7fc3cc008040 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:40.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:40.850+0000 7fc3ca7fc700 1 -- 172.21.15.46:0/2582724355 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc3d0050d50 con 0x7fc3dc0b3060 2024-01-27T01:03:41.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.136+0000 7fc3e0ae0700 1 -- 172.21.15.46:0/2582724355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc3dc04c710 con 0x7fc3dc0b3060 2024-01-27T01:03:41.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.137+0000 7fc3ca7fc700 1 -- 172.21.15.46:0/2582724355 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v51) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc3d0017020 con 0x7fc3dc0b3060 2024-01-27T01:03:41.136 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:41.136 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:03:41.136 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-01-27T01:03:41.136 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.137 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-01-27T01:03:41.137 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlM6dKKBAAPpeWeDZWrFgc/4zuxKCvHw==", 2024-01-27T01:03:41.137 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.137 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:03:41.137 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:03:41.137 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:03:41.137 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:03:41.137 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.137 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.137 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.137 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-01-27T01:03:41.137 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlmzHRNRAAflWYZ/p4AXeMy54tvLpugA==", 2024-01-27T01:03:41.137 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.137 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:03:41.138 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:03:41.138 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:03:41.138 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:03:41.138 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.138 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.138 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.138 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-01-27T01:03:41.138 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlMZxoMBAAG5kyfm+LLG86y/p2A8BfDQ==", 2024-01-27T01:03:41.138 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.138 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:03:41.138 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:03:41.138 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:03:41.138 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:03:41.138 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.138 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.139 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.139 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-01-27T01:03:41.139 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl5jx/ARAA3cwtLAOWQ/n2NG+xwEZ3QQ==", 2024-01-27T01:03:41.139 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.139 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:03:41.139 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:03:41.139 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:03:41.139 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:03:41.139 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.139 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.139 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.139 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-01-27T01:03:41.139 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4QLRlJ2OKJRAAWVhxEdQkUW20294w/KLL6g==", 2024-01-27T01:03:41.139 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.139 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:03:41.140 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:03:41.140 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:03:41.140 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.140 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.140 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.140 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-01-27T01:03:41.140 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7QLRlE2tAJRAAzNlTiCr6a2eWiabRy1SqSw==", 2024-01-27T01:03:41.140 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.140 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:03:41.141 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:03:41.141 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:03:41.141 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.141 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.141 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.141 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-01-27T01:03:41.141 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+QLRlyySaJBAA5072ba5HUNi2dYE3D02M/Q==", 2024-01-27T01:03:41.141 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.141 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:03:41.141 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:03:41.141 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:03:41.142 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.142 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.142 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.142 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-01-27T01:03:41.142 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBQLRl9+tAIxAApZk0btXIpQPWSb3BYUYsFQ==", 2024-01-27T01:03:41.142 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.142 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:03:41.142 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:03:41.142 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:03:41.142 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.142 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.142 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.142 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-01-27T01:03:41.142 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHQLRlurUtJxAAT9fwKrAWl62VOXwShg7NJg==", 2024-01-27T01:03:41.143 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.143 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:03:41.143 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:03:41.143 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:03:41.143 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.143 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.143 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.143 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-01-27T01:03:41.143 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKQLRl7aqoJRAAXu9fykWELsW3SwnahoIOLA==", 2024-01-27T01:03:41.143 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.143 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:03:41.143 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:03:41.143 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:03:41.143 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.144 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.144 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.144 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-01-27T01:03:41.144 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNQLRlWf7pJBAAopmFLIKb8s32YZs46VwbGg==", 2024-01-27T01:03:41.144 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.144 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:03:41.144 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:03:41.144 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:03:41.144 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.144 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.144 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.144 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-01-27T01:03:41.144 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQQLRlNGfhIxAAmCgmOB3QAGfSIYHBb5BUZw==", 2024-01-27T01:03:41.144 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.145 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:03:41.145 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:03:41.145 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:03:41.145 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.145 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.145 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.145 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-01-27T01:03:41.145 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRleO8wBhAA1018AcO3KcyOZxgB0Itxaw==", 2024-01-27T01:03:41.145 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.145 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:03:41.145 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-01-27T01:03:41.145 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-27T01:03:41.145 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.145 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.145 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.146 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-01-27T01:03:41.146 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlsiZEDRAAjXSamHe2bZcDqVsqXqIcGg==", 2024-01-27T01:03:41.146 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.146 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:03:41.146 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T01:03:41.146 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T01:03:41.146 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T01:03:41.146 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.146 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.146 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.146 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-01-27T01:03:41.146 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl6YT1CRAApXlN4j20xzwda0tDMy94Eg==", 2024-01-27T01:03:41.146 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.146 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:03:41.146 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T01:03:41.147 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T01:03:41.147 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T01:03:41.147 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.147 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.147 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.147 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-01-27T01:03:41.147 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlHvDQEhAAxMhgWJsZC5/6Af5zFV6XIA==", 2024-01-27T01:03:41.147 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.147 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:03:41.147 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T01:03:41.147 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T01:03:41.147 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T01:03:41.147 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.147 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.147 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.148 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-01-27T01:03:41.148 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlvMwDCBAAXnFyMduz7PfWdLWbjytIiQ==", 2024-01-27T01:03:41.148 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.148 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T01:03:41.148 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:03:41.148 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:03:41.148 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:03:41.148 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.148 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.148 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.148 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-01-27T01:03:41.148 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlhMNSJBAA38QU6DrCvFjO+u9zQ8bkEA==", 2024-01-27T01:03:41.148 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.148 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T01:03:41.148 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-27T01:03:41.149 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:03:41.149 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.149 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:41.149 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:41.149 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-01-27T01:03:41.149 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlI9EbIBAACTpzAO+8A/VFJZcT/WWBng==", 2024-01-27T01:03:41.149 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:03:41.149 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T01:03:41.149 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-27T01:03:41.149 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:03:41.149 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.149 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:41.149 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-01-27T01:03:41.149 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:03:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.139+0000 7fc3e0ae0700 1 -- 172.21.15.46:0/2582724355 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc3bc066f30 msgr2=0x7fc3bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.139+0000 7fc3e0ae0700 1 --2- 172.21.15.46:0/2582724355 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc3bc066f30 0x7fc3bc0693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc009ca0 tx=0x7fc3cc008040 comp rx=0 tx=0).stop 2024-01-27T01:03:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.139+0000 7fc3e0ae0700 1 -- 172.21.15.46:0/2582724355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b3060 msgr2=0x7fc3dc16cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.139+0000 7fc3e0ae0700 1 --2- 172.21.15.46:0/2582724355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b3060 0x7fc3dc16cad0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fc3d000c590 tx=0x7fc3d000c950 comp rx=0 tx=0).stop 2024-01-27T01:03:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.139+0000 7fc3e0ae0700 1 -- 172.21.15.46:0/2582724355 shutdown_connections 2024-01-27T01:03:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.139+0000 7fc3e0ae0700 1 --2- 172.21.15.46:0/2582724355 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc3bc066f30 0x7fc3bc0693e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.139+0000 7fc3e0ae0700 1 --2- 172.21.15.46:0/2582724355 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3dc159760 0x7fc3dc16d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.139+0000 7fc3e0ae0700 1 --2- 172.21.15.46:0/2582724355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3dc0b12c0 0x7fc3dc155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.139+0000 7fc3e0ae0700 1 --2- 172.21.15.46:0/2582724355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc0b3060 0x7fc3dc16cad0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.139+0000 7fc3e0ae0700 1 -- 172.21.15.46:0/2582724355 >> 172.21.15.46:0/2582724355 conn(0x7fc3dc069510 msgr2=0x7fc3dc0b04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.139+0000 7fc3e0ae0700 1 -- 172.21.15.46:0/2582724355 shutdown_connections 2024-01-27T01:03:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.139+0000 7fc3e0ae0700 1 -- 172.21.15.46:0/2582724355 wait complete. 2024-01-27T01:03:41.152 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:03:41.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.286+0000 7fda7289c700 1 Processor -- start 2024-01-27T01:03:41.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.286+0000 7fda7289c700 1 -- start start 2024-01-27T01:03:41.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.287+0000 7fda7289c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fda6c0ccc20 0x7fda6c068340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:41.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.287+0000 7fda7289c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda6c0ccff0 0x7fda6c06eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:41.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.287+0000 7fda7289c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda6c0b2590 0x7fda6c0b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:41.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.287+0000 7fda7289c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda6c0b2f10 con 0x7fda6c0b2590 2024-01-27T01:03:41.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.287+0000 7fda7289c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda6c158020 con 0x7fda6c0ccc20 2024-01-27T01:03:41.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.287+0000 7fda7289c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c158300 con 0x7fda6c0ccff0 2024-01-27T01:03:41.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.287+0000 7fda70e39700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda6c0b2590 0x7fda6c0b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:41.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.287+0000 7fda70e39700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda6c0b2590 0x7fda6c0b29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38882/0 (socket says 172.21.15.46:38882) 2024-01-27T01:03:41.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.287+0000 7fda70e39700 1 -- 172.21.15.46:0/3565033114 learned_addr learned my addr 172.21.15.46:0/3565033114 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.287+0000 7fda6b7fe700 1 --2- 172.21.15.46:0/3565033114 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda6c0ccff0 0x7fda6c06eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.287+0000 7fda6bfff700 1 --2- 172.21.15.46:0/3565033114 >> v2:172.21.15.106:3300/0 conn(0x7fda6c0ccc20 0x7fda6c068340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.287+0000 7fda70e39700 1 -- 172.21.15.46:0/3565033114 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda6c0ccff0 msgr2=0x7fda6c06eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.287+0000 7fda70e39700 1 --2- 172.21.15.46:0/3565033114 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda6c0ccff0 0x7fda6c06eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.287+0000 7fda70e39700 1 -- 172.21.15.46:0/3565033114 >> v2:172.21.15.106:3300/0 conn(0x7fda6c0ccc20 msgr2=0x7fda6c068340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.287+0000 7fda70e39700 1 --2- 172.21.15.46:0/3565033114 >> v2:172.21.15.106:3300/0 conn(0x7fda6c0ccc20 0x7fda6c068340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.287+0000 7fda70e39700 1 -- 172.21.15.46:0/3565033114 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c1585e0 con 0x7fda6c0b2590 2024-01-27T01:03:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.288+0000 7fda70e39700 1 --2- 172.21.15.46:0/3565033114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6c0b2590 0x7fda6c0b29d0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fda6c06e840 tx=0x7fda64009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dae877923c81ec9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.288+0000 7fda6affd700 1 -- 172.21.15.46:0/3565033114 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6400e070 con 0x7fda6c0b2590 2024-01-27T01:03:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.288+0000 7fda6affd700 1 -- 172.21.15.46:0/3565033114 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda6400b970 con 0x7fda6c0b2590 2024-01-27T01:03:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.288+0000 7fda6affd700 1 -- 172.21.15.46:0/3565033114 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda640092e0 con 0x7fda6c0b2590 2024-01-27T01:03:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.288+0000 7fda7289c700 1 -- 172.21.15.46:0/3565033114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6c0b2590 msgr2=0x7fda6c0b29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:41.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.288+0000 7fda7289c700 1 --2- 172.21.15.46:0/3565033114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6c0b2590 0x7fda6c0b29d0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fda6c06e840 tx=0x7fda64009840 comp rx=0 tx=0).stop 2024-01-27T01:03:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.289+0000 7fda7289c700 1 -- 172.21.15.46:0/3565033114 shutdown_connections 2024-01-27T01:03:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.289+0000 7fda7289c700 1 --2- 172.21.15.46:0/3565033114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6c0b2590 0x7fda6c0b29d0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.289+0000 7fda7289c700 1 --2- 172.21.15.46:0/3565033114 >> v2:172.21.15.106:3300/0 conn(0x7fda6c0ccc20 0x7fda6c068340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.289+0000 7fda7289c700 1 --2- 172.21.15.46:0/3565033114 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda6c0ccff0 0x7fda6c06eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.289+0000 7fda7289c700 1 -- 172.21.15.46:0/3565033114 >> 172.21.15.46:0/3565033114 conn(0x7fda6c06b630 msgr2=0x7fda6c0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.289+0000 7fda7289c700 1 -- 172.21.15.46:0/3565033114 shutdown_connections 2024-01-27T01:03:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.289+0000 7fda7289c700 1 -- 172.21.15.46:0/3565033114 wait complete. 2024-01-27T01:03:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.290+0000 7fda7289c700 1 Processor -- start 2024-01-27T01:03:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.290+0000 7fda7289c700 1 -- start start 2024-01-27T01:03:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.291+0000 7fda7289c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6c0b2590 0x7fda6c0aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.291+0000 7fda7289c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda6c0ccc20 0x7fda6c0a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.291+0000 7fda7289c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda6c0ccff0 0x7fda6c0a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.291+0000 7fda7289c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c0a5eb0 con 0x7fda6c0b2590 2024-01-27T01:03:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.291+0000 7fda7289c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c16a9b0 con 0x7fda6c0ccff0 2024-01-27T01:03:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.291+0000 7fda7289c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c0af1a0 con 0x7fda6c0ccc20 2024-01-27T01:03:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.291+0000 7fda6bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6c0b2590 0x7fda6c0aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.291+0000 7fda6bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6c0b2590 0x7fda6c0aec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38886/0 (socket says 172.21.15.46:38886) 2024-01-27T01:03:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.291+0000 7fda6bfff700 1 -- 172.21.15.46:0/2590511003 learned_addr learned my addr 172.21.15.46:0/2590511003 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.291+0000 7fda6b7fe700 1 --2- 172.21.15.46:0/2590511003 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda6c0ccc20 0x7fda6c0a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.291+0000 7fda70e39700 1 --2- 172.21.15.46:0/2590511003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda6c0ccff0 0x7fda6c0a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.291+0000 7fda6bfff700 1 -- 172.21.15.46:0/2590511003 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda6c0ccc20 msgr2=0x7fda6c0a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.291+0000 7fda6bfff700 1 --2- 172.21.15.46:0/2590511003 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda6c0ccc20 0x7fda6c0a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.291+0000 7fda6bfff700 1 -- 172.21.15.46:0/2590511003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda6c0ccff0 msgr2=0x7fda6c0a5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.291+0000 7fda6bfff700 1 --2- 172.21.15.46:0/2590511003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda6c0ccff0 0x7fda6c0a5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.291+0000 7fda6bfff700 1 -- 172.21.15.46:0/2590511003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda64009ae0 con 0x7fda6c0b2590 2024-01-27T01:03:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.291+0000 7fda6bfff700 1 --2- 172.21.15.46:0/2590511003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6c0b2590 0x7fda6c0aec60 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00ec80 tx=0x7fda5c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.292+0000 7fda53fff700 1 -- 172.21.15.46:0/2590511003 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5c00c9d0 con 0x7fda6c0b2590 2024-01-27T01:03:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.292+0000 7fda53fff700 1 -- 172.21.15.46:0/2590511003 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5c00cb30 con 0x7fda6c0b2590 2024-01-27T01:03:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.292+0000 7fda7289c700 1 -- 172.21.15.46:0/2590511003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda6c16ac00 con 0x7fda6c0b2590 2024-01-27T01:03:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.292+0000 7fda53fff700 1 -- 172.21.15.46:0/2590511003 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5c013470 con 0x7fda6c0b2590 2024-01-27T01:03:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.292+0000 7fda7289c700 1 -- 172.21.15.46:0/2590511003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda6c16b1c0 con 0x7fda6c0b2590 2024-01-27T01:03:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.293+0000 7fda7289c700 1 -- 172.21.15.46:0/2590511003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6c04c710 con 0x7fda6c0b2590 2024-01-27T01:03:41.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.294+0000 7fda53fff700 1 -- 172.21.15.46:0/2590511003 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fda5c013820 con 0x7fda6c0b2590 2024-01-27T01:03:41.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.294+0000 7fda53fff700 1 --2- 172.21.15.46:0/2590511003 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fda58066f30 0x7fda580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:41.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.294+0000 7fda53fff700 1 -- 172.21.15.46:0/2590511003 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5637+0+0 (secure 0 0 0) 0x7fda5c085020 con 0x7fda6c0b2590 2024-01-27T01:03:41.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.294+0000 7fda6b7fe700 1 --2- 172.21.15.46:0/2590511003 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fda58066f30 0x7fda580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:41.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.295+0000 7fda6b7fe700 1 --2- 172.21.15.46:0/2590511003 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fda58066f30 0x7fda580693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fda6c16a930 tx=0x7fda60008040 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:41.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.298+0000 7fda53fff700 1 -- 172.21.15.46:0/2590511003 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fda5c052350 con 0x7fda6c0b2590 2024-01-27T01:03:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.504+0000 7fda7289c700 1 -- 172.21.15.46:0/2590511003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda6c1588c0 con 0x7fda6c0b2590 2024-01-27T01:03:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.505+0000 7fda53fff700 1 -- 172.21.15.46:0/2590511003 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v201) v1 ==== 74+0+13060 (secure 0 0 0) 0x7fda5c055970 con 0x7fda6c0b2590 2024-01-27T01:03:41.506 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:41.506 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":201,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:03:40.678832+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:03:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.508+0000 7fda7289c700 1 -- 172.21.15.46:0/2590511003 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fda58066f30 msgr2=0x7fda580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.508+0000 7fda7289c700 1 --2- 172.21.15.46:0/2590511003 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fda58066f30 0x7fda580693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fda6c16a930 tx=0x7fda60008040 comp rx=0 tx=0).stop 2024-01-27T01:03:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.508+0000 7fda7289c700 1 -- 172.21.15.46:0/2590511003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6c0b2590 msgr2=0x7fda6c0aec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.508+0000 7fda7289c700 1 --2- 172.21.15.46:0/2590511003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6c0b2590 0x7fda6c0aec60 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00ec80 tx=0x7fda5c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.508+0000 7fda7289c700 1 -- 172.21.15.46:0/2590511003 shutdown_connections 2024-01-27T01:03:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.508+0000 7fda7289c700 1 --2- 172.21.15.46:0/2590511003 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fda58066f30 0x7fda580693e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.508+0000 7fda7289c700 1 --2- 172.21.15.46:0/2590511003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda6c0b2590 0x7fda6c0aec60 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.508+0000 7fda7289c700 1 --2- 172.21.15.46:0/2590511003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda6c0ccff0 0x7fda6c0a5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.508+0000 7fda7289c700 1 --2- 172.21.15.46:0/2590511003 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda6c0ccc20 0x7fda6c0a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.508+0000 7fda7289c700 1 -- 172.21.15.46:0/2590511003 >> 172.21.15.46:0/2590511003 conn(0x7fda6c06b630 msgr2=0x7fda6c158f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.508+0000 7fda7289c700 1 -- 172.21.15.46:0/2590511003 shutdown_connections 2024-01-27T01:03:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.508+0000 7fda7289c700 1 -- 172.21.15.46:0/2590511003 wait complete. 2024-01-27T01:03:41.522 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-27T01:03:41.522 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-27T01:03:41.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.656+0000 7f71787b9700 1 Processor -- start 2024-01-27T01:03:41.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.656+0000 7f71787b9700 1 -- start start 2024-01-27T01:03:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.657+0000 7f71787b9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f717015d110 0x7f7170067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.657+0000 7f71787b9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7170067e70 0x7f717006eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:41.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.657+0000 7f71787b9700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f717006caa0 con 0x7f717015cd40 2024-01-27T01:03:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.657+0000 7f71787b9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71700c7ac0 con 0x7f7170067e70 2024-01-27T01:03:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.657+0000 7f71787b9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71700682f0 con 0x7f717015d110 2024-01-27T01:03:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.658+0000 7f7176d56700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7170067e70 0x7f717006eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.658+0000 7f7176d56700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7170067e70 0x7f717006eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38888/0 (socket says 172.21.15.46:38888) 2024-01-27T01:03:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.658+0000 7f7176d56700 1 -- 172.21.15.46:0/1095871762 learned_addr learned my addr 172.21.15.46:0/1095871762 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.658+0000 7f7175d54700 1 --2- 172.21.15.46:0/1095871762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f717015d110 0x7f7170067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.658+0000 7f7176d56700 1 -- 172.21.15.46:0/1095871762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f717015d110 msgr2=0x7f7170067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.658+0000 7f7176d56700 1 --2- 172.21.15.46:0/1095871762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f717015d110 0x7f7170067930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.658+0000 7f7176d56700 1 -- 172.21.15.46:0/1095871762 >> v1:172.21.15.46:6789/0 conn(0x7f717015cd40 legacy=0x7f71700cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.658+0000 7f7176d56700 1 -- 172.21.15.46:0/1095871762 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71700ca8b0 con 0x7f7170067e70 2024-01-27T01:03:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.658+0000 7f7176d56700 1 --2- 172.21.15.46:0/1095871762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7170067e70 0x7f717006eb90 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f7168009f20 tx=0x7f7168009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e666a6c9238173 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.658+0000 7f7175553700 1 -- 172.21.15.46:0/1095871762 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716800e070 con 0x7f7170067e70 2024-01-27T01:03:41.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.658+0000 7f7175553700 1 -- 172.21.15.46:0/1095871762 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f716800ba80 con 0x7f7170067e70 2024-01-27T01:03:41.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.658+0000 7f7175553700 1 -- 172.21.15.46:0/1095871762 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71680092e0 con 0x7f7170067e70 2024-01-27T01:03:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.659+0000 7f71787b9700 1 -- 172.21.15.46:0/1095871762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7170067e70 msgr2=0x7f717006eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.659+0000 7f71787b9700 1 --2- 172.21.15.46:0/1095871762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7170067e70 0x7f717006eb90 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f7168009f20 tx=0x7f7168009840 comp rx=0 tx=0).stop 2024-01-27T01:03:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.659+0000 7f71787b9700 1 -- 172.21.15.46:0/1095871762 shutdown_connections 2024-01-27T01:03:41.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.659+0000 7f71787b9700 1 --2- 172.21.15.46:0/1095871762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7170067e70 0x7f717006eb90 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.659+0000 7f71787b9700 1 --2- 172.21.15.46:0/1095871762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f717015d110 0x7f7170067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.659+0000 7f71787b9700 1 -- 172.21.15.46:0/1095871762 >> 172.21.15.46:0/1095871762 conn(0x7f71700cdf50 msgr2=0x7f71700c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.659+0000 7f71787b9700 1 -- 172.21.15.46:0/1095871762 shutdown_connections 2024-01-27T01:03:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.659+0000 7f71787b9700 1 -- 172.21.15.46:0/1095871762 wait complete. 2024-01-27T01:03:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.660+0000 7f71787b9700 1 Processor -- start 2024-01-27T01:03:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.660+0000 7f71787b9700 1 -- start start 2024-01-27T01:03:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.661+0000 7f71787b9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7170067e70 0x7f7170149150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.661+0000 7f71787b9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f717015cd40 0x7f717015add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.661+0000 7f71787b9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f717015d110 0x7f71700c4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.661+0000 7f71787b9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7170158f20 con 0x7f717015cd40 2024-01-27T01:03:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.661+0000 7f71787b9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7170149690 con 0x7f7170067e70 2024-01-27T01:03:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.661+0000 7f71787b9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7170159130 con 0x7f717015d110 2024-01-27T01:03:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.661+0000 7f7175d54700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f717015cd40 0x7f717015add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:41.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.661+0000 7f7175d54700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f717015cd40 0x7f717015add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38896/0 (socket says 172.21.15.46:38896) 2024-01-27T01:03:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.661+0000 7f7175d54700 1 -- 172.21.15.46:0/2265259070 learned_addr learned my addr 172.21.15.46:0/2265259070 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.661+0000 7f7176d56700 1 --2- 172.21.15.46:0/2265259070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f717015d110 0x7f71700c4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.661+0000 7f7176555700 1 --2- 172.21.15.46:0/2265259070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7170067e70 0x7f7170149150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.662+0000 7f7175d54700 1 -- 172.21.15.46:0/2265259070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f717015d110 msgr2=0x7f71700c4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.662+0000 7f7175d54700 1 --2- 172.21.15.46:0/2265259070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f717015d110 0x7f71700c4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.662+0000 7f7175d54700 1 -- 172.21.15.46:0/2265259070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7170067e70 msgr2=0x7f7170149150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.662+0000 7f7175d54700 1 --2- 172.21.15.46:0/2265259070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7170067e70 0x7f7170149150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.662+0000 7f7175d54700 1 -- 172.21.15.46:0/2265259070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7168009ae0 con 0x7f717015cd40 2024-01-27T01:03:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.662+0000 7f7175d54700 1 --2- 172.21.15.46:0/2265259070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f717015cd40 0x7f717015add0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f716c00d890 tx=0x7f716c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.662+0000 7f71627fc700 1 -- 172.21.15.46:0/2265259070 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716c00b600 con 0x7f717015cd40 2024-01-27T01:03:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.662+0000 7f71627fc700 1 -- 172.21.15.46:0/2265259070 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f716c00b760 con 0x7f717015cd40 2024-01-27T01:03:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.662+0000 7f71787b9700 1 -- 172.21.15.46:0/2265259070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7170159410 con 0x7f717015cd40 2024-01-27T01:03:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.662+0000 7f71627fc700 1 -- 172.21.15.46:0/2265259070 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716c00bae0 con 0x7f717015cd40 2024-01-27T01:03:41.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.662+0000 7f71787b9700 1 -- 172.21.15.46:0/2265259070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71701599d0 con 0x7f717015cd40 2024-01-27T01:03:41.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.663+0000 7f71787b9700 1 -- 172.21.15.46:0/2265259070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f717004c710 con 0x7f717015cd40 2024-01-27T01:03:41.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.663+0000 7f71627fc700 1 -- 172.21.15.46:0/2265259070 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f716c01e460 con 0x7f717015cd40 2024-01-27T01:03:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.664+0000 7f71627fc700 1 --2- 172.21.15.46:0/2265259070 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7154066f30 0x7f71540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.664+0000 7f71627fc700 1 -- 172.21.15.46:0/2265259070 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5637+0+0 (secure 0 0 0) 0x7f716c084bd0 con 0x7f717015cd40 2024-01-27T01:03:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.664+0000 7f7176555700 1 --2- 172.21.15.46:0/2265259070 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7154066f30 0x7f71540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:41.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.665+0000 7f7176555700 1 --2- 172.21.15.46:0/2265259070 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7154066f30 0x7f71540693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7164003860 tx=0x7f7164009040 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.668+0000 7f71627fc700 1 -- 172.21.15.46:0/2265259070 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f716c051e80 con 0x7f717015cd40 2024-01-27T01:03:41.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.875+0000 7f71787b9700 1 -- 172.21.15.46:0/2265259070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f717006a530 con 0x7f717015cd40 2024-01-27T01:03:41.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.876+0000 7f71627fc700 1 -- 172.21.15.46:0/2265259070 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v201) v1 ==== 93+0+10 (secure 0 0 0) 0x7f716c0554a0 con 0x7f717015cd40 2024-01-27T01:03:41.876 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-27T01:03:41.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.878+0000 7f71787b9700 1 -- 172.21.15.46:0/2265259070 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7154066f30 msgr2=0x7f71540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:41.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.878+0000 7f71787b9700 1 --2- 172.21.15.46:0/2265259070 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7154066f30 0x7f71540693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7164003860 tx=0x7f7164009040 comp rx=0 tx=0).stop 2024-01-27T01:03:41.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.878+0000 7f71787b9700 1 -- 172.21.15.46:0/2265259070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f717015cd40 msgr2=0x7f717015add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:41.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.878+0000 7f71787b9700 1 --2- 172.21.15.46:0/2265259070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f717015cd40 0x7f717015add0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f716c00d890 tx=0x7f716c008040 comp rx=0 tx=0).stop 2024-01-27T01:03:41.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.878+0000 7f71787b9700 1 -- 172.21.15.46:0/2265259070 shutdown_connections 2024-01-27T01:03:41.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.878+0000 7f71787b9700 1 --2- 172.21.15.46:0/2265259070 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7154066f30 0x7f71540693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.878+0000 7f71787b9700 1 --2- 172.21.15.46:0/2265259070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f717015d110 0x7f71700c4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.878+0000 7f71787b9700 1 --2- 172.21.15.46:0/2265259070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7170067e70 0x7f7170149150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.878+0000 7f71787b9700 1 --2- 172.21.15.46:0/2265259070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f717015cd40 0x7f717015add0 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:41.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.878+0000 7f71787b9700 1 -- 172.21.15.46:0/2265259070 >> 172.21.15.46:0/2265259070 conn(0x7f71700cdf50 msgr2=0x7f71700c8060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:41.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.878+0000 7f71787b9700 1 -- 172.21.15.46:0/2265259070 shutdown_connections 2024-01-27T01:03:41.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:41.878+0000 7f71787b9700 1 -- 172.21.15.46:0/2265259070 wait complete. 2024-01-27T01:03:41.892 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-27T01:03:42.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.023+0000 7f809bc60700 1 Processor -- start 2024-01-27T01:03:42.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.023+0000 7f809bc60700 1 -- start start 2024-01-27T01:03:42.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.024+0000 7f809bc60700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f809415d210 0x7f80940cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:42.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.024+0000 7f809bc60700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f80940cbff0 0x7f80940ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:42.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.024+0000 7f809bc60700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80940cc450 con 0x7f809415ce40 2024-01-27T01:03:42.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.024+0000 7f809bc60700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80940cff70 con 0x7f809415d210 2024-01-27T01:03:42.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.024+0000 7f809bc60700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80940cf400 con 0x7f80940cbff0 2024-01-27T01:03:42.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.025+0000 7f80991fb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f809415d210 0x7f80940cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:42.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.025+0000 7f80991fb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f809415d210 0x7f80940cbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38912/0 (socket says 172.21.15.46:38912) 2024-01-27T01:03:42.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.025+0000 7f80991fb700 1 -- 172.21.15.46:0/1733631715 learned_addr learned my addr 172.21.15.46:0/1733631715 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:42.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.025+0000 7f809a1fd700 1 --2- 172.21.15.46:0/1733631715 >> v2:172.21.15.106:3300/0 conn(0x7f80940cbff0 0x7f80940ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:42.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.025+0000 7f80991fb700 1 -- 172.21.15.46:0/1733631715 >> v2:172.21.15.106:3300/0 conn(0x7f80940cbff0 msgr2=0x7f80940ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:42.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.025+0000 7f80991fb700 1 --2- 172.21.15.46:0/1733631715 >> v2:172.21.15.106:3300/0 conn(0x7f80940cbff0 0x7f80940ade30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.025+0000 7f80991fb700 1 -- 172.21.15.46:0/1733631715 >> v1:172.21.15.106:6789/0 conn(0x7f809415ce40 legacy=0x7f80940d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:42.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.025+0000 7f80991fb700 1 -- 172.21.15.46:0/1733631715 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80940cf660 con 0x7f809415d210 2024-01-27T01:03:42.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.025+0000 7f80991fb700 1 --2- 172.21.15.46:0/1733631715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f809415d210 0x7f80940cbab0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f80940d01c0 tx=0x7f8084009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1af95c6b6a31a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:42.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.025+0000 7f80989fa700 1 -- 172.21.15.46:0/1733631715 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808400b820 con 0x7f809415d210 2024-01-27T01:03:42.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.025+0000 7f80989fa700 1 -- 172.21.15.46:0/1733631715 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808400b980 con 0x7f809415d210 2024-01-27T01:03:42.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.025+0000 7f80989fa700 1 -- 172.21.15.46:0/1733631715 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808400bd20 con 0x7f809415d210 2024-01-27T01:03:42.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.026+0000 7f809bc60700 1 -- 172.21.15.46:0/1733631715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f809415d210 msgr2=0x7f80940cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:42.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.026+0000 7f809bc60700 1 --2- 172.21.15.46:0/1733631715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f809415d210 0x7f80940cbab0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f80940d01c0 tx=0x7f8084009250 comp rx=0 tx=0).stop 2024-01-27T01:03:42.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.026+0000 7f809bc60700 1 -- 172.21.15.46:0/1733631715 shutdown_connections 2024-01-27T01:03:42.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.026+0000 7f809bc60700 1 --2- 172.21.15.46:0/1733631715 >> v2:172.21.15.106:3300/0 conn(0x7f80940cbff0 0x7f80940ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.026+0000 7f809bc60700 1 --2- 172.21.15.46:0/1733631715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f809415d210 0x7f80940cbab0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.026+0000 7f809bc60700 1 -- 172.21.15.46:0/1733631715 >> 172.21.15.46:0/1733631715 conn(0x7f809406f840 msgr2=0x7f80940d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:42.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.026+0000 7f809bc60700 1 -- 172.21.15.46:0/1733631715 shutdown_connections 2024-01-27T01:03:42.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.026+0000 7f809bc60700 1 -- 172.21.15.46:0/1733631715 wait complete. 2024-01-27T01:03:42.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.027+0000 7f809bc60700 1 Processor -- start 2024-01-27T01:03:42.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.027+0000 7f809bc60700 1 -- start start 2024-01-27T01:03:42.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.028+0000 7f809bc60700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80940cbff0 0x7f809416aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:42.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.028+0000 7f809bc60700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f809415ce40 0x7f809409d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:42.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.028+0000 7f809bc60700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f809415d210 0x7f80940689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:42.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.028+0000 7f809bc60700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809416af70 con 0x7f809415d210 2024-01-27T01:03:42.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.028+0000 7f809bc60700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809409c890 con 0x7f80940cbff0 2024-01-27T01:03:42.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.028+0000 7f809bc60700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f809409d130 con 0x7f809415ce40 2024-01-27T01:03:42.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.028+0000 7f809a1fd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f809415d210 0x7f80940689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:42.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.028+0000 7f809a1fd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f809415d210 0x7f80940689b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38918/0 (socket says 172.21.15.46:38918) 2024-01-27T01:03:42.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.028+0000 7f809a1fd700 1 -- 172.21.15.46:0/147945256 learned_addr learned my addr 172.21.15.46:0/147945256 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:42.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.028+0000 7f80991fb700 1 --2- 172.21.15.46:0/147945256 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f809415ce40 0x7f809409d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:42.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.028+0000 7f80999fc700 1 --2- 172.21.15.46:0/147945256 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80940cbff0 0x7f809416aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:42.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.029+0000 7f809a1fd700 1 -- 172.21.15.46:0/147945256 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f809415ce40 msgr2=0x7f809409d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:42.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.029+0000 7f809a1fd700 1 --2- 172.21.15.46:0/147945256 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f809415ce40 0x7f809409d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.029+0000 7f809a1fd700 1 -- 172.21.15.46:0/147945256 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80940cbff0 msgr2=0x7f809416aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:42.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.029+0000 7f809a1fd700 1 --2- 172.21.15.46:0/147945256 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80940cbff0 0x7f809416aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.029+0000 7f809a1fd700 1 -- 172.21.15.46:0/147945256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80840094f0 con 0x7f809415d210 2024-01-27T01:03:42.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.029+0000 7f809a1fd700 1 --2- 172.21.15.46:0/147945256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f809415d210 0x7f80940689b0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f808c007f50 tx=0x7f808c009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:42.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.029+0000 7f8089ffb700 1 -- 172.21.15.46:0/147945256 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c00eb70 con 0x7f809415d210 2024-01-27T01:03:42.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.029+0000 7f8089ffb700 1 -- 172.21.15.46:0/147945256 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808c00ecf0 con 0x7f809415d210 2024-01-27T01:03:42.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.029+0000 7f809bc60700 1 -- 172.21.15.46:0/147945256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f809409c150 con 0x7f809415d210 2024-01-27T01:03:42.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.029+0000 7f8089ffb700 1 -- 172.21.15.46:0/147945256 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c014670 con 0x7f809415d210 2024-01-27T01:03:42.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.029+0000 7f809bc60700 1 -- 172.21.15.46:0/147945256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f809409cae0 con 0x7f809415d210 2024-01-27T01:03:42.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.030+0000 7f809bc60700 1 -- 172.21.15.46:0/147945256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80940d15b0 con 0x7f809415d210 2024-01-27T01:03:42.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.031+0000 7f8089ffb700 1 -- 172.21.15.46:0/147945256 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f808c0149e0 con 0x7f809415d210 2024-01-27T01:03:42.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.031+0000 7f8089ffb700 1 --2- 172.21.15.46:0/147945256 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8078066f30 0x7f80780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:42.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.031+0000 7f8089ffb700 1 -- 172.21.15.46:0/147945256 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5637+0+0 (secure 0 0 0) 0x7f808c084ae0 con 0x7f809415d210 2024-01-27T01:03:42.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.031+0000 7f80999fc700 1 --2- 172.21.15.46:0/147945256 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8078066f30 0x7f80780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:42.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.032+0000 7f80999fc700 1 --2- 172.21.15.46:0/147945256 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8078066f30 0x7f80780693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8090003a70 tx=0x7f8090007ef0 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:42.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.036+0000 7f8089ffb700 1 -- 172.21.15.46:0/147945256 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f808c051d90 con 0x7f809415d210 2024-01-27T01:03:42.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.241+0000 7f809bc60700 1 -- 172.21.15.46:0/147945256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f809404c710 con 0x7f809415d210 2024-01-27T01:03:42.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.241+0000 7f8089ffb700 1 -- 172.21.15.46:0/147945256 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v201) v1 ==== 84+0+65 (secure 0 0 0) 0x7f808c0553b0 con 0x7f809415d210 2024-01-27T01:03:42.241 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:42.241 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-27T01:03:42.241 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:42.241 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-01-27T01:03:42.242 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-01-27T01:03:42.242 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:42.242 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-27T01:03:42.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.243+0000 7f809bc60700 1 -- 172.21.15.46:0/147945256 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8078066f30 msgr2=0x7f80780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:42.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.243+0000 7f809bc60700 1 --2- 172.21.15.46:0/147945256 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8078066f30 0x7f80780693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8090003a70 tx=0x7f8090007ef0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.243+0000 7f809bc60700 1 -- 172.21.15.46:0/147945256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f809415d210 msgr2=0x7f80940689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:42.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.243+0000 7f809bc60700 1 --2- 172.21.15.46:0/147945256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f809415d210 0x7f80940689b0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f808c007f50 tx=0x7f808c009860 comp rx=0 tx=0).stop 2024-01-27T01:03:42.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.243+0000 7f809bc60700 1 -- 172.21.15.46:0/147945256 shutdown_connections 2024-01-27T01:03:42.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.243+0000 7f809bc60700 1 --2- 172.21.15.46:0/147945256 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8078066f30 0x7f80780693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.243+0000 7f809bc60700 1 --2- 172.21.15.46:0/147945256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f809415d210 0x7f80940689b0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.243+0000 7f809bc60700 1 --2- 172.21.15.46:0/147945256 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80940cbff0 0x7f809416aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.243+0000 7f809bc60700 1 --2- 172.21.15.46:0/147945256 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f809415ce40 0x7f809409d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.243+0000 7f809bc60700 1 -- 172.21.15.46:0/147945256 >> 172.21.15.46:0/147945256 conn(0x7f809406f840 msgr2=0x7f80940cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:42.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.243+0000 7f809bc60700 1 -- 172.21.15.46:0/147945256 shutdown_connections 2024-01-27T01:03:42.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.243+0000 7f809bc60700 1 -- 172.21.15.46:0/147945256 wait complete. 2024-01-27T01:03:42.257 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-27T01:03:42.258 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-27T01:03:42.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.393+0000 7f9bb184d700 1 Processor -- start 2024-01-27T01:03:42.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.393+0000 7f9bb184d700 1 -- start start 2024-01-27T01:03:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.395+0000 7f9bb184d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bac15ce40 0x7f9bac0d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.395+0000 7f9bb184d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bac0cff70 con 0x7f9bac15d210 2024-01-27T01:03:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.395+0000 7f9bb184d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bac0cf400 con 0x7f9bac0cc1c0 2024-01-27T01:03:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.395+0000 7f9bb184d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac0cf660 con 0x7f9bac15ce40 2024-01-27T01:03:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.395+0000 7f9baa7fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9bac15d210 0x7f9bac0cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57294/0 (socket says 172.21.15.46:57294) 2024-01-27T01:03:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.395+0000 7f9baa7fc700 1 -- 172.21.15.46:0/2616073122 learned_addr learned my addr 172.21.15.46:0/2616073122 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.395+0000 7f9baaffd700 1 --2- 172.21.15.46:0/2616073122 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bac15ce40 0x7f9bac0d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.395+0000 7f9ba9ffb700 1 -- 172.21.15.46:0/2616073122 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915521998 0 0) 0x7f9bac0cff70 con 0x7f9bac15d210 2024-01-27T01:03:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.395+0000 7f9ba9ffb700 1 -- 172.21.15.46:0/2616073122 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b98003570 con 0x7f9bac15d210 2024-01-27T01:03:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.395+0000 7f9ba9ffb700 1 -- 172.21.15.46:0/2616073122 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471846179 0 0) 0x7f9bac0cf400 con 0x7f9bac0cc1c0 2024-01-27T01:03:42.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.395+0000 7f9ba9ffb700 1 -- 172.21.15.46:0/2616073122 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bac0cff70 con 0x7f9bac0cc1c0 2024-01-27T01:03:42.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.395+0000 7f9baaffd700 1 -- 172.21.15.46:0/2616073122 >> v1:172.21.15.106:6789/0 conn(0x7f9bac0cc1c0 legacy=0x7f9bac0ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:42.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.395+0000 7f9baaffd700 1 -- 172.21.15.46:0/2616073122 >> v1:172.21.15.46:6789/0 conn(0x7f9bac15d210 legacy=0x7f9bac0cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:42.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.395+0000 7f9baaffd700 1 -- 172.21.15.46:0/2616073122 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac0d0500 con 0x7f9bac15ce40 2024-01-27T01:03:42.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.396+0000 7f9baaffd700 1 --2- 172.21.15.46:0/2616073122 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bac15ce40 0x7f9bac0d0e50 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c009910 tx=0x7f9b9c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9eb8c956f0742680 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:42.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.396+0000 7f9ba9ffb700 1 -- 172.21.15.46:0/2616073122 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c011630 con 0x7f9bac15ce40 2024-01-27T01:03:42.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.396+0000 7f9ba9ffb700 1 -- 172.21.15.46:0/2616073122 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9c011790 con 0x7f9bac15ce40 2024-01-27T01:03:42.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.396+0000 7f9ba9ffb700 1 -- 172.21.15.46:0/2616073122 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c011b30 con 0x7f9bac15ce40 2024-01-27T01:03:42.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.396+0000 7f9bb184d700 1 -- 172.21.15.46:0/2616073122 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bac15ce40 msgr2=0x7f9bac0d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:42.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.396+0000 7f9bb184d700 1 --2- 172.21.15.46:0/2616073122 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bac15ce40 0x7f9bac0d0e50 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c009910 tx=0x7f9b9c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.396+0000 7f9bb184d700 1 -- 172.21.15.46:0/2616073122 shutdown_connections 2024-01-27T01:03:42.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.396+0000 7f9bb184d700 1 --2- 172.21.15.46:0/2616073122 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bac15ce40 0x7f9bac0d0e50 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.396+0000 7f9bb184d700 1 -- 172.21.15.46:0/2616073122 >> 172.21.15.46:0/2616073122 conn(0x7f9bac06f840 msgr2=0x7f9bac0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:42.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.397+0000 7f9bb184d700 1 -- 172.21.15.46:0/2616073122 shutdown_connections 2024-01-27T01:03:42.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.397+0000 7f9bb184d700 1 -- 172.21.15.46:0/2616073122 wait complete. 2024-01-27T01:03:42.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.398+0000 7f9bb184d700 1 Processor -- start 2024-01-27T01:03:42.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.398+0000 7f9bb184d700 1 -- start start 2024-01-27T01:03:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.399+0000 7f9bb184d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bac0cc1c0 0x7f9bac0c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.399+0000 7f9bb184d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bac15ce40 0x7f9bac16a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.399+0000 7f9bb184d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bac15d210 0x7f9bac16a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.399+0000 7f9bb184d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac09db70 con 0x7f9bac15ce40 2024-01-27T01:03:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.399+0000 7f9bb184d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac0c7ba0 con 0x7f9bac15d210 2024-01-27T01:03:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.399+0000 7f9bb184d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac0c7e80 con 0x7f9bac0cc1c0 2024-01-27T01:03:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.400+0000 7f9baa7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bac15ce40 0x7f9bac16a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.400+0000 7f9baa7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bac15ce40 0x7f9bac16a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38926/0 (socket says 172.21.15.46:38926) 2024-01-27T01:03:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.400+0000 7f9baa7fc700 1 -- 172.21.15.46:0/1457117575 learned_addr learned my addr 172.21.15.46:0/1457117575 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.400+0000 7f9baaffd700 1 --2- 172.21.15.46:0/1457117575 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bac0cc1c0 0x7f9bac0c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.400+0000 7f9bab7fe700 1 --2- 172.21.15.46:0/1457117575 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bac15d210 0x7f9bac16a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.400+0000 7f9baa7fc700 1 -- 172.21.15.46:0/1457117575 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bac0cc1c0 msgr2=0x7f9bac0c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.400+0000 7f9baa7fc700 1 --2- 172.21.15.46:0/1457117575 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bac0cc1c0 0x7f9bac0c9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.400+0000 7f9baa7fc700 1 -- 172.21.15.46:0/1457117575 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bac15d210 msgr2=0x7f9bac16a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.400+0000 7f9baa7fc700 1 --2- 172.21.15.46:0/1457117575 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bac15d210 0x7f9bac16a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.400+0000 7f9baa7fc700 1 -- 172.21.15.46:0/1457117575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c009420 con 0x7f9bac15ce40 2024-01-27T01:03:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.400+0000 7f9bab7fe700 1 --2- 172.21.15.46:0/1457117575 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bac15d210 0x7f9bac16a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:03:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.400+0000 7f9baa7fc700 1 --2- 172.21.15.46:0/1457117575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bac15ce40 0x7f9bac16a130 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400c580 tx=0x7f9b94007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.400+0000 7f9ba2ffd700 1 -- 172.21.15.46:0/1457117575 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94005b30 con 0x7f9bac15ce40 2024-01-27T01:03:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.400+0000 7f9ba2ffd700 1 -- 172.21.15.46:0/1457117575 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b94005c90 con 0x7f9bac15ce40 2024-01-27T01:03:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.400+0000 7f9bb184d700 1 -- 172.21.15.46:0/1457117575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bac0c8160 con 0x7f9bac15ce40 2024-01-27T01:03:42.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.400+0000 7f9ba2ffd700 1 -- 172.21.15.46:0/1457117575 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b940072e0 con 0x7f9bac15ce40 2024-01-27T01:03:42.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.400+0000 7f9bb184d700 1 -- 172.21.15.46:0/1457117575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bac0c8720 con 0x7f9bac15ce40 2024-01-27T01:03:42.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.402+0000 7f9bb184d700 1 -- 172.21.15.46:0/1457117575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bac09b270 con 0x7f9bac15ce40 2024-01-27T01:03:42.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.402+0000 7f9ba2ffd700 1 -- 172.21.15.46:0/1457117575 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9b9401b3b0 con 0x7f9bac15ce40 2024-01-27T01:03:42.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.402+0000 7f9ba2ffd700 1 --2- 172.21.15.46:0/1457117575 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b8c066f30 0x7f9b8c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:42.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.402+0000 7f9ba2ffd700 1 -- 172.21.15.46:0/1457117575 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5637+0+0 (secure 0 0 0) 0x7f9b940838a0 con 0x7f9bac15ce40 2024-01-27T01:03:42.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.402+0000 7f9baaffd700 1 --2- 172.21.15.46:0/1457117575 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b8c066f30 0x7f9b8c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:42.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.403+0000 7f9baaffd700 1 --2- 172.21.15.46:0/1457117575 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b8c066f30 0x7f9b8c0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0091e0 tx=0x7f9b9c017000 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:42.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.407+0000 7f9ba2ffd700 1 -- 172.21.15.46:0/1457117575 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9b9404b5a0 con 0x7f9bac15ce40 2024-01-27T01:03:42.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.611+0000 7f9bb184d700 1 -- 172.21.15.46:0/1457117575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9bac04c710 con 0x7f9bac15ce40 2024-01-27T01:03:42.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.691+0000 7f9ba2ffd700 1 -- 172.21.15.46:0/1457117575 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v202) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9b94017020 con 0x7f9bac15ce40 2024-01-27T01:03:42.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.695+0000 7f9bb184d700 1 -- 172.21.15.46:0/1457117575 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b8c066f30 msgr2=0x7f9b8c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:42.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.695+0000 7f9bb184d700 1 --2- 172.21.15.46:0/1457117575 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b8c066f30 0x7f9b8c0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0091e0 tx=0x7f9b9c017000 comp rx=0 tx=0).stop 2024-01-27T01:03:42.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.695+0000 7f9bb184d700 1 -- 172.21.15.46:0/1457117575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bac15ce40 msgr2=0x7f9bac16a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:42.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.695+0000 7f9bb184d700 1 --2- 172.21.15.46:0/1457117575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bac15ce40 0x7f9bac16a130 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400c580 tx=0x7f9b94007e40 comp rx=0 tx=0).stop 2024-01-27T01:03:42.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.696+0000 7f9bb184d700 1 -- 172.21.15.46:0/1457117575 shutdown_connections 2024-01-27T01:03:42.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.696+0000 7f9bb184d700 1 --2- 172.21.15.46:0/1457117575 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b8c066f30 0x7f9b8c0693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.696+0000 7f9bb184d700 1 --2- 172.21.15.46:0/1457117575 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bac15d210 0x7f9bac16a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.696+0000 7f9bb184d700 1 --2- 172.21.15.46:0/1457117575 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bac0cc1c0 0x7f9bac0c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.696+0000 7f9bb184d700 1 --2- 172.21.15.46:0/1457117575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bac15ce40 0x7f9bac16a130 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.696+0000 7f9bb184d700 1 -- 172.21.15.46:0/1457117575 >> 172.21.15.46:0/1457117575 conn(0x7f9bac06f840 msgr2=0x7f9bac15c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:42.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.697+0000 7f9bb184d700 1 -- 172.21.15.46:0/1457117575 shutdown_connections 2024-01-27T01:03:42.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.697+0000 7f9bb184d700 1 -- 172.21.15.46:0/1457117575 wait complete. 2024-01-27T01:03:42.698 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-01-27T01:03:42.712 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-27T01:03:42.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.852+0000 7fac0aae6700 1 Processor -- start 2024-01-27T01:03:42.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.852+0000 7fac0aae6700 1 -- start start 2024-01-27T01:03:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.853+0000 7fac0aae6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fac04157860 0x7fac040cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.853+0000 7fac0aae6700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fac0406ddb0 0x7fac040cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.853+0000 7fac0aae6700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac040cea80 con 0x7fac04157490 2024-01-27T01:03:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.853+0000 7fac0aae6700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac040cdc60 con 0x7fac04157860 2024-01-27T01:03:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.853+0000 7fac0aae6700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac040b2ae0 con 0x7fac0406ddb0 2024-01-27T01:03:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.853+0000 7fac03fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fac04157860 0x7fac040cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.853+0000 7fac03fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fac04157860 0x7fac040cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38942/0 (socket says 172.21.15.46:38942) 2024-01-27T01:03:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.853+0000 7fac03fff700 1 -- 172.21.15.46:0/2058168329 learned_addr learned my addr 172.21.15.46:0/2058168329 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.853+0000 7fac09083700 1 --2- 172.21.15.46:0/2058168329 >> v2:172.21.15.106:3300/0 conn(0x7fac0406ddb0 0x7fac040cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.853+0000 7fac03fff700 1 -- 172.21.15.46:0/2058168329 >> v2:172.21.15.106:3300/0 conn(0x7fac0406ddb0 msgr2=0x7fac040cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.853+0000 7fac03fff700 1 --2- 172.21.15.46:0/2058168329 >> v2:172.21.15.106:3300/0 conn(0x7fac0406ddb0 0x7fac040cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.853+0000 7fac03fff700 1 -- 172.21.15.46:0/2058168329 >> v1:172.21.15.106:6789/0 conn(0x7fac04157490 legacy=0x7fac0406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:42.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.853+0000 7fac03fff700 1 -- 172.21.15.46:0/2058168329 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac040cde70 con 0x7fac04157860 2024-01-27T01:03:42.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.854+0000 7fac03fff700 1 --2- 172.21.15.46:0/2058168329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac04157860 0x7fac040cc600 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fac04068680 tx=0x7fabf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934d03207cdeeb88 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:42.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.854+0000 7fac037fe700 1 -- 172.21.15.46:0/2058168329 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf000b860 con 0x7fac04157860 2024-01-27T01:03:42.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.854+0000 7fac037fe700 1 -- 172.21.15.46:0/2058168329 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf000b9c0 con 0x7fac04157860 2024-01-27T01:03:42.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.854+0000 7fac037fe700 1 -- 172.21.15.46:0/2058168329 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf000bd60 con 0x7fac04157860 2024-01-27T01:03:42.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.854+0000 7fac0aae6700 1 -- 172.21.15.46:0/2058168329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac04157860 msgr2=0x7fac040cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:42.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.854+0000 7fac0aae6700 1 --2- 172.21.15.46:0/2058168329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac04157860 0x7fac040cc600 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fac04068680 tx=0x7fabf0009250 comp rx=0 tx=0).stop 2024-01-27T01:03:42.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.854+0000 7fac0aae6700 1 -- 172.21.15.46:0/2058168329 shutdown_connections 2024-01-27T01:03:42.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.854+0000 7fac0aae6700 1 --2- 172.21.15.46:0/2058168329 >> v2:172.21.15.106:3300/0 conn(0x7fac0406ddb0 0x7fac040cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.854+0000 7fac0aae6700 1 --2- 172.21.15.46:0/2058168329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac04157860 0x7fac040cc600 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.854+0000 7fac0aae6700 1 -- 172.21.15.46:0/2058168329 >> 172.21.15.46:0/2058168329 conn(0x7fac04067930 msgr2=0x7fac04067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:42.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.855+0000 7fac0aae6700 1 -- 172.21.15.46:0/2058168329 shutdown_connections 2024-01-27T01:03:42.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.855+0000 7fac0aae6700 1 -- 172.21.15.46:0/2058168329 wait complete. 2024-01-27T01:03:42.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.856+0000 7fac0aae6700 1 Processor -- start 2024-01-27T01:03:42.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.856+0000 7fac0aae6700 1 -- start start 2024-01-27T01:03:42.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.857+0000 7fac0aae6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac0406ddb0 0x7fac0414f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:42.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.857+0000 7fac0aae6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac04157490 0x7fac040ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:42.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.857+0000 7fac0aae6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac04157860 0x7fac040aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:42.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.857+0000 7fac0aae6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac041514b0 con 0x7fac04157860 2024-01-27T01:03:42.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.857+0000 7fac0aae6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac04151790 con 0x7fac0406ddb0 2024-01-27T01:03:42.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.857+0000 7fac0aae6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac04151a70 con 0x7fac04157490 2024-01-27T01:03:42.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.857+0000 7fac09083700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac04157860 0x7fac040aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:42.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.857+0000 7fac09083700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac04157860 0x7fac040aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38950/0 (socket says 172.21.15.46:38950) 2024-01-27T01:03:42.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.857+0000 7fac09083700 1 -- 172.21.15.46:0/2095182704 learned_addr learned my addr 172.21.15.46:0/2095182704 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:42.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.857+0000 7fac03fff700 1 --2- 172.21.15.46:0/2095182704 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac04157490 0x7fac040ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:42.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.857+0000 7fac08882700 1 --2- 172.21.15.46:0/2095182704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac0406ddb0 0x7fac0414f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:42.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.857+0000 7fac09083700 1 -- 172.21.15.46:0/2095182704 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac04157490 msgr2=0x7fac040ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:42.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.857+0000 7fac09083700 1 --2- 172.21.15.46:0/2095182704 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac04157490 0x7fac040ae640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.857+0000 7fac09083700 1 -- 172.21.15.46:0/2095182704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac0406ddb0 msgr2=0x7fac0414f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:42.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.857+0000 7fac09083700 1 --2- 172.21.15.46:0/2095182704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac0406ddb0 0x7fac0414f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:42.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.857+0000 7fac09083700 1 -- 172.21.15.46:0/2095182704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf00094f0 con 0x7fac04157860 2024-01-27T01:03:42.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.858+0000 7fac09083700 1 --2- 172.21.15.46:0/2095182704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac04157860 0x7fac040aeb80 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fabfc007f30 tx=0x7fabfc0098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:42.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.858+0000 7fac00ff9700 1 -- 172.21.15.46:0/2095182704 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc00eb90 con 0x7fac04157860 2024-01-27T01:03:42.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.858+0000 7fac00ff9700 1 -- 172.21.15.46:0/2095182704 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabfc00ecf0 con 0x7fac04157860 2024-01-27T01:03:42.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.858+0000 7fac0aae6700 1 -- 172.21.15.46:0/2095182704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac04151d50 con 0x7fac04157860 2024-01-27T01:03:42.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.858+0000 7fac00ff9700 1 -- 172.21.15.46:0/2095182704 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc014670 con 0x7fac04157860 2024-01-27T01:03:42.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.858+0000 7fac0aae6700 1 -- 172.21.15.46:0/2095182704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0414f830 con 0x7fac04157860 2024-01-27T01:03:42.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.859+0000 7fac0aae6700 1 -- 172.21.15.46:0/2095182704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac041521f0 con 0x7fac04157860 2024-01-27T01:03:42.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.860+0000 7fac00ff9700 1 -- 172.21.15.46:0/2095182704 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fabfc014a20 con 0x7fac04157860 2024-01-27T01:03:42.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.860+0000 7fac00ff9700 1 --2- 172.21.15.46:0/2095182704 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fabe8066f30 0x7fabe80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:42.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.860+0000 7fac00ff9700 1 -- 172.21.15.46:0/2095182704 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6007+0+0 (secure 0 0 0) 0x7fabfc085780 con 0x7fac04157860 2024-01-27T01:03:42.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.860+0000 7fac08882700 1 --2- 172.21.15.46:0/2095182704 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fabe8066f30 0x7fabe80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:42.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.861+0000 7fac08882700 1 --2- 172.21.15.46:0/2095182704 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fabe8066f30 0x7fabe80693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fabf8003a80 tx=0x7fabf8007ec0 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:42.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:42.864+0000 7fac00ff9700 1 -- 172.21.15.46:0/2095182704 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fabfc052940 con 0x7fac04157860 2024-01-27T01:03:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.069+0000 7fac0aae6700 1 -- 172.21.15.46:0/2095182704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fac0404c710 con 0x7fac04157860 2024-01-27T01:03:43.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.698+0000 7fac00ff9700 1 -- 172.21.15.46:0/2095182704 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v203) v1 ==== 170+0+0 (secure 0 0 0) 0x7fabfc00ee60 con 0x7fac04157860 2024-01-27T01:03:43.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.704+0000 7fac0aae6700 1 -- 172.21.15.46:0/2095182704 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fabe8066f30 msgr2=0x7fabe80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:43.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.704+0000 7fac0aae6700 1 --2- 172.21.15.46:0/2095182704 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fabe8066f30 0x7fabe80693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fabf8003a80 tx=0x7fabf8007ec0 comp rx=0 tx=0).stop 2024-01-27T01:03:43.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.704+0000 7fac0aae6700 1 -- 172.21.15.46:0/2095182704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac04157860 msgr2=0x7fac040aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:43.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.704+0000 7fac0aae6700 1 --2- 172.21.15.46:0/2095182704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac04157860 0x7fac040aeb80 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fabfc007f30 tx=0x7fabfc0098e0 comp rx=0 tx=0).stop 2024-01-27T01:03:43.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.705+0000 7fac0aae6700 1 -- 172.21.15.46:0/2095182704 shutdown_connections 2024-01-27T01:03:43.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.705+0000 7fac0aae6700 1 --2- 172.21.15.46:0/2095182704 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fabe8066f30 0x7fabe80693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:43.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.705+0000 7fac0aae6700 1 --2- 172.21.15.46:0/2095182704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac04157860 0x7fac040aeb80 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:43.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.705+0000 7fac0aae6700 1 --2- 172.21.15.46:0/2095182704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac0406ddb0 0x7fac0414f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:43.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.705+0000 7fac0aae6700 1 --2- 172.21.15.46:0/2095182704 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac04157490 0x7fac040ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:43.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.705+0000 7fac0aae6700 1 -- 172.21.15.46:0/2095182704 >> 172.21.15.46:0/2095182704 conn(0x7fac04067930 msgr2=0x7fac040b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:43.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.705+0000 7fac0aae6700 1 -- 172.21.15.46:0/2095182704 shutdown_connections 2024-01-27T01:03:43.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.707+0000 7fac0aae6700 1 -- 172.21.15.46:0/2095182704 wait complete. 2024-01-27T01:03:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-01-27T01:03:43.735 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-27T01:03:43.816 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:03:43.818 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:03:43.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.876+0000 7f1f23baa700 1 Processor -- start 2024-01-27T01:03:43.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.876+0000 7f1f23baa700 1 -- start start 2024-01-27T01:03:43.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.877+0000 7f1f23baa700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1f1c157490 0x7f1f1c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:43.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.877+0000 7f1f23baa700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1f1c157860 0x7f1f1c0cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:43.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.877+0000 7f1f23baa700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f1c06d6a0 0x7f1f1c06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:43.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.877+0000 7f1f23baa700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f1c0cdc60 con 0x7f1f1c157490 2024-01-27T01:03:43.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.877+0000 7f1f23baa700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f1c06e040 con 0x7f1f1c157860 2024-01-27T01:03:43.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.877+0000 7f1f23baa700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c0cea80 con 0x7f1f1c06d6a0 2024-01-27T01:03:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.877+0000 7f1f21946700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1f1c157490 0x7f1f1c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.877+0000 7f1f21946700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1f1c157490 0x7f1f1c0cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38966/0 (socket says 172.21.15.46:38966) 2024-01-27T01:03:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.877+0000 7f1f21946700 1 -- 172.21.15.46:0/371256638 learned_addr learned my addr 172.21.15.46:0/371256638 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.877+0000 7f1f22147700 1 --2- 172.21.15.46:0/371256638 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f1c06d6a0 0x7f1f1c06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.877+0000 7f1f21145700 1 --2- 172.21.15.46:0/371256638 >> v2:172.21.15.106:3300/0 conn(0x7f1f1c157860 0x7f1f1c0cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.877+0000 7f1f21946700 1 -- 172.21.15.46:0/371256638 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f1c06d6a0 msgr2=0x7f1f1c06db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.877+0000 7f1f21946700 1 --2- 172.21.15.46:0/371256638 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f1c06d6a0 0x7f1f1c06db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.877+0000 7f1f21946700 1 -- 172.21.15.46:0/371256638 >> v2:172.21.15.106:3300/0 conn(0x7f1f1c157860 msgr2=0x7f1f1c0cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.877+0000 7f1f21946700 1 --2- 172.21.15.46:0/371256638 >> v2:172.21.15.106:3300/0 conn(0x7f1f1c157860 0x7f1f1c0cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.877+0000 7f1f21946700 1 -- 172.21.15.46:0/371256638 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c0cde70 con 0x7f1f1c157490 2024-01-27T01:03:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.878+0000 7f1f21946700 1 --2- 172.21.15.46:0/371256638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f1c157490 0x7f1f1c0cc600 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c06f9a0 tx=0x7f1f18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6fceeec5362dec1 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.878+0000 7f1f20944700 1 -- 172.21.15.46:0/371256638 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1800b640 con 0x7f1f1c157490 2024-01-27T01:03:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.878+0000 7f1f20944700 1 -- 172.21.15.46:0/371256638 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1800b7a0 con 0x7f1f1c157490 2024-01-27T01:03:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.878+0000 7f1f20944700 1 -- 172.21.15.46:0/371256638 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1800bb60 con 0x7f1f1c157490 2024-01-27T01:03:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.878+0000 7f1f23baa700 1 -- 172.21.15.46:0/371256638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f1c157490 msgr2=0x7f1f1c0cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.878+0000 7f1f23baa700 1 --2- 172.21.15.46:0/371256638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f1c157490 0x7f1f1c0cc600 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c06f9a0 tx=0x7f1f18009250 comp rx=0 tx=0).stop 2024-01-27T01:03:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.879+0000 7f1f23baa700 1 -- 172.21.15.46:0/371256638 shutdown_connections 2024-01-27T01:03:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.879+0000 7f1f23baa700 1 --2- 172.21.15.46:0/371256638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f1c157490 0x7f1f1c0cc600 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.879+0000 7f1f23baa700 1 --2- 172.21.15.46:0/371256638 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f1c06d6a0 0x7f1f1c06db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.879+0000 7f1f23baa700 1 --2- 172.21.15.46:0/371256638 >> v2:172.21.15.106:3300/0 conn(0x7f1f1c157860 0x7f1f1c0cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.879+0000 7f1f23baa700 1 -- 172.21.15.46:0/371256638 >> 172.21.15.46:0/371256638 conn(0x7f1f1c067930 msgr2=0x7f1f1c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.879+0000 7f1f23baa700 1 -- 172.21.15.46:0/371256638 shutdown_connections 2024-01-27T01:03:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.879+0000 7f1f23baa700 1 -- 172.21.15.46:0/371256638 wait complete. 2024-01-27T01:03:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.880+0000 7f1f23baa700 1 Processor -- start 2024-01-27T01:03:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.880+0000 7f1f23baa700 1 -- start start 2024-01-27T01:03:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f23baa700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f1c06d6a0 0x7f1f1c150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f23baa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f1c157490 0x7f1f1c15a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f23baa700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f1c157860 0x7f1f1c157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f23baa700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c06a3d0 con 0x7f1f1c157490 2024-01-27T01:03:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f23baa700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c0697c0 con 0x7f1f1c157860 2024-01-27T01:03:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f23baa700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c069aa0 con 0x7f1f1c06d6a0 2024-01-27T01:03:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f21145700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f1c157490 0x7f1f1c15a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f21145700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f1c157490 0x7f1f1c15a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38978/0 (socket says 172.21.15.46:38978) 2024-01-27T01:03:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f21145700 1 -- 172.21.15.46:0/84617056 learned_addr learned my addr 172.21.15.46:0/84617056 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f22147700 1 --2- 172.21.15.46:0/84617056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f1c157860 0x7f1f1c157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f21946700 1 --2- 172.21.15.46:0/84617056 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f1c06d6a0 0x7f1f1c150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f21145700 1 -- 172.21.15.46:0/84617056 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f1c06d6a0 msgr2=0x7f1f1c150f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f21145700 1 --2- 172.21.15.46:0/84617056 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f1c06d6a0 0x7f1f1c150f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f21145700 1 -- 172.21.15.46:0/84617056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f1c157860 msgr2=0x7f1f1c157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f21145700 1 --2- 172.21.15.46:0/84617056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f1c157860 0x7f1f1c157c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f21145700 1 -- 172.21.15.46:0/84617056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f180094f0 con 0x7f1f1c157490 2024-01-27T01:03:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f22147700 1 --2- 172.21.15.46:0/84617056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f1c157860 0x7f1f1c157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:03:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f21145700 1 --2- 172.21.15.46:0/84617056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f1c157490 0x7f1f1c15a250 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c00e930 tx=0x7f1f0c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.881+0000 7f1f11ffb700 1 -- 172.21.15.46:0/84617056 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c00eed0 con 0x7f1f1c157490 2024-01-27T01:03:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.882+0000 7f1f11ffb700 1 -- 172.21.15.46:0/84617056 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0c00c850 con 0x7f1f1c157490 2024-01-27T01:03:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.882+0000 7f1f11ffb700 1 -- 172.21.15.46:0/84617056 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c00cbf0 con 0x7f1f1c157490 2024-01-27T01:03:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.882+0000 7f1f23baa700 1 -- 172.21.15.46:0/84617056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1c069d80 con 0x7f1f1c157490 2024-01-27T01:03:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.882+0000 7f1f23baa700 1 -- 172.21.15.46:0/84617056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1c069400 con 0x7f1f1c157490 2024-01-27T01:03:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.883+0000 7f1f23baa700 1 -- 172.21.15.46:0/84617056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1c04c710 con 0x7f1f1c157490 2024-01-27T01:03:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.883+0000 7f1f11ffb700 1 -- 172.21.15.46:0/84617056 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1f0c013470 con 0x7f1f1c157490 2024-01-27T01:03:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.884+0000 7f1f11ffb700 1 --2- 172.21.15.46:0/84617056 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1f00066f30 0x7f1f000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.884+0000 7f1f11ffb700 1 -- 172.21.15.46:0/84617056 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f1f0c0851f0 con 0x7f1f1c157490 2024-01-27T01:03:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.884+0000 7f1f21946700 1 --2- 172.21.15.46:0/84617056 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1f00066f30 0x7f1f000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.885+0000 7f1f21946700 1 --2- 172.21.15.46:0/84617056 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1f00066f30 0x7f1f000693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c06f9a0 tx=0x7f1f18017000 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:43.888+0000 7f1f11ffb700 1 -- 172.21.15.46:0/84617056 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1f0c052190 con 0x7f1f1c157490 2024-01-27T01:03:44.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.138+0000 7f1f23baa700 1 -- 172.21.15.46:0/84617056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1f1c0b14d0 con 0x7f1f1c157490 2024-01-27T01:03:44.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.709+0000 7f1f11ffb700 1 -- 172.21.15.46:0/84617056 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 25 and data pool 26 v103) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1f0c0557b0 con 0x7f1f1c157490 2024-01-27T01:03:44.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.727+0000 7f1f23baa700 1 -- 172.21.15.46:0/84617056 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1f00066f30 msgr2=0x7f1f000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:44.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.727+0000 7f1f23baa700 1 --2- 172.21.15.46:0/84617056 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1f00066f30 0x7f1f000693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c06f9a0 tx=0x7f1f18017000 comp rx=0 tx=0).stop 2024-01-27T01:03:44.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.727+0000 7f1f23baa700 1 -- 172.21.15.46:0/84617056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f1c157490 msgr2=0x7f1f1c15a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:44.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.727+0000 7f1f23baa700 1 --2- 172.21.15.46:0/84617056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f1c157490 0x7f1f1c15a250 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c00e930 tx=0x7f1f0c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:44.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.729+0000 7f1f23baa700 1 -- 172.21.15.46:0/84617056 shutdown_connections 2024-01-27T01:03:44.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.729+0000 7f1f23baa700 1 --2- 172.21.15.46:0/84617056 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1f00066f30 0x7f1f000693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:44.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.729+0000 7f1f23baa700 1 --2- 172.21.15.46:0/84617056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f1c157860 0x7f1f1c157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:44.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.729+0000 7f1f23baa700 1 --2- 172.21.15.46:0/84617056 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f1c06d6a0 0x7f1f1c150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:44.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.729+0000 7f1f23baa700 1 --2- 172.21.15.46:0/84617056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f1c157490 0x7f1f1c15a250 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:44.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.729+0000 7f1f23baa700 1 -- 172.21.15.46:0/84617056 >> 172.21.15.46:0/84617056 conn(0x7f1f1c067930 msgr2=0x7f1f1c152e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:44.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.730+0000 7f1f23baa700 1 -- 172.21.15.46:0/84617056 shutdown_connections 2024-01-27T01:03:44.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.731+0000 7f1f23baa700 1 -- 172.21.15.46:0/84617056 wait complete. 2024-01-27T01:03:44.735 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 25 and data pool 26 2024-01-27T01:03:44.748 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:03:44.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.925+0000 7fcb1b6bc700 1 Processor -- start 2024-01-27T01:03:44.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.925+0000 7fcb1b6bc700 1 -- start start 2024-01-27T01:03:44.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.926+0000 7fcb1b6bc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fcb140c16e0 0x7fcb1406bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:44.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.926+0000 7fcb1b6bc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb14161eb0 0x7fcb1406c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:44.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.926+0000 7fcb1b6bc700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb14162700 con 0x7fcb140c1ab0 2024-01-27T01:03:44.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.926+0000 7fcb1b6bc700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb140b0e40 con 0x7fcb140c16e0 2024-01-27T01:03:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.926+0000 7fcb1b6bc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb140691c0 con 0x7fcb14161eb0 2024-01-27T01:03:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.926+0000 7fcb19458700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fcb140c16e0 0x7fcb1406bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.926+0000 7fcb19c59700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb14161eb0 0x7fcb1406c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.926+0000 7fcb19458700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fcb140c16e0 0x7fcb1406bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:37014/0 (socket says 172.21.15.46:37014) 2024-01-27T01:03:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.926+0000 7fcb19c59700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb14161eb0 0x7fcb1406c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:39986/0 (socket says 172.21.15.46:39986) 2024-01-27T01:03:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.926+0000 7fcb19458700 1 -- 172.21.15.46:0/1627338524 learned_addr learned my addr 172.21.15.46:0/1627338524 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.926+0000 7fcb18c57700 1 --1- 172.21.15.46:0/1627338524 >> v1:172.21.15.106:6789/0 conn(0x7fcb140c1ab0 0x7fcb141617a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:52356/0 (socket says 172.21.15.46:52356) 2024-01-27T01:03:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.927+0000 7fcb0bfff700 1 -- 172.21.15.46:0/1627338524 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213946415 0 0) 0x7fcb14162700 con 0x7fcb140c1ab0 2024-01-27T01:03:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.927+0000 7fcb19c59700 1 -- 172.21.15.46:0/1627338524 >> v2:172.21.15.106:3300/0 conn(0x7fcb140c16e0 msgr2=0x7fcb1406bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.927+0000 7fcb19c59700 1 --2- 172.21.15.46:0/1627338524 >> v2:172.21.15.106:3300/0 conn(0x7fcb140c16e0 0x7fcb1406bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.927+0000 7fcb19c59700 1 -- 172.21.15.46:0/1627338524 >> v1:172.21.15.106:6789/0 conn(0x7fcb140c1ab0 legacy=0x7fcb141617a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.927+0000 7fcb19c59700 1 -- 172.21.15.46:0/1627338524 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb14162420 con 0x7fcb14161eb0 2024-01-27T01:03:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.927+0000 7fcb19c59700 1 --2- 172.21.15.46:0/1627338524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb14161eb0 0x7fcb1406c1f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c009e90 tx=0x7fcb0c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d760043563ec0560 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:44.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.927+0000 7fcb0bfff700 1 -- 172.21.15.46:0/1627338524 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c00e070 con 0x7fcb14161eb0 2024-01-27T01:03:44.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.927+0000 7fcb0bfff700 1 -- 172.21.15.46:0/1627338524 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0c00bbf0 con 0x7fcb14161eb0 2024-01-27T01:03:44.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.927+0000 7fcb0bfff700 1 -- 172.21.15.46:0/1627338524 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c012430 con 0x7fcb14161eb0 2024-01-27T01:03:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.928+0000 7fcb1b6bc700 1 -- 172.21.15.46:0/1627338524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb14161eb0 msgr2=0x7fcb1406c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.928+0000 7fcb1b6bc700 1 --2- 172.21.15.46:0/1627338524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb14161eb0 0x7fcb1406c1f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c009e90 tx=0x7fcb0c009780 comp rx=0 tx=0).stop 2024-01-27T01:03:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.928+0000 7fcb1b6bc700 1 -- 172.21.15.46:0/1627338524 shutdown_connections 2024-01-27T01:03:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.928+0000 7fcb1b6bc700 1 --2- 172.21.15.46:0/1627338524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb14161eb0 0x7fcb1406c1f0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.928+0000 7fcb1b6bc700 1 --2- 172.21.15.46:0/1627338524 >> v2:172.21.15.106:3300/0 conn(0x7fcb140c16e0 0x7fcb1406bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.928+0000 7fcb1b6bc700 1 -- 172.21.15.46:0/1627338524 >> 172.21.15.46:0/1627338524 conn(0x7fcb140ce530 msgr2=0x7fcb140ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.928+0000 7fcb1b6bc700 1 -- 172.21.15.46:0/1627338524 shutdown_connections 2024-01-27T01:03:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.928+0000 7fcb1b6bc700 1 -- 172.21.15.46:0/1627338524 wait complete. 2024-01-27T01:03:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.929+0000 7fcb1b6bc700 1 Processor -- start 2024-01-27T01:03:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.929+0000 7fcb1b6bc700 1 -- start start 2024-01-27T01:03:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.930+0000 7fcb1b6bc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb140c16e0 0x7fcb1416ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.930+0000 7fcb1b6bc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb140c1ab0 0x7fcb14170a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.930+0000 7fcb1b6bc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb14161eb0 0x7fcb140c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.930+0000 7fcb1b6bc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1414e290 con 0x7fcb14161eb0 2024-01-27T01:03:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.930+0000 7fcb1b6bc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb140c7660 con 0x7fcb140c1ab0 2024-01-27T01:03:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.930+0000 7fcb1b6bc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb140c51e0 con 0x7fcb140c16e0 2024-01-27T01:03:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.930+0000 7fcb19c59700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb14161eb0 0x7fcb140c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.930+0000 7fcb19c59700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb14161eb0 0x7fcb140c4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38986/0 (socket says 172.21.15.46:38986) 2024-01-27T01:03:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.930+0000 7fcb19c59700 1 -- 172.21.15.46:0/983918079 learned_addr learned my addr 172.21.15.46:0/983918079 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.930+0000 7fcb19458700 1 --2- 172.21.15.46:0/983918079 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb140c16e0 0x7fcb1416ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.930+0000 7fcb18c57700 1 --2- 172.21.15.46:0/983918079 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb140c1ab0 0x7fcb14170a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.931+0000 7fcb19c59700 1 -- 172.21.15.46:0/983918079 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb140c16e0 msgr2=0x7fcb1416ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.931+0000 7fcb19c59700 1 --2- 172.21.15.46:0/983918079 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb140c16e0 0x7fcb1416ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.931+0000 7fcb19c59700 1 -- 172.21.15.46:0/983918079 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb140c1ab0 msgr2=0x7fcb14170a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.931+0000 7fcb19c59700 1 --2- 172.21.15.46:0/983918079 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb140c1ab0 0x7fcb14170a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.931+0000 7fcb19c59700 1 -- 172.21.15.46:0/983918079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c009a20 con 0x7fcb14161eb0 2024-01-27T01:03:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.931+0000 7fcb19c59700 1 --2- 172.21.15.46:0/983918079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb14161eb0 0x7fcb140c4af0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c00b6c0 tx=0x7fcb0c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.931+0000 7fcb097fa700 1 -- 172.21.15.46:0/983918079 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c025440 con 0x7fcb14161eb0 2024-01-27T01:03:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.931+0000 7fcb1b6bc700 1 -- 172.21.15.46:0/983918079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb140c6d30 con 0x7fcb14161eb0 2024-01-27T01:03:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.931+0000 7fcb097fa700 1 -- 172.21.15.46:0/983918079 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0c0255c0 con 0x7fcb14161eb0 2024-01-27T01:03:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.931+0000 7fcb097fa700 1 -- 172.21.15.46:0/983918079 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c025980 con 0x7fcb14161eb0 2024-01-27T01:03:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.931+0000 7fcb1b6bc700 1 -- 172.21.15.46:0/983918079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb140c72f0 con 0x7fcb14161eb0 2024-01-27T01:03:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.932+0000 7fcb1b6bc700 1 -- 172.21.15.46:0/983918079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1404c710 con 0x7fcb14161eb0 2024-01-27T01:03:44.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.933+0000 7fcb097fa700 1 -- 172.21.15.46:0/983918079 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fcb0c003050 con 0x7fcb14161eb0 2024-01-27T01:03:44.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.933+0000 7fcb097fa700 1 --2- 172.21.15.46:0/983918079 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcaf8066f30 0x7fcaf80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:44.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.933+0000 7fcb097fa700 1 -- 172.21.15.46:0/983918079 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fcb0c086cf0 con 0x7fcb14161eb0 2024-01-27T01:03:44.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.933+0000 7fcb19458700 1 --2- 172.21.15.46:0/983918079 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcaf8066f30 0x7fcaf80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:44.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.934+0000 7fcb19458700 1 --2- 172.21.15.46:0/983918079 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcaf8066f30 0x7fcaf80693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcb10006fd0 tx=0x7fcb10008040 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:44.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:44.938+0000 7fcb097fa700 1 -- 172.21.15.46:0/983918079 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcb0c08b070 con 0x7fcb14161eb0 2024-01-27T01:03:45.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.141+0000 7fcb1b6bc700 1 -- 172.21.15.46:0/983918079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb14068b70 con 0x7fcb14161eb0 2024-01-27T01:03:45.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.142+0000 7fcb097fa700 1 -- 172.21.15.46:0/983918079 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v204) v1 ==== 74+0+16293 (secure 0 0 0) 0x7fcb0c053c50 con 0x7fcb14161eb0 2024-01-27T01:03:45.142 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:45.142 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":204,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:03:44.140802+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:03:42.648202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-01-27T01:03:43.106225+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:03:45.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.144+0000 7fcb1b6bc700 1 -- 172.21.15.46:0/983918079 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcaf8066f30 msgr2=0x7fcaf80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:45.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.144+0000 7fcb1b6bc700 1 --2- 172.21.15.46:0/983918079 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcaf8066f30 0x7fcaf80693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcb10006fd0 tx=0x7fcb10008040 comp rx=0 tx=0).stop 2024-01-27T01:03:45.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.144+0000 7fcb1b6bc700 1 -- 172.21.15.46:0/983918079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb14161eb0 msgr2=0x7fcb140c4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:45.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.144+0000 7fcb1b6bc700 1 --2- 172.21.15.46:0/983918079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb14161eb0 0x7fcb140c4af0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c00b6c0 tx=0x7fcb0c019000 comp rx=0 tx=0).stop 2024-01-27T01:03:45.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.145+0000 7fcb1b6bc700 1 -- 172.21.15.46:0/983918079 shutdown_connections 2024-01-27T01:03:45.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.145+0000 7fcb1b6bc700 1 --2- 172.21.15.46:0/983918079 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcaf8066f30 0x7fcaf80693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:45.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.145+0000 7fcb1b6bc700 1 --2- 172.21.15.46:0/983918079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb14161eb0 0x7fcb140c4af0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:45.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.145+0000 7fcb1b6bc700 1 --2- 172.21.15.46:0/983918079 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb140c16e0 0x7fcb1416ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:45.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.145+0000 7fcb1b6bc700 1 --2- 172.21.15.46:0/983918079 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb140c1ab0 0x7fcb14170a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:45.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.145+0000 7fcb1b6bc700 1 -- 172.21.15.46:0/983918079 >> 172.21.15.46:0/983918079 conn(0x7fcb140ce530 msgr2=0x7fcb140cd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:45.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.145+0000 7fcb1b6bc700 1 -- 172.21.15.46:0/983918079 shutdown_connections 2024-01-27T01:03:45.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.145+0000 7fcb1b6bc700 1 -- 172.21.15.46:0/983918079 wait complete. 2024-01-27T01:03:45.159 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:03:45.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.290+0000 7f26cdc9d700 1 Processor -- start 2024-01-27T01:03:45.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.290+0000 7f26cdc9d700 1 -- start start 2024-01-27T01:03:45.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.291+0000 7f26cdc9d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f26c80c3cf0 0x7f26c80ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:45.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.291+0000 7f26cdc9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f26c80cd080 0x7f26c80ab1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:45.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.291+0000 7f26cdc9d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c80cbf10 con 0x7f26c80acf30 2024-01-27T01:03:45.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.291+0000 7f26cdc9d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26c80ada10 con 0x7f26c80cd080 2024-01-27T01:03:45.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.291+0000 7f26cdc9d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26c8157be0 con 0x7f26c80c3cf0 2024-01-27T01:03:45.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.291+0000 7f26c7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f26c80cd080 0x7f26c80ab1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:45.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.291+0000 7f26c7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f26c80cd080 0x7f26c80ab1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38992/0 (socket says 172.21.15.46:38992) 2024-01-27T01:03:45.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.291+0000 7f26c7fff700 1 -- 172.21.15.46:0/1706671619 learned_addr learned my addr 172.21.15.46:0/1706671619 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:45.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.291+0000 7f26c6ffd700 1 --2- 172.21.15.46:0/1706671619 >> v2:172.21.15.106:3300/0 conn(0x7f26c80c3cf0 0x7f26c80ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:45.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.291+0000 7f26c7fff700 1 -- 172.21.15.46:0/1706671619 >> v2:172.21.15.106:3300/0 conn(0x7f26c80c3cf0 msgr2=0x7f26c80ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:45.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.291+0000 7f26c7fff700 1 --2- 172.21.15.46:0/1706671619 >> v2:172.21.15.106:3300/0 conn(0x7f26c80c3cf0 0x7f26c80ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:45.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.291+0000 7f26c7fff700 1 -- 172.21.15.46:0/1706671619 >> v1:172.21.15.106:6789/0 conn(0x7f26c80acf30 legacy=0x7f26c80ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:45.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.291+0000 7f26c7fff700 1 -- 172.21.15.46:0/1706671619 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c80cd450 con 0x7f26c80cd080 2024-01-27T01:03:45.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.292+0000 7f26c7fff700 1 --2- 172.21.15.46:0/1706671619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80cd080 0x7f26c80ab1b0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f26c80b1060 tx=0x7f26c0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c020bd4a4a5c8f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:45.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.292+0000 7f26c67fc700 1 -- 172.21.15.46:0/1706671619 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c000e070 con 0x7f26c80cd080 2024-01-27T01:03:45.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.292+0000 7f26c67fc700 1 -- 172.21.15.46:0/1706671619 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c000ba10 con 0x7f26c80cd080 2024-01-27T01:03:45.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.292+0000 7f26c67fc700 1 -- 172.21.15.46:0/1706671619 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c00092e0 con 0x7f26c80cd080 2024-01-27T01:03:45.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.292+0000 7f26cdc9d700 1 -- 172.21.15.46:0/1706671619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80cd080 msgr2=0x7f26c80ab1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:45.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.292+0000 7f26cdc9d700 1 --2- 172.21.15.46:0/1706671619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80cd080 0x7f26c80ab1b0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f26c80b1060 tx=0x7f26c0009840 comp rx=0 tx=0).stop 2024-01-27T01:03:45.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.292+0000 7f26cdc9d700 1 -- 172.21.15.46:0/1706671619 shutdown_connections 2024-01-27T01:03:45.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.292+0000 7f26cdc9d700 1 --2- 172.21.15.46:0/1706671619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80cd080 0x7f26c80ab1b0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:45.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.292+0000 7f26cdc9d700 1 --2- 172.21.15.46:0/1706671619 >> v2:172.21.15.106:3300/0 conn(0x7f26c80c3cf0 0x7f26c80ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:45.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.292+0000 7f26cdc9d700 1 -- 172.21.15.46:0/1706671619 >> 172.21.15.46:0/1706671619 conn(0x7f26c8097820 msgr2=0x7f26c8068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:45.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.292+0000 7f26cdc9d700 1 -- 172.21.15.46:0/1706671619 shutdown_connections 2024-01-27T01:03:45.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.293+0000 7f26cdc9d700 1 -- 172.21.15.46:0/1706671619 wait complete. 2024-01-27T01:03:45.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.294+0000 7f26cdc9d700 1 Processor -- start 2024-01-27T01:03:45.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.294+0000 7f26cdc9d700 1 -- start start 2024-01-27T01:03:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.295+0000 7f26cdc9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26c80acf30 0x7f26c81537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.295+0000 7f26cdc9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80c3cf0 0x7f26c80b0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.295+0000 7f26cdc9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26c80cd080 0x7f26c80b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.295+0000 7f26cdc9d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c80b1d20 con 0x7f26c80c3cf0 2024-01-27T01:03:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.295+0000 7f26cdc9d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c80b0d20 con 0x7f26c80cd080 2024-01-27T01:03:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.295+0000 7f26cdc9d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c80b21b0 con 0x7f26c80acf30 2024-01-27T01:03:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.295+0000 7f26c6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80c3cf0 0x7f26c80b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.295+0000 7f26c6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80c3cf0 0x7f26c80b0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39000/0 (socket says 172.21.15.46:39000) 2024-01-27T01:03:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.295+0000 7f26c6ffd700 1 -- 172.21.15.46:0/523890336 learned_addr learned my addr 172.21.15.46:0/523890336 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.295+0000 7f26c7fff700 1 --2- 172.21.15.46:0/523890336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26c80cd080 0x7f26c80b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.295+0000 7f26c77fe700 1 --2- 172.21.15.46:0/523890336 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26c80acf30 0x7f26c81537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.295+0000 7f26c6ffd700 1 -- 172.21.15.46:0/523890336 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26c80acf30 msgr2=0x7f26c81537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.295+0000 7f26c6ffd700 1 --2- 172.21.15.46:0/523890336 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26c80acf30 0x7f26c81537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:45.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.295+0000 7f26c6ffd700 1 -- 172.21.15.46:0/523890336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26c80cd080 msgr2=0x7f26c80b06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:45.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.295+0000 7f26c6ffd700 1 --2- 172.21.15.46:0/523890336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26c80cd080 0x7f26c80b06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:45.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.295+0000 7f26c6ffd700 1 -- 172.21.15.46:0/523890336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c0009ae0 con 0x7f26c80c3cf0 2024-01-27T01:03:45.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.295+0000 7f26c6ffd700 1 --2- 172.21.15.46:0/523890336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80c3cf0 0x7f26c80b0180 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f26bc00d8c0 tx=0x7f26bc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:45.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.296+0000 7f26af7fe700 1 -- 172.21.15.46:0/523890336 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26bc00de60 con 0x7f26c80c3cf0 2024-01-27T01:03:45.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.296+0000 7f26af7fe700 1 -- 172.21.15.46:0/523890336 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26bc00b780 con 0x7f26c80c3cf0 2024-01-27T01:03:45.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.296+0000 7f26af7fe700 1 -- 172.21.15.46:0/523890336 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26bc00bb20 con 0x7f26c80c3cf0 2024-01-27T01:03:45.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.296+0000 7f26cdc9d700 1 -- 172.21.15.46:0/523890336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c80b13f0 con 0x7f26c80c3cf0 2024-01-27T01:03:45.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.296+0000 7f26cdc9d700 1 -- 172.21.15.46:0/523890336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c80b19b0 con 0x7f26c80c3cf0 2024-01-27T01:03:45.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.297+0000 7f26cdc9d700 1 -- 172.21.15.46:0/523890336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c804c710 con 0x7f26c80c3cf0 2024-01-27T01:03:45.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.297+0000 7f26af7fe700 1 -- 172.21.15.46:0/523890336 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f26bc01e460 con 0x7f26c80c3cf0 2024-01-27T01:03:45.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.298+0000 7f26af7fe700 1 --2- 172.21.15.46:0/523890336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f26b4066f30 0x7f26b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:45.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.298+0000 7f26af7fe700 1 -- 172.21.15.46:0/523890336 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f26bc084ec0 con 0x7f26c80c3cf0 2024-01-27T01:03:45.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.298+0000 7f26c77fe700 1 --2- 172.21.15.46:0/523890336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f26b4066f30 0x7f26b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:45.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.298+0000 7f26c77fe700 1 --2- 172.21.15.46:0/523890336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f26b4066f30 0x7f26b40693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f26b80038c0 tx=0x7f26b8009040 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:45.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.302+0000 7f26af7fe700 1 -- 172.21.15.46:0/523890336 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f26bc051e20 con 0x7f26c80c3cf0 2024-01-27T01:03:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.507+0000 7f26cdc9d700 1 -- 172.21.15.46:0/523890336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26c80c4460 con 0x7f26c80c3cf0 2024-01-27T01:03:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.508+0000 7f26af7fe700 1 -- 172.21.15.46:0/523890336 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v204) v1 ==== 74+0+16293 (secure 0 0 0) 0x7f26bc055440 con 0x7f26c80c3cf0 2024-01-27T01:03:45.508 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:45.508 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":204,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:03:44.140802+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:03:42.648202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-01-27T01:03:43.106225+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:03:45.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.510+0000 7f26cdc9d700 1 -- 172.21.15.46:0/523890336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f26b4066f30 msgr2=0x7f26b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:45.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.510+0000 7f26cdc9d700 1 --2- 172.21.15.46:0/523890336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f26b4066f30 0x7f26b40693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f26b80038c0 tx=0x7f26b8009040 comp rx=0 tx=0).stop 2024-01-27T01:03:45.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.510+0000 7f26cdc9d700 1 -- 172.21.15.46:0/523890336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80c3cf0 msgr2=0x7f26c80b0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:45.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.510+0000 7f26cdc9d700 1 --2- 172.21.15.46:0/523890336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80c3cf0 0x7f26c80b0180 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f26bc00d8c0 tx=0x7f26bc008040 comp rx=0 tx=0).stop 2024-01-27T01:03:45.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.510+0000 7f26cdc9d700 1 -- 172.21.15.46:0/523890336 shutdown_connections 2024-01-27T01:03:45.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.510+0000 7f26cdc9d700 1 --2- 172.21.15.46:0/523890336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f26b4066f30 0x7f26b40693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:45.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.510+0000 7f26cdc9d700 1 --2- 172.21.15.46:0/523890336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26c80cd080 0x7f26c80b06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:45.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.510+0000 7f26cdc9d700 1 --2- 172.21.15.46:0/523890336 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26c80acf30 0x7f26c81537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:45.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.510+0000 7f26cdc9d700 1 --2- 172.21.15.46:0/523890336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26c80c3cf0 0x7f26c80b0180 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:45.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.510+0000 7f26cdc9d700 1 -- 172.21.15.46:0/523890336 >> 172.21.15.46:0/523890336 conn(0x7f26c8097820 msgr2=0x7f26c80cc330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:45.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.510+0000 7f26cdc9d700 1 -- 172.21.15.46:0/523890336 shutdown_connections 2024-01-27T01:03:45.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.510+0000 7f26cdc9d700 1 -- 172.21.15.46:0/523890336 wait complete. 2024-01-27T01:03:45.524 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-27T01:03:45.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-01-27T01:03:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.659+0000 7f995d7eb700 1 Processor -- start 2024-01-27T01:03:45.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.659+0000 7f995d7eb700 1 -- start start 2024-01-27T01:03:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.660+0000 7f995d7eb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9958159ef0 0x7f99580b09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.660+0000 7f995d7eb700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99580cd0f0 con 0x7f99580b1ec0 2024-01-27T01:03:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.660+0000 7f995d7eb700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99580b10a0 con 0x7f9958159b20 2024-01-27T01:03:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.660+0000 7f995d7eb700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99580b1380 con 0x7f9958159ef0 2024-01-27T01:03:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.660+0000 7f99567fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9958159ef0 0x7f99580b09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.660+0000 7f99567fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9958159ef0 0x7f99580b09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39006/0 (socket says 172.21.15.46:39006) 2024-01-27T01:03:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.660+0000 7f99577fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f99580b1ec0 0x7f99580b22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57306/0 (socket says 172.21.15.46:57306) 2024-01-27T01:03:45.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.660+0000 7f99567fc700 1 -- 172.21.15.46:0/997663263 learned_addr learned my addr 172.21.15.46:0/997663263 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:45.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.661+0000 7f99567fc700 1 -- 172.21.15.46:0/997663263 >> v1:172.21.15.106:6789/0 conn(0x7f9958159b20 legacy=0x7f99580b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:45.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.661+0000 7f99567fc700 1 -- 172.21.15.46:0/997663263 >> v1:172.21.15.46:6789/0 conn(0x7f99580b1ec0 legacy=0x7f99580b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:45.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.661+0000 7f99567fc700 1 -- 172.21.15.46:0/997663263 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99580b1660 con 0x7f9958159ef0 2024-01-27T01:03:45.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.661+0000 7f99567fc700 1 --2- 172.21.15.46:0/997663263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9958159ef0 0x7f99580b09d0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f994c0099b0 tx=0x7f994c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60e431a6475ff79d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:45.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.661+0000 7f9955ffb700 1 -- 172.21.15.46:0/997663263 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994c00b860 con 0x7f9958159ef0 2024-01-27T01:03:45.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.661+0000 7f9955ffb700 1 -- 172.21.15.46:0/997663263 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994c00b9c0 con 0x7f9958159ef0 2024-01-27T01:03:45.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.661+0000 7f9955ffb700 1 -- 172.21.15.46:0/997663263 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994c00bd60 con 0x7f9958159ef0 2024-01-27T01:03:45.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.662+0000 7f995d7eb700 1 -- 172.21.15.46:0/997663263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9958159ef0 msgr2=0x7f99580b09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:45.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.662+0000 7f995d7eb700 1 --2- 172.21.15.46:0/997663263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9958159ef0 0x7f99580b09d0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f994c0099b0 tx=0x7f994c009250 comp rx=0 tx=0).stop 2024-01-27T01:03:45.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.662+0000 7f995d7eb700 1 -- 172.21.15.46:0/997663263 shutdown_connections 2024-01-27T01:03:45.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.662+0000 7f995d7eb700 1 --2- 172.21.15.46:0/997663263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9958159ef0 0x7f99580b09d0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:45.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.662+0000 7f995d7eb700 1 -- 172.21.15.46:0/997663263 >> 172.21.15.46:0/997663263 conn(0x7f9958087f60 msgr2=0x7f995806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:45.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.662+0000 7f995d7eb700 1 -- 172.21.15.46:0/997663263 shutdown_connections 2024-01-27T01:03:45.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.662+0000 7f995d7eb700 1 -- 172.21.15.46:0/997663263 wait complete. 2024-01-27T01:03:45.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.663+0000 7f995d7eb700 1 Processor -- start 2024-01-27T01:03:45.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.663+0000 7f995d7eb700 1 -- start start 2024-01-27T01:03:45.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.664+0000 7f995d7eb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99580b1ec0 0x7f995814d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:45.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.664+0000 7f995d7eb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9958159b20 0x7f995809d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:45.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.664+0000 7f995d7eb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9958159ef0 0x7f995809de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:45.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.664+0000 7f995d7eb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995816e2f0 con 0x7f99580b1ec0 2024-01-27T01:03:45.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.664+0000 7f995d7eb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995814dcd0 con 0x7f9958159ef0 2024-01-27T01:03:45.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.664+0000 7f995d7eb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f995816d670 con 0x7f9958159b20 2024-01-27T01:03:45.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.664+0000 7f9956ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99580b1ec0 0x7f995814d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:45.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.664+0000 7f9956ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99580b1ec0 0x7f995814d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39018/0 (socket says 172.21.15.46:39018) 2024-01-27T01:03:45.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.664+0000 7f9956ffd700 1 -- 172.21.15.46:0/31025139 learned_addr learned my addr 172.21.15.46:0/31025139 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:45.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.664+0000 7f99567fc700 1 --2- 172.21.15.46:0/31025139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9958159b20 0x7f995809d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:45.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.664+0000 7f99577fe700 1 --2- 172.21.15.46:0/31025139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9958159ef0 0x7f995809de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:45.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.664+0000 7f9956ffd700 1 -- 172.21.15.46:0/31025139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9958159b20 msgr2=0x7f995809d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:45.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.664+0000 7f9956ffd700 1 --2- 172.21.15.46:0/31025139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9958159b20 0x7f995809d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:45.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.664+0000 7f9956ffd700 1 -- 172.21.15.46:0/31025139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9958159ef0 msgr2=0x7f995809de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:45.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.664+0000 7f9956ffd700 1 --2- 172.21.15.46:0/31025139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9958159ef0 0x7f995809de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:45.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.665+0000 7f9956ffd700 1 -- 172.21.15.46:0/31025139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0094f0 con 0x7f99580b1ec0 2024-01-27T01:03:45.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.665+0000 7f9956ffd700 1 --2- 172.21.15.46:0/31025139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99580b1ec0 0x7f995814d790 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f994800c7c0 tx=0x7f9948007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:45.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.665+0000 7f993effd700 1 -- 172.21.15.46:0/31025139 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9948011590 con 0x7f99580b1ec0 2024-01-27T01:03:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.665+0000 7f993effd700 1 -- 172.21.15.46:0/31025139 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9948011710 con 0x7f99580b1ec0 2024-01-27T01:03:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.665+0000 7f995d7eb700 1 -- 172.21.15.46:0/31025139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f995816d950 con 0x7f99580b1ec0 2024-01-27T01:03:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.665+0000 7f993effd700 1 -- 172.21.15.46:0/31025139 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9948011ad0 con 0x7f99580b1ec0 2024-01-27T01:03:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.665+0000 7f995d7eb700 1 -- 172.21.15.46:0/31025139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f995816df10 con 0x7f99580b1ec0 2024-01-27T01:03:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.666+0000 7f995d7eb700 1 -- 172.21.15.46:0/31025139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f995804c710 con 0x7f99580b1ec0 2024-01-27T01:03:45.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.666+0000 7f993effd700 1 -- 172.21.15.46:0/31025139 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f994801b3b0 con 0x7f99580b1ec0 2024-01-27T01:03:45.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.667+0000 7f993effd700 1 --2- 172.21.15.46:0/31025139 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9944066f30 0x7f99440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:45.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.667+0000 7f993effd700 1 -- 172.21.15.46:0/31025139 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f9948083b20 con 0x7f99580b1ec0 2024-01-27T01:03:45.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.667+0000 7f99567fc700 1 --2- 172.21.15.46:0/31025139 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9944066f30 0x7f99440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:45.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.668+0000 7f99567fc700 1 --2- 172.21.15.46:0/31025139 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9944066f30 0x7f99440693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f994c0099b0 tx=0x7f994c017000 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:45.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.671+0000 7f993effd700 1 -- 172.21.15.46:0/31025139 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9948050b30 con 0x7f99580b1ec0 2024-01-27T01:03:45.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:45.925+0000 7f995d7eb700 1 -- 172.21.15.46:0/31025139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f99580b1940 con 0x7f99580b1ec0 2024-01-27T01:03:46.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.727+0000 7f993effd700 1 -- 172.21.15.46:0/31025139 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v106) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9948017070 con 0x7f99580b1ec0 2024-01-27T01:03:46.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.730+0000 7f995d7eb700 1 -- 172.21.15.46:0/31025139 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9944066f30 msgr2=0x7f99440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.730+0000 7f995d7eb700 1 --2- 172.21.15.46:0/31025139 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9944066f30 0x7f99440693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f994c0099b0 tx=0x7f994c017000 comp rx=0 tx=0).stop 2024-01-27T01:03:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.730+0000 7f995d7eb700 1 -- 172.21.15.46:0/31025139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99580b1ec0 msgr2=0x7f995814d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.730+0000 7f995d7eb700 1 --2- 172.21.15.46:0/31025139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99580b1ec0 0x7f995814d790 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f994800c7c0 tx=0x7f9948007f70 comp rx=0 tx=0).stop 2024-01-27T01:03:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.730+0000 7f995d7eb700 1 -- 172.21.15.46:0/31025139 shutdown_connections 2024-01-27T01:03:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.730+0000 7f995d7eb700 1 --2- 172.21.15.46:0/31025139 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9944066f30 0x7f99440693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.730+0000 7f995d7eb700 1 --2- 172.21.15.46:0/31025139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99580b1ec0 0x7f995814d790 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.730+0000 7f995d7eb700 1 --2- 172.21.15.46:0/31025139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9958159ef0 0x7f995809de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.730+0000 7f995d7eb700 1 --2- 172.21.15.46:0/31025139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9958159b20 0x7f995809d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.730+0000 7f995d7eb700 1 -- 172.21.15.46:0/31025139 >> 172.21.15.46:0/31025139 conn(0x7f9958087f60 msgr2=0x7f99580d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.730+0000 7f995d7eb700 1 -- 172.21.15.46:0/31025139 shutdown_connections 2024-01-27T01:03:46.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.730+0000 7f995d7eb700 1 -- 172.21.15.46:0/31025139 wait complete. 2024-01-27T01:03:46.746 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-27T01:03:46.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.886+0000 7fa12e685700 1 Processor -- start 2024-01-27T01:03:46.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.886+0000 7fa12e685700 1 -- start start 2024-01-27T01:03:46.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.887+0000 7fa12e685700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa1280c8da0 0x7fa1280c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:46.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.887+0000 7fa12e685700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa128140af0 0x7fa128140f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:46.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.887+0000 7fa12e685700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1280ccc00 con 0x7fa1280caf30 2024-01-27T01:03:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.887+0000 7fa12e685700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1280cbbf0 con 0x7fa1280c8da0 2024-01-27T01:03:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.887+0000 7fa12e685700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1280cc310 con 0x7fa128140af0 2024-01-27T01:03:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.887+0000 7fa127fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa1280c8da0 0x7fa1280c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.887+0000 7fa127fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa1280c8da0 0x7fa1280c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39026/0 (socket says 172.21.15.46:39026) 2024-01-27T01:03:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.887+0000 7fa127fff700 1 -- 172.21.15.46:0/3282298175 learned_addr learned my addr 172.21.15.46:0/3282298175 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.888+0000 7fa1277fe700 1 --2- 172.21.15.46:0/3282298175 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa128140af0 0x7fa128140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.888+0000 7fa127fff700 1 -- 172.21.15.46:0/3282298175 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa128140af0 msgr2=0x7fa128140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.888+0000 7fa127fff700 1 --2- 172.21.15.46:0/3282298175 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa128140af0 0x7fa128140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.888+0000 7fa127fff700 1 -- 172.21.15.46:0/3282298175 >> v1:172.21.15.106:6789/0 conn(0x7fa1280caf30 legacy=0x7fa1280cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.888+0000 7fa127fff700 1 -- 172.21.15.46:0/3282298175 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1280cc5b0 con 0x7fa1280c8da0 2024-01-27T01:03:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.888+0000 7fa1277fe700 1 --2- 172.21.15.46:0/3282298175 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa128140af0 0x7fa128140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:03:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.888+0000 7fa127fff700 1 --2- 172.21.15.46:0/3282298175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1280c8da0 0x7fa1280c9170 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fa1180099b0 tx=0x7fa118009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fddcae994f26db68 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.888+0000 7fa126ffd700 1 -- 172.21.15.46:0/3282298175 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11800b820 con 0x7fa1280c8da0 2024-01-27T01:03:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.888+0000 7fa126ffd700 1 -- 172.21.15.46:0/3282298175 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11800b980 con 0x7fa1280c8da0 2024-01-27T01:03:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.888+0000 7fa126ffd700 1 -- 172.21.15.46:0/3282298175 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11800bd20 con 0x7fa1280c8da0 2024-01-27T01:03:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.889+0000 7fa12e685700 1 -- 172.21.15.46:0/3282298175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1280c8da0 msgr2=0x7fa1280c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.889+0000 7fa12e685700 1 --2- 172.21.15.46:0/3282298175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1280c8da0 0x7fa1280c9170 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fa1180099b0 tx=0x7fa118009250 comp rx=0 tx=0).stop 2024-01-27T01:03:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.889+0000 7fa12e685700 1 -- 172.21.15.46:0/3282298175 shutdown_connections 2024-01-27T01:03:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.889+0000 7fa12e685700 1 --2- 172.21.15.46:0/3282298175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1280c8da0 0x7fa1280c9170 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.889+0000 7fa12e685700 1 --2- 172.21.15.46:0/3282298175 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa128140af0 0x7fa128140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.889+0000 7fa12e685700 1 -- 172.21.15.46:0/3282298175 >> 172.21.15.46:0/3282298175 conn(0x7fa12806bd90 msgr2=0x7fa12806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.889+0000 7fa12e685700 1 -- 172.21.15.46:0/3282298175 shutdown_connections 2024-01-27T01:03:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.889+0000 7fa12e685700 1 -- 172.21.15.46:0/3282298175 wait complete. 2024-01-27T01:03:46.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.890+0000 7fa12e685700 1 Processor -- start 2024-01-27T01:03:46.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.890+0000 7fa12e685700 1 -- start start 2024-01-27T01:03:46.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.891+0000 7fa12e685700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1280c8da0 0x7fa128068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:46.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.891+0000 7fa12e685700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1280caf30 0x7fa1280c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:46.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.891+0000 7fa12e685700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa128140af0 0x7fa12813d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:46.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.891+0000 7fa12e685700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa128169580 con 0x7fa1280caf30 2024-01-27T01:03:46.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.891+0000 7fa12e685700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12806a190 con 0x7fa128140af0 2024-01-27T01:03:46.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.891+0000 7fa12e685700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa128069020 con 0x7fa1280c8da0 2024-01-27T01:03:46.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.891+0000 7fa1277fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1280caf30 0x7fa1280c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:46.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.891+0000 7fa1277fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1280caf30 0x7fa1280c7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39042/0 (socket says 172.21.15.46:39042) 2024-01-27T01:03:46.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.891+0000 7fa1277fe700 1 -- 172.21.15.46:0/1132609988 learned_addr learned my addr 172.21.15.46:0/1132609988 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:46.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.891+0000 7fa127fff700 1 --2- 172.21.15.46:0/1132609988 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1280c8da0 0x7fa128068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:46.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.891+0000 7fa12cc22700 1 --2- 172.21.15.46:0/1132609988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa128140af0 0x7fa12813d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:46.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.891+0000 7fa1277fe700 1 -- 172.21.15.46:0/1132609988 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1280c8da0 msgr2=0x7fa128068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:46.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.891+0000 7fa1277fe700 1 --2- 172.21.15.46:0/1132609988 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1280c8da0 0x7fa128068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:46.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.891+0000 7fa1277fe700 1 -- 172.21.15.46:0/1132609988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa128140af0 msgr2=0x7fa12813d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:46.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.891+0000 7fa1277fe700 1 --2- 172.21.15.46:0/1132609988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa128140af0 0x7fa12813d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:46.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.891+0000 7fa1277fe700 1 -- 172.21.15.46:0/1132609988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1180094f0 con 0x7fa1280caf30 2024-01-27T01:03:46.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.892+0000 7fa1277fe700 1 --2- 172.21.15.46:0/1132609988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1280caf30 0x7fa1280c7930 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fa11000c850 tx=0x7fa110008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:46.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.892+0000 7fa11ffff700 1 -- 172.21.15.46:0/1132609988 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110011600 con 0x7fa1280caf30 2024-01-27T01:03:46.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.892+0000 7fa11ffff700 1 -- 172.21.15.46:0/1132609988 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa110011760 con 0x7fa1280caf30 2024-01-27T01:03:46.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.892+0000 7fa12e685700 1 -- 172.21.15.46:0/1132609988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa128069300 con 0x7fa1280caf30 2024-01-27T01:03:46.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.892+0000 7fa11ffff700 1 -- 172.21.15.46:0/1132609988 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110011b00 con 0x7fa1280caf30 2024-01-27T01:03:46.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.892+0000 7fa12e685700 1 -- 172.21.15.46:0/1132609988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa12813d610 con 0x7fa1280caf30 2024-01-27T01:03:46.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.893+0000 7fa12e685700 1 -- 172.21.15.46:0/1132609988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa12804c710 con 0x7fa1280caf30 2024-01-27T01:03:46.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.893+0000 7fa11ffff700 1 -- 172.21.15.46:0/1132609988 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa1100073b0 con 0x7fa1280caf30 2024-01-27T01:03:46.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.894+0000 7fa11ffff700 1 --2- 172.21.15.46:0/1132609988 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa108066f30 0x7fa1080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:46.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.894+0000 7fa11ffff700 1 -- 172.21.15.46:0/1132609988 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fa1100847c0 con 0x7fa1280caf30 2024-01-27T01:03:46.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.894+0000 7fa127fff700 1 --2- 172.21.15.46:0/1132609988 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa108066f30 0x7fa1080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:46.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.895+0000 7fa127fff700 1 --2- 172.21.15.46:0/1132609988 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa108066f30 0x7fa1080693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa1180099b0 tx=0x7fa118016000 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:46.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:46.899+0000 7fa11ffff700 1 -- 172.21.15.46:0/1132609988 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa1100551b0 con 0x7fa1280caf30 2024-01-27T01:03:47.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.153+0000 7fa12e685700 1 -- 172.21.15.46:0/1132609988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa1280cc890 con 0x7fa1280caf30 2024-01-27T01:03:47.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.729+0000 7fa11ffff700 1 -- 172.21.15.46:0/1132609988 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v107) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa110054d40 con 0x7fa1280caf30 2024-01-27T01:03:47.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.732+0000 7fa12e685700 1 -- 172.21.15.46:0/1132609988 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa108066f30 msgr2=0x7fa1080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:47.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.732+0000 7fa12e685700 1 --2- 172.21.15.46:0/1132609988 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa108066f30 0x7fa1080693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa1180099b0 tx=0x7fa118016000 comp rx=0 tx=0).stop 2024-01-27T01:03:47.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.732+0000 7fa12e685700 1 -- 172.21.15.46:0/1132609988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1280caf30 msgr2=0x7fa1280c7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:47.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.732+0000 7fa12e685700 1 --2- 172.21.15.46:0/1132609988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1280caf30 0x7fa1280c7930 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fa11000c850 tx=0x7fa110008040 comp rx=0 tx=0).stop 2024-01-27T01:03:47.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.732+0000 7fa12e685700 1 -- 172.21.15.46:0/1132609988 shutdown_connections 2024-01-27T01:03:47.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.732+0000 7fa12e685700 1 --2- 172.21.15.46:0/1132609988 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa108066f30 0x7fa1080693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:47.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.732+0000 7fa12e685700 1 --2- 172.21.15.46:0/1132609988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa128140af0 0x7fa12813d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:47.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.732+0000 7fa12e685700 1 --2- 172.21.15.46:0/1132609988 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1280c8da0 0x7fa128068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:47.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.732+0000 7fa12e685700 1 --2- 172.21.15.46:0/1132609988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1280caf30 0x7fa1280c7930 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:47.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.732+0000 7fa12e685700 1 -- 172.21.15.46:0/1132609988 >> 172.21.15.46:0/1132609988 conn(0x7fa12806bd90 msgr2=0x7fa12813d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:47.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.732+0000 7fa12e685700 1 -- 172.21.15.46:0/1132609988 shutdown_connections 2024-01-27T01:03:47.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.732+0000 7fa12e685700 1 -- 172.21.15.46:0/1132609988 wait complete. 2024-01-27T01:03:47.747 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:03:47.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.884+0000 7f1ff2ee6700 1 Processor -- start 2024-01-27T01:03:47.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.884+0000 7f1ff2ee6700 1 -- start start 2024-01-27T01:03:47.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.885+0000 7f1ff2ee6700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1fec0b12c0 0x7f1fec1646a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:47.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.885+0000 7f1ff2ee6700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fec06f940 con 0x7f1fec159b30 2024-01-27T01:03:47.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.885+0000 7f1ff2ee6700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fec0b3230 con 0x7f1fec159760 2024-01-27T01:03:47.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.885+0000 7f1ff2ee6700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fec06ed30 con 0x7f1fec0b12c0 2024-01-27T01:03:47.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.885+0000 7f1febfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1fec159b30 0x7f1fec0b0bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57322/0 (socket says 172.21.15.46:57322) 2024-01-27T01:03:47.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.885+0000 7f1febfff700 1 -- 172.21.15.46:0/3279617016 learned_addr learned my addr 172.21.15.46:0/3279617016 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:47.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.885+0000 7f1ff1483700 1 --2- 172.21.15.46:0/3279617016 >> v2:172.21.15.106:3300/0 conn(0x7f1fec0b12c0 0x7f1fec1646a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:47.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1feb7fe700 1 -- 172.21.15.46:0/3279617016 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992872135 0 0) 0x7f1fec06f940 con 0x7f1fec159b30 2024-01-27T01:03:47.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1feb7fe700 1 -- 172.21.15.46:0/3279617016 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fdc003570 con 0x7f1fec159b30 2024-01-27T01:03:47.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1feb7fe700 1 -- 172.21.15.46:0/3279617016 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1628036385 0 0) 0x7f1fdc003570 con 0x7f1fec159b30 2024-01-27T01:03:47.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1feb7fe700 1 -- 172.21.15.46:0/3279617016 >> v2:172.21.15.106:3300/0 conn(0x7f1fec0b12c0 msgr2=0x7f1fec1646a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:47.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1feb7fe700 1 --2- 172.21.15.46:0/3279617016 >> v2:172.21.15.106:3300/0 conn(0x7f1fec0b12c0 0x7f1fec1646a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:47.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1feb7fe700 1 -- 172.21.15.46:0/3279617016 >> v1:172.21.15.106:6789/0 conn(0x7f1fec159760 legacy=0x7f1fec0b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:47.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1feb7fe700 1 -- 172.21.15.46:0/3279617016 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec06f010 con 0x7f1fec159b30 2024-01-27T01:03:47.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1feb7fe700 1 -- 172.21.15.46:0/3279617016 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f1fd8001470 con 0x7f1fec159b30 2024-01-27T01:03:47.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1ff1483700 1 --2- 172.21.15.46:0/3279617016 >> v2:172.21.15.106:3300/0 conn(0x7f1fec0b12c0 0x7f1fec1646a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:47.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1feb7fe700 1 -- 172.21.15.46:0/3279617016 >> v1:172.21.15.46:6789/0 conn(0x7f1fec159b30 legacy=0x7f1fec0b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:47.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1feb7fe700 1 --2- 172.21.15.46:0/3279617016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1fdc003a60 0x7f1fdc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:47.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1feb7fe700 1 --2- 172.21.15.46:0/3279617016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fdc004810 0x7f1fdc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:47.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1feb7fe700 1 --2- 172.21.15.46:0/3279617016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1fdc0051c0 0x7f1fdc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:47.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1feb7fe700 1 -- 172.21.15.46:0/3279617016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec06ed30 con 0x7f1fdc004810 2024-01-27T01:03:47.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1feb7fe700 1 -- 172.21.15.46:0/3279617016 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc00c0e0 con 0x7f1fdc003a60 2024-01-27T01:03:47.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1feb7fe700 1 -- 172.21.15.46:0/3279617016 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc00c2a0 con 0x7f1fdc0051c0 2024-01-27T01:03:47.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1febfff700 1 --2- 172.21.15.46:0/3279617016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fdc004810 0x7f1fdc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:47.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1ff0c82700 1 --2- 172.21.15.46:0/3279617016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1fdc003a60 0x7f1fdc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:47.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1ff1483700 1 --2- 172.21.15.46:0/3279617016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1fdc0051c0 0x7f1fdc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:47.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1febfff700 1 -- 172.21.15.46:0/3279617016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1fdc0051c0 msgr2=0x7f1fdc00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:47.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1febfff700 1 --2- 172.21.15.46:0/3279617016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1fdc0051c0 0x7f1fdc00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:47.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1febfff700 1 -- 172.21.15.46:0/3279617016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1fdc003a60 msgr2=0x7f1fdc003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:47.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1febfff700 1 --2- 172.21.15.46:0/3279617016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1fdc003a60 0x7f1fdc003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:47.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1febfff700 1 -- 172.21.15.46:0/3279617016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc003570 con 0x7f1fdc004810 2024-01-27T01:03:47.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1ff0c82700 1 --2- 172.21.15.46:0/3279617016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1fdc003a60 0x7f1fdc003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:03:47.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.886+0000 7f1ff1483700 1 -- 172.21.15.46:0/3279617016 reap_dead start 2024-01-27T01:03:47.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.887+0000 7f1febfff700 1 --2- 172.21.15.46:0/3279617016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fdc004810 0x7f1fdc004c80 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800c3c0 tx=0x7f1fd8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f42c10e9a1682c97 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:47.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.887+0000 7f1feb7fe700 1 -- 172.21.15.46:0/3279617016 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd800cc90 con 0x7f1fdc004810 2024-01-27T01:03:47.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.887+0000 7f1feb7fe700 1 -- 172.21.15.46:0/3279617016 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd800cdf0 con 0x7f1fdc004810 2024-01-27T01:03:47.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.887+0000 7f1feb7fe700 1 -- 172.21.15.46:0/3279617016 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8005b70 con 0x7f1fdc004810 2024-01-27T01:03:47.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.887+0000 7f1ff2ee6700 1 -- 172.21.15.46:0/3279617016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fdc004810 msgr2=0x7f1fdc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:47.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.887+0000 7f1ff2ee6700 1 --2- 172.21.15.46:0/3279617016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fdc004810 0x7f1fdc004c80 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800c3c0 tx=0x7f1fd8001e70 comp rx=0 tx=0).stop 2024-01-27T01:03:47.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.887+0000 7f1ff2ee6700 1 -- 172.21.15.46:0/3279617016 shutdown_connections 2024-01-27T01:03:47.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.887+0000 7f1ff2ee6700 1 --2- 172.21.15.46:0/3279617016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fdc004810 0x7f1fdc004c80 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:47.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.887+0000 7f1ff2ee6700 1 -- 172.21.15.46:0/3279617016 >> 172.21.15.46:0/3279617016 conn(0x7f1fec069510 msgr2=0x7f1fec0aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:47.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.887+0000 7f1ff2ee6700 1 -- 172.21.15.46:0/3279617016 shutdown_connections 2024-01-27T01:03:47.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.887+0000 7f1ff2ee6700 1 -- 172.21.15.46:0/3279617016 wait complete. 2024-01-27T01:03:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.889+0000 7f1ff2ee6700 1 Processor -- start 2024-01-27T01:03:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.889+0000 7f1ff2ee6700 1 -- start start 2024-01-27T01:03:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.890+0000 7f1ff2ee6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1fdc004810 0x7f1fec15c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.890+0000 7f1ff2ee6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1fec158860 0x7f1fec155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.890+0000 7f1ff2ee6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fec14fa50 0x7f1fec1542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.890+0000 7f1ff2ee6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec15ae30 con 0x7f1fec14fa50 2024-01-27T01:03:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.890+0000 7f1ff2ee6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec155f60 con 0x7f1fdc004810 2024-01-27T01:03:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.890+0000 7f1ff2ee6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec156240 con 0x7f1fec158860 2024-01-27T01:03:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.890+0000 7f1ff1483700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fec14fa50 0x7f1fec1542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.891+0000 7f1ff1483700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fec14fa50 0x7f1fec1542e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39054/0 (socket says 172.21.15.46:39054) 2024-01-27T01:03:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.891+0000 7f1ff1483700 1 -- 172.21.15.46:0/1539560260 learned_addr learned my addr 172.21.15.46:0/1539560260 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:47.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.891+0000 7f1febfff700 1 --2- 172.21.15.46:0/1539560260 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1fec158860 0x7f1fec155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:47.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.891+0000 7f1ff0c82700 1 --2- 172.21.15.46:0/1539560260 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1fdc004810 0x7f1fec15c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:47.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.891+0000 7f1ff1483700 1 -- 172.21.15.46:0/1539560260 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1fec158860 msgr2=0x7f1fec155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:47.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.891+0000 7f1ff1483700 1 --2- 172.21.15.46:0/1539560260 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1fec158860 0x7f1fec155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:47.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.891+0000 7f1ff1483700 1 -- 172.21.15.46:0/1539560260 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1fdc004810 msgr2=0x7f1fec15c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:47.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.891+0000 7f1ff1483700 1 --2- 172.21.15.46:0/1539560260 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1fdc004810 0x7f1fec15c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:47.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.891+0000 7f1ff1483700 1 -- 172.21.15.46:0/1539560260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec156520 con 0x7f1fec14fa50 2024-01-27T01:03:47.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.891+0000 7f1ff0c82700 1 --2- 172.21.15.46:0/1539560260 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1fdc004810 0x7f1fec15c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:03:47.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.891+0000 7f1ff1483700 1 --2- 172.21.15.46:0/1539560260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fec14fa50 0x7f1fec1542e0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0b16c0 tx=0x7f1fe400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:47.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.891+0000 7f1fe8ff9700 1 -- 172.21.15.46:0/1539560260 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe400e070 con 0x7f1fec14fa50 2024-01-27T01:03:47.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.891+0000 7f1fe8ff9700 1 -- 172.21.15.46:0/1539560260 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe4011d00 con 0x7f1fec14fa50 2024-01-27T01:03:47.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.891+0000 7f1ff2ee6700 1 -- 172.21.15.46:0/1539560260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fec156800 con 0x7f1fec14fa50 2024-01-27T01:03:47.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.891+0000 7f1fe8ff9700 1 -- 172.21.15.46:0/1539560260 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe40166c0 con 0x7f1fec14fa50 2024-01-27T01:03:47.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.891+0000 7f1ff2ee6700 1 -- 172.21.15.46:0/1539560260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fec14fec0 con 0x7f1fec14fa50 2024-01-27T01:03:47.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.892+0000 7f1ff2ee6700 1 -- 172.21.15.46:0/1539560260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fec156d20 con 0x7f1fec14fa50 2024-01-27T01:03:47.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.893+0000 7f1fe8ff9700 1 -- 172.21.15.46:0/1539560260 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1fe4016af0 con 0x7f1fec14fa50 2024-01-27T01:03:47.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.893+0000 7f1fe8ff9700 1 --2- 172.21.15.46:0/1539560260 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1fd0066f30 0x7f1fd00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:47.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.893+0000 7f1fe8ff9700 1 -- 172.21.15.46:0/1539560260 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f1fe4085440 con 0x7f1fec14fa50 2024-01-27T01:03:47.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.893+0000 7f1ff0c82700 1 --2- 172.21.15.46:0/1539560260 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1fd0066f30 0x7f1fd00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:47.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.894+0000 7f1ff0c82700 1 --2- 172.21.15.46:0/1539560260 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1fd0066f30 0x7f1fd00693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0b1780 tx=0x7f1fe0009040 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:47.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:47.898+0000 7f1fe8ff9700 1 -- 172.21.15.46:0/1539560260 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1fec156d20 con 0x7f1fec14fa50 2024-01-27T01:03:48.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.150+0000 7f1ff2ee6700 1 -- 172.21.15.46:0/1539560260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fec156d20 con 0x7f1fec14fa50 2024-01-27T01:03:48.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.151+0000 7f1fe8ff9700 1 -- 172.21.15.46:0/1539560260 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4612 (secure 0 0 0) 0x7f1fec156d20 con 0x7f1fec14fa50 2024-01-27T01:03:48.151 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:48.151 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":107,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8891,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/975115379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":975115379},{"type":"v1","addr":"172.21.15.106:6833","nonce":975115379}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10036,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3860120510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3860120510},{"type":"v1","addr":"172.21.15.46:6834","nonce":3860120510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10045,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3603549739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3603549739},{"type":"v1","addr":"172.21.15.106:6835","nonce":3603549739}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:03:44.141242+0000","modified":"2024-01-27T01:03:47.727201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10392":{"gid":10392,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/976738644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":976738644},{"type":"v1","addr":"172.21.15.46:6835","nonce":976738644}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2024-01-27T01:03:48.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.153+0000 7f1ff2ee6700 1 -- 172.21.15.46:0/1539560260 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1fd0066f30 msgr2=0x7f1fd00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:48.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.153+0000 7f1ff2ee6700 1 --2- 172.21.15.46:0/1539560260 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1fd0066f30 0x7f1fd00693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0b1780 tx=0x7f1fe0009040 comp rx=0 tx=0).stop 2024-01-27T01:03:48.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.153+0000 7f1ff2ee6700 1 -- 172.21.15.46:0/1539560260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fec14fa50 msgr2=0x7f1fec1542e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:48.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.153+0000 7f1ff2ee6700 1 --2- 172.21.15.46:0/1539560260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fec14fa50 0x7f1fec1542e0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0b16c0 tx=0x7f1fe400b040 comp rx=0 tx=0).stop 2024-01-27T01:03:48.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.153+0000 7f1ff2ee6700 1 -- 172.21.15.46:0/1539560260 shutdown_connections 2024-01-27T01:03:48.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.153+0000 7f1ff2ee6700 1 --2- 172.21.15.46:0/1539560260 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1fd0066f30 0x7f1fd00693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.153+0000 7f1ff2ee6700 1 --2- 172.21.15.46:0/1539560260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fec14fa50 0x7f1fec1542e0 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.153+0000 7f1ff2ee6700 1 --2- 172.21.15.46:0/1539560260 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1fdc004810 0x7f1fec15c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.153+0000 7f1ff2ee6700 1 --2- 172.21.15.46:0/1539560260 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1fec158860 0x7f1fec155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.153+0000 7f1ff2ee6700 1 -- 172.21.15.46:0/1539560260 >> 172.21.15.46:0/1539560260 conn(0x7f1fec069510 msgr2=0x7f1fec0add80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:48.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.153+0000 7f1ff2ee6700 1 -- 172.21.15.46:0/1539560260 shutdown_connections 2024-01-27T01:03:48.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.153+0000 7f1ff2ee6700 1 -- 172.21.15.46:0/1539560260 wait complete. 2024-01-27T01:03:48.155 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 107 2024-01-27T01:03:48.167 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:03:48.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.299+0000 7fc1da788700 1 Processor -- start 2024-01-27T01:03:48.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.299+0000 7fc1da788700 1 -- start start 2024-01-27T01:03:48.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.300+0000 7fc1da788700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1d4159590 0x7fc1d415d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:48.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.300+0000 7fc1da788700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1d406c3c0 0x7fc1d4069370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:48.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.300+0000 7fc1da788700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d415dd90 con 0x7fc1d415d8f0 2024-01-27T01:03:48.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.300+0000 7fc1da788700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1d415e350 con 0x7fc1d4159590 2024-01-27T01:03:48.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.300+0000 7fc1da788700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d40c97c0 con 0x7fc1d406c3c0 2024-01-27T01:03:48.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.300+0000 7fc1d3fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1d4159590 0x7fc1d415d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:48.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.300+0000 7fc1d3fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1d4159590 0x7fc1d415d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39060/0 (socket says 172.21.15.46:39060) 2024-01-27T01:03:48.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.300+0000 7fc1d3fff700 1 -- 172.21.15.46:0/3457233925 learned_addr learned my addr 172.21.15.46:0/3457233925 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:48.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.301+0000 7fc1d8d25700 1 --2- 172.21.15.46:0/3457233925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1d406c3c0 0x7fc1d4069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:48.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.301+0000 7fc1d3fff700 1 -- 172.21.15.46:0/3457233925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1d406c3c0 msgr2=0x7fc1d4069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:48.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.301+0000 7fc1d3fff700 1 --2- 172.21.15.46:0/3457233925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1d406c3c0 0x7fc1d4069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.301+0000 7fc1d3fff700 1 -- 172.21.15.46:0/3457233925 >> v1:172.21.15.46:6789/0 conn(0x7fc1d415d8f0 legacy=0x7fc1d406bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:48.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.301+0000 7fc1d3fff700 1 -- 172.21.15.46:0/3457233925 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d415e070 con 0x7fc1d4159590 2024-01-27T01:03:48.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.301+0000 7fc1d3fff700 1 --2- 172.21.15.46:0/3457233925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1d4159590 0x7fc1d415d3b0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fc1c40099e0 tx=0x7fc1c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3ac377110628d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:48.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.301+0000 7fc1d2ffd700 1 -- 172.21.15.46:0/3457233925 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c400b820 con 0x7fc1d4159590 2024-01-27T01:03:48.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.301+0000 7fc1d2ffd700 1 -- 172.21.15.46:0/3457233925 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c400b980 con 0x7fc1d4159590 2024-01-27T01:03:48.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.301+0000 7fc1d2ffd700 1 -- 172.21.15.46:0/3457233925 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c400bd40 con 0x7fc1d4159590 2024-01-27T01:03:48.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.302+0000 7fc1da788700 1 -- 172.21.15.46:0/3457233925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1d4159590 msgr2=0x7fc1d415d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:48.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.302+0000 7fc1da788700 1 --2- 172.21.15.46:0/3457233925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1d4159590 0x7fc1d415d3b0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fc1c40099e0 tx=0x7fc1c4009250 comp rx=0 tx=0).stop 2024-01-27T01:03:48.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.302+0000 7fc1da788700 1 -- 172.21.15.46:0/3457233925 shutdown_connections 2024-01-27T01:03:48.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.302+0000 7fc1da788700 1 --2- 172.21.15.46:0/3457233925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1d406c3c0 0x7fc1d4069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.302+0000 7fc1da788700 1 --2- 172.21.15.46:0/3457233925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1d4159590 0x7fc1d415d3b0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.302+0000 7fc1da788700 1 -- 172.21.15.46:0/3457233925 >> 172.21.15.46:0/3457233925 conn(0x7fc1d40ce530 msgr2=0x7fc1d40ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:48.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.302+0000 7fc1da788700 1 -- 172.21.15.46:0/3457233925 shutdown_connections 2024-01-27T01:03:48.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.302+0000 7fc1da788700 1 -- 172.21.15.46:0/3457233925 wait complete. 2024-01-27T01:03:48.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.303+0000 7fc1da788700 1 Processor -- start 2024-01-27T01:03:48.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.304+0000 7fc1da788700 1 -- start start 2024-01-27T01:03:48.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.305+0000 7fc1da788700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1d406c3c0 0x7fc1d4169b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:48.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.305+0000 7fc1da788700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1d4159590 0x7fc1d40c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:48.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.305+0000 7fc1da788700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1d415d8f0 0x7fc1d414c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:48.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.305+0000 7fc1da788700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d416a080 con 0x7fc1d406c3c0 2024-01-27T01:03:48.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.305+0000 7fc1da788700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d40c1d50 con 0x7fc1d4159590 2024-01-27T01:03:48.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.305+0000 7fc1da788700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d40c2310 con 0x7fc1d415d8f0 2024-01-27T01:03:48.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.305+0000 7fc1d3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1d406c3c0 0x7fc1d4169b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:48.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.305+0000 7fc1d3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1d406c3c0 0x7fc1d4169b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39064/0 (socket says 172.21.15.46:39064) 2024-01-27T01:03:48.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.305+0000 7fc1d3fff700 1 -- 172.21.15.46:0/243260973 learned_addr learned my addr 172.21.15.46:0/243260973 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:48.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.305+0000 7fc1d3fff700 1 -- 172.21.15.46:0/243260973 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1d415d8f0 msgr2=0x7fc1d414c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:48.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.305+0000 7fc1d3fff700 1 --2- 172.21.15.46:0/243260973 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1d415d8f0 0x7fc1d414c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.305+0000 7fc1d3fff700 1 -- 172.21.15.46:0/243260973 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1d4159590 msgr2=0x7fc1d40c78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:48.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.305+0000 7fc1d3fff700 1 --2- 172.21.15.46:0/243260973 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1d4159590 0x7fc1d40c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.305+0000 7fc1d3fff700 1 -- 172.21.15.46:0/243260973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c40094f0 con 0x7fc1d406c3c0 2024-01-27T01:03:48.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.305+0000 7fc1d3fff700 1 --2- 172.21.15.46:0/243260973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1d406c3c0 0x7fc1d4169b40 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4006fd0 tx=0x7fc1c4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:48.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.305+0000 7fc1bbfff700 1 -- 172.21.15.46:0/243260973 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c401f070 con 0x7fc1d406c3c0 2024-01-27T01:03:48.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.305+0000 7fc1bbfff700 1 -- 172.21.15.46:0/243260973 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c4002f30 con 0x7fc1d406c3c0 2024-01-27T01:03:48.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.306+0000 7fc1da788700 1 -- 172.21.15.46:0/243260973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d40c4530 con 0x7fc1d406c3c0 2024-01-27T01:03:48.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.306+0000 7fc1bbfff700 1 -- 172.21.15.46:0/243260973 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4011bc0 con 0x7fc1d406c3c0 2024-01-27T01:03:48.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.306+0000 7fc1da788700 1 -- 172.21.15.46:0/243260973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d40c1fa0 con 0x7fc1d406c3c0 2024-01-27T01:03:48.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.307+0000 7fc1da788700 1 -- 172.21.15.46:0/243260973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d404c710 con 0x7fc1d406c3c0 2024-01-27T01:03:48.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.307+0000 7fc1bbfff700 1 -- 172.21.15.46:0/243260973 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc1c4023070 con 0x7fc1d406c3c0 2024-01-27T01:03:48.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.308+0000 7fc1bbfff700 1 --2- 172.21.15.46:0/243260973 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc1c0066f30 0x7fc1c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:48.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.308+0000 7fc1bbfff700 1 -- 172.21.15.46:0/243260973 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fc1c4086200 con 0x7fc1d406c3c0 2024-01-27T01:03:48.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.308+0000 7fc1d37fe700 1 --2- 172.21.15.46:0/243260973 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc1c0066f30 0x7fc1c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:48.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.309+0000 7fc1d37fe700 1 --2- 172.21.15.46:0/243260973 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc1c0066f30 0x7fc1c00693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc1d415d540 tx=0x7fc1c8007b40 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:48.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.312+0000 7fc1bbfff700 1 -- 172.21.15.46:0/243260973 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc1c40531e0 con 0x7fc1d406c3c0 2024-01-27T01:03:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.518+0000 7fc1da788700 1 -- 172.21.15.46:0/243260973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1d4068b70 con 0x7fc1d406c3c0 2024-01-27T01:03:48.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.519+0000 7fc1bbfff700 1 -- 172.21.15.46:0/243260973 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v205) v1 ==== 74+0+16280 (secure 0 0 0) 0x7fc1c4015090 con 0x7fc1d406c3c0 2024-01-27T01:03:48.519 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:48.519 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":205,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:03:45.706493+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:03:42.648202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-01-27T01:03:43.106225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:03:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.521+0000 7fc1da788700 1 -- 172.21.15.46:0/243260973 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc1c0066f30 msgr2=0x7fc1c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.521+0000 7fc1da788700 1 --2- 172.21.15.46:0/243260973 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc1c0066f30 0x7fc1c00693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc1d415d540 tx=0x7fc1c8007b40 comp rx=0 tx=0).stop 2024-01-27T01:03:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.521+0000 7fc1da788700 1 -- 172.21.15.46:0/243260973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1d406c3c0 msgr2=0x7fc1d4169b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.521+0000 7fc1da788700 1 --2- 172.21.15.46:0/243260973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1d406c3c0 0x7fc1d4169b40 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4006fd0 tx=0x7fc1c4017000 comp rx=0 tx=0).stop 2024-01-27T01:03:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.522+0000 7fc1da788700 1 -- 172.21.15.46:0/243260973 shutdown_connections 2024-01-27T01:03:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.522+0000 7fc1da788700 1 --2- 172.21.15.46:0/243260973 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc1c0066f30 0x7fc1c00693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.522+0000 7fc1da788700 1 --2- 172.21.15.46:0/243260973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1d406c3c0 0x7fc1d4169b40 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.522+0000 7fc1da788700 1 --2- 172.21.15.46:0/243260973 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1d415d8f0 0x7fc1d414c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.522+0000 7fc1da788700 1 --2- 172.21.15.46:0/243260973 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1d4159590 0x7fc1d40c78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.522+0000 7fc1da788700 1 -- 172.21.15.46:0/243260973 >> 172.21.15.46:0/243260973 conn(0x7fc1d40ce530 msgr2=0x7fc1d4158920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:48.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.522+0000 7fc1da788700 1 -- 172.21.15.46:0/243260973 shutdown_connections 2024-01-27T01:03:48.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.522+0000 7fc1da788700 1 -- 172.21.15.46:0/243260973 wait complete. 2024-01-27T01:03:48.536 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-27T01:03:48.536 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:03:48.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.669+0000 7f2081474700 1 Processor -- start 2024-01-27T01:03:48.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.669+0000 7f2081474700 1 -- start start 2024-01-27T01:03:48.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f2081474700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f207c1596c0 0x7f207c06bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:48.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f2081474700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f207c15d460 0x7f207c06c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f2081474700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f207c159a90 0x7f207c06d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f2081474700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f207c0c98c0 con 0x7f207c15d460 2024-01-27T01:03:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f2081474700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f207c15d8f0 con 0x7f207c159a90 2024-01-27T01:03:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f2081474700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f207c15dbd0 con 0x7f207c1596c0 2024-01-27T01:03:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f207a7fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f207c15d460 0x7f207c06c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f207a7fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f207c15d460 0x7f207c06c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39080/0 (socket says 172.21.15.46:39080) 2024-01-27T01:03:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f207a7fc700 1 -- 172.21.15.46:0/3470083919 learned_addr learned my addr 172.21.15.46:0/3470083919 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f207b7fe700 1 --2- 172.21.15.46:0/3470083919 >> v2:172.21.15.106:3300/0 conn(0x7f207c159a90 0x7f207c06d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f207affd700 1 --2- 172.21.15.46:0/3470083919 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f207c1596c0 0x7f207c06bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f207a7fc700 1 -- 172.21.15.46:0/3470083919 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f207c1596c0 msgr2=0x7f207c06bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f207a7fc700 1 --2- 172.21.15.46:0/3470083919 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f207c1596c0 0x7f207c06bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f207a7fc700 1 -- 172.21.15.46:0/3470083919 >> v2:172.21.15.106:3300/0 conn(0x7f207c159a90 msgr2=0x7f207c06d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f207a7fc700 1 --2- 172.21.15.46:0/3470083919 >> v2:172.21.15.106:3300/0 conn(0x7f207c159a90 0x7f207c06d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f207a7fc700 1 -- 172.21.15.46:0/3470083919 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f207c15deb0 con 0x7f207c15d460 2024-01-27T01:03:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f207b7fe700 1 --2- 172.21.15.46:0/3470083919 >> v2:172.21.15.106:3300/0 conn(0x7f207c159a90 0x7f207c06d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:03:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f207a7fc700 1 --2- 172.21.15.46:0/3470083919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f207c15d460 0x7f207c06c210 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f207c0b0fe0 tx=0x7f2070009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=849e622b62077a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.670+0000 7f2079ffb700 1 -- 172.21.15.46:0/3470083919 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207000b860 con 0x7f207c15d460 2024-01-27T01:03:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.671+0000 7f2079ffb700 1 -- 172.21.15.46:0/3470083919 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207000b9c0 con 0x7f207c15d460 2024-01-27T01:03:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.671+0000 7f2079ffb700 1 -- 172.21.15.46:0/3470083919 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207000bd60 con 0x7f207c15d460 2024-01-27T01:03:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.671+0000 7f2081474700 1 -- 172.21.15.46:0/3470083919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f207c15d460 msgr2=0x7f207c06c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.671+0000 7f2081474700 1 --2- 172.21.15.46:0/3470083919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f207c15d460 0x7f207c06c210 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f207c0b0fe0 tx=0x7f2070009250 comp rx=0 tx=0).stop 2024-01-27T01:03:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.671+0000 7f2081474700 1 -- 172.21.15.46:0/3470083919 shutdown_connections 2024-01-27T01:03:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.671+0000 7f2081474700 1 --2- 172.21.15.46:0/3470083919 >> v2:172.21.15.106:3300/0 conn(0x7f207c159a90 0x7f207c06d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.671+0000 7f2081474700 1 --2- 172.21.15.46:0/3470083919 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f207c1596c0 0x7f207c06bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.671+0000 7f2081474700 1 --2- 172.21.15.46:0/3470083919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f207c15d460 0x7f207c06c210 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.671+0000 7f2081474700 1 -- 172.21.15.46:0/3470083919 >> 172.21.15.46:0/3470083919 conn(0x7f207c0ce530 msgr2=0x7f207c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.671+0000 7f2081474700 1 -- 172.21.15.46:0/3470083919 shutdown_connections 2024-01-27T01:03:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.671+0000 7f2081474700 1 -- 172.21.15.46:0/3470083919 wait complete. 2024-01-27T01:03:48.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.672+0000 7f2081474700 1 Processor -- start 2024-01-27T01:03:48.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.673+0000 7f2081474700 1 -- start start 2024-01-27T01:03:48.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.673+0000 7f2081474700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f207c1596c0 0x7f207c157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:48.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.673+0000 7f2081474700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f207c159a90 0x7f207c0c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:48.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.673+0000 7f2081474700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f207c15d460 0x7f207c0c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:48.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.673+0000 7f2081474700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207c0b3300 con 0x7f207c1596c0 2024-01-27T01:03:48.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.673+0000 7f2081474700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207c0b4110 con 0x7f207c15d460 2024-01-27T01:03:48.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.673+0000 7f2081474700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f207c0b3500 con 0x7f207c159a90 2024-01-27T01:03:48.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.673+0000 7f207affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f207c1596c0 0x7f207c157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:48.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.673+0000 7f207affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f207c1596c0 0x7f207c157c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39094/0 (socket says 172.21.15.46:39094) 2024-01-27T01:03:48.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.673+0000 7f207affd700 1 -- 172.21.15.46:0/795774397 learned_addr learned my addr 172.21.15.46:0/795774397 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:48.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.674+0000 7f207a7fc700 1 --2- 172.21.15.46:0/795774397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f207c159a90 0x7f207c0c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:48.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.674+0000 7f207b7fe700 1 --2- 172.21.15.46:0/795774397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f207c15d460 0x7f207c0c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:48.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.674+0000 7f207affd700 1 -- 172.21.15.46:0/795774397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f207c159a90 msgr2=0x7f207c0c5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:48.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.674+0000 7f207affd700 1 --2- 172.21.15.46:0/795774397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f207c159a90 0x7f207c0c5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.674+0000 7f207affd700 1 -- 172.21.15.46:0/795774397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f207c15d460 msgr2=0x7f207c0c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:48.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.674+0000 7f207affd700 1 --2- 172.21.15.46:0/795774397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f207c15d460 0x7f207c0c7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.674+0000 7f207affd700 1 -- 172.21.15.46:0/795774397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20700094f0 con 0x7f207c1596c0 2024-01-27T01:03:48.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.674+0000 7f207affd700 1 --2- 172.21.15.46:0/795774397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f207c1596c0 0x7f207c157c00 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f206c00d8c0 tx=0x7f206c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:48.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.674+0000 7f2062ffd700 1 -- 172.21.15.46:0/795774397 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206c0125f0 con 0x7f207c1596c0 2024-01-27T01:03:48.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.674+0000 7f2062ffd700 1 -- 172.21.15.46:0/795774397 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f206c012750 con 0x7f207c1596c0 2024-01-27T01:03:48.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.674+0000 7f2081474700 1 -- 172.21.15.46:0/795774397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f207c0b37e0 con 0x7f207c1596c0 2024-01-27T01:03:48.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.674+0000 7f2062ffd700 1 -- 172.21.15.46:0/795774397 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206c012af0 con 0x7f207c1596c0 2024-01-27T01:03:48.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.674+0000 7f2081474700 1 -- 172.21.15.46:0/795774397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f207c0b3e20 con 0x7f207c1596c0 2024-01-27T01:03:48.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.675+0000 7f2081474700 1 -- 172.21.15.46:0/795774397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f207c0aeff0 con 0x7f207c1596c0 2024-01-27T01:03:48.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.676+0000 7f2062ffd700 1 -- 172.21.15.46:0/795774397 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f206c01e460 con 0x7f207c1596c0 2024-01-27T01:03:48.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.676+0000 7f2062ffd700 1 --2- 172.21.15.46:0/795774397 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2068066f30 0x7f20680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:48.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.676+0000 7f2062ffd700 1 -- 172.21.15.46:0/795774397 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f206c084dc0 con 0x7f207c1596c0 2024-01-27T01:03:48.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.677+0000 7f207a7fc700 1 --2- 172.21.15.46:0/795774397 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2068066f30 0x7f20680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:48.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.677+0000 7f207a7fc700 1 --2- 172.21.15.46:0/795774397 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2068066f30 0x7f20680693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f207c0b0fe0 tx=0x7f2070017000 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:48.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.681+0000 7f2062ffd700 1 -- 172.21.15.46:0/795774397 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f206c051d20 con 0x7f207c1596c0 2024-01-27T01:03:48.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.882+0000 7f2081474700 1 -- 172.21.15.46:0/795774397 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f207c04c710 con 0x7f2068066f30 2024-01-27T01:03:48.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.891+0000 7f2062ffd700 1 -- 172.21.15.46:0/795774397 <== mgr.10182 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289686 (secure 0 0 0) 0x7f207c04c710 con 0x7f2068066f30 2024-01-27T01:03:48.896 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:48.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.894+0000 7f2081474700 1 -- 172.21.15.46:0/795774397 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2068066f30 msgr2=0x7f20680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:48.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.894+0000 7f2081474700 1 --2- 172.21.15.46:0/795774397 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2068066f30 0x7f20680693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f207c0b0fe0 tx=0x7f2070017000 comp rx=0 tx=0).stop 2024-01-27T01:03:48.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.894+0000 7f2081474700 1 -- 172.21.15.46:0/795774397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f207c1596c0 msgr2=0x7f207c157c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:48.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.894+0000 7f2081474700 1 --2- 172.21.15.46:0/795774397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f207c1596c0 0x7f207c157c00 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f206c00d8c0 tx=0x7f206c008040 comp rx=0 tx=0).stop 2024-01-27T01:03:48.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.895+0000 7f2081474700 1 -- 172.21.15.46:0/795774397 shutdown_connections 2024-01-27T01:03:48.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.895+0000 7f2081474700 1 --2- 172.21.15.46:0/795774397 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2068066f30 0x7f20680693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.895+0000 7f2081474700 1 --2- 172.21.15.46:0/795774397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f207c1596c0 0x7f207c157c00 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.895+0000 7f2081474700 1 --2- 172.21.15.46:0/795774397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f207c15d460 0x7f207c0c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.895+0000 7f2081474700 1 --2- 172.21.15.46:0/795774397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f207c159a90 0x7f207c0c5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:48.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.895+0000 7f2081474700 1 -- 172.21.15.46:0/795774397 >> 172.21.15.46:0/795774397 conn(0x7f207c0ce530 msgr2=0x7f207c15c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:48.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.895+0000 7f2081474700 1 -- 172.21.15.46:0/795774397 shutdown_connections 2024-01-27T01:03:48.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:48.895+0000 7f2081474700 1 -- 172.21.15.46:0/795774397 wait complete. 2024-01-27T01:03:48.899 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:03:48.908 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-01-27T01:03:47.756109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2072,"num_read_kb":4396,"num_write":634,"num_write_kb":3475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8766,"ondisk_log_size":8766,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2465148,"kb_used_data":21308,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752509572,"statfs":{"total":773094113280,"available":770569801728,"internally_reserved":0,"allocated":21819392,"data_stored":11957572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.065200"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.720972+0000","last_change":"2024-01-27T01:03:44.716770+0000","last_active":"2024-01-27T01:03:44.720972+0000","last_peered":"2024-01-27T01:03:44.720972+0000","last_clean":"2024-01-27T01:03:44.720972+0000","last_became_active":"2024-01-27T01:03:44.716316+0000","last_became_peered":"2024-01-27T01:03:44.716316+0000","last_unstale":"2024-01-27T01:03:44.720972+0000","last_undegraded":"2024-01-27T01:03:44.720972+0000","last_fullsized":"2024-01-27T01:03:44.720972+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:50:32.225758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.721285+0000","last_change":"2024-01-27T01:03:44.709818+0000","last_active":"2024-01-27T01:03:44.721285+0000","last_peered":"2024-01-27T01:03:44.721285+0000","last_clean":"2024-01-27T01:03:44.721285+0000","last_became_active":"2024-01-27T01:03:44.709330+0000","last_became_peered":"2024-01-27T01:03:44.709330+0000","last_unstale":"2024-01-27T01:03:44.721285+0000","last_undegraded":"2024-01-27T01:03:44.721285+0000","last_fullsized":"2024-01-27T01:03:44.721285+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:32:36.671785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.728735+0000","last_change":"2024-01-27T01:03:44.727679+0000","last_active":"2024-01-27T01:03:44.728735+0000","last_peered":"2024-01-27T01:03:44.728735+0000","last_clean":"2024-01-27T01:03:44.728735+0000","last_became_active":"2024-01-27T01:03:44.727407+0000","last_became_peered":"2024-01-27T01:03:44.727407+0000","last_unstale":"2024-01-27T01:03:44.728735+0000","last_undegraded":"2024-01-27T01:03:44.728735+0000","last_fullsized":"2024-01-27T01:03:44.728735+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:22:50.984272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.732987+0000","last_change":"2024-01-27T01:03:44.726023+0000","last_active":"2024-01-27T01:03:44.732987+0000","last_peered":"2024-01-27T01:03:44.732987+0000","last_clean":"2024-01-27T01:03:44.732987+0000","last_became_active":"2024-01-27T01:03:44.725684+0000","last_became_peered":"2024-01-27T01:03:44.725684+0000","last_unstale":"2024-01-27T01:03:44.732987+0000","last_undegraded":"2024-01-27T01:03:44.732987+0000","last_fullsized":"2024-01-27T01:03:44.732987+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:58:38.516519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.722400+0000","last_change":"2024-01-27T01:03:44.713015+0000","last_active":"2024-01-27T01:03:44.722400+0000","last_peered":"2024-01-27T01:03:44.722400+0000","last_clean":"2024-01-27T01:03:44.722400+0000","last_became_active":"2024-01-27T01:03:44.712092+0000","last_became_peered":"2024-01-27T01:03:44.712092+0000","last_unstale":"2024-01-27T01:03:44.722400+0000","last_undegraded":"2024-01-27T01:03:44.722400+0000","last_fullsized":"2024-01-27T01:03:44.722400+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:13.213228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.723478+0000","last_change":"2024-01-27T01:03:44.718409+0000","last_active":"2024-01-27T01:03:44.723478+0000","last_peered":"2024-01-27T01:03:44.723478+0000","last_clean":"2024-01-27T01:03:44.723478+0000","last_became_active":"2024-01-27T01:03:44.718084+0000","last_became_peered":"2024-01-27T01:03:44.718084+0000","last_unstale":"2024-01-27T01:03:44.723478+0000","last_undegraded":"2024-01-27T01:03:44.723478+0000","last_fullsized":"2024-01-27T01:03:44.723478+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:13:34.377333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.735734+0000","last_change":"2024-01-27T01:03:44.732216+0000","last_active":"2024-01-27T01:03:44.735734+0000","last_peered":"2024-01-27T01:03:44.735734+0000","last_clean":"2024-01-27T01:03:44.735734+0000","last_became_active":"2024-01-27T01:03:44.731911+0000","last_became_peered":"2024-01-27T01:03:44.731911+0000","last_unstale":"2024-01-27T01:03:44.735734+0000","last_undegraded":"2024-01-27T01:03:44.735734+0000","last_fullsized":"2024-01-27T01:03:44.735734+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:54:55.134131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.718838+0000","last_change":"2024-01-27T01:03:44.715232+0000","last_active":"2024-01-27T01:03:44.718838+0000","last_peered":"2024-01-27T01:03:44.718838+0000","last_clean":"2024-01-27T01:03:44.718838+0000","last_became_active":"2024-01-27T01:03:44.714947+0000","last_became_peered":"2024-01-27T01:03:44.714947+0000","last_unstale":"2024-01-27T01:03:44.718838+0000","last_undegraded":"2024-01-27T01:03:44.718838+0000","last_fullsized":"2024-01-27T01:03:44.718838+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:51:48.157181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.720719+0000","last_change":"2024-01-27T01:03:44.714156+0000","last_active":"2024-01-27T01:03:44.720719+0000","last_peered":"2024-01-27T01:03:44.720719+0000","last_clean":"2024-01-27T01:03:44.720719+0000","last_became_active":"2024-01-27T01:03:44.713824+0000","last_became_peered":"2024-01-27T01:03:44.713824+0000","last_unstale":"2024-01-27T01:03:44.720719+0000","last_undegraded":"2024-01-27T01:03:44.720719+0000","last_fullsized":"2024-01-27T01:03:44.720719+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:21:09.094003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.723019+0000","last_change":"2024-01-27T01:03:44.721167+0000","last_active":"2024-01-27T01:03:44.723019+0000","last_peered":"2024-01-27T01:03:44.723019+0000","last_clean":"2024-01-27T01:03:44.723019+0000","last_became_active":"2024-01-27T01:03:44.720888+0000","last_became_peered":"2024-01-27T01:03:44.720888+0000","last_unstale":"2024-01-27T01:03:44.723019+0000","last_undegraded":"2024-01-27T01:03:44.723019+0000","last_fullsized":"2024-01-27T01:03:44.723019+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:55:49.742669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.714010+0000","last_change":"2024-01-27T01:03:44.712099+0000","last_active":"2024-01-27T01:03:44.714010+0000","last_peered":"2024-01-27T01:03:44.714010+0000","last_clean":"2024-01-27T01:03:44.714010+0000","last_became_active":"2024-01-27T01:03:44.711831+0000","last_became_peered":"2024-01-27T01:03:44.711831+0000","last_unstale":"2024-01-27T01:03:44.714010+0000","last_undegraded":"2024-01-27T01:03:44.714010+0000","last_fullsized":"2024-01-27T01:03:44.714010+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:06.223570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.730397+0000","last_change":"2024-01-27T01:03:44.715834+0000","last_active":"2024-01-27T01:03:44.730397+0000","last_peered":"2024-01-27T01:03:44.730397+0000","last_clean":"2024-01-27T01:03:44.730397+0000","last_became_active":"2024-01-27T01:03:44.715512+0000","last_became_peered":"2024-01-27T01:03:44.715512+0000","last_unstale":"2024-01-27T01:03:44.730397+0000","last_undegraded":"2024-01-27T01:03:44.730397+0000","last_fullsized":"2024-01-27T01:03:44.730397+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:54:22.369203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.726719+0000","last_change":"2024-01-27T01:03:44.723710+0000","last_active":"2024-01-27T01:03:44.726719+0000","last_peered":"2024-01-27T01:03:44.726719+0000","last_clean":"2024-01-27T01:03:44.726719+0000","last_became_active":"2024-01-27T01:03:44.723234+0000","last_became_peered":"2024-01-27T01:03:44.723234+0000","last_unstale":"2024-01-27T01:03:44.726719+0000","last_undegraded":"2024-01-27T01:03:44.726719+0000","last_fullsized":"2024-01-27T01:03:44.726719+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:06.223570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.727197+0000","last_change":"2024-01-27T01:03:44.721866+0000","last_active":"2024-01-27T01:03:44.727197+0000","last_peered":"2024-01-27T01:03:44.727197+0000","last_clean":"2024-01-27T01:03:44.727197+0000","last_became_active":"2024-01-27T01:03:44.721513+0000","last_became_peered":"2024-01-27T01:03:44.721513+0000","last_unstale":"2024-01-27T01:03:44.727197+0000","last_undegraded":"2024-01-27T01:03:44.727197+0000","last_fullsized":"2024-01-27T01:03:44.727197+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:05:23.070236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.730155+0000","last_change":"2024-01-27T01:03:44.725309+0000","last_active":"2024-01-27T01:03:44.730155+0000","last_peered":"2024-01-27T01:03:44.730155+0000","last_clean":"2024-01-27T01:03:44.730155+0000","last_became_active":"2024-01-27T01:03:44.725034+0000","last_became_peered":"2024-01-27T01:03:44.725034+0000","last_unstale":"2024-01-27T01:03:44.730155+0000","last_undegraded":"2024-01-27T01:03:44.730155+0000","last_fullsized":"2024-01-27T01:03:44.730155+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:17:29.492503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.728169+0000","last_change":"2024-01-27T01:03:44.723851+0000","last_active":"2024-01-27T01:03:44.728169+0000","last_peered":"2024-01-27T01:03:44.728169+0000","last_clean":"2024-01-27T01:03:44.728169+0000","last_became_active":"2024-01-27T01:03:44.723316+0000","last_became_peered":"2024-01-27T01:03:44.723316+0000","last_unstale":"2024-01-27T01:03:44.728169+0000","last_undegraded":"2024-01-27T01:03:44.728169+0000","last_fullsized":"2024-01-27T01:03:44.728169+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:12:00.859544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.723162+0000","last_change":"2024-01-27T01:03:44.718612+0000","last_active":"2024-01-27T01:03:44.723162+0000","last_peered":"2024-01-27T01:03:44.723162+0000","last_clean":"2024-01-27T01:03:44.723162+0000","last_became_active":"2024-01-27T01:03:44.718264+0000","last_became_peered":"2024-01-27T01:03:44.718264+0000","last_unstale":"2024-01-27T01:03:44.723162+0000","last_undegraded":"2024-01-27T01:03:44.723162+0000","last_fullsized":"2024-01-27T01:03:44.723162+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:49:32.367602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.726273+0000","last_change":"2024-01-27T01:03:44.720922+0000","last_active":"2024-01-27T01:03:44.726273+0000","last_peered":"2024-01-27T01:03:44.726273+0000","last_clean":"2024-01-27T01:03:44.726273+0000","last_became_active":"2024-01-27T01:03:44.720639+0000","last_became_peered":"2024-01-27T01:03:44.720639+0000","last_unstale":"2024-01-27T01:03:44.726273+0000","last_undegraded":"2024-01-27T01:03:44.726273+0000","last_fullsized":"2024-01-27T01:03:44.726273+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:32:53.294651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.733389+0000","last_change":"2024-01-27T01:03:44.732930+0000","last_active":"2024-01-27T01:03:44.733389+0000","last_peered":"2024-01-27T01:03:44.733389+0000","last_clean":"2024-01-27T01:03:44.733389+0000","last_became_active":"2024-01-27T01:03:44.732629+0000","last_became_peered":"2024-01-27T01:03:44.732629+0000","last_unstale":"2024-01-27T01:03:44.733389+0000","last_undegraded":"2024-01-27T01:03:44.733389+0000","last_fullsized":"2024-01-27T01:03:44.733389+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:22:50.984272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.722684+0000","last_change":"2024-01-27T01:03:44.712507+0000","last_active":"2024-01-27T01:03:44.722684+0000","last_peered":"2024-01-27T01:03:44.722684+0000","last_clean":"2024-01-27T01:03:44.722684+0000","last_became_active":"2024-01-27T01:03:44.712179+0000","last_became_peered":"2024-01-27T01:03:44.712179+0000","last_unstale":"2024-01-27T01:03:44.722684+0000","last_undegraded":"2024-01-27T01:03:44.722684+0000","last_fullsized":"2024-01-27T01:03:44.722684+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:37:07.278285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.720542+0000","last_change":"2024-01-27T01:03:44.714127+0000","last_active":"2024-01-27T01:03:44.720542+0000","last_peered":"2024-01-27T01:03:44.720542+0000","last_clean":"2024-01-27T01:03:44.720542+0000","last_became_active":"2024-01-27T01:03:44.713797+0000","last_became_peered":"2024-01-27T01:03:44.713797+0000","last_unstale":"2024-01-27T01:03:44.720542+0000","last_undegraded":"2024-01-27T01:03:44.720542+0000","last_fullsized":"2024-01-27T01:03:44.720542+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:28:50.109807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.719504+0000","last_change":"2024-01-27T01:03:44.714072+0000","last_active":"2024-01-27T01:03:44.719504+0000","last_peered":"2024-01-27T01:03:44.719504+0000","last_clean":"2024-01-27T01:03:44.719504+0000","last_became_active":"2024-01-27T01:03:44.713785+0000","last_became_peered":"2024-01-27T01:03:44.713785+0000","last_unstale":"2024-01-27T01:03:44.719504+0000","last_undegraded":"2024-01-27T01:03:44.719504+0000","last_fullsized":"2024-01-27T01:03:44.719504+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:41:01.705159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.718934+0000","last_change":"2024-01-27T01:03:44.716659+0000","last_active":"2024-01-27T01:03:44.718934+0000","last_peered":"2024-01-27T01:03:44.718934+0000","last_clean":"2024-01-27T01:03:44.718934+0000","last_became_active":"2024-01-27T01:03:44.716336+0000","last_became_peered":"2024-01-27T01:03:44.716336+0000","last_unstale":"2024-01-27T01:03:44.718934+0000","last_undegraded":"2024-01-27T01:03:44.718934+0000","last_fullsized":"2024-01-27T01:03:44.718934+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:57:26.315979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.720852+0000","last_change":"2024-01-27T01:03:44.715662+0000","last_active":"2024-01-27T01:03:44.720852+0000","last_peered":"2024-01-27T01:03:44.720852+0000","last_clean":"2024-01-27T01:03:44.720852+0000","last_became_active":"2024-01-27T01:03:44.715373+0000","last_became_peered":"2024-01-27T01:03:44.715373+0000","last_unstale":"2024-01-27T01:03:44.720852+0000","last_undegraded":"2024-01-27T01:03:44.720852+0000","last_fullsized":"2024-01-27T01:03:44.720852+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:39:26.398910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.735207+0000","last_change":"2024-01-27T01:03:44.731607+0000","last_active":"2024-01-27T01:03:44.735207+0000","last_peered":"2024-01-27T01:03:44.735207+0000","last_clean":"2024-01-27T01:03:44.735207+0000","last_became_active":"2024-01-27T01:03:44.727541+0000","last_became_peered":"2024-01-27T01:03:44.727541+0000","last_unstale":"2024-01-27T01:03:44.735207+0000","last_undegraded":"2024-01-27T01:03:44.735207+0000","last_fullsized":"2024-01-27T01:03:44.735207+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:54.792398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.717439+0000","last_change":"2024-01-27T01:03:44.713245+0000","last_active":"2024-01-27T01:03:44.717439+0000","last_peered":"2024-01-27T01:03:44.717439+0000","last_clean":"2024-01-27T01:03:44.717439+0000","last_became_active":"2024-01-27T01:03:44.712817+0000","last_became_peered":"2024-01-27T01:03:44.712817+0000","last_unstale":"2024-01-27T01:03:44.717439+0000","last_undegraded":"2024-01-27T01:03:44.717439+0000","last_fullsized":"2024-01-27T01:03:44.717439+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:19:51.053608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.729475+0000","last_change":"2024-01-27T01:03:44.727971+0000","last_active":"2024-01-27T01:03:44.729475+0000","last_peered":"2024-01-27T01:03:44.729475+0000","last_clean":"2024-01-27T01:03:44.729475+0000","last_became_active":"2024-01-27T01:03:44.727672+0000","last_became_peered":"2024-01-27T01:03:44.727672+0000","last_unstale":"2024-01-27T01:03:44.729475+0000","last_undegraded":"2024-01-27T01:03:44.729475+0000","last_fullsized":"2024-01-27T01:03:44.729475+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:29:16.068435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.722706+0000","last_change":"2024-01-27T01:03:44.712870+0000","last_active":"2024-01-27T01:03:44.722706+0000","last_peered":"2024-01-27T01:03:44.722706+0000","last_clean":"2024-01-27T01:03:44.722706+0000","last_became_active":"2024-01-27T01:03:44.712106+0000","last_became_peered":"2024-01-27T01:03:44.712106+0000","last_unstale":"2024-01-27T01:03:44.722706+0000","last_undegraded":"2024-01-27T01:03:44.722706+0000","last_fullsized":"2024-01-27T01:03:44.722706+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:53:40.835858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.722144+0000","last_change":"2024-01-27T01:03:44.720569+0000","last_active":"2024-01-27T01:03:44.722144+0000","last_peered":"2024-01-27T01:03:44.722144+0000","last_clean":"2024-01-27T01:03:44.722144+0000","last_became_active":"2024-01-27T01:03:44.720292+0000","last_became_peered":"2024-01-27T01:03:44.720292+0000","last_unstale":"2024-01-27T01:03:44.722144+0000","last_undegraded":"2024-01-27T01:03:44.722144+0000","last_fullsized":"2024-01-27T01:03:44.722144+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:22:50.984272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.717829+0000","last_change":"2024-01-27T01:03:44.714245+0000","last_active":"2024-01-27T01:03:44.717829+0000","last_peered":"2024-01-27T01:03:44.717829+0000","last_clean":"2024-01-27T01:03:44.717829+0000","last_became_active":"2024-01-27T01:03:44.714021+0000","last_became_peered":"2024-01-27T01:03:44.714021+0000","last_unstale":"2024-01-27T01:03:44.717829+0000","last_undegraded":"2024-01-27T01:03:44.717829+0000","last_fullsized":"2024-01-27T01:03:44.717829+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:19:33.472237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.735212+0000","last_change":"2024-01-27T01:03:44.731737+0000","last_active":"2024-01-27T01:03:44.735212+0000","last_peered":"2024-01-27T01:03:44.735212+0000","last_clean":"2024-01-27T01:03:44.735212+0000","last_became_active":"2024-01-27T01:03:44.731430+0000","last_became_peered":"2024-01-27T01:03:44.731430+0000","last_unstale":"2024-01-27T01:03:44.735212+0000","last_undegraded":"2024-01-27T01:03:44.735212+0000","last_fullsized":"2024-01-27T01:03:44.735212+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:39:55.734791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.734232+0000","last_change":"2024-01-27T01:03:44.728456+0000","last_active":"2024-01-27T01:03:44.734232+0000","last_peered":"2024-01-27T01:03:44.734232+0000","last_clean":"2024-01-27T01:03:44.734232+0000","last_became_active":"2024-01-27T01:03:44.727965+0000","last_became_peered":"2024-01-27T01:03:44.727965+0000","last_unstale":"2024-01-27T01:03:44.734232+0000","last_undegraded":"2024-01-27T01:03:44.734232+0000","last_fullsized":"2024-01-27T01:03:44.734232+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:06:05.222812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.723605+0000","last_change":"2024-01-27T01:03:44.722503+0000","last_active":"2024-01-27T01:03:44.723605+0000","last_peered":"2024-01-27T01:03:44.723605+0000","last_clean":"2024-01-27T01:03:44.723605+0000","last_became_active":"2024-01-27T01:03:44.722230+0000","last_became_peered":"2024-01-27T01:03:44.722230+0000","last_unstale":"2024-01-27T01:03:44.723605+0000","last_undegraded":"2024-01-27T01:03:44.723605+0000","last_fullsized":"2024-01-27T01:03:44.723605+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:51:19.304072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"204'4","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.736213+0000","last_change":"2024-01-27T01:03:43.715832+0000","last_active":"2024-01-27T01:03:44.736213+0000","last_peered":"2024-01-27T01:03:44.736213+0000","last_clean":"2024-01-27T01:03:44.736213+0000","last_became_active":"2024-01-27T01:03:43.715526+0000","last_became_peered":"2024-01-27T01:03:43.715526+0000","last_unstale":"2024-01-27T01:03:44.736213+0000","last_undegraded":"2024-01-27T01:03:44.736213+0000","last_fullsized":"2024-01-27T01:03:44.736213+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:27:19.173864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.735757+0000","last_change":"2024-01-27T01:03:44.732480+0000","last_active":"2024-01-27T01:03:44.735757+0000","last_peered":"2024-01-27T01:03:44.735757+0000","last_clean":"2024-01-27T01:03:44.735757+0000","last_became_active":"2024-01-27T01:03:44.727517+0000","last_became_peered":"2024-01-27T01:03:44.727517+0000","last_unstale":"2024-01-27T01:03:44.735757+0000","last_undegraded":"2024-01-27T01:03:44.735757+0000","last_fullsized":"2024-01-27T01:03:44.735757+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:00:43.760382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"204'1","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.730832+0000","last_change":"2024-01-27T01:03:43.723182+0000","last_active":"2024-01-27T01:03:44.730832+0000","last_peered":"2024-01-27T01:03:44.730832+0000","last_clean":"2024-01-27T01:03:44.730832+0000","last_became_active":"2024-01-27T01:03:43.720060+0000","last_became_peered":"2024-01-27T01:03:43.720060+0000","last_unstale":"2024-01-27T01:03:44.730832+0000","last_undegraded":"2024-01-27T01:03:44.730832+0000","last_fullsized":"2024-01-27T01:03:44.730832+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:17:28.487577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.718159+0000","last_change":"2024-01-27T01:03:44.715360+0000","last_active":"2024-01-27T01:03:44.718159+0000","last_peered":"2024-01-27T01:03:44.718159+0000","last_clean":"2024-01-27T01:03:44.718159+0000","last_became_active":"2024-01-27T01:03:44.715129+0000","last_became_peered":"2024-01-27T01:03:44.715129+0000","last_unstale":"2024-01-27T01:03:44.718159+0000","last_undegraded":"2024-01-27T01:03:44.718159+0000","last_fullsized":"2024-01-27T01:03:44.718159+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:03:07.526894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"204'2","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.729986+0000","last_change":"2024-01-27T01:03:43.716856+0000","last_active":"2024-01-27T01:03:44.729986+0000","last_peered":"2024-01-27T01:03:44.729986+0000","last_clean":"2024-01-27T01:03:44.729986+0000","last_became_active":"2024-01-27T01:03:43.716162+0000","last_became_peered":"2024-01-27T01:03:43.716162+0000","last_unstale":"2024-01-27T01:03:44.729986+0000","last_undegraded":"2024-01-27T01:03:44.729986+0000","last_fullsized":"2024-01-27T01:03:44.729986+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:13:25.722707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.713618+0000","last_change":"2024-01-27T01:03:44.712278+0000","last_active":"2024-01-27T01:03:44.713618+0000","last_peered":"2024-01-27T01:03:44.713618+0000","last_clean":"2024-01-27T01:03:44.713618+0000","last_became_active":"2024-01-27T01:03:44.711998+0000","last_became_peered":"2024-01-27T01:03:44.711998+0000","last_unstale":"2024-01-27T01:03:44.713618+0000","last_undegraded":"2024-01-27T01:03:44.713618+0000","last_fullsized":"2024-01-27T01:03:44.713618+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:10:33.418151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"204'2","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.734297+0000","last_change":"2024-01-27T01:03:43.717831+0000","last_active":"2024-01-27T01:03:44.734297+0000","last_peered":"2024-01-27T01:03:44.734297+0000","last_clean":"2024-01-27T01:03:44.734297+0000","last_became_active":"2024-01-27T01:03:43.717581+0000","last_became_peered":"2024-01-27T01:03:43.717581+0000","last_unstale":"2024-01-27T01:03:44.734297+0000","last_undegraded":"2024-01-27T01:03:44.734297+0000","last_fullsized":"2024-01-27T01:03:44.734297+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:58:04.065236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.730053+0000","last_change":"2024-01-27T01:03:44.729189+0000","last_active":"2024-01-27T01:03:44.730053+0000","last_peered":"2024-01-27T01:03:44.730053+0000","last_clean":"2024-01-27T01:03:44.730053+0000","last_became_active":"2024-01-27T01:03:44.728899+0000","last_became_peered":"2024-01-27T01:03:44.728899+0000","last_unstale":"2024-01-27T01:03:44.730053+0000","last_undegraded":"2024-01-27T01:03:44.730053+0000","last_fullsized":"2024-01-27T01:03:44.730053+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:19:51.053608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"204'1","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.730861+0000","last_change":"2024-01-27T01:03:43.723249+0000","last_active":"2024-01-27T01:03:44.730861+0000","last_peered":"2024-01-27T01:03:44.730861+0000","last_clean":"2024-01-27T01:03:44.730861+0000","last_became_active":"2024-01-27T01:03:43.722901+0000","last_became_peered":"2024-01-27T01:03:43.722901+0000","last_unstale":"2024-01-27T01:03:44.730861+0000","last_undegraded":"2024-01-27T01:03:44.730861+0000","last_fullsized":"2024-01-27T01:03:44.730861+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:40:44.043969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.730134+0000","last_change":"2024-01-27T01:03:44.724625+0000","last_active":"2024-01-27T01:03:44.730134+0000","last_peered":"2024-01-27T01:03:44.730134+0000","last_clean":"2024-01-27T01:03:44.730134+0000","last_became_active":"2024-01-27T01:03:44.724302+0000","last_became_peered":"2024-01-27T01:03:44.724302+0000","last_unstale":"2024-01-27T01:03:44.730134+0000","last_undegraded":"2024-01-27T01:03:44.730134+0000","last_fullsized":"2024-01-27T01:03:44.730134+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:11:26.057436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"204'1","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.732880+0000","last_change":"2024-01-27T01:03:43.720619+0000","last_active":"2024-01-27T01:03:44.732880+0000","last_peered":"2024-01-27T01:03:44.732880+0000","last_clean":"2024-01-27T01:03:44.732880+0000","last_became_active":"2024-01-27T01:03:43.720318+0000","last_became_peered":"2024-01-27T01:03:43.720318+0000","last_unstale":"2024-01-27T01:03:44.732880+0000","last_undegraded":"2024-01-27T01:03:44.732880+0000","last_fullsized":"2024-01-27T01:03:44.732880+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:14:24.287472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.728926+0000","last_change":"2024-01-27T01:03:44.721762+0000","last_active":"2024-01-27T01:03:44.728926+0000","last_peered":"2024-01-27T01:03:44.728926+0000","last_clean":"2024-01-27T01:03:44.728926+0000","last_became_active":"2024-01-27T01:03:44.721273+0000","last_became_peered":"2024-01-27T01:03:44.721273+0000","last_unstale":"2024-01-27T01:03:44.728926+0000","last_undegraded":"2024-01-27T01:03:44.728926+0000","last_fullsized":"2024-01-27T01:03:44.728926+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:47:07.080276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.712402+0000","last_change":"2024-01-27T01:03:43.716959+0000","last_active":"2024-01-27T01:03:44.712402+0000","last_peered":"2024-01-27T01:03:44.712402+0000","last_clean":"2024-01-27T01:03:44.712402+0000","last_became_active":"2024-01-27T01:03:43.716750+0000","last_became_peered":"2024-01-27T01:03:43.716750+0000","last_unstale":"2024-01-27T01:03:44.712402+0000","last_undegraded":"2024-01-27T01:03:44.712402+0000","last_fullsized":"2024-01-27T01:03:44.712402+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:41:39.450463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.725870+0000","last_change":"2024-01-27T01:03:44.725313+0000","last_active":"2024-01-27T01:03:44.725870+0000","last_peered":"2024-01-27T01:03:44.725870+0000","last_clean":"2024-01-27T01:03:44.725870+0000","last_became_active":"2024-01-27T01:03:44.724942+0000","last_became_peered":"2024-01-27T01:03:44.724942+0000","last_unstale":"2024-01-27T01:03:44.725870+0000","last_undegraded":"2024-01-27T01:03:44.725870+0000","last_fullsized":"2024-01-27T01:03:44.725870+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:34:28.730691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.706505+0000","last_change":"2024-01-27T01:03:43.719422+0000","last_active":"2024-01-27T01:03:44.706505+0000","last_peered":"2024-01-27T01:03:44.706505+0000","last_clean":"2024-01-27T01:03:44.706505+0000","last_became_active":"2024-01-27T01:03:43.719138+0000","last_became_peered":"2024-01-27T01:03:43.719138+0000","last_unstale":"2024-01-27T01:03:44.706505+0000","last_undegraded":"2024-01-27T01:03:44.706505+0000","last_fullsized":"2024-01-27T01:03:44.706505+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:54:29.307945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.725203+0000","last_change":"2024-01-27T01:03:44.712901+0000","last_active":"2024-01-27T01:03:44.725203+0000","last_peered":"2024-01-27T01:03:44.725203+0000","last_clean":"2024-01-27T01:03:44.725203+0000","last_became_active":"2024-01-27T01:03:44.712614+0000","last_became_peered":"2024-01-27T01:03:44.712614+0000","last_unstale":"2024-01-27T01:03:44.725203+0000","last_undegraded":"2024-01-27T01:03:44.725203+0000","last_fullsized":"2024-01-27T01:03:44.725203+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:37:57.304264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.701928+0000","last_change":"2024-01-27T01:03:43.725915+0000","last_active":"2024-01-27T01:03:44.701928+0000","last_peered":"2024-01-27T01:03:44.701928+0000","last_clean":"2024-01-27T01:03:44.701928+0000","last_became_active":"2024-01-27T01:03:43.725624+0000","last_became_peered":"2024-01-27T01:03:43.725624+0000","last_unstale":"2024-01-27T01:03:44.701928+0000","last_undegraded":"2024-01-27T01:03:44.701928+0000","last_fullsized":"2024-01-27T01:03:44.701928+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:03:06.521968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.726293+0000","last_change":"2024-01-27T01:03:44.722018+0000","last_active":"2024-01-27T01:03:44.726293+0000","last_peered":"2024-01-27T01:03:44.726293+0000","last_clean":"2024-01-27T01:03:44.726293+0000","last_became_active":"2024-01-27T01:03:44.721602+0000","last_became_peered":"2024-01-27T01:03:44.721602+0000","last_unstale":"2024-01-27T01:03:44.726293+0000","last_undegraded":"2024-01-27T01:03:44.726293+0000","last_fullsized":"2024-01-27T01:03:44.726293+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:03:07.526894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"204'3","reported_seq":13,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.733551+0000","last_change":"2024-01-27T01:03:43.724110+0000","last_active":"2024-01-27T01:03:44.733551+0000","last_peered":"2024-01-27T01:03:44.733551+0000","last_clean":"2024-01-27T01:03:44.733551+0000","last_became_active":"2024-01-27T01:03:43.723813+0000","last_became_peered":"2024-01-27T01:03:43.723813+0000","last_unstale":"2024-01-27T01:03:44.733551+0000","last_undegraded":"2024-01-27T01:03:44.733551+0000","last_fullsized":"2024-01-27T01:03:44.733551+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:57:13.729066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.720951+0000","last_change":"2024-01-27T01:03:44.716587+0000","last_active":"2024-01-27T01:03:44.720951+0000","last_peered":"2024-01-27T01:03:44.720951+0000","last_clean":"2024-01-27T01:03:44.720951+0000","last_became_active":"2024-01-27T01:03:44.715931+0000","last_became_peered":"2024-01-27T01:03:44.715931+0000","last_unstale":"2024-01-27T01:03:44.720951+0000","last_undegraded":"2024-01-27T01:03:44.720951+0000","last_fullsized":"2024-01-27T01:03:44.720951+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:40:45.048895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"204'6","reported_seq":16,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.735434+0000","last_change":"2024-01-27T01:03:43.717696+0000","last_active":"2024-01-27T01:03:44.735434+0000","last_peered":"2024-01-27T01:03:44.735434+0000","last_clean":"2024-01-27T01:03:44.735434+0000","last_became_active":"2024-01-27T01:03:43.717449+0000","last_became_peered":"2024-01-27T01:03:43.717449+0000","last_unstale":"2024-01-27T01:03:44.735434+0000","last_undegraded":"2024-01-27T01:03:44.735434+0000","last_fullsized":"2024-01-27T01:03:44.735434+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:23.259629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.727234+0000","last_change":"2024-01-27T01:03:44.718112+0000","last_active":"2024-01-27T01:03:44.727234+0000","last_peered":"2024-01-27T01:03:44.727234+0000","last_clean":"2024-01-27T01:03:44.727234+0000","last_became_active":"2024-01-27T01:03:44.717066+0000","last_became_peered":"2024-01-27T01:03:44.717066+0000","last_unstale":"2024-01-27T01:03:44.727234+0000","last_undegraded":"2024-01-27T01:03:44.727234+0000","last_fullsized":"2024-01-27T01:03:44.727234+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:33:29.458783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"204'2","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.733415+0000","last_change":"2024-01-27T01:03:43.716408+0000","last_active":"2024-01-27T01:03:44.733415+0000","last_peered":"2024-01-27T01:03:44.733415+0000","last_clean":"2024-01-27T01:03:44.733415+0000","last_became_active":"2024-01-27T01:03:43.716151+0000","last_became_peered":"2024-01-27T01:03:43.716151+0000","last_unstale":"2024-01-27T01:03:44.733415+0000","last_undegraded":"2024-01-27T01:03:44.733415+0000","last_fullsized":"2024-01-27T01:03:44.733415+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:32.872628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.722357+0000","last_change":"2024-01-27T01:03:44.712162+0000","last_active":"2024-01-27T01:03:44.722357+0000","last_peered":"2024-01-27T01:03:44.722357+0000","last_clean":"2024-01-27T01:03:44.722357+0000","last_became_active":"2024-01-27T01:03:44.711786+0000","last_became_peered":"2024-01-27T01:03:44.711786+0000","last_unstale":"2024-01-27T01:03:44.722357+0000","last_undegraded":"2024-01-27T01:03:44.722357+0000","last_fullsized":"2024-01-27T01:03:44.722357+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:38:48.821118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.702448+0000","last_change":"2024-01-27T01:03:43.725752+0000","last_active":"2024-01-27T01:03:44.702448+0000","last_peered":"2024-01-27T01:03:44.702448+0000","last_clean":"2024-01-27T01:03:44.702448+0000","last_became_active":"2024-01-27T01:03:43.719772+0000","last_became_peered":"2024-01-27T01:03:43.719772+0000","last_unstale":"2024-01-27T01:03:44.702448+0000","last_undegraded":"2024-01-27T01:03:44.702448+0000","last_fullsized":"2024-01-27T01:03:44.702448+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:22:49.979346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.732178+0000","last_change":"2024-01-27T01:03:44.731427+0000","last_active":"2024-01-27T01:03:44.732178+0000","last_peered":"2024-01-27T01:03:44.732178+0000","last_clean":"2024-01-27T01:03:44.732178+0000","last_became_active":"2024-01-27T01:03:44.731128+0000","last_became_peered":"2024-01-27T01:03:44.731128+0000","last_unstale":"2024-01-27T01:03:44.732178+0000","last_undegraded":"2024-01-27T01:03:44.732178+0000","last_fullsized":"2024-01-27T01:03:44.732178+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:19:33.472237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.707093+0000","last_change":"2024-01-27T01:03:43.715244+0000","last_active":"2024-01-27T01:03:44.707093+0000","last_peered":"2024-01-27T01:03:44.707093+0000","last_clean":"2024-01-27T01:03:44.707093+0000","last_became_active":"2024-01-27T01:03:43.714956+0000","last_became_peered":"2024-01-27T01:03:43.714956+0000","last_unstale":"2024-01-27T01:03:44.707093+0000","last_undegraded":"2024-01-27T01:03:44.707093+0000","last_fullsized":"2024-01-27T01:03:44.707093+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:18:05.795670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.729831+0000","last_change":"2024-01-27T01:03:44.726440+0000","last_active":"2024-01-27T01:03:44.729831+0000","last_peered":"2024-01-27T01:03:44.729831+0000","last_clean":"2024-01-27T01:03:44.729831+0000","last_became_active":"2024-01-27T01:03:44.724748+0000","last_became_peered":"2024-01-27T01:03:44.724748+0000","last_unstale":"2024-01-27T01:03:44.729831+0000","last_undegraded":"2024-01-27T01:03:44.729831+0000","last_fullsized":"2024-01-27T01:03:44.729831+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:22.370885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"204'8","reported_seq":18,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.734892+0000","last_change":"2024-01-27T01:03:43.712094+0000","last_active":"2024-01-27T01:03:44.734892+0000","last_peered":"2024-01-27T01:03:44.734892+0000","last_clean":"2024-01-27T01:03:44.734892+0000","last_became_active":"2024-01-27T01:03:43.711811+0000","last_became_peered":"2024-01-27T01:03:43.711811+0000","last_unstale":"2024-01-27T01:03:44.734892+0000","last_undegraded":"2024-01-27T01:03:44.734892+0000","last_fullsized":"2024-01-27T01:03:44.734892+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:17:42.427577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.735646+0000","last_change":"2024-01-27T01:03:44.731801+0000","last_active":"2024-01-27T01:03:44.735646+0000","last_peered":"2024-01-27T01:03:44.735646+0000","last_clean":"2024-01-27T01:03:44.735646+0000","last_became_active":"2024-01-27T01:03:44.731183+0000","last_became_peered":"2024-01-27T01:03:44.731183+0000","last_unstale":"2024-01-27T01:03:44.735646+0000","last_undegraded":"2024-01-27T01:03:44.735646+0000","last_fullsized":"2024-01-27T01:03:44.735646+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:19.929140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"204'1","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.733460+0000","last_change":"2024-01-27T01:03:43.713381+0000","last_active":"2024-01-27T01:03:44.733460+0000","last_peered":"2024-01-27T01:03:44.733460+0000","last_clean":"2024-01-27T01:03:44.733460+0000","last_became_active":"2024-01-27T01:03:43.713110+0000","last_became_peered":"2024-01-27T01:03:43.713110+0000","last_unstale":"2024-01-27T01:03:44.733460+0000","last_undegraded":"2024-01-27T01:03:44.733460+0000","last_fullsized":"2024-01-27T01:03:44.733460+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:15:54.281492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.729204+0000","last_change":"2024-01-27T01:03:44.726121+0000","last_active":"2024-01-27T01:03:44.729204+0000","last_peered":"2024-01-27T01:03:44.729204+0000","last_clean":"2024-01-27T01:03:44.729204+0000","last_became_active":"2024-01-27T01:03:44.725028+0000","last_became_peered":"2024-01-27T01:03:44.725028+0000","last_unstale":"2024-01-27T01:03:44.729204+0000","last_undegraded":"2024-01-27T01:03:44.729204+0000","last_fullsized":"2024-01-27T01:03:44.729204+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:41:15.819900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"204'2","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.732909+0000","last_change":"2024-01-27T01:03:43.718894+0000","last_active":"2024-01-27T01:03:44.732909+0000","last_peered":"2024-01-27T01:03:44.732909+0000","last_clean":"2024-01-27T01:03:44.732909+0000","last_became_active":"2024-01-27T01:03:43.718609+0000","last_became_peered":"2024-01-27T01:03:43.718609+0000","last_unstale":"2024-01-27T01:03:44.732909+0000","last_undegraded":"2024-01-27T01:03:44.732909+0000","last_fullsized":"2024-01-27T01:03:44.732909+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:51:25.645321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.717415+0000","last_change":"2024-01-27T01:03:44.713061+0000","last_active":"2024-01-27T01:03:44.717415+0000","last_peered":"2024-01-27T01:03:44.717415+0000","last_clean":"2024-01-27T01:03:44.717415+0000","last_became_active":"2024-01-27T01:03:44.712508+0000","last_became_peered":"2024-01-27T01:03:44.712508+0000","last_unstale":"2024-01-27T01:03:44.717415+0000","last_undegraded":"2024-01-27T01:03:44.717415+0000","last_fullsized":"2024-01-27T01:03:44.717415+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:32:53.294651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"204'1","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.729629+0000","last_change":"2024-01-27T01:03:43.723623+0000","last_active":"2024-01-27T01:03:44.729629+0000","last_peered":"2024-01-27T01:03:44.729629+0000","last_clean":"2024-01-27T01:03:44.729629+0000","last_became_active":"2024-01-27T01:03:43.723348+0000","last_became_peered":"2024-01-27T01:03:43.723348+0000","last_unstale":"2024-01-27T01:03:44.729629+0000","last_undegraded":"2024-01-27T01:03:44.729629+0000","last_fullsized":"2024-01-27T01:03:44.729629+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:17:42.427577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.730073+0000","last_change":"2024-01-27T01:03:44.724054+0000","last_active":"2024-01-27T01:03:44.730073+0000","last_peered":"2024-01-27T01:03:44.730073+0000","last_clean":"2024-01-27T01:03:44.730073+0000","last_became_active":"2024-01-27T01:03:44.723771+0000","last_became_peered":"2024-01-27T01:03:44.723771+0000","last_unstale":"2024-01-27T01:03:44.730073+0000","last_undegraded":"2024-01-27T01:03:44.730073+0000","last_fullsized":"2024-01-27T01:03:44.730073+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:28.400436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.702278+0000","last_change":"2024-01-27T01:03:43.716871+0000","last_active":"2024-01-27T01:03:44.702278+0000","last_peered":"2024-01-27T01:03:44.702278+0000","last_clean":"2024-01-27T01:03:44.702278+0000","last_became_active":"2024-01-27T01:03:43.716462+0000","last_became_peered":"2024-01-27T01:03:43.716462+0000","last_unstale":"2024-01-27T01:03:44.702278+0000","last_undegraded":"2024-01-27T01:03:44.702278+0000","last_fullsized":"2024-01-27T01:03:44.702278+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:23:07.323492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.729184+0000","last_change":"2024-01-27T01:03:44.722215+0000","last_active":"2024-01-27T01:03:44.729184+0000","last_peered":"2024-01-27T01:03:44.729184+0000","last_clean":"2024-01-27T01:03:44.729184+0000","last_became_active":"2024-01-27T01:03:44.721927+0000","last_became_peered":"2024-01-27T01:03:44.721927+0000","last_unstale":"2024-01-27T01:03:44.729184+0000","last_undegraded":"2024-01-27T01:03:44.729184+0000","last_fullsized":"2024-01-27T01:03:44.729184+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:54:30.312872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"204'2","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.734264+0000","last_change":"2024-01-27T01:03:43.725099+0000","last_active":"2024-01-27T01:03:44.734264+0000","last_peered":"2024-01-27T01:03:44.734264+0000","last_clean":"2024-01-27T01:03:44.734264+0000","last_became_active":"2024-01-27T01:03:43.724812+0000","last_became_peered":"2024-01-27T01:03:43.724812+0000","last_unstale":"2024-01-27T01:03:44.734264+0000","last_undegraded":"2024-01-27T01:03:44.734264+0000","last_fullsized":"2024-01-27T01:03:44.734264+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:32:31.472856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.733234+0000","last_change":"2024-01-27T01:03:44.725409+0000","last_active":"2024-01-27T01:03:44.733234+0000","last_peered":"2024-01-27T01:03:44.733234+0000","last_clean":"2024-01-27T01:03:44.733234+0000","last_became_active":"2024-01-27T01:03:44.725120+0000","last_became_peered":"2024-01-27T01:03:44.725120+0000","last_unstale":"2024-01-27T01:03:44.733234+0000","last_undegraded":"2024-01-27T01:03:44.733234+0000","last_fullsized":"2024-01-27T01:03:44.733234+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:51:28.069637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.711817+0000","last_change":"2024-01-27T01:03:43.709415+0000","last_active":"2024-01-27T01:03:44.711817+0000","last_peered":"2024-01-27T01:03:44.711817+0000","last_clean":"2024-01-27T01:03:44.711817+0000","last_became_active":"2024-01-27T01:03:43.709067+0000","last_became_peered":"2024-01-27T01:03:43.709067+0000","last_unstale":"2024-01-27T01:03:44.711817+0000","last_undegraded":"2024-01-27T01:03:44.711817+0000","last_fullsized":"2024-01-27T01:03:44.711817+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:14:39.132003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.733673+0000","last_change":"2024-01-27T01:03:44.730564+0000","last_active":"2024-01-27T01:03:44.733673+0000","last_peered":"2024-01-27T01:03:44.733673+0000","last_clean":"2024-01-27T01:03:44.733673+0000","last_became_active":"2024-01-27T01:03:44.730270+0000","last_became_peered":"2024-01-27T01:03:44.730270+0000","last_unstale":"2024-01-27T01:03:44.733673+0000","last_undegraded":"2024-01-27T01:03:44.733673+0000","last_fullsized":"2024-01-27T01:03:44.733673+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:58:37.015025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.717470+0000","last_change":"2024-01-27T01:03:43.721493+0000","last_active":"2024-01-27T01:03:44.717470+0000","last_peered":"2024-01-27T01:03:44.717470+0000","last_clean":"2024-01-27T01:03:44.717470+0000","last_became_active":"2024-01-27T01:03:43.721147+0000","last_became_peered":"2024-01-27T01:03:43.721147+0000","last_unstale":"2024-01-27T01:03:44.717470+0000","last_undegraded":"2024-01-27T01:03:44.717470+0000","last_fullsized":"2024-01-27T01:03:44.717470+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:21:32.902395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.734113+0000","last_change":"2024-01-27T01:03:44.729906+0000","last_active":"2024-01-27T01:03:44.734113+0000","last_peered":"2024-01-27T01:03:44.734113+0000","last_clean":"2024-01-27T01:03:44.734113+0000","last_became_active":"2024-01-27T01:03:44.729622+0000","last_became_peered":"2024-01-27T01:03:44.729622+0000","last_unstale":"2024-01-27T01:03:44.734113+0000","last_undegraded":"2024-01-27T01:03:44.734113+0000","last_fullsized":"2024-01-27T01:03:44.734113+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:53:11.353764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.708745+0000","last_change":"2024-01-27T01:03:43.715588+0000","last_active":"2024-01-27T01:03:44.708745+0000","last_peered":"2024-01-27T01:03:44.708745+0000","last_clean":"2024-01-27T01:03:44.708745+0000","last_became_active":"2024-01-27T01:03:43.715312+0000","last_became_peered":"2024-01-27T01:03:43.715312+0000","last_unstale":"2024-01-27T01:03:44.708745+0000","last_undegraded":"2024-01-27T01:03:44.708745+0000","last_fullsized":"2024-01-27T01:03:44.708745+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:23:07.323492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.734777+0000","last_change":"2024-01-27T01:03:44.731519+0000","last_active":"2024-01-27T01:03:44.734777+0000","last_peered":"2024-01-27T01:03:44.734777+0000","last_clean":"2024-01-27T01:03:44.734777+0000","last_became_active":"2024-01-27T01:03:44.727396+0000","last_became_peered":"2024-01-27T01:03:44.727396+0000","last_unstale":"2024-01-27T01:03:44.734777+0000","last_undegraded":"2024-01-27T01:03:44.734777+0000","last_fullsized":"2024-01-27T01:03:44.734777+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:50:28.451748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"204'2","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.733193+0000","last_change":"2024-01-27T01:03:43.715941+0000","last_active":"2024-01-27T01:03:44.733193+0000","last_peered":"2024-01-27T01:03:44.733193+0000","last_clean":"2024-01-27T01:03:44.733193+0000","last_became_active":"2024-01-27T01:03:43.715648+0000","last_became_peered":"2024-01-27T01:03:43.715648+0000","last_unstale":"2024-01-27T01:03:44.733193+0000","last_undegraded":"2024-01-27T01:03:44.733193+0000","last_fullsized":"2024-01-27T01:03:44.733193+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:43:32.112762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.736148+0000","last_change":"2024-01-27T01:03:44.733916+0000","last_active":"2024-01-27T01:03:44.736148+0000","last_peered":"2024-01-27T01:03:44.736148+0000","last_clean":"2024-01-27T01:03:44.736148+0000","last_became_active":"2024-01-27T01:03:44.733528+0000","last_became_peered":"2024-01-27T01:03:44.733528+0000","last_unstale":"2024-01-27T01:03:44.736148+0000","last_undegraded":"2024-01-27T01:03:44.736148+0000","last_fullsized":"2024-01-27T01:03:44.736148+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:58:04.855152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.719031+0000","last_change":"2024-01-27T01:03:43.727716+0000","last_active":"2024-01-27T01:03:44.719031+0000","last_peered":"2024-01-27T01:03:44.719031+0000","last_clean":"2024-01-27T01:03:44.719031+0000","last_became_active":"2024-01-27T01:03:43.727384+0000","last_became_peered":"2024-01-27T01:03:43.727384+0000","last_unstale":"2024-01-27T01:03:44.719031+0000","last_undegraded":"2024-01-27T01:03:44.719031+0000","last_fullsized":"2024-01-27T01:03:44.719031+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:54:34.708513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.718690+0000","last_change":"2024-01-27T01:03:44.715180+0000","last_active":"2024-01-27T01:03:44.718690+0000","last_peered":"2024-01-27T01:03:44.718690+0000","last_clean":"2024-01-27T01:03:44.718690+0000","last_became_active":"2024-01-27T01:03:44.714610+0000","last_became_peered":"2024-01-27T01:03:44.714610+0000","last_unstale":"2024-01-27T01:03:44.718690+0000","last_undegraded":"2024-01-27T01:03:44.718690+0000","last_fullsized":"2024-01-27T01:03:44.718690+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:21:36.129471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.704439+0000","last_change":"2024-01-27T01:03:43.720074+0000","last_active":"2024-01-27T01:03:44.704439+0000","last_peered":"2024-01-27T01:03:44.704439+0000","last_clean":"2024-01-27T01:03:44.704439+0000","last_became_active":"2024-01-27T01:03:43.719776+0000","last_became_peered":"2024-01-27T01:03:43.719776+0000","last_unstale":"2024-01-27T01:03:44.704439+0000","last_undegraded":"2024-01-27T01:03:44.704439+0000","last_fullsized":"2024-01-27T01:03:44.704439+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:21:08.089077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.734914+0000","last_change":"2024-01-27T01:03:44.727464+0000","last_active":"2024-01-27T01:03:44.734914+0000","last_peered":"2024-01-27T01:03:44.734914+0000","last_clean":"2024-01-27T01:03:44.734914+0000","last_became_active":"2024-01-27T01:03:44.727139+0000","last_became_peered":"2024-01-27T01:03:44.727139+0000","last_unstale":"2024-01-27T01:03:44.734914+0000","last_undegraded":"2024-01-27T01:03:44.734914+0000","last_fullsized":"2024-01-27T01:03:44.734914+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:56:17.799789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.707731+0000","last_change":"2024-01-27T01:03:43.727865+0000","last_active":"2024-01-27T01:03:44.707731+0000","last_peered":"2024-01-27T01:03:44.707731+0000","last_clean":"2024-01-27T01:03:44.707731+0000","last_became_active":"2024-01-27T01:03:43.727581+0000","last_became_peered":"2024-01-27T01:03:43.727581+0000","last_unstale":"2024-01-27T01:03:44.707731+0000","last_undegraded":"2024-01-27T01:03:44.707731+0000","last_fullsized":"2024-01-27T01:03:44.707731+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:41:00.700232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.731794+0000","last_change":"2024-01-27T01:03:44.731312+0000","last_active":"2024-01-27T01:03:44.731794+0000","last_peered":"2024-01-27T01:03:44.731794+0000","last_clean":"2024-01-27T01:03:44.731794+0000","last_became_active":"2024-01-27T01:03:44.731055+0000","last_became_peered":"2024-01-27T01:03:44.731055+0000","last_unstale":"2024-01-27T01:03:44.731794+0000","last_undegraded":"2024-01-27T01:03:44.731794+0000","last_fullsized":"2024-01-27T01:03:44.731794+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:03:07.526894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.706810+0000","last_change":"2024-01-27T01:03:43.720294+0000","last_active":"2024-01-27T01:03:44.706810+0000","last_peered":"2024-01-27T01:03:44.706810+0000","last_clean":"2024-01-27T01:03:44.706810+0000","last_became_active":"2024-01-27T01:03:43.720011+0000","last_became_peered":"2024-01-27T01:03:43.720011+0000","last_unstale":"2024-01-27T01:03:44.706810+0000","last_undegraded":"2024-01-27T01:03:44.706810+0000","last_fullsized":"2024-01-27T01:03:44.706810+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:19:32.467311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.727867+0000","last_change":"2024-01-27T01:03:44.723224+0000","last_active":"2024-01-27T01:03:44.727867+0000","last_peered":"2024-01-27T01:03:44.727867+0000","last_clean":"2024-01-27T01:03:44.727867+0000","last_became_active":"2024-01-27T01:03:44.721825+0000","last_became_peered":"2024-01-27T01:03:44.721825+0000","last_unstale":"2024-01-27T01:03:44.727867+0000","last_undegraded":"2024-01-27T01:03:44.727867+0000","last_fullsized":"2024-01-27T01:03:44.727867+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:03:41.778611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.706007+0000","last_change":"2024-01-27T01:03:43.724947+0000","last_active":"2024-01-27T01:03:44.706007+0000","last_peered":"2024-01-27T01:03:44.706007+0000","last_clean":"2024-01-27T01:03:44.706007+0000","last_became_active":"2024-01-27T01:03:43.724646+0000","last_became_peered":"2024-01-27T01:03:43.724646+0000","last_unstale":"2024-01-27T01:03:44.706007+0000","last_undegraded":"2024-01-27T01:03:44.706007+0000","last_fullsized":"2024-01-27T01:03:44.706007+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:22:49.979346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.731901+0000","last_change":"2024-01-27T01:03:44.722316+0000","last_active":"2024-01-27T01:03:44.731901+0000","last_peered":"2024-01-27T01:03:44.731901+0000","last_clean":"2024-01-27T01:03:44.731901+0000","last_became_active":"2024-01-27T01:03:44.722046+0000","last_became_peered":"2024-01-27T01:03:44.722046+0000","last_unstale":"2024-01-27T01:03:44.731901+0000","last_undegraded":"2024-01-27T01:03:44.731901+0000","last_fullsized":"2024-01-27T01:03:44.731901+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:18:09.693234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.717596+0000","last_change":"2024-01-27T01:03:43.713126+0000","last_active":"2024-01-27T01:03:44.717596+0000","last_peered":"2024-01-27T01:03:44.717596+0000","last_clean":"2024-01-27T01:03:44.717596+0000","last_became_active":"2024-01-27T01:03:43.712850+0000","last_became_peered":"2024-01-27T01:03:43.712850+0000","last_unstale":"2024-01-27T01:03:44.717596+0000","last_undegraded":"2024-01-27T01:03:44.717596+0000","last_fullsized":"2024-01-27T01:03:44.717596+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:41:52.241070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.718977+0000","last_change":"2024-01-27T01:03:44.717845+0000","last_active":"2024-01-27T01:03:44.718977+0000","last_peered":"2024-01-27T01:03:44.718977+0000","last_clean":"2024-01-27T01:03:44.718977+0000","last_became_active":"2024-01-27T01:03:44.717539+0000","last_became_peered":"2024-01-27T01:03:44.717539+0000","last_unstale":"2024-01-27T01:03:44.718977+0000","last_undegraded":"2024-01-27T01:03:44.718977+0000","last_fullsized":"2024-01-27T01:03:44.718977+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:28:50.109807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.708843+0000","last_change":"2024-01-27T01:03:43.719995+0000","last_active":"2024-01-27T01:03:44.708843+0000","last_peered":"2024-01-27T01:03:44.708843+0000","last_clean":"2024-01-27T01:03:44.708843+0000","last_became_active":"2024-01-27T01:03:43.719699+0000","last_became_peered":"2024-01-27T01:03:43.719699+0000","last_unstale":"2024-01-27T01:03:44.708843+0000","last_undegraded":"2024-01-27T01:03:44.708843+0000","last_fullsized":"2024-01-27T01:03:44.708843+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:15:54.281492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.722470+0000","last_change":"2024-01-27T01:03:44.718614+0000","last_active":"2024-01-27T01:03:44.722470+0000","last_peered":"2024-01-27T01:03:44.722470+0000","last_clean":"2024-01-27T01:03:44.722470+0000","last_became_active":"2024-01-27T01:03:44.718383+0000","last_became_peered":"2024-01-27T01:03:44.718383+0000","last_unstale":"2024-01-27T01:03:44.722470+0000","last_undegraded":"2024-01-27T01:03:44.722470+0000","last_fullsized":"2024-01-27T01:03:44.722470+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:22.370885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.707232+0000","last_change":"2024-01-27T01:03:43.713997+0000","last_active":"2024-01-27T01:03:44.707232+0000","last_peered":"2024-01-27T01:03:44.707232+0000","last_clean":"2024-01-27T01:03:44.707232+0000","last_became_active":"2024-01-27T01:03:43.713668+0000","last_became_peered":"2024-01-27T01:03:43.713668+0000","last_unstale":"2024-01-27T01:03:44.707232+0000","last_undegraded":"2024-01-27T01:03:44.707232+0000","last_fullsized":"2024-01-27T01:03:44.707232+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:13:33.372407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"195'8728","reported_seq":9570,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.707680+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:03:44.707680+0000","last_peered":"2024-01-27T01:03:44.707680+0000","last_clean":"2024-01-27T01:03:44.707680+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:03:44.707680+0000","last_undegraded":"2024-01-27T01:03:44.707680+0000","last_fullsized":"2024-01-27T01:03:44.707680+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8728,"log_dups_size":0,"ondisk_log_size":8728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2072,"num_read_kb":4396,"num_write":590,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2072,"num_read_kb":4396,"num_write":590,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8728,"ondisk_log_size":8728,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575971,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300380,"kb_used_data":3092,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071460,"statfs":{"total":96636764160,"available":96329175040,"internally_reserved":0,"allocated":3166208,"data_stored":1937780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.622,"15min":0.61199999999999999},"min":{"1min":0.56699999999999995,"5min":0.41299999999999998,"15min":0.312},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.626,"15min":0.62},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.33000000000000002},"max":{"1min":0.76900000000000002,"5min":0.78100000000000003,"15min":0.86399999999999999},"last":0.498}]},{"osd":1,"last update":"Sat Jan 27 01:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.622,"15min":0.61799999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.64900000000000002,"15min":0.63400000000000001},"min":{"1min":0.53300000000000003,"5min":0.45400000000000001,"15min":0.34300000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.89100000000000001},"last":0.55400000000000005}]},{"osd":2,"last update":"Sat Jan 27 01:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.62,"15min":0.60999999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.39200000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.86599999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64300000000000002,"15min":0.63},"min":{"1min":0.44700000000000001,"5min":0.433,"15min":0.40400000000000003},"max":{"1min":0.85699999999999998,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.56999999999999995}]},{"osd":3,"last update":"Sat Jan 27 01:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.627,"15min":0.62},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.39100000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.51100000000000001,"5min":0.41099999999999998,"15min":0.40899999999999997},"max":{"1min":0.79200000000000004,"5min":0.85799999999999998,"15min":0.88},"last":0.59899999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.48999999999999999,"15min":0.48399999999999999},"min":{"1min":0.35099999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72699999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.432},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.38400000000000001,"5min":0.34599999999999997,"15min":0.32200000000000001},"max":{"1min":0.77300000000000002,"5min":0.77700000000000002,"15min":0.86599999999999999},"last":0.53500000000000003}]},{"osd":5,"last update":"Sat Jan 27 01:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.52500000000000002,"15min":0.51600000000000001},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.32600000000000001},"max":{"1min":0.76200000000000001,"5min":0.80400000000000005,"15min":0.84999999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56299999999999994,"15min":0.56200000000000006},"min":{"1min":0.42499999999999999,"5min":0.34399999999999997,"15min":0.33700000000000002},"max":{"1min":0.81499999999999995,"5min":0.84199999999999997,"15min":0.84299999999999997},"last":0.58599999999999997}]},{"osd":6,"last update":"Sat Jan 27 01:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.58399999999999996},"min":{"1min":0.46300000000000002,"5min":0.38900000000000001,"15min":0.37},"max":{"1min":0.80000000000000004,"5min":0.81399999999999995,"15min":0.84799999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.59799999999999998,"15min":0.57999999999999996},"min":{"1min":0.443,"5min":0.371,"15min":0.35699999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.44800000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575971,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307996,"kb_used_data":2516,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063844,"statfs":{"total":96636764160,"available":96321376256,"internally_reserved":0,"allocated":2576384,"data_stored":1347482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.45600000000000002,"5min":0.38600000000000001,"15min":0.36399999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.86399999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.63,"15min":0.62},"min":{"1min":0.47099999999999997,"5min":0.435,"15min":0.376},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.86299999999999999},"last":0.57299999999999995}]},{"osd":1,"last update":"Sat Jan 27 01:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.61799999999999999,"15min":0.61099999999999999},"min":{"1min":0.45100000000000001,"5min":0.44500000000000001,"15min":0.377},"max":{"1min":0.78800000000000003,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.626,"15min":0.624},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.83999999999999997},"last":0.59099999999999997}]},{"osd":2,"last update":"Sat Jan 27 01:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63300000000000001,"15min":0.626},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.40899999999999997},"max":{"1min":0.77100000000000002,"5min":0.80600000000000005,"15min":0.82399999999999995},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63300000000000001,"15min":0.625},"min":{"1min":0.48099999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.753,"5min":0.878,"15min":0.878},"last":0.60299999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.624,"15min":0.629},"min":{"1min":0.498,"5min":0.442,"15min":0.43099999999999999},"max":{"1min":0.78100000000000003,"5min":0.82599999999999996,"15min":0.90100000000000002},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.67300000000000004,"15min":0.66300000000000003},"min":{"1min":0.48399999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.79900000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.624}]},{"osd":4,"last update":"Sat Jan 27 01:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.5,"15min":0.502},"min":{"1min":0.371,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.86599999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.36599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.88900000000000001},"last":0.46400000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.53200000000000003,"15min":0.52200000000000002},"min":{"1min":0.38700000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.75900000000000001,"5min":0.77200000000000002,"15min":0.80500000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57899999999999996,"15min":0.56999999999999995},"min":{"1min":0.38800000000000001,"5min":0.38200000000000001,"15min":0.28899999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.874},"last":0.54300000000000004}]},{"osd":7,"last update":"Sat Jan 27 01:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57599999999999996,"15min":0.57399999999999995},"min":{"1min":0.41999999999999998,"5min":0.373,"15min":0.35699999999999998},"max":{"1min":0.83499999999999996,"5min":0.84399999999999997,"15min":0.92900000000000005},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59599999999999997,"15min":0.59099999999999997},"min":{"1min":0.438,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.83399999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.503}]}]},{"osd":1,"up_from":13,"seq":55834575971,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299800,"kb_used_data":2512,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072040,"statfs":{"total":96636764160,"available":96329768960,"internally_reserved":0,"allocated":2572288,"data_stored":1347412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41299999999999998,"15min":0.42299999999999999},"min":{"1min":0.28799999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.55800000000000005,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.317},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45900000000000002,"15min":0.45800000000000002},"min":{"1min":0.28399999999999997,"5min":0.27900000000000003,"15min":0.26800000000000002},"max":{"1min":0.67800000000000005,"5min":0.73999999999999999,"15min":0.76600000000000001},"last":0.41199999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.443,"15min":0.442},"min":{"1min":0.32800000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.495,"15min":0.49399999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.28399999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.39500000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53300000000000003,"15min":0.53600000000000003},"min":{"1min":0.46000000000000002,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":1.2130000000000001},"last":0.437},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.35799999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.125,"5min":1.125,"15min":1.22},"last":0.47099999999999997}]},{"osd":4,"last update":"Sat Jan 27 01:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.66900000000000004,"15min":0.67700000000000005},"min":{"1min":0.43099999999999999,"5min":0.42999999999999999,"15min":0.39800000000000002},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.70099999999999996,"15min":0.69999999999999996},"min":{"1min":0.57199999999999995,"5min":0.52800000000000002,"15min":0.35699999999999998},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.72199999999999998}]},{"osd":5,"last update":"Sat Jan 27 01:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70499999999999996,"15min":0.70199999999999996},"min":{"1min":0.53200000000000003,"5min":0.50900000000000001,"15min":0.44500000000000001},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.69799999999999995,"15min":0.70499999999999996},"min":{"1min":0.55000000000000004,"5min":0.47599999999999998,"15min":0.41999999999999998},"max":{"1min":1.079,"5min":1.079,"15min":1.119},"last":0.67400000000000004}]},{"osd":6,"last update":"Sat Jan 27 01:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.71999999999999997,"15min":0.71599999999999997},"min":{"1min":0.56000000000000005,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.70899999999999996,"15min":0.70999999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.435},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.59999999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.751,"15min":0.75600000000000001},"min":{"1min":0.55000000000000004,"5min":0.51100000000000001,"15min":0.47599999999999998},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.76700000000000002,"15min":0.76500000000000001},"min":{"1min":0.61299999999999999,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.76300000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575970,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316768,"kb_used_data":3096,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055072,"statfs":{"total":96636764160,"available":96312393728,"internally_reserved":0,"allocated":3170304,"data_stored":1931408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42599999999999999,"15min":0.42299999999999999},"min":{"1min":0.313,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.46899999999999997,"15min":0.46200000000000002},"min":{"1min":0.33300000000000002,"5min":0.313,"15min":0.26200000000000001},"max":{"1min":0.60899999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.55800000000000005}]},{"osd":2,"last update":"Sat Jan 27 01:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.44900000000000001,"15min":0.437},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.27300000000000002},"max":{"1min":0.53000000000000003,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.43099999999999999}]},{"osd":3,"last update":"Sat Jan 27 01:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.496,"15min":0.48699999999999999},"min":{"1min":0.36899999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.57099999999999995,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.40999999999999998,"5min":0.35999999999999999,"15min":0.33900000000000002},"max":{"1min":0.66300000000000003,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.48299999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67300000000000004,"15min":0.67700000000000005},"min":{"1min":0.5,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.91500000000000004,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.51300000000000001,"5min":0.47099999999999997,"15min":0.437},"max":{"1min":0.78000000000000003,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.72199999999999998}]},{"osd":5,"last update":"Sat Jan 27 01:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.70199999999999996,"15min":0.70299999999999996},"min":{"1min":0.51000000000000001,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.98199999999999998,"5min":2.165,"15min":2.165},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.71099999999999997,"15min":0.71599999999999997},"min":{"1min":0.48799999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.86499999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.77100000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.70999999999999996,"15min":0.71299999999999997},"min":{"1min":0.52600000000000002,"5min":0.48399999999999999,"15min":0.47699999999999998},"max":{"1min":0.83399999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.749,"15min":0.74399999999999999},"min":{"1min":0.57499999999999996,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0109999999999999,"5min":1.425,"15min":1.425},"last":0.84799999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.76500000000000001,"15min":0.76300000000000001},"min":{"1min":0.60399999999999998,"5min":0.52700000000000002,"15min":0.50900000000000001},"max":{"1min":0.95099999999999996,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.76600000000000001,"15min":0.77000000000000002},"min":{"1min":0.57399999999999995,"5min":0.52700000000000002,"15min":0.48499999999999999},"max":{"1min":0.94299999999999995,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.82799999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575970,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312092,"kb_used_data":2516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059748,"statfs":{"total":96636764160,"available":96317181952,"internally_reserved":0,"allocated":2576384,"data_stored":1347966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40600000000000003,"15min":0.41299999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.52600000000000002,"5min":0.746,"15min":0.78700000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.44800000000000001,"15min":0.44900000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.58799999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56399999999999995}]},{"osd":1,"last update":"Sat Jan 27 01:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.27000000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.51300000000000001,"15min":0.52400000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.78100000000000003},"last":0.64300000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.499,"15min":0.501},"min":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.495},"min":{"1min":0.33100000000000002,"5min":0.32800000000000001,"15min":0.31900000000000001},"max":{"1min":0.61299999999999999,"5min":0.66600000000000004,"15min":0.75700000000000001},"last":0.59099999999999997}]},{"osd":4,"last update":"Sat Jan 27 01:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66700000000000004,"15min":0.67900000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.442},"max":{"1min":0.88200000000000001,"5min":0.97599999999999998,"15min":1.091},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68899999999999995,"15min":0.69899999999999995},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.378},"max":{"1min":0.81799999999999995,"5min":0.89900000000000002,"15min":1.107},"last":0.81399999999999995}]},{"osd":5,"last update":"Sat Jan 27 01:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68899999999999995,"15min":0.70699999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.89500000000000002,"5min":0.995,"15min":1.167},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.71699999999999997,"15min":0.71599999999999997},"min":{"1min":0.46300000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.97499999999999998,"5min":1.0229999999999999,"15min":1.05},"last":0.93100000000000005}]},{"osd":6,"last update":"Sat Jan 27 01:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.71399999999999997,"15min":0.71599999999999997},"min":{"1min":0.50800000000000001,"5min":0.46700000000000003,"15min":0.45500000000000002},"max":{"1min":0.97899999999999998,"5min":0.996,"15min":1.077},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.71099999999999997,"15min":0.72599999999999998},"min":{"1min":0.53000000000000003,"5min":0.48199999999999998,"15min":0.39200000000000002},"max":{"1min":0.84199999999999997,"5min":0.96699999999999997,"15min":1.135},"last":0.66200000000000003}]},{"osd":7,"last update":"Sat Jan 27 01:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.745,"15min":0.752},"min":{"1min":0.50800000000000001,"5min":0.501,"15min":0.47099999999999997},"max":{"1min":0.89300000000000002,"5min":1.04,"15min":1.198},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.76600000000000001,"15min":0.77400000000000002},"min":{"1min":0.56499999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.94499999999999995,"5min":1.054,"15min":1.0880000000000001},"last":0.85799999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575969,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303904,"kb_used_data":2520,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067936,"statfs":{"total":96636764160,"available":96325566464,"internally_reserved":0,"allocated":2580480,"data_stored":1347572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:03:32 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40699999999999997,"15min":0.40999999999999998},"min":{"1min":0.28299999999999997,"5min":0.28199999999999997,"15min":0.26000000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.63700000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.441,"15min":0.45300000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.52500000000000002,"5min":0.64700000000000002,"15min":0.84699999999999998},"last":0.34100000000000003}]},{"osd":1,"last update":"Sat Jan 27 01:03:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47299999999999998,"15min":0.48299999999999998},"min":{"1min":0.33900000000000002,"5min":0.28899999999999998,"15min":0.27400000000000002},"max":{"1min":0.70199999999999996,"5min":1.012,"15min":1.0600000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.51100000000000001,"15min":0.51300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.33200000000000002},"max":{"1min":0.68200000000000005,"5min":0.995,"15min":1.018},"last":0.45700000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:03:32 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.495,"15min":0.5},"min":{"1min":0.36399999999999999,"5min":0.35699999999999998,"15min":0.34599999999999997},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.96499999999999997},"last":0.372},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47799999999999998,"15min":0.49299999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64900000000000002,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.54700000000000004}]},{"osd":4,"last update":"Sat Jan 27 01:03:32 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.69499999999999995,"15min":0.70399999999999996},"min":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.439},"max":{"1min":0.88700000000000001,"5min":1.089,"15min":1.089},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.67500000000000004,"15min":0.69599999999999995},"min":{"1min":0.48799999999999999,"5min":0.46000000000000002,"15min":0.42199999999999999},"max":{"1min":0.94599999999999995,"5min":1.069,"15min":1.069},"last":0.69799999999999995}]},{"osd":5,"last update":"Sat Jan 27 01:03:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68899999999999995,"15min":0.69999999999999996},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.46200000000000002},"max":{"1min":0.86699999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.68999999999999995,"15min":0.71099999999999997},"min":{"1min":0.54600000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.98599999999999999,"5min":1.123,"15min":1.123},"last":0.56499999999999995}]},{"osd":6,"last update":"Sat Jan 27 01:03:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69799999999999995,"15min":0.70699999999999996},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.44700000000000001},"max":{"1min":0.93100000000000005,"5min":0.98099999999999998,"15min":1.0589999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71499999999999997,"15min":0.73399999999999999},"min":{"1min":0.54500000000000004,"5min":0.52700000000000002,"15min":0.49199999999999999},"max":{"1min":0.89100000000000001,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.65100000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:03:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.70399999999999996,"15min":0.72299999999999998},"min":{"1min":0.55900000000000005,"5min":0.52500000000000002,"15min":0.45600000000000002},"max":{"1min":0.94999999999999996,"5min":1.157,"15min":1.157},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76300000000000001,"15min":0.77200000000000002},"min":{"1min":0.63100000000000001,"5min":0.55500000000000005,"15min":0.499},"max":{"1min":0.95899999999999996,"5min":1.103,"15min":1.103},"last":0.75700000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575971,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312104,"kb_used_data":2528,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059736,"statfs":{"total":96636764160,"available":96317169664,"internally_reserved":0,"allocated":2588672,"data_stored":1348982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.624,"15min":0.60899999999999999},"min":{"1min":0.441,"5min":0.40799999999999997,"15min":0.30299999999999999},"max":{"1min":0.69699999999999995,"5min":0.80000000000000004,"15min":0.84199999999999997},"last":0.377},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.61199999999999999,"15min":0.60999999999999999},"min":{"1min":0.44500000000000001,"5min":0.40000000000000002,"15min":0.35799999999999998},"max":{"1min":0.73199999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.67600000000000005}]},{"osd":1,"last update":"Sat Jan 27 01:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.63400000000000001,"15min":0.626},"min":{"1min":0.48499999999999999,"5min":0.38800000000000001,"15min":0.378},"max":{"1min":0.755,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.621,"15min":0.61499999999999999},"min":{"1min":0.502,"5min":0.42099999999999999,"15min":0.35799999999999998},"max":{"1min":0.72499999999999998,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.45300000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60099999999999998,"15min":0.59799999999999998},"min":{"1min":0.42999999999999999,"5min":0.40400000000000003,"15min":0.35299999999999998},"max":{"1min":0.70299999999999996,"5min":0.872,"15min":0.88700000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63800000000000001,"15min":0.628},"min":{"1min":0.51400000000000001,"5min":0.47999999999999998,"15min":0.40200000000000002},"max":{"1min":0.71699999999999997,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.60699999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.63200000000000001,"15min":0.61899999999999999},"min":{"1min":0.49099999999999999,"5min":0.41499999999999998,"15min":0.38200000000000001},"max":{"1min":0.76900000000000002,"5min":0.83599999999999997,"15min":0.84199999999999997},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.62,"15min":0.61899999999999999},"min":{"1min":0.48399999999999999,"5min":0.42399999999999999,"15min":0.40100000000000002},"max":{"1min":0.71199999999999997,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.58299999999999996}]},{"osd":5,"last update":"Sat Jan 27 01:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.48299999999999998,"15min":0.47799999999999998},"min":{"1min":0.34100000000000003,"5min":0.32900000000000001,"15min":0.32000000000000001},"max":{"1min":0.66400000000000003,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.503,"15min":0.5},"min":{"1min":0.35099999999999998,"5min":0.312,"15min":0.30399999999999999},"max":{"1min":0.73199999999999998,"5min":0.878,"15min":0.878},"last":0.499}]},{"osd":6,"last update":"Sat Jan 27 01:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.495,"15min":0.48999999999999999},"min":{"1min":0.379,"5min":0.35799999999999998,"15min":0.317},"max":{"1min":0.70099999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54600000000000004,"15min":0.54100000000000004},"min":{"1min":0.39000000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.73299999999999998,"5min":0.79300000000000004,"15min":0.81499999999999995},"last":0.65600000000000003}]},{"osd":7,"last update":"Sat Jan 27 01:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56699999999999995,"15min":0.55700000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36199999999999999},"max":{"1min":0.80400000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.746,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.624}]}]},{"osd":5,"up_from":13,"seq":55834575971,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312104,"kb_used_data":2528,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059736,"statfs":{"total":96636764160,"available":96317169664,"internally_reserved":0,"allocated":2588672,"data_stored":1348970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61199999999999999,"15min":0.60699999999999998},"min":{"1min":0.48199999999999998,"5min":0.40500000000000003,"15min":0.27300000000000002},"max":{"1min":0.77500000000000002,"5min":1.006,"15min":1.006},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.45200000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.77100000000000002,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.60199999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66500000000000004,"15min":0.65800000000000003},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.41399999999999998},"max":{"1min":0.876,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.502},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.55400000000000005,"5min":0.377,"15min":0.377},"max":{"1min":0.86399999999999999,"5min":1.212,"15min":1.212},"last":0.65400000000000003}]},{"osd":2,"last update":"Sat Jan 27 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75,"5min":1.631,"15min":1.631},"last":0.629},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.625,"15min":0.624},"min":{"1min":0.39300000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80800000000000005,"5min":0.82599999999999996,"15min":0.83299999999999996},"last":0.55100000000000005}]},{"osd":3,"last update":"Sat Jan 27 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60899999999999999,"15min":0.60399999999999998},"min":{"1min":0.42099999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.755,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64000000000000001,"15min":0.63400000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.84999999999999998,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.52500000000000002}]},{"osd":4,"last update":"Sat Jan 27 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.502,"15min":0.48199999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.317},"max":{"1min":0.76100000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.62},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.50900000000000001},"min":{"1min":0.371,"5min":0.34100000000000003,"15min":0.32500000000000001},"max":{"1min":0.747,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.69399999999999995}]},{"osd":6,"last update":"Sat Jan 27 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52500000000000002,"15min":0.501},"min":{"1min":0.372,"5min":0.314,"15min":0.314},"max":{"1min":0.82299999999999995,"5min":1.732,"15min":1.732},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.55100000000000005,"15min":0.53700000000000003},"min":{"1min":0.45200000000000001,"5min":0.374,"15min":0.33500000000000002},"max":{"1min":0.82599999999999996,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.64800000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58099999999999996,"15min":0.56499999999999995},"min":{"1min":0.434,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.628,"15min":0.60699999999999998},"min":{"1min":0.48399999999999999,"5min":0.40300000000000002,"15min":0.39100000000000001},"max":{"1min":0.85999999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.70699999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:03:48.914 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:03:49.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.084+0000 7f8c3ba73700 1 Processor -- start 2024-01-27T01:03:49.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.084+0000 7f8c3ba73700 1 -- start start 2024-01-27T01:03:49.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.085+0000 7f8c3ba73700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c3415dad0 0x7f8c34067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.085+0000 7f8c3ba73700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c34067e70 0x7f8c340b0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.085+0000 7f8c3ba73700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c3406cfe0 con 0x7f8c340b0e70 2024-01-27T01:03:49.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.085+0000 7f8c3ba73700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c340b16a0 con 0x7f8c3415dad0 2024-01-27T01:03:49.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.085+0000 7f8c3ba73700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c34068310 con 0x7f8c34067e70 2024-01-27T01:03:49.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.085+0000 7f8c3980f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c3415dad0 0x7f8c34067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.085+0000 7f8c3980f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c3415dad0 0x7f8c34067930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39102/0 (socket says 172.21.15.46:39102) 2024-01-27T01:03:49.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.085+0000 7f8c3980f700 1 -- 172.21.15.46:0/2976581322 learned_addr learned my addr 172.21.15.46:0/2976581322 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:49.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.085+0000 7f8c3900e700 1 --2- 172.21.15.46:0/2976581322 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c34067e70 0x7f8c340b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.085+0000 7f8c3980f700 1 -- 172.21.15.46:0/2976581322 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c34067e70 msgr2=0x7f8c340b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:49.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.085+0000 7f8c3980f700 1 --2- 172.21.15.46:0/2976581322 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c34067e70 0x7f8c340b0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.085+0000 7f8c3980f700 1 -- 172.21.15.46:0/2976581322 >> v1:172.21.15.46:6789/0 conn(0x7f8c340b0e70 legacy=0x7f8c3406eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:49.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.085+0000 7f8c3980f700 1 -- 172.21.15.46:0/2976581322 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c3406c6b0 con 0x7f8c3415dad0 2024-01-27T01:03:49.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.086+0000 7f8c3980f700 1 --2- 172.21.15.46:0/2976581322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c3415dad0 0x7f8c34067930 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f8c300099e0 tx=0x7f8c30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32dd948b1710c8a7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:49.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.086+0000 7f8c3880d700 1 -- 172.21.15.46:0/2976581322 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3000b820 con 0x7f8c3415dad0 2024-01-27T01:03:49.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.086+0000 7f8c3880d700 1 -- 172.21.15.46:0/2976581322 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c3000b980 con 0x7f8c3415dad0 2024-01-27T01:03:49.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.086+0000 7f8c3880d700 1 -- 172.21.15.46:0/2976581322 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3000bd20 con 0x7f8c3415dad0 2024-01-27T01:03:49.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.086+0000 7f8c3ba73700 1 -- 172.21.15.46:0/2976581322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c3415dad0 msgr2=0x7f8c34067930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:49.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.086+0000 7f8c3ba73700 1 --2- 172.21.15.46:0/2976581322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c3415dad0 0x7f8c34067930 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f8c300099e0 tx=0x7f8c30009250 comp rx=0 tx=0).stop 2024-01-27T01:03:49.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.086+0000 7f8c3ba73700 1 -- 172.21.15.46:0/2976581322 shutdown_connections 2024-01-27T01:03:49.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.086+0000 7f8c3ba73700 1 --2- 172.21.15.46:0/2976581322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c3415dad0 0x7f8c34067930 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.086+0000 7f8c3ba73700 1 --2- 172.21.15.46:0/2976581322 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c34067e70 0x7f8c340b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.086+0000 7f8c3ba73700 1 -- 172.21.15.46:0/2976581322 >> 172.21.15.46:0/2976581322 conn(0x7f8c3406b630 msgr2=0x7f8c340aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:49.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.087+0000 7f8c3ba73700 1 -- 172.21.15.46:0/2976581322 shutdown_connections 2024-01-27T01:03:49.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.087+0000 7f8c3ba73700 1 -- 172.21.15.46:0/2976581322 wait complete. 2024-01-27T01:03:49.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.088+0000 7f8c3ba73700 1 Processor -- start 2024-01-27T01:03:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.088+0000 7f8c3ba73700 1 -- start start 2024-01-27T01:03:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.088+0000 7f8c3ba73700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c34067e70 0x7f8c34149510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.088+0000 7f8c3ba73700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c340b0e70 0x7f8c341601b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.088+0000 7f8c3ba73700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c3415dad0 0x7f8c340ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.088+0000 7f8c3ba73700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3416efa0 con 0x7f8c34067e70 2024-01-27T01:03:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.088+0000 7f8c3ba73700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c340c9640 con 0x7f8c3415dad0 2024-01-27T01:03:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.088+0000 7f8c3ba73700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c340c9920 con 0x7f8c340b0e70 2024-01-27T01:03:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.089+0000 7f8c3980f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c34067e70 0x7f8c34149510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.089+0000 7f8c3980f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c34067e70 0x7f8c34149510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39108/0 (socket says 172.21.15.46:39108) 2024-01-27T01:03:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.089+0000 7f8c3980f700 1 -- 172.21.15.46:0/1251364454 learned_addr learned my addr 172.21.15.46:0/1251364454 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.089+0000 7f8c3900e700 1 --2- 172.21.15.46:0/1251364454 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c340b0e70 0x7f8c341601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.089+0000 7f8c3a010700 1 --2- 172.21.15.46:0/1251364454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c3415dad0 0x7f8c340ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.089+0000 7f8c3980f700 1 -- 172.21.15.46:0/1251364454 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c340b0e70 msgr2=0x7f8c341601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:49.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.089+0000 7f8c3980f700 1 --2- 172.21.15.46:0/1251364454 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c340b0e70 0x7f8c341601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.089+0000 7f8c3980f700 1 -- 172.21.15.46:0/1251364454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c3415dad0 msgr2=0x7f8c340ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:49.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.089+0000 7f8c3980f700 1 --2- 172.21.15.46:0/1251364454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c3415dad0 0x7f8c340ca400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.089+0000 7f8c3980f700 1 -- 172.21.15.46:0/1251364454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c300094f0 con 0x7f8c34067e70 2024-01-27T01:03:49.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.089+0000 7f8c3a010700 1 --2- 172.21.15.46:0/1251364454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c3415dad0 0x7f8c340ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:03:49.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.089+0000 7f8c3980f700 1 --2- 172.21.15.46:0/1251364454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c34067e70 0x7f8c34149510 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f8c30009290 tx=0x7f8c30017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:49.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.089+0000 7f8c29ffb700 1 -- 172.21.15.46:0/1251364454 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c30011dc0 con 0x7f8c34067e70 2024-01-27T01:03:49.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.089+0000 7f8c29ffb700 1 -- 172.21.15.46:0/1251364454 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c30026450 con 0x7f8c34067e70 2024-01-27T01:03:49.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.089+0000 7f8c29ffb700 1 -- 172.21.15.46:0/1251364454 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c30026810 con 0x7f8c34067e70 2024-01-27T01:03:49.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.089+0000 7f8c3ba73700 1 -- 172.21.15.46:0/1251364454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c340c9c00 con 0x7f8c34067e70 2024-01-27T01:03:49.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.090+0000 7f8c3ba73700 1 -- 172.21.15.46:0/1251364454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c34149a50 con 0x7f8c34067e70 2024-01-27T01:03:49.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.091+0000 7f8c3ba73700 1 -- 172.21.15.46:0/1251364454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c34149c40 con 0x7f8c34067e70 2024-01-27T01:03:49.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.091+0000 7f8c29ffb700 1 -- 172.21.15.46:0/1251364454 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8c3001a420 con 0x7f8c34067e70 2024-01-27T01:03:49.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.092+0000 7f8c29ffb700 1 --2- 172.21.15.46:0/1251364454 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8c18066f30 0x7f8c180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.092+0000 7f8c29ffb700 1 -- 172.21.15.46:0/1251364454 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f8c300863e0 con 0x7f8c34067e70 2024-01-27T01:03:49.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.092+0000 7f8c3900e700 1 --2- 172.21.15.46:0/1251364454 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8c18066f30 0x7f8c180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.092+0000 7f8c3900e700 1 --2- 172.21.15.46:0/1251364454 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8c18066f30 0x7f8c180693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8c3415fd20 tx=0x7f8c24008040 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:49.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.096+0000 7f8c29ffb700 1 -- 172.21.15.46:0/1251364454 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8c30053320 con 0x7f8c34067e70 2024-01-27T01:03:49.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.294+0000 7f8c3ba73700 1 -- 172.21.15.46:0/1251364454 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c3404c710 con 0x7f8c18066f30 2024-01-27T01:03:49.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.303+0000 7f8c29ffb700 1 -- 172.21.15.46:0/1251364454 <== mgr.10182 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289686 (secure 0 0 0) 0x7f8c3404c710 con 0x7f8c18066f30 2024-01-27T01:03:49.306 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:49.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.306+0000 7f8c3ba73700 1 -- 172.21.15.46:0/1251364454 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8c18066f30 msgr2=0x7f8c180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:49.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.306+0000 7f8c3ba73700 1 --2- 172.21.15.46:0/1251364454 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8c18066f30 0x7f8c180693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8c3415fd20 tx=0x7f8c24008040 comp rx=0 tx=0).stop 2024-01-27T01:03:49.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.306+0000 7f8c3ba73700 1 -- 172.21.15.46:0/1251364454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c34067e70 msgr2=0x7f8c34149510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:49.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.306+0000 7f8c3ba73700 1 --2- 172.21.15.46:0/1251364454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c34067e70 0x7f8c34149510 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f8c30009290 tx=0x7f8c30017000 comp rx=0 tx=0).stop 2024-01-27T01:03:49.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.306+0000 7f8c3ba73700 1 -- 172.21.15.46:0/1251364454 shutdown_connections 2024-01-27T01:03:49.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.306+0000 7f8c3ba73700 1 --2- 172.21.15.46:0/1251364454 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8c18066f30 0x7f8c180693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.306+0000 7f8c3ba73700 1 --2- 172.21.15.46:0/1251364454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c34067e70 0x7f8c34149510 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.306+0000 7f8c3ba73700 1 --2- 172.21.15.46:0/1251364454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c3415dad0 0x7f8c340ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.306+0000 7f8c3ba73700 1 --2- 172.21.15.46:0/1251364454 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8c340b0e70 0x7f8c341601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.306+0000 7f8c3ba73700 1 -- 172.21.15.46:0/1251364454 >> 172.21.15.46:0/1251364454 conn(0x7f8c3406b630 msgr2=0x7f8c3415a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:49.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.306+0000 7f8c3ba73700 1 -- 172.21.15.46:0/1251364454 shutdown_connections 2024-01-27T01:03:49.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.306+0000 7f8c3ba73700 1 -- 172.21.15.46:0/1251364454 wait complete. 2024-01-27T01:03:49.310 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:03:49.320 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-01-27T01:03:47.756109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2072,"num_read_kb":4396,"num_write":634,"num_write_kb":3475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8766,"ondisk_log_size":8766,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2465148,"kb_used_data":21308,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752509572,"statfs":{"total":773094113280,"available":770569801728,"internally_reserved":0,"allocated":21819392,"data_stored":11957572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.065200"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.720972+0000","last_change":"2024-01-27T01:03:44.716770+0000","last_active":"2024-01-27T01:03:44.720972+0000","last_peered":"2024-01-27T01:03:44.720972+0000","last_clean":"2024-01-27T01:03:44.720972+0000","last_became_active":"2024-01-27T01:03:44.716316+0000","last_became_peered":"2024-01-27T01:03:44.716316+0000","last_unstale":"2024-01-27T01:03:44.720972+0000","last_undegraded":"2024-01-27T01:03:44.720972+0000","last_fullsized":"2024-01-27T01:03:44.720972+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:50:32.225758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.721285+0000","last_change":"2024-01-27T01:03:44.709818+0000","last_active":"2024-01-27T01:03:44.721285+0000","last_peered":"2024-01-27T01:03:44.721285+0000","last_clean":"2024-01-27T01:03:44.721285+0000","last_became_active":"2024-01-27T01:03:44.709330+0000","last_became_peered":"2024-01-27T01:03:44.709330+0000","last_unstale":"2024-01-27T01:03:44.721285+0000","last_undegraded":"2024-01-27T01:03:44.721285+0000","last_fullsized":"2024-01-27T01:03:44.721285+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:32:36.671785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.728735+0000","last_change":"2024-01-27T01:03:44.727679+0000","last_active":"2024-01-27T01:03:44.728735+0000","last_peered":"2024-01-27T01:03:44.728735+0000","last_clean":"2024-01-27T01:03:44.728735+0000","last_became_active":"2024-01-27T01:03:44.727407+0000","last_became_peered":"2024-01-27T01:03:44.727407+0000","last_unstale":"2024-01-27T01:03:44.728735+0000","last_undegraded":"2024-01-27T01:03:44.728735+0000","last_fullsized":"2024-01-27T01:03:44.728735+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:22:50.984272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.732987+0000","last_change":"2024-01-27T01:03:44.726023+0000","last_active":"2024-01-27T01:03:44.732987+0000","last_peered":"2024-01-27T01:03:44.732987+0000","last_clean":"2024-01-27T01:03:44.732987+0000","last_became_active":"2024-01-27T01:03:44.725684+0000","last_became_peered":"2024-01-27T01:03:44.725684+0000","last_unstale":"2024-01-27T01:03:44.732987+0000","last_undegraded":"2024-01-27T01:03:44.732987+0000","last_fullsized":"2024-01-27T01:03:44.732987+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:58:38.516519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.722400+0000","last_change":"2024-01-27T01:03:44.713015+0000","last_active":"2024-01-27T01:03:44.722400+0000","last_peered":"2024-01-27T01:03:44.722400+0000","last_clean":"2024-01-27T01:03:44.722400+0000","last_became_active":"2024-01-27T01:03:44.712092+0000","last_became_peered":"2024-01-27T01:03:44.712092+0000","last_unstale":"2024-01-27T01:03:44.722400+0000","last_undegraded":"2024-01-27T01:03:44.722400+0000","last_fullsized":"2024-01-27T01:03:44.722400+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:13.213228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.723478+0000","last_change":"2024-01-27T01:03:44.718409+0000","last_active":"2024-01-27T01:03:44.723478+0000","last_peered":"2024-01-27T01:03:44.723478+0000","last_clean":"2024-01-27T01:03:44.723478+0000","last_became_active":"2024-01-27T01:03:44.718084+0000","last_became_peered":"2024-01-27T01:03:44.718084+0000","last_unstale":"2024-01-27T01:03:44.723478+0000","last_undegraded":"2024-01-27T01:03:44.723478+0000","last_fullsized":"2024-01-27T01:03:44.723478+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:13:34.377333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.735734+0000","last_change":"2024-01-27T01:03:44.732216+0000","last_active":"2024-01-27T01:03:44.735734+0000","last_peered":"2024-01-27T01:03:44.735734+0000","last_clean":"2024-01-27T01:03:44.735734+0000","last_became_active":"2024-01-27T01:03:44.731911+0000","last_became_peered":"2024-01-27T01:03:44.731911+0000","last_unstale":"2024-01-27T01:03:44.735734+0000","last_undegraded":"2024-01-27T01:03:44.735734+0000","last_fullsized":"2024-01-27T01:03:44.735734+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:54:55.134131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.718838+0000","last_change":"2024-01-27T01:03:44.715232+0000","last_active":"2024-01-27T01:03:44.718838+0000","last_peered":"2024-01-27T01:03:44.718838+0000","last_clean":"2024-01-27T01:03:44.718838+0000","last_became_active":"2024-01-27T01:03:44.714947+0000","last_became_peered":"2024-01-27T01:03:44.714947+0000","last_unstale":"2024-01-27T01:03:44.718838+0000","last_undegraded":"2024-01-27T01:03:44.718838+0000","last_fullsized":"2024-01-27T01:03:44.718838+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:51:48.157181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.720719+0000","last_change":"2024-01-27T01:03:44.714156+0000","last_active":"2024-01-27T01:03:44.720719+0000","last_peered":"2024-01-27T01:03:44.720719+0000","last_clean":"2024-01-27T01:03:44.720719+0000","last_became_active":"2024-01-27T01:03:44.713824+0000","last_became_peered":"2024-01-27T01:03:44.713824+0000","last_unstale":"2024-01-27T01:03:44.720719+0000","last_undegraded":"2024-01-27T01:03:44.720719+0000","last_fullsized":"2024-01-27T01:03:44.720719+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:21:09.094003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.723019+0000","last_change":"2024-01-27T01:03:44.721167+0000","last_active":"2024-01-27T01:03:44.723019+0000","last_peered":"2024-01-27T01:03:44.723019+0000","last_clean":"2024-01-27T01:03:44.723019+0000","last_became_active":"2024-01-27T01:03:44.720888+0000","last_became_peered":"2024-01-27T01:03:44.720888+0000","last_unstale":"2024-01-27T01:03:44.723019+0000","last_undegraded":"2024-01-27T01:03:44.723019+0000","last_fullsized":"2024-01-27T01:03:44.723019+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:55:49.742669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.714010+0000","last_change":"2024-01-27T01:03:44.712099+0000","last_active":"2024-01-27T01:03:44.714010+0000","last_peered":"2024-01-27T01:03:44.714010+0000","last_clean":"2024-01-27T01:03:44.714010+0000","last_became_active":"2024-01-27T01:03:44.711831+0000","last_became_peered":"2024-01-27T01:03:44.711831+0000","last_unstale":"2024-01-27T01:03:44.714010+0000","last_undegraded":"2024-01-27T01:03:44.714010+0000","last_fullsized":"2024-01-27T01:03:44.714010+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:06.223570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.730397+0000","last_change":"2024-01-27T01:03:44.715834+0000","last_active":"2024-01-27T01:03:44.730397+0000","last_peered":"2024-01-27T01:03:44.730397+0000","last_clean":"2024-01-27T01:03:44.730397+0000","last_became_active":"2024-01-27T01:03:44.715512+0000","last_became_peered":"2024-01-27T01:03:44.715512+0000","last_unstale":"2024-01-27T01:03:44.730397+0000","last_undegraded":"2024-01-27T01:03:44.730397+0000","last_fullsized":"2024-01-27T01:03:44.730397+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:54:22.369203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.726719+0000","last_change":"2024-01-27T01:03:44.723710+0000","last_active":"2024-01-27T01:03:44.726719+0000","last_peered":"2024-01-27T01:03:44.726719+0000","last_clean":"2024-01-27T01:03:44.726719+0000","last_became_active":"2024-01-27T01:03:44.723234+0000","last_became_peered":"2024-01-27T01:03:44.723234+0000","last_unstale":"2024-01-27T01:03:44.726719+0000","last_undegraded":"2024-01-27T01:03:44.726719+0000","last_fullsized":"2024-01-27T01:03:44.726719+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:06.223570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.727197+0000","last_change":"2024-01-27T01:03:44.721866+0000","last_active":"2024-01-27T01:03:44.727197+0000","last_peered":"2024-01-27T01:03:44.727197+0000","last_clean":"2024-01-27T01:03:44.727197+0000","last_became_active":"2024-01-27T01:03:44.721513+0000","last_became_peered":"2024-01-27T01:03:44.721513+0000","last_unstale":"2024-01-27T01:03:44.727197+0000","last_undegraded":"2024-01-27T01:03:44.727197+0000","last_fullsized":"2024-01-27T01:03:44.727197+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:05:23.070236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.730155+0000","last_change":"2024-01-27T01:03:44.725309+0000","last_active":"2024-01-27T01:03:44.730155+0000","last_peered":"2024-01-27T01:03:44.730155+0000","last_clean":"2024-01-27T01:03:44.730155+0000","last_became_active":"2024-01-27T01:03:44.725034+0000","last_became_peered":"2024-01-27T01:03:44.725034+0000","last_unstale":"2024-01-27T01:03:44.730155+0000","last_undegraded":"2024-01-27T01:03:44.730155+0000","last_fullsized":"2024-01-27T01:03:44.730155+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:17:29.492503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.728169+0000","last_change":"2024-01-27T01:03:44.723851+0000","last_active":"2024-01-27T01:03:44.728169+0000","last_peered":"2024-01-27T01:03:44.728169+0000","last_clean":"2024-01-27T01:03:44.728169+0000","last_became_active":"2024-01-27T01:03:44.723316+0000","last_became_peered":"2024-01-27T01:03:44.723316+0000","last_unstale":"2024-01-27T01:03:44.728169+0000","last_undegraded":"2024-01-27T01:03:44.728169+0000","last_fullsized":"2024-01-27T01:03:44.728169+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:12:00.859544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.723162+0000","last_change":"2024-01-27T01:03:44.718612+0000","last_active":"2024-01-27T01:03:44.723162+0000","last_peered":"2024-01-27T01:03:44.723162+0000","last_clean":"2024-01-27T01:03:44.723162+0000","last_became_active":"2024-01-27T01:03:44.718264+0000","last_became_peered":"2024-01-27T01:03:44.718264+0000","last_unstale":"2024-01-27T01:03:44.723162+0000","last_undegraded":"2024-01-27T01:03:44.723162+0000","last_fullsized":"2024-01-27T01:03:44.723162+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:49:32.367602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.726273+0000","last_change":"2024-01-27T01:03:44.720922+0000","last_active":"2024-01-27T01:03:44.726273+0000","last_peered":"2024-01-27T01:03:44.726273+0000","last_clean":"2024-01-27T01:03:44.726273+0000","last_became_active":"2024-01-27T01:03:44.720639+0000","last_became_peered":"2024-01-27T01:03:44.720639+0000","last_unstale":"2024-01-27T01:03:44.726273+0000","last_undegraded":"2024-01-27T01:03:44.726273+0000","last_fullsized":"2024-01-27T01:03:44.726273+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:32:53.294651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.733389+0000","last_change":"2024-01-27T01:03:44.732930+0000","last_active":"2024-01-27T01:03:44.733389+0000","last_peered":"2024-01-27T01:03:44.733389+0000","last_clean":"2024-01-27T01:03:44.733389+0000","last_became_active":"2024-01-27T01:03:44.732629+0000","last_became_peered":"2024-01-27T01:03:44.732629+0000","last_unstale":"2024-01-27T01:03:44.733389+0000","last_undegraded":"2024-01-27T01:03:44.733389+0000","last_fullsized":"2024-01-27T01:03:44.733389+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:22:50.984272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.722684+0000","last_change":"2024-01-27T01:03:44.712507+0000","last_active":"2024-01-27T01:03:44.722684+0000","last_peered":"2024-01-27T01:03:44.722684+0000","last_clean":"2024-01-27T01:03:44.722684+0000","last_became_active":"2024-01-27T01:03:44.712179+0000","last_became_peered":"2024-01-27T01:03:44.712179+0000","last_unstale":"2024-01-27T01:03:44.722684+0000","last_undegraded":"2024-01-27T01:03:44.722684+0000","last_fullsized":"2024-01-27T01:03:44.722684+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:37:07.278285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.720542+0000","last_change":"2024-01-27T01:03:44.714127+0000","last_active":"2024-01-27T01:03:44.720542+0000","last_peered":"2024-01-27T01:03:44.720542+0000","last_clean":"2024-01-27T01:03:44.720542+0000","last_became_active":"2024-01-27T01:03:44.713797+0000","last_became_peered":"2024-01-27T01:03:44.713797+0000","last_unstale":"2024-01-27T01:03:44.720542+0000","last_undegraded":"2024-01-27T01:03:44.720542+0000","last_fullsized":"2024-01-27T01:03:44.720542+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:28:50.109807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.719504+0000","last_change":"2024-01-27T01:03:44.714072+0000","last_active":"2024-01-27T01:03:44.719504+0000","last_peered":"2024-01-27T01:03:44.719504+0000","last_clean":"2024-01-27T01:03:44.719504+0000","last_became_active":"2024-01-27T01:03:44.713785+0000","last_became_peered":"2024-01-27T01:03:44.713785+0000","last_unstale":"2024-01-27T01:03:44.719504+0000","last_undegraded":"2024-01-27T01:03:44.719504+0000","last_fullsized":"2024-01-27T01:03:44.719504+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:41:01.705159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.718934+0000","last_change":"2024-01-27T01:03:44.716659+0000","last_active":"2024-01-27T01:03:44.718934+0000","last_peered":"2024-01-27T01:03:44.718934+0000","last_clean":"2024-01-27T01:03:44.718934+0000","last_became_active":"2024-01-27T01:03:44.716336+0000","last_became_peered":"2024-01-27T01:03:44.716336+0000","last_unstale":"2024-01-27T01:03:44.718934+0000","last_undegraded":"2024-01-27T01:03:44.718934+0000","last_fullsized":"2024-01-27T01:03:44.718934+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:57:26.315979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.720852+0000","last_change":"2024-01-27T01:03:44.715662+0000","last_active":"2024-01-27T01:03:44.720852+0000","last_peered":"2024-01-27T01:03:44.720852+0000","last_clean":"2024-01-27T01:03:44.720852+0000","last_became_active":"2024-01-27T01:03:44.715373+0000","last_became_peered":"2024-01-27T01:03:44.715373+0000","last_unstale":"2024-01-27T01:03:44.720852+0000","last_undegraded":"2024-01-27T01:03:44.720852+0000","last_fullsized":"2024-01-27T01:03:44.720852+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:39:26.398910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.735207+0000","last_change":"2024-01-27T01:03:44.731607+0000","last_active":"2024-01-27T01:03:44.735207+0000","last_peered":"2024-01-27T01:03:44.735207+0000","last_clean":"2024-01-27T01:03:44.735207+0000","last_became_active":"2024-01-27T01:03:44.727541+0000","last_became_peered":"2024-01-27T01:03:44.727541+0000","last_unstale":"2024-01-27T01:03:44.735207+0000","last_undegraded":"2024-01-27T01:03:44.735207+0000","last_fullsized":"2024-01-27T01:03:44.735207+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:54.792398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.717439+0000","last_change":"2024-01-27T01:03:44.713245+0000","last_active":"2024-01-27T01:03:44.717439+0000","last_peered":"2024-01-27T01:03:44.717439+0000","last_clean":"2024-01-27T01:03:44.717439+0000","last_became_active":"2024-01-27T01:03:44.712817+0000","last_became_peered":"2024-01-27T01:03:44.712817+0000","last_unstale":"2024-01-27T01:03:44.717439+0000","last_undegraded":"2024-01-27T01:03:44.717439+0000","last_fullsized":"2024-01-27T01:03:44.717439+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:19:51.053608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.729475+0000","last_change":"2024-01-27T01:03:44.727971+0000","last_active":"2024-01-27T01:03:44.729475+0000","last_peered":"2024-01-27T01:03:44.729475+0000","last_clean":"2024-01-27T01:03:44.729475+0000","last_became_active":"2024-01-27T01:03:44.727672+0000","last_became_peered":"2024-01-27T01:03:44.727672+0000","last_unstale":"2024-01-27T01:03:44.729475+0000","last_undegraded":"2024-01-27T01:03:44.729475+0000","last_fullsized":"2024-01-27T01:03:44.729475+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:29:16.068435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.722706+0000","last_change":"2024-01-27T01:03:44.712870+0000","last_active":"2024-01-27T01:03:44.722706+0000","last_peered":"2024-01-27T01:03:44.722706+0000","last_clean":"2024-01-27T01:03:44.722706+0000","last_became_active":"2024-01-27T01:03:44.712106+0000","last_became_peered":"2024-01-27T01:03:44.712106+0000","last_unstale":"2024-01-27T01:03:44.722706+0000","last_undegraded":"2024-01-27T01:03:44.722706+0000","last_fullsized":"2024-01-27T01:03:44.722706+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:53:40.835858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.722144+0000","last_change":"2024-01-27T01:03:44.720569+0000","last_active":"2024-01-27T01:03:44.722144+0000","last_peered":"2024-01-27T01:03:44.722144+0000","last_clean":"2024-01-27T01:03:44.722144+0000","last_became_active":"2024-01-27T01:03:44.720292+0000","last_became_peered":"2024-01-27T01:03:44.720292+0000","last_unstale":"2024-01-27T01:03:44.722144+0000","last_undegraded":"2024-01-27T01:03:44.722144+0000","last_fullsized":"2024-01-27T01:03:44.722144+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:22:50.984272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.717829+0000","last_change":"2024-01-27T01:03:44.714245+0000","last_active":"2024-01-27T01:03:44.717829+0000","last_peered":"2024-01-27T01:03:44.717829+0000","last_clean":"2024-01-27T01:03:44.717829+0000","last_became_active":"2024-01-27T01:03:44.714021+0000","last_became_peered":"2024-01-27T01:03:44.714021+0000","last_unstale":"2024-01-27T01:03:44.717829+0000","last_undegraded":"2024-01-27T01:03:44.717829+0000","last_fullsized":"2024-01-27T01:03:44.717829+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:19:33.472237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.735212+0000","last_change":"2024-01-27T01:03:44.731737+0000","last_active":"2024-01-27T01:03:44.735212+0000","last_peered":"2024-01-27T01:03:44.735212+0000","last_clean":"2024-01-27T01:03:44.735212+0000","last_became_active":"2024-01-27T01:03:44.731430+0000","last_became_peered":"2024-01-27T01:03:44.731430+0000","last_unstale":"2024-01-27T01:03:44.735212+0000","last_undegraded":"2024-01-27T01:03:44.735212+0000","last_fullsized":"2024-01-27T01:03:44.735212+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:39:55.734791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.734232+0000","last_change":"2024-01-27T01:03:44.728456+0000","last_active":"2024-01-27T01:03:44.734232+0000","last_peered":"2024-01-27T01:03:44.734232+0000","last_clean":"2024-01-27T01:03:44.734232+0000","last_became_active":"2024-01-27T01:03:44.727965+0000","last_became_peered":"2024-01-27T01:03:44.727965+0000","last_unstale":"2024-01-27T01:03:44.734232+0000","last_undegraded":"2024-01-27T01:03:44.734232+0000","last_fullsized":"2024-01-27T01:03:44.734232+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:06:05.222812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.723605+0000","last_change":"2024-01-27T01:03:44.722503+0000","last_active":"2024-01-27T01:03:44.723605+0000","last_peered":"2024-01-27T01:03:44.723605+0000","last_clean":"2024-01-27T01:03:44.723605+0000","last_became_active":"2024-01-27T01:03:44.722230+0000","last_became_peered":"2024-01-27T01:03:44.722230+0000","last_unstale":"2024-01-27T01:03:44.723605+0000","last_undegraded":"2024-01-27T01:03:44.723605+0000","last_fullsized":"2024-01-27T01:03:44.723605+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:51:19.304072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"204'4","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.736213+0000","last_change":"2024-01-27T01:03:43.715832+0000","last_active":"2024-01-27T01:03:44.736213+0000","last_peered":"2024-01-27T01:03:44.736213+0000","last_clean":"2024-01-27T01:03:44.736213+0000","last_became_active":"2024-01-27T01:03:43.715526+0000","last_became_peered":"2024-01-27T01:03:43.715526+0000","last_unstale":"2024-01-27T01:03:44.736213+0000","last_undegraded":"2024-01-27T01:03:44.736213+0000","last_fullsized":"2024-01-27T01:03:44.736213+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:27:19.173864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.735757+0000","last_change":"2024-01-27T01:03:44.732480+0000","last_active":"2024-01-27T01:03:44.735757+0000","last_peered":"2024-01-27T01:03:44.735757+0000","last_clean":"2024-01-27T01:03:44.735757+0000","last_became_active":"2024-01-27T01:03:44.727517+0000","last_became_peered":"2024-01-27T01:03:44.727517+0000","last_unstale":"2024-01-27T01:03:44.735757+0000","last_undegraded":"2024-01-27T01:03:44.735757+0000","last_fullsized":"2024-01-27T01:03:44.735757+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:00:43.760382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"204'1","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.730832+0000","last_change":"2024-01-27T01:03:43.723182+0000","last_active":"2024-01-27T01:03:44.730832+0000","last_peered":"2024-01-27T01:03:44.730832+0000","last_clean":"2024-01-27T01:03:44.730832+0000","last_became_active":"2024-01-27T01:03:43.720060+0000","last_became_peered":"2024-01-27T01:03:43.720060+0000","last_unstale":"2024-01-27T01:03:44.730832+0000","last_undegraded":"2024-01-27T01:03:44.730832+0000","last_fullsized":"2024-01-27T01:03:44.730832+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:17:28.487577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.718159+0000","last_change":"2024-01-27T01:03:44.715360+0000","last_active":"2024-01-27T01:03:44.718159+0000","last_peered":"2024-01-27T01:03:44.718159+0000","last_clean":"2024-01-27T01:03:44.718159+0000","last_became_active":"2024-01-27T01:03:44.715129+0000","last_became_peered":"2024-01-27T01:03:44.715129+0000","last_unstale":"2024-01-27T01:03:44.718159+0000","last_undegraded":"2024-01-27T01:03:44.718159+0000","last_fullsized":"2024-01-27T01:03:44.718159+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:03:07.526894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"204'2","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.729986+0000","last_change":"2024-01-27T01:03:43.716856+0000","last_active":"2024-01-27T01:03:44.729986+0000","last_peered":"2024-01-27T01:03:44.729986+0000","last_clean":"2024-01-27T01:03:44.729986+0000","last_became_active":"2024-01-27T01:03:43.716162+0000","last_became_peered":"2024-01-27T01:03:43.716162+0000","last_unstale":"2024-01-27T01:03:44.729986+0000","last_undegraded":"2024-01-27T01:03:44.729986+0000","last_fullsized":"2024-01-27T01:03:44.729986+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:13:25.722707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.713618+0000","last_change":"2024-01-27T01:03:44.712278+0000","last_active":"2024-01-27T01:03:44.713618+0000","last_peered":"2024-01-27T01:03:44.713618+0000","last_clean":"2024-01-27T01:03:44.713618+0000","last_became_active":"2024-01-27T01:03:44.711998+0000","last_became_peered":"2024-01-27T01:03:44.711998+0000","last_unstale":"2024-01-27T01:03:44.713618+0000","last_undegraded":"2024-01-27T01:03:44.713618+0000","last_fullsized":"2024-01-27T01:03:44.713618+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:10:33.418151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"204'2","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.734297+0000","last_change":"2024-01-27T01:03:43.717831+0000","last_active":"2024-01-27T01:03:44.734297+0000","last_peered":"2024-01-27T01:03:44.734297+0000","last_clean":"2024-01-27T01:03:44.734297+0000","last_became_active":"2024-01-27T01:03:43.717581+0000","last_became_peered":"2024-01-27T01:03:43.717581+0000","last_unstale":"2024-01-27T01:03:44.734297+0000","last_undegraded":"2024-01-27T01:03:44.734297+0000","last_fullsized":"2024-01-27T01:03:44.734297+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:58:04.065236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.730053+0000","last_change":"2024-01-27T01:03:44.729189+0000","last_active":"2024-01-27T01:03:44.730053+0000","last_peered":"2024-01-27T01:03:44.730053+0000","last_clean":"2024-01-27T01:03:44.730053+0000","last_became_active":"2024-01-27T01:03:44.728899+0000","last_became_peered":"2024-01-27T01:03:44.728899+0000","last_unstale":"2024-01-27T01:03:44.730053+0000","last_undegraded":"2024-01-27T01:03:44.730053+0000","last_fullsized":"2024-01-27T01:03:44.730053+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:19:51.053608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"204'1","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.730861+0000","last_change":"2024-01-27T01:03:43.723249+0000","last_active":"2024-01-27T01:03:44.730861+0000","last_peered":"2024-01-27T01:03:44.730861+0000","last_clean":"2024-01-27T01:03:44.730861+0000","last_became_active":"2024-01-27T01:03:43.722901+0000","last_became_peered":"2024-01-27T01:03:43.722901+0000","last_unstale":"2024-01-27T01:03:44.730861+0000","last_undegraded":"2024-01-27T01:03:44.730861+0000","last_fullsized":"2024-01-27T01:03:44.730861+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:40:44.043969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.730134+0000","last_change":"2024-01-27T01:03:44.724625+0000","last_active":"2024-01-27T01:03:44.730134+0000","last_peered":"2024-01-27T01:03:44.730134+0000","last_clean":"2024-01-27T01:03:44.730134+0000","last_became_active":"2024-01-27T01:03:44.724302+0000","last_became_peered":"2024-01-27T01:03:44.724302+0000","last_unstale":"2024-01-27T01:03:44.730134+0000","last_undegraded":"2024-01-27T01:03:44.730134+0000","last_fullsized":"2024-01-27T01:03:44.730134+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:11:26.057436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"204'1","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.732880+0000","last_change":"2024-01-27T01:03:43.720619+0000","last_active":"2024-01-27T01:03:44.732880+0000","last_peered":"2024-01-27T01:03:44.732880+0000","last_clean":"2024-01-27T01:03:44.732880+0000","last_became_active":"2024-01-27T01:03:43.720318+0000","last_became_peered":"2024-01-27T01:03:43.720318+0000","last_unstale":"2024-01-27T01:03:44.732880+0000","last_undegraded":"2024-01-27T01:03:44.732880+0000","last_fullsized":"2024-01-27T01:03:44.732880+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:14:24.287472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.728926+0000","last_change":"2024-01-27T01:03:44.721762+0000","last_active":"2024-01-27T01:03:44.728926+0000","last_peered":"2024-01-27T01:03:44.728926+0000","last_clean":"2024-01-27T01:03:44.728926+0000","last_became_active":"2024-01-27T01:03:44.721273+0000","last_became_peered":"2024-01-27T01:03:44.721273+0000","last_unstale":"2024-01-27T01:03:44.728926+0000","last_undegraded":"2024-01-27T01:03:44.728926+0000","last_fullsized":"2024-01-27T01:03:44.728926+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:47:07.080276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.712402+0000","last_change":"2024-01-27T01:03:43.716959+0000","last_active":"2024-01-27T01:03:44.712402+0000","last_peered":"2024-01-27T01:03:44.712402+0000","last_clean":"2024-01-27T01:03:44.712402+0000","last_became_active":"2024-01-27T01:03:43.716750+0000","last_became_peered":"2024-01-27T01:03:43.716750+0000","last_unstale":"2024-01-27T01:03:44.712402+0000","last_undegraded":"2024-01-27T01:03:44.712402+0000","last_fullsized":"2024-01-27T01:03:44.712402+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:41:39.450463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.725870+0000","last_change":"2024-01-27T01:03:44.725313+0000","last_active":"2024-01-27T01:03:44.725870+0000","last_peered":"2024-01-27T01:03:44.725870+0000","last_clean":"2024-01-27T01:03:44.725870+0000","last_became_active":"2024-01-27T01:03:44.724942+0000","last_became_peered":"2024-01-27T01:03:44.724942+0000","last_unstale":"2024-01-27T01:03:44.725870+0000","last_undegraded":"2024-01-27T01:03:44.725870+0000","last_fullsized":"2024-01-27T01:03:44.725870+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:34:28.730691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.706505+0000","last_change":"2024-01-27T01:03:43.719422+0000","last_active":"2024-01-27T01:03:44.706505+0000","last_peered":"2024-01-27T01:03:44.706505+0000","last_clean":"2024-01-27T01:03:44.706505+0000","last_became_active":"2024-01-27T01:03:43.719138+0000","last_became_peered":"2024-01-27T01:03:43.719138+0000","last_unstale":"2024-01-27T01:03:44.706505+0000","last_undegraded":"2024-01-27T01:03:44.706505+0000","last_fullsized":"2024-01-27T01:03:44.706505+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:54:29.307945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.725203+0000","last_change":"2024-01-27T01:03:44.712901+0000","last_active":"2024-01-27T01:03:44.725203+0000","last_peered":"2024-01-27T01:03:44.725203+0000","last_clean":"2024-01-27T01:03:44.725203+0000","last_became_active":"2024-01-27T01:03:44.712614+0000","last_became_peered":"2024-01-27T01:03:44.712614+0000","last_unstale":"2024-01-27T01:03:44.725203+0000","last_undegraded":"2024-01-27T01:03:44.725203+0000","last_fullsized":"2024-01-27T01:03:44.725203+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:37:57.304264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.701928+0000","last_change":"2024-01-27T01:03:43.725915+0000","last_active":"2024-01-27T01:03:44.701928+0000","last_peered":"2024-01-27T01:03:44.701928+0000","last_clean":"2024-01-27T01:03:44.701928+0000","last_became_active":"2024-01-27T01:03:43.725624+0000","last_became_peered":"2024-01-27T01:03:43.725624+0000","last_unstale":"2024-01-27T01:03:44.701928+0000","last_undegraded":"2024-01-27T01:03:44.701928+0000","last_fullsized":"2024-01-27T01:03:44.701928+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:03:06.521968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.726293+0000","last_change":"2024-01-27T01:03:44.722018+0000","last_active":"2024-01-27T01:03:44.726293+0000","last_peered":"2024-01-27T01:03:44.726293+0000","last_clean":"2024-01-27T01:03:44.726293+0000","last_became_active":"2024-01-27T01:03:44.721602+0000","last_became_peered":"2024-01-27T01:03:44.721602+0000","last_unstale":"2024-01-27T01:03:44.726293+0000","last_undegraded":"2024-01-27T01:03:44.726293+0000","last_fullsized":"2024-01-27T01:03:44.726293+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:03:07.526894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"204'3","reported_seq":13,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.733551+0000","last_change":"2024-01-27T01:03:43.724110+0000","last_active":"2024-01-27T01:03:44.733551+0000","last_peered":"2024-01-27T01:03:44.733551+0000","last_clean":"2024-01-27T01:03:44.733551+0000","last_became_active":"2024-01-27T01:03:43.723813+0000","last_became_peered":"2024-01-27T01:03:43.723813+0000","last_unstale":"2024-01-27T01:03:44.733551+0000","last_undegraded":"2024-01-27T01:03:44.733551+0000","last_fullsized":"2024-01-27T01:03:44.733551+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:57:13.729066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.720951+0000","last_change":"2024-01-27T01:03:44.716587+0000","last_active":"2024-01-27T01:03:44.720951+0000","last_peered":"2024-01-27T01:03:44.720951+0000","last_clean":"2024-01-27T01:03:44.720951+0000","last_became_active":"2024-01-27T01:03:44.715931+0000","last_became_peered":"2024-01-27T01:03:44.715931+0000","last_unstale":"2024-01-27T01:03:44.720951+0000","last_undegraded":"2024-01-27T01:03:44.720951+0000","last_fullsized":"2024-01-27T01:03:44.720951+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:40:45.048895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"204'6","reported_seq":16,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.735434+0000","last_change":"2024-01-27T01:03:43.717696+0000","last_active":"2024-01-27T01:03:44.735434+0000","last_peered":"2024-01-27T01:03:44.735434+0000","last_clean":"2024-01-27T01:03:44.735434+0000","last_became_active":"2024-01-27T01:03:43.717449+0000","last_became_peered":"2024-01-27T01:03:43.717449+0000","last_unstale":"2024-01-27T01:03:44.735434+0000","last_undegraded":"2024-01-27T01:03:44.735434+0000","last_fullsized":"2024-01-27T01:03:44.735434+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:23.259629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.727234+0000","last_change":"2024-01-27T01:03:44.718112+0000","last_active":"2024-01-27T01:03:44.727234+0000","last_peered":"2024-01-27T01:03:44.727234+0000","last_clean":"2024-01-27T01:03:44.727234+0000","last_became_active":"2024-01-27T01:03:44.717066+0000","last_became_peered":"2024-01-27T01:03:44.717066+0000","last_unstale":"2024-01-27T01:03:44.727234+0000","last_undegraded":"2024-01-27T01:03:44.727234+0000","last_fullsized":"2024-01-27T01:03:44.727234+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:33:29.458783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"204'2","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.733415+0000","last_change":"2024-01-27T01:03:43.716408+0000","last_active":"2024-01-27T01:03:44.733415+0000","last_peered":"2024-01-27T01:03:44.733415+0000","last_clean":"2024-01-27T01:03:44.733415+0000","last_became_active":"2024-01-27T01:03:43.716151+0000","last_became_peered":"2024-01-27T01:03:43.716151+0000","last_unstale":"2024-01-27T01:03:44.733415+0000","last_undegraded":"2024-01-27T01:03:44.733415+0000","last_fullsized":"2024-01-27T01:03:44.733415+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:39:32.872628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.722357+0000","last_change":"2024-01-27T01:03:44.712162+0000","last_active":"2024-01-27T01:03:44.722357+0000","last_peered":"2024-01-27T01:03:44.722357+0000","last_clean":"2024-01-27T01:03:44.722357+0000","last_became_active":"2024-01-27T01:03:44.711786+0000","last_became_peered":"2024-01-27T01:03:44.711786+0000","last_unstale":"2024-01-27T01:03:44.722357+0000","last_undegraded":"2024-01-27T01:03:44.722357+0000","last_fullsized":"2024-01-27T01:03:44.722357+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:38:48.821118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.702448+0000","last_change":"2024-01-27T01:03:43.725752+0000","last_active":"2024-01-27T01:03:44.702448+0000","last_peered":"2024-01-27T01:03:44.702448+0000","last_clean":"2024-01-27T01:03:44.702448+0000","last_became_active":"2024-01-27T01:03:43.719772+0000","last_became_peered":"2024-01-27T01:03:43.719772+0000","last_unstale":"2024-01-27T01:03:44.702448+0000","last_undegraded":"2024-01-27T01:03:44.702448+0000","last_fullsized":"2024-01-27T01:03:44.702448+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:22:49.979346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.732178+0000","last_change":"2024-01-27T01:03:44.731427+0000","last_active":"2024-01-27T01:03:44.732178+0000","last_peered":"2024-01-27T01:03:44.732178+0000","last_clean":"2024-01-27T01:03:44.732178+0000","last_became_active":"2024-01-27T01:03:44.731128+0000","last_became_peered":"2024-01-27T01:03:44.731128+0000","last_unstale":"2024-01-27T01:03:44.732178+0000","last_undegraded":"2024-01-27T01:03:44.732178+0000","last_fullsized":"2024-01-27T01:03:44.732178+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:19:33.472237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.707093+0000","last_change":"2024-01-27T01:03:43.715244+0000","last_active":"2024-01-27T01:03:44.707093+0000","last_peered":"2024-01-27T01:03:44.707093+0000","last_clean":"2024-01-27T01:03:44.707093+0000","last_became_active":"2024-01-27T01:03:43.714956+0000","last_became_peered":"2024-01-27T01:03:43.714956+0000","last_unstale":"2024-01-27T01:03:44.707093+0000","last_undegraded":"2024-01-27T01:03:44.707093+0000","last_fullsized":"2024-01-27T01:03:44.707093+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:18:05.795670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.729831+0000","last_change":"2024-01-27T01:03:44.726440+0000","last_active":"2024-01-27T01:03:44.729831+0000","last_peered":"2024-01-27T01:03:44.729831+0000","last_clean":"2024-01-27T01:03:44.729831+0000","last_became_active":"2024-01-27T01:03:44.724748+0000","last_became_peered":"2024-01-27T01:03:44.724748+0000","last_unstale":"2024-01-27T01:03:44.729831+0000","last_undegraded":"2024-01-27T01:03:44.729831+0000","last_fullsized":"2024-01-27T01:03:44.729831+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:22.370885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"204'8","reported_seq":18,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.734892+0000","last_change":"2024-01-27T01:03:43.712094+0000","last_active":"2024-01-27T01:03:44.734892+0000","last_peered":"2024-01-27T01:03:44.734892+0000","last_clean":"2024-01-27T01:03:44.734892+0000","last_became_active":"2024-01-27T01:03:43.711811+0000","last_became_peered":"2024-01-27T01:03:43.711811+0000","last_unstale":"2024-01-27T01:03:44.734892+0000","last_undegraded":"2024-01-27T01:03:44.734892+0000","last_fullsized":"2024-01-27T01:03:44.734892+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:17:42.427577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.735646+0000","last_change":"2024-01-27T01:03:44.731801+0000","last_active":"2024-01-27T01:03:44.735646+0000","last_peered":"2024-01-27T01:03:44.735646+0000","last_clean":"2024-01-27T01:03:44.735646+0000","last_became_active":"2024-01-27T01:03:44.731183+0000","last_became_peered":"2024-01-27T01:03:44.731183+0000","last_unstale":"2024-01-27T01:03:44.735646+0000","last_undegraded":"2024-01-27T01:03:44.735646+0000","last_fullsized":"2024-01-27T01:03:44.735646+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:19.929140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"204'1","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.733460+0000","last_change":"2024-01-27T01:03:43.713381+0000","last_active":"2024-01-27T01:03:44.733460+0000","last_peered":"2024-01-27T01:03:44.733460+0000","last_clean":"2024-01-27T01:03:44.733460+0000","last_became_active":"2024-01-27T01:03:43.713110+0000","last_became_peered":"2024-01-27T01:03:43.713110+0000","last_unstale":"2024-01-27T01:03:44.733460+0000","last_undegraded":"2024-01-27T01:03:44.733460+0000","last_fullsized":"2024-01-27T01:03:44.733460+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:15:54.281492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.729204+0000","last_change":"2024-01-27T01:03:44.726121+0000","last_active":"2024-01-27T01:03:44.729204+0000","last_peered":"2024-01-27T01:03:44.729204+0000","last_clean":"2024-01-27T01:03:44.729204+0000","last_became_active":"2024-01-27T01:03:44.725028+0000","last_became_peered":"2024-01-27T01:03:44.725028+0000","last_unstale":"2024-01-27T01:03:44.729204+0000","last_undegraded":"2024-01-27T01:03:44.729204+0000","last_fullsized":"2024-01-27T01:03:44.729204+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:41:15.819900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"204'2","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.732909+0000","last_change":"2024-01-27T01:03:43.718894+0000","last_active":"2024-01-27T01:03:44.732909+0000","last_peered":"2024-01-27T01:03:44.732909+0000","last_clean":"2024-01-27T01:03:44.732909+0000","last_became_active":"2024-01-27T01:03:43.718609+0000","last_became_peered":"2024-01-27T01:03:43.718609+0000","last_unstale":"2024-01-27T01:03:44.732909+0000","last_undegraded":"2024-01-27T01:03:44.732909+0000","last_fullsized":"2024-01-27T01:03:44.732909+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:51:25.645321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.717415+0000","last_change":"2024-01-27T01:03:44.713061+0000","last_active":"2024-01-27T01:03:44.717415+0000","last_peered":"2024-01-27T01:03:44.717415+0000","last_clean":"2024-01-27T01:03:44.717415+0000","last_became_active":"2024-01-27T01:03:44.712508+0000","last_became_peered":"2024-01-27T01:03:44.712508+0000","last_unstale":"2024-01-27T01:03:44.717415+0000","last_undegraded":"2024-01-27T01:03:44.717415+0000","last_fullsized":"2024-01-27T01:03:44.717415+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:32:53.294651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"204'1","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.729629+0000","last_change":"2024-01-27T01:03:43.723623+0000","last_active":"2024-01-27T01:03:44.729629+0000","last_peered":"2024-01-27T01:03:44.729629+0000","last_clean":"2024-01-27T01:03:44.729629+0000","last_became_active":"2024-01-27T01:03:43.723348+0000","last_became_peered":"2024-01-27T01:03:43.723348+0000","last_unstale":"2024-01-27T01:03:44.729629+0000","last_undegraded":"2024-01-27T01:03:44.729629+0000","last_fullsized":"2024-01-27T01:03:44.729629+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:17:42.427577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.730073+0000","last_change":"2024-01-27T01:03:44.724054+0000","last_active":"2024-01-27T01:03:44.730073+0000","last_peered":"2024-01-27T01:03:44.730073+0000","last_clean":"2024-01-27T01:03:44.730073+0000","last_became_active":"2024-01-27T01:03:44.723771+0000","last_became_peered":"2024-01-27T01:03:44.723771+0000","last_unstale":"2024-01-27T01:03:44.730073+0000","last_undegraded":"2024-01-27T01:03:44.730073+0000","last_fullsized":"2024-01-27T01:03:44.730073+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:28.400436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.702278+0000","last_change":"2024-01-27T01:03:43.716871+0000","last_active":"2024-01-27T01:03:44.702278+0000","last_peered":"2024-01-27T01:03:44.702278+0000","last_clean":"2024-01-27T01:03:44.702278+0000","last_became_active":"2024-01-27T01:03:43.716462+0000","last_became_peered":"2024-01-27T01:03:43.716462+0000","last_unstale":"2024-01-27T01:03:44.702278+0000","last_undegraded":"2024-01-27T01:03:44.702278+0000","last_fullsized":"2024-01-27T01:03:44.702278+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:23:07.323492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.729184+0000","last_change":"2024-01-27T01:03:44.722215+0000","last_active":"2024-01-27T01:03:44.729184+0000","last_peered":"2024-01-27T01:03:44.729184+0000","last_clean":"2024-01-27T01:03:44.729184+0000","last_became_active":"2024-01-27T01:03:44.721927+0000","last_became_peered":"2024-01-27T01:03:44.721927+0000","last_unstale":"2024-01-27T01:03:44.729184+0000","last_undegraded":"2024-01-27T01:03:44.729184+0000","last_fullsized":"2024-01-27T01:03:44.729184+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:54:30.312872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"204'2","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.734264+0000","last_change":"2024-01-27T01:03:43.725099+0000","last_active":"2024-01-27T01:03:44.734264+0000","last_peered":"2024-01-27T01:03:44.734264+0000","last_clean":"2024-01-27T01:03:44.734264+0000","last_became_active":"2024-01-27T01:03:43.724812+0000","last_became_peered":"2024-01-27T01:03:43.724812+0000","last_unstale":"2024-01-27T01:03:44.734264+0000","last_undegraded":"2024-01-27T01:03:44.734264+0000","last_fullsized":"2024-01-27T01:03:44.734264+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:32:31.472856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.733234+0000","last_change":"2024-01-27T01:03:44.725409+0000","last_active":"2024-01-27T01:03:44.733234+0000","last_peered":"2024-01-27T01:03:44.733234+0000","last_clean":"2024-01-27T01:03:44.733234+0000","last_became_active":"2024-01-27T01:03:44.725120+0000","last_became_peered":"2024-01-27T01:03:44.725120+0000","last_unstale":"2024-01-27T01:03:44.733234+0000","last_undegraded":"2024-01-27T01:03:44.733234+0000","last_fullsized":"2024-01-27T01:03:44.733234+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:51:28.069637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.711817+0000","last_change":"2024-01-27T01:03:43.709415+0000","last_active":"2024-01-27T01:03:44.711817+0000","last_peered":"2024-01-27T01:03:44.711817+0000","last_clean":"2024-01-27T01:03:44.711817+0000","last_became_active":"2024-01-27T01:03:43.709067+0000","last_became_peered":"2024-01-27T01:03:43.709067+0000","last_unstale":"2024-01-27T01:03:44.711817+0000","last_undegraded":"2024-01-27T01:03:44.711817+0000","last_fullsized":"2024-01-27T01:03:44.711817+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:14:39.132003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.733673+0000","last_change":"2024-01-27T01:03:44.730564+0000","last_active":"2024-01-27T01:03:44.733673+0000","last_peered":"2024-01-27T01:03:44.733673+0000","last_clean":"2024-01-27T01:03:44.733673+0000","last_became_active":"2024-01-27T01:03:44.730270+0000","last_became_peered":"2024-01-27T01:03:44.730270+0000","last_unstale":"2024-01-27T01:03:44.733673+0000","last_undegraded":"2024-01-27T01:03:44.733673+0000","last_fullsized":"2024-01-27T01:03:44.733673+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:58:37.015025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.717470+0000","last_change":"2024-01-27T01:03:43.721493+0000","last_active":"2024-01-27T01:03:44.717470+0000","last_peered":"2024-01-27T01:03:44.717470+0000","last_clean":"2024-01-27T01:03:44.717470+0000","last_became_active":"2024-01-27T01:03:43.721147+0000","last_became_peered":"2024-01-27T01:03:43.721147+0000","last_unstale":"2024-01-27T01:03:44.717470+0000","last_undegraded":"2024-01-27T01:03:44.717470+0000","last_fullsized":"2024-01-27T01:03:44.717470+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:21:32.902395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.734113+0000","last_change":"2024-01-27T01:03:44.729906+0000","last_active":"2024-01-27T01:03:44.734113+0000","last_peered":"2024-01-27T01:03:44.734113+0000","last_clean":"2024-01-27T01:03:44.734113+0000","last_became_active":"2024-01-27T01:03:44.729622+0000","last_became_peered":"2024-01-27T01:03:44.729622+0000","last_unstale":"2024-01-27T01:03:44.734113+0000","last_undegraded":"2024-01-27T01:03:44.734113+0000","last_fullsized":"2024-01-27T01:03:44.734113+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:53:11.353764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.708745+0000","last_change":"2024-01-27T01:03:43.715588+0000","last_active":"2024-01-27T01:03:44.708745+0000","last_peered":"2024-01-27T01:03:44.708745+0000","last_clean":"2024-01-27T01:03:44.708745+0000","last_became_active":"2024-01-27T01:03:43.715312+0000","last_became_peered":"2024-01-27T01:03:43.715312+0000","last_unstale":"2024-01-27T01:03:44.708745+0000","last_undegraded":"2024-01-27T01:03:44.708745+0000","last_fullsized":"2024-01-27T01:03:44.708745+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:23:07.323492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.734777+0000","last_change":"2024-01-27T01:03:44.731519+0000","last_active":"2024-01-27T01:03:44.734777+0000","last_peered":"2024-01-27T01:03:44.734777+0000","last_clean":"2024-01-27T01:03:44.734777+0000","last_became_active":"2024-01-27T01:03:44.727396+0000","last_became_peered":"2024-01-27T01:03:44.727396+0000","last_unstale":"2024-01-27T01:03:44.734777+0000","last_undegraded":"2024-01-27T01:03:44.734777+0000","last_fullsized":"2024-01-27T01:03:44.734777+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:50:28.451748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"204'2","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.733193+0000","last_change":"2024-01-27T01:03:43.715941+0000","last_active":"2024-01-27T01:03:44.733193+0000","last_peered":"2024-01-27T01:03:44.733193+0000","last_clean":"2024-01-27T01:03:44.733193+0000","last_became_active":"2024-01-27T01:03:43.715648+0000","last_became_peered":"2024-01-27T01:03:43.715648+0000","last_unstale":"2024-01-27T01:03:44.733193+0000","last_undegraded":"2024-01-27T01:03:44.733193+0000","last_fullsized":"2024-01-27T01:03:44.733193+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:43:32.112762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.736148+0000","last_change":"2024-01-27T01:03:44.733916+0000","last_active":"2024-01-27T01:03:44.736148+0000","last_peered":"2024-01-27T01:03:44.736148+0000","last_clean":"2024-01-27T01:03:44.736148+0000","last_became_active":"2024-01-27T01:03:44.733528+0000","last_became_peered":"2024-01-27T01:03:44.733528+0000","last_unstale":"2024-01-27T01:03:44.736148+0000","last_undegraded":"2024-01-27T01:03:44.736148+0000","last_fullsized":"2024-01-27T01:03:44.736148+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:58:04.855152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.719031+0000","last_change":"2024-01-27T01:03:43.727716+0000","last_active":"2024-01-27T01:03:44.719031+0000","last_peered":"2024-01-27T01:03:44.719031+0000","last_clean":"2024-01-27T01:03:44.719031+0000","last_became_active":"2024-01-27T01:03:43.727384+0000","last_became_peered":"2024-01-27T01:03:43.727384+0000","last_unstale":"2024-01-27T01:03:44.719031+0000","last_undegraded":"2024-01-27T01:03:44.719031+0000","last_fullsized":"2024-01-27T01:03:44.719031+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:54:34.708513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.718690+0000","last_change":"2024-01-27T01:03:44.715180+0000","last_active":"2024-01-27T01:03:44.718690+0000","last_peered":"2024-01-27T01:03:44.718690+0000","last_clean":"2024-01-27T01:03:44.718690+0000","last_became_active":"2024-01-27T01:03:44.714610+0000","last_became_peered":"2024-01-27T01:03:44.714610+0000","last_unstale":"2024-01-27T01:03:44.718690+0000","last_undegraded":"2024-01-27T01:03:44.718690+0000","last_fullsized":"2024-01-27T01:03:44.718690+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:21:36.129471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.704439+0000","last_change":"2024-01-27T01:03:43.720074+0000","last_active":"2024-01-27T01:03:44.704439+0000","last_peered":"2024-01-27T01:03:44.704439+0000","last_clean":"2024-01-27T01:03:44.704439+0000","last_became_active":"2024-01-27T01:03:43.719776+0000","last_became_peered":"2024-01-27T01:03:43.719776+0000","last_unstale":"2024-01-27T01:03:44.704439+0000","last_undegraded":"2024-01-27T01:03:44.704439+0000","last_fullsized":"2024-01-27T01:03:44.704439+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:21:08.089077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.734914+0000","last_change":"2024-01-27T01:03:44.727464+0000","last_active":"2024-01-27T01:03:44.734914+0000","last_peered":"2024-01-27T01:03:44.734914+0000","last_clean":"2024-01-27T01:03:44.734914+0000","last_became_active":"2024-01-27T01:03:44.727139+0000","last_became_peered":"2024-01-27T01:03:44.727139+0000","last_unstale":"2024-01-27T01:03:44.734914+0000","last_undegraded":"2024-01-27T01:03:44.734914+0000","last_fullsized":"2024-01-27T01:03:44.734914+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:56:17.799789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.707731+0000","last_change":"2024-01-27T01:03:43.727865+0000","last_active":"2024-01-27T01:03:44.707731+0000","last_peered":"2024-01-27T01:03:44.707731+0000","last_clean":"2024-01-27T01:03:44.707731+0000","last_became_active":"2024-01-27T01:03:43.727581+0000","last_became_peered":"2024-01-27T01:03:43.727581+0000","last_unstale":"2024-01-27T01:03:44.707731+0000","last_undegraded":"2024-01-27T01:03:44.707731+0000","last_fullsized":"2024-01-27T01:03:44.707731+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:41:00.700232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.731794+0000","last_change":"2024-01-27T01:03:44.731312+0000","last_active":"2024-01-27T01:03:44.731794+0000","last_peered":"2024-01-27T01:03:44.731794+0000","last_clean":"2024-01-27T01:03:44.731794+0000","last_became_active":"2024-01-27T01:03:44.731055+0000","last_became_peered":"2024-01-27T01:03:44.731055+0000","last_unstale":"2024-01-27T01:03:44.731794+0000","last_undegraded":"2024-01-27T01:03:44.731794+0000","last_fullsized":"2024-01-27T01:03:44.731794+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:03:07.526894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.706810+0000","last_change":"2024-01-27T01:03:43.720294+0000","last_active":"2024-01-27T01:03:44.706810+0000","last_peered":"2024-01-27T01:03:44.706810+0000","last_clean":"2024-01-27T01:03:44.706810+0000","last_became_active":"2024-01-27T01:03:43.720011+0000","last_became_peered":"2024-01-27T01:03:43.720011+0000","last_unstale":"2024-01-27T01:03:44.706810+0000","last_undegraded":"2024-01-27T01:03:44.706810+0000","last_fullsized":"2024-01-27T01:03:44.706810+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:19:32.467311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.727867+0000","last_change":"2024-01-27T01:03:44.723224+0000","last_active":"2024-01-27T01:03:44.727867+0000","last_peered":"2024-01-27T01:03:44.727867+0000","last_clean":"2024-01-27T01:03:44.727867+0000","last_became_active":"2024-01-27T01:03:44.721825+0000","last_became_peered":"2024-01-27T01:03:44.721825+0000","last_unstale":"2024-01-27T01:03:44.727867+0000","last_undegraded":"2024-01-27T01:03:44.727867+0000","last_fullsized":"2024-01-27T01:03:44.727867+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:03:41.778611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.706007+0000","last_change":"2024-01-27T01:03:43.724947+0000","last_active":"2024-01-27T01:03:44.706007+0000","last_peered":"2024-01-27T01:03:44.706007+0000","last_clean":"2024-01-27T01:03:44.706007+0000","last_became_active":"2024-01-27T01:03:43.724646+0000","last_became_peered":"2024-01-27T01:03:43.724646+0000","last_unstale":"2024-01-27T01:03:44.706007+0000","last_undegraded":"2024-01-27T01:03:44.706007+0000","last_fullsized":"2024-01-27T01:03:44.706007+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:22:49.979346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.731901+0000","last_change":"2024-01-27T01:03:44.722316+0000","last_active":"2024-01-27T01:03:44.731901+0000","last_peered":"2024-01-27T01:03:44.731901+0000","last_clean":"2024-01-27T01:03:44.731901+0000","last_became_active":"2024-01-27T01:03:44.722046+0000","last_became_peered":"2024-01-27T01:03:44.722046+0000","last_unstale":"2024-01-27T01:03:44.731901+0000","last_undegraded":"2024-01-27T01:03:44.731901+0000","last_fullsized":"2024-01-27T01:03:44.731901+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:18:09.693234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.717596+0000","last_change":"2024-01-27T01:03:43.713126+0000","last_active":"2024-01-27T01:03:44.717596+0000","last_peered":"2024-01-27T01:03:44.717596+0000","last_clean":"2024-01-27T01:03:44.717596+0000","last_became_active":"2024-01-27T01:03:43.712850+0000","last_became_peered":"2024-01-27T01:03:43.712850+0000","last_unstale":"2024-01-27T01:03:44.717596+0000","last_undegraded":"2024-01-27T01:03:44.717596+0000","last_fullsized":"2024-01-27T01:03:44.717596+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:41:52.241070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.718977+0000","last_change":"2024-01-27T01:03:44.717845+0000","last_active":"2024-01-27T01:03:44.718977+0000","last_peered":"2024-01-27T01:03:44.718977+0000","last_clean":"2024-01-27T01:03:44.718977+0000","last_became_active":"2024-01-27T01:03:44.717539+0000","last_became_peered":"2024-01-27T01:03:44.717539+0000","last_unstale":"2024-01-27T01:03:44.718977+0000","last_undegraded":"2024-01-27T01:03:44.718977+0000","last_fullsized":"2024-01-27T01:03:44.718977+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:28:50.109807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.708843+0000","last_change":"2024-01-27T01:03:43.719995+0000","last_active":"2024-01-27T01:03:44.708843+0000","last_peered":"2024-01-27T01:03:44.708843+0000","last_clean":"2024-01-27T01:03:44.708843+0000","last_became_active":"2024-01-27T01:03:43.719699+0000","last_became_peered":"2024-01-27T01:03:43.719699+0000","last_unstale":"2024-01-27T01:03:44.708843+0000","last_undegraded":"2024-01-27T01:03:44.708843+0000","last_fullsized":"2024-01-27T01:03:44.708843+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:15:54.281492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.722470+0000","last_change":"2024-01-27T01:03:44.718614+0000","last_active":"2024-01-27T01:03:44.722470+0000","last_peered":"2024-01-27T01:03:44.722470+0000","last_clean":"2024-01-27T01:03:44.722470+0000","last_became_active":"2024-01-27T01:03:44.718383+0000","last_became_peered":"2024-01-27T01:03:44.718383+0000","last_unstale":"2024-01-27T01:03:44.722470+0000","last_undegraded":"2024-01-27T01:03:44.722470+0000","last_fullsized":"2024-01-27T01:03:44.722470+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:43.691046+0000","last_clean_scrub_stamp":"2024-01-27T01:03:43.691046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:22.370885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":10,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.707232+0000","last_change":"2024-01-27T01:03:43.713997+0000","last_active":"2024-01-27T01:03:44.707232+0000","last_peered":"2024-01-27T01:03:44.707232+0000","last_clean":"2024-01-27T01:03:44.707232+0000","last_became_active":"2024-01-27T01:03:43.713668+0000","last_became_peered":"2024-01-27T01:03:43.713668+0000","last_unstale":"2024-01-27T01:03:44.707232+0000","last_undegraded":"2024-01-27T01:03:44.707232+0000","last_fullsized":"2024-01-27T01:03:44.707232+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:03:42.686120+0000","last_clean_scrub_stamp":"2024-01-27T01:03:42.686120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:13:33.372407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"195'8728","reported_seq":9570,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-01-27T01:03:44.707680+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:03:44.707680+0000","last_peered":"2024-01-27T01:03:44.707680+0000","last_clean":"2024-01-27T01:03:44.707680+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:03:44.707680+0000","last_undegraded":"2024-01-27T01:03:44.707680+0000","last_fullsized":"2024-01-27T01:03:44.707680+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8728,"log_dups_size":0,"ondisk_log_size":8728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2072,"num_read_kb":4396,"num_write":590,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2072,"num_read_kb":4396,"num_write":590,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8728,"ondisk_log_size":8728,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575971,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300380,"kb_used_data":3092,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071460,"statfs":{"total":96636764160,"available":96329175040,"internally_reserved":0,"allocated":3166208,"data_stored":1937780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.622,"15min":0.61199999999999999},"min":{"1min":0.56699999999999995,"5min":0.41299999999999998,"15min":0.312},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.626,"15min":0.62},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.33000000000000002},"max":{"1min":0.76900000000000002,"5min":0.78100000000000003,"15min":0.86399999999999999},"last":0.498}]},{"osd":1,"last update":"Sat Jan 27 01:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.622,"15min":0.61799999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.64900000000000002,"15min":0.63400000000000001},"min":{"1min":0.53300000000000003,"5min":0.45400000000000001,"15min":0.34300000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.89100000000000001},"last":0.55400000000000005}]},{"osd":2,"last update":"Sat Jan 27 01:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.62,"15min":0.60999999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.39200000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.86599999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64300000000000002,"15min":0.63},"min":{"1min":0.44700000000000001,"5min":0.433,"15min":0.40400000000000003},"max":{"1min":0.85699999999999998,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.56999999999999995}]},{"osd":3,"last update":"Sat Jan 27 01:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.627,"15min":0.62},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.39100000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.51100000000000001,"5min":0.41099999999999998,"15min":0.40899999999999997},"max":{"1min":0.79200000000000004,"5min":0.85799999999999998,"15min":0.88},"last":0.59899999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.48999999999999999,"15min":0.48399999999999999},"min":{"1min":0.35099999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72699999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.432},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.38400000000000001,"5min":0.34599999999999997,"15min":0.32200000000000001},"max":{"1min":0.77300000000000002,"5min":0.77700000000000002,"15min":0.86599999999999999},"last":0.53500000000000003}]},{"osd":5,"last update":"Sat Jan 27 01:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.52500000000000002,"15min":0.51600000000000001},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.32600000000000001},"max":{"1min":0.76200000000000001,"5min":0.80400000000000005,"15min":0.84999999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56299999999999994,"15min":0.56200000000000006},"min":{"1min":0.42499999999999999,"5min":0.34399999999999997,"15min":0.33700000000000002},"max":{"1min":0.81499999999999995,"5min":0.84199999999999997,"15min":0.84299999999999997},"last":0.58599999999999997}]},{"osd":6,"last update":"Sat Jan 27 01:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.58399999999999996},"min":{"1min":0.46300000000000002,"5min":0.38900000000000001,"15min":0.37},"max":{"1min":0.80000000000000004,"5min":0.81399999999999995,"15min":0.84799999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.59799999999999998,"15min":0.57999999999999996},"min":{"1min":0.443,"5min":0.371,"15min":0.35699999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.44800000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575971,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307996,"kb_used_data":2516,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063844,"statfs":{"total":96636764160,"available":96321376256,"internally_reserved":0,"allocated":2576384,"data_stored":1347482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.45600000000000002,"5min":0.38600000000000001,"15min":0.36399999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.86399999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.63,"15min":0.62},"min":{"1min":0.47099999999999997,"5min":0.435,"15min":0.376},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.86299999999999999},"last":0.57299999999999995}]},{"osd":1,"last update":"Sat Jan 27 01:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.61799999999999999,"15min":0.61099999999999999},"min":{"1min":0.45100000000000001,"5min":0.44500000000000001,"15min":0.377},"max":{"1min":0.78800000000000003,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.626,"15min":0.624},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.83999999999999997},"last":0.59099999999999997}]},{"osd":2,"last update":"Sat Jan 27 01:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63300000000000001,"15min":0.626},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.40899999999999997},"max":{"1min":0.77100000000000002,"5min":0.80600000000000005,"15min":0.82399999999999995},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63300000000000001,"15min":0.625},"min":{"1min":0.48099999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.753,"5min":0.878,"15min":0.878},"last":0.60299999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.624,"15min":0.629},"min":{"1min":0.498,"5min":0.442,"15min":0.43099999999999999},"max":{"1min":0.78100000000000003,"5min":0.82599999999999996,"15min":0.90100000000000002},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.67300000000000004,"15min":0.66300000000000003},"min":{"1min":0.48399999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.79900000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.624}]},{"osd":4,"last update":"Sat Jan 27 01:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.5,"15min":0.502},"min":{"1min":0.371,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.86599999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.36599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.88900000000000001},"last":0.46400000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.53200000000000003,"15min":0.52200000000000002},"min":{"1min":0.38700000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.75900000000000001,"5min":0.77200000000000002,"15min":0.80500000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57899999999999996,"15min":0.56999999999999995},"min":{"1min":0.38800000000000001,"5min":0.38200000000000001,"15min":0.28899999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.874},"last":0.54300000000000004}]},{"osd":7,"last update":"Sat Jan 27 01:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57599999999999996,"15min":0.57399999999999995},"min":{"1min":0.41999999999999998,"5min":0.373,"15min":0.35699999999999998},"max":{"1min":0.83499999999999996,"5min":0.84399999999999997,"15min":0.92900000000000005},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59599999999999997,"15min":0.59099999999999997},"min":{"1min":0.438,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.83399999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.503}]}]},{"osd":1,"up_from":13,"seq":55834575971,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299800,"kb_used_data":2512,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072040,"statfs":{"total":96636764160,"available":96329768960,"internally_reserved":0,"allocated":2572288,"data_stored":1347412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41299999999999998,"15min":0.42299999999999999},"min":{"1min":0.28799999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.55800000000000005,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.317},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45900000000000002,"15min":0.45800000000000002},"min":{"1min":0.28399999999999997,"5min":0.27900000000000003,"15min":0.26800000000000002},"max":{"1min":0.67800000000000005,"5min":0.73999999999999999,"15min":0.76600000000000001},"last":0.41199999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.443,"15min":0.442},"min":{"1min":0.32800000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.495,"15min":0.49399999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.28399999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.39500000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53300000000000003,"15min":0.53600000000000003},"min":{"1min":0.46000000000000002,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":1.2130000000000001},"last":0.437},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.35799999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.125,"5min":1.125,"15min":1.22},"last":0.47099999999999997}]},{"osd":4,"last update":"Sat Jan 27 01:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.66900000000000004,"15min":0.67700000000000005},"min":{"1min":0.43099999999999999,"5min":0.42999999999999999,"15min":0.39800000000000002},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.70099999999999996,"15min":0.69999999999999996},"min":{"1min":0.57199999999999995,"5min":0.52800000000000002,"15min":0.35699999999999998},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.72199999999999998}]},{"osd":5,"last update":"Sat Jan 27 01:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70499999999999996,"15min":0.70199999999999996},"min":{"1min":0.53200000000000003,"5min":0.50900000000000001,"15min":0.44500000000000001},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.69799999999999995,"15min":0.70499999999999996},"min":{"1min":0.55000000000000004,"5min":0.47599999999999998,"15min":0.41999999999999998},"max":{"1min":1.079,"5min":1.079,"15min":1.119},"last":0.67400000000000004}]},{"osd":6,"last update":"Sat Jan 27 01:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.71999999999999997,"15min":0.71599999999999997},"min":{"1min":0.56000000000000005,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.70899999999999996,"15min":0.70999999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.435},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.59999999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.751,"15min":0.75600000000000001},"min":{"1min":0.55000000000000004,"5min":0.51100000000000001,"15min":0.47599999999999998},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.76700000000000002,"15min":0.76500000000000001},"min":{"1min":0.61299999999999999,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.76300000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575970,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316768,"kb_used_data":3096,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055072,"statfs":{"total":96636764160,"available":96312393728,"internally_reserved":0,"allocated":3170304,"data_stored":1931408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42599999999999999,"15min":0.42299999999999999},"min":{"1min":0.313,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.46899999999999997,"15min":0.46200000000000002},"min":{"1min":0.33300000000000002,"5min":0.313,"15min":0.26200000000000001},"max":{"1min":0.60899999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.55800000000000005}]},{"osd":2,"last update":"Sat Jan 27 01:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.44900000000000001,"15min":0.437},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.27300000000000002},"max":{"1min":0.53000000000000003,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.43099999999999999}]},{"osd":3,"last update":"Sat Jan 27 01:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.496,"15min":0.48699999999999999},"min":{"1min":0.36899999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.57099999999999995,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.40999999999999998,"5min":0.35999999999999999,"15min":0.33900000000000002},"max":{"1min":0.66300000000000003,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.48299999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67300000000000004,"15min":0.67700000000000005},"min":{"1min":0.5,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.91500000000000004,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.51300000000000001,"5min":0.47099999999999997,"15min":0.437},"max":{"1min":0.78000000000000003,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.72199999999999998}]},{"osd":5,"last update":"Sat Jan 27 01:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.70199999999999996,"15min":0.70299999999999996},"min":{"1min":0.51000000000000001,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.98199999999999998,"5min":2.165,"15min":2.165},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.71099999999999997,"15min":0.71599999999999997},"min":{"1min":0.48799999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.86499999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.77100000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.70999999999999996,"15min":0.71299999999999997},"min":{"1min":0.52600000000000002,"5min":0.48399999999999999,"15min":0.47699999999999998},"max":{"1min":0.83399999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.749,"15min":0.74399999999999999},"min":{"1min":0.57499999999999996,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0109999999999999,"5min":1.425,"15min":1.425},"last":0.84799999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.76500000000000001,"15min":0.76300000000000001},"min":{"1min":0.60399999999999998,"5min":0.52700000000000002,"15min":0.50900000000000001},"max":{"1min":0.95099999999999996,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.76600000000000001,"15min":0.77000000000000002},"min":{"1min":0.57399999999999995,"5min":0.52700000000000002,"15min":0.48499999999999999},"max":{"1min":0.94299999999999995,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.82799999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575970,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312092,"kb_used_data":2516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059748,"statfs":{"total":96636764160,"available":96317181952,"internally_reserved":0,"allocated":2576384,"data_stored":1347966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40600000000000003,"15min":0.41299999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.52600000000000002,"5min":0.746,"15min":0.78700000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.44800000000000001,"15min":0.44900000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.58799999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56399999999999995}]},{"osd":1,"last update":"Sat Jan 27 01:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.27000000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.51300000000000001,"15min":0.52400000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.78100000000000003},"last":0.64300000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.499,"15min":0.501},"min":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.495},"min":{"1min":0.33100000000000002,"5min":0.32800000000000001,"15min":0.31900000000000001},"max":{"1min":0.61299999999999999,"5min":0.66600000000000004,"15min":0.75700000000000001},"last":0.59099999999999997}]},{"osd":4,"last update":"Sat Jan 27 01:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66700000000000004,"15min":0.67900000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.442},"max":{"1min":0.88200000000000001,"5min":0.97599999999999998,"15min":1.091},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68899999999999995,"15min":0.69899999999999995},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.378},"max":{"1min":0.81799999999999995,"5min":0.89900000000000002,"15min":1.107},"last":0.81399999999999995}]},{"osd":5,"last update":"Sat Jan 27 01:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68899999999999995,"15min":0.70699999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.89500000000000002,"5min":0.995,"15min":1.167},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.71699999999999997,"15min":0.71599999999999997},"min":{"1min":0.46300000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.97499999999999998,"5min":1.0229999999999999,"15min":1.05},"last":0.93100000000000005}]},{"osd":6,"last update":"Sat Jan 27 01:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.71399999999999997,"15min":0.71599999999999997},"min":{"1min":0.50800000000000001,"5min":0.46700000000000003,"15min":0.45500000000000002},"max":{"1min":0.97899999999999998,"5min":0.996,"15min":1.077},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.71099999999999997,"15min":0.72599999999999998},"min":{"1min":0.53000000000000003,"5min":0.48199999999999998,"15min":0.39200000000000002},"max":{"1min":0.84199999999999997,"5min":0.96699999999999997,"15min":1.135},"last":0.66200000000000003}]},{"osd":7,"last update":"Sat Jan 27 01:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.745,"15min":0.752},"min":{"1min":0.50800000000000001,"5min":0.501,"15min":0.47099999999999997},"max":{"1min":0.89300000000000002,"5min":1.04,"15min":1.198},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.76600000000000001,"15min":0.77400000000000002},"min":{"1min":0.56499999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.94499999999999995,"5min":1.054,"15min":1.0880000000000001},"last":0.85799999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575969,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303904,"kb_used_data":2520,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067936,"statfs":{"total":96636764160,"available":96325566464,"internally_reserved":0,"allocated":2580480,"data_stored":1347572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:03:32 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40699999999999997,"15min":0.40999999999999998},"min":{"1min":0.28299999999999997,"5min":0.28199999999999997,"15min":0.26000000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.63700000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.441,"15min":0.45300000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.52500000000000002,"5min":0.64700000000000002,"15min":0.84699999999999998},"last":0.34100000000000003}]},{"osd":1,"last update":"Sat Jan 27 01:03:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47299999999999998,"15min":0.48299999999999998},"min":{"1min":0.33900000000000002,"5min":0.28899999999999998,"15min":0.27400000000000002},"max":{"1min":0.70199999999999996,"5min":1.012,"15min":1.0600000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.51100000000000001,"15min":0.51300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.33200000000000002},"max":{"1min":0.68200000000000005,"5min":0.995,"15min":1.018},"last":0.45700000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:03:32 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.495,"15min":0.5},"min":{"1min":0.36399999999999999,"5min":0.35699999999999998,"15min":0.34599999999999997},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.96499999999999997},"last":0.372},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47799999999999998,"15min":0.49299999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64900000000000002,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.54700000000000004}]},{"osd":4,"last update":"Sat Jan 27 01:03:32 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.69499999999999995,"15min":0.70399999999999996},"min":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.439},"max":{"1min":0.88700000000000001,"5min":1.089,"15min":1.089},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.67500000000000004,"15min":0.69599999999999995},"min":{"1min":0.48799999999999999,"5min":0.46000000000000002,"15min":0.42199999999999999},"max":{"1min":0.94599999999999995,"5min":1.069,"15min":1.069},"last":0.69799999999999995}]},{"osd":5,"last update":"Sat Jan 27 01:03:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68899999999999995,"15min":0.69999999999999996},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.46200000000000002},"max":{"1min":0.86699999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.68999999999999995,"15min":0.71099999999999997},"min":{"1min":0.54600000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.98599999999999999,"5min":1.123,"15min":1.123},"last":0.56499999999999995}]},{"osd":6,"last update":"Sat Jan 27 01:03:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69799999999999995,"15min":0.70699999999999996},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.44700000000000001},"max":{"1min":0.93100000000000005,"5min":0.98099999999999998,"15min":1.0589999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71499999999999997,"15min":0.73399999999999999},"min":{"1min":0.54500000000000004,"5min":0.52700000000000002,"15min":0.49199999999999999},"max":{"1min":0.89100000000000001,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.65100000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:03:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.70399999999999996,"15min":0.72299999999999998},"min":{"1min":0.55900000000000005,"5min":0.52500000000000002,"15min":0.45600000000000002},"max":{"1min":0.94999999999999996,"5min":1.157,"15min":1.157},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76300000000000001,"15min":0.77200000000000002},"min":{"1min":0.63100000000000001,"5min":0.55500000000000005,"15min":0.499},"max":{"1min":0.95899999999999996,"5min":1.103,"15min":1.103},"last":0.75700000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575971,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312104,"kb_used_data":2528,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059736,"statfs":{"total":96636764160,"available":96317169664,"internally_reserved":0,"allocated":2588672,"data_stored":1348982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.624,"15min":0.60899999999999999},"min":{"1min":0.441,"5min":0.40799999999999997,"15min":0.30299999999999999},"max":{"1min":0.69699999999999995,"5min":0.80000000000000004,"15min":0.84199999999999997},"last":0.377},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.61199999999999999,"15min":0.60999999999999999},"min":{"1min":0.44500000000000001,"5min":0.40000000000000002,"15min":0.35799999999999998},"max":{"1min":0.73199999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.67600000000000005}]},{"osd":1,"last update":"Sat Jan 27 01:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.63400000000000001,"15min":0.626},"min":{"1min":0.48499999999999999,"5min":0.38800000000000001,"15min":0.378},"max":{"1min":0.755,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.621,"15min":0.61499999999999999},"min":{"1min":0.502,"5min":0.42099999999999999,"15min":0.35799999999999998},"max":{"1min":0.72499999999999998,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.45300000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60099999999999998,"15min":0.59799999999999998},"min":{"1min":0.42999999999999999,"5min":0.40400000000000003,"15min":0.35299999999999998},"max":{"1min":0.70299999999999996,"5min":0.872,"15min":0.88700000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63800000000000001,"15min":0.628},"min":{"1min":0.51400000000000001,"5min":0.47999999999999998,"15min":0.40200000000000002},"max":{"1min":0.71699999999999997,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.60699999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.63200000000000001,"15min":0.61899999999999999},"min":{"1min":0.49099999999999999,"5min":0.41499999999999998,"15min":0.38200000000000001},"max":{"1min":0.76900000000000002,"5min":0.83599999999999997,"15min":0.84199999999999997},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.62,"15min":0.61899999999999999},"min":{"1min":0.48399999999999999,"5min":0.42399999999999999,"15min":0.40100000000000002},"max":{"1min":0.71199999999999997,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.58299999999999996}]},{"osd":5,"last update":"Sat Jan 27 01:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.48299999999999998,"15min":0.47799999999999998},"min":{"1min":0.34100000000000003,"5min":0.32900000000000001,"15min":0.32000000000000001},"max":{"1min":0.66400000000000003,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.503,"15min":0.5},"min":{"1min":0.35099999999999998,"5min":0.312,"15min":0.30399999999999999},"max":{"1min":0.73199999999999998,"5min":0.878,"15min":0.878},"last":0.499}]},{"osd":6,"last update":"Sat Jan 27 01:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.495,"15min":0.48999999999999999},"min":{"1min":0.379,"5min":0.35799999999999998,"15min":0.317},"max":{"1min":0.70099999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54600000000000004,"15min":0.54100000000000004},"min":{"1min":0.39000000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.73299999999999998,"5min":0.79300000000000004,"15min":0.81499999999999995},"last":0.65600000000000003}]},{"osd":7,"last update":"Sat Jan 27 01:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56699999999999995,"15min":0.55700000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36199999999999999},"max":{"1min":0.80400000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.746,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.624}]}]},{"osd":5,"up_from":13,"seq":55834575971,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312104,"kb_used_data":2528,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059736,"statfs":{"total":96636764160,"available":96317169664,"internally_reserved":0,"allocated":2588672,"data_stored":1348970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61199999999999999,"15min":0.60699999999999998},"min":{"1min":0.48199999999999998,"5min":0.40500000000000003,"15min":0.27300000000000002},"max":{"1min":0.77500000000000002,"5min":1.006,"15min":1.006},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.45200000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.77100000000000002,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.60199999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66500000000000004,"15min":0.65800000000000003},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.41399999999999998},"max":{"1min":0.876,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.502},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.55400000000000005,"5min":0.377,"15min":0.377},"max":{"1min":0.86399999999999999,"5min":1.212,"15min":1.212},"last":0.65400000000000003}]},{"osd":2,"last update":"Sat Jan 27 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75,"5min":1.631,"15min":1.631},"last":0.629},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.625,"15min":0.624},"min":{"1min":0.39300000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80800000000000005,"5min":0.82599999999999996,"15min":0.83299999999999996},"last":0.55100000000000005}]},{"osd":3,"last update":"Sat Jan 27 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60899999999999999,"15min":0.60399999999999998},"min":{"1min":0.42099999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.755,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64000000000000001,"15min":0.63400000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.84999999999999998,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.52500000000000002}]},{"osd":4,"last update":"Sat Jan 27 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.502,"15min":0.48199999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.317},"max":{"1min":0.76100000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.62},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.50900000000000001},"min":{"1min":0.371,"5min":0.34100000000000003,"15min":0.32500000000000001},"max":{"1min":0.747,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.69399999999999995}]},{"osd":6,"last update":"Sat Jan 27 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52500000000000002,"15min":0.501},"min":{"1min":0.372,"5min":0.314,"15min":0.314},"max":{"1min":0.82299999999999995,"5min":1.732,"15min":1.732},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.55100000000000005,"15min":0.53700000000000003},"min":{"1min":0.45200000000000001,"5min":0.374,"15min":0.33500000000000002},"max":{"1min":0.82599999999999996,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.64800000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58099999999999996,"15min":0.56499999999999995},"min":{"1min":0.434,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.628,"15min":0.60699999999999998},"min":{"1min":0.48399999999999999,"5min":0.40300000000000002,"15min":0.39100000000000001},"max":{"1min":0.85999999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.70699999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:03:49.325 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-27T01:03:49.325 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-27T01:03:49.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.502+0000 7f0a26b1d700 1 Processor -- start 2024-01-27T01:03:49.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.502+0000 7f0a26b1d700 1 -- start start 2024-01-27T01:03:49.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.503+0000 7f0a26b1d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0a200c93a0 0x7f0a200c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.503+0000 7f0a26b1d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a20067d50 0x7f0a20159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.503+0000 7f0a26b1d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2015e390 con 0x7f0a200681c0 2024-01-27T01:03:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.503+0000 7f0a26b1d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a2015d810 con 0x7f0a200c93a0 2024-01-27T01:03:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.503+0000 7f0a26b1d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2015daf0 con 0x7f0a20067d50 2024-01-27T01:03:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.504+0000 7f0a248b9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0a200c93a0 0x7f0a200c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.504+0000 7f0a17fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a20067d50 0x7f0a20159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.504+0000 7f0a250ba700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f0a200681c0 0x7f0a20159760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:52410/0 (socket says 172.21.15.46:52410) 2024-01-27T01:03:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.504+0000 7f0a17fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a20067d50 0x7f0a20159220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:40134/0 (socket says 172.21.15.46:40134) 2024-01-27T01:03:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.504+0000 7f0a248b9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0a200c93a0 0x7f0a200c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:37106/0 (socket says 172.21.15.46:37106) 2024-01-27T01:03:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.504+0000 7f0a250ba700 1 -- 172.21.15.46:0/4051204107 learned_addr learned my addr 172.21.15.46:0/4051204107 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.504+0000 7f0a177fe700 1 -- 172.21.15.46:0/4051204107 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596041354 0 0) 0x7f0a2015e390 con 0x7f0a200681c0 2024-01-27T01:03:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.504+0000 7f0a177fe700 1 -- 172.21.15.46:0/4051204107 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a08003570 con 0x7f0a200681c0 2024-01-27T01:03:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.504+0000 7f0a248b9700 1 -- 172.21.15.46:0/4051204107 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a20067d50 msgr2=0x7f0a20159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.504+0000 7f0a248b9700 1 --2- 172.21.15.46:0/4051204107 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a20067d50 0x7f0a20159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.504+0000 7f0a248b9700 1 -- 172.21.15.46:0/4051204107 >> v1:172.21.15.106:6789/0 conn(0x7f0a200681c0 legacy=0x7f0a20159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.504+0000 7f0a248b9700 1 -- 172.21.15.46:0/4051204107 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2015ddd0 con 0x7f0a200c93a0 2024-01-27T01:03:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.504+0000 7f0a17fff700 1 --2- 172.21.15.46:0/4051204107 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a20067d50 0x7f0a20159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:03:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.504+0000 7f0a248b9700 1 --2- 172.21.15.46:0/4051204107 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a200c93a0 0x7f0a200c9770 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c0098e0 tx=0x7f0a1c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d08832a1f94aef38 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.504+0000 7f0a177fe700 1 -- 172.21.15.46:0/4051204107 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c011630 con 0x7f0a200c93a0 2024-01-27T01:03:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.504+0000 7f0a177fe700 1 -- 172.21.15.46:0/4051204107 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1c011790 con 0x7f0a200c93a0 2024-01-27T01:03:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.504+0000 7f0a177fe700 1 -- 172.21.15.46:0/4051204107 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c011b30 con 0x7f0a200c93a0 2024-01-27T01:03:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.505+0000 7f0a26b1d700 1 -- 172.21.15.46:0/4051204107 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a200c93a0 msgr2=0x7f0a200c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.505+0000 7f0a26b1d700 1 --2- 172.21.15.46:0/4051204107 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a200c93a0 0x7f0a200c9770 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c0098e0 tx=0x7f0a1c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.505+0000 7f0a26b1d700 1 -- 172.21.15.46:0/4051204107 shutdown_connections 2024-01-27T01:03:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.505+0000 7f0a26b1d700 1 --2- 172.21.15.46:0/4051204107 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a200c93a0 0x7f0a200c9770 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.505+0000 7f0a26b1d700 1 --2- 172.21.15.46:0/4051204107 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a20067d50 0x7f0a20159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.505+0000 7f0a26b1d700 1 -- 172.21.15.46:0/4051204107 >> 172.21.15.46:0/4051204107 conn(0x7f0a200ca9a0 msgr2=0x7f0a20068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.505+0000 7f0a26b1d700 1 -- 172.21.15.46:0/4051204107 shutdown_connections 2024-01-27T01:03:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.505+0000 7f0a26b1d700 1 -- 172.21.15.46:0/4051204107 wait complete. 2024-01-27T01:03:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.507+0000 7f0a26b1d700 1 Processor -- start 2024-01-27T01:03:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.507+0000 7f0a26b1d700 1 -- start start 2024-01-27T01:03:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.508+0000 7f0a26b1d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20067d50 0x7f0a2015f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.508+0000 7f0a26b1d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a200681c0 0x7f0a20156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.508+0000 7f0a26b1d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a200c93a0 0x7f0a200c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.508+0000 7f0a26b1d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a20158360 con 0x7f0a20067d50 2024-01-27T01:03:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.508+0000 7f0a26b1d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a20157ed0 con 0x7f0a200681c0 2024-01-27T01:03:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.508+0000 7f0a26b1d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2015fb80 con 0x7f0a200c93a0 2024-01-27T01:03:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.508+0000 7f0a248b9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20067d50 0x7f0a2015f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.508+0000 7f0a248b9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20067d50 0x7f0a2015f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39122/0 (socket says 172.21.15.46:39122) 2024-01-27T01:03:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.508+0000 7f0a248b9700 1 -- 172.21.15.46:0/3130798617 learned_addr learned my addr 172.21.15.46:0/3130798617 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.508+0000 7f0a17fff700 1 --2- 172.21.15.46:0/3130798617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a200681c0 0x7f0a20156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.508+0000 7f0a250ba700 1 --2- 172.21.15.46:0/3130798617 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a200c93a0 0x7f0a200c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.508+0000 7f0a248b9700 1 -- 172.21.15.46:0/3130798617 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a200c93a0 msgr2=0x7f0a200c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.509+0000 7f0a248b9700 1 --2- 172.21.15.46:0/3130798617 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a200c93a0 0x7f0a200c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.509+0000 7f0a248b9700 1 -- 172.21.15.46:0/3130798617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a200681c0 msgr2=0x7f0a20156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.509+0000 7f0a248b9700 1 --2- 172.21.15.46:0/3130798617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a200681c0 0x7f0a20156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.509+0000 7f0a248b9700 1 -- 172.21.15.46:0/3130798617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a10009420 con 0x7f0a20067d50 2024-01-27T01:03:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.509+0000 7f0a17fff700 1 --2- 172.21.15.46:0/3130798617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a200681c0 0x7f0a20156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:03:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.509+0000 7f0a250ba700 1 --2- 172.21.15.46:0/3130798617 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a200c93a0 0x7f0a200c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:03:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.509+0000 7f0a248b9700 1 --2- 172.21.15.46:0/3130798617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20067d50 0x7f0a2015f640 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c011f20 tx=0x7f0a1c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.509+0000 7f0a14ff9700 1 -- 172.21.15.46:0/3130798617 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c011b10 con 0x7f0a20067d50 2024-01-27T01:03:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.509+0000 7f0a14ff9700 1 -- 172.21.15.46:0/3130798617 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1c0253f0 con 0x7f0a20067d50 2024-01-27T01:03:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.509+0000 7f0a26b1d700 1 -- 172.21.15.46:0/3130798617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1c009420 con 0x7f0a20067d50 2024-01-27T01:03:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.509+0000 7f0a14ff9700 1 -- 172.21.15.46:0/3130798617 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c0257b0 con 0x7f0a20067d50 2024-01-27T01:03:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.509+0000 7f0a26b1d700 1 -- 172.21.15.46:0/3130798617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a201575a0 con 0x7f0a20067d50 2024-01-27T01:03:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.510+0000 7f0a26b1d700 1 -- 172.21.15.46:0/3130798617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2004c710 con 0x7f0a20067d50 2024-01-27T01:03:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.510+0000 7f0a14ff9700 1 -- 172.21.15.46:0/3130798617 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0a1c025b20 con 0x7f0a20067d50 2024-01-27T01:03:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.511+0000 7f0a14ff9700 1 --2- 172.21.15.46:0/3130798617 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a0c066f30 0x7f0a0c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.511+0000 7f0a14ff9700 1 -- 172.21.15.46:0/3130798617 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f0a1c085fa0 con 0x7f0a20067d50 2024-01-27T01:03:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.511+0000 7f0a17fff700 1 --2- 172.21.15.46:0/3130798617 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a0c066f30 0x7f0a0c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.512+0000 7f0a17fff700 1 --2- 172.21.15.46:0/3130798617 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a0c066f30 0x7f0a0c0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0a100093f0 tx=0x7f0a10008040 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.515+0000 7f0a14ff9700 1 -- 172.21.15.46:0/3130798617 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a1c0b75a0 con 0x7f0a20067d50 2024-01-27T01:03:49.786 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-01-27T01:03:49.786+0000 7f16f4f6f700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-27T01:03:49.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.804+0000 7f0a26b1d700 1 -- 172.21.15.46:0/3130798617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f0a2015e0b0 con 0x7f0a20067d50 2024-01-27T01:03:49.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.808+0000 7f0a14ff9700 1 -- 172.21.15.46:0/3130798617 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v52) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0a1c0150e0 con 0x7f0a20067d50 2024-01-27T01:03:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.810+0000 7f0a26b1d700 1 -- 172.21.15.46:0/3130798617 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a0c066f30 msgr2=0x7f0a0c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.810+0000 7f0a26b1d700 1 --2- 172.21.15.46:0/3130798617 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a0c066f30 0x7f0a0c0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0a100093f0 tx=0x7f0a10008040 comp rx=0 tx=0).stop 2024-01-27T01:03:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.810+0000 7f0a26b1d700 1 -- 172.21.15.46:0/3130798617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20067d50 msgr2=0x7f0a2015f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.810+0000 7f0a26b1d700 1 --2- 172.21.15.46:0/3130798617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20067d50 0x7f0a2015f640 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c011f20 tx=0x7f0a1c017000 comp rx=0 tx=0).stop 2024-01-27T01:03:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.810+0000 7f0a26b1d700 1 -- 172.21.15.46:0/3130798617 shutdown_connections 2024-01-27T01:03:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.810+0000 7f0a26b1d700 1 --2- 172.21.15.46:0/3130798617 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a0c066f30 0x7f0a0c0693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.810+0000 7f0a26b1d700 1 --2- 172.21.15.46:0/3130798617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20067d50 0x7f0a2015f640 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.810+0000 7f0a26b1d700 1 --2- 172.21.15.46:0/3130798617 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a200c93a0 0x7f0a200c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.810+0000 7f0a26b1d700 1 --2- 172.21.15.46:0/3130798617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a200681c0 0x7f0a20156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.810+0000 7f0a26b1d700 1 -- 172.21.15.46:0/3130798617 >> 172.21.15.46:0/3130798617 conn(0x7f0a200ca9a0 msgr2=0x7f0a2006f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.810+0000 7f0a26b1d700 1 -- 172.21.15.46:0/3130798617 shutdown_connections 2024-01-27T01:03:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.810+0000 7f0a26b1d700 1 -- 172.21.15.46:0/3130798617 wait complete. 2024-01-27T01:03:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-01-27T01:03:49.824 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:03:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.956+0000 7f654156d700 1 Processor -- start 2024-01-27T01:03:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.956+0000 7f654156d700 1 -- start start 2024-01-27T01:03:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f654156d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f653c0b2b40 0x7f653c0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f654156d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f653c0ae960 0x7f653c0ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f654156d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f653c0aedd0 0x7f653c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f654156d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f653c099fc0 con 0x7f653c0aedd0 2024-01-27T01:03:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f654156d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f653c15a0b0 con 0x7f653c0b2b40 2024-01-27T01:03:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f654156d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f653c0ae4d0 con 0x7f653c0ae960 2024-01-27T01:03:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f653b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f653c0aedd0 0x7f653c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f653b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f653c0aedd0 0x7f653c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39132/0 (socket says 172.21.15.46:39132) 2024-01-27T01:03:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f653b7fe700 1 -- 172.21.15.46:0/2370534730 learned_addr learned my addr 172.21.15.46:0/2370534730 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f653affd700 1 --2- 172.21.15.46:0/2370534730 >> v2:172.21.15.106:3300/0 conn(0x7f653c0b2b40 0x7f653c0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f653a7fc700 1 --2- 172.21.15.46:0/2370534730 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f653c0ae960 0x7f653c0ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f653b7fe700 1 -- 172.21.15.46:0/2370534730 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f653c0ae960 msgr2=0x7f653c0ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f653b7fe700 1 --2- 172.21.15.46:0/2370534730 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f653c0ae960 0x7f653c0ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f653b7fe700 1 -- 172.21.15.46:0/2370534730 >> v2:172.21.15.106:3300/0 conn(0x7f653c0b2b40 msgr2=0x7f653c0ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f653b7fe700 1 --2- 172.21.15.46:0/2370534730 >> v2:172.21.15.106:3300/0 conn(0x7f653c0b2b40 0x7f653c0ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f653b7fe700 1 -- 172.21.15.46:0/2370534730 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653c0ad8c0 con 0x7f653c0aedd0 2024-01-27T01:03:49.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f653a7fc700 1 --2- 172.21.15.46:0/2370534730 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f653c0ae960 0x7f653c0ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:03:49.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.957+0000 7f653b7fe700 1 --2- 172.21.15.46:0/2370534730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c0aedd0 0x7f653c0cf290 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f653c088850 tx=0x7f6534009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8811372e2438d469 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:49.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.958+0000 7f6539ffb700 1 -- 172.21.15.46:0/2370534730 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653400e070 con 0x7f653c0aedd0 2024-01-27T01:03:49.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.958+0000 7f6539ffb700 1 -- 172.21.15.46:0/2370534730 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653400ba50 con 0x7f653c0aedd0 2024-01-27T01:03:49.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.958+0000 7f6539ffb700 1 -- 172.21.15.46:0/2370534730 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65340092e0 con 0x7f653c0aedd0 2024-01-27T01:03:49.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.958+0000 7f654156d700 1 -- 172.21.15.46:0/2370534730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c0aedd0 msgr2=0x7f653c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:49.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.958+0000 7f654156d700 1 --2- 172.21.15.46:0/2370534730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c0aedd0 0x7f653c0cf290 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f653c088850 tx=0x7f6534009840 comp rx=0 tx=0).stop 2024-01-27T01:03:49.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.958+0000 7f654156d700 1 -- 172.21.15.46:0/2370534730 shutdown_connections 2024-01-27T01:03:49.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.958+0000 7f654156d700 1 --2- 172.21.15.46:0/2370534730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c0aedd0 0x7f653c0cf290 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.958+0000 7f654156d700 1 --2- 172.21.15.46:0/2370534730 >> v2:172.21.15.106:3300/0 conn(0x7f653c0b2b40 0x7f653c0ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.958+0000 7f654156d700 1 --2- 172.21.15.46:0/2370534730 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f653c0ae960 0x7f653c0ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.958+0000 7f654156d700 1 -- 172.21.15.46:0/2370534730 >> 172.21.15.46:0/2370534730 conn(0x7f653c099a30 msgr2=0x7f653c06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:49.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.958+0000 7f654156d700 1 -- 172.21.15.46:0/2370534730 shutdown_connections 2024-01-27T01:03:49.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.958+0000 7f654156d700 1 -- 172.21.15.46:0/2370534730 wait complete. 2024-01-27T01:03:49.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.960+0000 7f654156d700 1 Processor -- start 2024-01-27T01:03:49.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.960+0000 7f654156d700 1 -- start start 2024-01-27T01:03:49.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.960+0000 7f654156d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f653c0ae960 0x7f653c09db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.960+0000 7f654156d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f653c0b2b40 0x7f653c0c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.960+0000 7f654156d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c156b30 0x7f653c0c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.960+0000 7f654156d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c09e0c0 con 0x7f653c156b30 2024-01-27T01:03:49.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.960+0000 7f654156d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c0af9e0 con 0x7f653c0ae960 2024-01-27T01:03:49.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.960+0000 7f654156d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f653c0af420 con 0x7f653c0b2b40 2024-01-27T01:03:49.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.961+0000 7f653b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c156b30 0x7f653c0c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.961+0000 7f653b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c156b30 0x7f653c0c8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39146/0 (socket says 172.21.15.46:39146) 2024-01-27T01:03:49.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.961+0000 7f653b7fe700 1 -- 172.21.15.46:0/4133810366 learned_addr learned my addr 172.21.15.46:0/4133810366 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:49.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.961+0000 7f653a7fc700 1 --2- 172.21.15.46:0/4133810366 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f653c0b2b40 0x7f653c0c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.961+0000 7f653affd700 1 --2- 172.21.15.46:0/4133810366 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f653c0ae960 0x7f653c09db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.961+0000 7f653b7fe700 1 -- 172.21.15.46:0/4133810366 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f653c0b2b40 msgr2=0x7f653c0c87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:49.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.961+0000 7f653b7fe700 1 --2- 172.21.15.46:0/4133810366 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f653c0b2b40 0x7f653c0c87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.961+0000 7f653b7fe700 1 -- 172.21.15.46:0/4133810366 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f653c0ae960 msgr2=0x7f653c09db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:49.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.961+0000 7f653b7fe700 1 --2- 172.21.15.46:0/4133810366 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f653c0ae960 0x7f653c09db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:49.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.961+0000 7f653b7fe700 1 -- 172.21.15.46:0/4133810366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6534009ae0 con 0x7f653c156b30 2024-01-27T01:03:49.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.961+0000 7f653b7fe700 1 --2- 172.21.15.46:0/4133810366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c156b30 0x7f653c0c8d30 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f6534012000 tx=0x7f6534018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:49.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.961+0000 7f6522ffd700 1 -- 172.21.15.46:0/4133810366 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65340032f0 con 0x7f653c156b30 2024-01-27T01:03:49.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.961+0000 7f6522ffd700 1 -- 172.21.15.46:0/4133810366 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6534008d20 con 0x7f653c156b30 2024-01-27T01:03:49.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.961+0000 7f654156d700 1 -- 172.21.15.46:0/4133810366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f653c0af700 con 0x7f653c156b30 2024-01-27T01:03:49.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.961+0000 7f6522ffd700 1 -- 172.21.15.46:0/4133810366 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65340256b0 con 0x7f653c156b30 2024-01-27T01:03:49.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.961+0000 7f654156d700 1 -- 172.21.15.46:0/4133810366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f653c0b3890 con 0x7f653c156b30 2024-01-27T01:03:49.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.963+0000 7f654156d700 1 -- 172.21.15.46:0/4133810366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f653c0b3a80 con 0x7f653c156b30 2024-01-27T01:03:49.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.963+0000 7f6522ffd700 1 -- 172.21.15.46:0/4133810366 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6534003040 con 0x7f653c156b30 2024-01-27T01:03:49.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.963+0000 7f6522ffd700 1 --2- 172.21.15.46:0/4133810366 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6528066f30 0x7f65280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:49.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.963+0000 7f6522ffd700 1 -- 172.21.15.46:0/4133810366 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f6534085d40 con 0x7f653c156b30 2024-01-27T01:03:49.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.963+0000 7f653affd700 1 --2- 172.21.15.46:0/4133810366 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6528066f30 0x7f65280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:49.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.964+0000 7f653affd700 1 --2- 172.21.15.46:0/4133810366 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6528066f30 0x7f65280693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f652c006aa0 tx=0x7f652c008040 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:49.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:49.968+0000 7f6522ffd700 1 -- 172.21.15.46:0/4133810366 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6534052d50 con 0x7f653c156b30 2024-01-27T01:03:50.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.218+0000 7f654156d700 1 -- 172.21.15.46:0/4133810366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f653c04c710 con 0x7f653c156b30 2024-01-27T01:03:50.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.219+0000 7f6522ffd700 1 -- 172.21.15.46:0/4133810366 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4612 (secure 0 0 0) 0x7f6534009940 con 0x7f653c156b30 2024-01-27T01:03:50.219 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:03:50.219 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":107,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8891,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/975115379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":975115379},{"type":"v1","addr":"172.21.15.106:6833","nonce":975115379}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10036,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3860120510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3860120510},{"type":"v1","addr":"172.21.15.46:6834","nonce":3860120510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10045,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3603549739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3603549739},{"type":"v1","addr":"172.21.15.106:6835","nonce":3603549739}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:03:44.141242+0000","modified":"2024-01-27T01:03:47.727201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10392":{"gid":10392,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/976738644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":976738644},{"type":"v1","addr":"172.21.15.46:6835","nonce":976738644}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2024-01-27T01:03:50.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.221+0000 7f654156d700 1 -- 172.21.15.46:0/4133810366 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6528066f30 msgr2=0x7f65280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:50.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.221+0000 7f654156d700 1 --2- 172.21.15.46:0/4133810366 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6528066f30 0x7f65280693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f652c006aa0 tx=0x7f652c008040 comp rx=0 tx=0).stop 2024-01-27T01:03:50.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.221+0000 7f654156d700 1 -- 172.21.15.46:0/4133810366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c156b30 msgr2=0x7f653c0c8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:50.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.221+0000 7f654156d700 1 --2- 172.21.15.46:0/4133810366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c156b30 0x7f653c0c8d30 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f6534012000 tx=0x7f6534018000 comp rx=0 tx=0).stop 2024-01-27T01:03:50.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.221+0000 7f654156d700 1 -- 172.21.15.46:0/4133810366 shutdown_connections 2024-01-27T01:03:50.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.221+0000 7f654156d700 1 --2- 172.21.15.46:0/4133810366 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6528066f30 0x7f65280693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.221+0000 7f654156d700 1 --2- 172.21.15.46:0/4133810366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f653c156b30 0x7f653c0c8d30 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.221+0000 7f654156d700 1 --2- 172.21.15.46:0/4133810366 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f653c0ae960 0x7f653c09db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.221+0000 7f654156d700 1 --2- 172.21.15.46:0/4133810366 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f653c0b2b40 0x7f653c0c87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.221+0000 7f654156d700 1 -- 172.21.15.46:0/4133810366 >> 172.21.15.46:0/4133810366 conn(0x7f653c099a30 msgr2=0x7f653c158df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:50.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.221+0000 7f654156d700 1 -- 172.21.15.46:0/4133810366 shutdown_connections 2024-01-27T01:03:50.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.221+0000 7f654156d700 1 -- 172.21.15.46:0/4133810366 wait complete. 2024-01-27T01:03:50.223 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 107 2024-01-27T01:03:50.637 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-27T01:03:44.141242+0000', 'modified': '2024-01-27T01:03:47.727201+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10392}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10392': {'gid': 10392, 'name': 'c', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.46:6835/976738644', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 976738644}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 976738644}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-27T01:03:50.638 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-27T01:03:50.638 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-27T01:03:50.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.780+0000 7f5e97c0f700 1 Processor -- start 2024-01-27T01:03:50.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.780+0000 7f5e97c0f700 1 -- start start 2024-01-27T01:03:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.781+0000 7f5e97c0f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5e90068040 0x7f5e900cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.781+0000 7f5e97c0f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5e90159d30 0x7f5e900ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.781+0000 7f5e97c0f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9006d760 con 0x7f5e90159960 2024-01-27T01:03:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.781+0000 7f5e97c0f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e9006f820 con 0x7f5e90159d30 2024-01-27T01:03:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.781+0000 7f5e97c0f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e900c9f00 con 0x7f5e90068040 2024-01-27T01:03:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.781+0000 7f5e961ac700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5e90159d30 0x7f5e900ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.781+0000 7f5e961ac700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5e90159d30 0x7f5e900ce1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33828/0 (socket says 172.21.15.46:33828) 2024-01-27T01:03:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.781+0000 7f5e961ac700 1 -- 172.21.15.46:0/2062357731 learned_addr learned my addr 172.21.15.46:0/2062357731 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.781+0000 7f5e951aa700 1 --2- 172.21.15.46:0/2062357731 >> v2:172.21.15.106:3300/0 conn(0x7f5e90068040 0x7f5e900cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.781+0000 7f5e961ac700 1 -- 172.21.15.46:0/2062357731 >> v2:172.21.15.106:3300/0 conn(0x7f5e90068040 msgr2=0x7f5e900cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.781+0000 7f5e961ac700 1 --2- 172.21.15.46:0/2062357731 >> v2:172.21.15.106:3300/0 conn(0x7f5e90068040 0x7f5e900cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.781+0000 7f5e961ac700 1 -- 172.21.15.46:0/2062357731 >> v1:172.21.15.46:6789/0 conn(0x7f5e90159960 legacy=0x7f5e90067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.781+0000 7f5e961ac700 1 -- 172.21.15.46:0/2062357731 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9006fcb0 con 0x7f5e90159d30 2024-01-27T01:03:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.781+0000 7f5e951aa700 1 --2- 172.21.15.46:0/2062357731 >> v2:172.21.15.106:3300/0 conn(0x7f5e90068040 0x7f5e900cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:03:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.782+0000 7f5e961ac700 1 --2- 172.21.15.46:0/2062357731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e90159d30 0x7f5e900ce1a0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f5e90064a40 tx=0x7f5e88009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=323d679471c1a542 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.782+0000 7f5e949a9700 1 -- 172.21.15.46:0/2062357731 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8800e070 con 0x7f5e90159d30 2024-01-27T01:03:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.782+0000 7f5e949a9700 1 -- 172.21.15.46:0/2062357731 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8800ba50 con 0x7f5e90159d30 2024-01-27T01:03:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.782+0000 7f5e949a9700 1 -- 172.21.15.46:0/2062357731 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e880092e0 con 0x7f5e90159d30 2024-01-27T01:03:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.782+0000 7f5e97c0f700 1 -- 172.21.15.46:0/2062357731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e90159d30 msgr2=0x7f5e900ce1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.782+0000 7f5e97c0f700 1 --2- 172.21.15.46:0/2062357731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e90159d30 0x7f5e900ce1a0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f5e90064a40 tx=0x7f5e88009840 comp rx=0 tx=0).stop 2024-01-27T01:03:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.782+0000 7f5e97c0f700 1 -- 172.21.15.46:0/2062357731 shutdown_connections 2024-01-27T01:03:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.782+0000 7f5e97c0f700 1 --2- 172.21.15.46:0/2062357731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e90159d30 0x7f5e900ce1a0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.782+0000 7f5e97c0f700 1 --2- 172.21.15.46:0/2062357731 >> v2:172.21.15.106:3300/0 conn(0x7f5e90068040 0x7f5e900cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.782+0000 7f5e97c0f700 1 -- 172.21.15.46:0/2062357731 >> 172.21.15.46:0/2062357731 conn(0x7f5e9006b850 msgr2=0x7f5e900cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.782+0000 7f5e97c0f700 1 -- 172.21.15.46:0/2062357731 shutdown_connections 2024-01-27T01:03:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.782+0000 7f5e97c0f700 1 -- 172.21.15.46:0/2062357731 wait complete. 2024-01-27T01:03:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.784+0000 7f5e97c0f700 1 Processor -- start 2024-01-27T01:03:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.784+0000 7f5e97c0f700 1 -- start start 2024-01-27T01:03:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.785+0000 7f5e97c0f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e90068040 0x7f5e901536d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.785+0000 7f5e97c0f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5e90159960 0x7f5e9016cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.785+0000 7f5e97c0f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5e90159d30 0x7f5e900c31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.785+0000 7f5e97c0f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e900c4700 con 0x7f5e90068040 2024-01-27T01:03:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.785+0000 7f5e97c0f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e900b1910 con 0x7f5e90159960 2024-01-27T01:03:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.785+0000 7f5e97c0f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e900b1e90 con 0x7f5e90159d30 2024-01-27T01:03:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.785+0000 7f5e959ab700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e90068040 0x7f5e901536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.785+0000 7f5e959ab700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e90068040 0x7f5e901536d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33832/0 (socket says 172.21.15.46:33832) 2024-01-27T01:03:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.785+0000 7f5e959ab700 1 -- 172.21.15.46:0/601382899 learned_addr learned my addr 172.21.15.46:0/601382899 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.786+0000 7f5e951aa700 1 --2- 172.21.15.46:0/601382899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5e90159960 0x7f5e9016cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.786+0000 7f5e961ac700 1 --2- 172.21.15.46:0/601382899 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5e90159d30 0x7f5e900c31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.786+0000 7f5e959ab700 1 -- 172.21.15.46:0/601382899 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5e90159d30 msgr2=0x7f5e900c31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.786+0000 7f5e959ab700 1 --2- 172.21.15.46:0/601382899 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5e90159d30 0x7f5e900c31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.786+0000 7f5e959ab700 1 -- 172.21.15.46:0/601382899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5e90159960 msgr2=0x7f5e9016cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.786+0000 7f5e959ab700 1 --2- 172.21.15.46:0/601382899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5e90159960 0x7f5e9016cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.786+0000 7f5e959ab700 1 -- 172.21.15.46:0/601382899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e88009ae0 con 0x7f5e90068040 2024-01-27T01:03:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.786+0000 7f5e959ab700 1 --2- 172.21.15.46:0/601382899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e90068040 0x7f5e901536d0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00c590 tx=0x7f5e8c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.786+0000 7f5e85ffb700 1 -- 172.21.15.46:0/601382899 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c011590 con 0x7f5e90068040 2024-01-27T01:03:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.786+0000 7f5e85ffb700 1 -- 172.21.15.46:0/601382899 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8c0116f0 con 0x7f5e90068040 2024-01-27T01:03:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.786+0000 7f5e85ffb700 1 -- 172.21.15.46:0/601382899 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c011ab0 con 0x7f5e90068040 2024-01-27T01:03:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.786+0000 7f5e97c0f700 1 -- 172.21.15.46:0/601382899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e9016da00 con 0x7f5e90068040 2024-01-27T01:03:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.786+0000 7f5e97c0f700 1 -- 172.21.15.46:0/601382899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e900b1b20 con 0x7f5e90068040 2024-01-27T01:03:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.788+0000 7f5e85ffb700 1 -- 172.21.15.46:0/601382899 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5e8c01b3b0 con 0x7f5e90068040 2024-01-27T01:03:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.788+0000 7f5e85ffb700 1 --2- 172.21.15.46:0/601382899 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5e74066f30 0x7f5e740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.788+0000 7f5e85ffb700 1 -- 172.21.15.46:0/601382899 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f5e8c083cf0 con 0x7f5e90068040 2024-01-27T01:03:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.788+0000 7f5e951aa700 1 --2- 172.21.15.46:0/601382899 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5e74066f30 0x7f5e740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.789+0000 7f5e951aa700 1 --2- 172.21.15.46:0/601382899 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5e74066f30 0x7f5e740693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e80006f00 tx=0x7f5e80008040 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:50.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.790+0000 7f5ea0f12b80 1 Processor -- start 2024-01-27T01:03:50.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.790+0000 7f5ea0f12b80 1 -- start start 2024-01-27T01:03:50.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.791+0000 7f5ea0f12b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17ce010 0x5596f17ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.791+0000 7f5ea0f12b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f1816360 0x5596f17f8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.791+0000 7f5ea0f12b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f17cd5d0 0x5596f17f8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.791+0000 7f5ea0f12b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596f16d19f0 con 0x5596f1816360 2024-01-27T01:03:50.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.791+0000 7f5ea0f12b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596f1723130 con 0x5596f17cd5d0 2024-01-27T01:03:50.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.791+0000 7f5ea0f12b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5596f1720930 con 0x5596f17ce010 2024-01-27T01:03:50.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.791+0000 7f5e951aa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f1816360 0x5596f17f8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.791+0000 7f5e951aa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f1816360 0x5596f17f8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33844/0 (socket says 172.21.15.46:33844) 2024-01-27T01:03:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.791+0000 7f5e951aa700 1 -- 172.21.15.46:0/983256218 learned_addr learned my addr 172.21.15.46:0/983256218 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.791+0000 7f5e961ac700 1 --2- 172.21.15.46:0/983256218 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f17cd5d0 0x5596f17f8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.791+0000 7f5e959ab700 1 --2- 172.21.15.46:0/983256218 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17ce010 0x5596f17ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.791+0000 7f5e951aa700 1 -- 172.21.15.46:0/983256218 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17ce010 msgr2=0x5596f17ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.791+0000 7f5e951aa700 1 --2- 172.21.15.46:0/983256218 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17ce010 0x5596f17ce3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.791+0000 7f5e951aa700 1 -- 172.21.15.46:0/983256218 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f17cd5d0 msgr2=0x5596f17f8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.791+0000 7f5e951aa700 1 --2- 172.21.15.46:0/983256218 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f17cd5d0 0x5596f17f8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.791+0000 7f5e951aa700 1 -- 172.21.15.46:0/983256218 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596f155bb10 con 0x5596f1816360 2024-01-27T01:03:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.792+0000 7f5e951aa700 1 --2- 172.21.15.46:0/983256218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f1816360 0x5596f17f8840 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f5e800109c0 tx=0x7f5e80017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a16c40c9109ba1 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.792+0000 7f5e72ffd700 1 -- 172.21.15.46:0/983256218 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e80015800 con 0x5596f1816360 2024-01-27T01:03:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.792+0000 7f5e72ffd700 1 -- 172.21.15.46:0/983256218 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8001a070 con 0x5596f1816360 2024-01-27T01:03:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.792+0000 7f5e72ffd700 1 -- 172.21.15.46:0/983256218 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e80015b70 con 0x5596f1816360 2024-01-27T01:03:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.792+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/983256218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f1816360 msgr2=0x5596f17f8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.792+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/983256218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f1816360 0x5596f17f8840 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f5e800109c0 tx=0x7f5e80017000 comp rx=0 tx=0).stop 2024-01-27T01:03:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.793+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/983256218 shutdown_connections 2024-01-27T01:03:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.793+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/983256218 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f17cd5d0 0x5596f17f8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.793+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/983256218 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17ce010 0x5596f17ce3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.793+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/983256218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f1816360 0x5596f17f8840 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.793+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/983256218 >> 172.21.15.46:0/983256218 conn(0x5596f17cda70 msgr2=0x5596f1816820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.793+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/983256218 shutdown_connections 2024-01-27T01:03:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.793+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/983256218 wait complete. 2024-01-27T01:03:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.793+0000 7f5ea0f12b80 1 -- start start 2024-01-27T01:03:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.793+0000 7f5ea0f12b80 1 Processor -- start 2024-01-27T01:03:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.795+0000 7f5ea0f12b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17cd5d0 0x5596f17f3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.795+0000 7f5ea0f12b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f17ce010 0x5596f180f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.795+0000 7f5ea0f12b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f1816360 0x5596f1811fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.795+0000 7f5ea0f12b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596f171c720 con 0x5596f17ce010 2024-01-27T01:03:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.795+0000 7f5ea0f12b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596f16c81b0 con 0x5596f1816360 2024-01-27T01:03:50.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.795+0000 7f5ea0f12b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5596f16c7e50 con 0x5596f17cd5d0 2024-01-27T01:03:50.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.795+0000 7f5ea0f12b80 10 client.0 _ll_register_callbacks cb 0x5596f152a7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-27T01:03:50.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.795+0000 7f5e951aa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f17ce010 0x5596f180f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.795+0000 7f5e951aa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f17ce010 0x5596f180f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33858/0 (socket says 172.21.15.46:33858) 2024-01-27T01:03:50.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.795+0000 7f5e951aa700 1 -- 172.21.15.46:0/1340315895 learned_addr learned my addr 172.21.15.46:0/1340315895 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:50.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.795+0000 7f5e959ab700 1 --2- 172.21.15.46:0/1340315895 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17cd5d0 0x5596f17f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.795+0000 7f5e961ac700 1 --2- 172.21.15.46:0/1340315895 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f1816360 0x5596f1811fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.795+0000 7f5e951aa700 1 -- 172.21.15.46:0/1340315895 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17cd5d0 msgr2=0x5596f17f3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:50.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.795+0000 7f5e951aa700 1 --2- 172.21.15.46:0/1340315895 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17cd5d0 0x5596f17f3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.795+0000 7f5e951aa700 1 -- 172.21.15.46:0/1340315895 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f1816360 msgr2=0x5596f1811fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:50.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.795+0000 7f5e951aa700 1 --2- 172.21.15.46:0/1340315895 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f1816360 0x5596f1811fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.795+0000 7f5e951aa700 1 -- 172.21.15.46:0/1340315895 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8000e050 con 0x5596f17ce010 2024-01-27T01:03:50.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.796+0000 7f5e951aa700 1 --2- 172.21.15.46:0/1340315895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f17ce010 0x5596f180f200 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f5e80023040 tx=0x7f5e8001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551e6f09ad9f94cd server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:50.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.796+0000 7f5e6bfff700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-27T01:03:50.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.796+0000 7f5e6bfff700 1 -- 172.21.15.46:0/1340315895 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e80002f70 con 0x5596f17ce010 2024-01-27T01:03:50.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.796+0000 7f5e6bfff700 1 -- 172.21.15.46:0/1340315895 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(205..205 src has 1..205) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f5e80021be0 con 0x5596f17ce010 2024-01-27T01:03:50.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.796+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/1340315895 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5596f17c1190 con 0x5596f17ce010 2024-01-27T01:03:50.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.796+0000 7f5e6bfff700 1 -- 172.21.15.46:0/1340315895 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e80016420 con 0x5596f17ce010 2024-01-27T01:03:50.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.796+0000 7f5ea0f12b80 10 client.10524 fetch_fsmap learned FSMap version 107 2024-01-27T01:03:50.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.796+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/1340315895 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) v3 -- 0x5596f16ea290 con 0x5596f17ce010 2024-01-27T01:03:50.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.796+0000 7f5e6bfff700 1 -- 172.21.15.46:0/1340315895 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5596f17c1190 con 0x5596f17ce010 2024-01-27T01:03:50.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.796+0000 7f5e6bfff700 1 -- 172.21.15.46:0/1340315895 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 107) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f5e80016580 con 0x5596f17ce010 2024-01-27T01:03:50.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.796+0000 7f5ea0f12b80 10 client.10524 fetch_fsmap finished waiting for FSMap version 107 2024-01-27T01:03:50.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.796+0000 7f5ea0f12b80 10 client.10524 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-27T01:03:50.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.796+0000 7f5ea0f12b80 20 client.10524 populate_metadata read hostname 'smithi046' 2024-01-27T01:03:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.796+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/1340315895 >> [v2:172.21.15.46:6833/976738644,v1:172.21.15.46:6835/976738644] conn(0x5596f180e2d0 0x5596f180e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.796+0000 7f5ea0f12b80 4 client.10524 mds_command: new command op to 10392 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-27T01:03:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.796+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/1340315895 --> [v2:172.21.15.46:6833/976738644,v1:172.21.15.46:6835/976738644] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5596f16beee0 con 0x5596f180e2d0 2024-01-27T01:03:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.797+0000 7f5e959ab700 1 --2- 172.21.15.46:0/1340315895 >> [v2:172.21.15.46:6833/976738644,v1:172.21.15.46:6835/976738644] conn(0x5596f180e2d0 0x5596f180e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.797+0000 7f5e6bfff700 10 client.10524 ms_handle_connect on v2:172.21.15.46:6833/976738644 2024-01-27T01:03:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.797+0000 7f5e959ab700 1 --2- 172.21.15.46:0/1340315895 >> [v2:172.21.15.46:6833/976738644,v1:172.21.15.46:6835/976738644] conn(0x5596f180e2d0 0x5596f180e6a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21c39c235b71b10c server_cookie=6c20a3d57547535f in_seq=0 out_seq=0 2024-01-27T01:03:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.798+0000 7f5e6bfff700 1 -- 172.21.15.46:0/1340315895 <== mds.0 v2:172.21.15.46:6833/976738644 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x5596f16beee0 con 0x5596f180e2d0 2024-01-27T01:03:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.798+0000 7f5e6bfff700 10 client.10524 handle_command_reply: tid=0 2024-01-27T01:03:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.798+0000 7f5ea0f12b80 1 client.10524 shutdown 2024-01-27T01:03:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.798+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/1340315895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f17ce010 msgr2=0x5596f180f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.798+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/1340315895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f17ce010 0x5596f180f200 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f5e80023040 tx=0x7f5e8001a000 comp rx=0 tx=0).stop 2024-01-27T01:03:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.798+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/1340315895 shutdown_connections 2024-01-27T01:03:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.798+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/1340315895 >> [v2:172.21.15.46:6833/976738644,v1:172.21.15.46:6835/976738644] conn(0x5596f180e2d0 0x5596f180e6a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.798+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/1340315895 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f1816360 0x5596f1811fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.798+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/1340315895 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17cd5d0 0x5596f17f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.798+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/1340315895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f17ce010 0x5596f180f200 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.798+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/1340315895 >> 172.21.15.46:0/1340315895 conn(0x5596f17cda70 msgr2=0x5596f17f5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:50.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.798+0000 7f5e6bfff700 0 client.10524 ms_handle_reset on v2:172.21.15.46:6833/976738644 2024-01-27T01:03:50.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.798+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/1340315895 shutdown_connections 2024-01-27T01:03:50.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.798+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/1340315895 wait complete. 2024-01-27T01:03:50.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.798+0000 7f5ea0f12b80 20 client.10524 trim_cache size 0 max 16384 2024-01-27T01:03:50.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.817+0000 7f5ea0f12b80 1 Processor -- start 2024-01-27T01:03:50.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.817+0000 7f5ea0f12b80 1 -- start start 2024-01-27T01:03:50.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.818+0000 7f5ea0f12b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f17cd5d0 0x5596f17c9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.818+0000 7f5ea0f12b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17ce010 0x5596f17c8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.818+0000 7f5ea0f12b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f1816360 0x5596f17ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.818+0000 7f5ea0f12b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596f1614e40 con 0x5596f1816360 2024-01-27T01:03:50.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.818+0000 7f5ea0f12b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596f17f81e0 con 0x5596f17cd5d0 2024-01-27T01:03:50.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.818+0000 7f5ea0f12b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5596f1817310 con 0x5596f17ce010 2024-01-27T01:03:50.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.818+0000 7f5e961ac700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f1816360 0x5596f17ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.818+0000 7f5e961ac700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f1816360 0x5596f17ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33866/0 (socket says 172.21.15.46:33866) 2024-01-27T01:03:50.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.818+0000 7f5e961ac700 1 -- 172.21.15.46:0/1389981374 learned_addr learned my addr 172.21.15.46:0/1389981374 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:50.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.818+0000 7f5e961ac700 1 -- 172.21.15.46:0/1389981374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17ce010 msgr2=0x5596f17c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:50.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.818+0000 7f5e959ab700 1 --2- 172.21.15.46:0/1389981374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f17cd5d0 0x5596f17c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.818+0000 7f5e951aa700 1 --2- 172.21.15.46:0/1389981374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17ce010 0x5596f17c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.818+0000 7f5e961ac700 1 --2- 172.21.15.46:0/1389981374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17ce010 0x5596f17c8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.818+0000 7f5e961ac700 1 -- 172.21.15.46:0/1389981374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f17cd5d0 msgr2=0x5596f17c9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:50.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.818+0000 7f5e961ac700 1 --2- 172.21.15.46:0/1389981374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f17cd5d0 0x5596f17c9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.818+0000 7f5e961ac700 1 -- 172.21.15.46:0/1389981374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596f15b3c80 con 0x5596f1816360 2024-01-27T01:03:50.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.819+0000 7f5e951aa700 1 --2- 172.21.15.46:0/1389981374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17ce010 0x5596f17c8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:03:50.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.819+0000 7f5e959ab700 1 --2- 172.21.15.46:0/1389981374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f17cd5d0 0x5596f17c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:03:50.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.819+0000 7f5e961ac700 1 --2- 172.21.15.46:0/1389981374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f1816360 0x5596f17ca1e0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f5e9006c160 tx=0x7f5e88018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a42cdb326924a80b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:50.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.819+0000 7f5e737fe700 1 -- 172.21.15.46:0/1389981374 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e88022050 con 0x5596f1816360 2024-01-27T01:03:50.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.819+0000 7f5e737fe700 1 -- 172.21.15.46:0/1389981374 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e88023040 con 0x5596f1816360 2024-01-27T01:03:50.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.819+0000 7f5e737fe700 1 -- 172.21.15.46:0/1389981374 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e88007280 con 0x5596f1816360 2024-01-27T01:03:50.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.819+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/1389981374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f1816360 msgr2=0x5596f17ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:50.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.819+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/1389981374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f1816360 0x5596f17ca1e0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f5e9006c160 tx=0x7f5e88018000 comp rx=0 tx=0).stop 2024-01-27T01:03:50.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.820+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/1389981374 shutdown_connections 2024-01-27T01:03:50.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.820+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/1389981374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f1816360 0x5596f17ca1e0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.820+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/1389981374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f17cd5d0 0x5596f17c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.820+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/1389981374 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17ce010 0x5596f17c8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.820+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/1389981374 >> 172.21.15.46:0/1389981374 conn(0x5596f17cda70 msgr2=0x5596f18151c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:50.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.820+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/1389981374 shutdown_connections 2024-01-27T01:03:50.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.820+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/1389981374 wait complete. 2024-01-27T01:03:50.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.820+0000 7f5ea0f12b80 1 -- start start 2024-01-27T01:03:50.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.820+0000 7f5ea0f12b80 1 Processor -- start 2024-01-27T01:03:50.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.822+0000 7f5ea0f12b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17cd5d0 0x5596f17c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.822+0000 7f5ea0f12b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f17ce010 0x5596f17c8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.822+0000 7f5ea0f12b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f1816360 0x5596f17f3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.822+0000 7f5ea0f12b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596f182f050 con 0x5596f17ce010 2024-01-27T01:03:50.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.822+0000 7f5ea0f12b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596f18702d0 con 0x5596f1816360 2024-01-27T01:03:50.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.822+0000 7f5ea0f12b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5596f1870f00 con 0x5596f17cd5d0 2024-01-27T01:03:50.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.822+0000 7f5ea0f12b80 10 client.0 _ll_register_callbacks cb 0x5596f152a7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-27T01:03:50.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.822+0000 7f5e951aa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f17ce010 0x5596f17c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.822+0000 7f5e951aa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f17ce010 0x5596f17c8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33878/0 (socket says 172.21.15.46:33878) 2024-01-27T01:03:50.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.822+0000 7f5e951aa700 1 -- 172.21.15.46:0/2985671782 learned_addr learned my addr 172.21.15.46:0/2985671782 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:50.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.822+0000 7f5e951aa700 1 -- 172.21.15.46:0/2985671782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17cd5d0 msgr2=0x5596f17c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:50.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.822+0000 7f5e961ac700 1 --2- 172.21.15.46:0/2985671782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f1816360 0x5596f17f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.822+0000 7f5e951aa700 1 --2- 172.21.15.46:0/2985671782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17cd5d0 0x5596f17c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.822+0000 7f5e951aa700 1 -- 172.21.15.46:0/2985671782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f1816360 msgr2=0x5596f17f3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:50.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.822+0000 7f5e951aa700 1 --2- 172.21.15.46:0/2985671782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f1816360 0x5596f17f3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.822+0000 7f5e951aa700 1 -- 172.21.15.46:0/2985671782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9006fcb0 con 0x5596f17ce010 2024-01-27T01:03:50.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.822+0000 7f5e961ac700 1 --2- 172.21.15.46:0/2985671782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f1816360 0x5596f17f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:03:50.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.823+0000 7f5e951aa700 1 --2- 172.21.15.46:0/2985671782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f17ce010 0x5596f17c8190 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f5e80010860 tx=0x7f5e8002c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67423091b7fe30b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:50.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.823+0000 7f5e6bfff700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-27T01:03:50.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.823+0000 7f5e6bfff700 1 -- 172.21.15.46:0/2985671782 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e80021be0 con 0x5596f17ce010 2024-01-27T01:03:50.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.823+0000 7f5e6bfff700 1 -- 172.21.15.46:0/2985671782 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(205..205 src has 1..205) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f5e80016a90 con 0x5596f17ce010 2024-01-27T01:03:50.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.823+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/2985671782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e9005e3d0 con 0x5596f17ce010 2024-01-27T01:03:50.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.823+0000 7f5e6bfff700 1 -- 172.21.15.46:0/2985671782 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e80015680 con 0x5596f17ce010 2024-01-27T01:03:50.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.823+0000 7f5ea0f12b80 10 client.10530 fetch_fsmap learned FSMap version 107 2024-01-27T01:03:50.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.823+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/2985671782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) v3 -- 0x5596f1559fe0 con 0x5596f17ce010 2024-01-27T01:03:50.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.823+0000 7f5e6bfff700 1 -- 172.21.15.46:0/2985671782 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9005e3d0 con 0x5596f17ce010 2024-01-27T01:03:50.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.823+0000 7f5e6bfff700 1 -- 172.21.15.46:0/2985671782 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 107) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f5e8004a470 con 0x5596f17ce010 2024-01-27T01:03:50.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.823+0000 7f5ea0f12b80 10 client.10530 fetch_fsmap finished waiting for FSMap version 107 2024-01-27T01:03:50.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.823+0000 7f5ea0f12b80 10 client.10530 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-27T01:03:50.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.823+0000 7f5ea0f12b80 20 client.10530 populate_metadata read hostname 'smithi046' 2024-01-27T01:03:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.823+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/2985671782 >> [v2:172.21.15.46:6833/976738644,v1:172.21.15.46:6835/976738644] conn(0x5596f17f4390 0x5596f17f4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.823+0000 7f5ea0f12b80 4 client.10530 mds_command: new command op to 10392 tid=0[{"prefix": "status"}] 2024-01-27T01:03:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.823+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/2985671782 --> [v2:172.21.15.46:6833/976738644,v1:172.21.15.46:6835/976738644] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f5e8001a070 con 0x5596f17f4390 2024-01-27T01:03:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.823+0000 7f5e959ab700 1 --2- 172.21.15.46:0/2985671782 >> [v2:172.21.15.46:6833/976738644,v1:172.21.15.46:6835/976738644] conn(0x5596f17f4390 0x5596f17f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.824+0000 7f5e959ab700 1 --2- 172.21.15.46:0/2985671782 >> [v2:172.21.15.46:6833/976738644,v1:172.21.15.46:6835/976738644] conn(0x5596f17f4390 0x5596f17f4760 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82f6b2dc18d89979 server_cookie=5eba47dc34a00301 in_seq=0 out_seq=0 2024-01-27T01:03:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.824+0000 7f5e6bfff700 10 client.10530 ms_handle_connect on v2:172.21.15.46:6833/976738644 2024-01-27T01:03:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.824+0000 7f5e6bfff700 1 -- 172.21.15.46:0/2985671782 <== mds.0 v2:172.21.15.46:6833/976738644 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x7f5e8001a070 con 0x5596f17f4390 2024-01-27T01:03:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.824+0000 7f5e6bfff700 10 client.10530 handle_command_reply: tid=0 2024-01-27T01:03:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.824+0000 7f5ea0f12b80 1 client.10530 shutdown 2024-01-27T01:03:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.824+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/2985671782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f17ce010 msgr2=0x5596f17c8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.824+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/2985671782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f17ce010 0x5596f17c8190 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f5e80010860 tx=0x7f5e8002c000 comp rx=0 tx=0).stop 2024-01-27T01:03:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.824+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/2985671782 shutdown_connections 2024-01-27T01:03:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.824+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/2985671782 >> [v2:172.21.15.46:6833/976738644,v1:172.21.15.46:6835/976738644] conn(0x5596f17f4390 0x5596f17f4760 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.824+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/2985671782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5596f1816360 0x5596f17f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.824+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/2985671782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5596f17cd5d0 0x5596f17c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.824+0000 7f5e6bfff700 0 client.10530 ms_handle_reset on v2:172.21.15.46:6833/976738644 2024-01-27T01:03:50.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.824+0000 7f5ea0f12b80 1 --2- 172.21.15.46:0/2985671782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596f17ce010 0x5596f17c8190 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.824+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/2985671782 >> 172.21.15.46:0/2985671782 conn(0x5596f17cda70 msgr2=0x5596f17ca1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:50.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.824+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/2985671782 shutdown_connections 2024-01-27T01:03:50.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.824+0000 7f5ea0f12b80 1 -- 172.21.15.46:0/2985671782 wait complete. 2024-01-27T01:03:50.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.824+0000 7f5ea0f12b80 20 client.10530 trim_cache size 0 max 16384 2024-01-27T01:03:50.830 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:03:50.830 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-27T01:03:50.830 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-27T01:03:50.831 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 10392, 2024-01-27T01:03:50.831 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-27T01:03:50.831 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-27T01:03:50.831 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-01-27T01:03:50.831 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.1164562360000003, 2024-01-27T01:03:50.831 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 107, 2024-01-27T01:03:50.831 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 205, 2024-01-27T01:03:50.831 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 205, 2024-01-27T01:03:50.831 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 18.000597133999999 2024-01-27T01:03:50.831 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:03:50.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.825+0000 7f5e97c0f700 1 -- 172.21.15.46:0/601382899 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5e74066f30 msgr2=0x7f5e740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:50.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.825+0000 7f5e97c0f700 1 --2- 172.21.15.46:0/601382899 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5e74066f30 0x7f5e740693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e80006f00 tx=0x7f5e80008040 comp rx=0 tx=0).stop 2024-01-27T01:03:50.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.825+0000 7f5e97c0f700 1 -- 172.21.15.46:0/601382899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e90068040 msgr2=0x7f5e901536d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:50.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.825+0000 7f5e97c0f700 1 --2- 172.21.15.46:0/601382899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e90068040 0x7f5e901536d0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00c590 tx=0x7f5e8c00c950 comp rx=0 tx=0).stop 2024-01-27T01:03:50.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.825+0000 7f5e97c0f700 1 -- 172.21.15.46:0/601382899 shutdown_connections 2024-01-27T01:03:50.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.825+0000 7f5e97c0f700 1 --2- 172.21.15.46:0/601382899 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5e74066f30 0x7f5e740693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.825+0000 7f5e97c0f700 1 --2- 172.21.15.46:0/601382899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e90068040 0x7f5e901536d0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.825+0000 7f5e97c0f700 1 --2- 172.21.15.46:0/601382899 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5e90159d30 0x7f5e900c31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.825+0000 7f5e97c0f700 1 --2- 172.21.15.46:0/601382899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5e90159960 0x7f5e9016cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.825+0000 7f5e97c0f700 1 -- 172.21.15.46:0/601382899 >> 172.21.15.46:0/601382899 conn(0x7f5e9006b850 msgr2=0x7f5e90067930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:50.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.825+0000 7f5e97c0f700 1 -- 172.21.15.46:0/601382899 shutdown_connections 2024-01-27T01:03:50.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.825+0000 7f5e97c0f700 1 -- 172.21.15.46:0/601382899 wait complete. 2024-01-27T01:03:50.839 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-27T01:03:50.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.970+0000 7fef42a8c700 1 Processor -- start 2024-01-27T01:03:50.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.970+0000 7fef42a8c700 1 -- start start 2024-01-27T01:03:50.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.971+0000 7fef42a8c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fef3c159590 0x7fef3c15d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.971+0000 7fef42a8c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fef3c06c3c0 0x7fef3c069370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.971+0000 7fef42a8c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef3c15dd90 con 0x7fef3c15d8f0 2024-01-27T01:03:50.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.971+0000 7fef42a8c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef3c15e350 con 0x7fef3c159590 2024-01-27T01:03:50.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.971+0000 7fef42a8c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef3c0c97c0 con 0x7fef3c06c3c0 2024-01-27T01:03:50.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.971+0000 7fef40828700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fef3c159590 0x7fef3c15d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.971+0000 7fef40828700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fef3c159590 0x7fef3c15d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33882/0 (socket says 172.21.15.46:33882) 2024-01-27T01:03:50.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.971+0000 7fef40828700 1 -- 172.21.15.46:0/2283951961 learned_addr learned my addr 172.21.15.46:0/2283951961 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:50.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.971+0000 7fef41029700 1 --2- 172.21.15.46:0/2283951961 >> v2:172.21.15.106:3300/0 conn(0x7fef3c06c3c0 0x7fef3c069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.971+0000 7fef40828700 1 -- 172.21.15.46:0/2283951961 >> v2:172.21.15.106:3300/0 conn(0x7fef3c06c3c0 msgr2=0x7fef3c069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:50.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.971+0000 7fef40828700 1 --2- 172.21.15.46:0/2283951961 >> v2:172.21.15.106:3300/0 conn(0x7fef3c06c3c0 0x7fef3c069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.971+0000 7fef40828700 1 -- 172.21.15.46:0/2283951961 >> v1:172.21.15.46:6789/0 conn(0x7fef3c15d8f0 legacy=0x7fef3c06bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:50.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.971+0000 7fef40828700 1 -- 172.21.15.46:0/2283951961 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3c15e070 con 0x7fef3c159590 2024-01-27T01:03:50.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.971+0000 7fef40828700 1 --2- 172.21.15.46:0/2283951961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c159590 0x7fef3c15d3b0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fef3c06cd80 tx=0x7fef2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5db56e7aee35354 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:50.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.971+0000 7fef3b7fe700 1 -- 172.21.15.46:0/2283951961 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c00b820 con 0x7fef3c159590 2024-01-27T01:03:50.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.972+0000 7fef3b7fe700 1 -- 172.21.15.46:0/2283951961 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2c00b980 con 0x7fef3c159590 2024-01-27T01:03:50.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.972+0000 7fef3b7fe700 1 -- 172.21.15.46:0/2283951961 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c00bd40 con 0x7fef3c159590 2024-01-27T01:03:50.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.972+0000 7fef42a8c700 1 -- 172.21.15.46:0/2283951961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c159590 msgr2=0x7fef3c15d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:50.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.972+0000 7fef42a8c700 1 --2- 172.21.15.46:0/2283951961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c159590 0x7fef3c15d3b0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fef3c06cd80 tx=0x7fef2c009250 comp rx=0 tx=0).stop 2024-01-27T01:03:50.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.972+0000 7fef42a8c700 1 -- 172.21.15.46:0/2283951961 shutdown_connections 2024-01-27T01:03:50.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.972+0000 7fef42a8c700 1 --2- 172.21.15.46:0/2283951961 >> v2:172.21.15.106:3300/0 conn(0x7fef3c06c3c0 0x7fef3c069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.972+0000 7fef42a8c700 1 --2- 172.21.15.46:0/2283951961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c159590 0x7fef3c15d3b0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.972+0000 7fef42a8c700 1 -- 172.21.15.46:0/2283951961 >> 172.21.15.46:0/2283951961 conn(0x7fef3c0ce530 msgr2=0x7fef3c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:50.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.972+0000 7fef42a8c700 1 -- 172.21.15.46:0/2283951961 shutdown_connections 2024-01-27T01:03:50.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.972+0000 7fef42a8c700 1 -- 172.21.15.46:0/2283951961 wait complete. 2024-01-27T01:03:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.973+0000 7fef42a8c700 1 Processor -- start 2024-01-27T01:03:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.974+0000 7fef42a8c700 1 -- start start 2024-01-27T01:03:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.974+0000 7fef42a8c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c06c3c0 0x7fef3c15a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.974+0000 7fef42a8c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef3c159590 0x7fef3c15c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.974+0000 7fef42a8c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef3c15d8f0 0x7fef3c0c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.974+0000 7fef42a8c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c0c3370 con 0x7fef3c06c3c0 2024-01-27T01:03:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.974+0000 7fef42a8c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c0c8470 con 0x7fef3c15d8f0 2024-01-27T01:03:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.974+0000 7fef42a8c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c0c8750 con 0x7fef3c159590 2024-01-27T01:03:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.974+0000 7fef40828700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c06c3c0 0x7fef3c15a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.974+0000 7fef40828700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c06c3c0 0x7fef3c15a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33886/0 (socket says 172.21.15.46:33886) 2024-01-27T01:03:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.975+0000 7fef40828700 1 -- 172.21.15.46:0/1842911028 learned_addr learned my addr 172.21.15.46:0/1842911028 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.975+0000 7fef3bfff700 1 --2- 172.21.15.46:0/1842911028 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef3c159590 0x7fef3c15c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.975+0000 7fef41029700 1 --2- 172.21.15.46:0/1842911028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef3c15d8f0 0x7fef3c0c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.975+0000 7fef40828700 1 -- 172.21.15.46:0/1842911028 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef3c159590 msgr2=0x7fef3c15c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.975+0000 7fef40828700 1 --2- 172.21.15.46:0/1842911028 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef3c159590 0x7fef3c15c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.975+0000 7fef40828700 1 -- 172.21.15.46:0/1842911028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef3c15d8f0 msgr2=0x7fef3c0c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.975+0000 7fef40828700 1 --2- 172.21.15.46:0/1842911028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef3c15d8f0 0x7fef3c0c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.975+0000 7fef40828700 1 -- 172.21.15.46:0/1842911028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c0094f0 con 0x7fef3c06c3c0 2024-01-27T01:03:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.975+0000 7fef3bfff700 1 --2- 172.21.15.46:0/1842911028 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef3c159590 0x7fef3c15c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:03:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.975+0000 7fef41029700 1 --2- 172.21.15.46:0/1842911028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef3c15d8f0 0x7fef3c0c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:03:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.975+0000 7fef40828700 1 --2- 172.21.15.46:0/1842911028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c06c3c0 0x7fef3c15a130 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fef3c06f390 tx=0x7fef2c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.975+0000 7fef38ff9700 1 -- 172.21.15.46:0/1842911028 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c01f070 con 0x7fef3c06c3c0 2024-01-27T01:03:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.975+0000 7fef38ff9700 1 -- 172.21.15.46:0/1842911028 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2c002f30 con 0x7fef3c06c3c0 2024-01-27T01:03:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.975+0000 7fef42a8c700 1 -- 172.21.15.46:0/1842911028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef3c0c2d20 con 0x7fef3c06c3c0 2024-01-27T01:03:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.975+0000 7fef38ff9700 1 -- 172.21.15.46:0/1842911028 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c00bd20 con 0x7fef3c06c3c0 2024-01-27T01:03:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.975+0000 7fef42a8c700 1 -- 172.21.15.46:0/1842911028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef3c15a670 con 0x7fef3c06c3c0 2024-01-27T01:03:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.976+0000 7fef42a8c700 1 -- 172.21.15.46:0/1842911028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef3c04c710 con 0x7fef3c06c3c0 2024-01-27T01:03:50.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.977+0000 7fef38ff9700 1 -- 172.21.15.46:0/1842911028 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fef2c023070 con 0x7fef3c06c3c0 2024-01-27T01:03:50.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.977+0000 7fef38ff9700 1 --2- 172.21.15.46:0/1842911028 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fef28066f30 0x7fef280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:50.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.977+0000 7fef38ff9700 1 -- 172.21.15.46:0/1842911028 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fef2c085bd0 con 0x7fef3c06c3c0 2024-01-27T01:03:50.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.978+0000 7fef3bfff700 1 --2- 172.21.15.46:0/1842911028 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fef28066f30 0x7fef280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:50.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.978+0000 7fef3bfff700 1 --2- 172.21.15.46:0/1842911028 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fef28066f30 0x7fef280693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fef3c06e480 tx=0x7fef30009040 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:50.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:50.982+0000 7fef38ff9700 1 -- 172.21.15.46:0/1842911028 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fef2c052b30 con 0x7fef3c06c3c0 2024-01-27T01:03:51.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:51.239+0000 7fef42a8c700 1 -- 172.21.15.46:0/1842911028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fef3c068b70 con 0x7fef3c06c3c0 2024-01-27T01:03:51.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:51.239+0000 7fef38ff9700 1 -- 172.21.15.46:0/1842911028 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v107) v1 ==== 53+0+73 (secure 0 0 0) 0x7fef2c056150 con 0x7fef3c06c3c0 2024-01-27T01:03:51.239 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-27T01:03:51.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:51.241+0000 7fef42a8c700 1 -- 172.21.15.46:0/1842911028 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fef28066f30 msgr2=0x7fef280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:51.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:51.241+0000 7fef42a8c700 1 --2- 172.21.15.46:0/1842911028 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fef28066f30 0x7fef280693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fef3c06e480 tx=0x7fef30009040 comp rx=0 tx=0).stop 2024-01-27T01:03:51.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:51.241+0000 7fef42a8c700 1 -- 172.21.15.46:0/1842911028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c06c3c0 msgr2=0x7fef3c15a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:51.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:51.241+0000 7fef42a8c700 1 --2- 172.21.15.46:0/1842911028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c06c3c0 0x7fef3c15a130 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fef3c06f390 tx=0x7fef2c017000 comp rx=0 tx=0).stop 2024-01-27T01:03:51.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:51.241+0000 7fef42a8c700 1 -- 172.21.15.46:0/1842911028 shutdown_connections 2024-01-27T01:03:51.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:51.241+0000 7fef42a8c700 1 --2- 172.21.15.46:0/1842911028 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fef28066f30 0x7fef280693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:51.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:51.241+0000 7fef42a8c700 1 --2- 172.21.15.46:0/1842911028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c06c3c0 0x7fef3c15a130 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:51.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:51.241+0000 7fef42a8c700 1 --2- 172.21.15.46:0/1842911028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef3c15d8f0 0x7fef3c0c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:51.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:51.241+0000 7fef42a8c700 1 --2- 172.21.15.46:0/1842911028 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef3c159590 0x7fef3c15c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:51.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:51.241+0000 7fef42a8c700 1 -- 172.21.15.46:0/1842911028 >> 172.21.15.46:0/1842911028 conn(0x7fef3c0ce530 msgr2=0x7fef3c1589b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:51.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:51.241+0000 7fef42a8c700 1 -- 172.21.15.46:0/1842911028 shutdown_connections 2024-01-27T01:03:51.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:51.241+0000 7fef42a8c700 1 -- 172.21.15.46:0/1842911028 wait complete. 2024-01-27T01:03:51.255 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-27T01:03:51.255 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-27T01:03:51.255 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-27T01:03:51.255 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-27T01:03:51.255 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-27T01:03:51.255 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-27T01:03:51.255 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-27T01:03:51.255 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-27T01:03:51.255 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-27T01:03:51.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:03:51.256 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-27T01:03:51.275 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-27T01:03:51.276 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-27T01:03:51.276 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-27T01:03:51.276 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:03:51.276 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-27T01:03:51.276 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:03:51.276 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T01:03:51.276 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:03:51.276 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-27T01:03:51.276 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T01:03:51.276 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:03:51.276 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-27T01:03:51.276 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-27T01:03:51.276 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:03:51.277 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-27T01:03:51.277 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-27T01:03:51.277 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 36699sec preferred_lft 36699sec 2024-01-27T01:03:51.277 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-27T01:03:51.277 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:03:51.277 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T01:03:51.277 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:03:51.277 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-27T01:03:51.277 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-27T01:03:51.277 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:03:51.277 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-27T01:03:51.277 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:03:51.277 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::cce6:a7ff:fe4f:38d/64 scope link 2024-01-27T01:03:51.277 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:03:51.277 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-27T01:03:51.278 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 76:ef:90:4c:9d:6f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-27T01:03:51.278 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::74ef:90ff:fe4c:9d6f/64 scope link 2024-01-27T01:03:51.278 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:03:51.278 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-27T01:03:51.278 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-27T01:03:51.278 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::2c73:afff:fef0:6f39/64 scope link 2024-01-27T01:03:51.278 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:03:51.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:03:51.278 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-27T01:03:51.346 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-27T01:03:51.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:03:51.348 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-27T01:03:51.413 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-27T01:03:51.413 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-27T01:03:51.414 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-27T01:03:51.414 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-27T01:03:51.414 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-27T01:03:51.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:03:51.414 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-27T01:03:51.414 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-27T01:03:51.414 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-27T01:03:51.415 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-27T01:03:51.415 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-27T01:03:51.415 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-27T01:03:51.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:03:51.539 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-27T01:03:51.559 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-27T01:03:51.559 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-27T01:03:51.560 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-27T01:03:51.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:03:51.560 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-01-27T01:03:51.634 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-27T01:03:51.635 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-27T01:03:51.635 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-27T01:03:51.698 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-27T01:03:51.698 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-27T01:03:51.698 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-27T01:03:51.759 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-27T01:03:51.836 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-27T01:03:51.836 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-27T01:03:51.911 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-27T01:03:51.912 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-27T01:03:51.913 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:51.913 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-27T01:03:51.975 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-27T01:03:51.975 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-27T01:03:52.017 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-27T01:03:52.053 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-27T01:03:52.053 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-27T01:03:52.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-01-27T01:03:52.121+0000 7f033f51b540 -1 init, newargv = 0x55eec9ea4030 newargc=15 2024-01-27T01:03:52.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[106264]: starting ceph client 2024-01-27T01:03:52.126 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-27T01:03:52.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[106264]: starting fuse 2024-01-27T01:03:52.128 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-27T01:03:52.131 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:52.132 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-27T01:03:52.192 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-27T01:03:52.192 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-27T01:03:52.192 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-27T01:03:52.193 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-27T01:03:52.193 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-27T01:03:52.193 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-27T01:03:52.193 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-27T01:03:52.193 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:03:52.193 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-27T01:03:52.193 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-27T01:03:52.193 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-27T01:03:52.193 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-27T01:03:52.193 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:03:52.193 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-27T01:03:52.193 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-27T01:03:52.193 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-27T01:03:52.193 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-27T01:03:52.194 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:03:52.194 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-27T01:03:52.194 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-27T01:03:52.194 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-27T01:03:52.194 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-27T01:03:52.194 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-27T01:03:52.194 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-27T01:03:52.194 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-27T01:03:52.194 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-27T01:03:52.194 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:03:52.194 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-01-27T01:03:52.194 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-27T01:03:52.335 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.106264.asok 2024-01-27T01:03:52.514 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.106264.asok 2024-01-27T01:03:52.515 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:52.515 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.106264.asok status 2024-01-27T01:03:52.691 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:03:52.691 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-27T01:03:52.691 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-27T01:03:52.691 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-27T01:03:52.691 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-01-27T01:03:52.691 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-27T01:03:52.691 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-27T01:03:52.691 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "106264", 2024-01-27T01:03:52.691 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-27T01:03:52.691 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:52.692 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-27T01:03:52.692 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-27T01:03:52.692 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 10542, 2024-01-27T01:03:52.692 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-27T01:03:52.692 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-27T01:03:52.692 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-27T01:03:52.692 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 10542 2024-01-27T01:03:52.692 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:52.692 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-27T01:03:52.692 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-27T01:03:52.692 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-27T01:03:52.692 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1112373313 2024-01-27T01:03:52.692 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:52.692 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:52.692 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-27T01:03:52.693 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-27T01:03:52.693 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-27T01:03:52.693 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1112373313 2024-01-27T01:03:52.693 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:03:52.693 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.10542 192.168.0.1:0/1112373313", 2024-01-27T01:03:52.693 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/1112373313", 2024-01-27T01:03:52.693 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-27T01:03:52.693 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 107, 2024-01-27T01:03:52.693 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 205, 2024-01-27T01:03:52.693 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-27T01:03:52.693 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-27T01:03:52.693 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-27T01:03:52.693 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:03:52.702 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:52.702 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-27T01:03:52.702 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-27T01:03:52.722 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-27T01:03:52.722 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-27T01:03:52.722 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:52.723 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-27T01:03:52.797 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-27T01:03:52.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.979+0000 7f2e60f31700 1 Processor -- start 2024-01-27T01:03:52.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.979+0000 7f2e60f31700 1 -- start start 2024-01-27T01:03:52.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.980+0000 7f2e60f31700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e5c0cbdf0 0x7f2e5c0ae610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:52.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.980+0000 7f2e60f31700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c0ae180 con 0x7f2e5c0cc1c0 2024-01-27T01:03:52.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.980+0000 7f2e60f31700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c156c60 con 0x7f2e5c0ad070 2024-01-27T01:03:52.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.980+0000 7f2e60f31700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c0aeb50 con 0x7f2e5c0cbdf0 2024-01-27T01:03:52.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.980+0000 7f2e59d9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2e5c0cc1c0 0x7f2e5c0b0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58494/0 (socket says 172.21.15.46:58494) 2024-01-27T01:03:52.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.980+0000 7f2e59d9b700 1 -- 172.21.15.46:0/706102521 learned_addr learned my addr 172.21.15.46:0/706102521 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:52.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.980+0000 7f2e5a59c700 1 --2- 172.21.15.46:0/706102521 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e5c0cbdf0 0x7f2e5c0ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:52.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.980+0000 7f2e5959a700 1 -- 172.21.15.46:0/706102521 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183678203 0 0) 0x7f2e5c0ae180 con 0x7f2e5c0cc1c0 2024-01-27T01:03:52.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.980+0000 7f2e5959a700 1 -- 172.21.15.46:0/706102521 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e48003570 con 0x7f2e5c0cc1c0 2024-01-27T01:03:52.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.980+0000 7f2e5959a700 1 -- 172.21.15.46:0/706102521 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1462288986 0 0) 0x7f2e48003570 con 0x7f2e5c0cc1c0 2024-01-27T01:03:52.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.980+0000 7f2e5959a700 1 -- 172.21.15.46:0/706102521 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e5c0cbdf0 msgr2=0x7f2e5c0ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:52.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.980+0000 7f2e5959a700 1 --2- 172.21.15.46:0/706102521 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e5c0cbdf0 0x7f2e5c0ae610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:52.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.980+0000 7f2e5959a700 1 -- 172.21.15.46:0/706102521 >> v1:172.21.15.106:6789/0 conn(0x7f2e5c0ad070 legacy=0x7f2e5c0c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:52.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.980+0000 7f2e5959a700 1 -- 172.21.15.46:0/706102521 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5c0ad570 con 0x7f2e5c0cc1c0 2024-01-27T01:03:52.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.980+0000 7f2e5959a700 1 -- 172.21.15.46:0/706102521 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f2e44001470 con 0x7f2e5c0cc1c0 2024-01-27T01:03:52.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.981+0000 7f2e5959a700 1 -- 172.21.15.46:0/706102521 >> v1:172.21.15.46:6789/0 conn(0x7f2e5c0cc1c0 legacy=0x7f2e5c0b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:52.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.981+0000 7f2e5959a700 1 --2- 172.21.15.46:0/706102521 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e5c0cbdf0 0x7f2e48003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:52.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.981+0000 7f2e5959a700 1 --2- 172.21.15.46:0/706102521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e48004060 0x7f2e48004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:52.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.981+0000 7f2e5959a700 1 --2- 172.21.15.46:0/706102521 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e48004d50 0x7f2e480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:52.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.981+0000 7f2e5959a700 1 -- 172.21.15.46:0/706102521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c0aeb50 con 0x7f2e48004060 2024-01-27T01:03:52.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.981+0000 7f2e5959a700 1 -- 172.21.15.46:0/706102521 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4800bc20 con 0x7f2e48004d50 2024-01-27T01:03:52.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.981+0000 7f2e5959a700 1 -- 172.21.15.46:0/706102521 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4800bde0 con 0x7f2e5c0cbdf0 2024-01-27T01:03:52.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.981+0000 7f2e59d9b700 1 --2- 172.21.15.46:0/706102521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e48004060 0x7f2e48004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:52.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.981+0000 7f2e5a59c700 1 --2- 172.21.15.46:0/706102521 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e5c0cbdf0 0x7f2e48003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:52.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.981+0000 7f2e5ad9d700 1 --2- 172.21.15.46:0/706102521 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e48004d50 0x7f2e480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:52.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.981+0000 7f2e59d9b700 1 -- 172.21.15.46:0/706102521 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e5c0cbdf0 msgr2=0x7f2e48003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:52.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.981+0000 7f2e59d9b700 1 --2- 172.21.15.46:0/706102521 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e5c0cbdf0 0x7f2e48003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:52.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.981+0000 7f2e59d9b700 1 -- 172.21.15.46:0/706102521 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e48004d50 msgr2=0x7f2e480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:52.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.981+0000 7f2e59d9b700 1 --2- 172.21.15.46:0/706102521 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e48004d50 0x7f2e480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:52.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.981+0000 7f2e59d9b700 1 -- 172.21.15.46:0/706102521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e48003570 con 0x7f2e48004060 2024-01-27T01:03:52.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.981+0000 7f2e59d9b700 1 --2- 172.21.15.46:0/706102521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e48004060 0x7f2e48004810 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400c2d0 tx=0x7f2e44001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c4e4a957f0708 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:52.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.981+0000 7f2e5959a700 1 -- 172.21.15.46:0/706102521 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4400cba0 con 0x7f2e48004060 2024-01-27T01:03:52.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.982+0000 7f2e5959a700 1 -- 172.21.15.46:0/706102521 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4400cd20 con 0x7f2e48004060 2024-01-27T01:03:52.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.982+0000 7f2e5959a700 1 -- 172.21.15.46:0/706102521 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e44005b70 con 0x7f2e48004060 2024-01-27T01:03:52.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.982+0000 7f2e60f31700 1 -- 172.21.15.46:0/706102521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e48004060 msgr2=0x7f2e48004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:52.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.982+0000 7f2e60f31700 1 --2- 172.21.15.46:0/706102521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e48004060 0x7f2e48004810 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400c2d0 tx=0x7f2e44001d80 comp rx=0 tx=0).stop 2024-01-27T01:03:52.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.982+0000 7f2e5ad9d700 1 -- 172.21.15.46:0/706102521 reap_dead start 2024-01-27T01:03:52.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.982+0000 7f2e60f31700 1 -- 172.21.15.46:0/706102521 shutdown_connections 2024-01-27T01:03:52.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.982+0000 7f2e60f31700 1 -- 172.21.15.46:0/706102521 >> 172.21.15.46:0/706102521 conn(0x7f2e5c068400 msgr2=0x7f2e5c0cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:52.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.982+0000 7f2e60f31700 1 -- 172.21.15.46:0/706102521 shutdown_connections 2024-01-27T01:03:52.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.982+0000 7f2e60f31700 1 -- 172.21.15.46:0/706102521 wait complete. 2024-01-27T01:03:52.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.984+0000 7f2e60f31700 1 Processor -- start 2024-01-27T01:03:52.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.984+0000 7f2e60f31700 1 -- start start 2024-01-27T01:03:52.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.985+0000 7f2e60f31700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e48004060 0x7f2e5c16a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:52.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.985+0000 7f2e60f31700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e5c0a5e40 0x7f2e5c167100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:52.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.985+0000 7f2e60f31700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e5c169e70 0x7f2e5c0af780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:52.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.985+0000 7f2e60f31700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c0a70f0 con 0x7f2e48004060 2024-01-27T01:03:52.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.985+0000 7f2e60f31700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c0a76f0 con 0x7f2e5c0a5e40 2024-01-27T01:03:52.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.985+0000 7f2e60f31700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c167640 con 0x7f2e5c169e70 2024-01-27T01:03:52.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.985+0000 7f2e5a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e48004060 0x7f2e5c16a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:52.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.985+0000 7f2e5a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e48004060 0x7f2e5c16a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33904/0 (socket says 172.21.15.46:33904) 2024-01-27T01:03:52.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.985+0000 7f2e5a59c700 1 -- 172.21.15.46:0/2420911789 learned_addr learned my addr 172.21.15.46:0/2420911789 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:52.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.985+0000 7f2e5ad9d700 1 --2- 172.21.15.46:0/2420911789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e5c169e70 0x7f2e5c0af780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:52.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.985+0000 7f2e59d9b700 1 --2- 172.21.15.46:0/2420911789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e5c0a5e40 0x7f2e5c167100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:52.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.985+0000 7f2e5a59c700 1 -- 172.21.15.46:0/2420911789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e5c169e70 msgr2=0x7f2e5c0af780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:52.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.985+0000 7f2e5a59c700 1 --2- 172.21.15.46:0/2420911789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e5c169e70 0x7f2e5c0af780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:52.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.985+0000 7f2e5a59c700 1 -- 172.21.15.46:0/2420911789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e5c0a5e40 msgr2=0x7f2e5c167100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:52.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.985+0000 7f2e5a59c700 1 --2- 172.21.15.46:0/2420911789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e5c0a5e40 0x7f2e5c167100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:52.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.985+0000 7f2e5a59c700 1 -- 172.21.15.46:0/2420911789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5c16ad20 con 0x7f2e48004060 2024-01-27T01:03:52.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.986+0000 7f2e5a59c700 1 --2- 172.21.15.46:0/2420911789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e48004060 0x7f2e5c16a740 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c011da0 tx=0x7f2e4c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:52.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.986+0000 7f2e527fc700 1 -- 172.21.15.46:0/2420911789 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c009e20 con 0x7f2e48004060 2024-01-27T01:03:52.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.986+0000 7f2e527fc700 1 -- 172.21.15.46:0/2420911789 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4c00fcd0 con 0x7f2e48004060 2024-01-27T01:03:52.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.986+0000 7f2e60f31700 1 -- 172.21.15.46:0/2420911789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e5c16b000 con 0x7f2e48004060 2024-01-27T01:03:52.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.986+0000 7f2e527fc700 1 -- 172.21.15.46:0/2420911789 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c014650 con 0x7f2e48004060 2024-01-27T01:03:52.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.986+0000 7f2e60f31700 1 -- 172.21.15.46:0/2420911789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e5c16b5c0 con 0x7f2e48004060 2024-01-27T01:03:52.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.987+0000 7f2e60f31700 1 -- 172.21.15.46:0/2420911789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e5c0abd60 con 0x7f2e48004060 2024-01-27T01:03:52.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.987+0000 7f2e527fc700 1 -- 172.21.15.46:0/2420911789 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2e4c014a80 con 0x7f2e48004060 2024-01-27T01:03:52.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.988+0000 7f2e527fc700 1 --2- 172.21.15.46:0/2420911789 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2e3c066f30 0x7f2e3c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:52.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.988+0000 7f2e527fc700 1 -- 172.21.15.46:0/2420911789 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f2e4c018030 con 0x7f2e48004060 2024-01-27T01:03:52.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.988+0000 7f2e59d9b700 1 --2- 172.21.15.46:0/2420911789 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2e3c066f30 0x7f2e3c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:52.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.989+0000 7f2e59d9b700 1 --2- 172.21.15.46:0/2420911789 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2e3c066f30 0x7f2e3c0693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c0a7570 tx=0x7f2e44015000 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:52.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:52.992+0000 7f2e527fc700 1 -- 172.21.15.46:0/2420911789 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2e4c052300 con 0x7f2e48004060 2024-01-27T01:03:53.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.226+0000 7f2e60f31700 1 -- 172.21.15.46:0/2420911789 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e5c0a7830 con 0x7f2e3c066f30 2024-01-27T01:03:53.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.228+0000 7f2e527fc700 1 -- 172.21.15.46:0/2420911789 <== mgr.10182 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f2e44022070 con 0x7f2e3c066f30 2024-01-27T01:03:53.228 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-27T01:03:53.228 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:03:53.228 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephfs" 2024-01-27T01:03:53.228 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:03:53.228 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-27T01:03:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.230+0000 7f2e60f31700 1 -- 172.21.15.46:0/2420911789 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2e3c066f30 msgr2=0x7f2e3c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.230+0000 7f2e60f31700 1 --2- 172.21.15.46:0/2420911789 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2e3c066f30 0x7f2e3c0693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c0a7570 tx=0x7f2e44015000 comp rx=0 tx=0).stop 2024-01-27T01:03:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.230+0000 7f2e60f31700 1 -- 172.21.15.46:0/2420911789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e48004060 msgr2=0x7f2e5c16a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.230+0000 7f2e60f31700 1 --2- 172.21.15.46:0/2420911789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e48004060 0x7f2e5c16a740 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c011da0 tx=0x7f2e4c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.230+0000 7f2e60f31700 1 -- 172.21.15.46:0/2420911789 shutdown_connections 2024-01-27T01:03:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.230+0000 7f2e60f31700 1 --2- 172.21.15.46:0/2420911789 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2e3c066f30 0x7f2e3c0693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.230+0000 7f2e60f31700 1 --2- 172.21.15.46:0/2420911789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e48004060 0x7f2e5c16a740 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.230+0000 7f2e60f31700 1 --2- 172.21.15.46:0/2420911789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2e5c169e70 0x7f2e5c0af780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.230+0000 7f2e60f31700 1 --2- 172.21.15.46:0/2420911789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e5c0a5e40 0x7f2e5c167100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.230+0000 7f2e60f31700 1 -- 172.21.15.46:0/2420911789 >> 172.21.15.46:0/2420911789 conn(0x7f2e5c068400 msgr2=0x7f2e5c1548e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:53.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.230+0000 7f2e60f31700 1 -- 172.21.15.46:0/2420911789 shutdown_connections 2024-01-27T01:03:53.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.230+0000 7f2e60f31700 1 -- 172.21.15.46:0/2420911789 wait complete. 2024-01-27T01:03:53.244 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-27T01:03:53.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.377+0000 7fef43837700 1 Processor -- start 2024-01-27T01:03:53.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.378+0000 7fef43837700 1 -- start start 2024-01-27T01:03:53.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.379+0000 7fef43837700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fef3c156940 0x7fef3c06c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:53.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.379+0000 7fef43837700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef3c06f9a0 0x7fef3c06eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:53.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.379+0000 7fef43837700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef3c0c8b90 con 0x7fef3c156570 2024-01-27T01:03:53.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.379+0000 7fef43837700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef3c06d0b0 con 0x7fef3c156940 2024-01-27T01:03:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.379+0000 7fef43837700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c0c8e50 con 0x7fef3c06f9a0 2024-01-27T01:03:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.379+0000 7fef40dd2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fef3c156940 0x7fef3c06c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.379+0000 7fef415d3700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fef3c156570 0x7fef3c0c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58502/0 (socket says 172.21.15.46:58502) 2024-01-27T01:03:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.379+0000 7fef415d3700 1 -- 172.21.15.46:0/3146604843 learned_addr learned my addr 172.21.15.46:0/3146604843 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.379+0000 7fef40dd2700 1 --2- 172.21.15.46:0/3146604843 >> v2:172.21.15.46:3300/0 conn(0x7fef3c156940 0x7fef3c06c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33908/0 (socket says 172.21.15.46:33908) 2024-01-27T01:03:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.379+0000 7fef41dd4700 1 --2- 172.21.15.46:0/3146604843 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef3c06f9a0 0x7fef3c06eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.379+0000 7fef40dd2700 1 -- 172.21.15.46:0/3146604843 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef3c06f9a0 msgr2=0x7fef3c06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:03:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.379+0000 7fef40dd2700 1 --2- 172.21.15.46:0/3146604843 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef3c06f9a0 0x7fef3c06eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.379+0000 7fef40dd2700 1 -- 172.21.15.46:0/3146604843 >> v1:172.21.15.46:6789/0 conn(0x7fef3c156570 legacy=0x7fef3c0c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.379+0000 7fef2ffff700 1 -- 172.21.15.46:0/3146604843 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234553442 0 0) 0x7fef3c0c8b90 con 0x7fef3c156570 2024-01-27T01:03:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.379+0000 7fef40dd2700 1 -- 172.21.15.46:0/3146604843 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3c0c4c40 con 0x7fef3c156940 2024-01-27T01:03:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.379+0000 7fef40dd2700 1 --2- 172.21.15.46:0/3146604843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c156940 0x7fef3c06c110 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fef380098e0 tx=0x7fef380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d91794b5d283d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.379+0000 7fef2ffff700 1 -- 172.21.15.46:0/3146604843 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3800b640 con 0x7fef3c156940 2024-01-27T01:03:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.380+0000 7fef2ffff700 1 -- 172.21.15.46:0/3146604843 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3800b7a0 con 0x7fef3c156940 2024-01-27T01:03:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.380+0000 7fef2ffff700 1 -- 172.21.15.46:0/3146604843 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3800bb40 con 0x7fef3c156940 2024-01-27T01:03:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.380+0000 7fef43837700 1 -- 172.21.15.46:0/3146604843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c156940 msgr2=0x7fef3c06c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.380+0000 7fef43837700 1 --2- 172.21.15.46:0/3146604843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c156940 0x7fef3c06c110 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fef380098e0 tx=0x7fef380091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.380+0000 7fef43837700 1 -- 172.21.15.46:0/3146604843 shutdown_connections 2024-01-27T01:03:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.380+0000 7fef43837700 1 --2- 172.21.15.46:0/3146604843 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef3c06f9a0 0x7fef3c06eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.380+0000 7fef43837700 1 --2- 172.21.15.46:0/3146604843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c156940 0x7fef3c06c110 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.380+0000 7fef43837700 1 -- 172.21.15.46:0/3146604843 >> 172.21.15.46:0/3146604843 conn(0x7fef3c0cdf50 msgr2=0x7fef3c0c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.380+0000 7fef43837700 1 -- 172.21.15.46:0/3146604843 shutdown_connections 2024-01-27T01:03:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.380+0000 7fef43837700 1 -- 172.21.15.46:0/3146604843 wait complete. 2024-01-27T01:03:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.381+0000 7fef43837700 1 Processor -- start 2024-01-27T01:03:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.382+0000 7fef43837700 1 -- start start 2024-01-27T01:03:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.382+0000 7fef43837700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c06f9a0 0x7fef3c143f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.382+0000 7fef43837700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef3c156570 0x7fef3c158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.382+0000 7fef43837700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef3c156940 0x7fef3c1590c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.382+0000 7fef43837700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c0ad2c0 con 0x7fef3c06f9a0 2024-01-27T01:03:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.382+0000 7fef43837700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c0b12f0 con 0x7fef3c156570 2024-01-27T01:03:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.382+0000 7fef43837700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c0b15d0 con 0x7fef3c156940 2024-01-27T01:03:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef415d3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c06f9a0 0x7fef3c143f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef415d3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c06f9a0 0x7fef3c143f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33924/0 (socket says 172.21.15.46:33924) 2024-01-27T01:03:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef415d3700 1 -- 172.21.15.46:0/4136546763 learned_addr learned my addr 172.21.15.46:0/4136546763 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef40dd2700 1 --2- 172.21.15.46:0/4136546763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef3c156570 0x7fef3c158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef41dd4700 1 --2- 172.21.15.46:0/4136546763 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef3c156940 0x7fef3c1590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef415d3700 1 -- 172.21.15.46:0/4136546763 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef3c156940 msgr2=0x7fef3c1590c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef415d3700 1 --2- 172.21.15.46:0/4136546763 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef3c156940 0x7fef3c1590c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef415d3700 1 -- 172.21.15.46:0/4136546763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef3c156570 msgr2=0x7fef3c158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef415d3700 1 --2- 172.21.15.46:0/4136546763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef3c156570 0x7fef3c158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef415d3700 1 -- 172.21.15.46:0/4136546763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef38009420 con 0x7fef3c06f9a0 2024-01-27T01:03:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef41dd4700 1 --2- 172.21.15.46:0/4136546763 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef3c156940 0x7fef3c1590c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:03:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef40dd2700 1 --2- 172.21.15.46:0/4136546763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef3c156570 0x7fef3c158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:03:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef415d3700 1 --2- 172.21.15.46:0/4136546763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c06f9a0 0x7fef3c143f90 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fef3000c7e0 tx=0x7fef30007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef2d7fa700 1 -- 172.21.15.46:0/4136546763 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef30005b30 con 0x7fef3c06f9a0 2024-01-27T01:03:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef2d7fa700 1 -- 172.21.15.46:0/4136546763 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef30005c90 con 0x7fef3c06f9a0 2024-01-27T01:03:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef43837700 1 -- 172.21.15.46:0/4136546763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef3c0b18b0 con 0x7fef3c06f9a0 2024-01-27T01:03:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef2d7fa700 1 -- 172.21.15.46:0/4136546763 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef30007300 con 0x7fef3c06f9a0 2024-01-27T01:03:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.383+0000 7fef43837700 1 -- 172.21.15.46:0/4136546763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef3c1444d0 con 0x7fef3c06f9a0 2024-01-27T01:03:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.384+0000 7fef43837700 1 -- 172.21.15.46:0/4136546763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef3c0c87c0 con 0x7fef3c06f9a0 2024-01-27T01:03:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.385+0000 7fef2d7fa700 1 -- 172.21.15.46:0/4136546763 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 114) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fef3001b400 con 0x7fef3c06f9a0 2024-01-27T01:03:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.385+0000 7fef2d7fa700 1 --2- 172.21.15.46:0/4136546763 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fef20066f30 0x7fef200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:53.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.385+0000 7fef2d7fa700 1 -- 172.21.15.46:0/4136546763 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fef30084390 con 0x7fef3c06f9a0 2024-01-27T01:03:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.386+0000 7fef40dd2700 1 --2- 172.21.15.46:0/4136546763 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fef20066f30 0x7fef200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.386+0000 7fef40dd2700 1 --2- 172.21.15.46:0/4136546763 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fef20066f30 0x7fef200693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fef3c16ade0 tx=0x7fef38017000 comp rx=0 tx=0).ready entity=mgr.10182 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.390+0000 7fef2d7fa700 1 -- 172.21.15.46:0/4136546763 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fef30051370 con 0x7fef3c06f9a0 2024-01-27T01:03:53.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.657+0000 7fef43837700 1 -- 172.21.15.46:0/4136546763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fef3c0ad400 con 0x7fef3c06f9a0 2024-01-27T01:03:53.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.755+0000 7fef2d7fa700 1 -- 172.21.15.46:0/4136546763 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 115) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fef30051740 con 0x7fef3c06f9a0 2024-01-27T01:03:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.756+0000 7fef2d7fa700 1 -- 172.21.15.46:0/4136546763 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v115) v1 ==== 92+0+0 (secure 0 0 0) 0x7fef3001ae40 con 0x7fef3c06f9a0 2024-01-27T01:03:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.759+0000 7fef43837700 1 -- 172.21.15.46:0/4136546763 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fef20066f30 msgr2=0x7fef200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.759+0000 7fef43837700 1 --2- 172.21.15.46:0/4136546763 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fef20066f30 0x7fef200693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fef3c16ade0 tx=0x7fef38017000 comp rx=0 tx=0).stop 2024-01-27T01:03:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.759+0000 7fef43837700 1 -- 172.21.15.46:0/4136546763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c06f9a0 msgr2=0x7fef3c143f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.759+0000 7fef43837700 1 --2- 172.21.15.46:0/4136546763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c06f9a0 0x7fef3c143f90 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fef3000c7e0 tx=0x7fef30007df0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.759+0000 7fef43837700 1 -- 172.21.15.46:0/4136546763 shutdown_connections 2024-01-27T01:03:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.759+0000 7fef43837700 1 --2- 172.21.15.46:0/4136546763 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fef20066f30 0x7fef200693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.759+0000 7fef43837700 1 --2- 172.21.15.46:0/4136546763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef3c06f9a0 0x7fef3c143f90 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.759+0000 7fef43837700 1 --2- 172.21.15.46:0/4136546763 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef3c156940 0x7fef3c1590c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.759+0000 7fef43837700 1 --2- 172.21.15.46:0/4136546763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef3c156570 0x7fef3c158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.759+0000 7fef43837700 1 -- 172.21.15.46:0/4136546763 >> 172.21.15.46:0/4136546763 conn(0x7fef3c0cdf50 msgr2=0x7fef3c0c9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.759+0000 7fef43837700 1 -- 172.21.15.46:0/4136546763 shutdown_connections 2024-01-27T01:03:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.759+0000 7fef43837700 1 -- 172.21.15.46:0/4136546763 wait complete. 2024-01-27T01:03:53.777 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-27T01:03:53.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.915+0000 7f1bc4957700 1 Processor -- start 2024-01-27T01:03:53.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.915+0000 7f1bc4957700 1 -- start start 2024-01-27T01:03:53.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.916+0000 7f1bc4957700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1bc00afb40 0x7f1bc00c5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:53.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.916+0000 7f1bc4957700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc00cc100 con 0x7f1bc00cf050 2024-01-27T01:03:53.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.916+0000 7f1bc4957700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc00cb850 con 0x7f1bc0159cb0 2024-01-27T01:03:53.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.916+0000 7f1bc4957700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bc00ccca0 con 0x7f1bc00afb40 2024-01-27T01:03:53.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.916+0000 7f1bbed9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1bc00afb40 0x7f1bc00c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:53.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.916+0000 7f1bbed9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1bc00afb40 0x7f1bc00c5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33948/0 (socket says 172.21.15.46:33948) 2024-01-27T01:03:53.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.916+0000 7f1bbed9d700 1 -- 172.21.15.46:0/1876432552 learned_addr learned my addr 172.21.15.46:0/1876432552 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:53.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.916+0000 7f1bb5d9b700 1 --1- 172.21.15.46:0/1876432552 >> v1:172.21.15.46:6789/0 conn(0x7f1bc00cf050 0x7f1bc00af430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58520/0 (socket says 172.21.15.46:58520) 2024-01-27T01:03:53.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.916+0000 7f1bbed9d700 1 -- 172.21.15.46:0/1876432552 >> v1:172.21.15.106:6789/0 conn(0x7f1bc0159cb0 legacy=0x7f1bc00cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:53.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.916+0000 7f1bbdd9b700 1 -- 172.21.15.46:0/1876432552 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701971852 0 0) 0x7f1bc00cb850 con 0x7f1bc0159cb0 2024-01-27T01:03:53.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.916+0000 7f1bbed9d700 1 -- 172.21.15.46:0/1876432552 >> v1:172.21.15.46:6789/0 conn(0x7f1bc00cf050 legacy=0x7f1bc00af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:53.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.916+0000 7f1bbed9d700 1 -- 172.21.15.46:0/1876432552 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc00cbaf0 con 0x7f1bc00afb40 2024-01-27T01:03:53.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.917+0000 7f1bbed9d700 1 --2- 172.21.15.46:0/1876432552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bc00afb40 0x7f1bc00c5530 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8009f20 tx=0x7f1bb8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f1daa50acbc1694 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:53.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.917+0000 7f1bbdd9b700 1 -- 172.21.15.46:0/1876432552 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb800e070 con 0x7f1bc00afb40 2024-01-27T01:03:53.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.917+0000 7f1bbdd9b700 1 -- 172.21.15.46:0/1876432552 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb800ba80 con 0x7f1bc00afb40 2024-01-27T01:03:53.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.917+0000 7f1bbdd9b700 1 -- 172.21.15.46:0/1876432552 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb80092e0 con 0x7f1bc00afb40 2024-01-27T01:03:53.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.917+0000 7f1bc4957700 1 -- 172.21.15.46:0/1876432552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bc00afb40 msgr2=0x7f1bc00c5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:53.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.917+0000 7f1bc4957700 1 --2- 172.21.15.46:0/1876432552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bc00afb40 0x7f1bc00c5530 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8009f20 tx=0x7f1bb8009840 comp rx=0 tx=0).stop 2024-01-27T01:03:53.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.918+0000 7f1bc4957700 1 -- 172.21.15.46:0/1876432552 shutdown_connections 2024-01-27T01:03:53.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.918+0000 7f1bc4957700 1 --2- 172.21.15.46:0/1876432552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bc00afb40 0x7f1bc00c5530 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.918+0000 7f1bc4957700 1 -- 172.21.15.46:0/1876432552 >> 172.21.15.46:0/1876432552 conn(0x7f1bc0087f60 msgr2=0x7f1bc006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:53.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.918+0000 7f1bc4957700 1 -- 172.21.15.46:0/1876432552 shutdown_connections 2024-01-27T01:03:53.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.918+0000 7f1bc4957700 1 -- 172.21.15.46:0/1876432552 wait complete. 2024-01-27T01:03:53.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.919+0000 7f1bc4957700 1 Processor -- start 2024-01-27T01:03:53.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.919+0000 7f1bc4957700 1 -- start start 2024-01-27T01:03:53.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.920+0000 7f1bc4957700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bc00afb40 0x7f1bc00b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:53.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.920+0000 7f1bc4957700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bc00cf050 0x7f1bc016a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:53.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.920+0000 7f1bc4957700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bc0159cb0 0x7f1bc016cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:53.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.920+0000 7f1bc4957700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc0140f40 con 0x7f1bc00cf050 2024-01-27T01:03:53.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.920+0000 7f1bc4957700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc00b0910 con 0x7f1bc0159cb0 2024-01-27T01:03:53.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.920+0000 7f1bc4957700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc00b3ad0 con 0x7f1bc00afb40 2024-01-27T01:03:53.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.920+0000 7f1bb5d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bc00cf050 0x7f1bc016a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:53.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.920+0000 7f1bb5d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bc00cf050 0x7f1bc016a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33950/0 (socket says 172.21.15.46:33950) 2024-01-27T01:03:53.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.920+0000 7f1bb5d9b700 1 -- 172.21.15.46:0/543703851 learned_addr learned my addr 172.21.15.46:0/543703851 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:53.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.920+0000 7f1bbed9d700 1 --2- 172.21.15.46:0/543703851 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bc0159cb0 0x7f1bc016cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:53.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.920+0000 7f1bbe59c700 1 --2- 172.21.15.46:0/543703851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bc00afb40 0x7f1bc00b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:53.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.920+0000 7f1bb5d9b700 1 -- 172.21.15.46:0/543703851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bc00afb40 msgr2=0x7f1bc00b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:53.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.920+0000 7f1bb5d9b700 1 --2- 172.21.15.46:0/543703851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bc00afb40 0x7f1bc00b00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.920+0000 7f1bb5d9b700 1 -- 172.21.15.46:0/543703851 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bc0159cb0 msgr2=0x7f1bc016cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:53.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.920+0000 7f1bb5d9b700 1 --2- 172.21.15.46:0/543703851 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bc0159cb0 0x7f1bc016cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:53.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.920+0000 7f1bb5d9b700 1 -- 172.21.15.46:0/543703851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb8009ae0 con 0x7f1bc00cf050 2024-01-27T01:03:53.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.921+0000 7f1bb5d9b700 1 --2- 172.21.15.46:0/543703851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bc00cf050 0x7f1bc016a830 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800c7c0 tx=0x7f1ba8007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:53.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.921+0000 7f1bb6ffd700 1 -- 172.21.15.46:0/543703851 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8005a70 con 0x7f1bc00cf050 2024-01-27T01:03:53.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.921+0000 7f1bb6ffd700 1 -- 172.21.15.46:0/543703851 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba8005bd0 con 0x7f1bc00cf050 2024-01-27T01:03:53.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.921+0000 7f1bc4957700 1 -- 172.21.15.46:0/543703851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc00b3fc0 con 0x7f1bc00cf050 2024-01-27T01:03:53.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.921+0000 7f1bb6ffd700 1 -- 172.21.15.46:0/543703851 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba80070d0 con 0x7f1bc00cf050 2024-01-27T01:03:53.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.921+0000 7f1bc4957700 1 -- 172.21.15.46:0/543703851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc00b4580 con 0x7f1bc00cf050 2024-01-27T01:03:53.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.922+0000 7f1bc4957700 1 -- 172.21.15.46:0/543703851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc00b47a0 con 0x7f1bc00cf050 2024-01-27T01:03:53.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.923+0000 7f1bb6ffd700 1 -- 172.21.15.46:0/543703851 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 115) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f1ba8007440 con 0x7f1bc00cf050 2024-01-27T01:03:53.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.923+0000 7f1bb6ffd700 1 --2- 172.21.15.46:0/543703851 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ba0066f80 0x7f1ba0069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:53.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.923+0000 7f1bb6ffd700 1 -- 172.21.15.46:0/543703851 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f1ba8083cb0 con 0x7f1bc00cf050 2024-01-27T01:03:53.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.923+0000 7f1bbe59c700 1 -- 172.21.15.46:0/543703851 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ba0066f80 msgr2=0x7f1ba0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:53.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.923+0000 7f1bbe59c700 1 --2- 172.21.15.46:0/543703851 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ba0066f80 0x7f1ba0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:03:53.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:53.927+0000 7f1bb6ffd700 1 -- 172.21.15.46:0/543703851 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1ba8050c10 con 0x7f1bc00cf050 2024-01-27T01:03:54.093 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:54.093+0000 7f7a830b21c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:03:54.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.124+0000 7f1bbe59c700 1 -- 172.21.15.46:0/543703851 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ba0066f80 msgr2=0x7f1ba0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:54.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.124+0000 7f1bbe59c700 1 --2- 172.21.15.46:0/543703851 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ba0066f80 0x7f1ba0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:03:54.152 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:54.152+0000 7fbc50f561c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:03:54.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.158+0000 7f1bc4957700 1 -- 172.21.15.46:0/543703851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f1bc0068360 con 0x7f1bc00cf050 2024-01-27T01:03:54.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.162+0000 7f1bb6ffd700 1 -- 172.21.15.46:0/543703851 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v71)=0 v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1ba8017070 con 0x7f1bc00cf050 2024-01-27T01:03:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.164+0000 7f1bc4957700 1 -- 172.21.15.46:0/543703851 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ba0066f80 msgr2=0x7f1ba0069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:03:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.164+0000 7f1bc4957700 1 --2- 172.21.15.46:0/543703851 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ba0066f80 0x7f1ba0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.164+0000 7f1bc4957700 1 -- 172.21.15.46:0/543703851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bc00cf050 msgr2=0x7f1bc016a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.164+0000 7f1bc4957700 1 --2- 172.21.15.46:0/543703851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bc00cf050 0x7f1bc016a830 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800c7c0 tx=0x7f1ba8007f30 comp rx=0 tx=0).stop 2024-01-27T01:03:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.164+0000 7f1bc4957700 1 -- 172.21.15.46:0/543703851 shutdown_connections 2024-01-27T01:03:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.165+0000 7f1bc4957700 1 --2- 172.21.15.46:0/543703851 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ba0066f80 0x7f1ba0069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.165+0000 7f1bc4957700 1 --2- 172.21.15.46:0/543703851 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bc0159cb0 0x7f1bc016cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.165+0000 7f1bc4957700 1 --2- 172.21.15.46:0/543703851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bc00afb40 0x7f1bc00b00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.165+0000 7f1bc4957700 1 --2- 172.21.15.46:0/543703851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bc00cf050 0x7f1bc016a830 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.165+0000 7f1bc4957700 1 -- 172.21.15.46:0/543703851 >> 172.21.15.46:0/543703851 conn(0x7f1bc0087f60 msgr2=0x7f1bc01556e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.165+0000 7f1bc4957700 1 -- 172.21.15.46:0/543703851 shutdown_connections 2024-01-27T01:03:54.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.165+0000 7f1bc4957700 1 -- 172.21.15.46:0/543703851 wait complete. 2024-01-27T01:03:54.171 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:54.172+0000 7f7a830b21c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:03:54.179 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-27T01:03:54.242 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:54.242+0000 7f7a830b21c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:03:54.244 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:54.244+0000 7fbc50f561c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:03:54.307 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:54.307+0000 7f7a830b21c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:03:54.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.315+0000 7f66d0676700 1 Processor -- start 2024-01-27T01:03:54.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.315+0000 7f66d0676700 1 -- start start 2024-01-27T01:03:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.316+0000 7f66d0676700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f66c8159da0 0x7f66c80ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.316+0000 7f66d0676700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c815a200 con 0x7f66c815d8c0 2024-01-27T01:03:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.316+0000 7f66d0676700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c80b1a60 con 0x7f66c81599d0 2024-01-27T01:03:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.316+0000 7f66d0676700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c80b11c0 con 0x7f66c8159da0 2024-01-27T01:03:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.317+0000 7f66cdc11700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f66c8159da0 0x7f66c80ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:54.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.317+0000 7f66cdc11700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f66c8159da0 0x7f66c80ab1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33966/0 (socket says 172.21.15.46:33966) 2024-01-27T01:03:54.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.317+0000 7f66cdc11700 1 -- 172.21.15.46:0/2156865825 learned_addr learned my addr 172.21.15.46:0/2156865825 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:54.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.317+0000 7f66cd410700 1 -- 172.21.15.46:0/2156865825 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074066345 0 0) 0x7f66c80b1a60 con 0x7f66c81599d0 2024-01-27T01:03:54.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.317+0000 7f66cdc11700 1 -- 172.21.15.46:0/2156865825 >> v1:172.21.15.106:6789/0 conn(0x7f66c81599d0 legacy=0x7f66c80b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:54.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.317+0000 7f66cdc11700 1 -- 172.21.15.46:0/2156865825 >> v1:172.21.15.46:6789/0 conn(0x7f66c815d8c0 legacy=0x7f66c80b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:54.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.317+0000 7f66cdc11700 1 -- 172.21.15.46:0/2156865825 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c80b14a0 con 0x7f66c8159da0 2024-01-27T01:03:54.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.317+0000 7f66cdc11700 1 --2- 172.21.15.46:0/2156865825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66c8159da0 0x7f66c80ab1c0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f66b80099b0 tx=0x7f66b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f1a46f92fef10 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:54.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.317+0000 7f66cd410700 1 -- 172.21.15.46:0/2156865825 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b800b860 con 0x7f66c8159da0 2024-01-27T01:03:54.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.317+0000 7f66cd410700 1 -- 172.21.15.46:0/2156865825 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b800b9c0 con 0x7f66c8159da0 2024-01-27T01:03:54.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.317+0000 7f66cd410700 1 -- 172.21.15.46:0/2156865825 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b800bd60 con 0x7f66c8159da0 2024-01-27T01:03:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.318+0000 7f66d0676700 1 -- 172.21.15.46:0/2156865825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66c8159da0 msgr2=0x7f66c80ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.318+0000 7f66d0676700 1 --2- 172.21.15.46:0/2156865825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66c8159da0 0x7f66c80ab1c0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f66b80099b0 tx=0x7f66b8009250 comp rx=0 tx=0).stop 2024-01-27T01:03:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.318+0000 7f66d0676700 1 -- 172.21.15.46:0/2156865825 shutdown_connections 2024-01-27T01:03:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.318+0000 7f66d0676700 1 --2- 172.21.15.46:0/2156865825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66c8159da0 0x7f66c80ab1c0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.318+0000 7f66d0676700 1 -- 172.21.15.46:0/2156865825 >> 172.21.15.46:0/2156865825 conn(0x7f66c806f840 msgr2=0x7f66c8087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:54.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.318+0000 7f66d0676700 1 -- 172.21.15.46:0/2156865825 shutdown_connections 2024-01-27T01:03:54.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.318+0000 7f66d0676700 1 -- 172.21.15.46:0/2156865825 wait complete. 2024-01-27T01:03:54.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.320+0000 7f66d0676700 1 Processor -- start 2024-01-27T01:03:54.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.320+0000 7f66d0676700 1 -- start start 2024-01-27T01:03:54.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.321+0000 7f66d0676700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66c81599d0 0x7f66c8153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:54.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.321+0000 7f66d0676700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66c8159da0 0x7f66c815c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:54.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.321+0000 7f66d0676700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66c815d8c0 0x7f66c815cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:54.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.321+0000 7f66d0676700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c815f0b0 con 0x7f66c8159da0 2024-01-27T01:03:54.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.321+0000 7f66d0676700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c8155040 con 0x7f66c81599d0 2024-01-27T01:03:54.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.321+0000 7f66d0676700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c81552e0 con 0x7f66c815d8c0 2024-01-27T01:03:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.321+0000 7f66cdc11700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66c8159da0 0x7f66c815c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.321+0000 7f66cdc11700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66c8159da0 0x7f66c815c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33982/0 (socket says 172.21.15.46:33982) 2024-01-27T01:03:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.321+0000 7f66cdc11700 1 -- 172.21.15.46:0/2784215384 learned_addr learned my addr 172.21.15.46:0/2784215384 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.321+0000 7f66ce412700 1 --2- 172.21.15.46:0/2784215384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66c81599d0 0x7f66c8153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.321+0000 7f66cec13700 1 --2- 172.21.15.46:0/2784215384 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66c815d8c0 0x7f66c815cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.321+0000 7f66cdc11700 1 -- 172.21.15.46:0/2784215384 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66c815d8c0 msgr2=0x7f66c815cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.321+0000 7f66cdc11700 1 --2- 172.21.15.46:0/2784215384 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66c815d8c0 0x7f66c815cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.321+0000 7f66cdc11700 1 -- 172.21.15.46:0/2784215384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66c81599d0 msgr2=0x7f66c8153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.321+0000 7f66cdc11700 1 --2- 172.21.15.46:0/2784215384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66c81599d0 0x7f66c8153100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.321+0000 7f66cdc11700 1 -- 172.21.15.46:0/2784215384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b80094f0 con 0x7f66c8159da0 2024-01-27T01:03:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.322+0000 7f66cdc11700 1 --2- 172.21.15.46:0/2784215384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66c8159da0 0x7f66c815c9d0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f66b8006f50 tx=0x7f66b8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.322+0000 7f66be7fc700 1 -- 172.21.15.46:0/2784215384 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b801f070 con 0x7f66c8159da0 2024-01-27T01:03:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.322+0000 7f66be7fc700 1 -- 172.21.15.46:0/2784215384 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b8011bb0 con 0x7f66c8159da0 2024-01-27T01:03:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.322+0000 7f66d0676700 1 -- 172.21.15.46:0/2784215384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c8153640 con 0x7f66c8159da0 2024-01-27T01:03:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.322+0000 7f66be7fc700 1 -- 172.21.15.46:0/2784215384 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b8025420 con 0x7f66c8159da0 2024-01-27T01:03:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.322+0000 7f66d0676700 1 -- 172.21.15.46:0/2784215384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c81547c0 con 0x7f66c8159da0 2024-01-27T01:03:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.323+0000 7f66d0676700 1 -- 172.21.15.46:0/2784215384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c804c710 con 0x7f66c8159da0 2024-01-27T01:03:54.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.324+0000 7f66be7fc700 1 -- 172.21.15.46:0/2784215384 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 115) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f66b8023070 con 0x7f66c8159da0 2024-01-27T01:03:54.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.324+0000 7f66be7fc700 1 --2- 172.21.15.46:0/2784215384 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f66ac066f80 0x7f66ac069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:54.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.324+0000 7f66be7fc700 1 -- 172.21.15.46:0/2784215384 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f66b8085880 con 0x7f66c8159da0 2024-01-27T01:03:54.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.324+0000 7f66ce412700 1 -- 172.21.15.46:0/2784215384 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f66ac066f80 msgr2=0x7f66ac069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:54.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.324+0000 7f66ce412700 1 --2- 172.21.15.46:0/2784215384 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f66ac066f80 0x7f66ac069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:03:54.328 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:54.328+0000 7fbc50f561c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:03:54.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.329+0000 7f66be7fc700 1 -- 172.21.15.46:0/2784215384 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f66b8052890 con 0x7f66c8159da0 2024-01-27T01:03:54.405 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:54.405+0000 7fbc50f561c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:03:54.463 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:54.463+0000 7f7a830b21c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:03:54.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.525+0000 7f66ce412700 1 -- 172.21.15.46:0/2784215384 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f66ac066f80 msgr2=0x7f66ac069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:54.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.525+0000 7f66ce412700 1 --2- 172.21.15.46:0/2784215384 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f66ac066f80 0x7f66ac069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:03:54.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.535+0000 7f66d0676700 1 -- 172.21.15.46:0/2784215384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f66c80d0560 con 0x7f66c8159da0 2024-01-27T01:03:54.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.539+0000 7f66be7fc700 1 -- 172.21.15.46:0/2784215384 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v72)=0 v72) v1 ==== 133+0+0 (secure 0 0 0) 0x7f66b8055eb0 con 0x7f66c8159da0 2024-01-27T01:03:54.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.542+0000 7f66d0676700 1 -- 172.21.15.46:0/2784215384 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f66ac066f80 msgr2=0x7f66ac069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:03:54.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.542+0000 7f66d0676700 1 --2- 172.21.15.46:0/2784215384 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f66ac066f80 0x7f66ac069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:54.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.542+0000 7f66d0676700 1 -- 172.21.15.46:0/2784215384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66c8159da0 msgr2=0x7f66c815c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:54.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.542+0000 7f66d0676700 1 --2- 172.21.15.46:0/2784215384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66c8159da0 0x7f66c815c9d0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f66b8006f50 tx=0x7f66b8017000 comp rx=0 tx=0).stop 2024-01-27T01:03:54.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.542+0000 7f66d0676700 1 -- 172.21.15.46:0/2784215384 shutdown_connections 2024-01-27T01:03:54.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.542+0000 7f66d0676700 1 --2- 172.21.15.46:0/2784215384 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f66ac066f80 0x7f66ac069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:54.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.542+0000 7f66d0676700 1 --2- 172.21.15.46:0/2784215384 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66c815d8c0 0x7f66c815cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:54.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.542+0000 7f66d0676700 1 --2- 172.21.15.46:0/2784215384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66c81599d0 0x7f66c8153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:54.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.542+0000 7f66d0676700 1 --2- 172.21.15.46:0/2784215384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66c8159da0 0x7f66c815c9d0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:54.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.542+0000 7f66d0676700 1 -- 172.21.15.46:0/2784215384 >> 172.21.15.46:0/2784215384 conn(0x7f66c806f840 msgr2=0x7f66c80b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:54.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.542+0000 7f66d0676700 1 -- 172.21.15.46:0/2784215384 shutdown_connections 2024-01-27T01:03:54.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.542+0000 7f66d0676700 1 -- 172.21.15.46:0/2784215384 wait complete. 2024-01-27T01:03:54.557 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:03:54.557 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-27T01:03:54.580 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-01-27T01:03:54.586 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:54.586+0000 7fbc50f561c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:03:54.603 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:54.604+0000 7f7a830b21c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:03:54.670 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:54.670+0000 7f7a830b21c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:03:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.722+0000 7fbb360d0700 1 Processor -- start 2024-01-27T01:03:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.722+0000 7fbb360d0700 1 -- start start 2024-01-27T01:03:54.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.723+0000 7fbb360d0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb3006b4e0 0x7fbb3006a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:54.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.723+0000 7fbb360d0700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb300a9a80 con 0x7fbb30068b00 2024-01-27T01:03:54.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.723+0000 7fbb360d0700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3015a680 con 0x7fbb30068ed0 2024-01-27T01:03:54.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.723+0000 7fbb360d0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb30069320 con 0x7fbb3006b4e0 2024-01-27T01:03:54.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.723+0000 7fbb2f7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbb30068b00 0x7fbb30159ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58542/0 (socket says 172.21.15.46:58542) 2024-01-27T01:03:54.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.723+0000 7fbb2f7fe700 1 -- 172.21.15.46:0/1479977888 learned_addr learned my addr 172.21.15.46:0/1479977888 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:54.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.723+0000 7fbb2ffff700 1 --2- 172.21.15.46:0/1479977888 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb3006b4e0 0x7fbb3006a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:54.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.724+0000 7fbb2effd700 1 -- 172.21.15.46:0/1479977888 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519450095 0 0) 0x7fbb300a9a80 con 0x7fbb30068b00 2024-01-27T01:03:54.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.724+0000 7fbb2effd700 1 -- 172.21.15.46:0/1479977888 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb1c003570 con 0x7fbb30068b00 2024-01-27T01:03:54.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.724+0000 7fbb2effd700 1 -- 172.21.15.46:0/1479977888 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364675607 0 0) 0x7fbb3015a680 con 0x7fbb30068ed0 2024-01-27T01:03:54.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.724+0000 7fbb2effd700 1 -- 172.21.15.46:0/1479977888 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb300a9a80 con 0x7fbb30068ed0 2024-01-27T01:03:54.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.724+0000 7fbb2ffff700 1 -- 172.21.15.46:0/1479977888 >> v1:172.21.15.106:6789/0 conn(0x7fbb30068ed0 legacy=0x7fbb3006add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.724+0000 7fbb2ffff700 1 -- 172.21.15.46:0/1479977888 >> v1:172.21.15.46:6789/0 conn(0x7fbb30068b00 legacy=0x7fbb30159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.724+0000 7fbb2ffff700 1 -- 172.21.15.46:0/1479977888 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3015a960 con 0x7fbb3006b4e0 2024-01-27T01:03:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.724+0000 7fbb2ffff700 1 --2- 172.21.15.46:0/1479977888 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb3006b4e0 0x7fbb3006a660 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009e90 tx=0x7fbb28009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8659aa0bb7f15f2d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.724+0000 7fbb2effd700 1 -- 172.21.15.46:0/1479977888 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2800e070 con 0x7fbb3006b4e0 2024-01-27T01:03:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.724+0000 7fbb2effd700 1 -- 172.21.15.46:0/1479977888 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2800bb80 con 0x7fbb3006b4e0 2024-01-27T01:03:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.724+0000 7fbb2effd700 1 -- 172.21.15.46:0/1479977888 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb28012430 con 0x7fbb3006b4e0 2024-01-27T01:03:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.725+0000 7fbb360d0700 1 -- 172.21.15.46:0/1479977888 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb3006b4e0 msgr2=0x7fbb3006a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.725+0000 7fbb360d0700 1 --2- 172.21.15.46:0/1479977888 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb3006b4e0 0x7fbb3006a660 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009e90 tx=0x7fbb28009780 comp rx=0 tx=0).stop 2024-01-27T01:03:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.725+0000 7fbb360d0700 1 -- 172.21.15.46:0/1479977888 shutdown_connections 2024-01-27T01:03:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.725+0000 7fbb360d0700 1 --2- 172.21.15.46:0/1479977888 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb3006b4e0 0x7fbb3006a660 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.725+0000 7fbb360d0700 1 -- 172.21.15.46:0/1479977888 >> 172.21.15.46:0/1479977888 conn(0x7fbb300ca100 msgr2=0x7fbb3006e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.725+0000 7fbb360d0700 1 -- 172.21.15.46:0/1479977888 shutdown_connections 2024-01-27T01:03:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.725+0000 7fbb360d0700 1 -- 172.21.15.46:0/1479977888 wait complete. 2024-01-27T01:03:54.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.726+0000 7fbb360d0700 1 Processor -- start 2024-01-27T01:03:54.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.726+0000 7fbb360d0700 1 -- start start 2024-01-27T01:03:54.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.727+0000 7fbb360d0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb30068b00 0x7fbb30157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:54.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.727+0000 7fbb360d0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb30068ed0 0x7fbb30169770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:54.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.727+0000 7fbb360d0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb3006b4e0 0x7fbb300ab940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:54.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.727+0000 7fbb360d0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb300b0d70 con 0x7fbb3006b4e0 2024-01-27T01:03:54.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.727+0000 7fbb360d0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb300b18f0 con 0x7fbb30068ed0 2024-01-27T01:03:54.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.727+0000 7fbb360d0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb301576b0 con 0x7fbb30068b00 2024-01-27T01:03:54.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.727+0000 7fbb2ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb3006b4e0 0x7fbb300ab940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:54.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.727+0000 7fbb2f7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb30068b00 0x7fbb30157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:54.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.727+0000 7fbb2ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb3006b4e0 0x7fbb300ab940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33992/0 (socket says 172.21.15.46:33992) 2024-01-27T01:03:54.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.727+0000 7fbb27fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb30068ed0 0x7fbb30169770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:54.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.727+0000 7fbb2ffff700 1 -- 172.21.15.46:0/3370872804 learned_addr learned my addr 172.21.15.46:0/3370872804 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:54.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.728+0000 7fbb2ffff700 1 -- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb30068b00 msgr2=0x7fbb30157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:54.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.728+0000 7fbb2ffff700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb30068b00 0x7fbb30157170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:54.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.728+0000 7fbb2ffff700 1 -- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb30068ed0 msgr2=0x7fbb30169770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:54.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.728+0000 7fbb2ffff700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb30068ed0 0x7fbb30169770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:54.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.728+0000 7fbb2ffff700 1 -- 172.21.15.46:0/3370872804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb28009a20 con 0x7fbb3006b4e0 2024-01-27T01:03:54.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.728+0000 7fbb2ffff700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb3006b4e0 0x7fbb300ab940 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fbb300b2ca0 tx=0x7fbb28019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:54.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.728+0000 7fbb277fe700 1 -- 172.21.15.46:0/3370872804 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb28027470 con 0x7fbb3006b4e0 2024-01-27T01:03:54.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.728+0000 7fbb277fe700 1 -- 172.21.15.46:0/3370872804 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb280275f0 con 0x7fbb3006b4e0 2024-01-27T01:03:54.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.728+0000 7fbb360d0700 1 -- 172.21.15.46:0/3370872804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb300b0fc0 con 0x7fbb3006b4e0 2024-01-27T01:03:54.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.728+0000 7fbb277fe700 1 -- 172.21.15.46:0/3370872804 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb280279b0 con 0x7fbb3006b4e0 2024-01-27T01:03:54.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.728+0000 7fbb360d0700 1 -- 172.21.15.46:0/3370872804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb300b1580 con 0x7fbb3006b4e0 2024-01-27T01:03:54.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.729+0000 7fbb360d0700 1 -- 172.21.15.46:0/3370872804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb300b1a80 con 0x7fbb3006b4e0 2024-01-27T01:03:54.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.730+0000 7fbb277fe700 1 -- 172.21.15.46:0/3370872804 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 115) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fbb28024070 con 0x7fbb3006b4e0 2024-01-27T01:03:54.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.730+0000 7fbb277fe700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 0x7fbb10069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:54.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.731+0000 7fbb2f7fe700 1 -- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 msgr2=0x7fbb10069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:54.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.731+0000 7fbb277fe700 1 -- 172.21.15.46:0/3370872804 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fbb28087060 con 0x7fbb3006b4e0 2024-01-27T01:03:54.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.731+0000 7fbb2f7fe700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 0x7fbb10069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:03:54.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.735+0000 7fbb277fe700 1 -- 172.21.15.46:0/3370872804 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbb28053fc0 con 0x7fbb3006b4e0 2024-01-27T01:03:54.752 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:54.752+0000 7fbc50f561c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:03:54.793 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:54.793+0000 7f7a830b21c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:03:54.830 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:54.830+0000 7fbc50f561c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:03:54.866 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:54.867+0000 7f7a830b21c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:03:54.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.932+0000 7fbb2f7fe700 1 -- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 msgr2=0x7fbb10069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:54.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.932+0000 7fbb2f7fe700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 0x7fbb10069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:03:54.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:54.967+0000 7fbb360d0700 1 -- 172.21.15.46:0/3370872804 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb3004c710 con 0x7fbb10066f80 2024-01-27T01:03:54.969 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:54.970+0000 7fbc50f561c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:03:55.006 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:55.006+0000 7f7a830b21c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:03:55.052 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:55.052+0000 7fbc50f561c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:03:55.131 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:55.132+0000 7f7a830b21c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:03:55.229 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:55.229+0000 7fbc50f561c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:03:55.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:55.333+0000 7fbb2f7fe700 1 -- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 msgr2=0x7fbb10069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:55.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:55.333+0000 7fbb2f7fe700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 0x7fbb10069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-27T01:03:55.370 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:55.370+0000 7fbc50f561c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:03:55.444 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:55.445+0000 7f7a830b21c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:03:55.656 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:55.656+0000 7f7a830b21c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:03:55.733 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:55.733+0000 7fbc50f561c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:03:55.880 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:55.882+0000 7f7a830b21c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:03:55.951 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:55.952+0000 7f7a830b21c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:03:55.973 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:55.973+0000 7fbc50f561c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:03:56.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:56.134+0000 7fbb2f7fe700 1 -- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 msgr2=0x7fbb10069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:56.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:56.134+0000 7fbb2f7fe700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 0x7fbb10069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-27T01:03:56.233 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:56.233+0000 7fbc50f561c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:03:56.314 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:56.314+0000 7fbc50f561c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:03:56.550 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:56.551+0000 7f7a830b21c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:03:56.624 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:56.625+0000 7f7a830b21c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:03:56.745 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:56.747+0000 7f7a830b21c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:03:57.011 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:57.011+0000 7fbc50f561c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:03:57.096 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:57.096+0000 7fbc50f561c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:03:57.240 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:57.238+0000 7fbc50f561c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:03:57.245 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:57.246+0000 7f7a830b21c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:03:57.395 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:57.396+0000 7f7a830b21c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:03:57.545 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:57.545+0000 7f7a830b21c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:03:57.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:57.736+0000 7fbb2f7fe700 1 -- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 msgr2=0x7fbb10069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:03:57.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:57.736+0000 7fbb2f7fe700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 0x7fbb10069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-27T01:03:57.824 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:57.824+0000 7fbc50f561c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:03:57.996 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:57.997+0000 7fbc50f561c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:03:58.170 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:58.170+0000 7fbc50f561c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:03:58.245 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:58.246+0000 7f7a830b21c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:03:58.463 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:58.464+0000 7f7a830b21c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:03:58.536 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:58.537+0000 7f7a830b21c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:03:58.692 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:58.692+0000 7f7a830b21c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:03:58.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:58.711+0000 7fbb277fe700 1 -- 172.21.15.46:0/3370872804 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 116) v1 ==== 84470+0+0 (secure 0 0 0) 0x7fbb28003040 con 0x7fbb3006b4e0 2024-01-27T01:03:58.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:58.711+0000 7fbb277fe700 1 -- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 msgr2=0x7fbb10069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:03:58.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:58.711+0000 7fbb277fe700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 0x7fbb10069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:58.816 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:58.816+0000 7f79f9990700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:58.816 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:58.816+0000 7f79f9990700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:58.816 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:58.816+0000 7f79f9990700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:58.816 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:58.816+0000 7f79f9990700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:58.816 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:58.816+0000 7f79f9990700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:58.817 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:58.816+0000 7f79fb994700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:58.817 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:58.816+0000 7f79fb994700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:58.817 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:58.816+0000 7f79fb994700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:58.817 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:58.816+0000 7f79fb994700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:58.817 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:03:58.816+0000 7f79fb994700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:03:59.000 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:59.000+0000 7fbc50f561c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:03:59.251 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:59.251+0000 7fbc50f561c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:03:59.335 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:59.335+0000 7fbc50f561c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:03:59.516 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:03:59.516+0000 7fbc50f561c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:03:59.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.715+0000 7fbb277fe700 1 -- 172.21.15.46:0/3370872804 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 117) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fbb28054390 con 0x7fbb3006b4e0 2024-01-27T01:03:59.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.716+0000 7fbb277fe700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 0x7fbb10069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:59.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.716+0000 7fbb277fe700 1 -- 172.21.15.46:0/3370872804 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb3004c710 con 0x7fbb10066f80 2024-01-27T01:03:59.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.716+0000 7fbb2f7fe700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 0x7fbb10069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:59.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.719+0000 7fbb2f7fe700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 0x7fbb10069430 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbb200079e0 tx=0x7fbb20009040 comp rx=0 tx=0).ready entity=mgr.9011 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:59.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.775+0000 7fbb277fe700 1 -- 172.21.15.46:0/3370872804 <== mgr.9011 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbb3004c710 con 0x7fbb10066f80 2024-01-27T01:03:59.775 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path snap_test_dir1 2024-01-27T01:03:59.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.777+0000 7fbb360d0700 1 -- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 msgr2=0x7fbb10069430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:59.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.777+0000 7fbb360d0700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 0x7fbb10069430 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbb200079e0 tx=0x7fbb20009040 comp rx=0 tx=0).stop 2024-01-27T01:03:59.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.777+0000 7fbb360d0700 1 -- 172.21.15.46:0/3370872804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb3006b4e0 msgr2=0x7fbb300ab940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:59.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.777+0000 7fbb360d0700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb3006b4e0 0x7fbb300ab940 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fbb300b2ca0 tx=0x7fbb28019000 comp rx=0 tx=0).stop 2024-01-27T01:03:59.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.777+0000 7fbb360d0700 1 -- 172.21.15.46:0/3370872804 shutdown_connections 2024-01-27T01:03:59.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.777+0000 7fbb360d0700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbb10066f80 0x7fbb10069430 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:59.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.777+0000 7fbb360d0700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb3006b4e0 0x7fbb300ab940 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:59.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.777+0000 7fbb360d0700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb30068b00 0x7fbb30157170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:59.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.777+0000 7fbb360d0700 1 --2- 172.21.15.46:0/3370872804 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb30068ed0 0x7fbb30169770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:59.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.777+0000 7fbb360d0700 1 -- 172.21.15.46:0/3370872804 >> 172.21.15.46:0/3370872804 conn(0x7fbb300ca100 msgr2=0x7fbb300a9f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:59.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.777+0000 7fbb360d0700 1 -- 172.21.15.46:0/3370872804 shutdown_connections 2024-01-27T01:03:59.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.777+0000 7fbb360d0700 1 -- 172.21.15.46:0/3370872804 wait complete. 2024-01-27T01:03:59.793 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-01-27T01:03:59.793 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-01-27T01:03:59.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.933+0000 7fc997e8c700 1 Processor -- start 2024-01-27T01:03:59.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.934+0000 7fc997e8c700 1 -- start start 2024-01-27T01:03:59.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.934+0000 7fc997e8c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc9900ca4b0 0x7fc9900afe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:59.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.935+0000 7fc997e8c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc990158010 con 0x7fc9900ca0e0 2024-01-27T01:03:59.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.935+0000 7fc997e8c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9901582f0 con 0x7fc9900b3a50 2024-01-27T01:03:59.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.935+0000 7fc997e8c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc99006c7c0 con 0x7fc9900ca4b0 2024-01-27T01:03:59.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.935+0000 7fc995c28700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc9900ca0e0 0x7fc99006c0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58554/0 (socket says 172.21.15.46:58554) 2024-01-27T01:03:59.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.935+0000 7fc995c28700 1 -- 172.21.15.46:0/1947216418 learned_addr learned my addr 172.21.15.46:0/1947216418 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:03:59.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.935+0000 7fc995427700 1 --2- 172.21.15.46:0/1947216418 >> v2:172.21.15.106:3300/0 conn(0x7fc9900ca4b0 0x7fc9900afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:59.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.935+0000 7fc994c26700 1 -- 172.21.15.46:0/1947216418 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103463393 0 0) 0x7fc990158010 con 0x7fc9900ca0e0 2024-01-27T01:03:59.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.935+0000 7fc994c26700 1 -- 172.21.15.46:0/1947216418 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc97c003570 con 0x7fc9900ca0e0 2024-01-27T01:03:59.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.935+0000 7fc994c26700 1 -- 172.21.15.46:0/1947216418 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130078852 0 0) 0x7fc9901582f0 con 0x7fc9900b3a50 2024-01-27T01:03:59.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.935+0000 7fc994c26700 1 -- 172.21.15.46:0/1947216418 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc990158010 con 0x7fc9900b3a50 2024-01-27T01:03:59.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.935+0000 7fc995427700 1 -- 172.21.15.46:0/1947216418 >> v1:172.21.15.106:6789/0 conn(0x7fc9900b3a50 legacy=0x7fc99015b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:59.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.935+0000 7fc995427700 1 -- 172.21.15.46:0/1947216418 >> v1:172.21.15.46:6789/0 conn(0x7fc9900ca0e0 legacy=0x7fc99006c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:59.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.935+0000 7fc995427700 1 -- 172.21.15.46:0/1947216418 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9901585d0 con 0x7fc9900ca4b0 2024-01-27T01:03:59.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.935+0000 7fc995427700 1 --2- 172.21.15.46:0/1947216418 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9900ca4b0 0x7fc9900afe40 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fc9800098e0 tx=0x7fc9800091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=359bd5de3d7da9d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:59.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.936+0000 7fc994c26700 1 -- 172.21.15.46:0/1947216418 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980011630 con 0x7fc9900ca4b0 2024-01-27T01:03:59.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.936+0000 7fc994c26700 1 -- 172.21.15.46:0/1947216418 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc980011790 con 0x7fc9900ca4b0 2024-01-27T01:03:59.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.936+0000 7fc994c26700 1 -- 172.21.15.46:0/1947216418 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980011b30 con 0x7fc9900ca4b0 2024-01-27T01:03:59.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.936+0000 7fc997e8c700 1 -- 172.21.15.46:0/1947216418 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9900ca4b0 msgr2=0x7fc9900afe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:59.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.936+0000 7fc997e8c700 1 --2- 172.21.15.46:0/1947216418 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9900ca4b0 0x7fc9900afe40 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fc9800098e0 tx=0x7fc9800091a0 comp rx=0 tx=0).stop 2024-01-27T01:03:59.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.936+0000 7fc997e8c700 1 -- 172.21.15.46:0/1947216418 shutdown_connections 2024-01-27T01:03:59.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.936+0000 7fc997e8c700 1 --2- 172.21.15.46:0/1947216418 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9900ca4b0 0x7fc9900afe40 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:59.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.936+0000 7fc997e8c700 1 -- 172.21.15.46:0/1947216418 >> 172.21.15.46:0/1947216418 conn(0x7fc9900ce530 msgr2=0x7fc9900ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:03:59.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.936+0000 7fc997e8c700 1 -- 172.21.15.46:0/1947216418 shutdown_connections 2024-01-27T01:03:59.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.936+0000 7fc997e8c700 1 -- 172.21.15.46:0/1947216418 wait complete. 2024-01-27T01:03:59.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.937+0000 7fc997e8c700 1 Processor -- start 2024-01-27T01:03:59.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.938+0000 7fc997e8c700 1 -- start start 2024-01-27T01:03:59.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.938+0000 7fc997e8c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9900b3a50 0x7fc9900c7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:59.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.938+0000 7fc997e8c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9900ca0e0 0x7fc9900c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:59.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.938+0000 7fc997e8c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9900ca4b0 0x7fc9900b27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:59.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.938+0000 7fc997e8c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9900c5540 con 0x7fc9900ca0e0 2024-01-27T01:03:59.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.938+0000 7fc997e8c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99015da60 con 0x7fc9900ca4b0 2024-01-27T01:03:59.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.938+0000 7fc997e8c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc99015dd40 con 0x7fc9900b3a50 2024-01-27T01:03:59.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.938+0000 7fc995427700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9900ca0e0 0x7fc9900c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:59.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.938+0000 7fc995427700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9900ca0e0 0x7fc9900c7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33994/0 (socket says 172.21.15.46:33994) 2024-01-27T01:03:59.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.938+0000 7fc995427700 1 -- 172.21.15.46:0/2851587683 learned_addr learned my addr 172.21.15.46:0/2851587683 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:03:59.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.938+0000 7fc995c28700 1 --2- 172.21.15.46:0/2851587683 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9900b3a50 0x7fc9900c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:59.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.939+0000 7fc996429700 1 --2- 172.21.15.46:0/2851587683 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9900ca4b0 0x7fc9900b27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:59.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.939+0000 7fc995427700 1 -- 172.21.15.46:0/2851587683 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9900b3a50 msgr2=0x7fc9900c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:59.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.939+0000 7fc995427700 1 --2- 172.21.15.46:0/2851587683 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9900b3a50 0x7fc9900c7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:59.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.939+0000 7fc995427700 1 -- 172.21.15.46:0/2851587683 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9900ca4b0 msgr2=0x7fc9900b27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:03:59.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.939+0000 7fc995427700 1 --2- 172.21.15.46:0/2851587683 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9900ca4b0 0x7fc9900b27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:03:59.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.939+0000 7fc995427700 1 -- 172.21.15.46:0/2851587683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc980009420 con 0x7fc9900ca0e0 2024-01-27T01:03:59.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.939+0000 7fc995427700 1 --2- 172.21.15.46:0/2851587683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9900ca0e0 0x7fc9900c7ec0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fc99006f390 tx=0x7fc980017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:59.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.939+0000 7fc985ffb700 1 -- 172.21.15.46:0/2851587683 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98001f070 con 0x7fc9900ca0e0 2024-01-27T01:03:59.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.939+0000 7fc985ffb700 1 -- 172.21.15.46:0/2851587683 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc980002f70 con 0x7fc9900ca0e0 2024-01-27T01:03:59.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.939+0000 7fc997e8c700 1 -- 172.21.15.46:0/2851587683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99015e020 con 0x7fc9900ca0e0 2024-01-27T01:03:59.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.939+0000 7fc985ffb700 1 -- 172.21.15.46:0/2851587683 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980011900 con 0x7fc9900ca0e0 2024-01-27T01:03:59.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.939+0000 7fc997e8c700 1 -- 172.21.15.46:0/2851587683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9900c78d0 con 0x7fc9900ca0e0 2024-01-27T01:03:59.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.941+0000 7fc997e8c700 1 -- 172.21.15.46:0/2851587683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc99004c710 con 0x7fc9900ca0e0 2024-01-27T01:03:59.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.941+0000 7fc985ffb700 1 -- 172.21.15.46:0/2851587683 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 117) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fc980023070 con 0x7fc9900ca0e0 2024-01-27T01:03:59.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.942+0000 7fc985ffb700 1 --2- 172.21.15.46:0/2851587683 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc974066dd0 0x7fc974069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:03:59.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.942+0000 7fc985ffb700 1 -- 172.21.15.46:0/2851587683 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6747+0+0 (secure 0 0 0) 0x7fc980085770 con 0x7fc9900ca0e0 2024-01-27T01:03:59.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.942+0000 7fc995c28700 1 --2- 172.21.15.46:0/2851587683 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc974066dd0 0x7fc974069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:03:59.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.942+0000 7fc995c28700 1 --2- 172.21.15.46:0/2851587683 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc974066dd0 0x7fc974069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc99015b330 tx=0x7fc98c009040 comp rx=0 tx=0).ready entity=mgr.9011 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:03:59.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:03:59.946+0000 7fc985ffb700 1 -- 172.21.15.46:0/2851587683 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc980052880 con 0x7fc9900ca0e0 2024-01-27T01:04:00.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.180+0000 7fc997e8c700 1 -- 172.21.15.46:0/2851587683 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc99006c930 con 0x7fc974066dd0 2024-01-27T01:04:00.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.231+0000 7fc985ffb700 1 -- 172.21.15.46:0/2851587683 <== mgr.9011 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7fc99006c930 con 0x7fc974066dd0 2024-01-27T01:04:00.231 INFO:teuthology.orchestra.run.smithi046.stdout:Retention added to path snap_test_dir1 2024-01-27T01:04:00.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.233+0000 7fc997e8c700 1 -- 172.21.15.46:0/2851587683 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc974066dd0 msgr2=0x7fc974069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:04:00.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.233+0000 7fc997e8c700 1 --2- 172.21.15.46:0/2851587683 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc974066dd0 0x7fc974069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc99015b330 tx=0x7fc98c009040 comp rx=0 tx=0).stop 2024-01-27T01:04:00.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.233+0000 7fc997e8c700 1 -- 172.21.15.46:0/2851587683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9900ca0e0 msgr2=0x7fc9900c7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:04:00.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.233+0000 7fc997e8c700 1 --2- 172.21.15.46:0/2851587683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9900ca0e0 0x7fc9900c7ec0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fc99006f390 tx=0x7fc980017000 comp rx=0 tx=0).stop 2024-01-27T01:04:00.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.233+0000 7fc997e8c700 1 -- 172.21.15.46:0/2851587683 shutdown_connections 2024-01-27T01:04:00.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.233+0000 7fc997e8c700 1 --2- 172.21.15.46:0/2851587683 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc974066dd0 0x7fc974069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:04:00.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.233+0000 7fc997e8c700 1 --2- 172.21.15.46:0/2851587683 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9900ca4b0 0x7fc9900b27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:04:00.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.233+0000 7fc997e8c700 1 --2- 172.21.15.46:0/2851587683 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9900b3a50 0x7fc9900c7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:04:00.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.233+0000 7fc997e8c700 1 --2- 172.21.15.46:0/2851587683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9900ca0e0 0x7fc9900c7ec0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:04:00.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.233+0000 7fc997e8c700 1 -- 172.21.15.46:0/2851587683 >> 172.21.15.46:0/2851587683 conn(0x7fc9900ce530 msgr2=0x7fc9900ae680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:04:00.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.233+0000 7fc997e8c700 1 -- 172.21.15.46:0/2851587683 shutdown_connections 2024-01-27T01:04:00.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.234+0000 7fc997e8c700 1 -- 172.21.15.46:0/2851587683 wait complete. 2024-01-27T01:04:00.248 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-01-27T01:04:00.248 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-01-27-01_05_00 in ~60s... 2024-01-27T01:04:00.248 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-01-27T01:04:00.249 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-01-27T01:04:00.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.385+0000 7fbd0d105700 1 Processor -- start 2024-01-27T01:04:00.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.385+0000 7fbd0d105700 1 -- start start 2024-01-27T01:04:00.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.386+0000 7fbd0d105700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd080c8c50 0x7fbd080c7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:04:00.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.386+0000 7fbd0d105700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbd080c76a0 0x7fbd080af990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:04:00.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.386+0000 7fbd0d105700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd080c87c0 con 0x7fbd08068940 2024-01-27T01:04:00.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.386+0000 7fbd0d105700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd080c7bb0 con 0x7fbd080c8c50 2024-01-27T01:04:00.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.386+0000 7fbd0d105700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd080c7e90 con 0x7fbd080c76a0 2024-01-27T01:04:00.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.386+0000 7fbd0659c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd080c8c50 0x7fbd080c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:04:00.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.386+0000 7fbd0659c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd080c8c50 0x7fbd080c7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42334/0 (socket says 172.21.15.46:42334) 2024-01-27T01:04:00.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.386+0000 7fbd0659c700 1 -- 172.21.15.46:0/3998859324 learned_addr learned my addr 172.21.15.46:0/3998859324 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:04:00.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.386+0000 7fbd0759e700 1 --2- 172.21.15.46:0/3998859324 >> v2:172.21.15.106:3300/0 conn(0x7fbd080c76a0 0x7fbd080af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:04:00.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.387+0000 7fbd0659c700 1 -- 172.21.15.46:0/3998859324 >> v2:172.21.15.106:3300/0 conn(0x7fbd080c76a0 msgr2=0x7fbd080af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:04:00.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.387+0000 7fbd0659c700 1 --2- 172.21.15.46:0/3998859324 >> v2:172.21.15.106:3300/0 conn(0x7fbd080c76a0 0x7fbd080af990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:04:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.387+0000 7fbd0659c700 1 -- 172.21.15.46:0/3998859324 >> v1:172.21.15.46:6789/0 conn(0x7fbd08068940 legacy=0x7fbd08068d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:04:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.387+0000 7fbd0659c700 1 -- 172.21.15.46:0/3998859324 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd080c8170 con 0x7fbd080c8c50 2024-01-27T01:04:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.387+0000 7fbd0659c700 1 --2- 172.21.15.46:0/3998859324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd080c8c50 0x7fbd080c7160 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fbd08069420 tx=0x7fbcfc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f39e1d78d4cd3c01 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:04:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.387+0000 7fbd05d9b700 1 -- 172.21.15.46:0/3998859324 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc00b860 con 0x7fbd080c8c50 2024-01-27T01:04:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.387+0000 7fbd05d9b700 1 -- 172.21.15.46:0/3998859324 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcfc00b9c0 con 0x7fbd080c8c50 2024-01-27T01:04:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.387+0000 7fbd05d9b700 1 -- 172.21.15.46:0/3998859324 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc00bd60 con 0x7fbd080c8c50 2024-01-27T01:04:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.387+0000 7fbd0d105700 1 -- 172.21.15.46:0/3998859324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd080c8c50 msgr2=0x7fbd080c7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:04:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.387+0000 7fbd0d105700 1 --2- 172.21.15.46:0/3998859324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd080c8c50 0x7fbd080c7160 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fbd08069420 tx=0x7fbcfc009250 comp rx=0 tx=0).stop 2024-01-27T01:04:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.388+0000 7fbd0d105700 1 -- 172.21.15.46:0/3998859324 shutdown_connections 2024-01-27T01:04:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.388+0000 7fbd0d105700 1 --2- 172.21.15.46:0/3998859324 >> v2:172.21.15.106:3300/0 conn(0x7fbd080c76a0 0x7fbd080af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:04:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.388+0000 7fbd0d105700 1 --2- 172.21.15.46:0/3998859324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd080c8c50 0x7fbd080c7160 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:04:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.388+0000 7fbd0d105700 1 -- 172.21.15.46:0/3998859324 >> 172.21.15.46:0/3998859324 conn(0x7fbd080ca100 msgr2=0x7fbd0806e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:04:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.388+0000 7fbd0d105700 1 -- 172.21.15.46:0/3998859324 shutdown_connections 2024-01-27T01:04:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.388+0000 7fbd0d105700 1 -- 172.21.15.46:0/3998859324 wait complete. 2024-01-27T01:04:00.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.389+0000 7fbd0d105700 1 Processor -- start 2024-01-27T01:04:00.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.389+0000 7fbd0d105700 1 -- start start 2024-01-27T01:04:00.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.390+0000 7fbd0d105700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd08068940 0x7fbd0815a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:04:00.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.390+0000 7fbd0d105700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd080c76a0 0x7fbd0816aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:04:00.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.390+0000 7fbd0d105700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd080c8c50 0x7fbd0816b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:04:00.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.390+0000 7fbd0d105700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd080c3440 con 0x7fbd080c76a0 2024-01-27T01:04:00.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.390+0000 7fbd0d105700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd080c2830 con 0x7fbd080c8c50 2024-01-27T01:04:00.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.390+0000 7fbd0d105700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd080c2b10 con 0x7fbd08068940 2024-01-27T01:04:00.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.390+0000 7fbd0659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd080c76a0 0x7fbd0816aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:04:00.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.390+0000 7fbd0659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd080c76a0 0x7fbd0816aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42350/0 (socket says 172.21.15.46:42350) 2024-01-27T01:04:00.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.390+0000 7fbd0659c700 1 -- 172.21.15.46:0/2682363433 learned_addr learned my addr 172.21.15.46:0/2682363433 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:04:00.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.390+0000 7fbd0759e700 1 --2- 172.21.15.46:0/2682363433 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd080c8c50 0x7fbd0816b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:04:00.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.390+0000 7fbd06d9d700 1 --2- 172.21.15.46:0/2682363433 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd08068940 0x7fbd0815a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:04:00.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.390+0000 7fbd0659c700 1 -- 172.21.15.46:0/2682363433 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd08068940 msgr2=0x7fbd0815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:04:00.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.390+0000 7fbd0659c700 1 --2- 172.21.15.46:0/2682363433 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd08068940 0x7fbd0815a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:04:00.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.390+0000 7fbd0659c700 1 -- 172.21.15.46:0/2682363433 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd080c8c50 msgr2=0x7fbd0816b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:04:00.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.390+0000 7fbd0659c700 1 --2- 172.21.15.46:0/2682363433 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd080c8c50 0x7fbd0816b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:04:00.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.390+0000 7fbd0659c700 1 -- 172.21.15.46:0/2682363433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcfc0094f0 con 0x7fbd080c76a0 2024-01-27T01:04:00.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.391+0000 7fbd0659c700 1 --2- 172.21.15.46:0/2682363433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd080c76a0 0x7fbd0816aee0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fbd08069d70 tx=0x7fbcfc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:04:00.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.391+0000 7fbceeffd700 1 -- 172.21.15.46:0/2682363433 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc024440 con 0x7fbd080c76a0 2024-01-27T01:04:00.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.391+0000 7fbceeffd700 1 -- 172.21.15.46:0/2682363433 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcfc0245a0 con 0x7fbd080c76a0 2024-01-27T01:04:00.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.391+0000 7fbd0d105700 1 -- 172.21.15.46:0/2682363433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd080c2df0 con 0x7fbd080c76a0 2024-01-27T01:04:00.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.391+0000 7fbceeffd700 1 -- 172.21.15.46:0/2682363433 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc024960 con 0x7fbd080c76a0 2024-01-27T01:04:00.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.391+0000 7fbd0d105700 1 -- 172.21.15.46:0/2682363433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd080c1b00 con 0x7fbd080c76a0 2024-01-27T01:04:00.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.392+0000 7fbd0d105700 1 -- 172.21.15.46:0/2682363433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd0804c710 con 0x7fbd080c76a0 2024-01-27T01:04:00.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.393+0000 7fbceeffd700 1 -- 172.21.15.46:0/2682363433 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 117) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fbcfc01a420 con 0x7fbd080c76a0 2024-01-27T01:04:00.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.393+0000 7fbceeffd700 1 --2- 172.21.15.46:0/2682363433 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbcf4066dd0 0x7fbcf4069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:04:00.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.393+0000 7fbceeffd700 1 -- 172.21.15.46:0/2682363433 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6747+0+0 (secure 0 0 0) 0x7fbcfc086cc0 con 0x7fbd080c76a0 2024-01-27T01:04:00.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.393+0000 7fbd06d9d700 1 --2- 172.21.15.46:0/2682363433 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbcf4066dd0 0x7fbcf4069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:04:00.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.394+0000 7fbd06d9d700 1 --2- 172.21.15.46:0/2682363433 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbcf4066dd0 0x7fbcf4069280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbd080c42c0 tx=0x7fbcf8009040 comp rx=0 tx=0).ready entity=mgr.9011 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:04:00.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.397+0000 7fbceeffd700 1 -- 172.21.15.46:0/2682363433 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbcfc053d80 con 0x7fbd080c76a0 2024-01-27T01:04:00.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.633+0000 7fbd0d105700 1 -- 172.21.15.46:0/2682363433 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbd080c35f0 con 0x7fbcf4066dd0 2024-01-27T01:04:00.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.644+0000 7fbceeffd700 1 -- 172.21.15.46:0/2682363433 <== mgr.9011 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7fbcf80013a0 con 0x7fbcf4066dd0 2024-01-27T01:04:00.644 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:04:00.644 INFO:teuthology.orchestra.run.smithi046.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-01-27T01:04:00.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.646+0000 7fbd0d105700 1 -- 172.21.15.46:0/2682363433 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbcf4066dd0 msgr2=0x7fbcf4069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:04:00.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.646+0000 7fbd0d105700 1 --2- 172.21.15.46:0/2682363433 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbcf4066dd0 0x7fbcf4069280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbd080c42c0 tx=0x7fbcf8009040 comp rx=0 tx=0).stop 2024-01-27T01:04:00.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.646+0000 7fbd0d105700 1 -- 172.21.15.46:0/2682363433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd080c76a0 msgr2=0x7fbd0816aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:04:00.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.646+0000 7fbd0d105700 1 --2- 172.21.15.46:0/2682363433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd080c76a0 0x7fbd0816aee0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fbd08069d70 tx=0x7fbcfc017000 comp rx=0 tx=0).stop 2024-01-27T01:04:00.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.646+0000 7fbd0d105700 1 -- 172.21.15.46:0/2682363433 shutdown_connections 2024-01-27T01:04:00.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.646+0000 7fbd0d105700 1 --2- 172.21.15.46:0/2682363433 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbcf4066dd0 0x7fbcf4069280 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:04:00.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.646+0000 7fbd0d105700 1 --2- 172.21.15.46:0/2682363433 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd080c8c50 0x7fbd0816b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:04:00.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.646+0000 7fbd0d105700 1 --2- 172.21.15.46:0/2682363433 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbd08068940 0x7fbd0815a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:04:00.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.646+0000 7fbd0d105700 1 --2- 172.21.15.46:0/2682363433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd080c76a0 0x7fbd0816aee0 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:04:00.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.646+0000 7fbd0d105700 1 -- 172.21.15.46:0/2682363433 >> 172.21.15.46:0/2682363433 conn(0x7fbd080ca100 msgr2=0x7fbd080cb580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:04:00.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.646+0000 7fbd0d105700 1 -- 172.21.15.46:0/2682363433 shutdown_connections 2024-01-27T01:04:00.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:04:00.646+0000 7fbd0d105700 1 -- 172.21.15.46:0/2682363433 wait complete. 2024-01-27T01:04:00.660 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-01-27T01:04:00.660 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-01-27T01:04:00.660 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:00.661 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:00.684 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:00.684 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:00.684 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:01.685 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:01.686 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:01.714 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:01.714 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:01.714 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:02.716 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:02.716 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:02.743 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:02.743 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:02.743 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:03.744 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:03.744 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:03.771 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:03.771 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:03.771 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:04.772 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:04.772 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:04.799 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:04.799 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:04.799 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:05.799 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:05.800 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:05.827 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:05.828 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:05.828 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:06.828 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:06.829 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:06.857 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:06.857 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:06.857 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:07.858 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:07.858 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:07.886 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:07.886 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:07.886 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:08.887 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:08.887 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:08.915 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:08.915 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:08.915 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:09.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:09.916 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:09.943 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:09.943 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:09.944 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:10.944 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:10.944 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:10.972 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:10.972 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:10.972 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:11.973 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:11.973 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:12.000 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:12.000 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:12.000 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:13.001 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:13.001 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:13.030 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:13.030 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:13.030 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:13.947 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:04:13.950 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:04:14.031 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:14.031 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:14.058 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:14.058 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:14.058 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:15.059 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:15.059 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:15.086 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:15.086 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:15.086 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:16.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:16.087 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:16.115 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:16.115 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:16.115 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:17.115 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:17.116 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:17.143 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:17.143 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:17.143 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:18.144 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:18.145 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:18.172 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:18.172 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:18.172 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:19.173 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:19.173 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:19.201 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:19.201 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:19.201 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:20.202 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:20.202 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:20.229 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:20.230 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:20.230 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:21.230 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:21.231 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:21.258 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:21.258 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:21.259 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:22.259 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:22.260 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:22.287 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:22.287 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:22.287 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:23.288 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:23.288 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:23.316 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:23.317 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:23.317 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:24.317 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:24.318 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:24.345 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:24.345 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:24.345 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:25.346 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:25.346 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:25.374 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:25.374 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:25.374 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:26.375 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:26.375 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:26.402 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:26.403 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:26.403 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:27.403 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:27.404 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:27.431 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:27.431 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:27.431 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:28.432 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:28.432 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:28.460 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:28.460 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:28.460 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:29.460 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:29.461 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:29.488 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:29.488 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:29.488 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:30.488 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:30.489 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:30.516 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:30.516 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:30.516 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:31.517 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:31.518 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:31.545 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:31.545 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:31.545 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:32.546 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:32.546 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:32.573 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:32.573 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:32.573 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:33.574 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:33.574 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:33.602 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:33.602 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:33.602 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:34.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:34.603 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:34.630 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:34.631 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:34.631 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:35.631 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:35.632 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:35.659 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:35.659 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:35.659 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:36.660 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:36.661 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:36.688 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:36.688 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:36.688 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:37.689 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:37.689 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:37.716 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:37.716 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:37.717 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:38.717 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:38.717 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:38.745 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:38.745 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:38.745 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:39.746 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:39.746 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:39.774 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:39.774 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:39.774 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:40.775 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:40.775 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:40.803 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:40.803 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:40.804 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:41.805 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:41.805 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:41.832 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:41.832 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:41.832 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:42.833 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:42.833 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:42.861 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:42.861 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:42.861 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:43.862 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:43.863 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:43.890 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:43.890 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:43.890 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:44.111 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:04:44.114 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:04:44.891 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:44.892 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:44.919 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:44.919 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:44.919 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:45.920 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:45.920 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:45.948 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:45.948 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:45.948 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:46.949 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:46.949 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:46.977 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:46.977 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:46.977 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:47.978 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:47.978 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:48.005 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:48.005 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:48.005 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:49.006 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:49.007 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:49.034 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:49.034 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:49.034 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:50.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:50.036 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:50.064 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:50.064 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:50.064 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:51.065 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:51.065 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:51.092 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:51.093 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:51.093 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:52.094 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:52.094 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:52.121 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:52.121 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:52.121 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:53.122 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:53.123 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:53.150 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:53.150 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:53.151 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:54.152 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:54.152 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:54.180 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:54.180 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:54.180 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:55.181 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:55.181 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:55.208 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:55.209 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:55.209 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:56.209 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:56.209 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:56.237 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:56.237 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:56.237 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:57.238 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:57.239 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:57.266 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:57.266 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:57.266 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:58.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:58.267 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:58.295 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:58.295 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:58.295 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:04:59.295 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:04:59.296 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:04:59.323 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-27T01:04:59.323 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:04:59.323 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:00.242 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:05:00.242+0000 7f79eb974700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:05:00.242 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:05:00.242+0000 7f79eb974700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:05:00.243 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:05:00.242+0000 7f79eb974700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:05:00.243 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:05:00.242+0000 7f79eb974700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:05:00.243 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:05:00.242+0000 7f79eb974700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:05:00.324 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:00.325 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:00.353 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:00.353 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:00.354 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:00.354 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:00.354 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-01-27-01_05_00_UTC'] 2024-01-27T01:05:00.355 DEBUG:tasks.cephfs.test_snap_schedules:exec=1706317440.2487593, now = 1706317500.354986, timo = 60 2024-01-27T01:05:01.356 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2024-01-27-01_05_00 in ~120s... 2024-01-27T01:05:01.356 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:01.356 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:01.383 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:01.384 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:01.384 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:01.384 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:02.385 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:02.385 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:02.412 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:02.413 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:02.413 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:02.413 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:03.414 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:03.414 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:03.441 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:03.441 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:03.441 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:03.441 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:04.442 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:04.443 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:04.469 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:04.470 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:04.470 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:04.470 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:05.471 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:05.471 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:05.498 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:05.499 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:05.499 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:05.499 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:06.499 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:06.500 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:06.526 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:06.527 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:06.527 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:06.527 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:07.528 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:07.528 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:07.556 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:07.557 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:07.557 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:07.557 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:08.558 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:08.558 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:08.586 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:08.586 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:08.586 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:08.586 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:09.587 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:09.588 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:09.616 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:09.616 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:09.616 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:09.616 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:10.617 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:10.617 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:10.645 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:10.646 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:10.646 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:10.646 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:11.646 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:11.646 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:11.675 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:11.675 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:11.675 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:11.675 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:12.676 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:12.676 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:12.705 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:12.705 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:12.705 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:12.705 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:13.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:13.706 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:13.733 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:13.734 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:13.734 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:13.734 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:14.276 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:05:14.279 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:05:14.735 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:14.735 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:14.763 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:14.763 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:14.763 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:14.763 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:15.764 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:15.764 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:15.792 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:15.793 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:15.793 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:15.793 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:16.794 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:16.794 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:16.821 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:16.821 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:16.822 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:16.822 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:17.823 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:17.823 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:17.850 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:17.851 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:17.851 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:17.851 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:18.852 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:18.853 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:18.880 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:18.880 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:18.880 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:18.880 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:19.881 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:19.881 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:19.907 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:19.908 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:19.908 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:19.908 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:20.909 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:20.909 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:20.937 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:20.937 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:20.938 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:20.938 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:21.939 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:21.939 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:21.966 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:21.966 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:21.967 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:21.967 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:22.968 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:22.968 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:22.995 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:22.995 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:22.995 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:22.995 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:23.996 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:23.996 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:24.022 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:24.023 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:24.023 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:24.023 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:25.025 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:25.025 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:25.052 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:25.052 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:25.052 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:25.053 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:26.053 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:26.053 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:26.081 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:26.081 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:26.082 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:26.082 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:27.082 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:27.082 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:27.109 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:27.109 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:27.110 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:27.110 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:28.110 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:28.110 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:28.137 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:28.138 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:28.138 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:28.138 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:29.138 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:29.139 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:29.165 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:29.166 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:29.166 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:29.166 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:30.167 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:30.167 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:30.194 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:30.195 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:30.195 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:30.195 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:31.196 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:31.197 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:31.223 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:31.224 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:31.224 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:31.224 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:32.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:32.225 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:32.252 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:32.253 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:32.253 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:32.253 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:33.254 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:33.254 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:33.282 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:33.282 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:33.283 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:33.283 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:34.284 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:34.284 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:34.310 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:34.311 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:34.312 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:34.312 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:35.312 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:35.313 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:35.340 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:35.341 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:35.341 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:35.341 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:36.342 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:36.342 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:36.368 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:36.369 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:36.370 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:36.370 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:37.370 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:37.371 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:37.397 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:37.398 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:37.398 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:37.398 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:38.398 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:38.399 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:38.425 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:38.426 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:38.426 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:38.426 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:39.427 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:39.427 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:39.454 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:39.455 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:39.455 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:39.455 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:40.456 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:40.456 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:40.484 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:40.484 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:40.484 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:40.484 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:41.485 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:41.485 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:41.512 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:41.512 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:41.512 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:41.512 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:42.513 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:42.513 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:42.539 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:42.540 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:42.540 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:42.540 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:43.542 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:43.542 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:43.568 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:43.569 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:43.569 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:43.570 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:44.432 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:05:44.435 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:05:44.571 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:44.571 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:44.600 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:44.600 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:44.600 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:44.600 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:45.601 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:45.601 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:45.629 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:45.629 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:45.629 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:45.629 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:46.630 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:46.630 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:46.660 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:46.660 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:46.660 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:46.660 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:47.661 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:47.662 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:47.689 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:47.689 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:47.689 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:47.689 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:48.690 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:48.690 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:48.717 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:48.717 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:48.717 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:48.717 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:49.718 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:49.718 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:49.745 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:49.746 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:49.746 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:49.746 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:50.747 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:50.747 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:50.774 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:50.775 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:50.775 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:50.775 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:51.775 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:51.776 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:51.803 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:51.803 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:51.803 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:51.803 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:52.804 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:52.804 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:52.831 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:52.831 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:52.831 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:52.831 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:53.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:53.832 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:53.859 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:53.860 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:53.860 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:53.860 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:54.860 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:54.861 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:54.888 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:54.888 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:54.888 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:54.889 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:55.890 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:55.890 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:55.917 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:55.918 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:55.918 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:55.918 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:56.919 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:56.920 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:56.947 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:56.948 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:56.948 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:56.948 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:57.949 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:57.949 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:57.977 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:57.977 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:57.977 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:57.978 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:05:58.979 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:05:58.979 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:05:59.006 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:05:59.006 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:05:59.006 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:05:59.006 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:06:00.007 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:06:00.007 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:06:00.035 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_05_00_UTC 2024-01-27T01:06:00.035 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:06:00.035 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-27T01:06:00.036 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-27T01:06:01.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:06:01.036 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:06:01.064 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_06_00_UTC 2024-01-27T01:06:01.064 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-27-01_06_00_UTC'} 2024-01-27T01:06:01.064 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-01-27-01_06_00_UTC'} 2024-01-27T01:06:01.064 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2024-01-27-01_05_00_UTC'} 2024-01-27T01:06:01.064 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2024-01-27-01_05_00_UTC'] 2024-01-27T01:06:01.065 DEBUG:tasks.cephfs.test_snap_schedules:exec=1706317440.2487593, now = 1706317561.0650005, timo = 120 2024-01-27T01:06:02.066 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-01-27T01:06:02.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.209+0000 7f35c471f700 1 Processor -- start 2024-01-27T01:06:02.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.209+0000 7f35c471f700 1 -- start start 2024-01-27T01:06:02.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.210+0000 7f35c471f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35bc06ebe0 0x7f35bc067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:02.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.210+0000 7f35c471f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f35bc1597e0 0x7f35bc159c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:02.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.210+0000 7f35c471f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc06fdb0 con 0x7f35bc067af0 2024-01-27T01:06:02.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.210+0000 7f35c471f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35bc06f1a0 con 0x7f35bc1597e0 2024-01-27T01:06:02.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.210+0000 7f35c471f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc06f480 con 0x7f35bc06ebe0 2024-01-27T01:06:02.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.210+0000 7f35c2cbc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f35bc1597e0 0x7f35bc159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:02.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.210+0000 7f35c2cbc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f35bc1597e0 0x7f35bc159c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44534/0 (socket says 172.21.15.46:44534) 2024-01-27T01:06:02.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.210+0000 7f35c2cbc700 1 -- 172.21.15.46:0/281036402 learned_addr learned my addr 172.21.15.46:0/281036402 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:02.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.210+0000 7f35c1cba700 1 --2- 172.21.15.46:0/281036402 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35bc06ebe0 0x7f35bc067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:02.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.210+0000 7f35c2cbc700 1 -- 172.21.15.46:0/281036402 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35bc06ebe0 msgr2=0x7f35bc067ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:02.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.210+0000 7f35c2cbc700 1 --2- 172.21.15.46:0/281036402 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35bc06ebe0 0x7f35bc067ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:02.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.210+0000 7f35c2cbc700 1 -- 172.21.15.46:0/281036402 >> v1:172.21.15.106:6789/0 conn(0x7f35bc067af0 legacy=0x7f35bc0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:02.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.210+0000 7f35c2cbc700 1 -- 172.21.15.46:0/281036402 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc06f760 con 0x7f35bc1597e0 2024-01-27T01:06:02.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.211+0000 7f35c2cbc700 1 --2- 172.21.15.46:0/281036402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35bc1597e0 0x7f35bc159c20 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009f20 tx=0x7f35b4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80bf0cc87ac6596a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:02.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.211+0000 7f35c14b9700 1 -- 172.21.15.46:0/281036402 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b400e070 con 0x7f35bc1597e0 2024-01-27T01:06:02.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.211+0000 7f35c14b9700 1 -- 172.21.15.46:0/281036402 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b400ba80 con 0x7f35bc1597e0 2024-01-27T01:06:02.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.211+0000 7f35c14b9700 1 -- 172.21.15.46:0/281036402 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b40092e0 con 0x7f35bc1597e0 2024-01-27T01:06:02.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.211+0000 7f35c471f700 1 -- 172.21.15.46:0/281036402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35bc1597e0 msgr2=0x7f35bc159c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:02.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.211+0000 7f35c471f700 1 --2- 172.21.15.46:0/281036402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35bc1597e0 0x7f35bc159c20 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009f20 tx=0x7f35b4009840 comp rx=0 tx=0).stop 2024-01-27T01:06:02.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.211+0000 7f35c471f700 1 -- 172.21.15.46:0/281036402 shutdown_connections 2024-01-27T01:06:02.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.211+0000 7f35c471f700 1 --2- 172.21.15.46:0/281036402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35bc1597e0 0x7f35bc159c20 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:02.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.211+0000 7f35c471f700 1 --2- 172.21.15.46:0/281036402 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35bc06ebe0 0x7f35bc067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:02.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.211+0000 7f35c471f700 1 -- 172.21.15.46:0/281036402 >> 172.21.15.46:0/281036402 conn(0x7f35bc0cdf80 msgr2=0x7f35bc06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:02.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.211+0000 7f35c471f700 1 -- 172.21.15.46:0/281036402 shutdown_connections 2024-01-27T01:06:02.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.211+0000 7f35c471f700 1 -- 172.21.15.46:0/281036402 wait complete. 2024-01-27T01:06:02.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.213+0000 7f35c471f700 1 Processor -- start 2024-01-27T01:06:02.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.213+0000 7f35c471f700 1 -- start start 2024-01-27T01:06:02.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.213+0000 7f35c471f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35bc067af0 0x7f35bc16a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:02.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35c471f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35bc06ebe0 0x7f35bc157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:02.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35c471f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35bc1597e0 0x7f35bc158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:02.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35c471f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc15ba70 con 0x7f35bc067af0 2024-01-27T01:06:02.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35c471f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc16a650 con 0x7f35bc1597e0 2024-01-27T01:06:02.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35c471f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc15a0e0 con 0x7f35bc06ebe0 2024-01-27T01:06:02.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35c24bb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35bc067af0 0x7f35bc16a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:02.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35c24bb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35bc067af0 0x7f35bc16a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44540/0 (socket says 172.21.15.46:44540) 2024-01-27T01:06:02.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35c24bb700 1 -- 172.21.15.46:0/2847858723 learned_addr learned my addr 172.21.15.46:0/2847858723 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:02.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35c2cbc700 1 --2- 172.21.15.46:0/2847858723 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35bc1597e0 0x7f35bc158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:02.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35c1cba700 1 --2- 172.21.15.46:0/2847858723 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35bc06ebe0 0x7f35bc157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:02.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35c24bb700 1 -- 172.21.15.46:0/2847858723 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35bc06ebe0 msgr2=0x7f35bc157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:02.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35c24bb700 1 --2- 172.21.15.46:0/2847858723 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35bc06ebe0 0x7f35bc157320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:02.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35c24bb700 1 -- 172.21.15.46:0/2847858723 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35bc1597e0 msgr2=0x7f35bc158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:02.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35c24bb700 1 --2- 172.21.15.46:0/2847858723 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35bc1597e0 0x7f35bc158540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:02.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35c24bb700 1 -- 172.21.15.46:0/2847858723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b4009ae0 con 0x7f35bc067af0 2024-01-27T01:06:02.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35c24bb700 1 --2- 172.21.15.46:0/2847858723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35bc067af0 0x7f35bc16a110 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f35b800c4e0 tx=0x7f35b800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:02.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35b27fc700 1 -- 172.21.15.46:0/2847858723 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b8011590 con 0x7f35bc067af0 2024-01-27T01:06:02.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35b27fc700 1 -- 172.21.15.46:0/2847858723 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b8011710 con 0x7f35bc067af0 2024-01-27T01:06:02.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35c471f700 1 -- 172.21.15.46:0/2847858723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35bc15a3c0 con 0x7f35bc067af0 2024-01-27T01:06:02.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.214+0000 7f35b27fc700 1 -- 172.21.15.46:0/2847858723 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b8011ad0 con 0x7f35bc067af0 2024-01-27T01:06:02.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.215+0000 7f35c471f700 1 -- 172.21.15.46:0/2847858723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35bc15a980 con 0x7f35bc067af0 2024-01-27T01:06:02.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.216+0000 7f35c471f700 1 -- 172.21.15.46:0/2847858723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35bc04c710 con 0x7f35bc067af0 2024-01-27T01:06:02.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.216+0000 7f35b27fc700 1 -- 172.21.15.46:0/2847858723 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 121) v1 ==== 84873+0+0 (secure 0 0 0) 0x7f35b801b3b0 con 0x7f35bc067af0 2024-01-27T01:06:02.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.216+0000 7f35b27fc700 1 --2- 172.21.15.46:0/2847858723 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f35a00670a0 0x7f35a0069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:02.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.217+0000 7f35b27fc700 1 -- 172.21.15.46:0/2847858723 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6834+0+0 (secure 0 0 0) 0x7f35b8083f80 con 0x7f35bc067af0 2024-01-27T01:06:02.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.217+0000 7f35c1cba700 1 --2- 172.21.15.46:0/2847858723 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f35a00670a0 0x7f35a0069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:02.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.217+0000 7f35c1cba700 1 --2- 172.21.15.46:0/2847858723 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f35a00670a0 0x7f35a0069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f35bc152a80 tx=0x7f35ac008040 comp rx=0 tx=0).ready entity=mgr.9011 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:02.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.221+0000 7f35b27fc700 1 -- 172.21.15.46:0/2847858723 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f35b8050c70 con 0x7f35bc067af0 2024-01-27T01:06:02.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.453+0000 7f35c471f700 1 -- 172.21.15.46:0/2847858723 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f35bc068690 con 0x7f35a00670a0 2024-01-27T01:06:02.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.518+0000 7f35b27fc700 1 -- 172.21.15.46:0/2847858723 <== mgr.9011 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f35bc068690 con 0x7f35a00670a0 2024-01-27T01:06:02.518 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path snap_test_dir1 2024-01-27T01:06:02.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.520+0000 7f35c471f700 1 -- 172.21.15.46:0/2847858723 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f35a00670a0 msgr2=0x7f35a0069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:02.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.520+0000 7f35c471f700 1 --2- 172.21.15.46:0/2847858723 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f35a00670a0 0x7f35a0069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f35bc152a80 tx=0x7f35ac008040 comp rx=0 tx=0).stop 2024-01-27T01:06:02.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.520+0000 7f35c471f700 1 -- 172.21.15.46:0/2847858723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35bc067af0 msgr2=0x7f35bc16a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:02.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.520+0000 7f35c471f700 1 --2- 172.21.15.46:0/2847858723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35bc067af0 0x7f35bc16a110 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f35b800c4e0 tx=0x7f35b800c7f0 comp rx=0 tx=0).stop 2024-01-27T01:06:02.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.521+0000 7f35c471f700 1 -- 172.21.15.46:0/2847858723 shutdown_connections 2024-01-27T01:06:02.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.521+0000 7f35c471f700 1 --2- 172.21.15.46:0/2847858723 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f35a00670a0 0x7f35a0069550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:02.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.521+0000 7f35c471f700 1 --2- 172.21.15.46:0/2847858723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35bc067af0 0x7f35bc16a110 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:02.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.521+0000 7f35c471f700 1 --2- 172.21.15.46:0/2847858723 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35bc1597e0 0x7f35bc158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:02.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.521+0000 7f35c471f700 1 --2- 172.21.15.46:0/2847858723 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35bc06ebe0 0x7f35bc157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:02.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.521+0000 7f35c471f700 1 -- 172.21.15.46:0/2847858723 >> 172.21.15.46:0/2847858723 conn(0x7f35bc0cdf80 msgr2=0x7f35bc0cc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:02.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.521+0000 7f35c471f700 1 -- 172.21.15.46:0/2847858723 shutdown_connections 2024-01-27T01:06:02.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.521+0000 7f35c471f700 1 -- 172.21.15.46:0/2847858723 wait complete. 2024-01-27T01:06:02.535 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-01-27T01:06:02.535 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:06:02.535 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:06:02.559 INFO:teuthology.orchestra.run.smithi046.stdout:scheduled-2024-01-27-01_06_00_UTC 2024-01-27T01:06:02.559 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-27-01_06_00_UTC 2024-01-27T01:06:02.559 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:06:02.560 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-27-01_06_00_UTC) 2024-01-27T01:06:02.629 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:06:02.629 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-01-27T01:06:02.695 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-27T01:06:02.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.878+0000 7f533f336700 1 Processor -- start 2024-01-27T01:06:02.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.879+0000 7f533f336700 1 -- start start 2024-01-27T01:06:02.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.880+0000 7f533f336700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5338159510 0x7f53381598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:02.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.880+0000 7f533f336700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f53380ca690 0x7f53380cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:02.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.880+0000 7f533f336700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5338069260 con 0x7f53380ca220 2024-01-27T01:06:02.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.880+0000 7f533f336700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f533815d670 con 0x7f53380ca690 2024-01-27T01:06:02.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.880+0000 7f533f336700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53380c9750 con 0x7f5338159510 2024-01-27T01:06:02.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.880+0000 7f533d8d3700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f53380ca690 0x7f53380cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:02.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.880+0000 7f533d0d2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5338159510 0x7f53381598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:02.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.880+0000 7f533d8d3700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f53380ca690 0x7f53380cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:34168/0 (socket says 172.21.15.46:34168) 2024-01-27T01:06:02.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.880+0000 7f533d8d3700 1 -- 172.21.15.46:0/3543777451 learned_addr learned my addr 172.21.15.46:0/3543777451 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:02.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.880+0000 7f533c8d1700 1 --1- 172.21.15.46:0/3543777451 >> v1:172.21.15.106:6789/0 conn(0x7f53380ca220 0x7f53380c8be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:39268/0 (socket says 172.21.15.46:39268) 2024-01-27T01:06:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.880+0000 7f533d0d2700 1 -- 172.21.15.46:0/3543777451 >> v2:172.21.15.106:3300/0 conn(0x7f53380ca690 msgr2=0x7f53380cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.880+0000 7f533d0d2700 1 --2- 172.21.15.46:0/3543777451 >> v2:172.21.15.106:3300/0 conn(0x7f53380ca690 0x7f53380cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.880+0000 7f533d0d2700 1 -- 172.21.15.46:0/3543777451 >> v1:172.21.15.106:6789/0 conn(0x7f53380ca220 legacy=0x7f53380c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.880+0000 7f532ffff700 1 -- 172.21.15.46:0/3543777451 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316995465 0 0) 0x7f5338069260 con 0x7f53380ca220 2024-01-27T01:06:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.880+0000 7f533d0d2700 1 -- 172.21.15.46:0/3543777451 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53380c9a30 con 0x7f5338159510 2024-01-27T01:06:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.881+0000 7f533d8d3700 1 --2- 172.21.15.46:0/3543777451 >> v2:172.21.15.106:3300/0 conn(0x7f53380ca690 0x7f53380cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.881+0000 7f533d0d2700 1 --2- 172.21.15.46:0/3543777451 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5338159510 0x7f53381598e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5328009910 tx=0x7f53280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4484e3aa4eeb1a05 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.881+0000 7f532ffff700 1 -- 172.21.15.46:0/3543777451 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532800b640 con 0x7f5338159510 2024-01-27T01:06:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.881+0000 7f532ffff700 1 -- 172.21.15.46:0/3543777451 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532800b7a0 con 0x7f5338159510 2024-01-27T01:06:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.881+0000 7f532ffff700 1 -- 172.21.15.46:0/3543777451 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532800bb60 con 0x7f5338159510 2024-01-27T01:06:02.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.881+0000 7f533f336700 1 -- 172.21.15.46:0/3543777451 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5338159510 msgr2=0x7f53381598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:02.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.881+0000 7f533f336700 1 --2- 172.21.15.46:0/3543777451 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5338159510 0x7f53381598e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5328009910 tx=0x7f53280091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.881+0000 7f533f336700 1 -- 172.21.15.46:0/3543777451 shutdown_connections 2024-01-27T01:06:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.881+0000 7f533f336700 1 --2- 172.21.15.46:0/3543777451 >> v2:172.21.15.106:3300/0 conn(0x7f53380ca690 0x7f53380cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.881+0000 7f533f336700 1 --2- 172.21.15.46:0/3543777451 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5338159510 0x7f53381598e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.881+0000 7f533f336700 1 -- 172.21.15.46:0/3543777451 >> 172.21.15.46:0/3543777451 conn(0x7f533806bd90 msgr2=0x7f533806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.882+0000 7f533f336700 1 -- 172.21.15.46:0/3543777451 shutdown_connections 2024-01-27T01:06:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.882+0000 7f533f336700 1 -- 172.21.15.46:0/3543777451 wait complete. 2024-01-27T01:06:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.883+0000 7f533f336700 1 Processor -- start 2024-01-27T01:06:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.883+0000 7f533f336700 1 -- start start 2024-01-27T01:06:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.884+0000 7f533f336700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53380ca220 0x7f53380c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.884+0000 7f533f336700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5338159510 0x7f53381460b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.884+0000 7f533f336700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f533815ceb0 0x7f53380c3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.884+0000 7f533f336700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53380c6ab0 con 0x7f5338159510 2024-01-27T01:06:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.884+0000 7f533f336700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53380c5ea0 con 0x7f53380ca220 2024-01-27T01:06:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.884+0000 7f533f336700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53380c6180 con 0x7f533815ceb0 2024-01-27T01:06:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.884+0000 7f533c8d1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5338159510 0x7f53381460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.884+0000 7f533c8d1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5338159510 0x7f53381460b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44554/0 (socket says 172.21.15.46:44554) 2024-01-27T01:06:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.884+0000 7f533c8d1700 1 -- 172.21.15.46:0/3351499183 learned_addr learned my addr 172.21.15.46:0/3351499183 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.884+0000 7f533d8d3700 1 --2- 172.21.15.46:0/3351499183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f533815ceb0 0x7f53380c3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.884+0000 7f533d0d2700 1 --2- 172.21.15.46:0/3351499183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53380ca220 0x7f53380c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.884+0000 7f533c8d1700 1 -- 172.21.15.46:0/3351499183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f533815ceb0 msgr2=0x7f53380c3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.884+0000 7f533c8d1700 1 --2- 172.21.15.46:0/3351499183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f533815ceb0 0x7f53380c3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.884+0000 7f533c8d1700 1 -- 172.21.15.46:0/3351499183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53380ca220 msgr2=0x7f53380c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.884+0000 7f533c8d1700 1 --2- 172.21.15.46:0/3351499183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53380ca220 0x7f53380c5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.884+0000 7f533c8d1700 1 -- 172.21.15.46:0/3351499183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5330009ae0 con 0x7f5338159510 2024-01-27T01:06:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.884+0000 7f533c8d1700 1 --2- 172.21.15.46:0/3351499183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5338159510 0x7f53381460b0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f53340011e0 tx=0x7f5334007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.885+0000 7f532d7fa700 1 -- 172.21.15.46:0/3351499183 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5334007300 con 0x7f5338159510 2024-01-27T01:06:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.885+0000 7f532d7fa700 1 -- 172.21.15.46:0/3351499183 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5334007460 con 0x7f5338159510 2024-01-27T01:06:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.885+0000 7f533f336700 1 -- 172.21.15.46:0/3351499183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5328009420 con 0x7f5338159510 2024-01-27T01:06:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.885+0000 7f532d7fa700 1 -- 172.21.15.46:0/3351499183 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5334004830 con 0x7f5338159510 2024-01-27T01:06:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.885+0000 7f533f336700 1 -- 172.21.15.46:0/3351499183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53380c6740 con 0x7f5338159510 2024-01-27T01:06:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.886+0000 7f533f336700 1 -- 172.21.15.46:0/3351499183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f533804c710 con 0x7f5338159510 2024-01-27T01:06:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.886+0000 7f532d7fa700 1 -- 172.21.15.46:0/3351499183 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 121) v1 ==== 84873+0+0 (secure 0 0 0) 0x7f5334004be0 con 0x7f5338159510 2024-01-27T01:06:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.887+0000 7f532d7fa700 1 --2- 172.21.15.46:0/3351499183 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f531c0670a0 0x7f531c069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.887+0000 7f532d7fa700 1 -- 172.21.15.46:0/3351499183 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6834+0+0 (secure 0 0 0) 0x7f5334084120 con 0x7f5338159510 2024-01-27T01:06:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.887+0000 7f533d0d2700 1 --2- 172.21.15.46:0/3351499183 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f531c0670a0 0x7f531c069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.888+0000 7f533d0d2700 1 --2- 172.21.15.46:0/3351499183 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f531c0670a0 0x7f531c069550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f53280098e0 tx=0x7f5328017000 comp rx=0 tx=0).ready entity=mgr.9011 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:02.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:02.891+0000 7f532d7fa700 1 -- 172.21.15.46:0/3351499183 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5334050e10 con 0x7f5338159510 2024-01-27T01:06:03.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:03.155+0000 7f533f336700 1 -- 172.21.15.46:0/3351499183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f53380c6bf0 con 0x7f5338159510 2024-01-27T01:06:03.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:03.722+0000 7f532d7fa700 1 -- 172.21.15.46:0/3351499183 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 122) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f53340511e0 con 0x7f5338159510 2024-01-27T01:06:03.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:03.724+0000 7f532d7fa700 1 -- 172.21.15.46:0/3351499183 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v122) v1 ==== 93+0+0 (secure 0 0 0) 0x7f5334050e10 con 0x7f5338159510 2024-01-27T01:06:03.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:03.727+0000 7f533f336700 1 -- 172.21.15.46:0/3351499183 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f531c0670a0 msgr2=0x7f531c069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:03.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:03.727+0000 7f533f336700 1 --2- 172.21.15.46:0/3351499183 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f531c0670a0 0x7f531c069550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f53280098e0 tx=0x7f5328017000 comp rx=0 tx=0).stop 2024-01-27T01:06:03.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:03.727+0000 7f533f336700 1 -- 172.21.15.46:0/3351499183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5338159510 msgr2=0x7f53381460b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:03.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:03.727+0000 7f533f336700 1 --2- 172.21.15.46:0/3351499183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5338159510 0x7f53381460b0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f53340011e0 tx=0x7f5334007e10 comp rx=0 tx=0).stop 2024-01-27T01:06:03.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:03.727+0000 7f533f336700 1 -- 172.21.15.46:0/3351499183 shutdown_connections 2024-01-27T01:06:03.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:03.727+0000 7f533f336700 1 --2- 172.21.15.46:0/3351499183 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f531c0670a0 0x7f531c069550 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:03.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:03.727+0000 7f533f336700 1 --2- 172.21.15.46:0/3351499183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f533815ceb0 0x7f53380c3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:03.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:03.727+0000 7f533f336700 1 --2- 172.21.15.46:0/3351499183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53380ca220 0x7f53380c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:03.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:03.727+0000 7f533f336700 1 --2- 172.21.15.46:0/3351499183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5338159510 0x7f53381460b0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:03.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:03.727+0000 7f533f336700 1 -- 172.21.15.46:0/3351499183 >> 172.21.15.46:0/3351499183 conn(0x7f533806bd90 msgr2=0x7f5338152af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:03.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:03.727+0000 7f533f336700 1 -- 172.21.15.46:0/3351499183 shutdown_connections 2024-01-27T01:06:03.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:03.727+0000 7f533f336700 1 -- 172.21.15.46:0/3351499183 wait complete. 2024-01-27T01:06:03.743 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-27T01:06:03.743 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:06:03.795 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:06:03.831 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-27T01:06:03.831 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-27T01:06:03.831 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-27T01:06:03.831 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-27T01:06:03.877 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[106264]: fuse finished with error 0 and tester_r 0 2024-01-27T01:06:03.878 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T01:06:03.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:06:03.879 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-27T01:06:03.902 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T01:06:03.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:06:03.903 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-27T01:06:03.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:06:03.919 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-27T01:06:03.919 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T01:06:03.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:06:03.919 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-27T01:06:03.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:06:03.963 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-27T01:06:03.963 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T01:06:03.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:06:03.964 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-27T01:06:03.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:06:03.980 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-27T01:06:03.981 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:06:04.068 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:04.069+0000 7f86191b01c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:06:04.114 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:04.114+0000 7f3d64b1e1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:06:04.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.138+0000 7fe9cc93c700 1 Processor -- start 2024-01-27T01:06:04.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.138+0000 7fe9cc93c700 1 -- start start 2024-01-27T01:06:04.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.139+0000 7fe9cc93c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9c815ce40 0x7fe9c80d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.139+0000 7fe9cc93c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe9c815d210 0x7fe9c80cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.139+0000 7fe9cc93c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe9c80d1390 0x7fe9c80cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.139+0000 7fe9cc93c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c80cff70 con 0x7fe9c80d1390 2024-01-27T01:06:04.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.139+0000 7fe9cc93c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c80cf400 con 0x7fe9c815d210 2024-01-27T01:06:04.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.139+0000 7fe9cc93c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c80cf660 con 0x7fe9c815ce40 2024-01-27T01:06:04.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.140+0000 7fe9c6d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe9c80d1390 0x7fe9c80cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:04.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.140+0000 7fe9c6d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe9c80d1390 0x7fe9c80cbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44570/0 (socket says 172.21.15.46:44570) 2024-01-27T01:06:04.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.140+0000 7fe9c5d9b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe9c815d210 0x7fe9c80cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:04.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.140+0000 7fe9c6d9d700 1 -- 172.21.15.46:0/1938742016 learned_addr learned my addr 172.21.15.46:0/1938742016 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:04.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.140+0000 7fe9c659c700 1 --2- 172.21.15.46:0/1938742016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9c815ce40 0x7fe9c80d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:04.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.140+0000 7fe9c5d9b700 1 -- 172.21.15.46:0/1938742016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9c815ce40 msgr2=0x7fe9c80d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.140+0000 7fe9c5d9b700 1 --2- 172.21.15.46:0/1938742016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9c815ce40 0x7fe9c80d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.140+0000 7fe9c5d9b700 1 -- 172.21.15.46:0/1938742016 >> v2:172.21.15.46:3300/0 conn(0x7fe9c80d1390 msgr2=0x7fe9c80cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.140+0000 7fe9c5d9b700 1 --2- 172.21.15.46:0/1938742016 >> v2:172.21.15.46:3300/0 conn(0x7fe9c80d1390 0x7fe9c80cbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.140+0000 7fe9c5d9b700 1 -- 172.21.15.46:0/1938742016 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c80d0500 con 0x7fe9c815d210 2024-01-27T01:06:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.140+0000 7fe9c659c700 1 --2- 172.21.15.46:0/1938742016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9c815ce40 0x7fe9c80d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.140+0000 7fe9c6d9d700 1 --2- 172.21.15.46:0/1938742016 >> v2:172.21.15.46:3300/0 conn(0x7fe9c80d1390 0x7fe9c80cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.140+0000 7fe9c5d9b700 1 --2- 172.21.15.46:0/1938742016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9c815d210 0x7fe9c80cbab0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8088cb0 tx=0x7fe9bc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a2ca00066c6de25 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.140+0000 7fe9c559a700 1 -- 172.21.15.46:0/1938742016 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc011630 con 0x7fe9c815d210 2024-01-27T01:06:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.140+0000 7fe9c559a700 1 -- 172.21.15.46:0/1938742016 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc0117b0 con 0x7fe9c815d210 2024-01-27T01:06:04.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.140+0000 7fe9c559a700 1 -- 172.21.15.46:0/1938742016 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc011b70 con 0x7fe9c815d210 2024-01-27T01:06:04.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.141+0000 7fe9cc93c700 1 -- 172.21.15.46:0/1938742016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9c815d210 msgr2=0x7fe9c80cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:04.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.141+0000 7fe9cc93c700 1 --2- 172.21.15.46:0/1938742016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9c815d210 0x7fe9c80cbab0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8088cb0 tx=0x7fe9bc0091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.141+0000 7fe9cc93c700 1 -- 172.21.15.46:0/1938742016 shutdown_connections 2024-01-27T01:06:04.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.141+0000 7fe9cc93c700 1 --2- 172.21.15.46:0/1938742016 >> v2:172.21.15.46:3300/0 conn(0x7fe9c80d1390 0x7fe9c80cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.141+0000 7fe9cc93c700 1 --2- 172.21.15.46:0/1938742016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9c815ce40 0x7fe9c80d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.141+0000 7fe9cc93c700 1 --2- 172.21.15.46:0/1938742016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9c815d210 0x7fe9c80cbab0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.141+0000 7fe9cc93c700 1 -- 172.21.15.46:0/1938742016 >> 172.21.15.46:0/1938742016 conn(0x7fe9c806f840 msgr2=0x7fe9c80d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:04.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.141+0000 7fe9cc93c700 1 -- 172.21.15.46:0/1938742016 shutdown_connections 2024-01-27T01:06:04.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.142+0000 7fe9cc93c700 1 -- 172.21.15.46:0/1938742016 wait complete. 2024-01-27T01:06:04.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.143+0000 7fe9cc93c700 1 Processor -- start 2024-01-27T01:06:04.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.144+0000 7fe9cc93c700 1 -- start start 2024-01-27T01:06:04.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.145+0000 7fe9cc93c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9c80d1390 0x7fe9c8159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.145+0000 7fe9cc93c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9c815ce40 0x7fe9c8069cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.145+0000 7fe9cc93c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9c815d210 0x7fe9c806a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.145+0000 7fe9cc93c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c816a570 con 0x7fe9c815d210 2024-01-27T01:06:04.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.145+0000 7fe9cc93c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c80c86c0 con 0x7fe9c80d1390 2024-01-27T01:06:04.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.145+0000 7fe9cc93c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c80c89a0 con 0x7fe9c815ce40 2024-01-27T01:06:04.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.145+0000 7fe9c6d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9c815d210 0x7fe9c806a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:04.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.145+0000 7fe9c6d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9c815d210 0x7fe9c806a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44584/0 (socket says 172.21.15.46:44584) 2024-01-27T01:06:04.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.145+0000 7fe9c659c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9c80d1390 0x7fe9c8159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:04.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.145+0000 7fe9c6d9d700 1 -- 172.21.15.46:0/732441652 learned_addr learned my addr 172.21.15.46:0/732441652 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:04.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.145+0000 7fe9c5d9b700 1 --2- 172.21.15.46:0/732441652 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9c815ce40 0x7fe9c8069cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:04.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.145+0000 7fe9c659c700 1 -- 172.21.15.46:0/732441652 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9c815ce40 msgr2=0x7fe9c8069cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:04.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.145+0000 7fe9c659c700 1 --2- 172.21.15.46:0/732441652 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9c815ce40 0x7fe9c8069cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.145+0000 7fe9c659c700 1 -- 172.21.15.46:0/732441652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9c815d210 msgr2=0x7fe9c806a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:04.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.145+0000 7fe9c659c700 1 --2- 172.21.15.46:0/732441652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9c815d210 0x7fe9c806a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.145+0000 7fe9c659c700 1 -- 172.21.15.46:0/732441652 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c0009ae0 con 0x7fe9c80d1390 2024-01-27T01:06:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.145+0000 7fe9c6d9d700 1 --2- 172.21.15.46:0/732441652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9c815d210 0x7fe9c806a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.146+0000 7fe9c659c700 1 --2- 172.21.15.46:0/732441652 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9c80d1390 0x7fe9c8159a40 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe9b800ec80 tx=0x7fe9b8009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.146+0000 7fe9b67fc700 1 -- 172.21.15.46:0/732441652 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b8005a00 con 0x7fe9c80d1390 2024-01-27T01:06:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.146+0000 7fe9b67fc700 1 -- 172.21.15.46:0/732441652 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b8005b60 con 0x7fe9c80d1390 2024-01-27T01:06:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.146+0000 7fe9b67fc700 1 -- 172.21.15.46:0/732441652 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b800c850 con 0x7fe9c80d1390 2024-01-27T01:06:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.146+0000 7fe9cc93c700 1 -- 172.21.15.46:0/732441652 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b80093c0 con 0x7fe9c80d1390 2024-01-27T01:06:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.146+0000 7fe9cc93c700 1 -- 172.21.15.46:0/732441652 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9bc009420 con 0x7fe9c80d1390 2024-01-27T01:06:04.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.147+0000 7fe9cc93c700 1 -- 172.21.15.46:0/732441652 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c80cb810 con 0x7fe9c80d1390 2024-01-27T01:06:04.149 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:04.148+0000 7f86191b01c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:06:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.147+0000 7fe9b67fc700 1 -- 172.21.15.46:0/732441652 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 122) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fe9b800cc00 con 0x7fe9c80d1390 2024-01-27T01:06:04.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.148+0000 7fe9b67fc700 1 --2- 172.21.15.46:0/732441652 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe9a8067050 0x7fe9a8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.148+0000 7fe9c5d9b700 1 -- 172.21.15.46:0/732441652 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe9a8067050 msgr2=0x7fe9a8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:04.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.148+0000 7fe9b67fc700 1 -- 172.21.15.46:0/732441652 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6922+0+0 (secure 0 0 0) 0x7fe9b8084fe0 con 0x7fe9c80d1390 2024-01-27T01:06:04.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.148+0000 7fe9c5d9b700 1 --2- 172.21.15.46:0/732441652 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe9a8067050 0x7fe9a8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:06:04.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.152+0000 7fe9b67fc700 1 -- 172.21.15.46:0/732441652 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe9b8051bf0 con 0x7fe9c80d1390 2024-01-27T01:06:04.206 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:04.207+0000 7f3d64b1e1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:06:04.219 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:04.220+0000 7f86191b01c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:06:04.287 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:04.287+0000 7f86191b01c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:06:04.293 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:04.293+0000 7f3d64b1e1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:06:04.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.349+0000 7fe9c5d9b700 1 -- 172.21.15.46:0/732441652 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe9a8067050 msgr2=0x7fe9a8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:04.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.349+0000 7fe9c5d9b700 1 --2- 172.21.15.46:0/732441652 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe9a8067050 0x7fe9a8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:06:04.368 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:04.368+0000 7f3d64b1e1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:06:04.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.406+0000 7fe9cc93c700 1 -- 172.21.15.46:0/732441652 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9c804c710 con 0x7fe9c80d1390 2024-01-27T01:06:04.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.407+0000 7fe9b67fc700 1 -- 172.21.15.46:0/732441652 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4612 (secure 0 0 0) 0x7fe9b8055210 con 0x7fe9c80d1390 2024-01-27T01:06:04.407 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:04.407 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":107,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8891,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/975115379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":975115379},{"type":"v1","addr":"172.21.15.106:6833","nonce":975115379}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10036,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3860120510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3860120510},{"type":"v1","addr":"172.21.15.46:6834","nonce":3860120510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10045,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3603549739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3603549739},{"type":"v1","addr":"172.21.15.106:6835","nonce":3603549739}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:03:44.141242+0000","modified":"2024-01-27T01:03:47.727201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10392":{"gid":10392,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/976738644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":976738644},{"type":"v1","addr":"172.21.15.46:6835","nonce":976738644}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2024-01-27T01:06:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.409+0000 7fe9cc93c700 1 -- 172.21.15.46:0/732441652 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe9a8067050 msgr2=0x7fe9a8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:06:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.409+0000 7fe9cc93c700 1 --2- 172.21.15.46:0/732441652 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe9a8067050 0x7fe9a8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.409+0000 7fe9cc93c700 1 -- 172.21.15.46:0/732441652 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9c80d1390 msgr2=0x7fe9c8159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.409+0000 7fe9cc93c700 1 --2- 172.21.15.46:0/732441652 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9c80d1390 0x7fe9c8159a40 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe9b800ec80 tx=0x7fe9b8009120 comp rx=0 tx=0).stop 2024-01-27T01:06:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.409+0000 7fe9cc93c700 1 -- 172.21.15.46:0/732441652 shutdown_connections 2024-01-27T01:06:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.409+0000 7fe9cc93c700 1 --2- 172.21.15.46:0/732441652 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe9a8067050 0x7fe9a8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.409+0000 7fe9cc93c700 1 --2- 172.21.15.46:0/732441652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9c815d210 0x7fe9c806a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.409+0000 7fe9cc93c700 1 --2- 172.21.15.46:0/732441652 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9c80d1390 0x7fe9c8159a40 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.409+0000 7fe9cc93c700 1 --2- 172.21.15.46:0/732441652 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9c815ce40 0x7fe9c8069cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.409+0000 7fe9cc93c700 1 -- 172.21.15.46:0/732441652 >> 172.21.15.46:0/732441652 conn(0x7fe9c806f840 msgr2=0x7fe9c80b5910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:04.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.410+0000 7fe9cc93c700 1 -- 172.21.15.46:0/732441652 shutdown_connections 2024-01-27T01:06:04.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.410+0000 7fe9cc93c700 1 -- 172.21.15.46:0/732441652 wait complete. 2024-01-27T01:06:04.411 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 107 2024-01-27T01:06:04.426 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:06:04.441 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:04.442+0000 7f86191b01c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:06:04.549 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:04.549+0000 7f3d64b1e1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:06:04.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.559+0000 7fb194eab700 1 Processor -- start 2024-01-27T01:06:04.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.559+0000 7fb194eab700 1 -- start start 2024-01-27T01:06:04.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.560+0000 7fb194eab700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb19015d210 0x7fb1900cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.560+0000 7fb194eab700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1900cc450 con 0x7fb1900cbff0 2024-01-27T01:06:04.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.560+0000 7fb194eab700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1900cff70 con 0x7fb19015ce40 2024-01-27T01:06:04.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.560+0000 7fb194eab700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1900cf400 con 0x7fb19015d210 2024-01-27T01:06:04.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.560+0000 7fb18dd9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb19015d210 0x7fb1900cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:04.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.560+0000 7fb18dd9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb19015d210 0x7fb1900cbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44586/0 (socket says 172.21.15.46:44586) 2024-01-27T01:06:04.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.560+0000 7fb18dd9b700 1 -- 172.21.15.46:0/348760945 learned_addr learned my addr 172.21.15.46:0/348760945 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:04.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.560+0000 7fb18dd9b700 1 -- 172.21.15.46:0/348760945 >> v1:172.21.15.106:6789/0 conn(0x7fb19015ce40 legacy=0x7fb1900d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:04.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.560+0000 7fb18dd9b700 1 -- 172.21.15.46:0/348760945 >> v1:172.21.15.46:6789/0 conn(0x7fb1900cbff0 legacy=0x7fb1900ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:04.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.560+0000 7fb18dd9b700 1 -- 172.21.15.46:0/348760945 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1900cf660 con 0x7fb19015d210 2024-01-27T01:06:04.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.560+0000 7fb18dd9b700 1 --2- 172.21.15.46:0/348760945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb19015d210 0x7fb1900cbab0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fb1840099b0 tx=0x7fb184009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=843f8321ba785f61 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:04.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.561+0000 7fb18d59a700 1 -- 172.21.15.46:0/348760945 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18400b860 con 0x7fb19015d210 2024-01-27T01:06:04.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.561+0000 7fb18d59a700 1 -- 172.21.15.46:0/348760945 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18400b9c0 con 0x7fb19015d210 2024-01-27T01:06:04.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.561+0000 7fb18d59a700 1 -- 172.21.15.46:0/348760945 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18400bd60 con 0x7fb19015d210 2024-01-27T01:06:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.561+0000 7fb194eab700 1 -- 172.21.15.46:0/348760945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb19015d210 msgr2=0x7fb1900cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.561+0000 7fb194eab700 1 --2- 172.21.15.46:0/348760945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb19015d210 0x7fb1900cbab0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fb1840099b0 tx=0x7fb184009250 comp rx=0 tx=0).stop 2024-01-27T01:06:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.561+0000 7fb194eab700 1 -- 172.21.15.46:0/348760945 shutdown_connections 2024-01-27T01:06:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.561+0000 7fb194eab700 1 --2- 172.21.15.46:0/348760945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb19015d210 0x7fb1900cbab0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.561+0000 7fb194eab700 1 -- 172.21.15.46:0/348760945 >> 172.21.15.46:0/348760945 conn(0x7fb19006f840 msgr2=0x7fb1900d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.561+0000 7fb194eab700 1 -- 172.21.15.46:0/348760945 shutdown_connections 2024-01-27T01:06:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.561+0000 7fb194eab700 1 -- 172.21.15.46:0/348760945 wait complete. 2024-01-27T01:06:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.563+0000 7fb194eab700 1 Processor -- start 2024-01-27T01:06:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.563+0000 7fb194eab700 1 -- start start 2024-01-27T01:06:04.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.563+0000 7fb194eab700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1900cbff0 0x7fb190151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.563+0000 7fb194eab700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb19015ce40 0x7fb19016af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.563+0000 7fb194eab700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb19015d210 0x7fb190155b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.563+0000 7fb194eab700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb190158360 con 0x7fb1900cbff0 2024-01-27T01:06:04.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.563+0000 7fb194eab700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb190157500 con 0x7fb19015ce40 2024-01-27T01:06:04.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.563+0000 7fb194eab700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1901523d0 con 0x7fb19015d210 2024-01-27T01:06:04.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.564+0000 7fb18e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1900cbff0 0x7fb190151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:04.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.564+0000 7fb18e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1900cbff0 0x7fb190151e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44598/0 (socket says 172.21.15.46:44598) 2024-01-27T01:06:04.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.564+0000 7fb18e59c700 1 -- 172.21.15.46:0/2991230134 learned_addr learned my addr 172.21.15.46:0/2991230134 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:04.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.564+0000 7fb18dd9b700 1 --2- 172.21.15.46:0/2991230134 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb19015ce40 0x7fb19016af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:04.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.564+0000 7fb18ed9d700 1 --2- 172.21.15.46:0/2991230134 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb19015d210 0x7fb190155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:04.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.564+0000 7fb18e59c700 1 -- 172.21.15.46:0/2991230134 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb19015d210 msgr2=0x7fb190155b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:04.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.564+0000 7fb18e59c700 1 --2- 172.21.15.46:0/2991230134 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb19015d210 0x7fb190155b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.564+0000 7fb18e59c700 1 -- 172.21.15.46:0/2991230134 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb19015ce40 msgr2=0x7fb19016af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:04.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.564+0000 7fb18e59c700 1 --2- 172.21.15.46:0/2991230134 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb19015ce40 0x7fb19016af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.564+0000 7fb18e59c700 1 -- 172.21.15.46:0/2991230134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1840094f0 con 0x7fb1900cbff0 2024-01-27T01:06:04.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.564+0000 7fb18e59c700 1 --2- 172.21.15.46:0/2991230134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1900cbff0 0x7fb190151e90 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fb18000cde0 tx=0x7fb180001c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:04.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.564+0000 7fb17e7fc700 1 -- 172.21.15.46:0/2991230134 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18000ab30 con 0x7fb1900cbff0 2024-01-27T01:06:04.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.564+0000 7fb17e7fc700 1 -- 172.21.15.46:0/2991230134 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18000ac90 con 0x7fb1900cbff0 2024-01-27T01:06:04.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.564+0000 7fb17e7fc700 1 -- 172.21.15.46:0/2991230134 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb180012670 con 0x7fb1900cbff0 2024-01-27T01:06:04.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.564+0000 7fb194eab700 1 -- 172.21.15.46:0/2991230134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb19016b440 con 0x7fb1900cbff0 2024-01-27T01:06:04.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.564+0000 7fb194eab700 1 -- 172.21.15.46:0/2991230134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb190157a30 con 0x7fb1900cbff0 2024-01-27T01:06:04.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.565+0000 7fb194eab700 1 -- 172.21.15.46:0/2991230134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb19004c710 con 0x7fb1900cbff0 2024-01-27T01:06:04.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.566+0000 7fb17e7fc700 1 -- 172.21.15.46:0/2991230134 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 122) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fb180012a20 con 0x7fb1900cbff0 2024-01-27T01:06:04.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.566+0000 7fb17e7fc700 1 --2- 172.21.15.46:0/2991230134 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb170067050 0x7fb170069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.566+0000 7fb18dd9b700 1 -- 172.21.15.46:0/2991230134 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb170067050 msgr2=0x7fb170069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.567+0000 7fb17e7fc700 1 -- 172.21.15.46:0/2991230134 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6922+0+0 (secure 0 0 0) 0x7fb180085390 con 0x7fb1900cbff0 2024-01-27T01:06:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.567+0000 7fb18dd9b700 1 --2- 172.21.15.46:0/2991230134 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb170067050 0x7fb170069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:06:04.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.571+0000 7fb17e7fc700 1 -- 172.21.15.46:0/2991230134 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb180051f90 con 0x7fb1900cbff0 2024-01-27T01:06:04.578 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:04.578+0000 7f86191b01c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:06:04.645 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:04.645+0000 7f86191b01c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:06:04.707 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:04.707+0000 7f3d64b1e1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:06:04.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.768+0000 7fb18dd9b700 1 -- 172.21.15.46:0/2991230134 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb170067050 msgr2=0x7fb170069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:04.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.768+0000 7fb18dd9b700 1 --2- 172.21.15.46:0/2991230134 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb170067050 0x7fb170069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:06:04.768 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:04.768+0000 7f86191b01c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:06:04.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.788+0000 7fb194eab700 1 -- 172.21.15.46:0/2991230134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1900d0500 con 0x7fb1900cbff0 2024-01-27T01:06:04.789 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:04.789+0000 7f3d64b1e1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:06:04.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.789+0000 7fb17e7fc700 1 -- 172.21.15.46:0/2991230134 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v210) v1 ==== 74+0+16934 (secure 0 0 0) 0x7fb1800555b0 con 0x7fb1900cbff0 2024-01-27T01:06:04.790 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:04.790 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":210,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:06:04.711162+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:03:42.648202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-01-27T01:03:43.106225+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"210","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":210,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.1b","mappings":[{"from":3,"to":0}]},{"pgid":"26.31","mappings":[{"from":1,"to":0}]},{"pgid":"26.3a","mappings":[{"from":1,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/1813035004":"2024-01-28T01:03:58.706905+0000","172.21.15.106:0/2234269173":"2024-01-28T01:03:58.706905+0000","172.21.15.106:6837/62434":"2024-01-28T01:03:58.706905+0000","172.21.15.106:6836/62434":"2024-01-28T01:03:58.706905+0000","172.21.15.106:0/3361483200":"2024-01-28T01:03:58.706905+0000","172.21.15.106:0/1169106698":"2024-01-28T01:03:58.706905+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":26,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:06:04.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.792+0000 7fb194eab700 1 -- 172.21.15.46:0/2991230134 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb170067050 msgr2=0x7fb170069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:06:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.792+0000 7fb194eab700 1 --2- 172.21.15.46:0/2991230134 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb170067050 0x7fb170069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.792+0000 7fb194eab700 1 -- 172.21.15.46:0/2991230134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1900cbff0 msgr2=0x7fb190151e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.792+0000 7fb194eab700 1 --2- 172.21.15.46:0/2991230134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1900cbff0 0x7fb190151e90 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fb18000cde0 tx=0x7fb180001c20 comp rx=0 tx=0).stop 2024-01-27T01:06:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.792+0000 7fb194eab700 1 -- 172.21.15.46:0/2991230134 shutdown_connections 2024-01-27T01:06:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.792+0000 7fb194eab700 1 --2- 172.21.15.46:0/2991230134 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb170067050 0x7fb170069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.792+0000 7fb194eab700 1 --2- 172.21.15.46:0/2991230134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1900cbff0 0x7fb190151e90 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.792+0000 7fb194eab700 1 --2- 172.21.15.46:0/2991230134 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb19015d210 0x7fb190155b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.792+0000 7fb194eab700 1 --2- 172.21.15.46:0/2991230134 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb19015ce40 0x7fb19016af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.792+0000 7fb194eab700 1 -- 172.21.15.46:0/2991230134 >> 172.21.15.46:0/2991230134 conn(0x7fb19006f840 msgr2=0x7fb1900b4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.792+0000 7fb194eab700 1 -- 172.21.15.46:0/2991230134 shutdown_connections 2024-01-27T01:06:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.792+0000 7fb194eab700 1 -- 172.21.15.46:0/2991230134 wait complete. 2024-01-27T01:06:04.807 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-27T01:03:42.648202+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '204', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2024-01-27T01:03:43.106225+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '210', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 210, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-27T01:06:04.807 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-27T01:06:04.841 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:04.842+0000 7f86191b01c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:06:04.934 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:04.934+0000 7f3d64b1e1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:06:04.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.945+0000 7f82eac75700 1 Processor -- start 2024-01-27T01:06:04.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.945+0000 7f82eac75700 1 -- start start 2024-01-27T01:06:04.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.946+0000 7f82eac75700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82e41598e0 0x7f82e40c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.946+0000 7f82eac75700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f82e40c9120 0x7f82e40cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.946+0000 7f82eac75700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e4069260 con 0x7f82e4159510 2024-01-27T01:06:04.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.946+0000 7f82eac75700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e40ca930 con 0x7f82e40c9120 2024-01-27T01:06:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.946+0000 7f82eac75700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e4159d50 con 0x7f82e41598e0 2024-01-27T01:06:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.946+0000 7f82e8a11700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f82e4159510 0x7f82e40ca220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57968/0 (socket says 172.21.15.46:57968) 2024-01-27T01:06:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.946+0000 7f82e8a11700 1 -- 172.21.15.46:0/3036905311 learned_addr learned my addr 172.21.15.46:0/3036905311 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.946+0000 7f82e3fff700 1 --2- 172.21.15.46:0/3036905311 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82e41598e0 0x7f82e40c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.946+0000 7f82e9212700 1 --2- 172.21.15.46:0/3036905311 >> v2:172.21.15.106:3300/0 conn(0x7f82e40c9120 0x7f82e40cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.947+0000 7f82e37fe700 1 -- 172.21.15.46:0/3036905311 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806865283 0 0) 0x7f82e4069260 con 0x7f82e4159510 2024-01-27T01:06:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.947+0000 7f82e37fe700 1 -- 172.21.15.46:0/3036905311 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82cc003570 con 0x7f82e4159510 2024-01-27T01:06:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.947+0000 7f82e3fff700 1 -- 172.21.15.46:0/3036905311 >> v2:172.21.15.106:3300/0 conn(0x7f82e40c9120 msgr2=0x7f82e40cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.947+0000 7f82e3fff700 1 --2- 172.21.15.46:0/3036905311 >> v2:172.21.15.106:3300/0 conn(0x7f82e40c9120 0x7f82e40cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.947+0000 7f82e3fff700 1 -- 172.21.15.46:0/3036905311 >> v1:172.21.15.46:6789/0 conn(0x7f82e4159510 legacy=0x7f82e40ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.947+0000 7f82e3fff700 1 -- 172.21.15.46:0/3036905311 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e415d670 con 0x7f82e41598e0 2024-01-27T01:06:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.947+0000 7f82e9212700 1 --2- 172.21.15.46:0/3036905311 >> v2:172.21.15.106:3300/0 conn(0x7f82e40c9120 0x7f82e40cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.947+0000 7f82e3fff700 1 --2- 172.21.15.46:0/3036905311 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82e41598e0 0x7f82e40c8be0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f82d80098e0 tx=0x7f82d80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63cf68d27c99b2f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.947+0000 7f82e37fe700 1 -- 172.21.15.46:0/3036905311 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d8011630 con 0x7f82e41598e0 2024-01-27T01:06:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.947+0000 7f82e37fe700 1 -- 172.21.15.46:0/3036905311 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d8011790 con 0x7f82e41598e0 2024-01-27T01:06:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.947+0000 7f82e37fe700 1 -- 172.21.15.46:0/3036905311 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d8011b30 con 0x7f82e41598e0 2024-01-27T01:06:04.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.948+0000 7f82eac75700 1 -- 172.21.15.46:0/3036905311 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82e41598e0 msgr2=0x7f82e40c8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:04.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.948+0000 7f82eac75700 1 --2- 172.21.15.46:0/3036905311 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82e41598e0 0x7f82e40c8be0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f82d80098e0 tx=0x7f82d80091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.948+0000 7f82eac75700 1 -- 172.21.15.46:0/3036905311 shutdown_connections 2024-01-27T01:06:04.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.948+0000 7f82eac75700 1 --2- 172.21.15.46:0/3036905311 >> v2:172.21.15.106:3300/0 conn(0x7f82e40c9120 0x7f82e40cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.948+0000 7f82eac75700 1 --2- 172.21.15.46:0/3036905311 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82e41598e0 0x7f82e40c8be0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.948+0000 7f82eac75700 1 -- 172.21.15.46:0/3036905311 >> 172.21.15.46:0/3036905311 conn(0x7f82e406bd90 msgr2=0x7f82e406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:04.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.948+0000 7f82eac75700 1 -- 172.21.15.46:0/3036905311 shutdown_connections 2024-01-27T01:06:04.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.948+0000 7f82eac75700 1 -- 172.21.15.46:0/3036905311 wait complete. 2024-01-27T01:06:04.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.949+0000 7f82eac75700 1 Processor -- start 2024-01-27T01:06:04.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.949+0000 7f82eac75700 1 -- start start 2024-01-27T01:06:04.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.950+0000 7f82eac75700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82e40c9120 0x7f82e4169950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.950+0000 7f82eac75700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e4159510 0x7f82e416c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.950+0000 7f82eac75700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82e41598e0 0x7f82e40c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.950+0000 7f82eac75700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e40b1470 con 0x7f82e4159510 2024-01-27T01:06:04.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.950+0000 7f82eac75700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e415c640 con 0x7f82e41598e0 2024-01-27T01:06:04.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.950+0000 7f82eac75700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e415c920 con 0x7f82e40c9120 2024-01-27T01:06:04.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.950+0000 7f82e3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e4159510 0x7f82e416c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:04.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.950+0000 7f82e8a11700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82e40c9120 0x7f82e4169950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:04.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.950+0000 7f82e3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e4159510 0x7f82e416c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44600/0 (socket says 172.21.15.46:44600) 2024-01-27T01:06:04.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.950+0000 7f82e3fff700 1 -- 172.21.15.46:0/3660869773 learned_addr learned my addr 172.21.15.46:0/3660869773 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:04.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.951+0000 7f82e9212700 1 --2- 172.21.15.46:0/3660869773 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82e41598e0 0x7f82e40c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:04.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.951+0000 7f82e3fff700 1 -- 172.21.15.46:0/3660869773 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82e40c9120 msgr2=0x7f82e4169950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:04.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.951+0000 7f82e3fff700 1 --2- 172.21.15.46:0/3660869773 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82e40c9120 0x7f82e4169950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.951+0000 7f82e3fff700 1 -- 172.21.15.46:0/3660869773 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82e41598e0 msgr2=0x7f82e40c4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:04.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.951+0000 7f82e3fff700 1 --2- 172.21.15.46:0/3660869773 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82e41598e0 0x7f82e40c4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:04.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.951+0000 7f82e3fff700 1 -- 172.21.15.46:0/3660869773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d4009420 con 0x7f82e4159510 2024-01-27T01:06:04.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.951+0000 7f82e9212700 1 --2- 172.21.15.46:0/3660869773 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82e41598e0 0x7f82e40c4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:04.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.951+0000 7f82e3fff700 1 --2- 172.21.15.46:0/3660869773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e4159510 0x7f82e416c580 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f82e40691d0 tx=0x7f82d8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:04.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.951+0000 7f82e0ff9700 1 -- 172.21.15.46:0/3660869773 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d8011b10 con 0x7f82e4159510 2024-01-27T01:06:04.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.951+0000 7f82e0ff9700 1 -- 172.21.15.46:0/3660869773 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d8000cc0 con 0x7f82e4159510 2024-01-27T01:06:04.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.951+0000 7f82eac75700 1 -- 172.21.15.46:0/3660869773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d8009420 con 0x7f82e4159510 2024-01-27T01:06:04.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.951+0000 7f82e0ff9700 1 -- 172.21.15.46:0/3660869773 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d8025620 con 0x7f82e4159510 2024-01-27T01:06:04.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.951+0000 7f82eac75700 1 -- 172.21.15.46:0/3660869773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e415cee0 con 0x7f82e4159510 2024-01-27T01:06:04.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.952+0000 7f82eac75700 1 -- 172.21.15.46:0/3660869773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e40b15e0 con 0x7f82e4159510 2024-01-27T01:06:04.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.953+0000 7f82e0ff9700 1 -- 172.21.15.46:0/3660869773 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 122) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f82d8025a10 con 0x7f82e4159510 2024-01-27T01:06:04.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.954+0000 7f82e0ff9700 1 --2- 172.21.15.46:0/3660869773 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f82d0067050 0x7f82d0069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:04.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.954+0000 7f82e8a11700 1 -- 172.21.15.46:0/3660869773 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f82d0067050 msgr2=0x7f82d0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:04.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.954+0000 7f82e0ff9700 1 -- 172.21.15.46:0/3660869773 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6906+0+0 (secure 0 0 0) 0x7f82d8086960 con 0x7f82e4159510 2024-01-27T01:06:04.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.954+0000 7f82e8a11700 1 --2- 172.21.15.46:0/3660869773 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f82d0067050 0x7f82d0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:06:04.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:04.958+0000 7f82e0ff9700 1 -- 172.21.15.46:0/3660869773 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f82d80536b0 con 0x7f82e4159510 2024-01-27T01:06:04.981 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:04.982+0000 7f86191b01c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:06:05.020 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:05.020+0000 7f3d64b1e1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:06:05.107 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:05.107+0000 7f86191b01c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:06:05.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.155+0000 7f82e8a11700 1 -- 172.21.15.46:0/3660869773 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f82d0067050 msgr2=0x7f82d0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:05.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.155+0000 7f82e8a11700 1 --2- 172.21.15.46:0/3660869773 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f82d0067050 0x7f82d0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:06:05.192 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:05.192+0000 7f3d64b1e1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:06:05.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.208+0000 7f82eac75700 1 -- 172.21.15.46:0/3660869773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f82e40b15b0 con 0x7f82e4159510 2024-01-27T01:06:05.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.208+0000 7f82e0ff9700 1 -- 172.21.15.46:0/3660869773 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v210) v1 ==== 93+0+10 (secure 0 0 0) 0x7f82d8056cd0 con 0x7f82e4159510 2024-01-27T01:06:05.208 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-27T01:06:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.211+0000 7f82eac75700 1 -- 172.21.15.46:0/3660869773 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f82d0067050 msgr2=0x7f82d0069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:06:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.211+0000 7f82eac75700 1 --2- 172.21.15.46:0/3660869773 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f82d0067050 0x7f82d0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.211+0000 7f82eac75700 1 -- 172.21.15.46:0/3660869773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e4159510 msgr2=0x7f82e416c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.211+0000 7f82eac75700 1 --2- 172.21.15.46:0/3660869773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e4159510 0x7f82e416c580 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f82e40691d0 tx=0x7f82d8017000 comp rx=0 tx=0).stop 2024-01-27T01:06:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.211+0000 7f82eac75700 1 -- 172.21.15.46:0/3660869773 shutdown_connections 2024-01-27T01:06:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.211+0000 7f82eac75700 1 --2- 172.21.15.46:0/3660869773 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f82d0067050 0x7f82d0069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.211+0000 7f82eac75700 1 --2- 172.21.15.46:0/3660869773 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82e41598e0 0x7f82e40c4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.211+0000 7f82eac75700 1 --2- 172.21.15.46:0/3660869773 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82e40c9120 0x7f82e4169950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.211+0000 7f82eac75700 1 --2- 172.21.15.46:0/3660869773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82e4159510 0x7f82e416c580 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.211+0000 7f82eac75700 1 -- 172.21.15.46:0/3660869773 >> 172.21.15.46:0/3660869773 conn(0x7f82e406bd90 msgr2=0x7f82e4068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:05.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.211+0000 7f82eac75700 1 -- 172.21.15.46:0/3660869773 shutdown_connections 2024-01-27T01:06:05.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.211+0000 7f82eac75700 1 -- 172.21.15.46:0/3660869773 wait complete. 2024-01-27T01:06:05.226 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-27T01:06:05.338 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:05.338+0000 7f3d64b1e1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:06:05.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.363+0000 7f7cd454a700 1 Processor -- start 2024-01-27T01:06:05.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.363+0000 7f7cd454a700 1 -- start start 2024-01-27T01:06:05.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.364+0000 7f7cd454a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ccc068b00 0x7f7ccc068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:05.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.364+0000 7f7cd454a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7ccc06a660 0x7f7ccc159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:05.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.364+0000 7f7cd454a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7ccc06add0 0x7f7ccc06b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:05.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.364+0000 7f7cd454a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ccc0a9a80 con 0x7f7ccc06add0 2024-01-27T01:06:05.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.364+0000 7f7cd454a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ccc06aad0 con 0x7f7ccc06a660 2024-01-27T01:06:05.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.364+0000 7f7cd454a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc06b790 con 0x7f7ccc068b00 2024-01-27T01:06:05.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.364+0000 7f7cd2ae7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7ccc06add0 0x7f7ccc06b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:05.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.364+0000 7f7cd2ae7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7ccc06add0 0x7f7ccc06b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44610/0 (socket says 172.21.15.46:44610) 2024-01-27T01:06:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.364+0000 7f7cd2ae7700 1 -- 172.21.15.46:0/881462826 learned_addr learned my addr 172.21.15.46:0/881462826 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.365+0000 7f7cd22e6700 1 --2- 172.21.15.46:0/881462826 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ccc068b00 0x7f7ccc068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.365+0000 7f7cd1ae5700 1 --2- 172.21.15.46:0/881462826 >> v2:172.21.15.106:3300/0 conn(0x7f7ccc06a660 0x7f7ccc159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.365+0000 7f7cd2ae7700 1 -- 172.21.15.46:0/881462826 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ccc068b00 msgr2=0x7f7ccc068ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.365+0000 7f7cd2ae7700 1 --2- 172.21.15.46:0/881462826 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ccc068b00 0x7f7ccc068ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.365+0000 7f7cd2ae7700 1 -- 172.21.15.46:0/881462826 >> v2:172.21.15.106:3300/0 conn(0x7f7ccc06a660 msgr2=0x7f7ccc159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.365+0000 7f7cd2ae7700 1 --2- 172.21.15.46:0/881462826 >> v2:172.21.15.106:3300/0 conn(0x7f7ccc06a660 0x7f7ccc159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.365+0000 7f7cd2ae7700 1 -- 172.21.15.46:0/881462826 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc15a3e0 con 0x7f7ccc06add0 2024-01-27T01:06:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.365+0000 7f7cd2ae7700 1 --2- 172.21.15.46:0/881462826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc06add0 0x7f7ccc06b250 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc0ccb20 tx=0x7f7cc4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70495d660001ab5 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.365+0000 7f7cd12e4700 1 -- 172.21.15.46:0/881462826 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc400e070 con 0x7f7ccc06add0 2024-01-27T01:06:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.365+0000 7f7cd12e4700 1 -- 172.21.15.46:0/881462826 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc400b970 con 0x7f7ccc06add0 2024-01-27T01:06:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.365+0000 7f7cd12e4700 1 -- 172.21.15.46:0/881462826 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc40092e0 con 0x7f7ccc06add0 2024-01-27T01:06:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.366+0000 7f7cd454a700 1 -- 172.21.15.46:0/881462826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc06add0 msgr2=0x7f7ccc06b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.366+0000 7f7cd454a700 1 --2- 172.21.15.46:0/881462826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc06add0 0x7f7ccc06b250 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc0ccb20 tx=0x7f7cc4009840 comp rx=0 tx=0).stop 2024-01-27T01:06:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.366+0000 7f7cd454a700 1 -- 172.21.15.46:0/881462826 shutdown_connections 2024-01-27T01:06:05.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.366+0000 7f7cd454a700 1 --2- 172.21.15.46:0/881462826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc06add0 0x7f7ccc06b250 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.366+0000 7f7cd454a700 1 --2- 172.21.15.46:0/881462826 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ccc068b00 0x7f7ccc068ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.366+0000 7f7cd454a700 1 --2- 172.21.15.46:0/881462826 >> v2:172.21.15.106:3300/0 conn(0x7f7ccc06a660 0x7f7ccc159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.366+0000 7f7cd454a700 1 -- 172.21.15.46:0/881462826 >> 172.21.15.46:0/881462826 conn(0x7f7ccc0ca100 msgr2=0x7f7ccc06e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:05.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.366+0000 7f7cd454a700 1 -- 172.21.15.46:0/881462826 shutdown_connections 2024-01-27T01:06:05.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.366+0000 7f7cd454a700 1 -- 172.21.15.46:0/881462826 wait complete. 2024-01-27T01:06:05.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.367+0000 7f7cd454a700 1 Processor -- start 2024-01-27T01:06:05.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.367+0000 7f7cd454a700 1 -- start start 2024-01-27T01:06:05.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.368+0000 7f7cd454a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc068b00 0x7f7ccc0ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:05.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.368+0000 7f7cd454a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ccc06a660 0x7f7ccc0b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:05.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.368+0000 7f7cd454a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ccc06add0 0x7f7ccc0addc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:05.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.368+0000 7f7cd454a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc0b16f0 con 0x7f7ccc068b00 2024-01-27T01:06:05.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.368+0000 7f7cd454a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc0b0ae0 con 0x7f7ccc06a660 2024-01-27T01:06:05.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.368+0000 7f7cd454a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc0b0dc0 con 0x7f7ccc06add0 2024-01-27T01:06:05.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.368+0000 7f7cd22e6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc068b00 0x7f7ccc0ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:05.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.368+0000 7f7cd22e6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc068b00 0x7f7ccc0ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44616/0 (socket says 172.21.15.46:44616) 2024-01-27T01:06:05.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.368+0000 7f7cd22e6700 1 -- 172.21.15.46:0/2484158346 learned_addr learned my addr 172.21.15.46:0/2484158346 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:05.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.368+0000 7f7cd2ae7700 1 --2- 172.21.15.46:0/2484158346 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ccc06add0 0x7f7ccc0addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:05.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.368+0000 7f7cd1ae5700 1 --2- 172.21.15.46:0/2484158346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ccc06a660 0x7f7ccc0b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:05.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.368+0000 7f7cd22e6700 1 -- 172.21.15.46:0/2484158346 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ccc06add0 msgr2=0x7f7ccc0addc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:05.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.368+0000 7f7cd22e6700 1 --2- 172.21.15.46:0/2484158346 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ccc06add0 0x7f7ccc0addc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.368+0000 7f7cd22e6700 1 -- 172.21.15.46:0/2484158346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ccc06a660 msgr2=0x7f7ccc0b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:05.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.368+0000 7f7cd22e6700 1 --2- 172.21.15.46:0/2484158346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ccc06a660 0x7f7ccc0b1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.369+0000 7f7cd22e6700 1 -- 172.21.15.46:0/2484158346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc4009ae0 con 0x7f7ccc068b00 2024-01-27T01:06:05.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.369+0000 7f7cd22e6700 1 --2- 172.21.15.46:0/2484158346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc068b00 0x7f7ccc0ac970 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800ea00 tx=0x7f7cc8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:05.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.369+0000 7f7cc27fc700 1 -- 172.21.15.46:0/2484158346 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc800c6a0 con 0x7f7ccc068b00 2024-01-27T01:06:05.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.369+0000 7f7cc27fc700 1 -- 172.21.15.46:0/2484158346 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc800c800 con 0x7f7ccc068b00 2024-01-27T01:06:05.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.369+0000 7f7cd454a700 1 -- 172.21.15.46:0/2484158346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ccc0b10a0 con 0x7f7ccc068b00 2024-01-27T01:06:05.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.369+0000 7f7cc27fc700 1 -- 172.21.15.46:0/2484158346 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc800cb80 con 0x7f7ccc068b00 2024-01-27T01:06:05.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.369+0000 7f7cd454a700 1 -- 172.21.15.46:0/2484158346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ccc0abcd0 con 0x7f7ccc068b00 2024-01-27T01:06:05.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.370+0000 7f7cd454a700 1 -- 172.21.15.46:0/2484158346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ccc0b1540 con 0x7f7ccc068b00 2024-01-27T01:06:05.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.370+0000 7f7cc27fc700 1 -- 172.21.15.46:0/2484158346 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 122) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f7cc8013470 con 0x7f7ccc068b00 2024-01-27T01:06:05.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.371+0000 7f7cc27fc700 1 --2- 172.21.15.46:0/2484158346 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7cb0067050 0x7f7cb0069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:05.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.371+0000 7f7cd1ae5700 1 -- 172.21.15.46:0/2484158346 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7cb0067050 msgr2=0x7f7cb0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:05.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.371+0000 7f7cc27fc700 1 -- 172.21.15.46:0/2484158346 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6906+0+0 (secure 0 0 0) 0x7f7cc8085050 con 0x7f7ccc068b00 2024-01-27T01:06:05.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.371+0000 7f7cd1ae5700 1 --2- 172.21.15.46:0/2484158346 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7cb0067050 0x7f7cb0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:06:05.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.376+0000 7f7cc27fc700 1 -- 172.21.15.46:0/2484158346 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7cc8051d00 con 0x7f7ccc068b00 2024-01-27T01:06:05.424 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:05.425+0000 7f86191b01c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:06:05.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.572+0000 7f7cd1ae5700 1 -- 172.21.15.46:0/2484158346 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7cb0067050 msgr2=0x7f7cb0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:05.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.572+0000 7f7cd1ae5700 1 --2- 172.21.15.46:0/2484158346 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7cb0067050 0x7f7cb0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:06:05.637 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:05.636+0000 7f86191b01c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:06:05.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.583+0000 7f7cd454a700 1 -- 172.21.15.46:0/2484158346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7ccc068ed0 con 0x7f7ccc068b00 2024-01-27T01:06:05.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.584+0000 7f7cc27fc700 1 -- 172.21.15.46:0/2484158346 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v210) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7cc8017020 con 0x7f7ccc068b00 2024-01-27T01:06:05.638 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-27T01:06:05.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.586+0000 7f7cd454a700 1 -- 172.21.15.46:0/2484158346 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7cb0067050 msgr2=0x7f7cb0069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:06:05.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.586+0000 7f7cd454a700 1 --2- 172.21.15.46:0/2484158346 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7cb0067050 0x7f7cb0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.587+0000 7f7cd454a700 1 -- 172.21.15.46:0/2484158346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc068b00 msgr2=0x7f7ccc0ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:05.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.587+0000 7f7cd454a700 1 --2- 172.21.15.46:0/2484158346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc068b00 0x7f7ccc0ac970 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800ea00 tx=0x7f7cc8009120 comp rx=0 tx=0).stop 2024-01-27T01:06:05.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.587+0000 7f7cd454a700 1 -- 172.21.15.46:0/2484158346 shutdown_connections 2024-01-27T01:06:05.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.587+0000 7f7cd454a700 1 --2- 172.21.15.46:0/2484158346 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7cb0067050 0x7f7cb0069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.587+0000 7f7cd454a700 1 --2- 172.21.15.46:0/2484158346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc068b00 0x7f7ccc0ac970 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.587+0000 7f7cd454a700 1 --2- 172.21.15.46:0/2484158346 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ccc06add0 0x7f7ccc0addc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.587+0000 7f7cd454a700 1 --2- 172.21.15.46:0/2484158346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ccc06a660 0x7f7ccc0b1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.587+0000 7f7cd454a700 1 -- 172.21.15.46:0/2484158346 >> 172.21.15.46:0/2484158346 conn(0x7f7ccc0ca100 msgr2=0x7f7ccc159aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:05.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.587+0000 7f7cd454a700 1 -- 172.21.15.46:0/2484158346 shutdown_connections 2024-01-27T01:06:05.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.587+0000 7f7cd454a700 1 -- 172.21.15.46:0/2484158346 wait complete. 2024-01-27T01:06:05.641 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-27T01:06:05.713 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:05.713+0000 7f3d64b1e1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:06:05.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.782+0000 7f97cd446700 1 Processor -- start 2024-01-27T01:06:05.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.782+0000 7f97cd446700 1 -- start start 2024-01-27T01:06:05.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.783+0000 7f97cd446700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97c806c450 0x7f97c8157e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:05.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.783+0000 7f97cd446700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c806c8f0 con 0x7f97c8158340 2024-01-27T01:06:05.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.783+0000 7f97cd446700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c81587c0 con 0x7f97c8157a30 2024-01-27T01:06:05.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.783+0000 7f97cd446700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c81598a0 con 0x7f97c806c450 2024-01-27T01:06:05.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.783+0000 7f97c67fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97c806c450 0x7f97c8157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:05.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.783+0000 7f97c67fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97c806c450 0x7f97c8157e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44626/0 (socket says 172.21.15.46:44626) 2024-01-27T01:06:05.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.783+0000 7f97c67fc700 1 -- 172.21.15.46:0/1799791173 learned_addr learned my addr 172.21.15.46:0/1799791173 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:05.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.783+0000 7f97c67fc700 1 -- 172.21.15.46:0/1799791173 >> v1:172.21.15.106:6789/0 conn(0x7f97c8157a30 legacy=0x7f97c8159190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:05.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.783+0000 7f97c67fc700 1 -- 172.21.15.46:0/1799791173 >> v1:172.21.15.46:6789/0 conn(0x7f97c8158340 legacy=0x7f97c80cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:05.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.783+0000 7f97c67fc700 1 -- 172.21.15.46:0/1799791173 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c8158aa0 con 0x7f97c806c450 2024-01-27T01:06:05.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.784+0000 7f97c67fc700 1 --2- 172.21.15.46:0/1799791173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c806c450 0x7f97c8157e00 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0099b0 tx=0x7f97bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae3547e19f3854d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:05.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.784+0000 7f97c5ffb700 1 -- 172.21.15.46:0/1799791173 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc00b860 con 0x7f97c806c450 2024-01-27T01:06:05.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.784+0000 7f97c5ffb700 1 -- 172.21.15.46:0/1799791173 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97bc00b9c0 con 0x7f97c806c450 2024-01-27T01:06:05.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.784+0000 7f97c5ffb700 1 -- 172.21.15.46:0/1799791173 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc00bd80 con 0x7f97c806c450 2024-01-27T01:06:05.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.784+0000 7f97cd446700 1 -- 172.21.15.46:0/1799791173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c806c450 msgr2=0x7f97c8157e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:05.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.784+0000 7f97cd446700 1 --2- 172.21.15.46:0/1799791173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c806c450 0x7f97c8157e00 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0099b0 tx=0x7f97bc009250 comp rx=0 tx=0).stop 2024-01-27T01:06:05.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.785+0000 7f97cd446700 1 -- 172.21.15.46:0/1799791173 shutdown_connections 2024-01-27T01:06:05.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.785+0000 7f97cd446700 1 --2- 172.21.15.46:0/1799791173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c806c450 0x7f97c8157e00 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.785+0000 7f97cd446700 1 -- 172.21.15.46:0/1799791173 >> 172.21.15.46:0/1799791173 conn(0x7f97c80ce570 msgr2=0x7f97c80ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:05.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.785+0000 7f97cd446700 1 -- 172.21.15.46:0/1799791173 shutdown_connections 2024-01-27T01:06:05.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.785+0000 7f97cd446700 1 -- 172.21.15.46:0/1799791173 wait complete. 2024-01-27T01:06:05.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.786+0000 7f97cd446700 1 Processor -- start 2024-01-27T01:06:05.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.787+0000 7f97cd446700 1 -- start start 2024-01-27T01:06:05.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97cd446700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c806c450 0x7f97c80c7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:05.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97cd446700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c8157a30 0x7f97c814ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:05.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97cd446700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97c80c6990 0x7f97c80c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:05.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97cd446700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c80c8450 con 0x7f97c806c450 2024-01-27T01:06:05.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97cd446700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c814f4d0 con 0x7f97c8157a30 2024-01-27T01:06:05.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97cd446700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c80c6e70 con 0x7f97c80c6990 2024-01-27T01:06:05.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97c6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c806c450 0x7f97c80c7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:05.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97c6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c806c450 0x7f97c80c7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44642/0 (socket says 172.21.15.46:44642) 2024-01-27T01:06:05.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97c6ffd700 1 -- 172.21.15.46:0/1389918308 learned_addr learned my addr 172.21.15.46:0/1389918308 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:05.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97c67fc700 1 --2- 172.21.15.46:0/1389918308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c8157a30 0x7f97c814ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:05.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97c77fe700 1 --2- 172.21.15.46:0/1389918308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97c80c6990 0x7f97c80c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:05.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97c6ffd700 1 -- 172.21.15.46:0/1389918308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97c80c6990 msgr2=0x7f97c80c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:05.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97c6ffd700 1 --2- 172.21.15.46:0/1389918308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97c80c6990 0x7f97c80c3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97c6ffd700 1 -- 172.21.15.46:0/1389918308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c8157a30 msgr2=0x7f97c814ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:05.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97c6ffd700 1 --2- 172.21.15.46:0/1389918308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c8157a30 0x7f97c814ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:05.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97c6ffd700 1 -- 172.21.15.46:0/1389918308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc0094f0 con 0x7f97c806c450 2024-01-27T01:06:05.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97c6ffd700 1 --2- 172.21.15.46:0/1389918308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c806c450 0x7f97c80c7f10 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f97b800c590 tx=0x7f97b800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:05.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.788+0000 7f97aeffd700 1 -- 172.21.15.46:0/1389918308 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b8011590 con 0x7f97c806c450 2024-01-27T01:06:05.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.789+0000 7f97aeffd700 1 -- 172.21.15.46:0/1389918308 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b8011710 con 0x7f97c806c450 2024-01-27T01:06:05.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.789+0000 7f97cd446700 1 -- 172.21.15.46:0/1389918308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c80c7150 con 0x7f97c806c450 2024-01-27T01:06:05.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.789+0000 7f97aeffd700 1 -- 172.21.15.46:0/1389918308 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b8011ad0 con 0x7f97c806c450 2024-01-27T01:06:05.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.789+0000 7f97cd446700 1 -- 172.21.15.46:0/1389918308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c80c7710 con 0x7f97c806c450 2024-01-27T01:06:05.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.790+0000 7f97cd446700 1 -- 172.21.15.46:0/1389918308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c8070270 con 0x7f97c806c450 2024-01-27T01:06:05.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.790+0000 7f97aeffd700 1 -- 172.21.15.46:0/1389918308 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 122) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f97b801b3b0 con 0x7f97c806c450 2024-01-27T01:06:05.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.790+0000 7f97aeffd700 1 --2- 172.21.15.46:0/1389918308 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f97b4067050 0x7f97b4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:05.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.791+0000 7f97c67fc700 1 -- 172.21.15.46:0/1389918308 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f97b4067050 msgr2=0x7f97b4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:05.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.791+0000 7f97aeffd700 1 -- 172.21.15.46:0/1389918308 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6906+0+0 (secure 0 0 0) 0x7f97b8083f70 con 0x7f97c806c450 2024-01-27T01:06:05.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.791+0000 7f97c67fc700 1 --2- 172.21.15.46:0/1389918308 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f97b4067050 0x7f97b4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:06:05.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.795+0000 7f97aeffd700 1 -- 172.21.15.46:0/1389918308 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f97b8050c20 con 0x7f97c806c450 2024-01-27T01:06:05.863 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:05.864+0000 7f86191b01c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:06:05.934 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:05.934+0000 7f86191b01c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:06:05.967 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:05.967+0000 7f3d64b1e1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:06:05.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.992+0000 7f97c67fc700 1 -- 172.21.15.46:0/1389918308 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f97b4067050 msgr2=0x7f97b4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:05.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:05.992+0000 7f97c67fc700 1 --2- 172.21.15.46:0/1389918308 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f97b4067050 0x7f97b4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:06:06.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.014+0000 7f97cd446700 1 -- 172.21.15.46:0/1389918308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f97c804c710 con 0x7f97c806c450 2024-01-27T01:06:06.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.015+0000 7f97aeffd700 1 -- 172.21.15.46:0/1389918308 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v210) v1 ==== 100+0+11 (secure 0 0 0) 0x7f97b8017070 con 0x7f97c806c450 2024-01-27T01:06:06.015 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-27T01:06:06.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.017+0000 7f97cd446700 1 -- 172.21.15.46:0/1389918308 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f97b4067050 msgr2=0x7f97b4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:06:06.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.017+0000 7f97cd446700 1 --2- 172.21.15.46:0/1389918308 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f97b4067050 0x7f97b4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.017+0000 7f97cd446700 1 -- 172.21.15.46:0/1389918308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c806c450 msgr2=0x7f97c80c7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:06.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.017+0000 7f97cd446700 1 --2- 172.21.15.46:0/1389918308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c806c450 0x7f97c80c7f10 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f97b800c590 tx=0x7f97b800c8a0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.017+0000 7f97cd446700 1 -- 172.21.15.46:0/1389918308 shutdown_connections 2024-01-27T01:06:06.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.017+0000 7f97cd446700 1 --2- 172.21.15.46:0/1389918308 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f97b4067050 0x7f97b4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.017+0000 7f97cd446700 1 --2- 172.21.15.46:0/1389918308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97c806c450 0x7f97c80c7f10 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.017+0000 7f97cd446700 1 --2- 172.21.15.46:0/1389918308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97c80c6990 0x7f97c80c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.017+0000 7f97cd446700 1 --2- 172.21.15.46:0/1389918308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97c8157a30 0x7f97c814ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.017+0000 7f97cd446700 1 -- 172.21.15.46:0/1389918308 >> 172.21.15.46:0/1389918308 conn(0x7f97c80ce570 msgr2=0x7f97c806e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:06.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.017+0000 7f97cd446700 1 -- 172.21.15.46:0/1389918308 shutdown_connections 2024-01-27T01:06:06.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.017+0000 7f97cd446700 1 -- 172.21.15.46:0/1389918308 wait complete. 2024-01-27T01:06:06.032 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:06:06.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.168+0000 7f7088910700 1 Processor -- start 2024-01-27T01:06:06.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.168+0000 7f7088910700 1 -- start start 2024-01-27T01:06:06.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.170+0000 7f7088910700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f708415d210 0x7f70840cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:06.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.170+0000 7f7088910700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f70840cbff0 0x7f70840ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:06.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.170+0000 7f7088910700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70840cc450 con 0x7f708415ce40 2024-01-27T01:06:06.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.170+0000 7f7088910700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70840cff70 con 0x7f70840cbff0 2024-01-27T01:06:06.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.170+0000 7f7088910700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70840cf400 con 0x7f708415d210 2024-01-27T01:06:06.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.170+0000 7f7082d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f70840cbff0 0x7f70840ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:06.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.170+0000 7f7082d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f70840cbff0 0x7f70840ade30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44650/0 (socket says 172.21.15.46:44650) 2024-01-27T01:06:06.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.170+0000 7f7082d9d700 1 -- 172.21.15.46:0/4191053372 learned_addr learned my addr 172.21.15.46:0/4191053372 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:06.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.170+0000 7f7081d9b700 1 --2- 172.21.15.46:0/4191053372 >> v2:172.21.15.106:3300/0 conn(0x7f708415d210 0x7f70840cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:06.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.171+0000 7f7082d9d700 1 -- 172.21.15.46:0/4191053372 >> v2:172.21.15.106:3300/0 conn(0x7f708415d210 msgr2=0x7f70840cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:06.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.171+0000 7f7082d9d700 1 --2- 172.21.15.46:0/4191053372 >> v2:172.21.15.106:3300/0 conn(0x7f708415d210 0x7f70840cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.171+0000 7f7082d9d700 1 -- 172.21.15.46:0/4191053372 >> v1:172.21.15.106:6789/0 conn(0x7f708415ce40 legacy=0x7f70840d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:06.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.171+0000 7f7082d9d700 1 -- 172.21.15.46:0/4191053372 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70840cf660 con 0x7f70840cbff0 2024-01-27T01:06:06.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.171+0000 7f7082d9d700 1 --2- 172.21.15.46:0/4191053372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70840cbff0 0x7f70840ade30 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f70840895f0 tx=0x7f707c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2e906ce3604557 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:06.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.171+0000 7f708159a700 1 -- 172.21.15.46:0/4191053372 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707c009e70 con 0x7f70840cbff0 2024-01-27T01:06:06.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.171+0000 7f708159a700 1 -- 172.21.15.46:0/4191053372 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707c00b860 con 0x7f70840cbff0 2024-01-27T01:06:06.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.171+0000 7f708159a700 1 -- 172.21.15.46:0/4191053372 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707c00bc00 con 0x7f70840cbff0 2024-01-27T01:06:06.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.172+0000 7f7088910700 1 -- 172.21.15.46:0/4191053372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70840cbff0 msgr2=0x7f70840ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:06.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.172+0000 7f7088910700 1 --2- 172.21.15.46:0/4191053372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70840cbff0 0x7f70840ade30 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f70840895f0 tx=0x7f707c009250 comp rx=0 tx=0).stop 2024-01-27T01:06:06.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.172+0000 7f7088910700 1 -- 172.21.15.46:0/4191053372 shutdown_connections 2024-01-27T01:06:06.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.172+0000 7f7088910700 1 --2- 172.21.15.46:0/4191053372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70840cbff0 0x7f70840ade30 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.172+0000 7f7088910700 1 --2- 172.21.15.46:0/4191053372 >> v2:172.21.15.106:3300/0 conn(0x7f708415d210 0x7f70840cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.172+0000 7f7088910700 1 -- 172.21.15.46:0/4191053372 >> 172.21.15.46:0/4191053372 conn(0x7f708406f840 msgr2=0x7f70840d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:06.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.172+0000 7f7088910700 1 -- 172.21.15.46:0/4191053372 shutdown_connections 2024-01-27T01:06:06.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.172+0000 7f7088910700 1 -- 172.21.15.46:0/4191053372 wait complete. 2024-01-27T01:06:06.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.173+0000 7f7088910700 1 Processor -- start 2024-01-27T01:06:06.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.173+0000 7f7088910700 1 -- start start 2024-01-27T01:06:06.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.174+0000 7f7088910700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70840cbff0 0x7f70840c7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:06.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.174+0000 7f7088910700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f708415ce40 0x7f70841474c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:06.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.174+0000 7f7088910700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f708415d210 0x7f70840b4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:06.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.174+0000 7f7088910700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70840c8cc0 con 0x7f70840cbff0 2024-01-27T01:06:06.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.174+0000 7f7088910700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70840b4e80 con 0x7f708415ce40 2024-01-27T01:06:06.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.174+0000 7f7088910700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7084147bb0 con 0x7f708415d210 2024-01-27T01:06:06.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.174+0000 7f708259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70840cbff0 0x7f70840c7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:06.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.174+0000 7f708259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70840cbff0 0x7f70840c7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44660/0 (socket says 172.21.15.46:44660) 2024-01-27T01:06:06.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.174+0000 7f708259c700 1 -- 172.21.15.46:0/472069481 learned_addr learned my addr 172.21.15.46:0/472069481 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:06.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.174+0000 7f7081d9b700 1 --2- 172.21.15.46:0/472069481 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f708415ce40 0x7f70841474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:06.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.174+0000 7f7082d9d700 1 --2- 172.21.15.46:0/472069481 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f708415d210 0x7f70840b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:06.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.175+0000 7f708259c700 1 -- 172.21.15.46:0/472069481 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f708415d210 msgr2=0x7f70840b4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:06.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.175+0000 7f708259c700 1 --2- 172.21.15.46:0/472069481 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f708415d210 0x7f70840b4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.175+0000 7f708259c700 1 -- 172.21.15.46:0/472069481 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f708415ce40 msgr2=0x7f70841474c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:06.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.175+0000 7f708259c700 1 --2- 172.21.15.46:0/472069481 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f708415ce40 0x7f70841474c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.175+0000 7f708259c700 1 -- 172.21.15.46:0/472069481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707c0094f0 con 0x7f70840cbff0 2024-01-27T01:06:06.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.175+0000 7f708259c700 1 --2- 172.21.15.46:0/472069481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70840cbff0 0x7f70840c7a60 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f707400cdb0 tx=0x7f7074001be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:06.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.175+0000 7f70727fc700 1 -- 172.21.15.46:0/472069481 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707400aac0 con 0x7f70840cbff0 2024-01-27T01:06:06.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.175+0000 7f70727fc700 1 -- 172.21.15.46:0/472069481 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707400ac20 con 0x7f70840cbff0 2024-01-27T01:06:06.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.175+0000 7f7088910700 1 -- 172.21.15.46:0/472069481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70840c80b0 con 0x7f70840cbff0 2024-01-27T01:06:06.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.175+0000 7f70727fc700 1 -- 172.21.15.46:0/472069481 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7074012420 con 0x7f70840cbff0 2024-01-27T01:06:06.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.175+0000 7f7088910700 1 -- 172.21.15.46:0/472069481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70840c8670 con 0x7f70840cbff0 2024-01-27T01:06:06.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.176+0000 7f7088910700 1 -- 172.21.15.46:0/472069481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f708404c710 con 0x7f70840cbff0 2024-01-27T01:06:06.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.177+0000 7f70727fc700 1 -- 172.21.15.46:0/472069481 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 122) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f7074012810 con 0x7f70840cbff0 2024-01-27T01:06:06.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.177+0000 7f70727fc700 1 --2- 172.21.15.46:0/472069481 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7064067050 0x7f7064069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:06.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.177+0000 7f70727fc700 1 -- 172.21.15.46:0/472069481 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6906+0+0 (secure 0 0 0) 0x7f7074085040 con 0x7f70840cbff0 2024-01-27T01:06:06.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.177+0000 7f7081d9b700 1 -- 172.21.15.46:0/472069481 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7064067050 msgr2=0x7f7064069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:06.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.177+0000 7f7081d9b700 1 --2- 172.21.15.46:0/472069481 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7064067050 0x7f7064069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:06:06.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.182+0000 7f70727fc700 1 -- 172.21.15.46:0/472069481 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7074051cf0 con 0x7f70840cbff0 2024-01-27T01:06:06.233 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:06.233+0000 7f3d64b1e1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:06:06.315 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:06.315+0000 7f3d64b1e1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:06:06.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.378+0000 7f7081d9b700 1 -- 172.21.15.46:0/472069481 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7064067050 msgr2=0x7f7064069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:06.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.378+0000 7f7081d9b700 1 --2- 172.21.15.46:0/472069481 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7064067050 0x7f7064069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:06:06.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.441+0000 7f7088910700 1 -- 172.21.15.46:0/472069481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70840cc450 con 0x7f70840cbff0 2024-01-27T01:06:06.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.442+0000 7f70727fc700 1 -- 172.21.15.46:0/472069481 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4612 (secure 0 0 0) 0x7f70840cc450 con 0x7f70840cbff0 2024-01-27T01:06:06.442 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:06.442 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":107,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8891,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/975115379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":975115379},{"type":"v1","addr":"172.21.15.106:6833","nonce":975115379}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10036,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3860120510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3860120510},{"type":"v1","addr":"172.21.15.46:6834","nonce":3860120510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10045,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3603549739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3603549739},{"type":"v1","addr":"172.21.15.106:6835","nonce":3603549739}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:03:44.141242+0000","modified":"2024-01-27T01:03:47.727201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10392":{"gid":10392,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/976738644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":976738644},{"type":"v1","addr":"172.21.15.46:6835","nonce":976738644}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2024-01-27T01:06:06.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.444+0000 7f7088910700 1 -- 172.21.15.46:0/472069481 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7064067050 msgr2=0x7f7064069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:06:06.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.444+0000 7f7088910700 1 --2- 172.21.15.46:0/472069481 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7064067050 0x7f7064069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.444+0000 7f7088910700 1 -- 172.21.15.46:0/472069481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70840cbff0 msgr2=0x7f70840c7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:06.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.444+0000 7f7088910700 1 --2- 172.21.15.46:0/472069481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70840cbff0 0x7f70840c7a60 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f707400cdb0 tx=0x7f7074001be0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.444+0000 7f7088910700 1 -- 172.21.15.46:0/472069481 shutdown_connections 2024-01-27T01:06:06.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.444+0000 7f7088910700 1 --2- 172.21.15.46:0/472069481 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7064067050 0x7f7064069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.444+0000 7f7088910700 1 --2- 172.21.15.46:0/472069481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70840cbff0 0x7f70840c7a60 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.444+0000 7f7088910700 1 --2- 172.21.15.46:0/472069481 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f708415d210 0x7f70840b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.444+0000 7f7088910700 1 --2- 172.21.15.46:0/472069481 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f708415ce40 0x7f70841474c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.444+0000 7f7088910700 1 -- 172.21.15.46:0/472069481 >> 172.21.15.46:0/472069481 conn(0x7f708406f840 msgr2=0x7f708415d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:06.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.444+0000 7f7088910700 1 -- 172.21.15.46:0/472069481 shutdown_connections 2024-01-27T01:06:06.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.444+0000 7f7088910700 1 -- 172.21.15.46:0/472069481 wait complete. 2024-01-27T01:06:06.446 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 107 2024-01-27T01:06:06.458 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:06:06.535 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:06.536+0000 7f86191b01c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:06:06.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.593+0000 7f24f3284700 1 Processor -- start 2024-01-27T01:06:06.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.593+0000 7f24f3284700 1 -- start start 2024-01-27T01:06:06.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.594+0000 7f24f3284700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f24ec159b30 0x7f24ec0b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:06.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.594+0000 7f24f3284700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f24ec0b10f0 0x7f24ec0a9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:06.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.594+0000 7f24f3284700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ec06f940 con 0x7f24ec159760 2024-01-27T01:06:06.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.594+0000 7f24f3284700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24ec0b1530 con 0x7f24ec159b30 2024-01-27T01:06:06.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.594+0000 7f24f3284700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24ec0b3230 con 0x7f24ec0b10f0 2024-01-27T01:06:06.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.594+0000 7f24f081f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f24ec159b30 0x7f24ec0b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.594+0000 7f24f081f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f24ec159b30 0x7f24ec0b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44668/0 (socket says 172.21.15.46:44668) 2024-01-27T01:06:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.594+0000 7f24f081f700 1 -- 172.21.15.46:0/613469404 learned_addr learned my addr 172.21.15.46:0/613469404 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.594+0000 7f24f1821700 1 --2- 172.21.15.46:0/613469404 >> v2:172.21.15.106:3300/0 conn(0x7f24ec0b10f0 0x7f24ec0a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.595+0000 7f24f081f700 1 -- 172.21.15.46:0/613469404 >> v2:172.21.15.106:3300/0 conn(0x7f24ec0b10f0 msgr2=0x7f24ec0a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.595+0000 7f24f081f700 1 --2- 172.21.15.46:0/613469404 >> v2:172.21.15.106:3300/0 conn(0x7f24ec0b10f0 0x7f24ec0a9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.595+0000 7f24f081f700 1 -- 172.21.15.46:0/613469404 >> v1:172.21.15.106:6789/0 conn(0x7f24ec159760 legacy=0x7f24ec0b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.595+0000 7f24f081f700 1 -- 172.21.15.46:0/613469404 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec06ed30 con 0x7f24ec159b30 2024-01-27T01:06:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.595+0000 7f24e3fff700 1 -- 172.21.15.46:0/613469404 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280100416 0 0) 0x7f24ec06f940 con 0x7f24ec159760 2024-01-27T01:06:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.595+0000 7f24f081f700 1 --2- 172.21.15.46:0/613469404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24ec159b30 0x7f24ec0b0bb0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f24dc0099b0 tx=0x7f24dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b610c12133b5296 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.595+0000 7f24e3fff700 1 -- 172.21.15.46:0/613469404 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc009ea0 con 0x7f24ec159b30 2024-01-27T01:06:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.595+0000 7f24e3fff700 1 -- 172.21.15.46:0/613469404 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24dc00b820 con 0x7f24ec159b30 2024-01-27T01:06:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.595+0000 7f24e3fff700 1 -- 172.21.15.46:0/613469404 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc00bbc0 con 0x7f24ec159b30 2024-01-27T01:06:06.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.596+0000 7f24f3284700 1 -- 172.21.15.46:0/613469404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24ec159b30 msgr2=0x7f24ec0b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:06.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.596+0000 7f24f3284700 1 --2- 172.21.15.46:0/613469404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24ec159b30 0x7f24ec0b0bb0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f24dc0099b0 tx=0x7f24dc009250 comp rx=0 tx=0).stop 2024-01-27T01:06:06.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.596+0000 7f24f3284700 1 -- 172.21.15.46:0/613469404 shutdown_connections 2024-01-27T01:06:06.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.596+0000 7f24f3284700 1 --2- 172.21.15.46:0/613469404 >> v2:172.21.15.106:3300/0 conn(0x7f24ec0b10f0 0x7f24ec0a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.596+0000 7f24f3284700 1 --2- 172.21.15.46:0/613469404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24ec159b30 0x7f24ec0b0bb0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.596+0000 7f24f3284700 1 -- 172.21.15.46:0/613469404 >> 172.21.15.46:0/613469404 conn(0x7f24ec069510 msgr2=0x7f24ec0aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:06.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.596+0000 7f24f3284700 1 -- 172.21.15.46:0/613469404 shutdown_connections 2024-01-27T01:06:06.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.596+0000 7f24f3284700 1 -- 172.21.15.46:0/613469404 wait complete. 2024-01-27T01:06:06.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.597+0000 7f24f3284700 1 Processor -- start 2024-01-27T01:06:06.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.597+0000 7f24f3284700 1 -- start start 2024-01-27T01:06:06.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.598+0000 7f24f3284700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f24ec0b10f0 0x7f24ec153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:06.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.598+0000 7f24f3284700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24ec159760 0x7f24ec154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:06.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.598+0000 7f24f3284700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24ec159b30 0x7f24ec1699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:06.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.598+0000 7f24f3284700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec13b950 con 0x7f24ec159760 2024-01-27T01:06:06.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.598+0000 7f24f3284700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec154640 con 0x7f24ec159b30 2024-01-27T01:06:06.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.598+0000 7f24f3284700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec16ac40 con 0x7f24ec0b10f0 2024-01-27T01:06:06.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.598+0000 7f24f081f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24ec159760 0x7f24ec154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.598+0000 7f24f081f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24ec159760 0x7f24ec154100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44682/0 (socket says 172.21.15.46:44682) 2024-01-27T01:06:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.598+0000 7f24f081f700 1 -- 172.21.15.46:0/2384054286 learned_addr learned my addr 172.21.15.46:0/2384054286 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.598+0000 7f24f1020700 1 --2- 172.21.15.46:0/2384054286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f24ec0b10f0 0x7f24ec153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.598+0000 7f24f1821700 1 --2- 172.21.15.46:0/2384054286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24ec159b30 0x7f24ec1699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.599+0000 7f24f081f700 1 -- 172.21.15.46:0/2384054286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f24ec0b10f0 msgr2=0x7f24ec153bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.599+0000 7f24f081f700 1 --2- 172.21.15.46:0/2384054286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f24ec0b10f0 0x7f24ec153bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.599+0000 7f24f081f700 1 -- 172.21.15.46:0/2384054286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24ec159b30 msgr2=0x7f24ec1699d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.599+0000 7f24f081f700 1 --2- 172.21.15.46:0/2384054286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24ec159b30 0x7f24ec1699d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.599+0000 7f24f081f700 1 -- 172.21.15.46:0/2384054286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc0094f0 con 0x7f24ec159760 2024-01-27T01:06:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.599+0000 7f24f1821700 1 --2- 172.21.15.46:0/2384054286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24ec159b30 0x7f24ec1699d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.599+0000 7f24f081f700 1 --2- 172.21.15.46:0/2384054286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24ec159760 0x7f24ec154100 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f24ec0ce460 tx=0x7f24dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.599+0000 7f24e17fa700 1 -- 172.21.15.46:0/2384054286 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc000d10 con 0x7f24ec159760 2024-01-27T01:06:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.599+0000 7f24e17fa700 1 -- 172.21.15.46:0/2384054286 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24dc00bba0 con 0x7f24ec159760 2024-01-27T01:06:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.599+0000 7f24f3284700 1 -- 172.21.15.46:0/2384054286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24ec154880 con 0x7f24ec159760 2024-01-27T01:06:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.599+0000 7f24e17fa700 1 -- 172.21.15.46:0/2384054286 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc01a6a0 con 0x7f24ec159760 2024-01-27T01:06:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.599+0000 7f24f3284700 1 -- 172.21.15.46:0/2384054286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24ec16b240 con 0x7f24ec159760 2024-01-27T01:06:06.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.600+0000 7f24f3284700 1 -- 172.21.15.46:0/2384054286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24ec16b3d0 con 0x7f24ec159760 2024-01-27T01:06:06.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.601+0000 7f24e17fa700 1 -- 172.21.15.46:0/2384054286 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 122) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f24dc01aa50 con 0x7f24ec159760 2024-01-27T01:06:06.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.601+0000 7f24e17fa700 1 --2- 172.21.15.46:0/2384054286 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f24d0067050 0x7f24d0069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:06.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.602+0000 7f24f1020700 1 -- 172.21.15.46:0/2384054286 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f24d0067050 msgr2=0x7f24d0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:06.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.602+0000 7f24e17fa700 1 -- 172.21.15.46:0/2384054286 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6906+0+0 (secure 0 0 0) 0x7f24dc086380 con 0x7f24ec159760 2024-01-27T01:06:06.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.602+0000 7f24f1020700 1 --2- 172.21.15.46:0/2384054286 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f24d0067050 0x7f24d0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:06:06.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.606+0000 7f24e17fa700 1 -- 172.21.15.46:0/2384054286 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f24dc0530e0 con 0x7f24ec159760 2024-01-27T01:06:06.609 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:06.610+0000 7f86191b01c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:06:06.733 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:06.733+0000 7f86191b01c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:06:06.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.803+0000 7f24f1020700 1 -- 172.21.15.46:0/2384054286 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f24d0067050 msgr2=0x7f24d0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:06.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.803+0000 7f24f1020700 1 --2- 172.21.15.46:0/2384054286 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f24d0067050 0x7f24d0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:06:06.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.817+0000 7f24f3284700 1 -- 172.21.15.46:0/2384054286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24ec04c710 con 0x7f24ec159760 2024-01-27T01:06:06.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.818+0000 7f24e17fa700 1 -- 172.21.15.46:0/2384054286 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v210) v1 ==== 74+0+16934 (secure 0 0 0) 0x7f24dc056700 con 0x7f24ec159760 2024-01-27T01:06:06.819 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:06.819 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":210,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:06:04.711162+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:03:42.648202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-01-27T01:03:43.106225+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"210","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":210,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.1b","mappings":[{"from":3,"to":0}]},{"pgid":"26.31","mappings":[{"from":1,"to":0}]},{"pgid":"26.3a","mappings":[{"from":1,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/1813035004":"2024-01-28T01:03:58.706905+0000","172.21.15.106:0/2234269173":"2024-01-28T01:03:58.706905+0000","172.21.15.106:6837/62434":"2024-01-28T01:03:58.706905+0000","172.21.15.106:6836/62434":"2024-01-28T01:03:58.706905+0000","172.21.15.106:0/3361483200":"2024-01-28T01:03:58.706905+0000","172.21.15.106:0/1169106698":"2024-01-28T01:03:58.706905+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":26,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:06:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.821+0000 7f24f3284700 1 -- 172.21.15.46:0/2384054286 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f24d0067050 msgr2=0x7f24d0069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:06:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.821+0000 7f24f3284700 1 --2- 172.21.15.46:0/2384054286 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f24d0067050 0x7f24d0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.821+0000 7f24f3284700 1 -- 172.21.15.46:0/2384054286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24ec159760 msgr2=0x7f24ec154100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.821+0000 7f24f3284700 1 --2- 172.21.15.46:0/2384054286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24ec159760 0x7f24ec154100 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f24ec0ce460 tx=0x7f24dc017000 comp rx=0 tx=0).stop 2024-01-27T01:06:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.821+0000 7f24f3284700 1 -- 172.21.15.46:0/2384054286 shutdown_connections 2024-01-27T01:06:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.821+0000 7f24f3284700 1 --2- 172.21.15.46:0/2384054286 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f24d0067050 0x7f24d0069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.821+0000 7f24f3284700 1 --2- 172.21.15.46:0/2384054286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24ec159b30 0x7f24ec1699d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.821+0000 7f24f3284700 1 --2- 172.21.15.46:0/2384054286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f24ec0b10f0 0x7f24ec153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.821+0000 7f24f3284700 1 --2- 172.21.15.46:0/2384054286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24ec159760 0x7f24ec154100 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.821+0000 7f24f3284700 1 -- 172.21.15.46:0/2384054286 >> 172.21.15.46:0/2384054286 conn(0x7f24ec069510 msgr2=0x7f24ec1519b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.821+0000 7f24f3284700 1 -- 172.21.15.46:0/2384054286 shutdown_connections 2024-01-27T01:06:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.821+0000 7f24f3284700 1 -- 172.21.15.46:0/2384054286 wait complete. 2024-01-27T01:06:06.837 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-27T01:06:06.837 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:06:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.972+0000 7f4b7f4d8700 1 Processor -- start 2024-01-27T01:06:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.972+0000 7f4b7f4d8700 1 -- start start 2024-01-27T01:06:06.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.973+0000 7f4b7f4d8700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4b7815ce40 0x7f4b780d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:06.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.973+0000 7f4b7f4d8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b780cff70 con 0x7f4b7815d210 2024-01-27T01:06:06.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.973+0000 7f4b7f4d8700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b780cf400 con 0x7f4b780cc1c0 2024-01-27T01:06:06.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.973+0000 7f4b7f4d8700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b780cf660 con 0x7f4b7815ce40 2024-01-27T01:06:06.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.973+0000 7f4b7ca73700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4b7815d210 0x7f4b780cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57984/0 (socket says 172.21.15.46:57984) 2024-01-27T01:06:06.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.973+0000 7f4b7ca73700 1 -- 172.21.15.46:0/3843170728 learned_addr learned my addr 172.21.15.46:0/3843170728 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:06.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.974+0000 7f4b7d274700 1 --2- 172.21.15.46:0/3843170728 >> v2:172.21.15.106:3300/0 conn(0x7f4b7815ce40 0x7f4b780d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:06.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.974+0000 7f4b77fff700 1 -- 172.21.15.46:0/3843170728 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959367700 0 0) 0x7f4b780cff70 con 0x7f4b7815d210 2024-01-27T01:06:06.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.974+0000 7f4b77fff700 1 -- 172.21.15.46:0/3843170728 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b60003570 con 0x7f4b7815d210 2024-01-27T01:06:06.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.974+0000 7f4b7d274700 1 -- 172.21.15.46:0/3843170728 >> v1:172.21.15.106:6789/0 conn(0x7f4b780cc1c0 legacy=0x7f4b780ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:06.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.974+0000 7f4b7d274700 1 -- 172.21.15.46:0/3843170728 >> v1:172.21.15.46:6789/0 conn(0x7f4b7815d210 legacy=0x7f4b780cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:06.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.974+0000 7f4b77fff700 1 -- 172.21.15.46:0/3843170728 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96679240 0 0) 0x7f4b780cf400 con 0x7f4b780cc1c0 2024-01-27T01:06:06.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.974+0000 7f4b7d274700 1 -- 172.21.15.46:0/3843170728 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b780d0500 con 0x7f4b7815ce40 2024-01-27T01:06:06.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.974+0000 7f4b7d274700 1 --2- 172.21.15.46:0/3843170728 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b7815ce40 0x7f4b780d0e50 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009eb0 tx=0x7f4b68009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2604440f2c1003a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:06.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.974+0000 7f4b77fff700 1 -- 172.21.15.46:0/3843170728 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6800e070 con 0x7f4b7815ce40 2024-01-27T01:06:06.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.974+0000 7f4b77fff700 1 -- 172.21.15.46:0/3843170728 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6800bc30 con 0x7f4b7815ce40 2024-01-27T01:06:06.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.974+0000 7f4b77fff700 1 -- 172.21.15.46:0/3843170728 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68012430 con 0x7f4b7815ce40 2024-01-27T01:06:06.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.975+0000 7f4b7f4d8700 1 -- 172.21.15.46:0/3843170728 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b7815ce40 msgr2=0x7f4b780d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:06.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.975+0000 7f4b7f4d8700 1 --2- 172.21.15.46:0/3843170728 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b7815ce40 0x7f4b780d0e50 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009eb0 tx=0x7f4b68009780 comp rx=0 tx=0).stop 2024-01-27T01:06:06.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.975+0000 7f4b7f4d8700 1 -- 172.21.15.46:0/3843170728 shutdown_connections 2024-01-27T01:06:06.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.975+0000 7f4b7f4d8700 1 --2- 172.21.15.46:0/3843170728 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b7815ce40 0x7f4b780d0e50 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.975+0000 7f4b7f4d8700 1 -- 172.21.15.46:0/3843170728 >> 172.21.15.46:0/3843170728 conn(0x7f4b7806f840 msgr2=0x7f4b780d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:06.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.975+0000 7f4b7f4d8700 1 -- 172.21.15.46:0/3843170728 shutdown_connections 2024-01-27T01:06:06.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.975+0000 7f4b7f4d8700 1 -- 172.21.15.46:0/3843170728 wait complete. 2024-01-27T01:06:06.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.976+0000 7f4b7f4d8700 1 Processor -- start 2024-01-27T01:06:06.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.976+0000 7f4b7f4d8700 1 -- start start 2024-01-27T01:06:06.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.977+0000 7f4b7f4d8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b780cc1c0 0x7f4b781558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:06.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.977+0000 7f4b7f4d8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b7815ce40 0x7f4b780c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:06.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.977+0000 7f4b7f4d8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b7815d210 0x7f4b780c6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:06.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.977+0000 7f4b7f4d8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b780c4120 con 0x7f4b7815d210 2024-01-27T01:06:06.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.977+0000 7f4b7f4d8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b780c4400 con 0x7f4b780cc1c0 2024-01-27T01:06:06.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.977+0000 7f4b7f4d8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b780c46e0 con 0x7f4b7815ce40 2024-01-27T01:06:06.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.978+0000 7f4b7da75700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b7815d210 0x7f4b780c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:06.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.978+0000 7f4b7da75700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b7815d210 0x7f4b780c6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44684/0 (socket says 172.21.15.46:44684) 2024-01-27T01:06:06.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.978+0000 7f4b7da75700 1 -- 172.21.15.46:0/2391645116 learned_addr learned my addr 172.21.15.46:0/2391645116 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:06.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.978+0000 7f4b7d274700 1 --2- 172.21.15.46:0/2391645116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b780cc1c0 0x7f4b781558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:06.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.978+0000 7f4b7ca73700 1 --2- 172.21.15.46:0/2391645116 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b7815ce40 0x7f4b780c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:06.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.978+0000 7f4b7da75700 1 -- 172.21.15.46:0/2391645116 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b7815ce40 msgr2=0x7f4b780c79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:06.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.978+0000 7f4b7da75700 1 --2- 172.21.15.46:0/2391645116 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b7815ce40 0x7f4b780c79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.978+0000 7f4b7da75700 1 -- 172.21.15.46:0/2391645116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b780cc1c0 msgr2=0x7f4b781558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:06.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.978+0000 7f4b7da75700 1 --2- 172.21.15.46:0/2391645116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b780cc1c0 0x7f4b781558b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:06.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.978+0000 7f4b7da75700 1 -- 172.21.15.46:0/2391645116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b68009a20 con 0x7f4b7815d210 2024-01-27T01:06:06.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.978+0000 7f4b7da75700 1 --2- 172.21.15.46:0/2391645116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b7815d210 0x7f4b780c6370 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f4b700011e0 tx=0x7f4b70007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:06.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.978+0000 7f4b757fa700 1 -- 172.21.15.46:0/2391645116 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b70011590 con 0x7f4b7815d210 2024-01-27T01:06:06.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.978+0000 7f4b757fa700 1 -- 172.21.15.46:0/2391645116 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b700116f0 con 0x7f4b7815d210 2024-01-27T01:06:06.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.978+0000 7f4b7f4d8700 1 -- 172.21.15.46:0/2391645116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b780c49c0 con 0x7f4b7815d210 2024-01-27T01:06:06.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.978+0000 7f4b757fa700 1 -- 172.21.15.46:0/2391645116 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b70011ab0 con 0x7f4b7815d210 2024-01-27T01:06:06.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.978+0000 7f4b7f4d8700 1 -- 172.21.15.46:0/2391645116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b78155df0 con 0x7f4b7815d210 2024-01-27T01:06:06.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.979+0000 7f4b7f4d8700 1 -- 172.21.15.46:0/2391645116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b7804c710 con 0x7f4b7815d210 2024-01-27T01:06:06.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.980+0000 7f4b757fa700 1 -- 172.21.15.46:0/2391645116 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 122) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f4b7001b3b0 con 0x7f4b7815d210 2024-01-27T01:06:06.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.980+0000 7f4b757fa700 1 --2- 172.21.15.46:0/2391645116 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4b64067050 0x7f4b64069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:06.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.981+0000 7f4b7d274700 1 -- 172.21.15.46:0/2391645116 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4b64067050 msgr2=0x7f4b64069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:06.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.981+0000 7f4b757fa700 1 -- 172.21.15.46:0/2391645116 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6906+0+0 (secure 0 0 0) 0x7f4b70083fa0 con 0x7f4b7815d210 2024-01-27T01:06:06.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.981+0000 7f4b7d274700 1 --2- 172.21.15.46:0/2391645116 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4b64067050 0x7f4b64069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:06:06.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:06.985+0000 7f4b757fa700 1 -- 172.21.15.46:0/2391645116 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4b70050bb0 con 0x7f4b7815d210 2024-01-27T01:06:07.035 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:07.035+0000 7f3d64b1e1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:06:07.121 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:07.121+0000 7f3d64b1e1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:06:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.182+0000 7f4b7d274700 1 -- 172.21.15.46:0/2391645116 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4b64067050 msgr2=0x7f4b64069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.182+0000 7f4b7d274700 1 --2- 172.21.15.46:0/2391645116 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4b64067050 0x7f4b64069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:06:07.235 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:07.235+0000 7f86191b01c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:06:07.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.245+0000 7f4b7f4d8700 1 -- 172.21.15.46:0/2391645116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b7815b000 con 0x7f4b7815d210 2024-01-27T01:06:07.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.245+0000 7f4b757fa700 1 -- 172.21.15.46:0/2391645116 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4612 (secure 0 0 0) 0x7f4b70017070 con 0x7f4b7815d210 2024-01-27T01:06:07.245 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:07.246 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":107,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8891,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/975115379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":975115379},{"type":"v1","addr":"172.21.15.106:6833","nonce":975115379}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10036,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3860120510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3860120510},{"type":"v1","addr":"172.21.15.46:6834","nonce":3860120510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10045,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3603549739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3603549739},{"type":"v1","addr":"172.21.15.106:6835","nonce":3603549739}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:03:44.141242+0000","modified":"2024-01-27T01:03:47.727201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10392":{"gid":10392,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/976738644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":976738644},{"type":"v1","addr":"172.21.15.46:6835","nonce":976738644}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2024-01-27T01:06:07.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.248+0000 7f4b7f4d8700 1 -- 172.21.15.46:0/2391645116 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4b64067050 msgr2=0x7f4b64069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:06:07.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.248+0000 7f4b7f4d8700 1 --2- 172.21.15.46:0/2391645116 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4b64067050 0x7f4b64069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.248+0000 7f4b7f4d8700 1 -- 172.21.15.46:0/2391645116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b7815d210 msgr2=0x7f4b780c6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:07.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.248+0000 7f4b7f4d8700 1 --2- 172.21.15.46:0/2391645116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b7815d210 0x7f4b780c6370 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f4b700011e0 tx=0x7f4b70007d10 comp rx=0 tx=0).stop 2024-01-27T01:06:07.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.248+0000 7f4b7f4d8700 1 -- 172.21.15.46:0/2391645116 shutdown_connections 2024-01-27T01:06:07.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.248+0000 7f4b7f4d8700 1 --2- 172.21.15.46:0/2391645116 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4b64067050 0x7f4b64069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.248+0000 7f4b7f4d8700 1 --2- 172.21.15.46:0/2391645116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b7815d210 0x7f4b780c6370 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.248+0000 7f4b7f4d8700 1 --2- 172.21.15.46:0/2391645116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4b780cc1c0 0x7f4b781558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.248+0000 7f4b7f4d8700 1 --2- 172.21.15.46:0/2391645116 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4b7815ce40 0x7f4b780c79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.248+0000 7f4b7f4d8700 1 -- 172.21.15.46:0/2391645116 >> 172.21.15.46:0/2391645116 conn(0x7f4b7806f840 msgr2=0x7f4b780b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:07.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.248+0000 7f4b7f4d8700 1 -- 172.21.15.46:0/2391645116 shutdown_connections 2024-01-27T01:06:07.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.248+0000 7f4b7f4d8700 1 -- 172.21.15.46:0/2391645116 wait complete. 2024-01-27T01:06:07.250 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 107 2024-01-27T01:06:07.264 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:06:07.267 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:07.266+0000 7f3d64b1e1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:06:07.383 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:07.383+0000 7f86191b01c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:06:07.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.397+0000 7fe6d6027700 1 Processor -- start 2024-01-27T01:06:07.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.397+0000 7fe6d6027700 1 -- start start 2024-01-27T01:06:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.398+0000 7fe6d6027700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe6d015ce40 0x7fe6d00d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.398+0000 7fe6d6027700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe6d00cc1c0 0x7fe6d00ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.398+0000 7fe6d6027700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d00cff70 con 0x7fe6d015d210 2024-01-27T01:06:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.398+0000 7fe6d6027700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d00cf400 con 0x7fe6d00cc1c0 2024-01-27T01:06:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.398+0000 7fe6d6027700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d00cf660 con 0x7fe6d015ce40 2024-01-27T01:06:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.398+0000 7fe6cffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe6d00cc1c0 0x7fe6d00ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.399+0000 7fe6cffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe6d00cc1c0 0x7fe6d00ade30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44700/0 (socket says 172.21.15.46:44700) 2024-01-27T01:06:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.399+0000 7fe6cf7fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe6d015ce40 0x7fe6d00d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.399+0000 7fe6cffff700 1 -- 172.21.15.46:0/456699066 learned_addr learned my addr 172.21.15.46:0/456699066 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.399+0000 7fe6ceffd700 1 --1- 172.21.15.46:0/456699066 >> v1:172.21.15.106:6789/0 conn(0x7fe6d015d210 0x7fe6d00cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:39322/0 (socket says 172.21.15.46:39322) 2024-01-27T01:06:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.399+0000 7fe6cffff700 1 -- 172.21.15.46:0/456699066 >> v2:172.21.15.106:3300/0 conn(0x7fe6d015ce40 msgr2=0x7fe6d00d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.399+0000 7fe6cffff700 1 --2- 172.21.15.46:0/456699066 >> v2:172.21.15.106:3300/0 conn(0x7fe6d015ce40 0x7fe6d00d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.399+0000 7fe6cffff700 1 -- 172.21.15.46:0/456699066 >> v1:172.21.15.106:6789/0 conn(0x7fe6d015d210 legacy=0x7fe6d00cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.399+0000 7fe6cffff700 1 -- 172.21.15.46:0/456699066 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d00d0500 con 0x7fe6d00cc1c0 2024-01-27T01:06:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.399+0000 7fe6cf7fe700 1 --2- 172.21.15.46:0/456699066 >> v2:172.21.15.106:3300/0 conn(0x7fe6d015ce40 0x7fe6d00d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.399+0000 7fe6cffff700 1 --2- 172.21.15.46:0/456699066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d00cc1c0 0x7fe6d00ade30 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0088e90 tx=0x7fe6c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=774e8de9af558644 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.399+0000 7fe6ce7fc700 1 -- 172.21.15.46:0/456699066 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8009e70 con 0x7fe6d00cc1c0 2024-01-27T01:06:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.399+0000 7fe6ce7fc700 1 -- 172.21.15.46:0/456699066 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c800b860 con 0x7fe6d00cc1c0 2024-01-27T01:06:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.399+0000 7fe6ce7fc700 1 -- 172.21.15.46:0/456699066 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c800bc00 con 0x7fe6d00cc1c0 2024-01-27T01:06:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.400+0000 7fe6d6027700 1 -- 172.21.15.46:0/456699066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d00cc1c0 msgr2=0x7fe6d00ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.400+0000 7fe6d6027700 1 --2- 172.21.15.46:0/456699066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d00cc1c0 0x7fe6d00ade30 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0088e90 tx=0x7fe6c8009250 comp rx=0 tx=0).stop 2024-01-27T01:06:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.400+0000 7fe6d6027700 1 -- 172.21.15.46:0/456699066 shutdown_connections 2024-01-27T01:06:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.400+0000 7fe6d6027700 1 --2- 172.21.15.46:0/456699066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d00cc1c0 0x7fe6d00ade30 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.400+0000 7fe6d6027700 1 --2- 172.21.15.46:0/456699066 >> v2:172.21.15.106:3300/0 conn(0x7fe6d015ce40 0x7fe6d00d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.400+0000 7fe6d6027700 1 -- 172.21.15.46:0/456699066 >> 172.21.15.46:0/456699066 conn(0x7fe6d006f840 msgr2=0x7fe6d00d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.400+0000 7fe6d6027700 1 -- 172.21.15.46:0/456699066 shutdown_connections 2024-01-27T01:06:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.400+0000 7fe6d6027700 1 -- 172.21.15.46:0/456699066 wait complete. 2024-01-27T01:06:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.401+0000 7fe6d6027700 1 Processor -- start 2024-01-27T01:06:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.402+0000 7fe6d6027700 1 -- start start 2024-01-27T01:06:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.402+0000 7fe6d6027700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6d00cc1c0 0x7fe6d015bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.402+0000 7fe6d6027700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d015ce40 0x7fe6d015b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.402+0000 7fe6ceffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d015ce40 0x7fe6d015b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.402+0000 7fe6d6027700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6d015d210 0x7fe6d00c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6cf7fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6d00cc1c0 0x7fe6d015bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6ceffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d015ce40 0x7fe6d015b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44714/0 (socket says 172.21.15.46:44714) 2024-01-27T01:06:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6ceffd700 1 -- 172.21.15.46:0/3594682029 learned_addr learned my addr 172.21.15.46:0/3594682029 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6d6027700 1 -- 172.21.15.46:0/3594682029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d015bab0 con 0x7fe6d015ce40 2024-01-27T01:06:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6d6027700 1 -- 172.21.15.46:0/3594682029 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d0169b70 con 0x7fe6d015d210 2024-01-27T01:06:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6d6027700 1 -- 172.21.15.46:0/3594682029 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d00c87b0 con 0x7fe6d00cc1c0 2024-01-27T01:06:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6cffff700 1 --2- 172.21.15.46:0/3594682029 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6d015d210 0x7fe6d00c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6ceffd700 1 -- 172.21.15.46:0/3594682029 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6d00cc1c0 msgr2=0x7fe6d015bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6ceffd700 1 --2- 172.21.15.46:0/3594682029 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6d00cc1c0 0x7fe6d015bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6ceffd700 1 -- 172.21.15.46:0/3594682029 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6d015d210 msgr2=0x7fe6d00c8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6ceffd700 1 --2- 172.21.15.46:0/3594682029 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6d015d210 0x7fe6d00c8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6ceffd700 1 -- 172.21.15.46:0/3594682029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c0009a20 con 0x7fe6d015ce40 2024-01-27T01:06:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6cf7fe700 1 --2- 172.21.15.46:0/3594682029 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6d00cc1c0 0x7fe6d015bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6ceffd700 1 --2- 172.21.15.46:0/3594682029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d015ce40 0x7fe6d015b570 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4001eb0 tx=0x7fe6c4007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6b77fe700 1 -- 172.21.15.46:0/3594682029 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4005a70 con 0x7fe6d015ce40 2024-01-27T01:06:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6b77fe700 1 -- 172.21.15.46:0/3594682029 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c4005bd0 con 0x7fe6d015ce40 2024-01-27T01:06:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6d6027700 1 -- 172.21.15.46:0/3594682029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c80094f0 con 0x7fe6d015ce40 2024-01-27T01:06:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6b77fe700 1 -- 172.21.15.46:0/3594682029 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c40070d0 con 0x7fe6d015ce40 2024-01-27T01:06:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.403+0000 7fe6d6027700 1 -- 172.21.15.46:0/3594682029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d016a0f0 con 0x7fe6d015ce40 2024-01-27T01:06:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.404+0000 7fe6d6027700 1 -- 172.21.15.46:0/3594682029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d004c710 con 0x7fe6d015ce40 2024-01-27T01:06:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.405+0000 7fe6b77fe700 1 -- 172.21.15.46:0/3594682029 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 122) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fe6c4007480 con 0x7fe6d015ce40 2024-01-27T01:06:07.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.405+0000 7fe6b77fe700 1 --2- 172.21.15.46:0/3594682029 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe6bc067050 0x7fe6bc069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:07.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.406+0000 7fe6cf7fe700 1 -- 172.21.15.46:0/3594682029 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe6bc067050 msgr2=0x7fe6bc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:07.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.406+0000 7fe6b77fe700 1 -- 172.21.15.46:0/3594682029 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6906+0+0 (secure 0 0 0) 0x7fe6c4083ec0 con 0x7fe6d015ce40 2024-01-27T01:06:07.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.406+0000 7fe6cf7fe700 1 --2- 172.21.15.46:0/3594682029 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe6bc067050 0x7fe6bc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:06:07.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.414+0000 7fe6b77fe700 1 -- 172.21.15.46:0/3594682029 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe6c4050b70 con 0x7fe6d015ce40 2024-01-27T01:06:07.535 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:07.535+0000 7f86191b01c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:06:07.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.607+0000 7fe6cf7fe700 1 -- 172.21.15.46:0/3594682029 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe6bc067050 msgr2=0x7fe6bc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:07.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.607+0000 7fe6cf7fe700 1 --2- 172.21.15.46:0/3594682029 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe6bc067050 0x7fe6bc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:06:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.669+0000 7fe6d6027700 1 -- 172.21.15.46:0/3594682029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6d00cf980 con 0x7fe6d015ce40 2024-01-27T01:06:07.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.669+0000 7fe6b77fe700 1 -- 172.21.15.46:0/3594682029 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4612 (secure 0 0 0) 0x7fe6c4017020 con 0x7fe6d015ce40 2024-01-27T01:06:07.670 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:07.670 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":107,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8891,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/975115379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":975115379},{"type":"v1","addr":"172.21.15.106:6833","nonce":975115379}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10036,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3860120510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3860120510},{"type":"v1","addr":"172.21.15.46:6834","nonce":3860120510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10045,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3603549739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3603549739},{"type":"v1","addr":"172.21.15.106:6835","nonce":3603549739}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:03:44.141242+0000","modified":"2024-01-27T01:03:47.727201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10392":{"gid":10392,"name":"c","rank":0,"incarnation":103,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/976738644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":976738644},{"type":"v1","addr":"172.21.15.46:6835","nonce":976738644}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2024-01-27T01:06:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.672+0000 7fe6d6027700 1 -- 172.21.15.46:0/3594682029 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe6bc067050 msgr2=0x7fe6bc069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:06:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.672+0000 7fe6d6027700 1 --2- 172.21.15.46:0/3594682029 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe6bc067050 0x7fe6bc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.672+0000 7fe6d6027700 1 -- 172.21.15.46:0/3594682029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d015ce40 msgr2=0x7fe6d015b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.672+0000 7fe6d6027700 1 --2- 172.21.15.46:0/3594682029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d015ce40 0x7fe6d015b570 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4001eb0 tx=0x7fe6c4007f40 comp rx=0 tx=0).stop 2024-01-27T01:06:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.672+0000 7fe6d6027700 1 -- 172.21.15.46:0/3594682029 shutdown_connections 2024-01-27T01:06:07.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.672+0000 7fe6d6027700 1 --2- 172.21.15.46:0/3594682029 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe6bc067050 0x7fe6bc069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.672+0000 7fe6d6027700 1 --2- 172.21.15.46:0/3594682029 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6d015d210 0x7fe6d00c8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.672+0000 7fe6d6027700 1 --2- 172.21.15.46:0/3594682029 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6d00cc1c0 0x7fe6d015bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.672+0000 7fe6d6027700 1 --2- 172.21.15.46:0/3594682029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d015ce40 0x7fe6d015b570 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.672+0000 7fe6d6027700 1 -- 172.21.15.46:0/3594682029 >> 172.21.15.46:0/3594682029 conn(0x7fe6d006f840 msgr2=0x7fe6d00b5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:07.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.672+0000 7fe6d6027700 1 -- 172.21.15.46:0/3594682029 shutdown_connections 2024-01-27T01:06:07.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.672+0000 7fe6d6027700 1 -- 172.21.15.46:0/3594682029 wait complete. 2024-01-27T01:06:07.673 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 107 2024-01-27T01:06:07.686 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:06:07.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.824+0000 7f4592b7e700 1 Processor -- start 2024-01-27T01:06:07.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.824+0000 7f4592b7e700 1 -- start start 2024-01-27T01:06:07.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.825+0000 7f4592b7e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f458c15ce40 0x7f458c0d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:07.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.825+0000 7f4592b7e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f458c0cc1c0 0x7f458c0ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:07.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.825+0000 7f4592b7e700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f458c0cff70 con 0x7f458c15d210 2024-01-27T01:06:07.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.825+0000 7f4592b7e700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f458c0cf400 con 0x7f458c0cc1c0 2024-01-27T01:06:07.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.825+0000 7f4592b7e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458c0cf660 con 0x7f458c15ce40 2024-01-27T01:06:07.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.825+0000 7f4583fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f458c15d210 0x7f458c0cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57994/0 (socket says 172.21.15.46:57994) 2024-01-27T01:06:07.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.825+0000 7f4583fff700 1 -- 172.21.15.46:0/1128180553 learned_addr learned my addr 172.21.15.46:0/1128180553 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:07.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.825+0000 7f459111b700 1 --2- 172.21.15.46:0/1128180553 >> v2:172.21.15.106:3300/0 conn(0x7f458c0cc1c0 0x7f458c0ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:07.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.825+0000 7f459091a700 1 --2- 172.21.15.46:0/1128180553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f458c15ce40 0x7f458c0d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:07.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.825+0000 7f45837fe700 1 -- 172.21.15.46:0/1128180553 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617072568 0 0) 0x7f458c0cff70 con 0x7f458c15d210 2024-01-27T01:06:07.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.825+0000 7f45837fe700 1 -- 172.21.15.46:0/1128180553 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4574003570 con 0x7f458c15d210 2024-01-27T01:06:07.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.825+0000 7f459111b700 1 -- 172.21.15.46:0/1128180553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f458c15ce40 msgr2=0x7f458c0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:07.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.825+0000 7f459111b700 1 --2- 172.21.15.46:0/1128180553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f458c15ce40 0x7f458c0d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.825+0000 7f459111b700 1 -- 172.21.15.46:0/1128180553 >> v1:172.21.15.46:6789/0 conn(0x7f458c15d210 legacy=0x7f458c0cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:07.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.825+0000 7f459111b700 1 -- 172.21.15.46:0/1128180553 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458c0d0500 con 0x7f458c0cc1c0 2024-01-27T01:06:07.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.826+0000 7f459111b700 1 --2- 172.21.15.46:0/1128180553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f458c0cc1c0 0x7f458c0ade30 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f4584009e60 tx=0x7f4584009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36b57ccb2f52f0be server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:07.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.826+0000 7f45837fe700 1 -- 172.21.15.46:0/1128180553 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458400e070 con 0x7f458c0cc1c0 2024-01-27T01:06:07.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.826+0000 7f45837fe700 1 -- 172.21.15.46:0/1128180553 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458400bab0 con 0x7f458c0cc1c0 2024-01-27T01:06:07.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.826+0000 7f45837fe700 1 -- 172.21.15.46:0/1128180553 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458400be70 con 0x7f458c0cc1c0 2024-01-27T01:06:07.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.826+0000 7f4592b7e700 1 -- 172.21.15.46:0/1128180553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f458c0cc1c0 msgr2=0x7f458c0ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:07.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.826+0000 7f4592b7e700 1 --2- 172.21.15.46:0/1128180553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f458c0cc1c0 0x7f458c0ade30 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f4584009e60 tx=0x7f4584009780 comp rx=0 tx=0).stop 2024-01-27T01:06:07.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.826+0000 7f4592b7e700 1 -- 172.21.15.46:0/1128180553 shutdown_connections 2024-01-27T01:06:07.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.826+0000 7f4592b7e700 1 --2- 172.21.15.46:0/1128180553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f458c0cc1c0 0x7f458c0ade30 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.827+0000 7f4592b7e700 1 --2- 172.21.15.46:0/1128180553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f458c15ce40 0x7f458c0d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.827+0000 7f4592b7e700 1 -- 172.21.15.46:0/1128180553 >> 172.21.15.46:0/1128180553 conn(0x7f458c06f840 msgr2=0x7f458c0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:07.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.827+0000 7f4592b7e700 1 -- 172.21.15.46:0/1128180553 shutdown_connections 2024-01-27T01:06:07.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.827+0000 7f4592b7e700 1 -- 172.21.15.46:0/1128180553 wait complete. 2024-01-27T01:06:07.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.828+0000 7f4592b7e700 1 Processor -- start 2024-01-27T01:06:07.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.828+0000 7f4592b7e700 1 -- start start 2024-01-27T01:06:07.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.829+0000 7f4592b7e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f458c0cc1c0 0x7f458c0c40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:07.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.829+0000 7f4592b7e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f458c15ce40 0x7f458c0c6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:07.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.829+0000 7f4592b7e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c15d210 0x7f458c159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:07.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.829+0000 7f4592b7e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c0c52b0 con 0x7f458c15d210 2024-01-27T01:06:07.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.829+0000 7f4592b7e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c0c7490 con 0x7f458c0cc1c0 2024-01-27T01:06:07.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.829+0000 7f4592b7e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458c0c46a0 con 0x7f458c15ce40 2024-01-27T01:06:07.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.829+0000 7f459111b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c15d210 0x7f458c159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:07.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.829+0000 7f4583fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f458c15ce40 0x7f458c0c6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:07.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.829+0000 7f459111b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c15d210 0x7f458c159ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44722/0 (socket says 172.21.15.46:44722) 2024-01-27T01:06:07.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.829+0000 7f459111b700 1 -- 172.21.15.46:0/1953471239 learned_addr learned my addr 172.21.15.46:0/1953471239 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:07.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.829+0000 7f459091a700 1 --2- 172.21.15.46:0/1953471239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f458c0cc1c0 0x7f458c0c40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:07.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.829+0000 7f459111b700 1 -- 172.21.15.46:0/1953471239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f458c15ce40 msgr2=0x7f458c0c6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:07.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.829+0000 7f459111b700 1 --2- 172.21.15.46:0/1953471239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f458c15ce40 0x7f458c0c6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.829+0000 7f459111b700 1 -- 172.21.15.46:0/1953471239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f458c0cc1c0 msgr2=0x7f458c0c40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:07.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.829+0000 7f459111b700 1 --2- 172.21.15.46:0/1953471239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f458c0cc1c0 0x7f458c0c40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:07.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.829+0000 7f459111b700 1 -- 172.21.15.46:0/1953471239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4584009a20 con 0x7f458c15d210 2024-01-27T01:06:07.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.829+0000 7f459091a700 1 --2- 172.21.15.46:0/1953471239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f458c0cc1c0 0x7f458c0c40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:07.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.830+0000 7f459111b700 1 --2- 172.21.15.46:0/1953471239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c15d210 0x7f458c159ea0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f4584013000 tx=0x7f4584019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:07.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.830+0000 7f4580ff9700 1 -- 172.21.15.46:0/1953471239 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4584012e30 con 0x7f458c15d210 2024-01-27T01:06:07.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.830+0000 7f4580ff9700 1 -- 172.21.15.46:0/1953471239 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4584027470 con 0x7f458c15d210 2024-01-27T01:06:07.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.830+0000 7f4580ff9700 1 -- 172.21.15.46:0/1953471239 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4584027830 con 0x7f458c15d210 2024-01-27T01:06:07.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.830+0000 7f4592b7e700 1 -- 172.21.15.46:0/1953471239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458c0c4980 con 0x7f458c15d210 2024-01-27T01:06:07.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.830+0000 7f4592b7e700 1 -- 172.21.15.46:0/1953471239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458c0c4f40 con 0x7f458c15d210 2024-01-27T01:06:07.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.831+0000 7f4592b7e700 1 -- 172.21.15.46:0/1953471239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458c0c5420 con 0x7f458c15d210 2024-01-27T01:06:07.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.832+0000 7f4580ff9700 1 -- 172.21.15.46:0/1953471239 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 122) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f4584024070 con 0x7f458c15d210 2024-01-27T01:06:07.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.833+0000 7f4580ff9700 1 --2- 172.21.15.46:0/1953471239 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4578067050 0x7f4578069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:07.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.833+0000 7f4580ff9700 1 -- 172.21.15.46:0/1953471239 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6906+0+0 (secure 0 0 0) 0x7f4584087430 con 0x7f458c15d210 2024-01-27T01:06:07.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.833+0000 7f459091a700 1 -- 172.21.15.46:0/1953471239 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4578067050 msgr2=0x7f4578069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:07.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.833+0000 7f459091a700 1 --2- 172.21.15.46:0/1953471239 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4578067050 0x7f4578069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:06:07.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:07.837+0000 7f4580ff9700 1 -- 172.21.15.46:0/1953471239 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f45840540e0 con 0x7f458c15d210 2024-01-27T01:06:07.876 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:07.876+0000 7f3d64b1e1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:06:08.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.034+0000 7f459091a700 1 -- 172.21.15.46:0/1953471239 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4578067050 msgr2=0x7f4578069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:08.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.034+0000 7f459091a700 1 --2- 172.21.15.46:0/1953471239 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4578067050 0x7f4578069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:06:08.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.047+0000 7f4592b7e700 1 -- 172.21.15.46:0/1953471239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f458c04c740 con 0x7f458c15d210 2024-01-27T01:06:08.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.048+0000 7f4580ff9700 1 -- 172.21.15.46:0/1953471239 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v210) v1 ==== 74+0+16934 (secure 0 0 0) 0x7f4584057700 con 0x7f458c15d210 2024-01-27T01:06:08.048 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:08.048 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":210,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:06:04.711162+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:03:42.648202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-01-27T01:03:43.106225+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"210","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":210,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.1b","mappings":[{"from":3,"to":0}]},{"pgid":"26.31","mappings":[{"from":1,"to":0}]},{"pgid":"26.3a","mappings":[{"from":1,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/1813035004":"2024-01-28T01:03:58.706905+0000","172.21.15.106:0/2234269173":"2024-01-28T01:03:58.706905+0000","172.21.15.106:6837/62434":"2024-01-28T01:03:58.706905+0000","172.21.15.106:6836/62434":"2024-01-28T01:03:58.706905+0000","172.21.15.106:0/3361483200":"2024-01-28T01:03:58.706905+0000","172.21.15.106:0/1169106698":"2024-01-28T01:03:58.706905+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":26,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:06:08.048 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:08.048+0000 7f3d64b1e1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:06:08.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.050+0000 7f4592b7e700 1 -- 172.21.15.46:0/1953471239 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4578067050 msgr2=0x7f4578069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:06:08.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.050+0000 7f4592b7e700 1 --2- 172.21.15.46:0/1953471239 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4578067050 0x7f4578069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.050+0000 7f4592b7e700 1 -- 172.21.15.46:0/1953471239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c15d210 msgr2=0x7f458c159ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:08.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.050+0000 7f4592b7e700 1 --2- 172.21.15.46:0/1953471239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c15d210 0x7f458c159ea0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f4584013000 tx=0x7f4584019000 comp rx=0 tx=0).stop 2024-01-27T01:06:08.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.050+0000 7f4592b7e700 1 -- 172.21.15.46:0/1953471239 shutdown_connections 2024-01-27T01:06:08.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.050+0000 7f4592b7e700 1 --2- 172.21.15.46:0/1953471239 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4578067050 0x7f4578069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.050+0000 7f4592b7e700 1 --2- 172.21.15.46:0/1953471239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c15d210 0x7f458c159ea0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.050+0000 7f4592b7e700 1 --2- 172.21.15.46:0/1953471239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f458c0cc1c0 0x7f458c0c40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.050+0000 7f4592b7e700 1 --2- 172.21.15.46:0/1953471239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f458c15ce40 0x7f458c0c6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.050+0000 7f4592b7e700 1 -- 172.21.15.46:0/1953471239 >> 172.21.15.46:0/1953471239 conn(0x7f458c06f840 msgr2=0x7f458c0b44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:08.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.050+0000 7f4592b7e700 1 -- 172.21.15.46:0/1953471239 shutdown_connections 2024-01-27T01:06:08.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.050+0000 7f4592b7e700 1 -- 172.21.15.46:0/1953471239 wait complete. 2024-01-27T01:06:08.064 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-27T01:06:08.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.198+0000 7fb5b8904700 1 Processor -- start 2024-01-27T01:06:08.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.198+0000 7fb5b8904700 1 -- start start 2024-01-27T01:06:08.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.200+0000 7fb5b8904700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5b40b2b40 0x7fb5b40ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:08.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.200+0000 7fb5b8904700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b415a0b0 con 0x7fb5b40ae960 2024-01-27T01:06:08.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.200+0000 7fb5b8904700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b40ae4d0 con 0x7fb5b40aedd0 2024-01-27T01:06:08.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.200+0000 7fb5b8904700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b4099fc0 con 0x7fb5b40b2b40 2024-01-27T01:06:08.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.200+0000 7fb5b259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5b40b2b40 0x7fb5b40ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.200+0000 7fb5b259c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5b40b2b40 0x7fb5b40ad1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44728/0 (socket says 172.21.15.46:44728) 2024-01-27T01:06:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.200+0000 7fb5b259c700 1 -- 172.21.15.46:0/3499892837 learned_addr learned my addr 172.21.15.46:0/3499892837 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.200+0000 7fb5b259c700 1 -- 172.21.15.46:0/3499892837 >> v1:172.21.15.106:6789/0 conn(0x7fb5b40aedd0 legacy=0x7fb5b40cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.201+0000 7fb5b259c700 1 -- 172.21.15.46:0/3499892837 >> v1:172.21.15.46:6789/0 conn(0x7fb5b40ae960 legacy=0x7fb5b40ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.200+0000 7fb5b159a700 1 -- 172.21.15.46:0/3499892837 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177136382 0 0) 0x7fb5b40ae4d0 con 0x7fb5b40aedd0 2024-01-27T01:06:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.201+0000 7fb5b259c700 1 -- 172.21.15.46:0/3499892837 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b40ad8c0 con 0x7fb5b40b2b40 2024-01-27T01:06:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.201+0000 7fb5b259c700 1 --2- 172.21.15.46:0/3499892837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b40b2b40 0x7fb5b40ad1c0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009f70 tx=0x7fb5a4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7f43d4467f7314e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.201+0000 7fb5b159a700 1 -- 172.21.15.46:0/3499892837 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a400e070 con 0x7fb5b40b2b40 2024-01-27T01:06:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.201+0000 7fb5b159a700 1 -- 172.21.15.46:0/3499892837 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a400bad0 con 0x7fb5b40b2b40 2024-01-27T01:06:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.201+0000 7fb5b159a700 1 -- 172.21.15.46:0/3499892837 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a40092e0 con 0x7fb5b40b2b40 2024-01-27T01:06:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.202+0000 7fb5b8904700 1 -- 172.21.15.46:0/3499892837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b40b2b40 msgr2=0x7fb5b40ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.202+0000 7fb5b8904700 1 --2- 172.21.15.46:0/3499892837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b40b2b40 0x7fb5b40ad1c0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009f70 tx=0x7fb5a4009840 comp rx=0 tx=0).stop 2024-01-27T01:06:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.202+0000 7fb5b8904700 1 -- 172.21.15.46:0/3499892837 shutdown_connections 2024-01-27T01:06:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.202+0000 7fb5b8904700 1 --2- 172.21.15.46:0/3499892837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b40b2b40 0x7fb5b40ad1c0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.202+0000 7fb5b8904700 1 -- 172.21.15.46:0/3499892837 >> 172.21.15.46:0/3499892837 conn(0x7fb5b4099a30 msgr2=0x7fb5b406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.202+0000 7fb5b8904700 1 -- 172.21.15.46:0/3499892837 shutdown_connections 2024-01-27T01:06:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.202+0000 7fb5b8904700 1 -- 172.21.15.46:0/3499892837 wait complete. 2024-01-27T01:06:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.203+0000 7fb5b8904700 1 Processor -- start 2024-01-27T01:06:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.203+0000 7fb5b8904700 1 -- start start 2024-01-27T01:06:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.204+0000 7fb5b8904700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5b40ae960 0x7fb5b416d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.204+0000 7fb5b8904700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b40b2b40 0x7fb5b40cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.204+0000 7fb5b8904700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb5b40b4470 0x7fb5b4153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.204+0000 7fb5b8904700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b40b0850 con 0x7fb5b40b2b40 2024-01-27T01:06:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.204+0000 7fb5b8904700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b416e100 con 0x7fb5b40ae960 2024-01-27T01:06:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.204+0000 7fb5b8904700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b4153da0 con 0x7fb5b40b4470 2024-01-27T01:06:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.204+0000 7fb5b1d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b40b2b40 0x7fb5b40cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.204+0000 7fb5b1d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b40b2b40 0x7fb5b40cbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44740/0 (socket says 172.21.15.46:44740) 2024-01-27T01:06:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.204+0000 7fb5b1d9b700 1 -- 172.21.15.46:0/3025434590 learned_addr learned my addr 172.21.15.46:0/3025434590 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.204+0000 7fb5b259c700 1 --2- 172.21.15.46:0/3025434590 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5b40ae960 0x7fb5b416d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.204+0000 7fb5b2d9d700 1 --2- 172.21.15.46:0/3025434590 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb5b40b4470 0x7fb5b4153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.204+0000 7fb5b1d9b700 1 -- 172.21.15.46:0/3025434590 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb5b40b4470 msgr2=0x7fb5b4153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.204+0000 7fb5b1d9b700 1 --2- 172.21.15.46:0/3025434590 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb5b40b4470 0x7fb5b4153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.204+0000 7fb5b1d9b700 1 -- 172.21.15.46:0/3025434590 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5b40ae960 msgr2=0x7fb5b416d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.204+0000 7fb5b1d9b700 1 --2- 172.21.15.46:0/3025434590 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5b40ae960 0x7fb5b416d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.204+0000 7fb5b1d9b700 1 -- 172.21.15.46:0/3025434590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a4009ae0 con 0x7fb5b40b2b40 2024-01-27T01:06:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.204+0000 7fb5b259c700 1 --2- 172.21.15.46:0/3025434590 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5b40ae960 0x7fb5b416d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.205+0000 7fb5b1d9b700 1 --2- 172.21.15.46:0/3025434590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b40b2b40 0x7fb5b40cbff0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800c790 tx=0x7fb5a8007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.205+0000 7fb5a27fc700 1 -- 172.21.15.46:0/3025434590 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8011590 con 0x7fb5b40b2b40 2024-01-27T01:06:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.205+0000 7fb5a27fc700 1 -- 172.21.15.46:0/3025434590 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a8011710 con 0x7fb5b40b2b40 2024-01-27T01:06:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.205+0000 7fb5b8904700 1 -- 172.21.15.46:0/3025434590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b416d860 con 0x7fb5b40b2b40 2024-01-27T01:06:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.205+0000 7fb5a27fc700 1 -- 172.21.15.46:0/3025434590 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8011ad0 con 0x7fb5b40b2b40 2024-01-27T01:06:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.205+0000 7fb5b8904700 1 -- 172.21.15.46:0/3025434590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b416de20 con 0x7fb5b40b2b40 2024-01-27T01:06:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.206+0000 7fb5b8904700 1 -- 172.21.15.46:0/3025434590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b40b09c0 con 0x7fb5b40b2b40 2024-01-27T01:06:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.207+0000 7fb5a27fc700 1 -- 172.21.15.46:0/3025434590 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 122) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fb5a801b3b0 con 0x7fb5b40b2b40 2024-01-27T01:06:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.207+0000 7fb5a27fc700 1 --2- 172.21.15.46:0/3025434590 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb594067050 0x7fb594069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.207+0000 7fb5a27fc700 1 -- 172.21.15.46:0/3025434590 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6906+0+0 (secure 0 0 0) 0x7fb5a8083f80 con 0x7fb5b40b2b40 2024-01-27T01:06:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.207+0000 7fb5b259c700 1 -- 172.21.15.46:0/3025434590 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb594067050 msgr2=0x7fb594069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.207+0000 7fb5b259c700 1 --2- 172.21.15.46:0/3025434590 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb594067050 0x7fb594069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:06:08.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.212+0000 7fb5a27fc700 1 -- 172.21.15.46:0/3025434590 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb5a8050b90 con 0x7fb5b40b2b40 2024-01-27T01:06:08.231 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:08.231+0000 7f3d64b1e1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:06:08.237 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:08.238+0000 7f86191b01c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:06:08.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.408+0000 7fb5b259c700 1 -- 172.21.15.46:0/3025434590 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb594067050 msgr2=0x7fb594069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:08.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.408+0000 7fb5b259c700 1 --2- 172.21.15.46:0/3025434590 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb594067050 0x7fb594069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:06:08.456 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:08.457+0000 7f86191b01c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:06:08.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.467+0000 7fb5b8904700 1 -- 172.21.15.46:0/3025434590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb5b404c710 con 0x7fb5b40b2b40 2024-01-27T01:06:08.468 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-27T01:06:08.468+0000 7f33ffe86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-27T01:06:08.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.472+0000 7fb5a27fc700 1 -- 172.21.15.46:0/3025434590 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v108) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb5a8017070 con 0x7fb5b40b2b40 2024-01-27T01:06:08.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.475+0000 7fb5b8904700 1 -- 172.21.15.46:0/3025434590 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb594067050 msgr2=0x7fb594069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:06:08.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.475+0000 7fb5b8904700 1 --2- 172.21.15.46:0/3025434590 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb594067050 0x7fb594069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.475+0000 7fb5b8904700 1 -- 172.21.15.46:0/3025434590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b40b2b40 msgr2=0x7fb5b40cbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:08.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.475+0000 7fb5b8904700 1 --2- 172.21.15.46:0/3025434590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b40b2b40 0x7fb5b40cbff0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800c790 tx=0x7fb5a8007f40 comp rx=0 tx=0).stop 2024-01-27T01:06:08.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.480+0000 7fb5b8904700 1 -- 172.21.15.46:0/3025434590 shutdown_connections 2024-01-27T01:06:08.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.480+0000 7fb5b8904700 1 --2- 172.21.15.46:0/3025434590 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb594067050 0x7fb594069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.480+0000 7fb5b8904700 1 --2- 172.21.15.46:0/3025434590 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb5b40b4470 0x7fb5b4153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.480+0000 7fb5b8904700 1 --2- 172.21.15.46:0/3025434590 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb5b40ae960 0x7fb5b416d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.480+0000 7fb5b8904700 1 --2- 172.21.15.46:0/3025434590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b40b2b40 0x7fb5b40cbff0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.480+0000 7fb5b8904700 1 -- 172.21.15.46:0/3025434590 >> 172.21.15.46:0/3025434590 conn(0x7fb5b4099a30 msgr2=0x7fb5b40cf010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:08.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.480+0000 7fb5b8904700 1 -- 172.21.15.46:0/3025434590 shutdown_connections 2024-01-27T01:06:08.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.480+0000 7fb5b8904700 1 -- 172.21.15.46:0/3025434590 wait complete. 2024-01-27T01:06:08.481 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-27T01:06:08.494 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-27T01:06:08.524 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-27T01:06:08.531 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:08.532+0000 7f86191b01c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:06:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.634+0000 7f2ff1086700 1 Processor -- start 2024-01-27T01:06:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.634+0000 7f2ff1086700 1 -- start start 2024-01-27T01:06:08.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.635+0000 7f2ff1086700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2fec157a30 0x7f2fec06c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:08.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.635+0000 7f2ff1086700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fec159190 0x7f2fec157e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:08.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.635+0000 7f2ff1086700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fec1587c0 con 0x7f2fec158340 2024-01-27T01:06:08.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.635+0000 7f2ff1086700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fec158aa0 con 0x7f2fec157a30 2024-01-27T01:06:08.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.635+0000 7f2ff1086700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec158d80 con 0x7f2fec159190 2024-01-27T01:06:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.635+0000 7f2feb59e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2fec158340 0x7f2fec0cd890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58018/0 (socket says 172.21.15.46:58018) 2024-01-27T01:06:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.635+0000 7f2feb59e700 1 -- 172.21.15.46:0/3286273649 learned_addr learned my addr 172.21.15.46:0/3286273649 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.635+0000 7f2fea59c700 1 --2- 172.21.15.46:0/3286273649 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fec159190 0x7f2fec157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.635+0000 7f2fead9d700 1 --2- 172.21.15.46:0/3286273649 >> v2:172.21.15.106:3300/0 conn(0x7f2fec157a30 0x7f2fec06c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.636+0000 7f2fe9d9b700 1 -- 172.21.15.46:0/3286273649 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148255126 0 0) 0x7f2fec1587c0 con 0x7f2fec158340 2024-01-27T01:06:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.636+0000 7f2fe9d9b700 1 -- 172.21.15.46:0/3286273649 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd4003570 con 0x7f2fec158340 2024-01-27T01:06:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.636+0000 7f2fead9d700 1 -- 172.21.15.46:0/3286273649 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fec159190 msgr2=0x7f2fec157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.636+0000 7f2fead9d700 1 --2- 172.21.15.46:0/3286273649 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fec159190 0x7f2fec157e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.636+0000 7f2fead9d700 1 -- 172.21.15.46:0/3286273649 >> v1:172.21.15.46:6789/0 conn(0x7f2fec158340 legacy=0x7f2fec0cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.636+0000 7f2fead9d700 1 -- 172.21.15.46:0/3286273649 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec0cc4f0 con 0x7f2fec157a30 2024-01-27T01:06:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.636+0000 7f2fea59c700 1 --2- 172.21.15.46:0/3286273649 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fec159190 0x7f2fec157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.636+0000 7f2fead9d700 1 --2- 172.21.15.46:0/3286273649 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fec157a30 0x7f2fec06c450 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc0098e0 tx=0x7f2fdc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9257c0f7b51ecfb2 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.636+0000 7f2fe9d9b700 1 -- 172.21.15.46:0/3286273649 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc011630 con 0x7f2fec157a30 2024-01-27T01:06:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.636+0000 7f2fe9d9b700 1 -- 172.21.15.46:0/3286273649 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fdc011790 con 0x7f2fec157a30 2024-01-27T01:06:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.636+0000 7f2fe9d9b700 1 -- 172.21.15.46:0/3286273649 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc011b30 con 0x7f2fec157a30 2024-01-27T01:06:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.637+0000 7f2ff1086700 1 -- 172.21.15.46:0/3286273649 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fec157a30 msgr2=0x7f2fec06c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.637+0000 7f2ff1086700 1 --2- 172.21.15.46:0/3286273649 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fec157a30 0x7f2fec06c450 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc0098e0 tx=0x7f2fdc0091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.637+0000 7f2ff1086700 1 -- 172.21.15.46:0/3286273649 shutdown_connections 2024-01-27T01:06:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.637+0000 7f2ff1086700 1 --2- 172.21.15.46:0/3286273649 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fec157a30 0x7f2fec06c450 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.637+0000 7f2ff1086700 1 --2- 172.21.15.46:0/3286273649 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fec159190 0x7f2fec157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.637+0000 7f2ff1086700 1 -- 172.21.15.46:0/3286273649 >> 172.21.15.46:0/3286273649 conn(0x7f2fec0ce570 msgr2=0x7f2fec0ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.637+0000 7f2ff1086700 1 -- 172.21.15.46:0/3286273649 shutdown_connections 2024-01-27T01:06:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.637+0000 7f2ff1086700 1 -- 172.21.15.46:0/3286273649 wait complete. 2024-01-27T01:06:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.638+0000 7f2ff1086700 1 Processor -- start 2024-01-27T01:06:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.638+0000 7f2ff1086700 1 -- start start 2024-01-27T01:06:08.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.639+0000 7f2ff1086700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fec157a30 0x7f2fec0b4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:08.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.639+0000 7f2ff1086700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fec159190 0x7f2fec0c6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:08.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.639+0000 7f2ff1086700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fec0b2a80 0x7f2fec154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:08.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.639+0000 7f2ff1086700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec0c5960 con 0x7f2fec0b2a80 2024-01-27T01:06:08.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.639+0000 7f2ff1086700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec0b3b70 con 0x7f2fec157a30 2024-01-27T01:06:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.639+0000 7f2ff1086700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec0b2f60 con 0x7f2fec159190 2024-01-27T01:06:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.639+0000 7f2feb59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fec0b2a80 0x7f2fec154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.639+0000 7f2fead9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fec157a30 0x7f2fec0b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.639+0000 7f2fea59c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fec159190 0x7f2fec0c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.639+0000 7f2feb59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fec0b2a80 0x7f2fec154d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44762/0 (socket says 172.21.15.46:44762) 2024-01-27T01:06:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.639+0000 7f2fead9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fec157a30 0x7f2fec0b4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:34364/0 (socket says 172.21.15.46:34364) 2024-01-27T01:06:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.639+0000 7f2feb59e700 1 -- 172.21.15.46:0/2853683553 learned_addr learned my addr 172.21.15.46:0/2853683553 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.640+0000 7f2fea59c700 1 -- 172.21.15.46:0/2853683553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fec157a30 msgr2=0x7f2fec0b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.640+0000 7f2fea59c700 1 --2- 172.21.15.46:0/2853683553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fec157a30 0x7f2fec0b4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.640+0000 7f2fea59c700 1 -- 172.21.15.46:0/2853683553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fec0b2a80 msgr2=0x7f2fec154d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.640+0000 7f2fea59c700 1 --2- 172.21.15.46:0/2853683553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fec0b2a80 0x7f2fec154d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.640+0000 7f2fea59c700 1 -- 172.21.15.46:0/2853683553 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe0009420 con 0x7f2fec159190 2024-01-27T01:06:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.640+0000 7f2fead9d700 1 --2- 172.21.15.46:0/2853683553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fec157a30 0x7f2fec0b4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.640+0000 7f2fea59c700 1 --2- 172.21.15.46:0/2853683553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fec159190 0x7f2fec0c6540 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000ea00 tx=0x7f2fe00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.640+0000 7f2fd2ffd700 1 -- 172.21.15.46:0/2853683553 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe000c6a0 con 0x7f2fec159190 2024-01-27T01:06:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.640+0000 7f2fd2ffd700 1 -- 172.21.15.46:0/2853683553 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe000c800 con 0x7f2fec159190 2024-01-27T01:06:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.640+0000 7f2ff1086700 1 -- 172.21.15.46:0/2853683553 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fdc009420 con 0x7f2fec159190 2024-01-27T01:06:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.640+0000 7f2fd2ffd700 1 -- 172.21.15.46:0/2853683553 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe000cba0 con 0x7f2fec159190 2024-01-27T01:06:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.640+0000 7f2ff1086700 1 -- 172.21.15.46:0/2853683553 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fec0b3520 con 0x7f2fec159190 2024-01-27T01:06:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.641+0000 7f2ff1086700 1 -- 172.21.15.46:0/2853683553 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fec070270 con 0x7f2fec159190 2024-01-27T01:06:08.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.642+0000 7f2fd2ffd700 1 -- 172.21.15.46:0/2853683553 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 122) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f2fe0013470 con 0x7f2fec159190 2024-01-27T01:06:08.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.642+0000 7f2fd2ffd700 1 --2- 172.21.15.46:0/2853683553 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2fd8067050 0x7f2fd8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:08.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.642+0000 7f2fead9d700 1 -- 172.21.15.46:0/2853683553 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2fd8067050 msgr2=0x7f2fd8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:08.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.642+0000 7f2fd2ffd700 1 -- 172.21.15.46:0/2853683553 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6948+0+0 (secure 0 0 0) 0x7f2fe0084d90 con 0x7f2fec159190 2024-01-27T01:06:08.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.642+0000 7f2fead9d700 1 --2- 172.21.15.46:0/2853683553 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2fd8067050 0x7f2fd8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:06:08.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.647+0000 7f2fd2ffd700 1 -- 172.21.15.46:0/2853683553 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2fe0051a10 con 0x7f2fec159190 2024-01-27T01:06:08.690 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:08.690+0000 7f86191b01c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:06:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.717+0000 7f2fd2ffd700 1 -- 172.21.15.46:0/2853683553 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mgrmap(e 123) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f2fe0051de0 con 0x7f2fec159190 2024-01-27T01:06:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.718+0000 7f2fd2ffd700 1 -- 172.21.15.46:0/2853683553 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2fd8067050 msgr2=0x7f2fd8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:06:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.718+0000 7f2fd2ffd700 1 --2- 172.21.15.46:0/2853683553 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2fd8067050 0x7f2fd8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:08.783 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:08.783+0000 7f8597a9e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:08.783 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:08.783+0000 7f8597a9e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:08.783 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:08.783+0000 7f8597a9e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:08.783 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:08.783+0000 7f8597a9e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:08.783 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:08.783+0000 7f8597a9e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:08.784 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:08.784+0000 7f8594297700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:08.784 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:08.784+0000 7f8594297700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:08.784 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:08.784+0000 7f8594297700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:08.784 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:08.784+0000 7f8594297700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:08.784 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:08.784+0000 7f8594297700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:08.908+0000 7f2ff1086700 1 -- 172.21.15.46:0/2853683553 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2fec04c710 con 0x7f2fec159190 2024-01-27T01:06:09.074 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:09.074+0000 7f3d64b1e1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:06:09.324 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:09.324+0000 7f3d64b1e1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:06:09.407 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:09.408+0000 7f3d64b1e1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:06:09.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.475+0000 7f2fd2ffd700 1 -- 172.21.15.46:0/2853683553 <== mon.2 v2:172.21.15.106:3301/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v109) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2fe006eec0 con 0x7f2fec159190 2024-01-27T01:06:09.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.479+0000 7f2ff1086700 1 -- 172.21.15.46:0/2853683553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fec159190 msgr2=0x7f2fec0c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:09.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.479+0000 7f2ff1086700 1 --2- 172.21.15.46:0/2853683553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fec159190 0x7f2fec0c6540 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000ea00 tx=0x7f2fe00091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:09.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.479+0000 7f2ff1086700 1 -- 172.21.15.46:0/2853683553 shutdown_connections 2024-01-27T01:06:09.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.480+0000 7f2ff1086700 1 --2- 172.21.15.46:0/2853683553 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2fd8067050 0x7f2fd8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:09.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.480+0000 7f2ff1086700 1 --2- 172.21.15.46:0/2853683553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fec0b2a80 0x7f2fec154d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:09.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.480+0000 7f2ff1086700 1 --2- 172.21.15.46:0/2853683553 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fec157a30 0x7f2fec0b4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:09.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.480+0000 7f2ff1086700 1 --2- 172.21.15.46:0/2853683553 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fec159190 0x7f2fec0c6540 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:09.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.480+0000 7f2ff1086700 1 -- 172.21.15.46:0/2853683553 >> 172.21.15.46:0/2853683553 conn(0x7f2fec0ce570 msgr2=0x7f2fec149d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:09.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.480+0000 7f2ff1086700 1 -- 172.21.15.46:0/2853683553 shutdown_connections 2024-01-27T01:06:09.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.480+0000 7f2ff1086700 1 -- 172.21.15.46:0/2853683553 wait complete. 2024-01-27T01:06:09.494 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-27T01:06:09.593 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:09.593+0000 7f3d64b1e1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:06:09.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.673+0000 7ffb0602d700 1 Processor -- start 2024-01-27T01:06:09.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.673+0000 7ffb0602d700 1 -- start start 2024-01-27T01:06:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.674+0000 7ffb0602d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffb001597e0 0x7ffb000abe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.674+0000 7ffb0602d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ffb0006aef0 0x7ffb0006b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.674+0000 7ffb0602d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb0006bcd0 con 0x7ffb0006b8a0 2024-01-27T01:06:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.674+0000 7ffb0602d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb000a9af0 con 0x7ffb0006aef0 2024-01-27T01:06:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.674+0000 7ffb0602d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb00068a90 con 0x7ffb001597e0 2024-01-27T01:06:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.674+0000 7ffafffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ffb0006b8a0 0x7ffb0015d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58034/0 (socket says 172.21.15.46:58034) 2024-01-27T01:06:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.674+0000 7ffafffff700 1 -- 172.21.15.46:0/2922122484 learned_addr learned my addr 172.21.15.46:0/2922122484 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.674+0000 7ffaff7fe700 1 --2- 172.21.15.46:0/2922122484 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffb001597e0 0x7ffb000abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.674+0000 7ffafeffd700 1 --2- 172.21.15.46:0/2922122484 >> v2:172.21.15.106:3300/0 conn(0x7ffb0006aef0 0x7ffb0006b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.674+0000 7ffafe7fc700 1 -- 172.21.15.46:0/2922122484 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730266308 0 0) 0x7ffb0006bcd0 con 0x7ffb0006b8a0 2024-01-27T01:06:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.674+0000 7ffafe7fc700 1 -- 172.21.15.46:0/2922122484 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaec003570 con 0x7ffb0006b8a0 2024-01-27T01:06:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.675+0000 7ffaff7fe700 1 -- 172.21.15.46:0/2922122484 >> v2:172.21.15.106:3300/0 conn(0x7ffb0006aef0 msgr2=0x7ffb0006b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.675+0000 7ffaff7fe700 1 --2- 172.21.15.46:0/2922122484 >> v2:172.21.15.106:3300/0 conn(0x7ffb0006aef0 0x7ffb0006b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.675+0000 7ffaff7fe700 1 -- 172.21.15.46:0/2922122484 >> v1:172.21.15.46:6789/0 conn(0x7ffb0006b8a0 legacy=0x7ffb0015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.675+0000 7ffaff7fe700 1 -- 172.21.15.46:0/2922122484 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb00068d70 con 0x7ffb001597e0 2024-01-27T01:06:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.675+0000 7ffafeffd700 1 --2- 172.21.15.46:0/2922122484 >> v2:172.21.15.106:3300/0 conn(0x7ffb0006aef0 0x7ffb0006b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.675+0000 7ffaff7fe700 1 --2- 172.21.15.46:0/2922122484 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffb001597e0 0x7ffb000abe60 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00098d0 tx=0x7ffaf0009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7cc2701b8fc685d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.675+0000 7ffafe7fc700 1 -- 172.21.15.46:0/2922122484 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0005c10 con 0x7ffb001597e0 2024-01-27T01:06:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.675+0000 7ffafe7fc700 1 -- 172.21.15.46:0/2922122484 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf0005d70 con 0x7ffb001597e0 2024-01-27T01:06:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.675+0000 7ffafe7fc700 1 -- 172.21.15.46:0/2922122484 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0006280 con 0x7ffb001597e0 2024-01-27T01:06:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.676+0000 7ffb0602d700 1 -- 172.21.15.46:0/2922122484 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffb001597e0 msgr2=0x7ffb000abe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.676+0000 7ffb0602d700 1 --2- 172.21.15.46:0/2922122484 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffb001597e0 0x7ffb000abe60 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00098d0 tx=0x7ffaf0009120 comp rx=0 tx=0).stop 2024-01-27T01:06:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.676+0000 7ffb0602d700 1 -- 172.21.15.46:0/2922122484 shutdown_connections 2024-01-27T01:06:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.676+0000 7ffb0602d700 1 --2- 172.21.15.46:0/2922122484 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffb001597e0 0x7ffb000abe60 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.676+0000 7ffb0602d700 1 --2- 172.21.15.46:0/2922122484 >> v2:172.21.15.106:3300/0 conn(0x7ffb0006aef0 0x7ffb0006b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.676+0000 7ffb0602d700 1 -- 172.21.15.46:0/2922122484 >> 172.21.15.46:0/2922122484 conn(0x7ffb0006e2c0 msgr2=0x7ffb000ae620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.676+0000 7ffb0602d700 1 -- 172.21.15.46:0/2922122484 shutdown_connections 2024-01-27T01:06:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.676+0000 7ffb0602d700 1 -- 172.21.15.46:0/2922122484 wait complete. 2024-01-27T01:06:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.677+0000 7ffb0602d700 1 Processor -- start 2024-01-27T01:06:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.677+0000 7ffb0602d700 1 -- start start 2024-01-27T01:06:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.678+0000 7ffb0602d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffb0006aef0 0x7ffb001571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.678+0000 7ffb0602d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb0006b8a0 0x7ffb0015a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.678+0000 7ffb0602d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb001597e0 0x7ffb00146580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.678+0000 7ffb0602d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb000c8c50 con 0x7ffb001597e0 2024-01-27T01:06:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.678+0000 7ffb0602d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0015aab0 con 0x7ffb0006b8a0 2024-01-27T01:06:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.678+0000 7ffb0602d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0015ad80 con 0x7ffb0006aef0 2024-01-27T01:06:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.678+0000 7ffafffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb001597e0 0x7ffb00146580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.678+0000 7ffafffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb001597e0 0x7ffb00146580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44772/0 (socket says 172.21.15.46:44772) 2024-01-27T01:06:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.678+0000 7ffafffff700 1 -- 172.21.15.46:0/1512928607 learned_addr learned my addr 172.21.15.46:0/1512928607 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.678+0000 7ffaff7fe700 1 --2- 172.21.15.46:0/1512928607 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffb0006aef0 0x7ffb001571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.678+0000 7ffafeffd700 1 --2- 172.21.15.46:0/1512928607 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb0006b8a0 0x7ffb0015a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.679+0000 7ffafffff700 1 -- 172.21.15.46:0/1512928607 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffb0006aef0 msgr2=0x7ffb001571a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.679+0000 7ffafffff700 1 --2- 172.21.15.46:0/1512928607 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffb0006aef0 0x7ffb001571a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.679+0000 7ffafffff700 1 -- 172.21.15.46:0/1512928607 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb0006b8a0 msgr2=0x7ffb0015a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.679+0000 7ffafffff700 1 --2- 172.21.15.46:0/1512928607 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb0006b8a0 0x7ffb0015a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.679+0000 7ffafffff700 1 -- 172.21.15.46:0/1512928607 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae8009420 con 0x7ffb001597e0 2024-01-27T01:06:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.679+0000 7ffafeffd700 1 --2- 172.21.15.46:0/1512928607 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb0006b8a0 0x7ffb0015a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.679+0000 7ffaff7fe700 1 --2- 172.21.15.46:0/1512928607 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffb0006aef0 0x7ffb001571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.679+0000 7ffafffff700 1 --2- 172.21.15.46:0/1512928607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb001597e0 0x7ffb00146580 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8001460 tx=0x7ffaf8001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.679+0000 7ffaf77fe700 1 -- 172.21.15.46:0/1512928607 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf800ce80 con 0x7ffb001597e0 2024-01-27T01:06:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.679+0000 7ffaf77fe700 1 -- 172.21.15.46:0/1512928607 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf800ab70 con 0x7ffb001597e0 2024-01-27T01:06:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.679+0000 7ffb0602d700 1 -- 172.21.15.46:0/1512928607 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf00093c0 con 0x7ffb001597e0 2024-01-27T01:06:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.679+0000 7ffaf77fe700 1 -- 172.21.15.46:0/1512928607 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf80070f0 con 0x7ffb001597e0 2024-01-27T01:06:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.679+0000 7ffb0602d700 1 -- 172.21.15.46:0/1512928607 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb0015b340 con 0x7ffb001597e0 2024-01-27T01:06:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.680+0000 7ffb0602d700 1 -- 172.21.15.46:0/1512928607 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb000c8e20 con 0x7ffb001597e0 2024-01-27T01:06:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.681+0000 7ffaf77fe700 1 -- 172.21.15.46:0/1512928607 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 123) v1 ==== 84453+0+0 (secure 0 0 0) 0x7ffaf8007520 con 0x7ffb001597e0 2024-01-27T01:06:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.681+0000 7ffaf77fe700 1 -- 172.21.15.46:0/1512928607 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 7206+0+0 (secure 0 0 0) 0x7ffaf8084ac0 con 0x7ffb001597e0 2024-01-27T01:06:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.686+0000 7ffaf77fe700 1 -- 172.21.15.46:0/1512928607 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffb000c8e20 con 0x7ffb001597e0 2024-01-27T01:06:09.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.736+0000 7ffaf77fe700 1 -- 172.21.15.46:0/1512928607 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 124) v1 ==== 84580+0+0 (secure 0 0 0) 0x7ffaf8051640 con 0x7ffb001597e0 2024-01-27T01:06:09.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.737+0000 7ffaf77fe700 1 --2- 172.21.15.46:0/1512928607 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ffae006d230 0x7ffae006f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:09.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.737+0000 7ffaff7fe700 1 --2- 172.21.15.46:0/1512928607 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ffae006d230 0x7ffae006f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:09.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.738+0000 7ffaff7fe700 1 --2- 172.21.15.46:0/1512928607 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ffae006d230 0x7ffae006f610 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00098d0 tx=0x7ffaf0016000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:09.895+0000 7ffb0602d700 1 -- 172.21.15.46:0/1512928607 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ffb0004c710 con 0x7ffb001597e0 2024-01-27T01:06:10.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.744+0000 7ffaf77fe700 1 -- 172.21.15.46:0/1512928607 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 125) v1 ==== 84686+0+0 (secure 0 0 0) 0x7ffaf8081b20 con 0x7ffb001597e0 2024-01-27T01:06:10.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.745+0000 7ffaf77fe700 1 -- 172.21.15.46:0/1512928607 <== mon.0 v2:172.21.15.46:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v213) v1 ==== 181+0+0 (secure 0 0 0) 0x7ffaf8052ba0 con 0x7ffb001597e0 2024-01-27T01:06:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.752+0000 7ffb0602d700 1 -- 172.21.15.46:0/1512928607 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ffae006d230 msgr2=0x7ffae006f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.752+0000 7ffb0602d700 1 --2- 172.21.15.46:0/1512928607 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ffae006d230 0x7ffae006f610 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00098d0 tx=0x7ffaf0016000 comp rx=0 tx=0).stop 2024-01-27T01:06:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.752+0000 7ffb0602d700 1 -- 172.21.15.46:0/1512928607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb001597e0 msgr2=0x7ffb00146580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.752+0000 7ffb0602d700 1 --2- 172.21.15.46:0/1512928607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb001597e0 0x7ffb00146580 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8001460 tx=0x7ffaf8001ae0 comp rx=0 tx=0).stop 2024-01-27T01:06:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.752+0000 7ffb0602d700 1 -- 172.21.15.46:0/1512928607 shutdown_connections 2024-01-27T01:06:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.752+0000 7ffb0602d700 1 --2- 172.21.15.46:0/1512928607 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ffae006d230 0x7ffae006f610 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:10.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.752+0000 7ffb0602d700 1 --2- 172.21.15.46:0/1512928607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb001597e0 0x7ffb00146580 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.752+0000 7ffb0602d700 1 --2- 172.21.15.46:0/1512928607 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffb0006aef0 0x7ffb001571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.752+0000 7ffb0602d700 1 --2- 172.21.15.46:0/1512928607 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb0006b8a0 0x7ffb0015a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.752+0000 7ffb0602d700 1 -- 172.21.15.46:0/1512928607 >> 172.21.15.46:0/1512928607 conn(0x7ffb0006e2c0 msgr2=0x7ffb000b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.752+0000 7ffb0602d700 1 -- 172.21.15.46:0/1512928607 shutdown_connections 2024-01-27T01:06:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.752+0000 7ffb0602d700 1 -- 172.21.15.46:0/1512928607 wait complete. 2024-01-27T01:06:10.753 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-01-27T01:06:10.768 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-27T01:06:10.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.911+0000 7f20a7f50700 1 Processor -- start 2024-01-27T01:06:10.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.911+0000 7f20a7f50700 1 -- start start 2024-01-27T01:06:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.912+0000 7f20a7f50700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f20a0156b00 0x7f20a015ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.912+0000 7f20a7f50700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a0157000 con 0x7f20a00c9d50 2024-01-27T01:06:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.912+0000 7f20a7f50700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a006f3e0 con 0x7f20a00c9980 2024-01-27T01:06:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.912+0000 7f20a7f50700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a006f6c0 con 0x7f20a0156b00 2024-01-27T01:06:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.912+0000 7f20a54eb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f20a0156b00 0x7f20a015ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.912+0000 7f20a54eb700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f20a0156b00 0x7f20a015ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57630/0 (socket says 172.21.15.46:57630) 2024-01-27T01:06:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.912+0000 7f20a64ed700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f20a00c9d50 0x7f20a00b1aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41128/0 (socket says 172.21.15.46:41128) 2024-01-27T01:06:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.912+0000 7f20a54eb700 1 -- 172.21.15.46:0/1539443330 learned_addr learned my addr 172.21.15.46:0/1539443330 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.913+0000 7f20a54eb700 1 -- 172.21.15.46:0/1539443330 >> v1:172.21.15.106:6789/0 conn(0x7f20a00c9980 legacy=0x7f20a006eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.913+0000 7f20a54eb700 1 -- 172.21.15.46:0/1539443330 >> v1:172.21.15.46:6789/0 conn(0x7f20a00c9d50 legacy=0x7f20a00b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.913+0000 7f20a4cea700 1 -- 172.21.15.46:0/1539443330 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885244460 0 0) 0x7f20a006f3e0 con 0x7f20a00c9980 2024-01-27T01:06:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.913+0000 7f20a54eb700 1 -- 172.21.15.46:0/1539443330 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a006f9a0 con 0x7f20a0156b00 2024-01-27T01:06:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.913+0000 7f20a54eb700 1 --2- 172.21.15.46:0/1539443330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a0156b00 0x7f20a015ae10 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f209c0099b0 tx=0x7f209c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30da46e3d15fdf51 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.913+0000 7f20a4cea700 1 -- 172.21.15.46:0/1539443330 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c00b760 con 0x7f20a0156b00 2024-01-27T01:06:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.913+0000 7f20a4cea700 1 -- 172.21.15.46:0/1539443330 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209c00b8c0 con 0x7f20a0156b00 2024-01-27T01:06:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.913+0000 7f20a4cea700 1 -- 172.21.15.46:0/1539443330 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c00bc80 con 0x7f20a0156b00 2024-01-27T01:06:10.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.914+0000 7f20a7f50700 1 -- 172.21.15.46:0/1539443330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a0156b00 msgr2=0x7f20a015ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:10.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.914+0000 7f20a7f50700 1 --2- 172.21.15.46:0/1539443330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a0156b00 0x7f20a015ae10 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f209c0099b0 tx=0x7f209c009250 comp rx=0 tx=0).stop 2024-01-27T01:06:10.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.914+0000 7f20a7f50700 1 -- 172.21.15.46:0/1539443330 shutdown_connections 2024-01-27T01:06:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.914+0000 7f20a7f50700 1 --2- 172.21.15.46:0/1539443330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a0156b00 0x7f20a015ae10 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.914+0000 7f20a7f50700 1 -- 172.21.15.46:0/1539443330 >> 172.21.15.46:0/1539443330 conn(0x7f20a0069510 msgr2=0x7f20a00aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.914+0000 7f20a7f50700 1 -- 172.21.15.46:0/1539443330 shutdown_connections 2024-01-27T01:06:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.914+0000 7f20a7f50700 1 -- 172.21.15.46:0/1539443330 wait complete. 2024-01-27T01:06:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.915+0000 7f20a7f50700 1 Processor -- start 2024-01-27T01:06:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.915+0000 7f20a7f50700 1 -- start start 2024-01-27T01:06:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.916+0000 7f20a7f50700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a00c9980 0x7f20a016bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.916+0000 7f20a7f50700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20a00c9d50 0x7f20a00cd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.916+0000 7f20a7f50700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20a0156b00 0x7f20a014bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.916+0000 7f20a7f50700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a00cd660 con 0x7f20a00c9980 2024-01-27T01:06:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.916+0000 7f20a7f50700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a00cb790 con 0x7f20a0156b00 2024-01-27T01:06:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.916+0000 7f20a7f50700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a00cd870 con 0x7f20a00c9d50 2024-01-27T01:06:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.916+0000 7f20a5cec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a00c9980 0x7f20a016bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.916+0000 7f20a5cec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a00c9980 0x7f20a016bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57634/0 (socket says 172.21.15.46:57634) 2024-01-27T01:06:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.916+0000 7f20a5cec700 1 -- 172.21.15.46:0/2215269034 learned_addr learned my addr 172.21.15.46:0/2215269034 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.916+0000 7f20a64ed700 1 --2- 172.21.15.46:0/2215269034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20a0156b00 0x7f20a014bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.916+0000 7f20a54eb700 1 --2- 172.21.15.46:0/2215269034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20a00c9d50 0x7f20a00cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.916+0000 7f20a5cec700 1 -- 172.21.15.46:0/2215269034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20a00c9d50 msgr2=0x7f20a00cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.916+0000 7f20a5cec700 1 --2- 172.21.15.46:0/2215269034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20a00c9d50 0x7f20a00cd120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.916+0000 7f20a5cec700 1 -- 172.21.15.46:0/2215269034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20a0156b00 msgr2=0x7f20a014bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.916+0000 7f20a5cec700 1 --2- 172.21.15.46:0/2215269034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20a0156b00 0x7f20a014bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.916+0000 7f20a5cec700 1 -- 172.21.15.46:0/2215269034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209c0094f0 con 0x7f20a00c9980 2024-01-27T01:06:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.916+0000 7f20a64ed700 1 --2- 172.21.15.46:0/2215269034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20a0156b00 0x7f20a014bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.917+0000 7f20a5cec700 1 --2- 172.21.15.46:0/2215269034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a00c9980 0x7f20a016bfc0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f209400c810 tx=0x7f2094007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.917+0000 7f2091ffb700 1 -- 172.21.15.46:0/2215269034 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2094012700 con 0x7f20a00c9980 2024-01-27T01:06:10.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.917+0000 7f2091ffb700 1 -- 172.21.15.46:0/2215269034 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20940128a0 con 0x7f20a00c9980 2024-01-27T01:06:10.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.917+0000 7f20a7f50700 1 -- 172.21.15.46:0/2215269034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a00cae60 con 0x7f20a00c9980 2024-01-27T01:06:10.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.917+0000 7f2091ffb700 1 -- 172.21.15.46:0/2215269034 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2094012c60 con 0x7f20a00c9980 2024-01-27T01:06:10.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.917+0000 7f20a7f50700 1 -- 172.21.15.46:0/2215269034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a00cb420 con 0x7f20a00c9980 2024-01-27T01:06:10.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.918+0000 7f20a7f50700 1 -- 172.21.15.46:0/2215269034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a004c710 con 0x7f20a00c9980 2024-01-27T01:06:10.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.918+0000 7f2091ffb700 1 -- 172.21.15.46:0/2215269034 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 125) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f209401b400 con 0x7f20a00c9980 2024-01-27T01:06:10.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.919+0000 7f2091ffb700 1 --2- 172.21.15.46:0/2215269034 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2084066ea0 0x7f2084069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:10.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.919+0000 7f2091ffb700 1 -- 172.21.15.46:0/2215269034 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6768+0+0 (secure 0 0 0) 0x7f2094084450 con 0x7f20a00c9980 2024-01-27T01:06:10.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.919+0000 7f20a54eb700 1 --2- 172.21.15.46:0/2215269034 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2084066ea0 0x7f2084069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:10.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.920+0000 7f20a54eb700 1 --2- 172.21.15.46:0/2215269034 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2084066ea0 0x7f2084069350 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f209c0099b0 tx=0x7f209c016000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:10.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:10.923+0000 7f2091ffb700 1 -- 172.21.15.46:0/2215269034 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2094051380 con 0x7f20a00c9980 2024-01-27T01:06:11.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.134+0000 7f20a7f50700 1 -- 172.21.15.46:0/2215269034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f20a00b2900 con 0x7f20a00c9980 2024-01-27T01:06:11.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.748+0000 7f2091ffb700 1 -- 172.21.15.46:0/2215269034 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v214) v1 ==== 169+0+0 (secure 0 0 0) 0x7f20940549a0 con 0x7f20a00c9980 2024-01-27T01:06:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.754+0000 7f20a7f50700 1 -- 172.21.15.46:0/2215269034 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2084066ea0 msgr2=0x7f2084069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.754+0000 7f20a7f50700 1 --2- 172.21.15.46:0/2215269034 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2084066ea0 0x7f2084069350 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f209c0099b0 tx=0x7f209c016000 comp rx=0 tx=0).stop 2024-01-27T01:06:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.754+0000 7f20a7f50700 1 -- 172.21.15.46:0/2215269034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a00c9980 msgr2=0x7f20a016bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.754+0000 7f20a7f50700 1 --2- 172.21.15.46:0/2215269034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a00c9980 0x7f20a016bfc0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f209400c810 tx=0x7f2094007df0 comp rx=0 tx=0).stop 2024-01-27T01:06:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.754+0000 7f20a7f50700 1 -- 172.21.15.46:0/2215269034 shutdown_connections 2024-01-27T01:06:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.754+0000 7f20a7f50700 1 --2- 172.21.15.46:0/2215269034 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2084066ea0 0x7f2084069350 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.754+0000 7f20a7f50700 1 --2- 172.21.15.46:0/2215269034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a00c9980 0x7f20a016bfc0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.754+0000 7f20a7f50700 1 --2- 172.21.15.46:0/2215269034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20a0156b00 0x7f20a014bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.754+0000 7f20a7f50700 1 --2- 172.21.15.46:0/2215269034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20a00c9d50 0x7f20a00cd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.754+0000 7f20a7f50700 1 -- 172.21.15.46:0/2215269034 >> 172.21.15.46:0/2215269034 conn(0x7f20a0069510 msgr2=0x7f20a0157ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:11.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.757+0000 7f20a7f50700 1 -- 172.21.15.46:0/2215269034 shutdown_connections 2024-01-27T01:06:11.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.757+0000 7f20a7f50700 1 -- 172.21.15.46:0/2215269034 wait complete. 2024-01-27T01:06:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-01-27T01:06:11.779 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-27T01:06:11.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.918+0000 7f28777f3700 1 Processor -- start 2024-01-27T01:06:11.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.919+0000 7f28777f3700 1 -- start start 2024-01-27T01:06:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.919+0000 7f28777f3700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f287006ebe0 0x7f2870067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.919+0000 7f28777f3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f28701597e0 0x7f2870159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.919+0000 7f28777f3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f287006fdb0 con 0x7f2870067af0 2024-01-27T01:06:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.919+0000 7f28777f3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f287006f1a0 con 0x7f28701597e0 2024-01-27T01:06:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.919+0000 7f28777f3700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f287006f480 con 0x7f287006ebe0 2024-01-27T01:06:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.920+0000 7f2875d90700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f28701597e0 0x7f2870159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.920+0000 7f2875d90700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f28701597e0 0x7f2870159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57650/0 (socket says 172.21.15.46:57650) 2024-01-27T01:06:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.920+0000 7f2875d90700 1 -- 172.21.15.46:0/4028422833 learned_addr learned my addr 172.21.15.46:0/4028422833 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.920+0000 7f2874d8e700 1 --2- 172.21.15.46:0/4028422833 >> v2:172.21.15.106:3300/0 conn(0x7f287006ebe0 0x7f2870067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.920+0000 7f2875d90700 1 -- 172.21.15.46:0/4028422833 >> v2:172.21.15.106:3300/0 conn(0x7f287006ebe0 msgr2=0x7f2870067ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.920+0000 7f2875d90700 1 --2- 172.21.15.46:0/4028422833 >> v2:172.21.15.106:3300/0 conn(0x7f287006ebe0 0x7f2870067ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.920+0000 7f2875d90700 1 -- 172.21.15.46:0/4028422833 >> v1:172.21.15.106:6789/0 conn(0x7f2870067af0 legacy=0x7f28700ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.920+0000 7f2875d90700 1 -- 172.21.15.46:0/4028422833 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287006f760 con 0x7f28701597e0 2024-01-27T01:06:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.920+0000 7f2875d90700 1 --2- 172.21.15.46:0/4028422833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28701597e0 0x7f2870159bf0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f28700ce590 tx=0x7f2868009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e0a70e286dc252 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.920+0000 7f2863fff700 1 -- 172.21.15.46:0/4028422833 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286800e070 con 0x7f28701597e0 2024-01-27T01:06:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.920+0000 7f2863fff700 1 -- 172.21.15.46:0/4028422833 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286800ba10 con 0x7f28701597e0 2024-01-27T01:06:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.920+0000 7f2863fff700 1 -- 172.21.15.46:0/4028422833 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28680092e0 con 0x7f28701597e0 2024-01-27T01:06:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.921+0000 7f28777f3700 1 -- 172.21.15.46:0/4028422833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28701597e0 msgr2=0x7f2870159bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.921+0000 7f28777f3700 1 --2- 172.21.15.46:0/4028422833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28701597e0 0x7f2870159bf0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f28700ce590 tx=0x7f2868009840 comp rx=0 tx=0).stop 2024-01-27T01:06:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.921+0000 7f28777f3700 1 -- 172.21.15.46:0/4028422833 shutdown_connections 2024-01-27T01:06:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.921+0000 7f28777f3700 1 --2- 172.21.15.46:0/4028422833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28701597e0 0x7f2870159bf0 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.921+0000 7f28777f3700 1 --2- 172.21.15.46:0/4028422833 >> v2:172.21.15.106:3300/0 conn(0x7f287006ebe0 0x7f2870067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.921+0000 7f28777f3700 1 -- 172.21.15.46:0/4028422833 >> 172.21.15.46:0/4028422833 conn(0x7f28700cdf80 msgr2=0x7f287006acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.921+0000 7f28777f3700 1 -- 172.21.15.46:0/4028422833 shutdown_connections 2024-01-27T01:06:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.921+0000 7f28777f3700 1 -- 172.21.15.46:0/4028422833 wait complete. 2024-01-27T01:06:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.922+0000 7f28777f3700 1 Processor -- start 2024-01-27T01:06:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.922+0000 7f28777f3700 1 -- start start 2024-01-27T01:06:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.923+0000 7f28777f3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2870067af0 0x7f287014d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.923+0000 7f28777f3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f287006ebe0 0x7f287015cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.923+0000 7f28777f3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28701597e0 0x7f28700b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.923+0000 7f28777f3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287015a670 con 0x7f28701597e0 2024-01-27T01:06:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.923+0000 7f28777f3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287014ea70 con 0x7f287006ebe0 2024-01-27T01:06:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.923+0000 7f28777f3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f287014f8d0 con 0x7f2870067af0 2024-01-27T01:06:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.923+0000 7f2875d90700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28701597e0 0x7f28700b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.923+0000 7f2875d90700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28701597e0 0x7f28700b18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57656/0 (socket says 172.21.15.46:57656) 2024-01-27T01:06:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.923+0000 7f2875d90700 1 -- 172.21.15.46:0/2910130227 learned_addr learned my addr 172.21.15.46:0/2910130227 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.924+0000 7f2874d8e700 1 --2- 172.21.15.46:0/2910130227 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f287006ebe0 0x7f287015cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.924+0000 7f287558f700 1 --2- 172.21.15.46:0/2910130227 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2870067af0 0x7f287014d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.924+0000 7f2875d90700 1 -- 172.21.15.46:0/2910130227 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2870067af0 msgr2=0x7f287014d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.924+0000 7f2875d90700 1 --2- 172.21.15.46:0/2910130227 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2870067af0 0x7f287014d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.924+0000 7f2875d90700 1 -- 172.21.15.46:0/2910130227 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f287006ebe0 msgr2=0x7f287015cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.924+0000 7f2875d90700 1 --2- 172.21.15.46:0/2910130227 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f287006ebe0 0x7f287015cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.924+0000 7f2875d90700 1 -- 172.21.15.46:0/2910130227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2868009ae0 con 0x7f28701597e0 2024-01-27T01:06:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.924+0000 7f2875d90700 1 --2- 172.21.15.46:0/2910130227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28701597e0 0x7f28700b18e0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f2868012970 tx=0x7f2868017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.924+0000 7f28617fa700 1 -- 172.21.15.46:0/2910130227 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2868009840 con 0x7f28701597e0 2024-01-27T01:06:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.924+0000 7f28617fa700 1 -- 172.21.15.46:0/2910130227 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28680092e0 con 0x7f28701597e0 2024-01-27T01:06:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.924+0000 7f28777f3700 1 -- 172.21.15.46:0/2910130227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f287014ecc0 con 0x7f28701597e0 2024-01-27T01:06:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.924+0000 7f28617fa700 1 -- 172.21.15.46:0/2910130227 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286801d650 con 0x7f28701597e0 2024-01-27T01:06:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.924+0000 7f28777f3700 1 -- 172.21.15.46:0/2910130227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f287014f280 con 0x7f28701597e0 2024-01-27T01:06:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.925+0000 7f28777f3700 1 -- 172.21.15.46:0/2910130227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f287004c710 con 0x7f28701597e0 2024-01-27T01:06:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.926+0000 7f28617fa700 1 -- 172.21.15.46:0/2910130227 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 125) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f286801da00 con 0x7f28701597e0 2024-01-27T01:06:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.926+0000 7f28617fa700 1 --2- 172.21.15.46:0/2910130227 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2854066ea0 0x7f2854069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.926+0000 7f28617fa700 1 -- 172.21.15.46:0/2910130227 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6267+0+0 (secure 0 0 0) 0x7f2868085d10 con 0x7f28701597e0 2024-01-27T01:06:11.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.927+0000 7f287558f700 1 --2- 172.21.15.46:0/2910130227 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2854066ea0 0x7f2854069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:11.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.927+0000 7f287558f700 1 --2- 172.21.15.46:0/2910130227 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2854066ea0 0x7f2854069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2864006fd0 tx=0x7f2864008040 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:11.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:11.931+0000 7f28617fa700 1 -- 172.21.15.46:0/2910130227 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f286808c080 con 0x7f28701597e0 2024-01-27T01:06:12.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.152+0000 7f28777f3700 1 -- 172.21.15.46:0/2910130227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f287014fa10 con 0x7f28701597e0 2024-01-27T01:06:12.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.156+0000 7f28617fa700 1 -- 172.21.15.46:0/2910130227 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v73) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2868052d90 con 0x7f28701597e0 2024-01-27T01:06:12.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.158+0000 7f28777f3700 1 -- 172.21.15.46:0/2910130227 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2854066ea0 msgr2=0x7f2854069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:12.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.158+0000 7f28777f3700 1 --2- 172.21.15.46:0/2910130227 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2854066ea0 0x7f2854069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2864006fd0 tx=0x7f2864008040 comp rx=0 tx=0).stop 2024-01-27T01:06:12.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.158+0000 7f28777f3700 1 -- 172.21.15.46:0/2910130227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28701597e0 msgr2=0x7f28700b18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:12.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.158+0000 7f28777f3700 1 --2- 172.21.15.46:0/2910130227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28701597e0 0x7f28700b18e0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f2868012970 tx=0x7f2868017000 comp rx=0 tx=0).stop 2024-01-27T01:06:12.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.158+0000 7f28777f3700 1 -- 172.21.15.46:0/2910130227 shutdown_connections 2024-01-27T01:06:12.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.158+0000 7f28777f3700 1 --2- 172.21.15.46:0/2910130227 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2854066ea0 0x7f2854069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.158+0000 7f28777f3700 1 --2- 172.21.15.46:0/2910130227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28701597e0 0x7f28700b18e0 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.158+0000 7f28777f3700 1 --2- 172.21.15.46:0/2910130227 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2870067af0 0x7f287014d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.158+0000 7f28777f3700 1 --2- 172.21.15.46:0/2910130227 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f287006ebe0 0x7f287015cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.158+0000 7f28777f3700 1 -- 172.21.15.46:0/2910130227 >> 172.21.15.46:0/2910130227 conn(0x7f28700cdf80 msgr2=0x7f28700cbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:12.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.158+0000 7f28777f3700 1 -- 172.21.15.46:0/2910130227 shutdown_connections 2024-01-27T01:06:12.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.158+0000 7f28777f3700 1 -- 172.21.15.46:0/2910130227 wait complete. 2024-01-27T01:06:12.172 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-27T01:06:12.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.305+0000 7faa6c723700 1 Processor -- start 2024-01-27T01:06:12.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.305+0000 7faa6c723700 1 -- start start 2024-01-27T01:06:12.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.306+0000 7faa6c723700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faa64068940 0x7faa64068d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:12.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.306+0000 7faa6c723700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7faa640c90c0 0x7faa640af990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:12.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.306+0000 7faa6c723700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa640c87c0 con 0x7faa640c8c50 2024-01-27T01:06:12.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.306+0000 7faa6c723700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa64069250 con 0x7faa64068940 2024-01-27T01:06:12.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.306+0000 7faa6c723700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa640c7bb0 con 0x7faa640c90c0 2024-01-27T01:06:12.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.306+0000 7faa6a4bf700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faa64068940 0x7faa64068d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:12.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.306+0000 7faa6a4bf700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faa64068940 0x7faa64068d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57672/0 (socket says 172.21.15.46:57672) 2024-01-27T01:06:12.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.306+0000 7faa69cbe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faa640c8c50 0x7faa640c7160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41144/0 (socket says 172.21.15.46:41144) 2024-01-27T01:06:12.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.306+0000 7faa6a4bf700 1 -- 172.21.15.46:0/445274148 learned_addr learned my addr 172.21.15.46:0/445274148 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:12.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.306+0000 7faa6acc0700 1 --2- 172.21.15.46:0/445274148 >> v2:172.21.15.106:3300/0 conn(0x7faa640c90c0 0x7faa640af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:12.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.306+0000 7faa6a4bf700 1 -- 172.21.15.46:0/445274148 >> v2:172.21.15.106:3300/0 conn(0x7faa640c90c0 msgr2=0x7faa640af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:12.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.306+0000 7faa6a4bf700 1 --2- 172.21.15.46:0/445274148 >> v2:172.21.15.106:3300/0 conn(0x7faa640c90c0 0x7faa640af990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.306+0000 7faa6a4bf700 1 -- 172.21.15.46:0/445274148 >> v1:172.21.15.46:6789/0 conn(0x7faa640c8c50 legacy=0x7faa640c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:12.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.306+0000 7faa6a4bf700 1 -- 172.21.15.46:0/445274148 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa640c7e90 con 0x7faa64068940 2024-01-27T01:06:12.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.306+0000 7faa694bd700 1 -- 172.21.15.46:0/445274148 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079784795 0 0) 0x7faa640c87c0 con 0x7faa640c8c50 2024-01-27T01:06:12.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.306+0000 7faa6acc0700 1 --2- 172.21.15.46:0/445274148 >> v2:172.21.15.106:3300/0 conn(0x7faa640c90c0 0x7faa640af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:12.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.307+0000 7faa6a4bf700 1 --2- 172.21.15.46:0/445274148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa64068940 0x7faa64068d10 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7faa640c8730 tx=0x7faa58009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52cb996e9e5f40f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:12.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.307+0000 7faa694bd700 1 -- 172.21.15.46:0/445274148 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa58005c10 con 0x7faa64068940 2024-01-27T01:06:12.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.307+0000 7faa694bd700 1 -- 172.21.15.46:0/445274148 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa58005d70 con 0x7faa64068940 2024-01-27T01:06:12.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.307+0000 7faa694bd700 1 -- 172.21.15.46:0/445274148 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5800b880 con 0x7faa64068940 2024-01-27T01:06:12.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.307+0000 7faa6c723700 1 -- 172.21.15.46:0/445274148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa64068940 msgr2=0x7faa64068d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:12.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.307+0000 7faa6c723700 1 --2- 172.21.15.46:0/445274148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa64068940 0x7faa64068d10 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7faa640c8730 tx=0x7faa58009220 comp rx=0 tx=0).stop 2024-01-27T01:06:12.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.307+0000 7faa6c723700 1 -- 172.21.15.46:0/445274148 shutdown_connections 2024-01-27T01:06:12.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.307+0000 7faa6c723700 1 --2- 172.21.15.46:0/445274148 >> v2:172.21.15.106:3300/0 conn(0x7faa640c90c0 0x7faa640af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.307+0000 7faa6c723700 1 --2- 172.21.15.46:0/445274148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa64068940 0x7faa64068d10 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.307+0000 7faa6c723700 1 -- 172.21.15.46:0/445274148 >> 172.21.15.46:0/445274148 conn(0x7faa640ca100 msgr2=0x7faa6406e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:12.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.308+0000 7faa6c723700 1 -- 172.21.15.46:0/445274148 shutdown_connections 2024-01-27T01:06:12.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.308+0000 7faa6c723700 1 -- 172.21.15.46:0/445274148 wait complete. 2024-01-27T01:06:12.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.309+0000 7faa6c723700 1 Processor -- start 2024-01-27T01:06:12.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.309+0000 7faa6c723700 1 -- start start 2024-01-27T01:06:12.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.310+0000 7faa6c723700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa64068940 0x7faa6415c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:12.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.310+0000 7faa6c723700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa640c8c50 0x7faa6414cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:12.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.310+0000 7faa6c723700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa640c90c0 0x7faa6414d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:12.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.310+0000 7faa6c723700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6415cd70 con 0x7faa640c90c0 2024-01-27T01:06:12.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.310+0000 7faa6c723700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa640c15f0 con 0x7faa64068940 2024-01-27T01:06:12.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.310+0000 7faa6c723700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa640c1bb0 con 0x7faa640c8c50 2024-01-27T01:06:12.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.310+0000 7faa6acc0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa640c90c0 0x7faa6414d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:12.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.310+0000 7faa6acc0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa640c90c0 0x7faa6414d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57686/0 (socket says 172.21.15.46:57686) 2024-01-27T01:06:12.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.310+0000 7faa6acc0700 1 -- 172.21.15.46:0/3191556701 learned_addr learned my addr 172.21.15.46:0/3191556701 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:12.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.311+0000 7faa6a4bf700 1 --2- 172.21.15.46:0/3191556701 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa64068940 0x7faa6415c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:12.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.311+0000 7faa69cbe700 1 --2- 172.21.15.46:0/3191556701 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa640c8c50 0x7faa6414cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:12.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.311+0000 7faa6acc0700 1 -- 172.21.15.46:0/3191556701 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa640c8c50 msgr2=0x7faa6414cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:12.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.311+0000 7faa6acc0700 1 --2- 172.21.15.46:0/3191556701 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa640c8c50 0x7faa6414cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.311+0000 7faa6acc0700 1 -- 172.21.15.46:0/3191556701 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa64068940 msgr2=0x7faa6415c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:12.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.311+0000 7faa6acc0700 1 --2- 172.21.15.46:0/3191556701 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa64068940 0x7faa6415c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.311+0000 7faa6acc0700 1 -- 172.21.15.46:0/3191556701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa58009480 con 0x7faa640c90c0 2024-01-27T01:06:12.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.311+0000 7faa69cbe700 1 --2- 172.21.15.46:0/3191556701 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa640c8c50 0x7faa6414cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:12.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.311+0000 7faa6acc0700 1 --2- 172.21.15.46:0/3191556701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa640c90c0 0x7faa6414d120 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7faa5c00d180 tx=0x7faa5c009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.311+0000 7faa567fc700 1 -- 172.21.15.46:0/3191556701 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c00dd70 con 0x7faa640c90c0 2024-01-27T01:06:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.311+0000 7faa567fc700 1 -- 172.21.15.46:0/3191556701 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5c00f040 con 0x7faa640c90c0 2024-01-27T01:06:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.311+0000 7faa6c723700 1 -- 172.21.15.46:0/3191556701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6415bca0 con 0x7faa640c90c0 2024-01-27T01:06:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.311+0000 7faa567fc700 1 -- 172.21.15.46:0/3191556701 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c015690 con 0x7faa640c90c0 2024-01-27T01:06:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.311+0000 7faa6c723700 1 -- 172.21.15.46:0/3191556701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa6415d240 con 0x7faa640c90c0 2024-01-27T01:06:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.312+0000 7faa6c723700 1 -- 172.21.15.46:0/3191556701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa6404c710 con 0x7faa640c90c0 2024-01-27T01:06:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.313+0000 7faa567fc700 1 -- 172.21.15.46:0/3191556701 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 125) v1 ==== 84686+0+0 (secure 0 0 0) 0x7faa5c01e3b0 con 0x7faa640c90c0 2024-01-27T01:06:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.313+0000 7faa567fc700 1 --2- 172.21.15.46:0/3191556701 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faa48066ea0 0x7faa48069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.313+0000 7faa567fc700 1 -- 172.21.15.46:0/3191556701 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6267+0+0 (secure 0 0 0) 0x7faa5c084b90 con 0x7faa640c90c0 2024-01-27T01:06:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.313+0000 7faa6a4bf700 1 --2- 172.21.15.46:0/3191556701 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faa48066ea0 0x7faa48069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.314+0000 7faa6a4bf700 1 --2- 172.21.15.46:0/3191556701 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faa48066ea0 0x7faa48069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faa58009260 tx=0x7faa58016000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:12.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.318+0000 7faa567fc700 1 -- 172.21.15.46:0/3191556701 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faa5c051cb0 con 0x7faa640c90c0 2024-01-27T01:06:12.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.524+0000 7faa6c723700 1 -- 172.21.15.46:0/3191556701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7faa6406c9c0 con 0x7faa640c90c0 2024-01-27T01:06:12.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.527+0000 7faa567fc700 1 -- 172.21.15.46:0/3191556701 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v74) v1 ==== 120+0+0 (secure 0 0 0) 0x7faa5c0552d0 con 0x7faa640c90c0 2024-01-27T01:06:12.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.530+0000 7faa6c723700 1 -- 172.21.15.46:0/3191556701 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faa48066ea0 msgr2=0x7faa48069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:12.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.530+0000 7faa6c723700 1 --2- 172.21.15.46:0/3191556701 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faa48066ea0 0x7faa48069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faa58009260 tx=0x7faa58016000 comp rx=0 tx=0).stop 2024-01-27T01:06:12.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.530+0000 7faa6c723700 1 -- 172.21.15.46:0/3191556701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa640c90c0 msgr2=0x7faa6414d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:12.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.530+0000 7faa6c723700 1 --2- 172.21.15.46:0/3191556701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa640c90c0 0x7faa6414d120 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7faa5c00d180 tx=0x7faa5c009a50 comp rx=0 tx=0).stop 2024-01-27T01:06:12.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.530+0000 7faa6c723700 1 -- 172.21.15.46:0/3191556701 shutdown_connections 2024-01-27T01:06:12.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.530+0000 7faa6c723700 1 --2- 172.21.15.46:0/3191556701 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faa48066ea0 0x7faa48069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.530+0000 7faa6c723700 1 --2- 172.21.15.46:0/3191556701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa640c90c0 0x7faa6414d120 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.530+0000 7faa6c723700 1 --2- 172.21.15.46:0/3191556701 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa64068940 0x7faa6415c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.530+0000 7faa6c723700 1 --2- 172.21.15.46:0/3191556701 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa640c8c50 0x7faa6414cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.530+0000 7faa6c723700 1 -- 172.21.15.46:0/3191556701 >> 172.21.15.46:0/3191556701 conn(0x7faa640ca100 msgr2=0x7faa640af970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:12.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.530+0000 7faa6c723700 1 -- 172.21.15.46:0/3191556701 shutdown_connections 2024-01-27T01:06:12.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.530+0000 7faa6c723700 1 -- 172.21.15.46:0/3191556701 wait complete. 2024-01-27T01:06:12.544 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-27T01:06:12.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.677+0000 7fb0c6e75700 1 Processor -- start 2024-01-27T01:06:12.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.677+0000 7fb0c6e75700 1 -- start start 2024-01-27T01:06:12.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.678+0000 7fb0c6e75700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb0c0141200 0x7fb0c00c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:12.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.678+0000 7fb0c6e75700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0c00caf30 0x7fb0c00cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:12.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.678+0000 7fb0c6e75700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c00ccc00 con 0x7fb0c00c8da0 2024-01-27T01:06:12.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.678+0000 7fb0c6e75700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c00cbbf0 con 0x7fb0c0141200 2024-01-27T01:06:12.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.678+0000 7fb0c6e75700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c00cc310 con 0x7fb0c00caf30 2024-01-27T01:06:12.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.679+0000 7fb0c5412700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0c00caf30 0x7fb0c00cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:12.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.679+0000 7fb0b7fff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb0c0141200 0x7fb0c00c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:12.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.679+0000 7fb0c4c11700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fb0c00c8da0 0x7fb0c0140af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:58910/0 (socket says 172.21.15.46:58910) 2024-01-27T01:06:12.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.679+0000 7fb0c5412700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0c00caf30 0x7fb0c00cb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:34692/0 (socket says 172.21.15.46:34692) 2024-01-27T01:06:12.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.679+0000 7fb0c4c11700 1 -- 172.21.15.46:0/1564704975 learned_addr learned my addr 172.21.15.46:0/1564704975 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:12.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.679+0000 7fb0b7fff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb0c0141200 0x7fb0c00c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:54024/0 (socket says 172.21.15.46:54024) 2024-01-27T01:06:12.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.679+0000 7fb0b77fe700 1 -- 172.21.15.46:0/1564704975 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026988101 0 0) 0x7fb0c00ccc00 con 0x7fb0c00c8da0 2024-01-27T01:06:12.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.679+0000 7fb0b77fe700 1 -- 172.21.15.46:0/1564704975 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0a8003570 con 0x7fb0c00c8da0 2024-01-27T01:06:12.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.679+0000 7fb0b7fff700 1 -- 172.21.15.46:0/1564704975 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0c00caf30 msgr2=0x7fb0c00cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:12.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.679+0000 7fb0b7fff700 1 --2- 172.21.15.46:0/1564704975 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0c00caf30 0x7fb0c00cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.679+0000 7fb0b7fff700 1 -- 172.21.15.46:0/1564704975 >> v1:172.21.15.106:6789/0 conn(0x7fb0c00c8da0 legacy=0x7fb0c0140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:12.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.679+0000 7fb0b7fff700 1 -- 172.21.15.46:0/1564704975 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c00cc5b0 con 0x7fb0c0141200 2024-01-27T01:06:12.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.679+0000 7fb0c5412700 1 --2- 172.21.15.46:0/1564704975 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0c00caf30 0x7fb0c00cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:12.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.679+0000 7fb0b7fff700 1 --2- 172.21.15.46:0/1564704975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0c0141200 0x7fb0c00c9170 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00098e0 tx=0x7fb0b00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5c1a50fe1745260 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:12.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.680+0000 7fb0b77fe700 1 -- 172.21.15.46:0/1564704975 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0011630 con 0x7fb0c0141200 2024-01-27T01:06:12.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.680+0000 7fb0b77fe700 1 -- 172.21.15.46:0/1564704975 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b0011790 con 0x7fb0c0141200 2024-01-27T01:06:12.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.680+0000 7fb0b77fe700 1 -- 172.21.15.46:0/1564704975 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0011b30 con 0x7fb0c0141200 2024-01-27T01:06:12.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.680+0000 7fb0c6e75700 1 -- 172.21.15.46:0/1564704975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0c0141200 msgr2=0x7fb0c00c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:12.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.680+0000 7fb0c6e75700 1 --2- 172.21.15.46:0/1564704975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0c0141200 0x7fb0c00c9170 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00098e0 tx=0x7fb0b00091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.680+0000 7fb0c6e75700 1 -- 172.21.15.46:0/1564704975 shutdown_connections 2024-01-27T01:06:12.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.680+0000 7fb0c6e75700 1 --2- 172.21.15.46:0/1564704975 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0c00caf30 0x7fb0c00cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.680+0000 7fb0c6e75700 1 --2- 172.21.15.46:0/1564704975 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0c0141200 0x7fb0c00c9170 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.680+0000 7fb0c6e75700 1 -- 172.21.15.46:0/1564704975 >> 172.21.15.46:0/1564704975 conn(0x7fb0c006bd90 msgr2=0x7fb0c006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:12.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.680+0000 7fb0c6e75700 1 -- 172.21.15.46:0/1564704975 shutdown_connections 2024-01-27T01:06:12.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.680+0000 7fb0c6e75700 1 -- 172.21.15.46:0/1564704975 wait complete. 2024-01-27T01:06:12.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.682+0000 7fb0c6e75700 1 Processor -- start 2024-01-27T01:06:12.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.682+0000 7fb0c6e75700 1 -- start start 2024-01-27T01:06:12.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.683+0000 7fb0c6e75700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c00c8da0 0x7fb0c00b0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:12.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.683+0000 7fb0c6e75700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0c00caf30 0x7fb0c0159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:12.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.683+0000 7fb0c6e75700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0c0141200 0x7fb0c013d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:12.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.683+0000 7fb0c6e75700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c0159a80 con 0x7fb0c00c8da0 2024-01-27T01:06:12.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.683+0000 7fb0c6e75700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c0158e70 con 0x7fb0c00caf30 2024-01-27T01:06:12.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.683+0000 7fb0c6e75700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c0159150 con 0x7fb0c0141200 2024-01-27T01:06:12.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.683+0000 7fb0c4c11700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c00c8da0 0x7fb0c00b0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:12.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.683+0000 7fb0c4c11700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c00c8da0 0x7fb0c00b0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57702/0 (socket says 172.21.15.46:57702) 2024-01-27T01:06:12.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.683+0000 7fb0c4c11700 1 -- 172.21.15.46:0/1043267232 learned_addr learned my addr 172.21.15.46:0/1043267232 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:12.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.684+0000 7fb0b7fff700 1 --2- 172.21.15.46:0/1043267232 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0c00caf30 0x7fb0c0159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:12.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.684+0000 7fb0c5412700 1 --2- 172.21.15.46:0/1043267232 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0c0141200 0x7fb0c013d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:12.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.684+0000 7fb0c4c11700 1 -- 172.21.15.46:0/1043267232 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0c0141200 msgr2=0x7fb0c013d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:12.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.684+0000 7fb0c4c11700 1 --2- 172.21.15.46:0/1043267232 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0c0141200 0x7fb0c013d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.684+0000 7fb0c4c11700 1 -- 172.21.15.46:0/1043267232 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0c00caf30 msgr2=0x7fb0c0159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:12.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.684+0000 7fb0c4c11700 1 --2- 172.21.15.46:0/1043267232 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0c00caf30 0x7fb0c0159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.684+0000 7fb0c4c11700 1 -- 172.21.15.46:0/1043267232 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b8009a20 con 0x7fb0c00c8da0 2024-01-27T01:06:12.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.684+0000 7fb0c4c11700 1 --2- 172.21.15.46:0/1043267232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c00c8da0 0x7fb0c00b0800 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00c7c0 tx=0x7fb0bc007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:12.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.684+0000 7fb0b4ff9700 1 -- 172.21.15.46:0/1043267232 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc005b30 con 0x7fb0c00c8da0 2024-01-27T01:06:12.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.684+0000 7fb0b4ff9700 1 -- 172.21.15.46:0/1043267232 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0bc005cb0 con 0x7fb0c00c8da0 2024-01-27T01:06:12.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.684+0000 7fb0c6e75700 1 -- 172.21.15.46:0/1043267232 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b0009420 con 0x7fb0c00c8da0 2024-01-27T01:06:12.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.684+0000 7fb0b4ff9700 1 -- 172.21.15.46:0/1043267232 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc007320 con 0x7fb0c00c8da0 2024-01-27T01:06:12.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.684+0000 7fb0c6e75700 1 -- 172.21.15.46:0/1043267232 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c0159710 con 0x7fb0c00c8da0 2024-01-27T01:06:12.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.685+0000 7fb0c6e75700 1 -- 172.21.15.46:0/1043267232 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c004c710 con 0x7fb0c00c8da0 2024-01-27T01:06:12.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.686+0000 7fb0b4ff9700 1 -- 172.21.15.46:0/1043267232 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 125) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fb0bc01b3b0 con 0x7fb0c00c8da0 2024-01-27T01:06:12.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.686+0000 7fb0b4ff9700 1 --2- 172.21.15.46:0/1043267232 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb0ac066ea0 0x7fb0ac069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:12.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.686+0000 7fb0b4ff9700 1 -- 172.21.15.46:0/1043267232 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6267+0+0 (secure 0 0 0) 0x7fb0bc083b00 con 0x7fb0c00c8da0 2024-01-27T01:06:12.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.687+0000 7fb0b7fff700 1 --2- 172.21.15.46:0/1043267232 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb0ac066ea0 0x7fb0ac069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:12.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.687+0000 7fb0b7fff700 1 --2- 172.21.15.46:0/1043267232 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb0ac066ea0 0x7fb0ac069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00091e0 tx=0x7fb0b0017000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:12.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.691+0000 7fb0b4ff9700 1 -- 172.21.15.46:0/1043267232 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb0bc050ce0 con 0x7fb0c00c8da0 2024-01-27T01:06:12.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.897+0000 7fb0c6e75700 1 -- 172.21.15.46:0/1043267232 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fb0c0159d60 con 0x7fb0c00c8da0 2024-01-27T01:06:12.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.901+0000 7fb0b4ff9700 1 -- 172.21.15.46:0/1043267232 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v75) v1 ==== 115+0+0 (secure 0 0 0) 0x7fb0bc083e60 con 0x7fb0c00c8da0 2024-01-27T01:06:12.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.903+0000 7fb0c6e75700 1 -- 172.21.15.46:0/1043267232 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb0ac066ea0 msgr2=0x7fb0ac069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:12.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.903+0000 7fb0c6e75700 1 --2- 172.21.15.46:0/1043267232 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb0ac066ea0 0x7fb0ac069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00091e0 tx=0x7fb0b0017000 comp rx=0 tx=0).stop 2024-01-27T01:06:12.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.903+0000 7fb0c6e75700 1 -- 172.21.15.46:0/1043267232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c00c8da0 msgr2=0x7fb0c00b0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:12.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.903+0000 7fb0c6e75700 1 --2- 172.21.15.46:0/1043267232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c00c8da0 0x7fb0c00b0800 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00c7c0 tx=0x7fb0bc007e00 comp rx=0 tx=0).stop 2024-01-27T01:06:12.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.903+0000 7fb0c6e75700 1 -- 172.21.15.46:0/1043267232 shutdown_connections 2024-01-27T01:06:12.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.903+0000 7fb0c6e75700 1 --2- 172.21.15.46:0/1043267232 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb0ac066ea0 0x7fb0ac069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.903+0000 7fb0c6e75700 1 --2- 172.21.15.46:0/1043267232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c00c8da0 0x7fb0c00b0800 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.903+0000 7fb0c6e75700 1 --2- 172.21.15.46:0/1043267232 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0c0141200 0x7fb0c013d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.903+0000 7fb0c6e75700 1 --2- 172.21.15.46:0/1043267232 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0c00caf30 0x7fb0c0159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:12.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.903+0000 7fb0c6e75700 1 -- 172.21.15.46:0/1043267232 >> 172.21.15.46:0/1043267232 conn(0x7fb0c006bd90 msgr2=0x7fb0c00c3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:12.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.903+0000 7fb0c6e75700 1 -- 172.21.15.46:0/1043267232 shutdown_connections 2024-01-27T01:06:12.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:12.903+0000 7fb0c6e75700 1 -- 172.21.15.46:0/1043267232 wait complete. 2024-01-27T01:06:12.917 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-01-27T01:06:13.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.048+0000 7f53c00c9700 1 Processor -- start 2024-01-27T01:06:13.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.048+0000 7f53c00c9700 1 -- start start 2024-01-27T01:06:13.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53c00c9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f53b8068040 0x7f53b8164430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:13.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53c00c9700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b815abc0 con 0x7f53b8159760 2024-01-27T01:06:13.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53c00c9700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b80b2e90 con 0x7f53b8159b30 2024-01-27T01:06:13.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53c00c9700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b8159fb0 con 0x7f53b8068040 2024-01-27T01:06:13.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53bde65700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f53b8159760 0x7f53b80b0c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41148/0 (socket says 172.21.15.46:41148) 2024-01-27T01:06:13.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53bde65700 1 -- 172.21.15.46:0/4255348621 learned_addr learned my addr 172.21.15.46:0/4255348621 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:13.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53be666700 1 --2- 172.21.15.46:0/4255348621 >> v2:172.21.15.106:3300/0 conn(0x7f53b8068040 0x7f53b8164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:13.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53bce63700 1 -- 172.21.15.46:0/4255348621 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083811607 0 0) 0x7f53b815abc0 con 0x7f53b8159760 2024-01-27T01:06:13.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53bce63700 1 -- 172.21.15.46:0/4255348621 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a4003570 con 0x7f53b8159760 2024-01-27T01:06:13.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53bce63700 1 -- 172.21.15.46:0/4255348621 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 919516815 0 0) 0x7f53a4003570 con 0x7f53b8159760 2024-01-27T01:06:13.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53bce63700 1 -- 172.21.15.46:0/4255348621 >> v2:172.21.15.106:3300/0 conn(0x7f53b8068040 msgr2=0x7f53b8164430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:13.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53bce63700 1 --2- 172.21.15.46:0/4255348621 >> v2:172.21.15.106:3300/0 conn(0x7f53b8068040 0x7f53b8164430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:13.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53bce63700 1 -- 172.21.15.46:0/4255348621 >> v1:172.21.15.106:6789/0 conn(0x7f53b8159b30 legacy=0x7f53b8067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:13.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53bce63700 1 -- 172.21.15.46:0/4255348621 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b815a290 con 0x7f53b8159760 2024-01-27T01:06:13.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53bce63700 1 -- 172.21.15.46:0/4255348621 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f53ac001470 con 0x7f53b8159760 2024-01-27T01:06:13.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53bce63700 1 -- 172.21.15.46:0/4255348621 >> v1:172.21.15.46:6789/0 conn(0x7f53b8159760 legacy=0x7f53b80b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:13.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53bce63700 1 --2- 172.21.15.46:0/4255348621 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53a4003a60 0x7f53a4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:13.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53bce63700 1 --2- 172.21.15.46:0/4255348621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a4004810 0x7f53a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:13.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53bce63700 1 --2- 172.21.15.46:0/4255348621 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53a40051c0 0x7f53a400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:13.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53bce63700 1 -- 172.21.15.46:0/4255348621 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b8159fb0 con 0x7f53a4004810 2024-01-27T01:06:13.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53bce63700 1 -- 172.21.15.46:0/4255348621 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a400c1e0 con 0x7f53a4003a60 2024-01-27T01:06:13.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.050+0000 7f53bce63700 1 -- 172.21.15.46:0/4255348621 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a400c3a0 con 0x7f53a40051c0 2024-01-27T01:06:13.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.051+0000 7f53bd664700 1 --2- 172.21.15.46:0/4255348621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a4004810 0x7f53a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:13.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.051+0000 7f53bde65700 1 --2- 172.21.15.46:0/4255348621 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53a4003a60 0x7f53a4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:13.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.051+0000 7f53be666700 1 --2- 172.21.15.46:0/4255348621 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53a40051c0 0x7f53a400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:13.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.051+0000 7f53bd664700 1 -- 172.21.15.46:0/4255348621 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53a40051c0 msgr2=0x7f53a400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:13.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.051+0000 7f53bd664700 1 --2- 172.21.15.46:0/4255348621 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53a40051c0 0x7f53a400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:13.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.051+0000 7f53bd664700 1 -- 172.21.15.46:0/4255348621 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53a4003a60 msgr2=0x7f53a4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:13.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.051+0000 7f53bd664700 1 --2- 172.21.15.46:0/4255348621 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53a4003a60 0x7f53a4003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:13.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.051+0000 7f53bd664700 1 -- 172.21.15.46:0/4255348621 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a4003570 con 0x7f53a4004810 2024-01-27T01:06:13.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.051+0000 7f53be666700 1 --2- 172.21.15.46:0/4255348621 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53a40051c0 0x7f53a400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:13.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.051+0000 7f53be666700 1 -- 172.21.15.46:0/4255348621 reap_dead start 2024-01-27T01:06:13.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.051+0000 7f53bd664700 1 --2- 172.21.15.46:0/4255348621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a4004810 0x7f53a4004c80 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f53b4001970 tx=0x7f53b4007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=156588bc245211f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:13.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.051+0000 7f53bce63700 1 -- 172.21.15.46:0/4255348621 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b400c5d0 con 0x7f53a4004810 2024-01-27T01:06:13.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.051+0000 7f53bce63700 1 -- 172.21.15.46:0/4255348621 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b400c730 con 0x7f53a4004810 2024-01-27T01:06:13.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.051+0000 7f53bce63700 1 -- 172.21.15.46:0/4255348621 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b400cad0 con 0x7f53a4004810 2024-01-27T01:06:13.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.052+0000 7f53c00c9700 1 -- 172.21.15.46:0/4255348621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a4004810 msgr2=0x7f53a4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:13.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.052+0000 7f53c00c9700 1 --2- 172.21.15.46:0/4255348621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a4004810 0x7f53a4004c80 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f53b4001970 tx=0x7f53b4007d10 comp rx=0 tx=0).stop 2024-01-27T01:06:13.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.052+0000 7f53c00c9700 1 -- 172.21.15.46:0/4255348621 shutdown_connections 2024-01-27T01:06:13.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.052+0000 7f53c00c9700 1 --2- 172.21.15.46:0/4255348621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a4004810 0x7f53a4004c80 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:13.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.052+0000 7f53c00c9700 1 -- 172.21.15.46:0/4255348621 >> 172.21.15.46:0/4255348621 conn(0x7f53b806b630 msgr2=0x7f53b80aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:13.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.052+0000 7f53c00c9700 1 -- 172.21.15.46:0/4255348621 shutdown_connections 2024-01-27T01:06:13.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.052+0000 7f53c00c9700 1 -- 172.21.15.46:0/4255348621 wait complete. 2024-01-27T01:06:13.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.053+0000 7f53c00c9700 1 Processor -- start 2024-01-27T01:06:13.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.053+0000 7f53c00c9700 1 -- start start 2024-01-27T01:06:13.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.054+0000 7f53c00c9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a4004810 0x7f53b80ac550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:13.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.054+0000 7f53c00c9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53b80ae7c0 0x7f53b80aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:13.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.054+0000 7f53c00c9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53b816ca10 0x7f53b816ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:13.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.054+0000 7f53c00c9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b80afae0 con 0x7f53a4004810 2024-01-27T01:06:13.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.054+0000 7f53c00c9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b816d3c0 con 0x7f53b816ca10 2024-01-27T01:06:13.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.054+0000 7f53c00c9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b816a770 con 0x7f53b80ae7c0 2024-01-27T01:06:13.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.054+0000 7f53bde65700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a4004810 0x7f53b80ac550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:13.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.054+0000 7f53bde65700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a4004810 0x7f53b80ac550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57726/0 (socket says 172.21.15.46:57726) 2024-01-27T01:06:13.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.054+0000 7f53bde65700 1 -- 172.21.15.46:0/3341655007 learned_addr learned my addr 172.21.15.46:0/3341655007 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:13.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.054+0000 7f53be666700 1 --2- 172.21.15.46:0/3341655007 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53b816ca10 0x7f53b816ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:13.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.054+0000 7f53bd664700 1 --2- 172.21.15.46:0/3341655007 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53b80ae7c0 0x7f53b80aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:13.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.054+0000 7f53bde65700 1 -- 172.21.15.46:0/3341655007 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53b80ae7c0 msgr2=0x7f53b80aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:13.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.054+0000 7f53bde65700 1 --2- 172.21.15.46:0/3341655007 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53b80ae7c0 0x7f53b80aec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:13.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.054+0000 7f53bde65700 1 -- 172.21.15.46:0/3341655007 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53b816ca10 msgr2=0x7f53b816ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:13.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.054+0000 7f53bde65700 1 --2- 172.21.15.46:0/3341655007 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53b816ca10 0x7f53b816ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:13.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.054+0000 7f53bde65700 1 -- 172.21.15.46:0/3341655007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b80af240 con 0x7f53a4004810 2024-01-27T01:06:13.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.055+0000 7f53bde65700 1 --2- 172.21.15.46:0/3341655007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a4004810 0x7f53b80ac550 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f53ac007390 tx=0x7f53b815ab30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:13.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.055+0000 7f53a9ffb700 1 -- 172.21.15.46:0/3341655007 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac018030 con 0x7f53a4004810 2024-01-27T01:06:13.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.055+0000 7f53a9ffb700 1 -- 172.21.15.46:0/3341655007 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac00f640 con 0x7f53a4004810 2024-01-27T01:06:13.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.055+0000 7f53a9ffb700 1 -- 172.21.15.46:0/3341655007 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac00f9e0 con 0x7f53a4004810 2024-01-27T01:06:13.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.055+0000 7f53c00c9700 1 -- 172.21.15.46:0/3341655007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b80af520 con 0x7f53a4004810 2024-01-27T01:06:13.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.055+0000 7f53c00c9700 1 -- 172.21.15.46:0/3341655007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b80aca90 con 0x7f53a4004810 2024-01-27T01:06:13.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.056+0000 7f53c00c9700 1 -- 172.21.15.46:0/3341655007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b816a520 con 0x7f53a4004810 2024-01-27T01:06:13.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.056+0000 7f53a9ffb700 1 -- 172.21.15.46:0/3341655007 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 125) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f53ac01b070 con 0x7f53a4004810 2024-01-27T01:06:13.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.057+0000 7f53a9ffb700 1 --2- 172.21.15.46:0/3341655007 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f539c066ea0 0x7f539c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:13.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.057+0000 7f53a9ffb700 1 -- 172.21.15.46:0/3341655007 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6267+0+0 (secure 0 0 0) 0x7f53ac019070 con 0x7f53a4004810 2024-01-27T01:06:13.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.057+0000 7f53bd664700 1 --2- 172.21.15.46:0/3341655007 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f539c066ea0 0x7f539c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:13.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.058+0000 7f53bd664700 1 --2- 172.21.15.46:0/3341655007 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f539c066ea0 0x7f539c069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f53b80b0780 tx=0x7f53b4015000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:13.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.061+0000 7f53a9ffb700 1 -- 172.21.15.46:0/3341655007 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f53ac0513c0 con 0x7f53a4004810 2024-01-27T01:06:13.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.349+0000 7f53c00c9700 1 -- 172.21.15.46:0/3341655007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f53b804c710 con 0x7f53a4004810 2024-01-27T01:06:13.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.716+0000 7f53a9ffb700 1 -- 172.21.15.46:0/3341655007 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f53ac00fd90 con 0x7f53a4004810 2024-01-27T01:06:13.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.753+0000 7f53a9ffb700 1 -- 172.21.15.46:0/3341655007 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3134) v1 ==== 163+0+0 (secure 0 0 0) 0x7f53ac0513c0 con 0x7f53a4004810 2024-01-27T01:06:13.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.756+0000 7f53c00c9700 1 -- 172.21.15.46:0/3341655007 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f539c066ea0 msgr2=0x7f539c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:13.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.756+0000 7f53c00c9700 1 --2- 172.21.15.46:0/3341655007 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f539c066ea0 0x7f539c069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f53b80b0780 tx=0x7f53b4015000 comp rx=0 tx=0).stop 2024-01-27T01:06:13.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.756+0000 7f53c00c9700 1 -- 172.21.15.46:0/3341655007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a4004810 msgr2=0x7f53b80ac550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:13.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.756+0000 7f53c00c9700 1 --2- 172.21.15.46:0/3341655007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a4004810 0x7f53b80ac550 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f53ac007390 tx=0x7f53b815ab30 comp rx=0 tx=0).stop 2024-01-27T01:06:13.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.756+0000 7f53c00c9700 1 -- 172.21.15.46:0/3341655007 shutdown_connections 2024-01-27T01:06:13.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.756+0000 7f53c00c9700 1 --2- 172.21.15.46:0/3341655007 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f539c066ea0 0x7f539c069350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:13.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.756+0000 7f53c00c9700 1 --2- 172.21.15.46:0/3341655007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a4004810 0x7f53b80ac550 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:13.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.756+0000 7f53c00c9700 1 --2- 172.21.15.46:0/3341655007 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53b816ca10 0x7f53b816ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:13.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.756+0000 7f53c00c9700 1 --2- 172.21.15.46:0/3341655007 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53b80ae7c0 0x7f53b80aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:13.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.756+0000 7f53c00c9700 1 -- 172.21.15.46:0/3341655007 >> 172.21.15.46:0/3341655007 conn(0x7f53b806b630 msgr2=0x7f53b81579f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:13.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.756+0000 7f53c00c9700 1 -- 172.21.15.46:0/3341655007 shutdown_connections 2024-01-27T01:06:13.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.756+0000 7f53c00c9700 1 -- 172.21.15.46:0/3341655007 wait complete. 2024-01-27T01:06:13.771 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2024-01-27T01:06:13.772 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command succes with --fs argument for non-default fs 2024-01-27T01:06:13.772 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-01-27T01:06:13.772 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-01-27T01:06:13.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.907+0000 7f087baee700 1 Processor -- start 2024-01-27T01:06:13.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.907+0000 7f087baee700 1 -- start start 2024-01-27T01:06:13.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.908+0000 7f087baee700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f08741596e0 0x7f0874159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:13.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.908+0000 7f087baee700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08740cb4c0 0x7f0874068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:13.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.908+0000 7f087baee700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08740c97f0 con 0x7f08740c4720 2024-01-27T01:06:13.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.908+0000 7f087baee700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08740cbf90 con 0x7f08740cb4c0 2024-01-27T01:06:13.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.908+0000 7f087baee700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08740c8c20 con 0x7f08741596e0 2024-01-27T01:06:13.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.908+0000 7f087a08b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08740cb4c0 0x7f0874068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:13.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.908+0000 7f0879089700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f08740c4720 0x7f08740cadb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41158/0 (socket says 172.21.15.46:41158) 2024-01-27T01:06:13.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.908+0000 7f087a08b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08740cb4c0 0x7f0874068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57738/0 (socket says 172.21.15.46:57738) 2024-01-27T01:06:13.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.908+0000 7f0879089700 1 -- 172.21.15.46:0/3948911920 learned_addr learned my addr 172.21.15.46:0/3948911920 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:13.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.909+0000 7f087988a700 1 --2- 172.21.15.46:0/3948911920 >> v2:172.21.15.106:3300/0 conn(0x7f08741596e0 0x7f0874159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:13.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.909+0000 7f087a08b700 1 -- 172.21.15.46:0/3948911920 >> v2:172.21.15.106:3300/0 conn(0x7f08741596e0 msgr2=0x7f0874159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:13.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.909+0000 7f087a08b700 1 --2- 172.21.15.46:0/3948911920 >> v2:172.21.15.106:3300/0 conn(0x7f08741596e0 0x7f0874159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:13.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.909+0000 7f087a08b700 1 -- 172.21.15.46:0/3948911920 >> v1:172.21.15.46:6789/0 conn(0x7f08740c4720 legacy=0x7f08740cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:13.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.909+0000 7f0878888700 1 -- 172.21.15.46:0/3948911920 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173957686 0 0) 0x7f08740c97f0 con 0x7f08740c4720 2024-01-27T01:06:13.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.909+0000 7f087a08b700 1 -- 172.21.15.46:0/3948911920 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08740c8ec0 con 0x7f08740cb4c0 2024-01-27T01:06:13.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.909+0000 7f087a08b700 1 --2- 172.21.15.46:0/3948911920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08740cb4c0 0x7f0874068a40 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f087406fa70 tx=0x7f086c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3ac06b0373c628 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:13.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.909+0000 7f0878888700 1 -- 172.21.15.46:0/3948911920 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c00e070 con 0x7f08740cb4c0 2024-01-27T01:06:13.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.909+0000 7f0878888700 1 -- 172.21.15.46:0/3948911920 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c00bc30 con 0x7f08740cb4c0 2024-01-27T01:06:13.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.909+0000 7f0878888700 1 -- 172.21.15.46:0/3948911920 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c012430 con 0x7f08740cb4c0 2024-01-27T01:06:13.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.910+0000 7f087baee700 1 -- 172.21.15.46:0/3948911920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08740cb4c0 msgr2=0x7f0874068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:13.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.910+0000 7f087baee700 1 --2- 172.21.15.46:0/3948911920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08740cb4c0 0x7f0874068a40 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f087406fa70 tx=0x7f086c009780 comp rx=0 tx=0).stop 2024-01-27T01:06:13.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.910+0000 7f087baee700 1 -- 172.21.15.46:0/3948911920 shutdown_connections 2024-01-27T01:06:13.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.910+0000 7f087baee700 1 --2- 172.21.15.46:0/3948911920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08740cb4c0 0x7f0874068a40 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:13.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.910+0000 7f087baee700 1 --2- 172.21.15.46:0/3948911920 >> v2:172.21.15.106:3300/0 conn(0x7f08741596e0 0x7f0874159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:13.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.910+0000 7f087baee700 1 -- 172.21.15.46:0/3948911920 >> 172.21.15.46:0/3948911920 conn(0x7f0874097820 msgr2=0x7f0874068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:13.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.910+0000 7f087baee700 1 -- 172.21.15.46:0/3948911920 shutdown_connections 2024-01-27T01:06:13.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.910+0000 7f087baee700 1 -- 172.21.15.46:0/3948911920 wait complete. 2024-01-27T01:06:13.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.911+0000 7f087baee700 1 Processor -- start 2024-01-27T01:06:13.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.911+0000 7f087baee700 1 -- start start 2024-01-27T01:06:13.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f087baee700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08740c4720 0x7f0874157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:13.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f087baee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08740cb4c0 0x7f08740c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:13.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f087baee700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08741596e0 0x7f08740c7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:13.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f087baee700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0874158640 con 0x7f08740cb4c0 2024-01-27T01:06:13.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f087baee700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08740b15a0 con 0x7f08740c4720 2024-01-27T01:06:13.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f087baee700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0874157da0 con 0x7f08741596e0 2024-01-27T01:06:13.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f0879089700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08740cb4c0 0x7f08740c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:13.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f0879089700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08740cb4c0 0x7f08740c6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57754/0 (socket says 172.21.15.46:57754) 2024-01-27T01:06:13.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f0879089700 1 -- 172.21.15.46:0/638505823 learned_addr learned my addr 172.21.15.46:0/638505823 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f087988a700 1 --2- 172.21.15.46:0/638505823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08740c4720 0x7f0874157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f087a08b700 1 --2- 172.21.15.46:0/638505823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08741596e0 0x7f08740c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f0879089700 1 -- 172.21.15.46:0/638505823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08741596e0 msgr2=0x7f08740c7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f0879089700 1 --2- 172.21.15.46:0/638505823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08741596e0 0x7f08740c7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f0879089700 1 -- 172.21.15.46:0/638505823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08740c4720 msgr2=0x7f0874157300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f0879089700 1 --2- 172.21.15.46:0/638505823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08740c4720 0x7f0874157300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f0879089700 1 -- 172.21.15.46:0/638505823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c009a20 con 0x7f08740cb4c0 2024-01-27T01:06:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f087a08b700 1 --2- 172.21.15.46:0/638505823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08741596e0 0x7f08740c7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.912+0000 7f087988a700 1 --2- 172.21.15.46:0/638505823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08740c4720 0x7f0874157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.913+0000 7f0879089700 1 --2- 172.21.15.46:0/638505823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08740cb4c0 0x7f08740c6b30 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f086800c810 tx=0x7f0868007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.913+0000 7f0865ffb700 1 -- 172.21.15.46:0/638505823 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0868005c00 con 0x7f08740cb4c0 2024-01-27T01:06:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.913+0000 7f0865ffb700 1 -- 172.21.15.46:0/638505823 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0868005d60 con 0x7f08740cb4c0 2024-01-27T01:06:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.913+0000 7f087baee700 1 -- 172.21.15.46:0/638505823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0874158080 con 0x7f08740cb4c0 2024-01-27T01:06:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.913+0000 7f0865ffb700 1 -- 172.21.15.46:0/638505823 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0868007320 con 0x7f08740cb4c0 2024-01-27T01:06:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.913+0000 7f087baee700 1 -- 172.21.15.46:0/638505823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08740c7dd0 con 0x7f08740cb4c0 2024-01-27T01:06:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.914+0000 7f087baee700 1 -- 172.21.15.46:0/638505823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08740b13e0 con 0x7f08740cb4c0 2024-01-27T01:06:13.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.914+0000 7f0865ffb700 1 -- 172.21.15.46:0/638505823 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f086801b400 con 0x7f08740cb4c0 2024-01-27T01:06:13.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.915+0000 7f0865ffb700 1 --2- 172.21.15.46:0/638505823 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0858066f30 0x7f08580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:13.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.915+0000 7f0865ffb700 1 -- 172.21.15.46:0/638505823 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6267+0+0 (secure 0 0 0) 0x7f08680842c0 con 0x7f08740cb4c0 2024-01-27T01:06:13.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.915+0000 7f087988a700 1 --2- 172.21.15.46:0/638505823 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0858066f30 0x7f08580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:13.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.916+0000 7f087988a700 1 --2- 172.21.15.46:0/638505823 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0858066f30 0x7f08580693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0870006fd0 tx=0x7f0870008040 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:13.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:13.919+0000 7f0865ffb700 1 -- 172.21.15.46:0/638505823 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f08740b13e0 con 0x7f08740cb4c0 2024-01-27T01:06:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.204+0000 7f087baee700 1 -- 172.21.15.46:0/638505823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f08740b13e0 con 0x7f08740cb4c0 2024-01-27T01:06:14.587 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:06:14.590 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:06:14.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.760+0000 7f0865ffb700 1 -- 172.21.15.46:0/638505823 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3135) v1 ==== 189+0+0 (secure 0 0 0) 0x7f08740b13e0 con 0x7f08740cb4c0 2024-01-27T01:06:14.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.763+0000 7f087baee700 1 -- 172.21.15.46:0/638505823 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0858066f30 msgr2=0x7f08580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.763+0000 7f087baee700 1 --2- 172.21.15.46:0/638505823 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0858066f30 0x7f08580693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0870006fd0 tx=0x7f0870008040 comp rx=0 tx=0).stop 2024-01-27T01:06:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.763+0000 7f087baee700 1 -- 172.21.15.46:0/638505823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08740cb4c0 msgr2=0x7f08740c6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.763+0000 7f087baee700 1 --2- 172.21.15.46:0/638505823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08740cb4c0 0x7f08740c6b30 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f086800c810 tx=0x7f0868007db0 comp rx=0 tx=0).stop 2024-01-27T01:06:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.763+0000 7f087baee700 1 -- 172.21.15.46:0/638505823 shutdown_connections 2024-01-27T01:06:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.763+0000 7f087baee700 1 --2- 172.21.15.46:0/638505823 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0858066f30 0x7f08580693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.763+0000 7f087baee700 1 --2- 172.21.15.46:0/638505823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08741596e0 0x7f08740c7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.763+0000 7f087baee700 1 --2- 172.21.15.46:0/638505823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08740c4720 0x7f0874157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.763+0000 7f087baee700 1 --2- 172.21.15.46:0/638505823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08740cb4c0 0x7f08740c6b30 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.763+0000 7f087baee700 1 -- 172.21.15.46:0/638505823 >> 172.21.15.46:0/638505823 conn(0x7f0874097820 msgr2=0x7f08740ca510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.763+0000 7f087baee700 1 -- 172.21.15.46:0/638505823 shutdown_connections 2024-01-27T01:06:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.763+0000 7f087baee700 1 -- 172.21.15.46:0/638505823 wait complete. 2024-01-27T01:06:14.779 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-27T01:06:14.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.918+0000 7fb714510700 1 Processor -- start 2024-01-27T01:06:14.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.918+0000 7fb714510700 1 -- start start 2024-01-27T01:06:14.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.919+0000 7fb714510700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb70c068ed0 0x7fb70c06a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:14.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.919+0000 7fb714510700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb70c0a9a80 con 0x7fb70c06add0 2024-01-27T01:06:14.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.919+0000 7fb714510700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb70c15a680 con 0x7fb70c068b00 2024-01-27T01:06:14.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.919+0000 7fb714510700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb70c069320 con 0x7fb70c068ed0 2024-01-27T01:06:14.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.919+0000 7fb712aad700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb70c06add0 0x7fb70c0a9f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41168/0 (socket says 172.21.15.46:41168) 2024-01-27T01:06:14.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.919+0000 7fb712aad700 1 -- 172.21.15.46:0/3379648530 learned_addr learned my addr 172.21.15.46:0/3379648530 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:14.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.919+0000 7fb711aab700 1 --2- 172.21.15.46:0/3379648530 >> v2:172.21.15.106:3300/0 conn(0x7fb70c068ed0 0x7fb70c06a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.919+0000 7fb7112aa700 1 -- 172.21.15.46:0/3379648530 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854776915 0 0) 0x7fb70c0a9a80 con 0x7fb70c06add0 2024-01-27T01:06:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.919+0000 7fb7112aa700 1 -- 172.21.15.46:0/3379648530 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6f8003570 con 0x7fb70c06add0 2024-01-27T01:06:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7112aa700 1 -- 172.21.15.46:0/3379648530 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148413884 0 0) 0x7fb70c15a680 con 0x7fb70c068b00 2024-01-27T01:06:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7112aa700 1 -- 172.21.15.46:0/3379648530 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb70c0a9a80 con 0x7fb70c068b00 2024-01-27T01:06:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7112aa700 1 -- 172.21.15.46:0/3379648530 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2591727416 0 0) 0x7fb6f8003570 con 0x7fb70c06add0 2024-01-27T01:06:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7112aa700 1 -- 172.21.15.46:0/3379648530 >> v2:172.21.15.106:3300/0 conn(0x7fb70c068ed0 msgr2=0x7fb70c06a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7112aa700 1 --2- 172.21.15.46:0/3379648530 >> v2:172.21.15.106:3300/0 conn(0x7fb70c068ed0 0x7fb70c06a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7112aa700 1 -- 172.21.15.46:0/3379648530 >> v1:172.21.15.106:6789/0 conn(0x7fb70c068b00 legacy=0x7fb70c159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7112aa700 1 -- 172.21.15.46:0/3379648530 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70c15a960 con 0x7fb70c06add0 2024-01-27T01:06:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7112aa700 1 -- 172.21.15.46:0/3379648530 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fb7040017f0 con 0x7fb70c06add0 2024-01-27T01:06:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb711aab700 1 --2- 172.21.15.46:0/3379648530 >> v2:172.21.15.106:3300/0 conn(0x7fb70c068ed0 0x7fb70c06a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7112aa700 1 -- 172.21.15.46:0/3379648530 >> v1:172.21.15.46:6789/0 conn(0x7fb70c06add0 legacy=0x7fb70c0a9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7112aa700 1 --2- 172.21.15.46:0/3379648530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6f8003bf0 0x7fb6f8003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7112aa700 1 --2- 172.21.15.46:0/3379648530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6f8004810 0x7fb6f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:14.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7112aa700 1 --2- 172.21.15.46:0/3379648530 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6f80051c0 0x7fb6f800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:14.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7112aa700 1 -- 172.21.15.46:0/3379648530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c069320 con 0x7fb6f8003bf0 2024-01-27T01:06:14.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7112aa700 1 -- 172.21.15.46:0/3379648530 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f800c160 con 0x7fb6f8004810 2024-01-27T01:06:14.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7112aa700 1 -- 172.21.15.46:0/3379648530 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f800c320 con 0x7fb6f80051c0 2024-01-27T01:06:14.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7122ac700 1 --2- 172.21.15.46:0/3379648530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6f8003bf0 0x7fb6f8003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:14.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb711aab700 1 --2- 172.21.15.46:0/3379648530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6f8004810 0x7fb6f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:14.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb712aad700 1 --2- 172.21.15.46:0/3379648530 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6f80051c0 0x7fb6f800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:14.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7122ac700 1 -- 172.21.15.46:0/3379648530 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6f80051c0 msgr2=0x7fb6f800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:14.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7122ac700 1 --2- 172.21.15.46:0/3379648530 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6f80051c0 0x7fb6f800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:14.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7122ac700 1 -- 172.21.15.46:0/3379648530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6f8004810 msgr2=0x7fb6f8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:14.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7122ac700 1 --2- 172.21.15.46:0/3379648530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6f8004810 0x7fb6f8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:14.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb7122ac700 1 -- 172.21.15.46:0/3379648530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f8003570 con 0x7fb6f8003bf0 2024-01-27T01:06:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.920+0000 7fb712aad700 1 -- 172.21.15.46:0/3379648530 reap_dead start 2024-01-27T01:06:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.921+0000 7fb7122ac700 1 --2- 172.21.15.46:0/3379648530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6f8003bf0 0x7fb6f8003fe0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fb708007fd0 tx=0x7fb708007d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb69d229532c39c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.921+0000 7fb7112aa700 1 -- 172.21.15.46:0/3379648530 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7080013f0 con 0x7fb6f8003bf0 2024-01-27T01:06:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.921+0000 7fb7112aa700 1 -- 172.21.15.46:0/3379648530 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70800c590 con 0x7fb6f8003bf0 2024-01-27T01:06:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.921+0000 7fb7112aa700 1 -- 172.21.15.46:0/3379648530 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70800c950 con 0x7fb6f8003bf0 2024-01-27T01:06:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.921+0000 7fb714510700 1 -- 172.21.15.46:0/3379648530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6f8003bf0 msgr2=0x7fb6f8003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.921+0000 7fb714510700 1 --2- 172.21.15.46:0/3379648530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6f8003bf0 0x7fb6f8003fe0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fb708007fd0 tx=0x7fb708007d80 comp rx=0 tx=0).stop 2024-01-27T01:06:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.921+0000 7fb714510700 1 -- 172.21.15.46:0/3379648530 shutdown_connections 2024-01-27T01:06:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.921+0000 7fb714510700 1 --2- 172.21.15.46:0/3379648530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6f8003bf0 0x7fb6f8003fe0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.921+0000 7fb714510700 1 -- 172.21.15.46:0/3379648530 >> 172.21.15.46:0/3379648530 conn(0x7fb70c0ca100 msgr2=0x7fb70c06e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.921+0000 7fb714510700 1 -- 172.21.15.46:0/3379648530 shutdown_connections 2024-01-27T01:06:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.921+0000 7fb714510700 1 -- 172.21.15.46:0/3379648530 wait complete. 2024-01-27T01:06:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.923+0000 7fb714510700 1 Processor -- start 2024-01-27T01:06:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.923+0000 7fb714510700 1 -- start start 2024-01-27T01:06:14.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.923+0000 7fb714510700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6f8003bf0 0x7fb70c0ad490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:14.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.923+0000 7fb714510700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb70c0b07d0 0x7fb70c1691b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:14.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.923+0000 7fb714510700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb70c1696f0 0x7fb70c169b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:14.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb714510700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c0ae460 con 0x7fb70c1696f0 2024-01-27T01:06:14.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb714510700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c0ad9d0 con 0x7fb6f8003bf0 2024-01-27T01:06:14.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb714510700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c0b1150 con 0x7fb70c0b07d0 2024-01-27T01:06:14.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb712aad700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb70c1696f0 0x7fb70c169b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:14.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb712aad700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb70c1696f0 0x7fb70c169b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57774/0 (socket says 172.21.15.46:57774) 2024-01-27T01:06:14.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb712aad700 1 -- 172.21.15.46:0/177254574 learned_addr learned my addr 172.21.15.46:0/177254574 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:14.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb7122ac700 1 --2- 172.21.15.46:0/177254574 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6f8003bf0 0x7fb70c0ad490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:14.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb711aab700 1 --2- 172.21.15.46:0/177254574 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb70c0b07d0 0x7fb70c1691b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:14.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb712aad700 1 -- 172.21.15.46:0/177254574 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb70c0b07d0 msgr2=0x7fb70c1691b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:14.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb712aad700 1 --2- 172.21.15.46:0/177254574 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb70c0b07d0 0x7fb70c1691b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:14.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb712aad700 1 -- 172.21.15.46:0/177254574 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6f8003bf0 msgr2=0x7fb70c0ad490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:14.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb712aad700 1 --2- 172.21.15.46:0/177254574 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6f8003bf0 0x7fb70c0ad490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:14.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb712aad700 1 -- 172.21.15.46:0/177254574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70c16a090 con 0x7fb70c1696f0 2024-01-27T01:06:14.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb712aad700 1 --2- 172.21.15.46:0/177254574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb70c1696f0 0x7fb70c169b50 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fb70c157a90 tx=0x7fb704002170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:14.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb7027fc700 1 -- 172.21.15.46:0/177254574 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb704007ac0 con 0x7fb70c1696f0 2024-01-27T01:06:14.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb7027fc700 1 -- 172.21.15.46:0/177254574 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70400ac90 con 0x7fb70c1696f0 2024-01-27T01:06:14.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb7027fc700 1 -- 172.21.15.46:0/177254574 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb704013640 con 0x7fb70c1696f0 2024-01-27T01:06:14.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb714510700 1 -- 172.21.15.46:0/177254574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70c157330 con 0x7fb70c1696f0 2024-01-27T01:06:14.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.924+0000 7fb714510700 1 -- 172.21.15.46:0/177254574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb70c0adea0 con 0x7fb70c1696f0 2024-01-27T01:06:14.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.926+0000 7fb714510700 1 -- 172.21.15.46:0/177254574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70c0b12e0 con 0x7fb70c1696f0 2024-01-27T01:06:14.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.926+0000 7fb7027fc700 1 -- 172.21.15.46:0/177254574 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb7040139f0 con 0x7fb70c1696f0 2024-01-27T01:06:14.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.926+0000 7fb7027fc700 1 --2- 172.21.15.46:0/177254574 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f0066f30 0x7fb6f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:14.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.927+0000 7fb7027fc700 1 -- 172.21.15.46:0/177254574 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6267+0+0 (secure 0 0 0) 0x7fb704084db0 con 0x7fb70c1696f0 2024-01-27T01:06:14.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.927+0000 7fb7122ac700 1 --2- 172.21.15.46:0/177254574 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f0066f30 0x7fb6f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:14.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.927+0000 7fb7122ac700 1 --2- 172.21.15.46:0/177254574 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f0066f30 0x7fb6f00693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb708000c00 tx=0x7fb708015000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:14.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:14.931+0000 7fb7027fc700 1 -- 172.21.15.46:0/177254574 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb704089070 con 0x7fb70c1696f0 2024-01-27T01:06:15.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.134+0000 7fb714510700 1 -- 172.21.15.46:0/177254574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb70c04c710 con 0x7fb70c1696f0 2024-01-27T01:06:15.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.138+0000 7fb7027fc700 1 -- 172.21.15.46:0/177254574 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v76)=0 v76) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb704051d50 con 0x7fb70c1696f0 2024-01-27T01:06:15.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.140+0000 7fb714510700 1 -- 172.21.15.46:0/177254574 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f0066f30 msgr2=0x7fb6f00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:15.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.140+0000 7fb714510700 1 --2- 172.21.15.46:0/177254574 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f0066f30 0x7fb6f00693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb708000c00 tx=0x7fb708015000 comp rx=0 tx=0).stop 2024-01-27T01:06:15.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.141+0000 7fb714510700 1 -- 172.21.15.46:0/177254574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb70c1696f0 msgr2=0x7fb70c169b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:15.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.141+0000 7fb714510700 1 --2- 172.21.15.46:0/177254574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb70c1696f0 0x7fb70c169b50 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fb70c157a90 tx=0x7fb704002170 comp rx=0 tx=0).stop 2024-01-27T01:06:15.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.141+0000 7fb714510700 1 -- 172.21.15.46:0/177254574 shutdown_connections 2024-01-27T01:06:15.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.141+0000 7fb714510700 1 --2- 172.21.15.46:0/177254574 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f0066f30 0x7fb6f00693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:15.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.141+0000 7fb714510700 1 --2- 172.21.15.46:0/177254574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb70c1696f0 0x7fb70c169b50 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:15.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.141+0000 7fb714510700 1 --2- 172.21.15.46:0/177254574 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6f8003bf0 0x7fb70c0ad490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:15.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.141+0000 7fb714510700 1 --2- 172.21.15.46:0/177254574 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb70c0b07d0 0x7fb70c1691b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:15.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.141+0000 7fb714510700 1 -- 172.21.15.46:0/177254574 >> 172.21.15.46:0/177254574 conn(0x7fb70c0ca100 msgr2=0x7fb70c0cb170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:15.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.141+0000 7fb714510700 1 -- 172.21.15.46:0/177254574 shutdown_connections 2024-01-27T01:06:15.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.141+0000 7fb714510700 1 -- 172.21.15.46:0/177254574 wait complete. 2024-01-27T01:06:15.154 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-27T01:06:15.154 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:06:15.212 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:06:15.265 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-27T01:06:15.266 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-27T01:06:15.266 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:06:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.403+0000 7f92f6c23700 1 Processor -- start 2024-01-27T01:06:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.404+0000 7f92f6c23700 1 -- start start 2024-01-27T01:06:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.404+0000 7f92f6c23700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f92f0153ec0 0x7f92f0154290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.404+0000 7f92f6c23700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92f00b22d0 0x7f92f00b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.404+0000 7f92f6c23700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f015b140 con 0x7f92f0157630 2024-01-27T01:06:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.404+0000 7f92f6c23700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f015a5c0 con 0x7f92f0153ec0 2024-01-27T01:06:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.404+0000 7f92f6c23700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f015a8a0 con 0x7f92f00b22d0 2024-01-27T01:06:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.405+0000 7f92f49bf700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f92f0153ec0 0x7f92f0154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.405+0000 7f92effff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f92f0157630 0x7f92f00b1bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41174/0 (socket says 172.21.15.46:41174) 2024-01-27T01:06:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.405+0000 7f92effff700 1 -- 172.21.15.46:0/4184067596 learned_addr learned my addr 172.21.15.46:0/4184067596 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.405+0000 7f92f51c0700 1 --2- 172.21.15.46:0/4184067596 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92f00b22d0 0x7f92f00b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.405+0000 7f92f49bf700 1 -- 172.21.15.46:0/4184067596 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92f00b22d0 msgr2=0x7f92f00b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.405+0000 7f92f49bf700 1 --2- 172.21.15.46:0/4184067596 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92f00b22d0 0x7f92f00b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.405+0000 7f92f49bf700 1 -- 172.21.15.46:0/4184067596 >> v1:172.21.15.46:6789/0 conn(0x7f92f0157630 legacy=0x7f92f00b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.405+0000 7f92f49bf700 1 -- 172.21.15.46:0/4184067596 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f015ab80 con 0x7f92f0153ec0 2024-01-27T01:06:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.405+0000 7f92ef7fe700 1 -- 172.21.15.46:0/4184067596 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888948755 0 0) 0x7f92f015b140 con 0x7f92f0157630 2024-01-27T01:06:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.405+0000 7f92f49bf700 1 --2- 172.21.15.46:0/4184067596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f0153ec0 0x7f92f0154290 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f92e40098e0 tx=0x7f92e40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb0f832ae2ba1f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.405+0000 7f92ef7fe700 1 -- 172.21.15.46:0/4184067596 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e4009e80 con 0x7f92f0153ec0 2024-01-27T01:06:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.405+0000 7f92ef7fe700 1 -- 172.21.15.46:0/4184067596 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e400b860 con 0x7f92f0153ec0 2024-01-27T01:06:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.405+0000 7f92ef7fe700 1 -- 172.21.15.46:0/4184067596 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e400bbe0 con 0x7f92f0153ec0 2024-01-27T01:06:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.406+0000 7f92f6c23700 1 -- 172.21.15.46:0/4184067596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f0153ec0 msgr2=0x7f92f0154290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.406+0000 7f92f6c23700 1 --2- 172.21.15.46:0/4184067596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f0153ec0 0x7f92f0154290 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f92e40098e0 tx=0x7f92e40091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.406+0000 7f92f6c23700 1 -- 172.21.15.46:0/4184067596 shutdown_connections 2024-01-27T01:06:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.406+0000 7f92f6c23700 1 --2- 172.21.15.46:0/4184067596 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92f00b22d0 0x7f92f00b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.406+0000 7f92f6c23700 1 --2- 172.21.15.46:0/4184067596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f0153ec0 0x7f92f0154290 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.406+0000 7f92f6c23700 1 -- 172.21.15.46:0/4184067596 >> 172.21.15.46:0/4184067596 conn(0x7f92f0069510 msgr2=0x7f92f00aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.406+0000 7f92f6c23700 1 -- 172.21.15.46:0/4184067596 shutdown_connections 2024-01-27T01:06:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.406+0000 7f92f6c23700 1 -- 172.21.15.46:0/4184067596 wait complete. 2024-01-27T01:06:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.407+0000 7f92f6c23700 1 Processor -- start 2024-01-27T01:06:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.407+0000 7f92f6c23700 1 -- start start 2024-01-27T01:06:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.408+0000 7f92f6c23700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f00b22d0 0x7f92f00aa380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.408+0000 7f92f6c23700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92f0153ec0 0x7f92f00cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.408+0000 7f92f6c23700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92f0157630 0x7f92f01527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.408+0000 7f92f6c23700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f00cd010 con 0x7f92f00b22d0 2024-01-27T01:06:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.408+0000 7f92f6c23700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f00aa8c0 con 0x7f92f0153ec0 2024-01-27T01:06:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.408+0000 7f92f6c23700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f00aab40 con 0x7f92f0157630 2024-01-27T01:06:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.408+0000 7f92f49bf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f00b22d0 0x7f92f00aa380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.408+0000 7f92f49bf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f00b22d0 0x7f92f00aa380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57790/0 (socket says 172.21.15.46:57790) 2024-01-27T01:06:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.408+0000 7f92f49bf700 1 -- 172.21.15.46:0/3741819175 learned_addr learned my addr 172.21.15.46:0/3741819175 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.409+0000 7f92f51c0700 1 --2- 172.21.15.46:0/3741819175 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92f0157630 0x7f92f01527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.409+0000 7f92effff700 1 --2- 172.21.15.46:0/3741819175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92f0153ec0 0x7f92f00cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.409+0000 7f92f49bf700 1 -- 172.21.15.46:0/3741819175 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92f0157630 msgr2=0x7f92f01527e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.409+0000 7f92f49bf700 1 --2- 172.21.15.46:0/3741819175 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92f0157630 0x7f92f01527e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.409+0000 7f92f49bf700 1 -- 172.21.15.46:0/3741819175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92f0153ec0 msgr2=0x7f92f00cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.409+0000 7f92f49bf700 1 --2- 172.21.15.46:0/3741819175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92f0153ec0 0x7f92f00cab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.409+0000 7f92f49bf700 1 -- 172.21.15.46:0/3741819175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e4009420 con 0x7f92f00b22d0 2024-01-27T01:06:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.409+0000 7f92f49bf700 1 --2- 172.21.15.46:0/3741819175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f00b22d0 0x7f92f00aa380 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f92f006d620 tx=0x7f92e4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.409+0000 7f92ecff9700 1 -- 172.21.15.46:0/3741819175 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e400b9d0 con 0x7f92f00b22d0 2024-01-27T01:06:15.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.409+0000 7f92ecff9700 1 -- 172.21.15.46:0/3741819175 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e400bb30 con 0x7f92f00b22d0 2024-01-27T01:06:15.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.409+0000 7f92ecff9700 1 -- 172.21.15.46:0/3741819175 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e4011650 con 0x7f92f00b22d0 2024-01-27T01:06:15.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.409+0000 7f92f6c23700 1 -- 172.21.15.46:0/3741819175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f016ac00 con 0x7f92f00b22d0 2024-01-27T01:06:15.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.409+0000 7f92f6c23700 1 -- 172.21.15.46:0/3741819175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f00ccca0 con 0x7f92f00b22d0 2024-01-27T01:06:15.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.410+0000 7f92f6c23700 1 -- 172.21.15.46:0/3741819175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f004c710 con 0x7f92f00b22d0 2024-01-27T01:06:15.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.411+0000 7f92ecff9700 1 -- 172.21.15.46:0/3741819175 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f92e4011a00 con 0x7f92f00b22d0 2024-01-27T01:06:15.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.411+0000 7f92ecff9700 1 --2- 172.21.15.46:0/3741819175 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f92d4066f30 0x7f92d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:15.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.411+0000 7f92ecff9700 1 -- 172.21.15.46:0/3741819175 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6267+0+0 (secure 0 0 0) 0x7f92e4085b80 con 0x7f92f00b22d0 2024-01-27T01:06:15.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.411+0000 7f92effff700 1 --2- 172.21.15.46:0/3741819175 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f92d4066f30 0x7f92d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:15.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.412+0000 7f92effff700 1 --2- 172.21.15.46:0/3741819175 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f92d4066f30 0x7f92d40693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f92f01512d0 tx=0x7f92dc007d30 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:15.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.415+0000 7f92ecff9700 1 -- 172.21.15.46:0/3741819175 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f92e4052c40 con 0x7f92f00b22d0 2024-01-27T01:06:15.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.667+0000 7f92f6c23700 1 -- 172.21.15.46:0/3741819175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92f006fa80 con 0x7f92f00b22d0 2024-01-27T01:06:15.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.667+0000 7f92ecff9700 1 -- 172.21.15.46:0/3741819175 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+3533 (secure 0 0 0) 0x7f92e4028090 con 0x7f92f00b22d0 2024-01-27T01:06:15.668 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:15.668 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":109,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8891,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/975115379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":975115379},{"type":"v1","addr":"172.21.15.106:6833","nonce":975115379}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":9155,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/26110835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":26110835},{"type":"v1","addr":"172.21.15.46:6835","nonce":26110835}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":10036,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3860120510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3860120510},{"type":"v1","addr":"172.21.15.46:6834","nonce":3860120510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101},{"gid":10045,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/3603549739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":3603549739},{"type":"v1","addr":"172.21.15.106:6835","nonce":3603549739}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":102}],"filesystems":[]} 2024-01-27T01:06:15.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.670+0000 7f92f6c23700 1 -- 172.21.15.46:0/3741819175 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f92d4066f30 msgr2=0x7f92d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:15.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.670+0000 7f92f6c23700 1 --2- 172.21.15.46:0/3741819175 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f92d4066f30 0x7f92d40693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f92f01512d0 tx=0x7f92dc007d30 comp rx=0 tx=0).stop 2024-01-27T01:06:15.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.670+0000 7f92f6c23700 1 -- 172.21.15.46:0/3741819175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f00b22d0 msgr2=0x7f92f00aa380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:15.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.670+0000 7f92f6c23700 1 --2- 172.21.15.46:0/3741819175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f00b22d0 0x7f92f00aa380 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f92f006d620 tx=0x7f92e4017000 comp rx=0 tx=0).stop 2024-01-27T01:06:15.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.670+0000 7f92f6c23700 1 -- 172.21.15.46:0/3741819175 shutdown_connections 2024-01-27T01:06:15.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.670+0000 7f92f6c23700 1 --2- 172.21.15.46:0/3741819175 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f92d4066f30 0x7f92d40693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:15.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.670+0000 7f92f6c23700 1 --2- 172.21.15.46:0/3741819175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f00b22d0 0x7f92f00aa380 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:15.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.670+0000 7f92f6c23700 1 --2- 172.21.15.46:0/3741819175 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92f0157630 0x7f92f01527e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:15.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.670+0000 7f92f6c23700 1 --2- 172.21.15.46:0/3741819175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92f0153ec0 0x7f92f00cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:15.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.670+0000 7f92f6c23700 1 -- 172.21.15.46:0/3741819175 >> 172.21.15.46:0/3741819175 conn(0x7f92f0069510 msgr2=0x7f92f0154800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:15.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.670+0000 7f92f6c23700 1 -- 172.21.15.46:0/3741819175 shutdown_connections 2024-01-27T01:06:15.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:15.670+0000 7f92f6c23700 1 -- 172.21.15.46:0/3741819175 wait complete. 2024-01-27T01:06:15.671 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 109 2024-01-27T01:06:15.685 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-27T01:06:15.685 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-27T01:06:15.685 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-27T01:06:15.685 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:06:15.685 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-27T01:06:15.685 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-27T01:06:15.685 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-27T01:06:15.686 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:06:15.686 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-27T01:06:15.686 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-27T01:06:15.686 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-27T01:06:15.686 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:06:15.686 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-27T01:06:15.686 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-27T01:06:15.686 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-27T01:06:15.687 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:06:15.695 INFO:tasks.ceph.mds.c:Stopped 2024-01-27T01:06:15.696 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-27T01:06:15.700 INFO:tasks.ceph.mds.d:Stopped 2024-01-27T01:06:15.700 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-27T01:06:15.701 INFO:tasks.ceph.mds.b:Stopped 2024-01-27T01:06:15.701 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-27T01:06:15.738 INFO:tasks.ceph.mds.c:Started 2024-01-27T01:06:15.739 DEBUG:teuthology.parallel:result is None 2024-01-27T01:06:15.744 INFO:tasks.ceph.mds.d:Started 2024-01-27T01:06:15.744 INFO:tasks.ceph.mds.b:Started 2024-01-27T01:06:15.745 DEBUG:teuthology.parallel:result is None 2024-01-27T01:06:15.745 DEBUG:teuthology.parallel:result is None 2024-01-27T01:06:15.824 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-27T01:06:15.828 INFO:tasks.ceph.mds.b.smithi106.stdout:starting mds.b at 2024-01-27T01:06:15.828 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-01-27T01:06:19.870 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-01-27T01:06:21.695 INFO:tasks.ceph.mds.a:Stopped 2024-01-27T01:06:21.696 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-27T01:06:21.699 INFO:tasks.ceph.mds.a:Started 2024-01-27T01:06:21.699 DEBUG:teuthology.parallel:result is None 2024-01-27T01:06:21.699 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-27T01:06:21.791 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-27T01:06:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.846+0000 7fb3a57dd700 1 Processor -- start 2024-01-27T01:06:21.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.846+0000 7fb3a57dd700 1 -- start start 2024-01-27T01:06:21.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.846+0000 7fb3a57dd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb3a00cbdf0 0x7fb3a00ae610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:21.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb3a57dd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb3a00cc1c0 0x7fb3a00b0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:21.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb3a57dd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3a00ad070 0x7fb3a00b1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:21.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb3a57dd700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a00aeb50 con 0x7fb3a00cc1c0 2024-01-27T01:06:21.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb3a57dd700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a00ae180 con 0x7fb3a00cbdf0 2024-01-27T01:06:21.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb3a57dd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a0156c60 con 0x7fb3a00ad070 2024-01-27T01:06:21.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb39e7fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb3a00cc1c0 0x7fb3a00b0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:21.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb39e7fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb3a00cc1c0 0x7fb3a00b0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44300/0 (socket says 172.21.15.46:44300) 2024-01-27T01:06:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb39e7fc700 1 -- 172.21.15.46:0/983962908 learned_addr learned my addr 172.21.15.46:0/983962908 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb39f7fe700 1 --2- 172.21.15.46:0/983962908 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3a00ad070 0x7fb3a00b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb39effd700 1 --2- 172.21.15.46:0/983962908 >> v2:172.21.15.106:3300/0 conn(0x7fb3a00cbdf0 0x7fb3a00ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb39e7fc700 1 -- 172.21.15.46:0/983962908 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3a00ad070 msgr2=0x7fb3a00b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb39e7fc700 1 --2- 172.21.15.46:0/983962908 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3a00ad070 0x7fb3a00b1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb39e7fc700 1 -- 172.21.15.46:0/983962908 >> v2:172.21.15.106:3300/0 conn(0x7fb3a00cbdf0 msgr2=0x7fb3a00ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb39e7fc700 1 --2- 172.21.15.46:0/983962908 >> v2:172.21.15.106:3300/0 conn(0x7fb3a00cbdf0 0x7fb3a00ae610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb39e7fc700 1 -- 172.21.15.46:0/983962908 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a00ad570 con 0x7fb3a00cc1c0 2024-01-27T01:06:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb39e7fc700 1 --2- 172.21.15.46:0/983962908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a00cc1c0 0x7fb3a00b0e10 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fb3a006ee20 tx=0x7fb38800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bf7f86ef9d00205 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb39dffb700 1 -- 172.21.15.46:0/983962908 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38800dc50 con 0x7fb3a00cc1c0 2024-01-27T01:06:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.847+0000 7fb39dffb700 1 -- 172.21.15.46:0/983962908 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38800ddb0 con 0x7fb3a00cc1c0 2024-01-27T01:06:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.848+0000 7fb39dffb700 1 -- 172.21.15.46:0/983962908 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3880196d0 con 0x7fb3a00cc1c0 2024-01-27T01:06:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.848+0000 7fb3a57dd700 1 -- 172.21.15.46:0/983962908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a00cc1c0 msgr2=0x7fb3a00b0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:21.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.848+0000 7fb3a57dd700 1 --2- 172.21.15.46:0/983962908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a00cc1c0 0x7fb3a00b0e10 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fb3a006ee20 tx=0x7fb38800b4f0 comp rx=0 tx=0).stop 2024-01-27T01:06:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.848+0000 7fb3a57dd700 1 -- 172.21.15.46:0/983962908 shutdown_connections 2024-01-27T01:06:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.848+0000 7fb3a57dd700 1 --2- 172.21.15.46:0/983962908 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3a00ad070 0x7fb3a00b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.848+0000 7fb3a57dd700 1 --2- 172.21.15.46:0/983962908 >> v2:172.21.15.106:3300/0 conn(0x7fb3a00cbdf0 0x7fb3a00ae610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.848+0000 7fb3a57dd700 1 --2- 172.21.15.46:0/983962908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a00cc1c0 0x7fb3a00b0e10 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.848+0000 7fb3a57dd700 1 -- 172.21.15.46:0/983962908 >> 172.21.15.46:0/983962908 conn(0x7fb3a0068400 msgr2=0x7fb3a00cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.848+0000 7fb3a57dd700 1 -- 172.21.15.46:0/983962908 shutdown_connections 2024-01-27T01:06:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.848+0000 7fb3a57dd700 1 -- 172.21.15.46:0/983962908 wait complete. 2024-01-27T01:06:21.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.850+0000 7fb3a57dd700 1 Processor -- start 2024-01-27T01:06:21.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.850+0000 7fb3a57dd700 1 -- start start 2024-01-27T01:06:21.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb3a57dd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a00ad070 0x7fb3a00c6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:21.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb3a57dd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3a00cbdf0 0x7fb3a01542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb3a57dd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3a006a9d0 0x7fb3a006ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb3a57dd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a006b380 con 0x7fb3a00ad070 2024-01-27T01:06:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb3a57dd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a00c4af0 con 0x7fb3a006a9d0 2024-01-27T01:06:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb3a57dd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a00c6580 con 0x7fb3a00cbdf0 2024-01-27T01:06:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb39effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a00ad070 0x7fb3a00c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb39effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a00ad070 0x7fb3a00c6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44308/0 (socket says 172.21.15.46:44308) 2024-01-27T01:06:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb39effd700 1 -- 172.21.15.46:0/1847879343 learned_addr learned my addr 172.21.15.46:0/1847879343 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb39f7fe700 1 --2- 172.21.15.46:0/1847879343 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3a006a9d0 0x7fb3a006ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb39e7fc700 1 --2- 172.21.15.46:0/1847879343 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3a00cbdf0 0x7fb3a01542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb39effd700 1 -- 172.21.15.46:0/1847879343 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3a00cbdf0 msgr2=0x7fb3a01542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb39effd700 1 --2- 172.21.15.46:0/1847879343 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3a00cbdf0 0x7fb3a01542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb39effd700 1 -- 172.21.15.46:0/1847879343 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3a006a9d0 msgr2=0x7fb3a006ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb39effd700 1 --2- 172.21.15.46:0/1847879343 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3a006a9d0 0x7fb3a006ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb39effd700 1 -- 172.21.15.46:0/1847879343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38800b790 con 0x7fb3a00ad070 2024-01-27T01:06:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb39effd700 1 --2- 172.21.15.46:0/1847879343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a00ad070 0x7fb3a00c6040 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fb39000e190 tx=0x7fb390009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb396ffd700 1 -- 172.21.15.46:0/1847879343 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39000ecd0 con 0x7fb3a00ad070 2024-01-27T01:06:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb396ffd700 1 -- 172.21.15.46:0/1847879343 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39000ee30 con 0x7fb3a00ad070 2024-01-27T01:06:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb3a57dd700 1 -- 172.21.15.46:0/1847879343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a00c4810 con 0x7fb3a00ad070 2024-01-27T01:06:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb396ffd700 1 -- 172.21.15.46:0/1847879343 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb390014670 con 0x7fb3a00ad070 2024-01-27T01:06:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.852+0000 7fb3a57dd700 1 -- 172.21.15.46:0/1847879343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a00c1a20 con 0x7fb3a00ad070 2024-01-27T01:06:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.853+0000 7fb3a57dd700 1 -- 172.21.15.46:0/1847879343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a00c2070 con 0x7fb3a00ad070 2024-01-27T01:06:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.854+0000 7fb396ffd700 1 -- 172.21.15.46:0/1847879343 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb390014a20 con 0x7fb3a00ad070 2024-01-27T01:06:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.854+0000 7fb396ffd700 1 --2- 172.21.15.46:0/1847879343 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb380066f30 0x7fb3800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.854+0000 7fb396ffd700 1 -- 172.21.15.46:0/1847879343 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6267+0+0 (secure 0 0 0) 0x7fb390084df0 con 0x7fb3a00ad070 2024-01-27T01:06:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.854+0000 7fb39e7fc700 1 --2- 172.21.15.46:0/1847879343 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb380066f30 0x7fb3800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.855+0000 7fb39e7fc700 1 --2- 172.21.15.46:0/1847879343 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb380066f30 0x7fb3800693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb3a006ee20 tx=0x7fb38801b000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:21.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:21.859+0000 7fb396ffd700 1 -- 172.21.15.46:0/1847879343 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb390051e30 con 0x7fb3a00ad070 2024-01-27T01:06:22.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.075+0000 7fb3a57dd700 1 -- 172.21.15.46:0/1847879343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb3a004c710 con 0x7fb3a00ad070 2024-01-27T01:06:22.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.831+0000 7fb396ffd700 1 -- 172.21.15.46:0/1847879343 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v219) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb390009a60 con 0x7fb3a00ad070 2024-01-27T01:06:22.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.834+0000 7fb3a57dd700 1 -- 172.21.15.46:0/1847879343 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb380066f30 msgr2=0x7fb3800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:22.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.834+0000 7fb3a57dd700 1 --2- 172.21.15.46:0/1847879343 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb380066f30 0x7fb3800693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb3a006ee20 tx=0x7fb38801b000 comp rx=0 tx=0).stop 2024-01-27T01:06:22.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.834+0000 7fb3a57dd700 1 -- 172.21.15.46:0/1847879343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a00ad070 msgr2=0x7fb3a00c6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:22.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.834+0000 7fb3a57dd700 1 --2- 172.21.15.46:0/1847879343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a00ad070 0x7fb3a00c6040 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fb39000e190 tx=0x7fb390009920 comp rx=0 tx=0).stop 2024-01-27T01:06:22.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.834+0000 7fb3a57dd700 1 -- 172.21.15.46:0/1847879343 shutdown_connections 2024-01-27T01:06:22.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.834+0000 7fb3a57dd700 1 --2- 172.21.15.46:0/1847879343 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb380066f30 0x7fb3800693e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:22.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.834+0000 7fb3a57dd700 1 --2- 172.21.15.46:0/1847879343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a00ad070 0x7fb3a00c6040 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:22.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.834+0000 7fb3a57dd700 1 --2- 172.21.15.46:0/1847879343 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3a006a9d0 0x7fb3a006ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:22.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.834+0000 7fb3a57dd700 1 --2- 172.21.15.46:0/1847879343 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3a00cbdf0 0x7fb3a01542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:22.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.834+0000 7fb3a57dd700 1 -- 172.21.15.46:0/1847879343 >> 172.21.15.46:0/1847879343 conn(0x7fb3a0068400 msgr2=0x7fb3a015a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:22.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.834+0000 7fb3a57dd700 1 -- 172.21.15.46:0/1847879343 shutdown_connections 2024-01-27T01:06:22.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.834+0000 7fb3a57dd700 1 -- 172.21.15.46:0/1847879343 wait complete. 2024-01-27T01:06:22.834 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-01-27T01:06:22.848 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-27T01:06:22.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.985+0000 7ff8d2bb7700 1 Processor -- start 2024-01-27T01:06:22.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.985+0000 7ff8d2bb7700 1 -- start start 2024-01-27T01:06:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.986+0000 7ff8d2bb7700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff8cc1561d0 0x7ff8cc0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.986+0000 7ff8d2bb7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8cc0b0280 0x7ff8cc1565a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.986+0000 7ff8d2bb7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8cc06eb70 0x7ff8cc06c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.986+0000 7ff8d2bb7700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8cc0b0760 con 0x7ff8cc06eb70 2024-01-27T01:06:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.986+0000 7ff8d2bb7700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8cc0b0a40 con 0x7ff8cc1561d0 2024-01-27T01:06:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.986+0000 7ff8d2bb7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc0b0d20 con 0x7ff8cc0b0280 2024-01-27T01:06:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.986+0000 7ff8d1154700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8cc06eb70 0x7ff8cc06c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.986+0000 7ff8d1154700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8cc06eb70 0x7ff8cc06c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44316/0 (socket says 172.21.15.46:44316) 2024-01-27T01:06:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.986+0000 7ff8d1154700 1 -- 172.21.15.46:0/1323437788 learned_addr learned my addr 172.21.15.46:0/1323437788 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.986+0000 7ff8cbfff700 1 --2- 172.21.15.46:0/1323437788 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8cc0b0280 0x7ff8cc1565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.986+0000 7ff8d0953700 1 --2- 172.21.15.46:0/1323437788 >> v2:172.21.15.106:3300/0 conn(0x7ff8cc1561d0 0x7ff8cc0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.986+0000 7ff8d1154700 1 -- 172.21.15.46:0/1323437788 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8cc0b0280 msgr2=0x7ff8cc1565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.986+0000 7ff8d1154700 1 --2- 172.21.15.46:0/1323437788 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8cc0b0280 0x7ff8cc1565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:22.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.986+0000 7ff8d1154700 1 -- 172.21.15.46:0/1323437788 >> v2:172.21.15.106:3300/0 conn(0x7ff8cc1561d0 msgr2=0x7ff8cc0ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:22.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.986+0000 7ff8d1154700 1 --2- 172.21.15.46:0/1323437788 >> v2:172.21.15.106:3300/0 conn(0x7ff8cc1561d0 0x7ff8cc0ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:22.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.986+0000 7ff8d1154700 1 -- 172.21.15.46:0/1323437788 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8cc0b1000 con 0x7ff8cc06eb70 2024-01-27T01:06:22.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.987+0000 7ff8d1154700 1 --2- 172.21.15.46:0/1323437788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8cc06eb70 0x7ff8cc06c4a0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc06bb50 tx=0x7ff8c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5256b8602d96dbc server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:22.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.987+0000 7ff8cb7fe700 1 -- 172.21.15.46:0/1323437788 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c400e070 con 0x7ff8cc06eb70 2024-01-27T01:06:22.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.987+0000 7ff8cb7fe700 1 -- 172.21.15.46:0/1323437788 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c400ba50 con 0x7ff8cc06eb70 2024-01-27T01:06:22.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.987+0000 7ff8cb7fe700 1 -- 172.21.15.46:0/1323437788 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c40092e0 con 0x7ff8cc06eb70 2024-01-27T01:06:22.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.987+0000 7ff8d2bb7700 1 -- 172.21.15.46:0/1323437788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8cc06eb70 msgr2=0x7ff8cc06c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:22.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.987+0000 7ff8d2bb7700 1 --2- 172.21.15.46:0/1323437788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8cc06eb70 0x7ff8cc06c4a0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc06bb50 tx=0x7ff8c4009840 comp rx=0 tx=0).stop 2024-01-27T01:06:22.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.987+0000 7ff8d2bb7700 1 -- 172.21.15.46:0/1323437788 shutdown_connections 2024-01-27T01:06:22.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.987+0000 7ff8d2bb7700 1 --2- 172.21.15.46:0/1323437788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8cc06eb70 0x7ff8cc06c4a0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:22.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.987+0000 7ff8d2bb7700 1 --2- 172.21.15.46:0/1323437788 >> v2:172.21.15.106:3300/0 conn(0x7ff8cc1561d0 0x7ff8cc0ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:22.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.987+0000 7ff8d2bb7700 1 --2- 172.21.15.46:0/1323437788 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8cc0b0280 0x7ff8cc1565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:22.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.987+0000 7ff8d2bb7700 1 -- 172.21.15.46:0/1323437788 >> 172.21.15.46:0/1323437788 conn(0x7ff8cc06b630 msgr2=0x7ff8cc0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:22.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.987+0000 7ff8d2bb7700 1 -- 172.21.15.46:0/1323437788 shutdown_connections 2024-01-27T01:06:22.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.987+0000 7ff8d2bb7700 1 -- 172.21.15.46:0/1323437788 wait complete. 2024-01-27T01:06:22.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.989+0000 7ff8d2bb7700 1 Processor -- start 2024-01-27T01:06:22.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.989+0000 7ff8d2bb7700 1 -- start start 2024-01-27T01:06:22.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8d2bb7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8cc06eb70 0x7ff8cc14e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:22.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8d2bb7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8cc0b0280 0x7ff8cc15a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:22.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8d2bb7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8cc1561d0 0x7ff8cc15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:22.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8d2bb7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc15b010 con 0x7ff8cc06eb70 2024-01-27T01:06:22.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8d2bb7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc159620 con 0x7ff8cc1561d0 2024-01-27T01:06:22.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8d2bb7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc14f3c0 con 0x7ff8cc0b0280 2024-01-27T01:06:22.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8d0953700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8cc06eb70 0x7ff8cc14e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:22.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8d0953700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8cc06eb70 0x7ff8cc14e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44322/0 (socket says 172.21.15.46:44322) 2024-01-27T01:06:22.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8d0953700 1 -- 172.21.15.46:0/3116464268 learned_addr learned my addr 172.21.15.46:0/3116464268 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:22.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8d1154700 1 --2- 172.21.15.46:0/3116464268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8cc1561d0 0x7ff8cc15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:22.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8cbfff700 1 --2- 172.21.15.46:0/3116464268 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8cc0b0280 0x7ff8cc15a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:22.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8d0953700 1 -- 172.21.15.46:0/3116464268 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8cc0b0280 msgr2=0x7ff8cc15a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:22.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8d0953700 1 --2- 172.21.15.46:0/3116464268 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8cc0b0280 0x7ff8cc15a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:22.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8d0953700 1 -- 172.21.15.46:0/3116464268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8cc1561d0 msgr2=0x7ff8cc15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:22.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8d0953700 1 --2- 172.21.15.46:0/3116464268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8cc1561d0 0x7ff8cc15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:22.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8d0953700 1 -- 172.21.15.46:0/3116464268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c4009ae0 con 0x7ff8cc06eb70 2024-01-27T01:06:22.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8d1154700 1 --2- 172.21.15.46:0/3116464268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8cc1561d0 0x7ff8cc15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:06:22.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.990+0000 7ff8d0953700 1 --2- 172.21.15.46:0/3116464268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8cc06eb70 0x7ff8cc14e940 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000d860 tx=0x7ff8c0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:22.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.991+0000 7ff8c8ff9700 1 -- 172.21.15.46:0/3116464268 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0009610 con 0x7ff8cc06eb70 2024-01-27T01:06:22.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.991+0000 7ff8c8ff9700 1 -- 172.21.15.46:0/3116464268 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c0009770 con 0x7ff8cc06eb70 2024-01-27T01:06:22.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.991+0000 7ff8c8ff9700 1 -- 172.21.15.46:0/3116464268 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0009b30 con 0x7ff8cc06eb70 2024-01-27T01:06:22.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.991+0000 7ff8d2bb7700 1 -- 172.21.15.46:0/3116464268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8cc14ee80 con 0x7ff8cc06eb70 2024-01-27T01:06:22.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.991+0000 7ff8d2bb7700 1 -- 172.21.15.46:0/3116464268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8cc158fd0 con 0x7ff8cc06eb70 2024-01-27T01:06:22.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.992+0000 7ff8d2bb7700 1 -- 172.21.15.46:0/3116464268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8cc04c710 con 0x7ff8cc06eb70 2024-01-27T01:06:22.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.992+0000 7ff8c8ff9700 1 -- 172.21.15.46:0/3116464268 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff8c001e460 con 0x7ff8cc06eb70 2024-01-27T01:06:22.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.993+0000 7ff8c8ff9700 1 --2- 172.21.15.46:0/3116464268 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff8b0066f30 0x7ff8b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:22.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.993+0000 7ff8c8ff9700 1 -- 172.21.15.46:0/3116464268 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5665+0+0 (secure 0 0 0) 0x7ff8c0084b30 con 0x7ff8cc06eb70 2024-01-27T01:06:22.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.993+0000 7ff8cbfff700 1 --2- 172.21.15.46:0/3116464268 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff8b0066f30 0x7ff8b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:22.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.993+0000 7ff8cbfff700 1 --2- 172.21.15.46:0/3116464268 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff8b0066f30 0x7ff8b00693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8006fd0 tx=0x7ff8b8008040 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:22.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:22.997+0000 7ff8c8ff9700 1 -- 172.21.15.46:0/3116464268 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff8c0051dc0 con 0x7ff8cc06eb70 2024-01-27T01:06:23.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.283+0000 7ff8d2bb7700 1 -- 172.21.15.46:0/3116464268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff8cc0b12e0 con 0x7ff8cc06eb70 2024-01-27T01:06:23.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.284+0000 7ff8c8ff9700 1 -- 172.21.15.46:0/3116464268 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v53) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ff8c00553e0 con 0x7ff8cc06eb70 2024-01-27T01:06:23.283 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:23.283 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:06:23.283 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-01-27T01:06:23.283 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.284 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-01-27T01:06:23.284 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlM6dKKBAAPpeWeDZWrFgc/4zuxKCvHw==", 2024-01-27T01:06:23.284 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.284 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:06:23.284 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:06:23.284 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:06:23.284 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:06:23.284 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.284 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.284 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.285 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-01-27T01:06:23.285 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlmzHRNRAAflWYZ/p4AXeMy54tvLpugA==", 2024-01-27T01:06:23.285 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.285 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:06:23.285 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:06:23.285 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:06:23.285 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:06:23.285 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.285 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.285 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.285 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-01-27T01:06:23.286 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlMZxoMBAAG5kyfm+LLG86y/p2A8BfDQ==", 2024-01-27T01:06:23.286 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.286 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:06:23.286 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:06:23.286 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:06:23.286 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:06:23.286 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.286 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.286 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.286 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-01-27T01:06:23.286 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl5jx/ARAA3cwtLAOWQ/n2NG+xwEZ3QQ==", 2024-01-27T01:06:23.286 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.286 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:06:23.286 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:06:23.286 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:06:23.287 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:06:23.287 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.287 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.287 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.287 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-01-27T01:06:23.287 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4QLRlJ2OKJRAAWVhxEdQkUW20294w/KLL6g==", 2024-01-27T01:06:23.287 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.287 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:06:23.287 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:06:23.287 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:06:23.287 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.287 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.287 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.287 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-01-27T01:06:23.287 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7QLRlE2tAJRAAzNlTiCr6a2eWiabRy1SqSw==", 2024-01-27T01:06:23.288 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.288 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:06:23.288 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:06:23.288 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:06:23.288 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.288 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.288 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.288 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-01-27T01:06:23.288 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+QLRlyySaJBAA5072ba5HUNi2dYE3D02M/Q==", 2024-01-27T01:06:23.288 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.288 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:06:23.288 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:06:23.288 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:06:23.288 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.289 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.289 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.289 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-01-27T01:06:23.289 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBQLRl9+tAIxAApZk0btXIpQPWSb3BYUYsFQ==", 2024-01-27T01:06:23.289 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.289 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:06:23.289 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:06:23.289 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:06:23.289 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.289 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.289 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.289 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-01-27T01:06:23.289 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHQLRlurUtJxAAT9fwKrAWl62VOXwShg7NJg==", 2024-01-27T01:06:23.289 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.289 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:06:23.290 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:06:23.290 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:06:23.290 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.290 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.290 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.290 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-01-27T01:06:23.290 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKQLRl7aqoJRAAXu9fykWELsW3SwnahoIOLA==", 2024-01-27T01:06:23.290 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.290 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:06:23.290 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:06:23.290 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:06:23.290 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.290 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.290 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.290 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-01-27T01:06:23.291 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNQLRlWf7pJBAAopmFLIKb8s32YZs46VwbGg==", 2024-01-27T01:06:23.291 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.291 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:06:23.291 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:06:23.291 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:06:23.291 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.291 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.291 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.291 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-01-27T01:06:23.291 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQQLRlNGfhIxAAmCgmOB3QAGfSIYHBb5BUZw==", 2024-01-27T01:06:23.291 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.291 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:06:23.291 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:06:23.291 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:06:23.291 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.292 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.292 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.292 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-01-27T01:06:23.292 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRleO8wBhAA1018AcO3KcyOZxgB0Itxaw==", 2024-01-27T01:06:23.292 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.292 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:06:23.292 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-01-27T01:06:23.292 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-27T01:06:23.292 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.292 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.292 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.292 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-01-27T01:06:23.292 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlsiZEDRAAjXSamHe2bZcDqVsqXqIcGg==", 2024-01-27T01:06:23.292 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.292 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:06:23.293 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T01:06:23.293 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T01:06:23.293 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T01:06:23.293 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.293 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.293 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.293 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-01-27T01:06:23.293 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl6YT1CRAApXlN4j20xzwda0tDMy94Eg==", 2024-01-27T01:06:23.293 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.293 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:06:23.293 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T01:06:23.293 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T01:06:23.293 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T01:06:23.293 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.293 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.294 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.294 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-01-27T01:06:23.294 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlHvDQEhAAxMhgWJsZC5/6Af5zFV6XIA==", 2024-01-27T01:06:23.294 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.294 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:06:23.294 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T01:06:23.294 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T01:06:23.294 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T01:06:23.294 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.294 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.294 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.294 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-01-27T01:06:23.294 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlvMwDCBAAXnFyMduz7PfWdLWbjytIiQ==", 2024-01-27T01:06:23.294 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.294 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T01:06:23.295 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:06:23.295 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:06:23.295 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:06:23.295 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.295 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.295 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.295 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-01-27T01:06:23.295 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlhMNSJBAA38QU6DrCvFjO+u9zQ8bkEA==", 2024-01-27T01:06:23.295 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.295 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T01:06:23.295 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-27T01:06:23.295 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:06:23.295 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.295 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:23.295 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:23.296 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-01-27T01:06:23.296 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlI9EbIBAACTpzAO+8A/VFJZcT/WWBng==", 2024-01-27T01:06:23.296 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:06:23.296 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T01:06:23.296 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-27T01:06:23.296 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:06:23.296 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.296 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:23.296 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-01-27T01:06:23.296 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:06:23.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.286+0000 7ff8d2bb7700 1 -- 172.21.15.46:0/3116464268 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff8b0066f30 msgr2=0x7ff8b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:23.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.286+0000 7ff8d2bb7700 1 --2- 172.21.15.46:0/3116464268 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff8b0066f30 0x7ff8b00693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8006fd0 tx=0x7ff8b8008040 comp rx=0 tx=0).stop 2024-01-27T01:06:23.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.286+0000 7ff8d2bb7700 1 -- 172.21.15.46:0/3116464268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8cc06eb70 msgr2=0x7ff8cc14e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:23.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.286+0000 7ff8d2bb7700 1 --2- 172.21.15.46:0/3116464268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8cc06eb70 0x7ff8cc14e940 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000d860 tx=0x7ff8c0008040 comp rx=0 tx=0).stop 2024-01-27T01:06:23.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.286+0000 7ff8d2bb7700 1 -- 172.21.15.46:0/3116464268 shutdown_connections 2024-01-27T01:06:23.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.286+0000 7ff8d2bb7700 1 --2- 172.21.15.46:0/3116464268 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff8b0066f30 0x7ff8b00693e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.286+0000 7ff8d2bb7700 1 --2- 172.21.15.46:0/3116464268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8cc06eb70 0x7ff8cc14e940 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.286+0000 7ff8d2bb7700 1 --2- 172.21.15.46:0/3116464268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8cc1561d0 0x7ff8cc15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.286+0000 7ff8d2bb7700 1 --2- 172.21.15.46:0/3116464268 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff8cc0b0280 0x7ff8cc15a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.286+0000 7ff8d2bb7700 1 -- 172.21.15.46:0/3116464268 >> 172.21.15.46:0/3116464268 conn(0x7ff8cc06b630 msgr2=0x7ff8cc152060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:23.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.286+0000 7ff8d2bb7700 1 -- 172.21.15.46:0/3116464268 shutdown_connections 2024-01-27T01:06:23.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.286+0000 7ff8d2bb7700 1 -- 172.21.15.46:0/3116464268 wait complete. 2024-01-27T01:06:23.299 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:06:23.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.431+0000 7f33edf11700 1 Processor -- start 2024-01-27T01:06:23.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.431+0000 7f33edf11700 1 -- start start 2024-01-27T01:06:23.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.432+0000 7f33edf11700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33e80c8dc0 0x7f33e80c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:23.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.432+0000 7f33edf11700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33e80cccd0 0x7f33e80c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:23.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.432+0000 7f33edf11700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e8068190 con 0x7f33e8068620 2024-01-27T01:06:23.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.432+0000 7f33edf11700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33e8067930 con 0x7f33e80cccd0 2024-01-27T01:06:23.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.432+0000 7f33edf11700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e80c40c0 con 0x7f33e80c8dc0 2024-01-27T01:06:23.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.432+0000 7f33e7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33e80cccd0 0x7f33e80c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:23.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.432+0000 7f33e7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33e80cccd0 0x7f33e80c7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44328/0 (socket says 172.21.15.46:44328) 2024-01-27T01:06:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.432+0000 7f33e7fff700 1 -- 172.21.15.46:0/2450621566 learned_addr learned my addr 172.21.15.46:0/2450621566 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.432+0000 7f33e77fe700 1 --2- 172.21.15.46:0/2450621566 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33e80c8dc0 0x7f33e80c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.433+0000 7f33e7fff700 1 -- 172.21.15.46:0/2450621566 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33e80c8dc0 msgr2=0x7f33e80c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.433+0000 7f33e7fff700 1 --2- 172.21.15.46:0/2450621566 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33e80c8dc0 0x7f33e80c3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.433+0000 7f33e7fff700 1 -- 172.21.15.46:0/2450621566 >> v1:172.21.15.106:6789/0 conn(0x7f33e8068620 legacy=0x7f33e80cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.433+0000 7f33e7fff700 1 -- 172.21.15.46:0/2450621566 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e8067b40 con 0x7f33e80cccd0 2024-01-27T01:06:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.433+0000 7f33e7fff700 1 --2- 172.21.15.46:0/2450621566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33e80cccd0 0x7f33e80c7ae0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f33e0009f20 tx=0x7f33e0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb07ad95de0fb294 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.433+0000 7f33e67fc700 1 -- 172.21.15.46:0/2450621566 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e000e070 con 0x7f33e80cccd0 2024-01-27T01:06:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.433+0000 7f33e67fc700 1 -- 172.21.15.46:0/2450621566 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e000ba80 con 0x7f33e80cccd0 2024-01-27T01:06:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.433+0000 7f33e67fc700 1 -- 172.21.15.46:0/2450621566 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e00092e0 con 0x7f33e80cccd0 2024-01-27T01:06:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.434+0000 7f33edf11700 1 -- 172.21.15.46:0/2450621566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33e80cccd0 msgr2=0x7f33e80c7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.434+0000 7f33edf11700 1 --2- 172.21.15.46:0/2450621566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33e80cccd0 0x7f33e80c7ae0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f33e0009f20 tx=0x7f33e0009840 comp rx=0 tx=0).stop 2024-01-27T01:06:23.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.434+0000 7f33edf11700 1 -- 172.21.15.46:0/2450621566 shutdown_connections 2024-01-27T01:06:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.434+0000 7f33edf11700 1 --2- 172.21.15.46:0/2450621566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33e80cccd0 0x7f33e80c7ae0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.434+0000 7f33edf11700 1 --2- 172.21.15.46:0/2450621566 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33e80c8dc0 0x7f33e80c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.434+0000 7f33edf11700 1 -- 172.21.15.46:0/2450621566 >> 172.21.15.46:0/2450621566 conn(0x7f33e8069220 msgr2=0x7f33e80cd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.434+0000 7f33edf11700 1 -- 172.21.15.46:0/2450621566 shutdown_connections 2024-01-27T01:06:23.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.434+0000 7f33edf11700 1 -- 172.21.15.46:0/2450621566 wait complete. 2024-01-27T01:06:23.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.435+0000 7f33edf11700 1 Processor -- start 2024-01-27T01:06:23.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.435+0000 7f33edf11700 1 -- start start 2024-01-27T01:06:23.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.436+0000 7f33edf11700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33e8068620 0x7f33e816cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:23.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.436+0000 7f33edf11700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33e80c8dc0 0x7f33e80b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.436+0000 7f33edf11700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33e80cccd0 0x7f33e80af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.436+0000 7f33edf11700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e80afcf0 con 0x7f33e8068620 2024-01-27T01:06:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.436+0000 7f33edf11700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e816de30 con 0x7f33e80c8dc0 2024-01-27T01:06:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.436+0000 7f33edf11700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e80adda0 con 0x7f33e80cccd0 2024-01-27T01:06:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.436+0000 7f33e77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33e8068620 0x7f33e816cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.436+0000 7f33e77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33e8068620 0x7f33e816cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44334/0 (socket says 172.21.15.46:44334) 2024-01-27T01:06:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.436+0000 7f33e77fe700 1 -- 172.21.15.46:0/2281061887 learned_addr learned my addr 172.21.15.46:0/2281061887 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.436+0000 7f33e6ffd700 1 --2- 172.21.15.46:0/2281061887 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33e80c8dc0 0x7f33e80b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.436+0000 7f33e7fff700 1 --2- 172.21.15.46:0/2281061887 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33e80cccd0 0x7f33e80af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.436+0000 7f33e77fe700 1 -- 172.21.15.46:0/2281061887 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33e80cccd0 msgr2=0x7f33e80af7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.436+0000 7f33e77fe700 1 --2- 172.21.15.46:0/2281061887 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33e80cccd0 0x7f33e80af7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.436+0000 7f33e77fe700 1 -- 172.21.15.46:0/2281061887 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33e80c8dc0 msgr2=0x7f33e80b0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.436+0000 7f33e77fe700 1 --2- 172.21.15.46:0/2281061887 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33e80c8dc0 0x7f33e80b0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.436+0000 7f33e77fe700 1 -- 172.21.15.46:0/2281061887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e0009ae0 con 0x7f33e8068620 2024-01-27T01:06:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.437+0000 7f33e77fe700 1 --2- 172.21.15.46:0/2281061887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33e8068620 0x7f33e816cf20 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f33d800d840 tx=0x7f33d8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.437+0000 7f33cf7fe700 1 -- 172.21.15.46:0/2281061887 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d80125f0 con 0x7f33e8068620 2024-01-27T01:06:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.437+0000 7f33cf7fe700 1 -- 172.21.15.46:0/2281061887 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d8012750 con 0x7f33e8068620 2024-01-27T01:06:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.437+0000 7f33edf11700 1 -- 172.21.15.46:0/2281061887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e816d590 con 0x7f33e8068620 2024-01-27T01:06:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.437+0000 7f33cf7fe700 1 -- 172.21.15.46:0/2281061887 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d8012af0 con 0x7f33e8068620 2024-01-27T01:06:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.437+0000 7f33edf11700 1 -- 172.21.15.46:0/2281061887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e816db50 con 0x7f33e8068620 2024-01-27T01:06:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.438+0000 7f33edf11700 1 -- 172.21.15.46:0/2281061887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33e80adb70 con 0x7f33e8068620 2024-01-27T01:06:23.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.438+0000 7f33cf7fe700 1 -- 172.21.15.46:0/2281061887 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f33d801e460 con 0x7f33e8068620 2024-01-27T01:06:23.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.439+0000 7f33cf7fe700 1 --2- 172.21.15.46:0/2281061887 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f33d4066f30 0x7f33d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:23.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.439+0000 7f33cf7fe700 1 -- 172.21.15.46:0/2281061887 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5665+0+0 (secure 0 0 0) 0x7f33d8084a10 con 0x7f33e8068620 2024-01-27T01:06:23.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.439+0000 7f33e6ffd700 1 --2- 172.21.15.46:0/2281061887 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f33d4066f30 0x7f33d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:23.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.439+0000 7f33e6ffd700 1 --2- 172.21.15.46:0/2281061887 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f33d4066f30 0x7f33d40693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f33e8068a10 tx=0x7f33dc007ec0 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.443+0000 7f33cf7fe700 1 -- 172.21.15.46:0/2281061887 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f33d8051ca0 con 0x7f33e8068620 2024-01-27T01:06:23.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.648+0000 7f33edf11700 1 -- 172.21.15.46:0/2281061887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33e804c710 con 0x7f33e8068620 2024-01-27T01:06:23.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.649+0000 7f33cf7fe700 1 -- 172.21.15.46:0/2281061887 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) v1 ==== 74+0+13105 (secure 0 0 0) 0x7f33d80552c0 con 0x7f33e8068620 2024-01-27T01:06:23.648 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:23.648 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":219,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:06:22.827858+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:06:23.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.651+0000 7f33edf11700 1 -- 172.21.15.46:0/2281061887 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f33d4066f30 msgr2=0x7f33d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.651+0000 7f33edf11700 1 --2- 172.21.15.46:0/2281061887 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f33d4066f30 0x7f33d40693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f33e8068a10 tx=0x7f33dc007ec0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.651+0000 7f33edf11700 1 -- 172.21.15.46:0/2281061887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33e8068620 msgr2=0x7f33e816cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.651+0000 7f33edf11700 1 --2- 172.21.15.46:0/2281061887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33e8068620 0x7f33e816cf20 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f33d800d840 tx=0x7f33d8008040 comp rx=0 tx=0).stop 2024-01-27T01:06:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.651+0000 7f33edf11700 1 -- 172.21.15.46:0/2281061887 shutdown_connections 2024-01-27T01:06:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.651+0000 7f33edf11700 1 --2- 172.21.15.46:0/2281061887 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f33d4066f30 0x7f33d40693e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.651+0000 7f33edf11700 1 --2- 172.21.15.46:0/2281061887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33e8068620 0x7f33e816cf20 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.651+0000 7f33edf11700 1 --2- 172.21.15.46:0/2281061887 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33e80cccd0 0x7f33e80af7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.651+0000 7f33edf11700 1 --2- 172.21.15.46:0/2281061887 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33e80c8dc0 0x7f33e80b0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.651+0000 7f33edf11700 1 -- 172.21.15.46:0/2281061887 >> 172.21.15.46:0/2281061887 conn(0x7f33e8069220 msgr2=0x7f33e80ac060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.651+0000 7f33edf11700 1 -- 172.21.15.46:0/2281061887 shutdown_connections 2024-01-27T01:06:23.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.651+0000 7f33edf11700 1 -- 172.21.15.46:0/2281061887 wait complete. 2024-01-27T01:06:23.664 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-27T01:06:23.664 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-27T01:06:23.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.801+0000 7f40716a3700 1 Processor -- start 2024-01-27T01:06:23.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.801+0000 7f40716a3700 1 -- start start 2024-01-27T01:06:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.802+0000 7f40716a3700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f406c159da0 0x7f406c0ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.802+0000 7f40716a3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f406c15d8c0 0x7f406c15dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.802+0000 7f40716a3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406c15a200 con 0x7f406c1599d0 2024-01-27T01:06:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.802+0000 7f40716a3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f406c15e260 con 0x7f406c15d8c0 2024-01-27T01:06:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.802+0000 7f40716a3700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f406c0b1a60 con 0x7f406c159da0 2024-01-27T01:06:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.802+0000 7f406b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f406c15d8c0 0x7f406c15dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.802+0000 7f406b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f406c15d8c0 0x7f406c15dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44346/0 (socket says 172.21.15.46:44346) 2024-01-27T01:06:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.802+0000 7f406b7fe700 1 -- 172.21.15.46:0/811337181 learned_addr learned my addr 172.21.15.46:0/811337181 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.802+0000 7f406a7fc700 1 --2- 172.21.15.46:0/811337181 >> v2:172.21.15.106:3300/0 conn(0x7f406c159da0 0x7f406c0ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.802+0000 7f406b7fe700 1 -- 172.21.15.46:0/811337181 >> v2:172.21.15.106:3300/0 conn(0x7f406c159da0 msgr2=0x7f406c0ab1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.802+0000 7f406b7fe700 1 --2- 172.21.15.46:0/811337181 >> v2:172.21.15.106:3300/0 conn(0x7f406c159da0 0x7f406c0ab1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.802+0000 7f406b7fe700 1 -- 172.21.15.46:0/811337181 >> v1:172.21.15.106:6789/0 conn(0x7f406c1599d0 legacy=0x7f406c0b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.802+0000 7f406b7fe700 1 -- 172.21.15.46:0/811337181 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c0b11c0 con 0x7f406c15d8c0 2024-01-27T01:06:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.803+0000 7f406b7fe700 1 --2- 172.21.15.46:0/811337181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c15d8c0 0x7f406c15dd20 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f406c15d7f0 tx=0x7f4064009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce1c59921648067 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.803+0000 7f4069ffb700 1 -- 172.21.15.46:0/811337181 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406400e070 con 0x7f406c15d8c0 2024-01-27T01:06:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.803+0000 7f4069ffb700 1 -- 172.21.15.46:0/811337181 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406400ba20 con 0x7f406c15d8c0 2024-01-27T01:06:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.803+0000 7f4069ffb700 1 -- 172.21.15.46:0/811337181 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40640092e0 con 0x7f406c15d8c0 2024-01-27T01:06:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.803+0000 7f40716a3700 1 -- 172.21.15.46:0/811337181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c15d8c0 msgr2=0x7f406c15dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.803+0000 7f40716a3700 1 --2- 172.21.15.46:0/811337181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c15d8c0 0x7f406c15dd20 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f406c15d7f0 tx=0x7f4064009840 comp rx=0 tx=0).stop 2024-01-27T01:06:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.803+0000 7f40716a3700 1 -- 172.21.15.46:0/811337181 shutdown_connections 2024-01-27T01:06:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.803+0000 7f40716a3700 1 --2- 172.21.15.46:0/811337181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c15d8c0 0x7f406c15dd20 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.803+0000 7f40716a3700 1 --2- 172.21.15.46:0/811337181 >> v2:172.21.15.106:3300/0 conn(0x7f406c159da0 0x7f406c0ab1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.803+0000 7f40716a3700 1 -- 172.21.15.46:0/811337181 >> 172.21.15.46:0/811337181 conn(0x7f406c06f840 msgr2=0x7f406c087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.803+0000 7f40716a3700 1 -- 172.21.15.46:0/811337181 shutdown_connections 2024-01-27T01:06:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.804+0000 7f40716a3700 1 -- 172.21.15.46:0/811337181 wait complete. 2024-01-27T01:06:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.805+0000 7f40716a3700 1 Processor -- start 2024-01-27T01:06:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.805+0000 7f40716a3700 1 -- start start 2024-01-27T01:06:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f40716a3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c1599d0 0x7f406c156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f40716a3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f406c159da0 0x7f406c09d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f40716a3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f406c15d8c0 0x7f406c09b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f40716a3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c0cdf10 con 0x7f406c1599d0 2024-01-27T01:06:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f40716a3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c156560 con 0x7f406c159da0 2024-01-27T01:06:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f40716a3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406c16ac70 con 0x7f406c15d8c0 2024-01-27T01:06:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f406affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c1599d0 0x7f406c156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f406affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c1599d0 0x7f406c156020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44348/0 (socket says 172.21.15.46:44348) 2024-01-27T01:06:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f406affd700 1 -- 172.21.15.46:0/443667616 learned_addr learned my addr 172.21.15.46:0/443667616 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f406b7fe700 1 --2- 172.21.15.46:0/443667616 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f406c15d8c0 0x7f406c09b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f406a7fc700 1 --2- 172.21.15.46:0/443667616 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f406c159da0 0x7f406c09d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f406affd700 1 -- 172.21.15.46:0/443667616 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f406c15d8c0 msgr2=0x7f406c09b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f406affd700 1 --2- 172.21.15.46:0/443667616 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f406c15d8c0 0x7f406c09b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f406affd700 1 -- 172.21.15.46:0/443667616 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f406c159da0 msgr2=0x7f406c09d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f406affd700 1 --2- 172.21.15.46:0/443667616 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f406c159da0 0x7f406c09d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f406affd700 1 -- 172.21.15.46:0/443667616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4064009ae0 con 0x7f406c1599d0 2024-01-27T01:06:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f406a7fc700 1 --2- 172.21.15.46:0/443667616 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f406c159da0 0x7f406c09d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f406affd700 1 --2- 172.21.15.46:0/443667616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c1599d0 0x7f406c156020 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f405c00c7c0 tx=0x7f405c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.806+0000 7f4052ffd700 1 -- 172.21.15.46:0/443667616 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c011590 con 0x7f406c1599d0 2024-01-27T01:06:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.807+0000 7f4052ffd700 1 -- 172.21.15.46:0/443667616 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405c011710 con 0x7f406c1599d0 2024-01-27T01:06:23.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.807+0000 7f4052ffd700 1 -- 172.21.15.46:0/443667616 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c011ad0 con 0x7f406c1599d0 2024-01-27T01:06:23.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.807+0000 7f40716a3700 1 -- 172.21.15.46:0/443667616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f406c06a900 con 0x7f406c1599d0 2024-01-27T01:06:23.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.807+0000 7f40716a3700 1 -- 172.21.15.46:0/443667616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f406c16a620 con 0x7f406c1599d0 2024-01-27T01:06:23.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.808+0000 7f40716a3700 1 -- 172.21.15.46:0/443667616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406c04c710 con 0x7f406c1599d0 2024-01-27T01:06:23.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.808+0000 7f4052ffd700 1 -- 172.21.15.46:0/443667616 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f405c01a360 con 0x7f406c1599d0 2024-01-27T01:06:23.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.808+0000 7f4052ffd700 1 --2- 172.21.15.46:0/443667616 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4058066f30 0x7f40580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:23.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.809+0000 7f4052ffd700 1 -- 172.21.15.46:0/443667616 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5665+0+0 (secure 0 0 0) 0x7f405c0833f0 con 0x7f406c1599d0 2024-01-27T01:06:23.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.809+0000 7f406a7fc700 1 --2- 172.21.15.46:0/443667616 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4058066f30 0x7f40580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:23.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.809+0000 7f406a7fc700 1 --2- 172.21.15.46:0/443667616 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4058066f30 0x7f40580693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f406c0cfe60 tx=0x7f4060008040 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:23.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:23.813+0000 7f4052ffd700 1 -- 172.21.15.46:0/443667616 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f405c050730 con 0x7f406c1599d0 2024-01-27T01:06:24.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.020+0000 7f40716a3700 1 -- 172.21.15.46:0/443667616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f406c0b14d0 con 0x7f406c1599d0 2024-01-27T01:06:24.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.021+0000 7f4052ffd700 1 -- 172.21.15.46:0/443667616 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v219) v1 ==== 93+0+10 (secure 0 0 0) 0x7f405c053d50 con 0x7f406c1599d0 2024-01-27T01:06:24.020 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-27T01:06:24.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.023+0000 7f40716a3700 1 -- 172.21.15.46:0/443667616 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4058066f30 msgr2=0x7f40580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:24.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.023+0000 7f40716a3700 1 --2- 172.21.15.46:0/443667616 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4058066f30 0x7f40580693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f406c0cfe60 tx=0x7f4060008040 comp rx=0 tx=0).stop 2024-01-27T01:06:24.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.023+0000 7f40716a3700 1 -- 172.21.15.46:0/443667616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c1599d0 msgr2=0x7f406c156020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:24.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.023+0000 7f40716a3700 1 --2- 172.21.15.46:0/443667616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c1599d0 0x7f406c156020 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f405c00c7c0 tx=0x7f405c007f70 comp rx=0 tx=0).stop 2024-01-27T01:06:24.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.023+0000 7f40716a3700 1 -- 172.21.15.46:0/443667616 shutdown_connections 2024-01-27T01:06:24.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.023+0000 7f40716a3700 1 --2- 172.21.15.46:0/443667616 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4058066f30 0x7f40580693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.023+0000 7f40716a3700 1 --2- 172.21.15.46:0/443667616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c1599d0 0x7f406c156020 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.023+0000 7f40716a3700 1 --2- 172.21.15.46:0/443667616 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f406c15d8c0 0x7f406c09b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.023+0000 7f40716a3700 1 --2- 172.21.15.46:0/443667616 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f406c159da0 0x7f406c09d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.023+0000 7f40716a3700 1 -- 172.21.15.46:0/443667616 >> 172.21.15.46:0/443667616 conn(0x7f406c06f840 msgr2=0x7f406c0b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:24.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.023+0000 7f40716a3700 1 -- 172.21.15.46:0/443667616 shutdown_connections 2024-01-27T01:06:24.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.023+0000 7f40716a3700 1 -- 172.21.15.46:0/443667616 wait complete. 2024-01-27T01:06:24.036 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-27T01:06:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.169+0000 7f290920a700 1 Processor -- start 2024-01-27T01:06:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.169+0000 7f290920a700 1 -- start start 2024-01-27T01:06:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.170+0000 7f290920a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f29040c3cf0 0x7f29040ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.170+0000 7f290920a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29040cbf10 con 0x7f29040cd080 2024-01-27T01:06:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.170+0000 7f290920a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29040cd4a0 con 0x7f29040acf30 2024-01-27T01:06:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.170+0000 7f290920a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29040ada10 con 0x7f29040c3cf0 2024-01-27T01:06:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.170+0000 7f290359e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f29040cd080 0x7f29040ab1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53608/0 (socket says 172.21.15.46:53608) 2024-01-27T01:06:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.170+0000 7f290359e700 1 -- 172.21.15.46:0/4165638327 learned_addr learned my addr 172.21.15.46:0/4165638327 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.170+0000 7f290259c700 1 --2- 172.21.15.46:0/4165638327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f29040c3cf0 0x7f29040ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.171+0000 7f2901d9b700 1 -- 172.21.15.46:0/4165638327 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269056237 0 0) 0x7f29040cbf10 con 0x7f29040cd080 2024-01-27T01:06:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.171+0000 7f2901d9b700 1 -- 172.21.15.46:0/4165638327 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28f0003570 con 0x7f29040cd080 2024-01-27T01:06:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.171+0000 7f2901d9b700 1 -- 172.21.15.46:0/4165638327 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740915803 0 0) 0x7f29040cd4a0 con 0x7f29040acf30 2024-01-27T01:06:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.171+0000 7f2901d9b700 1 -- 172.21.15.46:0/4165638327 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29040cbf10 con 0x7f29040acf30 2024-01-27T01:06:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.171+0000 7f290259c700 1 -- 172.21.15.46:0/4165638327 >> v1:172.21.15.106:6789/0 conn(0x7f29040acf30 legacy=0x7f29040ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.171+0000 7f290259c700 1 -- 172.21.15.46:0/4165638327 >> v1:172.21.15.46:6789/0 conn(0x7f29040cd080 legacy=0x7f29040ab1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.171+0000 7f290259c700 1 -- 172.21.15.46:0/4165638327 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29040c4460 con 0x7f29040c3cf0 2024-01-27T01:06:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.171+0000 7f290259c700 1 --2- 172.21.15.46:0/4165638327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f29040c3cf0 0x7f29040ccb40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009910 tx=0x7f28ec0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3960ff3638a36769 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.171+0000 7f2901d9b700 1 -- 172.21.15.46:0/4165638327 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ec011630 con 0x7f29040c3cf0 2024-01-27T01:06:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.171+0000 7f2901d9b700 1 -- 172.21.15.46:0/4165638327 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ec011790 con 0x7f29040c3cf0 2024-01-27T01:06:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.171+0000 7f2901d9b700 1 -- 172.21.15.46:0/4165638327 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ec011b30 con 0x7f29040c3cf0 2024-01-27T01:06:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.172+0000 7f290920a700 1 -- 172.21.15.46:0/4165638327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f29040c3cf0 msgr2=0x7f29040ccb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.172+0000 7f290920a700 1 --2- 172.21.15.46:0/4165638327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f29040c3cf0 0x7f29040ccb40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009910 tx=0x7f28ec0091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.172+0000 7f290920a700 1 -- 172.21.15.46:0/4165638327 shutdown_connections 2024-01-27T01:06:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.172+0000 7f290920a700 1 --2- 172.21.15.46:0/4165638327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f29040c3cf0 0x7f29040ccb40 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.172+0000 7f290920a700 1 -- 172.21.15.46:0/4165638327 >> 172.21.15.46:0/4165638327 conn(0x7f2904097820 msgr2=0x7f2904068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.172+0000 7f290920a700 1 -- 172.21.15.46:0/4165638327 shutdown_connections 2024-01-27T01:06:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.172+0000 7f290920a700 1 -- 172.21.15.46:0/4165638327 wait complete. 2024-01-27T01:06:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.174+0000 7f290920a700 1 Processor -- start 2024-01-27T01:06:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.174+0000 7f290920a700 1 -- start start 2024-01-27T01:06:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f290920a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29040acf30 0x7f29041538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f290920a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f29040c3cf0 0x7f29040adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f290920a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29040cd080 0x7f29040b0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f290920a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2904153de0 con 0x7f29040acf30 2024-01-27T01:06:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f290920a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29040caa10 con 0x7f29040cd080 2024-01-27T01:06:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f290920a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29040c9e00 con 0x7f29040c3cf0 2024-01-27T01:06:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f2902d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29040acf30 0x7f29041538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f2902d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29040acf30 0x7f29041538a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44360/0 (socket says 172.21.15.46:44360) 2024-01-27T01:06:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f2902d9d700 1 -- 172.21.15.46:0/2449921412 learned_addr learned my addr 172.21.15.46:0/2449921412 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f290259c700 1 --2- 172.21.15.46:0/2449921412 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f29040c3cf0 0x7f29040adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f290359e700 1 --2- 172.21.15.46:0/2449921412 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29040cd080 0x7f29040b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f2902d9d700 1 -- 172.21.15.46:0/2449921412 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f29040c3cf0 msgr2=0x7f29040adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f2902d9d700 1 --2- 172.21.15.46:0/2449921412 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f29040c3cf0 0x7f29040adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f2902d9d700 1 -- 172.21.15.46:0/2449921412 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29040cd080 msgr2=0x7f29040b0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f2902d9d700 1 --2- 172.21.15.46:0/2449921412 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29040cd080 0x7f29040b0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f2902d9d700 1 -- 172.21.15.46:0/2449921412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ec009420 con 0x7f29040acf30 2024-01-27T01:06:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f2902d9d700 1 --2- 172.21.15.46:0/2449921412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29040acf30 0x7f29041538a0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f28f400c590 tx=0x7f28f4007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f28faffd700 1 -- 172.21.15.46:0/2449921412 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4010510 con 0x7f29040acf30 2024-01-27T01:06:24.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.175+0000 7f28faffd700 1 -- 172.21.15.46:0/2449921412 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f4010690 con 0x7f29040acf30 2024-01-27T01:06:24.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.176+0000 7f290920a700 1 -- 172.21.15.46:0/2449921412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29040ca0e0 con 0x7f29040acf30 2024-01-27T01:06:24.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.176+0000 7f28faffd700 1 -- 172.21.15.46:0/2449921412 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4010a50 con 0x7f29040acf30 2024-01-27T01:06:24.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.176+0000 7f290920a700 1 -- 172.21.15.46:0/2449921412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29040ca6a0 con 0x7f29040acf30 2024-01-27T01:06:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.177+0000 7f290920a700 1 -- 172.21.15.46:0/2449921412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f290404c710 con 0x7f29040acf30 2024-01-27T01:06:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.177+0000 7f28faffd700 1 -- 172.21.15.46:0/2449921412 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f28f401b400 con 0x7f29040acf30 2024-01-27T01:06:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.177+0000 7f28faffd700 1 --2- 172.21.15.46:0/2449921412 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f28e4066f30 0x7f28e40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.177+0000 7f28faffd700 1 -- 172.21.15.46:0/2449921412 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5665+0+0 (secure 0 0 0) 0x7f28f4084220 con 0x7f29040acf30 2024-01-27T01:06:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.178+0000 7f290259c700 1 --2- 172.21.15.46:0/2449921412 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f28e4066f30 0x7f28e40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.178+0000 7f290259c700 1 --2- 172.21.15.46:0/2449921412 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f28e4066f30 0x7f28e40693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009860 tx=0x7f28ec017000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:24.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.182+0000 7f28faffd700 1 -- 172.21.15.46:0/2449921412 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f28f40514b0 con 0x7f29040acf30 2024-01-27T01:06:24.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.391+0000 7f290920a700 1 -- 172.21.15.46:0/2449921412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f29040b0eb0 con 0x7f29040acf30 2024-01-27T01:06:24.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.391+0000 7f28faffd700 1 -- 172.21.15.46:0/2449921412 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v219) v1 ==== 84+0+65 (secure 0 0 0) 0x7f28f4054ad0 con 0x7f29040acf30 2024-01-27T01:06:24.391 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:24.391 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-27T01:06:24.391 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:24.391 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-01-27T01:06:24.391 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-01-27T01:06:24.391 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:24.391 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-27T01:06:24.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.394+0000 7f290920a700 1 -- 172.21.15.46:0/2449921412 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f28e4066f30 msgr2=0x7f28e40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:24.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.394+0000 7f290920a700 1 --2- 172.21.15.46:0/2449921412 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f28e4066f30 0x7f28e40693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009860 tx=0x7f28ec017000 comp rx=0 tx=0).stop 2024-01-27T01:06:24.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.394+0000 7f290920a700 1 -- 172.21.15.46:0/2449921412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29040acf30 msgr2=0x7f29041538a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:24.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.394+0000 7f290920a700 1 --2- 172.21.15.46:0/2449921412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29040acf30 0x7f29041538a0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f28f400c590 tx=0x7f28f4007dd0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.394+0000 7f290920a700 1 -- 172.21.15.46:0/2449921412 shutdown_connections 2024-01-27T01:06:24.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.394+0000 7f290920a700 1 --2- 172.21.15.46:0/2449921412 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f28e4066f30 0x7f28e40693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.394+0000 7f290920a700 1 --2- 172.21.15.46:0/2449921412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29040acf30 0x7f29041538a0 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.394+0000 7f290920a700 1 --2- 172.21.15.46:0/2449921412 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29040cd080 0x7f29040b0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.394+0000 7f290920a700 1 --2- 172.21.15.46:0/2449921412 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f29040c3cf0 0x7f29040adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.394+0000 7f290920a700 1 -- 172.21.15.46:0/2449921412 >> 172.21.15.46:0/2449921412 conn(0x7f2904097820 msgr2=0x7f29040cc330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.394+0000 7f290920a700 1 -- 172.21.15.46:0/2449921412 shutdown_connections 2024-01-27T01:06:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.394+0000 7f290920a700 1 -- 172.21.15.46:0/2449921412 wait complete. 2024-01-27T01:06:24.407 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-27T01:06:24.407 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-27T01:06:24.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.544+0000 7f71bb0cb700 1 Processor -- start 2024-01-27T01:06:24.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.544+0000 7f71bb0cb700 1 -- start start 2024-01-27T01:06:24.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.545+0000 7f71bb0cb700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f71b4159750 0x7f71b4069650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:24.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.545+0000 7f71bb0cb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71b4159b20 0x7f71b40ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:24.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.545+0000 7f71bb0cb700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b40691c0 con 0x7f71b4069b90 2024-01-27T01:06:24.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.545+0000 7f71bb0cb700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71b406a5e0 con 0x7f71b4159750 2024-01-27T01:06:24.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.545+0000 7f71bb0cb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b415dfd0 con 0x7f71b4159b20 2024-01-27T01:06:24.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.545+0000 7f71b8e67700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f71b4159750 0x7f71b4069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:24.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.545+0000 7f71b3fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71b4159b20 0x7f71b40ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:24.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.545+0000 7f71b9668700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f71b4069b90 0x7f71b415d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:49848/0 (socket says 172.21.15.46:49848) 2024-01-27T01:06:24.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.545+0000 7f71b9668700 1 -- 172.21.15.46:0/3396749192 learned_addr learned my addr 172.21.15.46:0/3396749192 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:24.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.545+0000 7f71b37fe700 1 -- 172.21.15.46:0/3396749192 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206129844 0 0) 0x7f71b40691c0 con 0x7f71b4069b90 2024-01-27T01:06:24.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.545+0000 7f71b37fe700 1 -- 172.21.15.46:0/3396749192 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71a4003570 con 0x7f71b4069b90 2024-01-27T01:06:24.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.546+0000 7f71b8e67700 1 -- 172.21.15.46:0/3396749192 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71b4159b20 msgr2=0x7f71b40ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:24.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.546+0000 7f71b8e67700 1 --2- 172.21.15.46:0/3396749192 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71b4159b20 0x7f71b40ade00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.546+0000 7f71b8e67700 1 -- 172.21.15.46:0/3396749192 >> v1:172.21.15.106:6789/0 conn(0x7f71b4069b90 legacy=0x7f71b415d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:24.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.546+0000 7f71b8e67700 1 -- 172.21.15.46:0/3396749192 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b4068b70 con 0x7f71b4159750 2024-01-27T01:06:24.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.546+0000 7f71b3fff700 1 --2- 172.21.15.46:0/3396749192 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71b4159b20 0x7f71b40ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:24.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.546+0000 7f71b8e67700 1 --2- 172.21.15.46:0/3396749192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71b4159750 0x7f71b4069650 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f71a80098e0 tx=0x7f71a80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85b926d566668d47 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:24.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.546+0000 7f71b37fe700 1 -- 172.21.15.46:0/3396749192 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a8011630 con 0x7f71b4159750 2024-01-27T01:06:24.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.546+0000 7f71b37fe700 1 -- 172.21.15.46:0/3396749192 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a8011790 con 0x7f71b4159750 2024-01-27T01:06:24.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.546+0000 7f71b37fe700 1 -- 172.21.15.46:0/3396749192 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a8011b30 con 0x7f71b4159750 2024-01-27T01:06:24.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.547+0000 7f71bb0cb700 1 -- 172.21.15.46:0/3396749192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71b4159750 msgr2=0x7f71b4069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:24.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.547+0000 7f71bb0cb700 1 --2- 172.21.15.46:0/3396749192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71b4159750 0x7f71b4069650 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f71a80098e0 tx=0x7f71a80091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.547+0000 7f71bb0cb700 1 -- 172.21.15.46:0/3396749192 shutdown_connections 2024-01-27T01:06:24.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.547+0000 7f71bb0cb700 1 --2- 172.21.15.46:0/3396749192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71b4159750 0x7f71b4069650 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.547+0000 7f71bb0cb700 1 --2- 172.21.15.46:0/3396749192 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71b4159b20 0x7f71b40ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.547+0000 7f71bb0cb700 1 -- 172.21.15.46:0/3396749192 >> 172.21.15.46:0/3396749192 conn(0x7f71b40ce530 msgr2=0x7f71b40ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:24.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.547+0000 7f71bb0cb700 1 -- 172.21.15.46:0/3396749192 shutdown_connections 2024-01-27T01:06:24.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.547+0000 7f71bb0cb700 1 -- 172.21.15.46:0/3396749192 wait complete. 2024-01-27T01:06:24.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.548+0000 7f71bb0cb700 1 Processor -- start 2024-01-27T01:06:24.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.548+0000 7f71bb0cb700 1 -- start start 2024-01-27T01:06:24.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.549+0000 7f71bb0cb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71b4069b90 0x7f71b40c8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:24.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.549+0000 7f71bb0cb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b4159750 0x7f71b41569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:24.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.549+0000 7f71bb0cb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71b4159b20 0x7f71b40aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:24.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.549+0000 7f71bb0cb700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b40b01c0 con 0x7f71b4159750 2024-01-27T01:06:24.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.549+0000 7f71bb0cb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b4156f10 con 0x7f71b4069b90 2024-01-27T01:06:24.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.549+0000 7f71bb0cb700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b40af310 con 0x7f71b4159b20 2024-01-27T01:06:24.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.549+0000 7f71b3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b4159750 0x7f71b41569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:24.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.549+0000 7f71b3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b4159750 0x7f71b41569d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44366/0 (socket says 172.21.15.46:44366) 2024-01-27T01:06:24.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.549+0000 7f71b3fff700 1 -- 172.21.15.46:0/2321055228 learned_addr learned my addr 172.21.15.46:0/2321055228 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:24.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.549+0000 7f71b8e67700 1 --2- 172.21.15.46:0/2321055228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71b4069b90 0x7f71b40c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:24.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.549+0000 7f71b9668700 1 --2- 172.21.15.46:0/2321055228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71b4159b20 0x7f71b40aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:24.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.549+0000 7f71b3fff700 1 -- 172.21.15.46:0/2321055228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71b4159b20 msgr2=0x7f71b40aedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:24.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.549+0000 7f71b3fff700 1 --2- 172.21.15.46:0/2321055228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71b4159b20 0x7f71b40aedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.549+0000 7f71b3fff700 1 -- 172.21.15.46:0/2321055228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71b4069b90 msgr2=0x7f71b40c8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:24.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.549+0000 7f71b3fff700 1 --2- 172.21.15.46:0/2321055228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71b4069b90 0x7f71b40c8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.549+0000 7f71b3fff700 1 -- 172.21.15.46:0/2321055228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a0009420 con 0x7f71b4159750 2024-01-27T01:06:24.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.550+0000 7f71b3fff700 1 --2- 172.21.15.46:0/2321055228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b4159750 0x7f71b41569d0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f71a000ec80 tx=0x7f71a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:24.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.550+0000 7f71b0ff9700 1 -- 172.21.15.46:0/2321055228 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a000c9d0 con 0x7f71b4159750 2024-01-27T01:06:24.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.550+0000 7f71b0ff9700 1 -- 172.21.15.46:0/2321055228 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a000cb30 con 0x7f71b4159750 2024-01-27T01:06:24.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.550+0000 7f71bb0cb700 1 -- 172.21.15.46:0/2321055228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a8009420 con 0x7f71b4159750 2024-01-27T01:06:24.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.550+0000 7f71b0ff9700 1 -- 172.21.15.46:0/2321055228 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a000ceb0 con 0x7f71b4159750 2024-01-27T01:06:24.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.550+0000 7f71bb0cb700 1 -- 172.21.15.46:0/2321055228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b40af890 con 0x7f71b4159750 2024-01-27T01:06:24.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.551+0000 7f71bb0cb700 1 -- 172.21.15.46:0/2321055228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b40b0330 con 0x7f71b4159750 2024-01-27T01:06:24.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.551+0000 7f71b0ff9700 1 -- 172.21.15.46:0/2321055228 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f71a0013680 con 0x7f71b4159750 2024-01-27T01:06:24.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.552+0000 7f71b0ff9700 1 --2- 172.21.15.46:0/2321055228 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7198066f30 0x7f71980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:24.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.552+0000 7f71b0ff9700 1 -- 172.21.15.46:0/2321055228 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5665+0+0 (secure 0 0 0) 0x7f71a00849c0 con 0x7f71b4159750 2024-01-27T01:06:24.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.552+0000 7f71b8e67700 1 --2- 172.21.15.46:0/2321055228 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7198066f30 0x7f71980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:24.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.553+0000 7f71b8e67700 1 --2- 172.21.15.46:0/2321055228 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7198066f30 0x7f71980693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f71a80091e0 tx=0x7f71a8017000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:24.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.556+0000 7f71b0ff9700 1 -- 172.21.15.46:0/2321055228 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f71a0051c50 con 0x7f71b4159750 2024-01-27T01:06:24.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.762+0000 7f71bb0cb700 1 -- 172.21.15.46:0/2321055228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f71b404c710 con 0x7f71b4159750 2024-01-27T01:06:24.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.837+0000 7f71b0ff9700 1 -- 172.21.15.46:0/2321055228 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v220) v1 ==== 138+0+0 (secure 0 0 0) 0x7f71a00170b0 con 0x7f71b4159750 2024-01-27T01:06:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.840+0000 7f71bb0cb700 1 -- 172.21.15.46:0/2321055228 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7198066f30 msgr2=0x7f71980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.840+0000 7f71bb0cb700 1 --2- 172.21.15.46:0/2321055228 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7198066f30 0x7f71980693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f71a80091e0 tx=0x7f71a8017000 comp rx=0 tx=0).stop 2024-01-27T01:06:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.840+0000 7f71bb0cb700 1 -- 172.21.15.46:0/2321055228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b4159750 msgr2=0x7f71b41569d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.840+0000 7f71bb0cb700 1 --2- 172.21.15.46:0/2321055228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b4159750 0x7f71b41569d0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f71a000ec80 tx=0x7f71a00091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.840+0000 7f71bb0cb700 1 -- 172.21.15.46:0/2321055228 shutdown_connections 2024-01-27T01:06:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.840+0000 7f71bb0cb700 1 --2- 172.21.15.46:0/2321055228 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7198066f30 0x7f71980693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.840+0000 7f71bb0cb700 1 --2- 172.21.15.46:0/2321055228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71b4159b20 0x7f71b40aedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.840+0000 7f71bb0cb700 1 --2- 172.21.15.46:0/2321055228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71b4069b90 0x7f71b40c8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.840+0000 7f71bb0cb700 1 --2- 172.21.15.46:0/2321055228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b4159750 0x7f71b41569d0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.840+0000 7f71bb0cb700 1 -- 172.21.15.46:0/2321055228 >> 172.21.15.46:0/2321055228 conn(0x7f71b40ce530 msgr2=0x7f71b4157b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.840+0000 7f71bb0cb700 1 -- 172.21.15.46:0/2321055228 shutdown_connections 2024-01-27T01:06:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.840+0000 7f71bb0cb700 1 -- 172.21.15.46:0/2321055228 wait complete. 2024-01-27T01:06:24.841 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-01-27T01:06:24.856 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-27T01:06:24.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.996+0000 7f8a09207700 1 Processor -- start 2024-01-27T01:06:24.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.996+0000 7f8a09207700 1 -- start start 2024-01-27T01:06:24.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.997+0000 7f8a09207700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8a040c93a0 0x7f8a040c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:24.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.997+0000 7f8a09207700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8a04067d50 0x7f8a04159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:24.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.997+0000 7f8a09207700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0415e390 con 0x7f8a040681f0 2024-01-27T01:06:24.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.997+0000 7f8a09207700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a0415d810 con 0x7f8a040c93a0 2024-01-27T01:06:24.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.997+0000 7f8a09207700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a0415daf0 con 0x7f8a04067d50 2024-01-27T01:06:24.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.997+0000 7f8a02d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8a040c93a0 0x7f8a040c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:24.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.997+0000 7f8a02d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8a040c93a0 0x7f8a040c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44378/0 (socket says 172.21.15.46:44378) 2024-01-27T01:06:24.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.997+0000 7f8a02d9d700 1 -- 172.21.15.46:0/3234197883 learned_addr learned my addr 172.21.15.46:0/3234197883 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:24.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.998+0000 7f8a0259c700 1 --2- 172.21.15.46:0/3234197883 >> v2:172.21.15.106:3300/0 conn(0x7f8a04067d50 0x7f8a04159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:24.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.998+0000 7f8a02d9d700 1 -- 172.21.15.46:0/3234197883 >> v2:172.21.15.106:3300/0 conn(0x7f8a04067d50 msgr2=0x7f8a04159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:24.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.998+0000 7f8a02d9d700 1 --2- 172.21.15.46:0/3234197883 >> v2:172.21.15.106:3300/0 conn(0x7f8a04067d50 0x7f8a04159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:24.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.998+0000 7f8a02d9d700 1 -- 172.21.15.46:0/3234197883 >> v1:172.21.15.106:6789/0 conn(0x7f8a040681f0 legacy=0x7f8a04159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:24.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.998+0000 7f8a02d9d700 1 -- 172.21.15.46:0/3234197883 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0415ddd0 con 0x7f8a040c93a0 2024-01-27T01:06:24.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.998+0000 7f8a02d9d700 1 --2- 172.21.15.46:0/3234197883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a040c93a0 0x7f8a040c9770 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f8a04064390 tx=0x7f89f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad628e818daa024 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:24.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.998+0000 7f8a01d9b700 1 -- 172.21.15.46:0/3234197883 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f4009e70 con 0x7f8a040c93a0 2024-01-27T01:06:25.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.998+0000 7f8a01d9b700 1 -- 172.21.15.46:0/3234197883 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f400b860 con 0x7f8a040c93a0 2024-01-27T01:06:25.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.998+0000 7f8a01d9b700 1 -- 172.21.15.46:0/3234197883 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f400bc20 con 0x7f8a040c93a0 2024-01-27T01:06:25.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.999+0000 7f8a09207700 1 -- 172.21.15.46:0/3234197883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a040c93a0 msgr2=0x7f8a040c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:25.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.999+0000 7f8a09207700 1 --2- 172.21.15.46:0/3234197883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a040c93a0 0x7f8a040c9770 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f8a04064390 tx=0x7f89f4009250 comp rx=0 tx=0).stop 2024-01-27T01:06:25.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.999+0000 7f8a09207700 1 -- 172.21.15.46:0/3234197883 shutdown_connections 2024-01-27T01:06:25.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.999+0000 7f8a09207700 1 --2- 172.21.15.46:0/3234197883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a040c93a0 0x7f8a040c9770 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:25.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.999+0000 7f8a09207700 1 --2- 172.21.15.46:0/3234197883 >> v2:172.21.15.106:3300/0 conn(0x7f8a04067d50 0x7f8a04159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:25.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.999+0000 7f8a09207700 1 -- 172.21.15.46:0/3234197883 >> 172.21.15.46:0/3234197883 conn(0x7f8a040ca9a0 msgr2=0x7f8a04068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:25.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.999+0000 7f8a09207700 1 -- 172.21.15.46:0/3234197883 shutdown_connections 2024-01-27T01:06:25.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:24.999+0000 7f8a09207700 1 -- 172.21.15.46:0/3234197883 wait complete. 2024-01-27T01:06:25.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.000+0000 7f8a09207700 1 Processor -- start 2024-01-27T01:06:25.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.000+0000 7f8a09207700 1 -- start start 2024-01-27T01:06:25.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.001+0000 7f8a09207700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a04067d50 0x7f8a040b0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:25.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.001+0000 7f8a09207700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a040681f0 0x7f8a04159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:25.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.001+0000 7f8a09207700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a040c93a0 0x7f8a04149640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.001+0000 7f8a09207700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a040b0740 con 0x7f8a040c93a0 2024-01-27T01:06:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.001+0000 7f8a09207700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a040b22f0 con 0x7f8a04067d50 2024-01-27T01:06:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.001+0000 7f8a09207700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a040b1ce0 con 0x7f8a040681f0 2024-01-27T01:06:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.001+0000 7f8a0359e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a040c93a0 0x7f8a04149640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.001+0000 7f8a0359e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a040c93a0 0x7f8a04149640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44388/0 (socket says 172.21.15.46:44388) 2024-01-27T01:06:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.001+0000 7f8a0359e700 1 -- 172.21.15.46:0/1917869207 learned_addr learned my addr 172.21.15.46:0/1917869207 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.001+0000 7f8a0259c700 1 --2- 172.21.15.46:0/1917869207 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a040681f0 0x7f8a04159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.001+0000 7f8a02d9d700 1 --2- 172.21.15.46:0/1917869207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a04067d50 0x7f8a040b0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.001+0000 7f8a0359e700 1 -- 172.21.15.46:0/1917869207 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a040681f0 msgr2=0x7f8a04159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.001+0000 7f8a0359e700 1 --2- 172.21.15.46:0/1917869207 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a040681f0 0x7f8a04159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.001+0000 7f8a0359e700 1 -- 172.21.15.46:0/1917869207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a04067d50 msgr2=0x7f8a040b0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.001+0000 7f8a0359e700 1 --2- 172.21.15.46:0/1917869207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a04067d50 0x7f8a040b0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.001+0000 7f8a0359e700 1 -- 172.21.15.46:0/1917869207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f40094f0 con 0x7f8a040c93a0 2024-01-27T01:06:25.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.002+0000 7f8a0359e700 1 --2- 172.21.15.46:0/1917869207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a040c93a0 0x7f8a04149640 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00c910 tx=0x7f89fc0016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:25.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.002+0000 7f89faffd700 1 -- 172.21.15.46:0/1917869207 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc00a7e0 con 0x7f8a040c93a0 2024-01-27T01:06:25.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.002+0000 7f89faffd700 1 -- 172.21.15.46:0/1917869207 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89fc00a940 con 0x7f8a040c93a0 2024-01-27T01:06:25.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.002+0000 7f8a09207700 1 -- 172.21.15.46:0/1917869207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a0414a230 con 0x7f8a040c93a0 2024-01-27T01:06:25.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.002+0000 7f89faffd700 1 -- 172.21.15.46:0/1917869207 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc00ad00 con 0x7f8a040c93a0 2024-01-27T01:06:25.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.002+0000 7f8a09207700 1 -- 172.21.15.46:0/1917869207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a0415cab0 con 0x7f8a040c93a0 2024-01-27T01:06:25.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.003+0000 7f8a09207700 1 -- 172.21.15.46:0/1917869207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a0404c710 con 0x7f8a040c93a0 2024-01-27T01:06:25.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.004+0000 7f89faffd700 1 -- 172.21.15.46:0/1917869207 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f89fc012630 con 0x7f8a040c93a0 2024-01-27T01:06:25.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.004+0000 7f89faffd700 1 --2- 172.21.15.46:0/1917869207 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f89e4066f30 0x7f89e40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:25.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.004+0000 7f89faffd700 1 -- 172.21.15.46:0/1917869207 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 6035+0+0 (secure 0 0 0) 0x7f89fc0848f0 con 0x7f8a040c93a0 2024-01-27T01:06:25.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.004+0000 7f8a02d9d700 1 --2- 172.21.15.46:0/1917869207 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f89e4066f30 0x7f89e40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:25.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.005+0000 7f8a02d9d700 1 --2- 172.21.15.46:0/1917869207 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f89e4066f30 0x7f89e40693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8a04064390 tx=0x7f89f4017000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:25.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.009+0000 7f89faffd700 1 -- 172.21.15.46:0/1917869207 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f89fc0554a0 con 0x7f8a040c93a0 2024-01-27T01:06:25.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.213+0000 7f8a09207700 1 -- 172.21.15.46:0/1917869207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8a040b2480 con 0x7f8a040c93a0 2024-01-27T01:06:25.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.843+0000 7f89faffd700 1 -- 172.21.15.46:0/1917869207 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v221) v1 ==== 170+0+0 (secure 0 0 0) 0x7f89fc016020 con 0x7f8a040c93a0 2024-01-27T01:06:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.860+0000 7f8a09207700 1 -- 172.21.15.46:0/1917869207 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f89e4066f30 msgr2=0x7f89e40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.860+0000 7f8a09207700 1 --2- 172.21.15.46:0/1917869207 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f89e4066f30 0x7f89e40693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8a04064390 tx=0x7f89f4017000 comp rx=0 tx=0).stop 2024-01-27T01:06:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.860+0000 7f8a09207700 1 -- 172.21.15.46:0/1917869207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a040c93a0 msgr2=0x7f8a04149640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.860+0000 7f8a09207700 1 --2- 172.21.15.46:0/1917869207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a040c93a0 0x7f8a04149640 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00c910 tx=0x7f89fc0016f0 comp rx=0 tx=0).stop 2024-01-27T01:06:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.860+0000 7f8a09207700 1 -- 172.21.15.46:0/1917869207 shutdown_connections 2024-01-27T01:06:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.860+0000 7f8a09207700 1 --2- 172.21.15.46:0/1917869207 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f89e4066f30 0x7f89e40693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.860+0000 7f8a09207700 1 --2- 172.21.15.46:0/1917869207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a040c93a0 0x7f8a04149640 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.860+0000 7f8a09207700 1 --2- 172.21.15.46:0/1917869207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a04067d50 0x7f8a040b0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.860+0000 7f8a09207700 1 --2- 172.21.15.46:0/1917869207 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a040681f0 0x7f8a04159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.860+0000 7f8a09207700 1 -- 172.21.15.46:0/1917869207 >> 172.21.15.46:0/1917869207 conn(0x7f8a040ca9a0 msgr2=0x7f8a0414d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:25.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.862+0000 7f8a09207700 1 -- 172.21.15.46:0/1917869207 shutdown_connections 2024-01-27T01:06:25.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:25.862+0000 7f8a09207700 1 -- 172.21.15.46:0/1917869207 wait complete. 2024-01-27T01:06:25.862 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-01-27T01:06:25.877 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-27T01:06:26.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.016+0000 7f169f23a700 1 Processor -- start 2024-01-27T01:06:26.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.017+0000 7f169f23a700 1 -- start start 2024-01-27T01:06:26.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.017+0000 7f169f23a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16980caa60 0x7f1698159d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.017+0000 7f169f23a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16980c8f70 con 0x7f16980ccc50 2024-01-27T01:06:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.017+0000 7f169f23a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16980cd0c0 con 0x7f16981599c0 2024-01-27T01:06:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.017+0000 7f169f23a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169815a9c0 con 0x7f16980caa60 2024-01-27T01:06:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.018+0000 7f1697fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f16980ccc50 0x7f16980ca350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53616/0 (socket says 172.21.15.46:53616) 2024-01-27T01:06:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.018+0000 7f1697fff700 1 -- 172.21.15.46:0/3070243544 learned_addr learned my addr 172.21.15.46:0/3070243544 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.018+0000 7f169d7d7700 1 --2- 172.21.15.46:0/3070243544 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16980caa60 0x7f1698159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.018+0000 7f16977fe700 1 -- 172.21.15.46:0/3070243544 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442241100 0 0) 0x7f16980c8f70 con 0x7f16980ccc50 2024-01-27T01:06:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.018+0000 7f16977fe700 1 -- 172.21.15.46:0/3070243544 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1688003570 con 0x7f16980ccc50 2024-01-27T01:06:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.018+0000 7f16977fe700 1 -- 172.21.15.46:0/3070243544 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632343434 0 0) 0x7f16980cd0c0 con 0x7f16981599c0 2024-01-27T01:06:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.018+0000 7f16977fe700 1 -- 172.21.15.46:0/3070243544 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16980c8f70 con 0x7f16981599c0 2024-01-27T01:06:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.018+0000 7f169d7d7700 1 -- 172.21.15.46:0/3070243544 >> v1:172.21.15.106:6789/0 conn(0x7f16981599c0 legacy=0x7f1698068c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.018+0000 7f169d7d7700 1 -- 172.21.15.46:0/3070243544 >> v1:172.21.15.46:6789/0 conn(0x7f16980ccc50 legacy=0x7f16980ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.018+0000 7f169d7d7700 1 -- 172.21.15.46:0/3070243544 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169815aca0 con 0x7f16980caa60 2024-01-27T01:06:26.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.018+0000 7f169d7d7700 1 --2- 172.21.15.46:0/3070243544 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16980caa60 0x7f1698159d90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1690009e90 tx=0x7f1690009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e69267f511882a77 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:26.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.019+0000 7f16977fe700 1 -- 172.21.15.46:0/3070243544 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169000e070 con 0x7f16980caa60 2024-01-27T01:06:26.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.019+0000 7f16977fe700 1 -- 172.21.15.46:0/3070243544 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169000bb80 con 0x7f16980caa60 2024-01-27T01:06:26.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.019+0000 7f16977fe700 1 -- 172.21.15.46:0/3070243544 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690012430 con 0x7f16980caa60 2024-01-27T01:06:26.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.019+0000 7f169f23a700 1 -- 172.21.15.46:0/3070243544 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16980caa60 msgr2=0x7f1698159d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:26.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.019+0000 7f169f23a700 1 --2- 172.21.15.46:0/3070243544 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16980caa60 0x7f1698159d90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1690009e90 tx=0x7f1690009780 comp rx=0 tx=0).stop 2024-01-27T01:06:26.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.019+0000 7f169f23a700 1 -- 172.21.15.46:0/3070243544 shutdown_connections 2024-01-27T01:06:26.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.019+0000 7f169f23a700 1 --2- 172.21.15.46:0/3070243544 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16980caa60 0x7f1698159d90 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:26.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.019+0000 7f169f23a700 1 -- 172.21.15.46:0/3070243544 >> 172.21.15.46:0/3070243544 conn(0x7f16980ce570 msgr2=0x7f16980ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:26.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.019+0000 7f169f23a700 1 -- 172.21.15.46:0/3070243544 shutdown_connections 2024-01-27T01:06:26.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.019+0000 7f169f23a700 1 -- 172.21.15.46:0/3070243544 wait complete. 2024-01-27T01:06:26.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.021+0000 7f169f23a700 1 Processor -- start 2024-01-27T01:06:26.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.021+0000 7f169f23a700 1 -- start start 2024-01-27T01:06:26.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.022+0000 7f169f23a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16980caa60 0x7f1698154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:26.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.022+0000 7f169f23a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16980ccc50 0x7f16980b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:26.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.022+0000 7f169f23a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16981599c0 0x7f16980b1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:26.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.022+0000 7f169f23a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16980b4430 con 0x7f16980ccc50 2024-01-27T01:06:26.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.022+0000 7f169f23a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1698169a40 con 0x7f16981599c0 2024-01-27T01:06:26.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.022+0000 7f169f23a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1698169d20 con 0x7f16980caa60 2024-01-27T01:06:26.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.022+0000 7f1697fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16980ccc50 0x7f16980b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:26.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.022+0000 7f1697fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16980ccc50 0x7f16980b3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44392/0 (socket says 172.21.15.46:44392) 2024-01-27T01:06:26.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.022+0000 7f1697fff700 1 -- 172.21.15.46:0/2034227169 learned_addr learned my addr 172.21.15.46:0/2034227169 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:26.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.022+0000 7f169cfd6700 1 --2- 172.21.15.46:0/2034227169 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16980caa60 0x7f1698154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:26.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.022+0000 7f169d7d7700 1 --2- 172.21.15.46:0/2034227169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16981599c0 0x7f16980b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:26.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.022+0000 7f1697fff700 1 -- 172.21.15.46:0/2034227169 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16980caa60 msgr2=0x7f1698154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:26.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.022+0000 7f1697fff700 1 --2- 172.21.15.46:0/2034227169 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16980caa60 0x7f1698154c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:26.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.022+0000 7f1697fff700 1 -- 172.21.15.46:0/2034227169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16981599c0 msgr2=0x7f16980b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:26.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.022+0000 7f1697fff700 1 --2- 172.21.15.46:0/2034227169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16981599c0 0x7f16980b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:26.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.022+0000 7f1697fff700 1 -- 172.21.15.46:0/2034227169 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1690009a20 con 0x7f16980ccc50 2024-01-27T01:06:26.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.022+0000 7f1697fff700 1 --2- 172.21.15.46:0/2034227169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16980ccc50 0x7f16980b3ef0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f168400c790 tx=0x7f1684007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:26.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.023+0000 7f1694ff9700 1 -- 172.21.15.46:0/2034227169 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684005b30 con 0x7f16980ccc50 2024-01-27T01:06:26.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.023+0000 7f1694ff9700 1 -- 172.21.15.46:0/2034227169 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1684005c90 con 0x7f16980ccc50 2024-01-27T01:06:26.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.023+0000 7f1694ff9700 1 -- 172.21.15.46:0/2034227169 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684007300 con 0x7f16980ccc50 2024-01-27T01:06:26.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.023+0000 7f169f23a700 1 -- 172.21.15.46:0/2034227169 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169816a000 con 0x7f16980ccc50 2024-01-27T01:06:26.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.023+0000 7f169f23a700 1 -- 172.21.15.46:0/2034227169 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16980c6020 con 0x7f16980ccc50 2024-01-27T01:06:26.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.024+0000 7f169f23a700 1 -- 172.21.15.46:0/2034227169 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169815b380 con 0x7f16980ccc50 2024-01-27T01:06:26.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.024+0000 7f1694ff9700 1 -- 172.21.15.46:0/2034227169 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f168401b3b0 con 0x7f16980ccc50 2024-01-27T01:06:26.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.025+0000 7f1694ff9700 1 --2- 172.21.15.46:0/2034227169 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f167c066f30 0x7f167c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:26.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.025+0000 7f1694ff9700 1 -- 172.21.15.46:0/2034227169 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f1684083d00 con 0x7f16980ccc50 2024-01-27T01:06:26.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.025+0000 7f169cfd6700 1 --2- 172.21.15.46:0/2034227169 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f167c066f30 0x7f167c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:26.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.026+0000 7f169cfd6700 1 --2- 172.21.15.46:0/2034227169 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f167c066f30 0x7f167c0693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f168c007fd0 tx=0x7f168c007d10 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:26.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.029+0000 7f1694ff9700 1 -- 172.21.15.46:0/2034227169 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f169815b380 con 0x7f16980ccc50 2024-01-27T01:06:26.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.284+0000 7f169f23a700 1 -- 172.21.15.46:0/2034227169 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f169815b380 con 0x7f16980ccc50 2024-01-27T01:06:26.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-01-27T01:06:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.884+0000 7f1694ff9700 1 -- 172.21.15.46:0/2034227169 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 27 and data pool 28 v113) v1 ==== 174+0+0 (secure 0 0 0) 0x7f169815b380 con 0x7f16980ccc50 2024-01-27T01:06:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.888+0000 7f169f23a700 1 -- 172.21.15.46:0/2034227169 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f167c066f30 msgr2=0x7f167c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.888+0000 7f169f23a700 1 --2- 172.21.15.46:0/2034227169 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f167c066f30 0x7f167c0693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f168c007fd0 tx=0x7f168c007d10 comp rx=0 tx=0).stop 2024-01-27T01:06:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.888+0000 7f169f23a700 1 -- 172.21.15.46:0/2034227169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16980ccc50 msgr2=0x7f16980b3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.888+0000 7f169f23a700 1 --2- 172.21.15.46:0/2034227169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16980ccc50 0x7f16980b3ef0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f168400c790 tx=0x7f1684007ea0 comp rx=0 tx=0).stop 2024-01-27T01:06:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.889+0000 7f169f23a700 1 -- 172.21.15.46:0/2034227169 shutdown_connections 2024-01-27T01:06:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.889+0000 7f169f23a700 1 --2- 172.21.15.46:0/2034227169 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f167c066f30 0x7f167c0693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.889+0000 7f169f23a700 1 --2- 172.21.15.46:0/2034227169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f16981599c0 0x7f16980b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.889+0000 7f169f23a700 1 --2- 172.21.15.46:0/2034227169 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16980caa60 0x7f1698154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.889+0000 7f169f23a700 1 --2- 172.21.15.46:0/2034227169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16980ccc50 0x7f16980b3ef0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.889+0000 7f169f23a700 1 -- 172.21.15.46:0/2034227169 >> 172.21.15.46:0/2034227169 conn(0x7f16980ce570 msgr2=0x7f16980c9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.889+0000 7f169f23a700 1 -- 172.21.15.46:0/2034227169 shutdown_connections 2024-01-27T01:06:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:26.889+0000 7f169f23a700 1 -- 172.21.15.46:0/2034227169 wait complete. 2024-01-27T01:06:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 27 and data pool 28 2024-01-27T01:06:26.904 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:06:27.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.041+0000 7fb694e42700 1 Processor -- start 2024-01-27T01:06:27.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.041+0000 7fb694e42700 1 -- start start 2024-01-27T01:06:27.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.042+0000 7fb694e42700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb6900acf30 0x7fb6900c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.042+0000 7fb694e42700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6900ad300 0x7fb6900ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.042+0000 7fb694e42700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6900c4230 con 0x7fb6900ad750 2024-01-27T01:06:27.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.042+0000 7fb694e42700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb690157be0 con 0x7fb6900acf30 2024-01-27T01:06:27.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.042+0000 7fb694e42700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6900cbf10 con 0x7fb6900ad300 2024-01-27T01:06:27.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.042+0000 7fb68ed9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb6900ad750 0x7fb69015b270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53630/0 (socket says 172.21.15.46:53630) 2024-01-27T01:06:27.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.042+0000 7fb68ed9d700 1 -- 172.21.15.46:0/3174880191 learned_addr learned my addr 172.21.15.46:0/3174880191 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:27.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.042+0000 7fb68dd9b700 1 --2- 172.21.15.46:0/3174880191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6900ad300 0x7fb6900ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.042+0000 7fb68e59c700 1 --2- 172.21.15.46:0/3174880191 >> v2:172.21.15.106:3300/0 conn(0x7fb6900acf30 0x7fb6900c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.042+0000 7fb68d59a700 1 -- 172.21.15.46:0/3174880191 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067243494 0 0) 0x7fb6900c4230 con 0x7fb6900ad750 2024-01-27T01:06:27.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.043+0000 7fb68d59a700 1 -- 172.21.15.46:0/3174880191 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb67c003570 con 0x7fb6900ad750 2024-01-27T01:06:27.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.043+0000 7fb68e59c700 1 -- 172.21.15.46:0/3174880191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6900ad300 msgr2=0x7fb6900ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:27.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.043+0000 7fb68e59c700 1 --2- 172.21.15.46:0/3174880191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6900ad300 0x7fb6900ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.043+0000 7fb68d59a700 1 -- 172.21.15.46:0/3174880191 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1742945963 0 0) 0x7fb67c003570 con 0x7fb6900ad750 2024-01-27T01:06:27.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.043+0000 7fb68e59c700 1 -- 172.21.15.46:0/3174880191 >> v1:172.21.15.46:6789/0 conn(0x7fb6900ad750 legacy=0x7fb69015b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:27.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.043+0000 7fb68e59c700 1 -- 172.21.15.46:0/3174880191 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6900c4460 con 0x7fb6900acf30 2024-01-27T01:06:27.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.043+0000 7fb68dd9b700 1 --2- 172.21.15.46:0/3174880191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6900ad300 0x7fb6900ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:27.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.043+0000 7fb68e59c700 1 --2- 172.21.15.46:0/3174880191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6900acf30 0x7fb6900c3cf0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fb6780098e0 tx=0x7fb6780091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35b4697bd821ac3b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:27.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.043+0000 7fb68d59a700 1 -- 172.21.15.46:0/3174880191 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6880017f0 con 0x7fb6900acf30 2024-01-27T01:06:27.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.043+0000 7fb68d59a700 1 -- 172.21.15.46:0/3174880191 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb678011420 con 0x7fb6900acf30 2024-01-27T01:06:27.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.043+0000 7fb68d59a700 1 -- 172.21.15.46:0/3174880191 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6780117a0 con 0x7fb6900acf30 2024-01-27T01:06:27.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.044+0000 7fb694e42700 1 -- 172.21.15.46:0/3174880191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6900acf30 msgr2=0x7fb6900c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:27.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.044+0000 7fb694e42700 1 --2- 172.21.15.46:0/3174880191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6900acf30 0x7fb6900c3cf0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fb6780098e0 tx=0x7fb6780091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.044+0000 7fb694e42700 1 -- 172.21.15.46:0/3174880191 shutdown_connections 2024-01-27T01:06:27.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.044+0000 7fb694e42700 1 --2- 172.21.15.46:0/3174880191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6900acf30 0x7fb6900c3cf0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.044+0000 7fb694e42700 1 --2- 172.21.15.46:0/3174880191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6900ad300 0x7fb6900ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.044+0000 7fb694e42700 1 -- 172.21.15.46:0/3174880191 >> 172.21.15.46:0/3174880191 conn(0x7fb690097820 msgr2=0x7fb690068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:27.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.044+0000 7fb694e42700 1 -- 172.21.15.46:0/3174880191 shutdown_connections 2024-01-27T01:06:27.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.044+0000 7fb694e42700 1 -- 172.21.15.46:0/3174880191 wait complete. 2024-01-27T01:06:27.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.045+0000 7fb694e42700 1 Processor -- start 2024-01-27T01:06:27.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.045+0000 7fb694e42700 1 -- start start 2024-01-27T01:06:27.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.046+0000 7fb694e42700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6900acf30 0x7fb6901538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.046+0000 7fb694e42700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6900ad300 0x7fb6900b3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.046+0000 7fb694e42700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6900ad750 0x7fb6900b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.046+0000 7fb694e42700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6900b1930 con 0x7fb6900ad300 2024-01-27T01:06:27.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.046+0000 7fb694e42700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6900afb10 con 0x7fb6900acf30 2024-01-27T01:06:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.046+0000 7fb694e42700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6900afdf0 con 0x7fb6900ad750 2024-01-27T01:06:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.046+0000 7fb68dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6900ad300 0x7fb6900b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.046+0000 7fb68dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6900ad300 0x7fb6900b3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44396/0 (socket says 172.21.15.46:44396) 2024-01-27T01:06:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.046+0000 7fb68dd9b700 1 -- 172.21.15.46:0/777273938 learned_addr learned my addr 172.21.15.46:0/777273938 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.046+0000 7fb68e59c700 1 --2- 172.21.15.46:0/777273938 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6900acf30 0x7fb6901538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.046+0000 7fb68ed9d700 1 --2- 172.21.15.46:0/777273938 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6900ad750 0x7fb6900b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.046+0000 7fb68dd9b700 1 -- 172.21.15.46:0/777273938 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6900ad750 msgr2=0x7fb6900b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.046+0000 7fb68dd9b700 1 --2- 172.21.15.46:0/777273938 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6900ad750 0x7fb6900b3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.046+0000 7fb68dd9b700 1 -- 172.21.15.46:0/777273938 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6900acf30 msgr2=0x7fb6901538c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.046+0000 7fb68dd9b700 1 --2- 172.21.15.46:0/777273938 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6900acf30 0x7fb6901538c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.047+0000 7fb68dd9b700 1 -- 172.21.15.46:0/777273938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb680009420 con 0x7fb6900ad300 2024-01-27T01:06:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.047+0000 7fb68dd9b700 1 --2- 172.21.15.46:0/777273938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6900ad300 0x7fb6900b3020 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fb68000ec80 tx=0x7fb6800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.047+0000 7fb6867fc700 1 -- 172.21.15.46:0/777273938 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68000c9d0 con 0x7fb6900ad300 2024-01-27T01:06:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.047+0000 7fb6867fc700 1 -- 172.21.15.46:0/777273938 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68000cb30 con 0x7fb6900ad300 2024-01-27T01:06:27.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.047+0000 7fb6867fc700 1 -- 172.21.15.46:0/777273938 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68000ceb0 con 0x7fb6900ad300 2024-01-27T01:06:27.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.047+0000 7fb694e42700 1 -- 172.21.15.46:0/777273938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb678009420 con 0x7fb6900ad300 2024-01-27T01:06:27.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.047+0000 7fb694e42700 1 -- 172.21.15.46:0/777273938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6900b15c0 con 0x7fb6900ad300 2024-01-27T01:06:27.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.048+0000 7fb694e42700 1 -- 172.21.15.46:0/777273938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb69004c710 con 0x7fb6900ad300 2024-01-27T01:06:27.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.048+0000 7fb6867fc700 1 -- 172.21.15.46:0/777273938 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb680013680 con 0x7fb6900ad300 2024-01-27T01:06:27.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.049+0000 7fb6867fc700 1 --2- 172.21.15.46:0/777273938 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb670066f30 0x7fb6700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.049+0000 7fb6867fc700 1 -- 172.21.15.46:0/777273938 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fb680084ca0 con 0x7fb6900ad300 2024-01-27T01:06:27.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.049+0000 7fb68e59c700 1 --2- 172.21.15.46:0/777273938 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb670066f30 0x7fb6700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.050+0000 7fb68e59c700 1 --2- 172.21.15.46:0/777273938 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb670066f30 0x7fb6700693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb6780091e0 tx=0x7fb678016000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.053+0000 7fb6867fc700 1 -- 172.21.15.46:0/777273938 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb680051be0 con 0x7fb6900ad300 2024-01-27T01:06:27.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.259+0000 7fb694e42700 1 -- 172.21.15.46:0/777273938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6900c4740 con 0x7fb6900ad300 2024-01-27T01:06:27.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.259+0000 7fb6867fc700 1 -- 172.21.15.46:0/777273938 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v222) v1 ==== 74+0+16338 (secure 0 0 0) 0x7fb680055200 con 0x7fb6900ad300 2024-01-27T01:06:27.259 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:27.259 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":222,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:06:26.285700+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:06:24.798791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-01-27T01:06:25.249215+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:06:27.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.262+0000 7fb694e42700 1 -- 172.21.15.46:0/777273938 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb670066f30 msgr2=0x7fb6700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:27.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.262+0000 7fb694e42700 1 --2- 172.21.15.46:0/777273938 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb670066f30 0x7fb6700693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb6780091e0 tx=0x7fb678016000 comp rx=0 tx=0).stop 2024-01-27T01:06:27.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.262+0000 7fb694e42700 1 -- 172.21.15.46:0/777273938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6900ad300 msgr2=0x7fb6900b3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:27.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.262+0000 7fb694e42700 1 --2- 172.21.15.46:0/777273938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6900ad300 0x7fb6900b3020 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fb68000ec80 tx=0x7fb6800091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.262+0000 7fb694e42700 1 -- 172.21.15.46:0/777273938 shutdown_connections 2024-01-27T01:06:27.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.262+0000 7fb694e42700 1 --2- 172.21.15.46:0/777273938 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb670066f30 0x7fb6700693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.262+0000 7fb694e42700 1 --2- 172.21.15.46:0/777273938 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6900ad750 0x7fb6900b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.262+0000 7fb694e42700 1 --2- 172.21.15.46:0/777273938 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6900acf30 0x7fb6901538c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.262+0000 7fb694e42700 1 --2- 172.21.15.46:0/777273938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6900ad300 0x7fb6900b3020 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.262+0000 7fb694e42700 1 -- 172.21.15.46:0/777273938 >> 172.21.15.46:0/777273938 conn(0x7fb690097820 msgr2=0x7fb6900cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:27.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.262+0000 7fb694e42700 1 -- 172.21.15.46:0/777273938 shutdown_connections 2024-01-27T01:06:27.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.262+0000 7fb694e42700 1 -- 172.21.15.46:0/777273938 wait complete. 2024-01-27T01:06:27.275 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:06:27.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.408+0000 7f62bcb64700 1 Processor -- start 2024-01-27T01:06:27.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.408+0000 7f62bcb64700 1 -- start start 2024-01-27T01:06:27.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.409+0000 7f62bcb64700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62b806cdb0 0x7f62b806eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.409+0000 7f62bcb64700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f62b81579f0 0x7f62b806f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.409+0000 7f62bcb64700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62b8157560 con 0x7f62b81541c0 2024-01-27T01:06:27.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.409+0000 7f62bcb64700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62b8156d00 con 0x7f62b81579f0 2024-01-27T01:06:27.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.409+0000 7f62bcb64700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b80cc940 con 0x7f62b806cdb0 2024-01-27T01:06:27.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.409+0000 7f62b6d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f62b81579f0 0x7f62b806f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.409+0000 7f62b6d9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f62b81579f0 0x7f62b806f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44400/0 (socket says 172.21.15.46:44400) 2024-01-27T01:06:27.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.409+0000 7f62b6d9d700 1 -- 172.21.15.46:0/727809926 learned_addr learned my addr 172.21.15.46:0/727809926 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:27.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.410+0000 7f62b5d9b700 1 --2- 172.21.15.46:0/727809926 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62b806cdb0 0x7f62b806eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.410+0000 7f62b6d9d700 1 -- 172.21.15.46:0/727809926 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62b806cdb0 msgr2=0x7f62b806eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:27.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.410+0000 7f62b6d9d700 1 --2- 172.21.15.46:0/727809926 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62b806cdb0 0x7f62b806eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.410+0000 7f62b6d9d700 1 -- 172.21.15.46:0/727809926 >> v1:172.21.15.106:6789/0 conn(0x7f62b81541c0 legacy=0x7f62b80cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:27.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.410+0000 7f62b6d9d700 1 -- 172.21.15.46:0/727809926 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b80ccc10 con 0x7f62b81579f0 2024-01-27T01:06:27.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.410+0000 7f62b6d9d700 1 --2- 172.21.15.46:0/727809926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62b81579f0 0x7f62b806f060 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009f20 tx=0x7f62b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9309ec8c95240c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:27.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.410+0000 7f62b559a700 1 -- 172.21.15.46:0/727809926 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b000e070 con 0x7f62b81579f0 2024-01-27T01:06:27.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.410+0000 7f62b559a700 1 -- 172.21.15.46:0/727809926 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b000b9d0 con 0x7f62b81579f0 2024-01-27T01:06:27.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.410+0000 7f62b559a700 1 -- 172.21.15.46:0/727809926 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b00092e0 con 0x7f62b81579f0 2024-01-27T01:06:27.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.411+0000 7f62bcb64700 1 -- 172.21.15.46:0/727809926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62b81579f0 msgr2=0x7f62b806f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:27.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.411+0000 7f62bcb64700 1 --2- 172.21.15.46:0/727809926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62b81579f0 0x7f62b806f060 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009f20 tx=0x7f62b0009840 comp rx=0 tx=0).stop 2024-01-27T01:06:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.411+0000 7f62bcb64700 1 -- 172.21.15.46:0/727809926 shutdown_connections 2024-01-27T01:06:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.411+0000 7f62bcb64700 1 --2- 172.21.15.46:0/727809926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62b81579f0 0x7f62b806f060 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.411+0000 7f62bcb64700 1 --2- 172.21.15.46:0/727809926 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62b806cdb0 0x7f62b806eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.411+0000 7f62bcb64700 1 -- 172.21.15.46:0/727809926 >> 172.21.15.46:0/727809926 conn(0x7f62b80cdf10 msgr2=0x7f62b80c9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.411+0000 7f62bcb64700 1 -- 172.21.15.46:0/727809926 shutdown_connections 2024-01-27T01:06:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.411+0000 7f62bcb64700 1 -- 172.21.15.46:0/727809926 wait complete. 2024-01-27T01:06:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.413+0000 7f62bcb64700 1 Processor -- start 2024-01-27T01:06:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.413+0000 7f62bcb64700 1 -- start start 2024-01-27T01:06:27.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62bcb64700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62b806cdb0 0x7f62b80c68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62bcb64700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62b81541c0 0x7f62b8145fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62bcb64700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62b81579f0 0x7f62b80679f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62bcb64700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b815a980 con 0x7f62b806cdb0 2024-01-27T01:06:27.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62bcb64700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b80c7480 con 0x7f62b81579f0 2024-01-27T01:06:27.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62bcb64700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b80c71a0 con 0x7f62b81541c0 2024-01-27T01:06:27.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62b659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62b806cdb0 0x7f62b80c68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62b659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62b806cdb0 0x7f62b80c68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44412/0 (socket says 172.21.15.46:44412) 2024-01-27T01:06:27.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62b659c700 1 -- 172.21.15.46:0/3885021485 learned_addr learned my addr 172.21.15.46:0/3885021485 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:27.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62b5d9b700 1 --2- 172.21.15.46:0/3885021485 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62b81541c0 0x7f62b8145fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62b6d9d700 1 --2- 172.21.15.46:0/3885021485 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62b81579f0 0x7f62b80679f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62b659c700 1 -- 172.21.15.46:0/3885021485 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62b81541c0 msgr2=0x7f62b8145fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:27.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62b659c700 1 --2- 172.21.15.46:0/3885021485 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62b81541c0 0x7f62b8145fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62b659c700 1 -- 172.21.15.46:0/3885021485 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62b81579f0 msgr2=0x7f62b80679f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:27.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62b659c700 1 --2- 172.21.15.46:0/3885021485 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62b81579f0 0x7f62b80679f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62b659c700 1 -- 172.21.15.46:0/3885021485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b0009ae0 con 0x7f62b806cdb0 2024-01-27T01:06:27.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62b6d9d700 1 --2- 172.21.15.46:0/3885021485 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62b81579f0 0x7f62b80679f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:27.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.414+0000 7f62b659c700 1 --2- 172.21.15.46:0/3885021485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62b806cdb0 0x7f62b80c68a0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f62a000c590 tx=0x7f62a000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.415+0000 7f62ae7fc700 1 -- 172.21.15.46:0/3885021485 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a0011590 con 0x7f62b806cdb0 2024-01-27T01:06:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.415+0000 7f62ae7fc700 1 -- 172.21.15.46:0/3885021485 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a00116f0 con 0x7f62b806cdb0 2024-01-27T01:06:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.415+0000 7f62bcb64700 1 -- 172.21.15.46:0/3885021485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b815a330 con 0x7f62b806cdb0 2024-01-27T01:06:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.415+0000 7f62ae7fc700 1 -- 172.21.15.46:0/3885021485 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a0011ab0 con 0x7f62b806cdb0 2024-01-27T01:06:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.415+0000 7f62bcb64700 1 -- 172.21.15.46:0/3885021485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b80c6de0 con 0x7f62b806cdb0 2024-01-27T01:06:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.416+0000 7f62bcb64700 1 -- 172.21.15.46:0/3885021485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b804c710 con 0x7f62b806cdb0 2024-01-27T01:06:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.416+0000 7f62ae7fc700 1 -- 172.21.15.46:0/3885021485 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f62a001b3b0 con 0x7f62b806cdb0 2024-01-27T01:06:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.417+0000 7f62ae7fc700 1 --2- 172.21.15.46:0/3885021485 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6298066f30 0x7f62980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.417+0000 7f62ae7fc700 1 -- 172.21.15.46:0/3885021485 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f62a0083e10 con 0x7f62b806cdb0 2024-01-27T01:06:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.417+0000 7f62b5d9b700 1 --2- 172.21.15.46:0/3885021485 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6298066f30 0x7f62980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.417+0000 7f62b5d9b700 1 --2- 172.21.15.46:0/3885021485 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6298066f30 0x7f62980693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f62b80b2ec0 tx=0x7f62a8008040 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.421+0000 7f62ae7fc700 1 -- 172.21.15.46:0/3885021485 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f62a0050d50 con 0x7f62b806cdb0 2024-01-27T01:06:27.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.628+0000 7f62bcb64700 1 -- 172.21.15.46:0/3885021485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62b8156f10 con 0x7f62b806cdb0 2024-01-27T01:06:27.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.629+0000 7f62ae7fc700 1 -- 172.21.15.46:0/3885021485 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v222) v1 ==== 74+0+16338 (secure 0 0 0) 0x7f62a0017070 con 0x7f62b806cdb0 2024-01-27T01:06:27.628 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:27.628 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":222,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:06:26.285700+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:06:24.798791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-01-27T01:06:25.249215+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:06:27.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.631+0000 7f62bcb64700 1 -- 172.21.15.46:0/3885021485 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6298066f30 msgr2=0x7f62980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:27.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.631+0000 7f62bcb64700 1 --2- 172.21.15.46:0/3885021485 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6298066f30 0x7f62980693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f62b80b2ec0 tx=0x7f62a8008040 comp rx=0 tx=0).stop 2024-01-27T01:06:27.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.631+0000 7f62bcb64700 1 -- 172.21.15.46:0/3885021485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62b806cdb0 msgr2=0x7f62b80c68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:27.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.631+0000 7f62bcb64700 1 --2- 172.21.15.46:0/3885021485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62b806cdb0 0x7f62b80c68a0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f62a000c590 tx=0x7f62a000c950 comp rx=0 tx=0).stop 2024-01-27T01:06:27.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.631+0000 7f62bcb64700 1 -- 172.21.15.46:0/3885021485 shutdown_connections 2024-01-27T01:06:27.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.631+0000 7f62bcb64700 1 --2- 172.21.15.46:0/3885021485 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6298066f30 0x7f62980693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.631+0000 7f62bcb64700 1 --2- 172.21.15.46:0/3885021485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62b806cdb0 0x7f62b80c68a0 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.631+0000 7f62bcb64700 1 --2- 172.21.15.46:0/3885021485 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62b81579f0 0x7f62b80679f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.631+0000 7f62bcb64700 1 --2- 172.21.15.46:0/3885021485 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62b81541c0 0x7f62b8145fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.631+0000 7f62bcb64700 1 -- 172.21.15.46:0/3885021485 >> 172.21.15.46:0/3885021485 conn(0x7f62b80cdf10 msgr2=0x7f62b8159780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:27.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.631+0000 7f62bcb64700 1 -- 172.21.15.46:0/3885021485 shutdown_connections 2024-01-27T01:06:27.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.631+0000 7f62bcb64700 1 -- 172.21.15.46:0/3885021485 wait complete. 2024-01-27T01:06:27.644 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-27T01:06:27.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.780+0000 7f0a4fc73700 1 Processor -- start 2024-01-27T01:06:27.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.780+0000 7f0a4fc73700 1 -- start start 2024-01-27T01:06:27.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.781+0000 7f0a4fc73700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a48159aa0 0x7f0a480cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.781+0000 7f0a4fc73700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0a48159e70 0x7f0a48067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.781+0000 7f0a4fc73700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a480cce30 con 0x7f0a480cd2c0 2024-01-27T01:06:27.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.781+0000 7f0a4fc73700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a480c9970 con 0x7f0a48159aa0 2024-01-27T01:06:27.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.781+0000 7f0a4fc73700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a480c9c50 con 0x7f0a48159e70 2024-01-27T01:06:27.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.782+0000 7f0a4da0f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a48159aa0 0x7f0a480cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.782+0000 7f0a4da0f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a48159aa0 0x7f0a480cdff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44416/0 (socket says 172.21.15.46:44416) 2024-01-27T01:06:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.782+0000 7f0a4da0f700 1 -- 172.21.15.46:0/1829008000 learned_addr learned my addr 172.21.15.46:0/1829008000 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.782+0000 7f0a4e210700 1 --2- 172.21.15.46:0/1829008000 >> v2:172.21.15.106:3300/0 conn(0x7f0a48159e70 0x7f0a48067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.782+0000 7f0a4da0f700 1 -- 172.21.15.46:0/1829008000 >> v2:172.21.15.106:3300/0 conn(0x7f0a48159e70 msgr2=0x7f0a48067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.782+0000 7f0a4da0f700 1 --2- 172.21.15.46:0/1829008000 >> v2:172.21.15.106:3300/0 conn(0x7f0a48159e70 0x7f0a48067930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.782+0000 7f0a4da0f700 1 -- 172.21.15.46:0/1829008000 >> v1:172.21.15.106:6789/0 conn(0x7f0a480cd2c0 legacy=0x7f0a480c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.782+0000 7f0a4da0f700 1 -- 172.21.15.46:0/1829008000 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a480c9620 con 0x7f0a48159aa0 2024-01-27T01:06:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.782+0000 7f0a4da0f700 1 --2- 172.21.15.46:0/1829008000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a48159aa0 0x7f0a480cdff0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f0a4806e1f0 tx=0x7f0a44009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b96ccf624c83a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.782+0000 7f0a4ca0d700 1 -- 172.21.15.46:0/1829008000 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a44005c10 con 0x7f0a48159aa0 2024-01-27T01:06:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.782+0000 7f0a4ca0d700 1 -- 172.21.15.46:0/1829008000 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a44005d70 con 0x7f0a48159aa0 2024-01-27T01:06:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.782+0000 7f0a4ca0d700 1 -- 172.21.15.46:0/1829008000 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4400b8a0 con 0x7f0a48159aa0 2024-01-27T01:06:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.783+0000 7f0a4fc73700 1 -- 172.21.15.46:0/1829008000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a48159aa0 msgr2=0x7f0a480cdff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.783+0000 7f0a4fc73700 1 --2- 172.21.15.46:0/1829008000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a48159aa0 0x7f0a480cdff0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f0a4806e1f0 tx=0x7f0a44009220 comp rx=0 tx=0).stop 2024-01-27T01:06:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.783+0000 7f0a4fc73700 1 -- 172.21.15.46:0/1829008000 shutdown_connections 2024-01-27T01:06:27.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.783+0000 7f0a4fc73700 1 --2- 172.21.15.46:0/1829008000 >> v2:172.21.15.106:3300/0 conn(0x7f0a48159e70 0x7f0a48067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.783+0000 7f0a4fc73700 1 --2- 172.21.15.46:0/1829008000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a48159aa0 0x7f0a480cdff0 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.783+0000 7f0a4fc73700 1 -- 172.21.15.46:0/1829008000 >> 172.21.15.46:0/1829008000 conn(0x7f0a48069220 msgr2=0x7f0a480cd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:27.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.783+0000 7f0a4fc73700 1 -- 172.21.15.46:0/1829008000 shutdown_connections 2024-01-27T01:06:27.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.783+0000 7f0a4fc73700 1 -- 172.21.15.46:0/1829008000 wait complete. 2024-01-27T01:06:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.784+0000 7f0a4fc73700 1 Processor -- start 2024-01-27T01:06:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.784+0000 7f0a4fc73700 1 -- start start 2024-01-27T01:06:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.785+0000 7f0a4fc73700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a480cd2c0 0x7f0a480b1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.785+0000 7f0a4fc73700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a48159aa0 0x7f0a480c18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.785+0000 7f0a4fc73700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a48159e70 0x7f0a48158400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.785+0000 7f0a4fc73700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a480c2070 con 0x7f0a480cd2c0 2024-01-27T01:06:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.785+0000 7f0a4fc73700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a480c1e20 con 0x7f0a48159aa0 2024-01-27T01:06:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.785+0000 7f0a4fc73700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a480b1760 con 0x7f0a48159e70 2024-01-27T01:06:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.785+0000 7f0a4da0f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a480cd2c0 0x7f0a480b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.785+0000 7f0a4da0f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a480cd2c0 0x7f0a480b1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44424/0 (socket says 172.21.15.46:44424) 2024-01-27T01:06:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.785+0000 7f0a4da0f700 1 -- 172.21.15.46:0/2508590277 learned_addr learned my addr 172.21.15.46:0/2508590277 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.785+0000 7f0a4e210700 1 --2- 172.21.15.46:0/2508590277 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a48159e70 0x7f0a48158400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.785+0000 7f0a4d20e700 1 --2- 172.21.15.46:0/2508590277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a48159aa0 0x7f0a480c18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.786+0000 7f0a4da0f700 1 -- 172.21.15.46:0/2508590277 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a48159e70 msgr2=0x7f0a48158400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.786+0000 7f0a4da0f700 1 --2- 172.21.15.46:0/2508590277 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a48159e70 0x7f0a48158400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.786+0000 7f0a4da0f700 1 -- 172.21.15.46:0/2508590277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a48159aa0 msgr2=0x7f0a480c18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.786+0000 7f0a4da0f700 1 --2- 172.21.15.46:0/2508590277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a48159aa0 0x7f0a480c18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.786+0000 7f0a4da0f700 1 -- 172.21.15.46:0/2508590277 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a44009480 con 0x7f0a480cd2c0 2024-01-27T01:06:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.786+0000 7f0a4da0f700 1 --2- 172.21.15.46:0/2508590277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a480cd2c0 0x7f0a480b1080 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f0a4806dfe0 tx=0x7f0a44016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.786+0000 7f0a3dffb700 1 -- 172.21.15.46:0/2508590277 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4401a420 con 0x7f0a480cd2c0 2024-01-27T01:06:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.786+0000 7f0a3dffb700 1 -- 172.21.15.46:0/2508590277 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4401a580 con 0x7f0a480cd2c0 2024-01-27T01:06:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.786+0000 7f0a4fc73700 1 -- 172.21.15.46:0/2508590277 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a480b1a40 con 0x7f0a480cd2c0 2024-01-27T01:06:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.786+0000 7f0a3dffb700 1 -- 172.21.15.46:0/2508590277 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4401a940 con 0x7f0a480cd2c0 2024-01-27T01:06:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.786+0000 7f0a4fc73700 1 -- 172.21.15.46:0/2508590277 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a480b2000 con 0x7f0a480cd2c0 2024-01-27T01:06:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.787+0000 7f0a4fc73700 1 -- 172.21.15.46:0/2508590277 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4804c710 con 0x7f0a480cd2c0 2024-01-27T01:06:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.788+0000 7f0a3dffb700 1 -- 172.21.15.46:0/2508590277 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0a44011420 con 0x7f0a480cd2c0 2024-01-27T01:06:27.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.788+0000 7f0a3dffb700 1 --2- 172.21.15.46:0/2508590277 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a2c066f30 0x7f0a2c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:27.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.788+0000 7f0a3dffb700 1 -- 172.21.15.46:0/2508590277 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f0a44085d50 con 0x7f0a480cd2c0 2024-01-27T01:06:27.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.788+0000 7f0a4d20e700 1 --2- 172.21.15.46:0/2508590277 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a2c066f30 0x7f0a2c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:27.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.789+0000 7f0a4d20e700 1 --2- 172.21.15.46:0/2508590277 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a2c066f30 0x7f0a2c0693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0a38003a80 tx=0x7f0a38007ec0 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:27.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:27.793+0000 7f0a3dffb700 1 -- 172.21.15.46:0/2508590277 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a44089030 con 0x7f0a480cd2c0 2024-01-27T01:06:28.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:28.050+0000 7f0a4fc73700 1 -- 172.21.15.46:0/2508590277 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0a480642f0 con 0x7f0a480cd2c0 2024-01-27T01:06:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:28.874+0000 7f0a3dffb700 1 -- 172.21.15.46:0/2508590277 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v115) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0a440290b0 con 0x7f0a480cd2c0 2024-01-27T01:06:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:28.877+0000 7f0a4fc73700 1 -- 172.21.15.46:0/2508590277 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a2c066f30 msgr2=0x7f0a2c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:28.877+0000 7f0a4fc73700 1 --2- 172.21.15.46:0/2508590277 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a2c066f30 0x7f0a2c0693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0a38003a80 tx=0x7f0a38007ec0 comp rx=0 tx=0).stop 2024-01-27T01:06:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:28.877+0000 7f0a4fc73700 1 -- 172.21.15.46:0/2508590277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a480cd2c0 msgr2=0x7f0a480b1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:28.877+0000 7f0a4fc73700 1 --2- 172.21.15.46:0/2508590277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a480cd2c0 0x7f0a480b1080 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f0a4806dfe0 tx=0x7f0a44016000 comp rx=0 tx=0).stop 2024-01-27T01:06:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:28.877+0000 7f0a4fc73700 1 -- 172.21.15.46:0/2508590277 shutdown_connections 2024-01-27T01:06:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:28.877+0000 7f0a4fc73700 1 --2- 172.21.15.46:0/2508590277 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a2c066f30 0x7f0a2c0693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:28.877+0000 7f0a4fc73700 1 --2- 172.21.15.46:0/2508590277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a480cd2c0 0x7f0a480b1080 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:28.877+0000 7f0a4fc73700 1 --2- 172.21.15.46:0/2508590277 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a48159e70 0x7f0a48158400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:28.877+0000 7f0a4fc73700 1 --2- 172.21.15.46:0/2508590277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a48159aa0 0x7f0a480c18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:28.877+0000 7f0a4fc73700 1 -- 172.21.15.46:0/2508590277 >> 172.21.15.46:0/2508590277 conn(0x7f0a48069220 msgr2=0x7f0a480aa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:28.878+0000 7f0a4fc73700 1 -- 172.21.15.46:0/2508590277 shutdown_connections 2024-01-27T01:06:28.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:28.878+0000 7f0a4fc73700 1 -- 172.21.15.46:0/2508590277 wait complete. 2024-01-27T01:06:28.891 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-27T01:06:29.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.031+0000 7f9f5112c700 1 Processor -- start 2024-01-27T01:06:29.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.031+0000 7f9f5112c700 1 -- start start 2024-01-27T01:06:29.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.032+0000 7f9f5112c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9f4c0cb640 0x7f9f4c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:29.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.032+0000 7f9f5112c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4c0ccc00 con 0x7f9f4c0c8da0 2024-01-27T01:06:29.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.032+0000 7f9f5112c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4c0cc310 con 0x7f9f4c141200 2024-01-27T01:06:29.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.032+0000 7f9f5112c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f4c0cbbf0 con 0x7f9f4c0cb640 2024-01-27T01:06:29.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.032+0000 7f9f4ad9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9f4c0c8da0 0x7f9f4c140af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53634/0 (socket says 172.21.15.46:53634) 2024-01-27T01:06:29.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.032+0000 7f9f4ad9d700 1 -- 172.21.15.46:0/2074781651 learned_addr learned my addr 172.21.15.46:0/2074781651 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:29.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4b59e700 1 --2- 172.21.15.46:0/2074781651 >> v2:172.21.15.106:3300/0 conn(0x7f9f4c0cb640 0x7f9f4c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:29.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4a59c700 1 -- 172.21.15.46:0/2074781651 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648066282 0 0) 0x7f9f4c0ccc00 con 0x7f9f4c0c8da0 2024-01-27T01:06:29.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4a59c700 1 -- 172.21.15.46:0/2074781651 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f38003570 con 0x7f9f4c0c8da0 2024-01-27T01:06:29.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4a59c700 1 -- 172.21.15.46:0/2074781651 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417458390 0 0) 0x7f9f4c0cc310 con 0x7f9f4c141200 2024-01-27T01:06:29.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4a59c700 1 -- 172.21.15.46:0/2074781651 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f4c0ccc00 con 0x7f9f4c141200 2024-01-27T01:06:29.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4a59c700 1 -- 172.21.15.46:0/2074781651 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1626298863 0 0) 0x7f9f38003570 con 0x7f9f4c0c8da0 2024-01-27T01:06:29.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4a59c700 1 -- 172.21.15.46:0/2074781651 >> v2:172.21.15.106:3300/0 conn(0x7f9f4c0cb640 msgr2=0x7f9f4c0c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:29.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4a59c700 1 --2- 172.21.15.46:0/2074781651 >> v2:172.21.15.106:3300/0 conn(0x7f9f4c0cb640 0x7f9f4c0c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:29.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4a59c700 1 -- 172.21.15.46:0/2074781651 >> v1:172.21.15.106:6789/0 conn(0x7f9f4c141200 legacy=0x7f9f4c0caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:29.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4a59c700 1 -- 172.21.15.46:0/2074781651 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c0cc5b0 con 0x7f9f4c0c8da0 2024-01-27T01:06:29.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4a59c700 1 -- 172.21.15.46:0/2074781651 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f9f3c001470 con 0x7f9f4c0c8da0 2024-01-27T01:06:29.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4a59c700 1 -- 172.21.15.46:0/2074781651 >> v1:172.21.15.46:6789/0 conn(0x7f9f4c0c8da0 legacy=0x7f9f4c140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:29.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4a59c700 1 --2- 172.21.15.46:0/2074781651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f38003bf0 0x7f9f38003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4a59c700 1 --2- 172.21.15.46:0/2074781651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f38004810 0x7f9f38004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4a59c700 1 --2- 172.21.15.46:0/2074781651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f380051c0 0x7f9f3800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4a59c700 1 -- 172.21.15.46:0/2074781651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c0cbbf0 con 0x7f9f38004810 2024-01-27T01:06:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4a59c700 1 -- 172.21.15.46:0/2074781651 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3800c160 con 0x7f9f38003bf0 2024-01-27T01:06:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4a59c700 1 -- 172.21.15.46:0/2074781651 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3800c320 con 0x7f9f380051c0 2024-01-27T01:06:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f43fff700 1 --2- 172.21.15.46:0/2074781651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f38004810 0x7f9f38004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4ad9d700 1 --2- 172.21.15.46:0/2074781651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f38003bf0 0x7f9f38003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.033+0000 7f9f4b59e700 1 --2- 172.21.15.46:0/2074781651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f380051c0 0x7f9f3800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.034+0000 7f9f43fff700 1 -- 172.21.15.46:0/2074781651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f380051c0 msgr2=0x7f9f3800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.034+0000 7f9f43fff700 1 --2- 172.21.15.46:0/2074781651 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f380051c0 0x7f9f3800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.034+0000 7f9f43fff700 1 -- 172.21.15.46:0/2074781651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f38003bf0 msgr2=0x7f9f38003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.034+0000 7f9f43fff700 1 --2- 172.21.15.46:0/2074781651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f38003bf0 0x7f9f38003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.034+0000 7f9f43fff700 1 -- 172.21.15.46:0/2074781651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f38003570 con 0x7f9f38004810 2024-01-27T01:06:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.034+0000 7f9f4b59e700 1 -- 172.21.15.46:0/2074781651 reap_dead start 2024-01-27T01:06:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.034+0000 7f9f43fff700 1 --2- 172.21.15.46:0/2074781651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f38004810 0x7f9f38004c80 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f9f340019e0 tx=0x7f9f34007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f5c26e698a7b8e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.034+0000 7f9f4a59c700 1 -- 172.21.15.46:0/2074781651 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3400c590 con 0x7f9f38004810 2024-01-27T01:06:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.034+0000 7f9f4a59c700 1 -- 172.21.15.46:0/2074781651 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3400c6f0 con 0x7f9f38004810 2024-01-27T01:06:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.034+0000 7f9f4a59c700 1 -- 172.21.15.46:0/2074781651 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3400ca90 con 0x7f9f38004810 2024-01-27T01:06:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.034+0000 7f9f5112c700 1 -- 172.21.15.46:0/2074781651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f38004810 msgr2=0x7f9f38004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.034+0000 7f9f5112c700 1 --2- 172.21.15.46:0/2074781651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f38004810 0x7f9f38004c80 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f9f340019e0 tx=0x7f9f34007d30 comp rx=0 tx=0).stop 2024-01-27T01:06:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.034+0000 7f9f5112c700 1 -- 172.21.15.46:0/2074781651 shutdown_connections 2024-01-27T01:06:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.034+0000 7f9f5112c700 1 --2- 172.21.15.46:0/2074781651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f38004810 0x7f9f38004c80 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.034+0000 7f9f5112c700 1 -- 172.21.15.46:0/2074781651 >> 172.21.15.46:0/2074781651 conn(0x7f9f4c06bd90 msgr2=0x7f9f4c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.035+0000 7f9f5112c700 1 -- 172.21.15.46:0/2074781651 shutdown_connections 2024-01-27T01:06:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.035+0000 7f9f5112c700 1 -- 172.21.15.46:0/2074781651 wait complete. 2024-01-27T01:06:29.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.036+0000 7f9f5112c700 1 Processor -- start 2024-01-27T01:06:29.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.036+0000 7f9f5112c700 1 -- start start 2024-01-27T01:06:29.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f5112c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f38004810 0x7f9f4c0b2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:29.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f5112c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f4c0c6030 0x7f9f4c0b1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:29.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f5112c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f4c1570c0 0x7f9f4c13bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:29.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f5112c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c0c6af0 con 0x7f9f38004810 2024-01-27T01:06:29.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f5112c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c0c5a10 con 0x7f9f4c0c6030 2024-01-27T01:06:29.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f5112c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c0b1850 con 0x7f9f4c1570c0 2024-01-27T01:06:29.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f4ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f38004810 0x7f9f4c0b2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:29.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f4ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f38004810 0x7f9f4c0b2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44436/0 (socket says 172.21.15.46:44436) 2024-01-27T01:06:29.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f4ad9d700 1 -- 172.21.15.46:0/3481891380 learned_addr learned my addr 172.21.15.46:0/3481891380 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:29.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f4b59e700 1 --2- 172.21.15.46:0/3481891380 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f4c1570c0 0x7f9f4c13bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:29.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f43fff700 1 --2- 172.21.15.46:0/3481891380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f4c0c6030 0x7f9f4c0b1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:29.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f4ad9d700 1 -- 172.21.15.46:0/3481891380 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f4c1570c0 msgr2=0x7f9f4c13bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:29.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f4ad9d700 1 --2- 172.21.15.46:0/3481891380 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f4c1570c0 0x7f9f4c13bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:29.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f4ad9d700 1 -- 172.21.15.46:0/3481891380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f4c0c6030 msgr2=0x7f9f4c0b1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:29.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f4ad9d700 1 --2- 172.21.15.46:0/3481891380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f4c0c6030 0x7f9f4c0b1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:29.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f4ad9d700 1 -- 172.21.15.46:0/3481891380 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c0c6530 con 0x7f9f38004810 2024-01-27T01:06:29.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f4ad9d700 1 --2- 172.21.15.46:0/3481891380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f38004810 0x7f9f4c0b2540 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c000e10 tx=0x7f9f4c0c96b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:29.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f42ffd700 1 -- 172.21.15.46:0/3481891380 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c018030 con 0x7f9f38004810 2024-01-27T01:06:29.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f42ffd700 1 -- 172.21.15.46:0/3481891380 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3c0077d0 con 0x7f9f38004810 2024-01-27T01:06:29.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f5112c700 1 -- 172.21.15.46:0/3481891380 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f4c0c6810 con 0x7f9f38004810 2024-01-27T01:06:29.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f42ffd700 1 -- 172.21.15.46:0/3481891380 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c00fa80 con 0x7f9f38004810 2024-01-27T01:06:29.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.037+0000 7f9f5112c700 1 -- 172.21.15.46:0/3481891380 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f4c0b2a80 con 0x7f9f38004810 2024-01-27T01:06:29.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.038+0000 7f9f5112c700 1 -- 172.21.15.46:0/3481891380 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f4c0b1340 con 0x7f9f38004810 2024-01-27T01:06:29.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.039+0000 7f9f42ffd700 1 -- 172.21.15.46:0/3481891380 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9f3c01b070 con 0x7f9f38004810 2024-01-27T01:06:29.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.039+0000 7f9f42ffd700 1 --2- 172.21.15.46:0/3481891380 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9f2c066f30 0x7f9f2c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:29.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.040+0000 7f9f42ffd700 1 -- 172.21.15.46:0/3481891380 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f9f3c019070 con 0x7f9f38004810 2024-01-27T01:06:29.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.040+0000 7f9f43fff700 1 --2- 172.21.15.46:0/3481891380 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9f2c066f30 0x7f9f2c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:29.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.040+0000 7f9f43fff700 1 --2- 172.21.15.46:0/3481891380 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9f2c066f30 0x7f9f2c0693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9f34000c00 tx=0x7f9f34015000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:29.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.044+0000 7f9f42ffd700 1 -- 172.21.15.46:0/3481891380 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9f4c0b1340 con 0x7f9f38004810 2024-01-27T01:06:29.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.298+0000 7f9f5112c700 1 -- 172.21.15.46:0/3481891380 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f9f4c0b1340 con 0x7f9f38004810 2024-01-27T01:06:29.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.879+0000 7f9f42ffd700 1 -- 172.21.15.46:0/3481891380 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v116) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9f4c0b1340 con 0x7f9f38004810 2024-01-27T01:06:29.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.882+0000 7f9f5112c700 1 -- 172.21.15.46:0/3481891380 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9f2c066f30 msgr2=0x7f9f2c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:29.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.882+0000 7f9f5112c700 1 --2- 172.21.15.46:0/3481891380 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9f2c066f30 0x7f9f2c0693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9f34000c00 tx=0x7f9f34015000 comp rx=0 tx=0).stop 2024-01-27T01:06:29.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.882+0000 7f9f5112c700 1 -- 172.21.15.46:0/3481891380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f38004810 msgr2=0x7f9f4c0b2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.882+0000 7f9f5112c700 1 --2- 172.21.15.46:0/3481891380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f38004810 0x7f9f4c0b2540 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c000e10 tx=0x7f9f4c0c96b0 comp rx=0 tx=0).stop 2024-01-27T01:06:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.882+0000 7f9f5112c700 1 -- 172.21.15.46:0/3481891380 shutdown_connections 2024-01-27T01:06:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.882+0000 7f9f5112c700 1 --2- 172.21.15.46:0/3481891380 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9f2c066f30 0x7f9f2c0693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.882+0000 7f9f5112c700 1 --2- 172.21.15.46:0/3481891380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f38004810 0x7f9f4c0b2540 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.882+0000 7f9f5112c700 1 --2- 172.21.15.46:0/3481891380 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9f4c1570c0 0x7f9f4c13bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.882+0000 7f9f5112c700 1 --2- 172.21.15.46:0/3481891380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f4c0c6030 0x7f9f4c0b1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.882+0000 7f9f5112c700 1 -- 172.21.15.46:0/3481891380 >> 172.21.15.46:0/3481891380 conn(0x7f9f4c06bd90 msgr2=0x7f9f4c0c3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.882+0000 7f9f5112c700 1 -- 172.21.15.46:0/3481891380 shutdown_connections 2024-01-27T01:06:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:29.882+0000 7f9f5112c700 1 -- 172.21.15.46:0/3481891380 wait complete. 2024-01-27T01:06:29.897 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:06:30.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.034+0000 7f532fdcc700 1 Processor -- start 2024-01-27T01:06:30.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.034+0000 7f532fdcc700 1 -- start start 2024-01-27T01:06:30.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.035+0000 7f532fdcc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5328157340 0x7f532815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:30.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.035+0000 7f532fdcc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f53280b4180 0x7f5328157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:30.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.035+0000 7f532fdcc700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5328158ae0 con 0x7f53280ae520 2024-01-27T01:06:30.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.035+0000 7f532fdcc700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f532815b650 con 0x7f5328157340 2024-01-27T01:06:30.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.035+0000 7f532fdcc700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f532815c020 con 0x7f53280b4180 2024-01-27T01:06:30.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.035+0000 7f532db68700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5328157340 0x7f532815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:30.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.035+0000 7f532d367700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f53280ae520 0x7f53280b3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53646/0 (socket says 172.21.15.46:53646) 2024-01-27T01:06:30.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.035+0000 7f532d367700 1 -- 172.21.15.46:0/2564184823 learned_addr learned my addr 172.21.15.46:0/2564184823 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:30.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.035+0000 7f532e369700 1 --2- 172.21.15.46:0/2564184823 >> v2:172.21.15.106:3300/0 conn(0x7f53280b4180 0x7f5328157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:30.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.035+0000 7f532db68700 1 -- 172.21.15.46:0/2564184823 >> v2:172.21.15.106:3300/0 conn(0x7f53280b4180 msgr2=0x7f5328157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:30.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.035+0000 7f532db68700 1 --2- 172.21.15.46:0/2564184823 >> v2:172.21.15.106:3300/0 conn(0x7f53280b4180 0x7f5328157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.035+0000 7f532db68700 1 -- 172.21.15.46:0/2564184823 >> v1:172.21.15.46:6789/0 conn(0x7f53280ae520 legacy=0x7f53280b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:30.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.035+0000 7f532db68700 1 -- 172.21.15.46:0/2564184823 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5328158520 con 0x7f5328157340 2024-01-27T01:06:30.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.035+0000 7f532cb66700 1 -- 172.21.15.46:0/2564184823 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466397090 0 0) 0x7f5328158ae0 con 0x7f53280ae520 2024-01-27T01:06:30.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.036+0000 7f532e369700 1 --2- 172.21.15.46:0/2564184823 >> v2:172.21.15.106:3300/0 conn(0x7f53280b4180 0x7f5328157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:06:30.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.036+0000 7f532db68700 1 --2- 172.21.15.46:0/2564184823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5328157340 0x7f532815bae0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f53280caeb0 tx=0x7f53180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4af27a93befa7f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:30.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.036+0000 7f532cb66700 1 -- 172.21.15.46:0/2564184823 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531800b6b0 con 0x7f5328157340 2024-01-27T01:06:30.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.036+0000 7f532cb66700 1 -- 172.21.15.46:0/2564184823 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531800b810 con 0x7f5328157340 2024-01-27T01:06:30.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.036+0000 7f532cb66700 1 -- 172.21.15.46:0/2564184823 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531800bbb0 con 0x7f5328157340 2024-01-27T01:06:30.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.036+0000 7f532fdcc700 1 -- 172.21.15.46:0/2564184823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5328157340 msgr2=0x7f532815bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:30.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.036+0000 7f532fdcc700 1 --2- 172.21.15.46:0/2564184823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5328157340 0x7f532815bae0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f53280caeb0 tx=0x7f53180091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.036+0000 7f532fdcc700 1 -- 172.21.15.46:0/2564184823 shutdown_connections 2024-01-27T01:06:30.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.036+0000 7f532fdcc700 1 --2- 172.21.15.46:0/2564184823 >> v2:172.21.15.106:3300/0 conn(0x7f53280b4180 0x7f5328157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.036+0000 7f532fdcc700 1 --2- 172.21.15.46:0/2564184823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5328157340 0x7f532815bae0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.036+0000 7f532fdcc700 1 -- 172.21.15.46:0/2564184823 >> 172.21.15.46:0/2564184823 conn(0x7f53280ce530 msgr2=0x7f53280ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:30.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.037+0000 7f532fdcc700 1 -- 172.21.15.46:0/2564184823 shutdown_connections 2024-01-27T01:06:30.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.037+0000 7f532fdcc700 1 -- 172.21.15.46:0/2564184823 wait complete. 2024-01-27T01:06:30.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.038+0000 7f532fdcc700 1 Processor -- start 2024-01-27T01:06:30.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.038+0000 7f532fdcc700 1 -- start start 2024-01-27T01:06:30.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532fdcc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53280ae520 0x7f53281568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:30.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532fdcc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53280b4180 0x7f53280c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:30.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532fdcc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5328157340 0x7f53280ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:30.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532fdcc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53280b1510 con 0x7f5328157340 2024-01-27T01:06:30.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532fdcc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532806a280 con 0x7f53280b4180 2024-01-27T01:06:30.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532fdcc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532806a560 con 0x7f53280ae520 2024-01-27T01:06:30.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532e369700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5328157340 0x7f53280ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:30.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532e369700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5328157340 0x7f53280ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44464/0 (socket says 172.21.15.46:44464) 2024-01-27T01:06:30.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532e369700 1 -- 172.21.15.46:0/3805458575 learned_addr learned my addr 172.21.15.46:0/3805458575 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:30.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532d367700 1 --2- 172.21.15.46:0/3805458575 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53280b4180 0x7f53280c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:30.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532db68700 1 --2- 172.21.15.46:0/3805458575 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53280ae520 0x7f53281568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:30.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532e369700 1 -- 172.21.15.46:0/3805458575 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53280ae520 msgr2=0x7f53281568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:30.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532e369700 1 --2- 172.21.15.46:0/3805458575 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53280ae520 0x7f53281568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532e369700 1 -- 172.21.15.46:0/3805458575 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53280b4180 msgr2=0x7f53280c9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:30.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532e369700 1 --2- 172.21.15.46:0/3805458575 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53280b4180 0x7f53280c9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532e369700 1 -- 172.21.15.46:0/3805458575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5318009420 con 0x7f5328157340 2024-01-27T01:06:30.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532db68700 1 --2- 172.21.15.46:0/3805458575 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53280ae520 0x7f53281568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:30.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.039+0000 7f532e369700 1 --2- 172.21.15.46:0/3805458575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5328157340 0x7f53280ca4f0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f532000cef0 tx=0x7f5320007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:30.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.040+0000 7f531dffb700 1 -- 172.21.15.46:0/3805458575 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532000add0 con 0x7f5328157340 2024-01-27T01:06:30.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.040+0000 7f531dffb700 1 -- 172.21.15.46:0/3805458575 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5320016420 con 0x7f5328157340 2024-01-27T01:06:30.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.040+0000 7f531dffb700 1 -- 172.21.15.46:0/3805458575 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53200167e0 con 0x7f5328157340 2024-01-27T01:06:30.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.040+0000 7f532fdcc700 1 -- 172.21.15.46:0/3805458575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532806a840 con 0x7f5328157340 2024-01-27T01:06:30.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.040+0000 7f532fdcc700 1 -- 172.21.15.46:0/3805458575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5328156df0 con 0x7f5328157340 2024-01-27T01:06:30.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.041+0000 7f532fdcc700 1 -- 172.21.15.46:0/3805458575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53280b12e0 con 0x7f5328157340 2024-01-27T01:06:30.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.041+0000 7f531dffb700 1 -- 172.21.15.46:0/3805458575 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5320016bd0 con 0x7f5328157340 2024-01-27T01:06:30.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.041+0000 7f531dffb700 1 --2- 172.21.15.46:0/3805458575 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f530c066f30 0x7f530c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:30.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.042+0000 7f531dffb700 1 -- 172.21.15.46:0/3805458575 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f5320012070 con 0x7f5328157340 2024-01-27T01:06:30.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.042+0000 7f532db68700 1 --2- 172.21.15.46:0/3805458575 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f530c066f30 0x7f530c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:30.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.042+0000 7f532db68700 1 --2- 172.21.15.46:0/3805458575 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f530c066f30 0x7f530c0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f53180093f0 tx=0x7f5318017000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:30.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.046+0000 7f531dffb700 1 -- 172.21.15.46:0/3805458575 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f532004e5a0 con 0x7f5328157340 2024-01-27T01:06:30.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.298+0000 7f532fdcc700 1 -- 172.21.15.46:0/3805458575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f532806e430 con 0x7f5328157340 2024-01-27T01:06:30.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.299+0000 7f531dffb700 1 -- 172.21.15.46:0/3805458575 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+4618 (secure 0 0 0) 0x7f532806e430 con 0x7f5328157340 2024-01-27T01:06:30.298 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:30.298 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":116,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/1999300009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1999300009},{"type":"v1","addr":"172.21.15.106:6835","nonce":1999300009}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":110},{"gid":10486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/4174263148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4174263148},{"type":"v1","addr":"172.21.15.106:6833","nonce":4174263148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":10854,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1263484152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1263484152},{"type":"v1","addr":"172.21.15.46:6833","nonce":1263484152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:06:26.286041+0000","modified":"2024-01-27T01:06:29.873402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10866},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10866":{"gid":10866,"name":"a","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2483163449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2483163449},{"type":"v1","addr":"172.21.15.46:6835","nonce":2483163449}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2024-01-27T01:06:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.301+0000 7f532fdcc700 1 -- 172.21.15.46:0/3805458575 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f530c066f30 msgr2=0x7f530c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.301+0000 7f532fdcc700 1 --2- 172.21.15.46:0/3805458575 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f530c066f30 0x7f530c0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f53180093f0 tx=0x7f5318017000 comp rx=0 tx=0).stop 2024-01-27T01:06:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.301+0000 7f532fdcc700 1 -- 172.21.15.46:0/3805458575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5328157340 msgr2=0x7f53280ca4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.301+0000 7f532fdcc700 1 --2- 172.21.15.46:0/3805458575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5328157340 0x7f53280ca4f0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f532000cef0 tx=0x7f5320007860 comp rx=0 tx=0).stop 2024-01-27T01:06:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.302+0000 7f532fdcc700 1 -- 172.21.15.46:0/3805458575 shutdown_connections 2024-01-27T01:06:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.302+0000 7f532fdcc700 1 --2- 172.21.15.46:0/3805458575 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f530c066f30 0x7f530c0693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.302+0000 7f532fdcc700 1 --2- 172.21.15.46:0/3805458575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5328157340 0x7f53280ca4f0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.302+0000 7f532fdcc700 1 --2- 172.21.15.46:0/3805458575 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53280ae520 0x7f53281568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.302+0000 7f532fdcc700 1 --2- 172.21.15.46:0/3805458575 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53280b4180 0x7f53280c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.302+0000 7f532fdcc700 1 -- 172.21.15.46:0/3805458575 >> 172.21.15.46:0/3805458575 conn(0x7f53280ce530 msgr2=0x7f5328157710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.302+0000 7f532fdcc700 1 -- 172.21.15.46:0/3805458575 shutdown_connections 2024-01-27T01:06:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.302+0000 7f532fdcc700 1 -- 172.21.15.46:0/3805458575 wait complete. 2024-01-27T01:06:30.302 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 116 2024-01-27T01:06:30.314 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:06:30.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.447+0000 7f0dd37d3700 1 Processor -- start 2024-01-27T01:06:30.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.447+0000 7f0dd37d3700 1 -- start start 2024-01-27T01:06:30.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.448+0000 7f0dd37d3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0dcc0ccc20 0x7f0dcc068340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.448+0000 7f0dd37d3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dcc0ccff0 0x7f0dcc06eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.448+0000 7f0dd37d3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dcc158020 con 0x7f0dcc0b2590 2024-01-27T01:06:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.448+0000 7f0dd37d3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dcc158300 con 0x7f0dcc0ccc20 2024-01-27T01:06:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.448+0000 7f0dd37d3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc1585e0 con 0x7f0dcc0ccff0 2024-01-27T01:06:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.448+0000 7f0dd156f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0dcc0ccc20 0x7f0dcc068340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.448+0000 7f0dd156f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0dcc0ccc20 0x7f0dcc068340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33692/0 (socket says 172.21.15.46:33692) 2024-01-27T01:06:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.448+0000 7f0dd156f700 1 -- 172.21.15.46:0/224198191 learned_addr learned my addr 172.21.15.46:0/224198191 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.448+0000 7f0dd0d6e700 1 --2- 172.21.15.46:0/224198191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dcc0ccff0 0x7f0dcc06eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.449+0000 7f0dd156f700 1 -- 172.21.15.46:0/224198191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dcc0ccff0 msgr2=0x7f0dcc06eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.449+0000 7f0dd156f700 1 --2- 172.21.15.46:0/224198191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dcc0ccff0 0x7f0dcc06eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.449+0000 7f0dd156f700 1 -- 172.21.15.46:0/224198191 >> v1:172.21.15.106:6789/0 conn(0x7f0dcc0b2590 legacy=0x7f0dcc15b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.449+0000 7f0dd156f700 1 -- 172.21.15.46:0/224198191 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc1588c0 con 0x7f0dcc0ccc20 2024-01-27T01:06:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.449+0000 7f0dd156f700 1 --2- 172.21.15.46:0/224198191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dcc0ccc20 0x7f0dcc068340 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f0dc80099b0 tx=0x7f0dc8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76782c09bd49664 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:30.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.449+0000 7f0dc3fff700 1 -- 172.21.15.46:0/224198191 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8009ea0 con 0x7f0dcc0ccc20 2024-01-27T01:06:30.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.449+0000 7f0dc3fff700 1 -- 172.21.15.46:0/224198191 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc800b860 con 0x7f0dcc0ccc20 2024-01-27T01:06:30.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.449+0000 7f0dc3fff700 1 -- 172.21.15.46:0/224198191 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc800bc00 con 0x7f0dcc0ccc20 2024-01-27T01:06:30.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.449+0000 7f0dd37d3700 1 -- 172.21.15.46:0/224198191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dcc0ccc20 msgr2=0x7f0dcc068340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:30.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.449+0000 7f0dd37d3700 1 --2- 172.21.15.46:0/224198191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dcc0ccc20 0x7f0dcc068340 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f0dc80099b0 tx=0x7f0dc8009250 comp rx=0 tx=0).stop 2024-01-27T01:06:30.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.449+0000 7f0dd37d3700 1 -- 172.21.15.46:0/224198191 shutdown_connections 2024-01-27T01:06:30.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.450+0000 7f0dd37d3700 1 --2- 172.21.15.46:0/224198191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dcc0ccc20 0x7f0dcc068340 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.450+0000 7f0dd37d3700 1 --2- 172.21.15.46:0/224198191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dcc0ccff0 0x7f0dcc06eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.450+0000 7f0dd37d3700 1 -- 172.21.15.46:0/224198191 >> 172.21.15.46:0/224198191 conn(0x7f0dcc06b630 msgr2=0x7f0dcc0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:30.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.450+0000 7f0dd37d3700 1 -- 172.21.15.46:0/224198191 shutdown_connections 2024-01-27T01:06:30.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.450+0000 7f0dd37d3700 1 -- 172.21.15.46:0/224198191 wait complete. 2024-01-27T01:06:30.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.451+0000 7f0dd37d3700 1 Processor -- start 2024-01-27T01:06:30.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.451+0000 7f0dd37d3700 1 -- start start 2024-01-27T01:06:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.452+0000 7f0dd37d3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dcc0b2590 0x7f0dcc144e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.452+0000 7f0dd37d3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dcc0ccc20 0x7f0dcc0c5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.452+0000 7f0dd37d3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0dcc0ccff0 0x7f0dcc145860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.452+0000 7f0dd37d3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc0c6e00 con 0x7f0dcc0b2590 2024-01-27T01:06:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.452+0000 7f0dd37d3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc0c61f0 con 0x7f0dcc0ccff0 2024-01-27T01:06:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.452+0000 7f0dd37d3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc0c64d0 con 0x7f0dcc0ccc20 2024-01-27T01:06:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.452+0000 7f0dd156f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dcc0b2590 0x7f0dcc144e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.452+0000 7f0dd156f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dcc0b2590 0x7f0dcc144e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33706/0 (socket says 172.21.15.46:33706) 2024-01-27T01:06:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.452+0000 7f0dd156f700 1 -- 172.21.15.46:0/3415558317 learned_addr learned my addr 172.21.15.46:0/3415558317 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.452+0000 7f0dd0d6e700 1 --2- 172.21.15.46:0/3415558317 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dcc0ccc20 0x7f0dcc0c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.452+0000 7f0dd1d70700 1 --2- 172.21.15.46:0/3415558317 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0dcc0ccff0 0x7f0dcc145860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.452+0000 7f0dd156f700 1 -- 172.21.15.46:0/3415558317 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dcc0ccc20 msgr2=0x7f0dcc0c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.452+0000 7f0dd156f700 1 --2- 172.21.15.46:0/3415558317 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dcc0ccc20 0x7f0dcc0c5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.452+0000 7f0dd156f700 1 -- 172.21.15.46:0/3415558317 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0dcc0ccff0 msgr2=0x7f0dcc145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:30.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.452+0000 7f0dd156f700 1 --2- 172.21.15.46:0/3415558317 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0dcc0ccff0 0x7f0dcc145860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.452+0000 7f0dd156f700 1 -- 172.21.15.46:0/3415558317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc80094f0 con 0x7f0dcc0b2590 2024-01-27T01:06:30.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.452+0000 7f0dd0d6e700 1 --2- 172.21.15.46:0/3415558317 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dcc0ccc20 0x7f0dcc0c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:30.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.453+0000 7f0dd156f700 1 --2- 172.21.15.46:0/3415558317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dcc0b2590 0x7f0dcc144e90 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc06de10 tx=0x7f0dc8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:30.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.453+0000 7f0dc17fa700 1 -- 172.21.15.46:0/3415558317 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc800bbe0 con 0x7f0dcc0b2590 2024-01-27T01:06:30.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.453+0000 7f0dc17fa700 1 -- 172.21.15.46:0/3415558317 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc8011420 con 0x7f0dcc0b2590 2024-01-27T01:06:30.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.453+0000 7f0dd37d3700 1 -- 172.21.15.46:0/3415558317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dcc0c67b0 con 0x7f0dcc0b2590 2024-01-27T01:06:30.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.453+0000 7f0dc17fa700 1 -- 172.21.15.46:0/3415558317 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc80117c0 con 0x7f0dcc0b2590 2024-01-27T01:06:30.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.453+0000 7f0dd37d3700 1 -- 172.21.15.46:0/3415558317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dcc1453d0 con 0x7f0dcc0b2590 2024-01-27T01:06:30.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.454+0000 7f0dd37d3700 1 -- 172.21.15.46:0/3415558317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dcc04c710 con 0x7f0dcc0b2590 2024-01-27T01:06:30.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.454+0000 7f0dc17fa700 1 -- 172.21.15.46:0/3415558317 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0dc8011bf0 con 0x7f0dcc0b2590 2024-01-27T01:06:30.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.455+0000 7f0dc17fa700 1 --2- 172.21.15.46:0/3415558317 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0db0066f30 0x7f0db00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:30.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.455+0000 7f0dc17fa700 1 -- 172.21.15.46:0/3415558317 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f0dc80861a0 con 0x7f0dcc0b2590 2024-01-27T01:06:30.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.455+0000 7f0dd0d6e700 1 --2- 172.21.15.46:0/3415558317 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0db0066f30 0x7f0db00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:30.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.455+0000 7f0dd0d6e700 1 --2- 172.21.15.46:0/3415558317 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0db0066f30 0x7f0db00693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc0af610 tx=0x7f0dbc008040 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:30.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.459+0000 7f0dc17fa700 1 -- 172.21.15.46:0/3415558317 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0dc80530e0 con 0x7f0dcc0b2590 2024-01-27T01:06:30.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.664+0000 7f0dd37d3700 1 -- 172.21.15.46:0/3415558317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0dcc06f0b0 con 0x7f0dcc0b2590 2024-01-27T01:06:30.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.665+0000 7f0dc17fa700 1 -- 172.21.15.46:0/3415558317 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v223) v1 ==== 74+0+16325 (secure 0 0 0) 0x7f0dc8056700 con 0x7f0dcc0b2590 2024-01-27T01:06:30.664 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:30.665 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":223,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:06:27.861661+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:06:24.798791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-01-27T01:06:25.249215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:06:30.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.667+0000 7f0dd37d3700 1 -- 172.21.15.46:0/3415558317 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0db0066f30 msgr2=0x7f0db00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:30.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.667+0000 7f0dd37d3700 1 --2- 172.21.15.46:0/3415558317 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0db0066f30 0x7f0db00693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc0af610 tx=0x7f0dbc008040 comp rx=0 tx=0).stop 2024-01-27T01:06:30.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.667+0000 7f0dd37d3700 1 -- 172.21.15.46:0/3415558317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dcc0b2590 msgr2=0x7f0dcc144e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:30.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.667+0000 7f0dd37d3700 1 --2- 172.21.15.46:0/3415558317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dcc0b2590 0x7f0dcc144e90 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc06de10 tx=0x7f0dc8017000 comp rx=0 tx=0).stop 2024-01-27T01:06:30.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.667+0000 7f0dd37d3700 1 -- 172.21.15.46:0/3415558317 shutdown_connections 2024-01-27T01:06:30.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.667+0000 7f0dd37d3700 1 --2- 172.21.15.46:0/3415558317 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0db0066f30 0x7f0db00693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.667+0000 7f0dd37d3700 1 --2- 172.21.15.46:0/3415558317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dcc0b2590 0x7f0dcc144e90 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.667+0000 7f0dd37d3700 1 --2- 172.21.15.46:0/3415558317 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0dcc0ccff0 0x7f0dcc145860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.667+0000 7f0dd37d3700 1 --2- 172.21.15.46:0/3415558317 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dcc0ccc20 0x7f0dcc0c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.667+0000 7f0dd37d3700 1 -- 172.21.15.46:0/3415558317 >> 172.21.15.46:0/3415558317 conn(0x7f0dcc06b630 msgr2=0x7f0dcc0cb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:30.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.667+0000 7f0dd37d3700 1 -- 172.21.15.46:0/3415558317 shutdown_connections 2024-01-27T01:06:30.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.668+0000 7f0dd37d3700 1 -- 172.21.15.46:0/3415558317 wait complete. 2024-01-27T01:06:30.680 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-27T01:06:30.680 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:06:30.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.814+0000 7fce36de4700 1 Processor -- start 2024-01-27T01:06:30.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.814+0000 7fce36de4700 1 -- start start 2024-01-27T01:06:30.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.815+0000 7fce36de4700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fce300b2160 0x7fce3015a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:30.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.815+0000 7fce36de4700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce300a9b90 con 0x7fce3015da10 2024-01-27T01:06:30.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.815+0000 7fce36de4700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce300aa210 con 0x7fce300b2530 2024-01-27T01:06:30.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.815+0000 7fce36de4700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce3015d580 con 0x7fce300b2160 2024-01-27T01:06:30.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.815+0000 7fce35381700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fce3015da10 0x7fce3006a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54130/0 (socket says 172.21.15.46:54130) 2024-01-27T01:06:30.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.815+0000 7fce35381700 1 -- 172.21.15.46:0/1075873662 learned_addr learned my addr 172.21.15.46:0/1075873662 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.815+0000 7fce34b80700 1 --2- 172.21.15.46:0/1075873662 >> v2:172.21.15.106:3300/0 conn(0x7fce300b2160 0x7fce3015a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.815+0000 7fce2f7fe700 1 -- 172.21.15.46:0/1075873662 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894999741 0 0) 0x7fce300a9b90 con 0x7fce3015da10 2024-01-27T01:06:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.816+0000 7fce2f7fe700 1 -- 172.21.15.46:0/1075873662 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce20003570 con 0x7fce3015da10 2024-01-27T01:06:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.816+0000 7fce2f7fe700 1 -- 172.21.15.46:0/1075873662 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334625974 0 0) 0x7fce300aa210 con 0x7fce300b2530 2024-01-27T01:06:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.816+0000 7fce2f7fe700 1 -- 172.21.15.46:0/1075873662 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce300a9b90 con 0x7fce300b2530 2024-01-27T01:06:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.816+0000 7fce34b80700 1 -- 172.21.15.46:0/1075873662 >> v1:172.21.15.106:6789/0 conn(0x7fce300b2530 legacy=0x7fce300abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.816+0000 7fce34b80700 1 -- 172.21.15.46:0/1075873662 >> v1:172.21.15.46:6789/0 conn(0x7fce3015da10 legacy=0x7fce3006a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.816+0000 7fce34b80700 1 -- 172.21.15.46:0/1075873662 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce300aa460 con 0x7fce300b2160 2024-01-27T01:06:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.816+0000 7fce34b80700 1 --2- 172.21.15.46:0/1075873662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce300b2160 0x7fce3015a7a0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fce240098e0 tx=0x7fce240091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2b9df54270644fb server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.816+0000 7fce2f7fe700 1 -- 172.21.15.46:0/1075873662 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24011630 con 0x7fce300b2160 2024-01-27T01:06:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.816+0000 7fce2f7fe700 1 -- 172.21.15.46:0/1075873662 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce24011790 con 0x7fce300b2160 2024-01-27T01:06:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.816+0000 7fce2f7fe700 1 -- 172.21.15.46:0/1075873662 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24011b30 con 0x7fce300b2160 2024-01-27T01:06:30.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.817+0000 7fce36de4700 1 -- 172.21.15.46:0/1075873662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce300b2160 msgr2=0x7fce3015a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:30.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.817+0000 7fce36de4700 1 --2- 172.21.15.46:0/1075873662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce300b2160 0x7fce3015a7a0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fce240098e0 tx=0x7fce240091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.817+0000 7fce36de4700 1 -- 172.21.15.46:0/1075873662 shutdown_connections 2024-01-27T01:06:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.817+0000 7fce36de4700 1 --2- 172.21.15.46:0/1075873662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce300b2160 0x7fce3015a7a0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.817+0000 7fce36de4700 1 -- 172.21.15.46:0/1075873662 >> 172.21.15.46:0/1075873662 conn(0x7fce300ccbc0 msgr2=0x7fce300ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.817+0000 7fce36de4700 1 -- 172.21.15.46:0/1075873662 shutdown_connections 2024-01-27T01:06:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.817+0000 7fce36de4700 1 -- 172.21.15.46:0/1075873662 wait complete. 2024-01-27T01:06:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.818+0000 7fce36de4700 1 Processor -- start 2024-01-27T01:06:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.818+0000 7fce36de4700 1 -- start start 2024-01-27T01:06:30.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.819+0000 7fce36de4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce300b2160 0x7fce3014bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:30.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.819+0000 7fce36de4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce300b2530 0x7fce3013d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:30.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.819+0000 7fce36de4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce3015da10 0x7fce30150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:30.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.819+0000 7fce36de4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce300acb60 con 0x7fce300b2160 2024-01-27T01:06:30.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.819+0000 7fce36de4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce300ace40 con 0x7fce300b2530 2024-01-27T01:06:30.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.819+0000 7fce36de4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce300ad120 con 0x7fce3015da10 2024-01-27T01:06:30.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.819+0000 7fce34b80700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce300b2160 0x7fce3014bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:30.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.819+0000 7fce34b80700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce300b2160 0x7fce3014bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33718/0 (socket says 172.21.15.46:33718) 2024-01-27T01:06:30.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.819+0000 7fce34b80700 1 -- 172.21.15.46:0/2708094903 learned_addr learned my addr 172.21.15.46:0/2708094903 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:30.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.819+0000 7fce35381700 1 --2- 172.21.15.46:0/2708094903 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce3015da10 0x7fce30150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:30.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.819+0000 7fce2ffff700 1 --2- 172.21.15.46:0/2708094903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce300b2530 0x7fce3013d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:30.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.819+0000 7fce34b80700 1 -- 172.21.15.46:0/2708094903 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce3015da10 msgr2=0x7fce30150140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:30.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.819+0000 7fce34b80700 1 --2- 172.21.15.46:0/2708094903 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce3015da10 0x7fce30150140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.819+0000 7fce34b80700 1 -- 172.21.15.46:0/2708094903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce300b2530 msgr2=0x7fce3013d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:30.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.819+0000 7fce34b80700 1 --2- 172.21.15.46:0/2708094903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce300b2530 0x7fce3013d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:30.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.820+0000 7fce34b80700 1 -- 172.21.15.46:0/2708094903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce24009420 con 0x7fce300b2160 2024-01-27T01:06:30.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.820+0000 7fce34b80700 1 --2- 172.21.15.46:0/2708094903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce300b2160 0x7fce3014bbe0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fce3006f610 tx=0x7fce24017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:30.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.820+0000 7fce2cff9700 1 -- 172.21.15.46:0/2708094903 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2401f070 con 0x7fce300b2160 2024-01-27T01:06:30.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.820+0000 7fce2cff9700 1 -- 172.21.15.46:0/2708094903 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce24000c90 con 0x7fce300b2160 2024-01-27T01:06:30.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.820+0000 7fce36de4700 1 -- 172.21.15.46:0/2708094903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce300ad400 con 0x7fce300b2160 2024-01-27T01:06:30.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.820+0000 7fce2cff9700 1 -- 172.21.15.46:0/2708094903 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24011b30 con 0x7fce300b2160 2024-01-27T01:06:30.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.820+0000 7fce36de4700 1 -- 172.21.15.46:0/2708094903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce3013dac0 con 0x7fce300b2160 2024-01-27T01:06:30.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.821+0000 7fce36de4700 1 -- 172.21.15.46:0/2708094903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3004c710 con 0x7fce300b2160 2024-01-27T01:06:30.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.821+0000 7fce2cff9700 1 -- 172.21.15.46:0/2708094903 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fce24023070 con 0x7fce300b2160 2024-01-27T01:06:30.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.822+0000 7fce2cff9700 1 --2- 172.21.15.46:0/2708094903 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fce14066f30 0x7fce140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:30.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.822+0000 7fce2cff9700 1 -- 172.21.15.46:0/2708094903 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fce24085910 con 0x7fce300b2160 2024-01-27T01:06:30.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.822+0000 7fce2ffff700 1 --2- 172.21.15.46:0/2708094903 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fce14066f30 0x7fce140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:30.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.822+0000 7fce2ffff700 1 --2- 172.21.15.46:0/2708094903 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fce14066f30 0x7fce140693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fce1c003870 tx=0x7fce1c009040 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:30.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:30.827+0000 7fce2cff9700 1 -- 172.21.15.46:0/2708094903 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fce24052900 con 0x7fce300b2160 2024-01-27T01:06:31.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.029+0000 7fce36de4700 1 -- 172.21.15.46:0/2708094903 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fce300ad5f0 con 0x7fce14066f30 2024-01-27T01:06:31.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.039+0000 7fce2cff9700 1 -- 172.21.15.46:0/2708094903 <== mgr.9101 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289869 (secure 0 0 0) 0x7fce300ad5f0 con 0x7fce14066f30 2024-01-27T01:06:31.042 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:31.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.043+0000 7fce36de4700 1 -- 172.21.15.46:0/2708094903 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fce14066f30 msgr2=0x7fce140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:31.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.043+0000 7fce36de4700 1 --2- 172.21.15.46:0/2708094903 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fce14066f30 0x7fce140693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fce1c003870 tx=0x7fce1c009040 comp rx=0 tx=0).stop 2024-01-27T01:06:31.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.043+0000 7fce36de4700 1 -- 172.21.15.46:0/2708094903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce300b2160 msgr2=0x7fce3014bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:31.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.043+0000 7fce36de4700 1 --2- 172.21.15.46:0/2708094903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce300b2160 0x7fce3014bbe0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fce3006f610 tx=0x7fce24017000 comp rx=0 tx=0).stop 2024-01-27T01:06:31.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.043+0000 7fce36de4700 1 -- 172.21.15.46:0/2708094903 shutdown_connections 2024-01-27T01:06:31.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.043+0000 7fce36de4700 1 --2- 172.21.15.46:0/2708094903 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fce14066f30 0x7fce140693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.043+0000 7fce36de4700 1 --2- 172.21.15.46:0/2708094903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce300b2160 0x7fce3014bbe0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.043+0000 7fce36de4700 1 --2- 172.21.15.46:0/2708094903 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce3015da10 0x7fce30150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.043+0000 7fce36de4700 1 --2- 172.21.15.46:0/2708094903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce300b2530 0x7fce3013d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.043+0000 7fce36de4700 1 -- 172.21.15.46:0/2708094903 >> 172.21.15.46:0/2708094903 conn(0x7fce300ccbc0 msgr2=0x7fce30152060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.043+0000 7fce36de4700 1 -- 172.21.15.46:0/2708094903 shutdown_connections 2024-01-27T01:06:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.043+0000 7fce36de4700 1 -- 172.21.15.46:0/2708094903 wait complete. 2024-01-27T01:06:31.045 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:06:31.055 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-01-27T01:06:30.730734+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":674,"num_write_kb":3475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8788,"ondisk_log_size":8788,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2466952,"kb_used_data":23112,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752507768,"statfs":{"total":773094113280,"available":770567954432,"internally_reserved":0,"allocated":23666688,"data_stored":12975028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.889576"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.878856+0000","last_change":"2024-01-27T01:06:26.873715+0000","last_active":"2024-01-27T01:06:26.878856+0000","last_peered":"2024-01-27T01:06:26.878856+0000","last_clean":"2024-01-27T01:06:26.878856+0000","last_became_active":"2024-01-27T01:06:26.873430+0000","last_became_peered":"2024-01-27T01:06:26.873430+0000","last_unstale":"2024-01-27T01:06:26.878856+0000","last_undegraded":"2024-01-27T01:06:26.878856+0000","last_fullsized":"2024-01-27T01:06:26.878856+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:44.342963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.875217+0000","last_change":"2024-01-27T01:06:26.871135+0000","last_active":"2024-01-27T01:06:26.875217+0000","last_peered":"2024-01-27T01:06:26.875217+0000","last_clean":"2024-01-27T01:06:26.875217+0000","last_became_active":"2024-01-27T01:06:26.870854+0000","last_became_peered":"2024-01-27T01:06:26.870854+0000","last_unstale":"2024-01-27T01:06:26.875217+0000","last_undegraded":"2024-01-27T01:06:26.875217+0000","last_fullsized":"2024-01-27T01:06:26.875217+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:12:59.875769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.878303+0000","last_change":"2024-01-27T01:06:26.868462+0000","last_active":"2024-01-27T01:06:26.878303+0000","last_peered":"2024-01-27T01:06:26.878303+0000","last_clean":"2024-01-27T01:06:26.878303+0000","last_became_active":"2024-01-27T01:06:26.868203+0000","last_became_peered":"2024-01-27T01:06:26.868203+0000","last_unstale":"2024-01-27T01:06:26.878303+0000","last_undegraded":"2024-01-27T01:06:26.878303+0000","last_fullsized":"2024-01-27T01:06:26.878303+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:10.965469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.873487+0000","last_change":"2024-01-27T01:06:26.868177+0000","last_active":"2024-01-27T01:06:26.873487+0000","last_peered":"2024-01-27T01:06:26.873487+0000","last_clean":"2024-01-27T01:06:26.873487+0000","last_became_active":"2024-01-27T01:06:26.867906+0000","last_became_peered":"2024-01-27T01:06:26.867906+0000","last_unstale":"2024-01-27T01:06:26.873487+0000","last_undegraded":"2024-01-27T01:06:26.873487+0000","last_fullsized":"2024-01-27T01:06:26.873487+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:32.271904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.875579+0000","last_change":"2024-01-27T01:06:26.870422+0000","last_active":"2024-01-27T01:06:26.875579+0000","last_peered":"2024-01-27T01:06:26.875579+0000","last_clean":"2024-01-27T01:06:26.875579+0000","last_became_active":"2024-01-27T01:06:26.870117+0000","last_became_peered":"2024-01-27T01:06:26.870117+0000","last_unstale":"2024-01-27T01:06:26.875579+0000","last_undegraded":"2024-01-27T01:06:26.875579+0000","last_fullsized":"2024-01-27T01:06:26.875579+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:42:55.987157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.883527+0000","last_change":"2024-01-27T01:06:26.879291+0000","last_active":"2024-01-27T01:06:26.883527+0000","last_peered":"2024-01-27T01:06:26.883527+0000","last_clean":"2024-01-27T01:06:26.883527+0000","last_became_active":"2024-01-27T01:06:26.879063+0000","last_became_peered":"2024-01-27T01:06:26.879063+0000","last_unstale":"2024-01-27T01:06:26.883527+0000","last_undegraded":"2024-01-27T01:06:26.883527+0000","last_fullsized":"2024-01-27T01:06:26.883527+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:22:23.545688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.878514+0000","last_change":"2024-01-27T01:06:26.872094+0000","last_active":"2024-01-27T01:06:26.878514+0000","last_peered":"2024-01-27T01:06:26.878514+0000","last_clean":"2024-01-27T01:06:26.878514+0000","last_became_active":"2024-01-27T01:06:26.871711+0000","last_became_peered":"2024-01-27T01:06:26.871711+0000","last_unstale":"2024-01-27T01:06:26.878514+0000","last_undegraded":"2024-01-27T01:06:26.878514+0000","last_fullsized":"2024-01-27T01:06:26.878514+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:24:32.821570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884998+0000","last_change":"2024-01-27T01:06:26.881786+0000","last_active":"2024-01-27T01:06:26.884998+0000","last_peered":"2024-01-27T01:06:26.884998+0000","last_clean":"2024-01-27T01:06:26.884998+0000","last_became_active":"2024-01-27T01:06:26.881527+0000","last_became_peered":"2024-01-27T01:06:26.881527+0000","last_unstale":"2024-01-27T01:06:26.884998+0000","last_undegraded":"2024-01-27T01:06:26.884998+0000","last_fullsized":"2024-01-27T01:06:26.884998+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:43.630831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.881095+0000","last_change":"2024-01-27T01:06:26.873409+0000","last_active":"2024-01-27T01:06:26.881095+0000","last_peered":"2024-01-27T01:06:26.881095+0000","last_clean":"2024-01-27T01:06:26.881095+0000","last_became_active":"2024-01-27T01:06:26.868026+0000","last_became_peered":"2024-01-27T01:06:26.868026+0000","last_unstale":"2024-01-27T01:06:26.881095+0000","last_undegraded":"2024-01-27T01:06:26.881095+0000","last_fullsized":"2024-01-27T01:06:26.881095+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:15.664932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884549+0000","last_change":"2024-01-27T01:06:26.876736+0000","last_active":"2024-01-27T01:06:26.884549+0000","last_peered":"2024-01-27T01:06:26.884549+0000","last_clean":"2024-01-27T01:06:26.884549+0000","last_became_active":"2024-01-27T01:06:26.875870+0000","last_became_peered":"2024-01-27T01:06:26.875870+0000","last_unstale":"2024-01-27T01:06:26.884549+0000","last_undegraded":"2024-01-27T01:06:26.884549+0000","last_fullsized":"2024-01-27T01:06:26.884549+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:43.223271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.881062+0000","last_change":"2024-01-27T01:06:26.879535+0000","last_active":"2024-01-27T01:06:26.881062+0000","last_peered":"2024-01-27T01:06:26.881062+0000","last_clean":"2024-01-27T01:06:26.881062+0000","last_became_active":"2024-01-27T01:06:26.879219+0000","last_became_peered":"2024-01-27T01:06:26.879219+0000","last_unstale":"2024-01-27T01:06:26.881062+0000","last_undegraded":"2024-01-27T01:06:26.881062+0000","last_fullsized":"2024-01-27T01:06:26.881062+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:58.495062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.874822+0000","last_change":"2024-01-27T01:06:26.865833+0000","last_active":"2024-01-27T01:06:26.874822+0000","last_peered":"2024-01-27T01:06:26.874822+0000","last_clean":"2024-01-27T01:06:26.874822+0000","last_became_active":"2024-01-27T01:06:26.865525+0000","last_became_peered":"2024-01-27T01:06:26.865525+0000","last_unstale":"2024-01-27T01:06:26.874822+0000","last_undegraded":"2024-01-27T01:06:26.874822+0000","last_fullsized":"2024-01-27T01:06:26.874822+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:42:10.990703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.880927+0000","last_change":"2024-01-27T01:06:26.877777+0000","last_active":"2024-01-27T01:06:26.880927+0000","last_peered":"2024-01-27T01:06:26.880927+0000","last_clean":"2024-01-27T01:06:26.880927+0000","last_became_active":"2024-01-27T01:06:26.877515+0000","last_became_peered":"2024-01-27T01:06:26.877515+0000","last_unstale":"2024-01-27T01:06:26.880927+0000","last_undegraded":"2024-01-27T01:06:26.880927+0000","last_fullsized":"2024-01-27T01:06:26.880927+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:27:14.653781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884229+0000","last_change":"2024-01-27T01:06:26.876674+0000","last_active":"2024-01-27T01:06:26.884229+0000","last_peered":"2024-01-27T01:06:26.884229+0000","last_clean":"2024-01-27T01:06:26.884229+0000","last_became_active":"2024-01-27T01:06:26.876413+0000","last_became_peered":"2024-01-27T01:06:26.876413+0000","last_unstale":"2024-01-27T01:06:26.884229+0000","last_undegraded":"2024-01-27T01:06:26.884229+0000","last_fullsized":"2024-01-27T01:06:26.884229+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:08.822886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.883563+0000","last_change":"2024-01-27T01:06:26.877929+0000","last_active":"2024-01-27T01:06:26.883563+0000","last_peered":"2024-01-27T01:06:26.883563+0000","last_clean":"2024-01-27T01:06:26.883563+0000","last_became_active":"2024-01-27T01:06:26.877679+0000","last_became_peered":"2024-01-27T01:06:26.877679+0000","last_unstale":"2024-01-27T01:06:26.883563+0000","last_undegraded":"2024-01-27T01:06:26.883563+0000","last_fullsized":"2024-01-27T01:06:26.883563+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:53:15.482586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.877545+0000","last_change":"2024-01-27T01:06:26.863380+0000","last_active":"2024-01-27T01:06:26.877545+0000","last_peered":"2024-01-27T01:06:26.877545+0000","last_clean":"2024-01-27T01:06:26.877545+0000","last_became_active":"2024-01-27T01:06:26.863034+0000","last_became_peered":"2024-01-27T01:06:26.863034+0000","last_unstale":"2024-01-27T01:06:26.877545+0000","last_undegraded":"2024-01-27T01:06:26.877545+0000","last_fullsized":"2024-01-27T01:06:26.877545+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:32.271904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.881114+0000","last_change":"2024-01-27T01:06:26.874473+0000","last_active":"2024-01-27T01:06:26.881114+0000","last_peered":"2024-01-27T01:06:26.881114+0000","last_clean":"2024-01-27T01:06:26.881114+0000","last_became_active":"2024-01-27T01:06:26.874203+0000","last_became_peered":"2024-01-27T01:06:26.874203+0000","last_unstale":"2024-01-27T01:06:26.881114+0000","last_undegraded":"2024-01-27T01:06:26.881114+0000","last_fullsized":"2024-01-27T01:06:26.881114+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:33:37.975146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.872176+0000","last_change":"2024-01-27T01:06:26.864890+0000","last_active":"2024-01-27T01:06:26.872176+0000","last_peered":"2024-01-27T01:06:26.872176+0000","last_clean":"2024-01-27T01:06:26.872176+0000","last_became_active":"2024-01-27T01:06:26.864556+0000","last_became_peered":"2024-01-27T01:06:26.864556+0000","last_unstale":"2024-01-27T01:06:26.872176+0000","last_undegraded":"2024-01-27T01:06:26.872176+0000","last_fullsized":"2024-01-27T01:06:26.872176+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:41.033580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.878220+0000","last_change":"2024-01-27T01:06:26.873401+0000","last_active":"2024-01-27T01:06:26.878220+0000","last_peered":"2024-01-27T01:06:26.878220+0000","last_clean":"2024-01-27T01:06:26.878220+0000","last_became_active":"2024-01-27T01:06:26.873094+0000","last_became_peered":"2024-01-27T01:06:26.873094+0000","last_unstale":"2024-01-27T01:06:26.878220+0000","last_undegraded":"2024-01-27T01:06:26.878220+0000","last_fullsized":"2024-01-27T01:06:26.878220+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:24.082749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884181+0000","last_change":"2024-01-27T01:06:26.876823+0000","last_active":"2024-01-27T01:06:26.884181+0000","last_peered":"2024-01-27T01:06:26.884181+0000","last_clean":"2024-01-27T01:06:26.884181+0000","last_became_active":"2024-01-27T01:06:26.876304+0000","last_became_peered":"2024-01-27T01:06:26.876304+0000","last_unstale":"2024-01-27T01:06:26.884181+0000","last_undegraded":"2024-01-27T01:06:26.884181+0000","last_fullsized":"2024-01-27T01:06:26.884181+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:28.378677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.885074+0000","last_change":"2024-01-27T01:06:26.879441+0000","last_active":"2024-01-27T01:06:26.885074+0000","last_peered":"2024-01-27T01:06:26.885074+0000","last_clean":"2024-01-27T01:06:26.885074+0000","last_became_active":"2024-01-27T01:06:26.879148+0000","last_became_peered":"2024-01-27T01:06:26.879148+0000","last_unstale":"2024-01-27T01:06:26.885074+0000","last_undegraded":"2024-01-27T01:06:26.885074+0000","last_fullsized":"2024-01-27T01:06:26.885074+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:05:08.012027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884672+0000","last_change":"2024-01-27T01:06:26.876390+0000","last_active":"2024-01-27T01:06:26.884672+0000","last_peered":"2024-01-27T01:06:26.884672+0000","last_clean":"2024-01-27T01:06:26.884672+0000","last_became_active":"2024-01-27T01:06:26.876056+0000","last_became_peered":"2024-01-27T01:06:26.876056+0000","last_unstale":"2024-01-27T01:06:26.884672+0000","last_undegraded":"2024-01-27T01:06:26.884672+0000","last_fullsized":"2024-01-27T01:06:26.884672+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:38:36.236958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.876121+0000","last_change":"2024-01-27T01:06:26.874471+0000","last_active":"2024-01-27T01:06:26.876121+0000","last_peered":"2024-01-27T01:06:26.876121+0000","last_clean":"2024-01-27T01:06:26.876121+0000","last_became_active":"2024-01-27T01:06:26.874271+0000","last_became_peered":"2024-01-27T01:06:26.874271+0000","last_unstale":"2024-01-27T01:06:26.876121+0000","last_undegraded":"2024-01-27T01:06:26.876121+0000","last_fullsized":"2024-01-27T01:06:26.876121+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:15:11.295547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.875043+0000","last_change":"2024-01-27T01:06:26.867241+0000","last_active":"2024-01-27T01:06:26.875043+0000","last_peered":"2024-01-27T01:06:26.875043+0000","last_clean":"2024-01-27T01:06:26.875043+0000","last_became_active":"2024-01-27T01:06:26.866917+0000","last_became_peered":"2024-01-27T01:06:26.866917+0000","last_unstale":"2024-01-27T01:06:26.875043+0000","last_undegraded":"2024-01-27T01:06:26.875043+0000","last_fullsized":"2024-01-27T01:06:26.875043+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:24:25.818592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.876177+0000","last_change":"2024-01-27T01:06:26.865149+0000","last_active":"2024-01-27T01:06:26.876177+0000","last_peered":"2024-01-27T01:06:26.876177+0000","last_clean":"2024-01-27T01:06:26.876177+0000","last_became_active":"2024-01-27T01:06:26.864853+0000","last_became_peered":"2024-01-27T01:06:26.864853+0000","last_unstale":"2024-01-27T01:06:26.876177+0000","last_undegraded":"2024-01-27T01:06:26.876177+0000","last_fullsized":"2024-01-27T01:06:26.876177+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:10:44.194323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884005+0000","last_change":"2024-01-27T01:06:26.878911+0000","last_active":"2024-01-27T01:06:26.884005+0000","last_peered":"2024-01-27T01:06:26.884005+0000","last_clean":"2024-01-27T01:06:26.884005+0000","last_became_active":"2024-01-27T01:06:26.878637+0000","last_became_peered":"2024-01-27T01:06:26.878637+0000","last_unstale":"2024-01-27T01:06:26.884005+0000","last_undegraded":"2024-01-27T01:06:26.884005+0000","last_fullsized":"2024-01-27T01:06:26.884005+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:33:03.751565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.881049+0000","last_change":"2024-01-27T01:06:26.878202+0000","last_active":"2024-01-27T01:06:26.881049+0000","last_peered":"2024-01-27T01:06:26.881049+0000","last_clean":"2024-01-27T01:06:26.881049+0000","last_became_active":"2024-01-27T01:06:26.877900+0000","last_became_peered":"2024-01-27T01:06:26.877900+0000","last_unstale":"2024-01-27T01:06:26.881049+0000","last_undegraded":"2024-01-27T01:06:26.881049+0000","last_fullsized":"2024-01-27T01:06:26.881049+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:42.766155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.875568+0000","last_change":"2024-01-27T01:06:26.870649+0000","last_active":"2024-01-27T01:06:26.875568+0000","last_peered":"2024-01-27T01:06:26.875568+0000","last_clean":"2024-01-27T01:06:26.875568+0000","last_became_active":"2024-01-27T01:06:26.870433+0000","last_became_peered":"2024-01-27T01:06:26.870433+0000","last_unstale":"2024-01-27T01:06:26.875568+0000","last_undegraded":"2024-01-27T01:06:26.875568+0000","last_fullsized":"2024-01-27T01:06:26.875568+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:11:27.402943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.873029+0000","last_change":"2024-01-27T01:06:26.872013+0000","last_active":"2024-01-27T01:06:26.873029+0000","last_peered":"2024-01-27T01:06:26.873029+0000","last_clean":"2024-01-27T01:06:26.873029+0000","last_became_active":"2024-01-27T01:06:26.867577+0000","last_became_peered":"2024-01-27T01:06:26.867577+0000","last_unstale":"2024-01-27T01:06:26.873029+0000","last_undegraded":"2024-01-27T01:06:26.873029+0000","last_fullsized":"2024-01-27T01:06:26.873029+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:27.604574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.874991+0000","last_change":"2024-01-27T01:06:26.871676+0000","last_active":"2024-01-27T01:06:26.874991+0000","last_peered":"2024-01-27T01:06:26.874991+0000","last_clean":"2024-01-27T01:06:26.874991+0000","last_became_active":"2024-01-27T01:06:26.871423+0000","last_became_peered":"2024-01-27T01:06:26.871423+0000","last_unstale":"2024-01-27T01:06:26.874991+0000","last_undegraded":"2024-01-27T01:06:26.874991+0000","last_fullsized":"2024-01-27T01:06:26.874991+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:29:58.146876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.880900+0000","last_change":"2024-01-27T01:06:26.876343+0000","last_active":"2024-01-27T01:06:26.880900+0000","last_peered":"2024-01-27T01:06:26.880900+0000","last_clean":"2024-01-27T01:06:26.880900+0000","last_became_active":"2024-01-27T01:06:26.875570+0000","last_became_peered":"2024-01-27T01:06:26.875570+0000","last_unstale":"2024-01-27T01:06:26.880900+0000","last_undegraded":"2024-01-27T01:06:26.880900+0000","last_fullsized":"2024-01-27T01:06:26.880900+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:07:51.640059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.872421+0000","last_change":"2024-01-27T01:06:26.869029+0000","last_active":"2024-01-27T01:06:26.872421+0000","last_peered":"2024-01-27T01:06:26.872421+0000","last_clean":"2024-01-27T01:06:26.872421+0000","last_became_active":"2024-01-27T01:06:26.868793+0000","last_became_peered":"2024-01-27T01:06:26.868793+0000","last_unstale":"2024-01-27T01:06:26.872421+0000","last_undegraded":"2024-01-27T01:06:26.872421+0000","last_fullsized":"2024-01-27T01:06:26.872421+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:10.965469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.874093+0000","last_change":"2024-01-27T01:06:26.869588+0000","last_active":"2024-01-27T01:06:26.874093+0000","last_peered":"2024-01-27T01:06:26.874093+0000","last_clean":"2024-01-27T01:06:26.874093+0000","last_became_active":"2024-01-27T01:06:26.869264+0000","last_became_peered":"2024-01-27T01:06:26.869264+0000","last_unstale":"2024-01-27T01:06:26.874093+0000","last_undegraded":"2024-01-27T01:06:26.874093+0000","last_fullsized":"2024-01-27T01:06:26.874093+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:42:54.364688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"222'4","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.889105+0000","last_change":"2024-01-27T01:06:25.856458+0000","last_active":"2024-01-27T01:06:26.889105+0000","last_peered":"2024-01-27T01:06:26.889105+0000","last_clean":"2024-01-27T01:06:26.889105+0000","last_became_active":"2024-01-27T01:06:25.856162+0000","last_became_peered":"2024-01-27T01:06:25.856162+0000","last_unstale":"2024-01-27T01:06:26.889105+0000","last_undegraded":"2024-01-27T01:06:26.889105+0000","last_fullsized":"2024-01-27T01:06:26.889105+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:51:44.378880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.883957+0000","last_change":"2024-01-27T01:06:26.878625+0000","last_active":"2024-01-27T01:06:26.883957+0000","last_peered":"2024-01-27T01:06:26.883957+0000","last_clean":"2024-01-27T01:06:26.883957+0000","last_became_active":"2024-01-27T01:06:26.878361+0000","last_became_peered":"2024-01-27T01:06:26.878361+0000","last_unstale":"2024-01-27T01:06:26.883957+0000","last_undegraded":"2024-01-27T01:06:26.883957+0000","last_fullsized":"2024-01-27T01:06:26.883957+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:37:06.977689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"222'1","reported_seq":11,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.891631+0000","last_change":"2024-01-27T01:06:25.856504+0000","last_active":"2024-01-27T01:06:26.891631+0000","last_peered":"2024-01-27T01:06:26.891631+0000","last_clean":"2024-01-27T01:06:26.891631+0000","last_became_active":"2024-01-27T01:06:25.856272+0000","last_became_peered":"2024-01-27T01:06:25.856272+0000","last_unstale":"2024-01-27T01:06:26.891631+0000","last_undegraded":"2024-01-27T01:06:26.891631+0000","last_fullsized":"2024-01-27T01:06:26.891631+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:20:56.736428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884207+0000","last_change":"2024-01-27T01:06:26.875831+0000","last_active":"2024-01-27T01:06:26.884207+0000","last_peered":"2024-01-27T01:06:26.884207+0000","last_clean":"2024-01-27T01:06:26.884207+0000","last_became_active":"2024-01-27T01:06:26.875488+0000","last_became_peered":"2024-01-27T01:06:26.875488+0000","last_unstale":"2024-01-27T01:06:26.884207+0000","last_undegraded":"2024-01-27T01:06:26.884207+0000","last_fullsized":"2024-01-27T01:06:26.884207+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:07:51.640059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"222'2","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.887982+0000","last_change":"2024-01-27T01:06:25.855991+0000","last_active":"2024-01-27T01:06:26.887982+0000","last_peered":"2024-01-27T01:06:26.887982+0000","last_clean":"2024-01-27T01:06:26.887982+0000","last_became_active":"2024-01-27T01:06:25.855679+0000","last_became_peered":"2024-01-27T01:06:25.855679+0000","last_unstale":"2024-01-27T01:06:26.887982+0000","last_undegraded":"2024-01-27T01:06:26.887982+0000","last_fullsized":"2024-01-27T01:06:26.887982+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:31:04.295025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.875454+0000","last_change":"2024-01-27T01:06:26.869128+0000","last_active":"2024-01-27T01:06:26.875454+0000","last_peered":"2024-01-27T01:06:26.875454+0000","last_clean":"2024-01-27T01:06:26.875454+0000","last_became_active":"2024-01-27T01:06:26.868892+0000","last_became_peered":"2024-01-27T01:06:26.868892+0000","last_unstale":"2024-01-27T01:06:26.875454+0000","last_undegraded":"2024-01-27T01:06:26.875454+0000","last_fullsized":"2024-01-27T01:06:26.875454+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:40:40.869516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"222'2","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.889282+0000","last_change":"2024-01-27T01:06:25.855114+0000","last_active":"2024-01-27T01:06:26.889282+0000","last_peered":"2024-01-27T01:06:26.889282+0000","last_clean":"2024-01-27T01:06:26.889282+0000","last_became_active":"2024-01-27T01:06:25.854790+0000","last_became_peered":"2024-01-27T01:06:25.854790+0000","last_unstale":"2024-01-27T01:06:26.889282+0000","last_undegraded":"2024-01-27T01:06:26.889282+0000","last_fullsized":"2024-01-27T01:06:26.889282+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:58:58.498377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.873014+0000","last_change":"2024-01-27T01:06:26.866121+0000","last_active":"2024-01-27T01:06:26.873014+0000","last_peered":"2024-01-27T01:06:26.873014+0000","last_clean":"2024-01-27T01:06:26.873014+0000","last_became_active":"2024-01-27T01:06:26.865391+0000","last_became_peered":"2024-01-27T01:06:26.865391+0000","last_unstale":"2024-01-27T01:06:26.873014+0000","last_undegraded":"2024-01-27T01:06:26.873014+0000","last_fullsized":"2024-01-27T01:06:26.873014+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:02:03.812030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"222'1","reported_seq":11,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.888794+0000","last_change":"2024-01-27T01:06:25.854193+0000","last_active":"2024-01-27T01:06:26.888794+0000","last_peered":"2024-01-27T01:06:26.888794+0000","last_clean":"2024-01-27T01:06:26.888794+0000","last_became_active":"2024-01-27T01:06:25.853956+0000","last_became_peered":"2024-01-27T01:06:25.853956+0000","last_unstale":"2024-01-27T01:06:26.888794+0000","last_undegraded":"2024-01-27T01:06:26.888794+0000","last_fullsized":"2024-01-27T01:06:26.888794+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:33:02.747087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.880717+0000","last_change":"2024-01-27T01:06:26.876963+0000","last_active":"2024-01-27T01:06:26.880717+0000","last_peered":"2024-01-27T01:06:26.880717+0000","last_clean":"2024-01-27T01:06:26.880717+0000","last_became_active":"2024-01-27T01:06:26.876709+0000","last_became_peered":"2024-01-27T01:06:26.876709+0000","last_unstale":"2024-01-27T01:06:26.880717+0000","last_undegraded":"2024-01-27T01:06:26.880717+0000","last_fullsized":"2024-01-27T01:06:26.880717+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:58:14.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.859854+0000","last_change":"2024-01-27T01:06:25.854878+0000","last_active":"2024-01-27T01:06:26.859854+0000","last_peered":"2024-01-27T01:06:26.859854+0000","last_clean":"2024-01-27T01:06:26.859854+0000","last_became_active":"2024-01-27T01:06:25.854623+0000","last_became_peered":"2024-01-27T01:06:25.854623+0000","last_unstale":"2024-01-27T01:06:26.859854+0000","last_undegraded":"2024-01-27T01:06:26.859854+0000","last_fullsized":"2024-01-27T01:06:26.859854+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:21.007449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.881070+0000","last_change":"2024-01-27T01:06:26.873199+0000","last_active":"2024-01-27T01:06:26.881070+0000","last_peered":"2024-01-27T01:06:26.881070+0000","last_clean":"2024-01-27T01:06:26.881070+0000","last_became_active":"2024-01-27T01:06:26.872658+0000","last_became_peered":"2024-01-27T01:06:26.872658+0000","last_unstale":"2024-01-27T01:06:26.881070+0000","last_undegraded":"2024-01-27T01:06:26.881070+0000","last_fullsized":"2024-01-27T01:06:26.881070+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:06:45.526347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.857016+0000","last_change":"2024-01-27T01:06:25.851888+0000","last_active":"2024-01-27T01:06:26.857016+0000","last_peered":"2024-01-27T01:06:26.857016+0000","last_clean":"2024-01-27T01:06:26.857016+0000","last_became_active":"2024-01-27T01:06:25.851605+0000","last_became_peered":"2024-01-27T01:06:25.851605+0000","last_unstale":"2024-01-27T01:06:26.857016+0000","last_undegraded":"2024-01-27T01:06:26.857016+0000","last_fullsized":"2024-01-27T01:06:26.857016+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:09.386094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.875063+0000","last_change":"2024-01-27T01:06:26.870806+0000","last_active":"2024-01-27T01:06:26.875063+0000","last_peered":"2024-01-27T01:06:26.875063+0000","last_clean":"2024-01-27T01:06:26.875063+0000","last_became_active":"2024-01-27T01:06:26.870263+0000","last_became_peered":"2024-01-27T01:06:26.870263+0000","last_unstale":"2024-01-27T01:06:26.875063+0000","last_undegraded":"2024-01-27T01:06:26.875063+0000","last_fullsized":"2024-01-27T01:06:26.875063+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:26.214953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.856798+0000","last_change":"2024-01-27T01:06:25.865314+0000","last_active":"2024-01-27T01:06:26.856798+0000","last_peered":"2024-01-27T01:06:26.856798+0000","last_clean":"2024-01-27T01:06:26.856798+0000","last_became_active":"2024-01-27T01:06:25.864940+0000","last_became_peered":"2024-01-27T01:06:25.864940+0000","last_unstale":"2024-01-27T01:06:26.856798+0000","last_undegraded":"2024-01-27T01:06:26.856798+0000","last_fullsized":"2024-01-27T01:06:26.856798+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:24.113986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.877921+0000","last_change":"2024-01-27T01:06:26.868859+0000","last_active":"2024-01-27T01:06:26.877921+0000","last_peered":"2024-01-27T01:06:26.877921+0000","last_clean":"2024-01-27T01:06:26.877921+0000","last_became_active":"2024-01-27T01:06:26.868623+0000","last_became_peered":"2024-01-27T01:06:26.868623+0000","last_unstale":"2024-01-27T01:06:26.877921+0000","last_undegraded":"2024-01-27T01:06:26.877921+0000","last_fullsized":"2024-01-27T01:06:26.877921+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:18:47.232067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.859277+0000","last_change":"2024-01-27T01:06:25.860762+0000","last_active":"2024-01-27T01:06:26.859277+0000","last_peered":"2024-01-27T01:06:26.859277+0000","last_clean":"2024-01-27T01:06:26.859277+0000","last_became_active":"2024-01-27T01:06:25.860459+0000","last_became_peered":"2024-01-27T01:06:25.860459+0000","last_unstale":"2024-01-27T01:06:26.859277+0000","last_undegraded":"2024-01-27T01:06:26.859277+0000","last_fullsized":"2024-01-27T01:06:26.859277+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:31.687265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.877812+0000","last_change":"2024-01-27T01:06:26.876685+0000","last_active":"2024-01-27T01:06:26.877812+0000","last_peered":"2024-01-27T01:06:26.877812+0000","last_clean":"2024-01-27T01:06:26.877812+0000","last_became_active":"2024-01-27T01:06:26.872939+0000","last_became_peered":"2024-01-27T01:06:26.872939+0000","last_unstale":"2024-01-27T01:06:26.877812+0000","last_undegraded":"2024-01-27T01:06:26.877812+0000","last_fullsized":"2024-01-27T01:06:26.877812+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:56:48.730726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"222'3","reported_seq":13,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.888743+0000","last_change":"2024-01-27T01:06:25.857771+0000","last_active":"2024-01-27T01:06:26.888743+0000","last_peered":"2024-01-27T01:06:26.888743+0000","last_clean":"2024-01-27T01:06:26.888743+0000","last_became_active":"2024-01-27T01:06:25.857445+0000","last_became_peered":"2024-01-27T01:06:25.857445+0000","last_unstale":"2024-01-27T01:06:26.888743+0000","last_undegraded":"2024-01-27T01:06:26.888743+0000","last_fullsized":"2024-01-27T01:06:26.888743+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:52.927866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.887220+0000","last_change":"2024-01-27T01:06:26.879652+0000","last_active":"2024-01-27T01:06:26.887220+0000","last_peered":"2024-01-27T01:06:26.887220+0000","last_clean":"2024-01-27T01:06:26.887220+0000","last_became_active":"2024-01-27T01:06:26.879271+0000","last_became_peered":"2024-01-27T01:06:26.879271+0000","last_unstale":"2024-01-27T01:06:26.887220+0000","last_undegraded":"2024-01-27T01:06:26.887220+0000","last_fullsized":"2024-01-27T01:06:26.887220+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:57:23.456731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"222'6","reported_seq":16,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.894314+0000","last_change":"2024-01-27T01:06:25.864160+0000","last_active":"2024-01-27T01:06:26.894314+0000","last_peered":"2024-01-27T01:06:26.894314+0000","last_clean":"2024-01-27T01:06:26.894314+0000","last_became_active":"2024-01-27T01:06:25.863867+0000","last_became_peered":"2024-01-27T01:06:25.863867+0000","last_unstale":"2024-01-27T01:06:26.894314+0000","last_undegraded":"2024-01-27T01:06:26.894314+0000","last_fullsized":"2024-01-27T01:06:26.894314+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:43.338484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.875502+0000","last_change":"2024-01-27T01:06:26.873464+0000","last_active":"2024-01-27T01:06:26.875502+0000","last_peered":"2024-01-27T01:06:26.875502+0000","last_clean":"2024-01-27T01:06:26.875502+0000","last_became_active":"2024-01-27T01:06:26.873063+0000","last_became_peered":"2024-01-27T01:06:26.873063+0000","last_unstale":"2024-01-27T01:06:26.875502+0000","last_undegraded":"2024-01-27T01:06:26.875502+0000","last_fullsized":"2024-01-27T01:06:26.875502+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:33:32.535574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"222'2","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.891983+0000","last_change":"2024-01-27T01:06:25.850061+0000","last_active":"2024-01-27T01:06:26.891983+0000","last_peered":"2024-01-27T01:06:26.891983+0000","last_clean":"2024-01-27T01:06:26.891983+0000","last_became_active":"2024-01-27T01:06:25.849763+0000","last_became_peered":"2024-01-27T01:06:25.849763+0000","last_unstale":"2024-01-27T01:06:26.891983+0000","last_undegraded":"2024-01-27T01:06:26.891983+0000","last_fullsized":"2024-01-27T01:06:26.891983+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:09:56.243562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.877332+0000","last_change":"2024-01-27T01:06:26.862905+0000","last_active":"2024-01-27T01:06:26.877332+0000","last_peered":"2024-01-27T01:06:26.877332+0000","last_clean":"2024-01-27T01:06:26.877332+0000","last_became_active":"2024-01-27T01:06:26.862584+0000","last_became_peered":"2024-01-27T01:06:26.862584+0000","last_unstale":"2024-01-27T01:06:26.877332+0000","last_undegraded":"2024-01-27T01:06:26.877332+0000","last_fullsized":"2024-01-27T01:06:26.877332+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:27:14.653781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.857286+0000","last_change":"2024-01-27T01:06:25.854192+0000","last_active":"2024-01-27T01:06:26.857286+0000","last_peered":"2024-01-27T01:06:26.857286+0000","last_clean":"2024-01-27T01:06:26.857286+0000","last_became_active":"2024-01-27T01:06:25.853601+0000","last_became_peered":"2024-01-27T01:06:25.853601+0000","last_unstale":"2024-01-27T01:06:26.857286+0000","last_undegraded":"2024-01-27T01:06:26.857286+0000","last_fullsized":"2024-01-27T01:06:26.857286+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:42.218792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.872363+0000","last_change":"2024-01-27T01:06:26.866613+0000","last_active":"2024-01-27T01:06:26.872363+0000","last_peered":"2024-01-27T01:06:26.872363+0000","last_clean":"2024-01-27T01:06:26.872363+0000","last_became_active":"2024-01-27T01:06:26.866298+0000","last_became_peered":"2024-01-27T01:06:26.866298+0000","last_unstale":"2024-01-27T01:06:26.872363+0000","last_undegraded":"2024-01-27T01:06:26.872363+0000","last_fullsized":"2024-01-27T01:06:26.872363+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:27:03.934654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.852979+0000","last_change":"2024-01-27T01:06:25.853597+0000","last_active":"2024-01-27T01:06:26.852979+0000","last_peered":"2024-01-27T01:06:26.852979+0000","last_clean":"2024-01-27T01:06:26.852979+0000","last_became_active":"2024-01-27T01:06:25.853257+0000","last_became_peered":"2024-01-27T01:06:25.853257+0000","last_unstale":"2024-01-27T01:06:26.852979+0000","last_undegraded":"2024-01-27T01:06:26.852979+0000","last_fullsized":"2024-01-27T01:06:26.852979+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:54:35.011796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.872381+0000","last_change":"2024-01-27T01:06:26.867077+0000","last_active":"2024-01-27T01:06:26.872381+0000","last_peered":"2024-01-27T01:06:26.872381+0000","last_clean":"2024-01-27T01:06:26.872381+0000","last_became_active":"2024-01-27T01:06:26.866787+0000","last_became_peered":"2024-01-27T01:06:26.866787+0000","last_unstale":"2024-01-27T01:06:26.872381+0000","last_undegraded":"2024-01-27T01:06:26.872381+0000","last_fullsized":"2024-01-27T01:06:26.872381+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:02.000909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"222'8","reported_seq":18,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.892509+0000","last_change":"2024-01-27T01:06:25.865239+0000","last_active":"2024-01-27T01:06:26.892509+0000","last_peered":"2024-01-27T01:06:26.892509+0000","last_clean":"2024-01-27T01:06:26.892509+0000","last_became_active":"2024-01-27T01:06:25.864943+0000","last_became_peered":"2024-01-27T01:06:25.864943+0000","last_unstale":"2024-01-27T01:06:26.892509+0000","last_undegraded":"2024-01-27T01:06:26.892509+0000","last_fullsized":"2024-01-27T01:06:26.892509+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:41:25.254751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.880063+0000","last_change":"2024-01-27T01:06:26.875234+0000","last_active":"2024-01-27T01:06:26.880063+0000","last_peered":"2024-01-27T01:06:26.880063+0000","last_clean":"2024-01-27T01:06:26.880063+0000","last_became_active":"2024-01-27T01:06:26.874927+0000","last_became_peered":"2024-01-27T01:06:26.874927+0000","last_unstale":"2024-01-27T01:06:26.880063+0000","last_undegraded":"2024-01-27T01:06:26.880063+0000","last_fullsized":"2024-01-27T01:06:26.880063+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:10.390573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"222'1","reported_seq":11,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.891755+0000","last_change":"2024-01-27T01:06:25.848900+0000","last_active":"2024-01-27T01:06:26.891755+0000","last_peered":"2024-01-27T01:06:26.891755+0000","last_clean":"2024-01-27T01:06:26.891755+0000","last_became_active":"2024-01-27T01:06:25.848483+0000","last_became_peered":"2024-01-27T01:06:25.848483+0000","last_unstale":"2024-01-27T01:06:26.891755+0000","last_undegraded":"2024-01-27T01:06:26.891755+0000","last_fullsized":"2024-01-27T01:06:26.891755+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:27:13.471892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884185+0000","last_change":"2024-01-27T01:06:26.879356+0000","last_active":"2024-01-27T01:06:26.884185+0000","last_peered":"2024-01-27T01:06:26.884185+0000","last_clean":"2024-01-27T01:06:26.884185+0000","last_became_active":"2024-01-27T01:06:26.878891+0000","last_became_peered":"2024-01-27T01:06:26.878891+0000","last_unstale":"2024-01-27T01:06:26.884185+0000","last_undegraded":"2024-01-27T01:06:26.884185+0000","last_fullsized":"2024-01-27T01:06:26.884185+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:22.011928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"222'2","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.892036+0000","last_change":"2024-01-27T01:06:25.852245+0000","last_active":"2024-01-27T01:06:26.892036+0000","last_peered":"2024-01-27T01:06:26.892036+0000","last_clean":"2024-01-27T01:06:26.892036+0000","last_became_active":"2024-01-27T01:06:25.851982+0000","last_became_peered":"2024-01-27T01:06:25.851982+0000","last_unstale":"2024-01-27T01:06:26.892036+0000","last_undegraded":"2024-01-27T01:06:26.892036+0000","last_fullsized":"2024-01-27T01:06:26.892036+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:05.596045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.879459+0000","last_change":"2024-01-27T01:06:26.870758+0000","last_active":"2024-01-27T01:06:26.879459+0000","last_peered":"2024-01-27T01:06:26.879459+0000","last_clean":"2024-01-27T01:06:26.879459+0000","last_became_active":"2024-01-27T01:06:26.870130+0000","last_became_peered":"2024-01-27T01:06:26.870130+0000","last_unstale":"2024-01-27T01:06:26.879459+0000","last_undegraded":"2024-01-27T01:06:26.879459+0000","last_fullsized":"2024-01-27T01:06:26.879459+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:36:31.625857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"222'1","reported_seq":11,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.887461+0000","last_change":"2024-01-27T01:06:25.851720+0000","last_active":"2024-01-27T01:06:26.887461+0000","last_peered":"2024-01-27T01:06:26.887461+0000","last_clean":"2024-01-27T01:06:26.887461+0000","last_became_active":"2024-01-27T01:06:25.851416+0000","last_became_peered":"2024-01-27T01:06:25.851416+0000","last_unstale":"2024-01-27T01:06:26.887461+0000","last_undegraded":"2024-01-27T01:06:26.887461+0000","last_fullsized":"2024-01-27T01:06:26.887461+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:07.951814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884204+0000","last_change":"2024-01-27T01:06:26.876619+0000","last_active":"2024-01-27T01:06:26.884204+0000","last_peered":"2024-01-27T01:06:26.884204+0000","last_clean":"2024-01-27T01:06:26.884204+0000","last_became_active":"2024-01-27T01:06:26.875927+0000","last_became_peered":"2024-01-27T01:06:26.875927+0000","last_unstale":"2024-01-27T01:06:26.884204+0000","last_undegraded":"2024-01-27T01:06:26.884204+0000","last_fullsized":"2024-01-27T01:06:26.884204+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:02.000909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.859698+0000","last_change":"2024-01-27T01:06:25.861650+0000","last_active":"2024-01-27T01:06:26.859698+0000","last_peered":"2024-01-27T01:06:26.859698+0000","last_clean":"2024-01-27T01:06:26.859698+0000","last_became_active":"2024-01-27T01:06:25.861342+0000","last_became_peered":"2024-01-27T01:06:25.861342+0000","last_unstale":"2024-01-27T01:06:26.859698+0000","last_undegraded":"2024-01-27T01:06:26.859698+0000","last_fullsized":"2024-01-27T01:06:26.859698+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:44:15.953217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.875392+0000","last_change":"2024-01-27T01:06:26.871346+0000","last_active":"2024-01-27T01:06:26.875392+0000","last_peered":"2024-01-27T01:06:26.875392+0000","last_clean":"2024-01-27T01:06:26.875392+0000","last_became_active":"2024-01-27T01:06:26.870999+0000","last_became_peered":"2024-01-27T01:06:26.870999+0000","last_unstale":"2024-01-27T01:06:26.875392+0000","last_undegraded":"2024-01-27T01:06:26.875392+0000","last_fullsized":"2024-01-27T01:06:26.875392+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:27.779157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"222'2","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.887697+0000","last_change":"2024-01-27T01:06:25.860572+0000","last_active":"2024-01-27T01:06:26.887697+0000","last_peered":"2024-01-27T01:06:26.887697+0000","last_clean":"2024-01-27T01:06:26.887697+0000","last_became_active":"2024-01-27T01:06:25.860248+0000","last_became_peered":"2024-01-27T01:06:25.860248+0000","last_unstale":"2024-01-27T01:06:26.887697+0000","last_undegraded":"2024-01-27T01:06:26.887697+0000","last_fullsized":"2024-01-27T01:06:26.887697+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:01:25.741123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.879329+0000","last_change":"2024-01-27T01:06:26.875858+0000","last_active":"2024-01-27T01:06:26.879329+0000","last_peered":"2024-01-27T01:06:26.879329+0000","last_clean":"2024-01-27T01:06:26.879329+0000","last_became_active":"2024-01-27T01:06:26.875564+0000","last_became_peered":"2024-01-27T01:06:26.875564+0000","last_unstale":"2024-01-27T01:06:26.879329+0000","last_undegraded":"2024-01-27T01:06:26.879329+0000","last_fullsized":"2024-01-27T01:06:26.879329+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:43.630831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.857921+0000","last_change":"2024-01-27T01:06:25.851595+0000","last_active":"2024-01-27T01:06:26.857921+0000","last_peered":"2024-01-27T01:06:26.857921+0000","last_clean":"2024-01-27T01:06:26.857921+0000","last_became_active":"2024-01-27T01:06:25.851260+0000","last_became_peered":"2024-01-27T01:06:25.851260+0000","last_unstale":"2024-01-27T01:06:26.857921+0000","last_undegraded":"2024-01-27T01:06:26.857921+0000","last_fullsized":"2024-01-27T01:06:26.857921+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:56:53.179347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.872399+0000","last_change":"2024-01-27T01:06:26.867570+0000","last_active":"2024-01-27T01:06:26.872399+0000","last_peered":"2024-01-27T01:06:26.872399+0000","last_clean":"2024-01-27T01:06:26.872399+0000","last_became_active":"2024-01-27T01:06:26.866517+0000","last_became_peered":"2024-01-27T01:06:26.866517+0000","last_unstale":"2024-01-27T01:06:26.872399+0000","last_undegraded":"2024-01-27T01:06:26.872399+0000","last_fullsized":"2024-01-27T01:06:26.872399+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:09:54.515702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.859850+0000","last_change":"2024-01-27T01:06:25.862199+0000","last_active":"2024-01-27T01:06:26.859850+0000","last_peered":"2024-01-27T01:06:26.859850+0000","last_clean":"2024-01-27T01:06:26.859850+0000","last_became_active":"2024-01-27T01:06:25.861876+0000","last_became_peered":"2024-01-27T01:06:25.861876+0000","last_unstale":"2024-01-27T01:06:26.859850+0000","last_undegraded":"2024-01-27T01:06:26.859850+0000","last_fullsized":"2024-01-27T01:06:26.859850+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:00:54.297085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884265+0000","last_change":"2024-01-27T01:06:26.875536+0000","last_active":"2024-01-27T01:06:26.884265+0000","last_peered":"2024-01-27T01:06:26.884265+0000","last_clean":"2024-01-27T01:06:26.884265+0000","last_became_active":"2024-01-27T01:06:26.875267+0000","last_became_peered":"2024-01-27T01:06:26.875267+0000","last_unstale":"2024-01-27T01:06:26.884265+0000","last_undegraded":"2024-01-27T01:06:26.884265+0000","last_fullsized":"2024-01-27T01:06:26.884265+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:32.534622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.858847+0000","last_change":"2024-01-27T01:06:25.853605+0000","last_active":"2024-01-27T01:06:26.858847+0000","last_peered":"2024-01-27T01:06:26.858847+0000","last_clean":"2024-01-27T01:06:26.858847+0000","last_became_active":"2024-01-27T01:06:25.853331+0000","last_became_peered":"2024-01-27T01:06:25.853331+0000","last_unstale":"2024-01-27T01:06:26.858847+0000","last_undegraded":"2024-01-27T01:06:26.858847+0000","last_fullsized":"2024-01-27T01:06:26.858847+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:09:08.458850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.880032+0000","last_change":"2024-01-27T01:06:26.875222+0000","last_active":"2024-01-27T01:06:26.880032+0000","last_peered":"2024-01-27T01:06:26.880032+0000","last_clean":"2024-01-27T01:06:26.880032+0000","last_became_active":"2024-01-27T01:06:26.874903+0000","last_became_peered":"2024-01-27T01:06:26.874903+0000","last_unstale":"2024-01-27T01:06:26.880032+0000","last_undegraded":"2024-01-27T01:06:26.880032+0000","last_fullsized":"2024-01-27T01:06:26.880032+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:09:33.903445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"222'2","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.892395+0000","last_change":"2024-01-27T01:06:25.857933+0000","last_active":"2024-01-27T01:06:26.892395+0000","last_peered":"2024-01-27T01:06:26.892395+0000","last_clean":"2024-01-27T01:06:26.892395+0000","last_became_active":"2024-01-27T01:06:25.857406+0000","last_became_peered":"2024-01-27T01:06:25.857406+0000","last_unstale":"2024-01-27T01:06:26.892395+0000","last_undegraded":"2024-01-27T01:06:26.892395+0000","last_fullsized":"2024-01-27T01:06:26.892395+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:20:20.239633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.880984+0000","last_change":"2024-01-27T01:06:26.878650+0000","last_active":"2024-01-27T01:06:26.880984+0000","last_peered":"2024-01-27T01:06:26.880984+0000","last_clean":"2024-01-27T01:06:26.880984+0000","last_became_active":"2024-01-27T01:06:26.877596+0000","last_became_peered":"2024-01-27T01:06:26.877596+0000","last_unstale":"2024-01-27T01:06:26.880984+0000","last_undegraded":"2024-01-27T01:06:26.880984+0000","last_fullsized":"2024-01-27T01:06:26.880984+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:42:48.942202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.864255+0000","last_change":"2024-01-27T01:06:25.861717+0000","last_active":"2024-01-27T01:06:26.864255+0000","last_peered":"2024-01-27T01:06:26.864255+0000","last_clean":"2024-01-27T01:06:26.864255+0000","last_became_active":"2024-01-27T01:06:25.861177+0000","last_became_peered":"2024-01-27T01:06:25.861177+0000","last_unstale":"2024-01-27T01:06:26.864255+0000","last_undegraded":"2024-01-27T01:06:26.864255+0000","last_fullsized":"2024-01-27T01:06:26.864255+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:05.596045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.876760+0000","last_change":"2024-01-27T01:06:26.872190+0000","last_active":"2024-01-27T01:06:26.876760+0000","last_peered":"2024-01-27T01:06:26.876760+0000","last_clean":"2024-01-27T01:06:26.876760+0000","last_became_active":"2024-01-27T01:06:26.867718+0000","last_became_peered":"2024-01-27T01:06:26.867718+0000","last_unstale":"2024-01-27T01:06:26.876760+0000","last_undegraded":"2024-01-27T01:06:26.876760+0000","last_fullsized":"2024-01-27T01:06:26.876760+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:23.301132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.858620+0000","last_change":"2024-01-27T01:06:25.858032+0000","last_active":"2024-01-27T01:06:26.858620+0000","last_peered":"2024-01-27T01:06:26.858620+0000","last_clean":"2024-01-27T01:06:26.858620+0000","last_became_active":"2024-01-27T01:06:25.857777+0000","last_became_peered":"2024-01-27T01:06:25.857777+0000","last_unstale":"2024-01-27T01:06:26.858620+0000","last_undegraded":"2024-01-27T01:06:26.858620+0000","last_fullsized":"2024-01-27T01:06:26.858620+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:09:08.458850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.876438+0000","last_change":"2024-01-27T01:06:26.870597+0000","last_active":"2024-01-27T01:06:26.876438+0000","last_peered":"2024-01-27T01:06:26.876438+0000","last_clean":"2024-01-27T01:06:26.876438+0000","last_became_active":"2024-01-27T01:06:26.870334+0000","last_became_peered":"2024-01-27T01:06:26.870334+0000","last_unstale":"2024-01-27T01:06:26.876438+0000","last_undegraded":"2024-01-27T01:06:26.876438+0000","last_fullsized":"2024-01-27T01:06:26.876438+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:01:23.266072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.861167+0000","last_change":"2024-01-27T01:06:25.859342+0000","last_active":"2024-01-27T01:06:26.861167+0000","last_peered":"2024-01-27T01:06:26.861167+0000","last_clean":"2024-01-27T01:06:26.861167+0000","last_became_active":"2024-01-27T01:06:25.859065+0000","last_became_peered":"2024-01-27T01:06:25.859065+0000","last_unstale":"2024-01-27T01:06:26.861167+0000","last_undegraded":"2024-01-27T01:06:26.861167+0000","last_fullsized":"2024-01-27T01:06:26.861167+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:00:54.297085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.885135+0000","last_change":"2024-01-27T01:06:26.880503+0000","last_active":"2024-01-27T01:06:26.885135+0000","last_peered":"2024-01-27T01:06:26.885135+0000","last_clean":"2024-01-27T01:06:26.885135+0000","last_became_active":"2024-01-27T01:06:26.880224+0000","last_became_peered":"2024-01-27T01:06:26.880224+0000","last_unstale":"2024-01-27T01:06:26.885135+0000","last_undegraded":"2024-01-27T01:06:26.885135+0000","last_fullsized":"2024-01-27T01:06:26.885135+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:27:33.886740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.865748+0000","last_change":"2024-01-27T01:06:25.858142+0000","last_active":"2024-01-27T01:06:26.865748+0000","last_peered":"2024-01-27T01:06:26.865748+0000","last_clean":"2024-01-27T01:06:26.865748+0000","last_became_active":"2024-01-27T01:06:25.857867+0000","last_became_peered":"2024-01-27T01:06:25.857867+0000","last_unstale":"2024-01-27T01:06:26.865748+0000","last_undegraded":"2024-01-27T01:06:26.865748+0000","last_fullsized":"2024-01-27T01:06:26.865748+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:10:59.780192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.880953+0000","last_change":"2024-01-27T01:06:26.876182+0000","last_active":"2024-01-27T01:06:26.880953+0000","last_peered":"2024-01-27T01:06:26.880953+0000","last_clean":"2024-01-27T01:06:26.880953+0000","last_became_active":"2024-01-27T01:06:26.875894+0000","last_became_peered":"2024-01-27T01:06:26.875894+0000","last_unstale":"2024-01-27T01:06:26.880953+0000","last_undegraded":"2024-01-27T01:06:26.880953+0000","last_fullsized":"2024-01-27T01:06:26.880953+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:58.495062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.854249+0000","last_change":"2024-01-27T01:06:25.855015+0000","last_active":"2024-01-27T01:06:26.854249+0000","last_peered":"2024-01-27T01:06:26.854249+0000","last_clean":"2024-01-27T01:06:26.854249+0000","last_became_active":"2024-01-27T01:06:25.854708+0000","last_became_peered":"2024-01-27T01:06:25.854708+0000","last_unstale":"2024-01-27T01:06:26.854249+0000","last_undegraded":"2024-01-27T01:06:26.854249+0000","last_fullsized":"2024-01-27T01:06:26.854249+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:03.971945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.876472+0000","last_change":"2024-01-27T01:06:26.870626+0000","last_active":"2024-01-27T01:06:26.876472+0000","last_peered":"2024-01-27T01:06:26.876472+0000","last_clean":"2024-01-27T01:06:26.876472+0000","last_became_active":"2024-01-27T01:06:26.870457+0000","last_became_peered":"2024-01-27T01:06:26.870457+0000","last_unstale":"2024-01-27T01:06:26.876472+0000","last_undegraded":"2024-01-27T01:06:26.876472+0000","last_fullsized":"2024-01-27T01:06:26.876472+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:42.766155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.865136+0000","last_change":"2024-01-27T01:06:25.856035+0000","last_active":"2024-01-27T01:06:26.865136+0000","last_peered":"2024-01-27T01:06:26.865136+0000","last_clean":"2024-01-27T01:06:26.865136+0000","last_became_active":"2024-01-27T01:06:25.855762+0000","last_became_peered":"2024-01-27T01:06:25.855762+0000","last_unstale":"2024-01-27T01:06:26.865136+0000","last_undegraded":"2024-01-27T01:06:26.865136+0000","last_fullsized":"2024-01-27T01:06:26.865136+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:51:44.378880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.881016+0000","last_change":"2024-01-27T01:06:26.879306+0000","last_active":"2024-01-27T01:06:26.881016+0000","last_peered":"2024-01-27T01:06:26.881016+0000","last_clean":"2024-01-27T01:06:26.881016+0000","last_became_active":"2024-01-27T01:06:26.879071+0000","last_became_peered":"2024-01-27T01:06:26.879071+0000","last_unstale":"2024-01-27T01:06:26.881016+0000","last_undegraded":"2024-01-27T01:06:26.881016+0000","last_fullsized":"2024-01-27T01:06:26.881016+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:50:46.965359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.857104+0000","last_change":"2024-01-27T01:06:25.865235+0000","last_active":"2024-01-27T01:06:26.857104+0000","last_peered":"2024-01-27T01:06:26.857104+0000","last_clean":"2024-01-27T01:06:26.857104+0000","last_became_active":"2024-01-27T01:06:25.864872+0000","last_became_peered":"2024-01-27T01:06:25.864872+0000","last_unstale":"2024-01-27T01:06:26.857104+0000","last_undegraded":"2024-01-27T01:06:26.857104+0000","last_fullsized":"2024-01-27T01:06:26.857104+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:53:38.618410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.878104+0000","last_change":"2024-01-27T01:06:26.873783+0000","last_active":"2024-01-27T01:06:26.878104+0000","last_peered":"2024-01-27T01:06:26.878104+0000","last_clean":"2024-01-27T01:06:26.878104+0000","last_became_active":"2024-01-27T01:06:26.873490+0000","last_became_peered":"2024-01-27T01:06:26.873490+0000","last_unstale":"2024-01-27T01:06:26.878104+0000","last_undegraded":"2024-01-27T01:06:26.878104+0000","last_fullsized":"2024-01-27T01:06:26.878104+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:32.691743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"222'1","reported_seq":11,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.888523+0000","last_change":"2024-01-27T01:06:25.858609+0000","last_active":"2024-01-27T01:06:26.888523+0000","last_peered":"2024-01-27T01:06:26.888523+0000","last_clean":"2024-01-27T01:06:26.888523+0000","last_became_active":"2024-01-27T01:06:25.857638+0000","last_became_peered":"2024-01-27T01:06:25.857638+0000","last_unstale":"2024-01-27T01:06:26.888523+0000","last_undegraded":"2024-01-27T01:06:26.888523+0000","last_fullsized":"2024-01-27T01:06:26.888523+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:35:59.109571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"213'8750","reported_seq":9664,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.859832+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:06:26.859832+0000","last_peered":"2024-01-27T01:06:26.859832+0000","last_clean":"2024-01-27T01:06:26.859832+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:06:26.859832+0000","last_undegraded":"2024-01-27T01:06:26.859832+0000","last_fullsized":"2024-01-27T01:06:26.859832+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8750,"log_dups_size":0,"ondisk_log_size":8750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":630,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":630,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8750,"ondisk_log_size":8750,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576002,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300620,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071220,"statfs":{"total":96636764160,"available":96328929280,"internally_reserved":0,"allocated":3411968,"data_stored":2065684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63200000000000001,"15min":0.61399999999999999},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.312},"max":{"1min":0.82099999999999995,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.63700000000000001,"15min":0.623},"min":{"1min":0.498,"5min":0.39900000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":0.879,"15min":0.879},"last":0.73199999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.623,"15min":0.61599999999999999},"min":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65600000000000003,"15min":0.63700000000000001},"min":{"1min":0.505,"5min":0.42099999999999999,"15min":0.34300000000000003},"max":{"1min":0.76800000000000002,"5min":0.84099999999999997,"15min":0.84399999999999997},"last":0.60799999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63400000000000001,"15min":0.61399999999999999},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82099999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64500000000000002,"15min":0.63600000000000001},"min":{"1min":0.49399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79400000000000004,"5min":0.872,"15min":0.88900000000000001},"last":0.69499999999999995}]},{"osd":3,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.628,"15min":0.62},"min":{"1min":0.52100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79900000000000004,"5min":0.876,"15min":0.876},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66800000000000004,"15min":0.64700000000000002},"min":{"1min":0.53200000000000003,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84599999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.75}]},{"osd":4,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.50900000000000001,"15min":0.48999999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.29999999999999999},"max":{"1min":0.73399999999999999,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.51900000000000002},"min":{"1min":0.35699999999999998,"5min":0.34699999999999998,"15min":0.32200000000000001},"max":{"1min":0.77500000000000002,"5min":0.80300000000000005,"15min":0.86599999999999999},"last":0.629}]},{"osd":5,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55800000000000005,"15min":0.52600000000000002},"min":{"1min":0.40999999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.753,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57999999999999996,"15min":0.56499999999999995},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.33700000000000002},"max":{"1min":0.76700000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.79200000000000004}]},{"osd":6,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60999999999999999,"15min":0.58899999999999997},"min":{"1min":0.42499999999999999,"5min":0.42199999999999999,"15min":0.37},"max":{"1min":0.81100000000000005,"5min":0.83999999999999997,"15min":0.84799999999999998},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61299999999999999,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.39100000000000001,"15min":0.35699999999999998},"max":{"1min":0.76100000000000001,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.71999999999999997}]}]},{"osd":6,"up_from":13,"seq":55834576002,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308232,"kb_used_data":2752,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063608,"statfs":{"total":96636764160,"available":96321134592,"internally_reserved":0,"allocated":2818048,"data_stored":1475800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59899999999999998,"15min":0.60699999999999998},"min":{"1min":0.44900000000000001,"5min":0.40200000000000002,"15min":0.36399999999999999},"max":{"1min":0.77800000000000002,"5min":0.81999999999999995,"15min":0.86399999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61699999999999999,"15min":0.61799999999999999},"min":{"1min":0.48299999999999998,"5min":0.432,"15min":0.376},"max":{"1min":0.78500000000000003,"5min":0.82899999999999996,"15min":0.86299999999999999},"last":0.76300000000000001}]},{"osd":1,"last update":"Sat Jan 27 01:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.504,"5min":0.44500000000000001,"15min":0.41499999999999998},"max":{"1min":0.72399999999999998,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.60999999999999999,"15min":0.621},"min":{"1min":0.53000000000000003,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.70899999999999996,"5min":0.80900000000000005,"15min":0.83999999999999997},"last":0.64000000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.51300000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72699999999999998,"5min":0.79800000000000004,"15min":0.82399999999999995},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61399999999999999,"15min":0.622},"min":{"1min":0.53300000000000003,"5min":0.39700000000000002,"15min":0.377},"max":{"1min":0.76600000000000001,"5min":0.878,"15min":0.878},"last":0.74399999999999999}]},{"osd":3,"last update":"Sat Jan 27 01:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61599999999999999,"15min":0.627},"min":{"1min":0.51800000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.748,"5min":0.82599999999999996,"15min":0.90100000000000002},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.66200000000000003,"15min":0.66100000000000003},"min":{"1min":0.52400000000000002,"5min":0.47299999999999998,"15min":0.42899999999999999},"max":{"1min":0.77700000000000002,"5min":0.84099999999999997,"15min":0.88100000000000001},"last":0.83399999999999996}]},{"osd":4,"last update":"Sat Jan 27 01:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50700000000000001,"15min":0.50600000000000001},"min":{"1min":0.375,"5min":0.33300000000000002,"15min":0.32500000000000001},"max":{"1min":0.73799999999999999,"5min":0.75700000000000001,"15min":0.86599999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.41599999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.75700000000000001,"5min":0.78500000000000003,"15min":0.88900000000000001},"last":0.62}]},{"osd":5,"last update":"Sat Jan 27 01:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51700000000000002,"15min":0.52400000000000002},"min":{"1min":0.39900000000000002,"5min":0.33600000000000002,"15min":0.31900000000000001},"max":{"1min":0.71099999999999997,"5min":0.75900000000000001,"15min":0.80500000000000005},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58199999999999996,"15min":0.57299999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.28899999999999998},"max":{"1min":0.79200000000000004,"5min":0.84899999999999998,"15min":0.874},"last":0.77700000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.41199999999999998,"5min":0.38200000000000001,"15min":0.35799999999999998},"max":{"1min":0.80400000000000005,"5min":0.83499999999999996,"15min":0.92900000000000005},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.45100000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.70699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834576002,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300024,"kb_used_data":2736,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071816,"statfs":{"total":96636764160,"available":96329539584,"internally_reserved":0,"allocated":2801664,"data_stored":1473710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.41099999999999998,"15min":0.42099999999999999},"min":{"1min":0.26400000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.64400000000000002,"5min":0.66100000000000003,"15min":0.69599999999999995},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44500000000000001,"15min":0.45500000000000002},"min":{"1min":0.34000000000000002,"5min":0.28399999999999997,"15min":0.26800000000000002},"max":{"1min":0.57999999999999996,"5min":0.73999999999999999,"15min":0.76600000000000001},"last":0.41099999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.439,"15min":0.438},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.26200000000000001},"max":{"1min":0.748,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.47599999999999998,"15min":0.48999999999999999},"min":{"1min":0.379,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.67400000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44500000000000001}]},{"osd":3,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52300000000000002,"15min":0.53300000000000003},"min":{"1min":0.39300000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73699999999999999,"5min":0.84799999999999998,"15min":1.2130000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.51400000000000001},"min":{"1min":0.41699999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68200000000000005,"5min":1.125,"15min":1.22},"last":0.35899999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.67200000000000004,"15min":0.67400000000000004},"min":{"1min":0.42099999999999999,"5min":0.41099999999999998,"15min":0.39800000000000002},"max":{"1min":0.82699999999999996,"5min":1.242,"15min":1.242},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69099999999999995,"15min":0.69599999999999995},"min":{"1min":0.53700000000000003,"5min":0.496,"15min":0.35699999999999998},"max":{"1min":0.90600000000000003,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.66600000000000004}]},{"osd":5,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73499999999999999,"15min":0.70899999999999996},"min":{"1min":0.53300000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.876,"5min":5.444,"15min":5.444},"last":0.628},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.71699999999999997,"15min":0.70799999999999996},"min":{"1min":0.502,"5min":0.47599999999999998,"15min":0.435},"max":{"1min":0.97799999999999998,"5min":2.6669999999999998,"15min":2.6669999999999998},"last":0.57999999999999996}]},{"osd":6,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.74199999999999999,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":0.93899999999999995,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.81299999999999994,"15min":0.745},"min":{"1min":0.48799999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.02,"5min":11.223000000000001,"15min":11.223000000000001},"last":0.64600000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.752,"15min":0.75600000000000001},"min":{"1min":0.58999999999999997,"5min":0.52800000000000002,"15min":0.47599999999999998},"max":{"1min":0.96299999999999997,"5min":2.9660000000000002,"15min":2.9660000000000002},"last":0.748},{"interface":"front","average":{"1min":0.752,"5min":0.77700000000000002,"15min":0.76800000000000002},"min":{"1min":0.58799999999999997,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.94499999999999995,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.77000000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576001,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316992,"kb_used_data":3320,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94054848,"statfs":{"total":96636764160,"available":96312164352,"internally_reserved":0,"allocated":3399680,"data_stored":2064168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42899999999999999,"15min":0.42099999999999999},"min":{"1min":0.33300000000000002,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":0.61099999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46800000000000003,"15min":0.46000000000000002},"min":{"1min":0.36299999999999999,"5min":0.313,"15min":0.26200000000000001},"max":{"1min":0.65600000000000003,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.35199999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.43099999999999999,"15min":0.435},"min":{"1min":0.31,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.76800000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.48999999999999999},"min":{"1min":0.314,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65500000000000003,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.42399999999999999}]},{"osd":3,"last update":"Sat Jan 27 01:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.498,"15min":0.48499999999999999},"min":{"1min":0.39500000000000002,"5min":0.33300000000000002,"15min":0.30499999999999999},"max":{"1min":0.68999999999999995,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.316},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.51400000000000001,"15min":0.502},"min":{"1min":0.38600000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.44}]},{"osd":4,"last update":"Sat Jan 27 01:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.68000000000000005,"15min":0.67600000000000005},"min":{"1min":0.52000000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82699999999999996,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68000000000000005,"15min":0.68200000000000005},"min":{"1min":0.50900000000000001,"5min":0.47099999999999997,"15min":0.437},"max":{"1min":0.98999999999999999,"5min":1.002,"15min":1.002},"last":0.67900000000000005}]},{"osd":5,"last update":"Sat Jan 27 01:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.68799999999999994,"15min":0.69699999999999995},"min":{"1min":0.52700000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89300000000000002,"5min":1.093,"15min":2.165},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.71399999999999997,"15min":0.71299999999999997},"min":{"1min":0.59299999999999997,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.96699999999999997,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.58599999999999997}]},{"osd":6,"last update":"Sat Jan 27 01:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70699999999999996,"15min":0.71199999999999997},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.95199999999999996,"5min":0.98899999999999999,"15min":1.0860000000000001},"last":0.622},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73599999999999999,"15min":0.73999999999999999},"min":{"1min":0.56899999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.96899999999999997,"5min":1.0429999999999999,"15min":1.425},"last":0.749}]},{"osd":7,"last update":"Sat Jan 27 01:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.75900000000000001,"15min":0.75700000000000001},"min":{"1min":0.52000000000000002,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0169999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.77100000000000002,"15min":0.77000000000000002},"min":{"1min":0.59999999999999998,"5min":0.51100000000000001,"15min":0.48499999999999999},"max":{"1min":1.01,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.66700000000000004}]}]},{"osd":2,"up_from":13,"seq":55834576001,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312316,"kb_used_data":2740,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059524,"statfs":{"total":96636764160,"available":96316952576,"internally_reserved":0,"allocated":2805760,"data_stored":1473800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40500000000000003,"15min":0.40799999999999997},"min":{"1min":0.26600000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.46000000000000002,"5min":0.746,"15min":0.746},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.44500000000000001,"15min":0.44400000000000001},"min":{"1min":0.32300000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.59099999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.41199999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.435,"15min":0.44500000000000001},"min":{"1min":0.32000000000000001,"5min":0.29099999999999998,"15min":0.27000000000000002},"max":{"1min":0.61299999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.439,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.55600000000000005,"5min":0.73599999999999999,"15min":0.78100000000000003},"last":0.51500000000000001}]},{"osd":3,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.498,"15min":0.497},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.33200000000000002},"max":{"1min":0.627,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.496},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48699999999999999,"15min":0.49199999999999999},"min":{"1min":0.373,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":0.59099999999999997,"5min":0.66600000000000004,"15min":0.75700000000000001},"last":0.47099999999999997}]},{"osd":4,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68200000000000005,"15min":0.68000000000000005},"min":{"1min":0.52800000000000002,"5min":0.495,"15min":0.442},"max":{"1min":0.92700000000000005,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68999999999999995,"15min":0.69399999999999995},"min":{"1min":0.50900000000000001,"5min":0.47099999999999997,"15min":0.378},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":1.107},"last":0.70899999999999996}]},{"osd":5,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.69099999999999995,"15min":0.70299999999999996},"min":{"1min":0.52400000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.89500000000000002,"15min":1.167},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70899999999999996,"15min":0.70999999999999996},"min":{"1min":0.47499999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89600000000000002,"5min":0.97499999999999998,"15min":1.05},"last":0.65400000000000003}]},{"osd":6,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71499999999999997,"15min":0.71199999999999997},"min":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.45500000000000002},"max":{"1min":0.81399999999999995,"5min":0.996,"15min":1.077},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70499999999999996,"15min":0.71999999999999997},"min":{"1min":0.497,"5min":0.497,"15min":0.39200000000000002},"max":{"1min":0.876,"5min":0.96699999999999997,"15min":1.135},"last":0.67100000000000004}]},{"osd":7,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.749,"15min":0.75},"min":{"1min":0.56000000000000005,"5min":0.50800000000000001,"15min":0.47099999999999997},"max":{"1min":0.94099999999999995,"5min":0.97399999999999998,"15min":1.198},"last":0.745},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.76500000000000001,"15min":0.76900000000000002},"min":{"1min":0.55000000000000004,"5min":0.53100000000000003,"15min":0.48199999999999998},"max":{"1min":0.94599999999999995,"5min":1.028,"15min":1.0880000000000001},"last":0.78600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576000,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304132,"kb_used_data":2748,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067708,"statfs":{"total":96636764160,"available":96325332992,"internally_reserved":0,"allocated":2813952,"data_stored":1473836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.44700000000000001,"15min":0.42299999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.54800000000000004,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.372,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.63500000000000001,"5min":0.66700000000000004,"15min":0.84699999999999998},"last":0.438}]},{"osd":1,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.63100000000000001,"15min":0.53500000000000003},"min":{"1min":0.34499999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.629,"5min":15.391,"15min":15.391},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.54600000000000004,"15min":0.52400000000000002},"min":{"1min":0.36699999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65100000000000002,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.53700000000000003}]},{"osd":2,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54600000000000004,"15min":0.51700000000000002},"min":{"1min":0.36499999999999999,"5min":0.36399999999999999,"15min":0.34599999999999997},"max":{"1min":0.62,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.53100000000000003,"15min":0.50700000000000001},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.60699999999999998,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":1.5529999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.74199999999999999,"15min":0.72099999999999997},"min":{"1min":0.497,"5min":0.47699999999999998,"15min":0.439},"max":{"1min":0.88600000000000001,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.71999999999999997,"15min":0.70799999999999996},"min":{"1min":0.46300000000000002,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":0.94399999999999995,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.73899999999999999}]},{"osd":5,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.74299999999999999,"15min":0.71699999999999997},"min":{"1min":0.51100000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.83999999999999997,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72799999999999998,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.97699999999999998,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.78000000000000003}]},{"osd":6,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.754,"15min":0.72599999999999998},"min":{"1min":0.53500000000000003,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.81200000000000006,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.748,"15min":0.74299999999999999},"min":{"1min":0.56100000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.92700000000000005,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.81699999999999995}]},{"osd":7,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.75600000000000001,"15min":0.73799999999999999},"min":{"1min":0.58599999999999997,"5min":0.52500000000000002,"15min":0.46700000000000003},"max":{"1min":1.0029999999999999,"5min":4.569,"15min":4.569},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.80600000000000005,"15min":0.78500000000000003},"min":{"1min":0.55700000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0169999999999999,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.64800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834576002,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312316,"kb_used_data":2740,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059524,"statfs":{"total":96636764160,"available":96316952576,"internally_reserved":0,"allocated":2805760,"data_stored":1474264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.46899999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.84199999999999997},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.61499999999999999,"15min":0.61399999999999999},"min":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78700000000000003,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.60999999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61899999999999999,"15min":0.627},"min":{"1min":0.46100000000000002,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.76400000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61499999999999999,"15min":0.62},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.79800000000000004,"5min":0.874,"15min":0.874},"last":0.59799999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.59999999999999998},"min":{"1min":0.42399999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.88700000000000001},"last":0.624},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.51100000000000001,"5min":0.42899999999999999,"15min":0.42799999999999999},"max":{"1min":0.81100000000000005,"5min":0.81499999999999995,"15min":0.89200000000000002},"last":0.74299999999999999}]},{"osd":3,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.61599999999999999,"15min":0.623},"min":{"1min":0.38800000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.83299999999999996,"5min":0.83599999999999997,"15min":0.84199999999999997},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.627},"min":{"1min":0.41299999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.88300000000000001},"last":0.66000000000000003}]},{"osd":5,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.75800000000000001,"15min":0.85899999999999999},"last":0.504},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.502},"min":{"1min":0.376,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.878},"last":0.53000000000000003}]},{"osd":6,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34999999999999998},"max":{"1min":0.63200000000000001,"5min":0.70099999999999996,"15min":0.88700000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.70699999999999996,"5min":0.79200000000000004,"15min":0.81499999999999995},"last":0.64500000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.42399999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72599999999999998,"5min":0.80400000000000005,"15min":0.90200000000000002},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59499999999999997,"15min":0.59399999999999997},"min":{"1min":0.41599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.73799999999999999,"5min":0.78700000000000003,"15min":0.92600000000000005},"last":0.70199999999999996}]}]},{"osd":5,"up_from":13,"seq":55834576002,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312320,"kb_used_data":2744,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059520,"statfs":{"total":96636764160,"available":96316948480,"internally_reserved":0,"allocated":2809856,"data_stored":1473766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.54500000000000004,"5min":0.40500000000000003,"15min":0.27300000000000002},"max":{"1min":0.70999999999999996,"5min":1.006,"15min":1.006},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81499999999999995,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.66900000000000004}]},{"osd":1,"last update":"Sat Jan 27 01:05:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.65200000000000002,"15min":0.65600000000000003},"min":{"1min":0.53600000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80300000000000005,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.63},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.67700000000000005,"15min":0.67100000000000004},"min":{"1min":0.57099999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.79100000000000004,"5min":1.212,"15min":1.212},"last":0.64100000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:05:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.59599999999999997},"min":{"1min":0.434,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67900000000000005,"5min":1.631,"15min":1.631},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.623},"min":{"1min":0.56699999999999995,"5min":0.38800000000000001,"15min":0.34000000000000002},"max":{"1min":0.72399999999999998,"5min":0.82599999999999996,"15min":0.83299999999999996},"last":0.51100000000000001}]},{"osd":3,"last update":"Sat Jan 27 01:05:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60599999999999998,"15min":0.59999999999999998},"min":{"1min":0.40600000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.51500000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.754,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.57299999999999995}]},{"osd":4,"last update":"Sat Jan 27 01:05:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.505,"15min":0.48599999999999999},"min":{"1min":0.377,"5min":0.32200000000000001,"15min":0.317},"max":{"1min":0.54000000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.53100000000000003,"15min":0.51100000000000001},"min":{"1min":0.35599999999999998,"5min":0.35199999999999998,"15min":0.33900000000000002},"max":{"1min":0.68999999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.48999999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:05:29 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52500000000000002,"15min":0.503},"min":{"1min":0.315,"5min":0.315,"15min":0.314},"max":{"1min":0.59599999999999997,"5min":1.732,"15min":1.732},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55800000000000005,"15min":0.54200000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70099999999999996,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.55700000000000005}]},{"osd":7,"last update":"Sat Jan 27 01:05:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.39400000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.63,"15min":0.60999999999999999},"min":{"1min":0.442,"5min":0.40799999999999997,"15min":0.39100000000000001},"max":{"1min":0.77700000000000002,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.61299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:06:31.061 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:06:31.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.232+0000 7f71df6ec700 1 Processor -- start 2024-01-27T01:06:31.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.232+0000 7f71df6ec700 1 -- start start 2024-01-27T01:06:31.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.233+0000 7f71df6ec700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71d81541c0 0x7f71d80cc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:31.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.233+0000 7f71df6ec700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71d80cc770 0x7f71d806eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:31.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.233+0000 7f71df6ec700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f71d806cdb0 0x7f71d806f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:31.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.233+0000 7f71df6ec700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d8156d00 con 0x7f71d81541c0 2024-01-27T01:06:31.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.233+0000 7f71df6ec700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d8157560 con 0x7f71d806cdb0 2024-01-27T01:06:31.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.233+0000 7f71df6ec700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d80ccc10 con 0x7f71d80cc770 2024-01-27T01:06:31.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.233+0000 7f71dd488700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71d81541c0 0x7f71d80cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:31.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.233+0000 7f71dd488700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71d81541c0 0x7f71d80cc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33724/0 (socket says 172.21.15.46:33724) 2024-01-27T01:06:31.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.233+0000 7f71dd488700 1 -- 172.21.15.46:0/168594655 learned_addr learned my addr 172.21.15.46:0/168594655 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:31.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.234+0000 7f71dcc87700 1 --2- 172.21.15.46:0/168594655 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71d80cc770 0x7f71d806eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:31.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.234+0000 7f71ddc89700 1 --2- 172.21.15.46:0/168594655 >> v2:172.21.15.106:3300/0 conn(0x7f71d806cdb0 0x7f71d806f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:31.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.234+0000 7f71dd488700 1 -- 172.21.15.46:0/168594655 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71d80cc770 msgr2=0x7f71d806eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:31.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.234+0000 7f71dd488700 1 --2- 172.21.15.46:0/168594655 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71d80cc770 0x7f71d806eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.234+0000 7f71dd488700 1 -- 172.21.15.46:0/168594655 >> v2:172.21.15.106:3300/0 conn(0x7f71d806cdb0 msgr2=0x7f71d806f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:31.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.234+0000 7f71dd488700 1 --2- 172.21.15.46:0/168594655 >> v2:172.21.15.106:3300/0 conn(0x7f71d806cdb0 0x7f71d806f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.234+0000 7f71dd488700 1 -- 172.21.15.46:0/168594655 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d8156f10 con 0x7f71d81541c0 2024-01-27T01:06:31.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.234+0000 7f71dd488700 1 --2- 172.21.15.46:0/168594655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d81541c0 0x7f71d80cc230 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f71d80c7a10 tx=0x7f71cc00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94afd32df3e5f95b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:31.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.234+0000 7f71cbfff700 1 -- 172.21.15.46:0/168594655 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc00db40 con 0x7f71d81541c0 2024-01-27T01:06:31.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.234+0000 7f71cbfff700 1 -- 172.21.15.46:0/168594655 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71cc00dca0 con 0x7f71d81541c0 2024-01-27T01:06:31.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.234+0000 7f71cbfff700 1 -- 172.21.15.46:0/168594655 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc0196d0 con 0x7f71d81541c0 2024-01-27T01:06:31.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.235+0000 7f71df6ec700 1 -- 172.21.15.46:0/168594655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d81541c0 msgr2=0x7f71d80cc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:31.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.235+0000 7f71df6ec700 1 --2- 172.21.15.46:0/168594655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d81541c0 0x7f71d80cc230 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f71d80c7a10 tx=0x7f71cc00b4f0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.235+0000 7f71df6ec700 1 -- 172.21.15.46:0/168594655 shutdown_connections 2024-01-27T01:06:31.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.235+0000 7f71df6ec700 1 --2- 172.21.15.46:0/168594655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d81541c0 0x7f71d80cc230 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.235+0000 7f71df6ec700 1 --2- 172.21.15.46:0/168594655 >> v2:172.21.15.106:3300/0 conn(0x7f71d806cdb0 0x7f71d806f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.235+0000 7f71df6ec700 1 --2- 172.21.15.46:0/168594655 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71d80cc770 0x7f71d806eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.235+0000 7f71df6ec700 1 -- 172.21.15.46:0/168594655 >> 172.21.15.46:0/168594655 conn(0x7f71d80cdf10 msgr2=0x7f71d80c9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:31.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.235+0000 7f71df6ec700 1 -- 172.21.15.46:0/168594655 shutdown_connections 2024-01-27T01:06:31.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.235+0000 7f71df6ec700 1 -- 172.21.15.46:0/168594655 wait complete. 2024-01-27T01:06:31.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.236+0000 7f71df6ec700 1 Processor -- start 2024-01-27T01:06:31.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.236+0000 7f71df6ec700 1 -- start start 2024-01-27T01:06:31.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.237+0000 7f71df6ec700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71d806cdb0 0x7f71d814daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:31.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.237+0000 7f71df6ec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d80cc770 0x7f71d8156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:31.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.237+0000 7f71df6ec700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71d81541c0 0x7f71d81565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:31.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.237+0000 7f71df6ec700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d80b2c60 con 0x7f71d80cc770 2024-01-27T01:06:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.237+0000 7f71df6ec700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d80b2330 con 0x7f71d806cdb0 2024-01-27T01:06:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.237+0000 7f71df6ec700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d80b2610 con 0x7f71d81541c0 2024-01-27T01:06:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.237+0000 7f71dcc87700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d80cc770 0x7f71d8156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.237+0000 7f71dcc87700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d80cc770 0x7f71d8156080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33732/0 (socket says 172.21.15.46:33732) 2024-01-27T01:06:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.237+0000 7f71dcc87700 1 -- 172.21.15.46:0/3985794405 learned_addr learned my addr 172.21.15.46:0/3985794405 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.237+0000 7f71ddc89700 1 --2- 172.21.15.46:0/3985794405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71d81541c0 0x7f71d81565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.237+0000 7f71dd488700 1 --2- 172.21.15.46:0/3985794405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71d806cdb0 0x7f71d814daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.238+0000 7f71dcc87700 1 -- 172.21.15.46:0/3985794405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71d81541c0 msgr2=0x7f71d81565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.238+0000 7f71dcc87700 1 --2- 172.21.15.46:0/3985794405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71d81541c0 0x7f71d81565c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.238+0000 7f71dcc87700 1 -- 172.21.15.46:0/3985794405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71d806cdb0 msgr2=0x7f71d814daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.238+0000 7f71dcc87700 1 --2- 172.21.15.46:0/3985794405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71d806cdb0 0x7f71d814daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.238+0000 7f71dcc87700 1 -- 172.21.15.46:0/3985794405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71cc00b790 con 0x7f71d80cc770 2024-01-27T01:06:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.238+0000 7f71dcc87700 1 --2- 172.21.15.46:0/3985794405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d80cc770 0x7f71d8156080 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f71d400ec50 tx=0x7f71d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:31.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.238+0000 7f71c97fa700 1 -- 172.21.15.46:0/3985794405 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d4009690 con 0x7f71d80cc770 2024-01-27T01:06:31.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.238+0000 7f71c97fa700 1 -- 172.21.15.46:0/3985794405 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d400c850 con 0x7f71d80cc770 2024-01-27T01:06:31.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.238+0000 7f71df6ec700 1 -- 172.21.15.46:0/3985794405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d80b28f0 con 0x7f71d80cc770 2024-01-27T01:06:31.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.238+0000 7f71c97fa700 1 -- 172.21.15.46:0/3985794405 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d400cc10 con 0x7f71d80cc770 2024-01-27T01:06:31.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.238+0000 7f71df6ec700 1 -- 172.21.15.46:0/3985794405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d814e030 con 0x7f71d80cc770 2024-01-27T01:06:31.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.239+0000 7f71df6ec700 1 -- 172.21.15.46:0/3985794405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d804c710 con 0x7f71d80cc770 2024-01-27T01:06:31.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.239+0000 7f71c97fa700 1 -- 172.21.15.46:0/3985794405 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f71d4013470 con 0x7f71d80cc770 2024-01-27T01:06:31.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.240+0000 7f71c97fa700 1 --2- 172.21.15.46:0/3985794405 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f71bc066f30 0x7f71bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:31.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.240+0000 7f71c97fa700 1 -- 172.21.15.46:0/3985794405 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f71d4084a20 con 0x7f71d80cc770 2024-01-27T01:06:31.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.240+0000 7f71dd488700 1 --2- 172.21.15.46:0/3985794405 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f71bc066f30 0x7f71bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:31.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.241+0000 7f71dd488700 1 --2- 172.21.15.46:0/3985794405 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f71bc066f30 0x7f71bc0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f71d80c7a10 tx=0x7f71cc01b000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:31.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.244+0000 7f71c97fa700 1 -- 172.21.15.46:0/3985794405 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f71d4051960 con 0x7f71d80cc770 2024-01-27T01:06:31.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.441+0000 7f71df6ec700 1 -- 172.21.15.46:0/3985794405 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f71d80b24a0 con 0x7f71bc066f30 2024-01-27T01:06:31.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.450+0000 7f71c97fa700 1 -- 172.21.15.46:0/3985794405 <== mgr.9101 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289869 (secure 0 0 0) 0x7f71d80b24a0 con 0x7f71bc066f30 2024-01-27T01:06:31.452 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:31.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.453+0000 7f71df6ec700 1 -- 172.21.15.46:0/3985794405 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f71bc066f30 msgr2=0x7f71bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:31.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.453+0000 7f71df6ec700 1 --2- 172.21.15.46:0/3985794405 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f71bc066f30 0x7f71bc0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f71d80c7a10 tx=0x7f71cc01b000 comp rx=0 tx=0).stop 2024-01-27T01:06:31.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.453+0000 7f71df6ec700 1 -- 172.21.15.46:0/3985794405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d80cc770 msgr2=0x7f71d8156080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:31.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.453+0000 7f71df6ec700 1 --2- 172.21.15.46:0/3985794405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d80cc770 0x7f71d8156080 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f71d400ec50 tx=0x7f71d40091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.454+0000 7f71df6ec700 1 -- 172.21.15.46:0/3985794405 shutdown_connections 2024-01-27T01:06:31.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.454+0000 7f71df6ec700 1 --2- 172.21.15.46:0/3985794405 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f71bc066f30 0x7f71bc0693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.454+0000 7f71df6ec700 1 --2- 172.21.15.46:0/3985794405 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71d81541c0 0x7f71d81565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.454+0000 7f71df6ec700 1 --2- 172.21.15.46:0/3985794405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71d806cdb0 0x7f71d814daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.454+0000 7f71df6ec700 1 --2- 172.21.15.46:0/3985794405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d80cc770 0x7f71d8156080 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.454+0000 7f71df6ec700 1 -- 172.21.15.46:0/3985794405 >> 172.21.15.46:0/3985794405 conn(0x7f71d80cdf10 msgr2=0x7f71d8155260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:31.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.454+0000 7f71df6ec700 1 -- 172.21.15.46:0/3985794405 shutdown_connections 2024-01-27T01:06:31.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.454+0000 7f71df6ec700 1 -- 172.21.15.46:0/3985794405 wait complete. 2024-01-27T01:06:31.456 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:06:31.466 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-01-27T01:06:30.730734+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":674,"num_write_kb":3475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8788,"ondisk_log_size":8788,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2466952,"kb_used_data":23112,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752507768,"statfs":{"total":773094113280,"available":770567954432,"internally_reserved":0,"allocated":23666688,"data_stored":12975028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.889576"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.878856+0000","last_change":"2024-01-27T01:06:26.873715+0000","last_active":"2024-01-27T01:06:26.878856+0000","last_peered":"2024-01-27T01:06:26.878856+0000","last_clean":"2024-01-27T01:06:26.878856+0000","last_became_active":"2024-01-27T01:06:26.873430+0000","last_became_peered":"2024-01-27T01:06:26.873430+0000","last_unstale":"2024-01-27T01:06:26.878856+0000","last_undegraded":"2024-01-27T01:06:26.878856+0000","last_fullsized":"2024-01-27T01:06:26.878856+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:44.342963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.875217+0000","last_change":"2024-01-27T01:06:26.871135+0000","last_active":"2024-01-27T01:06:26.875217+0000","last_peered":"2024-01-27T01:06:26.875217+0000","last_clean":"2024-01-27T01:06:26.875217+0000","last_became_active":"2024-01-27T01:06:26.870854+0000","last_became_peered":"2024-01-27T01:06:26.870854+0000","last_unstale":"2024-01-27T01:06:26.875217+0000","last_undegraded":"2024-01-27T01:06:26.875217+0000","last_fullsized":"2024-01-27T01:06:26.875217+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:12:59.875769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.878303+0000","last_change":"2024-01-27T01:06:26.868462+0000","last_active":"2024-01-27T01:06:26.878303+0000","last_peered":"2024-01-27T01:06:26.878303+0000","last_clean":"2024-01-27T01:06:26.878303+0000","last_became_active":"2024-01-27T01:06:26.868203+0000","last_became_peered":"2024-01-27T01:06:26.868203+0000","last_unstale":"2024-01-27T01:06:26.878303+0000","last_undegraded":"2024-01-27T01:06:26.878303+0000","last_fullsized":"2024-01-27T01:06:26.878303+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:10.965469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.873487+0000","last_change":"2024-01-27T01:06:26.868177+0000","last_active":"2024-01-27T01:06:26.873487+0000","last_peered":"2024-01-27T01:06:26.873487+0000","last_clean":"2024-01-27T01:06:26.873487+0000","last_became_active":"2024-01-27T01:06:26.867906+0000","last_became_peered":"2024-01-27T01:06:26.867906+0000","last_unstale":"2024-01-27T01:06:26.873487+0000","last_undegraded":"2024-01-27T01:06:26.873487+0000","last_fullsized":"2024-01-27T01:06:26.873487+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:32.271904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.875579+0000","last_change":"2024-01-27T01:06:26.870422+0000","last_active":"2024-01-27T01:06:26.875579+0000","last_peered":"2024-01-27T01:06:26.875579+0000","last_clean":"2024-01-27T01:06:26.875579+0000","last_became_active":"2024-01-27T01:06:26.870117+0000","last_became_peered":"2024-01-27T01:06:26.870117+0000","last_unstale":"2024-01-27T01:06:26.875579+0000","last_undegraded":"2024-01-27T01:06:26.875579+0000","last_fullsized":"2024-01-27T01:06:26.875579+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:42:55.987157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.883527+0000","last_change":"2024-01-27T01:06:26.879291+0000","last_active":"2024-01-27T01:06:26.883527+0000","last_peered":"2024-01-27T01:06:26.883527+0000","last_clean":"2024-01-27T01:06:26.883527+0000","last_became_active":"2024-01-27T01:06:26.879063+0000","last_became_peered":"2024-01-27T01:06:26.879063+0000","last_unstale":"2024-01-27T01:06:26.883527+0000","last_undegraded":"2024-01-27T01:06:26.883527+0000","last_fullsized":"2024-01-27T01:06:26.883527+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:22:23.545688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.878514+0000","last_change":"2024-01-27T01:06:26.872094+0000","last_active":"2024-01-27T01:06:26.878514+0000","last_peered":"2024-01-27T01:06:26.878514+0000","last_clean":"2024-01-27T01:06:26.878514+0000","last_became_active":"2024-01-27T01:06:26.871711+0000","last_became_peered":"2024-01-27T01:06:26.871711+0000","last_unstale":"2024-01-27T01:06:26.878514+0000","last_undegraded":"2024-01-27T01:06:26.878514+0000","last_fullsized":"2024-01-27T01:06:26.878514+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:24:32.821570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884998+0000","last_change":"2024-01-27T01:06:26.881786+0000","last_active":"2024-01-27T01:06:26.884998+0000","last_peered":"2024-01-27T01:06:26.884998+0000","last_clean":"2024-01-27T01:06:26.884998+0000","last_became_active":"2024-01-27T01:06:26.881527+0000","last_became_peered":"2024-01-27T01:06:26.881527+0000","last_unstale":"2024-01-27T01:06:26.884998+0000","last_undegraded":"2024-01-27T01:06:26.884998+0000","last_fullsized":"2024-01-27T01:06:26.884998+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:43.630831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.881095+0000","last_change":"2024-01-27T01:06:26.873409+0000","last_active":"2024-01-27T01:06:26.881095+0000","last_peered":"2024-01-27T01:06:26.881095+0000","last_clean":"2024-01-27T01:06:26.881095+0000","last_became_active":"2024-01-27T01:06:26.868026+0000","last_became_peered":"2024-01-27T01:06:26.868026+0000","last_unstale":"2024-01-27T01:06:26.881095+0000","last_undegraded":"2024-01-27T01:06:26.881095+0000","last_fullsized":"2024-01-27T01:06:26.881095+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:15.664932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884549+0000","last_change":"2024-01-27T01:06:26.876736+0000","last_active":"2024-01-27T01:06:26.884549+0000","last_peered":"2024-01-27T01:06:26.884549+0000","last_clean":"2024-01-27T01:06:26.884549+0000","last_became_active":"2024-01-27T01:06:26.875870+0000","last_became_peered":"2024-01-27T01:06:26.875870+0000","last_unstale":"2024-01-27T01:06:26.884549+0000","last_undegraded":"2024-01-27T01:06:26.884549+0000","last_fullsized":"2024-01-27T01:06:26.884549+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:43.223271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.881062+0000","last_change":"2024-01-27T01:06:26.879535+0000","last_active":"2024-01-27T01:06:26.881062+0000","last_peered":"2024-01-27T01:06:26.881062+0000","last_clean":"2024-01-27T01:06:26.881062+0000","last_became_active":"2024-01-27T01:06:26.879219+0000","last_became_peered":"2024-01-27T01:06:26.879219+0000","last_unstale":"2024-01-27T01:06:26.881062+0000","last_undegraded":"2024-01-27T01:06:26.881062+0000","last_fullsized":"2024-01-27T01:06:26.881062+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:58.495062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.874822+0000","last_change":"2024-01-27T01:06:26.865833+0000","last_active":"2024-01-27T01:06:26.874822+0000","last_peered":"2024-01-27T01:06:26.874822+0000","last_clean":"2024-01-27T01:06:26.874822+0000","last_became_active":"2024-01-27T01:06:26.865525+0000","last_became_peered":"2024-01-27T01:06:26.865525+0000","last_unstale":"2024-01-27T01:06:26.874822+0000","last_undegraded":"2024-01-27T01:06:26.874822+0000","last_fullsized":"2024-01-27T01:06:26.874822+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:42:10.990703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.880927+0000","last_change":"2024-01-27T01:06:26.877777+0000","last_active":"2024-01-27T01:06:26.880927+0000","last_peered":"2024-01-27T01:06:26.880927+0000","last_clean":"2024-01-27T01:06:26.880927+0000","last_became_active":"2024-01-27T01:06:26.877515+0000","last_became_peered":"2024-01-27T01:06:26.877515+0000","last_unstale":"2024-01-27T01:06:26.880927+0000","last_undegraded":"2024-01-27T01:06:26.880927+0000","last_fullsized":"2024-01-27T01:06:26.880927+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:27:14.653781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884229+0000","last_change":"2024-01-27T01:06:26.876674+0000","last_active":"2024-01-27T01:06:26.884229+0000","last_peered":"2024-01-27T01:06:26.884229+0000","last_clean":"2024-01-27T01:06:26.884229+0000","last_became_active":"2024-01-27T01:06:26.876413+0000","last_became_peered":"2024-01-27T01:06:26.876413+0000","last_unstale":"2024-01-27T01:06:26.884229+0000","last_undegraded":"2024-01-27T01:06:26.884229+0000","last_fullsized":"2024-01-27T01:06:26.884229+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:08.822886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.883563+0000","last_change":"2024-01-27T01:06:26.877929+0000","last_active":"2024-01-27T01:06:26.883563+0000","last_peered":"2024-01-27T01:06:26.883563+0000","last_clean":"2024-01-27T01:06:26.883563+0000","last_became_active":"2024-01-27T01:06:26.877679+0000","last_became_peered":"2024-01-27T01:06:26.877679+0000","last_unstale":"2024-01-27T01:06:26.883563+0000","last_undegraded":"2024-01-27T01:06:26.883563+0000","last_fullsized":"2024-01-27T01:06:26.883563+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:53:15.482586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.877545+0000","last_change":"2024-01-27T01:06:26.863380+0000","last_active":"2024-01-27T01:06:26.877545+0000","last_peered":"2024-01-27T01:06:26.877545+0000","last_clean":"2024-01-27T01:06:26.877545+0000","last_became_active":"2024-01-27T01:06:26.863034+0000","last_became_peered":"2024-01-27T01:06:26.863034+0000","last_unstale":"2024-01-27T01:06:26.877545+0000","last_undegraded":"2024-01-27T01:06:26.877545+0000","last_fullsized":"2024-01-27T01:06:26.877545+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:32.271904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.881114+0000","last_change":"2024-01-27T01:06:26.874473+0000","last_active":"2024-01-27T01:06:26.881114+0000","last_peered":"2024-01-27T01:06:26.881114+0000","last_clean":"2024-01-27T01:06:26.881114+0000","last_became_active":"2024-01-27T01:06:26.874203+0000","last_became_peered":"2024-01-27T01:06:26.874203+0000","last_unstale":"2024-01-27T01:06:26.881114+0000","last_undegraded":"2024-01-27T01:06:26.881114+0000","last_fullsized":"2024-01-27T01:06:26.881114+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:33:37.975146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.872176+0000","last_change":"2024-01-27T01:06:26.864890+0000","last_active":"2024-01-27T01:06:26.872176+0000","last_peered":"2024-01-27T01:06:26.872176+0000","last_clean":"2024-01-27T01:06:26.872176+0000","last_became_active":"2024-01-27T01:06:26.864556+0000","last_became_peered":"2024-01-27T01:06:26.864556+0000","last_unstale":"2024-01-27T01:06:26.872176+0000","last_undegraded":"2024-01-27T01:06:26.872176+0000","last_fullsized":"2024-01-27T01:06:26.872176+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:41.033580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.878220+0000","last_change":"2024-01-27T01:06:26.873401+0000","last_active":"2024-01-27T01:06:26.878220+0000","last_peered":"2024-01-27T01:06:26.878220+0000","last_clean":"2024-01-27T01:06:26.878220+0000","last_became_active":"2024-01-27T01:06:26.873094+0000","last_became_peered":"2024-01-27T01:06:26.873094+0000","last_unstale":"2024-01-27T01:06:26.878220+0000","last_undegraded":"2024-01-27T01:06:26.878220+0000","last_fullsized":"2024-01-27T01:06:26.878220+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:24.082749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884181+0000","last_change":"2024-01-27T01:06:26.876823+0000","last_active":"2024-01-27T01:06:26.884181+0000","last_peered":"2024-01-27T01:06:26.884181+0000","last_clean":"2024-01-27T01:06:26.884181+0000","last_became_active":"2024-01-27T01:06:26.876304+0000","last_became_peered":"2024-01-27T01:06:26.876304+0000","last_unstale":"2024-01-27T01:06:26.884181+0000","last_undegraded":"2024-01-27T01:06:26.884181+0000","last_fullsized":"2024-01-27T01:06:26.884181+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:28.378677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.885074+0000","last_change":"2024-01-27T01:06:26.879441+0000","last_active":"2024-01-27T01:06:26.885074+0000","last_peered":"2024-01-27T01:06:26.885074+0000","last_clean":"2024-01-27T01:06:26.885074+0000","last_became_active":"2024-01-27T01:06:26.879148+0000","last_became_peered":"2024-01-27T01:06:26.879148+0000","last_unstale":"2024-01-27T01:06:26.885074+0000","last_undegraded":"2024-01-27T01:06:26.885074+0000","last_fullsized":"2024-01-27T01:06:26.885074+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:05:08.012027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884672+0000","last_change":"2024-01-27T01:06:26.876390+0000","last_active":"2024-01-27T01:06:26.884672+0000","last_peered":"2024-01-27T01:06:26.884672+0000","last_clean":"2024-01-27T01:06:26.884672+0000","last_became_active":"2024-01-27T01:06:26.876056+0000","last_became_peered":"2024-01-27T01:06:26.876056+0000","last_unstale":"2024-01-27T01:06:26.884672+0000","last_undegraded":"2024-01-27T01:06:26.884672+0000","last_fullsized":"2024-01-27T01:06:26.884672+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:38:36.236958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.876121+0000","last_change":"2024-01-27T01:06:26.874471+0000","last_active":"2024-01-27T01:06:26.876121+0000","last_peered":"2024-01-27T01:06:26.876121+0000","last_clean":"2024-01-27T01:06:26.876121+0000","last_became_active":"2024-01-27T01:06:26.874271+0000","last_became_peered":"2024-01-27T01:06:26.874271+0000","last_unstale":"2024-01-27T01:06:26.876121+0000","last_undegraded":"2024-01-27T01:06:26.876121+0000","last_fullsized":"2024-01-27T01:06:26.876121+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:15:11.295547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.875043+0000","last_change":"2024-01-27T01:06:26.867241+0000","last_active":"2024-01-27T01:06:26.875043+0000","last_peered":"2024-01-27T01:06:26.875043+0000","last_clean":"2024-01-27T01:06:26.875043+0000","last_became_active":"2024-01-27T01:06:26.866917+0000","last_became_peered":"2024-01-27T01:06:26.866917+0000","last_unstale":"2024-01-27T01:06:26.875043+0000","last_undegraded":"2024-01-27T01:06:26.875043+0000","last_fullsized":"2024-01-27T01:06:26.875043+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:24:25.818592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.876177+0000","last_change":"2024-01-27T01:06:26.865149+0000","last_active":"2024-01-27T01:06:26.876177+0000","last_peered":"2024-01-27T01:06:26.876177+0000","last_clean":"2024-01-27T01:06:26.876177+0000","last_became_active":"2024-01-27T01:06:26.864853+0000","last_became_peered":"2024-01-27T01:06:26.864853+0000","last_unstale":"2024-01-27T01:06:26.876177+0000","last_undegraded":"2024-01-27T01:06:26.876177+0000","last_fullsized":"2024-01-27T01:06:26.876177+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:10:44.194323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884005+0000","last_change":"2024-01-27T01:06:26.878911+0000","last_active":"2024-01-27T01:06:26.884005+0000","last_peered":"2024-01-27T01:06:26.884005+0000","last_clean":"2024-01-27T01:06:26.884005+0000","last_became_active":"2024-01-27T01:06:26.878637+0000","last_became_peered":"2024-01-27T01:06:26.878637+0000","last_unstale":"2024-01-27T01:06:26.884005+0000","last_undegraded":"2024-01-27T01:06:26.884005+0000","last_fullsized":"2024-01-27T01:06:26.884005+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:33:03.751565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.881049+0000","last_change":"2024-01-27T01:06:26.878202+0000","last_active":"2024-01-27T01:06:26.881049+0000","last_peered":"2024-01-27T01:06:26.881049+0000","last_clean":"2024-01-27T01:06:26.881049+0000","last_became_active":"2024-01-27T01:06:26.877900+0000","last_became_peered":"2024-01-27T01:06:26.877900+0000","last_unstale":"2024-01-27T01:06:26.881049+0000","last_undegraded":"2024-01-27T01:06:26.881049+0000","last_fullsized":"2024-01-27T01:06:26.881049+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:42.766155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.875568+0000","last_change":"2024-01-27T01:06:26.870649+0000","last_active":"2024-01-27T01:06:26.875568+0000","last_peered":"2024-01-27T01:06:26.875568+0000","last_clean":"2024-01-27T01:06:26.875568+0000","last_became_active":"2024-01-27T01:06:26.870433+0000","last_became_peered":"2024-01-27T01:06:26.870433+0000","last_unstale":"2024-01-27T01:06:26.875568+0000","last_undegraded":"2024-01-27T01:06:26.875568+0000","last_fullsized":"2024-01-27T01:06:26.875568+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:11:27.402943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.873029+0000","last_change":"2024-01-27T01:06:26.872013+0000","last_active":"2024-01-27T01:06:26.873029+0000","last_peered":"2024-01-27T01:06:26.873029+0000","last_clean":"2024-01-27T01:06:26.873029+0000","last_became_active":"2024-01-27T01:06:26.867577+0000","last_became_peered":"2024-01-27T01:06:26.867577+0000","last_unstale":"2024-01-27T01:06:26.873029+0000","last_undegraded":"2024-01-27T01:06:26.873029+0000","last_fullsized":"2024-01-27T01:06:26.873029+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:27.604574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.874991+0000","last_change":"2024-01-27T01:06:26.871676+0000","last_active":"2024-01-27T01:06:26.874991+0000","last_peered":"2024-01-27T01:06:26.874991+0000","last_clean":"2024-01-27T01:06:26.874991+0000","last_became_active":"2024-01-27T01:06:26.871423+0000","last_became_peered":"2024-01-27T01:06:26.871423+0000","last_unstale":"2024-01-27T01:06:26.874991+0000","last_undegraded":"2024-01-27T01:06:26.874991+0000","last_fullsized":"2024-01-27T01:06:26.874991+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:29:58.146876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.880900+0000","last_change":"2024-01-27T01:06:26.876343+0000","last_active":"2024-01-27T01:06:26.880900+0000","last_peered":"2024-01-27T01:06:26.880900+0000","last_clean":"2024-01-27T01:06:26.880900+0000","last_became_active":"2024-01-27T01:06:26.875570+0000","last_became_peered":"2024-01-27T01:06:26.875570+0000","last_unstale":"2024-01-27T01:06:26.880900+0000","last_undegraded":"2024-01-27T01:06:26.880900+0000","last_fullsized":"2024-01-27T01:06:26.880900+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:07:51.640059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.872421+0000","last_change":"2024-01-27T01:06:26.869029+0000","last_active":"2024-01-27T01:06:26.872421+0000","last_peered":"2024-01-27T01:06:26.872421+0000","last_clean":"2024-01-27T01:06:26.872421+0000","last_became_active":"2024-01-27T01:06:26.868793+0000","last_became_peered":"2024-01-27T01:06:26.868793+0000","last_unstale":"2024-01-27T01:06:26.872421+0000","last_undegraded":"2024-01-27T01:06:26.872421+0000","last_fullsized":"2024-01-27T01:06:26.872421+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:10.965469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.874093+0000","last_change":"2024-01-27T01:06:26.869588+0000","last_active":"2024-01-27T01:06:26.874093+0000","last_peered":"2024-01-27T01:06:26.874093+0000","last_clean":"2024-01-27T01:06:26.874093+0000","last_became_active":"2024-01-27T01:06:26.869264+0000","last_became_peered":"2024-01-27T01:06:26.869264+0000","last_unstale":"2024-01-27T01:06:26.874093+0000","last_undegraded":"2024-01-27T01:06:26.874093+0000","last_fullsized":"2024-01-27T01:06:26.874093+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:42:54.364688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"222'4","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.889105+0000","last_change":"2024-01-27T01:06:25.856458+0000","last_active":"2024-01-27T01:06:26.889105+0000","last_peered":"2024-01-27T01:06:26.889105+0000","last_clean":"2024-01-27T01:06:26.889105+0000","last_became_active":"2024-01-27T01:06:25.856162+0000","last_became_peered":"2024-01-27T01:06:25.856162+0000","last_unstale":"2024-01-27T01:06:26.889105+0000","last_undegraded":"2024-01-27T01:06:26.889105+0000","last_fullsized":"2024-01-27T01:06:26.889105+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:51:44.378880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.883957+0000","last_change":"2024-01-27T01:06:26.878625+0000","last_active":"2024-01-27T01:06:26.883957+0000","last_peered":"2024-01-27T01:06:26.883957+0000","last_clean":"2024-01-27T01:06:26.883957+0000","last_became_active":"2024-01-27T01:06:26.878361+0000","last_became_peered":"2024-01-27T01:06:26.878361+0000","last_unstale":"2024-01-27T01:06:26.883957+0000","last_undegraded":"2024-01-27T01:06:26.883957+0000","last_fullsized":"2024-01-27T01:06:26.883957+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:37:06.977689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"222'1","reported_seq":11,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.891631+0000","last_change":"2024-01-27T01:06:25.856504+0000","last_active":"2024-01-27T01:06:26.891631+0000","last_peered":"2024-01-27T01:06:26.891631+0000","last_clean":"2024-01-27T01:06:26.891631+0000","last_became_active":"2024-01-27T01:06:25.856272+0000","last_became_peered":"2024-01-27T01:06:25.856272+0000","last_unstale":"2024-01-27T01:06:26.891631+0000","last_undegraded":"2024-01-27T01:06:26.891631+0000","last_fullsized":"2024-01-27T01:06:26.891631+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:20:56.736428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884207+0000","last_change":"2024-01-27T01:06:26.875831+0000","last_active":"2024-01-27T01:06:26.884207+0000","last_peered":"2024-01-27T01:06:26.884207+0000","last_clean":"2024-01-27T01:06:26.884207+0000","last_became_active":"2024-01-27T01:06:26.875488+0000","last_became_peered":"2024-01-27T01:06:26.875488+0000","last_unstale":"2024-01-27T01:06:26.884207+0000","last_undegraded":"2024-01-27T01:06:26.884207+0000","last_fullsized":"2024-01-27T01:06:26.884207+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:07:51.640059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"222'2","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.887982+0000","last_change":"2024-01-27T01:06:25.855991+0000","last_active":"2024-01-27T01:06:26.887982+0000","last_peered":"2024-01-27T01:06:26.887982+0000","last_clean":"2024-01-27T01:06:26.887982+0000","last_became_active":"2024-01-27T01:06:25.855679+0000","last_became_peered":"2024-01-27T01:06:25.855679+0000","last_unstale":"2024-01-27T01:06:26.887982+0000","last_undegraded":"2024-01-27T01:06:26.887982+0000","last_fullsized":"2024-01-27T01:06:26.887982+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:31:04.295025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.875454+0000","last_change":"2024-01-27T01:06:26.869128+0000","last_active":"2024-01-27T01:06:26.875454+0000","last_peered":"2024-01-27T01:06:26.875454+0000","last_clean":"2024-01-27T01:06:26.875454+0000","last_became_active":"2024-01-27T01:06:26.868892+0000","last_became_peered":"2024-01-27T01:06:26.868892+0000","last_unstale":"2024-01-27T01:06:26.875454+0000","last_undegraded":"2024-01-27T01:06:26.875454+0000","last_fullsized":"2024-01-27T01:06:26.875454+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:40:40.869516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"222'2","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.889282+0000","last_change":"2024-01-27T01:06:25.855114+0000","last_active":"2024-01-27T01:06:26.889282+0000","last_peered":"2024-01-27T01:06:26.889282+0000","last_clean":"2024-01-27T01:06:26.889282+0000","last_became_active":"2024-01-27T01:06:25.854790+0000","last_became_peered":"2024-01-27T01:06:25.854790+0000","last_unstale":"2024-01-27T01:06:26.889282+0000","last_undegraded":"2024-01-27T01:06:26.889282+0000","last_fullsized":"2024-01-27T01:06:26.889282+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:58:58.498377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.873014+0000","last_change":"2024-01-27T01:06:26.866121+0000","last_active":"2024-01-27T01:06:26.873014+0000","last_peered":"2024-01-27T01:06:26.873014+0000","last_clean":"2024-01-27T01:06:26.873014+0000","last_became_active":"2024-01-27T01:06:26.865391+0000","last_became_peered":"2024-01-27T01:06:26.865391+0000","last_unstale":"2024-01-27T01:06:26.873014+0000","last_undegraded":"2024-01-27T01:06:26.873014+0000","last_fullsized":"2024-01-27T01:06:26.873014+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:02:03.812030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"222'1","reported_seq":11,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.888794+0000","last_change":"2024-01-27T01:06:25.854193+0000","last_active":"2024-01-27T01:06:26.888794+0000","last_peered":"2024-01-27T01:06:26.888794+0000","last_clean":"2024-01-27T01:06:26.888794+0000","last_became_active":"2024-01-27T01:06:25.853956+0000","last_became_peered":"2024-01-27T01:06:25.853956+0000","last_unstale":"2024-01-27T01:06:26.888794+0000","last_undegraded":"2024-01-27T01:06:26.888794+0000","last_fullsized":"2024-01-27T01:06:26.888794+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:33:02.747087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.880717+0000","last_change":"2024-01-27T01:06:26.876963+0000","last_active":"2024-01-27T01:06:26.880717+0000","last_peered":"2024-01-27T01:06:26.880717+0000","last_clean":"2024-01-27T01:06:26.880717+0000","last_became_active":"2024-01-27T01:06:26.876709+0000","last_became_peered":"2024-01-27T01:06:26.876709+0000","last_unstale":"2024-01-27T01:06:26.880717+0000","last_undegraded":"2024-01-27T01:06:26.880717+0000","last_fullsized":"2024-01-27T01:06:26.880717+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:58:14.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.859854+0000","last_change":"2024-01-27T01:06:25.854878+0000","last_active":"2024-01-27T01:06:26.859854+0000","last_peered":"2024-01-27T01:06:26.859854+0000","last_clean":"2024-01-27T01:06:26.859854+0000","last_became_active":"2024-01-27T01:06:25.854623+0000","last_became_peered":"2024-01-27T01:06:25.854623+0000","last_unstale":"2024-01-27T01:06:26.859854+0000","last_undegraded":"2024-01-27T01:06:26.859854+0000","last_fullsized":"2024-01-27T01:06:26.859854+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:21.007449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.881070+0000","last_change":"2024-01-27T01:06:26.873199+0000","last_active":"2024-01-27T01:06:26.881070+0000","last_peered":"2024-01-27T01:06:26.881070+0000","last_clean":"2024-01-27T01:06:26.881070+0000","last_became_active":"2024-01-27T01:06:26.872658+0000","last_became_peered":"2024-01-27T01:06:26.872658+0000","last_unstale":"2024-01-27T01:06:26.881070+0000","last_undegraded":"2024-01-27T01:06:26.881070+0000","last_fullsized":"2024-01-27T01:06:26.881070+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:06:45.526347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.857016+0000","last_change":"2024-01-27T01:06:25.851888+0000","last_active":"2024-01-27T01:06:26.857016+0000","last_peered":"2024-01-27T01:06:26.857016+0000","last_clean":"2024-01-27T01:06:26.857016+0000","last_became_active":"2024-01-27T01:06:25.851605+0000","last_became_peered":"2024-01-27T01:06:25.851605+0000","last_unstale":"2024-01-27T01:06:26.857016+0000","last_undegraded":"2024-01-27T01:06:26.857016+0000","last_fullsized":"2024-01-27T01:06:26.857016+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:09.386094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.875063+0000","last_change":"2024-01-27T01:06:26.870806+0000","last_active":"2024-01-27T01:06:26.875063+0000","last_peered":"2024-01-27T01:06:26.875063+0000","last_clean":"2024-01-27T01:06:26.875063+0000","last_became_active":"2024-01-27T01:06:26.870263+0000","last_became_peered":"2024-01-27T01:06:26.870263+0000","last_unstale":"2024-01-27T01:06:26.875063+0000","last_undegraded":"2024-01-27T01:06:26.875063+0000","last_fullsized":"2024-01-27T01:06:26.875063+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:26.214953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.856798+0000","last_change":"2024-01-27T01:06:25.865314+0000","last_active":"2024-01-27T01:06:26.856798+0000","last_peered":"2024-01-27T01:06:26.856798+0000","last_clean":"2024-01-27T01:06:26.856798+0000","last_became_active":"2024-01-27T01:06:25.864940+0000","last_became_peered":"2024-01-27T01:06:25.864940+0000","last_unstale":"2024-01-27T01:06:26.856798+0000","last_undegraded":"2024-01-27T01:06:26.856798+0000","last_fullsized":"2024-01-27T01:06:26.856798+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:24.113986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.877921+0000","last_change":"2024-01-27T01:06:26.868859+0000","last_active":"2024-01-27T01:06:26.877921+0000","last_peered":"2024-01-27T01:06:26.877921+0000","last_clean":"2024-01-27T01:06:26.877921+0000","last_became_active":"2024-01-27T01:06:26.868623+0000","last_became_peered":"2024-01-27T01:06:26.868623+0000","last_unstale":"2024-01-27T01:06:26.877921+0000","last_undegraded":"2024-01-27T01:06:26.877921+0000","last_fullsized":"2024-01-27T01:06:26.877921+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:18:47.232067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.859277+0000","last_change":"2024-01-27T01:06:25.860762+0000","last_active":"2024-01-27T01:06:26.859277+0000","last_peered":"2024-01-27T01:06:26.859277+0000","last_clean":"2024-01-27T01:06:26.859277+0000","last_became_active":"2024-01-27T01:06:25.860459+0000","last_became_peered":"2024-01-27T01:06:25.860459+0000","last_unstale":"2024-01-27T01:06:26.859277+0000","last_undegraded":"2024-01-27T01:06:26.859277+0000","last_fullsized":"2024-01-27T01:06:26.859277+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:31.687265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.877812+0000","last_change":"2024-01-27T01:06:26.876685+0000","last_active":"2024-01-27T01:06:26.877812+0000","last_peered":"2024-01-27T01:06:26.877812+0000","last_clean":"2024-01-27T01:06:26.877812+0000","last_became_active":"2024-01-27T01:06:26.872939+0000","last_became_peered":"2024-01-27T01:06:26.872939+0000","last_unstale":"2024-01-27T01:06:26.877812+0000","last_undegraded":"2024-01-27T01:06:26.877812+0000","last_fullsized":"2024-01-27T01:06:26.877812+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:56:48.730726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"222'3","reported_seq":13,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.888743+0000","last_change":"2024-01-27T01:06:25.857771+0000","last_active":"2024-01-27T01:06:26.888743+0000","last_peered":"2024-01-27T01:06:26.888743+0000","last_clean":"2024-01-27T01:06:26.888743+0000","last_became_active":"2024-01-27T01:06:25.857445+0000","last_became_peered":"2024-01-27T01:06:25.857445+0000","last_unstale":"2024-01-27T01:06:26.888743+0000","last_undegraded":"2024-01-27T01:06:26.888743+0000","last_fullsized":"2024-01-27T01:06:26.888743+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:52.927866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.887220+0000","last_change":"2024-01-27T01:06:26.879652+0000","last_active":"2024-01-27T01:06:26.887220+0000","last_peered":"2024-01-27T01:06:26.887220+0000","last_clean":"2024-01-27T01:06:26.887220+0000","last_became_active":"2024-01-27T01:06:26.879271+0000","last_became_peered":"2024-01-27T01:06:26.879271+0000","last_unstale":"2024-01-27T01:06:26.887220+0000","last_undegraded":"2024-01-27T01:06:26.887220+0000","last_fullsized":"2024-01-27T01:06:26.887220+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:57:23.456731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"222'6","reported_seq":16,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.894314+0000","last_change":"2024-01-27T01:06:25.864160+0000","last_active":"2024-01-27T01:06:26.894314+0000","last_peered":"2024-01-27T01:06:26.894314+0000","last_clean":"2024-01-27T01:06:26.894314+0000","last_became_active":"2024-01-27T01:06:25.863867+0000","last_became_peered":"2024-01-27T01:06:25.863867+0000","last_unstale":"2024-01-27T01:06:26.894314+0000","last_undegraded":"2024-01-27T01:06:26.894314+0000","last_fullsized":"2024-01-27T01:06:26.894314+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:43.338484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.875502+0000","last_change":"2024-01-27T01:06:26.873464+0000","last_active":"2024-01-27T01:06:26.875502+0000","last_peered":"2024-01-27T01:06:26.875502+0000","last_clean":"2024-01-27T01:06:26.875502+0000","last_became_active":"2024-01-27T01:06:26.873063+0000","last_became_peered":"2024-01-27T01:06:26.873063+0000","last_unstale":"2024-01-27T01:06:26.875502+0000","last_undegraded":"2024-01-27T01:06:26.875502+0000","last_fullsized":"2024-01-27T01:06:26.875502+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:33:32.535574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"222'2","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.891983+0000","last_change":"2024-01-27T01:06:25.850061+0000","last_active":"2024-01-27T01:06:26.891983+0000","last_peered":"2024-01-27T01:06:26.891983+0000","last_clean":"2024-01-27T01:06:26.891983+0000","last_became_active":"2024-01-27T01:06:25.849763+0000","last_became_peered":"2024-01-27T01:06:25.849763+0000","last_unstale":"2024-01-27T01:06:26.891983+0000","last_undegraded":"2024-01-27T01:06:26.891983+0000","last_fullsized":"2024-01-27T01:06:26.891983+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:09:56.243562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.877332+0000","last_change":"2024-01-27T01:06:26.862905+0000","last_active":"2024-01-27T01:06:26.877332+0000","last_peered":"2024-01-27T01:06:26.877332+0000","last_clean":"2024-01-27T01:06:26.877332+0000","last_became_active":"2024-01-27T01:06:26.862584+0000","last_became_peered":"2024-01-27T01:06:26.862584+0000","last_unstale":"2024-01-27T01:06:26.877332+0000","last_undegraded":"2024-01-27T01:06:26.877332+0000","last_fullsized":"2024-01-27T01:06:26.877332+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:27:14.653781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.857286+0000","last_change":"2024-01-27T01:06:25.854192+0000","last_active":"2024-01-27T01:06:26.857286+0000","last_peered":"2024-01-27T01:06:26.857286+0000","last_clean":"2024-01-27T01:06:26.857286+0000","last_became_active":"2024-01-27T01:06:25.853601+0000","last_became_peered":"2024-01-27T01:06:25.853601+0000","last_unstale":"2024-01-27T01:06:26.857286+0000","last_undegraded":"2024-01-27T01:06:26.857286+0000","last_fullsized":"2024-01-27T01:06:26.857286+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:42.218792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.872363+0000","last_change":"2024-01-27T01:06:26.866613+0000","last_active":"2024-01-27T01:06:26.872363+0000","last_peered":"2024-01-27T01:06:26.872363+0000","last_clean":"2024-01-27T01:06:26.872363+0000","last_became_active":"2024-01-27T01:06:26.866298+0000","last_became_peered":"2024-01-27T01:06:26.866298+0000","last_unstale":"2024-01-27T01:06:26.872363+0000","last_undegraded":"2024-01-27T01:06:26.872363+0000","last_fullsized":"2024-01-27T01:06:26.872363+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:27:03.934654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.852979+0000","last_change":"2024-01-27T01:06:25.853597+0000","last_active":"2024-01-27T01:06:26.852979+0000","last_peered":"2024-01-27T01:06:26.852979+0000","last_clean":"2024-01-27T01:06:26.852979+0000","last_became_active":"2024-01-27T01:06:25.853257+0000","last_became_peered":"2024-01-27T01:06:25.853257+0000","last_unstale":"2024-01-27T01:06:26.852979+0000","last_undegraded":"2024-01-27T01:06:26.852979+0000","last_fullsized":"2024-01-27T01:06:26.852979+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:54:35.011796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.872381+0000","last_change":"2024-01-27T01:06:26.867077+0000","last_active":"2024-01-27T01:06:26.872381+0000","last_peered":"2024-01-27T01:06:26.872381+0000","last_clean":"2024-01-27T01:06:26.872381+0000","last_became_active":"2024-01-27T01:06:26.866787+0000","last_became_peered":"2024-01-27T01:06:26.866787+0000","last_unstale":"2024-01-27T01:06:26.872381+0000","last_undegraded":"2024-01-27T01:06:26.872381+0000","last_fullsized":"2024-01-27T01:06:26.872381+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:02.000909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"222'8","reported_seq":18,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.892509+0000","last_change":"2024-01-27T01:06:25.865239+0000","last_active":"2024-01-27T01:06:26.892509+0000","last_peered":"2024-01-27T01:06:26.892509+0000","last_clean":"2024-01-27T01:06:26.892509+0000","last_became_active":"2024-01-27T01:06:25.864943+0000","last_became_peered":"2024-01-27T01:06:25.864943+0000","last_unstale":"2024-01-27T01:06:26.892509+0000","last_undegraded":"2024-01-27T01:06:26.892509+0000","last_fullsized":"2024-01-27T01:06:26.892509+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:41:25.254751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.880063+0000","last_change":"2024-01-27T01:06:26.875234+0000","last_active":"2024-01-27T01:06:26.880063+0000","last_peered":"2024-01-27T01:06:26.880063+0000","last_clean":"2024-01-27T01:06:26.880063+0000","last_became_active":"2024-01-27T01:06:26.874927+0000","last_became_peered":"2024-01-27T01:06:26.874927+0000","last_unstale":"2024-01-27T01:06:26.880063+0000","last_undegraded":"2024-01-27T01:06:26.880063+0000","last_fullsized":"2024-01-27T01:06:26.880063+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:10.390573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"222'1","reported_seq":11,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.891755+0000","last_change":"2024-01-27T01:06:25.848900+0000","last_active":"2024-01-27T01:06:26.891755+0000","last_peered":"2024-01-27T01:06:26.891755+0000","last_clean":"2024-01-27T01:06:26.891755+0000","last_became_active":"2024-01-27T01:06:25.848483+0000","last_became_peered":"2024-01-27T01:06:25.848483+0000","last_unstale":"2024-01-27T01:06:26.891755+0000","last_undegraded":"2024-01-27T01:06:26.891755+0000","last_fullsized":"2024-01-27T01:06:26.891755+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:27:13.471892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884185+0000","last_change":"2024-01-27T01:06:26.879356+0000","last_active":"2024-01-27T01:06:26.884185+0000","last_peered":"2024-01-27T01:06:26.884185+0000","last_clean":"2024-01-27T01:06:26.884185+0000","last_became_active":"2024-01-27T01:06:26.878891+0000","last_became_peered":"2024-01-27T01:06:26.878891+0000","last_unstale":"2024-01-27T01:06:26.884185+0000","last_undegraded":"2024-01-27T01:06:26.884185+0000","last_fullsized":"2024-01-27T01:06:26.884185+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:22.011928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"222'2","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.892036+0000","last_change":"2024-01-27T01:06:25.852245+0000","last_active":"2024-01-27T01:06:26.892036+0000","last_peered":"2024-01-27T01:06:26.892036+0000","last_clean":"2024-01-27T01:06:26.892036+0000","last_became_active":"2024-01-27T01:06:25.851982+0000","last_became_peered":"2024-01-27T01:06:25.851982+0000","last_unstale":"2024-01-27T01:06:26.892036+0000","last_undegraded":"2024-01-27T01:06:26.892036+0000","last_fullsized":"2024-01-27T01:06:26.892036+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:05.596045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.879459+0000","last_change":"2024-01-27T01:06:26.870758+0000","last_active":"2024-01-27T01:06:26.879459+0000","last_peered":"2024-01-27T01:06:26.879459+0000","last_clean":"2024-01-27T01:06:26.879459+0000","last_became_active":"2024-01-27T01:06:26.870130+0000","last_became_peered":"2024-01-27T01:06:26.870130+0000","last_unstale":"2024-01-27T01:06:26.879459+0000","last_undegraded":"2024-01-27T01:06:26.879459+0000","last_fullsized":"2024-01-27T01:06:26.879459+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:36:31.625857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"222'1","reported_seq":11,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.887461+0000","last_change":"2024-01-27T01:06:25.851720+0000","last_active":"2024-01-27T01:06:26.887461+0000","last_peered":"2024-01-27T01:06:26.887461+0000","last_clean":"2024-01-27T01:06:26.887461+0000","last_became_active":"2024-01-27T01:06:25.851416+0000","last_became_peered":"2024-01-27T01:06:25.851416+0000","last_unstale":"2024-01-27T01:06:26.887461+0000","last_undegraded":"2024-01-27T01:06:26.887461+0000","last_fullsized":"2024-01-27T01:06:26.887461+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:07.951814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884204+0000","last_change":"2024-01-27T01:06:26.876619+0000","last_active":"2024-01-27T01:06:26.884204+0000","last_peered":"2024-01-27T01:06:26.884204+0000","last_clean":"2024-01-27T01:06:26.884204+0000","last_became_active":"2024-01-27T01:06:26.875927+0000","last_became_peered":"2024-01-27T01:06:26.875927+0000","last_unstale":"2024-01-27T01:06:26.884204+0000","last_undegraded":"2024-01-27T01:06:26.884204+0000","last_fullsized":"2024-01-27T01:06:26.884204+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:02.000909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.859698+0000","last_change":"2024-01-27T01:06:25.861650+0000","last_active":"2024-01-27T01:06:26.859698+0000","last_peered":"2024-01-27T01:06:26.859698+0000","last_clean":"2024-01-27T01:06:26.859698+0000","last_became_active":"2024-01-27T01:06:25.861342+0000","last_became_peered":"2024-01-27T01:06:25.861342+0000","last_unstale":"2024-01-27T01:06:26.859698+0000","last_undegraded":"2024-01-27T01:06:26.859698+0000","last_fullsized":"2024-01-27T01:06:26.859698+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:44:15.953217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.875392+0000","last_change":"2024-01-27T01:06:26.871346+0000","last_active":"2024-01-27T01:06:26.875392+0000","last_peered":"2024-01-27T01:06:26.875392+0000","last_clean":"2024-01-27T01:06:26.875392+0000","last_became_active":"2024-01-27T01:06:26.870999+0000","last_became_peered":"2024-01-27T01:06:26.870999+0000","last_unstale":"2024-01-27T01:06:26.875392+0000","last_undegraded":"2024-01-27T01:06:26.875392+0000","last_fullsized":"2024-01-27T01:06:26.875392+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:27.779157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"222'2","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.887697+0000","last_change":"2024-01-27T01:06:25.860572+0000","last_active":"2024-01-27T01:06:26.887697+0000","last_peered":"2024-01-27T01:06:26.887697+0000","last_clean":"2024-01-27T01:06:26.887697+0000","last_became_active":"2024-01-27T01:06:25.860248+0000","last_became_peered":"2024-01-27T01:06:25.860248+0000","last_unstale":"2024-01-27T01:06:26.887697+0000","last_undegraded":"2024-01-27T01:06:26.887697+0000","last_fullsized":"2024-01-27T01:06:26.887697+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:01:25.741123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.879329+0000","last_change":"2024-01-27T01:06:26.875858+0000","last_active":"2024-01-27T01:06:26.879329+0000","last_peered":"2024-01-27T01:06:26.879329+0000","last_clean":"2024-01-27T01:06:26.879329+0000","last_became_active":"2024-01-27T01:06:26.875564+0000","last_became_peered":"2024-01-27T01:06:26.875564+0000","last_unstale":"2024-01-27T01:06:26.879329+0000","last_undegraded":"2024-01-27T01:06:26.879329+0000","last_fullsized":"2024-01-27T01:06:26.879329+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:43.630831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.857921+0000","last_change":"2024-01-27T01:06:25.851595+0000","last_active":"2024-01-27T01:06:26.857921+0000","last_peered":"2024-01-27T01:06:26.857921+0000","last_clean":"2024-01-27T01:06:26.857921+0000","last_became_active":"2024-01-27T01:06:25.851260+0000","last_became_peered":"2024-01-27T01:06:25.851260+0000","last_unstale":"2024-01-27T01:06:26.857921+0000","last_undegraded":"2024-01-27T01:06:26.857921+0000","last_fullsized":"2024-01-27T01:06:26.857921+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:56:53.179347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.872399+0000","last_change":"2024-01-27T01:06:26.867570+0000","last_active":"2024-01-27T01:06:26.872399+0000","last_peered":"2024-01-27T01:06:26.872399+0000","last_clean":"2024-01-27T01:06:26.872399+0000","last_became_active":"2024-01-27T01:06:26.866517+0000","last_became_peered":"2024-01-27T01:06:26.866517+0000","last_unstale":"2024-01-27T01:06:26.872399+0000","last_undegraded":"2024-01-27T01:06:26.872399+0000","last_fullsized":"2024-01-27T01:06:26.872399+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:09:54.515702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.859850+0000","last_change":"2024-01-27T01:06:25.862199+0000","last_active":"2024-01-27T01:06:26.859850+0000","last_peered":"2024-01-27T01:06:26.859850+0000","last_clean":"2024-01-27T01:06:26.859850+0000","last_became_active":"2024-01-27T01:06:25.861876+0000","last_became_peered":"2024-01-27T01:06:25.861876+0000","last_unstale":"2024-01-27T01:06:26.859850+0000","last_undegraded":"2024-01-27T01:06:26.859850+0000","last_fullsized":"2024-01-27T01:06:26.859850+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:00:54.297085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.884265+0000","last_change":"2024-01-27T01:06:26.875536+0000","last_active":"2024-01-27T01:06:26.884265+0000","last_peered":"2024-01-27T01:06:26.884265+0000","last_clean":"2024-01-27T01:06:26.884265+0000","last_became_active":"2024-01-27T01:06:26.875267+0000","last_became_peered":"2024-01-27T01:06:26.875267+0000","last_unstale":"2024-01-27T01:06:26.884265+0000","last_undegraded":"2024-01-27T01:06:26.884265+0000","last_fullsized":"2024-01-27T01:06:26.884265+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:32.534622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.858847+0000","last_change":"2024-01-27T01:06:25.853605+0000","last_active":"2024-01-27T01:06:26.858847+0000","last_peered":"2024-01-27T01:06:26.858847+0000","last_clean":"2024-01-27T01:06:26.858847+0000","last_became_active":"2024-01-27T01:06:25.853331+0000","last_became_peered":"2024-01-27T01:06:25.853331+0000","last_unstale":"2024-01-27T01:06:26.858847+0000","last_undegraded":"2024-01-27T01:06:26.858847+0000","last_fullsized":"2024-01-27T01:06:26.858847+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:09:08.458850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.880032+0000","last_change":"2024-01-27T01:06:26.875222+0000","last_active":"2024-01-27T01:06:26.880032+0000","last_peered":"2024-01-27T01:06:26.880032+0000","last_clean":"2024-01-27T01:06:26.880032+0000","last_became_active":"2024-01-27T01:06:26.874903+0000","last_became_peered":"2024-01-27T01:06:26.874903+0000","last_unstale":"2024-01-27T01:06:26.880032+0000","last_undegraded":"2024-01-27T01:06:26.880032+0000","last_fullsized":"2024-01-27T01:06:26.880032+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:09:33.903445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"222'2","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.892395+0000","last_change":"2024-01-27T01:06:25.857933+0000","last_active":"2024-01-27T01:06:26.892395+0000","last_peered":"2024-01-27T01:06:26.892395+0000","last_clean":"2024-01-27T01:06:26.892395+0000","last_became_active":"2024-01-27T01:06:25.857406+0000","last_became_peered":"2024-01-27T01:06:25.857406+0000","last_unstale":"2024-01-27T01:06:26.892395+0000","last_undegraded":"2024-01-27T01:06:26.892395+0000","last_fullsized":"2024-01-27T01:06:26.892395+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:20:20.239633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.880984+0000","last_change":"2024-01-27T01:06:26.878650+0000","last_active":"2024-01-27T01:06:26.880984+0000","last_peered":"2024-01-27T01:06:26.880984+0000","last_clean":"2024-01-27T01:06:26.880984+0000","last_became_active":"2024-01-27T01:06:26.877596+0000","last_became_peered":"2024-01-27T01:06:26.877596+0000","last_unstale":"2024-01-27T01:06:26.880984+0000","last_undegraded":"2024-01-27T01:06:26.880984+0000","last_fullsized":"2024-01-27T01:06:26.880984+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:42:48.942202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.864255+0000","last_change":"2024-01-27T01:06:25.861717+0000","last_active":"2024-01-27T01:06:26.864255+0000","last_peered":"2024-01-27T01:06:26.864255+0000","last_clean":"2024-01-27T01:06:26.864255+0000","last_became_active":"2024-01-27T01:06:25.861177+0000","last_became_peered":"2024-01-27T01:06:25.861177+0000","last_unstale":"2024-01-27T01:06:26.864255+0000","last_undegraded":"2024-01-27T01:06:26.864255+0000","last_fullsized":"2024-01-27T01:06:26.864255+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:05.596045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.876760+0000","last_change":"2024-01-27T01:06:26.872190+0000","last_active":"2024-01-27T01:06:26.876760+0000","last_peered":"2024-01-27T01:06:26.876760+0000","last_clean":"2024-01-27T01:06:26.876760+0000","last_became_active":"2024-01-27T01:06:26.867718+0000","last_became_peered":"2024-01-27T01:06:26.867718+0000","last_unstale":"2024-01-27T01:06:26.876760+0000","last_undegraded":"2024-01-27T01:06:26.876760+0000","last_fullsized":"2024-01-27T01:06:26.876760+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:23.301132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.858620+0000","last_change":"2024-01-27T01:06:25.858032+0000","last_active":"2024-01-27T01:06:26.858620+0000","last_peered":"2024-01-27T01:06:26.858620+0000","last_clean":"2024-01-27T01:06:26.858620+0000","last_became_active":"2024-01-27T01:06:25.857777+0000","last_became_peered":"2024-01-27T01:06:25.857777+0000","last_unstale":"2024-01-27T01:06:26.858620+0000","last_undegraded":"2024-01-27T01:06:26.858620+0000","last_fullsized":"2024-01-27T01:06:26.858620+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:09:08.458850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.876438+0000","last_change":"2024-01-27T01:06:26.870597+0000","last_active":"2024-01-27T01:06:26.876438+0000","last_peered":"2024-01-27T01:06:26.876438+0000","last_clean":"2024-01-27T01:06:26.876438+0000","last_became_active":"2024-01-27T01:06:26.870334+0000","last_became_peered":"2024-01-27T01:06:26.870334+0000","last_unstale":"2024-01-27T01:06:26.876438+0000","last_undegraded":"2024-01-27T01:06:26.876438+0000","last_fullsized":"2024-01-27T01:06:26.876438+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:01:23.266072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.861167+0000","last_change":"2024-01-27T01:06:25.859342+0000","last_active":"2024-01-27T01:06:26.861167+0000","last_peered":"2024-01-27T01:06:26.861167+0000","last_clean":"2024-01-27T01:06:26.861167+0000","last_became_active":"2024-01-27T01:06:25.859065+0000","last_became_peered":"2024-01-27T01:06:25.859065+0000","last_unstale":"2024-01-27T01:06:26.861167+0000","last_undegraded":"2024-01-27T01:06:26.861167+0000","last_fullsized":"2024-01-27T01:06:26.861167+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:00:54.297085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.885135+0000","last_change":"2024-01-27T01:06:26.880503+0000","last_active":"2024-01-27T01:06:26.885135+0000","last_peered":"2024-01-27T01:06:26.885135+0000","last_clean":"2024-01-27T01:06:26.885135+0000","last_became_active":"2024-01-27T01:06:26.880224+0000","last_became_peered":"2024-01-27T01:06:26.880224+0000","last_unstale":"2024-01-27T01:06:26.885135+0000","last_undegraded":"2024-01-27T01:06:26.885135+0000","last_fullsized":"2024-01-27T01:06:26.885135+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:27:33.886740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.865748+0000","last_change":"2024-01-27T01:06:25.858142+0000","last_active":"2024-01-27T01:06:26.865748+0000","last_peered":"2024-01-27T01:06:26.865748+0000","last_clean":"2024-01-27T01:06:26.865748+0000","last_became_active":"2024-01-27T01:06:25.857867+0000","last_became_peered":"2024-01-27T01:06:25.857867+0000","last_unstale":"2024-01-27T01:06:26.865748+0000","last_undegraded":"2024-01-27T01:06:26.865748+0000","last_fullsized":"2024-01-27T01:06:26.865748+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:10:59.780192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.880953+0000","last_change":"2024-01-27T01:06:26.876182+0000","last_active":"2024-01-27T01:06:26.880953+0000","last_peered":"2024-01-27T01:06:26.880953+0000","last_clean":"2024-01-27T01:06:26.880953+0000","last_became_active":"2024-01-27T01:06:26.875894+0000","last_became_peered":"2024-01-27T01:06:26.875894+0000","last_unstale":"2024-01-27T01:06:26.880953+0000","last_undegraded":"2024-01-27T01:06:26.880953+0000","last_fullsized":"2024-01-27T01:06:26.880953+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:58.495062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.854249+0000","last_change":"2024-01-27T01:06:25.855015+0000","last_active":"2024-01-27T01:06:26.854249+0000","last_peered":"2024-01-27T01:06:26.854249+0000","last_clean":"2024-01-27T01:06:26.854249+0000","last_became_active":"2024-01-27T01:06:25.854708+0000","last_became_peered":"2024-01-27T01:06:25.854708+0000","last_unstale":"2024-01-27T01:06:26.854249+0000","last_undegraded":"2024-01-27T01:06:26.854249+0000","last_fullsized":"2024-01-27T01:06:26.854249+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:03.971945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.876472+0000","last_change":"2024-01-27T01:06:26.870626+0000","last_active":"2024-01-27T01:06:26.876472+0000","last_peered":"2024-01-27T01:06:26.876472+0000","last_clean":"2024-01-27T01:06:26.876472+0000","last_became_active":"2024-01-27T01:06:26.870457+0000","last_became_peered":"2024-01-27T01:06:26.870457+0000","last_unstale":"2024-01-27T01:06:26.876472+0000","last_undegraded":"2024-01-27T01:06:26.876472+0000","last_fullsized":"2024-01-27T01:06:26.876472+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:42.766155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.865136+0000","last_change":"2024-01-27T01:06:25.856035+0000","last_active":"2024-01-27T01:06:26.865136+0000","last_peered":"2024-01-27T01:06:26.865136+0000","last_clean":"2024-01-27T01:06:26.865136+0000","last_became_active":"2024-01-27T01:06:25.855762+0000","last_became_peered":"2024-01-27T01:06:25.855762+0000","last_unstale":"2024-01-27T01:06:26.865136+0000","last_undegraded":"2024-01-27T01:06:26.865136+0000","last_fullsized":"2024-01-27T01:06:26.865136+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:51:44.378880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.881016+0000","last_change":"2024-01-27T01:06:26.879306+0000","last_active":"2024-01-27T01:06:26.881016+0000","last_peered":"2024-01-27T01:06:26.881016+0000","last_clean":"2024-01-27T01:06:26.881016+0000","last_became_active":"2024-01-27T01:06:26.879071+0000","last_became_peered":"2024-01-27T01:06:26.879071+0000","last_unstale":"2024-01-27T01:06:26.881016+0000","last_undegraded":"2024-01-27T01:06:26.881016+0000","last_fullsized":"2024-01-27T01:06:26.881016+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:50:46.965359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":10,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.857104+0000","last_change":"2024-01-27T01:06:25.865235+0000","last_active":"2024-01-27T01:06:26.857104+0000","last_peered":"2024-01-27T01:06:26.857104+0000","last_clean":"2024-01-27T01:06:26.857104+0000","last_became_active":"2024-01-27T01:06:25.864872+0000","last_became_peered":"2024-01-27T01:06:25.864872+0000","last_unstale":"2024-01-27T01:06:26.857104+0000","last_undegraded":"2024-01-27T01:06:26.857104+0000","last_fullsized":"2024-01-27T01:06:26.857104+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:53:38.618410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":9,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.878104+0000","last_change":"2024-01-27T01:06:26.873783+0000","last_active":"2024-01-27T01:06:26.878104+0000","last_peered":"2024-01-27T01:06:26.878104+0000","last_clean":"2024-01-27T01:06:26.878104+0000","last_became_active":"2024-01-27T01:06:26.873490+0000","last_became_peered":"2024-01-27T01:06:26.873490+0000","last_unstale":"2024-01-27T01:06:26.878104+0000","last_undegraded":"2024-01-27T01:06:26.878104+0000","last_fullsized":"2024-01-27T01:06:26.878104+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:32.691743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"222'1","reported_seq":11,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.888523+0000","last_change":"2024-01-27T01:06:25.858609+0000","last_active":"2024-01-27T01:06:26.888523+0000","last_peered":"2024-01-27T01:06:26.888523+0000","last_clean":"2024-01-27T01:06:26.888523+0000","last_became_active":"2024-01-27T01:06:25.857638+0000","last_became_peered":"2024-01-27T01:06:25.857638+0000","last_unstale":"2024-01-27T01:06:26.888523+0000","last_undegraded":"2024-01-27T01:06:26.888523+0000","last_fullsized":"2024-01-27T01:06:26.888523+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:35:59.109571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"213'8750","reported_seq":9664,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-01-27T01:06:26.859832+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:06:26.859832+0000","last_peered":"2024-01-27T01:06:26.859832+0000","last_clean":"2024-01-27T01:06:26.859832+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:06:26.859832+0000","last_undegraded":"2024-01-27T01:06:26.859832+0000","last_fullsized":"2024-01-27T01:06:26.859832+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8750,"log_dups_size":0,"ondisk_log_size":8750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":630,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":630,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8750,"ondisk_log_size":8750,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576002,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300620,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071220,"statfs":{"total":96636764160,"available":96328929280,"internally_reserved":0,"allocated":3411968,"data_stored":2065684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63200000000000001,"15min":0.61399999999999999},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.312},"max":{"1min":0.82099999999999995,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.63700000000000001,"15min":0.623},"min":{"1min":0.498,"5min":0.39900000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":0.879,"15min":0.879},"last":0.73199999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.623,"15min":0.61599999999999999},"min":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65600000000000003,"15min":0.63700000000000001},"min":{"1min":0.505,"5min":0.42099999999999999,"15min":0.34300000000000003},"max":{"1min":0.76800000000000002,"5min":0.84099999999999997,"15min":0.84399999999999997},"last":0.60799999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63400000000000001,"15min":0.61399999999999999},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82099999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64500000000000002,"15min":0.63600000000000001},"min":{"1min":0.49399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79400000000000004,"5min":0.872,"15min":0.88900000000000001},"last":0.69499999999999995}]},{"osd":3,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.628,"15min":0.62},"min":{"1min":0.52100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79900000000000004,"5min":0.876,"15min":0.876},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66800000000000004,"15min":0.64700000000000002},"min":{"1min":0.53200000000000003,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84599999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.75}]},{"osd":4,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.50900000000000001,"15min":0.48999999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.29999999999999999},"max":{"1min":0.73399999999999999,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.51900000000000002},"min":{"1min":0.35699999999999998,"5min":0.34699999999999998,"15min":0.32200000000000001},"max":{"1min":0.77500000000000002,"5min":0.80300000000000005,"15min":0.86599999999999999},"last":0.629}]},{"osd":5,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55800000000000005,"15min":0.52600000000000002},"min":{"1min":0.40999999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.753,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57999999999999996,"15min":0.56499999999999995},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.33700000000000002},"max":{"1min":0.76700000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.79200000000000004}]},{"osd":6,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60999999999999999,"15min":0.58899999999999997},"min":{"1min":0.42499999999999999,"5min":0.42199999999999999,"15min":0.37},"max":{"1min":0.81100000000000005,"5min":0.83999999999999997,"15min":0.84799999999999998},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61299999999999999,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.39100000000000001,"15min":0.35699999999999998},"max":{"1min":0.76100000000000001,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.71999999999999997}]}]},{"osd":6,"up_from":13,"seq":55834576002,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308232,"kb_used_data":2752,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063608,"statfs":{"total":96636764160,"available":96321134592,"internally_reserved":0,"allocated":2818048,"data_stored":1475800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59899999999999998,"15min":0.60699999999999998},"min":{"1min":0.44900000000000001,"5min":0.40200000000000002,"15min":0.36399999999999999},"max":{"1min":0.77800000000000002,"5min":0.81999999999999995,"15min":0.86399999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61699999999999999,"15min":0.61799999999999999},"min":{"1min":0.48299999999999998,"5min":0.432,"15min":0.376},"max":{"1min":0.78500000000000003,"5min":0.82899999999999996,"15min":0.86299999999999999},"last":0.76300000000000001}]},{"osd":1,"last update":"Sat Jan 27 01:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.504,"5min":0.44500000000000001,"15min":0.41499999999999998},"max":{"1min":0.72399999999999998,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.60999999999999999,"15min":0.621},"min":{"1min":0.53000000000000003,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.70899999999999996,"5min":0.80900000000000005,"15min":0.83999999999999997},"last":0.64000000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.51300000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72699999999999998,"5min":0.79800000000000004,"15min":0.82399999999999995},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61399999999999999,"15min":0.622},"min":{"1min":0.53300000000000003,"5min":0.39700000000000002,"15min":0.377},"max":{"1min":0.76600000000000001,"5min":0.878,"15min":0.878},"last":0.74399999999999999}]},{"osd":3,"last update":"Sat Jan 27 01:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61599999999999999,"15min":0.627},"min":{"1min":0.51800000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.748,"5min":0.82599999999999996,"15min":0.90100000000000002},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.66200000000000003,"15min":0.66100000000000003},"min":{"1min":0.52400000000000002,"5min":0.47299999999999998,"15min":0.42899999999999999},"max":{"1min":0.77700000000000002,"5min":0.84099999999999997,"15min":0.88100000000000001},"last":0.83399999999999996}]},{"osd":4,"last update":"Sat Jan 27 01:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50700000000000001,"15min":0.50600000000000001},"min":{"1min":0.375,"5min":0.33300000000000002,"15min":0.32500000000000001},"max":{"1min":0.73799999999999999,"5min":0.75700000000000001,"15min":0.86599999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.41599999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.75700000000000001,"5min":0.78500000000000003,"15min":0.88900000000000001},"last":0.62}]},{"osd":5,"last update":"Sat Jan 27 01:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51700000000000002,"15min":0.52400000000000002},"min":{"1min":0.39900000000000002,"5min":0.33600000000000002,"15min":0.31900000000000001},"max":{"1min":0.71099999999999997,"5min":0.75900000000000001,"15min":0.80500000000000005},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58199999999999996,"15min":0.57299999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.28899999999999998},"max":{"1min":0.79200000000000004,"5min":0.84899999999999998,"15min":0.874},"last":0.77700000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.41199999999999998,"5min":0.38200000000000001,"15min":0.35799999999999998},"max":{"1min":0.80400000000000005,"5min":0.83499999999999996,"15min":0.92900000000000005},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.45100000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.70699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834576002,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300024,"kb_used_data":2736,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071816,"statfs":{"total":96636764160,"available":96329539584,"internally_reserved":0,"allocated":2801664,"data_stored":1473710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.41099999999999998,"15min":0.42099999999999999},"min":{"1min":0.26400000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.64400000000000002,"5min":0.66100000000000003,"15min":0.69599999999999995},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44500000000000001,"15min":0.45500000000000002},"min":{"1min":0.34000000000000002,"5min":0.28399999999999997,"15min":0.26800000000000002},"max":{"1min":0.57999999999999996,"5min":0.73999999999999999,"15min":0.76600000000000001},"last":0.41099999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.439,"15min":0.438},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.26200000000000001},"max":{"1min":0.748,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.47599999999999998,"15min":0.48999999999999999},"min":{"1min":0.379,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.67400000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44500000000000001}]},{"osd":3,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52300000000000002,"15min":0.53300000000000003},"min":{"1min":0.39300000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73699999999999999,"5min":0.84799999999999998,"15min":1.2130000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.51400000000000001},"min":{"1min":0.41699999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68200000000000005,"5min":1.125,"15min":1.22},"last":0.35899999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.67200000000000004,"15min":0.67400000000000004},"min":{"1min":0.42099999999999999,"5min":0.41099999999999998,"15min":0.39800000000000002},"max":{"1min":0.82699999999999996,"5min":1.242,"15min":1.242},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69099999999999995,"15min":0.69599999999999995},"min":{"1min":0.53700000000000003,"5min":0.496,"15min":0.35699999999999998},"max":{"1min":0.90600000000000003,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.66600000000000004}]},{"osd":5,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73499999999999999,"15min":0.70899999999999996},"min":{"1min":0.53300000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.876,"5min":5.444,"15min":5.444},"last":0.628},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.71699999999999997,"15min":0.70799999999999996},"min":{"1min":0.502,"5min":0.47599999999999998,"15min":0.435},"max":{"1min":0.97799999999999998,"5min":2.6669999999999998,"15min":2.6669999999999998},"last":0.57999999999999996}]},{"osd":6,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.74199999999999999,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":0.93899999999999995,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.81299999999999994,"15min":0.745},"min":{"1min":0.48799999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.02,"5min":11.223000000000001,"15min":11.223000000000001},"last":0.64600000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.752,"15min":0.75600000000000001},"min":{"1min":0.58999999999999997,"5min":0.52800000000000002,"15min":0.47599999999999998},"max":{"1min":0.96299999999999997,"5min":2.9660000000000002,"15min":2.9660000000000002},"last":0.748},{"interface":"front","average":{"1min":0.752,"5min":0.77700000000000002,"15min":0.76800000000000002},"min":{"1min":0.58799999999999997,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.94499999999999995,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.77000000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576001,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316992,"kb_used_data":3320,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94054848,"statfs":{"total":96636764160,"available":96312164352,"internally_reserved":0,"allocated":3399680,"data_stored":2064168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42899999999999999,"15min":0.42099999999999999},"min":{"1min":0.33300000000000002,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":0.61099999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46800000000000003,"15min":0.46000000000000002},"min":{"1min":0.36299999999999999,"5min":0.313,"15min":0.26200000000000001},"max":{"1min":0.65600000000000003,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.35199999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.43099999999999999,"15min":0.435},"min":{"1min":0.31,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.76800000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.48999999999999999},"min":{"1min":0.314,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65500000000000003,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.42399999999999999}]},{"osd":3,"last update":"Sat Jan 27 01:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.498,"15min":0.48499999999999999},"min":{"1min":0.39500000000000002,"5min":0.33300000000000002,"15min":0.30499999999999999},"max":{"1min":0.68999999999999995,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.316},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.51400000000000001,"15min":0.502},"min":{"1min":0.38600000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.44}]},{"osd":4,"last update":"Sat Jan 27 01:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.68000000000000005,"15min":0.67600000000000005},"min":{"1min":0.52000000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82699999999999996,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68000000000000005,"15min":0.68200000000000005},"min":{"1min":0.50900000000000001,"5min":0.47099999999999997,"15min":0.437},"max":{"1min":0.98999999999999999,"5min":1.002,"15min":1.002},"last":0.67900000000000005}]},{"osd":5,"last update":"Sat Jan 27 01:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.68799999999999994,"15min":0.69699999999999995},"min":{"1min":0.52700000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89300000000000002,"5min":1.093,"15min":2.165},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.71399999999999997,"15min":0.71299999999999997},"min":{"1min":0.59299999999999997,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.96699999999999997,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.58599999999999997}]},{"osd":6,"last update":"Sat Jan 27 01:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70699999999999996,"15min":0.71199999999999997},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.95199999999999996,"5min":0.98899999999999999,"15min":1.0860000000000001},"last":0.622},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73599999999999999,"15min":0.73999999999999999},"min":{"1min":0.56899999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.96899999999999997,"5min":1.0429999999999999,"15min":1.425},"last":0.749}]},{"osd":7,"last update":"Sat Jan 27 01:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.75900000000000001,"15min":0.75700000000000001},"min":{"1min":0.52000000000000002,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0169999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.77100000000000002,"15min":0.77000000000000002},"min":{"1min":0.59999999999999998,"5min":0.51100000000000001,"15min":0.48499999999999999},"max":{"1min":1.01,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.66700000000000004}]}]},{"osd":2,"up_from":13,"seq":55834576001,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312316,"kb_used_data":2740,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059524,"statfs":{"total":96636764160,"available":96316952576,"internally_reserved":0,"allocated":2805760,"data_stored":1473800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40500000000000003,"15min":0.40799999999999997},"min":{"1min":0.26600000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.46000000000000002,"5min":0.746,"15min":0.746},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.44500000000000001,"15min":0.44400000000000001},"min":{"1min":0.32300000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.59099999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.41199999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.435,"15min":0.44500000000000001},"min":{"1min":0.32000000000000001,"5min":0.29099999999999998,"15min":0.27000000000000002},"max":{"1min":0.61299999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.439,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.55600000000000005,"5min":0.73599999999999999,"15min":0.78100000000000003},"last":0.51500000000000001}]},{"osd":3,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.498,"15min":0.497},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.33200000000000002},"max":{"1min":0.627,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.496},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48699999999999999,"15min":0.49199999999999999},"min":{"1min":0.373,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":0.59099999999999997,"5min":0.66600000000000004,"15min":0.75700000000000001},"last":0.47099999999999997}]},{"osd":4,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68200000000000005,"15min":0.68000000000000005},"min":{"1min":0.52800000000000002,"5min":0.495,"15min":0.442},"max":{"1min":0.92700000000000005,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68999999999999995,"15min":0.69399999999999995},"min":{"1min":0.50900000000000001,"5min":0.47099999999999997,"15min":0.378},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":1.107},"last":0.70899999999999996}]},{"osd":5,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.69099999999999995,"15min":0.70299999999999996},"min":{"1min":0.52400000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.89500000000000002,"15min":1.167},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70899999999999996,"15min":0.70999999999999996},"min":{"1min":0.47499999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89600000000000002,"5min":0.97499999999999998,"15min":1.05},"last":0.65400000000000003}]},{"osd":6,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71499999999999997,"15min":0.71199999999999997},"min":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.45500000000000002},"max":{"1min":0.81399999999999995,"5min":0.996,"15min":1.077},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70499999999999996,"15min":0.71999999999999997},"min":{"1min":0.497,"5min":0.497,"15min":0.39200000000000002},"max":{"1min":0.876,"5min":0.96699999999999997,"15min":1.135},"last":0.67100000000000004}]},{"osd":7,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.749,"15min":0.75},"min":{"1min":0.56000000000000005,"5min":0.50800000000000001,"15min":0.47099999999999997},"max":{"1min":0.94099999999999995,"5min":0.97399999999999998,"15min":1.198},"last":0.745},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.76500000000000001,"15min":0.76900000000000002},"min":{"1min":0.55000000000000004,"5min":0.53100000000000003,"15min":0.48199999999999998},"max":{"1min":0.94599999999999995,"5min":1.028,"15min":1.0880000000000001},"last":0.78600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576000,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304132,"kb_used_data":2748,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067708,"statfs":{"total":96636764160,"available":96325332992,"internally_reserved":0,"allocated":2813952,"data_stored":1473836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.44700000000000001,"15min":0.42299999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.54800000000000004,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.372,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.63500000000000001,"5min":0.66700000000000004,"15min":0.84699999999999998},"last":0.438}]},{"osd":1,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.63100000000000001,"15min":0.53500000000000003},"min":{"1min":0.34499999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.629,"5min":15.391,"15min":15.391},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.54600000000000004,"15min":0.52400000000000002},"min":{"1min":0.36699999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65100000000000002,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.53700000000000003}]},{"osd":2,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54600000000000004,"15min":0.51700000000000002},"min":{"1min":0.36499999999999999,"5min":0.36399999999999999,"15min":0.34599999999999997},"max":{"1min":0.62,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.53100000000000003,"15min":0.50700000000000001},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.60699999999999998,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":1.5529999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.74199999999999999,"15min":0.72099999999999997},"min":{"1min":0.497,"5min":0.47699999999999998,"15min":0.439},"max":{"1min":0.88600000000000001,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.71999999999999997,"15min":0.70799999999999996},"min":{"1min":0.46300000000000002,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":0.94399999999999995,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.73899999999999999}]},{"osd":5,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.74299999999999999,"15min":0.71699999999999997},"min":{"1min":0.51100000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.83999999999999997,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72799999999999998,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.97699999999999998,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.78000000000000003}]},{"osd":6,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.754,"15min":0.72599999999999998},"min":{"1min":0.53500000000000003,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.81200000000000006,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.748,"15min":0.74299999999999999},"min":{"1min":0.56100000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.92700000000000005,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.81699999999999995}]},{"osd":7,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.75600000000000001,"15min":0.73799999999999999},"min":{"1min":0.58599999999999997,"5min":0.52500000000000002,"15min":0.46700000000000003},"max":{"1min":1.0029999999999999,"5min":4.569,"15min":4.569},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.80600000000000005,"15min":0.78500000000000003},"min":{"1min":0.55700000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0169999999999999,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.64800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834576002,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312316,"kb_used_data":2740,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059524,"statfs":{"total":96636764160,"available":96316952576,"internally_reserved":0,"allocated":2805760,"data_stored":1474264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.46899999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.84199999999999997},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.61499999999999999,"15min":0.61399999999999999},"min":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78700000000000003,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.60999999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61899999999999999,"15min":0.627},"min":{"1min":0.46100000000000002,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.76400000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61499999999999999,"15min":0.62},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.79800000000000004,"5min":0.874,"15min":0.874},"last":0.59799999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.59999999999999998},"min":{"1min":0.42399999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.88700000000000001},"last":0.624},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.51100000000000001,"5min":0.42899999999999999,"15min":0.42799999999999999},"max":{"1min":0.81100000000000005,"5min":0.81499999999999995,"15min":0.89200000000000002},"last":0.74299999999999999}]},{"osd":3,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.61599999999999999,"15min":0.623},"min":{"1min":0.38800000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.83299999999999996,"5min":0.83599999999999997,"15min":0.84199999999999997},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.627},"min":{"1min":0.41299999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.88300000000000001},"last":0.66000000000000003}]},{"osd":5,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.75800000000000001,"15min":0.85899999999999999},"last":0.504},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.502},"min":{"1min":0.376,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.878},"last":0.53000000000000003}]},{"osd":6,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34999999999999998},"max":{"1min":0.63200000000000001,"5min":0.70099999999999996,"15min":0.88700000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.70699999999999996,"5min":0.79200000000000004,"15min":0.81499999999999995},"last":0.64500000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.42399999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72599999999999998,"5min":0.80400000000000005,"15min":0.90200000000000002},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59499999999999997,"15min":0.59399999999999997},"min":{"1min":0.41599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.73799999999999999,"5min":0.78700000000000003,"15min":0.92600000000000005},"last":0.70199999999999996}]}]},{"osd":5,"up_from":13,"seq":55834576002,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312320,"kb_used_data":2744,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059520,"statfs":{"total":96636764160,"available":96316948480,"internally_reserved":0,"allocated":2809856,"data_stored":1473766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.54500000000000004,"5min":0.40500000000000003,"15min":0.27300000000000002},"max":{"1min":0.70999999999999996,"5min":1.006,"15min":1.006},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81499999999999995,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.66900000000000004}]},{"osd":1,"last update":"Sat Jan 27 01:05:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.65200000000000002,"15min":0.65600000000000003},"min":{"1min":0.53600000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80300000000000005,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.63},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.67700000000000005,"15min":0.67100000000000004},"min":{"1min":0.57099999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.79100000000000004,"5min":1.212,"15min":1.212},"last":0.64100000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:05:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.59599999999999997},"min":{"1min":0.434,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67900000000000005,"5min":1.631,"15min":1.631},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.623},"min":{"1min":0.56699999999999995,"5min":0.38800000000000001,"15min":0.34000000000000002},"max":{"1min":0.72399999999999998,"5min":0.82599999999999996,"15min":0.83299999999999996},"last":0.51100000000000001}]},{"osd":3,"last update":"Sat Jan 27 01:05:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60599999999999998,"15min":0.59999999999999998},"min":{"1min":0.40600000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.51500000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.754,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.57299999999999995}]},{"osd":4,"last update":"Sat Jan 27 01:05:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.505,"15min":0.48599999999999999},"min":{"1min":0.377,"5min":0.32200000000000001,"15min":0.317},"max":{"1min":0.54000000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.53100000000000003,"15min":0.51100000000000001},"min":{"1min":0.35599999999999998,"5min":0.35199999999999998,"15min":0.33900000000000002},"max":{"1min":0.68999999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.48999999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:05:29 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52500000000000002,"15min":0.503},"min":{"1min":0.315,"5min":0.315,"15min":0.314},"max":{"1min":0.59599999999999997,"5min":1.732,"15min":1.732},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55800000000000005,"15min":0.54200000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70099999999999996,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.55700000000000005}]},{"osd":7,"last update":"Sat Jan 27 01:05:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.39400000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.63,"15min":0.60999999999999999},"min":{"1min":0.442,"5min":0.40799999999999997,"15min":0.39100000000000001},"max":{"1min":0.77700000000000002,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.61299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:06:31.471 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-27T01:06:31.471 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-27T01:06:31.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.649+0000 7fa3399f4700 1 Processor -- start 2024-01-27T01:06:31.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.649+0000 7fa3399f4700 1 -- start start 2024-01-27T01:06:31.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.650+0000 7fa3399f4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3340ca650 0x7fa334067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:31.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.650+0000 7fa3399f4700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33406fdb0 con 0x7fa33406ebe0 2024-01-27T01:06:31.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.650+0000 7fa3399f4700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33406f1a0 con 0x7fa334067af0 2024-01-27T01:06:31.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.650+0000 7fa3399f4700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa33406f480 con 0x7fa3340ca650 2024-01-27T01:06:31.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.650+0000 7fa3337fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3340ca650 0x7fa334067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:31.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.651+0000 7fa3337fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3340ca650 0x7fa334067ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33740/0 (socket says 172.21.15.46:33740) 2024-01-27T01:06:31.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.651+0000 7fa3327fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa33406ebe0 0x7fa3341597e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54134/0 (socket says 172.21.15.46:54134) 2024-01-27T01:06:31.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.651+0000 7fa3337fe700 1 -- 172.21.15.46:0/3639756636 learned_addr learned my addr 172.21.15.46:0/3639756636 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:31.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.651+0000 7fa3337fe700 1 -- 172.21.15.46:0/3639756636 >> v1:172.21.15.106:6789/0 conn(0x7fa334067af0 legacy=0x7fa3340ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:31.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.651+0000 7fa3337fe700 1 -- 172.21.15.46:0/3639756636 >> v1:172.21.15.46:6789/0 conn(0x7fa33406ebe0 legacy=0x7fa3341597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:31.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.651+0000 7fa3337fe700 1 -- 172.21.15.46:0/3639756636 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33406f760 con 0x7fa3340ca650 2024-01-27T01:06:31.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.651+0000 7fa331ffb700 1 -- 172.21.15.46:0/3639756636 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276094354 0 0) 0x7fa33406fdb0 con 0x7fa33406ebe0 2024-01-27T01:06:31.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.651+0000 7fa3337fe700 1 --2- 172.21.15.46:0/3639756636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3340ca650 0x7fa334067ec0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fa32c009f20 tx=0x7fa32c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa5a19d5f947cccb server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:31.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.651+0000 7fa331ffb700 1 -- 172.21.15.46:0/3639756636 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c00e070 con 0x7fa3340ca650 2024-01-27T01:06:31.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.651+0000 7fa331ffb700 1 -- 172.21.15.46:0/3639756636 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32c00ba80 con 0x7fa3340ca650 2024-01-27T01:06:31.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.651+0000 7fa331ffb700 1 -- 172.21.15.46:0/3639756636 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c0092e0 con 0x7fa3340ca650 2024-01-27T01:06:31.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.652+0000 7fa3399f4700 1 -- 172.21.15.46:0/3639756636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3340ca650 msgr2=0x7fa334067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:31.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.652+0000 7fa3399f4700 1 --2- 172.21.15.46:0/3639756636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3340ca650 0x7fa334067ec0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fa32c009f20 tx=0x7fa32c009840 comp rx=0 tx=0).stop 2024-01-27T01:06:31.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.652+0000 7fa3399f4700 1 -- 172.21.15.46:0/3639756636 shutdown_connections 2024-01-27T01:06:31.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.652+0000 7fa3399f4700 1 --2- 172.21.15.46:0/3639756636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3340ca650 0x7fa334067ec0 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.652+0000 7fa3399f4700 1 -- 172.21.15.46:0/3639756636 >> 172.21.15.46:0/3639756636 conn(0x7fa3340cdf80 msgr2=0x7fa33406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:31.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.652+0000 7fa3399f4700 1 -- 172.21.15.46:0/3639756636 shutdown_connections 2024-01-27T01:06:31.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.652+0000 7fa3399f4700 1 -- 172.21.15.46:0/3639756636 wait complete. 2024-01-27T01:06:31.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.653+0000 7fa3399f4700 1 Processor -- start 2024-01-27T01:06:31.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.653+0000 7fa3399f4700 1 -- start start 2024-01-27T01:06:31.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.654+0000 7fa3399f4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa334067af0 0x7fa334151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:31.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.654+0000 7fa3399f4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa33406ebe0 0x7fa334157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:31.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.654+0000 7fa3399f4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3340ca650 0x7fa334154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:31.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.654+0000 7fa3399f4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3341520f0 con 0x7fa33406ebe0 2024-01-27T01:06:31.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.654+0000 7fa3399f4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3341595b0 con 0x7fa3340ca650 2024-01-27T01:06:31.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.654+0000 7fa3399f4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33415a410 con 0x7fa334067af0 2024-01-27T01:06:31.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.654+0000 7fa3327fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa33406ebe0 0x7fa334157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:31.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.654+0000 7fa3327fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa33406ebe0 0x7fa334157480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33742/0 (socket says 172.21.15.46:33742) 2024-01-27T01:06:31.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.654+0000 7fa3327fc700 1 -- 172.21.15.46:0/1313073940 learned_addr learned my addr 172.21.15.46:0/1313073940 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:31.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.654+0000 7fa3337fe700 1 --2- 172.21.15.46:0/1313073940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3340ca650 0x7fa334154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:31.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.654+0000 7fa332ffd700 1 --2- 172.21.15.46:0/1313073940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa334067af0 0x7fa334151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:31.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.655+0000 7fa3327fc700 1 -- 172.21.15.46:0/1313073940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa334067af0 msgr2=0x7fa334151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:31.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.655+0000 7fa3327fc700 1 --2- 172.21.15.46:0/1313073940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa334067af0 0x7fa334151bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.655+0000 7fa3327fc700 1 -- 172.21.15.46:0/1313073940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3340ca650 msgr2=0x7fa334154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:31.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.655+0000 7fa3327fc700 1 --2- 172.21.15.46:0/1313073940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3340ca650 0x7fa334154ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.655+0000 7fa3327fc700 1 -- 172.21.15.46:0/1313073940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32c009ae0 con 0x7fa33406ebe0 2024-01-27T01:06:31.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.655+0000 7fa332ffd700 1 --2- 172.21.15.46:0/1313073940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa334067af0 0x7fa334151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:06:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.655+0000 7fa3337fe700 1 --2- 172.21.15.46:0/1313073940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3340ca650 0x7fa334154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:06:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.655+0000 7fa3327fc700 1 --2- 172.21.15.46:0/1313073940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa33406ebe0 0x7fa334157480 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00c560 tx=0x7fa31c007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.655+0000 7fa32affd700 1 -- 172.21.15.46:0/1313073940 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31c005a70 con 0x7fa33406ebe0 2024-01-27T01:06:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.655+0000 7fa32affd700 1 -- 172.21.15.46:0/1313073940 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa31c005bd0 con 0x7fa33406ebe0 2024-01-27T01:06:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.655+0000 7fa3399f4700 1 -- 172.21.15.46:0/1313073940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa334159800 con 0x7fa33406ebe0 2024-01-27T01:06:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.655+0000 7fa32affd700 1 -- 172.21.15.46:0/1313073940 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31c0070d0 con 0x7fa33406ebe0 2024-01-27T01:06:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.655+0000 7fa3399f4700 1 -- 172.21.15.46:0/1313073940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa334159e40 con 0x7fa33406ebe0 2024-01-27T01:06:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.656+0000 7fa3399f4700 1 -- 172.21.15.46:0/1313073940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa33415a580 con 0x7fa33406ebe0 2024-01-27T01:06:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.657+0000 7fa32affd700 1 -- 172.21.15.46:0/1313073940 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa31c0074c0 con 0x7fa33406ebe0 2024-01-27T01:06:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.657+0000 7fa32affd700 1 --2- 172.21.15.46:0/1313073940 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa314066f30 0x7fa3140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.657+0000 7fa32affd700 1 -- 172.21.15.46:0/1313073940 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fa31c083dd0 con 0x7fa33406ebe0 2024-01-27T01:06:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.657+0000 7fa332ffd700 1 --2- 172.21.15.46:0/1313073940 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa314066f30 0x7fa3140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.658+0000 7fa332ffd700 1 --2- 172.21.15.46:0/1313073940 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa314066f30 0x7fa3140693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa334158f60 tx=0x7fa324009040 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:31.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.662+0000 7fa32affd700 1 -- 172.21.15.46:0/1313073940 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa31c050d10 con 0x7fa33406ebe0 2024-01-27T01:06:31.832 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-01-27T01:06:31.833+0000 7f60d04b8700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-27T01:06:31.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.954+0000 7fa3399f4700 1 -- 172.21.15.46:0/1313073940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa33404c710 con 0x7fa33406ebe0 2024-01-27T01:06:31.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.957+0000 7fa32affd700 1 -- 172.21.15.46:0/1313073940 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v54) v1 ==== 190+0+0 (secure 0 0 0) 0x7fa31c085020 con 0x7fa33406ebe0 2024-01-27T01:06:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.960+0000 7fa3399f4700 1 -- 172.21.15.46:0/1313073940 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa314066f30 msgr2=0x7fa3140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.960+0000 7fa3399f4700 1 --2- 172.21.15.46:0/1313073940 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa314066f30 0x7fa3140693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa334158f60 tx=0x7fa324009040 comp rx=0 tx=0).stop 2024-01-27T01:06:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.960+0000 7fa3399f4700 1 -- 172.21.15.46:0/1313073940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa33406ebe0 msgr2=0x7fa334157480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.960+0000 7fa3399f4700 1 --2- 172.21.15.46:0/1313073940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa33406ebe0 0x7fa334157480 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00c560 tx=0x7fa31c007f60 comp rx=0 tx=0).stop 2024-01-27T01:06:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.960+0000 7fa3399f4700 1 -- 172.21.15.46:0/1313073940 shutdown_connections 2024-01-27T01:06:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.960+0000 7fa3399f4700 1 --2- 172.21.15.46:0/1313073940 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa314066f30 0x7fa3140693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.960+0000 7fa3399f4700 1 --2- 172.21.15.46:0/1313073940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3340ca650 0x7fa334154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.960+0000 7fa3399f4700 1 --2- 172.21.15.46:0/1313073940 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa334067af0 0x7fa334151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.960+0000 7fa3399f4700 1 --2- 172.21.15.46:0/1313073940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa33406ebe0 0x7fa334157480 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:31.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.960+0000 7fa3399f4700 1 -- 172.21.15.46:0/1313073940 >> 172.21.15.46:0/1313073940 conn(0x7fa3340cdf80 msgr2=0x7fa3340cb940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:31.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.960+0000 7fa3399f4700 1 -- 172.21.15.46:0/1313073940 shutdown_connections 2024-01-27T01:06:31.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:31.960+0000 7fa3399f4700 1 -- 172.21.15.46:0/1313073940 wait complete. 2024-01-27T01:06:31.960 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-01-27T01:06:31.973 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:06:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.105+0000 7fe44107f700 1 Processor -- start 2024-01-27T01:06:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.105+0000 7fe44107f700 1 -- start start 2024-01-27T01:06:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.106+0000 7fe44107f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe43c0b2160 0x7fe43c15a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.106+0000 7fe44107f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe43c0b2530 0x7fe43c0abe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.106+0000 7fe44107f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe43c15da10 0x7fe43c15de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.106+0000 7fe44107f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe43c0a9b90 con 0x7fe43c0b2530 2024-01-27T01:06:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.106+0000 7fe44107f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe43c0aa210 con 0x7fe43c0b2160 2024-01-27T01:06:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.106+0000 7fe44107f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe43c15d580 con 0x7fe43c15da10 2024-01-27T01:06:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.106+0000 7fe43a59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe43c0b2530 0x7fe43c0abe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.106+0000 7fe43a59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe43c0b2530 0x7fe43c0abe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33752/0 (socket says 172.21.15.46:33752) 2024-01-27T01:06:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.106+0000 7fe43a59c700 1 -- 172.21.15.46:0/2922495798 learned_addr learned my addr 172.21.15.46:0/2922495798 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.106+0000 7fe43b59e700 1 --2- 172.21.15.46:0/2922495798 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe43c15da10 0x7fe43c15de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.106+0000 7fe43ad9d700 1 --2- 172.21.15.46:0/2922495798 >> v2:172.21.15.106:3300/0 conn(0x7fe43c0b2160 0x7fe43c15a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.106+0000 7fe43a59c700 1 -- 172.21.15.46:0/2922495798 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe43c15da10 msgr2=0x7fe43c15de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.106+0000 7fe43a59c700 1 --2- 172.21.15.46:0/2922495798 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe43c15da10 0x7fe43c15de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.106+0000 7fe43a59c700 1 -- 172.21.15.46:0/2922495798 >> v2:172.21.15.106:3300/0 conn(0x7fe43c0b2160 msgr2=0x7fe43c15a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.106+0000 7fe43a59c700 1 --2- 172.21.15.46:0/2922495798 >> v2:172.21.15.106:3300/0 conn(0x7fe43c0b2160 0x7fe43c15a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.106+0000 7fe43a59c700 1 -- 172.21.15.46:0/2922495798 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43c0ac390 con 0x7fe43c0b2530 2024-01-27T01:06:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.106+0000 7fe43a59c700 1 --2- 172.21.15.46:0/2922495798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43c0b2530 0x7fe43c0abe50 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fe43c063ca0 tx=0x7fe430009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3ca0d495d37395b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.107+0000 7fe439d9b700 1 -- 172.21.15.46:0/2922495798 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43000b640 con 0x7fe43c0b2530 2024-01-27T01:06:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.107+0000 7fe439d9b700 1 -- 172.21.15.46:0/2922495798 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe43000b7a0 con 0x7fe43c0b2530 2024-01-27T01:06:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.107+0000 7fe439d9b700 1 -- 172.21.15.46:0/2922495798 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43000bb40 con 0x7fe43c0b2530 2024-01-27T01:06:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.107+0000 7fe44107f700 1 -- 172.21.15.46:0/2922495798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43c0b2530 msgr2=0x7fe43c0abe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.107+0000 7fe44107f700 1 --2- 172.21.15.46:0/2922495798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43c0b2530 0x7fe43c0abe50 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fe43c063ca0 tx=0x7fe430009250 comp rx=0 tx=0).stop 2024-01-27T01:06:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.107+0000 7fe44107f700 1 -- 172.21.15.46:0/2922495798 shutdown_connections 2024-01-27T01:06:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.107+0000 7fe44107f700 1 --2- 172.21.15.46:0/2922495798 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe43c15da10 0x7fe43c15de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.107+0000 7fe44107f700 1 --2- 172.21.15.46:0/2922495798 >> v2:172.21.15.106:3300/0 conn(0x7fe43c0b2160 0x7fe43c15a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.107+0000 7fe44107f700 1 --2- 172.21.15.46:0/2922495798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43c0b2530 0x7fe43c0abe50 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.107+0000 7fe44107f700 1 -- 172.21.15.46:0/2922495798 >> 172.21.15.46:0/2922495798 conn(0x7fe43c0ccbc0 msgr2=0x7fe43c0ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.107+0000 7fe44107f700 1 -- 172.21.15.46:0/2922495798 shutdown_connections 2024-01-27T01:06:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.107+0000 7fe44107f700 1 -- 172.21.15.46:0/2922495798 wait complete. 2024-01-27T01:06:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.109+0000 7fe44107f700 1 Processor -- start 2024-01-27T01:06:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.109+0000 7fe44107f700 1 -- start start 2024-01-27T01:06:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.109+0000 7fe44107f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe43c0b2160 0x7fe43c14fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.109+0000 7fe44107f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe43c0b2530 0x7fe43c157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.109+0000 7fe44107f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43c15da10 0x7fe43c067b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.109+0000 7fe44107f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43c16e3b0 con 0x7fe43c15da10 2024-01-27T01:06:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.109+0000 7fe44107f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43c154e70 con 0x7fe43c0b2530 2024-01-27T01:06:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.109+0000 7fe44107f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe43c0ae520 con 0x7fe43c0b2160 2024-01-27T01:06:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.110+0000 7fe43b59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43c15da10 0x7fe43c067b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.110+0000 7fe43b59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43c15da10 0x7fe43c067b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33764/0 (socket says 172.21.15.46:33764) 2024-01-27T01:06:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.110+0000 7fe43b59e700 1 -- 172.21.15.46:0/1181567190 learned_addr learned my addr 172.21.15.46:0/1181567190 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.110+0000 7fe43ad9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe43c0b2160 0x7fe43c14fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.110+0000 7fe43a59c700 1 --2- 172.21.15.46:0/1181567190 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe43c0b2530 0x7fe43c157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.110+0000 7fe43b59e700 1 -- 172.21.15.46:0/1181567190 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe43c0b2160 msgr2=0x7fe43c14fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.110+0000 7fe43b59e700 1 --2- 172.21.15.46:0/1181567190 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe43c0b2160 0x7fe43c14fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.110+0000 7fe43b59e700 1 -- 172.21.15.46:0/1181567190 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe43c0b2530 msgr2=0x7fe43c157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.110+0000 7fe43b59e700 1 --2- 172.21.15.46:0/1181567190 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe43c0b2530 0x7fe43c157170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.110+0000 7fe43b59e700 1 -- 172.21.15.46:0/1181567190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4300094f0 con 0x7fe43c15da10 2024-01-27T01:06:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.110+0000 7fe43b59e700 1 --2- 172.21.15.46:0/1181567190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43c15da10 0x7fe43c067b70 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fe434009d70 tx=0x7fe43400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.110+0000 7fe422ffd700 1 -- 172.21.15.46:0/1181567190 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43400caa0 con 0x7fe43c15da10 2024-01-27T01:06:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.110+0000 7fe422ffd700 1 -- 172.21.15.46:0/1181567190 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe43400f040 con 0x7fe43c15da10 2024-01-27T01:06:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.110+0000 7fe44107f700 1 -- 172.21.15.46:0/1181567190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe43c16e0d0 con 0x7fe43c15da10 2024-01-27T01:06:32.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.110+0000 7fe422ffd700 1 -- 172.21.15.46:0/1181567190 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43400ce50 con 0x7fe43c15da10 2024-01-27T01:06:32.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.110+0000 7fe44107f700 1 -- 172.21.15.46:0/1181567190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe43c0ae1b0 con 0x7fe43c15da10 2024-01-27T01:06:32.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.111+0000 7fe44107f700 1 -- 172.21.15.46:0/1181567190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe43c04c710 con 0x7fe43c15da10 2024-01-27T01:06:32.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.112+0000 7fe422ffd700 1 -- 172.21.15.46:0/1181567190 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe434014690 con 0x7fe43c15da10 2024-01-27T01:06:32.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.112+0000 7fe422ffd700 1 --2- 172.21.15.46:0/1181567190 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe428066f30 0x7fe4280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.112+0000 7fe422ffd700 1 -- 172.21.15.46:0/1181567190 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fe434085380 con 0x7fe43c15da10 2024-01-27T01:06:32.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.113+0000 7fe43ad9d700 1 --2- 172.21.15.46:0/1181567190 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe428066f30 0x7fe4280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.113+0000 7fe43ad9d700 1 --2- 172.21.15.46:0/1181567190 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe428066f30 0x7fe4280693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009d20 tx=0x7fe42c0091a0 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:32.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.117+0000 7fe422ffd700 1 -- 172.21.15.46:0/1181567190 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe4340522c0 con 0x7fe43c15da10 2024-01-27T01:06:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.370+0000 7fe44107f700 1 -- 172.21.15.46:0/1181567190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe43c0ae660 con 0x7fe43c15da10 2024-01-27T01:06:32.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.371+0000 7fe422ffd700 1 -- 172.21.15.46:0/1181567190 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+4618 (secure 0 0 0) 0x7fe4340558e0 con 0x7fe43c15da10 2024-01-27T01:06:32.370 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:32.370 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":116,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/1999300009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1999300009},{"type":"v1","addr":"172.21.15.106:6835","nonce":1999300009}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":110},{"gid":10486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/4174263148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4174263148},{"type":"v1","addr":"172.21.15.106:6833","nonce":4174263148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":10854,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1263484152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1263484152},{"type":"v1","addr":"172.21.15.46:6833","nonce":1263484152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:06:26.286041+0000","modified":"2024-01-27T01:06:29.873402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10866},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10866":{"gid":10866,"name":"a","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2483163449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2483163449},{"type":"v1","addr":"172.21.15.46:6835","nonce":2483163449}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2024-01-27T01:06:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.373+0000 7fe44107f700 1 -- 172.21.15.46:0/1181567190 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe428066f30 msgr2=0x7fe4280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.373+0000 7fe44107f700 1 --2- 172.21.15.46:0/1181567190 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe428066f30 0x7fe4280693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009d20 tx=0x7fe42c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.373+0000 7fe44107f700 1 -- 172.21.15.46:0/1181567190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43c15da10 msgr2=0x7fe43c067b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.373+0000 7fe44107f700 1 --2- 172.21.15.46:0/1181567190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43c15da10 0x7fe43c067b70 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fe434009d70 tx=0x7fe43400b040 comp rx=0 tx=0).stop 2024-01-27T01:06:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.373+0000 7fe44107f700 1 -- 172.21.15.46:0/1181567190 shutdown_connections 2024-01-27T01:06:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.373+0000 7fe44107f700 1 --2- 172.21.15.46:0/1181567190 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe428066f30 0x7fe4280693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.373+0000 7fe44107f700 1 --2- 172.21.15.46:0/1181567190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43c15da10 0x7fe43c067b70 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.373+0000 7fe44107f700 1 --2- 172.21.15.46:0/1181567190 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe43c0b2160 0x7fe43c14fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.373+0000 7fe44107f700 1 --2- 172.21.15.46:0/1181567190 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe43c0b2530 0x7fe43c157170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.373+0000 7fe44107f700 1 -- 172.21.15.46:0/1181567190 >> 172.21.15.46:0/1181567190 conn(0x7fe43c0ccbc0 msgr2=0x7fe43c15aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:32.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.373+0000 7fe44107f700 1 -- 172.21.15.46:0/1181567190 shutdown_connections 2024-01-27T01:06:32.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.374+0000 7fe44107f700 1 -- 172.21.15.46:0/1181567190 wait complete. 2024-01-27T01:06:32.374 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 116 2024-01-27T01:06:32.789 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-27T01:06:26.286041+0000', 'modified': '2024-01-27T01:06:29.873402+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10866}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10866': {'gid': 10866, 'name': 'a', 'rank': 0, 'incarnation': 113, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6835/2483163449', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6834', 'nonce': 2483163449}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 2483163449}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-27T01:06:32.790 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-27T01:06:32.790 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-27T01:06:32.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.935+0000 7f8ed1a20700 1 Processor -- start 2024-01-27T01:06:32.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.935+0000 7f8ed1a20700 1 -- start start 2024-01-27T01:06:32.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.935+0000 7f8ed1a20700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8ecc0af850 0x7f8ecc159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.935+0000 7f8ed1a20700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ecc15a000 0x7f8ecc0afc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.936+0000 7f8ed1a20700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc1509d0 con 0x7f8ecc0b0160 2024-01-27T01:06:32.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.936+0000 7f8ed1a20700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ecc157820 con 0x7f8ecc0af850 2024-01-27T01:06:32.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.936+0000 7f8ed1a20700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc15a450 con 0x7f8ecc15a000 2024-01-27T01:06:32.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.936+0000 7f8ecb7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8ecc0b0160 0x7f8ecc156470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54144/0 (socket says 172.21.15.46:54144) 2024-01-27T01:06:32.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.936+0000 7f8ecb7fe700 1 -- 172.21.15.46:0/3409722847 learned_addr learned my addr 172.21.15.46:0/3409722847 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:32.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.936+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/3409722847 >> v2:172.21.15.106:3300/0 conn(0x7f8ecc0af850 0x7f8ecc159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.936+0000 7f8eca7fc700 1 --2- 172.21.15.46:0/3409722847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ecc15a000 0x7f8ecc0afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.936+0000 7f8ec9ffb700 1 -- 172.21.15.46:0/3409722847 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189106150 0 0) 0x7f8ec4001490 con 0x7f8ecc0b0160 2024-01-27T01:06:32.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.936+0000 7f8ec9ffb700 1 -- 172.21.15.46:0/3409722847 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eb4003570 con 0x7f8ecc0b0160 2024-01-27T01:06:32.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.936+0000 7f8ec9ffb700 1 -- 172.21.15.46:0/3409722847 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1131528742 0 0) 0x7f8eb4003570 con 0x7f8ecc0b0160 2024-01-27T01:06:32.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.936+0000 7f8ec9ffb700 1 -- 172.21.15.46:0/3409722847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ecc15a000 msgr2=0x7f8ecc0afc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:32.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.936+0000 7f8ec9ffb700 1 --2- 172.21.15.46:0/3409722847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ecc15a000 0x7f8ecc0afc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.936+0000 7f8ec9ffb700 1 -- 172.21.15.46:0/3409722847 >> v2:172.21.15.106:3300/0 conn(0x7f8ecc0af850 msgr2=0x7f8ecc159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:32.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.936+0000 7f8ec9ffb700 1 --2- 172.21.15.46:0/3409722847 >> v2:172.21.15.106:3300/0 conn(0x7f8ecc0af850 0x7f8ecc159ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.936+0000 7f8ec9ffb700 1 -- 172.21.15.46:0/3409722847 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc0b05e0 con 0x7f8ecc0b0160 2024-01-27T01:06:32.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.936+0000 7f8ec9ffb700 1 -- 172.21.15.46:0/3409722847 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f8ec4001980 con 0x7f8ecc0b0160 2024-01-27T01:06:32.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.936+0000 7f8ec9ffb700 1 -- 172.21.15.46:0/3409722847 >> v1:172.21.15.46:6789/0 conn(0x7f8ecc0b0160 legacy=0x7f8ecc156470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:32.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ec9ffb700 1 --2- 172.21.15.46:0/3409722847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb4003a60 0x7f8eb4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ec9ffb700 1 --2- 172.21.15.46:0/3409722847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ecc15a000 0x7f8eb4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ec9ffb700 1 --2- 172.21.15.46:0/3409722847 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8eb4004d50 0x7f8eb40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ec9ffb700 1 -- 172.21.15.46:0/3409722847 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc157820 con 0x7f8eb4003a60 2024-01-27T01:06:32.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ec9ffb700 1 -- 172.21.15.46:0/3409722847 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc15a450 con 0x7f8eb4004d50 2024-01-27T01:06:32.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ec9ffb700 1 -- 172.21.15.46:0/3409722847 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb400ba60 con 0x7f8ecc15a000 2024-01-27T01:06:32.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/3409722847 >> v2:172.21.15.106:3300/0 conn(0x7f8ecc0af850 0x7f8ecc159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:32.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/3409722847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb4003a60 0x7f8eb4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8eca7fc700 1 --2- 172.21.15.46:0/3409722847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ecc15a000 0x7f8eb4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ecb7fe700 1 --2- 172.21.15.46:0/3409722847 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8eb4004d50 0x7f8eb40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ecaffd700 1 -- 172.21.15.46:0/3409722847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ecc15a000 msgr2=0x7f8eb4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:32.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/3409722847 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ecc15a000 0x7f8eb4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ecaffd700 1 -- 172.21.15.46:0/3409722847 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8eb4004d50 msgr2=0x7f8eb40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:32.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/3409722847 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8eb4004d50 0x7f8eb40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ecaffd700 1 -- 172.21.15.46:0/3409722847 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb4003570 con 0x7f8eb4003a60 2024-01-27T01:06:32.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/3409722847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb4003a60 0x7f8eb4003e70 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009db0 tx=0x7f8ebc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff0eb5a0416cddc server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:32.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ec9ffb700 1 -- 172.21.15.46:0/3409722847 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc009740 con 0x7f8eb4003a60 2024-01-27T01:06:32.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ec9ffb700 1 -- 172.21.15.46:0/3409722847 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc00e8c0 con 0x7f8eb4003a60 2024-01-27T01:06:32.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.937+0000 7f8ec9ffb700 1 -- 172.21.15.46:0/3409722847 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc00ec40 con 0x7f8eb4003a60 2024-01-27T01:06:32.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.938+0000 7f8ed1a20700 1 -- 172.21.15.46:0/3409722847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb4003a60 msgr2=0x7f8eb4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:32.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.938+0000 7f8ed1a20700 1 --2- 172.21.15.46:0/3409722847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb4003a60 0x7f8eb4003e70 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009db0 tx=0x7f8ebc0091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.938+0000 7f8ecb7fe700 1 -- 172.21.15.46:0/3409722847 reap_dead start 2024-01-27T01:06:32.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.938+0000 7f8ed1a20700 1 -- 172.21.15.46:0/3409722847 shutdown_connections 2024-01-27T01:06:32.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.938+0000 7f8ed1a20700 1 -- 172.21.15.46:0/3409722847 >> 172.21.15.46:0/3409722847 conn(0x7f8ecc068400 msgr2=0x7f8ecc0cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:32.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.938+0000 7f8ed1a20700 1 -- 172.21.15.46:0/3409722847 shutdown_connections 2024-01-27T01:06:32.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.938+0000 7f8ed1a20700 1 -- 172.21.15.46:0/3409722847 wait complete. 2024-01-27T01:06:32.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.939+0000 7f8ed1a20700 1 Processor -- start 2024-01-27T01:06:32.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.939+0000 7f8ed1a20700 1 -- start start 2024-01-27T01:06:32.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.940+0000 7f8ed1a20700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb4003a60 0x7f8ecc15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.940+0000 7f8ed1a20700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ecc0ca780 0x7f8ecc0cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.940+0000 7f8ed1a20700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ecc166750 0x7f8ecc166bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.940+0000 7f8ed1a20700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc0c7840 con 0x7f8eb4003a60 2024-01-27T01:06:32.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.940+0000 7f8ed1a20700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc15b200 con 0x7f8ecc0ca780 2024-01-27T01:06:32.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.940+0000 7f8ed1a20700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc0cb4f0 con 0x7f8ecc166750 2024-01-27T01:06:32.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.940+0000 7f8ecaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb4003a60 0x7f8ecc15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.940+0000 7f8ecaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb4003a60 0x7f8ecc15d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33788/0 (socket says 172.21.15.46:33788) 2024-01-27T01:06:32.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.940+0000 7f8ecaffd700 1 -- 172.21.15.46:0/817718176 learned_addr learned my addr 172.21.15.46:0/817718176 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:32.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.940+0000 7f8eca7fc700 1 --2- 172.21.15.46:0/817718176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ecc0ca780 0x7f8ecc0cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.940+0000 7f8ecb7fe700 1 --2- 172.21.15.46:0/817718176 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ecc166750 0x7f8ecc166bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.941+0000 7f8ecaffd700 1 -- 172.21.15.46:0/817718176 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ecc166750 msgr2=0x7f8ecc166bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:32.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.941+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/817718176 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ecc166750 0x7f8ecc166bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.941+0000 7f8ecaffd700 1 -- 172.21.15.46:0/817718176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ecc0ca780 msgr2=0x7f8ecc0cabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:32.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.941+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/817718176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ecc0ca780 0x7f8ecc0cabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.941+0000 7f8ecaffd700 1 -- 172.21.15.46:0/817718176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc15b450 con 0x7f8eb4003a60 2024-01-27T01:06:32.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.941+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/817718176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb4003a60 0x7f8ecc15d5f0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4004730 tx=0x7f8ebc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:32.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.941+0000 7f8eb2ffd700 1 -- 172.21.15.46:0/817718176 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc00ec40 con 0x7f8eb4003a60 2024-01-27T01:06:32.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.941+0000 7f8eb2ffd700 1 -- 172.21.15.46:0/817718176 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc021040 con 0x7f8eb4003a60 2024-01-27T01:06:32.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.941+0000 7f8ed1a20700 1 -- 172.21.15.46:0/817718176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ecc15b730 con 0x7f8eb4003a60 2024-01-27T01:06:32.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.941+0000 7f8eb2ffd700 1 -- 172.21.15.46:0/817718176 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc025620 con 0x7f8eb4003a60 2024-01-27T01:06:32.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.941+0000 7f8ed1a20700 1 -- 172.21.15.46:0/817718176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ecc0cb130 con 0x7f8eb4003a60 2024-01-27T01:06:32.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.943+0000 7f8eb2ffd700 1 -- 172.21.15.46:0/817718176 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8ebc025990 con 0x7f8eb4003a60 2024-01-27T01:06:32.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.943+0000 7f8eb2ffd700 1 --2- 172.21.15.46:0/817718176 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8eb8066f30 0x7f8eb80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.943+0000 7f8eb2ffd700 1 -- 172.21.15.46:0/817718176 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f8ebc0860e0 con 0x7f8eb4003a60 2024-01-27T01:06:32.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.943+0000 7f8eca7fc700 1 --2- 172.21.15.46:0/817718176 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8eb8066f30 0x7f8eb80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.944+0000 7f8eca7fc700 1 --2- 172.21.15.46:0/817718176 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8eb8066f30 0x7f8eb80693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc06e440 tx=0x7f8ec00091a0 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.945+0000 7f8edad23b80 1 Processor -- start 2024-01-27T01:06:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.945+0000 7f8edad23b80 1 -- start start 2024-01-27T01:06:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.946+0000 7f8edad23b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697278bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.946+0000 7f8edad23b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55697278b360 0x55697276ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.946+0000 7f8edad23b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556972742e40 0x5569727432b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.946+0000 7f8edad23b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569726469f0 con 0x55697276a740 2024-01-27T01:06:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.946+0000 7f8edad23b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x556972698130 con 0x556972742e40 2024-01-27T01:06:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.946+0000 7f8edad23b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x556972695930 con 0x55697278b360 2024-01-27T01:06:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.946+0000 7f8ecaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697278bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.946+0000 7f8ecaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697278bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33794/0 (socket says 172.21.15.46:33794) 2024-01-27T01:06:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.946+0000 7f8ecaffd700 1 -- 172.21.15.46:0/3115567283 learned_addr learned my addr 172.21.15.46:0/3115567283 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.946+0000 7f8eca7fc700 1 --2- 172.21.15.46:0/3115567283 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55697278b360 0x55697276ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.946+0000 7f8ecb7fe700 1 --2- 172.21.15.46:0/3115567283 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556972742e40 0x5569727432b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.947+0000 7f8ecaffd700 1 -- 172.21.15.46:0/3115567283 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55697278b360 msgr2=0x55697276ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.947+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/3115567283 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55697278b360 0x55697276ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.947+0000 7f8ecaffd700 1 -- 172.21.15.46:0/3115567283 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556972742e40 msgr2=0x5569727432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.947+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/3115567283 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556972742e40 0x5569727432b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.947+0000 7f8ecaffd700 1 -- 172.21.15.46:0/3115567283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569724d0b10 con 0x55697276a740 2024-01-27T01:06:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.947+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/3115567283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697278bf30 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc056c70 tx=0x7f8ebc084230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef15479cffcfd1c2 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.947+0000 7f8eabfff700 1 -- 172.21.15.46:0/3115567283 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc056640 con 0x55697276a740 2024-01-27T01:06:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.947+0000 7f8eabfff700 1 -- 172.21.15.46:0/3115567283 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc0573e0 con 0x55697276a740 2024-01-27T01:06:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.947+0000 7f8eabfff700 1 -- 172.21.15.46:0/3115567283 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc056ce0 con 0x55697276a740 2024-01-27T01:06:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.948+0000 7f8edad23b80 1 -- 172.21.15.46:0/3115567283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 msgr2=0x55697278bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.948+0000 7f8edad23b80 1 --2- 172.21.15.46:0/3115567283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697278bf30 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc056c70 tx=0x7f8ebc084230 comp rx=0 tx=0).stop 2024-01-27T01:06:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.948+0000 7f8edad23b80 1 -- 172.21.15.46:0/3115567283 shutdown_connections 2024-01-27T01:06:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.948+0000 7f8edad23b80 1 --2- 172.21.15.46:0/3115567283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697278bf30 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.948+0000 7f8edad23b80 1 --2- 172.21.15.46:0/3115567283 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556972742e40 0x5569727432b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.948+0000 7f8edad23b80 1 --2- 172.21.15.46:0/3115567283 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55697278b360 0x55697276ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.948+0000 7f8edad23b80 1 -- 172.21.15.46:0/3115567283 >> 172.21.15.46:0/3115567283 conn(0x556972742a70 msgr2=0x55697278b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:32.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.948+0000 7f8edad23b80 1 -- 172.21.15.46:0/3115567283 shutdown_connections 2024-01-27T01:06:32.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.948+0000 7f8edad23b80 1 -- 172.21.15.46:0/3115567283 wait complete. 2024-01-27T01:06:32.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.949+0000 7f8edad23b80 1 -- start start 2024-01-27T01:06:32.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.949+0000 7f8edad23b80 1 Processor -- start 2024-01-27T01:06:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.950+0000 7f8edad23b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556972742e40 0x55697273dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.950+0000 7f8edad23b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697273e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.950+0000 7f8edad23b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55697278b360 0x556972784ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.950+0000 7f8edad23b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x556972691720 con 0x55697276a740 2024-01-27T01:06:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.950+0000 7f8edad23b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55697263d1b0 con 0x55697278b360 2024-01-27T01:06:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.950+0000 7f8edad23b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55697263ce50 con 0x556972742e40 2024-01-27T01:06:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.950+0000 7f8edad23b80 10 client.0 _ll_register_callbacks cb 0x55697249f7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-27T01:06:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.950+0000 7f8eca7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697273e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.950+0000 7f8eca7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697273e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33802/0 (socket says 172.21.15.46:33802) 2024-01-27T01:06:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.950+0000 7f8eca7fc700 1 -- 172.21.15.46:0/3011377698 learned_addr learned my addr 172.21.15.46:0/3011377698 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.950+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/3011377698 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556972742e40 0x55697273dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.951+0000 7f8ecb7fe700 1 --2- 172.21.15.46:0/3011377698 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55697278b360 0x556972784ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.951+0000 7f8eca7fc700 1 -- 172.21.15.46:0/3011377698 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556972742e40 msgr2=0x55697273dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.951+0000 7f8eca7fc700 1 --2- 172.21.15.46:0/3011377698 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556972742e40 0x55697273dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.951+0000 7f8eca7fc700 1 -- 172.21.15.46:0/3011377698 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55697278b360 msgr2=0x556972784ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.951+0000 7f8eca7fc700 1 --2- 172.21.15.46:0/3011377698 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55697278b360 0x556972784ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.951+0000 7f8eca7fc700 1 -- 172.21.15.46:0/3011377698 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ecc0cb130 con 0x55697276a740 2024-01-27T01:06:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.951+0000 7f8eca7fc700 1 --2- 172.21.15.46:0/3011377698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697273e010 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0015aa0 tx=0x7f8ec0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa28f8ec275c470 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.951+0000 7f8ea97fa700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-27T01:06:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.951+0000 7f8ea97fa700 1 -- 172.21.15.46:0/3011377698 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec00229c0 con 0x55697276a740 2024-01-27T01:06:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.951+0000 7f8ea97fa700 1 -- 172.21.15.46:0/3011377698 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f8ec0022b20 con 0x55697276a740 2024-01-27T01:06:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.951+0000 7f8edad23b80 1 -- 172.21.15.46:0/3011377698 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556972736190 con 0x55697276a740 2024-01-27T01:06:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.951+0000 7f8ea97fa700 1 -- 172.21.15.46:0/3011377698 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ec0005a00 con 0x55697276a740 2024-01-27T01:06:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.951+0000 7f8edad23b80 10 client.10989 fetch_fsmap learned FSMap version 116 2024-01-27T01:06:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.951+0000 7f8edad23b80 1 -- 172.21.15.46:0/3011377698 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=2+}) v3 -- 0x55697265f290 con 0x55697276a740 2024-01-27T01:06:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.952+0000 7f8ea97fa700 1 -- 172.21.15.46:0/3011377698 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556972736190 con 0x55697276a740 2024-01-27T01:06:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.952+0000 7f8ea97fa700 1 -- 172.21.15.46:0/3011377698 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 116) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f8ec0005b60 con 0x55697276a740 2024-01-27T01:06:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.952+0000 7f8edad23b80 10 client.10989 fetch_fsmap finished waiting for FSMap version 116 2024-01-27T01:06:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.952+0000 7f8edad23b80 10 client.10989 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-27T01:06:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.952+0000 7f8edad23b80 20 client.10989 populate_metadata read hostname 'smithi046' 2024-01-27T01:06:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.952+0000 7f8edad23b80 1 --2- 172.21.15.46:0/3011377698 >> [v2:172.21.15.46:6834/2483163449,v1:172.21.15.46:6835/2483163449] conn(0x556972788160 0x556972788530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.952+0000 7f8edad23b80 4 client.10989 mds_command: new command op to 10866 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-27T01:06:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.952+0000 7f8edad23b80 1 -- 172.21.15.46:0/3011377698 --> [v2:172.21.15.46:6834/2483163449,v1:172.21.15.46:6835/2483163449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556972633ee0 con 0x556972788160 2024-01-27T01:06:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.952+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/3011377698 >> [v2:172.21.15.46:6834/2483163449,v1:172.21.15.46:6835/2483163449] conn(0x556972788160 0x556972788530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.952+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/3011377698 >> [v2:172.21.15.46:6834/2483163449,v1:172.21.15.46:6835/2483163449] conn(0x556972788160 0x556972788530 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44c6499558fa549 server_cookie=f2e08e1e1da639f0 in_seq=0 out_seq=0 2024-01-27T01:06:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.952+0000 7f8ea97fa700 10 client.10989 ms_handle_connect on v2:172.21.15.46:6834/2483163449 2024-01-27T01:06:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.953+0000 7f8ea97fa700 1 -- 172.21.15.46:0/3011377698 <== mds.0 v2:172.21.15.46:6834/2483163449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x556972633ee0 con 0x556972788160 2024-01-27T01:06:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.953+0000 7f8ea97fa700 10 client.10989 handle_command_reply: tid=0 2024-01-27T01:06:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.953+0000 7f8edad23b80 1 client.10989 shutdown 2024-01-27T01:06:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.953+0000 7f8edad23b80 1 -- 172.21.15.46:0/3011377698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 msgr2=0x55697273e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:32.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.953+0000 7f8edad23b80 1 --2- 172.21.15.46:0/3011377698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697273e010 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0015aa0 tx=0x7f8ec0019000 comp rx=0 tx=0).stop 2024-01-27T01:06:32.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.953+0000 7f8edad23b80 1 -- 172.21.15.46:0/3011377698 shutdown_connections 2024-01-27T01:06:32.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.953+0000 7f8edad23b80 1 --2- 172.21.15.46:0/3011377698 >> [v2:172.21.15.46:6834/2483163449,v1:172.21.15.46:6835/2483163449] conn(0x556972788160 0x556972788530 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.953+0000 7f8edad23b80 1 --2- 172.21.15.46:0/3011377698 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55697278b360 0x556972784ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.953+0000 7f8edad23b80 1 --2- 172.21.15.46:0/3011377698 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556972742e40 0x55697273dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.953+0000 7f8ea97fa700 0 client.10989 ms_handle_reset on v2:172.21.15.46:6834/2483163449 2024-01-27T01:06:32.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.953+0000 7f8edad23b80 1 --2- 172.21.15.46:0/3011377698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697273e010 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.953+0000 7f8edad23b80 1 -- 172.21.15.46:0/3011377698 >> 172.21.15.46:0/3011377698 conn(0x556972742a70 msgr2=0x55697273c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:32.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.953+0000 7f8edad23b80 1 -- 172.21.15.46:0/3011377698 shutdown_connections 2024-01-27T01:06:32.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.953+0000 7f8edad23b80 1 -- 172.21.15.46:0/3011377698 wait complete. 2024-01-27T01:06:32.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.953+0000 7f8edad23b80 20 client.10989 trim_cache size 0 max 16384 2024-01-27T01:06:32.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.971+0000 7f8edad23b80 1 Processor -- start 2024-01-27T01:06:32.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.971+0000 7f8edad23b80 1 -- start start 2024-01-27T01:06:32.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.972+0000 7f8edad23b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556972742e40 0x5569727433a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.972+0000 7f8edad23b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697273c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.972+0000 7f8edad23b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55697278b360 0x55697277dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.972+0000 7f8edad23b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x556972589e40 con 0x55697276a740 2024-01-27T01:06:32.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.972+0000 7f8edad23b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55697273e800 con 0x55697278b360 2024-01-27T01:06:32.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.972+0000 7f8edad23b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x556972764eb0 con 0x556972742e40 2024-01-27T01:06:32.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.972+0000 7f8eca7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697273c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.972+0000 7f8eca7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697273c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33816/0 (socket says 172.21.15.46:33816) 2024-01-27T01:06:32.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.972+0000 7f8eca7fc700 1 -- 172.21.15.46:0/1510507558 learned_addr learned my addr 172.21.15.46:0/1510507558 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:32.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.973+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/1510507558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556972742e40 0x5569727433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.973+0000 7f8ecb7fe700 1 --2- 172.21.15.46:0/1510507558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55697278b360 0x55697277dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.973+0000 7f8eca7fc700 1 -- 172.21.15.46:0/1510507558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556972742e40 msgr2=0x5569727433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:32.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.973+0000 7f8eca7fc700 1 --2- 172.21.15.46:0/1510507558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556972742e40 0x5569727433a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.973+0000 7f8eca7fc700 1 -- 172.21.15.46:0/1510507558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55697278b360 msgr2=0x55697277dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:32.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.973+0000 7f8eca7fc700 1 --2- 172.21.15.46:0/1510507558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55697278b360 0x55697277dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.973+0000 7f8eca7fc700 1 -- 172.21.15.46:0/1510507558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556972528c80 con 0x55697276a740 2024-01-27T01:06:32.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.973+0000 7f8ecb7fe700 1 --2- 172.21.15.46:0/1510507558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55697278b360 0x55697277dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:06:32.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.973+0000 7f8eca7fc700 1 --2- 172.21.15.46:0/1510507558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697273c410 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0000c00 tx=0x7f8ec0020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1097a0506319dd3 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:32.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.973+0000 7f8eb0ff9700 1 -- 172.21.15.46:0/1510507558 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0022a90 con 0x55697276a740 2024-01-27T01:06:32.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.973+0000 7f8eb0ff9700 1 -- 172.21.15.46:0/1510507558 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec0005a00 con 0x55697276a740 2024-01-27T01:06:32.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.973+0000 7f8eb0ff9700 1 -- 172.21.15.46:0/1510507558 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0006070 con 0x55697276a740 2024-01-27T01:06:32.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.974+0000 7f8edad23b80 1 -- 172.21.15.46:0/1510507558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 msgr2=0x55697273c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:32.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.974+0000 7f8edad23b80 1 --2- 172.21.15.46:0/1510507558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697273c410 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0000c00 tx=0x7f8ec0020000 comp rx=0 tx=0).stop 2024-01-27T01:06:32.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.974+0000 7f8edad23b80 1 -- 172.21.15.46:0/1510507558 shutdown_connections 2024-01-27T01:06:32.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.974+0000 7f8edad23b80 1 --2- 172.21.15.46:0/1510507558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55697278b360 0x55697277dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.974+0000 7f8edad23b80 1 --2- 172.21.15.46:0/1510507558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556972742e40 0x5569727433a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.974+0000 7f8edad23b80 1 --2- 172.21.15.46:0/1510507558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x55697273c410 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.974+0000 7f8edad23b80 1 -- 172.21.15.46:0/1510507558 >> 172.21.15.46:0/1510507558 conn(0x556972742a70 msgr2=0x55697278b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:32.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.974+0000 7f8edad23b80 1 -- 172.21.15.46:0/1510507558 shutdown_connections 2024-01-27T01:06:32.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.974+0000 7f8edad23b80 1 -- 172.21.15.46:0/1510507558 wait complete. 2024-01-27T01:06:32.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.974+0000 7f8edad23b80 1 -- start start 2024-01-27T01:06:32.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.974+0000 7f8edad23b80 1 Processor -- start 2024-01-27T01:06:32.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.975+0000 7f8edad23b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556972742e40 0x5569727f1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.975+0000 7f8edad23b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x5569727eaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.975+0000 7f8edad23b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55697278b360 0x556972786bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.975+0000 7f8edad23b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569727eaf70 con 0x55697276a740 2024-01-27T01:06:32.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.975+0000 7f8edad23b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x556972787940 con 0x55697278b360 2024-01-27T01:06:32.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.975+0000 7f8edad23b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5569727f11c0 con 0x556972742e40 2024-01-27T01:06:32.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.975+0000 7f8edad23b80 10 client.0 _ll_register_callbacks cb 0x55697249f7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-27T01:06:32.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.975+0000 7f8eca7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x5569727eaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.975+0000 7f8eca7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x5569727eaa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33820/0 (socket says 172.21.15.46:33820) 2024-01-27T01:06:32.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.976+0000 7f8eca7fc700 1 -- 172.21.15.46:0/3712922558 learned_addr learned my addr 172.21.15.46:0/3712922558 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:32.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.976+0000 7f8ecb7fe700 1 --2- 172.21.15.46:0/3712922558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55697278b360 0x556972786bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.976+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/3712922558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556972742e40 0x5569727f1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.976+0000 7f8eca7fc700 1 -- 172.21.15.46:0/3712922558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556972742e40 msgr2=0x5569727f1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:32.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.976+0000 7f8eca7fc700 1 --2- 172.21.15.46:0/3712922558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556972742e40 0x5569727f1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.976+0000 7f8eca7fc700 1 -- 172.21.15.46:0/3712922558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55697278b360 msgr2=0x556972786bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:32.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.976+0000 7f8eca7fc700 1 --2- 172.21.15.46:0/3712922558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55697278b360 0x556972786bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.976+0000 7f8eca7fc700 1 -- 172.21.15.46:0/3712922558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55697265f290 con 0x55697276a740 2024-01-27T01:06:32.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.976+0000 7f8eca7fc700 1 --2- 172.21.15.46:0/3712922558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x5569727eaa30 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0009270 tx=0x7f8ec003a070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=585a44523b0f5893 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:32.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.976+0000 7f8ea97fa700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-27T01:06:32.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.976+0000 7f8ea97fa700 1 -- 172.21.15.46:0/3712922558 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0006070 con 0x55697276a740 2024-01-27T01:06:32.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.976+0000 7f8edad23b80 1 -- 172.21.15.46:0/3712922558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ecc15cf20 con 0x55697276a740 2024-01-27T01:06:32.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.976+0000 7f8ea97fa700 1 -- 172.21.15.46:0/3712922558 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f8ec0006460 con 0x55697276a740 2024-01-27T01:06:32.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.976+0000 7f8ea97fa700 1 -- 172.21.15.46:0/3712922558 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ec0006830 con 0x55697276a740 2024-01-27T01:06:32.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.976+0000 7f8edad23b80 10 client.10995 fetch_fsmap learned FSMap version 116 2024-01-27T01:06:32.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.976+0000 7f8edad23b80 1 -- 172.21.15.46:0/3712922558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=2+}) v3 -- 0x5569724cefe0 con 0x55697276a740 2024-01-27T01:06:32.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.977+0000 7f8ea97fa700 1 -- 172.21.15.46:0/3712922558 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ecc15cf20 con 0x55697276a740 2024-01-27T01:06:32.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.977+0000 7f8ea97fa700 1 -- 172.21.15.46:0/3712922558 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 116) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f8ec0049320 con 0x55697276a740 2024-01-27T01:06:32.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.977+0000 7f8edad23b80 10 client.10995 fetch_fsmap finished waiting for FSMap version 116 2024-01-27T01:06:32.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.977+0000 7f8edad23b80 10 client.10995 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-27T01:06:32.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.977+0000 7f8edad23b80 20 client.10995 populate_metadata read hostname 'smithi046' 2024-01-27T01:06:32.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.977+0000 7f8edad23b80 1 --2- 172.21.15.46:0/3712922558 >> [v2:172.21.15.46:6834/2483163449,v1:172.21.15.46:6835/2483163449] conn(0x55697277dfb0 0x55697277e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:32.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.977+0000 7f8edad23b80 4 client.10995 mds_command: new command op to 10866 tid=0[{"prefix": "status"}] 2024-01-27T01:06:32.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.977+0000 7f8edad23b80 1 -- 172.21.15.46:0/3712922558 --> [v2:172.21.15.46:6834/2483163449,v1:172.21.15.46:6835/2483163449] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f8ebc0573e0 con 0x55697277dfb0 2024-01-27T01:06:32.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.977+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/3712922558 >> [v2:172.21.15.46:6834/2483163449,v1:172.21.15.46:6835/2483163449] conn(0x55697277dfb0 0x55697277e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:32.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.977+0000 7f8ecaffd700 1 --2- 172.21.15.46:0/3712922558 >> [v2:172.21.15.46:6834/2483163449,v1:172.21.15.46:6835/2483163449] conn(0x55697277dfb0 0x55697277e380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fe33fd8615d96a0 server_cookie=e2bdc5b1270872e1 in_seq=0 out_seq=0 2024-01-27T01:06:32.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.977+0000 7f8ea97fa700 10 client.10995 ms_handle_connect on v2:172.21.15.46:6834/2483163449 2024-01-27T01:06:32.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.977+0000 7f8ea97fa700 1 -- 172.21.15.46:0/3712922558 <== mds.0 v2:172.21.15.46:6834/2483163449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x7f8ebc0573e0 con 0x55697277dfb0 2024-01-27T01:06:32.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.977+0000 7f8ea97fa700 10 client.10995 handle_command_reply: tid=0 2024-01-27T01:06:32.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.977+0000 7f8edad23b80 1 client.10995 shutdown 2024-01-27T01:06:32.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8edad23b80 1 -- 172.21.15.46:0/3712922558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 msgr2=0x5569727eaa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:32.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8edad23b80 1 --2- 172.21.15.46:0/3712922558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x5569727eaa30 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0009270 tx=0x7f8ec003a070 comp rx=0 tx=0).stop 2024-01-27T01:06:32.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8edad23b80 1 -- 172.21.15.46:0/3712922558 shutdown_connections 2024-01-27T01:06:32.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8edad23b80 1 --2- 172.21.15.46:0/3712922558 >> [v2:172.21.15.46:6834/2483163449,v1:172.21.15.46:6835/2483163449] conn(0x55697277dfb0 0x55697277e380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8edad23b80 1 --2- 172.21.15.46:0/3712922558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55697278b360 0x556972786bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8edad23b80 1 --2- 172.21.15.46:0/3712922558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556972742e40 0x5569727f1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8edad23b80 1 --2- 172.21.15.46:0/3712922558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55697276a740 0x5569727eaa30 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8ea97fa700 0 client.10995 ms_handle_reset on v2:172.21.15.46:6834/2483163449 2024-01-27T01:06:32.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8edad23b80 1 -- 172.21.15.46:0/3712922558 >> 172.21.15.46:0/3712922558 conn(0x556972742a70 msgr2=0x556972770fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:32.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8edad23b80 1 -- 172.21.15.46:0/3712922558 shutdown_connections 2024-01-27T01:06:32.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8edad23b80 1 -- 172.21.15.46:0/3712922558 wait complete. 2024-01-27T01:06:32.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8edad23b80 20 client.10995 trim_cache size 0 max 16384 2024-01-27T01:06:32.983 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:06:32.983 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-27T01:06:32.983 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-27T01:06:32.983 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 10866, 2024-01-27T01:06:32.983 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-27T01:06:32.983 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-27T01:06:32.983 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-01-27T01:06:32.983 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.1149157079999998, 2024-01-27T01:06:32.984 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 116, 2024-01-27T01:06:32.984 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 223, 2024-01-27T01:06:32.984 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 223, 2024-01-27T01:06:32.984 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 11.183331899000001 2024-01-27T01:06:32.984 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:06:32.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8ed1a20700 1 -- 172.21.15.46:0/817718176 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8eb8066f30 msgr2=0x7f8eb80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:32.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8ed1a20700 1 --2- 172.21.15.46:0/817718176 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8eb8066f30 0x7f8eb80693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc06e440 tx=0x7f8ec00091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8ed1a20700 1 -- 172.21.15.46:0/817718176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb4003a60 msgr2=0x7f8ecc15d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:32.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8ed1a20700 1 --2- 172.21.15.46:0/817718176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb4003a60 0x7f8ecc15d5f0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4004730 tx=0x7f8ebc019000 comp rx=0 tx=0).stop 2024-01-27T01:06:32.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8ed1a20700 1 -- 172.21.15.46:0/817718176 shutdown_connections 2024-01-27T01:06:32.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8ed1a20700 1 --2- 172.21.15.46:0/817718176 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8eb8066f30 0x7f8eb80693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8ed1a20700 1 --2- 172.21.15.46:0/817718176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8eb4003a60 0x7f8ecc15d5f0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8ed1a20700 1 --2- 172.21.15.46:0/817718176 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ecc166750 0x7f8ecc166bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8ed1a20700 1 --2- 172.21.15.46:0/817718176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ecc0ca780 0x7f8ecc0cabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:32.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.978+0000 7f8ed1a20700 1 -- 172.21.15.46:0/817718176 >> 172.21.15.46:0/817718176 conn(0x7f8ecc068400 msgr2=0x7f8ecc1563b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:32.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.979+0000 7f8ed1a20700 1 -- 172.21.15.46:0/817718176 shutdown_connections 2024-01-27T01:06:32.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:32.979+0000 7f8ed1a20700 1 -- 172.21.15.46:0/817718176 wait complete. 2024-01-27T01:06:32.991 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-27T01:06:33.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.123+0000 7f9bfe9ca700 1 Processor -- start 2024-01-27T01:06:33.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.123+0000 7f9bfe9ca700 1 -- start start 2024-01-27T01:06:33.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.124+0000 7f9bfe9ca700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bf80b1fc0 0x7f9bf80b2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:33.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.124+0000 7f9bfe9ca700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9bf8068310 0x7f9bf8068780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:33.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.124+0000 7f9bfe9ca700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf80cac60 con 0x7f9bf8156f80 2024-01-27T01:06:33.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.124+0000 7f9bfe9ca700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf8068dc0 con 0x7f9bf8068310 2024-01-27T01:06:33.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.124+0000 7f9bfe9ca700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf80690a0 con 0x7f9bf80b1fc0 2024-01-27T01:06:33.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.124+0000 7f9bfcf67700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9bf8156f80 0x7f9bf81573b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54156/0 (socket says 172.21.15.46:54156) 2024-01-27T01:06:33.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.124+0000 7f9bfcf67700 1 -- 172.21.15.46:0/3086720173 learned_addr learned my addr 172.21.15.46:0/3086720173 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.124+0000 7f9bf77fe700 1 --2- 172.21.15.46:0/3086720173 >> v2:172.21.15.106:3300/0 conn(0x7f9bf8068310 0x7f9bf8068780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.124+0000 7f9bf7fff700 1 --2- 172.21.15.46:0/3086720173 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bf80b1fc0 0x7f9bf80b2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.124+0000 7f9bf6ffd700 1 -- 172.21.15.46:0/3086720173 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744707116 0 0) 0x7f9bf80cac60 con 0x7f9bf8156f80 2024-01-27T01:06:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.124+0000 7f9bf6ffd700 1 -- 172.21.15.46:0/3086720173 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be0003570 con 0x7f9bf8156f80 2024-01-27T01:06:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.125+0000 7f9bf77fe700 1 -- 172.21.15.46:0/3086720173 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bf80b1fc0 msgr2=0x7f9bf80b2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.125+0000 7f9bf77fe700 1 --2- 172.21.15.46:0/3086720173 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bf80b1fc0 0x7f9bf80b2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.125+0000 7f9bf77fe700 1 -- 172.21.15.46:0/3086720173 >> v1:172.21.15.46:6789/0 conn(0x7f9bf8156f80 legacy=0x7f9bf81573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.125+0000 7f9bf77fe700 1 -- 172.21.15.46:0/3086720173 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf80caee0 con 0x7f9bf8068310 2024-01-27T01:06:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.125+0000 7f9bf7fff700 1 --2- 172.21.15.46:0/3086720173 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bf80b1fc0 0x7f9bf80b2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.125+0000 7f9bf77fe700 1 --2- 172.21.15.46:0/3086720173 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bf8068310 0x7f9bf8068780 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0098e0 tx=0x7f9bec0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d94cc55b195800b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.125+0000 7f9bf6ffd700 1 -- 172.21.15.46:0/3086720173 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec011630 con 0x7f9bf8068310 2024-01-27T01:06:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.125+0000 7f9bf6ffd700 1 -- 172.21.15.46:0/3086720173 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bec011790 con 0x7f9bf8068310 2024-01-27T01:06:33.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.125+0000 7f9bf6ffd700 1 -- 172.21.15.46:0/3086720173 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec011b30 con 0x7f9bf8068310 2024-01-27T01:06:33.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.126+0000 7f9bfe9ca700 1 -- 172.21.15.46:0/3086720173 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bf8068310 msgr2=0x7f9bf8068780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:33.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.126+0000 7f9bfe9ca700 1 --2- 172.21.15.46:0/3086720173 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bf8068310 0x7f9bf8068780 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0098e0 tx=0x7f9bec0091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:33.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.126+0000 7f9bfe9ca700 1 -- 172.21.15.46:0/3086720173 shutdown_connections 2024-01-27T01:06:33.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.126+0000 7f9bfe9ca700 1 --2- 172.21.15.46:0/3086720173 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bf80b1fc0 0x7f9bf80b2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:33.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.126+0000 7f9bfe9ca700 1 --2- 172.21.15.46:0/3086720173 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bf8068310 0x7f9bf8068780 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:33.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.126+0000 7f9bfe9ca700 1 -- 172.21.15.46:0/3086720173 >> 172.21.15.46:0/3086720173 conn(0x7f9bf80ccbc0 msgr2=0x7f9bf80ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:33.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.126+0000 7f9bfe9ca700 1 -- 172.21.15.46:0/3086720173 shutdown_connections 2024-01-27T01:06:33.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.126+0000 7f9bfe9ca700 1 -- 172.21.15.46:0/3086720173 wait complete. 2024-01-27T01:06:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.127+0000 7f9bfe9ca700 1 Processor -- start 2024-01-27T01:06:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.127+0000 7f9bfe9ca700 1 -- start start 2024-01-27T01:06:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.128+0000 7f9bfe9ca700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bf8068310 0x7f9bf8146940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.128+0000 7f9bfe9ca700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf80b1fc0 0x7f9bf814d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.128+0000 7f9bfe9ca700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bf8156f80 0x7f9bf816ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.128+0000 7f9bfe9ca700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf8157c60 con 0x7f9bf80b1fc0 2024-01-27T01:06:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.128+0000 7f9bfe9ca700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf81471a0 con 0x7f9bf8068310 2024-01-27T01:06:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.128+0000 7f9bfe9ca700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf8148000 con 0x7f9bf8156f80 2024-01-27T01:06:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.128+0000 7f9bf77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf80b1fc0 0x7f9bf814d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.128+0000 7f9bf77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf80b1fc0 0x7f9bf814d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33830/0 (socket says 172.21.15.46:33830) 2024-01-27T01:06:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.128+0000 7f9bf77fe700 1 -- 172.21.15.46:0/3390356436 learned_addr learned my addr 172.21.15.46:0/3390356436 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.128+0000 7f9bf7fff700 1 --2- 172.21.15.46:0/3390356436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bf8068310 0x7f9bf8146940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.128+0000 7f9bfcf67700 1 --2- 172.21.15.46:0/3390356436 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bf8156f80 0x7f9bf816ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.128+0000 7f9bf77fe700 1 -- 172.21.15.46:0/3390356436 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bf8156f80 msgr2=0x7f9bf816ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.128+0000 7f9bf77fe700 1 --2- 172.21.15.46:0/3390356436 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bf8156f80 0x7f9bf816ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.128+0000 7f9bf77fe700 1 -- 172.21.15.46:0/3390356436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bf8068310 msgr2=0x7f9bf8146940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.128+0000 7f9bf77fe700 1 --2- 172.21.15.46:0/3390356436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bf8068310 0x7f9bf8146940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.128+0000 7f9bf77fe700 1 -- 172.21.15.46:0/3390356436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be8009420 con 0x7f9bf80b1fc0 2024-01-27T01:06:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.129+0000 7f9bf77fe700 1 --2- 172.21.15.46:0/3390356436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf80b1fc0 0x7f9bf814d2c0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f9bec009ed0 tx=0x7f9bec017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.129+0000 7f9bdffff700 1 -- 172.21.15.46:0/3390356436 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec000d10 con 0x7f9bf80b1fc0 2024-01-27T01:06:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.129+0000 7f9bdffff700 1 -- 172.21.15.46:0/3390356436 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bec011b10 con 0x7f9bf80b1fc0 2024-01-27T01:06:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.129+0000 7f9bfe9ca700 1 -- 172.21.15.46:0/3390356436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bec009420 con 0x7f9bf80b1fc0 2024-01-27T01:06:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.129+0000 7f9bdffff700 1 -- 172.21.15.46:0/3390356436 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec025660 con 0x7f9bf80b1fc0 2024-01-27T01:06:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.129+0000 7f9bfe9ca700 1 -- 172.21.15.46:0/3390356436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf81479b0 con 0x7f9bf80b1fc0 2024-01-27T01:06:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.130+0000 7f9bfe9ca700 1 -- 172.21.15.46:0/3390356436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf804c710 con 0x7f9bf80b1fc0 2024-01-27T01:06:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.130+0000 7f9bdffff700 1 -- 172.21.15.46:0/3390356436 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9bec025a50 con 0x7f9bf80b1fc0 2024-01-27T01:06:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.131+0000 7f9bdffff700 1 --2- 172.21.15.46:0/3390356436 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9be4066f30 0x7f9be40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.131+0000 7f9bdffff700 1 -- 172.21.15.46:0/3390356436 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f9bec086000 con 0x7f9bf80b1fc0 2024-01-27T01:06:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.131+0000 7f9bf7fff700 1 --2- 172.21.15.46:0/3390356436 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9be4066f30 0x7f9be40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.132+0000 7f9bf7fff700 1 --2- 172.21.15.46:0/3390356436 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9be4066f30 0x7f9be40693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9be8009f40 tx=0x7f9be80091a0 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:33.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.136+0000 7f9bdffff700 1 -- 172.21.15.46:0/3390356436 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9bec052f40 con 0x7f9bf80b1fc0 2024-01-27T01:06:33.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.389+0000 7f9bfe9ca700 1 -- 172.21.15.46:0/3390356436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9bf80cb1c0 con 0x7f9bf80b1fc0 2024-01-27T01:06:33.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.389+0000 7f9bdffff700 1 -- 172.21.15.46:0/3390356436 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v116) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9bec056560 con 0x7f9bf80b1fc0 2024-01-27T01:06:33.389 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-27T01:06:33.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.392+0000 7f9bfe9ca700 1 -- 172.21.15.46:0/3390356436 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9be4066f30 msgr2=0x7f9be40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:33.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.392+0000 7f9bfe9ca700 1 --2- 172.21.15.46:0/3390356436 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9be4066f30 0x7f9be40693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9be8009f40 tx=0x7f9be80091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:33.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.392+0000 7f9bfe9ca700 1 -- 172.21.15.46:0/3390356436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf80b1fc0 msgr2=0x7f9bf814d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:33.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.392+0000 7f9bfe9ca700 1 --2- 172.21.15.46:0/3390356436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf80b1fc0 0x7f9bf814d2c0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f9bec009ed0 tx=0x7f9bec017000 comp rx=0 tx=0).stop 2024-01-27T01:06:33.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.392+0000 7f9bfe9ca700 1 -- 172.21.15.46:0/3390356436 shutdown_connections 2024-01-27T01:06:33.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.392+0000 7f9bfe9ca700 1 --2- 172.21.15.46:0/3390356436 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9be4066f30 0x7f9be40693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:33.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.392+0000 7f9bfe9ca700 1 --2- 172.21.15.46:0/3390356436 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9bf8156f80 0x7f9bf816ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:33.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.392+0000 7f9bfe9ca700 1 --2- 172.21.15.46:0/3390356436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bf8068310 0x7f9bf8146940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:33.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.392+0000 7f9bfe9ca700 1 --2- 172.21.15.46:0/3390356436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf80b1fc0 0x7f9bf814d2c0 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:33.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.392+0000 7f9bfe9ca700 1 -- 172.21.15.46:0/3390356436 >> 172.21.15.46:0/3390356436 conn(0x7f9bf80ccbc0 msgr2=0x7f9bf81517b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:33.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.392+0000 7f9bfe9ca700 1 -- 172.21.15.46:0/3390356436 shutdown_connections 2024-01-27T01:06:33.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:33.392+0000 7f9bfe9ca700 1 -- 172.21.15.46:0/3390356436 wait complete. 2024-01-27T01:06:33.405 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-27T01:06:33.405 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-27T01:06:33.405 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-27T01:06:33.405 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-27T01:06:33.405 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-27T01:06:33.405 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-27T01:06:33.405 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-27T01:06:33.405 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-27T01:06:33.405 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-27T01:06:33.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:06:33.406 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-27T01:06:33.426 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-27T01:06:33.426 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-27T01:06:33.426 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-27T01:06:33.426 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:06:33.426 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-27T01:06:33.426 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:06:33.426 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T01:06:33.426 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:06:33.426 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-27T01:06:33.426 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T01:06:33.426 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:06:33.426 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-27T01:06:33.426 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-27T01:06:33.427 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:06:33.427 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-27T01:06:33.427 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-27T01:06:33.427 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 36537sec preferred_lft 36537sec 2024-01-27T01:06:33.427 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-27T01:06:33.427 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:06:33.427 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T01:06:33.427 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:06:33.427 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-27T01:06:33.427 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-27T01:06:33.427 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:06:33.427 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-27T01:06:33.427 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:06:33.427 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::cce6:a7ff:fe4f:38d/64 scope link 2024-01-27T01:06:33.428 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:06:33.428 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-27T01:06:33.428 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 76:ef:90:4c:9d:6f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-27T01:06:33.428 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::74ef:90ff:fe4c:9d6f/64 scope link 2024-01-27T01:06:33.428 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:06:33.428 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-27T01:06:33.428 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-27T01:06:33.428 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::2c73:afff:fef0:6f39/64 scope link 2024-01-27T01:06:33.428 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:06:33.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:06:33.429 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-27T01:06:33.496 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-27T01:06:33.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:06:33.498 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-27T01:06:33.563 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-27T01:06:33.563 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-27T01:06:33.563 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-27T01:06:33.563 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-27T01:06:33.563 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-27T01:06:33.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:06:33.564 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-27T01:06:33.564 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-27T01:06:33.564 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-27T01:06:33.564 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-27T01:06:33.564 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-27T01:06:33.564 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-27T01:06:33.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:06:33.689 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-27T01:06:33.709 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-27T01:06:33.710 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-27T01:06:33.710 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-27T01:06:33.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:06:33.710 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-01-27T01:06:33.786 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-27T01:06:33.786 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-27T01:06:33.786 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-27T01:06:33.848 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-27T01:06:33.848 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-27T01:06:33.848 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-27T01:06:33.911 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-27T01:06:33.987 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-27T01:06:33.987 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-27T01:06:34.061 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-27T01:06:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-27T01:06:34.063 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:06:34.063 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-27T01:06:34.125 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-27T01:06:34.125 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-27T01:06:34.168 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-27T01:06:34.203 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-27T01:06:34.203 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-27T01:06:34.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-01-27T01:06:34.272+0000 7fa47a33d540 -1 init, newargv = 0x561d09a34b90 newargc=15 2024-01-27T01:06:34.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[112010]: starting ceph client 2024-01-27T01:06:34.276 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-27T01:06:34.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[112010]: starting fuse 2024-01-27T01:06:34.278 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-27T01:06:34.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:06:34.278 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-27T01:06:34.339 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-27T01:06:34.339 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-27T01:06:34.339 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-27T01:06:34.339 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-27T01:06:34.339 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-27T01:06:34.339 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-27T01:06:34.339 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-27T01:06:34.340 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:06:34.340 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-27T01:06:34.340 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-27T01:06:34.340 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-27T01:06:34.340 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-27T01:06:34.340 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:06:34.340 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-27T01:06:34.340 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-27T01:06:34.340 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-27T01:06:34.340 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-27T01:06:34.340 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:06:34.340 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-27T01:06:34.340 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-27T01:06:34.340 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-27T01:06:34.340 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-27T01:06:34.341 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-27T01:06:34.341 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-27T01:06:34.341 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-27T01:06:34.341 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-27T01:06:34.341 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:06:34.341 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-01-27T01:06:34.341 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-27T01:06:34.480 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.112010.asok 2024-01-27T01:06:34.661 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.112010.asok 2024-01-27T01:06:34.662 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:06:34.662 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.112010.asok status 2024-01-27T01:06:34.837 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:06:34.837 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-27T01:06:34.837 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-27T01:06:34.837 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-27T01:06:34.837 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-01-27T01:06:34.837 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-27T01:06:34.837 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-27T01:06:34.837 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "112010", 2024-01-27T01:06:34.838 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-27T01:06:34.838 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:34.838 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-27T01:06:34.838 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-27T01:06:34.838 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 10630, 2024-01-27T01:06:34.838 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-27T01:06:34.838 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-27T01:06:34.838 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-27T01:06:34.838 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 10630 2024-01-27T01:06:34.838 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:34.838 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-27T01:06:34.838 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-27T01:06:34.838 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-27T01:06:34.838 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 415115610 2024-01-27T01:06:34.839 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:34.839 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:34.839 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-27T01:06:34.839 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-27T01:06:34.839 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-27T01:06:34.839 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 415115610 2024-01-27T01:06:34.839 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:34.839 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.10630 192.168.0.1:0/415115610", 2024-01-27T01:06:34.839 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/415115610", 2024-01-27T01:06:34.839 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-27T01:06:34.839 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 116, 2024-01-27T01:06:34.839 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 223, 2024-01-27T01:06:34.839 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-27T01:06:34.839 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-27T01:06:34.840 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-27T01:06:34.840 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:06:34.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:06:34.849 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-27T01:06:34.849 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-27T01:06:34.868 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-27T01:06:34.868 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-27T01:06:34.869 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:06:34.869 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-27T01:06:34.943 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-27T01:06:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.129+0000 7f5f8c26a700 1 Processor -- start 2024-01-27T01:06:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.129+0000 7f5f8c26a700 1 -- start start 2024-01-27T01:06:35.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.130+0000 7f5f8c26a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f8415ce40 0x7f5f840d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:35.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.130+0000 7f5f8c26a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5f8415d210 0x7f5f840cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:35.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.130+0000 7f5f8c26a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f840cff70 con 0x7f5f840d1390 2024-01-27T01:06:35.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.130+0000 7f5f8c26a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f840cf400 con 0x7f5f8415d210 2024-01-27T01:06:35.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.130+0000 7f5f8c26a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f840cf660 con 0x7f5f8415ce40 2024-01-27T01:06:35.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.130+0000 7f5f89805700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5f8415d210 0x7f5f840cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:35.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.130+0000 7f5f8a006700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f8415ce40 0x7f5f840d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:35.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.130+0000 7f5f8a807700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f5f840d1390 0x7f5f840ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:56964/0 (socket says 172.21.15.46:56964) 2024-01-27T01:06:35.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.130+0000 7f5f8a807700 1 -- 172.21.15.46:0/3525874198 learned_addr learned my addr 172.21.15.46:0/3525874198 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:35.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.131+0000 7f5f89805700 1 -- 172.21.15.46:0/3525874198 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f8415ce40 msgr2=0x7f5f840d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:35.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.131+0000 7f5f89805700 1 --2- 172.21.15.46:0/3525874198 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f8415ce40 0x7f5f840d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:35.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.131+0000 7f5f89805700 1 -- 172.21.15.46:0/3525874198 >> v1:172.21.15.106:6789/0 conn(0x7f5f840d1390 legacy=0x7f5f840ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:35.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.131+0000 7f5f89805700 1 -- 172.21.15.46:0/3525874198 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f840d0500 con 0x7f5f8415d210 2024-01-27T01:06:35.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.131+0000 7f5f89004700 1 -- 172.21.15.46:0/3525874198 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858664409 0 0) 0x7f5f840cff70 con 0x7f5f840d1390 2024-01-27T01:06:35.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.131+0000 7f5f8a006700 1 --2- 172.21.15.46:0/3525874198 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f8415ce40 0x7f5f840d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:35.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.131+0000 7f5f89805700 1 --2- 172.21.15.46:0/3525874198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f8415d210 0x7f5f840cbab0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5f780098e0 tx=0x7f5f780091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9480a9c94415949e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:35.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.131+0000 7f5f89004700 1 -- 172.21.15.46:0/3525874198 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7800b640 con 0x7f5f8415d210 2024-01-27T01:06:35.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.131+0000 7f5f89004700 1 -- 172.21.15.46:0/3525874198 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7800b7a0 con 0x7f5f8415d210 2024-01-27T01:06:35.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.131+0000 7f5f89004700 1 -- 172.21.15.46:0/3525874198 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7800bb40 con 0x7f5f8415d210 2024-01-27T01:06:35.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.132+0000 7f5f8c26a700 1 -- 172.21.15.46:0/3525874198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f8415d210 msgr2=0x7f5f840cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:35.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.132+0000 7f5f8c26a700 1 --2- 172.21.15.46:0/3525874198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f8415d210 0x7f5f840cbab0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5f780098e0 tx=0x7f5f780091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:35.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.132+0000 7f5f8c26a700 1 -- 172.21.15.46:0/3525874198 shutdown_connections 2024-01-27T01:06:35.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.132+0000 7f5f8c26a700 1 --2- 172.21.15.46:0/3525874198 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f8415ce40 0x7f5f840d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:35.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.132+0000 7f5f8c26a700 1 --2- 172.21.15.46:0/3525874198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f8415d210 0x7f5f840cbab0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:35.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.132+0000 7f5f8c26a700 1 -- 172.21.15.46:0/3525874198 >> 172.21.15.46:0/3525874198 conn(0x7f5f8406f840 msgr2=0x7f5f840d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:35.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.132+0000 7f5f8c26a700 1 -- 172.21.15.46:0/3525874198 shutdown_connections 2024-01-27T01:06:35.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.132+0000 7f5f8c26a700 1 -- 172.21.15.46:0/3525874198 wait complete. 2024-01-27T01:06:35.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.133+0000 7f5f8c26a700 1 Processor -- start 2024-01-27T01:06:35.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.133+0000 7f5f8c26a700 1 -- start start 2024-01-27T01:06:35.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.134+0000 7f5f8c26a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f840d1390 0x7f5f84154340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:35.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.134+0000 7f5f8c26a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f8415ce40 0x7f5f8409a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:35.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.134+0000 7f5f8c26a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f8415d210 0x7f5f8409ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:35.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.134+0000 7f5f8c26a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8409c890 con 0x7f5f8415ce40 2024-01-27T01:06:35.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.134+0000 7f5f8c26a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f84158460 con 0x7f5f8415d210 2024-01-27T01:06:35.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.134+0000 7f5f8c26a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8409cb50 con 0x7f5f840d1390 2024-01-27T01:06:35.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.134+0000 7f5f89805700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f8415ce40 0x7f5f8409a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:35.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.134+0000 7f5f89805700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f8415ce40 0x7f5f8409a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33836/0 (socket says 172.21.15.46:33836) 2024-01-27T01:06:35.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.134+0000 7f5f89805700 1 -- 172.21.15.46:0/187444906 learned_addr learned my addr 172.21.15.46:0/187444906 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:35.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.134+0000 7f5f8a807700 1 --2- 172.21.15.46:0/187444906 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f8415d210 0x7f5f8409ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:35.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.134+0000 7f5f8a006700 1 --2- 172.21.15.46:0/187444906 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f840d1390 0x7f5f84154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:35.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.135+0000 7f5f89805700 1 -- 172.21.15.46:0/187444906 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f840d1390 msgr2=0x7f5f84154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:35.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.135+0000 7f5f89805700 1 --2- 172.21.15.46:0/187444906 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f840d1390 0x7f5f84154340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:35.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.135+0000 7f5f89805700 1 -- 172.21.15.46:0/187444906 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f8415d210 msgr2=0x7f5f8409ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:35.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.135+0000 7f5f89805700 1 --2- 172.21.15.46:0/187444906 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f8415d210 0x7f5f8409ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:35.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.135+0000 7f5f89805700 1 -- 172.21.15.46:0/187444906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f80009420 con 0x7f5f8415ce40 2024-01-27T01:06:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.135+0000 7f5f8a807700 1 --2- 172.21.15.46:0/187444906 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f8415d210 0x7f5f8409ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.135+0000 7f5f89805700 1 --2- 172.21.15.46:0/187444906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f8415ce40 0x7f5f8409a7e0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f5f78013000 tx=0x7f5f78017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.135+0000 7f5f767fc700 1 -- 172.21.15.46:0/187444906 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7800bb20 con 0x7f5f8415ce40 2024-01-27T01:06:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.135+0000 7f5f767fc700 1 -- 172.21.15.46:0/187444906 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f78011420 con 0x7f5f8415ce40 2024-01-27T01:06:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.135+0000 7f5f8c26a700 1 -- 172.21.15.46:0/187444906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f78009420 con 0x7f5f8415ce40 2024-01-27T01:06:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.135+0000 7f5f767fc700 1 -- 172.21.15.46:0/187444906 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f780117c0 con 0x7f5f8415ce40 2024-01-27T01:06:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.135+0000 7f5f8c26a700 1 -- 172.21.15.46:0/187444906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f84157bc0 con 0x7f5f8415ce40 2024-01-27T01:06:35.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.136+0000 7f5f8c26a700 1 -- 172.21.15.46:0/187444906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f8409ccc0 con 0x7f5f8415ce40 2024-01-27T01:06:35.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.137+0000 7f5f767fc700 1 -- 172.21.15.46:0/187444906 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5f78011bb0 con 0x7f5f8415ce40 2024-01-27T01:06:35.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.137+0000 7f5f767fc700 1 --2- 172.21.15.46:0/187444906 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5f68066f30 0x7f5f680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:35.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.137+0000 7f5f767fc700 1 -- 172.21.15.46:0/187444906 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f5f780860a0 con 0x7f5f8415ce40 2024-01-27T01:06:35.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.137+0000 7f5f8a006700 1 --2- 172.21.15.46:0/187444906 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5f68066f30 0x7f5f680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:35.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.138+0000 7f5f8a006700 1 --2- 172.21.15.46:0/187444906 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5f68066f30 0x7f5f680693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5f800093f0 tx=0x7f5f80008040 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:35.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.142+0000 7f5f767fc700 1 -- 172.21.15.46:0/187444906 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5f78052fe0 con 0x7f5f8415ce40 2024-01-27T01:06:35.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.395+0000 7f5f8c26a700 1 -- 172.21.15.46:0/187444906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5f840cf920 con 0x7f5f8415ce40 2024-01-27T01:06:35.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.897+0000 7f5f767fc700 1 -- 172.21.15.46:0/187444906 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v117) v1 ==== 136+0+0 (secure 0 0 0) 0x7f5f78011e60 con 0x7f5f8415ce40 2024-01-27T01:06:35.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.900+0000 7f5f8c26a700 1 -- 172.21.15.46:0/187444906 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5f68066f30 msgr2=0x7f5f680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:35.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.900+0000 7f5f8c26a700 1 --2- 172.21.15.46:0/187444906 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5f68066f30 0x7f5f680693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5f800093f0 tx=0x7f5f80008040 comp rx=0 tx=0).stop 2024-01-27T01:06:35.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.900+0000 7f5f8c26a700 1 -- 172.21.15.46:0/187444906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f8415ce40 msgr2=0x7f5f8409a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:35.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.900+0000 7f5f8c26a700 1 --2- 172.21.15.46:0/187444906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f8415ce40 0x7f5f8409a7e0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f5f78013000 tx=0x7f5f78017000 comp rx=0 tx=0).stop 2024-01-27T01:06:35.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.900+0000 7f5f8c26a700 1 -- 172.21.15.46:0/187444906 shutdown_connections 2024-01-27T01:06:35.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.900+0000 7f5f8c26a700 1 --2- 172.21.15.46:0/187444906 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5f68066f30 0x7f5f680693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:35.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.900+0000 7f5f8c26a700 1 --2- 172.21.15.46:0/187444906 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f8415d210 0x7f5f8409ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:35.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.900+0000 7f5f8c26a700 1 --2- 172.21.15.46:0/187444906 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f840d1390 0x7f5f84154340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:35.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.900+0000 7f5f8c26a700 1 --2- 172.21.15.46:0/187444906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f8415ce40 0x7f5f8409a7e0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:35.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.900+0000 7f5f8c26a700 1 -- 172.21.15.46:0/187444906 >> 172.21.15.46:0/187444906 conn(0x7f5f8406f840 msgr2=0x7f5f8415b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:35.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.900+0000 7f5f8c26a700 1 -- 172.21.15.46:0/187444906 shutdown_connections 2024-01-27T01:06:35.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:35.900+0000 7f5f8c26a700 1 -- 172.21.15.46:0/187444906 wait complete. 2024-01-27T01:06:35.914 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:06:36.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.051+0000 7fc7cd20d700 1 Processor -- start 2024-01-27T01:06:36.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.051+0000 7fc7cd20d700 1 -- start start 2024-01-27T01:06:36.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.052+0000 7fc7cd20d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7c8159910 0x7fc7c8159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:36.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.052+0000 7fc7cd20d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc7c8068bf0 0x7fc7c80cb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:36.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.052+0000 7fc7cd20d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c80cb770 con 0x7fc7c8067930 2024-01-27T01:06:36.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.052+0000 7fc7cd20d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7c8069340 con 0x7fc7c8068bf0 2024-01-27T01:06:36.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.052+0000 7fc7cd20d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c80af9d0 con 0x7fc7c8159910 2024-01-27T01:06:36.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.052+0000 7fc7c759e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc7c8067930 0x7fc7c806a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54172/0 (socket says 172.21.15.46:54172) 2024-01-27T01:06:36.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.052+0000 7fc7c759e700 1 -- 172.21.15.46:0/4181235931 learned_addr learned my addr 172.21.15.46:0/4181235931 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.052+0000 7fc7c6d9d700 1 --2- 172.21.15.46:0/4181235931 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7c8159910 0x7fc7c8159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.052+0000 7fc7c659c700 1 --2- 172.21.15.46:0/4181235931 >> v2:172.21.15.106:3300/0 conn(0x7fc7c8068bf0 0x7fc7c80cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.052+0000 7fc7c5d9b700 1 -- 172.21.15.46:0/4181235931 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384551007 0 0) 0x7fc7c80cb770 con 0x7fc7c8067930 2024-01-27T01:06:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.052+0000 7fc7c5d9b700 1 -- 172.21.15.46:0/4181235931 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b0003570 con 0x7fc7c8067930 2024-01-27T01:06:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.052+0000 7fc7c6d9d700 1 -- 172.21.15.46:0/4181235931 >> v2:172.21.15.106:3300/0 conn(0x7fc7c8068bf0 msgr2=0x7fc7c80cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.052+0000 7fc7c6d9d700 1 --2- 172.21.15.46:0/4181235931 >> v2:172.21.15.106:3300/0 conn(0x7fc7c8068bf0 0x7fc7c80cb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.052+0000 7fc7c6d9d700 1 -- 172.21.15.46:0/4181235931 >> v1:172.21.15.46:6789/0 conn(0x7fc7c8067930 legacy=0x7fc7c806a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.052+0000 7fc7c5d9b700 1 -- 172.21.15.46:0/4181235931 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3016035536 0 0) 0x7fc7b0003570 con 0x7fc7c8067930 2024-01-27T01:06:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.052+0000 7fc7c6d9d700 1 -- 172.21.15.46:0/4181235931 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c80c7e90 con 0x7fc7c8159910 2024-01-27T01:06:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.053+0000 7fc7c6d9d700 1 --2- 172.21.15.46:0/4181235931 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7c8159910 0x7fc7c8159ce0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc7b80098e0 tx=0x7fc7b80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=766b502da50e375d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.053+0000 7fc7c5d9b700 1 -- 172.21.15.46:0/4181235931 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c00017f0 con 0x7fc7c8159910 2024-01-27T01:06:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.053+0000 7fc7c5d9b700 1 -- 172.21.15.46:0/4181235931 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b8011420 con 0x7fc7c8159910 2024-01-27T01:06:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.053+0000 7fc7c5d9b700 1 -- 172.21.15.46:0/4181235931 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b80117c0 con 0x7fc7c8159910 2024-01-27T01:06:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.053+0000 7fc7cd20d700 1 -- 172.21.15.46:0/4181235931 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7c8159910 msgr2=0x7fc7c8159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:36.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.053+0000 7fc7cd20d700 1 --2- 172.21.15.46:0/4181235931 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7c8159910 0x7fc7c8159ce0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc7b80098e0 tx=0x7fc7b80091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.053+0000 7fc7cd20d700 1 -- 172.21.15.46:0/4181235931 shutdown_connections 2024-01-27T01:06:36.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.053+0000 7fc7cd20d700 1 --2- 172.21.15.46:0/4181235931 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7c8159910 0x7fc7c8159ce0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.053+0000 7fc7cd20d700 1 --2- 172.21.15.46:0/4181235931 >> v2:172.21.15.106:3300/0 conn(0x7fc7c8068bf0 0x7fc7c80cb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.053+0000 7fc7cd20d700 1 -- 172.21.15.46:0/4181235931 >> 172.21.15.46:0/4181235931 conn(0x7fc7c80cd6e0 msgr2=0x7fc7c80c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:36.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.054+0000 7fc7cd20d700 1 -- 172.21.15.46:0/4181235931 shutdown_connections 2024-01-27T01:06:36.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.054+0000 7fc7cd20d700 1 -- 172.21.15.46:0/4181235931 wait complete. 2024-01-27T01:06:36.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.055+0000 7fc7cd20d700 1 Processor -- start 2024-01-27T01:06:36.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.055+0000 7fc7cd20d700 1 -- start start 2024-01-27T01:06:36.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.056+0000 7fc7cd20d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7c8067930 0x7fc7c815dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:36.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.056+0000 7fc7cd20d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c8068bf0 0x7fc7c815bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:36.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.056+0000 7fc7cd20d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7c8159910 0x7fc7c80afd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:36.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.056+0000 7fc7cd20d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c815cc10 con 0x7fc7c8068bf0 2024-01-27T01:06:36.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.056+0000 7fc7cd20d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c815e1a0 con 0x7fc7c8067930 2024-01-27T01:06:36.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.057+0000 7fc7cd20d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c815c180 con 0x7fc7c8159910 2024-01-27T01:06:36.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.057+0000 7fc7c659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c8068bf0 0x7fc7c815bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:36.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.057+0000 7fc7c659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c8068bf0 0x7fc7c815bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33840/0 (socket says 172.21.15.46:33840) 2024-01-27T01:06:36.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.057+0000 7fc7c659c700 1 -- 172.21.15.46:0/3708746275 learned_addr learned my addr 172.21.15.46:0/3708746275 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:36.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.057+0000 7fc7c6d9d700 1 --2- 172.21.15.46:0/3708746275 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7c8067930 0x7fc7c815dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:36.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.057+0000 7fc7c759e700 1 --2- 172.21.15.46:0/3708746275 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7c8159910 0x7fc7c80afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:36.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.057+0000 7fc7c659c700 1 -- 172.21.15.46:0/3708746275 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7c8159910 msgr2=0x7fc7c80afd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:36.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.057+0000 7fc7c659c700 1 --2- 172.21.15.46:0/3708746275 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7c8159910 0x7fc7c80afd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.057+0000 7fc7c659c700 1 -- 172.21.15.46:0/3708746275 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7c8067930 msgr2=0x7fc7c815dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:36.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.057+0000 7fc7c659c700 1 --2- 172.21.15.46:0/3708746275 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7c8067930 0x7fc7c815dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.057+0000 7fc7c659c700 1 -- 172.21.15.46:0/3708746275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b8009420 con 0x7fc7c8068bf0 2024-01-27T01:06:36.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.057+0000 7fc7c659c700 1 --2- 172.21.15.46:0/3708746275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c8068bf0 0x7fc7c815bc40 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc00ec80 tx=0x7fc7bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:36.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.057+0000 7fc7aeffd700 1 -- 172.21.15.46:0/3708746275 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc00c850 con 0x7fc7c8068bf0 2024-01-27T01:06:36.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.057+0000 7fc7aeffd700 1 -- 172.21.15.46:0/3708746275 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7bc00c9b0 con 0x7fc7c8068bf0 2024-01-27T01:06:36.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.057+0000 7fc7cd20d700 1 -- 172.21.15.46:0/3708746275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c813e4b0 con 0x7fc7c8068bf0 2024-01-27T01:06:36.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.057+0000 7fc7aeffd700 1 -- 172.21.15.46:0/3708746275 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc00cd50 con 0x7fc7c8068bf0 2024-01-27T01:06:36.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.058+0000 7fc7cd20d700 1 -- 172.21.15.46:0/3708746275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c815c930 con 0x7fc7c8068bf0 2024-01-27T01:06:36.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.058+0000 7fc7cd20d700 1 -- 172.21.15.46:0/3708746275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c804c710 con 0x7fc7c8068bf0 2024-01-27T01:06:36.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.059+0000 7fc7aeffd700 1 -- 172.21.15.46:0/3708746275 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc7bc013680 con 0x7fc7c8068bf0 2024-01-27T01:06:36.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.059+0000 7fc7aeffd700 1 --2- 172.21.15.46:0/3708746275 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc7b4066f30 0x7fc7b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:36.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.059+0000 7fc7aeffd700 1 -- 172.21.15.46:0/3708746275 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fc7bc084a80 con 0x7fc7c8068bf0 2024-01-27T01:06:36.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.060+0000 7fc7c6d9d700 1 --2- 172.21.15.46:0/3708746275 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc7b4066f30 0x7fc7b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:36.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.060+0000 7fc7c6d9d700 1 --2- 172.21.15.46:0/3708746275 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc7b4066f30 0x7fc7b40693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc7b80093f0 tx=0x7fc7b8016000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:36.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.064+0000 7fc7aeffd700 1 -- 172.21.15.46:0/3708746275 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc7bc0519c0 con 0x7fc7c8068bf0 2024-01-27T01:06:36.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.268+0000 7fc7cd20d700 1 -- 172.21.15.46:0/3708746275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7c80c1bd0 con 0x7fc7c8068bf0 2024-01-27T01:06:36.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.269+0000 7fc7aeffd700 1 -- 172.21.15.46:0/3708746275 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v223) v1 ==== 74+0+16325 (secure 0 0 0) 0x7fc7bc054fe0 con 0x7fc7c8068bf0 2024-01-27T01:06:36.268 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:36.268 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":223,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:06:27.861661+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:06:24.798791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-01-27T01:06:25.249215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:06:36.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.271+0000 7fc7cd20d700 1 -- 172.21.15.46:0/3708746275 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc7b4066f30 msgr2=0x7fc7b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:36.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.271+0000 7fc7cd20d700 1 --2- 172.21.15.46:0/3708746275 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc7b4066f30 0x7fc7b40693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc7b80093f0 tx=0x7fc7b8016000 comp rx=0 tx=0).stop 2024-01-27T01:06:36.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.271+0000 7fc7cd20d700 1 -- 172.21.15.46:0/3708746275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c8068bf0 msgr2=0x7fc7c815bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:36.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.271+0000 7fc7cd20d700 1 --2- 172.21.15.46:0/3708746275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c8068bf0 0x7fc7c815bc40 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc00ec80 tx=0x7fc7bc0091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.271+0000 7fc7cd20d700 1 -- 172.21.15.46:0/3708746275 shutdown_connections 2024-01-27T01:06:36.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.271+0000 7fc7cd20d700 1 --2- 172.21.15.46:0/3708746275 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc7b4066f30 0x7fc7b40693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.271+0000 7fc7cd20d700 1 --2- 172.21.15.46:0/3708746275 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7c8159910 0x7fc7c80afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.271+0000 7fc7cd20d700 1 --2- 172.21.15.46:0/3708746275 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7c8067930 0x7fc7c815dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.271+0000 7fc7cd20d700 1 --2- 172.21.15.46:0/3708746275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c8068bf0 0x7fc7c815bc40 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.271+0000 7fc7cd20d700 1 -- 172.21.15.46:0/3708746275 >> 172.21.15.46:0/3708746275 conn(0x7fc7c80cd6e0 msgr2=0x7fc7c806e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:36.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.271+0000 7fc7cd20d700 1 -- 172.21.15.46:0/3708746275 shutdown_connections 2024-01-27T01:06:36.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.271+0000 7fc7cd20d700 1 -- 172.21.15.46:0/3708746275 wait complete. 2024-01-27T01:06:36.285 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-27T01:06:24.798791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '222', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2024-01-27T01:06:25.249215+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '223', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-27T01:06:36.286 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-27T01:06:36.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.427+0000 7fb032148700 1 Processor -- start 2024-01-27T01:06:36.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.427+0000 7fb032148700 1 -- start start 2024-01-27T01:06:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.428+0000 7fb032148700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c068b00 0x7fb02c068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.428+0000 7fb032148700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb02c06b910 0x7fb02c15aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.428+0000 7fb032148700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb02c15a2b0 con 0x7fb02c0ad380 2024-01-27T01:06:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.428+0000 7fb032148700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02c15a620 con 0x7fb02c06b910 2024-01-27T01:06:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.428+0000 7fb032148700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c156b80 con 0x7fb02c068b00 2024-01-27T01:06:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.428+0000 7fb02bfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb02c0ad380 0x7fb02c0ad800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54188/0 (socket says 172.21.15.46:54188) 2024-01-27T01:06:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.428+0000 7fb02bfff700 1 -- 172.21.15.46:0/3313386972 learned_addr learned my addr 172.21.15.46:0/3313386972 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.429+0000 7fb02b7fe700 1 --2- 172.21.15.46:0/3313386972 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c068b00 0x7fb02c068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.429+0000 7fb02affd700 1 --2- 172.21.15.46:0/3313386972 >> v2:172.21.15.106:3300/0 conn(0x7fb02c06b910 0x7fb02c15aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.429+0000 7fb02a7fc700 1 -- 172.21.15.46:0/3313386972 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508710886 0 0) 0x7fb02c15a2b0 con 0x7fb02c0ad380 2024-01-27T01:06:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.429+0000 7fb02a7fc700 1 -- 172.21.15.46:0/3313386972 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb018003570 con 0x7fb02c0ad380 2024-01-27T01:06:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.429+0000 7fb02affd700 1 -- 172.21.15.46:0/3313386972 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c068b00 msgr2=0x7fb02c068ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.429+0000 7fb02a7fc700 1 -- 172.21.15.46:0/3313386972 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2475476056 0 0) 0x7fb018003570 con 0x7fb02c0ad380 2024-01-27T01:06:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.429+0000 7fb02affd700 1 --2- 172.21.15.46:0/3313386972 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c068b00 0x7fb02c068ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.429+0000 7fb02affd700 1 -- 172.21.15.46:0/3313386972 >> v1:172.21.15.46:6789/0 conn(0x7fb02c0ad380 legacy=0x7fb02c0ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.429+0000 7fb02affd700 1 -- 172.21.15.46:0/3313386972 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02c06c140 con 0x7fb02c06b910 2024-01-27T01:06:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.429+0000 7fb02b7fe700 1 --2- 172.21.15.46:0/3313386972 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c068b00 0x7fb02c068ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.429+0000 7fb02affd700 1 --2- 172.21.15.46:0/3313386972 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c06b910 0x7fb02c15aa40 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fb01c009910 tx=0x7fb01c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=324b74d3269a822 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.429+0000 7fb02a7fc700 1 -- 172.21.15.46:0/3313386972 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0240017f0 con 0x7fb02c06b910 2024-01-27T01:06:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.429+0000 7fb02a7fc700 1 -- 172.21.15.46:0/3313386972 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01c011420 con 0x7fb02c06b910 2024-01-27T01:06:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.430+0000 7fb02a7fc700 1 -- 172.21.15.46:0/3313386972 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c0117c0 con 0x7fb02c06b910 2024-01-27T01:06:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.430+0000 7fb032148700 1 -- 172.21.15.46:0/3313386972 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c06b910 msgr2=0x7fb02c15aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.430+0000 7fb032148700 1 --2- 172.21.15.46:0/3313386972 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c06b910 0x7fb02c15aa40 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fb01c009910 tx=0x7fb01c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.430+0000 7fb032148700 1 -- 172.21.15.46:0/3313386972 shutdown_connections 2024-01-27T01:06:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.430+0000 7fb032148700 1 --2- 172.21.15.46:0/3313386972 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c068b00 0x7fb02c068ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.430+0000 7fb032148700 1 --2- 172.21.15.46:0/3313386972 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c06b910 0x7fb02c15aa40 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.430+0000 7fb032148700 1 -- 172.21.15.46:0/3313386972 >> 172.21.15.46:0/3313386972 conn(0x7fb02c0ca100 msgr2=0x7fb02c06e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:36.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.430+0000 7fb032148700 1 -- 172.21.15.46:0/3313386972 shutdown_connections 2024-01-27T01:06:36.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.430+0000 7fb032148700 1 -- 172.21.15.46:0/3313386972 wait complete. 2024-01-27T01:06:36.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.432+0000 7fb032148700 1 Processor -- start 2024-01-27T01:06:36.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.432+0000 7fb032148700 1 -- start start 2024-01-27T01:06:36.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.432+0000 7fb032148700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c068b00 0x7fb02c0cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:36.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.432+0000 7fb032148700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c06b910 0x7fb02c14b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:36.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.432+0000 7fb032148700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c0ad380 0x7fb02c0c0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:36.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.432+0000 7fb032148700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c14b5f0 con 0x7fb02c06b910 2024-01-27T01:06:36.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.432+0000 7fb032148700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c0c0b80 con 0x7fb02c068b00 2024-01-27T01:06:36.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.432+0000 7fb032148700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c16db40 con 0x7fb02c0ad380 2024-01-27T01:06:36.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb02affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c06b910 0x7fb02c14b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:36.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb02b7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c068b00 0x7fb02c0cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:36.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb02b7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c068b00 0x7fb02c0cc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:52028/0 (socket says 172.21.15.46:52028) 2024-01-27T01:06:36.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb02affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c06b910 0x7fb02c14b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33856/0 (socket says 172.21.15.46:33856) 2024-01-27T01:06:36.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb02b7fe700 1 -- 172.21.15.46:0/2692098 learned_addr learned my addr 172.21.15.46:0/2692098 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:36.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb02bfff700 1 --2- 172.21.15.46:0/2692098 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c0ad380 0x7fb02c0c0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:36.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb02affd700 1 -- 172.21.15.46:0/2692098 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c0ad380 msgr2=0x7fb02c0c0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:36.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb02affd700 1 --2- 172.21.15.46:0/2692098 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c0ad380 0x7fb02c0c0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb02affd700 1 -- 172.21.15.46:0/2692098 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c068b00 msgr2=0x7fb02c0cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:36.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb02affd700 1 --2- 172.21.15.46:0/2692098 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c068b00 0x7fb02c0cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb02affd700 1 -- 172.21.15.46:0/2692098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0140093c0 con 0x7fb02c06b910 2024-01-27T01:06:36.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb02b7fe700 1 --2- 172.21.15.46:0/2692098 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c068b00 0x7fb02c0cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:36.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb02affd700 1 --2- 172.21.15.46:0/2692098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c06b910 0x7fb02c14b0b0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fb01c015000 tx=0x7fb01c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb0237fe700 1 -- 172.21.15.46:0/2692098 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c000d10 con 0x7fb02c06b910 2024-01-27T01:06:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb0237fe700 1 -- 172.21.15.46:0/2692098 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01c020070 con 0x7fb02c06b910 2024-01-27T01:06:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb032148700 1 -- 172.21.15.46:0/2692098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01c009420 con 0x7fb02c06b910 2024-01-27T01:06:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb0237fe700 1 -- 172.21.15.46:0/2692098 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c01bb60 con 0x7fb02c06b910 2024-01-27T01:06:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.433+0000 7fb032148700 1 -- 172.21.15.46:0/2692098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb02c16d210 con 0x7fb02c06b910 2024-01-27T01:06:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.434+0000 7fb032148700 1 -- 172.21.15.46:0/2692098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02c16dcb0 con 0x7fb02c06b910 2024-01-27T01:06:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.435+0000 7fb0237fe700 1 -- 172.21.15.46:0/2692098 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb01c022030 con 0x7fb02c06b910 2024-01-27T01:06:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.435+0000 7fb0237fe700 1 --2- 172.21.15.46:0/2692098 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb00c066f30 0x7fb00c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.435+0000 7fb0237fe700 1 -- 172.21.15.46:0/2692098 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fb01c085cc0 con 0x7fb02c06b910 2024-01-27T01:06:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.436+0000 7fb02b7fe700 1 --2- 172.21.15.46:0/2692098 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb00c066f30 0x7fb00c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.436+0000 7fb02b7fe700 1 --2- 172.21.15.46:0/2692098 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb00c066f30 0x7fb00c0693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb02c0c8070 tx=0x7fb014009120 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.440+0000 7fb0237fe700 1 -- 172.21.15.46:0/2692098 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb01c052c00 con 0x7fb02c06b910 2024-01-27T01:06:36.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.641+0000 7fb032148700 1 -- 172.21.15.46:0/2692098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb02c04c710 con 0x7fb02c06b910 2024-01-27T01:06:36.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.642+0000 7fb0237fe700 1 -- 172.21.15.46:0/2692098 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v223) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb01c056220 con 0x7fb02c06b910 2024-01-27T01:06:36.642 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-27T01:06:36.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.644+0000 7fb032148700 1 -- 172.21.15.46:0/2692098 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb00c066f30 msgr2=0x7fb00c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.644+0000 7fb032148700 1 --2- 172.21.15.46:0/2692098 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb00c066f30 0x7fb00c0693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb02c0c8070 tx=0x7fb014009120 comp rx=0 tx=0).stop 2024-01-27T01:06:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.644+0000 7fb032148700 1 -- 172.21.15.46:0/2692098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c06b910 msgr2=0x7fb02c14b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.644+0000 7fb032148700 1 --2- 172.21.15.46:0/2692098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c06b910 0x7fb02c14b0b0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fb01c015000 tx=0x7fb01c016000 comp rx=0 tx=0).stop 2024-01-27T01:06:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.644+0000 7fb032148700 1 -- 172.21.15.46:0/2692098 shutdown_connections 2024-01-27T01:06:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.644+0000 7fb032148700 1 --2- 172.21.15.46:0/2692098 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb00c066f30 0x7fb00c0693e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.644+0000 7fb032148700 1 --2- 172.21.15.46:0/2692098 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb02c0ad380 0x7fb02c0c0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.644+0000 7fb032148700 1 --2- 172.21.15.46:0/2692098 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb02c068b00 0x7fb02c0cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.644+0000 7fb032148700 1 --2- 172.21.15.46:0/2692098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c06b910 0x7fb02c14b0b0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.644+0000 7fb032148700 1 -- 172.21.15.46:0/2692098 >> 172.21.15.46:0/2692098 conn(0x7fb02c0ca100 msgr2=0x7fb02c070b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:36.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.644+0000 7fb032148700 1 -- 172.21.15.46:0/2692098 shutdown_connections 2024-01-27T01:06:36.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.644+0000 7fb032148700 1 -- 172.21.15.46:0/2692098 wait complete. 2024-01-27T01:06:36.657 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-27T01:06:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.793+0000 7fe7628d8700 1 Processor -- start 2024-01-27T01:06:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.793+0000 7fe7628d8700 1 -- start start 2024-01-27T01:06:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.794+0000 7fe7628d8700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe75c0c8da0 0x7fe75c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.794+0000 7fe7628d8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75c0ccc00 con 0x7fe75c0caf30 2024-01-27T01:06:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.794+0000 7fe7628d8700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75c0cc310 con 0x7fe75c140af0 2024-01-27T01:06:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.794+0000 7fe7628d8700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe75c0cbbf0 con 0x7fe75c0c8da0 2024-01-27T01:06:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.794+0000 7fe760e75700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe75c0caf30 0x7fe75c0cb370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54196/0 (socket says 172.21.15.46:54196) 2024-01-27T01:06:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.794+0000 7fe760e75700 1 -- 172.21.15.46:0/2372795228 learned_addr learned my addr 172.21.15.46:0/2372795228 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.794+0000 7fe75bfff700 1 --2- 172.21.15.46:0/2372795228 >> v2:172.21.15.106:3300/0 conn(0x7fe75c0c8da0 0x7fe75c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.794+0000 7fe75affd700 1 -- 172.21.15.46:0/2372795228 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996043636 0 0) 0x7fe75c0ccc00 con 0x7fe75c0caf30 2024-01-27T01:06:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.794+0000 7fe75affd700 1 -- 172.21.15.46:0/2372795228 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe748003570 con 0x7fe75c0caf30 2024-01-27T01:06:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.795+0000 7fe75affd700 1 -- 172.21.15.46:0/2372795228 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263773195 0 0) 0x7fe75c0cc310 con 0x7fe75c140af0 2024-01-27T01:06:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.795+0000 7fe75affd700 1 -- 172.21.15.46:0/2372795228 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe75c0ccc00 con 0x7fe75c140af0 2024-01-27T01:06:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.795+0000 7fe75bfff700 1 -- 172.21.15.46:0/2372795228 >> v1:172.21.15.106:6789/0 conn(0x7fe75c140af0 legacy=0x7fe75c140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.795+0000 7fe75bfff700 1 -- 172.21.15.46:0/2372795228 >> v1:172.21.15.46:6789/0 conn(0x7fe75c0caf30 legacy=0x7fe75c0cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.795+0000 7fe75bfff700 1 -- 172.21.15.46:0/2372795228 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c0cc5b0 con 0x7fe75c0c8da0 2024-01-27T01:06:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.795+0000 7fe75bfff700 1 --2- 172.21.15.46:0/2372795228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe75c0c8da0 0x7fe75c0c9170 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fe74c0098e0 tx=0x7fe74c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3aabcac4dea9b71 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.795+0000 7fe75affd700 1 -- 172.21.15.46:0/2372795228 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c011630 con 0x7fe75c0c8da0 2024-01-27T01:06:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.795+0000 7fe75affd700 1 -- 172.21.15.46:0/2372795228 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74c011790 con 0x7fe75c0c8da0 2024-01-27T01:06:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.795+0000 7fe75affd700 1 -- 172.21.15.46:0/2372795228 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c011b30 con 0x7fe75c0c8da0 2024-01-27T01:06:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.796+0000 7fe7628d8700 1 -- 172.21.15.46:0/2372795228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe75c0c8da0 msgr2=0x7fe75c0c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.796+0000 7fe7628d8700 1 --2- 172.21.15.46:0/2372795228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe75c0c8da0 0x7fe75c0c9170 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fe74c0098e0 tx=0x7fe74c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.796+0000 7fe7628d8700 1 -- 172.21.15.46:0/2372795228 shutdown_connections 2024-01-27T01:06:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.796+0000 7fe7628d8700 1 --2- 172.21.15.46:0/2372795228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe75c0c8da0 0x7fe75c0c9170 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.796+0000 7fe7628d8700 1 -- 172.21.15.46:0/2372795228 >> 172.21.15.46:0/2372795228 conn(0x7fe75c06bd90 msgr2=0x7fe75c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.796+0000 7fe7628d8700 1 -- 172.21.15.46:0/2372795228 shutdown_connections 2024-01-27T01:06:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.796+0000 7fe7628d8700 1 -- 172.21.15.46:0/2372795228 wait complete. 2024-01-27T01:06:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.797+0000 7fe7628d8700 1 Processor -- start 2024-01-27T01:06:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.797+0000 7fe7628d8700 1 -- start start 2024-01-27T01:06:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.798+0000 7fe7628d8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe75c0c8da0 0x7fe75c1571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.798+0000 7fe7628d8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe75c0caf30 0x7fe75c0c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.798+0000 7fe7628d8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe75c140af0 0x7fe75c1585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.798+0000 7fe7628d8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c0b2340 con 0x7fe75c0c8da0 2024-01-27T01:06:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.798+0000 7fe7628d8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c1576e0 con 0x7fe75c0caf30 2024-01-27T01:06:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.798+0000 7fe7628d8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c0c7d40 con 0x7fe75c140af0 2024-01-27T01:06:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.798+0000 7fe75bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe75c0c8da0 0x7fe75c1571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.798+0000 7fe75b7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe75c0caf30 0x7fe75c0c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:36.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.798+0000 7fe75bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe75c0c8da0 0x7fe75c1571a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33860/0 (socket says 172.21.15.46:33860) 2024-01-27T01:06:36.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.798+0000 7fe75bfff700 1 -- 172.21.15.46:0/3222951458 learned_addr learned my addr 172.21.15.46:0/3222951458 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:36.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.798+0000 7fe760e75700 1 --2- 172.21.15.46:0/3222951458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe75c140af0 0x7fe75c1585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:36.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.799+0000 7fe75bfff700 1 -- 172.21.15.46:0/3222951458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe75c140af0 msgr2=0x7fe75c1585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:36.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.799+0000 7fe75bfff700 1 --2- 172.21.15.46:0/3222951458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe75c140af0 0x7fe75c1585c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.799+0000 7fe75bfff700 1 -- 172.21.15.46:0/3222951458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe75c0caf30 msgr2=0x7fe75c0c7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:36.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.799+0000 7fe75bfff700 1 --2- 172.21.15.46:0/3222951458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe75c0caf30 0x7fe75c0c7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:36.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.799+0000 7fe75bfff700 1 -- 172.21.15.46:0/3222951458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c009420 con 0x7fe75c0c8da0 2024-01-27T01:06:36.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.799+0000 7fe75b7fe700 1 --2- 172.21.15.46:0/3222951458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe75c0caf30 0x7fe75c0c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:36.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.799+0000 7fe75bfff700 1 --2- 172.21.15.46:0/3222951458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe75c0c8da0 0x7fe75c1571a0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009ed0 tx=0x7fe74c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:36.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.799+0000 7fe753fff700 1 -- 172.21.15.46:0/3222951458 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c01f070 con 0x7fe75c0c8da0 2024-01-27T01:06:36.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.799+0000 7fe753fff700 1 -- 172.21.15.46:0/3222951458 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74c011900 con 0x7fe75c0c8da0 2024-01-27T01:06:36.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.799+0000 7fe7628d8700 1 -- 172.21.15.46:0/3222951458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75c0c8020 con 0x7fe75c0c8da0 2024-01-27T01:06:36.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.799+0000 7fe753fff700 1 -- 172.21.15.46:0/3222951458 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c023410 con 0x7fe75c0c8da0 2024-01-27T01:06:36.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.799+0000 7fe7628d8700 1 -- 172.21.15.46:0/3222951458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75c0c83b0 con 0x7fe75c0c8da0 2024-01-27T01:06:36.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.800+0000 7fe7628d8700 1 -- 172.21.15.46:0/3222951458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75c0b2790 con 0x7fe75c0c8da0 2024-01-27T01:06:36.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.800+0000 7fe753fff700 1 -- 172.21.15.46:0/3222951458 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe74c02c3b0 con 0x7fe75c0c8da0 2024-01-27T01:06:36.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.801+0000 7fe753fff700 1 --2- 172.21.15.46:0/3222951458 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe73c066f30 0x7fe73c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:36.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.801+0000 7fe753fff700 1 -- 172.21.15.46:0/3222951458 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fe74c0859b0 con 0x7fe75c0c8da0 2024-01-27T01:06:36.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.801+0000 7fe75b7fe700 1 --2- 172.21.15.46:0/3222951458 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe73c066f30 0x7fe73c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:36.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.802+0000 7fe75b7fe700 1 --2- 172.21.15.46:0/3222951458 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe73c066f30 0x7fe73c0693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0c4410 tx=0x7fe744007d50 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:36.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:36.805+0000 7fe753fff700 1 -- 172.21.15.46:0/3222951458 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe75c0b2790 con 0x7fe75c0c8da0 2024-01-27T01:06:37.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.009+0000 7fe7628d8700 1 -- 172.21.15.46:0/3222951458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe75c0b2790 con 0x7fe75c0c8da0 2024-01-27T01:06:37.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.009+0000 7fe753fff700 1 -- 172.21.15.46:0/3222951458 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v223) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe74c015020 con 0x7fe75c0c8da0 2024-01-27T01:06:37.008 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-27T01:06:37.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.011+0000 7fe7628d8700 1 -- 172.21.15.46:0/3222951458 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe73c066f30 msgr2=0x7fe73c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.011+0000 7fe7628d8700 1 --2- 172.21.15.46:0/3222951458 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe73c066f30 0x7fe73c0693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0c4410 tx=0x7fe744007d50 comp rx=0 tx=0).stop 2024-01-27T01:06:37.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.011+0000 7fe7628d8700 1 -- 172.21.15.46:0/3222951458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe75c0c8da0 msgr2=0x7fe75c1571a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.011+0000 7fe7628d8700 1 --2- 172.21.15.46:0/3222951458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe75c0c8da0 0x7fe75c1571a0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009ed0 tx=0x7fe74c017000 comp rx=0 tx=0).stop 2024-01-27T01:06:37.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.011+0000 7fe7628d8700 1 -- 172.21.15.46:0/3222951458 shutdown_connections 2024-01-27T01:06:37.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.011+0000 7fe7628d8700 1 --2- 172.21.15.46:0/3222951458 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe73c066f30 0x7fe73c0693e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.011+0000 7fe7628d8700 1 --2- 172.21.15.46:0/3222951458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe75c0c8da0 0x7fe75c1571a0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.011+0000 7fe7628d8700 1 --2- 172.21.15.46:0/3222951458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe75c140af0 0x7fe75c1585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.011+0000 7fe7628d8700 1 --2- 172.21.15.46:0/3222951458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe75c0caf30 0x7fe75c0c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.011+0000 7fe7628d8700 1 -- 172.21.15.46:0/3222951458 >> 172.21.15.46:0/3222951458 conn(0x7fe75c06bd90 msgr2=0x7fe75c0cb370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:37.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.011+0000 7fe7628d8700 1 -- 172.21.15.46:0/3222951458 shutdown_connections 2024-01-27T01:06:37.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.011+0000 7fe7628d8700 1 -- 172.21.15.46:0/3222951458 wait complete. 2024-01-27T01:06:37.024 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-27T01:06:37.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.159+0000 7f8faa9ae700 1 Processor -- start 2024-01-27T01:06:37.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.159+0000 7f8faa9ae700 1 -- start start 2024-01-27T01:06:37.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.160+0000 7f8faa9ae700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8fa4067af0 0x7f8fa4067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.160+0000 7f8faa9ae700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fa406ebe0 0x7f8fa40ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.160+0000 7f8faa9ae700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa406fdb0 con 0x7f8fa41597e0 2024-01-27T01:06:37.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.160+0000 7f8faa9ae700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa40ade10 con 0x7f8fa4067af0 2024-01-27T01:06:37.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.160+0000 7f8faa9ae700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa406f1a0 con 0x7f8fa406ebe0 2024-01-27T01:06:37.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.160+0000 7f8fa3fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8fa4067af0 0x7f8fa4067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.160+0000 7f8fa8f4b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8fa41597e0 0x7f8fa4159bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54200/0 (socket says 172.21.15.46:54200) 2024-01-27T01:06:37.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.160+0000 7f8fa8f4b700 1 -- 172.21.15.46:0/584401343 learned_addr learned my addr 172.21.15.46:0/584401343 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:37.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.160+0000 7f8fa37fe700 1 --2- 172.21.15.46:0/584401343 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fa406ebe0 0x7f8fa40ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.160+0000 7f8fa3fff700 1 -- 172.21.15.46:0/584401343 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fa406ebe0 msgr2=0x7f8fa40ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:37.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.160+0000 7f8fa3fff700 1 --2- 172.21.15.46:0/584401343 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fa406ebe0 0x7f8fa40ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.160+0000 7f8fa3fff700 1 -- 172.21.15.46:0/584401343 >> v1:172.21.15.46:6789/0 conn(0x7f8fa41597e0 legacy=0x7f8fa4159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.160+0000 7f8fa3fff700 1 -- 172.21.15.46:0/584401343 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa406f480 con 0x7f8fa4067af0 2024-01-27T01:06:37.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.160+0000 7f8fa2ffd700 1 -- 172.21.15.46:0/584401343 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198129812 0 0) 0x7f8fa406fdb0 con 0x7f8fa41597e0 2024-01-27T01:06:37.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.161+0000 7f8fa3fff700 1 --2- 172.21.15.46:0/584401343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa4067af0 0x7f8fa4067ec0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f8f940098e0 tx=0x7f8f940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56af0a39a0d5c0a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:37.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.161+0000 7f8fa2ffd700 1 -- 172.21.15.46:0/584401343 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f94009e80 con 0x7f8fa4067af0 2024-01-27T01:06:37.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.161+0000 7f8fa2ffd700 1 -- 172.21.15.46:0/584401343 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9400b860 con 0x7f8fa4067af0 2024-01-27T01:06:37.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.161+0000 7f8fa2ffd700 1 -- 172.21.15.46:0/584401343 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9400bbe0 con 0x7f8fa4067af0 2024-01-27T01:06:37.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.161+0000 7f8faa9ae700 1 -- 172.21.15.46:0/584401343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa4067af0 msgr2=0x7f8fa4067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.161+0000 7f8faa9ae700 1 --2- 172.21.15.46:0/584401343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa4067af0 0x7f8fa4067ec0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f8f940098e0 tx=0x7f8f940091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.161+0000 7f8faa9ae700 1 -- 172.21.15.46:0/584401343 shutdown_connections 2024-01-27T01:06:37.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.161+0000 7f8faa9ae700 1 --2- 172.21.15.46:0/584401343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa4067af0 0x7f8fa4067ec0 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.161+0000 7f8faa9ae700 1 --2- 172.21.15.46:0/584401343 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fa406ebe0 0x7f8fa40ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.161+0000 7f8faa9ae700 1 -- 172.21.15.46:0/584401343 >> 172.21.15.46:0/584401343 conn(0x7f8fa40cdf80 msgr2=0x7f8fa406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:37.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.161+0000 7f8faa9ae700 1 -- 172.21.15.46:0/584401343 shutdown_connections 2024-01-27T01:06:37.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.161+0000 7f8faa9ae700 1 -- 172.21.15.46:0/584401343 wait complete. 2024-01-27T01:06:37.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.163+0000 7f8faa9ae700 1 Processor -- start 2024-01-27T01:06:37.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.163+0000 7f8faa9ae700 1 -- start start 2024-01-27T01:06:37.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8faa9ae700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fa4067af0 0x7f8fa4140a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8faa9ae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa406ebe0 0x7f8fa416d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8faa9ae700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8fa41597e0 0x7f8fa40b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8faa9ae700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa416ac00 con 0x7f8fa406ebe0 2024-01-27T01:06:37.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8faa9ae700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa416b1a0 con 0x7f8fa41597e0 2024-01-27T01:06:37.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8faa9ae700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa416db20 con 0x7f8fa4067af0 2024-01-27T01:06:37.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8fa37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa406ebe0 0x7f8fa416d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8fa37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa406ebe0 0x7f8fa416d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33882/0 (socket says 172.21.15.46:33882) 2024-01-27T01:06:37.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8fa37fe700 1 -- 172.21.15.46:0/3725122415 learned_addr learned my addr 172.21.15.46:0/3725122415 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:37.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8fa8f4b700 1 --2- 172.21.15.46:0/3725122415 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8fa41597e0 0x7f8fa40b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8fa3fff700 1 --2- 172.21.15.46:0/3725122415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fa4067af0 0x7f8fa4140a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8fa37fe700 1 -- 172.21.15.46:0/3725122415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fa4067af0 msgr2=0x7f8fa4140a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8fa37fe700 1 --2- 172.21.15.46:0/3725122415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fa4067af0 0x7f8fa4140a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8fa37fe700 1 -- 172.21.15.46:0/3725122415 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8fa41597e0 msgr2=0x7f8fa40b17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8fa37fe700 1 --2- 172.21.15.46:0/3725122415 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8fa41597e0 0x7f8fa40b17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8fa37fe700 1 -- 172.21.15.46:0/3725122415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f94009420 con 0x7f8fa406ebe0 2024-01-27T01:06:37.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8fa3fff700 1 --2- 172.21.15.46:0/3725122415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fa4067af0 0x7f8fa4140a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:37.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.164+0000 7f8fa37fe700 1 --2- 172.21.15.46:0/3725122415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa406ebe0 0x7f8fa416d0f0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800d860 tx=0x7f8f98008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:37.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.165+0000 7f8f8bfff700 1 -- 172.21.15.46:0/3725122415 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f980125f0 con 0x7f8fa406ebe0 2024-01-27T01:06:37.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.165+0000 7f8f8bfff700 1 -- 172.21.15.46:0/3725122415 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f98012750 con 0x7f8fa406ebe0 2024-01-27T01:06:37.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.165+0000 7f8faa9ae700 1 -- 172.21.15.46:0/3725122415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa416de00 con 0x7f8fa406ebe0 2024-01-27T01:06:37.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.165+0000 7f8f8bfff700 1 -- 172.21.15.46:0/3725122415 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f98012b10 con 0x7f8fa406ebe0 2024-01-27T01:06:37.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.165+0000 7f8faa9ae700 1 -- 172.21.15.46:0/3725122415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa4140fb0 con 0x7f8fa406ebe0 2024-01-27T01:06:37.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.166+0000 7f8faa9ae700 1 -- 172.21.15.46:0/3725122415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa416b330 con 0x7f8fa406ebe0 2024-01-27T01:06:37.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.166+0000 7f8f8bfff700 1 -- 172.21.15.46:0/3725122415 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8f9801e460 con 0x7f8fa406ebe0 2024-01-27T01:06:37.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.167+0000 7f8f8bfff700 1 --2- 172.21.15.46:0/3725122415 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8f90066f30 0x7f8f900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.167+0000 7f8f8bfff700 1 -- 172.21.15.46:0/3725122415 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f8f98084d90 con 0x7f8fa406ebe0 2024-01-27T01:06:37.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.167+0000 7f8fa3fff700 1 --2- 172.21.15.46:0/3725122415 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8f90066f30 0x7f8f900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.167+0000 7f8fa3fff700 1 --2- 172.21.15.46:0/3725122415 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8f90066f30 0x7f8f900693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4158c40 tx=0x7f8f94017000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:37.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.171+0000 7f8f8bfff700 1 -- 172.21.15.46:0/3725122415 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8f98051cd0 con 0x7f8fa406ebe0 2024-01-27T01:06:37.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.380+0000 7f8faa9ae700 1 -- 172.21.15.46:0/3725122415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8fa404c710 con 0x7f8fa406ebe0 2024-01-27T01:06:37.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.380+0000 7f8f8bfff700 1 -- 172.21.15.46:0/3725122415 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v223) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8f980552f0 con 0x7f8fa406ebe0 2024-01-27T01:06:37.379 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-27T01:06:37.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.382+0000 7f8faa9ae700 1 -- 172.21.15.46:0/3725122415 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8f90066f30 msgr2=0x7f8f900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.382+0000 7f8faa9ae700 1 --2- 172.21.15.46:0/3725122415 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8f90066f30 0x7f8f900693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4158c40 tx=0x7f8f94017000 comp rx=0 tx=0).stop 2024-01-27T01:06:37.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.382+0000 7f8faa9ae700 1 -- 172.21.15.46:0/3725122415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa406ebe0 msgr2=0x7f8fa416d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.382+0000 7f8faa9ae700 1 --2- 172.21.15.46:0/3725122415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa406ebe0 0x7f8fa416d0f0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800d860 tx=0x7f8f98008040 comp rx=0 tx=0).stop 2024-01-27T01:06:37.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.382+0000 7f8faa9ae700 1 -- 172.21.15.46:0/3725122415 shutdown_connections 2024-01-27T01:06:37.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.382+0000 7f8faa9ae700 1 --2- 172.21.15.46:0/3725122415 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8f90066f30 0x7f8f900693e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.382+0000 7f8faa9ae700 1 --2- 172.21.15.46:0/3725122415 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8fa41597e0 0x7f8fa40b17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.382+0000 7f8faa9ae700 1 --2- 172.21.15.46:0/3725122415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8fa4067af0 0x7f8fa4140a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.382+0000 7f8faa9ae700 1 --2- 172.21.15.46:0/3725122415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa406ebe0 0x7f8fa416d0f0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.382+0000 7f8faa9ae700 1 -- 172.21.15.46:0/3725122415 >> 172.21.15.46:0/3725122415 conn(0x7f8fa40cdf80 msgr2=0x7f8fa40c9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:37.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.383+0000 7f8faa9ae700 1 -- 172.21.15.46:0/3725122415 shutdown_connections 2024-01-27T01:06:37.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.383+0000 7f8faa9ae700 1 -- 172.21.15.46:0/3725122415 wait complete. 2024-01-27T01:06:37.395 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-27T01:06:37.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.528+0000 7fb1ed43c700 1 Processor -- start 2024-01-27T01:06:37.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.528+0000 7fb1ed43c700 1 -- start start 2024-01-27T01:06:37.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.529+0000 7fb1ed43c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb1e80cbdf0 0x7fb1e80ae610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.529+0000 7fb1ed43c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e80ae180 con 0x7fb1e80cc1c0 2024-01-27T01:06:37.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.529+0000 7fb1ed43c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e8156c60 con 0x7fb1e80ad070 2024-01-27T01:06:37.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.529+0000 7fb1ed43c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e80aeb50 con 0x7fb1e80cbdf0 2024-01-27T01:06:37.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.529+0000 7fb1e67fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb1e80cc1c0 0x7fb1e80b0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54216/0 (socket says 172.21.15.46:54216) 2024-01-27T01:06:37.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.529+0000 7fb1e67fc700 1 -- 172.21.15.46:0/395870107 learned_addr learned my addr 172.21.15.46:0/395870107 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:37.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.529+0000 7fb1e6ffd700 1 --2- 172.21.15.46:0/395870107 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb1e80cbdf0 0x7fb1e80ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.529+0000 7fb1e5ffb700 1 -- 172.21.15.46:0/395870107 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717634594 0 0) 0x7fb1e80ae180 con 0x7fb1e80cc1c0 2024-01-27T01:06:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.529+0000 7fb1e5ffb700 1 -- 172.21.15.46:0/395870107 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d4003570 con 0x7fb1e80cc1c0 2024-01-27T01:06:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.529+0000 7fb1e5ffb700 1 -- 172.21.15.46:0/395870107 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114106688 0 0) 0x7fb1e8156c60 con 0x7fb1e80ad070 2024-01-27T01:06:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.529+0000 7fb1e5ffb700 1 -- 172.21.15.46:0/395870107 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e80ae180 con 0x7fb1e80ad070 2024-01-27T01:06:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.529+0000 7fb1e6ffd700 1 -- 172.21.15.46:0/395870107 >> v1:172.21.15.106:6789/0 conn(0x7fb1e80ad070 legacy=0x7fb1e80c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.530+0000 7fb1e6ffd700 1 -- 172.21.15.46:0/395870107 >> v1:172.21.15.46:6789/0 conn(0x7fb1e80cc1c0 legacy=0x7fb1e80b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.530+0000 7fb1e6ffd700 1 -- 172.21.15.46:0/395870107 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e80ad570 con 0x7fb1e80cbdf0 2024-01-27T01:06:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.530+0000 7fb1e6ffd700 1 --2- 172.21.15.46:0/395870107 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb1e80cbdf0 0x7fb1e80ae610 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009910 tx=0x7fb1d80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d94662e7d30bb87 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.530+0000 7fb1e5ffb700 1 -- 172.21.15.46:0/395870107 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8011630 con 0x7fb1e80cbdf0 2024-01-27T01:06:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.530+0000 7fb1e5ffb700 1 -- 172.21.15.46:0/395870107 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d8011790 con 0x7fb1e80cbdf0 2024-01-27T01:06:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.530+0000 7fb1e5ffb700 1 -- 172.21.15.46:0/395870107 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8011b30 con 0x7fb1e80cbdf0 2024-01-27T01:06:37.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.530+0000 7fb1ed43c700 1 -- 172.21.15.46:0/395870107 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb1e80cbdf0 msgr2=0x7fb1e80ae610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.530+0000 7fb1ed43c700 1 --2- 172.21.15.46:0/395870107 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb1e80cbdf0 0x7fb1e80ae610 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009910 tx=0x7fb1d80091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.531+0000 7fb1ed43c700 1 -- 172.21.15.46:0/395870107 shutdown_connections 2024-01-27T01:06:37.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.531+0000 7fb1ed43c700 1 --2- 172.21.15.46:0/395870107 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb1e80cbdf0 0x7fb1e80ae610 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.531+0000 7fb1ed43c700 1 -- 172.21.15.46:0/395870107 >> 172.21.15.46:0/395870107 conn(0x7fb1e8068400 msgr2=0x7fb1e80cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:37.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.531+0000 7fb1ed43c700 1 -- 172.21.15.46:0/395870107 shutdown_connections 2024-01-27T01:06:37.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.531+0000 7fb1ed43c700 1 -- 172.21.15.46:0/395870107 wait complete. 2024-01-27T01:06:37.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.532+0000 7fb1ed43c700 1 Processor -- start 2024-01-27T01:06:37.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.532+0000 7fb1ed43c700 1 -- start start 2024-01-27T01:06:37.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1ed43c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1e80ad070 0x7fb1e80aae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1ed43c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb1e80cbdf0 0x7fb1e814c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1ed43c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1e80a5430 0x7fb1e814cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1ed43c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e814a180 con 0x7fb1e80ad070 2024-01-27T01:06:37.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1ed43c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e814c0e0 con 0x7fb1e80a5430 2024-01-27T01:06:37.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1ed43c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e80a5930 con 0x7fb1e80cbdf0 2024-01-27T01:06:37.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1e6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1e80ad070 0x7fb1e80aae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1e6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1e80ad070 0x7fb1e80aae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33894/0 (socket says 172.21.15.46:33894) 2024-01-27T01:06:37.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1e6ffd700 1 -- 172.21.15.46:0/458456290 learned_addr learned my addr 172.21.15.46:0/458456290 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:37.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1e67fc700 1 --2- 172.21.15.46:0/458456290 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb1e80cbdf0 0x7fb1e814c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1e77fe700 1 --2- 172.21.15.46:0/458456290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1e80a5430 0x7fb1e814cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1e6ffd700 1 -- 172.21.15.46:0/458456290 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb1e80cbdf0 msgr2=0x7fb1e814c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1e6ffd700 1 --2- 172.21.15.46:0/458456290 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb1e80cbdf0 0x7fb1e814c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1e6ffd700 1 -- 172.21.15.46:0/458456290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1e80a5430 msgr2=0x7fb1e814cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1e6ffd700 1 --2- 172.21.15.46:0/458456290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1e80a5430 0x7fb1e814cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1e6ffd700 1 -- 172.21.15.46:0/458456290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d8009420 con 0x7fb1e80ad070 2024-01-27T01:06:37.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1e77fe700 1 --2- 172.21.15.46:0/458456290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1e80a5430 0x7fb1e814cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:06:37.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1e67fc700 1 --2- 172.21.15.46:0/458456290 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb1e80cbdf0 0x7fb1e814c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:06:37.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.533+0000 7fb1e6ffd700 1 --2- 172.21.15.46:0/458456290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1e80ad070 0x7fb1e80aae20 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009f00 tx=0x7fb1d8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:37.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.534+0000 7fb1deffd700 1 -- 172.21.15.46:0/458456290 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d801f070 con 0x7fb1e80ad070 2024-01-27T01:06:37.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.534+0000 7fb1deffd700 1 -- 172.21.15.46:0/458456290 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d8000bc0 con 0x7fb1e80ad070 2024-01-27T01:06:37.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.534+0000 7fb1deffd700 1 -- 172.21.15.46:0/458456290 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8025420 con 0x7fb1e80ad070 2024-01-27T01:06:37.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.534+0000 7fb1ed43c700 1 -- 172.21.15.46:0/458456290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e814ba90 con 0x7fb1e80ad070 2024-01-27T01:06:37.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.534+0000 7fb1ed43c700 1 -- 172.21.15.46:0/458456290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e814b760 con 0x7fb1e80ad070 2024-01-27T01:06:37.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.535+0000 7fb1ed43c700 1 -- 172.21.15.46:0/458456290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e814bf30 con 0x7fb1e80ad070 2024-01-27T01:06:37.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.535+0000 7fb1deffd700 1 -- 172.21.15.46:0/458456290 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb1d8023070 con 0x7fb1e80ad070 2024-01-27T01:06:37.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.536+0000 7fb1deffd700 1 --2- 172.21.15.46:0/458456290 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb1c8066f30 0x7fb1c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.536+0000 7fb1deffd700 1 -- 172.21.15.46:0/458456290 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fb1d8085bd0 con 0x7fb1e80ad070 2024-01-27T01:06:37.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.536+0000 7fb1e67fc700 1 --2- 172.21.15.46:0/458456290 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb1c8066f30 0x7fb1c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.536+0000 7fb1e67fc700 1 --2- 172.21.15.46:0/458456290 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb1c8066f30 0x7fb1c80693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0007fd0 tx=0x7fb1d0009040 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:37.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.540+0000 7fb1deffd700 1 -- 172.21.15.46:0/458456290 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb1d8052b10 con 0x7fb1e80ad070 2024-01-27T01:06:37.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.747+0000 7fb1ed43c700 1 -- 172.21.15.46:0/458456290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb1e814c3c0 con 0x7fb1e80ad070 2024-01-27T01:06:37.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.747+0000 7fb1deffd700 1 -- 172.21.15.46:0/458456290 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v223) v1 ==== 84+0+209 (secure 0 0 0) 0x7fb1d802a090 con 0x7fb1e80ad070 2024-01-27T01:06:37.747 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:37.747 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-27T01:06:37.747 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:37.747 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-01-27T01:06:37.747 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-01-27T01:06:37.747 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:37.747 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:37.747 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 27, 2024-01-27T01:06:37.747 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": "cephfs_metadata" 2024-01-27T01:06:37.747 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:37.747 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:37.747 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 28, 2024-01-27T01:06:37.747 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": "cephfs_data" 2024-01-27T01:06:37.748 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:37.748 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-27T01:06:37.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.749+0000 7fb1ed43c700 1 -- 172.21.15.46:0/458456290 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb1c8066f30 msgr2=0x7fb1c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.749+0000 7fb1ed43c700 1 --2- 172.21.15.46:0/458456290 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb1c8066f30 0x7fb1c80693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0007fd0 tx=0x7fb1d0009040 comp rx=0 tx=0).stop 2024-01-27T01:06:37.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.749+0000 7fb1ed43c700 1 -- 172.21.15.46:0/458456290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1e80ad070 msgr2=0x7fb1e80aae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.749+0000 7fb1ed43c700 1 --2- 172.21.15.46:0/458456290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1e80ad070 0x7fb1e80aae20 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009f00 tx=0x7fb1d8017000 comp rx=0 tx=0).stop 2024-01-27T01:06:37.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.749+0000 7fb1ed43c700 1 -- 172.21.15.46:0/458456290 shutdown_connections 2024-01-27T01:06:37.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.749+0000 7fb1ed43c700 1 --2- 172.21.15.46:0/458456290 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb1c8066f30 0x7fb1c80693e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.749+0000 7fb1ed43c700 1 --2- 172.21.15.46:0/458456290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1e80ad070 0x7fb1e80aae20 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.749+0000 7fb1ed43c700 1 --2- 172.21.15.46:0/458456290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1e80a5430 0x7fb1e814cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.749+0000 7fb1ed43c700 1 --2- 172.21.15.46:0/458456290 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb1e80cbdf0 0x7fb1e814c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.749+0000 7fb1ed43c700 1 -- 172.21.15.46:0/458456290 >> 172.21.15.46:0/458456290 conn(0x7fb1e8068400 msgr2=0x7fb1e80c8810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:37.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.749+0000 7fb1ed43c700 1 -- 172.21.15.46:0/458456290 shutdown_connections 2024-01-27T01:06:37.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.750+0000 7fb1ed43c700 1 -- 172.21.15.46:0/458456290 wait complete. 2024-01-27T01:06:37.762 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-01-27T01:06:37.762 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-01-27T01:06:37.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.901+0000 7f82496cd700 1 Processor -- start 2024-01-27T01:06:37.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.901+0000 7f82496cd700 1 -- start start 2024-01-27T01:06:37.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.902+0000 7f82496cd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f82440ccb60 0x7f82440b3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.902+0000 7f82496cd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82440b0bc0 0x7f82440b1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.902+0000 7f82496cd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f82440b1550 0x7f82440ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.902+0000 7f82496cd700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82440aff00 con 0x7f82440b1550 2024-01-27T01:06:37.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.902+0000 7f82496cd700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82440b3540 con 0x7f82440ccb60 2024-01-27T01:06:37.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.902+0000 7f82496cd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82440b19d0 con 0x7f82440b0bc0 2024-01-27T01:06:37.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.903+0000 7f82437fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f82440b1550 0x7f82440ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.903+0000 7f82437fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f82440b1550 0x7f82440ae0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33904/0 (socket says 172.21.15.46:33904) 2024-01-27T01:06:37.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.903+0000 7f82437fe700 1 -- 172.21.15.46:0/3379133726 learned_addr learned my addr 172.21.15.46:0/3379133726 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:37.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.903+0000 7f8242ffd700 1 --2- 172.21.15.46:0/3379133726 >> v2:172.21.15.106:3300/0 conn(0x7f82440ccb60 0x7f82440b3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.903+0000 7f82427fc700 1 --2- 172.21.15.46:0/3379133726 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82440b0bc0 0x7f82440b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.903+0000 7f82437fe700 1 -- 172.21.15.46:0/3379133726 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82440b0bc0 msgr2=0x7f82440b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:37.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.903+0000 7f82437fe700 1 --2- 172.21.15.46:0/3379133726 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82440b0bc0 0x7f82440b1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.903+0000 7f82437fe700 1 -- 172.21.15.46:0/3379133726 >> v2:172.21.15.106:3300/0 conn(0x7f82440ccb60 msgr2=0x7f82440b3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:37.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.903+0000 7f82437fe700 1 --2- 172.21.15.46:0/3379133726 >> v2:172.21.15.106:3300/0 conn(0x7f82440ccb60 0x7f82440b3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.903+0000 7f82437fe700 1 -- 172.21.15.46:0/3379133726 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82440af2f0 con 0x7f82440b1550 2024-01-27T01:06:37.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.903+0000 7f82437fe700 1 --2- 172.21.15.46:0/3379133726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82440b1550 0x7f82440ae0d0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f824406dec0 tx=0x7f823c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa8b124276b5ca6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:37.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.903+0000 7f8241ffb700 1 -- 172.21.15.46:0/3379133726 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c00e070 con 0x7f82440b1550 2024-01-27T01:06:37.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.903+0000 7f8241ffb700 1 -- 172.21.15.46:0/3379133726 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823c00b970 con 0x7f82440b1550 2024-01-27T01:06:37.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.903+0000 7f8241ffb700 1 -- 172.21.15.46:0/3379133726 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c0092e0 con 0x7f82440b1550 2024-01-27T01:06:37.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.904+0000 7f82496cd700 1 -- 172.21.15.46:0/3379133726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82440b1550 msgr2=0x7f82440ae0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.904+0000 7f82496cd700 1 --2- 172.21.15.46:0/3379133726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82440b1550 0x7f82440ae0d0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f824406dec0 tx=0x7f823c009840 comp rx=0 tx=0).stop 2024-01-27T01:06:37.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.904+0000 7f82496cd700 1 -- 172.21.15.46:0/3379133726 shutdown_connections 2024-01-27T01:06:37.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.904+0000 7f82496cd700 1 --2- 172.21.15.46:0/3379133726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82440b1550 0x7f82440ae0d0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.904+0000 7f82496cd700 1 --2- 172.21.15.46:0/3379133726 >> v2:172.21.15.106:3300/0 conn(0x7f82440ccb60 0x7f82440b3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.904+0000 7f82496cd700 1 --2- 172.21.15.46:0/3379133726 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82440b0bc0 0x7f82440b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.904+0000 7f82496cd700 1 -- 172.21.15.46:0/3379133726 >> 172.21.15.46:0/3379133726 conn(0x7f8244069220 msgr2=0x7f82440cd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:37.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.904+0000 7f82496cd700 1 -- 172.21.15.46:0/3379133726 shutdown_connections 2024-01-27T01:06:37.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.904+0000 7f82496cd700 1 -- 172.21.15.46:0/3379133726 wait complete. 2024-01-27T01:06:37.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.906+0000 7f82496cd700 1 Processor -- start 2024-01-27T01:06:37.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.906+0000 7f82496cd700 1 -- start start 2024-01-27T01:06:37.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.906+0000 7f82496cd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82440b0bc0 0x7f82440c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.906+0000 7f82496cd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82440b1550 0x7f82440c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.906+0000 7f82496cd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82440ccb60 0x7f82440c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.906+0000 7f82496cd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8244159750 con 0x7f82440ccb60 2024-01-27T01:06:37.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.906+0000 7f82496cd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82440ca7d0 con 0x7f82440b1550 2024-01-27T01:06:37.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.906+0000 7f82496cd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82440c9bc0 con 0x7f82440b0bc0 2024-01-27T01:06:37.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.907+0000 7f82437fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82440ccb60 0x7f82440c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.907+0000 7f8242ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82440b0bc0 0x7f82440c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.907+0000 7f82437fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82440ccb60 0x7f82440c7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33914/0 (socket says 172.21.15.46:33914) 2024-01-27T01:06:37.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.907+0000 7f82437fe700 1 -- 172.21.15.46:0/1566569034 learned_addr learned my addr 172.21.15.46:0/1566569034 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:37.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.907+0000 7f82427fc700 1 --2- 172.21.15.46:0/1566569034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82440b1550 0x7f82440c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.907+0000 7f82437fe700 1 -- 172.21.15.46:0/1566569034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82440b0bc0 msgr2=0x7f82440c6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.907+0000 7f82437fe700 1 --2- 172.21.15.46:0/1566569034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82440b0bc0 0x7f82440c6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.907+0000 7f82437fe700 1 -- 172.21.15.46:0/1566569034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82440b1550 msgr2=0x7f82440c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:37.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.907+0000 7f82437fe700 1 --2- 172.21.15.46:0/1566569034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82440b1550 0x7f82440c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:37.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.907+0000 7f82437fe700 1 -- 172.21.15.46:0/1566569034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c009ae0 con 0x7f82440ccb60 2024-01-27T01:06:37.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.907+0000 7f82437fe700 1 --2- 172.21.15.46:0/1566569034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82440ccb60 0x7f82440c7cf0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f824406df70 tx=0x7f823c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:37.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.907+0000 7f822affd700 1 -- 172.21.15.46:0/1566569034 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c008ad0 con 0x7f82440ccb60 2024-01-27T01:06:37.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.907+0000 7f822affd700 1 -- 172.21.15.46:0/1566569034 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823c008c50 con 0x7f82440ccb60 2024-01-27T01:06:37.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.907+0000 7f822affd700 1 -- 172.21.15.46:0/1566569034 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c025460 con 0x7f82440ccb60 2024-01-27T01:06:37.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.907+0000 7f82496cd700 1 -- 172.21.15.46:0/1566569034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82440c9ea0 con 0x7f82440ccb60 2024-01-27T01:06:37.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.908+0000 7f82496cd700 1 -- 172.21.15.46:0/1566569034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82440ca460 con 0x7f82440ccb60 2024-01-27T01:06:37.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.909+0000 7f82496cd700 1 -- 172.21.15.46:0/1566569034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82440ca980 con 0x7f82440ccb60 2024-01-27T01:06:37.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.909+0000 7f822affd700 1 -- 172.21.15.46:0/1566569034 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f823c003040 con 0x7f82440ccb60 2024-01-27T01:06:37.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.910+0000 7f822affd700 1 --2- 172.21.15.46:0/1566569034 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8230066f30 0x7f82300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:37.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.910+0000 7f822affd700 1 -- 172.21.15.46:0/1566569034 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f823c0861f0 con 0x7f82440ccb60 2024-01-27T01:06:37.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.910+0000 7f8242ffd700 1 --2- 172.21.15.46:0/1566569034 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8230066f30 0x7f82300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:37.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.910+0000 7f8242ffd700 1 --2- 172.21.15.46:0/1566569034 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8230066f30 0x7f82300693e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8234009da0 tx=0x7f8234009120 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:37.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:37.914+0000 7f822affd700 1 -- 172.21.15.46:0/1566569034 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f823c053130 con 0x7f82440ccb60 2024-01-27T01:06:38.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:38.120+0000 7f82496cd700 1 -- 172.21.15.46:0/1566569034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f824404c710 con 0x7f82440ccb60 2024-01-27T01:06:38.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:38.909+0000 7f822affd700 1 -- 172.21.15.46:0/1566569034 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v224) v1 ==== 144+0+0 (secure 0 0 0) 0x7f823c056750 con 0x7f82440ccb60 2024-01-27T01:06:38.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:38.915+0000 7f82496cd700 1 -- 172.21.15.46:0/1566569034 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8230066f30 msgr2=0x7f82300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:38.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:38.915+0000 7f82496cd700 1 --2- 172.21.15.46:0/1566569034 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8230066f30 0x7f82300693e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8234009da0 tx=0x7f8234009120 comp rx=0 tx=0).stop 2024-01-27T01:06:38.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:38.915+0000 7f82496cd700 1 -- 172.21.15.46:0/1566569034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82440ccb60 msgr2=0x7f82440c7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:38.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:38.915+0000 7f82496cd700 1 --2- 172.21.15.46:0/1566569034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82440ccb60 0x7f82440c7cf0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f824406df70 tx=0x7f823c018000 comp rx=0 tx=0).stop 2024-01-27T01:06:38.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:38.916+0000 7f82496cd700 1 -- 172.21.15.46:0/1566569034 shutdown_connections 2024-01-27T01:06:38.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:38.916+0000 7f82496cd700 1 --2- 172.21.15.46:0/1566569034 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8230066f30 0x7f82300693e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:38.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:38.916+0000 7f82496cd700 1 --2- 172.21.15.46:0/1566569034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82440ccb60 0x7f82440c7cf0 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:38.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:38.916+0000 7f82496cd700 1 --2- 172.21.15.46:0/1566569034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82440b0bc0 0x7f82440c6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:38.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:38.916+0000 7f82496cd700 1 --2- 172.21.15.46:0/1566569034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82440b1550 0x7f82440c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:38.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:38.916+0000 7f82496cd700 1 -- 172.21.15.46:0/1566569034 >> 172.21.15.46:0/1566569034 conn(0x7f8244069220 msgr2=0x7f82440974f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:38.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:38.916+0000 7f82496cd700 1 -- 172.21.15.46:0/1566569034 shutdown_connections 2024-01-27T01:06:38.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:38.917+0000 7f82496cd700 1 -- 172.21.15.46:0/1566569034 wait complete. 2024-01-27T01:06:38.917 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'backup_fs_metadata' created 2024-01-27T01:06:38.931 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-27T01:06:39.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.074+0000 7f8630f5a700 1 Processor -- start 2024-01-27T01:06:39.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.074+0000 7f8630f5a700 1 -- start start 2024-01-27T01:06:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.075+0000 7f8630f5a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f862c157490 0x7f862c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.075+0000 7f8630f5a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f862c157860 0x7f862c0cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.075+0000 7f8630f5a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f862c06d6a0 0x7f862c06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.075+0000 7f8630f5a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f862c0cdc60 con 0x7f862c157860 2024-01-27T01:06:39.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.075+0000 7f8630f5a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f862c06e040 con 0x7f862c157490 2024-01-27T01:06:39.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.075+0000 7f8630f5a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f862c0cea80 con 0x7f862c06d6a0 2024-01-27T01:06:39.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.076+0000 7f8629d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f862c157860 0x7f862c0cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:39.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.076+0000 7f8629d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f862c157860 0x7f862c0cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33918/0 (socket says 172.21.15.46:33918) 2024-01-27T01:06:39.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.076+0000 7f8629d9b700 1 -- 172.21.15.46:0/527101388 learned_addr learned my addr 172.21.15.46:0/527101388 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:39.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.076+0000 7f862a59c700 1 --2- 172.21.15.46:0/527101388 >> v2:172.21.15.106:3300/0 conn(0x7f862c157490 0x7f862c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:39.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.076+0000 7f862ad9d700 1 --2- 172.21.15.46:0/527101388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f862c06d6a0 0x7f862c06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:39.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.076+0000 7f8629d9b700 1 -- 172.21.15.46:0/527101388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f862c06d6a0 msgr2=0x7f862c06db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:39.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.076+0000 7f8629d9b700 1 --2- 172.21.15.46:0/527101388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f862c06d6a0 0x7f862c06db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:39.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.076+0000 7f8629d9b700 1 -- 172.21.15.46:0/527101388 >> v2:172.21.15.106:3300/0 conn(0x7f862c157490 msgr2=0x7f862c0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:39.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.076+0000 7f8629d9b700 1 --2- 172.21.15.46:0/527101388 >> v2:172.21.15.106:3300/0 conn(0x7f862c157490 0x7f862c0cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:39.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.076+0000 7f8629d9b700 1 -- 172.21.15.46:0/527101388 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c0cde70 con 0x7f862c157860 2024-01-27T01:06:39.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.076+0000 7f862a59c700 1 --2- 172.21.15.46:0/527101388 >> v2:172.21.15.106:3300/0 conn(0x7f862c157490 0x7f862c0cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:39.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.076+0000 7f862ad9d700 1 --2- 172.21.15.46:0/527101388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f862c06d6a0 0x7f862c06db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:39.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.076+0000 7f8629d9b700 1 --2- 172.21.15.46:0/527101388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f862c157860 0x7f862c0cd620 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f862c06f9d0 tx=0x7f861c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45a1f6c6821a36e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:39.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.076+0000 7f862959a700 1 -- 172.21.15.46:0/527101388 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c00b860 con 0x7f862c157860 2024-01-27T01:06:39.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.076+0000 7f862959a700 1 -- 172.21.15.46:0/527101388 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861c00b9c0 con 0x7f862c157860 2024-01-27T01:06:39.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.076+0000 7f862959a700 1 -- 172.21.15.46:0/527101388 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c00bd60 con 0x7f862c157860 2024-01-27T01:06:39.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.077+0000 7f8630f5a700 1 -- 172.21.15.46:0/527101388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f862c157860 msgr2=0x7f862c0cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:39.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.077+0000 7f8630f5a700 1 --2- 172.21.15.46:0/527101388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f862c157860 0x7f862c0cd620 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f862c06f9d0 tx=0x7f861c009250 comp rx=0 tx=0).stop 2024-01-27T01:06:39.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.077+0000 7f8630f5a700 1 -- 172.21.15.46:0/527101388 shutdown_connections 2024-01-27T01:06:39.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.077+0000 7f8630f5a700 1 --2- 172.21.15.46:0/527101388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f862c06d6a0 0x7f862c06db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:39.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.077+0000 7f8630f5a700 1 --2- 172.21.15.46:0/527101388 >> v2:172.21.15.106:3300/0 conn(0x7f862c157490 0x7f862c0cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:39.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.077+0000 7f8630f5a700 1 --2- 172.21.15.46:0/527101388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f862c157860 0x7f862c0cd620 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:39.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.077+0000 7f8630f5a700 1 -- 172.21.15.46:0/527101388 >> 172.21.15.46:0/527101388 conn(0x7f862c067930 msgr2=0x7f862c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:39.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.077+0000 7f8630f5a700 1 -- 172.21.15.46:0/527101388 shutdown_connections 2024-01-27T01:06:39.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.077+0000 7f8630f5a700 1 -- 172.21.15.46:0/527101388 wait complete. 2024-01-27T01:06:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.079+0000 7f8630f5a700 1 Processor -- start 2024-01-27T01:06:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.079+0000 7f8630f5a700 1 -- start start 2024-01-27T01:06:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.080+0000 7f8630f5a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f862c06d6a0 0x7f862c0af980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.080+0000 7f8630f5a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f862c157490 0x7f862c157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.080+0000 7f8630f5a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f862c157860 0x7f862c150eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.080+0000 7f8630f5a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c158ce0 con 0x7f862c157490 2024-01-27T01:06:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.080+0000 7f8630f5a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c158720 con 0x7f862c157860 2024-01-27T01:06:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.080+0000 7f8630f5a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f862c158a00 con 0x7f862c06d6a0 2024-01-27T01:06:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.080+0000 7f8629d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f862c157490 0x7f862c157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.080+0000 7f8629d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f862c157490 0x7f862c157c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33920/0 (socket says 172.21.15.46:33920) 2024-01-27T01:06:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.080+0000 7f8629d9b700 1 -- 172.21.15.46:0/1664534748 learned_addr learned my addr 172.21.15.46:0/1664534748 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.080+0000 7f862ad9d700 1 --2- 172.21.15.46:0/1664534748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f862c157860 0x7f862c150eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.080+0000 7f862a59c700 1 --2- 172.21.15.46:0/1664534748 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f862c06d6a0 0x7f862c0af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.080+0000 7f8629d9b700 1 -- 172.21.15.46:0/1664534748 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f862c06d6a0 msgr2=0x7f862c0af980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.080+0000 7f8629d9b700 1 --2- 172.21.15.46:0/1664534748 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f862c06d6a0 0x7f862c0af980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.080+0000 7f8629d9b700 1 -- 172.21.15.46:0/1664534748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f862c157860 msgr2=0x7f862c150eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.080+0000 7f8629d9b700 1 --2- 172.21.15.46:0/1664534748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f862c157860 0x7f862c150eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.080+0000 7f8629d9b700 1 -- 172.21.15.46:0/1664534748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c0094f0 con 0x7f862c157490 2024-01-27T01:06:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.081+0000 7f8629d9b700 1 --2- 172.21.15.46:0/1664534748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f862c157490 0x7f862c157c60 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f861c009290 tx=0x7f861c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.081+0000 7f86227fc700 1 -- 172.21.15.46:0/1664534748 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c011e40 con 0x7f862c157490 2024-01-27T01:06:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.081+0000 7f86227fc700 1 -- 172.21.15.46:0/1664534748 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861c01f070 con 0x7f862c157490 2024-01-27T01:06:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.081+0000 7f8630f5a700 1 -- 172.21.15.46:0/1664534748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f862c1592d0 con 0x7f862c157490 2024-01-27T01:06:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.081+0000 7f86227fc700 1 -- 172.21.15.46:0/1664534748 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c0266d0 con 0x7f862c157490 2024-01-27T01:06:39.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.081+0000 7f8630f5a700 1 -- 172.21.15.46:0/1664534748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f862c0c3520 con 0x7f862c157490 2024-01-27T01:06:39.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.082+0000 7f8630f5a700 1 -- 172.21.15.46:0/1664534748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f862c1597f0 con 0x7f862c157490 2024-01-27T01:06:39.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.082+0000 7f86227fc700 1 -- 172.21.15.46:0/1664534748 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f861c026a40 con 0x7f862c157490 2024-01-27T01:06:39.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.083+0000 7f86227fc700 1 --2- 172.21.15.46:0/1664534748 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f860c066f30 0x7f860c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:39.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.083+0000 7f86227fc700 1 -- 172.21.15.46:0/1664534748 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 6890+0+0 (secure 0 0 0) 0x7f861c021070 con 0x7f862c157490 2024-01-27T01:06:39.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.083+0000 7f862a59c700 1 --2- 172.21.15.46:0/1664534748 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f860c066f30 0x7f860c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:39.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.084+0000 7f862a59c700 1 --2- 172.21.15.46:0/1664534748 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f860c066f30 0x7f860c0693e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8614006fd0 tx=0x7f8614008040 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:39.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.088+0000 7f86227fc700 1 -- 172.21.15.46:0/1664534748 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f862c1597f0 con 0x7f862c157490 2024-01-27T01:06:39.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.295+0000 7f8630f5a700 1 -- 172.21.15.46:0/1664534748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f862c1597f0 con 0x7f862c157490 2024-01-27T01:06:39.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.918+0000 7f86227fc700 1 -- 172.21.15.46:0/1664534748 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v225) v1 ==== 176+0+0 (secure 0 0 0) 0x7f862c1597f0 con 0x7f862c157490 2024-01-27T01:06:39.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.924+0000 7f8630f5a700 1 -- 172.21.15.46:0/1664534748 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f860c066f30 msgr2=0x7f860c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:39.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.924+0000 7f8630f5a700 1 --2- 172.21.15.46:0/1664534748 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f860c066f30 0x7f860c0693e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8614006fd0 tx=0x7f8614008040 comp rx=0 tx=0).stop 2024-01-27T01:06:39.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.924+0000 7f8630f5a700 1 -- 172.21.15.46:0/1664534748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f862c157490 msgr2=0x7f862c157c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:39.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.924+0000 7f8630f5a700 1 --2- 172.21.15.46:0/1664534748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f862c157490 0x7f862c157c60 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f861c009290 tx=0x7f861c017000 comp rx=0 tx=0).stop 2024-01-27T01:06:39.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.926+0000 7f8630f5a700 1 -- 172.21.15.46:0/1664534748 shutdown_connections 2024-01-27T01:06:39.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.926+0000 7f8630f5a700 1 --2- 172.21.15.46:0/1664534748 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f860c066f30 0x7f860c0693e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:39.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.926+0000 7f8630f5a700 1 --2- 172.21.15.46:0/1664534748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f862c157860 0x7f862c150eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:39.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.926+0000 7f8630f5a700 1 --2- 172.21.15.46:0/1664534748 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f862c06d6a0 0x7f862c0af980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:39.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.926+0000 7f8630f5a700 1 --2- 172.21.15.46:0/1664534748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f862c157490 0x7f862c157c60 secure :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f861c009290 tx=0x7f861c017000 comp rx=0 tx=0).stop 2024-01-27T01:06:39.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.926+0000 7f8630f5a700 1 -- 172.21.15.46:0/1664534748 >> 172.21.15.46:0/1664534748 conn(0x7f862c067930 msgr2=0x7f862c1530d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:39.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.928+0000 7f8630f5a700 1 -- 172.21.15.46:0/1664534748 shutdown_connections 2024-01-27T01:06:39.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:39.928+0000 7f8630f5a700 1 -- 172.21.15.46:0/1664534748 wait complete. 2024-01-27T01:06:39.934 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'backup_fs_data' created 2024-01-27T01:06:39.950 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-01-27T01:06:40.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.088+0000 7f0c79b67700 1 Processor -- start 2024-01-27T01:06:40.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.088+0000 7f0c79b67700 1 -- start start 2024-01-27T01:06:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.089+0000 7f0c79b67700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0c74157490 0x7f0c740cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.089+0000 7f0c79b67700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c7406ddb0 0x7f0c740cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.089+0000 7f0c79b67700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c740cea80 con 0x7f0c74157860 2024-01-27T01:06:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.089+0000 7f0c79b67700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c740cdc60 con 0x7f0c74157490 2024-01-27T01:06:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.089+0000 7f0c79b67700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c740b2ae0 con 0x7f0c7406ddb0 2024-01-27T01:06:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.089+0000 7f0c73fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c7406ddb0 0x7f0c740cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.089+0000 7f0c73fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c7406ddb0 0x7f0c740cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:55080/0 (socket says 172.21.15.46:55080) 2024-01-27T01:06:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.089+0000 7f0c73fff700 1 -- 172.21.15.46:0/752879156 learned_addr learned my addr 172.21.15.46:0/752879156 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.089+0000 7f0c737fe700 1 --2- 172.21.15.46:0/752879156 >> v2:172.21.15.106:3300/0 conn(0x7f0c74157490 0x7f0c740cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.090+0000 7f0c73fff700 1 -- 172.21.15.46:0/752879156 >> v2:172.21.15.106:3300/0 conn(0x7f0c74157490 msgr2=0x7f0c740cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.090+0000 7f0c73fff700 1 --2- 172.21.15.46:0/752879156 >> v2:172.21.15.106:3300/0 conn(0x7f0c74157490 0x7f0c740cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.090+0000 7f0c73fff700 1 -- 172.21.15.46:0/752879156 >> v1:172.21.15.106:6789/0 conn(0x7f0c74157860 legacy=0x7f0c7406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.090+0000 7f0c73fff700 1 -- 172.21.15.46:0/752879156 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c740cde70 con 0x7f0c7406ddb0 2024-01-27T01:06:40.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.090+0000 7f0c73fff700 1 --2- 172.21.15.46:0/752879156 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c7406ddb0 0x7f0c740cd620 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c009f20 tx=0x7f0c6c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bccfb964dffcfb5 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:40.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.090+0000 7f0c727fc700 1 -- 172.21.15.46:0/752879156 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c00e070 con 0x7f0c7406ddb0 2024-01-27T01:06:40.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.090+0000 7f0c727fc700 1 -- 172.21.15.46:0/752879156 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6c00b9d0 con 0x7f0c7406ddb0 2024-01-27T01:06:40.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.090+0000 7f0c727fc700 1 -- 172.21.15.46:0/752879156 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c0092e0 con 0x7f0c7406ddb0 2024-01-27T01:06:40.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.090+0000 7f0c79b67700 1 -- 172.21.15.46:0/752879156 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c7406ddb0 msgr2=0x7f0c740cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:40.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.090+0000 7f0c79b67700 1 --2- 172.21.15.46:0/752879156 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c7406ddb0 0x7f0c740cd620 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c009f20 tx=0x7f0c6c009840 comp rx=0 tx=0).stop 2024-01-27T01:06:40.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.091+0000 7f0c79b67700 1 -- 172.21.15.46:0/752879156 shutdown_connections 2024-01-27T01:06:40.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.091+0000 7f0c79b67700 1 --2- 172.21.15.46:0/752879156 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c7406ddb0 0x7f0c740cd620 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:40.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.091+0000 7f0c79b67700 1 --2- 172.21.15.46:0/752879156 >> v2:172.21.15.106:3300/0 conn(0x7f0c74157490 0x7f0c740cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:40.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.091+0000 7f0c79b67700 1 -- 172.21.15.46:0/752879156 >> 172.21.15.46:0/752879156 conn(0x7f0c74067930 msgr2=0x7f0c74067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:40.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.091+0000 7f0c79b67700 1 -- 172.21.15.46:0/752879156 shutdown_connections 2024-01-27T01:06:40.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.091+0000 7f0c79b67700 1 -- 172.21.15.46:0/752879156 wait complete. 2024-01-27T01:06:40.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.092+0000 7f0c79b67700 1 Processor -- start 2024-01-27T01:06:40.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.093+0000 7f0c79b67700 1 -- start start 2024-01-27T01:06:40.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.093+0000 7f0c79b67700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c7406ddb0 0x7f0c74151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:40.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.093+0000 7f0c79b67700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c74157490 0x7f0c74159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:40.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.094+0000 7f0c79b67700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c74157860 0x7f0c74153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:40.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.094+0000 7f0c79b67700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c74167a10 con 0x7f0c7406ddb0 2024-01-27T01:06:40.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.094+0000 7f0c79b67700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7415a190 con 0x7f0c74157490 2024-01-27T01:06:40.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.094+0000 7f0c79b67700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7415a470 con 0x7f0c74157860 2024-01-27T01:06:40.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.094+0000 7f0c737fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c7406ddb0 0x7f0c74151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:40.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.094+0000 7f0c737fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c7406ddb0 0x7f0c74151cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33936/0 (socket says 172.21.15.46:33936) 2024-01-27T01:06:40.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.094+0000 7f0c737fe700 1 -- 172.21.15.46:0/1475861285 learned_addr learned my addr 172.21.15.46:0/1475861285 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.094+0000 7f0c72ffd700 1 --2- 172.21.15.46:0/1475861285 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c74157490 0x7f0c74159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.094+0000 7f0c73fff700 1 --2- 172.21.15.46:0/1475861285 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c74157860 0x7f0c74153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.094+0000 7f0c737fe700 1 -- 172.21.15.46:0/1475861285 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c74157860 msgr2=0x7f0c74153450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.094+0000 7f0c737fe700 1 --2- 172.21.15.46:0/1475861285 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c74157860 0x7f0c74153450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.094+0000 7f0c737fe700 1 -- 172.21.15.46:0/1475861285 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c74157490 msgr2=0x7f0c74159840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.094+0000 7f0c737fe700 1 --2- 172.21.15.46:0/1475861285 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c74157490 0x7f0c74159840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.094+0000 7f0c737fe700 1 -- 172.21.15.46:0/1475861285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c009ae0 con 0x7f0c7406ddb0 2024-01-27T01:06:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.094+0000 7f0c737fe700 1 --2- 172.21.15.46:0/1475861285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c7406ddb0 0x7f0c74151cb0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c00ec80 tx=0x7f0c5c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.094+0000 7f0c6b7fe700 1 -- 172.21.15.46:0/1475861285 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c00c850 con 0x7f0c7406ddb0 2024-01-27T01:06:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.095+0000 7f0c6b7fe700 1 -- 172.21.15.46:0/1475861285 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5c00c9b0 con 0x7f0c7406ddb0 2024-01-27T01:06:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.095+0000 7f0c79b67700 1 -- 172.21.15.46:0/1475861285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c7415a750 con 0x7f0c7406ddb0 2024-01-27T01:06:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.095+0000 7f0c6b7fe700 1 -- 172.21.15.46:0/1475861285 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c00cd50 con 0x7f0c7406ddb0 2024-01-27T01:06:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.095+0000 7f0c79b67700 1 -- 172.21.15.46:0/1475861285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c74159d80 con 0x7f0c7406ddb0 2024-01-27T01:06:40.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.096+0000 7f0c79b67700 1 -- 172.21.15.46:0/1475861285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7404c710 con 0x7f0c7406ddb0 2024-01-27T01:06:40.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.096+0000 7f0c6b7fe700 1 -- 172.21.15.46:0/1475861285 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0c5c013680 con 0x7f0c7406ddb0 2024-01-27T01:06:40.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.097+0000 7f0c6b7fe700 1 --2- 172.21.15.46:0/1475861285 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0c54066f30 0x7f0c540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:40.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.097+0000 7f0c6b7fe700 1 -- 172.21.15.46:0/1475861285 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 7275+0+0 (secure 0 0 0) 0x7f0c5c085540 con 0x7f0c7406ddb0 2024-01-27T01:06:40.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.097+0000 7f0c72ffd700 1 --2- 172.21.15.46:0/1475861285 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0c54066f30 0x7f0c540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:40.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.098+0000 7f0c72ffd700 1 --2- 172.21.15.46:0/1475861285 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0c54066f30 0x7f0c540693e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0c74156ed0 tx=0x7f0c64007ec0 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:40.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.101+0000 7f0c6b7fe700 1 -- 172.21.15.46:0/1475861285 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0c5c052210 con 0x7f0c7406ddb0 2024-01-27T01:06:40.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.352+0000 7f0c79b67700 1 -- 172.21.15.46:0/1475861285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7f0c741521f0 con 0x7f0c7406ddb0 2024-01-27T01:06:40.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.946+0000 7f0c6b7fe700 1 -- 172.21.15.46:0/1475861285 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 29 and data pool 30 v118) v1 ==== 183+0+0 (secure 0 0 0) 0x7f0c5c059e60 con 0x7f0c7406ddb0 2024-01-27T01:06:40.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.952+0000 7f0c79b67700 1 -- 172.21.15.46:0/1475861285 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0c54066f30 msgr2=0x7f0c540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:40.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.952+0000 7f0c79b67700 1 --2- 172.21.15.46:0/1475861285 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0c54066f30 0x7f0c540693e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0c74156ed0 tx=0x7f0c64007ec0 comp rx=0 tx=0).stop 2024-01-27T01:06:40.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.952+0000 7f0c79b67700 1 -- 172.21.15.46:0/1475861285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c7406ddb0 msgr2=0x7f0c74151cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:40.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.952+0000 7f0c79b67700 1 --2- 172.21.15.46:0/1475861285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c7406ddb0 0x7f0c74151cb0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c00ec80 tx=0x7f0c5c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:40.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.952+0000 7f0c79b67700 1 -- 172.21.15.46:0/1475861285 shutdown_connections 2024-01-27T01:06:40.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.952+0000 7f0c79b67700 1 --2- 172.21.15.46:0/1475861285 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0c54066f30 0x7f0c540693e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:40.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.952+0000 7f0c79b67700 1 --2- 172.21.15.46:0/1475861285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c7406ddb0 0x7f0c74151cb0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:40.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.952+0000 7f0c79b67700 1 --2- 172.21.15.46:0/1475861285 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c74157860 0x7f0c74153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:40.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.952+0000 7f0c79b67700 1 --2- 172.21.15.46:0/1475861285 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c74157490 0x7f0c74159840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:40.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.952+0000 7f0c79b67700 1 -- 172.21.15.46:0/1475861285 >> 172.21.15.46:0/1475861285 conn(0x7f0c74067930 msgr2=0x7f0c74155940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:40.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.952+0000 7f0c79b67700 1 -- 172.21.15.46:0/1475861285 shutdown_connections 2024-01-27T01:06:40.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:40.952+0000 7f0c79b67700 1 -- 172.21.15.46:0/1475861285 wait complete. 2024-01-27T01:06:40.953 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 29 and data pool 30 2024-01-27T01:06:40.968 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:06:41.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.103+0000 7ff4ea01f700 1 Processor -- start 2024-01-27T01:06:41.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.103+0000 7ff4ea01f700 1 -- start start 2024-01-27T01:06:41.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.103+0000 7ff4ea01f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff4e40cc1c0 0x7ff4e4164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:41.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.103+0000 7ff4ea01f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e40cff70 con 0x7ff4e415ce40 2024-01-27T01:06:41.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.104+0000 7ff4ea01f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e40cf400 con 0x7ff4e415d210 2024-01-27T01:06:41.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.104+0000 7ff4ea01f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e40cf660 con 0x7ff4e40cc1c0 2024-01-27T01:06:41.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.104+0000 7ff4e37fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff4e415ce40 0x7ff4e40d0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45382/0 (socket says 172.21.15.46:45382) 2024-01-27T01:06:41.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.104+0000 7ff4e3fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff4e40cc1c0 0x7ff4e4164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:41.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.104+0000 7ff4e37fe700 1 -- 172.21.15.46:0/3107005421 learned_addr learned my addr 172.21.15.46:0/3107005421 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:41.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.104+0000 7ff4e27fc700 1 -- 172.21.15.46:0/3107005421 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806959344 0 0) 0x7ff4e40cff70 con 0x7ff4e415ce40 2024-01-27T01:06:41.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.104+0000 7ff4e27fc700 1 -- 172.21.15.46:0/3107005421 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d0003570 con 0x7ff4e415ce40 2024-01-27T01:06:41.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.104+0000 7ff4e3fff700 1 -- 172.21.15.46:0/3107005421 >> v1:172.21.15.106:6789/0 conn(0x7ff4e415d210 legacy=0x7ff4e40cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:41.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.104+0000 7ff4e27fc700 1 -- 172.21.15.46:0/3107005421 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335324726 0 0) 0x7ff4e40cf400 con 0x7ff4e415d210 2024-01-27T01:06:41.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.104+0000 7ff4e3fff700 1 -- 172.21.15.46:0/3107005421 >> v1:172.21.15.46:6789/0 conn(0x7ff4e415ce40 legacy=0x7ff4e40d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:41.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.104+0000 7ff4e3fff700 1 -- 172.21.15.46:0/3107005421 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e40d0500 con 0x7ff4e40cc1c0 2024-01-27T01:06:41.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.104+0000 7ff4e3fff700 1 --2- 172.21.15.46:0/3107005421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e40cc1c0 0x7ff4e4164700 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc009e60 tx=0x7ff4dc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4723380d15de2a53 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:41.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.104+0000 7ff4e27fc700 1 -- 172.21.15.46:0/3107005421 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc00e070 con 0x7ff4e40cc1c0 2024-01-27T01:06:41.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.105+0000 7ff4e27fc700 1 -- 172.21.15.46:0/3107005421 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4dc00bb60 con 0x7ff4e40cc1c0 2024-01-27T01:06:41.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.105+0000 7ff4e27fc700 1 -- 172.21.15.46:0/3107005421 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc00bee0 con 0x7ff4e40cc1c0 2024-01-27T01:06:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.105+0000 7ff4ea01f700 1 -- 172.21.15.46:0/3107005421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e40cc1c0 msgr2=0x7ff4e4164700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.105+0000 7ff4ea01f700 1 --2- 172.21.15.46:0/3107005421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e40cc1c0 0x7ff4e4164700 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc009e60 tx=0x7ff4dc009780 comp rx=0 tx=0).stop 2024-01-27T01:06:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.105+0000 7ff4ea01f700 1 -- 172.21.15.46:0/3107005421 shutdown_connections 2024-01-27T01:06:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.105+0000 7ff4ea01f700 1 --2- 172.21.15.46:0/3107005421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e40cc1c0 0x7ff4e4164700 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.105+0000 7ff4ea01f700 1 -- 172.21.15.46:0/3107005421 >> 172.21.15.46:0/3107005421 conn(0x7ff4e406f840 msgr2=0x7ff4e40d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.105+0000 7ff4ea01f700 1 -- 172.21.15.46:0/3107005421 shutdown_connections 2024-01-27T01:06:41.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.105+0000 7ff4ea01f700 1 -- 172.21.15.46:0/3107005421 wait complete. 2024-01-27T01:06:41.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.107+0000 7ff4ea01f700 1 Processor -- start 2024-01-27T01:06:41.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.107+0000 7ff4ea01f700 1 -- start start 2024-01-27T01:06:41.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.107+0000 7ff4ea01f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4e40cc1c0 0x7ff4e416d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:41.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.107+0000 7ff4ea01f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4e415ce40 0x7ff4e41595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:41.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4ea01f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e415d210 0x7ff4e415eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:41.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4ea01f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e415a5e0 con 0x7ff4e415d210 2024-01-27T01:06:41.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4ea01f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e416de20 con 0x7ff4e40cc1c0 2024-01-27T01:06:41.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4ea01f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e4159b20 con 0x7ff4e415ce40 2024-01-27T01:06:41.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4e3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e415d210 0x7ff4e415eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:41.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4e3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e415d210 0x7ff4e415eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41762/0 (socket says 172.21.15.46:41762) 2024-01-27T01:06:41.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4e3fff700 1 -- 172.21.15.46:0/2472993388 learned_addr learned my addr 172.21.15.46:0/2472993388 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:41.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4e37fe700 1 --2- 172.21.15.46:0/2472993388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4e40cc1c0 0x7ff4e416d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:41.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4e2ffd700 1 --2- 172.21.15.46:0/2472993388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4e415ce40 0x7ff4e41595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:41.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4e3fff700 1 -- 172.21.15.46:0/2472993388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4e415ce40 msgr2=0x7ff4e41595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:41.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4e3fff700 1 --2- 172.21.15.46:0/2472993388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4e415ce40 0x7ff4e41595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4e3fff700 1 -- 172.21.15.46:0/2472993388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4e40cc1c0 msgr2=0x7ff4e416d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:41.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4e3fff700 1 --2- 172.21.15.46:0/2472993388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4e40cc1c0 0x7ff4e416d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4e3fff700 1 -- 172.21.15.46:0/2472993388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc009a20 con 0x7ff4e415d210 2024-01-27T01:06:41.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4e3fff700 1 --2- 172.21.15.46:0/2472993388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e415d210 0x7ff4e415eff0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc013000 tx=0x7ff4dc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:41.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4db7fe700 1 -- 172.21.15.46:0/2472993388 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc012e50 con 0x7ff4e415d210 2024-01-27T01:06:41.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4db7fe700 1 -- 172.21.15.46:0/2472993388 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4dc027470 con 0x7ff4e415d210 2024-01-27T01:06:41.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.108+0000 7ff4db7fe700 1 -- 172.21.15.46:0/2472993388 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc027830 con 0x7ff4e415d210 2024-01-27T01:06:41.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.109+0000 7ff4ea01f700 1 -- 172.21.15.46:0/2472993388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e40c7c10 con 0x7ff4e415d210 2024-01-27T01:06:41.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.109+0000 7ff4ea01f700 1 -- 172.21.15.46:0/2472993388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e40c81d0 con 0x7ff4e415d210 2024-01-27T01:06:41.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.110+0000 7ff4ea01f700 1 -- 172.21.15.46:0/2472993388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e409a540 con 0x7ff4e415d210 2024-01-27T01:06:41.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.110+0000 7ff4db7fe700 1 -- 172.21.15.46:0/2472993388 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff4dc024070 con 0x7ff4e415d210 2024-01-27T01:06:41.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.110+0000 7ff4db7fe700 1 --2- 172.21.15.46:0/2472993388 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff4c4066f30 0x7ff4c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:41.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.111+0000 7ff4db7fe700 1 -- 172.21.15.46:0/2472993388 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 7381+0+0 (secure 0 0 0) 0x7ff4dc087150 con 0x7ff4e415d210 2024-01-27T01:06:41.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.111+0000 7ff4e37fe700 1 --2- 172.21.15.46:0/2472993388 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff4c4066f30 0x7ff4c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:41.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.111+0000 7ff4e37fe700 1 --2- 172.21.15.46:0/2472993388 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff4c4066f30 0x7ff4c40693e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4000c00 tx=0x7ff4d4009040 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:41.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.115+0000 7ff4db7fe700 1 -- 172.21.15.46:0/2472993388 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff4e409a540 con 0x7ff4e415d210 2024-01-27T01:06:41.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.325+0000 7ff4ea01f700 1 -- 172.21.15.46:0/2472993388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4e409a540 con 0x7ff4e415d210 2024-01-27T01:06:41.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.326+0000 7ff4db7fe700 1 -- 172.21.15.46:0/2472993388 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v226) v1 ==== 74+0+19570 (secure 0 0 0) 0x7ff4e409a540 con 0x7ff4e415d210 2024-01-27T01:06:41.325 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:41.325 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":226,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:06:40.352897+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:06:24.798791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-01-27T01:06:25.249215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:06:38.156934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:06:39.330686+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:06:41.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.328+0000 7ff4ea01f700 1 -- 172.21.15.46:0/2472993388 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff4c4066f30 msgr2=0x7ff4c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:41.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.328+0000 7ff4ea01f700 1 --2- 172.21.15.46:0/2472993388 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff4c4066f30 0x7ff4c40693e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4000c00 tx=0x7ff4d4009040 comp rx=0 tx=0).stop 2024-01-27T01:06:41.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.328+0000 7ff4ea01f700 1 -- 172.21.15.46:0/2472993388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e415d210 msgr2=0x7ff4e415eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:41.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.328+0000 7ff4ea01f700 1 --2- 172.21.15.46:0/2472993388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e415d210 0x7ff4e415eff0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc013000 tx=0x7ff4dc019000 comp rx=0 tx=0).stop 2024-01-27T01:06:41.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.328+0000 7ff4ea01f700 1 -- 172.21.15.46:0/2472993388 shutdown_connections 2024-01-27T01:06:41.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.328+0000 7ff4ea01f700 1 --2- 172.21.15.46:0/2472993388 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff4c4066f30 0x7ff4c40693e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.328+0000 7ff4ea01f700 1 --2- 172.21.15.46:0/2472993388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e415d210 0x7ff4e415eff0 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.328+0000 7ff4ea01f700 1 --2- 172.21.15.46:0/2472993388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4e40cc1c0 0x7ff4e416d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.328+0000 7ff4ea01f700 1 --2- 172.21.15.46:0/2472993388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4e415ce40 0x7ff4e41595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.328+0000 7ff4ea01f700 1 -- 172.21.15.46:0/2472993388 >> 172.21.15.46:0/2472993388 conn(0x7ff4e406f840 msgr2=0x7ff4e40d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:41.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.328+0000 7ff4ea01f700 1 -- 172.21.15.46:0/2472993388 shutdown_connections 2024-01-27T01:06:41.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.328+0000 7ff4ea01f700 1 -- 172.21.15.46:0/2472993388 wait complete. 2024-01-27T01:06:41.341 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:06:41.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.475+0000 7f51a7530700 1 Processor -- start 2024-01-27T01:06:41.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.475+0000 7f51a7530700 1 -- start start 2024-01-27T01:06:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.476+0000 7f51a7530700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f51a00ccc20 0x7f51a0068340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.476+0000 7f51a7530700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51a00ccff0 0x7f51a006eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.476+0000 7f51a7530700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51a0158020 con 0x7f51a00b2590 2024-01-27T01:06:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.476+0000 7f51a7530700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51a0158300 con 0x7f51a00ccc20 2024-01-27T01:06:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.476+0000 7f51a7530700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51a01585e0 con 0x7f51a00ccff0 2024-01-27T01:06:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.476+0000 7f51a5acd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f51a00b2590 0x7f51a015b5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45388/0 (socket says 172.21.15.46:45388) 2024-01-27T01:06:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.476+0000 7f51a5acd700 1 -- 172.21.15.46:0/3864022971 learned_addr learned my addr 172.21.15.46:0/3864022971 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.476+0000 7f51a52cc700 1 --2- 172.21.15.46:0/3864022971 >> v2:172.21.15.106:3300/0 conn(0x7f51a00ccc20 0x7f51a0068340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.476+0000 7f51a4acb700 1 --2- 172.21.15.46:0/3864022971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51a00ccff0 0x7f51a006eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.476+0000 7f5197fff700 1 -- 172.21.15.46:0/3864022971 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17127082 0 0) 0x7f51a0158020 con 0x7f51a00b2590 2024-01-27T01:06:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.476+0000 7f5197fff700 1 -- 172.21.15.46:0/3864022971 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f518c003570 con 0x7f51a00b2590 2024-01-27T01:06:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.476+0000 7f51a52cc700 1 -- 172.21.15.46:0/3864022971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51a00ccff0 msgr2=0x7f51a006eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.476+0000 7f51a52cc700 1 --2- 172.21.15.46:0/3864022971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51a00ccff0 0x7f51a006eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.476+0000 7f51a52cc700 1 -- 172.21.15.46:0/3864022971 >> v1:172.21.15.46:6789/0 conn(0x7f51a00b2590 legacy=0x7f51a015b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.476+0000 7f51a52cc700 1 -- 172.21.15.46:0/3864022971 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a01588c0 con 0x7f51a00ccc20 2024-01-27T01:06:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.476+0000 7f51a4acb700 1 --2- 172.21.15.46:0/3864022971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51a00ccff0 0x7f51a006eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.477+0000 7f51a52cc700 1 --2- 172.21.15.46:0/3864022971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a00ccc20 0x7f51a0068340 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f519c0098e0 tx=0x7f519c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cb1c076e0e03f23 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.477+0000 7f5197fff700 1 -- 172.21.15.46:0/3864022971 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c011630 con 0x7f51a00ccc20 2024-01-27T01:06:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.477+0000 7f5197fff700 1 -- 172.21.15.46:0/3864022971 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519c011790 con 0x7f51a00ccc20 2024-01-27T01:06:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.477+0000 7f5197fff700 1 -- 172.21.15.46:0/3864022971 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c011b30 con 0x7f51a00ccc20 2024-01-27T01:06:41.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.477+0000 7f51a7530700 1 -- 172.21.15.46:0/3864022971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a00ccc20 msgr2=0x7f51a0068340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:41.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.477+0000 7f51a7530700 1 --2- 172.21.15.46:0/3864022971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a00ccc20 0x7f51a0068340 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f519c0098e0 tx=0x7f519c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.477+0000 7f51a7530700 1 -- 172.21.15.46:0/3864022971 shutdown_connections 2024-01-27T01:06:41.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.477+0000 7f51a7530700 1 --2- 172.21.15.46:0/3864022971 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a00ccc20 0x7f51a0068340 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.477+0000 7f51a7530700 1 --2- 172.21.15.46:0/3864022971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51a00ccff0 0x7f51a006eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.477+0000 7f51a7530700 1 -- 172.21.15.46:0/3864022971 >> 172.21.15.46:0/3864022971 conn(0x7f51a006b630 msgr2=0x7f51a00aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:41.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.478+0000 7f51a7530700 1 -- 172.21.15.46:0/3864022971 shutdown_connections 2024-01-27T01:06:41.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.478+0000 7f51a7530700 1 -- 172.21.15.46:0/3864022971 wait complete. 2024-01-27T01:06:41.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.479+0000 7f51a7530700 1 Processor -- start 2024-01-27T01:06:41.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.479+0000 7f51a7530700 1 -- start start 2024-01-27T01:06:41.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.480+0000 7f51a7530700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51a00b2590 0x7f51a00a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:41.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.480+0000 7f51a7530700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a00ccc20 0x7f51a00b0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:41.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.480+0000 7f51a7530700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51a00ccff0 0x7f51a00a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:41.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.480+0000 7f51a7530700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a00b1d30 con 0x7f51a00ccff0 2024-01-27T01:06:41.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.480+0000 7f51a7530700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a00b1280 con 0x7f51a00ccc20 2024-01-27T01:06:41.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.480+0000 7f51a7530700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51a00c77f0 con 0x7f51a00b2590 2024-01-27T01:06:41.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.480+0000 7f51a5acd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51a00ccff0 0x7f51a00a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:41.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.480+0000 7f51a5acd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51a00ccff0 0x7f51a00a5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41774/0 (socket says 172.21.15.46:41774) 2024-01-27T01:06:41.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.480+0000 7f51a5acd700 1 -- 172.21.15.46:0/1416922579 learned_addr learned my addr 172.21.15.46:0/1416922579 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:41.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.480+0000 7f51a52cc700 1 --2- 172.21.15.46:0/1416922579 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51a00b2590 0x7f51a00a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:41.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.480+0000 7f51a4acb700 1 --2- 172.21.15.46:0/1416922579 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a00ccc20 0x7f51a00b0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:41.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.480+0000 7f51a5acd700 1 -- 172.21.15.46:0/1416922579 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51a00b2590 msgr2=0x7f51a00a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:41.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.480+0000 7f51a5acd700 1 --2- 172.21.15.46:0/1416922579 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51a00b2590 0x7f51a00a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.480+0000 7f51a5acd700 1 -- 172.21.15.46:0/1416922579 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a00ccc20 msgr2=0x7f51a00b0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:41.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.480+0000 7f51a5acd700 1 --2- 172.21.15.46:0/1416922579 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a00ccc20 0x7f51a00b0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.480+0000 7f51a5acd700 1 -- 172.21.15.46:0/1416922579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5190009420 con 0x7f51a00ccff0 2024-01-27T01:06:41.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.480+0000 7f51a5acd700 1 --2- 172.21.15.46:0/1416922579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51a00ccff0 0x7f51a00a5ef0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f519800cca0 tx=0x7f5198001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:41.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.481+0000 7f51957fa700 1 -- 172.21.15.46:0/1416922579 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519800a8d0 con 0x7f51a00ccff0 2024-01-27T01:06:41.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.481+0000 7f51957fa700 1 -- 172.21.15.46:0/1416922579 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519800aa30 con 0x7f51a00ccff0 2024-01-27T01:06:41.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.481+0000 7f51957fa700 1 -- 172.21.15.46:0/1416922579 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519800adb0 con 0x7f51a00ccff0 2024-01-27T01:06:41.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.481+0000 7f51a7530700 1 -- 172.21.15.46:0/1416922579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f519c009420 con 0x7f51a00ccff0 2024-01-27T01:06:41.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.481+0000 7f51a7530700 1 -- 172.21.15.46:0/1416922579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a00b1770 con 0x7f51a00ccff0 2024-01-27T01:06:41.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.482+0000 7f51a7530700 1 -- 172.21.15.46:0/1416922579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a00c75c0 con 0x7f51a00ccff0 2024-01-27T01:06:41.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.482+0000 7f51957fa700 1 -- 172.21.15.46:0/1416922579 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5198007300 con 0x7f51a00ccff0 2024-01-27T01:06:41.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.483+0000 7f51957fa700 1 --2- 172.21.15.46:0/1416922579 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5184066f30 0x7f51840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:41.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.483+0000 7f51957fa700 1 -- 172.21.15.46:0/1416922579 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f5198084f50 con 0x7f51a00ccff0 2024-01-27T01:06:41.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.483+0000 7f51a52cc700 1 --2- 172.21.15.46:0/1416922579 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5184066f30 0x7f51840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:41.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.484+0000 7f51a52cc700 1 --2- 172.21.15.46:0/1416922579 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5184066f30 0x7f51840693e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f519c0091e0 tx=0x7f519c017000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:41.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.487+0000 7f51957fa700 1 -- 172.21.15.46:0/1416922579 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f51a00c75c0 con 0x7f51a00ccff0 2024-01-27T01:06:41.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.693+0000 7f51a7530700 1 -- 172.21.15.46:0/1416922579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51a00c75c0 con 0x7f51a00ccff0 2024-01-27T01:06:41.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.694+0000 7f51957fa700 1 -- 172.21.15.46:0/1416922579 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v226) v1 ==== 74+0+19570 (secure 0 0 0) 0x7f51a00c75c0 con 0x7f51a00ccff0 2024-01-27T01:06:41.693 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:41.694 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":226,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:06:40.352897+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:06:24.798791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-01-27T01:06:25.249215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:06:38.156934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:06:39.330686+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:06:41.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.696+0000 7f51a7530700 1 -- 172.21.15.46:0/1416922579 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5184066f30 msgr2=0x7f51840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:41.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.696+0000 7f51a7530700 1 --2- 172.21.15.46:0/1416922579 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5184066f30 0x7f51840693e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f519c0091e0 tx=0x7f519c017000 comp rx=0 tx=0).stop 2024-01-27T01:06:41.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.696+0000 7f51a7530700 1 -- 172.21.15.46:0/1416922579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51a00ccff0 msgr2=0x7f51a00a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:41.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.696+0000 7f51a7530700 1 --2- 172.21.15.46:0/1416922579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51a00ccff0 0x7f51a00a5ef0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f519800cca0 tx=0x7f5198001ae0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.696+0000 7f51a7530700 1 -- 172.21.15.46:0/1416922579 shutdown_connections 2024-01-27T01:06:41.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.696+0000 7f51a7530700 1 --2- 172.21.15.46:0/1416922579 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5184066f30 0x7f51840693e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.696+0000 7f51a7530700 1 --2- 172.21.15.46:0/1416922579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51a00ccff0 0x7f51a00a5ef0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.696+0000 7f51a7530700 1 --2- 172.21.15.46:0/1416922579 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51a00b2590 0x7f51a00a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.696+0000 7f51a7530700 1 --2- 172.21.15.46:0/1416922579 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a00ccc20 0x7f51a00b0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.696+0000 7f51a7530700 1 -- 172.21.15.46:0/1416922579 >> 172.21.15.46:0/1416922579 conn(0x7f51a006b630 msgr2=0x7f51a0159f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:41.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.696+0000 7f51a7530700 1 -- 172.21.15.46:0/1416922579 shutdown_connections 2024-01-27T01:06:41.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.697+0000 7f51a7530700 1 -- 172.21.15.46:0/1416922579 wait complete. 2024-01-27T01:06:41.709 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-01-27T01:06:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.845+0000 7f524bd99700 1 Processor -- start 2024-01-27T01:06:41.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.846+0000 7f524bd99700 1 -- start start 2024-01-27T01:06:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.846+0000 7f524bd99700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f52441565a0 0x7f5244156970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.846+0000 7f524bd99700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52440ae2f0 0x7f52440ae750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.846+0000 7f524bd99700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52440b0830 con 0x7f524415a520 2024-01-27T01:06:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.846+0000 7f524bd99700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52440ae130 con 0x7f52441565a0 2024-01-27T01:06:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.846+0000 7f524bd99700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52440aec90 con 0x7f52440ae2f0 2024-01-27T01:06:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.847+0000 7f5249334700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f524415a520 0x7f52440acc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45398/0 (socket says 172.21.15.46:45398) 2024-01-27T01:06:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.847+0000 7f5249334700 1 -- 172.21.15.46:0/2136377501 learned_addr learned my addr 172.21.15.46:0/2136377501 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.847+0000 7f5249b35700 1 --2- 172.21.15.46:0/2136377501 >> v2:172.21.15.106:3300/0 conn(0x7f52441565a0 0x7f5244156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.847+0000 7f524a336700 1 --2- 172.21.15.46:0/2136377501 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52440ae2f0 0x7f52440ae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.847+0000 7f5248b33700 1 -- 172.21.15.46:0/2136377501 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1781088755 0 0) 0x7f52440b0830 con 0x7f524415a520 2024-01-27T01:06:41.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.847+0000 7f5248b33700 1 -- 172.21.15.46:0/2136377501 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5230003570 con 0x7f524415a520 2024-01-27T01:06:41.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.847+0000 7f5248b33700 1 -- 172.21.15.46:0/2136377501 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 270809749 0 0) 0x7f5230003570 con 0x7f524415a520 2024-01-27T01:06:41.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.847+0000 7f524a336700 1 -- 172.21.15.46:0/2136377501 >> v2:172.21.15.106:3300/0 conn(0x7f52441565a0 msgr2=0x7f5244156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:41.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.847+0000 7f524a336700 1 --2- 172.21.15.46:0/2136377501 >> v2:172.21.15.106:3300/0 conn(0x7f52441565a0 0x7f5244156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.847+0000 7f524a336700 1 -- 172.21.15.46:0/2136377501 >> v1:172.21.15.46:6789/0 conn(0x7f524415a520 legacy=0x7f52440acc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:41.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.847+0000 7f524a336700 1 -- 172.21.15.46:0/2136377501 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52440ad350 con 0x7f52440ae2f0 2024-01-27T01:06:41.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.847+0000 7f5249b35700 1 --2- 172.21.15.46:0/2136377501 >> v2:172.21.15.106:3300/0 conn(0x7f52441565a0 0x7f5244156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:41.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.848+0000 7f524a336700 1 --2- 172.21.15.46:0/2136377501 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52440ae2f0 0x7f52440ae750 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f52440ad4e0 tx=0x7f523c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ab5024c78818550 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:41.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.848+0000 7f5248b33700 1 -- 172.21.15.46:0/2136377501 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523c00e070 con 0x7f52440ae2f0 2024-01-27T01:06:41.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.848+0000 7f5248b33700 1 -- 172.21.15.46:0/2136377501 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523c00b7a0 con 0x7f52440ae2f0 2024-01-27T01:06:41.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.848+0000 7f5248b33700 1 -- 172.21.15.46:0/2136377501 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5240001470 con 0x7f52440ae2f0 2024-01-27T01:06:41.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.848+0000 7f524bd99700 1 -- 172.21.15.46:0/2136377501 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52440ae2f0 msgr2=0x7f52440ae750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.848+0000 7f524bd99700 1 --2- 172.21.15.46:0/2136377501 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52440ae2f0 0x7f52440ae750 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f52440ad4e0 tx=0x7f523c009780 comp rx=0 tx=0).stop 2024-01-27T01:06:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.848+0000 7f524bd99700 1 -- 172.21.15.46:0/2136377501 shutdown_connections 2024-01-27T01:06:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.848+0000 7f524bd99700 1 --2- 172.21.15.46:0/2136377501 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52440ae2f0 0x7f52440ae750 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.848+0000 7f524bd99700 1 --2- 172.21.15.46:0/2136377501 >> v2:172.21.15.106:3300/0 conn(0x7f52441565a0 0x7f5244156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.848+0000 7f524bd99700 1 -- 172.21.15.46:0/2136377501 >> 172.21.15.46:0/2136377501 conn(0x7f52440cdf80 msgr2=0x7f524406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.849+0000 7f524bd99700 1 -- 172.21.15.46:0/2136377501 shutdown_connections 2024-01-27T01:06:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.849+0000 7f524bd99700 1 -- 172.21.15.46:0/2136377501 wait complete. 2024-01-27T01:06:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.850+0000 7f524bd99700 1 Processor -- start 2024-01-27T01:06:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.850+0000 7f524bd99700 1 -- start start 2024-01-27T01:06:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.851+0000 7f524bd99700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52440ae2f0 0x7f52440ac030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.851+0000 7f524bd99700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52441565a0 0x7f52440b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.851+0000 7f524bd99700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f524415a520 0x7f5244159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.851+0000 7f524bd99700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52440b1610 con 0x7f52441565a0 2024-01-27T01:06:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.851+0000 7f524bd99700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52440b18f0 con 0x7f524415a520 2024-01-27T01:06:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.851+0000 7f524bd99700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52440b1bd0 con 0x7f52440ae2f0 2024-01-27T01:06:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.851+0000 7f5249334700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52441565a0 0x7f52440b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.851+0000 7f5249334700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52441565a0 0x7f52440b0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41790/0 (socket says 172.21.15.46:41790) 2024-01-27T01:06:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.851+0000 7f5249334700 1 -- 172.21.15.46:0/847934803 learned_addr learned my addr 172.21.15.46:0/847934803 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.851+0000 7f5249b35700 1 --2- 172.21.15.46:0/847934803 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52440ae2f0 0x7f52440ac030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.851+0000 7f524a336700 1 --2- 172.21.15.46:0/847934803 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f524415a520 0x7f5244159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.851+0000 7f5249334700 1 -- 172.21.15.46:0/847934803 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52440ae2f0 msgr2=0x7f52440ac030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.851+0000 7f5249334700 1 --2- 172.21.15.46:0/847934803 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52440ae2f0 0x7f52440ac030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.851+0000 7f5249334700 1 -- 172.21.15.46:0/847934803 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f524415a520 msgr2=0x7f5244159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.851+0000 7f5249334700 1 --2- 172.21.15.46:0/847934803 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f524415a520 0x7f5244159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.851+0000 7f5249334700 1 -- 172.21.15.46:0/847934803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5238009420 con 0x7f52441565a0 2024-01-27T01:06:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.851+0000 7f5249334700 1 --2- 172.21.15.46:0/847934803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52441565a0 0x7f52440b0ca0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f5240001660 tx=0x7f5240001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.852+0000 7f5235ffb700 1 -- 172.21.15.46:0/847934803 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5240007150 con 0x7f52441565a0 2024-01-27T01:06:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.852+0000 7f5235ffb700 1 -- 172.21.15.46:0/847934803 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52400072b0 con 0x7f52441565a0 2024-01-27T01:06:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.852+0000 7f524bd99700 1 -- 172.21.15.46:0/847934803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523c009a20 con 0x7f52441565a0 2024-01-27T01:06:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.852+0000 7f5235ffb700 1 -- 172.21.15.46:0/847934803 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5240007670 con 0x7f52441565a0 2024-01-27T01:06:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.852+0000 7f524bd99700 1 -- 172.21.15.46:0/847934803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52440b2190 con 0x7f52441565a0 2024-01-27T01:06:41.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.853+0000 7f524bd99700 1 -- 172.21.15.46:0/847934803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52440b23b0 con 0x7f52441565a0 2024-01-27T01:06:41.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.853+0000 7f5235ffb700 1 -- 172.21.15.46:0/847934803 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5240004a40 con 0x7f52441565a0 2024-01-27T01:06:41.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.854+0000 7f5235ffb700 1 --2- 172.21.15.46:0/847934803 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5228066f30 0x7f52280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:41.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.854+0000 7f5235ffb700 1 -- 172.21.15.46:0/847934803 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f52400842d0 con 0x7f52441565a0 2024-01-27T01:06:41.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.854+0000 7f5249b35700 1 --2- 172.21.15.46:0/847934803 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5228066f30 0x7f52280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:41.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.854+0000 7f5249b35700 1 --2- 172.21.15.46:0/847934803 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5228066f30 0x7f52280693e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5238009f40 tx=0x7f52380091a0 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:41.858+0000 7f5235ffb700 1 -- 172.21.15.46:0/847934803 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5240050f30 con 0x7f52441565a0 2024-01-27T01:06:42.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:42.114+0000 7f524bd99700 1 -- 172.21.15.46:0/847934803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f524404c710 con 0x7f52441565a0 2024-01-27T01:06:42.772 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-01-27T01:06:42.773+0000 7f6f3f584700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-27T01:06:42.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:42.940+0000 7f5235ffb700 1 -- 172.21.15.46:0/847934803 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v121) v1 ==== 121+0+0 (secure 0 0 0) 0x7f5240054550 con 0x7f52441565a0 2024-01-27T01:06:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:42.943+0000 7f524bd99700 1 -- 172.21.15.46:0/847934803 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5228066f30 msgr2=0x7f52280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:42.943+0000 7f524bd99700 1 --2- 172.21.15.46:0/847934803 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5228066f30 0x7f52280693e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5238009f40 tx=0x7f52380091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:42.943+0000 7f524bd99700 1 -- 172.21.15.46:0/847934803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52441565a0 msgr2=0x7f52440b0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:42.943+0000 7f524bd99700 1 --2- 172.21.15.46:0/847934803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52441565a0 0x7f52440b0ca0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f5240001660 tx=0x7f5240001a80 comp rx=0 tx=0).stop 2024-01-27T01:06:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:42.943+0000 7f524bd99700 1 -- 172.21.15.46:0/847934803 shutdown_connections 2024-01-27T01:06:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:42.943+0000 7f524bd99700 1 --2- 172.21.15.46:0/847934803 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5228066f30 0x7f52280693e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:42.943+0000 7f524bd99700 1 --2- 172.21.15.46:0/847934803 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f524415a520 0x7f5244159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:42.943+0000 7f524bd99700 1 --2- 172.21.15.46:0/847934803 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52440ae2f0 0x7f52440ac030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:42.943+0000 7f524bd99700 1 --2- 172.21.15.46:0/847934803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52441565a0 0x7f52440b0ca0 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:42.943+0000 7f524bd99700 1 -- 172.21.15.46:0/847934803 >> 172.21.15.46:0/847934803 conn(0x7f52440cdf80 msgr2=0x7f5244151c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:42.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:42.943+0000 7f524bd99700 1 -- 172.21.15.46:0/847934803 shutdown_connections 2024-01-27T01:06:42.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:42.943+0000 7f524bd99700 1 -- 172.21.15.46:0/847934803 wait complete. 2024-01-27T01:06:42.957 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-01-27T01:06:43.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.096+0000 7f7dc5873700 1 Processor -- start 2024-01-27T01:06:43.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.096+0000 7f7dc5873700 1 -- start start 2024-01-27T01:06:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.097+0000 7f7dc5873700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7dc01599c0 0x7f7dc00ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.097+0000 7f7dc5873700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7dc00caa60 0x7f7dc0159d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.097+0000 7f7dc5873700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc00c8f70 con 0x7f7dc0068c50 2024-01-27T01:06:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.097+0000 7f7dc5873700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc015a9c0 con 0x7f7dc00caa60 2024-01-27T01:06:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.097+0000 7f7dc5873700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc015aca0 con 0x7f7dc01599c0 2024-01-27T01:06:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.097+0000 7f7dbf7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7dc00caa60 0x7f7dc0159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.097+0000 7f7db67fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7dc0068c50 0x7f7dc00ca350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45414/0 (socket says 172.21.15.46:45414) 2024-01-27T01:06:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.097+0000 7f7db67fc700 1 -- 172.21.15.46:0/3759378410 learned_addr learned my addr 172.21.15.46:0/3759378410 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.097+0000 7f7dbeffd700 1 --2- 172.21.15.46:0/3759378410 >> v2:172.21.15.106:3300/0 conn(0x7f7dc01599c0 0x7f7dc00ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.097+0000 7f7dbf7fe700 1 -- 172.21.15.46:0/3759378410 >> v2:172.21.15.106:3300/0 conn(0x7f7dc01599c0 msgr2=0x7f7dc00ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.097+0000 7f7dbf7fe700 1 --2- 172.21.15.46:0/3759378410 >> v2:172.21.15.106:3300/0 conn(0x7f7dc01599c0 0x7f7dc00ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.097+0000 7f7dbf7fe700 1 -- 172.21.15.46:0/3759378410 >> v1:172.21.15.46:6789/0 conn(0x7f7dc0068c50 legacy=0x7f7dc00ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.097+0000 7f7dbf7fe700 1 -- 172.21.15.46:0/3759378410 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc015af80 con 0x7f7dc00caa60 2024-01-27T01:06:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.098+0000 7f7dbf7fe700 1 --2- 172.21.15.46:0/3759378410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc00caa60 0x7f7dc0159d90 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f7dc00caf20 tx=0x7f7db8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcde4aeb589af97f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.098+0000 7f7dbe7fc700 1 -- 172.21.15.46:0/3759378410 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db800e070 con 0x7f7dc00caa60 2024-01-27T01:06:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.098+0000 7f7dbe7fc700 1 -- 172.21.15.46:0/3759378410 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db800bb60 con 0x7f7dc00caa60 2024-01-27T01:06:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.098+0000 7f7dbe7fc700 1 -- 172.21.15.46:0/3759378410 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db8012430 con 0x7f7dc00caa60 2024-01-27T01:06:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.098+0000 7f7dc5873700 1 -- 172.21.15.46:0/3759378410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc00caa60 msgr2=0x7f7dc0159d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.098+0000 7f7dc5873700 1 --2- 172.21.15.46:0/3759378410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc00caa60 0x7f7dc0159d90 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f7dc00caf20 tx=0x7f7db8009780 comp rx=0 tx=0).stop 2024-01-27T01:06:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.098+0000 7f7dc5873700 1 -- 172.21.15.46:0/3759378410 shutdown_connections 2024-01-27T01:06:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.098+0000 7f7dc5873700 1 --2- 172.21.15.46:0/3759378410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc00caa60 0x7f7dc0159d90 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.098+0000 7f7dc5873700 1 --2- 172.21.15.46:0/3759378410 >> v2:172.21.15.106:3300/0 conn(0x7f7dc01599c0 0x7f7dc00ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.098+0000 7f7dc5873700 1 -- 172.21.15.46:0/3759378410 >> 172.21.15.46:0/3759378410 conn(0x7f7dc00ce570 msgr2=0x7f7dc00ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.098+0000 7f7dc5873700 1 -- 172.21.15.46:0/3759378410 shutdown_connections 2024-01-27T01:06:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.099+0000 7f7dc5873700 1 -- 172.21.15.46:0/3759378410 wait complete. 2024-01-27T01:06:43.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.100+0000 7f7dc5873700 1 Processor -- start 2024-01-27T01:06:43.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.100+0000 7f7dc5873700 1 -- start start 2024-01-27T01:06:43.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.101+0000 7f7dc5873700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7dc0068c50 0x7f7dc0143390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:43.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.101+0000 7f7dc5873700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dc00caa60 0x7f7dc00c7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:43.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.101+0000 7f7dc5873700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc01599c0 0x7f7dc00c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:43.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.101+0000 7f7dc5873700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc00c8960 con 0x7f7dc01599c0 2024-01-27T01:06:43.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.101+0000 7f7dc5873700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc00c7de0 con 0x7f7dc00caa60 2024-01-27T01:06:43.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.101+0000 7f7dc5873700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc00c80c0 con 0x7f7dc0068c50 2024-01-27T01:06:43.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.102+0000 7f7dbf7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc01599c0 0x7f7dc00c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:43.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.102+0000 7f7dbf7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc01599c0 0x7f7dc00c7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41808/0 (socket says 172.21.15.46:41808) 2024-01-27T01:06:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.102+0000 7f7db67fc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dc00caa60 0x7f7dc00c7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.102+0000 7f7dbf7fe700 1 -- 172.21.15.46:0/3021556516 learned_addr learned my addr 172.21.15.46:0/3021556516 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.102+0000 7f7dbeffd700 1 --2- 172.21.15.46:0/3021556516 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7dc0068c50 0x7f7dc0143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.102+0000 7f7dbf7fe700 1 -- 172.21.15.46:0/3021556516 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7dc0068c50 msgr2=0x7f7dc0143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.102+0000 7f7dbf7fe700 1 --2- 172.21.15.46:0/3021556516 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7dc0068c50 0x7f7dc0143390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.102+0000 7f7dbf7fe700 1 -- 172.21.15.46:0/3021556516 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dc00caa60 msgr2=0x7f7dc00c7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.102+0000 7f7dbf7fe700 1 --2- 172.21.15.46:0/3021556516 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dc00caa60 0x7f7dc00c7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.102+0000 7f7dbf7fe700 1 -- 172.21.15.46:0/3021556516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db8009a20 con 0x7f7dc01599c0 2024-01-27T01:06:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.102+0000 7f7db67fc700 1 --2- 172.21.15.46:0/3021556516 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dc00caa60 0x7f7dc00c7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.102+0000 7f7dbf7fe700 1 --2- 172.21.15.46:0/3021556516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc01599c0 0x7f7dc00c7760 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f7db8009fd0 tx=0x7f7db8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.102+0000 7f7db77fe700 1 -- 172.21.15.46:0/3021556516 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db8027470 con 0x7f7dc01599c0 2024-01-27T01:06:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.102+0000 7f7db77fe700 1 -- 172.21.15.46:0/3021556516 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db80275f0 con 0x7f7dc01599c0 2024-01-27T01:06:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.102+0000 7f7dc5873700 1 -- 172.21.15.46:0/3021556516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc00c83a0 con 0x7f7dc01599c0 2024-01-27T01:06:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.102+0000 7f7db77fe700 1 -- 172.21.15.46:0/3021556516 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db80279b0 con 0x7f7dc01599c0 2024-01-27T01:06:43.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.102+0000 7f7dc5873700 1 -- 172.21.15.46:0/3021556516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc00c5890 con 0x7f7dc01599c0 2024-01-27T01:06:43.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.103+0000 7f7dc5873700 1 -- 172.21.15.46:0/3021556516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc016a920 con 0x7f7dc01599c0 2024-01-27T01:06:43.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.104+0000 7f7db77fe700 1 -- 172.21.15.46:0/3021556516 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7db8024070 con 0x7f7dc01599c0 2024-01-27T01:06:43.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.104+0000 7f7db77fe700 1 --2- 172.21.15.46:0/3021556516 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7da0066f30 0x7f7da00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:43.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.104+0000 7f7db77fe700 1 -- 172.21.15.46:0/3021556516 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f7db80871b0 con 0x7f7dc01599c0 2024-01-27T01:06:43.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.105+0000 7f7dbeffd700 1 --2- 172.21.15.46:0/3021556516 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7da0066f30 0x7f7da00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:43.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.105+0000 7f7dbeffd700 1 --2- 172.21.15.46:0/3021556516 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7da0066f30 0x7f7da00693e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7db0009d10 tx=0x7f7db00091a0 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:43.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.109+0000 7f7db77fe700 1 -- 172.21.15.46:0/3021556516 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7dc016a920 con 0x7f7dc01599c0 2024-01-27T01:06:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.360+0000 7f7dc5873700 1 -- 172.21.15.46:0/3021556516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7dc016a920 con 0x7f7dc01599c0 2024-01-27T01:06:43.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.945+0000 7f7db77fe700 1 -- 172.21.15.46:0/3021556516 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v122) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7dc016a920 con 0x7f7dc01599c0 2024-01-27T01:06:43.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.948+0000 7f7dc5873700 1 -- 172.21.15.46:0/3021556516 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7da0066f30 msgr2=0x7f7da00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:43.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.948+0000 7f7dc5873700 1 --2- 172.21.15.46:0/3021556516 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7da0066f30 0x7f7da00693e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7db0009d10 tx=0x7f7db00091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:43.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.948+0000 7f7dc5873700 1 -- 172.21.15.46:0/3021556516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc01599c0 msgr2=0x7f7dc00c7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:43.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.948+0000 7f7dc5873700 1 --2- 172.21.15.46:0/3021556516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc01599c0 0x7f7dc00c7760 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f7db8009fd0 tx=0x7f7db8019000 comp rx=0 tx=0).stop 2024-01-27T01:06:43.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.948+0000 7f7dc5873700 1 -- 172.21.15.46:0/3021556516 shutdown_connections 2024-01-27T01:06:43.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.948+0000 7f7dc5873700 1 --2- 172.21.15.46:0/3021556516 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7da0066f30 0x7f7da00693e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:43.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.948+0000 7f7dc5873700 1 --2- 172.21.15.46:0/3021556516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc01599c0 0x7f7dc00c7760 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:43.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.948+0000 7f7dc5873700 1 --2- 172.21.15.46:0/3021556516 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7dc0068c50 0x7f7dc0143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:43.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.948+0000 7f7dc5873700 1 --2- 172.21.15.46:0/3021556516 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7dc00caa60 0x7f7dc00c7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:43.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.948+0000 7f7dc5873700 1 -- 172.21.15.46:0/3021556516 >> 172.21.15.46:0/3021556516 conn(0x7f7dc00ce570 msgr2=0x7f7dc0154c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:43.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.948+0000 7f7dc5873700 1 -- 172.21.15.46:0/3021556516 shutdown_connections 2024-01-27T01:06:43.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:43.948+0000 7f7dc5873700 1 -- 172.21.15.46:0/3021556516 wait complete. 2024-01-27T01:06:43.963 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:06:44.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.099+0000 7efdb674d700 1 Processor -- start 2024-01-27T01:06:44.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.099+0000 7efdb674d700 1 -- start start 2024-01-27T01:06:44.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.100+0000 7efdb674d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efdb01596e0 0x7efdb0159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.100+0000 7efdb674d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efdb00c4720 0x7efdb0068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.100+0000 7efdb674d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7efdb00cadb0 0x7efdb00cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.100+0000 7efdb674d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb00cbf90 con 0x7efdb00c4720 2024-01-27T01:06:44.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.100+0000 7efdb674d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb00c97f0 con 0x7efdb00cadb0 2024-01-27T01:06:44.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.100+0000 7efdb674d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb00cb730 con 0x7efdb01596e0 2024-01-27T01:06:44.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.100+0000 7efdaf7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efdb00c4720 0x7efdb0068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.100+0000 7efdaf7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efdb00c4720 0x7efdb0068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41820/0 (socket says 172.21.15.46:41820) 2024-01-27T01:06:44.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.100+0000 7efdaf7fe700 1 -- 172.21.15.46:0/3156907404 learned_addr learned my addr 172.21.15.46:0/3156907404 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:44.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.100+0000 7efdaffff700 1 --2- 172.21.15.46:0/3156907404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efdb01596e0 0x7efdb0159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.100+0000 7efdb4cea700 1 --2- 172.21.15.46:0/3156907404 >> v2:172.21.15.106:3300/0 conn(0x7efdb00cadb0 0x7efdb00cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.101+0000 7efdaf7fe700 1 -- 172.21.15.46:0/3156907404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efdb01596e0 msgr2=0x7efdb0159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:44.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.101+0000 7efdaf7fe700 1 --2- 172.21.15.46:0/3156907404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efdb01596e0 0x7efdb0159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.101+0000 7efdaf7fe700 1 -- 172.21.15.46:0/3156907404 >> v2:172.21.15.106:3300/0 conn(0x7efdb00cadb0 msgr2=0x7efdb00cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:44.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.101+0000 7efdaf7fe700 1 --2- 172.21.15.46:0/3156907404 >> v2:172.21.15.106:3300/0 conn(0x7efdb00cadb0 0x7efdb00cb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.101+0000 7efdaf7fe700 1 -- 172.21.15.46:0/3156907404 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb00c8c20 con 0x7efdb00c4720 2024-01-27T01:06:44.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.101+0000 7efdb4cea700 1 --2- 172.21.15.46:0/3156907404 >> v2:172.21.15.106:3300/0 conn(0x7efdb00cadb0 0x7efdb00cb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:44.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.101+0000 7efdaf7fe700 1 --2- 172.21.15.46:0/3156907404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdb00c4720 0x7efdb0068a40 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7efdb006fc00 tx=0x7efda4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e4482228e664a4a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:44.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.101+0000 7efdaeffd700 1 -- 172.21.15.46:0/3156907404 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda4009e70 con 0x7efdb00c4720 2024-01-27T01:06:44.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.101+0000 7efdaeffd700 1 -- 172.21.15.46:0/3156907404 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda400b860 con 0x7efdb00c4720 2024-01-27T01:06:44.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.101+0000 7efdaeffd700 1 -- 172.21.15.46:0/3156907404 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda400bc00 con 0x7efdb00c4720 2024-01-27T01:06:44.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.102+0000 7efdb674d700 1 -- 172.21.15.46:0/3156907404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdb00c4720 msgr2=0x7efdb0068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:44.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.102+0000 7efdb674d700 1 --2- 172.21.15.46:0/3156907404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdb00c4720 0x7efdb0068a40 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7efdb006fc00 tx=0x7efda4009250 comp rx=0 tx=0).stop 2024-01-27T01:06:44.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.102+0000 7efdb674d700 1 -- 172.21.15.46:0/3156907404 shutdown_connections 2024-01-27T01:06:44.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.102+0000 7efdb674d700 1 --2- 172.21.15.46:0/3156907404 >> v2:172.21.15.106:3300/0 conn(0x7efdb00cadb0 0x7efdb00cb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.102+0000 7efdb674d700 1 --2- 172.21.15.46:0/3156907404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efdb01596e0 0x7efdb0159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.102+0000 7efdb674d700 1 --2- 172.21.15.46:0/3156907404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdb00c4720 0x7efdb0068a40 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.102+0000 7efdb674d700 1 -- 172.21.15.46:0/3156907404 >> 172.21.15.46:0/3156907404 conn(0x7efdb0097820 msgr2=0x7efdb0068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:44.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.102+0000 7efdb674d700 1 -- 172.21.15.46:0/3156907404 shutdown_connections 2024-01-27T01:06:44.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.102+0000 7efdb674d700 1 -- 172.21.15.46:0/3156907404 wait complete. 2024-01-27T01:06:44.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.103+0000 7efdb674d700 1 Processor -- start 2024-01-27T01:06:44.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.103+0000 7efdb674d700 1 -- start start 2024-01-27T01:06:44.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.104+0000 7efdb674d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdb00c4720 0x7efdb00b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.104+0000 7efdb674d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdb00cadb0 0x7efdb00b2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.104+0000 7efdb674d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efdb01596e0 0x7efdb0150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.104+0000 7efdb674d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb00b23f0 con 0x7efdb00c4720 2024-01-27T01:06:44.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.104+0000 7efdb674d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb00b17e0 con 0x7efdb00cadb0 2024-01-27T01:06:44.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.104+0000 7efdb674d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb00b1ac0 con 0x7efdb01596e0 2024-01-27T01:06:44.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.104+0000 7efdaffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdb00c4720 0x7efdb00b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.104+0000 7efdaffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdb00c4720 0x7efdb00b3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41828/0 (socket says 172.21.15.46:41828) 2024-01-27T01:06:44.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.104+0000 7efdaffff700 1 -- 172.21.15.46:0/1348824359 learned_addr learned my addr 172.21.15.46:0/1348824359 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:44.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.104+0000 7efdaf7fe700 1 --2- 172.21.15.46:0/1348824359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdb00cadb0 0x7efdb00b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.104+0000 7efdb4cea700 1 --2- 172.21.15.46:0/1348824359 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efdb01596e0 0x7efdb0150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.104+0000 7efdaffff700 1 -- 172.21.15.46:0/1348824359 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efdb01596e0 msgr2=0x7efdb0150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:44.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.104+0000 7efdaffff700 1 --2- 172.21.15.46:0/1348824359 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efdb01596e0 0x7efdb0150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.104+0000 7efdaffff700 1 -- 172.21.15.46:0/1348824359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdb00cadb0 msgr2=0x7efdb00b2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:44.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.104+0000 7efdaffff700 1 --2- 172.21.15.46:0/1348824359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdb00cadb0 0x7efdb00b2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.104+0000 7efdaffff700 1 -- 172.21.15.46:0/1348824359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda40094f0 con 0x7efdb00c4720 2024-01-27T01:06:44.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.105+0000 7efdaffff700 1 --2- 172.21.15.46:0/1348824359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdb00c4720 0x7efdb00b3ef0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7efda000e930 tx=0x7efda00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:44.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.105+0000 7efd97fff700 1 -- 172.21.15.46:0/1348824359 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda0013680 con 0x7efdb00c4720 2024-01-27T01:06:44.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.105+0000 7efd97fff700 1 -- 172.21.15.46:0/1348824359 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda00137e0 con 0x7efdb00c4720 2024-01-27T01:06:44.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.105+0000 7efdb674d700 1 -- 172.21.15.46:0/1348824359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb00b1da0 con 0x7efdb00c4720 2024-01-27T01:06:44.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.105+0000 7efd97fff700 1 -- 172.21.15.46:0/1348824359 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda0013b80 con 0x7efdb00c4720 2024-01-27T01:06:44.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.105+0000 7efdb674d700 1 -- 172.21.15.46:0/1348824359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb0150db0 con 0x7efdb00c4720 2024-01-27T01:06:44.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.106+0000 7efdb674d700 1 -- 172.21.15.46:0/1348824359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb004c710 con 0x7efdb00c4720 2024-01-27T01:06:44.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.106+0000 7efd97fff700 1 -- 172.21.15.46:0/1348824359 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7efda000c460 con 0x7efdb00c4720 2024-01-27T01:06:44.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.107+0000 7efd97fff700 1 --2- 172.21.15.46:0/1348824359 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efd9c066f30 0x7efd9c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.107+0000 7efd97fff700 1 -- 172.21.15.46:0/1348824359 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7efda0084d30 con 0x7efdb00c4720 2024-01-27T01:06:44.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.107+0000 7efdaf7fe700 1 --2- 172.21.15.46:0/1348824359 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efd9c066f30 0x7efd9c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.108+0000 7efdaf7fe700 1 --2- 172.21.15.46:0/1348824359 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efd9c066f30 0x7efd9c0693e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efdb006fc00 tx=0x7efda4017000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:44.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.111+0000 7efd97fff700 1 -- 172.21.15.46:0/1348824359 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efda0051910 con 0x7efdb00c4720 2024-01-27T01:06:44.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.365+0000 7efdb674d700 1 -- 172.21.15.46:0/1348824359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdb00b26d0 con 0x7efdb00c4720 2024-01-27T01:06:44.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.366+0000 7efd97fff700 1 -- 172.21.15.46:0/1348824359 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5697 (secure 0 0 0) 0x7efda0054f30 con 0x7efdb00c4720 2024-01-27T01:06:44.365 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:44.365 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":122,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/1999300009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1999300009},{"type":"v1","addr":"172.21.15.106:6835","nonce":1999300009}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":110},{"gid":10486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/4174263148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4174263148},{"type":"v1","addr":"172.21.15.106:6833","nonce":4174263148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:06:26.286041+0000","modified":"2024-01-27T01:06:29.873402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10866},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10866":{"gid":10866,"name":"a","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2483163449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2483163449},{"type":"v1","addr":"172.21.15.46:6835","nonce":2483163449}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:06:40.353248+0000","modified":"2024-01-27T01:06:43.938885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10854":{"gid":10854,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":8,"addr":"172.21.15.46:6833/1263484152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1263484152},{"type":"v1","addr":"172.21.15.46:6833","nonce":1263484152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-27T01:06:44.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.368+0000 7efdb674d700 1 -- 172.21.15.46:0/1348824359 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efd9c066f30 msgr2=0x7efd9c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:44.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.368+0000 7efdb674d700 1 --2- 172.21.15.46:0/1348824359 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efd9c066f30 0x7efd9c0693e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efdb006fc00 tx=0x7efda4017000 comp rx=0 tx=0).stop 2024-01-27T01:06:44.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.368+0000 7efdb674d700 1 -- 172.21.15.46:0/1348824359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdb00c4720 msgr2=0x7efdb00b3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:44.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.368+0000 7efdb674d700 1 --2- 172.21.15.46:0/1348824359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdb00c4720 0x7efdb00b3ef0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7efda000e930 tx=0x7efda00091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.369+0000 7efdb674d700 1 -- 172.21.15.46:0/1348824359 shutdown_connections 2024-01-27T01:06:44.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.369+0000 7efdb674d700 1 --2- 172.21.15.46:0/1348824359 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efd9c066f30 0x7efd9c0693e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.369+0000 7efdb674d700 1 --2- 172.21.15.46:0/1348824359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdb00c4720 0x7efdb00b3ef0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.369+0000 7efdb674d700 1 --2- 172.21.15.46:0/1348824359 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efdb01596e0 0x7efdb0150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.369+0000 7efdb674d700 1 --2- 172.21.15.46:0/1348824359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdb00cadb0 0x7efdb00b2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.369+0000 7efdb674d700 1 -- 172.21.15.46:0/1348824359 >> 172.21.15.46:0/1348824359 conn(0x7efdb0097820 msgr2=0x7efdb006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:44.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.369+0000 7efdb674d700 1 -- 172.21.15.46:0/1348824359 shutdown_connections 2024-01-27T01:06:44.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.369+0000 7efdb674d700 1 -- 172.21.15.46:0/1348824359 wait complete. 2024-01-27T01:06:44.369 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 122 2024-01-27T01:06:44.381 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:06:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.514+0000 7fd922b5d700 1 Processor -- start 2024-01-27T01:06:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.514+0000 7fd922b5d700 1 -- start start 2024-01-27T01:06:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.515+0000 7fd922b5d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd91c0b0280 0x7fd91c0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.515+0000 7fd922b5d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91c0b0760 con 0x7fd91c1561d0 2024-01-27T01:06:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.515+0000 7fd922b5d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91c0b0a40 con 0x7fd91c06cbb0 2024-01-27T01:06:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.515+0000 7fd922b5d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd91c0b0d20 con 0x7fd91c0b0280 2024-01-27T01:06:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.515+0000 7fd9208f9700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd91c1561d0 0x7fd91c06c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45422/0 (socket says 172.21.15.46:45422) 2024-01-27T01:06:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.515+0000 7fd9208f9700 1 -- 172.21.15.46:0/155869055 learned_addr learned my addr 172.21.15.46:0/155869055 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.516+0000 7fd9210fa700 1 --2- 172.21.15.46:0/155869055 >> v2:172.21.15.106:3300/0 conn(0x7fd91c0b0280 0x7fd91c0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.516+0000 7fd91b7fe700 1 -- 172.21.15.46:0/155869055 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585839394 0 0) 0x7fd91c0b0760 con 0x7fd91c1561d0 2024-01-27T01:06:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.516+0000 7fd91b7fe700 1 -- 172.21.15.46:0/155869055 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd90c003570 con 0x7fd91c1561d0 2024-01-27T01:06:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.516+0000 7fd9210fa700 1 -- 172.21.15.46:0/155869055 >> v1:172.21.15.106:6789/0 conn(0x7fd91c06cbb0 legacy=0x7fd91c0b1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.516+0000 7fd9210fa700 1 -- 172.21.15.46:0/155869055 >> v1:172.21.15.46:6789/0 conn(0x7fd91c1561d0 legacy=0x7fd91c06c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.516+0000 7fd9210fa700 1 -- 172.21.15.46:0/155869055 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91c0b1000 con 0x7fd91c0b0280 2024-01-27T01:06:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.516+0000 7fd9210fa700 1 --2- 172.21.15.46:0/155869055 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd91c0b0280 0x7fd91c0ca5f0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fd914009e60 tx=0x7fd914009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a64e620d2ed7b689 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.516+0000 7fd91b7fe700 1 -- 172.21.15.46:0/155869055 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91400e070 con 0x7fd91c0b0280 2024-01-27T01:06:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.516+0000 7fd91b7fe700 1 -- 172.21.15.46:0/155869055 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91400bb60 con 0x7fd91c0b0280 2024-01-27T01:06:44.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.516+0000 7fd91b7fe700 1 -- 172.21.15.46:0/155869055 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd914012430 con 0x7fd91c0b0280 2024-01-27T01:06:44.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.517+0000 7fd922b5d700 1 -- 172.21.15.46:0/155869055 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd91c0b0280 msgr2=0x7fd91c0ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:44.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.517+0000 7fd922b5d700 1 --2- 172.21.15.46:0/155869055 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd91c0b0280 0x7fd91c0ca5f0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fd914009e60 tx=0x7fd914009780 comp rx=0 tx=0).stop 2024-01-27T01:06:44.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.517+0000 7fd922b5d700 1 -- 172.21.15.46:0/155869055 shutdown_connections 2024-01-27T01:06:44.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.517+0000 7fd922b5d700 1 --2- 172.21.15.46:0/155869055 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd91c0b0280 0x7fd91c0ca5f0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.517+0000 7fd922b5d700 1 -- 172.21.15.46:0/155869055 >> 172.21.15.46:0/155869055 conn(0x7fd91c06b630 msgr2=0x7fd91c0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:44.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.517+0000 7fd922b5d700 1 -- 172.21.15.46:0/155869055 shutdown_connections 2024-01-27T01:06:44.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.517+0000 7fd922b5d700 1 -- 172.21.15.46:0/155869055 wait complete. 2024-01-27T01:06:44.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.518+0000 7fd922b5d700 1 Processor -- start 2024-01-27T01:06:44.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.518+0000 7fd922b5d700 1 -- start start 2024-01-27T01:06:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.519+0000 7fd922b5d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd91c06cbb0 0x7fd91c14dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.519+0000 7fd922b5d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd91c0b0280 0x7fd91c161840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.519+0000 7fd922b5d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd91c1561d0 0x7fd91c152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.519+0000 7fd922b5d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c169ff0 con 0x7fd91c1561d0 2024-01-27T01:06:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.519+0000 7fd922b5d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c16a2d0 con 0x7fd91c06cbb0 2024-01-27T01:06:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.519+0000 7fd922b5d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c16a5b0 con 0x7fd91c0b0280 2024-01-27T01:06:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.519+0000 7fd9210fa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd91c1561d0 0x7fd91c152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.519+0000 7fd9210fa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd91c1561d0 0x7fd91c152c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41830/0 (socket says 172.21.15.46:41830) 2024-01-27T01:06:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.519+0000 7fd9210fa700 1 -- 172.21.15.46:0/1504615454 learned_addr learned my addr 172.21.15.46:0/1504615454 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.519+0000 7fd9208f9700 1 --2- 172.21.15.46:0/1504615454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd91c06cbb0 0x7fd91c14dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.520+0000 7fd91bfff700 1 --2- 172.21.15.46:0/1504615454 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd91c0b0280 0x7fd91c161840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.520+0000 7fd9210fa700 1 -- 172.21.15.46:0/1504615454 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd91c0b0280 msgr2=0x7fd91c161840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.520+0000 7fd9210fa700 1 --2- 172.21.15.46:0/1504615454 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd91c0b0280 0x7fd91c161840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.520+0000 7fd9210fa700 1 -- 172.21.15.46:0/1504615454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd91c06cbb0 msgr2=0x7fd91c14dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.520+0000 7fd9210fa700 1 --2- 172.21.15.46:0/1504615454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd91c06cbb0 0x7fd91c14dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.520+0000 7fd9210fa700 1 -- 172.21.15.46:0/1504615454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd914009a20 con 0x7fd91c1561d0 2024-01-27T01:06:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.520+0000 7fd9210fa700 1 --2- 172.21.15.46:0/1504615454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd91c1561d0 0x7fd91c152c20 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fd91c0ce590 tx=0x7fd914019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.520+0000 7fd918ff9700 1 -- 172.21.15.46:0/1504615454 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9140032b0 con 0x7fd91c1561d0 2024-01-27T01:06:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.520+0000 7fd918ff9700 1 -- 172.21.15.46:0/1504615454 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9140263f0 con 0x7fd91c1561d0 2024-01-27T01:06:44.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.520+0000 7fd922b5d700 1 -- 172.21.15.46:0/1504615454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd91c16a890 con 0x7fd91c1561d0 2024-01-27T01:06:44.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.520+0000 7fd918ff9700 1 -- 172.21.15.46:0/1504615454 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9140267b0 con 0x7fd91c1561d0 2024-01-27T01:06:44.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.520+0000 7fd922b5d700 1 -- 172.21.15.46:0/1504615454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91c169ba0 con 0x7fd91c1561d0 2024-01-27T01:06:44.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.521+0000 7fd922b5d700 1 -- 172.21.15.46:0/1504615454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd91c04c710 con 0x7fd91c1561d0 2024-01-27T01:06:44.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.522+0000 7fd918ff9700 1 -- 172.21.15.46:0/1504615454 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd914026ba0 con 0x7fd91c1561d0 2024-01-27T01:06:44.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.522+0000 7fd918ff9700 1 --2- 172.21.15.46:0/1504615454 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd900066f30 0x7fd9000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.522+0000 7fd918ff9700 1 -- 172.21.15.46:0/1504615454 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7fd914086b20 con 0x7fd91c1561d0 2024-01-27T01:06:44.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.523+0000 7fd9208f9700 1 --2- 172.21.15.46:0/1504615454 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd900066f30 0x7fd9000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.523+0000 7fd9208f9700 1 --2- 172.21.15.46:0/1504615454 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd900066f30 0x7fd9000693e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd910007fd0 tx=0x7fd910009040 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:44.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.527+0000 7fd918ff9700 1 -- 172.21.15.46:0/1504615454 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd914053660 con 0x7fd91c1561d0 2024-01-27T01:06:44.693 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:06:44.696 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:06:44.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.736+0000 7fd922b5d700 1 -- 172.21.15.46:0/1504615454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd91c0b12e0 con 0x7fd91c1561d0 2024-01-27T01:06:44.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.737+0000 7fd918ff9700 1 -- 172.21.15.46:0/1504615454 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v227) v1 ==== 74+0+19557 (secure 0 0 0) 0x7fd914056c80 con 0x7fd91c1561d0 2024-01-27T01:06:44.736 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:44.736 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":227,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:06:41.916724+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:06:24.798791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-01-27T01:06:25.249215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:06:38.156934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:06:39.330686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:06:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.739+0000 7fd922b5d700 1 -- 172.21.15.46:0/1504615454 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd900066f30 msgr2=0x7fd9000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.739+0000 7fd922b5d700 1 --2- 172.21.15.46:0/1504615454 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd900066f30 0x7fd9000693e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd910007fd0 tx=0x7fd910009040 comp rx=0 tx=0).stop 2024-01-27T01:06:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.739+0000 7fd922b5d700 1 -- 172.21.15.46:0/1504615454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd91c1561d0 msgr2=0x7fd91c152c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.739+0000 7fd922b5d700 1 --2- 172.21.15.46:0/1504615454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd91c1561d0 0x7fd91c152c20 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fd91c0ce590 tx=0x7fd914019000 comp rx=0 tx=0).stop 2024-01-27T01:06:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.739+0000 7fd922b5d700 1 -- 172.21.15.46:0/1504615454 shutdown_connections 2024-01-27T01:06:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.739+0000 7fd922b5d700 1 --2- 172.21.15.46:0/1504615454 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd900066f30 0x7fd9000693e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.739+0000 7fd922b5d700 1 --2- 172.21.15.46:0/1504615454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd91c1561d0 0x7fd91c152c20 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.739+0000 7fd922b5d700 1 --2- 172.21.15.46:0/1504615454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd91c06cbb0 0x7fd91c14dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.739+0000 7fd922b5d700 1 --2- 172.21.15.46:0/1504615454 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd91c0b0280 0x7fd91c161840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.739+0000 7fd922b5d700 1 -- 172.21.15.46:0/1504615454 >> 172.21.15.46:0/1504615454 conn(0x7fd91c06b630 msgr2=0x7fd91c0aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.739+0000 7fd922b5d700 1 -- 172.21.15.46:0/1504615454 shutdown_connections 2024-01-27T01:06:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.739+0000 7fd922b5d700 1 -- 172.21.15.46:0/1504615454 wait complete. 2024-01-27T01:06:44.752 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-27T01:06:44.752 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:06:44.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.928+0000 7f1b660d2700 1 Processor -- start 2024-01-27T01:06:44.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.928+0000 7f1b660d2700 1 -- start start 2024-01-27T01:06:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.929+0000 7f1b660d2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1b601596e0 0x7f1b60159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.929+0000 7f1b660d2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b600c4720 0x7f1b60068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.929+0000 7f1b660d2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b600c97f0 con 0x7f1b600cadb0 2024-01-27T01:06:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.929+0000 7f1b660d2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b600cbf90 con 0x7f1b601596e0 2024-01-27T01:06:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.929+0000 7f1b660d2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b600c8c20 con 0x7f1b600c4720 2024-01-27T01:06:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.929+0000 7f1b5f7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1b601596e0 0x7f1b60159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.929+0000 7f1b5ffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1b600cadb0 0x7f1b600c9c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45428/0 (socket says 172.21.15.46:45428) 2024-01-27T01:06:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.929+0000 7f1b5ffff700 1 -- 172.21.15.46:0/1029701827 learned_addr learned my addr 172.21.15.46:0/1029701827 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.929+0000 7f1b5f7fe700 1 -- 172.21.15.46:0/1029701827 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b600c4720 msgr2=0x7f1b60068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.929+0000 7f1b5effd700 1 --2- 172.21.15.46:0/1029701827 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b600c4720 0x7f1b60068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.929+0000 7f1b5f7fe700 1 --2- 172.21.15.46:0/1029701827 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b600c4720 0x7f1b60068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.929+0000 7f1b5f7fe700 1 -- 172.21.15.46:0/1029701827 >> v1:172.21.15.46:6789/0 conn(0x7f1b600cadb0 legacy=0x7f1b600c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.929+0000 7f1b5e7fc700 1 -- 172.21.15.46:0/1029701827 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484358151 0 0) 0x7f1b600c97f0 con 0x7f1b600cadb0 2024-01-27T01:06:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.929+0000 7f1b5f7fe700 1 -- 172.21.15.46:0/1029701827 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b600c8ec0 con 0x7f1b601596e0 2024-01-27T01:06:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.929+0000 7f1b5effd700 1 --2- 172.21.15.46:0/1029701827 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b600c4720 0x7f1b60068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:06:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.930+0000 7f1b5f7fe700 1 --2- 172.21.15.46:0/1029701827 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b601596e0 0x7f1b60159ab0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f1b500098e0 tx=0x7f1b500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2fd60b805c0ffa server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.930+0000 7f1b5e7fc700 1 -- 172.21.15.46:0/1029701827 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b50009e80 con 0x7f1b601596e0 2024-01-27T01:06:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.930+0000 7f1b5e7fc700 1 -- 172.21.15.46:0/1029701827 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5000b860 con 0x7f1b601596e0 2024-01-27T01:06:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.930+0000 7f1b5e7fc700 1 -- 172.21.15.46:0/1029701827 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5000bbe0 con 0x7f1b601596e0 2024-01-27T01:06:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.930+0000 7f1b660d2700 1 -- 172.21.15.46:0/1029701827 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b601596e0 msgr2=0x7f1b60159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.930+0000 7f1b660d2700 1 --2- 172.21.15.46:0/1029701827 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b601596e0 0x7f1b60159ab0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f1b500098e0 tx=0x7f1b500091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.930+0000 7f1b660d2700 1 -- 172.21.15.46:0/1029701827 shutdown_connections 2024-01-27T01:06:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.930+0000 7f1b660d2700 1 --2- 172.21.15.46:0/1029701827 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b601596e0 0x7f1b60159ab0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.930+0000 7f1b660d2700 1 --2- 172.21.15.46:0/1029701827 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b600c4720 0x7f1b60068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.930+0000 7f1b660d2700 1 -- 172.21.15.46:0/1029701827 >> 172.21.15.46:0/1029701827 conn(0x7f1b60097820 msgr2=0x7f1b60068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.930+0000 7f1b660d2700 1 -- 172.21.15.46:0/1029701827 shutdown_connections 2024-01-27T01:06:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.930+0000 7f1b660d2700 1 -- 172.21.15.46:0/1029701827 wait complete. 2024-01-27T01:06:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.932+0000 7f1b660d2700 1 Processor -- start 2024-01-27T01:06:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.932+0000 7f1b660d2700 1 -- start start 2024-01-27T01:06:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b660d2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b600c4720 0x7f1b60152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b660d2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b600cadb0 0x7f1b600c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b660d2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b601596e0 0x7f1b6015e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b660d2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b601531a0 con 0x7f1b601596e0 2024-01-27T01:06:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b660d2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b600c7bd0 con 0x7f1b600c4720 2024-01-27T01:06:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b660d2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b600c7950 con 0x7f1b600cadb0 2024-01-27T01:06:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b5ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b601596e0 0x7f1b6015e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b5ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b601596e0 0x7f1b6015e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41848/0 (socket says 172.21.15.46:41848) 2024-01-27T01:06:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b5ffff700 1 -- 172.21.15.46:0/2002056443 learned_addr learned my addr 172.21.15.46:0/2002056443 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b5effd700 1 --2- 172.21.15.46:0/2002056443 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b600cadb0 0x7f1b600c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b5f7fe700 1 --2- 172.21.15.46:0/2002056443 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b600c4720 0x7f1b60152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b5ffff700 1 -- 172.21.15.46:0/2002056443 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b600cadb0 msgr2=0x7f1b600c7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b5ffff700 1 --2- 172.21.15.46:0/2002056443 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b600cadb0 0x7f1b600c7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b5ffff700 1 -- 172.21.15.46:0/2002056443 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b600c4720 msgr2=0x7f1b60152c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b5ffff700 1 --2- 172.21.15.46:0/2002056443 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b600c4720 0x7f1b60152c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b5ffff700 1 -- 172.21.15.46:0/2002056443 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b50009420 con 0x7f1b601596e0 2024-01-27T01:06:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b5f7fe700 1 --2- 172.21.15.46:0/2002056443 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b600c4720 0x7f1b60152c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b5ffff700 1 --2- 172.21.15.46:0/2002056443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b601596e0 0x7f1b6015e3f0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800cc90 tx=0x7f1b58001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.933+0000 7f1b477fe700 1 -- 172.21.15.46:0/2002056443 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5800a960 con 0x7f1b601596e0 2024-01-27T01:06:44.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.934+0000 7f1b477fe700 1 -- 172.21.15.46:0/2002056443 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5800aac0 con 0x7f1b601596e0 2024-01-27T01:06:44.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.934+0000 7f1b477fe700 1 -- 172.21.15.46:0/2002056443 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5800ae80 con 0x7f1b601596e0 2024-01-27T01:06:44.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.934+0000 7f1b660d2700 1 -- 172.21.15.46:0/2002056443 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6016ab60 con 0x7f1b601596e0 2024-01-27T01:06:44.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.934+0000 7f1b660d2700 1 -- 172.21.15.46:0/2002056443 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6016a080 con 0x7f1b601596e0 2024-01-27T01:06:44.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.935+0000 7f1b660d2700 1 -- 172.21.15.46:0/2002056443 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b6004c710 con 0x7f1b601596e0 2024-01-27T01:06:44.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.935+0000 7f1b477fe700 1 -- 172.21.15.46:0/2002056443 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1b58007300 con 0x7f1b601596e0 2024-01-27T01:06:44.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.936+0000 7f1b477fe700 1 --2- 172.21.15.46:0/2002056443 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b4c066f30 0x7f1b4c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:44.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.936+0000 7f1b477fe700 1 -- 172.21.15.46:0/2002056443 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f1b58084f40 con 0x7f1b601596e0 2024-01-27T01:06:44.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.936+0000 7f1b5f7fe700 1 --2- 172.21.15.46:0/2002056443 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b4c066f30 0x7f1b4c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:44.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.936+0000 7f1b5f7fe700 1 --2- 172.21.15.46:0/2002056443 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b4c066f30 0x7f1b4c0693e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1b500091e0 tx=0x7f1b50017000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:44.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:44.940+0000 7f1b477fe700 1 -- 172.21.15.46:0/2002056443 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1b58051ba0 con 0x7f1b601596e0 2024-01-27T01:06:45.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.139+0000 7f1b660d2700 1 -- 172.21.15.46:0/2002056443 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1b600c91a0 con 0x7f1b4c066f30 2024-01-27T01:06:45.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.153+0000 7f1b477fe700 1 -- 172.21.15.46:0/2002056443 <== mgr.9101 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+517877 (secure 0 0 0) 0x7f1b600c91a0 con 0x7f1b4c066f30 2024-01-27T01:06:45.160 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:45.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.158+0000 7f1b660d2700 1 -- 172.21.15.46:0/2002056443 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b4c066f30 msgr2=0x7f1b4c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:45.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.158+0000 7f1b660d2700 1 --2- 172.21.15.46:0/2002056443 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b4c066f30 0x7f1b4c0693e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1b500091e0 tx=0x7f1b50017000 comp rx=0 tx=0).stop 2024-01-27T01:06:45.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.158+0000 7f1b660d2700 1 -- 172.21.15.46:0/2002056443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b601596e0 msgr2=0x7f1b6015e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:45.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.158+0000 7f1b660d2700 1 --2- 172.21.15.46:0/2002056443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b601596e0 0x7f1b6015e3f0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800cc90 tx=0x7f1b58001a80 comp rx=0 tx=0).stop 2024-01-27T01:06:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.158+0000 7f1b660d2700 1 -- 172.21.15.46:0/2002056443 shutdown_connections 2024-01-27T01:06:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.158+0000 7f1b660d2700 1 --2- 172.21.15.46:0/2002056443 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b4c066f30 0x7f1b4c0693e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.158+0000 7f1b660d2700 1 --2- 172.21.15.46:0/2002056443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b601596e0 0x7f1b6015e3f0 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.158+0000 7f1b660d2700 1 --2- 172.21.15.46:0/2002056443 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b600c4720 0x7f1b60152c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.158+0000 7f1b660d2700 1 --2- 172.21.15.46:0/2002056443 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b600cadb0 0x7f1b600c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.158+0000 7f1b660d2700 1 -- 172.21.15.46:0/2002056443 >> 172.21.15.46:0/2002056443 conn(0x7f1b60097820 msgr2=0x7f1b60069a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.158+0000 7f1b660d2700 1 -- 172.21.15.46:0/2002056443 shutdown_connections 2024-01-27T01:06:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.158+0000 7f1b660d2700 1 -- 172.21.15.46:0/2002056443 wait complete. 2024-01-27T01:06:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:06:45.171 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-01-27T01:06:44.734501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593906,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":677,"num_write_kb":3477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8791,"ondisk_log_size":8791,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":202,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2467036,"kb_used_data":23196,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752507684,"statfs":{"total":773094113280,"available":770567868416,"internally_reserved":0,"allocated":23752704,"data_stored":13029817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":787,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.826236"},"pg_stats":[{"pgid":"1.0","version":"213'8750","reported_seq":9665,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.037979+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:06:28.037979+0000","last_peered":"2024-01-27T01:06:28.037979+0000","last_clean":"2024-01-27T01:06:28.037979+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:06:28.037979+0000","last_undegraded":"2024-01-27T01:06:28.037979+0000","last_fullsized":"2024-01-27T01:06:28.037979+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8750,"log_dups_size":0,"ondisk_log_size":8750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":630,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865902+0000","last_change":"2024-01-27T01:06:25.858609+0000","last_active":"2024-01-27T01:06:27.865902+0000","last_peered":"2024-01-27T01:06:27.865902+0000","last_clean":"2024-01-27T01:06:27.865902+0000","last_became_active":"2024-01-27T01:06:25.857638+0000","last_became_peered":"2024-01-27T01:06:25.857638+0000","last_unstale":"2024-01-27T01:06:27.865902+0000","last_undegraded":"2024-01-27T01:06:27.865902+0000","last_fullsized":"2024-01-27T01:06:27.865902+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:35:59.109571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665626+0000","last_change":"2024-01-27T01:06:26.873783+0000","last_active":"2024-01-27T01:06:28.665626+0000","last_peered":"2024-01-27T01:06:28.665626+0000","last_clean":"2024-01-27T01:06:28.665626+0000","last_became_active":"2024-01-27T01:06:26.873490+0000","last_became_peered":"2024-01-27T01:06:26.873490+0000","last_unstale":"2024-01-27T01:06:28.665626+0000","last_undegraded":"2024-01-27T01:06:28.665626+0000","last_fullsized":"2024-01-27T01:06:28.665626+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:32.691743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916373+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:55:49.427110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865572+0000","last_change":"2024-01-27T01:06:25.865235+0000","last_active":"2024-01-27T01:06:27.865572+0000","last_peered":"2024-01-27T01:06:27.865572+0000","last_clean":"2024-01-27T01:06:27.865572+0000","last_became_active":"2024-01-27T01:06:25.864872+0000","last_became_peered":"2024-01-27T01:06:25.864872+0000","last_unstale":"2024-01-27T01:06:27.865572+0000","last_undegraded":"2024-01-27T01:06:27.865572+0000","last_fullsized":"2024-01-27T01:06:27.865572+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:53:38.618410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.241058+0000","last_change":"2024-01-27T01:06:26.879306+0000","last_active":"2024-01-27T01:06:28.241058+0000","last_peered":"2024-01-27T01:06:28.241058+0000","last_clean":"2024-01-27T01:06:28.241058+0000","last_became_active":"2024-01-27T01:06:26.879071+0000","last_became_peered":"2024-01-27T01:06:26.879071+0000","last_unstale":"2024-01-27T01:06:28.241058+0000","last_undegraded":"2024-01-27T01:06:28.241058+0000","last_fullsized":"2024-01-27T01:06:28.241058+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:50:46.965359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910860+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:36:16.613510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666145+0000","last_change":"2024-01-27T01:06:25.856035+0000","last_active":"2024-01-27T01:06:28.666145+0000","last_peered":"2024-01-27T01:06:28.666145+0000","last_clean":"2024-01-27T01:06:28.666145+0000","last_became_active":"2024-01-27T01:06:25.855762+0000","last_became_peered":"2024-01-27T01:06:25.855762+0000","last_unstale":"2024-01-27T01:06:28.666145+0000","last_undegraded":"2024-01-27T01:06:28.666145+0000","last_fullsized":"2024-01-27T01:06:28.666145+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:51:44.378880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038160+0000","last_change":"2024-01-27T01:06:26.870626+0000","last_active":"2024-01-27T01:06:28.038160+0000","last_peered":"2024-01-27T01:06:28.038160+0000","last_clean":"2024-01-27T01:06:28.038160+0000","last_became_active":"2024-01-27T01:06:26.870457+0000","last_became_peered":"2024-01-27T01:06:26.870457+0000","last_unstale":"2024-01-27T01:06:28.038160+0000","last_undegraded":"2024-01-27T01:06:28.038160+0000","last_fullsized":"2024-01-27T01:06:28.038160+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:42.766155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913817+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:22.022827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865995+0000","last_change":"2024-01-27T01:06:25.855015+0000","last_active":"2024-01-27T01:06:27.865995+0000","last_peered":"2024-01-27T01:06:27.865995+0000","last_clean":"2024-01-27T01:06:27.865995+0000","last_became_active":"2024-01-27T01:06:25.854708+0000","last_became_peered":"2024-01-27T01:06:25.854708+0000","last_unstale":"2024-01-27T01:06:27.865995+0000","last_undegraded":"2024-01-27T01:06:27.865995+0000","last_fullsized":"2024-01-27T01:06:27.865995+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:03.971945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038100+0000","last_change":"2024-01-27T01:06:26.876182+0000","last_active":"2024-01-27T01:06:28.038100+0000","last_peered":"2024-01-27T01:06:28.038100+0000","last_clean":"2024-01-27T01:06:28.038100+0000","last_became_active":"2024-01-27T01:06:26.875894+0000","last_became_peered":"2024-01-27T01:06:26.875894+0000","last_unstale":"2024-01-27T01:06:28.038100+0000","last_undegraded":"2024-01-27T01:06:28.038100+0000","last_fullsized":"2024-01-27T01:06:28.038100+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:58.495062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.914162+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:57:07.250360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666039+0000","last_change":"2024-01-27T01:06:25.858142+0000","last_active":"2024-01-27T01:06:28.666039+0000","last_peered":"2024-01-27T01:06:28.666039+0000","last_clean":"2024-01-27T01:06:28.666039+0000","last_became_active":"2024-01-27T01:06:25.857867+0000","last_became_peered":"2024-01-27T01:06:25.857867+0000","last_unstale":"2024-01-27T01:06:28.666039+0000","last_undegraded":"2024-01-27T01:06:28.666039+0000","last_fullsized":"2024-01-27T01:06:28.666039+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:10:59.780192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666343+0000","last_change":"2024-01-27T01:06:26.880503+0000","last_active":"2024-01-27T01:06:28.666343+0000","last_peered":"2024-01-27T01:06:28.666343+0000","last_clean":"2024-01-27T01:06:28.666343+0000","last_became_active":"2024-01-27T01:06:26.880224+0000","last_became_peered":"2024-01-27T01:06:26.880224+0000","last_unstale":"2024-01-27T01:06:28.666343+0000","last_undegraded":"2024-01-27T01:06:28.666343+0000","last_fullsized":"2024-01-27T01:06:28.666343+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:27:33.886740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.911198+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:26:34.409044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665860+0000","last_change":"2024-01-27T01:06:25.859342+0000","last_active":"2024-01-27T01:06:28.665860+0000","last_peered":"2024-01-27T01:06:28.665860+0000","last_clean":"2024-01-27T01:06:28.665860+0000","last_became_active":"2024-01-27T01:06:25.859065+0000","last_became_peered":"2024-01-27T01:06:25.859065+0000","last_unstale":"2024-01-27T01:06:28.665860+0000","last_undegraded":"2024-01-27T01:06:28.665860+0000","last_fullsized":"2024-01-27T01:06:28.665860+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:00:54.297085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666399+0000","last_change":"2024-01-27T01:06:26.870597+0000","last_active":"2024-01-27T01:06:28.666399+0000","last_peered":"2024-01-27T01:06:28.666399+0000","last_clean":"2024-01-27T01:06:28.666399+0000","last_became_active":"2024-01-27T01:06:26.870334+0000","last_became_peered":"2024-01-27T01:06:26.870334+0000","last_unstale":"2024-01-27T01:06:28.666399+0000","last_undegraded":"2024-01-27T01:06:28.666399+0000","last_fullsized":"2024-01-27T01:06:28.666399+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:01:23.266072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913094+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:08:04.706593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355235+0000","last_change":"2024-01-27T01:06:25.858032+0000","last_active":"2024-01-27T01:06:28.355235+0000","last_peered":"2024-01-27T01:06:28.355235+0000","last_clean":"2024-01-27T01:06:28.355235+0000","last_became_active":"2024-01-27T01:06:25.857777+0000","last_became_peered":"2024-01-27T01:06:25.857777+0000","last_unstale":"2024-01-27T01:06:28.355235+0000","last_undegraded":"2024-01-27T01:06:28.355235+0000","last_fullsized":"2024-01-27T01:06:28.355235+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:09:08.458850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865801+0000","last_change":"2024-01-27T01:06:26.872190+0000","last_active":"2024-01-27T01:06:27.865801+0000","last_peered":"2024-01-27T01:06:27.865801+0000","last_clean":"2024-01-27T01:06:27.865801+0000","last_became_active":"2024-01-27T01:06:26.867718+0000","last_became_peered":"2024-01-27T01:06:26.867718+0000","last_unstale":"2024-01-27T01:06:27.865801+0000","last_undegraded":"2024-01-27T01:06:27.865801+0000","last_fullsized":"2024-01-27T01:06:27.865801+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:23.301132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.915265+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:55:49.427110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.6","version":"223'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:31.798965+0000","last_change":"2024-01-27T01:06:25.861717+0000","last_active":"2024-01-27T01:06:31.798965+0000","last_peered":"2024-01-27T01:06:31.798965+0000","last_clean":"2024-01-27T01:06:31.798965+0000","last_became_active":"2024-01-27T01:06:25.861177+0000","last_became_peered":"2024-01-27T01:06:25.861177+0000","last_unstale":"2024-01-27T01:06:31.798965+0000","last_undegraded":"2024-01-27T01:06:31.798965+0000","last_fullsized":"2024-01-27T01:06:31.798965+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:05.596045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240815+0000","last_change":"2024-01-27T01:06:26.878650+0000","last_active":"2024-01-27T01:06:28.240815+0000","last_peered":"2024-01-27T01:06:28.240815+0000","last_clean":"2024-01-27T01:06:28.240815+0000","last_became_active":"2024-01-27T01:06:26.877596+0000","last_became_peered":"2024-01-27T01:06:26.877596+0000","last_unstale":"2024-01-27T01:06:28.240815+0000","last_undegraded":"2024-01-27T01:06:28.240815+0000","last_fullsized":"2024-01-27T01:06:28.240815+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:42:48.942202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.914590+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:04.634853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.7","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866083+0000","last_change":"2024-01-27T01:06:25.857933+0000","last_active":"2024-01-27T01:06:27.866083+0000","last_peered":"2024-01-27T01:06:27.866083+0000","last_clean":"2024-01-27T01:06:27.866083+0000","last_became_active":"2024-01-27T01:06:25.857406+0000","last_became_peered":"2024-01-27T01:06:25.857406+0000","last_unstale":"2024-01-27T01:06:27.866083+0000","last_undegraded":"2024-01-27T01:06:27.866083+0000","last_fullsized":"2024-01-27T01:06:27.866083+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:20:20.239633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665632+0000","last_change":"2024-01-27T01:06:26.875222+0000","last_active":"2024-01-27T01:06:28.665632+0000","last_peered":"2024-01-27T01:06:28.665632+0000","last_clean":"2024-01-27T01:06:28.665632+0000","last_became_active":"2024-01-27T01:06:26.874903+0000","last_became_peered":"2024-01-27T01:06:26.874903+0000","last_unstale":"2024-01-27T01:06:28.665632+0000","last_undegraded":"2024-01-27T01:06:28.665632+0000","last_fullsized":"2024-01-27T01:06:28.665632+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:09:33.903445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916305+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:24.566246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665720+0000","last_change":"2024-01-27T01:06:25.853605+0000","last_active":"2024-01-27T01:06:28.665720+0000","last_peered":"2024-01-27T01:06:28.665720+0000","last_clean":"2024-01-27T01:06:28.665720+0000","last_became_active":"2024-01-27T01:06:25.853331+0000","last_became_peered":"2024-01-27T01:06:25.853331+0000","last_unstale":"2024-01-27T01:06:28.665720+0000","last_undegraded":"2024-01-27T01:06:28.665720+0000","last_fullsized":"2024-01-27T01:06:28.665720+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:09:08.458850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666459+0000","last_change":"2024-01-27T01:06:26.875536+0000","last_active":"2024-01-27T01:06:28.666459+0000","last_peered":"2024-01-27T01:06:28.666459+0000","last_clean":"2024-01-27T01:06:28.666459+0000","last_became_active":"2024-01-27T01:06:26.875267+0000","last_became_peered":"2024-01-27T01:06:26.875267+0000","last_unstale":"2024-01-27T01:06:28.666459+0000","last_undegraded":"2024-01-27T01:06:28.666459+0000","last_fullsized":"2024-01-27T01:06:28.666459+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:32.534622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916585+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:37:20.044223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355214+0000","last_change":"2024-01-27T01:06:25.862199+0000","last_active":"2024-01-27T01:06:28.355214+0000","last_peered":"2024-01-27T01:06:28.355214+0000","last_clean":"2024-01-27T01:06:28.355214+0000","last_became_active":"2024-01-27T01:06:25.861876+0000","last_became_peered":"2024-01-27T01:06:25.861876+0000","last_unstale":"2024-01-27T01:06:28.355214+0000","last_undegraded":"2024-01-27T01:06:28.355214+0000","last_fullsized":"2024-01-27T01:06:28.355214+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:00:54.297085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865614+0000","last_change":"2024-01-27T01:06:26.867570+0000","last_active":"2024-01-27T01:06:27.865614+0000","last_peered":"2024-01-27T01:06:27.865614+0000","last_clean":"2024-01-27T01:06:27.865614+0000","last_became_active":"2024-01-27T01:06:26.866517+0000","last_became_peered":"2024-01-27T01:06:26.866517+0000","last_unstale":"2024-01-27T01:06:27.865614+0000","last_undegraded":"2024-01-27T01:06:27.865614+0000","last_fullsized":"2024-01-27T01:06:27.865614+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:09:54.515702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910624+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:47:30.145789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866689+0000","last_change":"2024-01-27T01:06:25.851595+0000","last_active":"2024-01-27T01:06:27.866689+0000","last_peered":"2024-01-27T01:06:27.866689+0000","last_clean":"2024-01-27T01:06:27.866689+0000","last_became_active":"2024-01-27T01:06:25.851260+0000","last_became_peered":"2024-01-27T01:06:25.851260+0000","last_unstale":"2024-01-27T01:06:27.866689+0000","last_undegraded":"2024-01-27T01:06:27.866689+0000","last_fullsized":"2024-01-27T01:06:27.866689+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:56:53.179347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038203+0000","last_change":"2024-01-27T01:06:26.875858+0000","last_active":"2024-01-27T01:06:28.038203+0000","last_peered":"2024-01-27T01:06:28.038203+0000","last_clean":"2024-01-27T01:06:28.038203+0000","last_became_active":"2024-01-27T01:06:26.875564+0000","last_became_peered":"2024-01-27T01:06:26.875564+0000","last_unstale":"2024-01-27T01:06:28.038203+0000","last_undegraded":"2024-01-27T01:06:28.038203+0000","last_fullsized":"2024-01-27T01:06:28.038203+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:43.630831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913526+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:07:02.246428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.b","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866591+0000","last_change":"2024-01-27T01:06:25.860572+0000","last_active":"2024-01-27T01:06:27.866591+0000","last_peered":"2024-01-27T01:06:27.866591+0000","last_clean":"2024-01-27T01:06:27.866591+0000","last_became_active":"2024-01-27T01:06:25.860248+0000","last_became_peered":"2024-01-27T01:06:25.860248+0000","last_unstale":"2024-01-27T01:06:27.866591+0000","last_undegraded":"2024-01-27T01:06:27.866591+0000","last_fullsized":"2024-01-27T01:06:27.866591+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:01:25.741123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866157+0000","last_change":"2024-01-27T01:06:26.871346+0000","last_active":"2024-01-27T01:06:27.866157+0000","last_peered":"2024-01-27T01:06:27.866157+0000","last_clean":"2024-01-27T01:06:27.866157+0000","last_became_active":"2024-01-27T01:06:26.870999+0000","last_became_peered":"2024-01-27T01:06:26.870999+0000","last_unstale":"2024-01-27T01:06:27.866157+0000","last_undegraded":"2024-01-27T01:06:27.866157+0000","last_fullsized":"2024-01-27T01:06:27.866157+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:27.779157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.914309+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:31:18.366038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866703+0000","last_change":"2024-01-27T01:06:25.861650+0000","last_active":"2024-01-27T01:06:27.866703+0000","last_peered":"2024-01-27T01:06:27.866703+0000","last_clean":"2024-01-27T01:06:27.866703+0000","last_became_active":"2024-01-27T01:06:25.861342+0000","last_became_peered":"2024-01-27T01:06:25.861342+0000","last_unstale":"2024-01-27T01:06:27.866703+0000","last_undegraded":"2024-01-27T01:06:27.866703+0000","last_fullsized":"2024-01-27T01:06:27.866703+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:44:15.953217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665953+0000","last_change":"2024-01-27T01:06:26.876619+0000","last_active":"2024-01-27T01:06:28.665953+0000","last_peered":"2024-01-27T01:06:28.665953+0000","last_clean":"2024-01-27T01:06:28.665953+0000","last_became_active":"2024-01-27T01:06:26.875927+0000","last_became_peered":"2024-01-27T01:06:26.875927+0000","last_unstale":"2024-01-27T01:06:28.665953+0000","last_undegraded":"2024-01-27T01:06:28.665953+0000","last_fullsized":"2024-01-27T01:06:28.665953+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:02.000909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.911280+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:43:07.431223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.d","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.867031+0000","last_change":"2024-01-27T01:06:25.851720+0000","last_active":"2024-01-27T01:06:27.867031+0000","last_peered":"2024-01-27T01:06:27.867031+0000","last_clean":"2024-01-27T01:06:27.867031+0000","last_became_active":"2024-01-27T01:06:25.851416+0000","last_became_peered":"2024-01-27T01:06:25.851416+0000","last_unstale":"2024-01-27T01:06:27.867031+0000","last_undegraded":"2024-01-27T01:06:27.867031+0000","last_fullsized":"2024-01-27T01:06:27.867031+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:07.951814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666130+0000","last_change":"2024-01-27T01:06:26.870758+0000","last_active":"2024-01-27T01:06:28.666130+0000","last_peered":"2024-01-27T01:06:28.666130+0000","last_clean":"2024-01-27T01:06:28.666130+0000","last_became_active":"2024-01-27T01:06:26.870130+0000","last_became_peered":"2024-01-27T01:06:26.870130+0000","last_unstale":"2024-01-27T01:06:28.666130+0000","last_undegraded":"2024-01-27T01:06:28.666130+0000","last_fullsized":"2024-01-27T01:06:28.666130+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:36:31.625857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913563+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:49:22.695065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.e","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665974+0000","last_change":"2024-01-27T01:06:25.852245+0000","last_active":"2024-01-27T01:06:28.665974+0000","last_peered":"2024-01-27T01:06:28.665974+0000","last_clean":"2024-01-27T01:06:28.665974+0000","last_became_active":"2024-01-27T01:06:25.851982+0000","last_became_peered":"2024-01-27T01:06:25.851982+0000","last_unstale":"2024-01-27T01:06:28.665974+0000","last_undegraded":"2024-01-27T01:06:28.665974+0000","last_fullsized":"2024-01-27T01:06:28.665974+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:05.596045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665930+0000","last_change":"2024-01-27T01:06:26.879356+0000","last_active":"2024-01-27T01:06:28.665930+0000","last_peered":"2024-01-27T01:06:28.665930+0000","last_clean":"2024-01-27T01:06:28.665930+0000","last_became_active":"2024-01-27T01:06:26.878891+0000","last_became_peered":"2024-01-27T01:06:26.878891+0000","last_unstale":"2024-01-27T01:06:28.665930+0000","last_undegraded":"2024-01-27T01:06:28.665930+0000","last_fullsized":"2024-01-27T01:06:28.665930+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:22.011928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916563+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:01:36.332606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.f","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666321+0000","last_change":"2024-01-27T01:06:25.848900+0000","last_active":"2024-01-27T01:06:28.666321+0000","last_peered":"2024-01-27T01:06:28.666321+0000","last_clean":"2024-01-27T01:06:28.666321+0000","last_became_active":"2024-01-27T01:06:25.848483+0000","last_became_peered":"2024-01-27T01:06:25.848483+0000","last_unstale":"2024-01-27T01:06:28.666321+0000","last_undegraded":"2024-01-27T01:06:28.666321+0000","last_fullsized":"2024-01-27T01:06:28.666321+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:27:13.471892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665407+0000","last_change":"2024-01-27T01:06:26.875234+0000","last_active":"2024-01-27T01:06:28.665407+0000","last_peered":"2024-01-27T01:06:28.665407+0000","last_clean":"2024-01-27T01:06:28.665407+0000","last_became_active":"2024-01-27T01:06:26.874927+0000","last_became_peered":"2024-01-27T01:06:26.874927+0000","last_unstale":"2024-01-27T01:06:28.665407+0000","last_undegraded":"2024-01-27T01:06:28.665407+0000","last_fullsized":"2024-01-27T01:06:28.665407+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:10.390573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.911685+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:31.265591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.10","version":"222'8","reported_seq":19,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865571+0000","last_change":"2024-01-27T01:06:25.865239+0000","last_active":"2024-01-27T01:06:27.865571+0000","last_peered":"2024-01-27T01:06:27.865571+0000","last_clean":"2024-01-27T01:06:27.865571+0000","last_became_active":"2024-01-27T01:06:25.864943+0000","last_became_peered":"2024-01-27T01:06:25.864943+0000","last_unstale":"2024-01-27T01:06:27.865571+0000","last_undegraded":"2024-01-27T01:06:27.865571+0000","last_fullsized":"2024-01-27T01:06:27.865571+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:41:25.254751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038170+0000","last_change":"2024-01-27T01:06:26.867077+0000","last_active":"2024-01-27T01:06:28.038170+0000","last_peered":"2024-01-27T01:06:28.038170+0000","last_clean":"2024-01-27T01:06:28.038170+0000","last_became_active":"2024-01-27T01:06:26.866787+0000","last_became_peered":"2024-01-27T01:06:26.866787+0000","last_unstale":"2024-01-27T01:06:28.038170+0000","last_undegraded":"2024-01-27T01:06:28.038170+0000","last_fullsized":"2024-01-27T01:06:28.038170+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:02.000909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913156+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:56.697365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865574+0000","last_change":"2024-01-27T01:06:25.853597+0000","last_active":"2024-01-27T01:06:27.865574+0000","last_peered":"2024-01-27T01:06:27.865574+0000","last_clean":"2024-01-27T01:06:27.865574+0000","last_became_active":"2024-01-27T01:06:25.853257+0000","last_became_peered":"2024-01-27T01:06:25.853257+0000","last_unstale":"2024-01-27T01:06:27.865574+0000","last_undegraded":"2024-01-27T01:06:27.865574+0000","last_fullsized":"2024-01-27T01:06:27.865574+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:54:35.011796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865516+0000","last_change":"2024-01-27T01:06:26.866613+0000","last_active":"2024-01-27T01:06:27.865516+0000","last_peered":"2024-01-27T01:06:27.865516+0000","last_clean":"2024-01-27T01:06:27.865516+0000","last_became_active":"2024-01-27T01:06:26.866298+0000","last_became_peered":"2024-01-27T01:06:26.866298+0000","last_unstale":"2024-01-27T01:06:27.865516+0000","last_undegraded":"2024-01-27T01:06:27.865516+0000","last_fullsized":"2024-01-27T01:06:27.865516+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:27:03.934654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.917128+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:57:15.231912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240972+0000","last_change":"2024-01-27T01:06:25.854192+0000","last_active":"2024-01-27T01:06:28.240972+0000","last_peered":"2024-01-27T01:06:28.240972+0000","last_clean":"2024-01-27T01:06:28.240972+0000","last_became_active":"2024-01-27T01:06:25.853601+0000","last_became_peered":"2024-01-27T01:06:25.853601+0000","last_unstale":"2024-01-27T01:06:28.240972+0000","last_undegraded":"2024-01-27T01:06:28.240972+0000","last_fullsized":"2024-01-27T01:06:28.240972+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:42.218792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.241137+0000","last_change":"2024-01-27T01:06:26.862905+0000","last_active":"2024-01-27T01:06:28.241137+0000","last_peered":"2024-01-27T01:06:28.241137+0000","last_clean":"2024-01-27T01:06:28.241137+0000","last_became_active":"2024-01-27T01:06:26.862584+0000","last_became_peered":"2024-01-27T01:06:26.862584+0000","last_unstale":"2024-01-27T01:06:28.241137+0000","last_undegraded":"2024-01-27T01:06:28.241137+0000","last_fullsized":"2024-01-27T01:06:28.241137+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:27:14.653781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910329+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:19:00.298602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.13","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866863+0000","last_change":"2024-01-27T01:06:25.850061+0000","last_active":"2024-01-27T01:06:27.866863+0000","last_peered":"2024-01-27T01:06:27.866863+0000","last_clean":"2024-01-27T01:06:27.866863+0000","last_became_active":"2024-01-27T01:06:25.849763+0000","last_became_peered":"2024-01-27T01:06:25.849763+0000","last_unstale":"2024-01-27T01:06:27.866863+0000","last_undegraded":"2024-01-27T01:06:27.866863+0000","last_fullsized":"2024-01-27T01:06:27.866863+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:09:56.243562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866051+0000","last_change":"2024-01-27T01:06:26.873464+0000","last_active":"2024-01-27T01:06:27.866051+0000","last_peered":"2024-01-27T01:06:27.866051+0000","last_clean":"2024-01-27T01:06:27.866051+0000","last_became_active":"2024-01-27T01:06:26.873063+0000","last_became_peered":"2024-01-27T01:06:26.873063+0000","last_unstale":"2024-01-27T01:06:27.866051+0000","last_undegraded":"2024-01-27T01:06:27.866051+0000","last_fullsized":"2024-01-27T01:06:27.866051+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:33:32.535574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916917+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:36:16.613510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.14","version":"223'8","reported_seq":19,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:36.798321+0000","last_change":"2024-01-27T01:06:25.864160+0000","last_active":"2024-01-27T01:06:36.798321+0000","last_peered":"2024-01-27T01:06:36.798321+0000","last_clean":"2024-01-27T01:06:36.798321+0000","last_became_active":"2024-01-27T01:06:25.863867+0000","last_became_peered":"2024-01-27T01:06:25.863867+0000","last_unstale":"2024-01-27T01:06:36.798321+0000","last_undegraded":"2024-01-27T01:06:36.798321+0000","last_fullsized":"2024-01-27T01:06:36.798321+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:43.338484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2768,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866494+0000","last_change":"2024-01-27T01:06:26.879652+0000","last_active":"2024-01-27T01:06:27.866494+0000","last_peered":"2024-01-27T01:06:27.866494+0000","last_clean":"2024-01-27T01:06:27.866494+0000","last_became_active":"2024-01-27T01:06:26.879271+0000","last_became_peered":"2024-01-27T01:06:26.879271+0000","last_unstale":"2024-01-27T01:06:27.866494+0000","last_undegraded":"2024-01-27T01:06:27.866494+0000","last_fullsized":"2024-01-27T01:06:27.866494+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:57:23.456731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.914165+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:27:24.223960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.15","version":"222'3","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866134+0000","last_change":"2024-01-27T01:06:25.857771+0000","last_active":"2024-01-27T01:06:27.866134+0000","last_peered":"2024-01-27T01:06:27.866134+0000","last_clean":"2024-01-27T01:06:27.866134+0000","last_became_active":"2024-01-27T01:06:25.857445+0000","last_became_peered":"2024-01-27T01:06:25.857445+0000","last_unstale":"2024-01-27T01:06:27.866134+0000","last_undegraded":"2024-01-27T01:06:27.866134+0000","last_fullsized":"2024-01-27T01:06:27.866134+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:52.927866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865698+0000","last_change":"2024-01-27T01:06:26.876685+0000","last_active":"2024-01-27T01:06:27.865698+0000","last_peered":"2024-01-27T01:06:27.865698+0000","last_clean":"2024-01-27T01:06:27.865698+0000","last_became_active":"2024-01-27T01:06:26.872939+0000","last_became_peered":"2024-01-27T01:06:26.872939+0000","last_unstale":"2024-01-27T01:06:27.865698+0000","last_undegraded":"2024-01-27T01:06:27.865698+0000","last_fullsized":"2024-01-27T01:06:27.865698+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:56:48.730726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.915793+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:36:16.613510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038259+0000","last_change":"2024-01-27T01:06:25.860762+0000","last_active":"2024-01-27T01:06:28.038259+0000","last_peered":"2024-01-27T01:06:28.038259+0000","last_clean":"2024-01-27T01:06:28.038259+0000","last_became_active":"2024-01-27T01:06:25.860459+0000","last_became_peered":"2024-01-27T01:06:25.860459+0000","last_unstale":"2024-01-27T01:06:28.038259+0000","last_undegraded":"2024-01-27T01:06:28.038259+0000","last_fullsized":"2024-01-27T01:06:28.038259+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:31.687265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240842+0000","last_change":"2024-01-27T01:06:26.868859+0000","last_active":"2024-01-27T01:06:28.240842+0000","last_peered":"2024-01-27T01:06:28.240842+0000","last_clean":"2024-01-27T01:06:28.240842+0000","last_became_active":"2024-01-27T01:06:26.868623+0000","last_became_peered":"2024-01-27T01:06:26.868623+0000","last_unstale":"2024-01-27T01:06:28.240842+0000","last_undegraded":"2024-01-27T01:06:28.240842+0000","last_fullsized":"2024-01-27T01:06:28.240842+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:18:47.232067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.915065+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:24.566246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866791+0000","last_change":"2024-01-27T01:06:25.865314+0000","last_active":"2024-01-27T01:06:27.866791+0000","last_peered":"2024-01-27T01:06:27.866791+0000","last_clean":"2024-01-27T01:06:27.866791+0000","last_became_active":"2024-01-27T01:06:25.864940+0000","last_became_peered":"2024-01-27T01:06:25.864940+0000","last_unstale":"2024-01-27T01:06:27.866791+0000","last_undegraded":"2024-01-27T01:06:27.866791+0000","last_fullsized":"2024-01-27T01:06:27.866791+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:24.113986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355011+0000","last_change":"2024-01-27T01:06:26.870806+0000","last_active":"2024-01-27T01:06:28.355011+0000","last_peered":"2024-01-27T01:06:28.355011+0000","last_clean":"2024-01-27T01:06:28.355011+0000","last_became_active":"2024-01-27T01:06:26.870263+0000","last_became_peered":"2024-01-27T01:06:26.870263+0000","last_unstale":"2024-01-27T01:06:28.355011+0000","last_undegraded":"2024-01-27T01:06:28.355011+0000","last_fullsized":"2024-01-27T01:06:28.355011+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:26.214953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910328+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:24.032004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240953+0000","last_change":"2024-01-27T01:06:25.851888+0000","last_active":"2024-01-27T01:06:28.240953+0000","last_peered":"2024-01-27T01:06:28.240953+0000","last_clean":"2024-01-27T01:06:28.240953+0000","last_became_active":"2024-01-27T01:06:25.851605+0000","last_became_peered":"2024-01-27T01:06:25.851605+0000","last_unstale":"2024-01-27T01:06:28.240953+0000","last_undegraded":"2024-01-27T01:06:28.240953+0000","last_fullsized":"2024-01-27T01:06:28.240953+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:09.386094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865985+0000","last_change":"2024-01-27T01:06:26.873199+0000","last_active":"2024-01-27T01:06:27.865985+0000","last_peered":"2024-01-27T01:06:27.865985+0000","last_clean":"2024-01-27T01:06:27.865985+0000","last_became_active":"2024-01-27T01:06:26.872658+0000","last_became_peered":"2024-01-27T01:06:26.872658+0000","last_unstale":"2024-01-27T01:06:27.865985+0000","last_undegraded":"2024-01-27T01:06:27.865985+0000","last_fullsized":"2024-01-27T01:06:27.865985+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:06:45.526347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913764+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:10:10.314575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.037877+0000","last_change":"2024-01-27T01:06:25.854878+0000","last_active":"2024-01-27T01:06:28.037877+0000","last_peered":"2024-01-27T01:06:28.037877+0000","last_clean":"2024-01-27T01:06:28.037877+0000","last_became_active":"2024-01-27T01:06:25.854623+0000","last_became_peered":"2024-01-27T01:06:25.854623+0000","last_unstale":"2024-01-27T01:06:28.037877+0000","last_undegraded":"2024-01-27T01:06:28.037877+0000","last_fullsized":"2024-01-27T01:06:28.037877+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:21.007449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355292+0000","last_change":"2024-01-27T01:06:26.876963+0000","last_active":"2024-01-27T01:06:28.355292+0000","last_peered":"2024-01-27T01:06:28.355292+0000","last_clean":"2024-01-27T01:06:28.355292+0000","last_became_active":"2024-01-27T01:06:26.876709+0000","last_became_peered":"2024-01-27T01:06:26.876709+0000","last_unstale":"2024-01-27T01:06:28.355292+0000","last_undegraded":"2024-01-27T01:06:28.355292+0000","last_fullsized":"2024-01-27T01:06:28.355292+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:58:14.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916960+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:57.409497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1b","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038235+0000","last_change":"2024-01-27T01:06:25.854193+0000","last_active":"2024-01-27T01:06:28.038235+0000","last_peered":"2024-01-27T01:06:28.038235+0000","last_clean":"2024-01-27T01:06:28.038235+0000","last_became_active":"2024-01-27T01:06:25.853956+0000","last_became_peered":"2024-01-27T01:06:25.853956+0000","last_unstale":"2024-01-27T01:06:28.038235+0000","last_undegraded":"2024-01-27T01:06:28.038235+0000","last_fullsized":"2024-01-27T01:06:28.038235+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:33:02.747087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355322+0000","last_change":"2024-01-27T01:06:26.866121+0000","last_active":"2024-01-27T01:06:28.355322+0000","last_peered":"2024-01-27T01:06:28.355322+0000","last_clean":"2024-01-27T01:06:28.355322+0000","last_became_active":"2024-01-27T01:06:26.865391+0000","last_became_peered":"2024-01-27T01:06:26.865391+0000","last_unstale":"2024-01-27T01:06:28.355322+0000","last_undegraded":"2024-01-27T01:06:28.355322+0000","last_fullsized":"2024-01-27T01:06:28.355322+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:02:03.812030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910329+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:27:27.720315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1c","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866341+0000","last_change":"2024-01-27T01:06:25.855114+0000","last_active":"2024-01-27T01:06:27.866341+0000","last_peered":"2024-01-27T01:06:27.866341+0000","last_clean":"2024-01-27T01:06:27.866341+0000","last_became_active":"2024-01-27T01:06:25.854790+0000","last_became_peered":"2024-01-27T01:06:25.854790+0000","last_unstale":"2024-01-27T01:06:27.866341+0000","last_undegraded":"2024-01-27T01:06:27.866341+0000","last_fullsized":"2024-01-27T01:06:27.866341+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:58:58.498377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866394+0000","last_change":"2024-01-27T01:06:26.869128+0000","last_active":"2024-01-27T01:06:27.866394+0000","last_peered":"2024-01-27T01:06:27.866394+0000","last_clean":"2024-01-27T01:06:27.866394+0000","last_became_active":"2024-01-27T01:06:26.868892+0000","last_became_peered":"2024-01-27T01:06:26.868892+0000","last_unstale":"2024-01-27T01:06:27.866394+0000","last_undegraded":"2024-01-27T01:06:27.866394+0000","last_fullsized":"2024-01-27T01:06:27.866394+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:40:40.869516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916917+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:31.265591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1d","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866926+0000","last_change":"2024-01-27T01:06:25.855991+0000","last_active":"2024-01-27T01:06:27.866926+0000","last_peered":"2024-01-27T01:06:27.866926+0000","last_clean":"2024-01-27T01:06:27.866926+0000","last_became_active":"2024-01-27T01:06:25.855679+0000","last_became_peered":"2024-01-27T01:06:25.855679+0000","last_unstale":"2024-01-27T01:06:27.866926+0000","last_undegraded":"2024-01-27T01:06:27.866926+0000","last_fullsized":"2024-01-27T01:06:27.866926+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:31:04.295025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666155+0000","last_change":"2024-01-27T01:06:26.875831+0000","last_active":"2024-01-27T01:06:28.666155+0000","last_peered":"2024-01-27T01:06:28.666155+0000","last_clean":"2024-01-27T01:06:28.666155+0000","last_became_active":"2024-01-27T01:06:26.875488+0000","last_became_peered":"2024-01-27T01:06:26.875488+0000","last_unstale":"2024-01-27T01:06:28.666155+0000","last_undegraded":"2024-01-27T01:06:28.666155+0000","last_fullsized":"2024-01-27T01:06:28.666155+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:07:51.640059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910981+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:57:15.231912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1e","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665879+0000","last_change":"2024-01-27T01:06:25.856504+0000","last_active":"2024-01-27T01:06:28.665879+0000","last_peered":"2024-01-27T01:06:28.665879+0000","last_clean":"2024-01-27T01:06:28.665879+0000","last_became_active":"2024-01-27T01:06:25.856272+0000","last_became_peered":"2024-01-27T01:06:25.856272+0000","last_unstale":"2024-01-27T01:06:28.665879+0000","last_undegraded":"2024-01-27T01:06:28.665879+0000","last_fullsized":"2024-01-27T01:06:28.665879+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:20:56.736428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666038+0000","last_change":"2024-01-27T01:06:26.878625+0000","last_active":"2024-01-27T01:06:28.666038+0000","last_peered":"2024-01-27T01:06:28.666038+0000","last_clean":"2024-01-27T01:06:28.666038+0000","last_became_active":"2024-01-27T01:06:26.878361+0000","last_became_peered":"2024-01-27T01:06:26.878361+0000","last_unstale":"2024-01-27T01:06:28.666038+0000","last_undegraded":"2024-01-27T01:06:28.666038+0000","last_fullsized":"2024-01-27T01:06:28.666038+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:37:06.977689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910594+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:31:50.703251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1f","version":"222'4","reported_seq":15,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355378+0000","last_change":"2024-01-27T01:06:25.856458+0000","last_active":"2024-01-27T01:06:28.355378+0000","last_peered":"2024-01-27T01:06:28.355378+0000","last_clean":"2024-01-27T01:06:28.355378+0000","last_became_active":"2024-01-27T01:06:25.856162+0000","last_became_peered":"2024-01-27T01:06:25.856162+0000","last_unstale":"2024-01-27T01:06:28.355378+0000","last_undegraded":"2024-01-27T01:06:28.355378+0000","last_fullsized":"2024-01-27T01:06:28.355378+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:51:44.378880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.867070+0000","last_change":"2024-01-27T01:06:26.869588+0000","last_active":"2024-01-27T01:06:27.867070+0000","last_peered":"2024-01-27T01:06:27.867070+0000","last_clean":"2024-01-27T01:06:27.867070+0000","last_became_active":"2024-01-27T01:06:26.869264+0000","last_became_peered":"2024-01-27T01:06:26.869264+0000","last_unstale":"2024-01-27T01:06:27.867070+0000","last_undegraded":"2024-01-27T01:06:27.867070+0000","last_fullsized":"2024-01-27T01:06:27.867070+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:42:54.364688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913160+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:15.067443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.037884+0000","last_change":"2024-01-27T01:06:26.869029+0000","last_active":"2024-01-27T01:06:28.037884+0000","last_peered":"2024-01-27T01:06:28.037884+0000","last_clean":"2024-01-27T01:06:28.037884+0000","last_became_active":"2024-01-27T01:06:26.868793+0000","last_became_peered":"2024-01-27T01:06:26.868793+0000","last_unstale":"2024-01-27T01:06:28.037884+0000","last_undegraded":"2024-01-27T01:06:28.037884+0000","last_fullsized":"2024-01-27T01:06:28.037884+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:10.965469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.037888+0000","last_change":"2024-01-27T01:06:26.876343+0000","last_active":"2024-01-27T01:06:28.037888+0000","last_peered":"2024-01-27T01:06:28.037888+0000","last_clean":"2024-01-27T01:06:28.037888+0000","last_became_active":"2024-01-27T01:06:26.875570+0000","last_became_peered":"2024-01-27T01:06:26.875570+0000","last_unstale":"2024-01-27T01:06:28.037888+0000","last_undegraded":"2024-01-27T01:06:28.037888+0000","last_fullsized":"2024-01-27T01:06:28.037888+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:07:51.640059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355113+0000","last_change":"2024-01-27T01:06:26.871676+0000","last_active":"2024-01-27T01:06:28.355113+0000","last_peered":"2024-01-27T01:06:28.355113+0000","last_clean":"2024-01-27T01:06:28.355113+0000","last_became_active":"2024-01-27T01:06:26.871423+0000","last_became_peered":"2024-01-27T01:06:26.871423+0000","last_unstale":"2024-01-27T01:06:28.355113+0000","last_undegraded":"2024-01-27T01:06:28.355113+0000","last_fullsized":"2024-01-27T01:06:28.355113+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:29:58.146876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866078+0000","last_change":"2024-01-27T01:06:26.872013+0000","last_active":"2024-01-27T01:06:27.866078+0000","last_peered":"2024-01-27T01:06:27.866078+0000","last_clean":"2024-01-27T01:06:27.866078+0000","last_became_active":"2024-01-27T01:06:26.867577+0000","last_became_peered":"2024-01-27T01:06:26.867577+0000","last_unstale":"2024-01-27T01:06:27.866078+0000","last_undegraded":"2024-01-27T01:06:27.866078+0000","last_fullsized":"2024-01-27T01:06:27.866078+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:27.604574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866848+0000","last_change":"2024-01-27T01:06:26.870649+0000","last_active":"2024-01-27T01:06:27.866848+0000","last_peered":"2024-01-27T01:06:27.866848+0000","last_clean":"2024-01-27T01:06:27.866848+0000","last_became_active":"2024-01-27T01:06:26.870433+0000","last_became_peered":"2024-01-27T01:06:26.870433+0000","last_unstale":"2024-01-27T01:06:27.866848+0000","last_undegraded":"2024-01-27T01:06:27.866848+0000","last_fullsized":"2024-01-27T01:06:27.866848+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:11:27.402943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.241052+0000","last_change":"2024-01-27T01:06:26.878202+0000","last_active":"2024-01-27T01:06:28.241052+0000","last_peered":"2024-01-27T01:06:28.241052+0000","last_clean":"2024-01-27T01:06:28.241052+0000","last_became_active":"2024-01-27T01:06:26.877900+0000","last_became_peered":"2024-01-27T01:06:26.877900+0000","last_unstale":"2024-01-27T01:06:28.241052+0000","last_undegraded":"2024-01-27T01:06:28.241052+0000","last_fullsized":"2024-01-27T01:06:28.241052+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:42.766155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665916+0000","last_change":"2024-01-27T01:06:26.878911+0000","last_active":"2024-01-27T01:06:28.665916+0000","last_peered":"2024-01-27T01:06:28.665916+0000","last_clean":"2024-01-27T01:06:28.665916+0000","last_became_active":"2024-01-27T01:06:26.878637+0000","last_became_peered":"2024-01-27T01:06:26.878637+0000","last_unstale":"2024-01-27T01:06:28.665916+0000","last_undegraded":"2024-01-27T01:06:28.665916+0000","last_fullsized":"2024-01-27T01:06:28.665916+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:33:03.751565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865886+0000","last_change":"2024-01-27T01:06:26.865149+0000","last_active":"2024-01-27T01:06:27.865886+0000","last_peered":"2024-01-27T01:06:27.865886+0000","last_clean":"2024-01-27T01:06:27.865886+0000","last_became_active":"2024-01-27T01:06:26.864853+0000","last_became_peered":"2024-01-27T01:06:26.864853+0000","last_unstale":"2024-01-27T01:06:27.865886+0000","last_undegraded":"2024-01-27T01:06:27.865886+0000","last_fullsized":"2024-01-27T01:06:27.865886+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:10:44.194323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865471+0000","last_change":"2024-01-27T01:06:26.867241+0000","last_active":"2024-01-27T01:06:27.865471+0000","last_peered":"2024-01-27T01:06:27.865471+0000","last_clean":"2024-01-27T01:06:27.865471+0000","last_became_active":"2024-01-27T01:06:26.866917+0000","last_became_peered":"2024-01-27T01:06:26.866917+0000","last_unstale":"2024-01-27T01:06:27.865471+0000","last_undegraded":"2024-01-27T01:06:27.865471+0000","last_fullsized":"2024-01-27T01:06:27.865471+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:24:25.818592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865972+0000","last_change":"2024-01-27T01:06:26.874471+0000","last_active":"2024-01-27T01:06:27.865972+0000","last_peered":"2024-01-27T01:06:27.865972+0000","last_clean":"2024-01-27T01:06:27.865972+0000","last_became_active":"2024-01-27T01:06:26.874271+0000","last_became_peered":"2024-01-27T01:06:26.874271+0000","last_unstale":"2024-01-27T01:06:27.865972+0000","last_undegraded":"2024-01-27T01:06:27.865972+0000","last_fullsized":"2024-01-27T01:06:27.865972+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:15:11.295547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665512+0000","last_change":"2024-01-27T01:06:26.876390+0000","last_active":"2024-01-27T01:06:28.665512+0000","last_peered":"2024-01-27T01:06:28.665512+0000","last_clean":"2024-01-27T01:06:28.665512+0000","last_became_active":"2024-01-27T01:06:26.876056+0000","last_became_peered":"2024-01-27T01:06:26.876056+0000","last_unstale":"2024-01-27T01:06:28.665512+0000","last_undegraded":"2024-01-27T01:06:28.665512+0000","last_fullsized":"2024-01-27T01:06:28.665512+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:38:36.236958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665852+0000","last_change":"2024-01-27T01:06:26.879441+0000","last_active":"2024-01-27T01:06:28.665852+0000","last_peered":"2024-01-27T01:06:28.665852+0000","last_clean":"2024-01-27T01:06:28.665852+0000","last_became_active":"2024-01-27T01:06:26.879148+0000","last_became_peered":"2024-01-27T01:06:26.879148+0000","last_unstale":"2024-01-27T01:06:28.665852+0000","last_undegraded":"2024-01-27T01:06:28.665852+0000","last_fullsized":"2024-01-27T01:06:28.665852+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:05:08.012027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666259+0000","last_change":"2024-01-27T01:06:26.876823+0000","last_active":"2024-01-27T01:06:28.666259+0000","last_peered":"2024-01-27T01:06:28.666259+0000","last_clean":"2024-01-27T01:06:28.666259+0000","last_became_active":"2024-01-27T01:06:26.876304+0000","last_became_peered":"2024-01-27T01:06:26.876304+0000","last_unstale":"2024-01-27T01:06:28.666259+0000","last_undegraded":"2024-01-27T01:06:28.666259+0000","last_fullsized":"2024-01-27T01:06:28.666259+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:28.378677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666099+0000","last_change":"2024-01-27T01:06:26.873401+0000","last_active":"2024-01-27T01:06:28.666099+0000","last_peered":"2024-01-27T01:06:28.666099+0000","last_clean":"2024-01-27T01:06:28.666099+0000","last_became_active":"2024-01-27T01:06:26.873094+0000","last_became_peered":"2024-01-27T01:06:26.873094+0000","last_unstale":"2024-01-27T01:06:28.666099+0000","last_undegraded":"2024-01-27T01:06:28.666099+0000","last_fullsized":"2024-01-27T01:06:28.666099+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:24.082749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865407+0000","last_change":"2024-01-27T01:06:26.864890+0000","last_active":"2024-01-27T01:06:27.865407+0000","last_peered":"2024-01-27T01:06:27.865407+0000","last_clean":"2024-01-27T01:06:27.865407+0000","last_became_active":"2024-01-27T01:06:26.864556+0000","last_became_peered":"2024-01-27T01:06:26.864556+0000","last_unstale":"2024-01-27T01:06:27.865407+0000","last_undegraded":"2024-01-27T01:06:27.865407+0000","last_fullsized":"2024-01-27T01:06:27.865407+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:41.033580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865791+0000","last_change":"2024-01-27T01:06:26.874473+0000","last_active":"2024-01-27T01:06:27.865791+0000","last_peered":"2024-01-27T01:06:27.865791+0000","last_clean":"2024-01-27T01:06:27.865791+0000","last_became_active":"2024-01-27T01:06:26.874203+0000","last_became_peered":"2024-01-27T01:06:26.874203+0000","last_unstale":"2024-01-27T01:06:27.865791+0000","last_undegraded":"2024-01-27T01:06:27.865791+0000","last_fullsized":"2024-01-27T01:06:27.865791+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:33:37.975146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240809+0000","last_change":"2024-01-27T01:06:26.863380+0000","last_active":"2024-01-27T01:06:28.240809+0000","last_peered":"2024-01-27T01:06:28.240809+0000","last_clean":"2024-01-27T01:06:28.240809+0000","last_became_active":"2024-01-27T01:06:26.863034+0000","last_became_peered":"2024-01-27T01:06:26.863034+0000","last_unstale":"2024-01-27T01:06:28.240809+0000","last_undegraded":"2024-01-27T01:06:28.240809+0000","last_fullsized":"2024-01-27T01:06:28.240809+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:32.271904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355189+0000","last_change":"2024-01-27T01:06:26.877929+0000","last_active":"2024-01-27T01:06:28.355189+0000","last_peered":"2024-01-27T01:06:28.355189+0000","last_clean":"2024-01-27T01:06:28.355189+0000","last_became_active":"2024-01-27T01:06:26.877679+0000","last_became_peered":"2024-01-27T01:06:26.877679+0000","last_unstale":"2024-01-27T01:06:28.355189+0000","last_undegraded":"2024-01-27T01:06:28.355189+0000","last_fullsized":"2024-01-27T01:06:28.355189+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:53:15.482586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665563+0000","last_change":"2024-01-27T01:06:26.876674+0000","last_active":"2024-01-27T01:06:28.665563+0000","last_peered":"2024-01-27T01:06:28.665563+0000","last_clean":"2024-01-27T01:06:28.665563+0000","last_became_active":"2024-01-27T01:06:26.876413+0000","last_became_peered":"2024-01-27T01:06:26.876413+0000","last_unstale":"2024-01-27T01:06:28.665563+0000","last_undegraded":"2024-01-27T01:06:28.665563+0000","last_fullsized":"2024-01-27T01:06:28.665563+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:08.822886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038257+0000","last_change":"2024-01-27T01:06:26.877777+0000","last_active":"2024-01-27T01:06:28.038257+0000","last_peered":"2024-01-27T01:06:28.038257+0000","last_clean":"2024-01-27T01:06:28.038257+0000","last_became_active":"2024-01-27T01:06:26.877515+0000","last_became_peered":"2024-01-27T01:06:26.877515+0000","last_unstale":"2024-01-27T01:06:28.038257+0000","last_undegraded":"2024-01-27T01:06:28.038257+0000","last_fullsized":"2024-01-27T01:06:28.038257+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:27:14.653781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866589+0000","last_change":"2024-01-27T01:06:26.865833+0000","last_active":"2024-01-27T01:06:27.866589+0000","last_peered":"2024-01-27T01:06:27.866589+0000","last_clean":"2024-01-27T01:06:27.866589+0000","last_became_active":"2024-01-27T01:06:26.865525+0000","last_became_peered":"2024-01-27T01:06:26.865525+0000","last_unstale":"2024-01-27T01:06:27.866589+0000","last_undegraded":"2024-01-27T01:06:27.866589+0000","last_fullsized":"2024-01-27T01:06:27.866589+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:42:10.990703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240896+0000","last_change":"2024-01-27T01:06:26.879535+0000","last_active":"2024-01-27T01:06:28.240896+0000","last_peered":"2024-01-27T01:06:28.240896+0000","last_clean":"2024-01-27T01:06:28.240896+0000","last_became_active":"2024-01-27T01:06:26.879219+0000","last_became_peered":"2024-01-27T01:06:26.879219+0000","last_unstale":"2024-01-27T01:06:28.240896+0000","last_undegraded":"2024-01-27T01:06:28.240896+0000","last_fullsized":"2024-01-27T01:06:28.240896+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:58.495062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666051+0000","last_change":"2024-01-27T01:06:26.876736+0000","last_active":"2024-01-27T01:06:28.666051+0000","last_peered":"2024-01-27T01:06:28.666051+0000","last_clean":"2024-01-27T01:06:28.666051+0000","last_became_active":"2024-01-27T01:06:26.875870+0000","last_became_peered":"2024-01-27T01:06:26.875870+0000","last_unstale":"2024-01-27T01:06:28.666051+0000","last_undegraded":"2024-01-27T01:06:28.666051+0000","last_fullsized":"2024-01-27T01:06:28.666051+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:43.223271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865691+0000","last_change":"2024-01-27T01:06:26.873409+0000","last_active":"2024-01-27T01:06:27.865691+0000","last_peered":"2024-01-27T01:06:27.865691+0000","last_clean":"2024-01-27T01:06:27.865691+0000","last_became_active":"2024-01-27T01:06:26.868026+0000","last_became_peered":"2024-01-27T01:06:26.868026+0000","last_unstale":"2024-01-27T01:06:27.865691+0000","last_undegraded":"2024-01-27T01:06:27.865691+0000","last_fullsized":"2024-01-27T01:06:27.865691+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:15.664932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665379+0000","last_change":"2024-01-27T01:06:26.881786+0000","last_active":"2024-01-27T01:06:28.665379+0000","last_peered":"2024-01-27T01:06:28.665379+0000","last_clean":"2024-01-27T01:06:28.665379+0000","last_became_active":"2024-01-27T01:06:26.881527+0000","last_became_peered":"2024-01-27T01:06:26.881527+0000","last_unstale":"2024-01-27T01:06:28.665379+0000","last_undegraded":"2024-01-27T01:06:28.665379+0000","last_fullsized":"2024-01-27T01:06:28.665379+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:43.630831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865757+0000","last_change":"2024-01-27T01:06:26.872094+0000","last_active":"2024-01-27T01:06:27.865757+0000","last_peered":"2024-01-27T01:06:27.865757+0000","last_clean":"2024-01-27T01:06:27.865757+0000","last_became_active":"2024-01-27T01:06:26.871711+0000","last_became_peered":"2024-01-27T01:06:26.871711+0000","last_unstale":"2024-01-27T01:06:27.865757+0000","last_undegraded":"2024-01-27T01:06:27.865757+0000","last_fullsized":"2024-01-27T01:06:27.865757+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:24:32.821570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.037950+0000","last_change":"2024-01-27T01:06:26.879291+0000","last_active":"2024-01-27T01:06:28.037950+0000","last_peered":"2024-01-27T01:06:28.037950+0000","last_clean":"2024-01-27T01:06:28.037950+0000","last_became_active":"2024-01-27T01:06:26.879063+0000","last_became_peered":"2024-01-27T01:06:26.879063+0000","last_unstale":"2024-01-27T01:06:28.037950+0000","last_undegraded":"2024-01-27T01:06:28.037950+0000","last_fullsized":"2024-01-27T01:06:28.037950+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:22:23.545688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866284+0000","last_change":"2024-01-27T01:06:26.870422+0000","last_active":"2024-01-27T01:06:27.866284+0000","last_peered":"2024-01-27T01:06:27.866284+0000","last_clean":"2024-01-27T01:06:27.866284+0000","last_became_active":"2024-01-27T01:06:26.870117+0000","last_became_peered":"2024-01-27T01:06:26.870117+0000","last_unstale":"2024-01-27T01:06:27.866284+0000","last_undegraded":"2024-01-27T01:06:27.866284+0000","last_fullsized":"2024-01-27T01:06:27.866284+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:42:55.987157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038255+0000","last_change":"2024-01-27T01:06:26.868177+0000","last_active":"2024-01-27T01:06:28.038255+0000","last_peered":"2024-01-27T01:06:28.038255+0000","last_clean":"2024-01-27T01:06:28.038255+0000","last_became_active":"2024-01-27T01:06:26.867906+0000","last_became_peered":"2024-01-27T01:06:26.867906+0000","last_unstale":"2024-01-27T01:06:28.038255+0000","last_undegraded":"2024-01-27T01:06:28.038255+0000","last_fullsized":"2024-01-27T01:06:28.038255+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:32.271904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.241039+0000","last_change":"2024-01-27T01:06:26.868462+0000","last_active":"2024-01-27T01:06:28.241039+0000","last_peered":"2024-01-27T01:06:28.241039+0000","last_clean":"2024-01-27T01:06:28.241039+0000","last_became_active":"2024-01-27T01:06:26.868203+0000","last_became_peered":"2024-01-27T01:06:26.868203+0000","last_unstale":"2024-01-27T01:06:28.241039+0000","last_undegraded":"2024-01-27T01:06:28.241039+0000","last_fullsized":"2024-01-27T01:06:28.241039+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:10.965469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866164+0000","last_change":"2024-01-27T01:06:26.871135+0000","last_active":"2024-01-27T01:06:27.866164+0000","last_peered":"2024-01-27T01:06:27.866164+0000","last_clean":"2024-01-27T01:06:27.866164+0000","last_became_active":"2024-01-27T01:06:26.870854+0000","last_became_peered":"2024-01-27T01:06:26.870854+0000","last_unstale":"2024-01-27T01:06:27.866164+0000","last_undegraded":"2024-01-27T01:06:27.866164+0000","last_fullsized":"2024-01-27T01:06:27.866164+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:12:59.875769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666262+0000","last_change":"2024-01-27T01:06:26.873715+0000","last_active":"2024-01-27T01:06:28.666262+0000","last_peered":"2024-01-27T01:06:28.666262+0000","last_clean":"2024-01-27T01:06:28.666262+0000","last_became_active":"2024-01-27T01:06:26.873430+0000","last_became_peered":"2024-01-27T01:06:26.873430+0000","last_unstale":"2024-01-27T01:06:28.666262+0000","last_undegraded":"2024-01-27T01:06:28.666262+0000","last_fullsized":"2024-01-27T01:06:28.666262+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:44.342963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3538,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":630,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8750,"ondisk_log_size":8750,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576004,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300632,"kb_used_data":3344,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071208,"statfs":{"total":96636764160,"available":96328916992,"internally_reserved":0,"allocated":3424256,"data_stored":2074391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63200000000000001,"15min":0.61399999999999999},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.312},"max":{"1min":0.82099999999999995,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.63700000000000001,"15min":0.623},"min":{"1min":0.498,"5min":0.39900000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":0.879,"15min":0.879},"last":0.55200000000000005}]},{"osd":1,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.623,"15min":0.61599999999999999},"min":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65600000000000003,"15min":0.63700000000000001},"min":{"1min":0.505,"5min":0.42099999999999999,"15min":0.34300000000000003},"max":{"1min":0.76800000000000002,"5min":0.84099999999999997,"15min":0.84399999999999997},"last":0.60599999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63400000000000001,"15min":0.61399999999999999},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82099999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.628},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64500000000000002,"15min":0.63600000000000001},"min":{"1min":0.49399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79400000000000004,"5min":0.872,"15min":0.88900000000000001},"last":0.59499999999999997}]},{"osd":3,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.628,"15min":0.62},"min":{"1min":0.52100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79900000000000004,"5min":0.876,"15min":0.876},"last":0.504},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66800000000000004,"15min":0.64700000000000002},"min":{"1min":0.53200000000000003,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84599999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.69099999999999995}]},{"osd":4,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.50900000000000001,"15min":0.48999999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.29999999999999999},"max":{"1min":0.73399999999999999,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.51900000000000002},"min":{"1min":0.35699999999999998,"5min":0.34699999999999998,"15min":0.32200000000000001},"max":{"1min":0.77500000000000002,"5min":0.80300000000000005,"15min":0.86599999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55800000000000005,"15min":0.52600000000000002},"min":{"1min":0.40999999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.753,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57999999999999996,"15min":0.56499999999999995},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.33700000000000002},"max":{"1min":0.76700000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.66300000000000003}]},{"osd":6,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60999999999999999,"15min":0.58899999999999997},"min":{"1min":0.42499999999999999,"5min":0.42199999999999999,"15min":0.37},"max":{"1min":0.81100000000000005,"5min":0.83999999999999997,"15min":0.84799999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61299999999999999,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.39100000000000001,"15min":0.35699999999999998},"max":{"1min":0.76100000000000001,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.68000000000000005}]}]},{"osd":6,"up_from":13,"seq":55834576004,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308244,"kb_used_data":2764,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063596,"statfs":{"total":96636764160,"available":96321122304,"internally_reserved":0,"allocated":2830336,"data_stored":1484507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61099999999999999,"15min":0.60799999999999998},"min":{"1min":0.41999999999999998,"5min":0.40799999999999997,"15min":0.36399999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.496},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.441,"5min":0.432,"15min":0.376},"max":{"1min":0.82499999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.67600000000000005}]},{"osd":1,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79000000000000004,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.40799999999999997,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.628}]},{"osd":2,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.39700000000000002,"15min":0.377},"max":{"1min":0.80400000000000005,"5min":0.878,"15min":0.878},"last":0.65500000000000003}]},{"osd":3,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.622,"15min":0.627},"min":{"1min":0.5,"5min":0.378,"15min":0.378},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.90100000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66300000000000003,"15min":0.66000000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.59899999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.36299999999999999,"5min":0.33300000000000002,"15min":0.32500000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.86599999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33900000000000002,"5min":0.33800000000000002,"15min":0.29799999999999999},"max":{"1min":0.77300000000000002,"5min":0.78500000000000003,"15min":0.88900000000000001},"last":0.54700000000000004}]},{"osd":5,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.31900000000000001},"max":{"1min":0.68999999999999995,"5min":0.75900000000000001,"15min":0.80500000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.42199999999999999,"5min":0.37,"15min":0.28899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.874},"last":0.58499999999999996}]},{"osd":7,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45800000000000002,"5min":0.39000000000000001,"15min":0.35799999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.92900000000000005},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.44500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78900000000000003,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.61399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834576004,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300024,"kb_used_data":2736,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071816,"statfs":{"total":96636764160,"available":96329539584,"internally_reserved":0,"allocated":2801664,"data_stored":1473710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.41099999999999998,"15min":0.42099999999999999},"min":{"1min":0.26400000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.64400000000000002,"5min":0.66100000000000003,"15min":0.69599999999999995},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44500000000000001,"15min":0.45500000000000002},"min":{"1min":0.34000000000000002,"5min":0.28399999999999997,"15min":0.26800000000000002},"max":{"1min":0.57999999999999996,"5min":0.73999999999999999,"15min":0.76600000000000001},"last":0.35699999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.439,"15min":0.438},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.26200000000000001},"max":{"1min":0.748,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.47599999999999998,"15min":0.48999999999999999},"min":{"1min":0.379,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.67400000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.5}]},{"osd":3,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52300000000000002,"15min":0.53300000000000003},"min":{"1min":0.39300000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73699999999999999,"5min":0.84799999999999998,"15min":1.2130000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.51400000000000001},"min":{"1min":0.41699999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68200000000000005,"5min":1.125,"15min":1.22},"last":0.42099999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.67200000000000004,"15min":0.67400000000000004},"min":{"1min":0.42099999999999999,"5min":0.41099999999999998,"15min":0.39800000000000002},"max":{"1min":0.82699999999999996,"5min":1.242,"15min":1.242},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69099999999999995,"15min":0.69599999999999995},"min":{"1min":0.53700000000000003,"5min":0.496,"15min":0.35699999999999998},"max":{"1min":0.90600000000000003,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.45900000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73499999999999999,"15min":0.70899999999999996},"min":{"1min":0.53300000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.876,"5min":5.444,"15min":5.444},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.71699999999999997,"15min":0.70799999999999996},"min":{"1min":0.502,"5min":0.47599999999999998,"15min":0.435},"max":{"1min":0.97799999999999998,"5min":2.6669999999999998,"15min":2.6669999999999998},"last":0.47599999999999998}]},{"osd":6,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.74199999999999999,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":0.93899999999999995,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.81299999999999994,"15min":0.745},"min":{"1min":0.48799999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.02,"5min":11.223000000000001,"15min":11.223000000000001},"last":0.52400000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.752,"15min":0.75600000000000001},"min":{"1min":0.58999999999999997,"5min":0.52800000000000002,"15min":0.47599999999999998},"max":{"1min":0.96299999999999997,"5min":2.9660000000000002,"15min":2.9660000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.77700000000000002,"15min":0.76800000000000002},"min":{"1min":0.58799999999999997,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.94499999999999995,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.53400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834576003,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317004,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94054836,"statfs":{"total":96636764160,"available":96312152064,"internally_reserved":0,"allocated":3411968,"data_stored":2071643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42899999999999999,"15min":0.42199999999999999},"min":{"1min":0.30299999999999999,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":0.55600000000000005,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.436,"5min":0.45600000000000002,"15min":0.46100000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":0.56499999999999995,"5min":0.72699999999999998,"15min":0.73399999999999999},"last":0.26700000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.29699999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.58299999999999996,"5min":0.66900000000000004,"15min":0.76800000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48399999999999999,"15min":0.48899999999999999},"min":{"1min":0.38600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64600000000000002,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.45400000000000001}]},{"osd":3,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.66100000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.504},"min":{"1min":0.36699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64100000000000001,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.36699999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.68300000000000005,"15min":0.67700000000000005},"min":{"1min":0.57799999999999996,"5min":0.47999999999999998,"15min":0.42199999999999999},"max":{"1min":0.86399999999999999,"5min":0.91500000000000004,"15min":1.0720000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.56999999999999995,"5min":0.47099999999999997,"15min":0.437},"max":{"1min":0.82699999999999996,"5min":1.002,"15min":1.002},"last":0.66200000000000003}]},{"osd":5,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.68999999999999995,"15min":0.70099999999999996},"min":{"1min":0.58199999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.876,"5min":1.0209999999999999,"15min":2.165},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70599999999999996,"15min":0.71199999999999997},"min":{"1min":0.498,"5min":0.48799999999999999,"15min":0.45600000000000002},"max":{"1min":0.89000000000000001,"5min":0.97399999999999998,"15min":1.1399999999999999},"last":0.64700000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69299999999999995,"15min":0.71099999999999997},"min":{"1min":0.56499999999999995,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.84599999999999997,"5min":0.98899999999999999,"15min":1.0860000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72599999999999998,"15min":0.74099999999999999},"min":{"1min":0.54300000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.93400000000000005,"5min":1.0109999999999999,"15min":1.425},"last":0.73299999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.74399999999999999,"15min":0.75700000000000001},"min":{"1min":0.61299999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.98899999999999999,"5min":1.044,"15min":1.1259999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.76300000000000001,"15min":0.76900000000000002},"min":{"1min":0.58299999999999996,"5min":0.51100000000000001,"15min":0.48499999999999999},"max":{"1min":0.91700000000000004,"5min":1.0289999999999999,"15min":1.1910000000000001},"last":0.76600000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576003,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312328,"kb_used_data":2752,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059512,"statfs":{"total":96636764160,"available":96316940288,"internally_reserved":0,"allocated":2818048,"data_stored":1481275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40500000000000003,"15min":0.40799999999999997},"min":{"1min":0.26600000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.46000000000000002,"5min":0.746,"15min":0.746},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.44500000000000001,"15min":0.44400000000000001},"min":{"1min":0.32300000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.59099999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.436}]},{"osd":1,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.435,"15min":0.44500000000000001},"min":{"1min":0.32000000000000001,"5min":0.29099999999999998,"15min":0.27000000000000002},"max":{"1min":0.61299999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.439,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.55600000000000005,"5min":0.73599999999999999,"15min":0.78100000000000003},"last":0.54800000000000004}]},{"osd":3,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.498,"15min":0.497},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.33200000000000002},"max":{"1min":0.627,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.497},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48699999999999999,"15min":0.49199999999999999},"min":{"1min":0.373,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":0.59099999999999997,"5min":0.66600000000000004,"15min":0.75700000000000001},"last":0.52000000000000002}]},{"osd":4,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68200000000000005,"15min":0.68000000000000005},"min":{"1min":0.52800000000000002,"5min":0.495,"15min":0.442},"max":{"1min":0.92700000000000005,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68999999999999995,"15min":0.69399999999999995},"min":{"1min":0.50900000000000001,"5min":0.47099999999999997,"15min":0.378},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":1.107},"last":0.77200000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.69099999999999995,"15min":0.70299999999999996},"min":{"1min":0.52400000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.89500000000000002,"15min":1.167},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70899999999999996,"15min":0.70999999999999996},"min":{"1min":0.47499999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89600000000000002,"5min":0.97499999999999998,"15min":1.05},"last":0.78600000000000003}]},{"osd":6,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71499999999999997,"15min":0.71199999999999997},"min":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.45500000000000002},"max":{"1min":0.81399999999999995,"5min":0.996,"15min":1.077},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70499999999999996,"15min":0.71999999999999997},"min":{"1min":0.497,"5min":0.497,"15min":0.39200000000000002},"max":{"1min":0.876,"5min":0.96699999999999997,"15min":1.135},"last":0.73499999999999999}]},{"osd":7,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.749,"15min":0.75},"min":{"1min":0.56000000000000005,"5min":0.50800000000000001,"15min":0.47099999999999997},"max":{"1min":0.94099999999999995,"5min":0.97399999999999998,"15min":1.198},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.76500000000000001,"15min":0.76900000000000002},"min":{"1min":0.55000000000000004,"5min":0.53100000000000003,"15min":0.48199999999999998},"max":{"1min":0.94599999999999995,"5min":1.028,"15min":1.0880000000000001},"last":0.752}]}]},{"osd":3,"up_from":13,"seq":55834576002,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304144,"kb_used_data":2760,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067696,"statfs":{"total":96636764160,"available":96325320704,"internally_reserved":0,"allocated":2826240,"data_stored":1481311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.44700000000000001,"15min":0.42299999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.54800000000000004,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.309},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.372,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.63500000000000001,"5min":0.66700000000000004,"15min":0.84699999999999998},"last":0.34300000000000003}]},{"osd":1,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.63100000000000001,"15min":0.53500000000000003},"min":{"1min":0.34499999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.629,"5min":15.391,"15min":15.391},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.54600000000000004,"15min":0.52400000000000002},"min":{"1min":0.36699999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65100000000000002,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.42999999999999999}]},{"osd":2,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54600000000000004,"15min":0.51700000000000002},"min":{"1min":0.36499999999999999,"5min":0.36399999999999999,"15min":0.34599999999999997},"max":{"1min":0.62,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.53100000000000003,"15min":0.50700000000000001},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.60699999999999998,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":0.39600000000000002}]},{"osd":4,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.74199999999999999,"15min":0.72099999999999997},"min":{"1min":0.497,"5min":0.47699999999999998,"15min":0.439},"max":{"1min":0.88600000000000001,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.71999999999999997,"15min":0.70799999999999996},"min":{"1min":0.46300000000000002,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":0.94399999999999995,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.52000000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.74299999999999999,"15min":0.71699999999999997},"min":{"1min":0.51100000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.83999999999999997,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72799999999999998,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.97699999999999998,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.57399999999999995}]},{"osd":6,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.754,"15min":0.72599999999999998},"min":{"1min":0.53500000000000003,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.81200000000000006,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.748,"15min":0.74299999999999999},"min":{"1min":0.56100000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.92700000000000005,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.60799999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.75600000000000001,"15min":0.73799999999999999},"min":{"1min":0.58599999999999997,"5min":0.52500000000000002,"15min":0.46700000000000003},"max":{"1min":1.0029999999999999,"5min":4.569,"15min":4.569},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.80600000000000005,"15min":0.78500000000000003},"min":{"1min":0.55700000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0169999999999999,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.59199999999999997}]}]},{"osd":4,"up_from":13,"seq":55834576004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312328,"kb_used_data":2752,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059512,"statfs":{"total":96636764160,"available":96316940288,"internally_reserved":0,"allocated":2818048,"data_stored":1481739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.46899999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.84199999999999997},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.61499999999999999,"15min":0.61399999999999999},"min":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78700000000000003,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.753}]},{"osd":1,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61899999999999999,"15min":0.627},"min":{"1min":0.46100000000000002,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.76400000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61499999999999999,"15min":0.62},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.79800000000000004,"5min":0.874,"15min":0.874},"last":0.69699999999999995}]},{"osd":2,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.59999999999999998},"min":{"1min":0.42399999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.88700000000000001},"last":0.628},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.51100000000000001,"5min":0.42899999999999999,"15min":0.42799999999999999},"max":{"1min":0.81100000000000005,"5min":0.81499999999999995,"15min":0.89200000000000002},"last":0.79200000000000004}]},{"osd":3,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.61599999999999999,"15min":0.623},"min":{"1min":0.38800000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.83299999999999996,"5min":0.83599999999999997,"15min":0.84199999999999997},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.627},"min":{"1min":0.41299999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.88300000000000001},"last":0.72999999999999998}]},{"osd":5,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.75800000000000001,"15min":0.85899999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.502},"min":{"1min":0.376,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.878},"last":0.74099999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34999999999999998},"max":{"1min":0.63200000000000001,"5min":0.70099999999999996,"15min":0.88700000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.70699999999999996,"5min":0.79200000000000004,"15min":0.81499999999999995},"last":0.64800000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.42399999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72599999999999998,"5min":0.80400000000000005,"15min":0.90200000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59499999999999997,"15min":0.59399999999999997},"min":{"1min":0.41599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.73799999999999999,"5min":0.78700000000000003,"15min":0.92600000000000005},"last":0.76700000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576004,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312332,"kb_used_data":2756,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059508,"statfs":{"total":96636764160,"available":96316936192,"internally_reserved":0,"allocated":2822144,"data_stored":1481241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59399999999999997,"15min":0.60299999999999998},"min":{"1min":0.42999999999999999,"5min":0.40500000000000003,"15min":0.27300000000000002},"max":{"1min":0.80700000000000005,"5min":0.82799999999999996,"15min":1.006},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59399999999999997,"15min":0.60699999999999998},"min":{"1min":0.36599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":1.1719999999999999},"last":0.73499999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63700000000000001,"15min":0.65200000000000002},"min":{"1min":0.39100000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.95899999999999996},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.52100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":1.212},"last":0.77900000000000003}]},{"osd":2,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57499999999999996,"15min":0.59399999999999997},"min":{"1min":0.42299999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":1.631},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.623,"5min":0.625,"15min":0.621},"min":{"1min":0.497,"5min":0.38800000000000001,"15min":0.34000000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.60499999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.59999999999999998},"min":{"1min":0.39600000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83299999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.625,"15min":0.63},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.879,"5min":0.879,"15min":1.4690000000000001},"last":0.754}]},{"osd":4,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.505,"15min":0.48799999999999999},"min":{"1min":0.34899999999999998,"5min":0.32200000000000001,"15min":0.317},"max":{"1min":0.76700000000000002,"5min":0.90200000000000002,"15min":0.94399999999999995},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53500000000000003,"15min":0.51500000000000001},"min":{"1min":0.439,"5min":0.35199999999999998,"15min":0.33900000000000002},"max":{"1min":0.80800000000000005,"5min":0.92500000000000004,"15min":0.93300000000000005},"last":0.58299999999999996}]},{"osd":6,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51800000000000002,"15min":0.504},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79400000000000004,"5min":0.96799999999999997,"15min":1.732},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56299999999999994,"15min":0.54500000000000004},"min":{"1min":0.42899999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.753,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.53800000000000003}]},{"osd":7,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.46200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80400000000000005,"5min":0.95399999999999996,"15min":1.0740000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.626,"15min":0.61299999999999999},"min":{"1min":0.504,"5min":0.40799999999999997,"15min":0.39100000000000001},"max":{"1min":0.85199999999999998,"5min":0.996,"15min":1.6919999999999999},"last":0.61599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:06:45.182 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:06:45.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.347+0000 7fe792587700 1 Processor -- start 2024-01-27T01:06:45.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.347+0000 7fe792587700 1 -- start start 2024-01-27T01:06:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.348+0000 7fe792587700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe78c0c6520 0x7fe78c0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.348+0000 7fe792587700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78c15a0b0 con 0x7fe78c0ced50 2024-01-27T01:06:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.348+0000 7fe792587700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78c0ae4d0 con 0x7fe78c0b2b40 2024-01-27T01:06:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.348+0000 7fe792587700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c0af070 con 0x7fe78c0c6520 2024-01-27T01:06:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.349+0000 7fe790b24700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe78c0ced50 0x7fe78c0cb5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45440/0 (socket says 172.21.15.46:45440) 2024-01-27T01:06:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.349+0000 7fe790b24700 1 -- 172.21.15.46:0/250370890 learned_addr learned my addr 172.21.15.46:0/250370890 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.349+0000 7fe78b7fe700 1 --2- 172.21.15.46:0/250370890 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe78c0c6520 0x7fe78c0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.349+0000 7fe78affd700 1 -- 172.21.15.46:0/250370890 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280566771 0 0) 0x7fe78c15a0b0 con 0x7fe78c0ced50 2024-01-27T01:06:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.349+0000 7fe78affd700 1 -- 172.21.15.46:0/250370890 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe774003570 con 0x7fe78c0ced50 2024-01-27T01:06:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.349+0000 7fe78affd700 1 -- 172.21.15.46:0/250370890 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733745176 0 0) 0x7fe78c0ae4d0 con 0x7fe78c0b2b40 2024-01-27T01:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.349+0000 7fe78affd700 1 -- 172.21.15.46:0/250370890 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe78c15a0b0 con 0x7fe78c0b2b40 2024-01-27T01:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.349+0000 7fe78affd700 1 -- 172.21.15.46:0/250370890 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3598931715 0 0) 0x7fe774003570 con 0x7fe78c0ced50 2024-01-27T01:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.349+0000 7fe78b7fe700 1 -- 172.21.15.46:0/250370890 >> v1:172.21.15.106:6789/0 conn(0x7fe78c0b2b40 legacy=0x7fe78c0ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.349+0000 7fe78b7fe700 1 -- 172.21.15.46:0/250370890 >> v1:172.21.15.46:6789/0 conn(0x7fe78c0ced50 legacy=0x7fe78c0cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.349+0000 7fe78b7fe700 1 -- 172.21.15.46:0/250370890 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78c0ad8c0 con 0x7fe78c0c6520 2024-01-27T01:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.349+0000 7fe78b7fe700 1 --2- 172.21.15.46:0/250370890 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe78c0c6520 0x7fe78c0ad1c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fe780009910 tx=0x7fe7800091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e44dabbebd59dce2 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.350+0000 7fe78affd700 1 -- 172.21.15.46:0/250370890 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7840017f0 con 0x7fe78c0c6520 2024-01-27T01:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.350+0000 7fe78affd700 1 -- 172.21.15.46:0/250370890 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe780011420 con 0x7fe78c0c6520 2024-01-27T01:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.350+0000 7fe78affd700 1 -- 172.21.15.46:0/250370890 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7800117a0 con 0x7fe78c0c6520 2024-01-27T01:06:45.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.350+0000 7fe792587700 1 -- 172.21.15.46:0/250370890 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe78c0c6520 msgr2=0x7fe78c0ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:45.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.350+0000 7fe792587700 1 --2- 172.21.15.46:0/250370890 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe78c0c6520 0x7fe78c0ad1c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fe780009910 tx=0x7fe7800091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:45.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.350+0000 7fe792587700 1 -- 172.21.15.46:0/250370890 shutdown_connections 2024-01-27T01:06:45.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.350+0000 7fe792587700 1 --2- 172.21.15.46:0/250370890 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe78c0c6520 0x7fe78c0ad1c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:45.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.350+0000 7fe792587700 1 -- 172.21.15.46:0/250370890 >> 172.21.15.46:0/250370890 conn(0x7fe78c099a30 msgr2=0x7fe78c06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:45.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.350+0000 7fe792587700 1 -- 172.21.15.46:0/250370890 shutdown_connections 2024-01-27T01:06:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.350+0000 7fe792587700 1 -- 172.21.15.46:0/250370890 wait complete. 2024-01-27T01:06:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.351+0000 7fe792587700 1 Processor -- start 2024-01-27T01:06:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.352+0000 7fe792587700 1 -- start start 2024-01-27T01:06:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.352+0000 7fe792587700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe78c0b2b40 0x7fe78c16a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.352+0000 7fe792587700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe78c0c6520 0x7fe78c147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.352+0000 7fe792587700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe78c0ced50 0x7fe78c158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe792587700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c0c87f0 con 0x7fe78c0ced50 2024-01-27T01:06:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe792587700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c0c8db0 con 0x7fe78c0b2b40 2024-01-27T01:06:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe792587700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c159270 con 0x7fe78c0c6520 2024-01-27T01:06:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe790b24700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe78c0ced50 0x7fe78c158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe790b24700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe78c0ced50 0x7fe78c158320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41850/0 (socket says 172.21.15.46:41850) 2024-01-27T01:06:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe790b24700 1 -- 172.21.15.46:0/2686720874 learned_addr learned my addr 172.21.15.46:0/2686720874 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe78bfff700 1 --2- 172.21.15.46:0/2686720874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe78c0b2b40 0x7fe78c16a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe78b7fe700 1 --2- 172.21.15.46:0/2686720874 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe78c0c6520 0x7fe78c147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe790b24700 1 -- 172.21.15.46:0/2686720874 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe78c0c6520 msgr2=0x7fe78c147470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe790b24700 1 --2- 172.21.15.46:0/2686720874 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe78c0c6520 0x7fe78c147470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe790b24700 1 -- 172.21.15.46:0/2686720874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe78c0b2b40 msgr2=0x7fe78c16a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe790b24700 1 --2- 172.21.15.46:0/2686720874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe78c0b2b40 0x7fe78c16a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe790b24700 1 -- 172.21.15.46:0/2686720874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe780009420 con 0x7fe78c0ced50 2024-01-27T01:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe78bfff700 1 --2- 172.21.15.46:0/2686720874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe78c0b2b40 0x7fe78c16a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe790b24700 1 --2- 172.21.15.46:0/2686720874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe78c0ced50 0x7fe78c158320 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fe78400cb70 tx=0x7fe784007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe773fff700 1 -- 172.21.15.46:0/2686720874 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784004dc0 con 0x7fe78c0ced50 2024-01-27T01:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe773fff700 1 -- 172.21.15.46:0/2686720874 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78400a6a0 con 0x7fe78c0ced50 2024-01-27T01:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe773fff700 1 -- 172.21.15.46:0/2686720874 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78400aa60 con 0x7fe78c0ced50 2024-01-27T01:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe792587700 1 -- 172.21.15.46:0/2686720874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe78c156950 con 0x7fe78c0ced50 2024-01-27T01:06:45.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.353+0000 7fe792587700 1 -- 172.21.15.46:0/2686720874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe78c0c8a40 con 0x7fe78c0ced50 2024-01-27T01:06:45.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.354+0000 7fe792587700 1 -- 172.21.15.46:0/2686720874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe78c0c8f20 con 0x7fe78c0ced50 2024-01-27T01:06:45.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.355+0000 7fe773fff700 1 -- 172.21.15.46:0/2686720874 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe784007150 con 0x7fe78c0ced50 2024-01-27T01:06:45.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.355+0000 7fe773fff700 1 --2- 172.21.15.46:0/2686720874 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe778066f30 0x7fe7780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:45.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.355+0000 7fe773fff700 1 -- 172.21.15.46:0/2686720874 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7fe784084340 con 0x7fe78c0ced50 2024-01-27T01:06:45.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.356+0000 7fe78bfff700 1 --2- 172.21.15.46:0/2686720874 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe778066f30 0x7fe7780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:45.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.356+0000 7fe78bfff700 1 --2- 172.21.15.46:0/2686720874 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe778066f30 0x7fe7780693e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe77c003980 tx=0x7fe77c009040 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:45.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.360+0000 7fe773fff700 1 -- 172.21.15.46:0/2686720874 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe784050fa0 con 0x7fe78c0ced50 2024-01-27T01:06:45.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.560+0000 7fe792587700 1 -- 172.21.15.46:0/2686720874 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe78c04c710 con 0x7fe778066f30 2024-01-27T01:06:45.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.574+0000 7fe773fff700 1 -- 172.21.15.46:0/2686720874 <== mgr.9101 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+517877 (secure 0 0 0) 0x7fe78c04c710 con 0x7fe778066f30 2024-01-27T01:06:45.579 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:45.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.579+0000 7fe792587700 1 -- 172.21.15.46:0/2686720874 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe778066f30 msgr2=0x7fe7780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:45.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.579+0000 7fe792587700 1 --2- 172.21.15.46:0/2686720874 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe778066f30 0x7fe7780693e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe77c003980 tx=0x7fe77c009040 comp rx=0 tx=0).stop 2024-01-27T01:06:45.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.579+0000 7fe792587700 1 -- 172.21.15.46:0/2686720874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe78c0ced50 msgr2=0x7fe78c158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:45.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.579+0000 7fe792587700 1 --2- 172.21.15.46:0/2686720874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe78c0ced50 0x7fe78c158320 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fe78400cb70 tx=0x7fe784007c70 comp rx=0 tx=0).stop 2024-01-27T01:06:45.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.579+0000 7fe792587700 1 -- 172.21.15.46:0/2686720874 shutdown_connections 2024-01-27T01:06:45.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.579+0000 7fe792587700 1 --2- 172.21.15.46:0/2686720874 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe778066f30 0x7fe7780693e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:45.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.579+0000 7fe792587700 1 --2- 172.21.15.46:0/2686720874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe78c0ced50 0x7fe78c158320 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:45.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.579+0000 7fe792587700 1 --2- 172.21.15.46:0/2686720874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe78c0b2b40 0x7fe78c16a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:45.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.579+0000 7fe792587700 1 --2- 172.21.15.46:0/2686720874 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe78c0c6520 0x7fe78c147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:45.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.579+0000 7fe792587700 1 -- 172.21.15.46:0/2686720874 >> 172.21.15.46:0/2686720874 conn(0x7fe78c099a30 msgr2=0x7fe78c069f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:45.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.579+0000 7fe792587700 1 -- 172.21.15.46:0/2686720874 shutdown_connections 2024-01-27T01:06:45.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.579+0000 7fe792587700 1 -- 172.21.15.46:0/2686720874 wait complete. 2024-01-27T01:06:45.585 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:06:45.592 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-01-27T01:06:44.734501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593906,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":677,"num_write_kb":3477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8791,"ondisk_log_size":8791,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":202,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2467036,"kb_used_data":23196,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752507684,"statfs":{"total":773094113280,"available":770567868416,"internally_reserved":0,"allocated":23752704,"data_stored":13029817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":787,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.826236"},"pg_stats":[{"pgid":"1.0","version":"213'8750","reported_seq":9665,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.037979+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:06:28.037979+0000","last_peered":"2024-01-27T01:06:28.037979+0000","last_clean":"2024-01-27T01:06:28.037979+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:06:28.037979+0000","last_undegraded":"2024-01-27T01:06:28.037979+0000","last_fullsized":"2024-01-27T01:06:28.037979+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8750,"log_dups_size":0,"ondisk_log_size":8750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":630,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865902+0000","last_change":"2024-01-27T01:06:25.858609+0000","last_active":"2024-01-27T01:06:27.865902+0000","last_peered":"2024-01-27T01:06:27.865902+0000","last_clean":"2024-01-27T01:06:27.865902+0000","last_became_active":"2024-01-27T01:06:25.857638+0000","last_became_peered":"2024-01-27T01:06:25.857638+0000","last_unstale":"2024-01-27T01:06:27.865902+0000","last_undegraded":"2024-01-27T01:06:27.865902+0000","last_fullsized":"2024-01-27T01:06:27.865902+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:35:59.109571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665626+0000","last_change":"2024-01-27T01:06:26.873783+0000","last_active":"2024-01-27T01:06:28.665626+0000","last_peered":"2024-01-27T01:06:28.665626+0000","last_clean":"2024-01-27T01:06:28.665626+0000","last_became_active":"2024-01-27T01:06:26.873490+0000","last_became_peered":"2024-01-27T01:06:26.873490+0000","last_unstale":"2024-01-27T01:06:28.665626+0000","last_undegraded":"2024-01-27T01:06:28.665626+0000","last_fullsized":"2024-01-27T01:06:28.665626+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:32.691743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916373+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:55:49.427110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865572+0000","last_change":"2024-01-27T01:06:25.865235+0000","last_active":"2024-01-27T01:06:27.865572+0000","last_peered":"2024-01-27T01:06:27.865572+0000","last_clean":"2024-01-27T01:06:27.865572+0000","last_became_active":"2024-01-27T01:06:25.864872+0000","last_became_peered":"2024-01-27T01:06:25.864872+0000","last_unstale":"2024-01-27T01:06:27.865572+0000","last_undegraded":"2024-01-27T01:06:27.865572+0000","last_fullsized":"2024-01-27T01:06:27.865572+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:53:38.618410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.241058+0000","last_change":"2024-01-27T01:06:26.879306+0000","last_active":"2024-01-27T01:06:28.241058+0000","last_peered":"2024-01-27T01:06:28.241058+0000","last_clean":"2024-01-27T01:06:28.241058+0000","last_became_active":"2024-01-27T01:06:26.879071+0000","last_became_peered":"2024-01-27T01:06:26.879071+0000","last_unstale":"2024-01-27T01:06:28.241058+0000","last_undegraded":"2024-01-27T01:06:28.241058+0000","last_fullsized":"2024-01-27T01:06:28.241058+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:50:46.965359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910860+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:36:16.613510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666145+0000","last_change":"2024-01-27T01:06:25.856035+0000","last_active":"2024-01-27T01:06:28.666145+0000","last_peered":"2024-01-27T01:06:28.666145+0000","last_clean":"2024-01-27T01:06:28.666145+0000","last_became_active":"2024-01-27T01:06:25.855762+0000","last_became_peered":"2024-01-27T01:06:25.855762+0000","last_unstale":"2024-01-27T01:06:28.666145+0000","last_undegraded":"2024-01-27T01:06:28.666145+0000","last_fullsized":"2024-01-27T01:06:28.666145+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:51:44.378880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038160+0000","last_change":"2024-01-27T01:06:26.870626+0000","last_active":"2024-01-27T01:06:28.038160+0000","last_peered":"2024-01-27T01:06:28.038160+0000","last_clean":"2024-01-27T01:06:28.038160+0000","last_became_active":"2024-01-27T01:06:26.870457+0000","last_became_peered":"2024-01-27T01:06:26.870457+0000","last_unstale":"2024-01-27T01:06:28.038160+0000","last_undegraded":"2024-01-27T01:06:28.038160+0000","last_fullsized":"2024-01-27T01:06:28.038160+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:42.766155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913817+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:22.022827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865995+0000","last_change":"2024-01-27T01:06:25.855015+0000","last_active":"2024-01-27T01:06:27.865995+0000","last_peered":"2024-01-27T01:06:27.865995+0000","last_clean":"2024-01-27T01:06:27.865995+0000","last_became_active":"2024-01-27T01:06:25.854708+0000","last_became_peered":"2024-01-27T01:06:25.854708+0000","last_unstale":"2024-01-27T01:06:27.865995+0000","last_undegraded":"2024-01-27T01:06:27.865995+0000","last_fullsized":"2024-01-27T01:06:27.865995+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:03.971945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038100+0000","last_change":"2024-01-27T01:06:26.876182+0000","last_active":"2024-01-27T01:06:28.038100+0000","last_peered":"2024-01-27T01:06:28.038100+0000","last_clean":"2024-01-27T01:06:28.038100+0000","last_became_active":"2024-01-27T01:06:26.875894+0000","last_became_peered":"2024-01-27T01:06:26.875894+0000","last_unstale":"2024-01-27T01:06:28.038100+0000","last_undegraded":"2024-01-27T01:06:28.038100+0000","last_fullsized":"2024-01-27T01:06:28.038100+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:58.495062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.914162+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:57:07.250360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666039+0000","last_change":"2024-01-27T01:06:25.858142+0000","last_active":"2024-01-27T01:06:28.666039+0000","last_peered":"2024-01-27T01:06:28.666039+0000","last_clean":"2024-01-27T01:06:28.666039+0000","last_became_active":"2024-01-27T01:06:25.857867+0000","last_became_peered":"2024-01-27T01:06:25.857867+0000","last_unstale":"2024-01-27T01:06:28.666039+0000","last_undegraded":"2024-01-27T01:06:28.666039+0000","last_fullsized":"2024-01-27T01:06:28.666039+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:10:59.780192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666343+0000","last_change":"2024-01-27T01:06:26.880503+0000","last_active":"2024-01-27T01:06:28.666343+0000","last_peered":"2024-01-27T01:06:28.666343+0000","last_clean":"2024-01-27T01:06:28.666343+0000","last_became_active":"2024-01-27T01:06:26.880224+0000","last_became_peered":"2024-01-27T01:06:26.880224+0000","last_unstale":"2024-01-27T01:06:28.666343+0000","last_undegraded":"2024-01-27T01:06:28.666343+0000","last_fullsized":"2024-01-27T01:06:28.666343+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:27:33.886740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.911198+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:26:34.409044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665860+0000","last_change":"2024-01-27T01:06:25.859342+0000","last_active":"2024-01-27T01:06:28.665860+0000","last_peered":"2024-01-27T01:06:28.665860+0000","last_clean":"2024-01-27T01:06:28.665860+0000","last_became_active":"2024-01-27T01:06:25.859065+0000","last_became_peered":"2024-01-27T01:06:25.859065+0000","last_unstale":"2024-01-27T01:06:28.665860+0000","last_undegraded":"2024-01-27T01:06:28.665860+0000","last_fullsized":"2024-01-27T01:06:28.665860+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:00:54.297085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666399+0000","last_change":"2024-01-27T01:06:26.870597+0000","last_active":"2024-01-27T01:06:28.666399+0000","last_peered":"2024-01-27T01:06:28.666399+0000","last_clean":"2024-01-27T01:06:28.666399+0000","last_became_active":"2024-01-27T01:06:26.870334+0000","last_became_peered":"2024-01-27T01:06:26.870334+0000","last_unstale":"2024-01-27T01:06:28.666399+0000","last_undegraded":"2024-01-27T01:06:28.666399+0000","last_fullsized":"2024-01-27T01:06:28.666399+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:01:23.266072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913094+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:08:04.706593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355235+0000","last_change":"2024-01-27T01:06:25.858032+0000","last_active":"2024-01-27T01:06:28.355235+0000","last_peered":"2024-01-27T01:06:28.355235+0000","last_clean":"2024-01-27T01:06:28.355235+0000","last_became_active":"2024-01-27T01:06:25.857777+0000","last_became_peered":"2024-01-27T01:06:25.857777+0000","last_unstale":"2024-01-27T01:06:28.355235+0000","last_undegraded":"2024-01-27T01:06:28.355235+0000","last_fullsized":"2024-01-27T01:06:28.355235+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:09:08.458850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865801+0000","last_change":"2024-01-27T01:06:26.872190+0000","last_active":"2024-01-27T01:06:27.865801+0000","last_peered":"2024-01-27T01:06:27.865801+0000","last_clean":"2024-01-27T01:06:27.865801+0000","last_became_active":"2024-01-27T01:06:26.867718+0000","last_became_peered":"2024-01-27T01:06:26.867718+0000","last_unstale":"2024-01-27T01:06:27.865801+0000","last_undegraded":"2024-01-27T01:06:27.865801+0000","last_fullsized":"2024-01-27T01:06:27.865801+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:23.301132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.915265+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:55:49.427110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.6","version":"223'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:31.798965+0000","last_change":"2024-01-27T01:06:25.861717+0000","last_active":"2024-01-27T01:06:31.798965+0000","last_peered":"2024-01-27T01:06:31.798965+0000","last_clean":"2024-01-27T01:06:31.798965+0000","last_became_active":"2024-01-27T01:06:25.861177+0000","last_became_peered":"2024-01-27T01:06:25.861177+0000","last_unstale":"2024-01-27T01:06:31.798965+0000","last_undegraded":"2024-01-27T01:06:31.798965+0000","last_fullsized":"2024-01-27T01:06:31.798965+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:05.596045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240815+0000","last_change":"2024-01-27T01:06:26.878650+0000","last_active":"2024-01-27T01:06:28.240815+0000","last_peered":"2024-01-27T01:06:28.240815+0000","last_clean":"2024-01-27T01:06:28.240815+0000","last_became_active":"2024-01-27T01:06:26.877596+0000","last_became_peered":"2024-01-27T01:06:26.877596+0000","last_unstale":"2024-01-27T01:06:28.240815+0000","last_undegraded":"2024-01-27T01:06:28.240815+0000","last_fullsized":"2024-01-27T01:06:28.240815+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:42:48.942202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.914590+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:04.634853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.7","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866083+0000","last_change":"2024-01-27T01:06:25.857933+0000","last_active":"2024-01-27T01:06:27.866083+0000","last_peered":"2024-01-27T01:06:27.866083+0000","last_clean":"2024-01-27T01:06:27.866083+0000","last_became_active":"2024-01-27T01:06:25.857406+0000","last_became_peered":"2024-01-27T01:06:25.857406+0000","last_unstale":"2024-01-27T01:06:27.866083+0000","last_undegraded":"2024-01-27T01:06:27.866083+0000","last_fullsized":"2024-01-27T01:06:27.866083+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:20:20.239633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665632+0000","last_change":"2024-01-27T01:06:26.875222+0000","last_active":"2024-01-27T01:06:28.665632+0000","last_peered":"2024-01-27T01:06:28.665632+0000","last_clean":"2024-01-27T01:06:28.665632+0000","last_became_active":"2024-01-27T01:06:26.874903+0000","last_became_peered":"2024-01-27T01:06:26.874903+0000","last_unstale":"2024-01-27T01:06:28.665632+0000","last_undegraded":"2024-01-27T01:06:28.665632+0000","last_fullsized":"2024-01-27T01:06:28.665632+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:09:33.903445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916305+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:24.566246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665720+0000","last_change":"2024-01-27T01:06:25.853605+0000","last_active":"2024-01-27T01:06:28.665720+0000","last_peered":"2024-01-27T01:06:28.665720+0000","last_clean":"2024-01-27T01:06:28.665720+0000","last_became_active":"2024-01-27T01:06:25.853331+0000","last_became_peered":"2024-01-27T01:06:25.853331+0000","last_unstale":"2024-01-27T01:06:28.665720+0000","last_undegraded":"2024-01-27T01:06:28.665720+0000","last_fullsized":"2024-01-27T01:06:28.665720+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:09:08.458850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666459+0000","last_change":"2024-01-27T01:06:26.875536+0000","last_active":"2024-01-27T01:06:28.666459+0000","last_peered":"2024-01-27T01:06:28.666459+0000","last_clean":"2024-01-27T01:06:28.666459+0000","last_became_active":"2024-01-27T01:06:26.875267+0000","last_became_peered":"2024-01-27T01:06:26.875267+0000","last_unstale":"2024-01-27T01:06:28.666459+0000","last_undegraded":"2024-01-27T01:06:28.666459+0000","last_fullsized":"2024-01-27T01:06:28.666459+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:32.534622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916585+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:37:20.044223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355214+0000","last_change":"2024-01-27T01:06:25.862199+0000","last_active":"2024-01-27T01:06:28.355214+0000","last_peered":"2024-01-27T01:06:28.355214+0000","last_clean":"2024-01-27T01:06:28.355214+0000","last_became_active":"2024-01-27T01:06:25.861876+0000","last_became_peered":"2024-01-27T01:06:25.861876+0000","last_unstale":"2024-01-27T01:06:28.355214+0000","last_undegraded":"2024-01-27T01:06:28.355214+0000","last_fullsized":"2024-01-27T01:06:28.355214+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:00:54.297085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865614+0000","last_change":"2024-01-27T01:06:26.867570+0000","last_active":"2024-01-27T01:06:27.865614+0000","last_peered":"2024-01-27T01:06:27.865614+0000","last_clean":"2024-01-27T01:06:27.865614+0000","last_became_active":"2024-01-27T01:06:26.866517+0000","last_became_peered":"2024-01-27T01:06:26.866517+0000","last_unstale":"2024-01-27T01:06:27.865614+0000","last_undegraded":"2024-01-27T01:06:27.865614+0000","last_fullsized":"2024-01-27T01:06:27.865614+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:09:54.515702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910624+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:47:30.145789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866689+0000","last_change":"2024-01-27T01:06:25.851595+0000","last_active":"2024-01-27T01:06:27.866689+0000","last_peered":"2024-01-27T01:06:27.866689+0000","last_clean":"2024-01-27T01:06:27.866689+0000","last_became_active":"2024-01-27T01:06:25.851260+0000","last_became_peered":"2024-01-27T01:06:25.851260+0000","last_unstale":"2024-01-27T01:06:27.866689+0000","last_undegraded":"2024-01-27T01:06:27.866689+0000","last_fullsized":"2024-01-27T01:06:27.866689+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:56:53.179347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038203+0000","last_change":"2024-01-27T01:06:26.875858+0000","last_active":"2024-01-27T01:06:28.038203+0000","last_peered":"2024-01-27T01:06:28.038203+0000","last_clean":"2024-01-27T01:06:28.038203+0000","last_became_active":"2024-01-27T01:06:26.875564+0000","last_became_peered":"2024-01-27T01:06:26.875564+0000","last_unstale":"2024-01-27T01:06:28.038203+0000","last_undegraded":"2024-01-27T01:06:28.038203+0000","last_fullsized":"2024-01-27T01:06:28.038203+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:43.630831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913526+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:07:02.246428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.b","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866591+0000","last_change":"2024-01-27T01:06:25.860572+0000","last_active":"2024-01-27T01:06:27.866591+0000","last_peered":"2024-01-27T01:06:27.866591+0000","last_clean":"2024-01-27T01:06:27.866591+0000","last_became_active":"2024-01-27T01:06:25.860248+0000","last_became_peered":"2024-01-27T01:06:25.860248+0000","last_unstale":"2024-01-27T01:06:27.866591+0000","last_undegraded":"2024-01-27T01:06:27.866591+0000","last_fullsized":"2024-01-27T01:06:27.866591+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:01:25.741123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866157+0000","last_change":"2024-01-27T01:06:26.871346+0000","last_active":"2024-01-27T01:06:27.866157+0000","last_peered":"2024-01-27T01:06:27.866157+0000","last_clean":"2024-01-27T01:06:27.866157+0000","last_became_active":"2024-01-27T01:06:26.870999+0000","last_became_peered":"2024-01-27T01:06:26.870999+0000","last_unstale":"2024-01-27T01:06:27.866157+0000","last_undegraded":"2024-01-27T01:06:27.866157+0000","last_fullsized":"2024-01-27T01:06:27.866157+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:27.779157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.914309+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:31:18.366038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866703+0000","last_change":"2024-01-27T01:06:25.861650+0000","last_active":"2024-01-27T01:06:27.866703+0000","last_peered":"2024-01-27T01:06:27.866703+0000","last_clean":"2024-01-27T01:06:27.866703+0000","last_became_active":"2024-01-27T01:06:25.861342+0000","last_became_peered":"2024-01-27T01:06:25.861342+0000","last_unstale":"2024-01-27T01:06:27.866703+0000","last_undegraded":"2024-01-27T01:06:27.866703+0000","last_fullsized":"2024-01-27T01:06:27.866703+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:44:15.953217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665953+0000","last_change":"2024-01-27T01:06:26.876619+0000","last_active":"2024-01-27T01:06:28.665953+0000","last_peered":"2024-01-27T01:06:28.665953+0000","last_clean":"2024-01-27T01:06:28.665953+0000","last_became_active":"2024-01-27T01:06:26.875927+0000","last_became_peered":"2024-01-27T01:06:26.875927+0000","last_unstale":"2024-01-27T01:06:28.665953+0000","last_undegraded":"2024-01-27T01:06:28.665953+0000","last_fullsized":"2024-01-27T01:06:28.665953+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:02.000909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.911280+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:43:07.431223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.d","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.867031+0000","last_change":"2024-01-27T01:06:25.851720+0000","last_active":"2024-01-27T01:06:27.867031+0000","last_peered":"2024-01-27T01:06:27.867031+0000","last_clean":"2024-01-27T01:06:27.867031+0000","last_became_active":"2024-01-27T01:06:25.851416+0000","last_became_peered":"2024-01-27T01:06:25.851416+0000","last_unstale":"2024-01-27T01:06:27.867031+0000","last_undegraded":"2024-01-27T01:06:27.867031+0000","last_fullsized":"2024-01-27T01:06:27.867031+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:07.951814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666130+0000","last_change":"2024-01-27T01:06:26.870758+0000","last_active":"2024-01-27T01:06:28.666130+0000","last_peered":"2024-01-27T01:06:28.666130+0000","last_clean":"2024-01-27T01:06:28.666130+0000","last_became_active":"2024-01-27T01:06:26.870130+0000","last_became_peered":"2024-01-27T01:06:26.870130+0000","last_unstale":"2024-01-27T01:06:28.666130+0000","last_undegraded":"2024-01-27T01:06:28.666130+0000","last_fullsized":"2024-01-27T01:06:28.666130+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:36:31.625857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913563+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:49:22.695065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.e","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665974+0000","last_change":"2024-01-27T01:06:25.852245+0000","last_active":"2024-01-27T01:06:28.665974+0000","last_peered":"2024-01-27T01:06:28.665974+0000","last_clean":"2024-01-27T01:06:28.665974+0000","last_became_active":"2024-01-27T01:06:25.851982+0000","last_became_peered":"2024-01-27T01:06:25.851982+0000","last_unstale":"2024-01-27T01:06:28.665974+0000","last_undegraded":"2024-01-27T01:06:28.665974+0000","last_fullsized":"2024-01-27T01:06:28.665974+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:05.596045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665930+0000","last_change":"2024-01-27T01:06:26.879356+0000","last_active":"2024-01-27T01:06:28.665930+0000","last_peered":"2024-01-27T01:06:28.665930+0000","last_clean":"2024-01-27T01:06:28.665930+0000","last_became_active":"2024-01-27T01:06:26.878891+0000","last_became_peered":"2024-01-27T01:06:26.878891+0000","last_unstale":"2024-01-27T01:06:28.665930+0000","last_undegraded":"2024-01-27T01:06:28.665930+0000","last_fullsized":"2024-01-27T01:06:28.665930+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:22.011928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916563+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:01:36.332606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.f","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666321+0000","last_change":"2024-01-27T01:06:25.848900+0000","last_active":"2024-01-27T01:06:28.666321+0000","last_peered":"2024-01-27T01:06:28.666321+0000","last_clean":"2024-01-27T01:06:28.666321+0000","last_became_active":"2024-01-27T01:06:25.848483+0000","last_became_peered":"2024-01-27T01:06:25.848483+0000","last_unstale":"2024-01-27T01:06:28.666321+0000","last_undegraded":"2024-01-27T01:06:28.666321+0000","last_fullsized":"2024-01-27T01:06:28.666321+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:27:13.471892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665407+0000","last_change":"2024-01-27T01:06:26.875234+0000","last_active":"2024-01-27T01:06:28.665407+0000","last_peered":"2024-01-27T01:06:28.665407+0000","last_clean":"2024-01-27T01:06:28.665407+0000","last_became_active":"2024-01-27T01:06:26.874927+0000","last_became_peered":"2024-01-27T01:06:26.874927+0000","last_unstale":"2024-01-27T01:06:28.665407+0000","last_undegraded":"2024-01-27T01:06:28.665407+0000","last_fullsized":"2024-01-27T01:06:28.665407+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:10.390573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.911685+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:31.265591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.10","version":"222'8","reported_seq":19,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865571+0000","last_change":"2024-01-27T01:06:25.865239+0000","last_active":"2024-01-27T01:06:27.865571+0000","last_peered":"2024-01-27T01:06:27.865571+0000","last_clean":"2024-01-27T01:06:27.865571+0000","last_became_active":"2024-01-27T01:06:25.864943+0000","last_became_peered":"2024-01-27T01:06:25.864943+0000","last_unstale":"2024-01-27T01:06:27.865571+0000","last_undegraded":"2024-01-27T01:06:27.865571+0000","last_fullsized":"2024-01-27T01:06:27.865571+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:41:25.254751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038170+0000","last_change":"2024-01-27T01:06:26.867077+0000","last_active":"2024-01-27T01:06:28.038170+0000","last_peered":"2024-01-27T01:06:28.038170+0000","last_clean":"2024-01-27T01:06:28.038170+0000","last_became_active":"2024-01-27T01:06:26.866787+0000","last_became_peered":"2024-01-27T01:06:26.866787+0000","last_unstale":"2024-01-27T01:06:28.038170+0000","last_undegraded":"2024-01-27T01:06:28.038170+0000","last_fullsized":"2024-01-27T01:06:28.038170+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:02.000909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913156+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:56.697365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865574+0000","last_change":"2024-01-27T01:06:25.853597+0000","last_active":"2024-01-27T01:06:27.865574+0000","last_peered":"2024-01-27T01:06:27.865574+0000","last_clean":"2024-01-27T01:06:27.865574+0000","last_became_active":"2024-01-27T01:06:25.853257+0000","last_became_peered":"2024-01-27T01:06:25.853257+0000","last_unstale":"2024-01-27T01:06:27.865574+0000","last_undegraded":"2024-01-27T01:06:27.865574+0000","last_fullsized":"2024-01-27T01:06:27.865574+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:54:35.011796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865516+0000","last_change":"2024-01-27T01:06:26.866613+0000","last_active":"2024-01-27T01:06:27.865516+0000","last_peered":"2024-01-27T01:06:27.865516+0000","last_clean":"2024-01-27T01:06:27.865516+0000","last_became_active":"2024-01-27T01:06:26.866298+0000","last_became_peered":"2024-01-27T01:06:26.866298+0000","last_unstale":"2024-01-27T01:06:27.865516+0000","last_undegraded":"2024-01-27T01:06:27.865516+0000","last_fullsized":"2024-01-27T01:06:27.865516+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:27:03.934654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.917128+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:57:15.231912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240972+0000","last_change":"2024-01-27T01:06:25.854192+0000","last_active":"2024-01-27T01:06:28.240972+0000","last_peered":"2024-01-27T01:06:28.240972+0000","last_clean":"2024-01-27T01:06:28.240972+0000","last_became_active":"2024-01-27T01:06:25.853601+0000","last_became_peered":"2024-01-27T01:06:25.853601+0000","last_unstale":"2024-01-27T01:06:28.240972+0000","last_undegraded":"2024-01-27T01:06:28.240972+0000","last_fullsized":"2024-01-27T01:06:28.240972+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:42.218792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.241137+0000","last_change":"2024-01-27T01:06:26.862905+0000","last_active":"2024-01-27T01:06:28.241137+0000","last_peered":"2024-01-27T01:06:28.241137+0000","last_clean":"2024-01-27T01:06:28.241137+0000","last_became_active":"2024-01-27T01:06:26.862584+0000","last_became_peered":"2024-01-27T01:06:26.862584+0000","last_unstale":"2024-01-27T01:06:28.241137+0000","last_undegraded":"2024-01-27T01:06:28.241137+0000","last_fullsized":"2024-01-27T01:06:28.241137+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:27:14.653781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910329+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:19:00.298602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.13","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866863+0000","last_change":"2024-01-27T01:06:25.850061+0000","last_active":"2024-01-27T01:06:27.866863+0000","last_peered":"2024-01-27T01:06:27.866863+0000","last_clean":"2024-01-27T01:06:27.866863+0000","last_became_active":"2024-01-27T01:06:25.849763+0000","last_became_peered":"2024-01-27T01:06:25.849763+0000","last_unstale":"2024-01-27T01:06:27.866863+0000","last_undegraded":"2024-01-27T01:06:27.866863+0000","last_fullsized":"2024-01-27T01:06:27.866863+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:09:56.243562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866051+0000","last_change":"2024-01-27T01:06:26.873464+0000","last_active":"2024-01-27T01:06:27.866051+0000","last_peered":"2024-01-27T01:06:27.866051+0000","last_clean":"2024-01-27T01:06:27.866051+0000","last_became_active":"2024-01-27T01:06:26.873063+0000","last_became_peered":"2024-01-27T01:06:26.873063+0000","last_unstale":"2024-01-27T01:06:27.866051+0000","last_undegraded":"2024-01-27T01:06:27.866051+0000","last_fullsized":"2024-01-27T01:06:27.866051+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:33:32.535574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916917+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:36:16.613510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.14","version":"223'8","reported_seq":19,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:36.798321+0000","last_change":"2024-01-27T01:06:25.864160+0000","last_active":"2024-01-27T01:06:36.798321+0000","last_peered":"2024-01-27T01:06:36.798321+0000","last_clean":"2024-01-27T01:06:36.798321+0000","last_became_active":"2024-01-27T01:06:25.863867+0000","last_became_peered":"2024-01-27T01:06:25.863867+0000","last_unstale":"2024-01-27T01:06:36.798321+0000","last_undegraded":"2024-01-27T01:06:36.798321+0000","last_fullsized":"2024-01-27T01:06:36.798321+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:43.338484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2768,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866494+0000","last_change":"2024-01-27T01:06:26.879652+0000","last_active":"2024-01-27T01:06:27.866494+0000","last_peered":"2024-01-27T01:06:27.866494+0000","last_clean":"2024-01-27T01:06:27.866494+0000","last_became_active":"2024-01-27T01:06:26.879271+0000","last_became_peered":"2024-01-27T01:06:26.879271+0000","last_unstale":"2024-01-27T01:06:27.866494+0000","last_undegraded":"2024-01-27T01:06:27.866494+0000","last_fullsized":"2024-01-27T01:06:27.866494+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:57:23.456731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.914165+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:27:24.223960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.15","version":"222'3","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866134+0000","last_change":"2024-01-27T01:06:25.857771+0000","last_active":"2024-01-27T01:06:27.866134+0000","last_peered":"2024-01-27T01:06:27.866134+0000","last_clean":"2024-01-27T01:06:27.866134+0000","last_became_active":"2024-01-27T01:06:25.857445+0000","last_became_peered":"2024-01-27T01:06:25.857445+0000","last_unstale":"2024-01-27T01:06:27.866134+0000","last_undegraded":"2024-01-27T01:06:27.866134+0000","last_fullsized":"2024-01-27T01:06:27.866134+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:52.927866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865698+0000","last_change":"2024-01-27T01:06:26.876685+0000","last_active":"2024-01-27T01:06:27.865698+0000","last_peered":"2024-01-27T01:06:27.865698+0000","last_clean":"2024-01-27T01:06:27.865698+0000","last_became_active":"2024-01-27T01:06:26.872939+0000","last_became_peered":"2024-01-27T01:06:26.872939+0000","last_unstale":"2024-01-27T01:06:27.865698+0000","last_undegraded":"2024-01-27T01:06:27.865698+0000","last_fullsized":"2024-01-27T01:06:27.865698+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:56:48.730726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.915793+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:36:16.613510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038259+0000","last_change":"2024-01-27T01:06:25.860762+0000","last_active":"2024-01-27T01:06:28.038259+0000","last_peered":"2024-01-27T01:06:28.038259+0000","last_clean":"2024-01-27T01:06:28.038259+0000","last_became_active":"2024-01-27T01:06:25.860459+0000","last_became_peered":"2024-01-27T01:06:25.860459+0000","last_unstale":"2024-01-27T01:06:28.038259+0000","last_undegraded":"2024-01-27T01:06:28.038259+0000","last_fullsized":"2024-01-27T01:06:28.038259+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:31.687265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240842+0000","last_change":"2024-01-27T01:06:26.868859+0000","last_active":"2024-01-27T01:06:28.240842+0000","last_peered":"2024-01-27T01:06:28.240842+0000","last_clean":"2024-01-27T01:06:28.240842+0000","last_became_active":"2024-01-27T01:06:26.868623+0000","last_became_peered":"2024-01-27T01:06:26.868623+0000","last_unstale":"2024-01-27T01:06:28.240842+0000","last_undegraded":"2024-01-27T01:06:28.240842+0000","last_fullsized":"2024-01-27T01:06:28.240842+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:18:47.232067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.915065+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:24.566246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866791+0000","last_change":"2024-01-27T01:06:25.865314+0000","last_active":"2024-01-27T01:06:27.866791+0000","last_peered":"2024-01-27T01:06:27.866791+0000","last_clean":"2024-01-27T01:06:27.866791+0000","last_became_active":"2024-01-27T01:06:25.864940+0000","last_became_peered":"2024-01-27T01:06:25.864940+0000","last_unstale":"2024-01-27T01:06:27.866791+0000","last_undegraded":"2024-01-27T01:06:27.866791+0000","last_fullsized":"2024-01-27T01:06:27.866791+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:24.113986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355011+0000","last_change":"2024-01-27T01:06:26.870806+0000","last_active":"2024-01-27T01:06:28.355011+0000","last_peered":"2024-01-27T01:06:28.355011+0000","last_clean":"2024-01-27T01:06:28.355011+0000","last_became_active":"2024-01-27T01:06:26.870263+0000","last_became_peered":"2024-01-27T01:06:26.870263+0000","last_unstale":"2024-01-27T01:06:28.355011+0000","last_undegraded":"2024-01-27T01:06:28.355011+0000","last_fullsized":"2024-01-27T01:06:28.355011+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:26.214953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910328+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:24.032004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240953+0000","last_change":"2024-01-27T01:06:25.851888+0000","last_active":"2024-01-27T01:06:28.240953+0000","last_peered":"2024-01-27T01:06:28.240953+0000","last_clean":"2024-01-27T01:06:28.240953+0000","last_became_active":"2024-01-27T01:06:25.851605+0000","last_became_peered":"2024-01-27T01:06:25.851605+0000","last_unstale":"2024-01-27T01:06:28.240953+0000","last_undegraded":"2024-01-27T01:06:28.240953+0000","last_fullsized":"2024-01-27T01:06:28.240953+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:09.386094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865985+0000","last_change":"2024-01-27T01:06:26.873199+0000","last_active":"2024-01-27T01:06:27.865985+0000","last_peered":"2024-01-27T01:06:27.865985+0000","last_clean":"2024-01-27T01:06:27.865985+0000","last_became_active":"2024-01-27T01:06:26.872658+0000","last_became_peered":"2024-01-27T01:06:26.872658+0000","last_unstale":"2024-01-27T01:06:27.865985+0000","last_undegraded":"2024-01-27T01:06:27.865985+0000","last_fullsized":"2024-01-27T01:06:27.865985+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:06:45.526347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913764+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:10:10.314575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.037877+0000","last_change":"2024-01-27T01:06:25.854878+0000","last_active":"2024-01-27T01:06:28.037877+0000","last_peered":"2024-01-27T01:06:28.037877+0000","last_clean":"2024-01-27T01:06:28.037877+0000","last_became_active":"2024-01-27T01:06:25.854623+0000","last_became_peered":"2024-01-27T01:06:25.854623+0000","last_unstale":"2024-01-27T01:06:28.037877+0000","last_undegraded":"2024-01-27T01:06:28.037877+0000","last_fullsized":"2024-01-27T01:06:28.037877+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:21.007449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355292+0000","last_change":"2024-01-27T01:06:26.876963+0000","last_active":"2024-01-27T01:06:28.355292+0000","last_peered":"2024-01-27T01:06:28.355292+0000","last_clean":"2024-01-27T01:06:28.355292+0000","last_became_active":"2024-01-27T01:06:26.876709+0000","last_became_peered":"2024-01-27T01:06:26.876709+0000","last_unstale":"2024-01-27T01:06:28.355292+0000","last_undegraded":"2024-01-27T01:06:28.355292+0000","last_fullsized":"2024-01-27T01:06:28.355292+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:58:14.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916960+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:57.409497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1b","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038235+0000","last_change":"2024-01-27T01:06:25.854193+0000","last_active":"2024-01-27T01:06:28.038235+0000","last_peered":"2024-01-27T01:06:28.038235+0000","last_clean":"2024-01-27T01:06:28.038235+0000","last_became_active":"2024-01-27T01:06:25.853956+0000","last_became_peered":"2024-01-27T01:06:25.853956+0000","last_unstale":"2024-01-27T01:06:28.038235+0000","last_undegraded":"2024-01-27T01:06:28.038235+0000","last_fullsized":"2024-01-27T01:06:28.038235+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:33:02.747087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355322+0000","last_change":"2024-01-27T01:06:26.866121+0000","last_active":"2024-01-27T01:06:28.355322+0000","last_peered":"2024-01-27T01:06:28.355322+0000","last_clean":"2024-01-27T01:06:28.355322+0000","last_became_active":"2024-01-27T01:06:26.865391+0000","last_became_peered":"2024-01-27T01:06:26.865391+0000","last_unstale":"2024-01-27T01:06:28.355322+0000","last_undegraded":"2024-01-27T01:06:28.355322+0000","last_fullsized":"2024-01-27T01:06:28.355322+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:02:03.812030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910329+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:27:27.720315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1c","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866341+0000","last_change":"2024-01-27T01:06:25.855114+0000","last_active":"2024-01-27T01:06:27.866341+0000","last_peered":"2024-01-27T01:06:27.866341+0000","last_clean":"2024-01-27T01:06:27.866341+0000","last_became_active":"2024-01-27T01:06:25.854790+0000","last_became_peered":"2024-01-27T01:06:25.854790+0000","last_unstale":"2024-01-27T01:06:27.866341+0000","last_undegraded":"2024-01-27T01:06:27.866341+0000","last_fullsized":"2024-01-27T01:06:27.866341+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:58:58.498377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866394+0000","last_change":"2024-01-27T01:06:26.869128+0000","last_active":"2024-01-27T01:06:27.866394+0000","last_peered":"2024-01-27T01:06:27.866394+0000","last_clean":"2024-01-27T01:06:27.866394+0000","last_became_active":"2024-01-27T01:06:26.868892+0000","last_became_peered":"2024-01-27T01:06:26.868892+0000","last_unstale":"2024-01-27T01:06:27.866394+0000","last_undegraded":"2024-01-27T01:06:27.866394+0000","last_fullsized":"2024-01-27T01:06:27.866394+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:40:40.869516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916917+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:31.265591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1d","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866926+0000","last_change":"2024-01-27T01:06:25.855991+0000","last_active":"2024-01-27T01:06:27.866926+0000","last_peered":"2024-01-27T01:06:27.866926+0000","last_clean":"2024-01-27T01:06:27.866926+0000","last_became_active":"2024-01-27T01:06:25.855679+0000","last_became_peered":"2024-01-27T01:06:25.855679+0000","last_unstale":"2024-01-27T01:06:27.866926+0000","last_undegraded":"2024-01-27T01:06:27.866926+0000","last_fullsized":"2024-01-27T01:06:27.866926+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:31:04.295025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666155+0000","last_change":"2024-01-27T01:06:26.875831+0000","last_active":"2024-01-27T01:06:28.666155+0000","last_peered":"2024-01-27T01:06:28.666155+0000","last_clean":"2024-01-27T01:06:28.666155+0000","last_became_active":"2024-01-27T01:06:26.875488+0000","last_became_peered":"2024-01-27T01:06:26.875488+0000","last_unstale":"2024-01-27T01:06:28.666155+0000","last_undegraded":"2024-01-27T01:06:28.666155+0000","last_fullsized":"2024-01-27T01:06:28.666155+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:07:51.640059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910981+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:57:15.231912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1e","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665879+0000","last_change":"2024-01-27T01:06:25.856504+0000","last_active":"2024-01-27T01:06:28.665879+0000","last_peered":"2024-01-27T01:06:28.665879+0000","last_clean":"2024-01-27T01:06:28.665879+0000","last_became_active":"2024-01-27T01:06:25.856272+0000","last_became_peered":"2024-01-27T01:06:25.856272+0000","last_unstale":"2024-01-27T01:06:28.665879+0000","last_undegraded":"2024-01-27T01:06:28.665879+0000","last_fullsized":"2024-01-27T01:06:28.665879+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:20:56.736428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666038+0000","last_change":"2024-01-27T01:06:26.878625+0000","last_active":"2024-01-27T01:06:28.666038+0000","last_peered":"2024-01-27T01:06:28.666038+0000","last_clean":"2024-01-27T01:06:28.666038+0000","last_became_active":"2024-01-27T01:06:26.878361+0000","last_became_peered":"2024-01-27T01:06:26.878361+0000","last_unstale":"2024-01-27T01:06:28.666038+0000","last_undegraded":"2024-01-27T01:06:28.666038+0000","last_fullsized":"2024-01-27T01:06:28.666038+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:37:06.977689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910594+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:31:50.703251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1f","version":"222'4","reported_seq":15,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355378+0000","last_change":"2024-01-27T01:06:25.856458+0000","last_active":"2024-01-27T01:06:28.355378+0000","last_peered":"2024-01-27T01:06:28.355378+0000","last_clean":"2024-01-27T01:06:28.355378+0000","last_became_active":"2024-01-27T01:06:25.856162+0000","last_became_peered":"2024-01-27T01:06:25.856162+0000","last_unstale":"2024-01-27T01:06:28.355378+0000","last_undegraded":"2024-01-27T01:06:28.355378+0000","last_fullsized":"2024-01-27T01:06:28.355378+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:51:44.378880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.867070+0000","last_change":"2024-01-27T01:06:26.869588+0000","last_active":"2024-01-27T01:06:27.867070+0000","last_peered":"2024-01-27T01:06:27.867070+0000","last_clean":"2024-01-27T01:06:27.867070+0000","last_became_active":"2024-01-27T01:06:26.869264+0000","last_became_peered":"2024-01-27T01:06:26.869264+0000","last_unstale":"2024-01-27T01:06:27.867070+0000","last_undegraded":"2024-01-27T01:06:27.867070+0000","last_fullsized":"2024-01-27T01:06:27.867070+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:42:54.364688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913160+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:15.067443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.037884+0000","last_change":"2024-01-27T01:06:26.869029+0000","last_active":"2024-01-27T01:06:28.037884+0000","last_peered":"2024-01-27T01:06:28.037884+0000","last_clean":"2024-01-27T01:06:28.037884+0000","last_became_active":"2024-01-27T01:06:26.868793+0000","last_became_peered":"2024-01-27T01:06:26.868793+0000","last_unstale":"2024-01-27T01:06:28.037884+0000","last_undegraded":"2024-01-27T01:06:28.037884+0000","last_fullsized":"2024-01-27T01:06:28.037884+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:10.965469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.037888+0000","last_change":"2024-01-27T01:06:26.876343+0000","last_active":"2024-01-27T01:06:28.037888+0000","last_peered":"2024-01-27T01:06:28.037888+0000","last_clean":"2024-01-27T01:06:28.037888+0000","last_became_active":"2024-01-27T01:06:26.875570+0000","last_became_peered":"2024-01-27T01:06:26.875570+0000","last_unstale":"2024-01-27T01:06:28.037888+0000","last_undegraded":"2024-01-27T01:06:28.037888+0000","last_fullsized":"2024-01-27T01:06:28.037888+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:07:51.640059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355113+0000","last_change":"2024-01-27T01:06:26.871676+0000","last_active":"2024-01-27T01:06:28.355113+0000","last_peered":"2024-01-27T01:06:28.355113+0000","last_clean":"2024-01-27T01:06:28.355113+0000","last_became_active":"2024-01-27T01:06:26.871423+0000","last_became_peered":"2024-01-27T01:06:26.871423+0000","last_unstale":"2024-01-27T01:06:28.355113+0000","last_undegraded":"2024-01-27T01:06:28.355113+0000","last_fullsized":"2024-01-27T01:06:28.355113+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:29:58.146876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866078+0000","last_change":"2024-01-27T01:06:26.872013+0000","last_active":"2024-01-27T01:06:27.866078+0000","last_peered":"2024-01-27T01:06:27.866078+0000","last_clean":"2024-01-27T01:06:27.866078+0000","last_became_active":"2024-01-27T01:06:26.867577+0000","last_became_peered":"2024-01-27T01:06:26.867577+0000","last_unstale":"2024-01-27T01:06:27.866078+0000","last_undegraded":"2024-01-27T01:06:27.866078+0000","last_fullsized":"2024-01-27T01:06:27.866078+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:27.604574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866848+0000","last_change":"2024-01-27T01:06:26.870649+0000","last_active":"2024-01-27T01:06:27.866848+0000","last_peered":"2024-01-27T01:06:27.866848+0000","last_clean":"2024-01-27T01:06:27.866848+0000","last_became_active":"2024-01-27T01:06:26.870433+0000","last_became_peered":"2024-01-27T01:06:26.870433+0000","last_unstale":"2024-01-27T01:06:27.866848+0000","last_undegraded":"2024-01-27T01:06:27.866848+0000","last_fullsized":"2024-01-27T01:06:27.866848+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:11:27.402943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.241052+0000","last_change":"2024-01-27T01:06:26.878202+0000","last_active":"2024-01-27T01:06:28.241052+0000","last_peered":"2024-01-27T01:06:28.241052+0000","last_clean":"2024-01-27T01:06:28.241052+0000","last_became_active":"2024-01-27T01:06:26.877900+0000","last_became_peered":"2024-01-27T01:06:26.877900+0000","last_unstale":"2024-01-27T01:06:28.241052+0000","last_undegraded":"2024-01-27T01:06:28.241052+0000","last_fullsized":"2024-01-27T01:06:28.241052+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:42.766155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665916+0000","last_change":"2024-01-27T01:06:26.878911+0000","last_active":"2024-01-27T01:06:28.665916+0000","last_peered":"2024-01-27T01:06:28.665916+0000","last_clean":"2024-01-27T01:06:28.665916+0000","last_became_active":"2024-01-27T01:06:26.878637+0000","last_became_peered":"2024-01-27T01:06:26.878637+0000","last_unstale":"2024-01-27T01:06:28.665916+0000","last_undegraded":"2024-01-27T01:06:28.665916+0000","last_fullsized":"2024-01-27T01:06:28.665916+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:33:03.751565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865886+0000","last_change":"2024-01-27T01:06:26.865149+0000","last_active":"2024-01-27T01:06:27.865886+0000","last_peered":"2024-01-27T01:06:27.865886+0000","last_clean":"2024-01-27T01:06:27.865886+0000","last_became_active":"2024-01-27T01:06:26.864853+0000","last_became_peered":"2024-01-27T01:06:26.864853+0000","last_unstale":"2024-01-27T01:06:27.865886+0000","last_undegraded":"2024-01-27T01:06:27.865886+0000","last_fullsized":"2024-01-27T01:06:27.865886+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:10:44.194323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865471+0000","last_change":"2024-01-27T01:06:26.867241+0000","last_active":"2024-01-27T01:06:27.865471+0000","last_peered":"2024-01-27T01:06:27.865471+0000","last_clean":"2024-01-27T01:06:27.865471+0000","last_became_active":"2024-01-27T01:06:26.866917+0000","last_became_peered":"2024-01-27T01:06:26.866917+0000","last_unstale":"2024-01-27T01:06:27.865471+0000","last_undegraded":"2024-01-27T01:06:27.865471+0000","last_fullsized":"2024-01-27T01:06:27.865471+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:24:25.818592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865972+0000","last_change":"2024-01-27T01:06:26.874471+0000","last_active":"2024-01-27T01:06:27.865972+0000","last_peered":"2024-01-27T01:06:27.865972+0000","last_clean":"2024-01-27T01:06:27.865972+0000","last_became_active":"2024-01-27T01:06:26.874271+0000","last_became_peered":"2024-01-27T01:06:26.874271+0000","last_unstale":"2024-01-27T01:06:27.865972+0000","last_undegraded":"2024-01-27T01:06:27.865972+0000","last_fullsized":"2024-01-27T01:06:27.865972+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:15:11.295547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665512+0000","last_change":"2024-01-27T01:06:26.876390+0000","last_active":"2024-01-27T01:06:28.665512+0000","last_peered":"2024-01-27T01:06:28.665512+0000","last_clean":"2024-01-27T01:06:28.665512+0000","last_became_active":"2024-01-27T01:06:26.876056+0000","last_became_peered":"2024-01-27T01:06:26.876056+0000","last_unstale":"2024-01-27T01:06:28.665512+0000","last_undegraded":"2024-01-27T01:06:28.665512+0000","last_fullsized":"2024-01-27T01:06:28.665512+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:38:36.236958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665852+0000","last_change":"2024-01-27T01:06:26.879441+0000","last_active":"2024-01-27T01:06:28.665852+0000","last_peered":"2024-01-27T01:06:28.665852+0000","last_clean":"2024-01-27T01:06:28.665852+0000","last_became_active":"2024-01-27T01:06:26.879148+0000","last_became_peered":"2024-01-27T01:06:26.879148+0000","last_unstale":"2024-01-27T01:06:28.665852+0000","last_undegraded":"2024-01-27T01:06:28.665852+0000","last_fullsized":"2024-01-27T01:06:28.665852+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:05:08.012027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666259+0000","last_change":"2024-01-27T01:06:26.876823+0000","last_active":"2024-01-27T01:06:28.666259+0000","last_peered":"2024-01-27T01:06:28.666259+0000","last_clean":"2024-01-27T01:06:28.666259+0000","last_became_active":"2024-01-27T01:06:26.876304+0000","last_became_peered":"2024-01-27T01:06:26.876304+0000","last_unstale":"2024-01-27T01:06:28.666259+0000","last_undegraded":"2024-01-27T01:06:28.666259+0000","last_fullsized":"2024-01-27T01:06:28.666259+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:28.378677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666099+0000","last_change":"2024-01-27T01:06:26.873401+0000","last_active":"2024-01-27T01:06:28.666099+0000","last_peered":"2024-01-27T01:06:28.666099+0000","last_clean":"2024-01-27T01:06:28.666099+0000","last_became_active":"2024-01-27T01:06:26.873094+0000","last_became_peered":"2024-01-27T01:06:26.873094+0000","last_unstale":"2024-01-27T01:06:28.666099+0000","last_undegraded":"2024-01-27T01:06:28.666099+0000","last_fullsized":"2024-01-27T01:06:28.666099+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:24.082749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865407+0000","last_change":"2024-01-27T01:06:26.864890+0000","last_active":"2024-01-27T01:06:27.865407+0000","last_peered":"2024-01-27T01:06:27.865407+0000","last_clean":"2024-01-27T01:06:27.865407+0000","last_became_active":"2024-01-27T01:06:26.864556+0000","last_became_peered":"2024-01-27T01:06:26.864556+0000","last_unstale":"2024-01-27T01:06:27.865407+0000","last_undegraded":"2024-01-27T01:06:27.865407+0000","last_fullsized":"2024-01-27T01:06:27.865407+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:41.033580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865791+0000","last_change":"2024-01-27T01:06:26.874473+0000","last_active":"2024-01-27T01:06:27.865791+0000","last_peered":"2024-01-27T01:06:27.865791+0000","last_clean":"2024-01-27T01:06:27.865791+0000","last_became_active":"2024-01-27T01:06:26.874203+0000","last_became_peered":"2024-01-27T01:06:26.874203+0000","last_unstale":"2024-01-27T01:06:27.865791+0000","last_undegraded":"2024-01-27T01:06:27.865791+0000","last_fullsized":"2024-01-27T01:06:27.865791+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:33:37.975146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240809+0000","last_change":"2024-01-27T01:06:26.863380+0000","last_active":"2024-01-27T01:06:28.240809+0000","last_peered":"2024-01-27T01:06:28.240809+0000","last_clean":"2024-01-27T01:06:28.240809+0000","last_became_active":"2024-01-27T01:06:26.863034+0000","last_became_peered":"2024-01-27T01:06:26.863034+0000","last_unstale":"2024-01-27T01:06:28.240809+0000","last_undegraded":"2024-01-27T01:06:28.240809+0000","last_fullsized":"2024-01-27T01:06:28.240809+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:32.271904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355189+0000","last_change":"2024-01-27T01:06:26.877929+0000","last_active":"2024-01-27T01:06:28.355189+0000","last_peered":"2024-01-27T01:06:28.355189+0000","last_clean":"2024-01-27T01:06:28.355189+0000","last_became_active":"2024-01-27T01:06:26.877679+0000","last_became_peered":"2024-01-27T01:06:26.877679+0000","last_unstale":"2024-01-27T01:06:28.355189+0000","last_undegraded":"2024-01-27T01:06:28.355189+0000","last_fullsized":"2024-01-27T01:06:28.355189+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:53:15.482586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665563+0000","last_change":"2024-01-27T01:06:26.876674+0000","last_active":"2024-01-27T01:06:28.665563+0000","last_peered":"2024-01-27T01:06:28.665563+0000","last_clean":"2024-01-27T01:06:28.665563+0000","last_became_active":"2024-01-27T01:06:26.876413+0000","last_became_peered":"2024-01-27T01:06:26.876413+0000","last_unstale":"2024-01-27T01:06:28.665563+0000","last_undegraded":"2024-01-27T01:06:28.665563+0000","last_fullsized":"2024-01-27T01:06:28.665563+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:08.822886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038257+0000","last_change":"2024-01-27T01:06:26.877777+0000","last_active":"2024-01-27T01:06:28.038257+0000","last_peered":"2024-01-27T01:06:28.038257+0000","last_clean":"2024-01-27T01:06:28.038257+0000","last_became_active":"2024-01-27T01:06:26.877515+0000","last_became_peered":"2024-01-27T01:06:26.877515+0000","last_unstale":"2024-01-27T01:06:28.038257+0000","last_undegraded":"2024-01-27T01:06:28.038257+0000","last_fullsized":"2024-01-27T01:06:28.038257+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:27:14.653781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866589+0000","last_change":"2024-01-27T01:06:26.865833+0000","last_active":"2024-01-27T01:06:27.866589+0000","last_peered":"2024-01-27T01:06:27.866589+0000","last_clean":"2024-01-27T01:06:27.866589+0000","last_became_active":"2024-01-27T01:06:26.865525+0000","last_became_peered":"2024-01-27T01:06:26.865525+0000","last_unstale":"2024-01-27T01:06:27.866589+0000","last_undegraded":"2024-01-27T01:06:27.866589+0000","last_fullsized":"2024-01-27T01:06:27.866589+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:42:10.990703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240896+0000","last_change":"2024-01-27T01:06:26.879535+0000","last_active":"2024-01-27T01:06:28.240896+0000","last_peered":"2024-01-27T01:06:28.240896+0000","last_clean":"2024-01-27T01:06:28.240896+0000","last_became_active":"2024-01-27T01:06:26.879219+0000","last_became_peered":"2024-01-27T01:06:26.879219+0000","last_unstale":"2024-01-27T01:06:28.240896+0000","last_undegraded":"2024-01-27T01:06:28.240896+0000","last_fullsized":"2024-01-27T01:06:28.240896+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:58.495062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666051+0000","last_change":"2024-01-27T01:06:26.876736+0000","last_active":"2024-01-27T01:06:28.666051+0000","last_peered":"2024-01-27T01:06:28.666051+0000","last_clean":"2024-01-27T01:06:28.666051+0000","last_became_active":"2024-01-27T01:06:26.875870+0000","last_became_peered":"2024-01-27T01:06:26.875870+0000","last_unstale":"2024-01-27T01:06:28.666051+0000","last_undegraded":"2024-01-27T01:06:28.666051+0000","last_fullsized":"2024-01-27T01:06:28.666051+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:43.223271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865691+0000","last_change":"2024-01-27T01:06:26.873409+0000","last_active":"2024-01-27T01:06:27.865691+0000","last_peered":"2024-01-27T01:06:27.865691+0000","last_clean":"2024-01-27T01:06:27.865691+0000","last_became_active":"2024-01-27T01:06:26.868026+0000","last_became_peered":"2024-01-27T01:06:26.868026+0000","last_unstale":"2024-01-27T01:06:27.865691+0000","last_undegraded":"2024-01-27T01:06:27.865691+0000","last_fullsized":"2024-01-27T01:06:27.865691+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:15.664932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665379+0000","last_change":"2024-01-27T01:06:26.881786+0000","last_active":"2024-01-27T01:06:28.665379+0000","last_peered":"2024-01-27T01:06:28.665379+0000","last_clean":"2024-01-27T01:06:28.665379+0000","last_became_active":"2024-01-27T01:06:26.881527+0000","last_became_peered":"2024-01-27T01:06:26.881527+0000","last_unstale":"2024-01-27T01:06:28.665379+0000","last_undegraded":"2024-01-27T01:06:28.665379+0000","last_fullsized":"2024-01-27T01:06:28.665379+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:43.630831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865757+0000","last_change":"2024-01-27T01:06:26.872094+0000","last_active":"2024-01-27T01:06:27.865757+0000","last_peered":"2024-01-27T01:06:27.865757+0000","last_clean":"2024-01-27T01:06:27.865757+0000","last_became_active":"2024-01-27T01:06:26.871711+0000","last_became_peered":"2024-01-27T01:06:26.871711+0000","last_unstale":"2024-01-27T01:06:27.865757+0000","last_undegraded":"2024-01-27T01:06:27.865757+0000","last_fullsized":"2024-01-27T01:06:27.865757+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:24:32.821570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.037950+0000","last_change":"2024-01-27T01:06:26.879291+0000","last_active":"2024-01-27T01:06:28.037950+0000","last_peered":"2024-01-27T01:06:28.037950+0000","last_clean":"2024-01-27T01:06:28.037950+0000","last_became_active":"2024-01-27T01:06:26.879063+0000","last_became_peered":"2024-01-27T01:06:26.879063+0000","last_unstale":"2024-01-27T01:06:28.037950+0000","last_undegraded":"2024-01-27T01:06:28.037950+0000","last_fullsized":"2024-01-27T01:06:28.037950+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:22:23.545688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866284+0000","last_change":"2024-01-27T01:06:26.870422+0000","last_active":"2024-01-27T01:06:27.866284+0000","last_peered":"2024-01-27T01:06:27.866284+0000","last_clean":"2024-01-27T01:06:27.866284+0000","last_became_active":"2024-01-27T01:06:26.870117+0000","last_became_peered":"2024-01-27T01:06:26.870117+0000","last_unstale":"2024-01-27T01:06:27.866284+0000","last_undegraded":"2024-01-27T01:06:27.866284+0000","last_fullsized":"2024-01-27T01:06:27.866284+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:42:55.987157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038255+0000","last_change":"2024-01-27T01:06:26.868177+0000","last_active":"2024-01-27T01:06:28.038255+0000","last_peered":"2024-01-27T01:06:28.038255+0000","last_clean":"2024-01-27T01:06:28.038255+0000","last_became_active":"2024-01-27T01:06:26.867906+0000","last_became_peered":"2024-01-27T01:06:26.867906+0000","last_unstale":"2024-01-27T01:06:28.038255+0000","last_undegraded":"2024-01-27T01:06:28.038255+0000","last_fullsized":"2024-01-27T01:06:28.038255+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:32.271904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.241039+0000","last_change":"2024-01-27T01:06:26.868462+0000","last_active":"2024-01-27T01:06:28.241039+0000","last_peered":"2024-01-27T01:06:28.241039+0000","last_clean":"2024-01-27T01:06:28.241039+0000","last_became_active":"2024-01-27T01:06:26.868203+0000","last_became_peered":"2024-01-27T01:06:26.868203+0000","last_unstale":"2024-01-27T01:06:28.241039+0000","last_undegraded":"2024-01-27T01:06:28.241039+0000","last_fullsized":"2024-01-27T01:06:28.241039+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:10.965469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866164+0000","last_change":"2024-01-27T01:06:26.871135+0000","last_active":"2024-01-27T01:06:27.866164+0000","last_peered":"2024-01-27T01:06:27.866164+0000","last_clean":"2024-01-27T01:06:27.866164+0000","last_became_active":"2024-01-27T01:06:26.870854+0000","last_became_peered":"2024-01-27T01:06:26.870854+0000","last_unstale":"2024-01-27T01:06:27.866164+0000","last_undegraded":"2024-01-27T01:06:27.866164+0000","last_fullsized":"2024-01-27T01:06:27.866164+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:12:59.875769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666262+0000","last_change":"2024-01-27T01:06:26.873715+0000","last_active":"2024-01-27T01:06:28.666262+0000","last_peered":"2024-01-27T01:06:28.666262+0000","last_clean":"2024-01-27T01:06:28.666262+0000","last_became_active":"2024-01-27T01:06:26.873430+0000","last_became_peered":"2024-01-27T01:06:26.873430+0000","last_unstale":"2024-01-27T01:06:28.666262+0000","last_undegraded":"2024-01-27T01:06:28.666262+0000","last_fullsized":"2024-01-27T01:06:28.666262+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:44.342963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3538,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":630,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8750,"ondisk_log_size":8750,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576004,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300632,"kb_used_data":3344,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071208,"statfs":{"total":96636764160,"available":96328916992,"internally_reserved":0,"allocated":3424256,"data_stored":2074391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63200000000000001,"15min":0.61399999999999999},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.312},"max":{"1min":0.82099999999999995,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.63700000000000001,"15min":0.623},"min":{"1min":0.498,"5min":0.39900000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":0.879,"15min":0.879},"last":0.55200000000000005}]},{"osd":1,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.623,"15min":0.61599999999999999},"min":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65600000000000003,"15min":0.63700000000000001},"min":{"1min":0.505,"5min":0.42099999999999999,"15min":0.34300000000000003},"max":{"1min":0.76800000000000002,"5min":0.84099999999999997,"15min":0.84399999999999997},"last":0.60599999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63400000000000001,"15min":0.61399999999999999},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82099999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.628},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64500000000000002,"15min":0.63600000000000001},"min":{"1min":0.49399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79400000000000004,"5min":0.872,"15min":0.88900000000000001},"last":0.59499999999999997}]},{"osd":3,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.628,"15min":0.62},"min":{"1min":0.52100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79900000000000004,"5min":0.876,"15min":0.876},"last":0.504},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66800000000000004,"15min":0.64700000000000002},"min":{"1min":0.53200000000000003,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84599999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.69099999999999995}]},{"osd":4,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.50900000000000001,"15min":0.48999999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.29999999999999999},"max":{"1min":0.73399999999999999,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.51900000000000002},"min":{"1min":0.35699999999999998,"5min":0.34699999999999998,"15min":0.32200000000000001},"max":{"1min":0.77500000000000002,"5min":0.80300000000000005,"15min":0.86599999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55800000000000005,"15min":0.52600000000000002},"min":{"1min":0.40999999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.753,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57999999999999996,"15min":0.56499999999999995},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.33700000000000002},"max":{"1min":0.76700000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.66300000000000003}]},{"osd":6,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60999999999999999,"15min":0.58899999999999997},"min":{"1min":0.42499999999999999,"5min":0.42199999999999999,"15min":0.37},"max":{"1min":0.81100000000000005,"5min":0.83999999999999997,"15min":0.84799999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61299999999999999,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.39100000000000001,"15min":0.35699999999999998},"max":{"1min":0.76100000000000001,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.68000000000000005}]}]},{"osd":6,"up_from":13,"seq":55834576004,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308244,"kb_used_data":2764,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063596,"statfs":{"total":96636764160,"available":96321122304,"internally_reserved":0,"allocated":2830336,"data_stored":1484507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61099999999999999,"15min":0.60799999999999998},"min":{"1min":0.41999999999999998,"5min":0.40799999999999997,"15min":0.36399999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.496},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.441,"5min":0.432,"15min":0.376},"max":{"1min":0.82499999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.67600000000000005}]},{"osd":1,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79000000000000004,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.40799999999999997,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.628}]},{"osd":2,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.39700000000000002,"15min":0.377},"max":{"1min":0.80400000000000005,"5min":0.878,"15min":0.878},"last":0.65500000000000003}]},{"osd":3,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.622,"15min":0.627},"min":{"1min":0.5,"5min":0.378,"15min":0.378},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.90100000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66300000000000003,"15min":0.66000000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.59899999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.36299999999999999,"5min":0.33300000000000002,"15min":0.32500000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.86599999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33900000000000002,"5min":0.33800000000000002,"15min":0.29799999999999999},"max":{"1min":0.77300000000000002,"5min":0.78500000000000003,"15min":0.88900000000000001},"last":0.54700000000000004}]},{"osd":5,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.31900000000000001},"max":{"1min":0.68999999999999995,"5min":0.75900000000000001,"15min":0.80500000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.42199999999999999,"5min":0.37,"15min":0.28899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.874},"last":0.58499999999999996}]},{"osd":7,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45800000000000002,"5min":0.39000000000000001,"15min":0.35799999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.92900000000000005},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.44500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78900000000000003,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.61399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834576004,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300024,"kb_used_data":2736,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071816,"statfs":{"total":96636764160,"available":96329539584,"internally_reserved":0,"allocated":2801664,"data_stored":1473710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.41099999999999998,"15min":0.42099999999999999},"min":{"1min":0.26400000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.64400000000000002,"5min":0.66100000000000003,"15min":0.69599999999999995},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44500000000000001,"15min":0.45500000000000002},"min":{"1min":0.34000000000000002,"5min":0.28399999999999997,"15min":0.26800000000000002},"max":{"1min":0.57999999999999996,"5min":0.73999999999999999,"15min":0.76600000000000001},"last":0.35699999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.439,"15min":0.438},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.26200000000000001},"max":{"1min":0.748,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.47599999999999998,"15min":0.48999999999999999},"min":{"1min":0.379,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.67400000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.5}]},{"osd":3,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52300000000000002,"15min":0.53300000000000003},"min":{"1min":0.39300000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73699999999999999,"5min":0.84799999999999998,"15min":1.2130000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.51400000000000001},"min":{"1min":0.41699999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68200000000000005,"5min":1.125,"15min":1.22},"last":0.42099999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.67200000000000004,"15min":0.67400000000000004},"min":{"1min":0.42099999999999999,"5min":0.41099999999999998,"15min":0.39800000000000002},"max":{"1min":0.82699999999999996,"5min":1.242,"15min":1.242},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69099999999999995,"15min":0.69599999999999995},"min":{"1min":0.53700000000000003,"5min":0.496,"15min":0.35699999999999998},"max":{"1min":0.90600000000000003,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.45900000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73499999999999999,"15min":0.70899999999999996},"min":{"1min":0.53300000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.876,"5min":5.444,"15min":5.444},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.71699999999999997,"15min":0.70799999999999996},"min":{"1min":0.502,"5min":0.47599999999999998,"15min":0.435},"max":{"1min":0.97799999999999998,"5min":2.6669999999999998,"15min":2.6669999999999998},"last":0.47599999999999998}]},{"osd":6,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.74199999999999999,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":0.93899999999999995,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.81299999999999994,"15min":0.745},"min":{"1min":0.48799999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.02,"5min":11.223000000000001,"15min":11.223000000000001},"last":0.52400000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.752,"15min":0.75600000000000001},"min":{"1min":0.58999999999999997,"5min":0.52800000000000002,"15min":0.47599999999999998},"max":{"1min":0.96299999999999997,"5min":2.9660000000000002,"15min":2.9660000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.77700000000000002,"15min":0.76800000000000002},"min":{"1min":0.58799999999999997,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.94499999999999995,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.53400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834576003,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317004,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94054836,"statfs":{"total":96636764160,"available":96312152064,"internally_reserved":0,"allocated":3411968,"data_stored":2071643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42899999999999999,"15min":0.42199999999999999},"min":{"1min":0.30299999999999999,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":0.55600000000000005,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.436,"5min":0.45600000000000002,"15min":0.46100000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":0.56499999999999995,"5min":0.72699999999999998,"15min":0.73399999999999999},"last":0.26700000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.29699999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.58299999999999996,"5min":0.66900000000000004,"15min":0.76800000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48399999999999999,"15min":0.48899999999999999},"min":{"1min":0.38600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64600000000000002,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.45400000000000001}]},{"osd":3,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.66100000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.504},"min":{"1min":0.36699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64100000000000001,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.36699999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.68300000000000005,"15min":0.67700000000000005},"min":{"1min":0.57799999999999996,"5min":0.47999999999999998,"15min":0.42199999999999999},"max":{"1min":0.86399999999999999,"5min":0.91500000000000004,"15min":1.0720000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.56999999999999995,"5min":0.47099999999999997,"15min":0.437},"max":{"1min":0.82699999999999996,"5min":1.002,"15min":1.002},"last":0.66200000000000003}]},{"osd":5,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.68999999999999995,"15min":0.70099999999999996},"min":{"1min":0.58199999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.876,"5min":1.0209999999999999,"15min":2.165},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70599999999999996,"15min":0.71199999999999997},"min":{"1min":0.498,"5min":0.48799999999999999,"15min":0.45600000000000002},"max":{"1min":0.89000000000000001,"5min":0.97399999999999998,"15min":1.1399999999999999},"last":0.64700000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69299999999999995,"15min":0.71099999999999997},"min":{"1min":0.56499999999999995,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.84599999999999997,"5min":0.98899999999999999,"15min":1.0860000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72599999999999998,"15min":0.74099999999999999},"min":{"1min":0.54300000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.93400000000000005,"5min":1.0109999999999999,"15min":1.425},"last":0.73299999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.74399999999999999,"15min":0.75700000000000001},"min":{"1min":0.61299999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.98899999999999999,"5min":1.044,"15min":1.1259999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.76300000000000001,"15min":0.76900000000000002},"min":{"1min":0.58299999999999996,"5min":0.51100000000000001,"15min":0.48499999999999999},"max":{"1min":0.91700000000000004,"5min":1.0289999999999999,"15min":1.1910000000000001},"last":0.76600000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576003,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312328,"kb_used_data":2752,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059512,"statfs":{"total":96636764160,"available":96316940288,"internally_reserved":0,"allocated":2818048,"data_stored":1481275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40500000000000003,"15min":0.40799999999999997},"min":{"1min":0.26600000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.46000000000000002,"5min":0.746,"15min":0.746},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.44500000000000001,"15min":0.44400000000000001},"min":{"1min":0.32300000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.59099999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.436}]},{"osd":1,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.435,"15min":0.44500000000000001},"min":{"1min":0.32000000000000001,"5min":0.29099999999999998,"15min":0.27000000000000002},"max":{"1min":0.61299999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.439,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.55600000000000005,"5min":0.73599999999999999,"15min":0.78100000000000003},"last":0.54800000000000004}]},{"osd":3,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.498,"15min":0.497},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.33200000000000002},"max":{"1min":0.627,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.497},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48699999999999999,"15min":0.49199999999999999},"min":{"1min":0.373,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":0.59099999999999997,"5min":0.66600000000000004,"15min":0.75700000000000001},"last":0.52000000000000002}]},{"osd":4,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68200000000000005,"15min":0.68000000000000005},"min":{"1min":0.52800000000000002,"5min":0.495,"15min":0.442},"max":{"1min":0.92700000000000005,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68999999999999995,"15min":0.69399999999999995},"min":{"1min":0.50900000000000001,"5min":0.47099999999999997,"15min":0.378},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":1.107},"last":0.77200000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.69099999999999995,"15min":0.70299999999999996},"min":{"1min":0.52400000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.89500000000000002,"15min":1.167},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70899999999999996,"15min":0.70999999999999996},"min":{"1min":0.47499999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89600000000000002,"5min":0.97499999999999998,"15min":1.05},"last":0.78600000000000003}]},{"osd":6,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71499999999999997,"15min":0.71199999999999997},"min":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.45500000000000002},"max":{"1min":0.81399999999999995,"5min":0.996,"15min":1.077},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70499999999999996,"15min":0.71999999999999997},"min":{"1min":0.497,"5min":0.497,"15min":0.39200000000000002},"max":{"1min":0.876,"5min":0.96699999999999997,"15min":1.135},"last":0.73499999999999999}]},{"osd":7,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.749,"15min":0.75},"min":{"1min":0.56000000000000005,"5min":0.50800000000000001,"15min":0.47099999999999997},"max":{"1min":0.94099999999999995,"5min":0.97399999999999998,"15min":1.198},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.76500000000000001,"15min":0.76900000000000002},"min":{"1min":0.55000000000000004,"5min":0.53100000000000003,"15min":0.48199999999999998},"max":{"1min":0.94599999999999995,"5min":1.028,"15min":1.0880000000000001},"last":0.752}]}]},{"osd":3,"up_from":13,"seq":55834576002,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304144,"kb_used_data":2760,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067696,"statfs":{"total":96636764160,"available":96325320704,"internally_reserved":0,"allocated":2826240,"data_stored":1481311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.44700000000000001,"15min":0.42299999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.54800000000000004,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.309},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.372,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.63500000000000001,"5min":0.66700000000000004,"15min":0.84699999999999998},"last":0.34300000000000003}]},{"osd":1,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.63100000000000001,"15min":0.53500000000000003},"min":{"1min":0.34499999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.629,"5min":15.391,"15min":15.391},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.54600000000000004,"15min":0.52400000000000002},"min":{"1min":0.36699999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65100000000000002,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.42999999999999999}]},{"osd":2,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54600000000000004,"15min":0.51700000000000002},"min":{"1min":0.36499999999999999,"5min":0.36399999999999999,"15min":0.34599999999999997},"max":{"1min":0.62,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.53100000000000003,"15min":0.50700000000000001},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.60699999999999998,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":0.39600000000000002}]},{"osd":4,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.74199999999999999,"15min":0.72099999999999997},"min":{"1min":0.497,"5min":0.47699999999999998,"15min":0.439},"max":{"1min":0.88600000000000001,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.71999999999999997,"15min":0.70799999999999996},"min":{"1min":0.46300000000000002,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":0.94399999999999995,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.52000000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.74299999999999999,"15min":0.71699999999999997},"min":{"1min":0.51100000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.83999999999999997,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72799999999999998,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.97699999999999998,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.57399999999999995}]},{"osd":6,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.754,"15min":0.72599999999999998},"min":{"1min":0.53500000000000003,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.81200000000000006,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.748,"15min":0.74299999999999999},"min":{"1min":0.56100000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.92700000000000005,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.60799999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.75600000000000001,"15min":0.73799999999999999},"min":{"1min":0.58599999999999997,"5min":0.52500000000000002,"15min":0.46700000000000003},"max":{"1min":1.0029999999999999,"5min":4.569,"15min":4.569},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.80600000000000005,"15min":0.78500000000000003},"min":{"1min":0.55700000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0169999999999999,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.59199999999999997}]}]},{"osd":4,"up_from":13,"seq":55834576004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312328,"kb_used_data":2752,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059512,"statfs":{"total":96636764160,"available":96316940288,"internally_reserved":0,"allocated":2818048,"data_stored":1481739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.46899999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.84199999999999997},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.61499999999999999,"15min":0.61399999999999999},"min":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78700000000000003,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.753}]},{"osd":1,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61899999999999999,"15min":0.627},"min":{"1min":0.46100000000000002,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.76400000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61499999999999999,"15min":0.62},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.79800000000000004,"5min":0.874,"15min":0.874},"last":0.69699999999999995}]},{"osd":2,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.59999999999999998},"min":{"1min":0.42399999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.88700000000000001},"last":0.628},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.51100000000000001,"5min":0.42899999999999999,"15min":0.42799999999999999},"max":{"1min":0.81100000000000005,"5min":0.81499999999999995,"15min":0.89200000000000002},"last":0.79200000000000004}]},{"osd":3,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.61599999999999999,"15min":0.623},"min":{"1min":0.38800000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.83299999999999996,"5min":0.83599999999999997,"15min":0.84199999999999997},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.627},"min":{"1min":0.41299999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.88300000000000001},"last":0.72999999999999998}]},{"osd":5,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.75800000000000001,"15min":0.85899999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.502},"min":{"1min":0.376,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.878},"last":0.74099999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34999999999999998},"max":{"1min":0.63200000000000001,"5min":0.70099999999999996,"15min":0.88700000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.70699999999999996,"5min":0.79200000000000004,"15min":0.81499999999999995},"last":0.64800000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.42399999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72599999999999998,"5min":0.80400000000000005,"15min":0.90200000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59499999999999997,"15min":0.59399999999999997},"min":{"1min":0.41599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.73799999999999999,"5min":0.78700000000000003,"15min":0.92600000000000005},"last":0.76700000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576004,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312332,"kb_used_data":2756,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059508,"statfs":{"total":96636764160,"available":96316936192,"internally_reserved":0,"allocated":2822144,"data_stored":1481241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59399999999999997,"15min":0.60299999999999998},"min":{"1min":0.42999999999999999,"5min":0.40500000000000003,"15min":0.27300000000000002},"max":{"1min":0.80700000000000005,"5min":0.82799999999999996,"15min":1.006},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59399999999999997,"15min":0.60699999999999998},"min":{"1min":0.36599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":1.1719999999999999},"last":0.73499999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63700000000000001,"15min":0.65200000000000002},"min":{"1min":0.39100000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.95899999999999996},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.52100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":1.212},"last":0.77900000000000003}]},{"osd":2,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57499999999999996,"15min":0.59399999999999997},"min":{"1min":0.42299999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":1.631},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.623,"5min":0.625,"15min":0.621},"min":{"1min":0.497,"5min":0.38800000000000001,"15min":0.34000000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.60499999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.59999999999999998},"min":{"1min":0.39600000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83299999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.625,"15min":0.63},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.879,"5min":0.879,"15min":1.4690000000000001},"last":0.754}]},{"osd":4,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.505,"15min":0.48799999999999999},"min":{"1min":0.34899999999999998,"5min":0.32200000000000001,"15min":0.317},"max":{"1min":0.76700000000000002,"5min":0.90200000000000002,"15min":0.94399999999999995},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53500000000000003,"15min":0.51500000000000001},"min":{"1min":0.439,"5min":0.35199999999999998,"15min":0.33900000000000002},"max":{"1min":0.80800000000000005,"5min":0.92500000000000004,"15min":0.93300000000000005},"last":0.58299999999999996}]},{"osd":6,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51800000000000002,"15min":0.504},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79400000000000004,"5min":0.96799999999999997,"15min":1.732},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56299999999999994,"15min":0.54500000000000004},"min":{"1min":0.42899999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.753,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.53800000000000003}]},{"osd":7,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.46200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80400000000000005,"5min":0.95399999999999996,"15min":1.0740000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.626,"15min":0.61299999999999999},"min":{"1min":0.504,"5min":0.40799999999999997,"15min":0.39100000000000001},"max":{"1min":0.85199999999999998,"5min":0.996,"15min":1.6919999999999999},"last":0.61599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:06:45.599 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:06:45.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.768+0000 7f105e45d700 1 Processor -- start 2024-01-27T01:06:45.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.768+0000 7f105e45d700 1 -- start start 2024-01-27T01:06:45.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.769+0000 7f105e45d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f105815d210 0x7f10580cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:45.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.769+0000 7f105e45d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10580cc450 con 0x7f10580cbff0 2024-01-27T01:06:45.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.769+0000 7f105e45d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10580cff70 con 0x7f105815ce40 2024-01-27T01:06:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.769+0000 7f105e45d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10580cf400 con 0x7f105815d210 2024-01-27T01:06:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.769+0000 7f105c9fa700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f10580cbff0 0x7f10580ade30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45452/0 (socket says 172.21.15.46:45452) 2024-01-27T01:06:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.769+0000 7f105c9fa700 1 -- 172.21.15.46:0/3422345733 learned_addr learned my addr 172.21.15.46:0/3422345733 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.769+0000 7f10577fe700 1 --2- 172.21.15.46:0/3422345733 >> v2:172.21.15.106:3300/0 conn(0x7f105815d210 0x7f10580cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.769+0000 7f1056ffd700 1 -- 172.21.15.46:0/3422345733 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690277206 0 0) 0x7f1050001490 con 0x7f10580cbff0 2024-01-27T01:06:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.769+0000 7f1056ffd700 1 -- 172.21.15.46:0/3422345733 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1040003570 con 0x7f10580cbff0 2024-01-27T01:06:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.769+0000 7f1056ffd700 1 -- 172.21.15.46:0/3422345733 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201872006 0 0) 0x7f10580cff70 con 0x7f105815ce40 2024-01-27T01:06:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.769+0000 7f1056ffd700 1 -- 172.21.15.46:0/3422345733 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1050001490 con 0x7f105815ce40 2024-01-27T01:06:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.769+0000 7f10577fe700 1 -- 172.21.15.46:0/3422345733 >> v1:172.21.15.106:6789/0 conn(0x7f105815ce40 legacy=0x7f10580d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.769+0000 7f10577fe700 1 -- 172.21.15.46:0/3422345733 >> v1:172.21.15.46:6789/0 conn(0x7f10580cbff0 legacy=0x7f10580ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.769+0000 7f1056ffd700 1 -- 172.21.15.46:0/3422345733 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 920860152 0 0) 0x7f1040003570 con 0x7f10580cbff0 2024-01-27T01:06:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.769+0000 7f10577fe700 1 -- 172.21.15.46:0/3422345733 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10580cf660 con 0x7f105815d210 2024-01-27T01:06:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.770+0000 7f10577fe700 1 --2- 172.21.15.46:0/3422345733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f105815d210 0x7f10580cbab0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f104c0098e0 tx=0x7f104c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25d8d810e742e5e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.770+0000 7f1056ffd700 1 -- 172.21.15.46:0/3422345733 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1050001980 con 0x7f105815d210 2024-01-27T01:06:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.770+0000 7f1056ffd700 1 -- 172.21.15.46:0/3422345733 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104c011420 con 0x7f105815d210 2024-01-27T01:06:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.770+0000 7f1056ffd700 1 -- 172.21.15.46:0/3422345733 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c0117c0 con 0x7f105815d210 2024-01-27T01:06:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.770+0000 7f105e45d700 1 -- 172.21.15.46:0/3422345733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f105815d210 msgr2=0x7f10580cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.770+0000 7f105e45d700 1 --2- 172.21.15.46:0/3422345733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f105815d210 0x7f10580cbab0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f104c0098e0 tx=0x7f104c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.771+0000 7f105e45d700 1 -- 172.21.15.46:0/3422345733 shutdown_connections 2024-01-27T01:06:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.771+0000 7f105e45d700 1 --2- 172.21.15.46:0/3422345733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f105815d210 0x7f10580cbab0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.771+0000 7f105e45d700 1 -- 172.21.15.46:0/3422345733 >> 172.21.15.46:0/3422345733 conn(0x7f105806f840 msgr2=0x7f10580d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.771+0000 7f105e45d700 1 -- 172.21.15.46:0/3422345733 shutdown_connections 2024-01-27T01:06:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.771+0000 7f105e45d700 1 -- 172.21.15.46:0/3422345733 wait complete. 2024-01-27T01:06:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.772+0000 7f105e45d700 1 Processor -- start 2024-01-27T01:06:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.772+0000 7f105e45d700 1 -- start start 2024-01-27T01:06:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.773+0000 7f105e45d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10580cbff0 0x7f10581474c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.773+0000 7f105e45d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f105815ce40 0x7f105809a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.773+0000 7f105e45d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f105815d210 0x7f10580c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.773+0000 7f105e45d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10580cb8c0 con 0x7f105815ce40 2024-01-27T01:06:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.773+0000 7f105e45d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105809abc0 con 0x7f105815d210 2024-01-27T01:06:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.773+0000 7f105e45d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10580c6150 con 0x7f10580cbff0 2024-01-27T01:06:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.773+0000 7f10577fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f105815ce40 0x7f105809a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.773+0000 7f10577fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f105815ce40 0x7f105809a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41852/0 (socket says 172.21.15.46:41852) 2024-01-27T01:06:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.773+0000 7f10577fe700 1 -- 172.21.15.46:0/1482212408 learned_addr learned my addr 172.21.15.46:0/1482212408 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.773+0000 7f105c9fa700 1 --2- 172.21.15.46:0/1482212408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f105815d210 0x7f10580c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.773+0000 7f1057fff700 1 --2- 172.21.15.46:0/1482212408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10580cbff0 0x7f10581474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.773+0000 7f10577fe700 1 -- 172.21.15.46:0/1482212408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10580cbff0 msgr2=0x7f10581474c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.773+0000 7f10577fe700 1 --2- 172.21.15.46:0/1482212408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10580cbff0 0x7f10581474c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.773+0000 7f10577fe700 1 -- 172.21.15.46:0/1482212408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f105815d210 msgr2=0x7f10580c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.773+0000 7f10577fe700 1 --2- 172.21.15.46:0/1482212408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f105815d210 0x7f10580c58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.773+0000 7f10577fe700 1 -- 172.21.15.46:0/1482212408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104c009420 con 0x7f105815ce40 2024-01-27T01:06:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.773+0000 7f1057fff700 1 --2- 172.21.15.46:0/1482212408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10580cbff0 0x7f10581474c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.774+0000 7f10577fe700 1 --2- 172.21.15.46:0/1482212408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f105815ce40 0x7f105809a330 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f104c002000 tx=0x7f104c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.774+0000 7f103ffff700 1 -- 172.21.15.46:0/1482212408 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c01cbe0 con 0x7f105815ce40 2024-01-27T01:06:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.774+0000 7f103ffff700 1 -- 172.21.15.46:0/1482212408 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104c01cd40 con 0x7f105815ce40 2024-01-27T01:06:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.774+0000 7f105e45d700 1 -- 172.21.15.46:0/1482212408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10580c6430 con 0x7f105815ce40 2024-01-27T01:06:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.774+0000 7f103ffff700 1 -- 172.21.15.46:0/1482212408 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c01a6a0 con 0x7f105815ce40 2024-01-27T01:06:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.774+0000 7f105e45d700 1 -- 172.21.15.46:0/1482212408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10580c5df0 con 0x7f105815ce40 2024-01-27T01:06:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.775+0000 7f105e45d700 1 -- 172.21.15.46:0/1482212408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10580c6950 con 0x7f105815ce40 2024-01-27T01:06:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.775+0000 7f103ffff700 1 -- 172.21.15.46:0/1482212408 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f104c01aa90 con 0x7f105815ce40 2024-01-27T01:06:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.776+0000 7f103ffff700 1 --2- 172.21.15.46:0/1482212408 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1044066f30 0x7f10440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.776+0000 7f103ffff700 1 -- 172.21.15.46:0/1482212408 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f104c086360 con 0x7f105815ce40 2024-01-27T01:06:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.776+0000 7f1057fff700 1 --2- 172.21.15.46:0/1482212408 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1044066f30 0x7f10440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.776+0000 7f1057fff700 1 --2- 172.21.15.46:0/1482212408 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1044066f30 0x7f10440693e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1048001170 tx=0x7f1048009040 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:45.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.780+0000 7f103ffff700 1 -- 172.21.15.46:0/1482212408 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f10580c6950 con 0x7f105815ce40 2024-01-27T01:06:45.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.981+0000 7f105e45d700 1 -- 172.21.15.46:0/1482212408 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f10580c6950 con 0x7f1044066f30 2024-01-27T01:06:45.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:45.995+0000 7f103ffff700 1 -- 172.21.15.46:0/1482212408 <== mgr.9101 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+517877 (secure 0 0 0) 0x7f10580c6950 con 0x7f1044066f30 2024-01-27T01:06:46.006 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:46.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.000+0000 7f105e45d700 1 -- 172.21.15.46:0/1482212408 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1044066f30 msgr2=0x7f10440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:46.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.000+0000 7f105e45d700 1 --2- 172.21.15.46:0/1482212408 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1044066f30 0x7f10440693e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1048001170 tx=0x7f1048009040 comp rx=0 tx=0).stop 2024-01-27T01:06:46.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.000+0000 7f105e45d700 1 -- 172.21.15.46:0/1482212408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f105815ce40 msgr2=0x7f105809a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:46.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.000+0000 7f105e45d700 1 --2- 172.21.15.46:0/1482212408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f105815ce40 0x7f105809a330 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f104c002000 tx=0x7f104c016000 comp rx=0 tx=0).stop 2024-01-27T01:06:46.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.000+0000 7f105e45d700 1 -- 172.21.15.46:0/1482212408 shutdown_connections 2024-01-27T01:06:46.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.000+0000 7f105e45d700 1 --2- 172.21.15.46:0/1482212408 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1044066f30 0x7f10440693e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.000+0000 7f105e45d700 1 --2- 172.21.15.46:0/1482212408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f105815d210 0x7f10580c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.000+0000 7f105e45d700 1 --2- 172.21.15.46:0/1482212408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f10580cbff0 0x7f10581474c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.000+0000 7f105e45d700 1 --2- 172.21.15.46:0/1482212408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f105815ce40 0x7f105809a330 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.001+0000 7f105e45d700 1 -- 172.21.15.46:0/1482212408 >> 172.21.15.46:0/1482212408 conn(0x7f105806f840 msgr2=0x7f10580cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:46.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.001+0000 7f105e45d700 1 -- 172.21.15.46:0/1482212408 shutdown_connections 2024-01-27T01:06:46.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.001+0000 7f105e45d700 1 -- 172.21.15.46:0/1482212408 wait complete. 2024-01-27T01:06:46.016 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:06:46.017 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-01-27T01:06:44.734501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593906,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":677,"num_write_kb":3477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8791,"ondisk_log_size":8791,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":202,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2467036,"kb_used_data":23196,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752507684,"statfs":{"total":773094113280,"available":770567868416,"internally_reserved":0,"allocated":23752704,"data_stored":13029817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":787,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.826236"},"pg_stats":[{"pgid":"1.0","version":"213'8750","reported_seq":9665,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.037979+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:06:28.037979+0000","last_peered":"2024-01-27T01:06:28.037979+0000","last_clean":"2024-01-27T01:06:28.037979+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:06:28.037979+0000","last_undegraded":"2024-01-27T01:06:28.037979+0000","last_fullsized":"2024-01-27T01:06:28.037979+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8750,"log_dups_size":0,"ondisk_log_size":8750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":630,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865902+0000","last_change":"2024-01-27T01:06:25.858609+0000","last_active":"2024-01-27T01:06:27.865902+0000","last_peered":"2024-01-27T01:06:27.865902+0000","last_clean":"2024-01-27T01:06:27.865902+0000","last_became_active":"2024-01-27T01:06:25.857638+0000","last_became_peered":"2024-01-27T01:06:25.857638+0000","last_unstale":"2024-01-27T01:06:27.865902+0000","last_undegraded":"2024-01-27T01:06:27.865902+0000","last_fullsized":"2024-01-27T01:06:27.865902+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:35:59.109571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665626+0000","last_change":"2024-01-27T01:06:26.873783+0000","last_active":"2024-01-27T01:06:28.665626+0000","last_peered":"2024-01-27T01:06:28.665626+0000","last_clean":"2024-01-27T01:06:28.665626+0000","last_became_active":"2024-01-27T01:06:26.873490+0000","last_became_peered":"2024-01-27T01:06:26.873490+0000","last_unstale":"2024-01-27T01:06:28.665626+0000","last_undegraded":"2024-01-27T01:06:28.665626+0000","last_fullsized":"2024-01-27T01:06:28.665626+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:32.691743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916373+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:55:49.427110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865572+0000","last_change":"2024-01-27T01:06:25.865235+0000","last_active":"2024-01-27T01:06:27.865572+0000","last_peered":"2024-01-27T01:06:27.865572+0000","last_clean":"2024-01-27T01:06:27.865572+0000","last_became_active":"2024-01-27T01:06:25.864872+0000","last_became_peered":"2024-01-27T01:06:25.864872+0000","last_unstale":"2024-01-27T01:06:27.865572+0000","last_undegraded":"2024-01-27T01:06:27.865572+0000","last_fullsized":"2024-01-27T01:06:27.865572+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:53:38.618410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.241058+0000","last_change":"2024-01-27T01:06:26.879306+0000","last_active":"2024-01-27T01:06:28.241058+0000","last_peered":"2024-01-27T01:06:28.241058+0000","last_clean":"2024-01-27T01:06:28.241058+0000","last_became_active":"2024-01-27T01:06:26.879071+0000","last_became_peered":"2024-01-27T01:06:26.879071+0000","last_unstale":"2024-01-27T01:06:28.241058+0000","last_undegraded":"2024-01-27T01:06:28.241058+0000","last_fullsized":"2024-01-27T01:06:28.241058+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:50:46.965359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910860+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:36:16.613510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666145+0000","last_change":"2024-01-27T01:06:25.856035+0000","last_active":"2024-01-27T01:06:28.666145+0000","last_peered":"2024-01-27T01:06:28.666145+0000","last_clean":"2024-01-27T01:06:28.666145+0000","last_became_active":"2024-01-27T01:06:25.855762+0000","last_became_peered":"2024-01-27T01:06:25.855762+0000","last_unstale":"2024-01-27T01:06:28.666145+0000","last_undegraded":"2024-01-27T01:06:28.666145+0000","last_fullsized":"2024-01-27T01:06:28.666145+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:51:44.378880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038160+0000","last_change":"2024-01-27T01:06:26.870626+0000","last_active":"2024-01-27T01:06:28.038160+0000","last_peered":"2024-01-27T01:06:28.038160+0000","last_clean":"2024-01-27T01:06:28.038160+0000","last_became_active":"2024-01-27T01:06:26.870457+0000","last_became_peered":"2024-01-27T01:06:26.870457+0000","last_unstale":"2024-01-27T01:06:28.038160+0000","last_undegraded":"2024-01-27T01:06:28.038160+0000","last_fullsized":"2024-01-27T01:06:28.038160+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:42.766155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913817+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:22.022827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865995+0000","last_change":"2024-01-27T01:06:25.855015+0000","last_active":"2024-01-27T01:06:27.865995+0000","last_peered":"2024-01-27T01:06:27.865995+0000","last_clean":"2024-01-27T01:06:27.865995+0000","last_became_active":"2024-01-27T01:06:25.854708+0000","last_became_peered":"2024-01-27T01:06:25.854708+0000","last_unstale":"2024-01-27T01:06:27.865995+0000","last_undegraded":"2024-01-27T01:06:27.865995+0000","last_fullsized":"2024-01-27T01:06:27.865995+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:03.971945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038100+0000","last_change":"2024-01-27T01:06:26.876182+0000","last_active":"2024-01-27T01:06:28.038100+0000","last_peered":"2024-01-27T01:06:28.038100+0000","last_clean":"2024-01-27T01:06:28.038100+0000","last_became_active":"2024-01-27T01:06:26.875894+0000","last_became_peered":"2024-01-27T01:06:26.875894+0000","last_unstale":"2024-01-27T01:06:28.038100+0000","last_undegraded":"2024-01-27T01:06:28.038100+0000","last_fullsized":"2024-01-27T01:06:28.038100+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:58.495062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.914162+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:57:07.250360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666039+0000","last_change":"2024-01-27T01:06:25.858142+0000","last_active":"2024-01-27T01:06:28.666039+0000","last_peered":"2024-01-27T01:06:28.666039+0000","last_clean":"2024-01-27T01:06:28.666039+0000","last_became_active":"2024-01-27T01:06:25.857867+0000","last_became_peered":"2024-01-27T01:06:25.857867+0000","last_unstale":"2024-01-27T01:06:28.666039+0000","last_undegraded":"2024-01-27T01:06:28.666039+0000","last_fullsized":"2024-01-27T01:06:28.666039+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:10:59.780192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666343+0000","last_change":"2024-01-27T01:06:26.880503+0000","last_active":"2024-01-27T01:06:28.666343+0000","last_peered":"2024-01-27T01:06:28.666343+0000","last_clean":"2024-01-27T01:06:28.666343+0000","last_became_active":"2024-01-27T01:06:26.880224+0000","last_became_peered":"2024-01-27T01:06:26.880224+0000","last_unstale":"2024-01-27T01:06:28.666343+0000","last_undegraded":"2024-01-27T01:06:28.666343+0000","last_fullsized":"2024-01-27T01:06:28.666343+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:27:33.886740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.911198+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:26:34.409044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665860+0000","last_change":"2024-01-27T01:06:25.859342+0000","last_active":"2024-01-27T01:06:28.665860+0000","last_peered":"2024-01-27T01:06:28.665860+0000","last_clean":"2024-01-27T01:06:28.665860+0000","last_became_active":"2024-01-27T01:06:25.859065+0000","last_became_peered":"2024-01-27T01:06:25.859065+0000","last_unstale":"2024-01-27T01:06:28.665860+0000","last_undegraded":"2024-01-27T01:06:28.665860+0000","last_fullsized":"2024-01-27T01:06:28.665860+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:00:54.297085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666399+0000","last_change":"2024-01-27T01:06:26.870597+0000","last_active":"2024-01-27T01:06:28.666399+0000","last_peered":"2024-01-27T01:06:28.666399+0000","last_clean":"2024-01-27T01:06:28.666399+0000","last_became_active":"2024-01-27T01:06:26.870334+0000","last_became_peered":"2024-01-27T01:06:26.870334+0000","last_unstale":"2024-01-27T01:06:28.666399+0000","last_undegraded":"2024-01-27T01:06:28.666399+0000","last_fullsized":"2024-01-27T01:06:28.666399+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:01:23.266072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913094+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:08:04.706593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355235+0000","last_change":"2024-01-27T01:06:25.858032+0000","last_active":"2024-01-27T01:06:28.355235+0000","last_peered":"2024-01-27T01:06:28.355235+0000","last_clean":"2024-01-27T01:06:28.355235+0000","last_became_active":"2024-01-27T01:06:25.857777+0000","last_became_peered":"2024-01-27T01:06:25.857777+0000","last_unstale":"2024-01-27T01:06:28.355235+0000","last_undegraded":"2024-01-27T01:06:28.355235+0000","last_fullsized":"2024-01-27T01:06:28.355235+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:09:08.458850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865801+0000","last_change":"2024-01-27T01:06:26.872190+0000","last_active":"2024-01-27T01:06:27.865801+0000","last_peered":"2024-01-27T01:06:27.865801+0000","last_clean":"2024-01-27T01:06:27.865801+0000","last_became_active":"2024-01-27T01:06:26.867718+0000","last_became_peered":"2024-01-27T01:06:26.867718+0000","last_unstale":"2024-01-27T01:06:27.865801+0000","last_undegraded":"2024-01-27T01:06:27.865801+0000","last_fullsized":"2024-01-27T01:06:27.865801+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:23.301132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.915265+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:55:49.427110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.6","version":"223'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:31.798965+0000","last_change":"2024-01-27T01:06:25.861717+0000","last_active":"2024-01-27T01:06:31.798965+0000","last_peered":"2024-01-27T01:06:31.798965+0000","last_clean":"2024-01-27T01:06:31.798965+0000","last_became_active":"2024-01-27T01:06:25.861177+0000","last_became_peered":"2024-01-27T01:06:25.861177+0000","last_unstale":"2024-01-27T01:06:31.798965+0000","last_undegraded":"2024-01-27T01:06:31.798965+0000","last_fullsized":"2024-01-27T01:06:31.798965+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:05.596045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240815+0000","last_change":"2024-01-27T01:06:26.878650+0000","last_active":"2024-01-27T01:06:28.240815+0000","last_peered":"2024-01-27T01:06:28.240815+0000","last_clean":"2024-01-27T01:06:28.240815+0000","last_became_active":"2024-01-27T01:06:26.877596+0000","last_became_peered":"2024-01-27T01:06:26.877596+0000","last_unstale":"2024-01-27T01:06:28.240815+0000","last_undegraded":"2024-01-27T01:06:28.240815+0000","last_fullsized":"2024-01-27T01:06:28.240815+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:42:48.942202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.914590+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:38:04.634853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.7","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866083+0000","last_change":"2024-01-27T01:06:25.857933+0000","last_active":"2024-01-27T01:06:27.866083+0000","last_peered":"2024-01-27T01:06:27.866083+0000","last_clean":"2024-01-27T01:06:27.866083+0000","last_became_active":"2024-01-27T01:06:25.857406+0000","last_became_peered":"2024-01-27T01:06:25.857406+0000","last_unstale":"2024-01-27T01:06:27.866083+0000","last_undegraded":"2024-01-27T01:06:27.866083+0000","last_fullsized":"2024-01-27T01:06:27.866083+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:20:20.239633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665632+0000","last_change":"2024-01-27T01:06:26.875222+0000","last_active":"2024-01-27T01:06:28.665632+0000","last_peered":"2024-01-27T01:06:28.665632+0000","last_clean":"2024-01-27T01:06:28.665632+0000","last_became_active":"2024-01-27T01:06:26.874903+0000","last_became_peered":"2024-01-27T01:06:26.874903+0000","last_unstale":"2024-01-27T01:06:28.665632+0000","last_undegraded":"2024-01-27T01:06:28.665632+0000","last_fullsized":"2024-01-27T01:06:28.665632+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:09:33.903445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916305+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:24.566246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665720+0000","last_change":"2024-01-27T01:06:25.853605+0000","last_active":"2024-01-27T01:06:28.665720+0000","last_peered":"2024-01-27T01:06:28.665720+0000","last_clean":"2024-01-27T01:06:28.665720+0000","last_became_active":"2024-01-27T01:06:25.853331+0000","last_became_peered":"2024-01-27T01:06:25.853331+0000","last_unstale":"2024-01-27T01:06:28.665720+0000","last_undegraded":"2024-01-27T01:06:28.665720+0000","last_fullsized":"2024-01-27T01:06:28.665720+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:09:08.458850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666459+0000","last_change":"2024-01-27T01:06:26.875536+0000","last_active":"2024-01-27T01:06:28.666459+0000","last_peered":"2024-01-27T01:06:28.666459+0000","last_clean":"2024-01-27T01:06:28.666459+0000","last_became_active":"2024-01-27T01:06:26.875267+0000","last_became_peered":"2024-01-27T01:06:26.875267+0000","last_unstale":"2024-01-27T01:06:28.666459+0000","last_undegraded":"2024-01-27T01:06:28.666459+0000","last_fullsized":"2024-01-27T01:06:28.666459+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:32.534622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916585+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:37:20.044223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355214+0000","last_change":"2024-01-27T01:06:25.862199+0000","last_active":"2024-01-27T01:06:28.355214+0000","last_peered":"2024-01-27T01:06:28.355214+0000","last_clean":"2024-01-27T01:06:28.355214+0000","last_became_active":"2024-01-27T01:06:25.861876+0000","last_became_peered":"2024-01-27T01:06:25.861876+0000","last_unstale":"2024-01-27T01:06:28.355214+0000","last_undegraded":"2024-01-27T01:06:28.355214+0000","last_fullsized":"2024-01-27T01:06:28.355214+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:00:54.297085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865614+0000","last_change":"2024-01-27T01:06:26.867570+0000","last_active":"2024-01-27T01:06:27.865614+0000","last_peered":"2024-01-27T01:06:27.865614+0000","last_clean":"2024-01-27T01:06:27.865614+0000","last_became_active":"2024-01-27T01:06:26.866517+0000","last_became_peered":"2024-01-27T01:06:26.866517+0000","last_unstale":"2024-01-27T01:06:27.865614+0000","last_undegraded":"2024-01-27T01:06:27.865614+0000","last_fullsized":"2024-01-27T01:06:27.865614+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:09:54.515702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910624+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:47:30.145789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866689+0000","last_change":"2024-01-27T01:06:25.851595+0000","last_active":"2024-01-27T01:06:27.866689+0000","last_peered":"2024-01-27T01:06:27.866689+0000","last_clean":"2024-01-27T01:06:27.866689+0000","last_became_active":"2024-01-27T01:06:25.851260+0000","last_became_peered":"2024-01-27T01:06:25.851260+0000","last_unstale":"2024-01-27T01:06:27.866689+0000","last_undegraded":"2024-01-27T01:06:27.866689+0000","last_fullsized":"2024-01-27T01:06:27.866689+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:56:53.179347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038203+0000","last_change":"2024-01-27T01:06:26.875858+0000","last_active":"2024-01-27T01:06:28.038203+0000","last_peered":"2024-01-27T01:06:28.038203+0000","last_clean":"2024-01-27T01:06:28.038203+0000","last_became_active":"2024-01-27T01:06:26.875564+0000","last_became_peered":"2024-01-27T01:06:26.875564+0000","last_unstale":"2024-01-27T01:06:28.038203+0000","last_undegraded":"2024-01-27T01:06:28.038203+0000","last_fullsized":"2024-01-27T01:06:28.038203+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:43.630831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913526+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:07:02.246428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.b","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866591+0000","last_change":"2024-01-27T01:06:25.860572+0000","last_active":"2024-01-27T01:06:27.866591+0000","last_peered":"2024-01-27T01:06:27.866591+0000","last_clean":"2024-01-27T01:06:27.866591+0000","last_became_active":"2024-01-27T01:06:25.860248+0000","last_became_peered":"2024-01-27T01:06:25.860248+0000","last_unstale":"2024-01-27T01:06:27.866591+0000","last_undegraded":"2024-01-27T01:06:27.866591+0000","last_fullsized":"2024-01-27T01:06:27.866591+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:01:25.741123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866157+0000","last_change":"2024-01-27T01:06:26.871346+0000","last_active":"2024-01-27T01:06:27.866157+0000","last_peered":"2024-01-27T01:06:27.866157+0000","last_clean":"2024-01-27T01:06:27.866157+0000","last_became_active":"2024-01-27T01:06:26.870999+0000","last_became_peered":"2024-01-27T01:06:26.870999+0000","last_unstale":"2024-01-27T01:06:27.866157+0000","last_undegraded":"2024-01-27T01:06:27.866157+0000","last_fullsized":"2024-01-27T01:06:27.866157+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:27.779157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.914309+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:31:18.366038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866703+0000","last_change":"2024-01-27T01:06:25.861650+0000","last_active":"2024-01-27T01:06:27.866703+0000","last_peered":"2024-01-27T01:06:27.866703+0000","last_clean":"2024-01-27T01:06:27.866703+0000","last_became_active":"2024-01-27T01:06:25.861342+0000","last_became_peered":"2024-01-27T01:06:25.861342+0000","last_unstale":"2024-01-27T01:06:27.866703+0000","last_undegraded":"2024-01-27T01:06:27.866703+0000","last_fullsized":"2024-01-27T01:06:27.866703+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:44:15.953217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665953+0000","last_change":"2024-01-27T01:06:26.876619+0000","last_active":"2024-01-27T01:06:28.665953+0000","last_peered":"2024-01-27T01:06:28.665953+0000","last_clean":"2024-01-27T01:06:28.665953+0000","last_became_active":"2024-01-27T01:06:26.875927+0000","last_became_peered":"2024-01-27T01:06:26.875927+0000","last_unstale":"2024-01-27T01:06:28.665953+0000","last_undegraded":"2024-01-27T01:06:28.665953+0000","last_fullsized":"2024-01-27T01:06:28.665953+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:02.000909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.911280+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:43:07.431223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.d","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.867031+0000","last_change":"2024-01-27T01:06:25.851720+0000","last_active":"2024-01-27T01:06:27.867031+0000","last_peered":"2024-01-27T01:06:27.867031+0000","last_clean":"2024-01-27T01:06:27.867031+0000","last_became_active":"2024-01-27T01:06:25.851416+0000","last_became_peered":"2024-01-27T01:06:25.851416+0000","last_unstale":"2024-01-27T01:06:27.867031+0000","last_undegraded":"2024-01-27T01:06:27.867031+0000","last_fullsized":"2024-01-27T01:06:27.867031+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:07.951814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666130+0000","last_change":"2024-01-27T01:06:26.870758+0000","last_active":"2024-01-27T01:06:28.666130+0000","last_peered":"2024-01-27T01:06:28.666130+0000","last_clean":"2024-01-27T01:06:28.666130+0000","last_became_active":"2024-01-27T01:06:26.870130+0000","last_became_peered":"2024-01-27T01:06:26.870130+0000","last_unstale":"2024-01-27T01:06:28.666130+0000","last_undegraded":"2024-01-27T01:06:28.666130+0000","last_fullsized":"2024-01-27T01:06:28.666130+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:36:31.625857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913563+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:49:22.695065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.e","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665974+0000","last_change":"2024-01-27T01:06:25.852245+0000","last_active":"2024-01-27T01:06:28.665974+0000","last_peered":"2024-01-27T01:06:28.665974+0000","last_clean":"2024-01-27T01:06:28.665974+0000","last_became_active":"2024-01-27T01:06:25.851982+0000","last_became_peered":"2024-01-27T01:06:25.851982+0000","last_unstale":"2024-01-27T01:06:28.665974+0000","last_undegraded":"2024-01-27T01:06:28.665974+0000","last_fullsized":"2024-01-27T01:06:28.665974+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:05.596045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665930+0000","last_change":"2024-01-27T01:06:26.879356+0000","last_active":"2024-01-27T01:06:28.665930+0000","last_peered":"2024-01-27T01:06:28.665930+0000","last_clean":"2024-01-27T01:06:28.665930+0000","last_became_active":"2024-01-27T01:06:26.878891+0000","last_became_peered":"2024-01-27T01:06:26.878891+0000","last_unstale":"2024-01-27T01:06:28.665930+0000","last_undegraded":"2024-01-27T01:06:28.665930+0000","last_fullsized":"2024-01-27T01:06:28.665930+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:22.011928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916563+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:01:36.332606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.f","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666321+0000","last_change":"2024-01-27T01:06:25.848900+0000","last_active":"2024-01-27T01:06:28.666321+0000","last_peered":"2024-01-27T01:06:28.666321+0000","last_clean":"2024-01-27T01:06:28.666321+0000","last_became_active":"2024-01-27T01:06:25.848483+0000","last_became_peered":"2024-01-27T01:06:25.848483+0000","last_unstale":"2024-01-27T01:06:28.666321+0000","last_undegraded":"2024-01-27T01:06:28.666321+0000","last_fullsized":"2024-01-27T01:06:28.666321+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:27:13.471892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665407+0000","last_change":"2024-01-27T01:06:26.875234+0000","last_active":"2024-01-27T01:06:28.665407+0000","last_peered":"2024-01-27T01:06:28.665407+0000","last_clean":"2024-01-27T01:06:28.665407+0000","last_became_active":"2024-01-27T01:06:26.874927+0000","last_became_peered":"2024-01-27T01:06:26.874927+0000","last_unstale":"2024-01-27T01:06:28.665407+0000","last_undegraded":"2024-01-27T01:06:28.665407+0000","last_fullsized":"2024-01-27T01:06:28.665407+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:10.390573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.911685+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:31.265591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.10","version":"222'8","reported_seq":19,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865571+0000","last_change":"2024-01-27T01:06:25.865239+0000","last_active":"2024-01-27T01:06:27.865571+0000","last_peered":"2024-01-27T01:06:27.865571+0000","last_clean":"2024-01-27T01:06:27.865571+0000","last_became_active":"2024-01-27T01:06:25.864943+0000","last_became_peered":"2024-01-27T01:06:25.864943+0000","last_unstale":"2024-01-27T01:06:27.865571+0000","last_undegraded":"2024-01-27T01:06:27.865571+0000","last_fullsized":"2024-01-27T01:06:27.865571+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:41:25.254751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038170+0000","last_change":"2024-01-27T01:06:26.867077+0000","last_active":"2024-01-27T01:06:28.038170+0000","last_peered":"2024-01-27T01:06:28.038170+0000","last_clean":"2024-01-27T01:06:28.038170+0000","last_became_active":"2024-01-27T01:06:26.866787+0000","last_became_peered":"2024-01-27T01:06:26.866787+0000","last_unstale":"2024-01-27T01:06:28.038170+0000","last_undegraded":"2024-01-27T01:06:28.038170+0000","last_fullsized":"2024-01-27T01:06:28.038170+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:02.000909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913156+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:56.697365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865574+0000","last_change":"2024-01-27T01:06:25.853597+0000","last_active":"2024-01-27T01:06:27.865574+0000","last_peered":"2024-01-27T01:06:27.865574+0000","last_clean":"2024-01-27T01:06:27.865574+0000","last_became_active":"2024-01-27T01:06:25.853257+0000","last_became_peered":"2024-01-27T01:06:25.853257+0000","last_unstale":"2024-01-27T01:06:27.865574+0000","last_undegraded":"2024-01-27T01:06:27.865574+0000","last_fullsized":"2024-01-27T01:06:27.865574+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:54:35.011796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865516+0000","last_change":"2024-01-27T01:06:26.866613+0000","last_active":"2024-01-27T01:06:27.865516+0000","last_peered":"2024-01-27T01:06:27.865516+0000","last_clean":"2024-01-27T01:06:27.865516+0000","last_became_active":"2024-01-27T01:06:26.866298+0000","last_became_peered":"2024-01-27T01:06:26.866298+0000","last_unstale":"2024-01-27T01:06:27.865516+0000","last_undegraded":"2024-01-27T01:06:27.865516+0000","last_fullsized":"2024-01-27T01:06:27.865516+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:27:03.934654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.917128+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:57:15.231912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240972+0000","last_change":"2024-01-27T01:06:25.854192+0000","last_active":"2024-01-27T01:06:28.240972+0000","last_peered":"2024-01-27T01:06:28.240972+0000","last_clean":"2024-01-27T01:06:28.240972+0000","last_became_active":"2024-01-27T01:06:25.853601+0000","last_became_peered":"2024-01-27T01:06:25.853601+0000","last_unstale":"2024-01-27T01:06:28.240972+0000","last_undegraded":"2024-01-27T01:06:28.240972+0000","last_fullsized":"2024-01-27T01:06:28.240972+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:42.218792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.241137+0000","last_change":"2024-01-27T01:06:26.862905+0000","last_active":"2024-01-27T01:06:28.241137+0000","last_peered":"2024-01-27T01:06:28.241137+0000","last_clean":"2024-01-27T01:06:28.241137+0000","last_became_active":"2024-01-27T01:06:26.862584+0000","last_became_peered":"2024-01-27T01:06:26.862584+0000","last_unstale":"2024-01-27T01:06:28.241137+0000","last_undegraded":"2024-01-27T01:06:28.241137+0000","last_fullsized":"2024-01-27T01:06:28.241137+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:27:14.653781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910329+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:19:00.298602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.13","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866863+0000","last_change":"2024-01-27T01:06:25.850061+0000","last_active":"2024-01-27T01:06:27.866863+0000","last_peered":"2024-01-27T01:06:27.866863+0000","last_clean":"2024-01-27T01:06:27.866863+0000","last_became_active":"2024-01-27T01:06:25.849763+0000","last_became_peered":"2024-01-27T01:06:25.849763+0000","last_unstale":"2024-01-27T01:06:27.866863+0000","last_undegraded":"2024-01-27T01:06:27.866863+0000","last_fullsized":"2024-01-27T01:06:27.866863+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:09:56.243562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866051+0000","last_change":"2024-01-27T01:06:26.873464+0000","last_active":"2024-01-27T01:06:27.866051+0000","last_peered":"2024-01-27T01:06:27.866051+0000","last_clean":"2024-01-27T01:06:27.866051+0000","last_became_active":"2024-01-27T01:06:26.873063+0000","last_became_peered":"2024-01-27T01:06:26.873063+0000","last_unstale":"2024-01-27T01:06:27.866051+0000","last_undegraded":"2024-01-27T01:06:27.866051+0000","last_fullsized":"2024-01-27T01:06:27.866051+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:33:32.535574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916917+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:36:16.613510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.14","version":"223'8","reported_seq":19,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:36.798321+0000","last_change":"2024-01-27T01:06:25.864160+0000","last_active":"2024-01-27T01:06:36.798321+0000","last_peered":"2024-01-27T01:06:36.798321+0000","last_clean":"2024-01-27T01:06:36.798321+0000","last_became_active":"2024-01-27T01:06:25.863867+0000","last_became_peered":"2024-01-27T01:06:25.863867+0000","last_unstale":"2024-01-27T01:06:36.798321+0000","last_undegraded":"2024-01-27T01:06:36.798321+0000","last_fullsized":"2024-01-27T01:06:36.798321+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:43.338484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2768,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866494+0000","last_change":"2024-01-27T01:06:26.879652+0000","last_active":"2024-01-27T01:06:27.866494+0000","last_peered":"2024-01-27T01:06:27.866494+0000","last_clean":"2024-01-27T01:06:27.866494+0000","last_became_active":"2024-01-27T01:06:26.879271+0000","last_became_peered":"2024-01-27T01:06:26.879271+0000","last_unstale":"2024-01-27T01:06:27.866494+0000","last_undegraded":"2024-01-27T01:06:27.866494+0000","last_fullsized":"2024-01-27T01:06:27.866494+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:57:23.456731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.914165+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:27:24.223960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.15","version":"222'3","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866134+0000","last_change":"2024-01-27T01:06:25.857771+0000","last_active":"2024-01-27T01:06:27.866134+0000","last_peered":"2024-01-27T01:06:27.866134+0000","last_clean":"2024-01-27T01:06:27.866134+0000","last_became_active":"2024-01-27T01:06:25.857445+0000","last_became_peered":"2024-01-27T01:06:25.857445+0000","last_unstale":"2024-01-27T01:06:27.866134+0000","last_undegraded":"2024-01-27T01:06:27.866134+0000","last_fullsized":"2024-01-27T01:06:27.866134+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:52.927866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865698+0000","last_change":"2024-01-27T01:06:26.876685+0000","last_active":"2024-01-27T01:06:27.865698+0000","last_peered":"2024-01-27T01:06:27.865698+0000","last_clean":"2024-01-27T01:06:27.865698+0000","last_became_active":"2024-01-27T01:06:26.872939+0000","last_became_peered":"2024-01-27T01:06:26.872939+0000","last_unstale":"2024-01-27T01:06:27.865698+0000","last_undegraded":"2024-01-27T01:06:27.865698+0000","last_fullsized":"2024-01-27T01:06:27.865698+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:56:48.730726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.915793+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:36:16.613510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038259+0000","last_change":"2024-01-27T01:06:25.860762+0000","last_active":"2024-01-27T01:06:28.038259+0000","last_peered":"2024-01-27T01:06:28.038259+0000","last_clean":"2024-01-27T01:06:28.038259+0000","last_became_active":"2024-01-27T01:06:25.860459+0000","last_became_peered":"2024-01-27T01:06:25.860459+0000","last_unstale":"2024-01-27T01:06:28.038259+0000","last_undegraded":"2024-01-27T01:06:28.038259+0000","last_fullsized":"2024-01-27T01:06:28.038259+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:31.687265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240842+0000","last_change":"2024-01-27T01:06:26.868859+0000","last_active":"2024-01-27T01:06:28.240842+0000","last_peered":"2024-01-27T01:06:28.240842+0000","last_clean":"2024-01-27T01:06:28.240842+0000","last_became_active":"2024-01-27T01:06:26.868623+0000","last_became_peered":"2024-01-27T01:06:26.868623+0000","last_unstale":"2024-01-27T01:06:28.240842+0000","last_undegraded":"2024-01-27T01:06:28.240842+0000","last_fullsized":"2024-01-27T01:06:28.240842+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:18:47.232067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.915065+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:24.566246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866791+0000","last_change":"2024-01-27T01:06:25.865314+0000","last_active":"2024-01-27T01:06:27.866791+0000","last_peered":"2024-01-27T01:06:27.866791+0000","last_clean":"2024-01-27T01:06:27.866791+0000","last_became_active":"2024-01-27T01:06:25.864940+0000","last_became_peered":"2024-01-27T01:06:25.864940+0000","last_unstale":"2024-01-27T01:06:27.866791+0000","last_undegraded":"2024-01-27T01:06:27.866791+0000","last_fullsized":"2024-01-27T01:06:27.866791+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:24.113986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355011+0000","last_change":"2024-01-27T01:06:26.870806+0000","last_active":"2024-01-27T01:06:28.355011+0000","last_peered":"2024-01-27T01:06:28.355011+0000","last_clean":"2024-01-27T01:06:28.355011+0000","last_became_active":"2024-01-27T01:06:26.870263+0000","last_became_peered":"2024-01-27T01:06:26.870263+0000","last_unstale":"2024-01-27T01:06:28.355011+0000","last_undegraded":"2024-01-27T01:06:28.355011+0000","last_fullsized":"2024-01-27T01:06:28.355011+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:26.214953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910328+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:24.032004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240953+0000","last_change":"2024-01-27T01:06:25.851888+0000","last_active":"2024-01-27T01:06:28.240953+0000","last_peered":"2024-01-27T01:06:28.240953+0000","last_clean":"2024-01-27T01:06:28.240953+0000","last_became_active":"2024-01-27T01:06:25.851605+0000","last_became_peered":"2024-01-27T01:06:25.851605+0000","last_unstale":"2024-01-27T01:06:28.240953+0000","last_undegraded":"2024-01-27T01:06:28.240953+0000","last_fullsized":"2024-01-27T01:06:28.240953+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:09.386094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865985+0000","last_change":"2024-01-27T01:06:26.873199+0000","last_active":"2024-01-27T01:06:27.865985+0000","last_peered":"2024-01-27T01:06:27.865985+0000","last_clean":"2024-01-27T01:06:27.865985+0000","last_became_active":"2024-01-27T01:06:26.872658+0000","last_became_peered":"2024-01-27T01:06:26.872658+0000","last_unstale":"2024-01-27T01:06:27.865985+0000","last_undegraded":"2024-01-27T01:06:27.865985+0000","last_fullsized":"2024-01-27T01:06:27.865985+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:06:45.526347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913764+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:10:10.314575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.037877+0000","last_change":"2024-01-27T01:06:25.854878+0000","last_active":"2024-01-27T01:06:28.037877+0000","last_peered":"2024-01-27T01:06:28.037877+0000","last_clean":"2024-01-27T01:06:28.037877+0000","last_became_active":"2024-01-27T01:06:25.854623+0000","last_became_peered":"2024-01-27T01:06:25.854623+0000","last_unstale":"2024-01-27T01:06:28.037877+0000","last_undegraded":"2024-01-27T01:06:28.037877+0000","last_fullsized":"2024-01-27T01:06:28.037877+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:21.007449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355292+0000","last_change":"2024-01-27T01:06:26.876963+0000","last_active":"2024-01-27T01:06:28.355292+0000","last_peered":"2024-01-27T01:06:28.355292+0000","last_clean":"2024-01-27T01:06:28.355292+0000","last_became_active":"2024-01-27T01:06:26.876709+0000","last_became_peered":"2024-01-27T01:06:26.876709+0000","last_unstale":"2024-01-27T01:06:28.355292+0000","last_undegraded":"2024-01-27T01:06:28.355292+0000","last_fullsized":"2024-01-27T01:06:28.355292+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:58:14.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916960+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:57.409497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1b","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038235+0000","last_change":"2024-01-27T01:06:25.854193+0000","last_active":"2024-01-27T01:06:28.038235+0000","last_peered":"2024-01-27T01:06:28.038235+0000","last_clean":"2024-01-27T01:06:28.038235+0000","last_became_active":"2024-01-27T01:06:25.853956+0000","last_became_peered":"2024-01-27T01:06:25.853956+0000","last_unstale":"2024-01-27T01:06:28.038235+0000","last_undegraded":"2024-01-27T01:06:28.038235+0000","last_fullsized":"2024-01-27T01:06:28.038235+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:33:02.747087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355322+0000","last_change":"2024-01-27T01:06:26.866121+0000","last_active":"2024-01-27T01:06:28.355322+0000","last_peered":"2024-01-27T01:06:28.355322+0000","last_clean":"2024-01-27T01:06:28.355322+0000","last_became_active":"2024-01-27T01:06:26.865391+0000","last_became_peered":"2024-01-27T01:06:26.865391+0000","last_unstale":"2024-01-27T01:06:28.355322+0000","last_undegraded":"2024-01-27T01:06:28.355322+0000","last_fullsized":"2024-01-27T01:06:28.355322+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:02:03.812030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910329+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:27:27.720315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1c","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866341+0000","last_change":"2024-01-27T01:06:25.855114+0000","last_active":"2024-01-27T01:06:27.866341+0000","last_peered":"2024-01-27T01:06:27.866341+0000","last_clean":"2024-01-27T01:06:27.866341+0000","last_became_active":"2024-01-27T01:06:25.854790+0000","last_became_peered":"2024-01-27T01:06:25.854790+0000","last_unstale":"2024-01-27T01:06:27.866341+0000","last_undegraded":"2024-01-27T01:06:27.866341+0000","last_fullsized":"2024-01-27T01:06:27.866341+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:58:58.498377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866394+0000","last_change":"2024-01-27T01:06:26.869128+0000","last_active":"2024-01-27T01:06:27.866394+0000","last_peered":"2024-01-27T01:06:27.866394+0000","last_clean":"2024-01-27T01:06:27.866394+0000","last_became_active":"2024-01-27T01:06:26.868892+0000","last_became_peered":"2024-01-27T01:06:26.868892+0000","last_unstale":"2024-01-27T01:06:27.866394+0000","last_undegraded":"2024-01-27T01:06:27.866394+0000","last_fullsized":"2024-01-27T01:06:27.866394+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:40:40.869516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.916917+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:31.265591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1d","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866926+0000","last_change":"2024-01-27T01:06:25.855991+0000","last_active":"2024-01-27T01:06:27.866926+0000","last_peered":"2024-01-27T01:06:27.866926+0000","last_clean":"2024-01-27T01:06:27.866926+0000","last_became_active":"2024-01-27T01:06:25.855679+0000","last_became_peered":"2024-01-27T01:06:25.855679+0000","last_unstale":"2024-01-27T01:06:27.866926+0000","last_undegraded":"2024-01-27T01:06:27.866926+0000","last_fullsized":"2024-01-27T01:06:27.866926+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:31:04.295025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666155+0000","last_change":"2024-01-27T01:06:26.875831+0000","last_active":"2024-01-27T01:06:28.666155+0000","last_peered":"2024-01-27T01:06:28.666155+0000","last_clean":"2024-01-27T01:06:28.666155+0000","last_became_active":"2024-01-27T01:06:26.875488+0000","last_became_peered":"2024-01-27T01:06:26.875488+0000","last_unstale":"2024-01-27T01:06:28.666155+0000","last_undegraded":"2024-01-27T01:06:28.666155+0000","last_fullsized":"2024-01-27T01:06:28.666155+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:07:51.640059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910981+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:57:15.231912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1e","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665879+0000","last_change":"2024-01-27T01:06:25.856504+0000","last_active":"2024-01-27T01:06:28.665879+0000","last_peered":"2024-01-27T01:06:28.665879+0000","last_clean":"2024-01-27T01:06:28.665879+0000","last_became_active":"2024-01-27T01:06:25.856272+0000","last_became_peered":"2024-01-27T01:06:25.856272+0000","last_unstale":"2024-01-27T01:06:28.665879+0000","last_undegraded":"2024-01-27T01:06:28.665879+0000","last_fullsized":"2024-01-27T01:06:28.665879+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:20:56.736428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666038+0000","last_change":"2024-01-27T01:06:26.878625+0000","last_active":"2024-01-27T01:06:28.666038+0000","last_peered":"2024-01-27T01:06:28.666038+0000","last_clean":"2024-01-27T01:06:28.666038+0000","last_became_active":"2024-01-27T01:06:26.878361+0000","last_became_peered":"2024-01-27T01:06:26.878361+0000","last_unstale":"2024-01-27T01:06:28.666038+0000","last_undegraded":"2024-01-27T01:06:28.666038+0000","last_fullsized":"2024-01-27T01:06:28.666038+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:37:06.977689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.910594+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:31:50.703251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1f","version":"222'4","reported_seq":15,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355378+0000","last_change":"2024-01-27T01:06:25.856458+0000","last_active":"2024-01-27T01:06:28.355378+0000","last_peered":"2024-01-27T01:06:28.355378+0000","last_clean":"2024-01-27T01:06:28.355378+0000","last_became_active":"2024-01-27T01:06:25.856162+0000","last_became_peered":"2024-01-27T01:06:25.856162+0000","last_unstale":"2024-01-27T01:06:28.355378+0000","last_undegraded":"2024-01-27T01:06:28.355378+0000","last_fullsized":"2024-01-27T01:06:28.355378+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:51:44.378880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.867070+0000","last_change":"2024-01-27T01:06:26.869588+0000","last_active":"2024-01-27T01:06:27.867070+0000","last_peered":"2024-01-27T01:06:27.867070+0000","last_clean":"2024-01-27T01:06:27.867070+0000","last_became_active":"2024-01-27T01:06:26.869264+0000","last_became_peered":"2024-01-27T01:06:26.869264+0000","last_unstale":"2024-01-27T01:06:27.867070+0000","last_undegraded":"2024-01-27T01:06:27.867070+0000","last_fullsized":"2024-01-27T01:06:27.867070+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:42:54.364688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-27T01:06:38.901811+0000","last_change":"2024-01-27T01:06:38.913160+0000","last_active":"2024-01-27T01:06:38.901811+0000","last_peered":"2024-01-27T01:06:38.901811+0000","last_clean":"2024-01-27T01:06:38.901811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:38.901811+0000","last_undegraded":"2024-01-27T01:06:38.901811+0000","last_fullsized":"2024-01-27T01:06:38.901811+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:15.067443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.037884+0000","last_change":"2024-01-27T01:06:26.869029+0000","last_active":"2024-01-27T01:06:28.037884+0000","last_peered":"2024-01-27T01:06:28.037884+0000","last_clean":"2024-01-27T01:06:28.037884+0000","last_became_active":"2024-01-27T01:06:26.868793+0000","last_became_peered":"2024-01-27T01:06:26.868793+0000","last_unstale":"2024-01-27T01:06:28.037884+0000","last_undegraded":"2024-01-27T01:06:28.037884+0000","last_fullsized":"2024-01-27T01:06:28.037884+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:10.965469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.037888+0000","last_change":"2024-01-27T01:06:26.876343+0000","last_active":"2024-01-27T01:06:28.037888+0000","last_peered":"2024-01-27T01:06:28.037888+0000","last_clean":"2024-01-27T01:06:28.037888+0000","last_became_active":"2024-01-27T01:06:26.875570+0000","last_became_peered":"2024-01-27T01:06:26.875570+0000","last_unstale":"2024-01-27T01:06:28.037888+0000","last_undegraded":"2024-01-27T01:06:28.037888+0000","last_fullsized":"2024-01-27T01:06:28.037888+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:07:51.640059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355113+0000","last_change":"2024-01-27T01:06:26.871676+0000","last_active":"2024-01-27T01:06:28.355113+0000","last_peered":"2024-01-27T01:06:28.355113+0000","last_clean":"2024-01-27T01:06:28.355113+0000","last_became_active":"2024-01-27T01:06:26.871423+0000","last_became_peered":"2024-01-27T01:06:26.871423+0000","last_unstale":"2024-01-27T01:06:28.355113+0000","last_undegraded":"2024-01-27T01:06:28.355113+0000","last_fullsized":"2024-01-27T01:06:28.355113+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:29:58.146876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866078+0000","last_change":"2024-01-27T01:06:26.872013+0000","last_active":"2024-01-27T01:06:27.866078+0000","last_peered":"2024-01-27T01:06:27.866078+0000","last_clean":"2024-01-27T01:06:27.866078+0000","last_became_active":"2024-01-27T01:06:26.867577+0000","last_became_peered":"2024-01-27T01:06:26.867577+0000","last_unstale":"2024-01-27T01:06:27.866078+0000","last_undegraded":"2024-01-27T01:06:27.866078+0000","last_fullsized":"2024-01-27T01:06:27.866078+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:27.604574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866848+0000","last_change":"2024-01-27T01:06:26.870649+0000","last_active":"2024-01-27T01:06:27.866848+0000","last_peered":"2024-01-27T01:06:27.866848+0000","last_clean":"2024-01-27T01:06:27.866848+0000","last_became_active":"2024-01-27T01:06:26.870433+0000","last_became_peered":"2024-01-27T01:06:26.870433+0000","last_unstale":"2024-01-27T01:06:27.866848+0000","last_undegraded":"2024-01-27T01:06:27.866848+0000","last_fullsized":"2024-01-27T01:06:27.866848+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:11:27.402943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.241052+0000","last_change":"2024-01-27T01:06:26.878202+0000","last_active":"2024-01-27T01:06:28.241052+0000","last_peered":"2024-01-27T01:06:28.241052+0000","last_clean":"2024-01-27T01:06:28.241052+0000","last_became_active":"2024-01-27T01:06:26.877900+0000","last_became_peered":"2024-01-27T01:06:26.877900+0000","last_unstale":"2024-01-27T01:06:28.241052+0000","last_undegraded":"2024-01-27T01:06:28.241052+0000","last_fullsized":"2024-01-27T01:06:28.241052+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:42.766155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665916+0000","last_change":"2024-01-27T01:06:26.878911+0000","last_active":"2024-01-27T01:06:28.665916+0000","last_peered":"2024-01-27T01:06:28.665916+0000","last_clean":"2024-01-27T01:06:28.665916+0000","last_became_active":"2024-01-27T01:06:26.878637+0000","last_became_peered":"2024-01-27T01:06:26.878637+0000","last_unstale":"2024-01-27T01:06:28.665916+0000","last_undegraded":"2024-01-27T01:06:28.665916+0000","last_fullsized":"2024-01-27T01:06:28.665916+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:33:03.751565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865886+0000","last_change":"2024-01-27T01:06:26.865149+0000","last_active":"2024-01-27T01:06:27.865886+0000","last_peered":"2024-01-27T01:06:27.865886+0000","last_clean":"2024-01-27T01:06:27.865886+0000","last_became_active":"2024-01-27T01:06:26.864853+0000","last_became_peered":"2024-01-27T01:06:26.864853+0000","last_unstale":"2024-01-27T01:06:27.865886+0000","last_undegraded":"2024-01-27T01:06:27.865886+0000","last_fullsized":"2024-01-27T01:06:27.865886+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:10:44.194323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865471+0000","last_change":"2024-01-27T01:06:26.867241+0000","last_active":"2024-01-27T01:06:27.865471+0000","last_peered":"2024-01-27T01:06:27.865471+0000","last_clean":"2024-01-27T01:06:27.865471+0000","last_became_active":"2024-01-27T01:06:26.866917+0000","last_became_peered":"2024-01-27T01:06:26.866917+0000","last_unstale":"2024-01-27T01:06:27.865471+0000","last_undegraded":"2024-01-27T01:06:27.865471+0000","last_fullsized":"2024-01-27T01:06:27.865471+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:24:25.818592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865972+0000","last_change":"2024-01-27T01:06:26.874471+0000","last_active":"2024-01-27T01:06:27.865972+0000","last_peered":"2024-01-27T01:06:27.865972+0000","last_clean":"2024-01-27T01:06:27.865972+0000","last_became_active":"2024-01-27T01:06:26.874271+0000","last_became_peered":"2024-01-27T01:06:26.874271+0000","last_unstale":"2024-01-27T01:06:27.865972+0000","last_undegraded":"2024-01-27T01:06:27.865972+0000","last_fullsized":"2024-01-27T01:06:27.865972+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:15:11.295547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665512+0000","last_change":"2024-01-27T01:06:26.876390+0000","last_active":"2024-01-27T01:06:28.665512+0000","last_peered":"2024-01-27T01:06:28.665512+0000","last_clean":"2024-01-27T01:06:28.665512+0000","last_became_active":"2024-01-27T01:06:26.876056+0000","last_became_peered":"2024-01-27T01:06:26.876056+0000","last_unstale":"2024-01-27T01:06:28.665512+0000","last_undegraded":"2024-01-27T01:06:28.665512+0000","last_fullsized":"2024-01-27T01:06:28.665512+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:38:36.236958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665852+0000","last_change":"2024-01-27T01:06:26.879441+0000","last_active":"2024-01-27T01:06:28.665852+0000","last_peered":"2024-01-27T01:06:28.665852+0000","last_clean":"2024-01-27T01:06:28.665852+0000","last_became_active":"2024-01-27T01:06:26.879148+0000","last_became_peered":"2024-01-27T01:06:26.879148+0000","last_unstale":"2024-01-27T01:06:28.665852+0000","last_undegraded":"2024-01-27T01:06:28.665852+0000","last_fullsized":"2024-01-27T01:06:28.665852+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:05:08.012027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666259+0000","last_change":"2024-01-27T01:06:26.876823+0000","last_active":"2024-01-27T01:06:28.666259+0000","last_peered":"2024-01-27T01:06:28.666259+0000","last_clean":"2024-01-27T01:06:28.666259+0000","last_became_active":"2024-01-27T01:06:26.876304+0000","last_became_peered":"2024-01-27T01:06:26.876304+0000","last_unstale":"2024-01-27T01:06:28.666259+0000","last_undegraded":"2024-01-27T01:06:28.666259+0000","last_fullsized":"2024-01-27T01:06:28.666259+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:28.378677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666099+0000","last_change":"2024-01-27T01:06:26.873401+0000","last_active":"2024-01-27T01:06:28.666099+0000","last_peered":"2024-01-27T01:06:28.666099+0000","last_clean":"2024-01-27T01:06:28.666099+0000","last_became_active":"2024-01-27T01:06:26.873094+0000","last_became_peered":"2024-01-27T01:06:26.873094+0000","last_unstale":"2024-01-27T01:06:28.666099+0000","last_undegraded":"2024-01-27T01:06:28.666099+0000","last_fullsized":"2024-01-27T01:06:28.666099+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:24.082749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865407+0000","last_change":"2024-01-27T01:06:26.864890+0000","last_active":"2024-01-27T01:06:27.865407+0000","last_peered":"2024-01-27T01:06:27.865407+0000","last_clean":"2024-01-27T01:06:27.865407+0000","last_became_active":"2024-01-27T01:06:26.864556+0000","last_became_peered":"2024-01-27T01:06:26.864556+0000","last_unstale":"2024-01-27T01:06:27.865407+0000","last_undegraded":"2024-01-27T01:06:27.865407+0000","last_fullsized":"2024-01-27T01:06:27.865407+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:41.033580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865791+0000","last_change":"2024-01-27T01:06:26.874473+0000","last_active":"2024-01-27T01:06:27.865791+0000","last_peered":"2024-01-27T01:06:27.865791+0000","last_clean":"2024-01-27T01:06:27.865791+0000","last_became_active":"2024-01-27T01:06:26.874203+0000","last_became_peered":"2024-01-27T01:06:26.874203+0000","last_unstale":"2024-01-27T01:06:27.865791+0000","last_undegraded":"2024-01-27T01:06:27.865791+0000","last_fullsized":"2024-01-27T01:06:27.865791+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:33:37.975146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240809+0000","last_change":"2024-01-27T01:06:26.863380+0000","last_active":"2024-01-27T01:06:28.240809+0000","last_peered":"2024-01-27T01:06:28.240809+0000","last_clean":"2024-01-27T01:06:28.240809+0000","last_became_active":"2024-01-27T01:06:26.863034+0000","last_became_peered":"2024-01-27T01:06:26.863034+0000","last_unstale":"2024-01-27T01:06:28.240809+0000","last_undegraded":"2024-01-27T01:06:28.240809+0000","last_fullsized":"2024-01-27T01:06:28.240809+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:32.271904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.355189+0000","last_change":"2024-01-27T01:06:26.877929+0000","last_active":"2024-01-27T01:06:28.355189+0000","last_peered":"2024-01-27T01:06:28.355189+0000","last_clean":"2024-01-27T01:06:28.355189+0000","last_became_active":"2024-01-27T01:06:26.877679+0000","last_became_peered":"2024-01-27T01:06:26.877679+0000","last_unstale":"2024-01-27T01:06:28.355189+0000","last_undegraded":"2024-01-27T01:06:28.355189+0000","last_fullsized":"2024-01-27T01:06:28.355189+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:53:15.482586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665563+0000","last_change":"2024-01-27T01:06:26.876674+0000","last_active":"2024-01-27T01:06:28.665563+0000","last_peered":"2024-01-27T01:06:28.665563+0000","last_clean":"2024-01-27T01:06:28.665563+0000","last_became_active":"2024-01-27T01:06:26.876413+0000","last_became_peered":"2024-01-27T01:06:26.876413+0000","last_unstale":"2024-01-27T01:06:28.665563+0000","last_undegraded":"2024-01-27T01:06:28.665563+0000","last_fullsized":"2024-01-27T01:06:28.665563+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:08.822886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038257+0000","last_change":"2024-01-27T01:06:26.877777+0000","last_active":"2024-01-27T01:06:28.038257+0000","last_peered":"2024-01-27T01:06:28.038257+0000","last_clean":"2024-01-27T01:06:28.038257+0000","last_became_active":"2024-01-27T01:06:26.877515+0000","last_became_peered":"2024-01-27T01:06:26.877515+0000","last_unstale":"2024-01-27T01:06:28.038257+0000","last_undegraded":"2024-01-27T01:06:28.038257+0000","last_fullsized":"2024-01-27T01:06:28.038257+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:27:14.653781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866589+0000","last_change":"2024-01-27T01:06:26.865833+0000","last_active":"2024-01-27T01:06:27.866589+0000","last_peered":"2024-01-27T01:06:27.866589+0000","last_clean":"2024-01-27T01:06:27.866589+0000","last_became_active":"2024-01-27T01:06:26.865525+0000","last_became_peered":"2024-01-27T01:06:26.865525+0000","last_unstale":"2024-01-27T01:06:27.866589+0000","last_undegraded":"2024-01-27T01:06:27.866589+0000","last_fullsized":"2024-01-27T01:06:27.866589+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:42:10.990703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.240896+0000","last_change":"2024-01-27T01:06:26.879535+0000","last_active":"2024-01-27T01:06:28.240896+0000","last_peered":"2024-01-27T01:06:28.240896+0000","last_clean":"2024-01-27T01:06:28.240896+0000","last_became_active":"2024-01-27T01:06:26.879219+0000","last_became_peered":"2024-01-27T01:06:26.879219+0000","last_unstale":"2024-01-27T01:06:28.240896+0000","last_undegraded":"2024-01-27T01:06:28.240896+0000","last_fullsized":"2024-01-27T01:06:28.240896+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:58.495062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666051+0000","last_change":"2024-01-27T01:06:26.876736+0000","last_active":"2024-01-27T01:06:28.666051+0000","last_peered":"2024-01-27T01:06:28.666051+0000","last_clean":"2024-01-27T01:06:28.666051+0000","last_became_active":"2024-01-27T01:06:26.875870+0000","last_became_peered":"2024-01-27T01:06:26.875870+0000","last_unstale":"2024-01-27T01:06:28.666051+0000","last_undegraded":"2024-01-27T01:06:28.666051+0000","last_fullsized":"2024-01-27T01:06:28.666051+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:43.223271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865691+0000","last_change":"2024-01-27T01:06:26.873409+0000","last_active":"2024-01-27T01:06:27.865691+0000","last_peered":"2024-01-27T01:06:27.865691+0000","last_clean":"2024-01-27T01:06:27.865691+0000","last_became_active":"2024-01-27T01:06:26.868026+0000","last_became_peered":"2024-01-27T01:06:26.868026+0000","last_unstale":"2024-01-27T01:06:27.865691+0000","last_undegraded":"2024-01-27T01:06:27.865691+0000","last_fullsized":"2024-01-27T01:06:27.865691+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:15.664932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.665379+0000","last_change":"2024-01-27T01:06:26.881786+0000","last_active":"2024-01-27T01:06:28.665379+0000","last_peered":"2024-01-27T01:06:28.665379+0000","last_clean":"2024-01-27T01:06:28.665379+0000","last_became_active":"2024-01-27T01:06:26.881527+0000","last_became_peered":"2024-01-27T01:06:26.881527+0000","last_unstale":"2024-01-27T01:06:28.665379+0000","last_undegraded":"2024-01-27T01:06:28.665379+0000","last_fullsized":"2024-01-27T01:06:28.665379+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:43.630831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.865757+0000","last_change":"2024-01-27T01:06:26.872094+0000","last_active":"2024-01-27T01:06:27.865757+0000","last_peered":"2024-01-27T01:06:27.865757+0000","last_clean":"2024-01-27T01:06:27.865757+0000","last_became_active":"2024-01-27T01:06:26.871711+0000","last_became_peered":"2024-01-27T01:06:26.871711+0000","last_unstale":"2024-01-27T01:06:27.865757+0000","last_undegraded":"2024-01-27T01:06:27.865757+0000","last_fullsized":"2024-01-27T01:06:27.865757+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:24:32.821570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.037950+0000","last_change":"2024-01-27T01:06:26.879291+0000","last_active":"2024-01-27T01:06:28.037950+0000","last_peered":"2024-01-27T01:06:28.037950+0000","last_clean":"2024-01-27T01:06:28.037950+0000","last_became_active":"2024-01-27T01:06:26.879063+0000","last_became_peered":"2024-01-27T01:06:26.879063+0000","last_unstale":"2024-01-27T01:06:28.037950+0000","last_undegraded":"2024-01-27T01:06:28.037950+0000","last_fullsized":"2024-01-27T01:06:28.037950+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:22:23.545688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866284+0000","last_change":"2024-01-27T01:06:26.870422+0000","last_active":"2024-01-27T01:06:27.866284+0000","last_peered":"2024-01-27T01:06:27.866284+0000","last_clean":"2024-01-27T01:06:27.866284+0000","last_became_active":"2024-01-27T01:06:26.870117+0000","last_became_peered":"2024-01-27T01:06:26.870117+0000","last_unstale":"2024-01-27T01:06:27.866284+0000","last_undegraded":"2024-01-27T01:06:27.866284+0000","last_fullsized":"2024-01-27T01:06:27.866284+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:42:55.987157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.038255+0000","last_change":"2024-01-27T01:06:26.868177+0000","last_active":"2024-01-27T01:06:28.038255+0000","last_peered":"2024-01-27T01:06:28.038255+0000","last_clean":"2024-01-27T01:06:28.038255+0000","last_became_active":"2024-01-27T01:06:26.867906+0000","last_became_peered":"2024-01-27T01:06:26.867906+0000","last_unstale":"2024-01-27T01:06:28.038255+0000","last_undegraded":"2024-01-27T01:06:28.038255+0000","last_fullsized":"2024-01-27T01:06:28.038255+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:32.271904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.241039+0000","last_change":"2024-01-27T01:06:26.868462+0000","last_active":"2024-01-27T01:06:28.241039+0000","last_peered":"2024-01-27T01:06:28.241039+0000","last_clean":"2024-01-27T01:06:28.241039+0000","last_became_active":"2024-01-27T01:06:26.868203+0000","last_became_peered":"2024-01-27T01:06:26.868203+0000","last_unstale":"2024-01-27T01:06:28.241039+0000","last_undegraded":"2024-01-27T01:06:28.241039+0000","last_fullsized":"2024-01-27T01:06:28.241039+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:10.965469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:27.866164+0000","last_change":"2024-01-27T01:06:26.871135+0000","last_active":"2024-01-27T01:06:27.866164+0000","last_peered":"2024-01-27T01:06:27.866164+0000","last_clean":"2024-01-27T01:06:27.866164+0000","last_became_active":"2024-01-27T01:06:26.870854+0000","last_became_peered":"2024-01-27T01:06:26.870854+0000","last_unstale":"2024-01-27T01:06:27.866164+0000","last_undegraded":"2024-01-27T01:06:27.866164+0000","last_fullsized":"2024-01-27T01:06:27.866164+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:12:59.875769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-27T01:06:39.907896+0000","last_change":"2024-01-27T01:06:39.907896+0000","last_active":"2024-01-27T01:06:39.907896+0000","last_peered":"2024-01-27T01:06:39.907896+0000","last_clean":"2024-01-27T01:06:39.907896+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-27T01:06:39.907896+0000","last_undegraded":"2024-01-27T01:06:39.907896+0000","last_fullsized":"2024-01-27T01:06:39.907896+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-01-27T01:06:28.666262+0000","last_change":"2024-01-27T01:06:26.873715+0000","last_active":"2024-01-27T01:06:28.666262+0000","last_peered":"2024-01-27T01:06:28.666262+0000","last_clean":"2024-01-27T01:06:28.666262+0000","last_became_active":"2024-01-27T01:06:26.873430+0000","last_became_peered":"2024-01-27T01:06:26.873430+0000","last_unstale":"2024-01-27T01:06:28.666262+0000","last_undegraded":"2024-01-27T01:06:28.666262+0000","last_fullsized":"2024-01-27T01:06:28.666262+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:44.342963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3538,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":630,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8750,"ondisk_log_size":8750,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576004,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300632,"kb_used_data":3344,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071208,"statfs":{"total":96636764160,"available":96328916992,"internally_reserved":0,"allocated":3424256,"data_stored":2074391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63200000000000001,"15min":0.61399999999999999},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.312},"max":{"1min":0.82099999999999995,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.63700000000000001,"15min":0.623},"min":{"1min":0.498,"5min":0.39900000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":0.879,"15min":0.879},"last":0.55200000000000005}]},{"osd":1,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.623,"15min":0.61599999999999999},"min":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65600000000000003,"15min":0.63700000000000001},"min":{"1min":0.505,"5min":0.42099999999999999,"15min":0.34300000000000003},"max":{"1min":0.76800000000000002,"5min":0.84099999999999997,"15min":0.84399999999999997},"last":0.60599999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63400000000000001,"15min":0.61399999999999999},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82099999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.628},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64500000000000002,"15min":0.63600000000000001},"min":{"1min":0.49399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79400000000000004,"5min":0.872,"15min":0.88900000000000001},"last":0.59499999999999997}]},{"osd":3,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.628,"15min":0.62},"min":{"1min":0.52100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79900000000000004,"5min":0.876,"15min":0.876},"last":0.504},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66800000000000004,"15min":0.64700000000000002},"min":{"1min":0.53200000000000003,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84599999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.69099999999999995}]},{"osd":4,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.50900000000000001,"15min":0.48999999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.29999999999999999},"max":{"1min":0.73399999999999999,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.51900000000000002},"min":{"1min":0.35699999999999998,"5min":0.34699999999999998,"15min":0.32200000000000001},"max":{"1min":0.77500000000000002,"5min":0.80300000000000005,"15min":0.86599999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55800000000000005,"15min":0.52600000000000002},"min":{"1min":0.40999999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.753,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57999999999999996,"15min":0.56499999999999995},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.33700000000000002},"max":{"1min":0.76700000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.66300000000000003}]},{"osd":6,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60999999999999999,"15min":0.58899999999999997},"min":{"1min":0.42499999999999999,"5min":0.42199999999999999,"15min":0.37},"max":{"1min":0.81100000000000005,"5min":0.83999999999999997,"15min":0.84799999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61299999999999999,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.39100000000000001,"15min":0.35699999999999998},"max":{"1min":0.76100000000000001,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.68000000000000005}]}]},{"osd":6,"up_from":13,"seq":55834576004,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308244,"kb_used_data":2764,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063596,"statfs":{"total":96636764160,"available":96321122304,"internally_reserved":0,"allocated":2830336,"data_stored":1484507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61099999999999999,"15min":0.60799999999999998},"min":{"1min":0.41999999999999998,"5min":0.40799999999999997,"15min":0.36399999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.496},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.441,"5min":0.432,"15min":0.376},"max":{"1min":0.82499999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.67600000000000005}]},{"osd":1,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79000000000000004,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.40799999999999997,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.628}]},{"osd":2,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.39700000000000002,"15min":0.377},"max":{"1min":0.80400000000000005,"5min":0.878,"15min":0.878},"last":0.65500000000000003}]},{"osd":3,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.622,"15min":0.627},"min":{"1min":0.5,"5min":0.378,"15min":0.378},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.90100000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66300000000000003,"15min":0.66000000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.59899999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.36299999999999999,"5min":0.33300000000000002,"15min":0.32500000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.86599999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33900000000000002,"5min":0.33800000000000002,"15min":0.29799999999999999},"max":{"1min":0.77300000000000002,"5min":0.78500000000000003,"15min":0.88900000000000001},"last":0.54700000000000004}]},{"osd":5,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.31900000000000001},"max":{"1min":0.68999999999999995,"5min":0.75900000000000001,"15min":0.80500000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.42199999999999999,"5min":0.37,"15min":0.28899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.874},"last":0.58499999999999996}]},{"osd":7,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45800000000000002,"5min":0.39000000000000001,"15min":0.35799999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.92900000000000005},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.44500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78900000000000003,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.61399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834576004,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300024,"kb_used_data":2736,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071816,"statfs":{"total":96636764160,"available":96329539584,"internally_reserved":0,"allocated":2801664,"data_stored":1473710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.41099999999999998,"15min":0.42099999999999999},"min":{"1min":0.26400000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.64400000000000002,"5min":0.66100000000000003,"15min":0.69599999999999995},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44500000000000001,"15min":0.45500000000000002},"min":{"1min":0.34000000000000002,"5min":0.28399999999999997,"15min":0.26800000000000002},"max":{"1min":0.57999999999999996,"5min":0.73999999999999999,"15min":0.76600000000000001},"last":0.35699999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.439,"15min":0.438},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.26200000000000001},"max":{"1min":0.748,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.47599999999999998,"15min":0.48999999999999999},"min":{"1min":0.379,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.67400000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.5}]},{"osd":3,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52300000000000002,"15min":0.53300000000000003},"min":{"1min":0.39300000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73699999999999999,"5min":0.84799999999999998,"15min":1.2130000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.51400000000000001},"min":{"1min":0.41699999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68200000000000005,"5min":1.125,"15min":1.22},"last":0.42099999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.67200000000000004,"15min":0.67400000000000004},"min":{"1min":0.42099999999999999,"5min":0.41099999999999998,"15min":0.39800000000000002},"max":{"1min":0.82699999999999996,"5min":1.242,"15min":1.242},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69099999999999995,"15min":0.69599999999999995},"min":{"1min":0.53700000000000003,"5min":0.496,"15min":0.35699999999999998},"max":{"1min":0.90600000000000003,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.45900000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73499999999999999,"15min":0.70899999999999996},"min":{"1min":0.53300000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.876,"5min":5.444,"15min":5.444},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.71699999999999997,"15min":0.70799999999999996},"min":{"1min":0.502,"5min":0.47599999999999998,"15min":0.435},"max":{"1min":0.97799999999999998,"5min":2.6669999999999998,"15min":2.6669999999999998},"last":0.47599999999999998}]},{"osd":6,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.74199999999999999,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":0.93899999999999995,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.81299999999999994,"15min":0.745},"min":{"1min":0.48799999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.02,"5min":11.223000000000001,"15min":11.223000000000001},"last":0.52400000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.752,"15min":0.75600000000000001},"min":{"1min":0.58999999999999997,"5min":0.52800000000000002,"15min":0.47599999999999998},"max":{"1min":0.96299999999999997,"5min":2.9660000000000002,"15min":2.9660000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.77700000000000002,"15min":0.76800000000000002},"min":{"1min":0.58799999999999997,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.94499999999999995,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.53400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834576003,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317004,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94054836,"statfs":{"total":96636764160,"available":96312152064,"internally_reserved":0,"allocated":3411968,"data_stored":2071643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42899999999999999,"15min":0.42199999999999999},"min":{"1min":0.30299999999999999,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":0.55600000000000005,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.436,"5min":0.45600000000000002,"15min":0.46100000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":0.56499999999999995,"5min":0.72699999999999998,"15min":0.73399999999999999},"last":0.26700000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.29699999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.58299999999999996,"5min":0.66900000000000004,"15min":0.76800000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48399999999999999,"15min":0.48899999999999999},"min":{"1min":0.38600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64600000000000002,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.45400000000000001}]},{"osd":3,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.66100000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.504},"min":{"1min":0.36699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64100000000000001,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.36699999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.68300000000000005,"15min":0.67700000000000005},"min":{"1min":0.57799999999999996,"5min":0.47999999999999998,"15min":0.42199999999999999},"max":{"1min":0.86399999999999999,"5min":0.91500000000000004,"15min":1.0720000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.56999999999999995,"5min":0.47099999999999997,"15min":0.437},"max":{"1min":0.82699999999999996,"5min":1.002,"15min":1.002},"last":0.66200000000000003}]},{"osd":5,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.68999999999999995,"15min":0.70099999999999996},"min":{"1min":0.58199999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.876,"5min":1.0209999999999999,"15min":2.165},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70599999999999996,"15min":0.71199999999999997},"min":{"1min":0.498,"5min":0.48799999999999999,"15min":0.45600000000000002},"max":{"1min":0.89000000000000001,"5min":0.97399999999999998,"15min":1.1399999999999999},"last":0.64700000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69299999999999995,"15min":0.71099999999999997},"min":{"1min":0.56499999999999995,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.84599999999999997,"5min":0.98899999999999999,"15min":1.0860000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72599999999999998,"15min":0.74099999999999999},"min":{"1min":0.54300000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.93400000000000005,"5min":1.0109999999999999,"15min":1.425},"last":0.73299999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.74399999999999999,"15min":0.75700000000000001},"min":{"1min":0.61299999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.98899999999999999,"5min":1.044,"15min":1.1259999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.76300000000000001,"15min":0.76900000000000002},"min":{"1min":0.58299999999999996,"5min":0.51100000000000001,"15min":0.48499999999999999},"max":{"1min":0.91700000000000004,"5min":1.0289999999999999,"15min":1.1910000000000001},"last":0.76600000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576003,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312328,"kb_used_data":2752,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059512,"statfs":{"total":96636764160,"available":96316940288,"internally_reserved":0,"allocated":2818048,"data_stored":1481275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40500000000000003,"15min":0.40799999999999997},"min":{"1min":0.26600000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.46000000000000002,"5min":0.746,"15min":0.746},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.44500000000000001,"15min":0.44400000000000001},"min":{"1min":0.32300000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.59099999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.436}]},{"osd":1,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.435,"15min":0.44500000000000001},"min":{"1min":0.32000000000000001,"5min":0.29099999999999998,"15min":0.27000000000000002},"max":{"1min":0.61299999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.439,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.55600000000000005,"5min":0.73599999999999999,"15min":0.78100000000000003},"last":0.54800000000000004}]},{"osd":3,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.498,"15min":0.497},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.33200000000000002},"max":{"1min":0.627,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.497},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48699999999999999,"15min":0.49199999999999999},"min":{"1min":0.373,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":0.59099999999999997,"5min":0.66600000000000004,"15min":0.75700000000000001},"last":0.52000000000000002}]},{"osd":4,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68200000000000005,"15min":0.68000000000000005},"min":{"1min":0.52800000000000002,"5min":0.495,"15min":0.442},"max":{"1min":0.92700000000000005,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68999999999999995,"15min":0.69399999999999995},"min":{"1min":0.50900000000000001,"5min":0.47099999999999997,"15min":0.378},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":1.107},"last":0.77200000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.69099999999999995,"15min":0.70299999999999996},"min":{"1min":0.52400000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.89500000000000002,"15min":1.167},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70899999999999996,"15min":0.70999999999999996},"min":{"1min":0.47499999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89600000000000002,"5min":0.97499999999999998,"15min":1.05},"last":0.78600000000000003}]},{"osd":6,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71499999999999997,"15min":0.71199999999999997},"min":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.45500000000000002},"max":{"1min":0.81399999999999995,"5min":0.996,"15min":1.077},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70499999999999996,"15min":0.71999999999999997},"min":{"1min":0.497,"5min":0.497,"15min":0.39200000000000002},"max":{"1min":0.876,"5min":0.96699999999999997,"15min":1.135},"last":0.73499999999999999}]},{"osd":7,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.749,"15min":0.75},"min":{"1min":0.56000000000000005,"5min":0.50800000000000001,"15min":0.47099999999999997},"max":{"1min":0.94099999999999995,"5min":0.97399999999999998,"15min":1.198},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.76500000000000001,"15min":0.76900000000000002},"min":{"1min":0.55000000000000004,"5min":0.53100000000000003,"15min":0.48199999999999998},"max":{"1min":0.94599999999999995,"5min":1.028,"15min":1.0880000000000001},"last":0.752}]}]},{"osd":3,"up_from":13,"seq":55834576002,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304144,"kb_used_data":2760,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067696,"statfs":{"total":96636764160,"available":96325320704,"internally_reserved":0,"allocated":2826240,"data_stored":1481311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.44700000000000001,"15min":0.42299999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.54800000000000004,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.309},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.372,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.63500000000000001,"5min":0.66700000000000004,"15min":0.84699999999999998},"last":0.34300000000000003}]},{"osd":1,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.63100000000000001,"15min":0.53500000000000003},"min":{"1min":0.34499999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.629,"5min":15.391,"15min":15.391},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.54600000000000004,"15min":0.52400000000000002},"min":{"1min":0.36699999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65100000000000002,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.42999999999999999}]},{"osd":2,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54600000000000004,"15min":0.51700000000000002},"min":{"1min":0.36499999999999999,"5min":0.36399999999999999,"15min":0.34599999999999997},"max":{"1min":0.62,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.53100000000000003,"15min":0.50700000000000001},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.60699999999999998,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":0.39600000000000002}]},{"osd":4,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.74199999999999999,"15min":0.72099999999999997},"min":{"1min":0.497,"5min":0.47699999999999998,"15min":0.439},"max":{"1min":0.88600000000000001,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.71999999999999997,"15min":0.70799999999999996},"min":{"1min":0.46300000000000002,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":0.94399999999999995,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.52000000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.74299999999999999,"15min":0.71699999999999997},"min":{"1min":0.51100000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.83999999999999997,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72799999999999998,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.97699999999999998,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.57399999999999995}]},{"osd":6,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.754,"15min":0.72599999999999998},"min":{"1min":0.53500000000000003,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.81200000000000006,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.748,"15min":0.74299999999999999},"min":{"1min":0.56100000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.92700000000000005,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.60799999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.75600000000000001,"15min":0.73799999999999999},"min":{"1min":0.58599999999999997,"5min":0.52500000000000002,"15min":0.46700000000000003},"max":{"1min":1.0029999999999999,"5min":4.569,"15min":4.569},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.80600000000000005,"15min":0.78500000000000003},"min":{"1min":0.55700000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0169999999999999,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.59199999999999997}]}]},{"osd":4,"up_from":13,"seq":55834576004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312328,"kb_used_data":2752,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059512,"statfs":{"total":96636764160,"available":96316940288,"internally_reserved":0,"allocated":2818048,"data_stored":1481739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.46899999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.84199999999999997},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.61499999999999999,"15min":0.61399999999999999},"min":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78700000000000003,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.753}]},{"osd":1,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61899999999999999,"15min":0.627},"min":{"1min":0.46100000000000002,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.76400000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61499999999999999,"15min":0.62},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.79800000000000004,"5min":0.874,"15min":0.874},"last":0.69699999999999995}]},{"osd":2,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.59999999999999998},"min":{"1min":0.42399999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.88700000000000001},"last":0.628},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.51100000000000001,"5min":0.42899999999999999,"15min":0.42799999999999999},"max":{"1min":0.81100000000000005,"5min":0.81499999999999995,"15min":0.89200000000000002},"last":0.79200000000000004}]},{"osd":3,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.61599999999999999,"15min":0.623},"min":{"1min":0.38800000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.83299999999999996,"5min":0.83599999999999997,"15min":0.84199999999999997},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.627},"min":{"1min":0.41299999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.88300000000000001},"last":0.72999999999999998}]},{"osd":5,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.75800000000000001,"15min":0.85899999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.502},"min":{"1min":0.376,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.878},"last":0.74099999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34999999999999998},"max":{"1min":0.63200000000000001,"5min":0.70099999999999996,"15min":0.88700000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.70699999999999996,"5min":0.79200000000000004,"15min":0.81499999999999995},"last":0.64800000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.42399999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72599999999999998,"5min":0.80400000000000005,"15min":0.90200000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59499999999999997,"15min":0.59399999999999997},"min":{"1min":0.41599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.73799999999999999,"5min":0.78700000000000003,"15min":0.92600000000000005},"last":0.76700000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576004,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312332,"kb_used_data":2756,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059508,"statfs":{"total":96636764160,"available":96316936192,"internally_reserved":0,"allocated":2822144,"data_stored":1481241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59399999999999997,"15min":0.60299999999999998},"min":{"1min":0.42999999999999999,"5min":0.40500000000000003,"15min":0.27300000000000002},"max":{"1min":0.80700000000000005,"5min":0.82799999999999996,"15min":1.006},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59399999999999997,"15min":0.60699999999999998},"min":{"1min":0.36599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":1.1719999999999999},"last":0.73499999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63700000000000001,"15min":0.65200000000000002},"min":{"1min":0.39100000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.95899999999999996},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.52100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":1.212},"last":0.77900000000000003}]},{"osd":2,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57499999999999996,"15min":0.59399999999999997},"min":{"1min":0.42299999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":1.631},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.623,"5min":0.625,"15min":0.621},"min":{"1min":0.497,"5min":0.38800000000000001,"15min":0.34000000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.60499999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.59999999999999998},"min":{"1min":0.39600000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83299999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.625,"15min":0.63},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.879,"5min":0.879,"15min":1.4690000000000001},"last":0.754}]},{"osd":4,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.505,"15min":0.48799999999999999},"min":{"1min":0.34899999999999998,"5min":0.32200000000000001,"15min":0.317},"max":{"1min":0.76700000000000002,"5min":0.90200000000000002,"15min":0.94399999999999995},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53500000000000003,"15min":0.51500000000000001},"min":{"1min":0.439,"5min":0.35199999999999998,"15min":0.33900000000000002},"max":{"1min":0.80800000000000005,"5min":0.92500000000000004,"15min":0.93300000000000005},"last":0.58299999999999996}]},{"osd":6,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51800000000000002,"15min":0.504},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79400000000000004,"5min":0.96799999999999997,"15min":1.732},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56299999999999994,"15min":0.54500000000000004},"min":{"1min":0.42899999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.753,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.53800000000000003}]},{"osd":7,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.46200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80400000000000005,"5min":0.95399999999999996,"15min":1.0740000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.626,"15min":0.61299999999999999},"min":{"1min":0.504,"5min":0.40799999999999997,"15min":0.39100000000000001},"max":{"1min":0.85199999999999998,"5min":0.996,"15min":1.6919999999999999},"last":0.61599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:06:46.030 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1c is not active+clean 2024-01-27T01:06:46.031 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.916373+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:55:49.427110+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-27T01:06:46.031 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1f is not active+clean 2024-01-27T01:06:46.031 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.031 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.6 is not active+clean 2024-01-27T01:06:46.031 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.910860+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T03:36:16.613510+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-27T01:06:46.031 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.5 is not active+clean 2024-01-27T01:06:46.031 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.031 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.7 is not active+clean 2024-01-27T01:06:46.031 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.913817+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T01:28:22.022827+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-27T01:06:46.032 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.4 is not active+clean 2024-01-27T01:06:46.032 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.032 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.4 is not active+clean 2024-01-27T01:06:46.032 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.914162+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T03:57:07.250360+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-27T01:06:46.032 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.7 is not active+clean 2024-01-27T01:06:46.032 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.032 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.5 is not active+clean 2024-01-27T01:06:46.032 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.911198+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T03:26:34.409044+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-27T01:06:46.032 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.6 is not active+clean 2024-01-27T01:06:46.033 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.033 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.2 is not active+clean 2024-01-27T01:06:46.033 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.913094+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T03:08:04.706593+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-27T01:06:46.033 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1 is not active+clean 2024-01-27T01:06:46.033 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.033 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.3 is not active+clean 2024-01-27T01:06:46.033 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.915265+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:55:49.427110+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-27T01:06:46.033 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.0 is not active+clean 2024-01-27T01:06:46.033 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.033 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.0 is not active+clean 2024-01-27T01:06:46.034 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.914590+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T08:38:04.634853+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-27T01:06:46.034 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3 is not active+clean 2024-01-27T01:06:46.034 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.034 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1 is not active+clean 2024-01-27T01:06:46.034 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.916305+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T05:54:24.566246+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-27T01:06:46.034 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2 is not active+clean 2024-01-27T01:06:46.034 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.034 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.e is not active+clean 2024-01-27T01:06:46.034 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.916585+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T05:37:20.044223+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-27T01:06:46.034 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.d is not active+clean 2024-01-27T01:06:46.034 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.035 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.f is not active+clean 2024-01-27T01:06:46.035 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.910624+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T01:47:30.145789+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-27T01:06:46.035 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.c is not active+clean 2024-01-27T01:06:46.035 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.035 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.c is not active+clean 2024-01-27T01:06:46.035 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.913526+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T11:07:02.246428+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-27T01:06:46.035 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.f is not active+clean 2024-01-27T01:06:46.035 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.035 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.d is not active+clean 2024-01-27T01:06:46.035 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.914309+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T12:31:18.366038+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-27T01:06:46.035 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.e is not active+clean 2024-01-27T01:06:46.035 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.036 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.a is not active+clean 2024-01-27T01:06:46.036 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.911280+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:43:07.431223+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-27T01:06:46.036 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.9 is not active+clean 2024-01-27T01:06:46.036 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.036 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.b is not active+clean 2024-01-27T01:06:46.036 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.913563+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T03:49:22.695065+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-27T01:06:46.036 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.8 is not active+clean 2024-01-27T01:06:46.036 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.036 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.8 is not active+clean 2024-01-27T01:06:46.036 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.916563+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T04:01:36.332606+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-27T01:06:46.037 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.b is not active+clean 2024-01-27T01:06:46.037 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.037 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.9 is not active+clean 2024-01-27T01:06:46.037 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.911685+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:21:31.265591+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-27T01:06:46.037 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.a is not active+clean 2024-01-27T01:06:46.037 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.037 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.16 is not active+clean 2024-01-27T01:06:46.037 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.913156+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T12:56:56.697365+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-27T01:06:46.037 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.15 is not active+clean 2024-01-27T01:06:46.037 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.037 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.17 is not active+clean 2024-01-27T01:06:46.037 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.917128+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T08:57:15.231912+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-27T01:06:46.038 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.14 is not active+clean 2024-01-27T01:06:46.038 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.038 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.14 is not active+clean 2024-01-27T01:06:46.038 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.910329+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T09:19:00.298602+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-27T01:06:46.038 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.17 is not active+clean 2024-01-27T01:06:46.038 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.038 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.15 is not active+clean 2024-01-27T01:06:46.038 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.916917+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T03:36:16.613510+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-27T01:06:46.038 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.16 is not active+clean 2024-01-27T01:06:46.038 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.039 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.12 is not active+clean 2024-01-27T01:06:46.039 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.914165+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T10:27:24.223960+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-27T01:06:46.039 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.11 is not active+clean 2024-01-27T01:06:46.039 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.039 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.13 is not active+clean 2024-01-27T01:06:46.039 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.915793+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T03:36:16.613510+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-27T01:06:46.039 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.10 is not active+clean 2024-01-27T01:06:46.039 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.039 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.10 is not active+clean 2024-01-27T01:06:46.039 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.915065+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T05:54:24.566246+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-27T01:06:46.039 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.13 is not active+clean 2024-01-27T01:06:46.039 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.040 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.11 is not active+clean 2024-01-27T01:06:46.040 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.910328+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:47:24.032004+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-27T01:06:46.040 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.12 is not active+clean 2024-01-27T01:06:46.040 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.040 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1e is not active+clean 2024-01-27T01:06:46.040 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.913764+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T07:10:10.314575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-27T01:06:46.040 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1d is not active+clean 2024-01-27T01:06:46.040 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.040 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1f is not active+clean 2024-01-27T01:06:46.040 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.916960+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T02:25:57.409497+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-27T01:06:46.040 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1c is not active+clean 2024-01-27T01:06:46.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.041 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1d is not active+clean 2024-01-27T01:06:46.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.910329+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T07:27:27.720315+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-27T01:06:46.041 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1e is not active+clean 2024-01-27T01:06:46.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.041 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1a is not active+clean 2024-01-27T01:06:46.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.916917+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T06:21:31.265591+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-27T01:06:46.041 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.19 is not active+clean 2024-01-27T01:06:46.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.041 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1b is not active+clean 2024-01-27T01:06:46.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.910981+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T08:57:15.231912+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-27T01:06:46.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.18 is not active+clean 2024-01-27T01:06:46.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.18 is not active+clean 2024-01-27T01:06:46.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.910594+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T11:31:50.703251+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-27T01:06:46.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1b is not active+clean 2024-01-27T01:06:46.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.19 is not active+clean 2024-01-27T01:06:46.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2024-01-27T01:06:38.901811+0000', 'last_change': '2024-01-27T01:06:38.913160+0000', 'last_active': '2024-01-27T01:06:38.901811+0000', 'last_peered': '2024-01-27T01:06:38.901811+0000', 'last_clean': '2024-01-27T01:06:38.901811+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:38.901811+0000', 'last_undegraded': '2024-01-27T01:06:38.901811+0000', 'last_fullsized': '2024-01-27T01:06:38.901811+0000', 'mapping_epoch': 224, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 224, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:38.901811+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-28T01:28:15.067443+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-27T01:06:46.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1a is not active+clean 2024-01-27T01:06:46.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.22 is not active+clean 2024-01-27T01:06:46.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.23 is not active+clean 2024-01-27T01:06:46.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.20 is not active+clean 2024-01-27T01:06:46.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.21 is not active+clean 2024-01-27T01:06:46.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.26 is not active+clean 2024-01-27T01:06:46.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.27 is not active+clean 2024-01-27T01:06:46.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.044 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.24 is not active+clean 2024-01-27T01:06:46.044 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.044 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.25 is not active+clean 2024-01-27T01:06:46.044 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.044 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2a is not active+clean 2024-01-27T01:06:46.044 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.048 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2b is not active+clean 2024-01-27T01:06:46.048 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.048 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.28 is not active+clean 2024-01-27T01:06:46.048 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.048 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.29 is not active+clean 2024-01-27T01:06:46.048 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.048 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2e is not active+clean 2024-01-27T01:06:46.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2f is not active+clean 2024-01-27T01:06:46.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2c is not active+clean 2024-01-27T01:06:46.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2d is not active+clean 2024-01-27T01:06:46.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.32 is not active+clean 2024-01-27T01:06:46.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.33 is not active+clean 2024-01-27T01:06:46.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.30 is not active+clean 2024-01-27T01:06:46.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.31 is not active+clean 2024-01-27T01:06:46.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.36 is not active+clean 2024-01-27T01:06:46.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.37 is not active+clean 2024-01-27T01:06:46.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.34 is not active+clean 2024-01-27T01:06:46.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.35 is not active+clean 2024-01-27T01:06:46.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3a is not active+clean 2024-01-27T01:06:46.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3b is not active+clean 2024-01-27T01:06:46.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.38 is not active+clean 2024-01-27T01:06:46.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.39 is not active+clean 2024-01-27T01:06:46.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3e is not active+clean 2024-01-27T01:06:46.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3f is not active+clean 2024-01-27T01:06:46.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3c is not active+clean 2024-01-27T01:06:46.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3d is not active+clean 2024-01-27T01:06:46.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-27T01:06:39.907896+0000', 'last_change': '2024-01-27T01:06:39.907896+0000', 'last_active': '2024-01-27T01:06:39.907896+0000', 'last_peered': '2024-01-27T01:06:39.907896+0000', 'last_clean': '2024-01-27T01:06:39.907896+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-27T01:06:39.907896+0000', 'last_undegraded': '2024-01-27T01:06:39.907896+0000', 'last_fullsized': '2024-01-27T01:06:39.907896+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'last_clean_scrub_stamp': '2024-01-27T01:06:39.907896+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-27T01:06:46.053 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-27T01:06:46.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.190+0000 7f667e7dd700 1 Processor -- start 2024-01-27T01:06:46.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.190+0000 7f667e7dd700 1 -- start start 2024-01-27T01:06:46.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.191+0000 7f667e7dd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f66780c3cf0 0x7f66780ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:46.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.191+0000 7f667e7dd700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66780cbf10 con 0x7f66780cd080 2024-01-27T01:06:46.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.191+0000 7f667e7dd700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66780cd450 con 0x7f66780acf30 2024-01-27T01:06:46.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.191+0000 7f667e7dd700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66780ada10 con 0x7f66780c3cf0 2024-01-27T01:06:46.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.192+0000 7f667cd7a700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f66780cd080 0x7f66780ab1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45456/0 (socket says 172.21.15.46:45456) 2024-01-27T01:06:46.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.192+0000 7f667cd7a700 1 -- 172.21.15.46:0/1755179455 learned_addr learned my addr 172.21.15.46:0/1755179455 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:46.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.192+0000 7f66777fe700 1 --2- 172.21.15.46:0/1755179455 >> v2:172.21.15.106:3300/0 conn(0x7f66780c3cf0 0x7f66780ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:46.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.192+0000 7f6676ffd700 1 -- 172.21.15.46:0/1755179455 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686608345 0 0) 0x7f66780cbf10 con 0x7f66780cd080 2024-01-27T01:06:46.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.192+0000 7f6676ffd700 1 -- 172.21.15.46:0/1755179455 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6660003570 con 0x7f66780cd080 2024-01-27T01:06:46.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.192+0000 7f66777fe700 1 -- 172.21.15.46:0/1755179455 >> v1:172.21.15.106:6789/0 conn(0x7f66780acf30 legacy=0x7f66780ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:46.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.192+0000 7f66777fe700 1 -- 172.21.15.46:0/1755179455 >> v1:172.21.15.46:6789/0 conn(0x7f66780cd080 legacy=0x7f66780ab1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:46.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.192+0000 7f66777fe700 1 -- 172.21.15.46:0/1755179455 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66780c4460 con 0x7f66780c3cf0 2024-01-27T01:06:46.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.192+0000 7f66777fe700 1 --2- 172.21.15.46:0/1755179455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66780c3cf0 0x7f66780ccb40 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f666c0098e0 tx=0x7f666c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=befc25743c104245 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:46.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.192+0000 7f6676ffd700 1 -- 172.21.15.46:0/1755179455 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c00b640 con 0x7f66780c3cf0 2024-01-27T01:06:46.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.193+0000 7f6676ffd700 1 -- 172.21.15.46:0/1755179455 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666c00b7a0 con 0x7f66780c3cf0 2024-01-27T01:06:46.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.193+0000 7f6676ffd700 1 -- 172.21.15.46:0/1755179455 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c00bb40 con 0x7f66780c3cf0 2024-01-27T01:06:46.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.193+0000 7f667e7dd700 1 -- 172.21.15.46:0/1755179455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66780c3cf0 msgr2=0x7f66780ccb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:46.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.193+0000 7f667e7dd700 1 --2- 172.21.15.46:0/1755179455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66780c3cf0 0x7f66780ccb40 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f666c0098e0 tx=0x7f666c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.193+0000 7f667e7dd700 1 -- 172.21.15.46:0/1755179455 shutdown_connections 2024-01-27T01:06:46.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.193+0000 7f667e7dd700 1 --2- 172.21.15.46:0/1755179455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66780c3cf0 0x7f66780ccb40 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.193+0000 7f667e7dd700 1 -- 172.21.15.46:0/1755179455 >> 172.21.15.46:0/1755179455 conn(0x7f6678097820 msgr2=0x7f6678068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.193+0000 7f667e7dd700 1 -- 172.21.15.46:0/1755179455 shutdown_connections 2024-01-27T01:06:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.193+0000 7f667e7dd700 1 -- 172.21.15.46:0/1755179455 wait complete. 2024-01-27T01:06:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.195+0000 7f667e7dd700 1 Processor -- start 2024-01-27T01:06:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.195+0000 7f667e7dd700 1 -- start start 2024-01-27T01:06:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.196+0000 7f667e7dd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66780acf30 0x7f6678140310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.196+0000 7f667e7dd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66780c3cf0 0x7f66780b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.196+0000 7f667e7dd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66780cd080 0x7f667816b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.196+0000 7f667e7dd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66780c99e0 con 0x7f66780c3cf0 2024-01-27T01:06:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.196+0000 7f667e7dd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66780c93b0 con 0x7f66780cd080 2024-01-27T01:06:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.196+0000 7f667e7dd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66780b3d50 con 0x7f66780acf30 2024-01-27T01:06:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.196+0000 7f66777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66780c3cf0 0x7f66780b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.196+0000 7f66777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66780c3cf0 0x7f66780b3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41854/0 (socket says 172.21.15.46:41854) 2024-01-27T01:06:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.196+0000 7f66777fe700 1 -- 172.21.15.46:0/2253407436 learned_addr learned my addr 172.21.15.46:0/2253407436 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.196+0000 7f667cd7a700 1 --2- 172.21.15.46:0/2253407436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66780cd080 0x7f667816b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.196+0000 7f6677fff700 1 --2- 172.21.15.46:0/2253407436 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66780acf30 0x7f6678140310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.197+0000 7f66777fe700 1 -- 172.21.15.46:0/2253407436 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66780acf30 msgr2=0x7f6678140310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.197+0000 7f66777fe700 1 --2- 172.21.15.46:0/2253407436 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66780acf30 0x7f6678140310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.197+0000 7f66777fe700 1 -- 172.21.15.46:0/2253407436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66780cd080 msgr2=0x7f667816b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.197+0000 7f66777fe700 1 --2- 172.21.15.46:0/2253407436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66780cd080 0x7f667816b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.197+0000 7f66777fe700 1 -- 172.21.15.46:0/2253407436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f666c009420 con 0x7f66780c3cf0 2024-01-27T01:06:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.197+0000 7f66777fe700 1 --2- 172.21.15.46:0/2253407436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66780c3cf0 0x7f66780b3460 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f666c0093f0 tx=0x7f666c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.197+0000 7f665ffff700 1 -- 172.21.15.46:0/2253407436 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c000d10 con 0x7f66780c3cf0 2024-01-27T01:06:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.197+0000 7f665ffff700 1 -- 172.21.15.46:0/2253407436 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666c00b910 con 0x7f66780c3cf0 2024-01-27T01:06:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.197+0000 7f665ffff700 1 -- 172.21.15.46:0/2253407436 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c025420 con 0x7f66780c3cf0 2024-01-27T01:06:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.197+0000 7f667e7dd700 1 -- 172.21.15.46:0/2253407436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66780b4030 con 0x7f66780c3cf0 2024-01-27T01:06:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.197+0000 7f667e7dd700 1 -- 172.21.15.46:0/2253407436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66780b39a0 con 0x7f66780c3cf0 2024-01-27T01:06:46.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.198+0000 7f667e7dd700 1 -- 172.21.15.46:0/2253407436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66780c9b50 con 0x7f66780c3cf0 2024-01-27T01:06:46.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.198+0000 7f665ffff700 1 -- 172.21.15.46:0/2253407436 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f666c023070 con 0x7f66780c3cf0 2024-01-27T01:06:46.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.199+0000 7f665ffff700 1 --2- 172.21.15.46:0/2253407436 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6664066f30 0x7f66640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:46.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.199+0000 7f665ffff700 1 -- 172.21.15.46:0/2253407436 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f666c085fd0 con 0x7f66780c3cf0 2024-01-27T01:06:46.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.199+0000 7f6677fff700 1 --2- 172.21.15.46:0/2253407436 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6664066f30 0x7f66640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:46.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.200+0000 7f6677fff700 1 --2- 172.21.15.46:0/2253407436 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6664066f30 0x7f66640693e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6668000c00 tx=0x7f6668007ea0 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:46.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.203+0000 7f665ffff700 1 -- 172.21.15.46:0/2253407436 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f666c052bb0 con 0x7f66780c3cf0 2024-01-27T01:06:46.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.494+0000 7f667e7dd700 1 -- 172.21.15.46:0/2253407436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f667804c710 con 0x7f66780c3cf0 2024-01-27T01:06:46.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.494+0000 7f665ffff700 1 -- 172.21.15.46:0/2253407436 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1288 (secure 0 0 0) 0x7f666c0561d0 con 0x7f66780c3cf0 2024-01-27T01:06:46.493 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:46.493 INFO:teuthology.orchestra.run.smithi046.stdout:{"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":5839,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":227,"num_osds":8,"num_up_osds":8,"osd_up_since":1706311777,"num_in_osds":8,"osd_in_since":1706311769,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97},{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32}],"num_pgs":193,"num_pools":5,"num_objects":24,"data_bytes":593906,"bytes_used":2526244864,"bytes_avail":770567868416,"bytes_total":773094113280,"unknown_pgs_ratio":0.33160620927810669,"inactive_pgs_ratio":0.16580310463905334,"write_bytes_sec":175,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":122,"id":13,"up":1,"in":1,"max":1,"id":14,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":14,"rank":0,"name":"c","status":"up:active","gid":10854},{"filesystem_id":13,"rank":0,"name":"a","status":"up:active","gid":10866}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-01-27T01:05:00.737161+0000","services":{}},"progress_events":{}} 2024-01-27T01:06:46.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.496+0000 7f667e7dd700 1 -- 172.21.15.46:0/2253407436 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6664066f30 msgr2=0x7f66640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:46.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.496+0000 7f667e7dd700 1 --2- 172.21.15.46:0/2253407436 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6664066f30 0x7f66640693e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6668000c00 tx=0x7f6668007ea0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.496+0000 7f667e7dd700 1 -- 172.21.15.46:0/2253407436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66780c3cf0 msgr2=0x7f66780b3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:46.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.496+0000 7f667e7dd700 1 --2- 172.21.15.46:0/2253407436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66780c3cf0 0x7f66780b3460 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f666c0093f0 tx=0x7f666c017000 comp rx=0 tx=0).stop 2024-01-27T01:06:46.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.496+0000 7f667e7dd700 1 -- 172.21.15.46:0/2253407436 shutdown_connections 2024-01-27T01:06:46.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.496+0000 7f667e7dd700 1 --2- 172.21.15.46:0/2253407436 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6664066f30 0x7f66640693e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.496+0000 7f667e7dd700 1 --2- 172.21.15.46:0/2253407436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66780cd080 0x7f667816b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.496+0000 7f667e7dd700 1 --2- 172.21.15.46:0/2253407436 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66780acf30 0x7f6678140310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.496+0000 7f667e7dd700 1 --2- 172.21.15.46:0/2253407436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66780c3cf0 0x7f66780b3460 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.496+0000 7f667e7dd700 1 -- 172.21.15.46:0/2253407436 >> 172.21.15.46:0/2253407436 conn(0x7f6678097820 msgr2=0x7f667806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:46.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.497+0000 7f667e7dd700 1 -- 172.21.15.46:0/2253407436 shutdown_connections 2024-01-27T01:06:46.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.497+0000 7f667e7dd700 1 -- 172.21.15.46:0/2253407436 wait complete. 2024-01-27T01:06:46.511 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-27T01:06:46.511 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:06:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.643+0000 7f7bd019b700 1 Processor -- start 2024-01-27T01:06:46.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.644+0000 7f7bd019b700 1 -- start start 2024-01-27T01:06:46.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.644+0000 7f7bd019b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7bc8159760 0x7f7bc80b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:46.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.644+0000 7f7bd019b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7bc80b12c0 0x7f7bc80a9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:46.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.644+0000 7f7bd019b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc806f940 con 0x7f7bc80b3060 2024-01-27T01:06:46.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.644+0000 7f7bd019b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bc806ed30 con 0x7f7bc8159760 2024-01-27T01:06:46.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.644+0000 7f7bd019b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bc806f010 con 0x7f7bc80b12c0 2024-01-27T01:06:46.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.645+0000 7f7bcdf37700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7bc8159760 0x7f7bc80b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:46.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.645+0000 7f7bcdf37700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7bc8159760 0x7f7bc80b2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41870/0 (socket says 172.21.15.46:41870) 2024-01-27T01:06:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.645+0000 7f7bcdf37700 1 -- 172.21.15.46:0/1127078336 learned_addr learned my addr 172.21.15.46:0/1127078336 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.645+0000 7f7bce738700 1 --2- 172.21.15.46:0/1127078336 >> v2:172.21.15.106:3300/0 conn(0x7f7bc80b12c0 0x7f7bc80a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.645+0000 7f7bcdf37700 1 -- 172.21.15.46:0/1127078336 >> v2:172.21.15.106:3300/0 conn(0x7f7bc80b12c0 msgr2=0x7f7bc80a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.645+0000 7f7bcdf37700 1 --2- 172.21.15.46:0/1127078336 >> v2:172.21.15.106:3300/0 conn(0x7f7bc80b12c0 0x7f7bc80a9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.645+0000 7f7bcdf37700 1 -- 172.21.15.46:0/1127078336 >> v1:172.21.15.106:6789/0 conn(0x7f7bc80b3060 legacy=0x7f7bc80b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.645+0000 7f7bcdf37700 1 -- 172.21.15.46:0/1127078336 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc806f2f0 con 0x7f7bc8159760 2024-01-27T01:06:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.645+0000 7f7bcdf37700 1 --2- 172.21.15.46:0/1127078336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc8159760 0x7f7bc80b2b20 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f7bc806f8b0 tx=0x7f7bc4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0fbfc139ad8a176 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.645+0000 7f7bccf35700 1 -- 172.21.15.46:0/1127078336 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc400b860 con 0x7f7bc8159760 2024-01-27T01:06:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.645+0000 7f7bccf35700 1 -- 172.21.15.46:0/1127078336 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc400b9c0 con 0x7f7bc8159760 2024-01-27T01:06:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.645+0000 7f7bccf35700 1 -- 172.21.15.46:0/1127078336 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc400bd60 con 0x7f7bc8159760 2024-01-27T01:06:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.646+0000 7f7bd019b700 1 -- 172.21.15.46:0/1127078336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc8159760 msgr2=0x7f7bc80b2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.646+0000 7f7bd019b700 1 --2- 172.21.15.46:0/1127078336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc8159760 0x7f7bc80b2b20 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f7bc806f8b0 tx=0x7f7bc4009250 comp rx=0 tx=0).stop 2024-01-27T01:06:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.646+0000 7f7bd019b700 1 -- 172.21.15.46:0/1127078336 shutdown_connections 2024-01-27T01:06:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.646+0000 7f7bd019b700 1 --2- 172.21.15.46:0/1127078336 >> v2:172.21.15.106:3300/0 conn(0x7f7bc80b12c0 0x7f7bc80a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.646+0000 7f7bd019b700 1 --2- 172.21.15.46:0/1127078336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc8159760 0x7f7bc80b2b20 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.646+0000 7f7bd019b700 1 -- 172.21.15.46:0/1127078336 >> 172.21.15.46:0/1127078336 conn(0x7f7bc8069510 msgr2=0x7f7bc80aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:46.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.646+0000 7f7bd019b700 1 -- 172.21.15.46:0/1127078336 shutdown_connections 2024-01-27T01:06:46.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.646+0000 7f7bd019b700 1 -- 172.21.15.46:0/1127078336 wait complete. 2024-01-27T01:06:46.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.648+0000 7f7bd019b700 1 Processor -- start 2024-01-27T01:06:46.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.648+0000 7f7bd019b700 1 -- start start 2024-01-27T01:06:46.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.649+0000 7f7bd019b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bc80b12c0 0x7f7bc816dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:46.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.649+0000 7f7bd019b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bc80b3060 0x7f7bc8142e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:46.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.649+0000 7f7bd019b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc8159760 0x7f7bc80ad5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:46.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.649+0000 7f7bd019b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc816a700 con 0x7f7bc8159760 2024-01-27T01:06:46.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.649+0000 7f7bd019b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc816a9e0 con 0x7f7bc80b12c0 2024-01-27T01:06:46.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.649+0000 7f7bd019b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc816d120 con 0x7f7bc80b3060 2024-01-27T01:06:46.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.649+0000 7f7bce738700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc8159760 0x7f7bc80ad5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.649+0000 7f7bce738700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc8159760 0x7f7bc80ad5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41884/0 (socket says 172.21.15.46:41884) 2024-01-27T01:06:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.649+0000 7f7bce738700 1 -- 172.21.15.46:0/2341084152 learned_addr learned my addr 172.21.15.46:0/2341084152 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.649+0000 7f7bcdf37700 1 --2- 172.21.15.46:0/2341084152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bc80b12c0 0x7f7bc816dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.649+0000 7f7bcd736700 1 --2- 172.21.15.46:0/2341084152 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bc80b3060 0x7f7bc8142e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.649+0000 7f7bce738700 1 -- 172.21.15.46:0/2341084152 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bc80b3060 msgr2=0x7f7bc8142e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.649+0000 7f7bce738700 1 --2- 172.21.15.46:0/2341084152 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bc80b3060 0x7f7bc8142e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.649+0000 7f7bce738700 1 -- 172.21.15.46:0/2341084152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bc80b12c0 msgr2=0x7f7bc816dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.649+0000 7f7bce738700 1 --2- 172.21.15.46:0/2341084152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bc80b12c0 0x7f7bc816dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.649+0000 7f7bce738700 1 -- 172.21.15.46:0/2341084152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc40094f0 con 0x7f7bc8159760 2024-01-27T01:06:46.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.649+0000 7f7bce738700 1 --2- 172.21.15.46:0/2341084152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc8159760 0x7f7bc80ad5f0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0007f30 tx=0x7f7bc00098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:46.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.650+0000 7f7bbdffb700 1 -- 172.21.15.46:0/2341084152 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc000ecf0 con 0x7f7bc8159760 2024-01-27T01:06:46.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.650+0000 7f7bbdffb700 1 -- 172.21.15.46:0/2341084152 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc000ee50 con 0x7f7bc8159760 2024-01-27T01:06:46.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.650+0000 7f7bd019b700 1 -- 172.21.15.46:0/2341084152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc816d400 con 0x7f7bc8159760 2024-01-27T01:06:46.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.650+0000 7f7bbdffb700 1 -- 172.21.15.46:0/2341084152 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc0014670 con 0x7f7bc8159760 2024-01-27T01:06:46.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.650+0000 7f7bd019b700 1 -- 172.21.15.46:0/2341084152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc8150450 con 0x7f7bc8159760 2024-01-27T01:06:46.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.651+0000 7f7bd019b700 1 -- 172.21.15.46:0/2341084152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc804c710 con 0x7f7bc8159760 2024-01-27T01:06:46.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.651+0000 7f7bbdffb700 1 -- 172.21.15.46:0/2341084152 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7bc00149e0 con 0x7f7bc8159760 2024-01-27T01:06:46.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.652+0000 7f7bbdffb700 1 --2- 172.21.15.46:0/2341084152 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bac066f30 0x7f7bac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:46.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.652+0000 7f7bbdffb700 1 -- 172.21.15.46:0/2341084152 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f7bc0085390 con 0x7f7bc8159760 2024-01-27T01:06:46.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.652+0000 7f7bcdf37700 1 --2- 172.21.15.46:0/2341084152 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bac066f30 0x7f7bac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:46.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.653+0000 7f7bcdf37700 1 --2- 172.21.15.46:0/2341084152 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bac066f30 0x7f7bac0693e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7bc806f8b0 tx=0x7f7bc4017000 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:46.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.656+0000 7f7bbdffb700 1 -- 172.21.15.46:0/2341084152 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7bc0055a00 con 0x7f7bc8159760 2024-01-27T01:06:46.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.855+0000 7f7bd019b700 1 -- 172.21.15.46:0/2341084152 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7bc816a840 con 0x7f7bac066f30 2024-01-27T01:06:46.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.870+0000 7f7bbdffb700 1 -- 172.21.15.46:0/2341084152 <== mgr.9101 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528848 (secure 0 0 0) 0x7f7bc816a840 con 0x7f7bac066f30 2024-01-27T01:06:46.876 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:46.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.874+0000 7f7bd019b700 1 -- 172.21.15.46:0/2341084152 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bac066f30 msgr2=0x7f7bac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:46.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.874+0000 7f7bd019b700 1 --2- 172.21.15.46:0/2341084152 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bac066f30 0x7f7bac0693e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7bc806f8b0 tx=0x7f7bc4017000 comp rx=0 tx=0).stop 2024-01-27T01:06:46.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.874+0000 7f7bd019b700 1 -- 172.21.15.46:0/2341084152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc8159760 msgr2=0x7f7bc80ad5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:46.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.874+0000 7f7bd019b700 1 --2- 172.21.15.46:0/2341084152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc8159760 0x7f7bc80ad5f0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0007f30 tx=0x7f7bc00098e0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.874+0000 7f7bd019b700 1 -- 172.21.15.46:0/2341084152 shutdown_connections 2024-01-27T01:06:46.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.874+0000 7f7bd019b700 1 --2- 172.21.15.46:0/2341084152 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bac066f30 0x7f7bac0693e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.874+0000 7f7bd019b700 1 --2- 172.21.15.46:0/2341084152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc8159760 0x7f7bc80ad5f0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.874+0000 7f7bd019b700 1 --2- 172.21.15.46:0/2341084152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bc80b12c0 0x7f7bc816dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.874+0000 7f7bd019b700 1 --2- 172.21.15.46:0/2341084152 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bc80b3060 0x7f7bc8142e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:46.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.874+0000 7f7bd019b700 1 -- 172.21.15.46:0/2341084152 >> 172.21.15.46:0/2341084152 conn(0x7f7bc8069510 msgr2=0x7f7bc80acdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:46.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.874+0000 7f7bd019b700 1 -- 172.21.15.46:0/2341084152 shutdown_connections 2024-01-27T01:06:46.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:46.874+0000 7f7bd019b700 1 -- 172.21.15.46:0/2341084152 wait complete. 2024-01-27T01:06:46.881 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:06:46.887 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-01-27T01:06:46.734927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596212,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":721,"num_write_kb":3490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8829,"ondisk_log_size":8829,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":376,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2467400,"kb_used_data":23560,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752507320,"statfs":{"total":773094113280,"available":770567495680,"internally_reserved":0,"allocated":24125440,"data_stored":13236688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.818889"},"pg_stats":[{"pgid":"1.0","version":"213'8750","reported_seq":9668,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.932851+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:06:40.932851+0000","last_peered":"2024-01-27T01:06:40.932851+0000","last_clean":"2024-01-27T01:06:40.932851+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:06:40.932851+0000","last_undegraded":"2024-01-27T01:06:40.932851+0000","last_fullsized":"2024-01-27T01:06:40.932851+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8750,"log_dups_size":0,"ondisk_log_size":8750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":630,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"222'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923976+0000","last_change":"2024-01-27T01:06:25.858609+0000","last_active":"2024-01-27T01:06:40.923976+0000","last_peered":"2024-01-27T01:06:40.923976+0000","last_clean":"2024-01-27T01:06:40.923976+0000","last_became_active":"2024-01-27T01:06:25.857638+0000","last_became_peered":"2024-01-27T01:06:25.857638+0000","last_unstale":"2024-01-27T01:06:40.923976+0000","last_undegraded":"2024-01-27T01:06:40.923976+0000","last_fullsized":"2024-01-27T01:06:40.923976+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:35:59.109571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.927920+0000","last_change":"2024-01-27T01:06:26.873783+0000","last_active":"2024-01-27T01:06:40.927920+0000","last_peered":"2024-01-27T01:06:40.927920+0000","last_clean":"2024-01-27T01:06:40.927920+0000","last_became_active":"2024-01-27T01:06:26.873490+0000","last_became_peered":"2024-01-27T01:06:26.873490+0000","last_unstale":"2024-01-27T01:06:40.927920+0000","last_undegraded":"2024-01-27T01:06:40.927920+0000","last_fullsized":"2024-01-27T01:06:40.927920+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:32.691743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.954737+0000","last_change":"2024-01-27T01:06:39.937077+0000","last_active":"2024-01-27T01:06:40.954737+0000","last_peered":"2024-01-27T01:06:40.954737+0000","last_clean":"2024-01-27T01:06:40.954737+0000","last_became_active":"2024-01-27T01:06:39.936782+0000","last_became_peered":"2024-01-27T01:06:39.936782+0000","last_unstale":"2024-01-27T01:06:40.954737+0000","last_undegraded":"2024-01-27T01:06:40.954737+0000","last_fullsized":"2024-01-27T01:06:40.954737+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:40.845691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.943895+0000","last_change":"2024-01-27T01:06:40.942047+0000","last_active":"2024-01-27T01:06:40.943895+0000","last_peered":"2024-01-27T01:06:40.943895+0000","last_clean":"2024-01-27T01:06:40.943895+0000","last_became_active":"2024-01-27T01:06:40.941754+0000","last_became_peered":"2024-01-27T01:06:40.941754+0000","last_unstale":"2024-01-27T01:06:40.943895+0000","last_undegraded":"2024-01-27T01:06:40.943895+0000","last_fullsized":"2024-01-27T01:06:40.943895+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:27:01.280156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.938272+0000","last_change":"2024-01-27T01:06:25.865235+0000","last_active":"2024-01-27T01:06:40.938272+0000","last_peered":"2024-01-27T01:06:40.938272+0000","last_clean":"2024-01-27T01:06:40.938272+0000","last_became_active":"2024-01-27T01:06:25.864872+0000","last_became_peered":"2024-01-27T01:06:25.864872+0000","last_unstale":"2024-01-27T01:06:40.938272+0000","last_undegraded":"2024-01-27T01:06:40.938272+0000","last_fullsized":"2024-01-27T01:06:40.938272+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:53:38.618410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919351+0000","last_change":"2024-01-27T01:06:26.879306+0000","last_active":"2024-01-27T01:06:40.919351+0000","last_peered":"2024-01-27T01:06:40.919351+0000","last_clean":"2024-01-27T01:06:40.919351+0000","last_became_active":"2024-01-27T01:06:26.879071+0000","last_became_peered":"2024-01-27T01:06:26.879071+0000","last_unstale":"2024-01-27T01:06:40.919351+0000","last_undegraded":"2024-01-27T01:06:40.919351+0000","last_fullsized":"2024-01-27T01:06:40.919351+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:50:46.965359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919137+0000","last_change":"2024-01-27T01:06:39.930566+0000","last_active":"2024-01-27T01:06:40.919137+0000","last_peered":"2024-01-27T01:06:40.919137+0000","last_clean":"2024-01-27T01:06:40.919137+0000","last_became_active":"2024-01-27T01:06:39.930266+0000","last_became_peered":"2024-01-27T01:06:39.930266+0000","last_unstale":"2024-01-27T01:06:40.919137+0000","last_undegraded":"2024-01-27T01:06:40.919137+0000","last_fullsized":"2024-01-27T01:06:40.919137+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:33:45.602108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.948093+0000","last_change":"2024-01-27T01:06:40.944710+0000","last_active":"2024-01-27T01:06:40.948093+0000","last_peered":"2024-01-27T01:06:40.948093+0000","last_clean":"2024-01-27T01:06:40.948093+0000","last_became_active":"2024-01-27T01:06:40.944030+0000","last_became_peered":"2024-01-27T01:06:40.944030+0000","last_unstale":"2024-01-27T01:06:40.948093+0000","last_undegraded":"2024-01-27T01:06:40.948093+0000","last_fullsized":"2024-01-27T01:06:40.948093+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:12:42.423061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923325+0000","last_change":"2024-01-27T01:06:25.856035+0000","last_active":"2024-01-27T01:06:40.923325+0000","last_peered":"2024-01-27T01:06:40.923325+0000","last_clean":"2024-01-27T01:06:40.923325+0000","last_became_active":"2024-01-27T01:06:25.855762+0000","last_became_peered":"2024-01-27T01:06:25.855762+0000","last_unstale":"2024-01-27T01:06:40.923325+0000","last_undegraded":"2024-01-27T01:06:40.923325+0000","last_fullsized":"2024-01-27T01:06:40.923325+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:51:44.378880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931904+0000","last_change":"2024-01-27T01:06:26.870626+0000","last_active":"2024-01-27T01:06:40.931904+0000","last_peered":"2024-01-27T01:06:40.931904+0000","last_clean":"2024-01-27T01:06:40.931904+0000","last_became_active":"2024-01-27T01:06:26.870457+0000","last_became_peered":"2024-01-27T01:06:26.870457+0000","last_unstale":"2024-01-27T01:06:40.931904+0000","last_undegraded":"2024-01-27T01:06:40.931904+0000","last_fullsized":"2024-01-27T01:06:40.931904+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:42.766155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.953645+0000","last_change":"2024-01-27T01:06:39.934530+0000","last_active":"2024-01-27T01:06:40.953645+0000","last_peered":"2024-01-27T01:06:40.953645+0000","last_clean":"2024-01-27T01:06:40.953645+0000","last_became_active":"2024-01-27T01:06:39.934264+0000","last_became_peered":"2024-01-27T01:06:39.934264+0000","last_unstale":"2024-01-27T01:06:40.953645+0000","last_undegraded":"2024-01-27T01:06:40.953645+0000","last_fullsized":"2024-01-27T01:06:40.953645+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:26:34.409044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.946078+0000","last_change":"2024-01-27T01:06:40.941588+0000","last_active":"2024-01-27T01:06:40.946078+0000","last_peered":"2024-01-27T01:06:40.946078+0000","last_clean":"2024-01-27T01:06:40.946078+0000","last_became_active":"2024-01-27T01:06:40.936001+0000","last_became_peered":"2024-01-27T01:06:40.936001+0000","last_unstale":"2024-01-27T01:06:40.946078+0000","last_undegraded":"2024-01-27T01:06:40.946078+0000","last_fullsized":"2024-01-27T01:06:40.946078+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:37:42.686040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923654+0000","last_change":"2024-01-27T01:06:25.855015+0000","last_active":"2024-01-27T01:06:40.923654+0000","last_peered":"2024-01-27T01:06:40.923654+0000","last_clean":"2024-01-27T01:06:40.923654+0000","last_became_active":"2024-01-27T01:06:25.854708+0000","last_became_peered":"2024-01-27T01:06:25.854708+0000","last_unstale":"2024-01-27T01:06:40.923654+0000","last_undegraded":"2024-01-27T01:06:40.923654+0000","last_fullsized":"2024-01-27T01:06:40.923654+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:03.971945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.933878+0000","last_change":"2024-01-27T01:06:26.876182+0000","last_active":"2024-01-27T01:06:40.933878+0000","last_peered":"2024-01-27T01:06:40.933878+0000","last_clean":"2024-01-27T01:06:40.933878+0000","last_became_active":"2024-01-27T01:06:26.875894+0000","last_became_peered":"2024-01-27T01:06:26.875894+0000","last_unstale":"2024-01-27T01:06:40.933878+0000","last_undegraded":"2024-01-27T01:06:40.933878+0000","last_fullsized":"2024-01-27T01:06:40.933878+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:58.495062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.927024+0000","last_change":"2024-01-27T01:06:39.939172+0000","last_active":"2024-01-27T01:06:40.927024+0000","last_peered":"2024-01-27T01:06:40.927024+0000","last_clean":"2024-01-27T01:06:40.927024+0000","last_became_active":"2024-01-27T01:06:39.938766+0000","last_became_peered":"2024-01-27T01:06:39.938766+0000","last_unstale":"2024-01-27T01:06:40.927024+0000","last_undegraded":"2024-01-27T01:06:40.927024+0000","last_fullsized":"2024-01-27T01:06:40.927024+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:33:45.602108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.952716+0000","last_change":"2024-01-27T01:06:40.943896+0000","last_active":"2024-01-27T01:06:40.952716+0000","last_peered":"2024-01-27T01:06:40.952716+0000","last_clean":"2024-01-27T01:06:40.952716+0000","last_became_active":"2024-01-27T01:06:40.943570+0000","last_became_peered":"2024-01-27T01:06:40.943570+0000","last_unstale":"2024-01-27T01:06:40.952716+0000","last_undegraded":"2024-01-27T01:06:40.952716+0000","last_fullsized":"2024-01-27T01:06:40.952716+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:42.451296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926823+0000","last_change":"2024-01-27T01:06:25.858142+0000","last_active":"2024-01-27T01:06:40.926823+0000","last_peered":"2024-01-27T01:06:40.926823+0000","last_clean":"2024-01-27T01:06:40.926823+0000","last_became_active":"2024-01-27T01:06:25.857867+0000","last_became_peered":"2024-01-27T01:06:25.857867+0000","last_unstale":"2024-01-27T01:06:40.926823+0000","last_undegraded":"2024-01-27T01:06:40.926823+0000","last_fullsized":"2024-01-27T01:06:40.926823+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:10:59.780192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.927155+0000","last_change":"2024-01-27T01:06:26.880503+0000","last_active":"2024-01-27T01:06:40.927155+0000","last_peered":"2024-01-27T01:06:40.927155+0000","last_clean":"2024-01-27T01:06:40.927155+0000","last_became_active":"2024-01-27T01:06:26.880224+0000","last_became_peered":"2024-01-27T01:06:26.880224+0000","last_unstale":"2024-01-27T01:06:40.927155+0000","last_undegraded":"2024-01-27T01:06:40.927155+0000","last_fullsized":"2024-01-27T01:06:40.927155+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:27:33.886740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919769+0000","last_change":"2024-01-27T01:06:39.928493+0000","last_active":"2024-01-27T01:06:40.919769+0000","last_peered":"2024-01-27T01:06:40.919769+0000","last_clean":"2024-01-27T01:06:40.919769+0000","last_became_active":"2024-01-27T01:06:39.928166+0000","last_became_peered":"2024-01-27T01:06:39.928166+0000","last_unstale":"2024-01-27T01:06:40.919769+0000","last_undegraded":"2024-01-27T01:06:40.919769+0000","last_fullsized":"2024-01-27T01:06:40.919769+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:15:24.362081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942986+0000","last_change":"2024-01-27T01:06:40.938135+0000","last_active":"2024-01-27T01:06:40.942986+0000","last_peered":"2024-01-27T01:06:40.942986+0000","last_clean":"2024-01-27T01:06:40.942986+0000","last_became_active":"2024-01-27T01:06:40.937878+0000","last_became_peered":"2024-01-27T01:06:40.937878+0000","last_unstale":"2024-01-27T01:06:40.942986+0000","last_undegraded":"2024-01-27T01:06:40.942986+0000","last_fullsized":"2024-01-27T01:06:40.942986+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:31:19.372123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.927955+0000","last_change":"2024-01-27T01:06:25.859342+0000","last_active":"2024-01-27T01:06:40.927955+0000","last_peered":"2024-01-27T01:06:40.927955+0000","last_clean":"2024-01-27T01:06:40.927955+0000","last_became_active":"2024-01-27T01:06:25.859065+0000","last_became_peered":"2024-01-27T01:06:25.859065+0000","last_unstale":"2024-01-27T01:06:40.927955+0000","last_undegraded":"2024-01-27T01:06:40.927955+0000","last_fullsized":"2024-01-27T01:06:40.927955+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:00:54.297085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925207+0000","last_change":"2024-01-27T01:06:26.870597+0000","last_active":"2024-01-27T01:06:40.925207+0000","last_peered":"2024-01-27T01:06:40.925207+0000","last_clean":"2024-01-27T01:06:40.925207+0000","last_became_active":"2024-01-27T01:06:26.870334+0000","last_became_peered":"2024-01-27T01:06:26.870334+0000","last_unstale":"2024-01-27T01:06:40.925207+0000","last_undegraded":"2024-01-27T01:06:40.925207+0000","last_fullsized":"2024-01-27T01:06:40.925207+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:01:23.266072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.933049+0000","last_change":"2024-01-27T01:06:39.933497+0000","last_active":"2024-01-27T01:06:40.933049+0000","last_peered":"2024-01-27T01:06:40.933049+0000","last_clean":"2024-01-27T01:06:40.933049+0000","last_became_active":"2024-01-27T01:06:39.933215+0000","last_became_peered":"2024-01-27T01:06:39.933215+0000","last_unstale":"2024-01-27T01:06:40.933049+0000","last_undegraded":"2024-01-27T01:06:40.933049+0000","last_fullsized":"2024-01-27T01:06:40.933049+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:22:36.612223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942697+0000","last_change":"2024-01-27T01:06:40.939651+0000","last_active":"2024-01-27T01:06:40.942697+0000","last_peered":"2024-01-27T01:06:40.942697+0000","last_clean":"2024-01-27T01:06:40.942697+0000","last_became_active":"2024-01-27T01:06:40.939320+0000","last_became_peered":"2024-01-27T01:06:40.939320+0000","last_unstale":"2024-01-27T01:06:40.942697+0000","last_undegraded":"2024-01-27T01:06:40.942697+0000","last_fullsized":"2024-01-27T01:06:40.942697+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:05:34.497339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.932963+0000","last_change":"2024-01-27T01:06:25.858032+0000","last_active":"2024-01-27T01:06:40.932963+0000","last_peered":"2024-01-27T01:06:40.932963+0000","last_clean":"2024-01-27T01:06:40.932963+0000","last_became_active":"2024-01-27T01:06:25.857777+0000","last_became_peered":"2024-01-27T01:06:25.857777+0000","last_unstale":"2024-01-27T01:06:40.932963+0000","last_undegraded":"2024-01-27T01:06:40.932963+0000","last_fullsized":"2024-01-27T01:06:40.932963+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:09:08.458850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923226+0000","last_change":"2024-01-27T01:06:26.872190+0000","last_active":"2024-01-27T01:06:40.923226+0000","last_peered":"2024-01-27T01:06:40.923226+0000","last_clean":"2024-01-27T01:06:40.923226+0000","last_became_active":"2024-01-27T01:06:26.867718+0000","last_became_peered":"2024-01-27T01:06:26.867718+0000","last_unstale":"2024-01-27T01:06:40.923226+0000","last_undegraded":"2024-01-27T01:06:40.923226+0000","last_fullsized":"2024-01-27T01:06:40.923226+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:23.301132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.927192+0000","last_change":"2024-01-27T01:06:39.941456+0000","last_active":"2024-01-27T01:06:40.927192+0000","last_peered":"2024-01-27T01:06:40.927192+0000","last_clean":"2024-01-27T01:06:40.927192+0000","last_became_active":"2024-01-27T01:06:39.940373+0000","last_became_peered":"2024-01-27T01:06:39.940373+0000","last_unstale":"2024-01-27T01:06:40.927192+0000","last_undegraded":"2024-01-27T01:06:40.927192+0000","last_fullsized":"2024-01-27T01:06:40.927192+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:18:25.975883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.949946+0000","last_change":"2024-01-27T01:06:40.942226+0000","last_active":"2024-01-27T01:06:40.949946+0000","last_peered":"2024-01-27T01:06:40.949946+0000","last_clean":"2024-01-27T01:06:40.949946+0000","last_became_active":"2024-01-27T01:06:40.941911+0000","last_became_peered":"2024-01-27T01:06:40.941911+0000","last_unstale":"2024-01-27T01:06:40.949946+0000","last_undegraded":"2024-01-27T01:06:40.949946+0000","last_fullsized":"2024-01-27T01:06:40.949946+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:27:18.007273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.6","version":"223'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.928859+0000","last_change":"2024-01-27T01:06:25.861717+0000","last_active":"2024-01-27T01:06:40.928859+0000","last_peered":"2024-01-27T01:06:40.928859+0000","last_clean":"2024-01-27T01:06:40.928859+0000","last_became_active":"2024-01-27T01:06:25.861177+0000","last_became_peered":"2024-01-27T01:06:25.861177+0000","last_unstale":"2024-01-27T01:06:40.928859+0000","last_undegraded":"2024-01-27T01:06:40.928859+0000","last_fullsized":"2024-01-27T01:06:40.928859+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:05.596045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919205+0000","last_change":"2024-01-27T01:06:26.878650+0000","last_active":"2024-01-27T01:06:40.919205+0000","last_peered":"2024-01-27T01:06:40.919205+0000","last_clean":"2024-01-27T01:06:40.919205+0000","last_became_active":"2024-01-27T01:06:26.877596+0000","last_became_peered":"2024-01-27T01:06:26.877596+0000","last_unstale":"2024-01-27T01:06:40.919205+0000","last_undegraded":"2024-01-27T01:06:40.919205+0000","last_fullsized":"2024-01-27T01:06:40.919205+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:42:48.942202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926127+0000","last_change":"2024-01-27T01:06:39.934375+0000","last_active":"2024-01-27T01:06:40.926127+0000","last_peered":"2024-01-27T01:06:40.926127+0000","last_clean":"2024-01-27T01:06:40.926127+0000","last_became_active":"2024-01-27T01:06:39.934098+0000","last_became_peered":"2024-01-27T01:06:39.934098+0000","last_unstale":"2024-01-27T01:06:40.926127+0000","last_undegraded":"2024-01-27T01:06:40.926127+0000","last_fullsized":"2024-01-27T01:06:40.926127+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:13:12.942303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.945722+0000","last_change":"2024-01-27T01:06:40.940649+0000","last_active":"2024-01-27T01:06:40.945722+0000","last_peered":"2024-01-27T01:06:40.945722+0000","last_clean":"2024-01-27T01:06:40.945722+0000","last_became_active":"2024-01-27T01:06:40.940389+0000","last_became_peered":"2024-01-27T01:06:40.940389+0000","last_unstale":"2024-01-27T01:06:40.945722+0000","last_undegraded":"2024-01-27T01:06:40.945722+0000","last_fullsized":"2024-01-27T01:06:40.945722+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:08:47.665701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"222'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925560+0000","last_change":"2024-01-27T01:06:25.857933+0000","last_active":"2024-01-27T01:06:40.925560+0000","last_peered":"2024-01-27T01:06:40.925560+0000","last_clean":"2024-01-27T01:06:40.925560+0000","last_became_active":"2024-01-27T01:06:25.857406+0000","last_became_peered":"2024-01-27T01:06:25.857406+0000","last_unstale":"2024-01-27T01:06:40.925560+0000","last_undegraded":"2024-01-27T01:06:40.925560+0000","last_fullsized":"2024-01-27T01:06:40.925560+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:20:20.239633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926501+0000","last_change":"2024-01-27T01:06:26.875222+0000","last_active":"2024-01-27T01:06:40.926501+0000","last_peered":"2024-01-27T01:06:40.926501+0000","last_clean":"2024-01-27T01:06:40.926501+0000","last_became_active":"2024-01-27T01:06:26.874903+0000","last_became_peered":"2024-01-27T01:06:26.874903+0000","last_unstale":"2024-01-27T01:06:40.926501+0000","last_undegraded":"2024-01-27T01:06:40.926501+0000","last_fullsized":"2024-01-27T01:06:40.926501+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:09:33.903445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.932982+0000","last_change":"2024-01-27T01:06:39.938367+0000","last_active":"2024-01-27T01:06:40.932982+0000","last_peered":"2024-01-27T01:06:40.932982+0000","last_clean":"2024-01-27T01:06:40.932982+0000","last_became_active":"2024-01-27T01:06:39.938071+0000","last_became_peered":"2024-01-27T01:06:39.938071+0000","last_unstale":"2024-01-27T01:06:40.932982+0000","last_undegraded":"2024-01-27T01:06:40.932982+0000","last_fullsized":"2024-01-27T01:06:40.932982+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:26:34.409044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.947647+0000","last_change":"2024-01-27T01:06:40.939338+0000","last_active":"2024-01-27T01:06:40.947647+0000","last_peered":"2024-01-27T01:06:40.947647+0000","last_clean":"2024-01-27T01:06:40.947647+0000","last_became_active":"2024-01-27T01:06:40.939008+0000","last_became_peered":"2024-01-27T01:06:40.939008+0000","last_unstale":"2024-01-27T01:06:40.947647+0000","last_undegraded":"2024-01-27T01:06:40.947647+0000","last_fullsized":"2024-01-27T01:06:40.947647+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:08:05.712678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.924020+0000","last_change":"2024-01-27T01:06:25.853605+0000","last_active":"2024-01-27T01:06:40.924020+0000","last_peered":"2024-01-27T01:06:40.924020+0000","last_clean":"2024-01-27T01:06:40.924020+0000","last_became_active":"2024-01-27T01:06:25.853331+0000","last_became_peered":"2024-01-27T01:06:25.853331+0000","last_unstale":"2024-01-27T01:06:40.924020+0000","last_undegraded":"2024-01-27T01:06:40.924020+0000","last_fullsized":"2024-01-27T01:06:40.924020+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:09:08.458850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.928716+0000","last_change":"2024-01-27T01:06:26.875536+0000","last_active":"2024-01-27T01:06:40.928716+0000","last_peered":"2024-01-27T01:06:40.928716+0000","last_clean":"2024-01-27T01:06:40.928716+0000","last_became_active":"2024-01-27T01:06:26.875267+0000","last_became_peered":"2024-01-27T01:06:26.875267+0000","last_unstale":"2024-01-27T01:06:40.928716+0000","last_undegraded":"2024-01-27T01:06:40.928716+0000","last_fullsized":"2024-01-27T01:06:40.928716+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:32.534622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.955063+0000","last_change":"2024-01-27T01:06:39.937034+0000","last_active":"2024-01-27T01:06:40.955063+0000","last_peered":"2024-01-27T01:06:40.955063+0000","last_clean":"2024-01-27T01:06:40.955063+0000","last_became_active":"2024-01-27T01:06:39.936709+0000","last_became_peered":"2024-01-27T01:06:39.936709+0000","last_unstale":"2024-01-27T01:06:40.955063+0000","last_undegraded":"2024-01-27T01:06:40.955063+0000","last_fullsized":"2024-01-27T01:06:40.955063+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:35.078462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.947436+0000","last_change":"2024-01-27T01:06:40.939425+0000","last_active":"2024-01-27T01:06:40.947436+0000","last_peered":"2024-01-27T01:06:40.947436+0000","last_clean":"2024-01-27T01:06:40.947436+0000","last_became_active":"2024-01-27T01:06:40.939162+0000","last_became_peered":"2024-01-27T01:06:40.939162+0000","last_unstale":"2024-01-27T01:06:40.947436+0000","last_undegraded":"2024-01-27T01:06:40.947436+0000","last_fullsized":"2024-01-27T01:06:40.947436+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:14:19.010092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.934093+0000","last_change":"2024-01-27T01:06:25.862199+0000","last_active":"2024-01-27T01:06:40.934093+0000","last_peered":"2024-01-27T01:06:40.934093+0000","last_clean":"2024-01-27T01:06:40.934093+0000","last_became_active":"2024-01-27T01:06:25.861876+0000","last_became_peered":"2024-01-27T01:06:25.861876+0000","last_unstale":"2024-01-27T01:06:40.934093+0000","last_undegraded":"2024-01-27T01:06:40.934093+0000","last_fullsized":"2024-01-27T01:06:40.934093+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:00:54.297085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925485+0000","last_change":"2024-01-27T01:06:26.867570+0000","last_active":"2024-01-27T01:06:40.925485+0000","last_peered":"2024-01-27T01:06:40.925485+0000","last_clean":"2024-01-27T01:06:40.925485+0000","last_became_active":"2024-01-27T01:06:26.866517+0000","last_became_peered":"2024-01-27T01:06:26.866517+0000","last_unstale":"2024-01-27T01:06:40.925485+0000","last_undegraded":"2024-01-27T01:06:40.925485+0000","last_fullsized":"2024-01-27T01:06:40.925485+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:09:54.515702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.955677+0000","last_change":"2024-01-27T01:06:39.938159+0000","last_active":"2024-01-27T01:06:40.955677+0000","last_peered":"2024-01-27T01:06:40.955677+0000","last_clean":"2024-01-27T01:06:40.955677+0000","last_became_active":"2024-01-27T01:06:39.937871+0000","last_became_peered":"2024-01-27T01:06:39.937871+0000","last_unstale":"2024-01-27T01:06:40.955677+0000","last_undegraded":"2024-01-27T01:06:40.955677+0000","last_fullsized":"2024-01-27T01:06:40.955677+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:36.367666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.946099+0000","last_change":"2024-01-27T01:06:40.941996+0000","last_active":"2024-01-27T01:06:40.946099+0000","last_peered":"2024-01-27T01:06:40.946099+0000","last_clean":"2024-01-27T01:06:40.946099+0000","last_became_active":"2024-01-27T01:06:40.936467+0000","last_became_peered":"2024-01-27T01:06:40.936467+0000","last_unstale":"2024-01-27T01:06:40.946099+0000","last_undegraded":"2024-01-27T01:06:40.946099+0000","last_fullsized":"2024-01-27T01:06:40.946099+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:47:39.379533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.934281+0000","last_change":"2024-01-27T01:06:25.851595+0000","last_active":"2024-01-27T01:06:40.934281+0000","last_peered":"2024-01-27T01:06:40.934281+0000","last_clean":"2024-01-27T01:06:40.934281+0000","last_became_active":"2024-01-27T01:06:25.851260+0000","last_became_peered":"2024-01-27T01:06:25.851260+0000","last_unstale":"2024-01-27T01:06:40.934281+0000","last_undegraded":"2024-01-27T01:06:40.934281+0000","last_fullsized":"2024-01-27T01:06:40.934281+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:56:53.179347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.933644+0000","last_change":"2024-01-27T01:06:26.875858+0000","last_active":"2024-01-27T01:06:40.933644+0000","last_peered":"2024-01-27T01:06:40.933644+0000","last_clean":"2024-01-27T01:06:40.933644+0000","last_became_active":"2024-01-27T01:06:26.875564+0000","last_became_peered":"2024-01-27T01:06:26.875564+0000","last_unstale":"2024-01-27T01:06:40.933644+0000","last_undegraded":"2024-01-27T01:06:40.933644+0000","last_fullsized":"2024-01-27T01:06:40.933644+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:43.630831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926797+0000","last_change":"2024-01-27T01:06:39.938832+0000","last_active":"2024-01-27T01:06:40.926797+0000","last_peered":"2024-01-27T01:06:40.926797+0000","last_clean":"2024-01-27T01:06:40.926797+0000","last_became_active":"2024-01-27T01:06:39.938365+0000","last_became_peered":"2024-01-27T01:06:39.938365+0000","last_unstale":"2024-01-27T01:06:40.926797+0000","last_undegraded":"2024-01-27T01:06:40.926797+0000","last_fullsized":"2024-01-27T01:06:40.926797+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:31.265591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.944315+0000","last_change":"2024-01-27T01:06:40.940315+0000","last_active":"2024-01-27T01:06:40.944315+0000","last_peered":"2024-01-27T01:06:40.944315+0000","last_clean":"2024-01-27T01:06:40.944315+0000","last_became_active":"2024-01-27T01:06:40.940058+0000","last_became_peered":"2024-01-27T01:06:40.940058+0000","last_unstale":"2024-01-27T01:06:40.944315+0000","last_undegraded":"2024-01-27T01:06:40.944315+0000","last_fullsized":"2024-01-27T01:06:40.944315+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:37.373751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.b","version":"222'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.939300+0000","last_change":"2024-01-27T01:06:25.860572+0000","last_active":"2024-01-27T01:06:40.939300+0000","last_peered":"2024-01-27T01:06:40.939300+0000","last_clean":"2024-01-27T01:06:40.939300+0000","last_became_active":"2024-01-27T01:06:25.860248+0000","last_became_peered":"2024-01-27T01:06:25.860248+0000","last_unstale":"2024-01-27T01:06:40.939300+0000","last_undegraded":"2024-01-27T01:06:40.939300+0000","last_fullsized":"2024-01-27T01:06:40.939300+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:01:25.741123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931847+0000","last_change":"2024-01-27T01:06:26.871346+0000","last_active":"2024-01-27T01:06:40.931847+0000","last_peered":"2024-01-27T01:06:40.931847+0000","last_clean":"2024-01-27T01:06:40.931847+0000","last_became_active":"2024-01-27T01:06:26.870999+0000","last_became_peered":"2024-01-27T01:06:26.870999+0000","last_unstale":"2024-01-27T01:06:40.931847+0000","last_undegraded":"2024-01-27T01:06:40.931847+0000","last_fullsized":"2024-01-27T01:06:40.931847+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:27.779157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.954555+0000","last_change":"2024-01-27T01:06:39.932938+0000","last_active":"2024-01-27T01:06:40.954555+0000","last_peered":"2024-01-27T01:06:40.954555+0000","last_clean":"2024-01-27T01:06:40.954555+0000","last_became_active":"2024-01-27T01:06:39.930096+0000","last_became_peered":"2024-01-27T01:06:39.930096+0000","last_unstale":"2024-01-27T01:06:40.954555+0000","last_undegraded":"2024-01-27T01:06:40.954555+0000","last_fullsized":"2024-01-27T01:06:40.954555+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:57:15.231912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.944057+0000","last_change":"2024-01-27T01:06:40.943101+0000","last_active":"2024-01-27T01:06:40.944057+0000","last_peered":"2024-01-27T01:06:40.944057+0000","last_clean":"2024-01-27T01:06:40.944057+0000","last_became_active":"2024-01-27T01:06:40.942787+0000","last_became_peered":"2024-01-27T01:06:40.942787+0000","last_unstale":"2024-01-27T01:06:40.944057+0000","last_undegraded":"2024-01-27T01:06:40.944057+0000","last_fullsized":"2024-01-27T01:06:40.944057+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:27:25.230045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.941219+0000","last_change":"2024-01-27T01:06:25.861650+0000","last_active":"2024-01-27T01:06:40.941219+0000","last_peered":"2024-01-27T01:06:40.941219+0000","last_clean":"2024-01-27T01:06:40.941219+0000","last_became_active":"2024-01-27T01:06:25.861342+0000","last_became_peered":"2024-01-27T01:06:25.861342+0000","last_unstale":"2024-01-27T01:06:40.941219+0000","last_undegraded":"2024-01-27T01:06:40.941219+0000","last_fullsized":"2024-01-27T01:06:40.941219+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:44:15.953217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926988+0000","last_change":"2024-01-27T01:06:26.876619+0000","last_active":"2024-01-27T01:06:40.926988+0000","last_peered":"2024-01-27T01:06:40.926988+0000","last_clean":"2024-01-27T01:06:40.926988+0000","last_became_active":"2024-01-27T01:06:26.875927+0000","last_became_peered":"2024-01-27T01:06:26.875927+0000","last_unstale":"2024-01-27T01:06:40.926988+0000","last_undegraded":"2024-01-27T01:06:40.926988+0000","last_fullsized":"2024-01-27T01:06:40.926988+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:02.000909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919397+0000","last_change":"2024-01-27T01:06:39.929131+0000","last_active":"2024-01-27T01:06:40.919397+0000","last_peered":"2024-01-27T01:06:40.919397+0000","last_clean":"2024-01-27T01:06:40.919397+0000","last_became_active":"2024-01-27T01:06:39.927860+0000","last_became_peered":"2024-01-27T01:06:39.927860+0000","last_unstale":"2024-01-27T01:06:40.919397+0000","last_undegraded":"2024-01-27T01:06:40.919397+0000","last_fullsized":"2024-01-27T01:06:40.919397+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:13:12.942303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.948177+0000","last_change":"2024-01-27T01:06:40.944322+0000","last_active":"2024-01-27T01:06:40.948177+0000","last_peered":"2024-01-27T01:06:40.948177+0000","last_clean":"2024-01-27T01:06:40.948177+0000","last_became_active":"2024-01-27T01:06:40.944008+0000","last_became_peered":"2024-01-27T01:06:40.944008+0000","last_unstale":"2024-01-27T01:06:40.948177+0000","last_undegraded":"2024-01-27T01:06:40.948177+0000","last_fullsized":"2024-01-27T01:06:40.948177+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:09:11.010297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.d","version":"222'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931425+0000","last_change":"2024-01-27T01:06:25.851720+0000","last_active":"2024-01-27T01:06:40.931425+0000","last_peered":"2024-01-27T01:06:40.931425+0000","last_clean":"2024-01-27T01:06:40.931425+0000","last_became_active":"2024-01-27T01:06:25.851416+0000","last_became_peered":"2024-01-27T01:06:25.851416+0000","last_unstale":"2024-01-27T01:06:40.931425+0000","last_undegraded":"2024-01-27T01:06:40.931425+0000","last_fullsized":"2024-01-27T01:06:40.931425+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:07.951814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925914+0000","last_change":"2024-01-27T01:06:26.870758+0000","last_active":"2024-01-27T01:06:40.925914+0000","last_peered":"2024-01-27T01:06:40.925914+0000","last_clean":"2024-01-27T01:06:40.925914+0000","last_became_active":"2024-01-27T01:06:26.870130+0000","last_became_peered":"2024-01-27T01:06:26.870130+0000","last_unstale":"2024-01-27T01:06:40.925914+0000","last_undegraded":"2024-01-27T01:06:40.925914+0000","last_fullsized":"2024-01-27T01:06:40.925914+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:36:31.625857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.954611+0000","last_change":"2024-01-27T01:06:39.934635+0000","last_active":"2024-01-27T01:06:40.954611+0000","last_peered":"2024-01-27T01:06:40.954611+0000","last_clean":"2024-01-27T01:06:40.954611+0000","last_became_active":"2024-01-27T01:06:39.934370+0000","last_became_peered":"2024-01-27T01:06:39.934370+0000","last_unstale":"2024-01-27T01:06:40.954611+0000","last_undegraded":"2024-01-27T01:06:40.954611+0000","last_fullsized":"2024-01-27T01:06:40.954611+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:40.845691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.945691+0000","last_change":"2024-01-27T01:06:40.938827+0000","last_active":"2024-01-27T01:06:40.945691+0000","last_peered":"2024-01-27T01:06:40.945691+0000","last_clean":"2024-01-27T01:06:40.945691+0000","last_became_active":"2024-01-27T01:06:40.938572+0000","last_became_peered":"2024-01-27T01:06:40.938572+0000","last_unstale":"2024-01-27T01:06:40.945691+0000","last_undegraded":"2024-01-27T01:06:40.945691+0000","last_fullsized":"2024-01-27T01:06:40.945691+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:38:42.855259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.e","version":"222'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.928649+0000","last_change":"2024-01-27T01:06:25.852245+0000","last_active":"2024-01-27T01:06:40.928649+0000","last_peered":"2024-01-27T01:06:40.928649+0000","last_clean":"2024-01-27T01:06:40.928649+0000","last_became_active":"2024-01-27T01:06:25.851982+0000","last_became_peered":"2024-01-27T01:06:25.851982+0000","last_unstale":"2024-01-27T01:06:40.928649+0000","last_undegraded":"2024-01-27T01:06:40.928649+0000","last_fullsized":"2024-01-27T01:06:40.928649+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:05.596045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923733+0000","last_change":"2024-01-27T01:06:26.879356+0000","last_active":"2024-01-27T01:06:40.923733+0000","last_peered":"2024-01-27T01:06:40.923733+0000","last_clean":"2024-01-27T01:06:40.923733+0000","last_became_active":"2024-01-27T01:06:26.878891+0000","last_became_peered":"2024-01-27T01:06:26.878891+0000","last_unstale":"2024-01-27T01:06:40.923733+0000","last_undegraded":"2024-01-27T01:06:40.923733+0000","last_fullsized":"2024-01-27T01:06:40.923733+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:22.011928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.937963+0000","last_change":"2024-01-27T01:06:39.937159+0000","last_active":"2024-01-27T01:06:40.937963+0000","last_peered":"2024-01-27T01:06:40.937963+0000","last_clean":"2024-01-27T01:06:40.937963+0000","last_became_active":"2024-01-27T01:06:39.936720+0000","last_became_peered":"2024-01-27T01:06:39.936720+0000","last_unstale":"2024-01-27T01:06:40.937963+0000","last_undegraded":"2024-01-27T01:06:40.937963+0000","last_fullsized":"2024-01-27T01:06:40.937963+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:45.758278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.940354+0000","last_change":"2024-01-27T01:06:40.935737+0000","last_active":"2024-01-27T01:06:40.940354+0000","last_peered":"2024-01-27T01:06:40.940354+0000","last_clean":"2024-01-27T01:06:40.940354+0000","last_became_active":"2024-01-27T01:06:40.935414+0000","last_became_peered":"2024-01-27T01:06:40.935414+0000","last_unstale":"2024-01-27T01:06:40.940354+0000","last_undegraded":"2024-01-27T01:06:40.940354+0000","last_fullsized":"2024-01-27T01:06:40.940354+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:16:08.266882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.f","version":"222'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.928246+0000","last_change":"2024-01-27T01:06:25.848900+0000","last_active":"2024-01-27T01:06:40.928246+0000","last_peered":"2024-01-27T01:06:40.928246+0000","last_clean":"2024-01-27T01:06:40.928246+0000","last_became_active":"2024-01-27T01:06:25.848483+0000","last_became_peered":"2024-01-27T01:06:25.848483+0000","last_unstale":"2024-01-27T01:06:40.928246+0000","last_undegraded":"2024-01-27T01:06:40.928246+0000","last_fullsized":"2024-01-27T01:06:40.928246+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:27:13.471892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926411+0000","last_change":"2024-01-27T01:06:26.875234+0000","last_active":"2024-01-27T01:06:40.926411+0000","last_peered":"2024-01-27T01:06:40.926411+0000","last_clean":"2024-01-27T01:06:40.926411+0000","last_became_active":"2024-01-27T01:06:26.874927+0000","last_became_peered":"2024-01-27T01:06:26.874927+0000","last_unstale":"2024-01-27T01:06:40.926411+0000","last_undegraded":"2024-01-27T01:06:40.926411+0000","last_fullsized":"2024-01-27T01:06:40.926411+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:10.390573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.918887+0000","last_change":"2024-01-27T01:06:39.931922+0000","last_active":"2024-01-27T01:06:40.918887+0000","last_peered":"2024-01-27T01:06:40.918887+0000","last_clean":"2024-01-27T01:06:40.918887+0000","last_became_active":"2024-01-27T01:06:39.931611+0000","last_became_peered":"2024-01-27T01:06:39.931611+0000","last_unstale":"2024-01-27T01:06:40.918887+0000","last_undegraded":"2024-01-27T01:06:40.918887+0000","last_fullsized":"2024-01-27T01:06:40.918887+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:40.845691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.945315+0000","last_change":"2024-01-27T01:06:40.941775+0000","last_active":"2024-01-27T01:06:40.945315+0000","last_peered":"2024-01-27T01:06:40.945315+0000","last_clean":"2024-01-27T01:06:40.945315+0000","last_became_active":"2024-01-27T01:06:40.941438+0000","last_became_peered":"2024-01-27T01:06:40.941438+0000","last_unstale":"2024-01-27T01:06:40.945315+0000","last_undegraded":"2024-01-27T01:06:40.945315+0000","last_fullsized":"2024-01-27T01:06:40.945315+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:50:46.505356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.10","version":"222'8","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.938162+0000","last_change":"2024-01-27T01:06:25.865239+0000","last_active":"2024-01-27T01:06:40.938162+0000","last_peered":"2024-01-27T01:06:40.938162+0000","last_clean":"2024-01-27T01:06:40.938162+0000","last_became_active":"2024-01-27T01:06:25.864943+0000","last_became_peered":"2024-01-27T01:06:25.864943+0000","last_unstale":"2024-01-27T01:06:40.938162+0000","last_undegraded":"2024-01-27T01:06:40.938162+0000","last_fullsized":"2024-01-27T01:06:40.938162+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:41:25.254751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.932277+0000","last_change":"2024-01-27T01:06:26.867077+0000","last_active":"2024-01-27T01:06:40.932277+0000","last_peered":"2024-01-27T01:06:40.932277+0000","last_clean":"2024-01-27T01:06:40.932277+0000","last_became_active":"2024-01-27T01:06:26.866787+0000","last_became_peered":"2024-01-27T01:06:26.866787+0000","last_unstale":"2024-01-27T01:06:40.932277+0000","last_undegraded":"2024-01-27T01:06:40.932277+0000","last_fullsized":"2024-01-27T01:06:40.932277+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:02.000909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931562+0000","last_change":"2024-01-27T01:06:39.936310+0000","last_active":"2024-01-27T01:06:40.931562+0000","last_peered":"2024-01-27T01:06:40.931562+0000","last_clean":"2024-01-27T01:06:40.931562+0000","last_became_active":"2024-01-27T01:06:39.936032+0000","last_became_peered":"2024-01-27T01:06:39.936032+0000","last_unstale":"2024-01-27T01:06:40.931562+0000","last_undegraded":"2024-01-27T01:06:40.931562+0000","last_fullsized":"2024-01-27T01:06:40.931562+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:55.832690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.948759+0000","last_change":"2024-01-27T01:06:40.941007+0000","last_active":"2024-01-27T01:06:40.948759+0000","last_peered":"2024-01-27T01:06:40.948759+0000","last_clean":"2024-01-27T01:06:40.948759+0000","last_became_active":"2024-01-27T01:06:40.940379+0000","last_became_peered":"2024-01-27T01:06:40.940379+0000","last_unstale":"2024-01-27T01:06:40.948759+0000","last_undegraded":"2024-01-27T01:06:40.948759+0000","last_fullsized":"2024-01-27T01:06:40.948759+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:14:19.010092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.922797+0000","last_change":"2024-01-27T01:06:25.853597+0000","last_active":"2024-01-27T01:06:40.922797+0000","last_peered":"2024-01-27T01:06:40.922797+0000","last_clean":"2024-01-27T01:06:40.922797+0000","last_became_active":"2024-01-27T01:06:25.853257+0000","last_became_peered":"2024-01-27T01:06:25.853257+0000","last_unstale":"2024-01-27T01:06:40.922797+0000","last_undegraded":"2024-01-27T01:06:40.922797+0000","last_fullsized":"2024-01-27T01:06:40.922797+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:54:35.011796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925065+0000","last_change":"2024-01-27T01:06:26.866613+0000","last_active":"2024-01-27T01:06:40.925065+0000","last_peered":"2024-01-27T01:06:40.925065+0000","last_clean":"2024-01-27T01:06:40.925065+0000","last_became_active":"2024-01-27T01:06:26.866298+0000","last_became_peered":"2024-01-27T01:06:26.866298+0000","last_unstale":"2024-01-27T01:06:40.925065+0000","last_undegraded":"2024-01-27T01:06:40.925065+0000","last_fullsized":"2024-01-27T01:06:40.925065+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:27:03.934654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.932185+0000","last_change":"2024-01-27T01:06:39.938479+0000","last_active":"2024-01-27T01:06:40.932185+0000","last_peered":"2024-01-27T01:06:40.932185+0000","last_clean":"2024-01-27T01:06:40.932185+0000","last_became_active":"2024-01-27T01:06:39.938172+0000","last_became_peered":"2024-01-27T01:06:39.938172+0000","last_unstale":"2024-01-27T01:06:40.932185+0000","last_undegraded":"2024-01-27T01:06:40.932185+0000","last_fullsized":"2024-01-27T01:06:40.932185+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:43:07.431223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.948795+0000","last_change":"2024-01-27T01:06:40.941899+0000","last_active":"2024-01-27T01:06:40.948795+0000","last_peered":"2024-01-27T01:06:40.948795+0000","last_clean":"2024-01-27T01:06:40.948795+0000","last_became_active":"2024-01-27T01:06:40.941618+0000","last_became_peered":"2024-01-27T01:06:40.941618+0000","last_unstale":"2024-01-27T01:06:40.948795+0000","last_undegraded":"2024-01-27T01:06:40.948795+0000","last_fullsized":"2024-01-27T01:06:40.948795+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:09:11.010297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919701+0000","last_change":"2024-01-27T01:06:25.854192+0000","last_active":"2024-01-27T01:06:40.919701+0000","last_peered":"2024-01-27T01:06:40.919701+0000","last_clean":"2024-01-27T01:06:40.919701+0000","last_became_active":"2024-01-27T01:06:25.853601+0000","last_became_peered":"2024-01-27T01:06:25.853601+0000","last_unstale":"2024-01-27T01:06:40.919701+0000","last_undegraded":"2024-01-27T01:06:40.919701+0000","last_fullsized":"2024-01-27T01:06:40.919701+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:42.218792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919249+0000","last_change":"2024-01-27T01:06:26.862905+0000","last_active":"2024-01-27T01:06:40.919249+0000","last_peered":"2024-01-27T01:06:40.919249+0000","last_clean":"2024-01-27T01:06:40.919249+0000","last_became_active":"2024-01-27T01:06:26.862584+0000","last_became_peered":"2024-01-27T01:06:26.862584+0000","last_unstale":"2024-01-27T01:06:40.919249+0000","last_undegraded":"2024-01-27T01:06:40.919249+0000","last_fullsized":"2024-01-27T01:06:40.919249+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:27:14.653781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"226'6","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.956444+0000","last_change":"2024-01-27T01:06:39.940319+0000","last_active":"2024-01-27T01:06:40.956444+0000","last_peered":"2024-01-27T01:06:40.956444+0000","last_clean":"2024-01-27T01:06:40.956444+0000","last_became_active":"2024-01-27T01:06:39.940015+0000","last_became_peered":"2024-01-27T01:06:39.940015+0000","last_unstale":"2024-01-27T01:06:40.956444+0000","last_undegraded":"2024-01-27T01:06:40.956444+0000","last_fullsized":"2024-01-27T01:06:40.956444+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:05:49.945911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.951066+0000","last_change":"2024-01-27T01:06:40.945965+0000","last_active":"2024-01-27T01:06:40.951066+0000","last_peered":"2024-01-27T01:06:40.951066+0000","last_clean":"2024-01-27T01:06:40.951066+0000","last_became_active":"2024-01-27T01:06:40.945646+0000","last_became_peered":"2024-01-27T01:06:40.945646+0000","last_unstale":"2024-01-27T01:06:40.951066+0000","last_undegraded":"2024-01-27T01:06:40.951066+0000","last_fullsized":"2024-01-27T01:06:40.951066+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:05:50.951996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"222'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.930339+0000","last_change":"2024-01-27T01:06:25.850061+0000","last_active":"2024-01-27T01:06:40.930339+0000","last_peered":"2024-01-27T01:06:40.930339+0000","last_clean":"2024-01-27T01:06:40.930339+0000","last_became_active":"2024-01-27T01:06:25.849763+0000","last_became_peered":"2024-01-27T01:06:25.849763+0000","last_unstale":"2024-01-27T01:06:40.930339+0000","last_undegraded":"2024-01-27T01:06:40.930339+0000","last_fullsized":"2024-01-27T01:06:40.930339+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:09:56.243562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931751+0000","last_change":"2024-01-27T01:06:26.873464+0000","last_active":"2024-01-27T01:06:40.931751+0000","last_peered":"2024-01-27T01:06:40.931751+0000","last_clean":"2024-01-27T01:06:40.931751+0000","last_became_active":"2024-01-27T01:06:26.873063+0000","last_became_peered":"2024-01-27T01:06:26.873063+0000","last_unstale":"2024-01-27T01:06:40.931751+0000","last_undegraded":"2024-01-27T01:06:40.931751+0000","last_fullsized":"2024-01-27T01:06:40.931751+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:33:32.535574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"226'3","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.952973+0000","last_change":"2024-01-27T01:06:39.940058+0000","last_active":"2024-01-27T01:06:40.952973+0000","last_peered":"2024-01-27T01:06:40.952973+0000","last_clean":"2024-01-27T01:06:40.952973+0000","last_became_active":"2024-01-27T01:06:39.939776+0000","last_became_peered":"2024-01-27T01:06:39.939776+0000","last_unstale":"2024-01-27T01:06:40.952973+0000","last_undegraded":"2024-01-27T01:06:40.952973+0000","last_fullsized":"2024-01-27T01:06:40.952973+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:15:24.362081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.938133+0000","last_change":"2024-01-27T01:06:40.935256+0000","last_active":"2024-01-27T01:06:40.938133+0000","last_peered":"2024-01-27T01:06:40.938133+0000","last_clean":"2024-01-27T01:06:40.938133+0000","last_became_active":"2024-01-27T01:06:40.934889+0000","last_became_peered":"2024-01-27T01:06:40.934889+0000","last_unstale":"2024-01-27T01:06:40.938133+0000","last_undegraded":"2024-01-27T01:06:40.938133+0000","last_fullsized":"2024-01-27T01:06:40.938133+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:42:17.119687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"223'8","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931618+0000","last_change":"2024-01-27T01:06:25.864160+0000","last_active":"2024-01-27T01:06:40.931618+0000","last_peered":"2024-01-27T01:06:40.931618+0000","last_clean":"2024-01-27T01:06:40.931618+0000","last_became_active":"2024-01-27T01:06:25.863867+0000","last_became_peered":"2024-01-27T01:06:25.863867+0000","last_unstale":"2024-01-27T01:06:40.931618+0000","last_undegraded":"2024-01-27T01:06:40.931618+0000","last_fullsized":"2024-01-27T01:06:40.931618+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:43.338484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2768,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.941328+0000","last_change":"2024-01-27T01:06:26.879652+0000","last_active":"2024-01-27T01:06:40.941328+0000","last_peered":"2024-01-27T01:06:40.941328+0000","last_clean":"2024-01-27T01:06:40.941328+0000","last_became_active":"2024-01-27T01:06:26.879271+0000","last_became_peered":"2024-01-27T01:06:26.879271+0000","last_unstale":"2024-01-27T01:06:40.941328+0000","last_undegraded":"2024-01-27T01:06:40.941328+0000","last_fullsized":"2024-01-27T01:06:40.941328+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:57:23.456731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923013+0000","last_change":"2024-01-27T01:06:39.936805+0000","last_active":"2024-01-27T01:06:40.923013+0000","last_peered":"2024-01-27T01:06:40.923013+0000","last_clean":"2024-01-27T01:06:40.923013+0000","last_became_active":"2024-01-27T01:06:39.936499+0000","last_became_peered":"2024-01-27T01:06:39.936499+0000","last_unstale":"2024-01-27T01:06:40.923013+0000","last_undegraded":"2024-01-27T01:06:40.923013+0000","last_fullsized":"2024-01-27T01:06:40.923013+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:16:07.260797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.938022+0000","last_change":"2024-01-27T01:06:40.935378+0000","last_active":"2024-01-27T01:06:40.938022+0000","last_peered":"2024-01-27T01:06:40.938022+0000","last_clean":"2024-01-27T01:06:40.938022+0000","last_became_active":"2024-01-27T01:06:40.934801+0000","last_became_peered":"2024-01-27T01:06:40.934801+0000","last_unstale":"2024-01-27T01:06:40.938022+0000","last_undegraded":"2024-01-27T01:06:40.938022+0000","last_fullsized":"2024-01-27T01:06:40.938022+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:10:08.588321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.15","version":"222'3","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923278+0000","last_change":"2024-01-27T01:06:25.857771+0000","last_active":"2024-01-27T01:06:40.923278+0000","last_peered":"2024-01-27T01:06:40.923278+0000","last_clean":"2024-01-27T01:06:40.923278+0000","last_became_active":"2024-01-27T01:06:25.857445+0000","last_became_peered":"2024-01-27T01:06:25.857445+0000","last_unstale":"2024-01-27T01:06:40.923278+0000","last_undegraded":"2024-01-27T01:06:40.923278+0000","last_fullsized":"2024-01-27T01:06:40.923278+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:52.927866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923155+0000","last_change":"2024-01-27T01:06:26.876685+0000","last_active":"2024-01-27T01:06:40.923155+0000","last_peered":"2024-01-27T01:06:40.923155+0000","last_clean":"2024-01-27T01:06:40.923155+0000","last_became_active":"2024-01-27T01:06:26.872939+0000","last_became_peered":"2024-01-27T01:06:26.872939+0000","last_unstale":"2024-01-27T01:06:40.923155+0000","last_undegraded":"2024-01-27T01:06:40.923155+0000","last_fullsized":"2024-01-27T01:06:40.923155+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:56:48.730726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.955282+0000","last_change":"2024-01-27T01:06:39.941476+0000","last_active":"2024-01-27T01:06:40.955282+0000","last_peered":"2024-01-27T01:06:40.955282+0000","last_clean":"2024-01-27T01:06:40.955282+0000","last_became_active":"2024-01-27T01:06:39.940676+0000","last_became_peered":"2024-01-27T01:06:39.940676+0000","last_unstale":"2024-01-27T01:06:40.955282+0000","last_undegraded":"2024-01-27T01:06:40.955282+0000","last_fullsized":"2024-01-27T01:06:40.955282+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:36:13.180584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.945422+0000","last_change":"2024-01-27T01:06:40.942851+0000","last_active":"2024-01-27T01:06:40.945422+0000","last_peered":"2024-01-27T01:06:40.945422+0000","last_clean":"2024-01-27T01:06:40.945422+0000","last_became_active":"2024-01-27T01:06:40.942563+0000","last_became_peered":"2024-01-27T01:06:40.942563+0000","last_unstale":"2024-01-27T01:06:40.945422+0000","last_undegraded":"2024-01-27T01:06:40.945422+0000","last_fullsized":"2024-01-27T01:06:40.945422+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:24:51.130614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931731+0000","last_change":"2024-01-27T01:06:25.860762+0000","last_active":"2024-01-27T01:06:40.931731+0000","last_peered":"2024-01-27T01:06:40.931731+0000","last_clean":"2024-01-27T01:06:40.931731+0000","last_became_active":"2024-01-27T01:06:25.860459+0000","last_became_peered":"2024-01-27T01:06:25.860459+0000","last_unstale":"2024-01-27T01:06:40.931731+0000","last_undegraded":"2024-01-27T01:06:40.931731+0000","last_fullsized":"2024-01-27T01:06:40.931731+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:31.687265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919076+0000","last_change":"2024-01-27T01:06:26.868859+0000","last_active":"2024-01-27T01:06:40.919076+0000","last_peered":"2024-01-27T01:06:40.919076+0000","last_clean":"2024-01-27T01:06:40.919076+0000","last_became_active":"2024-01-27T01:06:26.868623+0000","last_became_peered":"2024-01-27T01:06:26.868623+0000","last_unstale":"2024-01-27T01:06:40.919076+0000","last_undegraded":"2024-01-27T01:06:40.919076+0000","last_fullsized":"2024-01-27T01:06:40.919076+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:18:47.232067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"226'8","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.955575+0000","last_change":"2024-01-27T01:06:39.934980+0000","last_active":"2024-01-27T01:06:40.955575+0000","last_peered":"2024-01-27T01:06:40.955575+0000","last_clean":"2024-01-27T01:06:40.955575+0000","last_became_active":"2024-01-27T01:06:39.934711+0000","last_became_peered":"2024-01-27T01:06:39.934711+0000","last_unstale":"2024-01-27T01:06:40.955575+0000","last_undegraded":"2024-01-27T01:06:40.955575+0000","last_fullsized":"2024-01-27T01:06:40.955575+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:15:24.362081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.924980+0000","last_change":"2024-01-27T01:06:40.924260+0000","last_active":"2024-01-27T01:06:40.924980+0000","last_peered":"2024-01-27T01:06:40.924980+0000","last_clean":"2024-01-27T01:06:40.924980+0000","last_became_active":"2024-01-27T01:06:40.923949+0000","last_became_peered":"2024-01-27T01:06:40.923949+0000","last_unstale":"2024-01-27T01:06:40.924980+0000","last_undegraded":"2024-01-27T01:06:40.924980+0000","last_fullsized":"2024-01-27T01:06:40.924980+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:18.204468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942513+0000","last_change":"2024-01-27T01:06:25.865314+0000","last_active":"2024-01-27T01:06:40.942513+0000","last_peered":"2024-01-27T01:06:40.942513+0000","last_clean":"2024-01-27T01:06:40.942513+0000","last_became_active":"2024-01-27T01:06:25.864940+0000","last_became_peered":"2024-01-27T01:06:25.864940+0000","last_unstale":"2024-01-27T01:06:40.942513+0000","last_undegraded":"2024-01-27T01:06:40.942513+0000","last_fullsized":"2024-01-27T01:06:40.942513+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:24.113986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931333+0000","last_change":"2024-01-27T01:06:26.870806+0000","last_active":"2024-01-27T01:06:40.931333+0000","last_peered":"2024-01-27T01:06:40.931333+0000","last_clean":"2024-01-27T01:06:40.931333+0000","last_became_active":"2024-01-27T01:06:26.870263+0000","last_became_peered":"2024-01-27T01:06:26.870263+0000","last_unstale":"2024-01-27T01:06:40.931333+0000","last_undegraded":"2024-01-27T01:06:40.931333+0000","last_fullsized":"2024-01-27T01:06:40.931333+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:26.214953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923552+0000","last_change":"2024-01-27T01:06:39.934885+0000","last_active":"2024-01-27T01:06:40.923552+0000","last_peered":"2024-01-27T01:06:40.923552+0000","last_clean":"2024-01-27T01:06:40.923552+0000","last_became_active":"2024-01-27T01:06:39.934619+0000","last_became_peered":"2024-01-27T01:06:39.934619+0000","last_unstale":"2024-01-27T01:06:40.923552+0000","last_undegraded":"2024-01-27T01:06:40.923552+0000","last_fullsized":"2024-01-27T01:06:40.923552+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:43:02.008737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942725+0000","last_change":"2024-01-27T01:06:40.937822+0000","last_active":"2024-01-27T01:06:40.942725+0000","last_peered":"2024-01-27T01:06:40.942725+0000","last_clean":"2024-01-27T01:06:40.942725+0000","last_became_active":"2024-01-27T01:06:40.937489+0000","last_became_peered":"2024-01-27T01:06:40.937489+0000","last_unstale":"2024-01-27T01:06:40.942725+0000","last_undegraded":"2024-01-27T01:06:40.942725+0000","last_fullsized":"2024-01-27T01:06:40.942725+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:03:13.414840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.918934+0000","last_change":"2024-01-27T01:06:25.851888+0000","last_active":"2024-01-27T01:06:40.918934+0000","last_peered":"2024-01-27T01:06:40.918934+0000","last_clean":"2024-01-27T01:06:40.918934+0000","last_became_active":"2024-01-27T01:06:25.851605+0000","last_became_peered":"2024-01-27T01:06:25.851605+0000","last_unstale":"2024-01-27T01:06:40.918934+0000","last_undegraded":"2024-01-27T01:06:40.918934+0000","last_fullsized":"2024-01-27T01:06:40.918934+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:09.386094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925489+0000","last_change":"2024-01-27T01:06:26.873199+0000","last_active":"2024-01-27T01:06:40.925489+0000","last_peered":"2024-01-27T01:06:40.925489+0000","last_clean":"2024-01-27T01:06:40.925489+0000","last_became_active":"2024-01-27T01:06:26.872658+0000","last_became_peered":"2024-01-27T01:06:26.872658+0000","last_unstale":"2024-01-27T01:06:40.925489+0000","last_undegraded":"2024-01-27T01:06:40.925489+0000","last_fullsized":"2024-01-27T01:06:40.925489+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:06:45.526347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.951919+0000","last_change":"2024-01-27T01:06:39.936324+0000","last_active":"2024-01-27T01:06:40.951919+0000","last_peered":"2024-01-27T01:06:40.951919+0000","last_clean":"2024-01-27T01:06:40.951919+0000","last_became_active":"2024-01-27T01:06:39.935917+0000","last_became_peered":"2024-01-27T01:06:39.935917+0000","last_unstale":"2024-01-27T01:06:40.951919+0000","last_undegraded":"2024-01-27T01:06:40.951919+0000","last_fullsized":"2024-01-27T01:06:40.951919+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:43:07.431223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.946155+0000","last_change":"2024-01-27T01:06:40.937572+0000","last_active":"2024-01-27T01:06:40.946155+0000","last_peered":"2024-01-27T01:06:40.946155+0000","last_clean":"2024-01-27T01:06:40.946155+0000","last_became_active":"2024-01-27T01:06:40.937143+0000","last_became_peered":"2024-01-27T01:06:40.937143+0000","last_unstale":"2024-01-27T01:06:40.946155+0000","last_undegraded":"2024-01-27T01:06:40.946155+0000","last_fullsized":"2024-01-27T01:06:40.946155+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:36:14.604522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.933332+0000","last_change":"2024-01-27T01:06:25.854878+0000","last_active":"2024-01-27T01:06:40.933332+0000","last_peered":"2024-01-27T01:06:40.933332+0000","last_clean":"2024-01-27T01:06:40.933332+0000","last_became_active":"2024-01-27T01:06:25.854623+0000","last_became_peered":"2024-01-27T01:06:25.854623+0000","last_unstale":"2024-01-27T01:06:40.933332+0000","last_undegraded":"2024-01-27T01:06:40.933332+0000","last_fullsized":"2024-01-27T01:06:40.933332+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:21.007449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931642+0000","last_change":"2024-01-27T01:06:26.876963+0000","last_active":"2024-01-27T01:06:40.931642+0000","last_peered":"2024-01-27T01:06:40.931642+0000","last_clean":"2024-01-27T01:06:40.931642+0000","last_became_active":"2024-01-27T01:06:26.876709+0000","last_became_peered":"2024-01-27T01:06:26.876709+0000","last_unstale":"2024-01-27T01:06:40.931642+0000","last_undegraded":"2024-01-27T01:06:40.931642+0000","last_fullsized":"2024-01-27T01:06:40.931642+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:58:14.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"226'4","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.954938+0000","last_change":"2024-01-27T01:06:39.938669+0000","last_active":"2024-01-27T01:06:40.954938+0000","last_peered":"2024-01-27T01:06:40.954938+0000","last_clean":"2024-01-27T01:06:40.954938+0000","last_became_active":"2024-01-27T01:06:39.938238+0000","last_became_peered":"2024-01-27T01:06:39.938238+0000","last_unstale":"2024-01-27T01:06:40.954938+0000","last_undegraded":"2024-01-27T01:06:40.954938+0000","last_fullsized":"2024-01-27T01:06:40.954938+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:33:16.818100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.953919+0000","last_change":"2024-01-27T01:06:40.945961+0000","last_active":"2024-01-27T01:06:40.953919+0000","last_peered":"2024-01-27T01:06:40.953919+0000","last_clean":"2024-01-27T01:06:40.953919+0000","last_became_active":"2024-01-27T01:06:40.945643+0000","last_became_peered":"2024-01-27T01:06:40.945643+0000","last_unstale":"2024-01-27T01:06:40.953919+0000","last_undegraded":"2024-01-27T01:06:40.953919+0000","last_fullsized":"2024-01-27T01:06:40.953919+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:23:56.549640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1b","version":"222'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931590+0000","last_change":"2024-01-27T01:06:25.854193+0000","last_active":"2024-01-27T01:06:40.931590+0000","last_peered":"2024-01-27T01:06:40.931590+0000","last_clean":"2024-01-27T01:06:40.931590+0000","last_became_active":"2024-01-27T01:06:25.853956+0000","last_became_peered":"2024-01-27T01:06:25.853956+0000","last_unstale":"2024-01-27T01:06:40.931590+0000","last_undegraded":"2024-01-27T01:06:40.931590+0000","last_fullsized":"2024-01-27T01:06:40.931590+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:33:02.747087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931528+0000","last_change":"2024-01-27T01:06:26.866121+0000","last_active":"2024-01-27T01:06:40.931528+0000","last_peered":"2024-01-27T01:06:40.931528+0000","last_clean":"2024-01-27T01:06:40.931528+0000","last_became_active":"2024-01-27T01:06:26.865391+0000","last_became_peered":"2024-01-27T01:06:26.865391+0000","last_unstale":"2024-01-27T01:06:40.931528+0000","last_undegraded":"2024-01-27T01:06:40.931528+0000","last_fullsized":"2024-01-27T01:06:40.931528+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:02:03.812030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.953895+0000","last_change":"2024-01-27T01:06:39.938268+0000","last_active":"2024-01-27T01:06:40.953895+0000","last_peered":"2024-01-27T01:06:40.953895+0000","last_clean":"2024-01-27T01:06:40.953895+0000","last_became_active":"2024-01-27T01:06:39.937991+0000","last_became_peered":"2024-01-27T01:06:39.937991+0000","last_unstale":"2024-01-27T01:06:40.953895+0000","last_undegraded":"2024-01-27T01:06:40.953895+0000","last_fullsized":"2024-01-27T01:06:40.953895+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:51:00.031893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.944610+0000","last_change":"2024-01-27T01:06:40.937290+0000","last_active":"2024-01-27T01:06:40.944610+0000","last_peered":"2024-01-27T01:06:40.944610+0000","last_clean":"2024-01-27T01:06:40.944610+0000","last_became_active":"2024-01-27T01:06:40.936962+0000","last_became_peered":"2024-01-27T01:06:40.936962+0000","last_unstale":"2024-01-27T01:06:40.944610+0000","last_undegraded":"2024-01-27T01:06:40.944610+0000","last_fullsized":"2024-01-27T01:06:40.944610+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:31:51.709336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1c","version":"222'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923515+0000","last_change":"2024-01-27T01:06:25.855114+0000","last_active":"2024-01-27T01:06:40.923515+0000","last_peered":"2024-01-27T01:06:40.923515+0000","last_clean":"2024-01-27T01:06:40.923515+0000","last_became_active":"2024-01-27T01:06:25.854790+0000","last_became_peered":"2024-01-27T01:06:25.854790+0000","last_unstale":"2024-01-27T01:06:40.923515+0000","last_undegraded":"2024-01-27T01:06:40.923515+0000","last_fullsized":"2024-01-27T01:06:40.923515+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:58:58.498377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.941528+0000","last_change":"2024-01-27T01:06:26.869128+0000","last_active":"2024-01-27T01:06:40.941528+0000","last_peered":"2024-01-27T01:06:40.941528+0000","last_clean":"2024-01-27T01:06:40.941528+0000","last_became_active":"2024-01-27T01:06:26.868892+0000","last_became_peered":"2024-01-27T01:06:26.868892+0000","last_unstale":"2024-01-27T01:06:40.941528+0000","last_undegraded":"2024-01-27T01:06:40.941528+0000","last_fullsized":"2024-01-27T01:06:40.941528+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:40:40.869516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.950661+0000","last_change":"2024-01-27T01:06:39.937444+0000","last_active":"2024-01-27T01:06:40.950661+0000","last_peered":"2024-01-27T01:06:40.950661+0000","last_clean":"2024-01-27T01:06:40.950661+0000","last_became_active":"2024-01-27T01:06:39.937157+0000","last_became_peered":"2024-01-27T01:06:39.937157+0000","last_unstale":"2024-01-27T01:06:40.950661+0000","last_undegraded":"2024-01-27T01:06:40.950661+0000","last_fullsized":"2024-01-27T01:06:40.950661+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:13:12.942303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.944137+0000","last_change":"2024-01-27T01:06:40.939552+0000","last_active":"2024-01-27T01:06:40.944137+0000","last_peered":"2024-01-27T01:06:40.944137+0000","last_clean":"2024-01-27T01:06:40.944137+0000","last_became_active":"2024-01-27T01:06:40.939300+0000","last_became_peered":"2024-01-27T01:06:40.939300+0000","last_unstale":"2024-01-27T01:06:40.944137+0000","last_undegraded":"2024-01-27T01:06:40.944137+0000","last_fullsized":"2024-01-27T01:06:40.944137+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:06:59.598967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"222'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931303+0000","last_change":"2024-01-27T01:06:25.855991+0000","last_active":"2024-01-27T01:06:40.931303+0000","last_peered":"2024-01-27T01:06:40.931303+0000","last_clean":"2024-01-27T01:06:40.931303+0000","last_became_active":"2024-01-27T01:06:25.855679+0000","last_became_peered":"2024-01-27T01:06:25.855679+0000","last_unstale":"2024-01-27T01:06:40.931303+0000","last_undegraded":"2024-01-27T01:06:40.931303+0000","last_fullsized":"2024-01-27T01:06:40.931303+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:31:04.295025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926141+0000","last_change":"2024-01-27T01:06:26.875831+0000","last_active":"2024-01-27T01:06:40.926141+0000","last_peered":"2024-01-27T01:06:40.926141+0000","last_clean":"2024-01-27T01:06:40.926141+0000","last_became_active":"2024-01-27T01:06:26.875488+0000","last_became_peered":"2024-01-27T01:06:26.875488+0000","last_unstale":"2024-01-27T01:06:40.926141+0000","last_undegraded":"2024-01-27T01:06:40.926141+0000","last_fullsized":"2024-01-27T01:06:40.926141+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:07:51.640059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.951407+0000","last_change":"2024-01-27T01:06:39.938441+0000","last_active":"2024-01-27T01:06:40.951407+0000","last_peered":"2024-01-27T01:06:40.951407+0000","last_clean":"2024-01-27T01:06:40.951407+0000","last_became_active":"2024-01-27T01:06:39.938144+0000","last_became_peered":"2024-01-27T01:06:39.938144+0000","last_unstale":"2024-01-27T01:06:40.951407+0000","last_undegraded":"2024-01-27T01:06:40.951407+0000","last_fullsized":"2024-01-27T01:06:40.951407+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:18:25.975883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.940548+0000","last_change":"2024-01-27T01:06:40.936165+0000","last_active":"2024-01-27T01:06:40.940548+0000","last_peered":"2024-01-27T01:06:40.940548+0000","last_clean":"2024-01-27T01:06:40.940548+0000","last_became_active":"2024-01-27T01:06:40.935054+0000","last_became_peered":"2024-01-27T01:06:40.935054+0000","last_unstale":"2024-01-27T01:06:40.940548+0000","last_undegraded":"2024-01-27T01:06:40.940548+0000","last_fullsized":"2024-01-27T01:06:40.940548+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:50:27.972686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1e","version":"222'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.928525+0000","last_change":"2024-01-27T01:06:25.856504+0000","last_active":"2024-01-27T01:06:40.928525+0000","last_peered":"2024-01-27T01:06:40.928525+0000","last_clean":"2024-01-27T01:06:40.928525+0000","last_became_active":"2024-01-27T01:06:25.856272+0000","last_became_peered":"2024-01-27T01:06:25.856272+0000","last_unstale":"2024-01-27T01:06:40.928525+0000","last_undegraded":"2024-01-27T01:06:40.928525+0000","last_fullsized":"2024-01-27T01:06:40.928525+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:20:56.736428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923432+0000","last_change":"2024-01-27T01:06:26.878625+0000","last_active":"2024-01-27T01:06:40.923432+0000","last_peered":"2024-01-27T01:06:40.923432+0000","last_clean":"2024-01-27T01:06:40.923432+0000","last_became_active":"2024-01-27T01:06:26.878361+0000","last_became_peered":"2024-01-27T01:06:26.878361+0000","last_unstale":"2024-01-27T01:06:40.923432+0000","last_undegraded":"2024-01-27T01:06:40.923432+0000","last_fullsized":"2024-01-27T01:06:40.923432+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:37:06.977689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.930028+0000","last_change":"2024-01-27T01:06:39.932404+0000","last_active":"2024-01-27T01:06:40.930028+0000","last_peered":"2024-01-27T01:06:40.930028+0000","last_clean":"2024-01-27T01:06:40.930028+0000","last_became_active":"2024-01-27T01:06:39.932068+0000","last_became_peered":"2024-01-27T01:06:39.932068+0000","last_unstale":"2024-01-27T01:06:40.930028+0000","last_undegraded":"2024-01-27T01:06:40.930028+0000","last_fullsized":"2024-01-27T01:06:40.930028+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:37:20.881203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.946296+0000","last_change":"2024-01-27T01:06:40.938927+0000","last_active":"2024-01-27T01:06:40.946296+0000","last_peered":"2024-01-27T01:06:40.946296+0000","last_clean":"2024-01-27T01:06:40.946296+0000","last_became_active":"2024-01-27T01:06:40.938546+0000","last_became_peered":"2024-01-27T01:06:40.938546+0000","last_unstale":"2024-01-27T01:06:40.946296+0000","last_undegraded":"2024-01-27T01:06:40.946296+0000","last_fullsized":"2024-01-27T01:06:40.946296+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:57.703450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1f","version":"222'4","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.932836+0000","last_change":"2024-01-27T01:06:25.856458+0000","last_active":"2024-01-27T01:06:40.932836+0000","last_peered":"2024-01-27T01:06:40.932836+0000","last_clean":"2024-01-27T01:06:40.932836+0000","last_became_active":"2024-01-27T01:06:25.856162+0000","last_became_peered":"2024-01-27T01:06:25.856162+0000","last_unstale":"2024-01-27T01:06:40.932836+0000","last_undegraded":"2024-01-27T01:06:40.932836+0000","last_fullsized":"2024-01-27T01:06:40.932836+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:51:44.378880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.930272+0000","last_change":"2024-01-27T01:06:26.869588+0000","last_active":"2024-01-27T01:06:40.930272+0000","last_peered":"2024-01-27T01:06:40.930272+0000","last_clean":"2024-01-27T01:06:40.930272+0000","last_became_active":"2024-01-27T01:06:26.869264+0000","last_became_peered":"2024-01-27T01:06:26.869264+0000","last_unstale":"2024-01-27T01:06:40.930272+0000","last_undegraded":"2024-01-27T01:06:40.930272+0000","last_fullsized":"2024-01-27T01:06:40.930272+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:42:54.364688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.933810+0000","last_change":"2024-01-27T01:06:39.937481+0000","last_active":"2024-01-27T01:06:40.933810+0000","last_peered":"2024-01-27T01:06:40.933810+0000","last_clean":"2024-01-27T01:06:40.933810+0000","last_became_active":"2024-01-27T01:06:39.937225+0000","last_became_peered":"2024-01-27T01:06:39.937225+0000","last_unstale":"2024-01-27T01:06:40.933810+0000","last_undegraded":"2024-01-27T01:06:40.933810+0000","last_fullsized":"2024-01-27T01:06:40.933810+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:45.338439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.933146+0000","last_change":"2024-01-27T01:06:40.929946+0000","last_active":"2024-01-27T01:06:40.933146+0000","last_peered":"2024-01-27T01:06:40.933146+0000","last_clean":"2024-01-27T01:06:40.933146+0000","last_became_active":"2024-01-27T01:06:40.929262+0000","last_became_peered":"2024-01-27T01:06:40.929262+0000","last_unstale":"2024-01-27T01:06:40.933146+0000","last_undegraded":"2024-01-27T01:06:40.933146+0000","last_fullsized":"2024-01-27T01:06:40.933146+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:08:47.665701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.932763+0000","last_change":"2024-01-27T01:06:26.869029+0000","last_active":"2024-01-27T01:06:40.932763+0000","last_peered":"2024-01-27T01:06:40.932763+0000","last_clean":"2024-01-27T01:06:40.932763+0000","last_became_active":"2024-01-27T01:06:26.868793+0000","last_became_peered":"2024-01-27T01:06:26.868793+0000","last_unstale":"2024-01-27T01:06:40.932763+0000","last_undegraded":"2024-01-27T01:06:40.932763+0000","last_fullsized":"2024-01-27T01:06:40.932763+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:10.965469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942063+0000","last_change":"2024-01-27T01:06:40.935281+0000","last_active":"2024-01-27T01:06:40.942063+0000","last_peered":"2024-01-27T01:06:40.942063+0000","last_clean":"2024-01-27T01:06:40.942063+0000","last_became_active":"2024-01-27T01:06:40.935011+0000","last_became_peered":"2024-01-27T01:06:40.935011+0000","last_unstale":"2024-01-27T01:06:40.942063+0000","last_undegraded":"2024-01-27T01:06:40.942063+0000","last_fullsized":"2024-01-27T01:06:40.942063+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:25.572331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.933065+0000","last_change":"2024-01-27T01:06:26.876343+0000","last_active":"2024-01-27T01:06:40.933065+0000","last_peered":"2024-01-27T01:06:40.933065+0000","last_clean":"2024-01-27T01:06:40.933065+0000","last_became_active":"2024-01-27T01:06:26.875570+0000","last_became_peered":"2024-01-27T01:06:26.875570+0000","last_unstale":"2024-01-27T01:06:40.933065+0000","last_undegraded":"2024-01-27T01:06:40.933065+0000","last_fullsized":"2024-01-27T01:06:40.933065+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:07:51.640059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.945810+0000","last_change":"2024-01-27T01:06:40.942956+0000","last_active":"2024-01-27T01:06:40.945810+0000","last_peered":"2024-01-27T01:06:40.945810+0000","last_clean":"2024-01-27T01:06:40.945810+0000","last_became_active":"2024-01-27T01:06:40.942672+0000","last_became_peered":"2024-01-27T01:06:40.942672+0000","last_unstale":"2024-01-27T01:06:40.945810+0000","last_undegraded":"2024-01-27T01:06:40.945810+0000","last_fullsized":"2024-01-27T01:06:40.945810+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:33:33.988432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.933140+0000","last_change":"2024-01-27T01:06:26.871676+0000","last_active":"2024-01-27T01:06:40.933140+0000","last_peered":"2024-01-27T01:06:40.933140+0000","last_clean":"2024-01-27T01:06:40.933140+0000","last_became_active":"2024-01-27T01:06:26.871423+0000","last_became_peered":"2024-01-27T01:06:26.871423+0000","last_unstale":"2024-01-27T01:06:40.933140+0000","last_undegraded":"2024-01-27T01:06:40.933140+0000","last_fullsized":"2024-01-27T01:06:40.933140+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:29:58.146876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.944066+0000","last_change":"2024-01-27T01:06:40.942844+0000","last_active":"2024-01-27T01:06:40.944066+0000","last_peered":"2024-01-27T01:06:40.944066+0000","last_clean":"2024-01-27T01:06:40.944066+0000","last_became_active":"2024-01-27T01:06:40.942392+0000","last_became_peered":"2024-01-27T01:06:40.942392+0000","last_unstale":"2024-01-27T01:06:40.944066+0000","last_undegraded":"2024-01-27T01:06:40.944066+0000","last_fullsized":"2024-01-27T01:06:40.944066+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:57:08.256445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923341+0000","last_change":"2024-01-27T01:06:26.872013+0000","last_active":"2024-01-27T01:06:40.923341+0000","last_peered":"2024-01-27T01:06:40.923341+0000","last_clean":"2024-01-27T01:06:40.923341+0000","last_became_active":"2024-01-27T01:06:26.867577+0000","last_became_peered":"2024-01-27T01:06:26.867577+0000","last_unstale":"2024-01-27T01:06:40.923341+0000","last_undegraded":"2024-01-27T01:06:40.923341+0000","last_fullsized":"2024-01-27T01:06:40.923341+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:27.604574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.947745+0000","last_change":"2024-01-27T01:06:40.944023+0000","last_active":"2024-01-27T01:06:40.947745+0000","last_peered":"2024-01-27T01:06:40.947745+0000","last_clean":"2024-01-27T01:06:40.947745+0000","last_became_active":"2024-01-27T01:06:40.943733+0000","last_became_peered":"2024-01-27T01:06:40.943733+0000","last_unstale":"2024-01-27T01:06:40.947745+0000","last_undegraded":"2024-01-27T01:06:40.947745+0000","last_fullsized":"2024-01-27T01:06:40.947745+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.073528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.943369+0000","last_change":"2024-01-27T01:06:40.938941+0000","last_active":"2024-01-27T01:06:40.943369+0000","last_peered":"2024-01-27T01:06:40.943369+0000","last_clean":"2024-01-27T01:06:40.943369+0000","last_became_active":"2024-01-27T01:06:40.938667+0000","last_became_peered":"2024-01-27T01:06:40.938667+0000","last_unstale":"2024-01-27T01:06:40.943369+0000","last_undegraded":"2024-01-27T01:06:40.943369+0000","last_fullsized":"2024-01-27T01:06:40.943369+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:52:37.031485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.941082+0000","last_change":"2024-01-27T01:06:26.870649+0000","last_active":"2024-01-27T01:06:40.941082+0000","last_peered":"2024-01-27T01:06:40.941082+0000","last_clean":"2024-01-27T01:06:40.941082+0000","last_became_active":"2024-01-27T01:06:26.870433+0000","last_became_peered":"2024-01-27T01:06:26.870433+0000","last_unstale":"2024-01-27T01:06:40.941082+0000","last_undegraded":"2024-01-27T01:06:40.941082+0000","last_fullsized":"2024-01-27T01:06:40.941082+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:11:27.402943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.951687+0000","last_change":"2024-01-27T01:06:40.948958+0000","last_active":"2024-01-27T01:06:40.951687+0000","last_peered":"2024-01-27T01:06:40.951687+0000","last_clean":"2024-01-27T01:06:40.951687+0000","last_became_active":"2024-01-27T01:06:40.948336+0000","last_became_peered":"2024-01-27T01:06:40.948336+0000","last_unstale":"2024-01-27T01:06:40.951687+0000","last_undegraded":"2024-01-27T01:06:40.951687+0000","last_fullsized":"2024-01-27T01:06:40.951687+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:10:08.588321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919157+0000","last_change":"2024-01-27T01:06:26.878202+0000","last_active":"2024-01-27T01:06:40.919157+0000","last_peered":"2024-01-27T01:06:40.919157+0000","last_clean":"2024-01-27T01:06:40.919157+0000","last_became_active":"2024-01-27T01:06:26.877900+0000","last_became_peered":"2024-01-27T01:06:26.877900+0000","last_unstale":"2024-01-27T01:06:40.919157+0000","last_undegraded":"2024-01-27T01:06:40.919157+0000","last_fullsized":"2024-01-27T01:06:40.919157+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:42.766155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.928440+0000","last_change":"2024-01-27T01:06:26.878911+0000","last_active":"2024-01-27T01:06:40.928440+0000","last_peered":"2024-01-27T01:06:40.928440+0000","last_clean":"2024-01-27T01:06:40.928440+0000","last_became_active":"2024-01-27T01:06:26.878637+0000","last_became_peered":"2024-01-27T01:06:26.878637+0000","last_unstale":"2024-01-27T01:06:40.928440+0000","last_undegraded":"2024-01-27T01:06:40.928440+0000","last_fullsized":"2024-01-27T01:06:40.928440+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:33:03.751565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.944263+0000","last_change":"2024-01-27T01:06:40.941505+0000","last_active":"2024-01-27T01:06:40.944263+0000","last_peered":"2024-01-27T01:06:40.944263+0000","last_clean":"2024-01-27T01:06:40.944263+0000","last_became_active":"2024-01-27T01:06:40.940953+0000","last_became_peered":"2024-01-27T01:06:40.940953+0000","last_unstale":"2024-01-27T01:06:40.944263+0000","last_undegraded":"2024-01-27T01:06:40.944263+0000","last_fullsized":"2024-01-27T01:06:40.944263+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:47:31.151874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.949826+0000","last_change":"2024-01-27T01:06:40.942124+0000","last_active":"2024-01-27T01:06:40.949826+0000","last_peered":"2024-01-27T01:06:40.949826+0000","last_clean":"2024-01-27T01:06:40.949826+0000","last_became_active":"2024-01-27T01:06:40.941857+0000","last_became_peered":"2024-01-27T01:06:40.941857+0000","last_unstale":"2024-01-27T01:06:40.949826+0000","last_undegraded":"2024-01-27T01:06:40.949826+0000","last_fullsized":"2024-01-27T01:06:40.949826+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:21:06.828538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925446+0000","last_change":"2024-01-27T01:06:26.865149+0000","last_active":"2024-01-27T01:06:40.925446+0000","last_peered":"2024-01-27T01:06:40.925446+0000","last_clean":"2024-01-27T01:06:40.925446+0000","last_became_active":"2024-01-27T01:06:26.864853+0000","last_became_peered":"2024-01-27T01:06:26.864853+0000","last_unstale":"2024-01-27T01:06:40.925446+0000","last_undegraded":"2024-01-27T01:06:40.925446+0000","last_fullsized":"2024-01-27T01:06:40.925446+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:10:44.194323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.936461+0000","last_change":"2024-01-27T01:06:40.930136+0000","last_active":"2024-01-27T01:06:40.936461+0000","last_peered":"2024-01-27T01:06:40.936461+0000","last_clean":"2024-01-27T01:06:40.936461+0000","last_became_active":"2024-01-27T01:06:40.929659+0000","last_became_peered":"2024-01-27T01:06:40.929659+0000","last_unstale":"2024-01-27T01:06:40.936461+0000","last_undegraded":"2024-01-27T01:06:40.936461+0000","last_fullsized":"2024-01-27T01:06:40.936461+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:38:42.855259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.938062+0000","last_change":"2024-01-27T01:06:26.867241+0000","last_active":"2024-01-27T01:06:40.938062+0000","last_peered":"2024-01-27T01:06:40.938062+0000","last_clean":"2024-01-27T01:06:40.938062+0000","last_became_active":"2024-01-27T01:06:26.866917+0000","last_became_peered":"2024-01-27T01:06:26.866917+0000","last_unstale":"2024-01-27T01:06:40.938062+0000","last_undegraded":"2024-01-27T01:06:40.938062+0000","last_fullsized":"2024-01-27T01:06:40.938062+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:24:25.818592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.954534+0000","last_change":"2024-01-27T01:06:40.950563+0000","last_active":"2024-01-27T01:06:40.954534+0000","last_peered":"2024-01-27T01:06:40.954534+0000","last_clean":"2024-01-27T01:06:40.954534+0000","last_became_active":"2024-01-27T01:06:40.950227+0000","last_became_peered":"2024-01-27T01:06:40.950227+0000","last_unstale":"2024-01-27T01:06:40.954534+0000","last_undegraded":"2024-01-27T01:06:40.954534+0000","last_fullsized":"2024-01-27T01:06:40.954534+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:18:06.269258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.930535+0000","last_change":"2024-01-27T01:06:26.874471+0000","last_active":"2024-01-27T01:06:40.930535+0000","last_peered":"2024-01-27T01:06:40.930535+0000","last_clean":"2024-01-27T01:06:40.930535+0000","last_became_active":"2024-01-27T01:06:26.874271+0000","last_became_peered":"2024-01-27T01:06:26.874271+0000","last_unstale":"2024-01-27T01:06:40.930535+0000","last_undegraded":"2024-01-27T01:06:40.930535+0000","last_fullsized":"2024-01-27T01:06:40.930535+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:15:11.295547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.951309+0000","last_change":"2024-01-27T01:06:40.948707+0000","last_active":"2024-01-27T01:06:40.951309+0000","last_peered":"2024-01-27T01:06:40.951309+0000","last_clean":"2024-01-27T01:06:40.951309+0000","last_became_active":"2024-01-27T01:06:40.948426+0000","last_became_peered":"2024-01-27T01:06:40.948426+0000","last_unstale":"2024-01-27T01:06:40.951309+0000","last_undegraded":"2024-01-27T01:06:40.951309+0000","last_fullsized":"2024-01-27T01:06:40.951309+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:39:27.523918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926235+0000","last_change":"2024-01-27T01:06:26.876390+0000","last_active":"2024-01-27T01:06:40.926235+0000","last_peered":"2024-01-27T01:06:40.926235+0000","last_clean":"2024-01-27T01:06:40.926235+0000","last_became_active":"2024-01-27T01:06:26.876056+0000","last_became_peered":"2024-01-27T01:06:26.876056+0000","last_unstale":"2024-01-27T01:06:40.926235+0000","last_undegraded":"2024-01-27T01:06:40.926235+0000","last_fullsized":"2024-01-27T01:06:40.926235+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:38:36.236958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.947768+0000","last_change":"2024-01-27T01:06:40.944841+0000","last_active":"2024-01-27T01:06:40.947768+0000","last_peered":"2024-01-27T01:06:40.947768+0000","last_clean":"2024-01-27T01:06:40.947768+0000","last_became_active":"2024-01-27T01:06:40.941811+0000","last_became_peered":"2024-01-27T01:06:40.941811+0000","last_unstale":"2024-01-27T01:06:40.947768+0000","last_undegraded":"2024-01-27T01:06:40.947768+0000","last_fullsized":"2024-01-27T01:06:40.947768+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:38.155368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.927179+0000","last_change":"2024-01-27T01:06:26.879441+0000","last_active":"2024-01-27T01:06:40.927179+0000","last_peered":"2024-01-27T01:06:40.927179+0000","last_clean":"2024-01-27T01:06:40.927179+0000","last_became_active":"2024-01-27T01:06:26.879148+0000","last_became_peered":"2024-01-27T01:06:26.879148+0000","last_unstale":"2024-01-27T01:06:40.927179+0000","last_undegraded":"2024-01-27T01:06:40.927179+0000","last_fullsized":"2024-01-27T01:06:40.927179+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:05:08.012027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.954692+0000","last_change":"2024-01-27T01:06:40.951207+0000","last_active":"2024-01-27T01:06:40.954692+0000","last_peered":"2024-01-27T01:06:40.954692+0000","last_clean":"2024-01-27T01:06:40.954692+0000","last_became_active":"2024-01-27T01:06:40.950903+0000","last_became_peered":"2024-01-27T01:06:40.950903+0000","last_unstale":"2024-01-27T01:06:40.954692+0000","last_undegraded":"2024-01-27T01:06:40.954692+0000","last_fullsized":"2024-01-27T01:06:40.954692+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:13:18.790129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.927030+0000","last_change":"2024-01-27T01:06:26.876823+0000","last_active":"2024-01-27T01:06:40.927030+0000","last_peered":"2024-01-27T01:06:40.927030+0000","last_clean":"2024-01-27T01:06:40.927030+0000","last_became_active":"2024-01-27T01:06:26.876304+0000","last_became_peered":"2024-01-27T01:06:26.876304+0000","last_unstale":"2024-01-27T01:06:40.927030+0000","last_undegraded":"2024-01-27T01:06:40.927030+0000","last_fullsized":"2024-01-27T01:06:40.927030+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:28.378677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.943663+0000","last_change":"2024-01-27T01:06:40.936906+0000","last_active":"2024-01-27T01:06:40.943663+0000","last_peered":"2024-01-27T01:06:40.943663+0000","last_clean":"2024-01-27T01:06:40.943663+0000","last_became_active":"2024-01-27T01:06:40.936631+0000","last_became_peered":"2024-01-27T01:06:40.936631+0000","last_unstale":"2024-01-27T01:06:40.943663+0000","last_undegraded":"2024-01-27T01:06:40.943663+0000","last_fullsized":"2024-01-27T01:06:40.943663+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:41.677193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926899+0000","last_change":"2024-01-27T01:06:26.873401+0000","last_active":"2024-01-27T01:06:40.926899+0000","last_peered":"2024-01-27T01:06:40.926899+0000","last_clean":"2024-01-27T01:06:40.926899+0000","last_became_active":"2024-01-27T01:06:26.873094+0000","last_became_peered":"2024-01-27T01:06:26.873094+0000","last_unstale":"2024-01-27T01:06:40.926899+0000","last_undegraded":"2024-01-27T01:06:40.926899+0000","last_fullsized":"2024-01-27T01:06:40.926899+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:24.082749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.929355+0000","last_change":"2024-01-27T01:06:40.928273+0000","last_active":"2024-01-27T01:06:40.929355+0000","last_peered":"2024-01-27T01:06:40.929355+0000","last_clean":"2024-01-27T01:06:40.929355+0000","last_became_active":"2024-01-27T01:06:40.927715+0000","last_became_peered":"2024-01-27T01:06:40.927715+0000","last_unstale":"2024-01-27T01:06:40.929355+0000","last_undegraded":"2024-01-27T01:06:40.929355+0000","last_fullsized":"2024-01-27T01:06:40.929355+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:49:23.701149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923728+0000","last_change":"2024-01-27T01:06:26.864890+0000","last_active":"2024-01-27T01:06:40.923728+0000","last_peered":"2024-01-27T01:06:40.923728+0000","last_clean":"2024-01-27T01:06:40.923728+0000","last_became_active":"2024-01-27T01:06:26.864556+0000","last_became_peered":"2024-01-27T01:06:26.864556+0000","last_unstale":"2024-01-27T01:06:40.923728+0000","last_undegraded":"2024-01-27T01:06:40.923728+0000","last_fullsized":"2024-01-27T01:06:40.923728+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:41.033580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.948110+0000","last_change":"2024-01-27T01:06:40.944879+0000","last_active":"2024-01-27T01:06:40.948110+0000","last_peered":"2024-01-27T01:06:40.948110+0000","last_clean":"2024-01-27T01:06:40.948110+0000","last_became_active":"2024-01-27T01:06:40.944457+0000","last_became_peered":"2024-01-27T01:06:40.944457+0000","last_unstale":"2024-01-27T01:06:40.948110+0000","last_undegraded":"2024-01-27T01:06:40.948110+0000","last_fullsized":"2024-01-27T01:06:40.948110+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:57:02.803345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925347+0000","last_change":"2024-01-27T01:06:26.874473+0000","last_active":"2024-01-27T01:06:40.925347+0000","last_peered":"2024-01-27T01:06:40.925347+0000","last_clean":"2024-01-27T01:06:40.925347+0000","last_became_active":"2024-01-27T01:06:26.874203+0000","last_became_peered":"2024-01-27T01:06:26.874203+0000","last_unstale":"2024-01-27T01:06:40.925347+0000","last_undegraded":"2024-01-27T01:06:40.925347+0000","last_fullsized":"2024-01-27T01:06:40.925347+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:33:37.975146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.940476+0000","last_change":"2024-01-27T01:06:40.935406+0000","last_active":"2024-01-27T01:06:40.940476+0000","last_peered":"2024-01-27T01:06:40.940476+0000","last_clean":"2024-01-27T01:06:40.940476+0000","last_became_active":"2024-01-27T01:06:40.935179+0000","last_became_peered":"2024-01-27T01:06:40.935179+0000","last_unstale":"2024-01-27T01:06:40.940476+0000","last_undegraded":"2024-01-27T01:06:40.940476+0000","last_fullsized":"2024-01-27T01:06:40.940476+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:37:21.887288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919006+0000","last_change":"2024-01-27T01:06:26.863380+0000","last_active":"2024-01-27T01:06:40.919006+0000","last_peered":"2024-01-27T01:06:40.919006+0000","last_clean":"2024-01-27T01:06:40.919006+0000","last_became_active":"2024-01-27T01:06:26.863034+0000","last_became_peered":"2024-01-27T01:06:26.863034+0000","last_unstale":"2024-01-27T01:06:40.919006+0000","last_undegraded":"2024-01-27T01:06:40.919006+0000","last_fullsized":"2024-01-27T01:06:40.919006+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:32.271904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942025+0000","last_change":"2024-01-27T01:06:40.939294+0000","last_active":"2024-01-27T01:06:40.942025+0000","last_peered":"2024-01-27T01:06:40.942025+0000","last_clean":"2024-01-27T01:06:40.942025+0000","last_became_active":"2024-01-27T01:06:40.938975+0000","last_became_peered":"2024-01-27T01:06:40.938975+0000","last_unstale":"2024-01-27T01:06:40.942025+0000","last_undegraded":"2024-01-27T01:06:40.942025+0000","last_fullsized":"2024-01-27T01:06:40.942025+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:23.028911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.934074+0000","last_change":"2024-01-27T01:06:26.877929+0000","last_active":"2024-01-27T01:06:40.934074+0000","last_peered":"2024-01-27T01:06:40.934074+0000","last_clean":"2024-01-27T01:06:40.934074+0000","last_became_active":"2024-01-27T01:06:26.877679+0000","last_became_peered":"2024-01-27T01:06:26.877679+0000","last_unstale":"2024-01-27T01:06:40.934074+0000","last_undegraded":"2024-01-27T01:06:40.934074+0000","last_fullsized":"2024-01-27T01:06:40.934074+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:53:15.482586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942625+0000","last_change":"2024-01-27T01:06:40.940817+0000","last_active":"2024-01-27T01:06:40.942625+0000","last_peered":"2024-01-27T01:06:40.942625+0000","last_clean":"2024-01-27T01:06:40.942625+0000","last_became_active":"2024-01-27T01:06:40.940551+0000","last_became_peered":"2024-01-27T01:06:40.940551+0000","last_unstale":"2024-01-27T01:06:40.942625+0000","last_undegraded":"2024-01-27T01:06:40.942625+0000","last_fullsized":"2024-01-27T01:06:40.942625+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:36:17.619595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926330+0000","last_change":"2024-01-27T01:06:26.876674+0000","last_active":"2024-01-27T01:06:40.926330+0000","last_peered":"2024-01-27T01:06:40.926330+0000","last_clean":"2024-01-27T01:06:40.926330+0000","last_became_active":"2024-01-27T01:06:26.876413+0000","last_became_peered":"2024-01-27T01:06:26.876413+0000","last_unstale":"2024-01-27T01:06:40.926330+0000","last_undegraded":"2024-01-27T01:06:40.926330+0000","last_fullsized":"2024-01-27T01:06:40.926330+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:08.822886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.940760+0000","last_change":"2024-01-27T01:06:40.933472+0000","last_active":"2024-01-27T01:06:40.940760+0000","last_peered":"2024-01-27T01:06:40.940760+0000","last_clean":"2024-01-27T01:06:40.940760+0000","last_became_active":"2024-01-27T01:06:40.932482+0000","last_became_peered":"2024-01-27T01:06:40.932482+0000","last_unstale":"2024-01-27T01:06:40.940760+0000","last_undegraded":"2024-01-27T01:06:40.940760+0000","last_fullsized":"2024-01-27T01:06:40.940760+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:10:08.588321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931482+0000","last_change":"2024-01-27T01:06:26.877777+0000","last_active":"2024-01-27T01:06:40.931482+0000","last_peered":"2024-01-27T01:06:40.931482+0000","last_clean":"2024-01-27T01:06:40.931482+0000","last_became_active":"2024-01-27T01:06:26.877515+0000","last_became_peered":"2024-01-27T01:06:26.877515+0000","last_unstale":"2024-01-27T01:06:40.931482+0000","last_undegraded":"2024-01-27T01:06:40.931482+0000","last_fullsized":"2024-01-27T01:06:40.931482+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:27:14.653781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.954673+0000","last_change":"2024-01-27T01:06:40.950670+0000","last_active":"2024-01-27T01:06:40.954673+0000","last_peered":"2024-01-27T01:06:40.954673+0000","last_clean":"2024-01-27T01:06:40.954673+0000","last_became_active":"2024-01-27T01:06:40.949809+0000","last_became_peered":"2024-01-27T01:06:40.949809+0000","last_unstale":"2024-01-27T01:06:40.954673+0000","last_undegraded":"2024-01-27T01:06:40.954673+0000","last_fullsized":"2024-01-27T01:06:40.954673+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:12:59.099059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.939378+0000","last_change":"2024-01-27T01:06:26.865833+0000","last_active":"2024-01-27T01:06:40.939378+0000","last_peered":"2024-01-27T01:06:40.939378+0000","last_clean":"2024-01-27T01:06:40.939378+0000","last_became_active":"2024-01-27T01:06:26.865525+0000","last_became_peered":"2024-01-27T01:06:26.865525+0000","last_unstale":"2024-01-27T01:06:40.939378+0000","last_undegraded":"2024-01-27T01:06:40.939378+0000","last_fullsized":"2024-01-27T01:06:40.939378+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:42:10.990703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.937365+0000","last_change":"2024-01-27T01:06:40.934179+0000","last_active":"2024-01-27T01:06:40.937365+0000","last_peered":"2024-01-27T01:06:40.937365+0000","last_clean":"2024-01-27T01:06:40.937365+0000","last_became_active":"2024-01-27T01:06:40.933756+0000","last_became_peered":"2024-01-27T01:06:40.933756+0000","last_unstale":"2024-01-27T01:06:40.937365+0000","last_undegraded":"2024-01-27T01:06:40.937365+0000","last_fullsized":"2024-01-27T01:06:40.937365+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:46.607241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919877+0000","last_change":"2024-01-27T01:06:26.879535+0000","last_active":"2024-01-27T01:06:40.919877+0000","last_peered":"2024-01-27T01:06:40.919877+0000","last_clean":"2024-01-27T01:06:40.919877+0000","last_became_active":"2024-01-27T01:06:26.879219+0000","last_became_peered":"2024-01-27T01:06:26.879219+0000","last_unstale":"2024-01-27T01:06:40.919877+0000","last_undegraded":"2024-01-27T01:06:40.919877+0000","last_fullsized":"2024-01-27T01:06:40.919877+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:58.495062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.943965+0000","last_change":"2024-01-27T01:06:40.935281+0000","last_active":"2024-01-27T01:06:40.943965+0000","last_peered":"2024-01-27T01:06:40.943965+0000","last_clean":"2024-01-27T01:06:40.943965+0000","last_became_active":"2024-01-27T01:06:40.934857+0000","last_became_peered":"2024-01-27T01:06:40.934857+0000","last_unstale":"2024-01-27T01:06:40.943965+0000","last_undegraded":"2024-01-27T01:06:40.943965+0000","last_fullsized":"2024-01-27T01:06:40.943965+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:12:42.423061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.928744+0000","last_change":"2024-01-27T01:06:26.876736+0000","last_active":"2024-01-27T01:06:40.928744+0000","last_peered":"2024-01-27T01:06:40.928744+0000","last_clean":"2024-01-27T01:06:40.928744+0000","last_became_active":"2024-01-27T01:06:26.875870+0000","last_became_peered":"2024-01-27T01:06:26.875870+0000","last_unstale":"2024-01-27T01:06:40.928744+0000","last_undegraded":"2024-01-27T01:06:40.928744+0000","last_fullsized":"2024-01-27T01:06:40.928744+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:43.223271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.949848+0000","last_change":"2024-01-27T01:06:40.943329+0000","last_active":"2024-01-27T01:06:40.949848+0000","last_peered":"2024-01-27T01:06:40.949848+0000","last_clean":"2024-01-27T01:06:40.949848+0000","last_became_active":"2024-01-27T01:06:40.943072+0000","last_became_peered":"2024-01-27T01:06:40.943072+0000","last_unstale":"2024-01-27T01:06:40.949848+0000","last_undegraded":"2024-01-27T01:06:40.949848+0000","last_fullsized":"2024-01-27T01:06:40.949848+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:00:48.382771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925282+0000","last_change":"2024-01-27T01:06:26.873409+0000","last_active":"2024-01-27T01:06:40.925282+0000","last_peered":"2024-01-27T01:06:40.925282+0000","last_clean":"2024-01-27T01:06:40.925282+0000","last_became_active":"2024-01-27T01:06:26.868026+0000","last_became_peered":"2024-01-27T01:06:26.868026+0000","last_unstale":"2024-01-27T01:06:40.925282+0000","last_undegraded":"2024-01-27T01:06:40.925282+0000","last_fullsized":"2024-01-27T01:06:40.925282+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:15.664932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942082+0000","last_change":"2024-01-27T01:06:40.939648+0000","last_active":"2024-01-27T01:06:40.942082+0000","last_peered":"2024-01-27T01:06:40.942082+0000","last_clean":"2024-01-27T01:06:40.942082+0000","last_became_active":"2024-01-27T01:06:40.939090+0000","last_became_peered":"2024-01-27T01:06:40.939090+0000","last_unstale":"2024-01-27T01:06:40.942082+0000","last_undegraded":"2024-01-27T01:06:40.942082+0000","last_fullsized":"2024-01-27T01:06:40.942082+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:10:11.320660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926301+0000","last_change":"2024-01-27T01:06:26.881786+0000","last_active":"2024-01-27T01:06:40.926301+0000","last_peered":"2024-01-27T01:06:40.926301+0000","last_clean":"2024-01-27T01:06:40.926301+0000","last_became_active":"2024-01-27T01:06:26.881527+0000","last_became_peered":"2024-01-27T01:06:26.881527+0000","last_unstale":"2024-01-27T01:06:40.926301+0000","last_undegraded":"2024-01-27T01:06:40.926301+0000","last_fullsized":"2024-01-27T01:06:40.926301+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:43.630831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.948058+0000","last_change":"2024-01-27T01:06:40.945117+0000","last_active":"2024-01-27T01:06:40.948058+0000","last_peered":"2024-01-27T01:06:40.948058+0000","last_clean":"2024-01-27T01:06:40.948058+0000","last_became_active":"2024-01-27T01:06:40.944795+0000","last_became_peered":"2024-01-27T01:06:40.944795+0000","last_unstale":"2024-01-27T01:06:40.948058+0000","last_undegraded":"2024-01-27T01:06:40.948058+0000","last_fullsized":"2024-01-27T01:06:40.948058+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:35:02.237373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.938130+0000","last_change":"2024-01-27T01:06:26.872094+0000","last_active":"2024-01-27T01:06:40.938130+0000","last_peered":"2024-01-27T01:06:40.938130+0000","last_clean":"2024-01-27T01:06:40.938130+0000","last_became_active":"2024-01-27T01:06:26.871711+0000","last_became_peered":"2024-01-27T01:06:26.871711+0000","last_unstale":"2024-01-27T01:06:40.938130+0000","last_undegraded":"2024-01-27T01:06:40.938130+0000","last_fullsized":"2024-01-27T01:06:40.938130+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:24:32.821570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.944084+0000","last_change":"2024-01-27T01:06:40.943231+0000","last_active":"2024-01-27T01:06:40.944084+0000","last_peered":"2024-01-27T01:06:40.944084+0000","last_clean":"2024-01-27T01:06:40.944084+0000","last_became_active":"2024-01-27T01:06:40.942831+0000","last_became_peered":"2024-01-27T01:06:40.942831+0000","last_unstale":"2024-01-27T01:06:40.944084+0000","last_undegraded":"2024-01-27T01:06:40.944084+0000","last_fullsized":"2024-01-27T01:06:40.944084+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:55:50.433195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.932394+0000","last_change":"2024-01-27T01:06:26.879291+0000","last_active":"2024-01-27T01:06:40.932394+0000","last_peered":"2024-01-27T01:06:40.932394+0000","last_clean":"2024-01-27T01:06:40.932394+0000","last_became_active":"2024-01-27T01:06:26.879063+0000","last_became_peered":"2024-01-27T01:06:26.879063+0000","last_unstale":"2024-01-27T01:06:40.932394+0000","last_undegraded":"2024-01-27T01:06:40.932394+0000","last_fullsized":"2024-01-27T01:06:40.932394+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:22:23.545688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942365+0000","last_change":"2024-01-27T01:06:40.938252+0000","last_active":"2024-01-27T01:06:40.942365+0000","last_peered":"2024-01-27T01:06:40.942365+0000","last_clean":"2024-01-27T01:06:40.942365+0000","last_became_active":"2024-01-27T01:06:40.937940+0000","last_became_peered":"2024-01-27T01:06:40.937940+0000","last_unstale":"2024-01-27T01:06:40.942365+0000","last_undegraded":"2024-01-27T01:06:40.942365+0000","last_fullsized":"2024-01-27T01:06:40.942365+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:59.510860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.941679+0000","last_change":"2024-01-27T01:06:26.870422+0000","last_active":"2024-01-27T01:06:40.941679+0000","last_peered":"2024-01-27T01:06:40.941679+0000","last_clean":"2024-01-27T01:06:40.941679+0000","last_became_active":"2024-01-27T01:06:26.870117+0000","last_became_peered":"2024-01-27T01:06:26.870117+0000","last_unstale":"2024-01-27T01:06:40.941679+0000","last_undegraded":"2024-01-27T01:06:40.941679+0000","last_fullsized":"2024-01-27T01:06:40.941679+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:42:55.987157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926643+0000","last_change":"2024-01-27T01:06:40.925614+0000","last_active":"2024-01-27T01:06:40.926643+0000","last_peered":"2024-01-27T01:06:40.926643+0000","last_clean":"2024-01-27T01:06:40.926643+0000","last_became_active":"2024-01-27T01:06:40.925180+0000","last_became_peered":"2024-01-27T01:06:40.925180+0000","last_unstale":"2024-01-27T01:06:40.926643+0000","last_undegraded":"2024-01-27T01:06:40.926643+0000","last_fullsized":"2024-01-27T01:06:40.926643+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:14:51.519402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931414+0000","last_change":"2024-01-27T01:06:26.868177+0000","last_active":"2024-01-27T01:06:40.931414+0000","last_peered":"2024-01-27T01:06:40.931414+0000","last_clean":"2024-01-27T01:06:40.931414+0000","last_became_active":"2024-01-27T01:06:26.867906+0000","last_became_peered":"2024-01-27T01:06:26.867906+0000","last_unstale":"2024-01-27T01:06:40.931414+0000","last_undegraded":"2024-01-27T01:06:40.931414+0000","last_fullsized":"2024-01-27T01:06:40.931414+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:32.271904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.937853+0000","last_change":"2024-01-27T01:06:40.936971+0000","last_active":"2024-01-27T01:06:40.937853+0000","last_peered":"2024-01-27T01:06:40.937853+0000","last_clean":"2024-01-27T01:06:40.937853+0000","last_became_active":"2024-01-27T01:06:40.936636+0000","last_became_peered":"2024-01-27T01:06:40.936636+0000","last_unstale":"2024-01-27T01:06:40.937853+0000","last_undegraded":"2024-01-27T01:06:40.937853+0000","last_fullsized":"2024-01-27T01:06:40.937853+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:18.204468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919361+0000","last_change":"2024-01-27T01:06:26.868462+0000","last_active":"2024-01-27T01:06:40.919361+0000","last_peered":"2024-01-27T01:06:40.919361+0000","last_clean":"2024-01-27T01:06:40.919361+0000","last_became_active":"2024-01-27T01:06:26.868203+0000","last_became_peered":"2024-01-27T01:06:26.868203+0000","last_unstale":"2024-01-27T01:06:40.919361+0000","last_undegraded":"2024-01-27T01:06:40.919361+0000","last_fullsized":"2024-01-27T01:06:40.919361+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:10.965469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.943784+0000","last_change":"2024-01-27T01:06:40.939071+0000","last_active":"2024-01-27T01:06:40.943784+0000","last_peered":"2024-01-27T01:06:40.943784+0000","last_clean":"2024-01-27T01:06:40.943784+0000","last_became_active":"2024-01-27T01:06:40.938084+0000","last_became_peered":"2024-01-27T01:06:40.938084+0000","last_unstale":"2024-01-27T01:06:40.943784+0000","last_undegraded":"2024-01-27T01:06:40.943784+0000","last_fullsized":"2024-01-27T01:06:40.943784+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:51:01.037978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931515+0000","last_change":"2024-01-27T01:06:26.871135+0000","last_active":"2024-01-27T01:06:40.931515+0000","last_peered":"2024-01-27T01:06:40.931515+0000","last_clean":"2024-01-27T01:06:40.931515+0000","last_became_active":"2024-01-27T01:06:26.870854+0000","last_became_peered":"2024-01-27T01:06:26.870854+0000","last_unstale":"2024-01-27T01:06:40.931515+0000","last_undegraded":"2024-01-27T01:06:40.931515+0000","last_fullsized":"2024-01-27T01:06:40.931515+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:12:59.875769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.953931+0000","last_change":"2024-01-27T01:06:40.946055+0000","last_active":"2024-01-27T01:06:40.953931+0000","last_peered":"2024-01-27T01:06:40.953931+0000","last_clean":"2024-01-27T01:06:40.953931+0000","last_became_active":"2024-01-27T01:06:40.945714+0000","last_became_peered":"2024-01-27T01:06:40.945714+0000","last_unstale":"2024-01-27T01:06:40.953931+0000","last_undegraded":"2024-01-27T01:06:40.953931+0000","last_fullsized":"2024-01-27T01:06:40.953931+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:33:43.502121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.928152+0000","last_change":"2024-01-27T01:06:26.873715+0000","last_active":"2024-01-27T01:06:40.928152+0000","last_peered":"2024-01-27T01:06:40.928152+0000","last_clean":"2024-01-27T01:06:40.928152+0000","last_became_active":"2024-01-27T01:06:26.873430+0000","last_became_peered":"2024-01-27T01:06:26.873430+0000","last_unstale":"2024-01-27T01:06:40.928152+0000","last_undegraded":"2024-01-27T01:06:40.928152+0000","last_fullsized":"2024-01-27T01:06:40.928152+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:44.342963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3538,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":630,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8750,"ondisk_log_size":8750,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576005,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300676,"kb_used_data":3388,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071164,"statfs":{"total":96636764160,"available":96328871936,"internally_reserved":0,"allocated":3469312,"data_stored":2099383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63200000000000001,"15min":0.61399999999999999},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.312},"max":{"1min":0.82099999999999995,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.63700000000000001,"15min":0.623},"min":{"1min":0.498,"5min":0.39900000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":0.879,"15min":0.879},"last":0.69599999999999995}]},{"osd":1,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.623,"15min":0.61599999999999999},"min":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.624},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65600000000000003,"15min":0.63700000000000001},"min":{"1min":0.505,"5min":0.42099999999999999,"15min":0.34300000000000003},"max":{"1min":0.76800000000000002,"5min":0.84099999999999997,"15min":0.84399999999999997},"last":0.48299999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63400000000000001,"15min":0.61399999999999999},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82099999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64500000000000002,"15min":0.63600000000000001},"min":{"1min":0.49399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79400000000000004,"5min":0.872,"15min":0.88900000000000001},"last":0.66700000000000004}]},{"osd":3,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.628,"15min":0.62},"min":{"1min":0.52100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79900000000000004,"5min":0.876,"15min":0.876},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66800000000000004,"15min":0.64700000000000002},"min":{"1min":0.53200000000000003,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84599999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.72299999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.50900000000000001,"15min":0.48999999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.29999999999999999},"max":{"1min":0.73399999999999999,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.51900000000000002},"min":{"1min":0.35699999999999998,"5min":0.34699999999999998,"15min":0.32200000000000001},"max":{"1min":0.77500000000000002,"5min":0.80300000000000005,"15min":0.86599999999999999},"last":0.60299999999999998}]},{"osd":5,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55800000000000005,"15min":0.52600000000000002},"min":{"1min":0.40999999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.753,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57999999999999996,"15min":0.56499999999999995},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.33700000000000002},"max":{"1min":0.76700000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.65200000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60999999999999999,"15min":0.58899999999999997},"min":{"1min":0.42499999999999999,"5min":0.42199999999999999,"15min":0.37},"max":{"1min":0.81100000000000005,"5min":0.83999999999999997,"15min":0.84799999999999998},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61299999999999999,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.39100000000000001,"15min":0.35699999999999998},"max":{"1min":0.76100000000000001,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.64100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576005,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308304,"kb_used_data":2824,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063536,"statfs":{"total":96636764160,"available":96321060864,"internally_reserved":0,"allocated":2891776,"data_stored":1510517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61099999999999999,"15min":0.60799999999999998},"min":{"1min":0.41999999999999998,"5min":0.40799999999999997,"15min":0.36399999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.441,"5min":0.432,"15min":0.376},"max":{"1min":0.82499999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.64700000000000002}]},{"osd":1,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79000000000000004,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.40799999999999997,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.56299999999999994}]},{"osd":2,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.39700000000000002,"15min":0.377},"max":{"1min":0.80400000000000005,"5min":0.878,"15min":0.878},"last":0.55000000000000004}]},{"osd":3,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.622,"15min":0.627},"min":{"1min":0.5,"5min":0.378,"15min":0.378},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.90100000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66300000000000003,"15min":0.66000000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.71899999999999997}]},{"osd":4,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.36299999999999999,"5min":0.33300000000000002,"15min":0.32500000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.86599999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33900000000000002,"5min":0.33800000000000002,"15min":0.29799999999999999},"max":{"1min":0.77300000000000002,"5min":0.78500000000000003,"15min":0.88900000000000001},"last":0.58399999999999996}]},{"osd":5,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.31900000000000001},"max":{"1min":0.68999999999999995,"5min":0.75900000000000001,"15min":0.80500000000000005},"last":0.498},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.42199999999999999,"5min":0.37,"15min":0.28899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.874},"last":0.63300000000000001}]},{"osd":7,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45800000000000002,"5min":0.39000000000000001,"15min":0.35799999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.92900000000000005},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.44500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78900000000000003,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.66800000000000004}]}]},{"osd":1,"up_from":13,"seq":55834576005,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300072,"kb_used_data":2784,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071768,"statfs":{"total":96636764160,"available":96329490432,"internally_reserved":0,"allocated":2850816,"data_stored":1505533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.41099999999999998,"15min":0.42099999999999999},"min":{"1min":0.26400000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.64400000000000002,"5min":0.66100000000000003,"15min":0.69599999999999995},"last":0.309},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44500000000000001,"15min":0.45500000000000002},"min":{"1min":0.34000000000000002,"5min":0.28399999999999997,"15min":0.26800000000000002},"max":{"1min":0.57999999999999996,"5min":0.73999999999999999,"15min":0.76600000000000001},"last":0.59499999999999997}]},{"osd":2,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.439,"15min":0.438},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.26200000000000001},"max":{"1min":0.748,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.47599999999999998,"15min":0.48999999999999999},"min":{"1min":0.379,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.67400000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.50700000000000001}]},{"osd":3,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52300000000000002,"15min":0.53300000000000003},"min":{"1min":0.39300000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73699999999999999,"5min":0.84799999999999998,"15min":1.2130000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.51400000000000001},"min":{"1min":0.41699999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68200000000000005,"5min":1.125,"15min":1.22},"last":0.54900000000000004}]},{"osd":4,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.67200000000000004,"15min":0.67400000000000004},"min":{"1min":0.42099999999999999,"5min":0.41099999999999998,"15min":0.39800000000000002},"max":{"1min":0.82699999999999996,"5min":1.242,"15min":1.242},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69099999999999995,"15min":0.69599999999999995},"min":{"1min":0.53700000000000003,"5min":0.496,"15min":0.35699999999999998},"max":{"1min":0.90600000000000003,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.56899999999999995}]},{"osd":5,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73499999999999999,"15min":0.70899999999999996},"min":{"1min":0.53300000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.876,"5min":5.444,"15min":5.444},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.71699999999999997,"15min":0.70799999999999996},"min":{"1min":0.502,"5min":0.47599999999999998,"15min":0.435},"max":{"1min":0.97799999999999998,"5min":2.6669999999999998,"15min":2.6669999999999998},"last":0.46400000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.74199999999999999,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":0.93899999999999995,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.81299999999999994,"15min":0.745},"min":{"1min":0.48799999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.02,"5min":11.223000000000001,"15min":11.223000000000001},"last":0.52700000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.752,"15min":0.75600000000000001},"min":{"1min":0.58999999999999997,"5min":0.52800000000000002,"15min":0.47599999999999998},"max":{"1min":0.96299999999999997,"5min":2.9660000000000002,"15min":2.9660000000000002},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.77700000000000002,"15min":0.76800000000000002},"min":{"1min":0.58799999999999997,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.94499999999999995,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.64400000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576004,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317040,"kb_used_data":3368,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94054800,"statfs":{"total":96636764160,"available":96312115200,"internally_reserved":0,"allocated":3448832,"data_stored":2095991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42899999999999999,"15min":0.42199999999999999},"min":{"1min":0.30299999999999999,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":0.55600000000000005,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.436,"5min":0.45600000000000002,"15min":0.46100000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":0.56499999999999995,"5min":0.72699999999999998,"15min":0.73399999999999999},"last":0.57199999999999995}]},{"osd":2,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.29699999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.58299999999999996,"5min":0.66900000000000004,"15min":0.76800000000000002},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48399999999999999,"15min":0.48899999999999999},"min":{"1min":0.38600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64600000000000002,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.46899999999999997}]},{"osd":3,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.66100000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.504},"min":{"1min":0.36699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64100000000000001,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.497}]},{"osd":4,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.68300000000000005,"15min":0.67700000000000005},"min":{"1min":0.57799999999999996,"5min":0.47999999999999998,"15min":0.42199999999999999},"max":{"1min":0.86399999999999999,"5min":0.91500000000000004,"15min":1.0720000000000001},"last":0.749},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.56999999999999995,"5min":0.47099999999999997,"15min":0.437},"max":{"1min":0.82699999999999996,"5min":1.002,"15min":1.002},"last":0.84099999999999997}]},{"osd":5,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.68999999999999995,"15min":0.70099999999999996},"min":{"1min":0.58199999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.876,"5min":1.0209999999999999,"15min":2.165},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70599999999999996,"15min":0.71199999999999997},"min":{"1min":0.498,"5min":0.48799999999999999,"15min":0.45600000000000002},"max":{"1min":0.89000000000000001,"5min":0.97399999999999998,"15min":1.1399999999999999},"last":0.80700000000000005}]},{"osd":6,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69299999999999995,"15min":0.71099999999999997},"min":{"1min":0.56499999999999995,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.84599999999999997,"5min":0.98899999999999999,"15min":1.0860000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72599999999999998,"15min":0.74099999999999999},"min":{"1min":0.54300000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.93400000000000005,"5min":1.0109999999999999,"15min":1.425},"last":0.82199999999999995}]},{"osd":7,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.74399999999999999,"15min":0.75700000000000001},"min":{"1min":0.61299999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.98899999999999999,"5min":1.044,"15min":1.1259999999999999},"last":0.88},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.76300000000000001,"15min":0.76900000000000002},"min":{"1min":0.58299999999999996,"5min":0.51100000000000001,"15min":0.48499999999999999},"max":{"1min":0.91700000000000004,"5min":1.0289999999999999,"15min":1.1910000000000001},"last":0.89100000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576004,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312372,"kb_used_data":2796,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059468,"statfs":{"total":96636764160,"available":96316895232,"internally_reserved":0,"allocated":2863104,"data_stored":1506211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40500000000000003,"15min":0.40799999999999997},"min":{"1min":0.26600000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.46000000000000002,"5min":0.746,"15min":0.746},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.44500000000000001,"15min":0.44400000000000001},"min":{"1min":0.32300000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.59099999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.35399999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.435,"15min":0.44500000000000001},"min":{"1min":0.32000000000000001,"5min":0.29099999999999998,"15min":0.27000000000000002},"max":{"1min":0.61299999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.439,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.55600000000000005,"5min":0.73599999999999999,"15min":0.78100000000000003},"last":0.40699999999999997}]},{"osd":3,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.498,"15min":0.497},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.33200000000000002},"max":{"1min":0.627,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.439},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48699999999999999,"15min":0.49199999999999999},"min":{"1min":0.373,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":0.59099999999999997,"5min":0.66600000000000004,"15min":0.75700000000000001},"last":0.46300000000000002}]},{"osd":4,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68200000000000005,"15min":0.68000000000000005},"min":{"1min":0.52800000000000002,"5min":0.495,"15min":0.442},"max":{"1min":0.92700000000000005,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68999999999999995,"15min":0.69399999999999995},"min":{"1min":0.50900000000000001,"5min":0.47099999999999997,"15min":0.378},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":1.107},"last":0.60199999999999998}]},{"osd":5,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.69099999999999995,"15min":0.70299999999999996},"min":{"1min":0.52400000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.89500000000000002,"15min":1.167},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70899999999999996,"15min":0.70999999999999996},"min":{"1min":0.47499999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89600000000000002,"5min":0.97499999999999998,"15min":1.05},"last":0.58899999999999997}]},{"osd":6,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71499999999999997,"15min":0.71199999999999997},"min":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.45500000000000002},"max":{"1min":0.81399999999999995,"5min":0.996,"15min":1.077},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70499999999999996,"15min":0.71999999999999997},"min":{"1min":0.497,"5min":0.497,"15min":0.39200000000000002},"max":{"1min":0.876,"5min":0.96699999999999997,"15min":1.135},"last":0.70299999999999996}]},{"osd":7,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.749,"15min":0.75},"min":{"1min":0.56000000000000005,"5min":0.50800000000000001,"15min":0.47099999999999997},"max":{"1min":0.94099999999999995,"5min":0.97399999999999998,"15min":1.198},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.76500000000000001,"15min":0.76900000000000002},"min":{"1min":0.55000000000000004,"5min":0.53100000000000003,"15min":0.48199999999999998},"max":{"1min":0.94599999999999995,"5min":1.028,"15min":1.0880000000000001},"last":0.68100000000000005}]}]},{"osd":3,"up_from":13,"seq":55834576003,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304184,"kb_used_data":2800,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067656,"statfs":{"total":96636764160,"available":96325279744,"internally_reserved":0,"allocated":2867200,"data_stored":1505681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44700000000000001,"15min":0.42399999999999999},"min":{"1min":0.309,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.59499999999999997,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.31},{"interface":"front","average":{"1min":0.439,"5min":0.44,"15min":0.45500000000000002},"min":{"1min":0.32400000000000001,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.64000000000000001,"5min":0.66700000000000004,"15min":0.84699999999999998},"last":0.66400000000000003}]},{"osd":1,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.624,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66400000000000003,"5min":15.391,"15min":15.391},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.53500000000000003,"15min":0.52400000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70499999999999996,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.59199999999999997}]},{"osd":2,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.54300000000000004,"15min":0.51600000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.60299999999999998,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53900000000000003,"15min":0.51000000000000001},"min":{"1min":0.374,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.5529999999999999,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":0.39000000000000001}]},{"osd":4,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.745,"15min":0.71899999999999997},"min":{"1min":0.56399999999999995,"5min":0.47699999999999998,"15min":0.439},"max":{"1min":0.82699999999999996,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.70899999999999996},"min":{"1min":0.52000000000000002,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":1.0089999999999999,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.64600000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.745,"15min":0.71399999999999997},"min":{"1min":0.52000000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.036,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73899999999999999,"15min":0.72299999999999998},"min":{"1min":0.54900000000000004,"5min":0.435,"15min":0.435},"max":{"1min":1.127,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.495}]},{"osd":6,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.746,"15min":0.72199999999999998},"min":{"1min":0.50700000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.92100000000000004,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.753,"15min":0.74099999999999999},"min":{"1min":0.54900000000000004,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.077,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.54800000000000004}]},{"osd":7,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.755,"15min":0.73899999999999999},"min":{"1min":0.53300000000000003,"5min":0.52500000000000002,"15min":0.46700000000000003},"max":{"1min":1.103,"5min":4.569,"15min":4.569},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.80600000000000005,"15min":0.78300000000000003},"min":{"1min":0.59199999999999997,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.98999999999999999,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.51700000000000002}]}]},{"osd":4,"up_from":13,"seq":55834576005,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312368,"kb_used_data":2792,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059472,"statfs":{"total":96636764160,"available":96316899328,"internally_reserved":0,"allocated":2859008,"data_stored":1506177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.46899999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.84199999999999997},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.61499999999999999,"15min":0.61399999999999999},"min":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78700000000000003,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.68100000000000005}]},{"osd":1,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61899999999999999,"15min":0.627},"min":{"1min":0.46100000000000002,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.76400000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61499999999999999,"15min":0.62},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.79800000000000004,"5min":0.874,"15min":0.874},"last":0.72199999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.59999999999999998},"min":{"1min":0.42399999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.88700000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.51100000000000001,"5min":0.42899999999999999,"15min":0.42799999999999999},"max":{"1min":0.81100000000000005,"5min":0.81499999999999995,"15min":0.89200000000000002},"last":0.76300000000000001}]},{"osd":3,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.61599999999999999,"15min":0.623},"min":{"1min":0.38800000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.83299999999999996,"5min":0.83599999999999997,"15min":0.84199999999999997},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.627},"min":{"1min":0.41299999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.88300000000000001},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.75800000000000001,"15min":0.85899999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.502},"min":{"1min":0.376,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.878},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34999999999999998},"max":{"1min":0.63200000000000001,"5min":0.70099999999999996,"15min":0.88700000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.70699999999999996,"5min":0.79200000000000004,"15min":0.81499999999999995},"last":0.60599999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.42399999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72599999999999998,"5min":0.80400000000000005,"15min":0.90200000000000002},"last":0.629},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59499999999999997,"15min":0.59399999999999997},"min":{"1min":0.41599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.73799999999999999,"5min":0.78700000000000003,"15min":0.92600000000000005},"last":0.70499999999999996}]}]},{"osd":5,"up_from":13,"seq":55834576005,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312384,"kb_used_data":2808,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059456,"statfs":{"total":96636764160,"available":96316882944,"internally_reserved":0,"allocated":2875392,"data_stored":1507195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59399999999999997,"15min":0.60299999999999998},"min":{"1min":0.42999999999999999,"5min":0.40500000000000003,"15min":0.27300000000000002},"max":{"1min":0.80700000000000005,"5min":0.82799999999999996,"15min":1.006},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59399999999999997,"15min":0.60699999999999998},"min":{"1min":0.36599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":1.1719999999999999},"last":0.61799999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63700000000000001,"15min":0.65200000000000002},"min":{"1min":0.39100000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.95899999999999996},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.52100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":1.212},"last":0.51900000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57499999999999996,"15min":0.59399999999999997},"min":{"1min":0.42299999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":1.631},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.625,"15min":0.621},"min":{"1min":0.497,"5min":0.38800000000000001,"15min":0.34000000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.54000000000000004}]},{"osd":3,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.59999999999999998},"min":{"1min":0.39600000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83299999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.625,"15min":0.63},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.879,"5min":0.879,"15min":1.4690000000000001},"last":0.50700000000000001}]},{"osd":4,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.505,"15min":0.48799999999999999},"min":{"1min":0.34899999999999998,"5min":0.32200000000000001,"15min":0.317},"max":{"1min":0.76700000000000002,"5min":0.90200000000000002,"15min":0.94399999999999995},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53500000000000003,"15min":0.51500000000000001},"min":{"1min":0.439,"5min":0.35199999999999998,"15min":0.33900000000000002},"max":{"1min":0.80800000000000005,"5min":0.92500000000000004,"15min":0.93300000000000005},"last":0.46500000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51800000000000002,"15min":0.504},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79400000000000004,"5min":0.96799999999999997,"15min":1.732},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56299999999999994,"15min":0.54500000000000004},"min":{"1min":0.42899999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.753,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.48999999999999999}]},{"osd":7,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.46200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80400000000000005,"5min":0.95399999999999996,"15min":1.0740000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.626,"15min":0.61299999999999999},"min":{"1min":0.504,"5min":0.40799999999999997,"15min":0.39100000000000001},"max":{"1min":0.85199999999999998,"5min":0.996,"15min":1.6919999999999999},"last":0.58599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:06:49.902 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:06:50.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.044+0000 7f0b29e9e700 1 Processor -- start 2024-01-27T01:06:50.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.044+0000 7f0b29e9e700 1 -- start start 2024-01-27T01:06:50.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.045+0000 7f0b29e9e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0b240acf30 0x7f0b240c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:50.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.045+0000 7f0b29e9e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0b240ad300 0x7f0b240ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:50.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.045+0000 7f0b29e9e700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b240c4230 con 0x7f0b240ad780 2024-01-27T01:06:50.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.045+0000 7f0b29e9e700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b24157be0 con 0x7f0b240acf30 2024-01-27T01:06:50.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.045+0000 7f0b29e9e700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b240cbf10 con 0x7f0b240ad300 2024-01-27T01:06:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.045+0000 7f0b237fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0b240acf30 0x7f0b240c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.045+0000 7f0b23fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0b240ad780 0x7f0b2415b270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45464/0 (socket says 172.21.15.46:45464) 2024-01-27T01:06:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.045+0000 7f0b23fff700 1 -- 172.21.15.46:0/2279890682 learned_addr learned my addr 172.21.15.46:0/2279890682 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.046+0000 7f0b22ffd700 1 --2- 172.21.15.46:0/2279890682 >> v2:172.21.15.106:3300/0 conn(0x7f0b240ad300 0x7f0b240ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.046+0000 7f0b237fe700 1 -- 172.21.15.46:0/2279890682 >> v2:172.21.15.106:3300/0 conn(0x7f0b240ad300 msgr2=0x7f0b240ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.046+0000 7f0b237fe700 1 --2- 172.21.15.46:0/2279890682 >> v2:172.21.15.106:3300/0 conn(0x7f0b240ad300 0x7f0b240ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.046+0000 7f0b237fe700 1 -- 172.21.15.46:0/2279890682 >> v1:172.21.15.46:6789/0 conn(0x7f0b240ad780 legacy=0x7f0b2415b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.046+0000 7f0b237fe700 1 -- 172.21.15.46:0/2279890682 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b240c4460 con 0x7f0b240acf30 2024-01-27T01:06:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.046+0000 7f0b227fc700 1 -- 172.21.15.46:0/2279890682 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688576998 0 0) 0x7f0b240c4230 con 0x7f0b240ad780 2024-01-27T01:06:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.046+0000 7f0b237fe700 1 --2- 172.21.15.46:0/2279890682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b240acf30 0x7f0b240c3cf0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f0b240b1060 tx=0x7f0b140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d51380f69363e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.046+0000 7f0b227fc700 1 -- 172.21.15.46:0/2279890682 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b14011630 con 0x7f0b240acf30 2024-01-27T01:06:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.046+0000 7f0b227fc700 1 -- 172.21.15.46:0/2279890682 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b14011790 con 0x7f0b240acf30 2024-01-27T01:06:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.046+0000 7f0b227fc700 1 -- 172.21.15.46:0/2279890682 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b14011b30 con 0x7f0b240acf30 2024-01-27T01:06:50.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.047+0000 7f0b29e9e700 1 -- 172.21.15.46:0/2279890682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b240acf30 msgr2=0x7f0b240c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.047+0000 7f0b29e9e700 1 --2- 172.21.15.46:0/2279890682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b240acf30 0x7f0b240c3cf0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f0b240b1060 tx=0x7f0b140091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.047+0000 7f0b29e9e700 1 -- 172.21.15.46:0/2279890682 shutdown_connections 2024-01-27T01:06:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.047+0000 7f0b29e9e700 1 --2- 172.21.15.46:0/2279890682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b240acf30 0x7f0b240c3cf0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.047+0000 7f0b29e9e700 1 --2- 172.21.15.46:0/2279890682 >> v2:172.21.15.106:3300/0 conn(0x7f0b240ad300 0x7f0b240ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.047+0000 7f0b29e9e700 1 -- 172.21.15.46:0/2279890682 >> 172.21.15.46:0/2279890682 conn(0x7f0b24097820 msgr2=0x7f0b24068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.047+0000 7f0b29e9e700 1 -- 172.21.15.46:0/2279890682 shutdown_connections 2024-01-27T01:06:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.047+0000 7f0b29e9e700 1 -- 172.21.15.46:0/2279890682 wait complete. 2024-01-27T01:06:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.048+0000 7f0b29e9e700 1 Processor -- start 2024-01-27T01:06:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.048+0000 7f0b29e9e700 1 -- start start 2024-01-27T01:06:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.049+0000 7f0b29e9e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b240acf30 0x7f0b24148430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.049+0000 7f0b29e9e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0b240ad300 0x7f0b240b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.049+0000 7f0b29e9e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0b240ad780 0x7f0b24156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.049+0000 7f0b29e9e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b24148970 con 0x7f0b240acf30 2024-01-27T01:06:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.049+0000 7f0b29e9e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b240caa10 con 0x7f0b240ad300 2024-01-27T01:06:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.049+0000 7f0b29e9e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2406a950 con 0x7f0b240ad780 2024-01-27T01:06:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.049+0000 7f0b237fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b240acf30 0x7f0b24148430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.049+0000 7f0b237fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b240acf30 0x7f0b24148430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41910/0 (socket says 172.21.15.46:41910) 2024-01-27T01:06:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.049+0000 7f0b237fe700 1 -- 172.21.15.46:0/1119946519 learned_addr learned my addr 172.21.15.46:0/1119946519 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.049+0000 7f0b22ffd700 1 --2- 172.21.15.46:0/1119946519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0b240ad300 0x7f0b240b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.049+0000 7f0b23fff700 1 --2- 172.21.15.46:0/1119946519 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0b240ad780 0x7f0b24156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.049+0000 7f0b237fe700 1 -- 172.21.15.46:0/1119946519 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0b240ad780 msgr2=0x7f0b24156270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.049+0000 7f0b237fe700 1 --2- 172.21.15.46:0/1119946519 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0b240ad780 0x7f0b24156270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.049+0000 7f0b237fe700 1 -- 172.21.15.46:0/1119946519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0b240ad300 msgr2=0x7f0b240b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.049+0000 7f0b237fe700 1 --2- 172.21.15.46:0/1119946519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0b240ad300 0x7f0b240b12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.050+0000 7f0b237fe700 1 -- 172.21.15.46:0/1119946519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b14009420 con 0x7f0b240acf30 2024-01-27T01:06:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.050+0000 7f0b23fff700 1 --2- 172.21.15.46:0/1119946519 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0b240ad780 0x7f0b24156270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.050+0000 7f0b237fe700 1 --2- 172.21.15.46:0/1119946519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b240acf30 0x7f0b24148430 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f0b2406ef50 tx=0x7f0b14017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.050+0000 7f0b0b7fe700 1 -- 172.21.15.46:0/1119946519 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b14011b10 con 0x7f0b240acf30 2024-01-27T01:06:50.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.050+0000 7f0b0b7fe700 1 -- 172.21.15.46:0/1119946519 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b140253f0 con 0x7f0b240acf30 2024-01-27T01:06:50.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.050+0000 7f0b29e9e700 1 -- 172.21.15.46:0/1119946519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2406a390 con 0x7f0b240acf30 2024-01-27T01:06:50.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.050+0000 7f0b0b7fe700 1 -- 172.21.15.46:0/1119946519 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b14025790 con 0x7f0b240acf30 2024-01-27T01:06:50.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.050+0000 7f0b29e9e700 1 -- 172.21.15.46:0/1119946519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b240ca3c0 con 0x7f0b240acf30 2024-01-27T01:06:50.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.051+0000 7f0b29e9e700 1 -- 172.21.15.46:0/1119946519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b240b2d60 con 0x7f0b240acf30 2024-01-27T01:06:50.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.051+0000 7f0b0b7fe700 1 -- 172.21.15.46:0/1119946519 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0b14025b40 con 0x7f0b240acf30 2024-01-27T01:06:50.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.052+0000 7f0b0b7fe700 1 --2- 172.21.15.46:0/1119946519 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0b10066f30 0x7f0b100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:50.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.052+0000 7f0b0b7fe700 1 -- 172.21.15.46:0/1119946519 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f0b140863b0 con 0x7f0b240acf30 2024-01-27T01:06:50.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.052+0000 7f0b22ffd700 1 --2- 172.21.15.46:0/1119946519 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0b10066f30 0x7f0b100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:50.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.053+0000 7f0b22ffd700 1 --2- 172.21.15.46:0/1119946519 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0b10066f30 0x7f0b100693e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0b2416d040 tx=0x7f0b180091a0 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:50.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.057+0000 7f0b0b7fe700 1 -- 172.21.15.46:0/1119946519 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0b240b2d60 con 0x7f0b240acf30 2024-01-27T01:06:50.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.257+0000 7f0b29e9e700 1 -- 172.21.15.46:0/1119946519 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0b240b2d60 con 0x7f0b10066f30 2024-01-27T01:06:50.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.271+0000 7f0b0b7fe700 1 -- 172.21.15.46:0/1119946519 <== mgr.9101 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528848 (secure 0 0 0) 0x7f0b240b2d60 con 0x7f0b10066f30 2024-01-27T01:06:50.284 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:50.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.276+0000 7f0b29e9e700 1 -- 172.21.15.46:0/1119946519 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0b10066f30 msgr2=0x7f0b100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:50.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.276+0000 7f0b29e9e700 1 --2- 172.21.15.46:0/1119946519 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0b10066f30 0x7f0b100693e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0b2416d040 tx=0x7f0b180091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.276+0000 7f0b29e9e700 1 -- 172.21.15.46:0/1119946519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b240acf30 msgr2=0x7f0b24148430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:50.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.276+0000 7f0b29e9e700 1 --2- 172.21.15.46:0/1119946519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b240acf30 0x7f0b24148430 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f0b2406ef50 tx=0x7f0b14017000 comp rx=0 tx=0).stop 2024-01-27T01:06:50.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.276+0000 7f0b29e9e700 1 -- 172.21.15.46:0/1119946519 shutdown_connections 2024-01-27T01:06:50.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.276+0000 7f0b29e9e700 1 --2- 172.21.15.46:0/1119946519 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0b10066f30 0x7f0b100693e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.276+0000 7f0b29e9e700 1 --2- 172.21.15.46:0/1119946519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b240acf30 0x7f0b24148430 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.276+0000 7f0b29e9e700 1 --2- 172.21.15.46:0/1119946519 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0b240ad780 0x7f0b24156270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.276+0000 7f0b29e9e700 1 --2- 172.21.15.46:0/1119946519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0b240ad300 0x7f0b240b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.276+0000 7f0b29e9e700 1 -- 172.21.15.46:0/1119946519 >> 172.21.15.46:0/1119946519 conn(0x7f0b24097820 msgr2=0x7f0b240ab6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:50.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.276+0000 7f0b29e9e700 1 -- 172.21.15.46:0/1119946519 shutdown_connections 2024-01-27T01:06:50.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.276+0000 7f0b29e9e700 1 -- 172.21.15.46:0/1119946519 wait complete. 2024-01-27T01:06:50.295 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:06:50.296 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-01-27T01:06:48.735761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596212,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":721,"num_write_kb":3490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8829,"ondisk_log_size":8829,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":376,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2467400,"kb_used_data":23560,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752507320,"statfs":{"total":773094113280,"available":770567495680,"internally_reserved":0,"allocated":24125440,"data_stored":13236688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002216"},"pg_stats":[{"pgid":"1.0","version":"213'8750","reported_seq":9668,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.932851+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:06:40.932851+0000","last_peered":"2024-01-27T01:06:40.932851+0000","last_clean":"2024-01-27T01:06:40.932851+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:06:40.932851+0000","last_undegraded":"2024-01-27T01:06:40.932851+0000","last_fullsized":"2024-01-27T01:06:40.932851+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8750,"log_dups_size":0,"ondisk_log_size":8750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":630,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"222'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923976+0000","last_change":"2024-01-27T01:06:25.858609+0000","last_active":"2024-01-27T01:06:40.923976+0000","last_peered":"2024-01-27T01:06:40.923976+0000","last_clean":"2024-01-27T01:06:40.923976+0000","last_became_active":"2024-01-27T01:06:25.857638+0000","last_became_peered":"2024-01-27T01:06:25.857638+0000","last_unstale":"2024-01-27T01:06:40.923976+0000","last_undegraded":"2024-01-27T01:06:40.923976+0000","last_fullsized":"2024-01-27T01:06:40.923976+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:35:59.109571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.927920+0000","last_change":"2024-01-27T01:06:26.873783+0000","last_active":"2024-01-27T01:06:40.927920+0000","last_peered":"2024-01-27T01:06:40.927920+0000","last_clean":"2024-01-27T01:06:40.927920+0000","last_became_active":"2024-01-27T01:06:26.873490+0000","last_became_peered":"2024-01-27T01:06:26.873490+0000","last_unstale":"2024-01-27T01:06:40.927920+0000","last_undegraded":"2024-01-27T01:06:40.927920+0000","last_fullsized":"2024-01-27T01:06:40.927920+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:32.691743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.954737+0000","last_change":"2024-01-27T01:06:39.937077+0000","last_active":"2024-01-27T01:06:40.954737+0000","last_peered":"2024-01-27T01:06:40.954737+0000","last_clean":"2024-01-27T01:06:40.954737+0000","last_became_active":"2024-01-27T01:06:39.936782+0000","last_became_peered":"2024-01-27T01:06:39.936782+0000","last_unstale":"2024-01-27T01:06:40.954737+0000","last_undegraded":"2024-01-27T01:06:40.954737+0000","last_fullsized":"2024-01-27T01:06:40.954737+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:40.845691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.943895+0000","last_change":"2024-01-27T01:06:40.942047+0000","last_active":"2024-01-27T01:06:40.943895+0000","last_peered":"2024-01-27T01:06:40.943895+0000","last_clean":"2024-01-27T01:06:40.943895+0000","last_became_active":"2024-01-27T01:06:40.941754+0000","last_became_peered":"2024-01-27T01:06:40.941754+0000","last_unstale":"2024-01-27T01:06:40.943895+0000","last_undegraded":"2024-01-27T01:06:40.943895+0000","last_fullsized":"2024-01-27T01:06:40.943895+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:27:01.280156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.938272+0000","last_change":"2024-01-27T01:06:25.865235+0000","last_active":"2024-01-27T01:06:40.938272+0000","last_peered":"2024-01-27T01:06:40.938272+0000","last_clean":"2024-01-27T01:06:40.938272+0000","last_became_active":"2024-01-27T01:06:25.864872+0000","last_became_peered":"2024-01-27T01:06:25.864872+0000","last_unstale":"2024-01-27T01:06:40.938272+0000","last_undegraded":"2024-01-27T01:06:40.938272+0000","last_fullsized":"2024-01-27T01:06:40.938272+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:53:38.618410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919351+0000","last_change":"2024-01-27T01:06:26.879306+0000","last_active":"2024-01-27T01:06:40.919351+0000","last_peered":"2024-01-27T01:06:40.919351+0000","last_clean":"2024-01-27T01:06:40.919351+0000","last_became_active":"2024-01-27T01:06:26.879071+0000","last_became_peered":"2024-01-27T01:06:26.879071+0000","last_unstale":"2024-01-27T01:06:40.919351+0000","last_undegraded":"2024-01-27T01:06:40.919351+0000","last_fullsized":"2024-01-27T01:06:40.919351+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:50:46.965359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919137+0000","last_change":"2024-01-27T01:06:39.930566+0000","last_active":"2024-01-27T01:06:40.919137+0000","last_peered":"2024-01-27T01:06:40.919137+0000","last_clean":"2024-01-27T01:06:40.919137+0000","last_became_active":"2024-01-27T01:06:39.930266+0000","last_became_peered":"2024-01-27T01:06:39.930266+0000","last_unstale":"2024-01-27T01:06:40.919137+0000","last_undegraded":"2024-01-27T01:06:40.919137+0000","last_fullsized":"2024-01-27T01:06:40.919137+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:33:45.602108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.948093+0000","last_change":"2024-01-27T01:06:40.944710+0000","last_active":"2024-01-27T01:06:40.948093+0000","last_peered":"2024-01-27T01:06:40.948093+0000","last_clean":"2024-01-27T01:06:40.948093+0000","last_became_active":"2024-01-27T01:06:40.944030+0000","last_became_peered":"2024-01-27T01:06:40.944030+0000","last_unstale":"2024-01-27T01:06:40.948093+0000","last_undegraded":"2024-01-27T01:06:40.948093+0000","last_fullsized":"2024-01-27T01:06:40.948093+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:12:42.423061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923325+0000","last_change":"2024-01-27T01:06:25.856035+0000","last_active":"2024-01-27T01:06:40.923325+0000","last_peered":"2024-01-27T01:06:40.923325+0000","last_clean":"2024-01-27T01:06:40.923325+0000","last_became_active":"2024-01-27T01:06:25.855762+0000","last_became_peered":"2024-01-27T01:06:25.855762+0000","last_unstale":"2024-01-27T01:06:40.923325+0000","last_undegraded":"2024-01-27T01:06:40.923325+0000","last_fullsized":"2024-01-27T01:06:40.923325+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:51:44.378880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931904+0000","last_change":"2024-01-27T01:06:26.870626+0000","last_active":"2024-01-27T01:06:40.931904+0000","last_peered":"2024-01-27T01:06:40.931904+0000","last_clean":"2024-01-27T01:06:40.931904+0000","last_became_active":"2024-01-27T01:06:26.870457+0000","last_became_peered":"2024-01-27T01:06:26.870457+0000","last_unstale":"2024-01-27T01:06:40.931904+0000","last_undegraded":"2024-01-27T01:06:40.931904+0000","last_fullsized":"2024-01-27T01:06:40.931904+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:42.766155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.953645+0000","last_change":"2024-01-27T01:06:39.934530+0000","last_active":"2024-01-27T01:06:40.953645+0000","last_peered":"2024-01-27T01:06:40.953645+0000","last_clean":"2024-01-27T01:06:40.953645+0000","last_became_active":"2024-01-27T01:06:39.934264+0000","last_became_peered":"2024-01-27T01:06:39.934264+0000","last_unstale":"2024-01-27T01:06:40.953645+0000","last_undegraded":"2024-01-27T01:06:40.953645+0000","last_fullsized":"2024-01-27T01:06:40.953645+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:26:34.409044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.946078+0000","last_change":"2024-01-27T01:06:40.941588+0000","last_active":"2024-01-27T01:06:40.946078+0000","last_peered":"2024-01-27T01:06:40.946078+0000","last_clean":"2024-01-27T01:06:40.946078+0000","last_became_active":"2024-01-27T01:06:40.936001+0000","last_became_peered":"2024-01-27T01:06:40.936001+0000","last_unstale":"2024-01-27T01:06:40.946078+0000","last_undegraded":"2024-01-27T01:06:40.946078+0000","last_fullsized":"2024-01-27T01:06:40.946078+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:37:42.686040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923654+0000","last_change":"2024-01-27T01:06:25.855015+0000","last_active":"2024-01-27T01:06:40.923654+0000","last_peered":"2024-01-27T01:06:40.923654+0000","last_clean":"2024-01-27T01:06:40.923654+0000","last_became_active":"2024-01-27T01:06:25.854708+0000","last_became_peered":"2024-01-27T01:06:25.854708+0000","last_unstale":"2024-01-27T01:06:40.923654+0000","last_undegraded":"2024-01-27T01:06:40.923654+0000","last_fullsized":"2024-01-27T01:06:40.923654+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:03.971945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.933878+0000","last_change":"2024-01-27T01:06:26.876182+0000","last_active":"2024-01-27T01:06:40.933878+0000","last_peered":"2024-01-27T01:06:40.933878+0000","last_clean":"2024-01-27T01:06:40.933878+0000","last_became_active":"2024-01-27T01:06:26.875894+0000","last_became_peered":"2024-01-27T01:06:26.875894+0000","last_unstale":"2024-01-27T01:06:40.933878+0000","last_undegraded":"2024-01-27T01:06:40.933878+0000","last_fullsized":"2024-01-27T01:06:40.933878+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:58.495062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.927024+0000","last_change":"2024-01-27T01:06:39.939172+0000","last_active":"2024-01-27T01:06:40.927024+0000","last_peered":"2024-01-27T01:06:40.927024+0000","last_clean":"2024-01-27T01:06:40.927024+0000","last_became_active":"2024-01-27T01:06:39.938766+0000","last_became_peered":"2024-01-27T01:06:39.938766+0000","last_unstale":"2024-01-27T01:06:40.927024+0000","last_undegraded":"2024-01-27T01:06:40.927024+0000","last_fullsized":"2024-01-27T01:06:40.927024+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:33:45.602108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.952716+0000","last_change":"2024-01-27T01:06:40.943896+0000","last_active":"2024-01-27T01:06:40.952716+0000","last_peered":"2024-01-27T01:06:40.952716+0000","last_clean":"2024-01-27T01:06:40.952716+0000","last_became_active":"2024-01-27T01:06:40.943570+0000","last_became_peered":"2024-01-27T01:06:40.943570+0000","last_unstale":"2024-01-27T01:06:40.952716+0000","last_undegraded":"2024-01-27T01:06:40.952716+0000","last_fullsized":"2024-01-27T01:06:40.952716+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:42.451296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926823+0000","last_change":"2024-01-27T01:06:25.858142+0000","last_active":"2024-01-27T01:06:40.926823+0000","last_peered":"2024-01-27T01:06:40.926823+0000","last_clean":"2024-01-27T01:06:40.926823+0000","last_became_active":"2024-01-27T01:06:25.857867+0000","last_became_peered":"2024-01-27T01:06:25.857867+0000","last_unstale":"2024-01-27T01:06:40.926823+0000","last_undegraded":"2024-01-27T01:06:40.926823+0000","last_fullsized":"2024-01-27T01:06:40.926823+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:10:59.780192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.927155+0000","last_change":"2024-01-27T01:06:26.880503+0000","last_active":"2024-01-27T01:06:40.927155+0000","last_peered":"2024-01-27T01:06:40.927155+0000","last_clean":"2024-01-27T01:06:40.927155+0000","last_became_active":"2024-01-27T01:06:26.880224+0000","last_became_peered":"2024-01-27T01:06:26.880224+0000","last_unstale":"2024-01-27T01:06:40.927155+0000","last_undegraded":"2024-01-27T01:06:40.927155+0000","last_fullsized":"2024-01-27T01:06:40.927155+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:27:33.886740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919769+0000","last_change":"2024-01-27T01:06:39.928493+0000","last_active":"2024-01-27T01:06:40.919769+0000","last_peered":"2024-01-27T01:06:40.919769+0000","last_clean":"2024-01-27T01:06:40.919769+0000","last_became_active":"2024-01-27T01:06:39.928166+0000","last_became_peered":"2024-01-27T01:06:39.928166+0000","last_unstale":"2024-01-27T01:06:40.919769+0000","last_undegraded":"2024-01-27T01:06:40.919769+0000","last_fullsized":"2024-01-27T01:06:40.919769+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:15:24.362081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942986+0000","last_change":"2024-01-27T01:06:40.938135+0000","last_active":"2024-01-27T01:06:40.942986+0000","last_peered":"2024-01-27T01:06:40.942986+0000","last_clean":"2024-01-27T01:06:40.942986+0000","last_became_active":"2024-01-27T01:06:40.937878+0000","last_became_peered":"2024-01-27T01:06:40.937878+0000","last_unstale":"2024-01-27T01:06:40.942986+0000","last_undegraded":"2024-01-27T01:06:40.942986+0000","last_fullsized":"2024-01-27T01:06:40.942986+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:31:19.372123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.927955+0000","last_change":"2024-01-27T01:06:25.859342+0000","last_active":"2024-01-27T01:06:40.927955+0000","last_peered":"2024-01-27T01:06:40.927955+0000","last_clean":"2024-01-27T01:06:40.927955+0000","last_became_active":"2024-01-27T01:06:25.859065+0000","last_became_peered":"2024-01-27T01:06:25.859065+0000","last_unstale":"2024-01-27T01:06:40.927955+0000","last_undegraded":"2024-01-27T01:06:40.927955+0000","last_fullsized":"2024-01-27T01:06:40.927955+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:00:54.297085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925207+0000","last_change":"2024-01-27T01:06:26.870597+0000","last_active":"2024-01-27T01:06:40.925207+0000","last_peered":"2024-01-27T01:06:40.925207+0000","last_clean":"2024-01-27T01:06:40.925207+0000","last_became_active":"2024-01-27T01:06:26.870334+0000","last_became_peered":"2024-01-27T01:06:26.870334+0000","last_unstale":"2024-01-27T01:06:40.925207+0000","last_undegraded":"2024-01-27T01:06:40.925207+0000","last_fullsized":"2024-01-27T01:06:40.925207+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:01:23.266072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.933049+0000","last_change":"2024-01-27T01:06:39.933497+0000","last_active":"2024-01-27T01:06:40.933049+0000","last_peered":"2024-01-27T01:06:40.933049+0000","last_clean":"2024-01-27T01:06:40.933049+0000","last_became_active":"2024-01-27T01:06:39.933215+0000","last_became_peered":"2024-01-27T01:06:39.933215+0000","last_unstale":"2024-01-27T01:06:40.933049+0000","last_undegraded":"2024-01-27T01:06:40.933049+0000","last_fullsized":"2024-01-27T01:06:40.933049+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:22:36.612223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942697+0000","last_change":"2024-01-27T01:06:40.939651+0000","last_active":"2024-01-27T01:06:40.942697+0000","last_peered":"2024-01-27T01:06:40.942697+0000","last_clean":"2024-01-27T01:06:40.942697+0000","last_became_active":"2024-01-27T01:06:40.939320+0000","last_became_peered":"2024-01-27T01:06:40.939320+0000","last_unstale":"2024-01-27T01:06:40.942697+0000","last_undegraded":"2024-01-27T01:06:40.942697+0000","last_fullsized":"2024-01-27T01:06:40.942697+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:05:34.497339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.932963+0000","last_change":"2024-01-27T01:06:25.858032+0000","last_active":"2024-01-27T01:06:40.932963+0000","last_peered":"2024-01-27T01:06:40.932963+0000","last_clean":"2024-01-27T01:06:40.932963+0000","last_became_active":"2024-01-27T01:06:25.857777+0000","last_became_peered":"2024-01-27T01:06:25.857777+0000","last_unstale":"2024-01-27T01:06:40.932963+0000","last_undegraded":"2024-01-27T01:06:40.932963+0000","last_fullsized":"2024-01-27T01:06:40.932963+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:09:08.458850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923226+0000","last_change":"2024-01-27T01:06:26.872190+0000","last_active":"2024-01-27T01:06:40.923226+0000","last_peered":"2024-01-27T01:06:40.923226+0000","last_clean":"2024-01-27T01:06:40.923226+0000","last_became_active":"2024-01-27T01:06:26.867718+0000","last_became_peered":"2024-01-27T01:06:26.867718+0000","last_unstale":"2024-01-27T01:06:40.923226+0000","last_undegraded":"2024-01-27T01:06:40.923226+0000","last_fullsized":"2024-01-27T01:06:40.923226+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:23.301132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.927192+0000","last_change":"2024-01-27T01:06:39.941456+0000","last_active":"2024-01-27T01:06:40.927192+0000","last_peered":"2024-01-27T01:06:40.927192+0000","last_clean":"2024-01-27T01:06:40.927192+0000","last_became_active":"2024-01-27T01:06:39.940373+0000","last_became_peered":"2024-01-27T01:06:39.940373+0000","last_unstale":"2024-01-27T01:06:40.927192+0000","last_undegraded":"2024-01-27T01:06:40.927192+0000","last_fullsized":"2024-01-27T01:06:40.927192+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:18:25.975883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.949946+0000","last_change":"2024-01-27T01:06:40.942226+0000","last_active":"2024-01-27T01:06:40.949946+0000","last_peered":"2024-01-27T01:06:40.949946+0000","last_clean":"2024-01-27T01:06:40.949946+0000","last_became_active":"2024-01-27T01:06:40.941911+0000","last_became_peered":"2024-01-27T01:06:40.941911+0000","last_unstale":"2024-01-27T01:06:40.949946+0000","last_undegraded":"2024-01-27T01:06:40.949946+0000","last_fullsized":"2024-01-27T01:06:40.949946+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:27:18.007273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.6","version":"223'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.928859+0000","last_change":"2024-01-27T01:06:25.861717+0000","last_active":"2024-01-27T01:06:40.928859+0000","last_peered":"2024-01-27T01:06:40.928859+0000","last_clean":"2024-01-27T01:06:40.928859+0000","last_became_active":"2024-01-27T01:06:25.861177+0000","last_became_peered":"2024-01-27T01:06:25.861177+0000","last_unstale":"2024-01-27T01:06:40.928859+0000","last_undegraded":"2024-01-27T01:06:40.928859+0000","last_fullsized":"2024-01-27T01:06:40.928859+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:05.596045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919205+0000","last_change":"2024-01-27T01:06:26.878650+0000","last_active":"2024-01-27T01:06:40.919205+0000","last_peered":"2024-01-27T01:06:40.919205+0000","last_clean":"2024-01-27T01:06:40.919205+0000","last_became_active":"2024-01-27T01:06:26.877596+0000","last_became_peered":"2024-01-27T01:06:26.877596+0000","last_unstale":"2024-01-27T01:06:40.919205+0000","last_undegraded":"2024-01-27T01:06:40.919205+0000","last_fullsized":"2024-01-27T01:06:40.919205+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:42:48.942202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926127+0000","last_change":"2024-01-27T01:06:39.934375+0000","last_active":"2024-01-27T01:06:40.926127+0000","last_peered":"2024-01-27T01:06:40.926127+0000","last_clean":"2024-01-27T01:06:40.926127+0000","last_became_active":"2024-01-27T01:06:39.934098+0000","last_became_peered":"2024-01-27T01:06:39.934098+0000","last_unstale":"2024-01-27T01:06:40.926127+0000","last_undegraded":"2024-01-27T01:06:40.926127+0000","last_fullsized":"2024-01-27T01:06:40.926127+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:13:12.942303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.945722+0000","last_change":"2024-01-27T01:06:40.940649+0000","last_active":"2024-01-27T01:06:40.945722+0000","last_peered":"2024-01-27T01:06:40.945722+0000","last_clean":"2024-01-27T01:06:40.945722+0000","last_became_active":"2024-01-27T01:06:40.940389+0000","last_became_peered":"2024-01-27T01:06:40.940389+0000","last_unstale":"2024-01-27T01:06:40.945722+0000","last_undegraded":"2024-01-27T01:06:40.945722+0000","last_fullsized":"2024-01-27T01:06:40.945722+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:08:47.665701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"222'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925560+0000","last_change":"2024-01-27T01:06:25.857933+0000","last_active":"2024-01-27T01:06:40.925560+0000","last_peered":"2024-01-27T01:06:40.925560+0000","last_clean":"2024-01-27T01:06:40.925560+0000","last_became_active":"2024-01-27T01:06:25.857406+0000","last_became_peered":"2024-01-27T01:06:25.857406+0000","last_unstale":"2024-01-27T01:06:40.925560+0000","last_undegraded":"2024-01-27T01:06:40.925560+0000","last_fullsized":"2024-01-27T01:06:40.925560+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:20:20.239633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926501+0000","last_change":"2024-01-27T01:06:26.875222+0000","last_active":"2024-01-27T01:06:40.926501+0000","last_peered":"2024-01-27T01:06:40.926501+0000","last_clean":"2024-01-27T01:06:40.926501+0000","last_became_active":"2024-01-27T01:06:26.874903+0000","last_became_peered":"2024-01-27T01:06:26.874903+0000","last_unstale":"2024-01-27T01:06:40.926501+0000","last_undegraded":"2024-01-27T01:06:40.926501+0000","last_fullsized":"2024-01-27T01:06:40.926501+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:09:33.903445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.932982+0000","last_change":"2024-01-27T01:06:39.938367+0000","last_active":"2024-01-27T01:06:40.932982+0000","last_peered":"2024-01-27T01:06:40.932982+0000","last_clean":"2024-01-27T01:06:40.932982+0000","last_became_active":"2024-01-27T01:06:39.938071+0000","last_became_peered":"2024-01-27T01:06:39.938071+0000","last_unstale":"2024-01-27T01:06:40.932982+0000","last_undegraded":"2024-01-27T01:06:40.932982+0000","last_fullsized":"2024-01-27T01:06:40.932982+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:26:34.409044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.947647+0000","last_change":"2024-01-27T01:06:40.939338+0000","last_active":"2024-01-27T01:06:40.947647+0000","last_peered":"2024-01-27T01:06:40.947647+0000","last_clean":"2024-01-27T01:06:40.947647+0000","last_became_active":"2024-01-27T01:06:40.939008+0000","last_became_peered":"2024-01-27T01:06:40.939008+0000","last_unstale":"2024-01-27T01:06:40.947647+0000","last_undegraded":"2024-01-27T01:06:40.947647+0000","last_fullsized":"2024-01-27T01:06:40.947647+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:08:05.712678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.924020+0000","last_change":"2024-01-27T01:06:25.853605+0000","last_active":"2024-01-27T01:06:40.924020+0000","last_peered":"2024-01-27T01:06:40.924020+0000","last_clean":"2024-01-27T01:06:40.924020+0000","last_became_active":"2024-01-27T01:06:25.853331+0000","last_became_peered":"2024-01-27T01:06:25.853331+0000","last_unstale":"2024-01-27T01:06:40.924020+0000","last_undegraded":"2024-01-27T01:06:40.924020+0000","last_fullsized":"2024-01-27T01:06:40.924020+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:09:08.458850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.928716+0000","last_change":"2024-01-27T01:06:26.875536+0000","last_active":"2024-01-27T01:06:40.928716+0000","last_peered":"2024-01-27T01:06:40.928716+0000","last_clean":"2024-01-27T01:06:40.928716+0000","last_became_active":"2024-01-27T01:06:26.875267+0000","last_became_peered":"2024-01-27T01:06:26.875267+0000","last_unstale":"2024-01-27T01:06:40.928716+0000","last_undegraded":"2024-01-27T01:06:40.928716+0000","last_fullsized":"2024-01-27T01:06:40.928716+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:32.534622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.955063+0000","last_change":"2024-01-27T01:06:39.937034+0000","last_active":"2024-01-27T01:06:40.955063+0000","last_peered":"2024-01-27T01:06:40.955063+0000","last_clean":"2024-01-27T01:06:40.955063+0000","last_became_active":"2024-01-27T01:06:39.936709+0000","last_became_peered":"2024-01-27T01:06:39.936709+0000","last_unstale":"2024-01-27T01:06:40.955063+0000","last_undegraded":"2024-01-27T01:06:40.955063+0000","last_fullsized":"2024-01-27T01:06:40.955063+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:35.078462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.947436+0000","last_change":"2024-01-27T01:06:40.939425+0000","last_active":"2024-01-27T01:06:40.947436+0000","last_peered":"2024-01-27T01:06:40.947436+0000","last_clean":"2024-01-27T01:06:40.947436+0000","last_became_active":"2024-01-27T01:06:40.939162+0000","last_became_peered":"2024-01-27T01:06:40.939162+0000","last_unstale":"2024-01-27T01:06:40.947436+0000","last_undegraded":"2024-01-27T01:06:40.947436+0000","last_fullsized":"2024-01-27T01:06:40.947436+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:14:19.010092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.934093+0000","last_change":"2024-01-27T01:06:25.862199+0000","last_active":"2024-01-27T01:06:40.934093+0000","last_peered":"2024-01-27T01:06:40.934093+0000","last_clean":"2024-01-27T01:06:40.934093+0000","last_became_active":"2024-01-27T01:06:25.861876+0000","last_became_peered":"2024-01-27T01:06:25.861876+0000","last_unstale":"2024-01-27T01:06:40.934093+0000","last_undegraded":"2024-01-27T01:06:40.934093+0000","last_fullsized":"2024-01-27T01:06:40.934093+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:00:54.297085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925485+0000","last_change":"2024-01-27T01:06:26.867570+0000","last_active":"2024-01-27T01:06:40.925485+0000","last_peered":"2024-01-27T01:06:40.925485+0000","last_clean":"2024-01-27T01:06:40.925485+0000","last_became_active":"2024-01-27T01:06:26.866517+0000","last_became_peered":"2024-01-27T01:06:26.866517+0000","last_unstale":"2024-01-27T01:06:40.925485+0000","last_undegraded":"2024-01-27T01:06:40.925485+0000","last_fullsized":"2024-01-27T01:06:40.925485+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:09:54.515702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.955677+0000","last_change":"2024-01-27T01:06:39.938159+0000","last_active":"2024-01-27T01:06:40.955677+0000","last_peered":"2024-01-27T01:06:40.955677+0000","last_clean":"2024-01-27T01:06:40.955677+0000","last_became_active":"2024-01-27T01:06:39.937871+0000","last_became_peered":"2024-01-27T01:06:39.937871+0000","last_unstale":"2024-01-27T01:06:40.955677+0000","last_undegraded":"2024-01-27T01:06:40.955677+0000","last_fullsized":"2024-01-27T01:06:40.955677+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:36.367666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.946099+0000","last_change":"2024-01-27T01:06:40.941996+0000","last_active":"2024-01-27T01:06:40.946099+0000","last_peered":"2024-01-27T01:06:40.946099+0000","last_clean":"2024-01-27T01:06:40.946099+0000","last_became_active":"2024-01-27T01:06:40.936467+0000","last_became_peered":"2024-01-27T01:06:40.936467+0000","last_unstale":"2024-01-27T01:06:40.946099+0000","last_undegraded":"2024-01-27T01:06:40.946099+0000","last_fullsized":"2024-01-27T01:06:40.946099+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:47:39.379533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.934281+0000","last_change":"2024-01-27T01:06:25.851595+0000","last_active":"2024-01-27T01:06:40.934281+0000","last_peered":"2024-01-27T01:06:40.934281+0000","last_clean":"2024-01-27T01:06:40.934281+0000","last_became_active":"2024-01-27T01:06:25.851260+0000","last_became_peered":"2024-01-27T01:06:25.851260+0000","last_unstale":"2024-01-27T01:06:40.934281+0000","last_undegraded":"2024-01-27T01:06:40.934281+0000","last_fullsized":"2024-01-27T01:06:40.934281+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:56:53.179347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.933644+0000","last_change":"2024-01-27T01:06:26.875858+0000","last_active":"2024-01-27T01:06:40.933644+0000","last_peered":"2024-01-27T01:06:40.933644+0000","last_clean":"2024-01-27T01:06:40.933644+0000","last_became_active":"2024-01-27T01:06:26.875564+0000","last_became_peered":"2024-01-27T01:06:26.875564+0000","last_unstale":"2024-01-27T01:06:40.933644+0000","last_undegraded":"2024-01-27T01:06:40.933644+0000","last_fullsized":"2024-01-27T01:06:40.933644+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:43.630831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926797+0000","last_change":"2024-01-27T01:06:39.938832+0000","last_active":"2024-01-27T01:06:40.926797+0000","last_peered":"2024-01-27T01:06:40.926797+0000","last_clean":"2024-01-27T01:06:40.926797+0000","last_became_active":"2024-01-27T01:06:39.938365+0000","last_became_peered":"2024-01-27T01:06:39.938365+0000","last_unstale":"2024-01-27T01:06:40.926797+0000","last_undegraded":"2024-01-27T01:06:40.926797+0000","last_fullsized":"2024-01-27T01:06:40.926797+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:31.265591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.944315+0000","last_change":"2024-01-27T01:06:40.940315+0000","last_active":"2024-01-27T01:06:40.944315+0000","last_peered":"2024-01-27T01:06:40.944315+0000","last_clean":"2024-01-27T01:06:40.944315+0000","last_became_active":"2024-01-27T01:06:40.940058+0000","last_became_peered":"2024-01-27T01:06:40.940058+0000","last_unstale":"2024-01-27T01:06:40.944315+0000","last_undegraded":"2024-01-27T01:06:40.944315+0000","last_fullsized":"2024-01-27T01:06:40.944315+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:19:37.373751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.b","version":"222'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.939300+0000","last_change":"2024-01-27T01:06:25.860572+0000","last_active":"2024-01-27T01:06:40.939300+0000","last_peered":"2024-01-27T01:06:40.939300+0000","last_clean":"2024-01-27T01:06:40.939300+0000","last_became_active":"2024-01-27T01:06:25.860248+0000","last_became_peered":"2024-01-27T01:06:25.860248+0000","last_unstale":"2024-01-27T01:06:40.939300+0000","last_undegraded":"2024-01-27T01:06:40.939300+0000","last_fullsized":"2024-01-27T01:06:40.939300+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:01:25.741123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931847+0000","last_change":"2024-01-27T01:06:26.871346+0000","last_active":"2024-01-27T01:06:40.931847+0000","last_peered":"2024-01-27T01:06:40.931847+0000","last_clean":"2024-01-27T01:06:40.931847+0000","last_became_active":"2024-01-27T01:06:26.870999+0000","last_became_peered":"2024-01-27T01:06:26.870999+0000","last_unstale":"2024-01-27T01:06:40.931847+0000","last_undegraded":"2024-01-27T01:06:40.931847+0000","last_fullsized":"2024-01-27T01:06:40.931847+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:27.779157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.954555+0000","last_change":"2024-01-27T01:06:39.932938+0000","last_active":"2024-01-27T01:06:40.954555+0000","last_peered":"2024-01-27T01:06:40.954555+0000","last_clean":"2024-01-27T01:06:40.954555+0000","last_became_active":"2024-01-27T01:06:39.930096+0000","last_became_peered":"2024-01-27T01:06:39.930096+0000","last_unstale":"2024-01-27T01:06:40.954555+0000","last_undegraded":"2024-01-27T01:06:40.954555+0000","last_fullsized":"2024-01-27T01:06:40.954555+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:57:15.231912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.944057+0000","last_change":"2024-01-27T01:06:40.943101+0000","last_active":"2024-01-27T01:06:40.944057+0000","last_peered":"2024-01-27T01:06:40.944057+0000","last_clean":"2024-01-27T01:06:40.944057+0000","last_became_active":"2024-01-27T01:06:40.942787+0000","last_became_peered":"2024-01-27T01:06:40.942787+0000","last_unstale":"2024-01-27T01:06:40.944057+0000","last_undegraded":"2024-01-27T01:06:40.944057+0000","last_fullsized":"2024-01-27T01:06:40.944057+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:27:25.230045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.941219+0000","last_change":"2024-01-27T01:06:25.861650+0000","last_active":"2024-01-27T01:06:40.941219+0000","last_peered":"2024-01-27T01:06:40.941219+0000","last_clean":"2024-01-27T01:06:40.941219+0000","last_became_active":"2024-01-27T01:06:25.861342+0000","last_became_peered":"2024-01-27T01:06:25.861342+0000","last_unstale":"2024-01-27T01:06:40.941219+0000","last_undegraded":"2024-01-27T01:06:40.941219+0000","last_fullsized":"2024-01-27T01:06:40.941219+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:44:15.953217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926988+0000","last_change":"2024-01-27T01:06:26.876619+0000","last_active":"2024-01-27T01:06:40.926988+0000","last_peered":"2024-01-27T01:06:40.926988+0000","last_clean":"2024-01-27T01:06:40.926988+0000","last_became_active":"2024-01-27T01:06:26.875927+0000","last_became_peered":"2024-01-27T01:06:26.875927+0000","last_unstale":"2024-01-27T01:06:40.926988+0000","last_undegraded":"2024-01-27T01:06:40.926988+0000","last_fullsized":"2024-01-27T01:06:40.926988+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:02.000909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919397+0000","last_change":"2024-01-27T01:06:39.929131+0000","last_active":"2024-01-27T01:06:40.919397+0000","last_peered":"2024-01-27T01:06:40.919397+0000","last_clean":"2024-01-27T01:06:40.919397+0000","last_became_active":"2024-01-27T01:06:39.927860+0000","last_became_peered":"2024-01-27T01:06:39.927860+0000","last_unstale":"2024-01-27T01:06:40.919397+0000","last_undegraded":"2024-01-27T01:06:40.919397+0000","last_fullsized":"2024-01-27T01:06:40.919397+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:13:12.942303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.948177+0000","last_change":"2024-01-27T01:06:40.944322+0000","last_active":"2024-01-27T01:06:40.948177+0000","last_peered":"2024-01-27T01:06:40.948177+0000","last_clean":"2024-01-27T01:06:40.948177+0000","last_became_active":"2024-01-27T01:06:40.944008+0000","last_became_peered":"2024-01-27T01:06:40.944008+0000","last_unstale":"2024-01-27T01:06:40.948177+0000","last_undegraded":"2024-01-27T01:06:40.948177+0000","last_fullsized":"2024-01-27T01:06:40.948177+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:09:11.010297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.d","version":"222'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931425+0000","last_change":"2024-01-27T01:06:25.851720+0000","last_active":"2024-01-27T01:06:40.931425+0000","last_peered":"2024-01-27T01:06:40.931425+0000","last_clean":"2024-01-27T01:06:40.931425+0000","last_became_active":"2024-01-27T01:06:25.851416+0000","last_became_peered":"2024-01-27T01:06:25.851416+0000","last_unstale":"2024-01-27T01:06:40.931425+0000","last_undegraded":"2024-01-27T01:06:40.931425+0000","last_fullsized":"2024-01-27T01:06:40.931425+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:07.951814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925914+0000","last_change":"2024-01-27T01:06:26.870758+0000","last_active":"2024-01-27T01:06:40.925914+0000","last_peered":"2024-01-27T01:06:40.925914+0000","last_clean":"2024-01-27T01:06:40.925914+0000","last_became_active":"2024-01-27T01:06:26.870130+0000","last_became_peered":"2024-01-27T01:06:26.870130+0000","last_unstale":"2024-01-27T01:06:40.925914+0000","last_undegraded":"2024-01-27T01:06:40.925914+0000","last_fullsized":"2024-01-27T01:06:40.925914+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:36:31.625857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.954611+0000","last_change":"2024-01-27T01:06:39.934635+0000","last_active":"2024-01-27T01:06:40.954611+0000","last_peered":"2024-01-27T01:06:40.954611+0000","last_clean":"2024-01-27T01:06:40.954611+0000","last_became_active":"2024-01-27T01:06:39.934370+0000","last_became_peered":"2024-01-27T01:06:39.934370+0000","last_unstale":"2024-01-27T01:06:40.954611+0000","last_undegraded":"2024-01-27T01:06:40.954611+0000","last_fullsized":"2024-01-27T01:06:40.954611+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:40.845691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.945691+0000","last_change":"2024-01-27T01:06:40.938827+0000","last_active":"2024-01-27T01:06:40.945691+0000","last_peered":"2024-01-27T01:06:40.945691+0000","last_clean":"2024-01-27T01:06:40.945691+0000","last_became_active":"2024-01-27T01:06:40.938572+0000","last_became_peered":"2024-01-27T01:06:40.938572+0000","last_unstale":"2024-01-27T01:06:40.945691+0000","last_undegraded":"2024-01-27T01:06:40.945691+0000","last_fullsized":"2024-01-27T01:06:40.945691+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:38:42.855259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.e","version":"222'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.928649+0000","last_change":"2024-01-27T01:06:25.852245+0000","last_active":"2024-01-27T01:06:40.928649+0000","last_peered":"2024-01-27T01:06:40.928649+0000","last_clean":"2024-01-27T01:06:40.928649+0000","last_became_active":"2024-01-27T01:06:25.851982+0000","last_became_peered":"2024-01-27T01:06:25.851982+0000","last_unstale":"2024-01-27T01:06:40.928649+0000","last_undegraded":"2024-01-27T01:06:40.928649+0000","last_fullsized":"2024-01-27T01:06:40.928649+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:17:05.596045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923733+0000","last_change":"2024-01-27T01:06:26.879356+0000","last_active":"2024-01-27T01:06:40.923733+0000","last_peered":"2024-01-27T01:06:40.923733+0000","last_clean":"2024-01-27T01:06:40.923733+0000","last_became_active":"2024-01-27T01:06:26.878891+0000","last_became_peered":"2024-01-27T01:06:26.878891+0000","last_unstale":"2024-01-27T01:06:40.923733+0000","last_undegraded":"2024-01-27T01:06:40.923733+0000","last_fullsized":"2024-01-27T01:06:40.923733+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:22.011928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.937963+0000","last_change":"2024-01-27T01:06:39.937159+0000","last_active":"2024-01-27T01:06:40.937963+0000","last_peered":"2024-01-27T01:06:40.937963+0000","last_clean":"2024-01-27T01:06:40.937963+0000","last_became_active":"2024-01-27T01:06:39.936720+0000","last_became_peered":"2024-01-27T01:06:39.936720+0000","last_unstale":"2024-01-27T01:06:40.937963+0000","last_undegraded":"2024-01-27T01:06:40.937963+0000","last_fullsized":"2024-01-27T01:06:40.937963+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:45.758278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.940354+0000","last_change":"2024-01-27T01:06:40.935737+0000","last_active":"2024-01-27T01:06:40.940354+0000","last_peered":"2024-01-27T01:06:40.940354+0000","last_clean":"2024-01-27T01:06:40.940354+0000","last_became_active":"2024-01-27T01:06:40.935414+0000","last_became_peered":"2024-01-27T01:06:40.935414+0000","last_unstale":"2024-01-27T01:06:40.940354+0000","last_undegraded":"2024-01-27T01:06:40.940354+0000","last_fullsized":"2024-01-27T01:06:40.940354+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:16:08.266882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.f","version":"222'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.928246+0000","last_change":"2024-01-27T01:06:25.848900+0000","last_active":"2024-01-27T01:06:40.928246+0000","last_peered":"2024-01-27T01:06:40.928246+0000","last_clean":"2024-01-27T01:06:40.928246+0000","last_became_active":"2024-01-27T01:06:25.848483+0000","last_became_peered":"2024-01-27T01:06:25.848483+0000","last_unstale":"2024-01-27T01:06:40.928246+0000","last_undegraded":"2024-01-27T01:06:40.928246+0000","last_fullsized":"2024-01-27T01:06:40.928246+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:27:13.471892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926411+0000","last_change":"2024-01-27T01:06:26.875234+0000","last_active":"2024-01-27T01:06:40.926411+0000","last_peered":"2024-01-27T01:06:40.926411+0000","last_clean":"2024-01-27T01:06:40.926411+0000","last_became_active":"2024-01-27T01:06:26.874927+0000","last_became_peered":"2024-01-27T01:06:26.874927+0000","last_unstale":"2024-01-27T01:06:40.926411+0000","last_undegraded":"2024-01-27T01:06:40.926411+0000","last_fullsized":"2024-01-27T01:06:40.926411+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:10.390573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.918887+0000","last_change":"2024-01-27T01:06:39.931922+0000","last_active":"2024-01-27T01:06:40.918887+0000","last_peered":"2024-01-27T01:06:40.918887+0000","last_clean":"2024-01-27T01:06:40.918887+0000","last_became_active":"2024-01-27T01:06:39.931611+0000","last_became_peered":"2024-01-27T01:06:39.931611+0000","last_unstale":"2024-01-27T01:06:40.918887+0000","last_undegraded":"2024-01-27T01:06:40.918887+0000","last_fullsized":"2024-01-27T01:06:40.918887+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:40.845691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.945315+0000","last_change":"2024-01-27T01:06:40.941775+0000","last_active":"2024-01-27T01:06:40.945315+0000","last_peered":"2024-01-27T01:06:40.945315+0000","last_clean":"2024-01-27T01:06:40.945315+0000","last_became_active":"2024-01-27T01:06:40.941438+0000","last_became_peered":"2024-01-27T01:06:40.941438+0000","last_unstale":"2024-01-27T01:06:40.945315+0000","last_undegraded":"2024-01-27T01:06:40.945315+0000","last_fullsized":"2024-01-27T01:06:40.945315+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:50:46.505356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.10","version":"222'8","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.938162+0000","last_change":"2024-01-27T01:06:25.865239+0000","last_active":"2024-01-27T01:06:40.938162+0000","last_peered":"2024-01-27T01:06:40.938162+0000","last_clean":"2024-01-27T01:06:40.938162+0000","last_became_active":"2024-01-27T01:06:25.864943+0000","last_became_peered":"2024-01-27T01:06:25.864943+0000","last_unstale":"2024-01-27T01:06:40.938162+0000","last_undegraded":"2024-01-27T01:06:40.938162+0000","last_fullsized":"2024-01-27T01:06:40.938162+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:41:25.254751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.932277+0000","last_change":"2024-01-27T01:06:26.867077+0000","last_active":"2024-01-27T01:06:40.932277+0000","last_peered":"2024-01-27T01:06:40.932277+0000","last_clean":"2024-01-27T01:06:40.932277+0000","last_became_active":"2024-01-27T01:06:26.866787+0000","last_became_peered":"2024-01-27T01:06:26.866787+0000","last_unstale":"2024-01-27T01:06:40.932277+0000","last_undegraded":"2024-01-27T01:06:40.932277+0000","last_fullsized":"2024-01-27T01:06:40.932277+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:02.000909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931562+0000","last_change":"2024-01-27T01:06:39.936310+0000","last_active":"2024-01-27T01:06:40.931562+0000","last_peered":"2024-01-27T01:06:40.931562+0000","last_clean":"2024-01-27T01:06:40.931562+0000","last_became_active":"2024-01-27T01:06:39.936032+0000","last_became_peered":"2024-01-27T01:06:39.936032+0000","last_unstale":"2024-01-27T01:06:40.931562+0000","last_undegraded":"2024-01-27T01:06:40.931562+0000","last_fullsized":"2024-01-27T01:06:40.931562+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:55.832690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.948759+0000","last_change":"2024-01-27T01:06:40.941007+0000","last_active":"2024-01-27T01:06:40.948759+0000","last_peered":"2024-01-27T01:06:40.948759+0000","last_clean":"2024-01-27T01:06:40.948759+0000","last_became_active":"2024-01-27T01:06:40.940379+0000","last_became_peered":"2024-01-27T01:06:40.940379+0000","last_unstale":"2024-01-27T01:06:40.948759+0000","last_undegraded":"2024-01-27T01:06:40.948759+0000","last_fullsized":"2024-01-27T01:06:40.948759+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:14:19.010092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.922797+0000","last_change":"2024-01-27T01:06:25.853597+0000","last_active":"2024-01-27T01:06:40.922797+0000","last_peered":"2024-01-27T01:06:40.922797+0000","last_clean":"2024-01-27T01:06:40.922797+0000","last_became_active":"2024-01-27T01:06:25.853257+0000","last_became_peered":"2024-01-27T01:06:25.853257+0000","last_unstale":"2024-01-27T01:06:40.922797+0000","last_undegraded":"2024-01-27T01:06:40.922797+0000","last_fullsized":"2024-01-27T01:06:40.922797+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:54:35.011796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925065+0000","last_change":"2024-01-27T01:06:26.866613+0000","last_active":"2024-01-27T01:06:40.925065+0000","last_peered":"2024-01-27T01:06:40.925065+0000","last_clean":"2024-01-27T01:06:40.925065+0000","last_became_active":"2024-01-27T01:06:26.866298+0000","last_became_peered":"2024-01-27T01:06:26.866298+0000","last_unstale":"2024-01-27T01:06:40.925065+0000","last_undegraded":"2024-01-27T01:06:40.925065+0000","last_fullsized":"2024-01-27T01:06:40.925065+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:27:03.934654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.932185+0000","last_change":"2024-01-27T01:06:39.938479+0000","last_active":"2024-01-27T01:06:40.932185+0000","last_peered":"2024-01-27T01:06:40.932185+0000","last_clean":"2024-01-27T01:06:40.932185+0000","last_became_active":"2024-01-27T01:06:39.938172+0000","last_became_peered":"2024-01-27T01:06:39.938172+0000","last_unstale":"2024-01-27T01:06:40.932185+0000","last_undegraded":"2024-01-27T01:06:40.932185+0000","last_fullsized":"2024-01-27T01:06:40.932185+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:43:07.431223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.948795+0000","last_change":"2024-01-27T01:06:40.941899+0000","last_active":"2024-01-27T01:06:40.948795+0000","last_peered":"2024-01-27T01:06:40.948795+0000","last_clean":"2024-01-27T01:06:40.948795+0000","last_became_active":"2024-01-27T01:06:40.941618+0000","last_became_peered":"2024-01-27T01:06:40.941618+0000","last_unstale":"2024-01-27T01:06:40.948795+0000","last_undegraded":"2024-01-27T01:06:40.948795+0000","last_fullsized":"2024-01-27T01:06:40.948795+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:09:11.010297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919701+0000","last_change":"2024-01-27T01:06:25.854192+0000","last_active":"2024-01-27T01:06:40.919701+0000","last_peered":"2024-01-27T01:06:40.919701+0000","last_clean":"2024-01-27T01:06:40.919701+0000","last_became_active":"2024-01-27T01:06:25.853601+0000","last_became_peered":"2024-01-27T01:06:25.853601+0000","last_unstale":"2024-01-27T01:06:40.919701+0000","last_undegraded":"2024-01-27T01:06:40.919701+0000","last_fullsized":"2024-01-27T01:06:40.919701+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:42.218792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919249+0000","last_change":"2024-01-27T01:06:26.862905+0000","last_active":"2024-01-27T01:06:40.919249+0000","last_peered":"2024-01-27T01:06:40.919249+0000","last_clean":"2024-01-27T01:06:40.919249+0000","last_became_active":"2024-01-27T01:06:26.862584+0000","last_became_peered":"2024-01-27T01:06:26.862584+0000","last_unstale":"2024-01-27T01:06:40.919249+0000","last_undegraded":"2024-01-27T01:06:40.919249+0000","last_fullsized":"2024-01-27T01:06:40.919249+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:27:14.653781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"226'6","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.956444+0000","last_change":"2024-01-27T01:06:39.940319+0000","last_active":"2024-01-27T01:06:40.956444+0000","last_peered":"2024-01-27T01:06:40.956444+0000","last_clean":"2024-01-27T01:06:40.956444+0000","last_became_active":"2024-01-27T01:06:39.940015+0000","last_became_peered":"2024-01-27T01:06:39.940015+0000","last_unstale":"2024-01-27T01:06:40.956444+0000","last_undegraded":"2024-01-27T01:06:40.956444+0000","last_fullsized":"2024-01-27T01:06:40.956444+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:05:49.945911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.951066+0000","last_change":"2024-01-27T01:06:40.945965+0000","last_active":"2024-01-27T01:06:40.951066+0000","last_peered":"2024-01-27T01:06:40.951066+0000","last_clean":"2024-01-27T01:06:40.951066+0000","last_became_active":"2024-01-27T01:06:40.945646+0000","last_became_peered":"2024-01-27T01:06:40.945646+0000","last_unstale":"2024-01-27T01:06:40.951066+0000","last_undegraded":"2024-01-27T01:06:40.951066+0000","last_fullsized":"2024-01-27T01:06:40.951066+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:05:50.951996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"222'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.930339+0000","last_change":"2024-01-27T01:06:25.850061+0000","last_active":"2024-01-27T01:06:40.930339+0000","last_peered":"2024-01-27T01:06:40.930339+0000","last_clean":"2024-01-27T01:06:40.930339+0000","last_became_active":"2024-01-27T01:06:25.849763+0000","last_became_peered":"2024-01-27T01:06:25.849763+0000","last_unstale":"2024-01-27T01:06:40.930339+0000","last_undegraded":"2024-01-27T01:06:40.930339+0000","last_fullsized":"2024-01-27T01:06:40.930339+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:09:56.243562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931751+0000","last_change":"2024-01-27T01:06:26.873464+0000","last_active":"2024-01-27T01:06:40.931751+0000","last_peered":"2024-01-27T01:06:40.931751+0000","last_clean":"2024-01-27T01:06:40.931751+0000","last_became_active":"2024-01-27T01:06:26.873063+0000","last_became_peered":"2024-01-27T01:06:26.873063+0000","last_unstale":"2024-01-27T01:06:40.931751+0000","last_undegraded":"2024-01-27T01:06:40.931751+0000","last_fullsized":"2024-01-27T01:06:40.931751+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:33:32.535574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"226'3","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.952973+0000","last_change":"2024-01-27T01:06:39.940058+0000","last_active":"2024-01-27T01:06:40.952973+0000","last_peered":"2024-01-27T01:06:40.952973+0000","last_clean":"2024-01-27T01:06:40.952973+0000","last_became_active":"2024-01-27T01:06:39.939776+0000","last_became_peered":"2024-01-27T01:06:39.939776+0000","last_unstale":"2024-01-27T01:06:40.952973+0000","last_undegraded":"2024-01-27T01:06:40.952973+0000","last_fullsized":"2024-01-27T01:06:40.952973+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:15:24.362081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.938133+0000","last_change":"2024-01-27T01:06:40.935256+0000","last_active":"2024-01-27T01:06:40.938133+0000","last_peered":"2024-01-27T01:06:40.938133+0000","last_clean":"2024-01-27T01:06:40.938133+0000","last_became_active":"2024-01-27T01:06:40.934889+0000","last_became_peered":"2024-01-27T01:06:40.934889+0000","last_unstale":"2024-01-27T01:06:40.938133+0000","last_undegraded":"2024-01-27T01:06:40.938133+0000","last_fullsized":"2024-01-27T01:06:40.938133+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:42:17.119687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"223'8","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931618+0000","last_change":"2024-01-27T01:06:25.864160+0000","last_active":"2024-01-27T01:06:40.931618+0000","last_peered":"2024-01-27T01:06:40.931618+0000","last_clean":"2024-01-27T01:06:40.931618+0000","last_became_active":"2024-01-27T01:06:25.863867+0000","last_became_peered":"2024-01-27T01:06:25.863867+0000","last_unstale":"2024-01-27T01:06:40.931618+0000","last_undegraded":"2024-01-27T01:06:40.931618+0000","last_fullsized":"2024-01-27T01:06:40.931618+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:43.338484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2768,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.941328+0000","last_change":"2024-01-27T01:06:26.879652+0000","last_active":"2024-01-27T01:06:40.941328+0000","last_peered":"2024-01-27T01:06:40.941328+0000","last_clean":"2024-01-27T01:06:40.941328+0000","last_became_active":"2024-01-27T01:06:26.879271+0000","last_became_peered":"2024-01-27T01:06:26.879271+0000","last_unstale":"2024-01-27T01:06:40.941328+0000","last_undegraded":"2024-01-27T01:06:40.941328+0000","last_fullsized":"2024-01-27T01:06:40.941328+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:57:23.456731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923013+0000","last_change":"2024-01-27T01:06:39.936805+0000","last_active":"2024-01-27T01:06:40.923013+0000","last_peered":"2024-01-27T01:06:40.923013+0000","last_clean":"2024-01-27T01:06:40.923013+0000","last_became_active":"2024-01-27T01:06:39.936499+0000","last_became_peered":"2024-01-27T01:06:39.936499+0000","last_unstale":"2024-01-27T01:06:40.923013+0000","last_undegraded":"2024-01-27T01:06:40.923013+0000","last_fullsized":"2024-01-27T01:06:40.923013+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:16:07.260797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.938022+0000","last_change":"2024-01-27T01:06:40.935378+0000","last_active":"2024-01-27T01:06:40.938022+0000","last_peered":"2024-01-27T01:06:40.938022+0000","last_clean":"2024-01-27T01:06:40.938022+0000","last_became_active":"2024-01-27T01:06:40.934801+0000","last_became_peered":"2024-01-27T01:06:40.934801+0000","last_unstale":"2024-01-27T01:06:40.938022+0000","last_undegraded":"2024-01-27T01:06:40.938022+0000","last_fullsized":"2024-01-27T01:06:40.938022+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:10:08.588321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.15","version":"222'3","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923278+0000","last_change":"2024-01-27T01:06:25.857771+0000","last_active":"2024-01-27T01:06:40.923278+0000","last_peered":"2024-01-27T01:06:40.923278+0000","last_clean":"2024-01-27T01:06:40.923278+0000","last_became_active":"2024-01-27T01:06:25.857445+0000","last_became_peered":"2024-01-27T01:06:25.857445+0000","last_unstale":"2024-01-27T01:06:40.923278+0000","last_undegraded":"2024-01-27T01:06:40.923278+0000","last_fullsized":"2024-01-27T01:06:40.923278+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:45:52.927866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923155+0000","last_change":"2024-01-27T01:06:26.876685+0000","last_active":"2024-01-27T01:06:40.923155+0000","last_peered":"2024-01-27T01:06:40.923155+0000","last_clean":"2024-01-27T01:06:40.923155+0000","last_became_active":"2024-01-27T01:06:26.872939+0000","last_became_peered":"2024-01-27T01:06:26.872939+0000","last_unstale":"2024-01-27T01:06:40.923155+0000","last_undegraded":"2024-01-27T01:06:40.923155+0000","last_fullsized":"2024-01-27T01:06:40.923155+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:56:48.730726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.955282+0000","last_change":"2024-01-27T01:06:39.941476+0000","last_active":"2024-01-27T01:06:40.955282+0000","last_peered":"2024-01-27T01:06:40.955282+0000","last_clean":"2024-01-27T01:06:40.955282+0000","last_became_active":"2024-01-27T01:06:39.940676+0000","last_became_peered":"2024-01-27T01:06:39.940676+0000","last_unstale":"2024-01-27T01:06:40.955282+0000","last_undegraded":"2024-01-27T01:06:40.955282+0000","last_fullsized":"2024-01-27T01:06:40.955282+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:36:13.180584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.945422+0000","last_change":"2024-01-27T01:06:40.942851+0000","last_active":"2024-01-27T01:06:40.945422+0000","last_peered":"2024-01-27T01:06:40.945422+0000","last_clean":"2024-01-27T01:06:40.945422+0000","last_became_active":"2024-01-27T01:06:40.942563+0000","last_became_peered":"2024-01-27T01:06:40.942563+0000","last_unstale":"2024-01-27T01:06:40.945422+0000","last_undegraded":"2024-01-27T01:06:40.945422+0000","last_fullsized":"2024-01-27T01:06:40.945422+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:24:51.130614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931731+0000","last_change":"2024-01-27T01:06:25.860762+0000","last_active":"2024-01-27T01:06:40.931731+0000","last_peered":"2024-01-27T01:06:40.931731+0000","last_clean":"2024-01-27T01:06:40.931731+0000","last_became_active":"2024-01-27T01:06:25.860459+0000","last_became_peered":"2024-01-27T01:06:25.860459+0000","last_unstale":"2024-01-27T01:06:40.931731+0000","last_undegraded":"2024-01-27T01:06:40.931731+0000","last_fullsized":"2024-01-27T01:06:40.931731+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:31.687265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919076+0000","last_change":"2024-01-27T01:06:26.868859+0000","last_active":"2024-01-27T01:06:40.919076+0000","last_peered":"2024-01-27T01:06:40.919076+0000","last_clean":"2024-01-27T01:06:40.919076+0000","last_became_active":"2024-01-27T01:06:26.868623+0000","last_became_peered":"2024-01-27T01:06:26.868623+0000","last_unstale":"2024-01-27T01:06:40.919076+0000","last_undegraded":"2024-01-27T01:06:40.919076+0000","last_fullsized":"2024-01-27T01:06:40.919076+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:18:47.232067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"226'8","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.955575+0000","last_change":"2024-01-27T01:06:39.934980+0000","last_active":"2024-01-27T01:06:40.955575+0000","last_peered":"2024-01-27T01:06:40.955575+0000","last_clean":"2024-01-27T01:06:40.955575+0000","last_became_active":"2024-01-27T01:06:39.934711+0000","last_became_peered":"2024-01-27T01:06:39.934711+0000","last_unstale":"2024-01-27T01:06:40.955575+0000","last_undegraded":"2024-01-27T01:06:40.955575+0000","last_fullsized":"2024-01-27T01:06:40.955575+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:15:24.362081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.924980+0000","last_change":"2024-01-27T01:06:40.924260+0000","last_active":"2024-01-27T01:06:40.924980+0000","last_peered":"2024-01-27T01:06:40.924980+0000","last_clean":"2024-01-27T01:06:40.924980+0000","last_became_active":"2024-01-27T01:06:40.923949+0000","last_became_peered":"2024-01-27T01:06:40.923949+0000","last_unstale":"2024-01-27T01:06:40.924980+0000","last_undegraded":"2024-01-27T01:06:40.924980+0000","last_fullsized":"2024-01-27T01:06:40.924980+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:18.204468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942513+0000","last_change":"2024-01-27T01:06:25.865314+0000","last_active":"2024-01-27T01:06:40.942513+0000","last_peered":"2024-01-27T01:06:40.942513+0000","last_clean":"2024-01-27T01:06:40.942513+0000","last_became_active":"2024-01-27T01:06:25.864940+0000","last_became_peered":"2024-01-27T01:06:25.864940+0000","last_unstale":"2024-01-27T01:06:40.942513+0000","last_undegraded":"2024-01-27T01:06:40.942513+0000","last_fullsized":"2024-01-27T01:06:40.942513+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:24.113986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931333+0000","last_change":"2024-01-27T01:06:26.870806+0000","last_active":"2024-01-27T01:06:40.931333+0000","last_peered":"2024-01-27T01:06:40.931333+0000","last_clean":"2024-01-27T01:06:40.931333+0000","last_became_active":"2024-01-27T01:06:26.870263+0000","last_became_peered":"2024-01-27T01:06:26.870263+0000","last_unstale":"2024-01-27T01:06:40.931333+0000","last_undegraded":"2024-01-27T01:06:40.931333+0000","last_fullsized":"2024-01-27T01:06:40.931333+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:26.214953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923552+0000","last_change":"2024-01-27T01:06:39.934885+0000","last_active":"2024-01-27T01:06:40.923552+0000","last_peered":"2024-01-27T01:06:40.923552+0000","last_clean":"2024-01-27T01:06:40.923552+0000","last_became_active":"2024-01-27T01:06:39.934619+0000","last_became_peered":"2024-01-27T01:06:39.934619+0000","last_unstale":"2024-01-27T01:06:40.923552+0000","last_undegraded":"2024-01-27T01:06:40.923552+0000","last_fullsized":"2024-01-27T01:06:40.923552+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:43:02.008737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942725+0000","last_change":"2024-01-27T01:06:40.937822+0000","last_active":"2024-01-27T01:06:40.942725+0000","last_peered":"2024-01-27T01:06:40.942725+0000","last_clean":"2024-01-27T01:06:40.942725+0000","last_became_active":"2024-01-27T01:06:40.937489+0000","last_became_peered":"2024-01-27T01:06:40.937489+0000","last_unstale":"2024-01-27T01:06:40.942725+0000","last_undegraded":"2024-01-27T01:06:40.942725+0000","last_fullsized":"2024-01-27T01:06:40.942725+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:03:13.414840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.918934+0000","last_change":"2024-01-27T01:06:25.851888+0000","last_active":"2024-01-27T01:06:40.918934+0000","last_peered":"2024-01-27T01:06:40.918934+0000","last_clean":"2024-01-27T01:06:40.918934+0000","last_became_active":"2024-01-27T01:06:25.851605+0000","last_became_peered":"2024-01-27T01:06:25.851605+0000","last_unstale":"2024-01-27T01:06:40.918934+0000","last_undegraded":"2024-01-27T01:06:40.918934+0000","last_fullsized":"2024-01-27T01:06:40.918934+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:14:09.386094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925489+0000","last_change":"2024-01-27T01:06:26.873199+0000","last_active":"2024-01-27T01:06:40.925489+0000","last_peered":"2024-01-27T01:06:40.925489+0000","last_clean":"2024-01-27T01:06:40.925489+0000","last_became_active":"2024-01-27T01:06:26.872658+0000","last_became_peered":"2024-01-27T01:06:26.872658+0000","last_unstale":"2024-01-27T01:06:40.925489+0000","last_undegraded":"2024-01-27T01:06:40.925489+0000","last_fullsized":"2024-01-27T01:06:40.925489+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:06:45.526347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.951919+0000","last_change":"2024-01-27T01:06:39.936324+0000","last_active":"2024-01-27T01:06:40.951919+0000","last_peered":"2024-01-27T01:06:40.951919+0000","last_clean":"2024-01-27T01:06:40.951919+0000","last_became_active":"2024-01-27T01:06:39.935917+0000","last_became_peered":"2024-01-27T01:06:39.935917+0000","last_unstale":"2024-01-27T01:06:40.951919+0000","last_undegraded":"2024-01-27T01:06:40.951919+0000","last_fullsized":"2024-01-27T01:06:40.951919+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:43:07.431223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.946155+0000","last_change":"2024-01-27T01:06:40.937572+0000","last_active":"2024-01-27T01:06:40.946155+0000","last_peered":"2024-01-27T01:06:40.946155+0000","last_clean":"2024-01-27T01:06:40.946155+0000","last_became_active":"2024-01-27T01:06:40.937143+0000","last_became_peered":"2024-01-27T01:06:40.937143+0000","last_unstale":"2024-01-27T01:06:40.946155+0000","last_undegraded":"2024-01-27T01:06:40.946155+0000","last_fullsized":"2024-01-27T01:06:40.946155+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:36:14.604522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.933332+0000","last_change":"2024-01-27T01:06:25.854878+0000","last_active":"2024-01-27T01:06:40.933332+0000","last_peered":"2024-01-27T01:06:40.933332+0000","last_clean":"2024-01-27T01:06:40.933332+0000","last_became_active":"2024-01-27T01:06:25.854623+0000","last_became_peered":"2024-01-27T01:06:25.854623+0000","last_unstale":"2024-01-27T01:06:40.933332+0000","last_undegraded":"2024-01-27T01:06:40.933332+0000","last_fullsized":"2024-01-27T01:06:40.933332+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:21.007449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931642+0000","last_change":"2024-01-27T01:06:26.876963+0000","last_active":"2024-01-27T01:06:40.931642+0000","last_peered":"2024-01-27T01:06:40.931642+0000","last_clean":"2024-01-27T01:06:40.931642+0000","last_became_active":"2024-01-27T01:06:26.876709+0000","last_became_peered":"2024-01-27T01:06:26.876709+0000","last_unstale":"2024-01-27T01:06:40.931642+0000","last_undegraded":"2024-01-27T01:06:40.931642+0000","last_fullsized":"2024-01-27T01:06:40.931642+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:58:14.572262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"226'4","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.954938+0000","last_change":"2024-01-27T01:06:39.938669+0000","last_active":"2024-01-27T01:06:40.954938+0000","last_peered":"2024-01-27T01:06:40.954938+0000","last_clean":"2024-01-27T01:06:40.954938+0000","last_became_active":"2024-01-27T01:06:39.938238+0000","last_became_peered":"2024-01-27T01:06:39.938238+0000","last_unstale":"2024-01-27T01:06:40.954938+0000","last_undegraded":"2024-01-27T01:06:40.954938+0000","last_fullsized":"2024-01-27T01:06:40.954938+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:33:16.818100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.953919+0000","last_change":"2024-01-27T01:06:40.945961+0000","last_active":"2024-01-27T01:06:40.953919+0000","last_peered":"2024-01-27T01:06:40.953919+0000","last_clean":"2024-01-27T01:06:40.953919+0000","last_became_active":"2024-01-27T01:06:40.945643+0000","last_became_peered":"2024-01-27T01:06:40.945643+0000","last_unstale":"2024-01-27T01:06:40.953919+0000","last_undegraded":"2024-01-27T01:06:40.953919+0000","last_fullsized":"2024-01-27T01:06:40.953919+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:23:56.549640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1b","version":"222'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931590+0000","last_change":"2024-01-27T01:06:25.854193+0000","last_active":"2024-01-27T01:06:40.931590+0000","last_peered":"2024-01-27T01:06:40.931590+0000","last_clean":"2024-01-27T01:06:40.931590+0000","last_became_active":"2024-01-27T01:06:25.853956+0000","last_became_peered":"2024-01-27T01:06:25.853956+0000","last_unstale":"2024-01-27T01:06:40.931590+0000","last_undegraded":"2024-01-27T01:06:40.931590+0000","last_fullsized":"2024-01-27T01:06:40.931590+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:33:02.747087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931528+0000","last_change":"2024-01-27T01:06:26.866121+0000","last_active":"2024-01-27T01:06:40.931528+0000","last_peered":"2024-01-27T01:06:40.931528+0000","last_clean":"2024-01-27T01:06:40.931528+0000","last_became_active":"2024-01-27T01:06:26.865391+0000","last_became_peered":"2024-01-27T01:06:26.865391+0000","last_unstale":"2024-01-27T01:06:40.931528+0000","last_undegraded":"2024-01-27T01:06:40.931528+0000","last_fullsized":"2024-01-27T01:06:40.931528+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:02:03.812030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"226'2","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.953895+0000","last_change":"2024-01-27T01:06:39.938268+0000","last_active":"2024-01-27T01:06:40.953895+0000","last_peered":"2024-01-27T01:06:40.953895+0000","last_clean":"2024-01-27T01:06:40.953895+0000","last_became_active":"2024-01-27T01:06:39.937991+0000","last_became_peered":"2024-01-27T01:06:39.937991+0000","last_unstale":"2024-01-27T01:06:40.953895+0000","last_undegraded":"2024-01-27T01:06:40.953895+0000","last_fullsized":"2024-01-27T01:06:40.953895+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:51:00.031893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.944610+0000","last_change":"2024-01-27T01:06:40.937290+0000","last_active":"2024-01-27T01:06:40.944610+0000","last_peered":"2024-01-27T01:06:40.944610+0000","last_clean":"2024-01-27T01:06:40.944610+0000","last_became_active":"2024-01-27T01:06:40.936962+0000","last_became_peered":"2024-01-27T01:06:40.936962+0000","last_unstale":"2024-01-27T01:06:40.944610+0000","last_undegraded":"2024-01-27T01:06:40.944610+0000","last_fullsized":"2024-01-27T01:06:40.944610+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:31:51.709336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1c","version":"222'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923515+0000","last_change":"2024-01-27T01:06:25.855114+0000","last_active":"2024-01-27T01:06:40.923515+0000","last_peered":"2024-01-27T01:06:40.923515+0000","last_clean":"2024-01-27T01:06:40.923515+0000","last_became_active":"2024-01-27T01:06:25.854790+0000","last_became_peered":"2024-01-27T01:06:25.854790+0000","last_unstale":"2024-01-27T01:06:40.923515+0000","last_undegraded":"2024-01-27T01:06:40.923515+0000","last_fullsized":"2024-01-27T01:06:40.923515+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:58:58.498377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.941528+0000","last_change":"2024-01-27T01:06:26.869128+0000","last_active":"2024-01-27T01:06:40.941528+0000","last_peered":"2024-01-27T01:06:40.941528+0000","last_clean":"2024-01-27T01:06:40.941528+0000","last_became_active":"2024-01-27T01:06:26.868892+0000","last_became_peered":"2024-01-27T01:06:26.868892+0000","last_unstale":"2024-01-27T01:06:40.941528+0000","last_undegraded":"2024-01-27T01:06:40.941528+0000","last_fullsized":"2024-01-27T01:06:40.941528+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:40:40.869516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.950661+0000","last_change":"2024-01-27T01:06:39.937444+0000","last_active":"2024-01-27T01:06:40.950661+0000","last_peered":"2024-01-27T01:06:40.950661+0000","last_clean":"2024-01-27T01:06:40.950661+0000","last_became_active":"2024-01-27T01:06:39.937157+0000","last_became_peered":"2024-01-27T01:06:39.937157+0000","last_unstale":"2024-01-27T01:06:40.950661+0000","last_undegraded":"2024-01-27T01:06:40.950661+0000","last_fullsized":"2024-01-27T01:06:40.950661+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:13:12.942303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.944137+0000","last_change":"2024-01-27T01:06:40.939552+0000","last_active":"2024-01-27T01:06:40.944137+0000","last_peered":"2024-01-27T01:06:40.944137+0000","last_clean":"2024-01-27T01:06:40.944137+0000","last_became_active":"2024-01-27T01:06:40.939300+0000","last_became_peered":"2024-01-27T01:06:40.939300+0000","last_unstale":"2024-01-27T01:06:40.944137+0000","last_undegraded":"2024-01-27T01:06:40.944137+0000","last_fullsized":"2024-01-27T01:06:40.944137+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:06:59.598967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"222'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931303+0000","last_change":"2024-01-27T01:06:25.855991+0000","last_active":"2024-01-27T01:06:40.931303+0000","last_peered":"2024-01-27T01:06:40.931303+0000","last_clean":"2024-01-27T01:06:40.931303+0000","last_became_active":"2024-01-27T01:06:25.855679+0000","last_became_peered":"2024-01-27T01:06:25.855679+0000","last_unstale":"2024-01-27T01:06:40.931303+0000","last_undegraded":"2024-01-27T01:06:40.931303+0000","last_fullsized":"2024-01-27T01:06:40.931303+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:31:04.295025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926141+0000","last_change":"2024-01-27T01:06:26.875831+0000","last_active":"2024-01-27T01:06:40.926141+0000","last_peered":"2024-01-27T01:06:40.926141+0000","last_clean":"2024-01-27T01:06:40.926141+0000","last_became_active":"2024-01-27T01:06:26.875488+0000","last_became_peered":"2024-01-27T01:06:26.875488+0000","last_unstale":"2024-01-27T01:06:40.926141+0000","last_undegraded":"2024-01-27T01:06:40.926141+0000","last_fullsized":"2024-01-27T01:06:40.926141+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:07:51.640059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"226'1","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.951407+0000","last_change":"2024-01-27T01:06:39.938441+0000","last_active":"2024-01-27T01:06:40.951407+0000","last_peered":"2024-01-27T01:06:40.951407+0000","last_clean":"2024-01-27T01:06:40.951407+0000","last_became_active":"2024-01-27T01:06:39.938144+0000","last_became_peered":"2024-01-27T01:06:39.938144+0000","last_unstale":"2024-01-27T01:06:40.951407+0000","last_undegraded":"2024-01-27T01:06:40.951407+0000","last_fullsized":"2024-01-27T01:06:40.951407+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:18:25.975883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.940548+0000","last_change":"2024-01-27T01:06:40.936165+0000","last_active":"2024-01-27T01:06:40.940548+0000","last_peered":"2024-01-27T01:06:40.940548+0000","last_clean":"2024-01-27T01:06:40.940548+0000","last_became_active":"2024-01-27T01:06:40.935054+0000","last_became_peered":"2024-01-27T01:06:40.935054+0000","last_unstale":"2024-01-27T01:06:40.940548+0000","last_undegraded":"2024-01-27T01:06:40.940548+0000","last_fullsized":"2024-01-27T01:06:40.940548+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:50:27.972686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1e","version":"222'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.928525+0000","last_change":"2024-01-27T01:06:25.856504+0000","last_active":"2024-01-27T01:06:40.928525+0000","last_peered":"2024-01-27T01:06:40.928525+0000","last_clean":"2024-01-27T01:06:40.928525+0000","last_became_active":"2024-01-27T01:06:25.856272+0000","last_became_peered":"2024-01-27T01:06:25.856272+0000","last_unstale":"2024-01-27T01:06:40.928525+0000","last_undegraded":"2024-01-27T01:06:40.928525+0000","last_fullsized":"2024-01-27T01:06:40.928525+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:20:56.736428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923432+0000","last_change":"2024-01-27T01:06:26.878625+0000","last_active":"2024-01-27T01:06:40.923432+0000","last_peered":"2024-01-27T01:06:40.923432+0000","last_clean":"2024-01-27T01:06:40.923432+0000","last_became_active":"2024-01-27T01:06:26.878361+0000","last_became_peered":"2024-01-27T01:06:26.878361+0000","last_unstale":"2024-01-27T01:06:40.923432+0000","last_undegraded":"2024-01-27T01:06:40.923432+0000","last_fullsized":"2024-01-27T01:06:40.923432+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:37:06.977689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.930028+0000","last_change":"2024-01-27T01:06:39.932404+0000","last_active":"2024-01-27T01:06:40.930028+0000","last_peered":"2024-01-27T01:06:40.930028+0000","last_clean":"2024-01-27T01:06:40.930028+0000","last_became_active":"2024-01-27T01:06:39.932068+0000","last_became_peered":"2024-01-27T01:06:39.932068+0000","last_unstale":"2024-01-27T01:06:40.930028+0000","last_undegraded":"2024-01-27T01:06:40.930028+0000","last_fullsized":"2024-01-27T01:06:40.930028+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:37:20.881203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.946296+0000","last_change":"2024-01-27T01:06:40.938927+0000","last_active":"2024-01-27T01:06:40.946296+0000","last_peered":"2024-01-27T01:06:40.946296+0000","last_clean":"2024-01-27T01:06:40.946296+0000","last_became_active":"2024-01-27T01:06:40.938546+0000","last_became_peered":"2024-01-27T01:06:40.938546+0000","last_unstale":"2024-01-27T01:06:40.946296+0000","last_undegraded":"2024-01-27T01:06:40.946296+0000","last_fullsized":"2024-01-27T01:06:40.946296+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:57.703450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1f","version":"222'4","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.932836+0000","last_change":"2024-01-27T01:06:25.856458+0000","last_active":"2024-01-27T01:06:40.932836+0000","last_peered":"2024-01-27T01:06:40.932836+0000","last_clean":"2024-01-27T01:06:40.932836+0000","last_became_active":"2024-01-27T01:06:25.856162+0000","last_became_peered":"2024-01-27T01:06:25.856162+0000","last_unstale":"2024-01-27T01:06:40.932836+0000","last_undegraded":"2024-01-27T01:06:40.932836+0000","last_fullsized":"2024-01-27T01:06:40.932836+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:24.830798+0000","last_clean_scrub_stamp":"2024-01-27T01:06:24.830798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:51:44.378880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.930272+0000","last_change":"2024-01-27T01:06:26.869588+0000","last_active":"2024-01-27T01:06:40.930272+0000","last_peered":"2024-01-27T01:06:40.930272+0000","last_clean":"2024-01-27T01:06:40.930272+0000","last_became_active":"2024-01-27T01:06:26.869264+0000","last_became_peered":"2024-01-27T01:06:26.869264+0000","last_unstale":"2024-01-27T01:06:40.930272+0000","last_undegraded":"2024-01-27T01:06:40.930272+0000","last_fullsized":"2024-01-27T01:06:40.930272+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:42:54.364688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":10,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.933810+0000","last_change":"2024-01-27T01:06:39.937481+0000","last_active":"2024-01-27T01:06:40.933810+0000","last_peered":"2024-01-27T01:06:40.933810+0000","last_clean":"2024-01-27T01:06:40.933810+0000","last_became_active":"2024-01-27T01:06:39.937225+0000","last_became_peered":"2024-01-27T01:06:39.937225+0000","last_unstale":"2024-01-27T01:06:40.933810+0000","last_undegraded":"2024-01-27T01:06:40.933810+0000","last_fullsized":"2024-01-27T01:06:40.933810+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:38.901811+0000","last_clean_scrub_stamp":"2024-01-27T01:06:38.901811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:45.338439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.933146+0000","last_change":"2024-01-27T01:06:40.929946+0000","last_active":"2024-01-27T01:06:40.933146+0000","last_peered":"2024-01-27T01:06:40.933146+0000","last_clean":"2024-01-27T01:06:40.933146+0000","last_became_active":"2024-01-27T01:06:40.929262+0000","last_became_peered":"2024-01-27T01:06:40.929262+0000","last_unstale":"2024-01-27T01:06:40.933146+0000","last_undegraded":"2024-01-27T01:06:40.933146+0000","last_fullsized":"2024-01-27T01:06:40.933146+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:08:47.665701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.932763+0000","last_change":"2024-01-27T01:06:26.869029+0000","last_active":"2024-01-27T01:06:40.932763+0000","last_peered":"2024-01-27T01:06:40.932763+0000","last_clean":"2024-01-27T01:06:40.932763+0000","last_became_active":"2024-01-27T01:06:26.868793+0000","last_became_peered":"2024-01-27T01:06:26.868793+0000","last_unstale":"2024-01-27T01:06:40.932763+0000","last_undegraded":"2024-01-27T01:06:40.932763+0000","last_fullsized":"2024-01-27T01:06:40.932763+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:10.965469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942063+0000","last_change":"2024-01-27T01:06:40.935281+0000","last_active":"2024-01-27T01:06:40.942063+0000","last_peered":"2024-01-27T01:06:40.942063+0000","last_clean":"2024-01-27T01:06:40.942063+0000","last_became_active":"2024-01-27T01:06:40.935011+0000","last_became_peered":"2024-01-27T01:06:40.935011+0000","last_unstale":"2024-01-27T01:06:40.942063+0000","last_undegraded":"2024-01-27T01:06:40.942063+0000","last_fullsized":"2024-01-27T01:06:40.942063+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:25.572331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.933065+0000","last_change":"2024-01-27T01:06:26.876343+0000","last_active":"2024-01-27T01:06:40.933065+0000","last_peered":"2024-01-27T01:06:40.933065+0000","last_clean":"2024-01-27T01:06:40.933065+0000","last_became_active":"2024-01-27T01:06:26.875570+0000","last_became_peered":"2024-01-27T01:06:26.875570+0000","last_unstale":"2024-01-27T01:06:40.933065+0000","last_undegraded":"2024-01-27T01:06:40.933065+0000","last_fullsized":"2024-01-27T01:06:40.933065+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:07:51.640059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.945810+0000","last_change":"2024-01-27T01:06:40.942956+0000","last_active":"2024-01-27T01:06:40.945810+0000","last_peered":"2024-01-27T01:06:40.945810+0000","last_clean":"2024-01-27T01:06:40.945810+0000","last_became_active":"2024-01-27T01:06:40.942672+0000","last_became_peered":"2024-01-27T01:06:40.942672+0000","last_unstale":"2024-01-27T01:06:40.945810+0000","last_undegraded":"2024-01-27T01:06:40.945810+0000","last_fullsized":"2024-01-27T01:06:40.945810+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:33:33.988432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.933140+0000","last_change":"2024-01-27T01:06:26.871676+0000","last_active":"2024-01-27T01:06:40.933140+0000","last_peered":"2024-01-27T01:06:40.933140+0000","last_clean":"2024-01-27T01:06:40.933140+0000","last_became_active":"2024-01-27T01:06:26.871423+0000","last_became_peered":"2024-01-27T01:06:26.871423+0000","last_unstale":"2024-01-27T01:06:40.933140+0000","last_undegraded":"2024-01-27T01:06:40.933140+0000","last_fullsized":"2024-01-27T01:06:40.933140+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:29:58.146876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.944066+0000","last_change":"2024-01-27T01:06:40.942844+0000","last_active":"2024-01-27T01:06:40.944066+0000","last_peered":"2024-01-27T01:06:40.944066+0000","last_clean":"2024-01-27T01:06:40.944066+0000","last_became_active":"2024-01-27T01:06:40.942392+0000","last_became_peered":"2024-01-27T01:06:40.942392+0000","last_unstale":"2024-01-27T01:06:40.944066+0000","last_undegraded":"2024-01-27T01:06:40.944066+0000","last_fullsized":"2024-01-27T01:06:40.944066+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:57:08.256445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923341+0000","last_change":"2024-01-27T01:06:26.872013+0000","last_active":"2024-01-27T01:06:40.923341+0000","last_peered":"2024-01-27T01:06:40.923341+0000","last_clean":"2024-01-27T01:06:40.923341+0000","last_became_active":"2024-01-27T01:06:26.867577+0000","last_became_peered":"2024-01-27T01:06:26.867577+0000","last_unstale":"2024-01-27T01:06:40.923341+0000","last_undegraded":"2024-01-27T01:06:40.923341+0000","last_fullsized":"2024-01-27T01:06:40.923341+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:27.604574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.947745+0000","last_change":"2024-01-27T01:06:40.944023+0000","last_active":"2024-01-27T01:06:40.947745+0000","last_peered":"2024-01-27T01:06:40.947745+0000","last_clean":"2024-01-27T01:06:40.947745+0000","last_became_active":"2024-01-27T01:06:40.943733+0000","last_became_peered":"2024-01-27T01:06:40.943733+0000","last_unstale":"2024-01-27T01:06:40.947745+0000","last_undegraded":"2024-01-27T01:06:40.947745+0000","last_fullsized":"2024-01-27T01:06:40.947745+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:16.073528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.943369+0000","last_change":"2024-01-27T01:06:40.938941+0000","last_active":"2024-01-27T01:06:40.943369+0000","last_peered":"2024-01-27T01:06:40.943369+0000","last_clean":"2024-01-27T01:06:40.943369+0000","last_became_active":"2024-01-27T01:06:40.938667+0000","last_became_peered":"2024-01-27T01:06:40.938667+0000","last_unstale":"2024-01-27T01:06:40.943369+0000","last_undegraded":"2024-01-27T01:06:40.943369+0000","last_fullsized":"2024-01-27T01:06:40.943369+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:52:37.031485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.941082+0000","last_change":"2024-01-27T01:06:26.870649+0000","last_active":"2024-01-27T01:06:40.941082+0000","last_peered":"2024-01-27T01:06:40.941082+0000","last_clean":"2024-01-27T01:06:40.941082+0000","last_became_active":"2024-01-27T01:06:26.870433+0000","last_became_peered":"2024-01-27T01:06:26.870433+0000","last_unstale":"2024-01-27T01:06:40.941082+0000","last_undegraded":"2024-01-27T01:06:40.941082+0000","last_fullsized":"2024-01-27T01:06:40.941082+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:11:27.402943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.951687+0000","last_change":"2024-01-27T01:06:40.948958+0000","last_active":"2024-01-27T01:06:40.951687+0000","last_peered":"2024-01-27T01:06:40.951687+0000","last_clean":"2024-01-27T01:06:40.951687+0000","last_became_active":"2024-01-27T01:06:40.948336+0000","last_became_peered":"2024-01-27T01:06:40.948336+0000","last_unstale":"2024-01-27T01:06:40.951687+0000","last_undegraded":"2024-01-27T01:06:40.951687+0000","last_fullsized":"2024-01-27T01:06:40.951687+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:10:08.588321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919157+0000","last_change":"2024-01-27T01:06:26.878202+0000","last_active":"2024-01-27T01:06:40.919157+0000","last_peered":"2024-01-27T01:06:40.919157+0000","last_clean":"2024-01-27T01:06:40.919157+0000","last_became_active":"2024-01-27T01:06:26.877900+0000","last_became_peered":"2024-01-27T01:06:26.877900+0000","last_unstale":"2024-01-27T01:06:40.919157+0000","last_undegraded":"2024-01-27T01:06:40.919157+0000","last_fullsized":"2024-01-27T01:06:40.919157+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:42.766155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.928440+0000","last_change":"2024-01-27T01:06:26.878911+0000","last_active":"2024-01-27T01:06:40.928440+0000","last_peered":"2024-01-27T01:06:40.928440+0000","last_clean":"2024-01-27T01:06:40.928440+0000","last_became_active":"2024-01-27T01:06:26.878637+0000","last_became_peered":"2024-01-27T01:06:26.878637+0000","last_unstale":"2024-01-27T01:06:40.928440+0000","last_undegraded":"2024-01-27T01:06:40.928440+0000","last_fullsized":"2024-01-27T01:06:40.928440+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:33:03.751565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.944263+0000","last_change":"2024-01-27T01:06:40.941505+0000","last_active":"2024-01-27T01:06:40.944263+0000","last_peered":"2024-01-27T01:06:40.944263+0000","last_clean":"2024-01-27T01:06:40.944263+0000","last_became_active":"2024-01-27T01:06:40.940953+0000","last_became_peered":"2024-01-27T01:06:40.940953+0000","last_unstale":"2024-01-27T01:06:40.944263+0000","last_undegraded":"2024-01-27T01:06:40.944263+0000","last_fullsized":"2024-01-27T01:06:40.944263+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:47:31.151874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.949826+0000","last_change":"2024-01-27T01:06:40.942124+0000","last_active":"2024-01-27T01:06:40.949826+0000","last_peered":"2024-01-27T01:06:40.949826+0000","last_clean":"2024-01-27T01:06:40.949826+0000","last_became_active":"2024-01-27T01:06:40.941857+0000","last_became_peered":"2024-01-27T01:06:40.941857+0000","last_unstale":"2024-01-27T01:06:40.949826+0000","last_undegraded":"2024-01-27T01:06:40.949826+0000","last_fullsized":"2024-01-27T01:06:40.949826+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:21:06.828538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925446+0000","last_change":"2024-01-27T01:06:26.865149+0000","last_active":"2024-01-27T01:06:40.925446+0000","last_peered":"2024-01-27T01:06:40.925446+0000","last_clean":"2024-01-27T01:06:40.925446+0000","last_became_active":"2024-01-27T01:06:26.864853+0000","last_became_peered":"2024-01-27T01:06:26.864853+0000","last_unstale":"2024-01-27T01:06:40.925446+0000","last_undegraded":"2024-01-27T01:06:40.925446+0000","last_fullsized":"2024-01-27T01:06:40.925446+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:10:44.194323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.936461+0000","last_change":"2024-01-27T01:06:40.930136+0000","last_active":"2024-01-27T01:06:40.936461+0000","last_peered":"2024-01-27T01:06:40.936461+0000","last_clean":"2024-01-27T01:06:40.936461+0000","last_became_active":"2024-01-27T01:06:40.929659+0000","last_became_peered":"2024-01-27T01:06:40.929659+0000","last_unstale":"2024-01-27T01:06:40.936461+0000","last_undegraded":"2024-01-27T01:06:40.936461+0000","last_fullsized":"2024-01-27T01:06:40.936461+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:38:42.855259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.938062+0000","last_change":"2024-01-27T01:06:26.867241+0000","last_active":"2024-01-27T01:06:40.938062+0000","last_peered":"2024-01-27T01:06:40.938062+0000","last_clean":"2024-01-27T01:06:40.938062+0000","last_became_active":"2024-01-27T01:06:26.866917+0000","last_became_peered":"2024-01-27T01:06:26.866917+0000","last_unstale":"2024-01-27T01:06:40.938062+0000","last_undegraded":"2024-01-27T01:06:40.938062+0000","last_fullsized":"2024-01-27T01:06:40.938062+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:24:25.818592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.954534+0000","last_change":"2024-01-27T01:06:40.950563+0000","last_active":"2024-01-27T01:06:40.954534+0000","last_peered":"2024-01-27T01:06:40.954534+0000","last_clean":"2024-01-27T01:06:40.954534+0000","last_became_active":"2024-01-27T01:06:40.950227+0000","last_became_peered":"2024-01-27T01:06:40.950227+0000","last_unstale":"2024-01-27T01:06:40.954534+0000","last_undegraded":"2024-01-27T01:06:40.954534+0000","last_fullsized":"2024-01-27T01:06:40.954534+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:18:06.269258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.930535+0000","last_change":"2024-01-27T01:06:26.874471+0000","last_active":"2024-01-27T01:06:40.930535+0000","last_peered":"2024-01-27T01:06:40.930535+0000","last_clean":"2024-01-27T01:06:40.930535+0000","last_became_active":"2024-01-27T01:06:26.874271+0000","last_became_peered":"2024-01-27T01:06:26.874271+0000","last_unstale":"2024-01-27T01:06:40.930535+0000","last_undegraded":"2024-01-27T01:06:40.930535+0000","last_fullsized":"2024-01-27T01:06:40.930535+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:15:11.295547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.951309+0000","last_change":"2024-01-27T01:06:40.948707+0000","last_active":"2024-01-27T01:06:40.951309+0000","last_peered":"2024-01-27T01:06:40.951309+0000","last_clean":"2024-01-27T01:06:40.951309+0000","last_became_active":"2024-01-27T01:06:40.948426+0000","last_became_peered":"2024-01-27T01:06:40.948426+0000","last_unstale":"2024-01-27T01:06:40.951309+0000","last_undegraded":"2024-01-27T01:06:40.951309+0000","last_fullsized":"2024-01-27T01:06:40.951309+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:39:27.523918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926235+0000","last_change":"2024-01-27T01:06:26.876390+0000","last_active":"2024-01-27T01:06:40.926235+0000","last_peered":"2024-01-27T01:06:40.926235+0000","last_clean":"2024-01-27T01:06:40.926235+0000","last_became_active":"2024-01-27T01:06:26.876056+0000","last_became_peered":"2024-01-27T01:06:26.876056+0000","last_unstale":"2024-01-27T01:06:40.926235+0000","last_undegraded":"2024-01-27T01:06:40.926235+0000","last_fullsized":"2024-01-27T01:06:40.926235+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:38:36.236958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.947768+0000","last_change":"2024-01-27T01:06:40.944841+0000","last_active":"2024-01-27T01:06:40.947768+0000","last_peered":"2024-01-27T01:06:40.947768+0000","last_clean":"2024-01-27T01:06:40.947768+0000","last_became_active":"2024-01-27T01:06:40.941811+0000","last_became_peered":"2024-01-27T01:06:40.941811+0000","last_unstale":"2024-01-27T01:06:40.947768+0000","last_undegraded":"2024-01-27T01:06:40.947768+0000","last_fullsized":"2024-01-27T01:06:40.947768+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:38.155368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.927179+0000","last_change":"2024-01-27T01:06:26.879441+0000","last_active":"2024-01-27T01:06:40.927179+0000","last_peered":"2024-01-27T01:06:40.927179+0000","last_clean":"2024-01-27T01:06:40.927179+0000","last_became_active":"2024-01-27T01:06:26.879148+0000","last_became_peered":"2024-01-27T01:06:26.879148+0000","last_unstale":"2024-01-27T01:06:40.927179+0000","last_undegraded":"2024-01-27T01:06:40.927179+0000","last_fullsized":"2024-01-27T01:06:40.927179+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:05:08.012027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.954692+0000","last_change":"2024-01-27T01:06:40.951207+0000","last_active":"2024-01-27T01:06:40.954692+0000","last_peered":"2024-01-27T01:06:40.954692+0000","last_clean":"2024-01-27T01:06:40.954692+0000","last_became_active":"2024-01-27T01:06:40.950903+0000","last_became_peered":"2024-01-27T01:06:40.950903+0000","last_unstale":"2024-01-27T01:06:40.954692+0000","last_undegraded":"2024-01-27T01:06:40.954692+0000","last_fullsized":"2024-01-27T01:06:40.954692+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:13:18.790129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.927030+0000","last_change":"2024-01-27T01:06:26.876823+0000","last_active":"2024-01-27T01:06:40.927030+0000","last_peered":"2024-01-27T01:06:40.927030+0000","last_clean":"2024-01-27T01:06:40.927030+0000","last_became_active":"2024-01-27T01:06:26.876304+0000","last_became_peered":"2024-01-27T01:06:26.876304+0000","last_unstale":"2024-01-27T01:06:40.927030+0000","last_undegraded":"2024-01-27T01:06:40.927030+0000","last_fullsized":"2024-01-27T01:06:40.927030+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:32:28.378677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.943663+0000","last_change":"2024-01-27T01:06:40.936906+0000","last_active":"2024-01-27T01:06:40.943663+0000","last_peered":"2024-01-27T01:06:40.943663+0000","last_clean":"2024-01-27T01:06:40.943663+0000","last_became_active":"2024-01-27T01:06:40.936631+0000","last_became_peered":"2024-01-27T01:06:40.936631+0000","last_unstale":"2024-01-27T01:06:40.943663+0000","last_undegraded":"2024-01-27T01:06:40.943663+0000","last_fullsized":"2024-01-27T01:06:40.943663+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:41.677193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926899+0000","last_change":"2024-01-27T01:06:26.873401+0000","last_active":"2024-01-27T01:06:40.926899+0000","last_peered":"2024-01-27T01:06:40.926899+0000","last_clean":"2024-01-27T01:06:40.926899+0000","last_became_active":"2024-01-27T01:06:26.873094+0000","last_became_peered":"2024-01-27T01:06:26.873094+0000","last_unstale":"2024-01-27T01:06:40.926899+0000","last_undegraded":"2024-01-27T01:06:40.926899+0000","last_fullsized":"2024-01-27T01:06:40.926899+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:11:24.082749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.929355+0000","last_change":"2024-01-27T01:06:40.928273+0000","last_active":"2024-01-27T01:06:40.929355+0000","last_peered":"2024-01-27T01:06:40.929355+0000","last_clean":"2024-01-27T01:06:40.929355+0000","last_became_active":"2024-01-27T01:06:40.927715+0000","last_became_peered":"2024-01-27T01:06:40.927715+0000","last_unstale":"2024-01-27T01:06:40.929355+0000","last_undegraded":"2024-01-27T01:06:40.929355+0000","last_fullsized":"2024-01-27T01:06:40.929355+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:49:23.701149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.923728+0000","last_change":"2024-01-27T01:06:26.864890+0000","last_active":"2024-01-27T01:06:40.923728+0000","last_peered":"2024-01-27T01:06:40.923728+0000","last_clean":"2024-01-27T01:06:40.923728+0000","last_became_active":"2024-01-27T01:06:26.864556+0000","last_became_peered":"2024-01-27T01:06:26.864556+0000","last_unstale":"2024-01-27T01:06:40.923728+0000","last_undegraded":"2024-01-27T01:06:40.923728+0000","last_fullsized":"2024-01-27T01:06:40.923728+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:41.033580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.948110+0000","last_change":"2024-01-27T01:06:40.944879+0000","last_active":"2024-01-27T01:06:40.948110+0000","last_peered":"2024-01-27T01:06:40.948110+0000","last_clean":"2024-01-27T01:06:40.948110+0000","last_became_active":"2024-01-27T01:06:40.944457+0000","last_became_peered":"2024-01-27T01:06:40.944457+0000","last_unstale":"2024-01-27T01:06:40.948110+0000","last_undegraded":"2024-01-27T01:06:40.948110+0000","last_fullsized":"2024-01-27T01:06:40.948110+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:57:02.803345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925347+0000","last_change":"2024-01-27T01:06:26.874473+0000","last_active":"2024-01-27T01:06:40.925347+0000","last_peered":"2024-01-27T01:06:40.925347+0000","last_clean":"2024-01-27T01:06:40.925347+0000","last_became_active":"2024-01-27T01:06:26.874203+0000","last_became_peered":"2024-01-27T01:06:26.874203+0000","last_unstale":"2024-01-27T01:06:40.925347+0000","last_undegraded":"2024-01-27T01:06:40.925347+0000","last_fullsized":"2024-01-27T01:06:40.925347+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:33:37.975146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.940476+0000","last_change":"2024-01-27T01:06:40.935406+0000","last_active":"2024-01-27T01:06:40.940476+0000","last_peered":"2024-01-27T01:06:40.940476+0000","last_clean":"2024-01-27T01:06:40.940476+0000","last_became_active":"2024-01-27T01:06:40.935179+0000","last_became_peered":"2024-01-27T01:06:40.935179+0000","last_unstale":"2024-01-27T01:06:40.940476+0000","last_undegraded":"2024-01-27T01:06:40.940476+0000","last_fullsized":"2024-01-27T01:06:40.940476+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:37:21.887288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919006+0000","last_change":"2024-01-27T01:06:26.863380+0000","last_active":"2024-01-27T01:06:40.919006+0000","last_peered":"2024-01-27T01:06:40.919006+0000","last_clean":"2024-01-27T01:06:40.919006+0000","last_became_active":"2024-01-27T01:06:26.863034+0000","last_became_peered":"2024-01-27T01:06:26.863034+0000","last_unstale":"2024-01-27T01:06:40.919006+0000","last_undegraded":"2024-01-27T01:06:40.919006+0000","last_fullsized":"2024-01-27T01:06:40.919006+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:32.271904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942025+0000","last_change":"2024-01-27T01:06:40.939294+0000","last_active":"2024-01-27T01:06:40.942025+0000","last_peered":"2024-01-27T01:06:40.942025+0000","last_clean":"2024-01-27T01:06:40.942025+0000","last_became_active":"2024-01-27T01:06:40.938975+0000","last_became_peered":"2024-01-27T01:06:40.938975+0000","last_unstale":"2024-01-27T01:06:40.942025+0000","last_undegraded":"2024-01-27T01:06:40.942025+0000","last_fullsized":"2024-01-27T01:06:40.942025+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:28:23.028911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.934074+0000","last_change":"2024-01-27T01:06:26.877929+0000","last_active":"2024-01-27T01:06:40.934074+0000","last_peered":"2024-01-27T01:06:40.934074+0000","last_clean":"2024-01-27T01:06:40.934074+0000","last_became_active":"2024-01-27T01:06:26.877679+0000","last_became_peered":"2024-01-27T01:06:26.877679+0000","last_unstale":"2024-01-27T01:06:40.934074+0000","last_undegraded":"2024-01-27T01:06:40.934074+0000","last_fullsized":"2024-01-27T01:06:40.934074+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:53:15.482586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942625+0000","last_change":"2024-01-27T01:06:40.940817+0000","last_active":"2024-01-27T01:06:40.942625+0000","last_peered":"2024-01-27T01:06:40.942625+0000","last_clean":"2024-01-27T01:06:40.942625+0000","last_became_active":"2024-01-27T01:06:40.940551+0000","last_became_peered":"2024-01-27T01:06:40.940551+0000","last_unstale":"2024-01-27T01:06:40.942625+0000","last_undegraded":"2024-01-27T01:06:40.942625+0000","last_fullsized":"2024-01-27T01:06:40.942625+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:36:17.619595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926330+0000","last_change":"2024-01-27T01:06:26.876674+0000","last_active":"2024-01-27T01:06:40.926330+0000","last_peered":"2024-01-27T01:06:40.926330+0000","last_clean":"2024-01-27T01:06:40.926330+0000","last_became_active":"2024-01-27T01:06:26.876413+0000","last_became_peered":"2024-01-27T01:06:26.876413+0000","last_unstale":"2024-01-27T01:06:40.926330+0000","last_undegraded":"2024-01-27T01:06:40.926330+0000","last_fullsized":"2024-01-27T01:06:40.926330+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:09:08.822886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.940760+0000","last_change":"2024-01-27T01:06:40.933472+0000","last_active":"2024-01-27T01:06:40.940760+0000","last_peered":"2024-01-27T01:06:40.940760+0000","last_clean":"2024-01-27T01:06:40.940760+0000","last_became_active":"2024-01-27T01:06:40.932482+0000","last_became_peered":"2024-01-27T01:06:40.932482+0000","last_unstale":"2024-01-27T01:06:40.940760+0000","last_undegraded":"2024-01-27T01:06:40.940760+0000","last_fullsized":"2024-01-27T01:06:40.940760+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:10:08.588321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931482+0000","last_change":"2024-01-27T01:06:26.877777+0000","last_active":"2024-01-27T01:06:40.931482+0000","last_peered":"2024-01-27T01:06:40.931482+0000","last_clean":"2024-01-27T01:06:40.931482+0000","last_became_active":"2024-01-27T01:06:26.877515+0000","last_became_peered":"2024-01-27T01:06:26.877515+0000","last_unstale":"2024-01-27T01:06:40.931482+0000","last_undegraded":"2024-01-27T01:06:40.931482+0000","last_fullsized":"2024-01-27T01:06:40.931482+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:27:14.653781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.954673+0000","last_change":"2024-01-27T01:06:40.950670+0000","last_active":"2024-01-27T01:06:40.954673+0000","last_peered":"2024-01-27T01:06:40.954673+0000","last_clean":"2024-01-27T01:06:40.954673+0000","last_became_active":"2024-01-27T01:06:40.949809+0000","last_became_peered":"2024-01-27T01:06:40.949809+0000","last_unstale":"2024-01-27T01:06:40.954673+0000","last_undegraded":"2024-01-27T01:06:40.954673+0000","last_fullsized":"2024-01-27T01:06:40.954673+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:12:59.099059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.939378+0000","last_change":"2024-01-27T01:06:26.865833+0000","last_active":"2024-01-27T01:06:40.939378+0000","last_peered":"2024-01-27T01:06:40.939378+0000","last_clean":"2024-01-27T01:06:40.939378+0000","last_became_active":"2024-01-27T01:06:26.865525+0000","last_became_peered":"2024-01-27T01:06:26.865525+0000","last_unstale":"2024-01-27T01:06:40.939378+0000","last_undegraded":"2024-01-27T01:06:40.939378+0000","last_fullsized":"2024-01-27T01:06:40.939378+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:42:10.990703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.937365+0000","last_change":"2024-01-27T01:06:40.934179+0000","last_active":"2024-01-27T01:06:40.937365+0000","last_peered":"2024-01-27T01:06:40.937365+0000","last_clean":"2024-01-27T01:06:40.937365+0000","last_became_active":"2024-01-27T01:06:40.933756+0000","last_became_peered":"2024-01-27T01:06:40.933756+0000","last_unstale":"2024-01-27T01:06:40.937365+0000","last_undegraded":"2024-01-27T01:06:40.937365+0000","last_fullsized":"2024-01-27T01:06:40.937365+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:46.607241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919877+0000","last_change":"2024-01-27T01:06:26.879535+0000","last_active":"2024-01-27T01:06:40.919877+0000","last_peered":"2024-01-27T01:06:40.919877+0000","last_clean":"2024-01-27T01:06:40.919877+0000","last_became_active":"2024-01-27T01:06:26.879219+0000","last_became_peered":"2024-01-27T01:06:26.879219+0000","last_unstale":"2024-01-27T01:06:40.919877+0000","last_undegraded":"2024-01-27T01:06:40.919877+0000","last_fullsized":"2024-01-27T01:06:40.919877+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:58.495062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.943965+0000","last_change":"2024-01-27T01:06:40.935281+0000","last_active":"2024-01-27T01:06:40.943965+0000","last_peered":"2024-01-27T01:06:40.943965+0000","last_clean":"2024-01-27T01:06:40.943965+0000","last_became_active":"2024-01-27T01:06:40.934857+0000","last_became_peered":"2024-01-27T01:06:40.934857+0000","last_unstale":"2024-01-27T01:06:40.943965+0000","last_undegraded":"2024-01-27T01:06:40.943965+0000","last_fullsized":"2024-01-27T01:06:40.943965+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:12:42.423061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.928744+0000","last_change":"2024-01-27T01:06:26.876736+0000","last_active":"2024-01-27T01:06:40.928744+0000","last_peered":"2024-01-27T01:06:40.928744+0000","last_clean":"2024-01-27T01:06:40.928744+0000","last_became_active":"2024-01-27T01:06:26.875870+0000","last_became_peered":"2024-01-27T01:06:26.875870+0000","last_unstale":"2024-01-27T01:06:40.928744+0000","last_undegraded":"2024-01-27T01:06:40.928744+0000","last_fullsized":"2024-01-27T01:06:40.928744+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:43.223271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.949848+0000","last_change":"2024-01-27T01:06:40.943329+0000","last_active":"2024-01-27T01:06:40.949848+0000","last_peered":"2024-01-27T01:06:40.949848+0000","last_clean":"2024-01-27T01:06:40.949848+0000","last_became_active":"2024-01-27T01:06:40.943072+0000","last_became_peered":"2024-01-27T01:06:40.943072+0000","last_unstale":"2024-01-27T01:06:40.949848+0000","last_undegraded":"2024-01-27T01:06:40.949848+0000","last_fullsized":"2024-01-27T01:06:40.949848+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:00:48.382771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.925282+0000","last_change":"2024-01-27T01:06:26.873409+0000","last_active":"2024-01-27T01:06:40.925282+0000","last_peered":"2024-01-27T01:06:40.925282+0000","last_clean":"2024-01-27T01:06:40.925282+0000","last_became_active":"2024-01-27T01:06:26.868026+0000","last_became_peered":"2024-01-27T01:06:26.868026+0000","last_unstale":"2024-01-27T01:06:40.925282+0000","last_undegraded":"2024-01-27T01:06:40.925282+0000","last_fullsized":"2024-01-27T01:06:40.925282+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:15.664932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942082+0000","last_change":"2024-01-27T01:06:40.939648+0000","last_active":"2024-01-27T01:06:40.942082+0000","last_peered":"2024-01-27T01:06:40.942082+0000","last_clean":"2024-01-27T01:06:40.942082+0000","last_became_active":"2024-01-27T01:06:40.939090+0000","last_became_peered":"2024-01-27T01:06:40.939090+0000","last_unstale":"2024-01-27T01:06:40.942082+0000","last_undegraded":"2024-01-27T01:06:40.942082+0000","last_fullsized":"2024-01-27T01:06:40.942082+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:10:11.320660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926301+0000","last_change":"2024-01-27T01:06:26.881786+0000","last_active":"2024-01-27T01:06:40.926301+0000","last_peered":"2024-01-27T01:06:40.926301+0000","last_clean":"2024-01-27T01:06:40.926301+0000","last_became_active":"2024-01-27T01:06:26.881527+0000","last_became_peered":"2024-01-27T01:06:26.881527+0000","last_unstale":"2024-01-27T01:06:40.926301+0000","last_undegraded":"2024-01-27T01:06:40.926301+0000","last_fullsized":"2024-01-27T01:06:40.926301+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:43.630831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.948058+0000","last_change":"2024-01-27T01:06:40.945117+0000","last_active":"2024-01-27T01:06:40.948058+0000","last_peered":"2024-01-27T01:06:40.948058+0000","last_clean":"2024-01-27T01:06:40.948058+0000","last_became_active":"2024-01-27T01:06:40.944795+0000","last_became_peered":"2024-01-27T01:06:40.944795+0000","last_unstale":"2024-01-27T01:06:40.948058+0000","last_undegraded":"2024-01-27T01:06:40.948058+0000","last_fullsized":"2024-01-27T01:06:40.948058+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:35:02.237373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.938130+0000","last_change":"2024-01-27T01:06:26.872094+0000","last_active":"2024-01-27T01:06:40.938130+0000","last_peered":"2024-01-27T01:06:40.938130+0000","last_clean":"2024-01-27T01:06:40.938130+0000","last_became_active":"2024-01-27T01:06:26.871711+0000","last_became_peered":"2024-01-27T01:06:26.871711+0000","last_unstale":"2024-01-27T01:06:40.938130+0000","last_undegraded":"2024-01-27T01:06:40.938130+0000","last_fullsized":"2024-01-27T01:06:40.938130+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:24:32.821570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.944084+0000","last_change":"2024-01-27T01:06:40.943231+0000","last_active":"2024-01-27T01:06:40.944084+0000","last_peered":"2024-01-27T01:06:40.944084+0000","last_clean":"2024-01-27T01:06:40.944084+0000","last_became_active":"2024-01-27T01:06:40.942831+0000","last_became_peered":"2024-01-27T01:06:40.942831+0000","last_unstale":"2024-01-27T01:06:40.944084+0000","last_undegraded":"2024-01-27T01:06:40.944084+0000","last_fullsized":"2024-01-27T01:06:40.944084+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:55:50.433195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.932394+0000","last_change":"2024-01-27T01:06:26.879291+0000","last_active":"2024-01-27T01:06:40.932394+0000","last_peered":"2024-01-27T01:06:40.932394+0000","last_clean":"2024-01-27T01:06:40.932394+0000","last_became_active":"2024-01-27T01:06:26.879063+0000","last_became_peered":"2024-01-27T01:06:26.879063+0000","last_unstale":"2024-01-27T01:06:40.932394+0000","last_undegraded":"2024-01-27T01:06:40.932394+0000","last_fullsized":"2024-01-27T01:06:40.932394+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:22:23.545688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.942365+0000","last_change":"2024-01-27T01:06:40.938252+0000","last_active":"2024-01-27T01:06:40.942365+0000","last_peered":"2024-01-27T01:06:40.942365+0000","last_clean":"2024-01-27T01:06:40.942365+0000","last_became_active":"2024-01-27T01:06:40.937940+0000","last_became_peered":"2024-01-27T01:06:40.937940+0000","last_unstale":"2024-01-27T01:06:40.942365+0000","last_undegraded":"2024-01-27T01:06:40.942365+0000","last_fullsized":"2024-01-27T01:06:40.942365+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:59.510860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.941679+0000","last_change":"2024-01-27T01:06:26.870422+0000","last_active":"2024-01-27T01:06:40.941679+0000","last_peered":"2024-01-27T01:06:40.941679+0000","last_clean":"2024-01-27T01:06:40.941679+0000","last_became_active":"2024-01-27T01:06:26.870117+0000","last_became_peered":"2024-01-27T01:06:26.870117+0000","last_unstale":"2024-01-27T01:06:40.941679+0000","last_undegraded":"2024-01-27T01:06:40.941679+0000","last_fullsized":"2024-01-27T01:06:40.941679+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:42:55.987157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.926643+0000","last_change":"2024-01-27T01:06:40.925614+0000","last_active":"2024-01-27T01:06:40.926643+0000","last_peered":"2024-01-27T01:06:40.926643+0000","last_clean":"2024-01-27T01:06:40.926643+0000","last_became_active":"2024-01-27T01:06:40.925180+0000","last_became_peered":"2024-01-27T01:06:40.925180+0000","last_unstale":"2024-01-27T01:06:40.926643+0000","last_undegraded":"2024-01-27T01:06:40.926643+0000","last_fullsized":"2024-01-27T01:06:40.926643+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:14:51.519402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931414+0000","last_change":"2024-01-27T01:06:26.868177+0000","last_active":"2024-01-27T01:06:40.931414+0000","last_peered":"2024-01-27T01:06:40.931414+0000","last_clean":"2024-01-27T01:06:40.931414+0000","last_became_active":"2024-01-27T01:06:26.867906+0000","last_became_peered":"2024-01-27T01:06:26.867906+0000","last_unstale":"2024-01-27T01:06:40.931414+0000","last_undegraded":"2024-01-27T01:06:40.931414+0000","last_fullsized":"2024-01-27T01:06:40.931414+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:17:32.271904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.937853+0000","last_change":"2024-01-27T01:06:40.936971+0000","last_active":"2024-01-27T01:06:40.937853+0000","last_peered":"2024-01-27T01:06:40.937853+0000","last_clean":"2024-01-27T01:06:40.937853+0000","last_became_active":"2024-01-27T01:06:40.936636+0000","last_became_peered":"2024-01-27T01:06:40.936636+0000","last_unstale":"2024-01-27T01:06:40.937853+0000","last_undegraded":"2024-01-27T01:06:40.937853+0000","last_fullsized":"2024-01-27T01:06:40.937853+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:18.204468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.919361+0000","last_change":"2024-01-27T01:06:26.868462+0000","last_active":"2024-01-27T01:06:40.919361+0000","last_peered":"2024-01-27T01:06:40.919361+0000","last_clean":"2024-01-27T01:06:40.919361+0000","last_became_active":"2024-01-27T01:06:26.868203+0000","last_became_peered":"2024-01-27T01:06:26.868203+0000","last_unstale":"2024-01-27T01:06:40.919361+0000","last_undegraded":"2024-01-27T01:06:40.919361+0000","last_fullsized":"2024-01-27T01:06:40.919361+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:10.965469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.943784+0000","last_change":"2024-01-27T01:06:40.939071+0000","last_active":"2024-01-27T01:06:40.943784+0000","last_peered":"2024-01-27T01:06:40.943784+0000","last_clean":"2024-01-27T01:06:40.943784+0000","last_became_active":"2024-01-27T01:06:40.938084+0000","last_became_peered":"2024-01-27T01:06:40.938084+0000","last_unstale":"2024-01-27T01:06:40.943784+0000","last_undegraded":"2024-01-27T01:06:40.943784+0000","last_fullsized":"2024-01-27T01:06:40.943784+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:51:01.037978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.931515+0000","last_change":"2024-01-27T01:06:26.871135+0000","last_active":"2024-01-27T01:06:40.931515+0000","last_peered":"2024-01-27T01:06:40.931515+0000","last_clean":"2024-01-27T01:06:40.931515+0000","last_became_active":"2024-01-27T01:06:26.870854+0000","last_became_peered":"2024-01-27T01:06:26.870854+0000","last_unstale":"2024-01-27T01:06:40.931515+0000","last_undegraded":"2024-01-27T01:06:40.931515+0000","last_fullsized":"2024-01-27T01:06:40.931515+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:12:59.875769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.953931+0000","last_change":"2024-01-27T01:06:40.946055+0000","last_active":"2024-01-27T01:06:40.953931+0000","last_peered":"2024-01-27T01:06:40.953931+0000","last_clean":"2024-01-27T01:06:40.953931+0000","last_became_active":"2024-01-27T01:06:40.945714+0000","last_became_peered":"2024-01-27T01:06:40.945714+0000","last_unstale":"2024-01-27T01:06:40.953931+0000","last_undegraded":"2024-01-27T01:06:40.953931+0000","last_fullsized":"2024-01-27T01:06:40.953931+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:39.907896+0000","last_clean_scrub_stamp":"2024-01-27T01:06:39.907896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:33:43.502121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-01-27T01:06:40.928152+0000","last_change":"2024-01-27T01:06:26.873715+0000","last_active":"2024-01-27T01:06:40.928152+0000","last_peered":"2024-01-27T01:06:40.928152+0000","last_clean":"2024-01-27T01:06:40.928152+0000","last_became_active":"2024-01-27T01:06:26.873430+0000","last_became_peered":"2024-01-27T01:06:26.873430+0000","last_unstale":"2024-01-27T01:06:40.928152+0000","last_undegraded":"2024-01-27T01:06:40.928152+0000","last_fullsized":"2024-01-27T01:06:40.928152+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:06:25.835277+0000","last_clean_scrub_stamp":"2024-01-27T01:06:25.835277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:25:44.342963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3538,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2264,"num_read_kb":4808,"num_write":630,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8750,"ondisk_log_size":8750,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576005,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300676,"kb_used_data":3388,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071164,"statfs":{"total":96636764160,"available":96328871936,"internally_reserved":0,"allocated":3469312,"data_stored":2099383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63200000000000001,"15min":0.61399999999999999},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.312},"max":{"1min":0.82099999999999995,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.63700000000000001,"15min":0.623},"min":{"1min":0.498,"5min":0.39900000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":0.879,"15min":0.879},"last":0.69599999999999995}]},{"osd":1,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.623,"15min":0.61599999999999999},"min":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.624},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65600000000000003,"15min":0.63700000000000001},"min":{"1min":0.505,"5min":0.42099999999999999,"15min":0.34300000000000003},"max":{"1min":0.76800000000000002,"5min":0.84099999999999997,"15min":0.84399999999999997},"last":0.48299999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63400000000000001,"15min":0.61399999999999999},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82099999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64500000000000002,"15min":0.63600000000000001},"min":{"1min":0.49399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79400000000000004,"5min":0.872,"15min":0.88900000000000001},"last":0.66700000000000004}]},{"osd":3,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.628,"15min":0.62},"min":{"1min":0.52100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79900000000000004,"5min":0.876,"15min":0.876},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66800000000000004,"15min":0.64700000000000002},"min":{"1min":0.53200000000000003,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84599999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.72299999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.50900000000000001,"15min":0.48999999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.29999999999999999},"max":{"1min":0.73399999999999999,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.51900000000000002},"min":{"1min":0.35699999999999998,"5min":0.34699999999999998,"15min":0.32200000000000001},"max":{"1min":0.77500000000000002,"5min":0.80300000000000005,"15min":0.86599999999999999},"last":0.60299999999999998}]},{"osd":5,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55800000000000005,"15min":0.52600000000000002},"min":{"1min":0.40999999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.753,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57999999999999996,"15min":0.56499999999999995},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.33700000000000002},"max":{"1min":0.76700000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.65200000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:06:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60999999999999999,"15min":0.58899999999999997},"min":{"1min":0.42499999999999999,"5min":0.42199999999999999,"15min":0.37},"max":{"1min":0.81100000000000005,"5min":0.83999999999999997,"15min":0.84799999999999998},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61299999999999999,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.39100000000000001,"15min":0.35699999999999998},"max":{"1min":0.76100000000000001,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.64100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576005,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308304,"kb_used_data":2824,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063536,"statfs":{"total":96636764160,"available":96321060864,"internally_reserved":0,"allocated":2891776,"data_stored":1510517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61099999999999999,"15min":0.60799999999999998},"min":{"1min":0.41999999999999998,"5min":0.40799999999999997,"15min":0.36399999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.441,"5min":0.432,"15min":0.376},"max":{"1min":0.82499999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.64700000000000002}]},{"osd":1,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79000000000000004,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.40799999999999997,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.56299999999999994}]},{"osd":2,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.39700000000000002,"15min":0.377},"max":{"1min":0.80400000000000005,"5min":0.878,"15min":0.878},"last":0.55000000000000004}]},{"osd":3,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.622,"15min":0.627},"min":{"1min":0.5,"5min":0.378,"15min":0.378},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.90100000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66300000000000003,"15min":0.66000000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.71899999999999997}]},{"osd":4,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.36299999999999999,"5min":0.33300000000000002,"15min":0.32500000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.86599999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33900000000000002,"5min":0.33800000000000002,"15min":0.29799999999999999},"max":{"1min":0.77300000000000002,"5min":0.78500000000000003,"15min":0.88900000000000001},"last":0.58399999999999996}]},{"osd":5,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.31900000000000001},"max":{"1min":0.68999999999999995,"5min":0.75900000000000001,"15min":0.80500000000000005},"last":0.498},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.42199999999999999,"5min":0.37,"15min":0.28899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.874},"last":0.63300000000000001}]},{"osd":7,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45800000000000002,"5min":0.39000000000000001,"15min":0.35799999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.92900000000000005},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.44500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78900000000000003,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.66800000000000004}]}]},{"osd":1,"up_from":13,"seq":55834576005,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300072,"kb_used_data":2784,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071768,"statfs":{"total":96636764160,"available":96329490432,"internally_reserved":0,"allocated":2850816,"data_stored":1505533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.41099999999999998,"15min":0.42099999999999999},"min":{"1min":0.26400000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.64400000000000002,"5min":0.66100000000000003,"15min":0.69599999999999995},"last":0.309},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44500000000000001,"15min":0.45500000000000002},"min":{"1min":0.34000000000000002,"5min":0.28399999999999997,"15min":0.26800000000000002},"max":{"1min":0.57999999999999996,"5min":0.73999999999999999,"15min":0.76600000000000001},"last":0.59499999999999997}]},{"osd":2,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.439,"15min":0.438},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.26200000000000001},"max":{"1min":0.748,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.47599999999999998,"15min":0.48999999999999999},"min":{"1min":0.379,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.67400000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.50700000000000001}]},{"osd":3,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52300000000000002,"15min":0.53300000000000003},"min":{"1min":0.39300000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73699999999999999,"5min":0.84799999999999998,"15min":1.2130000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.51400000000000001},"min":{"1min":0.41699999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68200000000000005,"5min":1.125,"15min":1.22},"last":0.54900000000000004}]},{"osd":4,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.67200000000000004,"15min":0.67400000000000004},"min":{"1min":0.42099999999999999,"5min":0.41099999999999998,"15min":0.39800000000000002},"max":{"1min":0.82699999999999996,"5min":1.242,"15min":1.242},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69099999999999995,"15min":0.69599999999999995},"min":{"1min":0.53700000000000003,"5min":0.496,"15min":0.35699999999999998},"max":{"1min":0.90600000000000003,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.56899999999999995}]},{"osd":5,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73499999999999999,"15min":0.70899999999999996},"min":{"1min":0.53300000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.876,"5min":5.444,"15min":5.444},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.71699999999999997,"15min":0.70799999999999996},"min":{"1min":0.502,"5min":0.47599999999999998,"15min":0.435},"max":{"1min":0.97799999999999998,"5min":2.6669999999999998,"15min":2.6669999999999998},"last":0.46400000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.74199999999999999,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":0.93899999999999995,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.81299999999999994,"15min":0.745},"min":{"1min":0.48799999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.02,"5min":11.223000000000001,"15min":11.223000000000001},"last":0.52700000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.752,"15min":0.75600000000000001},"min":{"1min":0.58999999999999997,"5min":0.52800000000000002,"15min":0.47599999999999998},"max":{"1min":0.96299999999999997,"5min":2.9660000000000002,"15min":2.9660000000000002},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.77700000000000002,"15min":0.76800000000000002},"min":{"1min":0.58799999999999997,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.94499999999999995,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.64400000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576004,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317040,"kb_used_data":3368,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94054800,"statfs":{"total":96636764160,"available":96312115200,"internally_reserved":0,"allocated":3448832,"data_stored":2095991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42899999999999999,"15min":0.42199999999999999},"min":{"1min":0.30299999999999999,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":0.55600000000000005,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.436,"5min":0.45600000000000002,"15min":0.46100000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":0.56499999999999995,"5min":0.72699999999999998,"15min":0.73399999999999999},"last":0.57199999999999995}]},{"osd":2,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.29699999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.58299999999999996,"5min":0.66900000000000004,"15min":0.76800000000000002},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48399999999999999,"15min":0.48899999999999999},"min":{"1min":0.38600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64600000000000002,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.46899999999999997}]},{"osd":3,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.66100000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.504},"min":{"1min":0.36699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64100000000000001,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.497}]},{"osd":4,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.68300000000000005,"15min":0.67700000000000005},"min":{"1min":0.57799999999999996,"5min":0.47999999999999998,"15min":0.42199999999999999},"max":{"1min":0.86399999999999999,"5min":0.91500000000000004,"15min":1.0720000000000001},"last":0.749},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.56999999999999995,"5min":0.47099999999999997,"15min":0.437},"max":{"1min":0.82699999999999996,"5min":1.002,"15min":1.002},"last":0.84099999999999997}]},{"osd":5,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.68999999999999995,"15min":0.70099999999999996},"min":{"1min":0.58199999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.876,"5min":1.0209999999999999,"15min":2.165},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70599999999999996,"15min":0.71199999999999997},"min":{"1min":0.498,"5min":0.48799999999999999,"15min":0.45600000000000002},"max":{"1min":0.89000000000000001,"5min":0.97399999999999998,"15min":1.1399999999999999},"last":0.80700000000000005}]},{"osd":6,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69299999999999995,"15min":0.71099999999999997},"min":{"1min":0.56499999999999995,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.84599999999999997,"5min":0.98899999999999999,"15min":1.0860000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72599999999999998,"15min":0.74099999999999999},"min":{"1min":0.54300000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.93400000000000005,"5min":1.0109999999999999,"15min":1.425},"last":0.82199999999999995}]},{"osd":7,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.74399999999999999,"15min":0.75700000000000001},"min":{"1min":0.61299999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.98899999999999999,"5min":1.044,"15min":1.1259999999999999},"last":0.88},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.76300000000000001,"15min":0.76900000000000002},"min":{"1min":0.58299999999999996,"5min":0.51100000000000001,"15min":0.48499999999999999},"max":{"1min":0.91700000000000004,"5min":1.0289999999999999,"15min":1.1910000000000001},"last":0.89100000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576004,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312372,"kb_used_data":2796,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059468,"statfs":{"total":96636764160,"available":96316895232,"internally_reserved":0,"allocated":2863104,"data_stored":1506211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40500000000000003,"15min":0.40799999999999997},"min":{"1min":0.26600000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.46000000000000002,"5min":0.746,"15min":0.746},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.44500000000000001,"15min":0.44400000000000001},"min":{"1min":0.32300000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.59099999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.35399999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.435,"15min":0.44500000000000001},"min":{"1min":0.32000000000000001,"5min":0.29099999999999998,"15min":0.27000000000000002},"max":{"1min":0.61299999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.439,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.55600000000000005,"5min":0.73599999999999999,"15min":0.78100000000000003},"last":0.40699999999999997}]},{"osd":3,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.498,"15min":0.497},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.33200000000000002},"max":{"1min":0.627,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.439},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48699999999999999,"15min":0.49199999999999999},"min":{"1min":0.373,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":0.59099999999999997,"5min":0.66600000000000004,"15min":0.75700000000000001},"last":0.46300000000000002}]},{"osd":4,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68200000000000005,"15min":0.68000000000000005},"min":{"1min":0.52800000000000002,"5min":0.495,"15min":0.442},"max":{"1min":0.92700000000000005,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68999999999999995,"15min":0.69399999999999995},"min":{"1min":0.50900000000000001,"5min":0.47099999999999997,"15min":0.378},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":1.107},"last":0.60199999999999998}]},{"osd":5,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.69099999999999995,"15min":0.70299999999999996},"min":{"1min":0.52400000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.89500000000000002,"15min":1.167},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70899999999999996,"15min":0.70999999999999996},"min":{"1min":0.47499999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89600000000000002,"5min":0.97499999999999998,"15min":1.05},"last":0.58899999999999997}]},{"osd":6,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71499999999999997,"15min":0.71199999999999997},"min":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.45500000000000002},"max":{"1min":0.81399999999999995,"5min":0.996,"15min":1.077},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70499999999999996,"15min":0.71999999999999997},"min":{"1min":0.497,"5min":0.497,"15min":0.39200000000000002},"max":{"1min":0.876,"5min":0.96699999999999997,"15min":1.135},"last":0.70299999999999996}]},{"osd":7,"last update":"Sat Jan 27 01:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.749,"15min":0.75},"min":{"1min":0.56000000000000005,"5min":0.50800000000000001,"15min":0.47099999999999997},"max":{"1min":0.94099999999999995,"5min":0.97399999999999998,"15min":1.198},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.76500000000000001,"15min":0.76900000000000002},"min":{"1min":0.55000000000000004,"5min":0.53100000000000003,"15min":0.48199999999999998},"max":{"1min":0.94599999999999995,"5min":1.028,"15min":1.0880000000000001},"last":0.68100000000000005}]}]},{"osd":3,"up_from":13,"seq":55834576003,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304184,"kb_used_data":2800,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067656,"statfs":{"total":96636764160,"available":96325279744,"internally_reserved":0,"allocated":2867200,"data_stored":1505681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44700000000000001,"15min":0.42399999999999999},"min":{"1min":0.309,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.59499999999999997,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.31},{"interface":"front","average":{"1min":0.439,"5min":0.44,"15min":0.45500000000000002},"min":{"1min":0.32400000000000001,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.64000000000000001,"5min":0.66700000000000004,"15min":0.84699999999999998},"last":0.66400000000000003}]},{"osd":1,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.624,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66400000000000003,"5min":15.391,"15min":15.391},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.53500000000000003,"15min":0.52400000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70499999999999996,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.59199999999999997}]},{"osd":2,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.54300000000000004,"15min":0.51600000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.60299999999999998,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53900000000000003,"15min":0.51000000000000001},"min":{"1min":0.374,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.5529999999999999,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":0.39000000000000001}]},{"osd":4,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.745,"15min":0.71899999999999997},"min":{"1min":0.56399999999999995,"5min":0.47699999999999998,"15min":0.439},"max":{"1min":0.82699999999999996,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.70899999999999996},"min":{"1min":0.52000000000000002,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":1.0089999999999999,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.64600000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.745,"15min":0.71399999999999997},"min":{"1min":0.52000000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.036,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73899999999999999,"15min":0.72299999999999998},"min":{"1min":0.54900000000000004,"5min":0.435,"15min":0.435},"max":{"1min":1.127,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.495}]},{"osd":6,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.746,"15min":0.72199999999999998},"min":{"1min":0.50700000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.92100000000000004,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.753,"15min":0.74099999999999999},"min":{"1min":0.54900000000000004,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.077,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.54800000000000004}]},{"osd":7,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.755,"15min":0.73899999999999999},"min":{"1min":0.53300000000000003,"5min":0.52500000000000002,"15min":0.46700000000000003},"max":{"1min":1.103,"5min":4.569,"15min":4.569},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.80600000000000005,"15min":0.78300000000000003},"min":{"1min":0.59199999999999997,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.98999999999999999,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.51700000000000002}]}]},{"osd":4,"up_from":13,"seq":55834576005,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312368,"kb_used_data":2792,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059472,"statfs":{"total":96636764160,"available":96316899328,"internally_reserved":0,"allocated":2859008,"data_stored":1506177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.46899999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.84199999999999997},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.61499999999999999,"15min":0.61399999999999999},"min":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78700000000000003,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.68100000000000005}]},{"osd":1,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61899999999999999,"15min":0.627},"min":{"1min":0.46100000000000002,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.76400000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61499999999999999,"15min":0.62},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.79800000000000004,"5min":0.874,"15min":0.874},"last":0.72199999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.59999999999999998},"min":{"1min":0.42399999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.88700000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.51100000000000001,"5min":0.42899999999999999,"15min":0.42799999999999999},"max":{"1min":0.81100000000000005,"5min":0.81499999999999995,"15min":0.89200000000000002},"last":0.76300000000000001}]},{"osd":3,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.61599999999999999,"15min":0.623},"min":{"1min":0.38800000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.83299999999999996,"5min":0.83599999999999997,"15min":0.84199999999999997},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.627},"min":{"1min":0.41299999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.88300000000000001},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.75800000000000001,"15min":0.85899999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.502},"min":{"1min":0.376,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.878},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34999999999999998},"max":{"1min":0.63200000000000001,"5min":0.70099999999999996,"15min":0.88700000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.70699999999999996,"5min":0.79200000000000004,"15min":0.81499999999999995},"last":0.60599999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.42399999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72599999999999998,"5min":0.80400000000000005,"15min":0.90200000000000002},"last":0.629},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59499999999999997,"15min":0.59399999999999997},"min":{"1min":0.41599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.73799999999999999,"5min":0.78700000000000003,"15min":0.92600000000000005},"last":0.70499999999999996}]}]},{"osd":5,"up_from":13,"seq":55834576005,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312384,"kb_used_data":2808,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059456,"statfs":{"total":96636764160,"available":96316882944,"internally_reserved":0,"allocated":2875392,"data_stored":1507195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59399999999999997,"15min":0.60299999999999998},"min":{"1min":0.42999999999999999,"5min":0.40500000000000003,"15min":0.27300000000000002},"max":{"1min":0.80700000000000005,"5min":0.82799999999999996,"15min":1.006},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59399999999999997,"15min":0.60699999999999998},"min":{"1min":0.36599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":1.1719999999999999},"last":0.61799999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63700000000000001,"15min":0.65200000000000002},"min":{"1min":0.39100000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.95899999999999996},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.52100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":1.212},"last":0.51900000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57499999999999996,"15min":0.59399999999999997},"min":{"1min":0.42299999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":1.631},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.625,"15min":0.621},"min":{"1min":0.497,"5min":0.38800000000000001,"15min":0.34000000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.54000000000000004}]},{"osd":3,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.59999999999999998},"min":{"1min":0.39600000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83299999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.625,"15min":0.63},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.879,"5min":0.879,"15min":1.4690000000000001},"last":0.50700000000000001}]},{"osd":4,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.505,"15min":0.48799999999999999},"min":{"1min":0.34899999999999998,"5min":0.32200000000000001,"15min":0.317},"max":{"1min":0.76700000000000002,"5min":0.90200000000000002,"15min":0.94399999999999995},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53500000000000003,"15min":0.51500000000000001},"min":{"1min":0.439,"5min":0.35199999999999998,"15min":0.33900000000000002},"max":{"1min":0.80800000000000005,"5min":0.92500000000000004,"15min":0.93300000000000005},"last":0.46500000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51800000000000002,"15min":0.504},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79400000000000004,"5min":0.96799999999999997,"15min":1.732},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56299999999999994,"15min":0.54500000000000004},"min":{"1min":0.42899999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.753,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.48999999999999999}]},{"osd":7,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.46200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80400000000000005,"5min":0.95399999999999996,"15min":1.0740000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.626,"15min":0.61299999999999999},"min":{"1min":0.504,"5min":0.40799999999999997,"15min":0.39100000000000001},"max":{"1min":0.85199999999999998,"5min":0.996,"15min":1.6919999999999999},"last":0.58599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:06:50.311 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-27T01:06:50.311 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:06:50.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.466+0000 7f432d688700 1 Processor -- start 2024-01-27T01:06:50.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.466+0000 7f432d688700 1 -- start start 2024-01-27T01:06:50.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.467+0000 7f432d688700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4328157810 0x7f432806c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:50.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.467+0000 7f432d688700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432806d680 con 0x7f4328157be0 2024-01-27T01:06:50.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.467+0000 7f432d688700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432806ca70 con 0x7f4328067930 2024-01-27T01:06:50.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.467+0000 7f432d688700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43280acff0 con 0x7f4328157810 2024-01-27T01:06:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.467+0000 7f431ffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4328157be0 0x7f43280cdeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55976/0 (socket says 172.21.15.46:55976) 2024-01-27T01:06:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.467+0000 7f431ffff700 1 -- 172.21.15.46:0/3440534714 learned_addr learned my addr 172.21.15.46:0/3440534714 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f4326ffd700 1 --2- 172.21.15.46:0/3440534714 >> v2:172.21.15.106:3300/0 conn(0x7f4328157810 0x7f432806c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43267fc700 1 -- 172.21.15.46:0/3440534714 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973634465 0 0) 0x7f432806d680 con 0x7f4328157be0 2024-01-27T01:06:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43267fc700 1 -- 172.21.15.46:0/3440534714 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4314003570 con 0x7f4328157be0 2024-01-27T01:06:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43267fc700 1 -- 172.21.15.46:0/3440534714 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 366119667 0 0) 0x7f4314003570 con 0x7f4328157be0 2024-01-27T01:06:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43267fc700 1 -- 172.21.15.46:0/3440534714 >> v2:172.21.15.106:3300/0 conn(0x7f4328157810 msgr2=0x7f432806c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43267fc700 1 --2- 172.21.15.46:0/3440534714 >> v2:172.21.15.106:3300/0 conn(0x7f4328157810 0x7f432806c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43267fc700 1 -- 172.21.15.46:0/3440534714 >> v1:172.21.15.106:6789/0 conn(0x7f4328067930 legacy=0x7f43280ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43267fc700 1 -- 172.21.15.46:0/3440534714 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432806cd50 con 0x7f4328157be0 2024-01-27T01:06:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43267fc700 1 -- 172.21.15.46:0/3440534714 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f4310001470 con 0x7f4328157be0 2024-01-27T01:06:50.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43267fc700 1 -- 172.21.15.46:0/3440534714 >> v1:172.21.15.46:6789/0 conn(0x7f4328157be0 legacy=0x7f43280cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:50.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43267fc700 1 --2- 172.21.15.46:0/3440534714 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4314003a60 0x7f4314003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:50.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43267fc700 1 --2- 172.21.15.46:0/3440534714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4314004810 0x7f4314004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:50.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43267fc700 1 --2- 172.21.15.46:0/3440534714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43140051c0 0x7f431400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:50.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43267fc700 1 -- 172.21.15.46:0/3440534714 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43280acff0 con 0x7f43140051c0 2024-01-27T01:06:50.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43267fc700 1 -- 172.21.15.46:0/3440534714 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431400c160 con 0x7f4314004810 2024-01-27T01:06:50.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43267fc700 1 -- 172.21.15.46:0/3440534714 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f431400c320 con 0x7f4314003a60 2024-01-27T01:06:50.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43277fe700 1 --2- 172.21.15.46:0/3440534714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43140051c0 0x7f431400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:50.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f431ffff700 1 --2- 172.21.15.46:0/3440534714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4314004810 0x7f4314004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:50.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f4326ffd700 1 --2- 172.21.15.46:0/3440534714 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4314003a60 0x7f4314003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:50.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43277fe700 1 -- 172.21.15.46:0/3440534714 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4314003a60 msgr2=0x7f4314003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:50.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43277fe700 1 --2- 172.21.15.46:0/3440534714 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4314003a60 0x7f4314003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43277fe700 1 -- 172.21.15.46:0/3440534714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4314004810 msgr2=0x7f4314004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:50.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.468+0000 7f43277fe700 1 --2- 172.21.15.46:0/3440534714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4314004810 0x7f4314004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.469+0000 7f43277fe700 1 -- 172.21.15.46:0/3440534714 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4314003570 con 0x7f43140051c0 2024-01-27T01:06:50.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.469+0000 7f43277fe700 1 -- 172.21.15.46:0/3440534714 reap_dead start 2024-01-27T01:06:50.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.469+0000 7f43277fe700 1 --2- 172.21.15.46:0/3440534714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43140051c0 0x7f431400ba60 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f4320001fc0 tx=0x7f4320001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c262d59343df4d5f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:50.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.469+0000 7f43267fc700 1 -- 172.21.15.46:0/3440534714 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4320001950 con 0x7f43140051c0 2024-01-27T01:06:50.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.469+0000 7f43267fc700 1 -- 172.21.15.46:0/3440534714 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432000c8b0 con 0x7f43140051c0 2024-01-27T01:06:50.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.469+0000 7f43267fc700 1 -- 172.21.15.46:0/3440534714 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432000cc90 con 0x7f43140051c0 2024-01-27T01:06:50.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.469+0000 7f432d688700 1 -- 172.21.15.46:0/3440534714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43140051c0 msgr2=0x7f431400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:50.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.469+0000 7f432d688700 1 --2- 172.21.15.46:0/3440534714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43140051c0 0x7f431400ba60 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f4320001fc0 tx=0x7f4320001640 comp rx=0 tx=0).stop 2024-01-27T01:06:50.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.470+0000 7f432d688700 1 -- 172.21.15.46:0/3440534714 shutdown_connections 2024-01-27T01:06:50.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.470+0000 7f432d688700 1 --2- 172.21.15.46:0/3440534714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43140051c0 0x7f431400ba60 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.470+0000 7f432d688700 1 -- 172.21.15.46:0/3440534714 >> 172.21.15.46:0/3440534714 conn(0x7f432806b850 msgr2=0x7f43280cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:50.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.470+0000 7f432d688700 1 -- 172.21.15.46:0/3440534714 shutdown_connections 2024-01-27T01:06:50.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.470+0000 7f432d688700 1 -- 172.21.15.46:0/3440534714 wait complete. 2024-01-27T01:06:50.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.471+0000 7f432d688700 1 Processor -- start 2024-01-27T01:06:50.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.471+0000 7f432d688700 1 -- start start 2024-01-27T01:06:50.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f432d688700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43140051c0 0x7f43281499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:50.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f432d688700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43280b03f0 0x7f43280b0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f432d688700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43280b2110 0x7f43280b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f432d688700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4328149ee0 con 0x7f43280b03f0 2024-01-27T01:06:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f432d688700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43280c5130 con 0x7f43140051c0 2024-01-27T01:06:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f432d688700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43280c53a0 con 0x7f43280b2110 2024-01-27T01:06:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f431ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43280b03f0 0x7f43280b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f431ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43280b03f0 0x7f43280b0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37144/0 (socket says 172.21.15.46:37144) 2024-01-27T01:06:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f431ffff700 1 -- 172.21.15.46:0/3926352871 learned_addr learned my addr 172.21.15.46:0/3926352871 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f43277fe700 1 --2- 172.21.15.46:0/3926352871 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43280b2110 0x7f43280b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f4326ffd700 1 --2- 172.21.15.46:0/3926352871 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43140051c0 0x7f43281499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f431ffff700 1 -- 172.21.15.46:0/3926352871 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43280b2110 msgr2=0x7f43280b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f431ffff700 1 --2- 172.21.15.46:0/3926352871 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43280b2110 0x7f43280b25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f431ffff700 1 -- 172.21.15.46:0/3926352871 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43140051c0 msgr2=0x7f43281499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:50.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f431ffff700 1 --2- 172.21.15.46:0/3926352871 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43140051c0 0x7f43281499a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f431ffff700 1 -- 172.21.15.46:0/3926352871 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43280c5680 con 0x7f43280b03f0 2024-01-27T01:06:50.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f431ffff700 1 --2- 172.21.15.46:0/3926352871 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43280b03f0 0x7f43280b0860 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f43100073c0 tx=0x7f4310001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:50.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f431effd700 1 -- 172.21.15.46:0/3926352871 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4310011590 con 0x7f43280b03f0 2024-01-27T01:06:50.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f431effd700 1 -- 172.21.15.46:0/3926352871 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43100116f0 con 0x7f43280b03f0 2024-01-27T01:06:50.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f431effd700 1 -- 172.21.15.46:0/3926352871 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4310011a90 con 0x7f43280b03f0 2024-01-27T01:06:50.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.472+0000 7f432d688700 1 -- 172.21.15.46:0/3926352871 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43280c5960 con 0x7f43280b03f0 2024-01-27T01:06:50.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.473+0000 7f432d688700 1 -- 172.21.15.46:0/3926352871 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43280b2af0 con 0x7f43280b03f0 2024-01-27T01:06:50.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.473+0000 7f432d688700 1 -- 172.21.15.46:0/3926352871 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43280c5e80 con 0x7f43280b03f0 2024-01-27T01:06:50.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.474+0000 7f431effd700 1 -- 172.21.15.46:0/3926352871 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f43100133b0 con 0x7f43280b03f0 2024-01-27T01:06:50.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.475+0000 7f431effd700 1 --2- 172.21.15.46:0/3926352871 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4308066f30 0x7f43080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:50.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.475+0000 7f431effd700 1 -- 172.21.15.46:0/3926352871 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f4310084390 con 0x7f43280b03f0 2024-01-27T01:06:50.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.475+0000 7f4326ffd700 1 --2- 172.21.15.46:0/3926352871 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4308066f30 0x7f43080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:50.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.475+0000 7f4326ffd700 1 --2- 172.21.15.46:0/3926352871 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4308066f30 0x7f43080693e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f43280c7860 tx=0x7f43180091a0 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:50.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.479+0000 7f431effd700 1 -- 172.21.15.46:0/3926352871 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f43280c5e80 con 0x7f43280b03f0 2024-01-27T01:06:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.734+0000 7f432d688700 1 -- 172.21.15.46:0/3926352871 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43280c5e80 con 0x7f43280b03f0 2024-01-27T01:06:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.734+0000 7f431effd700 1 -- 172.21.15.46:0/3926352871 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5697 (secure 0 0 0) 0x7f43280c5e80 con 0x7f43280b03f0 2024-01-27T01:06:50.734 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:06:50.734 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":122,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/1999300009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1999300009},{"type":"v1","addr":"172.21.15.106:6835","nonce":1999300009}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":110},{"gid":10486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/4174263148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4174263148},{"type":"v1","addr":"172.21.15.106:6833","nonce":4174263148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:06:26.286041+0000","modified":"2024-01-27T01:06:29.873402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10866},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10866":{"gid":10866,"name":"a","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2483163449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2483163449},{"type":"v1","addr":"172.21.15.46:6835","nonce":2483163449}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:06:40.353248+0000","modified":"2024-01-27T01:06:43.938885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10854":{"gid":10854,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":8,"addr":"172.21.15.46:6833/1263484152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1263484152},{"type":"v1","addr":"172.21.15.46:6833","nonce":1263484152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-27T01:06:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.736+0000 7f432d688700 1 -- 172.21.15.46:0/3926352871 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4308066f30 msgr2=0x7f43080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.736+0000 7f432d688700 1 --2- 172.21.15.46:0/3926352871 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4308066f30 0x7f43080693e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f43280c7860 tx=0x7f43180091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.736+0000 7f432d688700 1 -- 172.21.15.46:0/3926352871 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43280b03f0 msgr2=0x7f43280b0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.736+0000 7f432d688700 1 --2- 172.21.15.46:0/3926352871 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43280b03f0 0x7f43280b0860 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f43100073c0 tx=0x7f4310001e50 comp rx=0 tx=0).stop 2024-01-27T01:06:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.736+0000 7f432d688700 1 -- 172.21.15.46:0/3926352871 shutdown_connections 2024-01-27T01:06:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.736+0000 7f432d688700 1 --2- 172.21.15.46:0/3926352871 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4308066f30 0x7f43080693e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.736+0000 7f432d688700 1 --2- 172.21.15.46:0/3926352871 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43280b2110 0x7f43280b25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.736+0000 7f432d688700 1 --2- 172.21.15.46:0/3926352871 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43140051c0 0x7f43281499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.736+0000 7f432d688700 1 --2- 172.21.15.46:0/3926352871 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43280b03f0 0x7f43280b0860 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.736+0000 7f432d688700 1 -- 172.21.15.46:0/3926352871 >> 172.21.15.46:0/3926352871 conn(0x7f432806b850 msgr2=0x7f43280cdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:50.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.737+0000 7f432d688700 1 -- 172.21.15.46:0/3926352871 shutdown_connections 2024-01-27T01:06:50.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:50.737+0000 7f432d688700 1 -- 172.21.15.46:0/3926352871 wait complete. 2024-01-27T01:06:50.737 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 122 2024-01-27T01:06:51.152 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-27T01:06:40.353248+0000', 'modified': '2024-01-27T01:06:43.938885+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10854}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10854': {'gid': 10854, 'name': 'c', 'rank': 0, 'incarnation': 118, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.46:6833/1263484152', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 1263484152}, {'type': 'v1', 'addr': '172.21.15.46:6833', 'nonce': 1263484152}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-27T01:06:51.152 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-27T01:06:51.152 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-27T01:06:51.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.296+0000 7fee76e19700 1 Processor -- start 2024-01-27T01:06:51.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.296+0000 7fee76e19700 1 -- start start 2024-01-27T01:06:51.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.297+0000 7fee76e19700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fee701573f0 0x7fee701577c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.297+0000 7fee76e19700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fee700b40a0 0x7fee700b1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.297+0000 7fee76e19700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee700aedf0 con 0x7fee700ade30 2024-01-27T01:06:51.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.297+0000 7fee76e19700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee700b3510 con 0x7fee700b40a0 2024-01-27T01:06:51.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.297+0000 7fee76e19700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee700af0d0 con 0x7fee701573f0 2024-01-27T01:06:51.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.297+0000 7fee6ffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fee700b40a0 0x7fee700b1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.297+0000 7fee6ffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fee700b40a0 0x7fee700b1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37158/0 (socket says 172.21.15.46:37158) 2024-01-27T01:06:51.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.297+0000 7fee6ffff700 1 -- 172.21.15.46:0/2263994079 learned_addr learned my addr 172.21.15.46:0/2263994079 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:51.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.297+0000 7fee74bb5700 1 --2- 172.21.15.46:0/2263994079 >> v2:172.21.15.106:3300/0 conn(0x7fee701573f0 0x7fee701577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.297+0000 7fee6ffff700 1 -- 172.21.15.46:0/2263994079 >> v2:172.21.15.106:3300/0 conn(0x7fee701573f0 msgr2=0x7fee701577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:51.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.297+0000 7fee6ffff700 1 --2- 172.21.15.46:0/2263994079 >> v2:172.21.15.106:3300/0 conn(0x7fee701573f0 0x7fee701577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.297+0000 7fee6ffff700 1 -- 172.21.15.46:0/2263994079 >> v1:172.21.15.106:6789/0 conn(0x7fee700ade30 legacy=0x7fee700afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.297+0000 7fee6ffff700 1 -- 172.21.15.46:0/2263994079 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee700af3b0 con 0x7fee700b40a0 2024-01-27T01:06:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.298+0000 7fee6ffff700 1 --2- 172.21.15.46:0/2263994079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee700b40a0 0x7fee700b1e60 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fee700649a0 tx=0x7fee5c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ca95ba1f2e1385 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.298+0000 7fee6f7fe700 1 -- 172.21.15.46:0/2263994079 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5c005c10 con 0x7fee700b40a0 2024-01-27T01:06:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.298+0000 7fee6f7fe700 1 -- 172.21.15.46:0/2263994079 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5c005d70 con 0x7fee700b40a0 2024-01-27T01:06:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.298+0000 7fee6f7fe700 1 -- 172.21.15.46:0/2263994079 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5c006260 con 0x7fee700b40a0 2024-01-27T01:06:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.298+0000 7fee76e19700 1 -- 172.21.15.46:0/2263994079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee700b40a0 msgr2=0x7fee700b1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.298+0000 7fee76e19700 1 --2- 172.21.15.46:0/2263994079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee700b40a0 0x7fee700b1e60 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fee700649a0 tx=0x7fee5c009220 comp rx=0 tx=0).stop 2024-01-27T01:06:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.298+0000 7fee76e19700 1 -- 172.21.15.46:0/2263994079 shutdown_connections 2024-01-27T01:06:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.298+0000 7fee76e19700 1 --2- 172.21.15.46:0/2263994079 >> v2:172.21.15.106:3300/0 conn(0x7fee701573f0 0x7fee701577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.298+0000 7fee76e19700 1 --2- 172.21.15.46:0/2263994079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee700b40a0 0x7fee700b1e60 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.298+0000 7fee76e19700 1 -- 172.21.15.46:0/2263994079 >> 172.21.15.46:0/2263994079 conn(0x7fee70087f60 msgr2=0x7fee70088330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.298+0000 7fee76e19700 1 -- 172.21.15.46:0/2263994079 shutdown_connections 2024-01-27T01:06:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.298+0000 7fee76e19700 1 -- 172.21.15.46:0/2263994079 wait complete. 2024-01-27T01:06:51.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.300+0000 7fee76e19700 1 Processor -- start 2024-01-27T01:06:51.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.300+0000 7fee76e19700 1 -- start start 2024-01-27T01:06:51.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.300+0000 7fee76e19700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee700ade30 0x7fee70141e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee76e19700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee700b40a0 0x7fee700acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee76e19700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee701573f0 0x7fee70099630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee76e19700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee70142350 con 0x7fee701573f0 2024-01-27T01:06:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee76e19700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee700aba90 con 0x7fee700b40a0 2024-01-27T01:06:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee76e19700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee700abd70 con 0x7fee700ade30 2024-01-27T01:06:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee753b6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee701573f0 0x7fee70099630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee753b6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee701573f0 0x7fee70099630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37172/0 (socket says 172.21.15.46:37172) 2024-01-27T01:06:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee753b6700 1 -- 172.21.15.46:0/426885328 learned_addr learned my addr 172.21.15.46:0/426885328 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee74bb5700 1 --2- 172.21.15.46:0/426885328 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee700ade30 0x7fee70141e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee6ffff700 1 --2- 172.21.15.46:0/426885328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee700b40a0 0x7fee700acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee753b6700 1 -- 172.21.15.46:0/426885328 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee700ade30 msgr2=0x7fee70141e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee753b6700 1 --2- 172.21.15.46:0/426885328 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee700ade30 0x7fee70141e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee753b6700 1 -- 172.21.15.46:0/426885328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee700b40a0 msgr2=0x7fee700acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee753b6700 1 --2- 172.21.15.46:0/426885328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee700b40a0 0x7fee700acb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee753b6700 1 -- 172.21.15.46:0/426885328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5c009480 con 0x7fee701573f0 2024-01-27T01:06:51.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee753b6700 1 --2- 172.21.15.46:0/426885328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee701573f0 0x7fee70099630 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fee6800c8e0 tx=0x7fee68001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:51.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee6cff9700 1 -- 172.21.15.46:0/426885328 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68012630 con 0x7fee701573f0 2024-01-27T01:06:51.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee6cff9700 1 -- 172.21.15.46:0/426885328 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee68012790 con 0x7fee701573f0 2024-01-27T01:06:51.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee76e19700 1 -- 172.21.15.46:0/426885328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee700ac050 con 0x7fee701573f0 2024-01-27T01:06:51.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.301+0000 7fee6cff9700 1 -- 172.21.15.46:0/426885328 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68012b50 con 0x7fee701573f0 2024-01-27T01:06:51.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.302+0000 7fee76e19700 1 -- 172.21.15.46:0/426885328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee700ac610 con 0x7fee701573f0 2024-01-27T01:06:51.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.303+0000 7fee6cff9700 1 -- 172.21.15.46:0/426885328 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fee6801a3a0 con 0x7fee701573f0 2024-01-27T01:06:51.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.303+0000 7fee6cff9700 1 --2- 172.21.15.46:0/426885328 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee54066f30 0x7fee540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.304+0000 7fee6cff9700 1 -- 172.21.15.46:0/426885328 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7fee68084dd0 con 0x7fee701573f0 2024-01-27T01:06:51.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.304+0000 7fee74bb5700 1 --2- 172.21.15.46:0/426885328 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee54066f30 0x7fee540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.304+0000 7fee74bb5700 1 --2- 172.21.15.46:0/426885328 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee54066f30 0x7fee540693e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fee64009d10 tx=0x7fee640091a0 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:51.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.306+0000 7fee8011cb80 1 Processor -- start 2024-01-27T01:06:51.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.306+0000 7fee8011cb80 1 -- start start 2024-01-27T01:06:51.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee8011cb80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556364630010 0x5563646303e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee8011cb80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556364678360 0x55636465a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee8011cb80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55636462f5d0 0x55636465ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee8011cb80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563645339f0 con 0x55636462f5d0 2024-01-27T01:06:51.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee8011cb80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x556364585130 con 0x556364630010 2024-01-27T01:06:51.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee8011cb80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x556364582930 con 0x556364678360 2024-01-27T01:06:51.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee753b6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55636462f5d0 0x55636465ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee753b6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55636462f5d0 0x55636465ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37174/0 (socket says 172.21.15.46:37174) 2024-01-27T01:06:51.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee753b6700 1 -- 172.21.15.46:0/3636484677 learned_addr learned my addr 172.21.15.46:0/3636484677 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:51.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee74bb5700 1 --2- 172.21.15.46:0/3636484677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556364630010 0x5563646303e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee6ffff700 1 --2- 172.21.15.46:0/3636484677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556364678360 0x55636465a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee753b6700 1 -- 172.21.15.46:0/3636484677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556364678360 msgr2=0x55636465a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:51.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee753b6700 1 --2- 172.21.15.46:0/3636484677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556364678360 0x55636465a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee753b6700 1 -- 172.21.15.46:0/3636484677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556364630010 msgr2=0x5563646303e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:51.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee753b6700 1 --2- 172.21.15.46:0/3636484677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556364630010 0x5563646303e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee753b6700 1 -- 172.21.15.46:0/3636484677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563643bdb10 con 0x55636462f5d0 2024-01-27T01:06:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee74bb5700 1 --2- 172.21.15.46:0/3636484677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556364630010 0x5563646303e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.307+0000 7fee753b6700 1 --2- 172.21.15.46:0/3636484677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55636462f5d0 0x55636465ad80 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fee68055dd0 tx=0x7fee68082b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1274693ae7b9a39 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.308+0000 7fee59ffb700 1 -- 172.21.15.46:0/3636484677 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68071270 con 0x55636462f5d0 2024-01-27T01:06:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.308+0000 7fee59ffb700 1 -- 172.21.15.46:0/3636484677 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee680588a0 con 0x55636462f5d0 2024-01-27T01:06:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.308+0000 7fee59ffb700 1 -- 172.21.15.46:0/3636484677 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6805c570 con 0x55636462f5d0 2024-01-27T01:06:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.308+0000 7fee8011cb80 1 -- 172.21.15.46:0/3636484677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55636462f5d0 msgr2=0x55636465ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.308+0000 7fee8011cb80 1 --2- 172.21.15.46:0/3636484677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55636462f5d0 0x55636465ad80 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fee68055dd0 tx=0x7fee68082b20 comp rx=0 tx=0).stop 2024-01-27T01:06:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.308+0000 7fee8011cb80 1 -- 172.21.15.46:0/3636484677 shutdown_connections 2024-01-27T01:06:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.308+0000 7fee8011cb80 1 --2- 172.21.15.46:0/3636484677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55636462f5d0 0x55636465ad80 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.308+0000 7fee8011cb80 1 --2- 172.21.15.46:0/3636484677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556364630010 0x5563646303e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.308+0000 7fee8011cb80 1 --2- 172.21.15.46:0/3636484677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556364678360 0x55636465a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.308+0000 7fee8011cb80 1 -- 172.21.15.46:0/3636484677 >> 172.21.15.46:0/3636484677 conn(0x55636462fa70 msgr2=0x556364678820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.308+0000 7fee8011cb80 1 -- 172.21.15.46:0/3636484677 shutdown_connections 2024-01-27T01:06:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.308+0000 7fee8011cb80 1 -- 172.21.15.46:0/3636484677 wait complete. 2024-01-27T01:06:51.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.309+0000 7fee8011cb80 1 -- start start 2024-01-27T01:06:51.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.309+0000 7fee8011cb80 1 Processor -- start 2024-01-27T01:06:51.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.310+0000 7fee8011cb80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55636462f5d0 0x556364655650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.310+0000 7fee8011cb80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556364630010 0x55636465b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.310+0000 7fee8011cb80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556364678360 0x556364675e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.310+0000 7fee8011cb80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55636457e720 con 0x556364630010 2024-01-27T01:06:51.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.310+0000 7fee8011cb80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55636452a1b0 con 0x556364678360 2024-01-27T01:06:51.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.310+0000 7fee8011cb80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x556364529e50 con 0x55636462f5d0 2024-01-27T01:06:51.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.310+0000 7fee8011cb80 10 client.0 _ll_register_callbacks cb 0x55636438c7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-27T01:06:51.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.310+0000 7fee6ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556364630010 0x55636465b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.310+0000 7fee6ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556364630010 0x55636465b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37188/0 (socket says 172.21.15.46:37188) 2024-01-27T01:06:51.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.310+0000 7fee6ffff700 1 -- 172.21.15.46:0/3624536872 learned_addr learned my addr 172.21.15.46:0/3624536872 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:51.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.310+0000 7fee74bb5700 1 --2- 172.21.15.46:0/3624536872 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55636462f5d0 0x556364655650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.310+0000 7fee753b6700 1 --2- 172.21.15.46:0/3624536872 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556364678360 0x556364675e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.311+0000 7fee6ffff700 1 -- 172.21.15.46:0/3624536872 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55636462f5d0 msgr2=0x556364655650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:51.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.311+0000 7fee6ffff700 1 --2- 172.21.15.46:0/3624536872 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55636462f5d0 0x556364655650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.311+0000 7fee6ffff700 1 -- 172.21.15.46:0/3624536872 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556364678360 msgr2=0x556364675e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:51.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.311+0000 7fee6ffff700 1 --2- 172.21.15.46:0/3624536872 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556364678360 0x556364675e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.311+0000 7fee6ffff700 1 -- 172.21.15.46:0/3624536872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee700ac610 con 0x556364630010 2024-01-27T01:06:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.311+0000 7fee6ffff700 1 --2- 172.21.15.46:0/3624536872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556364630010 0x55636465b350 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fee5c007fd0 tx=0x7fee5c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3588a1a764382b21 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.311+0000 7fee4effd700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-27T01:06:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.311+0000 7fee4effd700 1 -- 172.21.15.46:0/3624536872 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5c009320 con 0x556364630010 2024-01-27T01:06:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.311+0000 7fee4effd700 1 -- 172.21.15.46:0/3624536872 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7fee5c005ee0 con 0x556364630010 2024-01-27T01:06:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.311+0000 7fee8011cb80 1 -- 172.21.15.46:0/3624536872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556364623190 con 0x556364630010 2024-01-27T01:06:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.311+0000 7fee4effd700 1 -- 172.21.15.46:0/3624536872 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7fee5c0062b0 con 0x556364630010 2024-01-27T01:06:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.311+0000 7fee8011cb80 10 client.11160 fetch_fsmap learned FSMap version 122 2024-01-27T01:06:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.311+0000 7fee8011cb80 1 -- 172.21.15.46:0/3624536872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x55636454c290 con 0x556364630010 2024-01-27T01:06:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.311+0000 7fee4effd700 1 -- 172.21.15.46:0/3624536872 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556364623190 con 0x556364630010 2024-01-27T01:06:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.311+0000 7fee4effd700 1 -- 172.21.15.46:0/3624536872 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 122) v1 ==== 3653+0+0 (secure 0 0 0) 0x7fee5c025070 con 0x556364630010 2024-01-27T01:06:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.312+0000 7fee8011cb80 10 client.11160 fetch_fsmap finished waiting for FSMap version 122 2024-01-27T01:06:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.312+0000 7fee8011cb80 10 client.11160 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-27T01:06:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.312+0000 7fee8011cb80 20 client.11160 populate_metadata read hostname 'smithi046' 2024-01-27T01:06:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.312+0000 7fee8011cb80 1 --2- 172.21.15.46:0/3624536872 >> [v2:172.21.15.46:6832/1263484152,v1:172.21.15.46:6833/1263484152] conn(0x556364687a80 0x556364687e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.312+0000 7fee8011cb80 4 client.11160 mds_command: new command op to 10854 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-27T01:06:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.312+0000 7fee8011cb80 1 -- 172.21.15.46:0/3624536872 --> [v2:172.21.15.46:6832/1263484152,v1:172.21.15.46:6833/1263484152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556364520ee0 con 0x556364687a80 2024-01-27T01:06:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.312+0000 7fee74bb5700 1 --2- 172.21.15.46:0/3624536872 >> [v2:172.21.15.46:6832/1263484152,v1:172.21.15.46:6833/1263484152] conn(0x556364687a80 0x556364687e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.312+0000 7fee74bb5700 1 --2- 172.21.15.46:0/3624536872 >> [v2:172.21.15.46:6832/1263484152,v1:172.21.15.46:6833/1263484152] conn(0x556364687a80 0x556364687e50 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e22c23d8f71de036 server_cookie=e811f7cac7020037 in_seq=0 out_seq=0 2024-01-27T01:06:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.312+0000 7fee4effd700 10 client.11160 ms_handle_connect on v2:172.21.15.46:6832/1263484152 2024-01-27T01:06:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.313+0000 7fee4effd700 1 -- 172.21.15.46:0/3624536872 <== mds.0 v2:172.21.15.46:6832/1263484152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x556364520ee0 con 0x556364687a80 2024-01-27T01:06:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.313+0000 7fee4effd700 10 client.11160 handle_command_reply: tid=0 2024-01-27T01:06:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.313+0000 7fee8011cb80 1 client.11160 shutdown 2024-01-27T01:06:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.313+0000 7fee8011cb80 1 -- 172.21.15.46:0/3624536872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556364630010 msgr2=0x55636465b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.313+0000 7fee8011cb80 1 --2- 172.21.15.46:0/3624536872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556364630010 0x55636465b350 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fee5c007fd0 tx=0x7fee5c016000 comp rx=0 tx=0).stop 2024-01-27T01:06:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.313+0000 7fee8011cb80 1 -- 172.21.15.46:0/3624536872 shutdown_connections 2024-01-27T01:06:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.313+0000 7fee8011cb80 1 --2- 172.21.15.46:0/3624536872 >> [v2:172.21.15.46:6832/1263484152,v1:172.21.15.46:6833/1263484152] conn(0x556364687a80 0x556364687e50 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.313+0000 7fee8011cb80 1 --2- 172.21.15.46:0/3624536872 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556364678360 0x556364675e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.313+0000 7fee8011cb80 1 --2- 172.21.15.46:0/3624536872 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55636462f5d0 0x556364655650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.313+0000 7fee4effd700 0 client.11160 ms_handle_reset on v2:172.21.15.46:6832/1263484152 2024-01-27T01:06:51.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.313+0000 7fee8011cb80 1 --2- 172.21.15.46:0/3624536872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556364630010 0x55636465b350 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.313+0000 7fee8011cb80 1 -- 172.21.15.46:0/3624536872 >> 172.21.15.46:0/3624536872 conn(0x55636462fa70 msgr2=0x556364629240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:51.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.313+0000 7fee8011cb80 1 -- 172.21.15.46:0/3624536872 shutdown_connections 2024-01-27T01:06:51.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.313+0000 7fee8011cb80 1 -- 172.21.15.46:0/3624536872 wait complete. 2024-01-27T01:06:51.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.313+0000 7fee8011cb80 20 client.11160 trim_cache size 0 max 16384 2024-01-27T01:06:51.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.331+0000 7fee8011cb80 1 Processor -- start 2024-01-27T01:06:51.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.332+0000 7fee8011cb80 1 -- start start 2024-01-27T01:06:51.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.332+0000 7fee8011cb80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55636462f5d0 0x55636466f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.332+0000 7fee8011cb80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556364630010 0x55636468e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.332+0000 7fee8011cb80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556364678360 0x55636468e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.332+0000 7fee8011cb80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x556364476e40 con 0x556364678360 2024-01-27T01:06:51.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.332+0000 7fee8011cb80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563646306d0 con 0x55636462f5d0 2024-01-27T01:06:51.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.332+0000 7fee8011cb80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x556364677bf0 con 0x556364630010 2024-01-27T01:06:51.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.332+0000 7fee753b6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556364678360 0x55636468e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.332+0000 7fee753b6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556364678360 0x55636468e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37202/0 (socket says 172.21.15.46:37202) 2024-01-27T01:06:51.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.332+0000 7fee753b6700 1 -- 172.21.15.46:0/997752136 learned_addr learned my addr 172.21.15.46:0/997752136 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:51.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.332+0000 7fee74bb5700 1 --2- 172.21.15.46:0/997752136 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55636462f5d0 0x55636466f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.332+0000 7fee6ffff700 1 --2- 172.21.15.46:0/997752136 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556364630010 0x55636468e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.333+0000 7fee753b6700 1 -- 172.21.15.46:0/997752136 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556364630010 msgr2=0x55636468e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:51.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.333+0000 7fee753b6700 1 --2- 172.21.15.46:0/997752136 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556364630010 0x55636468e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.333+0000 7fee753b6700 1 -- 172.21.15.46:0/997752136 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55636462f5d0 msgr2=0x55636466f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:51.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.333+0000 7fee753b6700 1 --2- 172.21.15.46:0/997752136 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55636462f5d0 0x55636466f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.333+0000 7fee753b6700 1 -- 172.21.15.46:0/997752136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556364415c80 con 0x556364678360 2024-01-27T01:06:51.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.333+0000 7fee753b6700 1 --2- 172.21.15.46:0/997752136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556364678360 0x55636468e590 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fee6800f040 tx=0x7fee68070000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56ab825ed186a72 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:51.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.333+0000 7fee5a7fc700 1 -- 172.21.15.46:0/997752136 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68012420 con 0x556364678360 2024-01-27T01:06:51.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.333+0000 7fee5a7fc700 1 -- 172.21.15.46:0/997752136 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee68053970 con 0x556364678360 2024-01-27T01:06:51.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.333+0000 7fee5a7fc700 1 -- 172.21.15.46:0/997752136 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68055830 con 0x556364678360 2024-01-27T01:06:51.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.334+0000 7fee8011cb80 1 -- 172.21.15.46:0/997752136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556364678360 msgr2=0x55636468e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.334+0000 7fee8011cb80 1 --2- 172.21.15.46:0/997752136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556364678360 0x55636468e590 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fee6800f040 tx=0x7fee68070000 comp rx=0 tx=0).stop 2024-01-27T01:06:51.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.334+0000 7fee8011cb80 1 -- 172.21.15.46:0/997752136 shutdown_connections 2024-01-27T01:06:51.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.334+0000 7fee8011cb80 1 --2- 172.21.15.46:0/997752136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556364678360 0x55636468e590 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.334+0000 7fee8011cb80 1 --2- 172.21.15.46:0/997752136 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55636462f5d0 0x55636466f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.334+0000 7fee8011cb80 1 --2- 172.21.15.46:0/997752136 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556364630010 0x55636468e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.334+0000 7fee8011cb80 1 -- 172.21.15.46:0/997752136 >> 172.21.15.46:0/997752136 conn(0x55636462fa70 msgr2=0x556364676680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:51.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.334+0000 7fee8011cb80 1 -- 172.21.15.46:0/997752136 shutdown_connections 2024-01-27T01:06:51.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.334+0000 7fee8011cb80 1 -- 172.21.15.46:0/997752136 wait complete. 2024-01-27T01:06:51.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.334+0000 7fee8011cb80 1 -- start start 2024-01-27T01:06:51.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.335+0000 7fee8011cb80 1 Processor -- start 2024-01-27T01:06:51.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.335+0000 7fee8011cb80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55636462f5d0 0x5563646c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.335+0000 7fee8011cb80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556364630010 0x55636468e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.335+0000 7fee8011cb80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556364678360 0x55636468e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee8011cb80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55636462e930 con 0x55636462f5d0 2024-01-27T01:06:51.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee8011cb80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x556364629810 con 0x556364630010 2024-01-27T01:06:51.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee8011cb80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x556364684b90 con 0x556364678360 2024-01-27T01:06:51.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee8011cb80 10 client.0 _ll_register_callbacks cb 0x55636438c7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-27T01:06:51.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee74bb5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55636462f5d0 0x5563646c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee74bb5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55636462f5d0 0x5563646c62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37212/0 (socket says 172.21.15.46:37212) 2024-01-27T01:06:51.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee74bb5700 1 -- 172.21.15.46:0/1367967252 learned_addr learned my addr 172.21.15.46:0/1367967252 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:51.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee753b6700 1 --2- 172.21.15.46:0/1367967252 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556364678360 0x55636468e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee6ffff700 1 --2- 172.21.15.46:0/1367967252 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556364630010 0x55636468e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee74bb5700 1 -- 172.21.15.46:0/1367967252 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556364678360 msgr2=0x55636468e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:51.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee74bb5700 1 --2- 172.21.15.46:0/1367967252 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556364678360 0x55636468e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee74bb5700 1 -- 172.21.15.46:0/1367967252 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556364630010 msgr2=0x55636468e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:51.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee74bb5700 1 --2- 172.21.15.46:0/1367967252 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556364630010 0x55636468e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee74bb5700 1 -- 172.21.15.46:0/1367967252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5563643bdb10 con 0x55636462f5d0 2024-01-27T01:06:51.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee753b6700 1 --2- 172.21.15.46:0/1367967252 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556364678360 0x55636468e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:06:51.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee6ffff700 1 --2- 172.21.15.46:0/1367967252 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556364630010 0x55636468e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:06:51.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee74bb5700 1 --2- 172.21.15.46:0/1367967252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55636462f5d0 0x5563646c62d0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x55636465b0f0 tx=0x7fee64019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a403bf3353fadf0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:51.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee4effd700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-27T01:06:51.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee4effd700 1 -- 172.21.15.46:0/1367967252 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6402be70 con 0x55636462f5d0 2024-01-27T01:06:51.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee4effd700 1 -- 172.21.15.46:0/1367967252 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7fee64028040 con 0x55636462f5d0 2024-01-27T01:06:51.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.336+0000 7fee8011cb80 1 -- 172.21.15.46:0/1367967252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fee70098f20 con 0x55636462f5d0 2024-01-27T01:06:51.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.337+0000 7fee4effd700 1 -- 172.21.15.46:0/1367967252 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7fee64014860 con 0x55636462f5d0 2024-01-27T01:06:51.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.337+0000 7fee8011cb80 10 client.11166 fetch_fsmap learned FSMap version 122 2024-01-27T01:06:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.337+0000 7fee8011cb80 1 -- 172.21.15.46:0/1367967252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x5563643bbfe0 con 0x55636462f5d0 2024-01-27T01:06:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.337+0000 7fee4effd700 1 -- 172.21.15.46:0/1367967252 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee70098f20 con 0x55636462f5d0 2024-01-27T01:06:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.337+0000 7fee4effd700 1 -- 172.21.15.46:0/1367967252 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 122) v1 ==== 3653+0+0 (secure 0 0 0) 0x7fee640149c0 con 0x55636462f5d0 2024-01-27T01:06:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.337+0000 7fee8011cb80 10 client.11166 fetch_fsmap finished waiting for FSMap version 122 2024-01-27T01:06:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.337+0000 7fee8011cb80 10 client.11166 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-27T01:06:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.337+0000 7fee8011cb80 20 client.11166 populate_metadata read hostname 'smithi046' 2024-01-27T01:06:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.337+0000 7fee8011cb80 1 --2- 172.21.15.46:0/1367967252 >> [v2:172.21.15.46:6832/1263484152,v1:172.21.15.46:6833/1263484152] conn(0x5563646dd370 0x5563646dd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.337+0000 7fee8011cb80 4 client.11166 mds_command: new command op to 10854 tid=0[{"prefix": "status"}] 2024-01-27T01:06:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.337+0000 7fee8011cb80 1 -- 172.21.15.46:0/1367967252 --> [v2:172.21.15.46:6832/1263484152,v1:172.21.15.46:6833/1263484152] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fee680588a0 con 0x5563646dd370 2024-01-27T01:06:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.337+0000 7fee6ffff700 1 --2- 172.21.15.46:0/1367967252 >> [v2:172.21.15.46:6832/1263484152,v1:172.21.15.46:6833/1263484152] conn(0x5563646dd370 0x5563646dd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.337+0000 7fee4effd700 10 client.11166 ms_handle_connect on v2:172.21.15.46:6832/1263484152 2024-01-27T01:06:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.337+0000 7fee6ffff700 1 --2- 172.21.15.46:0/1367967252 >> [v2:172.21.15.46:6832/1263484152,v1:172.21.15.46:6833/1263484152] conn(0x5563646dd370 0x5563646dd740 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f01389e165380814 server_cookie=5ec54c3c5b0c66cc in_seq=0 out_seq=0 2024-01-27T01:06:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.338+0000 7fee4effd700 1 -- 172.21.15.46:0/1367967252 <== mds.0 v2:172.21.15.46:6832/1263484152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x7fee680588a0 con 0x5563646dd370 2024-01-27T01:06:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.338+0000 7fee4effd700 10 client.11166 handle_command_reply: tid=0 2024-01-27T01:06:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.338+0000 7fee8011cb80 1 client.11166 shutdown 2024-01-27T01:06:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.338+0000 7fee8011cb80 1 -- 172.21.15.46:0/1367967252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55636462f5d0 msgr2=0x5563646c62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.338+0000 7fee8011cb80 1 --2- 172.21.15.46:0/1367967252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55636462f5d0 0x5563646c62d0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x55636465b0f0 tx=0x7fee64019000 comp rx=0 tx=0).stop 2024-01-27T01:06:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.338+0000 7fee8011cb80 1 -- 172.21.15.46:0/1367967252 shutdown_connections 2024-01-27T01:06:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.338+0000 7fee8011cb80 1 --2- 172.21.15.46:0/1367967252 >> [v2:172.21.15.46:6832/1263484152,v1:172.21.15.46:6833/1263484152] conn(0x5563646dd370 0x5563646dd740 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.338+0000 7fee8011cb80 1 --2- 172.21.15.46:0/1367967252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55636462f5d0 0x5563646c62d0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.338+0000 7fee8011cb80 1 --2- 172.21.15.46:0/1367967252 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x556364678360 0x55636468e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.338+0000 7fee4effd700 0 client.11166 ms_handle_reset on v2:172.21.15.46:6832/1263484152 2024-01-27T01:06:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.338+0000 7fee8011cb80 1 --2- 172.21.15.46:0/1367967252 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556364630010 0x55636468e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.338+0000 7fee8011cb80 1 -- 172.21.15.46:0/1367967252 >> 172.21.15.46:0/1367967252 conn(0x55636462fa70 msgr2=0x556364676680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.338+0000 7fee8011cb80 1 -- 172.21.15.46:0/1367967252 shutdown_connections 2024-01-27T01:06:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.338+0000 7fee8011cb80 1 -- 172.21.15.46:0/1367967252 wait complete. 2024-01-27T01:06:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.338+0000 7fee8011cb80 20 client.11166 trim_cache size 0 max 16384 2024-01-27T01:06:51.344 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:06:51.344 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-27T01:06:51.344 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-27T01:06:51.344 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 10854, 2024-01-27T01:06:51.344 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-27T01:06:51.344 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-27T01:06:51.344 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "backup_fs", 2024-01-27T01:06:51.344 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 10.407178576, 2024-01-27T01:06:51.344 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 122, 2024-01-27T01:06:51.344 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 227, 2024-01-27T01:06:51.345 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 227, 2024-01-27T01:06:51.345 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 35.510911458000002 2024-01-27T01:06:51.345 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:06:51.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.339+0000 7fee76e19700 1 -- 172.21.15.46:0/426885328 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee54066f30 msgr2=0x7fee540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.339+0000 7fee76e19700 1 --2- 172.21.15.46:0/426885328 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee54066f30 0x7fee540693e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fee64009d10 tx=0x7fee640091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.339+0000 7fee76e19700 1 -- 172.21.15.46:0/426885328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee701573f0 msgr2=0x7fee70099630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.339+0000 7fee76e19700 1 --2- 172.21.15.46:0/426885328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee701573f0 0x7fee70099630 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fee6800c8e0 tx=0x7fee68001710 comp rx=0 tx=0).stop 2024-01-27T01:06:51.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.339+0000 7fee76e19700 1 -- 172.21.15.46:0/426885328 shutdown_connections 2024-01-27T01:06:51.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.339+0000 7fee76e19700 1 --2- 172.21.15.46:0/426885328 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee54066f30 0x7fee540693e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.339+0000 7fee76e19700 1 --2- 172.21.15.46:0/426885328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee701573f0 0x7fee70099630 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.339+0000 7fee76e19700 1 --2- 172.21.15.46:0/426885328 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee700ade30 0x7fee70141e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.339+0000 7fee76e19700 1 --2- 172.21.15.46:0/426885328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee700b40a0 0x7fee700acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.339+0000 7fee76e19700 1 -- 172.21.15.46:0/426885328 >> 172.21.15.46:0/426885328 conn(0x7fee70087f60 msgr2=0x7fee70156be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:51.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.339+0000 7fee76e19700 1 -- 172.21.15.46:0/426885328 shutdown_connections 2024-01-27T01:06:51.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.339+0000 7fee76e19700 1 -- 172.21.15.46:0/426885328 wait complete. 2024-01-27T01:06:51.351 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-27T01:06:51.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.485+0000 7f8a0ec38700 1 Processor -- start 2024-01-27T01:06:51.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.485+0000 7f8a0ec38700 1 -- start start 2024-01-27T01:06:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.486+0000 7f8a0ec38700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a08157340 0x7f8a0815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.486+0000 7f8a0ec38700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8a080ae520 0x7f8a080b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.486+0000 7f8a0ec38700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a08158ae0 con 0x7f8a080b3fb0 2024-01-27T01:06:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.486+0000 7f8a0ec38700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a080b4430 con 0x7f8a080ae520 2024-01-27T01:06:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.486+0000 7f8a0ec38700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0815b650 con 0x7f8a08157340 2024-01-27T01:06:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.486+0000 7f8a0d1d5700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8a080b3fb0 0x7f8a08157710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55990/0 (socket says 172.21.15.46:55990) 2024-01-27T01:06:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.486+0000 7f8a0d1d5700 1 -- 172.21.15.46:0/2444355449 learned_addr learned my addr 172.21.15.46:0/2444355449 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.486+0000 7f8a0c9d4700 1 --2- 172.21.15.46:0/2444355449 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a08157340 0x7f8a0815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.486+0000 7f8a07fff700 1 --2- 172.21.15.46:0/2444355449 >> v2:172.21.15.106:3300/0 conn(0x7f8a080ae520 0x7f8a080b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.486+0000 7f8a077fe700 1 -- 172.21.15.46:0/2444355449 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645172331 0 0) 0x7f8a08158ae0 con 0x7f8a080b3fb0 2024-01-27T01:06:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.487+0000 7f8a077fe700 1 -- 172.21.15.46:0/2444355449 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89f8003570 con 0x7f8a080b3fb0 2024-01-27T01:06:51.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.487+0000 7f8a0c9d4700 1 -- 172.21.15.46:0/2444355449 >> v2:172.21.15.106:3300/0 conn(0x7f8a080ae520 msgr2=0x7f8a080b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:51.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.487+0000 7f8a0c9d4700 1 --2- 172.21.15.46:0/2444355449 >> v2:172.21.15.106:3300/0 conn(0x7f8a080ae520 0x7f8a080b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.487+0000 7f8a0c9d4700 1 -- 172.21.15.46:0/2444355449 >> v1:172.21.15.46:6789/0 conn(0x7f8a080b3fb0 legacy=0x7f8a08157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.487+0000 7f8a0c9d4700 1 -- 172.21.15.46:0/2444355449 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0815c020 con 0x7f8a08157340 2024-01-27T01:06:51.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.487+0000 7f8a0c9d4700 1 --2- 172.21.15.46:0/2444355449 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a08157340 0x7f8a0815bae0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f89fc0098e0 tx=0x7f89fc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16e8be93b8cd2bef server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:51.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.487+0000 7f8a077fe700 1 -- 172.21.15.46:0/2444355449 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc011630 con 0x7f8a08157340 2024-01-27T01:06:51.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.487+0000 7f8a077fe700 1 -- 172.21.15.46:0/2444355449 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89fc011790 con 0x7f8a08157340 2024-01-27T01:06:51.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.487+0000 7f8a077fe700 1 -- 172.21.15.46:0/2444355449 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc011b30 con 0x7f8a08157340 2024-01-27T01:06:51.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.488+0000 7f8a0ec38700 1 -- 172.21.15.46:0/2444355449 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a08157340 msgr2=0x7f8a0815bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.488+0000 7f8a0ec38700 1 --2- 172.21.15.46:0/2444355449 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a08157340 0x7f8a0815bae0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f89fc0098e0 tx=0x7f89fc0091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.488+0000 7f8a0ec38700 1 -- 172.21.15.46:0/2444355449 shutdown_connections 2024-01-27T01:06:51.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.488+0000 7f8a0ec38700 1 --2- 172.21.15.46:0/2444355449 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a08157340 0x7f8a0815bae0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.488+0000 7f8a0ec38700 1 --2- 172.21.15.46:0/2444355449 >> v2:172.21.15.106:3300/0 conn(0x7f8a080ae520 0x7f8a080b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.488+0000 7f8a0ec38700 1 -- 172.21.15.46:0/2444355449 >> 172.21.15.46:0/2444355449 conn(0x7f8a080ce530 msgr2=0x7f8a080ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:51.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.488+0000 7f8a0ec38700 1 -- 172.21.15.46:0/2444355449 shutdown_connections 2024-01-27T01:06:51.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.488+0000 7f8a0ec38700 1 -- 172.21.15.46:0/2444355449 wait complete. 2024-01-27T01:06:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.489+0000 7f8a0ec38700 1 Processor -- start 2024-01-27T01:06:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.489+0000 7f8a0ec38700 1 -- start start 2024-01-27T01:06:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a0ec38700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a080ae520 0x7f8a080c5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a0ec38700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a080b3fb0 0x7f8a080c7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a0ec38700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a08157340 0x7f8a080c76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a0ec38700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0816d020 con 0x7f8a080ae520 2024-01-27T01:06:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a0ec38700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a08155a40 con 0x7f8a08157340 2024-01-27T01:06:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a0ec38700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a08155d20 con 0x7f8a080b3fb0 2024-01-27T01:06:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a0c9d4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a080ae520 0x7f8a080c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a0c9d4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a080ae520 0x7f8a080c5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37228/0 (socket says 172.21.15.46:37228) 2024-01-27T01:06:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a0c9d4700 1 -- 172.21.15.46:0/1242123494 learned_addr learned my addr 172.21.15.46:0/1242123494 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a0d1d5700 1 --2- 172.21.15.46:0/1242123494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a08157340 0x7f8a080c76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a07fff700 1 --2- 172.21.15.46:0/1242123494 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a080b3fb0 0x7f8a080c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a0c9d4700 1 -- 172.21.15.46:0/1242123494 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a080b3fb0 msgr2=0x7f8a080c7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a0c9d4700 1 --2- 172.21.15.46:0/1242123494 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a080b3fb0 0x7f8a080c7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a0c9d4700 1 -- 172.21.15.46:0/1242123494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a08157340 msgr2=0x7f8a080c76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a0c9d4700 1 --2- 172.21.15.46:0/1242123494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a08157340 0x7f8a080c76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a0c9d4700 1 -- 172.21.15.46:0/1242123494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89fc009420 con 0x7f8a080ae520 2024-01-27T01:06:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a07fff700 1 --2- 172.21.15.46:0/1242123494 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a080b3fb0 0x7f8a080c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:51.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.490+0000 7f8a0d1d5700 1 --2- 172.21.15.46:0/1242123494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a08157340 0x7f8a080c76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:51.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.491+0000 7f8a0c9d4700 1 --2- 172.21.15.46:0/1242123494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a080ae520 0x7f8a080c5690 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009ed0 tx=0x7f89fc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:51.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.491+0000 7f8a04ff9700 1 -- 172.21.15.46:0/1242123494 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc011b10 con 0x7f8a080ae520 2024-01-27T01:06:51.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.491+0000 7f8a04ff9700 1 -- 172.21.15.46:0/1242123494 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89fc0253f0 con 0x7f8a080ae520 2024-01-27T01:06:51.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.491+0000 7f8a0ec38700 1 -- 172.21.15.46:0/1242123494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a08156000 con 0x7f8a080ae520 2024-01-27T01:06:51.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.491+0000 7f8a04ff9700 1 -- 172.21.15.46:0/1242123494 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc0257b0 con 0x7f8a080ae520 2024-01-27T01:06:51.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.491+0000 7f8a0ec38700 1 -- 172.21.15.46:0/1242123494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a080c5bd0 con 0x7f8a080ae520 2024-01-27T01:06:51.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.492+0000 7f8a0ec38700 1 -- 172.21.15.46:0/1242123494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a081564a0 con 0x7f8a080ae520 2024-01-27T01:06:51.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.492+0000 7f8a04ff9700 1 -- 172.21.15.46:0/1242123494 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f89fc025ba0 con 0x7f8a080ae520 2024-01-27T01:06:51.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.493+0000 7f8a04ff9700 1 --2- 172.21.15.46:0/1242123494 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f89ec066f30 0x7f89ec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.493+0000 7f8a04ff9700 1 -- 172.21.15.46:0/1242123494 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f89fc086370 con 0x7f8a080ae520 2024-01-27T01:06:51.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.493+0000 7f8a07fff700 1 --2- 172.21.15.46:0/1242123494 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f89ec066f30 0x7f89ec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.493+0000 7f8a07fff700 1 --2- 172.21.15.46:0/1242123494 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f89ec066f30 0x7f89ec0693e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8a08156ea0 tx=0x7f89f4008040 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:51.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.497+0000 7f8a04ff9700 1 -- 172.21.15.46:0/1242123494 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f89fc052f50 con 0x7f8a080ae520 2024-01-27T01:06:51.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.727+0000 7f8a0ec38700 1 -- 172.21.15.46:0/1242123494 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a0804c710 con 0x7f89ec066f30 2024-01-27T01:06:51.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.730+0000 7f8a04ff9700 1 -- 172.21.15.46:0/1242123494 <== mgr.9101 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7f8a0804c710 con 0x7f89ec066f30 2024-01-27T01:06:51.729 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-27T01:06:51.729 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:51.729 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephfs" 2024-01-27T01:06:51.729 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:06:51.729 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:06:51.729 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "backup_fs" 2024-01-27T01:06:51.729 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:06:51.729 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-27T01:06:51.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.732+0000 7f8a0ec38700 1 -- 172.21.15.46:0/1242123494 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f89ec066f30 msgr2=0x7f89ec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.732+0000 7f8a0ec38700 1 --2- 172.21.15.46:0/1242123494 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f89ec066f30 0x7f89ec0693e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8a08156ea0 tx=0x7f89f4008040 comp rx=0 tx=0).stop 2024-01-27T01:06:51.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.732+0000 7f8a0ec38700 1 -- 172.21.15.46:0/1242123494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a080ae520 msgr2=0x7f8a080c5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.732+0000 7f8a0ec38700 1 --2- 172.21.15.46:0/1242123494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a080ae520 0x7f8a080c5690 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009ed0 tx=0x7f89fc017000 comp rx=0 tx=0).stop 2024-01-27T01:06:51.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.732+0000 7f8a0ec38700 1 -- 172.21.15.46:0/1242123494 shutdown_connections 2024-01-27T01:06:51.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.732+0000 7f8a0ec38700 1 --2- 172.21.15.46:0/1242123494 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f89ec066f30 0x7f89ec0693e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.732+0000 7f8a0ec38700 1 --2- 172.21.15.46:0/1242123494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a080ae520 0x7f8a080c5690 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.732+0000 7f8a0ec38700 1 --2- 172.21.15.46:0/1242123494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a08157340 0x7f8a080c76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.732+0000 7f8a0ec38700 1 --2- 172.21.15.46:0/1242123494 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8a080b3fb0 0x7f8a080c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.732+0000 7f8a0ec38700 1 -- 172.21.15.46:0/1242123494 >> 172.21.15.46:0/1242123494 conn(0x7f8a080ce530 msgr2=0x7f8a0806d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:51.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.732+0000 7f8a0ec38700 1 -- 172.21.15.46:0/1242123494 shutdown_connections 2024-01-27T01:06:51.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.732+0000 7f8a0ec38700 1 -- 172.21.15.46:0/1242123494 wait complete. 2024-01-27T01:06:51.745 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-27T01:06:51.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.880+0000 7f97ef84a700 1 Processor -- start 2024-01-27T01:06:51.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.880+0000 7f97ef84a700 1 -- start start 2024-01-27T01:06:51.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.881+0000 7f97ef84a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97e8162580 0x7f97e8067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.881+0000 7f97ef84a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97e806d410 0x7f97e815d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.881+0000 7f97ef84a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e806cf80 con 0x7f97e8067af0 2024-01-27T01:06:51.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.881+0000 7f97ef84a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e806c160 con 0x7f97e806d410 2024-01-27T01:06:51.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.881+0000 7f97ef84a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e806c370 con 0x7f97e8162580 2024-01-27T01:06:51.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.881+0000 7f97edde7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97e806d410 0x7f97e815d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.881+0000 7f97edde7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97e806d410 0x7f97e815d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37244/0 (socket says 172.21.15.46:37244) 2024-01-27T01:06:51.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.881+0000 7f97edde7700 1 -- 172.21.15.46:0/1456181604 learned_addr learned my addr 172.21.15.46:0/1456181604 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:51.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.882+0000 7f97ecde5700 1 --2- 172.21.15.46:0/1456181604 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97e8162580 0x7f97e8067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.882+0000 7f97edde7700 1 -- 172.21.15.46:0/1456181604 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97e8162580 msgr2=0x7f97e8067ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:51.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.882+0000 7f97edde7700 1 --2- 172.21.15.46:0/1456181604 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97e8162580 0x7f97e8067ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.882+0000 7f97edde7700 1 -- 172.21.15.46:0/1456181604 >> v1:172.21.15.106:6789/0 conn(0x7f97e8067af0 legacy=0x7f97e80cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.882+0000 7f97dbfff700 1 -- 172.21.15.46:0/1456181604 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879405188 0 0) 0x7f97e806cf80 con 0x7f97e8067af0 2024-01-27T01:06:51.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.882+0000 7f97edde7700 1 -- 172.21.15.46:0/1456181604 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e806c650 con 0x7f97e806d410 2024-01-27T01:06:51.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.882+0000 7f97edde7700 1 --2- 172.21.15.46:0/1456181604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e806d410 0x7f97e815d760 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f97e0009f20 tx=0x7f97e0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5829b916d5fc7a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:51.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.882+0000 7f97dbfff700 1 -- 172.21.15.46:0/1456181604 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e000e070 con 0x7f97e806d410 2024-01-27T01:06:51.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.882+0000 7f97dbfff700 1 -- 172.21.15.46:0/1456181604 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e000b9d0 con 0x7f97e806d410 2024-01-27T01:06:51.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.882+0000 7f97dbfff700 1 -- 172.21.15.46:0/1456181604 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e00092e0 con 0x7f97e806d410 2024-01-27T01:06:51.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.883+0000 7f97ef84a700 1 -- 172.21.15.46:0/1456181604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e806d410 msgr2=0x7f97e815d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.883+0000 7f97ef84a700 1 --2- 172.21.15.46:0/1456181604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e806d410 0x7f97e815d760 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f97e0009f20 tx=0x7f97e0009840 comp rx=0 tx=0).stop 2024-01-27T01:06:51.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.883+0000 7f97ef84a700 1 -- 172.21.15.46:0/1456181604 shutdown_connections 2024-01-27T01:06:51.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.883+0000 7f97ef84a700 1 --2- 172.21.15.46:0/1456181604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e806d410 0x7f97e815d760 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.883+0000 7f97ef84a700 1 --2- 172.21.15.46:0/1456181604 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97e8162580 0x7f97e8067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.883+0000 7f97ef84a700 1 -- 172.21.15.46:0/1456181604 >> 172.21.15.46:0/1456181604 conn(0x7f97e80ca9a0 msgr2=0x7f97e8068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:51.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.883+0000 7f97ef84a700 1 -- 172.21.15.46:0/1456181604 shutdown_connections 2024-01-27T01:06:51.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.883+0000 7f97ef84a700 1 -- 172.21.15.46:0/1456181604 wait complete. 2024-01-27T01:06:51.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.884+0000 7f97ef84a700 1 Processor -- start 2024-01-27T01:06:51.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.884+0000 7f97ef84a700 1 -- start start 2024-01-27T01:06:51.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.885+0000 7f97ef84a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97e8067af0 0x7f97e81496d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.885+0000 7f97ef84a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97e806d410 0x7f97e80b2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.885+0000 7f97ef84a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e8162580 0x7f97e80b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.885+0000 7f97ef84a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e80c5120 con 0x7f97e8162580 2024-01-27T01:06:51.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.885+0000 7f97ef84a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e8149c10 con 0x7f97e8067af0 2024-01-27T01:06:51.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.885+0000 7f97ef84a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e80c5310 con 0x7f97e806d410 2024-01-27T01:06:51.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.885+0000 7f97edde7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e8162580 0x7f97e80b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.885+0000 7f97ecde5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97e806d410 0x7f97e80b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.885+0000 7f97edde7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e8162580 0x7f97e80b28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37260/0 (socket says 172.21.15.46:37260) 2024-01-27T01:06:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.885+0000 7f97edde7700 1 -- 172.21.15.46:0/3861356683 learned_addr learned my addr 172.21.15.46:0/3861356683 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.885+0000 7f97ed5e6700 1 --2- 172.21.15.46:0/3861356683 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97e8067af0 0x7f97e81496d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.886+0000 7f97edde7700 1 -- 172.21.15.46:0/3861356683 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97e806d410 msgr2=0x7f97e80b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.886+0000 7f97edde7700 1 --2- 172.21.15.46:0/3861356683 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97e806d410 0x7f97e80b2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.886+0000 7f97edde7700 1 -- 172.21.15.46:0/3861356683 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97e8067af0 msgr2=0x7f97e81496d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.886+0000 7f97edde7700 1 --2- 172.21.15.46:0/3861356683 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97e8067af0 0x7f97e81496d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.886+0000 7f97edde7700 1 -- 172.21.15.46:0/3861356683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e0009ae0 con 0x7f97e8162580 2024-01-27T01:06:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.886+0000 7f97ecde5700 1 --2- 172.21.15.46:0/3861356683 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97e806d410 0x7f97e80b2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.886+0000 7f97edde7700 1 --2- 172.21.15.46:0/3861356683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e8162580 0x7f97e80b28a0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f97e0012000 tx=0x7f97e0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.886+0000 7f97d97fa700 1 -- 172.21.15.46:0/3861356683 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e0008b10 con 0x7f97e8162580 2024-01-27T01:06:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.886+0000 7f97d97fa700 1 -- 172.21.15.46:0/3861356683 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e0008c70 con 0x7f97e8162580 2024-01-27T01:06:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.886+0000 7f97ef84a700 1 -- 172.21.15.46:0/3861356683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e80c55f0 con 0x7f97e8162580 2024-01-27T01:06:51.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.886+0000 7f97d97fa700 1 -- 172.21.15.46:0/3861356683 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e0026460 con 0x7f97e8162580 2024-01-27T01:06:51.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.886+0000 7f97ef84a700 1 -- 172.21.15.46:0/3861356683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e80c5bb0 con 0x7f97e8162580 2024-01-27T01:06:51.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.887+0000 7f97ef84a700 1 -- 172.21.15.46:0/3861356683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e804c710 con 0x7f97e8162580 2024-01-27T01:06:51.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.888+0000 7f97d97fa700 1 -- 172.21.15.46:0/3861356683 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 126) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f97e0003040 con 0x7f97e8162580 2024-01-27T01:06:51.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.888+0000 7f97d97fa700 1 --2- 172.21.15.46:0/3861356683 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f97cc066f30 0x7f97cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:51.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.888+0000 7f97d97fa700 1 -- 172.21.15.46:0/3861356683 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f97e0086b90 con 0x7f97e8162580 2024-01-27T01:06:51.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.888+0000 7f97ed5e6700 1 --2- 172.21.15.46:0/3861356683 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f97cc066f30 0x7f97cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:51.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.889+0000 7f97ed5e6700 1 --2- 172.21.15.46:0/3861356683 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f97cc066f30 0x7f97cc0693e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f97e815c670 tx=0x7f97dc009040 comp rx=0 tx=0).ready entity=mgr.9101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:51.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:51.893+0000 7f97d97fa700 1 -- 172.21.15.46:0/3861356683 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f97e0053820 con 0x7f97e8162580 2024-01-27T01:06:52.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.157+0000 7f97ef84a700 1 -- 172.21.15.46:0/3861356683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f97e806c930 con 0x7f97e8162580 2024-01-27T01:06:52.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.805+0000 7f97d97fa700 1 -- 172.21.15.46:0/3861356683 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 127) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f97e0053bf0 con 0x7f97e8162580 2024-01-27T01:06:52.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.807+0000 7f97d97fa700 1 -- 172.21.15.46:0/3861356683 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v127) v1 ==== 92+0+0 (secure 0 0 0) 0x7f97e006da50 con 0x7f97e8162580 2024-01-27T01:06:52.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.810+0000 7f97ef84a700 1 -- 172.21.15.46:0/3861356683 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f97cc066f30 msgr2=0x7f97cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:52.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.810+0000 7f97ef84a700 1 --2- 172.21.15.46:0/3861356683 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f97cc066f30 0x7f97cc0693e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f97e815c670 tx=0x7f97dc009040 comp rx=0 tx=0).stop 2024-01-27T01:06:52.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.810+0000 7f97ef84a700 1 -- 172.21.15.46:0/3861356683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e8162580 msgr2=0x7f97e80b28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:52.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.810+0000 7f97ef84a700 1 --2- 172.21.15.46:0/3861356683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e8162580 0x7f97e80b28a0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f97e0012000 tx=0x7f97e0018000 comp rx=0 tx=0).stop 2024-01-27T01:06:52.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.810+0000 7f97ef84a700 1 -- 172.21.15.46:0/3861356683 shutdown_connections 2024-01-27T01:06:52.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.810+0000 7f97ef84a700 1 --2- 172.21.15.46:0/3861356683 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f97cc066f30 0x7f97cc0693e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:52.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.810+0000 7f97ef84a700 1 --2- 172.21.15.46:0/3861356683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e8162580 0x7f97e80b28a0 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:52.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.810+0000 7f97ef84a700 1 --2- 172.21.15.46:0/3861356683 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97e8067af0 0x7f97e81496d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:52.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.810+0000 7f97ef84a700 1 --2- 172.21.15.46:0/3861356683 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f97e806d410 0x7f97e80b2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:52.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.810+0000 7f97ef84a700 1 -- 172.21.15.46:0/3861356683 >> 172.21.15.46:0/3861356683 conn(0x7f97e80ca9a0 msgr2=0x7f97e80c2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:52.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.810+0000 7f97ef84a700 1 -- 172.21.15.46:0/3861356683 shutdown_connections 2024-01-27T01:06:52.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.810+0000 7f97ef84a700 1 -- 172.21.15.46:0/3861356683 wait complete. 2024-01-27T01:06:52.829 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-27T01:06:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.971+0000 7f944a889700 1 Processor -- start 2024-01-27T01:06:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.971+0000 7f944a889700 1 -- start start 2024-01-27T01:06:52.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.972+0000 7f944a889700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9444157840 0x7f94440ca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:52.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.972+0000 7f944a889700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9444068040 0x7f94440b0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:52.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.972+0000 7f944a889700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94440b02d0 con 0x7f944406c110 2024-01-27T01:06:52.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.972+0000 7f944a889700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f944406c680 con 0x7f9444068040 2024-01-27T01:06:52.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.972+0000 7f944a889700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f944406c960 con 0x7f9444157840 2024-01-27T01:06:52.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.972+0000 7f9448e26700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9444068040 0x7f94440b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:52.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.972+0000 7f9448e26700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9444068040 0x7f94440b0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37280/0 (socket says 172.21.15.46:37280) 2024-01-27T01:06:52.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.972+0000 7f9448e26700 1 -- 172.21.15.46:0/1437987164 learned_addr learned my addr 172.21.15.46:0/1437987164 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:52.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.972+0000 7f9443fff700 1 --2- 172.21.15.46:0/1437987164 >> v2:172.21.15.106:3300/0 conn(0x7f9444157840 0x7f94440ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:52.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.972+0000 7f9448e26700 1 -- 172.21.15.46:0/1437987164 >> v2:172.21.15.106:3300/0 conn(0x7f9444157840 msgr2=0x7f94440ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:52.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.972+0000 7f9448e26700 1 --2- 172.21.15.46:0/1437987164 >> v2:172.21.15.106:3300/0 conn(0x7f9444157840 0x7f94440ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:52.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.972+0000 7f9448e26700 1 -- 172.21.15.46:0/1437987164 >> v1:172.21.15.106:6789/0 conn(0x7f944406c110 legacy=0x7f9444067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:52.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.972+0000 7f9448e26700 1 -- 172.21.15.46:0/1437987164 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944406cc40 con 0x7f9444068040 2024-01-27T01:06:52.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.972+0000 7f9448e26700 1 --2- 172.21.15.46:0/1437987164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9444068040 0x7f94440b0760 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f944406e2c0 tx=0x7f943c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec6263892d63a535 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:52.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.973+0000 7f9442ffd700 1 -- 172.21.15.46:0/1437987164 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943c00e070 con 0x7f9444068040 2024-01-27T01:06:52.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.973+0000 7f9442ffd700 1 -- 172.21.15.46:0/1437987164 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f943c00ba20 con 0x7f9444068040 2024-01-27T01:06:52.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.973+0000 7f9442ffd700 1 -- 172.21.15.46:0/1437987164 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943c0092e0 con 0x7f9444068040 2024-01-27T01:06:52.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.973+0000 7f944a889700 1 -- 172.21.15.46:0/1437987164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9444068040 msgr2=0x7f94440b0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:52.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.973+0000 7f944a889700 1 --2- 172.21.15.46:0/1437987164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9444068040 0x7f94440b0760 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f944406e2c0 tx=0x7f943c009840 comp rx=0 tx=0).stop 2024-01-27T01:06:52.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.973+0000 7f944a889700 1 -- 172.21.15.46:0/1437987164 shutdown_connections 2024-01-27T01:06:52.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.973+0000 7f944a889700 1 --2- 172.21.15.46:0/1437987164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9444068040 0x7f94440b0760 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:52.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.973+0000 7f944a889700 1 --2- 172.21.15.46:0/1437987164 >> v2:172.21.15.106:3300/0 conn(0x7f9444157840 0x7f94440ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:52.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.973+0000 7f944a889700 1 -- 172.21.15.46:0/1437987164 >> 172.21.15.46:0/1437987164 conn(0x7f94440cdf50 msgr2=0x7f94440c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:52.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.973+0000 7f944a889700 1 -- 172.21.15.46:0/1437987164 shutdown_connections 2024-01-27T01:06:52.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.973+0000 7f944a889700 1 -- 172.21.15.46:0/1437987164 wait complete. 2024-01-27T01:06:52.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.975+0000 7f944a889700 1 Processor -- start 2024-01-27T01:06:52.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.975+0000 7f944a889700 1 -- start start 2024-01-27T01:06:52.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.976+0000 7f944a889700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9444068040 0x7f94440b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:52.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.976+0000 7f944a889700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f944406c110 0x7f9444146140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:52.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.976+0000 7f944a889700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9444157840 0x7f94440b1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:52.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.976+0000 7f944a889700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94440b2df0 con 0x7f944406c110 2024-01-27T01:06:52.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.976+0000 7f944a889700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944415cf70 con 0x7f9444157840 2024-01-27T01:06:52.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.976+0000 7f944a889700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f944415c360 con 0x7f9444068040 2024-01-27T01:06:52.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.977+0000 7f9448e26700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9444157840 0x7f94440b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:52.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.976+0000 7f94437fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f944406c110 0x7f9444146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:52.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.977+0000 7f9443fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9444068040 0x7f94440b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:52.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.977+0000 7f9448e26700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9444157840 0x7f94440b1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:54022/0 (socket says 172.21.15.46:54022) 2024-01-27T01:06:52.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.977+0000 7f94437fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f944406c110 0x7f9444146140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37294/0 (socket says 172.21.15.46:37294) 2024-01-27T01:06:52.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.977+0000 7f9443fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9444068040 0x7f94440b28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:52954/0 (socket says 172.21.15.46:52954) 2024-01-27T01:06:52.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.977+0000 7f9448e26700 1 -- 172.21.15.46:0/3984293933 learned_addr learned my addr 172.21.15.46:0/3984293933 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:52.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.977+0000 7f9448e26700 1 -- 172.21.15.46:0/3984293933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9444068040 msgr2=0x7f94440b28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:52.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.977+0000 7f9448e26700 1 --2- 172.21.15.46:0/3984293933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9444068040 0x7f94440b28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:52.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.977+0000 7f9448e26700 1 -- 172.21.15.46:0/3984293933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f944406c110 msgr2=0x7f9444146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:52.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.977+0000 7f9448e26700 1 --2- 172.21.15.46:0/3984293933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f944406c110 0x7f9444146140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:52.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.977+0000 7f9448e26700 1 -- 172.21.15.46:0/3984293933 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943c009ae0 con 0x7f9444157840 2024-01-27T01:06:52.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.977+0000 7f9448e26700 1 --2- 172.21.15.46:0/3984293933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9444157840 0x7f94440b1580 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f943c009880 tx=0x7f943c018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:52.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.977+0000 7f942bfff700 1 -- 172.21.15.46:0/3984293933 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943c008a80 con 0x7f9444157840 2024-01-27T01:06:52.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.977+0000 7f942bfff700 1 -- 172.21.15.46:0/3984293933 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f943c008be0 con 0x7f9444157840 2024-01-27T01:06:52.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.977+0000 7f942bfff700 1 -- 172.21.15.46:0/3984293933 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943c026460 con 0x7f9444157840 2024-01-27T01:06:52.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.977+0000 7f944a889700 1 -- 172.21.15.46:0/3984293933 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f944415c640 con 0x7f9444157840 2024-01-27T01:06:52.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.977+0000 7f944a889700 1 -- 172.21.15.46:0/3984293933 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f944415cc00 con 0x7f9444157840 2024-01-27T01:06:52.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.978+0000 7f944a889700 1 -- 172.21.15.46:0/3984293933 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f944404c710 con 0x7f9444157840 2024-01-27T01:06:52.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.979+0000 7f942bfff700 1 -- 172.21.15.46:0/3984293933 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 127) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f943c003040 con 0x7f9444157840 2024-01-27T01:06:52.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.979+0000 7f942bfff700 1 --2- 172.21.15.46:0/3984293933 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9430066f50 0x7f9430069400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:52.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.979+0000 7f9443fff700 1 -- 172.21.15.46:0/3984293933 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9430066f50 msgr2=0x7f9430069400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:52.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.979+0000 7f942bfff700 1 -- 172.21.15.46:0/3984293933 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f943c086a00 con 0x7f9444157840 2024-01-27T01:06:52.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.979+0000 7f9443fff700 1 --2- 172.21.15.46:0/3984293933 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9430066f50 0x7f9430069400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:06:52.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:52.984+0000 7f942bfff700 1 -- 172.21.15.46:0/3984293933 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f943c0535e0 con 0x7f9444157840 2024-01-27T01:06:53.140 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:53.141+0000 7f6dd013e1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:06:53.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.180+0000 7f9443fff700 1 -- 172.21.15.46:0/3984293933 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9430066f50 msgr2=0x7f9430069400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:53.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.180+0000 7f9443fff700 1 --2- 172.21.15.46:0/3984293933 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9430066f50 0x7f9430069400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:06:53.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.193+0000 7f944a889700 1 -- 172.21.15.46:0/3984293933 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f944406cf20 con 0x7f9444157840 2024-01-27T01:06:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.197+0000 7f942bfff700 1 -- 172.21.15.46:0/3984293933 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v77)=0 v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7f943c056c00 con 0x7f9444157840 2024-01-27T01:06:53.198 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:53.199+0000 7fddc51001c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:06:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.200+0000 7f944a889700 1 -- 172.21.15.46:0/3984293933 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9430066f50 msgr2=0x7f9430069400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:06:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.200+0000 7f944a889700 1 --2- 172.21.15.46:0/3984293933 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9430066f50 0x7f9430069400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.200+0000 7f944a889700 1 -- 172.21.15.46:0/3984293933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9444157840 msgr2=0x7f94440b1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.200+0000 7f944a889700 1 --2- 172.21.15.46:0/3984293933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9444157840 0x7f94440b1580 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f943c009880 tx=0x7f943c018000 comp rx=0 tx=0).stop 2024-01-27T01:06:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.200+0000 7f944a889700 1 -- 172.21.15.46:0/3984293933 shutdown_connections 2024-01-27T01:06:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.200+0000 7f944a889700 1 --2- 172.21.15.46:0/3984293933 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9430066f50 0x7f9430069400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.200+0000 7f944a889700 1 --2- 172.21.15.46:0/3984293933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9444157840 0x7f94440b1580 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.200+0000 7f944a889700 1 --2- 172.21.15.46:0/3984293933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9444068040 0x7f94440b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.200+0000 7f944a889700 1 --2- 172.21.15.46:0/3984293933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f944406c110 0x7f9444146140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.200+0000 7f944a889700 1 -- 172.21.15.46:0/3984293933 >> 172.21.15.46:0/3984293933 conn(0x7f94440cdf50 msgr2=0x7f94440c3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.200+0000 7f944a889700 1 -- 172.21.15.46:0/3984293933 shutdown_connections 2024-01-27T01:06:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.200+0000 7f944a889700 1 -- 172.21.15.46:0/3984293933 wait complete. 2024-01-27T01:06:53.213 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-27T01:06:53.217 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:53.218+0000 7f6dd013e1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:06:53.288 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:53.288+0000 7f6dd013e1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:06:53.289 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:53.291+0000 7fddc51001c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:06:53.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.352+0000 7f41e103e700 1 Processor -- start 2024-01-27T01:06:53.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.352+0000 7f41e103e700 1 -- start start 2024-01-27T01:06:53.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.352+0000 7f41e103e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41dc067af0 0x7f41dc067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:53.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.353+0000 7f41e103e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f41dc06ebe0 0x7f41dc0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:53.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.353+0000 7f41e103e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41dc06fdb0 con 0x7f41dc1597e0 2024-01-27T01:06:53.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.353+0000 7f41e103e700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41dc0ade10 con 0x7f41dc06ebe0 2024-01-27T01:06:53.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.353+0000 7f41e103e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc06f1a0 con 0x7f41dc067af0 2024-01-27T01:06:53.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.353+0000 7f41dad9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41dc067af0 0x7f41dc067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:53.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.353+0000 7f41da59c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f41dc06ebe0 0x7f41dc0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:53.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.353+0000 7f41db59e700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f41dc1597e0 0x7f41dc159bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:41218/0 (socket says 172.21.15.46:41218) 2024-01-27T01:06:53.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.353+0000 7f41db59e700 1 -- 172.21.15.46:0/3048938586 learned_addr learned my addr 172.21.15.46:0/3048938586 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:53.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.353+0000 7f41dad9d700 1 -- 172.21.15.46:0/3048938586 >> v2:172.21.15.106:3300/0 conn(0x7f41dc06ebe0 msgr2=0x7f41dc0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:53.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.353+0000 7f41dad9d700 1 --2- 172.21.15.46:0/3048938586 >> v2:172.21.15.106:3300/0 conn(0x7f41dc06ebe0 0x7f41dc0ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.353+0000 7f41dad9d700 1 -- 172.21.15.46:0/3048938586 >> v1:172.21.15.106:6789/0 conn(0x7f41dc1597e0 legacy=0x7f41dc159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:53.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.353+0000 7f41dad9d700 1 -- 172.21.15.46:0/3048938586 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41dc06f480 con 0x7f41dc067af0 2024-01-27T01:06:53.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.353+0000 7f41da59c700 1 --2- 172.21.15.46:0/3048938586 >> v2:172.21.15.106:3300/0 conn(0x7f41dc06ebe0 0x7f41dc0ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:53.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.353+0000 7f41d9d9b700 1 -- 172.21.15.46:0/3048938586 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265491045 0 0) 0x7f41dc06fdb0 con 0x7f41dc1597e0 2024-01-27T01:06:53.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.353+0000 7f41dad9d700 1 --2- 172.21.15.46:0/3048938586 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41dc067af0 0x7f41dc067ec0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0098e0 tx=0x7f41cc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cab998b992e6c20 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:53.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.354+0000 7f41d9d9b700 1 -- 172.21.15.46:0/3048938586 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc00b640 con 0x7f41dc067af0 2024-01-27T01:06:53.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.354+0000 7f41d9d9b700 1 -- 172.21.15.46:0/3048938586 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc00b7a0 con 0x7f41dc067af0 2024-01-27T01:06:53.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.354+0000 7f41d9d9b700 1 -- 172.21.15.46:0/3048938586 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc00bb40 con 0x7f41dc067af0 2024-01-27T01:06:53.355 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:53.353+0000 7f6dd013e1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:06:53.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.354+0000 7f41e103e700 1 -- 172.21.15.46:0/3048938586 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41dc067af0 msgr2=0x7f41dc067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:53.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.354+0000 7f41e103e700 1 --2- 172.21.15.46:0/3048938586 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41dc067af0 0x7f41dc067ec0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0098e0 tx=0x7f41cc0091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.354+0000 7f41e103e700 1 -- 172.21.15.46:0/3048938586 shutdown_connections 2024-01-27T01:06:53.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.354+0000 7f41e103e700 1 --2- 172.21.15.46:0/3048938586 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41dc067af0 0x7f41dc067ec0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.354+0000 7f41e103e700 1 --2- 172.21.15.46:0/3048938586 >> v2:172.21.15.106:3300/0 conn(0x7f41dc06ebe0 0x7f41dc0ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.354+0000 7f41e103e700 1 -- 172.21.15.46:0/3048938586 >> 172.21.15.46:0/3048938586 conn(0x7f41dc0cdf80 msgr2=0x7f41dc06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:53.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.354+0000 7f41e103e700 1 -- 172.21.15.46:0/3048938586 shutdown_connections 2024-01-27T01:06:53.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.354+0000 7f41e103e700 1 -- 172.21.15.46:0/3048938586 wait complete. 2024-01-27T01:06:53.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.356+0000 7f41e103e700 1 Processor -- start 2024-01-27T01:06:53.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.356+0000 7f41e103e700 1 -- start start 2024-01-27T01:06:53.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.356+0000 7f41e103e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc067af0 0x7f41dc1522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:53.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.356+0000 7f41e103e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41dc06ebe0 0x7f41dc15a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.356+0000 7f41e103e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41dc1597e0 0x7f41dc155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.356+0000 7f41e103e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc16a040 con 0x7f41dc067af0 2024-01-27T01:06:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.356+0000 7f41e103e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc1527e0 con 0x7f41dc06ebe0 2024-01-27T01:06:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.356+0000 7f41e103e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc169cf0 con 0x7f41dc1597e0 2024-01-27T01:06:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.357+0000 7f41dad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc067af0 0x7f41dc1522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.357+0000 7f41dad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc067af0 0x7f41dc1522a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37304/0 (socket says 172.21.15.46:37304) 2024-01-27T01:06:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.357+0000 7f41dad9d700 1 -- 172.21.15.46:0/2921829877 learned_addr learned my addr 172.21.15.46:0/2921829877 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.357+0000 7f41da59c700 1 --2- 172.21.15.46:0/2921829877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41dc06ebe0 0x7f41dc15a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.357+0000 7f41db59e700 1 --2- 172.21.15.46:0/2921829877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41dc1597e0 0x7f41dc155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.357+0000 7f41dad9d700 1 -- 172.21.15.46:0/2921829877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41dc1597e0 msgr2=0x7f41dc155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.357+0000 7f41dad9d700 1 --2- 172.21.15.46:0/2921829877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41dc1597e0 0x7f41dc155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.357+0000 7f41dad9d700 1 -- 172.21.15.46:0/2921829877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41dc06ebe0 msgr2=0x7f41dc15a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.357+0000 7f41dad9d700 1 --2- 172.21.15.46:0/2921829877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41dc06ebe0 0x7f41dc15a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.357+0000 7f41dad9d700 1 -- 172.21.15.46:0/2921829877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d0009420 con 0x7f41dc067af0 2024-01-27T01:06:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.357+0000 7f41dad9d700 1 --2- 172.21.15.46:0/2921829877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc067af0 0x7f41dc1522a0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0093f0 tx=0x7f41cc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.357+0000 7f41c2ffd700 1 -- 172.21.15.46:0/2921829877 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc000bc0 con 0x7f41dc067af0 2024-01-27T01:06:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.357+0000 7f41c2ffd700 1 -- 172.21.15.46:0/2921829877 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc00b910 con 0x7f41dc067af0 2024-01-27T01:06:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.357+0000 7f41e103e700 1 -- 172.21.15.46:0/2921829877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41cc009420 con 0x7f41dc067af0 2024-01-27T01:06:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.358+0000 7f41c2ffd700 1 -- 172.21.15.46:0/2921829877 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc000d20 con 0x7f41dc067af0 2024-01-27T01:06:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.358+0000 7f41e103e700 1 -- 172.21.15.46:0/2921829877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41dc15b1e0 con 0x7f41dc067af0 2024-01-27T01:06:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.359+0000 7f41e103e700 1 -- 172.21.15.46:0/2921829877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41dc04c710 con 0x7f41dc067af0 2024-01-27T01:06:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.359+0000 7f41c2ffd700 1 -- 172.21.15.46:0/2921829877 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 127) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f41cc023070 con 0x7f41dc067af0 2024-01-27T01:06:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.360+0000 7f41c2ffd700 1 --2- 172.21.15.46:0/2921829877 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f41c8066f80 0x7f41c8069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.360+0000 7f41da59c700 1 -- 172.21.15.46:0/2921829877 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f41c8066f80 msgr2=0x7f41c8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.360+0000 7f41c2ffd700 1 -- 172.21.15.46:0/2921829877 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f41cc085d10 con 0x7f41dc067af0 2024-01-27T01:06:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.360+0000 7f41da59c700 1 --2- 172.21.15.46:0/2921829877 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f41c8066f80 0x7f41c8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:06:53.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.364+0000 7f41c2ffd700 1 -- 172.21.15.46:0/2921829877 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f41cc0528f0 con 0x7f41dc067af0 2024-01-27T01:06:53.374 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:53.374+0000 7fddc51001c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:06:53.459 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:53.460+0000 7fddc51001c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:06:53.508 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:53.508+0000 7f6dd013e1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:06:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.561+0000 7f41da59c700 1 -- 172.21.15.46:0/2921829877 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f41c8066f80 msgr2=0x7f41c8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.561+0000 7f41da59c700 1 --2- 172.21.15.46:0/2921829877 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f41c8066f80 0x7f41c8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:06:53.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.583+0000 7f41e103e700 1 -- 172.21.15.46:0/2921829877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f41dc068690 con 0x7f41dc067af0 2024-01-27T01:06:53.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.590+0000 7f41c2ffd700 1 -- 172.21.15.46:0/2921829877 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v78)=0 v78) v1 ==== 133+0+0 (secure 0 0 0) 0x7f41cc055f10 con 0x7f41dc067af0 2024-01-27T01:06:53.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.592+0000 7f41e103e700 1 -- 172.21.15.46:0/2921829877 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f41c8066f80 msgr2=0x7f41c8069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:06:53.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.592+0000 7f41e103e700 1 --2- 172.21.15.46:0/2921829877 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f41c8066f80 0x7f41c8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.592+0000 7f41e103e700 1 -- 172.21.15.46:0/2921829877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc067af0 msgr2=0x7f41dc1522a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:53.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.592+0000 7f41e103e700 1 --2- 172.21.15.46:0/2921829877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc067af0 0x7f41dc1522a0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0093f0 tx=0x7f41cc017000 comp rx=0 tx=0).stop 2024-01-27T01:06:53.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.592+0000 7f41e103e700 1 -- 172.21.15.46:0/2921829877 shutdown_connections 2024-01-27T01:06:53.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.592+0000 7f41e103e700 1 --2- 172.21.15.46:0/2921829877 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f41c8066f80 0x7f41c8069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.592+0000 7f41e103e700 1 --2- 172.21.15.46:0/2921829877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc067af0 0x7f41dc1522a0 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.592+0000 7f41e103e700 1 --2- 172.21.15.46:0/2921829877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41dc1597e0 0x7f41dc155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.592+0000 7f41e103e700 1 --2- 172.21.15.46:0/2921829877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41dc06ebe0 0x7f41dc15a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.592+0000 7f41e103e700 1 -- 172.21.15.46:0/2921829877 >> 172.21.15.46:0/2921829877 conn(0x7f41dc0cdf80 msgr2=0x7f41dc0b2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:53.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.592+0000 7f41e103e700 1 -- 172.21.15.46:0/2921829877 shutdown_connections 2024-01-27T01:06:53.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.593+0000 7f41e103e700 1 -- 172.21.15.46:0/2921829877 wait complete. 2024-01-27T01:06:53.606 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:06:53.606 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-01-27T01:06:53.631 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-01-27T01:06:53.639 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:53.640+0000 7fddc51001c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:06:53.645 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:53.645+0000 7f6dd013e1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:06:53.711 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:53.711+0000 7f6dd013e1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:06:53.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.779+0000 7f0a2a272700 1 Processor -- start 2024-01-27T01:06:53.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.779+0000 7f0a2a272700 1 -- start start 2024-01-27T01:06:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.780+0000 7f0a2a272700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0a24067930 0x7f0a24067d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.780+0000 7f0a2a272700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2415ab40 con 0x7f0a24159630 2024-01-27T01:06:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.780+0000 7f0a2a272700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a24068270 con 0x7f0a240af570 2024-01-27T01:06:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.780+0000 7f0a2a272700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a2415a210 con 0x7f0a24067930 2024-01-27T01:06:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.780+0000 7f0a23fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0a24159630 0x7f0a24159a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56012/0 (socket says 172.21.15.46:56012) 2024-01-27T01:06:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.780+0000 7f0a23fff700 1 -- 172.21.15.46:0/36802558 learned_addr learned my addr 172.21.15.46:0/36802558 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.780+0000 7f0a2880f700 1 --2- 172.21.15.46:0/36802558 >> v2:172.21.15.106:3300/0 conn(0x7f0a24067930 0x7f0a24067d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.781+0000 7f0a22ffd700 1 -- 172.21.15.46:0/36802558 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851333024 0 0) 0x7f0a24068270 con 0x7f0a240af570 2024-01-27T01:06:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.781+0000 7f0a22ffd700 1 -- 172.21.15.46:0/36802558 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a0c003570 con 0x7f0a240af570 2024-01-27T01:06:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.781+0000 7f0a22ffd700 1 -- 172.21.15.46:0/36802558 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524025161 0 0) 0x7f0a2415ab40 con 0x7f0a24159630 2024-01-27T01:06:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.781+0000 7f0a22ffd700 1 -- 172.21.15.46:0/36802558 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a24068270 con 0x7f0a24159630 2024-01-27T01:06:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.781+0000 7f0a2880f700 1 -- 172.21.15.46:0/36802558 >> v1:172.21.15.106:6789/0 conn(0x7f0a240af570 legacy=0x7f0a240b2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.781+0000 7f0a2880f700 1 -- 172.21.15.46:0/36802558 >> v1:172.21.15.46:6789/0 conn(0x7f0a24159630 legacy=0x7f0a24159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.781+0000 7f0a2880f700 1 -- 172.21.15.46:0/36802558 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2415a4f0 con 0x7f0a24067930 2024-01-27T01:06:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.781+0000 7f0a2880f700 1 --2- 172.21.15.46:0/36802558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a24067930 0x7f0a24067d30 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009e60 tx=0x7f0a1c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30b86473f0b51ace server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.781+0000 7f0a22ffd700 1 -- 172.21.15.46:0/36802558 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c00e070 con 0x7f0a24067930 2024-01-27T01:06:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.781+0000 7f0a22ffd700 1 -- 172.21.15.46:0/36802558 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1c00bab0 con 0x7f0a24067930 2024-01-27T01:06:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.781+0000 7f0a22ffd700 1 -- 172.21.15.46:0/36802558 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c00be70 con 0x7f0a24067930 2024-01-27T01:06:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.782+0000 7f0a2a272700 1 -- 172.21.15.46:0/36802558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a24067930 msgr2=0x7f0a24067d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.782+0000 7f0a2a272700 1 --2- 172.21.15.46:0/36802558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a24067930 0x7f0a24067d30 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009e60 tx=0x7f0a1c009780 comp rx=0 tx=0).stop 2024-01-27T01:06:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.782+0000 7f0a2a272700 1 -- 172.21.15.46:0/36802558 shutdown_connections 2024-01-27T01:06:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.782+0000 7f0a2a272700 1 --2- 172.21.15.46:0/36802558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a24067930 0x7f0a24067d30 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.782+0000 7f0a2a272700 1 -- 172.21.15.46:0/36802558 >> 172.21.15.46:0/36802558 conn(0x7f0a24069510 msgr2=0x7f0a240aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.782+0000 7f0a2a272700 1 -- 172.21.15.46:0/36802558 shutdown_connections 2024-01-27T01:06:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.782+0000 7f0a2a272700 1 -- 172.21.15.46:0/36802558 wait complete. 2024-01-27T01:06:53.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.783+0000 7f0a2a272700 1 Processor -- start 2024-01-27T01:06:53.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.783+0000 7f0a2a272700 1 -- start start 2024-01-27T01:06:53.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.784+0000 7f0a2a272700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a24067930 0x7f0a2415d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:53.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.784+0000 7f0a2a272700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a240af570 0x7f0a240adec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.784+0000 7f0a2a272700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a24159630 0x7f0a240ae400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.784+0000 7f0a2a272700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2416a430 con 0x7f0a24159630 2024-01-27T01:06:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.784+0000 7f0a2a272700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2415dc30 con 0x7f0a24067930 2024-01-27T01:06:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.784+0000 7f0a2a272700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a24169de0 con 0x7f0a240af570 2024-01-27T01:06:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.784+0000 7f0a2880f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a24159630 0x7f0a240ae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.784+0000 7f0a2880f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a24159630 0x7f0a240ae400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37320/0 (socket says 172.21.15.46:37320) 2024-01-27T01:06:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.784+0000 7f0a2880f700 1 -- 172.21.15.46:0/4068930684 learned_addr learned my addr 172.21.15.46:0/4068930684 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.784+0000 7f0a23fff700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a24067930 0x7f0a2415d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.784+0000 7f0a237fe700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a240af570 0x7f0a240adec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.784+0000 7f0a2880f700 1 -- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a240af570 msgr2=0x7f0a240adec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.784+0000 7f0a2880f700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a240af570 0x7f0a240adec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.784+0000 7f0a2880f700 1 -- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a24067930 msgr2=0x7f0a2415d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.784+0000 7f0a2880f700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a24067930 0x7f0a2415d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.784+0000 7f0a2880f700 1 -- 172.21.15.46:0/4068930684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1c009a20 con 0x7f0a24159630 2024-01-27T01:06:53.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.784+0000 7f0a23fff700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a24067930 0x7f0a2415d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:53.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.785+0000 7f0a2880f700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a24159630 0x7f0a240ae400 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f0a240ce4b0 tx=0x7f0a1c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:53.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.785+0000 7f0a0bfff700 1 -- 172.21.15.46:0/4068930684 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c012db0 con 0x7f0a24159630 2024-01-27T01:06:53.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.785+0000 7f0a0bfff700 1 -- 172.21.15.46:0/4068930684 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1c026480 con 0x7f0a24159630 2024-01-27T01:06:53.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.785+0000 7f0a2a272700 1 -- 172.21.15.46:0/4068930684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a2416a0c0 con 0x7f0a24159630 2024-01-27T01:06:53.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.785+0000 7f0a0bfff700 1 -- 172.21.15.46:0/4068930684 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c026840 con 0x7f0a24159630 2024-01-27T01:06:53.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.785+0000 7f0a2a272700 1 -- 172.21.15.46:0/4068930684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a2415d790 con 0x7f0a24159630 2024-01-27T01:06:53.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.786+0000 7f0a2a272700 1 -- 172.21.15.46:0/4068930684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2416a5e0 con 0x7f0a24159630 2024-01-27T01:06:53.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.787+0000 7f0a0bfff700 1 -- 172.21.15.46:0/4068930684 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 127) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f0a1c01c400 con 0x7f0a24159630 2024-01-27T01:06:53.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.787+0000 7f0a0bfff700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 0x7f0a10069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:53.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.787+0000 7f0a0bfff700 1 -- 172.21.15.46:0/4068930684 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f0a1c086a10 con 0x7f0a24159630 2024-01-27T01:06:53.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.787+0000 7f0a23fff700 1 -- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 msgr2=0x7f0a10069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:53.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.787+0000 7f0a23fff700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 0x7f0a10069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:06:53.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.791+0000 7f0a0bfff700 1 -- 172.21.15.46:0/4068930684 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a1c0535f0 con 0x7f0a24159630 2024-01-27T01:06:53.801 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:53.802+0000 7fddc51001c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:06:53.836 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:53.834+0000 7f6dd013e1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:06:53.878 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:53.879+0000 7fddc51001c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:06:53.908 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:53.908+0000 7f6dd013e1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:06:53.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.988+0000 7f0a23fff700 1 -- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 msgr2=0x7f0a10069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:53.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:53.988+0000 7f0a23fff700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 0x7f0a10069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:06:54.018 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:54.019+0000 7fddc51001c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:06:54.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:54.026+0000 7f0a2a272700 1 -- 172.21.15.46:0/4068930684 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a2415d9b0 con 0x7f0a10066f80 2024-01-27T01:06:54.048 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:54.048+0000 7f6dd013e1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:06:54.101 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:54.102+0000 7fddc51001c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:06:54.174 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:54.174+0000 7f6dd013e1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:06:54.277 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:54.278+0000 7fddc51001c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:06:54.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:54.389+0000 7f0a23fff700 1 -- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 msgr2=0x7f0a10069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:54.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:54.389+0000 7f0a23fff700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 0x7f0a10069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-27T01:06:54.419 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:54.420+0000 7fddc51001c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:06:54.487 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:54.488+0000 7f6dd013e1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:06:54.697 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:54.698+0000 7f6dd013e1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:06:54.780 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:54.781+0000 7fddc51001c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:06:54.924 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:54.925+0000 7f6dd013e1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:06:54.995 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:54.995+0000 7f6dd013e1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:06:55.019 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:55.020+0000 7fddc51001c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:06:55.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:55.190+0000 7f0a23fff700 1 -- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 msgr2=0x7f0a10069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:55.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:55.190+0000 7f0a23fff700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 0x7f0a10069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-27T01:06:55.280 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:55.281+0000 7fddc51001c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:06:55.361 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:55.362+0000 7fddc51001c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:06:55.594 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:55.595+0000 7f6dd013e1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:06:55.668 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:55.669+0000 7f6dd013e1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:06:55.790 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:55.790+0000 7f6dd013e1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:06:56.056 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:56.057+0000 7fddc51001c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:06:56.141 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:56.142+0000 7fddc51001c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:06:56.282 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:56.283+0000 7fddc51001c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:06:56.293 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:56.293+0000 7f6dd013e1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:06:56.441 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:56.442+0000 7f6dd013e1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:06:56.591 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:56.591+0000 7f6dd013e1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:06:56.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:56.791+0000 7f0a23fff700 1 -- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 msgr2=0x7f0a10069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:06:56.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:56.791+0000 7f0a23fff700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 0x7f0a10069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-27T01:06:56.867 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:56.868+0000 7fddc51001c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:06:57.040 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:57.041+0000 7fddc51001c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:06:57.213 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:57.214+0000 7fddc51001c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:06:57.291 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.292+0000 7f6dd013e1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:06:57.511 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.512+0000 7f6dd013e1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:06:57.585 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.586+0000 7f6dd013e1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:06:57.742 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.742+0000 7f6dd013e1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:06:57.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:57.766+0000 7f0a0bfff700 1 -- 172.21.15.46:0/4068930684 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 128) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f0a1c0539c0 con 0x7f0a24159630 2024-01-27T01:06:57.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:57.767+0000 7f0a0bfff700 1 -- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 msgr2=0x7f0a10069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:06:57.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:57.767+0000 7f0a0bfff700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 0x7f0a10069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:57.904 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.904+0000 7f6d4621b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.904 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.904+0000 7f6d4621b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.904 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.904+0000 7f6d4621b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.904 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.904+0000 7f6d4621b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.904 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.904+0000 7f6d4621b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.904 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.904+0000 7f6d49221700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.904 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.904+0000 7f6d49221700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.904 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.904+0000 7f6d49221700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.904 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.904+0000 7f6d49221700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.904 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.904+0000 7f6d49221700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.914 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.914+0000 7f6d4621b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.915 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.915+0000 7f6d4621b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.915 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.915+0000 7f6d4621b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.915 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.915+0000 7f6d4621b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.915 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.915+0000 7f6d4621b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.915 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.915+0000 7f6d49221700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.915 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.915+0000 7f6d49221700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.915 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.915+0000 7f6d49221700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.915 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.915+0000 7f6d49221700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:57.915 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:06:57.915+0000 7f6d49221700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:06:58.045 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:58.046+0000 7fddc51001c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:06:58.296 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:58.297+0000 7fddc51001c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:06:58.381 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:58.382+0000 7fddc51001c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:06:58.559 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:06:58.560+0000 7fddc51001c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:06:58.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.772+0000 7f0a0bfff700 1 -- 172.21.15.46:0/4068930684 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 129) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f0a1c01a380 con 0x7f0a24159630 2024-01-27T01:06:58.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.772+0000 7f0a0bfff700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 0x7f0a100693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:58.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.772+0000 7f0a0bfff700 1 -- 172.21.15.46:0/4068930684 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a2415d9b0 con 0x7f0a10066f80 2024-01-27T01:06:58.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.773+0000 7f0a23fff700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 0x7f0a100693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:58.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.778+0000 7f0a23fff700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 0x7f0a100693b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0a14007fd0 tx=0x7f0a14009040 comp rx=0 tx=0).ready entity=mgr.9485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:58.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.833+0000 7f0a0bfff700 1 -- 172.21.15.46:0/4068930684 <== mgr.9485 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+53 (secure 0 0 0) 0x7f0a2415d9b0 con 0x7f0a10066f80 2024-01-27T01:06:58.833 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path mandatory_fs_argument_test_dir 2024-01-27T01:06:58.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.836+0000 7f0a2a272700 1 -- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 msgr2=0x7f0a100693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:58.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.836+0000 7f0a2a272700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 0x7f0a100693b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0a14007fd0 tx=0x7f0a14009040 comp rx=0 tx=0).stop 2024-01-27T01:06:58.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.836+0000 7f0a2a272700 1 -- 172.21.15.46:0/4068930684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a24159630 msgr2=0x7f0a240ae400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:58.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.836+0000 7f0a2a272700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a24159630 0x7f0a240ae400 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f0a240ce4b0 tx=0x7f0a1c019000 comp rx=0 tx=0).stop 2024-01-27T01:06:58.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.836+0000 7f0a2a272700 1 -- 172.21.15.46:0/4068930684 shutdown_connections 2024-01-27T01:06:58.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.836+0000 7f0a2a272700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10066f80 0x7f0a100693b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:58.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.836+0000 7f0a2a272700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a24159630 0x7f0a240ae400 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:58.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.836+0000 7f0a2a272700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a24067930 0x7f0a2415d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:58.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.836+0000 7f0a2a272700 1 --2- 172.21.15.46:0/4068930684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a240af570 0x7f0a240adec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:58.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.836+0000 7f0a2a272700 1 -- 172.21.15.46:0/4068930684 >> 172.21.15.46:0/4068930684 conn(0x7f0a24069510 msgr2=0x7f0a240ac0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:58.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.837+0000 7f0a2a272700 1 -- 172.21.15.46:0/4068930684 shutdown_connections 2024-01-27T01:06:58.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.837+0000 7f0a2a272700 1 -- 172.21.15.46:0/4068930684 wait complete. 2024-01-27T01:06:58.851 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path mandatory_fs_argument_test_dir 2024-01-27T01:06:58.852 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-01-27T01:06:58.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.995+0000 7f2f6e130700 1 Processor -- start 2024-01-27T01:06:58.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.995+0000 7f2f6e130700 1 -- start start 2024-01-27T01:06:58.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.996+0000 7f2f6e130700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2f680ca0e0 0x7f2f680ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:58.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.996+0000 7f2f6e130700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2f6806c0b0 0x7f2f680afe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:58.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.996+0000 7f2f6e130700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f68158010 con 0x7f2f6806c520 2024-01-27T01:06:58.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.996+0000 7f2f6e130700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f680cada0 con 0x7f2f6806c0b0 2024-01-27T01:06:58.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.996+0000 7f2f6e130700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f681582f0 con 0x7f2f680ca0e0 2024-01-27T01:06:58.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.996+0000 7f2f66ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2f6806c0b0 0x7f2f680afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:58.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.996+0000 7f2f66ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2f6806c0b0 0x7f2f680afe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37332/0 (socket says 172.21.15.46:37332) 2024-01-27T01:06:58.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.996+0000 7f2f66ffd700 1 -- 172.21.15.46:0/3535684696 learned_addr learned my addr 172.21.15.46:0/3535684696 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:58.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.997+0000 7f2f677fe700 1 --2- 172.21.15.46:0/3535684696 >> v2:172.21.15.106:3300/0 conn(0x7f2f680ca0e0 0x7f2f680ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:58.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.997+0000 7f2f66ffd700 1 -- 172.21.15.46:0/3535684696 >> v2:172.21.15.106:3300/0 conn(0x7f2f680ca0e0 msgr2=0x7f2f680ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:58.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.997+0000 7f2f66ffd700 1 --2- 172.21.15.46:0/3535684696 >> v2:172.21.15.106:3300/0 conn(0x7f2f680ca0e0 0x7f2f680ca4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:58.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.997+0000 7f2f66ffd700 1 -- 172.21.15.46:0/3535684696 >> v1:172.21.15.106:6789/0 conn(0x7f2f6806c520 legacy=0x7f2f680b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:58.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.997+0000 7f2f66ffd700 1 -- 172.21.15.46:0/3535684696 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f681585d0 con 0x7f2f6806c0b0 2024-01-27T01:06:58.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.997+0000 7f2f66ffd700 1 --2- 172.21.15.46:0/3535684696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f6806c0b0 0x7f2f680afe40 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f2f6806b850 tx=0x7f2f5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=706e752f85e6e9cd server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:58.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.997+0000 7f2f667fc700 1 -- 172.21.15.46:0/3535684696 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c009e70 con 0x7f2f6806c0b0 2024-01-27T01:06:58.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.997+0000 7f2f667fc700 1 -- 172.21.15.46:0/3535684696 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5c00b860 con 0x7f2f6806c0b0 2024-01-27T01:06:58.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.997+0000 7f2f667fc700 1 -- 172.21.15.46:0/3535684696 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c00bc00 con 0x7f2f6806c0b0 2024-01-27T01:06:58.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.998+0000 7f2f6e130700 1 -- 172.21.15.46:0/3535684696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f6806c0b0 msgr2=0x7f2f680afe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:58.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.998+0000 7f2f6e130700 1 --2- 172.21.15.46:0/3535684696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f6806c0b0 0x7f2f680afe40 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f2f6806b850 tx=0x7f2f5c009250 comp rx=0 tx=0).stop 2024-01-27T01:06:59.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.998+0000 7f2f6e130700 1 -- 172.21.15.46:0/3535684696 shutdown_connections 2024-01-27T01:06:59.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.998+0000 7f2f6e130700 1 --2- 172.21.15.46:0/3535684696 >> v2:172.21.15.106:3300/0 conn(0x7f2f680ca0e0 0x7f2f680ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.998+0000 7f2f6e130700 1 --2- 172.21.15.46:0/3535684696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f6806c0b0 0x7f2f680afe40 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.998+0000 7f2f6e130700 1 -- 172.21.15.46:0/3535684696 >> 172.21.15.46:0/3535684696 conn(0x7f2f680ce530 msgr2=0x7f2f680ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:59.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.998+0000 7f2f6e130700 1 -- 172.21.15.46:0/3535684696 shutdown_connections 2024-01-27T01:06:59.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.998+0000 7f2f6e130700 1 -- 172.21.15.46:0/3535684696 wait complete. 2024-01-27T01:06:59.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.999+0000 7f2f6e130700 1 Processor -- start 2024-01-27T01:06:59.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:58.999+0000 7f2f6e130700 1 -- start start 2024-01-27T01:06:59.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.000+0000 7f2f6e130700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f6806c0b0 0x7f2f680c4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:59.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.000+0000 7f2f6e130700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f6806c520 0x7f2f681565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:59.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.000+0000 7f2f6e130700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2f680ca0e0 0x7f2f68156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:59.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.000+0000 7f2f6e130700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f680c8740 con 0x7f2f6806c0b0 2024-01-27T01:06:59.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.000+0000 7f2f6e130700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f68169980 con 0x7f2f6806c520 2024-01-27T01:06:59.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.000+0000 7f2f6e130700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f680c52a0 con 0x7f2f680ca0e0 2024-01-27T01:06:59.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.000+0000 7f2f677fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f6806c0b0 0x7f2f680c4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:59.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.000+0000 7f2f677fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f6806c0b0 0x7f2f680c4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37344/0 (socket says 172.21.15.46:37344) 2024-01-27T01:06:59.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.000+0000 7f2f677fe700 1 -- 172.21.15.46:0/3461509259 learned_addr learned my addr 172.21.15.46:0/3461509259 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:59.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.000+0000 7f2f66ffd700 1 --2- 172.21.15.46:0/3461509259 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f6806c520 0x7f2f681565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:59.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.000+0000 7f2f67fff700 1 --2- 172.21.15.46:0/3461509259 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2f680ca0e0 0x7f2f68156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:59.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.001+0000 7f2f677fe700 1 -- 172.21.15.46:0/3461509259 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2f680ca0e0 msgr2=0x7f2f68156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:59.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.001+0000 7f2f677fe700 1 --2- 172.21.15.46:0/3461509259 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2f680ca0e0 0x7f2f68156b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.001+0000 7f2f677fe700 1 -- 172.21.15.46:0/3461509259 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f6806c520 msgr2=0x7f2f681565f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:59.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.001+0000 7f2f677fe700 1 --2- 172.21.15.46:0/3461509259 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f6806c520 0x7f2f681565f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.001+0000 7f2f677fe700 1 -- 172.21.15.46:0/3461509259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c0094f0 con 0x7f2f6806c0b0 2024-01-27T01:06:59.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.001+0000 7f2f677fe700 1 --2- 172.21.15.46:0/3461509259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f6806c0b0 0x7f2f680c4d60 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800e960 tx=0x7f2f580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:59.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.001+0000 7f2f4f7fe700 1 -- 172.21.15.46:0/3461509259 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f58013680 con 0x7f2f6806c0b0 2024-01-27T01:06:59.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.001+0000 7f2f4f7fe700 1 -- 172.21.15.46:0/3461509259 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f580137e0 con 0x7f2f6806c0b0 2024-01-27T01:06:59.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.001+0000 7f2f6e130700 1 -- 172.21.15.46:0/3461509259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f680b2190 con 0x7f2f6806c0b0 2024-01-27T01:06:59.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.001+0000 7f2f4f7fe700 1 -- 172.21.15.46:0/3461509259 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f58013ba0 con 0x7f2f6806c0b0 2024-01-27T01:06:59.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.001+0000 7f2f6e130700 1 -- 172.21.15.46:0/3461509259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f68169e90 con 0x7f2f6806c0b0 2024-01-27T01:06:59.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.002+0000 7f2f6e130700 1 -- 172.21.15.46:0/3461509259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6804c710 con 0x7f2f6806c0b0 2024-01-27T01:06:59.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.003+0000 7f2f4f7fe700 1 -- 172.21.15.46:0/3461509259 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 129) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f2f580123d0 con 0x7f2f6806c0b0 2024-01-27T01:06:59.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.003+0000 7f2f4f7fe700 1 --2- 172.21.15.46:0/3461509259 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2f54066dd0 0x7f2f54069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:59.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.003+0000 7f2f4f7fe700 1 -- 172.21.15.46:0/3461509259 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f2f58084a50 con 0x7f2f6806c0b0 2024-01-27T01:06:59.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.003+0000 7f2f66ffd700 1 --2- 172.21.15.46:0/3461509259 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2f54066dd0 0x7f2f54069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:59.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.004+0000 7f2f66ffd700 1 --2- 172.21.15.46:0/3461509259 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2f54066dd0 0x7f2f54069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2f6806b850 tx=0x7f2f5c017000 comp rx=0 tx=0).ready entity=mgr.9485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:59.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.008+0000 7f2f4f7fe700 1 -- 172.21.15.46:0/3461509259 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2f580517e0 con 0x7f2f6806c0b0 2024-01-27T01:06:59.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.241+0000 7f2f6e130700 1 -- 172.21.15.46:0/3461509259 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f681588b0 con 0x7f2f54066dd0 2024-01-27T01:06:59.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.290+0000 7f2f4f7fe700 1 -- 172.21.15.46:0/3461509259 <== mgr.9485 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f2f681588b0 con 0x7f2f54066dd0 2024-01-27T01:06:59.290 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule activated for path mandatory_fs_argument_test_dir 2024-01-27T01:06:59.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.293+0000 7f2f6e130700 1 -- 172.21.15.46:0/3461509259 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2f54066dd0 msgr2=0x7f2f54069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:59.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.293+0000 7f2f6e130700 1 --2- 172.21.15.46:0/3461509259 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2f54066dd0 0x7f2f54069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2f6806b850 tx=0x7f2f5c017000 comp rx=0 tx=0).stop 2024-01-27T01:06:59.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.293+0000 7f2f6e130700 1 -- 172.21.15.46:0/3461509259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f6806c0b0 msgr2=0x7f2f680c4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:59.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.293+0000 7f2f6e130700 1 --2- 172.21.15.46:0/3461509259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f6806c0b0 0x7f2f680c4d60 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800e960 tx=0x7f2f580091a0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.293+0000 7f2f6e130700 1 -- 172.21.15.46:0/3461509259 shutdown_connections 2024-01-27T01:06:59.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.293+0000 7f2f6e130700 1 --2- 172.21.15.46:0/3461509259 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2f54066dd0 0x7f2f54069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.293+0000 7f2f6e130700 1 --2- 172.21.15.46:0/3461509259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f6806c0b0 0x7f2f680c4d60 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.293+0000 7f2f6e130700 1 --2- 172.21.15.46:0/3461509259 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2f680ca0e0 0x7f2f68156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.293+0000 7f2f6e130700 1 --2- 172.21.15.46:0/3461509259 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2f6806c520 0x7f2f681565f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.293+0000 7f2f6e130700 1 -- 172.21.15.46:0/3461509259 >> 172.21.15.46:0/3461509259 conn(0x7f2f680ce530 msgr2=0x7f2f680cc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:59.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.293+0000 7f2f6e130700 1 -- 172.21.15.46:0/3461509259 shutdown_connections 2024-01-27T01:06:59.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.293+0000 7f2f6e130700 1 -- 172.21.15.46:0/3461509259 wait complete. 2024-01-27T01:06:59.307 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path mandatory_fs_argument_test_dir 2024-01-27T01:06:59.307 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-01-27T01:06:59.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.448+0000 7f42039de700 1 Processor -- start 2024-01-27T01:06:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.449+0000 7f42039de700 1 -- start start 2024-01-27T01:06:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.449+0000 7f42039de700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f41fc0c93a0 0x7f41fc0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.449+0000 7f42039de700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc067d50 0x7f41fc159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.449+0000 7f42039de700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc15e390 con 0x7f41fc0681c0 2024-01-27T01:06:59.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.450+0000 7f42039de700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41fc15d810 con 0x7f41fc0c93a0 2024-01-27T01:06:59.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.450+0000 7f42039de700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc15daf0 con 0x7f41fc067d50 2024-01-27T01:06:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.450+0000 7f4200f79700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc067d50 0x7f41fc159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.450+0000 7f420177a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f41fc0c93a0 0x7f41fc0c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.450+0000 7f4200f79700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc067d50 0x7f41fc159220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:53022/0 (socket says 172.21.15.46:53022) 2024-01-27T01:06:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.450+0000 7f4200f79700 1 -- 172.21.15.46:0/2229676796 learned_addr learned my addr 172.21.15.46:0/2229676796 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.450+0000 7f4200f79700 1 -- 172.21.15.46:0/2229676796 >> v2:172.21.15.106:3300/0 conn(0x7f41fc0c93a0 msgr2=0x7f41fc0c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:06:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.450+0000 7f4200f79700 1 --2- 172.21.15.46:0/2229676796 >> v2:172.21.15.106:3300/0 conn(0x7f41fc0c93a0 0x7f41fc0c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.450+0000 7f4200f79700 1 -- 172.21.15.46:0/2229676796 >> v1:172.21.15.106:6789/0 conn(0x7f41fc0681c0 legacy=0x7f41fc159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.450+0000 7f4200f79700 1 -- 172.21.15.46:0/2229676796 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41fc15ddd0 con 0x7f41fc067d50 2024-01-27T01:06:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.450+0000 7f420177a700 1 --2- 172.21.15.46:0/2229676796 >> v2:172.21.15.106:3300/0 conn(0x7f41fc0c93a0 0x7f41fc0c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:06:59.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.451+0000 7f4200f79700 1 --2- 172.21.15.46:0/2229676796 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc067d50 0x7f41fc159220 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f41f80099b0 tx=0x7f41f8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b240fc80c80a7c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:59.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.451+0000 7f41effff700 1 -- 172.21.15.46:0/2229676796 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f8009ea0 con 0x7f41fc067d50 2024-01-27T01:06:59.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.451+0000 7f41effff700 1 -- 172.21.15.46:0/2229676796 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f800b860 con 0x7f41fc067d50 2024-01-27T01:06:59.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.451+0000 7f41effff700 1 -- 172.21.15.46:0/2229676796 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f800bc20 con 0x7f41fc067d50 2024-01-27T01:06:59.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.451+0000 7f42039de700 1 -- 172.21.15.46:0/2229676796 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc067d50 msgr2=0x7f41fc159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.451+0000 7f42039de700 1 --2- 172.21.15.46:0/2229676796 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc067d50 0x7f41fc159220 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f41f80099b0 tx=0x7f41f8009250 comp rx=0 tx=0).stop 2024-01-27T01:06:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.451+0000 7f42039de700 1 -- 172.21.15.46:0/2229676796 shutdown_connections 2024-01-27T01:06:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.452+0000 7f42039de700 1 --2- 172.21.15.46:0/2229676796 >> v2:172.21.15.106:3300/0 conn(0x7f41fc0c93a0 0x7f41fc0c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.452+0000 7f42039de700 1 --2- 172.21.15.46:0/2229676796 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc067d50 0x7f41fc159220 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.452+0000 7f42039de700 1 -- 172.21.15.46:0/2229676796 >> 172.21.15.46:0/2229676796 conn(0x7f41fc0ca9a0 msgr2=0x7f41fc068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.452+0000 7f42039de700 1 -- 172.21.15.46:0/2229676796 shutdown_connections 2024-01-27T01:06:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.452+0000 7f42039de700 1 -- 172.21.15.46:0/2229676796 wait complete. 2024-01-27T01:06:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.453+0000 7f42039de700 1 Processor -- start 2024-01-27T01:06:59.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.453+0000 7f42039de700 1 -- start start 2024-01-27T01:06:59.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.454+0000 7f42039de700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41fc067d50 0x7f41fc0c7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:59.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f42039de700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc0681c0 0x7f41fc14cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:59.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f42039de700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41fc0c93a0 0x7f41fc14d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:59.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f42039de700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc0c4510 con 0x7f41fc0c93a0 2024-01-27T01:06:59.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f42039de700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc0c4760 con 0x7f41fc067d50 2024-01-27T01:06:59.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f42039de700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc0c4a40 con 0x7f41fc0681c0 2024-01-27T01:06:59.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f4201f7b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41fc0c93a0 0x7f41fc14d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:59.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f4201f7b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41fc0c93a0 0x7f41fc14d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37356/0 (socket says 172.21.15.46:37356) 2024-01-27T01:06:59.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f4201f7b700 1 -- 172.21.15.46:0/3313851003 learned_addr learned my addr 172.21.15.46:0/3313851003 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:59.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f4200f79700 1 --2- 172.21.15.46:0/3313851003 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc0681c0 0x7f41fc14cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:59.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f420177a700 1 --2- 172.21.15.46:0/3313851003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41fc067d50 0x7f41fc0c7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:59.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f4201f7b700 1 -- 172.21.15.46:0/3313851003 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc0681c0 msgr2=0x7f41fc14cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:59.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f4201f7b700 1 --2- 172.21.15.46:0/3313851003 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc0681c0 0x7f41fc14cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f4201f7b700 1 -- 172.21.15.46:0/3313851003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41fc067d50 msgr2=0x7f41fc0c7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:59.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f4201f7b700 1 --2- 172.21.15.46:0/3313851003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41fc067d50 0x7f41fc0c7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f4201f7b700 1 -- 172.21.15.46:0/3313851003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f0009420 con 0x7f41fc0c93a0 2024-01-27T01:06:59.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f420177a700 1 --2- 172.21.15.46:0/3313851003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41fc067d50 0x7f41fc0c7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:06:59.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f4201f7b700 1 --2- 172.21.15.46:0/3313851003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41fc0c93a0 0x7f41fc14d120 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f41f400c910 tx=0x7f41f40016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:59.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.455+0000 7f41ed7fa700 1 -- 172.21.15.46:0/3313851003 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4012630 con 0x7f41fc0c93a0 2024-01-27T01:06:59.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.456+0000 7f41ed7fa700 1 -- 172.21.15.46:0/3313851003 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f40127b0 con 0x7f41fc0c93a0 2024-01-27T01:06:59.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.456+0000 7f42039de700 1 -- 172.21.15.46:0/3313851003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f80094f0 con 0x7f41fc0c93a0 2024-01-27T01:06:59.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.456+0000 7f41ed7fa700 1 -- 172.21.15.46:0/3313851003 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4012b70 con 0x7f41fc0c93a0 2024-01-27T01:06:59.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.456+0000 7f42039de700 1 -- 172.21.15.46:0/3313851003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41fc0c5000 con 0x7f41fc0c93a0 2024-01-27T01:06:59.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.457+0000 7f42039de700 1 -- 172.21.15.46:0/3313851003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41fc04c710 con 0x7f41fc0c93a0 2024-01-27T01:06:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.457+0000 7f41ed7fa700 1 -- 172.21.15.46:0/3313851003 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 129) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f41f400a3b0 con 0x7f41fc0c93a0 2024-01-27T01:06:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.458+0000 7f41ed7fa700 1 --2- 172.21.15.46:0/3313851003 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f41e0066dd0 0x7f41e0069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.458+0000 7f41ed7fa700 1 -- 172.21.15.46:0/3313851003 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f41f4084c20 con 0x7f41fc0c93a0 2024-01-27T01:06:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.458+0000 7f420177a700 1 --2- 172.21.15.46:0/3313851003 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f41e0066dd0 0x7f41e0069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.458+0000 7f420177a700 1 --2- 172.21.15.46:0/3313851003 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f41e0066dd0 0x7f41e0069280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f41f00093f0 tx=0x7f41f0008040 comp rx=0 tx=0).ready entity=mgr.9485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:59.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.462+0000 7f41ed7fa700 1 -- 172.21.15.46:0/3313851003 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f41f4055440 con 0x7f41fc0c93a0 2024-01-27T01:06:59.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.695+0000 7f42039de700 1 -- 172.21.15.46:0/3313851003 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f41fc15e0b0 con 0x7f41e0066dd0 2024-01-27T01:06:59.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.747+0000 7f41ed7fa700 1 -- 172.21.15.46:0/3313851003 <== mgr.9485 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f41fc15e0b0 con 0x7f41e0066dd0 2024-01-27T01:06:59.747 INFO:teuthology.orchestra.run.smithi046.stdout:Retention added to path mandatory_fs_argument_test_dir 2024-01-27T01:06:59.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.750+0000 7f42039de700 1 -- 172.21.15.46:0/3313851003 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f41e0066dd0 msgr2=0x7f41e0069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:59.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.750+0000 7f42039de700 1 --2- 172.21.15.46:0/3313851003 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f41e0066dd0 0x7f41e0069280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f41f00093f0 tx=0x7f41f0008040 comp rx=0 tx=0).stop 2024-01-27T01:06:59.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.750+0000 7f42039de700 1 -- 172.21.15.46:0/3313851003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41fc0c93a0 msgr2=0x7f41fc14d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:59.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.750+0000 7f42039de700 1 --2- 172.21.15.46:0/3313851003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41fc0c93a0 0x7f41fc14d120 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f41f400c910 tx=0x7f41f40016f0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.750+0000 7f42039de700 1 -- 172.21.15.46:0/3313851003 shutdown_connections 2024-01-27T01:06:59.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.750+0000 7f42039de700 1 --2- 172.21.15.46:0/3313851003 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f41e0066dd0 0x7f41e0069280 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.750+0000 7f42039de700 1 --2- 172.21.15.46:0/3313851003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41fc0c93a0 0x7f41fc14d120 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.750+0000 7f42039de700 1 --2- 172.21.15.46:0/3313851003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41fc067d50 0x7f41fc0c7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.750+0000 7f42039de700 1 --2- 172.21.15.46:0/3313851003 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f41fc0681c0 0x7f41fc14cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.750+0000 7f42039de700 1 -- 172.21.15.46:0/3313851003 >> 172.21.15.46:0/3313851003 conn(0x7f41fc0ca9a0 msgr2=0x7f41fc156240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:59.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.750+0000 7f42039de700 1 -- 172.21.15.46:0/3313851003 shutdown_connections 2024-01-27T01:06:59.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.750+0000 7f42039de700 1 -- 172.21.15.46:0/3313851003 wait complete. 2024-01-27T01:06:59.764 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path mandatory_fs_argument_test_dir 2024-01-27T01:06:59.764 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-01-27T01:06:59.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.909+0000 7f49325ed700 1 Processor -- start 2024-01-27T01:06:59.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.909+0000 7f49325ed700 1 -- start start 2024-01-27T01:06:59.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.910+0000 7f49325ed700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f492c0c9d50 0x7f492c15ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:59.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.910+0000 7f49325ed700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492c157000 con 0x7f492c0c9980 2024-01-27T01:06:59.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.910+0000 7f49325ed700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492c06f3e0 con 0x7f492c156b00 2024-01-27T01:06:59.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.910+0000 7f49325ed700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f492c06f6c0 con 0x7f492c0c9d50 2024-01-27T01:06:59.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.910+0000 7f4930b8a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f492c0c9d50 0x7f492c15ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:59.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.910+0000 7f492bfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f492c0c9980 0x7f492c06eb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56016/0 (socket says 172.21.15.46:56016) 2024-01-27T01:06:59.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.910+0000 7f492bfff700 1 -- 172.21.15.46:0/3073643174 learned_addr learned my addr 172.21.15.46:0/3073643174 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:06:59.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.910+0000 7f492affd700 1 -- 172.21.15.46:0/3073643174 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297259881 0 0) 0x7f492c157000 con 0x7f492c0c9980 2024-01-27T01:06:59.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.911+0000 7f492affd700 1 -- 172.21.15.46:0/3073643174 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4914003570 con 0x7f492c0c9980 2024-01-27T01:06:59.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.911+0000 7f4930b8a700 1 -- 172.21.15.46:0/3073643174 >> v1:172.21.15.106:6789/0 conn(0x7f492c156b00 legacy=0x7f492c0b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:59.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.911+0000 7f4930b8a700 1 -- 172.21.15.46:0/3073643174 >> v1:172.21.15.46:6789/0 conn(0x7f492c0c9980 legacy=0x7f492c06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:59.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.911+0000 7f4930b8a700 1 -- 172.21.15.46:0/3073643174 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c06f9a0 con 0x7f492c0c9d50 2024-01-27T01:06:59.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.911+0000 7f4930b8a700 1 --2- 172.21.15.46:0/3073643174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f492c0c9d50 0x7f492c15ae10 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f4924009e60 tx=0x7f4924009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2184112a44f619d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:59.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.911+0000 7f492affd700 1 -- 172.21.15.46:0/3073643174 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492400e070 con 0x7f492c0c9d50 2024-01-27T01:06:59.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.911+0000 7f492affd700 1 -- 172.21.15.46:0/3073643174 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492400bb60 con 0x7f492c0c9d50 2024-01-27T01:06:59.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.911+0000 7f492affd700 1 -- 172.21.15.46:0/3073643174 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4924012430 con 0x7f492c0c9d50 2024-01-27T01:06:59.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.911+0000 7f49325ed700 1 -- 172.21.15.46:0/3073643174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f492c0c9d50 msgr2=0x7f492c15ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:59.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.911+0000 7f49325ed700 1 --2- 172.21.15.46:0/3073643174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f492c0c9d50 0x7f492c15ae10 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f4924009e60 tx=0x7f4924009780 comp rx=0 tx=0).stop 2024-01-27T01:06:59.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.912+0000 7f49325ed700 1 -- 172.21.15.46:0/3073643174 shutdown_connections 2024-01-27T01:06:59.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.912+0000 7f49325ed700 1 --2- 172.21.15.46:0/3073643174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f492c0c9d50 0x7f492c15ae10 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.912+0000 7f49325ed700 1 -- 172.21.15.46:0/3073643174 >> 172.21.15.46:0/3073643174 conn(0x7f492c069510 msgr2=0x7f492c0aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:06:59.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.912+0000 7f49325ed700 1 -- 172.21.15.46:0/3073643174 shutdown_connections 2024-01-27T01:06:59.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.912+0000 7f49325ed700 1 -- 172.21.15.46:0/3073643174 wait complete. 2024-01-27T01:06:59.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.913+0000 7f49325ed700 1 Processor -- start 2024-01-27T01:06:59.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.913+0000 7f49325ed700 1 -- start start 2024-01-27T01:06:59.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.914+0000 7f49325ed700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f492c0c9980 0x7f492c147030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:59.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.914+0000 7f49325ed700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f492c0c9d50 0x7f492c158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:59.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.914+0000 7f49325ed700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f492c156b00 0x7f492c1554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:59.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.914+0000 7f49325ed700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0a6170 con 0x7f492c0c9d50 2024-01-27T01:06:59.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.914+0000 7f49325ed700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0a5560 con 0x7f492c156b00 2024-01-27T01:06:59.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.914+0000 7f49325ed700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0a5840 con 0x7f492c0c9980 2024-01-27T01:06:59.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.914+0000 7f492b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f492c0c9d50 0x7f492c158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:59.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.914+0000 7f492b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f492c0c9d50 0x7f492c158e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37364/0 (socket says 172.21.15.46:37364) 2024-01-27T01:06:59.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.914+0000 7f492b7fe700 1 -- 172.21.15.46:0/1312633897 learned_addr learned my addr 172.21.15.46:0/1312633897 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:06:59.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.914+0000 7f492bfff700 1 --2- 172.21.15.46:0/1312633897 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f492c0c9980 0x7f492c147030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:59.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.914+0000 7f4930b8a700 1 --2- 172.21.15.46:0/1312633897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f492c156b00 0x7f492c1554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:59.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.914+0000 7f492b7fe700 1 -- 172.21.15.46:0/1312633897 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f492c0c9980 msgr2=0x7f492c147030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:59.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.914+0000 7f492b7fe700 1 --2- 172.21.15.46:0/1312633897 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f492c0c9980 0x7f492c147030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.914+0000 7f492b7fe700 1 -- 172.21.15.46:0/1312633897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f492c156b00 msgr2=0x7f492c1554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:06:59.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.914+0000 7f492b7fe700 1 --2- 172.21.15.46:0/1312633897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f492c156b00 0x7f492c1554e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:06:59.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.914+0000 7f492b7fe700 1 -- 172.21.15.46:0/1312633897 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4924009a20 con 0x7f492c0c9d50 2024-01-27T01:06:59.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.914+0000 7f492b7fe700 1 --2- 172.21.15.46:0/1312633897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f492c0c9d50 0x7f492c158e20 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f492000c4e0 tx=0x7f492000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:59.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.915+0000 7f4913fff700 1 -- 172.21.15.46:0/1312633897 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4920011590 con 0x7f492c0c9d50 2024-01-27T01:06:59.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.915+0000 7f4913fff700 1 -- 172.21.15.46:0/1312633897 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4920011710 con 0x7f492c0c9d50 2024-01-27T01:06:59.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.915+0000 7f49325ed700 1 -- 172.21.15.46:0/1312633897 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492c0a5b20 con 0x7f492c0c9d50 2024-01-27T01:06:59.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.915+0000 7f4913fff700 1 -- 172.21.15.46:0/1312633897 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4920011ad0 con 0x7f492c0c9d50 2024-01-27T01:06:59.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.915+0000 7f49325ed700 1 -- 172.21.15.46:0/1312633897 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492c147570 con 0x7f492c0c9d50 2024-01-27T01:06:59.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.916+0000 7f49325ed700 1 -- 172.21.15.46:0/1312633897 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f492c0cc390 con 0x7f492c0c9d50 2024-01-27T01:06:59.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.916+0000 7f4913fff700 1 -- 172.21.15.46:0/1312633897 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 130) v1 ==== 84964+0+0 (secure 0 0 0) 0x7f4920001020 con 0x7f492c0c9d50 2024-01-27T01:06:59.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.917+0000 7f4913fff700 1 --2- 172.21.15.46:0/1312633897 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f49180671c0 0x7f4918069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:06:59.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.917+0000 7f4913fff700 1 -- 172.21.15.46:0/1312633897 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f4920084380 con 0x7f492c0c9d50 2024-01-27T01:06:59.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.917+0000 7f492bfff700 1 --2- 172.21.15.46:0/1312633897 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f49180671c0 0x7f4918069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:06:59.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.917+0000 7f492bfff700 1 --2- 172.21.15.46:0/1312633897 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f49180671c0 0x7f4918069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f491c007fd0 tx=0x7f491c009040 comp rx=0 tx=0).ready entity=mgr.9485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:06:59.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:06:59.921+0000 7f4913fff700 1 -- 172.21.15.46:0/1312633897 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4920050bd0 con 0x7f492c0c9d50 2024-01-27T01:07:00.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.151+0000 7f49325ed700 1 -- 172.21.15.46:0/1312633897 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f492c0a5fc0 con 0x7f49180671c0 2024-01-27T01:07:00.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.158+0000 7f4913fff700 1 -- 172.21.15.46:0/1312633897 <== mgr.9485 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+88 (secure 0 0 0) 0x7f492c0a5fc0 con 0x7f49180671c0 2024-01-27T01:07:00.158 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:00.158 INFO:teuthology.orchestra.run.smithi046.stdout:{"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-01-27T01:07:00.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.161+0000 7f49325ed700 1 -- 172.21.15.46:0/1312633897 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f49180671c0 msgr2=0x7f4918069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:00.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.161+0000 7f49325ed700 1 --2- 172.21.15.46:0/1312633897 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f49180671c0 0x7f4918069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f491c007fd0 tx=0x7f491c009040 comp rx=0 tx=0).stop 2024-01-27T01:07:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.161+0000 7f49325ed700 1 -- 172.21.15.46:0/1312633897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f492c0c9d50 msgr2=0x7f492c158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.161+0000 7f49325ed700 1 --2- 172.21.15.46:0/1312633897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f492c0c9d50 0x7f492c158e20 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f492000c4e0 tx=0x7f492000c7f0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.161+0000 7f49325ed700 1 -- 172.21.15.46:0/1312633897 shutdown_connections 2024-01-27T01:07:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.161+0000 7f49325ed700 1 --2- 172.21.15.46:0/1312633897 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f49180671c0 0x7f4918069670 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.161+0000 7f49325ed700 1 --2- 172.21.15.46:0/1312633897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f492c156b00 0x7f492c1554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.161+0000 7f49325ed700 1 --2- 172.21.15.46:0/1312633897 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f492c0c9980 0x7f492c147030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.161+0000 7f49325ed700 1 --2- 172.21.15.46:0/1312633897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f492c0c9d50 0x7f492c158e20 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.161+0000 7f49325ed700 1 -- 172.21.15.46:0/1312633897 >> 172.21.15.46:0/1312633897 conn(0x7f492c069510 msgr2=0x7f492c0cacf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.161+0000 7f49325ed700 1 -- 172.21.15.46:0/1312633897 shutdown_connections 2024-01-27T01:07:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.161+0000 7f49325ed700 1 -- 172.21.15.46:0/1312633897 wait complete. 2024-01-27T01:07:00.174 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-01-27T01:07:00.174 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-01-27T01:07:00.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.311+0000 7f3b369ee700 1 Processor -- start 2024-01-27T01:07:00.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.311+0000 7f3b369ee700 1 -- start start 2024-01-27T01:07:00.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.312+0000 7f3b369ee700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3b30153ad0 0x7f3b30156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:00.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.312+0000 7f3b369ee700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b300afc50 con 0x7f3b3006a4e0 2024-01-27T01:07:00.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.312+0000 7f3b369ee700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b30068180 con 0x7f3b30153700 2024-01-27T01:07:00.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.312+0000 7f3b369ee700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b300af9b0 con 0x7f3b30153ad0 2024-01-27T01:07:00.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.312+0000 7f3b2f7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3b30153ad0 0x7f3b30156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:00.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.312+0000 7f3b34f8b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3b3006a4e0 0x7f3b300b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56028/0 (socket says 172.21.15.46:56028) 2024-01-27T01:07:00.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.312+0000 7f3b34f8b700 1 -- 172.21.15.46:0/4278785473 learned_addr learned my addr 172.21.15.46:0/4278785473 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:00.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.312+0000 7f3b2effd700 1 -- 172.21.15.46:0/4278785473 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636773487 0 0) 0x7f3b300afc50 con 0x7f3b3006a4e0 2024-01-27T01:07:00.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.312+0000 7f3b2effd700 1 -- 172.21.15.46:0/4278785473 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b18003570 con 0x7f3b3006a4e0 2024-01-27T01:07:00.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.312+0000 7f3b2f7fe700 1 -- 172.21.15.46:0/4278785473 >> v1:172.21.15.106:6789/0 conn(0x7f3b30153700 legacy=0x7f3b30067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:00.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.312+0000 7f3b2f7fe700 1 -- 172.21.15.46:0/4278785473 >> v1:172.21.15.46:6789/0 conn(0x7f3b3006a4e0 legacy=0x7f3b300b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:00.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.312+0000 7f3b2f7fe700 1 -- 172.21.15.46:0/4278785473 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b30068460 con 0x7f3b30153ad0 2024-01-27T01:07:00.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.313+0000 7f3b2f7fe700 1 --2- 172.21.15.46:0/4278785473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b30153ad0 0x7f3b30156eb0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f3b2400aa90 tx=0x7f3b2400a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d1364b7343a183 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:00.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.313+0000 7f3b2effd700 1 -- 172.21.15.46:0/4278785473 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2400c970 con 0x7f3b30153ad0 2024-01-27T01:07:00.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.313+0000 7f3b2effd700 1 -- 172.21.15.46:0/4278785473 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2400cad0 con 0x7f3b30153ad0 2024-01-27T01:07:00.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.313+0000 7f3b2effd700 1 -- 172.21.15.46:0/4278785473 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2400ce90 con 0x7f3b30153ad0 2024-01-27T01:07:00.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.313+0000 7f3b369ee700 1 -- 172.21.15.46:0/4278785473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b30153ad0 msgr2=0x7f3b30156eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:00.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.313+0000 7f3b369ee700 1 --2- 172.21.15.46:0/4278785473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b30153ad0 0x7f3b30156eb0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f3b2400aa90 tx=0x7f3b2400a330 comp rx=0 tx=0).stop 2024-01-27T01:07:00.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.313+0000 7f3b369ee700 1 -- 172.21.15.46:0/4278785473 shutdown_connections 2024-01-27T01:07:00.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.313+0000 7f3b369ee700 1 --2- 172.21.15.46:0/4278785473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b30153ad0 0x7f3b30156eb0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.313+0000 7f3b369ee700 1 -- 172.21.15.46:0/4278785473 >> 172.21.15.46:0/4278785473 conn(0x7f3b3006e2c0 msgr2=0x7f3b300ae620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:00.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.313+0000 7f3b369ee700 1 -- 172.21.15.46:0/4278785473 shutdown_connections 2024-01-27T01:07:00.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.313+0000 7f3b369ee700 1 -- 172.21.15.46:0/4278785473 wait complete. 2024-01-27T01:07:00.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.315+0000 7f3b369ee700 1 Processor -- start 2024-01-27T01:07:00.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.315+0000 7f3b369ee700 1 -- start start 2024-01-27T01:07:00.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b369ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b3006a4e0 0x7f3b30152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:00.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b369ee700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b30153700 0x7f3b3016e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:00.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b369ee700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b30153ad0 0x7f3b3014e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:00.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b369ee700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b300ad350 con 0x7f3b3006a4e0 2024-01-27T01:07:00.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b369ee700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3016ecb0 con 0x7f3b30153700 2024-01-27T01:07:00.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b369ee700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b300ac740 con 0x7f3b30153ad0 2024-01-27T01:07:00.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b2ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b3006a4e0 0x7f3b30152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:00.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b2ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b3006a4e0 0x7f3b30152730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37374/0 (socket says 172.21.15.46:37374) 2024-01-27T01:07:00.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b2ffff700 1 -- 172.21.15.46:0/1948135659 learned_addr learned my addr 172.21.15.46:0/1948135659 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:00.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b2f7fe700 1 --2- 172.21.15.46:0/1948135659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b30153700 0x7f3b3016e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:00.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b34f8b700 1 --2- 172.21.15.46:0/1948135659 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b30153ad0 0x7f3b3014e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:00.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b2ffff700 1 -- 172.21.15.46:0/1948135659 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b30153ad0 msgr2=0x7f3b3014e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:00.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b2ffff700 1 --2- 172.21.15.46:0/1948135659 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b30153ad0 0x7f3b3014e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b2ffff700 1 -- 172.21.15.46:0/1948135659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b30153700 msgr2=0x7f3b3016e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:00.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b2ffff700 1 --2- 172.21.15.46:0/1948135659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b30153700 0x7f3b3016e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b2ffff700 1 -- 172.21.15.46:0/1948135659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2400a5d0 con 0x7f3b3006a4e0 2024-01-27T01:07:00.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b2f7fe700 1 --2- 172.21.15.46:0/1948135659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b30153700 0x7f3b3016e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:07:00.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b2ffff700 1 --2- 172.21.15.46:0/1948135659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b3006a4e0 0x7f3b30152730 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000c570 tx=0x7f3b20001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:00.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b17fff700 1 -- 172.21.15.46:0/1948135659 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20011590 con 0x7f3b3006a4e0 2024-01-27T01:07:00.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.316+0000 7f3b17fff700 1 -- 172.21.15.46:0/1948135659 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b200116f0 con 0x7f3b3006a4e0 2024-01-27T01:07:00.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.317+0000 7f3b369ee700 1 -- 172.21.15.46:0/1948135659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b300aca20 con 0x7f3b3006a4e0 2024-01-27T01:07:00.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.317+0000 7f3b17fff700 1 -- 172.21.15.46:0/1948135659 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20011ab0 con 0x7f3b3006a4e0 2024-01-27T01:07:00.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.317+0000 7f3b369ee700 1 -- 172.21.15.46:0/1948135659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b300acfe0 con 0x7f3b3006a4e0 2024-01-27T01:07:00.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.317+0000 7f3b369ee700 1 -- 172.21.15.46:0/1948135659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3004c710 con 0x7f3b3006a4e0 2024-01-27T01:07:00.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.318+0000 7f3b17fff700 1 -- 172.21.15.46:0/1948135659 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 130) v1 ==== 84964+0+0 (secure 0 0 0) 0x7f3b2001b3b0 con 0x7f3b3006a4e0 2024-01-27T01:07:00.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.318+0000 7f3b17fff700 1 --2- 172.21.15.46:0/1948135659 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3b1c0671c0 0x7f3b1c069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:00.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.319+0000 7f3b17fff700 1 -- 172.21.15.46:0/1948135659 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f3b200844d0 con 0x7f3b3006a4e0 2024-01-27T01:07:00.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.319+0000 7f3b2f7fe700 1 --2- 172.21.15.46:0/1948135659 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3b1c0671c0 0x7f3b1c069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:00.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.319+0000 7f3b2f7fe700 1 --2- 172.21.15.46:0/1948135659 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3b1c0671c0 0x7f3b1c069670 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3b2400aa90 tx=0x7f3b2401a000 comp rx=0 tx=0).ready entity=mgr.9485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:00.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.323+0000 7f3b17fff700 1 -- 172.21.15.46:0/1948135659 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3b20050dd0 con 0x7f3b3006a4e0 2024-01-27T01:07:00.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.551+0000 7f3b369ee700 1 -- 172.21.15.46:0/1948135659 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b300ad490 con 0x7f3b1c0671c0 2024-01-27T01:07:00.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.559+0000 7f3b17fff700 1 -- 172.21.15.46:0/1948135659 <== mgr.9485 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+338 (secure 0 0 0) 0x7f3b2400cc40 con 0x7f3b1c0671c0 2024-01-27T01:07:00.558 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:00.559 INFO:teuthology.orchestra.run.smithi046.stdout:[{"fs": "backup_fs", "subvol": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-01-27T00:00:00", "created": "2024-01-27T01:06:58", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-01-27T01:07:00.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.561+0000 7f3b369ee700 1 -- 172.21.15.46:0/1948135659 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3b1c0671c0 msgr2=0x7f3b1c069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:00.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.561+0000 7f3b369ee700 1 --2- 172.21.15.46:0/1948135659 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3b1c0671c0 0x7f3b1c069670 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3b2400aa90 tx=0x7f3b2401a000 comp rx=0 tx=0).stop 2024-01-27T01:07:00.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.561+0000 7f3b369ee700 1 -- 172.21.15.46:0/1948135659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b3006a4e0 msgr2=0x7f3b30152730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:00.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.561+0000 7f3b369ee700 1 --2- 172.21.15.46:0/1948135659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b3006a4e0 0x7f3b30152730 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000c570 tx=0x7f3b20001070 comp rx=0 tx=0).stop 2024-01-27T01:07:00.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.561+0000 7f3b369ee700 1 -- 172.21.15.46:0/1948135659 shutdown_connections 2024-01-27T01:07:00.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.561+0000 7f3b369ee700 1 --2- 172.21.15.46:0/1948135659 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3b1c0671c0 0x7f3b1c069670 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.561+0000 7f3b369ee700 1 --2- 172.21.15.46:0/1948135659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b3006a4e0 0x7f3b30152730 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.561+0000 7f3b369ee700 1 --2- 172.21.15.46:0/1948135659 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b30153ad0 0x7f3b3014e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.561+0000 7f3b369ee700 1 --2- 172.21.15.46:0/1948135659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b30153700 0x7f3b3016e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.561+0000 7f3b369ee700 1 -- 172.21.15.46:0/1948135659 >> 172.21.15.46:0/1948135659 conn(0x7f3b3006e2c0 msgr2=0x7f3b30156eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:00.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.561+0000 7f3b369ee700 1 -- 172.21.15.46:0/1948135659 shutdown_connections 2024-01-27T01:07:00.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.561+0000 7f3b369ee700 1 -- 172.21.15.46:0/1948135659 wait complete. 2024-01-27T01:07:00.574 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "backup_fs", "subvol": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-01-27T00:00:00", "created": "2024-01-27T01:06:58", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-01-27T01:07:00.574 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-01-27T01:07:00.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.714+0000 7fee2ca13700 1 Processor -- start 2024-01-27T01:07:00.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.714+0000 7fee2ca13700 1 -- start start 2024-01-27T01:07:00.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.715+0000 7fee2ca13700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fee280cb640 0x7fee280c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.715+0000 7fee2ca13700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee280ccc00 con 0x7fee280c8da0 2024-01-27T01:07:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.715+0000 7fee2ca13700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee280cc310 con 0x7fee28141200 2024-01-27T01:07:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.715+0000 7fee2ca13700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee280cbbf0 con 0x7fee280cb640 2024-01-27T01:07:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.715+0000 7fee2659c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fee280c8da0 0x7fee28140af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46244/0 (socket says 172.21.15.46:46244) 2024-01-27T01:07:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.715+0000 7fee2659c700 1 -- 172.21.15.46:0/3620996723 learned_addr learned my addr 172.21.15.46:0/3620996723 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.715+0000 7fee26d9d700 1 --2- 172.21.15.46:0/3620996723 >> v2:172.21.15.106:3300/0 conn(0x7fee280cb640 0x7fee280c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.715+0000 7fee2559a700 1 -- 172.21.15.46:0/3620996723 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524543561 0 0) 0x7fee280ccc00 con 0x7fee280c8da0 2024-01-27T01:07:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.715+0000 7fee2559a700 1 -- 172.21.15.46:0/3620996723 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee14003570 con 0x7fee280c8da0 2024-01-27T01:07:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee2559a700 1 -- 172.21.15.46:0/3620996723 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1743704562 0 0) 0x7fee14003570 con 0x7fee280c8da0 2024-01-27T01:07:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee2559a700 1 -- 172.21.15.46:0/3620996723 >> v2:172.21.15.106:3300/0 conn(0x7fee280cb640 msgr2=0x7fee280c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee2559a700 1 --2- 172.21.15.46:0/3620996723 >> v2:172.21.15.106:3300/0 conn(0x7fee280cb640 0x7fee280c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee2559a700 1 -- 172.21.15.46:0/3620996723 >> v1:172.21.15.106:6789/0 conn(0x7fee28141200 legacy=0x7fee280caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee2559a700 1 -- 172.21.15.46:0/3620996723 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee280cc5b0 con 0x7fee280c8da0 2024-01-27T01:07:00.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee2559a700 1 -- 172.21.15.46:0/3620996723 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fee10001470 con 0x7fee280c8da0 2024-01-27T01:07:00.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee2559a700 1 -- 172.21.15.46:0/3620996723 >> v1:172.21.15.46:6789/0 conn(0x7fee280c8da0 legacy=0x7fee28140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:00.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee2559a700 1 --2- 172.21.15.46:0/3620996723 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee14003a60 0x7fee14003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:00.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee2559a700 1 --2- 172.21.15.46:0/3620996723 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee14004810 0x7fee14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:00.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee2559a700 1 --2- 172.21.15.46:0/3620996723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee140051c0 0x7fee1400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:00.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee2559a700 1 -- 172.21.15.46:0/3620996723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee280cbbf0 con 0x7fee140051c0 2024-01-27T01:07:00.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee2559a700 1 -- 172.21.15.46:0/3620996723 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1400c160 con 0x7fee14003a60 2024-01-27T01:07:00.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee2559a700 1 -- 172.21.15.46:0/3620996723 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1400c320 con 0x7fee14004810 2024-01-27T01:07:00.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee26d9d700 1 --2- 172.21.15.46:0/3620996723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee140051c0 0x7fee1400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:00.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee2659c700 1 --2- 172.21.15.46:0/3620996723 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee14003a60 0x7fee14003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:00.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee25d9b700 1 --2- 172.21.15.46:0/3620996723 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee14004810 0x7fee14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:00.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee26d9d700 1 -- 172.21.15.46:0/3620996723 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee14004810 msgr2=0x7fee14004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:00.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee26d9d700 1 --2- 172.21.15.46:0/3620996723 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee14004810 0x7fee14004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee26d9d700 1 -- 172.21.15.46:0/3620996723 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee14003a60 msgr2=0x7fee14003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee26d9d700 1 --2- 172.21.15.46:0/3620996723 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee14003a60 0x7fee14003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee26d9d700 1 -- 172.21.15.46:0/3620996723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee14003570 con 0x7fee140051c0 2024-01-27T01:07:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee26d9d700 1 -- 172.21.15.46:0/3620996723 reap_dead start 2024-01-27T01:07:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.716+0000 7fee26d9d700 1 --2- 172.21.15.46:0/3620996723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee140051c0 0x7fee1400ba60 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fee2000e210 tx=0x7fee2000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db3ff1af19457fd server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.717+0000 7fee2559a700 1 -- 172.21.15.46:0/3620996723 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2000eac0 con 0x7fee140051c0 2024-01-27T01:07:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.717+0000 7fee2559a700 1 -- 172.21.15.46:0/3620996723 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2000f040 con 0x7fee140051c0 2024-01-27T01:07:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.717+0000 7fee2559a700 1 -- 172.21.15.46:0/3620996723 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2000ee90 con 0x7fee140051c0 2024-01-27T01:07:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.717+0000 7fee2ca13700 1 -- 172.21.15.46:0/3620996723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee140051c0 msgr2=0x7fee1400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.717+0000 7fee2ca13700 1 --2- 172.21.15.46:0/3620996723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee140051c0 0x7fee1400ba60 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fee2000e210 tx=0x7fee2000b040 comp rx=0 tx=0).stop 2024-01-27T01:07:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.717+0000 7fee2ca13700 1 -- 172.21.15.46:0/3620996723 shutdown_connections 2024-01-27T01:07:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.717+0000 7fee2ca13700 1 --2- 172.21.15.46:0/3620996723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee140051c0 0x7fee1400ba60 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.717+0000 7fee2ca13700 1 -- 172.21.15.46:0/3620996723 >> 172.21.15.46:0/3620996723 conn(0x7fee2806bd90 msgr2=0x7fee2806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.717+0000 7fee2ca13700 1 -- 172.21.15.46:0/3620996723 shutdown_connections 2024-01-27T01:07:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.717+0000 7fee2ca13700 1 -- 172.21.15.46:0/3620996723 wait complete. 2024-01-27T01:07:00.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.718+0000 7fee2ca13700 1 Processor -- start 2024-01-27T01:07:00.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.719+0000 7fee2ca13700 1 -- start start 2024-01-27T01:07:00.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.719+0000 7fee2ca13700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee140051c0 0x7fee28156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:00.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.719+0000 7fee2ca13700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee280b1a80 0x7fee28069af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.719+0000 7fee2ca13700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee2806a030 0x7fee280a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.719+0000 7fee2ca13700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee28156b90 con 0x7fee140051c0 2024-01-27T01:07:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.719+0000 7fee2ca13700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2816ad90 con 0x7fee2806a030 2024-01-27T01:07:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.719+0000 7fee2ca13700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2816bbf0 con 0x7fee280b1a80 2024-01-27T01:07:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.719+0000 7fee2659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee140051c0 0x7fee28156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.719+0000 7fee2659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee140051c0 0x7fee28156650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59392/0 (socket says 172.21.15.46:59392) 2024-01-27T01:07:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.719+0000 7fee2659c700 1 -- 172.21.15.46:0/1035754168 learned_addr learned my addr 172.21.15.46:0/1035754168 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.720+0000 7fee26d9d700 1 --2- 172.21.15.46:0/1035754168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee2806a030 0x7fee280a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.720+0000 7fee25d9b700 1 --2- 172.21.15.46:0/1035754168 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee280b1a80 0x7fee28069af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.720+0000 7fee2659c700 1 -- 172.21.15.46:0/1035754168 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee280b1a80 msgr2=0x7fee28069af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.720+0000 7fee2659c700 1 --2- 172.21.15.46:0/1035754168 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee280b1a80 0x7fee28069af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.720+0000 7fee2659c700 1 -- 172.21.15.46:0/1035754168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee2806a030 msgr2=0x7fee280a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.720+0000 7fee2659c700 1 --2- 172.21.15.46:0/1035754168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee2806a030 0x7fee280a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.720+0000 7fee2659c700 1 -- 172.21.15.46:0/1035754168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2816afe0 con 0x7fee140051c0 2024-01-27T01:07:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.720+0000 7fee2659c700 1 --2- 172.21.15.46:0/1035754168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee140051c0 0x7fee28156650 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fee10007390 tx=0x7fee280c96b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.720+0000 7fee1e7fc700 1 -- 172.21.15.46:0/1035754168 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee10018030 con 0x7fee140051c0 2024-01-27T01:07:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.720+0000 7fee1e7fc700 1 -- 172.21.15.46:0/1035754168 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee10007820 con 0x7fee140051c0 2024-01-27T01:07:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.720+0000 7fee1e7fc700 1 -- 172.21.15.46:0/1035754168 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1000fb00 con 0x7fee140051c0 2024-01-27T01:07:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.720+0000 7fee2ca13700 1 -- 172.21.15.46:0/1035754168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2816b2c0 con 0x7fee140051c0 2024-01-27T01:07:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.720+0000 7fee2ca13700 1 -- 172.21.15.46:0/1035754168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2816b880 con 0x7fee140051c0 2024-01-27T01:07:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.721+0000 7fee2ca13700 1 -- 172.21.15.46:0/1035754168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2804c710 con 0x7fee140051c0 2024-01-27T01:07:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.722+0000 7fee1e7fc700 1 -- 172.21.15.46:0/1035754168 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 130) v1 ==== 84964+0+0 (secure 0 0 0) 0x7fee1001b070 con 0x7fee140051c0 2024-01-27T01:07:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.722+0000 7fee1e7fc700 1 --2- 172.21.15.46:0/1035754168 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee080671c0 0x7fee08069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.722+0000 7fee1e7fc700 1 -- 172.21.15.46:0/1035754168 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fee10019070 con 0x7fee140051c0 2024-01-27T01:07:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.722+0000 7fee25d9b700 1 --2- 172.21.15.46:0/1035754168 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee080671c0 0x7fee08069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.723+0000 7fee25d9b700 1 --2- 172.21.15.46:0/1035754168 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee080671c0 0x7fee08069670 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fee28140070 tx=0x7fee18007d30 comp rx=0 tx=0).ready entity=mgr.9485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:00.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.727+0000 7fee1e7fc700 1 -- 172.21.15.46:0/1035754168 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fee10051810 con 0x7fee140051c0 2024-01-27T01:07:00.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:00.960+0000 7fee2ca13700 1 -- 172.21.15.46:0/1035754168 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fee2816ba40 con 0x7fee080671c0 2024-01-27T01:07:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.016+0000 7fee1e7fc700 1 -- 172.21.15.46:0/1035754168 <== mgr.9485 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fee2816ba40 con 0x7fee080671c0 2024-01-27T01:07:01.016 INFO:teuthology.orchestra.run.smithi046.stdout:Retention removed from path mandatory_fs_argument_test_dir 2024-01-27T01:07:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.019+0000 7fee2ca13700 1 -- 172.21.15.46:0/1035754168 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee080671c0 msgr2=0x7fee08069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.019+0000 7fee2ca13700 1 --2- 172.21.15.46:0/1035754168 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee080671c0 0x7fee08069670 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fee28140070 tx=0x7fee18007d30 comp rx=0 tx=0).stop 2024-01-27T01:07:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.019+0000 7fee2ca13700 1 -- 172.21.15.46:0/1035754168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee140051c0 msgr2=0x7fee28156650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.019+0000 7fee2ca13700 1 --2- 172.21.15.46:0/1035754168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee140051c0 0x7fee28156650 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fee10007390 tx=0x7fee280c96b0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.019+0000 7fee2ca13700 1 -- 172.21.15.46:0/1035754168 shutdown_connections 2024-01-27T01:07:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.019+0000 7fee2ca13700 1 --2- 172.21.15.46:0/1035754168 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee080671c0 0x7fee08069670 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.019+0000 7fee2ca13700 1 --2- 172.21.15.46:0/1035754168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee140051c0 0x7fee28156650 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.019+0000 7fee2ca13700 1 --2- 172.21.15.46:0/1035754168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee2806a030 0x7fee280a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.019+0000 7fee2ca13700 1 --2- 172.21.15.46:0/1035754168 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee280b1a80 0x7fee28069af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.019+0000 7fee2ca13700 1 -- 172.21.15.46:0/1035754168 >> 172.21.15.46:0/1035754168 conn(0x7fee2806bd90 msgr2=0x7fee2806c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.019+0000 7fee2ca13700 1 -- 172.21.15.46:0/1035754168 shutdown_connections 2024-01-27T01:07:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.019+0000 7fee2ca13700 1 -- 172.21.15.46:0/1035754168 wait complete. 2024-01-27T01:07:01.033 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention removed from path mandatory_fs_argument_test_dir 2024-01-27T01:07:01.033 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-01-27T01:07:01.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.172+0000 7f34aec83700 1 Processor -- start 2024-01-27T01:07:01.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.172+0000 7f34aec83700 1 -- start start 2024-01-27T01:07:01.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.173+0000 7f34aec83700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f34a8153d20 0x7f34a80b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.173+0000 7f34aec83700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f34a81540f0 0x7f34a8157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.173+0000 7f34aec83700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a806eda0 con 0x7f34a8159f70 2024-01-27T01:07:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.173+0000 7f34aec83700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34a815af50 con 0x7f34a81540f0 2024-01-27T01:07:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.173+0000 7f34aec83700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34a8068830 con 0x7f34a8153d20 2024-01-27T01:07:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.173+0000 7f34a7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f34a81540f0 0x7f34a8157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.173+0000 7f34a7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f34a81540f0 0x7f34a8157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59396/0 (socket says 172.21.15.46:59396) 2024-01-27T01:07:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.173+0000 7f34a7fff700 1 -- 172.21.15.46:0/1223339401 learned_addr learned my addr 172.21.15.46:0/1223339401 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.173+0000 7f34aca1f700 1 --2- 172.21.15.46:0/1223339401 >> v2:172.21.15.106:3300/0 conn(0x7f34a8153d20 0x7f34a80b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.173+0000 7f34a7fff700 1 -- 172.21.15.46:0/1223339401 >> v2:172.21.15.106:3300/0 conn(0x7f34a8153d20 msgr2=0x7f34a80b2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.173+0000 7f34a7fff700 1 --2- 172.21.15.46:0/1223339401 >> v2:172.21.15.106:3300/0 conn(0x7f34a8153d20 0x7f34a80b2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.173+0000 7f34a7fff700 1 -- 172.21.15.46:0/1223339401 >> v1:172.21.15.106:6789/0 conn(0x7f34a8159f70 legacy=0x7f34a80b0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.174+0000 7f34a7fff700 1 -- 172.21.15.46:0/1223339401 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a815a6b0 con 0x7f34a81540f0 2024-01-27T01:07:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.174+0000 7f34a7fff700 1 --2- 172.21.15.46:0/1223339401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34a81540f0 0x7f34a8157ba0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f34a806e2c0 tx=0x7f3494009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351f31f65196904 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:01.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.174+0000 7f34a77fe700 1 -- 172.21.15.46:0/1223339401 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349400b860 con 0x7f34a81540f0 2024-01-27T01:07:01.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.174+0000 7f34a77fe700 1 -- 172.21.15.46:0/1223339401 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349400b9c0 con 0x7f34a81540f0 2024-01-27T01:07:01.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.174+0000 7f34a77fe700 1 -- 172.21.15.46:0/1223339401 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349400bd60 con 0x7f34a81540f0 2024-01-27T01:07:01.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.174+0000 7f34aec83700 1 -- 172.21.15.46:0/1223339401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34a81540f0 msgr2=0x7f34a8157ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:01.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.174+0000 7f34aec83700 1 --2- 172.21.15.46:0/1223339401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34a81540f0 0x7f34a8157ba0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f34a806e2c0 tx=0x7f3494009250 comp rx=0 tx=0).stop 2024-01-27T01:07:01.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.174+0000 7f34aec83700 1 -- 172.21.15.46:0/1223339401 shutdown_connections 2024-01-27T01:07:01.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.174+0000 7f34aec83700 1 --2- 172.21.15.46:0/1223339401 >> v2:172.21.15.106:3300/0 conn(0x7f34a8153d20 0x7f34a80b2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.175+0000 7f34aec83700 1 --2- 172.21.15.46:0/1223339401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34a81540f0 0x7f34a8157ba0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.175+0000 7f34aec83700 1 -- 172.21.15.46:0/1223339401 >> 172.21.15.46:0/1223339401 conn(0x7f34a80694d0 msgr2=0x7f34a80aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:01.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.175+0000 7f34aec83700 1 -- 172.21.15.46:0/1223339401 shutdown_connections 2024-01-27T01:07:01.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.175+0000 7f34aec83700 1 -- 172.21.15.46:0/1223339401 wait complete. 2024-01-27T01:07:01.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.176+0000 7f34aec83700 1 Processor -- start 2024-01-27T01:07:01.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.176+0000 7f34aec83700 1 -- start start 2024-01-27T01:07:01.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.177+0000 7f34aec83700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34a8153d20 0x7f34a814b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:01.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.177+0000 7f34aec83700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34a81540f0 0x7f34a815cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:01.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.177+0000 7f34aec83700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34a8159f70 0x7f34a815b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:01.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.177+0000 7f34aec83700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a815d250 con 0x7f34a8159f70 2024-01-27T01:07:01.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.177+0000 7f34aec83700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a815c880 con 0x7f34a81540f0 2024-01-27T01:07:01.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.177+0000 7f34aec83700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a814e990 con 0x7f34a8153d20 2024-01-27T01:07:01.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.177+0000 7f34ad220700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34a8159f70 0x7f34a815b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:01.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.177+0000 7f34ad220700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34a8159f70 0x7f34a815b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59406/0 (socket says 172.21.15.46:59406) 2024-01-27T01:07:01.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.177+0000 7f34ad220700 1 -- 172.21.15.46:0/3630172356 learned_addr learned my addr 172.21.15.46:0/3630172356 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:01.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.177+0000 7f34a7fff700 1 --2- 172.21.15.46:0/3630172356 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34a81540f0 0x7f34a815cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:01.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.177+0000 7f34aca1f700 1 --2- 172.21.15.46:0/3630172356 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34a8153d20 0x7f34a814b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:01.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.177+0000 7f34ad220700 1 -- 172.21.15.46:0/3630172356 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34a8153d20 msgr2=0x7f34a814b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:01.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.177+0000 7f34ad220700 1 --2- 172.21.15.46:0/3630172356 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34a8153d20 0x7f34a814b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.177+0000 7f34ad220700 1 -- 172.21.15.46:0/3630172356 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34a81540f0 msgr2=0x7f34a815cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:01.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.177+0000 7f34ad220700 1 --2- 172.21.15.46:0/3630172356 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34a81540f0 0x7f34a815cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.177+0000 7f34ad220700 1 -- 172.21.15.46:0/3630172356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34940094f0 con 0x7f34a8159f70 2024-01-27T01:07:01.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.178+0000 7f34ad220700 1 --2- 172.21.15.46:0/3630172356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34a8159f70 0x7f34a815b090 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f34a000c840 tx=0x7f34a0001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:01.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.178+0000 7f34a4ff9700 1 -- 172.21.15.46:0/3630172356 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a0012630 con 0x7f34a8159f70 2024-01-27T01:07:01.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.178+0000 7f34a4ff9700 1 -- 172.21.15.46:0/3630172356 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a00127b0 con 0x7f34a8159f70 2024-01-27T01:07:01.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.178+0000 7f34aec83700 1 -- 172.21.15.46:0/3630172356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a815bc70 con 0x7f34a8159f70 2024-01-27T01:07:01.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.178+0000 7f34a4ff9700 1 -- 172.21.15.46:0/3630172356 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a0012b70 con 0x7f34a8159f70 2024-01-27T01:07:01.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.178+0000 7f34aec83700 1 -- 172.21.15.46:0/3630172356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a815c2b0 con 0x7f34a8159f70 2024-01-27T01:07:01.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.179+0000 7f34aec83700 1 -- 172.21.15.46:0/3630172356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a815ca10 con 0x7f34a8159f70 2024-01-27T01:07:01.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.179+0000 7f34a4ff9700 1 -- 172.21.15.46:0/3630172356 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 130) v1 ==== 84964+0+0 (secure 0 0 0) 0x7f34a0003030 con 0x7f34a8159f70 2024-01-27T01:07:01.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.180+0000 7f34a4ff9700 1 --2- 172.21.15.46:0/3630172356 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f348c0671c0 0x7f348c069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:01.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.180+0000 7f34a4ff9700 1 -- 172.21.15.46:0/3630172356 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f34a0085020 con 0x7f34a8159f70 2024-01-27T01:07:01.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.180+0000 7f34aca1f700 1 --2- 172.21.15.46:0/3630172356 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f348c0671c0 0x7f348c069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:01.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.180+0000 7f34aca1f700 1 --2- 172.21.15.46:0/3630172356 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f348c0671c0 0x7f348c069670 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f349c006fd0 tx=0x7f349c008040 comp rx=0 tx=0).ready entity=mgr.9485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:01.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.184+0000 7f34a4ff9700 1 -- 172.21.15.46:0/3630172356 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f34a0051870 con 0x7f34a8159f70 2024-01-27T01:07:01.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.418+0000 7f34aec83700 1 -- 172.21.15.46:0/3630172356 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f34a804c710 con 0x7f348c0671c0 2024-01-27T01:07:01.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.476+0000 7f34a4ff9700 1 -- 172.21.15.46:0/3630172356 <== mgr.9485 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+61 (secure 0 0 0) 0x7f34a804c710 con 0x7f348c0671c0 2024-01-27T01:07:01.476 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule deactivated for path mandatory_fs_argument_test_dir 2024-01-27T01:07:01.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.478+0000 7f34aec83700 1 -- 172.21.15.46:0/3630172356 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f348c0671c0 msgr2=0x7f348c069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:01.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.479+0000 7f34aec83700 1 --2- 172.21.15.46:0/3630172356 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f348c0671c0 0x7f348c069670 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f349c006fd0 tx=0x7f349c008040 comp rx=0 tx=0).stop 2024-01-27T01:07:01.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.479+0000 7f34aec83700 1 -- 172.21.15.46:0/3630172356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34a8159f70 msgr2=0x7f34a815b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:01.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.479+0000 7f34aec83700 1 --2- 172.21.15.46:0/3630172356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34a8159f70 0x7f34a815b090 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f34a000c840 tx=0x7f34a0001670 comp rx=0 tx=0).stop 2024-01-27T01:07:01.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.479+0000 7f34aec83700 1 -- 172.21.15.46:0/3630172356 shutdown_connections 2024-01-27T01:07:01.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.479+0000 7f34aec83700 1 --2- 172.21.15.46:0/3630172356 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f348c0671c0 0x7f348c069670 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.479+0000 7f34aec83700 1 --2- 172.21.15.46:0/3630172356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34a8159f70 0x7f34a815b090 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.479+0000 7f34aec83700 1 --2- 172.21.15.46:0/3630172356 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34a8153d20 0x7f34a814b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.479+0000 7f34aec83700 1 --2- 172.21.15.46:0/3630172356 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34a81540f0 0x7f34a815cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.479+0000 7f34aec83700 1 -- 172.21.15.46:0/3630172356 >> 172.21.15.46:0/3630172356 conn(0x7f34a80694d0 msgr2=0x7f34a81549f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:01.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.479+0000 7f34aec83700 1 -- 172.21.15.46:0/3630172356 shutdown_connections 2024-01-27T01:07:01.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.479+0000 7f34aec83700 1 -- 172.21.15.46:0/3630172356 wait complete. 2024-01-27T01:07:01.494 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path mandatory_fs_argument_test_dir 2024-01-27T01:07:01.494 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-01-27T01:07:01.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.634+0000 7fe58f733700 1 Processor -- start 2024-01-27T01:07:01.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.635+0000 7fe58f733700 1 -- start start 2024-01-27T01:07:01.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.635+0000 7fe58f733700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe5880ca4b0 0x7fe5880afe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.635+0000 7fe58f733700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5880b3a50 0x7fe5880b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.635+0000 7fe58f733700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5880b43d0 con 0x7fe5880ca0e0 2024-01-27T01:07:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.635+0000 7fe58f733700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58806c7c0 con 0x7fe5880b3a50 2024-01-27T01:07:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.635+0000 7fe58f733700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5880cada0 con 0x7fe5880ca4b0 2024-01-27T01:07:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.636+0000 7fe58dcd0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5880b3a50 0x7fe5880b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.636+0000 7fe58dcd0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5880b3a50 0x7fe5880b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59410/0 (socket says 172.21.15.46:59410) 2024-01-27T01:07:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.636+0000 7fe58dcd0700 1 -- 172.21.15.46:0/4109335612 learned_addr learned my addr 172.21.15.46:0/4109335612 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.636+0000 7fe58ccce700 1 --2- 172.21.15.46:0/4109335612 >> v2:172.21.15.106:3300/0 conn(0x7fe5880ca4b0 0x7fe5880afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.636+0000 7fe58dcd0700 1 -- 172.21.15.46:0/4109335612 >> v2:172.21.15.106:3300/0 conn(0x7fe5880ca4b0 msgr2=0x7fe5880afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.636+0000 7fe58dcd0700 1 --2- 172.21.15.46:0/4109335612 >> v2:172.21.15.106:3300/0 conn(0x7fe5880ca4b0 0x7fe5880afe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.636+0000 7fe58dcd0700 1 -- 172.21.15.46:0/4109335612 >> v1:172.21.15.106:6789/0 conn(0x7fe5880ca0e0 legacy=0x7fe58806c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.636+0000 7fe58dcd0700 1 -- 172.21.15.46:0/4109335612 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe588158010 con 0x7fe5880b3a50 2024-01-27T01:07:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.636+0000 7fe58dcd0700 1 --2- 172.21.15.46:0/4109335612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5880b3a50 0x7fe5880b3e90 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fe588155790 tx=0x7fe580009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f2e450210f339 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:01.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.636+0000 7fe57ffff700 1 -- 172.21.15.46:0/4109335612 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58000e070 con 0x7fe5880b3a50 2024-01-27T01:07:01.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.636+0000 7fe57ffff700 1 -- 172.21.15.46:0/4109335612 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58000ba20 con 0x7fe5880b3a50 2024-01-27T01:07:01.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.636+0000 7fe57ffff700 1 -- 172.21.15.46:0/4109335612 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5800092e0 con 0x7fe5880b3a50 2024-01-27T01:07:01.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.637+0000 7fe58f733700 1 -- 172.21.15.46:0/4109335612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5880b3a50 msgr2=0x7fe5880b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:01.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.637+0000 7fe58f733700 1 --2- 172.21.15.46:0/4109335612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5880b3a50 0x7fe5880b3e90 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fe588155790 tx=0x7fe580009840 comp rx=0 tx=0).stop 2024-01-27T01:07:01.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.637+0000 7fe58f733700 1 -- 172.21.15.46:0/4109335612 shutdown_connections 2024-01-27T01:07:01.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.637+0000 7fe58f733700 1 --2- 172.21.15.46:0/4109335612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5880b3a50 0x7fe5880b3e90 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.637+0000 7fe58f733700 1 --2- 172.21.15.46:0/4109335612 >> v2:172.21.15.106:3300/0 conn(0x7fe5880ca4b0 0x7fe5880afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.637+0000 7fe58f733700 1 -- 172.21.15.46:0/4109335612 >> 172.21.15.46:0/4109335612 conn(0x7fe5880ce530 msgr2=0x7fe5880ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:01.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.637+0000 7fe58f733700 1 -- 172.21.15.46:0/4109335612 shutdown_connections 2024-01-27T01:07:01.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.637+0000 7fe58f733700 1 -- 172.21.15.46:0/4109335612 wait complete. 2024-01-27T01:07:01.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.639+0000 7fe58f733700 1 Processor -- start 2024-01-27T01:07:01.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.639+0000 7fe58f733700 1 -- start start 2024-01-27T01:07:01.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.640+0000 7fe58f733700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5880b3a50 0x7fe5880cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:01.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.640+0000 7fe58f733700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5880ca0e0 0x7fe5881521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:01.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.640+0000 7fe58f733700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5880ca4b0 0x7fe588150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:01.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.640+0000 7fe58f733700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe588153540 con 0x7fe5880ca0e0 2024-01-27T01:07:01.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.640+0000 7fe58f733700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5880c6840 con 0x7fe5880ca4b0 2024-01-27T01:07:01.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.640+0000 7fe58f733700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5880b3300 con 0x7fe5880b3a50 2024-01-27T01:07:01.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.640+0000 7fe58ccce700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5880ca0e0 0x7fe5881521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:01.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.640+0000 7fe58ccce700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5880ca0e0 0x7fe5881521e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59416/0 (socket says 172.21.15.46:59416) 2024-01-27T01:07:01.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.640+0000 7fe58ccce700 1 -- 172.21.15.46:0/145903210 learned_addr learned my addr 172.21.15.46:0/145903210 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:01.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.640+0000 7fe58dcd0700 1 --2- 172.21.15.46:0/145903210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5880ca4b0 0x7fe588150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:01.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.640+0000 7fe58d4cf700 1 --2- 172.21.15.46:0/145903210 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5880b3a50 0x7fe5880cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.641+0000 7fe58ccce700 1 -- 172.21.15.46:0/145903210 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5880b3a50 msgr2=0x7fe5880cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.641+0000 7fe58ccce700 1 --2- 172.21.15.46:0/145903210 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5880b3a50 0x7fe5880cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.641+0000 7fe58ccce700 1 -- 172.21.15.46:0/145903210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5880ca4b0 msgr2=0x7fe588150610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.641+0000 7fe58ccce700 1 --2- 172.21.15.46:0/145903210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5880ca4b0 0x7fe588150610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.641+0000 7fe58ccce700 1 -- 172.21.15.46:0/145903210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe580009ae0 con 0x7fe5880ca0e0 2024-01-27T01:07:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.641+0000 7fe58dcd0700 1 --2- 172.21.15.46:0/145903210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5880ca4b0 0x7fe588150610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.641+0000 7fe58ccce700 1 --2- 172.21.15.46:0/145903210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5880ca0e0 0x7fe5881521e0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fe57800ecb0 tx=0x7fe5780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.641+0000 7fe57d7fa700 1 -- 172.21.15.46:0/145903210 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57800c9d0 con 0x7fe5880ca0e0 2024-01-27T01:07:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.641+0000 7fe57d7fa700 1 -- 172.21.15.46:0/145903210 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57800cb30 con 0x7fe5880ca0e0 2024-01-27T01:07:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.641+0000 7fe57d7fa700 1 -- 172.21.15.46:0/145903210 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57800ced0 con 0x7fe5880ca0e0 2024-01-27T01:07:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.641+0000 7fe58f733700 1 -- 172.21.15.46:0/145903210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe588152d00 con 0x7fe5880ca0e0 2024-01-27T01:07:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.641+0000 7fe58f733700 1 -- 172.21.15.46:0/145903210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5880c6a70 con 0x7fe5880ca0e0 2024-01-27T01:07:01.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.642+0000 7fe58f733700 1 -- 172.21.15.46:0/145903210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe58804c710 con 0x7fe5880ca0e0 2024-01-27T01:07:01.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.642+0000 7fe57d7fa700 1 -- 172.21.15.46:0/145903210 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 130) v1 ==== 84964+0+0 (secure 0 0 0) 0x7fe578013680 con 0x7fe5880ca0e0 2024-01-27T01:07:01.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.643+0000 7fe57d7fa700 1 --2- 172.21.15.46:0/145903210 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe56c0671c0 0x7fe56c069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:01.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.643+0000 7fe57d7fa700 1 -- 172.21.15.46:0/145903210 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fe578085440 con 0x7fe5880ca0e0 2024-01-27T01:07:01.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.643+0000 7fe58d4cf700 1 --2- 172.21.15.46:0/145903210 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe56c0671c0 0x7fe56c069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:01.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.644+0000 7fe58d4cf700 1 --2- 172.21.15.46:0/145903210 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe56c0671c0 0x7fe56c069670 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe5840037c0 tx=0x7fe584009040 comp rx=0 tx=0).ready entity=mgr.9485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:01.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.648+0000 7fe57d7fa700 1 -- 172.21.15.46:0/145903210 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe578051c90 con 0x7fe5880ca0e0 2024-01-27T01:07:01.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.830+0000 7fe57d7fa700 1 -- 172.21.15.46:0/145903210 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 131) v1 ==== 85020+0+0 (secure 0 0 0) 0x7fe578052060 con 0x7fe5880ca0e0 2024-01-27T01:07:01.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.884+0000 7fe58f733700 1 -- 172.21.15.46:0/145903210 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fe588155570 con 0x7fe56c0671c0 2024-01-27T01:07:01.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.940+0000 7fe57d7fa700 1 -- 172.21.15.46:0/145903210 <== mgr.9485 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7fe588155570 con 0x7fe56c0671c0 2024-01-27T01:07:01.941 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path mandatory_fs_argument_test_dir 2024-01-27T01:07:01.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.943+0000 7fe58f733700 1 -- 172.21.15.46:0/145903210 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe56c0671c0 msgr2=0x7fe56c069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:01.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.943+0000 7fe58f733700 1 --2- 172.21.15.46:0/145903210 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe56c0671c0 0x7fe56c069670 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe5840037c0 tx=0x7fe584009040 comp rx=0 tx=0).stop 2024-01-27T01:07:01.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.943+0000 7fe58f733700 1 -- 172.21.15.46:0/145903210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5880ca0e0 msgr2=0x7fe5881521e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:01.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.943+0000 7fe58f733700 1 --2- 172.21.15.46:0/145903210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5880ca0e0 0x7fe5881521e0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fe57800ecb0 tx=0x7fe5780091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.943+0000 7fe58f733700 1 -- 172.21.15.46:0/145903210 shutdown_connections 2024-01-27T01:07:01.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.943+0000 7fe58f733700 1 --2- 172.21.15.46:0/145903210 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe56c0671c0 0x7fe56c069670 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.943+0000 7fe58f733700 1 --2- 172.21.15.46:0/145903210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5880ca4b0 0x7fe588150610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.943+0000 7fe58f733700 1 --2- 172.21.15.46:0/145903210 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5880b3a50 0x7fe5880cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.943+0000 7fe58f733700 1 --2- 172.21.15.46:0/145903210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5880ca0e0 0x7fe5881521e0 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:01.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.943+0000 7fe58f733700 1 -- 172.21.15.46:0/145903210 >> 172.21.15.46:0/145903210 conn(0x7fe5880ce530 msgr2=0x7fe588151920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:01.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.943+0000 7fe58f733700 1 -- 172.21.15.46:0/145903210 shutdown_connections 2024-01-27T01:07:01.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:01.943+0000 7fe58f733700 1 -- 172.21.15.46:0/145903210 wait complete. 2024-01-27T01:07:01.958 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path mandatory_fs_argument_test_dir 2024-01-27T01:07:01.959 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:07:01.959 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-01-27T01:07:01.982 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-27T01:07:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.164+0000 7f0f21aa4700 1 Processor -- start 2024-01-27T01:07:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.164+0000 7f0f21aa4700 1 -- start start 2024-01-27T01:07:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.165+0000 7f0f21aa4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f1c0c93a0 0x7f0f1c0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.165+0000 7f0f21aa4700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0f1c0681f0 0x7f0f1c159930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.165+0000 7f0f21aa4700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1c15e390 con 0x7f0f1c067d50 2024-01-27T01:07:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.165+0000 7f0f21aa4700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1c15d810 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.165+0000 7f0f21aa4700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1c15daf0 con 0x7f0f1c0681f0 2024-01-27T01:07:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.165+0000 7f0f1b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f1c0c93a0 0x7f0f1c0c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.165+0000 7f0f1affd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0f1c067d50 0x7f0f1c159220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46256/0 (socket says 172.21.15.46:46256) 2024-01-27T01:07:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.165+0000 7f0f1affd700 1 -- 172.21.15.46:0/925305361 learned_addr learned my addr 172.21.15.46:0/925305361 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.165+0000 7f0f1bfff700 1 --2- 172.21.15.46:0/925305361 >> v2:172.21.15.106:3300/0 conn(0x7f0f1c0681f0 0x7f0f1c159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.165+0000 7f0f1b7fe700 1 -- 172.21.15.46:0/925305361 >> v2:172.21.15.106:3300/0 conn(0x7f0f1c0681f0 msgr2=0x7f0f1c159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.166+0000 7f0f1b7fe700 1 --2- 172.21.15.46:0/925305361 >> v2:172.21.15.106:3300/0 conn(0x7f0f1c0681f0 0x7f0f1c159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.166+0000 7f0f1b7fe700 1 -- 172.21.15.46:0/925305361 >> v1:172.21.15.46:6789/0 conn(0x7f0f1c067d50 legacy=0x7f0f1c159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.166+0000 7f0f1b7fe700 1 -- 172.21.15.46:0/925305361 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c15ddd0 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.166+0000 7f0f1a7fc700 1 -- 172.21.15.46:0/925305361 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233998047 0 0) 0x7f0f1c15e390 con 0x7f0f1c067d50 2024-01-27T01:07:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.166+0000 7f0f1bfff700 1 --2- 172.21.15.46:0/925305361 >> v2:172.21.15.106:3300/0 conn(0x7f0f1c0681f0 0x7f0f1c159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.166+0000 7f0f1b7fe700 1 --2- 172.21.15.46:0/925305361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1c0c93a0 0x7f0f1c0c9770 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0cd300 tx=0x7f0f0c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5215df20ebb58a6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.166+0000 7f0f1a7fc700 1 -- 172.21.15.46:0/925305361 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c00b6b0 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.166+0000 7f0f1a7fc700 1 -- 172.21.15.46:0/925305361 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0c00b810 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.166+0000 7f0f1a7fc700 1 -- 172.21.15.46:0/925305361 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c00bbb0 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.166+0000 7f0f21aa4700 1 -- 172.21.15.46:0/925305361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1c0c93a0 msgr2=0x7f0f1c0c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.166+0000 7f0f21aa4700 1 --2- 172.21.15.46:0/925305361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1c0c93a0 0x7f0f1c0c9770 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0cd300 tx=0x7f0f0c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.167+0000 7f0f21aa4700 1 -- 172.21.15.46:0/925305361 shutdown_connections 2024-01-27T01:07:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.167+0000 7f0f21aa4700 1 --2- 172.21.15.46:0/925305361 >> v2:172.21.15.106:3300/0 conn(0x7f0f1c0681f0 0x7f0f1c159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.167+0000 7f0f21aa4700 1 --2- 172.21.15.46:0/925305361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1c0c93a0 0x7f0f1c0c9770 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.167+0000 7f0f21aa4700 1 -- 172.21.15.46:0/925305361 >> 172.21.15.46:0/925305361 conn(0x7f0f1c0ca9a0 msgr2=0x7f0f1c068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.167+0000 7f0f21aa4700 1 -- 172.21.15.46:0/925305361 shutdown_connections 2024-01-27T01:07:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.167+0000 7f0f21aa4700 1 -- 172.21.15.46:0/925305361 wait complete. 2024-01-27T01:07:02.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.168+0000 7f0f21aa4700 1 Processor -- start 2024-01-27T01:07:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.168+0000 7f0f21aa4700 1 -- start start 2024-01-27T01:07:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.169+0000 7f0f21aa4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f1c067d50 0x7f0f1c1520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.169+0000 7f0f21aa4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f1c0681f0 0x7f0f1c0b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.169+0000 7f0f21aa4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1c0c93a0 0x7f0f1c06c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.169+0000 7f0f21aa4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c159f30 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.169+0000 7f0f21aa4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c16a070 con 0x7f0f1c067d50 2024-01-27T01:07:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.169+0000 7f0f21aa4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c16a350 con 0x7f0f1c0681f0 2024-01-27T01:07:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.169+0000 7f0f1bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1c0c93a0 0x7f0f1c06c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.169+0000 7f0f1bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1c0c93a0 0x7f0f1c06c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59436/0 (socket says 172.21.15.46:59436) 2024-01-27T01:07:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.169+0000 7f0f1bfff700 1 -- 172.21.15.46:0/3211266109 learned_addr learned my addr 172.21.15.46:0/3211266109 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.169+0000 7f0f1affd700 1 --2- 172.21.15.46:0/3211266109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f1c0681f0 0x7f0f1c0b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.169+0000 7f0f1b7fe700 1 --2- 172.21.15.46:0/3211266109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f1c067d50 0x7f0f1c1520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.169+0000 7f0f1bfff700 1 -- 172.21.15.46:0/3211266109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f1c0681f0 msgr2=0x7f0f1c0b2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.169+0000 7f0f1bfff700 1 --2- 172.21.15.46:0/3211266109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f1c0681f0 0x7f0f1c0b2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.169+0000 7f0f1bfff700 1 -- 172.21.15.46:0/3211266109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f1c067d50 msgr2=0x7f0f1c1520d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.169+0000 7f0f1bfff700 1 --2- 172.21.15.46:0/3211266109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f1c067d50 0x7f0f1c1520d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.169+0000 7f0f1bfff700 1 -- 172.21.15.46:0/3211266109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0c009420 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.169+0000 7f0f1bfff700 1 --2- 172.21.15.46:0/3211266109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1c0c93a0 0x7f0f1c06c160 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f0f1400d180 tx=0x7f0f14009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.170+0000 7f0f037fe700 1 -- 172.21.15.46:0/3211266109 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1400ddc0 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.170+0000 7f0f037fe700 1 -- 172.21.15.46:0/3211266109 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1400f040 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.170+0000 7f0f21aa4700 1 -- 172.21.15.46:0/3211266109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1c16a630 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.170+0000 7f0f037fe700 1 -- 172.21.15.46:0/3211266109 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14015650 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.170+0000 7f0f21aa4700 1 -- 172.21.15.46:0/3211266109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1c152610 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.171+0000 7f0f21aa4700 1 -- 172.21.15.46:0/3211266109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1c15a0a0 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.171+0000 7f0f037fe700 1 -- 172.21.15.46:0/3211266109 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 131) v1 ==== 85020+0+0 (secure 0 0 0) 0x7f0f14015a40 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.172+0000 7f0f037fe700 1 --2- 172.21.15.46:0/3211266109 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0f08067250 0x7f0f08069700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.172+0000 7f0f037fe700 1 -- 172.21.15.46:0/3211266109 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f0f14085b90 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.172+0000 7f0f1b7fe700 1 --2- 172.21.15.46:0/3211266109 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0f08067250 0x7f0f08069700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:02.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.172+0000 7f0f1b7fe700 1 --2- 172.21.15.46:0/3211266109 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0f08067250 0x7f0f08069700 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c0093f0 tx=0x7f0f0c017000 comp rx=0 tx=0).ready entity=mgr.9485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:02.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.176+0000 7f0f037fe700 1 -- 172.21.15.46:0/3211266109 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0f14052210 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.438+0000 7f0f21aa4700 1 -- 172.21.15.46:0/3211266109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f0f1c04c710 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.842+0000 7f0f037fe700 1 -- 172.21.15.46:0/3211266109 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 132) v1 ==== 85003+0+0 (secure 0 0 0) 0x7f0f140525e0 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.844+0000 7f0f037fe700 1 -- 172.21.15.46:0/3211266109 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v132) v1 ==== 93+0+0 (secure 0 0 0) 0x7f0f14052210 con 0x7f0f1c0c93a0 2024-01-27T01:07:02.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.846+0000 7f0f21aa4700 1 -- 172.21.15.46:0/3211266109 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0f08067250 msgr2=0x7f0f08069700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:02.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.846+0000 7f0f21aa4700 1 --2- 172.21.15.46:0/3211266109 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0f08067250 0x7f0f08069700 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c0093f0 tx=0x7f0f0c017000 comp rx=0 tx=0).stop 2024-01-27T01:07:02.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.846+0000 7f0f21aa4700 1 -- 172.21.15.46:0/3211266109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1c0c93a0 msgr2=0x7f0f1c06c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:02.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.847+0000 7f0f21aa4700 1 --2- 172.21.15.46:0/3211266109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1c0c93a0 0x7f0f1c06c160 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f0f1400d180 tx=0x7f0f14009a50 comp rx=0 tx=0).stop 2024-01-27T01:07:02.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.847+0000 7f0f21aa4700 1 -- 172.21.15.46:0/3211266109 shutdown_connections 2024-01-27T01:07:02.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.847+0000 7f0f21aa4700 1 --2- 172.21.15.46:0/3211266109 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0f08067250 0x7f0f08069700 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:02.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.847+0000 7f0f21aa4700 1 --2- 172.21.15.46:0/3211266109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1c0c93a0 0x7f0f1c06c160 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:02.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.847+0000 7f0f21aa4700 1 --2- 172.21.15.46:0/3211266109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0f1c067d50 0x7f0f1c1520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:02.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.847+0000 7f0f21aa4700 1 --2- 172.21.15.46:0/3211266109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0f1c0681f0 0x7f0f1c0b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:02.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.847+0000 7f0f21aa4700 1 -- 172.21.15.46:0/3211266109 >> 172.21.15.46:0/3211266109 conn(0x7f0f1c0ca9a0 msgr2=0x7f0f1c06eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:02.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.847+0000 7f0f21aa4700 1 -- 172.21.15.46:0/3211266109 shutdown_connections 2024-01-27T01:07:02.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:02.847+0000 7f0f21aa4700 1 -- 172.21.15.46:0/3211266109 wait complete. 2024-01-27T01:07:02.863 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-27T01:07:02.864 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:07:02.914 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:07:02.953 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-27T01:07:02.953 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-27T01:07:02.953 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-27T01:07:02.954 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-27T01:07:02.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[112010]: fuse finished with error 0 and tester_r 0 2024-01-27T01:07:02.996 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T01:07:02.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:07:02.996 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-27T01:07:03.099 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T01:07:03.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:07:03.100 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-27T01:07:03.116 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-27T01:07:03.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:07:03.118 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T01:07:03.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:07:03.118 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-27T01:07:03.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:07:03.138 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-27T01:07:03.139 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T01:07:03.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:07:03.139 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-27T01:07:03.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:07:03.175 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-27T01:07:03.176 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:07:03.188 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:03.188+0000 7f09caa5b1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:07:03.234 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:03.234+0000 7f8dd1d5c1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:07:03.266 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:03.266+0000 7f09caa5b1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:07:03.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.312+0000 7f8ef4e47700 1 Processor -- start 2024-01-27T01:07:03.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.312+0000 7f8ef4e47700 1 -- start start 2024-01-27T01:07:03.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.313+0000 7f8ef4e47700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8ef00b2b40 0x7f8ef00ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:03.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.314+0000 7f8ef4e47700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ef00aedd0 0x7f8ef00cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:03.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.314+0000 7f8ef4e47700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef015a0b0 con 0x7f8ef00ae960 2024-01-27T01:07:03.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.314+0000 7f8ef4e47700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ef0099fc0 con 0x7f8ef00b2b40 2024-01-27T01:07:03.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.314+0000 7f8ef4e47700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef00ae4d0 con 0x7f8ef00aedd0 2024-01-27T01:07:03.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.314+0000 7f8eedd9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8ef00ae960 0x7f8ef00ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46264/0 (socket says 172.21.15.46:46264) 2024-01-27T01:07:03.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.314+0000 7f8eedd9b700 1 -- 172.21.15.46:0/54240425 learned_addr learned my addr 172.21.15.46:0/54240425 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:03.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.314+0000 7f8eee59c700 1 --2- 172.21.15.46:0/54240425 >> v2:172.21.15.106:3300/0 conn(0x7f8ef00b2b40 0x7f8ef00ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:03.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.314+0000 7f8eeed9d700 1 --2- 172.21.15.46:0/54240425 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ef00aedd0 0x7f8ef00cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:03.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.314+0000 7f8eed59a700 1 -- 172.21.15.46:0/54240425 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392378313 0 0) 0x7f8ef015a0b0 con 0x7f8ef00ae960 2024-01-27T01:07:03.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.314+0000 7f8eed59a700 1 -- 172.21.15.46:0/54240425 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ed8003570 con 0x7f8ef00ae960 2024-01-27T01:07:03.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.314+0000 7f8eeed9d700 1 -- 172.21.15.46:0/54240425 >> v2:172.21.15.106:3300/0 conn(0x7f8ef00b2b40 msgr2=0x7f8ef00ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:03.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.314+0000 7f8eeed9d700 1 --2- 172.21.15.46:0/54240425 >> v2:172.21.15.106:3300/0 conn(0x7f8ef00b2b40 0x7f8ef00ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.314+0000 7f8eeed9d700 1 -- 172.21.15.46:0/54240425 >> v1:172.21.15.46:6789/0 conn(0x7f8ef00ae960 legacy=0x7f8ef00ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:03.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.314+0000 7f8eeed9d700 1 -- 172.21.15.46:0/54240425 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef00ad8c0 con 0x7f8ef00aedd0 2024-01-27T01:07:03.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.314+0000 7f8eee59c700 1 --2- 172.21.15.46:0/54240425 >> v2:172.21.15.106:3300/0 conn(0x7f8ef00b2b40 0x7f8ef00ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:03.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.314+0000 7f8eeed9d700 1 --2- 172.21.15.46:0/54240425 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ef00aedd0 0x7f8ef00cb5e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8009e60 tx=0x7f8ee8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46ceceeacbab1e83 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:03.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.315+0000 7f8eed59a700 1 -- 172.21.15.46:0/54240425 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee800e070 con 0x7f8ef00aedd0 2024-01-27T01:07:03.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.315+0000 7f8eed59a700 1 -- 172.21.15.46:0/54240425 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee800bab0 con 0x7f8ef00aedd0 2024-01-27T01:07:03.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.315+0000 7f8eed59a700 1 -- 172.21.15.46:0/54240425 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee800be70 con 0x7f8ef00aedd0 2024-01-27T01:07:03.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.315+0000 7f8ef4e47700 1 -- 172.21.15.46:0/54240425 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ef00aedd0 msgr2=0x7f8ef00cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:03.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.315+0000 7f8ef4e47700 1 --2- 172.21.15.46:0/54240425 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ef00aedd0 0x7f8ef00cb5e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8009e60 tx=0x7f8ee8009780 comp rx=0 tx=0).stop 2024-01-27T01:07:03.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.315+0000 7f8ef4e47700 1 -- 172.21.15.46:0/54240425 shutdown_connections 2024-01-27T01:07:03.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.315+0000 7f8ef4e47700 1 --2- 172.21.15.46:0/54240425 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ef00aedd0 0x7f8ef00cb5e0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.315+0000 7f8ef4e47700 1 --2- 172.21.15.46:0/54240425 >> v2:172.21.15.106:3300/0 conn(0x7f8ef00b2b40 0x7f8ef00ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.315+0000 7f8ef4e47700 1 -- 172.21.15.46:0/54240425 >> 172.21.15.46:0/54240425 conn(0x7f8ef0099a30 msgr2=0x7f8ef006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:03.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.316+0000 7f8ef4e47700 1 -- 172.21.15.46:0/54240425 shutdown_connections 2024-01-27T01:07:03.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.316+0000 7f8ef4e47700 1 -- 172.21.15.46:0/54240425 wait complete. 2024-01-27T01:07:03.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.317+0000 7f8ef4e47700 1 Processor -- start 2024-01-27T01:07:03.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.317+0000 7f8ef4e47700 1 -- start start 2024-01-27T01:07:03.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.317+0000 7f8ef4e47700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ef00ae960 0x7f8ef0156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:03.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.317+0000 7f8ef4e47700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ef00b2b40 0x7f8ef0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:03.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.318+0000 7f8ef4e47700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef0157410 0x7f8ef00c9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:03.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.318+0000 7f8ef4e47700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef0156d60 con 0x7f8ef0157410 2024-01-27T01:07:03.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.318+0000 7f8ef4e47700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef00c7b70 con 0x7f8ef00b2b40 2024-01-27T01:07:03.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.318+0000 7f8ef4e47700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef00c81d0 con 0x7f8ef00ae960 2024-01-27T01:07:03.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.318+0000 7f8eeed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef0157410 0x7f8ef00c9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:03.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.318+0000 7f8eeed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef0157410 0x7f8ef00c9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59456/0 (socket says 172.21.15.46:59456) 2024-01-27T01:07:03.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.318+0000 7f8eeed9d700 1 -- 172.21.15.46:0/2260259211 learned_addr learned my addr 172.21.15.46:0/2260259211 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:03.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.318+0000 7f8eee59c700 1 --2- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ef00ae960 0x7f8ef0156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:03.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.318+0000 7f8eedd9b700 1 --2- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ef00b2b40 0x7f8ef0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:03.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.318+0000 7f8eeed9d700 1 -- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ef00ae960 msgr2=0x7f8ef0156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:03.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.318+0000 7f8eeed9d700 1 --2- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ef00ae960 0x7f8ef0156820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.318+0000 7f8eeed9d700 1 -- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ef00b2b40 msgr2=0x7f8ef0156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:03.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.318+0000 7f8eeed9d700 1 --2- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ef00b2b40 0x7f8ef0156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.318+0000 7f8eeed9d700 1 -- 172.21.15.46:0/2260259211 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee00093c0 con 0x7f8ef0157410 2024-01-27T01:07:03.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.318+0000 7f8eee59c700 1 --2- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ef00ae960 0x7f8ef0156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:03.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.318+0000 7f8eedd9b700 1 --2- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ef00b2b40 0x7f8ef0156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:03.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.318+0000 7f8eeed9d700 1 --2- 172.21.15.46:0/2260259211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef0157410 0x7f8ef00c9dc0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f8ef00ae610 tx=0x7f8ee8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:03.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.319+0000 7f8ede7fc700 1 -- 172.21.15.46:0/2260259211 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8012dd0 con 0x7f8ef0157410 2024-01-27T01:07:03.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.319+0000 7f8ede7fc700 1 -- 172.21.15.46:0/2260259211 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee8027470 con 0x7f8ef0157410 2024-01-27T01:07:03.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.319+0000 7f8ef4e47700 1 -- 172.21.15.46:0/2260259211 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee8009a20 con 0x7f8ef0157410 2024-01-27T01:07:03.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.319+0000 7f8ede7fc700 1 -- 172.21.15.46:0/2260259211 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8027830 con 0x7f8ef0157410 2024-01-27T01:07:03.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.319+0000 7f8ef4e47700 1 -- 172.21.15.46:0/2260259211 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef00b0510 con 0x7f8ef0157410 2024-01-27T01:07:03.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.320+0000 7f8ef4e47700 1 -- 172.21.15.46:0/2260259211 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef004c710 con 0x7f8ef0157410 2024-01-27T01:07:03.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.320+0000 7f8ede7fc700 1 -- 172.21.15.46:0/2260259211 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 132) v1 ==== 85003+0+0 (secure 0 0 0) 0x7f8ee8024070 con 0x7f8ef0157410 2024-01-27T01:07:03.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.321+0000 7f8ede7fc700 1 --2- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8ed0067200 0x7f8ed00696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:03.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.321+0000 7f8eee59c700 1 -- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8ed0067200 msgr2=0x7f8ed00696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:03.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.321+0000 7f8ede7fc700 1 -- 172.21.15.46:0/2260259211 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f8ee8087a90 con 0x7f8ef0157410 2024-01-27T01:07:03.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.321+0000 7f8eee59c700 1 --2- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8ed0067200 0x7f8ed00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:03.325 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:03.325+0000 7f8dd1d5c1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:07:03.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.326+0000 7f8ede7fc700 1 -- 172.21.15.46:0/2260259211 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8ee8054290 con 0x7f8ef0157410 2024-01-27T01:07:03.337 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:03.337+0000 7f09caa5b1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:07:03.402 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:03.403+0000 7f09caa5b1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:07:03.407 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:03.407+0000 7f8dd1d5c1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:07:03.482 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:03.482+0000 7f8dd1d5c1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:07:03.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.522+0000 7f8eee59c700 1 -- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8ed0067200 msgr2=0x7f8ed00696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:03.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.522+0000 7f8eee59c700 1 --2- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8ed0067200 0x7f8ed00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:03.557 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:03.558+0000 7f09caa5b1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:07:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.583+0000 7f8ef4e47700 1 -- 172.21.15.46:0/2260259211 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ef00682c0 con 0x7f8ef0157410 2024-01-27T01:07:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.584+0000 7f8ede7fc700 1 -- 172.21.15.46:0/2260259211 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5697 (secure 0 0 0) 0x7f8ee80578b0 con 0x7f8ef0157410 2024-01-27T01:07:03.584 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:03.584 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":122,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/1999300009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1999300009},{"type":"v1","addr":"172.21.15.106:6835","nonce":1999300009}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":110},{"gid":10486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/4174263148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4174263148},{"type":"v1","addr":"172.21.15.106:6833","nonce":4174263148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:06:26.286041+0000","modified":"2024-01-27T01:06:29.873402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10866},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10866":{"gid":10866,"name":"a","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2483163449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2483163449},{"type":"v1","addr":"172.21.15.46:6835","nonce":2483163449}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:06:40.353248+0000","modified":"2024-01-27T01:06:43.938885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10854":{"gid":10854,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":8,"addr":"172.21.15.46:6833/1263484152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1263484152},{"type":"v1","addr":"172.21.15.46:6833","nonce":1263484152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-27T01:07:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.586+0000 7f8ef4e47700 1 -- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8ed0067200 msgr2=0x7f8ed00696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.586+0000 7f8ef4e47700 1 --2- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8ed0067200 0x7f8ed00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.586+0000 7f8ef4e47700 1 -- 172.21.15.46:0/2260259211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef0157410 msgr2=0x7f8ef00c9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.586+0000 7f8ef4e47700 1 --2- 172.21.15.46:0/2260259211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef0157410 0x7f8ef00c9dc0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f8ef00ae610 tx=0x7f8ee8019000 comp rx=0 tx=0).stop 2024-01-27T01:07:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.586+0000 7f8ef4e47700 1 -- 172.21.15.46:0/2260259211 shutdown_connections 2024-01-27T01:07:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.586+0000 7f8ef4e47700 1 --2- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8ed0067200 0x7f8ed00696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.586+0000 7f8ef4e47700 1 --2- 172.21.15.46:0/2260259211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef0157410 0x7f8ef00c9dc0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.586+0000 7f8ef4e47700 1 --2- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ef00ae960 0x7f8ef0156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.586+0000 7f8ef4e47700 1 --2- 172.21.15.46:0/2260259211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ef00b2b40 0x7f8ef0156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.586+0000 7f8ef4e47700 1 -- 172.21.15.46:0/2260259211 >> 172.21.15.46:0/2260259211 conn(0x7f8ef0099a30 msgr2=0x7f8ef00c5da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.586+0000 7f8ef4e47700 1 -- 172.21.15.46:0/2260259211 shutdown_connections 2024-01-27T01:07:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.586+0000 7f8ef4e47700 1 -- 172.21.15.46:0/2260259211 wait complete. 2024-01-27T01:07:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 122 2024-01-27T01:07:03.600 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:07:03.663 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:03.663+0000 7f8dd1d5c1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:07:03.694 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:03.694+0000 7f09caa5b1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:07:03.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.734+0000 7fe26ca55700 1 Processor -- start 2024-01-27T01:07:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.734+0000 7fe26ca55700 1 -- start start 2024-01-27T01:07:03.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.735+0000 7fe26ca55700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2680b2b40 0x7fe2680ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:03.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.735+0000 7fe26ca55700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe2680ae960 0x7fe2680ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:03.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.735+0000 7fe26ca55700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26815a0b0 con 0x7fe2680aedd0 2024-01-27T01:07:03.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.735+0000 7fe26ca55700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe268099fc0 con 0x7fe2680ae960 2024-01-27T01:07:03.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.735+0000 7fe26ca55700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2680ae4d0 con 0x7fe2680b2b40 2024-01-27T01:07:03.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.735+0000 7fe266d9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe2680aedd0 0x7fe2680cb5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46280/0 (socket says 172.21.15.46:46280) 2024-01-27T01:07:03.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.735+0000 7fe266d9d700 1 -- 172.21.15.46:0/1909040063 learned_addr learned my addr 172.21.15.46:0/1909040063 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:03.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.736+0000 7fe26659c700 1 --2- 172.21.15.46:0/1909040063 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2680b2b40 0x7fe2680ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:03.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.736+0000 7fe265d9b700 1 --2- 172.21.15.46:0/1909040063 >> v2:172.21.15.106:3300/0 conn(0x7fe2680ae960 0x7fe2680ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:03.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.736+0000 7fe26559a700 1 -- 172.21.15.46:0/1909040063 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543945445 0 0) 0x7fe26815a0b0 con 0x7fe2680aedd0 2024-01-27T01:07:03.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.736+0000 7fe26559a700 1 -- 172.21.15.46:0/1909040063 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe254003570 con 0x7fe2680aedd0 2024-01-27T01:07:03.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.736+0000 7fe26659c700 1 -- 172.21.15.46:0/1909040063 >> v2:172.21.15.106:3300/0 conn(0x7fe2680ae960 msgr2=0x7fe2680ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:03.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.736+0000 7fe26659c700 1 --2- 172.21.15.46:0/1909040063 >> v2:172.21.15.106:3300/0 conn(0x7fe2680ae960 0x7fe2680ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.736+0000 7fe26659c700 1 -- 172.21.15.46:0/1909040063 >> v1:172.21.15.46:6789/0 conn(0x7fe2680aedd0 legacy=0x7fe2680cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:03.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.736+0000 7fe26659c700 1 -- 172.21.15.46:0/1909040063 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2680ad8c0 con 0x7fe2680b2b40 2024-01-27T01:07:03.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.736+0000 7fe265d9b700 1 --2- 172.21.15.46:0/1909040063 >> v2:172.21.15.106:3300/0 conn(0x7fe2680ae960 0x7fe2680ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:03.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.736+0000 7fe26659c700 1 --2- 172.21.15.46:0/1909040063 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2680b2b40 0x7fe2680ad1c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe258009eb0 tx=0x7fe258009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e7324acf90de45d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:03.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.736+0000 7fe26559a700 1 -- 172.21.15.46:0/1909040063 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25800e070 con 0x7fe2680b2b40 2024-01-27T01:07:03.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.736+0000 7fe26559a700 1 -- 172.21.15.46:0/1909040063 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25800bb80 con 0x7fe2680b2b40 2024-01-27T01:07:03.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.736+0000 7fe26559a700 1 -- 172.21.15.46:0/1909040063 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe258012430 con 0x7fe2680b2b40 2024-01-27T01:07:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.737+0000 7fe26ca55700 1 -- 172.21.15.46:0/1909040063 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2680b2b40 msgr2=0x7fe2680ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.737+0000 7fe26ca55700 1 --2- 172.21.15.46:0/1909040063 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2680b2b40 0x7fe2680ad1c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe258009eb0 tx=0x7fe258009780 comp rx=0 tx=0).stop 2024-01-27T01:07:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.737+0000 7fe26ca55700 1 -- 172.21.15.46:0/1909040063 shutdown_connections 2024-01-27T01:07:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.737+0000 7fe26ca55700 1 --2- 172.21.15.46:0/1909040063 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2680b2b40 0x7fe2680ad1c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.737+0000 7fe26ca55700 1 --2- 172.21.15.46:0/1909040063 >> v2:172.21.15.106:3300/0 conn(0x7fe2680ae960 0x7fe2680ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.737+0000 7fe26ca55700 1 -- 172.21.15.46:0/1909040063 >> 172.21.15.46:0/1909040063 conn(0x7fe268099a30 msgr2=0x7fe26806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.737+0000 7fe26ca55700 1 -- 172.21.15.46:0/1909040063 shutdown_connections 2024-01-27T01:07:03.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.737+0000 7fe26ca55700 1 -- 172.21.15.46:0/1909040063 wait complete. 2024-01-27T01:07:03.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.738+0000 7fe26ca55700 1 Processor -- start 2024-01-27T01:07:03.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.738+0000 7fe26ca55700 1 -- start start 2024-01-27T01:07:03.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.739+0000 7fe26ca55700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2680ae960 0x7fe268152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:03.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.739+0000 7fe26ca55700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2680b2b40 0x7fe2680b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:03.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.739+0000 7fe26ca55700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2681592c0 0x7fe2680af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:03.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.739+0000 7fe26ca55700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2680b0730 con 0x7fe2680b2b40 2024-01-27T01:07:03.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.739+0000 7fe26ca55700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2680afb20 con 0x7fe2681592c0 2024-01-27T01:07:03.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.739+0000 7fe26ca55700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2680afe00 con 0x7fe2680ae960 2024-01-27T01:07:03.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.739+0000 7fe26659c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2680ae960 0x7fe268152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:03.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.739+0000 7fe265d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2680b2b40 0x7fe2680b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:03.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.739+0000 7fe26659c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2680ae960 0x7fe268152540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:45542/0 (socket says 172.21.15.46:45542) 2024-01-27T01:07:03.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.739+0000 7fe26659c700 1 -- 172.21.15.46:0/2904921755 learned_addr learned my addr 172.21.15.46:0/2904921755 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:03.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.739+0000 7fe266d9d700 1 --2- 172.21.15.46:0/2904921755 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2681592c0 0x7fe2680af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:03.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.740+0000 7fe26659c700 1 -- 172.21.15.46:0/2904921755 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2681592c0 msgr2=0x7fe2680af1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:03.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.740+0000 7fe26659c700 1 --2- 172.21.15.46:0/2904921755 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2681592c0 0x7fe2680af1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.740+0000 7fe26659c700 1 -- 172.21.15.46:0/2904921755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2680b2b40 msgr2=0x7fe2680b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:03.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.740+0000 7fe26659c700 1 --2- 172.21.15.46:0/2904921755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2680b2b40 0x7fe2680b0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.740+0000 7fe26659c700 1 -- 172.21.15.46:0/2904921755 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe250009420 con 0x7fe2680ae960 2024-01-27T01:07:03.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.740+0000 7fe265d9b700 1 --2- 172.21.15.46:0/2904921755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2680b2b40 0x7fe2680b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:03.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.740+0000 7fe26659c700 1 --2- 172.21.15.46:0/2904921755 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2680ae960 0x7fe268152540 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe25800b610 tx=0x7fe258019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:03.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.740+0000 7fe25e7fc700 1 -- 172.21.15.46:0/2904921755 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe258012c00 con 0x7fe2680ae960 2024-01-27T01:07:03.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.740+0000 7fe25e7fc700 1 -- 172.21.15.46:0/2904921755 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe258012d60 con 0x7fe2680ae960 2024-01-27T01:07:03.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.740+0000 7fe26ca55700 1 -- 172.21.15.46:0/2904921755 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe258009a20 con 0x7fe2680ae960 2024-01-27T01:07:03.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.740+0000 7fe25e7fc700 1 -- 172.21.15.46:0/2904921755 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2580276c0 con 0x7fe2680ae960 2024-01-27T01:07:03.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.740+0000 7fe26ca55700 1 -- 172.21.15.46:0/2904921755 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2680b03c0 con 0x7fe2680ae960 2024-01-27T01:07:03.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.741+0000 7fe26ca55700 1 -- 172.21.15.46:0/2904921755 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26804c710 con 0x7fe2680ae960 2024-01-27T01:07:03.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.742+0000 7fe25e7fc700 1 -- 172.21.15.46:0/2904921755 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 132) v1 ==== 85003+0+0 (secure 0 0 0) 0x7fe258024070 con 0x7fe2680ae960 2024-01-27T01:07:03.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.742+0000 7fe25e7fc700 1 --2- 172.21.15.46:0/2904921755 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe248067200 0x7fe2480696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:03.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.742+0000 7fe25e7fc700 1 -- 172.21.15.46:0/2904921755 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fe2580875c0 con 0x7fe2680ae960 2024-01-27T01:07:03.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.742+0000 7fe265d9b700 1 -- 172.21.15.46:0/2904921755 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe248067200 msgr2=0x7fe2480696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:03.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.742+0000 7fe265d9b700 1 --2- 172.21.15.46:0/2904921755 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe248067200 0x7fe2480696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:03.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.746+0000 7fe25e7fc700 1 -- 172.21.15.46:0/2904921755 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe258053d10 con 0x7fe2680ae960 2024-01-27T01:07:03.760 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:03.761+0000 7f09caa5b1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:07:03.823 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:03.823+0000 7f8dd1d5c1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:07:03.884 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:03.884+0000 7f09caa5b1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:07:03.900 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:03.900+0000 7f8dd1d5c1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:07:03.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.943+0000 7fe265d9b700 1 -- 172.21.15.46:0/2904921755 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe248067200 msgr2=0x7fe2480696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:03.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.943+0000 7fe265d9b700 1 --2- 172.21.15.46:0/2904921755 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe248067200 0x7fe2480696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:03.958 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:03.959+0000 7f09caa5b1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:07:03.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.960+0000 7fe26ca55700 1 -- 172.21.15.46:0/2904921755 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2680682c0 con 0x7fe2680ae960 2024-01-27T01:07:03.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.962+0000 7fe25e7fc700 1 -- 172.21.15.46:0/2904921755 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v228) v1 ==== 74+0+19929 (secure 0 0 0) 0x7fe258057330 con 0x7fe2680ae960 2024-01-27T01:07:03.962 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:03.962 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":228,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:06:57.757011+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:06:24.798791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-01-27T01:06:25.249215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:06:38.156934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:06:39.330686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/3809430766":"2024-01-28T01:06:57.756975+0000","172.21.15.106:6837/62434":"2024-01-28T01:06:57.756975+0000","172.21.15.106:6836/62434":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/524220301":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/4002309435":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/2725250616":"2024-01-28T01:06:57.756975+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:07:03.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.964+0000 7fe26ca55700 1 -- 172.21.15.46:0/2904921755 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe248067200 msgr2=0x7fe2480696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:03.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.964+0000 7fe26ca55700 1 --2- 172.21.15.46:0/2904921755 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe248067200 0x7fe2480696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.964+0000 7fe26ca55700 1 -- 172.21.15.46:0/2904921755 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2680ae960 msgr2=0x7fe268152540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:03.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.964+0000 7fe26ca55700 1 --2- 172.21.15.46:0/2904921755 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2680ae960 0x7fe268152540 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe25800b610 tx=0x7fe258019000 comp rx=0 tx=0).stop 2024-01-27T01:07:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.964+0000 7fe26ca55700 1 -- 172.21.15.46:0/2904921755 shutdown_connections 2024-01-27T01:07:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.964+0000 7fe26ca55700 1 --2- 172.21.15.46:0/2904921755 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe248067200 0x7fe2480696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.964+0000 7fe26ca55700 1 --2- 172.21.15.46:0/2904921755 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2681592c0 0x7fe2680af1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.964+0000 7fe26ca55700 1 --2- 172.21.15.46:0/2904921755 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2680ae960 0x7fe268152540 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.964+0000 7fe26ca55700 1 --2- 172.21.15.46:0/2904921755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2680b2b40 0x7fe2680b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.964+0000 7fe26ca55700 1 -- 172.21.15.46:0/2904921755 >> 172.21.15.46:0/2904921755 conn(0x7fe268099a30 msgr2=0x7fe2680c6050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.964+0000 7fe26ca55700 1 -- 172.21.15.46:0/2904921755 shutdown_connections 2024-01-27T01:07:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:03.964+0000 7fe26ca55700 1 -- 172.21.15.46:0/2904921755 wait complete. 2024-01-27T01:07:03.978 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-27T01:06:24.798791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '222', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2024-01-27T01:06:25.249215+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '223', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-01-27T01:06:38.156934+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2024-01-27T01:06:39.330686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '227', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}}] 2024-01-27T01:07:03.979 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-27T01:07:04.047 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:04.047+0000 7f8dd1d5c1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:07:04.099 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:04.098+0000 7f09caa5b1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:07:04.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.116+0000 7fbe0d99a700 1 Processor -- start 2024-01-27T01:07:04.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.116+0000 7fbe0d99a700 1 -- start start 2024-01-27T01:07:04.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.117+0000 7fbe0d99a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbe081599d0 0x7fbe08159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:04.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.117+0000 7fbe0d99a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbe080ab1c0 0x7fbe080b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:04.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.117+0000 7fbe0d99a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe080ab620 con 0x7fbe0815d8c0 2024-01-27T01:07:04.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.117+0000 7fbe0d99a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe080b1a60 con 0x7fbe080ab1c0 2024-01-27T01:07:04.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.117+0000 7fbe0d99a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe080b0f40 con 0x7fbe081599d0 2024-01-27T01:07:04.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.117+0000 7fbe067fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbe080ab1c0 0x7fbe080b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:04.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.117+0000 7fbe077fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbe0815d8c0 0x7fbe080b20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46288/0 (socket says 172.21.15.46:46288) 2024-01-27T01:07:04.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.117+0000 7fbe077fe700 1 -- 172.21.15.46:0/2834634777 learned_addr learned my addr 172.21.15.46:0/2834634777 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:04.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.117+0000 7fbe06ffd700 1 --2- 172.21.15.46:0/2834634777 >> v2:172.21.15.106:3300/0 conn(0x7fbe081599d0 0x7fbe08159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:04.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.117+0000 7fbe067fc700 1 -- 172.21.15.46:0/2834634777 >> v2:172.21.15.106:3300/0 conn(0x7fbe081599d0 msgr2=0x7fbe08159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:04.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.117+0000 7fbe067fc700 1 --2- 172.21.15.46:0/2834634777 >> v2:172.21.15.106:3300/0 conn(0x7fbe081599d0 0x7fbe08159da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.117+0000 7fbe067fc700 1 -- 172.21.15.46:0/2834634777 >> v1:172.21.15.46:6789/0 conn(0x7fbe0815d8c0 legacy=0x7fbe080b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:04.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.117+0000 7fbe067fc700 1 -- 172.21.15.46:0/2834634777 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe080b11c0 con 0x7fbe080ab1c0 2024-01-27T01:07:04.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.117+0000 7fbe05ffb700 1 -- 172.21.15.46:0/2834634777 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890756066 0 0) 0x7fbe00001490 con 0x7fbe0815d8c0 2024-01-27T01:07:04.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.117+0000 7fbe067fc700 1 --2- 172.21.15.46:0/2834634777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe080ab1c0 0x7fbe080b0a00 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7fbe0808aa50 tx=0x7fbdfc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2cff988dcd2a555 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:04.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.118+0000 7fbe05ffb700 1 -- 172.21.15.46:0/2834634777 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc011630 con 0x7fbe080ab1c0 2024-01-27T01:07:04.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.118+0000 7fbe05ffb700 1 -- 172.21.15.46:0/2834634777 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdfc011790 con 0x7fbe080ab1c0 2024-01-27T01:07:04.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.118+0000 7fbe05ffb700 1 -- 172.21.15.46:0/2834634777 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc011b30 con 0x7fbe080ab1c0 2024-01-27T01:07:04.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.118+0000 7fbe0d99a700 1 -- 172.21.15.46:0/2834634777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe080ab1c0 msgr2=0x7fbe080b0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:04.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.118+0000 7fbe0d99a700 1 --2- 172.21.15.46:0/2834634777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe080ab1c0 0x7fbe080b0a00 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7fbe0808aa50 tx=0x7fbdfc0091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.118+0000 7fbe0d99a700 1 -- 172.21.15.46:0/2834634777 shutdown_connections 2024-01-27T01:07:04.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.118+0000 7fbe0d99a700 1 --2- 172.21.15.46:0/2834634777 >> v2:172.21.15.106:3300/0 conn(0x7fbe081599d0 0x7fbe08159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.118+0000 7fbe0d99a700 1 --2- 172.21.15.46:0/2834634777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe080ab1c0 0x7fbe080b0a00 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.118+0000 7fbe0d99a700 1 -- 172.21.15.46:0/2834634777 >> 172.21.15.46:0/2834634777 conn(0x7fbe0806f840 msgr2=0x7fbe08087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:04.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.118+0000 7fbe0d99a700 1 -- 172.21.15.46:0/2834634777 shutdown_connections 2024-01-27T01:07:04.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.118+0000 7fbe0d99a700 1 -- 172.21.15.46:0/2834634777 wait complete. 2024-01-27T01:07:04.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.120+0000 7fbe0d99a700 1 Processor -- start 2024-01-27T01:07:04.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.120+0000 7fbe0d99a700 1 -- start start 2024-01-27T01:07:04.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.121+0000 7fbe0d99a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe080ab1c0 0x7fbe0809acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:04.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.121+0000 7fbe0d99a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbe081599d0 0x7fbe08158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:04.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.121+0000 7fbe0d99a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe0815d8c0 0x7fbe0816d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:04.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.121+0000 7fbe0d99a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0816ded0 con 0x7fbe0815d8c0 2024-01-27T01:07:04.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.121+0000 7fbe0d99a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0809b210 con 0x7fbe080ab1c0 2024-01-27T01:07:04.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.121+0000 7fbe0d99a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0809bb40 con 0x7fbe081599d0 2024-01-27T01:07:04.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.121+0000 7fbe077fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe0815d8c0 0x7fbe0816d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:04.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.121+0000 7fbe077fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe0815d8c0 0x7fbe0816d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59488/0 (socket says 172.21.15.46:59488) 2024-01-27T01:07:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.121+0000 7fbe077fe700 1 -- 172.21.15.46:0/4108280742 learned_addr learned my addr 172.21.15.46:0/4108280742 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.121+0000 7fbe067fc700 1 --2- 172.21.15.46:0/4108280742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbe081599d0 0x7fbe08158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.121+0000 7fbe06ffd700 1 --2- 172.21.15.46:0/4108280742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe080ab1c0 0x7fbe0809acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.121+0000 7fbe077fe700 1 -- 172.21.15.46:0/4108280742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbe081599d0 msgr2=0x7fbe08158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.121+0000 7fbe077fe700 1 --2- 172.21.15.46:0/4108280742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbe081599d0 0x7fbe08158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.121+0000 7fbe077fe700 1 -- 172.21.15.46:0/4108280742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe080ab1c0 msgr2=0x7fbe0809acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.121+0000 7fbe077fe700 1 --2- 172.21.15.46:0/4108280742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe080ab1c0 0x7fbe0809acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.121+0000 7fbe077fe700 1 -- 172.21.15.46:0/4108280742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdfc009420 con 0x7fbe0815d8c0 2024-01-27T01:07:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.121+0000 7fbe077fe700 1 --2- 172.21.15.46:0/4108280742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe0815d8c0 0x7fbe0816d0d0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fbe0000cbf0 tx=0x7fbe000019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.122+0000 7fbdeeffd700 1 -- 172.21.15.46:0/4108280742 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0000a8b0 con 0x7fbe0815d8c0 2024-01-27T01:07:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.122+0000 7fbdeeffd700 1 -- 172.21.15.46:0/4108280742 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe0000aa10 con 0x7fbe0815d8c0 2024-01-27T01:07:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.122+0000 7fbe0d99a700 1 -- 172.21.15.46:0/4108280742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe0809be20 con 0x7fbe0815d8c0 2024-01-27T01:07:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.122+0000 7fbdeeffd700 1 -- 172.21.15.46:0/4108280742 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0000add0 con 0x7fbe0815d8c0 2024-01-27T01:07:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.122+0000 7fbe0d99a700 1 -- 172.21.15.46:0/4108280742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0809c3e0 con 0x7fbe0815d8c0 2024-01-27T01:07:04.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.123+0000 7fbe0d99a700 1 -- 172.21.15.46:0/4108280742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe0804c710 con 0x7fbe0815d8c0 2024-01-27T01:07:04.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.123+0000 7fbdeeffd700 1 -- 172.21.15.46:0/4108280742 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 132) v1 ==== 85003+0+0 (secure 0 0 0) 0x7fbe00007300 con 0x7fbe0815d8c0 2024-01-27T01:07:04.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.124+0000 7fbdeeffd700 1 --2- 172.21.15.46:0/4108280742 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbdf4067200 0x7fbdf40696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:04.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.124+0000 7fbdeeffd700 1 -- 172.21.15.46:0/4108280742 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fbe00085240 con 0x7fbe0815d8c0 2024-01-27T01:07:04.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.124+0000 7fbe06ffd700 1 -- 172.21.15.46:0/4108280742 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbdf4067200 msgr2=0x7fbdf40696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.124+0000 7fbe06ffd700 1 --2- 172.21.15.46:0/4108280742 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbdf4067200 0x7fbdf40696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:04.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.128+0000 7fbdeeffd700 1 -- 172.21.15.46:0/4108280742 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbe00051a50 con 0x7fbe0815d8c0 2024-01-27T01:07:04.133 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:04.133+0000 7f8dd1d5c1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:07:04.226 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:04.226+0000 7f09caa5b1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:07:04.308 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:04.307+0000 7f8dd1d5c1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:07:04.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.325+0000 7fbe06ffd700 1 -- 172.21.15.46:0/4108280742 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbdf4067200 msgr2=0x7fbdf40696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:04.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.325+0000 7fbe06ffd700 1 --2- 172.21.15.46:0/4108280742 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbdf4067200 0x7fbdf40696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:04.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.380+0000 7fbe0d99a700 1 -- 172.21.15.46:0/4108280742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbe080ab620 con 0x7fbe0815d8c0 2024-01-27T01:07:04.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.380+0000 7fbdeeffd700 1 -- 172.21.15.46:0/4108280742 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v228) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbe080ab620 con 0x7fbe0815d8c0 2024-01-27T01:07:04.380 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-27T01:07:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.382+0000 7fbe0d99a700 1 -- 172.21.15.46:0/4108280742 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbdf4067200 msgr2=0x7fbdf40696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.382+0000 7fbe0d99a700 1 --2- 172.21.15.46:0/4108280742 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbdf4067200 0x7fbdf40696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.382+0000 7fbe0d99a700 1 -- 172.21.15.46:0/4108280742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe0815d8c0 msgr2=0x7fbe0816d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.382+0000 7fbe0d99a700 1 --2- 172.21.15.46:0/4108280742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe0815d8c0 0x7fbe0816d0d0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fbe0000cbf0 tx=0x7fbe000019b0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.382+0000 7fbe0d99a700 1 -- 172.21.15.46:0/4108280742 shutdown_connections 2024-01-27T01:07:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.382+0000 7fbe0d99a700 1 --2- 172.21.15.46:0/4108280742 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbdf4067200 0x7fbdf40696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.382+0000 7fbe0d99a700 1 --2- 172.21.15.46:0/4108280742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe0815d8c0 0x7fbe0816d0d0 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.382+0000 7fbe0d99a700 1 --2- 172.21.15.46:0/4108280742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbe080ab1c0 0x7fbe0809acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.382+0000 7fbe0d99a700 1 --2- 172.21.15.46:0/4108280742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbe081599d0 0x7fbe08158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.382+0000 7fbe0d99a700 1 -- 172.21.15.46:0/4108280742 >> 172.21.15.46:0/4108280742 conn(0x7fbe0806f840 msgr2=0x7fbe0815b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:04.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.383+0000 7fbe0d99a700 1 -- 172.21.15.46:0/4108280742 shutdown_connections 2024-01-27T01:07:04.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.383+0000 7fbe0d99a700 1 -- 172.21.15.46:0/4108280742 wait complete. 2024-01-27T01:07:04.397 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-27T01:07:04.452 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:04.453+0000 7f8dd1d5c1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:07:04.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.532+0000 7fb2c630b700 1 Processor -- start 2024-01-27T01:07:04.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.532+0000 7fb2c630b700 1 -- start start 2024-01-27T01:07:04.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.533+0000 7fb2c630b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb2c00cf050 0x7fb2c00af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:04.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.533+0000 7fb2c630b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c00cc100 con 0x7fb2c0159cb0 2024-01-27T01:07:04.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.533+0000 7fb2c630b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c00cb850 con 0x7fb2c00cf4a0 2024-01-27T01:07:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.533+0000 7fb2c630b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2c00ccca0 con 0x7fb2c00cf050 2024-01-27T01:07:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.533+0000 7fb2bffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb2c0159cb0 0x7fb2c00cc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46300/0 (socket says 172.21.15.46:46300) 2024-01-27T01:07:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.533+0000 7fb2bffff700 1 -- 172.21.15.46:0/1063061704 learned_addr learned my addr 172.21.15.46:0/1063061704 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.533+0000 7fb2bf7fe700 1 --2- 172.21.15.46:0/1063061704 >> v2:172.21.15.106:3300/0 conn(0x7fb2c00cf050 0x7fb2c00af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.534+0000 7fb2beffd700 1 -- 172.21.15.46:0/1063061704 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199025978 0 0) 0x7fb2c00cc100 con 0x7fb2c0159cb0 2024-01-27T01:07:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.534+0000 7fb2beffd700 1 -- 172.21.15.46:0/1063061704 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a8003570 con 0x7fb2c0159cb0 2024-01-27T01:07:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.534+0000 7fb2beffd700 1 -- 172.21.15.46:0/1063061704 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043720935 0 0) 0x7fb2c00cb850 con 0x7fb2c00cf4a0 2024-01-27T01:07:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.534+0000 7fb2beffd700 1 -- 172.21.15.46:0/1063061704 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c00cc100 con 0x7fb2c00cf4a0 2024-01-27T01:07:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.534+0000 7fb2bf7fe700 1 -- 172.21.15.46:0/1063061704 >> v1:172.21.15.106:6789/0 conn(0x7fb2c00cf4a0 legacy=0x7fb2c00c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.534+0000 7fb2bf7fe700 1 -- 172.21.15.46:0/1063061704 >> v1:172.21.15.46:6789/0 conn(0x7fb2c0159cb0 legacy=0x7fb2c00cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.534+0000 7fb2bf7fe700 1 -- 172.21.15.46:0/1063061704 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c00cbaf0 con 0x7fb2c00cf050 2024-01-27T01:07:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.534+0000 7fb2bf7fe700 1 --2- 172.21.15.46:0/1063061704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2c00cf050 0x7fb2c00af430 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fb2b40098e0 tx=0x7fb2b40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=273583f6b7fff445 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.534+0000 7fb2beffd700 1 -- 172.21.15.46:0/1063061704 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4011630 con 0x7fb2c00cf050 2024-01-27T01:07:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.534+0000 7fb2beffd700 1 -- 172.21.15.46:0/1063061704 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b4011790 con 0x7fb2c00cf050 2024-01-27T01:07:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.534+0000 7fb2beffd700 1 -- 172.21.15.46:0/1063061704 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4011b30 con 0x7fb2c00cf050 2024-01-27T01:07:04.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.535+0000 7fb2c630b700 1 -- 172.21.15.46:0/1063061704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2c00cf050 msgr2=0x7fb2c00af430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:04.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.535+0000 7fb2c630b700 1 --2- 172.21.15.46:0/1063061704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2c00cf050 0x7fb2c00af430 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fb2b40098e0 tx=0x7fb2b40091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.535+0000 7fb2c630b700 1 -- 172.21.15.46:0/1063061704 shutdown_connections 2024-01-27T01:07:04.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.535+0000 7fb2c630b700 1 --2- 172.21.15.46:0/1063061704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2c00cf050 0x7fb2c00af430 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.535+0000 7fb2c630b700 1 -- 172.21.15.46:0/1063061704 >> 172.21.15.46:0/1063061704 conn(0x7fb2c0087f60 msgr2=0x7fb2c006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:04.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.535+0000 7fb2c630b700 1 -- 172.21.15.46:0/1063061704 shutdown_connections 2024-01-27T01:07:04.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.535+0000 7fb2c630b700 1 -- 172.21.15.46:0/1063061704 wait complete. 2024-01-27T01:07:04.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.536+0000 7fb2c630b700 1 Processor -- start 2024-01-27T01:07:04.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.537+0000 7fb2c630b700 1 -- start start 2024-01-27T01:07:04.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.537+0000 7fb2c630b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c00cf050 0x7fb2c009dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:04.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.537+0000 7fb2c630b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2c00cf4a0 0x7fb2c0158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:04.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.537+0000 7fb2c630b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2c0159cb0 0x7fb2c0152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:04.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.537+0000 7fb2c630b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c016aeb0 con 0x7fb2c00cf050 2024-01-27T01:07:04.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.537+0000 7fb2c630b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c016a2a0 con 0x7fb2c00cf4a0 2024-01-27T01:07:04.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.537+0000 7fb2c630b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c016a580 con 0x7fb2c0159cb0 2024-01-27T01:07:04.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.537+0000 7fb2bffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c00cf050 0x7fb2c009dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:04.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.537+0000 7fb2bffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c00cf050 0x7fb2c009dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59502/0 (socket says 172.21.15.46:59502) 2024-01-27T01:07:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.537+0000 7fb2bffff700 1 -- 172.21.15.46:0/3834646733 learned_addr learned my addr 172.21.15.46:0/3834646733 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.538+0000 7fb2bf7fe700 1 --2- 172.21.15.46:0/3834646733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2c00cf4a0 0x7fb2c0158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.538+0000 7fb2c48a8700 1 --2- 172.21.15.46:0/3834646733 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2c0159cb0 0x7fb2c0152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.538+0000 7fb2bffff700 1 -- 172.21.15.46:0/3834646733 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2c0159cb0 msgr2=0x7fb2c0152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.538+0000 7fb2bffff700 1 --2- 172.21.15.46:0/3834646733 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2c0159cb0 0x7fb2c0152230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.538+0000 7fb2bffff700 1 -- 172.21.15.46:0/3834646733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2c00cf4a0 msgr2=0x7fb2c0158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.538+0000 7fb2bffff700 1 --2- 172.21.15.46:0/3834646733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2c00cf4a0 0x7fb2c0158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.538+0000 7fb2bffff700 1 -- 172.21.15.46:0/3834646733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b4009420 con 0x7fb2c00cf050 2024-01-27T01:07:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.538+0000 7fb2bf7fe700 1 --2- 172.21.15.46:0/3834646733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2c00cf4a0 0x7fb2c0158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.538+0000 7fb2bffff700 1 --2- 172.21.15.46:0/3834646733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c00cf050 0x7fb2c009dd20 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fb2b000c790 tx=0x7fb2b0007cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.538+0000 7fb2a7fff700 1 -- 172.21.15.46:0/3834646733 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b00072e0 con 0x7fb2c00cf050 2024-01-27T01:07:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.538+0000 7fb2a7fff700 1 -- 172.21.15.46:0/3834646733 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b0007460 con 0x7fb2c00cf050 2024-01-27T01:07:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.538+0000 7fb2c630b700 1 -- 172.21.15.46:0/3834646733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c016a860 con 0x7fb2c00cf050 2024-01-27T01:07:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.538+0000 7fb2a7fff700 1 -- 172.21.15.46:0/3834646733 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b00057f0 con 0x7fb2c00cf050 2024-01-27T01:07:04.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.538+0000 7fb2c630b700 1 -- 172.21.15.46:0/3834646733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c0152db0 con 0x7fb2c00cf050 2024-01-27T01:07:04.541 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:04.539+0000 7f09caa5b1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:07:04.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.540+0000 7fb2c630b700 1 -- 172.21.15.46:0/3834646733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c016ad00 con 0x7fb2c00cf050 2024-01-27T01:07:04.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.540+0000 7fb2a7fff700 1 -- 172.21.15.46:0/3834646733 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 132) v1 ==== 85003+0+0 (secure 0 0 0) 0x7fb2b0005c20 con 0x7fb2c00cf050 2024-01-27T01:07:04.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.540+0000 7fb2a7fff700 1 --2- 172.21.15.46:0/3834646733 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb2ac067200 0x7fb2ac0696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:04.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.540+0000 7fb2a7fff700 1 -- 172.21.15.46:0/3834646733 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fb2b0084430 con 0x7fb2c00cf050 2024-01-27T01:07:04.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.540+0000 7fb2bf7fe700 1 -- 172.21.15.46:0/3834646733 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb2ac067200 msgr2=0x7fb2ac0696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:04.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.540+0000 7fb2bf7fe700 1 --2- 172.21.15.46:0/3834646733 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb2ac067200 0x7fb2ac0696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:04.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.549+0000 7fb2a7fff700 1 -- 172.21.15.46:0/3834646733 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb2b004b5a0 con 0x7fb2c00cf050 2024-01-27T01:07:04.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.741+0000 7fb2bf7fe700 1 -- 172.21.15.46:0/3834646733 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb2ac067200 msgr2=0x7fb2ac0696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:04.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.741+0000 7fb2bf7fe700 1 --2- 172.21.15.46:0/3834646733 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb2ac067200 0x7fb2ac0696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:04.748 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:04.749+0000 7f09caa5b1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:07:04.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.755+0000 7fb2c630b700 1 -- 172.21.15.46:0/3834646733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb2c00cfc80 con 0x7fb2c00cf050 2024-01-27T01:07:04.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.755+0000 7fb2a7fff700 1 -- 172.21.15.46:0/3834646733 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v228) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb2b0050c30 con 0x7fb2c00cf050 2024-01-27T01:07:04.755 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-27T01:07:04.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.758+0000 7fb2c630b700 1 -- 172.21.15.46:0/3834646733 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb2ac067200 msgr2=0x7fb2ac0696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:04.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.758+0000 7fb2c630b700 1 --2- 172.21.15.46:0/3834646733 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb2ac067200 0x7fb2ac0696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.758+0000 7fb2c630b700 1 -- 172.21.15.46:0/3834646733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c00cf050 msgr2=0x7fb2c009dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:04.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.758+0000 7fb2c630b700 1 --2- 172.21.15.46:0/3834646733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c00cf050 0x7fb2c009dd20 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fb2b000c790 tx=0x7fb2b0007cc0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.758+0000 7fb2c630b700 1 -- 172.21.15.46:0/3834646733 shutdown_connections 2024-01-27T01:07:04.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.758+0000 7fb2c630b700 1 --2- 172.21.15.46:0/3834646733 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb2ac067200 0x7fb2ac0696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.758+0000 7fb2c630b700 1 --2- 172.21.15.46:0/3834646733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c00cf050 0x7fb2c009dd20 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.758+0000 7fb2c630b700 1 --2- 172.21.15.46:0/3834646733 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2c0159cb0 0x7fb2c0152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.758+0000 7fb2c630b700 1 --2- 172.21.15.46:0/3834646733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2c00cf4a0 0x7fb2c0158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.758+0000 7fb2c630b700 1 -- 172.21.15.46:0/3834646733 >> 172.21.15.46:0/3834646733 conn(0x7fb2c0087f60 msgr2=0x7fb2c01556e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:04.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.758+0000 7fb2c630b700 1 -- 172.21.15.46:0/3834646733 shutdown_connections 2024-01-27T01:07:04.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.758+0000 7fb2c630b700 1 -- 172.21.15.46:0/3834646733 wait complete. 2024-01-27T01:07:04.773 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-27T01:07:04.830 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:04.830+0000 7f8dd1d5c1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:07:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.909+0000 7f587c7d1700 1 Processor -- start 2024-01-27T01:07:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.909+0000 7f587c7d1700 1 -- start start 2024-01-27T01:07:04.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.910+0000 7f587c7d1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5874068940 0x7f5874068d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:04.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.910+0000 7f587c7d1700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f58740c90c0 0x7f58740af990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:04.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.910+0000 7f587c7d1700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58740c87c0 con 0x7f58740c8c50 2024-01-27T01:07:04.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.910+0000 7f587c7d1700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5874069250 con 0x7f58740c90c0 2024-01-27T01:07:04.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.910+0000 7f587c7d1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58740c7bb0 con 0x7f5874068940 2024-01-27T01:07:04.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.911+0000 7f587ad6e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f58740c90c0 0x7f58740af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.911+0000 7f587a56d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5874068940 0x7f5874068d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.911+0000 7f5879d6c700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f58740c8c50 0x7f58740c7160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:56478/0 (socket says 172.21.15.46:56478) 2024-01-27T01:07:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.911+0000 7f5879d6c700 1 -- 172.21.15.46:0/1149662497 learned_addr learned my addr 172.21.15.46:0/1149662497 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.911+0000 7f587ad6e700 1 -- 172.21.15.46:0/1149662497 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5874068940 msgr2=0x7f5874068d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.911+0000 7f587ad6e700 1 --2- 172.21.15.46:0/1149662497 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5874068940 0x7f5874068d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.911+0000 7f587ad6e700 1 -- 172.21.15.46:0/1149662497 >> v1:172.21.15.106:6789/0 conn(0x7f58740c8c50 legacy=0x7f58740c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.911+0000 7f587ad6e700 1 -- 172.21.15.46:0/1149662497 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58740c7e90 con 0x7f58740c90c0 2024-01-27T01:07:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.911+0000 7f587ad6e700 1 --2- 172.21.15.46:0/1149662497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58740c90c0 0x7f58740af990 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f586c009e60 tx=0x7f586c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fbfc500af429fe server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.911+0000 7f587956b700 1 -- 172.21.15.46:0/1149662497 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586c00e070 con 0x7f58740c90c0 2024-01-27T01:07:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.911+0000 7f587956b700 1 -- 172.21.15.46:0/1149662497 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586c00bb60 con 0x7f58740c90c0 2024-01-27T01:07:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.911+0000 7f587956b700 1 -- 172.21.15.46:0/1149662497 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586c012430 con 0x7f58740c90c0 2024-01-27T01:07:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.912+0000 7f587c7d1700 1 -- 172.21.15.46:0/1149662497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58740c90c0 msgr2=0x7f58740af990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.912+0000 7f587c7d1700 1 --2- 172.21.15.46:0/1149662497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58740c90c0 0x7f58740af990 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f586c009e60 tx=0x7f586c009780 comp rx=0 tx=0).stop 2024-01-27T01:07:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.912+0000 7f587c7d1700 1 -- 172.21.15.46:0/1149662497 shutdown_connections 2024-01-27T01:07:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.912+0000 7f587c7d1700 1 --2- 172.21.15.46:0/1149662497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58740c90c0 0x7f58740af990 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.912+0000 7f587c7d1700 1 --2- 172.21.15.46:0/1149662497 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5874068940 0x7f5874068d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.912+0000 7f587c7d1700 1 -- 172.21.15.46:0/1149662497 >> 172.21.15.46:0/1149662497 conn(0x7f58740ca100 msgr2=0x7f587406e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.912+0000 7f587c7d1700 1 -- 172.21.15.46:0/1149662497 shutdown_connections 2024-01-27T01:07:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.912+0000 7f587c7d1700 1 -- 172.21.15.46:0/1149662497 wait complete. 2024-01-27T01:07:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.913+0000 7f587c7d1700 1 Processor -- start 2024-01-27T01:07:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.913+0000 7f587c7d1700 1 -- start start 2024-01-27T01:07:04.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.914+0000 7f587c7d1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5874068940 0x7f58741699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:04.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.914+0000 7f587c7d1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58740c8c50 0x7f587416d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:04.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.914+0000 7f587c7d1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58740c90c0 0x7f5874159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:04.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.914+0000 7f587c7d1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5874163ee0 con 0x7f5874068940 2024-01-27T01:07:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.914+0000 7f587c7d1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587406bde0 con 0x7f58740c90c0 2024-01-27T01:07:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.914+0000 7f587c7d1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f587415a210 con 0x7f58740c8c50 2024-01-27T01:07:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.914+0000 7f587a56d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5874068940 0x7f58741699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.914+0000 7f5879d6c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58740c8c50 0x7f587416d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.914+0000 7f587ad6e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58740c90c0 0x7f5874159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.914+0000 7f587a56d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5874068940 0x7f58741699b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59514/0 (socket says 172.21.15.46:59514) 2024-01-27T01:07:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.914+0000 7f587a56d700 1 -- 172.21.15.46:0/3130920622 learned_addr learned my addr 172.21.15.46:0/3130920622 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.915+0000 7f587a56d700 1 -- 172.21.15.46:0/3130920622 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58740c8c50 msgr2=0x7f587416d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.915+0000 7f587a56d700 1 --2- 172.21.15.46:0/3130920622 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58740c8c50 0x7f587416d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.915+0000 7f587a56d700 1 -- 172.21.15.46:0/3130920622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58740c90c0 msgr2=0x7f5874159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.915+0000 7f587a56d700 1 --2- 172.21.15.46:0/3130920622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58740c90c0 0x7f5874159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.915+0000 7f587a56d700 1 -- 172.21.15.46:0/3130920622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586c009a20 con 0x7f5874068940 2024-01-27T01:07:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.915+0000 7f587ad6e700 1 --2- 172.21.15.46:0/3130920622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58740c90c0 0x7f5874159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.915+0000 7f587a56d700 1 --2- 172.21.15.46:0/3130920622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5874068940 0x7f58741699b0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f586800ec80 tx=0x7f5868009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.915+0000 7f58667fc700 1 -- 172.21.15.46:0/3130920622 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586800c850 con 0x7f5874068940 2024-01-27T01:07:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.915+0000 7f58667fc700 1 -- 172.21.15.46:0/3130920622 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586800c9b0 con 0x7f5874068940 2024-01-27T01:07:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.915+0000 7f587c7d1700 1 -- 172.21.15.46:0/3130920622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f587406a4f0 con 0x7f5874068940 2024-01-27T01:07:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.915+0000 7f58667fc700 1 -- 172.21.15.46:0/3130920622 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586800cd30 con 0x7f5874068940 2024-01-27T01:07:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.915+0000 7f587c7d1700 1 -- 172.21.15.46:0/3130920622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f587416a2a0 con 0x7f5874068940 2024-01-27T01:07:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.916+0000 7f587c7d1700 1 -- 172.21.15.46:0/3130920622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f587404c710 con 0x7f5874068940 2024-01-27T01:07:04.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.917+0000 7f58667fc700 1 -- 172.21.15.46:0/3130920622 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 132) v1 ==== 85003+0+0 (secure 0 0 0) 0x7f5868013680 con 0x7f5874068940 2024-01-27T01:07:04.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.917+0000 7f58667fc700 1 --2- 172.21.15.46:0/3130920622 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5858067200 0x7f58580696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:04.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.917+0000 7f5879d6c700 1 -- 172.21.15.46:0/3130920622 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5858067200 msgr2=0x7f58580696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:04.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.917+0000 7f58667fc700 1 -- 172.21.15.46:0/3130920622 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f5868085730 con 0x7f5874068940 2024-01-27T01:07:04.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.917+0000 7f5879d6c700 1 --2- 172.21.15.46:0/3130920622 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5858067200 0x7f58580696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:04.922+0000 7f58667fc700 1 -- 172.21.15.46:0/3130920622 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5868051e80 con 0x7f5874068940 2024-01-27T01:07:04.979 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:04.979+0000 7f09caa5b1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:07:05.049 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:05.049+0000 7f09caa5b1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:07:05.079 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:05.079+0000 7f8dd1d5c1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:07:05.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.118+0000 7f5879d6c700 1 -- 172.21.15.46:0/3130920622 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5858067200 msgr2=0x7f58580696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:05.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.118+0000 7f5879d6c700 1 --2- 172.21.15.46:0/3130920622 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5858067200 0x7f58580696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:05.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.127+0000 7f587c7d1700 1 -- 172.21.15.46:0/3130920622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f587406c9c0 con 0x7f5874068940 2024-01-27T01:07:05.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.127+0000 7f58667fc700 1 -- 172.21.15.46:0/3130920622 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v228) v1 ==== 100+0+11 (secure 0 0 0) 0x7f58680554a0 con 0x7f5874068940 2024-01-27T01:07:05.127 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-27T01:07:05.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.129+0000 7f587c7d1700 1 -- 172.21.15.46:0/3130920622 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5858067200 msgr2=0x7f58580696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:05.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.129+0000 7f587c7d1700 1 --2- 172.21.15.46:0/3130920622 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5858067200 0x7f58580696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.129+0000 7f587c7d1700 1 -- 172.21.15.46:0/3130920622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5874068940 msgr2=0x7f58741699b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:05.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.129+0000 7f587c7d1700 1 --2- 172.21.15.46:0/3130920622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5874068940 0x7f58741699b0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f586800ec80 tx=0x7f5868009120 comp rx=0 tx=0).stop 2024-01-27T01:07:05.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.129+0000 7f587c7d1700 1 -- 172.21.15.46:0/3130920622 shutdown_connections 2024-01-27T01:07:05.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.129+0000 7f587c7d1700 1 --2- 172.21.15.46:0/3130920622 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5858067200 0x7f58580696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.129+0000 7f587c7d1700 1 --2- 172.21.15.46:0/3130920622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5874068940 0x7f58741699b0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.129+0000 7f587c7d1700 1 --2- 172.21.15.46:0/3130920622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f58740c90c0 0x7f5874159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.129+0000 7f587c7d1700 1 --2- 172.21.15.46:0/3130920622 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f58740c8c50 0x7f587416d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.129+0000 7f587c7d1700 1 -- 172.21.15.46:0/3130920622 >> 172.21.15.46:0/3130920622 conn(0x7f58740ca100 msgr2=0x7f58740b1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:05.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.129+0000 7f587c7d1700 1 -- 172.21.15.46:0/3130920622 shutdown_connections 2024-01-27T01:07:05.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.130+0000 7f587c7d1700 1 -- 172.21.15.46:0/3130920622 wait complete. 2024-01-27T01:07:05.144 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-01-27T01:07:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.283+0000 7fc2bd893700 1 Processor -- start 2024-01-27T01:07:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.283+0000 7fc2bd893700 1 -- start start 2024-01-27T01:07:05.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.284+0000 7fc2bd893700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc2b8159da0 0x7fc2b80ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:05.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.284+0000 7fc2bd893700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2b815d8c0 0x7fc2b815dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:05.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.284+0000 7fc2bd893700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2b815a200 con 0x7fc2b81599d0 2024-01-27T01:07:05.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.284+0000 7fc2bd893700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b80b1a60 con 0x7fc2b8159da0 2024-01-27T01:07:05.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.284+0000 7fc2bd893700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b815e230 con 0x7fc2b815d8c0 2024-01-27T01:07:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.284+0000 7fc2b6ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc2b81599d0 0x7fc2b80b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46304/0 (socket says 172.21.15.46:46304) 2024-01-27T01:07:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.284+0000 7fc2b6ffd700 1 -- 172.21.15.46:0/2682786297 learned_addr learned my addr 172.21.15.46:0/2682786297 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.285+0000 7fc2b67fc700 1 --2- 172.21.15.46:0/2682786297 >> v2:172.21.15.106:3300/0 conn(0x7fc2b8159da0 0x7fc2b80ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.285+0000 7fc2b77fe700 1 --2- 172.21.15.46:0/2682786297 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2b815d8c0 0x7fc2b815dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.285+0000 7fc2b5ffb700 1 -- 172.21.15.46:0/2682786297 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250229 0 0) 0x7fc2b815a200 con 0x7fc2b81599d0 2024-01-27T01:07:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.285+0000 7fc2b5ffb700 1 -- 172.21.15.46:0/2682786297 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a0003570 con 0x7fc2b81599d0 2024-01-27T01:07:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.285+0000 7fc2b67fc700 1 -- 172.21.15.46:0/2682786297 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2b815d8c0 msgr2=0x7fc2b815dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.285+0000 7fc2b67fc700 1 --2- 172.21.15.46:0/2682786297 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2b815d8c0 0x7fc2b815dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.285+0000 7fc2b67fc700 1 -- 172.21.15.46:0/2682786297 >> v1:172.21.15.46:6789/0 conn(0x7fc2b81599d0 legacy=0x7fc2b80b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.285+0000 7fc2b67fc700 1 -- 172.21.15.46:0/2682786297 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b80b11c0 con 0x7fc2b8159da0 2024-01-27T01:07:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.285+0000 7fc2b67fc700 1 --2- 172.21.15.46:0/2682786297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc2b8159da0 0x7fc2b80ab1c0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0098e0 tx=0x7fc2ac0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cc7ead23ef407ef server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.285+0000 7fc2b5ffb700 1 -- 172.21.15.46:0/2682786297 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac011630 con 0x7fc2b8159da0 2024-01-27T01:07:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.285+0000 7fc2b5ffb700 1 -- 172.21.15.46:0/2682786297 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ac011790 con 0x7fc2b8159da0 2024-01-27T01:07:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.285+0000 7fc2b5ffb700 1 -- 172.21.15.46:0/2682786297 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac011b30 con 0x7fc2b8159da0 2024-01-27T01:07:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.286+0000 7fc2bd893700 1 -- 172.21.15.46:0/2682786297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc2b8159da0 msgr2=0x7fc2b80ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:05.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.286+0000 7fc2bd893700 1 --2- 172.21.15.46:0/2682786297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc2b8159da0 0x7fc2b80ab1c0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0098e0 tx=0x7fc2ac0091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.286+0000 7fc2bd893700 1 -- 172.21.15.46:0/2682786297 shutdown_connections 2024-01-27T01:07:05.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.286+0000 7fc2bd893700 1 --2- 172.21.15.46:0/2682786297 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2b815d8c0 0x7fc2b815dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.286+0000 7fc2bd893700 1 --2- 172.21.15.46:0/2682786297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc2b8159da0 0x7fc2b80ab1c0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.286+0000 7fc2bd893700 1 -- 172.21.15.46:0/2682786297 >> 172.21.15.46:0/2682786297 conn(0x7fc2b806f840 msgr2=0x7fc2b8087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:05.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.286+0000 7fc2bd893700 1 -- 172.21.15.46:0/2682786297 shutdown_connections 2024-01-27T01:07:05.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.286+0000 7fc2bd893700 1 -- 172.21.15.46:0/2682786297 wait complete. 2024-01-27T01:07:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.287+0000 7fc2bd893700 1 Processor -- start 2024-01-27T01:07:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.288+0000 7fc2bd893700 1 -- start start 2024-01-27T01:07:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.288+0000 7fc2bd893700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc2b81599d0 0x7fc2b8147c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.288+0000 7fc2bd893700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2b8159da0 0x7fc2b809bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.288+0000 7fc2bd893700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2b815d8c0 0x7fc2b80aea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.288+0000 7fc2bd893700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b80ae5f0 con 0x7fc2b815d8c0 2024-01-27T01:07:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.288+0000 7fc2bd893700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b80ada20 con 0x7fc2b81599d0 2024-01-27T01:07:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.288+0000 7fc2bd893700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b80adcc0 con 0x7fc2b8159da0 2024-01-27T01:07:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.288+0000 7fc2b6ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc2b81599d0 0x7fc2b8147c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.288+0000 7fc2b77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2b815d8c0 0x7fc2b80aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:05.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.288+0000 7fc2b67fc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2b8159da0 0x7fc2b809bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:05.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.289+0000 7fc2b6ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc2b81599d0 0x7fc2b8147c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:52280/0 (socket says 172.21.15.46:52280) 2024-01-27T01:07:05.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.289+0000 7fc2b6ffd700 1 -- 172.21.15.46:0/125174982 learned_addr learned my addr 172.21.15.46:0/125174982 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:05.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.289+0000 7fc2b77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2b815d8c0 0x7fc2b80aea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59520/0 (socket says 172.21.15.46:59520) 2024-01-27T01:07:05.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.289+0000 7fc2b6ffd700 1 -- 172.21.15.46:0/125174982 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2b8159da0 msgr2=0x7fc2b809bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:05.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.289+0000 7fc2b6ffd700 1 --2- 172.21.15.46:0/125174982 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2b8159da0 0x7fc2b809bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.289+0000 7fc2b6ffd700 1 -- 172.21.15.46:0/125174982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2b815d8c0 msgr2=0x7fc2b80aea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:05.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.289+0000 7fc2b6ffd700 1 --2- 172.21.15.46:0/125174982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2b815d8c0 0x7fc2b80aea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.289+0000 7fc2b6ffd700 1 -- 172.21.15.46:0/125174982 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ac009420 con 0x7fc2b81599d0 2024-01-27T01:07:05.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.289+0000 7fc2b67fc700 1 --2- 172.21.15.46:0/125174982 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2b8159da0 0x7fc2b809bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:05.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.289+0000 7fc2b77fe700 1 --2- 172.21.15.46:0/125174982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2b815d8c0 0x7fc2b80aea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:05.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.289+0000 7fc2b6ffd700 1 --2- 172.21.15.46:0/125174982 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc2b81599d0 0x7fc2b8147c70 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800cb10 tx=0x7fc2a80027e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:05.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.289+0000 7fc29effd700 1 -- 172.21.15.46:0/125174982 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a800ce20 con 0x7fc2b81599d0 2024-01-27T01:07:05.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.289+0000 7fc29effd700 1 -- 172.21.15.46:0/125174982 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a8005d80 con 0x7fc2b81599d0 2024-01-27T01:07:05.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.289+0000 7fc2bd893700 1 -- 172.21.15.46:0/125174982 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b80adfa0 con 0x7fc2b81599d0 2024-01-27T01:07:05.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.289+0000 7fc29effd700 1 -- 172.21.15.46:0/125174982 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a800a7e0 con 0x7fc2b81599d0 2024-01-27T01:07:05.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.289+0000 7fc2bd893700 1 -- 172.21.15.46:0/125174982 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b81481b0 con 0x7fc2b81599d0 2024-01-27T01:07:05.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.290+0000 7fc2bd893700 1 -- 172.21.15.46:0/125174982 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2b804c710 con 0x7fc2b81599d0 2024-01-27T01:07:05.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.291+0000 7fc29effd700 1 -- 172.21.15.46:0/125174982 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 132) v1 ==== 85003+0+0 (secure 0 0 0) 0x7fc2a800ac10 con 0x7fc2b81599d0 2024-01-27T01:07:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.291+0000 7fc29effd700 1 --2- 172.21.15.46:0/125174982 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc2a4067200 0x7fc2a40696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.291+0000 7fc2b67fc700 1 -- 172.21.15.46:0/125174982 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc2a4067200 msgr2=0x7fc2a40696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.291+0000 7fc29effd700 1 -- 172.21.15.46:0/125174982 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fc2a8084e20 con 0x7fc2b81599d0 2024-01-27T01:07:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.291+0000 7fc2b67fc700 1 --2- 172.21.15.46:0/125174982 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc2a4067200 0x7fc2a40696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:05.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.300+0000 7fc29effd700 1 -- 172.21.15.46:0/125174982 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc2a8051630 con 0x7fc2b81599d0 2024-01-27T01:07:05.346 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:05.346+0000 7f8dd1d5c1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:07:05.428 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:05.428+0000 7f8dd1d5c1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:07:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.492+0000 7fc2b67fc700 1 -- 172.21.15.46:0/125174982 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc2a4067200 msgr2=0x7fc2a40696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.492+0000 7fc2b67fc700 1 --2- 172.21.15.46:0/125174982 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc2a4067200 0x7fc2a40696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:05.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.509+0000 7fc2bd893700 1 -- 172.21.15.46:0/125174982 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc2b80d0560 con 0x7fc2b81599d0 2024-01-27T01:07:05.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.509+0000 7fc29effd700 1 -- 172.21.15.46:0/125174982 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v228) v1 ==== 107+0+11 (secure 0 0 0) 0x7fc2a8054c50 con 0x7fc2b81599d0 2024-01-27T01:07:05.509 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-27T01:07:05.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.512+0000 7fc2bd893700 1 -- 172.21.15.46:0/125174982 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc2a4067200 msgr2=0x7fc2a40696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:05.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.512+0000 7fc2bd893700 1 --2- 172.21.15.46:0/125174982 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc2a4067200 0x7fc2a40696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.512+0000 7fc2bd893700 1 -- 172.21.15.46:0/125174982 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc2b81599d0 msgr2=0x7fc2b8147c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:05.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.512+0000 7fc2bd893700 1 --2- 172.21.15.46:0/125174982 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc2b81599d0 0x7fc2b8147c70 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800cb10 tx=0x7fc2a80027e0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.512+0000 7fc2bd893700 1 -- 172.21.15.46:0/125174982 shutdown_connections 2024-01-27T01:07:05.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.512+0000 7fc2bd893700 1 --2- 172.21.15.46:0/125174982 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc2a4067200 0x7fc2a40696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.512+0000 7fc2bd893700 1 --2- 172.21.15.46:0/125174982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2b815d8c0 0x7fc2b80aea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.512+0000 7fc2bd893700 1 --2- 172.21.15.46:0/125174982 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc2b81599d0 0x7fc2b8147c70 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.512+0000 7fc2bd893700 1 --2- 172.21.15.46:0/125174982 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc2b8159da0 0x7fc2b809bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.512+0000 7fc2bd893700 1 -- 172.21.15.46:0/125174982 >> 172.21.15.46:0/125174982 conn(0x7fc2b806f840 msgr2=0x7fc2b80b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:05.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.512+0000 7fc2bd893700 1 -- 172.21.15.46:0/125174982 shutdown_connections 2024-01-27T01:07:05.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.512+0000 7fc2bd893700 1 -- 172.21.15.46:0/125174982 wait complete. 2024-01-27T01:07:05.526 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-01-27T01:07:05.654 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:05.654+0000 7f09caa5b1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:07:05.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.662+0000 7f3fe30bd700 1 Processor -- start 2024-01-27T01:07:05.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.663+0000 7f3fe30bd700 1 -- start start 2024-01-27T01:07:05.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.664+0000 7f3fe30bd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3fdc067af0 0x7f3fdc067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:05.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.664+0000 7f3fe30bd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fdc157200 0x7f3fdc157680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:05.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.664+0000 7f3fe30bd700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fdc06c160 con 0x7f3fdc06ce90 2024-01-27T01:07:05.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.664+0000 7f3fe30bd700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fdc06ca00 con 0x7f3fdc067af0 2024-01-27T01:07:05.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.664+0000 7f3fe30bd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc0c84f0 con 0x7f3fdc157200 2024-01-27T01:07:05.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.664+0000 7f3fe0e59700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3fdc067af0 0x7f3fdc067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:05.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.664+0000 7f3fe0e59700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3fdc067af0 0x7f3fdc067ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59530/0 (socket says 172.21.15.46:59530) 2024-01-27T01:07:05.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.664+0000 7f3fe0e59700 1 -- 172.21.15.46:0/338533533 learned_addr learned my addr 172.21.15.46:0/338533533 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:05.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.664+0000 7f3fe165a700 1 --2- 172.21.15.46:0/338533533 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fdc157200 0x7f3fdc157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:05.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.664+0000 7f3fe0e59700 1 -- 172.21.15.46:0/338533533 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fdc157200 msgr2=0x7f3fdc157680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:05.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.664+0000 7f3fe0e59700 1 --2- 172.21.15.46:0/338533533 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fdc157200 0x7f3fdc157680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.664+0000 7f3fe0e59700 1 -- 172.21.15.46:0/338533533 >> v1:172.21.15.106:6789/0 conn(0x7f3fdc06ce90 legacy=0x7f3fdc0c7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:05.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.664+0000 7f3fdb7fe700 1 -- 172.21.15.46:0/338533533 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447679666 0 0) 0x7f3fdc06c160 con 0x7f3fdc06ce90 2024-01-27T01:07:05.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.664+0000 7f3fe0e59700 1 -- 172.21.15.46:0/338533533 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fdc06c3b0 con 0x7f3fdc067af0 2024-01-27T01:07:05.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.665+0000 7f3fe0e59700 1 --2- 172.21.15.46:0/338533533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fdc067af0 0x7f3fdc067ec0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f3fd00099b0 tx=0x7f3fd0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c595b19cb73b6c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:05.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.665+0000 7f3fdb7fe700 1 -- 172.21.15.46:0/338533533 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0009ea0 con 0x7f3fdc067af0 2024-01-27T01:07:05.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.665+0000 7f3fdb7fe700 1 -- 172.21.15.46:0/338533533 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd000b860 con 0x7f3fdc067af0 2024-01-27T01:07:05.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.665+0000 7f3fdb7fe700 1 -- 172.21.15.46:0/338533533 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd000bc00 con 0x7f3fdc067af0 2024-01-27T01:07:05.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.665+0000 7f3fe30bd700 1 -- 172.21.15.46:0/338533533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fdc067af0 msgr2=0x7f3fdc067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:05.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.665+0000 7f3fe30bd700 1 --2- 172.21.15.46:0/338533533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fdc067af0 0x7f3fdc067ec0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f3fd00099b0 tx=0x7f3fd0009250 comp rx=0 tx=0).stop 2024-01-27T01:07:05.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.665+0000 7f3fe30bd700 1 -- 172.21.15.46:0/338533533 shutdown_connections 2024-01-27T01:07:05.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.665+0000 7f3fe30bd700 1 --2- 172.21.15.46:0/338533533 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fdc157200 0x7f3fdc157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.665+0000 7f3fe30bd700 1 --2- 172.21.15.46:0/338533533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fdc067af0 0x7f3fdc067ec0 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.665+0000 7f3fe30bd700 1 -- 172.21.15.46:0/338533533 >> 172.21.15.46:0/338533533 conn(0x7f3fdc0ca9a0 msgr2=0x7f3fdc068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:05.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.666+0000 7f3fe30bd700 1 -- 172.21.15.46:0/338533533 shutdown_connections 2024-01-27T01:07:05.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.666+0000 7f3fe30bd700 1 -- 172.21.15.46:0/338533533 wait complete. 2024-01-27T01:07:05.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.667+0000 7f3fe30bd700 1 Processor -- start 2024-01-27T01:07:05.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.667+0000 7f3fe30bd700 1 -- start start 2024-01-27T01:07:05.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.668+0000 7f3fe30bd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fdc067af0 0x7f3fdc152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:05.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.668+0000 7f3fe30bd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fdc06ce90 0x7f3fdc0c76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:05.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.668+0000 7f3fe30bd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fdc157200 0x7f3fdc16dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:05.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.668+0000 7f3fe30bd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc0c5160 con 0x7f3fdc157200 2024-01-27T01:07:05.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.668+0000 7f3fe30bd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc0cc020 con 0x7f3fdc06ce90 2024-01-27T01:07:05.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.668+0000 7f3fe30bd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc152580 con 0x7f3fdc067af0 2024-01-27T01:07:05.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.668+0000 7f3fe165a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fdc157200 0x7f3fdc16dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:05.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.668+0000 7f3fe165a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fdc157200 0x7f3fdc16dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59532/0 (socket says 172.21.15.46:59532) 2024-01-27T01:07:05.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.668+0000 7f3fe165a700 1 -- 172.21.15.46:0/2023376460 learned_addr learned my addr 172.21.15.46:0/2023376460 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:05.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.668+0000 7f3fe0e59700 1 --2- 172.21.15.46:0/2023376460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fdc067af0 0x7f3fdc152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:05.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.668+0000 7f3fdbfff700 1 --2- 172.21.15.46:0/2023376460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fdc06ce90 0x7f3fdc0c76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:05.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.668+0000 7f3fe165a700 1 -- 172.21.15.46:0/2023376460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fdc067af0 msgr2=0x7f3fdc152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:05.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.668+0000 7f3fe165a700 1 --2- 172.21.15.46:0/2023376460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fdc067af0 0x7f3fdc152040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.668+0000 7f3fe165a700 1 -- 172.21.15.46:0/2023376460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fdc06ce90 msgr2=0x7f3fdc0c76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:05.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.668+0000 7f3fe165a700 1 --2- 172.21.15.46:0/2023376460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fdc06ce90 0x7f3fdc0c76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.668+0000 7f3fe165a700 1 -- 172.21.15.46:0/2023376460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd00094f0 con 0x7f3fdc157200 2024-01-27T01:07:05.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.668+0000 7f3fe165a700 1 --2- 172.21.15.46:0/2023376460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fdc157200 0x7f3fdc16dcd0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4009df0 tx=0x7f3fd400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:05.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.669+0000 7f3fd8ff9700 1 -- 172.21.15.46:0/2023376460 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd400caa0 con 0x7f3fdc157200 2024-01-27T01:07:05.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.669+0000 7f3fd8ff9700 1 -- 172.21.15.46:0/2023376460 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd400f040 con 0x7f3fdc157200 2024-01-27T01:07:05.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.669+0000 7f3fe30bd700 1 -- 172.21.15.46:0/2023376460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fdc0cb410 con 0x7f3fdc157200 2024-01-27T01:07:05.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.669+0000 7f3fd8ff9700 1 -- 172.21.15.46:0/2023376460 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd400ce70 con 0x7f3fdc157200 2024-01-27T01:07:05.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.669+0000 7f3fe30bd700 1 -- 172.21.15.46:0/2023376460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fdc0cb9d0 con 0x7f3fdc157200 2024-01-27T01:07:05.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.670+0000 7f3fe30bd700 1 -- 172.21.15.46:0/2023376460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fdc069a20 con 0x7f3fdc157200 2024-01-27T01:07:05.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.670+0000 7f3fd8ff9700 1 -- 172.21.15.46:0/2023376460 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 132) v1 ==== 85003+0+0 (secure 0 0 0) 0x7f3fd4014690 con 0x7f3fdc157200 2024-01-27T01:07:05.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.671+0000 7f3fd8ff9700 1 --2- 172.21.15.46:0/2023376460 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fc0067200 0x7f3fc00696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:05.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.671+0000 7f3fd8ff9700 1 -- 172.21.15.46:0/2023376460 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f3fd4085d40 con 0x7f3fdc157200 2024-01-27T01:07:05.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.671+0000 7f3fe0e59700 1 -- 172.21.15.46:0/2023376460 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fc0067200 msgr2=0x7f3fc00696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:05.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.671+0000 7f3fe0e59700 1 --2- 172.21.15.46:0/2023376460 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fc0067200 0x7f3fc00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:05.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.675+0000 7f3fd8ff9700 1 -- 172.21.15.46:0/2023376460 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3fd4055e50 con 0x7f3fdc157200 2024-01-27T01:07:05.730 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:05.730+0000 7f09caa5b1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:07:05.855 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:05.855+0000 7f09caa5b1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:07:05.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.872+0000 7f3fe0e59700 1 -- 172.21.15.46:0/2023376460 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fc0067200 msgr2=0x7f3fc00696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:05.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.872+0000 7f3fe0e59700 1 --2- 172.21.15.46:0/2023376460 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fc0067200 0x7f3fc00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.882+0000 7f3fe30bd700 1 -- 172.21.15.46:0/2023376460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f3fdc04c710 con 0x7f3fdc157200 2024-01-27T01:07:05.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.883+0000 7f3fd8ff9700 1 -- 172.21.15.46:0/2023376460 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v228) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3fd40559e0 con 0x7f3fdc157200 2024-01-27T01:07:05.883 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-27T01:07:05.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.885+0000 7f3fe30bd700 1 -- 172.21.15.46:0/2023376460 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fc0067200 msgr2=0x7f3fc00696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:05.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.885+0000 7f3fe30bd700 1 --2- 172.21.15.46:0/2023376460 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fc0067200 0x7f3fc00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.885+0000 7f3fe30bd700 1 -- 172.21.15.46:0/2023376460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fdc157200 msgr2=0x7f3fdc16dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:05.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.885+0000 7f3fe30bd700 1 --2- 172.21.15.46:0/2023376460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fdc157200 0x7f3fdc16dcd0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4009df0 tx=0x7f3fd400b040 comp rx=0 tx=0).stop 2024-01-27T01:07:05.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.886+0000 7f3fe30bd700 1 -- 172.21.15.46:0/2023376460 shutdown_connections 2024-01-27T01:07:05.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.886+0000 7f3fe30bd700 1 --2- 172.21.15.46:0/2023376460 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fc0067200 0x7f3fc00696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.886+0000 7f3fe30bd700 1 --2- 172.21.15.46:0/2023376460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fdc157200 0x7f3fdc16dcd0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.886+0000 7f3fe30bd700 1 --2- 172.21.15.46:0/2023376460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fdc067af0 0x7f3fdc152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.886+0000 7f3fe30bd700 1 --2- 172.21.15.46:0/2023376460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fdc06ce90 0x7f3fdc0c76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:05.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.886+0000 7f3fe30bd700 1 -- 172.21.15.46:0/2023376460 >> 172.21.15.46:0/2023376460 conn(0x7f3fdc0ca9a0 msgr2=0x7f3fdc155a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:05.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.886+0000 7f3fe30bd700 1 -- 172.21.15.46:0/2023376460 shutdown_connections 2024-01-27T01:07:05.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:05.886+0000 7f3fe30bd700 1 -- 172.21.15.46:0/2023376460 wait complete. 2024-01-27T01:07:05.900 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:07:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.037+0000 7ff00a4c2700 1 Processor -- start 2024-01-27T01:07:06.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.037+0000 7ff00a4c2700 1 -- start start 2024-01-27T01:07:06.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.038+0000 7ff00a4c2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff0040b2b40 0x7ff0040ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.038+0000 7ff00a4c2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff0040aedd0 0x7ff0040cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.038+0000 7ff00a4c2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00415a0b0 con 0x7ff0040ae960 2024-01-27T01:07:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.038+0000 7ff00a4c2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff004099fc0 con 0x7ff0040aedd0 2024-01-27T01:07:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.038+0000 7ff00a4c2700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0040ae4d0 con 0x7ff0040b2b40 2024-01-27T01:07:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.038+0000 7ff008a5f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff0040aedd0 0x7ff0040cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.038+0000 7ff0037fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff0040ae960 0x7ff0040ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46320/0 (socket says 172.21.15.46:46320) 2024-01-27T01:07:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.038+0000 7ff0037fe700 1 -- 172.21.15.46:0/2149842500 learned_addr learned my addr 172.21.15.46:0/2149842500 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.038+0000 7ff003fff700 1 --2- 172.21.15.46:0/2149842500 >> v2:172.21.15.106:3300/0 conn(0x7ff0040b2b40 0x7ff0040ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.039+0000 7ff002ffd700 1 -- 172.21.15.46:0/2149842500 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778624964 0 0) 0x7ff00415a0b0 con 0x7ff0040ae960 2024-01-27T01:07:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.039+0000 7ff008a5f700 1 -- 172.21.15.46:0/2149842500 >> v2:172.21.15.106:3300/0 conn(0x7ff0040b2b40 msgr2=0x7ff0040ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.039+0000 7ff008a5f700 1 --2- 172.21.15.46:0/2149842500 >> v2:172.21.15.106:3300/0 conn(0x7ff0040b2b40 0x7ff0040ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.039+0000 7ff008a5f700 1 -- 172.21.15.46:0/2149842500 >> v1:172.21.15.46:6789/0 conn(0x7ff0040ae960 legacy=0x7ff0040ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.039+0000 7ff008a5f700 1 -- 172.21.15.46:0/2149842500 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0040ad8c0 con 0x7ff0040aedd0 2024-01-27T01:07:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.039+0000 7ff008a5f700 1 --2- 172.21.15.46:0/2149842500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0040aedd0 0x7ff0040cb5e0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7ff004088850 tx=0x7feffc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13363120c8ce6be6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.039+0000 7ff002ffd700 1 -- 172.21.15.46:0/2149842500 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc00e070 con 0x7ff0040aedd0 2024-01-27T01:07:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.039+0000 7ff002ffd700 1 -- 172.21.15.46:0/2149842500 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feffc00bb60 con 0x7ff0040aedd0 2024-01-27T01:07:06.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.039+0000 7ff002ffd700 1 -- 172.21.15.46:0/2149842500 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc012430 con 0x7ff0040aedd0 2024-01-27T01:07:06.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.040+0000 7ff00a4c2700 1 -- 172.21.15.46:0/2149842500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0040aedd0 msgr2=0x7ff0040cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:06.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.040+0000 7ff00a4c2700 1 --2- 172.21.15.46:0/2149842500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0040aedd0 0x7ff0040cb5e0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7ff004088850 tx=0x7feffc009780 comp rx=0 tx=0).stop 2024-01-27T01:07:06.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.040+0000 7ff00a4c2700 1 -- 172.21.15.46:0/2149842500 shutdown_connections 2024-01-27T01:07:06.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.040+0000 7ff00a4c2700 1 --2- 172.21.15.46:0/2149842500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0040aedd0 0x7ff0040cb5e0 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.040+0000 7ff00a4c2700 1 --2- 172.21.15.46:0/2149842500 >> v2:172.21.15.106:3300/0 conn(0x7ff0040b2b40 0x7ff0040ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.040+0000 7ff00a4c2700 1 -- 172.21.15.46:0/2149842500 >> 172.21.15.46:0/2149842500 conn(0x7ff004099a30 msgr2=0x7ff00406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:06.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.040+0000 7ff00a4c2700 1 -- 172.21.15.46:0/2149842500 shutdown_connections 2024-01-27T01:07:06.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.040+0000 7ff00a4c2700 1 -- 172.21.15.46:0/2149842500 wait complete. 2024-01-27T01:07:06.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.041+0000 7ff00a4c2700 1 Processor -- start 2024-01-27T01:07:06.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.042+0000 7ff00a4c2700 1 -- start start 2024-01-27T01:07:06.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.042+0000 7ff00a4c2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0040ae960 0x7ff0040c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.042+0000 7ff00a4c2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0040b2b40 0x7ff0040b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.042+0000 7ff00a4c2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0040c7820 0x7ff004151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.042+0000 7ff00a4c2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0040c8b30 con 0x7ff0040c7820 2024-01-27T01:07:06.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.042+0000 7ff00a4c2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0040b59d0 con 0x7ff0040ae960 2024-01-27T01:07:06.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.042+0000 7ff00a4c2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0040b63a0 con 0x7ff0040b2b40 2024-01-27T01:07:06.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.042+0000 7ff008a5f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0040c7820 0x7ff004151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:06.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.043+0000 7ff003fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0040ae960 0x7ff0040c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:06.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.043+0000 7ff008a5f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0040c7820 0x7ff004151740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59548/0 (socket says 172.21.15.46:59548) 2024-01-27T01:07:06.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.043+0000 7ff0037fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0040b2b40 0x7ff0040b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:06.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.043+0000 7ff008a5f700 1 -- 172.21.15.46:0/2893613510 learned_addr learned my addr 172.21.15.46:0/2893613510 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:06.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.043+0000 7ff003fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0040ae960 0x7ff0040c85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:52300/0 (socket says 172.21.15.46:52300) 2024-01-27T01:07:06.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.043+0000 7ff008a5f700 1 -- 172.21.15.46:0/2893613510 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0040b2b40 msgr2=0x7ff0040b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:06.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.043+0000 7ff008a5f700 1 --2- 172.21.15.46:0/2893613510 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0040b2b40 0x7ff0040b5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.043+0000 7ff008a5f700 1 -- 172.21.15.46:0/2893613510 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0040ae960 msgr2=0x7ff0040c85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.043+0000 7ff008a5f700 1 --2- 172.21.15.46:0/2893613510 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0040ae960 0x7ff0040c85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.043+0000 7ff008a5f700 1 -- 172.21.15.46:0/2893613510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc009a20 con 0x7ff0040c7820 2024-01-27T01:07:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.043+0000 7ff003fff700 1 --2- 172.21.15.46:0/2893613510 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0040ae960 0x7ff0040c85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.043+0000 7ff008a5f700 1 --2- 172.21.15.46:0/2893613510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0040c7820 0x7ff004151740 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7feffc013000 tx=0x7feffc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.043+0000 7fefebfff700 1 -- 172.21.15.46:0/2893613510 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc027470 con 0x7ff0040c7820 2024-01-27T01:07:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.043+0000 7fefebfff700 1 -- 172.21.15.46:0/2893613510 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feffc0275d0 con 0x7ff0040c7820 2024-01-27T01:07:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.043+0000 7fefebfff700 1 -- 172.21.15.46:0/2893613510 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc027990 con 0x7ff0040c7820 2024-01-27T01:07:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.043+0000 7ff00a4c2700 1 -- 172.21.15.46:0/2893613510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0040c8d60 con 0x7ff0040c7820 2024-01-27T01:07:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.043+0000 7ff00a4c2700 1 -- 172.21.15.46:0/2893613510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0040b5380 con 0x7ff0040c7820 2024-01-27T01:07:06.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.045+0000 7ff00a4c2700 1 -- 172.21.15.46:0/2893613510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0040b5b80 con 0x7ff0040c7820 2024-01-27T01:07:06.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.046+0000 7fefebfff700 1 -- 172.21.15.46:0/2893613510 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 132) v1 ==== 85003+0+0 (secure 0 0 0) 0x7feffc024070 con 0x7ff0040c7820 2024-01-27T01:07:06.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.046+0000 7fefebfff700 1 --2- 172.21.15.46:0/2893613510 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feff0067200 0x7feff00696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.046+0000 7fefebfff700 1 -- 172.21.15.46:0/2893613510 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7feffc087a40 con 0x7ff0040c7820 2024-01-27T01:07:06.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.046+0000 7ff003fff700 1 -- 172.21.15.46:0/2893613510 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feff0067200 msgr2=0x7feff00696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:06.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.046+0000 7ff003fff700 1 --2- 172.21.15.46:0/2893613510 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feff0067200 0x7feff00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:06.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.051+0000 7fefebfff700 1 -- 172.21.15.46:0/2893613510 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff0040b5b80 con 0x7ff0040c7820 2024-01-27T01:07:06.139 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:06.139+0000 7f8dd1d5c1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:07:06.225 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:06.225+0000 7f8dd1d5c1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:07:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.247+0000 7ff003fff700 1 -- 172.21.15.46:0/2893613510 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feff0067200 msgr2=0x7feff00696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.247+0000 7ff003fff700 1 --2- 172.21.15.46:0/2893613510 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feff0067200 0x7feff00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:06.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.301+0000 7ff00a4c2700 1 -- 172.21.15.46:0/2893613510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0040b5b40 con 0x7ff0040c7820 2024-01-27T01:07:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.302+0000 7fefebfff700 1 -- 172.21.15.46:0/2893613510 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5697 (secure 0 0 0) 0x7feffc0541d0 con 0x7ff0040c7820 2024-01-27T01:07:06.302 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:06.302 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":122,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/1999300009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1999300009},{"type":"v1","addr":"172.21.15.106:6835","nonce":1999300009}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":110},{"gid":10486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/4174263148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4174263148},{"type":"v1","addr":"172.21.15.106:6833","nonce":4174263148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:06:26.286041+0000","modified":"2024-01-27T01:06:29.873402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10866},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10866":{"gid":10866,"name":"a","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2483163449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2483163449},{"type":"v1","addr":"172.21.15.46:6835","nonce":2483163449}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:06:40.353248+0000","modified":"2024-01-27T01:06:43.938885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10854":{"gid":10854,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":8,"addr":"172.21.15.46:6833/1263484152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1263484152},{"type":"v1","addr":"172.21.15.46:6833","nonce":1263484152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-27T01:07:06.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.304+0000 7ff00a4c2700 1 -- 172.21.15.46:0/2893613510 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feff0067200 msgr2=0x7feff00696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:06.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.304+0000 7ff00a4c2700 1 --2- 172.21.15.46:0/2893613510 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feff0067200 0x7feff00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.304+0000 7ff00a4c2700 1 -- 172.21.15.46:0/2893613510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0040c7820 msgr2=0x7ff004151740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:06.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.304+0000 7ff00a4c2700 1 --2- 172.21.15.46:0/2893613510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0040c7820 0x7ff004151740 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7feffc013000 tx=0x7feffc019000 comp rx=0 tx=0).stop 2024-01-27T01:07:06.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.304+0000 7ff00a4c2700 1 -- 172.21.15.46:0/2893613510 shutdown_connections 2024-01-27T01:07:06.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.304+0000 7ff00a4c2700 1 --2- 172.21.15.46:0/2893613510 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feff0067200 0x7feff00696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.304+0000 7ff00a4c2700 1 --2- 172.21.15.46:0/2893613510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0040c7820 0x7ff004151740 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.304+0000 7ff00a4c2700 1 --2- 172.21.15.46:0/2893613510 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0040ae960 0x7ff0040c85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.304+0000 7ff00a4c2700 1 --2- 172.21.15.46:0/2893613510 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0040b2b40 0x7ff0040b5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.304+0000 7ff00a4c2700 1 -- 172.21.15.46:0/2893613510 >> 172.21.15.46:0/2893613510 conn(0x7ff004099a30 msgr2=0x7ff0041590b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:06.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.304+0000 7ff00a4c2700 1 -- 172.21.15.46:0/2893613510 shutdown_connections 2024-01-27T01:07:06.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.304+0000 7ff00a4c2700 1 -- 172.21.15.46:0/2893613510 wait complete. 2024-01-27T01:07:06.306 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 122 2024-01-27T01:07:06.318 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:07:06.360 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:06.360+0000 7f09caa5b1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:07:06.373 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:06.373+0000 7f8dd1d5c1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:07:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.452+0000 7f8583f17700 1 Processor -- start 2024-01-27T01:07:06.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.452+0000 7f8583f17700 1 -- start start 2024-01-27T01:07:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.453+0000 7f8583f17700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f857c0b2b40 0x7f857c0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.453+0000 7f8583f17700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f857c0aedd0 0x7f857c0cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.453+0000 7f8583f17700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f857c15a0b0 con 0x7f857c0ae960 2024-01-27T01:07:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.453+0000 7f8583f17700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f857c099fc0 con 0x7f857c0b2b40 2024-01-27T01:07:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.453+0000 7f8583f17700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857c0ae4d0 con 0x7f857c0aedd0 2024-01-27T01:07:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.453+0000 7f8581cb3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f857c0b2b40 0x7f857c0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.453+0000 7f8581cb3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f857c0b2b40 0x7f857c0ad1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59562/0 (socket says 172.21.15.46:59562) 2024-01-27T01:07:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.453+0000 7f8581cb3700 1 -- 172.21.15.46:0/3064874283 learned_addr learned my addr 172.21.15.46:0/3064874283 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.453+0000 7f85824b4700 1 --2- 172.21.15.46:0/3064874283 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f857c0aedd0 0x7f857c0cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.454+0000 7f8581cb3700 1 -- 172.21.15.46:0/3064874283 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f857c0aedd0 msgr2=0x7f857c0cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.454+0000 7f8581cb3700 1 --2- 172.21.15.46:0/3064874283 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f857c0aedd0 0x7f857c0cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.454+0000 7f8581cb3700 1 -- 172.21.15.46:0/3064874283 >> v1:172.21.15.106:6789/0 conn(0x7f857c0ae960 legacy=0x7f857c0ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.454+0000 7f8581cb3700 1 -- 172.21.15.46:0/3064874283 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857c0ad8c0 con 0x7f857c0b2b40 2024-01-27T01:07:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.454+0000 7f8580cb1700 1 -- 172.21.15.46:0/3064874283 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267327607 0 0) 0x7f857c15a0b0 con 0x7f857c0ae960 2024-01-27T01:07:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.454+0000 7f8581cb3700 1 --2- 172.21.15.46:0/3064874283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f857c0b2b40 0x7f857c0ad1c0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f8578009990 tx=0x7f8578009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0faa57c4b24fc28 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.454+0000 7f8580cb1700 1 -- 172.21.15.46:0/3064874283 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8578009e80 con 0x7f857c0b2b40 2024-01-27T01:07:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.454+0000 7f8580cb1700 1 -- 172.21.15.46:0/3064874283 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8578005c10 con 0x7f857c0b2b40 2024-01-27T01:07:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.454+0000 7f8580cb1700 1 -- 172.21.15.46:0/3064874283 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857800b650 con 0x7f857c0b2b40 2024-01-27T01:07:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.455+0000 7f8583f17700 1 -- 172.21.15.46:0/3064874283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f857c0b2b40 msgr2=0x7f857c0ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.455+0000 7f8583f17700 1 --2- 172.21.15.46:0/3064874283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f857c0b2b40 0x7f857c0ad1c0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f8578009990 tx=0x7f8578009220 comp rx=0 tx=0).stop 2024-01-27T01:07:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.455+0000 7f8583f17700 1 -- 172.21.15.46:0/3064874283 shutdown_connections 2024-01-27T01:07:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.455+0000 7f8583f17700 1 --2- 172.21.15.46:0/3064874283 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f857c0aedd0 0x7f857c0cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.455+0000 7f8583f17700 1 --2- 172.21.15.46:0/3064874283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f857c0b2b40 0x7f857c0ad1c0 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.455+0000 7f8583f17700 1 -- 172.21.15.46:0/3064874283 >> 172.21.15.46:0/3064874283 conn(0x7f857c099a30 msgr2=0x7f857c06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.455+0000 7f8583f17700 1 -- 172.21.15.46:0/3064874283 shutdown_connections 2024-01-27T01:07:06.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.455+0000 7f8583f17700 1 -- 172.21.15.46:0/3064874283 wait complete. 2024-01-27T01:07:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.457+0000 7f8583f17700 1 Processor -- start 2024-01-27T01:07:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.457+0000 7f8583f17700 1 -- start start 2024-01-27T01:07:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f8583f17700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f857c0ae960 0x7f857c1573c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f8583f17700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f857c0b2b40 0x7f857c0b5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f8583f17700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f857c0b3f20 0x7f857c0b1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f8583f17700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857c15ce00 con 0x7f857c0b3f20 2024-01-27T01:07:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f8583f17700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857c0b4420 con 0x7f857c0b2b40 2024-01-27T01:07:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f8583f17700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857c0b4700 con 0x7f857c0ae960 2024-01-27T01:07:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f85824b4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f857c0b3f20 0x7f857c0b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f8581cb3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f857c0ae960 0x7f857c1573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f85824b4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f857c0b3f20 0x7f857c0b1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59576/0 (socket says 172.21.15.46:59576) 2024-01-27T01:07:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f85824b4700 1 -- 172.21.15.46:0/948845347 learned_addr learned my addr 172.21.15.46:0/948845347 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f85814b2700 1 --2- 172.21.15.46:0/948845347 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f857c0b2b40 0x7f857c0b5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f85824b4700 1 -- 172.21.15.46:0/948845347 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f857c0ae960 msgr2=0x7f857c1573c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f85824b4700 1 --2- 172.21.15.46:0/948845347 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f857c0ae960 0x7f857c1573c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f85824b4700 1 -- 172.21.15.46:0/948845347 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f857c0b2b40 msgr2=0x7f857c0b5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f85824b4700 1 --2- 172.21.15.46:0/948845347 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f857c0b2b40 0x7f857c0b5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f85824b4700 1 -- 172.21.15.46:0/948845347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8578009480 con 0x7f857c0b3f20 2024-01-27T01:07:06.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f8581cb3700 1 --2- 172.21.15.46:0/948845347 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f857c0ae960 0x7f857c1573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.458+0000 7f85824b4700 1 --2- 172.21.15.46:0/948845347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f857c0b3f20 0x7f857c0b1120 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f8574009d70 tx=0x7f857400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.459+0000 7f8571ffb700 1 -- 172.21.15.46:0/948845347 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857400caa0 con 0x7f857c0b3f20 2024-01-27T01:07:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.459+0000 7f8571ffb700 1 -- 172.21.15.46:0/948845347 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857400f040 con 0x7f857c0b3f20 2024-01-27T01:07:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.459+0000 7f8583f17700 1 -- 172.21.15.46:0/948845347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f857c0b49e0 con 0x7f857c0b3f20 2024-01-27T01:07:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.459+0000 7f8571ffb700 1 -- 172.21.15.46:0/948845347 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857400ce50 con 0x7f857c0b3f20 2024-01-27T01:07:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.459+0000 7f8583f17700 1 -- 172.21.15.46:0/948845347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f857c157900 con 0x7f857c0b3f20 2024-01-27T01:07:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.460+0000 7f8583f17700 1 -- 172.21.15.46:0/948845347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f857c15d000 con 0x7f857c0b3f20 2024-01-27T01:07:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.461+0000 7f8571ffb700 1 -- 172.21.15.46:0/948845347 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 132) v1 ==== 85003+0+0 (secure 0 0 0) 0x7f8574014690 con 0x7f857c0b3f20 2024-01-27T01:07:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.461+0000 7f8571ffb700 1 --2- 172.21.15.46:0/948845347 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8560067200 0x7f85600696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.462+0000 7f8581cb3700 1 -- 172.21.15.46:0/948845347 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8560067200 msgr2=0x7f85600696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.462+0000 7f8571ffb700 1 -- 172.21.15.46:0/948845347 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f8574085d40 con 0x7f857c0b3f20 2024-01-27T01:07:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.462+0000 7f8581cb3700 1 --2- 172.21.15.46:0/948845347 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8560067200 0x7f85600696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:06.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.466+0000 7f8571ffb700 1 -- 172.21.15.46:0/948845347 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f857c15d000 con 0x7f857c0b3f20 2024-01-27T01:07:06.511 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:06.511+0000 7f09caa5b1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:07:06.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.663+0000 7f8581cb3700 1 -- 172.21.15.46:0/948845347 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8560067200 msgr2=0x7f85600696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:06.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.663+0000 7f8581cb3700 1 --2- 172.21.15.46:0/948845347 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8560067200 0x7f85600696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:06.663 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:06.664+0000 7f09caa5b1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:07:06.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.673+0000 7f8583f17700 1 -- 172.21.15.46:0/948845347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f857c04c710 con 0x7f857c0b3f20 2024-01-27T01:07:06.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.675+0000 7f8571ffb700 1 -- 172.21.15.46:0/948845347 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v228) v1 ==== 74+0+19929 (secure 0 0 0) 0x7f85740523c0 con 0x7f857c0b3f20 2024-01-27T01:07:06.675 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:06.675 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":228,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:06:57.757011+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:06:24.798791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-01-27T01:06:25.249215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:06:38.156934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:06:39.330686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/3809430766":"2024-01-28T01:06:57.756975+0000","172.21.15.106:6837/62434":"2024-01-28T01:06:57.756975+0000","172.21.15.106:6836/62434":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/524220301":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/4002309435":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/2725250616":"2024-01-28T01:06:57.756975+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:07:06.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.677+0000 7f8583f17700 1 -- 172.21.15.46:0/948845347 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8560067200 msgr2=0x7f85600696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:06.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.677+0000 7f8583f17700 1 --2- 172.21.15.46:0/948845347 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8560067200 0x7f85600696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.677+0000 7f8583f17700 1 -- 172.21.15.46:0/948845347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f857c0b3f20 msgr2=0x7f857c0b1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:06.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.677+0000 7f8583f17700 1 --2- 172.21.15.46:0/948845347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f857c0b3f20 0x7f857c0b1120 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f8574009d70 tx=0x7f857400b040 comp rx=0 tx=0).stop 2024-01-27T01:07:06.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.677+0000 7f8583f17700 1 -- 172.21.15.46:0/948845347 shutdown_connections 2024-01-27T01:07:06.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.677+0000 7f8583f17700 1 --2- 172.21.15.46:0/948845347 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8560067200 0x7f85600696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.677+0000 7f8583f17700 1 --2- 172.21.15.46:0/948845347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f857c0b3f20 0x7f857c0b1120 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.677+0000 7f8583f17700 1 --2- 172.21.15.46:0/948845347 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f857c0ae960 0x7f857c1573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.677+0000 7f8583f17700 1 --2- 172.21.15.46:0/948845347 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f857c0b2b40 0x7f857c0b5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.677+0000 7f8583f17700 1 -- 172.21.15.46:0/948845347 >> 172.21.15.46:0/948845347 conn(0x7f857c099a30 msgr2=0x7f857c159460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:06.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.677+0000 7f8583f17700 1 -- 172.21.15.46:0/948845347 shutdown_connections 2024-01-27T01:07:06.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.677+0000 7f8583f17700 1 -- 172.21.15.46:0/948845347 wait complete. 2024-01-27T01:07:06.693 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-27T01:07:06.693 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:07:06.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.830+0000 7f34b3e94700 1 Processor -- start 2024-01-27T01:07:06.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.830+0000 7f34b3e94700 1 -- start start 2024-01-27T01:07:06.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.831+0000 7f34b3e94700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f34ac15d210 0x7f34ac0cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.831+0000 7f34b3e94700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34ac0cbff0 0x7f34ac0ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.831+0000 7f34b3e94700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ac0cff70 con 0x7f34ac15ce40 2024-01-27T01:07:06.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.831+0000 7f34b3e94700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34ac0cc490 con 0x7f34ac15d210 2024-01-27T01:07:06.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.831+0000 7f34b3e94700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac0cf400 con 0x7f34ac0cbff0 2024-01-27T01:07:06.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.831+0000 7f34b142f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f34ac15d210 0x7f34ac0cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:06.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.831+0000 7f34b142f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f34ac15d210 0x7f34ac0cbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59586/0 (socket says 172.21.15.46:59586) 2024-01-27T01:07:06.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.831+0000 7f34b142f700 1 -- 172.21.15.46:0/746723111 learned_addr learned my addr 172.21.15.46:0/746723111 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:06.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.831+0000 7f34b1c30700 1 --1- 172.21.15.46:0/746723111 >> v1:172.21.15.106:6789/0 conn(0x7f34ac15ce40 0x7f34ac0d0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:56504/0 (socket says 172.21.15.46:56504) 2024-01-27T01:07:06.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.831+0000 7f34b2431700 1 --2- 172.21.15.46:0/746723111 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34ac0cbff0 0x7f34ac0ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:06.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.831+0000 7f34b0c2e700 1 -- 172.21.15.46:0/746723111 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890179202 0 0) 0x7f34ac0cff70 con 0x7f34ac15ce40 2024-01-27T01:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.831+0000 7f34b0c2e700 1 -- 172.21.15.46:0/746723111 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3498003570 con 0x7f34ac15ce40 2024-01-27T01:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.831+0000 7f34b2431700 1 -- 172.21.15.46:0/746723111 >> v2:172.21.15.46:3300/0 conn(0x7f34ac15d210 msgr2=0x7f34ac0cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.831+0000 7f34b2431700 1 --2- 172.21.15.46:0/746723111 >> v2:172.21.15.46:3300/0 conn(0x7f34ac15d210 0x7f34ac0cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.831+0000 7f34b2431700 1 -- 172.21.15.46:0/746723111 >> v1:172.21.15.106:6789/0 conn(0x7f34ac15ce40 legacy=0x7f34ac0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.831+0000 7f34b2431700 1 -- 172.21.15.46:0/746723111 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac0cf660 con 0x7f34ac0cbff0 2024-01-27T01:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.831+0000 7f34b142f700 1 --2- 172.21.15.46:0/746723111 >> v2:172.21.15.46:3300/0 conn(0x7f34ac15d210 0x7f34ac0cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.832+0000 7f34b2431700 1 --2- 172.21.15.46:0/746723111 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34ac0cbff0 0x7f34ac0ade30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f34a4009e60 tx=0x7f34a4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bb5244f93f281bc server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.832+0000 7f34b0c2e700 1 -- 172.21.15.46:0/746723111 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a400e070 con 0x7f34ac0cbff0 2024-01-27T01:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.832+0000 7f34b0c2e700 1 -- 172.21.15.46:0/746723111 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a400bab0 con 0x7f34ac0cbff0 2024-01-27T01:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.832+0000 7f34b0c2e700 1 -- 172.21.15.46:0/746723111 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a400be70 con 0x7f34ac0cbff0 2024-01-27T01:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.832+0000 7f34b3e94700 1 -- 172.21.15.46:0/746723111 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34ac0cbff0 msgr2=0x7f34ac0ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.832+0000 7f34b3e94700 1 --2- 172.21.15.46:0/746723111 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34ac0cbff0 0x7f34ac0ade30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f34a4009e60 tx=0x7f34a4009780 comp rx=0 tx=0).stop 2024-01-27T01:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.833+0000 7f34b3e94700 1 -- 172.21.15.46:0/746723111 shutdown_connections 2024-01-27T01:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.833+0000 7f34b3e94700 1 --2- 172.21.15.46:0/746723111 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34ac0cbff0 0x7f34ac0ade30 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.833+0000 7f34b3e94700 1 --2- 172.21.15.46:0/746723111 >> v2:172.21.15.46:3300/0 conn(0x7f34ac15d210 0x7f34ac0cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.833+0000 7f34b3e94700 1 -- 172.21.15.46:0/746723111 >> 172.21.15.46:0/746723111 conn(0x7f34ac06f840 msgr2=0x7f34ac0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.833+0000 7f34b3e94700 1 -- 172.21.15.46:0/746723111 shutdown_connections 2024-01-27T01:07:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.833+0000 7f34b3e94700 1 -- 172.21.15.46:0/746723111 wait complete. 2024-01-27T01:07:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.834+0000 7f34b3e94700 1 Processor -- start 2024-01-27T01:07:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.834+0000 7f34b3e94700 1 -- start start 2024-01-27T01:07:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.835+0000 7f34b3e94700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34ac0cbff0 0x7f34ac0c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.835+0000 7f34b3e94700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34ac15ce40 0x7f34ac09a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.835+0000 7f34b3e94700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34ac15d210 0x7f34ac09a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.835+0000 7f34b3e94700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac0c8c40 con 0x7f34ac15d210 2024-01-27T01:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.835+0000 7f34b3e94700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac0c8030 con 0x7f34ac15ce40 2024-01-27T01:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.835+0000 7f34b3e94700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac0c8310 con 0x7f34ac0cbff0 2024-01-27T01:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.836+0000 7f34b2431700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34ac15d210 0x7f34ac09a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.836+0000 7f34b142f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34ac15ce40 0x7f34ac09a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.836+0000 7f34b1c30700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34ac0cbff0 0x7f34ac0c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.836+0000 7f34b142f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34ac15ce40 0x7f34ac09a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:52320/0 (socket says 172.21.15.46:52320) 2024-01-27T01:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.836+0000 7f34b142f700 1 -- 172.21.15.46:0/3222405024 learned_addr learned my addr 172.21.15.46:0/3222405024 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.836+0000 7f34b1c30700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34ac0cbff0 0x7f34ac0c7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:45670/0 (socket says 172.21.15.46:45670) 2024-01-27T01:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.836+0000 7f34b142f700 1 -- 172.21.15.46:0/3222405024 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34ac0cbff0 msgr2=0x7f34ac0c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.836+0000 7f34b142f700 1 --2- 172.21.15.46:0/3222405024 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34ac0cbff0 0x7f34ac0c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.836+0000 7f34b142f700 1 -- 172.21.15.46:0/3222405024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34ac15d210 msgr2=0x7f34ac09a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.836+0000 7f34b142f700 1 --2- 172.21.15.46:0/3222405024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34ac15d210 0x7f34ac09a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.836+0000 7f34b142f700 1 -- 172.21.15.46:0/3222405024 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c0094f0 con 0x7f34ac15ce40 2024-01-27T01:07:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.836+0000 7f34b1c30700 1 --2- 172.21.15.46:0/3222405024 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34ac0cbff0 0x7f34ac0c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.836+0000 7f34b142f700 1 --2- 172.21.15.46:0/3222405024 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34ac15ce40 0x7f34ac09a1a0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f349c00ec50 tx=0x7f349c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.836+0000 7f34a1ffb700 1 -- 172.21.15.46:0/3222405024 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c00c850 con 0x7f34ac15ce40 2024-01-27T01:07:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.836+0000 7f34a1ffb700 1 -- 172.21.15.46:0/3222405024 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349c00c9b0 con 0x7f34ac15ce40 2024-01-27T01:07:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.836+0000 7f34b3e94700 1 -- 172.21.15.46:0/3222405024 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a4009a20 con 0x7f34ac15ce40 2024-01-27T01:07:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.836+0000 7f34a1ffb700 1 -- 172.21.15.46:0/3222405024 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c00cd50 con 0x7f34ac15ce40 2024-01-27T01:07:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.837+0000 7f34b3e94700 1 -- 172.21.15.46:0/3222405024 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34ac0c88d0 con 0x7f34ac15ce40 2024-01-27T01:07:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.838+0000 7f34b3e94700 1 -- 172.21.15.46:0/3222405024 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34ac04c710 con 0x7f34ac15ce40 2024-01-27T01:07:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.838+0000 7f34a1ffb700 1 -- 172.21.15.46:0/3222405024 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 132) v1 ==== 85003+0+0 (secure 0 0 0) 0x7f349c013680 con 0x7f34ac15ce40 2024-01-27T01:07:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.838+0000 7f34a1ffb700 1 --2- 172.21.15.46:0/3222405024 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3490067200 0x7f34900696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.839+0000 7f34b1c30700 1 -- 172.21.15.46:0/3222405024 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3490067200 msgr2=0x7f34900696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.839+0000 7f34a1ffb700 1 -- 172.21.15.46:0/3222405024 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7f349c085410 con 0x7f34ac15ce40 2024-01-27T01:07:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.839+0000 7f34b1c30700 1 --2- 172.21.15.46:0/3222405024 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3490067200 0x7f34900696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:06.843+0000 7f34a1ffb700 1 -- 172.21.15.46:0/3222405024 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f349c051b60 con 0x7f34ac15ce40 2024-01-27T01:07:06.972 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:06.972+0000 7f8dd1d5c1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:07:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.040+0000 7f34b1c30700 1 -- 172.21.15.46:0/3222405024 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3490067200 msgr2=0x7f34900696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.040+0000 7f34b1c30700 1 --2- 172.21.15.46:0/3222405024 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3490067200 0x7f34900696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:07.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.101+0000 7f34b3e94700 1 -- 172.21.15.46:0/3222405024 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34ac0c8d80 con 0x7f34ac15ce40 2024-01-27T01:07:07.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.102+0000 7f34a1ffb700 1 -- 172.21.15.46:0/3222405024 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5697 (secure 0 0 0) 0x7f349c055180 con 0x7f34ac15ce40 2024-01-27T01:07:07.103 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:07.103 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":122,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/1999300009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1999300009},{"type":"v1","addr":"172.21.15.106:6835","nonce":1999300009}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":110},{"gid":10486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/4174263148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4174263148},{"type":"v1","addr":"172.21.15.106:6833","nonce":4174263148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:06:26.286041+0000","modified":"2024-01-27T01:06:29.873402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10866},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10866":{"gid":10866,"name":"a","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2483163449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2483163449},{"type":"v1","addr":"172.21.15.46:6835","nonce":2483163449}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:06:40.353248+0000","modified":"2024-01-27T01:06:43.938885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10854":{"gid":10854,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":8,"addr":"172.21.15.46:6833/1263484152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1263484152},{"type":"v1","addr":"172.21.15.46:6833","nonce":1263484152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-27T01:07:07.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.105+0000 7f34b3e94700 1 -- 172.21.15.46:0/3222405024 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3490067200 msgr2=0x7f34900696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:07.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.105+0000 7f34b3e94700 1 --2- 172.21.15.46:0/3222405024 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3490067200 0x7f34900696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.105+0000 7f34b3e94700 1 -- 172.21.15.46:0/3222405024 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34ac15ce40 msgr2=0x7f34ac09a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:07.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.105+0000 7f34b3e94700 1 --2- 172.21.15.46:0/3222405024 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34ac15ce40 0x7f34ac09a1a0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f349c00ec50 tx=0x7f349c009250 comp rx=0 tx=0).stop 2024-01-27T01:07:07.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.105+0000 7f34b3e94700 1 -- 172.21.15.46:0/3222405024 shutdown_connections 2024-01-27T01:07:07.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.105+0000 7f34b3e94700 1 --2- 172.21.15.46:0/3222405024 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3490067200 0x7f34900696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.105+0000 7f34b3e94700 1 --2- 172.21.15.46:0/3222405024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34ac15d210 0x7f34ac09a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.105+0000 7f34b3e94700 1 --2- 172.21.15.46:0/3222405024 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34ac0cbff0 0x7f34ac0c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.105+0000 7f34b3e94700 1 --2- 172.21.15.46:0/3222405024 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34ac15ce40 0x7f34ac09a1a0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.105+0000 7f34b3e94700 1 -- 172.21.15.46:0/3222405024 >> 172.21.15.46:0/3222405024 conn(0x7f34ac06f840 msgr2=0x7f34ac0b36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:07.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.105+0000 7f34b3e94700 1 -- 172.21.15.46:0/3222405024 shutdown_connections 2024-01-27T01:07:07.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.105+0000 7f34b3e94700 1 -- 172.21.15.46:0/3222405024 wait complete. 2024-01-27T01:07:07.106 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 122 2024-01-27T01:07:07.119 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:07:07.146 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:07.146+0000 7f8dd1d5c1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:07:07.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.251+0000 7fb4bac89700 1 Processor -- start 2024-01-27T01:07:07.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.251+0000 7fb4bac89700 1 -- start start 2024-01-27T01:07:07.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.252+0000 7fb4bac89700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb4b4159960 0x7fb4b406bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:07.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.252+0000 7fb4bac89700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b406c600 con 0x7fb4b406c1f0 2024-01-27T01:07:07.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.252+0000 7fb4bac89700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b415dac0 con 0x7fb4b4159590 2024-01-27T01:07:07.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.252+0000 7fb4bac89700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4b415e350 con 0x7fb4b4159960 2024-01-27T01:07:07.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.252+0000 7fb4b9226700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb4b406c1f0 0x7fb4b4069370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46332/0 (socket says 172.21.15.46:46332) 2024-01-27T01:07:07.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.252+0000 7fb4b9226700 1 -- 172.21.15.46:0/2628508908 learned_addr learned my addr 172.21.15.46:0/2628508908 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:07.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.252+0000 7fb4b3fff700 1 --2- 172.21.15.46:0/2628508908 >> v2:172.21.15.106:3300/0 conn(0x7fb4b4159960 0x7fb4b406bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:07.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.253+0000 7fb4b37fe700 1 -- 172.21.15.46:0/2628508908 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771796867 0 0) 0x7fb4b406c600 con 0x7fb4b406c1f0 2024-01-27T01:07:07.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.253+0000 7fb4b37fe700 1 -- 172.21.15.46:0/2628508908 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb49c003570 con 0x7fb4b406c1f0 2024-01-27T01:07:07.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.253+0000 7fb4b37fe700 1 -- 172.21.15.46:0/2628508908 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431851088 0 0) 0x7fb4b415dac0 con 0x7fb4b4159590 2024-01-27T01:07:07.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.253+0000 7fb4b37fe700 1 -- 172.21.15.46:0/2628508908 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4b406c600 con 0x7fb4b4159590 2024-01-27T01:07:07.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.253+0000 7fb4b3fff700 1 -- 172.21.15.46:0/2628508908 >> v1:172.21.15.106:6789/0 conn(0x7fb4b4159590 legacy=0x7fb4b415d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:07.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.253+0000 7fb4b3fff700 1 -- 172.21.15.46:0/2628508908 >> v1:172.21.15.46:6789/0 conn(0x7fb4b406c1f0 legacy=0x7fb4b4069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:07.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.253+0000 7fb4b3fff700 1 -- 172.21.15.46:0/2628508908 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b415dd90 con 0x7fb4b4159960 2024-01-27T01:07:07.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.253+0000 7fb4b3fff700 1 --2- 172.21.15.46:0/2628508908 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4b4159960 0x7fb4b406bcb0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fb4a80098e0 tx=0x7fb4a80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df43b95c31715fbe server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:07.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.253+0000 7fb4b37fe700 1 -- 172.21.15.46:0/2628508908 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a8011630 con 0x7fb4b4159960 2024-01-27T01:07:07.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.253+0000 7fb4b37fe700 1 -- 172.21.15.46:0/2628508908 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a8011790 con 0x7fb4b4159960 2024-01-27T01:07:07.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.253+0000 7fb4b37fe700 1 -- 172.21.15.46:0/2628508908 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a8011b10 con 0x7fb4b4159960 2024-01-27T01:07:07.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.254+0000 7fb4bac89700 1 -- 172.21.15.46:0/2628508908 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4b4159960 msgr2=0x7fb4b406bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:07.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.254+0000 7fb4bac89700 1 --2- 172.21.15.46:0/2628508908 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4b4159960 0x7fb4b406bcb0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fb4a80098e0 tx=0x7fb4a80091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.254+0000 7fb4bac89700 1 -- 172.21.15.46:0/2628508908 shutdown_connections 2024-01-27T01:07:07.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.254+0000 7fb4bac89700 1 --2- 172.21.15.46:0/2628508908 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4b4159960 0x7fb4b406bcb0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.254+0000 7fb4bac89700 1 -- 172.21.15.46:0/2628508908 >> 172.21.15.46:0/2628508908 conn(0x7fb4b40ce530 msgr2=0x7fb4b40ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:07.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.254+0000 7fb4bac89700 1 -- 172.21.15.46:0/2628508908 shutdown_connections 2024-01-27T01:07:07.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.254+0000 7fb4bac89700 1 -- 172.21.15.46:0/2628508908 wait complete. 2024-01-27T01:07:07.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.255+0000 7fb4bac89700 1 Processor -- start 2024-01-27T01:07:07.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.255+0000 7fb4bac89700 1 -- start start 2024-01-27T01:07:07.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.256+0000 7fb4bac89700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4b406c1f0 0x7fb4b416a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:07.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.256+0000 7fb4bac89700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b4159590 0x7fb4b40b3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:07.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.256+0000 7fb4bac89700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4b4159960 0x7fb4b40c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:07.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.256+0000 7fb4bac89700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b40c65b0 con 0x7fb4b4159590 2024-01-27T01:07:07.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.256+0000 7fb4bac89700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b416ad40 con 0x7fb4b4159960 2024-01-27T01:07:07.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.256+0000 7fb4bac89700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b40c5ff0 con 0x7fb4b406c1f0 2024-01-27T01:07:07.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.256+0000 7fb4b3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b4159590 0x7fb4b40b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:07.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.256+0000 7fb4b3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b4159590 0x7fb4b40b3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59604/0 (socket says 172.21.15.46:59604) 2024-01-27T01:07:07.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.256+0000 7fb4b3fff700 1 -- 172.21.15.46:0/3743880998 learned_addr learned my addr 172.21.15.46:0/3743880998 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:07.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.256+0000 7fb4b9226700 1 --2- 172.21.15.46:0/3743880998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4b4159960 0x7fb4b40c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:07.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.256+0000 7fb4b8a25700 1 --2- 172.21.15.46:0/3743880998 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4b406c1f0 0x7fb4b416a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:07.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.257+0000 7fb4b9226700 1 -- 172.21.15.46:0/3743880998 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4b406c1f0 msgr2=0x7fb4b416a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:07.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.257+0000 7fb4b9226700 1 --2- 172.21.15.46:0/3743880998 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4b406c1f0 0x7fb4b416a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.257+0000 7fb4b9226700 1 -- 172.21.15.46:0/3743880998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b4159590 msgr2=0x7fb4b40b3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:07.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.257+0000 7fb4b9226700 1 --2- 172.21.15.46:0/3743880998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b4159590 0x7fb4b40b3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.257+0000 7fb4b9226700 1 -- 172.21.15.46:0/3743880998 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a8009420 con 0x7fb4b4159960 2024-01-27T01:07:07.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.257+0000 7fb4b3fff700 1 --2- 172.21.15.46:0/3743880998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b4159590 0x7fb4b40b3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:07.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.257+0000 7fb4b9226700 1 --2- 172.21.15.46:0/3743880998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4b4159960 0x7fb4b40c79d0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac001460 tx=0x7fb4ac001a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:07.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.257+0000 7fb4b0ff9700 1 -- 172.21.15.46:0/3743880998 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac00a840 con 0x7fb4b4159960 2024-01-27T01:07:07.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.257+0000 7fb4b0ff9700 1 -- 172.21.15.46:0/3743880998 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ac00a9a0 con 0x7fb4b4159960 2024-01-27T01:07:07.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.257+0000 7fb4bac89700 1 -- 172.21.15.46:0/3743880998 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b40c62d0 con 0x7fb4b4159960 2024-01-27T01:07:07.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.257+0000 7fb4b0ff9700 1 -- 172.21.15.46:0/3743880998 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac00ad40 con 0x7fb4b4159960 2024-01-27T01:07:07.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.257+0000 7fb4bac89700 1 -- 172.21.15.46:0/3743880998 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b40af860 con 0x7fb4b4159960 2024-01-27T01:07:07.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.258+0000 7fb4bac89700 1 -- 172.21.15.46:0/3743880998 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b404c710 con 0x7fb4b4159960 2024-01-27T01:07:07.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.259+0000 7fb4b0ff9700 1 -- 172.21.15.46:0/3743880998 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 132) v1 ==== 85003+0+0 (secure 0 0 0) 0x7fb4ac007300 con 0x7fb4b4159960 2024-01-27T01:07:07.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.259+0000 7fb4b0ff9700 1 --2- 172.21.15.46:0/3743880998 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb4a0067200 0x7fb4a00696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:07.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.259+0000 7fb4b8a25700 1 -- 172.21.15.46:0/3743880998 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb4a0067200 msgr2=0x7fb4a00696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:07.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.259+0000 7fb4b0ff9700 1 -- 172.21.15.46:0/3743880998 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fb4ac085430 con 0x7fb4b4159960 2024-01-27T01:07:07.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.259+0000 7fb4b8a25700 1 --2- 172.21.15.46:0/3743880998 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb4a0067200 0x7fb4a00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:07.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.267+0000 7fb4b0ff9700 1 -- 172.21.15.46:0/3743880998 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb4ac051c00 con 0x7fb4b4159960 2024-01-27T01:07:07.326 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:07.326+0000 7f8dd1d5c1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:07:07.375 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.376+0000 7f09caa5b1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:07:07.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.460+0000 7fb4b8a25700 1 -- 172.21.15.46:0/3743880998 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb4a0067200 msgr2=0x7fb4a00696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:07.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.460+0000 7fb4b8a25700 1 --2- 172.21.15.46:0/3743880998 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb4a0067200 0x7fb4a00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:07.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.520+0000 7fb4bac89700 1 -- 172.21.15.46:0/3743880998 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4b406bcb0 con 0x7fb4b4159960 2024-01-27T01:07:07.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.521+0000 7fb4b0ff9700 1 -- 172.21.15.46:0/3743880998 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5697 (secure 0 0 0) 0x7fb4ac055220 con 0x7fb4b4159960 2024-01-27T01:07:07.521 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:07.521 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":122,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/1999300009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1999300009},{"type":"v1","addr":"172.21.15.106:6835","nonce":1999300009}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":110},{"gid":10486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/4174263148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4174263148},{"type":"v1","addr":"172.21.15.106:6833","nonce":4174263148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:06:26.286041+0000","modified":"2024-01-27T01:06:29.873402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10866},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10866":{"gid":10866,"name":"a","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2483163449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2483163449},{"type":"v1","addr":"172.21.15.46:6835","nonce":2483163449}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:06:40.353248+0000","modified":"2024-01-27T01:06:43.938885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10854":{"gid":10854,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":8,"addr":"172.21.15.46:6833/1263484152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1263484152},{"type":"v1","addr":"172.21.15.46:6833","nonce":1263484152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-27T01:07:07.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.523+0000 7fb4bac89700 1 -- 172.21.15.46:0/3743880998 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb4a0067200 msgr2=0x7fb4a00696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:07.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.523+0000 7fb4bac89700 1 --2- 172.21.15.46:0/3743880998 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb4a0067200 0x7fb4a00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.523+0000 7fb4bac89700 1 -- 172.21.15.46:0/3743880998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4b4159960 msgr2=0x7fb4b40c79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:07.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.523+0000 7fb4bac89700 1 --2- 172.21.15.46:0/3743880998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4b4159960 0x7fb4b40c79d0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac001460 tx=0x7fb4ac001a60 comp rx=0 tx=0).stop 2024-01-27T01:07:07.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.523+0000 7fb4bac89700 1 -- 172.21.15.46:0/3743880998 shutdown_connections 2024-01-27T01:07:07.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.523+0000 7fb4bac89700 1 --2- 172.21.15.46:0/3743880998 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb4a0067200 0x7fb4a00696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.523+0000 7fb4bac89700 1 --2- 172.21.15.46:0/3743880998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4b4159960 0x7fb4b40c79d0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.523+0000 7fb4bac89700 1 --2- 172.21.15.46:0/3743880998 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4b406c1f0 0x7fb4b416a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.523+0000 7fb4bac89700 1 --2- 172.21.15.46:0/3743880998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b4159590 0x7fb4b40b3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.523+0000 7fb4bac89700 1 -- 172.21.15.46:0/3743880998 >> 172.21.15.46:0/3743880998 conn(0x7fb4b40ce530 msgr2=0x7fb4b40cda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:07.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.523+0000 7fb4bac89700 1 -- 172.21.15.46:0/3743880998 shutdown_connections 2024-01-27T01:07:07.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.523+0000 7fb4bac89700 1 -- 172.21.15.46:0/3743880998 wait complete. 2024-01-27T01:07:07.527 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 122 2024-01-27T01:07:07.537 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:07:07.597 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.597+0000 7f09caa5b1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:07:07.671 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.671+0000 7f09caa5b1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:07:07.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.714+0000 7fa200d5b700 1 Processor -- start 2024-01-27T01:07:07.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.714+0000 7fa200d5b700 1 -- start start 2024-01-27T01:07:07.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.715+0000 7fa200d5b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1fc15ce40 0x7fa1fc0d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:07.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.715+0000 7fa200d5b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa1fc15d210 0x7fa1fc0cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:07.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.715+0000 7fa200d5b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1fc0cff70 con 0x7fa1fc0d1390 2024-01-27T01:07:07.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.715+0000 7fa200d5b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1fc0cf400 con 0x7fa1fc15d210 2024-01-27T01:07:07.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.715+0000 7fa200d5b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc0cf660 con 0x7fa1fc15ce40 2024-01-27T01:07:07.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.715+0000 7fa1fad9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa1fc0d1390 0x7fa1fc0ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46348/0 (socket says 172.21.15.46:46348) 2024-01-27T01:07:07.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.715+0000 7fa1fad9d700 1 -- 172.21.15.46:0/3100736150 learned_addr learned my addr 172.21.15.46:0/3100736150 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:07.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.716+0000 7fa1f9d9b700 1 --2- 172.21.15.46:0/3100736150 >> v2:172.21.15.106:3300/0 conn(0x7fa1fc15d210 0x7fa1fc0cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:07.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.716+0000 7fa1fa59c700 1 --2- 172.21.15.46:0/3100736150 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1fc15ce40 0x7fa1fc0d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:07.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.716+0000 7fa1f959a700 1 -- 172.21.15.46:0/3100736150 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850159967 0 0) 0x7fa1fc0cff70 con 0x7fa1fc0d1390 2024-01-27T01:07:07.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.716+0000 7fa1f959a700 1 -- 172.21.15.46:0/3100736150 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1e4003570 con 0x7fa1fc0d1390 2024-01-27T01:07:07.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.716+0000 7fa1f9d9b700 1 -- 172.21.15.46:0/3100736150 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1fc15ce40 msgr2=0x7fa1fc0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:07.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.716+0000 7fa1f9d9b700 1 --2- 172.21.15.46:0/3100736150 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1fc15ce40 0x7fa1fc0d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.716+0000 7fa1f9d9b700 1 -- 172.21.15.46:0/3100736150 >> v1:172.21.15.46:6789/0 conn(0x7fa1fc0d1390 legacy=0x7fa1fc0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:07.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.716+0000 7fa1f9d9b700 1 -- 172.21.15.46:0/3100736150 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1fc0d0500 con 0x7fa1fc15d210 2024-01-27T01:07:07.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.716+0000 7fa1f9d9b700 1 --2- 172.21.15.46:0/3100736150 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1fc15d210 0x7fa1fc0cbab0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00098e0 tx=0x7fa1f00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf4c62ff82c0c17b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:07.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.716+0000 7fa1f959a700 1 -- 172.21.15.46:0/3100736150 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0011630 con 0x7fa1fc15d210 2024-01-27T01:07:07.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.716+0000 7fa1f959a700 1 -- 172.21.15.46:0/3100736150 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f0011790 con 0x7fa1fc15d210 2024-01-27T01:07:07.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.716+0000 7fa1f959a700 1 -- 172.21.15.46:0/3100736150 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0011b30 con 0x7fa1fc15d210 2024-01-27T01:07:07.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.717+0000 7fa200d5b700 1 -- 172.21.15.46:0/3100736150 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1fc15d210 msgr2=0x7fa1fc0cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:07.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.717+0000 7fa200d5b700 1 --2- 172.21.15.46:0/3100736150 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1fc15d210 0x7fa1fc0cbab0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00098e0 tx=0x7fa1f00091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.717+0000 7fa200d5b700 1 -- 172.21.15.46:0/3100736150 shutdown_connections 2024-01-27T01:07:07.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.717+0000 7fa200d5b700 1 --2- 172.21.15.46:0/3100736150 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1fc15ce40 0x7fa1fc0d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.717+0000 7fa200d5b700 1 --2- 172.21.15.46:0/3100736150 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1fc15d210 0x7fa1fc0cbab0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.717+0000 7fa200d5b700 1 -- 172.21.15.46:0/3100736150 >> 172.21.15.46:0/3100736150 conn(0x7fa1fc06f840 msgr2=0x7fa1fc0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:07.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.717+0000 7fa200d5b700 1 -- 172.21.15.46:0/3100736150 shutdown_connections 2024-01-27T01:07:07.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.717+0000 7fa200d5b700 1 -- 172.21.15.46:0/3100736150 wait complete. 2024-01-27T01:07:07.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.719+0000 7fa200d5b700 1 Processor -- start 2024-01-27T01:07:07.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.719+0000 7fa200d5b700 1 -- start start 2024-01-27T01:07:07.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.719+0000 7fa200d5b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1fc0d1390 0x7fa1fc16c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:07.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.719+0000 7fa200d5b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1fc15ce40 0x7fa1fc0b5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:07.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.719+0000 7fa200d5b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1fc15d210 0x7fa1fc16a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:07.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.719+0000 7fa200d5b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc16cb00 con 0x7fa1fc0d1390 2024-01-27T01:07:07.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.719+0000 7fa200d5b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc16c130 con 0x7fa1fc15d210 2024-01-27T01:07:07.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.719+0000 7fa200d5b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc16b520 con 0x7fa1fc15ce40 2024-01-27T01:07:07.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.720+0000 7fa1fa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1fc0d1390 0x7fa1fc16c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:07.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.720+0000 7fa1fa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1fc0d1390 0x7fa1fc16c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59606/0 (socket says 172.21.15.46:59606) 2024-01-27T01:07:07.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.720+0000 7fa1fa59c700 1 -- 172.21.15.46:0/2852839792 learned_addr learned my addr 172.21.15.46:0/2852839792 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:07.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.720+0000 7fa1fad9d700 1 --2- 172.21.15.46:0/2852839792 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1fc15d210 0x7fa1fc16a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:07.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.720+0000 7fa1f9d9b700 1 --2- 172.21.15.46:0/2852839792 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1fc15ce40 0x7fa1fc0b5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:07.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.720+0000 7fa1fa59c700 1 -- 172.21.15.46:0/2852839792 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1fc15ce40 msgr2=0x7fa1fc0b5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:07.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.720+0000 7fa1fa59c700 1 --2- 172.21.15.46:0/2852839792 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1fc15ce40 0x7fa1fc0b5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.720+0000 7fa1fa59c700 1 -- 172.21.15.46:0/2852839792 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1fc15d210 msgr2=0x7fa1fc16a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:07.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.720+0000 7fa1fa59c700 1 --2- 172.21.15.46:0/2852839792 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1fc15d210 0x7fa1fc16a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.720+0000 7fa1fa59c700 1 -- 172.21.15.46:0/2852839792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f0009420 con 0x7fa1fc0d1390 2024-01-27T01:07:07.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.720+0000 7fa1f9d9b700 1 --2- 172.21.15.46:0/2852839792 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1fc15ce40 0x7fa1fc0b5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:07.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.720+0000 7fa1fad9d700 1 --2- 172.21.15.46:0/2852839792 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1fc15d210 0x7fa1fc16a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:07.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.720+0000 7fa1fa59c700 1 --2- 172.21.15.46:0/2852839792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1fc0d1390 0x7fa1fc16c5c0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009f40 tx=0x7fa1ec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:07.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.721+0000 7fa1ea7fc700 1 -- 172.21.15.46:0/2852839792 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec00cb70 con 0x7fa1fc0d1390 2024-01-27T01:07:07.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.721+0000 7fa1ea7fc700 1 -- 172.21.15.46:0/2852839792 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1ec00f040 con 0x7fa1fc0d1390 2024-01-27T01:07:07.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.721+0000 7fa200d5b700 1 -- 172.21.15.46:0/2852839792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1fc16b800 con 0x7fa1fc0d1390 2024-01-27T01:07:07.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.721+0000 7fa1ea7fc700 1 -- 172.21.15.46:0/2852839792 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec014480 con 0x7fa1fc0d1390 2024-01-27T01:07:07.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.721+0000 7fa200d5b700 1 -- 172.21.15.46:0/2852839792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1fc16bdc0 con 0x7fa1fc0d1390 2024-01-27T01:07:07.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.722+0000 7fa200d5b700 1 -- 172.21.15.46:0/2852839792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1fc04c710 con 0x7fa1fc0d1390 2024-01-27T01:07:07.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.723+0000 7fa1ea7fc700 1 -- 172.21.15.46:0/2852839792 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 132) v1 ==== 85003+0+0 (secure 0 0 0) 0x7fa1ec0147f0 con 0x7fa1fc0d1390 2024-01-27T01:07:07.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.723+0000 7fa1ea7fc700 1 --2- 172.21.15.46:0/2852839792 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa1dc067200 0x7fa1dc0696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:07.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.723+0000 7fa1f9d9b700 1 -- 172.21.15.46:0/2852839792 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa1dc067200 msgr2=0x7fa1dc0696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:07.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.723+0000 7fa1ea7fc700 1 -- 172.21.15.46:0/2852839792 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7639+0+0 (secure 0 0 0) 0x7fa1ec085960 con 0x7fa1fc0d1390 2024-01-27T01:07:07.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.723+0000 7fa1f9d9b700 1 --2- 172.21.15.46:0/2852839792 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa1dc067200 0x7fa1dc0696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:07.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.732+0000 7fa1ea7fc700 1 -- 172.21.15.46:0/2852839792 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa1ec051fa0 con 0x7fa1fc0d1390 2024-01-27T01:07:07.831 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.831+0000 7f09caa5b1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:07:07.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.857+0000 7fa1ea7fc700 1 -- 172.21.15.46:0/2852839792 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 133) v1 ==== 84453+0+0 (secure 0 0 0) 0x7fa1ec052370 con 0x7fa1fc0d1390 2024-01-27T01:07:07.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.857+0000 7fa1ea7fc700 1 -- 172.21.15.46:0/2852839792 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa1dc067200 msgr2=0x7fa1dc0696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:07.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.857+0000 7fa1ea7fc700 1 --2- 172.21.15.46:0/2852839792 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa1dc067200 0x7fa1dc0696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.916 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.916+0000 7f0946b44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.916 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.916+0000 7f0946b44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.916 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.916+0000 7f0946b44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.916 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.916+0000 7f0946b44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.916 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.916+0000 7f0946b44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.917 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.917+0000 7f0949b4a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.917 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.917+0000 7f0949b4a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.917 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.917+0000 7f0949b4a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.917 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.917+0000 7f0949b4a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.917 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.917+0000 7f0949b4a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.925 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.925+0000 7f0946b44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.925 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.925+0000 7f0946b44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.925 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.925+0000 7f0946b44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.925 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.925+0000 7f0946b44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.925 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.925+0000 7f0946b44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.926 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.927+0000 7f0949b4a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.926 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.927+0000 7f0949b4a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.926 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.927+0000 7f0949b4a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.926 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.927+0000 7f0949b4a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.926 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:07.927+0000 7f0949b4a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.947+0000 7fa200d5b700 1 -- 172.21.15.46:0/2852839792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1fc06fe70 con 0x7fa1fc0d1390 2024-01-27T01:07:07.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.948+0000 7fa1ea7fc700 1 -- 172.21.15.46:0/2852839792 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) v1 ==== 74+0+20491 (secure 0 0 0) 0x7fa1ec06fa00 con 0x7fa1fc0d1390 2024-01-27T01:07:07.949 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:07.949 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":229,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:07:07.849216+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:06:24.798791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-01-27T01:06:25.249215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:06:38.156934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:06:39.330686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/2812454604":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/3610030821":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/3431540914":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/42206641":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/3926539126":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/694514533":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/819986778":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/3809430766":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/981929816":"2024-01-28T01:07:07.849128+0000","172.21.15.106:6837/62434":"2024-01-28T01:06:57.756975+0000","172.21.15.106:6836/62434":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/524220301":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/4002309435":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/3400107086":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/2725250616":"2024-01-28T01:06:57.756975+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:07:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.951+0000 7fa200d5b700 1 -- 172.21.15.46:0/2852839792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1fc0d1390 msgr2=0x7fa1fc16c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.951+0000 7fa200d5b700 1 --2- 172.21.15.46:0/2852839792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1fc0d1390 0x7fa1fc16c5c0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009f40 tx=0x7fa1ec00b040 comp rx=0 tx=0).stop 2024-01-27T01:07:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.951+0000 7fa200d5b700 1 -- 172.21.15.46:0/2852839792 shutdown_connections 2024-01-27T01:07:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.951+0000 7fa200d5b700 1 --2- 172.21.15.46:0/2852839792 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa1dc067200 0x7fa1dc0696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.951+0000 7fa200d5b700 1 --2- 172.21.15.46:0/2852839792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1fc0d1390 0x7fa1fc16c5c0 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.951+0000 7fa200d5b700 1 --2- 172.21.15.46:0/2852839792 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1fc15d210 0x7fa1fc16a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.951+0000 7fa200d5b700 1 --2- 172.21.15.46:0/2852839792 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa1fc15ce40 0x7fa1fc0b5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.951+0000 7fa200d5b700 1 -- 172.21.15.46:0/2852839792 >> 172.21.15.46:0/2852839792 conn(0x7fa1fc06f840 msgr2=0x7fa1fc0b41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:07.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.951+0000 7fa200d5b700 1 -- 172.21.15.46:0/2852839792 shutdown_connections 2024-01-27T01:07:07.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:07.951+0000 7fa200d5b700 1 -- 172.21.15.46:0/2852839792 wait complete. 2024-01-27T01:07:07.965 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-27T01:07:08.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.100+0000 7ff2de1ca700 1 Processor -- start 2024-01-27T01:07:08.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.100+0000 7ff2de1ca700 1 -- start start 2024-01-27T01:07:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.101+0000 7ff2de1ca700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff2d80cf460 0x7ff2d80ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.101+0000 7ff2de1ca700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d815a0b0 con 0x7ff2d80b2b40 2024-01-27T01:07:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.101+0000 7ff2de1ca700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d80ae4d0 con 0x7ff2d80c6520 2024-01-27T01:07:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.101+0000 7ff2de1ca700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2d80af070 con 0x7ff2d80cf460 2024-01-27T01:07:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.101+0000 7ff2d77fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff2d80b2b40 0x7ff2d80ae960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46358/0 (socket says 172.21.15.46:46358) 2024-01-27T01:07:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.101+0000 7ff2d77fe700 1 -- 172.21.15.46:0/159661024 learned_addr learned my addr 172.21.15.46:0/159661024 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.101+0000 7ff2d7fff700 1 --2- 172.21.15.46:0/159661024 >> v2:172.21.15.106:3300/0 conn(0x7ff2d80cf460 0x7ff2d80ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.101+0000 7ff2d67fc700 1 -- 172.21.15.46:0/159661024 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644986776 0 0) 0x7ff2d815a0b0 con 0x7ff2d80b2b40 2024-01-27T01:07:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.101+0000 7ff2d67fc700 1 -- 172.21.15.46:0/159661024 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2c0003570 con 0x7ff2d80b2b40 2024-01-27T01:07:08.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.101+0000 7ff2d7fff700 1 -- 172.21.15.46:0/159661024 >> v1:172.21.15.106:6789/0 conn(0x7ff2d80c6520 legacy=0x7ff2d80ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:08.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.101+0000 7ff2d67fc700 1 -- 172.21.15.46:0/159661024 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735055712 0 0) 0x7ff2d80ae4d0 con 0x7ff2d80c6520 2024-01-27T01:07:08.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.101+0000 7ff2d7fff700 1 -- 172.21.15.46:0/159661024 >> v1:172.21.15.46:6789/0 conn(0x7ff2d80b2b40 legacy=0x7ff2d80ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:08.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.101+0000 7ff2d7fff700 1 -- 172.21.15.46:0/159661024 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d80ad8c0 con 0x7ff2d80cf460 2024-01-27T01:07:08.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.102+0000 7ff2d7fff700 1 --2- 172.21.15.46:0/159661024 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2d80cf460 0x7ff2d80ad1c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0009e60 tx=0x7ff2d0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bdaf641d950bc5e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:08.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.102+0000 7ff2d67fc700 1 -- 172.21.15.46:0/159661024 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d000e070 con 0x7ff2d80cf460 2024-01-27T01:07:08.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.102+0000 7ff2d67fc700 1 -- 172.21.15.46:0/159661024 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d000bb60 con 0x7ff2d80cf460 2024-01-27T01:07:08.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.102+0000 7ff2d67fc700 1 -- 172.21.15.46:0/159661024 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d0012430 con 0x7ff2d80cf460 2024-01-27T01:07:08.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.102+0000 7ff2de1ca700 1 -- 172.21.15.46:0/159661024 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2d80cf460 msgr2=0x7ff2d80ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:08.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.102+0000 7ff2de1ca700 1 --2- 172.21.15.46:0/159661024 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2d80cf460 0x7ff2d80ad1c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0009e60 tx=0x7ff2d0009780 comp rx=0 tx=0).stop 2024-01-27T01:07:08.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.103+0000 7ff2de1ca700 1 -- 172.21.15.46:0/159661024 shutdown_connections 2024-01-27T01:07:08.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.103+0000 7ff2de1ca700 1 --2- 172.21.15.46:0/159661024 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2d80cf460 0x7ff2d80ad1c0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:08.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.103+0000 7ff2de1ca700 1 -- 172.21.15.46:0/159661024 >> 172.21.15.46:0/159661024 conn(0x7ff2d8099a30 msgr2=0x7ff2d806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:08.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.103+0000 7ff2de1ca700 1 -- 172.21.15.46:0/159661024 shutdown_connections 2024-01-27T01:07:08.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.103+0000 7ff2de1ca700 1 -- 172.21.15.46:0/159661024 wait complete. 2024-01-27T01:07:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.104+0000 7ff2de1ca700 1 Processor -- start 2024-01-27T01:07:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.104+0000 7ff2de1ca700 1 -- start start 2024-01-27T01:07:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.105+0000 7ff2de1ca700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2d80b2b40 0x7ff2d80af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.105+0000 7ff2de1ca700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2d80c6520 0x7ff2d814eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.105+0000 7ff2de1ca700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2d80cf460 0x7ff2d80c85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.105+0000 7ff2de1ca700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d80b4590 con 0x7ff2d80c6520 2024-01-27T01:07:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.105+0000 7ff2de1ca700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d80b42b0 con 0x7ff2d80cf460 2024-01-27T01:07:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.105+0000 7ff2de1ca700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d80b4ab0 con 0x7ff2d80b2b40 2024-01-27T01:07:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.105+0000 7ff2d6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2d80c6520 0x7ff2d814eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.105+0000 7ff2d6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2d80c6520 0x7ff2d814eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59612/0 (socket says 172.21.15.46:59612) 2024-01-27T01:07:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.105+0000 7ff2d6ffd700 1 -- 172.21.15.46:0/3561901737 learned_addr learned my addr 172.21.15.46:0/3561901737 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.105+0000 7ff2d7fff700 1 --2- 172.21.15.46:0/3561901737 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2d80cf460 0x7ff2d80c85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.105+0000 7ff2d77fe700 1 --2- 172.21.15.46:0/3561901737 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2d80b2b40 0x7ff2d80af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.105+0000 7ff2d6ffd700 1 -- 172.21.15.46:0/3561901737 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2d80b2b40 msgr2=0x7ff2d80af1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.105+0000 7ff2d6ffd700 1 --2- 172.21.15.46:0/3561901737 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2d80b2b40 0x7ff2d80af1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.105+0000 7ff2d6ffd700 1 -- 172.21.15.46:0/3561901737 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2d80cf460 msgr2=0x7ff2d80c85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:08.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.105+0000 7ff2d6ffd700 1 --2- 172.21.15.46:0/3561901737 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2d80cf460 0x7ff2d80c85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:08.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.105+0000 7ff2d6ffd700 1 -- 172.21.15.46:0/3561901737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d0009a20 con 0x7ff2d80c6520 2024-01-27T01:07:08.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.105+0000 7ff2d7fff700 1 --2- 172.21.15.46:0/3561901737 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2d80cf460 0x7ff2d80c85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:08.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.106+0000 7ff2d6ffd700 1 --2- 172.21.15.46:0/3561901737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2d80c6520 0x7ff2d814eba0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00c7e0 tx=0x7ff2cc007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:08.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.106+0000 7ff2bf7fe700 1 -- 172.21.15.46:0/3561901737 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc0105c0 con 0x7ff2d80c6520 2024-01-27T01:07:08.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.106+0000 7ff2bf7fe700 1 -- 172.21.15.46:0/3561901737 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2cc010740 con 0x7ff2d80c6520 2024-01-27T01:07:08.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.106+0000 7ff2bf7fe700 1 -- 172.21.15.46:0/3561901737 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc010b00 con 0x7ff2d80c6520 2024-01-27T01:07:08.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.106+0000 7ff2de1ca700 1 -- 172.21.15.46:0/3561901737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d80b4d90 con 0x7ff2d80c6520 2024-01-27T01:07:08.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.106+0000 7ff2de1ca700 1 -- 172.21.15.46:0/3561901737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d814f0e0 con 0x7ff2d80c6520 2024-01-27T01:07:08.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.107+0000 7ff2de1ca700 1 -- 172.21.15.46:0/3561901737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2d814f360 con 0x7ff2d80c6520 2024-01-27T01:07:08.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.107+0000 7ff2bf7fe700 1 -- 172.21.15.46:0/3561901737 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 133) v1 ==== 84453+0+0 (secure 0 0 0) 0x7ff2cc01b3b0 con 0x7ff2d80c6520 2024-01-27T01:07:08.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.108+0000 7ff2bf7fe700 1 -- 172.21.15.46:0/3561901737 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 8026+0+0 (secure 0 0 0) 0x7ff2cc083f40 con 0x7ff2d80c6520 2024-01-27T01:07:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.112+0000 7ff2bf7fe700 1 -- 172.21.15.46:0/3561901737 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff2cc050c60 con 0x7ff2d80c6520 2024-01-27T01:07:08.195 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:08.195+0000 7f8dd1d5c1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:07:08.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.438+0000 7ff2de1ca700 1 -- 172.21.15.46:0/3561901737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff2d804c710 con 0x7ff2d80c6520 2024-01-27T01:07:08.438 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-27T01:07:08.438+0000 7f33ffe86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-27T01:07:08.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.444+0000 7ff2bf7fe700 1 -- 172.21.15.46:0/3561901737 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v123) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff2cc017030 con 0x7ff2d80c6520 2024-01-27T01:07:08.446 INFO:tasks.ceph.mds.a.smithi046.stderr: -3427> 2024-01-27T01:06:31.833+0000 7f60d04b8700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-27T01:07:08.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.447+0000 7ff2de1ca700 1 -- 172.21.15.46:0/3561901737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2d80c6520 msgr2=0x7ff2d814eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:08.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.447+0000 7ff2de1ca700 1 --2- 172.21.15.46:0/3561901737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2d80c6520 0x7ff2d814eba0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00c7e0 tx=0x7ff2cc007df0 comp rx=0 tx=0).stop 2024-01-27T01:07:08.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.447+0000 7ff2de1ca700 1 -- 172.21.15.46:0/3561901737 shutdown_connections 2024-01-27T01:07:08.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.447+0000 7ff2de1ca700 1 --2- 172.21.15.46:0/3561901737 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2d80cf460 0x7ff2d80c85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:08.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.447+0000 7ff2de1ca700 1 --2- 172.21.15.46:0/3561901737 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2d80b2b40 0x7ff2d80af1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:08.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.447+0000 7ff2de1ca700 1 --2- 172.21.15.46:0/3561901737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2d80c6520 0x7ff2d814eba0 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:08.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.447+0000 7ff2de1ca700 1 -- 172.21.15.46:0/3561901737 >> 172.21.15.46:0/3561901737 conn(0x7ff2d8099a30 msgr2=0x7ff2d80c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:08.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.448+0000 7ff2de1ca700 1 -- 172.21.15.46:0/3561901737 shutdown_connections 2024-01-27T01:07:08.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.448+0000 7ff2de1ca700 1 -- 172.21.15.46:0/3561901737 wait complete. 2024-01-27T01:07:08.449 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-27T01:07:08.468 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-27T01:07:08.473 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-27T01:07:08.484 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:08.484+0000 7f8dd1d5c1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:07:08.574 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:08.574+0000 7f8dd1d5c1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:07:08.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.606+0000 7fcbd3afe700 1 Processor -- start 2024-01-27T01:07:08.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.606+0000 7fcbd3afe700 1 -- start start 2024-01-27T01:07:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.608+0000 7fcbd3afe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fcbcc159da0 0x7fcbcc0ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.608+0000 7fcbd3afe700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbcc15a200 con 0x7fcbcc15d8c0 2024-01-27T01:07:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.608+0000 7fcbd3afe700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbcc0b1a60 con 0x7fcbcc1599d0 2024-01-27T01:07:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.608+0000 7fcbd3afe700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbcc0b11c0 con 0x7fcbcc159da0 2024-01-27T01:07:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.608+0000 7fcbd1099700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fcbcc159da0 0x7fcbcc0ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.608+0000 7fcbd209b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcbcc15d8c0 0x7fcbcc0b3b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46372/0 (socket says 172.21.15.46:46372) 2024-01-27T01:07:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.608+0000 7fcbd209b700 1 -- 172.21.15.46:0/1764450841 learned_addr learned my addr 172.21.15.46:0/1764450841 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.608+0000 7fcbd189a700 1 --1- 172.21.15.46:0/1764450841 >> v1:172.21.15.106:6789/0 conn(0x7fcbcc1599d0 0x7fcbcc0b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:56528/0 (socket says 172.21.15.46:56528) 2024-01-27T01:07:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.608+0000 7fcbd0898700 1 -- 172.21.15.46:0/1764450841 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418339780 0 0) 0x7fcbcc15a200 con 0x7fcbcc15d8c0 2024-01-27T01:07:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.608+0000 7fcbd1099700 1 -- 172.21.15.46:0/1764450841 >> v1:172.21.15.106:6789/0 conn(0x7fcbcc1599d0 legacy=0x7fcbcc0b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.608+0000 7fcbd1099700 1 -- 172.21.15.46:0/1764450841 >> v1:172.21.15.46:6789/0 conn(0x7fcbcc15d8c0 legacy=0x7fcbcc0b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.608+0000 7fcbd1099700 1 -- 172.21.15.46:0/1764450841 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc0b14a0 con 0x7fcbcc159da0 2024-01-27T01:07:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.608+0000 7fcbd1099700 1 --2- 172.21.15.46:0/1764450841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbcc159da0 0x7fcbcc0ab1c0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009860 tx=0x7fcbbc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ac32eeddd7120f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.609+0000 7fcbd0898700 1 -- 172.21.15.46:0/1764450841 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc011630 con 0x7fcbcc159da0 2024-01-27T01:07:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.609+0000 7fcbd0898700 1 -- 172.21.15.46:0/1764450841 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbbc011790 con 0x7fcbcc159da0 2024-01-27T01:07:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.609+0000 7fcbd0898700 1 -- 172.21.15.46:0/1764450841 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc011b50 con 0x7fcbcc159da0 2024-01-27T01:07:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.609+0000 7fcbd3afe700 1 -- 172.21.15.46:0/1764450841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbcc159da0 msgr2=0x7fcbcc0ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.609+0000 7fcbd3afe700 1 --2- 172.21.15.46:0/1764450841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbcc159da0 0x7fcbcc0ab1c0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009860 tx=0x7fcbbc0091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.609+0000 7fcbd3afe700 1 -- 172.21.15.46:0/1764450841 shutdown_connections 2024-01-27T01:07:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.609+0000 7fcbd3afe700 1 --2- 172.21.15.46:0/1764450841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbcc159da0 0x7fcbcc0ab1c0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.609+0000 7fcbd3afe700 1 -- 172.21.15.46:0/1764450841 >> 172.21.15.46:0/1764450841 conn(0x7fcbcc06f840 msgr2=0x7fcbcc087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.609+0000 7fcbd3afe700 1 -- 172.21.15.46:0/1764450841 shutdown_connections 2024-01-27T01:07:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.609+0000 7fcbd3afe700 1 -- 172.21.15.46:0/1764450841 wait complete. 2024-01-27T01:07:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.610+0000 7fcbd3afe700 1 Processor -- start 2024-01-27T01:07:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.611+0000 7fcbd3afe700 1 -- start start 2024-01-27T01:07:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.611+0000 7fcbd3afe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbcc1599d0 0x7fcbcc1574f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.611+0000 7fcbd3afe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbcc159da0 0x7fcbcc155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.611+0000 7fcbd3afe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbcc15d8c0 0x7fcbcc0cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.611+0000 7fcbd3afe700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc1561a0 con 0x7fcbcc15d8c0 2024-01-27T01:07:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.611+0000 7fcbd3afe700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc0ad0d0 con 0x7fcbcc159da0 2024-01-27T01:07:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.611+0000 7fcbd3afe700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc0acda0 con 0x7fcbcc1599d0 2024-01-27T01:07:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.611+0000 7fcbd209b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbcc15d8c0 0x7fcbcc0cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.611+0000 7fcbd1099700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbcc159da0 0x7fcbcc155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.611+0000 7fcbd209b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbcc15d8c0 0x7fcbcc0cfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59636/0 (socket says 172.21.15.46:59636) 2024-01-27T01:07:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.611+0000 7fcbd209b700 1 -- 172.21.15.46:0/1454856635 learned_addr learned my addr 172.21.15.46:0/1454856635 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.612+0000 7fcbd189a700 1 --2- 172.21.15.46:0/1454856635 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbcc1599d0 0x7fcbcc1574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.612+0000 7fcbd209b700 1 -- 172.21.15.46:0/1454856635 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbcc1599d0 msgr2=0x7fcbcc1574f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.612+0000 7fcbd209b700 1 --2- 172.21.15.46:0/1454856635 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbcc1599d0 0x7fcbcc1574f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.612+0000 7fcbd209b700 1 -- 172.21.15.46:0/1454856635 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbcc159da0 msgr2=0x7fcbcc155110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.612+0000 7fcbd209b700 1 --2- 172.21.15.46:0/1454856635 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbcc159da0 0x7fcbcc155110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.612+0000 7fcbd209b700 1 -- 172.21.15.46:0/1454856635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc009420 con 0x7fcbcc15d8c0 2024-01-27T01:07:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.612+0000 7fcbd189a700 1 --2- 172.21.15.46:0/1454856635 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbcc1599d0 0x7fcbcc1574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.612+0000 7fcbd1099700 1 --2- 172.21.15.46:0/1454856635 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbcc159da0 0x7fcbcc155110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.612+0000 7fcbd209b700 1 --2- 172.21.15.46:0/1454856635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbcc15d8c0 0x7fcbcc0cfd70 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fcbc400cc20 tx=0x7fcbc40019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.612+0000 7fcbc1ffb700 1 -- 172.21.15.46:0/1454856635 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc400a870 con 0x7fcbcc15d8c0 2024-01-27T01:07:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.612+0000 7fcbc1ffb700 1 -- 172.21.15.46:0/1454856635 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc400a9d0 con 0x7fcbcc15d8c0 2024-01-27T01:07:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.612+0000 7fcbd3afe700 1 -- 172.21.15.46:0/1454856635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbcc155650 con 0x7fcbcc15d8c0 2024-01-27T01:07:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.612+0000 7fcbc1ffb700 1 -- 172.21.15.46:0/1454856635 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc400ad90 con 0x7fcbcc15d8c0 2024-01-27T01:07:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.613+0000 7fcbd3afe700 1 -- 172.21.15.46:0/1454856635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbcc155be0 con 0x7fcbcc15d8c0 2024-01-27T01:07:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.614+0000 7fcbd3afe700 1 -- 172.21.15.46:0/1454856635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbcc04c710 con 0x7fcbcc15d8c0 2024-01-27T01:07:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.614+0000 7fcbc1ffb700 1 -- 172.21.15.46:0/1454856635 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 133) v1 ==== 84453+0+0 (secure 0 0 0) 0x7fcbc4007300 con 0x7fcbcc15d8c0 2024-01-27T01:07:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.615+0000 7fcbc1ffb700 1 -- 172.21.15.46:0/1454856635 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 8112+0+0 (secure 0 0 0) 0x7fcbc4084d00 con 0x7fcbcc15d8c0 2024-01-27T01:07:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.619+0000 7fcbc1ffb700 1 -- 172.21.15.46:0/1454856635 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcbc4051980 con 0x7fcbcc15d8c0 2024-01-27T01:07:08.760 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:08.760+0000 7f8dd1d5c1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:07:08.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.859+0000 7fcbc1ffb700 1 -- 172.21.15.46:0/1454856635 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 134) v1 ==== 84580+0+0 (secure 0 0 0) 0x7fcbc4051430 con 0x7fcbcc15d8c0 2024-01-27T01:07:08.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.859+0000 7fcbc1ffb700 1 --2- 172.21.15.46:0/1454856635 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcbb006f930 0x7fcbb0071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:08.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.860+0000 7fcbd189a700 1 --2- 172.21.15.46:0/1454856635 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcbb006f930 0x7fcbb0071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:08.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.860+0000 7fcbd189a700 1 --2- 172.21.15.46:0/1454856635 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcbb006f930 0x7fcbb0071d10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8000fd0 tx=0x7fcbc8007ea0 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:08.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:08.879+0000 7fcbd3afe700 1 -- 172.21.15.46:0/1454856635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcbcc0ad210 con 0x7fcbcc15d8c0 2024-01-27T01:07:09.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.445+0000 7fcbc1ffb700 1 -- 172.21.15.46:0/1454856635 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v124) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcbc4080f80 con 0x7fcbcc15d8c0 2024-01-27T01:07:09.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.447+0000 7fcbd3afe700 1 -- 172.21.15.46:0/1454856635 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcbb006f930 msgr2=0x7fcbb0071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:09.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.447+0000 7fcbd3afe700 1 --2- 172.21.15.46:0/1454856635 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcbb006f930 0x7fcbb0071d10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8000fd0 tx=0x7fcbc8007ea0 comp rx=0 tx=0).stop 2024-01-27T01:07:09.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.447+0000 7fcbd3afe700 1 -- 172.21.15.46:0/1454856635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbcc15d8c0 msgr2=0x7fcbcc0cfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:09.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.447+0000 7fcbd3afe700 1 --2- 172.21.15.46:0/1454856635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbcc15d8c0 0x7fcbcc0cfd70 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fcbc400cc20 tx=0x7fcbc40019b0 comp rx=0 tx=0).stop 2024-01-27T01:07:09.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.448+0000 7fcbd3afe700 1 -- 172.21.15.46:0/1454856635 shutdown_connections 2024-01-27T01:07:09.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.448+0000 7fcbd3afe700 1 --2- 172.21.15.46:0/1454856635 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcbb006f930 0x7fcbb0071d10 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:09.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.448+0000 7fcbd3afe700 1 --2- 172.21.15.46:0/1454856635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbcc15d8c0 0x7fcbcc0cfd70 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:09.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.448+0000 7fcbd3afe700 1 --2- 172.21.15.46:0/1454856635 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbcc1599d0 0x7fcbcc1574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:09.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.448+0000 7fcbd3afe700 1 --2- 172.21.15.46:0/1454856635 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbcc159da0 0x7fcbcc155110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:09.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.448+0000 7fcbd3afe700 1 -- 172.21.15.46:0/1454856635 >> 172.21.15.46:0/1454856635 conn(0x7fcbcc06f840 msgr2=0x7fcbcc15d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:09.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.448+0000 7fcbd3afe700 1 -- 172.21.15.46:0/1454856635 shutdown_connections 2024-01-27T01:07:09.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.448+0000 7fcbd3afe700 1 -- 172.21.15.46:0/1454856635 wait complete. 2024-01-27T01:07:09.463 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-27T01:07:09.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.647+0000 7f5208772700 1 Processor -- start 2024-01-27T01:07:09.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.647+0000 7f5208772700 1 -- start start 2024-01-27T01:07:09.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.648+0000 7f5208772700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52000b26e0 0x7f52000b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.648+0000 7f5208772700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f520015bd50 0x7f52001571f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.648+0000 7f5208772700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5200157730 0x7f5200067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.648+0000 7f5208772700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5200157b80 con 0x7f520015bd50 2024-01-27T01:07:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.648+0000 7f5208772700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52000cd4f0 con 0x7f5200157730 2024-01-27T01:07:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.648+0000 7f5208772700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52000cd7d0 con 0x7f52000b26e0 2024-01-27T01:07:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.648+0000 7f5205d0d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f520015bd50 0x7f52001571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.648+0000 7f5205d0d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f520015bd50 0x7f52001571f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59652/0 (socket says 172.21.15.46:59652) 2024-01-27T01:07:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.648+0000 7f5205d0d700 1 -- 172.21.15.46:0/2496883669 learned_addr learned my addr 172.21.15.46:0/2496883669 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.648+0000 7f520650e700 1 --2- 172.21.15.46:0/2496883669 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52000b26e0 0x7f52000b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.649+0000 7f5206d0f700 1 --2- 172.21.15.46:0/2496883669 >> v2:172.21.15.106:3300/0 conn(0x7f5200157730 0x7f5200067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.649+0000 7f5205d0d700 1 -- 172.21.15.46:0/2496883669 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52000b26e0 msgr2=0x7f52000b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.649+0000 7f5205d0d700 1 --2- 172.21.15.46:0/2496883669 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52000b26e0 0x7f52000b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.649+0000 7f5205d0d700 1 -- 172.21.15.46:0/2496883669 >> v2:172.21.15.106:3300/0 conn(0x7f5200157730 msgr2=0x7f5200067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.649+0000 7f5205d0d700 1 --2- 172.21.15.46:0/2496883669 >> v2:172.21.15.106:3300/0 conn(0x7f5200157730 0x7f5200067930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.649+0000 7f5205d0d700 1 -- 172.21.15.46:0/2496883669 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52000cdab0 con 0x7f520015bd50 2024-01-27T01:07:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.649+0000 7f520650e700 1 --2- 172.21.15.46:0/2496883669 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52000b26e0 0x7f52000b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.649+0000 7f5205d0d700 1 --2- 172.21.15.46:0/2496883669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f520015bd50 0x7f52001571f0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f52000aff80 tx=0x7f51fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc20f1107ce4928e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.649+0000 7f520550c700 1 -- 172.21.15.46:0/2496883669 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc00b860 con 0x7f520015bd50 2024-01-27T01:07:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.649+0000 7f520550c700 1 -- 172.21.15.46:0/2496883669 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51fc00b9c0 con 0x7f520015bd50 2024-01-27T01:07:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.649+0000 7f520550c700 1 -- 172.21.15.46:0/2496883669 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc00bd80 con 0x7f520015bd50 2024-01-27T01:07:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.650+0000 7f5208772700 1 -- 172.21.15.46:0/2496883669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f520015bd50 msgr2=0x7f52001571f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.650+0000 7f5208772700 1 --2- 172.21.15.46:0/2496883669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f520015bd50 0x7f52001571f0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f52000aff80 tx=0x7f51fc009250 comp rx=0 tx=0).stop 2024-01-27T01:07:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.650+0000 7f5208772700 1 -- 172.21.15.46:0/2496883669 shutdown_connections 2024-01-27T01:07:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.650+0000 7f5208772700 1 --2- 172.21.15.46:0/2496883669 >> v2:172.21.15.106:3300/0 conn(0x7f5200157730 0x7f5200067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.650+0000 7f5208772700 1 --2- 172.21.15.46:0/2496883669 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52000b26e0 0x7f52000b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.650+0000 7f5208772700 1 --2- 172.21.15.46:0/2496883669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f520015bd50 0x7f52001571f0 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:09.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.650+0000 7f5208772700 1 -- 172.21.15.46:0/2496883669 >> 172.21.15.46:0/2496883669 conn(0x7f5200069510 msgr2=0x7f52000aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:09.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.650+0000 7f5208772700 1 -- 172.21.15.46:0/2496883669 shutdown_connections 2024-01-27T01:07:09.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.650+0000 7f5208772700 1 -- 172.21.15.46:0/2496883669 wait complete. 2024-01-27T01:07:09.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.651+0000 7f5208772700 1 Processor -- start 2024-01-27T01:07:09.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.651+0000 7f5208772700 1 -- start start 2024-01-27T01:07:09.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.652+0000 7f5208772700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52000b26e0 0x7f52000ab3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:09.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.652+0000 7f5208772700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5200157730 0x7f520016dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:09.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.652+0000 7f5208772700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f520015bd50 0x7f5200159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:09.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.652+0000 7f5208772700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520016e130 con 0x7f5200157730 2024-01-27T01:07:09.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.652+0000 7f5208772700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52000ac350 con 0x7f520015bd50 2024-01-27T01:07:09.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.652+0000 7f5208772700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52000a9a00 con 0x7f52000b26e0 2024-01-27T01:07:09.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.652+0000 7f5205d0d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5200157730 0x7f520016dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:09.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.652+0000 7f5205d0d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5200157730 0x7f520016dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59660/0 (socket says 172.21.15.46:59660) 2024-01-27T01:07:09.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.652+0000 7f5205d0d700 1 -- 172.21.15.46:0/4147748130 learned_addr learned my addr 172.21.15.46:0/4147748130 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:09.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.652+0000 7f5206d0f700 1 --2- 172.21.15.46:0/4147748130 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f520015bd50 0x7f5200159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:09.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.652+0000 7f520650e700 1 --2- 172.21.15.46:0/4147748130 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52000b26e0 0x7f52000ab3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:09.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.652+0000 7f5205d0d700 1 -- 172.21.15.46:0/4147748130 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52000b26e0 msgr2=0x7f52000ab3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:09.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.652+0000 7f5205d0d700 1 --2- 172.21.15.46:0/4147748130 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52000b26e0 0x7f52000ab3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:09.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.652+0000 7f5205d0d700 1 -- 172.21.15.46:0/4147748130 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f520015bd50 msgr2=0x7f5200159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:09.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.652+0000 7f5205d0d700 1 --2- 172.21.15.46:0/4147748130 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f520015bd50 0x7f5200159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:09.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.652+0000 7f5205d0d700 1 -- 172.21.15.46:0/4147748130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51fc0094f0 con 0x7f5200157730 2024-01-27T01:07:09.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.653+0000 7f5205d0d700 1 --2- 172.21.15.46:0/4147748130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5200157730 0x7f520016dbf0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f51fc006fd0 tx=0x7f51fc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:09.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.653+0000 7f51f67fc700 1 -- 172.21.15.46:0/4147748130 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc01f070 con 0x7f5200157730 2024-01-27T01:07:09.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.653+0000 7f51f67fc700 1 -- 172.21.15.46:0/4147748130 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51fc006ab0 con 0x7f5200157730 2024-01-27T01:07:09.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.653+0000 7f51f67fc700 1 -- 172.21.15.46:0/4147748130 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc00bd80 con 0x7f5200157730 2024-01-27T01:07:09.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.653+0000 7f5208772700 1 -- 172.21.15.46:0/4147748130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5200159760 con 0x7f5200157730 2024-01-27T01:07:09.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.653+0000 7f5208772700 1 -- 172.21.15.46:0/4147748130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52000abd90 con 0x7f5200157730 2024-01-27T01:07:09.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.654+0000 7f5208772700 1 -- 172.21.15.46:0/4147748130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f520004c710 con 0x7f5200157730 2024-01-27T01:07:09.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.654+0000 7f51f67fc700 1 -- 172.21.15.46:0/4147748130 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 134) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f51fc023070 con 0x7f5200157730 2024-01-27T01:07:09.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.655+0000 7f51f67fc700 1 --2- 172.21.15.46:0/4147748130 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f51e4066d80 0x7f51e4069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:09.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.655+0000 7f51f67fc700 1 -- 172.21.15.46:0/4147748130 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 8112+0+0 (secure 0 0 0) 0x7f51fc085ec0 con 0x7f5200157730 2024-01-27T01:07:09.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.655+0000 7f520650e700 1 --2- 172.21.15.46:0/4147748130 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f51e4066d80 0x7f51e4069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:09.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.655+0000 7f520650e700 1 --2- 172.21.15.46:0/4147748130 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f51e4066d80 0x7f51e4069230 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f51f0006f00 tx=0x7f51f0008040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:09.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.659+0000 7f51f67fc700 1 -- 172.21.15.46:0/4147748130 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f51fc052b30 con 0x7f5200157730 2024-01-27T01:07:09.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:09.865+0000 7f5208772700 1 -- 172.21.15.46:0/4147748130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f52000af600 con 0x7f5200157730 2024-01-27T01:07:10.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:10.450+0000 7f51f67fc700 1 -- 172.21.15.46:0/4147748130 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 135) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f51fc052f00 con 0x7f5200157730 2024-01-27T01:07:10.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:10.872+0000 7f51f67fc700 1 -- 172.21.15.46:0/4147748130 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v231) v1 ==== 181+0+0 (secure 0 0 0) 0x7f51fc052b30 con 0x7f5200157730 2024-01-27T01:07:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:10.875+0000 7f5208772700 1 -- 172.21.15.46:0/4147748130 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f51e4066d80 msgr2=0x7f51e4069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:10.875+0000 7f5208772700 1 --2- 172.21.15.46:0/4147748130 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f51e4066d80 0x7f51e4069230 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f51f0006f00 tx=0x7f51f0008040 comp rx=0 tx=0).stop 2024-01-27T01:07:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:10.875+0000 7f5208772700 1 -- 172.21.15.46:0/4147748130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5200157730 msgr2=0x7f520016dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:10.875+0000 7f5208772700 1 --2- 172.21.15.46:0/4147748130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5200157730 0x7f520016dbf0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f51fc006fd0 tx=0x7f51fc017000 comp rx=0 tx=0).stop 2024-01-27T01:07:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:10.876+0000 7f5208772700 1 -- 172.21.15.46:0/4147748130 shutdown_connections 2024-01-27T01:07:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:10.876+0000 7f5208772700 1 --2- 172.21.15.46:0/4147748130 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f51e4066d80 0x7f51e4069230 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:10.876+0000 7f5208772700 1 --2- 172.21.15.46:0/4147748130 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f520015bd50 0x7f5200159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:10.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:10.876+0000 7f5208772700 1 --2- 172.21.15.46:0/4147748130 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52000b26e0 0x7f52000ab3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:10.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:10.876+0000 7f5208772700 1 --2- 172.21.15.46:0/4147748130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5200157730 0x7f520016dbf0 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:10.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:10.876+0000 7f5208772700 1 -- 172.21.15.46:0/4147748130 >> 172.21.15.46:0/4147748130 conn(0x7f5200069510 msgr2=0x7f520006eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:10.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:10.876+0000 7f5208772700 1 -- 172.21.15.46:0/4147748130 shutdown_connections 2024-01-27T01:07:10.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:10.876+0000 7f5208772700 1 -- 172.21.15.46:0/4147748130 wait complete. 2024-01-27T01:07:10.877 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-01-27T01:07:10.891 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-27T01:07:11.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.033+0000 7fde0f3f7700 1 Processor -- start 2024-01-27T01:07:11.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.033+0000 7fde0f3f7700 1 -- start start 2024-01-27T01:07:11.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.034+0000 7fde0f3f7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde080ca0e0 0x7fde080ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:11.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.034+0000 7fde0f3f7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde0806c0b0 0x7fde080afe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:11.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.034+0000 7fde0f3f7700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde08158010 con 0x7fde0806c520 2024-01-27T01:07:11.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.034+0000 7fde0f3f7700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde080cada0 con 0x7fde0806c0b0 2024-01-27T01:07:11.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.034+0000 7fde0f3f7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde081582f0 con 0x7fde080ca0e0 2024-01-27T01:07:11.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.034+0000 7fde0c992700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde0806c0b0 0x7fde080afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:11.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.034+0000 7fde0d994700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fde0806c520 0x7fde080b3a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52952/0 (socket says 172.21.15.46:52952) 2024-01-27T01:07:11.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.034+0000 7fde0d994700 1 -- 172.21.15.46:0/3971734810 learned_addr learned my addr 172.21.15.46:0/3971734810 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:11.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.034+0000 7fde0d193700 1 --2- 172.21.15.46:0/3971734810 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde080ca0e0 0x7fde080ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:11.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.035+0000 7fde0c992700 1 -- 172.21.15.46:0/3971734810 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde080ca0e0 msgr2=0x7fde080ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:11.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.035+0000 7fde0c992700 1 --2- 172.21.15.46:0/3971734810 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde080ca0e0 0x7fde080ca4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:11.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.035+0000 7fde0c992700 1 -- 172.21.15.46:0/3971734810 >> v1:172.21.15.46:6789/0 conn(0x7fde0806c520 legacy=0x7fde080b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:11.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.035+0000 7fde0c992700 1 -- 172.21.15.46:0/3971734810 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde081585d0 con 0x7fde0806c0b0 2024-01-27T01:07:11.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.035+0000 7fde0c992700 1 --2- 172.21.15.46:0/3971734810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde0806c0b0 0x7fde080afe40 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fddf80098e0 tx=0x7fddf80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da05778051bebcc5 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:11.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.035+0000 7fddfffff700 1 -- 172.21.15.46:0/3971734810 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8009e80 con 0x7fde0806c0b0 2024-01-27T01:07:11.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.035+0000 7fddfffff700 1 -- 172.21.15.46:0/3971734810 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf800b860 con 0x7fde0806c0b0 2024-01-27T01:07:11.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.035+0000 7fddfffff700 1 -- 172.21.15.46:0/3971734810 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf800bbe0 con 0x7fde0806c0b0 2024-01-27T01:07:11.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.036+0000 7fde0f3f7700 1 -- 172.21.15.46:0/3971734810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde0806c0b0 msgr2=0x7fde080afe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:11.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.036+0000 7fde0f3f7700 1 --2- 172.21.15.46:0/3971734810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde0806c0b0 0x7fde080afe40 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fddf80098e0 tx=0x7fddf80091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:11.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.036+0000 7fde0f3f7700 1 -- 172.21.15.46:0/3971734810 shutdown_connections 2024-01-27T01:07:11.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.036+0000 7fde0f3f7700 1 --2- 172.21.15.46:0/3971734810 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde080ca0e0 0x7fde080ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:11.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.036+0000 7fde0f3f7700 1 --2- 172.21.15.46:0/3971734810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde0806c0b0 0x7fde080afe40 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:11.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.036+0000 7fde0f3f7700 1 -- 172.21.15.46:0/3971734810 >> 172.21.15.46:0/3971734810 conn(0x7fde080ce530 msgr2=0x7fde080ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:11.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.036+0000 7fde0f3f7700 1 -- 172.21.15.46:0/3971734810 shutdown_connections 2024-01-27T01:07:11.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.036+0000 7fde0f3f7700 1 -- 172.21.15.46:0/3971734810 wait complete. 2024-01-27T01:07:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.037+0000 7fde0f3f7700 1 Processor -- start 2024-01-27T01:07:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.037+0000 7fde0f3f7700 1 -- start start 2024-01-27T01:07:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.038+0000 7fde0f3f7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde0806c0b0 0x7fde080c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.038+0000 7fde0f3f7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde0806c520 0x7fde0806d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.038+0000 7fde0f3f7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080ca0e0 0x7fde0806de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.038+0000 7fde0f3f7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde08167770 con 0x7fde080ca0e0 2024-01-27T01:07:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.038+0000 7fde0f3f7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde080c68a0 con 0x7fde0806c520 2024-01-27T01:07:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.038+0000 7fde0f3f7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde080c6b80 con 0x7fde0806c0b0 2024-01-27T01:07:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.038+0000 7fde0d994700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080ca0e0 0x7fde0806de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:11.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.038+0000 7fde0d994700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080ca0e0 0x7fde0806de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50528/0 (socket says 172.21.15.46:50528) 2024-01-27T01:07:11.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.038+0000 7fde0d994700 1 -- 172.21.15.46:0/2566906348 learned_addr learned my addr 172.21.15.46:0/2566906348 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:11.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.038+0000 7fde0d193700 1 --2- 172.21.15.46:0/2566906348 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde0806c0b0 0x7fde080c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:11.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.038+0000 7fde0c992700 1 --2- 172.21.15.46:0/2566906348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde0806c520 0x7fde0806d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:11.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.038+0000 7fde0d994700 1 -- 172.21.15.46:0/2566906348 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde0806c0b0 msgr2=0x7fde080c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:11.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.038+0000 7fde0d994700 1 --2- 172.21.15.46:0/2566906348 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde0806c0b0 0x7fde080c5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:11.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.039+0000 7fde0d994700 1 -- 172.21.15.46:0/2566906348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde0806c520 msgr2=0x7fde0806d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:11.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.039+0000 7fde0d994700 1 --2- 172.21.15.46:0/2566906348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde0806c520 0x7fde0806d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:11.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.039+0000 7fde0d994700 1 -- 172.21.15.46:0/2566906348 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf8009420 con 0x7fde080ca0e0 2024-01-27T01:07:11.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.039+0000 7fde0c992700 1 --2- 172.21.15.46:0/2566906348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde0806c520 0x7fde0806d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:11.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.039+0000 7fde0d994700 1 --2- 172.21.15.46:0/2566906348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080ca0e0 0x7fde0806de90 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fde0000cc70 tx=0x7fde00001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:11.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.039+0000 7fddfd7fa700 1 -- 172.21.15.46:0/2566906348 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0000a8e0 con 0x7fde080ca0e0 2024-01-27T01:07:11.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.039+0000 7fddfd7fa700 1 -- 172.21.15.46:0/2566906348 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0000aa40 con 0x7fde080ca0e0 2024-01-27T01:07:11.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.039+0000 7fde0f3f7700 1 -- 172.21.15.46:0/2566906348 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde080c6e60 con 0x7fde080ca0e0 2024-01-27T01:07:11.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.039+0000 7fddfd7fa700 1 -- 172.21.15.46:0/2566906348 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0000ae00 con 0x7fde080ca0e0 2024-01-27T01:07:11.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.039+0000 7fde0f3f7700 1 -- 172.21.15.46:0/2566906348 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde080c6440 con 0x7fde080ca0e0 2024-01-27T01:07:11.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.040+0000 7fde0f3f7700 1 -- 172.21.15.46:0/2566906348 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde080c7300 con 0x7fde080ca0e0 2024-01-27T01:07:11.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.040+0000 7fddfd7fa700 1 -- 172.21.15.46:0/2566906348 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 135) v1 ==== 84890+0+0 (secure 0 0 0) 0x7fde00007300 con 0x7fde080ca0e0 2024-01-27T01:07:11.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.041+0000 7fddfd7fa700 1 --2- 172.21.15.46:0/2566906348 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fddec0670e0 0x7fddec069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:11.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.041+0000 7fddfd7fa700 1 -- 172.21.15.46:0/2566906348 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7674+0+0 (secure 0 0 0) 0x7fde000852f0 con 0x7fde080ca0e0 2024-01-27T01:07:11.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.041+0000 7fde0d193700 1 --2- 172.21.15.46:0/2566906348 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fddec0670e0 0x7fddec069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:11.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.042+0000 7fde0d193700 1 --2- 172.21.15.46:0/2566906348 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fddec0670e0 0x7fddec069590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fde04006fd0 tx=0x7fde04008040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:11.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.045+0000 7fddfd7fa700 1 -- 172.21.15.46:0/2566906348 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fde00051bb0 con 0x7fde080ca0e0 2024-01-27T01:07:11.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.248+0000 7fde0f3f7700 1 -- 172.21.15.46:0/2566906348 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fde0804c710 con 0x7fde080ca0e0 2024-01-27T01:07:11.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.883+0000 7fddfd7fa700 1 -- 172.21.15.46:0/2566906348 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v232) v1 ==== 169+0+0 (secure 0 0 0) 0x7fde000551d0 con 0x7fde080ca0e0 2024-01-27T01:07:11.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.892+0000 7fde0f3f7700 1 -- 172.21.15.46:0/2566906348 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fddec0670e0 msgr2=0x7fddec069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:11.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.892+0000 7fde0f3f7700 1 --2- 172.21.15.46:0/2566906348 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fddec0670e0 0x7fddec069590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fde04006fd0 tx=0x7fde04008040 comp rx=0 tx=0).stop 2024-01-27T01:07:11.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.892+0000 7fde0f3f7700 1 -- 172.21.15.46:0/2566906348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080ca0e0 msgr2=0x7fde0806de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:11.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.892+0000 7fde0f3f7700 1 --2- 172.21.15.46:0/2566906348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080ca0e0 0x7fde0806de90 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fde0000cc70 tx=0x7fde00001a20 comp rx=0 tx=0).stop 2024-01-27T01:07:11.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.893+0000 7fde0f3f7700 1 -- 172.21.15.46:0/2566906348 shutdown_connections 2024-01-27T01:07:11.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.893+0000 7fde0f3f7700 1 --2- 172.21.15.46:0/2566906348 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fddec0670e0 0x7fddec069590 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:11.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.893+0000 7fde0f3f7700 1 --2- 172.21.15.46:0/2566906348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080ca0e0 0x7fde0806de90 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:11.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.893+0000 7fde0f3f7700 1 --2- 172.21.15.46:0/2566906348 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde0806c0b0 0x7fde080c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:11.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.893+0000 7fde0f3f7700 1 --2- 172.21.15.46:0/2566906348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde0806c520 0x7fde0806d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:11.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.893+0000 7fde0f3f7700 1 -- 172.21.15.46:0/2566906348 >> 172.21.15.46:0/2566906348 conn(0x7fde080ce530 msgr2=0x7fde080ad550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:11.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.894+0000 7fde0f3f7700 1 -- 172.21.15.46:0/2566906348 shutdown_connections 2024-01-27T01:07:11.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:11.895+0000 7fde0f3f7700 1 -- 172.21.15.46:0/2566906348 wait complete. 2024-01-27T01:07:11.898 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-01-27T01:07:11.914 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:07:12.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.049+0000 7fe700fb1700 1 Processor -- start 2024-01-27T01:07:12.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.050+0000 7fe700fb1700 1 -- start start 2024-01-27T01:07:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.051+0000 7fe700fb1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6fc0b2160 0x7fe6fc15a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.051+0000 7fe700fb1700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe6fc15da10 0x7fe6fc15de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.051+0000 7fe700fb1700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6fc0a9b90 con 0x7fe6fc0b2530 2024-01-27T01:07:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.051+0000 7fe700fb1700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6fc0aa210 con 0x7fe6fc15da10 2024-01-27T01:07:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.051+0000 7fe700fb1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc15d580 con 0x7fe6fc0b2160 2024-01-27T01:07:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.051+0000 7fe6f9d9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe6fc0b2530 0x7fe6fc0abe50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52958/0 (socket says 172.21.15.46:52958) 2024-01-27T01:07:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.051+0000 7fe6f9d9b700 1 -- 172.21.15.46:0/2307073842 learned_addr learned my addr 172.21.15.46:0/2307073842 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.051+0000 7fe6fa59c700 1 --2- 172.21.15.46:0/2307073842 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6fc0b2160 0x7fe6fc15a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.051+0000 7fe6fad9d700 1 --2- 172.21.15.46:0/2307073842 >> v2:172.21.15.106:3300/0 conn(0x7fe6fc15da10 0x7fe6fc15de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.051+0000 7fe6f959a700 1 -- 172.21.15.46:0/2307073842 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437121487 0 0) 0x7fe6fc0a9b90 con 0x7fe6fc0b2530 2024-01-27T01:07:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.051+0000 7fe6f959a700 1 -- 172.21.15.46:0/2307073842 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6e4003570 con 0x7fe6fc0b2530 2024-01-27T01:07:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.051+0000 7fe6fa59c700 1 -- 172.21.15.46:0/2307073842 >> v2:172.21.15.106:3300/0 conn(0x7fe6fc15da10 msgr2=0x7fe6fc15de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.051+0000 7fe6fa59c700 1 --2- 172.21.15.46:0/2307073842 >> v2:172.21.15.106:3300/0 conn(0x7fe6fc15da10 0x7fe6fc15de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.051+0000 7fe6fa59c700 1 -- 172.21.15.46:0/2307073842 >> v1:172.21.15.46:6789/0 conn(0x7fe6fc0b2530 legacy=0x7fe6fc0abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.051+0000 7fe6fa59c700 1 -- 172.21.15.46:0/2307073842 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc0aa460 con 0x7fe6fc0b2160 2024-01-27T01:07:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.051+0000 7fe6fad9d700 1 --2- 172.21.15.46:0/2307073842 >> v2:172.21.15.106:3300/0 conn(0x7fe6fc15da10 0x7fe6fc15de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.052+0000 7fe6fa59c700 1 --2- 172.21.15.46:0/2307073842 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6fc0b2160 0x7fe6fc15a7a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe6f00098e0 tx=0x7fe6f00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf63342cb6633f94 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.052+0000 7fe6f959a700 1 -- 172.21.15.46:0/2307073842 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0011630 con 0x7fe6fc0b2160 2024-01-27T01:07:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.052+0000 7fe6f959a700 1 -- 172.21.15.46:0/2307073842 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f0011790 con 0x7fe6fc0b2160 2024-01-27T01:07:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.052+0000 7fe6f959a700 1 -- 172.21.15.46:0/2307073842 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0011b30 con 0x7fe6fc0b2160 2024-01-27T01:07:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.052+0000 7fe700fb1700 1 -- 172.21.15.46:0/2307073842 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6fc0b2160 msgr2=0x7fe6fc15a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:12.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.052+0000 7fe700fb1700 1 --2- 172.21.15.46:0/2307073842 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6fc0b2160 0x7fe6fc15a7a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe6f00098e0 tx=0x7fe6f00091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.053+0000 7fe700fb1700 1 -- 172.21.15.46:0/2307073842 shutdown_connections 2024-01-27T01:07:12.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.053+0000 7fe700fb1700 1 --2- 172.21.15.46:0/2307073842 >> v2:172.21.15.106:3300/0 conn(0x7fe6fc15da10 0x7fe6fc15de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.053+0000 7fe700fb1700 1 --2- 172.21.15.46:0/2307073842 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6fc0b2160 0x7fe6fc15a7a0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.053+0000 7fe700fb1700 1 -- 172.21.15.46:0/2307073842 >> 172.21.15.46:0/2307073842 conn(0x7fe6fc0ccbc0 msgr2=0x7fe6fc0ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:12.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.053+0000 7fe700fb1700 1 -- 172.21.15.46:0/2307073842 shutdown_connections 2024-01-27T01:07:12.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.053+0000 7fe700fb1700 1 -- 172.21.15.46:0/2307073842 wait complete. 2024-01-27T01:07:12.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.054+0000 7fe700fb1700 1 Processor -- start 2024-01-27T01:07:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.054+0000 7fe700fb1700 1 -- start start 2024-01-27T01:07:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe700fb1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6fc0b2160 0x7fe6fc142640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe700fb1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6fc0b2530 0x7fe6fc0b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe700fb1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6fc15da10 0x7fe6fc0b11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe700fb1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc16a690 con 0x7fe6fc0b2530 2024-01-27T01:07:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe700fb1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc16aff0 con 0x7fe6fc0b2160 2024-01-27T01:07:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe700fb1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc142b80 con 0x7fe6fc15da10 2024-01-27T01:07:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe6f9d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6fc0b2530 0x7fe6fc0b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe6f9d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6fc0b2530 0x7fe6fc0b0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50534/0 (socket says 172.21.15.46:50534) 2024-01-27T01:07:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe6f9d9b700 1 -- 172.21.15.46:0/2434534941 learned_addr learned my addr 172.21.15.46:0/2434534941 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe6fad9d700 1 --2- 172.21.15.46:0/2434534941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6fc15da10 0x7fe6fc0b11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe6fa59c700 1 --2- 172.21.15.46:0/2434534941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6fc0b2160 0x7fe6fc142640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe6f9d9b700 1 -- 172.21.15.46:0/2434534941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6fc15da10 msgr2=0x7fe6fc0b11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:12.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe6f9d9b700 1 --2- 172.21.15.46:0/2434534941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6fc15da10 0x7fe6fc0b11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe6f9d9b700 1 -- 172.21.15.46:0/2434534941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6fc0b2160 msgr2=0x7fe6fc142640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:12.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe6f9d9b700 1 --2- 172.21.15.46:0/2434534941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6fc0b2160 0x7fe6fc142640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe6f9d9b700 1 -- 172.21.15.46:0/2434534941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f4009a20 con 0x7fe6fc0b2530 2024-01-27T01:07:12.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe6fa59c700 1 --2- 172.21.15.46:0/2434534941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6fc0b2160 0x7fe6fc142640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:07:12.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe6f9d9b700 1 --2- 172.21.15.46:0/2434534941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6fc0b2530 0x7fe6fc0b0cb0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec00c7c0 tx=0x7fe6ec007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:12.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe6ea7fc700 1 -- 172.21.15.46:0/2434534941 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec005b30 con 0x7fe6fc0b2530 2024-01-27T01:07:12.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe6ea7fc700 1 -- 172.21.15.46:0/2434534941 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ec005c90 con 0x7fe6fc0b2530 2024-01-27T01:07:12.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe700fb1700 1 -- 172.21.15.46:0/2434534941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f0009420 con 0x7fe6fc0b2530 2024-01-27T01:07:12.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.055+0000 7fe6ea7fc700 1 -- 172.21.15.46:0/2434534941 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec007340 con 0x7fe6fc0b2530 2024-01-27T01:07:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.056+0000 7fe700fb1700 1 -- 172.21.15.46:0/2434534941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6fc16a9a0 con 0x7fe6fc0b2530 2024-01-27T01:07:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.056+0000 7fe700fb1700 1 -- 172.21.15.46:0/2434534941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6fc16b180 con 0x7fe6fc0b2530 2024-01-27T01:07:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.057+0000 7fe6ea7fc700 1 -- 172.21.15.46:0/2434534941 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fe6ec01b400 con 0x7fe6fc0b2530 2024-01-27T01:07:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.057+0000 7fe6ea7fc700 1 --2- 172.21.15.46:0/2434534941 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe6dc067170 0x7fe6dc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.057+0000 7fe6ea7fc700 1 -- 172.21.15.46:0/2434534941 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7260+0+0 (secure 0 0 0) 0x7fe6ec084b00 con 0x7fe6fc0b2530 2024-01-27T01:07:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.058+0000 7fe6fa59c700 1 --2- 172.21.15.46:0/2434534941 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe6dc067170 0x7fe6dc069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.058+0000 7fe6fa59c700 1 --2- 172.21.15.46:0/2434534941 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe6dc067170 0x7fe6dc069620 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0003080 tx=0x7fe6f0017000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:12.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.062+0000 7fe6ea7fc700 1 -- 172.21.15.46:0/2434534941 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe6ec086e60 con 0x7fe6fc0b2530 2024-01-27T01:07:12.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.266+0000 7fe700fb1700 1 -- 172.21.15.46:0/2434534941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6fc04c710 con 0x7fe6fc0b2530 2024-01-27T01:07:12.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.267+0000 7fe6ea7fc700 1 -- 172.21.15.46:0/2434534941 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+17401 (secure 0 0 0) 0x7fe6ec0513e0 con 0x7fe6fc0b2530 2024-01-27T01:07:12.267 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:12.267 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":232,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:07:11.870581+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:06:38.156934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:06:39.330686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6834/2483163449":"2024-01-28T01:07:08.439983+0000","172.21.15.106:0/2812454604":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/3610030821":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/3431540914":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/42206641":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/3926539126":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/694514533":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/819986778":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/3809430766":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/981929816":"2024-01-28T01:07:07.849128+0000","172.21.15.106:6837/62434":"2024-01-28T01:06:57.756975+0000","172.21.15.106:6836/62434":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/524220301":"2024-01-28T01:06:57.756975+0000","172.21.15.46:6835/2483163449":"2024-01-28T01:07:08.439983+0000","172.21.15.106:0/4002309435":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/3400107086":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/2725250616":"2024-01-28T01:06:57.756975+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:07:12.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.269+0000 7fe700fb1700 1 -- 172.21.15.46:0/2434534941 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe6dc067170 msgr2=0x7fe6dc069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:12.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.269+0000 7fe700fb1700 1 --2- 172.21.15.46:0/2434534941 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe6dc067170 0x7fe6dc069620 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0003080 tx=0x7fe6f0017000 comp rx=0 tx=0).stop 2024-01-27T01:07:12.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.269+0000 7fe700fb1700 1 -- 172.21.15.46:0/2434534941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6fc0b2530 msgr2=0x7fe6fc0b0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:12.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.269+0000 7fe700fb1700 1 --2- 172.21.15.46:0/2434534941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6fc0b2530 0x7fe6fc0b0cb0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec00c7c0 tx=0x7fe6ec007e00 comp rx=0 tx=0).stop 2024-01-27T01:07:12.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.269+0000 7fe700fb1700 1 -- 172.21.15.46:0/2434534941 shutdown_connections 2024-01-27T01:07:12.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.269+0000 7fe700fb1700 1 --2- 172.21.15.46:0/2434534941 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe6dc067170 0x7fe6dc069620 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.269+0000 7fe700fb1700 1 --2- 172.21.15.46:0/2434534941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6fc15da10 0x7fe6fc0b11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.269+0000 7fe700fb1700 1 --2- 172.21.15.46:0/2434534941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6fc0b2160 0x7fe6fc142640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.269+0000 7fe700fb1700 1 --2- 172.21.15.46:0/2434534941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6fc0b2530 0x7fe6fc0b0cb0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.269+0000 7fe700fb1700 1 -- 172.21.15.46:0/2434534941 >> 172.21.15.46:0/2434534941 conn(0x7fe6fc0ccbc0 msgr2=0x7fe6fc15b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:12.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.270+0000 7fe700fb1700 1 -- 172.21.15.46:0/2434534941 shutdown_connections 2024-01-27T01:07:12.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.270+0000 7fe700fb1700 1 -- 172.21.15.46:0/2434534941 wait complete. 2024-01-27T01:07:12.284 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-01-27T01:06:38.156934+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2024-01-27T01:06:39.330686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '227', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}}] 2024-01-27T01:07:12.284 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-27T01:07:12.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.419+0000 7fd7fc51a700 1 Processor -- start 2024-01-27T01:07:12.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.419+0000 7fd7fc51a700 1 -- start start 2024-01-27T01:07:12.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.420+0000 7fd7fc51a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7f4157860 0x7fd7f40cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:12.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.420+0000 7fd7fc51a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd7f406ddb0 0x7fd7f40cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:12.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.420+0000 7fd7fc51a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f40cea80 con 0x7fd7f4157490 2024-01-27T01:07:12.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.420+0000 7fd7fc51a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7f40cdc60 con 0x7fd7f406ddb0 2024-01-27T01:07:12.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.420+0000 7fd7fc51a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f40b2ae0 con 0x7fd7f4157860 2024-01-27T01:07:12.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.421+0000 7fd7f9ab5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7f4157860 0x7fd7f40cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:12.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.421+0000 7fd7f9ab5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7f4157860 0x7fd7f40cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:59556/0 (socket says 172.21.15.46:59556) 2024-01-27T01:07:12.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.421+0000 7fd7f9ab5700 1 -- 172.21.15.46:0/1546026026 learned_addr learned my addr 172.21.15.46:0/1546026026 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:12.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.421+0000 7fd7faab7700 1 --2- 172.21.15.46:0/1546026026 >> v2:172.21.15.106:3300/0 conn(0x7fd7f406ddb0 0x7fd7f40cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:12.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.421+0000 7fd7f9ab5700 1 -- 172.21.15.46:0/1546026026 >> v2:172.21.15.106:3300/0 conn(0x7fd7f406ddb0 msgr2=0x7fd7f40cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:12.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.421+0000 7fd7f9ab5700 1 --2- 172.21.15.46:0/1546026026 >> v2:172.21.15.106:3300/0 conn(0x7fd7f406ddb0 0x7fd7f40cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.421+0000 7fd7f9ab5700 1 -- 172.21.15.46:0/1546026026 >> v1:172.21.15.106:6789/0 conn(0x7fd7f4157490 legacy=0x7fd7f406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:12.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.421+0000 7fd7f9ab5700 1 -- 172.21.15.46:0/1546026026 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f40cde70 con 0x7fd7f4157860 2024-01-27T01:07:12.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.421+0000 7fd7f9ab5700 1 --2- 172.21.15.46:0/1546026026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7f4157860 0x7fd7f40cc600 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00099b0 tx=0x7fd7f0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbf0d2c48f7b4edd server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:12.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.421+0000 7fd7f92b4700 1 -- 172.21.15.46:0/1546026026 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f0009ea0 con 0x7fd7f4157860 2024-01-27T01:07:12.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.422+0000 7fd7f92b4700 1 -- 172.21.15.46:0/1546026026 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f000b860 con 0x7fd7f4157860 2024-01-27T01:07:12.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.422+0000 7fd7f92b4700 1 -- 172.21.15.46:0/1546026026 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f000bc00 con 0x7fd7f4157860 2024-01-27T01:07:12.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.422+0000 7fd7fc51a700 1 -- 172.21.15.46:0/1546026026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7f4157860 msgr2=0x7fd7f40cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:12.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.422+0000 7fd7fc51a700 1 --2- 172.21.15.46:0/1546026026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7f4157860 0x7fd7f40cc600 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00099b0 tx=0x7fd7f0009250 comp rx=0 tx=0).stop 2024-01-27T01:07:12.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.422+0000 7fd7fc51a700 1 -- 172.21.15.46:0/1546026026 shutdown_connections 2024-01-27T01:07:12.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.422+0000 7fd7fc51a700 1 --2- 172.21.15.46:0/1546026026 >> v2:172.21.15.106:3300/0 conn(0x7fd7f406ddb0 0x7fd7f40cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.422+0000 7fd7fc51a700 1 --2- 172.21.15.46:0/1546026026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7f4157860 0x7fd7f40cc600 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.422+0000 7fd7fc51a700 1 -- 172.21.15.46:0/1546026026 >> 172.21.15.46:0/1546026026 conn(0x7fd7f4067930 msgr2=0x7fd7f4067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:12.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.422+0000 7fd7fc51a700 1 -- 172.21.15.46:0/1546026026 shutdown_connections 2024-01-27T01:07:12.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.422+0000 7fd7fc51a700 1 -- 172.21.15.46:0/1546026026 wait complete. 2024-01-27T01:07:12.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.424+0000 7fd7fc51a700 1 Processor -- start 2024-01-27T01:07:12.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.424+0000 7fd7fc51a700 1 -- start start 2024-01-27T01:07:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.425+0000 7fd7fc51a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7f406ddb0 0x7fd7f40c3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.425+0000 7fd7fc51a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7f4157490 0x7fd7f40c3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.425+0000 7fd7fc51a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7f4157860 0x7fd7f40af7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.425+0000 7fd7fc51a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f415b520 con 0x7fd7f406ddb0 2024-01-27T01:07:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.425+0000 7fd7fc51a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f415ac10 con 0x7fd7f4157860 2024-01-27T01:07:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.425+0000 7fd7fc51a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f40b0bb0 con 0x7fd7f4157490 2024-01-27T01:07:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.425+0000 7fd7fa2b6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7f406ddb0 0x7fd7f40c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.425+0000 7fd7fa2b6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7f406ddb0 0x7fd7f40c3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50548/0 (socket says 172.21.15.46:50548) 2024-01-27T01:07:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.425+0000 7fd7fa2b6700 1 -- 172.21.15.46:0/2595018313 learned_addr learned my addr 172.21.15.46:0/2595018313 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.425+0000 7fd7f9ab5700 1 --2- 172.21.15.46:0/2595018313 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7f4157490 0x7fd7f40c3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.425+0000 7fd7faab7700 1 --2- 172.21.15.46:0/2595018313 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7f4157860 0x7fd7f40af7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:12.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.425+0000 7fd7fa2b6700 1 -- 172.21.15.46:0/2595018313 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7f4157490 msgr2=0x7fd7f40c3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:12.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.425+0000 7fd7fa2b6700 1 --2- 172.21.15.46:0/2595018313 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7f4157490 0x7fd7f40c3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.425+0000 7fd7fa2b6700 1 -- 172.21.15.46:0/2595018313 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7f4157860 msgr2=0x7fd7f40af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:12.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.425+0000 7fd7fa2b6700 1 --2- 172.21.15.46:0/2595018313 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7f4157860 0x7fd7f40af7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.426+0000 7fd7fa2b6700 1 -- 172.21.15.46:0/2595018313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f00094f0 con 0x7fd7f406ddb0 2024-01-27T01:07:12.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.426+0000 7fd7fa2b6700 1 --2- 172.21.15.46:0/2595018313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7f406ddb0 0x7fd7f40c3350 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7fd7e800c590 tx=0x7fd7e800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:12.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.426+0000 7fd7e67fc700 1 -- 172.21.15.46:0/2595018313 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8011590 con 0x7fd7f406ddb0 2024-01-27T01:07:12.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.426+0000 7fd7e67fc700 1 -- 172.21.15.46:0/2595018313 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e8011710 con 0x7fd7f406ddb0 2024-01-27T01:07:12.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.426+0000 7fd7e67fc700 1 -- 172.21.15.46:0/2595018313 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8011ad0 con 0x7fd7f406ddb0 2024-01-27T01:07:12.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.426+0000 7fd7fc51a700 1 -- 172.21.15.46:0/2595018313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7f415aed0 con 0x7fd7f406ddb0 2024-01-27T01:07:12.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.426+0000 7fd7fc51a700 1 -- 172.21.15.46:0/2595018313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7f40afd10 con 0x7fd7f406ddb0 2024-01-27T01:07:12.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.427+0000 7fd7fc51a700 1 -- 172.21.15.46:0/2595018313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7f404c710 con 0x7fd7f406ddb0 2024-01-27T01:07:12.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.427+0000 7fd7e67fc700 1 -- 172.21.15.46:0/2595018313 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fd7e801b3b0 con 0x7fd7f406ddb0 2024-01-27T01:07:12.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.428+0000 7fd7e67fc700 1 --2- 172.21.15.46:0/2595018313 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd7d8067170 0x7fd7d8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:12.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.428+0000 7fd7e67fc700 1 -- 172.21.15.46:0/2595018313 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7260+0+0 (secure 0 0 0) 0x7fd7e8084310 con 0x7fd7f406ddb0 2024-01-27T01:07:12.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.428+0000 7fd7f9ab5700 1 --2- 172.21.15.46:0/2595018313 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd7d8067170 0x7fd7d8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.429+0000 7fd7f9ab5700 1 --2- 172.21.15.46:0/2595018313 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd7d8067170 0x7fd7d8069620 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00099b0 tx=0x7fd7f0017000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:12.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.432+0000 7fd7e67fc700 1 -- 172.21.15.46:0/2595018313 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd7e8050ce0 con 0x7fd7f406ddb0 2024-01-27T01:07:12.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.638+0000 7fd7fc51a700 1 -- 172.21.15.46:0/2595018313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd7f40b14d0 con 0x7fd7f406ddb0 2024-01-27T01:07:12.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.638+0000 7fd7e67fc700 1 -- 172.21.15.46:0/2595018313 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v232) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd7e80170a0 con 0x7fd7f406ddb0 2024-01-27T01:07:12.639 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-27T01:07:12.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.641+0000 7fd7fc51a700 1 -- 172.21.15.46:0/2595018313 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd7d8067170 msgr2=0x7fd7d8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:12.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.641+0000 7fd7fc51a700 1 --2- 172.21.15.46:0/2595018313 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd7d8067170 0x7fd7d8069620 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00099b0 tx=0x7fd7f0017000 comp rx=0 tx=0).stop 2024-01-27T01:07:12.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.641+0000 7fd7fc51a700 1 -- 172.21.15.46:0/2595018313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7f406ddb0 msgr2=0x7fd7f40c3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:12.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.641+0000 7fd7fc51a700 1 --2- 172.21.15.46:0/2595018313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7f406ddb0 0x7fd7f40c3350 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7fd7e800c590 tx=0x7fd7e800c8a0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.641+0000 7fd7fc51a700 1 -- 172.21.15.46:0/2595018313 shutdown_connections 2024-01-27T01:07:12.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.641+0000 7fd7fc51a700 1 --2- 172.21.15.46:0/2595018313 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd7d8067170 0x7fd7d8069620 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.641+0000 7fd7fc51a700 1 --2- 172.21.15.46:0/2595018313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7f406ddb0 0x7fd7f40c3350 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.641+0000 7fd7fc51a700 1 --2- 172.21.15.46:0/2595018313 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7f4157860 0x7fd7f40af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.641+0000 7fd7fc51a700 1 --2- 172.21.15.46:0/2595018313 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7f4157490 0x7fd7f40c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.641+0000 7fd7fc51a700 1 -- 172.21.15.46:0/2595018313 >> 172.21.15.46:0/2595018313 conn(0x7fd7f4067930 msgr2=0x7fd7f4153040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:12.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.641+0000 7fd7fc51a700 1 -- 172.21.15.46:0/2595018313 shutdown_connections 2024-01-27T01:07:12.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.641+0000 7fd7fc51a700 1 -- 172.21.15.46:0/2595018313 wait complete. 2024-01-27T01:07:12.654 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-01-27T01:07:12.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.788+0000 7f6d5122b700 1 Processor -- start 2024-01-27T01:07:12.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.788+0000 7f6d5122b700 1 -- start start 2024-01-27T01:07:12.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.789+0000 7f6d5122b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6d4c06ebe0 0x7f6d4c067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:12.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.789+0000 7f6d5122b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4c06fdb0 con 0x7f6d4c067af0 2024-01-27T01:07:12.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.789+0000 7f6d5122b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4c06f1a0 con 0x7f6d4c1597e0 2024-01-27T01:07:12.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.789+0000 7f6d5122b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d4c06f480 con 0x7f6d4c06ebe0 2024-01-27T01:07:12.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.789+0000 7f6d4a59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6d4c06ebe0 0x7f6d4c067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:12.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.789+0000 7f6d4ad9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6d4c067af0 0x7f6d4c0ad8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52964/0 (socket says 172.21.15.46:52964) 2024-01-27T01:07:12.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.789+0000 7f6d4ad9d700 1 -- 172.21.15.46:0/2168125623 learned_addr learned my addr 172.21.15.46:0/2168125623 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:12.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.790+0000 7f6d49d9b700 1 -- 172.21.15.46:0/2168125623 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846866883 0 0) 0x7f6d4c06fdb0 con 0x7f6d4c067af0 2024-01-27T01:07:12.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.790+0000 7f6d49d9b700 1 -- 172.21.15.46:0/2168125623 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d38003570 con 0x7f6d4c067af0 2024-01-27T01:07:12.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.790+0000 7f6d4a59c700 1 -- 172.21.15.46:0/2168125623 >> v1:172.21.15.106:6789/0 conn(0x7f6d4c1597e0 legacy=0x7f6d4c159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:12.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.790+0000 7f6d4a59c700 1 -- 172.21.15.46:0/2168125623 >> v1:172.21.15.46:6789/0 conn(0x7f6d4c067af0 legacy=0x7f6d4c0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:12.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.790+0000 7f6d4a59c700 1 -- 172.21.15.46:0/2168125623 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c06f760 con 0x7f6d4c06ebe0 2024-01-27T01:07:12.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.790+0000 7f6d4a59c700 1 --2- 172.21.15.46:0/2168125623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c06ebe0 0x7f6d4c067ec0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c0098e0 tx=0x7f6d3c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b50468679724d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:12.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.790+0000 7f6d49d9b700 1 -- 172.21.15.46:0/2168125623 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c00b640 con 0x7f6d4c06ebe0 2024-01-27T01:07:12.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.790+0000 7f6d49d9b700 1 -- 172.21.15.46:0/2168125623 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d3c00b7a0 con 0x7f6d4c06ebe0 2024-01-27T01:07:12.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.790+0000 7f6d49d9b700 1 -- 172.21.15.46:0/2168125623 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c00bb40 con 0x7f6d4c06ebe0 2024-01-27T01:07:12.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.791+0000 7f6d5122b700 1 -- 172.21.15.46:0/2168125623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c06ebe0 msgr2=0x7f6d4c067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:12.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.791+0000 7f6d5122b700 1 --2- 172.21.15.46:0/2168125623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c06ebe0 0x7f6d4c067ec0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c0098e0 tx=0x7f6d3c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.791+0000 7f6d5122b700 1 -- 172.21.15.46:0/2168125623 shutdown_connections 2024-01-27T01:07:12.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.791+0000 7f6d5122b700 1 --2- 172.21.15.46:0/2168125623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c06ebe0 0x7f6d4c067ec0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.791+0000 7f6d5122b700 1 -- 172.21.15.46:0/2168125623 >> 172.21.15.46:0/2168125623 conn(0x7f6d4c0cdf80 msgr2=0x7f6d4c06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:12.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.791+0000 7f6d5122b700 1 -- 172.21.15.46:0/2168125623 shutdown_connections 2024-01-27T01:07:12.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.791+0000 7f6d5122b700 1 -- 172.21.15.46:0/2168125623 wait complete. 2024-01-27T01:07:12.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.792+0000 7f6d5122b700 1 Processor -- start 2024-01-27T01:07:12.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.792+0000 7f6d5122b700 1 -- start start 2024-01-27T01:07:12.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.793+0000 7f6d5122b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d4c067af0 0x7f6d4c157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:12.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.793+0000 7f6d5122b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d4c06ebe0 0x7f6d4c156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:12.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.793+0000 7f6d5122b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c1597e0 0x7f6d4c0b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:12.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.793+0000 7f6d5122b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c158360 con 0x7f6d4c1597e0 2024-01-27T01:07:12.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.793+0000 7f6d5122b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c158640 con 0x7f6d4c067af0 2024-01-27T01:07:12.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.793+0000 7f6d5122b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c158920 con 0x7f6d4c06ebe0 2024-01-27T01:07:12.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.793+0000 7f6d4b59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c1597e0 0x7f6d4c0b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:12.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.793+0000 7f6d4b59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c1597e0 0x7f6d4c0b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50568/0 (socket says 172.21.15.46:50568) 2024-01-27T01:07:12.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.793+0000 7f6d4b59e700 1 -- 172.21.15.46:0/3038714070 learned_addr learned my addr 172.21.15.46:0/3038714070 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:12.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.793+0000 7f6d4a59c700 1 --2- 172.21.15.46:0/3038714070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d4c06ebe0 0x7f6d4c156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:12.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.793+0000 7f6d4ad9d700 1 --2- 172.21.15.46:0/3038714070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d4c067af0 0x7f6d4c157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:12.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.793+0000 7f6d4b59e700 1 -- 172.21.15.46:0/3038714070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d4c06ebe0 msgr2=0x7f6d4c156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:12.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.794+0000 7f6d4b59e700 1 --2- 172.21.15.46:0/3038714070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d4c06ebe0 0x7f6d4c156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.794+0000 7f6d4b59e700 1 -- 172.21.15.46:0/3038714070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d4c067af0 msgr2=0x7f6d4c157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:12.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.794+0000 7f6d4b59e700 1 --2- 172.21.15.46:0/3038714070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d4c067af0 0x7f6d4c157c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:12.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.794+0000 7f6d4b59e700 1 -- 172.21.15.46:0/3038714070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3c009420 con 0x7f6d4c1597e0 2024-01-27T01:07:12.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.794+0000 7f6d4a59c700 1 --2- 172.21.15.46:0/3038714070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d4c06ebe0 0x7f6d4c156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:12.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.794+0000 7f6d4ad9d700 1 --2- 172.21.15.46:0/3038714070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d4c067af0 0x7f6d4c157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:12.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.794+0000 7f6d4b59e700 1 --2- 172.21.15.46:0/3038714070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c1597e0 0x7f6d4c0b1a00 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400c8e0 tx=0x7f6d44001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:12.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.794+0000 7f6d42ffd700 1 -- 172.21.15.46:0/3038714070 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4400ce80 con 0x7f6d4c1597e0 2024-01-27T01:07:12.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.794+0000 7f6d42ffd700 1 -- 172.21.15.46:0/3038714070 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4400a7e0 con 0x7f6d4c1597e0 2024-01-27T01:07:12.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.794+0000 7f6d5122b700 1 -- 172.21.15.46:0/3038714070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d4c158c00 con 0x7f6d4c1597e0 2024-01-27T01:07:12.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.794+0000 7f6d42ffd700 1 -- 172.21.15.46:0/3038714070 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4400ab80 con 0x7f6d4c1597e0 2024-01-27T01:07:12.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.794+0000 7f6d5122b700 1 -- 172.21.15.46:0/3038714070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d4c14d430 con 0x7f6d4c1597e0 2024-01-27T01:07:12.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.795+0000 7f6d5122b700 1 -- 172.21.15.46:0/3038714070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d4c159120 con 0x7f6d4c1597e0 2024-01-27T01:07:12.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.796+0000 7f6d42ffd700 1 -- 172.21.15.46:0/3038714070 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f6d44012420 con 0x7f6d4c1597e0 2024-01-27T01:07:12.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.796+0000 7f6d42ffd700 1 --2- 172.21.15.46:0/3038714070 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6d2c067170 0x7f6d2c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:12.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.796+0000 7f6d42ffd700 1 -- 172.21.15.46:0/3038714070 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7260+0+0 (secure 0 0 0) 0x7f6d44085190 con 0x7f6d4c1597e0 2024-01-27T01:07:12.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.796+0000 7f6d4ad9d700 1 --2- 172.21.15.46:0/3038714070 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6d2c067170 0x7f6d2c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:12.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.797+0000 7f6d4ad9d700 1 --2- 172.21.15.46:0/3038714070 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6d2c067170 0x7f6d2c069620 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6d340058d0 tx=0x7f6d34009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:12.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:12.801+0000 7f6d42ffd700 1 -- 172.21.15.46:0/3038714070 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6d4c159120 con 0x7f6d4c1597e0 2024-01-27T01:07:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.010+0000 7f6d5122b700 1 -- 172.21.15.46:0/3038714070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6d4c159120 con 0x7f6d4c1597e0 2024-01-27T01:07:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.010+0000 7f6d42ffd700 1 -- 172.21.15.46:0/3038714070 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v232) v1 ==== 107+0+11 (secure 0 0 0) 0x7f6d4c159120 con 0x7f6d4c1597e0 2024-01-27T01:07:13.011 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-27T01:07:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.012+0000 7f6d5122b700 1 -- 172.21.15.46:0/3038714070 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6d2c067170 msgr2=0x7f6d2c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.012+0000 7f6d5122b700 1 --2- 172.21.15.46:0/3038714070 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6d2c067170 0x7f6d2c069620 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6d340058d0 tx=0x7f6d34009040 comp rx=0 tx=0).stop 2024-01-27T01:07:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.012+0000 7f6d5122b700 1 -- 172.21.15.46:0/3038714070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c1597e0 msgr2=0x7f6d4c0b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.012+0000 7f6d5122b700 1 --2- 172.21.15.46:0/3038714070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c1597e0 0x7f6d4c0b1a00 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400c8e0 tx=0x7f6d44001710 comp rx=0 tx=0).stop 2024-01-27T01:07:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.012+0000 7f6d5122b700 1 -- 172.21.15.46:0/3038714070 shutdown_connections 2024-01-27T01:07:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.012+0000 7f6d5122b700 1 --2- 172.21.15.46:0/3038714070 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6d2c067170 0x7f6d2c069620 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.012+0000 7f6d5122b700 1 --2- 172.21.15.46:0/3038714070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d4c1597e0 0x7f6d4c0b1a00 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.012+0000 7f6d5122b700 1 --2- 172.21.15.46:0/3038714070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d4c067af0 0x7f6d4c157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.012+0000 7f6d5122b700 1 --2- 172.21.15.46:0/3038714070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d4c06ebe0 0x7f6d4c156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.012+0000 7f6d5122b700 1 -- 172.21.15.46:0/3038714070 >> 172.21.15.46:0/3038714070 conn(0x7f6d4c0cdf80 msgr2=0x7f6d4c0ca3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:13.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.013+0000 7f6d5122b700 1 -- 172.21.15.46:0/3038714070 shutdown_connections 2024-01-27T01:07:13.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.013+0000 7f6d5122b700 1 -- 172.21.15.46:0/3038714070 wait complete. 2024-01-27T01:07:13.026 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-01-27T01:07:13.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.162+0000 7f7be16a1700 1 Processor -- start 2024-01-27T01:07:13.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.162+0000 7f7be16a1700 1 -- start start 2024-01-27T01:07:13.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.162+0000 7f7be16a1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7bdc06a6c0 0x7f7bdc06aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.162+0000 7f7be16a1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bdc14fd20 0x7f7bdc150190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.162+0000 7f7be16a1700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7bdc063580 0x7f7bdc0b0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.162+0000 7f7be16a1700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bdc15a4a0 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.162+0000 7f7be16a1700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bdc1506d0 con 0x7f7bdc063580 2024-01-27T01:07:13.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.162+0000 7f7be16a1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc159890 con 0x7f7bdc14fd20 2024-01-27T01:07:13.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.163+0000 7f7bdaffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7bdc06a6c0 0x7f7bdc06aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.163+0000 7f7bdaffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7bdc06a6c0 0x7f7bdc06aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50570/0 (socket says 172.21.15.46:50570) 2024-01-27T01:07:13.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.163+0000 7f7bdaffd700 1 -- 172.21.15.46:0/1353202596 learned_addr learned my addr 172.21.15.46:0/1353202596 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:13.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.163+0000 7f7bda7fc700 1 --2- 172.21.15.46:0/1353202596 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bdc14fd20 0x7f7bdc150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.163+0000 7f7bdb7fe700 1 --2- 172.21.15.46:0/1353202596 >> v2:172.21.15.106:3300/0 conn(0x7f7bdc063580 0x7f7bdc0b0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.163+0000 7f7bdaffd700 1 -- 172.21.15.46:0/1353202596 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bdc14fd20 msgr2=0x7f7bdc150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:13.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.163+0000 7f7bdaffd700 1 --2- 172.21.15.46:0/1353202596 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bdc14fd20 0x7f7bdc150190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.163+0000 7f7bdaffd700 1 -- 172.21.15.46:0/1353202596 >> v2:172.21.15.106:3300/0 conn(0x7f7bdc063580 msgr2=0x7f7bdc0b0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.163+0000 7f7bdaffd700 1 --2- 172.21.15.46:0/1353202596 >> v2:172.21.15.106:3300/0 conn(0x7f7bdc063580 0x7f7bdc0b0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.163+0000 7f7bdaffd700 1 -- 172.21.15.46:0/1353202596 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bdc159b70 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.163+0000 7f7bdaffd700 1 --2- 172.21.15.46:0/1353202596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bdc06a6c0 0x7f7bdc06aa90 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc153dc0 tx=0x7f7bc4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd380f7a6244903 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.163+0000 7f7bd9ffb700 1 -- 172.21.15.46:0/1353202596 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc400b640 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.164+0000 7f7bd9ffb700 1 -- 172.21.15.46:0/1353202596 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc400b7a0 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.164+0000 7f7bd9ffb700 1 -- 172.21.15.46:0/1353202596 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc400bb60 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.164+0000 7f7be16a1700 1 -- 172.21.15.46:0/1353202596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bdc06a6c0 msgr2=0x7f7bdc06aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.164+0000 7f7be16a1700 1 --2- 172.21.15.46:0/1353202596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bdc06a6c0 0x7f7bdc06aa90 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc153dc0 tx=0x7f7bc4009250 comp rx=0 tx=0).stop 2024-01-27T01:07:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.164+0000 7f7be16a1700 1 -- 172.21.15.46:0/1353202596 shutdown_connections 2024-01-27T01:07:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.164+0000 7f7be16a1700 1 --2- 172.21.15.46:0/1353202596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bdc06a6c0 0x7f7bdc06aa90 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.164+0000 7f7be16a1700 1 --2- 172.21.15.46:0/1353202596 >> v2:172.21.15.106:3300/0 conn(0x7f7bdc063580 0x7f7bdc0b0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.164+0000 7f7be16a1700 1 --2- 172.21.15.46:0/1353202596 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bdc14fd20 0x7f7bdc150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.164+0000 7f7be16a1700 1 -- 172.21.15.46:0/1353202596 >> 172.21.15.46:0/1353202596 conn(0x7f7bdc0ccc20 msgr2=0x7f7bdc0cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.164+0000 7f7be16a1700 1 -- 172.21.15.46:0/1353202596 shutdown_connections 2024-01-27T01:07:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.164+0000 7f7be16a1700 1 -- 172.21.15.46:0/1353202596 wait complete. 2024-01-27T01:07:13.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.166+0000 7f7be16a1700 1 Processor -- start 2024-01-27T01:07:13.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.166+0000 7f7be16a1700 1 -- start start 2024-01-27T01:07:13.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.167+0000 7f7be16a1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bdc063580 0x7f7bdc156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.167+0000 7f7be16a1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bdc06a6c0 0x7f7bdc0ab600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.167+0000 7f7be16a1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bdc14fd20 0x7f7bdc067f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.167+0000 7f7be16a1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc0abb40 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.167+0000 7f7be16a1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc157050 con 0x7f7bdc14fd20 2024-01-27T01:07:13.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.167+0000 7f7be16a1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc157330 con 0x7f7bdc063580 2024-01-27T01:07:13.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.167+0000 7f7bda7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bdc06a6c0 0x7f7bdc0ab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.167+0000 7f7bda7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bdc06a6c0 0x7f7bdc0ab600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50578/0 (socket says 172.21.15.46:50578) 2024-01-27T01:07:13.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.167+0000 7f7bda7fc700 1 -- 172.21.15.46:0/74141314 learned_addr learned my addr 172.21.15.46:0/74141314 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:13.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.167+0000 7f7bdaffd700 1 --2- 172.21.15.46:0/74141314 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bdc063580 0x7f7bdc156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.167+0000 7f7bdb7fe700 1 --2- 172.21.15.46:0/74141314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bdc14fd20 0x7f7bdc067f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.168+0000 7f7bda7fc700 1 -- 172.21.15.46:0/74141314 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bdc063580 msgr2=0x7f7bdc156a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:13.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.168+0000 7f7bda7fc700 1 --2- 172.21.15.46:0/74141314 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bdc063580 0x7f7bdc156a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.168+0000 7f7bda7fc700 1 -- 172.21.15.46:0/74141314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bdc14fd20 msgr2=0x7f7bdc067f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:13.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.168+0000 7f7bda7fc700 1 --2- 172.21.15.46:0/74141314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bdc14fd20 0x7f7bdc067f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.168+0000 7f7bda7fc700 1 -- 172.21.15.46:0/74141314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc40094f0 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.168+0000 7f7bda7fc700 1 --2- 172.21.15.46:0/74141314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bdc06a6c0 0x7f7bdc0ab600 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc00ec80 tx=0x7f7bcc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:13.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.168+0000 7f7bd2ffd700 1 -- 172.21.15.46:0/74141314 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc00c850 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.168+0000 7f7bd2ffd700 1 -- 172.21.15.46:0/74141314 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bcc00c9b0 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.168+0000 7f7bd2ffd700 1 -- 172.21.15.46:0/74141314 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc00cd50 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.168+0000 7f7be16a1700 1 -- 172.21.15.46:0/74141314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bdc157610 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.168+0000 7f7be16a1700 1 -- 172.21.15.46:0/74141314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bdc157bd0 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.169+0000 7f7be16a1700 1 -- 172.21.15.46:0/74141314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bdc1581b0 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.170+0000 7f7bd2ffd700 1 -- 172.21.15.46:0/74141314 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f7bcc013680 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.170+0000 7f7bd2ffd700 1 --2- 172.21.15.46:0/74141314 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bbc067170 0x7f7bbc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.170+0000 7f7bd2ffd700 1 -- 172.21.15.46:0/74141314 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7260+0+0 (secure 0 0 0) 0x7f7bcc085100 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.170+0000 7f7bdaffd700 1 --2- 172.21.15.46:0/74141314 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bbc067170 0x7f7bbc069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.171+0000 7f7bdaffd700 1 --2- 172.21.15.46:0/74141314 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bbc067170 0x7f7bbc069620 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc153dc0 tx=0x7f7bc4017000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:13.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.175+0000 7f7bd2ffd700 1 -- 172.21.15.46:0/74141314 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7bcc051ad0 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.377+0000 7f7be16a1700 1 -- 172.21.15.46:0/74141314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f7bdc04c710 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.377+0000 7f7bd2ffd700 1 -- 172.21.15.46:0/74141314 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v232) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7bcc0550f0 con 0x7f7bdc06a6c0 2024-01-27T01:07:13.377 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-27T01:07:13.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.379+0000 7f7be16a1700 1 -- 172.21.15.46:0/74141314 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bbc067170 msgr2=0x7f7bbc069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.379+0000 7f7be16a1700 1 --2- 172.21.15.46:0/74141314 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bbc067170 0x7f7bbc069620 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc153dc0 tx=0x7f7bc4017000 comp rx=0 tx=0).stop 2024-01-27T01:07:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.379+0000 7f7be16a1700 1 -- 172.21.15.46:0/74141314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bdc06a6c0 msgr2=0x7f7bdc0ab600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.379+0000 7f7be16a1700 1 --2- 172.21.15.46:0/74141314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bdc06a6c0 0x7f7bdc0ab600 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc00ec80 tx=0x7f7bcc0091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.379+0000 7f7be16a1700 1 -- 172.21.15.46:0/74141314 shutdown_connections 2024-01-27T01:07:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.379+0000 7f7be16a1700 1 --2- 172.21.15.46:0/74141314 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7bbc067170 0x7f7bbc069620 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.379+0000 7f7be16a1700 1 --2- 172.21.15.46:0/74141314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bdc14fd20 0x7f7bdc067f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.379+0000 7f7be16a1700 1 --2- 172.21.15.46:0/74141314 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7bdc063580 0x7f7bdc156a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.379+0000 7f7be16a1700 1 --2- 172.21.15.46:0/74141314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bdc06a6c0 0x7f7bdc0ab600 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.379+0000 7f7be16a1700 1 -- 172.21.15.46:0/74141314 >> 172.21.15.46:0/74141314 conn(0x7f7bdc0ccc20 msgr2=0x7f7bdc0ad6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.379+0000 7f7be16a1700 1 -- 172.21.15.46:0/74141314 shutdown_connections 2024-01-27T01:07:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.379+0000 7f7be16a1700 1 -- 172.21.15.46:0/74141314 wait complete. 2024-01-27T01:07:13.393 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:07:13.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.524+0000 7fccd2c5f700 1 Processor -- start 2024-01-27T01:07:13.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.524+0000 7fccd2c5f700 1 -- start start 2024-01-27T01:07:13.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.525+0000 7fccd2c5f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcccc15d210 0x7fcccc0cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.525+0000 7fccd2c5f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fcccc0cbff0 0x7fcccc0ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.525+0000 7fccd2c5f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcccc0cc450 con 0x7fcccc15ce40 2024-01-27T01:07:13.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.525+0000 7fccd2c5f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcccc0cff70 con 0x7fcccc15d210 2024-01-27T01:07:13.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.525+0000 7fccd2c5f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcccc0cf400 con 0x7fcccc0cbff0 2024-01-27T01:07:13.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.525+0000 7fcccbfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcccc15d210 0x7fcccc0cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.526+0000 7fcccbfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcccc15d210 0x7fcccc0cbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50594/0 (socket says 172.21.15.46:50594) 2024-01-27T01:07:13.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.526+0000 7fcccbfff700 1 -- 172.21.15.46:0/808732458 learned_addr learned my addr 172.21.15.46:0/808732458 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:13.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.526+0000 7fccd11fc700 1 --2- 172.21.15.46:0/808732458 >> v2:172.21.15.106:3300/0 conn(0x7fcccc0cbff0 0x7fcccc0ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.526+0000 7fcccbfff700 1 -- 172.21.15.46:0/808732458 >> v2:172.21.15.106:3300/0 conn(0x7fcccc0cbff0 msgr2=0x7fcccc0ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:13.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.526+0000 7fcccbfff700 1 --2- 172.21.15.46:0/808732458 >> v2:172.21.15.106:3300/0 conn(0x7fcccc0cbff0 0x7fcccc0ade30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.526+0000 7fcccbfff700 1 -- 172.21.15.46:0/808732458 >> v1:172.21.15.106:6789/0 conn(0x7fcccc15ce40 legacy=0x7fcccc0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:13.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.526+0000 7fcccbfff700 1 -- 172.21.15.46:0/808732458 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc0cf660 con 0x7fcccc15d210 2024-01-27T01:07:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.526+0000 7fccd11fc700 1 --2- 172.21.15.46:0/808732458 >> v2:172.21.15.106:3300/0 conn(0x7fcccc0cbff0 0x7fcccc0ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.526+0000 7fcccbfff700 1 --2- 172.21.15.46:0/808732458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc15d210 0x7fcccc0cbab0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fcccc0d01c0 tx=0x7fccb4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705ad11c7343c33b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.526+0000 7fcccb7fe700 1 -- 172.21.15.46:0/808732458 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb400b860 con 0x7fcccc15d210 2024-01-27T01:07:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.526+0000 7fcccb7fe700 1 -- 172.21.15.46:0/808732458 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb400b9c0 con 0x7fcccc15d210 2024-01-27T01:07:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.526+0000 7fcccb7fe700 1 -- 172.21.15.46:0/808732458 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb400bd60 con 0x7fcccc15d210 2024-01-27T01:07:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.527+0000 7fccd2c5f700 1 -- 172.21.15.46:0/808732458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc15d210 msgr2=0x7fcccc0cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.527+0000 7fccd2c5f700 1 --2- 172.21.15.46:0/808732458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc15d210 0x7fcccc0cbab0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fcccc0d01c0 tx=0x7fccb4009250 comp rx=0 tx=0).stop 2024-01-27T01:07:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.527+0000 7fccd2c5f700 1 -- 172.21.15.46:0/808732458 shutdown_connections 2024-01-27T01:07:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.527+0000 7fccd2c5f700 1 --2- 172.21.15.46:0/808732458 >> v2:172.21.15.106:3300/0 conn(0x7fcccc0cbff0 0x7fcccc0ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.527+0000 7fccd2c5f700 1 --2- 172.21.15.46:0/808732458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc15d210 0x7fcccc0cbab0 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.527+0000 7fccd2c5f700 1 -- 172.21.15.46:0/808732458 >> 172.21.15.46:0/808732458 conn(0x7fcccc06f840 msgr2=0x7fcccc0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.527+0000 7fccd2c5f700 1 -- 172.21.15.46:0/808732458 shutdown_connections 2024-01-27T01:07:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.527+0000 7fccd2c5f700 1 -- 172.21.15.46:0/808732458 wait complete. 2024-01-27T01:07:13.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.528+0000 7fccd2c5f700 1 Processor -- start 2024-01-27T01:07:13.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.528+0000 7fccd2c5f700 1 -- start start 2024-01-27T01:07:13.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.529+0000 7fccd2c5f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcccc0cbff0 0x7fcccc169b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.529+0000 7fccd2c5f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc15ce40 0x7fcccc156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.529+0000 7fccd2c5f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcccc15d210 0x7fcccc158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.529+0000 7fccd2c5f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc1538f0 con 0x7fcccc15ce40 2024-01-27T01:07:13.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.529+0000 7fccd2c5f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc153bd0 con 0x7fcccc0cbff0 2024-01-27T01:07:13.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.529+0000 7fccd2c5f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc153eb0 con 0x7fcccc15d210 2024-01-27T01:07:13.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.529+0000 7fcccbfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc15ce40 0x7fcccc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.529+0000 7fcccbfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc15ce40 0x7fcccc156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50608/0 (socket says 172.21.15.46:50608) 2024-01-27T01:07:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.529+0000 7fcccbfff700 1 -- 172.21.15.46:0/1591426811 learned_addr learned my addr 172.21.15.46:0/1591426811 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.529+0000 7fccd09fb700 1 --2- 172.21.15.46:0/1591426811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcccc0cbff0 0x7fcccc169b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.529+0000 7fccd11fc700 1 --2- 172.21.15.46:0/1591426811 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcccc15d210 0x7fcccc158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.530+0000 7fcccbfff700 1 -- 172.21.15.46:0/1591426811 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcccc15d210 msgr2=0x7fcccc158720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.530+0000 7fcccbfff700 1 --2- 172.21.15.46:0/1591426811 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcccc15d210 0x7fcccc158720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.530+0000 7fcccbfff700 1 -- 172.21.15.46:0/1591426811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcccc0cbff0 msgr2=0x7fcccc169b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.530+0000 7fcccbfff700 1 --2- 172.21.15.46:0/1591426811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcccc0cbff0 0x7fcccc169b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.530+0000 7fcccbfff700 1 -- 172.21.15.46:0/1591426811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb40094f0 con 0x7fcccc15ce40 2024-01-27T01:07:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.530+0000 7fcccbfff700 1 --2- 172.21.15.46:0/1591426811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc15ce40 0x7fcccc156430 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fccb40090b0 tx=0x7fccb4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.530+0000 7fccc8ff9700 1 -- 172.21.15.46:0/1591426811 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4011e60 con 0x7fcccc15ce40 2024-01-27T01:07:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.530+0000 7fccc8ff9700 1 -- 172.21.15.46:0/1591426811 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb401a630 con 0x7fcccc15ce40 2024-01-27T01:07:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.530+0000 7fccd2c5f700 1 -- 172.21.15.46:0/1591426811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc154190 con 0x7fcccc15ce40 2024-01-27T01:07:13.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.530+0000 7fccc8ff9700 1 -- 172.21.15.46:0/1591426811 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb401a9d0 con 0x7fcccc15ce40 2024-01-27T01:07:13.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.530+0000 7fccd2c5f700 1 -- 172.21.15.46:0/1591426811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc1514c0 con 0x7fcccc15ce40 2024-01-27T01:07:13.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.531+0000 7fccd2c5f700 1 -- 172.21.15.46:0/1591426811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcccc1546b0 con 0x7fcccc15ce40 2024-01-27T01:07:13.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.532+0000 7fccc8ff9700 1 -- 172.21.15.46:0/1591426811 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fccb402d440 con 0x7fcccc15ce40 2024-01-27T01:07:13.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.532+0000 7fccc8ff9700 1 --2- 172.21.15.46:0/1591426811 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fccac067170 0x7fccac069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.532+0000 7fccc8ff9700 1 -- 172.21.15.46:0/1591426811 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7260+0+0 (secure 0 0 0) 0x7fccb40869a0 con 0x7fcccc15ce40 2024-01-27T01:07:13.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.532+0000 7fccd09fb700 1 --2- 172.21.15.46:0/1591426811 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fccac067170 0x7fccac069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.533+0000 7fccd09fb700 1 --2- 172.21.15.46:0/1591426811 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fccac067170 0x7fccac069620 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fccbc003890 tx=0x7fccbc009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:13.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.536+0000 7fccc8ff9700 1 -- 172.21.15.46:0/1591426811 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcccc1546b0 con 0x7fcccc15ce40 2024-01-27T01:07:13.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.788+0000 7fccd2c5f700 1 -- 172.21.15.46:0/1591426811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcccc1546b0 con 0x7fcccc15ce40 2024-01-27T01:07:13.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.788+0000 7fccc8ff9700 1 -- 172.21.15.46:0/1591426811 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+4621 (secure 0 0 0) 0x7fcccc1546b0 con 0x7fcccc15ce40 2024-01-27T01:07:13.789 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:13.789 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":124,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/1999300009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1999300009},{"type":"v1","addr":"172.21.15.106:6835","nonce":1999300009}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":110},{"gid":10486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/4174263148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4174263148},{"type":"v1","addr":"172.21.15.106:6833","nonce":4174263148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":11454,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1260622833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":1260622833},{"type":"v1","addr":"172.21.15.46:6835","nonce":1260622833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:06:40.353248+0000","modified":"2024-01-27T01:06:43.938885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10854":{"gid":10854,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":8,"addr":"172.21.15.46:6833/1263484152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1263484152},{"type":"v1","addr":"172.21.15.46:6833","nonce":1263484152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-27T01:07:13.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.791+0000 7fccd2c5f700 1 -- 172.21.15.46:0/1591426811 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fccac067170 msgr2=0x7fccac069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:13.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.791+0000 7fccd2c5f700 1 --2- 172.21.15.46:0/1591426811 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fccac067170 0x7fccac069620 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fccbc003890 tx=0x7fccbc009040 comp rx=0 tx=0).stop 2024-01-27T01:07:13.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.791+0000 7fccd2c5f700 1 -- 172.21.15.46:0/1591426811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc15ce40 msgr2=0x7fcccc156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:13.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.791+0000 7fccd2c5f700 1 --2- 172.21.15.46:0/1591426811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc15ce40 0x7fcccc156430 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fccb40090b0 tx=0x7fccb4017000 comp rx=0 tx=0).stop 2024-01-27T01:07:13.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.791+0000 7fccd2c5f700 1 -- 172.21.15.46:0/1591426811 shutdown_connections 2024-01-27T01:07:13.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.791+0000 7fccd2c5f700 1 --2- 172.21.15.46:0/1591426811 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fccac067170 0x7fccac069620 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.791+0000 7fccd2c5f700 1 --2- 172.21.15.46:0/1591426811 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcccc15d210 0x7fcccc158720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.791+0000 7fccd2c5f700 1 --2- 172.21.15.46:0/1591426811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcccc0cbff0 0x7fcccc169b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.791+0000 7fccd2c5f700 1 --2- 172.21.15.46:0/1591426811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc15ce40 0x7fcccc156430 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.791+0000 7fccd2c5f700 1 -- 172.21.15.46:0/1591426811 >> 172.21.15.46:0/1591426811 conn(0x7fcccc06f840 msgr2=0x7fcccc0b3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:13.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.791+0000 7fccd2c5f700 1 -- 172.21.15.46:0/1591426811 shutdown_connections 2024-01-27T01:07:13.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.791+0000 7fccd2c5f700 1 -- 172.21.15.46:0/1591426811 wait complete. 2024-01-27T01:07:13.792 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 124 2024-01-27T01:07:13.804 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:07:13.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.938+0000 7f5b4bb10700 1 Processor -- start 2024-01-27T01:07:13.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.938+0000 7f5b4bb10700 1 -- start start 2024-01-27T01:07:13.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.939+0000 7f5b4bb10700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5b44157340 0x7f5b4415bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.939+0000 7f5b4bb10700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b440ae520 0x7f5b440b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.939+0000 7f5b4bb10700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b44158ae0 con 0x7f5b440b3fb0 2024-01-27T01:07:13.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.939+0000 7f5b4bb10700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b440b4430 con 0x7f5b44157340 2024-01-27T01:07:13.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.939+0000 7f5b4bb10700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4415b650 con 0x7f5b440ae520 2024-01-27T01:07:13.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.939+0000 7f5b490ab700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b440ae520 0x7f5b440b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.940+0000 7f5b498ac700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5b44157340 0x7f5b4415bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.940+0000 7f5b4a0ad700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f5b440b3fb0 0x7f5b44157710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:45876/0 (socket says 172.21.15.46:45876) 2024-01-27T01:07:13.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.940+0000 7f5b4a0ad700 1 -- 172.21.15.46:0/1734555819 learned_addr learned my addr 172.21.15.46:0/1734555819 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:13.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.940+0000 7f5b488aa700 1 -- 172.21.15.46:0/1734555819 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269610691 0 0) 0x7f5b44158ae0 con 0x7f5b440b3fb0 2024-01-27T01:07:13.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.940+0000 7f5b490ab700 1 -- 172.21.15.46:0/1734555819 >> v2:172.21.15.106:3300/0 conn(0x7f5b44157340 msgr2=0x7f5b4415bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:13.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.940+0000 7f5b490ab700 1 --2- 172.21.15.46:0/1734555819 >> v2:172.21.15.106:3300/0 conn(0x7f5b44157340 0x7f5b4415bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.940+0000 7f5b490ab700 1 -- 172.21.15.46:0/1734555819 >> v1:172.21.15.106:6789/0 conn(0x7f5b440b3fb0 legacy=0x7f5b44157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:13.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.940+0000 7f5b490ab700 1 -- 172.21.15.46:0/1734555819 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b4415c020 con 0x7f5b440ae520 2024-01-27T01:07:13.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.940+0000 7f5b490ab700 1 --2- 172.21.15.46:0/1734555819 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b440ae520 0x7f5b440b3a70 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5b340098e0 tx=0x7f5b340091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a427ce61d237150 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:13.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.940+0000 7f5b488aa700 1 -- 172.21.15.46:0/1734555819 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3400b640 con 0x7f5b440ae520 2024-01-27T01:07:13.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.940+0000 7f5b488aa700 1 -- 172.21.15.46:0/1734555819 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b3400b7a0 con 0x7f5b440ae520 2024-01-27T01:07:13.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.940+0000 7f5b488aa700 1 -- 172.21.15.46:0/1734555819 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3400bb40 con 0x7f5b440ae520 2024-01-27T01:07:13.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.941+0000 7f5b4bb10700 1 -- 172.21.15.46:0/1734555819 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b440ae520 msgr2=0x7f5b440b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.941+0000 7f5b4bb10700 1 --2- 172.21.15.46:0/1734555819 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b440ae520 0x7f5b440b3a70 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5b340098e0 tx=0x7f5b340091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.941+0000 7f5b4bb10700 1 -- 172.21.15.46:0/1734555819 shutdown_connections 2024-01-27T01:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.941+0000 7f5b4bb10700 1 --2- 172.21.15.46:0/1734555819 >> v2:172.21.15.106:3300/0 conn(0x7f5b44157340 0x7f5b4415bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.941+0000 7f5b4bb10700 1 --2- 172.21.15.46:0/1734555819 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b440ae520 0x7f5b440b3a70 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.941+0000 7f5b4bb10700 1 -- 172.21.15.46:0/1734555819 >> 172.21.15.46:0/1734555819 conn(0x7f5b440ce530 msgr2=0x7f5b440ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.941+0000 7f5b4bb10700 1 -- 172.21.15.46:0/1734555819 shutdown_connections 2024-01-27T01:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.941+0000 7f5b4bb10700 1 -- 172.21.15.46:0/1734555819 wait complete. 2024-01-27T01:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.942+0000 7f5b4bb10700 1 Processor -- start 2024-01-27T01:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.942+0000 7f5b4bb10700 1 -- start start 2024-01-27T01:07:13.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.943+0000 7f5b4bb10700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b440ae520 0x7f5b440acf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.943+0000 7f5b4bb10700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b440b3fb0 0x7f5b440b3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.943+0000 7f5b4bb10700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b44157340 0x7f5b4406a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.943+0000 7f5b4bb10700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b440ad470 con 0x7f5b44157340 2024-01-27T01:07:13.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.943+0000 7f5b4bb10700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b440b2ff0 con 0x7f5b440b3fb0 2024-01-27T01:07:13.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.943+0000 7f5b4bb10700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b440b23e0 con 0x7f5b440ae520 2024-01-27T01:07:13.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.943+0000 7f5b4a0ad700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b44157340 0x7f5b4406a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.943+0000 7f5b4a0ad700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b44157340 0x7f5b4406a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50624/0 (socket says 172.21.15.46:50624) 2024-01-27T01:07:13.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.943+0000 7f5b4a0ad700 1 -- 172.21.15.46:0/315003446 learned_addr learned my addr 172.21.15.46:0/315003446 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:13.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.943+0000 7f5b498ac700 1 --2- 172.21.15.46:0/315003446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b440ae520 0x7f5b440acf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.943+0000 7f5b490ab700 1 --2- 172.21.15.46:0/315003446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b440b3fb0 0x7f5b440b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.944+0000 7f5b4a0ad700 1 -- 172.21.15.46:0/315003446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b440ae520 msgr2=0x7f5b440acf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:13.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.944+0000 7f5b4a0ad700 1 --2- 172.21.15.46:0/315003446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b440ae520 0x7f5b440acf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.944+0000 7f5b4a0ad700 1 -- 172.21.15.46:0/315003446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b440b3fb0 msgr2=0x7f5b440b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:13.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.944+0000 7f5b4a0ad700 1 --2- 172.21.15.46:0/315003446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b440b3fb0 0x7f5b440b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:13.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.944+0000 7f5b4a0ad700 1 -- 172.21.15.46:0/315003446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b34009420 con 0x7f5b44157340 2024-01-27T01:07:13.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.944+0000 7f5b490ab700 1 --2- 172.21.15.46:0/315003446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b440b3fb0 0x7f5b440b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:13.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.944+0000 7f5b4a0ad700 1 --2- 172.21.15.46:0/315003446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b44157340 0x7f5b4406a2c0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00cc90 tx=0x7f5b3c001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:13.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.944+0000 7f5b39ffb700 1 -- 172.21.15.46:0/315003446 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c00a960 con 0x7f5b44157340 2024-01-27T01:07:13.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.944+0000 7f5b39ffb700 1 -- 172.21.15.46:0/315003446 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b3c00aac0 con 0x7f5b44157340 2024-01-27T01:07:13.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.944+0000 7f5b39ffb700 1 -- 172.21.15.46:0/315003446 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c00ae80 con 0x7f5b44157340 2024-01-27T01:07:13.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.944+0000 7f5b4bb10700 1 -- 172.21.15.46:0/315003446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b440b26c0 con 0x7f5b44157340 2024-01-27T01:07:13.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.944+0000 7f5b4bb10700 1 -- 172.21.15.46:0/315003446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b440b2c80 con 0x7f5b44157340 2024-01-27T01:07:13.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.945+0000 7f5b4bb10700 1 -- 172.21.15.46:0/315003446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b4404c710 con 0x7f5b44157340 2024-01-27T01:07:13.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.946+0000 7f5b39ffb700 1 -- 172.21.15.46:0/315003446 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f5b3c007300 con 0x7f5b44157340 2024-01-27T01:07:13.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.946+0000 7f5b39ffb700 1 --2- 172.21.15.46:0/315003446 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5b28067140 0x7f5b280695f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:13.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.946+0000 7f5b39ffb700 1 -- 172.21.15.46:0/315003446 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7260+0+0 (secure 0 0 0) 0x7f5b3c085280 con 0x7f5b44157340 2024-01-27T01:07:13.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.946+0000 7f5b498ac700 1 --2- 172.21.15.46:0/315003446 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5b28067140 0x7f5b280695f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:13.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.947+0000 7f5b498ac700 1 --2- 172.21.15.46:0/315003446 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5b28067140 0x7f5b280695f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5b40009d10 tx=0x7f5b400091a0 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:13.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:13.951+0000 7f5b39ffb700 1 -- 172.21.15.46:0/315003446 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5b3c051cd0 con 0x7f5b44157340 2024-01-27T01:07:14.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.160+0000 7f5b4bb10700 1 -- 172.21.15.46:0/315003446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b44158520 con 0x7f5b44157340 2024-01-27T01:07:14.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.161+0000 7f5b39ffb700 1 -- 172.21.15.46:0/315003446 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+17401 (secure 0 0 0) 0x7f5b3c0552f0 con 0x7f5b44157340 2024-01-27T01:07:14.161 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:14.161 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":232,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:07:11.870581+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:06:38.156934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:06:39.330686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6834/2483163449":"2024-01-28T01:07:08.439983+0000","172.21.15.106:0/2812454604":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/3610030821":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/3431540914":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/42206641":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/3926539126":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/694514533":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/819986778":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/3809430766":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/981929816":"2024-01-28T01:07:07.849128+0000","172.21.15.106:6837/62434":"2024-01-28T01:06:57.756975+0000","172.21.15.106:6836/62434":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/524220301":"2024-01-28T01:06:57.756975+0000","172.21.15.46:6835/2483163449":"2024-01-28T01:07:08.439983+0000","172.21.15.106:0/4002309435":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/3400107086":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/2725250616":"2024-01-28T01:06:57.756975+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:07:14.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.163+0000 7f5b4bb10700 1 -- 172.21.15.46:0/315003446 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5b28067140 msgr2=0x7f5b280695f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:14.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.163+0000 7f5b4bb10700 1 --2- 172.21.15.46:0/315003446 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5b28067140 0x7f5b280695f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5b40009d10 tx=0x7f5b400091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.163+0000 7f5b4bb10700 1 -- 172.21.15.46:0/315003446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b44157340 msgr2=0x7f5b4406a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:14.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.163+0000 7f5b4bb10700 1 --2- 172.21.15.46:0/315003446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b44157340 0x7f5b4406a2c0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00cc90 tx=0x7f5b3c001a80 comp rx=0 tx=0).stop 2024-01-27T01:07:14.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.163+0000 7f5b4bb10700 1 -- 172.21.15.46:0/315003446 shutdown_connections 2024-01-27T01:07:14.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.163+0000 7f5b4bb10700 1 --2- 172.21.15.46:0/315003446 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5b28067140 0x7f5b280695f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.163+0000 7f5b4bb10700 1 --2- 172.21.15.46:0/315003446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b44157340 0x7f5b4406a2c0 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.163+0000 7f5b4bb10700 1 --2- 172.21.15.46:0/315003446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b440ae520 0x7f5b440acf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.163+0000 7f5b4bb10700 1 --2- 172.21.15.46:0/315003446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b440b3fb0 0x7f5b440b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.163+0000 7f5b4bb10700 1 -- 172.21.15.46:0/315003446 >> 172.21.15.46:0/315003446 conn(0x7f5b440ce530 msgr2=0x7f5b44156900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:14.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.163+0000 7f5b4bb10700 1 -- 172.21.15.46:0/315003446 shutdown_connections 2024-01-27T01:07:14.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.163+0000 7f5b4bb10700 1 -- 172.21.15.46:0/315003446 wait complete. 2024-01-27T01:07:14.177 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-01-27T01:07:14.177 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:07:14.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.308+0000 7fe017f75700 1 Processor -- start 2024-01-27T01:07:14.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.309+0000 7fe017f75700 1 -- start start 2024-01-27T01:07:14.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.310+0000 7fe017f75700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe010159610 0x7fe0100b2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.310+0000 7fe017f75700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0100ce2b0 con 0x7fe010068040 2024-01-27T01:07:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.310+0000 7fe017f75700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0100afeb0 con 0x7fe0100b2f90 2024-01-27T01:07:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.310+0000 7fe017f75700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0100cd3c0 con 0x7fe010159610 2024-01-27T01:07:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.310+0000 7fe015d11700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe010159610 0x7fe0100b2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.310+0000 7fe016512700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe010068040 0x7fe01015a080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52976/0 (socket says 172.21.15.46:52976) 2024-01-27T01:07:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.310+0000 7fe016512700 1 -- 172.21.15.46:0/2144354979 learned_addr learned my addr 172.21.15.46:0/2144354979 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.310+0000 7fe015d11700 1 -- 172.21.15.46:0/2144354979 >> v1:172.21.15.106:6789/0 conn(0x7fe0100b2f90 legacy=0x7fe010067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.310+0000 7fe015d11700 1 -- 172.21.15.46:0/2144354979 >> v1:172.21.15.46:6789/0 conn(0x7fe010068040 legacy=0x7fe01015a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.310+0000 7fe015d11700 1 -- 172.21.15.46:0/2144354979 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0100cd6a0 con 0x7fe010159610 2024-01-27T01:07:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.310+0000 7fe014d0f700 1 -- 172.21.15.46:0/2144354979 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005014932 0 0) 0x7fe0100ce2b0 con 0x7fe010068040 2024-01-27T01:07:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.310+0000 7fe015d11700 1 --2- 172.21.15.46:0/2144354979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe010159610 0x7fe0100b2a50 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fe0100ce260 tx=0x7fe00c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e81a164a710bdbc server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.310+0000 7fe014d0f700 1 -- 172.21.15.46:0/2144354979 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c00b6b0 con 0x7fe010159610 2024-01-27T01:07:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.310+0000 7fe014d0f700 1 -- 172.21.15.46:0/2144354979 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00c00b810 con 0x7fe010159610 2024-01-27T01:07:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.311+0000 7fe014d0f700 1 -- 172.21.15.46:0/2144354979 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c00bbd0 con 0x7fe010159610 2024-01-27T01:07:14.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.311+0000 7fe017f75700 1 -- 172.21.15.46:0/2144354979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe010159610 msgr2=0x7fe0100b2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.311+0000 7fe017f75700 1 --2- 172.21.15.46:0/2144354979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe010159610 0x7fe0100b2a50 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fe0100ce260 tx=0x7fe00c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.311+0000 7fe017f75700 1 -- 172.21.15.46:0/2144354979 shutdown_connections 2024-01-27T01:07:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.311+0000 7fe017f75700 1 --2- 172.21.15.46:0/2144354979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe010159610 0x7fe0100b2a50 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.311+0000 7fe017f75700 1 -- 172.21.15.46:0/2144354979 >> 172.21.15.46:0/2144354979 conn(0x7fe0100694d0 msgr2=0x7fe0100aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.311+0000 7fe017f75700 1 -- 172.21.15.46:0/2144354979 shutdown_connections 2024-01-27T01:07:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.311+0000 7fe017f75700 1 -- 172.21.15.46:0/2144354979 wait complete. 2024-01-27T01:07:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.312+0000 7fe017f75700 1 Processor -- start 2024-01-27T01:07:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.312+0000 7fe017f75700 1 -- start start 2024-01-27T01:07:14.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.313+0000 7fe017f75700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe010068040 0x7fe01014d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:14.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.313+0000 7fe017f75700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0100b2f90 0x7fe0100af310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:14.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.313+0000 7fe017f75700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe010159610 0x7fe01014b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:14.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.313+0000 7fe017f75700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01014fe80 con 0x7fe0100b2f90 2024-01-27T01:07:14.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.313+0000 7fe017f75700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0100af850 con 0x7fe010068040 2024-01-27T01:07:14.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.313+0000 7fe017f75700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01014d850 con 0x7fe010159610 2024-01-27T01:07:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.313+0000 7fe015510700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0100b2f90 0x7fe0100af310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.313+0000 7fe015510700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0100b2f90 0x7fe0100af310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50644/0 (socket says 172.21.15.46:50644) 2024-01-27T01:07:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.313+0000 7fe015510700 1 -- 172.21.15.46:0/1113645199 learned_addr learned my addr 172.21.15.46:0/1113645199 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.313+0000 7fe015d11700 1 --2- 172.21.15.46:0/1113645199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe010068040 0x7fe01014d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.313+0000 7fe016512700 1 --2- 172.21.15.46:0/1113645199 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe010159610 0x7fe01014b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.314+0000 7fe015510700 1 -- 172.21.15.46:0/1113645199 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe010159610 msgr2=0x7fe01014b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.314+0000 7fe015510700 1 --2- 172.21.15.46:0/1113645199 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe010159610 0x7fe01014b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.314+0000 7fe015510700 1 -- 172.21.15.46:0/1113645199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe010068040 msgr2=0x7fe01014d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.314+0000 7fe015510700 1 --2- 172.21.15.46:0/1113645199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe010068040 0x7fe01014d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.314+0000 7fe015510700 1 -- 172.21.15.46:0/1113645199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c009420 con 0x7fe0100b2f90 2024-01-27T01:07:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.314+0000 7fe015510700 1 --2- 172.21.15.46:0/1113645199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0100b2f90 0x7fe0100af310 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fe00000c590 tx=0x7fe00000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.314+0000 7fe005ffb700 1 -- 172.21.15.46:0/1113645199 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe000011590 con 0x7fe0100b2f90 2024-01-27T01:07:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.314+0000 7fe005ffb700 1 -- 172.21.15.46:0/1113645199 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe000011710 con 0x7fe0100b2f90 2024-01-27T01:07:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.314+0000 7fe005ffb700 1 -- 172.21.15.46:0/1113645199 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe000011ad0 con 0x7fe0100b2f90 2024-01-27T01:07:14.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.314+0000 7fe017f75700 1 -- 172.21.15.46:0/1113645199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01014db30 con 0x7fe0100b2f90 2024-01-27T01:07:14.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.314+0000 7fe017f75700 1 -- 172.21.15.46:0/1113645199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01014e0f0 con 0x7fe0100b2f90 2024-01-27T01:07:14.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.315+0000 7fe017f75700 1 -- 172.21.15.46:0/1113645199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01004c710 con 0x7fe0100b2f90 2024-01-27T01:07:14.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.316+0000 7fe005ffb700 1 -- 172.21.15.46:0/1113645199 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fe00001b3b0 con 0x7fe0100b2f90 2024-01-27T01:07:14.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.316+0000 7fe005ffb700 1 --2- 172.21.15.46:0/1113645199 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdff4067170 0x7fdff4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:14.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.316+0000 7fe005ffb700 1 -- 172.21.15.46:0/1113645199 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7260+0+0 (secure 0 0 0) 0x7fe000084310 con 0x7fe0100b2f90 2024-01-27T01:07:14.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.317+0000 7fe015d11700 1 --2- 172.21.15.46:0/1113645199 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdff4067170 0x7fdff4069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:14.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.317+0000 7fe015d11700 1 --2- 172.21.15.46:0/1113645199 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdff4067170 0x7fdff4069620 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe00c0091e0 tx=0x7fe00c016000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:14.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.321+0000 7fe005ffb700 1 -- 172.21.15.46:0/1113645199 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe000050ce0 con 0x7fe0100b2f90 2024-01-27T01:07:14.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.574+0000 7fe017f75700 1 -- 172.21.15.46:0/1113645199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0100cd980 con 0x7fe0100b2f90 2024-01-27T01:07:14.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.574+0000 7fe005ffb700 1 -- 172.21.15.46:0/1113645199 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+4621 (secure 0 0 0) 0x7fe000017090 con 0x7fe0100b2f90 2024-01-27T01:07:14.575 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:14.575 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":124,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/1999300009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1999300009},{"type":"v1","addr":"172.21.15.106:6835","nonce":1999300009}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":110},{"gid":10486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/4174263148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4174263148},{"type":"v1","addr":"172.21.15.106:6833","nonce":4174263148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":11454,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1260622833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":1260622833},{"type":"v1","addr":"172.21.15.46:6835","nonce":1260622833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:06:40.353248+0000","modified":"2024-01-27T01:06:43.938885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10854":{"gid":10854,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":8,"addr":"172.21.15.46:6833/1263484152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1263484152},{"type":"v1","addr":"172.21.15.46:6833","nonce":1263484152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-27T01:07:14.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.576+0000 7fe017f75700 1 -- 172.21.15.46:0/1113645199 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdff4067170 msgr2=0x7fdff4069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:14.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.576+0000 7fe017f75700 1 --2- 172.21.15.46:0/1113645199 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdff4067170 0x7fdff4069620 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe00c0091e0 tx=0x7fe00c016000 comp rx=0 tx=0).stop 2024-01-27T01:07:14.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.576+0000 7fe017f75700 1 -- 172.21.15.46:0/1113645199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0100b2f90 msgr2=0x7fe0100af310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:14.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.576+0000 7fe017f75700 1 --2- 172.21.15.46:0/1113645199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0100b2f90 0x7fe0100af310 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fe00000c590 tx=0x7fe00000c8a0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.577+0000 7fe017f75700 1 -- 172.21.15.46:0/1113645199 shutdown_connections 2024-01-27T01:07:14.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.577+0000 7fe017f75700 1 --2- 172.21.15.46:0/1113645199 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdff4067170 0x7fdff4069620 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.577+0000 7fe017f75700 1 --2- 172.21.15.46:0/1113645199 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe010159610 0x7fe01014b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.577+0000 7fe017f75700 1 --2- 172.21.15.46:0/1113645199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe010068040 0x7fe01014d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.577+0000 7fe017f75700 1 --2- 172.21.15.46:0/1113645199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0100b2f90 0x7fe0100af310 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.577+0000 7fe017f75700 1 -- 172.21.15.46:0/1113645199 >> 172.21.15.46:0/1113645199 conn(0x7fe0100694d0 msgr2=0x7fe01015a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:14.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.577+0000 7fe017f75700 1 -- 172.21.15.46:0/1113645199 shutdown_connections 2024-01-27T01:07:14.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.577+0000 7fe017f75700 1 -- 172.21.15.46:0/1113645199 wait complete. 2024-01-27T01:07:14.578 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 124 2024-01-27T01:07:14.590 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:07:14.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.722+0000 7fcebce75700 1 Processor -- start 2024-01-27T01:07:14.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.722+0000 7fcebce75700 1 -- start start 2024-01-27T01:07:14.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.723+0000 7fcebce75700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fceb8157340 0x7fceb815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:14.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.723+0000 7fcebce75700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fceb80ae520 0x7fceb80b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:14.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.723+0000 7fcebce75700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb8158ae0 con 0x7fceb80b3fb0 2024-01-27T01:07:14.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.723+0000 7fcebce75700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb815b650 con 0x7fceb80ae520 2024-01-27T01:07:14.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.723+0000 7fcebce75700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb815c020 con 0x7fceb8157340 2024-01-27T01:07:14.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.723+0000 7fceb5d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fceb80ae520 0x7fceb80b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:14.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.723+0000 7fceb6d9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fceb80b3fb0 0x7fceb8157710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52992/0 (socket says 172.21.15.46:52992) 2024-01-27T01:07:14.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.723+0000 7fceb6d9d700 1 -- 172.21.15.46:0/2728822902 learned_addr learned my addr 172.21.15.46:0/2728822902 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:14.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.723+0000 7fceb659c700 1 --2- 172.21.15.46:0/2728822902 >> v2:172.21.15.106:3300/0 conn(0x7fceb8157340 0x7fceb815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:14.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.724+0000 7fceb5d9b700 1 -- 172.21.15.46:0/2728822902 >> v2:172.21.15.106:3300/0 conn(0x7fceb8157340 msgr2=0x7fceb815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:14.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.724+0000 7fceb5d9b700 1 --2- 172.21.15.46:0/2728822902 >> v2:172.21.15.106:3300/0 conn(0x7fceb8157340 0x7fceb815bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.724+0000 7fceb5d9b700 1 -- 172.21.15.46:0/2728822902 >> v1:172.21.15.46:6789/0 conn(0x7fceb80b3fb0 legacy=0x7fceb8157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:14.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.724+0000 7fceb559a700 1 -- 172.21.15.46:0/2728822902 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168828554 0 0) 0x7fceb8158ae0 con 0x7fceb80b3fb0 2024-01-27T01:07:14.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.724+0000 7fceb5d9b700 1 -- 172.21.15.46:0/2728822902 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb80b43c0 con 0x7fceb80ae520 2024-01-27T01:07:14.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.724+0000 7fceb5d9b700 1 --2- 172.21.15.46:0/2728822902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80ae520 0x7fceb80b3a70 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7fceb806f980 tx=0x7fceac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=173dada9c7b28aa2 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:14.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.724+0000 7fceb559a700 1 -- 172.21.15.46:0/2728822902 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceac011630 con 0x7fceb80ae520 2024-01-27T01:07:14.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.724+0000 7fceb559a700 1 -- 172.21.15.46:0/2728822902 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceac011790 con 0x7fceb80ae520 2024-01-27T01:07:14.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.724+0000 7fceb559a700 1 -- 172.21.15.46:0/2728822902 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceac011b30 con 0x7fceb80ae520 2024-01-27T01:07:14.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.724+0000 7fcebce75700 1 -- 172.21.15.46:0/2728822902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80ae520 msgr2=0x7fceb80b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:14.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.724+0000 7fcebce75700 1 --2- 172.21.15.46:0/2728822902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80ae520 0x7fceb80b3a70 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7fceb806f980 tx=0x7fceac0091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.725+0000 7fcebce75700 1 -- 172.21.15.46:0/2728822902 shutdown_connections 2024-01-27T01:07:14.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.725+0000 7fcebce75700 1 --2- 172.21.15.46:0/2728822902 >> v2:172.21.15.106:3300/0 conn(0x7fceb8157340 0x7fceb815bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.725+0000 7fcebce75700 1 --2- 172.21.15.46:0/2728822902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80ae520 0x7fceb80b3a70 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.725+0000 7fcebce75700 1 -- 172.21.15.46:0/2728822902 >> 172.21.15.46:0/2728822902 conn(0x7fceb80ce530 msgr2=0x7fceb80ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:14.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.725+0000 7fcebce75700 1 -- 172.21.15.46:0/2728822902 shutdown_connections 2024-01-27T01:07:14.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.725+0000 7fcebce75700 1 -- 172.21.15.46:0/2728822902 wait complete. 2024-01-27T01:07:14.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.726+0000 7fcebce75700 1 Processor -- start 2024-01-27T01:07:14.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.726+0000 7fcebce75700 1 -- start start 2024-01-27T01:07:14.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fcebce75700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80ae520 0x7fceb816a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:14.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fcebce75700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fceb80b3fb0 0x7fceb80c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:14.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fcebce75700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceb8157340 0x7fceb80b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:14.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fcebce75700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb816a110 con 0x7fceb80ae520 2024-01-27T01:07:14.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fcebce75700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb8169b20 con 0x7fceb80b3fb0 2024-01-27T01:07:14.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fcebce75700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb80c7f80 con 0x7fceb8157340 2024-01-27T01:07:14.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fceb659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80ae520 0x7fceb816a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:14.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fceb659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80ae520 0x7fceb816a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50668/0 (socket says 172.21.15.46:50668) 2024-01-27T01:07:14.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fceb659c700 1 -- 172.21.15.46:0/1743927695 learned_addr learned my addr 172.21.15.46:0/1743927695 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:14.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fceb5d9b700 1 --2- 172.21.15.46:0/1743927695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fceb80b3fb0 0x7fceb80c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:14.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fceb6d9d700 1 --2- 172.21.15.46:0/1743927695 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceb8157340 0x7fceb80b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:14.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fceb659c700 1 -- 172.21.15.46:0/1743927695 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceb8157340 msgr2=0x7fceb80b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:14.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fceb659c700 1 --2- 172.21.15.46:0/1743927695 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceb8157340 0x7fceb80b2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fceb659c700 1 -- 172.21.15.46:0/1743927695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fceb80b3fb0 msgr2=0x7fceb80c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:14.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fceb659c700 1 --2- 172.21.15.46:0/1743927695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fceb80b3fb0 0x7fceb80c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fceb659c700 1 -- 172.21.15.46:0/1743927695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceac009420 con 0x7fceb80ae520 2024-01-27T01:07:14.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fceb5d9b700 1 --2- 172.21.15.46:0/1743927695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fceb80b3fb0 0x7fceb80c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:07:14.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fceb6d9d700 1 --2- 172.21.15.46:0/1743927695 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceb8157340 0x7fceb80b2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:07:14.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fceb659c700 1 --2- 172.21.15.46:0/1743927695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80ae520 0x7fceb816a5a0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fcea800ec80 tx=0x7fcea80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:14.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fcea67fc700 1 -- 172.21.15.46:0/1743927695 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea800c9d0 con 0x7fceb80ae520 2024-01-27T01:07:14.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fcea67fc700 1 -- 172.21.15.46:0/1743927695 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea800cb50 con 0x7fceb80ae520 2024-01-27T01:07:14.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fcea67fc700 1 -- 172.21.15.46:0/1743927695 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea8013470 con 0x7fceb80ae520 2024-01-27T01:07:14.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.727+0000 7fcebce75700 1 -- 172.21.15.46:0/1743927695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb80c8260 con 0x7fceb80ae520 2024-01-27T01:07:14.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.728+0000 7fcebce75700 1 -- 172.21.15.46:0/1743927695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb816aae0 con 0x7fceb80ae520 2024-01-27T01:07:14.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.728+0000 7fcebce75700 1 -- 172.21.15.46:0/1743927695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb8169f60 con 0x7fceb80ae520 2024-01-27T01:07:14.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.729+0000 7fcea67fc700 1 -- 172.21.15.46:0/1743927695 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fcea8013860 con 0x7fceb80ae520 2024-01-27T01:07:14.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.729+0000 7fcea67fc700 1 --2- 172.21.15.46:0/1743927695 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fce98067170 0x7fce98069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:14.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.730+0000 7fcea67fc700 1 -- 172.21.15.46:0/1743927695 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7260+0+0 (secure 0 0 0) 0x7fcea8085270 con 0x7fceb80ae520 2024-01-27T01:07:14.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.730+0000 7fceb5d9b700 1 --2- 172.21.15.46:0/1743927695 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fce98067170 0x7fce98069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:14.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.730+0000 7fceb5d9b700 1 --2- 172.21.15.46:0/1743927695 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fce98067170 0x7fce98069620 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fceb80c1f20 tx=0x7fceac017000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:14.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.734+0000 7fcea67fc700 1 -- 172.21.15.46:0/1743927695 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcea8051c40 con 0x7fceb80ae520 2024-01-27T01:07:14.842 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:07:14.844 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:07:14.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.990+0000 7fcebce75700 1 -- 172.21.15.46:0/1743927695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fceb816a3f0 con 0x7fceb80ae520 2024-01-27T01:07:14.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.990+0000 7fcea67fc700 1 -- 172.21.15.46:0/1743927695 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+4621 (secure 0 0 0) 0x7fcea8055260 con 0x7fceb80ae520 2024-01-27T01:07:14.991 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:14.991 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":124,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/1999300009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1999300009},{"type":"v1","addr":"172.21.15.106:6835","nonce":1999300009}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":110},{"gid":10486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/4174263148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4174263148},{"type":"v1","addr":"172.21.15.106:6833","nonce":4174263148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":11454,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1260622833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":1260622833},{"type":"v1","addr":"172.21.15.46:6835","nonce":1260622833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:06:40.353248+0000","modified":"2024-01-27T01:06:43.938885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10854":{"gid":10854,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":8,"addr":"172.21.15.46:6833/1263484152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1263484152},{"type":"v1","addr":"172.21.15.46:6833","nonce":1263484152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-27T01:07:14.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.993+0000 7fcebce75700 1 -- 172.21.15.46:0/1743927695 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fce98067170 msgr2=0x7fce98069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:14.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.993+0000 7fcebce75700 1 --2- 172.21.15.46:0/1743927695 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fce98067170 0x7fce98069620 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fceb80c1f20 tx=0x7fceac017000 comp rx=0 tx=0).stop 2024-01-27T01:07:14.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.993+0000 7fcebce75700 1 -- 172.21.15.46:0/1743927695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80ae520 msgr2=0x7fceb816a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:14.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.993+0000 7fcebce75700 1 --2- 172.21.15.46:0/1743927695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80ae520 0x7fceb816a5a0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fcea800ec80 tx=0x7fcea80091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.993+0000 7fcebce75700 1 -- 172.21.15.46:0/1743927695 shutdown_connections 2024-01-27T01:07:14.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.993+0000 7fcebce75700 1 --2- 172.21.15.46:0/1743927695 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fce98067170 0x7fce98069620 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.993+0000 7fcebce75700 1 --2- 172.21.15.46:0/1743927695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceb80ae520 0x7fceb816a5a0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.993+0000 7fcebce75700 1 --2- 172.21.15.46:0/1743927695 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceb8157340 0x7fceb80b2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.993+0000 7fcebce75700 1 --2- 172.21.15.46:0/1743927695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fceb80b3fb0 0x7fceb80c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:14.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.993+0000 7fcebce75700 1 -- 172.21.15.46:0/1743927695 >> 172.21.15.46:0/1743927695 conn(0x7fceb80ce530 msgr2=0x7fceb80ade00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:14.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.993+0000 7fcebce75700 1 -- 172.21.15.46:0/1743927695 shutdown_connections 2024-01-27T01:07:14.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:14.993+0000 7fcebce75700 1 -- 172.21.15.46:0/1743927695 wait complete. 2024-01-27T01:07:14.994 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 124 2024-01-27T01:07:15.006 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:07:15.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.138+0000 7f2aa4cae700 1 Processor -- start 2024-01-27T01:07:15.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.138+0000 7f2aa4cae700 1 -- start start 2024-01-27T01:07:15.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.139+0000 7f2aa4cae700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2aa00cccd0 0x7f2aa00c3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:15.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.139+0000 7f2aa4cae700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa0068220 con 0x7f2aa015cd60 2024-01-27T01:07:15.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.139+0000 7f2aa4cae700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa0067930 con 0x7f2aa00c2240 2024-01-27T01:07:15.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.139+0000 7f2aa4cae700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa00c4210 con 0x7f2aa00cccd0 2024-01-27T01:07:15.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.139+0000 7f2a9dd9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2aa00cccd0 0x7f2aa00c3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:15.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.139+0000 7f2a9e59c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2aa015cd60 0x7f2aa00cc5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53000/0 (socket says 172.21.15.46:53000) 2024-01-27T01:07:15.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.139+0000 7f2a9e59c700 1 -- 172.21.15.46:0/1090852185 learned_addr learned my addr 172.21.15.46:0/1090852185 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:15.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.140+0000 7f2a9dd9b700 1 -- 172.21.15.46:0/1090852185 >> v1:172.21.15.106:6789/0 conn(0x7f2aa00c2240 legacy=0x7f2aa00c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:15.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.140+0000 7f2a9dd9b700 1 -- 172.21.15.46:0/1090852185 >> v1:172.21.15.46:6789/0 conn(0x7f2aa015cd60 legacy=0x7f2aa00cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:15.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.140+0000 7f2a9dd9b700 1 -- 172.21.15.46:0/1090852185 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa0067bd0 con 0x7f2aa00cccd0 2024-01-27T01:07:15.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.140+0000 7f2a9dd9b700 1 --2- 172.21.15.46:0/1090852185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2aa00cccd0 0x7f2aa00c3b30 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f2a940098e0 tx=0x7f2a940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c87b0d84eb805594 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:15.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.140+0000 7f2a9d59a700 1 -- 172.21.15.46:0/1090852185 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94009e80 con 0x7f2aa00cccd0 2024-01-27T01:07:15.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.140+0000 7f2a9d59a700 1 -- 172.21.15.46:0/1090852185 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9400b860 con 0x7f2aa00cccd0 2024-01-27T01:07:15.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.140+0000 7f2a9d59a700 1 -- 172.21.15.46:0/1090852185 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9400bbe0 con 0x7f2aa00cccd0 2024-01-27T01:07:15.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.141+0000 7f2aa4cae700 1 -- 172.21.15.46:0/1090852185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2aa00cccd0 msgr2=0x7f2aa00c3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:15.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.141+0000 7f2aa4cae700 1 --2- 172.21.15.46:0/1090852185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2aa00cccd0 0x7f2aa00c3b30 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f2a940098e0 tx=0x7f2a940091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.141+0000 7f2aa4cae700 1 -- 172.21.15.46:0/1090852185 shutdown_connections 2024-01-27T01:07:15.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.141+0000 7f2aa4cae700 1 --2- 172.21.15.46:0/1090852185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2aa00cccd0 0x7f2aa00c3b30 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.141+0000 7f2aa4cae700 1 -- 172.21.15.46:0/1090852185 >> 172.21.15.46:0/1090852185 conn(0x7f2aa0069220 msgr2=0x7f2aa00cd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:15.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.141+0000 7f2aa4cae700 1 -- 172.21.15.46:0/1090852185 shutdown_connections 2024-01-27T01:07:15.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.141+0000 7f2aa4cae700 1 -- 172.21.15.46:0/1090852185 wait complete. 2024-01-27T01:07:15.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.142+0000 7f2aa4cae700 1 Processor -- start 2024-01-27T01:07:15.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.142+0000 7f2aa4cae700 1 -- start start 2024-01-27T01:07:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2aa4cae700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2aa00c2240 0x7f2aa016a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2aa4cae700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2aa00cccd0 0x7f2aa00c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2aa4cae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2aa015cd60 0x7f2aa00996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2aa4cae700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa016b470 con 0x7f2aa015cd60 2024-01-27T01:07:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2aa4cae700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00c5470 con 0x7f2aa00c2240 2024-01-27T01:07:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2aa4cae700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00c56d0 con 0x7f2aa00cccd0 2024-01-27T01:07:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2a9ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2aa015cd60 0x7f2aa00996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2a9ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2aa015cd60 0x7f2aa00996b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50682/0 (socket says 172.21.15.46:50682) 2024-01-27T01:07:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2a9ed9d700 1 -- 172.21.15.46:0/640181807 learned_addr learned my addr 172.21.15.46:0/640181807 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2a9dd9b700 1 --2- 172.21.15.46:0/640181807 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2aa00cccd0 0x7f2aa00c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2a9e59c700 1 --2- 172.21.15.46:0/640181807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2aa00c2240 0x7f2aa016a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2a9ed9d700 1 -- 172.21.15.46:0/640181807 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2aa00cccd0 msgr2=0x7f2aa00c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2a9ed9d700 1 --2- 172.21.15.46:0/640181807 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2aa00cccd0 0x7f2aa00c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2a9ed9d700 1 -- 172.21.15.46:0/640181807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2aa00c2240 msgr2=0x7f2aa016a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2a9ed9d700 1 --2- 172.21.15.46:0/640181807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2aa00c2240 0x7f2aa016a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2a9ed9d700 1 -- 172.21.15.46:0/640181807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a94009420 con 0x7f2aa015cd60 2024-01-27T01:07:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2a9dd9b700 1 --2- 172.21.15.46:0/640181807 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2aa00cccd0 0x7f2aa00c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:07:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.143+0000 7f2a9e59c700 1 --2- 172.21.15.46:0/640181807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2aa00c2240 0x7f2aa016a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:07:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.144+0000 7f2a9ed9d700 1 --2- 172.21.15.46:0/640181807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2aa015cd60 0x7f2aa00996b0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f2a9800c8e0 tx=0x7f2a98001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.144+0000 7f2a8e7fc700 1 -- 172.21.15.46:0/640181807 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9800ce80 con 0x7f2aa015cd60 2024-01-27T01:07:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.144+0000 7f2a8e7fc700 1 -- 172.21.15.46:0/640181807 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9800a7e0 con 0x7f2aa015cd60 2024-01-27T01:07:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.144+0000 7f2a8e7fc700 1 -- 172.21.15.46:0/640181807 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9800aba0 con 0x7f2aa015cd60 2024-01-27T01:07:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.144+0000 7f2aa4cae700 1 -- 172.21.15.46:0/640181807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa00c59b0 con 0x7f2aa015cd60 2024-01-27T01:07:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.144+0000 7f2aa4cae700 1 -- 172.21.15.46:0/640181807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa016b100 con 0x7f2aa015cd60 2024-01-27T01:07:15.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.145+0000 7f2aa4cae700 1 -- 172.21.15.46:0/640181807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa016b5e0 con 0x7f2aa015cd60 2024-01-27T01:07:15.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.146+0000 7f2a8e7fc700 1 -- 172.21.15.46:0/640181807 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f2a98003030 con 0x7f2aa015cd60 2024-01-27T01:07:15.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.146+0000 7f2a8e7fc700 1 --2- 172.21.15.46:0/640181807 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2a80067170 0x7f2a80069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:15.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.146+0000 7f2a8e7fc700 1 -- 172.21.15.46:0/640181807 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7260+0+0 (secure 0 0 0) 0x7f2a98084e70 con 0x7f2aa015cd60 2024-01-27T01:07:15.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.146+0000 7f2a9e59c700 1 --2- 172.21.15.46:0/640181807 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2a80067170 0x7f2a80069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:15.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.147+0000 7f2a9e59c700 1 --2- 172.21.15.46:0/640181807 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2a80067170 0x7f2a80069620 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2a90001770 tx=0x7f2a90009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:15.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.151+0000 7f2a8e7fc700 1 -- 172.21.15.46:0/640181807 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2a98051840 con 0x7f2aa015cd60 2024-01-27T01:07:15.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.358+0000 7f2aa4cae700 1 -- 172.21.15.46:0/640181807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2aa004c710 con 0x7f2aa015cd60 2024-01-27T01:07:15.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.359+0000 7f2a8e7fc700 1 -- 172.21.15.46:0/640181807 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+17401 (secure 0 0 0) 0x7f2a98054e60 con 0x7f2aa015cd60 2024-01-27T01:07:15.359 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:15.360 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":232,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:07:11.870581+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:06:38.156934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:06:39.330686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6834/2483163449":"2024-01-28T01:07:08.439983+0000","172.21.15.106:0/2812454604":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/3610030821":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/3431540914":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/42206641":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/3926539126":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/694514533":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/819986778":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/3809430766":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/981929816":"2024-01-28T01:07:07.849128+0000","172.21.15.106:6837/62434":"2024-01-28T01:06:57.756975+0000","172.21.15.106:6836/62434":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/524220301":"2024-01-28T01:06:57.756975+0000","172.21.15.46:6835/2483163449":"2024-01-28T01:07:08.439983+0000","172.21.15.106:0/4002309435":"2024-01-28T01:06:57.756975+0000","172.21.15.106:0/3400107086":"2024-01-28T01:07:07.849128+0000","172.21.15.106:0/2725250616":"2024-01-28T01:06:57.756975+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:07:15.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.361+0000 7f2aa4cae700 1 -- 172.21.15.46:0/640181807 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2a80067170 msgr2=0x7f2a80069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:15.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.361+0000 7f2aa4cae700 1 --2- 172.21.15.46:0/640181807 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2a80067170 0x7f2a80069620 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2a90001770 tx=0x7f2a90009040 comp rx=0 tx=0).stop 2024-01-27T01:07:15.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.361+0000 7f2aa4cae700 1 -- 172.21.15.46:0/640181807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2aa015cd60 msgr2=0x7f2aa00996b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:15.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.361+0000 7f2aa4cae700 1 --2- 172.21.15.46:0/640181807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2aa015cd60 0x7f2aa00996b0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f2a9800c8e0 tx=0x7f2a98001710 comp rx=0 tx=0).stop 2024-01-27T01:07:15.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.361+0000 7f2aa4cae700 1 -- 172.21.15.46:0/640181807 shutdown_connections 2024-01-27T01:07:15.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.361+0000 7f2aa4cae700 1 --2- 172.21.15.46:0/640181807 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2a80067170 0x7f2a80069620 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.361+0000 7f2aa4cae700 1 --2- 172.21.15.46:0/640181807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2aa015cd60 0x7f2aa00996b0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.361+0000 7f2aa4cae700 1 --2- 172.21.15.46:0/640181807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2aa00c2240 0x7f2aa016a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.361+0000 7f2aa4cae700 1 --2- 172.21.15.46:0/640181807 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2aa00cccd0 0x7f2aa00c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.361+0000 7f2aa4cae700 1 -- 172.21.15.46:0/640181807 >> 172.21.15.46:0/640181807 conn(0x7f2aa0069220 msgr2=0x7f2aa006b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:15.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.361+0000 7f2aa4cae700 1 -- 172.21.15.46:0/640181807 shutdown_connections 2024-01-27T01:07:15.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.361+0000 7f2aa4cae700 1 -- 172.21.15.46:0/640181807 wait complete. 2024-01-27T01:07:15.375 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-01-27T01:07:15.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.508+0000 7f005603a700 1 Processor -- start 2024-01-27T01:07:15.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.508+0000 7f005603a700 1 -- start start 2024-01-27T01:07:15.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.509+0000 7f005603a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0050159760 0x7f00500b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.509+0000 7f005603a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f00500b3060 0x7f00500b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.509+0000 7f005603a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005006f940 con 0x7f0050159b30 2024-01-27T01:07:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.509+0000 7f005603a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f005006ed30 con 0x7f00500b3060 2024-01-27T01:07:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.509+0000 7f005603a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005006f010 con 0x7f0050159760 2024-01-27T01:07:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.509+0000 7f004effd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f00500b3060 0x7f00500b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.509+0000 7f004ffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0050159b30 0x7f005015d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53004/0 (socket says 172.21.15.46:53004) 2024-01-27T01:07:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.509+0000 7f004effd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f00500b3060 0x7f00500b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50698/0 (socket says 172.21.15.46:50698) 2024-01-27T01:07:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.509+0000 7f004ffff700 1 -- 172.21.15.46:0/1141866012 learned_addr learned my addr 172.21.15.46:0/1141866012 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.509+0000 7f004f7fe700 1 --2- 172.21.15.46:0/1141866012 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0050159760 0x7f00500b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.509+0000 7f004effd700 1 -- 172.21.15.46:0/1141866012 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0050159760 msgr2=0x7f00500b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.509+0000 7f004effd700 1 --2- 172.21.15.46:0/1141866012 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0050159760 0x7f00500b2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.509+0000 7f004effd700 1 -- 172.21.15.46:0/1141866012 >> v1:172.21.15.46:6789/0 conn(0x7f0050159b30 legacy=0x7f005015d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.509+0000 7f004effd700 1 -- 172.21.15.46:0/1141866012 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005006f2f0 con 0x7f00500b3060 2024-01-27T01:07:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.509+0000 7f004e7fc700 1 -- 172.21.15.46:0/1141866012 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1863423448 0 0) 0x7f005006f940 con 0x7f0050159b30 2024-01-27T01:07:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.510+0000 7f004effd700 1 --2- 172.21.15.46:0/1141866012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00500b3060 0x7f00500b0bb0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f00380098e0 tx=0x7f00380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f8ac85082d43e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.510+0000 7f004e7fc700 1 -- 172.21.15.46:0/1141866012 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0038009e80 con 0x7f00500b3060 2024-01-27T01:07:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.510+0000 7f004e7fc700 1 -- 172.21.15.46:0/1141866012 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003800b860 con 0x7f00500b3060 2024-01-27T01:07:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.510+0000 7f004e7fc700 1 -- 172.21.15.46:0/1141866012 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003800bbe0 con 0x7f00500b3060 2024-01-27T01:07:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.510+0000 7f005603a700 1 -- 172.21.15.46:0/1141866012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00500b3060 msgr2=0x7f00500b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.510+0000 7f005603a700 1 --2- 172.21.15.46:0/1141866012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00500b3060 0x7f00500b0bb0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f00380098e0 tx=0x7f00380091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.510+0000 7f005603a700 1 -- 172.21.15.46:0/1141866012 shutdown_connections 2024-01-27T01:07:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.510+0000 7f005603a700 1 --2- 172.21.15.46:0/1141866012 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0050159760 0x7f00500b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.510+0000 7f005603a700 1 --2- 172.21.15.46:0/1141866012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00500b3060 0x7f00500b0bb0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.510+0000 7f005603a700 1 -- 172.21.15.46:0/1141866012 >> 172.21.15.46:0/1141866012 conn(0x7f0050069510 msgr2=0x7f00500aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.511+0000 7f005603a700 1 -- 172.21.15.46:0/1141866012 shutdown_connections 2024-01-27T01:07:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.511+0000 7f005603a700 1 -- 172.21.15.46:0/1141866012 wait complete. 2024-01-27T01:07:15.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.512+0000 7f005603a700 1 Processor -- start 2024-01-27T01:07:15.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.512+0000 7f005603a700 1 -- start start 2024-01-27T01:07:15.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f005603a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00500b3060 0x7f0050169da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:15.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f005603a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0050159760 0x7f00500aa9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:15.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f005603a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0050159b30 0x7f00500aaf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:15.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f005603a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005016caf0 con 0x7f00500b3060 2024-01-27T01:07:15.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f005603a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005016d670 con 0x7f0050159760 2024-01-27T01:07:15.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f005603a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005016cd40 con 0x7f0050159b30 2024-01-27T01:07:15.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f004f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00500b3060 0x7f0050169da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f004f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00500b3060 0x7f0050169da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50708/0 (socket says 172.21.15.46:50708) 2024-01-27T01:07:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f004f7fe700 1 -- 172.21.15.46:0/2363026109 learned_addr learned my addr 172.21.15.46:0/2363026109 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f004ffff700 1 --2- 172.21.15.46:0/2363026109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0050159b30 0x7f00500aaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f004effd700 1 --2- 172.21.15.46:0/2363026109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0050159760 0x7f00500aa9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f004f7fe700 1 -- 172.21.15.46:0/2363026109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0050159b30 msgr2=0x7f00500aaf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f004f7fe700 1 --2- 172.21.15.46:0/2363026109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0050159b30 0x7f00500aaf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f004f7fe700 1 -- 172.21.15.46:0/2363026109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0050159760 msgr2=0x7f00500aa9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f004f7fe700 1 --2- 172.21.15.46:0/2363026109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0050159760 0x7f00500aa9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f004f7fe700 1 -- 172.21.15.46:0/2363026109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0038009420 con 0x7f00500b3060 2024-01-27T01:07:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f004effd700 1 --2- 172.21.15.46:0/2363026109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0050159760 0x7f00500aa9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:07:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.513+0000 7f004f7fe700 1 --2- 172.21.15.46:0/2363026109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00500b3060 0x7f0050169da0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f004000d8c0 tx=0x7f0040008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.514+0000 7f00477fe700 1 -- 172.21.15.46:0/2363026109 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004000de60 con 0x7f00500b3060 2024-01-27T01:07:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.514+0000 7f00477fe700 1 -- 172.21.15.46:0/2363026109 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f004000b780 con 0x7f00500b3060 2024-01-27T01:07:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.514+0000 7f005603a700 1 -- 172.21.15.46:0/2363026109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f005016d020 con 0x7f00500b3060 2024-01-27T01:07:15.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.514+0000 7f00477fe700 1 -- 172.21.15.46:0/2363026109 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004000bb20 con 0x7f00500b3060 2024-01-27T01:07:15.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.514+0000 7f005603a700 1 -- 172.21.15.46:0/2363026109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f005016a2e0 con 0x7f00500b3060 2024-01-27T01:07:15.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.515+0000 7f005603a700 1 -- 172.21.15.46:0/2363026109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f005004c710 con 0x7f00500b3060 2024-01-27T01:07:15.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.515+0000 7f00477fe700 1 -- 172.21.15.46:0/2363026109 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f004001e460 con 0x7f00500b3060 2024-01-27T01:07:15.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.516+0000 7f00477fe700 1 --2- 172.21.15.46:0/2363026109 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0030067170 0x7f0030069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:15.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.516+0000 7f00477fe700 1 -- 172.21.15.46:0/2363026109 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7260+0+0 (secure 0 0 0) 0x7f0040085540 con 0x7f00500b3060 2024-01-27T01:07:15.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.516+0000 7f004effd700 1 --2- 172.21.15.46:0/2363026109 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0030067170 0x7f0030069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:15.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.516+0000 7f004effd700 1 --2- 172.21.15.46:0/2363026109 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0030067170 0x7f0030069620 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f00500ab640 tx=0x7f0038017000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:15.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.520+0000 7f00477fe700 1 -- 172.21.15.46:0/2363026109 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0040051f10 con 0x7f00500b3060 2024-01-27T01:07:15.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.770+0000 7f005603a700 1 -- 172.21.15.46:0/2363026109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7f005006ddd0 con 0x7f00500b3060 2024-01-27T01:07:15.770 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-27T01:07:15.770+0000 7f33ffe86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-27T01:07:15.777 INFO:tasks.ceph.mds.c.smithi046.stderr: -1803> 2024-01-27T01:06:42.773+0000 7f6f3f584700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-27T01:07:15.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.780+0000 7f00477fe700 1 -- 172.21.15.46:0/2363026109 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v125) v1 ==== 167+0+0 (secure 0 0 0) 0x7f0040055530 con 0x7f00500b3060 2024-01-27T01:07:15.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.782+0000 7f005603a700 1 -- 172.21.15.46:0/2363026109 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0030067170 msgr2=0x7f0030069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:15.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.782+0000 7f005603a700 1 --2- 172.21.15.46:0/2363026109 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0030067170 0x7f0030069620 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f00500ab640 tx=0x7f0038017000 comp rx=0 tx=0).stop 2024-01-27T01:07:15.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.782+0000 7f005603a700 1 -- 172.21.15.46:0/2363026109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00500b3060 msgr2=0x7f0050169da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:15.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.782+0000 7f005603a700 1 --2- 172.21.15.46:0/2363026109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00500b3060 0x7f0050169da0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f004000d8c0 tx=0x7f0040008040 comp rx=0 tx=0).stop 2024-01-27T01:07:15.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.783+0000 7f005603a700 1 -- 172.21.15.46:0/2363026109 shutdown_connections 2024-01-27T01:07:15.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.783+0000 7f005603a700 1 --2- 172.21.15.46:0/2363026109 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0030067170 0x7f0030069620 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.783+0000 7f005603a700 1 --2- 172.21.15.46:0/2363026109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00500b3060 0x7f0050169da0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.783+0000 7f005603a700 1 --2- 172.21.15.46:0/2363026109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0050159b30 0x7f00500aaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.783+0000 7f005603a700 1 --2- 172.21.15.46:0/2363026109 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0050159760 0x7f00500aa9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.783+0000 7f005603a700 1 -- 172.21.15.46:0/2363026109 >> 172.21.15.46:0/2363026109 conn(0x7f0050069510 msgr2=0x7f0050158a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:15.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.783+0000 7f005603a700 1 -- 172.21.15.46:0/2363026109 shutdown_connections 2024-01-27T01:07:15.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.783+0000 7f005603a700 1 -- 172.21.15.46:0/2363026109 wait complete. 2024-01-27T01:07:15.784 INFO:teuthology.orchestra.run.smithi046.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-27T01:07:15.798 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-01-27T01:07:15.801 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-27T01:07:15.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.932+0000 7efff09d2700 1 Processor -- start 2024-01-27T01:07:15.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.932+0000 7efff09d2700 1 -- start start 2024-01-27T01:07:15.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.933+0000 7efff09d2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec156570 0x7effec06c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:15.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.933+0000 7efff09d2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7effec06f9a0 0x7effec06eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:15.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.933+0000 7efff09d2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effec0c8b90 con 0x7effec156940 2024-01-27T01:07:15.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.933+0000 7efff09d2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effec06d0b0 con 0x7effec06f9a0 2024-01-27T01:07:15.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.933+0000 7efff09d2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effec0c8e50 con 0x7effec156570 2024-01-27T01:07:15.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.934+0000 7effead9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7effec06f9a0 0x7effec06eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:15.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.934+0000 7effead9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7effec06f9a0 0x7effec06eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50728/0 (socket says 172.21.15.46:50728) 2024-01-27T01:07:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.934+0000 7effe9d9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7effec156940 0x7effec0c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53024/0 (socket says 172.21.15.46:53024) 2024-01-27T01:07:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.934+0000 7effead9d700 1 -- 172.21.15.46:0/3639593988 learned_addr learned my addr 172.21.15.46:0/3639593988 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.934+0000 7effea59c700 1 --2- 172.21.15.46:0/3639593988 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec156570 0x7effec06c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.934+0000 7effead9d700 1 -- 172.21.15.46:0/3639593988 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec156570 msgr2=0x7effec06c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.934+0000 7effead9d700 1 --2- 172.21.15.46:0/3639593988 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec156570 0x7effec06c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.934+0000 7effead9d700 1 -- 172.21.15.46:0/3639593988 >> v1:172.21.15.46:6789/0 conn(0x7effec156940 legacy=0x7effec0c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.934+0000 7effead9d700 1 -- 172.21.15.46:0/3639593988 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effec0c4c40 con 0x7effec06f9a0 2024-01-27T01:07:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.934+0000 7effe959a700 1 -- 172.21.15.46:0/3639593988 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095583622 0 0) 0x7effec0c8b90 con 0x7effec156940 2024-01-27T01:07:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.934+0000 7effead9d700 1 --2- 172.21.15.46:0/3639593988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec06f9a0 0x7effec06eb90 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7effe400d1d0 tx=0x7effe400ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d586529d76cd7c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.934+0000 7effe959a700 1 -- 172.21.15.46:0/3639593988 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe4010070 con 0x7effec06f9a0 2024-01-27T01:07:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.934+0000 7effe959a700 1 -- 172.21.15.46:0/3639593988 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe400de00 con 0x7effec06f9a0 2024-01-27T01:07:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.934+0000 7effe959a700 1 -- 172.21.15.46:0/3639593988 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe400b320 con 0x7effec06f9a0 2024-01-27T01:07:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.935+0000 7efff09d2700 1 -- 172.21.15.46:0/3639593988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec06f9a0 msgr2=0x7effec06eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.935+0000 7efff09d2700 1 --2- 172.21.15.46:0/3639593988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec06f9a0 0x7effec06eb90 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7effe400d1d0 tx=0x7effe400ba50 comp rx=0 tx=0).stop 2024-01-27T01:07:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.935+0000 7efff09d2700 1 -- 172.21.15.46:0/3639593988 shutdown_connections 2024-01-27T01:07:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.935+0000 7efff09d2700 1 --2- 172.21.15.46:0/3639593988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec06f9a0 0x7effec06eb90 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.935+0000 7efff09d2700 1 --2- 172.21.15.46:0/3639593988 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec156570 0x7effec06c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.935+0000 7efff09d2700 1 -- 172.21.15.46:0/3639593988 >> 172.21.15.46:0/3639593988 conn(0x7effec0cdf50 msgr2=0x7effec0c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.935+0000 7efff09d2700 1 -- 172.21.15.46:0/3639593988 shutdown_connections 2024-01-27T01:07:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.935+0000 7efff09d2700 1 -- 172.21.15.46:0/3639593988 wait complete. 2024-01-27T01:07:15.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.936+0000 7efff09d2700 1 Processor -- start 2024-01-27T01:07:15.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.936+0000 7efff09d2700 1 -- start start 2024-01-27T01:07:15.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.937+0000 7efff09d2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec06f9a0 0x7effec150690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:15.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.937+0000 7efff09d2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effec156570 0x7effec0cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:15.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.937+0000 7efff09d2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec156940 0x7effec0cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:15.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.937+0000 7efff09d2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effec164d50 con 0x7effec156940 2024-01-27T01:07:15.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.937+0000 7efff09d2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effec0cbc30 con 0x7effec156570 2024-01-27T01:07:15.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.937+0000 7efff09d2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effec165590 con 0x7effec06f9a0 2024-01-27T01:07:15.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.937+0000 7effead9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec156940 0x7effec0cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:15.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.937+0000 7effead9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec156940 0x7effec0cb320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50740/0 (socket says 172.21.15.46:50740) 2024-01-27T01:07:15.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.937+0000 7effead9d700 1 -- 172.21.15.46:0/1396266723 learned_addr learned my addr 172.21.15.46:0/1396266723 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:15.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.937+0000 7effea59c700 1 --2- 172.21.15.46:0/1396266723 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec06f9a0 0x7effec150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:15.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.937+0000 7effe9d9b700 1 --2- 172.21.15.46:0/1396266723 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effec156570 0x7effec0cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:15.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.938+0000 7effead9d700 1 -- 172.21.15.46:0/1396266723 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec06f9a0 msgr2=0x7effec150690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:15.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.938+0000 7effead9d700 1 --2- 172.21.15.46:0/1396266723 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec06f9a0 0x7effec150690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.938+0000 7effead9d700 1 -- 172.21.15.46:0/1396266723 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effec156570 msgr2=0x7effec0cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:15.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.938+0000 7effead9d700 1 --2- 172.21.15.46:0/1396266723 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effec156570 0x7effec0cade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:15.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.938+0000 7effead9d700 1 -- 172.21.15.46:0/1396266723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe400bcf0 con 0x7effec156940 2024-01-27T01:07:15.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.938+0000 7effe9d9b700 1 --2- 172.21.15.46:0/1396266723 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effec156570 0x7effec0cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:15.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.938+0000 7effead9d700 1 --2- 172.21.15.46:0/1396266723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec156940 0x7effec0cb320 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7effe400b660 tx=0x7effe401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:15.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.938+0000 7effe27fc700 1 -- 172.21.15.46:0/1396266723 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe400acd0 con 0x7effec156940 2024-01-27T01:07:15.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.938+0000 7effe27fc700 1 -- 172.21.15.46:0/1396266723 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe400ae30 con 0x7effec156940 2024-01-27T01:07:15.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.938+0000 7effe27fc700 1 -- 172.21.15.46:0/1396266723 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe40286b0 con 0x7effec156940 2024-01-27T01:07:15.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.938+0000 7efff09d2700 1 -- 172.21.15.46:0/1396266723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effec0b2b00 con 0x7effec156940 2024-01-27T01:07:15.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.938+0000 7efff09d2700 1 -- 172.21.15.46:0/1396266723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effec164f40 con 0x7effec156940 2024-01-27T01:07:15.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.939+0000 7efff09d2700 1 -- 172.21.15.46:0/1396266723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effec04c710 con 0x7effec156940 2024-01-27T01:07:15.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.939+0000 7effe27fc700 1 -- 172.21.15.46:0/1396266723 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7effe4003040 con 0x7effec156940 2024-01-27T01:07:15.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.940+0000 7effe27fc700 1 --2- 172.21.15.46:0/1396266723 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7effcc067170 0x7effcc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:15.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.940+0000 7effe27fc700 1 -- 172.21.15.46:0/1396266723 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7346+0+0 (secure 0 0 0) 0x7effe4089090 con 0x7effec156940 2024-01-27T01:07:15.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.940+0000 7effea59c700 1 --2- 172.21.15.46:0/1396266723 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7effcc067170 0x7effcc069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:15.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.941+0000 7effea59c700 1 --2- 172.21.15.46:0/1396266723 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7effcc067170 0x7effcc069620 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7effd4009db0 tx=0x7effd40091a0 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:15.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:15.945+0000 7effe27fc700 1 -- 172.21.15.46:0/1396266723 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7effe40558f0 con 0x7effec156940 2024-01-27T01:07:16.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.195+0000 7efff09d2700 1 -- 172.21.15.46:0/1396266723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7effec1656d0 con 0x7effec156940 2024-01-27T01:07:16.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.778+0000 7effe27fc700 1 -- 172.21.15.46:0/1396266723 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v126) v1 ==== 107+0+0 (secure 0 0 0) 0x7effe4058f10 con 0x7effec156940 2024-01-27T01:07:16.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.781+0000 7efff09d2700 1 -- 172.21.15.46:0/1396266723 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7effcc067170 msgr2=0x7effcc069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:16.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.781+0000 7efff09d2700 1 --2- 172.21.15.46:0/1396266723 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7effcc067170 0x7effcc069620 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7effd4009db0 tx=0x7effd40091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:16.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.781+0000 7efff09d2700 1 -- 172.21.15.46:0/1396266723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec156940 msgr2=0x7effec0cb320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:16.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.781+0000 7efff09d2700 1 --2- 172.21.15.46:0/1396266723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec156940 0x7effec0cb320 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7effe400b660 tx=0x7effe401a000 comp rx=0 tx=0).stop 2024-01-27T01:07:16.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.781+0000 7efff09d2700 1 -- 172.21.15.46:0/1396266723 shutdown_connections 2024-01-27T01:07:16.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.781+0000 7efff09d2700 1 --2- 172.21.15.46:0/1396266723 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7effcc067170 0x7effcc069620 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:16.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.781+0000 7efff09d2700 1 --2- 172.21.15.46:0/1396266723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effec156940 0x7effec0cb320 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:16.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.781+0000 7efff09d2700 1 --2- 172.21.15.46:0/1396266723 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7effec06f9a0 0x7effec150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:16.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.781+0000 7efff09d2700 1 --2- 172.21.15.46:0/1396266723 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effec156570 0x7effec0cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:16.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.781+0000 7efff09d2700 1 -- 172.21.15.46:0/1396266723 >> 172.21.15.46:0/1396266723 conn(0x7effec0cdf50 msgr2=0x7effec155760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:16.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.781+0000 7efff09d2700 1 -- 172.21.15.46:0/1396266723 shutdown_connections 2024-01-27T01:07:16.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.781+0000 7efff09d2700 1 -- 172.21.15.46:0/1396266723 wait complete. 2024-01-27T01:07:16.797 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-01-27T01:07:16.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.939+0000 7f0048180700 1 Processor -- start 2024-01-27T01:07:16.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.939+0000 7f0048180700 1 -- start start 2024-01-27T01:07:16.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.940+0000 7f0048180700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f00400c9d50 0x7f004015ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:16.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.940+0000 7f0048180700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0040157000 con 0x7f0040156b00 2024-01-27T01:07:16.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.940+0000 7f0048180700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f004006f3e0 con 0x7f00400c9980 2024-01-27T01:07:16.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.940+0000 7f0048180700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f004006f6c0 con 0x7f00400c9d50 2024-01-27T01:07:16.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.940+0000 7f004671d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f00400c9d50 0x7f004015ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:16.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.940+0000 7f004571b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0040156b00 0x7f00400b1aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53030/0 (socket says 172.21.15.46:53030) 2024-01-27T01:07:16.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.940+0000 7f004571b700 1 -- 172.21.15.46:0/4068728943 learned_addr learned my addr 172.21.15.46:0/4068728943 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:16.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.940+0000 7f004671d700 1 -- 172.21.15.46:0/4068728943 >> v1:172.21.15.106:6789/0 conn(0x7f00400c9980 legacy=0x7f004006eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:16.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.940+0000 7f004671d700 1 -- 172.21.15.46:0/4068728943 >> v1:172.21.15.46:6789/0 conn(0x7f0040156b00 legacy=0x7f00400b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:16.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.940+0000 7f004671d700 1 -- 172.21.15.46:0/4068728943 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f004006f9a0 con 0x7f00400c9d50 2024-01-27T01:07:16.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.940+0000 7f0044f1a700 1 -- 172.21.15.46:0/4068728943 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147495791 0 0) 0x7f0040157000 con 0x7f0040156b00 2024-01-27T01:07:16.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.941+0000 7f004671d700 1 --2- 172.21.15.46:0/4068728943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00400c9d50 0x7f004015ae10 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f0038009e60 tx=0x7f0038009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da367860f36d4f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:16.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.941+0000 7f0044f1a700 1 -- 172.21.15.46:0/4068728943 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003800e070 con 0x7f00400c9d50 2024-01-27T01:07:16.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.941+0000 7f0044f1a700 1 -- 172.21.15.46:0/4068728943 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003800bbf0 con 0x7f00400c9d50 2024-01-27T01:07:16.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.941+0000 7f0044f1a700 1 -- 172.21.15.46:0/4068728943 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0038012430 con 0x7f00400c9d50 2024-01-27T01:07:16.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.941+0000 7f0048180700 1 -- 172.21.15.46:0/4068728943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00400c9d50 msgr2=0x7f004015ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:16.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.941+0000 7f0048180700 1 --2- 172.21.15.46:0/4068728943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00400c9d50 0x7f004015ae10 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f0038009e60 tx=0x7f0038009780 comp rx=0 tx=0).stop 2024-01-27T01:07:16.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.941+0000 7f0048180700 1 -- 172.21.15.46:0/4068728943 shutdown_connections 2024-01-27T01:07:16.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.941+0000 7f0048180700 1 --2- 172.21.15.46:0/4068728943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00400c9d50 0x7f004015ae10 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:16.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.941+0000 7f0048180700 1 -- 172.21.15.46:0/4068728943 >> 172.21.15.46:0/4068728943 conn(0x7f0040069510 msgr2=0x7f00400aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:16.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.941+0000 7f0048180700 1 -- 172.21.15.46:0/4068728943 shutdown_connections 2024-01-27T01:07:16.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.941+0000 7f0048180700 1 -- 172.21.15.46:0/4068728943 wait complete. 2024-01-27T01:07:16.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.943+0000 7f0048180700 1 Processor -- start 2024-01-27T01:07:16.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.943+0000 7f0048180700 1 -- start start 2024-01-27T01:07:16.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.943+0000 7f0048180700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f00400c9980 0x7f0040152050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:16.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.943+0000 7f0048180700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00400c9d50 0x7f0040157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:16.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.943+0000 7f0048180700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0040156b00 0x7f00401554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:16.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.943+0000 7f0048180700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0040158a80 con 0x7f00400c9d50 2024-01-27T01:07:16.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.943+0000 7f0048180700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0040157e00 con 0x7f00400c9980 2024-01-27T01:07:16.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.943+0000 7f0048180700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00401580e0 con 0x7f0040156b00 2024-01-27T01:07:16.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.944+0000 7f004571b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00400c9d50 0x7f0040157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:16.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.944+0000 7f004571b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00400c9d50 0x7f0040157490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50752/0 (socket says 172.21.15.46:50752) 2024-01-27T01:07:16.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.944+0000 7f004571b700 1 -- 172.21.15.46:0/1214814646 learned_addr learned my addr 172.21.15.46:0/1214814646 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.944+0000 7f004671d700 1 --2- 172.21.15.46:0/1214814646 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0040156b00 0x7f00401554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.944+0000 7f0045f1c700 1 --2- 172.21.15.46:0/1214814646 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f00400c9980 0x7f0040152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.944+0000 7f004571b700 1 -- 172.21.15.46:0/1214814646 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0040156b00 msgr2=0x7f00401554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.944+0000 7f004571b700 1 --2- 172.21.15.46:0/1214814646 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0040156b00 0x7f00401554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.944+0000 7f004571b700 1 -- 172.21.15.46:0/1214814646 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f00400c9980 msgr2=0x7f0040152050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.944+0000 7f004571b700 1 --2- 172.21.15.46:0/1214814646 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f00400c9980 0x7f0040152050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.944+0000 7f004571b700 1 -- 172.21.15.46:0/1214814646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0038009a20 con 0x7f00400c9d50 2024-01-27T01:07:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.944+0000 7f004571b700 1 --2- 172.21.15.46:0/1214814646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00400c9d50 0x7f0040157490 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f003000c5b0 tx=0x7f0030001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.944+0000 7f0035ffb700 1 -- 172.21.15.46:0/1214814646 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0030005b30 con 0x7f00400c9d50 2024-01-27T01:07:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.944+0000 7f0035ffb700 1 -- 172.21.15.46:0/1214814646 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0030005c90 con 0x7f00400c9d50 2024-01-27T01:07:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.944+0000 7f0048180700 1 -- 172.21.15.46:0/1214814646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00401583c0 con 0x7f00400c9d50 2024-01-27T01:07:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.944+0000 7f0035ffb700 1 -- 172.21.15.46:0/1214814646 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0030007300 con 0x7f00400c9d50 2024-01-27T01:07:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.945+0000 7f0048180700 1 -- 172.21.15.46:0/1214814646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0040152590 con 0x7f00400c9d50 2024-01-27T01:07:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.945+0000 7f0048180700 1 -- 172.21.15.46:0/1214814646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0040158bc0 con 0x7f00400c9d50 2024-01-27T01:07:16.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.946+0000 7f0035ffb700 1 -- 172.21.15.46:0/1214814646 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f00300076b0 con 0x7f00400c9d50 2024-01-27T01:07:16.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.947+0000 7f0035ffb700 1 --2- 172.21.15.46:0/1214814646 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0024067170 0x7f0024069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:16.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.947+0000 7f0035ffb700 1 -- 172.21.15.46:0/1214814646 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7346+0+0 (secure 0 0 0) 0x7f0030084440 con 0x7f00400c9d50 2024-01-27T01:07:16.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.947+0000 7f0045f1c700 1 --2- 172.21.15.46:0/1214814646 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0024067170 0x7f0024069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:16.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.947+0000 7f0045f1c700 1 --2- 172.21.15.46:0/1214814646 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0024067170 0x7f0024069620 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f003c003870 tx=0x7f003c009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:16.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:16.951+0000 7f0035ffb700 1 -- 172.21.15.46:0/1214814646 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0040158bc0 con 0x7f00400c9d50 2024-01-27T01:07:17.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:17.153+0000 7f0048180700 1 -- 172.21.15.46:0/1214814646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0040158bc0 con 0x7f00400c9d50 2024-01-27T01:07:17.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:17.892+0000 7f0035ffb700 1 -- 172.21.15.46:0/1214814646 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v234) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0040158bc0 con 0x7f00400c9d50 2024-01-27T01:07:17.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:17.895+0000 7f0048180700 1 -- 172.21.15.46:0/1214814646 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0024067170 msgr2=0x7f0024069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:17.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:17.895+0000 7f0048180700 1 --2- 172.21.15.46:0/1214814646 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0024067170 0x7f0024069620 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f003c003870 tx=0x7f003c009040 comp rx=0 tx=0).stop 2024-01-27T01:07:17.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:17.895+0000 7f0048180700 1 -- 172.21.15.46:0/1214814646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00400c9d50 msgr2=0x7f0040157490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:17.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:17.895+0000 7f0048180700 1 --2- 172.21.15.46:0/1214814646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00400c9d50 0x7f0040157490 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f003000c5b0 tx=0x7f0030001070 comp rx=0 tx=0).stop 2024-01-27T01:07:17.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:17.895+0000 7f0048180700 1 -- 172.21.15.46:0/1214814646 shutdown_connections 2024-01-27T01:07:17.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:17.895+0000 7f0048180700 1 --2- 172.21.15.46:0/1214814646 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0024067170 0x7f0024069620 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:17.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:17.895+0000 7f0048180700 1 --2- 172.21.15.46:0/1214814646 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0040156b00 0x7f00401554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:17.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:17.895+0000 7f0048180700 1 --2- 172.21.15.46:0/1214814646 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f00400c9980 0x7f0040152050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:17.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:17.895+0000 7f0048180700 1 --2- 172.21.15.46:0/1214814646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00400c9d50 0x7f0040157490 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:17.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:17.895+0000 7f0048180700 1 -- 172.21.15.46:0/1214814646 >> 172.21.15.46:0/1214814646 conn(0x7f0040069510 msgr2=0x7f00400ae780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:17.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:17.895+0000 7f0048180700 1 -- 172.21.15.46:0/1214814646 shutdown_connections 2024-01-27T01:07:17.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:17.895+0000 7f0048180700 1 -- 172.21.15.46:0/1214814646 wait complete. 2024-01-27T01:07:17.897 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'backup_fs_metadata' removed 2024-01-27T01:07:17.911 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-01-27T01:07:18.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.051+0000 7fbf04be0700 1 Processor -- start 2024-01-27T01:07:18.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.051+0000 7fbf04be0700 1 -- start start 2024-01-27T01:07:18.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.052+0000 7fbf04be0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbf0015ce40 0x7fbf000d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:18.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.052+0000 7fbf04be0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf0015d210 0x7fbf000cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:18.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.052+0000 7fbf04be0700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbf000d1390 0x7fbf000cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:18.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.052+0000 7fbf04be0700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf000cff70 con 0x7fbf0015ce40 2024-01-27T01:07:18.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.052+0000 7fbf04be0700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf000cf400 con 0x7fbf000d1390 2024-01-27T01:07:18.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.052+0000 7fbf04be0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf000cf660 con 0x7fbf0015d210 2024-01-27T01:07:18.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.052+0000 7fbefe59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbf0015ce40 0x7fbf000d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:18.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.052+0000 7fbefe59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbf0015ce40 0x7fbf000d0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50766/0 (socket says 172.21.15.46:50766) 2024-01-27T01:07:18.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.052+0000 7fbefe59c700 1 -- 172.21.15.46:0/3059037926 learned_addr learned my addr 172.21.15.46:0/3059037926 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:18.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.052+0000 7fbefed9d700 1 --2- 172.21.15.46:0/3059037926 >> v2:172.21.15.106:3300/0 conn(0x7fbf000d1390 0x7fbf000cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:18.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.052+0000 7fbefdd9b700 1 --2- 172.21.15.46:0/3059037926 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf0015d210 0x7fbf000cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:18.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.053+0000 7fbefe59c700 1 -- 172.21.15.46:0/3059037926 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf0015d210 msgr2=0x7fbf000cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:18.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.053+0000 7fbefe59c700 1 --2- 172.21.15.46:0/3059037926 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf0015d210 0x7fbf000cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:18.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.053+0000 7fbefe59c700 1 -- 172.21.15.46:0/3059037926 >> v2:172.21.15.106:3300/0 conn(0x7fbf000d1390 msgr2=0x7fbf000cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:18.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.053+0000 7fbefe59c700 1 --2- 172.21.15.46:0/3059037926 >> v2:172.21.15.106:3300/0 conn(0x7fbf000d1390 0x7fbf000cbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:18.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.053+0000 7fbefe59c700 1 -- 172.21.15.46:0/3059037926 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf000d0500 con 0x7fbf0015ce40 2024-01-27T01:07:18.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.053+0000 7fbefe59c700 1 --2- 172.21.15.46:0/3059037926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf0015ce40 0x7fbf000d0e50 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fbf00089b70 tx=0x7fbef0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e81bbaf148c24b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:18.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.053+0000 7fbefd59a700 1 -- 172.21.15.46:0/3059037926 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef000b640 con 0x7fbf0015ce40 2024-01-27T01:07:18.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.053+0000 7fbefd59a700 1 -- 172.21.15.46:0/3059037926 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef000b7a0 con 0x7fbf0015ce40 2024-01-27T01:07:18.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.053+0000 7fbefd59a700 1 -- 172.21.15.46:0/3059037926 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef000bb60 con 0x7fbf0015ce40 2024-01-27T01:07:18.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.054+0000 7fbf04be0700 1 -- 172.21.15.46:0/3059037926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf0015ce40 msgr2=0x7fbf000d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:18.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.054+0000 7fbf04be0700 1 --2- 172.21.15.46:0/3059037926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf0015ce40 0x7fbf000d0e50 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fbf00089b70 tx=0x7fbef0009250 comp rx=0 tx=0).stop 2024-01-27T01:07:18.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.054+0000 7fbf04be0700 1 -- 172.21.15.46:0/3059037926 shutdown_connections 2024-01-27T01:07:18.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.054+0000 7fbf04be0700 1 --2- 172.21.15.46:0/3059037926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf0015ce40 0x7fbf000d0e50 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:18.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.054+0000 7fbf04be0700 1 --2- 172.21.15.46:0/3059037926 >> v2:172.21.15.106:3300/0 conn(0x7fbf000d1390 0x7fbf000cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:18.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.054+0000 7fbf04be0700 1 --2- 172.21.15.46:0/3059037926 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf0015d210 0x7fbf000cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:18.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.054+0000 7fbf04be0700 1 -- 172.21.15.46:0/3059037926 >> 172.21.15.46:0/3059037926 conn(0x7fbf0006f840 msgr2=0x7fbf000d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:18.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.054+0000 7fbf04be0700 1 -- 172.21.15.46:0/3059037926 shutdown_connections 2024-01-27T01:07:18.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.054+0000 7fbf04be0700 1 -- 172.21.15.46:0/3059037926 wait complete. 2024-01-27T01:07:18.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.055+0000 7fbf04be0700 1 Processor -- start 2024-01-27T01:07:18.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.055+0000 7fbf04be0700 1 -- start start 2024-01-27T01:07:18.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbf04be0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf000d1390 0x7fbf0015a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:18.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbf04be0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf0015ce40 0x7fbf0009a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:18.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbf04be0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf0015d210 0x7fbf0009c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:18.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbf04be0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0009b890 con 0x7fbf0015ce40 2024-01-27T01:07:18.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbf04be0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0009ac80 con 0x7fbf0015d210 2024-01-27T01:07:18.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbf04be0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0009af60 con 0x7fbf000d1390 2024-01-27T01:07:18.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbefdd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf0015ce40 0x7fbf0009a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:18.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbefdd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf0015ce40 0x7fbf0009a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50768/0 (socket says 172.21.15.46:50768) 2024-01-27T01:07:18.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbefdd9b700 1 -- 172.21.15.46:0/795422938 learned_addr learned my addr 172.21.15.46:0/795422938 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:18.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbefed9d700 1 --2- 172.21.15.46:0/795422938 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf0015d210 0x7fbf0009c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:18.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbefe59c700 1 --2- 172.21.15.46:0/795422938 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf000d1390 0x7fbf0015a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:18.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbefdd9b700 1 -- 172.21.15.46:0/795422938 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf000d1390 msgr2=0x7fbf0015a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:18.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbefdd9b700 1 --2- 172.21.15.46:0/795422938 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf000d1390 0x7fbf0015a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:18.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbefdd9b700 1 -- 172.21.15.46:0/795422938 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf0015d210 msgr2=0x7fbf0009c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:18.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbefdd9b700 1 --2- 172.21.15.46:0/795422938 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf0015d210 0x7fbf0009c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:18.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbefdd9b700 1 -- 172.21.15.46:0/795422938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef00094f0 con 0x7fbf0015ce40 2024-01-27T01:07:18.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbefe59c700 1 --2- 172.21.15.46:0/795422938 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf000d1390 0x7fbf0015a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:18.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.056+0000 7fbefdd9b700 1 --2- 172.21.15.46:0/795422938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf0015ce40 0x7fbf0009a2a0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fbee800e900 tx=0x7fbee80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:18.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.057+0000 7fbef67fc700 1 -- 172.21.15.46:0/795422938 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee8013680 con 0x7fbf0015ce40 2024-01-27T01:07:18.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.057+0000 7fbef67fc700 1 -- 172.21.15.46:0/795422938 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee80137e0 con 0x7fbf0015ce40 2024-01-27T01:07:18.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.057+0000 7fbf04be0700 1 -- 172.21.15.46:0/795422938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf0009b240 con 0x7fbf0015ce40 2024-01-27T01:07:18.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.057+0000 7fbef67fc700 1 -- 172.21.15.46:0/795422938 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee8013ba0 con 0x7fbf0015ce40 2024-01-27T01:07:18.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.057+0000 7fbf04be0700 1 -- 172.21.15.46:0/795422938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf0015ac70 con 0x7fbf0015ce40 2024-01-27T01:07:18.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.058+0000 7fbf04be0700 1 -- 172.21.15.46:0/795422938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf000ad470 con 0x7fbf0015ce40 2024-01-27T01:07:18.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.059+0000 7fbef67fc700 1 -- 172.21.15.46:0/795422938 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fbee800c460 con 0x7fbf0015ce40 2024-01-27T01:07:18.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.059+0000 7fbef67fc700 1 --2- 172.21.15.46:0/795422938 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbee0067170 0x7fbee0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:18.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.059+0000 7fbef67fc700 1 -- 172.21.15.46:0/795422938 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6902+0+0 (secure 0 0 0) 0x7fbee8084bf0 con 0x7fbf0015ce40 2024-01-27T01:07:18.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.059+0000 7fbefe59c700 1 --2- 172.21.15.46:0/795422938 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbee0067170 0x7fbee0069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:18.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.060+0000 7fbefe59c700 1 --2- 172.21.15.46:0/795422938 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbee0067170 0x7fbee0069620 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbf00089b70 tx=0x7fbef0017000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:18.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.064+0000 7fbef67fc700 1 -- 172.21.15.46:0/795422938 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbee8051720 con 0x7fbf0015ce40 2024-01-27T01:07:18.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.266+0000 7fbf04be0700 1 -- 172.21.15.46:0/795422938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbf000cf920 con 0x7fbf0015ce40 2024-01-27T01:07:18.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.894+0000 7fbef67fc700 1 -- 172.21.15.46:0/795422938 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v235) v1 ==== 178+0+0 (secure 0 0 0) 0x7fbee80170d0 con 0x7fbf0015ce40 2024-01-27T01:07:18.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.899+0000 7fbf04be0700 1 -- 172.21.15.46:0/795422938 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbee0067170 msgr2=0x7fbee0069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:18.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.899+0000 7fbf04be0700 1 --2- 172.21.15.46:0/795422938 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbee0067170 0x7fbee0069620 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbf00089b70 tx=0x7fbef0017000 comp rx=0 tx=0).stop 2024-01-27T01:07:18.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.899+0000 7fbf04be0700 1 -- 172.21.15.46:0/795422938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf0015ce40 msgr2=0x7fbf0009a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:18.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.899+0000 7fbf04be0700 1 --2- 172.21.15.46:0/795422938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf0015ce40 0x7fbf0009a2a0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fbee800e900 tx=0x7fbee80091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:18.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.899+0000 7fbf04be0700 1 -- 172.21.15.46:0/795422938 shutdown_connections 2024-01-27T01:07:18.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.899+0000 7fbf04be0700 1 --2- 172.21.15.46:0/795422938 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbee0067170 0x7fbee0069620 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:18.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.899+0000 7fbf04be0700 1 --2- 172.21.15.46:0/795422938 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf0015d210 0x7fbf0009c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:18.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.899+0000 7fbf04be0700 1 --2- 172.21.15.46:0/795422938 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf000d1390 0x7fbf0015a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:18.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.899+0000 7fbf04be0700 1 --2- 172.21.15.46:0/795422938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf0015ce40 0x7fbf0009a2a0 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:18.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.899+0000 7fbf04be0700 1 -- 172.21.15.46:0/795422938 >> 172.21.15.46:0/795422938 conn(0x7fbf0006f840 msgr2=0x7fbf000b5910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:18.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.899+0000 7fbf04be0700 1 -- 172.21.15.46:0/795422938 shutdown_connections 2024-01-27T01:07:18.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:18.900+0000 7fbf04be0700 1 -- 172.21.15.46:0/795422938 wait complete. 2024-01-27T01:07:18.901 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'backup_fs_data' removed 2024-01-27T01:07:18.917 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-27T01:07:19.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.066+0000 7fdcda71e700 1 Processor -- start 2024-01-27T01:07:19.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.066+0000 7fdcda71e700 1 -- start start 2024-01-27T01:07:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.067+0000 7fdcda71e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdcd4068b00 0x7fdcd4068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.067+0000 7fdcda71e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd406add0 0x7fdcd406b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.067+0000 7fdcda71e700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd40a9a80 con 0x7fdcd406a660 2024-01-27T01:07:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.067+0000 7fdcda71e700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd406b7b0 con 0x7fdcd4068b00 2024-01-27T01:07:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.067+0000 7fdcda71e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd406aab0 con 0x7fdcd406add0 2024-01-27T01:07:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.067+0000 7fdcd3fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdcd4068b00 0x7fdcd4068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.067+0000 7fdcd37fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdcd406a660 0x7fdcd4159ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53036/0 (socket says 172.21.15.46:53036) 2024-01-27T01:07:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.067+0000 7fdcd37fe700 1 -- 172.21.15.46:0/1637328549 learned_addr learned my addr 172.21.15.46:0/1637328549 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.067+0000 7fdcd8cbb700 1 --2- 172.21.15.46:0/1637328549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd406add0 0x7fdcd406b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.067+0000 7fdcd3fff700 1 -- 172.21.15.46:0/1637328549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd406add0 msgr2=0x7fdcd406b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.068+0000 7fdcd3fff700 1 --2- 172.21.15.46:0/1637328549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd406add0 0x7fdcd406b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.068+0000 7fdcd3fff700 1 -- 172.21.15.46:0/1637328549 >> v1:172.21.15.46:6789/0 conn(0x7fdcd406a660 legacy=0x7fdcd4159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.068+0000 7fdcd3fff700 1 -- 172.21.15.46:0/1637328549 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd415a680 con 0x7fdcd4068b00 2024-01-27T01:07:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.068+0000 7fdcd2ffd700 1 -- 172.21.15.46:0/1637328549 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821387135 0 0) 0x7fdcd40a9a80 con 0x7fdcd406a660 2024-01-27T01:07:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.068+0000 7fdcd3fff700 1 --2- 172.21.15.46:0/1637328549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd4068b00 0x7fdcd4068ed0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40098d0 tx=0x7fdcc4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=964cb398b2159af1 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.068+0000 7fdcd2ffd700 1 -- 172.21.15.46:0/1637328549 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4009e70 con 0x7fdcd4068b00 2024-01-27T01:07:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.068+0000 7fdcd2ffd700 1 -- 172.21.15.46:0/1637328549 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc4005c10 con 0x7fdcd4068b00 2024-01-27T01:07:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.068+0000 7fdcd2ffd700 1 -- 172.21.15.46:0/1637328549 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc400b650 con 0x7fdcd4068b00 2024-01-27T01:07:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.068+0000 7fdcda71e700 1 -- 172.21.15.46:0/1637328549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd4068b00 msgr2=0x7fdcd4068ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.068+0000 7fdcda71e700 1 --2- 172.21.15.46:0/1637328549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd4068b00 0x7fdcd4068ed0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40098d0 tx=0x7fdcc4009120 comp rx=0 tx=0).stop 2024-01-27T01:07:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.069+0000 7fdcda71e700 1 -- 172.21.15.46:0/1637328549 shutdown_connections 2024-01-27T01:07:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.069+0000 7fdcda71e700 1 --2- 172.21.15.46:0/1637328549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd406add0 0x7fdcd406b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.069+0000 7fdcda71e700 1 --2- 172.21.15.46:0/1637328549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd4068b00 0x7fdcd4068ed0 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.069+0000 7fdcda71e700 1 -- 172.21.15.46:0/1637328549 >> 172.21.15.46:0/1637328549 conn(0x7fdcd40ca100 msgr2=0x7fdcd406e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:19.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.069+0000 7fdcda71e700 1 -- 172.21.15.46:0/1637328549 shutdown_connections 2024-01-27T01:07:19.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.069+0000 7fdcda71e700 1 -- 172.21.15.46:0/1637328549 wait complete. 2024-01-27T01:07:19.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.070+0000 7fdcda71e700 1 Processor -- start 2024-01-27T01:07:19.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.070+0000 7fdcda71e700 1 -- start start 2024-01-27T01:07:19.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.071+0000 7fdcda71e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcd4068b00 0x7fdcd40adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.071+0000 7fdcda71e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd406a660 0x7fdcd4159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.071+0000 7fdcda71e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd406add0 0x7fdcd40ab940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.071+0000 7fdcda71e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd416d150 con 0x7fdcd406a660 2024-01-27T01:07:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.071+0000 7fdcda71e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd416d330 con 0x7fdcd4068b00 2024-01-27T01:07:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.071+0000 7fdcda71e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd416d610 con 0x7fdcd406add0 2024-01-27T01:07:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.071+0000 7fdcd37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd406a660 0x7fdcd4159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.071+0000 7fdcd37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd406a660 0x7fdcd4159760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50776/0 (socket says 172.21.15.46:50776) 2024-01-27T01:07:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.071+0000 7fdcd37fe700 1 -- 172.21.15.46:0/2636673424 learned_addr learned my addr 172.21.15.46:0/2636673424 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.071+0000 7fdcd8cbb700 1 --2- 172.21.15.46:0/2636673424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd406add0 0x7fdcd40ab940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.071+0000 7fdcd3fff700 1 --2- 172.21.15.46:0/2636673424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcd4068b00 0x7fdcd40adcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.071+0000 7fdcd37fe700 1 -- 172.21.15.46:0/2636673424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd406add0 msgr2=0x7fdcd40ab940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.071+0000 7fdcd37fe700 1 --2- 172.21.15.46:0/2636673424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd406add0 0x7fdcd40ab940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.071+0000 7fdcd37fe700 1 -- 172.21.15.46:0/2636673424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcd4068b00 msgr2=0x7fdcd40adcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.071+0000 7fdcd37fe700 1 --2- 172.21.15.46:0/2636673424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcd4068b00 0x7fdcd40adcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.071+0000 7fdcd37fe700 1 -- 172.21.15.46:0/2636673424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc40093c0 con 0x7fdcd406a660 2024-01-27T01:07:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.071+0000 7fdcd3fff700 1 --2- 172.21.15.46:0/2636673424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcd4068b00 0x7fdcd40adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.072+0000 7fdcd37fe700 1 --2- 172.21.15.46:0/2636673424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd406a660 0x7fdcd4159760 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc00c810 tx=0x7fdcbc007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.072+0000 7fdccbfff700 1 -- 172.21.15.46:0/2636673424 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc005c00 con 0x7fdcd406a660 2024-01-27T01:07:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.072+0000 7fdccbfff700 1 -- 172.21.15.46:0/2636673424 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcbc005d60 con 0x7fdcd406a660 2024-01-27T01:07:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.072+0000 7fdccbfff700 1 -- 172.21.15.46:0/2636673424 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc0072e0 con 0x7fdcd406a660 2024-01-27T01:07:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.072+0000 7fdcda71e700 1 -- 172.21.15.46:0/2636673424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd416d8f0 con 0x7fdcd406a660 2024-01-27T01:07:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.072+0000 7fdcda71e700 1 -- 172.21.15.46:0/2636673424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd416df30 con 0x7fdcd406a660 2024-01-27T01:07:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.073+0000 7fdcda71e700 1 -- 172.21.15.46:0/2636673424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd41578a0 con 0x7fdcd406a660 2024-01-27T01:07:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.073+0000 7fdccbfff700 1 -- 172.21.15.46:0/2636673424 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fdcbc007690 con 0x7fdcd406a660 2024-01-27T01:07:19.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.074+0000 7fdccbfff700 1 --2- 172.21.15.46:0/2636673424 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdcb4067170 0x7fdcb4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.074+0000 7fdccbfff700 1 -- 172.21.15.46:0/2636673424 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6482+0+0 (secure 0 0 0) 0x7fdcbc084060 con 0x7fdcd406a660 2024-01-27T01:07:19.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.074+0000 7fdcd3fff700 1 --2- 172.21.15.46:0/2636673424 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdcb4067170 0x7fdcb4069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.075+0000 7fdcd3fff700 1 --2- 172.21.15.46:0/2636673424 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdcb4067170 0x7fdcb4069620 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4009160 tx=0x7fdcc4016000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:19.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.078+0000 7fdccbfff700 1 -- 172.21.15.46:0/2636673424 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdcbc04b5a0 con 0x7fdcd406a660 2024-01-27T01:07:19.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.284+0000 7fdcda71e700 1 -- 172.21.15.46:0/2636673424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fdcd40ae290 con 0x7fdcd406a660 2024-01-27T01:07:19.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.288+0000 7fdccbfff700 1 -- 172.21.15.46:0/2636673424 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v79) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdcbc0170a0 con 0x7fdcd406a660 2024-01-27T01:07:19.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.290+0000 7fdcda71e700 1 -- 172.21.15.46:0/2636673424 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdcb4067170 msgr2=0x7fdcb4069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:19.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.290+0000 7fdcda71e700 1 --2- 172.21.15.46:0/2636673424 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdcb4067170 0x7fdcb4069620 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4009160 tx=0x7fdcc4016000 comp rx=0 tx=0).stop 2024-01-27T01:07:19.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.290+0000 7fdcda71e700 1 -- 172.21.15.46:0/2636673424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd406a660 msgr2=0x7fdcd4159760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:19.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.290+0000 7fdcda71e700 1 --2- 172.21.15.46:0/2636673424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd406a660 0x7fdcd4159760 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc00c810 tx=0x7fdcbc007db0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.291+0000 7fdcda71e700 1 -- 172.21.15.46:0/2636673424 shutdown_connections 2024-01-27T01:07:19.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.291+0000 7fdcda71e700 1 --2- 172.21.15.46:0/2636673424 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdcb4067170 0x7fdcb4069620 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.291+0000 7fdcda71e700 1 --2- 172.21.15.46:0/2636673424 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcd406add0 0x7fdcd40ab940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.291+0000 7fdcda71e700 1 --2- 172.21.15.46:0/2636673424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcd4068b00 0x7fdcd40adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.291+0000 7fdcda71e700 1 --2- 172.21.15.46:0/2636673424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd406a660 0x7fdcd4159760 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.291+0000 7fdcda71e700 1 -- 172.21.15.46:0/2636673424 >> 172.21.15.46:0/2636673424 conn(0x7fdcd40ca100 msgr2=0x7fdcd40aedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:19.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.291+0000 7fdcda71e700 1 -- 172.21.15.46:0/2636673424 shutdown_connections 2024-01-27T01:07:19.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.291+0000 7fdcda71e700 1 -- 172.21.15.46:0/2636673424 wait complete. 2024-01-27T01:07:19.305 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-27T01:07:19.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.438+0000 7f1d10d91700 1 Processor -- start 2024-01-27T01:07:19.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.438+0000 7f1d10d91700 1 -- start start 2024-01-27T01:07:19.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.439+0000 7f1d10d91700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d0c157490 0x7f1d0c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.439+0000 7f1d10d91700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1d0c06ddb0 0x7f1d0c0cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.439+0000 7f1d10d91700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0c0cea80 con 0x7f1d0c157860 2024-01-27T01:07:19.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.439+0000 7f1d10d91700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d0c0cdc60 con 0x7f1d0c06ddb0 2024-01-27T01:07:19.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.439+0000 7f1d10d91700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c0b2ae0 con 0x7f1d0c157490 2024-01-27T01:07:19.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.440+0000 7f1d0ad9d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1d0c06ddb0 0x7f1d0c0cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.440+0000 7f1d0a59c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d0c157490 0x7f1d0c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.440+0000 7f1d09d9b700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f1d0c157860 0x7f1d0c06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:45924/0 (socket says 172.21.15.46:45924) 2024-01-27T01:07:19.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.440+0000 7f1d09d9b700 1 -- 172.21.15.46:0/204021285 learned_addr learned my addr 172.21.15.46:0/204021285 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:19.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.440+0000 7f1d0a59c700 1 -- 172.21.15.46:0/204021285 >> v2:172.21.15.106:3300/0 conn(0x7f1d0c06ddb0 msgr2=0x7f1d0c0cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:19.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.440+0000 7f1d0a59c700 1 --2- 172.21.15.46:0/204021285 >> v2:172.21.15.106:3300/0 conn(0x7f1d0c06ddb0 0x7f1d0c0cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.440+0000 7f1d0a59c700 1 -- 172.21.15.46:0/204021285 >> v1:172.21.15.106:6789/0 conn(0x7f1d0c157860 legacy=0x7f1d0c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:19.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.440+0000 7f1d0a59c700 1 -- 172.21.15.46:0/204021285 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0c0cde70 con 0x7f1d0c157490 2024-01-27T01:07:19.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.440+0000 7f1d0ad9d700 1 --2- 172.21.15.46:0/204021285 >> v2:172.21.15.106:3300/0 conn(0x7f1d0c06ddb0 0x7f1d0c0cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:19.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.440+0000 7f1d0959a700 1 -- 172.21.15.46:0/204021285 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070626829 0 0) 0x7f1d0c0cea80 con 0x7f1d0c157860 2024-01-27T01:07:19.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.440+0000 7f1d0a59c700 1 --2- 172.21.15.46:0/204021285 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d0c157490 0x7f1d0c0cc600 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1cf40098e0 tx=0x7f1cf40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e45677e8c05e5f0f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:19.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.440+0000 7f1d0959a700 1 -- 172.21.15.46:0/204021285 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf400b640 con 0x7f1d0c157490 2024-01-27T01:07:19.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.441+0000 7f1d0959a700 1 -- 172.21.15.46:0/204021285 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf400b7a0 con 0x7f1d0c157490 2024-01-27T01:07:19.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.441+0000 7f1d0959a700 1 -- 172.21.15.46:0/204021285 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf400bb40 con 0x7f1d0c157490 2024-01-27T01:07:19.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.441+0000 7f1d10d91700 1 -- 172.21.15.46:0/204021285 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d0c157490 msgr2=0x7f1d0c0cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:19.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.441+0000 7f1d10d91700 1 --2- 172.21.15.46:0/204021285 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d0c157490 0x7f1d0c0cc600 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1cf40098e0 tx=0x7f1cf40091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.441+0000 7f1d10d91700 1 -- 172.21.15.46:0/204021285 shutdown_connections 2024-01-27T01:07:19.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.441+0000 7f1d10d91700 1 --2- 172.21.15.46:0/204021285 >> v2:172.21.15.106:3300/0 conn(0x7f1d0c06ddb0 0x7f1d0c0cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.441+0000 7f1d10d91700 1 --2- 172.21.15.46:0/204021285 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d0c157490 0x7f1d0c0cc600 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.441+0000 7f1d10d91700 1 -- 172.21.15.46:0/204021285 >> 172.21.15.46:0/204021285 conn(0x7f1d0c067930 msgr2=0x7f1d0c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:19.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.441+0000 7f1d10d91700 1 -- 172.21.15.46:0/204021285 shutdown_connections 2024-01-27T01:07:19.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.441+0000 7f1d10d91700 1 -- 172.21.15.46:0/204021285 wait complete. 2024-01-27T01:07:19.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.442+0000 7f1d10d91700 1 Processor -- start 2024-01-27T01:07:19.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.442+0000 7f1d10d91700 1 -- start start 2024-01-27T01:07:19.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.443+0000 7f1d10d91700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d0c06ddb0 0x7f1d0c151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.443+0000 7f1d10d91700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d0c157490 0x7f1d0c06a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.443+0000 7f1d10d91700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d0c157860 0x7f1d0c0af130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.443+0000 7f1d10d91700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c152290 con 0x7f1d0c157860 2024-01-27T01:07:19.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.443+0000 7f1d10d91700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c16ab70 con 0x7f1d0c06ddb0 2024-01-27T01:07:19.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.443+0000 7f1d10d91700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c169ff0 con 0x7f1d0c157490 2024-01-27T01:07:19.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.443+0000 7f1d0ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d0c157860 0x7f1d0c0af130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.443+0000 7f1d0ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d0c157860 0x7f1d0c0af130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50784/0 (socket says 172.21.15.46:50784) 2024-01-27T01:07:19.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.443+0000 7f1d0ad9d700 1 -- 172.21.15.46:0/2828657667 learned_addr learned my addr 172.21.15.46:0/2828657667 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:19.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.444+0000 7f1d09d9b700 1 --2- 172.21.15.46:0/2828657667 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d0c157490 0x7f1d0c06a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.444+0000 7f1d0a59c700 1 --2- 172.21.15.46:0/2828657667 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d0c06ddb0 0x7f1d0c151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.444+0000 7f1d0ad9d700 1 -- 172.21.15.46:0/2828657667 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d0c157490 msgr2=0x7f1d0c06a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:19.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.444+0000 7f1d0ad9d700 1 --2- 172.21.15.46:0/2828657667 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d0c157490 0x7f1d0c06a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.444+0000 7f1d0ad9d700 1 -- 172.21.15.46:0/2828657667 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d0c06ddb0 msgr2=0x7f1d0c151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:19.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.444+0000 7f1d0ad9d700 1 --2- 172.21.15.46:0/2828657667 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d0c06ddb0 0x7f1d0c151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.444+0000 7f1d0ad9d700 1 -- 172.21.15.46:0/2828657667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d04009a20 con 0x7f1d0c157860 2024-01-27T01:07:19.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.444+0000 7f1d0ad9d700 1 --2- 172.21.15.46:0/2828657667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d0c157860 0x7f1d0c0af130 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f1d04009fc0 tx=0x7f1d0400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:19.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.444+0000 7f1d027fc700 1 -- 172.21.15.46:0/2828657667 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0400cbd0 con 0x7f1d0c157860 2024-01-27T01:07:19.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.444+0000 7f1d027fc700 1 -- 172.21.15.46:0/2828657667 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0400f040 con 0x7f1d0c157860 2024-01-27T01:07:19.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.444+0000 7f1d10d91700 1 -- 172.21.15.46:0/2828657667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf4009420 con 0x7f1d0c157860 2024-01-27T01:07:19.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.444+0000 7f1d027fc700 1 -- 172.21.15.46:0/2828657667 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04014480 con 0x7f1d0c157860 2024-01-27T01:07:19.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.444+0000 7f1d10d91700 1 -- 172.21.15.46:0/2828657667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d0c16a5b0 con 0x7f1d0c157860 2024-01-27T01:07:19.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.445+0000 7f1d10d91700 1 -- 172.21.15.46:0/2828657667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d0c04c710 con 0x7f1d0c157860 2024-01-27T01:07:19.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.446+0000 7f1d027fc700 1 -- 172.21.15.46:0/2828657667 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f1d04003020 con 0x7f1d0c157860 2024-01-27T01:07:19.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.446+0000 7f1d027fc700 1 --2- 172.21.15.46:0/2828657667 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1cec067170 0x7f1cec069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.446+0000 7f1d027fc700 1 -- 172.21.15.46:0/2828657667 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6482+0+0 (secure 0 0 0) 0x7f1d04085720 con 0x7f1d0c157860 2024-01-27T01:07:19.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.447+0000 7f1d0a59c700 1 --2- 172.21.15.46:0/2828657667 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1cec067170 0x7f1cec069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.447+0000 7f1d0a59c700 1 --2- 172.21.15.46:0/2828657667 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1cec067170 0x7f1cec069620 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1cf40093f0 tx=0x7f1cf4017000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:19.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.451+0000 7f1d027fc700 1 -- 172.21.15.46:0/2828657667 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1d04052170 con 0x7f1d0c157860 2024-01-27T01:07:19.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.656+0000 7f1d10d91700 1 -- 172.21.15.46:0/2828657667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f1d0c0b14d0 con 0x7f1d0c157860 2024-01-27T01:07:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.660+0000 7f1d027fc700 1 -- 172.21.15.46:0/2828657667 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v80) v1 ==== 120+0+0 (secure 0 0 0) 0x7f1d04055790 con 0x7f1d0c157860 2024-01-27T01:07:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.663+0000 7f1d10d91700 1 -- 172.21.15.46:0/2828657667 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1cec067170 msgr2=0x7f1cec069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.663+0000 7f1d10d91700 1 --2- 172.21.15.46:0/2828657667 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1cec067170 0x7f1cec069620 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1cf40093f0 tx=0x7f1cf4017000 comp rx=0 tx=0).stop 2024-01-27T01:07:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.663+0000 7f1d10d91700 1 -- 172.21.15.46:0/2828657667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d0c157860 msgr2=0x7f1d0c0af130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.663+0000 7f1d10d91700 1 --2- 172.21.15.46:0/2828657667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d0c157860 0x7f1d0c0af130 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f1d04009fc0 tx=0x7f1d0400b040 comp rx=0 tx=0).stop 2024-01-27T01:07:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.663+0000 7f1d10d91700 1 -- 172.21.15.46:0/2828657667 shutdown_connections 2024-01-27T01:07:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.663+0000 7f1d10d91700 1 --2- 172.21.15.46:0/2828657667 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1cec067170 0x7f1cec069620 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.663+0000 7f1d10d91700 1 --2- 172.21.15.46:0/2828657667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d0c157860 0x7f1d0c0af130 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.663+0000 7f1d10d91700 1 --2- 172.21.15.46:0/2828657667 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d0c06ddb0 0x7f1d0c151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.663+0000 7f1d10d91700 1 --2- 172.21.15.46:0/2828657667 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d0c157490 0x7f1d0c06a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.663+0000 7f1d10d91700 1 -- 172.21.15.46:0/2828657667 >> 172.21.15.46:0/2828657667 conn(0x7f1d0c067930 msgr2=0x7f1d0c154fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.663+0000 7f1d10d91700 1 -- 172.21.15.46:0/2828657667 shutdown_connections 2024-01-27T01:07:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.663+0000 7f1d10d91700 1 -- 172.21.15.46:0/2828657667 wait complete. 2024-01-27T01:07:19.676 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-27T01:07:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.810+0000 7fbc7a96d700 1 Processor -- start 2024-01-27T01:07:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.810+0000 7fbc7a96d700 1 -- start start 2024-01-27T01:07:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.811+0000 7fbc7a96d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbc74068940 0x7fbc74068d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.811+0000 7fbc7a96d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbc740c90c0 0x7fbc740af990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.811+0000 7fbc7a96d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc740c87c0 con 0x7fbc740c8c50 2024-01-27T01:07:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.811+0000 7fbc7a96d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc74069250 con 0x7fbc740c90c0 2024-01-27T01:07:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.811+0000 7fbc7a96d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc740c7bb0 con 0x7fbc74068940 2024-01-27T01:07:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.812+0000 7fbc78f0a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbc740c90c0 0x7fbc740af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.812+0000 7fbc78f0a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbc740c90c0 0x7fbc740af990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50794/0 (socket says 172.21.15.46:50794) 2024-01-27T01:07:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.812+0000 7fbc78f0a700 1 -- 172.21.15.46:0/4117793281 learned_addr learned my addr 172.21.15.46:0/4117793281 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.812+0000 7fbc73fff700 1 --2- 172.21.15.46:0/4117793281 >> v2:172.21.15.106:3300/0 conn(0x7fbc74068940 0x7fbc74068d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.812+0000 7fbc78f0a700 1 -- 172.21.15.46:0/4117793281 >> v2:172.21.15.106:3300/0 conn(0x7fbc74068940 msgr2=0x7fbc74068d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.812+0000 7fbc78f0a700 1 --2- 172.21.15.46:0/4117793281 >> v2:172.21.15.106:3300/0 conn(0x7fbc74068940 0x7fbc74068d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.812+0000 7fbc78f0a700 1 -- 172.21.15.46:0/4117793281 >> v1:172.21.15.106:6789/0 conn(0x7fbc740c8c50 legacy=0x7fbc740c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.812+0000 7fbc78f0a700 1 -- 172.21.15.46:0/4117793281 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc740c7e90 con 0x7fbc740c90c0 2024-01-27T01:07:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.812+0000 7fbc78f0a700 1 --2- 172.21.15.46:0/4117793281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc740c90c0 0x7fbc740af990 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7fbc7406dc20 tx=0x7fbc6c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ee2ce08dd18a2b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.812+0000 7fbc72ffd700 1 -- 172.21.15.46:0/4117793281 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c00e070 con 0x7fbc740c90c0 2024-01-27T01:07:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.812+0000 7fbc72ffd700 1 -- 172.21.15.46:0/4117793281 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6c00ba50 con 0x7fbc740c90c0 2024-01-27T01:07:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.812+0000 7fbc72ffd700 1 -- 172.21.15.46:0/4117793281 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c0092e0 con 0x7fbc740c90c0 2024-01-27T01:07:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.813+0000 7fbc7a96d700 1 -- 172.21.15.46:0/4117793281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc740c90c0 msgr2=0x7fbc740af990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.813+0000 7fbc7a96d700 1 --2- 172.21.15.46:0/4117793281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc740c90c0 0x7fbc740af990 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7fbc7406dc20 tx=0x7fbc6c009840 comp rx=0 tx=0).stop 2024-01-27T01:07:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.813+0000 7fbc7a96d700 1 -- 172.21.15.46:0/4117793281 shutdown_connections 2024-01-27T01:07:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.813+0000 7fbc7a96d700 1 --2- 172.21.15.46:0/4117793281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc740c90c0 0x7fbc740af990 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.813+0000 7fbc7a96d700 1 --2- 172.21.15.46:0/4117793281 >> v2:172.21.15.106:3300/0 conn(0x7fbc74068940 0x7fbc74068d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.813+0000 7fbc7a96d700 1 -- 172.21.15.46:0/4117793281 >> 172.21.15.46:0/4117793281 conn(0x7fbc740ca100 msgr2=0x7fbc7406e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.813+0000 7fbc7a96d700 1 -- 172.21.15.46:0/4117793281 shutdown_connections 2024-01-27T01:07:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.813+0000 7fbc7a96d700 1 -- 172.21.15.46:0/4117793281 wait complete. 2024-01-27T01:07:19.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.815+0000 7fbc7a96d700 1 Processor -- start 2024-01-27T01:07:19.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.815+0000 7fbc7a96d700 1 -- start start 2024-01-27T01:07:19.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.816+0000 7fbc7a96d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc74068940 0x7fbc740ac9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.816+0000 7fbc7a96d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc740c8c50 0x7fbc740a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.816+0000 7fbc7a96d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc740c90c0 0x7fbc740a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.816+0000 7fbc7a96d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc74168560 con 0x7fbc74068940 2024-01-27T01:07:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.816+0000 7fbc7a96d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc740acf20 con 0x7fbc740c8c50 2024-01-27T01:07:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.816+0000 7fbc7a96d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc740c5ad0 con 0x7fbc740c90c0 2024-01-27T01:07:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.816+0000 7fbc73fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc74068940 0x7fbc740ac9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.816+0000 7fbc73fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc74068940 0x7fbc740ac9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50808/0 (socket says 172.21.15.46:50808) 2024-01-27T01:07:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.816+0000 7fbc73fff700 1 -- 172.21.15.46:0/1193425448 learned_addr learned my addr 172.21.15.46:0/1193425448 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.816+0000 7fbc78f0a700 1 --2- 172.21.15.46:0/1193425448 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc740c90c0 0x7fbc740a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.816+0000 7fbc737fe700 1 --2- 172.21.15.46:0/1193425448 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc740c8c50 0x7fbc740a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.816+0000 7fbc73fff700 1 -- 172.21.15.46:0/1193425448 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc740c90c0 msgr2=0x7fbc740a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.816+0000 7fbc73fff700 1 --2- 172.21.15.46:0/1193425448 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc740c90c0 0x7fbc740a5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.816+0000 7fbc73fff700 1 -- 172.21.15.46:0/1193425448 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc740c8c50 msgr2=0x7fbc740a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.816+0000 7fbc73fff700 1 --2- 172.21.15.46:0/1193425448 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc740c8c50 0x7fbc740a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.816+0000 7fbc73fff700 1 -- 172.21.15.46:0/1193425448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6c009ae0 con 0x7fbc74068940 2024-01-27T01:07:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.817+0000 7fbc73fff700 1 --2- 172.21.15.46:0/1193425448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc74068940 0x7fbc740ac9e0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400d840 tx=0x7fbc64008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.817+0000 7fbc6bfff700 1 -- 172.21.15.46:0/1193425448 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc64014620 con 0x7fbc74068940 2024-01-27T01:07:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.817+0000 7fbc6bfff700 1 -- 172.21.15.46:0/1193425448 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc64014780 con 0x7fbc74068940 2024-01-27T01:07:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.817+0000 7fbc7a96d700 1 -- 172.21.15.46:0/1193425448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc740c5db0 con 0x7fbc74068940 2024-01-27T01:07:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.817+0000 7fbc6bfff700 1 -- 172.21.15.46:0/1193425448 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc64014b40 con 0x7fbc74068940 2024-01-27T01:07:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.817+0000 7fbc7a96d700 1 -- 172.21.15.46:0/1193425448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc740c6370 con 0x7fbc74068940 2024-01-27T01:07:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.818+0000 7fbc7a96d700 1 -- 172.21.15.46:0/1193425448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc7404c710 con 0x7fbc74068940 2024-01-27T01:07:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.818+0000 7fbc6bfff700 1 -- 172.21.15.46:0/1193425448 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fbc6401d400 con 0x7fbc74068940 2024-01-27T01:07:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.819+0000 7fbc6bfff700 1 --2- 172.21.15.46:0/1193425448 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbc54067170 0x7fbc54069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.819+0000 7fbc6bfff700 1 -- 172.21.15.46:0/1193425448 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6482+0+0 (secure 0 0 0) 0x7fbc640853d0 con 0x7fbc74068940 2024-01-27T01:07:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.819+0000 7fbc737fe700 1 --2- 172.21.15.46:0/1193425448 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbc54067170 0x7fbc54069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.820+0000 7fbc737fe700 1 --2- 172.21.15.46:0/1193425448 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbc54067170 0x7fbc54069620 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbc74164f80 tx=0x7fbc5c000bc0 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:19.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:19.823+0000 7fbc6bfff700 1 -- 172.21.15.46:0/1193425448 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbc640520a0 con 0x7fbc74068940 2024-01-27T01:07:20.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.032+0000 7fbc7a96d700 1 -- 172.21.15.46:0/1193425448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fbc74168840 con 0x7fbc74068940 2024-01-27T01:07:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.036+0000 7fbc6bfff700 1 -- 172.21.15.46:0/1193425448 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v81) v1 ==== 115+0+0 (secure 0 0 0) 0x7fbc640556c0 con 0x7fbc74068940 2024-01-27T01:07:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.038+0000 7fbc7a96d700 1 -- 172.21.15.46:0/1193425448 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbc54067170 msgr2=0x7fbc54069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.038+0000 7fbc7a96d700 1 --2- 172.21.15.46:0/1193425448 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbc54067170 0x7fbc54069620 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbc74164f80 tx=0x7fbc5c000bc0 comp rx=0 tx=0).stop 2024-01-27T01:07:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.038+0000 7fbc7a96d700 1 -- 172.21.15.46:0/1193425448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc74068940 msgr2=0x7fbc740ac9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.038+0000 7fbc7a96d700 1 --2- 172.21.15.46:0/1193425448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc74068940 0x7fbc740ac9e0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400d840 tx=0x7fbc64008040 comp rx=0 tx=0).stop 2024-01-27T01:07:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.038+0000 7fbc7a96d700 1 -- 172.21.15.46:0/1193425448 shutdown_connections 2024-01-27T01:07:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.038+0000 7fbc7a96d700 1 --2- 172.21.15.46:0/1193425448 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbc54067170 0x7fbc54069620 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.038+0000 7fbc7a96d700 1 --2- 172.21.15.46:0/1193425448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc74068940 0x7fbc740ac9e0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.038+0000 7fbc7a96d700 1 --2- 172.21.15.46:0/1193425448 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc740c90c0 0x7fbc740a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.038+0000 7fbc7a96d700 1 --2- 172.21.15.46:0/1193425448 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc740c8c50 0x7fbc740a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.038+0000 7fbc7a96d700 1 -- 172.21.15.46:0/1193425448 >> 172.21.15.46:0/1193425448 conn(0x7fbc740ca100 msgr2=0x7fbc740b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.038+0000 7fbc7a96d700 1 -- 172.21.15.46:0/1193425448 shutdown_connections 2024-01-27T01:07:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.038+0000 7fbc7a96d700 1 -- 172.21.15.46:0/1193425448 wait complete. 2024-01-27T01:07:20.053 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-01-27T01:07:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.185+0000 7f12bdc20700 1 Processor -- start 2024-01-27T01:07:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.185+0000 7f12bdc20700 1 -- start start 2024-01-27T01:07:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.186+0000 7f12bdc20700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f12b80cf460 0x7f12b80ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.186+0000 7f12bdc20700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b815a0b0 con 0x7f12b80c6520 2024-01-27T01:07:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.186+0000 7f12bdc20700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b80ae4d0 con 0x7f12b80b2b40 2024-01-27T01:07:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.186+0000 7f12bdc20700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b80af070 con 0x7f12b80cf460 2024-01-27T01:07:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.186+0000 7f12b6ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f12b80c6520 0x7f12b80ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53046/0 (socket says 172.21.15.46:53046) 2024-01-27T01:07:20.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.186+0000 7f12b6ffd700 1 -- 172.21.15.46:0/4182914242 learned_addr learned my addr 172.21.15.46:0/4182914242 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.186+0000 7f12b7fff700 1 --2- 172.21.15.46:0/4182914242 >> v2:172.21.15.106:3300/0 conn(0x7f12b80cf460 0x7f12b80ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.186+0000 7f12b67fc700 1 -- 172.21.15.46:0/4182914242 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424626020 0 0) 0x7f12b815a0b0 con 0x7f12b80c6520 2024-01-27T01:07:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.186+0000 7f12b67fc700 1 -- 172.21.15.46:0/4182914242 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a4003570 con 0x7f12b80c6520 2024-01-27T01:07:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.187+0000 7f12b67fc700 1 -- 172.21.15.46:0/4182914242 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414630137 0 0) 0x7f12b80ae4d0 con 0x7f12b80b2b40 2024-01-27T01:07:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.187+0000 7f12b67fc700 1 -- 172.21.15.46:0/4182914242 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b815a0b0 con 0x7f12b80b2b40 2024-01-27T01:07:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.187+0000 7f12b7fff700 1 -- 172.21.15.46:0/4182914242 >> v1:172.21.15.106:6789/0 conn(0x7f12b80b2b40 legacy=0x7f12b80ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.187+0000 7f12b7fff700 1 -- 172.21.15.46:0/4182914242 >> v1:172.21.15.46:6789/0 conn(0x7f12b80c6520 legacy=0x7f12b80ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.187+0000 7f12b7fff700 1 -- 172.21.15.46:0/4182914242 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b80ad8c0 con 0x7f12b80cf460 2024-01-27T01:07:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.187+0000 7f12b7fff700 1 --2- 172.21.15.46:0/4182914242 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12b80cf460 0x7f12b80ad1c0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009e60 tx=0x7f12b0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9ec53484cf14a6c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.187+0000 7f12b67fc700 1 -- 172.21.15.46:0/4182914242 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b000e070 con 0x7f12b80cf460 2024-01-27T01:07:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.187+0000 7f12b67fc700 1 -- 172.21.15.46:0/4182914242 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b000bab0 con 0x7f12b80cf460 2024-01-27T01:07:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.187+0000 7f12b67fc700 1 -- 172.21.15.46:0/4182914242 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b000be70 con 0x7f12b80cf460 2024-01-27T01:07:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.188+0000 7f12bdc20700 1 -- 172.21.15.46:0/4182914242 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12b80cf460 msgr2=0x7f12b80ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.188+0000 7f12bdc20700 1 --2- 172.21.15.46:0/4182914242 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12b80cf460 0x7f12b80ad1c0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009e60 tx=0x7f12b0009780 comp rx=0 tx=0).stop 2024-01-27T01:07:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.188+0000 7f12bdc20700 1 -- 172.21.15.46:0/4182914242 shutdown_connections 2024-01-27T01:07:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.188+0000 7f12bdc20700 1 --2- 172.21.15.46:0/4182914242 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12b80cf460 0x7f12b80ad1c0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.188+0000 7f12bdc20700 1 -- 172.21.15.46:0/4182914242 >> 172.21.15.46:0/4182914242 conn(0x7f12b8099a30 msgr2=0x7f12b806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.188+0000 7f12bdc20700 1 -- 172.21.15.46:0/4182914242 shutdown_connections 2024-01-27T01:07:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.188+0000 7f12bdc20700 1 -- 172.21.15.46:0/4182914242 wait complete. 2024-01-27T01:07:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.189+0000 7f12bdc20700 1 Processor -- start 2024-01-27T01:07:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.189+0000 7f12bdc20700 1 -- start start 2024-01-27T01:07:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.190+0000 7f12bdc20700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12b80b2b40 0x7f12b8151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.190+0000 7f12bdc20700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12b80c6520 0x7f12b80b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.190+0000 7f12bdc20700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b80cf460 0x7f12b80b5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.190+0000 7f12bdc20700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b80b42f0 con 0x7f12b80cf460 2024-01-27T01:07:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.190+0000 7f12bdc20700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b80b5880 con 0x7f12b80c6520 2024-01-27T01:07:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.190+0000 7f12bdc20700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b80b4500 con 0x7f12b80b2b40 2024-01-27T01:07:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.190+0000 7f12b7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b80cf460 0x7f12b80b5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.190+0000 7f12b7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b80cf460 0x7f12b80b5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50812/0 (socket says 172.21.15.46:50812) 2024-01-27T01:07:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.190+0000 7f12b7fff700 1 -- 172.21.15.46:0/1253306963 learned_addr learned my addr 172.21.15.46:0/1253306963 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.190+0000 7f12b77fe700 1 --2- 172.21.15.46:0/1253306963 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12b80b2b40 0x7f12b8151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.190+0000 7f12b6ffd700 1 --2- 172.21.15.46:0/1253306963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12b80c6520 0x7f12b80b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.190+0000 7f12b7fff700 1 -- 172.21.15.46:0/1253306963 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12b80b2b40 msgr2=0x7f12b8151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.190+0000 7f12b7fff700 1 --2- 172.21.15.46:0/1253306963 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12b80b2b40 0x7f12b8151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.190+0000 7f12b7fff700 1 -- 172.21.15.46:0/1253306963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12b80c6520 msgr2=0x7f12b80b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.190+0000 7f12b7fff700 1 --2- 172.21.15.46:0/1253306963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12b80c6520 0x7f12b80b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.190+0000 7f12b7fff700 1 -- 172.21.15.46:0/1253306963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b0009a20 con 0x7f12b80cf460 2024-01-27T01:07:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.191+0000 7f12b7fff700 1 --2- 172.21.15.46:0/1253306963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b80cf460 0x7f12b80b5340 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f12b0013000 tx=0x7f12b0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.191+0000 7f12af7fe700 1 -- 172.21.15.46:0/1253306963 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b0026480 con 0x7f12b80cf460 2024-01-27T01:07:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.191+0000 7f12af7fe700 1 -- 172.21.15.46:0/1253306963 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b00265e0 con 0x7f12b80cf460 2024-01-27T01:07:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.191+0000 7f12bdc20700 1 -- 172.21.15.46:0/1253306963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b80b47e0 con 0x7f12b80cf460 2024-01-27T01:07:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.191+0000 7f12af7fe700 1 -- 172.21.15.46:0/1253306963 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b00269a0 con 0x7f12b80cf460 2024-01-27T01:07:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.191+0000 7f12bdc20700 1 -- 172.21.15.46:0/1253306963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b80b4da0 con 0x7f12b80cf460 2024-01-27T01:07:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.192+0000 7f12bdc20700 1 -- 172.21.15.46:0/1253306963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b804c710 con 0x7f12b80cf460 2024-01-27T01:07:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.192+0000 7f12af7fe700 1 -- 172.21.15.46:0/1253306963 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f12b001c400 con 0x7f12b80cf460 2024-01-27T01:07:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.193+0000 7f12af7fe700 1 --2- 172.21.15.46:0/1253306963 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1298067170 0x7f1298069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.193+0000 7f12af7fe700 1 -- 172.21.15.46:0/1253306963 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6482+0+0 (secure 0 0 0) 0x7f12b0086ed0 con 0x7f12b80cf460 2024-01-27T01:07:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.193+0000 7f12b77fe700 1 --2- 172.21.15.46:0/1253306963 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1298067170 0x7f1298069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:20.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.194+0000 7f12b77fe700 1 --2- 172.21.15.46:0/1253306963 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1298067170 0x7f1298069620 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f12a8003950 tx=0x7f12a8009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:20.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.198+0000 7f12af7fe700 1 -- 172.21.15.46:0/1253306963 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f12b0053a90 con 0x7f12b80cf460 2024-01-27T01:07:20.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.488+0000 7f12bdc20700 1 -- 172.21.15.46:0/1253306963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f12b80682c0 con 0x7f12b80cf460 2024-01-27T01:07:20.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.897+0000 7f12af7fe700 1 -- 172.21.15.46:0/1253306963 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3195) v1 ==== 186+0+0 (secure 0 0 0) 0x7f12b00570b0 con 0x7f12b80cf460 2024-01-27T01:07:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.900+0000 7f12bdc20700 1 -- 172.21.15.46:0/1253306963 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1298067170 msgr2=0x7f1298069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.900+0000 7f12bdc20700 1 --2- 172.21.15.46:0/1253306963 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1298067170 0x7f1298069620 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f12a8003950 tx=0x7f12a8009040 comp rx=0 tx=0).stop 2024-01-27T01:07:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.900+0000 7f12bdc20700 1 -- 172.21.15.46:0/1253306963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b80cf460 msgr2=0x7f12b80b5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.900+0000 7f12bdc20700 1 --2- 172.21.15.46:0/1253306963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b80cf460 0x7f12b80b5340 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f12b0013000 tx=0x7f12b0019000 comp rx=0 tx=0).stop 2024-01-27T01:07:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.900+0000 7f12bdc20700 1 -- 172.21.15.46:0/1253306963 shutdown_connections 2024-01-27T01:07:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.900+0000 7f12bdc20700 1 --2- 172.21.15.46:0/1253306963 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1298067170 0x7f1298069620 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.900+0000 7f12bdc20700 1 --2- 172.21.15.46:0/1253306963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b80cf460 0x7f12b80b5340 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.900+0000 7f12bdc20700 1 --2- 172.21.15.46:0/1253306963 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12b80b2b40 0x7f12b8151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.900+0000 7f12bdc20700 1 --2- 172.21.15.46:0/1253306963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12b80c6520 0x7f12b80b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:20.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.900+0000 7f12bdc20700 1 -- 172.21.15.46:0/1253306963 >> 172.21.15.46:0/1253306963 conn(0x7f12b8099a30 msgr2=0x7f12b80c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:20.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.900+0000 7f12bdc20700 1 -- 172.21.15.46:0/1253306963 shutdown_connections 2024-01-27T01:07:20.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:20.900+0000 7f12bdc20700 1 -- 172.21.15.46:0/1253306963 wait complete. 2024-01-27T01:07:20.916 INFO:tasks.cephfs_test_runner:Test command succes with --fs argument for non-default fs ... ok 2024-01-27T01:07:20.916 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command fails without --fs argument in presence of multiple fs 2024-01-27T01:07:20.917 INFO:tasks.cephfs_test_runner:test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-01-27T01:07:20.917 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-01-27T01:07:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.053+0000 7fe15adff700 1 Processor -- start 2024-01-27T01:07:21.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.053+0000 7fe15adff700 1 -- start start 2024-01-27T01:07:21.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.054+0000 7fe15adff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe15406abf0 0x7fe154159980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:21.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.054+0000 7fe15adff700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe154069100 con 0x7fe1541595b0 2024-01-27T01:07:21.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.054+0000 7fe15adff700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1540af5c0 con 0x7fe1540af190 2024-01-27T01:07:21.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.054+0000 7fe15adff700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe15406bc90 con 0x7fe15406abf0 2024-01-27T01:07:21.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.054+0000 7fe153fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe15406abf0 0x7fe154159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:21.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.054+0000 7fe158b9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe1541595b0 0x7fe15406a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47280/0 (socket says 172.21.15.46:47280) 2024-01-27T01:07:21.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.054+0000 7fe158b9b700 1 -- 172.21.15.46:0/1731339862 learned_addr learned my addr 172.21.15.46:0/1731339862 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:21.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.054+0000 7fe153fff700 1 -- 172.21.15.46:0/1731339862 >> v1:172.21.15.106:6789/0 conn(0x7fe1540af190 legacy=0x7fe1540a9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:21.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.054+0000 7fe153fff700 1 -- 172.21.15.46:0/1731339862 >> v1:172.21.15.46:6789/0 conn(0x7fe1541595b0 legacy=0x7fe15406a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:21.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.055+0000 7fe1537fe700 1 -- 172.21.15.46:0/1731339862 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974202605 0 0) 0x7fe154069100 con 0x7fe1541595b0 2024-01-27T01:07:21.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.055+0000 7fe153fff700 1 -- 172.21.15.46:0/1731339862 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15415def0 con 0x7fe15406abf0 2024-01-27T01:07:21.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.055+0000 7fe153fff700 1 --2- 172.21.15.46:0/1731339862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe15406abf0 0x7fe154159980 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fe1400098e0 tx=0x7fe1400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cdc7292665bde2b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:21.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.055+0000 7fe1537fe700 1 -- 172.21.15.46:0/1731339862 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140009e80 con 0x7fe15406abf0 2024-01-27T01:07:21.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.055+0000 7fe1537fe700 1 -- 172.21.15.46:0/1731339862 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14000b860 con 0x7fe15406abf0 2024-01-27T01:07:21.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.055+0000 7fe1537fe700 1 -- 172.21.15.46:0/1731339862 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14000bbe0 con 0x7fe15406abf0 2024-01-27T01:07:21.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.055+0000 7fe15adff700 1 -- 172.21.15.46:0/1731339862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe15406abf0 msgr2=0x7fe154159980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:21.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.055+0000 7fe15adff700 1 --2- 172.21.15.46:0/1731339862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe15406abf0 0x7fe154159980 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fe1400098e0 tx=0x7fe1400091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:21.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.056+0000 7fe15adff700 1 -- 172.21.15.46:0/1731339862 shutdown_connections 2024-01-27T01:07:21.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.056+0000 7fe15adff700 1 --2- 172.21.15.46:0/1731339862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe15406abf0 0x7fe154159980 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:21.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.056+0000 7fe15adff700 1 -- 172.21.15.46:0/1731339862 >> 172.21.15.46:0/1731339862 conn(0x7fe15406e2c0 msgr2=0x7fe1540ae620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:21.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.056+0000 7fe15adff700 1 -- 172.21.15.46:0/1731339862 shutdown_connections 2024-01-27T01:07:21.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.056+0000 7fe15adff700 1 -- 172.21.15.46:0/1731339862 wait complete. 2024-01-27T01:07:21.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.057+0000 7fe15adff700 1 Processor -- start 2024-01-27T01:07:21.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.057+0000 7fe15adff700 1 -- start start 2024-01-27T01:07:21.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe15adff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe15406abf0 0x7fe1540c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:21.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe15adff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1540af190 0x7fe15415b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:21.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe15adff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1541595b0 0x7fe1540c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:21.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe15adff700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe154169c40 con 0x7fe1540af190 2024-01-27T01:07:21.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe15adff700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15415b6c0 con 0x7fe1541595b0 2024-01-27T01:07:21.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe15adff700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15415c1c0 con 0x7fe15406abf0 2024-01-27T01:07:21.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe153fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1540af190 0x7fe15415b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:21.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe153fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1540af190 0x7fe15415b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42060/0 (socket says 172.21.15.46:42060) 2024-01-27T01:07:21.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe153fff700 1 -- 172.21.15.46:0/4126463991 learned_addr learned my addr 172.21.15.46:0/4126463991 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:21.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe158b9b700 1 --2- 172.21.15.46:0/4126463991 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe15406abf0 0x7fe1540c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:21.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe15939c700 1 --2- 172.21.15.46:0/4126463991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1541595b0 0x7fe1540c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:21.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe153fff700 1 -- 172.21.15.46:0/4126463991 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe15406abf0 msgr2=0x7fe1540c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:21.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe153fff700 1 --2- 172.21.15.46:0/4126463991 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe15406abf0 0x7fe1540c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:21.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe153fff700 1 -- 172.21.15.46:0/4126463991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1541595b0 msgr2=0x7fe1540c3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:21.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe153fff700 1 --2- 172.21.15.46:0/4126463991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1541595b0 0x7fe1540c3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:21.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe153fff700 1 -- 172.21.15.46:0/4126463991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe140009420 con 0x7fe1540af190 2024-01-27T01:07:21.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe158b9b700 1 --2- 172.21.15.46:0/4126463991 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe15406abf0 0x7fe1540c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:21.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.058+0000 7fe153fff700 1 --2- 172.21.15.46:0/4126463991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1540af190 0x7fe15415b180 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fe140013000 tx=0x7fe140017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:21.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.059+0000 7fe150ff9700 1 -- 172.21.15.46:0/4126463991 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14001f070 con 0x7fe1540af190 2024-01-27T01:07:21.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.059+0000 7fe150ff9700 1 -- 172.21.15.46:0/4126463991 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe140000c90 con 0x7fe1540af190 2024-01-27T01:07:21.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.059+0000 7fe150ff9700 1 -- 172.21.15.46:0/4126463991 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140025420 con 0x7fe1540af190 2024-01-27T01:07:21.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.059+0000 7fe15adff700 1 -- 172.21.15.46:0/4126463991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15415b920 con 0x7fe1540af190 2024-01-27T01:07:21.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.059+0000 7fe15adff700 1 -- 172.21.15.46:0/4126463991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15415bee0 con 0x7fe1540af190 2024-01-27T01:07:21.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.060+0000 7fe15adff700 1 -- 172.21.15.46:0/4126463991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe154169db0 con 0x7fe1540af190 2024-01-27T01:07:21.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.060+0000 7fe150ff9700 1 -- 172.21.15.46:0/4126463991 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fe140023070 con 0x7fe1540af190 2024-01-27T01:07:21.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.060+0000 7fe150ff9700 1 --2- 172.21.15.46:0/4126463991 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe138067170 0x7fe138069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:21.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.061+0000 7fe150ff9700 1 -- 172.21.15.46:0/4126463991 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6482+0+0 (secure 0 0 0) 0x7fe1400860d0 con 0x7fe1540af190 2024-01-27T01:07:21.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.061+0000 7fe158b9b700 1 --2- 172.21.15.46:0/4126463991 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe138067170 0x7fe138069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:21.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.061+0000 7fe158b9b700 1 --2- 172.21.15.46:0/4126463991 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe138067170 0x7fe138069620 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe148000c00 tx=0x7fe148009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:21.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.065+0000 7fe150ff9700 1 -- 172.21.15.46:0/4126463991 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe154169db0 con 0x7fe1540af190 2024-01-27T01:07:21.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.356+0000 7fe15adff700 1 -- 172.21.15.46:0/4126463991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7fe154169a40 con 0x7fe1540af190 2024-01-27T01:07:21.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.902+0000 7fe150ff9700 1 -- 172.21.15.46:0/4126463991 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3196) v1 ==== 190+0+0 (secure 0 0 0) 0x7fe140052da0 con 0x7fe1540af190 2024-01-27T01:07:21.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.904+0000 7fe15adff700 1 -- 172.21.15.46:0/4126463991 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe138067170 msgr2=0x7fe138069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:21.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.904+0000 7fe15adff700 1 --2- 172.21.15.46:0/4126463991 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe138067170 0x7fe138069620 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe148000c00 tx=0x7fe148009040 comp rx=0 tx=0).stop 2024-01-27T01:07:21.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.904+0000 7fe15adff700 1 -- 172.21.15.46:0/4126463991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1540af190 msgr2=0x7fe15415b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:21.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.904+0000 7fe15adff700 1 --2- 172.21.15.46:0/4126463991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1540af190 0x7fe15415b180 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fe140013000 tx=0x7fe140017000 comp rx=0 tx=0).stop 2024-01-27T01:07:21.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.904+0000 7fe15adff700 1 -- 172.21.15.46:0/4126463991 shutdown_connections 2024-01-27T01:07:21.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.904+0000 7fe15adff700 1 --2- 172.21.15.46:0/4126463991 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe138067170 0x7fe138069620 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:21.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.904+0000 7fe15adff700 1 --2- 172.21.15.46:0/4126463991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1541595b0 0x7fe1540c3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:21.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.904+0000 7fe15adff700 1 --2- 172.21.15.46:0/4126463991 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe15406abf0 0x7fe1540c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:21.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.904+0000 7fe15adff700 1 --2- 172.21.15.46:0/4126463991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1540af190 0x7fe15415b180 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:21.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.904+0000 7fe15adff700 1 -- 172.21.15.46:0/4126463991 >> 172.21.15.46:0/4126463991 conn(0x7fe15406e2c0 msgr2=0x7fe1540b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:21.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.905+0000 7fe15adff700 1 -- 172.21.15.46:0/4126463991 shutdown_connections 2024-01-27T01:07:21.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:21.905+0000 7fe15adff700 1 -- 172.21.15.46:0/4126463991 wait complete. 2024-01-27T01:07:21.920 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-27T01:07:22.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.060+0000 7f6b3a76c700 1 Processor -- start 2024-01-27T01:07:22.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.060+0000 7f6b3a76c700 1 -- start start 2024-01-27T01:07:22.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b3a76c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6b34153c40 0x7f6b3406d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:22.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b3a76c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b34154010 0x7f6b3406dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:22.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b3a76c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b340b1e80 0x7f6b3406e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:22.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b3a76c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b340b1d20 con 0x7f6b34154010 2024-01-27T01:07:22.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b3a76c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b3415ad60 con 0x7f6b34153c40 2024-01-27T01:07:22.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b3a76c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3415a150 con 0x7f6b340b1e80 2024-01-27T01:07:22.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b337fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b34154010 0x7f6b3406dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:22.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b337fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b34154010 0x7f6b3406dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42072/0 (socket says 172.21.15.46:42072) 2024-01-27T01:07:22.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b337fe700 1 -- 172.21.15.46:0/347388208 learned_addr learned my addr 172.21.15.46:0/347388208 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:22.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b38d09700 1 --2- 172.21.15.46:0/347388208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b340b1e80 0x7f6b3406e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:22.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b33fff700 1 --2- 172.21.15.46:0/347388208 >> v2:172.21.15.106:3300/0 conn(0x7f6b34153c40 0x7f6b3406d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:22.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b337fe700 1 -- 172.21.15.46:0/347388208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b340b1e80 msgr2=0x7f6b3406e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:22.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b337fe700 1 --2- 172.21.15.46:0/347388208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b340b1e80 0x7f6b3406e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b337fe700 1 -- 172.21.15.46:0/347388208 >> v2:172.21.15.106:3300/0 conn(0x7f6b34153c40 msgr2=0x7f6b3406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b337fe700 1 --2- 172.21.15.46:0/347388208 >> v2:172.21.15.106:3300/0 conn(0x7f6b34153c40 0x7f6b3406d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b337fe700 1 -- 172.21.15.46:0/347388208 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3415a430 con 0x7f6b34154010 2024-01-27T01:07:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b33fff700 1 --2- 172.21.15.46:0/347388208 >> v2:172.21.15.106:3300/0 conn(0x7f6b34153c40 0x7f6b3406d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b337fe700 1 --2- 172.21.15.46:0/347388208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b34154010 0x7f6b3406dbe0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f6b3406f8e0 tx=0x7f6b28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a78d3a55b636d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.061+0000 7f6b32ffd700 1 -- 172.21.15.46:0/347388208 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b28009e70 con 0x7f6b34154010 2024-01-27T01:07:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.062+0000 7f6b32ffd700 1 -- 172.21.15.46:0/347388208 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2800b860 con 0x7f6b34154010 2024-01-27T01:07:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.062+0000 7f6b32ffd700 1 -- 172.21.15.46:0/347388208 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2800bc00 con 0x7f6b34154010 2024-01-27T01:07:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.062+0000 7f6b3a76c700 1 -- 172.21.15.46:0/347388208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b34154010 msgr2=0x7f6b3406dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.062+0000 7f6b3a76c700 1 --2- 172.21.15.46:0/347388208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b34154010 0x7f6b3406dbe0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f6b3406f8e0 tx=0x7f6b28009250 comp rx=0 tx=0).stop 2024-01-27T01:07:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.062+0000 7f6b3a76c700 1 -- 172.21.15.46:0/347388208 shutdown_connections 2024-01-27T01:07:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.062+0000 7f6b3a76c700 1 --2- 172.21.15.46:0/347388208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b340b1e80 0x7f6b3406e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.062+0000 7f6b3a76c700 1 --2- 172.21.15.46:0/347388208 >> v2:172.21.15.106:3300/0 conn(0x7f6b34153c40 0x7f6b3406d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.062+0000 7f6b3a76c700 1 --2- 172.21.15.46:0/347388208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b34154010 0x7f6b3406dbe0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.062+0000 7f6b3a76c700 1 -- 172.21.15.46:0/347388208 >> 172.21.15.46:0/347388208 conn(0x7f6b34067930 msgr2=0x7f6b34067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:22.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.062+0000 7f6b3a76c700 1 -- 172.21.15.46:0/347388208 shutdown_connections 2024-01-27T01:07:22.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.062+0000 7f6b3a76c700 1 -- 172.21.15.46:0/347388208 wait complete. 2024-01-27T01:07:22.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.063+0000 7f6b3a76c700 1 Processor -- start 2024-01-27T01:07:22.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.064+0000 7f6b3a76c700 1 -- start start 2024-01-27T01:07:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.064+0000 7f6b3a76c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b340b1e80 0x7f6b3414d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.064+0000 7f6b3a76c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b34153c40 0x7f6b3414dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.064+0000 7f6b3a76c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b34154010 0x7f6b3416a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.064+0000 7f6b3a76c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3414ef90 con 0x7f6b34153c40 2024-01-27T01:07:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.064+0000 7f6b3a76c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b340c86e0 con 0x7f6b34154010 2024-01-27T01:07:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.064+0000 7f6b3a76c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b340ca0c0 con 0x7f6b340b1e80 2024-01-27T01:07:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.065+0000 7f6b337fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b34153c40 0x7f6b3414dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.065+0000 7f6b337fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b34153c40 0x7f6b3414dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42080/0 (socket says 172.21.15.46:42080) 2024-01-27T01:07:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.065+0000 7f6b337fe700 1 -- 172.21.15.46:0/69035845 learned_addr learned my addr 172.21.15.46:0/69035845 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.065+0000 7f6b33fff700 1 --2- 172.21.15.46:0/69035845 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b340b1e80 0x7f6b3414d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.065+0000 7f6b38d09700 1 --2- 172.21.15.46:0/69035845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b34154010 0x7f6b3416a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.065+0000 7f6b337fe700 1 -- 172.21.15.46:0/69035845 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b340b1e80 msgr2=0x7f6b3414d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.065+0000 7f6b337fe700 1 --2- 172.21.15.46:0/69035845 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b340b1e80 0x7f6b3414d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.065+0000 7f6b337fe700 1 -- 172.21.15.46:0/69035845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b34154010 msgr2=0x7f6b3416a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.065+0000 7f6b337fe700 1 --2- 172.21.15.46:0/69035845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b34154010 0x7f6b3416a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.065+0000 7f6b337fe700 1 -- 172.21.15.46:0/69035845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b280094f0 con 0x7f6b34153c40 2024-01-27T01:07:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.065+0000 7f6b337fe700 1 --2- 172.21.15.46:0/69035845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b34153c40 0x7f6b3414dcc0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f6b28009290 tx=0x7f6b28017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.065+0000 7f6b1bfff700 1 -- 172.21.15.46:0/69035845 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2800bbe0 con 0x7f6b34153c40 2024-01-27T01:07:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.065+0000 7f6b1bfff700 1 -- 172.21.15.46:0/69035845 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b28000c90 con 0x7f6b34153c40 2024-01-27T01:07:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.065+0000 7f6b3a76c700 1 -- 172.21.15.46:0/69035845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b340ca3a0 con 0x7f6b34153c40 2024-01-27T01:07:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.065+0000 7f6b1bfff700 1 -- 172.21.15.46:0/69035845 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2801a680 con 0x7f6b34153c40 2024-01-27T01:07:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.065+0000 7f6b3a76c700 1 -- 172.21.15.46:0/69035845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b340c9d70 con 0x7f6b34153c40 2024-01-27T01:07:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.067+0000 7f6b3a76c700 1 -- 172.21.15.46:0/69035845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3414f0d0 con 0x7f6b34153c40 2024-01-27T01:07:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.067+0000 7f6b1bfff700 1 -- 172.21.15.46:0/69035845 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f6b2801aa30 con 0x7f6b34153c40 2024-01-27T01:07:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.068+0000 7f6b1bfff700 1 --2- 172.21.15.46:0/69035845 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6b20067170 0x7f6b20069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.068+0000 7f6b1bfff700 1 -- 172.21.15.46:0/69035845 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6482+0+0 (secure 0 0 0) 0x7f6b28085f40 con 0x7f6b34153c40 2024-01-27T01:07:22.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.068+0000 7f6b33fff700 1 --2- 172.21.15.46:0/69035845 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6b20067170 0x7f6b20069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:22.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.068+0000 7f6b33fff700 1 --2- 172.21.15.46:0/69035845 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6b20067170 0x7f6b20069620 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6b24006f00 tx=0x7f6b24008040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:22.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.072+0000 7f6b1bfff700 1 -- 172.21.15.46:0/69035845 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6b3414f0d0 con 0x7f6b34153c40 2024-01-27T01:07:22.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.276+0000 7f6b3a76c700 1 -- 172.21.15.46:0/69035845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6b3414f0d0 con 0x7f6b34153c40 2024-01-27T01:07:22.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.281+0000 7f6b1bfff700 1 -- 172.21.15.46:0/69035845 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v82)=0 v82) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6b3414f0d0 con 0x7f6b34153c40 2024-01-27T01:07:22.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.283+0000 7f6b3a76c700 1 -- 172.21.15.46:0/69035845 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6b20067170 msgr2=0x7f6b20069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:22.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.283+0000 7f6b3a76c700 1 --2- 172.21.15.46:0/69035845 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6b20067170 0x7f6b20069620 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6b24006f00 tx=0x7f6b24008040 comp rx=0 tx=0).stop 2024-01-27T01:07:22.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.283+0000 7f6b3a76c700 1 -- 172.21.15.46:0/69035845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b34153c40 msgr2=0x7f6b3414dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:22.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.283+0000 7f6b3a76c700 1 --2- 172.21.15.46:0/69035845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b34153c40 0x7f6b3414dcc0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f6b28009290 tx=0x7f6b28017000 comp rx=0 tx=0).stop 2024-01-27T01:07:22.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.283+0000 7f6b3a76c700 1 -- 172.21.15.46:0/69035845 shutdown_connections 2024-01-27T01:07:22.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.283+0000 7f6b3a76c700 1 --2- 172.21.15.46:0/69035845 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6b20067170 0x7f6b20069620 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.283+0000 7f6b3a76c700 1 --2- 172.21.15.46:0/69035845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b34154010 0x7f6b3416a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.283+0000 7f6b3a76c700 1 --2- 172.21.15.46:0/69035845 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b340b1e80 0x7f6b3414d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.283+0000 7f6b3a76c700 1 --2- 172.21.15.46:0/69035845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b34153c40 0x7f6b3414dcc0 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.283+0000 7f6b3a76c700 1 -- 172.21.15.46:0/69035845 >> 172.21.15.46:0/69035845 conn(0x7f6b34067930 msgr2=0x7f6b340adc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:22.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.283+0000 7f6b3a76c700 1 -- 172.21.15.46:0/69035845 shutdown_connections 2024-01-27T01:07:22.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.283+0000 7f6b3a76c700 1 -- 172.21.15.46:0/69035845 wait complete. 2024-01-27T01:07:22.297 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-27T01:07:22.297 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:07:22.341 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:07:22.457 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-27T01:07:22.457 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-27T01:07:22.458 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:07:22.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.597+0000 7feeef094700 1 Processor -- start 2024-01-27T01:07:22.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.597+0000 7feeef094700 1 -- start start 2024-01-27T01:07:22.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.598+0000 7feeef094700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7feee8159760 0x7feee80b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:22.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.598+0000 7feeef094700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feee8159b30 0x7feee80b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:22.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.598+0000 7feeef094700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee815abc0 con 0x7feee815afc0 2024-01-27T01:07:22.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.598+0000 7feeef094700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee80b2e90 con 0x7feee8159b30 2024-01-27T01:07:22.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.598+0000 7feeef094700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee8159fb0 con 0x7feee8159760 2024-01-27T01:07:22.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.598+0000 7feee7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feee8159b30 0x7feee80b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:22.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.598+0000 7feee7fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feee8159b30 0x7feee80b11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42086/0 (socket says 172.21.15.46:42086) 2024-01-27T01:07:22.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.598+0000 7feee7fff700 1 -- 172.21.15.46:0/2513924862 learned_addr learned my addr 172.21.15.46:0/2513924862 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:22.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.598+0000 7feeece30700 1 --2- 172.21.15.46:0/2513924862 >> v2:172.21.15.106:3300/0 conn(0x7feee8159760 0x7feee80b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:22.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.598+0000 7feee7fff700 1 -- 172.21.15.46:0/2513924862 >> v2:172.21.15.106:3300/0 conn(0x7feee8159760 msgr2=0x7feee80b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:22.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.598+0000 7feee7fff700 1 --2- 172.21.15.46:0/2513924862 >> v2:172.21.15.106:3300/0 conn(0x7feee8159760 0x7feee80b0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.598+0000 7feee7fff700 1 -- 172.21.15.46:0/2513924862 >> v1:172.21.15.106:6789/0 conn(0x7feee815afc0 legacy=0x7feee8067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:22.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.598+0000 7feee7fff700 1 -- 172.21.15.46:0/2513924862 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee815a290 con 0x7feee8159b30 2024-01-27T01:07:22.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.599+0000 7feee7fff700 1 --2- 172.21.15.46:0/2513924862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee8159b30 0x7feee80b11b0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7feee80707a0 tx=0x7feed4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1932413802898f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:22.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.599+0000 7feee77fe700 1 -- 172.21.15.46:0/2513924862 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed4009e70 con 0x7feee8159b30 2024-01-27T01:07:22.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.599+0000 7feee77fe700 1 -- 172.21.15.46:0/2513924862 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed400b860 con 0x7feee8159b30 2024-01-27T01:07:22.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.599+0000 7feee77fe700 1 -- 172.21.15.46:0/2513924862 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed400bc20 con 0x7feee8159b30 2024-01-27T01:07:22.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.599+0000 7feeef094700 1 -- 172.21.15.46:0/2513924862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee8159b30 msgr2=0x7feee80b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:22.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.599+0000 7feeef094700 1 --2- 172.21.15.46:0/2513924862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee8159b30 0x7feee80b11b0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7feee80707a0 tx=0x7feed4009250 comp rx=0 tx=0).stop 2024-01-27T01:07:22.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.599+0000 7feeef094700 1 -- 172.21.15.46:0/2513924862 shutdown_connections 2024-01-27T01:07:22.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.599+0000 7feeef094700 1 --2- 172.21.15.46:0/2513924862 >> v2:172.21.15.106:3300/0 conn(0x7feee8159760 0x7feee80b0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.599+0000 7feeef094700 1 --2- 172.21.15.46:0/2513924862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee8159b30 0x7feee80b11b0 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.599+0000 7feeef094700 1 -- 172.21.15.46:0/2513924862 >> 172.21.15.46:0/2513924862 conn(0x7feee806b630 msgr2=0x7feee80aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:22.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.600+0000 7feeef094700 1 -- 172.21.15.46:0/2513924862 shutdown_connections 2024-01-27T01:07:22.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.600+0000 7feeef094700 1 -- 172.21.15.46:0/2513924862 wait complete. 2024-01-27T01:07:22.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.601+0000 7feeef094700 1 Processor -- start 2024-01-27T01:07:22.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.601+0000 7feeef094700 1 -- start start 2024-01-27T01:07:22.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.602+0000 7feeef094700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feee8159760 0x7feee816a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:22.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.602+0000 7feeef094700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee815afc0 0x7feee80c2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:22.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.602+0000 7feeef094700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feee80c3100 0x7feee815b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:22.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.602+0000 7feeef094700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee80c35a0 con 0x7feee815afc0 2024-01-27T01:07:22.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.602+0000 7feeef094700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee815d2a0 con 0x7feee8159760 2024-01-27T01:07:22.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.602+0000 7feeef094700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee815c690 con 0x7feee80c3100 2024-01-27T01:07:22.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.602+0000 7feee7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee815afc0 0x7feee80c2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:22.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.602+0000 7feee7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee815afc0 0x7feee80c2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42090/0 (socket says 172.21.15.46:42090) 2024-01-27T01:07:22.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.602+0000 7feee7fff700 1 -- 172.21.15.46:0/495544240 learned_addr learned my addr 172.21.15.46:0/495544240 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:22.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.602+0000 7feeece30700 1 --2- 172.21.15.46:0/495544240 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feee8159760 0x7feee816a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:22.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.602+0000 7feeed631700 1 --2- 172.21.15.46:0/495544240 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feee80c3100 0x7feee815b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:22.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.602+0000 7feee7fff700 1 -- 172.21.15.46:0/495544240 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feee80c3100 msgr2=0x7feee815b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:22.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.602+0000 7feee7fff700 1 --2- 172.21.15.46:0/495544240 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feee80c3100 0x7feee815b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.602+0000 7feee7fff700 1 -- 172.21.15.46:0/495544240 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feee8159760 msgr2=0x7feee816a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:22.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.602+0000 7feee7fff700 1 --2- 172.21.15.46:0/495544240 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feee8159760 0x7feee816a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.602+0000 7feee7fff700 1 -- 172.21.15.46:0/495544240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed40094f0 con 0x7feee815afc0 2024-01-27T01:07:22.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.603+0000 7feee7fff700 1 --2- 172.21.15.46:0/495544240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee815afc0 0x7feee80c2bc0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7feee80ce590 tx=0x7feed4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:22.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.603+0000 7feee4ff9700 1 -- 172.21.15.46:0/495544240 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed401f070 con 0x7feee815afc0 2024-01-27T01:07:22.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.603+0000 7feee4ff9700 1 -- 172.21.15.46:0/495544240 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed4002f30 con 0x7feee815afc0 2024-01-27T01:07:22.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.603+0000 7feeef094700 1 -- 172.21.15.46:0/495544240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee815c970 con 0x7feee815afc0 2024-01-27T01:07:22.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.603+0000 7feee4ff9700 1 -- 172.21.15.46:0/495544240 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed400b9d0 con 0x7feee815afc0 2024-01-27T01:07:22.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.603+0000 7feeef094700 1 -- 172.21.15.46:0/495544240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee815cf30 con 0x7feee815afc0 2024-01-27T01:07:22.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.604+0000 7feeef094700 1 -- 172.21.15.46:0/495544240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee815d430 con 0x7feee815afc0 2024-01-27T01:07:22.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.604+0000 7feee4ff9700 1 -- 172.21.15.46:0/495544240 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7feed4023070 con 0x7feee815afc0 2024-01-27T01:07:22.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.605+0000 7feee4ff9700 1 --2- 172.21.15.46:0/495544240 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feecc067170 0x7feecc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:22.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.605+0000 7feee4ff9700 1 -- 172.21.15.46:0/495544240 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6482+0+0 (secure 0 0 0) 0x7feed4085dd0 con 0x7feee815afc0 2024-01-27T01:07:22.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.605+0000 7feeece30700 1 --2- 172.21.15.46:0/495544240 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feecc067170 0x7feecc069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:22.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.606+0000 7feeece30700 1 --2- 172.21.15.46:0/495544240 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feecc067170 0x7feecc069620 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feedc009d10 tx=0x7feedc0091a0 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:22.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.609+0000 7feee4ff9700 1 -- 172.21.15.46:0/495544240 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feed4052aa0 con 0x7feee815afc0 2024-01-27T01:07:22.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.862+0000 7feeef094700 1 -- 172.21.15.46:0/495544240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feee804c710 con 0x7feee815afc0 2024-01-27T01:07:22.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.863+0000 7feee4ff9700 1 -- 172.21.15.46:0/495544240 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+3540 (secure 0 0 0) 0x7feed402adf0 con 0x7feee815afc0 2024-01-27T01:07:22.863 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:22.863 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":126,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/1999300009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":1999300009},{"type":"v1","addr":"172.21.15.106:6835","nonce":1999300009}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":110},{"gid":10486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/4174263148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":4174263148},{"type":"v1","addr":"172.21.15.106:6833","nonce":4174263148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":111},{"gid":11454,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1260622833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":1260622833},{"type":"v1","addr":"172.21.15.46:6835","nonce":1260622833}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":124},{"gid":11559,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/950444387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":950444387},{"type":"v1","addr":"172.21.15.46:6833","nonce":950444387}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126}],"filesystems":[]} 2024-01-27T01:07:22.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.865+0000 7feeef094700 1 -- 172.21.15.46:0/495544240 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feecc067170 msgr2=0x7feecc069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:22.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.865+0000 7feeef094700 1 --2- 172.21.15.46:0/495544240 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feecc067170 0x7feecc069620 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feedc009d10 tx=0x7feedc0091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.865+0000 7feeef094700 1 -- 172.21.15.46:0/495544240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee815afc0 msgr2=0x7feee80c2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:22.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.865+0000 7feeef094700 1 --2- 172.21.15.46:0/495544240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee815afc0 0x7feee80c2bc0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7feee80ce590 tx=0x7feed4017000 comp rx=0 tx=0).stop 2024-01-27T01:07:22.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.865+0000 7feeef094700 1 -- 172.21.15.46:0/495544240 shutdown_connections 2024-01-27T01:07:22.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.865+0000 7feeef094700 1 --2- 172.21.15.46:0/495544240 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feecc067170 0x7feecc069620 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.865+0000 7feeef094700 1 --2- 172.21.15.46:0/495544240 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feee80c3100 0x7feee815b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.865+0000 7feeef094700 1 --2- 172.21.15.46:0/495544240 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feee8159760 0x7feee816a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.865+0000 7feeef094700 1 --2- 172.21.15.46:0/495544240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee815afc0 0x7feee80c2bc0 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:22.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.865+0000 7feeef094700 1 -- 172.21.15.46:0/495544240 >> 172.21.15.46:0/495544240 conn(0x7feee806b630 msgr2=0x7feee80adb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:22.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.865+0000 7feeef094700 1 -- 172.21.15.46:0/495544240 shutdown_connections 2024-01-27T01:07:22.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:22.865+0000 7feeef094700 1 -- 172.21.15.46:0/495544240 wait complete. 2024-01-27T01:07:22.866 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 126 2024-01-27T01:07:22.879 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-27T01:07:22.879 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-27T01:07:22.879 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-27T01:07:22.879 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:07:22.880 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-27T01:07:22.880 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-27T01:07:22.880 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-27T01:07:22.880 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:07:22.880 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-27T01:07:22.880 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-27T01:07:22.880 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-27T01:07:22.880 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:07:22.881 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-27T01:07:22.881 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-27T01:07:22.881 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-27T01:07:22.881 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:07:22.893 INFO:tasks.ceph.mds.a:Stopped 2024-01-27T01:07:22.893 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-27T01:07:22.893 INFO:tasks.ceph.mds.b:Stopped 2024-01-27T01:07:22.894 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-27T01:07:22.894 INFO:tasks.ceph.mds.c:Stopped 2024-01-27T01:07:22.895 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-27T01:07:22.896 INFO:tasks.ceph.mds.d:Stopped 2024-01-27T01:07:22.896 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-27T01:07:22.935 INFO:tasks.ceph.mds.b:Started 2024-01-27T01:07:22.936 DEBUG:teuthology.parallel:result is None 2024-01-27T01:07:22.936 INFO:tasks.ceph.mds.a:Started 2024-01-27T01:07:22.937 INFO:tasks.ceph.mds.d:Started 2024-01-27T01:07:22.937 DEBUG:teuthology.parallel:result is None 2024-01-27T01:07:22.937 DEBUG:teuthology.parallel:result is None 2024-01-27T01:07:22.938 INFO:tasks.ceph.mds.c:Started 2024-01-27T01:07:22.938 DEBUG:teuthology.parallel:result is None 2024-01-27T01:07:22.938 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-27T01:07:23.020 INFO:tasks.ceph.mds.b.smithi106.stdout:starting mds.b at 2024-01-27T01:07:23.021 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-01-27T01:07:23.024 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-27T01:07:23.024 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-27T01:07:23.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.079+0000 7f076e142700 1 Processor -- start 2024-01-27T01:07:23.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.079+0000 7f076e142700 1 -- start start 2024-01-27T01:07:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.079+0000 7f076e142700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f07680c6520 0x7f07680ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:23.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.079+0000 7f076e142700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076815a0b0 con 0x7f07680ced50 2024-01-27T01:07:23.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.079+0000 7f076e142700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07680ae4d0 con 0x7f07680b2b40 2024-01-27T01:07:23.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.079+0000 7f076e142700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07680af070 con 0x7f07680c6520 2024-01-27T01:07:23.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.080+0000 7f0767fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f07680ced50 0x7f07680cb5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47320/0 (socket says 172.21.15.46:47320) 2024-01-27T01:07:23.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.080+0000 7f0767fff700 1 -- 172.21.15.46:0/2279900012 learned_addr learned my addr 172.21.15.46:0/2279900012 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:23.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.080+0000 7f0766ffd700 1 --2- 172.21.15.46:0/2279900012 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f07680c6520 0x7f07680ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:23.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.080+0000 7f07667fc700 1 -- 172.21.15.46:0/2279900012 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427011122 0 0) 0x7f07680ae4d0 con 0x7f07680b2b40 2024-01-27T01:07:23.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.080+0000 7f07667fc700 1 -- 172.21.15.46:0/2279900012 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0750003570 con 0x7f07680b2b40 2024-01-27T01:07:23.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.080+0000 7f07667fc700 1 -- 172.21.15.46:0/2279900012 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069500207 0 0) 0x7f076815a0b0 con 0x7f07680ced50 2024-01-27T01:07:23.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.080+0000 7f07667fc700 1 -- 172.21.15.46:0/2279900012 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07680ae4d0 con 0x7f07680ced50 2024-01-27T01:07:23.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.080+0000 7f0766ffd700 1 -- 172.21.15.46:0/2279900012 >> v1:172.21.15.106:6789/0 conn(0x7f07680b2b40 legacy=0x7f07680ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:23.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.080+0000 7f0766ffd700 1 -- 172.21.15.46:0/2279900012 >> v1:172.21.15.46:6789/0 conn(0x7f07680ced50 legacy=0x7f07680cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.080+0000 7f0766ffd700 1 -- 172.21.15.46:0/2279900012 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07680ad8c0 con 0x7f07680c6520 2024-01-27T01:07:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.080+0000 7f0766ffd700 1 --2- 172.21.15.46:0/2279900012 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f07680c6520 0x7f07680ad1c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f075c009910 tx=0x7f075c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef903aed8ddc5905 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.080+0000 7f07667fc700 1 -- 172.21.15.46:0/2279900012 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c011630 con 0x7f07680c6520 2024-01-27T01:07:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.081+0000 7f07667fc700 1 -- 172.21.15.46:0/2279900012 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c011790 con 0x7f07680c6520 2024-01-27T01:07:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.081+0000 7f07667fc700 1 -- 172.21.15.46:0/2279900012 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c011b30 con 0x7f07680c6520 2024-01-27T01:07:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.081+0000 7f076e142700 1 -- 172.21.15.46:0/2279900012 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f07680c6520 msgr2=0x7f07680ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.081+0000 7f076e142700 1 --2- 172.21.15.46:0/2279900012 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f07680c6520 0x7f07680ad1c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f075c009910 tx=0x7f075c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.081+0000 7f076e142700 1 -- 172.21.15.46:0/2279900012 shutdown_connections 2024-01-27T01:07:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.081+0000 7f076e142700 1 --2- 172.21.15.46:0/2279900012 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f07680c6520 0x7f07680ad1c0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.081+0000 7f076e142700 1 -- 172.21.15.46:0/2279900012 >> 172.21.15.46:0/2279900012 conn(0x7f0768099a30 msgr2=0x7f076806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.081+0000 7f076e142700 1 -- 172.21.15.46:0/2279900012 shutdown_connections 2024-01-27T01:07:23.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.081+0000 7f076e142700 1 -- 172.21.15.46:0/2279900012 wait complete. 2024-01-27T01:07:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.083+0000 7f076e142700 1 Processor -- start 2024-01-27T01:07:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.083+0000 7f076e142700 1 -- start start 2024-01-27T01:07:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.083+0000 7f076e142700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f07680b2b40 0x7f0768157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.083+0000 7f076e142700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f07680c6520 0x7f07680c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.083+0000 7f076e142700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07680ced50 0x7f07681512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:23.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.083+0000 7f076e142700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07680c8bc0 con 0x7f07680ced50 2024-01-27T01:07:23.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.083+0000 7f076e142700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07680c93b0 con 0x7f07680b2b40 2024-01-27T01:07:23.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.083+0000 7f076e142700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07680c8320 con 0x7f07680c6520 2024-01-27T01:07:23.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.083+0000 7f0767fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07680ced50 0x7f07681512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:23.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.083+0000 7f0766ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f07680c6520 0x7f07680c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:23.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.084+0000 7f0767fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07680ced50 0x7f07681512f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42156/0 (socket says 172.21.15.46:42156) 2024-01-27T01:07:23.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.083+0000 7f07677fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f07680b2b40 0x7f0768157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:23.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.084+0000 7f0767fff700 1 -- 172.21.15.46:0/1874547404 learned_addr learned my addr 172.21.15.46:0/1874547404 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:23.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.084+0000 7f0766ffd700 1 -- 172.21.15.46:0/1874547404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f07680b2b40 msgr2=0x7f0768157bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:23.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.084+0000 7f0766ffd700 1 --2- 172.21.15.46:0/1874547404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f07680b2b40 0x7f0768157bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:23.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.084+0000 7f0766ffd700 1 -- 172.21.15.46:0/1874547404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07680ced50 msgr2=0x7f07681512f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:23.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.084+0000 7f0766ffd700 1 --2- 172.21.15.46:0/1874547404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07680ced50 0x7f07681512f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:23.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.084+0000 7f0766ffd700 1 -- 172.21.15.46:0/1874547404 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c009420 con 0x7f07680c6520 2024-01-27T01:07:23.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.084+0000 7f07677fe700 1 --2- 172.21.15.46:0/1874547404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f07680b2b40 0x7f0768157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:23.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.084+0000 7f0767fff700 1 --2- 172.21.15.46:0/1874547404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07680ced50 0x7f07681512f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:23.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.084+0000 7f0766ffd700 1 --2- 172.21.15.46:0/1874547404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f07680c6520 0x7f07680c7820 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f07680644e0 tx=0x7f075c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:23.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.084+0000 7f074f7fe700 1 -- 172.21.15.46:0/1874547404 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c01f070 con 0x7f07680c6520 2024-01-27T01:07:23.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.084+0000 7f074f7fe700 1 -- 172.21.15.46:0/1874547404 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c002f70 con 0x7f07680c6520 2024-01-27T01:07:23.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.084+0000 7f076e142700 1 -- 172.21.15.46:0/1874547404 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07680c8600 con 0x7f07680c6520 2024-01-27T01:07:23.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.084+0000 7f074f7fe700 1 -- 172.21.15.46:0/1874547404 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c011900 con 0x7f07680c6520 2024-01-27T01:07:23.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.084+0000 7f076e142700 1 -- 172.21.15.46:0/1874547404 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0768157560 con 0x7f07680c6520 2024-01-27T01:07:23.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.085+0000 7f076e142700 1 -- 172.21.15.46:0/1874547404 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07680c9210 con 0x7f07680c6520 2024-01-27T01:07:23.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.086+0000 7f074f7fe700 1 -- 172.21.15.46:0/1874547404 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f075c023070 con 0x7f07680c6520 2024-01-27T01:07:23.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.086+0000 7f074f7fe700 1 --2- 172.21.15.46:0/1874547404 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0754067170 0x7f0754069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:23.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.086+0000 7f074f7fe700 1 -- 172.21.15.46:0/1874547404 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6482+0+0 (secure 0 0 0) 0x7f075c085c50 con 0x7f07680c6520 2024-01-27T01:07:23.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.086+0000 7f07677fe700 1 --2- 172.21.15.46:0/1874547404 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0754067170 0x7f0754069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:23.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.087+0000 7f07677fe700 1 --2- 172.21.15.46:0/1874547404 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0754067170 0x7f0754069620 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0768158750 tx=0x7f0758007d60 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:23.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.091+0000 7f074f7fe700 1 -- 172.21.15.46:0/1874547404 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f07680c9210 con 0x7f07680c6520 2024-01-27T01:07:23.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.315+0000 7f076e142700 1 -- 172.21.15.46:0/1874547404 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f07680c9210 con 0x7f07680c6520 2024-01-27T01:07:23.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.920+0000 7f074f7fe700 1 -- 172.21.15.46:0/1874547404 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v237) v1 ==== 97+0+0 (secure 0 0 0) 0x7f07680c9210 con 0x7f07680c6520 2024-01-27T01:07:23.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.923+0000 7f076e142700 1 -- 172.21.15.46:0/1874547404 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0754067170 msgr2=0x7f0754069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:23.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.923+0000 7f076e142700 1 --2- 172.21.15.46:0/1874547404 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0754067170 0x7f0754069620 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0768158750 tx=0x7f0758007d60 comp rx=0 tx=0).stop 2024-01-27T01:07:23.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.923+0000 7f076e142700 1 -- 172.21.15.46:0/1874547404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f07680c6520 msgr2=0x7f07680c7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:23.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.923+0000 7f076e142700 1 --2- 172.21.15.46:0/1874547404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f07680c6520 0x7f07680c7820 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f07680644e0 tx=0x7f075c017000 comp rx=0 tx=0).stop 2024-01-27T01:07:23.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.923+0000 7f076e142700 1 -- 172.21.15.46:0/1874547404 shutdown_connections 2024-01-27T01:07:23.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.923+0000 7f076e142700 1 --2- 172.21.15.46:0/1874547404 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0754067170 0x7f0754069620 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:23.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.923+0000 7f076e142700 1 --2- 172.21.15.46:0/1874547404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07680ced50 0x7f07681512f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:23.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.923+0000 7f076e142700 1 --2- 172.21.15.46:0/1874547404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f07680b2b40 0x7f0768157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:23.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.923+0000 7f076e142700 1 --2- 172.21.15.46:0/1874547404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f07680c6520 0x7f07680c7820 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:23.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.923+0000 7f076e142700 1 -- 172.21.15.46:0/1874547404 >> 172.21.15.46:0/1874547404 conn(0x7f0768099a30 msgr2=0x7f0768068910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:23.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.923+0000 7f076e142700 1 -- 172.21.15.46:0/1874547404 shutdown_connections 2024-01-27T01:07:23.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:23.923+0000 7f076e142700 1 -- 172.21.15.46:0/1874547404 wait complete. 2024-01-27T01:07:23.925 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-01-27T01:07:23.942 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-27T01:07:24.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.077+0000 7f57b8b94700 1 Processor -- start 2024-01-27T01:07:24.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.077+0000 7f57b8b94700 1 -- start start 2024-01-27T01:07:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.078+0000 7f57b8b94700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f57b406cbb0 0x7f57b40ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.078+0000 7f57b8b94700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57b40b0280 0x7f57b41565a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.078+0000 7f57b8b94700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b40b0760 con 0x7f57b41561d0 2024-01-27T01:07:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.078+0000 7f57b8b94700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57b40b0a40 con 0x7f57b406cbb0 2024-01-27T01:07:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.078+0000 7f57b8b94700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b40b0d20 con 0x7f57b40b0280 2024-01-27T01:07:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.078+0000 7f57b1d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f57b406cbb0 0x7f57b40ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.078+0000 7f57b1d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f57b406cbb0 0x7f57b40ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42162/0 (socket says 172.21.15.46:42162) 2024-01-27T01:07:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.078+0000 7f57b1d9b700 1 -- 172.21.15.46:0/3785528893 learned_addr learned my addr 172.21.15.46:0/3785528893 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.078+0000 7f57b2d9d700 1 --2- 172.21.15.46:0/3785528893 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57b40b0280 0x7f57b41565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.078+0000 7f57b1d9b700 1 -- 172.21.15.46:0/3785528893 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57b40b0280 msgr2=0x7f57b41565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.078+0000 7f57b1d9b700 1 --2- 172.21.15.46:0/3785528893 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57b40b0280 0x7f57b41565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.078+0000 7f57b1d9b700 1 -- 172.21.15.46:0/3785528893 >> v1:172.21.15.46:6789/0 conn(0x7f57b41561d0 legacy=0x7f57b406c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:24.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.078+0000 7f57b1d9b700 1 -- 172.21.15.46:0/3785528893 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b40b1000 con 0x7f57b406cbb0 2024-01-27T01:07:24.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.079+0000 7f57b1d9b700 1 --2- 172.21.15.46:0/3785528893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57b406cbb0 0x7f57b40ca5f0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f57a80099b0 tx=0x7f57a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab4eb2ecca565a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:24.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.079+0000 7f57b159a700 1 -- 172.21.15.46:0/3785528893 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a800b860 con 0x7f57b406cbb0 2024-01-27T01:07:24.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.079+0000 7f57b159a700 1 -- 172.21.15.46:0/3785528893 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a800b9c0 con 0x7f57b406cbb0 2024-01-27T01:07:24.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.079+0000 7f57b159a700 1 -- 172.21.15.46:0/3785528893 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a800bd80 con 0x7f57b406cbb0 2024-01-27T01:07:24.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.079+0000 7f57b8b94700 1 -- 172.21.15.46:0/3785528893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57b406cbb0 msgr2=0x7f57b40ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:24.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.079+0000 7f57b8b94700 1 --2- 172.21.15.46:0/3785528893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57b406cbb0 0x7f57b40ca5f0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f57a80099b0 tx=0x7f57a8009250 comp rx=0 tx=0).stop 2024-01-27T01:07:24.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.079+0000 7f57b8b94700 1 -- 172.21.15.46:0/3785528893 shutdown_connections 2024-01-27T01:07:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.079+0000 7f57b8b94700 1 --2- 172.21.15.46:0/3785528893 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57b40b0280 0x7f57b41565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.079+0000 7f57b8b94700 1 --2- 172.21.15.46:0/3785528893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57b406cbb0 0x7f57b40ca5f0 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.079+0000 7f57b8b94700 1 -- 172.21.15.46:0/3785528893 >> 172.21.15.46:0/3785528893 conn(0x7f57b406b630 msgr2=0x7f57b40aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.079+0000 7f57b8b94700 1 -- 172.21.15.46:0/3785528893 shutdown_connections 2024-01-27T01:07:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.079+0000 7f57b8b94700 1 -- 172.21.15.46:0/3785528893 wait complete. 2024-01-27T01:07:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.081+0000 7f57b8b94700 1 Processor -- start 2024-01-27T01:07:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.081+0000 7f57b8b94700 1 -- start start 2024-01-27T01:07:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.081+0000 7f57b8b94700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57b406cbb0 0x7f57b414e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.081+0000 7f57b8b94700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57b40b0280 0x7f57b414ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.081+0000 7f57b8b94700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57b41561d0 0x7f57b414f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.081+0000 7f57b8b94700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b414aa70 con 0x7f57b406cbb0 2024-01-27T01:07:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.081+0000 7f57b8b94700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b414b880 con 0x7f57b41561d0 2024-01-27T01:07:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.081+0000 7f57b8b94700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b4149e40 con 0x7f57b40b0280 2024-01-27T01:07:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.081+0000 7f57b259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57b406cbb0 0x7f57b414e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.082+0000 7f57b259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57b406cbb0 0x7f57b414e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42176/0 (socket says 172.21.15.46:42176) 2024-01-27T01:07:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.082+0000 7f57b259c700 1 -- 172.21.15.46:0/730660986 learned_addr learned my addr 172.21.15.46:0/730660986 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.082+0000 7f57b2d9d700 1 --2- 172.21.15.46:0/730660986 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57b41561d0 0x7f57b414f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.082+0000 7f57b1d9b700 1 --2- 172.21.15.46:0/730660986 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57b40b0280 0x7f57b414ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.082+0000 7f57b259c700 1 -- 172.21.15.46:0/730660986 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57b40b0280 msgr2=0x7f57b414ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.082+0000 7f57b259c700 1 --2- 172.21.15.46:0/730660986 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57b40b0280 0x7f57b414ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.082+0000 7f57b259c700 1 -- 172.21.15.46:0/730660986 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57b41561d0 msgr2=0x7f57b414f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.082+0000 7f57b259c700 1 --2- 172.21.15.46:0/730660986 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57b41561d0 0x7f57b414f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.082+0000 7f57b259c700 1 -- 172.21.15.46:0/730660986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a80094f0 con 0x7f57b406cbb0 2024-01-27T01:07:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.082+0000 7f57b259c700 1 --2- 172.21.15.46:0/730660986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57b406cbb0 0x7f57b414e900 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f57a400c4e0 tx=0x7f57a400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.082+0000 7f57a27fc700 1 -- 172.21.15.46:0/730660986 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a4011590 con 0x7f57b406cbb0 2024-01-27T01:07:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.082+0000 7f57a27fc700 1 -- 172.21.15.46:0/730660986 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a4011710 con 0x7f57b406cbb0 2024-01-27T01:07:24.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.082+0000 7f57b8b94700 1 -- 172.21.15.46:0/730660986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b414ac70 con 0x7f57b406cbb0 2024-01-27T01:07:24.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.082+0000 7f57a27fc700 1 -- 172.21.15.46:0/730660986 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a4011ad0 con 0x7f57b406cbb0 2024-01-27T01:07:24.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.082+0000 7f57b8b94700 1 -- 172.21.15.46:0/730660986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b414b230 con 0x7f57b406cbb0 2024-01-27T01:07:24.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.083+0000 7f57b8b94700 1 -- 172.21.15.46:0/730660986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b404c710 con 0x7f57b406cbb0 2024-01-27T01:07:24.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.084+0000 7f57a27fc700 1 -- 172.21.15.46:0/730660986 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f57a401b3b0 con 0x7f57b406cbb0 2024-01-27T01:07:24.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.084+0000 7f57a27fc700 1 --2- 172.21.15.46:0/730660986 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5794067170 0x7f5794069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:24.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.084+0000 7f57a27fc700 1 -- 172.21.15.46:0/730660986 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5665+0+0 (secure 0 0 0) 0x7f57a4083c30 con 0x7f57b406cbb0 2024-01-27T01:07:24.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.084+0000 7f57b1d9b700 1 --2- 172.21.15.46:0/730660986 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5794067170 0x7f5794069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:24.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.085+0000 7f57b1d9b700 1 --2- 172.21.15.46:0/730660986 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5794067170 0x7f5794069620 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f57a80099b0 tx=0x7f57a8017000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:24.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.089+0000 7f57a27fc700 1 -- 172.21.15.46:0/730660986 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f57a4050c30 con 0x7f57b406cbb0 2024-01-27T01:07:24.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.376+0000 7f57b8b94700 1 -- 172.21.15.46:0/730660986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f57b40b12e0 con 0x7f57b406cbb0 2024-01-27T01:07:24.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.377+0000 7f57a27fc700 1 -- 172.21.15.46:0/730660986 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v54) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f57a4017070 con 0x7f57b406cbb0 2024-01-27T01:07:24.377 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:24.377 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:07:24.377 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-01-27T01:07:24.377 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.377 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-01-27T01:07:24.377 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlM6dKKBAAPpeWeDZWrFgc/4zuxKCvHw==", 2024-01-27T01:07:24.377 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.377 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:07:24.377 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:07:24.377 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:07:24.377 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:07:24.378 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.378 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.378 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.378 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-01-27T01:07:24.378 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlmzHRNRAAflWYZ/p4AXeMy54tvLpugA==", 2024-01-27T01:07:24.378 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.378 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:07:24.378 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:07:24.378 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:07:24.378 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:07:24.378 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.378 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.378 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.378 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-01-27T01:07:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlMZxoMBAAG5kyfm+LLG86y/p2A8BfDQ==", 2024-01-27T01:07:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:07:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:07:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:07:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:07:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-01-27T01:07:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl5jx/ARAA3cwtLAOWQ/n2NG+xwEZ3QQ==", 2024-01-27T01:07:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:07:24.379 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:07:24.380 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:07:24.380 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:07:24.380 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.380 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.380 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.380 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-01-27T01:07:24.380 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4QLRlJ2OKJRAAWVhxEdQkUW20294w/KLL6g==", 2024-01-27T01:07:24.380 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.380 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:07:24.380 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:07:24.380 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:07:24.380 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.380 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.380 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.381 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-01-27T01:07:24.381 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7QLRlE2tAJRAAzNlTiCr6a2eWiabRy1SqSw==", 2024-01-27T01:07:24.381 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.381 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:07:24.381 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:07:24.381 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:07:24.381 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.381 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.381 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.381 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-01-27T01:07:24.381 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+QLRlyySaJBAA5072ba5HUNi2dYE3D02M/Q==", 2024-01-27T01:07:24.381 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.381 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:07:24.381 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:07:24.381 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:07:24.382 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.382 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.382 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.382 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-01-27T01:07:24.382 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBQLRl9+tAIxAApZk0btXIpQPWSb3BYUYsFQ==", 2024-01-27T01:07:24.382 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.382 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:07:24.382 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:07:24.382 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:07:24.382 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.383 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.383 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.383 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-01-27T01:07:24.383 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHQLRlurUtJxAAT9fwKrAWl62VOXwShg7NJg==", 2024-01-27T01:07:24.383 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.383 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:07:24.383 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:07:24.383 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:07:24.383 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.383 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.383 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.383 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-01-27T01:07:24.383 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKQLRl7aqoJRAAXu9fykWELsW3SwnahoIOLA==", 2024-01-27T01:07:24.383 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.384 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:07:24.384 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:07:24.384 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:07:24.384 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.384 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.384 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.384 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-01-27T01:07:24.384 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNQLRlWf7pJBAAopmFLIKb8s32YZs46VwbGg==", 2024-01-27T01:07:24.384 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.384 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:07:24.384 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:07:24.384 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:07:24.384 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.384 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.385 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.385 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-01-27T01:07:24.385 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQQLRlNGfhIxAAmCgmOB3QAGfSIYHBb5BUZw==", 2024-01-27T01:07:24.385 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.385 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:07:24.385 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:07:24.385 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:07:24.385 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.385 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.385 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.385 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-01-27T01:07:24.385 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRleO8wBhAA1018AcO3KcyOZxgB0Itxaw==", 2024-01-27T01:07:24.385 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.385 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:07:24.385 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-01-27T01:07:24.386 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-27T01:07:24.386 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.386 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.386 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.386 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-01-27T01:07:24.386 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlsiZEDRAAjXSamHe2bZcDqVsqXqIcGg==", 2024-01-27T01:07:24.386 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.386 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:07:24.386 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T01:07:24.386 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T01:07:24.386 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T01:07:24.386 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.386 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.386 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.386 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-01-27T01:07:24.387 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl6YT1CRAApXlN4j20xzwda0tDMy94Eg==", 2024-01-27T01:07:24.387 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.387 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:07:24.387 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T01:07:24.387 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T01:07:24.387 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T01:07:24.387 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.387 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.387 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.387 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-01-27T01:07:24.387 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlHvDQEhAAxMhgWJsZC5/6Af5zFV6XIA==", 2024-01-27T01:07:24.387 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.387 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:07:24.387 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T01:07:24.388 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T01:07:24.388 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T01:07:24.388 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.388 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.388 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.388 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-01-27T01:07:24.388 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlvMwDCBAAXnFyMduz7PfWdLWbjytIiQ==", 2024-01-27T01:07:24.388 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.388 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T01:07:24.388 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:07:24.388 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:07:24.388 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:07:24.388 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.388 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.389 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.389 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-01-27T01:07:24.389 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlhMNSJBAA38QU6DrCvFjO+u9zQ8bkEA==", 2024-01-27T01:07:24.389 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.389 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T01:07:24.389 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-27T01:07:24.389 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:07:24.389 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.389 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:24.389 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:24.389 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-01-27T01:07:24.389 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlI9EbIBAACTpzAO+8A/VFJZcT/WWBng==", 2024-01-27T01:07:24.389 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:07:24.389 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T01:07:24.389 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-27T01:07:24.390 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:07:24.390 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.390 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:24.390 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-01-27T01:07:24.390 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:07:24.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.379+0000 7f57b8b94700 1 -- 172.21.15.46:0/730660986 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5794067170 msgr2=0x7f5794069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:24.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.379+0000 7f57b8b94700 1 --2- 172.21.15.46:0/730660986 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5794067170 0x7f5794069620 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f57a80099b0 tx=0x7f57a8017000 comp rx=0 tx=0).stop 2024-01-27T01:07:24.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.379+0000 7f57b8b94700 1 -- 172.21.15.46:0/730660986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57b406cbb0 msgr2=0x7f57b414e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:24.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.379+0000 7f57b8b94700 1 --2- 172.21.15.46:0/730660986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57b406cbb0 0x7f57b414e900 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f57a400c4e0 tx=0x7f57a400c7f0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.379+0000 7f57b8b94700 1 -- 172.21.15.46:0/730660986 shutdown_connections 2024-01-27T01:07:24.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.379+0000 7f57b8b94700 1 --2- 172.21.15.46:0/730660986 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5794067170 0x7f5794069620 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.379+0000 7f57b8b94700 1 --2- 172.21.15.46:0/730660986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57b406cbb0 0x7f57b414e900 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.379+0000 7f57b8b94700 1 --2- 172.21.15.46:0/730660986 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57b41561d0 0x7f57b414f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.379+0000 7f57b8b94700 1 --2- 172.21.15.46:0/730660986 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f57b40b0280 0x7f57b414ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.379+0000 7f57b8b94700 1 -- 172.21.15.46:0/730660986 >> 172.21.15.46:0/730660986 conn(0x7f57b406b630 msgr2=0x7f57b4154530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:24.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.379+0000 7f57b8b94700 1 -- 172.21.15.46:0/730660986 shutdown_connections 2024-01-27T01:07:24.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.379+0000 7f57b8b94700 1 -- 172.21.15.46:0/730660986 wait complete. 2024-01-27T01:07:24.393 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:07:24.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.524+0000 7fc9e03ea700 1 Processor -- start 2024-01-27T01:07:24.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.524+0000 7fc9e03ea700 1 -- start start 2024-01-27T01:07:24.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.525+0000 7fc9e03ea700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9d80c2450 0x7fc9d80cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:24.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.525+0000 7fc9e03ea700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc9d80cc840 0x7fc9d81601d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:24.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.525+0000 7fc9e03ea700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d80ce400 con 0x7fc9d80695f0 2024-01-27T01:07:24.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.525+0000 7fc9e03ea700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9d80c3680 con 0x7fc9d80c2450 2024-01-27T01:07:24.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.525+0000 7fc9e03ea700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9d80afe90 con 0x7fc9d80cc840 2024-01-27T01:07:24.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.525+0000 7fc9de186700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9d80c2450 0x7fc9d80cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:24.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.525+0000 7fc9de186700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9d80c2450 0x7fc9d80cdbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42186/0 (socket says 172.21.15.46:42186) 2024-01-27T01:07:24.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.525+0000 7fc9de186700 1 -- 172.21.15.46:0/1812607486 learned_addr learned my addr 172.21.15.46:0/1812607486 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:24.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.526+0000 7fc9de987700 1 --2- 172.21.15.46:0/1812607486 >> v2:172.21.15.106:3300/0 conn(0x7fc9d80cc840 0x7fc9d81601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:24.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.526+0000 7fc9de186700 1 -- 172.21.15.46:0/1812607486 >> v2:172.21.15.106:3300/0 conn(0x7fc9d80cc840 msgr2=0x7fc9d81601d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:24.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.526+0000 7fc9de186700 1 --2- 172.21.15.46:0/1812607486 >> v2:172.21.15.106:3300/0 conn(0x7fc9d80cc840 0x7fc9d81601d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.526+0000 7fc9de186700 1 -- 172.21.15.46:0/1812607486 >> v1:172.21.15.106:6789/0 conn(0x7fc9d80695f0 legacy=0x7fc9d80c2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:24.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.526+0000 7fc9de186700 1 -- 172.21.15.46:0/1812607486 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d80cb3a0 con 0x7fc9d80c2450 2024-01-27T01:07:24.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.526+0000 7fc9de186700 1 --2- 172.21.15.46:0/1812607486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d80c2450 0x7fc9d80cdbb0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fc9d80c7cc0 tx=0x7fc9d4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=508f28ad0b33a23d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:24.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.526+0000 7fc9dd184700 1 -- 172.21.15.46:0/1812607486 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d4005c10 con 0x7fc9d80c2450 2024-01-27T01:07:24.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.526+0000 7fc9dd184700 1 -- 172.21.15.46:0/1812607486 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d4005d70 con 0x7fc9d80c2450 2024-01-27T01:07:24.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.526+0000 7fc9dd184700 1 -- 172.21.15.46:0/1812607486 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d4006260 con 0x7fc9d80c2450 2024-01-27T01:07:24.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.527+0000 7fc9e03ea700 1 -- 172.21.15.46:0/1812607486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d80c2450 msgr2=0x7fc9d80cdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:24.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.527+0000 7fc9e03ea700 1 --2- 172.21.15.46:0/1812607486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d80c2450 0x7fc9d80cdbb0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fc9d80c7cc0 tx=0x7fc9d4009220 comp rx=0 tx=0).stop 2024-01-27T01:07:24.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.527+0000 7fc9e03ea700 1 -- 172.21.15.46:0/1812607486 shutdown_connections 2024-01-27T01:07:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.527+0000 7fc9e03ea700 1 --2- 172.21.15.46:0/1812607486 >> v2:172.21.15.106:3300/0 conn(0x7fc9d80cc840 0x7fc9d81601d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.527+0000 7fc9e03ea700 1 --2- 172.21.15.46:0/1812607486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d80c2450 0x7fc9d80cdbb0 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.527+0000 7fc9e03ea700 1 -- 172.21.15.46:0/1812607486 >> 172.21.15.46:0/1812607486 conn(0x7fc9d8069220 msgr2=0x7fc9d80cd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.527+0000 7fc9e03ea700 1 -- 172.21.15.46:0/1812607486 shutdown_connections 2024-01-27T01:07:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.527+0000 7fc9e03ea700 1 -- 172.21.15.46:0/1812607486 wait complete. 2024-01-27T01:07:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.528+0000 7fc9e03ea700 1 Processor -- start 2024-01-27T01:07:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.528+0000 7fc9e03ea700 1 -- start start 2024-01-27T01:07:24.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.529+0000 7fc9e03ea700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d80695f0 0x7fc9d8148050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:24.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.529+0000 7fc9e03ea700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9d80c2450 0x7fc9d80c4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:24.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.529+0000 7fc9e03ea700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9d80cc840 0x7fc9d80c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:24.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.529+0000 7fc9e03ea700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d80c71d0 con 0x7fc9d80695f0 2024-01-27T01:07:24.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.529+0000 7fc9e03ea700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d8148590 con 0x7fc9d80c2450 2024-01-27T01:07:24.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.529+0000 7fc9e03ea700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d8156f20 con 0x7fc9d80cc840 2024-01-27T01:07:24.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.529+0000 7fc9de186700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d80695f0 0x7fc9d8148050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:24.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.529+0000 7fc9de186700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d80695f0 0x7fc9d8148050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42200/0 (socket says 172.21.15.46:42200) 2024-01-27T01:07:24.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.529+0000 7fc9de186700 1 -- 172.21.15.46:0/1538787693 learned_addr learned my addr 172.21.15.46:0/1538787693 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:24.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.529+0000 7fc9de987700 1 --2- 172.21.15.46:0/1538787693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9d80cc840 0x7fc9d80c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:24.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.529+0000 7fc9de186700 1 -- 172.21.15.46:0/1538787693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9d80cc840 msgr2=0x7fc9d80c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:24.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.529+0000 7fc9dd985700 1 --2- 172.21.15.46:0/1538787693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9d80c2450 0x7fc9d80c4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:24.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.529+0000 7fc9de186700 1 --2- 172.21.15.46:0/1538787693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9d80cc840 0x7fc9d80c61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.529+0000 7fc9de186700 1 -- 172.21.15.46:0/1538787693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9d80c2450 msgr2=0x7fc9d80c4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:24.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.529+0000 7fc9de186700 1 --2- 172.21.15.46:0/1538787693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9d80c2450 0x7fc9d80c4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.529+0000 7fc9de186700 1 -- 172.21.15.46:0/1538787693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d4009480 con 0x7fc9d80695f0 2024-01-27T01:07:24.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.530+0000 7fc9de186700 1 --2- 172.21.15.46:0/1538787693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d80695f0 0x7fc9d8148050 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4006360 tx=0x7fc9d4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:24.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.530+0000 7fc9ce7fc700 1 -- 172.21.15.46:0/1538787693 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d400bb70 con 0x7fc9d80695f0 2024-01-27T01:07:24.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.530+0000 7fc9ce7fc700 1 -- 172.21.15.46:0/1538787693 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d400bcd0 con 0x7fc9d80695f0 2024-01-27T01:07:24.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.530+0000 7fc9ce7fc700 1 -- 172.21.15.46:0/1538787693 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d401b640 con 0x7fc9d80695f0 2024-01-27T01:07:24.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.530+0000 7fc9e03ea700 1 -- 172.21.15.46:0/1538787693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d8157200 con 0x7fc9d80695f0 2024-01-27T01:07:24.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.530+0000 7fc9e03ea700 1 -- 172.21.15.46:0/1538787693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d81577c0 con 0x7fc9d80695f0 2024-01-27T01:07:24.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.531+0000 7fc9e03ea700 1 -- 172.21.15.46:0/1538787693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9d804c710 con 0x7fc9d80695f0 2024-01-27T01:07:24.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.531+0000 7fc9ce7fc700 1 -- 172.21.15.46:0/1538787693 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fc9d401ba70 con 0x7fc9d80695f0 2024-01-27T01:07:24.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.532+0000 7fc9ce7fc700 1 --2- 172.21.15.46:0/1538787693 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc9bc067170 0x7fc9bc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:24.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.532+0000 7fc9ce7fc700 1 -- 172.21.15.46:0/1538787693 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5665+0+0 (secure 0 0 0) 0x7fc9d4086220 con 0x7fc9d80695f0 2024-01-27T01:07:24.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.532+0000 7fc9dd985700 1 --2- 172.21.15.46:0/1538787693 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc9bc067170 0x7fc9bc069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:24.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.532+0000 7fc9dd985700 1 --2- 172.21.15.46:0/1538787693 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc9bc067170 0x7fc9bc069620 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8159650 tx=0x7fc9c8009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:24.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.536+0000 7fc9ce7fc700 1 -- 172.21.15.46:0/1538787693 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc9d4053220 con 0x7fc9d80695f0 2024-01-27T01:07:24.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.741+0000 7fc9e03ea700 1 -- 172.21.15.46:0/1538787693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9d80686c0 con 0x7fc9d80695f0 2024-01-27T01:07:24.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.742+0000 7fc9ce7fc700 1 -- 172.21.15.46:0/1538787693 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v239) v1 ==== 74+0+13105 (secure 0 0 0) 0x7fc9d4056840 con 0x7fc9d80695f0 2024-01-27T01:07:24.742 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:24.742 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":239,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:07:23.926695+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:07:24.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.744+0000 7fc9e03ea700 1 -- 172.21.15.46:0/1538787693 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc9bc067170 msgr2=0x7fc9bc069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:24.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.744+0000 7fc9e03ea700 1 --2- 172.21.15.46:0/1538787693 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc9bc067170 0x7fc9bc069620 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8159650 tx=0x7fc9c8009040 comp rx=0 tx=0).stop 2024-01-27T01:07:24.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.744+0000 7fc9e03ea700 1 -- 172.21.15.46:0/1538787693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d80695f0 msgr2=0x7fc9d8148050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:24.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.744+0000 7fc9e03ea700 1 --2- 172.21.15.46:0/1538787693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d80695f0 0x7fc9d8148050 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4006360 tx=0x7fc9d4016000 comp rx=0 tx=0).stop 2024-01-27T01:07:24.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.744+0000 7fc9e03ea700 1 -- 172.21.15.46:0/1538787693 shutdown_connections 2024-01-27T01:07:24.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.744+0000 7fc9e03ea700 1 --2- 172.21.15.46:0/1538787693 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc9bc067170 0x7fc9bc069620 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.744+0000 7fc9e03ea700 1 --2- 172.21.15.46:0/1538787693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d80695f0 0x7fc9d8148050 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.744+0000 7fc9e03ea700 1 --2- 172.21.15.46:0/1538787693 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9d80cc840 0x7fc9d80c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.744+0000 7fc9e03ea700 1 --2- 172.21.15.46:0/1538787693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9d80c2450 0x7fc9d80c4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.744+0000 7fc9e03ea700 1 -- 172.21.15.46:0/1538787693 >> 172.21.15.46:0/1538787693 conn(0x7fc9d8069220 msgr2=0x7fc9d8067930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:24.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.744+0000 7fc9e03ea700 1 -- 172.21.15.46:0/1538787693 shutdown_connections 2024-01-27T01:07:24.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.745+0000 7fc9e03ea700 1 -- 172.21.15.46:0/1538787693 wait complete. 2024-01-27T01:07:24.758 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-27T01:07:24.758 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-27T01:07:24.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.893+0000 7fbdfcfd3700 1 Processor -- start 2024-01-27T01:07:24.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.893+0000 7fbdfcfd3700 1 -- start start 2024-01-27T01:07:24.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.894+0000 7fbdfcfd3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbdf8068ed0 0x7fbdf806a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:24.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.894+0000 7fbdfcfd3700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf80a9a80 con 0x7fbdf8068b00 2024-01-27T01:07:24.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.894+0000 7fbdfcfd3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf815a680 con 0x7fbdf806add0 2024-01-27T01:07:24.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.894+0000 7fbdfcfd3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf8069320 con 0x7fbdf8068ed0 2024-01-27T01:07:24.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.894+0000 7fbdf5d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbdf8068ed0 0x7fbdf806a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:24.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.894+0000 7fbdf5d9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbdf8068ed0 0x7fbdf806a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42204/0 (socket says 172.21.15.46:42204) 2024-01-27T01:07:24.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.894+0000 7fbdf5d9b700 1 -- 172.21.15.46:0/1733852518 learned_addr learned my addr 172.21.15.46:0/1733852518 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:24.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.895+0000 7fbdf5d9b700 1 -- 172.21.15.46:0/1733852518 >> v1:172.21.15.106:6789/0 conn(0x7fbdf806add0 legacy=0x7fbdf80a9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:24.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.895+0000 7fbdf5d9b700 1 -- 172.21.15.46:0/1733852518 >> v1:172.21.15.46:6789/0 conn(0x7fbdf8068b00 legacy=0x7fbdf8159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:24.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.895+0000 7fbdf5d9b700 1 -- 172.21.15.46:0/1733852518 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf815a960 con 0x7fbdf8068ed0 2024-01-27T01:07:24.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.895+0000 7fbdf5d9b700 1 --2- 172.21.15.46:0/1733852518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf8068ed0 0x7fbdf806a660 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fbdec0099b0 tx=0x7fbdec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=435c57eeb9b1c222 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:24.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.895+0000 7fbdf559a700 1 -- 172.21.15.46:0/1733852518 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdec00b860 con 0x7fbdf8068ed0 2024-01-27T01:07:24.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.895+0000 7fbdf559a700 1 -- 172.21.15.46:0/1733852518 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdec00b9c0 con 0x7fbdf8068ed0 2024-01-27T01:07:24.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.895+0000 7fbdf559a700 1 -- 172.21.15.46:0/1733852518 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdec00bd60 con 0x7fbdf8068ed0 2024-01-27T01:07:24.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.896+0000 7fbdfcfd3700 1 -- 172.21.15.46:0/1733852518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf8068ed0 msgr2=0x7fbdf806a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:24.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.896+0000 7fbdfcfd3700 1 --2- 172.21.15.46:0/1733852518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf8068ed0 0x7fbdf806a660 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fbdec0099b0 tx=0x7fbdec009250 comp rx=0 tx=0).stop 2024-01-27T01:07:24.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.896+0000 7fbdfcfd3700 1 -- 172.21.15.46:0/1733852518 shutdown_connections 2024-01-27T01:07:24.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.896+0000 7fbdfcfd3700 1 --2- 172.21.15.46:0/1733852518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf8068ed0 0x7fbdf806a660 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.896+0000 7fbdfcfd3700 1 -- 172.21.15.46:0/1733852518 >> 172.21.15.46:0/1733852518 conn(0x7fbdf80ca100 msgr2=0x7fbdf806e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:24.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.896+0000 7fbdfcfd3700 1 -- 172.21.15.46:0/1733852518 shutdown_connections 2024-01-27T01:07:24.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.896+0000 7fbdfcfd3700 1 -- 172.21.15.46:0/1733852518 wait complete. 2024-01-27T01:07:24.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.897+0000 7fbdfcfd3700 1 Processor -- start 2024-01-27T01:07:24.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.897+0000 7fbdfcfd3700 1 -- start start 2024-01-27T01:07:24.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.898+0000 7fbdfcfd3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbdf8068b00 0x7fbdf80c3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:24.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.898+0000 7fbdfcfd3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbdf8068ed0 0x7fbdf8149a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:24.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.898+0000 7fbdfcfd3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf806add0 0x7fbdf80ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:24.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.898+0000 7fbdfcfd3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf80adbb0 con 0x7fbdf806add0 2024-01-27T01:07:24.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.898+0000 7fbdfcfd3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf80acd20 con 0x7fbdf8068ed0 2024-01-27T01:07:24.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.898+0000 7fbdfcfd3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf80b2610 con 0x7fbdf8068b00 2024-01-27T01:07:24.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.898+0000 7fbdf6d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf806add0 0x7fbdf80ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:24.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.898+0000 7fbdf6d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf806add0 0x7fbdf80ac7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42206/0 (socket says 172.21.15.46:42206) 2024-01-27T01:07:24.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.898+0000 7fbdf6d9d700 1 -- 172.21.15.46:0/2031533336 learned_addr learned my addr 172.21.15.46:0/2031533336 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:24.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.898+0000 7fbdf5d9b700 1 --2- 172.21.15.46:0/2031533336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbdf8068ed0 0x7fbdf8149a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:24.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.898+0000 7fbdf659c700 1 --2- 172.21.15.46:0/2031533336 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbdf8068b00 0x7fbdf80c3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:24.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.898+0000 7fbdf6d9d700 1 -- 172.21.15.46:0/2031533336 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbdf8068b00 msgr2=0x7fbdf80c3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:24.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.898+0000 7fbdf6d9d700 1 --2- 172.21.15.46:0/2031533336 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbdf8068b00 0x7fbdf80c3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.898+0000 7fbdf6d9d700 1 -- 172.21.15.46:0/2031533336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbdf8068ed0 msgr2=0x7fbdf8149a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:24.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.898+0000 7fbdf6d9d700 1 --2- 172.21.15.46:0/2031533336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbdf8068ed0 0x7fbdf8149a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:24.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.898+0000 7fbdf6d9d700 1 -- 172.21.15.46:0/2031533336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec0094f0 con 0x7fbdf806add0 2024-01-27T01:07:24.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.898+0000 7fbdf6d9d700 1 --2- 172.21.15.46:0/2031533336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf806add0 0x7fbdf80ac7e0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7fbdf000c910 tx=0x7fbdf0001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:24.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.899+0000 7fbde67fc700 1 -- 172.21.15.46:0/2031533336 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf000a7a0 con 0x7fbdf806add0 2024-01-27T01:07:24.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.899+0000 7fbde67fc700 1 -- 172.21.15.46:0/2031533336 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf000a900 con 0x7fbdf806add0 2024-01-27T01:07:24.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.899+0000 7fbdfcfd3700 1 -- 172.21.15.46:0/2031533336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf80acfa0 con 0x7fbdf806add0 2024-01-27T01:07:24.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.899+0000 7fbde67fc700 1 -- 172.21.15.46:0/2031533336 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf000acc0 con 0x7fbdf806add0 2024-01-27T01:07:24.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.899+0000 7fbdfcfd3700 1 -- 172.21.15.46:0/2031533336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf80ad560 con 0x7fbdf806add0 2024-01-27T01:07:24.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.900+0000 7fbdfcfd3700 1 -- 172.21.15.46:0/2031533336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf815ced0 con 0x7fbdf806add0 2024-01-27T01:07:24.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.900+0000 7fbde67fc700 1 -- 172.21.15.46:0/2031533336 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fbdf0012630 con 0x7fbdf806add0 2024-01-27T01:07:24.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.901+0000 7fbde67fc700 1 --2- 172.21.15.46:0/2031533336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbdd8067170 0x7fbdd8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:24.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.901+0000 7fbde67fc700 1 -- 172.21.15.46:0/2031533336 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5665+0+0 (secure 0 0 0) 0x7fbdf0084b20 con 0x7fbdf806add0 2024-01-27T01:07:24.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.901+0000 7fbdf659c700 1 --2- 172.21.15.46:0/2031533336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbdd8067170 0x7fbdd8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:24.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.901+0000 7fbdf659c700 1 --2- 172.21.15.46:0/2031533336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbdd8067170 0x7fbdd8069620 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbde8003770 tx=0x7fbde8009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:24.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:24.905+0000 7fbde67fc700 1 -- 172.21.15.46:0/2031533336 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbdf0051b20 con 0x7fbdf806add0 2024-01-27T01:07:25.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.110+0000 7fbdfcfd3700 1 -- 172.21.15.46:0/2031533336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbdf804c710 con 0x7fbdf806add0 2024-01-27T01:07:25.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.110+0000 7fbde67fc700 1 -- 172.21.15.46:0/2031533336 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v239) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbdf0055140 con 0x7fbdf806add0 2024-01-27T01:07:25.110 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-27T01:07:25.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.113+0000 7fbdfcfd3700 1 -- 172.21.15.46:0/2031533336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbdd8067170 msgr2=0x7fbdd8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:25.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.113+0000 7fbdfcfd3700 1 --2- 172.21.15.46:0/2031533336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbdd8067170 0x7fbdd8069620 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbde8003770 tx=0x7fbde8009040 comp rx=0 tx=0).stop 2024-01-27T01:07:25.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.113+0000 7fbdfcfd3700 1 -- 172.21.15.46:0/2031533336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf806add0 msgr2=0x7fbdf80ac7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:25.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.113+0000 7fbdfcfd3700 1 --2- 172.21.15.46:0/2031533336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf806add0 0x7fbdf80ac7e0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7fbdf000c910 tx=0x7fbdf0001710 comp rx=0 tx=0).stop 2024-01-27T01:07:25.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.113+0000 7fbdfcfd3700 1 -- 172.21.15.46:0/2031533336 shutdown_connections 2024-01-27T01:07:25.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.113+0000 7fbdfcfd3700 1 --2- 172.21.15.46:0/2031533336 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbdd8067170 0x7fbdd8069620 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.113+0000 7fbdfcfd3700 1 --2- 172.21.15.46:0/2031533336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf806add0 0x7fbdf80ac7e0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.113+0000 7fbdfcfd3700 1 --2- 172.21.15.46:0/2031533336 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbdf8068b00 0x7fbdf80c3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.113+0000 7fbdfcfd3700 1 --2- 172.21.15.46:0/2031533336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbdf8068ed0 0x7fbdf8149a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.113+0000 7fbdfcfd3700 1 -- 172.21.15.46:0/2031533336 >> 172.21.15.46:0/2031533336 conn(0x7fbdf80ca100 msgr2=0x7fbdf80caf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:25.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.113+0000 7fbdfcfd3700 1 -- 172.21.15.46:0/2031533336 shutdown_connections 2024-01-27T01:07:25.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.113+0000 7fbdfcfd3700 1 -- 172.21.15.46:0/2031533336 wait complete. 2024-01-27T01:07:25.126 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-27T01:07:25.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.258+0000 7f1ade749700 1 Processor -- start 2024-01-27T01:07:25.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.258+0000 7f1ade749700 1 -- start start 2024-01-27T01:07:25.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.259+0000 7f1ade749700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1ad8153ec0 0x7f1ad8154290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:25.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.259+0000 7f1ade749700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ad80b22d0 0x7f1ad80b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:25.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.259+0000 7f1ade749700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad815b140 con 0x7f1ad8157630 2024-01-27T01:07:25.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.259+0000 7f1ade749700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad815a5c0 con 0x7f1ad80b22d0 2024-01-27T01:07:25.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.259+0000 7f1ade749700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad815a8a0 con 0x7f1ad8153ec0 2024-01-27T01:07:25.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.259+0000 7f1adcce6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ad80b22d0 0x7f1ad80b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:25.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.259+0000 7f1adcce6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ad80b22d0 0x7f1ad80b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42216/0 (socket says 172.21.15.46:42216) 2024-01-27T01:07:25.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.259+0000 7f1adcce6700 1 -- 172.21.15.46:0/1208801316 learned_addr learned my addr 172.21.15.46:0/1208801316 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:25.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.259+0000 7f1ad7fff700 1 --2- 172.21.15.46:0/1208801316 >> v2:172.21.15.106:3300/0 conn(0x7f1ad8153ec0 0x7f1ad8154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:25.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.259+0000 7f1adcce6700 1 -- 172.21.15.46:0/1208801316 >> v2:172.21.15.106:3300/0 conn(0x7f1ad8153ec0 msgr2=0x7f1ad8154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:25.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.259+0000 7f1adcce6700 1 --2- 172.21.15.46:0/1208801316 >> v2:172.21.15.106:3300/0 conn(0x7f1ad8153ec0 0x7f1ad8154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.259+0000 7f1adcce6700 1 -- 172.21.15.46:0/1208801316 >> v1:172.21.15.106:6789/0 conn(0x7f1ad8157630 legacy=0x7f1ad80b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:25.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.259+0000 7f1adcce6700 1 -- 172.21.15.46:0/1208801316 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad815ab80 con 0x7f1ad80b22d0 2024-01-27T01:07:25.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.260+0000 7f1adcce6700 1 --2- 172.21.15.46:0/1208801316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad80b22d0 0x7f1ad80b04f0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8064990 tx=0x7f1ad0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32271d9c037619d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:25.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.260+0000 7f1ad6ffd700 1 -- 172.21.15.46:0/1208801316 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad000e070 con 0x7f1ad80b22d0 2024-01-27T01:07:25.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.260+0000 7f1ad6ffd700 1 -- 172.21.15.46:0/1208801316 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad000ba20 con 0x7f1ad80b22d0 2024-01-27T01:07:25.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.260+0000 7f1ad6ffd700 1 -- 172.21.15.46:0/1208801316 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad00092e0 con 0x7f1ad80b22d0 2024-01-27T01:07:25.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.260+0000 7f1ade749700 1 -- 172.21.15.46:0/1208801316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad80b22d0 msgr2=0x7f1ad80b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:25.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.260+0000 7f1ade749700 1 --2- 172.21.15.46:0/1208801316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad80b22d0 0x7f1ad80b04f0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8064990 tx=0x7f1ad0009840 comp rx=0 tx=0).stop 2024-01-27T01:07:25.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.260+0000 7f1ade749700 1 -- 172.21.15.46:0/1208801316 shutdown_connections 2024-01-27T01:07:25.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.260+0000 7f1ade749700 1 --2- 172.21.15.46:0/1208801316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad80b22d0 0x7f1ad80b04f0 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.260+0000 7f1ade749700 1 --2- 172.21.15.46:0/1208801316 >> v2:172.21.15.106:3300/0 conn(0x7f1ad8153ec0 0x7f1ad8154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.260+0000 7f1ade749700 1 -- 172.21.15.46:0/1208801316 >> 172.21.15.46:0/1208801316 conn(0x7f1ad8069510 msgr2=0x7f1ad80aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:25.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.260+0000 7f1ade749700 1 -- 172.21.15.46:0/1208801316 shutdown_connections 2024-01-27T01:07:25.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.260+0000 7f1ade749700 1 -- 172.21.15.46:0/1208801316 wait complete. 2024-01-27T01:07:25.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.262+0000 7f1ade749700 1 Processor -- start 2024-01-27T01:07:25.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.262+0000 7f1ade749700 1 -- start start 2024-01-27T01:07:25.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1ade749700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad80b22d0 0x7f1ad80ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:25.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1ade749700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ad8153ec0 0x7f1ad8140aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:25.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1ade749700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ad8157630 0x7f1ad8140fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:25.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1ade749700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad815ccf0 con 0x7f1ad80b22d0 2024-01-27T01:07:25.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1ade749700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80cadb0 con 0x7f1ad8157630 2024-01-27T01:07:25.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1ade749700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad815ca10 con 0x7f1ad8153ec0 2024-01-27T01:07:25.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1ad7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad80b22d0 0x7f1ad80ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:25.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1ad7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad80b22d0 0x7f1ad80ccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42228/0 (socket says 172.21.15.46:42228) 2024-01-27T01:07:25.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1ad7fff700 1 -- 172.21.15.46:0/622570440 learned_addr learned my addr 172.21.15.46:0/622570440 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:25.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1adcce6700 1 --2- 172.21.15.46:0/622570440 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ad8157630 0x7f1ad8140fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:25.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1ad77fe700 1 --2- 172.21.15.46:0/622570440 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ad8153ec0 0x7f1ad8140aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:25.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1ad7fff700 1 -- 172.21.15.46:0/622570440 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ad8153ec0 msgr2=0x7f1ad8140aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:25.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1ad7fff700 1 --2- 172.21.15.46:0/622570440 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ad8153ec0 0x7f1ad8140aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1ad7fff700 1 -- 172.21.15.46:0/622570440 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ad8157630 msgr2=0x7f1ad8140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:25.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1ad7fff700 1 --2- 172.21.15.46:0/622570440 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ad8157630 0x7f1ad8140fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1ad7fff700 1 -- 172.21.15.46:0/622570440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad0009ae0 con 0x7f1ad80b22d0 2024-01-27T01:07:25.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1ad7fff700 1 --2- 172.21.15.46:0/622570440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad80b22d0 0x7f1ad80ccba0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800e930 tx=0x7f1ac80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:25.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.263+0000 7f1acffff700 1 -- 172.21.15.46:0/622570440 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac800eed0 con 0x7f1ad80b22d0 2024-01-27T01:07:25.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.264+0000 7f1acffff700 1 -- 172.21.15.46:0/622570440 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac800c850 con 0x7f1ad80b22d0 2024-01-27T01:07:25.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.264+0000 7f1ade749700 1 -- 172.21.15.46:0/622570440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad80ca760 con 0x7f1ad80b22d0 2024-01-27T01:07:25.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.264+0000 7f1acffff700 1 -- 172.21.15.46:0/622570440 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac800cbf0 con 0x7f1ad80b22d0 2024-01-27T01:07:25.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.264+0000 7f1ade749700 1 -- 172.21.15.46:0/622570440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad8168a20 con 0x7f1ad80b22d0 2024-01-27T01:07:25.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.265+0000 7f1ade749700 1 -- 172.21.15.46:0/622570440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad80cb090 con 0x7f1ad80b22d0 2024-01-27T01:07:25.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.265+0000 7f1acffff700 1 -- 172.21.15.46:0/622570440 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f1ac8013470 con 0x7f1ad80b22d0 2024-01-27T01:07:25.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.265+0000 7f1acffff700 1 --2- 172.21.15.46:0/622570440 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ab8067170 0x7f1ab8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:25.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.266+0000 7f1acffff700 1 -- 172.21.15.46:0/622570440 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5665+0+0 (secure 0 0 0) 0x7f1ac8084aa0 con 0x7f1ad80b22d0 2024-01-27T01:07:25.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.266+0000 7f1ad77fe700 1 --2- 172.21.15.46:0/622570440 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ab8067170 0x7f1ab8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:25.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.266+0000 7f1ad77fe700 1 --2- 172.21.15.46:0/622570440 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ab8067170 0x7f1ab8069620 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1ad815d0b0 tx=0x7f1ac0009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:25.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.270+0000 7f1acffff700 1 -- 172.21.15.46:0/622570440 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1ac8051aa0 con 0x7f1ad80b22d0 2024-01-27T01:07:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.478+0000 7f1ade749700 1 -- 172.21.15.46:0/622570440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1ad8157300 con 0x7f1ad80b22d0 2024-01-27T01:07:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.478+0000 7f1acffff700 1 -- 172.21.15.46:0/622570440 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v239) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1ac80550c0 con 0x7f1ad80b22d0 2024-01-27T01:07:25.479 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:25.479 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-27T01:07:25.479 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:25.479 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-01-27T01:07:25.479 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-01-27T01:07:25.479 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:25.479 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-27T01:07:25.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.480+0000 7f1ade749700 1 -- 172.21.15.46:0/622570440 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ab8067170 msgr2=0x7f1ab8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:25.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.480+0000 7f1ade749700 1 --2- 172.21.15.46:0/622570440 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ab8067170 0x7f1ab8069620 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1ad815d0b0 tx=0x7f1ac0009040 comp rx=0 tx=0).stop 2024-01-27T01:07:25.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.480+0000 7f1ade749700 1 -- 172.21.15.46:0/622570440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad80b22d0 msgr2=0x7f1ad80ccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:25.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.480+0000 7f1ade749700 1 --2- 172.21.15.46:0/622570440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad80b22d0 0x7f1ad80ccba0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800e930 tx=0x7f1ac80091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.480+0000 7f1ade749700 1 -- 172.21.15.46:0/622570440 shutdown_connections 2024-01-27T01:07:25.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.480+0000 7f1ade749700 1 --2- 172.21.15.46:0/622570440 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ab8067170 0x7f1ab8069620 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.480+0000 7f1ade749700 1 --2- 172.21.15.46:0/622570440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad80b22d0 0x7f1ad80ccba0 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.480+0000 7f1ade749700 1 --2- 172.21.15.46:0/622570440 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ad8157630 0x7f1ad8140fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.480+0000 7f1ade749700 1 --2- 172.21.15.46:0/622570440 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ad8153ec0 0x7f1ad8140aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.480+0000 7f1ade749700 1 -- 172.21.15.46:0/622570440 >> 172.21.15.46:0/622570440 conn(0x7f1ad8069510 msgr2=0x7f1ad80b03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:25.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.481+0000 7f1ade749700 1 -- 172.21.15.46:0/622570440 shutdown_connections 2024-01-27T01:07:25.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.481+0000 7f1ade749700 1 -- 172.21.15.46:0/622570440 wait complete. 2024-01-27T01:07:25.494 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-27T01:07:25.494 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-27T01:07:25.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.629+0000 7faf77a9c700 1 Processor -- start 2024-01-27T01:07:25.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.630+0000 7faf77a9c700 1 -- start start 2024-01-27T01:07:25.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.630+0000 7faf77a9c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf70067af0 0x7faf70067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:25.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.630+0000 7faf77a9c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faf700b1320 0x7faf7015d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:25.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.630+0000 7faf77a9c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf700b1780 con 0x7faf7006fc90 2024-01-27T01:07:25.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.630+0000 7faf77a9c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7006f800 con 0x7faf700b1320 2024-01-27T01:07:25.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.630+0000 7faf77a9c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7006ec30 con 0x7faf70067af0 2024-01-27T01:07:25.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.631+0000 7faf76039700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faf700b1320 0x7faf7015d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:25.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.631+0000 7faf76039700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faf700b1320 0x7faf7015d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42238/0 (socket says 172.21.15.46:42238) 2024-01-27T01:07:25.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.631+0000 7faf76039700 1 -- 172.21.15.46:0/260770151 learned_addr learned my addr 172.21.15.46:0/260770151 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:25.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.631+0000 7faf75838700 1 --2- 172.21.15.46:0/260770151 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf70067af0 0x7faf70067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:25.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.631+0000 7faf76039700 1 -- 172.21.15.46:0/260770151 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf70067af0 msgr2=0x7faf70067ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:25.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.631+0000 7faf76039700 1 --2- 172.21.15.46:0/260770151 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf70067af0 0x7faf70067ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.631+0000 7faf76039700 1 -- 172.21.15.46:0/260770151 >> v1:172.21.15.46:6789/0 conn(0x7faf7006fc90 legacy=0x7faf700b2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:25.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.631+0000 7faf76039700 1 -- 172.21.15.46:0/260770151 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf700b32a0 con 0x7faf700b1320 2024-01-27T01:07:25.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.631+0000 7faf75838700 1 --2- 172.21.15.46:0/260770151 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf70067af0 0x7faf70067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:25.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.631+0000 7faf76039700 1 --2- 172.21.15.46:0/260770151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf700b1320 0x7faf7015d7f0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7faf68009f20 tx=0x7faf68009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f757c37ee86c4f8f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:25.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.631+0000 7faf74836700 1 -- 172.21.15.46:0/260770151 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6800e070 con 0x7faf700b1320 2024-01-27T01:07:25.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.631+0000 7faf74836700 1 -- 172.21.15.46:0/260770151 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6800ba80 con 0x7faf700b1320 2024-01-27T01:07:25.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.631+0000 7faf74836700 1 -- 172.21.15.46:0/260770151 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf680092e0 con 0x7faf700b1320 2024-01-27T01:07:25.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.632+0000 7faf77a9c700 1 -- 172.21.15.46:0/260770151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf700b1320 msgr2=0x7faf7015d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:25.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.632+0000 7faf77a9c700 1 --2- 172.21.15.46:0/260770151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf700b1320 0x7faf7015d7f0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7faf68009f20 tx=0x7faf68009840 comp rx=0 tx=0).stop 2024-01-27T01:07:25.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.632+0000 7faf77a9c700 1 -- 172.21.15.46:0/260770151 shutdown_connections 2024-01-27T01:07:25.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.632+0000 7faf77a9c700 1 --2- 172.21.15.46:0/260770151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf700b1320 0x7faf7015d7f0 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.632+0000 7faf77a9c700 1 --2- 172.21.15.46:0/260770151 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf70067af0 0x7faf70067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.632+0000 7faf77a9c700 1 -- 172.21.15.46:0/260770151 >> 172.21.15.46:0/260770151 conn(0x7faf70069510 msgr2=0x7faf700aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:25.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.632+0000 7faf77a9c700 1 -- 172.21.15.46:0/260770151 shutdown_connections 2024-01-27T01:07:25.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.632+0000 7faf77a9c700 1 -- 172.21.15.46:0/260770151 wait complete. 2024-01-27T01:07:25.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.633+0000 7faf77a9c700 1 Processor -- start 2024-01-27T01:07:25.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.633+0000 7faf77a9c700 1 -- start start 2024-01-27T01:07:25.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.634+0000 7faf77a9c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf70067af0 0x7faf7015e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:25.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.634+0000 7faf77a9c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf7006fc90 0x7faf7015ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:25.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.634+0000 7faf77a9c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf700b1320 0x7faf701547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:25.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.634+0000 7faf77a9c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf70160130 con 0x7faf700b1320 2024-01-27T01:07:25.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.634+0000 7faf77a9c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7015f520 con 0x7faf70067af0 2024-01-27T01:07:25.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.634+0000 7faf77a9c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7015f800 con 0x7faf7006fc90 2024-01-27T01:07:25.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.634+0000 7faf76039700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf700b1320 0x7faf701547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:25.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.634+0000 7faf76039700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf700b1320 0x7faf701547d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42248/0 (socket says 172.21.15.46:42248) 2024-01-27T01:07:25.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.634+0000 7faf76039700 1 -- 172.21.15.46:0/2977975619 learned_addr learned my addr 172.21.15.46:0/2977975619 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:25.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.635+0000 7faf75838700 1 --2- 172.21.15.46:0/2977975619 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf70067af0 0x7faf7015e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:25.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.635+0000 7faf75037700 1 --2- 172.21.15.46:0/2977975619 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf7006fc90 0x7faf7015ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:25.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.635+0000 7faf76039700 1 -- 172.21.15.46:0/2977975619 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf7006fc90 msgr2=0x7faf7015ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:25.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.635+0000 7faf76039700 1 --2- 172.21.15.46:0/2977975619 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf7006fc90 0x7faf7015ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.635+0000 7faf76039700 1 -- 172.21.15.46:0/2977975619 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf70067af0 msgr2=0x7faf7015e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:25.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.635+0000 7faf76039700 1 --2- 172.21.15.46:0/2977975619 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf70067af0 0x7faf7015e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.635+0000 7faf76039700 1 -- 172.21.15.46:0/2977975619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf68009ae0 con 0x7faf700b1320 2024-01-27T01:07:25.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.635+0000 7faf76039700 1 --2- 172.21.15.46:0/2977975619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf700b1320 0x7faf701547d0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7faf700ce4b0 tx=0x7faf68018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:25.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.635+0000 7faf65ffb700 1 -- 172.21.15.46:0/2977975619 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf680032f0 con 0x7faf700b1320 2024-01-27T01:07:25.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.635+0000 7faf65ffb700 1 -- 172.21.15.46:0/2977975619 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf68007e30 con 0x7faf700b1320 2024-01-27T01:07:25.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.635+0000 7faf77a9c700 1 -- 172.21.15.46:0/2977975619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7015fae0 con 0x7faf700b1320 2024-01-27T01:07:25.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.635+0000 7faf65ffb700 1 -- 172.21.15.46:0/2977975619 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6801d650 con 0x7faf700b1320 2024-01-27T01:07:25.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.635+0000 7faf77a9c700 1 -- 172.21.15.46:0/2977975619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7015f230 con 0x7faf700b1320 2024-01-27T01:07:25.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.636+0000 7faf77a9c700 1 -- 172.21.15.46:0/2977975619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7004c710 con 0x7faf700b1320 2024-01-27T01:07:25.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.637+0000 7faf65ffb700 1 -- 172.21.15.46:0/2977975619 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7faf68003040 con 0x7faf700b1320 2024-01-27T01:07:25.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.637+0000 7faf65ffb700 1 --2- 172.21.15.46:0/2977975619 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faf54067170 0x7faf54069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:25.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.637+0000 7faf65ffb700 1 -- 172.21.15.46:0/2977975619 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5665+0+0 (secure 0 0 0) 0x7faf68086880 con 0x7faf700b1320 2024-01-27T01:07:25.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.637+0000 7faf75838700 1 --2- 172.21.15.46:0/2977975619 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faf54067170 0x7faf54069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:25.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.638+0000 7faf75838700 1 --2- 172.21.15.46:0/2977975619 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faf54067170 0x7faf54069620 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faf60006f00 tx=0x7faf60008040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:25.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.642+0000 7faf65ffb700 1 -- 172.21.15.46:0/2977975619 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faf680536d0 con 0x7faf700b1320 2024-01-27T01:07:25.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.849+0000 7faf77a9c700 1 -- 172.21.15.46:0/2977975619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7faf70159940 con 0x7faf700b1320 2024-01-27T01:07:25.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.924+0000 7faf65ffb700 1 -- 172.21.15.46:0/2977975619 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v240) v1 ==== 138+0+0 (secure 0 0 0) 0x7faf68056cf0 con 0x7faf700b1320 2024-01-27T01:07:25.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.929+0000 7faf77a9c700 1 -- 172.21.15.46:0/2977975619 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faf54067170 msgr2=0x7faf54069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:25.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.929+0000 7faf77a9c700 1 --2- 172.21.15.46:0/2977975619 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faf54067170 0x7faf54069620 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faf60006f00 tx=0x7faf60008040 comp rx=0 tx=0).stop 2024-01-27T01:07:25.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.929+0000 7faf77a9c700 1 -- 172.21.15.46:0/2977975619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf700b1320 msgr2=0x7faf701547d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:25.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.929+0000 7faf77a9c700 1 --2- 172.21.15.46:0/2977975619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf700b1320 0x7faf701547d0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7faf700ce4b0 tx=0x7faf68018000 comp rx=0 tx=0).stop 2024-01-27T01:07:25.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.929+0000 7faf77a9c700 1 -- 172.21.15.46:0/2977975619 shutdown_connections 2024-01-27T01:07:25.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.929+0000 7faf77a9c700 1 --2- 172.21.15.46:0/2977975619 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7faf54067170 0x7faf54069620 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.929+0000 7faf77a9c700 1 --2- 172.21.15.46:0/2977975619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf700b1320 0x7faf701547d0 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.929+0000 7faf77a9c700 1 --2- 172.21.15.46:0/2977975619 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf70067af0 0x7faf7015e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.929+0000 7faf77a9c700 1 --2- 172.21.15.46:0/2977975619 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf7006fc90 0x7faf7015ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:25.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.929+0000 7faf77a9c700 1 -- 172.21.15.46:0/2977975619 >> 172.21.15.46:0/2977975619 conn(0x7faf70069510 msgr2=0x7faf701591a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:25.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.930+0000 7faf77a9c700 1 -- 172.21.15.46:0/2977975619 shutdown_connections 2024-01-27T01:07:25.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:25.930+0000 7faf77a9c700 1 -- 172.21.15.46:0/2977975619 wait complete. 2024-01-27T01:07:25.932 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-01-27T01:07:25.945 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-27T01:07:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.084+0000 7f2abd503700 1 Processor -- start 2024-01-27T01:07:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.084+0000 7f2abd503700 1 -- start start 2024-01-27T01:07:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.085+0000 7f2abd503700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ab815dfd0 0x7f2ab80ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.085+0000 7f2abd503700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab815a200 con 0x7f2ab8159da0 2024-01-27T01:07:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.085+0000 7f2abd503700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab80b1a60 con 0x7f2ab81599d0 2024-01-27T01:07:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.085+0000 7f2abd503700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab80b11c0 con 0x7f2ab815dfd0 2024-01-27T01:07:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.085+0000 7f2ab67fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2ab8159da0 0x7f2ab815d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47338/0 (socket says 172.21.15.46:47338) 2024-01-27T01:07:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.085+0000 7f2ab67fc700 1 -- 172.21.15.46:0/1419543802 learned_addr learned my addr 172.21.15.46:0/1419543802 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.085+0000 7f2ab77fe700 1 --2- 172.21.15.46:0/1419543802 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ab815dfd0 0x7f2ab80ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.086+0000 7f2ab5ffb700 1 -- 172.21.15.46:0/1419543802 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675674689 0 0) 0x7f2ab815a200 con 0x7f2ab8159da0 2024-01-27T01:07:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.086+0000 7f2ab5ffb700 1 -- 172.21.15.46:0/1419543802 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa0003570 con 0x7f2ab8159da0 2024-01-27T01:07:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.086+0000 7f2ab5ffb700 1 -- 172.21.15.46:0/1419543802 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731564751 0 0) 0x7f2ab80b1a60 con 0x7f2ab81599d0 2024-01-27T01:07:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.086+0000 7f2ab5ffb700 1 -- 172.21.15.46:0/1419543802 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab815a200 con 0x7f2ab81599d0 2024-01-27T01:07:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.086+0000 7f2ab77fe700 1 -- 172.21.15.46:0/1419543802 >> v1:172.21.15.106:6789/0 conn(0x7f2ab81599d0 legacy=0x7f2ab80b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.086+0000 7f2ab5ffb700 1 -- 172.21.15.46:0/1419543802 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1618268521 0 0) 0x7f2aa0003570 con 0x7f2ab8159da0 2024-01-27T01:07:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.086+0000 7f2ab77fe700 1 -- 172.21.15.46:0/1419543802 >> v1:172.21.15.46:6789/0 conn(0x7f2ab8159da0 legacy=0x7f2ab815d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.086+0000 7f2ab77fe700 1 -- 172.21.15.46:0/1419543802 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab80b14a0 con 0x7f2ab815dfd0 2024-01-27T01:07:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.086+0000 7f2ab77fe700 1 --2- 172.21.15.46:0/1419543802 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ab815dfd0 0x7f2ab80ab1c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0009e90 tx=0x7f2ab0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a745a2507c32d58a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.086+0000 7f2ab5ffb700 1 -- 172.21.15.46:0/1419543802 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab000e070 con 0x7f2ab815dfd0 2024-01-27T01:07:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.086+0000 7f2ab5ffb700 1 -- 172.21.15.46:0/1419543802 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab000b870 con 0x7f2ab815dfd0 2024-01-27T01:07:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.086+0000 7f2ab5ffb700 1 -- 172.21.15.46:0/1419543802 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac001470 con 0x7f2ab815dfd0 2024-01-27T01:07:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.087+0000 7f2abd503700 1 -- 172.21.15.46:0/1419543802 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ab815dfd0 msgr2=0x7f2ab80ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.087+0000 7f2abd503700 1 --2- 172.21.15.46:0/1419543802 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ab815dfd0 0x7f2ab80ab1c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0009e90 tx=0x7f2ab0009780 comp rx=0 tx=0).stop 2024-01-27T01:07:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.087+0000 7f2abd503700 1 -- 172.21.15.46:0/1419543802 shutdown_connections 2024-01-27T01:07:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.087+0000 7f2abd503700 1 --2- 172.21.15.46:0/1419543802 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ab815dfd0 0x7f2ab80ab1c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.087+0000 7f2abd503700 1 -- 172.21.15.46:0/1419543802 >> 172.21.15.46:0/1419543802 conn(0x7f2ab806f840 msgr2=0x7f2ab8087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.087+0000 7f2abd503700 1 -- 172.21.15.46:0/1419543802 shutdown_connections 2024-01-27T01:07:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.087+0000 7f2abd503700 1 -- 172.21.15.46:0/1419543802 wait complete. 2024-01-27T01:07:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.088+0000 7f2abd503700 1 Processor -- start 2024-01-27T01:07:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.088+0000 7f2abd503700 1 -- start start 2024-01-27T01:07:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.089+0000 7f2abd503700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ab81599d0 0x7f2ab80cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.089+0000 7f2abd503700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab8159da0 0x7f2ab8156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.089+0000 7f2abd503700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ab815dfd0 0x7f2ab80ab760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.089+0000 7f2abd503700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab80cd070 con 0x7f2ab815dfd0 2024-01-27T01:07:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.089+0000 7f2abd503700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab80d0050 con 0x7f2ab8159da0 2024-01-27T01:07:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.089+0000 7f2abd503700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab80ad430 con 0x7f2ab81599d0 2024-01-27T01:07:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.089+0000 7f2ab77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ab815dfd0 0x7f2ab80ab760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.089+0000 7f2ab77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ab815dfd0 0x7f2ab80ab760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42254/0 (socket says 172.21.15.46:42254) 2024-01-27T01:07:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.089+0000 7f2ab77fe700 1 -- 172.21.15.46:0/3416507322 learned_addr learned my addr 172.21.15.46:0/3416507322 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.089+0000 7f2ab6ffd700 1 --2- 172.21.15.46:0/3416507322 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ab81599d0 0x7f2ab80cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.089+0000 7f2ab67fc700 1 --2- 172.21.15.46:0/3416507322 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab8159da0 0x7f2ab8156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.090+0000 7f2ab77fe700 1 -- 172.21.15.46:0/3416507322 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ab81599d0 msgr2=0x7f2ab80cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.090+0000 7f2ab77fe700 1 --2- 172.21.15.46:0/3416507322 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ab81599d0 0x7f2ab80cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.090+0000 7f2ab77fe700 1 -- 172.21.15.46:0/3416507322 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab8159da0 msgr2=0x7f2ab8156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.090+0000 7f2ab77fe700 1 --2- 172.21.15.46:0/3416507322 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab8159da0 0x7f2ab8156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.090+0000 7f2ab77fe700 1 -- 172.21.15.46:0/3416507322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab0009a20 con 0x7f2ab815dfd0 2024-01-27T01:07:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.090+0000 7f2ab77fe700 1 --2- 172.21.15.46:0/3416507322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ab815dfd0 0x7f2ab80ab760 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0009de0 tx=0x7f2ab0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.090+0000 7f2a9effd700 1 -- 172.21.15.46:0/3416507322 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0012c20 con 0x7f2ab815dfd0 2024-01-27T01:07:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.090+0000 7f2a9effd700 1 -- 172.21.15.46:0/3416507322 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab0012d80 con 0x7f2ab815dfd0 2024-01-27T01:07:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.090+0000 7f2abd503700 1 -- 172.21.15.46:0/3416507322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab80ad710 con 0x7f2ab815dfd0 2024-01-27T01:07:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.090+0000 7f2a9effd700 1 -- 172.21.15.46:0/3416507322 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab001e6a0 con 0x7f2ab815dfd0 2024-01-27T01:07:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.090+0000 7f2abd503700 1 -- 172.21.15.46:0/3416507322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab80adcd0 con 0x7f2ab815dfd0 2024-01-27T01:07:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.091+0000 7f2abd503700 1 -- 172.21.15.46:0/3416507322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab804c710 con 0x7f2ab815dfd0 2024-01-27T01:07:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.092+0000 7f2a9effd700 1 -- 172.21.15.46:0/3416507322 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f2ab001ead0 con 0x7f2ab815dfd0 2024-01-27T01:07:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.092+0000 7f2a9effd700 1 --2- 172.21.15.46:0/3416507322 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2aa4067170 0x7f2aa4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.092+0000 7f2a9effd700 1 -- 172.21.15.46:0/3416507322 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6035+0+0 (secure 0 0 0) 0x7f2ab00868e0 con 0x7f2ab815dfd0 2024-01-27T01:07:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.092+0000 7f2ab6ffd700 1 --2- 172.21.15.46:0/3416507322 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2aa4067170 0x7f2aa4069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.093+0000 7f2ab6ffd700 1 --2- 172.21.15.46:0/3416507322 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2aa4067170 0x7f2aa4069620 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2ab815c2f0 tx=0x7f2aa8007d40 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.097+0000 7f2a9effd700 1 -- 172.21.15.46:0/3416507322 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2ab00535c0 con 0x7f2ab815dfd0 2024-01-27T01:07:26.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.302+0000 7f2abd503700 1 -- 172.21.15.46:0/3416507322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2ab80b1780 con 0x7f2ab815dfd0 2024-01-27T01:07:26.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.931+0000 7f2a9effd700 1 -- 172.21.15.46:0/3416507322 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v241) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2ab0056be0 con 0x7f2ab815dfd0 2024-01-27T01:07:26.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.937+0000 7f2abd503700 1 -- 172.21.15.46:0/3416507322 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2aa4067170 msgr2=0x7f2aa4069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:26.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.937+0000 7f2abd503700 1 --2- 172.21.15.46:0/3416507322 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2aa4067170 0x7f2aa4069620 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2ab815c2f0 tx=0x7f2aa8007d40 comp rx=0 tx=0).stop 2024-01-27T01:07:26.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.937+0000 7f2abd503700 1 -- 172.21.15.46:0/3416507322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ab815dfd0 msgr2=0x7f2ab80ab760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:26.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.937+0000 7f2abd503700 1 --2- 172.21.15.46:0/3416507322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ab815dfd0 0x7f2ab80ab760 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0009de0 tx=0x7f2ab0017000 comp rx=0 tx=0).stop 2024-01-27T01:07:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.940+0000 7f2abd503700 1 -- 172.21.15.46:0/3416507322 shutdown_connections 2024-01-27T01:07:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.940+0000 7f2abd503700 1 --2- 172.21.15.46:0/3416507322 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2aa4067170 0x7f2aa4069620 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.940+0000 7f2abd503700 1 --2- 172.21.15.46:0/3416507322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ab815dfd0 0x7f2ab80ab760 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.940+0000 7f2abd503700 1 --2- 172.21.15.46:0/3416507322 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ab81599d0 0x7f2ab80cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.940+0000 7f2abd503700 1 --2- 172.21.15.46:0/3416507322 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab8159da0 0x7f2ab8156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:26.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.940+0000 7f2abd503700 1 -- 172.21.15.46:0/3416507322 >> 172.21.15.46:0/3416507322 conn(0x7f2ab806f840 msgr2=0x7f2ab80b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:26.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.944+0000 7f2abd503700 1 -- 172.21.15.46:0/3416507322 shutdown_connections 2024-01-27T01:07:26.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:26.946+0000 7f2abd503700 1 -- 172.21.15.46:0/3416507322 wait complete. 2024-01-27T01:07:26.951 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-01-27T01:07:26.967 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-27T01:07:27.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.105+0000 7fb70dbb7700 1 Processor -- start 2024-01-27T01:07:27.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.105+0000 7fb70dbb7700 1 -- start start 2024-01-27T01:07:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.106+0000 7fb70dbb7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb708141200 0x7fb7080c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.106+0000 7fb70dbb7700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb7080caf30 0x7fb7080cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.106+0000 7fb70dbb7700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7080ccc00 con 0x7fb7080c8da0 2024-01-27T01:07:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.106+0000 7fb70dbb7700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7080cbbf0 con 0x7fb708141200 2024-01-27T01:07:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.106+0000 7fb70dbb7700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7080cc310 con 0x7fb7080caf30 2024-01-27T01:07:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.106+0000 7fb706ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb708141200 0x7fb7080c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.106+0000 7fb706ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb708141200 0x7fb7080c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42268/0 (socket says 172.21.15.46:42268) 2024-01-27T01:07:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.106+0000 7fb706ffd700 1 -- 172.21.15.46:0/4216381825 learned_addr learned my addr 172.21.15.46:0/4216381825 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:27.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.106+0000 7fb707fff700 1 --2- 172.21.15.46:0/4216381825 >> v2:172.21.15.106:3300/0 conn(0x7fb7080caf30 0x7fb7080cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:27.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.107+0000 7fb706ffd700 1 -- 172.21.15.46:0/4216381825 >> v2:172.21.15.106:3300/0 conn(0x7fb7080caf30 msgr2=0x7fb7080cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:27.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.107+0000 7fb706ffd700 1 --2- 172.21.15.46:0/4216381825 >> v2:172.21.15.106:3300/0 conn(0x7fb7080caf30 0x7fb7080cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:27.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.107+0000 7fb706ffd700 1 -- 172.21.15.46:0/4216381825 >> v1:172.21.15.46:6789/0 conn(0x7fb7080c8da0 legacy=0x7fb708140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:27.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.107+0000 7fb706ffd700 1 -- 172.21.15.46:0/4216381825 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7080cc5b0 con 0x7fb708141200 2024-01-27T01:07:27.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.107+0000 7fb706ffd700 1 --2- 172.21.15.46:0/4216381825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb708141200 0x7fb7080c9170 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7fb7081416d0 tx=0x7fb6fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f417d8b4c2cd0fd6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:27.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.107+0000 7fb7067fc700 1 -- 172.21.15.46:0/4216381825 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc00b860 con 0x7fb708141200 2024-01-27T01:07:27.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.107+0000 7fb7067fc700 1 -- 172.21.15.46:0/4216381825 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6fc00b9c0 con 0x7fb708141200 2024-01-27T01:07:27.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.107+0000 7fb7067fc700 1 -- 172.21.15.46:0/4216381825 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc00bd60 con 0x7fb708141200 2024-01-27T01:07:27.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.107+0000 7fb70dbb7700 1 -- 172.21.15.46:0/4216381825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb708141200 msgr2=0x7fb7080c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:27.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.107+0000 7fb70dbb7700 1 --2- 172.21.15.46:0/4216381825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb708141200 0x7fb7080c9170 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7fb7081416d0 tx=0x7fb6fc009250 comp rx=0 tx=0).stop 2024-01-27T01:07:27.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.108+0000 7fb70dbb7700 1 -- 172.21.15.46:0/4216381825 shutdown_connections 2024-01-27T01:07:27.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.108+0000 7fb70dbb7700 1 --2- 172.21.15.46:0/4216381825 >> v2:172.21.15.106:3300/0 conn(0x7fb7080caf30 0x7fb7080cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:27.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.108+0000 7fb70dbb7700 1 --2- 172.21.15.46:0/4216381825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb708141200 0x7fb7080c9170 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:27.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.108+0000 7fb70dbb7700 1 -- 172.21.15.46:0/4216381825 >> 172.21.15.46:0/4216381825 conn(0x7fb70806bd90 msgr2=0x7fb70806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:27.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.108+0000 7fb70dbb7700 1 -- 172.21.15.46:0/4216381825 shutdown_connections 2024-01-27T01:07:27.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.108+0000 7fb70dbb7700 1 -- 172.21.15.46:0/4216381825 wait complete. 2024-01-27T01:07:27.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.109+0000 7fb70dbb7700 1 Processor -- start 2024-01-27T01:07:27.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.109+0000 7fb70dbb7700 1 -- start start 2024-01-27T01:07:27.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.110+0000 7fb70dbb7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7080c8da0 0x7fb70814a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:27.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.110+0000 7fb70dbb7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7080caf30 0x7fb708157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:27.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.110+0000 7fb70dbb7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb708141200 0x7fb708157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:27.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.110+0000 7fb70dbb7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7080b2640 con 0x7fb7080c8da0 2024-01-27T01:07:27.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.110+0000 7fb70dbb7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70815a7b0 con 0x7fb7080caf30 2024-01-27T01:07:27.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.110+0000 7fb70dbb7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb708159e80 con 0x7fb708141200 2024-01-27T01:07:27.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.110+0000 7fb7077fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7080c8da0 0x7fb70814a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:27.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.110+0000 7fb7077fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7080c8da0 0x7fb70814a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42284/0 (socket says 172.21.15.46:42284) 2024-01-27T01:07:27.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.110+0000 7fb7077fe700 1 -- 172.21.15.46:0/2907662614 learned_addr learned my addr 172.21.15.46:0/2907662614 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.110+0000 7fb707fff700 1 --2- 172.21.15.46:0/2907662614 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb708141200 0x7fb708157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.110+0000 7fb706ffd700 1 --2- 172.21.15.46:0/2907662614 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7080caf30 0x7fb708157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.110+0000 7fb7077fe700 1 -- 172.21.15.46:0/2907662614 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb708141200 msgr2=0x7fb708157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.110+0000 7fb7077fe700 1 --2- 172.21.15.46:0/2907662614 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb708141200 0x7fb708157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.110+0000 7fb7077fe700 1 -- 172.21.15.46:0/2907662614 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7080caf30 msgr2=0x7fb708157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.110+0000 7fb7077fe700 1 --2- 172.21.15.46:0/2907662614 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7080caf30 0x7fb708157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.110+0000 7fb7077fe700 1 -- 172.21.15.46:0/2907662614 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6fc0094f0 con 0x7fb7080c8da0 2024-01-27T01:07:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.110+0000 7fb7077fe700 1 --2- 172.21.15.46:0/2907662614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7080c8da0 0x7fb70814a160 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7fb6f800c590 tx=0x7fb6f800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.111+0000 7fb6ef7fe700 1 -- 172.21.15.46:0/2907662614 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8011590 con 0x7fb7080c8da0 2024-01-27T01:07:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.111+0000 7fb6ef7fe700 1 -- 172.21.15.46:0/2907662614 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f80116f0 con 0x7fb7080c8da0 2024-01-27T01:07:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.111+0000 7fb70dbb7700 1 -- 172.21.15.46:0/2907662614 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70815a160 con 0x7fb7080c8da0 2024-01-27T01:07:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.111+0000 7fb6ef7fe700 1 -- 172.21.15.46:0/2907662614 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8011ab0 con 0x7fb7080c8da0 2024-01-27T01:07:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.111+0000 7fb70dbb7700 1 -- 172.21.15.46:0/2907662614 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb708159b30 con 0x7fb7080c8da0 2024-01-27T01:07:27.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.112+0000 7fb70dbb7700 1 -- 172.21.15.46:0/2907662614 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb708070f70 con 0x7fb7080c8da0 2024-01-27T01:07:27.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.113+0000 7fb6ef7fe700 1 -- 172.21.15.46:0/2907662614 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fb6f801b3b0 con 0x7fb7080c8da0 2024-01-27T01:07:27.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.113+0000 7fb6ef7fe700 1 --2- 172.21.15.46:0/2907662614 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f4067170 0x7fb6f4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:27.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.113+0000 7fb6ef7fe700 1 -- 172.21.15.46:0/2907662614 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fb6f80840a0 con 0x7fb7080c8da0 2024-01-27T01:07:27.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.114+0000 7fb706ffd700 1 --2- 172.21.15.46:0/2907662614 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f4067170 0x7fb6f4069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:27.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.114+0000 7fb706ffd700 1 --2- 172.21.15.46:0/2907662614 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f4067170 0x7fb6f4069620 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb7081416d0 tx=0x7fb6fc017000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:27.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.118+0000 7fb6ef7fe700 1 -- 172.21.15.46:0/2907662614 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb6f8050db0 con 0x7fb7080c8da0 2024-01-27T01:07:27.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.373+0000 7fb70dbb7700 1 -- 172.21.15.46:0/2907662614 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb70804c710 con 0x7fb7080c8da0 2024-01-27T01:07:27.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.932+0000 7fb6ef7fe700 1 -- 172.21.15.46:0/2907662614 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 31 and data pool 32 v129) v1 ==== 174+0+0 (secure 0 0 0) 0x7fb6f80170c0 con 0x7fb7080c8da0 2024-01-27T01:07:27.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.936+0000 7fb70dbb7700 1 -- 172.21.15.46:0/2907662614 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f4067170 msgr2=0x7fb6f4069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:27.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.936+0000 7fb70dbb7700 1 --2- 172.21.15.46:0/2907662614 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f4067170 0x7fb6f4069620 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb7081416d0 tx=0x7fb6fc017000 comp rx=0 tx=0).stop 2024-01-27T01:07:27.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.936+0000 7fb70dbb7700 1 -- 172.21.15.46:0/2907662614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7080c8da0 msgr2=0x7fb70814a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:27.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.936+0000 7fb70dbb7700 1 --2- 172.21.15.46:0/2907662614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7080c8da0 0x7fb70814a160 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7fb6f800c590 tx=0x7fb6f800c950 comp rx=0 tx=0).stop 2024-01-27T01:07:27.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.937+0000 7fb70dbb7700 1 -- 172.21.15.46:0/2907662614 shutdown_connections 2024-01-27T01:07:27.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.937+0000 7fb70dbb7700 1 --2- 172.21.15.46:0/2907662614 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f4067170 0x7fb6f4069620 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb7081416d0 tx=0x7fb6fc017000 comp rx=0 tx=0).stop 2024-01-27T01:07:27.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.937+0000 7fb70dbb7700 1 --2- 172.21.15.46:0/2907662614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7080c8da0 0x7fb70814a160 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:27.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.937+0000 7fb70dbb7700 1 --2- 172.21.15.46:0/2907662614 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb708141200 0x7fb708157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:27.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.937+0000 7fb70dbb7700 1 --2- 172.21.15.46:0/2907662614 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7080caf30 0x7fb708157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:27.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.937+0000 7fb70dbb7700 1 -- 172.21.15.46:0/2907662614 >> 172.21.15.46:0/2907662614 conn(0x7fb70806bd90 msgr2=0x7fb70813f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:27.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.940+0000 7fb70dbb7700 1 -- 172.21.15.46:0/2907662614 shutdown_connections 2024-01-27T01:07:27.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:27.940+0000 7fb70dbb7700 1 -- 172.21.15.46:0/2907662614 wait complete. 2024-01-27T01:07:27.944 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 31 and data pool 32 2024-01-27T01:07:27.958 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:07:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.093+0000 7f9d12e88700 1 Processor -- start 2024-01-27T01:07:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.093+0000 7f9d12e88700 1 -- start start 2024-01-27T01:07:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.093+0000 7f9d12e88700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9d0c0ad070 0x7f9d0c0ae610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d12e88700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c0ae180 con 0x7f9d0c0cc1c0 2024-01-27T01:07:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d12e88700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c156c60 con 0x7f9d0c0cbdf0 2024-01-27T01:07:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d12e88700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d0c0aeb50 con 0x7f9d0c0ad070 2024-01-27T01:07:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d0bfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9d0c0cc1c0 0x7f9d0c0c98b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47360/0 (socket says 172.21.15.46:47360) 2024-01-27T01:07:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d0bfff700 1 -- 172.21.15.46:0/2294572830 learned_addr learned my addr 172.21.15.46:0/2294572830 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d11425700 1 --2- 172.21.15.46:0/2294572830 >> v2:172.21.15.106:3300/0 conn(0x7f9d0c0ad070 0x7f9d0c0ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d0b7fe700 1 -- 172.21.15.46:0/2294572830 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394485896 0 0) 0x7f9d0c0ae180 con 0x7f9d0c0cc1c0 2024-01-27T01:07:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d0b7fe700 1 -- 172.21.15.46:0/2294572830 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cf8003570 con 0x7f9d0c0cc1c0 2024-01-27T01:07:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d0b7fe700 1 -- 172.21.15.46:0/2294572830 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1224569881 0 0) 0x7f9cf8003570 con 0x7f9d0c0cc1c0 2024-01-27T01:07:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d0b7fe700 1 -- 172.21.15.46:0/2294572830 >> v2:172.21.15.106:3300/0 conn(0x7f9d0c0ad070 msgr2=0x7f9d0c0ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d0b7fe700 1 --2- 172.21.15.46:0/2294572830 >> v2:172.21.15.106:3300/0 conn(0x7f9d0c0ad070 0x7f9d0c0ae610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d0b7fe700 1 -- 172.21.15.46:0/2294572830 >> v1:172.21.15.106:6789/0 conn(0x7f9d0c0cbdf0 legacy=0x7f9d0c0b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d0b7fe700 1 -- 172.21.15.46:0/2294572830 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0c0ad570 con 0x7f9d0c0cc1c0 2024-01-27T01:07:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d0b7fe700 1 -- 172.21.15.46:0/2294572830 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f9cf4001470 con 0x7f9d0c0cc1c0 2024-01-27T01:07:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d0b7fe700 1 -- 172.21.15.46:0/2294572830 >> v1:172.21.15.46:6789/0 conn(0x7f9d0c0cc1c0 legacy=0x7f9d0c0c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d0b7fe700 1 --2- 172.21.15.46:0/2294572830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf8003ab0 0x7f9cf8003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d0b7fe700 1 --2- 172.21.15.46:0/2294572830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cf8004810 0x7f9cf8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d0b7fe700 1 --2- 172.21.15.46:0/2294572830 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9cf8005210 0x7f9cf800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d0b7fe700 1 -- 172.21.15.46:0/2294572830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c0aeb50 con 0x7f9cf8003ab0 2024-01-27T01:07:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d0b7fe700 1 -- 172.21.15.46:0/2294572830 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf800c1b0 con 0x7f9cf8004810 2024-01-27T01:07:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.094+0000 7f9d0b7fe700 1 -- 172.21.15.46:0/2294572830 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf800c370 con 0x7f9cf8005210 2024-01-27T01:07:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.095+0000 7f9d10c24700 1 --2- 172.21.15.46:0/2294572830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf8003ab0 0x7f9cf8003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.095+0000 7f9d0bfff700 1 --2- 172.21.15.46:0/2294572830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cf8004810 0x7f9cf8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.095+0000 7f9d11425700 1 --2- 172.21.15.46:0/2294572830 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9cf8005210 0x7f9cf800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.095+0000 7f9d10c24700 1 -- 172.21.15.46:0/2294572830 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9cf8005210 msgr2=0x7f9cf800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.095+0000 7f9d10c24700 1 --2- 172.21.15.46:0/2294572830 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9cf8005210 0x7f9cf800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.095+0000 7f9d10c24700 1 -- 172.21.15.46:0/2294572830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cf8004810 msgr2=0x7f9cf8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.095+0000 7f9d10c24700 1 --2- 172.21.15.46:0/2294572830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cf8004810 0x7f9cf8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.095+0000 7f9d10c24700 1 -- 172.21.15.46:0/2294572830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf8003570 con 0x7f9cf8003ab0 2024-01-27T01:07:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.095+0000 7f9d11425700 1 --2- 172.21.15.46:0/2294572830 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9cf8005210 0x7f9cf800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.095+0000 7f9d11425700 1 -- 172.21.15.46:0/2294572830 reap_dead start 2024-01-27T01:07:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.095+0000 7f9d10c24700 1 --2- 172.21.15.46:0/2294572830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf8003ab0 0x7f9cf8003ec0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc0018b0 tx=0x7f9cfc007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb64d12fa23389d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.095+0000 7f9d0b7fe700 1 -- 172.21.15.46:0/2294572830 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc012420 con 0x7f9cf8003ab0 2024-01-27T01:07:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.095+0000 7f9d0b7fe700 1 -- 172.21.15.46:0/2294572830 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cfc0125a0 con 0x7f9cf8003ab0 2024-01-27T01:07:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.095+0000 7f9d0b7fe700 1 -- 172.21.15.46:0/2294572830 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc012960 con 0x7f9cf8003ab0 2024-01-27T01:07:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.096+0000 7f9d12e88700 1 -- 172.21.15.46:0/2294572830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf8003ab0 msgr2=0x7f9cf8003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.096+0000 7f9d12e88700 1 --2- 172.21.15.46:0/2294572830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf8003ab0 0x7f9cf8003ec0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc0018b0 tx=0x7f9cfc007d30 comp rx=0 tx=0).stop 2024-01-27T01:07:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.096+0000 7f9d12e88700 1 -- 172.21.15.46:0/2294572830 shutdown_connections 2024-01-27T01:07:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.096+0000 7f9d12e88700 1 --2- 172.21.15.46:0/2294572830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf8003ab0 0x7f9cf8003ec0 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.096+0000 7f9d12e88700 1 -- 172.21.15.46:0/2294572830 >> 172.21.15.46:0/2294572830 conn(0x7f9d0c068400 msgr2=0x7f9d0c0cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.096+0000 7f9d12e88700 1 -- 172.21.15.46:0/2294572830 shutdown_connections 2024-01-27T01:07:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.096+0000 7f9d12e88700 1 -- 172.21.15.46:0/2294572830 wait complete. 2024-01-27T01:07:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.097+0000 7f9d12e88700 1 Processor -- start 2024-01-27T01:07:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.097+0000 7f9d12e88700 1 -- start start 2024-01-27T01:07:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.098+0000 7f9d12e88700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf8003ab0 0x7f9d0c166b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.098+0000 7f9d12e88700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d0c154fd0 0x7f9d0c0a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.098+0000 7f9d12e88700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d0c155440 0x7f9d0c150d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.098+0000 7f9d12e88700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c154b40 con 0x7f9cf8003ab0 2024-01-27T01:07:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.098+0000 7f9d12e88700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c153f30 con 0x7f9d0c155440 2024-01-27T01:07:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.098+0000 7f9d12e88700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c154210 con 0x7f9d0c154fd0 2024-01-27T01:07:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.098+0000 7f9d10c24700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf8003ab0 0x7f9d0c166b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.098+0000 7f9d10c24700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf8003ab0 0x7f9d0c166b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42306/0 (socket says 172.21.15.46:42306) 2024-01-27T01:07:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.098+0000 7f9d10c24700 1 -- 172.21.15.46:0/4152470362 learned_addr learned my addr 172.21.15.46:0/4152470362 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.098+0000 7f9d0bfff700 1 --2- 172.21.15.46:0/4152470362 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d0c154fd0 0x7f9d0c0a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.098+0000 7f9d11425700 1 --2- 172.21.15.46:0/4152470362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d0c155440 0x7f9d0c150d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.099+0000 7f9d10c24700 1 -- 172.21.15.46:0/4152470362 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d0c154fd0 msgr2=0x7f9d0c0a5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.099+0000 7f9d10c24700 1 --2- 172.21.15.46:0/4152470362 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d0c154fd0 0x7f9d0c0a5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.099+0000 7f9d10c24700 1 -- 172.21.15.46:0/4152470362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d0c155440 msgr2=0x7f9d0c150d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.099+0000 7f9d10c24700 1 --2- 172.21.15.46:0/4152470362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d0c155440 0x7f9d0c150d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.099+0000 7f9d10c24700 1 -- 172.21.15.46:0/4152470362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0c1544f0 con 0x7f9cf8003ab0 2024-01-27T01:07:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.099+0000 7f9d11425700 1 --2- 172.21.15.46:0/4152470362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d0c155440 0x7f9d0c150d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.099+0000 7f9d0bfff700 1 --2- 172.21.15.46:0/4152470362 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d0c154fd0 0x7f9d0c0a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.099+0000 7f9d10c24700 1 --2- 172.21.15.46:0/4152470362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf8003ab0 0x7f9d0c166b00 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800c180 tx=0x7f9cfc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.099+0000 7f9d08ff9700 1 -- 172.21.15.46:0/4152470362 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc00c800 con 0x7f9cf8003ab0 2024-01-27T01:07:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.099+0000 7f9d08ff9700 1 -- 172.21.15.46:0/4152470362 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cfc00c960 con 0x7f9cf8003ab0 2024-01-27T01:07:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.099+0000 7f9d12e88700 1 -- 172.21.15.46:0/4152470362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0c1547d0 con 0x7f9cf8003ab0 2024-01-27T01:07:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.099+0000 7f9d08ff9700 1 -- 172.21.15.46:0/4152470362 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc00cd20 con 0x7f9cf8003ab0 2024-01-27T01:07:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.099+0000 7f9d12e88700 1 -- 172.21.15.46:0/4152470362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0c153be0 con 0x7f9cf8003ab0 2024-01-27T01:07:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.100+0000 7f9d12e88700 1 -- 172.21.15.46:0/4152470362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d0c156a30 con 0x7f9cf8003ab0 2024-01-27T01:07:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.102+0000 7f9d08ff9700 1 -- 172.21.15.46:0/4152470362 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f9cfc01f620 con 0x7f9cf8003ab0 2024-01-27T01:07:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.102+0000 7f9d08ff9700 1 --2- 172.21.15.46:0/4152470362 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9cec067170 0x7f9cec069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.102+0000 7f9d08ff9700 1 -- 172.21.15.46:0/4152470362 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f9cfc086890 con 0x7f9cf8003ab0 2024-01-27T01:07:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.102+0000 7f9d0bfff700 1 --2- 172.21.15.46:0/4152470362 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9cec067170 0x7f9cec069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.103+0000 7f9d0bfff700 1 --2- 172.21.15.46:0/4152470362 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9cec067170 0x7f9cec069620 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40015a0 tx=0x7f9cf4009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.105+0000 7f9d08ff9700 1 -- 172.21.15.46:0/4152470362 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9cfc0535c0 con 0x7f9cf8003ab0 2024-01-27T01:07:28.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.314+0000 7f9d12e88700 1 -- 172.21.15.46:0/4152470362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d0c04c710 con 0x7f9cf8003ab0 2024-01-27T01:07:28.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.315+0000 7f9d08ff9700 1 -- 172.21.15.46:0/4152470362 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v242) v1 ==== 74+0+16338 (secure 0 0 0) 0x7f9cfc056be0 con 0x7f9cf8003ab0 2024-01-27T01:07:28.315 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:28.315 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":242,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:07:27.375081+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:07:25.886791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-01-27T01:07:26.340041+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:07:28.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.317+0000 7f9d12e88700 1 -- 172.21.15.46:0/4152470362 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9cec067170 msgr2=0x7f9cec069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.317+0000 7f9d12e88700 1 --2- 172.21.15.46:0/4152470362 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9cec067170 0x7f9cec069620 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40015a0 tx=0x7f9cf4009040 comp rx=0 tx=0).stop 2024-01-27T01:07:28.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.317+0000 7f9d12e88700 1 -- 172.21.15.46:0/4152470362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf8003ab0 msgr2=0x7f9d0c166b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.317+0000 7f9d12e88700 1 --2- 172.21.15.46:0/4152470362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf8003ab0 0x7f9d0c166b00 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800c180 tx=0x7f9cfc015000 comp rx=0 tx=0).stop 2024-01-27T01:07:28.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.317+0000 7f9d12e88700 1 -- 172.21.15.46:0/4152470362 shutdown_connections 2024-01-27T01:07:28.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.317+0000 7f9d12e88700 1 --2- 172.21.15.46:0/4152470362 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9cec067170 0x7f9cec069620 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.318+0000 7f9d12e88700 1 --2- 172.21.15.46:0/4152470362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf8003ab0 0x7f9d0c166b00 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.318+0000 7f9d12e88700 1 --2- 172.21.15.46:0/4152470362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d0c155440 0x7f9d0c150d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.318+0000 7f9d12e88700 1 --2- 172.21.15.46:0/4152470362 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d0c154fd0 0x7f9d0c0a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.318+0000 7f9d12e88700 1 -- 172.21.15.46:0/4152470362 >> 172.21.15.46:0/4152470362 conn(0x7f9d0c068400 msgr2=0x7f9d0c06a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:28.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.318+0000 7f9d12e88700 1 -- 172.21.15.46:0/4152470362 shutdown_connections 2024-01-27T01:07:28.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.318+0000 7f9d12e88700 1 -- 172.21.15.46:0/4152470362 wait complete. 2024-01-27T01:07:28.332 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:07:28.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.464+0000 7fab57038700 1 Processor -- start 2024-01-27T01:07:28.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.464+0000 7fab57038700 1 -- start start 2024-01-27T01:07:28.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.465+0000 7fab57038700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fab50157920 0x7fab500b2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.465+0000 7fab57038700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab500b2a60 0x7fab500cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.465+0000 7fab57038700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab500cc550 con 0x7fab50157550 2024-01-27T01:07:28.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.465+0000 7fab57038700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab500ca020 con 0x7fab50157920 2024-01-27T01:07:28.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.465+0000 7fab57038700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab500b2ee0 con 0x7fab500b2a60 2024-01-27T01:07:28.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.465+0000 7fab54dd4700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fab50157550 0x7fab5006f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47374/0 (socket says 172.21.15.46:47374) 2024-01-27T01:07:28.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.465+0000 7fab54dd4700 1 -- 172.21.15.46:0/2766692770 learned_addr learned my addr 172.21.15.46:0/2766692770 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:28.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.465+0000 7fab555d5700 1 --2- 172.21.15.46:0/2766692770 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab500b2a60 0x7fab500cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.465+0000 7fab4ffff700 1 --2- 172.21.15.46:0/2766692770 >> v2:172.21.15.106:3300/0 conn(0x7fab50157920 0x7fab500b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.466+0000 7fab4f7fe700 1 -- 172.21.15.46:0/2766692770 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186581691 0 0) 0x7fab500cc550 con 0x7fab50157550 2024-01-27T01:07:28.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.466+0000 7fab4f7fe700 1 -- 172.21.15.46:0/2766692770 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab40003570 con 0x7fab50157550 2024-01-27T01:07:28.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.466+0000 7fab555d5700 1 -- 172.21.15.46:0/2766692770 >> v2:172.21.15.106:3300/0 conn(0x7fab50157920 msgr2=0x7fab500b2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:28.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.466+0000 7fab4f7fe700 1 -- 172.21.15.46:0/2766692770 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1270639501 0 0) 0x7fab40003570 con 0x7fab50157550 2024-01-27T01:07:28.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.466+0000 7fab555d5700 1 --2- 172.21.15.46:0/2766692770 >> v2:172.21.15.106:3300/0 conn(0x7fab50157920 0x7fab500b2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.466+0000 7fab555d5700 1 -- 172.21.15.46:0/2766692770 >> v1:172.21.15.46:6789/0 conn(0x7fab50157550 legacy=0x7fab5006f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.466+0000 7fab555d5700 1 -- 172.21.15.46:0/2766692770 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab500cb6a0 con 0x7fab500b2a60 2024-01-27T01:07:28.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.466+0000 7fab4ffff700 1 --2- 172.21.15.46:0/2766692770 >> v2:172.21.15.106:3300/0 conn(0x7fab50157920 0x7fab500b2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:28.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.466+0000 7fab555d5700 1 --2- 172.21.15.46:0/2766692770 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab500b2a60 0x7fab500cc9e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fab48009e60 tx=0x7fab48009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae7569cf8318849f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:28.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.466+0000 7fab4f7fe700 1 -- 172.21.15.46:0/2766692770 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4800e070 con 0x7fab500b2a60 2024-01-27T01:07:28.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.466+0000 7fab4f7fe700 1 -- 172.21.15.46:0/2766692770 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4800b7a0 con 0x7fab500b2a60 2024-01-27T01:07:28.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.466+0000 7fab4f7fe700 1 -- 172.21.15.46:0/2766692770 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab44001470 con 0x7fab500b2a60 2024-01-27T01:07:28.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.467+0000 7fab57038700 1 -- 172.21.15.46:0/2766692770 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab500b2a60 msgr2=0x7fab500cc9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.467+0000 7fab57038700 1 --2- 172.21.15.46:0/2766692770 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab500b2a60 0x7fab500cc9e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fab48009e60 tx=0x7fab48009780 comp rx=0 tx=0).stop 2024-01-27T01:07:28.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.467+0000 7fab57038700 1 -- 172.21.15.46:0/2766692770 shutdown_connections 2024-01-27T01:07:28.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.467+0000 7fab57038700 1 --2- 172.21.15.46:0/2766692770 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab500b2a60 0x7fab500cc9e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.467+0000 7fab57038700 1 --2- 172.21.15.46:0/2766692770 >> v2:172.21.15.106:3300/0 conn(0x7fab50157920 0x7fab500b2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.467+0000 7fab57038700 1 -- 172.21.15.46:0/2766692770 >> 172.21.15.46:0/2766692770 conn(0x7fab50069510 msgr2=0x7fab500aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:28.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.467+0000 7fab57038700 1 -- 172.21.15.46:0/2766692770 shutdown_connections 2024-01-27T01:07:28.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.467+0000 7fab57038700 1 -- 172.21.15.46:0/2766692770 wait complete. 2024-01-27T01:07:28.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.468+0000 7fab57038700 1 Processor -- start 2024-01-27T01:07:28.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.468+0000 7fab57038700 1 -- start start 2024-01-27T01:07:28.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.469+0000 7fab57038700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab500b2a60 0x7fab50152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.469+0000 7fab57038700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab50157550 0x7fab50157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.469+0000 7fab57038700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab50157920 0x7fab50158330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.469+0000 7fab57038700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab500c7f90 con 0x7fab50157920 2024-01-27T01:07:28.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.469+0000 7fab57038700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab501595c0 con 0x7fab50157550 2024-01-27T01:07:28.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.469+0000 7fab57038700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab501589b0 con 0x7fab500b2a60 2024-01-27T01:07:28.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.469+0000 7fab555d5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab50157920 0x7fab50158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.469+0000 7fab555d5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab50157920 0x7fab50158330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42312/0 (socket says 172.21.15.46:42312) 2024-01-27T01:07:28.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.469+0000 7fab555d5700 1 -- 172.21.15.46:0/3390576927 learned_addr learned my addr 172.21.15.46:0/3390576927 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.469+0000 7fab4ffff700 1 --2- 172.21.15.46:0/3390576927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab50157550 0x7fab50157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.469+0000 7fab54dd4700 1 --2- 172.21.15.46:0/3390576927 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab500b2a60 0x7fab50152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.470+0000 7fab555d5700 1 -- 172.21.15.46:0/3390576927 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab500b2a60 msgr2=0x7fab50152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.470+0000 7fab555d5700 1 --2- 172.21.15.46:0/3390576927 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab500b2a60 0x7fab50152600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.470+0000 7fab555d5700 1 -- 172.21.15.46:0/3390576927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab50157550 msgr2=0x7fab50157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.470+0000 7fab555d5700 1 --2- 172.21.15.46:0/3390576927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab50157550 0x7fab50157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.470+0000 7fab555d5700 1 -- 172.21.15.46:0/3390576927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3c009420 con 0x7fab50157920 2024-01-27T01:07:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.470+0000 7fab555d5700 1 --2- 172.21.15.46:0/3390576927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab50157920 0x7fab50158330 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7fab48012760 tx=0x7fab48018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.470+0000 7fab4cff9700 1 -- 172.21.15.46:0/3390576927 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab48012b90 con 0x7fab50157920 2024-01-27T01:07:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.470+0000 7fab4cff9700 1 -- 172.21.15.46:0/3390576927 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab48012cf0 con 0x7fab50157920 2024-01-27T01:07:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.470+0000 7fab57038700 1 -- 172.21.15.46:0/3390576927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab48009a20 con 0x7fab50157920 2024-01-27T01:07:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.470+0000 7fab4cff9700 1 -- 172.21.15.46:0/3390576927 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab480266b0 con 0x7fab50157920 2024-01-27T01:07:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.470+0000 7fab57038700 1 -- 172.21.15.46:0/3390576927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab50158f70 con 0x7fab50157920 2024-01-27T01:07:28.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.471+0000 7fab57038700 1 -- 172.21.15.46:0/3390576927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab5004c710 con 0x7fab50157920 2024-01-27T01:07:28.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.472+0000 7fab4cff9700 1 -- 172.21.15.46:0/3390576927 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fab48026ae0 con 0x7fab50157920 2024-01-27T01:07:28.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.472+0000 7fab4cff9700 1 --2- 172.21.15.46:0/3390576927 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fab34067170 0x7fab34069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.472+0000 7fab4cff9700 1 -- 172.21.15.46:0/3390576927 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fab48086d40 con 0x7fab50157920 2024-01-27T01:07:28.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.472+0000 7fab54dd4700 1 --2- 172.21.15.46:0/3390576927 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fab34067170 0x7fab34069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.473+0000 7fab54dd4700 1 --2- 172.21.15.46:0/3390576927 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fab34067170 0x7fab34069620 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fab440011e0 tx=0x7fab44001aa0 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:28.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.477+0000 7fab4cff9700 1 -- 172.21.15.46:0/3390576927 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fab480538e0 con 0x7fab50157920 2024-01-27T01:07:28.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.681+0000 7fab57038700 1 -- 172.21.15.46:0/3390576927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab50159700 con 0x7fab50157920 2024-01-27T01:07:28.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.682+0000 7fab4cff9700 1 -- 172.21.15.46:0/3390576927 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v242) v1 ==== 74+0+16338 (secure 0 0 0) 0x7fab48012e60 con 0x7fab50157920 2024-01-27T01:07:28.682 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:28.682 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":242,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:07:27.375081+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:07:25.886791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-01-27T01:07:26.340041+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:07:28.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.684+0000 7fab57038700 1 -- 172.21.15.46:0/3390576927 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fab34067170 msgr2=0x7fab34069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.684+0000 7fab57038700 1 --2- 172.21.15.46:0/3390576927 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fab34067170 0x7fab34069620 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fab440011e0 tx=0x7fab44001aa0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.684+0000 7fab57038700 1 -- 172.21.15.46:0/3390576927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab50157920 msgr2=0x7fab50158330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.684+0000 7fab57038700 1 --2- 172.21.15.46:0/3390576927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab50157920 0x7fab50158330 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7fab48012760 tx=0x7fab48018000 comp rx=0 tx=0).stop 2024-01-27T01:07:28.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.684+0000 7fab57038700 1 -- 172.21.15.46:0/3390576927 shutdown_connections 2024-01-27T01:07:28.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.684+0000 7fab57038700 1 --2- 172.21.15.46:0/3390576927 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fab34067170 0x7fab34069620 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.684+0000 7fab57038700 1 --2- 172.21.15.46:0/3390576927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab50157920 0x7fab50158330 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.684+0000 7fab57038700 1 --2- 172.21.15.46:0/3390576927 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab500b2a60 0x7fab50152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.684+0000 7fab57038700 1 --2- 172.21.15.46:0/3390576927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab50157550 0x7fab50157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.684+0000 7fab57038700 1 -- 172.21.15.46:0/3390576927 >> 172.21.15.46:0/3390576927 conn(0x7fab50069510 msgr2=0x7fab5006eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.684+0000 7fab57038700 1 -- 172.21.15.46:0/3390576927 shutdown_connections 2024-01-27T01:07:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.684+0000 7fab57038700 1 -- 172.21.15.46:0/3390576927 wait complete. 2024-01-27T01:07:28.698 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-27T01:07:28.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.834+0000 7f0a26c3e700 1 Processor -- start 2024-01-27T01:07:28.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.834+0000 7f0a26c3e700 1 -- start start 2024-01-27T01:07:28.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.835+0000 7f0a26c3e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a20159930 0x7f0a201647d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.835+0000 7f0a26c3e700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2015e390 con 0x7f0a200c93a0 2024-01-27T01:07:28.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.835+0000 7f0a26c3e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a20068460 con 0x7f0a200c9770 2024-01-27T01:07:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.835+0000 7f0a26c3e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a20159d50 con 0x7f0a20159930 2024-01-27T01:07:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.835+0000 7f0a249da700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0a200c93a0 0x7f0a20067d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47388/0 (socket says 172.21.15.46:47388) 2024-01-27T01:07:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.835+0000 7f0a249da700 1 -- 172.21.15.46:0/1859583578 learned_addr learned my addr 172.21.15.46:0/1859583578 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.835+0000 7f0a251db700 1 --2- 172.21.15.46:0/1859583578 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a20159930 0x7f0a201647d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.836+0000 7f0a1f7fe700 1 -- 172.21.15.46:0/1859583578 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538929409 0 0) 0x7f0a2015e390 con 0x7f0a200c93a0 2024-01-27T01:07:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.836+0000 7f0a1f7fe700 1 -- 172.21.15.46:0/1859583578 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a10003570 con 0x7f0a200c93a0 2024-01-27T01:07:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.836+0000 7f0a1f7fe700 1 -- 172.21.15.46:0/1859583578 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550170610 0 0) 0x7f0a20068460 con 0x7f0a200c9770 2024-01-27T01:07:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.836+0000 7f0a1f7fe700 1 -- 172.21.15.46:0/1859583578 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a2015e390 con 0x7f0a200c9770 2024-01-27T01:07:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.836+0000 7f0a251db700 1 -- 172.21.15.46:0/1859583578 >> v1:172.21.15.106:6789/0 conn(0x7f0a200c9770 legacy=0x7f0a20159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.836+0000 7f0a251db700 1 -- 172.21.15.46:0/1859583578 >> v1:172.21.15.46:6789/0 conn(0x7f0a200c93a0 legacy=0x7f0a20067d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.836+0000 7f0a251db700 1 -- 172.21.15.46:0/1859583578 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2015d810 con 0x7f0a20159930 2024-01-27T01:07:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.836+0000 7f0a251db700 1 --2- 172.21.15.46:0/1859583578 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a20159930 0x7f0a201647d0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009e90 tx=0x7f0a18009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63850229d491981 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.836+0000 7f0a1f7fe700 1 -- 172.21.15.46:0/1859583578 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1800e070 con 0x7f0a20159930 2024-01-27T01:07:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.836+0000 7f0a1f7fe700 1 -- 172.21.15.46:0/1859583578 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1800bb80 con 0x7f0a20159930 2024-01-27T01:07:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.836+0000 7f0a1f7fe700 1 -- 172.21.15.46:0/1859583578 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a18012430 con 0x7f0a20159930 2024-01-27T01:07:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.837+0000 7f0a26c3e700 1 -- 172.21.15.46:0/1859583578 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a20159930 msgr2=0x7f0a201647d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.837+0000 7f0a26c3e700 1 --2- 172.21.15.46:0/1859583578 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a20159930 0x7f0a201647d0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009e90 tx=0x7f0a18009780 comp rx=0 tx=0).stop 2024-01-27T01:07:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.837+0000 7f0a26c3e700 1 -- 172.21.15.46:0/1859583578 shutdown_connections 2024-01-27T01:07:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.837+0000 7f0a26c3e700 1 --2- 172.21.15.46:0/1859583578 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a20159930 0x7f0a201647d0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.837+0000 7f0a26c3e700 1 -- 172.21.15.46:0/1859583578 >> 172.21.15.46:0/1859583578 conn(0x7f0a200ca9a0 msgr2=0x7f0a20068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.837+0000 7f0a26c3e700 1 -- 172.21.15.46:0/1859583578 shutdown_connections 2024-01-27T01:07:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.837+0000 7f0a26c3e700 1 -- 172.21.15.46:0/1859583578 wait complete. 2024-01-27T01:07:28.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.838+0000 7f0a26c3e700 1 Processor -- start 2024-01-27T01:07:28.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.839+0000 7f0a26c3e700 1 -- start start 2024-01-27T01:07:28.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.839+0000 7f0a26c3e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a200c93a0 0x7f0a20143a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.839+0000 7f0a26c3e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a200c9770 0x7f0a2015cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.840+0000 7f0a26c3e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20159930 0x7f0a200c3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.840+0000 7f0a26c3e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2015bb80 con 0x7f0a20159930 2024-01-27T01:07:28.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.840+0000 7f0a26c3e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a20155690 con 0x7f0a200c9770 2024-01-27T01:07:28.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.840+0000 7f0a26c3e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2016adc0 con 0x7f0a200c93a0 2024-01-27T01:07:28.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.840+0000 7f0a251db700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20159930 0x7f0a200c3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.840+0000 7f0a251db700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20159930 0x7f0a200c3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42328/0 (socket says 172.21.15.46:42328) 2024-01-27T01:07:28.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.840+0000 7f0a251db700 1 -- 172.21.15.46:0/53275191 learned_addr learned my addr 172.21.15.46:0/53275191 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:28.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.840+0000 7f0a249da700 1 --2- 172.21.15.46:0/53275191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a200c93a0 0x7f0a20143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.840+0000 7f0a1ffff700 1 --2- 172.21.15.46:0/53275191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a200c9770 0x7f0a2015cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.840+0000 7f0a251db700 1 -- 172.21.15.46:0/53275191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a200c93a0 msgr2=0x7f0a20143a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.840+0000 7f0a251db700 1 --2- 172.21.15.46:0/53275191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a200c93a0 0x7f0a20143a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.840+0000 7f0a251db700 1 -- 172.21.15.46:0/53275191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a200c9770 msgr2=0x7f0a2015cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:28.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.840+0000 7f0a251db700 1 --2- 172.21.15.46:0/53275191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a200c9770 0x7f0a2015cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:28.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.840+0000 7f0a251db700 1 -- 172.21.15.46:0/53275191 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a18009a20 con 0x7f0a20159930 2024-01-27T01:07:28.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.840+0000 7f0a1ffff700 1 --2- 172.21.15.46:0/53275191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a200c9770 0x7f0a2015cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:28.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.840+0000 7f0a251db700 1 --2- 172.21.15.46:0/53275191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20159930 0x7f0a200c3210 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800b610 tx=0x7f0a18019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:28.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.840+0000 7f0a1cff9700 1 -- 172.21.15.46:0/53275191 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a18007000 con 0x7f0a20159930 2024-01-27T01:07:28.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.841+0000 7f0a1cff9700 1 -- 172.21.15.46:0/53275191 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a180263f0 con 0x7f0a20159930 2024-01-27T01:07:28.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.841+0000 7f0a1cff9700 1 -- 172.21.15.46:0/53275191 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a180267d0 con 0x7f0a20159930 2024-01-27T01:07:28.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.841+0000 7f0a26c3e700 1 -- 172.21.15.46:0/53275191 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a2015c6e0 con 0x7f0a20159930 2024-01-27T01:07:28.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.841+0000 7f0a26c3e700 1 -- 172.21.15.46:0/53275191 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a2015c090 con 0x7f0a20159930 2024-01-27T01:07:28.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.842+0000 7f0a26c3e700 1 -- 172.21.15.46:0/53275191 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2015c870 con 0x7f0a20159930 2024-01-27T01:07:28.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.842+0000 7f0a1cff9700 1 -- 172.21.15.46:0/53275191 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f0a18026c00 con 0x7f0a20159930 2024-01-27T01:07:28.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.842+0000 7f0a1cff9700 1 --2- 172.21.15.46:0/53275191 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a04067170 0x7f0a04069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:28.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.843+0000 7f0a1cff9700 1 -- 172.21.15.46:0/53275191 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f0a18086ce0 con 0x7f0a20159930 2024-01-27T01:07:28.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.843+0000 7f0a249da700 1 --2- 172.21.15.46:0/53275191 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a04067170 0x7f0a04069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:28.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.843+0000 7f0a249da700 1 --2- 172.21.15.46:0/53275191 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a04067170 0x7f0a04069620 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0a14007fd0 tx=0x7f0a14009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:28.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:28.847+0000 7f0a1cff9700 1 -- 172.21.15.46:0/53275191 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a18053880 con 0x7f0a20159930 2024-01-27T01:07:29.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:29.099+0000 7f0a26c3e700 1 -- 172.21.15.46:0/53275191 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0a2004c710 con 0x7f0a20159930 2024-01-27T01:07:29.920 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-01-27T01:07:29.920+0000 7f95e4017700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-27T01:07:29.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:29.946+0000 7f0a1cff9700 1 -- 172.21.15.46:0/53275191 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v132) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0a18056ea0 con 0x7f0a20159930 2024-01-27T01:07:29.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:29.949+0000 7f0a26c3e700 1 -- 172.21.15.46:0/53275191 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a04067170 msgr2=0x7f0a04069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:29.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:29.949+0000 7f0a26c3e700 1 --2- 172.21.15.46:0/53275191 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a04067170 0x7f0a04069620 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0a14007fd0 tx=0x7f0a14009040 comp rx=0 tx=0).stop 2024-01-27T01:07:29.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:29.949+0000 7f0a26c3e700 1 -- 172.21.15.46:0/53275191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20159930 msgr2=0x7f0a200c3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:29.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:29.949+0000 7f0a26c3e700 1 --2- 172.21.15.46:0/53275191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20159930 0x7f0a200c3210 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800b610 tx=0x7f0a18019000 comp rx=0 tx=0).stop 2024-01-27T01:07:29.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:29.949+0000 7f0a26c3e700 1 -- 172.21.15.46:0/53275191 shutdown_connections 2024-01-27T01:07:29.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:29.949+0000 7f0a26c3e700 1 --2- 172.21.15.46:0/53275191 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a04067170 0x7f0a04069620 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:29.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:29.949+0000 7f0a26c3e700 1 --2- 172.21.15.46:0/53275191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20159930 0x7f0a200c3210 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:29.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:29.949+0000 7f0a26c3e700 1 --2- 172.21.15.46:0/53275191 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a200c93a0 0x7f0a20143a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:29.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:29.949+0000 7f0a26c3e700 1 --2- 172.21.15.46:0/53275191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a200c9770 0x7f0a2015cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:29.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:29.949+0000 7f0a26c3e700 1 -- 172.21.15.46:0/53275191 >> 172.21.15.46:0/53275191 conn(0x7f0a200ca9a0 msgr2=0x7f0a20070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:29.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:29.949+0000 7f0a26c3e700 1 -- 172.21.15.46:0/53275191 shutdown_connections 2024-01-27T01:07:29.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:29.949+0000 7f0a26c3e700 1 -- 172.21.15.46:0/53275191 wait complete. 2024-01-27T01:07:29.965 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-27T01:07:30.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.104+0000 7f7dab286700 1 Processor -- start 2024-01-27T01:07:30.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.104+0000 7f7dab286700 1 -- start start 2024-01-27T01:07:30.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.105+0000 7f7dab286700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da41569b0 0x7f7da40a98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:30.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.105+0000 7f7dab286700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7da4069b20 0x7f7da40aa2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:30.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.105+0000 7f7dab286700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7da406ba40 con 0x7f7da41565e0 2024-01-27T01:07:30.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.105+0000 7f7dab286700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7da40acbb0 con 0x7f7da4069b20 2024-01-27T01:07:30.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.105+0000 7f7dab286700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da406e470 con 0x7f7da41569b0 2024-01-27T01:07:30.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.105+0000 7f7da9823700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7da4069b20 0x7f7da40aa2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:30.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.105+0000 7f7da9823700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7da4069b20 0x7f7da40aa2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42342/0 (socket says 172.21.15.46:42342) 2024-01-27T01:07:30.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.105+0000 7f7da9823700 1 -- 172.21.15.46:0/2296959228 learned_addr learned my addr 172.21.15.46:0/2296959228 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:30.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.105+0000 7f7da8821700 1 --2- 172.21.15.46:0/2296959228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da41569b0 0x7f7da40a98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:30.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.105+0000 7f7da9823700 1 -- 172.21.15.46:0/2296959228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da41569b0 msgr2=0x7f7da40a98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:30.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.106+0000 7f7da9823700 1 --2- 172.21.15.46:0/2296959228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da41569b0 0x7f7da40a98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:30.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.106+0000 7f7da9823700 1 -- 172.21.15.46:0/2296959228 >> v1:172.21.15.106:6789/0 conn(0x7f7da41565e0 legacy=0x7f7da40ac4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:30.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.106+0000 7f7da9823700 1 -- 172.21.15.46:0/2296959228 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da406db60 con 0x7f7da4069b20 2024-01-27T01:07:30.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.106+0000 7f7da9823700 1 --2- 172.21.15.46:0/2296959228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7da4069b20 0x7f7da40aa2a0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c00d1a0 tx=0x7f7d9c00ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f66f12aa12ac320e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:30.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.106+0000 7f7d9bfff700 1 -- 172.21.15.46:0/2296959228 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c010070 con 0x7f7da4069b20 2024-01-27T01:07:30.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.106+0000 7f7d9bfff700 1 -- 172.21.15.46:0/2296959228 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9c00de10 con 0x7f7da4069b20 2024-01-27T01:07:30.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.106+0000 7f7d9bfff700 1 -- 172.21.15.46:0/2296959228 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c00b320 con 0x7f7da4069b20 2024-01-27T01:07:30.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.106+0000 7f7dab286700 1 -- 172.21.15.46:0/2296959228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7da4069b20 msgr2=0x7f7da40aa2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:30.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.106+0000 7f7dab286700 1 --2- 172.21.15.46:0/2296959228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7da4069b20 0x7f7da40aa2a0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c00d1a0 tx=0x7f7d9c00ba50 comp rx=0 tx=0).stop 2024-01-27T01:07:30.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.106+0000 7f7dab286700 1 -- 172.21.15.46:0/2296959228 shutdown_connections 2024-01-27T01:07:30.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.106+0000 7f7dab286700 1 --2- 172.21.15.46:0/2296959228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7da4069b20 0x7f7da40aa2a0 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:30.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.106+0000 7f7dab286700 1 --2- 172.21.15.46:0/2296959228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da41569b0 0x7f7da40a98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:30.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.107+0000 7f7dab286700 1 -- 172.21.15.46:0/2296959228 >> 172.21.15.46:0/2296959228 conn(0x7f7da4067930 msgr2=0x7f7da4067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:30.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.107+0000 7f7dab286700 1 -- 172.21.15.46:0/2296959228 shutdown_connections 2024-01-27T01:07:30.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.107+0000 7f7dab286700 1 -- 172.21.15.46:0/2296959228 wait complete. 2024-01-27T01:07:30.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.108+0000 7f7dab286700 1 Processor -- start 2024-01-27T01:07:30.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.108+0000 7f7dab286700 1 -- start start 2024-01-27T01:07:30.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.109+0000 7f7dab286700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7da4069b20 0x7f7da416a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:30.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.109+0000 7f7dab286700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da41565e0 0x7f7da4157ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:30.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.109+0000 7f7dab286700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7da41569b0 0x7f7da4150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:30.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.109+0000 7f7dab286700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da4151710 con 0x7f7da41569b0 2024-01-27T01:07:30.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.109+0000 7f7dab286700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da416a870 con 0x7f7da4069b20 2024-01-27T01:07:30.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.109+0000 7f7dab286700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da4158430 con 0x7f7da41565e0 2024-01-27T01:07:30.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.109+0000 7f7da9823700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7da41569b0 0x7f7da4150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:30.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.109+0000 7f7da9823700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7da41569b0 0x7f7da4150840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42348/0 (socket says 172.21.15.46:42348) 2024-01-27T01:07:30.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.109+0000 7f7da9823700 1 -- 172.21.15.46:0/3198696917 learned_addr learned my addr 172.21.15.46:0/3198696917 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:30.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.109+0000 7f7da8821700 1 --2- 172.21.15.46:0/3198696917 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da41565e0 0x7f7da4157ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:30.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.109+0000 7f7da9022700 1 --2- 172.21.15.46:0/3198696917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7da4069b20 0x7f7da416a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:30.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.109+0000 7f7da9823700 1 -- 172.21.15.46:0/3198696917 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da41565e0 msgr2=0x7f7da4157ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:30.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.109+0000 7f7da9823700 1 --2- 172.21.15.46:0/3198696917 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da41565e0 0x7f7da4157ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:30.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.109+0000 7f7da9823700 1 -- 172.21.15.46:0/3198696917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7da4069b20 msgr2=0x7f7da416a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:30.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.109+0000 7f7da9823700 1 --2- 172.21.15.46:0/3198696917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7da4069b20 0x7f7da416a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:30.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.109+0000 7f7da9823700 1 -- 172.21.15.46:0/3198696917 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9c00bcf0 con 0x7f7da41569b0 2024-01-27T01:07:30.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.110+0000 7f7da9823700 1 --2- 172.21.15.46:0/3198696917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7da41569b0 0x7f7da4150840 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c00b4d0 tx=0x7f7d9c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:30.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.110+0000 7f7d997fa700 1 -- 172.21.15.46:0/3198696917 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c00ac90 con 0x7f7da41569b0 2024-01-27T01:07:30.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.110+0000 7f7d997fa700 1 -- 172.21.15.46:0/3198696917 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9c00adf0 con 0x7f7da41569b0 2024-01-27T01:07:30.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.110+0000 7f7dab286700 1 -- 172.21.15.46:0/3198696917 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7da4150d80 con 0x7f7da41569b0 2024-01-27T01:07:30.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.110+0000 7f7d997fa700 1 -- 172.21.15.46:0/3198696917 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c01d680 con 0x7f7da41569b0 2024-01-27T01:07:30.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.110+0000 7f7dab286700 1 -- 172.21.15.46:0/3198696917 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da4157410 con 0x7f7da41569b0 2024-01-27T01:07:30.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.111+0000 7f7dab286700 1 -- 172.21.15.46:0/3198696917 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7da404c710 con 0x7f7da41569b0 2024-01-27T01:07:30.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.111+0000 7f7d997fa700 1 -- 172.21.15.46:0/3198696917 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f7d9c003040 con 0x7f7da41569b0 2024-01-27T01:07:30.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.112+0000 7f7d997fa700 1 --2- 172.21.15.46:0/3198696917 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7d88067170 0x7f7d88069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:30.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.112+0000 7f7d997fa700 1 -- 172.21.15.46:0/3198696917 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f7d9c086340 con 0x7f7da41569b0 2024-01-27T01:07:30.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.112+0000 7f7da9022700 1 --2- 172.21.15.46:0/3198696917 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7d88067170 0x7f7d88069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:30.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.112+0000 7f7da9022700 1 --2- 172.21.15.46:0/3198696917 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7d88067170 0x7f7d88069620 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7da0003890 tx=0x7f7da0009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:30.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.117+0000 7f7d997fa700 1 -- 172.21.15.46:0/3198696917 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7d9c052ee0 con 0x7f7da41569b0 2024-01-27T01:07:30.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.375+0000 7f7dab286700 1 -- 172.21.15.46:0/3198696917 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7da40a9fc0 con 0x7f7da41569b0 2024-01-27T01:07:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.948+0000 7f7d997fa700 1 -- 172.21.15.46:0/3198696917 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v133) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7d9c056500 con 0x7f7da41569b0 2024-01-27T01:07:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.951+0000 7f7dab286700 1 -- 172.21.15.46:0/3198696917 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7d88067170 msgr2=0x7f7d88069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.951+0000 7f7dab286700 1 --2- 172.21.15.46:0/3198696917 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7d88067170 0x7f7d88069620 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7da0003890 tx=0x7f7da0009040 comp rx=0 tx=0).stop 2024-01-27T01:07:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.951+0000 7f7dab286700 1 -- 172.21.15.46:0/3198696917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7da41569b0 msgr2=0x7f7da4150840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.951+0000 7f7dab286700 1 --2- 172.21.15.46:0/3198696917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7da41569b0 0x7f7da4150840 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c00b4d0 tx=0x7f7d9c01a000 comp rx=0 tx=0).stop 2024-01-27T01:07:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.951+0000 7f7dab286700 1 -- 172.21.15.46:0/3198696917 shutdown_connections 2024-01-27T01:07:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.951+0000 7f7dab286700 1 --2- 172.21.15.46:0/3198696917 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7d88067170 0x7f7d88069620 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.951+0000 7f7dab286700 1 --2- 172.21.15.46:0/3198696917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7da41569b0 0x7f7da4150840 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.951+0000 7f7dab286700 1 --2- 172.21.15.46:0/3198696917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7da4069b20 0x7f7da416a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.951+0000 7f7dab286700 1 --2- 172.21.15.46:0/3198696917 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7da41565e0 0x7f7da4157ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.951+0000 7f7dab286700 1 -- 172.21.15.46:0/3198696917 >> 172.21.15.46:0/3198696917 conn(0x7f7da4067930 msgr2=0x7f7da40aa1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.951+0000 7f7dab286700 1 -- 172.21.15.46:0/3198696917 shutdown_connections 2024-01-27T01:07:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:30.951+0000 7f7dab286700 1 -- 172.21.15.46:0/3198696917 wait complete. 2024-01-27T01:07:30.967 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:07:31.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.103+0000 7f40ab5dd700 1 Processor -- start 2024-01-27T01:07:31.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.104+0000 7f40ab5dd700 1 -- start start 2024-01-27T01:07:31.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.104+0000 7f40ab5dd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f40a4159ad0 0x7f40a40c3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:31.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.104+0000 7f40ab5dd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40a40c4250 0x7f40a40c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:31.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.104+0000 7f40ab5dd700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40a40ce270 con 0x7f40a4159700 2024-01-27T01:07:31.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.104+0000 7f40ab5dd700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40a40c46d0 con 0x7f40a4159ad0 2024-01-27T01:07:31.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.104+0000 7f40ab5dd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a40cd660 con 0x7f40a40c4250 2024-01-27T01:07:31.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.105+0000 7f40a9379700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f40a4159700 0x7f40a40ccb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45524/0 (socket says 172.21.15.46:45524) 2024-01-27T01:07:31.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.105+0000 7f40a9379700 1 -- 172.21.15.46:0/1072810450 learned_addr learned my addr 172.21.15.46:0/1072810450 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:31.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.105+0000 7f40a8b78700 1 --2- 172.21.15.46:0/1072810450 >> v2:172.21.15.106:3300/0 conn(0x7f40a4159ad0 0x7f40a40c3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:31.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.105+0000 7f40a9b7a700 1 --2- 172.21.15.46:0/1072810450 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40a40c4250 0x7f40a40c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:31.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.105+0000 7f409bfff700 1 -- 172.21.15.46:0/1072810450 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268596571 0 0) 0x7f40a40ce270 con 0x7f40a4159700 2024-01-27T01:07:31.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.105+0000 7f409bfff700 1 -- 172.21.15.46:0/1072810450 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4090003570 con 0x7f40a4159700 2024-01-27T01:07:31.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.105+0000 7f409bfff700 1 -- 172.21.15.46:0/1072810450 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1554946807 0 0) 0x7f4090003570 con 0x7f40a4159700 2024-01-27T01:07:31.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.105+0000 7f40a8b78700 1 -- 172.21.15.46:0/1072810450 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40a40c4250 msgr2=0x7f40a40c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:31.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.105+0000 7f40a8b78700 1 --2- 172.21.15.46:0/1072810450 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40a40c4250 0x7f40a40c8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.105+0000 7f40a8b78700 1 -- 172.21.15.46:0/1072810450 >> v1:172.21.15.46:6789/0 conn(0x7f40a4159700 legacy=0x7f40a40ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:31.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.105+0000 7f40a8b78700 1 -- 172.21.15.46:0/1072810450 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a40cd940 con 0x7f40a4159ad0 2024-01-27T01:07:31.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.105+0000 7f40a8b78700 1 --2- 172.21.15.46:0/1072810450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40a4159ad0 0x7f40a40c3d10 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f40940098e0 tx=0x7f40940091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a64bc2cf5698f49a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:31.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.106+0000 7f409bfff700 1 -- 172.21.15.46:0/1072810450 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a00015f0 con 0x7f40a4159ad0 2024-01-27T01:07:31.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.106+0000 7f409bfff700 1 -- 172.21.15.46:0/1072810450 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4094011420 con 0x7f40a4159ad0 2024-01-27T01:07:31.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.106+0000 7f409bfff700 1 -- 172.21.15.46:0/1072810450 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40940117a0 con 0x7f40a4159ad0 2024-01-27T01:07:31.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.106+0000 7f40ab5dd700 1 -- 172.21.15.46:0/1072810450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40a4159ad0 msgr2=0x7f40a40c3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:31.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.106+0000 7f40ab5dd700 1 --2- 172.21.15.46:0/1072810450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40a4159ad0 0x7f40a40c3d10 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f40940098e0 tx=0x7f40940091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.106+0000 7f40ab5dd700 1 -- 172.21.15.46:0/1072810450 shutdown_connections 2024-01-27T01:07:31.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.106+0000 7f40ab5dd700 1 --2- 172.21.15.46:0/1072810450 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40a40c4250 0x7f40a40c8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.106+0000 7f40ab5dd700 1 --2- 172.21.15.46:0/1072810450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40a4159ad0 0x7f40a40c3d10 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.106+0000 7f40ab5dd700 1 -- 172.21.15.46:0/1072810450 >> 172.21.15.46:0/1072810450 conn(0x7f40a4097820 msgr2=0x7f40a4068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:31.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.106+0000 7f40ab5dd700 1 -- 172.21.15.46:0/1072810450 shutdown_connections 2024-01-27T01:07:31.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.106+0000 7f40ab5dd700 1 -- 172.21.15.46:0/1072810450 wait complete. 2024-01-27T01:07:31.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.107+0000 7f40ab5dd700 1 Processor -- start 2024-01-27T01:07:31.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.108+0000 7f40ab5dd700 1 -- start start 2024-01-27T01:07:31.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.108+0000 7f40ab5dd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a40c4250 0x7f40a415a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:31.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.108+0000 7f40ab5dd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40a4159700 0x7f40a4156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:31.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.108+0000 7f40ab5dd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40a4159ad0 0x7f40a40b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:31.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.108+0000 7f40ab5dd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a40b3260 con 0x7f40a40c4250 2024-01-27T01:07:31.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.108+0000 7f40ab5dd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a40c65e0 con 0x7f40a4159700 2024-01-27T01:07:31.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40ab5dd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a40c59d0 con 0x7f40a4159ad0 2024-01-27T01:07:31.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40a9379700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a40c4250 0x7f40a415a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:31.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40a9379700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a40c4250 0x7f40a415a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45784/0 (socket says 172.21.15.46:45784) 2024-01-27T01:07:31.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40a9379700 1 -- 172.21.15.46:0/3556054844 learned_addr learned my addr 172.21.15.46:0/3556054844 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:31.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40a8b78700 1 --2- 172.21.15.46:0/3556054844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40a4159700 0x7f40a4156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:31.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40a9b7a700 1 --2- 172.21.15.46:0/3556054844 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40a4159ad0 0x7f40a40b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:31.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40a9379700 1 -- 172.21.15.46:0/3556054844 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40a4159ad0 msgr2=0x7f40a40b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:31.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40a9379700 1 --2- 172.21.15.46:0/3556054844 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40a4159ad0 0x7f40a40b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40a9379700 1 -- 172.21.15.46:0/3556054844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40a4159700 msgr2=0x7f40a4156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:31.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40a9379700 1 --2- 172.21.15.46:0/3556054844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40a4159700 0x7f40a4156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40a9379700 1 -- 172.21.15.46:0/3556054844 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4094009420 con 0x7f40a40c4250 2024-01-27T01:07:31.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40a8b78700 1 --2- 172.21.15.46:0/3556054844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40a4159700 0x7f40a4156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:07:31.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40a9379700 1 --2- 172.21.15.46:0/3556054844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a40c4250 0x7f40a415a430 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f40a000c910 tx=0x7f40a0000f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:31.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40997fa700 1 -- 172.21.15.46:0/3556054844 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a0007300 con 0x7f40a40c4250 2024-01-27T01:07:31.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40997fa700 1 -- 172.21.15.46:0/3556054844 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a0007480 con 0x7f40a40c4250 2024-01-27T01:07:31.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40ab5dd700 1 -- 172.21.15.46:0/3556054844 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a40c5cb0 con 0x7f40a40c4250 2024-01-27T01:07:31.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40997fa700 1 -- 172.21.15.46:0/3556054844 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a00057f0 con 0x7f40a40c4250 2024-01-27T01:07:31.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.109+0000 7f40ab5dd700 1 -- 172.21.15.46:0/3556054844 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a415a970 con 0x7f40a40c4250 2024-01-27T01:07:31.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.110+0000 7f40ab5dd700 1 -- 172.21.15.46:0/3556054844 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a404c710 con 0x7f40a40c4250 2024-01-27T01:07:31.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.111+0000 7f40997fa700 1 -- 172.21.15.46:0/3556054844 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f40a0005c20 con 0x7f40a40c4250 2024-01-27T01:07:31.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.112+0000 7f40997fa700 1 --2- 172.21.15.46:0/3556054844 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4088067170 0x7f4088069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:31.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.112+0000 7f40997fa700 1 -- 172.21.15.46:0/3556054844 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f40a0085460 con 0x7f40a40c4250 2024-01-27T01:07:31.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.112+0000 7f40a8b78700 1 --2- 172.21.15.46:0/3556054844 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4088067170 0x7f4088069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:31.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.112+0000 7f40a8b78700 1 --2- 172.21.15.46:0/3556054844 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4088067170 0x7f4088069620 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f40a416cf80 tx=0x7f4094016000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:31.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.115+0000 7f40997fa700 1 -- 172.21.15.46:0/3556054844 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f40a0052270 con 0x7f40a40c4250 2024-01-27T01:07:31.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.366+0000 7f40ab5dd700 1 -- 172.21.15.46:0/3556054844 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40a4097c20 con 0x7f40a40c4250 2024-01-27T01:07:31.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.367+0000 7f40997fa700 1 -- 172.21.15.46:0/3556054844 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+4610 (secure 0 0 0) 0x7f40a0055890 con 0x7f40a40c4250 2024-01-27T01:07:31.368 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:31.369 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":133,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1114164873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1114164873},{"type":"v1","addr":"172.21.15.106:6834","nonce":1114164873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128},{"gid":9821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/702094428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":702094428},{"type":"v1","addr":"172.21.15.106:6835","nonce":702094428}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":9827,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/408985798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":408985798},{"type":"v1","addr":"172.21.15.46:6835","nonce":408985798}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:07:27.375435+0000","modified":"2024-01-27T01:07:30.945729+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11248":{"gid":11248,"name":"a","rank":0,"incarnation":129,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/1087205220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1087205220},{"type":"v1","addr":"172.21.15.46:6834","nonce":1087205220}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2024-01-27T01:07:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.369+0000 7f40ab5dd700 1 -- 172.21.15.46:0/3556054844 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4088067170 msgr2=0x7f4088069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.369+0000 7f40ab5dd700 1 --2- 172.21.15.46:0/3556054844 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4088067170 0x7f4088069620 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f40a416cf80 tx=0x7f4094016000 comp rx=0 tx=0).stop 2024-01-27T01:07:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.369+0000 7f40ab5dd700 1 -- 172.21.15.46:0/3556054844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a40c4250 msgr2=0x7f40a415a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.369+0000 7f40ab5dd700 1 --2- 172.21.15.46:0/3556054844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a40c4250 0x7f40a415a430 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f40a000c910 tx=0x7f40a0000f70 comp rx=0 tx=0).stop 2024-01-27T01:07:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.370+0000 7f40ab5dd700 1 -- 172.21.15.46:0/3556054844 shutdown_connections 2024-01-27T01:07:31.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.370+0000 7f40ab5dd700 1 --2- 172.21.15.46:0/3556054844 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4088067170 0x7f4088069620 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.370+0000 7f40ab5dd700 1 --2- 172.21.15.46:0/3556054844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a40c4250 0x7f40a415a430 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.370+0000 7f40ab5dd700 1 --2- 172.21.15.46:0/3556054844 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40a4159ad0 0x7f40a40b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.370+0000 7f40ab5dd700 1 --2- 172.21.15.46:0/3556054844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40a4159700 0x7f40a4156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.370+0000 7f40ab5dd700 1 -- 172.21.15.46:0/3556054844 >> 172.21.15.46:0/3556054844 conn(0x7f40a4097820 msgr2=0x7f40a40ccad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:31.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.370+0000 7f40ab5dd700 1 -- 172.21.15.46:0/3556054844 shutdown_connections 2024-01-27T01:07:31.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.370+0000 7f40ab5dd700 1 -- 172.21.15.46:0/3556054844 wait complete. 2024-01-27T01:07:31.371 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 133 2024-01-27T01:07:31.384 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:07:31.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.517+0000 7f3d84bed700 1 Processor -- start 2024-01-27T01:07:31.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.517+0000 7f3d84bed700 1 -- start start 2024-01-27T01:07:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.518+0000 7f3d84bed700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d800acf30 0x7f3d800c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.518+0000 7f3d84bed700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3d800ad770 0x7f3d800ab1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.518+0000 7f3d84bed700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d800c4230 con 0x7f3d800ad300 2024-01-27T01:07:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.518+0000 7f3d84bed700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d80157be0 con 0x7f3d800ad770 2024-01-27T01:07:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.518+0000 7f3d84bed700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d800cbf10 con 0x7f3d800acf30 2024-01-27T01:07:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.518+0000 7f3d7ed9d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3d800ad770 0x7f3d800ab1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.518+0000 7f3d7dd9b700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f3d800ad300 0x7f3d800ccb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:55588/0 (socket says 172.21.15.46:55588) 2024-01-27T01:07:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.518+0000 7f3d7dd9b700 1 -- 172.21.15.46:0/2976834579 learned_addr learned my addr 172.21.15.46:0/2976834579 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.518+0000 7f3d7e59c700 1 --2- 172.21.15.46:0/2976834579 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d800acf30 0x7f3d800c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.518+0000 7f3d7d59a700 1 -- 172.21.15.46:0/2976834579 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888534514 0 0) 0x7f3d800c4230 con 0x7f3d800ad300 2024-01-27T01:07:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.518+0000 7f3d7d59a700 1 -- 172.21.15.46:0/2976834579 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d68003570 con 0x7f3d800ad300 2024-01-27T01:07:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.519+0000 7f3d7ed9d700 1 -- 172.21.15.46:0/2976834579 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d800acf30 msgr2=0x7f3d800c3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.519+0000 7f3d7ed9d700 1 --2- 172.21.15.46:0/2976834579 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d800acf30 0x7f3d800c3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.519+0000 7f3d7ed9d700 1 -- 172.21.15.46:0/2976834579 >> v1:172.21.15.106:6789/0 conn(0x7f3d800ad300 legacy=0x7f3d800ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.519+0000 7f3d7ed9d700 1 -- 172.21.15.46:0/2976834579 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d800c4460 con 0x7f3d800ad770 2024-01-27T01:07:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.519+0000 7f3d7ed9d700 1 --2- 172.21.15.46:0/2976834579 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d800ad770 0x7f3d800ab1b0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009e60 tx=0x7f3d78009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a83c466af977d5b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.519+0000 7f3d7d59a700 1 -- 172.21.15.46:0/2976834579 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7800e070 con 0x7f3d800ad770 2024-01-27T01:07:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.519+0000 7f3d7d59a700 1 -- 172.21.15.46:0/2976834579 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7800bab0 con 0x7f3d800ad770 2024-01-27T01:07:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.519+0000 7f3d7d59a700 1 -- 172.21.15.46:0/2976834579 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7800be70 con 0x7f3d800ad770 2024-01-27T01:07:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.519+0000 7f3d84bed700 1 -- 172.21.15.46:0/2976834579 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d800ad770 msgr2=0x7f3d800ab1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.519+0000 7f3d84bed700 1 --2- 172.21.15.46:0/2976834579 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d800ad770 0x7f3d800ab1b0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009e60 tx=0x7f3d78009780 comp rx=0 tx=0).stop 2024-01-27T01:07:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.520+0000 7f3d84bed700 1 -- 172.21.15.46:0/2976834579 shutdown_connections 2024-01-27T01:07:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.520+0000 7f3d84bed700 1 --2- 172.21.15.46:0/2976834579 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d800ad770 0x7f3d800ab1b0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.520+0000 7f3d84bed700 1 --2- 172.21.15.46:0/2976834579 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d800acf30 0x7f3d800c3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.520+0000 7f3d84bed700 1 -- 172.21.15.46:0/2976834579 >> 172.21.15.46:0/2976834579 conn(0x7f3d80097820 msgr2=0x7f3d80068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:31.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.520+0000 7f3d84bed700 1 -- 172.21.15.46:0/2976834579 shutdown_connections 2024-01-27T01:07:31.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.520+0000 7f3d84bed700 1 -- 172.21.15.46:0/2976834579 wait complete. 2024-01-27T01:07:31.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.521+0000 7f3d84bed700 1 Processor -- start 2024-01-27T01:07:31.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.521+0000 7f3d84bed700 1 -- start start 2024-01-27T01:07:31.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.522+0000 7f3d84bed700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d800acf30 0x7f3d8016cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:31.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.522+0000 7f3d84bed700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d800ad300 0x7f3d8014d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:31.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.522+0000 7f3d84bed700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d8006a6c0 0x7f3d800b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:31.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.522+0000 7f3d84bed700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8016d0e0 con 0x7f3d8006a6c0 2024-01-27T01:07:31.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.522+0000 7f3d84bed700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d80068c70 con 0x7f3d800ad300 2024-01-27T01:07:31.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.522+0000 7f3d84bed700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8016d310 con 0x7f3d800acf30 2024-01-27T01:07:31.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.522+0000 7f3d7ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d8006a6c0 0x7f3d800b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:31.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.522+0000 7f3d7ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d8006a6c0 0x7f3d800b21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45792/0 (socket says 172.21.15.46:45792) 2024-01-27T01:07:31.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.522+0000 7f3d7ed9d700 1 -- 172.21.15.46:0/4122728654 learned_addr learned my addr 172.21.15.46:0/4122728654 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:31.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.522+0000 7f3d7dd9b700 1 --2- 172.21.15.46:0/4122728654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d800ad300 0x7f3d8014d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:31.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.522+0000 7f3d7e59c700 1 --2- 172.21.15.46:0/4122728654 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d800acf30 0x7f3d8016cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:31.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.522+0000 7f3d7ed9d700 1 -- 172.21.15.46:0/4122728654 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d800acf30 msgr2=0x7f3d8016cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:31.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.522+0000 7f3d7ed9d700 1 --2- 172.21.15.46:0/4122728654 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d800acf30 0x7f3d8016cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.522+0000 7f3d7ed9d700 1 -- 172.21.15.46:0/4122728654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d800ad300 msgr2=0x7f3d8014d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:31.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.522+0000 7f3d7ed9d700 1 --2- 172.21.15.46:0/4122728654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d800ad300 0x7f3d8014d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.522+0000 7f3d7ed9d700 1 -- 172.21.15.46:0/4122728654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d78009a20 con 0x7f3d8006a6c0 2024-01-27T01:07:31.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.522+0000 7f3d7dd9b700 1 --2- 172.21.15.46:0/4122728654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d800ad300 0x7f3d8014d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:31.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.523+0000 7f3d7ed9d700 1 --2- 172.21.15.46:0/4122728654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d8006a6c0 0x7f3d800b21f0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f3d8006ef50 tx=0x7f3d78019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:31.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.523+0000 7f3d6e7fc700 1 -- 172.21.15.46:0/4122728654 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d78012e10 con 0x7f3d8006a6c0 2024-01-27T01:07:31.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.523+0000 7f3d6e7fc700 1 -- 172.21.15.46:0/4122728654 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d78027470 con 0x7f3d8006a6c0 2024-01-27T01:07:31.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.523+0000 7f3d6e7fc700 1 -- 172.21.15.46:0/4122728654 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d78027830 con 0x7f3d8006a6c0 2024-01-27T01:07:31.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.523+0000 7f3d84bed700 1 -- 172.21.15.46:0/4122728654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d8016d5f0 con 0x7f3d8006a6c0 2024-01-27T01:07:31.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.523+0000 7f3d84bed700 1 -- 172.21.15.46:0/4122728654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d8014da50 con 0x7f3d8006a6c0 2024-01-27T01:07:31.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.524+0000 7f3d84bed700 1 -- 172.21.15.46:0/4122728654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d80068a40 con 0x7f3d8006a6c0 2024-01-27T01:07:31.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.524+0000 7f3d6e7fc700 1 -- 172.21.15.46:0/4122728654 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f3d78024070 con 0x7f3d8006a6c0 2024-01-27T01:07:31.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.525+0000 7f3d6e7fc700 1 --2- 172.21.15.46:0/4122728654 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3d60067170 0x7f3d60069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:31.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.525+0000 7f3d6e7fc700 1 -- 172.21.15.46:0/4122728654 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f3d78087520 con 0x7f3d8006a6c0 2024-01-27T01:07:31.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.525+0000 7f3d7e59c700 1 --2- 172.21.15.46:0/4122728654 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3d60067170 0x7f3d60069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:31.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.525+0000 7f3d7e59c700 1 --2- 172.21.15.46:0/4122728654 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3d60067170 0x7f3d60069620 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009db0 tx=0x7f3d700091a0 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:31.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.529+0000 7f3d6e7fc700 1 -- 172.21.15.46:0/4122728654 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3d80068a40 con 0x7f3d8006a6c0 2024-01-27T01:07:31.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.733+0000 7f3d84bed700 1 -- 172.21.15.46:0/4122728654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d80068a40 con 0x7f3d8006a6c0 2024-01-27T01:07:31.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.734+0000 7f3d6e7fc700 1 -- 172.21.15.46:0/4122728654 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v243) v1 ==== 74+0+16325 (secure 0 0 0) 0x7f3d80068a40 con 0x7f3d8006a6c0 2024-01-27T01:07:31.734 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:31.734 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":243,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:07:28.728011+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:07:25.886791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-01-27T01:07:26.340041+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:07:31.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.736+0000 7f3d84bed700 1 -- 172.21.15.46:0/4122728654 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3d60067170 msgr2=0x7f3d60069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:31.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.736+0000 7f3d84bed700 1 --2- 172.21.15.46:0/4122728654 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3d60067170 0x7f3d60069620 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009db0 tx=0x7f3d700091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.736+0000 7f3d84bed700 1 -- 172.21.15.46:0/4122728654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d8006a6c0 msgr2=0x7f3d800b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:31.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.736+0000 7f3d84bed700 1 --2- 172.21.15.46:0/4122728654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d8006a6c0 0x7f3d800b21f0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f3d8006ef50 tx=0x7f3d78019000 comp rx=0 tx=0).stop 2024-01-27T01:07:31.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.736+0000 7f3d84bed700 1 -- 172.21.15.46:0/4122728654 shutdown_connections 2024-01-27T01:07:31.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.736+0000 7f3d84bed700 1 --2- 172.21.15.46:0/4122728654 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3d60067170 0x7f3d60069620 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.736+0000 7f3d84bed700 1 --2- 172.21.15.46:0/4122728654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d8006a6c0 0x7f3d800b21f0 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.736+0000 7f3d84bed700 1 --2- 172.21.15.46:0/4122728654 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d800acf30 0x7f3d8016cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.736+0000 7f3d84bed700 1 --2- 172.21.15.46:0/4122728654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d800ad300 0x7f3d8014d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.736+0000 7f3d84bed700 1 -- 172.21.15.46:0/4122728654 >> 172.21.15.46:0/4122728654 conn(0x7f3d80097820 msgr2=0x7f3d800cc330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:31.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.736+0000 7f3d84bed700 1 -- 172.21.15.46:0/4122728654 shutdown_connections 2024-01-27T01:07:31.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.736+0000 7f3d84bed700 1 -- 172.21.15.46:0/4122728654 wait complete. 2024-01-27T01:07:31.750 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-27T01:07:31.750 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:07:31.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.883+0000 7f92717c8700 1 Processor -- start 2024-01-27T01:07:31.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.883+0000 7f92717c8700 1 -- start start 2024-01-27T01:07:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.884+0000 7f92717c8700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f926c1596c0 0x7f926c06bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.884+0000 7f92717c8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926c06e130 con 0x7f926c15d460 2024-01-27T01:07:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.884+0000 7f92717c8700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926c15d8f0 con 0x7f926c159a90 2024-01-27T01:07:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.884+0000 7f92717c8700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f926c0c98c0 con 0x7f926c1596c0 2024-01-27T01:07:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.884+0000 7f926a7fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f926c15d460 0x7f926c06d310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45528/0 (socket says 172.21.15.46:45528) 2024-01-27T01:07:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.884+0000 7f926a7fc700 1 -- 172.21.15.46:0/965487099 learned_addr learned my addr 172.21.15.46:0/965487099 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.884+0000 7f926affd700 1 --2- 172.21.15.46:0/965487099 >> v2:172.21.15.106:3300/0 conn(0x7f926c1596c0 0x7f926c06bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.884+0000 7f9269ffb700 1 -- 172.21.15.46:0/965487099 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005679797 0 0) 0x7f926c06e130 con 0x7f926c15d460 2024-01-27T01:07:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.885+0000 7f9269ffb700 1 -- 172.21.15.46:0/965487099 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9254003570 con 0x7f926c15d460 2024-01-27T01:07:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.885+0000 7f926affd700 1 -- 172.21.15.46:0/965487099 >> v1:172.21.15.106:6789/0 conn(0x7f926c159a90 legacy=0x7f926c06da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.885+0000 7f9269ffb700 1 -- 172.21.15.46:0/965487099 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3088012318 0 0) 0x7f9254003570 con 0x7f926c15d460 2024-01-27T01:07:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.885+0000 7f926affd700 1 -- 172.21.15.46:0/965487099 >> v1:172.21.15.46:6789/0 conn(0x7f926c15d460 legacy=0x7f926c06d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.885+0000 7f926affd700 1 -- 172.21.15.46:0/965487099 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c15dbd0 con 0x7f926c1596c0 2024-01-27T01:07:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.885+0000 7f926affd700 1 --2- 172.21.15.46:0/965487099 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f926c1596c0 0x7f926c06bcd0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f925c009e60 tx=0x7f925c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19b1c5436554ec05 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.885+0000 7f9269ffb700 1 -- 172.21.15.46:0/965487099 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925c00e070 con 0x7f926c1596c0 2024-01-27T01:07:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.885+0000 7f9269ffb700 1 -- 172.21.15.46:0/965487099 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925c00b850 con 0x7f926c1596c0 2024-01-27T01:07:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.885+0000 7f9269ffb700 1 -- 172.21.15.46:0/965487099 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9260001470 con 0x7f926c1596c0 2024-01-27T01:07:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.886+0000 7f92717c8700 1 -- 172.21.15.46:0/965487099 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f926c1596c0 msgr2=0x7f926c06bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.886+0000 7f92717c8700 1 --2- 172.21.15.46:0/965487099 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f926c1596c0 0x7f926c06bcd0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f925c009e60 tx=0x7f925c009780 comp rx=0 tx=0).stop 2024-01-27T01:07:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.886+0000 7f92717c8700 1 -- 172.21.15.46:0/965487099 shutdown_connections 2024-01-27T01:07:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.886+0000 7f92717c8700 1 --2- 172.21.15.46:0/965487099 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f926c1596c0 0x7f926c06bcd0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.886+0000 7f92717c8700 1 -- 172.21.15.46:0/965487099 >> 172.21.15.46:0/965487099 conn(0x7f926c0ce530 msgr2=0x7f926c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:31.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.886+0000 7f92717c8700 1 -- 172.21.15.46:0/965487099 shutdown_connections 2024-01-27T01:07:31.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.886+0000 7f92717c8700 1 -- 172.21.15.46:0/965487099 wait complete. 2024-01-27T01:07:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.887+0000 7f92717c8700 1 Processor -- start 2024-01-27T01:07:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.887+0000 7f92717c8700 1 -- start start 2024-01-27T01:07:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.888+0000 7f92717c8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f926c1596c0 0x7f926c0b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.888+0000 7f92717c8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f926c159a90 0x7f926c15af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.888+0000 7f92717c8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f926c15d460 0x7f926c14fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.888+0000 7f92717c8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c1500c0 con 0x7f926c159a90 2024-01-27T01:07:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.888+0000 7f92717c8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c1587e0 con 0x7f926c15d460 2024-01-27T01:07:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.888+0000 7f92717c8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926c15b590 con 0x7f926c1596c0 2024-01-27T01:07:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.888+0000 7f926a7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f926c159a90 0x7f926c15af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.888+0000 7f926a7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f926c159a90 0x7f926c15af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45798/0 (socket says 172.21.15.46:45798) 2024-01-27T01:07:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.888+0000 7f926a7fc700 1 -- 172.21.15.46:0/2107564097 learned_addr learned my addr 172.21.15.46:0/2107564097 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.888+0000 7f926affd700 1 --2- 172.21.15.46:0/2107564097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f926c1596c0 0x7f926c0b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.888+0000 7f926b7fe700 1 --2- 172.21.15.46:0/2107564097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f926c15d460 0x7f926c14fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.889+0000 7f926a7fc700 1 -- 172.21.15.46:0/2107564097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f926c1596c0 msgr2=0x7f926c0b34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.889+0000 7f926a7fc700 1 --2- 172.21.15.46:0/2107564097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f926c1596c0 0x7f926c0b34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.889+0000 7f926a7fc700 1 -- 172.21.15.46:0/2107564097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f926c15d460 msgr2=0x7f926c14fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.889+0000 7f926a7fc700 1 --2- 172.21.15.46:0/2107564097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f926c15d460 0x7f926c14fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.889+0000 7f926a7fc700 1 -- 172.21.15.46:0/2107564097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c009a20 con 0x7f926c159a90 2024-01-27T01:07:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.889+0000 7f926b7fe700 1 --2- 172.21.15.46:0/2107564097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f926c15d460 0x7f926c14fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.889+0000 7f926a7fc700 1 --2- 172.21.15.46:0/2107564097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f926c159a90 0x7f926c15af00 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f926000cb90 tx=0x7f9260001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.889+0000 7f9252ffd700 1 -- 172.21.15.46:0/2107564097 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9260007150 con 0x7f926c159a90 2024-01-27T01:07:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.889+0000 7f9252ffd700 1 -- 172.21.15.46:0/2107564097 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92600072d0 con 0x7f926c159a90 2024-01-27T01:07:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.889+0000 7f9252ffd700 1 -- 172.21.15.46:0/2107564097 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9260007690 con 0x7f926c159a90 2024-01-27T01:07:31.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.889+0000 7f92717c8700 1 -- 172.21.15.46:0/2107564097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926c15b870 con 0x7f926c159a90 2024-01-27T01:07:31.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.889+0000 7f92717c8700 1 -- 172.21.15.46:0/2107564097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f926c15be30 con 0x7f926c159a90 2024-01-27T01:07:31.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.890+0000 7f92717c8700 1 -- 172.21.15.46:0/2107564097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f926c0aeb10 con 0x7f926c159a90 2024-01-27T01:07:31.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.891+0000 7f9252ffd700 1 -- 172.21.15.46:0/2107564097 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f9260004a40 con 0x7f926c159a90 2024-01-27T01:07:31.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.891+0000 7f9252ffd700 1 --2- 172.21.15.46:0/2107564097 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9258067170 0x7f9258069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:31.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.891+0000 7f9252ffd700 1 -- 172.21.15.46:0/2107564097 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f9260084090 con 0x7f926c159a90 2024-01-27T01:07:31.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.891+0000 7f926affd700 1 --2- 172.21.15.46:0/2107564097 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9258067170 0x7f9258069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:31.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.892+0000 7f926affd700 1 --2- 172.21.15.46:0/2107564097 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9258067170 0x7f9258069620 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f925c0099f0 tx=0x7f925c017000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:31.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:31.896+0000 7f9252ffd700 1 -- 172.21.15.46:0/2107564097 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f926c0aeb10 con 0x7f926c159a90 2024-01-27T01:07:32.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.094+0000 7f92717c8700 1 -- 172.21.15.46:0/2107564097 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f926c0aeb10 con 0x7f9258067170 2024-01-27T01:07:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.104+0000 7f9252ffd700 1 -- 172.21.15.46:0/2107564097 <== mgr.9590 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289836 (secure 0 0 0) 0x7f926c0aeb10 con 0x7f9258067170 2024-01-27T01:07:32.109 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.107+0000 7f92717c8700 1 -- 172.21.15.46:0/2107564097 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9258067170 msgr2=0x7f9258069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.107+0000 7f92717c8700 1 --2- 172.21.15.46:0/2107564097 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9258067170 0x7f9258069620 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f925c0099f0 tx=0x7f925c017000 comp rx=0 tx=0).stop 2024-01-27T01:07:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.107+0000 7f92717c8700 1 -- 172.21.15.46:0/2107564097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f926c159a90 msgr2=0x7f926c15af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.107+0000 7f92717c8700 1 --2- 172.21.15.46:0/2107564097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f926c159a90 0x7f926c15af00 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f926000cb90 tx=0x7f9260001a30 comp rx=0 tx=0).stop 2024-01-27T01:07:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.107+0000 7f92717c8700 1 -- 172.21.15.46:0/2107564097 shutdown_connections 2024-01-27T01:07:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.107+0000 7f92717c8700 1 --2- 172.21.15.46:0/2107564097 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9258067170 0x7f9258069620 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.107+0000 7f92717c8700 1 --2- 172.21.15.46:0/2107564097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f926c15d460 0x7f926c14fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.107+0000 7f92717c8700 1 --2- 172.21.15.46:0/2107564097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f926c1596c0 0x7f926c0b34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.107+0000 7f92717c8700 1 --2- 172.21.15.46:0/2107564097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f926c159a90 0x7f926c15af00 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.107+0000 7f92717c8700 1 -- 172.21.15.46:0/2107564097 >> 172.21.15.46:0/2107564097 conn(0x7f926c0ce530 msgr2=0x7f926c0cd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.107+0000 7f92717c8700 1 -- 172.21.15.46:0/2107564097 shutdown_connections 2024-01-27T01:07:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.107+0000 7f92717c8700 1 -- 172.21.15.46:0/2107564097 wait complete. 2024-01-27T01:07:32.112 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:07:32.121 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-01-27T01:07:31.870335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2456,"num_read_kb":5220,"num_write":714,"num_write_kb":3475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6581,"ondisk_log_size":6581,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2456616,"kb_used_data":25064,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752518104,"statfs":{"total":773094113280,"available":770578538496,"internally_reserved":0,"allocated":25665536,"data_stored":14108052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.946518"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.748807+0000","last_change":"2024-01-27T01:07:27.744924+0000","last_active":"2024-01-27T01:07:27.748807+0000","last_peered":"2024-01-27T01:07:27.748807+0000","last_clean":"2024-01-27T01:07:27.748807+0000","last_became_active":"2024-01-27T01:07:27.744624+0000","last_became_peered":"2024-01-27T01:07:27.744624+0000","last_unstale":"2024-01-27T01:07:27.748807+0000","last_undegraded":"2024-01-27T01:07:27.748807+0000","last_fullsized":"2024-01-27T01:07:27.748807+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:44:00.209505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.756780+0000","last_change":"2024-01-27T01:07:27.749698+0000","last_active":"2024-01-27T01:07:27.756780+0000","last_peered":"2024-01-27T01:07:27.756780+0000","last_clean":"2024-01-27T01:07:27.756780+0000","last_became_active":"2024-01-27T01:07:27.749173+0000","last_became_peered":"2024-01-27T01:07:27.749173+0000","last_unstale":"2024-01-27T01:07:27.756780+0000","last_undegraded":"2024-01-27T01:07:27.756780+0000","last_fullsized":"2024-01-27T01:07:27.756780+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:52:40.618198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.741674+0000","last_change":"2024-01-27T01:07:27.741674+0000","last_active":"2024-01-27T01:07:27.741674+0000","last_peered":"2024-01-27T01:07:27.741674+0000","last_clean":"2024-01-27T01:07:27.741674+0000","last_became_active":"2024-01-27T01:07:27.741351+0000","last_became_peered":"2024-01-27T01:07:27.741351+0000","last_unstale":"2024-01-27T01:07:27.741674+0000","last_undegraded":"2024-01-27T01:07:27.741674+0000","last_fullsized":"2024-01-27T01:07:27.741674+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:48:18.465014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.749174+0000","last_change":"2024-01-27T01:07:27.749174+0000","last_active":"2024-01-27T01:07:27.749174+0000","last_peered":"2024-01-27T01:07:27.749174+0000","last_clean":"2024-01-27T01:07:27.749174+0000","last_became_active":"2024-01-27T01:07:27.748883+0000","last_became_peered":"2024-01-27T01:07:27.748883+0000","last_unstale":"2024-01-27T01:07:27.749174+0000","last_undegraded":"2024-01-27T01:07:27.749174+0000","last_fullsized":"2024-01-27T01:07:27.749174+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:03:47.280261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.749946+0000","last_change":"2024-01-27T01:07:27.749946+0000","last_active":"2024-01-27T01:07:27.749946+0000","last_peered":"2024-01-27T01:07:27.749946+0000","last_clean":"2024-01-27T01:07:27.749946+0000","last_became_active":"2024-01-27T01:07:27.749653+0000","last_became_peered":"2024-01-27T01:07:27.749653+0000","last_unstale":"2024-01-27T01:07:27.749946+0000","last_undegraded":"2024-01-27T01:07:27.749946+0000","last_fullsized":"2024-01-27T01:07:27.749946+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:37.006113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.755473+0000","last_change":"2024-01-27T01:07:27.753636+0000","last_active":"2024-01-27T01:07:27.755473+0000","last_peered":"2024-01-27T01:07:27.755473+0000","last_clean":"2024-01-27T01:07:27.755473+0000","last_became_active":"2024-01-27T01:07:27.753305+0000","last_became_peered":"2024-01-27T01:07:27.753305+0000","last_unstale":"2024-01-27T01:07:27.755473+0000","last_undegraded":"2024-01-27T01:07:27.755473+0000","last_fullsized":"2024-01-27T01:07:27.755473+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:28:31.571123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.743608+0000","last_change":"2024-01-27T01:07:27.743608+0000","last_active":"2024-01-27T01:07:27.743608+0000","last_peered":"2024-01-27T01:07:27.743608+0000","last_clean":"2024-01-27T01:07:27.743608+0000","last_became_active":"2024-01-27T01:07:27.743324+0000","last_became_peered":"2024-01-27T01:07:27.743324+0000","last_unstale":"2024-01-27T01:07:27.743608+0000","last_undegraded":"2024-01-27T01:07:27.743608+0000","last_fullsized":"2024-01-27T01:07:27.743608+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:25:38.146534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.744506+0000","last_change":"2024-01-27T01:07:27.744506+0000","last_active":"2024-01-27T01:07:27.744506+0000","last_peered":"2024-01-27T01:07:27.744506+0000","last_clean":"2024-01-27T01:07:27.744506+0000","last_became_active":"2024-01-27T01:07:27.744222+0000","last_became_peered":"2024-01-27T01:07:27.744222+0000","last_unstale":"2024-01-27T01:07:27.744506+0000","last_undegraded":"2024-01-27T01:07:27.744506+0000","last_fullsized":"2024-01-27T01:07:27.744506+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:24:38.456700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.754529+0000","last_change":"2024-01-27T01:07:27.753284+0000","last_active":"2024-01-27T01:07:27.754529+0000","last_peered":"2024-01-27T01:07:27.754529+0000","last_clean":"2024-01-27T01:07:27.754529+0000","last_became_active":"2024-01-27T01:07:27.752819+0000","last_became_peered":"2024-01-27T01:07:27.752819+0000","last_unstale":"2024-01-27T01:07:27.754529+0000","last_undegraded":"2024-01-27T01:07:27.754529+0000","last_fullsized":"2024-01-27T01:07:27.754529+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:58:33.718993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.743902+0000","last_change":"2024-01-27T01:07:27.743902+0000","last_active":"2024-01-27T01:07:27.743902+0000","last_peered":"2024-01-27T01:07:27.743902+0000","last_clean":"2024-01-27T01:07:27.743902+0000","last_became_active":"2024-01-27T01:07:27.743270+0000","last_became_peered":"2024-01-27T01:07:27.743270+0000","last_unstale":"2024-01-27T01:07:27.743902+0000","last_undegraded":"2024-01-27T01:07:27.743902+0000","last_fullsized":"2024-01-27T01:07:27.743902+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:24:38.456700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.743746+0000","last_change":"2024-01-27T01:07:27.743746+0000","last_active":"2024-01-27T01:07:27.743746+0000","last_peered":"2024-01-27T01:07:27.743746+0000","last_clean":"2024-01-27T01:07:27.743746+0000","last_became_active":"2024-01-27T01:07:27.743414+0000","last_became_peered":"2024-01-27T01:07:27.743414+0000","last_unstale":"2024-01-27T01:07:27.743746+0000","last_undegraded":"2024-01-27T01:07:27.743746+0000","last_fullsized":"2024-01-27T01:07:27.743746+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:37.006113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.750780+0000","last_change":"2024-01-27T01:07:27.748096+0000","last_active":"2024-01-27T01:07:27.750780+0000","last_peered":"2024-01-27T01:07:27.750780+0000","last_clean":"2024-01-27T01:07:27.750780+0000","last_became_active":"2024-01-27T01:07:27.747801+0000","last_became_peered":"2024-01-27T01:07:27.747801+0000","last_unstale":"2024-01-27T01:07:27.750780+0000","last_undegraded":"2024-01-27T01:07:27.750780+0000","last_fullsized":"2024-01-27T01:07:27.750780+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:20:52.644616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.746806+0000","last_change":"2024-01-27T01:07:27.746806+0000","last_active":"2024-01-27T01:07:27.746806+0000","last_peered":"2024-01-27T01:07:27.746806+0000","last_clean":"2024-01-27T01:07:27.746806+0000","last_became_active":"2024-01-27T01:07:27.745168+0000","last_became_peered":"2024-01-27T01:07:27.745168+0000","last_unstale":"2024-01-27T01:07:27.746806+0000","last_undegraded":"2024-01-27T01:07:27.746806+0000","last_fullsized":"2024-01-27T01:07:27.746806+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:34:21.004352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.753177+0000","last_change":"2024-01-27T01:07:27.749049+0000","last_active":"2024-01-27T01:07:27.753177+0000","last_peered":"2024-01-27T01:07:27.753177+0000","last_clean":"2024-01-27T01:07:27.753177+0000","last_became_active":"2024-01-27T01:07:27.748736+0000","last_became_peered":"2024-01-27T01:07:27.748736+0000","last_unstale":"2024-01-27T01:07:27.753177+0000","last_undegraded":"2024-01-27T01:07:27.753177+0000","last_fullsized":"2024-01-27T01:07:27.753177+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:23:10.979296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.755236+0000","last_change":"2024-01-27T01:07:27.755236+0000","last_active":"2024-01-27T01:07:27.755236+0000","last_peered":"2024-01-27T01:07:27.755236+0000","last_clean":"2024-01-27T01:07:27.755236+0000","last_became_active":"2024-01-27T01:07:27.754945+0000","last_became_peered":"2024-01-27T01:07:27.754945+0000","last_unstale":"2024-01-27T01:07:27.755236+0000","last_undegraded":"2024-01-27T01:07:27.755236+0000","last_fullsized":"2024-01-27T01:07:27.755236+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:07:48.619221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.745495+0000","last_change":"2024-01-27T01:07:27.745495+0000","last_active":"2024-01-27T01:07:27.745495+0000","last_peered":"2024-01-27T01:07:27.745495+0000","last_clean":"2024-01-27T01:07:27.745495+0000","last_became_active":"2024-01-27T01:07:27.744986+0000","last_became_peered":"2024-01-27T01:07:27.744986+0000","last_unstale":"2024-01-27T01:07:27.745495+0000","last_undegraded":"2024-01-27T01:07:27.745495+0000","last_fullsized":"2024-01-27T01:07:27.745495+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:08.374701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.749357+0000","last_change":"2024-01-27T01:07:27.749357+0000","last_active":"2024-01-27T01:07:27.749357+0000","last_peered":"2024-01-27T01:07:27.749357+0000","last_clean":"2024-01-27T01:07:27.749357+0000","last_became_active":"2024-01-27T01:07:27.745281+0000","last_became_peered":"2024-01-27T01:07:27.745281+0000","last_unstale":"2024-01-27T01:07:27.749357+0000","last_undegraded":"2024-01-27T01:07:27.749357+0000","last_fullsized":"2024-01-27T01:07:27.749357+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:11.619364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.736143+0000","last_change":"2024-01-27T01:07:27.736143+0000","last_active":"2024-01-27T01:07:27.736143+0000","last_peered":"2024-01-27T01:07:27.736143+0000","last_clean":"2024-01-27T01:07:27.736143+0000","last_became_active":"2024-01-27T01:07:27.735824+0000","last_became_peered":"2024-01-27T01:07:27.735824+0000","last_unstale":"2024-01-27T01:07:27.736143+0000","last_undegraded":"2024-01-27T01:07:27.736143+0000","last_fullsized":"2024-01-27T01:07:27.736143+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:05.536246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.756376+0000","last_change":"2024-01-27T01:07:27.756376+0000","last_active":"2024-01-27T01:07:27.756376+0000","last_peered":"2024-01-27T01:07:27.756376+0000","last_clean":"2024-01-27T01:07:27.756376+0000","last_became_active":"2024-01-27T01:07:27.756152+0000","last_became_peered":"2024-01-27T01:07:27.756152+0000","last_unstale":"2024-01-27T01:07:27.756376+0000","last_undegraded":"2024-01-27T01:07:27.756376+0000","last_fullsized":"2024-01-27T01:07:27.756376+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:02:04.959467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.758763+0000","last_change":"2024-01-27T01:07:27.758252+0000","last_active":"2024-01-27T01:07:27.758763+0000","last_peered":"2024-01-27T01:07:27.758763+0000","last_clean":"2024-01-27T01:07:27.758763+0000","last_became_active":"2024-01-27T01:07:27.757929+0000","last_became_peered":"2024-01-27T01:07:27.757929+0000","last_unstale":"2024-01-27T01:07:27.758763+0000","last_undegraded":"2024-01-27T01:07:27.758763+0000","last_fullsized":"2024-01-27T01:07:27.758763+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:30.518041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.752626+0000","last_change":"2024-01-27T01:07:27.752626+0000","last_active":"2024-01-27T01:07:27.752626+0000","last_peered":"2024-01-27T01:07:27.752626+0000","last_clean":"2024-01-27T01:07:27.752626+0000","last_became_active":"2024-01-27T01:07:27.752363+0000","last_became_peered":"2024-01-27T01:07:27.752363+0000","last_unstale":"2024-01-27T01:07:27.752626+0000","last_undegraded":"2024-01-27T01:07:27.752626+0000","last_fullsized":"2024-01-27T01:07:27.752626+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:03:44.824769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.748467+0000","last_change":"2024-01-27T01:07:27.748467+0000","last_active":"2024-01-27T01:07:27.748467+0000","last_peered":"2024-01-27T01:07:27.748467+0000","last_clean":"2024-01-27T01:07:27.748467+0000","last_became_active":"2024-01-27T01:07:27.748100+0000","last_became_peered":"2024-01-27T01:07:27.748100+0000","last_unstale":"2024-01-27T01:07:27.748467+0000","last_undegraded":"2024-01-27T01:07:27.748467+0000","last_fullsized":"2024-01-27T01:07:27.748467+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:38:29.701960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.755439+0000","last_change":"2024-01-27T01:07:27.755439+0000","last_active":"2024-01-27T01:07:27.755439+0000","last_peered":"2024-01-27T01:07:27.755439+0000","last_clean":"2024-01-27T01:07:27.755439+0000","last_became_active":"2024-01-27T01:07:27.755126+0000","last_became_peered":"2024-01-27T01:07:27.755126+0000","last_unstale":"2024-01-27T01:07:27.755439+0000","last_undegraded":"2024-01-27T01:07:27.755439+0000","last_fullsized":"2024-01-27T01:07:27.755439+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:43:37.067000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.745155+0000","last_change":"2024-01-27T01:07:27.745155+0000","last_active":"2024-01-27T01:07:27.745155+0000","last_peered":"2024-01-27T01:07:27.745155+0000","last_clean":"2024-01-27T01:07:27.745155+0000","last_became_active":"2024-01-27T01:07:27.744562+0000","last_became_peered":"2024-01-27T01:07:27.744562+0000","last_unstale":"2024-01-27T01:07:27.745155+0000","last_undegraded":"2024-01-27T01:07:27.745155+0000","last_fullsized":"2024-01-27T01:07:27.745155+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:28:42.362768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.749426+0000","last_change":"2024-01-27T01:07:27.745357+0000","last_active":"2024-01-27T01:07:27.749426+0000","last_peered":"2024-01-27T01:07:27.749426+0000","last_clean":"2024-01-27T01:07:27.749426+0000","last_became_active":"2024-01-27T01:07:27.745075+0000","last_became_peered":"2024-01-27T01:07:27.745075+0000","last_unstale":"2024-01-27T01:07:27.749426+0000","last_undegraded":"2024-01-27T01:07:27.749426+0000","last_fullsized":"2024-01-27T01:07:27.749426+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:37:01.620442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.744187+0000","last_change":"2024-01-27T01:07:27.744187+0000","last_active":"2024-01-27T01:07:27.744187+0000","last_peered":"2024-01-27T01:07:27.744187+0000","last_clean":"2024-01-27T01:07:27.744187+0000","last_became_active":"2024-01-27T01:07:27.743889+0000","last_became_peered":"2024-01-27T01:07:27.743889+0000","last_unstale":"2024-01-27T01:07:27.744187+0000","last_undegraded":"2024-01-27T01:07:27.744187+0000","last_fullsized":"2024-01-27T01:07:27.744187+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:11.619364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.743304+0000","last_change":"2024-01-27T01:07:27.743304+0000","last_active":"2024-01-27T01:07:27.743304+0000","last_peered":"2024-01-27T01:07:27.743304+0000","last_clean":"2024-01-27T01:07:27.743304+0000","last_became_active":"2024-01-27T01:07:27.742783+0000","last_became_peered":"2024-01-27T01:07:27.742783+0000","last_unstale":"2024-01-27T01:07:27.743304+0000","last_undegraded":"2024-01-27T01:07:27.743304+0000","last_fullsized":"2024-01-27T01:07:27.743304+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:45:07.998194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.748843+0000","last_change":"2024-01-27T01:07:27.746921+0000","last_active":"2024-01-27T01:07:27.748843+0000","last_peered":"2024-01-27T01:07:27.748843+0000","last_clean":"2024-01-27T01:07:27.748843+0000","last_became_active":"2024-01-27T01:07:27.746630+0000","last_became_peered":"2024-01-27T01:07:27.746630+0000","last_unstale":"2024-01-27T01:07:27.748843+0000","last_undegraded":"2024-01-27T01:07:27.748843+0000","last_fullsized":"2024-01-27T01:07:27.748843+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:03:41.111732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.753223+0000","last_change":"2024-01-27T01:07:27.749637+0000","last_active":"2024-01-27T01:07:27.753223+0000","last_peered":"2024-01-27T01:07:27.753223+0000","last_clean":"2024-01-27T01:07:27.753223+0000","last_became_active":"2024-01-27T01:07:27.749170+0000","last_became_peered":"2024-01-27T01:07:27.749170+0000","last_unstale":"2024-01-27T01:07:27.753223+0000","last_undegraded":"2024-01-27T01:07:27.753223+0000","last_fullsized":"2024-01-27T01:07:27.753223+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:10:06.064380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.752984+0000","last_change":"2024-01-27T01:07:27.748872+0000","last_active":"2024-01-27T01:07:27.752984+0000","last_peered":"2024-01-27T01:07:27.752984+0000","last_clean":"2024-01-27T01:07:27.752984+0000","last_became_active":"2024-01-27T01:07:27.748549+0000","last_became_peered":"2024-01-27T01:07:27.748549+0000","last_unstale":"2024-01-27T01:07:27.752984+0000","last_undegraded":"2024-01-27T01:07:27.752984+0000","last_fullsized":"2024-01-27T01:07:27.752984+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:58:33.718993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.757752+0000","last_change":"2024-01-27T01:07:27.755023+0000","last_active":"2024-01-27T01:07:27.757752+0000","last_peered":"2024-01-27T01:07:27.757752+0000","last_clean":"2024-01-27T01:07:27.757752+0000","last_became_active":"2024-01-27T01:07:27.754304+0000","last_became_peered":"2024-01-27T01:07:27.754304+0000","last_unstale":"2024-01-27T01:07:27.757752+0000","last_undegraded":"2024-01-27T01:07:27.757752+0000","last_fullsized":"2024-01-27T01:07:27.757752+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:28:12.245965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.751905+0000","last_change":"2024-01-27T01:07:27.747314+0000","last_active":"2024-01-27T01:07:27.751905+0000","last_peered":"2024-01-27T01:07:27.751905+0000","last_clean":"2024-01-27T01:07:27.751905+0000","last_became_active":"2024-01-27T01:07:27.746947+0000","last_became_peered":"2024-01-27T01:07:27.746947+0000","last_unstale":"2024-01-27T01:07:27.751905+0000","last_undegraded":"2024-01-27T01:07:27.751905+0000","last_fullsized":"2024-01-27T01:07:27.751905+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:25:01.603669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.758346+0000","last_change":"2024-01-27T01:07:27.755462+0000","last_active":"2024-01-27T01:07:27.758346+0000","last_peered":"2024-01-27T01:07:27.758346+0000","last_clean":"2024-01-27T01:07:27.758346+0000","last_became_active":"2024-01-27T01:07:27.755131+0000","last_became_peered":"2024-01-27T01:07:27.755131+0000","last_unstale":"2024-01-27T01:07:27.758346+0000","last_undegraded":"2024-01-27T01:07:27.758346+0000","last_fullsized":"2024-01-27T01:07:27.758346+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:32:22.760237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"242'4","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.945348+0000","last_change":"2024-01-27T01:07:26.950037+0000","last_active":"2024-01-27T01:07:27.945348+0000","last_peered":"2024-01-27T01:07:27.945348+0000","last_clean":"2024-01-27T01:07:27.945348+0000","last_became_active":"2024-01-27T01:07:26.949604+0000","last_became_peered":"2024-01-27T01:07:26.949604+0000","last_unstale":"2024-01-27T01:07:27.945348+0000","last_undegraded":"2024-01-27T01:07:27.945348+0000","last_fullsized":"2024-01-27T01:07:27.945348+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:01:17.838521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.755690+0000","last_change":"2024-01-27T01:07:27.755690+0000","last_active":"2024-01-27T01:07:27.755690+0000","last_peered":"2024-01-27T01:07:27.755690+0000","last_clean":"2024-01-27T01:07:27.755690+0000","last_became_active":"2024-01-27T01:07:27.755402+0000","last_became_peered":"2024-01-27T01:07:27.755402+0000","last_unstale":"2024-01-27T01:07:27.755690+0000","last_undegraded":"2024-01-27T01:07:27.755690+0000","last_fullsized":"2024-01-27T01:07:27.755690+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:25:59.841939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"242'2","reported_seq":11,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.943676+0000","last_change":"2024-01-27T01:07:26.951267+0000","last_active":"2024-01-27T01:07:27.943676+0000","last_peered":"2024-01-27T01:07:27.943676+0000","last_clean":"2024-01-27T01:07:27.943676+0000","last_became_active":"2024-01-27T01:07:26.950967+0000","last_became_peered":"2024-01-27T01:07:26.950967+0000","last_unstale":"2024-01-27T01:07:27.943676+0000","last_undegraded":"2024-01-27T01:07:27.943676+0000","last_fullsized":"2024-01-27T01:07:27.943676+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:37:00.613294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.753950+0000","last_change":"2024-01-27T01:07:27.752814+0000","last_active":"2024-01-27T01:07:27.753950+0000","last_peered":"2024-01-27T01:07:27.753950+0000","last_clean":"2024-01-27T01:07:27.753950+0000","last_became_active":"2024-01-27T01:07:27.752459+0000","last_became_peered":"2024-01-27T01:07:27.752459+0000","last_unstale":"2024-01-27T01:07:27.753950+0000","last_undegraded":"2024-01-27T01:07:27.753950+0000","last_fullsized":"2024-01-27T01:07:27.753950+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:44:32.042391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"242'2","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.947326+0000","last_change":"2024-01-27T01:07:26.945958+0000","last_active":"2024-01-27T01:07:27.947326+0000","last_peered":"2024-01-27T01:07:27.947326+0000","last_clean":"2024-01-27T01:07:27.947326+0000","last_became_active":"2024-01-27T01:07:26.945671+0000","last_became_peered":"2024-01-27T01:07:26.945671+0000","last_unstale":"2024-01-27T01:07:27.947326+0000","last_undegraded":"2024-01-27T01:07:27.947326+0000","last_fullsized":"2024-01-27T01:07:27.947326+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:20.901463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.757603+0000","last_change":"2024-01-27T01:07:27.756421+0000","last_active":"2024-01-27T01:07:27.757603+0000","last_peered":"2024-01-27T01:07:27.757603+0000","last_clean":"2024-01-27T01:07:27.757603+0000","last_became_active":"2024-01-27T01:07:27.756102+0000","last_became_peered":"2024-01-27T01:07:27.756102+0000","last_unstale":"2024-01-27T01:07:27.757603+0000","last_undegraded":"2024-01-27T01:07:27.757603+0000","last_fullsized":"2024-01-27T01:07:27.757603+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:23:49.531160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"242'1","reported_seq":10,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.747354+0000","last_change":"2024-01-27T01:07:26.957169+0000","last_active":"2024-01-27T01:07:27.747354+0000","last_peered":"2024-01-27T01:07:27.747354+0000","last_clean":"2024-01-27T01:07:27.747354+0000","last_became_active":"2024-01-27T01:07:26.956902+0000","last_became_peered":"2024-01-27T01:07:26.956902+0000","last_unstale":"2024-01-27T01:07:27.747354+0000","last_undegraded":"2024-01-27T01:07:27.747354+0000","last_fullsized":"2024-01-27T01:07:27.747354+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:27:55.458182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.752682+0000","last_change":"2024-01-27T01:07:27.752682+0000","last_active":"2024-01-27T01:07:27.752682+0000","last_peered":"2024-01-27T01:07:27.752682+0000","last_clean":"2024-01-27T01:07:27.752682+0000","last_became_active":"2024-01-27T01:07:27.752401+0000","last_became_peered":"2024-01-27T01:07:27.752401+0000","last_unstale":"2024-01-27T01:07:27.752682+0000","last_undegraded":"2024-01-27T01:07:27.752682+0000","last_fullsized":"2024-01-27T01:07:27.752682+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:55:40.962728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"242'1","reported_seq":11,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.943898+0000","last_change":"2024-01-27T01:07:26.951702+0000","last_active":"2024-01-27T01:07:27.943898+0000","last_peered":"2024-01-27T01:07:27.943898+0000","last_clean":"2024-01-27T01:07:27.943898+0000","last_became_active":"2024-01-27T01:07:26.951367+0000","last_became_peered":"2024-01-27T01:07:26.951367+0000","last_unstale":"2024-01-27T01:07:27.943898+0000","last_undegraded":"2024-01-27T01:07:27.943898+0000","last_fullsized":"2024-01-27T01:07:27.943898+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:25:00.596520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.750189+0000","last_change":"2024-01-27T01:07:27.750189+0000","last_active":"2024-01-27T01:07:27.750189+0000","last_peered":"2024-01-27T01:07:27.750189+0000","last_clean":"2024-01-27T01:07:27.750189+0000","last_became_active":"2024-01-27T01:07:27.749867+0000","last_became_peered":"2024-01-27T01:07:27.749867+0000","last_unstale":"2024-01-27T01:07:27.750189+0000","last_undegraded":"2024-01-27T01:07:27.750189+0000","last_fullsized":"2024-01-27T01:07:27.750189+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:39:46.126956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.958068+0000","last_change":"2024-01-27T01:07:26.954992+0000","last_active":"2024-01-27T01:07:26.958068+0000","last_peered":"2024-01-27T01:07:26.958068+0000","last_clean":"2024-01-27T01:07:26.958068+0000","last_became_active":"2024-01-27T01:07:26.954701+0000","last_became_peered":"2024-01-27T01:07:26.954701+0000","last_unstale":"2024-01-27T01:07:26.958068+0000","last_undegraded":"2024-01-27T01:07:26.958068+0000","last_fullsized":"2024-01-27T01:07:26.958068+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:24.976708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.743124+0000","last_change":"2024-01-27T01:07:27.743124+0000","last_active":"2024-01-27T01:07:27.743124+0000","last_peered":"2024-01-27T01:07:27.743124+0000","last_clean":"2024-01-27T01:07:27.743124+0000","last_became_active":"2024-01-27T01:07:27.742799+0000","last_became_peered":"2024-01-27T01:07:27.742799+0000","last_unstale":"2024-01-27T01:07:27.743124+0000","last_undegraded":"2024-01-27T01:07:27.743124+0000","last_fullsized":"2024-01-27T01:07:27.743124+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:14.714021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":10,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.740048+0000","last_change":"2024-01-27T01:07:26.946511+0000","last_active":"2024-01-27T01:07:27.740048+0000","last_peered":"2024-01-27T01:07:27.740048+0000","last_clean":"2024-01-27T01:07:27.740048+0000","last_became_active":"2024-01-27T01:07:26.946186+0000","last_became_peered":"2024-01-27T01:07:26.946186+0000","last_unstale":"2024-01-27T01:07:27.740048+0000","last_undegraded":"2024-01-27T01:07:27.740048+0000","last_fullsized":"2024-01-27T01:07:27.740048+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:24:39.506445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.749767+0000","last_change":"2024-01-27T01:07:27.749767+0000","last_active":"2024-01-27T01:07:27.749767+0000","last_peered":"2024-01-27T01:07:27.749767+0000","last_clean":"2024-01-27T01:07:27.749767+0000","last_became_active":"2024-01-27T01:07:27.749508+0000","last_became_peered":"2024-01-27T01:07:27.749508+0000","last_unstale":"2024-01-27T01:07:27.749767+0000","last_undegraded":"2024-01-27T01:07:27.749767+0000","last_fullsized":"2024-01-27T01:07:27.749767+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:56:10.391021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":10,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.734495+0000","last_change":"2024-01-27T01:07:26.947663+0000","last_active":"2024-01-27T01:07:27.734495+0000","last_peered":"2024-01-27T01:07:27.734495+0000","last_clean":"2024-01-27T01:07:27.734495+0000","last_became_active":"2024-01-27T01:07:26.947378+0000","last_became_peered":"2024-01-27T01:07:26.947378+0000","last_unstale":"2024-01-27T01:07:27.734495+0000","last_undegraded":"2024-01-27T01:07:27.734495+0000","last_fullsized":"2024-01-27T01:07:27.734495+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:28:30.563975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.749325+0000","last_change":"2024-01-27T01:07:27.749325+0000","last_active":"2024-01-27T01:07:27.749325+0000","last_peered":"2024-01-27T01:07:27.749325+0000","last_clean":"2024-01-27T01:07:27.749325+0000","last_became_active":"2024-01-27T01:07:27.744840+0000","last_became_peered":"2024-01-27T01:07:27.744840+0000","last_unstale":"2024-01-27T01:07:27.749325+0000","last_undegraded":"2024-01-27T01:07:27.749325+0000","last_fullsized":"2024-01-27T01:07:27.749325+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:15:38.535322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.949785+0000","last_change":"2024-01-27T01:07:26.948157+0000","last_active":"2024-01-27T01:07:26.949785+0000","last_peered":"2024-01-27T01:07:26.949785+0000","last_clean":"2024-01-27T01:07:26.949785+0000","last_became_active":"2024-01-27T01:07:26.947870+0000","last_became_peered":"2024-01-27T01:07:26.947870+0000","last_unstale":"2024-01-27T01:07:26.949785+0000","last_undegraded":"2024-01-27T01:07:26.949785+0000","last_fullsized":"2024-01-27T01:07:26.949785+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:04:16.755137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.744220+0000","last_change":"2024-01-27T01:07:27.744220+0000","last_active":"2024-01-27T01:07:27.744220+0000","last_peered":"2024-01-27T01:07:27.744220+0000","last_clean":"2024-01-27T01:07:27.744220+0000","last_became_active":"2024-01-27T01:07:27.742448+0000","last_became_peered":"2024-01-27T01:07:27.742448+0000","last_unstale":"2024-01-27T01:07:27.744220+0000","last_undegraded":"2024-01-27T01:07:27.744220+0000","last_fullsized":"2024-01-27T01:07:27.744220+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:03.017967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"242'3","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.942331+0000","last_change":"2024-01-27T01:07:26.953006+0000","last_active":"2024-01-27T01:07:27.942331+0000","last_peered":"2024-01-27T01:07:27.942331+0000","last_clean":"2024-01-27T01:07:27.942331+0000","last_became_active":"2024-01-27T01:07:26.952730+0000","last_became_peered":"2024-01-27T01:07:26.952730+0000","last_unstale":"2024-01-27T01:07:27.942331+0000","last_undegraded":"2024-01-27T01:07:27.942331+0000","last_fullsized":"2024-01-27T01:07:27.942331+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:14.273722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.747921+0000","last_change":"2024-01-27T01:07:27.747921+0000","last_active":"2024-01-27T01:07:27.747921+0000","last_peered":"2024-01-27T01:07:27.747921+0000","last_clean":"2024-01-27T01:07:27.747921+0000","last_became_active":"2024-01-27T01:07:27.747382+0000","last_became_peered":"2024-01-27T01:07:27.747382+0000","last_unstale":"2024-01-27T01:07:27.747921+0000","last_undegraded":"2024-01-27T01:07:27.747921+0000","last_fullsized":"2024-01-27T01:07:27.747921+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:43:44.312681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"242'6","reported_seq":16,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.948807+0000","last_change":"2024-01-27T01:07:26.951048+0000","last_active":"2024-01-27T01:07:27.948807+0000","last_peered":"2024-01-27T01:07:27.948807+0000","last_clean":"2024-01-27T01:07:27.948807+0000","last_became_active":"2024-01-27T01:07:26.950750+0000","last_became_peered":"2024-01-27T01:07:26.950750+0000","last_unstale":"2024-01-27T01:07:27.948807+0000","last_undegraded":"2024-01-27T01:07:27.948807+0000","last_fullsized":"2024-01-27T01:07:27.948807+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:48:29.363281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.758005+0000","last_change":"2024-01-27T01:07:27.756638+0000","last_active":"2024-01-27T01:07:27.758005+0000","last_peered":"2024-01-27T01:07:27.758005+0000","last_clean":"2024-01-27T01:07:27.758005+0000","last_became_active":"2024-01-27T01:07:27.756219+0000","last_became_peered":"2024-01-27T01:07:27.756219+0000","last_unstale":"2024-01-27T01:07:27.758005+0000","last_undegraded":"2024-01-27T01:07:27.758005+0000","last_fullsized":"2024-01-27T01:07:27.758005+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:24:43.565560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"242'2","reported_seq":11,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.945780+0000","last_change":"2024-01-27T01:07:26.957374+0000","last_active":"2024-01-27T01:07:27.945780+0000","last_peered":"2024-01-27T01:07:27.945780+0000","last_clean":"2024-01-27T01:07:27.945780+0000","last_became_active":"2024-01-27T01:07:26.957184+0000","last_became_peered":"2024-01-27T01:07:26.957184+0000","last_unstale":"2024-01-27T01:07:27.945780+0000","last_undegraded":"2024-01-27T01:07:27.945780+0000","last_fullsized":"2024-01-27T01:07:27.945780+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:43:44.001485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.741016+0000","last_change":"2024-01-27T01:07:27.741016+0000","last_active":"2024-01-27T01:07:27.741016+0000","last_peered":"2024-01-27T01:07:27.741016+0000","last_clean":"2024-01-27T01:07:27.741016+0000","last_became_active":"2024-01-27T01:07:27.740724+0000","last_became_peered":"2024-01-27T01:07:27.740724+0000","last_unstale":"2024-01-27T01:07:27.741016+0000","last_undegraded":"2024-01-27T01:07:27.741016+0000","last_fullsized":"2024-01-27T01:07:27.741016+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:19:56.759979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.955598+0000","last_change":"2024-01-27T01:07:26.951785+0000","last_active":"2024-01-27T01:07:26.955598+0000","last_peered":"2024-01-27T01:07:26.955598+0000","last_clean":"2024-01-27T01:07:26.955598+0000","last_became_active":"2024-01-27T01:07:26.951519+0000","last_became_peered":"2024-01-27T01:07:26.951519+0000","last_unstale":"2024-01-27T01:07:26.955598+0000","last_undegraded":"2024-01-27T01:07:26.955598+0000","last_fullsized":"2024-01-27T01:07:26.955598+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:06.511483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.755983+0000","last_change":"2024-01-27T01:07:27.755983+0000","last_active":"2024-01-27T01:07:27.755983+0000","last_peered":"2024-01-27T01:07:27.755983+0000","last_clean":"2024-01-27T01:07:27.755983+0000","last_became_active":"2024-01-27T01:07:27.755715+0000","last_became_peered":"2024-01-27T01:07:27.755715+0000","last_unstale":"2024-01-27T01:07:27.755983+0000","last_undegraded":"2024-01-27T01:07:27.755983+0000","last_fullsized":"2024-01-27T01:07:27.755983+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:11.556927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.955622+0000","last_change":"2024-01-27T01:07:26.952237+0000","last_active":"2024-01-27T01:07:26.955622+0000","last_peered":"2024-01-27T01:07:26.955622+0000","last_clean":"2024-01-27T01:07:26.955622+0000","last_became_active":"2024-01-27T01:07:26.951971+0000","last_became_peered":"2024-01-27T01:07:26.951971+0000","last_unstale":"2024-01-27T01:07:26.955622+0000","last_undegraded":"2024-01-27T01:07:26.955622+0000","last_fullsized":"2024-01-27T01:07:26.955622+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:43:36.059851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.746452+0000","last_change":"2024-01-27T01:07:27.746452+0000","last_active":"2024-01-27T01:07:27.746452+0000","last_peered":"2024-01-27T01:07:27.746452+0000","last_clean":"2024-01-27T01:07:27.746452+0000","last_became_active":"2024-01-27T01:07:27.746174+0000","last_became_peered":"2024-01-27T01:07:27.746174+0000","last_unstale":"2024-01-27T01:07:27.746452+0000","last_undegraded":"2024-01-27T01:07:27.746452+0000","last_fullsized":"2024-01-27T01:07:27.746452+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:32.778691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"242'8","reported_seq":18,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.948345+0000","last_change":"2024-01-27T01:07:26.946801+0000","last_active":"2024-01-27T01:07:27.948345+0000","last_peered":"2024-01-27T01:07:27.948345+0000","last_clean":"2024-01-27T01:07:27.948345+0000","last_became_active":"2024-01-27T01:07:26.946453+0000","last_became_peered":"2024-01-27T01:07:26.946453+0000","last_unstale":"2024-01-27T01:07:27.948345+0000","last_undegraded":"2024-01-27T01:07:27.948345+0000","last_fullsized":"2024-01-27T01:07:27.948345+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:11:44.275714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.746480+0000","last_change":"2024-01-27T01:07:27.746480+0000","last_active":"2024-01-27T01:07:27.746480+0000","last_peered":"2024-01-27T01:07:27.746480+0000","last_clean":"2024-01-27T01:07:27.746480+0000","last_became_active":"2024-01-27T01:07:27.746131+0000","last_became_peered":"2024-01-27T01:07:27.746131+0000","last_unstale":"2024-01-27T01:07:27.746480+0000","last_undegraded":"2024-01-27T01:07:27.746480+0000","last_fullsized":"2024-01-27T01:07:27.746480+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:06:55.090910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"242'1","reported_seq":11,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.945219+0000","last_change":"2024-01-27T01:07:26.958336+0000","last_active":"2024-01-27T01:07:27.945219+0000","last_peered":"2024-01-27T01:07:27.945219+0000","last_clean":"2024-01-27T01:07:27.945219+0000","last_became_active":"2024-01-27T01:07:26.957817+0000","last_became_peered":"2024-01-27T01:07:26.957817+0000","last_unstale":"2024-01-27T01:07:27.945219+0000","last_undegraded":"2024-01-27T01:07:27.945219+0000","last_fullsized":"2024-01-27T01:07:27.945219+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:45.033976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.743490+0000","last_change":"2024-01-27T01:07:27.743490+0000","last_active":"2024-01-27T01:07:27.743490+0000","last_peered":"2024-01-27T01:07:27.743490+0000","last_clean":"2024-01-27T01:07:27.743490+0000","last_became_active":"2024-01-27T01:07:27.743207+0000","last_became_peered":"2024-01-27T01:07:27.743207+0000","last_unstale":"2024-01-27T01:07:27.743490+0000","last_undegraded":"2024-01-27T01:07:27.743490+0000","last_fullsized":"2024-01-27T01:07:27.743490+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:19.339556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"242'2","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.944670+0000","last_change":"2024-01-27T01:07:26.946877+0000","last_active":"2024-01-27T01:07:27.944670+0000","last_peered":"2024-01-27T01:07:27.944670+0000","last_clean":"2024-01-27T01:07:27.944670+0000","last_became_active":"2024-01-27T01:07:26.946596+0000","last_became_peered":"2024-01-27T01:07:26.946596+0000","last_unstale":"2024-01-27T01:07:27.944670+0000","last_undegraded":"2024-01-27T01:07:27.944670+0000","last_fullsized":"2024-01-27T01:07:27.944670+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:27:41.114970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.749680+0000","last_change":"2024-01-27T01:07:27.746482+0000","last_active":"2024-01-27T01:07:27.749680+0000","last_peered":"2024-01-27T01:07:27.749680+0000","last_clean":"2024-01-27T01:07:27.749680+0000","last_became_active":"2024-01-27T01:07:27.746052+0000","last_became_peered":"2024-01-27T01:07:27.746052+0000","last_unstale":"2024-01-27T01:07:27.749680+0000","last_undegraded":"2024-01-27T01:07:27.749680+0000","last_fullsized":"2024-01-27T01:07:27.749680+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:32.387353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"242'1","reported_seq":10,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.740668+0000","last_change":"2024-01-27T01:07:26.950943+0000","last_active":"2024-01-27T01:07:27.740668+0000","last_peered":"2024-01-27T01:07:27.740668+0000","last_clean":"2024-01-27T01:07:27.740668+0000","last_became_active":"2024-01-27T01:07:26.950251+0000","last_became_peered":"2024-01-27T01:07:26.950251+0000","last_unstale":"2024-01-27T01:07:27.740668+0000","last_undegraded":"2024-01-27T01:07:27.740668+0000","last_fullsized":"2024-01-27T01:07:27.740668+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:31.380205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.742779+0000","last_change":"2024-01-27T01:07:27.742779+0000","last_active":"2024-01-27T01:07:27.742779+0000","last_peered":"2024-01-27T01:07:27.742779+0000","last_clean":"2024-01-27T01:07:27.742779+0000","last_became_active":"2024-01-27T01:07:27.742462+0000","last_became_peered":"2024-01-27T01:07:27.742462+0000","last_unstale":"2024-01-27T01:07:27.742779+0000","last_undegraded":"2024-01-27T01:07:27.742779+0000","last_fullsized":"2024-01-27T01:07:27.742779+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:28.493839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.958400+0000","last_change":"2024-01-27T01:07:26.956417+0000","last_active":"2024-01-27T01:07:26.958400+0000","last_peered":"2024-01-27T01:07:26.958400+0000","last_clean":"2024-01-27T01:07:26.958400+0000","last_became_active":"2024-01-27T01:07:26.956128+0000","last_became_peered":"2024-01-27T01:07:26.956128+0000","last_unstale":"2024-01-27T01:07:26.958400+0000","last_undegraded":"2024-01-27T01:07:26.958400+0000","last_fullsized":"2024-01-27T01:07:26.958400+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:22.165694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.752330+0000","last_change":"2024-01-27T01:07:27.752330+0000","last_active":"2024-01-27T01:07:27.752330+0000","last_peered":"2024-01-27T01:07:27.752330+0000","last_clean":"2024-01-27T01:07:27.752330+0000","last_became_active":"2024-01-27T01:07:27.752040+0000","last_became_peered":"2024-01-27T01:07:27.752040+0000","last_unstale":"2024-01-27T01:07:27.752330+0000","last_undegraded":"2024-01-27T01:07:27.752330+0000","last_fullsized":"2024-01-27T01:07:27.752330+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:36:14.454947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"242'2","reported_seq":11,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.944262+0000","last_change":"2024-01-27T01:07:26.947483+0000","last_active":"2024-01-27T01:07:27.944262+0000","last_peered":"2024-01-27T01:07:27.944262+0000","last_clean":"2024-01-27T01:07:27.944262+0000","last_became_active":"2024-01-27T01:07:26.947172+0000","last_became_peered":"2024-01-27T01:07:26.947172+0000","last_unstale":"2024-01-27T01:07:27.944262+0000","last_undegraded":"2024-01-27T01:07:27.944262+0000","last_fullsized":"2024-01-27T01:07:27.944262+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:03.604469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.757740+0000","last_change":"2024-01-27T01:07:27.755356+0000","last_active":"2024-01-27T01:07:27.757740+0000","last_peered":"2024-01-27T01:07:27.757740+0000","last_clean":"2024-01-27T01:07:27.757740+0000","last_became_active":"2024-01-27T01:07:27.755108+0000","last_became_peered":"2024-01-27T01:07:27.755108+0000","last_unstale":"2024-01-27T01:07:27.757740+0000","last_undegraded":"2024-01-27T01:07:27.757740+0000","last_fullsized":"2024-01-27T01:07:27.757740+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:51:14.988606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.953821+0000","last_change":"2024-01-27T01:07:26.950680+0000","last_active":"2024-01-27T01:07:26.953821+0000","last_peered":"2024-01-27T01:07:26.953821+0000","last_clean":"2024-01-27T01:07:26.953821+0000","last_became_active":"2024-01-27T01:07:26.950395+0000","last_became_peered":"2024-01-27T01:07:26.950395+0000","last_unstale":"2024-01-27T01:07:26.953821+0000","last_undegraded":"2024-01-27T01:07:26.953821+0000","last_fullsized":"2024-01-27T01:07:26.953821+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:28:11.238816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.748049+0000","last_change":"2024-01-27T01:07:27.748049+0000","last_active":"2024-01-27T01:07:27.748049+0000","last_peered":"2024-01-27T01:07:27.748049+0000","last_clean":"2024-01-27T01:07:27.748049+0000","last_became_active":"2024-01-27T01:07:27.747765+0000","last_became_peered":"2024-01-27T01:07:27.747765+0000","last_unstale":"2024-01-27T01:07:27.748049+0000","last_undegraded":"2024-01-27T01:07:27.748049+0000","last_fullsized":"2024-01-27T01:07:27.748049+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:51:33.521276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.953036+0000","last_change":"2024-01-27T01:07:26.950170+0000","last_active":"2024-01-27T01:07:26.953036+0000","last_peered":"2024-01-27T01:07:26.953036+0000","last_clean":"2024-01-27T01:07:26.953036+0000","last_became_active":"2024-01-27T01:07:26.949857+0000","last_became_peered":"2024-01-27T01:07:26.949857+0000","last_unstale":"2024-01-27T01:07:26.953036+0000","last_undegraded":"2024-01-27T01:07:26.953036+0000","last_fullsized":"2024-01-27T01:07:26.953036+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:29.510893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.753040+0000","last_change":"2024-01-27T01:07:27.753040+0000","last_active":"2024-01-27T01:07:27.753040+0000","last_peered":"2024-01-27T01:07:27.753040+0000","last_clean":"2024-01-27T01:07:27.753040+0000","last_became_active":"2024-01-27T01:07:27.752784+0000","last_became_peered":"2024-01-27T01:07:27.752784+0000","last_unstale":"2024-01-27T01:07:27.753040+0000","last_undegraded":"2024-01-27T01:07:27.753040+0000","last_fullsized":"2024-01-27T01:07:27.753040+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:09:57.375468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":10,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.734272+0000","last_change":"2024-01-27T01:07:26.944368+0000","last_active":"2024-01-27T01:07:27.734272+0000","last_peered":"2024-01-27T01:07:27.734272+0000","last_clean":"2024-01-27T01:07:27.734272+0000","last_became_active":"2024-01-27T01:07:26.944076+0000","last_became_peered":"2024-01-27T01:07:26.944076+0000","last_unstale":"2024-01-27T01:07:27.734272+0000","last_undegraded":"2024-01-27T01:07:27.734272+0000","last_fullsized":"2024-01-27T01:07:27.734272+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:59:38.148215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.747646+0000","last_change":"2024-01-27T01:07:27.747646+0000","last_active":"2024-01-27T01:07:27.747646+0000","last_peered":"2024-01-27T01:07:27.747646+0000","last_clean":"2024-01-27T01:07:27.747646+0000","last_became_active":"2024-01-27T01:07:27.747345+0000","last_became_peered":"2024-01-27T01:07:27.747345+0000","last_unstale":"2024-01-27T01:07:27.747646+0000","last_undegraded":"2024-01-27T01:07:27.747646+0000","last_fullsized":"2024-01-27T01:07:27.747646+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:30:45.126288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"242'2","reported_seq":11,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.944310+0000","last_change":"2024-01-27T01:07:26.950235+0000","last_active":"2024-01-27T01:07:27.944310+0000","last_peered":"2024-01-27T01:07:27.944310+0000","last_clean":"2024-01-27T01:07:27.944310+0000","last_became_active":"2024-01-27T01:07:26.949923+0000","last_became_peered":"2024-01-27T01:07:26.949923+0000","last_unstale":"2024-01-27T01:07:27.944310+0000","last_undegraded":"2024-01-27T01:07:27.944310+0000","last_fullsized":"2024-01-27T01:07:27.944310+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:27:55.458182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.758370+0000","last_change":"2024-01-27T01:07:27.758370+0000","last_active":"2024-01-27T01:07:27.758370+0000","last_peered":"2024-01-27T01:07:27.758370+0000","last_clean":"2024-01-27T01:07:27.758370+0000","last_became_active":"2024-01-27T01:07:27.758007+0000","last_became_peered":"2024-01-27T01:07:27.758007+0000","last_unstale":"2024-01-27T01:07:27.758370+0000","last_undegraded":"2024-01-27T01:07:27.758370+0000","last_fullsized":"2024-01-27T01:07:27.758370+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:07.518631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.955634+0000","last_change":"2024-01-27T01:07:26.953360+0000","last_active":"2024-01-27T01:07:26.955634+0000","last_peered":"2024-01-27T01:07:26.955634+0000","last_clean":"2024-01-27T01:07:26.955634+0000","last_became_active":"2024-01-27T01:07:26.953090+0000","last_became_peered":"2024-01-27T01:07:26.953090+0000","last_unstale":"2024-01-27T01:07:26.955634+0000","last_undegraded":"2024-01-27T01:07:26.955634+0000","last_fullsized":"2024-01-27T01:07:26.955634+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:10.549778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.750264+0000","last_change":"2024-01-27T01:07:27.750264+0000","last_active":"2024-01-27T01:07:27.750264+0000","last_peered":"2024-01-27T01:07:27.750264+0000","last_clean":"2024-01-27T01:07:27.750264+0000","last_became_active":"2024-01-27T01:07:27.749987+0000","last_became_peered":"2024-01-27T01:07:27.749987+0000","last_unstale":"2024-01-27T01:07:27.750264+0000","last_undegraded":"2024-01-27T01:07:27.750264+0000","last_fullsized":"2024-01-27T01:07:27.750264+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:19.339556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":10,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.734444+0000","last_change":"2024-01-27T01:07:26.942700+0000","last_active":"2024-01-27T01:07:27.734444+0000","last_peered":"2024-01-27T01:07:27.734444+0000","last_clean":"2024-01-27T01:07:27.734444+0000","last_became_active":"2024-01-27T01:07:26.942491+0000","last_became_peered":"2024-01-27T01:07:26.942491+0000","last_unstale":"2024-01-27T01:07:27.734444+0000","last_undegraded":"2024-01-27T01:07:27.734444+0000","last_fullsized":"2024-01-27T01:07:27.734444+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:03:40.104583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.746804+0000","last_change":"2024-01-27T01:07:27.746804+0000","last_active":"2024-01-27T01:07:27.746804+0000","last_peered":"2024-01-27T01:07:27.746804+0000","last_clean":"2024-01-27T01:07:27.746804+0000","last_became_active":"2024-01-27T01:07:27.746110+0000","last_became_peered":"2024-01-27T01:07:27.746110+0000","last_unstale":"2024-01-27T01:07:27.746804+0000","last_undegraded":"2024-01-27T01:07:27.746804+0000","last_fullsized":"2024-01-27T01:07:27.746804+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:48:18.465014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":10,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.734759+0000","last_change":"2024-01-27T01:07:26.943462+0000","last_active":"2024-01-27T01:07:27.734759+0000","last_peered":"2024-01-27T01:07:27.734759+0000","last_clean":"2024-01-27T01:07:27.734759+0000","last_became_active":"2024-01-27T01:07:26.943193+0000","last_became_peered":"2024-01-27T01:07:26.943193+0000","last_unstale":"2024-01-27T01:07:27.734759+0000","last_undegraded":"2024-01-27T01:07:27.734759+0000","last_fullsized":"2024-01-27T01:07:27.734759+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:21:20.952966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.743744+0000","last_change":"2024-01-27T01:07:27.739197+0000","last_active":"2024-01-27T01:07:27.743744+0000","last_peered":"2024-01-27T01:07:27.743744+0000","last_clean":"2024-01-27T01:07:27.743744+0000","last_became_active":"2024-01-27T01:07:27.738911+0000","last_became_peered":"2024-01-27T01:07:27.738911+0000","last_unstale":"2024-01-27T01:07:27.743744+0000","last_undegraded":"2024-01-27T01:07:27.743744+0000","last_fullsized":"2024-01-27T01:07:27.743744+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:41:00.947931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.952474+0000","last_change":"2024-01-27T01:07:26.949530+0000","last_active":"2024-01-27T01:07:26.952474+0000","last_peered":"2024-01-27T01:07:26.952474+0000","last_clean":"2024-01-27T01:07:26.952474+0000","last_became_active":"2024-01-27T01:07:26.949204+0000","last_became_peered":"2024-01-27T01:07:26.949204+0000","last_unstale":"2024-01-27T01:07:26.952474+0000","last_undegraded":"2024-01-27T01:07:26.952474+0000","last_fullsized":"2024-01-27T01:07:26.952474+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:16:54.275653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.751936+0000","last_change":"2024-01-27T01:07:27.744772+0000","last_active":"2024-01-27T01:07:27.751936+0000","last_peered":"2024-01-27T01:07:27.751936+0000","last_clean":"2024-01-27T01:07:27.751936+0000","last_became_active":"2024-01-27T01:07:27.744416+0000","last_became_peered":"2024-01-27T01:07:27.744416+0000","last_unstale":"2024-01-27T01:07:27.751936+0000","last_undegraded":"2024-01-27T01:07:27.751936+0000","last_fullsized":"2024-01-27T01:07:27.751936+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:59:39.155363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.955576+0000","last_change":"2024-01-27T01:07:26.947692+0000","last_active":"2024-01-27T01:07:26.955576+0000","last_peered":"2024-01-27T01:07:26.955576+0000","last_clean":"2024-01-27T01:07:26.955576+0000","last_became_active":"2024-01-27T01:07:26.947404+0000","last_became_peered":"2024-01-27T01:07:26.947404+0000","last_unstale":"2024-01-27T01:07:26.955576+0000","last_undegraded":"2024-01-27T01:07:26.955576+0000","last_fullsized":"2024-01-27T01:07:26.955576+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:12:55.245326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.756805+0000","last_change":"2024-01-27T01:07:27.749874+0000","last_active":"2024-01-27T01:07:27.756805+0000","last_peered":"2024-01-27T01:07:27.756805+0000","last_clean":"2024-01-27T01:07:27.756805+0000","last_became_active":"2024-01-27T01:07:27.749252+0000","last_became_peered":"2024-01-27T01:07:27.749252+0000","last_unstale":"2024-01-27T01:07:27.756805+0000","last_undegraded":"2024-01-27T01:07:27.756805+0000","last_fullsized":"2024-01-27T01:07:27.756805+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:18:53.285178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.949118+0000","last_change":"2024-01-27T01:07:26.947606+0000","last_active":"2024-01-27T01:07:26.949118+0000","last_peered":"2024-01-27T01:07:26.949118+0000","last_clean":"2024-01-27T01:07:26.949118+0000","last_became_active":"2024-01-27T01:07:26.947287+0000","last_became_peered":"2024-01-27T01:07:26.947287+0000","last_unstale":"2024-01-27T01:07:26.949118+0000","last_undegraded":"2024-01-27T01:07:26.949118+0000","last_fullsized":"2024-01-27T01:07:26.949118+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:34:14.649324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.749856+0000","last_change":"2024-01-27T01:07:27.749856+0000","last_active":"2024-01-27T01:07:27.749856+0000","last_peered":"2024-01-27T01:07:27.749856+0000","last_clean":"2024-01-27T01:07:27.749856+0000","last_became_active":"2024-01-27T01:07:27.749595+0000","last_became_peered":"2024-01-27T01:07:27.749595+0000","last_unstale":"2024-01-27T01:07:27.749856+0000","last_undegraded":"2024-01-27T01:07:27.749856+0000","last_fullsized":"2024-01-27T01:07:27.749856+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:25:38.146534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.953694+0000","last_change":"2024-01-27T01:07:26.950737+0000","last_active":"2024-01-27T01:07:26.953694+0000","last_peered":"2024-01-27T01:07:26.953694+0000","last_clean":"2024-01-27T01:07:26.953694+0000","last_became_active":"2024-01-27T01:07:26.950498+0000","last_became_peered":"2024-01-27T01:07:26.950498+0000","last_unstale":"2024-01-27T01:07:26.953694+0000","last_undegraded":"2024-01-27T01:07:26.953694+0000","last_fullsized":"2024-01-27T01:07:26.953694+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:24:42.558412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.748875+0000","last_change":"2024-01-27T01:07:27.746007+0000","last_active":"2024-01-27T01:07:27.748875+0000","last_peered":"2024-01-27T01:07:27.748875+0000","last_clean":"2024-01-27T01:07:27.748875+0000","last_became_active":"2024-01-27T01:07:27.745722+0000","last_became_peered":"2024-01-27T01:07:27.745722+0000","last_unstale":"2024-01-27T01:07:27.748875+0000","last_undegraded":"2024-01-27T01:07:27.748875+0000","last_fullsized":"2024-01-27T01:07:27.748875+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:09:23.112301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"242'1","reported_seq":10,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.736684+0000","last_change":"2024-01-27T01:07:26.951812+0000","last_active":"2024-01-27T01:07:27.736684+0000","last_peered":"2024-01-27T01:07:27.736684+0000","last_clean":"2024-01-27T01:07:27.736684+0000","last_became_active":"2024-01-27T01:07:26.951141+0000","last_became_peered":"2024-01-27T01:07:26.951141+0000","last_unstale":"2024-01-27T01:07:27.736684+0000","last_undegraded":"2024-01-27T01:07:27.736684+0000","last_fullsized":"2024-01-27T01:07:27.736684+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:06:20.506111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"230'8772","reported_seq":9758,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.952559+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:07:26.952559+0000","last_peered":"2024-01-27T01:07:26.952559+0000","last_clean":"2024-01-27T01:07:26.952559+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:07:26.952559+0000","last_undegraded":"2024-01-27T01:07:26.952559+0000","last_fullsized":"2024-01-27T01:07:26.952559+0000","mapping_epoch":14,"log_start":"98'2229","ondisk_log_start":"98'2229","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2456,"num_read_kb":5220,"num_write":670,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2456,"num_read_kb":5220,"num_write":670,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6543,"ondisk_log_size":6543,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576012,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300856,"kb_used_data":3568,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94070984,"statfs":{"total":96636764160,"available":96328687616,"internally_reserved":0,"allocated":3653632,"data_stored":2205830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.312},"max":{"1min":0.72499999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.63200000000000001,"15min":0.621},"min":{"1min":0.48099999999999998,"5min":0.39900000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":0.879,"15min":0.879},"last":0.76400000000000001}]},{"osd":1,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70299999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.63800000000000001,"15min":0.63100000000000001},"min":{"1min":0.44500000000000001,"5min":0.42099999999999999,"15min":0.34300000000000003},"max":{"1min":0.73499999999999999,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.81499999999999995}]},{"osd":2,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.61899999999999999,"15min":0.60999999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.72899999999999998,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.628,"15min":0.629},"min":{"1min":0.372,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.69499999999999995,"5min":0.872,"15min":0.88900000000000001},"last":0.77700000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.60899999999999999,"15min":0.61399999999999999},"min":{"1min":0.38500000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69099999999999995,"5min":0.876,"15min":0.876},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.65700000000000003,"15min":0.64200000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80700000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.753}]},{"osd":4,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.503,"15min":0.48799999999999999},"min":{"1min":0.35199999999999998,"5min":0.314,"15min":0.29999999999999999},"max":{"1min":0.65500000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53700000000000003,"15min":0.51600000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.629,"5min":0.80300000000000005,"15min":0.86599999999999999},"last":0.64200000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.55300000000000005,"15min":0.52500000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.80400000000000005},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57299999999999995,"15min":0.56399999999999995},"min":{"1min":0.41699999999999998,"5min":0.34799999999999998,"15min":0.34399999999999997},"max":{"1min":0.79200000000000004,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.68300000000000005}]},{"osd":6,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60499999999999998,"15min":0.58899999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.60499999999999998,"15min":0.58699999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.75,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.69999999999999996}]}]},{"osd":6,"up_from":13,"seq":55834576012,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308468,"kb_used_data":2988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063372,"statfs":{"total":96636764160,"available":96320892928,"internally_reserved":0,"allocated":3059712,"data_stored":1615450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61099999999999999,"15min":0.60799999999999998},"min":{"1min":0.41999999999999998,"5min":0.40799999999999997,"15min":0.36399999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.441,"5min":0.432,"15min":0.376},"max":{"1min":0.82499999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.58799999999999997}]},{"osd":1,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79000000000000004,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.40799999999999997,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.81200000000000006}]},{"osd":2,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.504},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.39700000000000002,"15min":0.377},"max":{"1min":0.80400000000000005,"5min":0.878,"15min":0.878},"last":0.54300000000000004}]},{"osd":3,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.622,"15min":0.627},"min":{"1min":0.5,"5min":0.378,"15min":0.378},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.90100000000000002},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66300000000000003,"15min":0.66000000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.754}]},{"osd":4,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.36299999999999999,"5min":0.33300000000000002,"15min":0.32500000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.86599999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33900000000000002,"5min":0.33800000000000002,"15min":0.29799999999999999},"max":{"1min":0.77300000000000002,"5min":0.78500000000000003,"15min":0.88900000000000001},"last":0.66800000000000004}]},{"osd":5,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.31900000000000001},"max":{"1min":0.68999999999999995,"5min":0.75900000000000001,"15min":0.80500000000000005},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.42199999999999999,"5min":0.37,"15min":0.28899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.874},"last":0.84199999999999997}]},{"osd":7,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45800000000000002,"5min":0.39000000000000001,"15min":0.35799999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.92900000000000005},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.44500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78900000000000003,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.73599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834576012,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300268,"kb_used_data":2980,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071572,"statfs":{"total":96636764160,"available":96329289728,"internally_reserved":0,"allocated":3051520,"data_stored":1615338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.28499999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.48899999999999999,"5min":0.66100000000000003,"15min":0.69599999999999995},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.625,"5min":0.67800000000000005,"15min":0.76600000000000001},"last":0.34999999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.42699999999999999,"15min":0.435},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.26200000000000001},"max":{"1min":0.54900000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.309},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":0.48799999999999999},"min":{"1min":0.36699999999999999,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.58899999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.40899999999999997}]},{"osd":3,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51800000000000002,"15min":0.53100000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66200000000000003,"5min":0.84799999999999998,"15min":1.2130000000000001},"last":0.377},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.5,"15min":0.51300000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.32300000000000001},"max":{"1min":0.65100000000000002,"5min":1.125,"15min":1.22},"last":0.39000000000000001}]},{"osd":4,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.66900000000000004,"15min":0.67000000000000004},"min":{"1min":0.41599999999999998,"5min":0.41099999999999998,"15min":0.39800000000000002},"max":{"1min":0.92100000000000004,"5min":1.242,"15min":1.242},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.68500000000000005,"15min":0.69199999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.35699999999999998},"max":{"1min":0.80500000000000005,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.59699999999999998}]},{"osd":5,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.72099999999999997,"15min":0.70499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1,"5min":5.444,"15min":5.444},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71399999999999997,"15min":0.70599999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.435},"max":{"1min":0.97599999999999998,"5min":2.6669999999999998,"15min":2.6669999999999998},"last":0.51300000000000001}]},{"osd":6,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.73599999999999999,"15min":0.72099999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.47099999999999997},"max":{"1min":0.89700000000000002,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.499},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.80600000000000005,"15min":0.74199999999999999},"min":{"1min":0.52400000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0600000000000001,"5min":11.223000000000001,"15min":11.223000000000001},"last":0.61599999999999999}]},{"osd":7,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.749,"15min":0.753},"min":{"1min":0.54100000000000004,"5min":0.52800000000000002,"15min":0.48999999999999999},"max":{"1min":1.0349999999999999,"5min":2.9660000000000002,"15min":2.9660000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.76900000000000002,"15min":0.76700000000000002},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.52000000000000002},"max":{"1min":0.95599999999999996,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.58799999999999997}]}]},{"osd":0,"up_from":13,"seq":55834576011,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300860,"kb_used_data":3572,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94070980,"statfs":{"total":96636764160,"available":96328683520,"internally_reserved":0,"allocated":3657728,"data_stored":2207242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42899999999999999,"15min":0.42199999999999999},"min":{"1min":0.30299999999999999,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":0.55600000000000005,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.436,"5min":0.45600000000000002,"15min":0.46100000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":0.56499999999999995,"5min":0.72699999999999998,"15min":0.73399999999999999},"last":0.64300000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.29699999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.58299999999999996,"5min":0.66900000000000004,"15min":0.76800000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48399999999999999,"15min":0.48899999999999999},"min":{"1min":0.38600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64600000000000002,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.502}]},{"osd":3,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.66100000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.504},"min":{"1min":0.36699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64100000000000001,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.55300000000000005}]},{"osd":4,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.68300000000000005,"15min":0.67700000000000005},"min":{"1min":0.57799999999999996,"5min":0.47999999999999998,"15min":0.42199999999999999},"max":{"1min":0.86399999999999999,"5min":0.91500000000000004,"15min":1.0720000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.56999999999999995,"5min":0.47099999999999997,"15min":0.437},"max":{"1min":0.82699999999999996,"5min":1.002,"15min":1.002},"last":0.90100000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.68999999999999995,"15min":0.70099999999999996},"min":{"1min":0.58199999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.876,"5min":1.0209999999999999,"15min":2.165},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70599999999999996,"15min":0.71199999999999997},"min":{"1min":0.498,"5min":0.48799999999999999,"15min":0.45600000000000002},"max":{"1min":0.89000000000000001,"5min":0.97399999999999998,"15min":1.1399999999999999},"last":0.92300000000000004}]},{"osd":6,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69299999999999995,"15min":0.71099999999999997},"min":{"1min":0.56499999999999995,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.84599999999999997,"5min":0.98899999999999999,"15min":1.0860000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72599999999999998,"15min":0.74099999999999999},"min":{"1min":0.54300000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.93400000000000005,"5min":1.0109999999999999,"15min":1.425},"last":0.86499999999999999}]},{"osd":7,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.74399999999999999,"15min":0.75700000000000001},"min":{"1min":0.61299999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.98899999999999999,"5min":1.044,"15min":1.1259999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.76300000000000001,"15min":0.76900000000000002},"min":{"1min":0.58299999999999996,"5min":0.51100000000000001,"15min":0.48499999999999999},"max":{"1min":0.91700000000000004,"5min":1.0289999999999999,"15min":1.1910000000000001},"last":0.88700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576011,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312564,"kb_used_data":2988,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059276,"statfs":{"total":96636764160,"available":96316698624,"internally_reserved":0,"allocated":3059712,"data_stored":1615962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39600000000000002,"15min":0.40400000000000003},"min":{"1min":0.27200000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.497,"5min":0.746,"15min":0.746},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.441,"15min":0.443},"min":{"1min":0.34599999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.40699999999999997}]},{"osd":1,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44500000000000001,"15min":0.44600000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.50900000000000001,"15min":0.51900000000000002},"min":{"1min":0.40699999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.48299999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.49299999999999999,"15min":0.496},"min":{"1min":0.38700000000000001,"5min":0.34699999999999998,"15min":0.33200000000000002},"max":{"1min":0.65000000000000002,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.371},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.47799999999999998,"15min":0.48699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.63900000000000001,"5min":0.65500000000000003,"15min":0.75700000000000001},"last":0.439}]},{"osd":4,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.77800000000000002,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.68100000000000005,"15min":0.69199999999999995},"min":{"1min":0.47899999999999998,"5min":0.47099999999999997,"15min":0.378},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.107},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.69699999999999995,"15min":0.70199999999999996},"min":{"1min":0.47399999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.075,"5min":1.075,"15min":1.167},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.69199999999999995,"15min":0.70499999999999996},"min":{"1min":0.52300000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81899999999999995,"5min":0.97499999999999998,"15min":1.05},"last":0.63900000000000001}]},{"osd":6,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.70099999999999996,"15min":0.70899999999999996},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.90200000000000002,"5min":0.97899999999999998,"15min":1.077},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.69599999999999995,"15min":0.71499999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.39200000000000002},"max":{"1min":0.85999999999999999,"5min":0.96699999999999997,"15min":1.135},"last":0.503}]},{"osd":7,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73399999999999999,"15min":0.74399999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.198},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.755,"15min":0.76500000000000001},"min":{"1min":0.53600000000000003,"5min":0.53100000000000003,"15min":0.48199999999999998},"max":{"1min":0.875,"5min":0.96399999999999997,"15min":1.0880000000000001},"last":0.67700000000000005}]}]},{"osd":3,"up_from":13,"seq":55834576010,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304388,"kb_used_data":3004,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067452,"statfs":{"total":96636764160,"available":96325070848,"internally_reserved":0,"allocated":3076096,"data_stored":1617484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44700000000000001,"15min":0.42399999999999999},"min":{"1min":0.309,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.59499999999999997,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.44,"15min":0.45500000000000002},"min":{"1min":0.32400000000000001,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.64000000000000001,"5min":0.66700000000000004,"15min":0.84699999999999998},"last":0.30599999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.624,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66400000000000003,"5min":15.391,"15min":15.391},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.53500000000000003,"15min":0.52400000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70499999999999996,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.432}]},{"osd":2,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.54300000000000004,"15min":0.51600000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.60299999999999998,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53900000000000003,"15min":0.51000000000000001},"min":{"1min":0.374,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.5529999999999999,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":0.35799999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.745,"15min":0.71899999999999997},"min":{"1min":0.56399999999999995,"5min":0.47699999999999998,"15min":0.439},"max":{"1min":0.82699999999999996,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.70899999999999996},"min":{"1min":0.52000000000000002,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":1.0089999999999999,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.63}]},{"osd":5,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.745,"15min":0.71399999999999997},"min":{"1min":0.52000000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.036,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73899999999999999,"15min":0.72299999999999998},"min":{"1min":0.54900000000000004,"5min":0.435,"15min":0.435},"max":{"1min":1.127,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.54200000000000004}]},{"osd":6,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.746,"15min":0.72199999999999998},"min":{"1min":0.50700000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.92100000000000004,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.753,"15min":0.74099999999999999},"min":{"1min":0.54900000000000004,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.077,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.59199999999999997}]},{"osd":7,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.755,"15min":0.73899999999999999},"min":{"1min":0.53300000000000003,"5min":0.52500000000000002,"15min":0.46700000000000003},"max":{"1min":1.103,"5min":4.569,"15min":4.569},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.80600000000000005,"15min":0.78300000000000003},"min":{"1min":0.59199999999999997,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.98999999999999999,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.61199999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576012,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316656,"kb_used_data":2984,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055184,"statfs":{"total":96636764160,"available":96312508416,"internally_reserved":0,"allocated":3055616,"data_stored":1615408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.436,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60199999999999998,"15min":0.61299999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.753,"5min":0.82199999999999995,"15min":0.85699999999999998},"last":0.59299999999999997}]},{"osd":1,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60999999999999999,"15min":0.628},"min":{"1min":0.40699999999999997,"5min":0.39100000000000001,"15min":0.38800000000000001},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60999999999999999,"15min":0.623},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82299999999999995,"5min":0.874,"15min":0.874},"last":0.42399999999999999}]},{"osd":2,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.84699999999999998,"5min":0.85599999999999998,"15min":0.88700000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.48999999999999999,"5min":0.42899999999999999,"15min":0.42799999999999999},"max":{"1min":0.79200000000000004,"5min":0.81499999999999995,"15min":0.89200000000000002},"last":0.39600000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60399999999999998,"15min":0.623},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.80800000000000005,"5min":0.83299999999999996,"15min":0.84199999999999997},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.621,"15min":0.629},"min":{"1min":0.42299999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.40999999999999998}]},{"osd":5,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47999999999999998,"15min":0.48399999999999999},"min":{"1min":0.32400000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.85899999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.503,"15min":0.505},"min":{"1min":0.36299999999999999,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.878},"last":0.61399999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.50600000000000001,"15min":0.499},"min":{"1min":0.38300000000000001,"5min":0.35799999999999998,"15min":0.34999999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.88700000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.375,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.502}]},{"osd":7,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56200000000000006,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.748,"5min":0.80400000000000005,"15min":0.90200000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.38600000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.92600000000000005},"last":0.56799999999999995}]}]},{"osd":5,"up_from":13,"seq":55834576012,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312556,"kb_used_data":2980,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059284,"statfs":{"total":96636764160,"available":96316706816,"internally_reserved":0,"allocated":3051520,"data_stored":1615338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59399999999999997,"15min":0.60299999999999998},"min":{"1min":0.42999999999999999,"5min":0.40500000000000003,"15min":0.27300000000000002},"max":{"1min":0.80700000000000005,"5min":0.82799999999999996,"15min":1.006},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59399999999999997,"15min":0.60699999999999998},"min":{"1min":0.36599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":1.1719999999999999},"last":0.55100000000000005}]},{"osd":1,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63700000000000001,"15min":0.65200000000000002},"min":{"1min":0.39100000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.95899999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.52100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":1.212},"last":0.53000000000000003}]},{"osd":2,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57499999999999996,"15min":0.59399999999999997},"min":{"1min":0.42299999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":1.631},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.625,"15min":0.621},"min":{"1min":0.497,"5min":0.38800000000000001,"15min":0.34000000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.40100000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.59999999999999998},"min":{"1min":0.39600000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83299999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.504},{"interface":"front","average":{"1min":0.62,"5min":0.625,"15min":0.63},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.879,"5min":0.879,"15min":1.4690000000000001},"last":0.46600000000000003}]},{"osd":4,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.505,"15min":0.48799999999999999},"min":{"1min":0.34899999999999998,"5min":0.32200000000000001,"15min":0.317},"max":{"1min":0.76700000000000002,"5min":0.90200000000000002,"15min":0.94399999999999995},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53500000000000003,"15min":0.51500000000000001},"min":{"1min":0.439,"5min":0.35199999999999998,"15min":0.33900000000000002},"max":{"1min":0.80800000000000005,"5min":0.92500000000000004,"15min":0.93300000000000005},"last":0.42899999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51800000000000002,"15min":0.504},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79400000000000004,"5min":0.96799999999999997,"15min":1.732},"last":0.38},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56299999999999994,"15min":0.54500000000000004},"min":{"1min":0.42899999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.753,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.49299999999999999}]},{"osd":7,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.46200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80400000000000005,"5min":0.95399999999999996,"15min":1.0740000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.626,"15min":0.61299999999999999},"min":{"1min":0.504,"5min":0.40799999999999997,"15min":0.39100000000000001},"max":{"1min":0.85199999999999998,"5min":0.996,"15min":1.6919999999999999},"last":0.51400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:07:32.127 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:07:32.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.296+0000 7f11d4b5f700 1 Processor -- start 2024-01-27T01:07:32.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.296+0000 7f11d4b5f700 1 -- start start 2024-01-27T01:07:32.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.297+0000 7f11d4b5f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f11d01596e0 0x7f11d0159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:32.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.297+0000 7f11d4b5f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11d00c4720 0x7f11d0068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:32.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.297+0000 7f11d4b5f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11d00cadb0 0x7f11d00cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:32.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.297+0000 7f11d4b5f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d00cbf90 con 0x7f11d00c4720 2024-01-27T01:07:32.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.297+0000 7f11d4b5f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d00c97f0 con 0x7f11d01596e0 2024-01-27T01:07:32.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.297+0000 7f11d4b5f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d00cb710 con 0x7f11d00cadb0 2024-01-27T01:07:32.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.297+0000 7f11cdd9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11d00c4720 0x7f11d0068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:32.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.297+0000 7f11cdd9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11d00c4720 0x7f11d0068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45810/0 (socket says 172.21.15.46:45810) 2024-01-27T01:07:32.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.297+0000 7f11cdd9b700 1 -- 172.21.15.46:0/3691177201 learned_addr learned my addr 172.21.15.46:0/3691177201 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:32.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.297+0000 7f11ced9d700 1 --2- 172.21.15.46:0/3691177201 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11d00cadb0 0x7f11d00cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:32.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.297+0000 7f11ce59c700 1 --2- 172.21.15.46:0/3691177201 >> v2:172.21.15.106:3300/0 conn(0x7f11d01596e0 0x7f11d0159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:32.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.297+0000 7f11cdd9b700 1 -- 172.21.15.46:0/3691177201 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11d00cadb0 msgr2=0x7f11d00cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:32.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.297+0000 7f11cdd9b700 1 --2- 172.21.15.46:0/3691177201 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11d00cadb0 0x7f11d00cb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.297+0000 7f11cdd9b700 1 -- 172.21.15.46:0/3691177201 >> v2:172.21.15.106:3300/0 conn(0x7f11d01596e0 msgr2=0x7f11d0159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:32.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.297+0000 7f11cdd9b700 1 --2- 172.21.15.46:0/3691177201 >> v2:172.21.15.106:3300/0 conn(0x7f11d01596e0 0x7f11d0159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.297+0000 7f11cdd9b700 1 -- 172.21.15.46:0/3691177201 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d00c8c20 con 0x7f11d00c4720 2024-01-27T01:07:32.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.298+0000 7f11cdd9b700 1 --2- 172.21.15.46:0/3691177201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d00c4720 0x7f11d0068a40 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f11d006fa70 tx=0x7f11c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbbe4545e2764a20 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:32.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.298+0000 7f11cd59a700 1 -- 172.21.15.46:0/3691177201 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c000b860 con 0x7f11d00c4720 2024-01-27T01:07:32.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.298+0000 7f11cd59a700 1 -- 172.21.15.46:0/3691177201 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c000b9c0 con 0x7f11d00c4720 2024-01-27T01:07:32.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.298+0000 7f11cd59a700 1 -- 172.21.15.46:0/3691177201 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c000bd60 con 0x7f11d00c4720 2024-01-27T01:07:32.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.298+0000 7f11d4b5f700 1 -- 172.21.15.46:0/3691177201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d00c4720 msgr2=0x7f11d0068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:32.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.298+0000 7f11d4b5f700 1 --2- 172.21.15.46:0/3691177201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d00c4720 0x7f11d0068a40 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f11d006fa70 tx=0x7f11c0009250 comp rx=0 tx=0).stop 2024-01-27T01:07:32.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.299+0000 7f11d4b5f700 1 -- 172.21.15.46:0/3691177201 shutdown_connections 2024-01-27T01:07:32.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.299+0000 7f11d4b5f700 1 --2- 172.21.15.46:0/3691177201 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11d00cadb0 0x7f11d00cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.299+0000 7f11d4b5f700 1 --2- 172.21.15.46:0/3691177201 >> v2:172.21.15.106:3300/0 conn(0x7f11d01596e0 0x7f11d0159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.299+0000 7f11d4b5f700 1 --2- 172.21.15.46:0/3691177201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d00c4720 0x7f11d0068a40 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.299+0000 7f11d4b5f700 1 -- 172.21.15.46:0/3691177201 >> 172.21.15.46:0/3691177201 conn(0x7f11d0097820 msgr2=0x7f11d0068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:32.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.299+0000 7f11d4b5f700 1 -- 172.21.15.46:0/3691177201 shutdown_connections 2024-01-27T01:07:32.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.299+0000 7f11d4b5f700 1 -- 172.21.15.46:0/3691177201 wait complete. 2024-01-27T01:07:32.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.300+0000 7f11d4b5f700 1 Processor -- start 2024-01-27T01:07:32.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.300+0000 7f11d4b5f700 1 -- start start 2024-01-27T01:07:32.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11d4b5f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11d00c4720 0x7f11d015b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:32.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11d4b5f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11d00cadb0 0x7f11d015c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:32.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11d4b5f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d01596e0 0x7f11d015a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:32.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11d4b5f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d015d180 con 0x7f11d01596e0 2024-01-27T01:07:32.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11d4b5f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d015cee0 con 0x7f11d00c4720 2024-01-27T01:07:32.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11d4b5f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d00c6580 con 0x7f11d00cadb0 2024-01-27T01:07:32.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11ced9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d01596e0 0x7f11d015a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:32.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11ced9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d01596e0 0x7f11d015a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45820/0 (socket says 172.21.15.46:45820) 2024-01-27T01:07:32.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11ced9d700 1 -- 172.21.15.46:0/4238595760 learned_addr learned my addr 172.21.15.46:0/4238595760 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:32.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11cdd9b700 1 --2- 172.21.15.46:0/4238595760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11d00cadb0 0x7f11d015c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:32.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11ce59c700 1 --2- 172.21.15.46:0/4238595760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11d00c4720 0x7f11d015b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11ced9d700 1 -- 172.21.15.46:0/4238595760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11d00cadb0 msgr2=0x7f11d015c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11ced9d700 1 --2- 172.21.15.46:0/4238595760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11d00cadb0 0x7f11d015c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11ced9d700 1 -- 172.21.15.46:0/4238595760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11d00c4720 msgr2=0x7f11d015b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11ced9d700 1 --2- 172.21.15.46:0/4238595760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11d00c4720 0x7f11d015b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11ced9d700 1 -- 172.21.15.46:0/4238595760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c00094f0 con 0x7f11d01596e0 2024-01-27T01:07:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11ce59c700 1 --2- 172.21.15.46:0/4238595760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11d00c4720 0x7f11d015b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:07:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11ced9d700 1 --2- 172.21.15.46:0/4238595760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d01596e0 0x7f11d015a720 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009fa0 tx=0x7f11c8009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.301+0000 7f11c67fc700 1 -- 172.21.15.46:0/4238595760 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c800ed20 con 0x7f11d01596e0 2024-01-27T01:07:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.302+0000 7f11c67fc700 1 -- 172.21.15.46:0/4238595760 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c800ee80 con 0x7f11d01596e0 2024-01-27T01:07:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.302+0000 7f11d4b5f700 1 -- 172.21.15.46:0/4238595760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d00c6860 con 0x7f11d01596e0 2024-01-27T01:07:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.302+0000 7f11c67fc700 1 -- 172.21.15.46:0/4238595760 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c8014670 con 0x7f11d01596e0 2024-01-27T01:07:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.302+0000 7f11d4b5f700 1 -- 172.21.15.46:0/4238595760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d00c6e20 con 0x7f11d01596e0 2024-01-27T01:07:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.303+0000 7f11d4b5f700 1 -- 172.21.15.46:0/4238595760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d004c710 con 0x7f11d01596e0 2024-01-27T01:07:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.303+0000 7f11c67fc700 1 -- 172.21.15.46:0/4238595760 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f11c80149e0 con 0x7f11d01596e0 2024-01-27T01:07:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.304+0000 7f11c67fc700 1 --2- 172.21.15.46:0/4238595760 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f11b0067170 0x7f11b0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:32.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.304+0000 7f11c67fc700 1 -- 172.21.15.46:0/4238595760 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f11c80853b0 con 0x7f11d01596e0 2024-01-27T01:07:32.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.304+0000 7f11ce59c700 1 --2- 172.21.15.46:0/4238595760 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f11b0067170 0x7f11b0069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:32.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.305+0000 7f11ce59c700 1 --2- 172.21.15.46:0/4238595760 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f11b0067170 0x7f11b0069620 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f11b8009990 tx=0x7f11b8008040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:32.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.308+0000 7f11c67fc700 1 -- 172.21.15.46:0/4238595760 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f11c8052060 con 0x7f11d01596e0 2024-01-27T01:07:32.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.514+0000 7f11d4b5f700 1 -- 172.21.15.46:0/4238595760 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f11d00c8ec0 con 0x7f11b0067170 2024-01-27T01:07:32.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.522+0000 7f11c67fc700 1 -- 172.21.15.46:0/4238595760 <== mgr.9590 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289836 (secure 0 0 0) 0x7f11d00c8ec0 con 0x7f11b0067170 2024-01-27T01:07:32.526 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:32.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.525+0000 7f11d4b5f700 1 -- 172.21.15.46:0/4238595760 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f11b0067170 msgr2=0x7f11b0069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:32.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.525+0000 7f11d4b5f700 1 --2- 172.21.15.46:0/4238595760 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f11b0067170 0x7f11b0069620 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f11b8009990 tx=0x7f11b8008040 comp rx=0 tx=0).stop 2024-01-27T01:07:32.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.525+0000 7f11d4b5f700 1 -- 172.21.15.46:0/4238595760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d01596e0 msgr2=0x7f11d015a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:32.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.525+0000 7f11d4b5f700 1 --2- 172.21.15.46:0/4238595760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d01596e0 0x7f11d015a720 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009fa0 tx=0x7f11c8009900 comp rx=0 tx=0).stop 2024-01-27T01:07:32.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.525+0000 7f11d4b5f700 1 -- 172.21.15.46:0/4238595760 shutdown_connections 2024-01-27T01:07:32.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.525+0000 7f11d4b5f700 1 --2- 172.21.15.46:0/4238595760 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f11b0067170 0x7f11b0069620 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.525+0000 7f11d4b5f700 1 --2- 172.21.15.46:0/4238595760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d01596e0 0x7f11d015a720 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.525+0000 7f11d4b5f700 1 --2- 172.21.15.46:0/4238595760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11d00c4720 0x7f11d015b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.525+0000 7f11d4b5f700 1 --2- 172.21.15.46:0/4238595760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11d00cadb0 0x7f11d015c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.526+0000 7f11d4b5f700 1 -- 172.21.15.46:0/4238595760 >> 172.21.15.46:0/4238595760 conn(0x7f11d0097820 msgr2=0x7f11d00cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:32.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.526+0000 7f11d4b5f700 1 -- 172.21.15.46:0/4238595760 shutdown_connections 2024-01-27T01:07:32.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.526+0000 7f11d4b5f700 1 -- 172.21.15.46:0/4238595760 wait complete. 2024-01-27T01:07:32.529 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:07:32.539 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-01-27T01:07:31.870335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2456,"num_read_kb":5220,"num_write":714,"num_write_kb":3475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6581,"ondisk_log_size":6581,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2456616,"kb_used_data":25064,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752518104,"statfs":{"total":773094113280,"available":770578538496,"internally_reserved":0,"allocated":25665536,"data_stored":14108052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.946518"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.748807+0000","last_change":"2024-01-27T01:07:27.744924+0000","last_active":"2024-01-27T01:07:27.748807+0000","last_peered":"2024-01-27T01:07:27.748807+0000","last_clean":"2024-01-27T01:07:27.748807+0000","last_became_active":"2024-01-27T01:07:27.744624+0000","last_became_peered":"2024-01-27T01:07:27.744624+0000","last_unstale":"2024-01-27T01:07:27.748807+0000","last_undegraded":"2024-01-27T01:07:27.748807+0000","last_fullsized":"2024-01-27T01:07:27.748807+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:44:00.209505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.756780+0000","last_change":"2024-01-27T01:07:27.749698+0000","last_active":"2024-01-27T01:07:27.756780+0000","last_peered":"2024-01-27T01:07:27.756780+0000","last_clean":"2024-01-27T01:07:27.756780+0000","last_became_active":"2024-01-27T01:07:27.749173+0000","last_became_peered":"2024-01-27T01:07:27.749173+0000","last_unstale":"2024-01-27T01:07:27.756780+0000","last_undegraded":"2024-01-27T01:07:27.756780+0000","last_fullsized":"2024-01-27T01:07:27.756780+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:52:40.618198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.741674+0000","last_change":"2024-01-27T01:07:27.741674+0000","last_active":"2024-01-27T01:07:27.741674+0000","last_peered":"2024-01-27T01:07:27.741674+0000","last_clean":"2024-01-27T01:07:27.741674+0000","last_became_active":"2024-01-27T01:07:27.741351+0000","last_became_peered":"2024-01-27T01:07:27.741351+0000","last_unstale":"2024-01-27T01:07:27.741674+0000","last_undegraded":"2024-01-27T01:07:27.741674+0000","last_fullsized":"2024-01-27T01:07:27.741674+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:48:18.465014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.749174+0000","last_change":"2024-01-27T01:07:27.749174+0000","last_active":"2024-01-27T01:07:27.749174+0000","last_peered":"2024-01-27T01:07:27.749174+0000","last_clean":"2024-01-27T01:07:27.749174+0000","last_became_active":"2024-01-27T01:07:27.748883+0000","last_became_peered":"2024-01-27T01:07:27.748883+0000","last_unstale":"2024-01-27T01:07:27.749174+0000","last_undegraded":"2024-01-27T01:07:27.749174+0000","last_fullsized":"2024-01-27T01:07:27.749174+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:03:47.280261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.749946+0000","last_change":"2024-01-27T01:07:27.749946+0000","last_active":"2024-01-27T01:07:27.749946+0000","last_peered":"2024-01-27T01:07:27.749946+0000","last_clean":"2024-01-27T01:07:27.749946+0000","last_became_active":"2024-01-27T01:07:27.749653+0000","last_became_peered":"2024-01-27T01:07:27.749653+0000","last_unstale":"2024-01-27T01:07:27.749946+0000","last_undegraded":"2024-01-27T01:07:27.749946+0000","last_fullsized":"2024-01-27T01:07:27.749946+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:37.006113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.755473+0000","last_change":"2024-01-27T01:07:27.753636+0000","last_active":"2024-01-27T01:07:27.755473+0000","last_peered":"2024-01-27T01:07:27.755473+0000","last_clean":"2024-01-27T01:07:27.755473+0000","last_became_active":"2024-01-27T01:07:27.753305+0000","last_became_peered":"2024-01-27T01:07:27.753305+0000","last_unstale":"2024-01-27T01:07:27.755473+0000","last_undegraded":"2024-01-27T01:07:27.755473+0000","last_fullsized":"2024-01-27T01:07:27.755473+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:28:31.571123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.743608+0000","last_change":"2024-01-27T01:07:27.743608+0000","last_active":"2024-01-27T01:07:27.743608+0000","last_peered":"2024-01-27T01:07:27.743608+0000","last_clean":"2024-01-27T01:07:27.743608+0000","last_became_active":"2024-01-27T01:07:27.743324+0000","last_became_peered":"2024-01-27T01:07:27.743324+0000","last_unstale":"2024-01-27T01:07:27.743608+0000","last_undegraded":"2024-01-27T01:07:27.743608+0000","last_fullsized":"2024-01-27T01:07:27.743608+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:25:38.146534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.744506+0000","last_change":"2024-01-27T01:07:27.744506+0000","last_active":"2024-01-27T01:07:27.744506+0000","last_peered":"2024-01-27T01:07:27.744506+0000","last_clean":"2024-01-27T01:07:27.744506+0000","last_became_active":"2024-01-27T01:07:27.744222+0000","last_became_peered":"2024-01-27T01:07:27.744222+0000","last_unstale":"2024-01-27T01:07:27.744506+0000","last_undegraded":"2024-01-27T01:07:27.744506+0000","last_fullsized":"2024-01-27T01:07:27.744506+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:24:38.456700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.754529+0000","last_change":"2024-01-27T01:07:27.753284+0000","last_active":"2024-01-27T01:07:27.754529+0000","last_peered":"2024-01-27T01:07:27.754529+0000","last_clean":"2024-01-27T01:07:27.754529+0000","last_became_active":"2024-01-27T01:07:27.752819+0000","last_became_peered":"2024-01-27T01:07:27.752819+0000","last_unstale":"2024-01-27T01:07:27.754529+0000","last_undegraded":"2024-01-27T01:07:27.754529+0000","last_fullsized":"2024-01-27T01:07:27.754529+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:58:33.718993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.743902+0000","last_change":"2024-01-27T01:07:27.743902+0000","last_active":"2024-01-27T01:07:27.743902+0000","last_peered":"2024-01-27T01:07:27.743902+0000","last_clean":"2024-01-27T01:07:27.743902+0000","last_became_active":"2024-01-27T01:07:27.743270+0000","last_became_peered":"2024-01-27T01:07:27.743270+0000","last_unstale":"2024-01-27T01:07:27.743902+0000","last_undegraded":"2024-01-27T01:07:27.743902+0000","last_fullsized":"2024-01-27T01:07:27.743902+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:24:38.456700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.743746+0000","last_change":"2024-01-27T01:07:27.743746+0000","last_active":"2024-01-27T01:07:27.743746+0000","last_peered":"2024-01-27T01:07:27.743746+0000","last_clean":"2024-01-27T01:07:27.743746+0000","last_became_active":"2024-01-27T01:07:27.743414+0000","last_became_peered":"2024-01-27T01:07:27.743414+0000","last_unstale":"2024-01-27T01:07:27.743746+0000","last_undegraded":"2024-01-27T01:07:27.743746+0000","last_fullsized":"2024-01-27T01:07:27.743746+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:37.006113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.750780+0000","last_change":"2024-01-27T01:07:27.748096+0000","last_active":"2024-01-27T01:07:27.750780+0000","last_peered":"2024-01-27T01:07:27.750780+0000","last_clean":"2024-01-27T01:07:27.750780+0000","last_became_active":"2024-01-27T01:07:27.747801+0000","last_became_peered":"2024-01-27T01:07:27.747801+0000","last_unstale":"2024-01-27T01:07:27.750780+0000","last_undegraded":"2024-01-27T01:07:27.750780+0000","last_fullsized":"2024-01-27T01:07:27.750780+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:20:52.644616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.746806+0000","last_change":"2024-01-27T01:07:27.746806+0000","last_active":"2024-01-27T01:07:27.746806+0000","last_peered":"2024-01-27T01:07:27.746806+0000","last_clean":"2024-01-27T01:07:27.746806+0000","last_became_active":"2024-01-27T01:07:27.745168+0000","last_became_peered":"2024-01-27T01:07:27.745168+0000","last_unstale":"2024-01-27T01:07:27.746806+0000","last_undegraded":"2024-01-27T01:07:27.746806+0000","last_fullsized":"2024-01-27T01:07:27.746806+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:34:21.004352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.753177+0000","last_change":"2024-01-27T01:07:27.749049+0000","last_active":"2024-01-27T01:07:27.753177+0000","last_peered":"2024-01-27T01:07:27.753177+0000","last_clean":"2024-01-27T01:07:27.753177+0000","last_became_active":"2024-01-27T01:07:27.748736+0000","last_became_peered":"2024-01-27T01:07:27.748736+0000","last_unstale":"2024-01-27T01:07:27.753177+0000","last_undegraded":"2024-01-27T01:07:27.753177+0000","last_fullsized":"2024-01-27T01:07:27.753177+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:23:10.979296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.755236+0000","last_change":"2024-01-27T01:07:27.755236+0000","last_active":"2024-01-27T01:07:27.755236+0000","last_peered":"2024-01-27T01:07:27.755236+0000","last_clean":"2024-01-27T01:07:27.755236+0000","last_became_active":"2024-01-27T01:07:27.754945+0000","last_became_peered":"2024-01-27T01:07:27.754945+0000","last_unstale":"2024-01-27T01:07:27.755236+0000","last_undegraded":"2024-01-27T01:07:27.755236+0000","last_fullsized":"2024-01-27T01:07:27.755236+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:07:48.619221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.745495+0000","last_change":"2024-01-27T01:07:27.745495+0000","last_active":"2024-01-27T01:07:27.745495+0000","last_peered":"2024-01-27T01:07:27.745495+0000","last_clean":"2024-01-27T01:07:27.745495+0000","last_became_active":"2024-01-27T01:07:27.744986+0000","last_became_peered":"2024-01-27T01:07:27.744986+0000","last_unstale":"2024-01-27T01:07:27.745495+0000","last_undegraded":"2024-01-27T01:07:27.745495+0000","last_fullsized":"2024-01-27T01:07:27.745495+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:08.374701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.749357+0000","last_change":"2024-01-27T01:07:27.749357+0000","last_active":"2024-01-27T01:07:27.749357+0000","last_peered":"2024-01-27T01:07:27.749357+0000","last_clean":"2024-01-27T01:07:27.749357+0000","last_became_active":"2024-01-27T01:07:27.745281+0000","last_became_peered":"2024-01-27T01:07:27.745281+0000","last_unstale":"2024-01-27T01:07:27.749357+0000","last_undegraded":"2024-01-27T01:07:27.749357+0000","last_fullsized":"2024-01-27T01:07:27.749357+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:11.619364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.736143+0000","last_change":"2024-01-27T01:07:27.736143+0000","last_active":"2024-01-27T01:07:27.736143+0000","last_peered":"2024-01-27T01:07:27.736143+0000","last_clean":"2024-01-27T01:07:27.736143+0000","last_became_active":"2024-01-27T01:07:27.735824+0000","last_became_peered":"2024-01-27T01:07:27.735824+0000","last_unstale":"2024-01-27T01:07:27.736143+0000","last_undegraded":"2024-01-27T01:07:27.736143+0000","last_fullsized":"2024-01-27T01:07:27.736143+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:05.536246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.756376+0000","last_change":"2024-01-27T01:07:27.756376+0000","last_active":"2024-01-27T01:07:27.756376+0000","last_peered":"2024-01-27T01:07:27.756376+0000","last_clean":"2024-01-27T01:07:27.756376+0000","last_became_active":"2024-01-27T01:07:27.756152+0000","last_became_peered":"2024-01-27T01:07:27.756152+0000","last_unstale":"2024-01-27T01:07:27.756376+0000","last_undegraded":"2024-01-27T01:07:27.756376+0000","last_fullsized":"2024-01-27T01:07:27.756376+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:02:04.959467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.758763+0000","last_change":"2024-01-27T01:07:27.758252+0000","last_active":"2024-01-27T01:07:27.758763+0000","last_peered":"2024-01-27T01:07:27.758763+0000","last_clean":"2024-01-27T01:07:27.758763+0000","last_became_active":"2024-01-27T01:07:27.757929+0000","last_became_peered":"2024-01-27T01:07:27.757929+0000","last_unstale":"2024-01-27T01:07:27.758763+0000","last_undegraded":"2024-01-27T01:07:27.758763+0000","last_fullsized":"2024-01-27T01:07:27.758763+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:30.518041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.752626+0000","last_change":"2024-01-27T01:07:27.752626+0000","last_active":"2024-01-27T01:07:27.752626+0000","last_peered":"2024-01-27T01:07:27.752626+0000","last_clean":"2024-01-27T01:07:27.752626+0000","last_became_active":"2024-01-27T01:07:27.752363+0000","last_became_peered":"2024-01-27T01:07:27.752363+0000","last_unstale":"2024-01-27T01:07:27.752626+0000","last_undegraded":"2024-01-27T01:07:27.752626+0000","last_fullsized":"2024-01-27T01:07:27.752626+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:03:44.824769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.748467+0000","last_change":"2024-01-27T01:07:27.748467+0000","last_active":"2024-01-27T01:07:27.748467+0000","last_peered":"2024-01-27T01:07:27.748467+0000","last_clean":"2024-01-27T01:07:27.748467+0000","last_became_active":"2024-01-27T01:07:27.748100+0000","last_became_peered":"2024-01-27T01:07:27.748100+0000","last_unstale":"2024-01-27T01:07:27.748467+0000","last_undegraded":"2024-01-27T01:07:27.748467+0000","last_fullsized":"2024-01-27T01:07:27.748467+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:38:29.701960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.755439+0000","last_change":"2024-01-27T01:07:27.755439+0000","last_active":"2024-01-27T01:07:27.755439+0000","last_peered":"2024-01-27T01:07:27.755439+0000","last_clean":"2024-01-27T01:07:27.755439+0000","last_became_active":"2024-01-27T01:07:27.755126+0000","last_became_peered":"2024-01-27T01:07:27.755126+0000","last_unstale":"2024-01-27T01:07:27.755439+0000","last_undegraded":"2024-01-27T01:07:27.755439+0000","last_fullsized":"2024-01-27T01:07:27.755439+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:43:37.067000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.745155+0000","last_change":"2024-01-27T01:07:27.745155+0000","last_active":"2024-01-27T01:07:27.745155+0000","last_peered":"2024-01-27T01:07:27.745155+0000","last_clean":"2024-01-27T01:07:27.745155+0000","last_became_active":"2024-01-27T01:07:27.744562+0000","last_became_peered":"2024-01-27T01:07:27.744562+0000","last_unstale":"2024-01-27T01:07:27.745155+0000","last_undegraded":"2024-01-27T01:07:27.745155+0000","last_fullsized":"2024-01-27T01:07:27.745155+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:28:42.362768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.749426+0000","last_change":"2024-01-27T01:07:27.745357+0000","last_active":"2024-01-27T01:07:27.749426+0000","last_peered":"2024-01-27T01:07:27.749426+0000","last_clean":"2024-01-27T01:07:27.749426+0000","last_became_active":"2024-01-27T01:07:27.745075+0000","last_became_peered":"2024-01-27T01:07:27.745075+0000","last_unstale":"2024-01-27T01:07:27.749426+0000","last_undegraded":"2024-01-27T01:07:27.749426+0000","last_fullsized":"2024-01-27T01:07:27.749426+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:37:01.620442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.744187+0000","last_change":"2024-01-27T01:07:27.744187+0000","last_active":"2024-01-27T01:07:27.744187+0000","last_peered":"2024-01-27T01:07:27.744187+0000","last_clean":"2024-01-27T01:07:27.744187+0000","last_became_active":"2024-01-27T01:07:27.743889+0000","last_became_peered":"2024-01-27T01:07:27.743889+0000","last_unstale":"2024-01-27T01:07:27.744187+0000","last_undegraded":"2024-01-27T01:07:27.744187+0000","last_fullsized":"2024-01-27T01:07:27.744187+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:11.619364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.743304+0000","last_change":"2024-01-27T01:07:27.743304+0000","last_active":"2024-01-27T01:07:27.743304+0000","last_peered":"2024-01-27T01:07:27.743304+0000","last_clean":"2024-01-27T01:07:27.743304+0000","last_became_active":"2024-01-27T01:07:27.742783+0000","last_became_peered":"2024-01-27T01:07:27.742783+0000","last_unstale":"2024-01-27T01:07:27.743304+0000","last_undegraded":"2024-01-27T01:07:27.743304+0000","last_fullsized":"2024-01-27T01:07:27.743304+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:45:07.998194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.748843+0000","last_change":"2024-01-27T01:07:27.746921+0000","last_active":"2024-01-27T01:07:27.748843+0000","last_peered":"2024-01-27T01:07:27.748843+0000","last_clean":"2024-01-27T01:07:27.748843+0000","last_became_active":"2024-01-27T01:07:27.746630+0000","last_became_peered":"2024-01-27T01:07:27.746630+0000","last_unstale":"2024-01-27T01:07:27.748843+0000","last_undegraded":"2024-01-27T01:07:27.748843+0000","last_fullsized":"2024-01-27T01:07:27.748843+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:03:41.111732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.753223+0000","last_change":"2024-01-27T01:07:27.749637+0000","last_active":"2024-01-27T01:07:27.753223+0000","last_peered":"2024-01-27T01:07:27.753223+0000","last_clean":"2024-01-27T01:07:27.753223+0000","last_became_active":"2024-01-27T01:07:27.749170+0000","last_became_peered":"2024-01-27T01:07:27.749170+0000","last_unstale":"2024-01-27T01:07:27.753223+0000","last_undegraded":"2024-01-27T01:07:27.753223+0000","last_fullsized":"2024-01-27T01:07:27.753223+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:10:06.064380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.752984+0000","last_change":"2024-01-27T01:07:27.748872+0000","last_active":"2024-01-27T01:07:27.752984+0000","last_peered":"2024-01-27T01:07:27.752984+0000","last_clean":"2024-01-27T01:07:27.752984+0000","last_became_active":"2024-01-27T01:07:27.748549+0000","last_became_peered":"2024-01-27T01:07:27.748549+0000","last_unstale":"2024-01-27T01:07:27.752984+0000","last_undegraded":"2024-01-27T01:07:27.752984+0000","last_fullsized":"2024-01-27T01:07:27.752984+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:58:33.718993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.757752+0000","last_change":"2024-01-27T01:07:27.755023+0000","last_active":"2024-01-27T01:07:27.757752+0000","last_peered":"2024-01-27T01:07:27.757752+0000","last_clean":"2024-01-27T01:07:27.757752+0000","last_became_active":"2024-01-27T01:07:27.754304+0000","last_became_peered":"2024-01-27T01:07:27.754304+0000","last_unstale":"2024-01-27T01:07:27.757752+0000","last_undegraded":"2024-01-27T01:07:27.757752+0000","last_fullsized":"2024-01-27T01:07:27.757752+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:28:12.245965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.751905+0000","last_change":"2024-01-27T01:07:27.747314+0000","last_active":"2024-01-27T01:07:27.751905+0000","last_peered":"2024-01-27T01:07:27.751905+0000","last_clean":"2024-01-27T01:07:27.751905+0000","last_became_active":"2024-01-27T01:07:27.746947+0000","last_became_peered":"2024-01-27T01:07:27.746947+0000","last_unstale":"2024-01-27T01:07:27.751905+0000","last_undegraded":"2024-01-27T01:07:27.751905+0000","last_fullsized":"2024-01-27T01:07:27.751905+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:25:01.603669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.758346+0000","last_change":"2024-01-27T01:07:27.755462+0000","last_active":"2024-01-27T01:07:27.758346+0000","last_peered":"2024-01-27T01:07:27.758346+0000","last_clean":"2024-01-27T01:07:27.758346+0000","last_became_active":"2024-01-27T01:07:27.755131+0000","last_became_peered":"2024-01-27T01:07:27.755131+0000","last_unstale":"2024-01-27T01:07:27.758346+0000","last_undegraded":"2024-01-27T01:07:27.758346+0000","last_fullsized":"2024-01-27T01:07:27.758346+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:32:22.760237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"242'4","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.945348+0000","last_change":"2024-01-27T01:07:26.950037+0000","last_active":"2024-01-27T01:07:27.945348+0000","last_peered":"2024-01-27T01:07:27.945348+0000","last_clean":"2024-01-27T01:07:27.945348+0000","last_became_active":"2024-01-27T01:07:26.949604+0000","last_became_peered":"2024-01-27T01:07:26.949604+0000","last_unstale":"2024-01-27T01:07:27.945348+0000","last_undegraded":"2024-01-27T01:07:27.945348+0000","last_fullsized":"2024-01-27T01:07:27.945348+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:01:17.838521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.755690+0000","last_change":"2024-01-27T01:07:27.755690+0000","last_active":"2024-01-27T01:07:27.755690+0000","last_peered":"2024-01-27T01:07:27.755690+0000","last_clean":"2024-01-27T01:07:27.755690+0000","last_became_active":"2024-01-27T01:07:27.755402+0000","last_became_peered":"2024-01-27T01:07:27.755402+0000","last_unstale":"2024-01-27T01:07:27.755690+0000","last_undegraded":"2024-01-27T01:07:27.755690+0000","last_fullsized":"2024-01-27T01:07:27.755690+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:25:59.841939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"242'2","reported_seq":11,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.943676+0000","last_change":"2024-01-27T01:07:26.951267+0000","last_active":"2024-01-27T01:07:27.943676+0000","last_peered":"2024-01-27T01:07:27.943676+0000","last_clean":"2024-01-27T01:07:27.943676+0000","last_became_active":"2024-01-27T01:07:26.950967+0000","last_became_peered":"2024-01-27T01:07:26.950967+0000","last_unstale":"2024-01-27T01:07:27.943676+0000","last_undegraded":"2024-01-27T01:07:27.943676+0000","last_fullsized":"2024-01-27T01:07:27.943676+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:37:00.613294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.753950+0000","last_change":"2024-01-27T01:07:27.752814+0000","last_active":"2024-01-27T01:07:27.753950+0000","last_peered":"2024-01-27T01:07:27.753950+0000","last_clean":"2024-01-27T01:07:27.753950+0000","last_became_active":"2024-01-27T01:07:27.752459+0000","last_became_peered":"2024-01-27T01:07:27.752459+0000","last_unstale":"2024-01-27T01:07:27.753950+0000","last_undegraded":"2024-01-27T01:07:27.753950+0000","last_fullsized":"2024-01-27T01:07:27.753950+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:44:32.042391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"242'2","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.947326+0000","last_change":"2024-01-27T01:07:26.945958+0000","last_active":"2024-01-27T01:07:27.947326+0000","last_peered":"2024-01-27T01:07:27.947326+0000","last_clean":"2024-01-27T01:07:27.947326+0000","last_became_active":"2024-01-27T01:07:26.945671+0000","last_became_peered":"2024-01-27T01:07:26.945671+0000","last_unstale":"2024-01-27T01:07:27.947326+0000","last_undegraded":"2024-01-27T01:07:27.947326+0000","last_fullsized":"2024-01-27T01:07:27.947326+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:20.901463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.757603+0000","last_change":"2024-01-27T01:07:27.756421+0000","last_active":"2024-01-27T01:07:27.757603+0000","last_peered":"2024-01-27T01:07:27.757603+0000","last_clean":"2024-01-27T01:07:27.757603+0000","last_became_active":"2024-01-27T01:07:27.756102+0000","last_became_peered":"2024-01-27T01:07:27.756102+0000","last_unstale":"2024-01-27T01:07:27.757603+0000","last_undegraded":"2024-01-27T01:07:27.757603+0000","last_fullsized":"2024-01-27T01:07:27.757603+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:23:49.531160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"242'1","reported_seq":10,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.747354+0000","last_change":"2024-01-27T01:07:26.957169+0000","last_active":"2024-01-27T01:07:27.747354+0000","last_peered":"2024-01-27T01:07:27.747354+0000","last_clean":"2024-01-27T01:07:27.747354+0000","last_became_active":"2024-01-27T01:07:26.956902+0000","last_became_peered":"2024-01-27T01:07:26.956902+0000","last_unstale":"2024-01-27T01:07:27.747354+0000","last_undegraded":"2024-01-27T01:07:27.747354+0000","last_fullsized":"2024-01-27T01:07:27.747354+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:27:55.458182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.752682+0000","last_change":"2024-01-27T01:07:27.752682+0000","last_active":"2024-01-27T01:07:27.752682+0000","last_peered":"2024-01-27T01:07:27.752682+0000","last_clean":"2024-01-27T01:07:27.752682+0000","last_became_active":"2024-01-27T01:07:27.752401+0000","last_became_peered":"2024-01-27T01:07:27.752401+0000","last_unstale":"2024-01-27T01:07:27.752682+0000","last_undegraded":"2024-01-27T01:07:27.752682+0000","last_fullsized":"2024-01-27T01:07:27.752682+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:55:40.962728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"242'1","reported_seq":11,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.943898+0000","last_change":"2024-01-27T01:07:26.951702+0000","last_active":"2024-01-27T01:07:27.943898+0000","last_peered":"2024-01-27T01:07:27.943898+0000","last_clean":"2024-01-27T01:07:27.943898+0000","last_became_active":"2024-01-27T01:07:26.951367+0000","last_became_peered":"2024-01-27T01:07:26.951367+0000","last_unstale":"2024-01-27T01:07:27.943898+0000","last_undegraded":"2024-01-27T01:07:27.943898+0000","last_fullsized":"2024-01-27T01:07:27.943898+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:25:00.596520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.750189+0000","last_change":"2024-01-27T01:07:27.750189+0000","last_active":"2024-01-27T01:07:27.750189+0000","last_peered":"2024-01-27T01:07:27.750189+0000","last_clean":"2024-01-27T01:07:27.750189+0000","last_became_active":"2024-01-27T01:07:27.749867+0000","last_became_peered":"2024-01-27T01:07:27.749867+0000","last_unstale":"2024-01-27T01:07:27.750189+0000","last_undegraded":"2024-01-27T01:07:27.750189+0000","last_fullsized":"2024-01-27T01:07:27.750189+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:39:46.126956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.958068+0000","last_change":"2024-01-27T01:07:26.954992+0000","last_active":"2024-01-27T01:07:26.958068+0000","last_peered":"2024-01-27T01:07:26.958068+0000","last_clean":"2024-01-27T01:07:26.958068+0000","last_became_active":"2024-01-27T01:07:26.954701+0000","last_became_peered":"2024-01-27T01:07:26.954701+0000","last_unstale":"2024-01-27T01:07:26.958068+0000","last_undegraded":"2024-01-27T01:07:26.958068+0000","last_fullsized":"2024-01-27T01:07:26.958068+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:24.976708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.743124+0000","last_change":"2024-01-27T01:07:27.743124+0000","last_active":"2024-01-27T01:07:27.743124+0000","last_peered":"2024-01-27T01:07:27.743124+0000","last_clean":"2024-01-27T01:07:27.743124+0000","last_became_active":"2024-01-27T01:07:27.742799+0000","last_became_peered":"2024-01-27T01:07:27.742799+0000","last_unstale":"2024-01-27T01:07:27.743124+0000","last_undegraded":"2024-01-27T01:07:27.743124+0000","last_fullsized":"2024-01-27T01:07:27.743124+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:14.714021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":10,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.740048+0000","last_change":"2024-01-27T01:07:26.946511+0000","last_active":"2024-01-27T01:07:27.740048+0000","last_peered":"2024-01-27T01:07:27.740048+0000","last_clean":"2024-01-27T01:07:27.740048+0000","last_became_active":"2024-01-27T01:07:26.946186+0000","last_became_peered":"2024-01-27T01:07:26.946186+0000","last_unstale":"2024-01-27T01:07:27.740048+0000","last_undegraded":"2024-01-27T01:07:27.740048+0000","last_fullsized":"2024-01-27T01:07:27.740048+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:24:39.506445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.749767+0000","last_change":"2024-01-27T01:07:27.749767+0000","last_active":"2024-01-27T01:07:27.749767+0000","last_peered":"2024-01-27T01:07:27.749767+0000","last_clean":"2024-01-27T01:07:27.749767+0000","last_became_active":"2024-01-27T01:07:27.749508+0000","last_became_peered":"2024-01-27T01:07:27.749508+0000","last_unstale":"2024-01-27T01:07:27.749767+0000","last_undegraded":"2024-01-27T01:07:27.749767+0000","last_fullsized":"2024-01-27T01:07:27.749767+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:56:10.391021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":10,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.734495+0000","last_change":"2024-01-27T01:07:26.947663+0000","last_active":"2024-01-27T01:07:27.734495+0000","last_peered":"2024-01-27T01:07:27.734495+0000","last_clean":"2024-01-27T01:07:27.734495+0000","last_became_active":"2024-01-27T01:07:26.947378+0000","last_became_peered":"2024-01-27T01:07:26.947378+0000","last_unstale":"2024-01-27T01:07:27.734495+0000","last_undegraded":"2024-01-27T01:07:27.734495+0000","last_fullsized":"2024-01-27T01:07:27.734495+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:28:30.563975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.749325+0000","last_change":"2024-01-27T01:07:27.749325+0000","last_active":"2024-01-27T01:07:27.749325+0000","last_peered":"2024-01-27T01:07:27.749325+0000","last_clean":"2024-01-27T01:07:27.749325+0000","last_became_active":"2024-01-27T01:07:27.744840+0000","last_became_peered":"2024-01-27T01:07:27.744840+0000","last_unstale":"2024-01-27T01:07:27.749325+0000","last_undegraded":"2024-01-27T01:07:27.749325+0000","last_fullsized":"2024-01-27T01:07:27.749325+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:15:38.535322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.949785+0000","last_change":"2024-01-27T01:07:26.948157+0000","last_active":"2024-01-27T01:07:26.949785+0000","last_peered":"2024-01-27T01:07:26.949785+0000","last_clean":"2024-01-27T01:07:26.949785+0000","last_became_active":"2024-01-27T01:07:26.947870+0000","last_became_peered":"2024-01-27T01:07:26.947870+0000","last_unstale":"2024-01-27T01:07:26.949785+0000","last_undegraded":"2024-01-27T01:07:26.949785+0000","last_fullsized":"2024-01-27T01:07:26.949785+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:04:16.755137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.744220+0000","last_change":"2024-01-27T01:07:27.744220+0000","last_active":"2024-01-27T01:07:27.744220+0000","last_peered":"2024-01-27T01:07:27.744220+0000","last_clean":"2024-01-27T01:07:27.744220+0000","last_became_active":"2024-01-27T01:07:27.742448+0000","last_became_peered":"2024-01-27T01:07:27.742448+0000","last_unstale":"2024-01-27T01:07:27.744220+0000","last_undegraded":"2024-01-27T01:07:27.744220+0000","last_fullsized":"2024-01-27T01:07:27.744220+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:03.017967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"242'3","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.942331+0000","last_change":"2024-01-27T01:07:26.953006+0000","last_active":"2024-01-27T01:07:27.942331+0000","last_peered":"2024-01-27T01:07:27.942331+0000","last_clean":"2024-01-27T01:07:27.942331+0000","last_became_active":"2024-01-27T01:07:26.952730+0000","last_became_peered":"2024-01-27T01:07:26.952730+0000","last_unstale":"2024-01-27T01:07:27.942331+0000","last_undegraded":"2024-01-27T01:07:27.942331+0000","last_fullsized":"2024-01-27T01:07:27.942331+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:14.273722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.747921+0000","last_change":"2024-01-27T01:07:27.747921+0000","last_active":"2024-01-27T01:07:27.747921+0000","last_peered":"2024-01-27T01:07:27.747921+0000","last_clean":"2024-01-27T01:07:27.747921+0000","last_became_active":"2024-01-27T01:07:27.747382+0000","last_became_peered":"2024-01-27T01:07:27.747382+0000","last_unstale":"2024-01-27T01:07:27.747921+0000","last_undegraded":"2024-01-27T01:07:27.747921+0000","last_fullsized":"2024-01-27T01:07:27.747921+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:43:44.312681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"242'6","reported_seq":16,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.948807+0000","last_change":"2024-01-27T01:07:26.951048+0000","last_active":"2024-01-27T01:07:27.948807+0000","last_peered":"2024-01-27T01:07:27.948807+0000","last_clean":"2024-01-27T01:07:27.948807+0000","last_became_active":"2024-01-27T01:07:26.950750+0000","last_became_peered":"2024-01-27T01:07:26.950750+0000","last_unstale":"2024-01-27T01:07:27.948807+0000","last_undegraded":"2024-01-27T01:07:27.948807+0000","last_fullsized":"2024-01-27T01:07:27.948807+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:48:29.363281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.758005+0000","last_change":"2024-01-27T01:07:27.756638+0000","last_active":"2024-01-27T01:07:27.758005+0000","last_peered":"2024-01-27T01:07:27.758005+0000","last_clean":"2024-01-27T01:07:27.758005+0000","last_became_active":"2024-01-27T01:07:27.756219+0000","last_became_peered":"2024-01-27T01:07:27.756219+0000","last_unstale":"2024-01-27T01:07:27.758005+0000","last_undegraded":"2024-01-27T01:07:27.758005+0000","last_fullsized":"2024-01-27T01:07:27.758005+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:24:43.565560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"242'2","reported_seq":11,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.945780+0000","last_change":"2024-01-27T01:07:26.957374+0000","last_active":"2024-01-27T01:07:27.945780+0000","last_peered":"2024-01-27T01:07:27.945780+0000","last_clean":"2024-01-27T01:07:27.945780+0000","last_became_active":"2024-01-27T01:07:26.957184+0000","last_became_peered":"2024-01-27T01:07:26.957184+0000","last_unstale":"2024-01-27T01:07:27.945780+0000","last_undegraded":"2024-01-27T01:07:27.945780+0000","last_fullsized":"2024-01-27T01:07:27.945780+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:43:44.001485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.741016+0000","last_change":"2024-01-27T01:07:27.741016+0000","last_active":"2024-01-27T01:07:27.741016+0000","last_peered":"2024-01-27T01:07:27.741016+0000","last_clean":"2024-01-27T01:07:27.741016+0000","last_became_active":"2024-01-27T01:07:27.740724+0000","last_became_peered":"2024-01-27T01:07:27.740724+0000","last_unstale":"2024-01-27T01:07:27.741016+0000","last_undegraded":"2024-01-27T01:07:27.741016+0000","last_fullsized":"2024-01-27T01:07:27.741016+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:19:56.759979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.955598+0000","last_change":"2024-01-27T01:07:26.951785+0000","last_active":"2024-01-27T01:07:26.955598+0000","last_peered":"2024-01-27T01:07:26.955598+0000","last_clean":"2024-01-27T01:07:26.955598+0000","last_became_active":"2024-01-27T01:07:26.951519+0000","last_became_peered":"2024-01-27T01:07:26.951519+0000","last_unstale":"2024-01-27T01:07:26.955598+0000","last_undegraded":"2024-01-27T01:07:26.955598+0000","last_fullsized":"2024-01-27T01:07:26.955598+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:06.511483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.755983+0000","last_change":"2024-01-27T01:07:27.755983+0000","last_active":"2024-01-27T01:07:27.755983+0000","last_peered":"2024-01-27T01:07:27.755983+0000","last_clean":"2024-01-27T01:07:27.755983+0000","last_became_active":"2024-01-27T01:07:27.755715+0000","last_became_peered":"2024-01-27T01:07:27.755715+0000","last_unstale":"2024-01-27T01:07:27.755983+0000","last_undegraded":"2024-01-27T01:07:27.755983+0000","last_fullsized":"2024-01-27T01:07:27.755983+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:11.556927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.955622+0000","last_change":"2024-01-27T01:07:26.952237+0000","last_active":"2024-01-27T01:07:26.955622+0000","last_peered":"2024-01-27T01:07:26.955622+0000","last_clean":"2024-01-27T01:07:26.955622+0000","last_became_active":"2024-01-27T01:07:26.951971+0000","last_became_peered":"2024-01-27T01:07:26.951971+0000","last_unstale":"2024-01-27T01:07:26.955622+0000","last_undegraded":"2024-01-27T01:07:26.955622+0000","last_fullsized":"2024-01-27T01:07:26.955622+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:43:36.059851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.746452+0000","last_change":"2024-01-27T01:07:27.746452+0000","last_active":"2024-01-27T01:07:27.746452+0000","last_peered":"2024-01-27T01:07:27.746452+0000","last_clean":"2024-01-27T01:07:27.746452+0000","last_became_active":"2024-01-27T01:07:27.746174+0000","last_became_peered":"2024-01-27T01:07:27.746174+0000","last_unstale":"2024-01-27T01:07:27.746452+0000","last_undegraded":"2024-01-27T01:07:27.746452+0000","last_fullsized":"2024-01-27T01:07:27.746452+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:32.778691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"242'8","reported_seq":18,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.948345+0000","last_change":"2024-01-27T01:07:26.946801+0000","last_active":"2024-01-27T01:07:27.948345+0000","last_peered":"2024-01-27T01:07:27.948345+0000","last_clean":"2024-01-27T01:07:27.948345+0000","last_became_active":"2024-01-27T01:07:26.946453+0000","last_became_peered":"2024-01-27T01:07:26.946453+0000","last_unstale":"2024-01-27T01:07:27.948345+0000","last_undegraded":"2024-01-27T01:07:27.948345+0000","last_fullsized":"2024-01-27T01:07:27.948345+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:11:44.275714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.746480+0000","last_change":"2024-01-27T01:07:27.746480+0000","last_active":"2024-01-27T01:07:27.746480+0000","last_peered":"2024-01-27T01:07:27.746480+0000","last_clean":"2024-01-27T01:07:27.746480+0000","last_became_active":"2024-01-27T01:07:27.746131+0000","last_became_peered":"2024-01-27T01:07:27.746131+0000","last_unstale":"2024-01-27T01:07:27.746480+0000","last_undegraded":"2024-01-27T01:07:27.746480+0000","last_fullsized":"2024-01-27T01:07:27.746480+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:06:55.090910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"242'1","reported_seq":11,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.945219+0000","last_change":"2024-01-27T01:07:26.958336+0000","last_active":"2024-01-27T01:07:27.945219+0000","last_peered":"2024-01-27T01:07:27.945219+0000","last_clean":"2024-01-27T01:07:27.945219+0000","last_became_active":"2024-01-27T01:07:26.957817+0000","last_became_peered":"2024-01-27T01:07:26.957817+0000","last_unstale":"2024-01-27T01:07:27.945219+0000","last_undegraded":"2024-01-27T01:07:27.945219+0000","last_fullsized":"2024-01-27T01:07:27.945219+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:45.033976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.743490+0000","last_change":"2024-01-27T01:07:27.743490+0000","last_active":"2024-01-27T01:07:27.743490+0000","last_peered":"2024-01-27T01:07:27.743490+0000","last_clean":"2024-01-27T01:07:27.743490+0000","last_became_active":"2024-01-27T01:07:27.743207+0000","last_became_peered":"2024-01-27T01:07:27.743207+0000","last_unstale":"2024-01-27T01:07:27.743490+0000","last_undegraded":"2024-01-27T01:07:27.743490+0000","last_fullsized":"2024-01-27T01:07:27.743490+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:19.339556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"242'2","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.944670+0000","last_change":"2024-01-27T01:07:26.946877+0000","last_active":"2024-01-27T01:07:27.944670+0000","last_peered":"2024-01-27T01:07:27.944670+0000","last_clean":"2024-01-27T01:07:27.944670+0000","last_became_active":"2024-01-27T01:07:26.946596+0000","last_became_peered":"2024-01-27T01:07:26.946596+0000","last_unstale":"2024-01-27T01:07:27.944670+0000","last_undegraded":"2024-01-27T01:07:27.944670+0000","last_fullsized":"2024-01-27T01:07:27.944670+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:27:41.114970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.749680+0000","last_change":"2024-01-27T01:07:27.746482+0000","last_active":"2024-01-27T01:07:27.749680+0000","last_peered":"2024-01-27T01:07:27.749680+0000","last_clean":"2024-01-27T01:07:27.749680+0000","last_became_active":"2024-01-27T01:07:27.746052+0000","last_became_peered":"2024-01-27T01:07:27.746052+0000","last_unstale":"2024-01-27T01:07:27.749680+0000","last_undegraded":"2024-01-27T01:07:27.749680+0000","last_fullsized":"2024-01-27T01:07:27.749680+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:32.387353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"242'1","reported_seq":10,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.740668+0000","last_change":"2024-01-27T01:07:26.950943+0000","last_active":"2024-01-27T01:07:27.740668+0000","last_peered":"2024-01-27T01:07:27.740668+0000","last_clean":"2024-01-27T01:07:27.740668+0000","last_became_active":"2024-01-27T01:07:26.950251+0000","last_became_peered":"2024-01-27T01:07:26.950251+0000","last_unstale":"2024-01-27T01:07:27.740668+0000","last_undegraded":"2024-01-27T01:07:27.740668+0000","last_fullsized":"2024-01-27T01:07:27.740668+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:31.380205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.742779+0000","last_change":"2024-01-27T01:07:27.742779+0000","last_active":"2024-01-27T01:07:27.742779+0000","last_peered":"2024-01-27T01:07:27.742779+0000","last_clean":"2024-01-27T01:07:27.742779+0000","last_became_active":"2024-01-27T01:07:27.742462+0000","last_became_peered":"2024-01-27T01:07:27.742462+0000","last_unstale":"2024-01-27T01:07:27.742779+0000","last_undegraded":"2024-01-27T01:07:27.742779+0000","last_fullsized":"2024-01-27T01:07:27.742779+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:28.493839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.958400+0000","last_change":"2024-01-27T01:07:26.956417+0000","last_active":"2024-01-27T01:07:26.958400+0000","last_peered":"2024-01-27T01:07:26.958400+0000","last_clean":"2024-01-27T01:07:26.958400+0000","last_became_active":"2024-01-27T01:07:26.956128+0000","last_became_peered":"2024-01-27T01:07:26.956128+0000","last_unstale":"2024-01-27T01:07:26.958400+0000","last_undegraded":"2024-01-27T01:07:26.958400+0000","last_fullsized":"2024-01-27T01:07:26.958400+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:22.165694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.752330+0000","last_change":"2024-01-27T01:07:27.752330+0000","last_active":"2024-01-27T01:07:27.752330+0000","last_peered":"2024-01-27T01:07:27.752330+0000","last_clean":"2024-01-27T01:07:27.752330+0000","last_became_active":"2024-01-27T01:07:27.752040+0000","last_became_peered":"2024-01-27T01:07:27.752040+0000","last_unstale":"2024-01-27T01:07:27.752330+0000","last_undegraded":"2024-01-27T01:07:27.752330+0000","last_fullsized":"2024-01-27T01:07:27.752330+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:36:14.454947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"242'2","reported_seq":11,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.944262+0000","last_change":"2024-01-27T01:07:26.947483+0000","last_active":"2024-01-27T01:07:27.944262+0000","last_peered":"2024-01-27T01:07:27.944262+0000","last_clean":"2024-01-27T01:07:27.944262+0000","last_became_active":"2024-01-27T01:07:26.947172+0000","last_became_peered":"2024-01-27T01:07:26.947172+0000","last_unstale":"2024-01-27T01:07:27.944262+0000","last_undegraded":"2024-01-27T01:07:27.944262+0000","last_fullsized":"2024-01-27T01:07:27.944262+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:03.604469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.757740+0000","last_change":"2024-01-27T01:07:27.755356+0000","last_active":"2024-01-27T01:07:27.757740+0000","last_peered":"2024-01-27T01:07:27.757740+0000","last_clean":"2024-01-27T01:07:27.757740+0000","last_became_active":"2024-01-27T01:07:27.755108+0000","last_became_peered":"2024-01-27T01:07:27.755108+0000","last_unstale":"2024-01-27T01:07:27.757740+0000","last_undegraded":"2024-01-27T01:07:27.757740+0000","last_fullsized":"2024-01-27T01:07:27.757740+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:51:14.988606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.953821+0000","last_change":"2024-01-27T01:07:26.950680+0000","last_active":"2024-01-27T01:07:26.953821+0000","last_peered":"2024-01-27T01:07:26.953821+0000","last_clean":"2024-01-27T01:07:26.953821+0000","last_became_active":"2024-01-27T01:07:26.950395+0000","last_became_peered":"2024-01-27T01:07:26.950395+0000","last_unstale":"2024-01-27T01:07:26.953821+0000","last_undegraded":"2024-01-27T01:07:26.953821+0000","last_fullsized":"2024-01-27T01:07:26.953821+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:28:11.238816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.748049+0000","last_change":"2024-01-27T01:07:27.748049+0000","last_active":"2024-01-27T01:07:27.748049+0000","last_peered":"2024-01-27T01:07:27.748049+0000","last_clean":"2024-01-27T01:07:27.748049+0000","last_became_active":"2024-01-27T01:07:27.747765+0000","last_became_peered":"2024-01-27T01:07:27.747765+0000","last_unstale":"2024-01-27T01:07:27.748049+0000","last_undegraded":"2024-01-27T01:07:27.748049+0000","last_fullsized":"2024-01-27T01:07:27.748049+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:51:33.521276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.953036+0000","last_change":"2024-01-27T01:07:26.950170+0000","last_active":"2024-01-27T01:07:26.953036+0000","last_peered":"2024-01-27T01:07:26.953036+0000","last_clean":"2024-01-27T01:07:26.953036+0000","last_became_active":"2024-01-27T01:07:26.949857+0000","last_became_peered":"2024-01-27T01:07:26.949857+0000","last_unstale":"2024-01-27T01:07:26.953036+0000","last_undegraded":"2024-01-27T01:07:26.953036+0000","last_fullsized":"2024-01-27T01:07:26.953036+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:29.510893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.753040+0000","last_change":"2024-01-27T01:07:27.753040+0000","last_active":"2024-01-27T01:07:27.753040+0000","last_peered":"2024-01-27T01:07:27.753040+0000","last_clean":"2024-01-27T01:07:27.753040+0000","last_became_active":"2024-01-27T01:07:27.752784+0000","last_became_peered":"2024-01-27T01:07:27.752784+0000","last_unstale":"2024-01-27T01:07:27.753040+0000","last_undegraded":"2024-01-27T01:07:27.753040+0000","last_fullsized":"2024-01-27T01:07:27.753040+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:09:57.375468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":10,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.734272+0000","last_change":"2024-01-27T01:07:26.944368+0000","last_active":"2024-01-27T01:07:27.734272+0000","last_peered":"2024-01-27T01:07:27.734272+0000","last_clean":"2024-01-27T01:07:27.734272+0000","last_became_active":"2024-01-27T01:07:26.944076+0000","last_became_peered":"2024-01-27T01:07:26.944076+0000","last_unstale":"2024-01-27T01:07:27.734272+0000","last_undegraded":"2024-01-27T01:07:27.734272+0000","last_fullsized":"2024-01-27T01:07:27.734272+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:59:38.148215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.747646+0000","last_change":"2024-01-27T01:07:27.747646+0000","last_active":"2024-01-27T01:07:27.747646+0000","last_peered":"2024-01-27T01:07:27.747646+0000","last_clean":"2024-01-27T01:07:27.747646+0000","last_became_active":"2024-01-27T01:07:27.747345+0000","last_became_peered":"2024-01-27T01:07:27.747345+0000","last_unstale":"2024-01-27T01:07:27.747646+0000","last_undegraded":"2024-01-27T01:07:27.747646+0000","last_fullsized":"2024-01-27T01:07:27.747646+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:30:45.126288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"242'2","reported_seq":11,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.944310+0000","last_change":"2024-01-27T01:07:26.950235+0000","last_active":"2024-01-27T01:07:27.944310+0000","last_peered":"2024-01-27T01:07:27.944310+0000","last_clean":"2024-01-27T01:07:27.944310+0000","last_became_active":"2024-01-27T01:07:26.949923+0000","last_became_peered":"2024-01-27T01:07:26.949923+0000","last_unstale":"2024-01-27T01:07:27.944310+0000","last_undegraded":"2024-01-27T01:07:27.944310+0000","last_fullsized":"2024-01-27T01:07:27.944310+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:27:55.458182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.758370+0000","last_change":"2024-01-27T01:07:27.758370+0000","last_active":"2024-01-27T01:07:27.758370+0000","last_peered":"2024-01-27T01:07:27.758370+0000","last_clean":"2024-01-27T01:07:27.758370+0000","last_became_active":"2024-01-27T01:07:27.758007+0000","last_became_peered":"2024-01-27T01:07:27.758007+0000","last_unstale":"2024-01-27T01:07:27.758370+0000","last_undegraded":"2024-01-27T01:07:27.758370+0000","last_fullsized":"2024-01-27T01:07:27.758370+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:07.518631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.955634+0000","last_change":"2024-01-27T01:07:26.953360+0000","last_active":"2024-01-27T01:07:26.955634+0000","last_peered":"2024-01-27T01:07:26.955634+0000","last_clean":"2024-01-27T01:07:26.955634+0000","last_became_active":"2024-01-27T01:07:26.953090+0000","last_became_peered":"2024-01-27T01:07:26.953090+0000","last_unstale":"2024-01-27T01:07:26.955634+0000","last_undegraded":"2024-01-27T01:07:26.955634+0000","last_fullsized":"2024-01-27T01:07:26.955634+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:10.549778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.750264+0000","last_change":"2024-01-27T01:07:27.750264+0000","last_active":"2024-01-27T01:07:27.750264+0000","last_peered":"2024-01-27T01:07:27.750264+0000","last_clean":"2024-01-27T01:07:27.750264+0000","last_became_active":"2024-01-27T01:07:27.749987+0000","last_became_peered":"2024-01-27T01:07:27.749987+0000","last_unstale":"2024-01-27T01:07:27.750264+0000","last_undegraded":"2024-01-27T01:07:27.750264+0000","last_fullsized":"2024-01-27T01:07:27.750264+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:19.339556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":10,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.734444+0000","last_change":"2024-01-27T01:07:26.942700+0000","last_active":"2024-01-27T01:07:27.734444+0000","last_peered":"2024-01-27T01:07:27.734444+0000","last_clean":"2024-01-27T01:07:27.734444+0000","last_became_active":"2024-01-27T01:07:26.942491+0000","last_became_peered":"2024-01-27T01:07:26.942491+0000","last_unstale":"2024-01-27T01:07:27.734444+0000","last_undegraded":"2024-01-27T01:07:27.734444+0000","last_fullsized":"2024-01-27T01:07:27.734444+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:03:40.104583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.746804+0000","last_change":"2024-01-27T01:07:27.746804+0000","last_active":"2024-01-27T01:07:27.746804+0000","last_peered":"2024-01-27T01:07:27.746804+0000","last_clean":"2024-01-27T01:07:27.746804+0000","last_became_active":"2024-01-27T01:07:27.746110+0000","last_became_peered":"2024-01-27T01:07:27.746110+0000","last_unstale":"2024-01-27T01:07:27.746804+0000","last_undegraded":"2024-01-27T01:07:27.746804+0000","last_fullsized":"2024-01-27T01:07:27.746804+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:48:18.465014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":10,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.734759+0000","last_change":"2024-01-27T01:07:26.943462+0000","last_active":"2024-01-27T01:07:27.734759+0000","last_peered":"2024-01-27T01:07:27.734759+0000","last_clean":"2024-01-27T01:07:27.734759+0000","last_became_active":"2024-01-27T01:07:26.943193+0000","last_became_peered":"2024-01-27T01:07:26.943193+0000","last_unstale":"2024-01-27T01:07:27.734759+0000","last_undegraded":"2024-01-27T01:07:27.734759+0000","last_fullsized":"2024-01-27T01:07:27.734759+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:21:20.952966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.743744+0000","last_change":"2024-01-27T01:07:27.739197+0000","last_active":"2024-01-27T01:07:27.743744+0000","last_peered":"2024-01-27T01:07:27.743744+0000","last_clean":"2024-01-27T01:07:27.743744+0000","last_became_active":"2024-01-27T01:07:27.738911+0000","last_became_peered":"2024-01-27T01:07:27.738911+0000","last_unstale":"2024-01-27T01:07:27.743744+0000","last_undegraded":"2024-01-27T01:07:27.743744+0000","last_fullsized":"2024-01-27T01:07:27.743744+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:41:00.947931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.952474+0000","last_change":"2024-01-27T01:07:26.949530+0000","last_active":"2024-01-27T01:07:26.952474+0000","last_peered":"2024-01-27T01:07:26.952474+0000","last_clean":"2024-01-27T01:07:26.952474+0000","last_became_active":"2024-01-27T01:07:26.949204+0000","last_became_peered":"2024-01-27T01:07:26.949204+0000","last_unstale":"2024-01-27T01:07:26.952474+0000","last_undegraded":"2024-01-27T01:07:26.952474+0000","last_fullsized":"2024-01-27T01:07:26.952474+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:16:54.275653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.751936+0000","last_change":"2024-01-27T01:07:27.744772+0000","last_active":"2024-01-27T01:07:27.751936+0000","last_peered":"2024-01-27T01:07:27.751936+0000","last_clean":"2024-01-27T01:07:27.751936+0000","last_became_active":"2024-01-27T01:07:27.744416+0000","last_became_peered":"2024-01-27T01:07:27.744416+0000","last_unstale":"2024-01-27T01:07:27.751936+0000","last_undegraded":"2024-01-27T01:07:27.751936+0000","last_fullsized":"2024-01-27T01:07:27.751936+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:59:39.155363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.955576+0000","last_change":"2024-01-27T01:07:26.947692+0000","last_active":"2024-01-27T01:07:26.955576+0000","last_peered":"2024-01-27T01:07:26.955576+0000","last_clean":"2024-01-27T01:07:26.955576+0000","last_became_active":"2024-01-27T01:07:26.947404+0000","last_became_peered":"2024-01-27T01:07:26.947404+0000","last_unstale":"2024-01-27T01:07:26.955576+0000","last_undegraded":"2024-01-27T01:07:26.955576+0000","last_fullsized":"2024-01-27T01:07:26.955576+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:12:55.245326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.756805+0000","last_change":"2024-01-27T01:07:27.749874+0000","last_active":"2024-01-27T01:07:27.756805+0000","last_peered":"2024-01-27T01:07:27.756805+0000","last_clean":"2024-01-27T01:07:27.756805+0000","last_became_active":"2024-01-27T01:07:27.749252+0000","last_became_peered":"2024-01-27T01:07:27.749252+0000","last_unstale":"2024-01-27T01:07:27.756805+0000","last_undegraded":"2024-01-27T01:07:27.756805+0000","last_fullsized":"2024-01-27T01:07:27.756805+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:18:53.285178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.949118+0000","last_change":"2024-01-27T01:07:26.947606+0000","last_active":"2024-01-27T01:07:26.949118+0000","last_peered":"2024-01-27T01:07:26.949118+0000","last_clean":"2024-01-27T01:07:26.949118+0000","last_became_active":"2024-01-27T01:07:26.947287+0000","last_became_peered":"2024-01-27T01:07:26.947287+0000","last_unstale":"2024-01-27T01:07:26.949118+0000","last_undegraded":"2024-01-27T01:07:26.949118+0000","last_fullsized":"2024-01-27T01:07:26.949118+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:34:14.649324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":8,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.749856+0000","last_change":"2024-01-27T01:07:27.749856+0000","last_active":"2024-01-27T01:07:27.749856+0000","last_peered":"2024-01-27T01:07:27.749856+0000","last_clean":"2024-01-27T01:07:27.749856+0000","last_became_active":"2024-01-27T01:07:27.749595+0000","last_became_peered":"2024-01-27T01:07:27.749595+0000","last_unstale":"2024-01-27T01:07:27.749856+0000","last_undegraded":"2024-01-27T01:07:27.749856+0000","last_fullsized":"2024-01-27T01:07:27.749856+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:25:38.146534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":9,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.953694+0000","last_change":"2024-01-27T01:07:26.950737+0000","last_active":"2024-01-27T01:07:26.953694+0000","last_peered":"2024-01-27T01:07:26.953694+0000","last_clean":"2024-01-27T01:07:26.953694+0000","last_became_active":"2024-01-27T01:07:26.950498+0000","last_became_peered":"2024-01-27T01:07:26.950498+0000","last_unstale":"2024-01-27T01:07:26.953694+0000","last_undegraded":"2024-01-27T01:07:26.953694+0000","last_fullsized":"2024-01-27T01:07:26.953694+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:24:42.558412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":9,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.748875+0000","last_change":"2024-01-27T01:07:27.746007+0000","last_active":"2024-01-27T01:07:27.748875+0000","last_peered":"2024-01-27T01:07:27.748875+0000","last_clean":"2024-01-27T01:07:27.748875+0000","last_became_active":"2024-01-27T01:07:27.745722+0000","last_became_peered":"2024-01-27T01:07:27.745722+0000","last_unstale":"2024-01-27T01:07:27.748875+0000","last_undegraded":"2024-01-27T01:07:27.748875+0000","last_fullsized":"2024-01-27T01:07:27.748875+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:09:23.112301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"242'1","reported_seq":10,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-01-27T01:07:27.736684+0000","last_change":"2024-01-27T01:07:26.951812+0000","last_active":"2024-01-27T01:07:27.736684+0000","last_peered":"2024-01-27T01:07:27.736684+0000","last_clean":"2024-01-27T01:07:27.736684+0000","last_became_active":"2024-01-27T01:07:26.951141+0000","last_became_peered":"2024-01-27T01:07:26.951141+0000","last_unstale":"2024-01-27T01:07:27.736684+0000","last_undegraded":"2024-01-27T01:07:27.736684+0000","last_fullsized":"2024-01-27T01:07:27.736684+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:06:20.506111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"230'8772","reported_seq":9758,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-01-27T01:07:26.952559+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:07:26.952559+0000","last_peered":"2024-01-27T01:07:26.952559+0000","last_clean":"2024-01-27T01:07:26.952559+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:07:26.952559+0000","last_undegraded":"2024-01-27T01:07:26.952559+0000","last_fullsized":"2024-01-27T01:07:26.952559+0000","mapping_epoch":14,"log_start":"98'2229","ondisk_log_start":"98'2229","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2456,"num_read_kb":5220,"num_write":670,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2456,"num_read_kb":5220,"num_write":670,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6543,"ondisk_log_size":6543,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576012,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300856,"kb_used_data":3568,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94070984,"statfs":{"total":96636764160,"available":96328687616,"internally_reserved":0,"allocated":3653632,"data_stored":2205830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.312},"max":{"1min":0.72499999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.63200000000000001,"15min":0.621},"min":{"1min":0.48099999999999998,"5min":0.39900000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":0.879,"15min":0.879},"last":0.76400000000000001}]},{"osd":1,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70299999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.63800000000000001,"15min":0.63100000000000001},"min":{"1min":0.44500000000000001,"5min":0.42099999999999999,"15min":0.34300000000000003},"max":{"1min":0.73499999999999999,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.81499999999999995}]},{"osd":2,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.61899999999999999,"15min":0.60999999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.72899999999999998,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.628,"15min":0.629},"min":{"1min":0.372,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.69499999999999995,"5min":0.872,"15min":0.88900000000000001},"last":0.77700000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.60899999999999999,"15min":0.61399999999999999},"min":{"1min":0.38500000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69099999999999995,"5min":0.876,"15min":0.876},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.65700000000000003,"15min":0.64200000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80700000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.753}]},{"osd":4,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.503,"15min":0.48799999999999999},"min":{"1min":0.35199999999999998,"5min":0.314,"15min":0.29999999999999999},"max":{"1min":0.65500000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53700000000000003,"15min":0.51600000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.629,"5min":0.80300000000000005,"15min":0.86599999999999999},"last":0.64200000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.55300000000000005,"15min":0.52500000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.80400000000000005},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57299999999999995,"15min":0.56399999999999995},"min":{"1min":0.41699999999999998,"5min":0.34799999999999998,"15min":0.34399999999999997},"max":{"1min":0.79200000000000004,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.68300000000000005}]},{"osd":6,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60499999999999998,"15min":0.58899999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.60499999999999998,"15min":0.58699999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.75,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.69999999999999996}]}]},{"osd":6,"up_from":13,"seq":55834576012,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308468,"kb_used_data":2988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063372,"statfs":{"total":96636764160,"available":96320892928,"internally_reserved":0,"allocated":3059712,"data_stored":1615450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61099999999999999,"15min":0.60799999999999998},"min":{"1min":0.41999999999999998,"5min":0.40799999999999997,"15min":0.36399999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.441,"5min":0.432,"15min":0.376},"max":{"1min":0.82499999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.58799999999999997}]},{"osd":1,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79000000000000004,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.40799999999999997,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.81200000000000006}]},{"osd":2,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.504},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.39700000000000002,"15min":0.377},"max":{"1min":0.80400000000000005,"5min":0.878,"15min":0.878},"last":0.54300000000000004}]},{"osd":3,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.622,"15min":0.627},"min":{"1min":0.5,"5min":0.378,"15min":0.378},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.90100000000000002},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66300000000000003,"15min":0.66000000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.754}]},{"osd":4,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.36299999999999999,"5min":0.33300000000000002,"15min":0.32500000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.86599999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33900000000000002,"5min":0.33800000000000002,"15min":0.29799999999999999},"max":{"1min":0.77300000000000002,"5min":0.78500000000000003,"15min":0.88900000000000001},"last":0.66800000000000004}]},{"osd":5,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.31900000000000001},"max":{"1min":0.68999999999999995,"5min":0.75900000000000001,"15min":0.80500000000000005},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.42199999999999999,"5min":0.37,"15min":0.28899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.874},"last":0.84199999999999997}]},{"osd":7,"last update":"Sat Jan 27 01:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45800000000000002,"5min":0.39000000000000001,"15min":0.35799999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.92900000000000005},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.44500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78900000000000003,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.73599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834576012,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300268,"kb_used_data":2980,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071572,"statfs":{"total":96636764160,"available":96329289728,"internally_reserved":0,"allocated":3051520,"data_stored":1615338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.28499999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.48899999999999999,"5min":0.66100000000000003,"15min":0.69599999999999995},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.625,"5min":0.67800000000000005,"15min":0.76600000000000001},"last":0.34999999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.42699999999999999,"15min":0.435},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.26200000000000001},"max":{"1min":0.54900000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.309},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":0.48799999999999999},"min":{"1min":0.36699999999999999,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.58899999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.40899999999999997}]},{"osd":3,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51800000000000002,"15min":0.53100000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66200000000000003,"5min":0.84799999999999998,"15min":1.2130000000000001},"last":0.377},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.5,"15min":0.51300000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.32300000000000001},"max":{"1min":0.65100000000000002,"5min":1.125,"15min":1.22},"last":0.39000000000000001}]},{"osd":4,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.66900000000000004,"15min":0.67000000000000004},"min":{"1min":0.41599999999999998,"5min":0.41099999999999998,"15min":0.39800000000000002},"max":{"1min":0.92100000000000004,"5min":1.242,"15min":1.242},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.68500000000000005,"15min":0.69199999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.35699999999999998},"max":{"1min":0.80500000000000005,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.59699999999999998}]},{"osd":5,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.72099999999999997,"15min":0.70499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1,"5min":5.444,"15min":5.444},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71399999999999997,"15min":0.70599999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.435},"max":{"1min":0.97599999999999998,"5min":2.6669999999999998,"15min":2.6669999999999998},"last":0.51300000000000001}]},{"osd":6,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.73599999999999999,"15min":0.72099999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.47099999999999997},"max":{"1min":0.89700000000000002,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.499},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.80600000000000005,"15min":0.74199999999999999},"min":{"1min":0.52400000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0600000000000001,"5min":11.223000000000001,"15min":11.223000000000001},"last":0.61599999999999999}]},{"osd":7,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.749,"15min":0.753},"min":{"1min":0.54100000000000004,"5min":0.52800000000000002,"15min":0.48999999999999999},"max":{"1min":1.0349999999999999,"5min":2.9660000000000002,"15min":2.9660000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.76900000000000002,"15min":0.76700000000000002},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.52000000000000002},"max":{"1min":0.95599999999999996,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.58799999999999997}]}]},{"osd":0,"up_from":13,"seq":55834576011,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300860,"kb_used_data":3572,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94070980,"statfs":{"total":96636764160,"available":96328683520,"internally_reserved":0,"allocated":3657728,"data_stored":2207242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42899999999999999,"15min":0.42199999999999999},"min":{"1min":0.30299999999999999,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":0.55600000000000005,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.436,"5min":0.45600000000000002,"15min":0.46100000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":0.56499999999999995,"5min":0.72699999999999998,"15min":0.73399999999999999},"last":0.64300000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.29699999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.58299999999999996,"5min":0.66900000000000004,"15min":0.76800000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48399999999999999,"15min":0.48899999999999999},"min":{"1min":0.38600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64600000000000002,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.502}]},{"osd":3,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.66100000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.504},"min":{"1min":0.36699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64100000000000001,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.55300000000000005}]},{"osd":4,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.68300000000000005,"15min":0.67700000000000005},"min":{"1min":0.57799999999999996,"5min":0.47999999999999998,"15min":0.42199999999999999},"max":{"1min":0.86399999999999999,"5min":0.91500000000000004,"15min":1.0720000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.56999999999999995,"5min":0.47099999999999997,"15min":0.437},"max":{"1min":0.82699999999999996,"5min":1.002,"15min":1.002},"last":0.90100000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.68999999999999995,"15min":0.70099999999999996},"min":{"1min":0.58199999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.876,"5min":1.0209999999999999,"15min":2.165},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70599999999999996,"15min":0.71199999999999997},"min":{"1min":0.498,"5min":0.48799999999999999,"15min":0.45600000000000002},"max":{"1min":0.89000000000000001,"5min":0.97399999999999998,"15min":1.1399999999999999},"last":0.92300000000000004}]},{"osd":6,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69299999999999995,"15min":0.71099999999999997},"min":{"1min":0.56499999999999995,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.84599999999999997,"5min":0.98899999999999999,"15min":1.0860000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72599999999999998,"15min":0.74099999999999999},"min":{"1min":0.54300000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.93400000000000005,"5min":1.0109999999999999,"15min":1.425},"last":0.86499999999999999}]},{"osd":7,"last update":"Sat Jan 27 01:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.74399999999999999,"15min":0.75700000000000001},"min":{"1min":0.61299999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.98899999999999999,"5min":1.044,"15min":1.1259999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.76300000000000001,"15min":0.76900000000000002},"min":{"1min":0.58299999999999996,"5min":0.51100000000000001,"15min":0.48499999999999999},"max":{"1min":0.91700000000000004,"5min":1.0289999999999999,"15min":1.1910000000000001},"last":0.88700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576011,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312564,"kb_used_data":2988,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059276,"statfs":{"total":96636764160,"available":96316698624,"internally_reserved":0,"allocated":3059712,"data_stored":1615962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39600000000000002,"15min":0.40400000000000003},"min":{"1min":0.27200000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.497,"5min":0.746,"15min":0.746},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.441,"15min":0.443},"min":{"1min":0.34599999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.40699999999999997}]},{"osd":1,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44500000000000001,"15min":0.44600000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.50900000000000001,"15min":0.51900000000000002},"min":{"1min":0.40699999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.48299999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.49299999999999999,"15min":0.496},"min":{"1min":0.38700000000000001,"5min":0.34699999999999998,"15min":0.33200000000000002},"max":{"1min":0.65000000000000002,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.371},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.47799999999999998,"15min":0.48699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.63900000000000001,"5min":0.65500000000000003,"15min":0.75700000000000001},"last":0.439}]},{"osd":4,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.77800000000000002,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.68100000000000005,"15min":0.69199999999999995},"min":{"1min":0.47899999999999998,"5min":0.47099999999999997,"15min":0.378},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.107},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.69699999999999995,"15min":0.70199999999999996},"min":{"1min":0.47399999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.075,"5min":1.075,"15min":1.167},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.69199999999999995,"15min":0.70499999999999996},"min":{"1min":0.52300000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81899999999999995,"5min":0.97499999999999998,"15min":1.05},"last":0.63900000000000001}]},{"osd":6,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.70099999999999996,"15min":0.70899999999999996},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.90200000000000002,"5min":0.97899999999999998,"15min":1.077},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.69599999999999995,"15min":0.71499999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.39200000000000002},"max":{"1min":0.85999999999999999,"5min":0.96699999999999997,"15min":1.135},"last":0.503}]},{"osd":7,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73399999999999999,"15min":0.74399999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.198},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.755,"15min":0.76500000000000001},"min":{"1min":0.53600000000000003,"5min":0.53100000000000003,"15min":0.48199999999999998},"max":{"1min":0.875,"5min":0.96399999999999997,"15min":1.0880000000000001},"last":0.67700000000000005}]}]},{"osd":3,"up_from":13,"seq":55834576010,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304388,"kb_used_data":3004,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067452,"statfs":{"total":96636764160,"available":96325070848,"internally_reserved":0,"allocated":3076096,"data_stored":1617484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44700000000000001,"15min":0.42399999999999999},"min":{"1min":0.309,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.59499999999999997,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.44,"15min":0.45500000000000002},"min":{"1min":0.32400000000000001,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.64000000000000001,"5min":0.66700000000000004,"15min":0.84699999999999998},"last":0.30599999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.624,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66400000000000003,"5min":15.391,"15min":15.391},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.53500000000000003,"15min":0.52400000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70499999999999996,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.432}]},{"osd":2,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.54300000000000004,"15min":0.51600000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.60299999999999998,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53900000000000003,"15min":0.51000000000000001},"min":{"1min":0.374,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.5529999999999999,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":0.35799999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.745,"15min":0.71899999999999997},"min":{"1min":0.56399999999999995,"5min":0.47699999999999998,"15min":0.439},"max":{"1min":0.82699999999999996,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.70899999999999996},"min":{"1min":0.52000000000000002,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":1.0089999999999999,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.63}]},{"osd":5,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.745,"15min":0.71399999999999997},"min":{"1min":0.52000000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.036,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73899999999999999,"15min":0.72299999999999998},"min":{"1min":0.54900000000000004,"5min":0.435,"15min":0.435},"max":{"1min":1.127,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.54200000000000004}]},{"osd":6,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.746,"15min":0.72199999999999998},"min":{"1min":0.50700000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.92100000000000004,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.753,"15min":0.74099999999999999},"min":{"1min":0.54900000000000004,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.077,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.59199999999999997}]},{"osd":7,"last update":"Sat Jan 27 01:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.755,"15min":0.73899999999999999},"min":{"1min":0.53300000000000003,"5min":0.52500000000000002,"15min":0.46700000000000003},"max":{"1min":1.103,"5min":4.569,"15min":4.569},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.80600000000000005,"15min":0.78300000000000003},"min":{"1min":0.59199999999999997,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.98999999999999999,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.61199999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576012,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316656,"kb_used_data":2984,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055184,"statfs":{"total":96636764160,"available":96312508416,"internally_reserved":0,"allocated":3055616,"data_stored":1615408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.436,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60199999999999998,"15min":0.61299999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.753,"5min":0.82199999999999995,"15min":0.85699999999999998},"last":0.59299999999999997}]},{"osd":1,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60999999999999999,"15min":0.628},"min":{"1min":0.40699999999999997,"5min":0.39100000000000001,"15min":0.38800000000000001},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60999999999999999,"15min":0.623},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82299999999999995,"5min":0.874,"15min":0.874},"last":0.42399999999999999}]},{"osd":2,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.84699999999999998,"5min":0.85599999999999998,"15min":0.88700000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.48999999999999999,"5min":0.42899999999999999,"15min":0.42799999999999999},"max":{"1min":0.79200000000000004,"5min":0.81499999999999995,"15min":0.89200000000000002},"last":0.39600000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60399999999999998,"15min":0.623},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.80800000000000005,"5min":0.83299999999999996,"15min":0.84199999999999997},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.621,"15min":0.629},"min":{"1min":0.42299999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.40999999999999998}]},{"osd":5,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47999999999999998,"15min":0.48399999999999999},"min":{"1min":0.32400000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.85899999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.503,"15min":0.505},"min":{"1min":0.36299999999999999,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.878},"last":0.61399999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.50600000000000001,"15min":0.499},"min":{"1min":0.38300000000000001,"5min":0.35799999999999998,"15min":0.34999999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.88700000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.375,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.502}]},{"osd":7,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56200000000000006,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.748,"5min":0.80400000000000005,"15min":0.90200000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.38600000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.92600000000000005},"last":0.56799999999999995}]}]},{"osd":5,"up_from":13,"seq":55834576012,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312556,"kb_used_data":2980,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059284,"statfs":{"total":96636764160,"available":96316706816,"internally_reserved":0,"allocated":3051520,"data_stored":1615338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59399999999999997,"15min":0.60299999999999998},"min":{"1min":0.42999999999999999,"5min":0.40500000000000003,"15min":0.27300000000000002},"max":{"1min":0.80700000000000005,"5min":0.82799999999999996,"15min":1.006},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59399999999999997,"15min":0.60699999999999998},"min":{"1min":0.36599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":1.1719999999999999},"last":0.55100000000000005}]},{"osd":1,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63700000000000001,"15min":0.65200000000000002},"min":{"1min":0.39100000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.95899999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.52100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":1.212},"last":0.53000000000000003}]},{"osd":2,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57499999999999996,"15min":0.59399999999999997},"min":{"1min":0.42299999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":1.631},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.625,"15min":0.621},"min":{"1min":0.497,"5min":0.38800000000000001,"15min":0.34000000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.40100000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.59999999999999998},"min":{"1min":0.39600000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83299999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.504},{"interface":"front","average":{"1min":0.62,"5min":0.625,"15min":0.63},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.879,"5min":0.879,"15min":1.4690000000000001},"last":0.46600000000000003}]},{"osd":4,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.505,"15min":0.48799999999999999},"min":{"1min":0.34899999999999998,"5min":0.32200000000000001,"15min":0.317},"max":{"1min":0.76700000000000002,"5min":0.90200000000000002,"15min":0.94399999999999995},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53500000000000003,"15min":0.51500000000000001},"min":{"1min":0.439,"5min":0.35199999999999998,"15min":0.33900000000000002},"max":{"1min":0.80800000000000005,"5min":0.92500000000000004,"15min":0.93300000000000005},"last":0.42899999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51800000000000002,"15min":0.504},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79400000000000004,"5min":0.96799999999999997,"15min":1.732},"last":0.38},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56299999999999994,"15min":0.54500000000000004},"min":{"1min":0.42899999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.753,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.49299999999999999}]},{"osd":7,"last update":"Sat Jan 27 01:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.46200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80400000000000005,"5min":0.95399999999999996,"15min":1.0740000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.626,"15min":0.61299999999999999},"min":{"1min":0.504,"5min":0.40799999999999997,"15min":0.39100000000000001},"max":{"1min":0.85199999999999998,"5min":0.996,"15min":1.6919999999999999},"last":0.51400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:07:32.545 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-27T01:07:32.545 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-27T01:07:32.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.720+0000 7ff965a11700 1 Processor -- start 2024-01-27T01:07:32.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.720+0000 7ff965a11700 1 -- start start 2024-01-27T01:07:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.721+0000 7ff965a11700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9600ccc50 0x7ff9600ca350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.721+0000 7ff965a11700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9600c8f70 con 0x7ff9601599c0 2024-01-27T01:07:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.721+0000 7ff965a11700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9600cd0c0 con 0x7ff9600ca890 2024-01-27T01:07:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.721+0000 7ff965a11700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9600cad10 con 0x7ff9600ccc50 2024-01-27T01:07:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.721+0000 7ff95effd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff9601599c0 0x7ff960068c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45536/0 (socket says 172.21.15.46:45536) 2024-01-27T01:07:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.721+0000 7ff95effd700 1 -- 172.21.15.46:0/4251687146 learned_addr learned my addr 172.21.15.46:0/4251687146 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.721+0000 7ff95e7fc700 1 --2- 172.21.15.46:0/4251687146 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9600ccc50 0x7ff9600ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.721+0000 7ff95dffb700 1 -- 172.21.15.46:0/4251687146 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472844762 0 0) 0x7ff9600c8f70 con 0x7ff9601599c0 2024-01-27T01:07:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.721+0000 7ff95dffb700 1 -- 172.21.15.46:0/4251687146 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff948003570 con 0x7ff9601599c0 2024-01-27T01:07:32.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.721+0000 7ff95dffb700 1 -- 172.21.15.46:0/4251687146 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778649859 0 0) 0x7ff9600cd0c0 con 0x7ff9600ca890 2024-01-27T01:07:32.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.721+0000 7ff95dffb700 1 -- 172.21.15.46:0/4251687146 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9600c8f70 con 0x7ff9600ca890 2024-01-27T01:07:32.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.722+0000 7ff95e7fc700 1 -- 172.21.15.46:0/4251687146 >> v1:172.21.15.106:6789/0 conn(0x7ff9600ca890 legacy=0x7ff960159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:32.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.722+0000 7ff95e7fc700 1 -- 172.21.15.46:0/4251687146 >> v1:172.21.15.46:6789/0 conn(0x7ff9601599c0 legacy=0x7ff960068c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:32.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.722+0000 7ff95e7fc700 1 -- 172.21.15.46:0/4251687146 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff96015a9c0 con 0x7ff9600ccc50 2024-01-27T01:07:32.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.722+0000 7ff95e7fc700 1 --2- 172.21.15.46:0/4251687146 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9600ccc50 0x7ff9600ca350 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff950009910 tx=0x7ff9500091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e51571c8317768c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:32.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.722+0000 7ff95dffb700 1 -- 172.21.15.46:0/4251687146 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff950011630 con 0x7ff9600ccc50 2024-01-27T01:07:32.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.722+0000 7ff95dffb700 1 -- 172.21.15.46:0/4251687146 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff950011790 con 0x7ff9600ccc50 2024-01-27T01:07:32.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.722+0000 7ff95dffb700 1 -- 172.21.15.46:0/4251687146 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff950011b10 con 0x7ff9600ccc50 2024-01-27T01:07:32.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.723+0000 7ff965a11700 1 -- 172.21.15.46:0/4251687146 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9600ccc50 msgr2=0x7ff9600ca350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:32.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.723+0000 7ff965a11700 1 --2- 172.21.15.46:0/4251687146 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9600ccc50 0x7ff9600ca350 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff950009910 tx=0x7ff9500091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.723+0000 7ff965a11700 1 -- 172.21.15.46:0/4251687146 shutdown_connections 2024-01-27T01:07:32.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.723+0000 7ff965a11700 1 --2- 172.21.15.46:0/4251687146 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9600ccc50 0x7ff9600ca350 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.723+0000 7ff965a11700 1 -- 172.21.15.46:0/4251687146 >> 172.21.15.46:0/4251687146 conn(0x7ff9600ce570 msgr2=0x7ff9600ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:32.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.723+0000 7ff965a11700 1 -- 172.21.15.46:0/4251687146 shutdown_connections 2024-01-27T01:07:32.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.723+0000 7ff965a11700 1 -- 172.21.15.46:0/4251687146 wait complete. 2024-01-27T01:07:32.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.724+0000 7ff965a11700 1 Processor -- start 2024-01-27T01:07:32.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.724+0000 7ff965a11700 1 -- start start 2024-01-27T01:07:32.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.725+0000 7ff965a11700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9600ca890 0x7ff960156450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:32.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.725+0000 7ff965a11700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9600ccc50 0x7ff9601581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:32.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.725+0000 7ff965a11700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9601599c0 0x7ff960158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:32.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.725+0000 7ff965a11700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9600c8770 con 0x7ff9600ca890 2024-01-27T01:07:32.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.725+0000 7ff965a11700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96016b0c0 con 0x7ff9601599c0 2024-01-27T01:07:32.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.725+0000 7ff965a11700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff96016aab0 con 0x7ff9600ccc50 2024-01-27T01:07:32.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.725+0000 7ff95effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9600ca890 0x7ff960156450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:32.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.725+0000 7ff95effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9600ca890 0x7ff960156450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45836/0 (socket says 172.21.15.46:45836) 2024-01-27T01:07:32.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.725+0000 7ff95effd700 1 -- 172.21.15.46:0/1725266877 learned_addr learned my addr 172.21.15.46:0/1725266877 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:32.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.725+0000 7ff95e7fc700 1 --2- 172.21.15.46:0/1725266877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9600ccc50 0x7ff9601581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:32.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.725+0000 7ff95f7fe700 1 --2- 172.21.15.46:0/1725266877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9601599c0 0x7ff960158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:32.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.725+0000 7ff95effd700 1 -- 172.21.15.46:0/1725266877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9600ccc50 msgr2=0x7ff9601581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:32.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.725+0000 7ff95effd700 1 --2- 172.21.15.46:0/1725266877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9600ccc50 0x7ff9601581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.725+0000 7ff95effd700 1 -- 172.21.15.46:0/1725266877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9601599c0 msgr2=0x7ff960158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:32.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.725+0000 7ff95effd700 1 --2- 172.21.15.46:0/1725266877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9601599c0 0x7ff960158710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:32.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.725+0000 7ff95effd700 1 -- 172.21.15.46:0/1725266877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff950009420 con 0x7ff9600ca890 2024-01-27T01:07:32.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.726+0000 7ff95effd700 1 --2- 172.21.15.46:0/1725266877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9600ca890 0x7ff960156450 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7ff95400c790 tx=0x7ff954007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:32.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.726+0000 7ff946ffd700 1 -- 172.21.15.46:0/1725266877 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff954005b30 con 0x7ff9600ca890 2024-01-27T01:07:32.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.726+0000 7ff946ffd700 1 -- 172.21.15.46:0/1725266877 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff954005c90 con 0x7ff9600ca890 2024-01-27T01:07:32.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.726+0000 7ff965a11700 1 -- 172.21.15.46:0/1725266877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff960156990 con 0x7ff9600ca890 2024-01-27T01:07:32.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.726+0000 7ff946ffd700 1 -- 172.21.15.46:0/1725266877 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff954007300 con 0x7ff9600ca890 2024-01-27T01:07:32.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.726+0000 7ff965a11700 1 -- 172.21.15.46:0/1725266877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff96016ad50 con 0x7ff9600ca890 2024-01-27T01:07:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.727+0000 7ff965a11700 1 -- 172.21.15.46:0/1725266877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff960070270 con 0x7ff9600ca890 2024-01-27T01:07:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.728+0000 7ff946ffd700 1 -- 172.21.15.46:0/1725266877 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7ff95401b3b0 con 0x7ff9600ca890 2024-01-27T01:07:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.729+0000 7ff946ffd700 1 --2- 172.21.15.46:0/1725266877 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff94c067170 0x7ff94c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.729+0000 7ff95e7fc700 1 --2- 172.21.15.46:0/1725266877 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff94c067170 0x7ff94c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:32.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.729+0000 7ff946ffd700 1 -- 172.21.15.46:0/1725266877 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7ff9540550c0 con 0x7ff9600ca890 2024-01-27T01:07:32.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.730+0000 7ff95e7fc700 1 --2- 172.21.15.46:0/1725266877 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff94c067170 0x7ff94c069620 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff950009860 tx=0x7ff950017000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:32.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:32.734+0000 7ff946ffd700 1 -- 172.21.15.46:0/1725266877 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff954054790 con 0x7ff9600ca890 2024-01-27T01:07:33.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.020+0000 7ff965a11700 1 -- 172.21.15.46:0/1725266877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff96004c710 con 0x7ff9600ca890 2024-01-27T01:07:33.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.024+0000 7ff946ffd700 1 -- 172.21.15.46:0/1725266877 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v55) v1 ==== 190+0+0 (secure 0 0 0) 0x7ff954017020 con 0x7ff9600ca890 2024-01-27T01:07:33.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.027+0000 7ff965a11700 1 -- 172.21.15.46:0/1725266877 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff94c067170 msgr2=0x7ff94c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:33.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.027+0000 7ff965a11700 1 --2- 172.21.15.46:0/1725266877 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff94c067170 0x7ff94c069620 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff950009860 tx=0x7ff950017000 comp rx=0 tx=0).stop 2024-01-27T01:07:33.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.027+0000 7ff965a11700 1 -- 172.21.15.46:0/1725266877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9600ca890 msgr2=0x7ff960156450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:33.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.027+0000 7ff965a11700 1 --2- 172.21.15.46:0/1725266877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9600ca890 0x7ff960156450 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7ff95400c790 tx=0x7ff954007ea0 comp rx=0 tx=0).stop 2024-01-27T01:07:33.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.027+0000 7ff965a11700 1 -- 172.21.15.46:0/1725266877 shutdown_connections 2024-01-27T01:07:33.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.027+0000 7ff965a11700 1 --2- 172.21.15.46:0/1725266877 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff94c067170 0x7ff94c069620 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:33.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.027+0000 7ff965a11700 1 --2- 172.21.15.46:0/1725266877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9600ca890 0x7ff960156450 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:33.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.027+0000 7ff965a11700 1 --2- 172.21.15.46:0/1725266877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9601599c0 0x7ff960158710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:33.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.027+0000 7ff965a11700 1 --2- 172.21.15.46:0/1725266877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9600ccc50 0x7ff9601581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:33.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.027+0000 7ff965a11700 1 -- 172.21.15.46:0/1725266877 >> 172.21.15.46:0/1725266877 conn(0x7ff9600ce570 msgr2=0x7ff9600c9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:33.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.027+0000 7ff965a11700 1 -- 172.21.15.46:0/1725266877 shutdown_connections 2024-01-27T01:07:33.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.027+0000 7ff965a11700 1 -- 172.21.15.46:0/1725266877 wait complete. 2024-01-27T01:07:33.029 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-01-27T01:07:33.041 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:07:33.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.172+0000 7f0a292d5700 1 Processor -- start 2024-01-27T01:07:33.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.172+0000 7f0a292d5700 1 -- start start 2024-01-27T01:07:33.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.172+0000 7f0a292d5700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a241596c0 0x7f0a2406bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:33.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.173+0000 7f0a292d5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a2415d460 0x7f0a2406c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:33.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.173+0000 7f0a292d5700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2415d8f0 con 0x7f0a24159a90 2024-01-27T01:07:33.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.173+0000 7f0a292d5700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a240c98c0 con 0x7f0a241596c0 2024-01-27T01:07:33.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.173+0000 7f0a292d5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2415dbd0 con 0x7f0a2415d460 2024-01-27T01:07:33.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.173+0000 7f0a22ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a241596c0 0x7f0a2406bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:33.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.173+0000 7f0a22ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a241596c0 0x7f0a2406bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45844/0 (socket says 172.21.15.46:45844) 2024-01-27T01:07:33.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.173+0000 7f0a22ffd700 1 -- 172.21.15.46:0/1222665714 learned_addr learned my addr 172.21.15.46:0/1222665714 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:33.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.173+0000 7f0a227fc700 1 --2- 172.21.15.46:0/1222665714 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a2415d460 0x7f0a2406c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:33.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.173+0000 7f0a22ffd700 1 -- 172.21.15.46:0/1222665714 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a2415d460 msgr2=0x7f0a2406c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:33.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.173+0000 7f0a22ffd700 1 --2- 172.21.15.46:0/1222665714 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a2415d460 0x7f0a2406c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:33.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.173+0000 7f0a22ffd700 1 -- 172.21.15.46:0/1222665714 >> v1:172.21.15.46:6789/0 conn(0x7f0a24159a90 legacy=0x7f0a2406d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:33.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.173+0000 7f0a22ffd700 1 -- 172.21.15.46:0/1222665714 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2415deb0 con 0x7f0a241596c0 2024-01-27T01:07:33.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.173+0000 7f0a22ffd700 1 --2- 172.21.15.46:0/1222665714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a241596c0 0x7f0a2406bcd0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f0a140099b0 tx=0x7f0a14009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3699d9f30301148c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:33.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.173+0000 7f0a21ffb700 1 -- 172.21.15.46:0/1222665714 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a14009ea0 con 0x7f0a241596c0 2024-01-27T01:07:33.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.174+0000 7f0a21ffb700 1 -- 172.21.15.46:0/1222665714 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1400b860 con 0x7f0a241596c0 2024-01-27T01:07:33.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.174+0000 7f0a21ffb700 1 -- 172.21.15.46:0/1222665714 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1400bc00 con 0x7f0a241596c0 2024-01-27T01:07:33.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.174+0000 7f0a292d5700 1 -- 172.21.15.46:0/1222665714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a241596c0 msgr2=0x7f0a2406bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:33.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.174+0000 7f0a292d5700 1 --2- 172.21.15.46:0/1222665714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a241596c0 0x7f0a2406bcd0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f0a140099b0 tx=0x7f0a14009250 comp rx=0 tx=0).stop 2024-01-27T01:07:33.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.174+0000 7f0a292d5700 1 -- 172.21.15.46:0/1222665714 shutdown_connections 2024-01-27T01:07:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.174+0000 7f0a292d5700 1 --2- 172.21.15.46:0/1222665714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a241596c0 0x7f0a2406bcd0 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.174+0000 7f0a292d5700 1 --2- 172.21.15.46:0/1222665714 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a2415d460 0x7f0a2406c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.174+0000 7f0a292d5700 1 -- 172.21.15.46:0/1222665714 >> 172.21.15.46:0/1222665714 conn(0x7f0a240ce530 msgr2=0x7f0a240ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.174+0000 7f0a292d5700 1 -- 172.21.15.46:0/1222665714 shutdown_connections 2024-01-27T01:07:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.174+0000 7f0a292d5700 1 -- 172.21.15.46:0/1222665714 wait complete. 2024-01-27T01:07:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.176+0000 7f0a292d5700 1 Processor -- start 2024-01-27T01:07:33.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.176+0000 7f0a292d5700 1 -- start start 2024-01-27T01:07:33.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.177+0000 7f0a292d5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a241596c0 0x7f0a24156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:33.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.177+0000 7f0a292d5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a24159a90 0x7f0a2415c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:33.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.177+0000 7f0a292d5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a2415d460 0x7f0a2415c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:33.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.177+0000 7f0a292d5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a240b3490 con 0x7f0a24159a90 2024-01-27T01:07:33.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.177+0000 7f0a292d5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a241570a0 con 0x7f0a241596c0 2024-01-27T01:07:33.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.177+0000 7f0a292d5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a240af3d0 con 0x7f0a2415d460 2024-01-27T01:07:33.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.177+0000 7f0a227fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a24159a90 0x7f0a2415c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:33.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.177+0000 7f0a227fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a24159a90 0x7f0a2415c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45852/0 (socket says 172.21.15.46:45852) 2024-01-27T01:07:33.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.177+0000 7f0a227fc700 1 -- 172.21.15.46:0/446112087 learned_addr learned my addr 172.21.15.46:0/446112087 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:33.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.177+0000 7f0a237fe700 1 --2- 172.21.15.46:0/446112087 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a2415d460 0x7f0a2415c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:33.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.178+0000 7f0a22ffd700 1 --2- 172.21.15.46:0/446112087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a241596c0 0x7f0a24156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:33.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.178+0000 7f0a227fc700 1 -- 172.21.15.46:0/446112087 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a2415d460 msgr2=0x7f0a2415c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:33.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.178+0000 7f0a227fc700 1 --2- 172.21.15.46:0/446112087 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a2415d460 0x7f0a2415c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:33.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.178+0000 7f0a227fc700 1 -- 172.21.15.46:0/446112087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a241596c0 msgr2=0x7f0a24156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:33.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.178+0000 7f0a227fc700 1 --2- 172.21.15.46:0/446112087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a241596c0 0x7f0a24156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:33.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.178+0000 7f0a227fc700 1 -- 172.21.15.46:0/446112087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a140094f0 con 0x7f0a24159a90 2024-01-27T01:07:33.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.178+0000 7f0a227fc700 1 --2- 172.21.15.46:0/446112087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a24159a90 0x7f0a2415c160 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800e930 tx=0x7f0a180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:33.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.178+0000 7f0a0affd700 1 -- 172.21.15.46:0/446112087 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1800eed0 con 0x7f0a24159a90 2024-01-27T01:07:33.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.178+0000 7f0a0affd700 1 -- 172.21.15.46:0/446112087 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1800c850 con 0x7f0a24159a90 2024-01-27T01:07:33.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.178+0000 7f0a292d5700 1 -- 172.21.15.46:0/446112087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a240af6b0 con 0x7f0a24159a90 2024-01-27T01:07:33.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.178+0000 7f0a0affd700 1 -- 172.21.15.46:0/446112087 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1800cbf0 con 0x7f0a24159a90 2024-01-27T01:07:33.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.178+0000 7f0a292d5700 1 -- 172.21.15.46:0/446112087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a240afc70 con 0x7f0a24159a90 2024-01-27T01:07:33.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.179+0000 7f0a292d5700 1 -- 172.21.15.46:0/446112087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2404c710 con 0x7f0a24159a90 2024-01-27T01:07:33.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.180+0000 7f0a0affd700 1 -- 172.21.15.46:0/446112087 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f0a18013470 con 0x7f0a24159a90 2024-01-27T01:07:33.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.180+0000 7f0a0affd700 1 --2- 172.21.15.46:0/446112087 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10067170 0x7f0a10069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:33.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.180+0000 7f0a0affd700 1 -- 172.21.15.46:0/446112087 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f0a18084db0 con 0x7f0a24159a90 2024-01-27T01:07:33.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.180+0000 7f0a22ffd700 1 --2- 172.21.15.46:0/446112087 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10067170 0x7f0a10069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:33.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.181+0000 7f0a22ffd700 1 --2- 172.21.15.46:0/446112087 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10067170 0x7f0a10069620 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0a140099b0 tx=0x7f0a14017000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.185+0000 7f0a0affd700 1 -- 172.21.15.46:0/446112087 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a18051a60 con 0x7f0a24159a90 2024-01-27T01:07:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.438+0000 7f0a292d5700 1 -- 172.21.15.46:0/446112087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a2415e190 con 0x7f0a24159a90 2024-01-27T01:07:33.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.439+0000 7f0a0affd700 1 -- 172.21.15.46:0/446112087 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+4610 (secure 0 0 0) 0x7f0a18055080 con 0x7f0a24159a90 2024-01-27T01:07:33.439 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:33.439 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":133,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1114164873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1114164873},{"type":"v1","addr":"172.21.15.106:6834","nonce":1114164873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128},{"gid":9821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/702094428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":702094428},{"type":"v1","addr":"172.21.15.106:6835","nonce":702094428}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":9827,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/408985798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":408985798},{"type":"v1","addr":"172.21.15.46:6835","nonce":408985798}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:07:27.375435+0000","modified":"2024-01-27T01:07:30.945729+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11248":{"gid":11248,"name":"a","rank":0,"incarnation":129,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/1087205220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1087205220},{"type":"v1","addr":"172.21.15.46:6834","nonce":1087205220}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2024-01-27T01:07:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.441+0000 7f0a292d5700 1 -- 172.21.15.46:0/446112087 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10067170 msgr2=0x7f0a10069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.441+0000 7f0a292d5700 1 --2- 172.21.15.46:0/446112087 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10067170 0x7f0a10069620 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0a140099b0 tx=0x7f0a14017000 comp rx=0 tx=0).stop 2024-01-27T01:07:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.441+0000 7f0a292d5700 1 -- 172.21.15.46:0/446112087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a24159a90 msgr2=0x7f0a2415c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.441+0000 7f0a292d5700 1 --2- 172.21.15.46:0/446112087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a24159a90 0x7f0a2415c160 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800e930 tx=0x7f0a180091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.441+0000 7f0a292d5700 1 -- 172.21.15.46:0/446112087 shutdown_connections 2024-01-27T01:07:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.441+0000 7f0a292d5700 1 --2- 172.21.15.46:0/446112087 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0a10067170 0x7f0a10069620 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.441+0000 7f0a292d5700 1 --2- 172.21.15.46:0/446112087 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a2415d460 0x7f0a2415c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.441+0000 7f0a292d5700 1 --2- 172.21.15.46:0/446112087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a241596c0 0x7f0a24156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.441+0000 7f0a292d5700 1 --2- 172.21.15.46:0/446112087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a24159a90 0x7f0a2415c160 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.441+0000 7f0a292d5700 1 -- 172.21.15.46:0/446112087 >> 172.21.15.46:0/446112087 conn(0x7f0a240ce530 msgr2=0x7f0a240cd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.441+0000 7f0a292d5700 1 -- 172.21.15.46:0/446112087 shutdown_connections 2024-01-27T01:07:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.441+0000 7f0a292d5700 1 -- 172.21.15.46:0/446112087 wait complete. 2024-01-27T01:07:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 133 2024-01-27T01:07:33.858 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 133, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-27T01:07:27.375435+0000', 'modified': '2024-01-27T01:07:30.945729+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11248}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11248': {'gid': 11248, 'name': 'a', 'rank': 0, 'incarnation': 129, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6834/1087205220', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 1087205220}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 1087205220}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [32], 'metadata_pool': 31, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-27T01:07:33.858 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-27T01:07:33.858 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-27T01:07:33.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.999+0000 7f42fb3a3700 1 Processor -- start 2024-01-27T01:07:33.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:33.999+0000 7f42fb3a3700 1 -- start start 2024-01-27T01:07:34.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.000+0000 7f42fb3a3700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f42f40cb5a0 0x7f42f40ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.000+0000 7f42fb3a3700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f40b0df0 con 0x7f42f40c4410 2024-01-27T01:07:34.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.000+0000 7f42fb3a3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f40c9df0 con 0x7f42f40b0a20 2024-01-27T01:07:34.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.000+0000 7f42fb3a3700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f4157c00 con 0x7f42f40cb5a0 2024-01-27T01:07:34.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.001+0000 7f42f9940700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f42f40c4410 0x7f42f4161300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45552/0 (socket says 172.21.15.46:45552) 2024-01-27T01:07:34.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.001+0000 7f42f9940700 1 -- 172.21.15.46:0/3976264815 learned_addr learned my addr 172.21.15.46:0/3976264815 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:34.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.001+0000 7f42f893e700 1 --2- 172.21.15.46:0/3976264815 >> v2:172.21.15.106:3300/0 conn(0x7f42f40cb5a0 0x7f42f40ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.001+0000 7f42ebfff700 1 -- 172.21.15.46:0/3976264815 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986645176 0 0) 0x7f42f40b0df0 con 0x7f42f40c4410 2024-01-27T01:07:34.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.001+0000 7f42ebfff700 1 -- 172.21.15.46:0/3976264815 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42e0003570 con 0x7f42f40c4410 2024-01-27T01:07:34.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.001+0000 7f42ebfff700 1 -- 172.21.15.46:0/3976264815 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949041029 0 0) 0x7f42f40c9df0 con 0x7f42f40b0a20 2024-01-27T01:07:34.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.001+0000 7f42ebfff700 1 -- 172.21.15.46:0/3976264815 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42f40b0df0 con 0x7f42f40b0a20 2024-01-27T01:07:34.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.001+0000 7f42ebfff700 1 -- 172.21.15.46:0/3976264815 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3866405765 0 0) 0x7f42e0003570 con 0x7f42f40c4410 2024-01-27T01:07:34.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.001+0000 7f42f893e700 1 -- 172.21.15.46:0/3976264815 >> v1:172.21.15.106:6789/0 conn(0x7f42f40b0a20 legacy=0x7f42f40cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:34.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.001+0000 7f42f893e700 1 -- 172.21.15.46:0/3976264815 >> v1:172.21.15.46:6789/0 conn(0x7f42f40c4410 legacy=0x7f42f4161300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:34.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.001+0000 7f42f893e700 1 -- 172.21.15.46:0/3976264815 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f40ca0a0 con 0x7f42f40cb5a0 2024-01-27T01:07:34.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.001+0000 7f42f893e700 1 --2- 172.21.15.46:0/3976264815 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42f40cb5a0 0x7f42f40ccc00 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f42e40098e0 tx=0x7f42e40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4ab30b9136e2b1c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:34.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.002+0000 7f42ebfff700 1 -- 172.21.15.46:0/3976264815 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec0017f0 con 0x7f42f40cb5a0 2024-01-27T01:07:34.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.002+0000 7f42ebfff700 1 -- 172.21.15.46:0/3976264815 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e4011420 con 0x7f42f40cb5a0 2024-01-27T01:07:34.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.002+0000 7f42ebfff700 1 -- 172.21.15.46:0/3976264815 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e40117c0 con 0x7f42f40cb5a0 2024-01-27T01:07:34.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.002+0000 7f42fb3a3700 1 -- 172.21.15.46:0/3976264815 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42f40cb5a0 msgr2=0x7f42f40ccc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:34.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.002+0000 7f42fb3a3700 1 --2- 172.21.15.46:0/3976264815 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42f40cb5a0 0x7f42f40ccc00 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f42e40098e0 tx=0x7f42e40091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.002+0000 7f42fb3a3700 1 -- 172.21.15.46:0/3976264815 shutdown_connections 2024-01-27T01:07:34.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.002+0000 7f42fb3a3700 1 --2- 172.21.15.46:0/3976264815 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42f40cb5a0 0x7f42f40ccc00 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.002+0000 7f42fb3a3700 1 -- 172.21.15.46:0/3976264815 >> 172.21.15.46:0/3976264815 conn(0x7f42f4097820 msgr2=0x7f42f4068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:34.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.002+0000 7f42fb3a3700 1 -- 172.21.15.46:0/3976264815 shutdown_connections 2024-01-27T01:07:34.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.002+0000 7f42fb3a3700 1 -- 172.21.15.46:0/3976264815 wait complete. 2024-01-27T01:07:34.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.004+0000 7f42fb3a3700 1 Processor -- start 2024-01-27T01:07:34.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.004+0000 7f42fb3a3700 1 -- start start 2024-01-27T01:07:34.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.005+0000 7f42fb3a3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f40b0a20 0x7f42f40b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.005+0000 7f42fb3a3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f40c4410 0x7f42f4157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.005+0000 7f42fb3a3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42f40cb5a0 0x7f42f4158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.005+0000 7f42fb3a3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f4151970 con 0x7f42f40c4410 2024-01-27T01:07:34.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.005+0000 7f42fb3a3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f40af770 con 0x7f42f40cb5a0 2024-01-27T01:07:34.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.005+0000 7f42fb3a3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f4068a40 con 0x7f42f40b0a20 2024-01-27T01:07:34.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.005+0000 7f42f893e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f40c4410 0x7f42f4157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.005+0000 7f42f893e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f40c4410 0x7f42f4157d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45858/0 (socket says 172.21.15.46:45858) 2024-01-27T01:07:34.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.005+0000 7f42f893e700 1 -- 172.21.15.46:0/3015138396 learned_addr learned my addr 172.21.15.46:0/3015138396 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:34.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.005+0000 7f42f9940700 1 --2- 172.21.15.46:0/3015138396 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42f40cb5a0 0x7f42f4158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.005+0000 7f42f913f700 1 --2- 172.21.15.46:0/3015138396 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f40b0a20 0x7f42f40b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.005+0000 7f42f893e700 1 -- 172.21.15.46:0/3015138396 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f40b0a20 msgr2=0x7f42f40b3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:34.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.005+0000 7f42f893e700 1 --2- 172.21.15.46:0/3015138396 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f40b0a20 0x7f42f40b3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.005+0000 7f42f893e700 1 -- 172.21.15.46:0/3015138396 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42f40cb5a0 msgr2=0x7f42f4158280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:34.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.005+0000 7f42f893e700 1 --2- 172.21.15.46:0/3015138396 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42f40cb5a0 0x7f42f4158280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.005+0000 7f42f893e700 1 -- 172.21.15.46:0/3015138396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e4009420 con 0x7f42f40c4410 2024-01-27T01:07:34.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.006+0000 7f42f893e700 1 --2- 172.21.15.46:0/3015138396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f40c4410 0x7f42f4157d40 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f42e4009ed0 tx=0x7f42e4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:34.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.006+0000 7f42e97fa700 1 -- 172.21.15.46:0/3015138396 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e401cb20 con 0x7f42f40c4410 2024-01-27T01:07:34.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.006+0000 7f42e97fa700 1 -- 172.21.15.46:0/3015138396 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e401cca0 con 0x7f42f40c4410 2024-01-27T01:07:34.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.006+0000 7f42fb3a3700 1 -- 172.21.15.46:0/3015138396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f4069000 con 0x7f42f40c4410 2024-01-27T01:07:34.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.006+0000 7f42e97fa700 1 -- 172.21.15.46:0/3015138396 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e4025680 con 0x7f42f40c4410 2024-01-27T01:07:34.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.006+0000 7f42fb3a3700 1 -- 172.21.15.46:0/3015138396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f4158c00 con 0x7f42f40c4410 2024-01-27T01:07:34.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.007+0000 7f42e97fa700 1 -- 172.21.15.46:0/3015138396 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f42e40193f0 con 0x7f42f40c4410 2024-01-27T01:07:34.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.008+0000 7f42e97fa700 1 --2- 172.21.15.46:0/3015138396 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f42d8067170 0x7f42d8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.008+0000 7f42e97fa700 1 -- 172.21.15.46:0/3015138396 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f42e4085e20 con 0x7f42f40c4410 2024-01-27T01:07:34.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.008+0000 7f42f913f700 1 --2- 172.21.15.46:0/3015138396 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f42d8067170 0x7f42d8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.008+0000 7f42f913f700 1 --2- 172.21.15.46:0/3015138396 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f42d8067170 0x7f42d8069620 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f42f0003a40 tx=0x7f42f0007d40 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:34.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.010+0000 7f43046a6b80 1 Processor -- start 2024-01-27T01:07:34.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.010+0000 7f43046a6b80 1 -- start start 2024-01-27T01:07:34.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f43046a6b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa5ee3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f43046a6b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa636360 0x55efaa618840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f43046a6b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa5ed5d0 0x55efaa618d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f43046a6b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efaa4f19f0 con 0x55efaa5ed5d0 2024-01-27T01:07:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f43046a6b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efaa543130 con 0x55efaa5ee010 2024-01-27T01:07:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f43046a6b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55efaa540930 con 0x55efaa636360 2024-01-27T01:07:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f42f9940700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa5ed5d0 0x55efaa618d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f42f9940700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa5ed5d0 0x55efaa618d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45870/0 (socket says 172.21.15.46:45870) 2024-01-27T01:07:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f42f9940700 1 -- 172.21.15.46:0/1195398627 learned_addr learned my addr 172.21.15.46:0/1195398627 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f42f913f700 1 --2- 172.21.15.46:0/1195398627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa5ee3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f42f893e700 1 --2- 172.21.15.46:0/1195398627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa636360 0x55efaa618840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f42f9940700 1 -- 172.21.15.46:0/1195398627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa636360 msgr2=0x55efaa618840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f42f9940700 1 --2- 172.21.15.46:0/1195398627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa636360 0x55efaa618840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f42f9940700 1 -- 172.21.15.46:0/1195398627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 msgr2=0x55efaa5ee3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f42f9940700 1 --2- 172.21.15.46:0/1195398627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa5ee3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f42f9940700 1 -- 172.21.15.46:0/1195398627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efaa37bb10 con 0x55efaa5ed5d0 2024-01-27T01:07:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f42f913f700 1 --2- 172.21.15.46:0/1195398627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa5ee3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:34.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f42f9940700 1 --2- 172.21.15.46:0/1195398627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa5ed5d0 0x55efaa618d80 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f42ec002070 tx=0x7f42ec000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bbda12b4e4a99a7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:34.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.011+0000 7f42de7fc700 1 -- 172.21.15.46:0/1195398627 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec0020a0 con 0x55efaa5ed5d0 2024-01-27T01:07:34.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.012+0000 7f42de7fc700 1 -- 172.21.15.46:0/1195398627 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ec0077d0 con 0x55efaa5ed5d0 2024-01-27T01:07:34.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.012+0000 7f42de7fc700 1 -- 172.21.15.46:0/1195398627 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec0133b0 con 0x55efaa5ed5d0 2024-01-27T01:07:34.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.012+0000 7f43046a6b80 1 -- 172.21.15.46:0/1195398627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa5ed5d0 msgr2=0x55efaa618d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:34.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.012+0000 7f43046a6b80 1 --2- 172.21.15.46:0/1195398627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa5ed5d0 0x55efaa618d80 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f42ec002070 tx=0x7f42ec000f40 comp rx=0 tx=0).stop 2024-01-27T01:07:34.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.012+0000 7f43046a6b80 1 -- 172.21.15.46:0/1195398627 shutdown_connections 2024-01-27T01:07:34.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.012+0000 7f43046a6b80 1 --2- 172.21.15.46:0/1195398627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa5ed5d0 0x55efaa618d80 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.012+0000 7f43046a6b80 1 --2- 172.21.15.46:0/1195398627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa5ee3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.012+0000 7f43046a6b80 1 --2- 172.21.15.46:0/1195398627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa636360 0x55efaa618840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.012+0000 7f43046a6b80 1 -- 172.21.15.46:0/1195398627 >> 172.21.15.46:0/1195398627 conn(0x55efaa5eda70 msgr2=0x55efaa636820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:34.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.012+0000 7f43046a6b80 1 -- 172.21.15.46:0/1195398627 shutdown_connections 2024-01-27T01:07:34.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.012+0000 7f43046a6b80 1 -- 172.21.15.46:0/1195398627 wait complete. 2024-01-27T01:07:34.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.013+0000 7f43046a6b80 1 -- start start 2024-01-27T01:07:34.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.013+0000 7f43046a6b80 1 Processor -- start 2024-01-27T01:07:34.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.014+0000 7f43046a6b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa5ed5d0 0x55efaa609540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.014+0000 7f43046a6b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa5eca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.014+0000 7f43046a6b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa636360 0x55efaa5e7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.014+0000 7f43046a6b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efaa53c720 con 0x55efaa636360 2024-01-27T01:07:34.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.014+0000 7f43046a6b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efaa4e81b0 con 0x55efaa5ee010 2024-01-27T01:07:34.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.014+0000 7f43046a6b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55efaa4e7e50 con 0x55efaa5ed5d0 2024-01-27T01:07:34.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.014+0000 7f43046a6b80 10 client.0 _ll_register_callbacks cb 0x55efaa34a7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-27T01:07:34.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.014+0000 7f42f9940700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa636360 0x55efaa5e7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.014+0000 7f42f9940700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa636360 0x55efaa5e7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45874/0 (socket says 172.21.15.46:45874) 2024-01-27T01:07:34.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.014+0000 7f42f9940700 1 -- 172.21.15.46:0/4139937744 learned_addr learned my addr 172.21.15.46:0/4139937744 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:34.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.014+0000 7f42f893e700 1 --2- 172.21.15.46:0/4139937744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa5eca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.014+0000 7f42f913f700 1 --2- 172.21.15.46:0/4139937744 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa5ed5d0 0x55efaa609540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.014+0000 7f42f9940700 1 -- 172.21.15.46:0/4139937744 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa5ed5d0 msgr2=0x55efaa609540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:34.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.014+0000 7f42f9940700 1 --2- 172.21.15.46:0/4139937744 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa5ed5d0 0x55efaa609540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.014+0000 7f42f9940700 1 -- 172.21.15.46:0/4139937744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 msgr2=0x55efaa5eca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:34.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.014+0000 7f42f9940700 1 --2- 172.21.15.46:0/4139937744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa5eca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f42f9940700 1 -- 172.21.15.46:0/4139937744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f40b0df0 con 0x55efaa636360 2024-01-27T01:07:34.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f42f893e700 1 --2- 172.21.15.46:0/4139937744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa5eca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:34.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f42f9940700 1 --2- 172.21.15.46:0/4139937744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa636360 0x55efaa5e7240 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f42ec013000 tx=0x7f42ec014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=392d48dbb360618b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:34.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f42d37fe700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-27T01:07:34.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f42d37fe700 1 -- 172.21.15.46:0/4139937744 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec0133b0 con 0x55efaa636360 2024-01-27T01:07:34.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f42d37fe700 1 -- 172.21.15.46:0/4139937744 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f42ec0020a0 con 0x55efaa636360 2024-01-27T01:07:34.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f43046a6b80 1 -- 172.21.15.46:0/4139937744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55efaa5e1190 con 0x55efaa636360 2024-01-27T01:07:34.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f42d37fe700 1 -- 172.21.15.46:0/4139937744 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=133) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42ec0016f0 con 0x55efaa636360 2024-01-27T01:07:34.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f43046a6b80 10 client.11781 fetch_fsmap learned FSMap version 133 2024-01-27T01:07:34.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f43046a6b80 1 -- 172.21.15.46:0/4139937744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=133,monmap=2+}) v3 -- 0x7f42ec001c70 con 0x55efaa636360 2024-01-27T01:07:34.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f42d37fe700 1 -- 172.21.15.46:0/4139937744 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55efaa5e1190 con 0x55efaa636360 2024-01-27T01:07:34.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f42d37fe700 1 -- 172.21.15.46:0/4139937744 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 133) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f42ec01c030 con 0x55efaa636360 2024-01-27T01:07:34.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f43046a6b80 10 client.11781 fetch_fsmap finished waiting for FSMap version 133 2024-01-27T01:07:34.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f43046a6b80 10 client.11781 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-27T01:07:34.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f43046a6b80 20 client.11781 populate_metadata read hostname 'smithi046' 2024-01-27T01:07:34.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f43046a6b80 1 --2- 172.21.15.46:0/4139937744 >> [v2:172.21.15.46:6832/1087205220,v1:172.21.15.46:6834/1087205220] conn(0x55efaa619bc0 0x55efaa619f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f43046a6b80 4 client.11781 mds_command: new command op to 11248 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-27T01:07:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.015+0000 7f43046a6b80 1 -- 172.21.15.46:0/4139937744 --> [v2:172.21.15.46:6832/1087205220,v1:172.21.15.46:6834/1087205220] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55efaa4deee0 con 0x55efaa619bc0 2024-01-27T01:07:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.016+0000 7f42f913f700 1 --2- 172.21.15.46:0/4139937744 >> [v2:172.21.15.46:6832/1087205220,v1:172.21.15.46:6834/1087205220] conn(0x55efaa619bc0 0x55efaa619f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.016+0000 7f42f913f700 1 --2- 172.21.15.46:0/4139937744 >> [v2:172.21.15.46:6832/1087205220,v1:172.21.15.46:6834/1087205220] conn(0x55efaa619bc0 0x55efaa619f90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1dba8e9d0f15545 server_cookie=acf88b23af0e2160 in_seq=0 out_seq=0 2024-01-27T01:07:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.016+0000 7f42d37fe700 10 client.11781 ms_handle_connect on v2:172.21.15.46:6832/1087205220 2024-01-27T01:07:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.017+0000 7f42d37fe700 1 -- 172.21.15.46:0/4139937744 <== mds.0 v2:172.21.15.46:6832/1087205220 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x55efaa4deee0 con 0x55efaa619bc0 2024-01-27T01:07:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.017+0000 7f42d37fe700 10 client.11781 handle_command_reply: tid=0 2024-01-27T01:07:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.017+0000 7f43046a6b80 1 client.11781 shutdown 2024-01-27T01:07:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.017+0000 7f43046a6b80 1 -- 172.21.15.46:0/4139937744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa636360 msgr2=0x55efaa5e7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.017+0000 7f43046a6b80 1 --2- 172.21.15.46:0/4139937744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa636360 0x55efaa5e7240 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f42ec013000 tx=0x7f42ec014000 comp rx=0 tx=0).stop 2024-01-27T01:07:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.017+0000 7f43046a6b80 1 -- 172.21.15.46:0/4139937744 shutdown_connections 2024-01-27T01:07:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.017+0000 7f43046a6b80 1 --2- 172.21.15.46:0/4139937744 >> [v2:172.21.15.46:6832/1087205220,v1:172.21.15.46:6834/1087205220] conn(0x55efaa619bc0 0x55efaa619f90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.017+0000 7f43046a6b80 1 --2- 172.21.15.46:0/4139937744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa636360 0x55efaa5e7240 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.017+0000 7f43046a6b80 1 --2- 172.21.15.46:0/4139937744 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa5ed5d0 0x55efaa609540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.017+0000 7f43046a6b80 1 --2- 172.21.15.46:0/4139937744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa5eca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.017+0000 7f42d37fe700 0 client.11781 ms_handle_reset on v2:172.21.15.46:6832/1087205220 2024-01-27T01:07:34.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.017+0000 7f43046a6b80 1 -- 172.21.15.46:0/4139937744 >> 172.21.15.46:0/4139937744 conn(0x55efaa5eda70 msgr2=0x55efaa5e8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:34.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.017+0000 7f43046a6b80 1 -- 172.21.15.46:0/4139937744 shutdown_connections 2024-01-27T01:07:34.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.017+0000 7f43046a6b80 1 -- 172.21.15.46:0/4139937744 wait complete. 2024-01-27T01:07:34.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.017+0000 7f43046a6b80 20 client.11781 trim_cache size 0 max 16384 2024-01-27T01:07:34.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.035+0000 7f43046a6b80 1 Processor -- start 2024-01-27T01:07:34.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.035+0000 7f43046a6b80 1 -- start start 2024-01-27T01:07:34.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f43046a6b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa5ed5d0 0x55efaa6351c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f43046a6b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa635700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f43046a6b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa636360 0x55efaa635c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f43046a6b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efaa434e40 con 0x55efaa5ed5d0 2024-01-27T01:07:34.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f43046a6b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efaa5ee550 con 0x55efaa5ee010 2024-01-27T01:07:34.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f43046a6b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55efaa5ebe60 con 0x55efaa636360 2024-01-27T01:07:34.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f42f913f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa5ed5d0 0x55efaa6351c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f42f913f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa5ed5d0 0x55efaa6351c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45888/0 (socket says 172.21.15.46:45888) 2024-01-27T01:07:34.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f42f913f700 1 -- 172.21.15.46:0/3459253524 learned_addr learned my addr 172.21.15.46:0/3459253524 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:34.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f42f9940700 1 --2- 172.21.15.46:0/3459253524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa636360 0x55efaa635c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f42f893e700 1 --2- 172.21.15.46:0/3459253524 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa635700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f42f913f700 1 -- 172.21.15.46:0/3459253524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa636360 msgr2=0x55efaa635c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:34.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f42f913f700 1 --2- 172.21.15.46:0/3459253524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa636360 0x55efaa635c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f42f913f700 1 -- 172.21.15.46:0/3459253524 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 msgr2=0x55efaa635700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:34.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f42f913f700 1 --2- 172.21.15.46:0/3459253524 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa635700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f42f913f700 1 -- 172.21.15.46:0/3459253524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efaa50a290 con 0x55efaa5ed5d0 2024-01-27T01:07:34.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f42f913f700 1 --2- 172.21.15.46:0/3459253524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa5ed5d0 0x55efaa6351c0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f42f00033d0 tx=0x7f42f0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25fc392f0c2c3ed9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:34.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.036+0000 7f42deffd700 1 -- 172.21.15.46:0/3459253524 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f000fdf0 con 0x55efaa5ed5d0 2024-01-27T01:07:34.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.037+0000 7f42deffd700 1 -- 172.21.15.46:0/3459253524 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f00240d0 con 0x55efaa5ed5d0 2024-01-27T01:07:34.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.037+0000 7f42deffd700 1 -- 172.21.15.46:0/3459253524 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f0024980 con 0x55efaa5ed5d0 2024-01-27T01:07:34.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.037+0000 7f43046a6b80 1 -- 172.21.15.46:0/3459253524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa5ed5d0 msgr2=0x55efaa6351c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:34.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.037+0000 7f43046a6b80 1 --2- 172.21.15.46:0/3459253524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa5ed5d0 0x55efaa6351c0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f42f00033d0 tx=0x7f42f0013000 comp rx=0 tx=0).stop 2024-01-27T01:07:34.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.037+0000 7f43046a6b80 1 -- 172.21.15.46:0/3459253524 shutdown_connections 2024-01-27T01:07:34.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.037+0000 7f43046a6b80 1 --2- 172.21.15.46:0/3459253524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa5ed5d0 0x55efaa6351c0 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.037+0000 7f43046a6b80 1 --2- 172.21.15.46:0/3459253524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa636360 0x55efaa635c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.037+0000 7f43046a6b80 1 --2- 172.21.15.46:0/3459253524 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa635700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.037+0000 7f43046a6b80 1 -- 172.21.15.46:0/3459253524 >> 172.21.15.46:0/3459253524 conn(0x55efaa5eda70 msgr2=0x55efaa5ea3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:34.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.037+0000 7f43046a6b80 1 -- 172.21.15.46:0/3459253524 shutdown_connections 2024-01-27T01:07:34.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.037+0000 7f43046a6b80 1 -- 172.21.15.46:0/3459253524 wait complete. 2024-01-27T01:07:34.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.038+0000 7f43046a6b80 1 -- start start 2024-01-27T01:07:34.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.038+0000 7f43046a6b80 1 Processor -- start 2024-01-27T01:07:34.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.039+0000 7f43046a6b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa5ed5d0 0x55efaa64ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.039+0000 7f43046a6b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa608a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f43046a6b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa636360 0x55efaa67d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f43046a6b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efaa64f260 con 0x55efaa636360 2024-01-27T01:07:34.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f43046a6b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efaa608f60 con 0x55efaa5ee010 2024-01-27T01:07:34.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f43046a6b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55efaa67e440 con 0x55efaa5ed5d0 2024-01-27T01:07:34.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f43046a6b80 10 client.0 _ll_register_callbacks cb 0x55efaa34a7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-27T01:07:34.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f42f9940700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa636360 0x55efaa67d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f42f9940700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa636360 0x55efaa67d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45904/0 (socket says 172.21.15.46:45904) 2024-01-27T01:07:34.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f42f9940700 1 -- 172.21.15.46:0/3149583174 learned_addr learned my addr 172.21.15.46:0/3149583174 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:34.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f42f913f700 1 --2- 172.21.15.46:0/3149583174 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa5ed5d0 0x55efaa64ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f42f893e700 1 --2- 172.21.15.46:0/3149583174 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa608a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f42f9940700 1 -- 172.21.15.46:0/3149583174 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa5ed5d0 msgr2=0x55efaa64ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:34.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f42f9940700 1 --2- 172.21.15.46:0/3149583174 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa5ed5d0 0x55efaa64ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f42f9940700 1 -- 172.21.15.46:0/3149583174 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 msgr2=0x55efaa608a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:34.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f42f9940700 1 --2- 172.21.15.46:0/3149583174 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa608a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f42f9940700 1 -- 172.21.15.46:0/3149583174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f000fb00 con 0x55efaa636360 2024-01-27T01:07:34.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f42f9940700 1 --2- 172.21.15.46:0/3149583174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa636360 0x55efaa67d7c0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f42ec013000 tx=0x7f42ec021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e920166b4ec2fd31 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:34.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f42d37fe700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-27T01:07:34.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f42d37fe700 1 -- 172.21.15.46:0/3149583174 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec0133b0 con 0x55efaa636360 2024-01-27T01:07:34.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f42d37fe700 1 -- 172.21.15.46:0/3149583174 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f42ec014db0 con 0x55efaa636360 2024-01-27T01:07:34.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.040+0000 7f43046a6b80 1 -- 172.21.15.46:0/3149583174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f42f405f2f0 con 0x55efaa636360 2024-01-27T01:07:34.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.041+0000 7f42d37fe700 1 -- 172.21.15.46:0/3149583174 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=133) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42ec001450 con 0x55efaa636360 2024-01-27T01:07:34.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.041+0000 7f43046a6b80 10 client.11787 fetch_fsmap learned FSMap version 133 2024-01-27T01:07:34.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.041+0000 7f43046a6b80 1 -- 172.21.15.46:0/3149583174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=133,monmap=2+}) v3 -- 0x55efaa3d3c80 con 0x55efaa636360 2024-01-27T01:07:34.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.041+0000 7f42d37fe700 1 -- 172.21.15.46:0/3149583174 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f405f2f0 con 0x55efaa636360 2024-01-27T01:07:34.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.041+0000 7f42d37fe700 1 -- 172.21.15.46:0/3149583174 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 133) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f42ec034020 con 0x55efaa636360 2024-01-27T01:07:34.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.041+0000 7f43046a6b80 10 client.11787 fetch_fsmap finished waiting for FSMap version 133 2024-01-27T01:07:34.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.041+0000 7f43046a6b80 10 client.11787 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-27T01:07:34.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.041+0000 7f43046a6b80 20 client.11787 populate_metadata read hostname 'smithi046' 2024-01-27T01:07:34.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.041+0000 7f43046a6b80 1 --2- 172.21.15.46:0/3149583174 >> [v2:172.21.15.46:6832/1087205220,v1:172.21.15.46:6834/1087205220] conn(0x55efaa62e4c0 0x55efaa62e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.041+0000 7f43046a6b80 4 client.11787 mds_command: new command op to 11248 tid=0[{"prefix": "status"}] 2024-01-27T01:07:34.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.041+0000 7f43046a6b80 1 -- 172.21.15.46:0/3149583174 --> [v2:172.21.15.46:6832/1087205220,v1:172.21.15.46:6834/1087205220] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f42ec0077d0 con 0x55efaa62e4c0 2024-01-27T01:07:34.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.041+0000 7f42f913f700 1 --2- 172.21.15.46:0/3149583174 >> [v2:172.21.15.46:6832/1087205220,v1:172.21.15.46:6834/1087205220] conn(0x55efaa62e4c0 0x55efaa62e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.041+0000 7f42d37fe700 10 client.11787 ms_handle_connect on v2:172.21.15.46:6832/1087205220 2024-01-27T01:07:34.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.041+0000 7f42f913f700 1 --2- 172.21.15.46:0/3149583174 >> [v2:172.21.15.46:6832/1087205220,v1:172.21.15.46:6834/1087205220] conn(0x55efaa62e4c0 0x55efaa62e890 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34031d09a0c2f9e5 server_cookie=281b9bc338b07bce in_seq=0 out_seq=0 2024-01-27T01:07:34.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.042+0000 7f42d37fe700 1 -- 172.21.15.46:0/3149583174 <== mds.0 v2:172.21.15.46:6832/1087205220 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x7f42ec0077d0 con 0x55efaa62e4c0 2024-01-27T01:07:34.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.042+0000 7f42d37fe700 10 client.11787 handle_command_reply: tid=0 2024-01-27T01:07:34.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.042+0000 7f43046a6b80 1 client.11787 shutdown 2024-01-27T01:07:34.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.042+0000 7f43046a6b80 1 -- 172.21.15.46:0/3149583174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa636360 msgr2=0x55efaa67d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:34.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.042+0000 7f43046a6b80 1 --2- 172.21.15.46:0/3149583174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa636360 0x55efaa67d7c0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f42ec013000 tx=0x7f42ec021000 comp rx=0 tx=0).stop 2024-01-27T01:07:34.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.042+0000 7f43046a6b80 1 -- 172.21.15.46:0/3149583174 shutdown_connections 2024-01-27T01:07:34.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.042+0000 7f43046a6b80 1 --2- 172.21.15.46:0/3149583174 >> [v2:172.21.15.46:6832/1087205220,v1:172.21.15.46:6834/1087205220] conn(0x55efaa62e4c0 0x55efaa62e890 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.042+0000 7f43046a6b80 1 --2- 172.21.15.46:0/3149583174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efaa636360 0x55efaa67d7c0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.042+0000 7f43046a6b80 1 --2- 172.21.15.46:0/3149583174 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55efaa5ed5d0 0x55efaa64ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.042+0000 7f42d37fe700 0 client.11787 ms_handle_reset on v2:172.21.15.46:6832/1087205220 2024-01-27T01:07:34.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.042+0000 7f43046a6b80 1 --2- 172.21.15.46:0/3149583174 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55efaa5ee010 0x55efaa608a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.042+0000 7f43046a6b80 1 -- 172.21.15.46:0/3149583174 >> 172.21.15.46:0/3149583174 conn(0x55efaa5eda70 msgr2=0x55efaa5ea3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:34.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.042+0000 7f43046a6b80 1 -- 172.21.15.46:0/3149583174 shutdown_connections 2024-01-27T01:07:34.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.042+0000 7f43046a6b80 1 -- 172.21.15.46:0/3149583174 wait complete. 2024-01-27T01:07:34.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.042+0000 7f43046a6b80 20 client.11787 trim_cache size 0 max 16384 2024-01-27T01:07:34.047 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:07:34.048 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-27T01:07:34.048 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-27T01:07:34.048 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 11248, 2024-01-27T01:07:34.048 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-27T01:07:34.048 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-27T01:07:34.048 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-01-27T01:07:34.048 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.1110971599999999, 2024-01-27T01:07:34.048 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 133, 2024-01-27T01:07:34.048 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 243, 2024-01-27T01:07:34.048 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 243, 2024-01-27T01:07:34.048 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 11.015964425 2024-01-27T01:07:34.048 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:07:34.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.043+0000 7f42fb3a3700 1 -- 172.21.15.46:0/3015138396 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f42d8067170 msgr2=0x7f42d8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:34.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.043+0000 7f42fb3a3700 1 --2- 172.21.15.46:0/3015138396 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f42d8067170 0x7f42d8069620 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f42f0003a40 tx=0x7f42f0007d40 comp rx=0 tx=0).stop 2024-01-27T01:07:34.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.043+0000 7f42fb3a3700 1 -- 172.21.15.46:0/3015138396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f40c4410 msgr2=0x7f42f4157d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:34.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.043+0000 7f42fb3a3700 1 --2- 172.21.15.46:0/3015138396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f40c4410 0x7f42f4157d40 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f42e4009ed0 tx=0x7f42e4016000 comp rx=0 tx=0).stop 2024-01-27T01:07:34.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.043+0000 7f42fb3a3700 1 -- 172.21.15.46:0/3015138396 shutdown_connections 2024-01-27T01:07:34.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.043+0000 7f42fb3a3700 1 --2- 172.21.15.46:0/3015138396 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f42d8067170 0x7f42d8069620 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.043+0000 7f42fb3a3700 1 --2- 172.21.15.46:0/3015138396 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f42f40cb5a0 0x7f42f4158280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.043+0000 7f42fb3a3700 1 --2- 172.21.15.46:0/3015138396 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f42f40b0a20 0x7f42f40b3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.043+0000 7f42fb3a3700 1 --2- 172.21.15.46:0/3015138396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f40c4410 0x7f42f4157d40 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.043+0000 7f42fb3a3700 1 -- 172.21.15.46:0/3015138396 >> 172.21.15.46:0/3015138396 conn(0x7f42f4097820 msgr2=0x7f42f406ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:34.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.043+0000 7f42fb3a3700 1 -- 172.21.15.46:0/3015138396 shutdown_connections 2024-01-27T01:07:34.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.043+0000 7f42fb3a3700 1 -- 172.21.15.46:0/3015138396 wait complete. 2024-01-27T01:07:34.056 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-27T01:07:34.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.188+0000 7f7771ff2700 1 Processor -- start 2024-01-27T01:07:34.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.188+0000 7f7771ff2700 1 -- start start 2024-01-27T01:07:34.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.189+0000 7f7771ff2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f776c159760 0x7f776c0b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.189+0000 7f7771ff2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f776c159b30 0x7f776c0b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.189+0000 7f7771ff2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776c15abc0 con 0x7f776c15afc0 2024-01-27T01:07:34.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.189+0000 7f7771ff2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f776c0b2e90 con 0x7f776c159760 2024-01-27T01:07:34.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.189+0000 7f7771ff2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776c159fb0 con 0x7f776c159b30 2024-01-27T01:07:34.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.189+0000 7f776b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f776c159760 0x7f776c0b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.189+0000 7f776b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f776c159760 0x7f776c0b0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45920/0 (socket says 172.21.15.46:45920) 2024-01-27T01:07:34.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.189+0000 7f776b7fe700 1 -- 172.21.15.46:0/825500383 learned_addr learned my addr 172.21.15.46:0/825500383 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:34.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.189+0000 7f776affd700 1 --2- 172.21.15.46:0/825500383 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f776c159b30 0x7f776c0b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.190+0000 7f776b7fe700 1 -- 172.21.15.46:0/825500383 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f776c159b30 msgr2=0x7f776c0b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:34.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.190+0000 7f776b7fe700 1 --2- 172.21.15.46:0/825500383 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f776c159b30 0x7f776c0b11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.190+0000 7f776b7fe700 1 -- 172.21.15.46:0/825500383 >> v1:172.21.15.46:6789/0 conn(0x7f776c15afc0 legacy=0x7f776c067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:34.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.190+0000 7f776b7fe700 1 -- 172.21.15.46:0/825500383 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776c15a290 con 0x7f776c159760 2024-01-27T01:07:34.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.190+0000 7f776b7fe700 1 --2- 172.21.15.46:0/825500383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776c159760 0x7f776c0b0c70 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f775c0099b0 tx=0x7f775c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c165e85a8d5ed087 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:34.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.190+0000 7f776a7fc700 1 -- 172.21.15.46:0/825500383 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775c009ea0 con 0x7f776c159760 2024-01-27T01:07:34.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.190+0000 7f776a7fc700 1 -- 172.21.15.46:0/825500383 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775c00b860 con 0x7f776c159760 2024-01-27T01:07:34.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.190+0000 7f776a7fc700 1 -- 172.21.15.46:0/825500383 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775c00bc20 con 0x7f776c159760 2024-01-27T01:07:34.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.190+0000 7f7771ff2700 1 -- 172.21.15.46:0/825500383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776c159760 msgr2=0x7f776c0b0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:34.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.190+0000 7f7771ff2700 1 --2- 172.21.15.46:0/825500383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776c159760 0x7f776c0b0c70 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f775c0099b0 tx=0x7f775c009250 comp rx=0 tx=0).stop 2024-01-27T01:07:34.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.191+0000 7f7771ff2700 1 -- 172.21.15.46:0/825500383 shutdown_connections 2024-01-27T01:07:34.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.191+0000 7f7771ff2700 1 --2- 172.21.15.46:0/825500383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776c159760 0x7f776c0b0c70 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.191+0000 7f7771ff2700 1 --2- 172.21.15.46:0/825500383 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f776c159b30 0x7f776c0b11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.191+0000 7f7771ff2700 1 -- 172.21.15.46:0/825500383 >> 172.21.15.46:0/825500383 conn(0x7f776c06b630 msgr2=0x7f776c0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:34.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.191+0000 7f7771ff2700 1 -- 172.21.15.46:0/825500383 shutdown_connections 2024-01-27T01:07:34.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.191+0000 7f7771ff2700 1 -- 172.21.15.46:0/825500383 wait complete. 2024-01-27T01:07:34.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.192+0000 7f7771ff2700 1 Processor -- start 2024-01-27T01:07:34.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.192+0000 7f7771ff2700 1 -- start start 2024-01-27T01:07:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f7771ff2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f776c159760 0x7f776c144e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f7771ff2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f776c15afc0 0x7f776c15cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f7771ff2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776c15d2e0 0x7f776c0c2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f7771ff2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776c155c00 con 0x7f776c15d2e0 2024-01-27T01:07:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f7771ff2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776c15d760 con 0x7f776c159760 2024-01-27T01:07:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f7771ff2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776c1457f0 con 0x7f776c15afc0 2024-01-27T01:07:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f776bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776c15d2e0 0x7f776c0c2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f776bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776c15d2e0 0x7f776c0c2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45932/0 (socket says 172.21.15.46:45932) 2024-01-27T01:07:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f776bfff700 1 -- 172.21.15.46:0/2915483263 learned_addr learned my addr 172.21.15.46:0/2915483263 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f776b7fe700 1 --2- 172.21.15.46:0/2915483263 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f776c159760 0x7f776c144e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f776affd700 1 --2- 172.21.15.46:0/2915483263 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f776c15afc0 0x7f776c15cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f776bfff700 1 -- 172.21.15.46:0/2915483263 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f776c15afc0 msgr2=0x7f776c15cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:34.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f776bfff700 1 --2- 172.21.15.46:0/2915483263 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f776c15afc0 0x7f776c15cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f776bfff700 1 -- 172.21.15.46:0/2915483263 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f776c159760 msgr2=0x7f776c144e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f776bfff700 1 --2- 172.21.15.46:0/2915483263 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f776c159760 0x7f776c144e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f776bfff700 1 -- 172.21.15.46:0/2915483263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775c0094f0 con 0x7f776c15d2e0 2024-01-27T01:07:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f776affd700 1 --2- 172.21.15.46:0/2915483263 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f776c15afc0 0x7f776c15cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:07:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f776b7fe700 1 --2- 172.21.15.46:0/2915483263 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f776c159760 0x7f776c144e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:07:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.193+0000 7f776bfff700 1 --2- 172.21.15.46:0/2915483263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776c15d2e0 0x7f776c0c2c30 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f776400c940 tx=0x7f77640016b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.194+0000 7f77537fe700 1 -- 172.21.15.46:0/2915483263 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776400ce80 con 0x7f776c15d2e0 2024-01-27T01:07:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.194+0000 7f77537fe700 1 -- 172.21.15.46:0/2915483263 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776400a7a0 con 0x7f776c15d2e0 2024-01-27T01:07:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.194+0000 7f7771ff2700 1 -- 172.21.15.46:0/2915483263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776c15d9d0 con 0x7f776c15d2e0 2024-01-27T01:07:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.194+0000 7f77537fe700 1 -- 172.21.15.46:0/2915483263 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776400ab20 con 0x7f776c15d2e0 2024-01-27T01:07:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.194+0000 7f7771ff2700 1 -- 172.21.15.46:0/2915483263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f776c145d70 con 0x7f776c15d2e0 2024-01-27T01:07:34.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.195+0000 7f7771ff2700 1 -- 172.21.15.46:0/2915483263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f776c145f90 con 0x7f776c15d2e0 2024-01-27T01:07:34.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.195+0000 7f77537fe700 1 -- 172.21.15.46:0/2915483263 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f7764012420 con 0x7f776c15d2e0 2024-01-27T01:07:34.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.196+0000 7f77537fe700 1 --2- 172.21.15.46:0/2915483263 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7758067170 0x7f7758069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:34.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.196+0000 7f77537fe700 1 -- 172.21.15.46:0/2915483263 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f7764084e50 con 0x7f776c15d2e0 2024-01-27T01:07:34.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.196+0000 7f776b7fe700 1 --2- 172.21.15.46:0/2915483263 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7758067170 0x7f7758069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:34.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.196+0000 7f776b7fe700 1 --2- 172.21.15.46:0/2915483263 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7758067170 0x7f7758069620 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f775c0099b0 tx=0x7f775c017000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:34.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.200+0000 7f77537fe700 1 -- 172.21.15.46:0/2915483263 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7764051b00 con 0x7f776c15d2e0 2024-01-27T01:07:34.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.457+0000 7f7771ff2700 1 -- 172.21.15.46:0/2915483263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f776c04c710 con 0x7f776c15d2e0 2024-01-27T01:07:34.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.457+0000 7f77537fe700 1 -- 172.21.15.46:0/2915483263 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v133) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7764055120 con 0x7f776c15d2e0 2024-01-27T01:07:34.457 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-27T01:07:34.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.459+0000 7f7771ff2700 1 -- 172.21.15.46:0/2915483263 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7758067170 msgr2=0x7f7758069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:34.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.459+0000 7f7771ff2700 1 --2- 172.21.15.46:0/2915483263 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7758067170 0x7f7758069620 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f775c0099b0 tx=0x7f775c017000 comp rx=0 tx=0).stop 2024-01-27T01:07:34.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.459+0000 7f7771ff2700 1 -- 172.21.15.46:0/2915483263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776c15d2e0 msgr2=0x7f776c0c2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:34.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.459+0000 7f7771ff2700 1 --2- 172.21.15.46:0/2915483263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776c15d2e0 0x7f776c0c2c30 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f776400c940 tx=0x7f77640016b0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.459+0000 7f7771ff2700 1 -- 172.21.15.46:0/2915483263 shutdown_connections 2024-01-27T01:07:34.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.459+0000 7f7771ff2700 1 --2- 172.21.15.46:0/2915483263 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7758067170 0x7f7758069620 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.459+0000 7f7771ff2700 1 --2- 172.21.15.46:0/2915483263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776c15d2e0 0x7f776c0c2c30 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.459+0000 7f7771ff2700 1 --2- 172.21.15.46:0/2915483263 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f776c159760 0x7f776c144e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.459+0000 7f7771ff2700 1 --2- 172.21.15.46:0/2915483263 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f776c15afc0 0x7f776c15cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:34.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.459+0000 7f7771ff2700 1 -- 172.21.15.46:0/2915483263 >> 172.21.15.46:0/2915483263 conn(0x7f776c06b630 msgr2=0x7f776c0ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:34.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.460+0000 7f7771ff2700 1 -- 172.21.15.46:0/2915483263 shutdown_connections 2024-01-27T01:07:34.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:34.460+0000 7f7771ff2700 1 -- 172.21.15.46:0/2915483263 wait complete. 2024-01-27T01:07:34.473 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-27T01:07:34.473 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-27T01:07:34.473 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-27T01:07:34.473 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-27T01:07:34.473 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-27T01:07:34.473 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-27T01:07:34.473 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-27T01:07:34.473 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-27T01:07:34.473 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-27T01:07:34.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:07:34.474 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-27T01:07:34.493 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-27T01:07:34.493 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-27T01:07:34.493 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-27T01:07:34.493 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:07:34.493 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-27T01:07:34.493 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:07:34.494 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T01:07:34.494 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:07:34.494 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-27T01:07:34.494 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T01:07:34.494 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:07:34.494 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-27T01:07:34.494 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-27T01:07:34.494 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:07:34.494 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-27T01:07:34.494 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-27T01:07:34.494 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 36476sec preferred_lft 36476sec 2024-01-27T01:07:34.494 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-27T01:07:34.494 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:07:34.494 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T01:07:34.494 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:07:34.495 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-27T01:07:34.495 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-27T01:07:34.495 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:07:34.495 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-27T01:07:34.495 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:07:34.495 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::cce6:a7ff:fe4f:38d/64 scope link 2024-01-27T01:07:34.495 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:07:34.495 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-27T01:07:34.495 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 76:ef:90:4c:9d:6f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-27T01:07:34.495 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::74ef:90ff:fe4c:9d6f/64 scope link 2024-01-27T01:07:34.495 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:07:34.495 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-27T01:07:34.495 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-27T01:07:34.495 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::2c73:afff:fef0:6f39/64 scope link 2024-01-27T01:07:34.496 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:07:34.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:07:34.497 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-27T01:07:34.565 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-27T01:07:34.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:07:34.567 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-27T01:07:34.631 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-27T01:07:34.631 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-27T01:07:34.631 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-27T01:07:34.631 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-27T01:07:34.631 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-27T01:07:34.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:07:34.632 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-27T01:07:34.632 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-27T01:07:34.632 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-27T01:07:34.632 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-27T01:07:34.632 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-27T01:07:34.632 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-27T01:07:34.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:07:34.755 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-27T01:07:34.775 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-27T01:07:34.775 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-27T01:07:34.776 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-27T01:07:34.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:07:34.776 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-01-27T01:07:34.851 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-27T01:07:34.851 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-27T01:07:34.851 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-27T01:07:34.913 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-27T01:07:34.913 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-27T01:07:34.913 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-27T01:07:34.974 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-27T01:07:35.050 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-27T01:07:35.050 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-27T01:07:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-27T01:07:35.126 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-27T01:07:35.126 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:07:35.126 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-27T01:07:35.189 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-27T01:07:35.189 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-27T01:07:35.231 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-27T01:07:35.268 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-27T01:07:35.268 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-27T01:07:35.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-01-27T01:07:35.333+0000 7fc97906b540 -1 init, newargv = 0x5570e71b40e0 newargc=15 2024-01-27T01:07:35.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[117260]: starting ceph client 2024-01-27T01:07:35.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[117260]: starting fuse 2024-01-27T01:07:35.341 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-27T01:07:35.342 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-27T01:07:35.343 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:07:35.343 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-27T01:07:35.403 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-27T01:07:35.403 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-27T01:07:35.403 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-27T01:07:35.403 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-27T01:07:35.403 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-27T01:07:35.403 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-27T01:07:35.404 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-27T01:07:35.404 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:07:35.404 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-27T01:07:35.404 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-27T01:07:35.404 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-27T01:07:35.404 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-27T01:07:35.404 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:07:35.404 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-27T01:07:35.404 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-27T01:07:35.404 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-27T01:07:35.404 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-27T01:07:35.404 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:07:35.404 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-27T01:07:35.404 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-27T01:07:35.405 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-27T01:07:35.405 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-27T01:07:35.405 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-27T01:07:35.405 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-27T01:07:35.405 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-27T01:07:35.405 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-27T01:07:35.405 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:07:35.405 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-01-27T01:07:35.405 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-27T01:07:35.544 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.117260.asok 2024-01-27T01:07:35.725 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.117260.asok 2024-01-27T01:07:35.725 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:07:35.725 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.117260.asok status 2024-01-27T01:07:35.899 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:07:35.899 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-27T01:07:35.899 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-27T01:07:35.899 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-27T01:07:35.899 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-01-27T01:07:35.899 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-27T01:07:35.899 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-27T01:07:35.900 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "117260", 2024-01-27T01:07:35.900 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-27T01:07:35.900 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:35.900 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-27T01:07:35.900 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-27T01:07:35.900 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 11802, 2024-01-27T01:07:35.900 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-27T01:07:35.900 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-27T01:07:35.900 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-27T01:07:35.900 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 11802 2024-01-27T01:07:35.900 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:35.900 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-27T01:07:35.900 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-27T01:07:35.900 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-27T01:07:35.901 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 4103102435 2024-01-27T01:07:35.901 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:35.901 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:35.901 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-27T01:07:35.901 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-27T01:07:35.901 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-27T01:07:35.901 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 4103102435 2024-01-27T01:07:35.901 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:35.901 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.11802 192.168.0.1:0/4103102435", 2024-01-27T01:07:35.901 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/4103102435", 2024-01-27T01:07:35.901 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-27T01:07:35.901 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 133, 2024-01-27T01:07:35.901 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 243, 2024-01-27T01:07:35.902 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-27T01:07:35.902 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-27T01:07:35.902 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-27T01:07:35.902 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:07:35.910 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:07:35.910 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-27T01:07:35.910 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-27T01:07:35.930 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-27T01:07:35.930 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-27T01:07:35.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:07:35.931 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-27T01:07:36.004 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-27T01:07:36.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.185+0000 7f3861bb1700 1 Processor -- start 2024-01-27T01:07:36.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.185+0000 7f3861bb1700 1 -- start start 2024-01-27T01:07:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.186+0000 7f3861bb1700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f385c0af570 0x7f385c0b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.186+0000 7f3861bb1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f385c067930 0x7f385c067d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.186+0000 7f3861bb1700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f385c15ab40 con 0x7f385c159630 2024-01-27T01:07:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.186+0000 7f3861bb1700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f385c068270 con 0x7f385c067930 2024-01-27T01:07:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.186+0000 7f3861bb1700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f385c15a210 con 0x7f385c0af570 2024-01-27T01:07:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.186+0000 7f385bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f385c067930 0x7f385c067d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.186+0000 7f385bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f385c067930 0x7f385c067d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45936/0 (socket says 172.21.15.46:45936) 2024-01-27T01:07:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.186+0000 7f385bfff700 1 -- 172.21.15.46:0/3695213710 learned_addr learned my addr 172.21.15.46:0/3695213710 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.186+0000 7f385b7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f385c159630 0x7f385c159a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45562/0 (socket says 172.21.15.46:45562) 2024-01-27T01:07:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.186+0000 7f3852ffd700 1 --2- 172.21.15.46:0/3695213710 >> v2:172.21.15.106:3300/0 conn(0x7f385c0af570 0x7f385c0b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.187+0000 7f385bfff700 1 -- 172.21.15.46:0/3695213710 >> v2:172.21.15.106:3300/0 conn(0x7f385c0af570 msgr2=0x7f385c0b2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.187+0000 7f385bfff700 1 --2- 172.21.15.46:0/3695213710 >> v2:172.21.15.106:3300/0 conn(0x7f385c0af570 0x7f385c0b2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.187+0000 7f385bfff700 1 -- 172.21.15.46:0/3695213710 >> v1:172.21.15.46:6789/0 conn(0x7f385c159630 legacy=0x7f385c159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.187+0000 7f385bfff700 1 -- 172.21.15.46:0/3695213710 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385c15a4f0 con 0x7f385c067930 2024-01-27T01:07:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.187+0000 7f385bfff700 1 --2- 172.21.15.46:0/3695213710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c067930 0x7f385c067d30 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f385c06d670 tx=0x7f3854009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e2cb15eb7bb1f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.187+0000 7f385affd700 1 -- 172.21.15.46:0/3695213710 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385400e070 con 0x7f385c067930 2024-01-27T01:07:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.187+0000 7f385affd700 1 -- 172.21.15.46:0/3695213710 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385400b950 con 0x7f385c067930 2024-01-27T01:07:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.187+0000 7f385affd700 1 -- 172.21.15.46:0/3695213710 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38540092e0 con 0x7f385c067930 2024-01-27T01:07:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.188+0000 7f3861bb1700 1 -- 172.21.15.46:0/3695213710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c067930 msgr2=0x7f385c067d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.188+0000 7f3861bb1700 1 --2- 172.21.15.46:0/3695213710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c067930 0x7f385c067d30 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f385c06d670 tx=0x7f3854009840 comp rx=0 tx=0).stop 2024-01-27T01:07:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.188+0000 7f3861bb1700 1 -- 172.21.15.46:0/3695213710 shutdown_connections 2024-01-27T01:07:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.188+0000 7f3861bb1700 1 --2- 172.21.15.46:0/3695213710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c067930 0x7f385c067d30 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.188+0000 7f3861bb1700 1 --2- 172.21.15.46:0/3695213710 >> v2:172.21.15.106:3300/0 conn(0x7f385c0af570 0x7f385c0b2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.188+0000 7f3861bb1700 1 -- 172.21.15.46:0/3695213710 >> 172.21.15.46:0/3695213710 conn(0x7f385c069510 msgr2=0x7f385c0aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.188+0000 7f3861bb1700 1 -- 172.21.15.46:0/3695213710 shutdown_connections 2024-01-27T01:07:36.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.188+0000 7f3861bb1700 1 -- 172.21.15.46:0/3695213710 wait complete. 2024-01-27T01:07:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.189+0000 7f3861bb1700 1 Processor -- start 2024-01-27T01:07:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.189+0000 7f3861bb1700 1 -- start start 2024-01-27T01:07:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f3861bb1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f385c067930 0x7f385c149b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f3861bb1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c0af570 0x7f385c14d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f3861bb1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f385c159630 0x7f385c15c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f3861bb1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385c14a0c0 con 0x7f385c0af570 2024-01-27T01:07:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f3861bb1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385c15bc10 con 0x7f385c067930 2024-01-27T01:07:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f3861bb1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385c15b070 con 0x7f385c159630 2024-01-27T01:07:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f3852ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c0af570 0x7f385c14d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f3852ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c0af570 0x7f385c14d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45938/0 (socket says 172.21.15.46:45938) 2024-01-27T01:07:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f3852ffd700 1 -- 172.21.15.46:0/1553421396 learned_addr learned my addr 172.21.15.46:0/1553421396 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f385bfff700 1 --2- 172.21.15.46:0/1553421396 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f385c159630 0x7f385c15c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f385b7fe700 1 --2- 172.21.15.46:0/1553421396 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f385c067930 0x7f385c149b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f3852ffd700 1 -- 172.21.15.46:0/1553421396 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f385c159630 msgr2=0x7f385c15c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f3852ffd700 1 --2- 172.21.15.46:0/1553421396 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f385c159630 0x7f385c15c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f3852ffd700 1 -- 172.21.15.46:0/1553421396 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f385c067930 msgr2=0x7f385c149b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f3852ffd700 1 --2- 172.21.15.46:0/1553421396 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f385c067930 0x7f385c149b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f3852ffd700 1 -- 172.21.15.46:0/1553421396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3854009ae0 con 0x7f385c0af570 2024-01-27T01:07:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f385bfff700 1 --2- 172.21.15.46:0/1553421396 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f385c159630 0x7f385c15c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:07:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f385b7fe700 1 --2- 172.21.15.46:0/1553421396 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f385c067930 0x7f385c149b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:07:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.190+0000 7f3852ffd700 1 --2- 172.21.15.46:0/1553421396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c0af570 0x7f385c14d7e0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f384400d8c0 tx=0x7f3844008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.191+0000 7f3853fff700 1 -- 172.21.15.46:0/1553421396 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384400de60 con 0x7f385c0af570 2024-01-27T01:07:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.191+0000 7f3853fff700 1 -- 172.21.15.46:0/1553421396 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384400b780 con 0x7f385c0af570 2024-01-27T01:07:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.191+0000 7f3861bb1700 1 -- 172.21.15.46:0/1553421396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385c14dd20 con 0x7f385c0af570 2024-01-27T01:07:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.191+0000 7f3853fff700 1 -- 172.21.15.46:0/1553421396 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384400bb20 con 0x7f385c0af570 2024-01-27T01:07:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.191+0000 7f3861bb1700 1 -- 172.21.15.46:0/1553421396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385c15b8a0 con 0x7f385c0af570 2024-01-27T01:07:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.192+0000 7f3861bb1700 1 -- 172.21.15.46:0/1553421396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f385c15bd80 con 0x7f385c0af570 2024-01-27T01:07:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.192+0000 7f3853fff700 1 -- 172.21.15.46:0/1553421396 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f384401e460 con 0x7f385c0af570 2024-01-27T01:07:36.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.193+0000 7f3853fff700 1 --2- 172.21.15.46:0/1553421396 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f383c067170 0x7f383c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:36.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.193+0000 7f3853fff700 1 -- 172.21.15.46:0/1553421396 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f3844085260 con 0x7f385c0af570 2024-01-27T01:07:36.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.193+0000 7f385b7fe700 1 --2- 172.21.15.46:0/1553421396 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f383c067170 0x7f383c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:36.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.194+0000 7f385b7fe700 1 --2- 172.21.15.46:0/1553421396 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f383c067170 0x7f383c069620 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f384c001920 tx=0x7f384c009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:36.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.197+0000 7f3853fff700 1 -- 172.21.15.46:0/1553421396 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3844051f10 con 0x7f385c0af570 2024-01-27T01:07:36.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.451+0000 7f3861bb1700 1 -- 172.21.15.46:0/1553421396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f385c04c710 con 0x7f385c0af570 2024-01-27T01:07:36.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.951+0000 7f3853fff700 1 -- 172.21.15.46:0/1553421396 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v134) v1 ==== 136+0+0 (secure 0 0 0) 0x7f3844055530 con 0x7f385c0af570 2024-01-27T01:07:36.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.954+0000 7f3861bb1700 1 -- 172.21.15.46:0/1553421396 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f383c067170 msgr2=0x7f383c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:36.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.954+0000 7f3861bb1700 1 --2- 172.21.15.46:0/1553421396 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f383c067170 0x7f383c069620 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f384c001920 tx=0x7f384c009040 comp rx=0 tx=0).stop 2024-01-27T01:07:36.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.954+0000 7f3861bb1700 1 -- 172.21.15.46:0/1553421396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c0af570 msgr2=0x7f385c14d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:36.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.954+0000 7f3861bb1700 1 --2- 172.21.15.46:0/1553421396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c0af570 0x7f385c14d7e0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f384400d8c0 tx=0x7f3844008040 comp rx=0 tx=0).stop 2024-01-27T01:07:36.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.954+0000 7f3861bb1700 1 -- 172.21.15.46:0/1553421396 shutdown_connections 2024-01-27T01:07:36.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.954+0000 7f3861bb1700 1 --2- 172.21.15.46:0/1553421396 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f383c067170 0x7f383c069620 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:36.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.954+0000 7f3861bb1700 1 --2- 172.21.15.46:0/1553421396 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f385c159630 0x7f385c15c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:36.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.954+0000 7f3861bb1700 1 --2- 172.21.15.46:0/1553421396 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f385c067930 0x7f385c149b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:36.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.954+0000 7f3861bb1700 1 --2- 172.21.15.46:0/1553421396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c0af570 0x7f385c14d7e0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:36.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.954+0000 7f3861bb1700 1 -- 172.21.15.46:0/1553421396 >> 172.21.15.46:0/1553421396 conn(0x7f385c069510 msgr2=0x7f385c06abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:36.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.955+0000 7f3861bb1700 1 -- 172.21.15.46:0/1553421396 shutdown_connections 2024-01-27T01:07:36.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:36.955+0000 7f3861bb1700 1 -- 172.21.15.46:0/1553421396 wait complete. 2024-01-27T01:07:36.971 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:07:37.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.111+0000 7f46b9cf0700 1 Processor -- start 2024-01-27T01:07:37.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.111+0000 7f46b9cf0700 1 -- start start 2024-01-27T01:07:37.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.112+0000 7f46b9cf0700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f46b4153ec0 0x7f46b4154290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:37.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.112+0000 7f46b9cf0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f46b40b22d0 0x7f46b40b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:37.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.112+0000 7f46b9cf0700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b415b140 con 0x7f46b4157630 2024-01-27T01:07:37.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.112+0000 7f46b9cf0700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46b415a5c0 con 0x7f46b40b22d0 2024-01-27T01:07:37.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.112+0000 7f46b9cf0700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46b415a8a0 con 0x7f46b4153ec0 2024-01-27T01:07:37.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.112+0000 7f46b3fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f46b40b22d0 0x7f46b40b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:37.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.112+0000 7f46b3fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f46b40b22d0 0x7f46b40b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45950/0 (socket says 172.21.15.46:45950) 2024-01-27T01:07:37.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.112+0000 7f46b3fff700 1 -- 172.21.15.46:0/4157528376 learned_addr learned my addr 172.21.15.46:0/4157528376 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:37.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.112+0000 7f46b37fe700 1 --2- 172.21.15.46:0/4157528376 >> v2:172.21.15.106:3300/0 conn(0x7f46b4153ec0 0x7f46b4154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:37.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.112+0000 7f46b3fff700 1 -- 172.21.15.46:0/4157528376 >> v2:172.21.15.106:3300/0 conn(0x7f46b4153ec0 msgr2=0x7f46b4154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:37.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.112+0000 7f46b3fff700 1 --2- 172.21.15.46:0/4157528376 >> v2:172.21.15.106:3300/0 conn(0x7f46b4153ec0 0x7f46b4154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.112+0000 7f46b3fff700 1 -- 172.21.15.46:0/4157528376 >> v1:172.21.15.106:6789/0 conn(0x7f46b4157630 legacy=0x7f46b40b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:37.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.112+0000 7f46b3fff700 1 -- 172.21.15.46:0/4157528376 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b415ab80 con 0x7f46b40b22d0 2024-01-27T01:07:37.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.112+0000 7f46b3fff700 1 --2- 172.21.15.46:0/4157528376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b40b22d0 0x7f46b40b04f0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f46b4064990 tx=0x7f46ac00ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90069e2c84dcdda4 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:37.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.113+0000 7f46b27fc700 1 -- 172.21.15.46:0/4157528376 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac010070 con 0x7f46b40b22d0 2024-01-27T01:07:37.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.113+0000 7f46b27fc700 1 -- 172.21.15.46:0/4157528376 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ac00de90 con 0x7f46b40b22d0 2024-01-27T01:07:37.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.113+0000 7f46b27fc700 1 -- 172.21.15.46:0/4157528376 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac00b320 con 0x7f46b40b22d0 2024-01-27T01:07:37.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.113+0000 7f46b9cf0700 1 -- 172.21.15.46:0/4157528376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b40b22d0 msgr2=0x7f46b40b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:37.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.113+0000 7f46b9cf0700 1 --2- 172.21.15.46:0/4157528376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b40b22d0 0x7f46b40b04f0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f46b4064990 tx=0x7f46ac00ba50 comp rx=0 tx=0).stop 2024-01-27T01:07:37.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.113+0000 7f46b9cf0700 1 -- 172.21.15.46:0/4157528376 shutdown_connections 2024-01-27T01:07:37.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.113+0000 7f46b9cf0700 1 --2- 172.21.15.46:0/4157528376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b40b22d0 0x7f46b40b04f0 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.113+0000 7f46b9cf0700 1 --2- 172.21.15.46:0/4157528376 >> v2:172.21.15.106:3300/0 conn(0x7f46b4153ec0 0x7f46b4154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.113+0000 7f46b9cf0700 1 -- 172.21.15.46:0/4157528376 >> 172.21.15.46:0/4157528376 conn(0x7f46b4069510 msgr2=0x7f46b40aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:37.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.113+0000 7f46b9cf0700 1 -- 172.21.15.46:0/4157528376 shutdown_connections 2024-01-27T01:07:37.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.113+0000 7f46b9cf0700 1 -- 172.21.15.46:0/4157528376 wait complete. 2024-01-27T01:07:37.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.115+0000 7f46b9cf0700 1 Processor -- start 2024-01-27T01:07:37.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.115+0000 7f46b9cf0700 1 -- start start 2024-01-27T01:07:37.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.115+0000 7f46b9cf0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f46b40b22d0 0x7f46b40ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:37.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.115+0000 7f46b9cf0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b4153ec0 0x7f46b40aa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:37.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.115+0000 7f46b9cf0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46b4157630 0x7f46b40aa820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:37.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f46b9cf0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b4151270 con 0x7f46b4153ec0 2024-01-27T01:07:37.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f46b9cf0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b40cbcf0 con 0x7f46b4157630 2024-01-27T01:07:37.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f46b9cf0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b40cbfd0 con 0x7f46b40b22d0 2024-01-27T01:07:37.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f46b2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b4153ec0 0x7f46b40aa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:37.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f46b2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b4153ec0 0x7f46b40aa2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45954/0 (socket says 172.21.15.46:45954) 2024-01-27T01:07:37.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f46b2ffd700 1 -- 172.21.15.46:0/3850944977 learned_addr learned my addr 172.21.15.46:0/3850944977 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:37.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f46b37fe700 1 --2- 172.21.15.46:0/3850944977 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f46b40b22d0 0x7f46b40ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:37.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f46b3fff700 1 --2- 172.21.15.46:0/3850944977 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46b4157630 0x7f46b40aa820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:37.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f46b2ffd700 1 -- 172.21.15.46:0/3850944977 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f46b40b22d0 msgr2=0x7f46b40ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:37.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f46b2ffd700 1 --2- 172.21.15.46:0/3850944977 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f46b40b22d0 0x7f46b40ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f46b2ffd700 1 -- 172.21.15.46:0/3850944977 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46b4157630 msgr2=0x7f46b40aa820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:37.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f46b2ffd700 1 --2- 172.21.15.46:0/3850944977 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46b4157630 0x7f46b40aa820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f46b2ffd700 1 -- 172.21.15.46:0/3850944977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ac00bcf0 con 0x7f46b4153ec0 2024-01-27T01:07:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f46b2ffd700 1 --2- 172.21.15.46:0/3850944977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b4153ec0 0x7f46b40aa2e0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f46a800c4e0 tx=0x7f46a800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f469b7fe700 1 -- 172.21.15.46:0/3850944977 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a8011590 con 0x7f46b4153ec0 2024-01-27T01:07:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f469b7fe700 1 -- 172.21.15.46:0/3850944977 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a8011710 con 0x7f46b4153ec0 2024-01-27T01:07:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f46b9cf0700 1 -- 172.21.15.46:0/3850944977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b40cc2b0 con 0x7f46b4153ec0 2024-01-27T01:07:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f469b7fe700 1 -- 172.21.15.46:0/3850944977 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a8011ad0 con 0x7f46b4153ec0 2024-01-27T01:07:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.116+0000 7f46b9cf0700 1 -- 172.21.15.46:0/3850944977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b40cc870 con 0x7f46b4153ec0 2024-01-27T01:07:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.117+0000 7f46b9cf0700 1 -- 172.21.15.46:0/3850944977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b4157300 con 0x7f46b4153ec0 2024-01-27T01:07:37.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.118+0000 7f469b7fe700 1 -- 172.21.15.46:0/3850944977 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f46a801b3b0 con 0x7f46b4153ec0 2024-01-27T01:07:37.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.118+0000 7f469b7fe700 1 --2- 172.21.15.46:0/3850944977 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f46a0067170 0x7f46a0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:37.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.118+0000 7f469b7fe700 1 -- 172.21.15.46:0/3850944977 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f46a8083fc0 con 0x7f46b4153ec0 2024-01-27T01:07:37.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.118+0000 7f46b37fe700 1 --2- 172.21.15.46:0/3850944977 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f46a0067170 0x7f46a0069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:37.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.119+0000 7f46b37fe700 1 --2- 172.21.15.46:0/3850944977 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f46a0067170 0x7f46a0069620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f46a4006fd0 tx=0x7f46a4008040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:37.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.123+0000 7f469b7fe700 1 -- 172.21.15.46:0/3850944977 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f46a8050c70 con 0x7f46b4153ec0 2024-01-27T01:07:37.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.328+0000 7f46b9cf0700 1 -- 172.21.15.46:0/3850944977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46b40cca90 con 0x7f46b4153ec0 2024-01-27T01:07:37.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.329+0000 7f469b7fe700 1 -- 172.21.15.46:0/3850944977 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v243) v1 ==== 74+0+16325 (secure 0 0 0) 0x7f46a8017070 con 0x7f46b4153ec0 2024-01-27T01:07:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:37.330 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":243,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:07:28.728011+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:07:25.886791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-01-27T01:07:26.340041+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:07:37.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.331+0000 7f46b9cf0700 1 -- 172.21.15.46:0/3850944977 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f46a0067170 msgr2=0x7f46a0069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:37.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.331+0000 7f46b9cf0700 1 --2- 172.21.15.46:0/3850944977 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f46a0067170 0x7f46a0069620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f46a4006fd0 tx=0x7f46a4008040 comp rx=0 tx=0).stop 2024-01-27T01:07:37.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.331+0000 7f46b9cf0700 1 -- 172.21.15.46:0/3850944977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b4153ec0 msgr2=0x7f46b40aa2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:37.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.331+0000 7f46b9cf0700 1 --2- 172.21.15.46:0/3850944977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b4153ec0 0x7f46b40aa2e0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f46a800c4e0 tx=0x7f46a800c7f0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.331+0000 7f46b9cf0700 1 -- 172.21.15.46:0/3850944977 shutdown_connections 2024-01-27T01:07:37.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.331+0000 7f46b9cf0700 1 --2- 172.21.15.46:0/3850944977 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f46a0067170 0x7f46a0069620 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.331+0000 7f46b9cf0700 1 --2- 172.21.15.46:0/3850944977 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46b4157630 0x7f46b40aa820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.331+0000 7f46b9cf0700 1 --2- 172.21.15.46:0/3850944977 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f46b40b22d0 0x7f46b40ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.331+0000 7f46b9cf0700 1 --2- 172.21.15.46:0/3850944977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b4153ec0 0x7f46b40aa2e0 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.331+0000 7f46b9cf0700 1 -- 172.21.15.46:0/3850944977 >> 172.21.15.46:0/3850944977 conn(0x7f46b4069510 msgr2=0x7f46b40b03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:37.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.331+0000 7f46b9cf0700 1 -- 172.21.15.46:0/3850944977 shutdown_connections 2024-01-27T01:07:37.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.331+0000 7f46b9cf0700 1 -- 172.21.15.46:0/3850944977 wait complete. 2024-01-27T01:07:37.345 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-27T01:07:25.886791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '242', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2024-01-27T01:07:26.340041+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '243', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-27T01:07:37.345 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-27T01:07:37.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.480+0000 7f3a5d0e8700 1 Processor -- start 2024-01-27T01:07:37.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.480+0000 7f3a5d0e8700 1 -- start start 2024-01-27T01:07:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.481+0000 7f3a5d0e8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a581598e0 0x7f3a580c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.481+0000 7f3a5d0e8700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3a580c9120 0x7f3a580cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.481+0000 7f3a5d0e8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a58069260 con 0x7f3a58159510 2024-01-27T01:07:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.481+0000 7f3a5d0e8700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a580ca930 con 0x7f3a580c9120 2024-01-27T01:07:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.481+0000 7f3a5d0e8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a58159d50 con 0x7f3a581598e0 2024-01-27T01:07:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.481+0000 7f3a56d9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3a58159510 0x7f3a580ca220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45576/0 (socket says 172.21.15.46:45576) 2024-01-27T01:07:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.481+0000 7f3a56d9d700 1 -- 172.21.15.46:0/3456514631 learned_addr learned my addr 172.21.15.46:0/3456514631 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.481+0000 7f3a5759e700 1 --2- 172.21.15.46:0/3456514631 >> v2:172.21.15.106:3300/0 conn(0x7f3a580c9120 0x7f3a580cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.481+0000 7f3a5659c700 1 --2- 172.21.15.46:0/3456514631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a581598e0 0x7f3a580c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.482+0000 7f3a55d9b700 1 -- 172.21.15.46:0/3456514631 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233325426 0 0) 0x7f3a58069260 con 0x7f3a58159510 2024-01-27T01:07:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.482+0000 7f3a55d9b700 1 -- 172.21.15.46:0/3456514631 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a40003570 con 0x7f3a58159510 2024-01-27T01:07:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.482+0000 7f3a5759e700 1 -- 172.21.15.46:0/3456514631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a581598e0 msgr2=0x7f3a580c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.482+0000 7f3a5759e700 1 --2- 172.21.15.46:0/3456514631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a581598e0 0x7f3a580c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.482+0000 7f3a5759e700 1 -- 172.21.15.46:0/3456514631 >> v1:172.21.15.46:6789/0 conn(0x7f3a58159510 legacy=0x7f3a580ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.482+0000 7f3a55d9b700 1 -- 172.21.15.46:0/3456514631 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2824613258 0 0) 0x7f3a40003570 con 0x7f3a58159510 2024-01-27T01:07:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.482+0000 7f3a5759e700 1 -- 172.21.15.46:0/3456514631 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5815d670 con 0x7f3a580c9120 2024-01-27T01:07:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.482+0000 7f3a5759e700 1 --2- 172.21.15.46:0/3456514631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a580c9120 0x7f3a580cc380 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009e60 tx=0x7f3a50009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4ad1eb110c2dcce server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.482+0000 7f3a55d9b700 1 -- 172.21.15.46:0/3456514631 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5000e070 con 0x7f3a580c9120 2024-01-27T01:07:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.482+0000 7f3a55d9b700 1 -- 172.21.15.46:0/3456514631 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5000b7a0 con 0x7f3a580c9120 2024-01-27T01:07:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.482+0000 7f3a55d9b700 1 -- 172.21.15.46:0/3456514631 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c001470 con 0x7f3a580c9120 2024-01-27T01:07:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.483+0000 7f3a5d0e8700 1 -- 172.21.15.46:0/3456514631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a580c9120 msgr2=0x7f3a580cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.483+0000 7f3a5d0e8700 1 --2- 172.21.15.46:0/3456514631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a580c9120 0x7f3a580cc380 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009e60 tx=0x7f3a50009780 comp rx=0 tx=0).stop 2024-01-27T01:07:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.483+0000 7f3a5d0e8700 1 -- 172.21.15.46:0/3456514631 shutdown_connections 2024-01-27T01:07:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.483+0000 7f3a5d0e8700 1 --2- 172.21.15.46:0/3456514631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a580c9120 0x7f3a580cc380 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.483+0000 7f3a5d0e8700 1 --2- 172.21.15.46:0/3456514631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a581598e0 0x7f3a580c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.483+0000 7f3a5d0e8700 1 -- 172.21.15.46:0/3456514631 >> 172.21.15.46:0/3456514631 conn(0x7f3a5806bd90 msgr2=0x7f3a5806c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.483+0000 7f3a5d0e8700 1 -- 172.21.15.46:0/3456514631 shutdown_connections 2024-01-27T01:07:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.483+0000 7f3a5d0e8700 1 -- 172.21.15.46:0/3456514631 wait complete. 2024-01-27T01:07:37.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.484+0000 7f3a5d0e8700 1 Processor -- start 2024-01-27T01:07:37.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.484+0000 7f3a5d0e8700 1 -- start start 2024-01-27T01:07:37.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.485+0000 7f3a5d0e8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a580c9120 0x7f3a581515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:37.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.485+0000 7f3a5d0e8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a58159510 0x7f3a5815c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:37.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.485+0000 7f3a5d0e8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a581598e0 0x7f3a5815c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.485+0000 7f3a5d0e8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a580b2210 con 0x7f3a58159510 2024-01-27T01:07:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.485+0000 7f3a5d0e8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a58151b00 con 0x7f3a581598e0 2024-01-27T01:07:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.485+0000 7f3a5d0e8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a580b1340 con 0x7f3a580c9120 2024-01-27T01:07:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.485+0000 7f3a5659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a58159510 0x7f3a5815c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.485+0000 7f3a5659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a58159510 0x7f3a5815c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45968/0 (socket says 172.21.15.46:45968) 2024-01-27T01:07:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.485+0000 7f3a5659c700 1 -- 172.21.15.46:0/824115778 learned_addr learned my addr 172.21.15.46:0/824115778 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.485+0000 7f3a56d9d700 1 --2- 172.21.15.46:0/824115778 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a580c9120 0x7f3a581515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.485+0000 7f3a5759e700 1 --2- 172.21.15.46:0/824115778 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a581598e0 0x7f3a5815c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.486+0000 7f3a5659c700 1 -- 172.21.15.46:0/824115778 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a580c9120 msgr2=0x7f3a581515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.486+0000 7f3a5659c700 1 --2- 172.21.15.46:0/824115778 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a580c9120 0x7f3a581515c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.486+0000 7f3a5659c700 1 -- 172.21.15.46:0/824115778 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a581598e0 msgr2=0x7f3a5815c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.486+0000 7f3a5659c700 1 --2- 172.21.15.46:0/824115778 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a581598e0 0x7f3a5815c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.486+0000 7f3a5659c700 1 -- 172.21.15.46:0/824115778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a50009a20 con 0x7f3a58159510 2024-01-27T01:07:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.486+0000 7f3a5659c700 1 --2- 172.21.15.46:0/824115778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a58159510 0x7f3a5815c110 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800e930 tx=0x7f3a480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.486+0000 7f3a3effd700 1 -- 172.21.15.46:0/824115778 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4800eed0 con 0x7f3a58159510 2024-01-27T01:07:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.486+0000 7f3a3effd700 1 -- 172.21.15.46:0/824115778 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a4800c850 con 0x7f3a58159510 2024-01-27T01:07:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.486+0000 7f3a5d0e8700 1 -- 172.21.15.46:0/824115778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a580b1600 con 0x7f3a58159510 2024-01-27T01:07:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.486+0000 7f3a3effd700 1 -- 172.21.15.46:0/824115778 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4800cbf0 con 0x7f3a58159510 2024-01-27T01:07:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.486+0000 7f3a5d0e8700 1 -- 172.21.15.46:0/824115778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a580b1bc0 con 0x7f3a58159510 2024-01-27T01:07:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.487+0000 7f3a5d0e8700 1 -- 172.21.15.46:0/824115778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a580b23c0 con 0x7f3a58159510 2024-01-27T01:07:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.487+0000 7f3a3effd700 1 -- 172.21.15.46:0/824115778 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f3a48013470 con 0x7f3a58159510 2024-01-27T01:07:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.488+0000 7f3a3effd700 1 --2- 172.21.15.46:0/824115778 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3a44067170 0x7f3a44069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.488+0000 7f3a3effd700 1 -- 172.21.15.46:0/824115778 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f3a48084db0 con 0x7f3a58159510 2024-01-27T01:07:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.488+0000 7f3a56d9d700 1 --2- 172.21.15.46:0/824115778 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3a44067170 0x7f3a44069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.489+0000 7f3a56d9d700 1 --2- 172.21.15.46:0/824115778 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3a44067170 0x7f3a44069620 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c005c20 tx=0x7f3a4c001120 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:37.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.492+0000 7f3a3effd700 1 -- 172.21.15.46:0/824115778 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3a580b23c0 con 0x7f3a58159510 2024-01-27T01:07:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.697+0000 7f3a5d0e8700 1 -- 172.21.15.46:0/824115778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3a5804c710 con 0x7f3a58159510 2024-01-27T01:07:37.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.698+0000 7f3a3effd700 1 -- 172.21.15.46:0/824115778 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v243) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3a48051a60 con 0x7f3a58159510 2024-01-27T01:07:37.700 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-27T01:07:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.700+0000 7f3a5d0e8700 1 -- 172.21.15.46:0/824115778 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3a44067170 msgr2=0x7f3a44069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.700+0000 7f3a5d0e8700 1 --2- 172.21.15.46:0/824115778 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3a44067170 0x7f3a44069620 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c005c20 tx=0x7f3a4c001120 comp rx=0 tx=0).stop 2024-01-27T01:07:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.700+0000 7f3a5d0e8700 1 -- 172.21.15.46:0/824115778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a58159510 msgr2=0x7f3a5815c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.700+0000 7f3a5d0e8700 1 --2- 172.21.15.46:0/824115778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a58159510 0x7f3a5815c110 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800e930 tx=0x7f3a480091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.700+0000 7f3a5d0e8700 1 -- 172.21.15.46:0/824115778 shutdown_connections 2024-01-27T01:07:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.700+0000 7f3a5d0e8700 1 --2- 172.21.15.46:0/824115778 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3a44067170 0x7f3a44069620 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.700+0000 7f3a5d0e8700 1 --2- 172.21.15.46:0/824115778 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a581598e0 0x7f3a5815c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.700+0000 7f3a5d0e8700 1 --2- 172.21.15.46:0/824115778 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3a580c9120 0x7f3a581515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.700+0000 7f3a5d0e8700 1 --2- 172.21.15.46:0/824115778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a58159510 0x7f3a5815c110 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.700+0000 7f3a5d0e8700 1 -- 172.21.15.46:0/824115778 >> 172.21.15.46:0/824115778 conn(0x7f3a5806bd90 msgr2=0x7f3a5806c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.700+0000 7f3a5d0e8700 1 -- 172.21.15.46:0/824115778 shutdown_connections 2024-01-27T01:07:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.700+0000 7f3a5d0e8700 1 -- 172.21.15.46:0/824115778 wait complete. 2024-01-27T01:07:37.714 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-27T01:07:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.848+0000 7f1ffd2d3700 1 Processor -- start 2024-01-27T01:07:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.848+0000 7f1ffd2d3700 1 -- start start 2024-01-27T01:07:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.849+0000 7f1ffd2d3700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1ff80aa9b0 0x7f1ff80a98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.849+0000 7f1ffd2d3700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff806ba40 con 0x7f1ff81565e0 2024-01-27T01:07:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.849+0000 7f1ffd2d3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff806e470 con 0x7f1ff81569b0 2024-01-27T01:07:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.849+0000 7f1ffd2d3700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff80acbb0 con 0x7f1ff80aa9b0 2024-01-27T01:07:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.849+0000 7f1ff6ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1ff81565e0 0x7f1ff80ac4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45582/0 (socket says 172.21.15.46:45582) 2024-01-27T01:07:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.849+0000 7f1ff6ffd700 1 -- 172.21.15.46:0/738992735 learned_addr learned my addr 172.21.15.46:0/738992735 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.849+0000 7f1ff77fe700 1 --2- 172.21.15.46:0/738992735 >> v2:172.21.15.106:3300/0 conn(0x7f1ff80aa9b0 0x7f1ff80a98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.849+0000 7f1ff5ffb700 1 -- 172.21.15.46:0/738992735 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126005035 0 0) 0x7f1ff806ba40 con 0x7f1ff81565e0 2024-01-27T01:07:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.849+0000 7f1ff5ffb700 1 -- 172.21.15.46:0/738992735 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe0003570 con 0x7f1ff81565e0 2024-01-27T01:07:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.850+0000 7f1ff77fe700 1 -- 172.21.15.46:0/738992735 >> v1:172.21.15.106:6789/0 conn(0x7f1ff81569b0 legacy=0x7f1ff80aa2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.850+0000 7f1ff77fe700 1 -- 172.21.15.46:0/738992735 >> v1:172.21.15.46:6789/0 conn(0x7f1ff81565e0 legacy=0x7f1ff80ac4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.850+0000 7f1ff5ffb700 1 -- 172.21.15.46:0/738992735 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431377679 0 0) 0x7f1ff806e470 con 0x7f1ff81569b0 2024-01-27T01:07:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.850+0000 7f1ff77fe700 1 -- 172.21.15.46:0/738992735 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff806db60 con 0x7f1ff80aa9b0 2024-01-27T01:07:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.850+0000 7f1ff77fe700 1 --2- 172.21.15.46:0/738992735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ff80aa9b0 0x7f1ff80a98e0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0009e60 tx=0x7f1ff0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a83f15626a304fbe server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.850+0000 7f1ff5ffb700 1 -- 172.21.15.46:0/738992735 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff000e070 con 0x7f1ff80aa9b0 2024-01-27T01:07:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.850+0000 7f1ff5ffb700 1 -- 172.21.15.46:0/738992735 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff000bb60 con 0x7f1ff80aa9b0 2024-01-27T01:07:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.850+0000 7f1ff5ffb700 1 -- 172.21.15.46:0/738992735 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0012430 con 0x7f1ff80aa9b0 2024-01-27T01:07:37.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.851+0000 7f1ffd2d3700 1 -- 172.21.15.46:0/738992735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ff80aa9b0 msgr2=0x7f1ff80a98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:37.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.851+0000 7f1ffd2d3700 1 --2- 172.21.15.46:0/738992735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ff80aa9b0 0x7f1ff80a98e0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0009e60 tx=0x7f1ff0009780 comp rx=0 tx=0).stop 2024-01-27T01:07:37.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.851+0000 7f1ffd2d3700 1 -- 172.21.15.46:0/738992735 shutdown_connections 2024-01-27T01:07:37.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.851+0000 7f1ffd2d3700 1 --2- 172.21.15.46:0/738992735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ff80aa9b0 0x7f1ff80a98e0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.851+0000 7f1ffd2d3700 1 -- 172.21.15.46:0/738992735 >> 172.21.15.46:0/738992735 conn(0x7f1ff8067930 msgr2=0x7f1ff8067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:37.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.851+0000 7f1ffd2d3700 1 -- 172.21.15.46:0/738992735 shutdown_connections 2024-01-27T01:07:37.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.851+0000 7f1ffd2d3700 1 -- 172.21.15.46:0/738992735 wait complete. 2024-01-27T01:07:37.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.852+0000 7f1ffd2d3700 1 Processor -- start 2024-01-27T01:07:37.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.852+0000 7f1ffd2d3700 1 -- start start 2024-01-27T01:07:37.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ffd2d3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ff80aa9b0 0x7f1ff806a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:37.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ffd2d3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ff81565e0 0x7f1ff8099570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:37.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ffd2d3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ff81569b0 0x7f1ff8144fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:37.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ffd2d3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff8159f40 con 0x7f1ff81569b0 2024-01-27T01:07:37.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ffd2d3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff8159330 con 0x7f1ff80aa9b0 2024-01-27T01:07:37.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ffd2d3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff8159610 con 0x7f1ff81565e0 2024-01-27T01:07:37.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ff77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ff81569b0 0x7f1ff8144fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:37.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ff77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ff81569b0 0x7f1ff8144fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45978/0 (socket says 172.21.15.46:45978) 2024-01-27T01:07:37.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ff77fe700 1 -- 172.21.15.46:0/2922394572 learned_addr learned my addr 172.21.15.46:0/2922394572 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:37.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ff6ffd700 1 --2- 172.21.15.46:0/2922394572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ff80aa9b0 0x7f1ff806a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:37.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ff67fc700 1 --2- 172.21.15.46:0/2922394572 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ff81565e0 0x7f1ff8099570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:37.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ff77fe700 1 -- 172.21.15.46:0/2922394572 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ff81565e0 msgr2=0x7f1ff8099570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:37.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ff77fe700 1 --2- 172.21.15.46:0/2922394572 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ff81565e0 0x7f1ff8099570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ff77fe700 1 -- 172.21.15.46:0/2922394572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ff80aa9b0 msgr2=0x7f1ff806a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:37.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ff77fe700 1 --2- 172.21.15.46:0/2922394572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ff80aa9b0 0x7f1ff806a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:37.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ff77fe700 1 -- 172.21.15.46:0/2922394572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff0009a20 con 0x7f1ff81569b0 2024-01-27T01:07:37.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ff67fc700 1 --2- 172.21.15.46:0/2922394572 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ff81565e0 0x7f1ff8099570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:07:37.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.853+0000 7f1ff77fe700 1 --2- 172.21.15.46:0/2922394572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ff81569b0 0x7f1ff8144fd0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8064090 tx=0x7f1ff0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:37.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.854+0000 7f1fdeffd700 1 -- 172.21.15.46:0/2922394572 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff001c610 con 0x7f1ff81569b0 2024-01-27T01:07:37.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.854+0000 7f1fdeffd700 1 -- 172.21.15.46:0/2922394572 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff001c790 con 0x7f1ff81569b0 2024-01-27T01:07:37.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.854+0000 7f1ffd2d3700 1 -- 172.21.15.46:0/2922394572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff81598f0 con 0x7f1ff81569b0 2024-01-27T01:07:37.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.854+0000 7f1fdeffd700 1 -- 172.21.15.46:0/2922394572 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff001cb50 con 0x7f1ff81569b0 2024-01-27T01:07:37.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.854+0000 7f1ffd2d3700 1 -- 172.21.15.46:0/2922394572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff8158fc0 con 0x7f1ff81569b0 2024-01-27T01:07:37.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.855+0000 7f1ffd2d3700 1 -- 172.21.15.46:0/2922394572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff815a0f0 con 0x7f1ff81569b0 2024-01-27T01:07:37.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.855+0000 7f1fdeffd700 1 -- 172.21.15.46:0/2922394572 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f1ff001b400 con 0x7f1ff81569b0 2024-01-27T01:07:37.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.856+0000 7f1fdeffd700 1 --2- 172.21.15.46:0/2922394572 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1fe4067170 0x7f1fe4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:37.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.856+0000 7f1fdeffd700 1 -- 172.21.15.46:0/2922394572 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f1ff0086e10 con 0x7f1ff81569b0 2024-01-27T01:07:37.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.856+0000 7f1ff6ffd700 1 --2- 172.21.15.46:0/2922394572 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1fe4067170 0x7f1fe4069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:37.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.856+0000 7f1ff6ffd700 1 --2- 172.21.15.46:0/2922394572 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1fe4067170 0x7f1fe4069620 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8000c00 tx=0x7f1fe8009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:37.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:37.860+0000 7f1fdeffd700 1 -- 172.21.15.46:0/2922394572 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1ff815a0f0 con 0x7f1ff81569b0 2024-01-27T01:07:38.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.063+0000 7f1ffd2d3700 1 -- 172.21.15.46:0/2922394572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1ff815a0f0 con 0x7f1ff81569b0 2024-01-27T01:07:38.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.063+0000 7f1fdeffd700 1 -- 172.21.15.46:0/2922394572 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v243) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1ff002b020 con 0x7f1ff81569b0 2024-01-27T01:07:38.064 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-27T01:07:38.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.066+0000 7f1ffd2d3700 1 -- 172.21.15.46:0/2922394572 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1fe4067170 msgr2=0x7f1fe4069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.066+0000 7f1ffd2d3700 1 --2- 172.21.15.46:0/2922394572 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1fe4067170 0x7f1fe4069620 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8000c00 tx=0x7f1fe8009040 comp rx=0 tx=0).stop 2024-01-27T01:07:38.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.066+0000 7f1ffd2d3700 1 -- 172.21.15.46:0/2922394572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ff81569b0 msgr2=0x7f1ff8144fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.066+0000 7f1ffd2d3700 1 --2- 172.21.15.46:0/2922394572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ff81569b0 0x7f1ff8144fd0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8064090 tx=0x7f1ff0019000 comp rx=0 tx=0).stop 2024-01-27T01:07:38.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.066+0000 7f1ffd2d3700 1 -- 172.21.15.46:0/2922394572 shutdown_connections 2024-01-27T01:07:38.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.066+0000 7f1ffd2d3700 1 --2- 172.21.15.46:0/2922394572 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1fe4067170 0x7f1fe4069620 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.066+0000 7f1ffd2d3700 1 --2- 172.21.15.46:0/2922394572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ff81569b0 0x7f1ff8144fd0 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.066+0000 7f1ffd2d3700 1 --2- 172.21.15.46:0/2922394572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ff80aa9b0 0x7f1ff806a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.066+0000 7f1ffd2d3700 1 --2- 172.21.15.46:0/2922394572 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ff81565e0 0x7f1ff8099570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.066+0000 7f1ffd2d3700 1 -- 172.21.15.46:0/2922394572 >> 172.21.15.46:0/2922394572 conn(0x7f1ff8067930 msgr2=0x7f1ff80acd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:38.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.066+0000 7f1ffd2d3700 1 -- 172.21.15.46:0/2922394572 shutdown_connections 2024-01-27T01:07:38.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.066+0000 7f1ffd2d3700 1 -- 172.21.15.46:0/2922394572 wait complete. 2024-01-27T01:07:38.079 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-27T01:07:38.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.214+0000 7f9c73041700 1 Processor -- start 2024-01-27T01:07:38.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.214+0000 7f9c73041700 1 -- start start 2024-01-27T01:07:38.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.215+0000 7f9c73041700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9c6c156570 0x7f9c6c06c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.215+0000 7f9c73041700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6c0c8b90 con 0x7f9c6c156940 2024-01-27T01:07:38.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.215+0000 7f9c73041700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6c0c8e50 con 0x7f9c6c06f9a0 2024-01-27T01:07:38.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.215+0000 7f9c73041700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c6c06d0b0 con 0x7f9c6c156570 2024-01-27T01:07:38.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.215+0000 7f9c70ddd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9c6c156570 0x7f9c6c06c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.215+0000 7f9c6bfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9c6c156940 0x7f9c6c0c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45590/0 (socket says 172.21.15.46:45590) 2024-01-27T01:07:38.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.215+0000 7f9c6bfff700 1 -- 172.21.15.46:0/699784869 learned_addr learned my addr 172.21.15.46:0/699784869 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:38.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.216+0000 7f9c70ddd700 1 -- 172.21.15.46:0/699784869 >> v1:172.21.15.106:6789/0 conn(0x7f9c6c06f9a0 legacy=0x7f9c6c06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.216+0000 7f9c70ddd700 1 -- 172.21.15.46:0/699784869 >> v1:172.21.15.46:6789/0 conn(0x7f9c6c156940 legacy=0x7f9c6c0c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.216+0000 7f9c70ddd700 1 -- 172.21.15.46:0/699784869 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6c0c4c40 con 0x7f9c6c156570 2024-01-27T01:07:38.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.216+0000 7f9c6b7fe700 1 -- 172.21.15.46:0/699784869 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554276614 0 0) 0x7f9c6c0c8b90 con 0x7f9c6c156940 2024-01-27T01:07:38.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.216+0000 7f9c70ddd700 1 --2- 172.21.15.46:0/699784869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c6c156570 0x7f9c6c06c110 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f9c600098e0 tx=0x7f9c600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6801cc84fc5b3d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:38.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.216+0000 7f9c6b7fe700 1 -- 172.21.15.46:0/699784869 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c60009e80 con 0x7f9c6c156570 2024-01-27T01:07:38.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.216+0000 7f9c6b7fe700 1 -- 172.21.15.46:0/699784869 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6000b860 con 0x7f9c6c156570 2024-01-27T01:07:38.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.216+0000 7f9c6b7fe700 1 -- 172.21.15.46:0/699784869 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6000bbe0 con 0x7f9c6c156570 2024-01-27T01:07:38.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.217+0000 7f9c73041700 1 -- 172.21.15.46:0/699784869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c6c156570 msgr2=0x7f9c6c06c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.217+0000 7f9c73041700 1 --2- 172.21.15.46:0/699784869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c6c156570 0x7f9c6c06c110 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f9c600098e0 tx=0x7f9c600091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.217+0000 7f9c73041700 1 -- 172.21.15.46:0/699784869 shutdown_connections 2024-01-27T01:07:38.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.217+0000 7f9c73041700 1 --2- 172.21.15.46:0/699784869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c6c156570 0x7f9c6c06c110 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.217+0000 7f9c73041700 1 -- 172.21.15.46:0/699784869 >> 172.21.15.46:0/699784869 conn(0x7f9c6c0cdf50 msgr2=0x7f9c6c0c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:38.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.217+0000 7f9c73041700 1 -- 172.21.15.46:0/699784869 shutdown_connections 2024-01-27T01:07:38.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.217+0000 7f9c73041700 1 -- 172.21.15.46:0/699784869 wait complete. 2024-01-27T01:07:38.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.218+0000 7f9c73041700 1 Processor -- start 2024-01-27T01:07:38.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.218+0000 7f9c73041700 1 -- start start 2024-01-27T01:07:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.219+0000 7f9c73041700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c6c06f9a0 0x7f9c6c14e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.219+0000 7f9c73041700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c6c156570 0x7f9c6c0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.219+0000 7f9c73041700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c6c156940 0x7f9c6c0b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.219+0000 7f9c73041700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c14ecf0 con 0x7f9c6c06f9a0 2024-01-27T01:07:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.219+0000 7f9c73041700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c0b17f0 con 0x7f9c6c156570 2024-01-27T01:07:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.219+0000 7f9c73041700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c1580b0 con 0x7f9c6c156940 2024-01-27T01:07:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.219+0000 7f9c70ddd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c6c06f9a0 0x7f9c6c14e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.219+0000 7f9c70ddd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c6c06f9a0 0x7f9c6c14e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45986/0 (socket says 172.21.15.46:45986) 2024-01-27T01:07:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.219+0000 7f9c70ddd700 1 -- 172.21.15.46:0/1781293909 learned_addr learned my addr 172.21.15.46:0/1781293909 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.219+0000 7f9c6bfff700 1 --2- 172.21.15.46:0/1781293909 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c6c156570 0x7f9c6c0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.219+0000 7f9c715de700 1 --2- 172.21.15.46:0/1781293909 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c6c156940 0x7f9c6c0b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.219+0000 7f9c70ddd700 1 -- 172.21.15.46:0/1781293909 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c6c156940 msgr2=0x7f9c6c0b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.219+0000 7f9c70ddd700 1 --2- 172.21.15.46:0/1781293909 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c6c156940 0x7f9c6c0b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.219+0000 7f9c70ddd700 1 -- 172.21.15.46:0/1781293909 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c6c156570 msgr2=0x7f9c6c0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.219+0000 7f9c70ddd700 1 --2- 172.21.15.46:0/1781293909 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c6c156570 0x7f9c6c0b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.219+0000 7f9c70ddd700 1 -- 172.21.15.46:0/1781293909 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c60009420 con 0x7f9c6c06f9a0 2024-01-27T01:07:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.220+0000 7f9c70ddd700 1 --2- 172.21.15.46:0/1781293909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c6c06f9a0 0x7f9c6c14e160 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c06e800 tx=0x7f9c60017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.220+0000 7f9c68ff9700 1 -- 172.21.15.46:0/1781293909 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6001f070 con 0x7f9c6c06f9a0 2024-01-27T01:07:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.220+0000 7f9c68ff9700 1 -- 172.21.15.46:0/1781293909 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c60011420 con 0x7f9c6c06f9a0 2024-01-27T01:07:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.220+0000 7f9c68ff9700 1 -- 172.21.15.46:0/1781293909 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6000bbe0 con 0x7f9c6c06f9a0 2024-01-27T01:07:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.220+0000 7f9c73041700 1 -- 172.21.15.46:0/1781293909 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c6c158390 con 0x7f9c6c06f9a0 2024-01-27T01:07:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.220+0000 7f9c73041700 1 -- 172.21.15.46:0/1781293909 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c6c158950 con 0x7f9c6c06f9a0 2024-01-27T01:07:38.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.221+0000 7f9c73041700 1 -- 172.21.15.46:0/1781293909 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c6c04c710 con 0x7f9c6c06f9a0 2024-01-27T01:07:38.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.221+0000 7f9c68ff9700 1 -- 172.21.15.46:0/1781293909 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f9c60011a50 con 0x7f9c6c06f9a0 2024-01-27T01:07:38.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.222+0000 7f9c68ff9700 1 --2- 172.21.15.46:0/1781293909 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9c50067170 0x7f9c50069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.222+0000 7f9c68ff9700 1 -- 172.21.15.46:0/1781293909 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f9c60086120 con 0x7f9c6c06f9a0 2024-01-27T01:07:38.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.222+0000 7f9c6bfff700 1 --2- 172.21.15.46:0/1781293909 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9c50067170 0x7f9c50069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.223+0000 7f9c6bfff700 1 --2- 172.21.15.46:0/1781293909 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9c50067170 0x7f9c50069620 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9c580011e0 tx=0x7f9c58009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:38.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.226+0000 7f9c68ff9700 1 -- 172.21.15.46:0/1781293909 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9c60052dd0 con 0x7f9c6c06f9a0 2024-01-27T01:07:38.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.430+0000 7f9c73041700 1 -- 172.21.15.46:0/1781293909 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9c6c06a530 con 0x7f9c6c06f9a0 2024-01-27T01:07:38.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.430+0000 7f9c68ff9700 1 -- 172.21.15.46:0/1781293909 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v243) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9c60015020 con 0x7f9c6c06f9a0 2024-01-27T01:07:38.431 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-27T01:07:38.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.433+0000 7f9c73041700 1 -- 172.21.15.46:0/1781293909 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9c50067170 msgr2=0x7f9c50069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.433+0000 7f9c73041700 1 --2- 172.21.15.46:0/1781293909 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9c50067170 0x7f9c50069620 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9c580011e0 tx=0x7f9c58009040 comp rx=0 tx=0).stop 2024-01-27T01:07:38.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.433+0000 7f9c73041700 1 -- 172.21.15.46:0/1781293909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c6c06f9a0 msgr2=0x7f9c6c14e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.433+0000 7f9c73041700 1 --2- 172.21.15.46:0/1781293909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c6c06f9a0 0x7f9c6c14e160 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c06e800 tx=0x7f9c60017000 comp rx=0 tx=0).stop 2024-01-27T01:07:38.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.433+0000 7f9c73041700 1 -- 172.21.15.46:0/1781293909 shutdown_connections 2024-01-27T01:07:38.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.433+0000 7f9c73041700 1 --2- 172.21.15.46:0/1781293909 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9c50067170 0x7f9c50069620 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.433+0000 7f9c73041700 1 --2- 172.21.15.46:0/1781293909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c6c06f9a0 0x7f9c6c14e160 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.433+0000 7f9c73041700 1 --2- 172.21.15.46:0/1781293909 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c6c156940 0x7f9c6c0b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.433+0000 7f9c73041700 1 --2- 172.21.15.46:0/1781293909 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c6c156570 0x7f9c6c0b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.433+0000 7f9c73041700 1 -- 172.21.15.46:0/1781293909 >> 172.21.15.46:0/1781293909 conn(0x7f9c6c0cdf50 msgr2=0x7f9c6c0ca470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:38.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.433+0000 7f9c73041700 1 -- 172.21.15.46:0/1781293909 shutdown_connections 2024-01-27T01:07:38.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.433+0000 7f9c73041700 1 -- 172.21.15.46:0/1781293909 wait complete. 2024-01-27T01:07:38.446 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-27T01:07:38.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.578+0000 7f1b6a053700 1 Processor -- start 2024-01-27T01:07:38.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.578+0000 7f1b6a053700 1 -- start start 2024-01-27T01:07:38.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.578+0000 7f1b6a053700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b64159960 0x7f1b6406bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b6a053700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1b6406c1f0 0x7f1b64069370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b6a053700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6415dac0 con 0x7f1b64159590 2024-01-27T01:07:38.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b6a053700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b6415e350 con 0x7f1b6406c1f0 2024-01-27T01:07:38.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b6a053700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b640c97c0 con 0x7f1b64159960 2024-01-27T01:07:38.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b637fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1b64159590 0x7f1b6415d3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45596/0 (socket says 172.21.15.46:45596) 2024-01-27T01:07:38.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b637fe700 1 -- 172.21.15.46:0/3569496716 learned_addr learned my addr 172.21.15.46:0/3569496716 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:38.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b63fff700 1 --2- 172.21.15.46:0/3569496716 >> v2:172.21.15.106:3300/0 conn(0x7f1b6406c1f0 0x7f1b64069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b62ffd700 1 --2- 172.21.15.46:0/3569496716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b64159960 0x7f1b6406bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b627fc700 1 -- 172.21.15.46:0/3569496716 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3799189973 0 0) 0x7f1b6415dac0 con 0x7f1b64159590 2024-01-27T01:07:38.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b627fc700 1 -- 172.21.15.46:0/3569496716 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b4c003570 con 0x7f1b64159590 2024-01-27T01:07:38.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b627fc700 1 -- 172.21.15.46:0/3569496716 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3340952179 0 0) 0x7f1b4c003570 con 0x7f1b64159590 2024-01-27T01:07:38.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b627fc700 1 -- 172.21.15.46:0/3569496716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b64159960 msgr2=0x7f1b6406bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:38.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b627fc700 1 --2- 172.21.15.46:0/3569496716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b64159960 0x7f1b6406bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b627fc700 1 -- 172.21.15.46:0/3569496716 >> v2:172.21.15.106:3300/0 conn(0x7f1b6406c1f0 msgr2=0x7f1b64069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:38.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b627fc700 1 --2- 172.21.15.46:0/3569496716 >> v2:172.21.15.106:3300/0 conn(0x7f1b6406c1f0 0x7f1b64069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b627fc700 1 -- 172.21.15.46:0/3569496716 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6415dd90 con 0x7f1b64159590 2024-01-27T01:07:38.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b627fc700 1 -- 172.21.15.46:0/3569496716 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f1b540017f0 con 0x7f1b64159590 2024-01-27T01:07:38.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b63fff700 1 --2- 172.21.15.46:0/3569496716 >> v2:172.21.15.106:3300/0 conn(0x7f1b6406c1f0 0x7f1b64069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:38.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b627fc700 1 -- 172.21.15.46:0/3569496716 >> v1:172.21.15.46:6789/0 conn(0x7f1b64159590 legacy=0x7f1b6415d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b627fc700 1 --2- 172.21.15.46:0/3569496716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b64159960 0x7f1b4c003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b627fc700 1 --2- 172.21.15.46:0/3569496716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b4c003fe0 0x7f1b4c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b627fc700 1 --2- 172.21.15.46:0/3569496716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b4c004d50 0x7f1b4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b627fc700 1 -- 172.21.15.46:0/3569496716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6415e350 con 0x7f1b4c004d50 2024-01-27T01:07:38.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b627fc700 1 -- 172.21.15.46:0/3569496716 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b640c97c0 con 0x7f1b4c003fe0 2024-01-27T01:07:38.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.579+0000 7f1b627fc700 1 -- 172.21.15.46:0/3569496716 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c00bb40 con 0x7f1b64159960 2024-01-27T01:07:38.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.580+0000 7f1b637fe700 1 --2- 172.21.15.46:0/3569496716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b64159960 0x7f1b4c003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.580+0000 7f1b63fff700 1 --2- 172.21.15.46:0/3569496716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b4c004d50 0x7f1b4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.580+0000 7f1b62ffd700 1 --2- 172.21.15.46:0/3569496716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b4c003fe0 0x7f1b4c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.580+0000 7f1b63fff700 1 -- 172.21.15.46:0/3569496716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b64159960 msgr2=0x7f1b4c003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:38.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.580+0000 7f1b63fff700 1 --2- 172.21.15.46:0/3569496716 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b64159960 0x7f1b4c003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.580+0000 7f1b63fff700 1 -- 172.21.15.46:0/3569496716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b4c003fe0 msgr2=0x7f1b4c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:38.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.580+0000 7f1b63fff700 1 --2- 172.21.15.46:0/3569496716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b4c003fe0 0x7f1b4c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.580+0000 7f1b63fff700 1 -- 172.21.15.46:0/3569496716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c003570 con 0x7f1b4c004d50 2024-01-27T01:07:38.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.580+0000 7f1b62ffd700 1 --2- 172.21.15.46:0/3569496716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b4c003fe0 0x7f1b4c004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:38.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.580+0000 7f1b63fff700 1 --2- 172.21.15.46:0/3569496716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b4c004d50 0x7f1b4c0051c0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c006f00 tx=0x7f1b5c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d67ef4a8f80a7a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:38.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.580+0000 7f1b627fc700 1 -- 172.21.15.46:0/3569496716 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c0097e0 con 0x7f1b4c004d50 2024-01-27T01:07:38.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.580+0000 7f1b627fc700 1 -- 172.21.15.46:0/3569496716 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5c00e920 con 0x7f1b4c004d50 2024-01-27T01:07:38.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.580+0000 7f1b627fc700 1 -- 172.21.15.46:0/3569496716 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c00ecc0 con 0x7f1b4c004d50 2024-01-27T01:07:38.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.581+0000 7f1b6a053700 1 -- 172.21.15.46:0/3569496716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b4c004d50 msgr2=0x7f1b4c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.581+0000 7f1b6a053700 1 --2- 172.21.15.46:0/3569496716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b4c004d50 0x7f1b4c0051c0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c006f00 tx=0x7f1b5c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.581+0000 7f1b63fff700 1 -- 172.21.15.46:0/3569496716 reap_dead start 2024-01-27T01:07:38.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.581+0000 7f1b6a053700 1 -- 172.21.15.46:0/3569496716 shutdown_connections 2024-01-27T01:07:38.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.581+0000 7f1b6a053700 1 -- 172.21.15.46:0/3569496716 >> 172.21.15.46:0/3569496716 conn(0x7f1b640ce530 msgr2=0x7f1b640ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:38.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.581+0000 7f1b6a053700 1 -- 172.21.15.46:0/3569496716 shutdown_connections 2024-01-27T01:07:38.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.581+0000 7f1b6a053700 1 -- 172.21.15.46:0/3569496716 wait complete. 2024-01-27T01:07:38.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.582+0000 7f1b6a053700 1 Processor -- start 2024-01-27T01:07:38.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.582+0000 7f1b6a053700 1 -- start start 2024-01-27T01:07:38.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.583+0000 7f1b6a053700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6416c880 0x7f1b640b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.583+0000 7f1b6a053700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b6416cc50 0x7f1b640c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.583+0000 7f1b6a053700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b640b27f0 0x7f1b640c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.583+0000 7f1b6a053700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b640b39f0 con 0x7f1b6416c880 2024-01-27T01:07:38.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.583+0000 7f1b6a053700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b640b2d70 con 0x7f1b6416cc50 2024-01-27T01:07:38.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.583+0000 7f1b6a053700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b640b3050 con 0x7f1b640b27f0 2024-01-27T01:07:38.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.583+0000 7f1b637fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6416c880 0x7f1b640b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.583+0000 7f1b637fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6416c880 0x7f1b640b22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46002/0 (socket says 172.21.15.46:46002) 2024-01-27T01:07:38.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.583+0000 7f1b637fe700 1 -- 172.21.15.46:0/1837461038 learned_addr learned my addr 172.21.15.46:0/1837461038 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:38.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.584+0000 7f1b62ffd700 1 --2- 172.21.15.46:0/1837461038 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b6416cc50 0x7f1b640c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.584+0000 7f1b63fff700 1 --2- 172.21.15.46:0/1837461038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b640b27f0 0x7f1b640c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.584+0000 7f1b637fe700 1 -- 172.21.15.46:0/1837461038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b640b27f0 msgr2=0x7f1b640c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.584+0000 7f1b637fe700 1 --2- 172.21.15.46:0/1837461038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b640b27f0 0x7f1b640c5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.584+0000 7f1b637fe700 1 -- 172.21.15.46:0/1837461038 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b6416cc50 msgr2=0x7f1b640c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.584+0000 7f1b637fe700 1 --2- 172.21.15.46:0/1837461038 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b6416cc50 0x7f1b640c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.584+0000 7f1b637fe700 1 -- 172.21.15.46:0/1837461038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b640b3330 con 0x7f1b6416c880 2024-01-27T01:07:38.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.584+0000 7f1b637fe700 1 --2- 172.21.15.46:0/1837461038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6416c880 0x7f1b640b22b0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f1b54001350 tx=0x7f1b54002210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:38.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.584+0000 7f1b4b7fe700 1 -- 172.21.15.46:0/1837461038 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b540133f0 con 0x7f1b6416c880 2024-01-27T01:07:38.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.584+0000 7f1b4b7fe700 1 -- 172.21.15.46:0/1837461038 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b54013550 con 0x7f1b6416c880 2024-01-27T01:07:38.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.584+0000 7f1b6a053700 1 -- 172.21.15.46:0/1837461038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b640b3610 con 0x7f1b6416c880 2024-01-27T01:07:38.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.584+0000 7f1b4b7fe700 1 -- 172.21.15.46:0/1837461038 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b540138d0 con 0x7f1b6416c880 2024-01-27T01:07:38.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.584+0000 7f1b6a053700 1 -- 172.21.15.46:0/1837461038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b64169d60 con 0x7f1b6416c880 2024-01-27T01:07:38.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.585+0000 7f1b6a053700 1 -- 172.21.15.46:0/1837461038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b640b3b30 con 0x7f1b6416c880 2024-01-27T01:07:38.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.586+0000 7f1b4b7fe700 1 -- 172.21.15.46:0/1837461038 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f1b54013cc0 con 0x7f1b6416c880 2024-01-27T01:07:38.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.586+0000 7f1b4b7fe700 1 --2- 172.21.15.46:0/1837461038 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b50067170 0x7f1b50069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.586+0000 7f1b4b7fe700 1 -- 172.21.15.46:0/1837461038 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f1b54085940 con 0x7f1b6416c880 2024-01-27T01:07:38.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.586+0000 7f1b62ffd700 1 --2- 172.21.15.46:0/1837461038 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b50067170 0x7f1b50069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.587+0000 7f1b62ffd700 1 --2- 172.21.15.46:0/1837461038 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b50067170 0x7f1b50069620 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1b6406e300 tx=0x7f1b580091a0 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:38.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.591+0000 7f1b4b7fe700 1 -- 172.21.15.46:0/1837461038 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1b640b3b30 con 0x7f1b6416c880 2024-01-27T01:07:38.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.798+0000 7f1b6a053700 1 -- 172.21.15.46:0/1837461038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1b640b3b30 con 0x7f1b6416c880 2024-01-27T01:07:38.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.798+0000 7f1b4b7fe700 1 -- 172.21.15.46:0/1837461038 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v243) v1 ==== 84+0+209 (secure 0 0 0) 0x7f1b640b3b30 con 0x7f1b6416c880 2024-01-27T01:07:38.798 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:38.799 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-27T01:07:38.799 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:38.799 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-01-27T01:07:38.799 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-01-27T01:07:38.799 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:38.799 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:38.799 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 31, 2024-01-27T01:07:38.799 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": "cephfs_metadata" 2024-01-27T01:07:38.799 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:38.799 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:38.799 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 32, 2024-01-27T01:07:38.799 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": "cephfs_data" 2024-01-27T01:07:38.799 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:38.799 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-27T01:07:38.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.801+0000 7f1b6a053700 1 -- 172.21.15.46:0/1837461038 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b50067170 msgr2=0x7f1b50069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.801+0000 7f1b6a053700 1 --2- 172.21.15.46:0/1837461038 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b50067170 0x7f1b50069620 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1b6406e300 tx=0x7f1b580091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.801+0000 7f1b6a053700 1 -- 172.21.15.46:0/1837461038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6416c880 msgr2=0x7f1b640b22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.801+0000 7f1b6a053700 1 --2- 172.21.15.46:0/1837461038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6416c880 0x7f1b640b22b0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f1b54001350 tx=0x7f1b54002210 comp rx=0 tx=0).stop 2024-01-27T01:07:38.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.801+0000 7f1b6a053700 1 -- 172.21.15.46:0/1837461038 shutdown_connections 2024-01-27T01:07:38.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.801+0000 7f1b6a053700 1 --2- 172.21.15.46:0/1837461038 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1b50067170 0x7f1b50069620 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.801+0000 7f1b6a053700 1 --2- 172.21.15.46:0/1837461038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6416c880 0x7f1b640b22b0 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.801+0000 7f1b6a053700 1 --2- 172.21.15.46:0/1837461038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b640b27f0 0x7f1b640c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.801+0000 7f1b6a053700 1 --2- 172.21.15.46:0/1837461038 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b6416cc50 0x7f1b640c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.801+0000 7f1b6a053700 1 -- 172.21.15.46:0/1837461038 >> 172.21.15.46:0/1837461038 conn(0x7f1b640ce530 msgr2=0x7f1b640c3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:38.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.801+0000 7f1b6a053700 1 -- 172.21.15.46:0/1837461038 shutdown_connections 2024-01-27T01:07:38.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.801+0000 7f1b6a053700 1 -- 172.21.15.46:0/1837461038 wait complete. 2024-01-27T01:07:38.814 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-01-27T01:07:38.815 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-01-27T01:07:38.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.950+0000 7f0207191700 1 Processor -- start 2024-01-27T01:07:38.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.950+0000 7f0207191700 1 -- start start 2024-01-27T01:07:38.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.951+0000 7f0207191700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02000b26e0 0x7f02000b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.951+0000 7f0207191700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f020015bd50 0x7f02001571f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.951+0000 7f0207191700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0200157bb0 con 0x7f0200157730 2024-01-27T01:07:38.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.951+0000 7f0207191700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02000cd4f0 con 0x7f02000b26e0 2024-01-27T01:07:38.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.951+0000 7f0207191700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02000cd7d0 con 0x7f020015bd50 2024-01-27T01:07:38.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.951+0000 7f0204f2d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02000b26e0 0x7f02000b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.951+0000 7f020572e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0200157730 0x7f0200067930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45610/0 (socket says 172.21.15.46:45610) 2024-01-27T01:07:38.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.951+0000 7f020572e700 1 -- 172.21.15.46:0/3893480762 learned_addr learned my addr 172.21.15.46:0/3893480762 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:38.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.951+0000 7f01fffff700 1 --2- 172.21.15.46:0/3893480762 >> v2:172.21.15.106:3300/0 conn(0x7f020015bd50 0x7f02001571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.952+0000 7f01ff7fe700 1 -- 172.21.15.46:0/3893480762 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990575208 0 0) 0x7f0200157bb0 con 0x7f0200157730 2024-01-27T01:07:38.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.952+0000 7f01ff7fe700 1 -- 172.21.15.46:0/3893480762 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f0003570 con 0x7f0200157730 2024-01-27T01:07:38.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.952+0000 7f0204f2d700 1 -- 172.21.15.46:0/3893480762 >> v2:172.21.15.106:3300/0 conn(0x7f020015bd50 msgr2=0x7f02001571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:38.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.952+0000 7f0204f2d700 1 --2- 172.21.15.46:0/3893480762 >> v2:172.21.15.106:3300/0 conn(0x7f020015bd50 0x7f02001571f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.952+0000 7f0204f2d700 1 -- 172.21.15.46:0/3893480762 >> v1:172.21.15.46:6789/0 conn(0x7f0200157730 legacy=0x7f0200067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.952+0000 7f0204f2d700 1 -- 172.21.15.46:0/3893480762 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02000cdab0 con 0x7f02000b26e0 2024-01-27T01:07:38.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.952+0000 7f0204f2d700 1 --2- 172.21.15.46:0/3893480762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02000b26e0 0x7f02000b2ab0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f02000affb0 tx=0x7f01f40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7559c2e4dd8cf2 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:38.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.955+0000 7f01ff7fe700 1 -- 172.21.15.46:0/3893480762 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f4011630 con 0x7f02000b26e0 2024-01-27T01:07:38.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.955+0000 7f01ff7fe700 1 -- 172.21.15.46:0/3893480762 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f40117b0 con 0x7f02000b26e0 2024-01-27T01:07:38.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.955+0000 7f01ff7fe700 1 -- 172.21.15.46:0/3893480762 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f4011b30 con 0x7f02000b26e0 2024-01-27T01:07:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.955+0000 7f0207191700 1 -- 172.21.15.46:0/3893480762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02000b26e0 msgr2=0x7f02000b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.955+0000 7f0207191700 1 --2- 172.21.15.46:0/3893480762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02000b26e0 0x7f02000b2ab0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f02000affb0 tx=0x7f01f40091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.955+0000 7f0207191700 1 -- 172.21.15.46:0/3893480762 shutdown_connections 2024-01-27T01:07:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.955+0000 7f0207191700 1 --2- 172.21.15.46:0/3893480762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02000b26e0 0x7f02000b2ab0 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.955+0000 7f0207191700 1 --2- 172.21.15.46:0/3893480762 >> v2:172.21.15.106:3300/0 conn(0x7f020015bd50 0x7f02001571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.955+0000 7f0207191700 1 -- 172.21.15.46:0/3893480762 >> 172.21.15.46:0/3893480762 conn(0x7f0200069510 msgr2=0x7f02000aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.955+0000 7f0207191700 1 -- 172.21.15.46:0/3893480762 shutdown_connections 2024-01-27T01:07:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.955+0000 7f0207191700 1 -- 172.21.15.46:0/3893480762 wait complete. 2024-01-27T01:07:38.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.957+0000 7f0207191700 1 Processor -- start 2024-01-27T01:07:38.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.957+0000 7f0207191700 1 -- start start 2024-01-27T01:07:38.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.958+0000 7f0207191700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02000b26e0 0x7f020015cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.958+0000 7f0207191700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0200157730 0x7f0200151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.958+0000 7f0207191700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f020015bd50 0x7f02001520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.958+0000 7f0207191700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0200152630 con 0x7f020015bd50 2024-01-27T01:07:38.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.958+0000 7f0207191700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0200155700 con 0x7f02000b26e0 2024-01-27T01:07:38.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.958+0000 7f0207191700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02001559e0 con 0x7f0200157730 2024-01-27T01:07:38.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.958+0000 7f020572e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f020015bd50 0x7f02001520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.958+0000 7f020572e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f020015bd50 0x7f02001520f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46008/0 (socket says 172.21.15.46:46008) 2024-01-27T01:07:38.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.958+0000 7f01fffff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0200157730 0x7f0200151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.958+0000 7f020572e700 1 -- 172.21.15.46:0/3495519084 learned_addr learned my addr 172.21.15.46:0/3495519084 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:38.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.958+0000 7f0204f2d700 1 --2- 172.21.15.46:0/3495519084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02000b26e0 0x7f020015cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.958+0000 7f020572e700 1 -- 172.21.15.46:0/3495519084 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0200157730 msgr2=0x7f0200151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.958+0000 7f020572e700 1 --2- 172.21.15.46:0/3495519084 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0200157730 0x7f0200151bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.958+0000 7f020572e700 1 -- 172.21.15.46:0/3495519084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02000b26e0 msgr2=0x7f020015cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:38.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.958+0000 7f020572e700 1 --2- 172.21.15.46:0/3495519084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02000b26e0 0x7f020015cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:38.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.958+0000 7f020572e700 1 -- 172.21.15.46:0/3495519084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f4009420 con 0x7f020015bd50 2024-01-27T01:07:38.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.958+0000 7f020572e700 1 --2- 172.21.15.46:0/3495519084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f020015bd50 0x7f02001520f0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f01f8001460 tx=0x7f01f8001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:38.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.959+0000 7f01fcff9700 1 -- 172.21.15.46:0/3495519084 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f800a890 con 0x7f020015bd50 2024-01-27T01:07:38.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.959+0000 7f01fcff9700 1 -- 172.21.15.46:0/3495519084 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f800a9f0 con 0x7f020015bd50 2024-01-27T01:07:38.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.959+0000 7f0207191700 1 -- 172.21.15.46:0/3495519084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0200155cc0 con 0x7f020015bd50 2024-01-27T01:07:38.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.959+0000 7f01fcff9700 1 -- 172.21.15.46:0/3495519084 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f800ad70 con 0x7f020015bd50 2024-01-27T01:07:38.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.959+0000 7f0207191700 1 -- 172.21.15.46:0/3495519084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0200156280 con 0x7f020015bd50 2024-01-27T01:07:38.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.960+0000 7f0207191700 1 -- 172.21.15.46:0/3495519084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f020004c710 con 0x7f020015bd50 2024-01-27T01:07:38.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.960+0000 7f01fcff9700 1 -- 172.21.15.46:0/3495519084 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f01f8007300 con 0x7f020015bd50 2024-01-27T01:07:38.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.961+0000 7f01fcff9700 1 --2- 172.21.15.46:0/3495519084 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f01e4067170 0x7f01e4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:38.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.961+0000 7f01fcff9700 1 -- 172.21.15.46:0/3495519084 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f01f8084f90 con 0x7f020015bd50 2024-01-27T01:07:38.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.961+0000 7f0204f2d700 1 --2- 172.21.15.46:0/3495519084 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f01e4067170 0x7f01e4069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:38.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.962+0000 7f0204f2d700 1 --2- 172.21.15.46:0/3495519084 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f01e4067170 0x7f01e4069620 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f02000affb0 tx=0x7f01f4017000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:38.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:38.965+0000 7f01fcff9700 1 -- 172.21.15.46:0/3495519084 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f01f8051c30 con 0x7f020015bd50 2024-01-27T01:07:39.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:39.171+0000 7f0207191700 1 -- 172.21.15.46:0/3495519084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f02000ccb80 con 0x7f020015bd50 2024-01-27T01:07:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:39.965+0000 7f01fcff9700 1 -- 172.21.15.46:0/3495519084 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v244) v1 ==== 144+0+0 (secure 0 0 0) 0x7f01f8055250 con 0x7f020015bd50 2024-01-27T01:07:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:39.971+0000 7f0207191700 1 -- 172.21.15.46:0/3495519084 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f01e4067170 msgr2=0x7f01e4069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:39.971+0000 7f0207191700 1 --2- 172.21.15.46:0/3495519084 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f01e4067170 0x7f01e4069620 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f02000affb0 tx=0x7f01f4017000 comp rx=0 tx=0).stop 2024-01-27T01:07:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:39.971+0000 7f0207191700 1 -- 172.21.15.46:0/3495519084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f020015bd50 msgr2=0x7f02001520f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:39.971+0000 7f0207191700 1 --2- 172.21.15.46:0/3495519084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f020015bd50 0x7f02001520f0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f01f8001460 tx=0x7f01f8001ae0 comp rx=0 tx=0).stop 2024-01-27T01:07:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:39.971+0000 7f0207191700 1 -- 172.21.15.46:0/3495519084 shutdown_connections 2024-01-27T01:07:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:39.971+0000 7f0207191700 1 --2- 172.21.15.46:0/3495519084 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f01e4067170 0x7f01e4069620 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:39.971+0000 7f0207191700 1 --2- 172.21.15.46:0/3495519084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f020015bd50 0x7f02001520f0 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:39.971+0000 7f0207191700 1 --2- 172.21.15.46:0/3495519084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f02000b26e0 0x7f020015cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:39.971+0000 7f0207191700 1 --2- 172.21.15.46:0/3495519084 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0200157730 0x7f0200151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:39.971+0000 7f0207191700 1 -- 172.21.15.46:0/3495519084 >> 172.21.15.46:0/3495519084 conn(0x7f0200069510 msgr2=0x7f02000b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:39.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:39.972+0000 7f0207191700 1 -- 172.21.15.46:0/3495519084 shutdown_connections 2024-01-27T01:07:39.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:39.972+0000 7f0207191700 1 -- 172.21.15.46:0/3495519084 wait complete. 2024-01-27T01:07:39.973 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'backup_fs_metadata' created 2024-01-27T01:07:39.987 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-27T01:07:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.131+0000 7fc5f3f66700 1 Processor -- start 2024-01-27T01:07:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.131+0000 7fc5f3f66700 1 -- start start 2024-01-27T01:07:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.132+0000 7fc5f3f66700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc5ec0cdbf0 0x7fc5ec0ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.132+0000 7fc5f3f66700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5ec0ae610 0x7fc5ec0abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.132+0000 7fc5f3f66700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ec0abb30 con 0x7fc5ec0b25b0 2024-01-27T01:07:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.132+0000 7fc5f3f66700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5ec0b2980 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.132+0000 7fc5f3f66700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec15a560 con 0x7fc5ec0ae610 2024-01-27T01:07:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.132+0000 7fc5f2503700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5ec0ae610 0x7fc5ec0abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.132+0000 7fc5f1501700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc5ec0cdbf0 0x7fc5ec0ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.132+0000 7fc5f1d02700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fc5ec0b25b0 0x7fc5ec0b04e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:55658/0 (socket says 172.21.15.46:55658) 2024-01-27T01:07:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.133+0000 7fc5f1d02700 1 -- 172.21.15.46:0/4285874742 learned_addr learned my addr 172.21.15.46:0/4285874742 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.133+0000 7fc5f0d00700 1 -- 172.21.15.46:0/4285874742 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2012650269 0 0) 0x7fc5ec0abb30 con 0x7fc5ec0b25b0 2024-01-27T01:07:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.133+0000 7fc5f0d00700 1 -- 172.21.15.46:0/4285874742 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d8003570 con 0x7fc5ec0b25b0 2024-01-27T01:07:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.133+0000 7fc5f1501700 1 -- 172.21.15.46:0/4285874742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5ec0ae610 msgr2=0x7fc5ec0abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.133+0000 7fc5f1501700 1 --2- 172.21.15.46:0/4285874742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5ec0ae610 0x7fc5ec0abfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.133+0000 7fc5f1501700 1 -- 172.21.15.46:0/4285874742 >> v1:172.21.15.106:6789/0 conn(0x7fc5ec0b25b0 legacy=0x7fc5ec0b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.133+0000 7fc5f1501700 1 -- 172.21.15.46:0/4285874742 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec0b3210 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.133+0000 7fc5f2503700 1 --2- 172.21.15.46:0/4285874742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5ec0ae610 0x7fc5ec0abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.133+0000 7fc5f1501700 1 --2- 172.21.15.46:0/4285874742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5ec0cdbf0 0x7fc5ec0ae0d0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0098e0 tx=0x7fc5dc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1de2bdd72ab0871a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.133+0000 7fc5f0d00700 1 -- 172.21.15.46:0/4285874742 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc011630 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.133+0000 7fc5f0d00700 1 -- 172.21.15.46:0/4285874742 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5dc011790 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.133+0000 7fc5f0d00700 1 -- 172.21.15.46:0/4285874742 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc011b30 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.134+0000 7fc5f3f66700 1 -- 172.21.15.46:0/4285874742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5ec0cdbf0 msgr2=0x7fc5ec0ae0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.134+0000 7fc5f3f66700 1 --2- 172.21.15.46:0/4285874742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5ec0cdbf0 0x7fc5ec0ae0d0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0098e0 tx=0x7fc5dc0091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.134+0000 7fc5f3f66700 1 -- 172.21.15.46:0/4285874742 shutdown_connections 2024-01-27T01:07:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.134+0000 7fc5f3f66700 1 --2- 172.21.15.46:0/4285874742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5ec0ae610 0x7fc5ec0abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.134+0000 7fc5f3f66700 1 --2- 172.21.15.46:0/4285874742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5ec0cdbf0 0x7fc5ec0ae0d0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.134+0000 7fc5f3f66700 1 -- 172.21.15.46:0/4285874742 >> 172.21.15.46:0/4285874742 conn(0x7fc5ec069220 msgr2=0x7fc5ec0cd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.134+0000 7fc5f3f66700 1 -- 172.21.15.46:0/4285874742 shutdown_connections 2024-01-27T01:07:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.134+0000 7fc5f3f66700 1 -- 172.21.15.46:0/4285874742 wait complete. 2024-01-27T01:07:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.135+0000 7fc5f3f66700 1 Processor -- start 2024-01-27T01:07:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.135+0000 7fc5f3f66700 1 -- start start 2024-01-27T01:07:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.136+0000 7fc5f3f66700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5ec0ae610 0x7fc5ec155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.136+0000 7fc5f3f66700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5ec0b25b0 0x7fc5ec0977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.136+0000 7fc5f3f66700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5ec0cdbf0 0x7fc5ec16a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.136+0000 7fc5f3f66700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec0cc4d0 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.136+0000 7fc5f3f66700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec156290 con 0x7fc5ec0ae610 2024-01-27T01:07:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.136+0000 7fc5f3f66700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec0cbba0 con 0x7fc5ec0b25b0 2024-01-27T01:07:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.136+0000 7fc5f2503700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5ec0cdbf0 0x7fc5ec16a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.136+0000 7fc5f2503700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5ec0cdbf0 0x7fc5ec16a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46016/0 (socket says 172.21.15.46:46016) 2024-01-27T01:07:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.136+0000 7fc5f2503700 1 -- 172.21.15.46:0/2634489470 learned_addr learned my addr 172.21.15.46:0/2634489470 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.136+0000 7fc5f1d02700 1 --2- 172.21.15.46:0/2634489470 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5ec0ae610 0x7fc5ec155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.136+0000 7fc5f1501700 1 --2- 172.21.15.46:0/2634489470 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5ec0b25b0 0x7fc5ec0977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.137+0000 7fc5f2503700 1 -- 172.21.15.46:0/2634489470 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5ec0b25b0 msgr2=0x7fc5ec0977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.137+0000 7fc5f2503700 1 --2- 172.21.15.46:0/2634489470 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5ec0b25b0 0x7fc5ec0977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.137+0000 7fc5f2503700 1 -- 172.21.15.46:0/2634489470 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5ec0ae610 msgr2=0x7fc5ec155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.137+0000 7fc5f2503700 1 --2- 172.21.15.46:0/2634489470 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5ec0ae610 0x7fc5ec155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.137+0000 7fc5f2503700 1 -- 172.21.15.46:0/2634489470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e4009a20 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.137+0000 7fc5f2503700 1 --2- 172.21.15.46:0/2634489470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5ec0cdbf0 0x7fc5ec16a710 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009fc0 tx=0x7fc5e400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.137+0000 7fc5e1ffb700 1 -- 172.21.15.46:0/2634489470 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e400cbd0 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.137+0000 7fc5e1ffb700 1 -- 172.21.15.46:0/2634489470 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e400f040 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.137+0000 7fc5f3f66700 1 -- 172.21.15.46:0/2634489470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5dc009420 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.137+0000 7fc5e1ffb700 1 -- 172.21.15.46:0/2634489470 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4014480 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.137+0000 7fc5f3f66700 1 -- 172.21.15.46:0/2634489470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ec0cc160 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.138+0000 7fc5f3f66700 1 -- 172.21.15.46:0/2634489470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ec0cc640 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.139+0000 7fc5e1ffb700 1 -- 172.21.15.46:0/2634489470 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fc5e4003020 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.139+0000 7fc5e1ffb700 1 --2- 172.21.15.46:0/2634489470 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc5d0067170 0x7fc5d0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:40.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.139+0000 7fc5e1ffb700 1 -- 172.21.15.46:0/2634489470 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6890+0+0 (secure 0 0 0) 0x7fc5e40858a0 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.140+0000 7fc5f1d02700 1 --2- 172.21.15.46:0/2634489470 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc5d0067170 0x7fc5d0069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:40.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.140+0000 7fc5f1d02700 1 --2- 172.21.15.46:0/2634489470 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc5d0067170 0x7fc5d0069620 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8007fd0 tx=0x7fc5e8009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:40.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.144+0000 7fc5e1ffb700 1 -- 172.21.15.46:0/2634489470 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc5e4052200 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.349+0000 7fc5f3f66700 1 -- 172.21.15.46:0/2634489470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc5ec04c710 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.968+0000 7fc5e1ffb700 1 -- 172.21.15.46:0/2634489470 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v245) v1 ==== 176+0+0 (secure 0 0 0) 0x7fc5e4055820 con 0x7fc5ec0cdbf0 2024-01-27T01:07:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.990+0000 7fc5f3f66700 1 -- 172.21.15.46:0/2634489470 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc5d0067170 msgr2=0x7fc5d0069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.990+0000 7fc5f3f66700 1 --2- 172.21.15.46:0/2634489470 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc5d0067170 0x7fc5d0069620 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8007fd0 tx=0x7fc5e8009040 comp rx=0 tx=0).stop 2024-01-27T01:07:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.990+0000 7fc5f3f66700 1 -- 172.21.15.46:0/2634489470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5ec0cdbf0 msgr2=0x7fc5ec16a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.990+0000 7fc5f3f66700 1 --2- 172.21.15.46:0/2634489470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5ec0cdbf0 0x7fc5ec16a710 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009fc0 tx=0x7fc5e400b040 comp rx=0 tx=0).stop 2024-01-27T01:07:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.991+0000 7fc5f3f66700 1 -- 172.21.15.46:0/2634489470 shutdown_connections 2024-01-27T01:07:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.991+0000 7fc5f3f66700 1 --2- 172.21.15.46:0/2634489470 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc5d0067170 0x7fc5d0069620 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.991+0000 7fc5f3f66700 1 --2- 172.21.15.46:0/2634489470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5ec0cdbf0 0x7fc5ec16a710 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.991+0000 7fc5f3f66700 1 --2- 172.21.15.46:0/2634489470 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5ec0ae610 0x7fc5ec155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.991+0000 7fc5f3f66700 1 --2- 172.21.15.46:0/2634489470 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5ec0b25b0 0x7fc5ec0977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.991+0000 7fc5f3f66700 1 -- 172.21.15.46:0/2634489470 >> 172.21.15.46:0/2634489470 conn(0x7fc5ec069220 msgr2=0x7fc5ec0cd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.991+0000 7fc5f3f66700 1 -- 172.21.15.46:0/2634489470 shutdown_connections 2024-01-27T01:07:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:40.991+0000 7fc5f3f66700 1 -- 172.21.15.46:0/2634489470 wait complete. 2024-01-27T01:07:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'backup_fs_data' created 2024-01-27T01:07:41.006 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-01-27T01:07:41.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.143+0000 7fe5c15a9700 1 Processor -- start 2024-01-27T01:07:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.143+0000 7fe5c15a9700 1 -- start start 2024-01-27T01:07:41.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.144+0000 7fe5c15a9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5bc157490 0x7fe5bc0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:41.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.144+0000 7fe5c15a9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe5bc157860 0x7fe5bc0cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:41.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.144+0000 7fe5c15a9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5bc06d6a0 0x7fe5bc06dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:41.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.144+0000 7fe5c15a9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5bc0cdc60 con 0x7fe5bc06d6a0 2024-01-27T01:07:41.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.144+0000 7fe5c15a9700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5bc0cea80 con 0x7fe5bc157860 2024-01-27T01:07:41.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.144+0000 7fe5c15a9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc06dff0 con 0x7fe5bc157490 2024-01-27T01:07:41.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.144+0000 7fe5bb7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5bc06d6a0 0x7fe5bc06dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:41.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.144+0000 7fe5bb7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5bc06d6a0 0x7fe5bc06dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37676/0 (socket says 172.21.15.46:37676) 2024-01-27T01:07:41.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.144+0000 7fe5bb7fe700 1 -- 172.21.15.46:0/2529725121 learned_addr learned my addr 172.21.15.46:0/2529725121 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:41.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.145+0000 7fe5ba7fc700 1 --2- 172.21.15.46:0/2529725121 >> v2:172.21.15.106:3300/0 conn(0x7fe5bc157860 0x7fe5bc0cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:41.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.145+0000 7fe5baffd700 1 --2- 172.21.15.46:0/2529725121 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5bc157490 0x7fe5bc0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:41.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.145+0000 7fe5bb7fe700 1 -- 172.21.15.46:0/2529725121 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5bc157490 msgr2=0x7fe5bc0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:41.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.145+0000 7fe5bb7fe700 1 --2- 172.21.15.46:0/2529725121 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5bc157490 0x7fe5bc0cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:41.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.145+0000 7fe5bb7fe700 1 -- 172.21.15.46:0/2529725121 >> v2:172.21.15.106:3300/0 conn(0x7fe5bc157860 msgr2=0x7fe5bc0cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:41.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.145+0000 7fe5bb7fe700 1 --2- 172.21.15.46:0/2529725121 >> v2:172.21.15.106:3300/0 conn(0x7fe5bc157860 0x7fe5bc0cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:41.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.145+0000 7fe5bb7fe700 1 -- 172.21.15.46:0/2529725121 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5bc0cde70 con 0x7fe5bc06d6a0 2024-01-27T01:07:41.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.145+0000 7fe5bb7fe700 1 --2- 172.21.15.46:0/2529725121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5bc06d6a0 0x7fe5bc06dab0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc06cfe0 tx=0x7fe5b4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb07d3e831b659cf server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:41.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.145+0000 7fe5b9ffb700 1 -- 172.21.15.46:0/2529725121 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b400e070 con 0x7fe5bc06d6a0 2024-01-27T01:07:41.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.145+0000 7fe5b9ffb700 1 -- 172.21.15.46:0/2529725121 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b400b970 con 0x7fe5bc06d6a0 2024-01-27T01:07:41.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.145+0000 7fe5b9ffb700 1 -- 172.21.15.46:0/2529725121 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b40092e0 con 0x7fe5bc06d6a0 2024-01-27T01:07:41.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.146+0000 7fe5c15a9700 1 -- 172.21.15.46:0/2529725121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5bc06d6a0 msgr2=0x7fe5bc06dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:41.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.146+0000 7fe5c15a9700 1 --2- 172.21.15.46:0/2529725121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5bc06d6a0 0x7fe5bc06dab0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc06cfe0 tx=0x7fe5b4009840 comp rx=0 tx=0).stop 2024-01-27T01:07:41.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.146+0000 7fe5c15a9700 1 -- 172.21.15.46:0/2529725121 shutdown_connections 2024-01-27T01:07:41.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.146+0000 7fe5c15a9700 1 --2- 172.21.15.46:0/2529725121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5bc06d6a0 0x7fe5bc06dab0 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:41.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.146+0000 7fe5c15a9700 1 --2- 172.21.15.46:0/2529725121 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5bc157490 0x7fe5bc0cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:41.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.146+0000 7fe5c15a9700 1 --2- 172.21.15.46:0/2529725121 >> v2:172.21.15.106:3300/0 conn(0x7fe5bc157860 0x7fe5bc0cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:41.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.146+0000 7fe5c15a9700 1 -- 172.21.15.46:0/2529725121 >> 172.21.15.46:0/2529725121 conn(0x7fe5bc067930 msgr2=0x7fe5bc067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:41.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.146+0000 7fe5c15a9700 1 -- 172.21.15.46:0/2529725121 shutdown_connections 2024-01-27T01:07:41.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.146+0000 7fe5c15a9700 1 -- 172.21.15.46:0/2529725121 wait complete. 2024-01-27T01:07:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.147+0000 7fe5c15a9700 1 Processor -- start 2024-01-27T01:07:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.147+0000 7fe5c15a9700 1 -- start start 2024-01-27T01:07:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.148+0000 7fe5c15a9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5bc06d6a0 0x7fe5bc1680b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.148+0000 7fe5c15a9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5bc157490 0x7fe5bc16a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.148+0000 7fe5c15a9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5bc157860 0x7fe5bc0b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.148+0000 7fe5c15a9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc1685f0 con 0x7fe5bc157490 2024-01-27T01:07:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.148+0000 7fe5c15a9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc0afe80 con 0x7fe5bc157860 2024-01-27T01:07:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.148+0000 7fe5c15a9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc0af270 con 0x7fe5bc06d6a0 2024-01-27T01:07:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.148+0000 7fe5ba7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5bc157490 0x7fe5bc16a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.148+0000 7fe5ba7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5bc157490 0x7fe5bc16a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37680/0 (socket says 172.21.15.46:37680) 2024-01-27T01:07:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.148+0000 7fe5ba7fc700 1 -- 172.21.15.46:0/1255080685 learned_addr learned my addr 172.21.15.46:0/1255080685 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.148+0000 7fe5bb7fe700 1 --2- 172.21.15.46:0/1255080685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5bc157860 0x7fe5bc0b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.148+0000 7fe5baffd700 1 --2- 172.21.15.46:0/1255080685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5bc06d6a0 0x7fe5bc1680b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.149+0000 7fe5ba7fc700 1 -- 172.21.15.46:0/1255080685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5bc06d6a0 msgr2=0x7fe5bc1680b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.149+0000 7fe5ba7fc700 1 --2- 172.21.15.46:0/1255080685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5bc06d6a0 0x7fe5bc1680b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.149+0000 7fe5ba7fc700 1 -- 172.21.15.46:0/1255080685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5bc157860 msgr2=0x7fe5bc0b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.149+0000 7fe5ba7fc700 1 --2- 172.21.15.46:0/1255080685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5bc157860 0x7fe5bc0b1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.149+0000 7fe5ba7fc700 1 -- 172.21.15.46:0/1255080685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b4009ae0 con 0x7fe5bc157490 2024-01-27T01:07:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.149+0000 7fe5baffd700 1 --2- 172.21.15.46:0/1255080685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5bc06d6a0 0x7fe5bc1680b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.149+0000 7fe5ba7fc700 1 --2- 172.21.15.46:0/1255080685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5bc157490 0x7fe5bc16a3d0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fe5a400e900 tx=0x7fe5a40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.149+0000 7fe5b2ffd700 1 -- 172.21.15.46:0/1255080685 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4013680 con 0x7fe5bc157490 2024-01-27T01:07:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.149+0000 7fe5b2ffd700 1 -- 172.21.15.46:0/1255080685 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a40137e0 con 0x7fe5bc157490 2024-01-27T01:07:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.149+0000 7fe5c15a9700 1 -- 172.21.15.46:0/1255080685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5bc0af550 con 0x7fe5bc157490 2024-01-27T01:07:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.149+0000 7fe5b2ffd700 1 -- 172.21.15.46:0/1255080685 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4013ba0 con 0x7fe5bc157490 2024-01-27T01:07:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.149+0000 7fe5c15a9700 1 -- 172.21.15.46:0/1255080685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5bc0afb10 con 0x7fe5bc157490 2024-01-27T01:07:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.150+0000 7fe5c15a9700 1 -- 172.21.15.46:0/1255080685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5bc04c710 con 0x7fe5bc157490 2024-01-27T01:07:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.151+0000 7fe5b2ffd700 1 -- 172.21.15.46:0/1255080685 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fe5a400c460 con 0x7fe5bc157490 2024-01-27T01:07:41.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.151+0000 7fe5b2ffd700 1 --2- 172.21.15.46:0/1255080685 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe59c067170 0x7fe59c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:41.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.151+0000 7fe5b2ffd700 1 -- 172.21.15.46:0/1255080685 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 7275+0+0 (secure 0 0 0) 0x7fe5a40857a0 con 0x7fe5bc157490 2024-01-27T01:07:41.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.151+0000 7fe5baffd700 1 --2- 172.21.15.46:0/1255080685 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe59c067170 0x7fe59c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:41.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.152+0000 7fe5baffd700 1 --2- 172.21.15.46:0/1255080685 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe59c067170 0x7fe59c069620 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac009d80 tx=0x7fe5ac008040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:41.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.155+0000 7fe5b2ffd700 1 -- 172.21.15.46:0/1255080685 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe5a40521e0 con 0x7fe5bc157490 2024-01-27T01:07:41.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.408+0000 7fe5c15a9700 1 -- 172.21.15.46:0/1255080685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7fe5bc0b14d0 con 0x7fe5bc157490 2024-01-27T01:07:41.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.981+0000 7fe5b2ffd700 1 -- 172.21.15.46:0/1255080685 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 33 and data pool 34 v135) v1 ==== 183+0+0 (secure 0 0 0) 0x7fe5a4055800 con 0x7fe5bc157490 2024-01-27T01:07:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.996+0000 7fe5c15a9700 1 -- 172.21.15.46:0/1255080685 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe59c067170 msgr2=0x7fe59c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.996+0000 7fe5c15a9700 1 --2- 172.21.15.46:0/1255080685 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe59c067170 0x7fe59c069620 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac009d80 tx=0x7fe5ac008040 comp rx=0 tx=0).stop 2024-01-27T01:07:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.996+0000 7fe5c15a9700 1 -- 172.21.15.46:0/1255080685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5bc157490 msgr2=0x7fe5bc16a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.996+0000 7fe5c15a9700 1 --2- 172.21.15.46:0/1255080685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5bc157490 0x7fe5bc16a3d0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fe5a400e900 tx=0x7fe5a40091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:41.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.998+0000 7fe5c15a9700 1 -- 172.21.15.46:0/1255080685 shutdown_connections 2024-01-27T01:07:41.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.998+0000 7fe5c15a9700 1 --2- 172.21.15.46:0/1255080685 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe59c067170 0x7fe59c069620 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:41.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.998+0000 7fe5c15a9700 1 --2- 172.21.15.46:0/1255080685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5bc157860 0x7fe5bc0b1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:41.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.998+0000 7fe5c15a9700 1 --2- 172.21.15.46:0/1255080685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe5bc06d6a0 0x7fe5bc1680b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:41.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.998+0000 7fe5c15a9700 1 --2- 172.21.15.46:0/1255080685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5bc157490 0x7fe5bc16a3d0 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:41.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.998+0000 7fe5c15a9700 1 -- 172.21.15.46:0/1255080685 >> 172.21.15.46:0/1255080685 conn(0x7fe5bc067930 msgr2=0x7fe5bc154500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:41.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.998+0000 7fe5c15a9700 1 -- 172.21.15.46:0/1255080685 shutdown_connections 2024-01-27T01:07:41.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:41.999+0000 7fe5c15a9700 1 -- 172.21.15.46:0/1255080685 wait complete. 2024-01-27T01:07:42.005 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 33 and data pool 34 2024-01-27T01:07:42.025 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:07:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.159+0000 7feb440a2700 1 Processor -- start 2024-01-27T01:07:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.159+0000 7feb440a2700 1 -- start start 2024-01-27T01:07:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.160+0000 7feb440a2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb3c0b08c0 0x7feb3c068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.160+0000 7feb440a2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3c0c98c0 con 0x7feb3c0cd9d0 2024-01-27T01:07:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.160+0000 7feb440a2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3c0c8cb0 con 0x7feb3c0cb4c0 2024-01-27T01:07:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.160+0000 7feb440a2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb3c0c8f90 con 0x7feb3c0b08c0 2024-01-27T01:07:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.160+0000 7feb41e3e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb3c0b08c0 0x7feb3c068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.160+0000 7feb41e3e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb3c0b08c0 0x7feb3c068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37694/0 (socket says 172.21.15.46:37694) 2024-01-27T01:07:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.160+0000 7feb41e3e700 1 -- 172.21.15.46:0/472390710 learned_addr learned my addr 172.21.15.46:0/472390710 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.160+0000 7feb4163d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7feb3c0cd9d0 0x7feb3c0cadb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42474/0 (socket says 172.21.15.46:42474) 2024-01-27T01:07:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.160+0000 7feb41e3e700 1 -- 172.21.15.46:0/472390710 >> v1:172.21.15.106:6789/0 conn(0x7feb3c0cb4c0 legacy=0x7feb3c0c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.160+0000 7feb41e3e700 1 -- 172.21.15.46:0/472390710 >> v1:172.21.15.46:6789/0 conn(0x7feb3c0cd9d0 legacy=0x7feb3c0cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.160+0000 7feb41e3e700 1 -- 172.21.15.46:0/472390710 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3c0c9270 con 0x7feb3c0b08c0 2024-01-27T01:07:42.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.161+0000 7feb41e3e700 1 --2- 172.21.15.46:0/472390710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b08c0 0x7feb3c068a40 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7feb380099b0 tx=0x7feb38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa36af8ecfa8bf21 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:42.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.161+0000 7feb40e3c700 1 -- 172.21.15.46:0/472390710 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3800b860 con 0x7feb3c0b08c0 2024-01-27T01:07:42.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.161+0000 7feb40e3c700 1 -- 172.21.15.46:0/472390710 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3800b9c0 con 0x7feb3c0b08c0 2024-01-27T01:07:42.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.161+0000 7feb40e3c700 1 -- 172.21.15.46:0/472390710 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3800bd80 con 0x7feb3c0b08c0 2024-01-27T01:07:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.161+0000 7feb440a2700 1 -- 172.21.15.46:0/472390710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b08c0 msgr2=0x7feb3c068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.161+0000 7feb440a2700 1 --2- 172.21.15.46:0/472390710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b08c0 0x7feb3c068a40 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7feb380099b0 tx=0x7feb38009250 comp rx=0 tx=0).stop 2024-01-27T01:07:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.161+0000 7feb440a2700 1 -- 172.21.15.46:0/472390710 shutdown_connections 2024-01-27T01:07:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.161+0000 7feb440a2700 1 --2- 172.21.15.46:0/472390710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b08c0 0x7feb3c068a40 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.161+0000 7feb440a2700 1 -- 172.21.15.46:0/472390710 >> 172.21.15.46:0/472390710 conn(0x7feb3c097820 msgr2=0x7feb3c068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.162+0000 7feb440a2700 1 -- 172.21.15.46:0/472390710 shutdown_connections 2024-01-27T01:07:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.162+0000 7feb440a2700 1 -- 172.21.15.46:0/472390710 wait complete. 2024-01-27T01:07:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.163+0000 7feb440a2700 1 Processor -- start 2024-01-27T01:07:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.163+0000 7feb440a2700 1 -- start start 2024-01-27T01:07:42.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb440a2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b08c0 0x7feb3c1710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:42.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb440a2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feb3c0cb4c0 0x7feb3c0b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:42.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb440a2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feb3c0cd9d0 0x7feb3c0c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:42.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb440a2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c1715e0 con 0x7feb3c0b08c0 2024-01-27T01:07:42.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb440a2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c0c67e0 con 0x7feb3c0cb4c0 2024-01-27T01:07:42.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb440a2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c0b19d0 con 0x7feb3c0cd9d0 2024-01-27T01:07:42.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb41e3e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b08c0 0x7feb3c1710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:42.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb41e3e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b08c0 0x7feb3c1710a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37710/0 (socket says 172.21.15.46:37710) 2024-01-27T01:07:42.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb41e3e700 1 -- 172.21.15.46:0/258299047 learned_addr learned my addr 172.21.15.46:0/258299047 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:42.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb4163d700 1 --2- 172.21.15.46:0/258299047 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feb3c0cb4c0 0x7feb3c0b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:42.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb4263f700 1 --2- 172.21.15.46:0/258299047 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feb3c0cd9d0 0x7feb3c0c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:42.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb41e3e700 1 -- 172.21.15.46:0/258299047 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feb3c0cd9d0 msgr2=0x7feb3c0c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb41e3e700 1 --2- 172.21.15.46:0/258299047 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feb3c0cd9d0 0x7feb3c0c71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb41e3e700 1 -- 172.21.15.46:0/258299047 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feb3c0cb4c0 msgr2=0x7feb3c0b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb41e3e700 1 --2- 172.21.15.46:0/258299047 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feb3c0cb4c0 0x7feb3c0b3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb41e3e700 1 -- 172.21.15.46:0/258299047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb380094f0 con 0x7feb3c0b08c0 2024-01-27T01:07:42.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb4163d700 1 --2- 172.21.15.46:0/258299047 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feb3c0cb4c0 0x7feb3c0b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:42.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb41e3e700 1 --2- 172.21.15.46:0/258299047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b08c0 0x7feb3c1710a0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7feb3c06ef50 tx=0x7feb38017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:42.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb31ffb700 1 -- 172.21.15.46:0/258299047 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3801f070 con 0x7feb3c0b08c0 2024-01-27T01:07:42.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb31ffb700 1 -- 172.21.15.46:0/258299047 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb38002f30 con 0x7feb3c0b08c0 2024-01-27T01:07:42.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb440a2700 1 -- 172.21.15.46:0/258299047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb3c0c6500 con 0x7feb3c0b08c0 2024-01-27T01:07:42.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.164+0000 7feb31ffb700 1 -- 172.21.15.46:0/258299047 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb38025420 con 0x7feb3c0b08c0 2024-01-27T01:07:42.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.165+0000 7feb440a2700 1 -- 172.21.15.46:0/258299047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb3c0b1660 con 0x7feb3c0b08c0 2024-01-27T01:07:42.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.166+0000 7feb440a2700 1 -- 172.21.15.46:0/258299047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb3c04c710 con 0x7feb3c0b08c0 2024-01-27T01:07:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.166+0000 7feb31ffb700 1 -- 172.21.15.46:0/258299047 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7feb38023070 con 0x7feb3c0b08c0 2024-01-27T01:07:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.167+0000 7feb31ffb700 1 --2- 172.21.15.46:0/258299047 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feb20067170 0x7feb20069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.167+0000 7feb31ffb700 1 -- 172.21.15.46:0/258299047 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 7381+0+0 (secure 0 0 0) 0x7feb380868a0 con 0x7feb3c0b08c0 2024-01-27T01:07:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.167+0000 7feb4163d700 1 --2- 172.21.15.46:0/258299047 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feb20067170 0x7feb20069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.168+0000 7feb4163d700 1 --2- 172.21.15.46:0/258299047 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feb20067170 0x7feb20069620 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7feb2c007fd0 tx=0x7feb2c007e10 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:42.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.171+0000 7feb31ffb700 1 -- 172.21.15.46:0/258299047 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feb38053270 con 0x7feb3c0b08c0 2024-01-27T01:07:42.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.378+0000 7feb440a2700 1 -- 172.21.15.46:0/258299047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb3c0c9550 con 0x7feb3c0b08c0 2024-01-27T01:07:42.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.379+0000 7feb31ffb700 1 -- 172.21.15.46:0/258299047 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v246) v1 ==== 74+0+19570 (secure 0 0 0) 0x7feb38056890 con 0x7feb3c0b08c0 2024-01-27T01:07:42.379 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:42.379 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":246,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:07:41.410619+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:07:25.886791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-01-27T01:07:26.340041+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:07:39.209497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:07:40.386606+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:07:42.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.381+0000 7feb440a2700 1 -- 172.21.15.46:0/258299047 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feb20067170 msgr2=0x7feb20069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.381+0000 7feb440a2700 1 --2- 172.21.15.46:0/258299047 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feb20067170 0x7feb20069620 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7feb2c007fd0 tx=0x7feb2c007e10 comp rx=0 tx=0).stop 2024-01-27T01:07:42.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.381+0000 7feb440a2700 1 -- 172.21.15.46:0/258299047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b08c0 msgr2=0x7feb3c1710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.381+0000 7feb440a2700 1 --2- 172.21.15.46:0/258299047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b08c0 0x7feb3c1710a0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7feb3c06ef50 tx=0x7feb38017000 comp rx=0 tx=0).stop 2024-01-27T01:07:42.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.382+0000 7feb440a2700 1 -- 172.21.15.46:0/258299047 shutdown_connections 2024-01-27T01:07:42.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.382+0000 7feb440a2700 1 --2- 172.21.15.46:0/258299047 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7feb20067170 0x7feb20069620 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.382+0000 7feb440a2700 1 --2- 172.21.15.46:0/258299047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b08c0 0x7feb3c1710a0 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.382+0000 7feb440a2700 1 --2- 172.21.15.46:0/258299047 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feb3c0cd9d0 0x7feb3c0c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.382+0000 7feb440a2700 1 --2- 172.21.15.46:0/258299047 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feb3c0cb4c0 0x7feb3c0b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.382+0000 7feb440a2700 1 -- 172.21.15.46:0/258299047 >> 172.21.15.46:0/258299047 conn(0x7feb3c097820 msgr2=0x7feb3c0697d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:42.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.382+0000 7feb440a2700 1 -- 172.21.15.46:0/258299047 shutdown_connections 2024-01-27T01:07:42.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.382+0000 7feb440a2700 1 -- 172.21.15.46:0/258299047 wait complete. 2024-01-27T01:07:42.395 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:07:42.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.527+0000 7f0c729f0700 1 Processor -- start 2024-01-27T01:07:42.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.527+0000 7f0c729f0700 1 -- start start 2024-01-27T01:07:42.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.528+0000 7f0c729f0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c6c068040 0x7f0c6c164430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:42.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.528+0000 7f0c729f0700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6c15abc0 con 0x7f0c6c159760 2024-01-27T01:07:42.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.528+0000 7f0c729f0700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6c0b2e90 con 0x7f0c6c159b30 2024-01-27T01:07:42.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.528+0000 7f0c729f0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c159fb0 con 0x7f0c6c068040 2024-01-27T01:07:42.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.528+0000 7f0c6bfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0c6c159760 0x7f0c6c0b0c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42490/0 (socket says 172.21.15.46:42490) 2024-01-27T01:07:42.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.528+0000 7f0c6bfff700 1 -- 172.21.15.46:0/1644792382 learned_addr learned my addr 172.21.15.46:0/1644792382 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:42.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.528+0000 7f0c70f8d700 1 --2- 172.21.15.46:0/1644792382 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c6c068040 0x7f0c6c164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:42.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.528+0000 7f0c6affd700 1 -- 172.21.15.46:0/1644792382 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606355028 0 0) 0x7f0c6c15abc0 con 0x7f0c6c159760 2024-01-27T01:07:42.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.528+0000 7f0c6affd700 1 -- 172.21.15.46:0/1644792382 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c58003570 con 0x7f0c6c159760 2024-01-27T01:07:42.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.528+0000 7f0c6affd700 1 -- 172.21.15.46:0/1644792382 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375881186 0 0) 0x7f0c6c0b2e90 con 0x7f0c6c159b30 2024-01-27T01:07:42.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.528+0000 7f0c6affd700 1 -- 172.21.15.46:0/1644792382 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c6c15abc0 con 0x7f0c6c159b30 2024-01-27T01:07:42.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.529+0000 7f0c70f8d700 1 -- 172.21.15.46:0/1644792382 >> v1:172.21.15.106:6789/0 conn(0x7f0c6c159b30 legacy=0x7f0c6c067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.529+0000 7f0c70f8d700 1 -- 172.21.15.46:0/1644792382 >> v1:172.21.15.46:6789/0 conn(0x7f0c6c159760 legacy=0x7f0c6c0b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.529+0000 7f0c6affd700 1 -- 172.21.15.46:0/1644792382 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 684802645 0 0) 0x7f0c58003570 con 0x7f0c6c159760 2024-01-27T01:07:42.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.529+0000 7f0c70f8d700 1 -- 172.21.15.46:0/1644792382 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c15a290 con 0x7f0c6c068040 2024-01-27T01:07:42.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.529+0000 7f0c70f8d700 1 --2- 172.21.15.46:0/1644792382 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c6c068040 0x7f0c6c164430 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f0c64009e90 tx=0x7f0c64009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b843a1b8df5134ad server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:42.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.529+0000 7f0c6affd700 1 -- 172.21.15.46:0/1644792382 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6400e070 con 0x7f0c6c068040 2024-01-27T01:07:42.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.529+0000 7f0c6affd700 1 -- 172.21.15.46:0/1644792382 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6400bb80 con 0x7f0c6c068040 2024-01-27T01:07:42.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.529+0000 7f0c6affd700 1 -- 172.21.15.46:0/1644792382 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c64012430 con 0x7f0c6c068040 2024-01-27T01:07:42.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.530+0000 7f0c729f0700 1 -- 172.21.15.46:0/1644792382 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c6c068040 msgr2=0x7f0c6c164430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.530+0000 7f0c729f0700 1 --2- 172.21.15.46:0/1644792382 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c6c068040 0x7f0c6c164430 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f0c64009e90 tx=0x7f0c64009780 comp rx=0 tx=0).stop 2024-01-27T01:07:42.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.530+0000 7f0c729f0700 1 -- 172.21.15.46:0/1644792382 shutdown_connections 2024-01-27T01:07:42.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.530+0000 7f0c729f0700 1 --2- 172.21.15.46:0/1644792382 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c6c068040 0x7f0c6c164430 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.530+0000 7f0c729f0700 1 -- 172.21.15.46:0/1644792382 >> 172.21.15.46:0/1644792382 conn(0x7f0c6c06b630 msgr2=0x7f0c6c0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:42.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.530+0000 7f0c729f0700 1 -- 172.21.15.46:0/1644792382 shutdown_connections 2024-01-27T01:07:42.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.530+0000 7f0c729f0700 1 -- 172.21.15.46:0/1644792382 wait complete. 2024-01-27T01:07:42.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.531+0000 7f0c729f0700 1 Processor -- start 2024-01-27T01:07:42.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.531+0000 7f0c729f0700 1 -- start start 2024-01-27T01:07:42.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.532+0000 7f0c729f0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c6c068040 0x7f0c6c15e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:42.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.532+0000 7f0c729f0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c6c159760 0x7f0c6c16caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:42.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.532+0000 7f0c729f0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c6c16d030 0x7f0c6c0cd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:42.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.532+0000 7f0c729f0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c16d490 con 0x7f0c6c16d030 2024-01-27T01:07:42.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.532+0000 7f0c729f0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c15f8d0 con 0x7f0c6c068040 2024-01-27T01:07:42.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.532+0000 7f0c729f0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c15ecc0 con 0x7f0c6c159760 2024-01-27T01:07:42.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.532+0000 7f0c70f8d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c6c16d030 0x7f0c6c0cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:42.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.532+0000 7f0c70f8d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c6c16d030 0x7f0c6c0cd900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37716/0 (socket says 172.21.15.46:37716) 2024-01-27T01:07:42.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.532+0000 7f0c70f8d700 1 -- 172.21.15.46:0/454008618 learned_addr learned my addr 172.21.15.46:0/454008618 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:42.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.532+0000 7f0c6bfff700 1 --2- 172.21.15.46:0/454008618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c6c068040 0x7f0c6c15e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:42.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.532+0000 7f0c6b7fe700 1 --2- 172.21.15.46:0/454008618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c6c159760 0x7f0c6c16caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:42.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.532+0000 7f0c70f8d700 1 -- 172.21.15.46:0/454008618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c6c159760 msgr2=0x7f0c6c16caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.532+0000 7f0c70f8d700 1 --2- 172.21.15.46:0/454008618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c6c159760 0x7f0c6c16caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.532+0000 7f0c70f8d700 1 -- 172.21.15.46:0/454008618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c6c068040 msgr2=0x7f0c6c15e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.532+0000 7f0c70f8d700 1 --2- 172.21.15.46:0/454008618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c6c068040 0x7f0c6c15e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.532+0000 7f0c70f8d700 1 -- 172.21.15.46:0/454008618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c64009a20 con 0x7f0c6c16d030 2024-01-27T01:07:42.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.533+0000 7f0c70f8d700 1 --2- 172.21.15.46:0/454008618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c6c16d030 0x7f0c6c0cd900 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0ce590 tx=0x7f0c64019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:42.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.533+0000 7f0c63fff700 1 -- 172.21.15.46:0/454008618 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c64007000 con 0x7f0c6c16d030 2024-01-27T01:07:42.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.533+0000 7f0c63fff700 1 -- 172.21.15.46:0/454008618 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c64012e40 con 0x7f0c6c16d030 2024-01-27T01:07:42.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.533+0000 7f0c729f0700 1 -- 172.21.15.46:0/454008618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6c15efa0 con 0x7f0c6c16d030 2024-01-27T01:07:42.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.533+0000 7f0c63fff700 1 -- 172.21.15.46:0/454008618 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6401b830 con 0x7f0c6c16d030 2024-01-27T01:07:42.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.533+0000 7f0c729f0700 1 -- 172.21.15.46:0/454008618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6c15f560 con 0x7f0c6c16d030 2024-01-27T01:07:42.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.534+0000 7f0c729f0700 1 -- 172.21.15.46:0/454008618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c6c064b00 con 0x7f0c6c16d030 2024-01-27T01:07:42.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.534+0000 7f0c63fff700 1 -- 172.21.15.46:0/454008618 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f0c6401bc60 con 0x7f0c6c16d030 2024-01-27T01:07:42.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.535+0000 7f0c63fff700 1 --2- 172.21.15.46:0/454008618 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0c4c067170 0x7f0c4c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:42.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.535+0000 7f0c63fff700 1 -- 172.21.15.46:0/454008618 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f0c640870f0 con 0x7f0c6c16d030 2024-01-27T01:07:42.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.535+0000 7f0c6bfff700 1 --2- 172.21.15.46:0/454008618 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0c4c067170 0x7f0c4c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:42.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.536+0000 7f0c6bfff700 1 --2- 172.21.15.46:0/454008618 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0c4c067170 0x7f0c4c069620 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0aa0d0 tx=0x7f0c54005950 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:42.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.539+0000 7f0c63fff700 1 -- 172.21.15.46:0/454008618 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0c640573c0 con 0x7f0c6c16d030 2024-01-27T01:07:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.747+0000 7f0c729f0700 1 -- 172.21.15.46:0/454008618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c6c04c710 con 0x7f0c6c16d030 2024-01-27T01:07:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.748+0000 7f0c63fff700 1 -- 172.21.15.46:0/454008618 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v246) v1 ==== 74+0+19570 (secure 0 0 0) 0x7f0c64056f50 con 0x7f0c6c16d030 2024-01-27T01:07:42.748 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:42.748 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":246,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:07:41.410619+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:07:25.886791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-01-27T01:07:26.340041+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:07:39.209497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:07:40.386606+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:07:42.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.750+0000 7f0c729f0700 1 -- 172.21.15.46:0/454008618 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0c4c067170 msgr2=0x7f0c4c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.750+0000 7f0c729f0700 1 --2- 172.21.15.46:0/454008618 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0c4c067170 0x7f0c4c069620 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0aa0d0 tx=0x7f0c54005950 comp rx=0 tx=0).stop 2024-01-27T01:07:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.750+0000 7f0c729f0700 1 -- 172.21.15.46:0/454008618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c6c16d030 msgr2=0x7f0c6c0cd900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.750+0000 7f0c729f0700 1 --2- 172.21.15.46:0/454008618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c6c16d030 0x7f0c6c0cd900 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0ce590 tx=0x7f0c64019000 comp rx=0 tx=0).stop 2024-01-27T01:07:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.750+0000 7f0c729f0700 1 -- 172.21.15.46:0/454008618 shutdown_connections 2024-01-27T01:07:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.750+0000 7f0c729f0700 1 --2- 172.21.15.46:0/454008618 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0c4c067170 0x7f0c4c069620 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.750+0000 7f0c729f0700 1 --2- 172.21.15.46:0/454008618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c6c16d030 0x7f0c6c0cd900 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.750+0000 7f0c729f0700 1 --2- 172.21.15.46:0/454008618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c6c068040 0x7f0c6c15e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.750+0000 7f0c729f0700 1 --2- 172.21.15.46:0/454008618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c6c159760 0x7f0c6c16caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.750+0000 7f0c729f0700 1 -- 172.21.15.46:0/454008618 >> 172.21.15.46:0/454008618 conn(0x7f0c6c06b630 msgr2=0x7f0c6c06c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.750+0000 7f0c729f0700 1 -- 172.21.15.46:0/454008618 shutdown_connections 2024-01-27T01:07:42.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.750+0000 7f0c729f0700 1 -- 172.21.15.46:0/454008618 wait complete. 2024-01-27T01:07:42.764 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-01-27T01:07:42.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.899+0000 7f6739057700 1 Processor -- start 2024-01-27T01:07:42.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.899+0000 7f6739057700 1 -- start start 2024-01-27T01:07:42.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.900+0000 7f6739057700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f67341565a0 0x7f6734156970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:42.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.900+0000 7f6739057700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f67340ae2f0 0x7f67340ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:42.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.900+0000 7f6739057700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67340b0830 con 0x7f673415a520 2024-01-27T01:07:42.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.900+0000 7f6739057700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67340ae130 con 0x7f67340ae2f0 2024-01-27T01:07:42.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.900+0000 7f6739057700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67340ad350 con 0x7f67341565a0 2024-01-27T01:07:42.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.900+0000 7f673359e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f67340ae2f0 0x7f67340ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:42.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.900+0000 7f673259c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f673415a520 0x7f67340acc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42498/0 (socket says 172.21.15.46:42498) 2024-01-27T01:07:42.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.900+0000 7f673359e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f67340ae2f0 0x7f67340ae700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37732/0 (socket says 172.21.15.46:37732) 2024-01-27T01:07:42.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.900+0000 7f673259c700 1 -- 172.21.15.46:0/4047032034 learned_addr learned my addr 172.21.15.46:0/4047032034 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:42.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.900+0000 7f6732d9d700 1 --2- 172.21.15.46:0/4047032034 >> v2:172.21.15.106:3300/0 conn(0x7f67341565a0 0x7f6734156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:42.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.900+0000 7f673359e700 1 -- 172.21.15.46:0/4047032034 >> v2:172.21.15.106:3300/0 conn(0x7f67341565a0 msgr2=0x7f6734156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:42.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.900+0000 7f673359e700 1 --2- 172.21.15.46:0/4047032034 >> v2:172.21.15.106:3300/0 conn(0x7f67341565a0 0x7f6734156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.900+0000 7f673359e700 1 -- 172.21.15.46:0/4047032034 >> v1:172.21.15.46:6789/0 conn(0x7f673415a520 legacy=0x7f67340acc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.900+0000 7f6731d9b700 1 -- 172.21.15.46:0/4047032034 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691345795 0 0) 0x7f67340b0830 con 0x7f673415a520 2024-01-27T01:07:42.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.900+0000 7f673359e700 1 -- 172.21.15.46:0/4047032034 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67340aec40 con 0x7f67340ae2f0 2024-01-27T01:07:42.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.900+0000 7f6732d9d700 1 --2- 172.21.15.46:0/4047032034 >> v2:172.21.15.106:3300/0 conn(0x7f67341565a0 0x7f6734156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:07:42.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.901+0000 7f673359e700 1 --2- 172.21.15.46:0/4047032034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67340ae2f0 0x7f67340ae700 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f6734064a50 tx=0x7f672c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be80cdb17a801a68 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:42.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.901+0000 7f6731d9b700 1 -- 172.21.15.46:0/4047032034 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c00e070 con 0x7f67340ae2f0 2024-01-27T01:07:42.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.901+0000 7f6731d9b700 1 -- 172.21.15.46:0/4047032034 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672c00bc30 con 0x7f67340ae2f0 2024-01-27T01:07:42.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.901+0000 7f6731d9b700 1 -- 172.21.15.46:0/4047032034 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c012430 con 0x7f67340ae2f0 2024-01-27T01:07:42.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.901+0000 7f6739057700 1 -- 172.21.15.46:0/4047032034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67340ae2f0 msgr2=0x7f67340ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.901+0000 7f6739057700 1 --2- 172.21.15.46:0/4047032034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67340ae2f0 0x7f67340ae700 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f6734064a50 tx=0x7f672c009780 comp rx=0 tx=0).stop 2024-01-27T01:07:42.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.901+0000 7f6739057700 1 -- 172.21.15.46:0/4047032034 shutdown_connections 2024-01-27T01:07:42.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.901+0000 7f6739057700 1 --2- 172.21.15.46:0/4047032034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67340ae2f0 0x7f67340ae700 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.901+0000 7f6739057700 1 --2- 172.21.15.46:0/4047032034 >> v2:172.21.15.106:3300/0 conn(0x7f67341565a0 0x7f6734156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.901+0000 7f6739057700 1 -- 172.21.15.46:0/4047032034 >> 172.21.15.46:0/4047032034 conn(0x7f67340cdf80 msgr2=0x7f673406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:42.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.901+0000 7f6739057700 1 -- 172.21.15.46:0/4047032034 shutdown_connections 2024-01-27T01:07:42.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.901+0000 7f6739057700 1 -- 172.21.15.46:0/4047032034 wait complete. 2024-01-27T01:07:42.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.903+0000 7f6739057700 1 Processor -- start 2024-01-27T01:07:42.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.903+0000 7f6739057700 1 -- start start 2024-01-27T01:07:42.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.903+0000 7f6739057700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67340ae2f0 0x7f6734144020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:42.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f6739057700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67341565a0 0x7f6734151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:42.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f6739057700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f673415a520 0x7f6734155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:42.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f6739057700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6734151be0 con 0x7f67340ae2f0 2024-01-27T01:07:42.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f6739057700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6734151ec0 con 0x7f67341565a0 2024-01-27T01:07:42.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f6739057700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67341521a0 con 0x7f673415a520 2024-01-27T01:07:42.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f6732d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67340ae2f0 0x7f6734144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:42.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f6732d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67340ae2f0 0x7f6734144020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37736/0 (socket says 172.21.15.46:37736) 2024-01-27T01:07:42.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f6732d9d700 1 -- 172.21.15.46:0/452247611 learned_addr learned my addr 172.21.15.46:0/452247611 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:42.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f673259c700 1 --2- 172.21.15.46:0/452247611 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67341565a0 0x7f6734151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:42.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f673359e700 1 --2- 172.21.15.46:0/452247611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f673415a520 0x7f6734155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:42.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f6732d9d700 1 -- 172.21.15.46:0/452247611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f673415a520 msgr2=0x7f6734155110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f6732d9d700 1 --2- 172.21.15.46:0/452247611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f673415a520 0x7f6734155110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f6732d9d700 1 -- 172.21.15.46:0/452247611 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67341565a0 msgr2=0x7f6734151270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:42.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f6732d9d700 1 --2- 172.21.15.46:0/452247611 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67341565a0 0x7f6734151270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:42.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f6732d9d700 1 -- 172.21.15.46:0/452247611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c009a20 con 0x7f67340ae2f0 2024-01-27T01:07:42.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f6732d9d700 1 --2- 172.21.15.46:0/452247611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67340ae2f0 0x7f6734144020 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f671c00c830 tx=0x7f671c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:42.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f672affd700 1 -- 172.21.15.46:0/452247611 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c011600 con 0x7f67340ae2f0 2024-01-27T01:07:42.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f672affd700 1 -- 172.21.15.46:0/452247611 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671c011760 con 0x7f67340ae2f0 2024-01-27T01:07:42.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f6739057700 1 -- 172.21.15.46:0/452247611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67340a9f90 con 0x7f67340ae2f0 2024-01-27T01:07:42.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f672affd700 1 -- 172.21.15.46:0/452247611 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c011b20 con 0x7f67340ae2f0 2024-01-27T01:07:42.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.904+0000 7f6739057700 1 -- 172.21.15.46:0/452247611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6734144560 con 0x7f67340ae2f0 2024-01-27T01:07:42.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.906+0000 7f6739057700 1 -- 172.21.15.46:0/452247611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67340a9d90 con 0x7f67340ae2f0 2024-01-27T01:07:42.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.906+0000 7f672affd700 1 -- 172.21.15.46:0/452247611 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f671c0193b0 con 0x7f67340ae2f0 2024-01-27T01:07:42.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.906+0000 7f672affd700 1 --2- 172.21.15.46:0/452247611 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6714067170 0x7f6714069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:42.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.906+0000 7f672affd700 1 -- 172.21.15.46:0/452247611 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f671c084fd0 con 0x7f67340ae2f0 2024-01-27T01:07:42.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.907+0000 7f673259c700 1 --2- 172.21.15.46:0/452247611 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6714067170 0x7f6714069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:42.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.907+0000 7f673259c700 1 --2- 172.21.15.46:0/452247611 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6714067170 0x7f6714069620 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f67340a99c0 tx=0x7f6724007d30 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:42.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:42.911+0000 7f672affd700 1 -- 172.21.15.46:0/452247611 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f671c051810 con 0x7f67340ae2f0 2024-01-27T01:07:43.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:43.173+0000 7f6739057700 1 -- 172.21.15.46:0/452247611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f673404c710 con 0x7f67340ae2f0 2024-01-27T01:07:44.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.003+0000 7f672affd700 1 -- 172.21.15.46:0/452247611 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v137) v1 ==== 121+0+0 (secure 0 0 0) 0x7f671c054e30 con 0x7f67340ae2f0 2024-01-27T01:07:44.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.006+0000 7f6739057700 1 -- 172.21.15.46:0/452247611 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6714067170 msgr2=0x7f6714069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:44.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.006+0000 7f6739057700 1 --2- 172.21.15.46:0/452247611 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6714067170 0x7f6714069620 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f67340a99c0 tx=0x7f6724007d30 comp rx=0 tx=0).stop 2024-01-27T01:07:44.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.006+0000 7f6739057700 1 -- 172.21.15.46:0/452247611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67340ae2f0 msgr2=0x7f6734144020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:44.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.006+0000 7f6739057700 1 --2- 172.21.15.46:0/452247611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67340ae2f0 0x7f6734144020 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f671c00c830 tx=0x7f671c008040 comp rx=0 tx=0).stop 2024-01-27T01:07:44.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.006+0000 7f6739057700 1 -- 172.21.15.46:0/452247611 shutdown_connections 2024-01-27T01:07:44.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.006+0000 7f6739057700 1 --2- 172.21.15.46:0/452247611 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f6714067170 0x7f6714069620 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:44.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.006+0000 7f6739057700 1 --2- 172.21.15.46:0/452247611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67340ae2f0 0x7f6734144020 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:44.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.006+0000 7f6739057700 1 --2- 172.21.15.46:0/452247611 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f673415a520 0x7f6734155110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:44.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.006+0000 7f6739057700 1 --2- 172.21.15.46:0/452247611 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f67341565a0 0x7f6734151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:44.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.006+0000 7f6739057700 1 -- 172.21.15.46:0/452247611 >> 172.21.15.46:0/452247611 conn(0x7f67340cdf80 msgr2=0x7f6734152ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:44.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.006+0000 7f6739057700 1 -- 172.21.15.46:0/452247611 shutdown_connections 2024-01-27T01:07:44.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.006+0000 7f6739057700 1 -- 172.21.15.46:0/452247611 wait complete. 2024-01-27T01:07:44.022 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-01-27T01:07:44.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.161+0000 7f23fb6c5700 1 Processor -- start 2024-01-27T01:07:44.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.161+0000 7f23fb6c5700 1 -- start start 2024-01-27T01:07:44.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.162+0000 7f23fb6c5700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f23f40c93a0 0x7f23f40c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:44.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.162+0000 7f23fb6c5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f23f4067d50 0x7f23f4159220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:44.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.162+0000 7f23fb6c5700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23f415e390 con 0x7f23f40681c0 2024-01-27T01:07:44.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.162+0000 7f23fb6c5700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f415d810 con 0x7f23f40c93a0 2024-01-27T01:07:44.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.162+0000 7f23fb6c5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f415daf0 con 0x7f23f4067d50 2024-01-27T01:07:44.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.162+0000 7f23f9461700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f23f40c93a0 0x7f23f40c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:44.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.162+0000 7f23f9c62700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f23f40681c0 0x7f23f4159760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42510/0 (socket says 172.21.15.46:42510) 2024-01-27T01:07:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.162+0000 7f23f9c62700 1 -- 172.21.15.46:0/1405962114 learned_addr learned my addr 172.21.15.46:0/1405962114 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.162+0000 7f23f9461700 1 --2- 172.21.15.46:0/1405962114 >> v2:172.21.15.46:3300/0 conn(0x7f23f40c93a0 0x7f23f40c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37740/0 (socket says 172.21.15.46:37740) 2024-01-27T01:07:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.162+0000 7f23f8c60700 1 --2- 172.21.15.46:0/1405962114 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f23f4067d50 0x7f23f4159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.163+0000 7f23f9461700 1 -- 172.21.15.46:0/1405962114 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f23f4067d50 msgr2=0x7f23f4159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.163+0000 7f23f9461700 1 --2- 172.21.15.46:0/1405962114 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f23f4067d50 0x7f23f4159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.163+0000 7f23f9461700 1 -- 172.21.15.46:0/1405962114 >> v1:172.21.15.46:6789/0 conn(0x7f23f40681c0 legacy=0x7f23f4159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.163+0000 7f23f9461700 1 -- 172.21.15.46:0/1405962114 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f415ddd0 con 0x7f23f40c93a0 2024-01-27T01:07:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.163+0000 7f23ebfff700 1 -- 172.21.15.46:0/1405962114 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280450147 0 0) 0x7f23f415e390 con 0x7f23f40681c0 2024-01-27T01:07:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.163+0000 7f23f9461700 1 --2- 172.21.15.46:0/1405962114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23f40c93a0 0x7f23f40c9770 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f23f00098e0 tx=0x7f23f00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed424b1c819d113e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.163+0000 7f23ebfff700 1 -- 172.21.15.46:0/1405962114 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f0009e80 con 0x7f23f40c93a0 2024-01-27T01:07:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.163+0000 7f23ebfff700 1 -- 172.21.15.46:0/1405962114 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f000b860 con 0x7f23f40c93a0 2024-01-27T01:07:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.163+0000 7f23ebfff700 1 -- 172.21.15.46:0/1405962114 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f000bbe0 con 0x7f23f40c93a0 2024-01-27T01:07:44.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.164+0000 7f23fb6c5700 1 -- 172.21.15.46:0/1405962114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23f40c93a0 msgr2=0x7f23f40c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:44.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.164+0000 7f23fb6c5700 1 --2- 172.21.15.46:0/1405962114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23f40c93a0 0x7f23f40c9770 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f23f00098e0 tx=0x7f23f00091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:44.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.164+0000 7f23fb6c5700 1 -- 172.21.15.46:0/1405962114 shutdown_connections 2024-01-27T01:07:44.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.164+0000 7f23fb6c5700 1 --2- 172.21.15.46:0/1405962114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23f40c93a0 0x7f23f40c9770 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:44.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.164+0000 7f23fb6c5700 1 --2- 172.21.15.46:0/1405962114 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f23f4067d50 0x7f23f4159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:44.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.164+0000 7f23fb6c5700 1 -- 172.21.15.46:0/1405962114 >> 172.21.15.46:0/1405962114 conn(0x7f23f40ca9a0 msgr2=0x7f23f4068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:44.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.164+0000 7f23fb6c5700 1 -- 172.21.15.46:0/1405962114 shutdown_connections 2024-01-27T01:07:44.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.164+0000 7f23fb6c5700 1 -- 172.21.15.46:0/1405962114 wait complete. 2024-01-27T01:07:44.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.165+0000 7f23fb6c5700 1 Processor -- start 2024-01-27T01:07:44.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.165+0000 7f23fb6c5700 1 -- start start 2024-01-27T01:07:44.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.166+0000 7f23fb6c5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f23f4067d50 0x7f23f41558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:44.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.166+0000 7f23fb6c5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23f40681c0 0x7f23f4154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:44.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.166+0000 7f23fb6c5700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f23f40c93a0 0x7f23f40c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:44.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.166+0000 7f23fb6c5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f41699f0 con 0x7f23f40681c0 2024-01-27T01:07:44.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.166+0000 7f23fb6c5700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f4155e00 con 0x7f23f4067d50 2024-01-27T01:07:44.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.166+0000 7f23fb6c5700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f41560a0 con 0x7f23f40c93a0 2024-01-27T01:07:44.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.166+0000 7f23f8c60700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23f40681c0 0x7f23f4154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:44.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.166+0000 7f23f8c60700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23f40681c0 0x7f23f4154f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37750/0 (socket says 172.21.15.46:37750) 2024-01-27T01:07:44.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.166+0000 7f23f8c60700 1 -- 172.21.15.46:0/1308565090 learned_addr learned my addr 172.21.15.46:0/1308565090 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:44.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.166+0000 7f23f9c62700 1 --2- 172.21.15.46:0/1308565090 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f23f40c93a0 0x7f23f40c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:44.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.166+0000 7f23f9461700 1 --2- 172.21.15.46:0/1308565090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f23f4067d50 0x7f23f41558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:44.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.166+0000 7f23f8c60700 1 -- 172.21.15.46:0/1308565090 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f23f40c93a0 msgr2=0x7f23f40c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:44.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.166+0000 7f23f8c60700 1 --2- 172.21.15.46:0/1308565090 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f23f40c93a0 0x7f23f40c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:44.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.166+0000 7f23f8c60700 1 -- 172.21.15.46:0/1308565090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f23f4067d50 msgr2=0x7f23f41558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:44.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.166+0000 7f23f8c60700 1 --2- 172.21.15.46:0/1308565090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f23f4067d50 0x7f23f41558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:44.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.166+0000 7f23f8c60700 1 -- 172.21.15.46:0/1308565090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f0009420 con 0x7f23f40681c0 2024-01-27T01:07:44.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.167+0000 7f23f8c60700 1 --2- 172.21.15.46:0/1308565090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23f40681c0 0x7f23f4154f00 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f23e400d890 tx=0x7f23e4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:44.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.167+0000 7f23e97fa700 1 -- 172.21.15.46:0/1308565090 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e400b600 con 0x7f23f40681c0 2024-01-27T01:07:44.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.167+0000 7f23e97fa700 1 -- 172.21.15.46:0/1308565090 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e400b760 con 0x7f23f40681c0 2024-01-27T01:07:44.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.167+0000 7f23fb6c5700 1 -- 172.21.15.46:0/1308565090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f4156380 con 0x7f23f40681c0 2024-01-27T01:07:44.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.167+0000 7f23e97fa700 1 -- 172.21.15.46:0/1308565090 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e400bae0 con 0x7f23f40681c0 2024-01-27T01:07:44.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.167+0000 7f23fb6c5700 1 -- 172.21.15.46:0/1308565090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f4156940 con 0x7f23f40681c0 2024-01-27T01:07:44.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.168+0000 7f23fb6c5700 1 -- 172.21.15.46:0/1308565090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f4169d20 con 0x7f23f40681c0 2024-01-27T01:07:44.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.168+0000 7f23e97fa700 1 -- 172.21.15.46:0/1308565090 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f23e401e460 con 0x7f23f40681c0 2024-01-27T01:07:44.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.169+0000 7f23e97fa700 1 --2- 172.21.15.46:0/1308565090 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23d8067170 0x7f23d8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:44.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.169+0000 7f23e97fa700 1 -- 172.21.15.46:0/1308565090 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f23e40855c0 con 0x7f23f40681c0 2024-01-27T01:07:44.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.169+0000 7f23f9461700 1 --2- 172.21.15.46:0/1308565090 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23d8067170 0x7f23d8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:44.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.170+0000 7f23f9461700 1 --2- 172.21.15.46:0/1308565090 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23d8067170 0x7f23d8069620 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f23f0003080 tx=0x7f23f0017000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:44.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.173+0000 7f23e97fa700 1 -- 172.21.15.46:0/1308565090 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f23e4051f10 con 0x7f23f40681c0 2024-01-27T01:07:44.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:44.428+0000 7f23fb6c5700 1 -- 172.21.15.46:0/1308565090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f23f4169b30 con 0x7f23f40681c0 2024-01-27T01:07:44.922 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-01-27T01:07:44.922+0000 7f45e261c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-27T01:07:44.981 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:07:44.984 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:07:45.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.005+0000 7f23e97fa700 1 -- 172.21.15.46:0/1308565090 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v138) v1 ==== 125+0+0 (secure 0 0 0) 0x7f23e4055530 con 0x7f23f40681c0 2024-01-27T01:07:45.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.008+0000 7f23fb6c5700 1 -- 172.21.15.46:0/1308565090 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23d8067170 msgr2=0x7f23d8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:45.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.008+0000 7f23fb6c5700 1 --2- 172.21.15.46:0/1308565090 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23d8067170 0x7f23d8069620 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f23f0003080 tx=0x7f23f0017000 comp rx=0 tx=0).stop 2024-01-27T01:07:45.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.008+0000 7f23fb6c5700 1 -- 172.21.15.46:0/1308565090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23f40681c0 msgr2=0x7f23f4154f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:45.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.008+0000 7f23fb6c5700 1 --2- 172.21.15.46:0/1308565090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23f40681c0 0x7f23f4154f00 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f23e400d890 tx=0x7f23e4008040 comp rx=0 tx=0).stop 2024-01-27T01:07:45.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.008+0000 7f23fb6c5700 1 -- 172.21.15.46:0/1308565090 shutdown_connections 2024-01-27T01:07:45.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.008+0000 7f23fb6c5700 1 --2- 172.21.15.46:0/1308565090 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23d8067170 0x7f23d8069620 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.008+0000 7f23fb6c5700 1 --2- 172.21.15.46:0/1308565090 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f23f40c93a0 0x7f23f40c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.008+0000 7f23fb6c5700 1 --2- 172.21.15.46:0/1308565090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f23f4067d50 0x7f23f41558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.008+0000 7f23fb6c5700 1 --2- 172.21.15.46:0/1308565090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23f40681c0 0x7f23f4154f00 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.008+0000 7f23fb6c5700 1 -- 172.21.15.46:0/1308565090 >> 172.21.15.46:0/1308565090 conn(0x7f23f40ca9a0 msgr2=0x7f23f415fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:45.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.008+0000 7f23fb6c5700 1 -- 172.21.15.46:0/1308565090 shutdown_connections 2024-01-27T01:07:45.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.008+0000 7f23fb6c5700 1 -- 172.21.15.46:0/1308565090 wait complete. 2024-01-27T01:07:45.025 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:07:45.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.236+0000 7fdb75492700 1 Processor -- start 2024-01-27T01:07:45.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.236+0000 7fdb75492700 1 -- start start 2024-01-27T01:07:45.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.238+0000 7fdb75492700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb700b2b40 0x7fdb700ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:45.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.238+0000 7fdb75492700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdb700ae960 0x7fdb700ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:45.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.238+0000 7fdb75492700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdb700aedd0 0x7fdb700cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:45.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.238+0000 7fdb75492700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb70099fc0 con 0x7fdb700ae960 2024-01-27T01:07:45.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.238+0000 7fdb75492700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb7015a0b0 con 0x7fdb700aedd0 2024-01-27T01:07:45.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.238+0000 7fdb75492700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb700ae4d0 con 0x7fdb700b2b40 2024-01-27T01:07:45.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.238+0000 7fdb6e7fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdb700ae960 0x7fdb700ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:45.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.238+0000 7fdb6e7fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdb700ae960 0x7fdb700ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37760/0 (socket says 172.21.15.46:37760) 2024-01-27T01:07:45.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.238+0000 7fdb6e7fc700 1 -- 172.21.15.46:0/2619159779 learned_addr learned my addr 172.21.15.46:0/2619159779 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:45.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.238+0000 7fdb6f7fe700 1 --2- 172.21.15.46:0/2619159779 >> v2:172.21.15.106:3300/0 conn(0x7fdb700aedd0 0x7fdb700cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:45.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.238+0000 7fdb6effd700 1 --2- 172.21.15.46:0/2619159779 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb700b2b40 0x7fdb700ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:45.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.238+0000 7fdb6e7fc700 1 -- 172.21.15.46:0/2619159779 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb700b2b40 msgr2=0x7fdb700ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:45.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.238+0000 7fdb6e7fc700 1 --2- 172.21.15.46:0/2619159779 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb700b2b40 0x7fdb700ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.238+0000 7fdb6e7fc700 1 -- 172.21.15.46:0/2619159779 >> v2:172.21.15.106:3300/0 conn(0x7fdb700aedd0 msgr2=0x7fdb700cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:45.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.238+0000 7fdb6e7fc700 1 --2- 172.21.15.46:0/2619159779 >> v2:172.21.15.106:3300/0 conn(0x7fdb700aedd0 0x7fdb700cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.238+0000 7fdb6e7fc700 1 -- 172.21.15.46:0/2619159779 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb700ad8c0 con 0x7fdb700ae960 2024-01-27T01:07:45.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.238+0000 7fdb6e7fc700 1 --2- 172.21.15.46:0/2619159779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb700ae960 0x7fdb700ced50 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fdb700ae440 tx=0x7fdb64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e5a518ebdfeb94 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:45.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.239+0000 7fdb6dffb700 1 -- 172.21.15.46:0/2619159779 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6400b860 con 0x7fdb700ae960 2024-01-27T01:07:45.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.239+0000 7fdb6dffb700 1 -- 172.21.15.46:0/2619159779 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6400b9c0 con 0x7fdb700ae960 2024-01-27T01:07:45.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.239+0000 7fdb6dffb700 1 -- 172.21.15.46:0/2619159779 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6400bd60 con 0x7fdb700ae960 2024-01-27T01:07:45.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.239+0000 7fdb75492700 1 -- 172.21.15.46:0/2619159779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb700ae960 msgr2=0x7fdb700ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:45.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.239+0000 7fdb75492700 1 --2- 172.21.15.46:0/2619159779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb700ae960 0x7fdb700ced50 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fdb700ae440 tx=0x7fdb64009250 comp rx=0 tx=0).stop 2024-01-27T01:07:45.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.239+0000 7fdb75492700 1 -- 172.21.15.46:0/2619159779 shutdown_connections 2024-01-27T01:07:45.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.239+0000 7fdb75492700 1 --2- 172.21.15.46:0/2619159779 >> v2:172.21.15.106:3300/0 conn(0x7fdb700aedd0 0x7fdb700cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.239+0000 7fdb75492700 1 --2- 172.21.15.46:0/2619159779 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb700b2b40 0x7fdb700ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.239+0000 7fdb75492700 1 --2- 172.21.15.46:0/2619159779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb700ae960 0x7fdb700ced50 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.239+0000 7fdb75492700 1 -- 172.21.15.46:0/2619159779 >> 172.21.15.46:0/2619159779 conn(0x7fdb70099a30 msgr2=0x7fdb7006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:45.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.239+0000 7fdb75492700 1 -- 172.21.15.46:0/2619159779 shutdown_connections 2024-01-27T01:07:45.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.239+0000 7fdb75492700 1 -- 172.21.15.46:0/2619159779 wait complete. 2024-01-27T01:07:45.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.241+0000 7fdb75492700 1 Processor -- start 2024-01-27T01:07:45.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.241+0000 7fdb75492700 1 -- start start 2024-01-27T01:07:45.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.242+0000 7fdb75492700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb700ae960 0x7fdb7016d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:45.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.242+0000 7fdb75492700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb700b2b40 0x7fdb7015e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:45.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.242+0000 7fdb75492700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb700b5190 0x7fdb700b3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:45.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.242+0000 7fdb75492700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7015e7c0 con 0x7fdb700ae960 2024-01-27T01:07:45.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.242+0000 7fdb75492700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7016dc00 con 0x7fdb700b2b40 2024-01-27T01:07:45.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.242+0000 7fdb75492700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7016dee0 con 0x7fdb700b5190 2024-01-27T01:07:45.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.242+0000 7fdb6effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb700ae960 0x7fdb7016d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:45.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.243+0000 7fdb6effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb700ae960 0x7fdb7016d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37764/0 (socket says 172.21.15.46:37764) 2024-01-27T01:07:45.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.243+0000 7fdb6effd700 1 -- 172.21.15.46:0/430776535 learned_addr learned my addr 172.21.15.46:0/430776535 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:45.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.243+0000 7fdb6e7fc700 1 --2- 172.21.15.46:0/430776535 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb700b2b40 0x7fdb7015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:45.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.243+0000 7fdb6f7fe700 1 --2- 172.21.15.46:0/430776535 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb700b5190 0x7fdb700b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:45.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.243+0000 7fdb6effd700 1 -- 172.21.15.46:0/430776535 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb700b5190 msgr2=0x7fdb700b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:45.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.243+0000 7fdb6effd700 1 --2- 172.21.15.46:0/430776535 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb700b5190 0x7fdb700b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.243+0000 7fdb6effd700 1 -- 172.21.15.46:0/430776535 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb700b2b40 msgr2=0x7fdb7015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:45.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.243+0000 7fdb6effd700 1 --2- 172.21.15.46:0/430776535 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb700b2b40 0x7fdb7015e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.243+0000 7fdb6effd700 1 -- 172.21.15.46:0/430776535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb640094f0 con 0x7fdb700ae960 2024-01-27T01:07:45.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.243+0000 7fdb6effd700 1 --2- 172.21.15.46:0/430776535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb700ae960 0x7fdb7016d570 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fdb6000d810 tx=0x7fdb60008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:45.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.243+0000 7fdb56ffd700 1 -- 172.21.15.46:0/430776535 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb600125f0 con 0x7fdb700ae960 2024-01-27T01:07:45.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.243+0000 7fdb56ffd700 1 -- 172.21.15.46:0/430776535 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb60012750 con 0x7fdb700ae960 2024-01-27T01:07:45.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.243+0000 7fdb56ffd700 1 -- 172.21.15.46:0/430776535 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb60012b10 con 0x7fdb700ae960 2024-01-27T01:07:45.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.243+0000 7fdb75492700 1 -- 172.21.15.46:0/430776535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb7016e1c0 con 0x7fdb700ae960 2024-01-27T01:07:45.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.243+0000 7fdb75492700 1 -- 172.21.15.46:0/430776535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb700b5600 con 0x7fdb700ae960 2024-01-27T01:07:45.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.245+0000 7fdb75492700 1 -- 172.21.15.46:0/430776535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb7015c280 con 0x7fdb700ae960 2024-01-27T01:07:45.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.245+0000 7fdb56ffd700 1 -- 172.21.15.46:0/430776535 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fdb6001e460 con 0x7fdb700ae960 2024-01-27T01:07:45.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.245+0000 7fdb56ffd700 1 --2- 172.21.15.46:0/430776535 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdb5c067170 0x7fdb5c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:45.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.245+0000 7fdb56ffd700 1 -- 172.21.15.46:0/430776535 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 7381+0+0 (secure 0 0 0) 0x7fdb600854e0 con 0x7fdb700ae960 2024-01-27T01:07:45.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.246+0000 7fdb6e7fc700 1 --2- 172.21.15.46:0/430776535 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdb5c067170 0x7fdb5c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:45.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.246+0000 7fdb6e7fc700 1 --2- 172.21.15.46:0/430776535 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdb5c067170 0x7fdb5c069620 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdb700ae440 tx=0x7fdb64017000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:45.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.250+0000 7fdb56ffd700 1 -- 172.21.15.46:0/430776535 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdb7015c280 con 0x7fdb700ae960 2024-01-27T01:07:45.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.503+0000 7fdb75492700 1 -- 172.21.15.46:0/430776535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb7015c280 con 0x7fdb700ae960 2024-01-27T01:07:45.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.504+0000 7fdb56ffd700 1 -- 172.21.15.46:0/430776535 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+5687 (secure 0 0 0) 0x7fdb7015c280 con 0x7fdb700ae960 2024-01-27T01:07:45.504 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:45.504 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":138,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1114164873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1114164873},{"type":"v1","addr":"172.21.15.106:6834","nonce":1114164873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128},{"gid":9821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/702094428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":702094428},{"type":"v1","addr":"172.21.15.106:6835","nonce":702094428}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:07:27.375435+0000","modified":"2024-01-27T01:07:30.945729+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11248":{"gid":11248,"name":"a","rank":0,"incarnation":129,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/1087205220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1087205220},{"type":"v1","addr":"172.21.15.46:6834","nonce":1087205220}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:07:41.411084+0000","modified":"2024-01-27T01:07:45.002753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9827":{"gid":9827,"name":"c","rank":0,"incarnation":135,"state":"up:active","state_seq":6,"addr":"172.21.15.46:6835/408985798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":408985798},{"type":"v1","addr":"172.21.15.46:6835","nonce":408985798}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-27T01:07:45.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.506+0000 7fdb75492700 1 -- 172.21.15.46:0/430776535 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdb5c067170 msgr2=0x7fdb5c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.506+0000 7fdb75492700 1 --2- 172.21.15.46:0/430776535 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdb5c067170 0x7fdb5c069620 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdb700ae440 tx=0x7fdb64017000 comp rx=0 tx=0).stop 2024-01-27T01:07:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.506+0000 7fdb75492700 1 -- 172.21.15.46:0/430776535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb700ae960 msgr2=0x7fdb7016d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.506+0000 7fdb75492700 1 --2- 172.21.15.46:0/430776535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb700ae960 0x7fdb7016d570 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fdb6000d810 tx=0x7fdb60008040 comp rx=0 tx=0).stop 2024-01-27T01:07:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.506+0000 7fdb75492700 1 -- 172.21.15.46:0/430776535 shutdown_connections 2024-01-27T01:07:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.506+0000 7fdb75492700 1 --2- 172.21.15.46:0/430776535 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdb5c067170 0x7fdb5c069620 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.506+0000 7fdb75492700 1 --2- 172.21.15.46:0/430776535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb700ae960 0x7fdb7016d570 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.506+0000 7fdb75492700 1 --2- 172.21.15.46:0/430776535 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb700b5190 0x7fdb700b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.506+0000 7fdb75492700 1 --2- 172.21.15.46:0/430776535 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb700b2b40 0x7fdb7015e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.506+0000 7fdb75492700 1 -- 172.21.15.46:0/430776535 >> 172.21.15.46:0/430776535 conn(0x7fdb70099a30 msgr2=0x7fdb70159040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.506+0000 7fdb75492700 1 -- 172.21.15.46:0/430776535 shutdown_connections 2024-01-27T01:07:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.506+0000 7fdb75492700 1 -- 172.21.15.46:0/430776535 wait complete. 2024-01-27T01:07:45.508 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 138 2024-01-27T01:07:45.520 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:07:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.651+0000 7f7608755700 1 Processor -- start 2024-01-27T01:07:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.651+0000 7f7608755700 1 -- start start 2024-01-27T01:07:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.652+0000 7f7608755700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f76000a9170 0x7f76000b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.652+0000 7f7608755700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76000b09a0 0x7f76000b1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.652+0000 7f7608755700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76000b2050 con 0x7f76000b1530 2024-01-27T01:07:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.652+0000 7f7608755700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76000b29f0 con 0x7f76000b09a0 2024-01-27T01:07:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.652+0000 7f7608755700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76000b0e20 con 0x7f76000a9170 2024-01-27T01:07:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.652+0000 7f7606cf2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76000b09a0 0x7f76000b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.652+0000 7f7606cf2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76000b09a0 0x7f76000b1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37768/0 (socket says 172.21.15.46:37768) 2024-01-27T01:07:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.652+0000 7f7606cf2700 1 -- 172.21.15.46:0/4206716707 learned_addr learned my addr 172.21.15.46:0/4206716707 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.652+0000 7f7605cf0700 1 --2- 172.21.15.46:0/4206716707 >> v2:172.21.15.106:3300/0 conn(0x7f76000a9170 0x7f76000b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.652+0000 7f7606cf2700 1 -- 172.21.15.46:0/4206716707 >> v2:172.21.15.106:3300/0 conn(0x7f76000a9170 msgr2=0x7f76000b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.652+0000 7f7606cf2700 1 --2- 172.21.15.46:0/4206716707 >> v2:172.21.15.106:3300/0 conn(0x7f76000a9170 0x7f76000b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.652+0000 7f7606cf2700 1 -- 172.21.15.46:0/4206716707 >> v1:172.21.15.106:6789/0 conn(0x7f76000b1530 legacy=0x7f76000a8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:45.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.652+0000 7f7606cf2700 1 -- 172.21.15.46:0/4206716707 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76000b2330 con 0x7f76000b09a0 2024-01-27T01:07:45.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.653+0000 7f7606cf2700 1 --2- 172.21.15.46:0/4206716707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76000b09a0 0x7f76000b1900 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f7600064100 tx=0x7f75f8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a965c8918d112982 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:45.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.653+0000 7f76054ef700 1 -- 172.21.15.46:0/4206716707 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f800e070 con 0x7f76000b09a0 2024-01-27T01:07:45.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.653+0000 7f76054ef700 1 -- 172.21.15.46:0/4206716707 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f800ba20 con 0x7f76000b09a0 2024-01-27T01:07:45.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.653+0000 7f76054ef700 1 -- 172.21.15.46:0/4206716707 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f80092e0 con 0x7f76000b09a0 2024-01-27T01:07:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.653+0000 7f7608755700 1 -- 172.21.15.46:0/4206716707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76000b09a0 msgr2=0x7f76000b1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.653+0000 7f7608755700 1 --2- 172.21.15.46:0/4206716707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76000b09a0 0x7f76000b1900 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f7600064100 tx=0x7f75f8009840 comp rx=0 tx=0).stop 2024-01-27T01:07:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.653+0000 7f7608755700 1 -- 172.21.15.46:0/4206716707 shutdown_connections 2024-01-27T01:07:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.653+0000 7f7608755700 1 --2- 172.21.15.46:0/4206716707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76000b09a0 0x7f76000b1900 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.653+0000 7f7608755700 1 --2- 172.21.15.46:0/4206716707 >> v2:172.21.15.106:3300/0 conn(0x7f76000a9170 0x7f76000b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.653+0000 7f7608755700 1 -- 172.21.15.46:0/4206716707 >> 172.21.15.46:0/4206716707 conn(0x7f7600087f60 msgr2=0x7f76000ce5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.653+0000 7f7608755700 1 -- 172.21.15.46:0/4206716707 shutdown_connections 2024-01-27T01:07:45.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.653+0000 7f7608755700 1 -- 172.21.15.46:0/4206716707 wait complete. 2024-01-27T01:07:45.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.655+0000 7f7608755700 1 Processor -- start 2024-01-27T01:07:45.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.655+0000 7f7608755700 1 -- start start 2024-01-27T01:07:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.656+0000 7f7608755700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76000a9170 0x7f76000ac390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.656+0000 7f7608755700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76000b09a0 0x7f76000c4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.656+0000 7f7608755700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76000b1530 0x7f76000c3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.656+0000 7f7608755700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76000c4770 con 0x7f76000b09a0 2024-01-27T01:07:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.656+0000 7f7608755700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76000c5140 con 0x7f76000a9170 2024-01-27T01:07:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.656+0000 7f7608755700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76000c3890 con 0x7f76000b1530 2024-01-27T01:07:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.656+0000 7f7605cf0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76000b09a0 0x7f76000c4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.656+0000 7f7605cf0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76000b09a0 0x7f76000c4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37770/0 (socket says 172.21.15.46:37770) 2024-01-27T01:07:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.656+0000 7f7605cf0700 1 -- 172.21.15.46:0/640406777 learned_addr learned my addr 172.21.15.46:0/640406777 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.656+0000 7f76064f1700 1 --2- 172.21.15.46:0/640406777 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76000a9170 0x7f76000ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.657+0000 7f7606cf2700 1 --2- 172.21.15.46:0/640406777 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76000b1530 0x7f76000c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.657+0000 7f7605cf0700 1 -- 172.21.15.46:0/640406777 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76000b1530 msgr2=0x7f76000c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.657+0000 7f7605cf0700 1 --2- 172.21.15.46:0/640406777 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76000b1530 0x7f76000c3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.657+0000 7f7605cf0700 1 -- 172.21.15.46:0/640406777 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76000a9170 msgr2=0x7f76000ac390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.657+0000 7f7605cf0700 1 --2- 172.21.15.46:0/640406777 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76000a9170 0x7f76000ac390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.657+0000 7f7605cf0700 1 -- 172.21.15.46:0/640406777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f8009ae0 con 0x7f76000b09a0 2024-01-27T01:07:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.657+0000 7f76064f1700 1 --2- 172.21.15.46:0/640406777 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76000a9170 0x7f76000ac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.657+0000 7f7605cf0700 1 --2- 172.21.15.46:0/640406777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76000b09a0 0x7f76000c4c00 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f75f000e9d0 tx=0x7f75f0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.657+0000 7f75f67fc700 1 -- 172.21.15.46:0/640406777 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f000c850 con 0x7f76000b09a0 2024-01-27T01:07:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.657+0000 7f75f67fc700 1 -- 172.21.15.46:0/640406777 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f000c9b0 con 0x7f76000b09a0 2024-01-27T01:07:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.657+0000 7f7608755700 1 -- 172.21.15.46:0/640406777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76000c3b60 con 0x7f76000b09a0 2024-01-27T01:07:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.657+0000 7f75f67fc700 1 -- 172.21.15.46:0/640406777 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f000cd30 con 0x7f76000b09a0 2024-01-27T01:07:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.657+0000 7f7608755700 1 -- 172.21.15.46:0/640406777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76000c4120 con 0x7f76000b09a0 2024-01-27T01:07:45.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.658+0000 7f7608755700 1 -- 172.21.15.46:0/640406777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76000c48e0 con 0x7f76000b09a0 2024-01-27T01:07:45.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.659+0000 7f75f67fc700 1 -- 172.21.15.46:0/640406777 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f75f0013680 con 0x7f76000b09a0 2024-01-27T01:07:45.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.659+0000 7f75f67fc700 1 --2- 172.21.15.46:0/640406777 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f75e4067170 0x7f75e4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:45.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.659+0000 7f75f67fc700 1 -- 172.21.15.46:0/640406777 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f75f0085610 con 0x7f76000b09a0 2024-01-27T01:07:45.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.659+0000 7f76064f1700 1 --2- 172.21.15.46:0/640406777 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f75e4067170 0x7f75e4069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:45.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.660+0000 7f76064f1700 1 --2- 172.21.15.46:0/640406777 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f75e4067170 0x7f75e4069620 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f75fc007fd0 tx=0x7f75fc009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:45.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.664+0000 7f75f67fc700 1 -- 172.21.15.46:0/640406777 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f75f0051f60 con 0x7f76000b09a0 2024-01-27T01:07:45.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.868+0000 7f7608755700 1 -- 172.21.15.46:0/640406777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f760004c710 con 0x7f76000b09a0 2024-01-27T01:07:45.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.869+0000 7f75f67fc700 1 -- 172.21.15.46:0/640406777 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v247) v1 ==== 74+0+19557 (secure 0 0 0) 0x7f75f0055580 con 0x7f76000b09a0 2024-01-27T01:07:45.869 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:45.870 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":247,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:07:42.991940+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:07:25.886791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-01-27T01:07:26.340041+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:07:39.209497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:07:40.386606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:07:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.871+0000 7f7608755700 1 -- 172.21.15.46:0/640406777 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f75e4067170 msgr2=0x7f75e4069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.871+0000 7f7608755700 1 --2- 172.21.15.46:0/640406777 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f75e4067170 0x7f75e4069620 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f75fc007fd0 tx=0x7f75fc009040 comp rx=0 tx=0).stop 2024-01-27T01:07:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.871+0000 7f7608755700 1 -- 172.21.15.46:0/640406777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76000b09a0 msgr2=0x7f76000c4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.871+0000 7f7608755700 1 --2- 172.21.15.46:0/640406777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76000b09a0 0x7f76000c4c00 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f75f000e9d0 tx=0x7f75f0009120 comp rx=0 tx=0).stop 2024-01-27T01:07:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.871+0000 7f7608755700 1 -- 172.21.15.46:0/640406777 shutdown_connections 2024-01-27T01:07:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.871+0000 7f7608755700 1 --2- 172.21.15.46:0/640406777 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f75e4067170 0x7f75e4069620 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.871+0000 7f7608755700 1 --2- 172.21.15.46:0/640406777 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76000b1530 0x7f76000c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.871+0000 7f7608755700 1 --2- 172.21.15.46:0/640406777 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76000a9170 0x7f76000ac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.871+0000 7f7608755700 1 --2- 172.21.15.46:0/640406777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76000b09a0 0x7f76000c4c00 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.871+0000 7f7608755700 1 -- 172.21.15.46:0/640406777 >> 172.21.15.46:0/640406777 conn(0x7f7600087f60 msgr2=0x7f7600098840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.871+0000 7f7608755700 1 -- 172.21.15.46:0/640406777 shutdown_connections 2024-01-27T01:07:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:45.871+0000 7f7608755700 1 -- 172.21.15.46:0/640406777 wait complete. 2024-01-27T01:07:45.885 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-27T01:07:45.885 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:07:46.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.018+0000 7f917067c700 1 Processor -- start 2024-01-27T01:07:46.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.018+0000 7f917067c700 1 -- start start 2024-01-27T01:07:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.019+0000 7f917067c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9168157340 0x7f916815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.019+0000 7f917067c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91680b4180 0x7f9168157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.019+0000 7f917067c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9168158ae0 con 0x7f91680ae520 2024-01-27T01:07:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.019+0000 7f917067c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f916815b650 con 0x7f9168157340 2024-01-27T01:07:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.019+0000 7f917067c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916815c020 con 0x7f91680b4180 2024-01-27T01:07:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.019+0000 7f916dc17700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f91680ae520 0x7f91680b3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42518/0 (socket says 172.21.15.46:42518) 2024-01-27T01:07:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.019+0000 7f916dc17700 1 -- 172.21.15.46:0/3626149602 learned_addr learned my addr 172.21.15.46:0/3626149602 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.019+0000 7f916e418700 1 --2- 172.21.15.46:0/3626149602 >> v2:172.21.15.106:3300/0 conn(0x7f9168157340 0x7f916815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.019+0000 7f916ec19700 1 --2- 172.21.15.46:0/3626149602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91680b4180 0x7f9168157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.019+0000 7f916d416700 1 -- 172.21.15.46:0/3626149602 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901098304 0 0) 0x7f9168158ae0 con 0x7f91680ae520 2024-01-27T01:07:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.019+0000 7f916d416700 1 -- 172.21.15.46:0/3626149602 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9154003570 con 0x7f91680ae520 2024-01-27T01:07:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.020+0000 7f916ec19700 1 -- 172.21.15.46:0/3626149602 >> v2:172.21.15.106:3300/0 conn(0x7f9168157340 msgr2=0x7f916815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.020+0000 7f916ec19700 1 --2- 172.21.15.46:0/3626149602 >> v2:172.21.15.106:3300/0 conn(0x7f9168157340 0x7f916815bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.020+0000 7f916ec19700 1 -- 172.21.15.46:0/3626149602 >> v1:172.21.15.46:6789/0 conn(0x7f91680ae520 legacy=0x7f91680b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.020+0000 7f916ec19700 1 -- 172.21.15.46:0/3626149602 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9168158520 con 0x7f91680b4180 2024-01-27T01:07:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.020+0000 7f916e418700 1 --2- 172.21.15.46:0/3626149602 >> v2:172.21.15.106:3300/0 conn(0x7f9168157340 0x7f916815bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.020+0000 7f916ec19700 1 --2- 172.21.15.46:0/3626149602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91680b4180 0x7f9168157710 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9160009e60 tx=0x7f9160009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff414ee234428cf0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.020+0000 7f916d416700 1 -- 172.21.15.46:0/3626149602 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916000e070 con 0x7f91680b4180 2024-01-27T01:07:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.020+0000 7f916d416700 1 -- 172.21.15.46:0/3626149602 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916000bab0 con 0x7f91680b4180 2024-01-27T01:07:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.020+0000 7f916d416700 1 -- 172.21.15.46:0/3626149602 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916000be30 con 0x7f91680b4180 2024-01-27T01:07:46.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.021+0000 7f917067c700 1 -- 172.21.15.46:0/3626149602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91680b4180 msgr2=0x7f9168157710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:46.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.021+0000 7f917067c700 1 --2- 172.21.15.46:0/3626149602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91680b4180 0x7f9168157710 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9160009e60 tx=0x7f9160009780 comp rx=0 tx=0).stop 2024-01-27T01:07:46.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.021+0000 7f917067c700 1 -- 172.21.15.46:0/3626149602 shutdown_connections 2024-01-27T01:07:46.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.021+0000 7f917067c700 1 --2- 172.21.15.46:0/3626149602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91680b4180 0x7f9168157710 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.021+0000 7f917067c700 1 --2- 172.21.15.46:0/3626149602 >> v2:172.21.15.106:3300/0 conn(0x7f9168157340 0x7f916815bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.021+0000 7f917067c700 1 -- 172.21.15.46:0/3626149602 >> 172.21.15.46:0/3626149602 conn(0x7f91680ce530 msgr2=0x7f91680ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.021+0000 7f917067c700 1 -- 172.21.15.46:0/3626149602 shutdown_connections 2024-01-27T01:07:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.021+0000 7f917067c700 1 -- 172.21.15.46:0/3626149602 wait complete. 2024-01-27T01:07:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.022+0000 7f917067c700 1 Processor -- start 2024-01-27T01:07:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.022+0000 7f917067c700 1 -- start start 2024-01-27T01:07:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.023+0000 7f917067c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91680ae520 0x7f91681689c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.023+0000 7f917067c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91680b4180 0x7f916816adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.023+0000 7f917067c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9168157340 0x7f916816b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.023+0000 7f917067c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91680c2340 con 0x7f9168157340 2024-01-27T01:07:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.023+0000 7f917067c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9168168f00 con 0x7f91680ae520 2024-01-27T01:07:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.023+0000 7f917067c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91680c52b0 con 0x7f91680b4180 2024-01-27T01:07:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.023+0000 7f916ec19700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9168157340 0x7f916816b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.023+0000 7f916ec19700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9168157340 0x7f916816b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37782/0 (socket says 172.21.15.46:37782) 2024-01-27T01:07:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.023+0000 7f916ec19700 1 -- 172.21.15.46:0/171338140 learned_addr learned my addr 172.21.15.46:0/171338140 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.023+0000 7f916dc17700 1 --2- 172.21.15.46:0/171338140 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91680b4180 0x7f916816adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.023+0000 7f916e418700 1 --2- 172.21.15.46:0/171338140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91680ae520 0x7f91681689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.023+0000 7f916ec19700 1 -- 172.21.15.46:0/171338140 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91680b4180 msgr2=0x7f916816adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.023+0000 7f916ec19700 1 --2- 172.21.15.46:0/171338140 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91680b4180 0x7f916816adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.023+0000 7f916ec19700 1 -- 172.21.15.46:0/171338140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91680ae520 msgr2=0x7f91681689c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.023+0000 7f916ec19700 1 --2- 172.21.15.46:0/171338140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91680ae520 0x7f91681689c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.023+0000 7f916ec19700 1 -- 172.21.15.46:0/171338140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9164009420 con 0x7f9168157340 2024-01-27T01:07:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.023+0000 7f916e418700 1 --2- 172.21.15.46:0/171338140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91680ae520 0x7f91681689c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.024+0000 7f916ec19700 1 --2- 172.21.15.46:0/171338140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9168157340 0x7f916816b330 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f91680b1080 tx=0x7f9160019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.024+0000 7f915e7fc700 1 -- 172.21.15.46:0/171338140 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160007060 con 0x7f9168157340 2024-01-27T01:07:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.024+0000 7f915e7fc700 1 -- 172.21.15.46:0/171338140 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9160012e30 con 0x7f9168157340 2024-01-27T01:07:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.024+0000 7f917067c700 1 -- 172.21.15.46:0/171338140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9160009a20 con 0x7f9168157340 2024-01-27T01:07:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.024+0000 7f915e7fc700 1 -- 172.21.15.46:0/171338140 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160026620 con 0x7f9168157340 2024-01-27T01:07:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.024+0000 7f917067c700 1 -- 172.21.15.46:0/171338140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91680b2630 con 0x7f9168157340 2024-01-27T01:07:46.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.025+0000 7f917067c700 1 -- 172.21.15.46:0/171338140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916804c710 con 0x7f9168157340 2024-01-27T01:07:46.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.025+0000 7f915e7fc700 1 -- 172.21.15.46:0/171338140 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f9160026990 con 0x7f9168157340 2024-01-27T01:07:46.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.026+0000 7f915e7fc700 1 --2- 172.21.15.46:0/171338140 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f914c067170 0x7f914c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:46.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.026+0000 7f915e7fc700 1 -- 172.21.15.46:0/171338140 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f9160086b70 con 0x7f9168157340 2024-01-27T01:07:46.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.026+0000 7f916e418700 1 --2- 172.21.15.46:0/171338140 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f914c067170 0x7f914c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:46.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.026+0000 7f916e418700 1 --2- 172.21.15.46:0/171338140 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f914c067170 0x7f914c069620 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f91640093f0 tx=0x7f9164008040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:46.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.030+0000 7f915e7fc700 1 -- 172.21.15.46:0/171338140 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f91600532e0 con 0x7f9168157340 2024-01-27T01:07:46.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.228+0000 7f917067c700 1 -- 172.21.15.46:0/171338140 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9168158800 con 0x7f914c067170 2024-01-27T01:07:46.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.243+0000 7f915e7fc700 1 -- 172.21.15.46:0/171338140 <== mgr.9590 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528865 (secure 0 0 0) 0x7f9168158800 con 0x7f914c067170 2024-01-27T01:07:46.250 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:46.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.248+0000 7f917067c700 1 -- 172.21.15.46:0/171338140 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f914c067170 msgr2=0x7f914c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:46.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.248+0000 7f917067c700 1 --2- 172.21.15.46:0/171338140 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f914c067170 0x7f914c069620 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f91640093f0 tx=0x7f9164008040 comp rx=0 tx=0).stop 2024-01-27T01:07:46.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.248+0000 7f917067c700 1 -- 172.21.15.46:0/171338140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9168157340 msgr2=0x7f916816b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:46.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.248+0000 7f917067c700 1 --2- 172.21.15.46:0/171338140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9168157340 0x7f916816b330 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f91680b1080 tx=0x7f9160019000 comp rx=0 tx=0).stop 2024-01-27T01:07:46.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.248+0000 7f917067c700 1 -- 172.21.15.46:0/171338140 shutdown_connections 2024-01-27T01:07:46.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.248+0000 7f917067c700 1 --2- 172.21.15.46:0/171338140 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f914c067170 0x7f914c069620 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.248+0000 7f917067c700 1 --2- 172.21.15.46:0/171338140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9168157340 0x7f916816b330 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.248+0000 7f917067c700 1 --2- 172.21.15.46:0/171338140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91680ae520 0x7f91681689c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.248+0000 7f917067c700 1 --2- 172.21.15.46:0/171338140 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91680b4180 0x7f916816adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.248+0000 7f917067c700 1 -- 172.21.15.46:0/171338140 >> 172.21.15.46:0/171338140 conn(0x7f91680ce530 msgr2=0x7f9168156970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:46.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.248+0000 7f917067c700 1 -- 172.21.15.46:0/171338140 shutdown_connections 2024-01-27T01:07:46.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.248+0000 7f917067c700 1 -- 172.21.15.46:0/171338140 wait complete. 2024-01-27T01:07:46.256 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:07:46.262 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-01-27T01:07:45.873756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596212,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2456,"num_read_kb":5220,"num_write":762,"num_write_kb":3490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6623,"ondisk_log_size":6623,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":298,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2461208,"kb_used_data":25560,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752513512,"statfs":{"total":773094113280,"available":770573836288,"internally_reserved":0,"allocated":26173440,"data_stored":14394644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.913487"},"pg_stats":[{"pgid":"34.1d","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.008459+0000","last_change":"2024-01-27T01:07:42.002670+0000","last_active":"2024-01-27T01:07:42.008459+0000","last_peered":"2024-01-27T01:07:42.008459+0000","last_clean":"2024-01-27T01:07:42.008459+0000","last_became_active":"2024-01-27T01:07:42.002370+0000","last_became_peered":"2024-01-27T01:07:42.002370+0000","last_unstale":"2024-01-27T01:07:42.008459+0000","last_undegraded":"2024-01-27T01:07:42.008459+0000","last_fullsized":"2024-01-27T01:07:42.008459+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:04:31.798327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"246'1","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.003265+0000","last_change":"2024-01-27T01:07:40.984398+0000","last_active":"2024-01-27T01:07:42.003265+0000","last_peered":"2024-01-27T01:07:42.003265+0000","last_clean":"2024-01-27T01:07:42.003265+0000","last_became_active":"2024-01-27T01:07:40.984090+0000","last_became_peered":"2024-01-27T01:07:40.984090+0000","last_unstale":"2024-01-27T01:07:42.003265+0000","last_undegraded":"2024-01-27T01:07:42.003265+0000","last_fullsized":"2024-01-27T01:07:42.003265+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:59:52.185923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.993028+0000","last_change":"2024-01-27T01:07:27.744924+0000","last_active":"2024-01-27T01:07:41.993028+0000","last_peered":"2024-01-27T01:07:41.993028+0000","last_clean":"2024-01-27T01:07:41.993028+0000","last_became_active":"2024-01-27T01:07:27.744624+0000","last_became_peered":"2024-01-27T01:07:27.744624+0000","last_unstale":"2024-01-27T01:07:41.993028+0000","last_undegraded":"2024-01-27T01:07:41.993028+0000","last_fullsized":"2024-01-27T01:07:41.993028+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:44:00.209505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.991610+0000","last_change":"2024-01-27T01:07:41.990441+0000","last_active":"2024-01-27T01:07:41.991610+0000","last_peered":"2024-01-27T01:07:41.991610+0000","last_clean":"2024-01-27T01:07:41.991610+0000","last_became_active":"2024-01-27T01:07:41.990162+0000","last_became_peered":"2024-01-27T01:07:41.990162+0000","last_unstale":"2024-01-27T01:07:41.991610+0000","last_undegraded":"2024-01-27T01:07:41.991610+0000","last_fullsized":"2024-01-27T01:07:41.991610+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:23:25.015337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"246'4","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.002548+0000","last_change":"2024-01-27T01:07:40.990025+0000","last_active":"2024-01-27T01:07:42.002548+0000","last_peered":"2024-01-27T01:07:42.002548+0000","last_clean":"2024-01-27T01:07:42.002548+0000","last_became_active":"2024-01-27T01:07:40.989621+0000","last_became_peered":"2024-01-27T01:07:40.989621+0000","last_unstale":"2024-01-27T01:07:42.002548+0000","last_undegraded":"2024-01-27T01:07:42.002548+0000","last_fullsized":"2024-01-27T01:07:42.002548+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:58:53.757686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.983079+0000","last_change":"2024-01-27T01:07:27.749698+0000","last_active":"2024-01-27T01:07:41.983079+0000","last_peered":"2024-01-27T01:07:41.983079+0000","last_clean":"2024-01-27T01:07:41.983079+0000","last_became_active":"2024-01-27T01:07:27.749173+0000","last_became_peered":"2024-01-27T01:07:27.749173+0000","last_unstale":"2024-01-27T01:07:41.983079+0000","last_undegraded":"2024-01-27T01:07:41.983079+0000","last_fullsized":"2024-01-27T01:07:41.983079+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:52:40.618198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.011037+0000","last_change":"2024-01-27T01:07:42.006726+0000","last_active":"2024-01-27T01:07:42.011037+0000","last_peered":"2024-01-27T01:07:42.011037+0000","last_clean":"2024-01-27T01:07:42.011037+0000","last_became_active":"2024-01-27T01:07:42.006328+0000","last_became_peered":"2024-01-27T01:07:42.006328+0000","last_unstale":"2024-01-27T01:07:42.011037+0000","last_undegraded":"2024-01-27T01:07:42.011037+0000","last_fullsized":"2024-01-27T01:07:42.011037+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:36:25.025260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"246'2","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.008343+0000","last_change":"2024-01-27T01:07:40.979841+0000","last_active":"2024-01-27T01:07:42.008343+0000","last_peered":"2024-01-27T01:07:42.008343+0000","last_clean":"2024-01-27T01:07:42.008343+0000","last_became_active":"2024-01-27T01:07:40.979581+0000","last_became_peered":"2024-01-27T01:07:40.979581+0000","last_unstale":"2024-01-27T01:07:42.008343+0000","last_undegraded":"2024-01-27T01:07:42.008343+0000","last_fullsized":"2024-01-27T01:07:42.008343+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:21:34.990674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981807+0000","last_change":"2024-01-27T01:07:27.741674+0000","last_active":"2024-01-27T01:07:41.981807+0000","last_peered":"2024-01-27T01:07:41.981807+0000","last_clean":"2024-01-27T01:07:41.981807+0000","last_became_active":"2024-01-27T01:07:27.741351+0000","last_became_peered":"2024-01-27T01:07:27.741351+0000","last_unstale":"2024-01-27T01:07:41.981807+0000","last_undegraded":"2024-01-27T01:07:41.981807+0000","last_fullsized":"2024-01-27T01:07:41.981807+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:48:18.465014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.014563+0000","last_change":"2024-01-27T01:07:42.007107+0000","last_active":"2024-01-27T01:07:42.014563+0000","last_peered":"2024-01-27T01:07:42.014563+0000","last_clean":"2024-01-27T01:07:42.014563+0000","last_became_active":"2024-01-27T01:07:42.006691+0000","last_became_peered":"2024-01-27T01:07:42.006691+0000","last_unstale":"2024-01-27T01:07:42.014563+0000","last_undegraded":"2024-01-27T01:07:42.014563+0000","last_fullsized":"2024-01-27T01:07:42.014563+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:31:16.217384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"246'2","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.009194+0000","last_change":"2024-01-27T01:07:40.988051+0000","last_active":"2024-01-27T01:07:42.009194+0000","last_peered":"2024-01-27T01:07:42.009194+0000","last_clean":"2024-01-27T01:07:42.009194+0000","last_became_active":"2024-01-27T01:07:40.987772+0000","last_became_peered":"2024-01-27T01:07:40.987772+0000","last_unstale":"2024-01-27T01:07:42.009194+0000","last_undegraded":"2024-01-27T01:07:42.009194+0000","last_fullsized":"2024-01-27T01:07:42.009194+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:04:30.792846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979851+0000","last_change":"2024-01-27T01:07:27.749174+0000","last_active":"2024-01-27T01:07:41.979851+0000","last_peered":"2024-01-27T01:07:41.979851+0000","last_clean":"2024-01-27T01:07:41.979851+0000","last_became_active":"2024-01-27T01:07:27.748883+0000","last_became_peered":"2024-01-27T01:07:27.748883+0000","last_unstale":"2024-01-27T01:07:41.979851+0000","last_undegraded":"2024-01-27T01:07:41.979851+0000","last_fullsized":"2024-01-27T01:07:41.979851+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:03:47.280261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.012112+0000","last_change":"2024-01-27T01:07:42.007776+0000","last_active":"2024-01-27T01:07:42.012112+0000","last_peered":"2024-01-27T01:07:42.012112+0000","last_clean":"2024-01-27T01:07:42.012112+0000","last_became_active":"2024-01-27T01:07:42.007390+0000","last_became_peered":"2024-01-27T01:07:42.007390+0000","last_unstale":"2024-01-27T01:07:42.012112+0000","last_undegraded":"2024-01-27T01:07:42.012112+0000","last_fullsized":"2024-01-27T01:07:42.012112+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:04.421041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"246'1","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.019016+0000","last_change":"2024-01-27T01:07:40.983010+0000","last_active":"2024-01-27T01:07:42.019016+0000","last_peered":"2024-01-27T01:07:42.019016+0000","last_clean":"2024-01-27T01:07:42.019016+0000","last_became_active":"2024-01-27T01:07:40.982756+0000","last_became_peered":"2024-01-27T01:07:40.982756+0000","last_unstale":"2024-01-27T01:07:42.019016+0000","last_undegraded":"2024-01-27T01:07:42.019016+0000","last_fullsized":"2024-01-27T01:07:42.019016+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:58:14.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981671+0000","last_change":"2024-01-27T01:07:27.749946+0000","last_active":"2024-01-27T01:07:41.981671+0000","last_peered":"2024-01-27T01:07:41.981671+0000","last_clean":"2024-01-27T01:07:41.981671+0000","last_became_active":"2024-01-27T01:07:27.749653+0000","last_became_peered":"2024-01-27T01:07:27.749653+0000","last_unstale":"2024-01-27T01:07:41.981671+0000","last_undegraded":"2024-01-27T01:07:41.981671+0000","last_fullsized":"2024-01-27T01:07:41.981671+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:37.006113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.011035+0000","last_change":"2024-01-27T01:07:42.004489+0000","last_active":"2024-01-27T01:07:42.011035+0000","last_peered":"2024-01-27T01:07:42.011035+0000","last_clean":"2024-01-27T01:07:42.011035+0000","last_became_active":"2024-01-27T01:07:42.003992+0000","last_became_peered":"2024-01-27T01:07:42.003992+0000","last_unstale":"2024-01-27T01:07:42.011035+0000","last_undegraded":"2024-01-27T01:07:42.011035+0000","last_fullsized":"2024-01-27T01:07:42.011035+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:42:28.810173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"246'1","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.009359+0000","last_change":"2024-01-27T01:07:40.991405+0000","last_active":"2024-01-27T01:07:42.009359+0000","last_peered":"2024-01-27T01:07:42.009359+0000","last_clean":"2024-01-27T01:07:42.009359+0000","last_became_active":"2024-01-27T01:07:40.991112+0000","last_became_peered":"2024-01-27T01:07:40.991112+0000","last_unstale":"2024-01-27T01:07:42.009359+0000","last_undegraded":"2024-01-27T01:07:42.009359+0000","last_fullsized":"2024-01-27T01:07:42.009359+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:21:34.990674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.992517+0000","last_change":"2024-01-27T01:07:27.753636+0000","last_active":"2024-01-27T01:07:41.992517+0000","last_peered":"2024-01-27T01:07:41.992517+0000","last_clean":"2024-01-27T01:07:41.992517+0000","last_became_active":"2024-01-27T01:07:27.753305+0000","last_became_peered":"2024-01-27T01:07:27.753305+0000","last_unstale":"2024-01-27T01:07:41.992517+0000","last_undegraded":"2024-01-27T01:07:41.992517+0000","last_fullsized":"2024-01-27T01:07:41.992517+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:28:31.571123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.020172+0000","last_change":"2024-01-27T01:07:42.017435+0000","last_active":"2024-01-27T01:07:42.020172+0000","last_peered":"2024-01-27T01:07:42.020172+0000","last_clean":"2024-01-27T01:07:42.020172+0000","last_became_active":"2024-01-27T01:07:42.015956+0000","last_became_peered":"2024-01-27T01:07:42.015956+0000","last_unstale":"2024-01-27T01:07:42.020172+0000","last_undegraded":"2024-01-27T01:07:42.020172+0000","last_fullsized":"2024-01-27T01:07:42.020172+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:49:15.861867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981446+0000","last_change":"2024-01-27T01:07:40.987713+0000","last_active":"2024-01-27T01:07:41.981446+0000","last_peered":"2024-01-27T01:07:41.981446+0000","last_clean":"2024-01-27T01:07:41.981446+0000","last_became_active":"2024-01-27T01:07:40.987468+0000","last_became_peered":"2024-01-27T01:07:40.987468+0000","last_unstale":"2024-01-27T01:07:41.981446+0000","last_undegraded":"2024-01-27T01:07:41.981446+0000","last_fullsized":"2024-01-27T01:07:41.981446+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:34.939172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985248+0000","last_change":"2024-01-27T01:07:27.743608+0000","last_active":"2024-01-27T01:07:41.985248+0000","last_peered":"2024-01-27T01:07:41.985248+0000","last_clean":"2024-01-27T01:07:41.985248+0000","last_became_active":"2024-01-27T01:07:27.743324+0000","last_became_peered":"2024-01-27T01:07:27.743324+0000","last_unstale":"2024-01-27T01:07:41.985248+0000","last_undegraded":"2024-01-27T01:07:41.985248+0000","last_fullsized":"2024-01-27T01:07:41.985248+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:25:38.146534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.007671+0000","last_change":"2024-01-27T01:07:42.004032+0000","last_active":"2024-01-27T01:07:42.007671+0000","last_peered":"2024-01-27T01:07:42.007671+0000","last_clean":"2024-01-27T01:07:42.007671+0000","last_became_active":"2024-01-27T01:07:42.003776+0000","last_became_peered":"2024-01-27T01:07:42.003776+0000","last_unstale":"2024-01-27T01:07:42.007671+0000","last_undegraded":"2024-01-27T01:07:42.007671+0000","last_fullsized":"2024-01-27T01:07:42.007671+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:42.529880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.980992+0000","last_change":"2024-01-27T01:07:40.989457+0000","last_active":"2024-01-27T01:07:41.980992+0000","last_peered":"2024-01-27T01:07:41.980992+0000","last_clean":"2024-01-27T01:07:41.980992+0000","last_became_active":"2024-01-27T01:07:40.989191+0000","last_became_peered":"2024-01-27T01:07:40.989191+0000","last_unstale":"2024-01-27T01:07:41.980992+0000","last_undegraded":"2024-01-27T01:07:41.980992+0000","last_fullsized":"2024-01-27T01:07:41.980992+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:03:54.142292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981523+0000","last_change":"2024-01-27T01:07:27.744506+0000","last_active":"2024-01-27T01:07:41.981523+0000","last_peered":"2024-01-27T01:07:41.981523+0000","last_clean":"2024-01-27T01:07:41.981523+0000","last_became_active":"2024-01-27T01:07:27.744222+0000","last_became_peered":"2024-01-27T01:07:27.744222+0000","last_unstale":"2024-01-27T01:07:41.981523+0000","last_undegraded":"2024-01-27T01:07:41.981523+0000","last_fullsized":"2024-01-27T01:07:41.981523+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:24:38.456700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.001462+0000","last_change":"2024-01-27T01:07:41.994975+0000","last_active":"2024-01-27T01:07:42.001462+0000","last_peered":"2024-01-27T01:07:42.001462+0000","last_clean":"2024-01-27T01:07:42.001462+0000","last_became_active":"2024-01-27T01:07:41.994692+0000","last_became_peered":"2024-01-27T01:07:41.994692+0000","last_unstale":"2024-01-27T01:07:42.001462+0000","last_undegraded":"2024-01-27T01:07:42.001462+0000","last_fullsized":"2024-01-27T01:07:42.001462+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:40.019898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981333+0000","last_change":"2024-01-27T01:07:40.989481+0000","last_active":"2024-01-27T01:07:41.981333+0000","last_peered":"2024-01-27T01:07:41.981333+0000","last_clean":"2024-01-27T01:07:41.981333+0000","last_became_active":"2024-01-27T01:07:40.989211+0000","last_became_peered":"2024-01-27T01:07:40.989211+0000","last_unstale":"2024-01-27T01:07:41.981333+0000","last_undegraded":"2024-01-27T01:07:41.981333+0000","last_fullsized":"2024-01-27T01:07:41.981333+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:28:44.601683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.993180+0000","last_change":"2024-01-27T01:07:27.753284+0000","last_active":"2024-01-27T01:07:41.993180+0000","last_peered":"2024-01-27T01:07:41.993180+0000","last_clean":"2024-01-27T01:07:41.993180+0000","last_became_active":"2024-01-27T01:07:27.752819+0000","last_became_peered":"2024-01-27T01:07:27.752819+0000","last_unstale":"2024-01-27T01:07:41.993180+0000","last_undegraded":"2024-01-27T01:07:41.993180+0000","last_fullsized":"2024-01-27T01:07:41.993180+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:58:33.718993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.015645+0000","last_change":"2024-01-27T01:07:42.011852+0000","last_active":"2024-01-27T01:07:42.015645+0000","last_peered":"2024-01-27T01:07:42.015645+0000","last_clean":"2024-01-27T01:07:42.015645+0000","last_became_active":"2024-01-27T01:07:42.011522+0000","last_became_peered":"2024-01-27T01:07:42.011522+0000","last_unstale":"2024-01-27T01:07:42.015645+0000","last_undegraded":"2024-01-27T01:07:42.015645+0000","last_fullsized":"2024-01-27T01:07:42.015645+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:49:05.586187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.992730+0000","last_change":"2024-01-27T01:07:40.984823+0000","last_active":"2024-01-27T01:07:41.992730+0000","last_peered":"2024-01-27T01:07:41.992730+0000","last_clean":"2024-01-27T01:07:41.992730+0000","last_became_active":"2024-01-27T01:07:40.984369+0000","last_became_peered":"2024-01-27T01:07:40.984369+0000","last_unstale":"2024-01-27T01:07:41.992730+0000","last_undegraded":"2024-01-27T01:07:41.992730+0000","last_fullsized":"2024-01-27T01:07:41.992730+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:59:25.911803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979458+0000","last_change":"2024-01-27T01:07:27.743902+0000","last_active":"2024-01-27T01:07:41.979458+0000","last_peered":"2024-01-27T01:07:41.979458+0000","last_clean":"2024-01-27T01:07:41.979458+0000","last_became_active":"2024-01-27T01:07:27.743270+0000","last_became_peered":"2024-01-27T01:07:27.743270+0000","last_unstale":"2024-01-27T01:07:41.979458+0000","last_undegraded":"2024-01-27T01:07:41.979458+0000","last_fullsized":"2024-01-27T01:07:41.979458+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:24:38.456700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.013239+0000","last_change":"2024-01-27T01:07:42.008312+0000","last_active":"2024-01-27T01:07:42.013239+0000","last_peered":"2024-01-27T01:07:42.013239+0000","last_clean":"2024-01-27T01:07:42.013239+0000","last_became_active":"2024-01-27T01:07:42.008091+0000","last_became_peered":"2024-01-27T01:07:42.008091+0000","last_unstale":"2024-01-27T01:07:42.013239+0000","last_undegraded":"2024-01-27T01:07:42.013239+0000","last_fullsized":"2024-01-27T01:07:42.013239+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:42.529880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"246'6","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.016378+0000","last_change":"2024-01-27T01:07:40.986467+0000","last_active":"2024-01-27T01:07:42.016378+0000","last_peered":"2024-01-27T01:07:42.016378+0000","last_clean":"2024-01-27T01:07:42.016378+0000","last_became_active":"2024-01-27T01:07:40.986128+0000","last_became_peered":"2024-01-27T01:07:40.986128+0000","last_unstale":"2024-01-27T01:07:42.016378+0000","last_undegraded":"2024-01-27T01:07:42.016378+0000","last_fullsized":"2024-01-27T01:07:42.016378+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:39.079815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981992+0000","last_change":"2024-01-27T01:07:27.743746+0000","last_active":"2024-01-27T01:07:41.981992+0000","last_peered":"2024-01-27T01:07:41.981992+0000","last_clean":"2024-01-27T01:07:41.981992+0000","last_became_active":"2024-01-27T01:07:27.743414+0000","last_became_peered":"2024-01-27T01:07:27.743414+0000","last_unstale":"2024-01-27T01:07:41.981992+0000","last_undegraded":"2024-01-27T01:07:41.981992+0000","last_fullsized":"2024-01-27T01:07:41.981992+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:37.006113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.000392+0000","last_change":"2024-01-27T01:07:41.992795+0000","last_active":"2024-01-27T01:07:42.000392+0000","last_peered":"2024-01-27T01:07:42.000392+0000","last_clean":"2024-01-27T01:07:42.000392+0000","last_became_active":"2024-01-27T01:07:41.992541+0000","last_became_peered":"2024-01-27T01:07:41.992541+0000","last_unstale":"2024-01-27T01:07:42.000392+0000","last_undegraded":"2024-01-27T01:07:42.000392+0000","last_fullsized":"2024-01-27T01:07:42.000392+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:18:33.128110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"246'3","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.017535+0000","last_change":"2024-01-27T01:07:40.985223+0000","last_active":"2024-01-27T01:07:42.017535+0000","last_peered":"2024-01-27T01:07:42.017535+0000","last_clean":"2024-01-27T01:07:42.017535+0000","last_became_active":"2024-01-27T01:07:40.984909+0000","last_became_peered":"2024-01-27T01:07:40.984909+0000","last_unstale":"2024-01-27T01:07:42.017535+0000","last_undegraded":"2024-01-27T01:07:42.017535+0000","last_fullsized":"2024-01-27T01:07:42.017535+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:58:14.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.994743+0000","last_change":"2024-01-27T01:07:27.748096+0000","last_active":"2024-01-27T01:07:41.994743+0000","last_peered":"2024-01-27T01:07:41.994743+0000","last_clean":"2024-01-27T01:07:41.994743+0000","last_became_active":"2024-01-27T01:07:27.747801+0000","last_became_peered":"2024-01-27T01:07:27.747801+0000","last_unstale":"2024-01-27T01:07:41.994743+0000","last_undegraded":"2024-01-27T01:07:41.994743+0000","last_fullsized":"2024-01-27T01:07:41.994743+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:20:52.644616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.013347+0000","last_change":"2024-01-27T01:07:42.009566+0000","last_active":"2024-01-27T01:07:42.013347+0000","last_peered":"2024-01-27T01:07:42.013347+0000","last_clean":"2024-01-27T01:07:42.013347+0000","last_became_active":"2024-01-27T01:07:42.008961+0000","last_became_peered":"2024-01-27T01:07:42.008961+0000","last_unstale":"2024-01-27T01:07:42.013347+0000","last_undegraded":"2024-01-27T01:07:42.013347+0000","last_fullsized":"2024-01-27T01:07:42.013347+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:00.162997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.992233+0000","last_change":"2024-01-27T01:07:40.980925+0000","last_active":"2024-01-27T01:07:41.992233+0000","last_peered":"2024-01-27T01:07:41.992233+0000","last_clean":"2024-01-27T01:07:41.992233+0000","last_became_active":"2024-01-27T01:07:40.980634+0000","last_became_peered":"2024-01-27T01:07:40.980634+0000","last_unstale":"2024-01-27T01:07:41.992233+0000","last_undegraded":"2024-01-27T01:07:41.992233+0000","last_fullsized":"2024-01-27T01:07:41.992233+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:17.897003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995189+0000","last_change":"2024-01-27T01:07:27.746806+0000","last_active":"2024-01-27T01:07:41.995189+0000","last_peered":"2024-01-27T01:07:41.995189+0000","last_clean":"2024-01-27T01:07:41.995189+0000","last_became_active":"2024-01-27T01:07:27.745168+0000","last_became_peered":"2024-01-27T01:07:27.745168+0000","last_unstale":"2024-01-27T01:07:41.995189+0000","last_undegraded":"2024-01-27T01:07:41.995189+0000","last_fullsized":"2024-01-27T01:07:41.995189+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:34:21.004352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.004700+0000","last_change":"2024-01-27T01:07:41.998067+0000","last_active":"2024-01-27T01:07:42.004700+0000","last_peered":"2024-01-27T01:07:42.004700+0000","last_clean":"2024-01-27T01:07:42.004700+0000","last_became_active":"2024-01-27T01:07:41.997808+0000","last_became_peered":"2024-01-27T01:07:41.997808+0000","last_unstale":"2024-01-27T01:07:42.004700+0000","last_undegraded":"2024-01-27T01:07:42.004700+0000","last_fullsized":"2024-01-27T01:07:42.004700+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:10:20.100421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"246'2","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.009489+0000","last_change":"2024-01-27T01:07:40.986956+0000","last_active":"2024-01-27T01:07:42.009489+0000","last_peered":"2024-01-27T01:07:42.009489+0000","last_clean":"2024-01-27T01:07:42.009489+0000","last_became_active":"2024-01-27T01:07:40.986683+0000","last_became_peered":"2024-01-27T01:07:40.986683+0000","last_unstale":"2024-01-27T01:07:42.009489+0000","last_undegraded":"2024-01-27T01:07:42.009489+0000","last_fullsized":"2024-01-27T01:07:42.009489+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:03:54.142292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995101+0000","last_change":"2024-01-27T01:07:27.749049+0000","last_active":"2024-01-27T01:07:41.995101+0000","last_peered":"2024-01-27T01:07:41.995101+0000","last_clean":"2024-01-27T01:07:41.995101+0000","last_became_active":"2024-01-27T01:07:27.748736+0000","last_became_peered":"2024-01-27T01:07:27.748736+0000","last_unstale":"2024-01-27T01:07:41.995101+0000","last_undegraded":"2024-01-27T01:07:41.995101+0000","last_fullsized":"2024-01-27T01:07:41.995101+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:23:10.979296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.009544+0000","last_change":"2024-01-27T01:07:42.006099+0000","last_active":"2024-01-27T01:07:42.009544+0000","last_peered":"2024-01-27T01:07:42.009544+0000","last_clean":"2024-01-27T01:07:42.009544+0000","last_became_active":"2024-01-27T01:07:42.005798+0000","last_became_peered":"2024-01-27T01:07:42.005798+0000","last_unstale":"2024-01-27T01:07:42.009544+0000","last_undegraded":"2024-01-27T01:07:42.009544+0000","last_fullsized":"2024-01-27T01:07:42.009544+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:41:14.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"246'8","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.014698+0000","last_change":"2024-01-27T01:07:40.988442+0000","last_active":"2024-01-27T01:07:42.014698+0000","last_peered":"2024-01-27T01:07:42.014698+0000","last_clean":"2024-01-27T01:07:42.014698+0000","last_became_active":"2024-01-27T01:07:40.988119+0000","last_became_peered":"2024-01-27T01:07:40.988119+0000","last_unstale":"2024-01-27T01:07:42.014698+0000","last_undegraded":"2024-01-27T01:07:42.014698+0000","last_fullsized":"2024-01-27T01:07:42.014698+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:54:53.395827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984376+0000","last_change":"2024-01-27T01:07:27.755236+0000","last_active":"2024-01-27T01:07:41.984376+0000","last_peered":"2024-01-27T01:07:41.984376+0000","last_clean":"2024-01-27T01:07:41.984376+0000","last_became_active":"2024-01-27T01:07:27.754945+0000","last_became_peered":"2024-01-27T01:07:27.754945+0000","last_unstale":"2024-01-27T01:07:41.984376+0000","last_undegraded":"2024-01-27T01:07:41.984376+0000","last_fullsized":"2024-01-27T01:07:41.984376+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:07:48.619221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.008758+0000","last_change":"2024-01-27T01:07:42.001263+0000","last_active":"2024-01-27T01:07:42.008758+0000","last_peered":"2024-01-27T01:07:42.008758+0000","last_clean":"2024-01-27T01:07:42.008758+0000","last_became_active":"2024-01-27T01:07:42.000947+0000","last_became_peered":"2024-01-27T01:07:42.000947+0000","last_unstale":"2024-01-27T01:07:42.008758+0000","last_undegraded":"2024-01-27T01:07:42.008758+0000","last_fullsized":"2024-01-27T01:07:42.008758+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:17.562068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.987205+0000","last_change":"2024-01-27T01:07:40.976555+0000","last_active":"2024-01-27T01:07:41.987205+0000","last_peered":"2024-01-27T01:07:41.987205+0000","last_clean":"2024-01-27T01:07:41.987205+0000","last_became_active":"2024-01-27T01:07:40.976259+0000","last_became_peered":"2024-01-27T01:07:40.976259+0000","last_unstale":"2024-01-27T01:07:41.987205+0000","last_undegraded":"2024-01-27T01:07:41.987205+0000","last_fullsized":"2024-01-27T01:07:41.987205+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:08:19.116626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981641+0000","last_change":"2024-01-27T01:07:27.745495+0000","last_active":"2024-01-27T01:07:41.981641+0000","last_peered":"2024-01-27T01:07:41.981641+0000","last_clean":"2024-01-27T01:07:41.981641+0000","last_became_active":"2024-01-27T01:07:27.744986+0000","last_became_peered":"2024-01-27T01:07:27.744986+0000","last_unstale":"2024-01-27T01:07:41.981641+0000","last_undegraded":"2024-01-27T01:07:41.981641+0000","last_fullsized":"2024-01-27T01:07:41.981641+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:08.374701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.006025+0000","last_change":"2024-01-27T01:07:42.003063+0000","last_active":"2024-01-27T01:07:42.006025+0000","last_peered":"2024-01-27T01:07:42.006025+0000","last_clean":"2024-01-27T01:07:42.006025+0000","last_became_active":"2024-01-27T01:07:42.002733+0000","last_became_peered":"2024-01-27T01:07:42.002733+0000","last_unstale":"2024-01-27T01:07:42.006025+0000","last_undegraded":"2024-01-27T01:07:42.006025+0000","last_fullsized":"2024-01-27T01:07:42.006025+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:22.410741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"246'2","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.002101+0000","last_change":"2024-01-27T01:07:40.990836+0000","last_active":"2024-01-27T01:07:42.002101+0000","last_peered":"2024-01-27T01:07:42.002101+0000","last_clean":"2024-01-27T01:07:42.002101+0000","last_became_active":"2024-01-27T01:07:40.990079+0000","last_became_peered":"2024-01-27T01:07:40.990079+0000","last_unstale":"2024-01-27T01:07:42.002101+0000","last_undegraded":"2024-01-27T01:07:42.002101+0000","last_fullsized":"2024-01-27T01:07:42.002101+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:52:32.284100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.998215+0000","last_change":"2024-01-27T01:07:27.749357+0000","last_active":"2024-01-27T01:07:41.998215+0000","last_peered":"2024-01-27T01:07:41.998215+0000","last_clean":"2024-01-27T01:07:41.998215+0000","last_became_active":"2024-01-27T01:07:27.745281+0000","last_became_peered":"2024-01-27T01:07:27.745281+0000","last_unstale":"2024-01-27T01:07:41.998215+0000","last_undegraded":"2024-01-27T01:07:41.998215+0000","last_fullsized":"2024-01-27T01:07:41.998215+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:11.619364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.005896+0000","last_change":"2024-01-27T01:07:42.002607+0000","last_active":"2024-01-27T01:07:42.005896+0000","last_peered":"2024-01-27T01:07:42.005896+0000","last_clean":"2024-01-27T01:07:42.005896+0000","last_became_active":"2024-01-27T01:07:42.002101+0000","last_became_peered":"2024-01-27T01:07:42.002101+0000","last_unstale":"2024-01-27T01:07:42.005896+0000","last_undegraded":"2024-01-27T01:07:42.005896+0000","last_fullsized":"2024-01-27T01:07:42.005896+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:19.572287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"246'1","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.005523+0000","last_change":"2024-01-27T01:07:40.989369+0000","last_active":"2024-01-27T01:07:42.005523+0000","last_peered":"2024-01-27T01:07:42.005523+0000","last_clean":"2024-01-27T01:07:42.005523+0000","last_became_active":"2024-01-27T01:07:40.989079+0000","last_became_peered":"2024-01-27T01:07:40.989079+0000","last_unstale":"2024-01-27T01:07:42.005523+0000","last_undegraded":"2024-01-27T01:07:42.005523+0000","last_fullsized":"2024-01-27T01:07:42.005523+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:45.809251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979563+0000","last_change":"2024-01-27T01:07:27.736143+0000","last_active":"2024-01-27T01:07:41.979563+0000","last_peered":"2024-01-27T01:07:41.979563+0000","last_clean":"2024-01-27T01:07:41.979563+0000","last_became_active":"2024-01-27T01:07:27.735824+0000","last_became_peered":"2024-01-27T01:07:27.735824+0000","last_unstale":"2024-01-27T01:07:41.979563+0000","last_undegraded":"2024-01-27T01:07:41.979563+0000","last_fullsized":"2024-01-27T01:07:41.979563+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:05.536246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.013345+0000","last_change":"2024-01-27T01:07:42.009502+0000","last_active":"2024-01-27T01:07:42.013345+0000","last_peered":"2024-01-27T01:07:42.013345+0000","last_clean":"2024-01-27T01:07:42.013345+0000","last_became_active":"2024-01-27T01:07:42.009217+0000","last_became_peered":"2024-01-27T01:07:42.009217+0000","last_unstale":"2024-01-27T01:07:42.013345+0000","last_undegraded":"2024-01-27T01:07:42.013345+0000","last_fullsized":"2024-01-27T01:07:42.013345+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:33.375597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982593+0000","last_change":"2024-01-27T01:07:40.990708+0000","last_active":"2024-01-27T01:07:41.982593+0000","last_peered":"2024-01-27T01:07:41.982593+0000","last_clean":"2024-01-27T01:07:41.982593+0000","last_became_active":"2024-01-27T01:07:40.990411+0000","last_became_peered":"2024-01-27T01:07:40.990411+0000","last_unstale":"2024-01-27T01:07:41.982593+0000","last_undegraded":"2024-01-27T01:07:41.982593+0000","last_fullsized":"2024-01-27T01:07:41.982593+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:41:37.115942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985631+0000","last_change":"2024-01-27T01:07:27.756376+0000","last_active":"2024-01-27T01:07:41.985631+0000","last_peered":"2024-01-27T01:07:41.985631+0000","last_clean":"2024-01-27T01:07:41.985631+0000","last_became_active":"2024-01-27T01:07:27.756152+0000","last_became_peered":"2024-01-27T01:07:27.756152+0000","last_unstale":"2024-01-27T01:07:41.985631+0000","last_undegraded":"2024-01-27T01:07:41.985631+0000","last_fullsized":"2024-01-27T01:07:41.985631+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:02:04.959467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.014625+0000","last_change":"2024-01-27T01:07:42.011536+0000","last_active":"2024-01-27T01:07:42.014625+0000","last_peered":"2024-01-27T01:07:42.014625+0000","last_clean":"2024-01-27T01:07:42.014625+0000","last_became_active":"2024-01-27T01:07:42.006921+0000","last_became_peered":"2024-01-27T01:07:42.006921+0000","last_unstale":"2024-01-27T01:07:42.014625+0000","last_undegraded":"2024-01-27T01:07:42.014625+0000","last_fullsized":"2024-01-27T01:07:42.014625+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:36:25.025260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"246'1","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.003675+0000","last_change":"2024-01-27T01:07:40.984795+0000","last_active":"2024-01-27T01:07:42.003675+0000","last_peered":"2024-01-27T01:07:42.003675+0000","last_clean":"2024-01-27T01:07:42.003675+0000","last_became_active":"2024-01-27T01:07:40.984185+0000","last_became_peered":"2024-01-27T01:07:40.984185+0000","last_unstale":"2024-01-27T01:07:42.003675+0000","last_undegraded":"2024-01-27T01:07:42.003675+0000","last_fullsized":"2024-01-27T01:07:42.003675+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:43:50.097560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982873+0000","last_change":"2024-01-27T01:07:27.758252+0000","last_active":"2024-01-27T01:07:41.982873+0000","last_peered":"2024-01-27T01:07:41.982873+0000","last_clean":"2024-01-27T01:07:41.982873+0000","last_became_active":"2024-01-27T01:07:27.757929+0000","last_became_peered":"2024-01-27T01:07:27.757929+0000","last_unstale":"2024-01-27T01:07:41.982873+0000","last_undegraded":"2024-01-27T01:07:41.982873+0000","last_fullsized":"2024-01-27T01:07:41.982873+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:30.518041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.010869+0000","last_change":"2024-01-27T01:07:42.006225+0000","last_active":"2024-01-27T01:07:42.010869+0000","last_peered":"2024-01-27T01:07:42.010869+0000","last_clean":"2024-01-27T01:07:42.010869+0000","last_became_active":"2024-01-27T01:07:42.005928+0000","last_became_peered":"2024-01-27T01:07:42.005928+0000","last_unstale":"2024-01-27T01:07:42.010869+0000","last_undegraded":"2024-01-27T01:07:42.010869+0000","last_fullsized":"2024-01-27T01:07:42.010869+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:38.943517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.978974+0000","last_change":"2024-01-27T01:07:40.992511+0000","last_active":"2024-01-27T01:07:41.978974+0000","last_peered":"2024-01-27T01:07:41.978974+0000","last_clean":"2024-01-27T01:07:41.978974+0000","last_became_active":"2024-01-27T01:07:40.992205+0000","last_became_peered":"2024-01-27T01:07:40.992205+0000","last_unstale":"2024-01-27T01:07:41.978974+0000","last_undegraded":"2024-01-27T01:07:41.978974+0000","last_fullsized":"2024-01-27T01:07:41.978974+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:13.573493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985737+0000","last_change":"2024-01-27T01:07:27.752626+0000","last_active":"2024-01-27T01:07:41.985737+0000","last_peered":"2024-01-27T01:07:41.985737+0000","last_clean":"2024-01-27T01:07:41.985737+0000","last_became_active":"2024-01-27T01:07:27.752363+0000","last_became_peered":"2024-01-27T01:07:27.752363+0000","last_unstale":"2024-01-27T01:07:41.985737+0000","last_undegraded":"2024-01-27T01:07:41.985737+0000","last_fullsized":"2024-01-27T01:07:41.985737+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:03:44.824769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.011005+0000","last_change":"2024-01-27T01:07:42.004276+0000","last_active":"2024-01-27T01:07:42.011005+0000","last_peered":"2024-01-27T01:07:42.011005+0000","last_clean":"2024-01-27T01:07:42.011005+0000","last_became_active":"2024-01-27T01:07:42.003559+0000","last_became_peered":"2024-01-27T01:07:42.003559+0000","last_unstale":"2024-01-27T01:07:42.011005+0000","last_undegraded":"2024-01-27T01:07:42.011005+0000","last_fullsized":"2024-01-27T01:07:42.011005+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:22:42.410257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"246'2","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.018015+0000","last_change":"2024-01-27T01:07:40.992336+0000","last_active":"2024-01-27T01:07:42.018015+0000","last_peered":"2024-01-27T01:07:42.018015+0000","last_clean":"2024-01-27T01:07:42.018015+0000","last_became_active":"2024-01-27T01:07:40.991815+0000","last_became_peered":"2024-01-27T01:07:40.991815+0000","last_unstale":"2024-01-27T01:07:42.018015+0000","last_undegraded":"2024-01-27T01:07:42.018015+0000","last_fullsized":"2024-01-27T01:07:42.018015+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:59:52.185923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995278+0000","last_change":"2024-01-27T01:07:27.748467+0000","last_active":"2024-01-27T01:07:41.995278+0000","last_peered":"2024-01-27T01:07:41.995278+0000","last_clean":"2024-01-27T01:07:41.995278+0000","last_became_active":"2024-01-27T01:07:27.748100+0000","last_became_peered":"2024-01-27T01:07:27.748100+0000","last_unstale":"2024-01-27T01:07:41.995278+0000","last_undegraded":"2024-01-27T01:07:41.995278+0000","last_fullsized":"2024-01-27T01:07:41.995278+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:38:29.701960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.012756+0000","last_change":"2024-01-27T01:07:42.004497+0000","last_active":"2024-01-27T01:07:42.012756+0000","last_peered":"2024-01-27T01:07:42.012756+0000","last_clean":"2024-01-27T01:07:42.012756+0000","last_became_active":"2024-01-27T01:07:42.003613+0000","last_became_peered":"2024-01-27T01:07:42.003613+0000","last_unstale":"2024-01-27T01:07:42.012756+0000","last_undegraded":"2024-01-27T01:07:42.012756+0000","last_fullsized":"2024-01-27T01:07:42.012756+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:41.431166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.988574+0000","last_change":"2024-01-27T01:07:40.978139+0000","last_active":"2024-01-27T01:07:41.988574+0000","last_peered":"2024-01-27T01:07:41.988574+0000","last_clean":"2024-01-27T01:07:41.988574+0000","last_became_active":"2024-01-27T01:07:40.977778+0000","last_became_peered":"2024-01-27T01:07:40.977778+0000","last_unstale":"2024-01-27T01:07:41.988574+0000","last_undegraded":"2024-01-27T01:07:41.988574+0000","last_fullsized":"2024-01-27T01:07:41.988574+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:30:38.944123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984213+0000","last_change":"2024-01-27T01:07:27.755439+0000","last_active":"2024-01-27T01:07:41.984213+0000","last_peered":"2024-01-27T01:07:41.984213+0000","last_clean":"2024-01-27T01:07:41.984213+0000","last_became_active":"2024-01-27T01:07:27.755126+0000","last_became_peered":"2024-01-27T01:07:27.755126+0000","last_unstale":"2024-01-27T01:07:41.984213+0000","last_undegraded":"2024-01-27T01:07:41.984213+0000","last_fullsized":"2024-01-27T01:07:41.984213+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:43:37.067000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.017713+0000","last_change":"2024-01-27T01:07:42.014667+0000","last_active":"2024-01-27T01:07:42.017713+0000","last_peered":"2024-01-27T01:07:42.017713+0000","last_clean":"2024-01-27T01:07:42.017713+0000","last_became_active":"2024-01-27T01:07:42.011648+0000","last_became_peered":"2024-01-27T01:07:42.011648+0000","last_unstale":"2024-01-27T01:07:42.017713+0000","last_undegraded":"2024-01-27T01:07:42.017713+0000","last_fullsized":"2024-01-27T01:07:42.017713+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:22:42.410257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.997076+0000","last_change":"2024-01-27T01:07:40.988279+0000","last_active":"2024-01-27T01:07:41.997076+0000","last_peered":"2024-01-27T01:07:41.997076+0000","last_clean":"2024-01-27T01:07:41.997076+0000","last_became_active":"2024-01-27T01:07:40.987982+0000","last_became_peered":"2024-01-27T01:07:40.987982+0000","last_unstale":"2024-01-27T01:07:41.997076+0000","last_undegraded":"2024-01-27T01:07:41.997076+0000","last_fullsized":"2024-01-27T01:07:41.997076+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:08:19.116626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995851+0000","last_change":"2024-01-27T01:07:27.745155+0000","last_active":"2024-01-27T01:07:41.995851+0000","last_peered":"2024-01-27T01:07:41.995851+0000","last_clean":"2024-01-27T01:07:41.995851+0000","last_became_active":"2024-01-27T01:07:27.744562+0000","last_became_peered":"2024-01-27T01:07:27.744562+0000","last_unstale":"2024-01-27T01:07:41.995851+0000","last_undegraded":"2024-01-27T01:07:41.995851+0000","last_fullsized":"2024-01-27T01:07:41.995851+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:28:42.362768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.002199+0000","last_change":"2024-01-27T01:07:41.999378+0000","last_active":"2024-01-27T01:07:42.002199+0000","last_peered":"2024-01-27T01:07:42.002199+0000","last_clean":"2024-01-27T01:07:42.002199+0000","last_became_active":"2024-01-27T01:07:41.999026+0000","last_became_peered":"2024-01-27T01:07:41.999026+0000","last_unstale":"2024-01-27T01:07:42.002199+0000","last_undegraded":"2024-01-27T01:07:42.002199+0000","last_fullsized":"2024-01-27T01:07:42.002199+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:44:21.352342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"247'1","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-01-27T01:07:43.033348+0000","last_change":"2024-01-27T01:07:40.991024+0000","last_active":"2024-01-27T01:07:43.033348+0000","last_peered":"2024-01-27T01:07:43.033348+0000","last_clean":"2024-01-27T01:07:43.033348+0000","last_became_active":"2024-01-27T01:07:40.990526+0000","last_became_peered":"2024-01-27T01:07:40.990526+0000","last_unstale":"2024-01-27T01:07:43.033348+0000","last_undegraded":"2024-01-27T01:07:43.033348+0000","last_fullsized":"2024-01-27T01:07:43.033348+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:34:29.680208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.983165+0000","last_change":"2024-01-27T01:07:27.745357+0000","last_active":"2024-01-27T01:07:41.983165+0000","last_peered":"2024-01-27T01:07:41.983165+0000","last_clean":"2024-01-27T01:07:41.983165+0000","last_became_active":"2024-01-27T01:07:27.745075+0000","last_became_peered":"2024-01-27T01:07:27.745075+0000","last_unstale":"2024-01-27T01:07:41.983165+0000","last_undegraded":"2024-01-27T01:07:41.983165+0000","last_fullsized":"2024-01-27T01:07:41.983165+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:37:01.620442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.000853+0000","last_change":"2024-01-27T01:07:41.993171+0000","last_active":"2024-01-27T01:07:42.000853+0000","last_peered":"2024-01-27T01:07:42.000853+0000","last_clean":"2024-01-27T01:07:42.000853+0000","last_became_active":"2024-01-27T01:07:41.991964+0000","last_became_peered":"2024-01-27T01:07:41.991964+0000","last_unstale":"2024-01-27T01:07:42.000853+0000","last_undegraded":"2024-01-27T01:07:42.000853+0000","last_fullsized":"2024-01-27T01:07:42.000853+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:14.578974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"246'2","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.011545+0000","last_change":"2024-01-27T01:07:40.985107+0000","last_active":"2024-01-27T01:07:42.011545+0000","last_peered":"2024-01-27T01:07:42.011545+0000","last_clean":"2024-01-27T01:07:42.011545+0000","last_became_active":"2024-01-27T01:07:40.984742+0000","last_became_peered":"2024-01-27T01:07:40.984742+0000","last_unstale":"2024-01-27T01:07:42.011545+0000","last_undegraded":"2024-01-27T01:07:42.011545+0000","last_fullsized":"2024-01-27T01:07:42.011545+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:07:12.229792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985979+0000","last_change":"2024-01-27T01:07:27.744187+0000","last_active":"2024-01-27T01:07:41.985979+0000","last_peered":"2024-01-27T01:07:41.985979+0000","last_clean":"2024-01-27T01:07:41.985979+0000","last_became_active":"2024-01-27T01:07:27.743889+0000","last_became_peered":"2024-01-27T01:07:27.743889+0000","last_unstale":"2024-01-27T01:07:41.985979+0000","last_undegraded":"2024-01-27T01:07:41.985979+0000","last_fullsized":"2024-01-27T01:07:41.985979+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:11.619364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.002866+0000","last_change":"2024-01-27T01:07:42.001717+0000","last_active":"2024-01-27T01:07:42.002866+0000","last_peered":"2024-01-27T01:07:42.002866+0000","last_clean":"2024-01-27T01:07:42.002866+0000","last_became_active":"2024-01-27T01:07:42.001443+0000","last_became_peered":"2024-01-27T01:07:42.001443+0000","last_unstale":"2024-01-27T01:07:42.002866+0000","last_undegraded":"2024-01-27T01:07:42.002866+0000","last_fullsized":"2024-01-27T01:07:42.002866+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:03:35.144430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985723+0000","last_change":"2024-01-27T01:07:40.979755+0000","last_active":"2024-01-27T01:07:41.985723+0000","last_peered":"2024-01-27T01:07:41.985723+0000","last_clean":"2024-01-27T01:07:41.985723+0000","last_became_active":"2024-01-27T01:07:40.979490+0000","last_became_peered":"2024-01-27T01:07:40.979490+0000","last_unstale":"2024-01-27T01:07:41.985723+0000","last_undegraded":"2024-01-27T01:07:41.985723+0000","last_fullsized":"2024-01-27T01:07:41.985723+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:24.587487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981626+0000","last_change":"2024-01-27T01:07:27.743304+0000","last_active":"2024-01-27T01:07:41.981626+0000","last_peered":"2024-01-27T01:07:41.981626+0000","last_clean":"2024-01-27T01:07:41.981626+0000","last_became_active":"2024-01-27T01:07:27.742783+0000","last_became_peered":"2024-01-27T01:07:27.742783+0000","last_unstale":"2024-01-27T01:07:41.981626+0000","last_undegraded":"2024-01-27T01:07:41.981626+0000","last_fullsized":"2024-01-27T01:07:41.981626+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:45:07.998194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.007642+0000","last_change":"2024-01-27T01:07:42.002777+0000","last_active":"2024-01-27T01:07:42.007642+0000","last_peered":"2024-01-27T01:07:42.007642+0000","last_clean":"2024-01-27T01:07:42.007642+0000","last_became_active":"2024-01-27T01:07:42.002493+0000","last_became_peered":"2024-01-27T01:07:42.002493+0000","last_unstale":"2024-01-27T01:07:42.007642+0000","last_undegraded":"2024-01-27T01:07:42.007642+0000","last_fullsized":"2024-01-27T01:07:42.007642+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:17.054007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.986261+0000","last_change":"2024-01-27T01:07:40.984956+0000","last_active":"2024-01-27T01:07:41.986261+0000","last_peered":"2024-01-27T01:07:41.986261+0000","last_clean":"2024-01-27T01:07:41.986261+0000","last_became_active":"2024-01-27T01:07:40.984539+0000","last_became_peered":"2024-01-27T01:07:40.984539+0000","last_unstale":"2024-01-27T01:07:41.986261+0000","last_undegraded":"2024-01-27T01:07:41.986261+0000","last_fullsized":"2024-01-27T01:07:41.986261+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:26:12.872499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.994833+0000","last_change":"2024-01-27T01:07:27.746921+0000","last_active":"2024-01-27T01:07:41.994833+0000","last_peered":"2024-01-27T01:07:41.994833+0000","last_clean":"2024-01-27T01:07:41.994833+0000","last_became_active":"2024-01-27T01:07:27.746630+0000","last_became_peered":"2024-01-27T01:07:27.746630+0000","last_unstale":"2024-01-27T01:07:41.994833+0000","last_undegraded":"2024-01-27T01:07:41.994833+0000","last_fullsized":"2024-01-27T01:07:41.994833+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:03:41.111732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.020811+0000","last_change":"2024-01-27T01:07:42.019567+0000","last_active":"2024-01-27T01:07:42.020811+0000","last_peered":"2024-01-27T01:07:42.020811+0000","last_clean":"2024-01-27T01:07:42.020811+0000","last_became_active":"2024-01-27T01:07:42.019261+0000","last_became_peered":"2024-01-27T01:07:42.019261+0000","last_unstale":"2024-01-27T01:07:42.020811+0000","last_undegraded":"2024-01-27T01:07:42.020811+0000","last_fullsized":"2024-01-27T01:07:42.020811+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:49:15.861867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982243+0000","last_change":"2024-01-27T01:07:40.987525+0000","last_active":"2024-01-27T01:07:41.982243+0000","last_peered":"2024-01-27T01:07:41.982243+0000","last_clean":"2024-01-27T01:07:41.982243+0000","last_became_active":"2024-01-27T01:07:40.986541+0000","last_became_peered":"2024-01-27T01:07:40.986541+0000","last_unstale":"2024-01-27T01:07:41.982243+0000","last_undegraded":"2024-01-27T01:07:41.982243+0000","last_fullsized":"2024-01-27T01:07:41.982243+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:22:57.587494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.992694+0000","last_change":"2024-01-27T01:07:27.749637+0000","last_active":"2024-01-27T01:07:41.992694+0000","last_peered":"2024-01-27T01:07:41.992694+0000","last_clean":"2024-01-27T01:07:41.992694+0000","last_became_active":"2024-01-27T01:07:27.749170+0000","last_became_peered":"2024-01-27T01:07:27.749170+0000","last_unstale":"2024-01-27T01:07:41.992694+0000","last_undegraded":"2024-01-27T01:07:41.992694+0000","last_fullsized":"2024-01-27T01:07:41.992694+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:10:06.064380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.017034+0000","last_change":"2024-01-27T01:07:42.014208+0000","last_active":"2024-01-27T01:07:42.017034+0000","last_peered":"2024-01-27T01:07:42.017034+0000","last_clean":"2024-01-27T01:07:42.017034+0000","last_became_active":"2024-01-27T01:07:42.013983+0000","last_became_peered":"2024-01-27T01:07:42.013983+0000","last_unstale":"2024-01-27T01:07:42.017034+0000","last_undegraded":"2024-01-27T01:07:42.017034+0000","last_fullsized":"2024-01-27T01:07:42.017034+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:02:31.556291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985310+0000","last_change":"2024-01-27T01:07:40.990547+0000","last_active":"2024-01-27T01:07:41.985310+0000","last_peered":"2024-01-27T01:07:41.985310+0000","last_clean":"2024-01-27T01:07:41.985310+0000","last_became_active":"2024-01-27T01:07:40.990249+0000","last_became_peered":"2024-01-27T01:07:40.990249+0000","last_unstale":"2024-01-27T01:07:41.985310+0000","last_undegraded":"2024-01-27T01:07:41.985310+0000","last_fullsized":"2024-01-27T01:07:41.985310+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:15:24.216671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995182+0000","last_change":"2024-01-27T01:07:27.748872+0000","last_active":"2024-01-27T01:07:41.995182+0000","last_peered":"2024-01-27T01:07:41.995182+0000","last_clean":"2024-01-27T01:07:41.995182+0000","last_became_active":"2024-01-27T01:07:27.748549+0000","last_became_peered":"2024-01-27T01:07:27.748549+0000","last_unstale":"2024-01-27T01:07:41.995182+0000","last_undegraded":"2024-01-27T01:07:41.995182+0000","last_fullsized":"2024-01-27T01:07:41.995182+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:58:33.718993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.992417+0000","last_change":"2024-01-27T01:07:41.991372+0000","last_active":"2024-01-27T01:07:41.992417+0000","last_peered":"2024-01-27T01:07:41.992417+0000","last_clean":"2024-01-27T01:07:41.992417+0000","last_became_active":"2024-01-27T01:07:41.991084+0000","last_became_peered":"2024-01-27T01:07:41.991084+0000","last_unstale":"2024-01-27T01:07:41.992417+0000","last_undegraded":"2024-01-27T01:07:41.992417+0000","last_fullsized":"2024-01-27T01:07:41.992417+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:23:17.453403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984768+0000","last_change":"2024-01-27T01:07:40.986837+0000","last_active":"2024-01-27T01:07:41.984768+0000","last_peered":"2024-01-27T01:07:41.984768+0000","last_clean":"2024-01-27T01:07:41.984768+0000","last_became_active":"2024-01-27T01:07:40.986445+0000","last_became_peered":"2024-01-27T01:07:40.986445+0000","last_unstale":"2024-01-27T01:07:41.984768+0000","last_undegraded":"2024-01-27T01:07:41.984768+0000","last_fullsized":"2024-01-27T01:07:41.984768+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:55:53.993289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982042+0000","last_change":"2024-01-27T01:07:27.755023+0000","last_active":"2024-01-27T01:07:41.982042+0000","last_peered":"2024-01-27T01:07:41.982042+0000","last_clean":"2024-01-27T01:07:41.982042+0000","last_became_active":"2024-01-27T01:07:27.754304+0000","last_became_peered":"2024-01-27T01:07:27.754304+0000","last_unstale":"2024-01-27T01:07:41.982042+0000","last_undegraded":"2024-01-27T01:07:41.982042+0000","last_fullsized":"2024-01-27T01:07:41.982042+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:28:12.245965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.016805+0000","last_change":"2024-01-27T01:07:42.016325+0000","last_active":"2024-01-27T01:07:42.016805+0000","last_peered":"2024-01-27T01:07:42.016805+0000","last_clean":"2024-01-27T01:07:42.016805+0000","last_became_active":"2024-01-27T01:07:42.015981+0000","last_became_peered":"2024-01-27T01:07:42.015981+0000","last_unstale":"2024-01-27T01:07:42.016805+0000","last_undegraded":"2024-01-27T01:07:42.016805+0000","last_fullsized":"2024-01-27T01:07:42.016805+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:49:46.679016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985162+0000","last_change":"2024-01-27T01:07:40.986796+0000","last_active":"2024-01-27T01:07:41.985162+0000","last_peered":"2024-01-27T01:07:41.985162+0000","last_clean":"2024-01-27T01:07:41.985162+0000","last_became_active":"2024-01-27T01:07:40.986511+0000","last_became_peered":"2024-01-27T01:07:40.986511+0000","last_unstale":"2024-01-27T01:07:41.985162+0000","last_undegraded":"2024-01-27T01:07:41.985162+0000","last_fullsized":"2024-01-27T01:07:41.985162+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:20.549192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.988350+0000","last_change":"2024-01-27T01:07:27.747314+0000","last_active":"2024-01-27T01:07:41.988350+0000","last_peered":"2024-01-27T01:07:41.988350+0000","last_clean":"2024-01-27T01:07:41.988350+0000","last_became_active":"2024-01-27T01:07:27.746947+0000","last_became_peered":"2024-01-27T01:07:27.746947+0000","last_unstale":"2024-01-27T01:07:41.988350+0000","last_undegraded":"2024-01-27T01:07:41.988350+0000","last_fullsized":"2024-01-27T01:07:41.988350+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:25:01.603669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.001204+0000","last_change":"2024-01-27T01:07:41.992826+0000","last_active":"2024-01-27T01:07:42.001204+0000","last_peered":"2024-01-27T01:07:42.001204+0000","last_clean":"2024-01-27T01:07:42.001204+0000","last_became_active":"2024-01-27T01:07:41.991941+0000","last_became_peered":"2024-01-27T01:07:41.991941+0000","last_unstale":"2024-01-27T01:07:42.001204+0000","last_undegraded":"2024-01-27T01:07:42.001204+0000","last_fullsized":"2024-01-27T01:07:42.001204+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:36:39.545539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984533+0000","last_change":"2024-01-27T01:07:27.755462+0000","last_active":"2024-01-27T01:07:41.984533+0000","last_peered":"2024-01-27T01:07:41.984533+0000","last_clean":"2024-01-27T01:07:41.984533+0000","last_became_active":"2024-01-27T01:07:27.755131+0000","last_became_peered":"2024-01-27T01:07:27.755131+0000","last_unstale":"2024-01-27T01:07:41.984533+0000","last_undegraded":"2024-01-27T01:07:41.984533+0000","last_fullsized":"2024-01-27T01:07:41.984533+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:32:22.760237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"242'4","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.992931+0000","last_change":"2024-01-27T01:07:26.950037+0000","last_active":"2024-01-27T01:07:41.992931+0000","last_peered":"2024-01-27T01:07:41.992931+0000","last_clean":"2024-01-27T01:07:41.992931+0000","last_became_active":"2024-01-27T01:07:26.949604+0000","last_became_peered":"2024-01-27T01:07:26.949604+0000","last_unstale":"2024-01-27T01:07:41.992931+0000","last_undegraded":"2024-01-27T01:07:41.992931+0000","last_fullsized":"2024-01-27T01:07:41.992931+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:01:17.838521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.011116+0000","last_change":"2024-01-27T01:07:42.007084+0000","last_active":"2024-01-27T01:07:42.011116+0000","last_peered":"2024-01-27T01:07:42.011116+0000","last_clean":"2024-01-27T01:07:42.011116+0000","last_became_active":"2024-01-27T01:07:42.006767+0000","last_became_peered":"2024-01-27T01:07:42.006767+0000","last_unstale":"2024-01-27T01:07:42.011116+0000","last_undegraded":"2024-01-27T01:07:42.011116+0000","last_fullsized":"2024-01-27T01:07:42.011116+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:29:57.281939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985207+0000","last_change":"2024-01-27T01:07:27.755690+0000","last_active":"2024-01-27T01:07:41.985207+0000","last_peered":"2024-01-27T01:07:41.985207+0000","last_clean":"2024-01-27T01:07:41.985207+0000","last_became_active":"2024-01-27T01:07:27.755402+0000","last_became_peered":"2024-01-27T01:07:27.755402+0000","last_unstale":"2024-01-27T01:07:41.985207+0000","last_undegraded":"2024-01-27T01:07:41.985207+0000","last_fullsized":"2024-01-27T01:07:41.985207+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:25:59.841939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"242'2","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.998721+0000","last_change":"2024-01-27T01:07:26.951267+0000","last_active":"2024-01-27T01:07:41.998721+0000","last_peered":"2024-01-27T01:07:41.998721+0000","last_clean":"2024-01-27T01:07:41.998721+0000","last_became_active":"2024-01-27T01:07:26.950967+0000","last_became_peered":"2024-01-27T01:07:26.950967+0000","last_unstale":"2024-01-27T01:07:41.998721+0000","last_undegraded":"2024-01-27T01:07:41.998721+0000","last_fullsized":"2024-01-27T01:07:41.998721+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:37:00.613294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.020790+0000","last_change":"2024-01-27T01:07:42.019332+0000","last_active":"2024-01-27T01:07:42.020790+0000","last_peered":"2024-01-27T01:07:42.020790+0000","last_clean":"2024-01-27T01:07:42.020790+0000","last_became_active":"2024-01-27T01:07:42.018863+0000","last_became_peered":"2024-01-27T01:07:42.018863+0000","last_unstale":"2024-01-27T01:07:42.020790+0000","last_undegraded":"2024-01-27T01:07:42.020790+0000","last_fullsized":"2024-01-27T01:07:42.020790+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:20.301704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.992509+0000","last_change":"2024-01-27T01:07:27.752814+0000","last_active":"2024-01-27T01:07:41.992509+0000","last_peered":"2024-01-27T01:07:41.992509+0000","last_clean":"2024-01-27T01:07:41.992509+0000","last_became_active":"2024-01-27T01:07:27.752459+0000","last_became_peered":"2024-01-27T01:07:27.752459+0000","last_unstale":"2024-01-27T01:07:41.992509+0000","last_undegraded":"2024-01-27T01:07:41.992509+0000","last_fullsized":"2024-01-27T01:07:41.992509+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:44:32.042391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"242'2","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.994509+0000","last_change":"2024-01-27T01:07:26.945958+0000","last_active":"2024-01-27T01:07:41.994509+0000","last_peered":"2024-01-27T01:07:41.994509+0000","last_clean":"2024-01-27T01:07:41.994509+0000","last_became_active":"2024-01-27T01:07:26.945671+0000","last_became_peered":"2024-01-27T01:07:26.945671+0000","last_unstale":"2024-01-27T01:07:41.994509+0000","last_undegraded":"2024-01-27T01:07:41.994509+0000","last_fullsized":"2024-01-27T01:07:41.994509+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:20.901463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.012905+0000","last_change":"2024-01-27T01:07:42.009910+0000","last_active":"2024-01-27T01:07:42.012905+0000","last_peered":"2024-01-27T01:07:42.012905+0000","last_clean":"2024-01-27T01:07:42.012905+0000","last_became_active":"2024-01-27T01:07:42.009517+0000","last_became_peered":"2024-01-27T01:07:42.009517+0000","last_unstale":"2024-01-27T01:07:42.012905+0000","last_undegraded":"2024-01-27T01:07:42.012905+0000","last_fullsized":"2024-01-27T01:07:42.012905+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:10:02.105512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.994311+0000","last_change":"2024-01-27T01:07:27.756421+0000","last_active":"2024-01-27T01:07:41.994311+0000","last_peered":"2024-01-27T01:07:41.994311+0000","last_clean":"2024-01-27T01:07:41.994311+0000","last_became_active":"2024-01-27T01:07:27.756102+0000","last_became_peered":"2024-01-27T01:07:27.756102+0000","last_unstale":"2024-01-27T01:07:41.994311+0000","last_undegraded":"2024-01-27T01:07:41.994311+0000","last_fullsized":"2024-01-27T01:07:41.994311+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:23:49.531160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"242'1","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985595+0000","last_change":"2024-01-27T01:07:26.957169+0000","last_active":"2024-01-27T01:07:41.985595+0000","last_peered":"2024-01-27T01:07:41.985595+0000","last_clean":"2024-01-27T01:07:41.985595+0000","last_became_active":"2024-01-27T01:07:26.956902+0000","last_became_peered":"2024-01-27T01:07:26.956902+0000","last_unstale":"2024-01-27T01:07:41.985595+0000","last_undegraded":"2024-01-27T01:07:41.985595+0000","last_fullsized":"2024-01-27T01:07:41.985595+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:27:55.458182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.993951+0000","last_change":"2024-01-27T01:07:41.991557+0000","last_active":"2024-01-27T01:07:41.993951+0000","last_peered":"2024-01-27T01:07:41.993951+0000","last_clean":"2024-01-27T01:07:41.993951+0000","last_became_active":"2024-01-27T01:07:41.991257+0000","last_became_peered":"2024-01-27T01:07:41.991257+0000","last_unstale":"2024-01-27T01:07:41.993951+0000","last_undegraded":"2024-01-27T01:07:41.993951+0000","last_fullsized":"2024-01-27T01:07:41.993951+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:00.162997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985629+0000","last_change":"2024-01-27T01:07:27.752682+0000","last_active":"2024-01-27T01:07:41.985629+0000","last_peered":"2024-01-27T01:07:41.985629+0000","last_clean":"2024-01-27T01:07:41.985629+0000","last_became_active":"2024-01-27T01:07:27.752401+0000","last_became_peered":"2024-01-27T01:07:27.752401+0000","last_unstale":"2024-01-27T01:07:41.985629+0000","last_undegraded":"2024-01-27T01:07:41.985629+0000","last_fullsized":"2024-01-27T01:07:41.985629+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:55:40.962728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"242'1","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995265+0000","last_change":"2024-01-27T01:07:26.951702+0000","last_active":"2024-01-27T01:07:41.995265+0000","last_peered":"2024-01-27T01:07:41.995265+0000","last_clean":"2024-01-27T01:07:41.995265+0000","last_became_active":"2024-01-27T01:07:26.951367+0000","last_became_peered":"2024-01-27T01:07:26.951367+0000","last_unstale":"2024-01-27T01:07:41.995265+0000","last_undegraded":"2024-01-27T01:07:41.995265+0000","last_fullsized":"2024-01-27T01:07:41.995265+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:25:00.596520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.013327+0000","last_change":"2024-01-27T01:07:42.009310+0000","last_active":"2024-01-27T01:07:42.013327+0000","last_peered":"2024-01-27T01:07:42.013327+0000","last_clean":"2024-01-27T01:07:42.013327+0000","last_became_active":"2024-01-27T01:07:42.008856+0000","last_became_peered":"2024-01-27T01:07:42.008856+0000","last_unstale":"2024-01-27T01:07:42.013327+0000","last_undegraded":"2024-01-27T01:07:42.013327+0000","last_fullsized":"2024-01-27T01:07:42.013327+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:04:01.316302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979338+0000","last_change":"2024-01-27T01:07:27.750189+0000","last_active":"2024-01-27T01:07:41.979338+0000","last_peered":"2024-01-27T01:07:41.979338+0000","last_clean":"2024-01-27T01:07:41.979338+0000","last_became_active":"2024-01-27T01:07:27.749867+0000","last_became_peered":"2024-01-27T01:07:27.749867+0000","last_unstale":"2024-01-27T01:07:41.979338+0000","last_undegraded":"2024-01-27T01:07:41.979338+0000","last_fullsized":"2024-01-27T01:07:41.979338+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:39:46.126956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979097+0000","last_change":"2024-01-27T01:07:26.954992+0000","last_active":"2024-01-27T01:07:41.979097+0000","last_peered":"2024-01-27T01:07:41.979097+0000","last_clean":"2024-01-27T01:07:41.979097+0000","last_became_active":"2024-01-27T01:07:26.954701+0000","last_became_peered":"2024-01-27T01:07:26.954701+0000","last_unstale":"2024-01-27T01:07:41.979097+0000","last_undegraded":"2024-01-27T01:07:41.979097+0000","last_fullsized":"2024-01-27T01:07:41.979097+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:24.976708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.013358+0000","last_change":"2024-01-27T01:07:42.009611+0000","last_active":"2024-01-27T01:07:42.013358+0000","last_peered":"2024-01-27T01:07:42.013358+0000","last_clean":"2024-01-27T01:07:42.013358+0000","last_became_active":"2024-01-27T01:07:42.009362+0000","last_became_peered":"2024-01-27T01:07:42.009362+0000","last_unstale":"2024-01-27T01:07:42.013358+0000","last_undegraded":"2024-01-27T01:07:42.013358+0000","last_fullsized":"2024-01-27T01:07:42.013358+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:51.042154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982305+0000","last_change":"2024-01-27T01:07:27.743124+0000","last_active":"2024-01-27T01:07:41.982305+0000","last_peered":"2024-01-27T01:07:41.982305+0000","last_clean":"2024-01-27T01:07:41.982305+0000","last_became_active":"2024-01-27T01:07:27.742799+0000","last_became_peered":"2024-01-27T01:07:27.742799+0000","last_unstale":"2024-01-27T01:07:41.982305+0000","last_undegraded":"2024-01-27T01:07:41.982305+0000","last_fullsized":"2024-01-27T01:07:41.982305+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:14.714021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.988678+0000","last_change":"2024-01-27T01:07:26.946511+0000","last_active":"2024-01-27T01:07:41.988678+0000","last_peered":"2024-01-27T01:07:41.988678+0000","last_clean":"2024-01-27T01:07:41.988678+0000","last_became_active":"2024-01-27T01:07:26.946186+0000","last_became_peered":"2024-01-27T01:07:26.946186+0000","last_unstale":"2024-01-27T01:07:41.988678+0000","last_undegraded":"2024-01-27T01:07:41.988678+0000","last_fullsized":"2024-01-27T01:07:41.988678+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:24:39.506445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.006017+0000","last_change":"2024-01-27T01:07:42.003996+0000","last_active":"2024-01-27T01:07:42.006017+0000","last_peered":"2024-01-27T01:07:42.006017+0000","last_clean":"2024-01-27T01:07:42.006017+0000","last_became_active":"2024-01-27T01:07:42.003482+0000","last_became_peered":"2024-01-27T01:07:42.003482+0000","last_unstale":"2024-01-27T01:07:42.006017+0000","last_undegraded":"2024-01-27T01:07:42.006017+0000","last_fullsized":"2024-01-27T01:07:42.006017+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:24:52.492741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984547+0000","last_change":"2024-01-27T01:07:27.749767+0000","last_active":"2024-01-27T01:07:41.984547+0000","last_peered":"2024-01-27T01:07:41.984547+0000","last_clean":"2024-01-27T01:07:41.984547+0000","last_became_active":"2024-01-27T01:07:27.749508+0000","last_became_peered":"2024-01-27T01:07:27.749508+0000","last_unstale":"2024-01-27T01:07:41.984547+0000","last_undegraded":"2024-01-27T01:07:41.984547+0000","last_fullsized":"2024-01-27T01:07:41.984547+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:56:10.391021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.993835+0000","last_change":"2024-01-27T01:07:26.947663+0000","last_active":"2024-01-27T01:07:41.993835+0000","last_peered":"2024-01-27T01:07:41.993835+0000","last_clean":"2024-01-27T01:07:41.993835+0000","last_became_active":"2024-01-27T01:07:26.947378+0000","last_became_peered":"2024-01-27T01:07:26.947378+0000","last_unstale":"2024-01-27T01:07:41.993835+0000","last_undegraded":"2024-01-27T01:07:41.993835+0000","last_fullsized":"2024-01-27T01:07:41.993835+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:28:30.563975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.020641+0000","last_change":"2024-01-27T01:07:42.019102+0000","last_active":"2024-01-27T01:07:42.020641+0000","last_peered":"2024-01-27T01:07:42.020641+0000","last_clean":"2024-01-27T01:07:42.020641+0000","last_became_active":"2024-01-27T01:07:42.018594+0000","last_became_peered":"2024-01-27T01:07:42.018594+0000","last_unstale":"2024-01-27T01:07:42.020641+0000","last_undegraded":"2024-01-27T01:07:42.020641+0000","last_fullsized":"2024-01-27T01:07:42.020641+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:23:25.015337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995991+0000","last_change":"2024-01-27T01:07:27.749325+0000","last_active":"2024-01-27T01:07:41.995991+0000","last_peered":"2024-01-27T01:07:41.995991+0000","last_clean":"2024-01-27T01:07:41.995991+0000","last_became_active":"2024-01-27T01:07:27.744840+0000","last_became_peered":"2024-01-27T01:07:27.744840+0000","last_unstale":"2024-01-27T01:07:41.995991+0000","last_undegraded":"2024-01-27T01:07:41.995991+0000","last_fullsized":"2024-01-27T01:07:41.995991+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:15:38.535322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979665+0000","last_change":"2024-01-27T01:07:26.948157+0000","last_active":"2024-01-27T01:07:41.979665+0000","last_peered":"2024-01-27T01:07:41.979665+0000","last_clean":"2024-01-27T01:07:41.979665+0000","last_became_active":"2024-01-27T01:07:26.947870+0000","last_became_peered":"2024-01-27T01:07:26.947870+0000","last_unstale":"2024-01-27T01:07:41.979665+0000","last_undegraded":"2024-01-27T01:07:41.979665+0000","last_fullsized":"2024-01-27T01:07:41.979665+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:04:16.755137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.000911+0000","last_change":"2024-01-27T01:07:41.997557+0000","last_active":"2024-01-27T01:07:42.000911+0000","last_peered":"2024-01-27T01:07:42.000911+0000","last_clean":"2024-01-27T01:07:42.000911+0000","last_became_active":"2024-01-27T01:07:41.997288+0000","last_became_peered":"2024-01-27T01:07:41.997288+0000","last_unstale":"2024-01-27T01:07:42.000911+0000","last_undegraded":"2024-01-27T01:07:42.000911+0000","last_fullsized":"2024-01-27T01:07:42.000911+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:36:28.490987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979190+0000","last_change":"2024-01-27T01:07:27.744220+0000","last_active":"2024-01-27T01:07:41.979190+0000","last_peered":"2024-01-27T01:07:41.979190+0000","last_clean":"2024-01-27T01:07:41.979190+0000","last_became_active":"2024-01-27T01:07:27.742448+0000","last_became_peered":"2024-01-27T01:07:27.742448+0000","last_unstale":"2024-01-27T01:07:41.979190+0000","last_undegraded":"2024-01-27T01:07:41.979190+0000","last_fullsized":"2024-01-27T01:07:41.979190+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:03.017967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"242'3","reported_seq":17,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981788+0000","last_change":"2024-01-27T01:07:26.953006+0000","last_active":"2024-01-27T01:07:41.981788+0000","last_peered":"2024-01-27T01:07:41.981788+0000","last_clean":"2024-01-27T01:07:41.981788+0000","last_became_active":"2024-01-27T01:07:26.952730+0000","last_became_peered":"2024-01-27T01:07:26.952730+0000","last_unstale":"2024-01-27T01:07:41.981788+0000","last_undegraded":"2024-01-27T01:07:41.981788+0000","last_fullsized":"2024-01-27T01:07:41.981788+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:14.273722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.012073+0000","last_change":"2024-01-27T01:07:42.007566+0000","last_active":"2024-01-27T01:07:42.012073+0000","last_peered":"2024-01-27T01:07:42.012073+0000","last_clean":"2024-01-27T01:07:42.012073+0000","last_became_active":"2024-01-27T01:07:42.007055+0000","last_became_peered":"2024-01-27T01:07:42.007055+0000","last_unstale":"2024-01-27T01:07:42.012073+0000","last_undegraded":"2024-01-27T01:07:42.012073+0000","last_fullsized":"2024-01-27T01:07:42.012073+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:27:59.374021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985613+0000","last_change":"2024-01-27T01:07:27.747921+0000","last_active":"2024-01-27T01:07:41.985613+0000","last_peered":"2024-01-27T01:07:41.985613+0000","last_clean":"2024-01-27T01:07:41.985613+0000","last_became_active":"2024-01-27T01:07:27.747382+0000","last_became_peered":"2024-01-27T01:07:27.747382+0000","last_unstale":"2024-01-27T01:07:41.985613+0000","last_undegraded":"2024-01-27T01:07:41.985613+0000","last_fullsized":"2024-01-27T01:07:41.985613+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:43:44.312681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"243'8","reported_seq":22,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995022+0000","last_change":"2024-01-27T01:07:26.951048+0000","last_active":"2024-01-27T01:07:41.995022+0000","last_peered":"2024-01-27T01:07:41.995022+0000","last_clean":"2024-01-27T01:07:41.995022+0000","last_became_active":"2024-01-27T01:07:26.950750+0000","last_became_peered":"2024-01-27T01:07:26.950750+0000","last_unstale":"2024-01-27T01:07:41.995022+0000","last_undegraded":"2024-01-27T01:07:41.995022+0000","last_fullsized":"2024-01-27T01:07:41.995022+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:48:29.363281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2768,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.009019+0000","last_change":"2024-01-27T01:07:42.002706+0000","last_active":"2024-01-27T01:07:42.009019+0000","last_peered":"2024-01-27T01:07:42.009019+0000","last_clean":"2024-01-27T01:07:42.009019+0000","last_became_active":"2024-01-27T01:07:42.002441+0000","last_became_peered":"2024-01-27T01:07:42.002441+0000","last_unstale":"2024-01-27T01:07:42.009019+0000","last_undegraded":"2024-01-27T01:07:42.009019+0000","last_fullsized":"2024-01-27T01:07:42.009019+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:35:39.491535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982749+0000","last_change":"2024-01-27T01:07:27.756638+0000","last_active":"2024-01-27T01:07:41.982749+0000","last_peered":"2024-01-27T01:07:41.982749+0000","last_clean":"2024-01-27T01:07:41.982749+0000","last_became_active":"2024-01-27T01:07:27.756219+0000","last_became_peered":"2024-01-27T01:07:27.756219+0000","last_unstale":"2024-01-27T01:07:41.982749+0000","last_undegraded":"2024-01-27T01:07:41.982749+0000","last_fullsized":"2024-01-27T01:07:41.982749+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:24:43.565560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"242'2","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985902+0000","last_change":"2024-01-27T01:07:26.957374+0000","last_active":"2024-01-27T01:07:41.985902+0000","last_peered":"2024-01-27T01:07:41.985902+0000","last_clean":"2024-01-27T01:07:41.985902+0000","last_became_active":"2024-01-27T01:07:26.957184+0000","last_became_peered":"2024-01-27T01:07:26.957184+0000","last_unstale":"2024-01-27T01:07:41.985902+0000","last_undegraded":"2024-01-27T01:07:41.985902+0000","last_fullsized":"2024-01-27T01:07:41.985902+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:43:44.001485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.000750+0000","last_change":"2024-01-27T01:07:41.999362+0000","last_active":"2024-01-27T01:07:42.000750+0000","last_peered":"2024-01-27T01:07:42.000750+0000","last_clean":"2024-01-27T01:07:42.000750+0000","last_became_active":"2024-01-27T01:07:41.999099+0000","last_became_peered":"2024-01-27T01:07:41.999099+0000","last_unstale":"2024-01-27T01:07:42.000750+0000","last_undegraded":"2024-01-27T01:07:42.000750+0000","last_fullsized":"2024-01-27T01:07:42.000750+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:41:14.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985717+0000","last_change":"2024-01-27T01:07:27.741016+0000","last_active":"2024-01-27T01:07:41.985717+0000","last_peered":"2024-01-27T01:07:41.985717+0000","last_clean":"2024-01-27T01:07:41.985717+0000","last_became_active":"2024-01-27T01:07:27.740724+0000","last_became_peered":"2024-01-27T01:07:27.740724+0000","last_unstale":"2024-01-27T01:07:41.985717+0000","last_undegraded":"2024-01-27T01:07:41.985717+0000","last_fullsized":"2024-01-27T01:07:41.985717+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:19:56.759979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981479+0000","last_change":"2024-01-27T01:07:26.951785+0000","last_active":"2024-01-27T01:07:41.981479+0000","last_peered":"2024-01-27T01:07:41.981479+0000","last_clean":"2024-01-27T01:07:41.981479+0000","last_became_active":"2024-01-27T01:07:26.951519+0000","last_became_peered":"2024-01-27T01:07:26.951519+0000","last_unstale":"2024-01-27T01:07:41.981479+0000","last_undegraded":"2024-01-27T01:07:41.981479+0000","last_fullsized":"2024-01-27T01:07:41.981479+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:06.511483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.000874+0000","last_change":"2024-01-27T01:07:41.994163+0000","last_active":"2024-01-27T01:07:42.000874+0000","last_peered":"2024-01-27T01:07:42.000874+0000","last_clean":"2024-01-27T01:07:42.000874+0000","last_became_active":"2024-01-27T01:07:41.993928+0000","last_became_peered":"2024-01-27T01:07:41.993928+0000","last_unstale":"2024-01-27T01:07:42.000874+0000","last_undegraded":"2024-01-27T01:07:42.000874+0000","last_fullsized":"2024-01-27T01:07:42.000874+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:14:34.374610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.986496+0000","last_change":"2024-01-27T01:07:27.755983+0000","last_active":"2024-01-27T01:07:41.986496+0000","last_peered":"2024-01-27T01:07:41.986496+0000","last_clean":"2024-01-27T01:07:41.986496+0000","last_became_active":"2024-01-27T01:07:27.755715+0000","last_became_peered":"2024-01-27T01:07:27.755715+0000","last_unstale":"2024-01-27T01:07:41.986496+0000","last_undegraded":"2024-01-27T01:07:41.986496+0000","last_fullsized":"2024-01-27T01:07:41.986496+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:11.556927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981153+0000","last_change":"2024-01-27T01:07:26.952237+0000","last_active":"2024-01-27T01:07:41.981153+0000","last_peered":"2024-01-27T01:07:41.981153+0000","last_clean":"2024-01-27T01:07:41.981153+0000","last_became_active":"2024-01-27T01:07:26.951971+0000","last_became_peered":"2024-01-27T01:07:26.951971+0000","last_unstale":"2024-01-27T01:07:41.981153+0000","last_undegraded":"2024-01-27T01:07:41.981153+0000","last_fullsized":"2024-01-27T01:07:41.981153+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:43:36.059851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.004338+0000","last_change":"2024-01-27T01:07:42.001580+0000","last_active":"2024-01-27T01:07:42.004338+0000","last_peered":"2024-01-27T01:07:42.004338+0000","last_clean":"2024-01-27T01:07:42.004338+0000","last_became_active":"2024-01-27T01:07:42.001300+0000","last_became_peered":"2024-01-27T01:07:42.001300+0000","last_unstale":"2024-01-27T01:07:42.004338+0000","last_undegraded":"2024-01-27T01:07:42.004338+0000","last_fullsized":"2024-01-27T01:07:42.004338+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:24:03.567201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982253+0000","last_change":"2024-01-27T01:07:27.746452+0000","last_active":"2024-01-27T01:07:41.982253+0000","last_peered":"2024-01-27T01:07:41.982253+0000","last_clean":"2024-01-27T01:07:41.982253+0000","last_became_active":"2024-01-27T01:07:27.746174+0000","last_became_peered":"2024-01-27T01:07:27.746174+0000","last_unstale":"2024-01-27T01:07:41.982253+0000","last_undegraded":"2024-01-27T01:07:41.982253+0000","last_fullsized":"2024-01-27T01:07:41.982253+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:32.778691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"242'8","reported_seq":22,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.980773+0000","last_change":"2024-01-27T01:07:26.946801+0000","last_active":"2024-01-27T01:07:41.980773+0000","last_peered":"2024-01-27T01:07:41.980773+0000","last_clean":"2024-01-27T01:07:41.980773+0000","last_became_active":"2024-01-27T01:07:26.946453+0000","last_became_peered":"2024-01-27T01:07:26.946453+0000","last_unstale":"2024-01-27T01:07:41.980773+0000","last_undegraded":"2024-01-27T01:07:41.980773+0000","last_fullsized":"2024-01-27T01:07:41.980773+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:11:44.275714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.016792+0000","last_change":"2024-01-27T01:07:42.016234+0000","last_active":"2024-01-27T01:07:42.016792+0000","last_peered":"2024-01-27T01:07:42.016792+0000","last_clean":"2024-01-27T01:07:42.016792+0000","last_became_active":"2024-01-27T01:07:42.015609+0000","last_became_peered":"2024-01-27T01:07:42.015609+0000","last_unstale":"2024-01-27T01:07:42.016792+0000","last_undegraded":"2024-01-27T01:07:42.016792+0000","last_fullsized":"2024-01-27T01:07:42.016792+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:40:45.932550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982340+0000","last_change":"2024-01-27T01:07:27.746480+0000","last_active":"2024-01-27T01:07:41.982340+0000","last_peered":"2024-01-27T01:07:41.982340+0000","last_clean":"2024-01-27T01:07:41.982340+0000","last_became_active":"2024-01-27T01:07:27.746131+0000","last_became_peered":"2024-01-27T01:07:27.746131+0000","last_unstale":"2024-01-27T01:07:41.982340+0000","last_undegraded":"2024-01-27T01:07:41.982340+0000","last_fullsized":"2024-01-27T01:07:41.982340+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:06:55.090910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"242'1","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.992831+0000","last_change":"2024-01-27T01:07:26.958336+0000","last_active":"2024-01-27T01:07:41.992831+0000","last_peered":"2024-01-27T01:07:41.992831+0000","last_clean":"2024-01-27T01:07:41.992831+0000","last_became_active":"2024-01-27T01:07:26.957817+0000","last_became_peered":"2024-01-27T01:07:26.957817+0000","last_unstale":"2024-01-27T01:07:41.992831+0000","last_undegraded":"2024-01-27T01:07:41.992831+0000","last_fullsized":"2024-01-27T01:07:41.992831+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:45.033976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.020153+0000","last_change":"2024-01-27T01:07:42.017341+0000","last_active":"2024-01-27T01:07:42.020153+0000","last_peered":"2024-01-27T01:07:42.020153+0000","last_clean":"2024-01-27T01:07:42.020153+0000","last_became_active":"2024-01-27T01:07:42.015689+0000","last_became_peered":"2024-01-27T01:07:42.015689+0000","last_unstale":"2024-01-27T01:07:42.020153+0000","last_undegraded":"2024-01-27T01:07:42.020153+0000","last_fullsized":"2024-01-27T01:07:42.020153+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:07:07.746380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981055+0000","last_change":"2024-01-27T01:07:27.743490+0000","last_active":"2024-01-27T01:07:41.981055+0000","last_peered":"2024-01-27T01:07:41.981055+0000","last_clean":"2024-01-27T01:07:41.981055+0000","last_became_active":"2024-01-27T01:07:27.743207+0000","last_became_peered":"2024-01-27T01:07:27.743207+0000","last_unstale":"2024-01-27T01:07:41.981055+0000","last_undegraded":"2024-01-27T01:07:41.981055+0000","last_fullsized":"2024-01-27T01:07:41.981055+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:19.339556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"242'2","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982911+0000","last_change":"2024-01-27T01:07:26.946877+0000","last_active":"2024-01-27T01:07:41.982911+0000","last_peered":"2024-01-27T01:07:41.982911+0000","last_clean":"2024-01-27T01:07:41.982911+0000","last_became_active":"2024-01-27T01:07:26.946596+0000","last_became_peered":"2024-01-27T01:07:26.946596+0000","last_unstale":"2024-01-27T01:07:41.982911+0000","last_undegraded":"2024-01-27T01:07:41.982911+0000","last_fullsized":"2024-01-27T01:07:41.982911+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:27:41.114970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.010761+0000","last_change":"2024-01-27T01:07:42.009184+0000","last_active":"2024-01-27T01:07:42.010761+0000","last_peered":"2024-01-27T01:07:42.010761+0000","last_clean":"2024-01-27T01:07:42.010761+0000","last_became_active":"2024-01-27T01:07:42.008857+0000","last_became_peered":"2024-01-27T01:07:42.008857+0000","last_unstale":"2024-01-27T01:07:42.010761+0000","last_undegraded":"2024-01-27T01:07:42.010761+0000","last_fullsized":"2024-01-27T01:07:42.010761+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:04:28.461950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982436+0000","last_change":"2024-01-27T01:07:27.746482+0000","last_active":"2024-01-27T01:07:41.982436+0000","last_peered":"2024-01-27T01:07:41.982436+0000","last_clean":"2024-01-27T01:07:41.982436+0000","last_became_active":"2024-01-27T01:07:27.746052+0000","last_became_peered":"2024-01-27T01:07:27.746052+0000","last_unstale":"2024-01-27T01:07:41.982436+0000","last_undegraded":"2024-01-27T01:07:41.982436+0000","last_fullsized":"2024-01-27T01:07:41.982436+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:32.387353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"242'1","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.998825+0000","last_change":"2024-01-27T01:07:26.950943+0000","last_active":"2024-01-27T01:07:41.998825+0000","last_peered":"2024-01-27T01:07:41.998825+0000","last_clean":"2024-01-27T01:07:41.998825+0000","last_became_active":"2024-01-27T01:07:26.950251+0000","last_became_peered":"2024-01-27T01:07:26.950251+0000","last_unstale":"2024-01-27T01:07:41.998825+0000","last_undegraded":"2024-01-27T01:07:41.998825+0000","last_fullsized":"2024-01-27T01:07:41.998825+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:31.380205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.012092+0000","last_change":"2024-01-27T01:07:42.007675+0000","last_active":"2024-01-27T01:07:42.012092+0000","last_peered":"2024-01-27T01:07:42.012092+0000","last_clean":"2024-01-27T01:07:42.012092+0000","last_became_active":"2024-01-27T01:07:42.007228+0000","last_became_peered":"2024-01-27T01:07:42.007228+0000","last_unstale":"2024-01-27T01:07:42.012092+0000","last_undegraded":"2024-01-27T01:07:42.012092+0000","last_fullsized":"2024-01-27T01:07:42.012092+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:06:58.292213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.980187+0000","last_change":"2024-01-27T01:07:27.742779+0000","last_active":"2024-01-27T01:07:41.980187+0000","last_peered":"2024-01-27T01:07:41.980187+0000","last_clean":"2024-01-27T01:07:41.980187+0000","last_became_active":"2024-01-27T01:07:27.742462+0000","last_became_peered":"2024-01-27T01:07:27.742462+0000","last_unstale":"2024-01-27T01:07:41.980187+0000","last_undegraded":"2024-01-27T01:07:41.980187+0000","last_fullsized":"2024-01-27T01:07:41.980187+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:28.493839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.986365+0000","last_change":"2024-01-27T01:07:26.956417+0000","last_active":"2024-01-27T01:07:41.986365+0000","last_peered":"2024-01-27T01:07:41.986365+0000","last_clean":"2024-01-27T01:07:41.986365+0000","last_became_active":"2024-01-27T01:07:26.956128+0000","last_became_peered":"2024-01-27T01:07:26.956128+0000","last_unstale":"2024-01-27T01:07:41.986365+0000","last_undegraded":"2024-01-27T01:07:41.986365+0000","last_fullsized":"2024-01-27T01:07:41.986365+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:22.165694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.004889+0000","last_change":"2024-01-27T01:07:42.002859+0000","last_active":"2024-01-27T01:07:42.004889+0000","last_peered":"2024-01-27T01:07:42.004889+0000","last_clean":"2024-01-27T01:07:42.004889+0000","last_became_active":"2024-01-27T01:07:42.002585+0000","last_became_peered":"2024-01-27T01:07:42.002585+0000","last_unstale":"2024-01-27T01:07:42.004889+0000","last_undegraded":"2024-01-27T01:07:42.004889+0000","last_fullsized":"2024-01-27T01:07:42.004889+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:17.562068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979747+0000","last_change":"2024-01-27T01:07:27.752330+0000","last_active":"2024-01-27T01:07:41.979747+0000","last_peered":"2024-01-27T01:07:41.979747+0000","last_clean":"2024-01-27T01:07:41.979747+0000","last_became_active":"2024-01-27T01:07:27.752040+0000","last_became_peered":"2024-01-27T01:07:27.752040+0000","last_unstale":"2024-01-27T01:07:41.979747+0000","last_undegraded":"2024-01-27T01:07:41.979747+0000","last_fullsized":"2024-01-27T01:07:41.979747+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:36:14.454947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"242'2","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.980077+0000","last_change":"2024-01-27T01:07:26.947483+0000","last_active":"2024-01-27T01:07:41.980077+0000","last_peered":"2024-01-27T01:07:41.980077+0000","last_clean":"2024-01-27T01:07:41.980077+0000","last_became_active":"2024-01-27T01:07:26.947172+0000","last_became_peered":"2024-01-27T01:07:26.947172+0000","last_unstale":"2024-01-27T01:07:41.980077+0000","last_undegraded":"2024-01-27T01:07:41.980077+0000","last_fullsized":"2024-01-27T01:07:41.980077+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:03.604469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.003800+0000","last_change":"2024-01-27T01:07:42.002232+0000","last_active":"2024-01-27T01:07:42.003800+0000","last_peered":"2024-01-27T01:07:42.003800+0000","last_clean":"2024-01-27T01:07:42.003800+0000","last_became_active":"2024-01-27T01:07:42.001905+0000","last_became_peered":"2024-01-27T01:07:42.001905+0000","last_unstale":"2024-01-27T01:07:42.003800+0000","last_undegraded":"2024-01-27T01:07:42.003800+0000","last_fullsized":"2024-01-27T01:07:42.003800+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:45:46.843500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984511+0000","last_change":"2024-01-27T01:07:27.755356+0000","last_active":"2024-01-27T01:07:41.984511+0000","last_peered":"2024-01-27T01:07:41.984511+0000","last_clean":"2024-01-27T01:07:41.984511+0000","last_became_active":"2024-01-27T01:07:27.755108+0000","last_became_peered":"2024-01-27T01:07:27.755108+0000","last_unstale":"2024-01-27T01:07:41.984511+0000","last_undegraded":"2024-01-27T01:07:41.984511+0000","last_fullsized":"2024-01-27T01:07:41.984511+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:51:14.988606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995367+0000","last_change":"2024-01-27T01:07:26.950680+0000","last_active":"2024-01-27T01:07:41.995367+0000","last_peered":"2024-01-27T01:07:41.995367+0000","last_clean":"2024-01-27T01:07:41.995367+0000","last_became_active":"2024-01-27T01:07:26.950395+0000","last_became_peered":"2024-01-27T01:07:26.950395+0000","last_unstale":"2024-01-27T01:07:41.995367+0000","last_undegraded":"2024-01-27T01:07:41.995367+0000","last_fullsized":"2024-01-27T01:07:41.995367+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:28:11.238816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.002928+0000","last_change":"2024-01-27T01:07:42.000220+0000","last_active":"2024-01-27T01:07:42.002928+0000","last_peered":"2024-01-27T01:07:42.002928+0000","last_clean":"2024-01-27T01:07:42.002928+0000","last_became_active":"2024-01-27T01:07:41.999872+0000","last_became_peered":"2024-01-27T01:07:41.999872+0000","last_unstale":"2024-01-27T01:07:42.002928+0000","last_undegraded":"2024-01-27T01:07:42.002928+0000","last_fullsized":"2024-01-27T01:07:42.002928+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:40:17.514841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995284+0000","last_change":"2024-01-27T01:07:27.748049+0000","last_active":"2024-01-27T01:07:41.995284+0000","last_peered":"2024-01-27T01:07:41.995284+0000","last_clean":"2024-01-27T01:07:41.995284+0000","last_became_active":"2024-01-27T01:07:27.747765+0000","last_became_peered":"2024-01-27T01:07:27.747765+0000","last_unstale":"2024-01-27T01:07:41.995284+0000","last_undegraded":"2024-01-27T01:07:41.995284+0000","last_fullsized":"2024-01-27T01:07:41.995284+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:51:33.521276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.996168+0000","last_change":"2024-01-27T01:07:26.950170+0000","last_active":"2024-01-27T01:07:41.996168+0000","last_peered":"2024-01-27T01:07:41.996168+0000","last_clean":"2024-01-27T01:07:41.996168+0000","last_became_active":"2024-01-27T01:07:26.949857+0000","last_became_peered":"2024-01-27T01:07:26.949857+0000","last_unstale":"2024-01-27T01:07:41.996168+0000","last_undegraded":"2024-01-27T01:07:41.996168+0000","last_fullsized":"2024-01-27T01:07:41.996168+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:29.510893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.007784+0000","last_change":"2024-01-27T01:07:42.005851+0000","last_active":"2024-01-27T01:07:42.007784+0000","last_peered":"2024-01-27T01:07:42.007784+0000","last_clean":"2024-01-27T01:07:42.007784+0000","last_became_active":"2024-01-27T01:07:42.004370+0000","last_became_peered":"2024-01-27T01:07:42.004370+0000","last_unstale":"2024-01-27T01:07:42.007784+0000","last_undegraded":"2024-01-27T01:07:42.007784+0000","last_fullsized":"2024-01-27T01:07:42.007784+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:59.153074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.997166+0000","last_change":"2024-01-27T01:07:27.753040+0000","last_active":"2024-01-27T01:07:41.997166+0000","last_peered":"2024-01-27T01:07:41.997166+0000","last_clean":"2024-01-27T01:07:41.997166+0000","last_became_active":"2024-01-27T01:07:27.752784+0000","last_became_peered":"2024-01-27T01:07:27.752784+0000","last_unstale":"2024-01-27T01:07:41.997166+0000","last_undegraded":"2024-01-27T01:07:41.997166+0000","last_fullsized":"2024-01-27T01:07:41.997166+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:09:57.375468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.997934+0000","last_change":"2024-01-27T01:07:26.944368+0000","last_active":"2024-01-27T01:07:41.997934+0000","last_peered":"2024-01-27T01:07:41.997934+0000","last_clean":"2024-01-27T01:07:41.997934+0000","last_became_active":"2024-01-27T01:07:26.944076+0000","last_became_peered":"2024-01-27T01:07:26.944076+0000","last_unstale":"2024-01-27T01:07:41.997934+0000","last_undegraded":"2024-01-27T01:07:41.997934+0000","last_fullsized":"2024-01-27T01:07:41.997934+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:59:38.148215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.999202+0000","last_change":"2024-01-27T01:07:41.991392+0000","last_active":"2024-01-27T01:07:41.999202+0000","last_peered":"2024-01-27T01:07:41.999202+0000","last_clean":"2024-01-27T01:07:41.999202+0000","last_became_active":"2024-01-27T01:07:41.991102+0000","last_became_peered":"2024-01-27T01:07:41.991102+0000","last_unstale":"2024-01-27T01:07:41.999202+0000","last_undegraded":"2024-01-27T01:07:41.999202+0000","last_fullsized":"2024-01-27T01:07:41.999202+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:25:15.639710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984464+0000","last_change":"2024-01-27T01:07:27.747646+0000","last_active":"2024-01-27T01:07:41.984464+0000","last_peered":"2024-01-27T01:07:41.984464+0000","last_clean":"2024-01-27T01:07:41.984464+0000","last_became_active":"2024-01-27T01:07:27.747345+0000","last_became_peered":"2024-01-27T01:07:27.747345+0000","last_unstale":"2024-01-27T01:07:41.984464+0000","last_undegraded":"2024-01-27T01:07:41.984464+0000","last_fullsized":"2024-01-27T01:07:41.984464+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:30:45.126288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"242'2","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979523+0000","last_change":"2024-01-27T01:07:26.950235+0000","last_active":"2024-01-27T01:07:41.979523+0000","last_peered":"2024-01-27T01:07:41.979523+0000","last_clean":"2024-01-27T01:07:41.979523+0000","last_became_active":"2024-01-27T01:07:26.949923+0000","last_became_peered":"2024-01-27T01:07:26.949923+0000","last_unstale":"2024-01-27T01:07:41.979523+0000","last_undegraded":"2024-01-27T01:07:41.979523+0000","last_fullsized":"2024-01-27T01:07:41.979523+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:27:55.458182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.020664+0000","last_change":"2024-01-27T01:07:42.019294+0000","last_active":"2024-01-27T01:07:42.020664+0000","last_peered":"2024-01-27T01:07:42.020664+0000","last_clean":"2024-01-27T01:07:42.020664+0000","last_became_active":"2024-01-27T01:07:42.018289+0000","last_became_peered":"2024-01-27T01:07:42.018289+0000","last_unstale":"2024-01-27T01:07:42.020664+0000","last_undegraded":"2024-01-27T01:07:42.020664+0000","last_fullsized":"2024-01-27T01:07:42.020664+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:24:03.567201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984578+0000","last_change":"2024-01-27T01:07:27.758370+0000","last_active":"2024-01-27T01:07:41.984578+0000","last_peered":"2024-01-27T01:07:41.984578+0000","last_clean":"2024-01-27T01:07:41.984578+0000","last_became_active":"2024-01-27T01:07:27.758007+0000","last_became_peered":"2024-01-27T01:07:27.758007+0000","last_unstale":"2024-01-27T01:07:41.984578+0000","last_undegraded":"2024-01-27T01:07:41.984578+0000","last_fullsized":"2024-01-27T01:07:41.984578+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:07.518631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"243'1","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981495+0000","last_change":"2024-01-27T01:07:26.953360+0000","last_active":"2024-01-27T01:07:41.981495+0000","last_peered":"2024-01-27T01:07:41.981495+0000","last_clean":"2024-01-27T01:07:41.981495+0000","last_became_active":"2024-01-27T01:07:26.953090+0000","last_became_peered":"2024-01-27T01:07:26.953090+0000","last_unstale":"2024-01-27T01:07:41.981495+0000","last_undegraded":"2024-01-27T01:07:41.981495+0000","last_fullsized":"2024-01-27T01:07:41.981495+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:10.549778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.017058+0000","last_change":"2024-01-27T01:07:42.014257+0000","last_active":"2024-01-27T01:07:42.017058+0000","last_peered":"2024-01-27T01:07:42.017058+0000","last_clean":"2024-01-27T01:07:42.017058+0000","last_became_active":"2024-01-27T01:07:42.006612+0000","last_became_peered":"2024-01-27T01:07:42.006612+0000","last_unstale":"2024-01-27T01:07:42.017058+0000","last_undegraded":"2024-01-27T01:07:42.017058+0000","last_fullsized":"2024-01-27T01:07:42.017058+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:20.301704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979085+0000","last_change":"2024-01-27T01:07:27.750264+0000","last_active":"2024-01-27T01:07:41.979085+0000","last_peered":"2024-01-27T01:07:41.979085+0000","last_clean":"2024-01-27T01:07:41.979085+0000","last_became_active":"2024-01-27T01:07:27.749987+0000","last_became_peered":"2024-01-27T01:07:27.749987+0000","last_unstale":"2024-01-27T01:07:41.979085+0000","last_undegraded":"2024-01-27T01:07:41.979085+0000","last_fullsized":"2024-01-27T01:07:41.979085+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:19.339556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.994047+0000","last_change":"2024-01-27T01:07:26.942700+0000","last_active":"2024-01-27T01:07:41.994047+0000","last_peered":"2024-01-27T01:07:41.994047+0000","last_clean":"2024-01-27T01:07:41.994047+0000","last_became_active":"2024-01-27T01:07:26.942491+0000","last_became_peered":"2024-01-27T01:07:26.942491+0000","last_unstale":"2024-01-27T01:07:41.994047+0000","last_undegraded":"2024-01-27T01:07:41.994047+0000","last_fullsized":"2024-01-27T01:07:41.994047+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:03:40.104583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.993239+0000","last_change":"2024-01-27T01:07:41.989219+0000","last_active":"2024-01-27T01:07:41.993239+0000","last_peered":"2024-01-27T01:07:41.993239+0000","last_clean":"2024-01-27T01:07:41.993239+0000","last_became_active":"2024-01-27T01:07:41.988917+0000","last_became_peered":"2024-01-27T01:07:41.988917+0000","last_unstale":"2024-01-27T01:07:41.993239+0000","last_undegraded":"2024-01-27T01:07:41.993239+0000","last_fullsized":"2024-01-27T01:07:41.993239+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:19.572287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981584+0000","last_change":"2024-01-27T01:07:27.746804+0000","last_active":"2024-01-27T01:07:41.981584+0000","last_peered":"2024-01-27T01:07:41.981584+0000","last_clean":"2024-01-27T01:07:41.981584+0000","last_became_active":"2024-01-27T01:07:27.746110+0000","last_became_peered":"2024-01-27T01:07:27.746110+0000","last_unstale":"2024-01-27T01:07:41.981584+0000","last_undegraded":"2024-01-27T01:07:41.981584+0000","last_fullsized":"2024-01-27T01:07:41.981584+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:48:18.465014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.994604+0000","last_change":"2024-01-27T01:07:26.943462+0000","last_active":"2024-01-27T01:07:41.994604+0000","last_peered":"2024-01-27T01:07:41.994604+0000","last_clean":"2024-01-27T01:07:41.994604+0000","last_became_active":"2024-01-27T01:07:26.943193+0000","last_became_peered":"2024-01-27T01:07:26.943193+0000","last_unstale":"2024-01-27T01:07:41.994604+0000","last_undegraded":"2024-01-27T01:07:41.994604+0000","last_fullsized":"2024-01-27T01:07:41.994604+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:21:20.952966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.010918+0000","last_change":"2024-01-27T01:07:42.004836+0000","last_active":"2024-01-27T01:07:42.010918+0000","last_peered":"2024-01-27T01:07:42.010918+0000","last_clean":"2024-01-27T01:07:42.010918+0000","last_became_active":"2024-01-27T01:07:42.004530+0000","last_became_peered":"2024-01-27T01:07:42.004530+0000","last_unstale":"2024-01-27T01:07:42.010918+0000","last_undegraded":"2024-01-27T01:07:42.010918+0000","last_fullsized":"2024-01-27T01:07:42.010918+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:58:40.048139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.994218+0000","last_change":"2024-01-27T01:07:27.739197+0000","last_active":"2024-01-27T01:07:41.994218+0000","last_peered":"2024-01-27T01:07:41.994218+0000","last_clean":"2024-01-27T01:07:41.994218+0000","last_became_active":"2024-01-27T01:07:27.738911+0000","last_became_peered":"2024-01-27T01:07:27.738911+0000","last_unstale":"2024-01-27T01:07:41.994218+0000","last_undegraded":"2024-01-27T01:07:41.994218+0000","last_fullsized":"2024-01-27T01:07:41.994218+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:41:00.947931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995492+0000","last_change":"2024-01-27T01:07:26.949530+0000","last_active":"2024-01-27T01:07:41.995492+0000","last_peered":"2024-01-27T01:07:41.995492+0000","last_clean":"2024-01-27T01:07:41.995492+0000","last_became_active":"2024-01-27T01:07:26.949204+0000","last_became_peered":"2024-01-27T01:07:26.949204+0000","last_unstale":"2024-01-27T01:07:41.995492+0000","last_undegraded":"2024-01-27T01:07:41.995492+0000","last_fullsized":"2024-01-27T01:07:41.995492+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:16:54.275653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3f","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.013308+0000","last_change":"2024-01-27T01:07:42.009249+0000","last_active":"2024-01-27T01:07:42.013308+0000","last_peered":"2024-01-27T01:07:42.013308+0000","last_clean":"2024-01-27T01:07:42.013308+0000","last_became_active":"2024-01-27T01:07:42.008946+0000","last_became_peered":"2024-01-27T01:07:42.008946+0000","last_unstale":"2024-01-27T01:07:42.013308+0000","last_undegraded":"2024-01-27T01:07:42.013308+0000","last_fullsized":"2024-01-27T01:07:42.013308+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:48:32.501055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.997013+0000","last_change":"2024-01-27T01:07:27.744772+0000","last_active":"2024-01-27T01:07:41.997013+0000","last_peered":"2024-01-27T01:07:41.997013+0000","last_clean":"2024-01-27T01:07:41.997013+0000","last_became_active":"2024-01-27T01:07:27.744416+0000","last_became_peered":"2024-01-27T01:07:27.744416+0000","last_unstale":"2024-01-27T01:07:41.997013+0000","last_undegraded":"2024-01-27T01:07:41.997013+0000","last_fullsized":"2024-01-27T01:07:41.997013+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:59:39.155363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981719+0000","last_change":"2024-01-27T01:07:26.947692+0000","last_active":"2024-01-27T01:07:41.981719+0000","last_peered":"2024-01-27T01:07:41.981719+0000","last_clean":"2024-01-27T01:07:41.981719+0000","last_became_active":"2024-01-27T01:07:26.947404+0000","last_became_peered":"2024-01-27T01:07:26.947404+0000","last_unstale":"2024-01-27T01:07:41.981719+0000","last_undegraded":"2024-01-27T01:07:41.981719+0000","last_fullsized":"2024-01-27T01:07:41.981719+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:12:55.245326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.017835+0000","last_change":"2024-01-27T01:07:42.015538+0000","last_active":"2024-01-27T01:07:42.017835+0000","last_peered":"2024-01-27T01:07:42.017835+0000","last_clean":"2024-01-27T01:07:42.017835+0000","last_became_active":"2024-01-27T01:07:42.011906+0000","last_became_peered":"2024-01-27T01:07:42.011906+0000","last_unstale":"2024-01-27T01:07:42.017835+0000","last_undegraded":"2024-01-27T01:07:42.017835+0000","last_fullsized":"2024-01-27T01:07:42.017835+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:49:46.679016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.983288+0000","last_change":"2024-01-27T01:07:27.749874+0000","last_active":"2024-01-27T01:07:41.983288+0000","last_peered":"2024-01-27T01:07:41.983288+0000","last_clean":"2024-01-27T01:07:41.983288+0000","last_became_active":"2024-01-27T01:07:27.749252+0000","last_became_peered":"2024-01-27T01:07:27.749252+0000","last_unstale":"2024-01-27T01:07:41.983288+0000","last_undegraded":"2024-01-27T01:07:41.983288+0000","last_fullsized":"2024-01-27T01:07:41.983288+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:18:53.285178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979216+0000","last_change":"2024-01-27T01:07:26.947606+0000","last_active":"2024-01-27T01:07:41.979216+0000","last_peered":"2024-01-27T01:07:41.979216+0000","last_clean":"2024-01-27T01:07:41.979216+0000","last_became_active":"2024-01-27T01:07:26.947287+0000","last_became_peered":"2024-01-27T01:07:26.947287+0000","last_unstale":"2024-01-27T01:07:41.979216+0000","last_undegraded":"2024-01-27T01:07:41.979216+0000","last_fullsized":"2024-01-27T01:07:41.979216+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:34:14.649324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.007691+0000","last_change":"2024-01-27T01:07:42.004792+0000","last_active":"2024-01-27T01:07:42.007691+0000","last_peered":"2024-01-27T01:07:42.007691+0000","last_clean":"2024-01-27T01:07:42.007691+0000","last_became_active":"2024-01-27T01:07:42.004529+0000","last_became_peered":"2024-01-27T01:07:42.004529+0000","last_unstale":"2024-01-27T01:07:42.007691+0000","last_undegraded":"2024-01-27T01:07:42.007691+0000","last_fullsized":"2024-01-27T01:07:42.007691+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:04:01.316302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.997262+0000","last_change":"2024-01-27T01:07:27.749856+0000","last_active":"2024-01-27T01:07:41.997262+0000","last_peered":"2024-01-27T01:07:41.997262+0000","last_clean":"2024-01-27T01:07:41.997262+0000","last_became_active":"2024-01-27T01:07:27.749595+0000","last_became_peered":"2024-01-27T01:07:27.749595+0000","last_unstale":"2024-01-27T01:07:41.997262+0000","last_undegraded":"2024-01-27T01:07:41.997262+0000","last_fullsized":"2024-01-27T01:07:41.997262+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:25:38.146534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995964+0000","last_change":"2024-01-27T01:07:26.950737+0000","last_active":"2024-01-27T01:07:41.995964+0000","last_peered":"2024-01-27T01:07:41.995964+0000","last_clean":"2024-01-27T01:07:41.995964+0000","last_became_active":"2024-01-27T01:07:26.950498+0000","last_became_peered":"2024-01-27T01:07:26.950498+0000","last_unstale":"2024-01-27T01:07:41.995964+0000","last_undegraded":"2024-01-27T01:07:41.995964+0000","last_fullsized":"2024-01-27T01:07:41.995964+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:24:42.558412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.020131+0000","last_change":"2024-01-27T01:07:42.017196+0000","last_active":"2024-01-27T01:07:42.020131+0000","last_peered":"2024-01-27T01:07:42.020131+0000","last_clean":"2024-01-27T01:07:42.020131+0000","last_became_active":"2024-01-27T01:07:42.012218+0000","last_became_peered":"2024-01-27T01:07:42.012218+0000","last_unstale":"2024-01-27T01:07:42.020131+0000","last_undegraded":"2024-01-27T01:07:42.020131+0000","last_fullsized":"2024-01-27T01:07:42.020131+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:41.431166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.987124+0000","last_change":"2024-01-27T01:07:27.746007+0000","last_active":"2024-01-27T01:07:41.987124+0000","last_peered":"2024-01-27T01:07:41.987124+0000","last_clean":"2024-01-27T01:07:41.987124+0000","last_became_active":"2024-01-27T01:07:27.745722+0000","last_became_peered":"2024-01-27T01:07:27.745722+0000","last_unstale":"2024-01-27T01:07:41.987124+0000","last_undegraded":"2024-01-27T01:07:41.987124+0000","last_fullsized":"2024-01-27T01:07:41.987124+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:09:23.112301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"242'1","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.986221+0000","last_change":"2024-01-27T01:07:26.951812+0000","last_active":"2024-01-27T01:07:41.986221+0000","last_peered":"2024-01-27T01:07:41.986221+0000","last_clean":"2024-01-27T01:07:41.986221+0000","last_became_active":"2024-01-27T01:07:26.951141+0000","last_became_peered":"2024-01-27T01:07:26.951141+0000","last_unstale":"2024-01-27T01:07:41.986221+0000","last_undegraded":"2024-01-27T01:07:41.986221+0000","last_fullsized":"2024-01-27T01:07:41.986221+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:06:20.506111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"230'8772","reported_seq":9763,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984186+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:07:41.984186+0000","last_peered":"2024-01-27T01:07:41.984186+0000","last_clean":"2024-01-27T01:07:41.984186+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:07:41.984186+0000","last_undegraded":"2024-01-27T01:07:41.984186+0000","last_fullsized":"2024-01-27T01:07:41.984186+0000","mapping_epoch":14,"log_start":"98'2229","ondisk_log_start":"98'2229","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2456,"num_read_kb":5220,"num_write":670,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":3538,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2456,"num_read_kb":5220,"num_write":670,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6543,"ondisk_log_size":6543,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576015,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300924,"kb_used_data":3636,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94070916,"statfs":{"total":96636764160,"available":96328617984,"internally_reserved":0,"allocated":3723264,"data_stored":2245329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.312},"max":{"1min":0.72499999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.63200000000000001,"15min":0.621},"min":{"1min":0.48099999999999998,"5min":0.39900000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":0.879,"15min":0.879},"last":0.72899999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70299999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.63800000000000001,"15min":0.63100000000000001},"min":{"1min":0.44500000000000001,"5min":0.42099999999999999,"15min":0.34300000000000003},"max":{"1min":0.73499999999999999,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.77400000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.61899999999999999,"15min":0.60999999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.72899999999999998,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.753},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.628,"15min":0.629},"min":{"1min":0.372,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.69499999999999995,"5min":0.872,"15min":0.88900000000000001},"last":0.85799999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.60899999999999999,"15min":0.61399999999999999},"min":{"1min":0.38500000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69099999999999995,"5min":0.876,"15min":0.876},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.65700000000000003,"15min":0.64200000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80700000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.80600000000000005}]},{"osd":4,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.503,"15min":0.48799999999999999},"min":{"1min":0.35199999999999998,"5min":0.314,"15min":0.29999999999999999},"max":{"1min":0.65500000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53700000000000003,"15min":0.51600000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.629,"5min":0.80300000000000005,"15min":0.86599999999999999},"last":0.63500000000000001}]},{"osd":5,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.55300000000000005,"15min":0.52500000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.80400000000000005},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57299999999999995,"15min":0.56399999999999995},"min":{"1min":0.41699999999999998,"5min":0.34799999999999998,"15min":0.34399999999999997},"max":{"1min":0.79200000000000004,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.66000000000000003}]},{"osd":6,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60499999999999998,"15min":0.58899999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.60499999999999998,"15min":0.58699999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.75,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.69999999999999996}]}]},{"osd":6,"up_from":13,"seq":55834576015,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308520,"kb_used_data":3040,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063320,"statfs":{"total":96636764160,"available":96320839680,"internally_reserved":0,"allocated":3112960,"data_stored":1646440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.59199999999999997,"15min":0.59899999999999998},"min":{"1min":0.40999999999999998,"5min":0.40799999999999997,"15min":0.36399999999999999},"max":{"1min":0.65400000000000003,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60199999999999998,"15min":0.61199999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.376},"max":{"1min":0.67600000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.73499999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.59199999999999997,"15min":0.60299999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.65600000000000003,"5min":0.80300000000000005,"15min":0.84499999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60199999999999998,"15min":0.61699999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81200000000000006,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.76400000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.58299999999999996,"15min":0.61099999999999999},"min":{"1min":0.39200000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70999999999999996,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69999999999999996,"5min":0.80400000000000005,"15min":0.878},"last":0.754}]},{"osd":3,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61399999999999999,"15min":0.623},"min":{"1min":0.379,"5min":0.378,"15min":0.378},"max":{"1min":0.72599999999999998,"5min":0.83699999999999997,"15min":0.90100000000000002},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64500000000000002,"15min":0.65400000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.754,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.877}]},{"osd":4,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51300000000000001,"15min":0.502},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.71399999999999997,"5min":0.77600000000000002,"15min":0.86599999999999999},"last":0.63},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.36599999999999999,"5min":0.33800000000000002,"15min":0.29799999999999999},"max":{"1min":0.69299999999999995,"5min":0.78500000000000003,"15min":0.88900000000000001},"last":0.81799999999999995}]},{"osd":5,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.69199999999999995,"5min":0.75900000000000001,"15min":0.79300000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.58299999999999996,"15min":0.57399999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.28899999999999998},"max":{"1min":0.84199999999999997,"5min":0.87,"15min":0.874},"last":0.71099999999999997}]},{"osd":7,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57799999999999996,"15min":0.57399999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82099999999999995,"5min":0.84899999999999998,"15min":0.92900000000000005},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58999999999999997,"15min":0.58899999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.34699999999999998},"max":{"1min":0.76500000000000001,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.79300000000000004}]}]},{"osd":1,"up_from":13,"seq":55834576015,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304420,"kb_used_data":3036,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067420,"statfs":{"total":96636764160,"available":96325038080,"internally_reserved":0,"allocated":3108864,"data_stored":1646418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.28499999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.48899999999999999,"5min":0.66100000000000003,"15min":0.69599999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.625,"5min":0.67800000000000005,"15min":0.76600000000000001},"last":0.308}]},{"osd":2,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.42699999999999999,"15min":0.435},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.26200000000000001},"max":{"1min":0.54900000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":0.48799999999999999},"min":{"1min":0.36699999999999999,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.58899999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.41599999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51800000000000002,"15min":0.53100000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66200000000000003,"5min":0.84799999999999998,"15min":1.2130000000000001},"last":0.441},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.5,"15min":0.51300000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.32300000000000001},"max":{"1min":0.65100000000000002,"5min":1.125,"15min":1.22},"last":0.38600000000000001}]},{"osd":4,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.66900000000000004,"15min":0.67000000000000004},"min":{"1min":0.41599999999999998,"5min":0.41099999999999998,"15min":0.39800000000000002},"max":{"1min":0.92100000000000004,"5min":1.242,"15min":1.242},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.68500000000000005,"15min":0.69199999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.35699999999999998},"max":{"1min":0.80500000000000005,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.495}]},{"osd":5,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.72099999999999997,"15min":0.70499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1,"5min":5.444,"15min":5.444},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71399999999999997,"15min":0.70599999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.435},"max":{"1min":0.97599999999999998,"5min":2.6669999999999998,"15min":2.6669999999999998},"last":0.57199999999999995}]},{"osd":6,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.73599999999999999,"15min":0.72099999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.47099999999999997},"max":{"1min":0.89700000000000002,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.80600000000000005,"15min":0.74199999999999999},"min":{"1min":0.52400000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0600000000000001,"5min":11.223000000000001,"15min":11.223000000000001},"last":0.502}]},{"osd":7,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.749,"15min":0.753},"min":{"1min":0.54100000000000004,"5min":0.52800000000000002,"15min":0.48999999999999999},"max":{"1min":1.0349999999999999,"5min":2.9660000000000002,"15min":2.9660000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.76900000000000002,"15min":0.76700000000000002},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.52000000000000002},"max":{"1min":0.95599999999999996,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.55400000000000005}]}]},{"osd":0,"up_from":13,"seq":55834576014,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300912,"kb_used_data":3624,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94070928,"statfs":{"total":96636764160,"available":96328630272,"internally_reserved":0,"allocated":3710976,"data_stored":2239500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.434,"15min":0.42499999999999999},"min":{"1min":0.28899999999999998,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.443,"5min":0.45100000000000001,"15min":0.46000000000000002},"min":{"1min":0.33200000000000002,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":0.64300000000000002,"5min":0.65600000000000003,"15min":0.73399999999999999},"last":0.33200000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.41899999999999998,"15min":0.433},"min":{"1min":0.27400000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.56499999999999995,"5min":0.66900000000000004,"15min":0.76800000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.47599999999999998,"15min":0.48599999999999999},"min":{"1min":0.32900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60299999999999998,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.378}]},{"osd":3,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.34599999999999997,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.83699999999999997},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.503,"15min":0.504},"min":{"1min":0.35899999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.65900000000000003,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.35899999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.67600000000000005,"15min":0.67200000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.42199999999999999},"max":{"1min":0.91200000000000003,"5min":0.91500000000000004,"15min":1.0720000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.65900000000000003,"15min":0.67600000000000005},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.90100000000000002,"5min":1.002,"15min":1.002},"last":0.59799999999999998}]},{"osd":5,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.68700000000000006,"15min":0.69799999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95799999999999996,"5min":1.0209999999999999,"15min":2.165},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69499999999999995,"15min":0.70999999999999996},"min":{"1min":0.50700000000000001,"5min":0.48799999999999999,"15min":0.45600000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.1399999999999999},"last":0.58099999999999996}]},{"osd":6,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68799999999999994,"15min":0.70999999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.71399999999999997,"15min":0.73699999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.88700000000000001,"5min":1.0109999999999999,"15min":1.425},"last":0.56000000000000005}]},{"osd":7,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.73099999999999998,"15min":0.752},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.99199999999999999,"5min":1.0169999999999999,"15min":1.1259999999999999},"last":0.495},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.755,"15min":0.76600000000000001},"min":{"1min":0.55000000000000004,"5min":0.51100000000000001,"15min":0.48499999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.1910000000000001},"last":0.61599999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576014,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312644,"kb_used_data":3068,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059196,"statfs":{"total":96636764160,"available":96316616704,"internally_reserved":0,"allocated":3141632,"data_stored":1656997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39600000000000002,"15min":0.40400000000000003},"min":{"1min":0.27200000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.497,"5min":0.746,"15min":0.746},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.441,"15min":0.443},"min":{"1min":0.34599999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.33000000000000002}]},{"osd":1,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44500000000000001,"15min":0.44600000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.50900000000000001,"15min":0.51900000000000002},"min":{"1min":0.40699999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.45800000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.49299999999999999,"15min":0.496},"min":{"1min":0.38700000000000001,"5min":0.34699999999999998,"15min":0.33200000000000002},"max":{"1min":0.65000000000000002,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.47799999999999998,"15min":0.48699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.63900000000000001,"5min":0.65500000000000003,"15min":0.75700000000000001},"last":0.38600000000000001}]},{"osd":4,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.77800000000000002,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.437},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.68100000000000005,"15min":0.69199999999999995},"min":{"1min":0.47899999999999998,"5min":0.47099999999999997,"15min":0.378},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.107},"last":0.55400000000000005}]},{"osd":5,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.69699999999999995,"15min":0.70199999999999996},"min":{"1min":0.47399999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.075,"5min":1.075,"15min":1.167},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.69199999999999995,"15min":0.70499999999999996},"min":{"1min":0.52300000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81899999999999995,"5min":0.97499999999999998,"15min":1.05},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.70099999999999996,"15min":0.70899999999999996},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.90200000000000002,"5min":0.97899999999999998,"15min":1.077},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.69599999999999995,"15min":0.71499999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.39200000000000002},"max":{"1min":0.85999999999999999,"5min":0.96699999999999997,"15min":1.135},"last":0.57699999999999996}]},{"osd":7,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73399999999999999,"15min":0.74399999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.198},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.755,"15min":0.76500000000000001},"min":{"1min":0.53600000000000003,"5min":0.53100000000000003,"15min":0.48199999999999998},"max":{"1min":0.875,"5min":0.96399999999999997,"15min":1.0880000000000001},"last":0.42799999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576013,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304452,"kb_used_data":3068,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067388,"statfs":{"total":96636764160,"available":96325005312,"internally_reserved":0,"allocated":3141632,"data_stored":1657729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.439,"15min":0.42199999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63400000000000001,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44,"15min":0.45400000000000001},"min":{"1min":0.30599999999999999,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.66400000000000003,"5min":0.66700000000000004,"15min":0.84699999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.61699999999999999,"15min":0.53200000000000003},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.28899999999999998},"max":{"1min":0.76100000000000001,"5min":15.391,"15min":15.391},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.52400000000000002,"15min":0.52300000000000002},"min":{"1min":0.35799999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.628,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.379}]},{"osd":2,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.53600000000000003,"15min":0.51300000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70299999999999996,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.50900000000000001},"min":{"1min":0.308,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67600000000000005,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":0.308}]},{"osd":4,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.72899999999999998,"15min":0.71499999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.439},"max":{"1min":0.89200000000000002,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.71499999999999997,"15min":0.70299999999999996},"min":{"1min":0.47499999999999998,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":0.96699999999999997,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.48899999999999999}]},{"osd":5,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.73099999999999998,"15min":0.70899999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.94499999999999995,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.5},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.72699999999999998,"15min":0.71499999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.435},"max":{"1min":0.872,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.51800000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.73799999999999999,"15min":0.71999999999999997},"min":{"1min":0.49199999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.96699999999999997,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92900000000000005,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.745,"15min":0.73199999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.91400000000000003,"5min":4.569,"15min":4.569},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.78900000000000003,"15min":0.77800000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97999999999999998,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.67700000000000005}]}]},{"osd":4,"up_from":13,"seq":55834576015,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316716,"kb_used_data":3044,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055124,"statfs":{"total":96636764160,"available":96312446976,"internally_reserved":0,"allocated":3117056,"data_stored":1647900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.436,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60199999999999998,"15min":0.61299999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.753,"5min":0.82199999999999995,"15min":0.85699999999999998},"last":0.35499999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60999999999999999,"15min":0.628},"min":{"1min":0.40699999999999997,"5min":0.39100000000000001,"15min":0.38800000000000001},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60999999999999999,"15min":0.623},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82299999999999995,"5min":0.874,"15min":0.874},"last":0.497}]},{"osd":2,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.84699999999999998,"5min":0.85599999999999998,"15min":0.88700000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.48999999999999999,"5min":0.42899999999999999,"15min":0.42799999999999999},"max":{"1min":0.79200000000000004,"5min":0.81499999999999995,"15min":0.89200000000000002},"last":0.51900000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60399999999999998,"15min":0.623},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.80800000000000005,"5min":0.83299999999999996,"15min":0.84199999999999997},"last":0.377},{"interface":"front","average":{"1min":0.628,"5min":0.621,"15min":0.629},"min":{"1min":0.42299999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.47099999999999997}]},{"osd":5,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47999999999999998,"15min":0.48399999999999999},"min":{"1min":0.32400000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.85899999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.503,"15min":0.505},"min":{"1min":0.36299999999999999,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.878},"last":0.44800000000000001}]},{"osd":6,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.50600000000000001,"15min":0.499},"min":{"1min":0.38300000000000001,"5min":0.35799999999999998,"15min":0.34999999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.88700000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.375,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.54600000000000004}]},{"osd":7,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56200000000000006,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.748,"5min":0.80400000000000005,"15min":0.90200000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.38600000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.92600000000000005},"last":0.53000000000000003}]}]},{"osd":5,"up_from":13,"seq":55834576015,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312620,"kb_used_data":3044,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059220,"statfs":{"total":96636764160,"available":96316641280,"internally_reserved":0,"allocated":3117056,"data_stored":1654331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.58799999999999997,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.27300000000000002},"max":{"1min":0.73999999999999999,"5min":0.82799999999999996,"15min":1.006},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.501,"5min":0.57099999999999995,"15min":0.59799999999999998},"min":{"1min":0.37,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73499999999999999,"5min":0.81499999999999995,"15min":1.1719999999999999},"last":0.54900000000000004}]},{"osd":1,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.61599999999999999,"15min":0.64300000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69699999999999995,"5min":0.89000000000000001,"15min":0.95899999999999996},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.65500000000000003,"15min":0.66400000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.377},"max":{"1min":0.77900000000000003,"5min":0.86899999999999999,"15min":1.212},"last":0.623}]},{"osd":2,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56200000000000006,"15min":0.58699999999999997},"min":{"1min":0.34699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70999999999999996,"5min":0.82099999999999995,"15min":1.631},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.60299999999999998,"15min":0.61299999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34000000000000002},"max":{"1min":0.67600000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.59999999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.373,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71099999999999997,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.61199999999999999,"15min":0.624},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.754,"5min":0.879,"15min":1.4690000000000001},"last":0.52900000000000003}]},{"osd":4,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.5,"15min":0.48699999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69199999999999995,"5min":0.90200000000000002,"15min":0.94399999999999995},"last":0.501},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.52300000000000002,"15min":0.51200000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66100000000000003,"5min":0.92500000000000004,"15min":0.93300000000000005},"last":0.58899999999999997}]},{"osd":6,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51100000000000001,"15min":0.504},"min":{"1min":0.312,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.629,"5min":0.96799999999999997,"15min":1.732},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56399999999999995,"15min":0.54700000000000004},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78300000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.63700000000000001}]},{"osd":7,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59099999999999997,"15min":0.57499999999999996},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.79700000000000004,"5min":0.95399999999999996,"15min":1.0740000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.621,"15min":0.61299999999999999},"min":{"1min":0.41299999999999998,"5min":0.40799999999999997,"15min":0.39900000000000002},"max":{"1min":0.76500000000000001,"5min":0.996,"15min":1.6919999999999999},"last":0.57999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:07:46.271 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:07:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.437+0000 7fe2b50fd700 1 Processor -- start 2024-01-27T01:07:46.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.437+0000 7fe2b50fd700 1 -- start start 2024-01-27T01:07:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.438+0000 7fe2b50fd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe2b00cc1c0 0x7fe2b0164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.438+0000 7fe2b50fd700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b00cff70 con 0x7fe2b015ce40 2024-01-27T01:07:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.438+0000 7fe2b50fd700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b00cf400 con 0x7fe2b015d210 2024-01-27T01:07:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.438+0000 7fe2b50fd700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b00cf660 con 0x7fe2b00cc1c0 2024-01-27T01:07:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.438+0000 7fe2aed9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe2b015ce40 0x7fe2b00d0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42520/0 (socket says 172.21.15.46:42520) 2024-01-27T01:07:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.438+0000 7fe2aed9d700 1 -- 172.21.15.46:0/3303819702 learned_addr learned my addr 172.21.15.46:0/3303819702 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.438+0000 7fe2af59e700 1 --2- 172.21.15.46:0/3303819702 >> v2:172.21.15.106:3300/0 conn(0x7fe2b00cc1c0 0x7fe2b0164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.439+0000 7fe2add9b700 1 -- 172.21.15.46:0/3303819702 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 449629319 0 0) 0x7fe2b00cff70 con 0x7fe2b015ce40 2024-01-27T01:07:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.439+0000 7fe2add9b700 1 -- 172.21.15.46:0/3303819702 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe298003570 con 0x7fe2b015ce40 2024-01-27T01:07:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.439+0000 7fe2af59e700 1 -- 172.21.15.46:0/3303819702 >> v1:172.21.15.106:6789/0 conn(0x7fe2b015d210 legacy=0x7fe2b00cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.439+0000 7fe2af59e700 1 -- 172.21.15.46:0/3303819702 >> v1:172.21.15.46:6789/0 conn(0x7fe2b015ce40 legacy=0x7fe2b00d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.439+0000 7fe2af59e700 1 -- 172.21.15.46:0/3303819702 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b00d0500 con 0x7fe2b00cc1c0 2024-01-27T01:07:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.439+0000 7fe2af59e700 1 --2- 172.21.15.46:0/3303819702 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2b00cc1c0 0x7fe2b0164700 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8009e60 tx=0x7fe2a8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3009774abafbdb0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.439+0000 7fe2add9b700 1 -- 172.21.15.46:0/3303819702 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a800e070 con 0x7fe2b00cc1c0 2024-01-27T01:07:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.439+0000 7fe2add9b700 1 -- 172.21.15.46:0/3303819702 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a800bb60 con 0x7fe2b00cc1c0 2024-01-27T01:07:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.439+0000 7fe2add9b700 1 -- 172.21.15.46:0/3303819702 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a8012430 con 0x7fe2b00cc1c0 2024-01-27T01:07:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.440+0000 7fe2b50fd700 1 -- 172.21.15.46:0/3303819702 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2b00cc1c0 msgr2=0x7fe2b0164700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.440+0000 7fe2b50fd700 1 --2- 172.21.15.46:0/3303819702 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2b00cc1c0 0x7fe2b0164700 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8009e60 tx=0x7fe2a8009780 comp rx=0 tx=0).stop 2024-01-27T01:07:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.440+0000 7fe2b50fd700 1 -- 172.21.15.46:0/3303819702 shutdown_connections 2024-01-27T01:07:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.440+0000 7fe2b50fd700 1 --2- 172.21.15.46:0/3303819702 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2b00cc1c0 0x7fe2b0164700 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.440+0000 7fe2b50fd700 1 -- 172.21.15.46:0/3303819702 >> 172.21.15.46:0/3303819702 conn(0x7fe2b006f840 msgr2=0x7fe2b00d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.440+0000 7fe2b50fd700 1 -- 172.21.15.46:0/3303819702 shutdown_connections 2024-01-27T01:07:46.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.440+0000 7fe2b50fd700 1 -- 172.21.15.46:0/3303819702 wait complete. 2024-01-27T01:07:46.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.442+0000 7fe2b50fd700 1 Processor -- start 2024-01-27T01:07:46.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.442+0000 7fe2b50fd700 1 -- start start 2024-01-27T01:07:46.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.443+0000 7fe2b50fd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b00cc1c0 0x7fe2b01547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:46.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.443+0000 7fe2b50fd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2b015ce40 0x7fe2b015d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:46.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.443+0000 7fe2b50fd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2b015d210 0x7fe2b015e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:46.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.443+0000 7fe2b50fd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b01608c0 con 0x7fe2b00cc1c0 2024-01-27T01:07:46.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.443+0000 7fe2b50fd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b00c7b00 con 0x7fe2b015ce40 2024-01-27T01:07:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.443+0000 7fe2b50fd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b0156d00 con 0x7fe2b015d210 2024-01-27T01:07:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.443+0000 7fe2aed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b00cc1c0 0x7fe2b01547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.443+0000 7fe2aed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b00cc1c0 0x7fe2b01547b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37784/0 (socket says 172.21.15.46:37784) 2024-01-27T01:07:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.443+0000 7fe2aed9d700 1 -- 172.21.15.46:0/3893528530 learned_addr learned my addr 172.21.15.46:0/3893528530 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.443+0000 7fe2af59e700 1 --2- 172.21.15.46:0/3893528530 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2b015d210 0x7fe2b015e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.443+0000 7fe2ae59c700 1 --2- 172.21.15.46:0/3893528530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2b015ce40 0x7fe2b015d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.443+0000 7fe2aed9d700 1 -- 172.21.15.46:0/3893528530 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2b015d210 msgr2=0x7fe2b015e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.443+0000 7fe2aed9d700 1 --2- 172.21.15.46:0/3893528530 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2b015d210 0x7fe2b015e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.443+0000 7fe2aed9d700 1 -- 172.21.15.46:0/3893528530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2b015ce40 msgr2=0x7fe2b015d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.443+0000 7fe2aed9d700 1 --2- 172.21.15.46:0/3893528530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2b015ce40 0x7fe2b015d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.443+0000 7fe2aed9d700 1 -- 172.21.15.46:0/3893528530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a8009a20 con 0x7fe2b00cc1c0 2024-01-27T01:07:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.443+0000 7fe2aed9d700 1 --2- 172.21.15.46:0/3893528530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b00cc1c0 0x7fe2b01547b0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fe2a000c790 tx=0x7fe2a0007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.444+0000 7fe296ffd700 1 -- 172.21.15.46:0/3893528530 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0007300 con 0x7fe2b00cc1c0 2024-01-27T01:07:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.444+0000 7fe296ffd700 1 -- 172.21.15.46:0/3893528530 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a0007460 con 0x7fe2b00cc1c0 2024-01-27T01:07:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.444+0000 7fe2b50fd700 1 -- 172.21.15.46:0/3893528530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b0160060 con 0x7fe2b00cc1c0 2024-01-27T01:07:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.444+0000 7fe296ffd700 1 -- 172.21.15.46:0/3893528530 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a00057f0 con 0x7fe2b00cc1c0 2024-01-27T01:07:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.444+0000 7fe2b50fd700 1 -- 172.21.15.46:0/3893528530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b01603b0 con 0x7fe2b00cc1c0 2024-01-27T01:07:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.445+0000 7fe2b50fd700 1 -- 172.21.15.46:0/3893528530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b004c710 con 0x7fe2b00cc1c0 2024-01-27T01:07:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.445+0000 7fe296ffd700 1 -- 172.21.15.46:0/3893528530 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fe2a0005c20 con 0x7fe2b00cc1c0 2024-01-27T01:07:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.446+0000 7fe296ffd700 1 --2- 172.21.15.46:0/3893528530 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe29c067170 0x7fe29c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.446+0000 7fe296ffd700 1 -- 172.21.15.46:0/3893528530 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 7381+0+0 (secure 0 0 0) 0x7fe2a0084ad0 con 0x7fe2b00cc1c0 2024-01-27T01:07:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.446+0000 7fe2ae59c700 1 --2- 172.21.15.46:0/3893528530 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe29c067170 0x7fe29c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.446+0000 7fe2ae59c700 1 --2- 172.21.15.46:0/3893528530 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe29c067170 0x7fe29c069620 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0160a50 tx=0x7fe2a4009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:46.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.450+0000 7fe296ffd700 1 -- 172.21.15.46:0/3893528530 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe2a0051400 con 0x7fe2b00cc1c0 2024-01-27T01:07:46.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.648+0000 7fe2b50fd700 1 -- 172.21.15.46:0/3893528530 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe2b00cf8f0 con 0x7fe29c067170 2024-01-27T01:07:46.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.662+0000 7fe296ffd700 1 -- 172.21.15.46:0/3893528530 <== mgr.9590 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528865 (secure 0 0 0) 0x7fe2b00cf8f0 con 0x7fe29c067170 2024-01-27T01:07:46.669 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.667+0000 7fe2b50fd700 1 -- 172.21.15.46:0/3893528530 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe29c067170 msgr2=0x7fe29c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.667+0000 7fe2b50fd700 1 --2- 172.21.15.46:0/3893528530 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe29c067170 0x7fe29c069620 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0160a50 tx=0x7fe2a4009040 comp rx=0 tx=0).stop 2024-01-27T01:07:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.667+0000 7fe2b50fd700 1 -- 172.21.15.46:0/3893528530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b00cc1c0 msgr2=0x7fe2b01547b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.667+0000 7fe2b50fd700 1 --2- 172.21.15.46:0/3893528530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b00cc1c0 0x7fe2b01547b0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fe2a000c790 tx=0x7fe2a0007c80 comp rx=0 tx=0).stop 2024-01-27T01:07:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.667+0000 7fe2b50fd700 1 -- 172.21.15.46:0/3893528530 shutdown_connections 2024-01-27T01:07:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.667+0000 7fe2b50fd700 1 --2- 172.21.15.46:0/3893528530 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe29c067170 0x7fe29c069620 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.667+0000 7fe2b50fd700 1 --2- 172.21.15.46:0/3893528530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b00cc1c0 0x7fe2b01547b0 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.667+0000 7fe2b50fd700 1 --2- 172.21.15.46:0/3893528530 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe2b015d210 0x7fe2b015e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.667+0000 7fe2b50fd700 1 --2- 172.21.15.46:0/3893528530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2b015ce40 0x7fe2b015d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.667+0000 7fe2b50fd700 1 -- 172.21.15.46:0/3893528530 >> 172.21.15.46:0/3893528530 conn(0x7fe2b006f840 msgr2=0x7fe2b00b3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.667+0000 7fe2b50fd700 1 -- 172.21.15.46:0/3893528530 shutdown_connections 2024-01-27T01:07:46.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.667+0000 7fe2b50fd700 1 -- 172.21.15.46:0/3893528530 wait complete. 2024-01-27T01:07:46.675 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:07:46.681 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-01-27T01:07:45.873756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596212,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2456,"num_read_kb":5220,"num_write":762,"num_write_kb":3490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6623,"ondisk_log_size":6623,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":298,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2461208,"kb_used_data":25560,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752513512,"statfs":{"total":773094113280,"available":770573836288,"internally_reserved":0,"allocated":26173440,"data_stored":14394644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.913487"},"pg_stats":[{"pgid":"34.1d","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.008459+0000","last_change":"2024-01-27T01:07:42.002670+0000","last_active":"2024-01-27T01:07:42.008459+0000","last_peered":"2024-01-27T01:07:42.008459+0000","last_clean":"2024-01-27T01:07:42.008459+0000","last_became_active":"2024-01-27T01:07:42.002370+0000","last_became_peered":"2024-01-27T01:07:42.002370+0000","last_unstale":"2024-01-27T01:07:42.008459+0000","last_undegraded":"2024-01-27T01:07:42.008459+0000","last_fullsized":"2024-01-27T01:07:42.008459+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:04:31.798327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"246'1","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.003265+0000","last_change":"2024-01-27T01:07:40.984398+0000","last_active":"2024-01-27T01:07:42.003265+0000","last_peered":"2024-01-27T01:07:42.003265+0000","last_clean":"2024-01-27T01:07:42.003265+0000","last_became_active":"2024-01-27T01:07:40.984090+0000","last_became_peered":"2024-01-27T01:07:40.984090+0000","last_unstale":"2024-01-27T01:07:42.003265+0000","last_undegraded":"2024-01-27T01:07:42.003265+0000","last_fullsized":"2024-01-27T01:07:42.003265+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:59:52.185923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.993028+0000","last_change":"2024-01-27T01:07:27.744924+0000","last_active":"2024-01-27T01:07:41.993028+0000","last_peered":"2024-01-27T01:07:41.993028+0000","last_clean":"2024-01-27T01:07:41.993028+0000","last_became_active":"2024-01-27T01:07:27.744624+0000","last_became_peered":"2024-01-27T01:07:27.744624+0000","last_unstale":"2024-01-27T01:07:41.993028+0000","last_undegraded":"2024-01-27T01:07:41.993028+0000","last_fullsized":"2024-01-27T01:07:41.993028+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:44:00.209505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.991610+0000","last_change":"2024-01-27T01:07:41.990441+0000","last_active":"2024-01-27T01:07:41.991610+0000","last_peered":"2024-01-27T01:07:41.991610+0000","last_clean":"2024-01-27T01:07:41.991610+0000","last_became_active":"2024-01-27T01:07:41.990162+0000","last_became_peered":"2024-01-27T01:07:41.990162+0000","last_unstale":"2024-01-27T01:07:41.991610+0000","last_undegraded":"2024-01-27T01:07:41.991610+0000","last_fullsized":"2024-01-27T01:07:41.991610+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:23:25.015337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"246'4","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.002548+0000","last_change":"2024-01-27T01:07:40.990025+0000","last_active":"2024-01-27T01:07:42.002548+0000","last_peered":"2024-01-27T01:07:42.002548+0000","last_clean":"2024-01-27T01:07:42.002548+0000","last_became_active":"2024-01-27T01:07:40.989621+0000","last_became_peered":"2024-01-27T01:07:40.989621+0000","last_unstale":"2024-01-27T01:07:42.002548+0000","last_undegraded":"2024-01-27T01:07:42.002548+0000","last_fullsized":"2024-01-27T01:07:42.002548+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:58:53.757686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.983079+0000","last_change":"2024-01-27T01:07:27.749698+0000","last_active":"2024-01-27T01:07:41.983079+0000","last_peered":"2024-01-27T01:07:41.983079+0000","last_clean":"2024-01-27T01:07:41.983079+0000","last_became_active":"2024-01-27T01:07:27.749173+0000","last_became_peered":"2024-01-27T01:07:27.749173+0000","last_unstale":"2024-01-27T01:07:41.983079+0000","last_undegraded":"2024-01-27T01:07:41.983079+0000","last_fullsized":"2024-01-27T01:07:41.983079+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:52:40.618198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.011037+0000","last_change":"2024-01-27T01:07:42.006726+0000","last_active":"2024-01-27T01:07:42.011037+0000","last_peered":"2024-01-27T01:07:42.011037+0000","last_clean":"2024-01-27T01:07:42.011037+0000","last_became_active":"2024-01-27T01:07:42.006328+0000","last_became_peered":"2024-01-27T01:07:42.006328+0000","last_unstale":"2024-01-27T01:07:42.011037+0000","last_undegraded":"2024-01-27T01:07:42.011037+0000","last_fullsized":"2024-01-27T01:07:42.011037+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:36:25.025260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"246'2","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.008343+0000","last_change":"2024-01-27T01:07:40.979841+0000","last_active":"2024-01-27T01:07:42.008343+0000","last_peered":"2024-01-27T01:07:42.008343+0000","last_clean":"2024-01-27T01:07:42.008343+0000","last_became_active":"2024-01-27T01:07:40.979581+0000","last_became_peered":"2024-01-27T01:07:40.979581+0000","last_unstale":"2024-01-27T01:07:42.008343+0000","last_undegraded":"2024-01-27T01:07:42.008343+0000","last_fullsized":"2024-01-27T01:07:42.008343+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:21:34.990674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981807+0000","last_change":"2024-01-27T01:07:27.741674+0000","last_active":"2024-01-27T01:07:41.981807+0000","last_peered":"2024-01-27T01:07:41.981807+0000","last_clean":"2024-01-27T01:07:41.981807+0000","last_became_active":"2024-01-27T01:07:27.741351+0000","last_became_peered":"2024-01-27T01:07:27.741351+0000","last_unstale":"2024-01-27T01:07:41.981807+0000","last_undegraded":"2024-01-27T01:07:41.981807+0000","last_fullsized":"2024-01-27T01:07:41.981807+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:48:18.465014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.014563+0000","last_change":"2024-01-27T01:07:42.007107+0000","last_active":"2024-01-27T01:07:42.014563+0000","last_peered":"2024-01-27T01:07:42.014563+0000","last_clean":"2024-01-27T01:07:42.014563+0000","last_became_active":"2024-01-27T01:07:42.006691+0000","last_became_peered":"2024-01-27T01:07:42.006691+0000","last_unstale":"2024-01-27T01:07:42.014563+0000","last_undegraded":"2024-01-27T01:07:42.014563+0000","last_fullsized":"2024-01-27T01:07:42.014563+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:31:16.217384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"246'2","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.009194+0000","last_change":"2024-01-27T01:07:40.988051+0000","last_active":"2024-01-27T01:07:42.009194+0000","last_peered":"2024-01-27T01:07:42.009194+0000","last_clean":"2024-01-27T01:07:42.009194+0000","last_became_active":"2024-01-27T01:07:40.987772+0000","last_became_peered":"2024-01-27T01:07:40.987772+0000","last_unstale":"2024-01-27T01:07:42.009194+0000","last_undegraded":"2024-01-27T01:07:42.009194+0000","last_fullsized":"2024-01-27T01:07:42.009194+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:04:30.792846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979851+0000","last_change":"2024-01-27T01:07:27.749174+0000","last_active":"2024-01-27T01:07:41.979851+0000","last_peered":"2024-01-27T01:07:41.979851+0000","last_clean":"2024-01-27T01:07:41.979851+0000","last_became_active":"2024-01-27T01:07:27.748883+0000","last_became_peered":"2024-01-27T01:07:27.748883+0000","last_unstale":"2024-01-27T01:07:41.979851+0000","last_undegraded":"2024-01-27T01:07:41.979851+0000","last_fullsized":"2024-01-27T01:07:41.979851+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:03:47.280261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.012112+0000","last_change":"2024-01-27T01:07:42.007776+0000","last_active":"2024-01-27T01:07:42.012112+0000","last_peered":"2024-01-27T01:07:42.012112+0000","last_clean":"2024-01-27T01:07:42.012112+0000","last_became_active":"2024-01-27T01:07:42.007390+0000","last_became_peered":"2024-01-27T01:07:42.007390+0000","last_unstale":"2024-01-27T01:07:42.012112+0000","last_undegraded":"2024-01-27T01:07:42.012112+0000","last_fullsized":"2024-01-27T01:07:42.012112+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:04.421041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"246'1","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.019016+0000","last_change":"2024-01-27T01:07:40.983010+0000","last_active":"2024-01-27T01:07:42.019016+0000","last_peered":"2024-01-27T01:07:42.019016+0000","last_clean":"2024-01-27T01:07:42.019016+0000","last_became_active":"2024-01-27T01:07:40.982756+0000","last_became_peered":"2024-01-27T01:07:40.982756+0000","last_unstale":"2024-01-27T01:07:42.019016+0000","last_undegraded":"2024-01-27T01:07:42.019016+0000","last_fullsized":"2024-01-27T01:07:42.019016+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:58:14.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981671+0000","last_change":"2024-01-27T01:07:27.749946+0000","last_active":"2024-01-27T01:07:41.981671+0000","last_peered":"2024-01-27T01:07:41.981671+0000","last_clean":"2024-01-27T01:07:41.981671+0000","last_became_active":"2024-01-27T01:07:27.749653+0000","last_became_peered":"2024-01-27T01:07:27.749653+0000","last_unstale":"2024-01-27T01:07:41.981671+0000","last_undegraded":"2024-01-27T01:07:41.981671+0000","last_fullsized":"2024-01-27T01:07:41.981671+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:37.006113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.011035+0000","last_change":"2024-01-27T01:07:42.004489+0000","last_active":"2024-01-27T01:07:42.011035+0000","last_peered":"2024-01-27T01:07:42.011035+0000","last_clean":"2024-01-27T01:07:42.011035+0000","last_became_active":"2024-01-27T01:07:42.003992+0000","last_became_peered":"2024-01-27T01:07:42.003992+0000","last_unstale":"2024-01-27T01:07:42.011035+0000","last_undegraded":"2024-01-27T01:07:42.011035+0000","last_fullsized":"2024-01-27T01:07:42.011035+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:42:28.810173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"246'1","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.009359+0000","last_change":"2024-01-27T01:07:40.991405+0000","last_active":"2024-01-27T01:07:42.009359+0000","last_peered":"2024-01-27T01:07:42.009359+0000","last_clean":"2024-01-27T01:07:42.009359+0000","last_became_active":"2024-01-27T01:07:40.991112+0000","last_became_peered":"2024-01-27T01:07:40.991112+0000","last_unstale":"2024-01-27T01:07:42.009359+0000","last_undegraded":"2024-01-27T01:07:42.009359+0000","last_fullsized":"2024-01-27T01:07:42.009359+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:21:34.990674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.992517+0000","last_change":"2024-01-27T01:07:27.753636+0000","last_active":"2024-01-27T01:07:41.992517+0000","last_peered":"2024-01-27T01:07:41.992517+0000","last_clean":"2024-01-27T01:07:41.992517+0000","last_became_active":"2024-01-27T01:07:27.753305+0000","last_became_peered":"2024-01-27T01:07:27.753305+0000","last_unstale":"2024-01-27T01:07:41.992517+0000","last_undegraded":"2024-01-27T01:07:41.992517+0000","last_fullsized":"2024-01-27T01:07:41.992517+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:28:31.571123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.020172+0000","last_change":"2024-01-27T01:07:42.017435+0000","last_active":"2024-01-27T01:07:42.020172+0000","last_peered":"2024-01-27T01:07:42.020172+0000","last_clean":"2024-01-27T01:07:42.020172+0000","last_became_active":"2024-01-27T01:07:42.015956+0000","last_became_peered":"2024-01-27T01:07:42.015956+0000","last_unstale":"2024-01-27T01:07:42.020172+0000","last_undegraded":"2024-01-27T01:07:42.020172+0000","last_fullsized":"2024-01-27T01:07:42.020172+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:49:15.861867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981446+0000","last_change":"2024-01-27T01:07:40.987713+0000","last_active":"2024-01-27T01:07:41.981446+0000","last_peered":"2024-01-27T01:07:41.981446+0000","last_clean":"2024-01-27T01:07:41.981446+0000","last_became_active":"2024-01-27T01:07:40.987468+0000","last_became_peered":"2024-01-27T01:07:40.987468+0000","last_unstale":"2024-01-27T01:07:41.981446+0000","last_undegraded":"2024-01-27T01:07:41.981446+0000","last_fullsized":"2024-01-27T01:07:41.981446+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:34.939172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985248+0000","last_change":"2024-01-27T01:07:27.743608+0000","last_active":"2024-01-27T01:07:41.985248+0000","last_peered":"2024-01-27T01:07:41.985248+0000","last_clean":"2024-01-27T01:07:41.985248+0000","last_became_active":"2024-01-27T01:07:27.743324+0000","last_became_peered":"2024-01-27T01:07:27.743324+0000","last_unstale":"2024-01-27T01:07:41.985248+0000","last_undegraded":"2024-01-27T01:07:41.985248+0000","last_fullsized":"2024-01-27T01:07:41.985248+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:25:38.146534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.007671+0000","last_change":"2024-01-27T01:07:42.004032+0000","last_active":"2024-01-27T01:07:42.007671+0000","last_peered":"2024-01-27T01:07:42.007671+0000","last_clean":"2024-01-27T01:07:42.007671+0000","last_became_active":"2024-01-27T01:07:42.003776+0000","last_became_peered":"2024-01-27T01:07:42.003776+0000","last_unstale":"2024-01-27T01:07:42.007671+0000","last_undegraded":"2024-01-27T01:07:42.007671+0000","last_fullsized":"2024-01-27T01:07:42.007671+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:42.529880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.980992+0000","last_change":"2024-01-27T01:07:40.989457+0000","last_active":"2024-01-27T01:07:41.980992+0000","last_peered":"2024-01-27T01:07:41.980992+0000","last_clean":"2024-01-27T01:07:41.980992+0000","last_became_active":"2024-01-27T01:07:40.989191+0000","last_became_peered":"2024-01-27T01:07:40.989191+0000","last_unstale":"2024-01-27T01:07:41.980992+0000","last_undegraded":"2024-01-27T01:07:41.980992+0000","last_fullsized":"2024-01-27T01:07:41.980992+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:03:54.142292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981523+0000","last_change":"2024-01-27T01:07:27.744506+0000","last_active":"2024-01-27T01:07:41.981523+0000","last_peered":"2024-01-27T01:07:41.981523+0000","last_clean":"2024-01-27T01:07:41.981523+0000","last_became_active":"2024-01-27T01:07:27.744222+0000","last_became_peered":"2024-01-27T01:07:27.744222+0000","last_unstale":"2024-01-27T01:07:41.981523+0000","last_undegraded":"2024-01-27T01:07:41.981523+0000","last_fullsized":"2024-01-27T01:07:41.981523+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:24:38.456700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.001462+0000","last_change":"2024-01-27T01:07:41.994975+0000","last_active":"2024-01-27T01:07:42.001462+0000","last_peered":"2024-01-27T01:07:42.001462+0000","last_clean":"2024-01-27T01:07:42.001462+0000","last_became_active":"2024-01-27T01:07:41.994692+0000","last_became_peered":"2024-01-27T01:07:41.994692+0000","last_unstale":"2024-01-27T01:07:42.001462+0000","last_undegraded":"2024-01-27T01:07:42.001462+0000","last_fullsized":"2024-01-27T01:07:42.001462+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:40.019898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981333+0000","last_change":"2024-01-27T01:07:40.989481+0000","last_active":"2024-01-27T01:07:41.981333+0000","last_peered":"2024-01-27T01:07:41.981333+0000","last_clean":"2024-01-27T01:07:41.981333+0000","last_became_active":"2024-01-27T01:07:40.989211+0000","last_became_peered":"2024-01-27T01:07:40.989211+0000","last_unstale":"2024-01-27T01:07:41.981333+0000","last_undegraded":"2024-01-27T01:07:41.981333+0000","last_fullsized":"2024-01-27T01:07:41.981333+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:28:44.601683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.993180+0000","last_change":"2024-01-27T01:07:27.753284+0000","last_active":"2024-01-27T01:07:41.993180+0000","last_peered":"2024-01-27T01:07:41.993180+0000","last_clean":"2024-01-27T01:07:41.993180+0000","last_became_active":"2024-01-27T01:07:27.752819+0000","last_became_peered":"2024-01-27T01:07:27.752819+0000","last_unstale":"2024-01-27T01:07:41.993180+0000","last_undegraded":"2024-01-27T01:07:41.993180+0000","last_fullsized":"2024-01-27T01:07:41.993180+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:58:33.718993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.015645+0000","last_change":"2024-01-27T01:07:42.011852+0000","last_active":"2024-01-27T01:07:42.015645+0000","last_peered":"2024-01-27T01:07:42.015645+0000","last_clean":"2024-01-27T01:07:42.015645+0000","last_became_active":"2024-01-27T01:07:42.011522+0000","last_became_peered":"2024-01-27T01:07:42.011522+0000","last_unstale":"2024-01-27T01:07:42.015645+0000","last_undegraded":"2024-01-27T01:07:42.015645+0000","last_fullsized":"2024-01-27T01:07:42.015645+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:49:05.586187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.992730+0000","last_change":"2024-01-27T01:07:40.984823+0000","last_active":"2024-01-27T01:07:41.992730+0000","last_peered":"2024-01-27T01:07:41.992730+0000","last_clean":"2024-01-27T01:07:41.992730+0000","last_became_active":"2024-01-27T01:07:40.984369+0000","last_became_peered":"2024-01-27T01:07:40.984369+0000","last_unstale":"2024-01-27T01:07:41.992730+0000","last_undegraded":"2024-01-27T01:07:41.992730+0000","last_fullsized":"2024-01-27T01:07:41.992730+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:59:25.911803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979458+0000","last_change":"2024-01-27T01:07:27.743902+0000","last_active":"2024-01-27T01:07:41.979458+0000","last_peered":"2024-01-27T01:07:41.979458+0000","last_clean":"2024-01-27T01:07:41.979458+0000","last_became_active":"2024-01-27T01:07:27.743270+0000","last_became_peered":"2024-01-27T01:07:27.743270+0000","last_unstale":"2024-01-27T01:07:41.979458+0000","last_undegraded":"2024-01-27T01:07:41.979458+0000","last_fullsized":"2024-01-27T01:07:41.979458+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:24:38.456700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.013239+0000","last_change":"2024-01-27T01:07:42.008312+0000","last_active":"2024-01-27T01:07:42.013239+0000","last_peered":"2024-01-27T01:07:42.013239+0000","last_clean":"2024-01-27T01:07:42.013239+0000","last_became_active":"2024-01-27T01:07:42.008091+0000","last_became_peered":"2024-01-27T01:07:42.008091+0000","last_unstale":"2024-01-27T01:07:42.013239+0000","last_undegraded":"2024-01-27T01:07:42.013239+0000","last_fullsized":"2024-01-27T01:07:42.013239+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:42.529880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"246'6","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.016378+0000","last_change":"2024-01-27T01:07:40.986467+0000","last_active":"2024-01-27T01:07:42.016378+0000","last_peered":"2024-01-27T01:07:42.016378+0000","last_clean":"2024-01-27T01:07:42.016378+0000","last_became_active":"2024-01-27T01:07:40.986128+0000","last_became_peered":"2024-01-27T01:07:40.986128+0000","last_unstale":"2024-01-27T01:07:42.016378+0000","last_undegraded":"2024-01-27T01:07:42.016378+0000","last_fullsized":"2024-01-27T01:07:42.016378+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:19:39.079815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981992+0000","last_change":"2024-01-27T01:07:27.743746+0000","last_active":"2024-01-27T01:07:41.981992+0000","last_peered":"2024-01-27T01:07:41.981992+0000","last_clean":"2024-01-27T01:07:41.981992+0000","last_became_active":"2024-01-27T01:07:27.743414+0000","last_became_peered":"2024-01-27T01:07:27.743414+0000","last_unstale":"2024-01-27T01:07:41.981992+0000","last_undegraded":"2024-01-27T01:07:41.981992+0000","last_fullsized":"2024-01-27T01:07:41.981992+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:37.006113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.000392+0000","last_change":"2024-01-27T01:07:41.992795+0000","last_active":"2024-01-27T01:07:42.000392+0000","last_peered":"2024-01-27T01:07:42.000392+0000","last_clean":"2024-01-27T01:07:42.000392+0000","last_became_active":"2024-01-27T01:07:41.992541+0000","last_became_peered":"2024-01-27T01:07:41.992541+0000","last_unstale":"2024-01-27T01:07:42.000392+0000","last_undegraded":"2024-01-27T01:07:42.000392+0000","last_fullsized":"2024-01-27T01:07:42.000392+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:18:33.128110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"246'3","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.017535+0000","last_change":"2024-01-27T01:07:40.985223+0000","last_active":"2024-01-27T01:07:42.017535+0000","last_peered":"2024-01-27T01:07:42.017535+0000","last_clean":"2024-01-27T01:07:42.017535+0000","last_became_active":"2024-01-27T01:07:40.984909+0000","last_became_peered":"2024-01-27T01:07:40.984909+0000","last_unstale":"2024-01-27T01:07:42.017535+0000","last_undegraded":"2024-01-27T01:07:42.017535+0000","last_fullsized":"2024-01-27T01:07:42.017535+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:58:14.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.994743+0000","last_change":"2024-01-27T01:07:27.748096+0000","last_active":"2024-01-27T01:07:41.994743+0000","last_peered":"2024-01-27T01:07:41.994743+0000","last_clean":"2024-01-27T01:07:41.994743+0000","last_became_active":"2024-01-27T01:07:27.747801+0000","last_became_peered":"2024-01-27T01:07:27.747801+0000","last_unstale":"2024-01-27T01:07:41.994743+0000","last_undegraded":"2024-01-27T01:07:41.994743+0000","last_fullsized":"2024-01-27T01:07:41.994743+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:20:52.644616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.013347+0000","last_change":"2024-01-27T01:07:42.009566+0000","last_active":"2024-01-27T01:07:42.013347+0000","last_peered":"2024-01-27T01:07:42.013347+0000","last_clean":"2024-01-27T01:07:42.013347+0000","last_became_active":"2024-01-27T01:07:42.008961+0000","last_became_peered":"2024-01-27T01:07:42.008961+0000","last_unstale":"2024-01-27T01:07:42.013347+0000","last_undegraded":"2024-01-27T01:07:42.013347+0000","last_fullsized":"2024-01-27T01:07:42.013347+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:00.162997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.992233+0000","last_change":"2024-01-27T01:07:40.980925+0000","last_active":"2024-01-27T01:07:41.992233+0000","last_peered":"2024-01-27T01:07:41.992233+0000","last_clean":"2024-01-27T01:07:41.992233+0000","last_became_active":"2024-01-27T01:07:40.980634+0000","last_became_peered":"2024-01-27T01:07:40.980634+0000","last_unstale":"2024-01-27T01:07:41.992233+0000","last_undegraded":"2024-01-27T01:07:41.992233+0000","last_fullsized":"2024-01-27T01:07:41.992233+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:17.897003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995189+0000","last_change":"2024-01-27T01:07:27.746806+0000","last_active":"2024-01-27T01:07:41.995189+0000","last_peered":"2024-01-27T01:07:41.995189+0000","last_clean":"2024-01-27T01:07:41.995189+0000","last_became_active":"2024-01-27T01:07:27.745168+0000","last_became_peered":"2024-01-27T01:07:27.745168+0000","last_unstale":"2024-01-27T01:07:41.995189+0000","last_undegraded":"2024-01-27T01:07:41.995189+0000","last_fullsized":"2024-01-27T01:07:41.995189+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:34:21.004352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.004700+0000","last_change":"2024-01-27T01:07:41.998067+0000","last_active":"2024-01-27T01:07:42.004700+0000","last_peered":"2024-01-27T01:07:42.004700+0000","last_clean":"2024-01-27T01:07:42.004700+0000","last_became_active":"2024-01-27T01:07:41.997808+0000","last_became_peered":"2024-01-27T01:07:41.997808+0000","last_unstale":"2024-01-27T01:07:42.004700+0000","last_undegraded":"2024-01-27T01:07:42.004700+0000","last_fullsized":"2024-01-27T01:07:42.004700+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:10:20.100421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"246'2","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.009489+0000","last_change":"2024-01-27T01:07:40.986956+0000","last_active":"2024-01-27T01:07:42.009489+0000","last_peered":"2024-01-27T01:07:42.009489+0000","last_clean":"2024-01-27T01:07:42.009489+0000","last_became_active":"2024-01-27T01:07:40.986683+0000","last_became_peered":"2024-01-27T01:07:40.986683+0000","last_unstale":"2024-01-27T01:07:42.009489+0000","last_undegraded":"2024-01-27T01:07:42.009489+0000","last_fullsized":"2024-01-27T01:07:42.009489+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:03:54.142292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995101+0000","last_change":"2024-01-27T01:07:27.749049+0000","last_active":"2024-01-27T01:07:41.995101+0000","last_peered":"2024-01-27T01:07:41.995101+0000","last_clean":"2024-01-27T01:07:41.995101+0000","last_became_active":"2024-01-27T01:07:27.748736+0000","last_became_peered":"2024-01-27T01:07:27.748736+0000","last_unstale":"2024-01-27T01:07:41.995101+0000","last_undegraded":"2024-01-27T01:07:41.995101+0000","last_fullsized":"2024-01-27T01:07:41.995101+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:23:10.979296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.009544+0000","last_change":"2024-01-27T01:07:42.006099+0000","last_active":"2024-01-27T01:07:42.009544+0000","last_peered":"2024-01-27T01:07:42.009544+0000","last_clean":"2024-01-27T01:07:42.009544+0000","last_became_active":"2024-01-27T01:07:42.005798+0000","last_became_peered":"2024-01-27T01:07:42.005798+0000","last_unstale":"2024-01-27T01:07:42.009544+0000","last_undegraded":"2024-01-27T01:07:42.009544+0000","last_fullsized":"2024-01-27T01:07:42.009544+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:41:14.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"246'8","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.014698+0000","last_change":"2024-01-27T01:07:40.988442+0000","last_active":"2024-01-27T01:07:42.014698+0000","last_peered":"2024-01-27T01:07:42.014698+0000","last_clean":"2024-01-27T01:07:42.014698+0000","last_became_active":"2024-01-27T01:07:40.988119+0000","last_became_peered":"2024-01-27T01:07:40.988119+0000","last_unstale":"2024-01-27T01:07:42.014698+0000","last_undegraded":"2024-01-27T01:07:42.014698+0000","last_fullsized":"2024-01-27T01:07:42.014698+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:54:53.395827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984376+0000","last_change":"2024-01-27T01:07:27.755236+0000","last_active":"2024-01-27T01:07:41.984376+0000","last_peered":"2024-01-27T01:07:41.984376+0000","last_clean":"2024-01-27T01:07:41.984376+0000","last_became_active":"2024-01-27T01:07:27.754945+0000","last_became_peered":"2024-01-27T01:07:27.754945+0000","last_unstale":"2024-01-27T01:07:41.984376+0000","last_undegraded":"2024-01-27T01:07:41.984376+0000","last_fullsized":"2024-01-27T01:07:41.984376+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:07:48.619221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.008758+0000","last_change":"2024-01-27T01:07:42.001263+0000","last_active":"2024-01-27T01:07:42.008758+0000","last_peered":"2024-01-27T01:07:42.008758+0000","last_clean":"2024-01-27T01:07:42.008758+0000","last_became_active":"2024-01-27T01:07:42.000947+0000","last_became_peered":"2024-01-27T01:07:42.000947+0000","last_unstale":"2024-01-27T01:07:42.008758+0000","last_undegraded":"2024-01-27T01:07:42.008758+0000","last_fullsized":"2024-01-27T01:07:42.008758+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:17.562068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.987205+0000","last_change":"2024-01-27T01:07:40.976555+0000","last_active":"2024-01-27T01:07:41.987205+0000","last_peered":"2024-01-27T01:07:41.987205+0000","last_clean":"2024-01-27T01:07:41.987205+0000","last_became_active":"2024-01-27T01:07:40.976259+0000","last_became_peered":"2024-01-27T01:07:40.976259+0000","last_unstale":"2024-01-27T01:07:41.987205+0000","last_undegraded":"2024-01-27T01:07:41.987205+0000","last_fullsized":"2024-01-27T01:07:41.987205+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:08:19.116626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981641+0000","last_change":"2024-01-27T01:07:27.745495+0000","last_active":"2024-01-27T01:07:41.981641+0000","last_peered":"2024-01-27T01:07:41.981641+0000","last_clean":"2024-01-27T01:07:41.981641+0000","last_became_active":"2024-01-27T01:07:27.744986+0000","last_became_peered":"2024-01-27T01:07:27.744986+0000","last_unstale":"2024-01-27T01:07:41.981641+0000","last_undegraded":"2024-01-27T01:07:41.981641+0000","last_fullsized":"2024-01-27T01:07:41.981641+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:08.374701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.006025+0000","last_change":"2024-01-27T01:07:42.003063+0000","last_active":"2024-01-27T01:07:42.006025+0000","last_peered":"2024-01-27T01:07:42.006025+0000","last_clean":"2024-01-27T01:07:42.006025+0000","last_became_active":"2024-01-27T01:07:42.002733+0000","last_became_peered":"2024-01-27T01:07:42.002733+0000","last_unstale":"2024-01-27T01:07:42.006025+0000","last_undegraded":"2024-01-27T01:07:42.006025+0000","last_fullsized":"2024-01-27T01:07:42.006025+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:11:22.410741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"246'2","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.002101+0000","last_change":"2024-01-27T01:07:40.990836+0000","last_active":"2024-01-27T01:07:42.002101+0000","last_peered":"2024-01-27T01:07:42.002101+0000","last_clean":"2024-01-27T01:07:42.002101+0000","last_became_active":"2024-01-27T01:07:40.990079+0000","last_became_peered":"2024-01-27T01:07:40.990079+0000","last_unstale":"2024-01-27T01:07:42.002101+0000","last_undegraded":"2024-01-27T01:07:42.002101+0000","last_fullsized":"2024-01-27T01:07:42.002101+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:52:32.284100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.998215+0000","last_change":"2024-01-27T01:07:27.749357+0000","last_active":"2024-01-27T01:07:41.998215+0000","last_peered":"2024-01-27T01:07:41.998215+0000","last_clean":"2024-01-27T01:07:41.998215+0000","last_became_active":"2024-01-27T01:07:27.745281+0000","last_became_peered":"2024-01-27T01:07:27.745281+0000","last_unstale":"2024-01-27T01:07:41.998215+0000","last_undegraded":"2024-01-27T01:07:41.998215+0000","last_fullsized":"2024-01-27T01:07:41.998215+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:11.619364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.005896+0000","last_change":"2024-01-27T01:07:42.002607+0000","last_active":"2024-01-27T01:07:42.005896+0000","last_peered":"2024-01-27T01:07:42.005896+0000","last_clean":"2024-01-27T01:07:42.005896+0000","last_became_active":"2024-01-27T01:07:42.002101+0000","last_became_peered":"2024-01-27T01:07:42.002101+0000","last_unstale":"2024-01-27T01:07:42.005896+0000","last_undegraded":"2024-01-27T01:07:42.005896+0000","last_fullsized":"2024-01-27T01:07:42.005896+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:19.572287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"246'1","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.005523+0000","last_change":"2024-01-27T01:07:40.989369+0000","last_active":"2024-01-27T01:07:42.005523+0000","last_peered":"2024-01-27T01:07:42.005523+0000","last_clean":"2024-01-27T01:07:42.005523+0000","last_became_active":"2024-01-27T01:07:40.989079+0000","last_became_peered":"2024-01-27T01:07:40.989079+0000","last_unstale":"2024-01-27T01:07:42.005523+0000","last_undegraded":"2024-01-27T01:07:42.005523+0000","last_fullsized":"2024-01-27T01:07:42.005523+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:45.809251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979563+0000","last_change":"2024-01-27T01:07:27.736143+0000","last_active":"2024-01-27T01:07:41.979563+0000","last_peered":"2024-01-27T01:07:41.979563+0000","last_clean":"2024-01-27T01:07:41.979563+0000","last_became_active":"2024-01-27T01:07:27.735824+0000","last_became_peered":"2024-01-27T01:07:27.735824+0000","last_unstale":"2024-01-27T01:07:41.979563+0000","last_undegraded":"2024-01-27T01:07:41.979563+0000","last_fullsized":"2024-01-27T01:07:41.979563+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:05.536246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.013345+0000","last_change":"2024-01-27T01:07:42.009502+0000","last_active":"2024-01-27T01:07:42.013345+0000","last_peered":"2024-01-27T01:07:42.013345+0000","last_clean":"2024-01-27T01:07:42.013345+0000","last_became_active":"2024-01-27T01:07:42.009217+0000","last_became_peered":"2024-01-27T01:07:42.009217+0000","last_unstale":"2024-01-27T01:07:42.013345+0000","last_undegraded":"2024-01-27T01:07:42.013345+0000","last_fullsized":"2024-01-27T01:07:42.013345+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:33.375597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982593+0000","last_change":"2024-01-27T01:07:40.990708+0000","last_active":"2024-01-27T01:07:41.982593+0000","last_peered":"2024-01-27T01:07:41.982593+0000","last_clean":"2024-01-27T01:07:41.982593+0000","last_became_active":"2024-01-27T01:07:40.990411+0000","last_became_peered":"2024-01-27T01:07:40.990411+0000","last_unstale":"2024-01-27T01:07:41.982593+0000","last_undegraded":"2024-01-27T01:07:41.982593+0000","last_fullsized":"2024-01-27T01:07:41.982593+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:41:37.115942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985631+0000","last_change":"2024-01-27T01:07:27.756376+0000","last_active":"2024-01-27T01:07:41.985631+0000","last_peered":"2024-01-27T01:07:41.985631+0000","last_clean":"2024-01-27T01:07:41.985631+0000","last_became_active":"2024-01-27T01:07:27.756152+0000","last_became_peered":"2024-01-27T01:07:27.756152+0000","last_unstale":"2024-01-27T01:07:41.985631+0000","last_undegraded":"2024-01-27T01:07:41.985631+0000","last_fullsized":"2024-01-27T01:07:41.985631+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:02:04.959467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.014625+0000","last_change":"2024-01-27T01:07:42.011536+0000","last_active":"2024-01-27T01:07:42.014625+0000","last_peered":"2024-01-27T01:07:42.014625+0000","last_clean":"2024-01-27T01:07:42.014625+0000","last_became_active":"2024-01-27T01:07:42.006921+0000","last_became_peered":"2024-01-27T01:07:42.006921+0000","last_unstale":"2024-01-27T01:07:42.014625+0000","last_undegraded":"2024-01-27T01:07:42.014625+0000","last_fullsized":"2024-01-27T01:07:42.014625+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:36:25.025260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"246'1","reported_seq":11,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.003675+0000","last_change":"2024-01-27T01:07:40.984795+0000","last_active":"2024-01-27T01:07:42.003675+0000","last_peered":"2024-01-27T01:07:42.003675+0000","last_clean":"2024-01-27T01:07:42.003675+0000","last_became_active":"2024-01-27T01:07:40.984185+0000","last_became_peered":"2024-01-27T01:07:40.984185+0000","last_unstale":"2024-01-27T01:07:42.003675+0000","last_undegraded":"2024-01-27T01:07:42.003675+0000","last_fullsized":"2024-01-27T01:07:42.003675+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:43:50.097560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982873+0000","last_change":"2024-01-27T01:07:27.758252+0000","last_active":"2024-01-27T01:07:41.982873+0000","last_peered":"2024-01-27T01:07:41.982873+0000","last_clean":"2024-01-27T01:07:41.982873+0000","last_became_active":"2024-01-27T01:07:27.757929+0000","last_became_peered":"2024-01-27T01:07:27.757929+0000","last_unstale":"2024-01-27T01:07:41.982873+0000","last_undegraded":"2024-01-27T01:07:41.982873+0000","last_fullsized":"2024-01-27T01:07:41.982873+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:30.518041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.010869+0000","last_change":"2024-01-27T01:07:42.006225+0000","last_active":"2024-01-27T01:07:42.010869+0000","last_peered":"2024-01-27T01:07:42.010869+0000","last_clean":"2024-01-27T01:07:42.010869+0000","last_became_active":"2024-01-27T01:07:42.005928+0000","last_became_peered":"2024-01-27T01:07:42.005928+0000","last_unstale":"2024-01-27T01:07:42.010869+0000","last_undegraded":"2024-01-27T01:07:42.010869+0000","last_fullsized":"2024-01-27T01:07:42.010869+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:38.943517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.978974+0000","last_change":"2024-01-27T01:07:40.992511+0000","last_active":"2024-01-27T01:07:41.978974+0000","last_peered":"2024-01-27T01:07:41.978974+0000","last_clean":"2024-01-27T01:07:41.978974+0000","last_became_active":"2024-01-27T01:07:40.992205+0000","last_became_peered":"2024-01-27T01:07:40.992205+0000","last_unstale":"2024-01-27T01:07:41.978974+0000","last_undegraded":"2024-01-27T01:07:41.978974+0000","last_fullsized":"2024-01-27T01:07:41.978974+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:13.573493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985737+0000","last_change":"2024-01-27T01:07:27.752626+0000","last_active":"2024-01-27T01:07:41.985737+0000","last_peered":"2024-01-27T01:07:41.985737+0000","last_clean":"2024-01-27T01:07:41.985737+0000","last_became_active":"2024-01-27T01:07:27.752363+0000","last_became_peered":"2024-01-27T01:07:27.752363+0000","last_unstale":"2024-01-27T01:07:41.985737+0000","last_undegraded":"2024-01-27T01:07:41.985737+0000","last_fullsized":"2024-01-27T01:07:41.985737+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:03:44.824769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.011005+0000","last_change":"2024-01-27T01:07:42.004276+0000","last_active":"2024-01-27T01:07:42.011005+0000","last_peered":"2024-01-27T01:07:42.011005+0000","last_clean":"2024-01-27T01:07:42.011005+0000","last_became_active":"2024-01-27T01:07:42.003559+0000","last_became_peered":"2024-01-27T01:07:42.003559+0000","last_unstale":"2024-01-27T01:07:42.011005+0000","last_undegraded":"2024-01-27T01:07:42.011005+0000","last_fullsized":"2024-01-27T01:07:42.011005+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:22:42.410257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"246'2","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.018015+0000","last_change":"2024-01-27T01:07:40.992336+0000","last_active":"2024-01-27T01:07:42.018015+0000","last_peered":"2024-01-27T01:07:42.018015+0000","last_clean":"2024-01-27T01:07:42.018015+0000","last_became_active":"2024-01-27T01:07:40.991815+0000","last_became_peered":"2024-01-27T01:07:40.991815+0000","last_unstale":"2024-01-27T01:07:42.018015+0000","last_undegraded":"2024-01-27T01:07:42.018015+0000","last_fullsized":"2024-01-27T01:07:42.018015+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:59:52.185923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995278+0000","last_change":"2024-01-27T01:07:27.748467+0000","last_active":"2024-01-27T01:07:41.995278+0000","last_peered":"2024-01-27T01:07:41.995278+0000","last_clean":"2024-01-27T01:07:41.995278+0000","last_became_active":"2024-01-27T01:07:27.748100+0000","last_became_peered":"2024-01-27T01:07:27.748100+0000","last_unstale":"2024-01-27T01:07:41.995278+0000","last_undegraded":"2024-01-27T01:07:41.995278+0000","last_fullsized":"2024-01-27T01:07:41.995278+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:38:29.701960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.012756+0000","last_change":"2024-01-27T01:07:42.004497+0000","last_active":"2024-01-27T01:07:42.012756+0000","last_peered":"2024-01-27T01:07:42.012756+0000","last_clean":"2024-01-27T01:07:42.012756+0000","last_became_active":"2024-01-27T01:07:42.003613+0000","last_became_peered":"2024-01-27T01:07:42.003613+0000","last_unstale":"2024-01-27T01:07:42.012756+0000","last_undegraded":"2024-01-27T01:07:42.012756+0000","last_fullsized":"2024-01-27T01:07:42.012756+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:41.431166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.988574+0000","last_change":"2024-01-27T01:07:40.978139+0000","last_active":"2024-01-27T01:07:41.988574+0000","last_peered":"2024-01-27T01:07:41.988574+0000","last_clean":"2024-01-27T01:07:41.988574+0000","last_became_active":"2024-01-27T01:07:40.977778+0000","last_became_peered":"2024-01-27T01:07:40.977778+0000","last_unstale":"2024-01-27T01:07:41.988574+0000","last_undegraded":"2024-01-27T01:07:41.988574+0000","last_fullsized":"2024-01-27T01:07:41.988574+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:30:38.944123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984213+0000","last_change":"2024-01-27T01:07:27.755439+0000","last_active":"2024-01-27T01:07:41.984213+0000","last_peered":"2024-01-27T01:07:41.984213+0000","last_clean":"2024-01-27T01:07:41.984213+0000","last_became_active":"2024-01-27T01:07:27.755126+0000","last_became_peered":"2024-01-27T01:07:27.755126+0000","last_unstale":"2024-01-27T01:07:41.984213+0000","last_undegraded":"2024-01-27T01:07:41.984213+0000","last_fullsized":"2024-01-27T01:07:41.984213+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:43:37.067000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.017713+0000","last_change":"2024-01-27T01:07:42.014667+0000","last_active":"2024-01-27T01:07:42.017713+0000","last_peered":"2024-01-27T01:07:42.017713+0000","last_clean":"2024-01-27T01:07:42.017713+0000","last_became_active":"2024-01-27T01:07:42.011648+0000","last_became_peered":"2024-01-27T01:07:42.011648+0000","last_unstale":"2024-01-27T01:07:42.017713+0000","last_undegraded":"2024-01-27T01:07:42.017713+0000","last_fullsized":"2024-01-27T01:07:42.017713+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:22:42.410257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.997076+0000","last_change":"2024-01-27T01:07:40.988279+0000","last_active":"2024-01-27T01:07:41.997076+0000","last_peered":"2024-01-27T01:07:41.997076+0000","last_clean":"2024-01-27T01:07:41.997076+0000","last_became_active":"2024-01-27T01:07:40.987982+0000","last_became_peered":"2024-01-27T01:07:40.987982+0000","last_unstale":"2024-01-27T01:07:41.997076+0000","last_undegraded":"2024-01-27T01:07:41.997076+0000","last_fullsized":"2024-01-27T01:07:41.997076+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:08:19.116626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995851+0000","last_change":"2024-01-27T01:07:27.745155+0000","last_active":"2024-01-27T01:07:41.995851+0000","last_peered":"2024-01-27T01:07:41.995851+0000","last_clean":"2024-01-27T01:07:41.995851+0000","last_became_active":"2024-01-27T01:07:27.744562+0000","last_became_peered":"2024-01-27T01:07:27.744562+0000","last_unstale":"2024-01-27T01:07:41.995851+0000","last_undegraded":"2024-01-27T01:07:41.995851+0000","last_fullsized":"2024-01-27T01:07:41.995851+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:28:42.362768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.002199+0000","last_change":"2024-01-27T01:07:41.999378+0000","last_active":"2024-01-27T01:07:42.002199+0000","last_peered":"2024-01-27T01:07:42.002199+0000","last_clean":"2024-01-27T01:07:42.002199+0000","last_became_active":"2024-01-27T01:07:41.999026+0000","last_became_peered":"2024-01-27T01:07:41.999026+0000","last_unstale":"2024-01-27T01:07:42.002199+0000","last_undegraded":"2024-01-27T01:07:42.002199+0000","last_fullsized":"2024-01-27T01:07:42.002199+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:44:21.352342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"247'1","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-01-27T01:07:43.033348+0000","last_change":"2024-01-27T01:07:40.991024+0000","last_active":"2024-01-27T01:07:43.033348+0000","last_peered":"2024-01-27T01:07:43.033348+0000","last_clean":"2024-01-27T01:07:43.033348+0000","last_became_active":"2024-01-27T01:07:40.990526+0000","last_became_peered":"2024-01-27T01:07:40.990526+0000","last_unstale":"2024-01-27T01:07:43.033348+0000","last_undegraded":"2024-01-27T01:07:43.033348+0000","last_fullsized":"2024-01-27T01:07:43.033348+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:34:29.680208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.983165+0000","last_change":"2024-01-27T01:07:27.745357+0000","last_active":"2024-01-27T01:07:41.983165+0000","last_peered":"2024-01-27T01:07:41.983165+0000","last_clean":"2024-01-27T01:07:41.983165+0000","last_became_active":"2024-01-27T01:07:27.745075+0000","last_became_peered":"2024-01-27T01:07:27.745075+0000","last_unstale":"2024-01-27T01:07:41.983165+0000","last_undegraded":"2024-01-27T01:07:41.983165+0000","last_fullsized":"2024-01-27T01:07:41.983165+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:37:01.620442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.000853+0000","last_change":"2024-01-27T01:07:41.993171+0000","last_active":"2024-01-27T01:07:42.000853+0000","last_peered":"2024-01-27T01:07:42.000853+0000","last_clean":"2024-01-27T01:07:42.000853+0000","last_became_active":"2024-01-27T01:07:41.991964+0000","last_became_peered":"2024-01-27T01:07:41.991964+0000","last_unstale":"2024-01-27T01:07:42.000853+0000","last_undegraded":"2024-01-27T01:07:42.000853+0000","last_fullsized":"2024-01-27T01:07:42.000853+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:22:14.578974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"246'2","reported_seq":12,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.011545+0000","last_change":"2024-01-27T01:07:40.985107+0000","last_active":"2024-01-27T01:07:42.011545+0000","last_peered":"2024-01-27T01:07:42.011545+0000","last_clean":"2024-01-27T01:07:42.011545+0000","last_became_active":"2024-01-27T01:07:40.984742+0000","last_became_peered":"2024-01-27T01:07:40.984742+0000","last_unstale":"2024-01-27T01:07:42.011545+0000","last_undegraded":"2024-01-27T01:07:42.011545+0000","last_fullsized":"2024-01-27T01:07:42.011545+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:07:12.229792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985979+0000","last_change":"2024-01-27T01:07:27.744187+0000","last_active":"2024-01-27T01:07:41.985979+0000","last_peered":"2024-01-27T01:07:41.985979+0000","last_clean":"2024-01-27T01:07:41.985979+0000","last_became_active":"2024-01-27T01:07:27.743889+0000","last_became_peered":"2024-01-27T01:07:27.743889+0000","last_unstale":"2024-01-27T01:07:41.985979+0000","last_undegraded":"2024-01-27T01:07:41.985979+0000","last_fullsized":"2024-01-27T01:07:41.985979+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:11.619364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.002866+0000","last_change":"2024-01-27T01:07:42.001717+0000","last_active":"2024-01-27T01:07:42.002866+0000","last_peered":"2024-01-27T01:07:42.002866+0000","last_clean":"2024-01-27T01:07:42.002866+0000","last_became_active":"2024-01-27T01:07:42.001443+0000","last_became_peered":"2024-01-27T01:07:42.001443+0000","last_unstale":"2024-01-27T01:07:42.002866+0000","last_undegraded":"2024-01-27T01:07:42.002866+0000","last_fullsized":"2024-01-27T01:07:42.002866+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:03:35.144430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985723+0000","last_change":"2024-01-27T01:07:40.979755+0000","last_active":"2024-01-27T01:07:41.985723+0000","last_peered":"2024-01-27T01:07:41.985723+0000","last_clean":"2024-01-27T01:07:41.985723+0000","last_became_active":"2024-01-27T01:07:40.979490+0000","last_became_peered":"2024-01-27T01:07:40.979490+0000","last_unstale":"2024-01-27T01:07:41.985723+0000","last_undegraded":"2024-01-27T01:07:41.985723+0000","last_fullsized":"2024-01-27T01:07:41.985723+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:24.587487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981626+0000","last_change":"2024-01-27T01:07:27.743304+0000","last_active":"2024-01-27T01:07:41.981626+0000","last_peered":"2024-01-27T01:07:41.981626+0000","last_clean":"2024-01-27T01:07:41.981626+0000","last_became_active":"2024-01-27T01:07:27.742783+0000","last_became_peered":"2024-01-27T01:07:27.742783+0000","last_unstale":"2024-01-27T01:07:41.981626+0000","last_undegraded":"2024-01-27T01:07:41.981626+0000","last_fullsized":"2024-01-27T01:07:41.981626+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:45:07.998194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.007642+0000","last_change":"2024-01-27T01:07:42.002777+0000","last_active":"2024-01-27T01:07:42.007642+0000","last_peered":"2024-01-27T01:07:42.007642+0000","last_clean":"2024-01-27T01:07:42.007642+0000","last_became_active":"2024-01-27T01:07:42.002493+0000","last_became_peered":"2024-01-27T01:07:42.002493+0000","last_unstale":"2024-01-27T01:07:42.007642+0000","last_undegraded":"2024-01-27T01:07:42.007642+0000","last_fullsized":"2024-01-27T01:07:42.007642+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:17.054007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.986261+0000","last_change":"2024-01-27T01:07:40.984956+0000","last_active":"2024-01-27T01:07:41.986261+0000","last_peered":"2024-01-27T01:07:41.986261+0000","last_clean":"2024-01-27T01:07:41.986261+0000","last_became_active":"2024-01-27T01:07:40.984539+0000","last_became_peered":"2024-01-27T01:07:40.984539+0000","last_unstale":"2024-01-27T01:07:41.986261+0000","last_undegraded":"2024-01-27T01:07:41.986261+0000","last_fullsized":"2024-01-27T01:07:41.986261+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:26:12.872499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.994833+0000","last_change":"2024-01-27T01:07:27.746921+0000","last_active":"2024-01-27T01:07:41.994833+0000","last_peered":"2024-01-27T01:07:41.994833+0000","last_clean":"2024-01-27T01:07:41.994833+0000","last_became_active":"2024-01-27T01:07:27.746630+0000","last_became_peered":"2024-01-27T01:07:27.746630+0000","last_unstale":"2024-01-27T01:07:41.994833+0000","last_undegraded":"2024-01-27T01:07:41.994833+0000","last_fullsized":"2024-01-27T01:07:41.994833+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:03:41.111732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.020811+0000","last_change":"2024-01-27T01:07:42.019567+0000","last_active":"2024-01-27T01:07:42.020811+0000","last_peered":"2024-01-27T01:07:42.020811+0000","last_clean":"2024-01-27T01:07:42.020811+0000","last_became_active":"2024-01-27T01:07:42.019261+0000","last_became_peered":"2024-01-27T01:07:42.019261+0000","last_unstale":"2024-01-27T01:07:42.020811+0000","last_undegraded":"2024-01-27T01:07:42.020811+0000","last_fullsized":"2024-01-27T01:07:42.020811+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:49:15.861867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982243+0000","last_change":"2024-01-27T01:07:40.987525+0000","last_active":"2024-01-27T01:07:41.982243+0000","last_peered":"2024-01-27T01:07:41.982243+0000","last_clean":"2024-01-27T01:07:41.982243+0000","last_became_active":"2024-01-27T01:07:40.986541+0000","last_became_peered":"2024-01-27T01:07:40.986541+0000","last_unstale":"2024-01-27T01:07:41.982243+0000","last_undegraded":"2024-01-27T01:07:41.982243+0000","last_fullsized":"2024-01-27T01:07:41.982243+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:22:57.587494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.992694+0000","last_change":"2024-01-27T01:07:27.749637+0000","last_active":"2024-01-27T01:07:41.992694+0000","last_peered":"2024-01-27T01:07:41.992694+0000","last_clean":"2024-01-27T01:07:41.992694+0000","last_became_active":"2024-01-27T01:07:27.749170+0000","last_became_peered":"2024-01-27T01:07:27.749170+0000","last_unstale":"2024-01-27T01:07:41.992694+0000","last_undegraded":"2024-01-27T01:07:41.992694+0000","last_fullsized":"2024-01-27T01:07:41.992694+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:10:06.064380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.017034+0000","last_change":"2024-01-27T01:07:42.014208+0000","last_active":"2024-01-27T01:07:42.017034+0000","last_peered":"2024-01-27T01:07:42.017034+0000","last_clean":"2024-01-27T01:07:42.017034+0000","last_became_active":"2024-01-27T01:07:42.013983+0000","last_became_peered":"2024-01-27T01:07:42.013983+0000","last_unstale":"2024-01-27T01:07:42.017034+0000","last_undegraded":"2024-01-27T01:07:42.017034+0000","last_fullsized":"2024-01-27T01:07:42.017034+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:02:31.556291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985310+0000","last_change":"2024-01-27T01:07:40.990547+0000","last_active":"2024-01-27T01:07:41.985310+0000","last_peered":"2024-01-27T01:07:41.985310+0000","last_clean":"2024-01-27T01:07:41.985310+0000","last_became_active":"2024-01-27T01:07:40.990249+0000","last_became_peered":"2024-01-27T01:07:40.990249+0000","last_unstale":"2024-01-27T01:07:41.985310+0000","last_undegraded":"2024-01-27T01:07:41.985310+0000","last_fullsized":"2024-01-27T01:07:41.985310+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:15:24.216671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995182+0000","last_change":"2024-01-27T01:07:27.748872+0000","last_active":"2024-01-27T01:07:41.995182+0000","last_peered":"2024-01-27T01:07:41.995182+0000","last_clean":"2024-01-27T01:07:41.995182+0000","last_became_active":"2024-01-27T01:07:27.748549+0000","last_became_peered":"2024-01-27T01:07:27.748549+0000","last_unstale":"2024-01-27T01:07:41.995182+0000","last_undegraded":"2024-01-27T01:07:41.995182+0000","last_fullsized":"2024-01-27T01:07:41.995182+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:58:33.718993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.992417+0000","last_change":"2024-01-27T01:07:41.991372+0000","last_active":"2024-01-27T01:07:41.992417+0000","last_peered":"2024-01-27T01:07:41.992417+0000","last_clean":"2024-01-27T01:07:41.992417+0000","last_became_active":"2024-01-27T01:07:41.991084+0000","last_became_peered":"2024-01-27T01:07:41.991084+0000","last_unstale":"2024-01-27T01:07:41.992417+0000","last_undegraded":"2024-01-27T01:07:41.992417+0000","last_fullsized":"2024-01-27T01:07:41.992417+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:23:17.453403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984768+0000","last_change":"2024-01-27T01:07:40.986837+0000","last_active":"2024-01-27T01:07:41.984768+0000","last_peered":"2024-01-27T01:07:41.984768+0000","last_clean":"2024-01-27T01:07:41.984768+0000","last_became_active":"2024-01-27T01:07:40.986445+0000","last_became_peered":"2024-01-27T01:07:40.986445+0000","last_unstale":"2024-01-27T01:07:41.984768+0000","last_undegraded":"2024-01-27T01:07:41.984768+0000","last_fullsized":"2024-01-27T01:07:41.984768+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:55:53.993289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982042+0000","last_change":"2024-01-27T01:07:27.755023+0000","last_active":"2024-01-27T01:07:41.982042+0000","last_peered":"2024-01-27T01:07:41.982042+0000","last_clean":"2024-01-27T01:07:41.982042+0000","last_became_active":"2024-01-27T01:07:27.754304+0000","last_became_peered":"2024-01-27T01:07:27.754304+0000","last_unstale":"2024-01-27T01:07:41.982042+0000","last_undegraded":"2024-01-27T01:07:41.982042+0000","last_fullsized":"2024-01-27T01:07:41.982042+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:28:12.245965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.016805+0000","last_change":"2024-01-27T01:07:42.016325+0000","last_active":"2024-01-27T01:07:42.016805+0000","last_peered":"2024-01-27T01:07:42.016805+0000","last_clean":"2024-01-27T01:07:42.016805+0000","last_became_active":"2024-01-27T01:07:42.015981+0000","last_became_peered":"2024-01-27T01:07:42.015981+0000","last_unstale":"2024-01-27T01:07:42.016805+0000","last_undegraded":"2024-01-27T01:07:42.016805+0000","last_fullsized":"2024-01-27T01:07:42.016805+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:49:46.679016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985162+0000","last_change":"2024-01-27T01:07:40.986796+0000","last_active":"2024-01-27T01:07:41.985162+0000","last_peered":"2024-01-27T01:07:41.985162+0000","last_clean":"2024-01-27T01:07:41.985162+0000","last_became_active":"2024-01-27T01:07:40.986511+0000","last_became_peered":"2024-01-27T01:07:40.986511+0000","last_unstale":"2024-01-27T01:07:41.985162+0000","last_undegraded":"2024-01-27T01:07:41.985162+0000","last_fullsized":"2024-01-27T01:07:41.985162+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:39.954376+0000","last_clean_scrub_stamp":"2024-01-27T01:07:39.954376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:20.549192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.988350+0000","last_change":"2024-01-27T01:07:27.747314+0000","last_active":"2024-01-27T01:07:41.988350+0000","last_peered":"2024-01-27T01:07:41.988350+0000","last_clean":"2024-01-27T01:07:41.988350+0000","last_became_active":"2024-01-27T01:07:27.746947+0000","last_became_peered":"2024-01-27T01:07:27.746947+0000","last_unstale":"2024-01-27T01:07:41.988350+0000","last_undegraded":"2024-01-27T01:07:41.988350+0000","last_fullsized":"2024-01-27T01:07:41.988350+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:25:01.603669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.001204+0000","last_change":"2024-01-27T01:07:41.992826+0000","last_active":"2024-01-27T01:07:42.001204+0000","last_peered":"2024-01-27T01:07:42.001204+0000","last_clean":"2024-01-27T01:07:42.001204+0000","last_became_active":"2024-01-27T01:07:41.991941+0000","last_became_peered":"2024-01-27T01:07:41.991941+0000","last_unstale":"2024-01-27T01:07:42.001204+0000","last_undegraded":"2024-01-27T01:07:42.001204+0000","last_fullsized":"2024-01-27T01:07:42.001204+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:36:39.545539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984533+0000","last_change":"2024-01-27T01:07:27.755462+0000","last_active":"2024-01-27T01:07:41.984533+0000","last_peered":"2024-01-27T01:07:41.984533+0000","last_clean":"2024-01-27T01:07:41.984533+0000","last_became_active":"2024-01-27T01:07:27.755131+0000","last_became_peered":"2024-01-27T01:07:27.755131+0000","last_unstale":"2024-01-27T01:07:41.984533+0000","last_undegraded":"2024-01-27T01:07:41.984533+0000","last_fullsized":"2024-01-27T01:07:41.984533+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:32:22.760237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"242'4","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.992931+0000","last_change":"2024-01-27T01:07:26.950037+0000","last_active":"2024-01-27T01:07:41.992931+0000","last_peered":"2024-01-27T01:07:41.992931+0000","last_clean":"2024-01-27T01:07:41.992931+0000","last_became_active":"2024-01-27T01:07:26.949604+0000","last_became_peered":"2024-01-27T01:07:26.949604+0000","last_unstale":"2024-01-27T01:07:41.992931+0000","last_undegraded":"2024-01-27T01:07:41.992931+0000","last_fullsized":"2024-01-27T01:07:41.992931+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:01:17.838521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.011116+0000","last_change":"2024-01-27T01:07:42.007084+0000","last_active":"2024-01-27T01:07:42.011116+0000","last_peered":"2024-01-27T01:07:42.011116+0000","last_clean":"2024-01-27T01:07:42.011116+0000","last_became_active":"2024-01-27T01:07:42.006767+0000","last_became_peered":"2024-01-27T01:07:42.006767+0000","last_unstale":"2024-01-27T01:07:42.011116+0000","last_undegraded":"2024-01-27T01:07:42.011116+0000","last_fullsized":"2024-01-27T01:07:42.011116+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:29:57.281939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985207+0000","last_change":"2024-01-27T01:07:27.755690+0000","last_active":"2024-01-27T01:07:41.985207+0000","last_peered":"2024-01-27T01:07:41.985207+0000","last_clean":"2024-01-27T01:07:41.985207+0000","last_became_active":"2024-01-27T01:07:27.755402+0000","last_became_peered":"2024-01-27T01:07:27.755402+0000","last_unstale":"2024-01-27T01:07:41.985207+0000","last_undegraded":"2024-01-27T01:07:41.985207+0000","last_fullsized":"2024-01-27T01:07:41.985207+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:25:59.841939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"242'2","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.998721+0000","last_change":"2024-01-27T01:07:26.951267+0000","last_active":"2024-01-27T01:07:41.998721+0000","last_peered":"2024-01-27T01:07:41.998721+0000","last_clean":"2024-01-27T01:07:41.998721+0000","last_became_active":"2024-01-27T01:07:26.950967+0000","last_became_peered":"2024-01-27T01:07:26.950967+0000","last_unstale":"2024-01-27T01:07:41.998721+0000","last_undegraded":"2024-01-27T01:07:41.998721+0000","last_fullsized":"2024-01-27T01:07:41.998721+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:37:00.613294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.020790+0000","last_change":"2024-01-27T01:07:42.019332+0000","last_active":"2024-01-27T01:07:42.020790+0000","last_peered":"2024-01-27T01:07:42.020790+0000","last_clean":"2024-01-27T01:07:42.020790+0000","last_became_active":"2024-01-27T01:07:42.018863+0000","last_became_peered":"2024-01-27T01:07:42.018863+0000","last_unstale":"2024-01-27T01:07:42.020790+0000","last_undegraded":"2024-01-27T01:07:42.020790+0000","last_fullsized":"2024-01-27T01:07:42.020790+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:20.301704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.992509+0000","last_change":"2024-01-27T01:07:27.752814+0000","last_active":"2024-01-27T01:07:41.992509+0000","last_peered":"2024-01-27T01:07:41.992509+0000","last_clean":"2024-01-27T01:07:41.992509+0000","last_became_active":"2024-01-27T01:07:27.752459+0000","last_became_peered":"2024-01-27T01:07:27.752459+0000","last_unstale":"2024-01-27T01:07:41.992509+0000","last_undegraded":"2024-01-27T01:07:41.992509+0000","last_fullsized":"2024-01-27T01:07:41.992509+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:44:32.042391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"242'2","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.994509+0000","last_change":"2024-01-27T01:07:26.945958+0000","last_active":"2024-01-27T01:07:41.994509+0000","last_peered":"2024-01-27T01:07:41.994509+0000","last_clean":"2024-01-27T01:07:41.994509+0000","last_became_active":"2024-01-27T01:07:26.945671+0000","last_became_peered":"2024-01-27T01:07:26.945671+0000","last_unstale":"2024-01-27T01:07:41.994509+0000","last_undegraded":"2024-01-27T01:07:41.994509+0000","last_fullsized":"2024-01-27T01:07:41.994509+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:20.901463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.012905+0000","last_change":"2024-01-27T01:07:42.009910+0000","last_active":"2024-01-27T01:07:42.012905+0000","last_peered":"2024-01-27T01:07:42.012905+0000","last_clean":"2024-01-27T01:07:42.012905+0000","last_became_active":"2024-01-27T01:07:42.009517+0000","last_became_peered":"2024-01-27T01:07:42.009517+0000","last_unstale":"2024-01-27T01:07:42.012905+0000","last_undegraded":"2024-01-27T01:07:42.012905+0000","last_fullsized":"2024-01-27T01:07:42.012905+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:10:02.105512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.994311+0000","last_change":"2024-01-27T01:07:27.756421+0000","last_active":"2024-01-27T01:07:41.994311+0000","last_peered":"2024-01-27T01:07:41.994311+0000","last_clean":"2024-01-27T01:07:41.994311+0000","last_became_active":"2024-01-27T01:07:27.756102+0000","last_became_peered":"2024-01-27T01:07:27.756102+0000","last_unstale":"2024-01-27T01:07:41.994311+0000","last_undegraded":"2024-01-27T01:07:41.994311+0000","last_fullsized":"2024-01-27T01:07:41.994311+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:23:49.531160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"242'1","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985595+0000","last_change":"2024-01-27T01:07:26.957169+0000","last_active":"2024-01-27T01:07:41.985595+0000","last_peered":"2024-01-27T01:07:41.985595+0000","last_clean":"2024-01-27T01:07:41.985595+0000","last_became_active":"2024-01-27T01:07:26.956902+0000","last_became_peered":"2024-01-27T01:07:26.956902+0000","last_unstale":"2024-01-27T01:07:41.985595+0000","last_undegraded":"2024-01-27T01:07:41.985595+0000","last_fullsized":"2024-01-27T01:07:41.985595+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:27:55.458182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.993951+0000","last_change":"2024-01-27T01:07:41.991557+0000","last_active":"2024-01-27T01:07:41.993951+0000","last_peered":"2024-01-27T01:07:41.993951+0000","last_clean":"2024-01-27T01:07:41.993951+0000","last_became_active":"2024-01-27T01:07:41.991257+0000","last_became_peered":"2024-01-27T01:07:41.991257+0000","last_unstale":"2024-01-27T01:07:41.993951+0000","last_undegraded":"2024-01-27T01:07:41.993951+0000","last_fullsized":"2024-01-27T01:07:41.993951+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:00.162997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985629+0000","last_change":"2024-01-27T01:07:27.752682+0000","last_active":"2024-01-27T01:07:41.985629+0000","last_peered":"2024-01-27T01:07:41.985629+0000","last_clean":"2024-01-27T01:07:41.985629+0000","last_became_active":"2024-01-27T01:07:27.752401+0000","last_became_peered":"2024-01-27T01:07:27.752401+0000","last_unstale":"2024-01-27T01:07:41.985629+0000","last_undegraded":"2024-01-27T01:07:41.985629+0000","last_fullsized":"2024-01-27T01:07:41.985629+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:55:40.962728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"242'1","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995265+0000","last_change":"2024-01-27T01:07:26.951702+0000","last_active":"2024-01-27T01:07:41.995265+0000","last_peered":"2024-01-27T01:07:41.995265+0000","last_clean":"2024-01-27T01:07:41.995265+0000","last_became_active":"2024-01-27T01:07:26.951367+0000","last_became_peered":"2024-01-27T01:07:26.951367+0000","last_unstale":"2024-01-27T01:07:41.995265+0000","last_undegraded":"2024-01-27T01:07:41.995265+0000","last_fullsized":"2024-01-27T01:07:41.995265+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:25:00.596520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.013327+0000","last_change":"2024-01-27T01:07:42.009310+0000","last_active":"2024-01-27T01:07:42.013327+0000","last_peered":"2024-01-27T01:07:42.013327+0000","last_clean":"2024-01-27T01:07:42.013327+0000","last_became_active":"2024-01-27T01:07:42.008856+0000","last_became_peered":"2024-01-27T01:07:42.008856+0000","last_unstale":"2024-01-27T01:07:42.013327+0000","last_undegraded":"2024-01-27T01:07:42.013327+0000","last_fullsized":"2024-01-27T01:07:42.013327+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:04:01.316302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979338+0000","last_change":"2024-01-27T01:07:27.750189+0000","last_active":"2024-01-27T01:07:41.979338+0000","last_peered":"2024-01-27T01:07:41.979338+0000","last_clean":"2024-01-27T01:07:41.979338+0000","last_became_active":"2024-01-27T01:07:27.749867+0000","last_became_peered":"2024-01-27T01:07:27.749867+0000","last_unstale":"2024-01-27T01:07:41.979338+0000","last_undegraded":"2024-01-27T01:07:41.979338+0000","last_fullsized":"2024-01-27T01:07:41.979338+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:39:46.126956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979097+0000","last_change":"2024-01-27T01:07:26.954992+0000","last_active":"2024-01-27T01:07:41.979097+0000","last_peered":"2024-01-27T01:07:41.979097+0000","last_clean":"2024-01-27T01:07:41.979097+0000","last_became_active":"2024-01-27T01:07:26.954701+0000","last_became_peered":"2024-01-27T01:07:26.954701+0000","last_unstale":"2024-01-27T01:07:41.979097+0000","last_undegraded":"2024-01-27T01:07:41.979097+0000","last_fullsized":"2024-01-27T01:07:41.979097+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:54:24.976708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.013358+0000","last_change":"2024-01-27T01:07:42.009611+0000","last_active":"2024-01-27T01:07:42.013358+0000","last_peered":"2024-01-27T01:07:42.013358+0000","last_clean":"2024-01-27T01:07:42.013358+0000","last_became_active":"2024-01-27T01:07:42.009362+0000","last_became_peered":"2024-01-27T01:07:42.009362+0000","last_unstale":"2024-01-27T01:07:42.013358+0000","last_undegraded":"2024-01-27T01:07:42.013358+0000","last_fullsized":"2024-01-27T01:07:42.013358+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:30:51.042154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982305+0000","last_change":"2024-01-27T01:07:27.743124+0000","last_active":"2024-01-27T01:07:41.982305+0000","last_peered":"2024-01-27T01:07:41.982305+0000","last_clean":"2024-01-27T01:07:41.982305+0000","last_became_active":"2024-01-27T01:07:27.742799+0000","last_became_peered":"2024-01-27T01:07:27.742799+0000","last_unstale":"2024-01-27T01:07:41.982305+0000","last_undegraded":"2024-01-27T01:07:41.982305+0000","last_fullsized":"2024-01-27T01:07:41.982305+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:14.714021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.988678+0000","last_change":"2024-01-27T01:07:26.946511+0000","last_active":"2024-01-27T01:07:41.988678+0000","last_peered":"2024-01-27T01:07:41.988678+0000","last_clean":"2024-01-27T01:07:41.988678+0000","last_became_active":"2024-01-27T01:07:26.946186+0000","last_became_peered":"2024-01-27T01:07:26.946186+0000","last_unstale":"2024-01-27T01:07:41.988678+0000","last_undegraded":"2024-01-27T01:07:41.988678+0000","last_fullsized":"2024-01-27T01:07:41.988678+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:24:39.506445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.006017+0000","last_change":"2024-01-27T01:07:42.003996+0000","last_active":"2024-01-27T01:07:42.006017+0000","last_peered":"2024-01-27T01:07:42.006017+0000","last_clean":"2024-01-27T01:07:42.006017+0000","last_became_active":"2024-01-27T01:07:42.003482+0000","last_became_peered":"2024-01-27T01:07:42.003482+0000","last_unstale":"2024-01-27T01:07:42.006017+0000","last_undegraded":"2024-01-27T01:07:42.006017+0000","last_fullsized":"2024-01-27T01:07:42.006017+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:24:52.492741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984547+0000","last_change":"2024-01-27T01:07:27.749767+0000","last_active":"2024-01-27T01:07:41.984547+0000","last_peered":"2024-01-27T01:07:41.984547+0000","last_clean":"2024-01-27T01:07:41.984547+0000","last_became_active":"2024-01-27T01:07:27.749508+0000","last_became_peered":"2024-01-27T01:07:27.749508+0000","last_unstale":"2024-01-27T01:07:41.984547+0000","last_undegraded":"2024-01-27T01:07:41.984547+0000","last_fullsized":"2024-01-27T01:07:41.984547+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:56:10.391021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.993835+0000","last_change":"2024-01-27T01:07:26.947663+0000","last_active":"2024-01-27T01:07:41.993835+0000","last_peered":"2024-01-27T01:07:41.993835+0000","last_clean":"2024-01-27T01:07:41.993835+0000","last_became_active":"2024-01-27T01:07:26.947378+0000","last_became_peered":"2024-01-27T01:07:26.947378+0000","last_unstale":"2024-01-27T01:07:41.993835+0000","last_undegraded":"2024-01-27T01:07:41.993835+0000","last_fullsized":"2024-01-27T01:07:41.993835+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:28:30.563975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.020641+0000","last_change":"2024-01-27T01:07:42.019102+0000","last_active":"2024-01-27T01:07:42.020641+0000","last_peered":"2024-01-27T01:07:42.020641+0000","last_clean":"2024-01-27T01:07:42.020641+0000","last_became_active":"2024-01-27T01:07:42.018594+0000","last_became_peered":"2024-01-27T01:07:42.018594+0000","last_unstale":"2024-01-27T01:07:42.020641+0000","last_undegraded":"2024-01-27T01:07:42.020641+0000","last_fullsized":"2024-01-27T01:07:42.020641+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:23:25.015337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995991+0000","last_change":"2024-01-27T01:07:27.749325+0000","last_active":"2024-01-27T01:07:41.995991+0000","last_peered":"2024-01-27T01:07:41.995991+0000","last_clean":"2024-01-27T01:07:41.995991+0000","last_became_active":"2024-01-27T01:07:27.744840+0000","last_became_peered":"2024-01-27T01:07:27.744840+0000","last_unstale":"2024-01-27T01:07:41.995991+0000","last_undegraded":"2024-01-27T01:07:41.995991+0000","last_fullsized":"2024-01-27T01:07:41.995991+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:15:38.535322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979665+0000","last_change":"2024-01-27T01:07:26.948157+0000","last_active":"2024-01-27T01:07:41.979665+0000","last_peered":"2024-01-27T01:07:41.979665+0000","last_clean":"2024-01-27T01:07:41.979665+0000","last_became_active":"2024-01-27T01:07:26.947870+0000","last_became_peered":"2024-01-27T01:07:26.947870+0000","last_unstale":"2024-01-27T01:07:41.979665+0000","last_undegraded":"2024-01-27T01:07:41.979665+0000","last_fullsized":"2024-01-27T01:07:41.979665+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:04:16.755137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.000911+0000","last_change":"2024-01-27T01:07:41.997557+0000","last_active":"2024-01-27T01:07:42.000911+0000","last_peered":"2024-01-27T01:07:42.000911+0000","last_clean":"2024-01-27T01:07:42.000911+0000","last_became_active":"2024-01-27T01:07:41.997288+0000","last_became_peered":"2024-01-27T01:07:41.997288+0000","last_unstale":"2024-01-27T01:07:42.000911+0000","last_undegraded":"2024-01-27T01:07:42.000911+0000","last_fullsized":"2024-01-27T01:07:42.000911+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:36:28.490987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979190+0000","last_change":"2024-01-27T01:07:27.744220+0000","last_active":"2024-01-27T01:07:41.979190+0000","last_peered":"2024-01-27T01:07:41.979190+0000","last_clean":"2024-01-27T01:07:41.979190+0000","last_became_active":"2024-01-27T01:07:27.742448+0000","last_became_peered":"2024-01-27T01:07:27.742448+0000","last_unstale":"2024-01-27T01:07:41.979190+0000","last_undegraded":"2024-01-27T01:07:41.979190+0000","last_fullsized":"2024-01-27T01:07:41.979190+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:56:03.017967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"242'3","reported_seq":17,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981788+0000","last_change":"2024-01-27T01:07:26.953006+0000","last_active":"2024-01-27T01:07:41.981788+0000","last_peered":"2024-01-27T01:07:41.981788+0000","last_clean":"2024-01-27T01:07:41.981788+0000","last_became_active":"2024-01-27T01:07:26.952730+0000","last_became_peered":"2024-01-27T01:07:26.952730+0000","last_unstale":"2024-01-27T01:07:41.981788+0000","last_undegraded":"2024-01-27T01:07:41.981788+0000","last_fullsized":"2024-01-27T01:07:41.981788+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:47:14.273722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.012073+0000","last_change":"2024-01-27T01:07:42.007566+0000","last_active":"2024-01-27T01:07:42.012073+0000","last_peered":"2024-01-27T01:07:42.012073+0000","last_clean":"2024-01-27T01:07:42.012073+0000","last_became_active":"2024-01-27T01:07:42.007055+0000","last_became_peered":"2024-01-27T01:07:42.007055+0000","last_unstale":"2024-01-27T01:07:42.012073+0000","last_undegraded":"2024-01-27T01:07:42.012073+0000","last_fullsized":"2024-01-27T01:07:42.012073+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:27:59.374021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985613+0000","last_change":"2024-01-27T01:07:27.747921+0000","last_active":"2024-01-27T01:07:41.985613+0000","last_peered":"2024-01-27T01:07:41.985613+0000","last_clean":"2024-01-27T01:07:41.985613+0000","last_became_active":"2024-01-27T01:07:27.747382+0000","last_became_peered":"2024-01-27T01:07:27.747382+0000","last_unstale":"2024-01-27T01:07:41.985613+0000","last_undegraded":"2024-01-27T01:07:41.985613+0000","last_fullsized":"2024-01-27T01:07:41.985613+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:43:44.312681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"243'8","reported_seq":22,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995022+0000","last_change":"2024-01-27T01:07:26.951048+0000","last_active":"2024-01-27T01:07:41.995022+0000","last_peered":"2024-01-27T01:07:41.995022+0000","last_clean":"2024-01-27T01:07:41.995022+0000","last_became_active":"2024-01-27T01:07:26.950750+0000","last_became_peered":"2024-01-27T01:07:26.950750+0000","last_unstale":"2024-01-27T01:07:41.995022+0000","last_undegraded":"2024-01-27T01:07:41.995022+0000","last_fullsized":"2024-01-27T01:07:41.995022+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:48:29.363281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2768,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.009019+0000","last_change":"2024-01-27T01:07:42.002706+0000","last_active":"2024-01-27T01:07:42.009019+0000","last_peered":"2024-01-27T01:07:42.009019+0000","last_clean":"2024-01-27T01:07:42.009019+0000","last_became_active":"2024-01-27T01:07:42.002441+0000","last_became_peered":"2024-01-27T01:07:42.002441+0000","last_unstale":"2024-01-27T01:07:42.009019+0000","last_undegraded":"2024-01-27T01:07:42.009019+0000","last_fullsized":"2024-01-27T01:07:42.009019+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:35:39.491535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982749+0000","last_change":"2024-01-27T01:07:27.756638+0000","last_active":"2024-01-27T01:07:41.982749+0000","last_peered":"2024-01-27T01:07:41.982749+0000","last_clean":"2024-01-27T01:07:41.982749+0000","last_became_active":"2024-01-27T01:07:27.756219+0000","last_became_peered":"2024-01-27T01:07:27.756219+0000","last_unstale":"2024-01-27T01:07:41.982749+0000","last_undegraded":"2024-01-27T01:07:41.982749+0000","last_fullsized":"2024-01-27T01:07:41.982749+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:24:43.565560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"242'2","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985902+0000","last_change":"2024-01-27T01:07:26.957374+0000","last_active":"2024-01-27T01:07:41.985902+0000","last_peered":"2024-01-27T01:07:41.985902+0000","last_clean":"2024-01-27T01:07:41.985902+0000","last_became_active":"2024-01-27T01:07:26.957184+0000","last_became_peered":"2024-01-27T01:07:26.957184+0000","last_unstale":"2024-01-27T01:07:41.985902+0000","last_undegraded":"2024-01-27T01:07:41.985902+0000","last_fullsized":"2024-01-27T01:07:41.985902+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:43:44.001485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.000750+0000","last_change":"2024-01-27T01:07:41.999362+0000","last_active":"2024-01-27T01:07:42.000750+0000","last_peered":"2024-01-27T01:07:42.000750+0000","last_clean":"2024-01-27T01:07:42.000750+0000","last_became_active":"2024-01-27T01:07:41.999099+0000","last_became_peered":"2024-01-27T01:07:41.999099+0000","last_unstale":"2024-01-27T01:07:42.000750+0000","last_undegraded":"2024-01-27T01:07:42.000750+0000","last_fullsized":"2024-01-27T01:07:42.000750+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:41:14.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.985717+0000","last_change":"2024-01-27T01:07:27.741016+0000","last_active":"2024-01-27T01:07:41.985717+0000","last_peered":"2024-01-27T01:07:41.985717+0000","last_clean":"2024-01-27T01:07:41.985717+0000","last_became_active":"2024-01-27T01:07:27.740724+0000","last_became_peered":"2024-01-27T01:07:27.740724+0000","last_unstale":"2024-01-27T01:07:41.985717+0000","last_undegraded":"2024-01-27T01:07:41.985717+0000","last_fullsized":"2024-01-27T01:07:41.985717+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:19:56.759979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981479+0000","last_change":"2024-01-27T01:07:26.951785+0000","last_active":"2024-01-27T01:07:41.981479+0000","last_peered":"2024-01-27T01:07:41.981479+0000","last_clean":"2024-01-27T01:07:41.981479+0000","last_became_active":"2024-01-27T01:07:26.951519+0000","last_became_peered":"2024-01-27T01:07:26.951519+0000","last_unstale":"2024-01-27T01:07:41.981479+0000","last_undegraded":"2024-01-27T01:07:41.981479+0000","last_fullsized":"2024-01-27T01:07:41.981479+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:06.511483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.000874+0000","last_change":"2024-01-27T01:07:41.994163+0000","last_active":"2024-01-27T01:07:42.000874+0000","last_peered":"2024-01-27T01:07:42.000874+0000","last_clean":"2024-01-27T01:07:42.000874+0000","last_became_active":"2024-01-27T01:07:41.993928+0000","last_became_peered":"2024-01-27T01:07:41.993928+0000","last_unstale":"2024-01-27T01:07:42.000874+0000","last_undegraded":"2024-01-27T01:07:42.000874+0000","last_fullsized":"2024-01-27T01:07:42.000874+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:14:34.374610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.986496+0000","last_change":"2024-01-27T01:07:27.755983+0000","last_active":"2024-01-27T01:07:41.986496+0000","last_peered":"2024-01-27T01:07:41.986496+0000","last_clean":"2024-01-27T01:07:41.986496+0000","last_became_active":"2024-01-27T01:07:27.755715+0000","last_became_peered":"2024-01-27T01:07:27.755715+0000","last_unstale":"2024-01-27T01:07:41.986496+0000","last_undegraded":"2024-01-27T01:07:41.986496+0000","last_fullsized":"2024-01-27T01:07:41.986496+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:11.556927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981153+0000","last_change":"2024-01-27T01:07:26.952237+0000","last_active":"2024-01-27T01:07:41.981153+0000","last_peered":"2024-01-27T01:07:41.981153+0000","last_clean":"2024-01-27T01:07:41.981153+0000","last_became_active":"2024-01-27T01:07:26.951971+0000","last_became_peered":"2024-01-27T01:07:26.951971+0000","last_unstale":"2024-01-27T01:07:41.981153+0000","last_undegraded":"2024-01-27T01:07:41.981153+0000","last_fullsized":"2024-01-27T01:07:41.981153+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:43:36.059851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.004338+0000","last_change":"2024-01-27T01:07:42.001580+0000","last_active":"2024-01-27T01:07:42.004338+0000","last_peered":"2024-01-27T01:07:42.004338+0000","last_clean":"2024-01-27T01:07:42.004338+0000","last_became_active":"2024-01-27T01:07:42.001300+0000","last_became_peered":"2024-01-27T01:07:42.001300+0000","last_unstale":"2024-01-27T01:07:42.004338+0000","last_undegraded":"2024-01-27T01:07:42.004338+0000","last_fullsized":"2024-01-27T01:07:42.004338+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:24:03.567201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982253+0000","last_change":"2024-01-27T01:07:27.746452+0000","last_active":"2024-01-27T01:07:41.982253+0000","last_peered":"2024-01-27T01:07:41.982253+0000","last_clean":"2024-01-27T01:07:41.982253+0000","last_became_active":"2024-01-27T01:07:27.746174+0000","last_became_peered":"2024-01-27T01:07:27.746174+0000","last_unstale":"2024-01-27T01:07:41.982253+0000","last_undegraded":"2024-01-27T01:07:41.982253+0000","last_fullsized":"2024-01-27T01:07:41.982253+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:32.778691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"242'8","reported_seq":22,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.980773+0000","last_change":"2024-01-27T01:07:26.946801+0000","last_active":"2024-01-27T01:07:41.980773+0000","last_peered":"2024-01-27T01:07:41.980773+0000","last_clean":"2024-01-27T01:07:41.980773+0000","last_became_active":"2024-01-27T01:07:26.946453+0000","last_became_peered":"2024-01-27T01:07:26.946453+0000","last_unstale":"2024-01-27T01:07:41.980773+0000","last_undegraded":"2024-01-27T01:07:41.980773+0000","last_fullsized":"2024-01-27T01:07:41.980773+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:11:44.275714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.016792+0000","last_change":"2024-01-27T01:07:42.016234+0000","last_active":"2024-01-27T01:07:42.016792+0000","last_peered":"2024-01-27T01:07:42.016792+0000","last_clean":"2024-01-27T01:07:42.016792+0000","last_became_active":"2024-01-27T01:07:42.015609+0000","last_became_peered":"2024-01-27T01:07:42.015609+0000","last_unstale":"2024-01-27T01:07:42.016792+0000","last_undegraded":"2024-01-27T01:07:42.016792+0000","last_fullsized":"2024-01-27T01:07:42.016792+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:40:45.932550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982340+0000","last_change":"2024-01-27T01:07:27.746480+0000","last_active":"2024-01-27T01:07:41.982340+0000","last_peered":"2024-01-27T01:07:41.982340+0000","last_clean":"2024-01-27T01:07:41.982340+0000","last_became_active":"2024-01-27T01:07:27.746131+0000","last_became_peered":"2024-01-27T01:07:27.746131+0000","last_unstale":"2024-01-27T01:07:41.982340+0000","last_undegraded":"2024-01-27T01:07:41.982340+0000","last_fullsized":"2024-01-27T01:07:41.982340+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:06:55.090910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"242'1","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.992831+0000","last_change":"2024-01-27T01:07:26.958336+0000","last_active":"2024-01-27T01:07:41.992831+0000","last_peered":"2024-01-27T01:07:41.992831+0000","last_clean":"2024-01-27T01:07:41.992831+0000","last_became_active":"2024-01-27T01:07:26.957817+0000","last_became_peered":"2024-01-27T01:07:26.957817+0000","last_unstale":"2024-01-27T01:07:41.992831+0000","last_undegraded":"2024-01-27T01:07:41.992831+0000","last_fullsized":"2024-01-27T01:07:41.992831+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:45.033976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.020153+0000","last_change":"2024-01-27T01:07:42.017341+0000","last_active":"2024-01-27T01:07:42.020153+0000","last_peered":"2024-01-27T01:07:42.020153+0000","last_clean":"2024-01-27T01:07:42.020153+0000","last_became_active":"2024-01-27T01:07:42.015689+0000","last_became_peered":"2024-01-27T01:07:42.015689+0000","last_unstale":"2024-01-27T01:07:42.020153+0000","last_undegraded":"2024-01-27T01:07:42.020153+0000","last_fullsized":"2024-01-27T01:07:42.020153+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:07:07.746380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981055+0000","last_change":"2024-01-27T01:07:27.743490+0000","last_active":"2024-01-27T01:07:41.981055+0000","last_peered":"2024-01-27T01:07:41.981055+0000","last_clean":"2024-01-27T01:07:41.981055+0000","last_became_active":"2024-01-27T01:07:27.743207+0000","last_became_peered":"2024-01-27T01:07:27.743207+0000","last_unstale":"2024-01-27T01:07:41.981055+0000","last_undegraded":"2024-01-27T01:07:41.981055+0000","last_fullsized":"2024-01-27T01:07:41.981055+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:19.339556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"242'2","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982911+0000","last_change":"2024-01-27T01:07:26.946877+0000","last_active":"2024-01-27T01:07:41.982911+0000","last_peered":"2024-01-27T01:07:41.982911+0000","last_clean":"2024-01-27T01:07:41.982911+0000","last_became_active":"2024-01-27T01:07:26.946596+0000","last_became_peered":"2024-01-27T01:07:26.946596+0000","last_unstale":"2024-01-27T01:07:41.982911+0000","last_undegraded":"2024-01-27T01:07:41.982911+0000","last_fullsized":"2024-01-27T01:07:41.982911+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:27:41.114970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.010761+0000","last_change":"2024-01-27T01:07:42.009184+0000","last_active":"2024-01-27T01:07:42.010761+0000","last_peered":"2024-01-27T01:07:42.010761+0000","last_clean":"2024-01-27T01:07:42.010761+0000","last_became_active":"2024-01-27T01:07:42.008857+0000","last_became_peered":"2024-01-27T01:07:42.008857+0000","last_unstale":"2024-01-27T01:07:42.010761+0000","last_undegraded":"2024-01-27T01:07:42.010761+0000","last_fullsized":"2024-01-27T01:07:42.010761+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:04:28.461950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.982436+0000","last_change":"2024-01-27T01:07:27.746482+0000","last_active":"2024-01-27T01:07:41.982436+0000","last_peered":"2024-01-27T01:07:41.982436+0000","last_clean":"2024-01-27T01:07:41.982436+0000","last_became_active":"2024-01-27T01:07:27.746052+0000","last_became_peered":"2024-01-27T01:07:27.746052+0000","last_unstale":"2024-01-27T01:07:41.982436+0000","last_undegraded":"2024-01-27T01:07:41.982436+0000","last_fullsized":"2024-01-27T01:07:41.982436+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:32.387353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"242'1","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.998825+0000","last_change":"2024-01-27T01:07:26.950943+0000","last_active":"2024-01-27T01:07:41.998825+0000","last_peered":"2024-01-27T01:07:41.998825+0000","last_clean":"2024-01-27T01:07:41.998825+0000","last_became_active":"2024-01-27T01:07:26.950251+0000","last_became_peered":"2024-01-27T01:07:26.950251+0000","last_unstale":"2024-01-27T01:07:41.998825+0000","last_undegraded":"2024-01-27T01:07:41.998825+0000","last_fullsized":"2024-01-27T01:07:41.998825+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:31.380205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.012092+0000","last_change":"2024-01-27T01:07:42.007675+0000","last_active":"2024-01-27T01:07:42.012092+0000","last_peered":"2024-01-27T01:07:42.012092+0000","last_clean":"2024-01-27T01:07:42.012092+0000","last_became_active":"2024-01-27T01:07:42.007228+0000","last_became_peered":"2024-01-27T01:07:42.007228+0000","last_unstale":"2024-01-27T01:07:42.012092+0000","last_undegraded":"2024-01-27T01:07:42.012092+0000","last_fullsized":"2024-01-27T01:07:42.012092+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:06:58.292213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.980187+0000","last_change":"2024-01-27T01:07:27.742779+0000","last_active":"2024-01-27T01:07:41.980187+0000","last_peered":"2024-01-27T01:07:41.980187+0000","last_clean":"2024-01-27T01:07:41.980187+0000","last_became_active":"2024-01-27T01:07:27.742462+0000","last_became_peered":"2024-01-27T01:07:27.742462+0000","last_unstale":"2024-01-27T01:07:41.980187+0000","last_undegraded":"2024-01-27T01:07:41.980187+0000","last_fullsized":"2024-01-27T01:07:41.980187+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:24:28.493839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.986365+0000","last_change":"2024-01-27T01:07:26.956417+0000","last_active":"2024-01-27T01:07:41.986365+0000","last_peered":"2024-01-27T01:07:41.986365+0000","last_clean":"2024-01-27T01:07:41.986365+0000","last_became_active":"2024-01-27T01:07:26.956128+0000","last_became_peered":"2024-01-27T01:07:26.956128+0000","last_unstale":"2024-01-27T01:07:41.986365+0000","last_undegraded":"2024-01-27T01:07:41.986365+0000","last_fullsized":"2024-01-27T01:07:41.986365+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:05:22.165694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.004889+0000","last_change":"2024-01-27T01:07:42.002859+0000","last_active":"2024-01-27T01:07:42.004889+0000","last_peered":"2024-01-27T01:07:42.004889+0000","last_clean":"2024-01-27T01:07:42.004889+0000","last_became_active":"2024-01-27T01:07:42.002585+0000","last_became_peered":"2024-01-27T01:07:42.002585+0000","last_unstale":"2024-01-27T01:07:42.004889+0000","last_undegraded":"2024-01-27T01:07:42.004889+0000","last_fullsized":"2024-01-27T01:07:42.004889+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:03:17.562068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979747+0000","last_change":"2024-01-27T01:07:27.752330+0000","last_active":"2024-01-27T01:07:41.979747+0000","last_peered":"2024-01-27T01:07:41.979747+0000","last_clean":"2024-01-27T01:07:41.979747+0000","last_became_active":"2024-01-27T01:07:27.752040+0000","last_became_peered":"2024-01-27T01:07:27.752040+0000","last_unstale":"2024-01-27T01:07:41.979747+0000","last_undegraded":"2024-01-27T01:07:41.979747+0000","last_fullsized":"2024-01-27T01:07:41.979747+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:36:14.454947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"242'2","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.980077+0000","last_change":"2024-01-27T01:07:26.947483+0000","last_active":"2024-01-27T01:07:41.980077+0000","last_peered":"2024-01-27T01:07:41.980077+0000","last_clean":"2024-01-27T01:07:41.980077+0000","last_became_active":"2024-01-27T01:07:26.947172+0000","last_became_peered":"2024-01-27T01:07:26.947172+0000","last_unstale":"2024-01-27T01:07:41.980077+0000","last_undegraded":"2024-01-27T01:07:41.980077+0000","last_fullsized":"2024-01-27T01:07:41.980077+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:34:03.604469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.003800+0000","last_change":"2024-01-27T01:07:42.002232+0000","last_active":"2024-01-27T01:07:42.003800+0000","last_peered":"2024-01-27T01:07:42.003800+0000","last_clean":"2024-01-27T01:07:42.003800+0000","last_became_active":"2024-01-27T01:07:42.001905+0000","last_became_peered":"2024-01-27T01:07:42.001905+0000","last_unstale":"2024-01-27T01:07:42.003800+0000","last_undegraded":"2024-01-27T01:07:42.003800+0000","last_fullsized":"2024-01-27T01:07:42.003800+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:45:46.843500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984511+0000","last_change":"2024-01-27T01:07:27.755356+0000","last_active":"2024-01-27T01:07:41.984511+0000","last_peered":"2024-01-27T01:07:41.984511+0000","last_clean":"2024-01-27T01:07:41.984511+0000","last_became_active":"2024-01-27T01:07:27.755108+0000","last_became_peered":"2024-01-27T01:07:27.755108+0000","last_unstale":"2024-01-27T01:07:41.984511+0000","last_undegraded":"2024-01-27T01:07:41.984511+0000","last_fullsized":"2024-01-27T01:07:41.984511+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:51:14.988606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995367+0000","last_change":"2024-01-27T01:07:26.950680+0000","last_active":"2024-01-27T01:07:41.995367+0000","last_peered":"2024-01-27T01:07:41.995367+0000","last_clean":"2024-01-27T01:07:41.995367+0000","last_became_active":"2024-01-27T01:07:26.950395+0000","last_became_peered":"2024-01-27T01:07:26.950395+0000","last_unstale":"2024-01-27T01:07:41.995367+0000","last_undegraded":"2024-01-27T01:07:41.995367+0000","last_fullsized":"2024-01-27T01:07:41.995367+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:28:11.238816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.002928+0000","last_change":"2024-01-27T01:07:42.000220+0000","last_active":"2024-01-27T01:07:42.002928+0000","last_peered":"2024-01-27T01:07:42.002928+0000","last_clean":"2024-01-27T01:07:42.002928+0000","last_became_active":"2024-01-27T01:07:41.999872+0000","last_became_peered":"2024-01-27T01:07:41.999872+0000","last_unstale":"2024-01-27T01:07:42.002928+0000","last_undegraded":"2024-01-27T01:07:42.002928+0000","last_fullsized":"2024-01-27T01:07:42.002928+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:40:17.514841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995284+0000","last_change":"2024-01-27T01:07:27.748049+0000","last_active":"2024-01-27T01:07:41.995284+0000","last_peered":"2024-01-27T01:07:41.995284+0000","last_clean":"2024-01-27T01:07:41.995284+0000","last_became_active":"2024-01-27T01:07:27.747765+0000","last_became_peered":"2024-01-27T01:07:27.747765+0000","last_unstale":"2024-01-27T01:07:41.995284+0000","last_undegraded":"2024-01-27T01:07:41.995284+0000","last_fullsized":"2024-01-27T01:07:41.995284+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:51:33.521276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.996168+0000","last_change":"2024-01-27T01:07:26.950170+0000","last_active":"2024-01-27T01:07:41.996168+0000","last_peered":"2024-01-27T01:07:41.996168+0000","last_clean":"2024-01-27T01:07:41.996168+0000","last_became_active":"2024-01-27T01:07:26.949857+0000","last_became_peered":"2024-01-27T01:07:26.949857+0000","last_unstale":"2024-01-27T01:07:41.996168+0000","last_undegraded":"2024-01-27T01:07:41.996168+0000","last_fullsized":"2024-01-27T01:07:41.996168+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:29.510893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.007784+0000","last_change":"2024-01-27T01:07:42.005851+0000","last_active":"2024-01-27T01:07:42.007784+0000","last_peered":"2024-01-27T01:07:42.007784+0000","last_clean":"2024-01-27T01:07:42.007784+0000","last_became_active":"2024-01-27T01:07:42.004370+0000","last_became_peered":"2024-01-27T01:07:42.004370+0000","last_unstale":"2024-01-27T01:07:42.007784+0000","last_undegraded":"2024-01-27T01:07:42.007784+0000","last_fullsized":"2024-01-27T01:07:42.007784+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:59.153074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.997166+0000","last_change":"2024-01-27T01:07:27.753040+0000","last_active":"2024-01-27T01:07:41.997166+0000","last_peered":"2024-01-27T01:07:41.997166+0000","last_clean":"2024-01-27T01:07:41.997166+0000","last_became_active":"2024-01-27T01:07:27.752784+0000","last_became_peered":"2024-01-27T01:07:27.752784+0000","last_unstale":"2024-01-27T01:07:41.997166+0000","last_undegraded":"2024-01-27T01:07:41.997166+0000","last_fullsized":"2024-01-27T01:07:41.997166+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:09:57.375468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.997934+0000","last_change":"2024-01-27T01:07:26.944368+0000","last_active":"2024-01-27T01:07:41.997934+0000","last_peered":"2024-01-27T01:07:41.997934+0000","last_clean":"2024-01-27T01:07:41.997934+0000","last_became_active":"2024-01-27T01:07:26.944076+0000","last_became_peered":"2024-01-27T01:07:26.944076+0000","last_unstale":"2024-01-27T01:07:41.997934+0000","last_undegraded":"2024-01-27T01:07:41.997934+0000","last_fullsized":"2024-01-27T01:07:41.997934+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:59:38.148215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.999202+0000","last_change":"2024-01-27T01:07:41.991392+0000","last_active":"2024-01-27T01:07:41.999202+0000","last_peered":"2024-01-27T01:07:41.999202+0000","last_clean":"2024-01-27T01:07:41.999202+0000","last_became_active":"2024-01-27T01:07:41.991102+0000","last_became_peered":"2024-01-27T01:07:41.991102+0000","last_unstale":"2024-01-27T01:07:41.999202+0000","last_undegraded":"2024-01-27T01:07:41.999202+0000","last_fullsized":"2024-01-27T01:07:41.999202+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:25:15.639710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984464+0000","last_change":"2024-01-27T01:07:27.747646+0000","last_active":"2024-01-27T01:07:41.984464+0000","last_peered":"2024-01-27T01:07:41.984464+0000","last_clean":"2024-01-27T01:07:41.984464+0000","last_became_active":"2024-01-27T01:07:27.747345+0000","last_became_peered":"2024-01-27T01:07:27.747345+0000","last_unstale":"2024-01-27T01:07:41.984464+0000","last_undegraded":"2024-01-27T01:07:41.984464+0000","last_fullsized":"2024-01-27T01:07:41.984464+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:30:45.126288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"242'2","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979523+0000","last_change":"2024-01-27T01:07:26.950235+0000","last_active":"2024-01-27T01:07:41.979523+0000","last_peered":"2024-01-27T01:07:41.979523+0000","last_clean":"2024-01-27T01:07:41.979523+0000","last_became_active":"2024-01-27T01:07:26.949923+0000","last_became_peered":"2024-01-27T01:07:26.949923+0000","last_unstale":"2024-01-27T01:07:41.979523+0000","last_undegraded":"2024-01-27T01:07:41.979523+0000","last_fullsized":"2024-01-27T01:07:41.979523+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:27:55.458182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.020664+0000","last_change":"2024-01-27T01:07:42.019294+0000","last_active":"2024-01-27T01:07:42.020664+0000","last_peered":"2024-01-27T01:07:42.020664+0000","last_clean":"2024-01-27T01:07:42.020664+0000","last_became_active":"2024-01-27T01:07:42.018289+0000","last_became_peered":"2024-01-27T01:07:42.018289+0000","last_unstale":"2024-01-27T01:07:42.020664+0000","last_undegraded":"2024-01-27T01:07:42.020664+0000","last_fullsized":"2024-01-27T01:07:42.020664+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:24:03.567201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984578+0000","last_change":"2024-01-27T01:07:27.758370+0000","last_active":"2024-01-27T01:07:41.984578+0000","last_peered":"2024-01-27T01:07:41.984578+0000","last_clean":"2024-01-27T01:07:41.984578+0000","last_became_active":"2024-01-27T01:07:27.758007+0000","last_became_peered":"2024-01-27T01:07:27.758007+0000","last_unstale":"2024-01-27T01:07:41.984578+0000","last_undegraded":"2024-01-27T01:07:41.984578+0000","last_fullsized":"2024-01-27T01:07:41.984578+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:46:07.518631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"243'1","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981495+0000","last_change":"2024-01-27T01:07:26.953360+0000","last_active":"2024-01-27T01:07:41.981495+0000","last_peered":"2024-01-27T01:07:41.981495+0000","last_clean":"2024-01-27T01:07:41.981495+0000","last_became_active":"2024-01-27T01:07:26.953090+0000","last_became_peered":"2024-01-27T01:07:26.953090+0000","last_unstale":"2024-01-27T01:07:41.981495+0000","last_undegraded":"2024-01-27T01:07:41.981495+0000","last_fullsized":"2024-01-27T01:07:41.981495+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:10.549778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.017058+0000","last_change":"2024-01-27T01:07:42.014257+0000","last_active":"2024-01-27T01:07:42.017058+0000","last_peered":"2024-01-27T01:07:42.017058+0000","last_clean":"2024-01-27T01:07:42.017058+0000","last_became_active":"2024-01-27T01:07:42.006612+0000","last_became_peered":"2024-01-27T01:07:42.006612+0000","last_unstale":"2024-01-27T01:07:42.017058+0000","last_undegraded":"2024-01-27T01:07:42.017058+0000","last_fullsized":"2024-01-27T01:07:42.017058+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:20.301704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979085+0000","last_change":"2024-01-27T01:07:27.750264+0000","last_active":"2024-01-27T01:07:41.979085+0000","last_peered":"2024-01-27T01:07:41.979085+0000","last_clean":"2024-01-27T01:07:41.979085+0000","last_became_active":"2024-01-27T01:07:27.749987+0000","last_became_peered":"2024-01-27T01:07:27.749987+0000","last_unstale":"2024-01-27T01:07:41.979085+0000","last_undegraded":"2024-01-27T01:07:41.979085+0000","last_fullsized":"2024-01-27T01:07:41.979085+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:36:19.339556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.994047+0000","last_change":"2024-01-27T01:07:26.942700+0000","last_active":"2024-01-27T01:07:41.994047+0000","last_peered":"2024-01-27T01:07:41.994047+0000","last_clean":"2024-01-27T01:07:41.994047+0000","last_became_active":"2024-01-27T01:07:26.942491+0000","last_became_peered":"2024-01-27T01:07:26.942491+0000","last_unstale":"2024-01-27T01:07:41.994047+0000","last_undegraded":"2024-01-27T01:07:41.994047+0000","last_fullsized":"2024-01-27T01:07:41.994047+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:03:40.104583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.993239+0000","last_change":"2024-01-27T01:07:41.989219+0000","last_active":"2024-01-27T01:07:41.993239+0000","last_peered":"2024-01-27T01:07:41.993239+0000","last_clean":"2024-01-27T01:07:41.993239+0000","last_became_active":"2024-01-27T01:07:41.988917+0000","last_became_peered":"2024-01-27T01:07:41.988917+0000","last_unstale":"2024-01-27T01:07:41.993239+0000","last_undegraded":"2024-01-27T01:07:41.993239+0000","last_fullsized":"2024-01-27T01:07:41.993239+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:45:19.572287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981584+0000","last_change":"2024-01-27T01:07:27.746804+0000","last_active":"2024-01-27T01:07:41.981584+0000","last_peered":"2024-01-27T01:07:41.981584+0000","last_clean":"2024-01-27T01:07:41.981584+0000","last_became_active":"2024-01-27T01:07:27.746110+0000","last_became_peered":"2024-01-27T01:07:27.746110+0000","last_unstale":"2024-01-27T01:07:41.981584+0000","last_undegraded":"2024-01-27T01:07:41.981584+0000","last_fullsized":"2024-01-27T01:07:41.981584+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:48:18.465014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.994604+0000","last_change":"2024-01-27T01:07:26.943462+0000","last_active":"2024-01-27T01:07:41.994604+0000","last_peered":"2024-01-27T01:07:41.994604+0000","last_clean":"2024-01-27T01:07:41.994604+0000","last_became_active":"2024-01-27T01:07:26.943193+0000","last_became_peered":"2024-01-27T01:07:26.943193+0000","last_unstale":"2024-01-27T01:07:41.994604+0000","last_undegraded":"2024-01-27T01:07:41.994604+0000","last_fullsized":"2024-01-27T01:07:41.994604+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:21:20.952966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.010918+0000","last_change":"2024-01-27T01:07:42.004836+0000","last_active":"2024-01-27T01:07:42.010918+0000","last_peered":"2024-01-27T01:07:42.010918+0000","last_clean":"2024-01-27T01:07:42.010918+0000","last_became_active":"2024-01-27T01:07:42.004530+0000","last_became_peered":"2024-01-27T01:07:42.004530+0000","last_unstale":"2024-01-27T01:07:42.010918+0000","last_undegraded":"2024-01-27T01:07:42.010918+0000","last_fullsized":"2024-01-27T01:07:42.010918+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:58:40.048139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.994218+0000","last_change":"2024-01-27T01:07:27.739197+0000","last_active":"2024-01-27T01:07:41.994218+0000","last_peered":"2024-01-27T01:07:41.994218+0000","last_clean":"2024-01-27T01:07:41.994218+0000","last_became_active":"2024-01-27T01:07:27.738911+0000","last_became_peered":"2024-01-27T01:07:27.738911+0000","last_unstale":"2024-01-27T01:07:41.994218+0000","last_undegraded":"2024-01-27T01:07:41.994218+0000","last_fullsized":"2024-01-27T01:07:41.994218+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:41:00.947931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995492+0000","last_change":"2024-01-27T01:07:26.949530+0000","last_active":"2024-01-27T01:07:41.995492+0000","last_peered":"2024-01-27T01:07:41.995492+0000","last_clean":"2024-01-27T01:07:41.995492+0000","last_became_active":"2024-01-27T01:07:26.949204+0000","last_became_peered":"2024-01-27T01:07:26.949204+0000","last_unstale":"2024-01-27T01:07:41.995492+0000","last_undegraded":"2024-01-27T01:07:41.995492+0000","last_fullsized":"2024-01-27T01:07:41.995492+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:16:54.275653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3f","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.013308+0000","last_change":"2024-01-27T01:07:42.009249+0000","last_active":"2024-01-27T01:07:42.013308+0000","last_peered":"2024-01-27T01:07:42.013308+0000","last_clean":"2024-01-27T01:07:42.013308+0000","last_became_active":"2024-01-27T01:07:42.008946+0000","last_became_peered":"2024-01-27T01:07:42.008946+0000","last_unstale":"2024-01-27T01:07:42.013308+0000","last_undegraded":"2024-01-27T01:07:42.013308+0000","last_fullsized":"2024-01-27T01:07:42.013308+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:48:32.501055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.997013+0000","last_change":"2024-01-27T01:07:27.744772+0000","last_active":"2024-01-27T01:07:41.997013+0000","last_peered":"2024-01-27T01:07:41.997013+0000","last_clean":"2024-01-27T01:07:41.997013+0000","last_became_active":"2024-01-27T01:07:27.744416+0000","last_became_peered":"2024-01-27T01:07:27.744416+0000","last_unstale":"2024-01-27T01:07:41.997013+0000","last_undegraded":"2024-01-27T01:07:41.997013+0000","last_fullsized":"2024-01-27T01:07:41.997013+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:59:39.155363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.981719+0000","last_change":"2024-01-27T01:07:26.947692+0000","last_active":"2024-01-27T01:07:41.981719+0000","last_peered":"2024-01-27T01:07:41.981719+0000","last_clean":"2024-01-27T01:07:41.981719+0000","last_became_active":"2024-01-27T01:07:26.947404+0000","last_became_peered":"2024-01-27T01:07:26.947404+0000","last_unstale":"2024-01-27T01:07:41.981719+0000","last_undegraded":"2024-01-27T01:07:41.981719+0000","last_fullsized":"2024-01-27T01:07:41.981719+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:12:55.245326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.017835+0000","last_change":"2024-01-27T01:07:42.015538+0000","last_active":"2024-01-27T01:07:42.017835+0000","last_peered":"2024-01-27T01:07:42.017835+0000","last_clean":"2024-01-27T01:07:42.017835+0000","last_became_active":"2024-01-27T01:07:42.011906+0000","last_became_peered":"2024-01-27T01:07:42.011906+0000","last_unstale":"2024-01-27T01:07:42.017835+0000","last_undegraded":"2024-01-27T01:07:42.017835+0000","last_fullsized":"2024-01-27T01:07:42.017835+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:49:46.679016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.983288+0000","last_change":"2024-01-27T01:07:27.749874+0000","last_active":"2024-01-27T01:07:41.983288+0000","last_peered":"2024-01-27T01:07:41.983288+0000","last_clean":"2024-01-27T01:07:41.983288+0000","last_became_active":"2024-01-27T01:07:27.749252+0000","last_became_peered":"2024-01-27T01:07:27.749252+0000","last_unstale":"2024-01-27T01:07:41.983288+0000","last_undegraded":"2024-01-27T01:07:41.983288+0000","last_fullsized":"2024-01-27T01:07:41.983288+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:18:53.285178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.979216+0000","last_change":"2024-01-27T01:07:26.947606+0000","last_active":"2024-01-27T01:07:41.979216+0000","last_peered":"2024-01-27T01:07:41.979216+0000","last_clean":"2024-01-27T01:07:41.979216+0000","last_became_active":"2024-01-27T01:07:26.947287+0000","last_became_peered":"2024-01-27T01:07:26.947287+0000","last_unstale":"2024-01-27T01:07:41.979216+0000","last_undegraded":"2024-01-27T01:07:41.979216+0000","last_fullsized":"2024-01-27T01:07:41.979216+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:34:14.649324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.007691+0000","last_change":"2024-01-27T01:07:42.004792+0000","last_active":"2024-01-27T01:07:42.007691+0000","last_peered":"2024-01-27T01:07:42.007691+0000","last_clean":"2024-01-27T01:07:42.007691+0000","last_became_active":"2024-01-27T01:07:42.004529+0000","last_became_peered":"2024-01-27T01:07:42.004529+0000","last_unstale":"2024-01-27T01:07:42.007691+0000","last_undegraded":"2024-01-27T01:07:42.007691+0000","last_fullsized":"2024-01-27T01:07:42.007691+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:04:01.316302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.997262+0000","last_change":"2024-01-27T01:07:27.749856+0000","last_active":"2024-01-27T01:07:41.997262+0000","last_peered":"2024-01-27T01:07:41.997262+0000","last_clean":"2024-01-27T01:07:41.997262+0000","last_became_active":"2024-01-27T01:07:27.749595+0000","last_became_peered":"2024-01-27T01:07:27.749595+0000","last_unstale":"2024-01-27T01:07:41.997262+0000","last_undegraded":"2024-01-27T01:07:41.997262+0000","last_fullsized":"2024-01-27T01:07:41.997262+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:25:38.146534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.995964+0000","last_change":"2024-01-27T01:07:26.950737+0000","last_active":"2024-01-27T01:07:41.995964+0000","last_peered":"2024-01-27T01:07:41.995964+0000","last_clean":"2024-01-27T01:07:41.995964+0000","last_became_active":"2024-01-27T01:07:26.950498+0000","last_became_peered":"2024-01-27T01:07:26.950498+0000","last_unstale":"2024-01-27T01:07:41.995964+0000","last_undegraded":"2024-01-27T01:07:41.995964+0000","last_fullsized":"2024-01-27T01:07:41.995964+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:24:42.558412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":9,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:42.020131+0000","last_change":"2024-01-27T01:07:42.017196+0000","last_active":"2024-01-27T01:07:42.020131+0000","last_peered":"2024-01-27T01:07:42.020131+0000","last_clean":"2024-01-27T01:07:42.020131+0000","last_became_active":"2024-01-27T01:07:42.012218+0000","last_became_peered":"2024-01-27T01:07:42.012218+0000","last_unstale":"2024-01-27T01:07:42.020131+0000","last_undegraded":"2024-01-27T01:07:42.020131+0000","last_fullsized":"2024-01-27T01:07:42.020131+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:40.959857+0000","last_clean_scrub_stamp":"2024-01-27T01:07:40.959857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:34:41.431166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.987124+0000","last_change":"2024-01-27T01:07:27.746007+0000","last_active":"2024-01-27T01:07:41.987124+0000","last_peered":"2024-01-27T01:07:41.987124+0000","last_clean":"2024-01-27T01:07:41.987124+0000","last_became_active":"2024-01-27T01:07:27.745722+0000","last_became_peered":"2024-01-27T01:07:27.745722+0000","last_unstale":"2024-01-27T01:07:41.987124+0000","last_undegraded":"2024-01-27T01:07:41.987124+0000","last_fullsized":"2024-01-27T01:07:41.987124+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:26.923816+0000","last_clean_scrub_stamp":"2024-01-27T01:07:26.923816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:09:23.112301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"242'1","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.986221+0000","last_change":"2024-01-27T01:07:26.951812+0000","last_active":"2024-01-27T01:07:41.986221+0000","last_peered":"2024-01-27T01:07:41.986221+0000","last_clean":"2024-01-27T01:07:41.986221+0000","last_became_active":"2024-01-27T01:07:26.951141+0000","last_became_peered":"2024-01-27T01:07:26.951141+0000","last_unstale":"2024-01-27T01:07:41.986221+0000","last_undegraded":"2024-01-27T01:07:41.986221+0000","last_fullsized":"2024-01-27T01:07:41.986221+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:07:25.916668+0000","last_clean_scrub_stamp":"2024-01-27T01:07:25.916668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:06:20.506111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"230'8772","reported_seq":9763,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-01-27T01:07:41.984186+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:07:41.984186+0000","last_peered":"2024-01-27T01:07:41.984186+0000","last_clean":"2024-01-27T01:07:41.984186+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:07:41.984186+0000","last_undegraded":"2024-01-27T01:07:41.984186+0000","last_fullsized":"2024-01-27T01:07:41.984186+0000","mapping_epoch":14,"log_start":"98'2229","ondisk_log_start":"98'2229","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2456,"num_read_kb":5220,"num_write":670,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":3538,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2456,"num_read_kb":5220,"num_write":670,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6543,"ondisk_log_size":6543,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576015,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300924,"kb_used_data":3636,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94070916,"statfs":{"total":96636764160,"available":96328617984,"internally_reserved":0,"allocated":3723264,"data_stored":2245329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.312},"max":{"1min":0.72499999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.63200000000000001,"15min":0.621},"min":{"1min":0.48099999999999998,"5min":0.39900000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":0.879,"15min":0.879},"last":0.72899999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70299999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.63800000000000001,"15min":0.63100000000000001},"min":{"1min":0.44500000000000001,"5min":0.42099999999999999,"15min":0.34300000000000003},"max":{"1min":0.73499999999999999,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.77400000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.61899999999999999,"15min":0.60999999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.72899999999999998,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.753},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.628,"15min":0.629},"min":{"1min":0.372,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.69499999999999995,"5min":0.872,"15min":0.88900000000000001},"last":0.85799999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.60899999999999999,"15min":0.61399999999999999},"min":{"1min":0.38500000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69099999999999995,"5min":0.876,"15min":0.876},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.65700000000000003,"15min":0.64200000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80700000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.80600000000000005}]},{"osd":4,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.503,"15min":0.48799999999999999},"min":{"1min":0.35199999999999998,"5min":0.314,"15min":0.29999999999999999},"max":{"1min":0.65500000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53700000000000003,"15min":0.51600000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.629,"5min":0.80300000000000005,"15min":0.86599999999999999},"last":0.63500000000000001}]},{"osd":5,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.55300000000000005,"15min":0.52500000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.80400000000000005},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57299999999999995,"15min":0.56399999999999995},"min":{"1min":0.41699999999999998,"5min":0.34799999999999998,"15min":0.34399999999999997},"max":{"1min":0.79200000000000004,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.66000000000000003}]},{"osd":6,"last update":"Sat Jan 27 01:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60499999999999998,"15min":0.58899999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.60499999999999998,"15min":0.58699999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.75,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.69999999999999996}]}]},{"osd":6,"up_from":13,"seq":55834576015,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308520,"kb_used_data":3040,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063320,"statfs":{"total":96636764160,"available":96320839680,"internally_reserved":0,"allocated":3112960,"data_stored":1646440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.59199999999999997,"15min":0.59899999999999998},"min":{"1min":0.40999999999999998,"5min":0.40799999999999997,"15min":0.36399999999999999},"max":{"1min":0.65400000000000003,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60199999999999998,"15min":0.61199999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.376},"max":{"1min":0.67600000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.73499999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.59199999999999997,"15min":0.60299999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.65600000000000003,"5min":0.80300000000000005,"15min":0.84499999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60199999999999998,"15min":0.61699999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81200000000000006,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.76400000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.58299999999999996,"15min":0.61099999999999999},"min":{"1min":0.39200000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70999999999999996,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69999999999999996,"5min":0.80400000000000005,"15min":0.878},"last":0.754}]},{"osd":3,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61399999999999999,"15min":0.623},"min":{"1min":0.379,"5min":0.378,"15min":0.378},"max":{"1min":0.72599999999999998,"5min":0.83699999999999997,"15min":0.90100000000000002},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64500000000000002,"15min":0.65400000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.754,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.877}]},{"osd":4,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51300000000000001,"15min":0.502},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.71399999999999997,"5min":0.77600000000000002,"15min":0.86599999999999999},"last":0.63},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.36599999999999999,"5min":0.33800000000000002,"15min":0.29799999999999999},"max":{"1min":0.69299999999999995,"5min":0.78500000000000003,"15min":0.88900000000000001},"last":0.81799999999999995}]},{"osd":5,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.69199999999999995,"5min":0.75900000000000001,"15min":0.79300000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.58299999999999996,"15min":0.57399999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.28899999999999998},"max":{"1min":0.84199999999999997,"5min":0.87,"15min":0.874},"last":0.71099999999999997}]},{"osd":7,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57799999999999996,"15min":0.57399999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82099999999999995,"5min":0.84899999999999998,"15min":0.92900000000000005},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58999999999999997,"15min":0.58899999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.34699999999999998},"max":{"1min":0.76500000000000001,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.79300000000000004}]}]},{"osd":1,"up_from":13,"seq":55834576015,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304420,"kb_used_data":3036,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067420,"statfs":{"total":96636764160,"available":96325038080,"internally_reserved":0,"allocated":3108864,"data_stored":1646418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.28499999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.48899999999999999,"5min":0.66100000000000003,"15min":0.69599999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.625,"5min":0.67800000000000005,"15min":0.76600000000000001},"last":0.308}]},{"osd":2,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.42699999999999999,"15min":0.435},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.26200000000000001},"max":{"1min":0.54900000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":0.48799999999999999},"min":{"1min":0.36699999999999999,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.58899999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.41599999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51800000000000002,"15min":0.53100000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66200000000000003,"5min":0.84799999999999998,"15min":1.2130000000000001},"last":0.441},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.5,"15min":0.51300000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.32300000000000001},"max":{"1min":0.65100000000000002,"5min":1.125,"15min":1.22},"last":0.38600000000000001}]},{"osd":4,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.66900000000000004,"15min":0.67000000000000004},"min":{"1min":0.41599999999999998,"5min":0.41099999999999998,"15min":0.39800000000000002},"max":{"1min":0.92100000000000004,"5min":1.242,"15min":1.242},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.68500000000000005,"15min":0.69199999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.35699999999999998},"max":{"1min":0.80500000000000005,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.495}]},{"osd":5,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.72099999999999997,"15min":0.70499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1,"5min":5.444,"15min":5.444},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71399999999999997,"15min":0.70599999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.435},"max":{"1min":0.97599999999999998,"5min":2.6669999999999998,"15min":2.6669999999999998},"last":0.57199999999999995}]},{"osd":6,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.73599999999999999,"15min":0.72099999999999997},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.47099999999999997},"max":{"1min":0.89700000000000002,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.80600000000000005,"15min":0.74199999999999999},"min":{"1min":0.52400000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0600000000000001,"5min":11.223000000000001,"15min":11.223000000000001},"last":0.502}]},{"osd":7,"last update":"Sat Jan 27 01:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.749,"15min":0.753},"min":{"1min":0.54100000000000004,"5min":0.52800000000000002,"15min":0.48999999999999999},"max":{"1min":1.0349999999999999,"5min":2.9660000000000002,"15min":2.9660000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.76900000000000002,"15min":0.76700000000000002},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.52000000000000002},"max":{"1min":0.95599999999999996,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.55400000000000005}]}]},{"osd":0,"up_from":13,"seq":55834576014,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300912,"kb_used_data":3624,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94070928,"statfs":{"total":96636764160,"available":96328630272,"internally_reserved":0,"allocated":3710976,"data_stored":2239500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.434,"15min":0.42499999999999999},"min":{"1min":0.28899999999999998,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.443,"5min":0.45100000000000001,"15min":0.46000000000000002},"min":{"1min":0.33200000000000002,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":0.64300000000000002,"5min":0.65600000000000003,"15min":0.73399999999999999},"last":0.33200000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.41899999999999998,"15min":0.433},"min":{"1min":0.27400000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.56499999999999995,"5min":0.66900000000000004,"15min":0.76800000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.47599999999999998,"15min":0.48599999999999999},"min":{"1min":0.32900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60299999999999998,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.378}]},{"osd":3,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.34599999999999997,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.83699999999999997},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.503,"15min":0.504},"min":{"1min":0.35899999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.65900000000000003,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.35899999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.67600000000000005,"15min":0.67200000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.42199999999999999},"max":{"1min":0.91200000000000003,"5min":0.91500000000000004,"15min":1.0720000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.65900000000000003,"15min":0.67600000000000005},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.90100000000000002,"5min":1.002,"15min":1.002},"last":0.59799999999999998}]},{"osd":5,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.68700000000000006,"15min":0.69799999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95799999999999996,"5min":1.0209999999999999,"15min":2.165},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69499999999999995,"15min":0.70999999999999996},"min":{"1min":0.50700000000000001,"5min":0.48799999999999999,"15min":0.45600000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.1399999999999999},"last":0.58099999999999996}]},{"osd":6,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68799999999999994,"15min":0.70999999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.71399999999999997,"15min":0.73699999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.88700000000000001,"5min":1.0109999999999999,"15min":1.425},"last":0.56000000000000005}]},{"osd":7,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.73099999999999998,"15min":0.752},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.99199999999999999,"5min":1.0169999999999999,"15min":1.1259999999999999},"last":0.495},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.755,"15min":0.76600000000000001},"min":{"1min":0.55000000000000004,"5min":0.51100000000000001,"15min":0.48499999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.1910000000000001},"last":0.61599999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576014,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312644,"kb_used_data":3068,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059196,"statfs":{"total":96636764160,"available":96316616704,"internally_reserved":0,"allocated":3141632,"data_stored":1656997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39600000000000002,"15min":0.40400000000000003},"min":{"1min":0.27200000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.497,"5min":0.746,"15min":0.746},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.441,"15min":0.443},"min":{"1min":0.34599999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.33000000000000002}]},{"osd":1,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44500000000000001,"15min":0.44600000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.50900000000000001,"15min":0.51900000000000002},"min":{"1min":0.40699999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.45800000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.49299999999999999,"15min":0.496},"min":{"1min":0.38700000000000001,"5min":0.34699999999999998,"15min":0.33200000000000002},"max":{"1min":0.65000000000000002,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.47799999999999998,"15min":0.48699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.63900000000000001,"5min":0.65500000000000003,"15min":0.75700000000000001},"last":0.38600000000000001}]},{"osd":4,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.77800000000000002,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.437},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.68100000000000005,"15min":0.69199999999999995},"min":{"1min":0.47899999999999998,"5min":0.47099999999999997,"15min":0.378},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.107},"last":0.55400000000000005}]},{"osd":5,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.69699999999999995,"15min":0.70199999999999996},"min":{"1min":0.47399999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.075,"5min":1.075,"15min":1.167},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.69199999999999995,"15min":0.70499999999999996},"min":{"1min":0.52300000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81899999999999995,"5min":0.97499999999999998,"15min":1.05},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.70099999999999996,"15min":0.70899999999999996},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.90200000000000002,"5min":0.97899999999999998,"15min":1.077},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.69599999999999995,"15min":0.71499999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.39200000000000002},"max":{"1min":0.85999999999999999,"5min":0.96699999999999997,"15min":1.135},"last":0.57699999999999996}]},{"osd":7,"last update":"Sat Jan 27 01:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73399999999999999,"15min":0.74399999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.198},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.755,"15min":0.76500000000000001},"min":{"1min":0.53600000000000003,"5min":0.53100000000000003,"15min":0.48199999999999998},"max":{"1min":0.875,"5min":0.96399999999999997,"15min":1.0880000000000001},"last":0.42799999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576013,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304452,"kb_used_data":3068,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067388,"statfs":{"total":96636764160,"available":96325005312,"internally_reserved":0,"allocated":3141632,"data_stored":1657729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.439,"15min":0.42199999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63400000000000001,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44,"15min":0.45400000000000001},"min":{"1min":0.30599999999999999,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.66400000000000003,"5min":0.66700000000000004,"15min":0.84699999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.61699999999999999,"15min":0.53200000000000003},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.28899999999999998},"max":{"1min":0.76100000000000001,"5min":15.391,"15min":15.391},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.52400000000000002,"15min":0.52300000000000002},"min":{"1min":0.35799999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.628,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.379}]},{"osd":2,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.53600000000000003,"15min":0.51300000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70299999999999996,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.50900000000000001},"min":{"1min":0.308,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67600000000000005,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":0.308}]},{"osd":4,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.72899999999999998,"15min":0.71499999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.439},"max":{"1min":0.89200000000000002,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.71499999999999997,"15min":0.70299999999999996},"min":{"1min":0.47499999999999998,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":0.96699999999999997,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.48899999999999999}]},{"osd":5,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.73099999999999998,"15min":0.70899999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.94499999999999995,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.5},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.72699999999999998,"15min":0.71499999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.435},"max":{"1min":0.872,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.51800000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.73799999999999999,"15min":0.71999999999999997},"min":{"1min":0.49199999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.96699999999999997,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92900000000000005,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.745,"15min":0.73199999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.91400000000000003,"5min":4.569,"15min":4.569},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.78900000000000003,"15min":0.77800000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97999999999999998,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.67700000000000005}]}]},{"osd":4,"up_from":13,"seq":55834576015,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316716,"kb_used_data":3044,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055124,"statfs":{"total":96636764160,"available":96312446976,"internally_reserved":0,"allocated":3117056,"data_stored":1647900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.436,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60199999999999998,"15min":0.61299999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.753,"5min":0.82199999999999995,"15min":0.85699999999999998},"last":0.35499999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60999999999999999,"15min":0.628},"min":{"1min":0.40699999999999997,"5min":0.39100000000000001,"15min":0.38800000000000001},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60999999999999999,"15min":0.623},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82299999999999995,"5min":0.874,"15min":0.874},"last":0.497}]},{"osd":2,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.84699999999999998,"5min":0.85599999999999998,"15min":0.88700000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.48999999999999999,"5min":0.42899999999999999,"15min":0.42799999999999999},"max":{"1min":0.79200000000000004,"5min":0.81499999999999995,"15min":0.89200000000000002},"last":0.51900000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60399999999999998,"15min":0.623},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.80800000000000005,"5min":0.83299999999999996,"15min":0.84199999999999997},"last":0.377},{"interface":"front","average":{"1min":0.628,"5min":0.621,"15min":0.629},"min":{"1min":0.42299999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.47099999999999997}]},{"osd":5,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47999999999999998,"15min":0.48399999999999999},"min":{"1min":0.32400000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.85899999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.503,"15min":0.505},"min":{"1min":0.36299999999999999,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.878},"last":0.44800000000000001}]},{"osd":6,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.50600000000000001,"15min":0.499},"min":{"1min":0.38300000000000001,"5min":0.35799999999999998,"15min":0.34999999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.88700000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.375,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.54600000000000004}]},{"osd":7,"last update":"Sat Jan 27 01:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56200000000000006,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.748,"5min":0.80400000000000005,"15min":0.90200000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.38600000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.92600000000000005},"last":0.53000000000000003}]}]},{"osd":5,"up_from":13,"seq":55834576015,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312620,"kb_used_data":3044,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059220,"statfs":{"total":96636764160,"available":96316641280,"internally_reserved":0,"allocated":3117056,"data_stored":1654331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.58799999999999997,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.27300000000000002},"max":{"1min":0.73999999999999999,"5min":0.82799999999999996,"15min":1.006},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.501,"5min":0.57099999999999995,"15min":0.59799999999999998},"min":{"1min":0.37,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73499999999999999,"5min":0.81499999999999995,"15min":1.1719999999999999},"last":0.54900000000000004}]},{"osd":1,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.61599999999999999,"15min":0.64300000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69699999999999995,"5min":0.89000000000000001,"15min":0.95899999999999996},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.65500000000000003,"15min":0.66400000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.377},"max":{"1min":0.77900000000000003,"5min":0.86899999999999999,"15min":1.212},"last":0.623}]},{"osd":2,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56200000000000006,"15min":0.58699999999999997},"min":{"1min":0.34699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70999999999999996,"5min":0.82099999999999995,"15min":1.631},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.60299999999999998,"15min":0.61299999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34000000000000002},"max":{"1min":0.67600000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.59999999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.373,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71099999999999997,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.61199999999999999,"15min":0.624},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.754,"5min":0.879,"15min":1.4690000000000001},"last":0.52900000000000003}]},{"osd":4,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.5,"15min":0.48699999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69199999999999995,"5min":0.90200000000000002,"15min":0.94399999999999995},"last":0.501},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.52300000000000002,"15min":0.51200000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66100000000000003,"5min":0.92500000000000004,"15min":0.93300000000000005},"last":0.58899999999999997}]},{"osd":6,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51100000000000001,"15min":0.504},"min":{"1min":0.312,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.629,"5min":0.96799999999999997,"15min":1.732},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56399999999999995,"15min":0.54700000000000004},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78300000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.63700000000000001}]},{"osd":7,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59099999999999997,"15min":0.57499999999999996},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.79700000000000004,"5min":0.95399999999999996,"15min":1.0740000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.621,"15min":0.61299999999999999},"min":{"1min":0.41299999999999998,"5min":0.40799999999999997,"15min":0.39900000000000002},"max":{"1min":0.76500000000000001,"5min":0.996,"15min":1.6919999999999999},"last":0.57999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:07:46.689 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-27T01:07:46.689 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:07:46.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.855+0000 7f741ee95700 1 Processor -- start 2024-01-27T01:07:46.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.855+0000 7f741ee95700 1 -- start start 2024-01-27T01:07:46.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.856+0000 7f741ee95700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f74180ccc20 0x7f7418068340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:46.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.856+0000 7f741ee95700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7418158020 con 0x7f74180b2590 2024-01-27T01:07:46.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.856+0000 7f741ee95700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7418158300 con 0x7f74180ccff0 2024-01-27T01:07:46.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.856+0000 7f741ee95700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74181585e0 con 0x7f74180ccc20 2024-01-27T01:07:46.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.856+0000 7f741cc31700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f74180ccc20 0x7f7418068340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:46.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.856+0000 7f741d432700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f74180b2590 0x7f741815b5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42532/0 (socket says 172.21.15.46:42532) 2024-01-27T01:07:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.856+0000 7f741d432700 1 -- 172.21.15.46:0/2256474248 learned_addr learned my addr 172.21.15.46:0/2256474248 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.857+0000 7f741cc31700 1 -- 172.21.15.46:0/2256474248 >> v1:172.21.15.106:6789/0 conn(0x7f74180ccff0 legacy=0x7f741806eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.857+0000 7f741cc31700 1 -- 172.21.15.46:0/2256474248 >> v1:172.21.15.46:6789/0 conn(0x7f74180b2590 legacy=0x7f741815b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.857+0000 7f741cc31700 1 -- 172.21.15.46:0/2256474248 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74181588c0 con 0x7f74180ccc20 2024-01-27T01:07:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.857+0000 7f74177fe700 1 -- 172.21.15.46:0/2256474248 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652026619 0 0) 0x7f7418158020 con 0x7f74180b2590 2024-01-27T01:07:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.857+0000 7f741cc31700 1 --2- 172.21.15.46:0/2256474248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74180ccc20 0x7f7418068340 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f74080098e0 tx=0x7f74080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30bd689783467796 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.857+0000 7f74177fe700 1 -- 172.21.15.46:0/2256474248 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7408009e80 con 0x7f74180ccc20 2024-01-27T01:07:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.857+0000 7f74177fe700 1 -- 172.21.15.46:0/2256474248 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f740800b860 con 0x7f74180ccc20 2024-01-27T01:07:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.857+0000 7f74177fe700 1 -- 172.21.15.46:0/2256474248 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740800bbe0 con 0x7f74180ccc20 2024-01-27T01:07:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.858+0000 7f741ee95700 1 -- 172.21.15.46:0/2256474248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74180ccc20 msgr2=0x7f7418068340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.858+0000 7f741ee95700 1 --2- 172.21.15.46:0/2256474248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74180ccc20 0x7f7418068340 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f74080098e0 tx=0x7f74080091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.858+0000 7f741ee95700 1 -- 172.21.15.46:0/2256474248 shutdown_connections 2024-01-27T01:07:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.858+0000 7f741ee95700 1 --2- 172.21.15.46:0/2256474248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74180ccc20 0x7f7418068340 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.858+0000 7f741ee95700 1 -- 172.21.15.46:0/2256474248 >> 172.21.15.46:0/2256474248 conn(0x7f741806b630 msgr2=0x7f74180aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:46.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.858+0000 7f741ee95700 1 -- 172.21.15.46:0/2256474248 shutdown_connections 2024-01-27T01:07:46.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.858+0000 7f741ee95700 1 -- 172.21.15.46:0/2256474248 wait complete. 2024-01-27T01:07:46.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.859+0000 7f741ee95700 1 Processor -- start 2024-01-27T01:07:46.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.859+0000 7f741ee95700 1 -- start start 2024-01-27T01:07:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.860+0000 7f741ee95700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74180b2590 0x7f74180b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.860+0000 7f741ee95700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74180ccc20 0x7f741813bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.860+0000 7f741ee95700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74180ccff0 0x7f741815b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.860+0000 7f741ee95700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741813c460 con 0x7f74180ccff0 2024-01-27T01:07:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.860+0000 7f741ee95700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741815ca10 con 0x7f74180ccc20 2024-01-27T01:07:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.860+0000 7f741ee95700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741815cea0 con 0x7f74180b2590 2024-01-27T01:07:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.860+0000 7f741d432700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74180ccff0 0x7f741815b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.860+0000 7f741d432700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74180ccff0 0x7f741815b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37806/0 (socket says 172.21.15.46:37806) 2024-01-27T01:07:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.860+0000 7f741d432700 1 -- 172.21.15.46:0/3866244309 learned_addr learned my addr 172.21.15.46:0/3866244309 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.860+0000 7f7417fff700 1 --2- 172.21.15.46:0/3866244309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74180ccc20 0x7f741813bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.860+0000 7f741cc31700 1 --2- 172.21.15.46:0/3866244309 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74180b2590 0x7f74180b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.860+0000 7f741d432700 1 -- 172.21.15.46:0/3866244309 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74180b2590 msgr2=0x7f74180b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.860+0000 7f741d432700 1 --2- 172.21.15.46:0/3866244309 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74180b2590 0x7f74180b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.860+0000 7f741d432700 1 -- 172.21.15.46:0/3866244309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74180ccc20 msgr2=0x7f741813bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.860+0000 7f741d432700 1 --2- 172.21.15.46:0/3866244309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74180ccc20 0x7f741813bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.860+0000 7f741d432700 1 -- 172.21.15.46:0/3866244309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7408009420 con 0x7f74180ccff0 2024-01-27T01:07:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.861+0000 7f741d432700 1 --2- 172.21.15.46:0/3866244309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74180ccff0 0x7f741815b5b0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f741000cc90 tx=0x7f7410001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.861+0000 7f7414ff9700 1 -- 172.21.15.46:0/3866244309 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741000cea0 con 0x7f74180ccff0 2024-01-27T01:07:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.861+0000 7f7414ff9700 1 -- 172.21.15.46:0/3866244309 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741000ab30 con 0x7f74180ccff0 2024-01-27T01:07:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.861+0000 7f741ee95700 1 -- 172.21.15.46:0/3866244309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f741815be00 con 0x7f74180ccff0 2024-01-27T01:07:46.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.861+0000 7f7414ff9700 1 -- 172.21.15.46:0/3866244309 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74100070f0 con 0x7f74180ccff0 2024-01-27T01:07:46.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.861+0000 7f741ee95700 1 -- 172.21.15.46:0/3866244309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f741815c3c0 con 0x7f74180ccff0 2024-01-27T01:07:46.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.862+0000 7f741ee95700 1 -- 172.21.15.46:0/3866244309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f741815cba0 con 0x7f74180ccff0 2024-01-27T01:07:46.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.863+0000 7f7414ff9700 1 -- 172.21.15.46:0/3866244309 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f74100074a0 con 0x7f74180ccff0 2024-01-27T01:07:46.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.863+0000 7f7414ff9700 1 --2- 172.21.15.46:0/3866244309 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f74040671a0 0x7f7404069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:46.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.863+0000 7f7414ff9700 1 -- 172.21.15.46:0/3866244309 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f7410085e40 con 0x7f74180ccff0 2024-01-27T01:07:46.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.863+0000 7f741cc31700 1 --2- 172.21.15.46:0/3866244309 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f74040671a0 0x7f7404069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:46.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.864+0000 7f741cc31700 1 --2- 172.21.15.46:0/3866244309 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f74040671a0 0x7f7404069650 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f74080093f0 tx=0x7f7408017000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:46.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:46.867+0000 7f7414ff9700 1 -- 172.21.15.46:0/3866244309 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7410052860 con 0x7f74180ccff0 2024-01-27T01:07:47.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.116+0000 7f741ee95700 1 -- 172.21.15.46:0/3866244309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f741804c710 con 0x7f74180ccff0 2024-01-27T01:07:47.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.117+0000 7f7414ff9700 1 -- 172.21.15.46:0/3866244309 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+5687 (secure 0 0 0) 0x7f7410055e80 con 0x7f74180ccff0 2024-01-27T01:07:47.117 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:47.117 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":138,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1114164873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1114164873},{"type":"v1","addr":"172.21.15.106:6834","nonce":1114164873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128},{"gid":9821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/702094428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":702094428},{"type":"v1","addr":"172.21.15.106:6835","nonce":702094428}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:07:27.375435+0000","modified":"2024-01-27T01:07:30.945729+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11248":{"gid":11248,"name":"a","rank":0,"incarnation":129,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/1087205220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1087205220},{"type":"v1","addr":"172.21.15.46:6834","nonce":1087205220}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:07:41.411084+0000","modified":"2024-01-27T01:07:45.002753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9827":{"gid":9827,"name":"c","rank":0,"incarnation":135,"state":"up:active","state_seq":6,"addr":"172.21.15.46:6835/408985798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":408985798},{"type":"v1","addr":"172.21.15.46:6835","nonce":408985798}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-27T01:07:47.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.119+0000 7f741ee95700 1 -- 172.21.15.46:0/3866244309 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f74040671a0 msgr2=0x7f7404069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:47.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.119+0000 7f741ee95700 1 --2- 172.21.15.46:0/3866244309 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f74040671a0 0x7f7404069650 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f74080093f0 tx=0x7f7408017000 comp rx=0 tx=0).stop 2024-01-27T01:07:47.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.119+0000 7f741ee95700 1 -- 172.21.15.46:0/3866244309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74180ccff0 msgr2=0x7f741815b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:47.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.119+0000 7f741ee95700 1 --2- 172.21.15.46:0/3866244309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74180ccff0 0x7f741815b5b0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f741000cc90 tx=0x7f7410001a20 comp rx=0 tx=0).stop 2024-01-27T01:07:47.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.119+0000 7f741ee95700 1 -- 172.21.15.46:0/3866244309 shutdown_connections 2024-01-27T01:07:47.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.119+0000 7f741ee95700 1 --2- 172.21.15.46:0/3866244309 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f74040671a0 0x7f7404069650 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.119+0000 7f741ee95700 1 --2- 172.21.15.46:0/3866244309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74180ccff0 0x7f741815b5b0 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.119+0000 7f741ee95700 1 --2- 172.21.15.46:0/3866244309 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74180b2590 0x7f74180b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.119+0000 7f741ee95700 1 --2- 172.21.15.46:0/3866244309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74180ccc20 0x7f741813bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.119+0000 7f741ee95700 1 -- 172.21.15.46:0/3866244309 >> 172.21.15.46:0/3866244309 conn(0x7f741806b630 msgr2=0x7f7418156b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:47.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.119+0000 7f741ee95700 1 -- 172.21.15.46:0/3866244309 shutdown_connections 2024-01-27T01:07:47.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.119+0000 7f741ee95700 1 -- 172.21.15.46:0/3866244309 wait complete. 2024-01-27T01:07:47.120 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 138 2024-01-27T01:07:47.535 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 138, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-27T01:07:41.411084+0000', 'modified': '2024-01-27T01:07:45.002753+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9827}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9827': {'gid': 9827, 'name': 'c', 'rank': 0, 'incarnation': 135, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.46:6835/408985798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 408985798}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 408985798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-27T01:07:47.535 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-27T01:07:47.535 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-27T01:07:47.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.678+0000 7fee09418700 1 Processor -- start 2024-01-27T01:07:47.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.678+0000 7fee09418700 1 -- start start 2024-01-27T01:07:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.679+0000 7fee09418700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fee040c7970 0x7fee04157880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.679+0000 7fee09418700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee040af1c0 0x7fee040ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.679+0000 7fee09418700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0406a570 con 0x7fee041574b0 2024-01-27T01:07:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.679+0000 7fee09418700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee040ca630 con 0x7fee040c7970 2024-01-27T01:07:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.679+0000 7fee09418700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee040682f0 con 0x7fee040af1c0 2024-01-27T01:07:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.679+0000 7fee027fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fee040c7970 0x7fee04157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.679+0000 7fee02ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fee041574b0 0x7fee040c7260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42536/0 (socket says 172.21.15.46:42536) 2024-01-27T01:07:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.679+0000 7fee02ffd700 1 -- 172.21.15.46:0/1861949730 learned_addr learned my addr 172.21.15.46:0/1861949730 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.680+0000 7fee037fe700 1 --2- 172.21.15.46:0/1861949730 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee040af1c0 0x7fee040ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.680+0000 7fee027fc700 1 -- 172.21.15.46:0/1861949730 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee040af1c0 msgr2=0x7fee040ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.680+0000 7fee027fc700 1 --2- 172.21.15.46:0/1861949730 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee040af1c0 0x7fee040ca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.680+0000 7fee027fc700 1 -- 172.21.15.46:0/1861949730 >> v1:172.21.15.46:6789/0 conn(0x7fee041574b0 legacy=0x7fee040c7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.680+0000 7fee027fc700 1 -- 172.21.15.46:0/1861949730 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee04068590 con 0x7fee040c7970 2024-01-27T01:07:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.680+0000 7fee037fe700 1 --2- 172.21.15.46:0/1861949730 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee040af1c0 0x7fee040ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.680+0000 7fee01ffb700 1 -- 172.21.15.46:0/1861949730 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045027602 0 0) 0x7fee0406a570 con 0x7fee041574b0 2024-01-27T01:07:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.680+0000 7fee027fc700 1 --2- 172.21.15.46:0/1861949730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee040c7970 0x7fee04157880 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fedf80098e0 tx=0x7fedf80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c89159a0dae4681d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.680+0000 7fee01ffb700 1 -- 172.21.15.46:0/1861949730 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8009e80 con 0x7fee040c7970 2024-01-27T01:07:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.680+0000 7fee01ffb700 1 -- 172.21.15.46:0/1861949730 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf800b860 con 0x7fee040c7970 2024-01-27T01:07:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.680+0000 7fee01ffb700 1 -- 172.21.15.46:0/1861949730 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf800bbe0 con 0x7fee040c7970 2024-01-27T01:07:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.681+0000 7fee09418700 1 -- 172.21.15.46:0/1861949730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee040c7970 msgr2=0x7fee04157880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.681+0000 7fee09418700 1 --2- 172.21.15.46:0/1861949730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee040c7970 0x7fee04157880 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fedf80098e0 tx=0x7fedf80091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.681+0000 7fee09418700 1 -- 172.21.15.46:0/1861949730 shutdown_connections 2024-01-27T01:07:47.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.681+0000 7fee09418700 1 --2- 172.21.15.46:0/1861949730 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee040af1c0 0x7fee040ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.681+0000 7fee09418700 1 --2- 172.21.15.46:0/1861949730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee040c7970 0x7fee04157880 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.681+0000 7fee09418700 1 -- 172.21.15.46:0/1861949730 >> 172.21.15.46:0/1861949730 conn(0x7fee040cd6f0 msgr2=0x7fee040c9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:47.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.681+0000 7fee09418700 1 -- 172.21.15.46:0/1861949730 shutdown_connections 2024-01-27T01:07:47.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.681+0000 7fee09418700 1 -- 172.21.15.46:0/1861949730 wait complete. 2024-01-27T01:07:47.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.683+0000 7fee09418700 1 Processor -- start 2024-01-27T01:07:47.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.683+0000 7fee09418700 1 -- start start 2024-01-27T01:07:47.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee09418700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee040af1c0 0x7fee04154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee09418700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee040c7970 0x7fee04154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee09418700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee041574b0 0x7fee040c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee09418700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee041561f0 con 0x7fee040c7970 2024-01-27T01:07:47.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee09418700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee040aecc0 con 0x7fee040af1c0 2024-01-27T01:07:47.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee09418700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee041555e0 con 0x7fee041574b0 2024-01-27T01:07:47.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee027fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee040c7970 0x7fee04154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee027fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee040c7970 0x7fee04154f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37810/0 (socket says 172.21.15.46:37810) 2024-01-27T01:07:47.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee027fc700 1 -- 172.21.15.46:0/3996307766 learned_addr learned my addr 172.21.15.46:0/3996307766 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:47.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee037fe700 1 --2- 172.21.15.46:0/3996307766 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee041574b0 0x7fee040c3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee02ffd700 1 --2- 172.21.15.46:0/3996307766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee040af1c0 0x7fee04154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee027fc700 1 -- 172.21.15.46:0/3996307766 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee041574b0 msgr2=0x7fee040c3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:47.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee027fc700 1 --2- 172.21.15.46:0/3996307766 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee041574b0 0x7fee040c3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee027fc700 1 -- 172.21.15.46:0/3996307766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee040af1c0 msgr2=0x7fee04154a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:47.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee027fc700 1 --2- 172.21.15.46:0/3996307766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee040af1c0 0x7fee04154a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee027fc700 1 -- 172.21.15.46:0/3996307766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf8009420 con 0x7fee040c7970 2024-01-27T01:07:47.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee037fe700 1 --2- 172.21.15.46:0/3996307766 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee041574b0 0x7fee040c3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:07:47.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee02ffd700 1 --2- 172.21.15.46:0/3996307766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee040af1c0 0x7fee04154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:07:47.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fee027fc700 1 --2- 172.21.15.46:0/3996307766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee040c7970 0x7fee04154f60 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fedf8013000 tx=0x7fedf8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:47.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fedeaffd700 1 -- 172.21.15.46:0/3996307766 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf800bbe0 con 0x7fee040c7970 2024-01-27T01:07:47.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.684+0000 7fedeaffd700 1 -- 172.21.15.46:0/3996307766 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf8000cc0 con 0x7fee040c7970 2024-01-27T01:07:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.685+0000 7fedeaffd700 1 -- 172.21.15.46:0/3996307766 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8011650 con 0x7fee040c7970 2024-01-27T01:07:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.685+0000 7fee09418700 1 -- 172.21.15.46:0/3996307766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee041558c0 con 0x7fee040c7970 2024-01-27T01:07:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.685+0000 7fee09418700 1 -- 172.21.15.46:0/3996307766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee04155e80 con 0x7fee040c7970 2024-01-27T01:07:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.686+0000 7fedeaffd700 1 -- 172.21.15.46:0/3996307766 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fedf8011a80 con 0x7fee040c7970 2024-01-27T01:07:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.686+0000 7fedeaffd700 1 --2- 172.21.15.46:0/3996307766 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fedf0067170 0x7fedf0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.687+0000 7fedeaffd700 1 -- 172.21.15.46:0/3996307766 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 7381+0+0 (secure 0 0 0) 0x7fedf80867c0 con 0x7fee040c7970 2024-01-27T01:07:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.687+0000 7fee02ffd700 1 --2- 172.21.15.46:0/3996307766 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fedf0067170 0x7fedf0069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.687+0000 7fee02ffd700 1 --2- 172.21.15.46:0/3996307766 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fedf0067170 0x7fedf0069620 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fedf40018d0 tx=0x7fedf4009040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.689+0000 7fee1271bb80 1 Processor -- start 2024-01-27T01:07:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.689+0000 7fee1271bb80 1 -- start start 2024-01-27T01:07:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fee1271bb80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d87010 0x5561e5d873e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fee1271bb80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5dcf360 0x5561e5db1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fee1271bb80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5d865d0 0x5561e5db1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fee1271bb80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e5c8a9f0 con 0x5561e5dcf360 2024-01-27T01:07:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fee1271bb80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e5cdc130 con 0x5561e5d87010 2024-01-27T01:07:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fee1271bb80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5561e5cd9930 con 0x5561e5d865d0 2024-01-27T01:07:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fee027fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5dcf360 0x5561e5db1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fee027fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5dcf360 0x5561e5db1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37814/0 (socket says 172.21.15.46:37814) 2024-01-27T01:07:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fee027fc700 1 -- 172.21.15.46:0/202671035 learned_addr learned my addr 172.21.15.46:0/202671035 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fee02ffd700 1 --2- 172.21.15.46:0/202671035 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d87010 0x5561e5d873e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fee037fe700 1 --2- 172.21.15.46:0/202671035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5d865d0 0x5561e5db1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fee027fc700 1 -- 172.21.15.46:0/202671035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5d865d0 msgr2=0x5561e5db1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fee027fc700 1 --2- 172.21.15.46:0/202671035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5d865d0 0x5561e5db1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fee027fc700 1 -- 172.21.15.46:0/202671035 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d87010 msgr2=0x5561e5d873e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fee027fc700 1 --2- 172.21.15.46:0/202671035 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d87010 0x5561e5d873e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fee027fc700 1 -- 172.21.15.46:0/202671035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561e5b14b10 con 0x5561e5dcf360 2024-01-27T01:07:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fee027fc700 1 --2- 172.21.15.46:0/202671035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5dcf360 0x5561e5db1840 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7fedf80842b0 tx=0x7fedf800bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d70f0446dd2213 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.690+0000 7fede3fff700 1 -- 172.21.15.46:0/202671035 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf805c0c0 con 0x5561e5dcf360 2024-01-27T01:07:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.691+0000 7fede3fff700 1 -- 172.21.15.46:0/202671035 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf8056870 con 0x5561e5dcf360 2024-01-27T01:07:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.691+0000 7fede3fff700 1 -- 172.21.15.46:0/202671035 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8056690 con 0x5561e5dcf360 2024-01-27T01:07:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.691+0000 7fee1271bb80 1 -- 172.21.15.46:0/202671035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5dcf360 msgr2=0x5561e5db1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.691+0000 7fee1271bb80 1 --2- 172.21.15.46:0/202671035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5dcf360 0x5561e5db1840 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7fedf80842b0 tx=0x7fedf800bf50 comp rx=0 tx=0).stop 2024-01-27T01:07:47.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.691+0000 7fee1271bb80 1 -- 172.21.15.46:0/202671035 shutdown_connections 2024-01-27T01:07:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.691+0000 7fee1271bb80 1 --2- 172.21.15.46:0/202671035 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5d865d0 0x5561e5db1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.691+0000 7fee1271bb80 1 --2- 172.21.15.46:0/202671035 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d87010 0x5561e5d873e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.691+0000 7fee1271bb80 1 --2- 172.21.15.46:0/202671035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5dcf360 0x5561e5db1840 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.691+0000 7fee1271bb80 1 -- 172.21.15.46:0/202671035 >> 172.21.15.46:0/202671035 conn(0x5561e5d86a70 msgr2=0x5561e5dcf820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.691+0000 7fee1271bb80 1 -- 172.21.15.46:0/202671035 shutdown_connections 2024-01-27T01:07:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.691+0000 7fee1271bb80 1 -- 172.21.15.46:0/202671035 wait complete. 2024-01-27T01:07:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.692+0000 7fee1271bb80 1 -- start start 2024-01-27T01:07:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.692+0000 7fee1271bb80 1 Processor -- start 2024-01-27T01:07:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.693+0000 7fee1271bb80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d865d0 0x5561e5daca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.693+0000 7fee1271bb80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5d87010 0x5561e5dceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.693+0000 7fee1271bb80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5dcf360 0x5561e5dde9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.693+0000 7fee1271bb80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e5cd5720 con 0x5561e5d87010 2024-01-27T01:07:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.693+0000 7fee1271bb80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e5c811b0 con 0x5561e5d865d0 2024-01-27T01:07:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.693+0000 7fee1271bb80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5561e5c80e50 con 0x5561e5dcf360 2024-01-27T01:07:47.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.693+0000 7fee1271bb80 10 client.0 _ll_register_callbacks cb 0x5561e5ae37b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-27T01:07:47.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.693+0000 7fee027fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5d87010 0x5561e5dceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.693+0000 7fee027fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5d87010 0x5561e5dceaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37820/0 (socket says 172.21.15.46:37820) 2024-01-27T01:07:47.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.693+0000 7fee027fc700 1 -- 172.21.15.46:0/480631828 learned_addr learned my addr 172.21.15.46:0/480631828 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:47.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.693+0000 7fee037fe700 1 --2- 172.21.15.46:0/480631828 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5dcf360 0x5561e5dde9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.693+0000 7fee02ffd700 1 --2- 172.21.15.46:0/480631828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d865d0 0x5561e5daca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.694+0000 7fee027fc700 1 -- 172.21.15.46:0/480631828 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5dcf360 msgr2=0x5561e5dde9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:47.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.694+0000 7fee027fc700 1 --2- 172.21.15.46:0/480631828 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5dcf360 0x5561e5dde9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.694+0000 7fee027fc700 1 -- 172.21.15.46:0/480631828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d865d0 msgr2=0x5561e5daca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:47.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.694+0000 7fee027fc700 1 --2- 172.21.15.46:0/480631828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d865d0 0x5561e5daca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.694+0000 7fee027fc700 1 -- 172.21.15.46:0/480631828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee04155e80 con 0x5561e5d87010 2024-01-27T01:07:47.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.694+0000 7fee027fc700 1 --2- 172.21.15.46:0/480631828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5d87010 0x5561e5dceaa0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7fedf800b3c0 tx=0x7fedf802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51963e5777562614 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:47.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.694+0000 7fede17fa700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-27T01:07:47.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.694+0000 7fede17fa700 1 -- 172.21.15.46:0/480631828 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8057430 con 0x5561e5d87010 2024-01-27T01:07:47.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.694+0000 7fee1271bb80 1 -- 172.21.15.46:0/480631828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5561e5d7a190 con 0x5561e5d87010 2024-01-27T01:07:47.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.694+0000 7fede17fa700 1 -- 172.21.15.46:0/480631828 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(247..247 src has 1..247) v4 ==== 7381+0+0 (secure 0 0 0) 0x7fedf8056d30 con 0x5561e5d87010 2024-01-27T01:07:47.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.694+0000 7fede17fa700 1 -- 172.21.15.46:0/480631828 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=138) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedf8059f80 con 0x5561e5d87010 2024-01-27T01:07:47.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.694+0000 7fee1271bb80 10 client.11943 fetch_fsmap learned FSMap version 138 2024-01-27T01:07:47.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.694+0000 7fee1271bb80 1 -- 172.21.15.46:0/480631828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=138,monmap=2+}) v3 -- 0x5561e5ca3290 con 0x5561e5d87010 2024-01-27T01:07:47.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.694+0000 7fede17fa700 1 -- 172.21.15.46:0/480631828 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5561e5d7a190 con 0x5561e5d87010 2024-01-27T01:07:47.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.694+0000 7fede17fa700 1 -- 172.21.15.46:0/480631828 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 138) v1 ==== 3653+0+0 (secure 0 0 0) 0x7fedf8084480 con 0x5561e5d87010 2024-01-27T01:07:47.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.695+0000 7fee1271bb80 10 client.11943 fetch_fsmap finished waiting for FSMap version 138 2024-01-27T01:07:47.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.695+0000 7fee1271bb80 10 client.11943 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-27T01:07:47.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.695+0000 7fee1271bb80 20 client.11943 populate_metadata read hostname 'smithi046' 2024-01-27T01:07:47.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.695+0000 7fee1271bb80 1 --2- 172.21.15.46:0/480631828 >> [v2:172.21.15.46:6833/408985798,v1:172.21.15.46:6835/408985798] conn(0x5561e5db3f10 0x5561e5db42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.695+0000 7fee1271bb80 4 client.11943 mds_command: new command op to 9827 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-27T01:07:47.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.695+0000 7fee1271bb80 1 -- 172.21.15.46:0/480631828 --> [v2:172.21.15.46:6833/408985798,v1:172.21.15.46:6835/408985798] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5561e5c77ee0 con 0x5561e5db3f10 2024-01-27T01:07:47.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.695+0000 7fee02ffd700 1 --2- 172.21.15.46:0/480631828 >> [v2:172.21.15.46:6833/408985798,v1:172.21.15.46:6835/408985798] conn(0x5561e5db3f10 0x5561e5db42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.695+0000 7fede17fa700 10 client.11943 ms_handle_connect on v2:172.21.15.46:6833/408985798 2024-01-27T01:07:47.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.695+0000 7fee02ffd700 1 --2- 172.21.15.46:0/480631828 >> [v2:172.21.15.46:6833/408985798,v1:172.21.15.46:6835/408985798] conn(0x5561e5db3f10 0x5561e5db42e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57aaf8ebd4ce18be server_cookie=f75096913a5bca61 in_seq=0 out_seq=0 2024-01-27T01:07:47.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.696+0000 7fede17fa700 1 -- 172.21.15.46:0/480631828 <== mds.0 v2:172.21.15.46:6833/408985798 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x5561e5c77ee0 con 0x5561e5db3f10 2024-01-27T01:07:47.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.696+0000 7fede17fa700 10 client.11943 handle_command_reply: tid=0 2024-01-27T01:07:47.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.696+0000 7fee1271bb80 1 client.11943 shutdown 2024-01-27T01:07:47.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.696+0000 7fee1271bb80 1 -- 172.21.15.46:0/480631828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5d87010 msgr2=0x5561e5dceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:47.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.696+0000 7fee1271bb80 1 --2- 172.21.15.46:0/480631828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5d87010 0x5561e5dceaa0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7fedf800b3c0 tx=0x7fedf802f000 comp rx=0 tx=0).stop 2024-01-27T01:07:47.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.696+0000 7fee1271bb80 1 -- 172.21.15.46:0/480631828 shutdown_connections 2024-01-27T01:07:47.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.696+0000 7fee1271bb80 1 --2- 172.21.15.46:0/480631828 >> [v2:172.21.15.46:6833/408985798,v1:172.21.15.46:6835/408985798] conn(0x5561e5db3f10 0x5561e5db42e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.696+0000 7fee1271bb80 1 --2- 172.21.15.46:0/480631828 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5dcf360 0x5561e5dde9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.696+0000 7fee1271bb80 1 --2- 172.21.15.46:0/480631828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d865d0 0x5561e5daca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.696+0000 7fede17fa700 0 client.11943 ms_handle_reset on v2:172.21.15.46:6833/408985798 2024-01-27T01:07:47.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.696+0000 7fee1271bb80 1 --2- 172.21.15.46:0/480631828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5d87010 0x5561e5dceaa0 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.696+0000 7fee1271bb80 1 -- 172.21.15.46:0/480631828 >> 172.21.15.46:0/480631828 conn(0x5561e5d86a70 msgr2=0x5561e5dae790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:47.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.696+0000 7fee1271bb80 1 -- 172.21.15.46:0/480631828 shutdown_connections 2024-01-27T01:07:47.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.696+0000 7fee1271bb80 1 -- 172.21.15.46:0/480631828 wait complete. 2024-01-27T01:07:47.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.696+0000 7fee1271bb80 20 client.11943 trim_cache size 0 max 16384 2024-01-27T01:07:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.714+0000 7fee1271bb80 1 Processor -- start 2024-01-27T01:07:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.714+0000 7fee1271bb80 1 -- start start 2024-01-27T01:07:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.715+0000 7fee1271bb80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d865d0 0x5561e5db15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.715+0000 7fee1271bb80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5d87010 0x5561e5d80fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.715+0000 7fee1271bb80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5dcf360 0x5561e5dcf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.715+0000 7fee1271bb80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e5bcde40 con 0x5561e5d87010 2024-01-27T01:07:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.715+0000 7fee1271bb80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e5d64020 con 0x5561e5d865d0 2024-01-27T01:07:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.715+0000 7fee1271bb80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5561e5dac4e0 con 0x5561e5dcf360 2024-01-27T01:07:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.715+0000 7fee027fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5d87010 0x5561e5d80fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.715+0000 7fee027fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5d87010 0x5561e5d80fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37826/0 (socket says 172.21.15.46:37826) 2024-01-27T01:07:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.715+0000 7fee027fc700 1 -- 172.21.15.46:0/3692589246 learned_addr learned my addr 172.21.15.46:0/3692589246 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.716+0000 7fee037fe700 1 --2- 172.21.15.46:0/3692589246 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5dcf360 0x5561e5dcf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.716+0000 7fee02ffd700 1 --2- 172.21.15.46:0/3692589246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d865d0 0x5561e5db15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.716+0000 7fee027fc700 1 -- 172.21.15.46:0/3692589246 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5dcf360 msgr2=0x5561e5dcf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.716+0000 7fee027fc700 1 --2- 172.21.15.46:0/3692589246 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5dcf360 0x5561e5dcf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.716+0000 7fee027fc700 1 -- 172.21.15.46:0/3692589246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d865d0 msgr2=0x5561e5db15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.716+0000 7fee027fc700 1 --2- 172.21.15.46:0/3692589246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d865d0 0x5561e5db15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.716+0000 7fee027fc700 1 -- 172.21.15.46:0/3692589246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561e5d87e60 con 0x5561e5d87010 2024-01-27T01:07:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.716+0000 7fee02ffd700 1 --2- 172.21.15.46:0/3692589246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d865d0 0x5561e5db15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.716+0000 7fee027fc700 1 --2- 172.21.15.46:0/3692589246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5d87010 0x5561e5d80fa0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fedf8056a30 tx=0x7fedf800b070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b5715b2774a0ea server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.716+0000 7fede8ff9700 1 -- 172.21.15.46:0/3692589246 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8057430 con 0x5561e5d87010 2024-01-27T01:07:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.716+0000 7fede8ff9700 1 -- 172.21.15.46:0/3692589246 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf8056690 con 0x5561e5d87010 2024-01-27T01:07:47.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.716+0000 7fede8ff9700 1 -- 172.21.15.46:0/3692589246 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8056d30 con 0x5561e5d87010 2024-01-27T01:07:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.717+0000 7fee1271bb80 1 -- 172.21.15.46:0/3692589246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5d87010 msgr2=0x5561e5d80fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.717+0000 7fee1271bb80 1 --2- 172.21.15.46:0/3692589246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5d87010 0x5561e5d80fa0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fedf8056a30 tx=0x7fedf800b070 comp rx=0 tx=0).stop 2024-01-27T01:07:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.717+0000 7fee1271bb80 1 -- 172.21.15.46:0/3692589246 shutdown_connections 2024-01-27T01:07:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.717+0000 7fee1271bb80 1 --2- 172.21.15.46:0/3692589246 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5dcf360 0x5561e5dcf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.717+0000 7fee1271bb80 1 --2- 172.21.15.46:0/3692589246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d865d0 0x5561e5db15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.717+0000 7fee1271bb80 1 --2- 172.21.15.46:0/3692589246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5d87010 0x5561e5d80fa0 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.717+0000 7fee1271bb80 1 -- 172.21.15.46:0/3692589246 >> 172.21.15.46:0/3692589246 conn(0x5561e5d86a70 msgr2=0x5561e5d81d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.717+0000 7fee1271bb80 1 -- 172.21.15.46:0/3692589246 shutdown_connections 2024-01-27T01:07:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.717+0000 7fee1271bb80 1 -- 172.21.15.46:0/3692589246 wait complete. 2024-01-27T01:07:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.717+0000 7fee1271bb80 1 -- start start 2024-01-27T01:07:47.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.718+0000 7fee1271bb80 1 Processor -- start 2024-01-27T01:07:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.719+0000 7fee1271bb80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d865d0 0x5561e5dce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.719+0000 7fee1271bb80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5d87010 0x5561e5e2f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.719+0000 7fee1271bb80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5dcf360 0x5561e5dab900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.719+0000 7fee1271bb80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e5e2fe40 con 0x5561e5dcf360 2024-01-27T01:07:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.719+0000 7fee1271bb80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561e5ddd870 con 0x5561e5d865d0 2024-01-27T01:07:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.719+0000 7fee1271bb80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x5561e5dced00 con 0x5561e5d87010 2024-01-27T01:07:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.719+0000 7fee1271bb80 10 client.0 _ll_register_callbacks cb 0x5561e5ae37b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-27T01:07:47.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.719+0000 7fee037fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5dcf360 0x5561e5dab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.719+0000 7fee037fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5dcf360 0x5561e5dab900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37838/0 (socket says 172.21.15.46:37838) 2024-01-27T01:07:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.719+0000 7fee037fe700 1 -- 172.21.15.46:0/1631946427 learned_addr learned my addr 172.21.15.46:0/1631946427 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.719+0000 7fee027fc700 1 --2- 172.21.15.46:0/1631946427 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5d87010 0x5561e5e2f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.719+0000 7fee02ffd700 1 --2- 172.21.15.46:0/1631946427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d865d0 0x5561e5dce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.720+0000 7fee037fe700 1 -- 172.21.15.46:0/1631946427 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5d87010 msgr2=0x5561e5e2f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.720+0000 7fee037fe700 1 --2- 172.21.15.46:0/1631946427 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5d87010 0x5561e5e2f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.720+0000 7fee037fe700 1 -- 172.21.15.46:0/1631946427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d865d0 msgr2=0x5561e5dce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.720+0000 7fee037fe700 1 --2- 172.21.15.46:0/1631946427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d865d0 0x5561e5dce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.720+0000 7fee037fe700 1 -- 172.21.15.46:0/1631946427 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5561e5ca3290 con 0x5561e5dcf360 2024-01-27T01:07:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.720+0000 7fee037fe700 1 --2- 172.21.15.46:0/1631946427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5dcf360 0x5561e5dab900 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fee040c57f0 tx=0x7fedfc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebedd613674fa047 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.720+0000 7fede17fa700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-27T01:07:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.720+0000 7fede17fa700 1 -- 172.21.15.46:0/1631946427 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc00e040 con 0x5561e5dcf360 2024-01-27T01:07:47.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.720+0000 7fee1271bb80 1 -- 172.21.15.46:0/1631946427 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fee040b0360 con 0x5561e5dcf360 2024-01-27T01:07:47.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.720+0000 7fede17fa700 1 -- 172.21.15.46:0/1631946427 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(247..247 src has 1..247) v4 ==== 7381+0+0 (secure 0 0 0) 0x7fedfc015670 con 0x5561e5dcf360 2024-01-27T01:07:47.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.720+0000 7fede17fa700 1 -- 172.21.15.46:0/1631946427 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=138) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedfc015a40 con 0x5561e5dcf360 2024-01-27T01:07:47.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.720+0000 7fee1271bb80 10 client.11949 fetch_fsmap learned FSMap version 138 2024-01-27T01:07:47.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.720+0000 7fee1271bb80 1 -- 172.21.15.46:0/1631946427 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=138,monmap=2+}) v3 -- 0x5561e5d84970 con 0x5561e5dcf360 2024-01-27T01:07:47.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.720+0000 7fede17fa700 1 -- 172.21.15.46:0/1631946427 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee040b0360 con 0x5561e5dcf360 2024-01-27T01:07:47.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.720+0000 7fede17fa700 1 -- 172.21.15.46:0/1631946427 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 138) v1 ==== 3653+0+0 (secure 0 0 0) 0x7fedfc015ba0 con 0x5561e5dcf360 2024-01-27T01:07:47.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.720+0000 7fee1271bb80 10 client.11949 fetch_fsmap finished waiting for FSMap version 138 2024-01-27T01:07:47.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.720+0000 7fee1271bb80 10 client.11949 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-27T01:07:47.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.721+0000 7fee1271bb80 20 client.11949 populate_metadata read hostname 'smithi046' 2024-01-27T01:07:47.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.721+0000 7fee1271bb80 1 --2- 172.21.15.46:0/1631946427 >> [v2:172.21.15.46:6833/408985798,v1:172.21.15.46:6835/408985798] conn(0x5561e5dccf40 0x5561e5dcd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.721+0000 7fee1271bb80 4 client.11949 mds_command: new command op to 9827 tid=0[{"prefix": "status"}] 2024-01-27T01:07:47.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.721+0000 7fee1271bb80 1 -- 172.21.15.46:0/1631946427 --> [v2:172.21.15.46:6833/408985798,v1:172.21.15.46:6835/408985798] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fedf8056870 con 0x5561e5dccf40 2024-01-27T01:07:47.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.721+0000 7fee02ffd700 1 --2- 172.21.15.46:0/1631946427 >> [v2:172.21.15.46:6833/408985798,v1:172.21.15.46:6835/408985798] conn(0x5561e5dccf40 0x5561e5dcd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.721+0000 7fede17fa700 10 client.11949 ms_handle_connect on v2:172.21.15.46:6833/408985798 2024-01-27T01:07:47.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.721+0000 7fee02ffd700 1 --2- 172.21.15.46:0/1631946427 >> [v2:172.21.15.46:6833/408985798,v1:172.21.15.46:6835/408985798] conn(0x5561e5dccf40 0x5561e5dcd310 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6533e3f6ceac4ff server_cookie=94d6e204fbc8071 in_seq=0 out_seq=0 2024-01-27T01:07:47.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.721+0000 7fede17fa700 1 -- 172.21.15.46:0/1631946427 <== mds.0 v2:172.21.15.46:6833/408985798 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x7fedf8056870 con 0x5561e5dccf40 2024-01-27T01:07:47.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.721+0000 7fede17fa700 10 client.11949 handle_command_reply: tid=0 2024-01-27T01:07:47.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.721+0000 7fee1271bb80 1 client.11949 shutdown 2024-01-27T01:07:47.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.721+0000 7fee1271bb80 1 -- 172.21.15.46:0/1631946427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5dcf360 msgr2=0x5561e5dab900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:47.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.721+0000 7fee1271bb80 1 --2- 172.21.15.46:0/1631946427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5dcf360 0x5561e5dab900 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fee040c57f0 tx=0x7fedfc00b040 comp rx=0 tx=0).stop 2024-01-27T01:07:47.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.721+0000 7fee1271bb80 1 -- 172.21.15.46:0/1631946427 shutdown_connections 2024-01-27T01:07:47.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.721+0000 7fee1271bb80 1 --2- 172.21.15.46:0/1631946427 >> [v2:172.21.15.46:6833/408985798,v1:172.21.15.46:6835/408985798] conn(0x5561e5dccf40 0x5561e5dcd310 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee1271bb80 1 --2- 172.21.15.46:0/1631946427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561e5dcf360 0x5561e5dab900 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee1271bb80 1 --2- 172.21.15.46:0/1631946427 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5561e5d865d0 0x5561e5dce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee1271bb80 1 --2- 172.21.15.46:0/1631946427 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5561e5d87010 0x5561e5e2f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fede17fa700 0 client.11949 ms_handle_reset on v2:172.21.15.46:6833/408985798 2024-01-27T01:07:47.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee1271bb80 1 -- 172.21.15.46:0/1631946427 >> 172.21.15.46:0/1631946427 conn(0x5561e5d86a70 msgr2=0x5561e5db2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:47.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee1271bb80 1 -- 172.21.15.46:0/1631946427 shutdown_connections 2024-01-27T01:07:47.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee1271bb80 1 -- 172.21.15.46:0/1631946427 wait complete. 2024-01-27T01:07:47.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee1271bb80 20 client.11949 trim_cache size 0 max 16384 2024-01-27T01:07:47.728 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:07:47.728 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-27T01:07:47.728 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-27T01:07:47.729 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 9827, 2024-01-27T01:07:47.729 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-27T01:07:47.729 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-27T01:07:47.729 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "backup_fs", 2024-01-27T01:07:47.729 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 5.7445663869999999, 2024-01-27T01:07:47.729 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 138, 2024-01-27T01:07:47.729 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 247, 2024-01-27T01:07:47.729 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 247, 2024-01-27T01:07:47.730 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 24.695144572 2024-01-27T01:07:47.730 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:07:47.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee09418700 1 -- 172.21.15.46:0/3996307766 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fedf0067170 msgr2=0x7fedf0069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:47.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee09418700 1 --2- 172.21.15.46:0/3996307766 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fedf0067170 0x7fedf0069620 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fedf40018d0 tx=0x7fedf4009040 comp rx=0 tx=0).stop 2024-01-27T01:07:47.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee09418700 1 -- 172.21.15.46:0/3996307766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee040c7970 msgr2=0x7fee04154f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:47.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee09418700 1 --2- 172.21.15.46:0/3996307766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee040c7970 0x7fee04154f60 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fedf8013000 tx=0x7fedf8017000 comp rx=0 tx=0).stop 2024-01-27T01:07:47.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee09418700 1 -- 172.21.15.46:0/3996307766 shutdown_connections 2024-01-27T01:07:47.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee09418700 1 --2- 172.21.15.46:0/3996307766 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fedf0067170 0x7fedf0069620 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee09418700 1 --2- 172.21.15.46:0/3996307766 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee041574b0 0x7fee040c3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee09418700 1 --2- 172.21.15.46:0/3996307766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee040af1c0 0x7fee04154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee09418700 1 --2- 172.21.15.46:0/3996307766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee040c7970 0x7fee04154f60 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee09418700 1 -- 172.21.15.46:0/3996307766 >> 172.21.15.46:0/3996307766 conn(0x7fee040cd6f0 msgr2=0x7fee0406aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:47.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee09418700 1 -- 172.21.15.46:0/3996307766 shutdown_connections 2024-01-27T01:07:47.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.722+0000 7fee09418700 1 -- 172.21.15.46:0/3996307766 wait complete. 2024-01-27T01:07:47.736 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-27T01:07:47.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.869+0000 7f569f7d2700 1 Processor -- start 2024-01-27T01:07:47.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.869+0000 7f569f7d2700 1 -- start start 2024-01-27T01:07:47.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.870+0000 7f569f7d2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f56980b2b40 0x7f56980ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.870+0000 7f569f7d2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56980aedd0 0x7f56980cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.870+0000 7f569f7d2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569815a0b0 con 0x7f56980ae960 2024-01-27T01:07:47.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.870+0000 7f569f7d2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5698099fc0 con 0x7f56980aedd0 2024-01-27T01:07:47.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.870+0000 7f569f7d2700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56980ae4d0 con 0x7f56980b2b40 2024-01-27T01:07:47.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.870+0000 7f569dd6f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56980aedd0 0x7f56980cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.870+0000 7f569dd6f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56980aedd0 0x7f56980cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37840/0 (socket says 172.21.15.46:37840) 2024-01-27T01:07:47.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.870+0000 7f569dd6f700 1 -- 172.21.15.46:0/436855508 learned_addr learned my addr 172.21.15.46:0/436855508 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:47.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.871+0000 7f569d56e700 1 --2- 172.21.15.46:0/436855508 >> v2:172.21.15.106:3300/0 conn(0x7f56980b2b40 0x7f56980ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.871+0000 7f569dd6f700 1 -- 172.21.15.46:0/436855508 >> v2:172.21.15.106:3300/0 conn(0x7f56980b2b40 msgr2=0x7f56980ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:47.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.871+0000 7f569dd6f700 1 --2- 172.21.15.46:0/436855508 >> v2:172.21.15.106:3300/0 conn(0x7f56980b2b40 0x7f56980ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.871+0000 7f569dd6f700 1 -- 172.21.15.46:0/436855508 >> v1:172.21.15.106:6789/0 conn(0x7f56980ae960 legacy=0x7f56980ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:47.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.871+0000 7f569dd6f700 1 -- 172.21.15.46:0/436855508 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56980ad8c0 con 0x7f56980aedd0 2024-01-27T01:07:47.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.871+0000 7f569d56e700 1 --2- 172.21.15.46:0/436855508 >> v2:172.21.15.106:3300/0 conn(0x7f56980b2b40 0x7f56980ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:47.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.871+0000 7f569dd6f700 1 --2- 172.21.15.46:0/436855508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56980aedd0 0x7f56980cb5e0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f5698088850 tx=0x7f5690009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbe7e366ba68e2d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:47.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.871+0000 7f568ffff700 1 -- 172.21.15.46:0/436855508 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569000e070 con 0x7f56980aedd0 2024-01-27T01:07:47.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.871+0000 7f568ffff700 1 -- 172.21.15.46:0/436855508 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569000ba50 con 0x7f56980aedd0 2024-01-27T01:07:47.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.871+0000 7f568ffff700 1 -- 172.21.15.46:0/436855508 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56900092e0 con 0x7f56980aedd0 2024-01-27T01:07:47.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.872+0000 7f569f7d2700 1 -- 172.21.15.46:0/436855508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56980aedd0 msgr2=0x7f56980cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:47.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.872+0000 7f569f7d2700 1 --2- 172.21.15.46:0/436855508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56980aedd0 0x7f56980cb5e0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f5698088850 tx=0x7f5690009840 comp rx=0 tx=0).stop 2024-01-27T01:07:47.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.872+0000 7f569f7d2700 1 -- 172.21.15.46:0/436855508 shutdown_connections 2024-01-27T01:07:47.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.872+0000 7f569f7d2700 1 --2- 172.21.15.46:0/436855508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56980aedd0 0x7f56980cb5e0 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.872+0000 7f569f7d2700 1 --2- 172.21.15.46:0/436855508 >> v2:172.21.15.106:3300/0 conn(0x7f56980b2b40 0x7f56980ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.872+0000 7f569f7d2700 1 -- 172.21.15.46:0/436855508 >> 172.21.15.46:0/436855508 conn(0x7f5698099a30 msgr2=0x7f569806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:47.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.872+0000 7f569f7d2700 1 -- 172.21.15.46:0/436855508 shutdown_connections 2024-01-27T01:07:47.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.872+0000 7f569f7d2700 1 -- 172.21.15.46:0/436855508 wait complete. 2024-01-27T01:07:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.873+0000 7f569f7d2700 1 Processor -- start 2024-01-27T01:07:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.873+0000 7f569f7d2700 1 -- start start 2024-01-27T01:07:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.874+0000 7f569f7d2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56980ae960 0x7f56980c7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.874+0000 7f569f7d2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56980b2b40 0x7f56980c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.874+0000 7f569f7d2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56980c9c30 0x7f5698147080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.874+0000 7f569f7d2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569816aa60 con 0x7f56980b2b40 2024-01-27T01:07:47.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.874+0000 7f569f7d2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56980ca0d0 con 0x7f56980ae960 2024-01-27T01:07:47.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.874+0000 7f569f7d2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56980ca3b0 con 0x7f56980c9c30 2024-01-27T01:07:47.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.874+0000 7f569cd6d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56980b2b40 0x7f56980c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.874+0000 7f569cd6d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56980b2b40 0x7f56980c96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37856/0 (socket says 172.21.15.46:37856) 2024-01-27T01:07:47.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.874+0000 7f569cd6d700 1 -- 172.21.15.46:0/636815684 learned_addr learned my addr 172.21.15.46:0/636815684 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:47.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.874+0000 7f569d56e700 1 --2- 172.21.15.46:0/636815684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56980ae960 0x7f56980c7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.875+0000 7f569dd6f700 1 --2- 172.21.15.46:0/636815684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56980c9c30 0x7f5698147080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.875+0000 7f569cd6d700 1 -- 172.21.15.46:0/636815684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56980c9c30 msgr2=0x7f5698147080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:47.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.875+0000 7f569cd6d700 1 --2- 172.21.15.46:0/636815684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56980c9c30 0x7f5698147080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.875+0000 7f569cd6d700 1 -- 172.21.15.46:0/636815684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56980ae960 msgr2=0x7f56980c7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:47.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.875+0000 7f569cd6d700 1 --2- 172.21.15.46:0/636815684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56980ae960 0x7f56980c7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:47.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.875+0000 7f569cd6d700 1 -- 172.21.15.46:0/636815684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5690009ae0 con 0x7f56980b2b40 2024-01-27T01:07:47.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.875+0000 7f569d56e700 1 --2- 172.21.15.46:0/636815684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56980ae960 0x7f56980c7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:07:47.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.875+0000 7f569cd6d700 1 --2- 172.21.15.46:0/636815684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56980b2b40 0x7f56980c96f0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f568800c4e0 tx=0x7f568800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:47.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.875+0000 7f568d7fa700 1 -- 172.21.15.46:0/636815684 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5688011590 con 0x7f56980b2b40 2024-01-27T01:07:47.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.875+0000 7f568d7fa700 1 -- 172.21.15.46:0/636815684 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5688011710 con 0x7f56980b2b40 2024-01-27T01:07:47.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.875+0000 7f569f7d2700 1 -- 172.21.15.46:0/636815684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56980ca690 con 0x7f56980b2b40 2024-01-27T01:07:47.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.875+0000 7f568d7fa700 1 -- 172.21.15.46:0/636815684 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5688011ad0 con 0x7f56980b2b40 2024-01-27T01:07:47.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.875+0000 7f569f7d2700 1 -- 172.21.15.46:0/636815684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56980b47c0 con 0x7f56980b2b40 2024-01-27T01:07:47.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.876+0000 7f569f7d2700 1 -- 172.21.15.46:0/636815684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56980b49e0 con 0x7f56980b2b40 2024-01-27T01:07:47.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.877+0000 7f568d7fa700 1 -- 172.21.15.46:0/636815684 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f568801b3b0 con 0x7f56980b2b40 2024-01-27T01:07:47.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.877+0000 7f568d7fa700 1 --2- 172.21.15.46:0/636815684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f567c067170 0x7f567c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:47.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.877+0000 7f568d7fa700 1 -- 172.21.15.46:0/636815684 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f5688084450 con 0x7f56980b2b40 2024-01-27T01:07:47.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.877+0000 7f569d56e700 1 --2- 172.21.15.46:0/636815684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f567c067170 0x7f567c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:47.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.878+0000 7f569d56e700 1 --2- 172.21.15.46:0/636815684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f567c067170 0x7f567c069620 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5694006ea0 tx=0x7f5694008040 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:47.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:47.882+0000 7f568d7fa700 1 -- 172.21.15.46:0/636815684 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5688050da0 con 0x7f56980b2b40 2024-01-27T01:07:48.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.115+0000 7f569f7d2700 1 -- 172.21.15.46:0/636815684 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f569804c710 con 0x7f567c067170 2024-01-27T01:07:48.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.117+0000 7f568d7fa700 1 -- 172.21.15.46:0/636815684 <== mgr.9590 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7f569804c710 con 0x7f567c067170 2024-01-27T01:07:48.117 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-27T01:07:48.117 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:48.117 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephfs" 2024-01-27T01:07:48.117 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:07:48.117 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:07:48.117 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "backup_fs" 2024-01-27T01:07:48.117 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:07:48.117 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-27T01:07:48.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.119+0000 7f569f7d2700 1 -- 172.21.15.46:0/636815684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f567c067170 msgr2=0x7f567c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:48.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.119+0000 7f569f7d2700 1 --2- 172.21.15.46:0/636815684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f567c067170 0x7f567c069620 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5694006ea0 tx=0x7f5694008040 comp rx=0 tx=0).stop 2024-01-27T01:07:48.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.119+0000 7f569f7d2700 1 -- 172.21.15.46:0/636815684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56980b2b40 msgr2=0x7f56980c96f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:48.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.119+0000 7f569f7d2700 1 --2- 172.21.15.46:0/636815684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56980b2b40 0x7f56980c96f0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f568800c4e0 tx=0x7f568800c8a0 comp rx=0 tx=0).stop 2024-01-27T01:07:48.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.119+0000 7f569f7d2700 1 -- 172.21.15.46:0/636815684 shutdown_connections 2024-01-27T01:07:48.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.119+0000 7f569f7d2700 1 --2- 172.21.15.46:0/636815684 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f567c067170 0x7f567c069620 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:48.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.119+0000 7f569f7d2700 1 --2- 172.21.15.46:0/636815684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56980c9c30 0x7f5698147080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:48.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.119+0000 7f569f7d2700 1 --2- 172.21.15.46:0/636815684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56980ae960 0x7f56980c7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:48.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.119+0000 7f569f7d2700 1 --2- 172.21.15.46:0/636815684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56980b2b40 0x7f56980c96f0 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:48.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.119+0000 7f569f7d2700 1 -- 172.21.15.46:0/636815684 >> 172.21.15.46:0/636815684 conn(0x7f5698099a30 msgr2=0x7f56980d13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:48.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.119+0000 7f569f7d2700 1 -- 172.21.15.46:0/636815684 shutdown_connections 2024-01-27T01:07:48.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.119+0000 7f569f7d2700 1 -- 172.21.15.46:0/636815684 wait complete. 2024-01-27T01:07:48.133 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-27T01:07:48.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.266+0000 7f15c0b21700 1 Processor -- start 2024-01-27T01:07:48.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.266+0000 7f15c0b21700 1 -- start start 2024-01-27T01:07:48.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.267+0000 7f15c0b21700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f15bc068b00 0x7f15bc068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:48.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.267+0000 7f15c0b21700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15bc06a660 0x7f15bc159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:48.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.267+0000 7f15c0b21700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15bc0a9a80 con 0x7f15bc06add0 2024-01-27T01:07:48.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.267+0000 7f15c0b21700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15bc06aab0 con 0x7f15bc068b00 2024-01-27T01:07:48.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.267+0000 7f15c0b21700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc15a3e0 con 0x7f15bc06a660 2024-01-27T01:07:48.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.267+0000 7f15ba59c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f15bc068b00 0x7f15bc068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:48.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.267+0000 7f15b9d9b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15bc06a660 0x7f15bc159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:48.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.268+0000 7f15bad9d700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f15bc06add0 0x7f15bc15d4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:45578/0 (socket says 172.21.15.46:45578) 2024-01-27T01:07:48.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.268+0000 7f15ba59c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f15bc068b00 0x7f15bc068ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:35172/0 (socket says 172.21.15.46:35172) 2024-01-27T01:07:48.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.268+0000 7f15bad9d700 1 -- 172.21.15.46:0/1883447627 learned_addr learned my addr 172.21.15.46:0/1883447627 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:48.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.268+0000 7f15ba59c700 1 -- 172.21.15.46:0/1883447627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15bc06a660 msgr2=0x7f15bc159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:48.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.268+0000 7f15ba59c700 1 --2- 172.21.15.46:0/1883447627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15bc06a660 0x7f15bc159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:48.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.268+0000 7f15ba59c700 1 -- 172.21.15.46:0/1883447627 >> v1:172.21.15.106:6789/0 conn(0x7f15bc06add0 legacy=0x7f15bc15d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:48.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.268+0000 7f15ba59c700 1 -- 172.21.15.46:0/1883447627 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15bc15a680 con 0x7f15bc068b00 2024-01-27T01:07:48.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.268+0000 7f15b959a700 1 -- 172.21.15.46:0/1883447627 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123339342 0 0) 0x7f15bc0a9a80 con 0x7f15bc06add0 2024-01-27T01:07:48.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.268+0000 7f15b9d9b700 1 --2- 172.21.15.46:0/1883447627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15bc06a660 0x7f15bc159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:48.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.268+0000 7f15ba59c700 1 --2- 172.21.15.46:0/1883447627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15bc068b00 0x7f15bc068ed0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0098d0 tx=0x7f15ac009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cda60876e856bad5 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:48.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.268+0000 7f15b959a700 1 -- 172.21.15.46:0/1883447627 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac005c10 con 0x7f15bc068b00 2024-01-27T01:07:48.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.268+0000 7f15b959a700 1 -- 172.21.15.46:0/1883447627 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15ac005d70 con 0x7f15bc068b00 2024-01-27T01:07:48.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.268+0000 7f15b959a700 1 -- 172.21.15.46:0/1883447627 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac006280 con 0x7f15bc068b00 2024-01-27T01:07:48.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.269+0000 7f15c0b21700 1 -- 172.21.15.46:0/1883447627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15bc068b00 msgr2=0x7f15bc068ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:48.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.269+0000 7f15c0b21700 1 --2- 172.21.15.46:0/1883447627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15bc068b00 0x7f15bc068ed0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0098d0 tx=0x7f15ac009120 comp rx=0 tx=0).stop 2024-01-27T01:07:48.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.269+0000 7f15c0b21700 1 -- 172.21.15.46:0/1883447627 shutdown_connections 2024-01-27T01:07:48.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.269+0000 7f15c0b21700 1 --2- 172.21.15.46:0/1883447627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15bc068b00 0x7f15bc068ed0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:48.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.269+0000 7f15c0b21700 1 --2- 172.21.15.46:0/1883447627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15bc06a660 0x7f15bc159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:48.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.269+0000 7f15c0b21700 1 -- 172.21.15.46:0/1883447627 >> 172.21.15.46:0/1883447627 conn(0x7f15bc0ca100 msgr2=0x7f15bc06e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:48.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.269+0000 7f15c0b21700 1 -- 172.21.15.46:0/1883447627 shutdown_connections 2024-01-27T01:07:48.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.269+0000 7f15c0b21700 1 -- 172.21.15.46:0/1883447627 wait complete. 2024-01-27T01:07:48.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.270+0000 7f15c0b21700 1 Processor -- start 2024-01-27T01:07:48.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.270+0000 7f15c0b21700 1 -- start start 2024-01-27T01:07:48.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.271+0000 7f15c0b21700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15bc068b00 0x7f15bc0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:48.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.271+0000 7f15c0b21700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15bc06a660 0x7f15bc0ada90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:48.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.271+0000 7f15c0b21700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15bc06add0 0x7f15bc16d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:48.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.271+0000 7f15c0b21700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc0b0e70 con 0x7f15bc06add0 2024-01-27T01:07:48.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.271+0000 7f15c0b21700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc0b0260 con 0x7f15bc068b00 2024-01-27T01:07:48.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.271+0000 7f15c0b21700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc0b0540 con 0x7f15bc06a660 2024-01-27T01:07:48.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.271+0000 7f15b9d9b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15bc06a660 0x7f15bc0ada90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:48.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.271+0000 7f15bad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15bc06add0 0x7f15bc16d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.271+0000 7f15ba59c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15bc068b00 0x7f15bc0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.271+0000 7f15b9d9b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15bc06a660 0x7f15bc0ada90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:41390/0 (socket says 172.21.15.46:41390) 2024-01-27T01:07:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.272+0000 7f15bad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15bc06add0 0x7f15bc16d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37866/0 (socket says 172.21.15.46:37866) 2024-01-27T01:07:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.272+0000 7f15b9d9b700 1 -- 172.21.15.46:0/984258878 learned_addr learned my addr 172.21.15.46:0/984258878 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.272+0000 7f15bad9d700 1 -- 172.21.15.46:0/984258878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15bc06a660 msgr2=0x7f15bc0ada90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.272+0000 7f15bad9d700 1 --2- 172.21.15.46:0/984258878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15bc06a660 0x7f15bc0ada90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.272+0000 7f15bad9d700 1 -- 172.21.15.46:0/984258878 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15bc068b00 msgr2=0x7f15bc0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.272+0000 7f15bad9d700 1 --2- 172.21.15.46:0/984258878 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15bc068b00 0x7f15bc0c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.272+0000 7f15bad9d700 1 -- 172.21.15.46:0/984258878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a4009420 con 0x7f15bc06add0 2024-01-27T01:07:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.272+0000 7f15ba59c700 1 --2- 172.21.15.46:0/984258878 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15bc068b00 0x7f15bc0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.272+0000 7f15bad9d700 1 --2- 172.21.15.46:0/984258878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15bc06add0 0x7f15bc16d590 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f15b400cbe0 tx=0x7f15b40019a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.272+0000 7f15b27fc700 1 -- 172.21.15.46:0/984258878 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b400a8b0 con 0x7f15bc06add0 2024-01-27T01:07:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.272+0000 7f15b27fc700 1 -- 172.21.15.46:0/984258878 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b400aa30 con 0x7f15bc06add0 2024-01-27T01:07:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.272+0000 7f15c0b21700 1 -- 172.21.15.46:0/984258878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ac0093c0 con 0x7f15bc06add0 2024-01-27T01:07:48.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.272+0000 7f15b27fc700 1 -- 172.21.15.46:0/984258878 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b400adf0 con 0x7f15bc06add0 2024-01-27T01:07:48.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.272+0000 7f15c0b21700 1 -- 172.21.15.46:0/984258878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15bc0b0b00 con 0x7f15bc06add0 2024-01-27T01:07:48.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.273+0000 7f15c0b21700 1 -- 172.21.15.46:0/984258878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15bc04c710 con 0x7f15bc06add0 2024-01-27T01:07:48.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.274+0000 7f15b27fc700 1 -- 172.21.15.46:0/984258878 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 136) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f15b4007300 con 0x7f15bc06add0 2024-01-27T01:07:48.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.274+0000 7f15b27fc700 1 --2- 172.21.15.46:0/984258878 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f159c067170 0x7f159c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:48.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.275+0000 7f15b27fc700 1 -- 172.21.15.46:0/984258878 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f15b4085ee0 con 0x7f15bc06add0 2024-01-27T01:07:48.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.275+0000 7f15ba59c700 1 --2- 172.21.15.46:0/984258878 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f159c067170 0x7f159c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:48.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.275+0000 7f15ba59c700 1 --2- 172.21.15.46:0/984258878 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f159c067170 0x7f159c069620 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f15ac009160 tx=0x7f15ac016000 comp rx=0 tx=0).ready entity=mgr.9590 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:48.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.278+0000 7f15b27fc700 1 -- 172.21.15.46:0/984258878 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f15b40528c0 con 0x7f15bc06add0 2024-01-27T01:07:48.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:48.544+0000 7f15c0b21700 1 -- 172.21.15.46:0/984258878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f15bc0b0fb0 con 0x7f15bc06add0 2024-01-27T01:07:49.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.028+0000 7f15b27fc700 1 -- 172.21.15.46:0/984258878 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 137) v1 ==== 84963+0+0 (secure 0 0 0) 0x7f15b4052c90 con 0x7f15bc06add0 2024-01-27T01:07:49.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.030+0000 7f15b27fc700 1 -- 172.21.15.46:0/984258878 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v137) v1 ==== 92+0+0 (secure 0 0 0) 0x7f15b40528c0 con 0x7f15bc06add0 2024-01-27T01:07:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.032+0000 7f15c0b21700 1 -- 172.21.15.46:0/984258878 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f159c067170 msgr2=0x7f159c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.032+0000 7f15c0b21700 1 --2- 172.21.15.46:0/984258878 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f159c067170 0x7f159c069620 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f15ac009160 tx=0x7f15ac016000 comp rx=0 tx=0).stop 2024-01-27T01:07:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.032+0000 7f15c0b21700 1 -- 172.21.15.46:0/984258878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15bc06add0 msgr2=0x7f15bc16d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.032+0000 7f15c0b21700 1 --2- 172.21.15.46:0/984258878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15bc06add0 0x7f15bc16d590 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f15b400cbe0 tx=0x7f15b40019a0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.033+0000 7f15c0b21700 1 -- 172.21.15.46:0/984258878 shutdown_connections 2024-01-27T01:07:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.033+0000 7f15c0b21700 1 --2- 172.21.15.46:0/984258878 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f159c067170 0x7f159c069620 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.033+0000 7f15c0b21700 1 --2- 172.21.15.46:0/984258878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15bc06add0 0x7f15bc16d590 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.033+0000 7f15c0b21700 1 --2- 172.21.15.46:0/984258878 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15bc068b00 0x7f15bc0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.033+0000 7f15c0b21700 1 --2- 172.21.15.46:0/984258878 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15bc06a660 0x7f15bc0ada90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.033+0000 7f15c0b21700 1 -- 172.21.15.46:0/984258878 >> 172.21.15.46:0/984258878 conn(0x7f15bc0ca100 msgr2=0x7f15bc0aebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:49.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.033+0000 7f15c0b21700 1 -- 172.21.15.46:0/984258878 shutdown_connections 2024-01-27T01:07:49.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.033+0000 7f15c0b21700 1 -- 172.21.15.46:0/984258878 wait complete. 2024-01-27T01:07:49.049 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-27T01:07:49.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.188+0000 7f5ed77de700 1 Processor -- start 2024-01-27T01:07:49.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.188+0000 7f5ed77de700 1 -- start start 2024-01-27T01:07:49.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.189+0000 7f5ed77de700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5ed0159cb0 0x7f5ed00cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.189+0000 7f5ed77de700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5ed00cf050 0x7f5ed00af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.189+0000 7f5ed77de700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed00cc100 con 0x7f5ed00cf4a0 2024-01-27T01:07:49.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.189+0000 7f5ed77de700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed00cb850 con 0x7f5ed0159cb0 2024-01-27T01:07:49.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.189+0000 7f5ed77de700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed00cbaf0 con 0x7f5ed00cf050 2024-01-27T01:07:49.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.189+0000 7f5ed557a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5ed0159cb0 0x7f5ed00cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:49.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.189+0000 7f5ed557a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5ed0159cb0 0x7f5ed00cc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37886/0 (socket says 172.21.15.46:37886) 2024-01-27T01:07:49.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.189+0000 7f5ed557a700 1 -- 172.21.15.46:0/1523519969 learned_addr learned my addr 172.21.15.46:0/1523519969 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:49.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.189+0000 7f5ed4d79700 1 --2- 172.21.15.46:0/1523519969 >> v2:172.21.15.106:3300/0 conn(0x7f5ed00cf050 0x7f5ed00af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:49.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.190+0000 7f5ed557a700 1 -- 172.21.15.46:0/1523519969 >> v2:172.21.15.106:3300/0 conn(0x7f5ed00cf050 msgr2=0x7f5ed00af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:49.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.190+0000 7f5ed557a700 1 --2- 172.21.15.46:0/1523519969 >> v2:172.21.15.106:3300/0 conn(0x7f5ed00cf050 0x7f5ed00af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.190+0000 7f5ed557a700 1 -- 172.21.15.46:0/1523519969 >> v1:172.21.15.106:6789/0 conn(0x7f5ed00cf4a0 legacy=0x7f5ed00c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:49.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.190+0000 7f5ed557a700 1 -- 172.21.15.46:0/1523519969 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed00cfc80 con 0x7f5ed0159cb0 2024-01-27T01:07:49.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.190+0000 7f5ed557a700 1 --2- 172.21.15.46:0/1523519969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ed0159cb0 0x7f5ed00cc590 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00887e0 tx=0x7f5ec0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84dfac82de1e1b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:49.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.190+0000 7f5ecffff700 1 -- 172.21.15.46:0/1523519969 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec000e070 con 0x7f5ed0159cb0 2024-01-27T01:07:49.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.190+0000 7f5ecffff700 1 -- 172.21.15.46:0/1523519969 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec000ba20 con 0x7f5ed0159cb0 2024-01-27T01:07:49.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.190+0000 7f5ecffff700 1 -- 172.21.15.46:0/1523519969 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec00092e0 con 0x7f5ed0159cb0 2024-01-27T01:07:49.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.191+0000 7f5ed77de700 1 -- 172.21.15.46:0/1523519969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ed0159cb0 msgr2=0x7f5ed00cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:49.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.191+0000 7f5ed77de700 1 --2- 172.21.15.46:0/1523519969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ed0159cb0 0x7f5ed00cc590 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00887e0 tx=0x7f5ec0009840 comp rx=0 tx=0).stop 2024-01-27T01:07:49.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.191+0000 7f5ed77de700 1 -- 172.21.15.46:0/1523519969 shutdown_connections 2024-01-27T01:07:49.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.191+0000 7f5ed77de700 1 --2- 172.21.15.46:0/1523519969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ed0159cb0 0x7f5ed00cc590 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.191+0000 7f5ed77de700 1 --2- 172.21.15.46:0/1523519969 >> v2:172.21.15.106:3300/0 conn(0x7f5ed00cf050 0x7f5ed00af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.191+0000 7f5ed77de700 1 -- 172.21.15.46:0/1523519969 >> 172.21.15.46:0/1523519969 conn(0x7f5ed0087f60 msgr2=0x7f5ed006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:49.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.191+0000 7f5ed77de700 1 -- 172.21.15.46:0/1523519969 shutdown_connections 2024-01-27T01:07:49.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.191+0000 7f5ed77de700 1 -- 172.21.15.46:0/1523519969 wait complete. 2024-01-27T01:07:49.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.193+0000 7f5ed77de700 1 Processor -- start 2024-01-27T01:07:49.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.193+0000 7f5ed77de700 1 -- start start 2024-01-27T01:07:49.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.194+0000 7f5ed77de700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ed00cf050 0x7f5ed0155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.194+0000 7f5ed77de700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ed00cf4a0 0x7f5ed00c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.194+0000 7f5ed77de700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ed0159cb0 0x7f5ed00ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.194+0000 7f5ed77de700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed0156700 con 0x7f5ed00cf050 2024-01-27T01:07:49.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.194+0000 7f5ed77de700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed015c250 con 0x7f5ed00cf4a0 2024-01-27T01:07:49.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.194+0000 7f5ed77de700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed0155d70 con 0x7f5ed0159cb0 2024-01-27T01:07:49.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.194+0000 7f5ed557a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ed00cf050 0x7f5ed0155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:49.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.194+0000 7f5ed557a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ed00cf050 0x7f5ed0155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37896/0 (socket says 172.21.15.46:37896) 2024-01-27T01:07:49.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.194+0000 7f5ed557a700 1 -- 172.21.15.46:0/1530580761 learned_addr learned my addr 172.21.15.46:0/1530580761 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:49.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.194+0000 7f5ed4d79700 1 --2- 172.21.15.46:0/1530580761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ed00cf4a0 0x7f5ed00c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:49.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.194+0000 7f5ed5d7b700 1 --2- 172.21.15.46:0/1530580761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ed0159cb0 0x7f5ed00ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:49.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.194+0000 7f5ed557a700 1 -- 172.21.15.46:0/1530580761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ed0159cb0 msgr2=0x7f5ed00ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:49.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.194+0000 7f5ed557a700 1 --2- 172.21.15.46:0/1530580761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ed0159cb0 0x7f5ed00ca010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.194+0000 7f5ed557a700 1 -- 172.21.15.46:0/1530580761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ed00cf4a0 msgr2=0x7f5ed00c9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:49.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.194+0000 7f5ed557a700 1 --2- 172.21.15.46:0/1530580761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ed00cf4a0 0x7f5ed00c9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.194+0000 7f5ed557a700 1 -- 172.21.15.46:0/1530580761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec0009ae0 con 0x7f5ed00cf050 2024-01-27T01:07:49.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.195+0000 7f5ed557a700 1 --2- 172.21.15.46:0/1530580761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ed00cf050 0x7f5ed0155830 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0012000 tx=0x7f5ec0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:49.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.195+0000 7f5ecd7fa700 1 -- 172.21.15.46:0/1530580761 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec00078f0 con 0x7f5ed00cf050 2024-01-27T01:07:49.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.195+0000 7f5ecd7fa700 1 -- 172.21.15.46:0/1530580761 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec0007a50 con 0x7f5ed00cf050 2024-01-27T01:07:49.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.195+0000 7f5ed77de700 1 -- 172.21.15.46:0/1530580761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed01560f0 con 0x7f5ed00cf050 2024-01-27T01:07:49.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.195+0000 7f5ecd7fa700 1 -- 172.21.15.46:0/1530580761 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0007e10 con 0x7f5ed00cf050 2024-01-27T01:07:49.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.195+0000 7f5ed77de700 1 -- 172.21.15.46:0/1530580761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed015bf70 con 0x7f5ed00cf050 2024-01-27T01:07:49.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.196+0000 7f5ed77de700 1 -- 172.21.15.46:0/1530580761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed004c710 con 0x7f5ed00cf050 2024-01-27T01:07:49.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.196+0000 7f5ecd7fa700 1 -- 172.21.15.46:0/1530580761 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 137) v1 ==== 84963+0+0 (secure 0 0 0) 0x7f5ec0003040 con 0x7f5ed00cf050 2024-01-27T01:07:49.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.197+0000 7f5ecd7fa700 1 --2- 172.21.15.46:0/1530580761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5ebc0671c0 0x7f5ebc069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.197+0000 7f5ed4d79700 1 -- 172.21.15.46:0/1530580761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5ebc0671c0 msgr2=0x7f5ebc069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:49.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.197+0000 7f5ecd7fa700 1 -- 172.21.15.46:0/1530580761 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f5ec00865b0 con 0x7f5ed00cf050 2024-01-27T01:07:49.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.197+0000 7f5ed4d79700 1 --2- 172.21.15.46:0/1530580761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5ebc0671c0 0x7f5ebc069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:49.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.202+0000 7f5ecd7fa700 1 -- 172.21.15.46:0/1530580761 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5ec0052df0 con 0x7f5ed00cf050 2024-01-27T01:07:49.361 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:49.361+0000 7ff61bf941c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:07:49.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.398+0000 7f5ed4d79700 1 -- 172.21.15.46:0/1530580761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5ebc0671c0 msgr2=0x7f5ebc069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:49.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.398+0000 7f5ed4d79700 1 --2- 172.21.15.46:0/1530580761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5ebc0671c0 0x7f5ebc069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.411+0000 7f5ed77de700 1 -- 172.21.15.46:0/1530580761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f5ed00cfa00 con 0x7f5ed00cf050 2024-01-27T01:07:49.411 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:49.411+0000 7fe9cc7211c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:07:49.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.416+0000 7f5ecd7fa700 1 -- 172.21.15.46:0/1530580761 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v83)=0 v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5ec0056410 con 0x7f5ed00cf050 2024-01-27T01:07:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.418+0000 7f5ed77de700 1 -- 172.21.15.46:0/1530580761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5ebc0671c0 msgr2=0x7f5ebc069670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.418+0000 7f5ed77de700 1 --2- 172.21.15.46:0/1530580761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5ebc0671c0 0x7f5ebc069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.418+0000 7f5ed77de700 1 -- 172.21.15.46:0/1530580761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ed00cf050 msgr2=0x7f5ed0155830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.418+0000 7f5ed77de700 1 --2- 172.21.15.46:0/1530580761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ed00cf050 0x7f5ed0155830 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0012000 tx=0x7f5ec0018000 comp rx=0 tx=0).stop 2024-01-27T01:07:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.418+0000 7f5ed77de700 1 -- 172.21.15.46:0/1530580761 shutdown_connections 2024-01-27T01:07:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.418+0000 7f5ed77de700 1 --2- 172.21.15.46:0/1530580761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5ebc0671c0 0x7f5ebc069670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.418+0000 7f5ed77de700 1 --2- 172.21.15.46:0/1530580761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ed00cf050 0x7f5ed0155830 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.418+0000 7f5ed77de700 1 --2- 172.21.15.46:0/1530580761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ed0159cb0 0x7f5ed00ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.418+0000 7f5ed77de700 1 --2- 172.21.15.46:0/1530580761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ed00cf4a0 0x7f5ed00c9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.418+0000 7f5ed77de700 1 -- 172.21.15.46:0/1530580761 >> 172.21.15.46:0/1530580761 conn(0x7f5ed0087f60 msgr2=0x7f5ed0068ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.418+0000 7f5ed77de700 1 -- 172.21.15.46:0/1530580761 shutdown_connections 2024-01-27T01:07:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.418+0000 7f5ed77de700 1 -- 172.21.15.46:0/1530580761 wait complete. 2024-01-27T01:07:49.432 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-27T01:07:49.437 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:49.438+0000 7ff61bf941c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:07:49.503 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:49.503+0000 7fe9cc7211c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:07:49.508 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:49.509+0000 7ff61bf941c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:07:49.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.569+0000 7f2030a1c700 1 Processor -- start 2024-01-27T01:07:49.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.569+0000 7f2030a1c700 1 -- start start 2024-01-27T01:07:49.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.570+0000 7f2030a1c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f202c0c8da0 0x7f202c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.570+0000 7f2030a1c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f202c140af0 0x7f202c140f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.570+0000 7f2030a1c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f202c0caf30 0x7f202c0cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.570+0000 7f2030a1c700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f202c0cbbf0 con 0x7f202c0c8da0 2024-01-27T01:07:49.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.570+0000 7f2030a1c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f202c0ccc00 con 0x7f202c0caf30 2024-01-27T01:07:49.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.570+0000 7f2030a1c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f202c0cc310 con 0x7f202c140af0 2024-01-27T01:07:49.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.571+0000 7f202a59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f202c0c8da0 0x7f202c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:49.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.571+0000 7f202a59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f202c0c8da0 0x7f202c0c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37902/0 (socket says 172.21.15.46:37902) 2024-01-27T01:07:49.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.571+0000 7f202a59c700 1 -- 172.21.15.46:0/3963616054 learned_addr learned my addr 172.21.15.46:0/3963616054 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:49.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.571+0000 7f2029d9b700 1 --2- 172.21.15.46:0/3963616054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f202c140af0 0x7f202c140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:49.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.571+0000 7f202ad9d700 1 --2- 172.21.15.46:0/3963616054 >> v2:172.21.15.106:3300/0 conn(0x7f202c0caf30 0x7f202c0cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:49.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.571+0000 7f202a59c700 1 -- 172.21.15.46:0/3963616054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f202c140af0 msgr2=0x7f202c140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:49.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.571+0000 7f202a59c700 1 --2- 172.21.15.46:0/3963616054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f202c140af0 0x7f202c140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.571+0000 7f202a59c700 1 -- 172.21.15.46:0/3963616054 >> v2:172.21.15.106:3300/0 conn(0x7f202c0caf30 msgr2=0x7f202c0cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:49.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.571+0000 7f202a59c700 1 --2- 172.21.15.46:0/3963616054 >> v2:172.21.15.106:3300/0 conn(0x7f202c0caf30 0x7f202c0cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.571+0000 7f202a59c700 1 -- 172.21.15.46:0/3963616054 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202c0cc5b0 con 0x7f202c0c8da0 2024-01-27T01:07:49.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.571+0000 7f202a59c700 1 --2- 172.21.15.46:0/3963616054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f202c0c8da0 0x7f202c0c9170 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f202c0ce550 tx=0x7f201c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28657e639d2ce62 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:49.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.571+0000 7f202959a700 1 -- 172.21.15.46:0/3963616054 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c00b640 con 0x7f202c0c8da0 2024-01-27T01:07:49.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.572+0000 7f202959a700 1 -- 172.21.15.46:0/3963616054 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f201c00b7a0 con 0x7f202c0c8da0 2024-01-27T01:07:49.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.572+0000 7f202959a700 1 -- 172.21.15.46:0/3963616054 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c00bb60 con 0x7f202c0c8da0 2024-01-27T01:07:49.573 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:49.573+0000 7ff61bf941c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:07:49.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.572+0000 7f2030a1c700 1 -- 172.21.15.46:0/3963616054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f202c0c8da0 msgr2=0x7f202c0c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:49.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.572+0000 7f2030a1c700 1 --2- 172.21.15.46:0/3963616054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f202c0c8da0 0x7f202c0c9170 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f202c0ce550 tx=0x7f201c009250 comp rx=0 tx=0).stop 2024-01-27T01:07:49.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.572+0000 7f2030a1c700 1 -- 172.21.15.46:0/3963616054 shutdown_connections 2024-01-27T01:07:49.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.572+0000 7f2030a1c700 1 --2- 172.21.15.46:0/3963616054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f202c0c8da0 0x7f202c0c9170 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.572+0000 7f2030a1c700 1 --2- 172.21.15.46:0/3963616054 >> v2:172.21.15.106:3300/0 conn(0x7f202c0caf30 0x7f202c0cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.572+0000 7f2030a1c700 1 --2- 172.21.15.46:0/3963616054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f202c140af0 0x7f202c140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.572+0000 7f2030a1c700 1 -- 172.21.15.46:0/3963616054 >> 172.21.15.46:0/3963616054 conn(0x7f202c06bd90 msgr2=0x7f202c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:49.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.572+0000 7f2030a1c700 1 -- 172.21.15.46:0/3963616054 shutdown_connections 2024-01-27T01:07:49.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.572+0000 7f2030a1c700 1 -- 172.21.15.46:0/3963616054 wait complete. 2024-01-27T01:07:49.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.574+0000 7f2030a1c700 1 Processor -- start 2024-01-27T01:07:49.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.574+0000 7f2030a1c700 1 -- start start 2024-01-27T01:07:49.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.575+0000 7f2030a1c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f202c0c8da0 0x7f202c169d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.575+0000 7f2030a1c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f202c0caf30 0x7f202c13b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.575+0000 7f2030a1c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f202c140af0 0x7f202c0afb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.575+0000 7f2030a1c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c069ab0 con 0x7f202c140af0 2024-01-27T01:07:49.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.575+0000 7f2030a1c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c068ee0 con 0x7f202c0caf30 2024-01-27T01:07:49.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.575+0000 7f2030a1c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f202c069180 con 0x7f202c0c8da0 2024-01-27T01:07:49.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.575+0000 7f202ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f202c140af0 0x7f202c0afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:49.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.575+0000 7f202a59c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f202c0c8da0 0x7f202c169d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:49.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.575+0000 7f202ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f202c140af0 0x7f202c0afb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37912/0 (socket says 172.21.15.46:37912) 2024-01-27T01:07:49.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.575+0000 7f202ad9d700 1 -- 172.21.15.46:0/2686156225 learned_addr learned my addr 172.21.15.46:0/2686156225 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:49.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.575+0000 7f2029d9b700 1 --2- 172.21.15.46:0/2686156225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f202c0caf30 0x7f202c13b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:49.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.576+0000 7f202ad9d700 1 -- 172.21.15.46:0/2686156225 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f202c0c8da0 msgr2=0x7f202c169d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:49.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.576+0000 7f202ad9d700 1 --2- 172.21.15.46:0/2686156225 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f202c0c8da0 0x7f202c169d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.576+0000 7f202ad9d700 1 -- 172.21.15.46:0/2686156225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f202c0caf30 msgr2=0x7f202c13b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:49.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.576+0000 7f202ad9d700 1 --2- 172.21.15.46:0/2686156225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f202c0caf30 0x7f202c13b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.576+0000 7f202ad9d700 1 -- 172.21.15.46:0/2686156225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201c0094f0 con 0x7f202c140af0 2024-01-27T01:07:49.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.576+0000 7f202a59c700 1 --2- 172.21.15.46:0/2686156225 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f202c0c8da0 0x7f202c169d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:49.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.576+0000 7f2029d9b700 1 --2- 172.21.15.46:0/2686156225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f202c0caf30 0x7f202c13b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:49.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.576+0000 7f202ad9d700 1 --2- 172.21.15.46:0/2686156225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f202c140af0 0x7f202c0afb50 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f2024009e90 tx=0x7f202400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:49.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.576+0000 7f201a7fc700 1 -- 172.21.15.46:0/2686156225 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202400cb70 con 0x7f202c140af0 2024-01-27T01:07:49.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.576+0000 7f201a7fc700 1 -- 172.21.15.46:0/2686156225 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202400f040 con 0x7f202c140af0 2024-01-27T01:07:49.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.576+0000 7f2030a1c700 1 -- 172.21.15.46:0/2686156225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f202c069460 con 0x7f202c140af0 2024-01-27T01:07:49.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.576+0000 7f201a7fc700 1 -- 172.21.15.46:0/2686156225 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2024014480 con 0x7f202c140af0 2024-01-27T01:07:49.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.576+0000 7f2030a1c700 1 -- 172.21.15.46:0/2686156225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f202c0b1520 con 0x7f202c140af0 2024-01-27T01:07:49.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.577+0000 7f2030a1c700 1 -- 172.21.15.46:0/2686156225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f202c069c60 con 0x7f202c140af0 2024-01-27T01:07:49.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.578+0000 7f201a7fc700 1 -- 172.21.15.46:0/2686156225 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 137) v1 ==== 84963+0+0 (secure 0 0 0) 0x7f2024014830 con 0x7f202c140af0 2024-01-27T01:07:49.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.578+0000 7f201a7fc700 1 --2- 172.21.15.46:0/2686156225 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f200c0671c0 0x7f200c069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.578+0000 7f202a59c700 1 -- 172.21.15.46:0/2686156225 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f200c0671c0 msgr2=0x7f200c069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:49.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.578+0000 7f201a7fc700 1 -- 172.21.15.46:0/2686156225 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 7381+0+0 (secure 0 0 0) 0x7f2024085d90 con 0x7f202c140af0 2024-01-27T01:07:49.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.578+0000 7f202a59c700 1 --2- 172.21.15.46:0/2686156225 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f200c0671c0 0x7f200c069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:49.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.583+0000 7f201a7fc700 1 -- 172.21.15.46:0/2686156225 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f202c069c60 con 0x7f202c140af0 2024-01-27T01:07:49.587 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:49.587+0000 7fe9cc7211c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:07:49.664 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:49.664+0000 7fe9cc7211c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:07:49.729 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:49.730+0000 7ff61bf941c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:07:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.779+0000 7f202a59c700 1 -- 172.21.15.46:0/2686156225 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f200c0671c0 msgr2=0x7f200c069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.779+0000 7f202a59c700 1 --2- 172.21.15.46:0/2686156225 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f200c0671c0 0x7f200c069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.792+0000 7f2030a1c700 1 -- 172.21.15.46:0/2686156225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f202c04c710 con 0x7f202c140af0 2024-01-27T01:07:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.795+0000 7f201a7fc700 1 -- 172.21.15.46:0/2686156225 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v84)=0 v84) v1 ==== 133+0+0 (secure 0 0 0) 0x7f20240525d0 con 0x7f202c140af0 2024-01-27T01:07:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.798+0000 7f2030a1c700 1 -- 172.21.15.46:0/2686156225 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f200c0671c0 msgr2=0x7f200c069670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.798+0000 7f2030a1c700 1 --2- 172.21.15.46:0/2686156225 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f200c0671c0 0x7f200c069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.798+0000 7f2030a1c700 1 -- 172.21.15.46:0/2686156225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f202c140af0 msgr2=0x7f202c0afb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.798+0000 7f2030a1c700 1 --2- 172.21.15.46:0/2686156225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f202c140af0 0x7f202c0afb50 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f2024009e90 tx=0x7f202400b040 comp rx=0 tx=0).stop 2024-01-27T01:07:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.798+0000 7f2030a1c700 1 -- 172.21.15.46:0/2686156225 shutdown_connections 2024-01-27T01:07:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.798+0000 7f2030a1c700 1 --2- 172.21.15.46:0/2686156225 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f200c0671c0 0x7f200c069670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.798+0000 7f2030a1c700 1 --2- 172.21.15.46:0/2686156225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f202c140af0 0x7f202c0afb50 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.798+0000 7f2030a1c700 1 --2- 172.21.15.46:0/2686156225 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f202c0c8da0 0x7f202c169d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.798+0000 7f2030a1c700 1 --2- 172.21.15.46:0/2686156225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f202c0caf30 0x7f202c13b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.798+0000 7f2030a1c700 1 -- 172.21.15.46:0/2686156225 >> 172.21.15.46:0/2686156225 conn(0x7f202c06bd90 msgr2=0x7f202c13fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.798+0000 7f2030a1c700 1 -- 172.21.15.46:0/2686156225 shutdown_connections 2024-01-27T01:07:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.798+0000 7f2030a1c700 1 -- 172.21.15.46:0/2686156225 wait complete. 2024-01-27T01:07:49.812 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:07:49.812 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-01-27T01:07:49.835 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir 1M 2024-01-27T01:07:49.843 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:49.843+0000 7fe9cc7211c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:07:49.868 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:49.868+0000 7ff61bf941c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:07:49.933 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:49.934+0000 7ff61bf941c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:07:49.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.986+0000 7fea375f9700 1 Processor -- start 2024-01-27T01:07:49.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.986+0000 7fea375f9700 1 -- start start 2024-01-27T01:07:49.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.987+0000 7fea375f9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fea30068940 0x7fea30068d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.987+0000 7fea375f9700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea300c87c0 con 0x7fea300c90c0 2024-01-27T01:07:49.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.987+0000 7fea375f9700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea30069250 con 0x7fea300c8c50 2024-01-27T01:07:49.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.987+0000 7fea375f9700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea300c7bb0 con 0x7fea30068940 2024-01-27T01:07:49.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.987+0000 7fea35b96700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fea300c90c0 0x7fea300af990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42558/0 (socket says 172.21.15.46:42558) 2024-01-27T01:07:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.987+0000 7fea35b96700 1 -- 172.21.15.46:0/582022520 learned_addr learned my addr 172.21.15.46:0/582022520 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.987+0000 7fea35395700 1 --2- 172.21.15.46:0/582022520 >> v2:172.21.15.106:3300/0 conn(0x7fea30068940 0x7fea30068d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.988+0000 7fea2ffff700 1 -- 172.21.15.46:0/582022520 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750566302 0 0) 0x7fea300c87c0 con 0x7fea300c90c0 2024-01-27T01:07:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.988+0000 7fea2ffff700 1 -- 172.21.15.46:0/582022520 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea18003570 con 0x7fea300c90c0 2024-01-27T01:07:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.988+0000 7fea2ffff700 1 -- 172.21.15.46:0/582022520 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948926330 0 0) 0x7fea30069250 con 0x7fea300c8c50 2024-01-27T01:07:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.988+0000 7fea2ffff700 1 -- 172.21.15.46:0/582022520 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea300c87c0 con 0x7fea300c8c50 2024-01-27T01:07:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.988+0000 7fea35395700 1 -- 172.21.15.46:0/582022520 >> v1:172.21.15.106:6789/0 conn(0x7fea300c8c50 legacy=0x7fea300c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.988+0000 7fea35395700 1 -- 172.21.15.46:0/582022520 >> v1:172.21.15.46:6789/0 conn(0x7fea300c90c0 legacy=0x7fea300af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.988+0000 7fea35395700 1 -- 172.21.15.46:0/582022520 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea300c7e90 con 0x7fea30068940 2024-01-27T01:07:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.988+0000 7fea35395700 1 --2- 172.21.15.46:0/582022520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea30068940 0x7fea30068d10 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fea20009eb0 tx=0x7fea20009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=186cd0653e18888c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.988+0000 7fea2ffff700 1 -- 172.21.15.46:0/582022520 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2000e070 con 0x7fea30068940 2024-01-27T01:07:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.988+0000 7fea2ffff700 1 -- 172.21.15.46:0/582022520 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2000bb80 con 0x7fea30068940 2024-01-27T01:07:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.988+0000 7fea2ffff700 1 -- 172.21.15.46:0/582022520 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea20012430 con 0x7fea30068940 2024-01-27T01:07:49.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.989+0000 7fea375f9700 1 -- 172.21.15.46:0/582022520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea30068940 msgr2=0x7fea30068d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:49.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.989+0000 7fea375f9700 1 --2- 172.21.15.46:0/582022520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea30068940 0x7fea30068d10 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fea20009eb0 tx=0x7fea20009780 comp rx=0 tx=0).stop 2024-01-27T01:07:49.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.989+0000 7fea375f9700 1 -- 172.21.15.46:0/582022520 shutdown_connections 2024-01-27T01:07:49.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.989+0000 7fea375f9700 1 --2- 172.21.15.46:0/582022520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea30068940 0x7fea30068d10 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.989+0000 7fea375f9700 1 -- 172.21.15.46:0/582022520 >> 172.21.15.46:0/582022520 conn(0x7fea300ca100 msgr2=0x7fea3006e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:49.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.989+0000 7fea375f9700 1 -- 172.21.15.46:0/582022520 shutdown_connections 2024-01-27T01:07:49.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.989+0000 7fea375f9700 1 -- 172.21.15.46:0/582022520 wait complete. 2024-01-27T01:07:49.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.991+0000 7fea375f9700 1 Processor -- start 2024-01-27T01:07:49.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.991+0000 7fea375f9700 1 -- start start 2024-01-27T01:07:49.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.993+0000 7fea375f9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea30068940 0x7fea300c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.993+0000 7fea375f9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea300c8c50 0x7fea3014b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.993+0000 7fea375f9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea300c90c0 0x7fea300c1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.993+0000 7fea375f9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea300c5630 con 0x7fea300c90c0 2024-01-27T01:07:49.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.993+0000 7fea375f9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea300c2400 con 0x7fea300c8c50 2024-01-27T01:07:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.993+0000 7fea375f9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea300c4d00 con 0x7fea30068940 2024-01-27T01:07:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.993+0000 7fea35395700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea30068940 0x7fea300c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.993+0000 7fea34b94700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea300c8c50 0x7fea3014b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.993+0000 7fea35395700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea30068940 0x7fea300c5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:41454/0 (socket says 172.21.15.46:41454) 2024-01-27T01:07:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.993+0000 7fea35395700 1 -- 172.21.15.46:0/2562617898 learned_addr learned my addr 172.21.15.46:0/2562617898 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.993+0000 7fea35b96700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea300c90c0 0x7fea300c1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.993+0000 7fea34b94700 1 -- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea30068940 msgr2=0x7fea300c5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.993+0000 7fea34b94700 1 --2- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea30068940 0x7fea300c5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.993+0000 7fea34b94700 1 -- 172.21.15.46:0/2562617898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea300c90c0 msgr2=0x7fea300c1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.993+0000 7fea34b94700 1 --2- 172.21.15.46:0/2562617898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea300c90c0 0x7fea300c1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.993+0000 7fea34b94700 1 -- 172.21.15.46:0/2562617898 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea20009a20 con 0x7fea300c8c50 2024-01-27T01:07:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.993+0000 7fea35b96700 1 --2- 172.21.15.46:0/2562617898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea300c90c0 0x7fea300c1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.994+0000 7fea34b94700 1 --2- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea300c8c50 0x7fea3014b1e0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fea2400c560 tx=0x7fea24007d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.994+0000 7fea2d7fa700 1 -- 172.21.15.46:0/2562617898 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea24011590 con 0x7fea300c8c50 2024-01-27T01:07:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.994+0000 7fea2d7fa700 1 -- 172.21.15.46:0/2562617898 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea240116f0 con 0x7fea300c8c50 2024-01-27T01:07:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.994+0000 7fea375f9700 1 -- 172.21.15.46:0/2562617898 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea300c4fe0 con 0x7fea300c8c50 2024-01-27T01:07:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.994+0000 7fea2d7fa700 1 -- 172.21.15.46:0/2562617898 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea24011ab0 con 0x7fea300c8c50 2024-01-27T01:07:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.994+0000 7fea375f9700 1 -- 172.21.15.46:0/2562617898 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea300c1430 con 0x7fea300c8c50 2024-01-27T01:07:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.995+0000 7fea375f9700 1 -- 172.21.15.46:0/2562617898 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3004c710 con 0x7fea300c8c50 2024-01-27T01:07:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.995+0000 7fea2d7fa700 1 -- 172.21.15.46:0/2562617898 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 137) v1 ==== 84963+0+0 (secure 0 0 0) 0x7fea2401c460 con 0x7fea300c8c50 2024-01-27T01:07:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.996+0000 7fea2d7fa700 1 --2- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea1c0671c0 0x7fea1c069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.996+0000 7fea2d7fa700 1 -- 172.21.15.46:0/2562617898 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 7381+0+0 (secure 0 0 0) 0x7fea24084b60 con 0x7fea300c8c50 2024-01-27T01:07:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.996+0000 7fea35395700 1 -- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea1c0671c0 msgr2=0x7fea1c069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:50.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:49.996+0000 7fea35395700 1 --2- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea1c0671c0 0x7fea1c069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:50.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:50.000+0000 7fea2d7fa700 1 -- 172.21.15.46:0/2562617898 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fea24051490 con 0x7fea300c8c50 2024-01-27T01:07:50.027 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:50.027+0000 7fe9cc7211c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:07:50.119 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:50.058+0000 7ff61bf941c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:07:50.130 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:50.131+0000 7ff61bf941c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:07:50.147 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:50.147+0000 7fe9cc7211c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:07:50.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:50.197+0000 7fea35395700 1 -- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea1c0671c0 msgr2=0x7fea1c069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:50.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:50.197+0000 7fea35395700 1 --2- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea1c0671c0 0x7fea1c069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:50.271 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:50.271+0000 7ff61bf941c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:07:50.287 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:50.287+0000 7fe9cc7211c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:07:50.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:50.307+0000 7fea375f9700 1 -- 172.21.15.46:0/2562617898 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fea3006c9c0 con 0x7fea1c0671c0 2024-01-27T01:07:50.369 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:50.369+0000 7fe9cc7211c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:07:50.397 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:50.397+0000 7ff61bf941c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:07:50.546 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:50.546+0000 7fe9cc7211c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:07:50.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:50.598+0000 7fea35395700 1 -- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea1c0671c0 msgr2=0x7fea1c069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:50.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:50.598+0000 7fea35395700 1 --2- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea1c0671c0 0x7fea1c069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-27T01:07:50.688 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:50.688+0000 7fe9cc7211c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:07:50.711 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:50.711+0000 7ff61bf941c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:07:50.923 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:50.923+0000 7ff61bf941c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:07:51.051 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:51.051+0000 7fe9cc7211c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:07:51.152 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:51.152+0000 7ff61bf941c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:07:51.222 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:51.222+0000 7ff61bf941c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:07:51.291 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:51.291+0000 7fe9cc7211c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:07:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:51.399+0000 7fea35395700 1 -- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea1c0671c0 msgr2=0x7fea1c069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:51.399+0000 7fea35395700 1 --2- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea1c0671c0 0x7fea1c069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-27T01:07:51.552 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:51.552+0000 7fe9cc7211c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:07:51.633 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:51.633+0000 7fe9cc7211c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:07:51.825 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:51.825+0000 7ff61bf941c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:07:51.899 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:51.899+0000 7ff61bf941c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:07:52.022 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:52.023+0000 7ff61bf941c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:07:52.333 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:52.333+0000 7fe9cc7211c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:07:52.418 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:52.418+0000 7fe9cc7211c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:07:52.523 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:52.523+0000 7ff61bf941c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:07:52.559 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:52.559+0000 7fe9cc7211c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:07:52.674 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:52.674+0000 7ff61bf941c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:07:52.827 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:52.827+0000 7ff61bf941c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:07:53.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:53.001+0000 7fea35395700 1 -- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea1c0671c0 msgr2=0x7fea1c069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:53.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:53.001+0000 7fea35395700 1 --2- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea1c0671c0 0x7fea1c069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-27T01:07:53.146 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:53.146+0000 7fe9cc7211c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:07:53.317 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:53.317+0000 7fe9cc7211c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:07:53.490 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:53.490+0000 7fe9cc7211c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:07:53.528 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:53.529+0000 7ff61bf941c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:07:53.749 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:53.749+0000 7ff61bf941c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:07:53.823 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:53.823+0000 7ff61bf941c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:07:53.981 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:53.981+0000 7ff61bf941c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:07:54.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:54.003+0000 7fea2d7fa700 1 -- 172.21.15.46:0/2562617898 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mgrmap(e 138) v1 ==== 84470+0+0 (secure 0 0 0) 0x7fea24051860 con 0x7fea300c8c50 2024-01-27T01:07:54.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:54.003+0000 7fea2d7fa700 1 -- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea1c0671c0 msgr2=0x7fea1c069670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:54.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:54.003+0000 7fea2d7fa700 1 --2- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea1c0671c0 0x7fea1c069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:54.143 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.143+0000 7ff595077700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.143 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.143+0000 7ff595077700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.143 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.143+0000 7ff595077700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.143 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.143+0000 7ff595077700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.143 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.143+0000 7ff595077700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.144 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.143+0000 7ff592071700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.144 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.143+0000 7ff592071700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.144 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.143+0000 7ff592071700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.144 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.143+0000 7ff592071700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.144 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.143+0000 7ff592071700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.154 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.154+0000 7ff595077700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.154 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.154+0000 7ff595077700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.154 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.154+0000 7ff595077700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.154 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.154+0000 7ff595077700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.154 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.154+0000 7ff595077700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.155 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.155+0000 7ff592071700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.155 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.155+0000 7ff592071700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.155 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.155+0000 7ff592071700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.155 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.155+0000 7ff592071700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.155 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:54.155+0000 7ff592071700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:07:54.322 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:54.322+0000 7fe9cc7211c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:07:54.574 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:54.574+0000 7fe9cc7211c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:07:54.659 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:54.659+0000 7fe9cc7211c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:07:54.840 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:54.839+0000 7fe9cc7211c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:07:55.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.008+0000 7fea2d7fa700 1 -- 172.21.15.46:0/2562617898 <== mon.1 v2:172.21.15.106:3300/0 8 ==== mgrmap(e 139) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fea2400f260 con 0x7fea300c8c50 2024-01-27T01:07:55.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.008+0000 7fea2d7fa700 1 --2- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea1c0671c0 0x7fea1c0695f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:55.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.008+0000 7fea2d7fa700 1 -- 172.21.15.46:0/2562617898 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fea3006c9c0 con 0x7fea1c0671c0 2024-01-27T01:07:55.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.012+0000 7fea35395700 1 --2- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea1c0671c0 0x7fea1c0695f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:55.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.013+0000 7fea35395700 1 --2- 172.21.15.46:0/2562617898 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea1c0671c0 0x7fea1c0695f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fea200097c0 tx=0x7fea20019000 comp rx=0 tx=0).ready entity=mgr.11524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:55.015 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:55.016+0000 7ff59c085700 -1 mgr.server reply reply (22) Invalid argument filesystem argument is required when there is more than one file system 2024-01-27T01:07:55.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.016+0000 7fea2d7fa700 1 -- 172.21.15.46:0/2562617898 <== mgr.11524 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: -22 filesystem argument is required when there is more than one file system) v1 ==== 79+0+0 (secure 0 0 0) 0x7fea3006c9c0 con 0x7fea1c0671c0 2024-01-27T01:07:55.018 INFO:teuthology.orchestra.run.smithi046.stderr:Error EINVAL: filesystem argument is required when there is more than one file system 2024-01-27T01:07:55.022 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-27T01:07:55.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:07:55.023 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-01-27T01:07:55.048 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-27T01:07:55.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.232+0000 7f0541058700 1 Processor -- start 2024-01-27T01:07:55.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.232+0000 7f0541058700 1 -- start start 2024-01-27T01:07:55.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.233+0000 7f0541058700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053c0c8da0 0x7f053c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:55.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.233+0000 7f0541058700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c0ccc00 con 0x7f053c0caf30 2024-01-27T01:07:55.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.233+0000 7f0541058700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c0cc310 con 0x7f053c140af0 2024-01-27T01:07:55.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.233+0000 7f0541058700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053c0cbbf0 con 0x7f053c0c8da0 2024-01-27T01:07:55.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.233+0000 7f053b59e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f053c0caf30 0x7f053c0cb370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60482/0 (socket says 172.21.15.46:60482) 2024-01-27T01:07:55.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.233+0000 7f053b59e700 1 -- 172.21.15.46:0/1364458332 learned_addr learned my addr 172.21.15.46:0/1364458332 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:55.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.233+0000 7f053ad9d700 1 --2- 172.21.15.46:0/1364458332 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053c0c8da0 0x7f053c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:55.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.233+0000 7f053a59c700 1 -- 172.21.15.46:0/1364458332 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069125016 0 0) 0x7f053c0ccc00 con 0x7f053c0caf30 2024-01-27T01:07:55.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.234+0000 7f053a59c700 1 -- 172.21.15.46:0/1364458332 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0528003570 con 0x7f053c0caf30 2024-01-27T01:07:55.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.234+0000 7f053a59c700 1 -- 172.21.15.46:0/1364458332 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376690697 0 0) 0x7f053c0cc310 con 0x7f053c140af0 2024-01-27T01:07:55.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.234+0000 7f053a59c700 1 -- 172.21.15.46:0/1364458332 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f053c0ccc00 con 0x7f053c140af0 2024-01-27T01:07:55.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.234+0000 7f053ad9d700 1 -- 172.21.15.46:0/1364458332 >> v1:172.21.15.106:6789/0 conn(0x7f053c140af0 legacy=0x7f053c140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:55.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.234+0000 7f053ad9d700 1 -- 172.21.15.46:0/1364458332 >> v1:172.21.15.46:6789/0 conn(0x7f053c0caf30 legacy=0x7f053c0cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:55.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.234+0000 7f053a59c700 1 -- 172.21.15.46:0/1364458332 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3928323970 0 0) 0x7f0528003570 con 0x7f053c0caf30 2024-01-27T01:07:55.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.234+0000 7f053ad9d700 1 -- 172.21.15.46:0/1364458332 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c0cc5b0 con 0x7f053c0c8da0 2024-01-27T01:07:55.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.234+0000 7f053ad9d700 1 --2- 172.21.15.46:0/1364458332 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053c0c8da0 0x7f053c0c9170 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f052c009910 tx=0x7f052c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfecc0c5a117dd4b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:55.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.234+0000 7f053a59c700 1 -- 172.21.15.46:0/1364458332 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05340017f0 con 0x7f053c0c8da0 2024-01-27T01:07:55.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.234+0000 7f053a59c700 1 -- 172.21.15.46:0/1364458332 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c011420 con 0x7f053c0c8da0 2024-01-27T01:07:55.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.234+0000 7f053a59c700 1 -- 172.21.15.46:0/1364458332 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c0117c0 con 0x7f053c0c8da0 2024-01-27T01:07:55.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.235+0000 7f0541058700 1 -- 172.21.15.46:0/1364458332 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053c0c8da0 msgr2=0x7f053c0c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:55.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.235+0000 7f0541058700 1 --2- 172.21.15.46:0/1364458332 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053c0c8da0 0x7f053c0c9170 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f052c009910 tx=0x7f052c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:55.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.235+0000 7f0541058700 1 -- 172.21.15.46:0/1364458332 shutdown_connections 2024-01-27T01:07:55.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.235+0000 7f0541058700 1 --2- 172.21.15.46:0/1364458332 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053c0c8da0 0x7f053c0c9170 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:55.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.235+0000 7f0541058700 1 -- 172.21.15.46:0/1364458332 >> 172.21.15.46:0/1364458332 conn(0x7f053c06bd90 msgr2=0x7f053c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:55.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.235+0000 7f0541058700 1 -- 172.21.15.46:0/1364458332 shutdown_connections 2024-01-27T01:07:55.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.235+0000 7f0541058700 1 -- 172.21.15.46:0/1364458332 wait complete. 2024-01-27T01:07:55.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.236+0000 7f0541058700 1 Processor -- start 2024-01-27T01:07:55.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.236+0000 7f0541058700 1 -- start start 2024-01-27T01:07:55.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.237+0000 7f0541058700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f053c0c8da0 0x7f053c0b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:55.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.237+0000 7f0541058700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053c0caf30 0x7f053c069f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:55.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.237+0000 7f0541058700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f053c140af0 0x7f053c1571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:55.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.237+0000 7f0541058700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c1576f0 con 0x7f053c140af0 2024-01-27T01:07:55.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.237+0000 7f0541058700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c068ed0 con 0x7f053c0c8da0 2024-01-27T01:07:55.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.237+0000 7f0541058700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053c0691b0 con 0x7f053c0caf30 2024-01-27T01:07:55.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.237+0000 7f053b59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f053c140af0 0x7f053c1571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:55.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.237+0000 7f053b59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f053c140af0 0x7f053c1571b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51862/0 (socket says 172.21.15.46:51862) 2024-01-27T01:07:55.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.237+0000 7f053b59e700 1 -- 172.21.15.46:0/4002756306 learned_addr learned my addr 172.21.15.46:0/4002756306 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:55.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.237+0000 7f053ad9d700 1 --2- 172.21.15.46:0/4002756306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f053c0c8da0 0x7f053c0b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:55.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.237+0000 7f0533fff700 1 --2- 172.21.15.46:0/4002756306 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053c0caf30 0x7f053c069f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:55.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.238+0000 7f053b59e700 1 -- 172.21.15.46:0/4002756306 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053c0caf30 msgr2=0x7f053c069f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:55.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.238+0000 7f053b59e700 1 --2- 172.21.15.46:0/4002756306 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053c0caf30 0x7f053c069f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:55.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.238+0000 7f053b59e700 1 -- 172.21.15.46:0/4002756306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f053c0c8da0 msgr2=0x7f053c0b3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:55.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.238+0000 7f053b59e700 1 --2- 172.21.15.46:0/4002756306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f053c0c8da0 0x7f053c0b3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:55.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.238+0000 7f053b59e700 1 -- 172.21.15.46:0/4002756306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c009420 con 0x7f053c140af0 2024-01-27T01:07:55.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.238+0000 7f053b59e700 1 --2- 172.21.15.46:0/4002756306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f053c140af0 0x7f053c1571b0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f053400cc80 tx=0x7f0534007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:55.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.238+0000 7f0532ffd700 1 -- 172.21.15.46:0/4002756306 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0534004dc0 con 0x7f053c140af0 2024-01-27T01:07:55.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.238+0000 7f0532ffd700 1 -- 172.21.15.46:0/4002756306 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053400a660 con 0x7f053c140af0 2024-01-27T01:07:55.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.238+0000 7f0532ffd700 1 -- 172.21.15.46:0/4002756306 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053400aa20 con 0x7f053c140af0 2024-01-27T01:07:55.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.238+0000 7f0541058700 1 -- 172.21.15.46:0/4002756306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053c069490 con 0x7f053c140af0 2024-01-27T01:07:55.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.238+0000 7f0541058700 1 -- 172.21.15.46:0/4002756306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053c069a50 con 0x7f053c140af0 2024-01-27T01:07:55.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.239+0000 7f0541058700 1 -- 172.21.15.46:0/4002756306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053c069c70 con 0x7f053c140af0 2024-01-27T01:07:55.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.240+0000 7f0532ffd700 1 -- 172.21.15.46:0/4002756306 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 139) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f0534007150 con 0x7f053c140af0 2024-01-27T01:07:55.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.240+0000 7f0532ffd700 1 --2- 172.21.15.46:0/4002756306 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f051c066dd0 0x7f051c069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:55.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.240+0000 7f0532ffd700 1 -- 172.21.15.46:0/4002756306 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7811+0+0 (secure 0 0 0) 0x7f0534084070 con 0x7f053c140af0 2024-01-27T01:07:55.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.240+0000 7f053ad9d700 1 --2- 172.21.15.46:0/4002756306 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f051c066dd0 0x7f051c069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:55.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.241+0000 7f053ad9d700 1 --2- 172.21.15.46:0/4002756306 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f051c066dd0 0x7f051c069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f052c0030c0 tx=0x7f052c016000 comp rx=0 tx=0).ready entity=mgr.11524 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:55.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.245+0000 7f0532ffd700 1 -- 172.21.15.46:0/4002756306 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0534050dd0 con 0x7f053c140af0 2024-01-27T01:07:55.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:55.506+0000 7f0541058700 1 -- 172.21.15.46:0/4002756306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f053c04c710 con 0x7f053c140af0 2024-01-27T01:07:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.060+0000 7f0532ffd700 1 -- 172.21.15.46:0/4002756306 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 140) v1 ==== 84947+0+0 (secure 0 0 0) 0x7f05340511a0 con 0x7f053c140af0 2024-01-27T01:07:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.061+0000 7f0532ffd700 1 -- 172.21.15.46:0/4002756306 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v140) v1 ==== 93+0+0 (secure 0 0 0) 0x7f0534050dd0 con 0x7f053c140af0 2024-01-27T01:07:56.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.065+0000 7f0541058700 1 -- 172.21.15.46:0/4002756306 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f051c066dd0 msgr2=0x7f051c069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:56.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.065+0000 7f0541058700 1 --2- 172.21.15.46:0/4002756306 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f051c066dd0 0x7f051c069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f052c0030c0 tx=0x7f052c016000 comp rx=0 tx=0).stop 2024-01-27T01:07:56.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.065+0000 7f0541058700 1 -- 172.21.15.46:0/4002756306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f053c140af0 msgr2=0x7f053c1571b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:56.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.065+0000 7f0541058700 1 --2- 172.21.15.46:0/4002756306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f053c140af0 0x7f053c1571b0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f053400cc80 tx=0x7f0534007d20 comp rx=0 tx=0).stop 2024-01-27T01:07:56.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.065+0000 7f0541058700 1 -- 172.21.15.46:0/4002756306 shutdown_connections 2024-01-27T01:07:56.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.065+0000 7f0541058700 1 --2- 172.21.15.46:0/4002756306 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f051c066dd0 0x7f051c069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:56.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.065+0000 7f0541058700 1 --2- 172.21.15.46:0/4002756306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f053c140af0 0x7f053c1571b0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:56.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.065+0000 7f0541058700 1 --2- 172.21.15.46:0/4002756306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f053c0c8da0 0x7f053c0b3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:56.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.065+0000 7f0541058700 1 --2- 172.21.15.46:0/4002756306 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f053c0caf30 0x7f053c069f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:56.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.065+0000 7f0541058700 1 -- 172.21.15.46:0/4002756306 >> 172.21.15.46:0/4002756306 conn(0x7f053c06bd90 msgr2=0x7f053c13f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:56.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.065+0000 7f0541058700 1 -- 172.21.15.46:0/4002756306 shutdown_connections 2024-01-27T01:07:56.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.065+0000 7f0541058700 1 -- 172.21.15.46:0/4002756306 wait complete. 2024-01-27T01:07:56.082 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-27T01:07:56.082 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:07:56.132 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:07:56.180 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-27T01:07:56.180 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-27T01:07:56.180 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-27T01:07:56.180 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-27T01:07:56.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[117260]: fuse finished with error 0 and tester_r 0 2024-01-27T01:07:56.221 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T01:07:56.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:07:56.222 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-27T01:07:56.352 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T01:07:56.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:07:56.352 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-27T01:07:56.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:07:56.369 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-27T01:07:56.369 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T01:07:56.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:07:56.369 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-27T01:07:56.389 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-27T01:07:56.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:07:56.390 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T01:07:56.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:07:56.390 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-27T01:07:56.406 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:56.406+0000 7f9b4c19d1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:07:56.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:07:56.421 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-27T01:07:56.422 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:07:56.457 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:56.457+0000 7f6ad5cab1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:07:56.485 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:56.485+0000 7f9b4c19d1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:07:56.548 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:56.548+0000 7f6ad5cab1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:07:56.555 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:56.555+0000 7f9b4c19d1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:07:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.567+0000 7f28c1085700 1 Processor -- start 2024-01-27T01:07:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.567+0000 7f28c1085700 1 -- start start 2024-01-27T01:07:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.568+0000 7f28c1085700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f28bc159590 0x7f28bc15d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.568+0000 7f28c1085700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28bc15dd90 con 0x7f28bc06c3c0 2024-01-27T01:07:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.568+0000 7f28c1085700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28bc15e070 con 0x7f28bc15d8f0 2024-01-27T01:07:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.568+0000 7f28c1085700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28bc15e350 con 0x7f28bc159590 2024-01-27T01:07:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.568+0000 7f28bb59e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f28bc06c3c0 0x7f28bc069370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60506/0 (socket says 172.21.15.46:60506) 2024-01-27T01:07:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.568+0000 7f28bb59e700 1 -- 172.21.15.46:0/4058578533 learned_addr learned my addr 172.21.15.46:0/4058578533 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.568+0000 7f28bad9d700 1 --2- 172.21.15.46:0/4058578533 >> v2:172.21.15.106:3300/0 conn(0x7f28bc159590 0x7f28bc15d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.569+0000 7f28b9d9b700 1 -- 172.21.15.46:0/4058578533 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659480324 0 0) 0x7f28bc15dd90 con 0x7f28bc06c3c0 2024-01-27T01:07:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.569+0000 7f28b9d9b700 1 -- 172.21.15.46:0/4058578533 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28a4003570 con 0x7f28bc06c3c0 2024-01-27T01:07:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.569+0000 7f28b9d9b700 1 -- 172.21.15.46:0/4058578533 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167470034 0 0) 0x7f28bc15e070 con 0x7f28bc15d8f0 2024-01-27T01:07:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.569+0000 7f28b9d9b700 1 -- 172.21.15.46:0/4058578533 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28bc15dd90 con 0x7f28bc15d8f0 2024-01-27T01:07:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.569+0000 7f28bad9d700 1 -- 172.21.15.46:0/4058578533 >> v1:172.21.15.106:6789/0 conn(0x7f28bc15d8f0 legacy=0x7f28bc06bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.569+0000 7f28bad9d700 1 -- 172.21.15.46:0/4058578533 >> v1:172.21.15.46:6789/0 conn(0x7f28bc06c3c0 legacy=0x7f28bc069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.569+0000 7f28bad9d700 1 -- 172.21.15.46:0/4058578533 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28bc068b70 con 0x7f28bc159590 2024-01-27T01:07:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.569+0000 7f28bad9d700 1 --2- 172.21.15.46:0/4058578533 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28bc159590 0x7f28bc15d3b0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f28ac0098e0 tx=0x7f28ac0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=658773d92971381d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.569+0000 7f28b9d9b700 1 -- 172.21.15.46:0/4058578533 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac011630 con 0x7f28bc159590 2024-01-27T01:07:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.569+0000 7f28b9d9b700 1 -- 172.21.15.46:0/4058578533 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ac011790 con 0x7f28bc159590 2024-01-27T01:07:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.569+0000 7f28b9d9b700 1 -- 172.21.15.46:0/4058578533 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac011b30 con 0x7f28bc159590 2024-01-27T01:07:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.570+0000 7f28c1085700 1 -- 172.21.15.46:0/4058578533 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28bc159590 msgr2=0x7f28bc15d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.570+0000 7f28c1085700 1 --2- 172.21.15.46:0/4058578533 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28bc159590 0x7f28bc15d3b0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f28ac0098e0 tx=0x7f28ac0091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.570+0000 7f28c1085700 1 -- 172.21.15.46:0/4058578533 shutdown_connections 2024-01-27T01:07:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.570+0000 7f28c1085700 1 --2- 172.21.15.46:0/4058578533 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28bc159590 0x7f28bc15d3b0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.570+0000 7f28c1085700 1 -- 172.21.15.46:0/4058578533 >> 172.21.15.46:0/4058578533 conn(0x7f28bc0ce530 msgr2=0x7f28bc0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.570+0000 7f28c1085700 1 -- 172.21.15.46:0/4058578533 shutdown_connections 2024-01-27T01:07:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.570+0000 7f28c1085700 1 -- 172.21.15.46:0/4058578533 wait complete. 2024-01-27T01:07:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.572+0000 7f28c1085700 1 Processor -- start 2024-01-27T01:07:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.572+0000 7f28c1085700 1 -- start start 2024-01-27T01:07:56.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.573+0000 7f28c1085700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28bc06c3c0 0x7f28bc0c2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:56.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.573+0000 7f28c1085700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28bc159590 0x7f28bc149a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:56.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.573+0000 7f28c1085700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f28bc15d8f0 0x7f28bc16a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.573+0000 7f28c1085700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc0c2d70 con 0x7f28bc06c3c0 2024-01-27T01:07:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.573+0000 7f28c1085700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc16abd0 con 0x7f28bc159590 2024-01-27T01:07:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.573+0000 7f28c1085700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc16b800 con 0x7f28bc15d8f0 2024-01-27T01:07:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.573+0000 7f28bad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28bc06c3c0 0x7f28bc0c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.573+0000 7f28bad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28bc06c3c0 0x7f28bc0c2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51886/0 (socket says 172.21.15.46:51886) 2024-01-27T01:07:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.573+0000 7f28bad9d700 1 -- 172.21.15.46:0/3848447558 learned_addr learned my addr 172.21.15.46:0/3848447558 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.573+0000 7f28ba59c700 1 --2- 172.21.15.46:0/3848447558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28bc159590 0x7f28bc149a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.573+0000 7f28bb59e700 1 --2- 172.21.15.46:0/3848447558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f28bc15d8f0 0x7f28bc16a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.573+0000 7f28bad9d700 1 -- 172.21.15.46:0/3848447558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f28bc15d8f0 msgr2=0x7f28bc16a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.573+0000 7f28bad9d700 1 --2- 172.21.15.46:0/3848447558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f28bc15d8f0 0x7f28bc16a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.573+0000 7f28bad9d700 1 -- 172.21.15.46:0/3848447558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28bc159590 msgr2=0x7f28bc149a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.573+0000 7f28bad9d700 1 --2- 172.21.15.46:0/3848447558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28bc159590 0x7f28bc149a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.573+0000 7f28bad9d700 1 -- 172.21.15.46:0/3848447558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ac009420 con 0x7f28bc06c3c0 2024-01-27T01:07:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.573+0000 7f28bad9d700 1 --2- 172.21.15.46:0/3848447558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28bc06c3c0 0x7f28bc0c2720 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f28ac009ed0 tx=0x7f28ac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.574+0000 7f28a2ffd700 1 -- 172.21.15.46:0/3848447558 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac01f070 con 0x7f28bc06c3c0 2024-01-27T01:07:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.574+0000 7f28a2ffd700 1 -- 172.21.15.46:0/3848447558 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ac002f70 con 0x7f28bc06c3c0 2024-01-27T01:07:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.574+0000 7f28c1085700 1 -- 172.21.15.46:0/3848447558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28bc16aed0 con 0x7f28bc06c3c0 2024-01-27T01:07:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.574+0000 7f28a2ffd700 1 -- 172.21.15.46:0/3848447558 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac006af0 con 0x7f28bc06c3c0 2024-01-27T01:07:56.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.574+0000 7f28c1085700 1 -- 172.21.15.46:0/3848447558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28bc16b490 con 0x7f28bc06c3c0 2024-01-27T01:07:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.575+0000 7f28c1085700 1 -- 172.21.15.46:0/3848447558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28bc04c710 con 0x7f28bc06c3c0 2024-01-27T01:07:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.576+0000 7f28a2ffd700 1 -- 172.21.15.46:0/3848447558 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 140) v1 ==== 84947+0+0 (secure 0 0 0) 0x7f28ac02c3b0 con 0x7f28bc06c3c0 2024-01-27T01:07:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.576+0000 7f28a2ffd700 1 --2- 172.21.15.46:0/3848447558 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f28a8067170 0x7f28a8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.576+0000 7f28a2ffd700 1 -- 172.21.15.46:0/3848447558 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7811+0+0 (secure 0 0 0) 0x7f28ac086260 con 0x7f28bc06c3c0 2024-01-27T01:07:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.576+0000 7f28ba59c700 1 -- 172.21.15.46:0/3848447558 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f28a8067170 msgr2=0x7f28a8069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:56.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.576+0000 7f28ba59c700 1 --2- 172.21.15.46:0/3848447558 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f28a8067170 0x7f28a8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:56.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.581+0000 7f28a2ffd700 1 -- 172.21.15.46:0/3848447558 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f28ac052930 con 0x7f28bc06c3c0 2024-01-27T01:07:56.621 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:56.622+0000 7f9b4c19d1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:07:56.632 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:56.632+0000 7f6ad5cab1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:07:56.709 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:56.709+0000 7f6ad5cab1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:07:56.775 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:56.776+0000 7f9b4c19d1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:07:56.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.777+0000 7f28ba59c700 1 -- 172.21.15.46:0/3848447558 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f28a8067170 msgr2=0x7f28a8069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:56.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.777+0000 7f28ba59c700 1 --2- 172.21.15.46:0/3848447558 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f28a8067170 0x7f28a8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.839+0000 7f28c1085700 1 -- 172.21.15.46:0/3848447558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28bc06c9a0 con 0x7f28bc06c3c0 2024-01-27T01:07:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.839+0000 7f28a2ffd700 1 -- 172.21.15.46:0/3848447558 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+5687 (secure 0 0 0) 0x7f28ac028090 con 0x7f28bc06c3c0 2024-01-27T01:07:56.839 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:56.840 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":138,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1114164873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1114164873},{"type":"v1","addr":"172.21.15.106:6834","nonce":1114164873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128},{"gid":9821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/702094428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":702094428},{"type":"v1","addr":"172.21.15.106:6835","nonce":702094428}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:07:27.375435+0000","modified":"2024-01-27T01:07:30.945729+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11248":{"gid":11248,"name":"a","rank":0,"incarnation":129,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/1087205220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1087205220},{"type":"v1","addr":"172.21.15.46:6834","nonce":1087205220}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:07:41.411084+0000","modified":"2024-01-27T01:07:45.002753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9827":{"gid":9827,"name":"c","rank":0,"incarnation":135,"state":"up:active","state_seq":6,"addr":"172.21.15.46:6835/408985798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":408985798},{"type":"v1","addr":"172.21.15.46:6835","nonce":408985798}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-27T01:07:56.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.842+0000 7f28c1085700 1 -- 172.21.15.46:0/3848447558 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f28a8067170 msgr2=0x7f28a8069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.842+0000 7f28c1085700 1 --2- 172.21.15.46:0/3848447558 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f28a8067170 0x7f28a8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.842+0000 7f28c1085700 1 -- 172.21.15.46:0/3848447558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28bc06c3c0 msgr2=0x7f28bc0c2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.842+0000 7f28c1085700 1 --2- 172.21.15.46:0/3848447558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28bc06c3c0 0x7f28bc0c2720 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f28ac009ed0 tx=0x7f28ac017000 comp rx=0 tx=0).stop 2024-01-27T01:07:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.842+0000 7f28c1085700 1 -- 172.21.15.46:0/3848447558 shutdown_connections 2024-01-27T01:07:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.842+0000 7f28c1085700 1 --2- 172.21.15.46:0/3848447558 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f28a8067170 0x7f28a8069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.842+0000 7f28c1085700 1 --2- 172.21.15.46:0/3848447558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28bc06c3c0 0x7f28bc0c2720 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.842+0000 7f28c1085700 1 --2- 172.21.15.46:0/3848447558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f28bc15d8f0 0x7f28bc16a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.842+0000 7f28c1085700 1 --2- 172.21.15.46:0/3848447558 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28bc159590 0x7f28bc149a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.842+0000 7f28c1085700 1 -- 172.21.15.46:0/3848447558 >> 172.21.15.46:0/3848447558 conn(0x7f28bc0ce530 msgr2=0x7f28bc158c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.842+0000 7f28c1085700 1 -- 172.21.15.46:0/3848447558 shutdown_connections 2024-01-27T01:07:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:56.842+0000 7f28c1085700 1 -- 172.21.15.46:0/3848447558 wait complete. 2024-01-27T01:07:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 138 2024-01-27T01:07:56.857 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:07:56.887 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:56.887+0000 7f6ad5cab1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:07:56.911 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:56.912+0000 7f9b4c19d1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:07:56.979 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:56.979+0000 7f9b4c19d1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:07:57.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.023+0000 7fb718f9b700 1 Processor -- start 2024-01-27T01:07:57.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.023+0000 7fb718f9b700 1 -- start start 2024-01-27T01:07:57.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.025+0000 7fb718f9b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb7140ae520 0x7fb7140b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:57.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.025+0000 7fb718f9b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb714158ae0 con 0x7fb714157340 2024-01-27T01:07:57.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.025+0000 7fb718f9b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb71415b650 con 0x7fb7140b3fb0 2024-01-27T01:07:57.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.025+0000 7fb718f9b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7140b43c0 con 0x7fb7140ae520 2024-01-27T01:07:57.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.025+0000 7fb71259c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb714157340 0x7fb71415bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60508/0 (socket says 172.21.15.46:60508) 2024-01-27T01:07:57.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.025+0000 7fb71259c700 1 -- 172.21.15.46:0/916283700 learned_addr learned my addr 172.21.15.46:0/916283700 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:57.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.025+0000 7fb711d9b700 1 --2- 172.21.15.46:0/916283700 >> v2:172.21.15.106:3300/0 conn(0x7fb7140ae520 0x7fb7140b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:57.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.025+0000 7fb71159a700 1 -- 172.21.15.46:0/916283700 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991436378 0 0) 0x7fb714158ae0 con 0x7fb714157340 2024-01-27T01:07:57.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.026+0000 7fb71159a700 1 -- 172.21.15.46:0/916283700 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6fc003570 con 0x7fb714157340 2024-01-27T01:07:57.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.026+0000 7fb71159a700 1 -- 172.21.15.46:0/916283700 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021594218 0 0) 0x7fb71415b650 con 0x7fb7140b3fb0 2024-01-27T01:07:57.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.026+0000 7fb71159a700 1 -- 172.21.15.46:0/916283700 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb714158ae0 con 0x7fb7140b3fb0 2024-01-27T01:07:57.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.026+0000 7fb711d9b700 1 -- 172.21.15.46:0/916283700 >> v1:172.21.15.106:6789/0 conn(0x7fb7140b3fb0 legacy=0x7fb714157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:57.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.026+0000 7fb711d9b700 1 -- 172.21.15.46:0/916283700 >> v1:172.21.15.46:6789/0 conn(0x7fb714157340 legacy=0x7fb71415bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:57.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.026+0000 7fb711d9b700 1 -- 172.21.15.46:0/916283700 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb714158520 con 0x7fb7140ae520 2024-01-27T01:07:57.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.026+0000 7fb711d9b700 1 --2- 172.21.15.46:0/916283700 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7140ae520 0x7fb7140b3a70 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fb7080098e0 tx=0x7fb7080091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2b4b842828b0567 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:57.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.026+0000 7fb71159a700 1 -- 172.21.15.46:0/916283700 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb708011630 con 0x7fb7140ae520 2024-01-27T01:07:57.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.026+0000 7fb71159a700 1 -- 172.21.15.46:0/916283700 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb708011790 con 0x7fb7140ae520 2024-01-27T01:07:57.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.026+0000 7fb71159a700 1 -- 172.21.15.46:0/916283700 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb708011b30 con 0x7fb7140ae520 2024-01-27T01:07:57.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.027+0000 7fb718f9b700 1 -- 172.21.15.46:0/916283700 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7140ae520 msgr2=0x7fb7140b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:57.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.027+0000 7fb718f9b700 1 --2- 172.21.15.46:0/916283700 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7140ae520 0x7fb7140b3a70 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fb7080098e0 tx=0x7fb7080091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.027+0000 7fb718f9b700 1 -- 172.21.15.46:0/916283700 shutdown_connections 2024-01-27T01:07:57.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.027+0000 7fb718f9b700 1 --2- 172.21.15.46:0/916283700 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7140ae520 0x7fb7140b3a70 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.027+0000 7fb718f9b700 1 -- 172.21.15.46:0/916283700 >> 172.21.15.46:0/916283700 conn(0x7fb7140ce530 msgr2=0x7fb7140ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:57.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.027+0000 7fb718f9b700 1 -- 172.21.15.46:0/916283700 shutdown_connections 2024-01-27T01:07:57.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.027+0000 7fb718f9b700 1 -- 172.21.15.46:0/916283700 wait complete. 2024-01-27T01:07:57.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.029+0000 7fb718f9b700 1 Processor -- start 2024-01-27T01:07:57.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.029+0000 7fb718f9b700 1 -- start start 2024-01-27T01:07:57.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.030+0000 7fb718f9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7140ae520 0x7fb7140c6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:57.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.030+0000 7fb718f9b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7140b3fb0 0x7fb71414b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:57.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.030+0000 7fb718f9b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb714157340 0x7fb7140c3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.030+0000 7fb718f9b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7140c9db0 con 0x7fb7140ae520 2024-01-27T01:07:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.030+0000 7fb718f9b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7140c91a0 con 0x7fb714157340 2024-01-27T01:07:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.030+0000 7fb718f9b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7140c9480 con 0x7fb7140b3fb0 2024-01-27T01:07:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.030+0000 7fb71259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7140ae520 0x7fb7140c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.030+0000 7fb71259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7140ae520 0x7fb7140c6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51898/0 (socket says 172.21.15.46:51898) 2024-01-27T01:07:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.030+0000 7fb71259c700 1 -- 172.21.15.46:0/1679509718 learned_addr learned my addr 172.21.15.46:0/1679509718 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.030+0000 7fb711d9b700 1 --2- 172.21.15.46:0/1679509718 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7140b3fb0 0x7fb71414b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.030+0000 7fb712d9d700 1 --2- 172.21.15.46:0/1679509718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb714157340 0x7fb7140c3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.030+0000 7fb71259c700 1 -- 172.21.15.46:0/1679509718 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7140b3fb0 msgr2=0x7fb71414b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.030+0000 7fb71259c700 1 --2- 172.21.15.46:0/1679509718 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7140b3fb0 0x7fb71414b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.030+0000 7fb71259c700 1 -- 172.21.15.46:0/1679509718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb714157340 msgr2=0x7fb7140c3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.030+0000 7fb71259c700 1 --2- 172.21.15.46:0/1679509718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb714157340 0x7fb7140c3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.030+0000 7fb71259c700 1 -- 172.21.15.46:0/1679509718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb708009420 con 0x7fb7140ae520 2024-01-27T01:07:57.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.031+0000 7fb71259c700 1 --2- 172.21.15.46:0/1679509718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7140ae520 0x7fb7140c6030 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fb70400c790 tx=0x7fb704007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:57.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.031+0000 7fb7027fc700 1 -- 172.21.15.46:0/1679509718 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb704005b30 con 0x7fb7140ae520 2024-01-27T01:07:57.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.031+0000 7fb7027fc700 1 -- 172.21.15.46:0/1679509718 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb704005c90 con 0x7fb7140ae520 2024-01-27T01:07:57.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.031+0000 7fb718f9b700 1 -- 172.21.15.46:0/1679509718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7140c9760 con 0x7fb7140ae520 2024-01-27T01:07:57.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.031+0000 7fb7027fc700 1 -- 172.21.15.46:0/1679509718 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb704007300 con 0x7fb7140ae520 2024-01-27T01:07:57.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.031+0000 7fb718f9b700 1 -- 172.21.15.46:0/1679509718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7140c6570 con 0x7fb7140ae520 2024-01-27T01:07:57.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.032+0000 7fb718f9b700 1 -- 172.21.15.46:0/1679509718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7140c6760 con 0x7fb7140ae520 2024-01-27T01:07:57.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.033+0000 7fb7027fc700 1 -- 172.21.15.46:0/1679509718 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 140) v1 ==== 84947+0+0 (secure 0 0 0) 0x7fb70401b3b0 con 0x7fb7140ae520 2024-01-27T01:07:57.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.033+0000 7fb7027fc700 1 --2- 172.21.15.46:0/1679509718 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f4067170 0x7fb6f4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:57.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.033+0000 7fb7027fc700 1 -- 172.21.15.46:0/1679509718 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7811+0+0 (secure 0 0 0) 0x7fb704084630 con 0x7fb7140ae520 2024-01-27T01:07:57.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.033+0000 7fb711d9b700 1 -- 172.21.15.46:0/1679509718 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f4067170 msgr2=0x7fb6f4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:57.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.033+0000 7fb711d9b700 1 --2- 172.21.15.46:0/1679509718 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f4067170 0x7fb6f4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:57.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.038+0000 7fb7027fc700 1 -- 172.21.15.46:0/1679509718 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb704050dd0 con 0x7fb7140ae520 2024-01-27T01:07:57.055 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:57.055+0000 7f6ad5cab1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:07:57.104 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:57.104+0000 7f9b4c19d1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:07:57.133 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:57.133+0000 7f6ad5cab1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:07:57.176 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:57.176+0000 7f9b4c19d1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:07:57.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.234+0000 7fb711d9b700 1 -- 172.21.15.46:0/1679509718 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f4067170 msgr2=0x7fb6f4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:57.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.234+0000 7fb711d9b700 1 --2- 172.21.15.46:0/1679509718 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f4067170 0x7fb6f4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:57.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.248+0000 7fb718f9b700 1 -- 172.21.15.46:0/1679509718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb71404c710 con 0x7fb7140ae520 2024-01-27T01:07:57.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.249+0000 7fb7027fc700 1 -- 172.21.15.46:0/1679509718 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) v1 ==== 74+0+20181 (secure 0 0 0) 0x7fb704017020 con 0x7fb7140ae520 2024-01-27T01:07:57.249 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:57.249 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":248,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:07:53.996939+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:07:25.886791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-01-27T01:07:26.340041+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:07:39.209497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:07:40.386606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/2241845238":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1948052707":"2024-01-28T01:07:53.996892+0000","172.21.15.106:6836/62434":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1918093439":"2024-01-28T01:07:53.996892+0000","172.21.15.106:6837/62434":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/469270357":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1388668891":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1199239718":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/2250157080":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/2626066195":"2024-01-28T01:07:53.996892+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:07:57.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.251+0000 7fb718f9b700 1 -- 172.21.15.46:0/1679509718 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f4067170 msgr2=0x7fb6f4069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:57.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.251+0000 7fb718f9b700 1 --2- 172.21.15.46:0/1679509718 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f4067170 0x7fb6f4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.251+0000 7fb718f9b700 1 -- 172.21.15.46:0/1679509718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7140ae520 msgr2=0x7fb7140c6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:57.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.251+0000 7fb718f9b700 1 --2- 172.21.15.46:0/1679509718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7140ae520 0x7fb7140c6030 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fb70400c790 tx=0x7fb704007ea0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.251+0000 7fb718f9b700 1 -- 172.21.15.46:0/1679509718 shutdown_connections 2024-01-27T01:07:57.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.251+0000 7fb718f9b700 1 --2- 172.21.15.46:0/1679509718 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6f4067170 0x7fb6f4069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.251+0000 7fb718f9b700 1 --2- 172.21.15.46:0/1679509718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7140ae520 0x7fb7140c6030 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.251+0000 7fb718f9b700 1 --2- 172.21.15.46:0/1679509718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb714157340 0x7fb7140c3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.251+0000 7fb718f9b700 1 --2- 172.21.15.46:0/1679509718 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7140b3fb0 0x7fb71414b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.251+0000 7fb718f9b700 1 -- 172.21.15.46:0/1679509718 >> 172.21.15.46:0/1679509718 conn(0x7fb7140ce530 msgr2=0x7fb71415a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:57.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.251+0000 7fb718f9b700 1 -- 172.21.15.46:0/1679509718 shutdown_connections 2024-01-27T01:07:57.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.251+0000 7fb718f9b700 1 -- 172.21.15.46:0/1679509718 wait complete. 2024-01-27T01:07:57.265 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-27T01:07:25.886791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '242', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2024-01-27T01:07:26.340041+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '243', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-01-27T01:07:39.209497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '246', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2024-01-27T01:07:40.386606+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '247', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}}] 2024-01-27T01:07:57.266 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-27T01:07:57.276 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:57.276+0000 7f6ad5cab1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:07:57.316 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:57.317+0000 7f9b4c19d1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:07:57.363 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:57.363+0000 7f6ad5cab1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:07:57.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.404+0000 7f6667b22700 1 Processor -- start 2024-01-27T01:07:57.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.404+0000 7f6667b22700 1 -- start start 2024-01-27T01:07:57.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.405+0000 7f6667b22700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6660159da0 0x7f66600ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:57.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.405+0000 7f6667b22700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f666015a200 con 0x7f66601599d0 2024-01-27T01:07:57.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.405+0000 7f6667b22700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66600b1a60 con 0x7f666015d8c0 2024-01-27T01:07:57.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.405+0000 7f6667b22700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66600b11c0 con 0x7f6660159da0 2024-01-27T01:07:57.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.406+0000 7f66658be700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f66601599d0 0x7f66600b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60518/0 (socket says 172.21.15.46:60518) 2024-01-27T01:07:57.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.406+0000 7f66658be700 1 -- 172.21.15.46:0/3641553546 learned_addr learned my addr 172.21.15.46:0/3641553546 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:57.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.406+0000 7f66650bd700 1 --2- 172.21.15.46:0/3641553546 >> v2:172.21.15.106:3300/0 conn(0x7f6660159da0 0x7f66600ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:57.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.406+0000 7f66648bc700 1 -- 172.21.15.46:0/3641553546 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092623651 0 0) 0x7f66600b1a60 con 0x7f666015d8c0 2024-01-27T01:07:57.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.406+0000 7f66648bc700 1 -- 172.21.15.46:0/3641553546 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6648003570 con 0x7f666015d8c0 2024-01-27T01:07:57.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.406+0000 7f66648bc700 1 -- 172.21.15.46:0/3641553546 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381537203 0 0) 0x7f666015a200 con 0x7f66601599d0 2024-01-27T01:07:57.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.406+0000 7f66648bc700 1 -- 172.21.15.46:0/3641553546 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66600b1a60 con 0x7f66601599d0 2024-01-27T01:07:57.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.406+0000 7f66650bd700 1 -- 172.21.15.46:0/3641553546 >> v1:172.21.15.106:6789/0 conn(0x7f666015d8c0 legacy=0x7f66600b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:57.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.406+0000 7f66650bd700 1 -- 172.21.15.46:0/3641553546 >> v1:172.21.15.46:6789/0 conn(0x7f66601599d0 legacy=0x7f66600b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:57.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.406+0000 7f66650bd700 1 -- 172.21.15.46:0/3641553546 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66600b14a0 con 0x7f6660159da0 2024-01-27T01:07:57.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.406+0000 7f66650bd700 1 --2- 172.21.15.46:0/3641553546 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660159da0 0x7f66600ab1c0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f66540098e0 tx=0x7f66540091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a262c10df958abce server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:57.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.406+0000 7f66648bc700 1 -- 172.21.15.46:0/3641553546 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6654011630 con 0x7f6660159da0 2024-01-27T01:07:57.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.406+0000 7f66648bc700 1 -- 172.21.15.46:0/3641553546 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6654011790 con 0x7f6660159da0 2024-01-27T01:07:57.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.406+0000 7f66648bc700 1 -- 172.21.15.46:0/3641553546 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6654011b30 con 0x7f6660159da0 2024-01-27T01:07:57.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.407+0000 7f6667b22700 1 -- 172.21.15.46:0/3641553546 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660159da0 msgr2=0x7f66600ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:57.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.407+0000 7f6667b22700 1 --2- 172.21.15.46:0/3641553546 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660159da0 0x7f66600ab1c0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f66540098e0 tx=0x7f66540091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.407+0000 7f6667b22700 1 -- 172.21.15.46:0/3641553546 shutdown_connections 2024-01-27T01:07:57.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.407+0000 7f6667b22700 1 --2- 172.21.15.46:0/3641553546 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6660159da0 0x7f66600ab1c0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.407+0000 7f6667b22700 1 -- 172.21.15.46:0/3641553546 >> 172.21.15.46:0/3641553546 conn(0x7f666006f840 msgr2=0x7f6660087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:57.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.407+0000 7f6667b22700 1 -- 172.21.15.46:0/3641553546 shutdown_connections 2024-01-27T01:07:57.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.407+0000 7f6667b22700 1 -- 172.21.15.46:0/3641553546 wait complete. 2024-01-27T01:07:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.408+0000 7f6667b22700 1 Processor -- start 2024-01-27T01:07:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.408+0000 7f6667b22700 1 -- start start 2024-01-27T01:07:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.409+0000 7f6667b22700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66601599d0 0x7f666015bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.409+0000 7f6667b22700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6660159da0 0x7f666015a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.409+0000 7f6667b22700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f666015d8c0 0x7f666015a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.409+0000 7f6667b22700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66600ab960 con 0x7f66601599d0 2024-01-27T01:07:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.409+0000 7f6667b22700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66600abc40 con 0x7f666015d8c0 2024-01-27T01:07:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.409+0000 7f6667b22700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66600abf20 con 0x7f6660159da0 2024-01-27T01:07:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.409+0000 7f66658be700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66601599d0 0x7f666015bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.409+0000 7f66658be700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66601599d0 0x7f666015bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51900/0 (socket says 172.21.15.46:51900) 2024-01-27T01:07:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.409+0000 7f66658be700 1 -- 172.21.15.46:0/1835963670 learned_addr learned my addr 172.21.15.46:0/1835963670 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.409+0000 7f66650bd700 1 --2- 172.21.15.46:0/1835963670 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6660159da0 0x7f666015a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.409+0000 7f66660bf700 1 --2- 172.21.15.46:0/1835963670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f666015d8c0 0x7f666015a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.410+0000 7f66658be700 1 -- 172.21.15.46:0/1835963670 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6660159da0 msgr2=0x7f666015a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:57.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.410+0000 7f66658be700 1 --2- 172.21.15.46:0/1835963670 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6660159da0 0x7f666015a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.410+0000 7f66658be700 1 -- 172.21.15.46:0/1835963670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f666015d8c0 msgr2=0x7f666015a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:57.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.410+0000 7f66658be700 1 --2- 172.21.15.46:0/1835963670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f666015d8c0 0x7f666015a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.410+0000 7f66658be700 1 -- 172.21.15.46:0/1835963670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6654009420 con 0x7f66601599d0 2024-01-27T01:07:57.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.410+0000 7f66658be700 1 --2- 172.21.15.46:0/1835963670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66601599d0 0x7f666015bf90 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f665000c790 tx=0x7f6650007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:57.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.410+0000 7f665dffb700 1 -- 172.21.15.46:0/1835963670 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66500072e0 con 0x7f66601599d0 2024-01-27T01:07:57.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.410+0000 7f665dffb700 1 -- 172.21.15.46:0/1835963670 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6650007440 con 0x7f66601599d0 2024-01-27T01:07:57.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.410+0000 7f6667b22700 1 -- 172.21.15.46:0/1835963670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66600ac200 con 0x7f66601599d0 2024-01-27T01:07:57.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.410+0000 7f665dffb700 1 -- 172.21.15.46:0/1835963670 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66500057f0 con 0x7f66601599d0 2024-01-27T01:07:57.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.410+0000 7f6667b22700 1 -- 172.21.15.46:0/1835963670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6660169d30 con 0x7f66601599d0 2024-01-27T01:07:57.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.412+0000 7f6667b22700 1 -- 172.21.15.46:0/1835963670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f666004c710 con 0x7f66601599d0 2024-01-27T01:07:57.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.413+0000 7f665dffb700 1 -- 172.21.15.46:0/1835963670 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 140) v1 ==== 84947+0+0 (secure 0 0 0) 0x7f6650005be0 con 0x7f66601599d0 2024-01-27T01:07:57.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.413+0000 7f665dffb700 1 --2- 172.21.15.46:0/1835963670 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f664c067170 0x7f664c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:57.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.413+0000 7f665dffb700 1 -- 172.21.15.46:0/1835963670 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7811+0+0 (secure 0 0 0) 0x7f66500849a0 con 0x7f66601599d0 2024-01-27T01:07:57.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.413+0000 7f66650bd700 1 -- 172.21.15.46:0/1835963670 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f664c067170 msgr2=0x7f664c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:57.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.413+0000 7f66650bd700 1 --2- 172.21.15.46:0/1835963670 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f664c067170 0x7f664c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:57.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.417+0000 7f665dffb700 1 -- 172.21.15.46:0/1835963670 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f66500511f0 con 0x7f66601599d0 2024-01-27T01:07:57.444 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:57.445+0000 7f9b4c19d1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:07:57.575 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:57.575+0000 7f6ad5cab1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:07:57.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.614+0000 7f66650bd700 1 -- 172.21.15.46:0/1835963670 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f664c067170 msgr2=0x7f664c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:57.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.614+0000 7f66650bd700 1 --2- 172.21.15.46:0/1835963670 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f664c067170 0x7f664c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:57.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.688+0000 7f6667b22700 1 -- 172.21.15.46:0/1835963670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f66600b1780 con 0x7f66601599d0 2024-01-27T01:07:57.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.688+0000 7f665dffb700 1 -- 172.21.15.46:0/1835963670 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v248) v1 ==== 93+0+10 (secure 0 0 0) 0x7f66500170a0 con 0x7f66601599d0 2024-01-27T01:07:57.689 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-27T01:07:57.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.691+0000 7f6667b22700 1 -- 172.21.15.46:0/1835963670 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f664c067170 msgr2=0x7f664c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:57.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.691+0000 7f6667b22700 1 --2- 172.21.15.46:0/1835963670 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f664c067170 0x7f664c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.691+0000 7f6667b22700 1 -- 172.21.15.46:0/1835963670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66601599d0 msgr2=0x7f666015bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:57.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.691+0000 7f6667b22700 1 --2- 172.21.15.46:0/1835963670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66601599d0 0x7f666015bf90 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f665000c790 tx=0x7f6650007c80 comp rx=0 tx=0).stop 2024-01-27T01:07:57.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.691+0000 7f6667b22700 1 -- 172.21.15.46:0/1835963670 shutdown_connections 2024-01-27T01:07:57.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.691+0000 7f6667b22700 1 --2- 172.21.15.46:0/1835963670 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f664c067170 0x7f664c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.691+0000 7f6667b22700 1 --2- 172.21.15.46:0/1835963670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66601599d0 0x7f666015bf90 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.691+0000 7f6667b22700 1 --2- 172.21.15.46:0/1835963670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f666015d8c0 0x7f666015a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.691+0000 7f6667b22700 1 --2- 172.21.15.46:0/1835963670 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6660159da0 0x7f666015a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.691+0000 7f6667b22700 1 -- 172.21.15.46:0/1835963670 >> 172.21.15.46:0/1835963670 conn(0x7f666006f840 msgr2=0x7f6660068b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:57.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.691+0000 7f6667b22700 1 -- 172.21.15.46:0/1835963670 shutdown_connections 2024-01-27T01:07:57.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.691+0000 7f6667b22700 1 -- 172.21.15.46:0/1835963670 wait complete. 2024-01-27T01:07:57.706 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-27T01:07:57.720 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:57.720+0000 7f6ad5cab1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:07:57.761 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:57.761+0000 7f9b4c19d1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:07:57.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.857+0000 7fa2e1624700 1 Processor -- start 2024-01-27T01:07:57.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.857+0000 7fa2e1624700 1 -- start start 2024-01-27T01:07:57.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.858+0000 7fa2e1624700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2dc1599d0 0x7fa2dc159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:57.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.858+0000 7fa2e1624700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa2dc0ab1c0 0x7fa2dc0b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:57.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.858+0000 7fa2e1624700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2dc0b1a60 con 0x7fa2dc15d8c0 2024-01-27T01:07:57.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.858+0000 7fa2e1624700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2dc0ab5f0 con 0x7fa2dc0ab1c0 2024-01-27T01:07:57.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.858+0000 7fa2e1624700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc0b0f40 con 0x7fa2dc1599d0 2024-01-27T01:07:57.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.858+0000 7fa2daffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2dc1599d0 0x7fa2dc159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:57.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.858+0000 7fa2da7fc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa2dc0ab1c0 0x7fa2dc0b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:57.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.858+0000 7fa2db7fe700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fa2dc15d8c0 0x7fa2dc0b20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:41376/0 (socket says 172.21.15.46:41376) 2024-01-27T01:07:57.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.858+0000 7fa2db7fe700 1 -- 172.21.15.46:0/1865117116 learned_addr learned my addr 172.21.15.46:0/1865117116 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:57.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.858+0000 7fa2d9ffb700 1 -- 172.21.15.46:0/1865117116 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728014363 0 0) 0x7fa2dc0b1a60 con 0x7fa2dc15d8c0 2024-01-27T01:07:57.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.858+0000 7fa2daffd700 1 -- 172.21.15.46:0/1865117116 >> v2:172.21.15.106:3300/0 conn(0x7fa2dc0ab1c0 msgr2=0x7fa2dc0b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:57.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.858+0000 7fa2daffd700 1 --2- 172.21.15.46:0/1865117116 >> v2:172.21.15.106:3300/0 conn(0x7fa2dc0ab1c0 0x7fa2dc0b0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.858+0000 7fa2daffd700 1 -- 172.21.15.46:0/1865117116 >> v1:172.21.15.106:6789/0 conn(0x7fa2dc15d8c0 legacy=0x7fa2dc0b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:57.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.858+0000 7fa2daffd700 1 -- 172.21.15.46:0/1865117116 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2dc0b11c0 con 0x7fa2dc1599d0 2024-01-27T01:07:57.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.858+0000 7fa2da7fc700 1 --2- 172.21.15.46:0/1865117116 >> v2:172.21.15.106:3300/0 conn(0x7fa2dc0ab1c0 0x7fa2dc0b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:57.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.858+0000 7fa2daffd700 1 --2- 172.21.15.46:0/1865117116 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2dc1599d0 0x7fa2dc159da0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc0098d0 tx=0x7fa2cc009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=521f145d2468d4b7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:57.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.859+0000 7fa2d9ffb700 1 -- 172.21.15.46:0/1865117116 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc005c10 con 0x7fa2dc1599d0 2024-01-27T01:07:57.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.859+0000 7fa2d9ffb700 1 -- 172.21.15.46:0/1865117116 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2cc005d70 con 0x7fa2dc1599d0 2024-01-27T01:07:57.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.859+0000 7fa2d9ffb700 1 -- 172.21.15.46:0/1865117116 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc006280 con 0x7fa2dc1599d0 2024-01-27T01:07:57.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.859+0000 7fa2e1624700 1 -- 172.21.15.46:0/1865117116 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2dc1599d0 msgr2=0x7fa2dc159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:57.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.859+0000 7fa2e1624700 1 --2- 172.21.15.46:0/1865117116 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2dc1599d0 0x7fa2dc159da0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc0098d0 tx=0x7fa2cc009120 comp rx=0 tx=0).stop 2024-01-27T01:07:57.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.859+0000 7fa2e1624700 1 -- 172.21.15.46:0/1865117116 shutdown_connections 2024-01-27T01:07:57.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.859+0000 7fa2e1624700 1 --2- 172.21.15.46:0/1865117116 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2dc1599d0 0x7fa2dc159da0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.859+0000 7fa2e1624700 1 --2- 172.21.15.46:0/1865117116 >> v2:172.21.15.106:3300/0 conn(0x7fa2dc0ab1c0 0x7fa2dc0b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.859+0000 7fa2e1624700 1 -- 172.21.15.46:0/1865117116 >> 172.21.15.46:0/1865117116 conn(0x7fa2dc06f840 msgr2=0x7fa2dc087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:57.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.859+0000 7fa2e1624700 1 -- 172.21.15.46:0/1865117116 shutdown_connections 2024-01-27T01:07:57.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.859+0000 7fa2e1624700 1 -- 172.21.15.46:0/1865117116 wait complete. 2024-01-27T01:07:57.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.861+0000 7fa2e1624700 1 Processor -- start 2024-01-27T01:07:57.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.861+0000 7fa2e1624700 1 -- start start 2024-01-27T01:07:57.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2e1624700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2dc0ab1c0 0x7fa2dc158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:57.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2e1624700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2dc1599d0 0x7fa2dc16a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:57.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2e1624700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2dc15d8c0 0x7fa2dc0aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:57.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2e1624700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc0ae450 con 0x7fa2dc15d8c0 2024-01-27T01:07:57.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2e1624700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc159030 con 0x7fa2dc1599d0 2024-01-27T01:07:57.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2e1624700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc0ad840 con 0x7fa2dc0ab1c0 2024-01-27T01:07:57.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2db7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2dc15d8c0 0x7fa2dc0aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:57.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2da7fc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2dc1599d0 0x7fa2dc16a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:57.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2daffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2dc0ab1c0 0x7fa2dc158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:57.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2db7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2dc15d8c0 0x7fa2dc0aecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51904/0 (socket says 172.21.15.46:51904) 2024-01-27T01:07:57.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2da7fc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2dc1599d0 0x7fa2dc16a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:34970/0 (socket says 172.21.15.46:34970) 2024-01-27T01:07:57.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2db7fe700 1 -- 172.21.15.46:0/2313728761 learned_addr learned my addr 172.21.15.46:0/2313728761 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:57.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2daffd700 1 -- 172.21.15.46:0/2313728761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2dc1599d0 msgr2=0x7fa2dc16a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:57.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2daffd700 1 --2- 172.21.15.46:0/2313728761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2dc1599d0 0x7fa2dc16a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2daffd700 1 -- 172.21.15.46:0/2313728761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2dc15d8c0 msgr2=0x7fa2dc0aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:57.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2daffd700 1 --2- 172.21.15.46:0/2313728761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2dc15d8c0 0x7fa2dc0aecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:57.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2daffd700 1 -- 172.21.15.46:0/2313728761 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d0009420 con 0x7fa2dc0ab1c0 2024-01-27T01:07:57.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2db7fe700 1 --2- 172.21.15.46:0/2313728761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2dc15d8c0 0x7fa2dc0aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:57.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2da7fc700 1 --2- 172.21.15.46:0/2313728761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2dc1599d0 0x7fa2dc16a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:57.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.862+0000 7fa2daffd700 1 --2- 172.21.15.46:0/2313728761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2dc0ab1c0 0x7fa2dc158af0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc015000 tx=0x7fa2cc016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:57.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.863+0000 7fa2c2ffd700 1 -- 172.21.15.46:0/2313728761 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc01a700 con 0x7fa2dc0ab1c0 2024-01-27T01:07:57.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.863+0000 7fa2c2ffd700 1 -- 172.21.15.46:0/2313728761 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2cc01a860 con 0x7fa2dc0ab1c0 2024-01-27T01:07:57.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.863+0000 7fa2e1624700 1 -- 172.21.15.46:0/2313728761 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2cc0093c0 con 0x7fa2dc0ab1c0 2024-01-27T01:07:57.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.863+0000 7fa2c2ffd700 1 -- 172.21.15.46:0/2313728761 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc01ac20 con 0x7fa2dc0ab1c0 2024-01-27T01:07:57.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.863+0000 7fa2e1624700 1 -- 172.21.15.46:0/2313728761 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2dc0ade00 con 0x7fa2dc0ab1c0 2024-01-27T01:07:57.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.864+0000 7fa2e1624700 1 -- 172.21.15.46:0/2313728761 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2dc04c710 con 0x7fa2dc0ab1c0 2024-01-27T01:07:57.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.865+0000 7fa2c2ffd700 1 -- 172.21.15.46:0/2313728761 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 140) v1 ==== 84947+0+0 (secure 0 0 0) 0x7fa2cc026430 con 0x7fa2dc0ab1c0 2024-01-27T01:07:57.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.865+0000 7fa2c2ffd700 1 --2- 172.21.15.46:0/2313728761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa2c8067170 0x7fa2c8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:57.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.865+0000 7fa2c2ffd700 1 -- 172.21.15.46:0/2313728761 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7811+0+0 (secure 0 0 0) 0x7fa2cc086910 con 0x7fa2dc0ab1c0 2024-01-27T01:07:57.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.865+0000 7fa2da7fc700 1 -- 172.21.15.46:0/2313728761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa2c8067170 msgr2=0x7fa2c8069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:57.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.865+0000 7fa2da7fc700 1 --2- 172.21.15.46:0/2313728761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa2c8067170 0x7fa2c8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:57.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:57.870+0000 7fa2c2ffd700 1 -- 172.21.15.46:0/2313728761 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa2cc0530b0 con 0x7fa2dc0ab1c0 2024-01-27T01:07:57.973 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:57.974+0000 7f9b4c19d1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:07:58.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.066+0000 7fa2da7fc700 1 -- 172.21.15.46:0/2313728761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa2c8067170 msgr2=0x7fa2c8069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:58.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.066+0000 7fa2da7fc700 1 --2- 172.21.15.46:0/2313728761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa2c8067170 0x7fa2c8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:58.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.079+0000 7fa2e1624700 1 -- 172.21.15.46:0/2313728761 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa2dc0b14a0 con 0x7fa2dc0ab1c0 2024-01-27T01:07:58.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.079+0000 7fa2c2ffd700 1 -- 172.21.15.46:0/2313728761 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v248) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa2cc0566d0 con 0x7fa2dc0ab1c0 2024-01-27T01:07:58.079 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-27T01:07:58.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.082+0000 7fa2e1624700 1 -- 172.21.15.46:0/2313728761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa2c8067170 msgr2=0x7fa2c8069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:58.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.082+0000 7fa2e1624700 1 --2- 172.21.15.46:0/2313728761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa2c8067170 0x7fa2c8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.082+0000 7fa2e1624700 1 -- 172.21.15.46:0/2313728761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2dc0ab1c0 msgr2=0x7fa2dc158af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:58.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.082+0000 7fa2e1624700 1 --2- 172.21.15.46:0/2313728761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2dc0ab1c0 0x7fa2dc158af0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc015000 tx=0x7fa2cc016000 comp rx=0 tx=0).stop 2024-01-27T01:07:58.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.082+0000 7fa2e1624700 1 -- 172.21.15.46:0/2313728761 shutdown_connections 2024-01-27T01:07:58.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.082+0000 7fa2e1624700 1 --2- 172.21.15.46:0/2313728761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa2c8067170 0x7fa2c8069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.082+0000 7fa2e1624700 1 --2- 172.21.15.46:0/2313728761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2dc15d8c0 0x7fa2dc0aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.082+0000 7fa2e1624700 1 --2- 172.21.15.46:0/2313728761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2dc0ab1c0 0x7fa2dc158af0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.082+0000 7fa2e1624700 1 --2- 172.21.15.46:0/2313728761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2dc1599d0 0x7fa2dc16a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.082+0000 7fa2e1624700 1 -- 172.21.15.46:0/2313728761 >> 172.21.15.46:0/2313728761 conn(0x7fa2dc06f840 msgr2=0x7fa2dc0d0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:58.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.082+0000 7fa2e1624700 1 -- 172.21.15.46:0/2313728761 shutdown_connections 2024-01-27T01:07:58.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.082+0000 7fa2e1624700 1 -- 172.21.15.46:0/2313728761 wait complete. 2024-01-27T01:07:58.094 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:58.094+0000 7f6ad5cab1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:07:58.096 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-27T01:07:58.206 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:58.206+0000 7f9b4c19d1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:07:58.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.232+0000 7fc700523700 1 Processor -- start 2024-01-27T01:07:58.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.232+0000 7fc700523700 1 -- start start 2024-01-27T01:07:58.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.233+0000 7fc700523700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc6f8068b00 0x7fc6f8068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:58.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.233+0000 7fc700523700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6f806a660 0x7fc6f8159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:58.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.233+0000 7fc700523700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc6f806add0 0x7fc6f806b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:58.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.233+0000 7fc700523700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f80a9a80 con 0x7fc6f806add0 2024-01-27T01:07:58.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.233+0000 7fc700523700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f806b790 con 0x7fc6f8068b00 2024-01-27T01:07:58.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.233+0000 7fc700523700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f806aab0 con 0x7fc6f806a660 2024-01-27T01:07:58.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.233+0000 7fc6feac0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc6f806add0 0x7fc6f806b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:58.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.233+0000 7fc6feac0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc6f806add0 0x7fc6f806b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51912/0 (socket says 172.21.15.46:51912) 2024-01-27T01:07:58.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.233+0000 7fc6fdabe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6f806a660 0x7fc6f8159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:58.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.233+0000 7fc6fe2bf700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc6f8068b00 0x7fc6f8068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:58.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.233+0000 7fc6feac0700 1 -- 172.21.15.46:0/994351045 learned_addr learned my addr 172.21.15.46:0/994351045 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:58.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.233+0000 7fc6feac0700 1 -- 172.21.15.46:0/994351045 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6f806a660 msgr2=0x7fc6f8159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.233+0000 7fc6feac0700 1 --2- 172.21.15.46:0/994351045 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6f806a660 0x7fc6f8159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.233+0000 7fc6feac0700 1 -- 172.21.15.46:0/994351045 >> v2:172.21.15.106:3300/0 conn(0x7fc6f8068b00 msgr2=0x7fc6f8068ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.233+0000 7fc6feac0700 1 --2- 172.21.15.46:0/994351045 >> v2:172.21.15.106:3300/0 conn(0x7fc6f8068b00 0x7fc6f8068ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.233+0000 7fc6feac0700 1 -- 172.21.15.46:0/994351045 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f815a3e0 con 0x7fc6f806add0 2024-01-27T01:07:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.233+0000 7fc6fe2bf700 1 --2- 172.21.15.46:0/994351045 >> v2:172.21.15.106:3300/0 conn(0x7fc6f8068b00 0x7fc6f8068ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.234+0000 7fc6feac0700 1 --2- 172.21.15.46:0/994351045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f806add0 0x7fc6f806b250 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7fc6f806dc50 tx=0x7fc6f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d217f400fccc7348 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.234+0000 7fc6fd2bd700 1 -- 172.21.15.46:0/994351045 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f000e070 con 0x7fc6f806add0 2024-01-27T01:07:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.234+0000 7fc6fd2bd700 1 -- 172.21.15.46:0/994351045 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f000b970 con 0x7fc6f806add0 2024-01-27T01:07:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.234+0000 7fc6fd2bd700 1 -- 172.21.15.46:0/994351045 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f00092e0 con 0x7fc6f806add0 2024-01-27T01:07:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.235+0000 7fc700523700 1 -- 172.21.15.46:0/994351045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f806add0 msgr2=0x7fc6f806b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.235+0000 7fc700523700 1 --2- 172.21.15.46:0/994351045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f806add0 0x7fc6f806b250 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7fc6f806dc50 tx=0x7fc6f0009840 comp rx=0 tx=0).stop 2024-01-27T01:07:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.235+0000 7fc700523700 1 -- 172.21.15.46:0/994351045 shutdown_connections 2024-01-27T01:07:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.235+0000 7fc700523700 1 --2- 172.21.15.46:0/994351045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f806add0 0x7fc6f806b250 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.235+0000 7fc700523700 1 --2- 172.21.15.46:0/994351045 >> v2:172.21.15.106:3300/0 conn(0x7fc6f8068b00 0x7fc6f8068ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.235+0000 7fc700523700 1 --2- 172.21.15.46:0/994351045 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6f806a660 0x7fc6f8159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.235+0000 7fc700523700 1 -- 172.21.15.46:0/994351045 >> 172.21.15.46:0/994351045 conn(0x7fc6f80ca100 msgr2=0x7fc6f806e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:58.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.235+0000 7fc700523700 1 -- 172.21.15.46:0/994351045 shutdown_connections 2024-01-27T01:07:58.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.235+0000 7fc700523700 1 -- 172.21.15.46:0/994351045 wait complete. 2024-01-27T01:07:58.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.236+0000 7fc700523700 1 Processor -- start 2024-01-27T01:07:58.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.236+0000 7fc700523700 1 -- start start 2024-01-27T01:07:58.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.237+0000 7fc700523700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6f8068b00 0x7fc6f8156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:58.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.237+0000 7fc700523700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f806a660 0x7fc6f81697a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:58.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.237+0000 7fc700523700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6f806add0 0x7fc6f81581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:58.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.237+0000 7fc700523700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f816a790 con 0x7fc6f806a660 2024-01-27T01:07:58.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.237+0000 7fc700523700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f816d300 con 0x7fc6f806add0 2024-01-27T01:07:58.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.237+0000 7fc700523700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f8156cc0 con 0x7fc6f8068b00 2024-01-27T01:07:58.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.237+0000 7fc6fdabe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f806a660 0x7fc6f81697a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:58.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.237+0000 7fc6fdabe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f806a660 0x7fc6f81697a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51918/0 (socket says 172.21.15.46:51918) 2024-01-27T01:07:58.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.237+0000 7fc6fdabe700 1 -- 172.21.15.46:0/4100241642 learned_addr learned my addr 172.21.15.46:0/4100241642 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:58.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.237+0000 7fc6fe2bf700 1 --2- 172.21.15.46:0/4100241642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6f8068b00 0x7fc6f8156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:58.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.237+0000 7fc6feac0700 1 --2- 172.21.15.46:0/4100241642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6f806add0 0x7fc6f81581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:58.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.237+0000 7fc6fdabe700 1 -- 172.21.15.46:0/4100241642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6f8068b00 msgr2=0x7fc6f8156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:58.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.237+0000 7fc6fdabe700 1 --2- 172.21.15.46:0/4100241642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6f8068b00 0x7fc6f8156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.237+0000 7fc6fdabe700 1 -- 172.21.15.46:0/4100241642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6f806add0 msgr2=0x7fc6f81581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:58.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.237+0000 7fc6fdabe700 1 --2- 172.21.15.46:0/4100241642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6f806add0 0x7fc6f81581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.237+0000 7fc6fdabe700 1 -- 172.21.15.46:0/4100241642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f40093c0 con 0x7fc6f806a660 2024-01-27T01:07:58.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.238+0000 7fc6fdabe700 1 --2- 172.21.15.46:0/4100241642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f806a660 0x7fc6f81697a0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fc6e800ec80 tx=0x7fc6e80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:58.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.238+0000 7fc6ee7fc700 1 -- 172.21.15.46:0/4100241642 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e800c850 con 0x7fc6f806a660 2024-01-27T01:07:58.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.238+0000 7fc6ee7fc700 1 -- 172.21.15.46:0/4100241642 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e800c9b0 con 0x7fc6f806a660 2024-01-27T01:07:58.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.238+0000 7fc700523700 1 -- 172.21.15.46:0/4100241642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f0009ae0 con 0x7fc6f806a660 2024-01-27T01:07:58.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.238+0000 7fc6ee7fc700 1 -- 172.21.15.46:0/4100241642 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e800cd50 con 0x7fc6f806a660 2024-01-27T01:07:58.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.238+0000 7fc700523700 1 -- 172.21.15.46:0/4100241642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f816a4b0 con 0x7fc6f806a660 2024-01-27T01:07:58.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.239+0000 7fc700523700 1 -- 172.21.15.46:0/4100241642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f816d490 con 0x7fc6f806a660 2024-01-27T01:07:58.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.240+0000 7fc6ee7fc700 1 -- 172.21.15.46:0/4100241642 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 140) v1 ==== 84947+0+0 (secure 0 0 0) 0x7fc6e8013680 con 0x7fc6f806a660 2024-01-27T01:07:58.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.240+0000 7fc6ee7fc700 1 --2- 172.21.15.46:0/4100241642 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc6dc067170 0x7fc6dc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:58.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.240+0000 7fc6fe2bf700 1 -- 172.21.15.46:0/4100241642 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc6dc067170 msgr2=0x7fc6dc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:58.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.240+0000 7fc6ee7fc700 1 -- 172.21.15.46:0/4100241642 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7811+0+0 (secure 0 0 0) 0x7fc6e8085350 con 0x7fc6f806a660 2024-01-27T01:07:58.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.240+0000 7fc6fe2bf700 1 --2- 172.21.15.46:0/4100241642 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc6dc067170 0x7fc6dc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:58.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.245+0000 7fc6ee7fc700 1 -- 172.21.15.46:0/4100241642 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc6e8051af0 con 0x7fc6f806a660 2024-01-27T01:07:58.278 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:58.278+0000 7f9b4c19d1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:07:58.369 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:58.369+0000 7f6ad5cab1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:07:58.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.441+0000 7fc6fe2bf700 1 -- 172.21.15.46:0/4100241642 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc6dc067170 msgr2=0x7fc6dc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:58.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.441+0000 7fc6fe2bf700 1 --2- 172.21.15.46:0/4100241642 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc6dc067170 0x7fc6dc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:58.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.479+0000 7fc700523700 1 -- 172.21.15.46:0/4100241642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc6f804c710 con 0x7fc6f806a660 2024-01-27T01:07:58.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.480+0000 7fc6ee7fc700 1 -- 172.21.15.46:0/4100241642 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v248) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc6e8055110 con 0x7fc6f806a660 2024-01-27T01:07:58.480 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-27T01:07:58.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.482+0000 7fc700523700 1 -- 172.21.15.46:0/4100241642 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc6dc067170 msgr2=0x7fc6dc069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:58.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.482+0000 7fc700523700 1 --2- 172.21.15.46:0/4100241642 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc6dc067170 0x7fc6dc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.482+0000 7fc700523700 1 -- 172.21.15.46:0/4100241642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f806a660 msgr2=0x7fc6f81697a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:58.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.482+0000 7fc700523700 1 --2- 172.21.15.46:0/4100241642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f806a660 0x7fc6f81697a0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fc6e800ec80 tx=0x7fc6e80091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.482+0000 7fc700523700 1 -- 172.21.15.46:0/4100241642 shutdown_connections 2024-01-27T01:07:58.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.482+0000 7fc700523700 1 --2- 172.21.15.46:0/4100241642 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc6dc067170 0x7fc6dc069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.482+0000 7fc700523700 1 --2- 172.21.15.46:0/4100241642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6f806add0 0x7fc6f81581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.482+0000 7fc700523700 1 --2- 172.21.15.46:0/4100241642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6f8068b00 0x7fc6f8156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.482+0000 7fc700523700 1 --2- 172.21.15.46:0/4100241642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f806a660 0x7fc6f81697a0 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.482+0000 7fc700523700 1 -- 172.21.15.46:0/4100241642 >> 172.21.15.46:0/4100241642 conn(0x7fc6f80ca100 msgr2=0x7fc6f80a9f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:58.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.482+0000 7fc700523700 1 -- 172.21.15.46:0/4100241642 shutdown_connections 2024-01-27T01:07:58.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.482+0000 7fc700523700 1 -- 172.21.15.46:0/4100241642 wait complete. 2024-01-27T01:07:58.497 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-01-27T01:07:58.633 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:58.633+0000 7f6ad5cab1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:07:58.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.634+0000 7f0af811e700 1 Processor -- start 2024-01-27T01:07:58.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.634+0000 7f0af811e700 1 -- start start 2024-01-27T01:07:58.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.635+0000 7f0af811e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0af0068ed0 0x7f0af006a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:58.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.635+0000 7f0af811e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0af006add0 0x7f0af006b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:58.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.635+0000 7f0af811e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af00a9a80 con 0x7f0af0068b00 2024-01-27T01:07:58.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.635+0000 7f0af811e700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0af0069320 con 0x7f0af0068ed0 2024-01-27T01:07:58.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.635+0000 7f0af811e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0af006b790 con 0x7f0af006add0 2024-01-27T01:07:58.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.635+0000 7f0af66bb700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0af006add0 0x7f0af006b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:58.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.635+0000 7f0af56b9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0af0068ed0 0x7f0af006a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:58.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.636+0000 7f0af56b9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0af0068ed0 0x7f0af006a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:34998/0 (socket says 172.21.15.46:34998) 2024-01-27T01:07:58.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.636+0000 7f0af56b9700 1 -- 172.21.15.46:0/3104790852 learned_addr learned my addr 172.21.15.46:0/3104790852 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:58.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.636+0000 7f0af5eba700 1 --1- 172.21.15.46:0/3104790852 >> v1:172.21.15.106:6789/0 conn(0x7f0af0068b00 0x7f0af0159ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:41392/0 (socket says 172.21.15.46:41392) 2024-01-27T01:07:58.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.636+0000 7f0af56b9700 1 -- 172.21.15.46:0/3104790852 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0af006add0 msgr2=0x7f0af006b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:58.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.636+0000 7f0af56b9700 1 --2- 172.21.15.46:0/3104790852 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0af006add0 0x7f0af006b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.636+0000 7f0af56b9700 1 -- 172.21.15.46:0/3104790852 >> v1:172.21.15.106:6789/0 conn(0x7f0af0068b00 legacy=0x7f0af0159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:58.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.636+0000 7f0af56b9700 1 -- 172.21.15.46:0/3104790852 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af015a680 con 0x7f0af0068ed0 2024-01-27T01:07:58.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.636+0000 7f0af4eb8700 1 -- 172.21.15.46:0/3104790852 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024213881 0 0) 0x7f0af00a9a80 con 0x7f0af0068b00 2024-01-27T01:07:58.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.636+0000 7f0af56b9700 1 --2- 172.21.15.46:0/3104790852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0af0068ed0 0x7f0af006a660 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f0ae40099b0 tx=0x7f0ae4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0a48d3743d6faed server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:58.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.636+0000 7f0af4eb8700 1 -- 172.21.15.46:0/3104790852 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4009ea0 con 0x7f0af0068ed0 2024-01-27T01:07:58.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.636+0000 7f0af4eb8700 1 -- 172.21.15.46:0/3104790852 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ae400b860 con 0x7f0af0068ed0 2024-01-27T01:07:58.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.636+0000 7f0af4eb8700 1 -- 172.21.15.46:0/3104790852 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae400bc00 con 0x7f0af0068ed0 2024-01-27T01:07:58.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.637+0000 7f0af811e700 1 -- 172.21.15.46:0/3104790852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0af0068ed0 msgr2=0x7f0af006a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:58.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.637+0000 7f0af811e700 1 --2- 172.21.15.46:0/3104790852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0af0068ed0 0x7f0af006a660 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f0ae40099b0 tx=0x7f0ae4009250 comp rx=0 tx=0).stop 2024-01-27T01:07:58.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.637+0000 7f0af811e700 1 -- 172.21.15.46:0/3104790852 shutdown_connections 2024-01-27T01:07:58.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.637+0000 7f0af811e700 1 --2- 172.21.15.46:0/3104790852 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0af006add0 0x7f0af006b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.637+0000 7f0af811e700 1 --2- 172.21.15.46:0/3104790852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0af0068ed0 0x7f0af006a660 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.637+0000 7f0af811e700 1 -- 172.21.15.46:0/3104790852 >> 172.21.15.46:0/3104790852 conn(0x7f0af00ca100 msgr2=0x7f0af006e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:58.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.637+0000 7f0af811e700 1 -- 172.21.15.46:0/3104790852 shutdown_connections 2024-01-27T01:07:58.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.637+0000 7f0af811e700 1 -- 172.21.15.46:0/3104790852 wait complete. 2024-01-27T01:07:58.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.638+0000 7f0af811e700 1 Processor -- start 2024-01-27T01:07:58.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.638+0000 7f0af811e700 1 -- start start 2024-01-27T01:07:58.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.639+0000 7f0af811e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0af0068b00 0x7f0af0145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:58.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.639+0000 7f0af811e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0af0068ed0 0x7f0af016c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:58.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.639+0000 7f0af811e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0af006add0 0x7f0af016ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:58.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.639+0000 7f0af811e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af016c290 con 0x7f0af006add0 2024-01-27T01:07:58.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.639+0000 7f0af811e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af0169c80 con 0x7f0af0068b00 2024-01-27T01:07:58.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.639+0000 7f0af811e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0af0145c50 con 0x7f0af0068ed0 2024-01-27T01:07:58.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.639+0000 7f0af66bb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0af006add0 0x7f0af016ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:58.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.639+0000 7f0af5eba700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0af0068b00 0x7f0af0145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:58.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.639+0000 7f0af66bb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0af006add0 0x7f0af016ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51934/0 (socket says 172.21.15.46:51934) 2024-01-27T01:07:58.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.639+0000 7f0af66bb700 1 -- 172.21.15.46:0/1588218556 learned_addr learned my addr 172.21.15.46:0/1588218556 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:58.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.639+0000 7f0af56b9700 1 --2- 172.21.15.46:0/1588218556 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0af0068ed0 0x7f0af016c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:58.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.640+0000 7f0af66bb700 1 -- 172.21.15.46:0/1588218556 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0af0068ed0 msgr2=0x7f0af016c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:58.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.640+0000 7f0af66bb700 1 --2- 172.21.15.46:0/1588218556 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0af0068ed0 0x7f0af016c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.640+0000 7f0af66bb700 1 -- 172.21.15.46:0/1588218556 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0af0068b00 msgr2=0x7f0af0145710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:58.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.640+0000 7f0af66bb700 1 --2- 172.21.15.46:0/1588218556 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0af0068b00 0x7f0af0145710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.640+0000 7f0af66bb700 1 -- 172.21.15.46:0/1588218556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae40094f0 con 0x7f0af006add0 2024-01-27T01:07:58.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.640+0000 7f0af5eba700 1 --2- 172.21.15.46:0/1588218556 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0af0068b00 0x7f0af0145710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:58.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.640+0000 7f0af56b9700 1 --2- 172.21.15.46:0/1588218556 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0af0068ed0 0x7f0af016c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:58.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.640+0000 7f0af66bb700 1 --2- 172.21.15.46:0/1588218556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0af006add0 0x7f0af016ab10 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f0ae800ea50 tx=0x7f0ae80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:58.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.640+0000 7f0aedffb700 1 -- 172.21.15.46:0/1588218556 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae800ee90 con 0x7f0af006add0 2024-01-27T01:07:58.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.640+0000 7f0aedffb700 1 -- 172.21.15.46:0/1588218556 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ae800c850 con 0x7f0af006add0 2024-01-27T01:07:58.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.640+0000 7f0af811e700 1 -- 172.21.15.46:0/1588218556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0af016cc60 con 0x7f0af006add0 2024-01-27T01:07:58.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.640+0000 7f0aedffb700 1 -- 172.21.15.46:0/1588218556 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae800cc10 con 0x7f0af006add0 2024-01-27T01:07:58.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.640+0000 7f0af811e700 1 -- 172.21.15.46:0/1588218556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0af016b960 con 0x7f0af006add0 2024-01-27T01:07:58.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.642+0000 7f0af811e700 1 -- 172.21.15.46:0/1588218556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0af016c400 con 0x7f0af006add0 2024-01-27T01:07:58.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.642+0000 7f0aedffb700 1 -- 172.21.15.46:0/1588218556 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 140) v1 ==== 84947+0+0 (secure 0 0 0) 0x7f0ae8013470 con 0x7f0af006add0 2024-01-27T01:07:58.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.642+0000 7f0aedffb700 1 --2- 172.21.15.46:0/1588218556 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0adc067170 0x7f0adc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:58.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.643+0000 7f0af5eba700 1 -- 172.21.15.46:0/1588218556 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0adc067170 msgr2=0x7f0adc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:58.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.643+0000 7f0aedffb700 1 -- 172.21.15.46:0/1588218556 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7811+0+0 (secure 0 0 0) 0x7f0ae8085320 con 0x7f0af006add0 2024-01-27T01:07:58.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.643+0000 7f0af5eba700 1 --2- 172.21.15.46:0/1588218556 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0adc067170 0x7f0adc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:58.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.647+0000 7f0aedffb700 1 -- 172.21.15.46:0/1588218556 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0ae8051ac0 con 0x7f0af006add0 2024-01-27T01:07:58.718 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:58.718+0000 7f6ad5cab1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:07:58.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.844+0000 7f0af5eba700 1 -- 172.21.15.46:0/1588218556 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0adc067170 msgr2=0x7f0adc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:58.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.844+0000 7f0af5eba700 1 --2- 172.21.15.46:0/1588218556 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0adc067170 0x7f0adc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:58.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.856+0000 7f0af811e700 1 -- 172.21.15.46:0/1588218556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0af004c710 con 0x7f0af006add0 2024-01-27T01:07:58.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.856+0000 7f0aedffb700 1 -- 172.21.15.46:0/1588218556 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v248) v1 ==== 107+0+11 (secure 0 0 0) 0x7f0ae80550e0 con 0x7f0af006add0 2024-01-27T01:07:58.856 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-27T01:07:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.858+0000 7f0af811e700 1 -- 172.21.15.46:0/1588218556 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0adc067170 msgr2=0x7f0adc069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.858+0000 7f0af811e700 1 --2- 172.21.15.46:0/1588218556 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0adc067170 0x7f0adc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.858+0000 7f0af811e700 1 -- 172.21.15.46:0/1588218556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0af006add0 msgr2=0x7f0af016ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.858+0000 7f0af811e700 1 --2- 172.21.15.46:0/1588218556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0af006add0 0x7f0af016ab10 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f0ae800ea50 tx=0x7f0ae80091a0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.859+0000 7f0af811e700 1 -- 172.21.15.46:0/1588218556 shutdown_connections 2024-01-27T01:07:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.859+0000 7f0af811e700 1 --2- 172.21.15.46:0/1588218556 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0adc067170 0x7f0adc069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.859+0000 7f0af811e700 1 --2- 172.21.15.46:0/1588218556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0af006add0 0x7f0af016ab10 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.859+0000 7f0af811e700 1 --2- 172.21.15.46:0/1588218556 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0af0068b00 0x7f0af0145710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.859+0000 7f0af811e700 1 --2- 172.21.15.46:0/1588218556 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0af0068ed0 0x7f0af016c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.859+0000 7f0af811e700 1 -- 172.21.15.46:0/1588218556 >> 172.21.15.46:0/1588218556 conn(0x7f0af00ca100 msgr2=0x7f0af00cb6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.859+0000 7f0af811e700 1 -- 172.21.15.46:0/1588218556 shutdown_connections 2024-01-27T01:07:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:58.859+0000 7f0af811e700 1 -- 172.21.15.46:0/1588218556 wait complete. 2024-01-27T01:07:58.873 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-01-27T01:07:58.876 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:58.877+0000 7f9b4c19d1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:07:58.952 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:58.952+0000 7f9b4c19d1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:07:59.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.010+0000 7f957f6c1700 1 Processor -- start 2024-01-27T01:07:59.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.010+0000 7f957f6c1700 1 -- start start 2024-01-27T01:07:59.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.011+0000 7f957f6c1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9578159cb0 0x7f95780cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:59.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.011+0000 7f957f6c1700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f95780afb40 0x7f95780c5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:59.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.011+0000 7f957f6c1700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95780cc100 con 0x7f95780cf050 2024-01-27T01:07:59.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.011+0000 7f957f6c1700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95780cb850 con 0x7f95780afb40 2024-01-27T01:07:59.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.011+0000 7f957f6c1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95780cbaf0 con 0x7f9578159cb0 2024-01-27T01:07:59.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.012+0000 7f957d45d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9578159cb0 0x7f95780cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:59.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.012+0000 7f957dc5e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f95780afb40 0x7f95780c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:59.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.012+0000 7f957cc5c700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f95780cf050 0x7f95780af430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:41408/0 (socket says 172.21.15.46:41408) 2024-01-27T01:07:59.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.012+0000 7f957cc5c700 1 -- 172.21.15.46:0/3965688647 learned_addr learned my addr 172.21.15.46:0/3965688647 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:59.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.012+0000 7f956ffff700 1 -- 172.21.15.46:0/3965688647 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585983258 0 0) 0x7f95780cc100 con 0x7f95780cf050 2024-01-27T01:07:59.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.012+0000 7f956ffff700 1 -- 172.21.15.46:0/3965688647 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9564003570 con 0x7f95780cf050 2024-01-27T01:07:59.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.012+0000 7f957d45d700 1 -- 172.21.15.46:0/3965688647 >> v2:172.21.15.106:3300/0 conn(0x7f95780afb40 msgr2=0x7f95780c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:59.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.012+0000 7f957d45d700 1 --2- 172.21.15.46:0/3965688647 >> v2:172.21.15.106:3300/0 conn(0x7f95780afb40 0x7f95780c5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.012+0000 7f957d45d700 1 -- 172.21.15.46:0/3965688647 >> v1:172.21.15.106:6789/0 conn(0x7f95780cf050 legacy=0x7f95780af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:59.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.012+0000 7f957d45d700 1 -- 172.21.15.46:0/3965688647 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95780cfc80 con 0x7f9578159cb0 2024-01-27T01:07:59.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.012+0000 7f957dc5e700 1 --2- 172.21.15.46:0/3965688647 >> v2:172.21.15.106:3300/0 conn(0x7f95780afb40 0x7f95780c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:59.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.012+0000 7f957d45d700 1 --2- 172.21.15.46:0/3965688647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9578159cb0 0x7f95780cc590 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f95740098d0 tx=0x7f9574009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc9df20b78873c33 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:59.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.012+0000 7f956ffff700 1 -- 172.21.15.46:0/3965688647 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9574005c10 con 0x7f9578159cb0 2024-01-27T01:07:59.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.012+0000 7f956ffff700 1 -- 172.21.15.46:0/3965688647 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9574005d70 con 0x7f9578159cb0 2024-01-27T01:07:59.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.012+0000 7f956ffff700 1 -- 172.21.15.46:0/3965688647 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9574006280 con 0x7f9578159cb0 2024-01-27T01:07:59.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.013+0000 7f957f6c1700 1 -- 172.21.15.46:0/3965688647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9578159cb0 msgr2=0x7f95780cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:59.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.013+0000 7f957f6c1700 1 --2- 172.21.15.46:0/3965688647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9578159cb0 0x7f95780cc590 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f95740098d0 tx=0x7f9574009120 comp rx=0 tx=0).stop 2024-01-27T01:07:59.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.013+0000 7f957f6c1700 1 -- 172.21.15.46:0/3965688647 shutdown_connections 2024-01-27T01:07:59.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.013+0000 7f957f6c1700 1 --2- 172.21.15.46:0/3965688647 >> v2:172.21.15.106:3300/0 conn(0x7f95780afb40 0x7f95780c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.013+0000 7f957f6c1700 1 --2- 172.21.15.46:0/3965688647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9578159cb0 0x7f95780cc590 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.013+0000 7f957f6c1700 1 -- 172.21.15.46:0/3965688647 >> 172.21.15.46:0/3965688647 conn(0x7f9578087f60 msgr2=0x7f957806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:59.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.013+0000 7f957f6c1700 1 -- 172.21.15.46:0/3965688647 shutdown_connections 2024-01-27T01:07:59.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.013+0000 7f957f6c1700 1 -- 172.21.15.46:0/3965688647 wait complete. 2024-01-27T01:07:59.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.015+0000 7f957f6c1700 1 Processor -- start 2024-01-27T01:07:59.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.015+0000 7f957f6c1700 1 -- start start 2024-01-27T01:07:59.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.016+0000 7f957f6c1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95780afb40 0x7f9578140f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:59.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.016+0000 7f957f6c1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95780cf050 0x7f95780b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:59.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.016+0000 7f957f6c1700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9578159cb0 0x7f95780b3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:59.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.016+0000 7f957f6c1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95781414a0 con 0x7f95780cf050 2024-01-27T01:07:59.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.016+0000 7f957f6c1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95780b0670 con 0x7f95780afb40 2024-01-27T01:07:59.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.016+0000 7f957f6c1700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95780b0910 con 0x7f9578159cb0 2024-01-27T01:07:59.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.016+0000 7f957d45d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95780afb40 0x7f9578140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:59.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.016+0000 7f957cc5c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95780cf050 0x7f95780b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:59.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.016+0000 7f957dc5e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9578159cb0 0x7f95780b3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:59.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.016+0000 7f957d45d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95780afb40 0x7f9578140f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:35024/0 (socket says 172.21.15.46:35024) 2024-01-27T01:07:59.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.016+0000 7f957d45d700 1 -- 172.21.15.46:0/1138084408 learned_addr learned my addr 172.21.15.46:0/1138084408 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:59.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.017+0000 7f957d45d700 1 -- 172.21.15.46:0/1138084408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9578159cb0 msgr2=0x7f95780b3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:59.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.017+0000 7f957d45d700 1 --2- 172.21.15.46:0/1138084408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9578159cb0 0x7f95780b3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.017+0000 7f957d45d700 1 -- 172.21.15.46:0/1138084408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95780cf050 msgr2=0x7f95780b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:59.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.017+0000 7f957d45d700 1 --2- 172.21.15.46:0/1138084408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95780cf050 0x7f95780b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.017+0000 7f957d45d700 1 -- 172.21.15.46:0/1138084408 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9570009a20 con 0x7f95780afb40 2024-01-27T01:07:59.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.017+0000 7f957d45d700 1 --2- 172.21.15.46:0/1138084408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95780afb40 0x7f9578140f60 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f95780cbc30 tx=0x7f9574016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:59.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.017+0000 7f956d7fa700 1 -- 172.21.15.46:0/1138084408 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9574006a90 con 0x7f95780afb40 2024-01-27T01:07:59.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.017+0000 7f956d7fa700 1 -- 172.21.15.46:0/1138084408 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957401d7b0 con 0x7f95780afb40 2024-01-27T01:07:59.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.017+0000 7f957f6c1700 1 -- 172.21.15.46:0/1138084408 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95740093c0 con 0x7f95780afb40 2024-01-27T01:07:59.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.017+0000 7f956d7fa700 1 -- 172.21.15.46:0/1138084408 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957401db90 con 0x7f95780afb40 2024-01-27T01:07:59.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.017+0000 7f957f6c1700 1 -- 172.21.15.46:0/1138084408 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95780b0ed0 con 0x7f95780afb40 2024-01-27T01:07:59.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.018+0000 7f957f6c1700 1 -- 172.21.15.46:0/1138084408 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f957804c710 con 0x7f95780afb40 2024-01-27T01:07:59.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.019+0000 7f956d7fa700 1 -- 172.21.15.46:0/1138084408 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 140) v1 ==== 84947+0+0 (secure 0 0 0) 0x7f957401b480 con 0x7f95780afb40 2024-01-27T01:07:59.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.019+0000 7f956d7fa700 1 --2- 172.21.15.46:0/1138084408 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f955c067170 0x7f955c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:59.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.019+0000 7f956d7fa700 1 -- 172.21.15.46:0/1138084408 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7811+0+0 (secure 0 0 0) 0x7f9574086cd0 con 0x7f95780afb40 2024-01-27T01:07:59.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.019+0000 7f957cc5c700 1 -- 172.21.15.46:0/1138084408 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f955c067170 msgr2=0x7f955c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:59.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.019+0000 7f957cc5c700 1 --2- 172.21.15.46:0/1138084408 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f955c067170 0x7f955c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:59.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.028+0000 7f956d7fa700 1 -- 172.21.15.46:0/1138084408 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9574053460 con 0x7f95780afb40 2024-01-27T01:07:59.082 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:59.083+0000 7f9b4c19d1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:07:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.220+0000 7f957cc5c700 1 -- 172.21.15.46:0/1138084408 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f955c067170 msgr2=0x7f955c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.220+0000 7f957cc5c700 1 --2- 172.21.15.46:0/1138084408 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f955c067170 0x7f955c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:59.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.234+0000 7f957f6c1700 1 -- 172.21.15.46:0/1138084408 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f95780cf9a0 con 0x7f95780afb40 2024-01-27T01:07:59.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.235+0000 7f956d7fa700 1 -- 172.21.15.46:0/1138084408 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v248) v1 ==== 103+0+11 (secure 0 0 0) 0x7f957402a030 con 0x7f95780afb40 2024-01-27T01:07:59.235 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-27T01:07:59.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.237+0000 7f957f6c1700 1 -- 172.21.15.46:0/1138084408 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f955c067170 msgr2=0x7f955c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:59.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.237+0000 7f957f6c1700 1 --2- 172.21.15.46:0/1138084408 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f955c067170 0x7f955c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.237+0000 7f957f6c1700 1 -- 172.21.15.46:0/1138084408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95780afb40 msgr2=0x7f9578140f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:59.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.237+0000 7f957f6c1700 1 --2- 172.21.15.46:0/1138084408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95780afb40 0x7f9578140f60 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f95780cbc30 tx=0x7f9574016000 comp rx=0 tx=0).stop 2024-01-27T01:07:59.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.238+0000 7f957f6c1700 1 -- 172.21.15.46:0/1138084408 shutdown_connections 2024-01-27T01:07:59.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.238+0000 7f957f6c1700 1 --2- 172.21.15.46:0/1138084408 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f955c067170 0x7f955c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.238+0000 7f957f6c1700 1 --2- 172.21.15.46:0/1138084408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9578159cb0 0x7f95780b3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.238+0000 7f957f6c1700 1 --2- 172.21.15.46:0/1138084408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95780afb40 0x7f9578140f60 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.238+0000 7f957f6c1700 1 --2- 172.21.15.46:0/1138084408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95780cf050 0x7f95780b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.238+0000 7f957f6c1700 1 -- 172.21.15.46:0/1138084408 >> 172.21.15.46:0/1138084408 conn(0x7f9578087f60 msgr2=0x7f9578155570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:59.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.238+0000 7f957f6c1700 1 -- 172.21.15.46:0/1138084408 shutdown_connections 2024-01-27T01:07:59.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.238+0000 7f957f6c1700 1 -- 172.21.15.46:0/1138084408 wait complete. 2024-01-27T01:07:59.253 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:07:59.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.387+0000 7ff74093a700 1 Processor -- start 2024-01-27T01:07:59.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.387+0000 7ff74093a700 1 -- start start 2024-01-27T01:07:59.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.388+0000 7ff74093a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff73c0cf460 0x7ff73c0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:59.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.388+0000 7ff74093a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff73c15a0b0 con 0x7ff73c0c6520 2024-01-27T01:07:59.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.388+0000 7ff74093a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff73c0ae4d0 con 0x7ff73c0b2b40 2024-01-27T01:07:59.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.388+0000 7ff74093a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c0af070 con 0x7ff73c0cf460 2024-01-27T01:07:59.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.388+0000 7ff739d9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff73c0c6520 0x7ff73c0ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60522/0 (socket says 172.21.15.46:60522) 2024-01-27T01:07:59.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.388+0000 7ff739d9b700 1 -- 172.21.15.46:0/3236206076 learned_addr learned my addr 172.21.15.46:0/3236206076 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:07:59.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.388+0000 7ff73ad9d700 1 --2- 172.21.15.46:0/3236206076 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff73c0cf460 0x7ff73c0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:59.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.388+0000 7ff73959a700 1 -- 172.21.15.46:0/3236206076 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177050828 0 0) 0x7ff73c15a0b0 con 0x7ff73c0c6520 2024-01-27T01:07:59.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.389+0000 7ff73959a700 1 -- 172.21.15.46:0/3236206076 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff724003570 con 0x7ff73c0c6520 2024-01-27T01:07:59.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.389+0000 7ff73959a700 1 -- 172.21.15.46:0/3236206076 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212929883 0 0) 0x7ff73c0ae4d0 con 0x7ff73c0b2b40 2024-01-27T01:07:59.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.389+0000 7ff73959a700 1 -- 172.21.15.46:0/3236206076 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff73c15a0b0 con 0x7ff73c0b2b40 2024-01-27T01:07:59.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.389+0000 7ff73ad9d700 1 -- 172.21.15.46:0/3236206076 >> v1:172.21.15.106:6789/0 conn(0x7ff73c0b2b40 legacy=0x7ff73c0ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:59.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.389+0000 7ff73ad9d700 1 -- 172.21.15.46:0/3236206076 >> v1:172.21.15.46:6789/0 conn(0x7ff73c0c6520 legacy=0x7ff73c0ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:59.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.389+0000 7ff73ad9d700 1 -- 172.21.15.46:0/3236206076 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73c0ad8c0 con 0x7ff73c0cf460 2024-01-27T01:07:59.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.389+0000 7ff73ad9d700 1 --2- 172.21.15.46:0/3236206076 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff73c0cf460 0x7ff73c0ad1c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff734009e90 tx=0x7ff734009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f3daa0cb5d9cad server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:59.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.389+0000 7ff73959a700 1 -- 172.21.15.46:0/3236206076 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73400e070 con 0x7ff73c0cf460 2024-01-27T01:07:59.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.389+0000 7ff73959a700 1 -- 172.21.15.46:0/3236206076 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73400bb80 con 0x7ff73c0cf460 2024-01-27T01:07:59.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.389+0000 7ff73959a700 1 -- 172.21.15.46:0/3236206076 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff734012430 con 0x7ff73c0cf460 2024-01-27T01:07:59.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.390+0000 7ff74093a700 1 -- 172.21.15.46:0/3236206076 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff73c0cf460 msgr2=0x7ff73c0ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:59.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.390+0000 7ff74093a700 1 --2- 172.21.15.46:0/3236206076 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff73c0cf460 0x7ff73c0ad1c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff734009e90 tx=0x7ff734009780 comp rx=0 tx=0).stop 2024-01-27T01:07:59.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.390+0000 7ff74093a700 1 -- 172.21.15.46:0/3236206076 shutdown_connections 2024-01-27T01:07:59.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.390+0000 7ff74093a700 1 --2- 172.21.15.46:0/3236206076 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff73c0cf460 0x7ff73c0ad1c0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.390+0000 7ff74093a700 1 -- 172.21.15.46:0/3236206076 >> 172.21.15.46:0/3236206076 conn(0x7ff73c099a30 msgr2=0x7ff73c06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:59.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.390+0000 7ff74093a700 1 -- 172.21.15.46:0/3236206076 shutdown_connections 2024-01-27T01:07:59.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.390+0000 7ff74093a700 1 -- 172.21.15.46:0/3236206076 wait complete. 2024-01-27T01:07:59.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.391+0000 7ff74093a700 1 Processor -- start 2024-01-27T01:07:59.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.391+0000 7ff74093a700 1 -- start start 2024-01-27T01:07:59.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.392+0000 7ff74093a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff73c0b2b40 0x7ff73c0b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:59.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.392+0000 7ff74093a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff73c0c6520 0x7ff73c16caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:59.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.392+0000 7ff74093a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff73c0cf460 0x7ff73c1694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:59.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.392+0000 7ff74093a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c16d030 con 0x7ff73c0b2b40 2024-01-27T01:07:59.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.392+0000 7ff74093a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c16d260 con 0x7ff73c0cf460 2024-01-27T01:07:59.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.392+0000 7ff74093a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c16d540 con 0x7ff73c0c6520 2024-01-27T01:07:59.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.392+0000 7ff73a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff73c0b2b40 0x7ff73c0b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:59.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.392+0000 7ff73a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff73c0b2b40 0x7ff73c0b57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51954/0 (socket says 172.21.15.46:51954) 2024-01-27T01:07:59.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.392+0000 7ff73a59c700 1 -- 172.21.15.46:0/503141565 learned_addr learned my addr 172.21.15.46:0/503141565 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:59.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.392+0000 7ff739d9b700 1 --2- 172.21.15.46:0/503141565 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff73c0c6520 0x7ff73c16caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:59.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.392+0000 7ff73ad9d700 1 --2- 172.21.15.46:0/503141565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff73c0cf460 0x7ff73c1694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:59.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.392+0000 7ff73a59c700 1 -- 172.21.15.46:0/503141565 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff73c0c6520 msgr2=0x7ff73c16caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:59.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.393+0000 7ff73a59c700 1 --2- 172.21.15.46:0/503141565 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff73c0c6520 0x7ff73c16caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.393+0000 7ff73a59c700 1 -- 172.21.15.46:0/503141565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff73c0cf460 msgr2=0x7ff73c1694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:59.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.393+0000 7ff73a59c700 1 --2- 172.21.15.46:0/503141565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff73c0cf460 0x7ff73c1694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.393+0000 7ff73a59c700 1 -- 172.21.15.46:0/503141565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff734009a20 con 0x7ff73c0b2b40 2024-01-27T01:07:59.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.393+0000 7ff73ad9d700 1 --2- 172.21.15.46:0/503141565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff73c0cf460 0x7ff73c1694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:59.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.393+0000 7ff73a59c700 1 --2- 172.21.15.46:0/503141565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff73c0b2b40 0x7ff73c0b57e0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7ff72c00c7c0 tx=0x7ff72c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:59.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.393+0000 7ff72a7fc700 1 -- 172.21.15.46:0/503141565 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72c001530 con 0x7ff73c0b2b40 2024-01-27T01:07:59.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.393+0000 7ff72a7fc700 1 -- 172.21.15.46:0/503141565 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72c0016b0 con 0x7ff73c0b2b40 2024-01-27T01:07:59.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.393+0000 7ff72a7fc700 1 -- 172.21.15.46:0/503141565 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72c001a70 con 0x7ff73c0b2b40 2024-01-27T01:07:59.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.393+0000 7ff74093a700 1 -- 172.21.15.46:0/503141565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff73c16d820 con 0x7ff73c0b2b40 2024-01-27T01:07:59.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.393+0000 7ff74093a700 1 -- 172.21.15.46:0/503141565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff73c169a30 con 0x7ff73c0b2b40 2024-01-27T01:07:59.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.394+0000 7ff74093a700 1 -- 172.21.15.46:0/503141565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff73c16da60 con 0x7ff73c0b2b40 2024-01-27T01:07:59.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.394+0000 7ff72a7fc700 1 -- 172.21.15.46:0/503141565 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 140) v1 ==== 84947+0+0 (secure 0 0 0) 0x7ff72c01b3b0 con 0x7ff73c0b2b40 2024-01-27T01:07:59.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.395+0000 7ff72a7fc700 1 --2- 172.21.15.46:0/503141565 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff71c067170 0x7ff71c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:59.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.395+0000 7ff739d9b700 1 -- 172.21.15.46:0/503141565 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff71c067170 msgr2=0x7ff71c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:59.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.395+0000 7ff72a7fc700 1 -- 172.21.15.46:0/503141565 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7811+0+0 (secure 0 0 0) 0x7ff72c084880 con 0x7ff73c0b2b40 2024-01-27T01:07:59.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.395+0000 7ff739d9b700 1 --2- 172.21.15.46:0/503141565 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff71c067170 0x7ff71c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:59.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.400+0000 7ff72a7fc700 1 -- 172.21.15.46:0/503141565 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff73c16da60 con 0x7ff73c0b2b40 2024-01-27T01:07:59.433 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:59.433+0000 7f6ad5cab1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:07:59.520 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:59.520+0000 7f6ad5cab1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:07:59.588 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:59.588+0000 7f9b4c19d1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:07:59.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.596+0000 7ff739d9b700 1 -- 172.21.15.46:0/503141565 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff71c067170 msgr2=0x7ff71c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:59.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.596+0000 7ff739d9b700 1 --2- 172.21.15.46:0/503141565 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff71c067170 0x7ff71c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:07:59.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.649+0000 7ff74093a700 1 -- 172.21.15.46:0/503141565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff73c0af1b0 con 0x7ff73c0b2b40 2024-01-27T01:07:59.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.649+0000 7ff72a7fc700 1 -- 172.21.15.46:0/503141565 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+5687 (secure 0 0 0) 0x7ff73c0af1b0 con 0x7ff73c0b2b40 2024-01-27T01:07:59.649 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:07:59.649 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":138,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1114164873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1114164873},{"type":"v1","addr":"172.21.15.106:6834","nonce":1114164873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128},{"gid":9821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/702094428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":702094428},{"type":"v1","addr":"172.21.15.106:6835","nonce":702094428}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:07:27.375435+0000","modified":"2024-01-27T01:07:30.945729+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11248":{"gid":11248,"name":"a","rank":0,"incarnation":129,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/1087205220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1087205220},{"type":"v1","addr":"172.21.15.46:6834","nonce":1087205220}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:07:41.411084+0000","modified":"2024-01-27T01:07:45.002753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9827":{"gid":9827,"name":"c","rank":0,"incarnation":135,"state":"up:active","state_seq":6,"addr":"172.21.15.46:6835/408985798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":408985798},{"type":"v1","addr":"172.21.15.46:6835","nonce":408985798}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-27T01:07:59.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.651+0000 7ff74093a700 1 -- 172.21.15.46:0/503141565 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff71c067170 msgr2=0x7ff71c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:07:59.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.651+0000 7ff74093a700 1 --2- 172.21.15.46:0/503141565 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff71c067170 0x7ff71c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.651+0000 7ff74093a700 1 -- 172.21.15.46:0/503141565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff73c0b2b40 msgr2=0x7ff73c0b57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:59.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.651+0000 7ff74093a700 1 --2- 172.21.15.46:0/503141565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff73c0b2b40 0x7ff73c0b57e0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7ff72c00c7c0 tx=0x7ff72c007e00 comp rx=0 tx=0).stop 2024-01-27T01:07:59.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.651+0000 7ff74093a700 1 -- 172.21.15.46:0/503141565 shutdown_connections 2024-01-27T01:07:59.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.651+0000 7ff74093a700 1 --2- 172.21.15.46:0/503141565 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff71c067170 0x7ff71c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.651+0000 7ff74093a700 1 --2- 172.21.15.46:0/503141565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff73c0b2b40 0x7ff73c0b57e0 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.651+0000 7ff74093a700 1 --2- 172.21.15.46:0/503141565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff73c0cf460 0x7ff73c1694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.651+0000 7ff74093a700 1 --2- 172.21.15.46:0/503141565 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff73c0c6520 0x7ff73c16caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.652+0000 7ff74093a700 1 -- 172.21.15.46:0/503141565 >> 172.21.15.46:0/503141565 conn(0x7ff73c099a30 msgr2=0x7ff73c0d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:59.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.652+0000 7ff74093a700 1 -- 172.21.15.46:0/503141565 shutdown_connections 2024-01-27T01:07:59.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.652+0000 7ff74093a700 1 -- 172.21.15.46:0/503141565 wait complete. 2024-01-27T01:07:59.653 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 138 2024-01-27T01:07:59.663 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:07:59.663+0000 7f6ad5cab1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:07:59.666 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:07:59.738 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:59.738+0000 7f9b4c19d1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:07:59.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.801+0000 7f4f5854b700 1 Processor -- start 2024-01-27T01:07:59.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.801+0000 7f4f5854b700 1 -- start start 2024-01-27T01:07:59.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.802+0000 7f4f5854b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4f5015d210 0x7f4f500cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:59.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.802+0000 7f4f5854b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4f500cbff0 0x7f4f500ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:59.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.802+0000 7f4f5854b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f500cc450 con 0x7f4f5015ce40 2024-01-27T01:07:59.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.802+0000 7f4f5854b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f500cff70 con 0x7f4f5015d210 2024-01-27T01:07:59.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.802+0000 7f4f5854b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f500cf400 con 0x7f4f500cbff0 2024-01-27T01:07:59.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.803+0000 7f4f55ae6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4f5015d210 0x7f4f500cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:59.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.803+0000 7f4f56ae8700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4f500cbff0 0x7f4f500ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:59.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.803+0000 7f4f55ae6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4f5015d210 0x7f4f500cbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51962/0 (socket says 172.21.15.46:51962) 2024-01-27T01:07:59.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.803+0000 7f4f55ae6700 1 -- 172.21.15.46:0/2322037322 learned_addr learned my addr 172.21.15.46:0/2322037322 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:59.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.803+0000 7f4f55ae6700 1 -- 172.21.15.46:0/2322037322 >> v2:172.21.15.106:3300/0 conn(0x7f4f500cbff0 msgr2=0x7f4f500ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:07:59.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.803+0000 7f4f55ae6700 1 --2- 172.21.15.46:0/2322037322 >> v2:172.21.15.106:3300/0 conn(0x7f4f500cbff0 0x7f4f500ade30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.803+0000 7f4f55ae6700 1 -- 172.21.15.46:0/2322037322 >> v1:172.21.15.106:6789/0 conn(0x7f4f5015ce40 legacy=0x7f4f500d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:59.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.803+0000 7f4f55ae6700 1 -- 172.21.15.46:0/2322037322 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f500cf660 con 0x7f4f5015d210 2024-01-27T01:07:59.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.803+0000 7f4f552e5700 1 -- 172.21.15.46:0/2322037322 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637627891 0 0) 0x7f4f4c001160 con 0x7f4f5015ce40 2024-01-27T01:07:59.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.803+0000 7f4f56ae8700 1 --2- 172.21.15.46:0/2322037322 >> v2:172.21.15.106:3300/0 conn(0x7f4f500cbff0 0x7f4f500ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:07:59.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.803+0000 7f4f55ae6700 1 --2- 172.21.15.46:0/2322037322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f5015d210 0x7f4f500cbab0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f4f500d01c0 tx=0x7f4f40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1fdc359ce555060 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:59.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.803+0000 7f4f552e5700 1 -- 172.21.15.46:0/2322037322 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f40009e70 con 0x7f4f5015d210 2024-01-27T01:07:59.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.803+0000 7f4f552e5700 1 -- 172.21.15.46:0/2322037322 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4000b860 con 0x7f4f5015d210 2024-01-27T01:07:59.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.803+0000 7f4f552e5700 1 -- 172.21.15.46:0/2322037322 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4000bc00 con 0x7f4f5015d210 2024-01-27T01:07:59.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.804+0000 7f4f5854b700 1 -- 172.21.15.46:0/2322037322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f5015d210 msgr2=0x7f4f500cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:59.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.804+0000 7f4f5854b700 1 --2- 172.21.15.46:0/2322037322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f5015d210 0x7f4f500cbab0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f4f500d01c0 tx=0x7f4f40009250 comp rx=0 tx=0).stop 2024-01-27T01:07:59.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.804+0000 7f4f5854b700 1 -- 172.21.15.46:0/2322037322 shutdown_connections 2024-01-27T01:07:59.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.804+0000 7f4f5854b700 1 --2- 172.21.15.46:0/2322037322 >> v2:172.21.15.106:3300/0 conn(0x7f4f500cbff0 0x7f4f500ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.804+0000 7f4f5854b700 1 --2- 172.21.15.46:0/2322037322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f5015d210 0x7f4f500cbab0 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.804+0000 7f4f5854b700 1 -- 172.21.15.46:0/2322037322 >> 172.21.15.46:0/2322037322 conn(0x7f4f5006f840 msgr2=0x7f4f500d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:07:59.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.804+0000 7f4f5854b700 1 -- 172.21.15.46:0/2322037322 shutdown_connections 2024-01-27T01:07:59.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.804+0000 7f4f5854b700 1 -- 172.21.15.46:0/2322037322 wait complete. 2024-01-27T01:07:59.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.805+0000 7f4f5854b700 1 Processor -- start 2024-01-27T01:07:59.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.805+0000 7f4f5854b700 1 -- start start 2024-01-27T01:07:59.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.806+0000 7f4f5854b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f500cbff0 0x7f4f5009a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:59.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.806+0000 7f4f5854b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f5015ce40 0x7f4f50158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:59.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.806+0000 7f4f5854b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f5015d210 0x7f4f5009b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:59.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.806+0000 7f4f5854b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5009b490 con 0x7f4f5015ce40 2024-01-27T01:07:59.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.806+0000 7f4f5854b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5009a880 con 0x7f4f5015d210 2024-01-27T01:07:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.806+0000 7f4f5854b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5009ab60 con 0x7f4f500cbff0 2024-01-27T01:07:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.806+0000 7f4f55ae6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f5015ce40 0x7f4f50158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.806+0000 7f4f55ae6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f5015ce40 0x7f4f50158b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51966/0 (socket says 172.21.15.46:51966) 2024-01-27T01:07:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.806+0000 7f4f562e7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f500cbff0 0x7f4f5009a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.806+0000 7f4f55ae6700 1 -- 172.21.15.46:0/2457488274 learned_addr learned my addr 172.21.15.46:0/2457488274 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:07:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.806+0000 7f4f56ae8700 1 --2- 172.21.15.46:0/2457488274 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f5015d210 0x7f4f5009b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:07:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.807+0000 7f4f55ae6700 1 -- 172.21.15.46:0/2457488274 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f500cbff0 msgr2=0x7f4f5009a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.807+0000 7f4f55ae6700 1 --2- 172.21.15.46:0/2457488274 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f500cbff0 0x7f4f5009a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.807+0000 7f4f55ae6700 1 -- 172.21.15.46:0/2457488274 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f5015d210 msgr2=0x7f4f5009b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:07:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.807+0000 7f4f55ae6700 1 --2- 172.21.15.46:0/2457488274 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f5015d210 0x7f4f5009b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:07:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.807+0000 7f4f55ae6700 1 -- 172.21.15.46:0/2457488274 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f48009a20 con 0x7f4f5015ce40 2024-01-27T01:07:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.807+0000 7f4f55ae6700 1 --2- 172.21.15.46:0/2457488274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f5015ce40 0x7f4f50158b30 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f4f400094c0 tx=0x7f4f40017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:07:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.807+0000 7f4f467fc700 1 -- 172.21.15.46:0/2457488274 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4000bbe0 con 0x7f4f5015ce40 2024-01-27T01:07:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.807+0000 7f4f467fc700 1 -- 172.21.15.46:0/2457488274 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4001a430 con 0x7f4f5015ce40 2024-01-27T01:07:59.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.807+0000 7f4f5854b700 1 -- 172.21.15.46:0/2457488274 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f400094f0 con 0x7f4f5015ce40 2024-01-27T01:07:59.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.807+0000 7f4f467fc700 1 -- 172.21.15.46:0/2457488274 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4001a7f0 con 0x7f4f5015ce40 2024-01-27T01:07:59.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.807+0000 7f4f5854b700 1 -- 172.21.15.46:0/2457488274 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5009b120 con 0x7f4f5015ce40 2024-01-27T01:07:59.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.808+0000 7f4f5854b700 1 -- 172.21.15.46:0/2457488274 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f5004c710 con 0x7f4f5015ce40 2024-01-27T01:07:59.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.809+0000 7f4f467fc700 1 -- 172.21.15.46:0/2457488274 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 140) v1 ==== 84947+0+0 (secure 0 0 0) 0x7f4f4001ab60 con 0x7f4f5015ce40 2024-01-27T01:07:59.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.809+0000 7f4f467fc700 1 --2- 172.21.15.46:0/2457488274 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4f34067170 0x7f4f34069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:07:59.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.809+0000 7f4f467fc700 1 -- 172.21.15.46:0/2457488274 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7811+0+0 (secure 0 0 0) 0x7f4f40086ce0 con 0x7f4f5015ce40 2024-01-27T01:07:59.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.809+0000 7f4f562e7700 1 -- 172.21.15.46:0/2457488274 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4f34067170 msgr2=0x7f4f34069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:07:59.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.809+0000 7f4f562e7700 1 --2- 172.21.15.46:0/2457488274 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4f34067170 0x7f4f34069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:07:59.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:07:59.814+0000 7f4f467fc700 1 -- 172.21.15.46:0/2457488274 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4f40053540 con 0x7f4f5015ce40 2024-01-27T01:07:59.893 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:07:59.894+0000 7f9b4c19d1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:08:00.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.010+0000 7f4f562e7700 1 -- 172.21.15.46:0/2457488274 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4f34067170 msgr2=0x7f4f34069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:00.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.010+0000 7f4f562e7700 1 --2- 172.21.15.46:0/2457488274 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4f34067170 0x7f4f34069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:08:00.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.021+0000 7f4f5854b700 1 -- 172.21.15.46:0/2457488274 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f5006fe70 con 0x7f4f5015ce40 2024-01-27T01:08:00.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.023+0000 7f4f467fc700 1 -- 172.21.15.46:0/2457488274 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) v1 ==== 74+0+20181 (secure 0 0 0) 0x7f4f40056b60 con 0x7f4f5015ce40 2024-01-27T01:08:00.023 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:00.023 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":248,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:07:53.996939+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:07:25.886791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-01-27T01:07:26.340041+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:07:39.209497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:07:40.386606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/2241845238":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1948052707":"2024-01-28T01:07:53.996892+0000","172.21.15.106:6836/62434":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1918093439":"2024-01-28T01:07:53.996892+0000","172.21.15.106:6837/62434":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/469270357":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1388668891":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1199239718":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/2250157080":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/2626066195":"2024-01-28T01:07:53.996892+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:08:00.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.025+0000 7f4f5854b700 1 -- 172.21.15.46:0/2457488274 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4f34067170 msgr2=0x7f4f34069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:00.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.025+0000 7f4f5854b700 1 --2- 172.21.15.46:0/2457488274 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4f34067170 0x7f4f34069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.025+0000 7f4f5854b700 1 -- 172.21.15.46:0/2457488274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f5015ce40 msgr2=0x7f4f50158b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:00.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.025+0000 7f4f5854b700 1 --2- 172.21.15.46:0/2457488274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f5015ce40 0x7f4f50158b30 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f4f400094c0 tx=0x7f4f40017000 comp rx=0 tx=0).stop 2024-01-27T01:08:00.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.025+0000 7f4f5854b700 1 -- 172.21.15.46:0/2457488274 shutdown_connections 2024-01-27T01:08:00.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.025+0000 7f4f5854b700 1 --2- 172.21.15.46:0/2457488274 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f4f34067170 0x7f4f34069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.025+0000 7f4f5854b700 1 --2- 172.21.15.46:0/2457488274 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f5015d210 0x7f4f5009b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.025+0000 7f4f5854b700 1 --2- 172.21.15.46:0/2457488274 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4f500cbff0 0x7f4f5009a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.025+0000 7f4f5854b700 1 --2- 172.21.15.46:0/2457488274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f5015ce40 0x7f4f50158b30 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.025+0000 7f4f5854b700 1 -- 172.21.15.46:0/2457488274 >> 172.21.15.46:0/2457488274 conn(0x7f4f5006f840 msgr2=0x7f4f5015aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:00.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.025+0000 7f4f5854b700 1 -- 172.21.15.46:0/2457488274 shutdown_connections 2024-01-27T01:08:00.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.025+0000 7f4f5854b700 1 -- 172.21.15.46:0/2457488274 wait complete. 2024-01-27T01:08:00.040 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-27T01:08:00.040 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:08:00.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.184+0000 7f0aebfc8700 1 Processor -- start 2024-01-27T01:08:00.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.184+0000 7f0aebfc8700 1 -- start start 2024-01-27T01:08:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.185+0000 7f0aebfc8700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0ae4159760 0x7f0ae40b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.185+0000 7f0aebfc8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae406f940 con 0x7f0ae40b3060 2024-01-27T01:08:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.185+0000 7f0aebfc8700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae406ed30 con 0x7f0ae40b12c0 2024-01-27T01:08:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.185+0000 7f0aebfc8700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ae406f010 con 0x7f0ae4159760 2024-01-27T01:08:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.185+0000 7f0ae9563700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0ae40b3060 0x7f0ae40b0bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60534/0 (socket says 172.21.15.46:60534) 2024-01-27T01:08:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.185+0000 7f0ae9563700 1 -- 172.21.15.46:0/2838288341 learned_addr learned my addr 172.21.15.46:0/2838288341 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.185+0000 7f0ae9d64700 1 --2- 172.21.15.46:0/2838288341 >> v2:172.21.15.106:3300/0 conn(0x7f0ae4159760 0x7f0ae40b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:00.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.185+0000 7f0ae8d62700 1 -- 172.21.15.46:0/2838288341 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344612932 0 0) 0x7f0ae406f940 con 0x7f0ae40b3060 2024-01-27T01:08:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.185+0000 7f0ae8d62700 1 -- 172.21.15.46:0/2838288341 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ad0003570 con 0x7f0ae40b3060 2024-01-27T01:08:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.185+0000 7f0ae9d64700 1 -- 172.21.15.46:0/2838288341 >> v1:172.21.15.106:6789/0 conn(0x7f0ae40b12c0 legacy=0x7f0ae40a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.185+0000 7f0ae9d64700 1 -- 172.21.15.46:0/2838288341 >> v1:172.21.15.46:6789/0 conn(0x7f0ae40b3060 legacy=0x7f0ae40b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.185+0000 7f0ae8d62700 1 -- 172.21.15.46:0/2838288341 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1974233637 0 0) 0x7f0ae406ed30 con 0x7f0ae40b12c0 2024-01-27T01:08:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.185+0000 7f0ae9d64700 1 -- 172.21.15.46:0/2838288341 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae406f2f0 con 0x7f0ae4159760 2024-01-27T01:08:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.186+0000 7f0ae9d64700 1 --2- 172.21.15.46:0/2838288341 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ae4159760 0x7f0ae40b2b20 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f0ae00098e0 tx=0x7f0ae00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=167942590c01116e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.186+0000 7f0ae8d62700 1 -- 172.21.15.46:0/2838288341 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae000b640 con 0x7f0ae4159760 2024-01-27T01:08:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.186+0000 7f0ae8d62700 1 -- 172.21.15.46:0/2838288341 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ae000b7a0 con 0x7f0ae4159760 2024-01-27T01:08:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.186+0000 7f0ae8d62700 1 -- 172.21.15.46:0/2838288341 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae000bb40 con 0x7f0ae4159760 2024-01-27T01:08:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.186+0000 7f0aebfc8700 1 -- 172.21.15.46:0/2838288341 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ae4159760 msgr2=0x7f0ae40b2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.186+0000 7f0aebfc8700 1 --2- 172.21.15.46:0/2838288341 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ae4159760 0x7f0ae40b2b20 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f0ae00098e0 tx=0x7f0ae00091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.187+0000 7f0aebfc8700 1 -- 172.21.15.46:0/2838288341 shutdown_connections 2024-01-27T01:08:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.187+0000 7f0aebfc8700 1 --2- 172.21.15.46:0/2838288341 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ae4159760 0x7f0ae40b2b20 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.187+0000 7f0aebfc8700 1 -- 172.21.15.46:0/2838288341 >> 172.21.15.46:0/2838288341 conn(0x7f0ae4069510 msgr2=0x7f0ae40aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.187+0000 7f0aebfc8700 1 -- 172.21.15.46:0/2838288341 shutdown_connections 2024-01-27T01:08:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.187+0000 7f0aebfc8700 1 -- 172.21.15.46:0/2838288341 wait complete. 2024-01-27T01:08:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.188+0000 7f0aebfc8700 1 Processor -- start 2024-01-27T01:08:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.188+0000 7f0aebfc8700 1 -- start start 2024-01-27T01:08:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.189+0000 7f0aebfc8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ae40b12c0 0x7f0ae4154e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.189+0000 7f0aebfc8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ae40b3060 0x7f0ae415ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.189+0000 7f0aebfc8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ae4159760 0x7f0ae4151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.189+0000 7f0aebfc8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae415c0b0 con 0x7f0ae40b3060 2024-01-27T01:08:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.189+0000 7f0aebfc8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae415c4d0 con 0x7f0ae4159760 2024-01-27T01:08:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.189+0000 7f0aebfc8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae415b430 con 0x7f0ae40b12c0 2024-01-27T01:08:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.190+0000 7f0ae9563700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ae40b3060 0x7f0ae415ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.190+0000 7f0ae9563700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ae40b3060 0x7f0ae415ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51980/0 (socket says 172.21.15.46:51980) 2024-01-27T01:08:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.190+0000 7f0ae9d64700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ae40b12c0 0x7f0ae4154e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.190+0000 7f0ae9563700 1 -- 172.21.15.46:0/202881559 learned_addr learned my addr 172.21.15.46:0/202881559 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.190+0000 7f0aea565700 1 --2- 172.21.15.46:0/202881559 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ae4159760 0x7f0ae4151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.190+0000 7f0ae9563700 1 -- 172.21.15.46:0/202881559 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ae40b12c0 msgr2=0x7f0ae4154e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.190+0000 7f0ae9563700 1 --2- 172.21.15.46:0/202881559 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ae40b12c0 0x7f0ae4154e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.190+0000 7f0ae9563700 1 -- 172.21.15.46:0/202881559 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ae4159760 msgr2=0x7f0ae4151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:00.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.190+0000 7f0ae9563700 1 --2- 172.21.15.46:0/202881559 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ae4159760 0x7f0ae4151730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.190+0000 7f0ae9563700 1 -- 172.21.15.46:0/202881559 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae0009420 con 0x7f0ae40b3060 2024-01-27T01:08:00.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.190+0000 7f0ae9563700 1 --2- 172.21.15.46:0/202881559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ae40b3060 0x7f0ae415ab00 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f0ad40011e0 tx=0x7f0ad4007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:00.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.190+0000 7f0ad9ffb700 1 -- 172.21.15.46:0/202881559 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4005b30 con 0x7f0ae40b3060 2024-01-27T01:08:00.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.190+0000 7f0ad9ffb700 1 -- 172.21.15.46:0/202881559 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ad4005c90 con 0x7f0ae40b3060 2024-01-27T01:08:00.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.190+0000 7f0ad9ffb700 1 -- 172.21.15.46:0/202881559 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4007300 con 0x7f0ae40b3060 2024-01-27T01:08:00.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.190+0000 7f0aebfc8700 1 -- 172.21.15.46:0/202881559 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae415b710 con 0x7f0ae40b3060 2024-01-27T01:08:00.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.191+0000 7f0aebfc8700 1 -- 172.21.15.46:0/202881559 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae415bcd0 con 0x7f0ae40b3060 2024-01-27T01:08:00.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.192+0000 7f0ad9ffb700 1 -- 172.21.15.46:0/202881559 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 140) v1 ==== 84947+0+0 (secure 0 0 0) 0x7f0ad4007670 con 0x7f0ae40b3060 2024-01-27T01:08:00.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.192+0000 7f0aebfc8700 1 -- 172.21.15.46:0/202881559 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae415c1f0 con 0x7f0ae40b3060 2024-01-27T01:08:00.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.192+0000 7f0ad9ffb700 1 --2- 172.21.15.46:0/202881559 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0ac8067170 0x7f0ac8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:00.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.193+0000 7f0ae9d64700 1 -- 172.21.15.46:0/202881559 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0ac8067170 msgr2=0x7f0ac8069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:00.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.193+0000 7f0ad9ffb700 1 -- 172.21.15.46:0/202881559 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7811+0+0 (secure 0 0 0) 0x7f0ad4084680 con 0x7f0ae40b3060 2024-01-27T01:08:00.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.193+0000 7f0ae9d64700 1 --2- 172.21.15.46:0/202881559 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0ac8067170 0x7f0ac8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:00.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.197+0000 7f0ad9ffb700 1 -- 172.21.15.46:0/202881559 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0ae415c1f0 con 0x7f0ae40b3060 2024-01-27T01:08:00.275 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:00.275+0000 7f6ad5cab1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:08:00.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.394+0000 7f0ae9d64700 1 -- 172.21.15.46:0/202881559 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0ac8067170 msgr2=0x7f0ac8069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:00.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.394+0000 7f0ae9d64700 1 --2- 172.21.15.46:0/202881559 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0ac8067170 0x7f0ac8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:08:00.447 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:00.447+0000 7f6ad5cab1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:08:00.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.447+0000 7f0aebfc8700 1 -- 172.21.15.46:0/202881559 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ae415c1f0 con 0x7f0ae40b3060 2024-01-27T01:08:00.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.448+0000 7f0ad9ffb700 1 -- 172.21.15.46:0/202881559 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+5687 (secure 0 0 0) 0x7f0ae415c1f0 con 0x7f0ae40b3060 2024-01-27T01:08:00.448 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:00.448 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":138,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1114164873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1114164873},{"type":"v1","addr":"172.21.15.106:6834","nonce":1114164873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128},{"gid":9821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/702094428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":702094428},{"type":"v1","addr":"172.21.15.106:6835","nonce":702094428}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:07:27.375435+0000","modified":"2024-01-27T01:07:30.945729+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11248":{"gid":11248,"name":"a","rank":0,"incarnation":129,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/1087205220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1087205220},{"type":"v1","addr":"172.21.15.46:6834","nonce":1087205220}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:07:41.411084+0000","modified":"2024-01-27T01:07:45.002753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9827":{"gid":9827,"name":"c","rank":0,"incarnation":135,"state":"up:active","state_seq":6,"addr":"172.21.15.46:6835/408985798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":408985798},{"type":"v1","addr":"172.21.15.46:6835","nonce":408985798}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-27T01:08:00.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.450+0000 7f0aebfc8700 1 -- 172.21.15.46:0/202881559 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0ac8067170 msgr2=0x7f0ac8069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:00.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.450+0000 7f0aebfc8700 1 --2- 172.21.15.46:0/202881559 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0ac8067170 0x7f0ac8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.450+0000 7f0aebfc8700 1 -- 172.21.15.46:0/202881559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ae40b3060 msgr2=0x7f0ae415ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:00.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.450+0000 7f0aebfc8700 1 --2- 172.21.15.46:0/202881559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ae40b3060 0x7f0ae415ab00 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f0ad40011e0 tx=0x7f0ad4007df0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.450+0000 7f0aebfc8700 1 -- 172.21.15.46:0/202881559 shutdown_connections 2024-01-27T01:08:00.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.450+0000 7f0aebfc8700 1 --2- 172.21.15.46:0/202881559 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0ac8067170 0x7f0ac8069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.450+0000 7f0aebfc8700 1 --2- 172.21.15.46:0/202881559 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ae4159760 0x7f0ae4151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.450+0000 7f0aebfc8700 1 --2- 172.21.15.46:0/202881559 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ae40b12c0 0x7f0ae4154e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.450+0000 7f0aebfc8700 1 --2- 172.21.15.46:0/202881559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ae40b3060 0x7f0ae415ab00 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.450+0000 7f0aebfc8700 1 -- 172.21.15.46:0/202881559 >> 172.21.15.46:0/202881559 conn(0x7f0ae4069510 msgr2=0x7f0ae40b04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:00.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.450+0000 7f0aebfc8700 1 -- 172.21.15.46:0/202881559 shutdown_connections 2024-01-27T01:08:00.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.450+0000 7f0aebfc8700 1 -- 172.21.15.46:0/202881559 wait complete. 2024-01-27T01:08:00.452 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 138 2024-01-27T01:08:00.465 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:08:00.596 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:00.597+0000 7f9b4c19d1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:08:00.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.598+0000 7fa4ae6bf700 1 Processor -- start 2024-01-27T01:08:00.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.598+0000 7fa4ae6bf700 1 -- start start 2024-01-27T01:08:00.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.599+0000 7fa4ae6bf700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4a80cf460 0x7fa4a80ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:00.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.599+0000 7fa4ae6bf700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a815a0b0 con 0x7fa4a80c6520 2024-01-27T01:08:00.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.599+0000 7fa4ae6bf700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a80ae4d0 con 0x7fa4a80b2b40 2024-01-27T01:08:00.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.599+0000 7fa4ae6bf700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a80af070 con 0x7fa4a80cf460 2024-01-27T01:08:00.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.599+0000 7fa4a77fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa4a80c6520 0x7fa4a80ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57156/0 (socket says 172.21.15.46:57156) 2024-01-27T01:08:00.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.599+0000 7fa4a77fe700 1 -- 172.21.15.46:0/257645462 learned_addr learned my addr 172.21.15.46:0/257645462 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:00.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.599+0000 7fa4acc5c700 1 --2- 172.21.15.46:0/257645462 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4a80cf460 0x7fa4a80ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:00.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.600+0000 7fa4a6ffd700 1 -- 172.21.15.46:0/257645462 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188871241 0 0) 0x7fa4a815a0b0 con 0x7fa4a80c6520 2024-01-27T01:08:00.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.600+0000 7fa4a6ffd700 1 -- 172.21.15.46:0/257645462 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa490003570 con 0x7fa4a80c6520 2024-01-27T01:08:00.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.600+0000 7fa4a6ffd700 1 -- 172.21.15.46:0/257645462 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170485535 0 0) 0x7fa4a80ae4d0 con 0x7fa4a80b2b40 2024-01-27T01:08:00.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.600+0000 7fa4a6ffd700 1 -- 172.21.15.46:0/257645462 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a815a0b0 con 0x7fa4a80b2b40 2024-01-27T01:08:00.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.600+0000 7fa4acc5c700 1 -- 172.21.15.46:0/257645462 >> v1:172.21.15.106:6789/0 conn(0x7fa4a80b2b40 legacy=0x7fa4a80ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:00.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.600+0000 7fa4acc5c700 1 -- 172.21.15.46:0/257645462 >> v1:172.21.15.46:6789/0 conn(0x7fa4a80c6520 legacy=0x7fa4a80ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:00.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.600+0000 7fa4acc5c700 1 -- 172.21.15.46:0/257645462 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a80ad8c0 con 0x7fa4a80cf460 2024-01-27T01:08:00.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.600+0000 7fa4acc5c700 1 --2- 172.21.15.46:0/257645462 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4a80cf460 0x7fa4a80ad1c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0009e90 tx=0x7fa4a0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8ef497267c4a414 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:00.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.600+0000 7fa4a6ffd700 1 -- 172.21.15.46:0/257645462 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a000e070 con 0x7fa4a80cf460 2024-01-27T01:08:00.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.600+0000 7fa4a6ffd700 1 -- 172.21.15.46:0/257645462 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a000bb80 con 0x7fa4a80cf460 2024-01-27T01:08:00.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.600+0000 7fa4a6ffd700 1 -- 172.21.15.46:0/257645462 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0012430 con 0x7fa4a80cf460 2024-01-27T01:08:00.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.601+0000 7fa4ae6bf700 1 -- 172.21.15.46:0/257645462 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4a80cf460 msgr2=0x7fa4a80ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:00.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.601+0000 7fa4ae6bf700 1 --2- 172.21.15.46:0/257645462 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4a80cf460 0x7fa4a80ad1c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0009e90 tx=0x7fa4a0009780 comp rx=0 tx=0).stop 2024-01-27T01:08:00.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.601+0000 7fa4ae6bf700 1 -- 172.21.15.46:0/257645462 shutdown_connections 2024-01-27T01:08:00.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.601+0000 7fa4ae6bf700 1 --2- 172.21.15.46:0/257645462 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4a80cf460 0x7fa4a80ad1c0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.601+0000 7fa4ae6bf700 1 -- 172.21.15.46:0/257645462 >> 172.21.15.46:0/257645462 conn(0x7fa4a8099a30 msgr2=0x7fa4a806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:00.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.601+0000 7fa4ae6bf700 1 -- 172.21.15.46:0/257645462 shutdown_connections 2024-01-27T01:08:00.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.601+0000 7fa4ae6bf700 1 -- 172.21.15.46:0/257645462 wait complete. 2024-01-27T01:08:00.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.602+0000 7fa4ae6bf700 1 Processor -- start 2024-01-27T01:08:00.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.602+0000 7fa4ae6bf700 1 -- start start 2024-01-27T01:08:00.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.603+0000 7fa4ae6bf700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4a80b2b40 0x7fa4a80b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:00.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.603+0000 7fa4ae6bf700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4a80c6520 0x7fa4a816caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:00.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.603+0000 7fa4ae6bf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4a80cf460 0x7fa4a81694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:00.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.603+0000 7fa4ae6bf700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a816d030 con 0x7fa4a80cf460 2024-01-27T01:08:00.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.603+0000 7fa4ae6bf700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a816d260 con 0x7fa4a80c6520 2024-01-27T01:08:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.603+0000 7fa4ae6bf700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a816d540 con 0x7fa4a80b2b40 2024-01-27T01:08:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.603+0000 7fa4a77fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4a80c6520 0x7fa4a816caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.603+0000 7fa4acc5c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4a80cf460 0x7fa4a81694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.603+0000 7fa4a7fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4a80b2b40 0x7fa4a80b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.603+0000 7fa4a77fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4a80c6520 0x7fa4a816caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56058/0 (socket says 172.21.15.46:56058) 2024-01-27T01:08:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.603+0000 7fa4acc5c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4a80cf460 0x7fa4a81694f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55426/0 (socket says 172.21.15.46:55426) 2024-01-27T01:08:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.603+0000 7fa4a77fe700 1 -- 172.21.15.46:0/3356046010 learned_addr learned my addr 172.21.15.46:0/3356046010 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.604+0000 7fa4a77fe700 1 -- 172.21.15.46:0/3356046010 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4a80b2b40 msgr2=0x7fa4a80b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.604+0000 7fa4a77fe700 1 --2- 172.21.15.46:0/3356046010 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4a80b2b40 0x7fa4a80b57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.604+0000 7fa4a77fe700 1 -- 172.21.15.46:0/3356046010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4a80cf460 msgr2=0x7fa4a81694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.604+0000 7fa4a77fe700 1 --2- 172.21.15.46:0/3356046010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4a80cf460 0x7fa4a81694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.604+0000 7fa4a77fe700 1 -- 172.21.15.46:0/3356046010 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a0009a20 con 0x7fa4a80c6520 2024-01-27T01:08:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.604+0000 7fa4a77fe700 1 --2- 172.21.15.46:0/3356046010 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4a80c6520 0x7fa4a816caf0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00c790 tx=0x7fa49c007e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:00.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.604+0000 7fa48ffff700 1 -- 172.21.15.46:0/3356046010 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c005b30 con 0x7fa4a80c6520 2024-01-27T01:08:00.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.604+0000 7fa48ffff700 1 -- 172.21.15.46:0/3356046010 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49c005cd0 con 0x7fa4a80c6520 2024-01-27T01:08:00.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.604+0000 7fa4ae6bf700 1 -- 172.21.15.46:0/3356046010 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a816d820 con 0x7fa4a80c6520 2024-01-27T01:08:00.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.604+0000 7fa48ffff700 1 -- 172.21.15.46:0/3356046010 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c007320 con 0x7fa4a80c6520 2024-01-27T01:08:00.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.604+0000 7fa4ae6bf700 1 -- 172.21.15.46:0/3356046010 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a8169a30 con 0x7fa4a80c6520 2024-01-27T01:08:00.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.605+0000 7fa4ae6bf700 1 -- 172.21.15.46:0/3356046010 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a816da60 con 0x7fa4a80c6520 2024-01-27T01:08:00.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.606+0000 7fa48ffff700 1 -- 172.21.15.46:0/3356046010 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 140) v1 ==== 84947+0+0 (secure 0 0 0) 0x7fa49c012400 con 0x7fa4a80c6520 2024-01-27T01:08:00.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.606+0000 7fa48ffff700 1 --2- 172.21.15.46:0/3356046010 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa494067170 0x7fa494069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:00.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.606+0000 7fa4a7fff700 1 -- 172.21.15.46:0/3356046010 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa494067170 msgr2=0x7fa494069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:00.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.606+0000 7fa48ffff700 1 -- 172.21.15.46:0/3356046010 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7811+0+0 (secure 0 0 0) 0x7fa49c0844f0 con 0x7fa4a80c6520 2024-01-27T01:08:00.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.606+0000 7fa4a7fff700 1 --2- 172.21.15.46:0/3356046010 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa494067170 0x7fa494069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:00.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.611+0000 7fa48ffff700 1 -- 172.21.15.46:0/3356046010 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa49c04b5a0 con 0x7fa4a80c6520 2024-01-27T01:08:00.630 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:00.630+0000 7f6ad5cab1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:08:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.807+0000 7fa4a7fff700 1 -- 172.21.15.46:0/3356046010 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa494067170 msgr2=0x7fa494069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.807+0000 7fa4a7fff700 1 --2- 172.21.15.46:0/3356046010 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa494067170 0x7fa494069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:08:00.820 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:00.822+0000 7f9b4c19d1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:08:00.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.862+0000 7fa4ae6bf700 1 -- 172.21.15.46:0/3356046010 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4a80af1b0 con 0x7fa4a80c6520 2024-01-27T01:08:00.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.862+0000 7fa48ffff700 1 -- 172.21.15.46:0/3356046010 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+5687 (secure 0 0 0) 0x7fa4a80af1b0 con 0x7fa4a80c6520 2024-01-27T01:08:00.862 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:00.863 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":138,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1114164873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1114164873},{"type":"v1","addr":"172.21.15.106:6834","nonce":1114164873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128},{"gid":9821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/702094428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":702094428},{"type":"v1","addr":"172.21.15.106:6835","nonce":702094428}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:07:27.375435+0000","modified":"2024-01-27T01:07:30.945729+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11248":{"gid":11248,"name":"a","rank":0,"incarnation":129,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/1087205220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1087205220},{"type":"v1","addr":"172.21.15.46:6834","nonce":1087205220}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:07:41.411084+0000","modified":"2024-01-27T01:07:45.002753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9827":{"gid":9827,"name":"c","rank":0,"incarnation":135,"state":"up:active","state_seq":6,"addr":"172.21.15.46:6835/408985798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":408985798},{"type":"v1","addr":"172.21.15.46:6835","nonce":408985798}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-27T01:08:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.864+0000 7fa4ae6bf700 1 -- 172.21.15.46:0/3356046010 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa494067170 msgr2=0x7fa494069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.864+0000 7fa4ae6bf700 1 --2- 172.21.15.46:0/3356046010 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa494067170 0x7fa494069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.864+0000 7fa4ae6bf700 1 -- 172.21.15.46:0/3356046010 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4a80c6520 msgr2=0x7fa4a816caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.864+0000 7fa4ae6bf700 1 --2- 172.21.15.46:0/3356046010 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4a80c6520 0x7fa4a816caf0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00c790 tx=0x7fa49c007e80 comp rx=0 tx=0).stop 2024-01-27T01:08:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.865+0000 7fa4ae6bf700 1 -- 172.21.15.46:0/3356046010 shutdown_connections 2024-01-27T01:08:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.865+0000 7fa4ae6bf700 1 --2- 172.21.15.46:0/3356046010 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa494067170 0x7fa494069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.865+0000 7fa4ae6bf700 1 --2- 172.21.15.46:0/3356046010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4a80cf460 0x7fa4a81694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.865+0000 7fa4ae6bf700 1 --2- 172.21.15.46:0/3356046010 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4a80b2b40 0x7fa4a80b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.865+0000 7fa4ae6bf700 1 --2- 172.21.15.46:0/3356046010 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4a80c6520 0x7fa4a816caf0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.865+0000 7fa4ae6bf700 1 -- 172.21.15.46:0/3356046010 >> 172.21.15.46:0/3356046010 conn(0x7fa4a8099a30 msgr2=0x7fa4a80d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.865+0000 7fa4ae6bf700 1 -- 172.21.15.46:0/3356046010 shutdown_connections 2024-01-27T01:08:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:00.865+0000 7fa4ae6bf700 1 -- 172.21.15.46:0/3356046010 wait complete. 2024-01-27T01:08:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 138 2024-01-27T01:08:00.879 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:08:00.893 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:00.895+0000 7f9b4c19d1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:08:01.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.012+0000 7fe01ad84700 1 Processor -- start 2024-01-27T01:08:01.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.012+0000 7fe01ad84700 1 -- start start 2024-01-27T01:08:01.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.013+0000 7fe01ad84700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe014157340 0x7fe01415bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:01.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.013+0000 7fe01ad84700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe0140b4180 0x7fe014157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:01.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.013+0000 7fe01ad84700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe014158ae0 con 0x7fe0140ae520 2024-01-27T01:08:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.013+0000 7fe01ad84700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe01415b650 con 0x7fe014157340 2024-01-27T01:08:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.013+0000 7fe01ad84700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe01415c020 con 0x7fe0140b4180 2024-01-27T01:08:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.013+0000 7fe018b20700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe014157340 0x7fe01415bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.013+0000 7fe018b20700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe014157340 0x7fe01415bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55434/0 (socket says 172.21.15.46:55434) 2024-01-27T01:08:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.013+0000 7fe018b20700 1 -- 172.21.15.46:0/1409634294 learned_addr learned my addr 172.21.15.46:0/1409634294 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.013+0000 7fe019321700 1 --2- 172.21.15.46:0/1409634294 >> v2:172.21.15.106:3300/0 conn(0x7fe0140b4180 0x7fe014157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.014+0000 7fe018b20700 1 -- 172.21.15.46:0/1409634294 >> v2:172.21.15.106:3300/0 conn(0x7fe0140b4180 msgr2=0x7fe014157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.014+0000 7fe018b20700 1 --2- 172.21.15.46:0/1409634294 >> v2:172.21.15.106:3300/0 conn(0x7fe0140b4180 0x7fe014157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.014+0000 7fe018b20700 1 -- 172.21.15.46:0/1409634294 >> v1:172.21.15.106:6789/0 conn(0x7fe0140ae520 legacy=0x7fe0140b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.014+0000 7fe018b20700 1 -- 172.21.15.46:0/1409634294 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe014158520 con 0x7fe014157340 2024-01-27T01:08:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.014+0000 7fe018b20700 1 --2- 172.21.15.46:0/1409634294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe014157340 0x7fe01415bae0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fe0140caeb0 tx=0x7fe004009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5530b15e8d4246 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.014+0000 7fe0137fe700 1 -- 172.21.15.46:0/1409634294 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00400e070 con 0x7fe014157340 2024-01-27T01:08:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.014+0000 7fe0137fe700 1 -- 172.21.15.46:0/1409634294 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00400ba20 con 0x7fe014157340 2024-01-27T01:08:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.014+0000 7fe0137fe700 1 -- 172.21.15.46:0/1409634294 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0040092e0 con 0x7fe014157340 2024-01-27T01:08:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.015+0000 7fe01ad84700 1 -- 172.21.15.46:0/1409634294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe014157340 msgr2=0x7fe01415bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.015+0000 7fe01ad84700 1 --2- 172.21.15.46:0/1409634294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe014157340 0x7fe01415bae0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fe0140caeb0 tx=0x7fe004009840 comp rx=0 tx=0).stop 2024-01-27T01:08:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.015+0000 7fe01ad84700 1 -- 172.21.15.46:0/1409634294 shutdown_connections 2024-01-27T01:08:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.015+0000 7fe01ad84700 1 --2- 172.21.15.46:0/1409634294 >> v2:172.21.15.106:3300/0 conn(0x7fe0140b4180 0x7fe014157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.015+0000 7fe01ad84700 1 --2- 172.21.15.46:0/1409634294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe014157340 0x7fe01415bae0 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.015+0000 7fe01ad84700 1 -- 172.21.15.46:0/1409634294 >> 172.21.15.46:0/1409634294 conn(0x7fe0140ce530 msgr2=0x7fe0140ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.015+0000 7fe01ad84700 1 -- 172.21.15.46:0/1409634294 shutdown_connections 2024-01-27T01:08:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.015+0000 7fe01ad84700 1 -- 172.21.15.46:0/1409634294 wait complete. 2024-01-27T01:08:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.016+0000 7fe01ad84700 1 Processor -- start 2024-01-27T01:08:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.017+0000 7fe01ad84700 1 -- start start 2024-01-27T01:08:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.017+0000 7fe01ad84700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe0140ae520 0x7fe014155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.017+0000 7fe01ad84700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0140b4180 0x7fe0140b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.017+0000 7fe01ad84700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe014157340 0x7fe0140c2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.017+0000 7fe01ad84700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0140c64c0 con 0x7fe0140b4180 2024-01-27T01:08:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.017+0000 7fe01ad84700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0140c58b0 con 0x7fe0140ae520 2024-01-27T01:08:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.017+0000 7fe01ad84700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0140c5b90 con 0x7fe014157340 2024-01-27T01:08:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe013fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0140b4180 0x7fe0140b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe018b20700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe0140ae520 0x7fe014155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe013fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0140b4180 0x7fe0140b23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55448/0 (socket says 172.21.15.46:55448) 2024-01-27T01:08:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe013fff700 1 -- 172.21.15.46:0/154981760 learned_addr learned my addr 172.21.15.46:0/154981760 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe018b20700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe0140ae520 0x7fe014155d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:56072/0 (socket says 172.21.15.46:56072) 2024-01-27T01:08:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe019321700 1 --2- 172.21.15.46:0/154981760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe014157340 0x7fe0140c2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe013fff700 1 -- 172.21.15.46:0/154981760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe014157340 msgr2=0x7fe0140c2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe013fff700 1 --2- 172.21.15.46:0/154981760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe014157340 0x7fe0140c2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe013fff700 1 -- 172.21.15.46:0/154981760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe0140ae520 msgr2=0x7fe014155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe013fff700 1 --2- 172.21.15.46:0/154981760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe0140ae520 0x7fe014155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe013fff700 1 -- 172.21.15.46:0/154981760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe004009ae0 con 0x7fe0140b4180 2024-01-27T01:08:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe018b20700 1 --2- 172.21.15.46:0/154981760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe0140ae520 0x7fe014155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe013fff700 1 --2- 172.21.15.46:0/154981760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0140b4180 0x7fe0140b23c0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fe00800c4e0 tx=0x7fe00800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe010ff9700 1 -- 172.21.15.46:0/154981760 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe008011590 con 0x7fe0140b4180 2024-01-27T01:08:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe010ff9700 1 -- 172.21.15.46:0/154981760 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe008011710 con 0x7fe0140b4180 2024-01-27T01:08:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe01ad84700 1 -- 172.21.15.46:0/154981760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0140c5e70 con 0x7fe0140b4180 2024-01-27T01:08:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe010ff9700 1 -- 172.21.15.46:0/154981760 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe008011ad0 con 0x7fe0140b4180 2024-01-27T01:08:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.018+0000 7fe01ad84700 1 -- 172.21.15.46:0/154981760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0140c5540 con 0x7fe0140b4180 2024-01-27T01:08:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.019+0000 7fe01ad84700 1 -- 172.21.15.46:0/154981760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01404c710 con 0x7fe0140b4180 2024-01-27T01:08:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.020+0000 7fe010ff9700 1 -- 172.21.15.46:0/154981760 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 140) v1 ==== 84947+0+0 (secure 0 0 0) 0x7fe00801b3b0 con 0x7fe0140b4180 2024-01-27T01:08:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.020+0000 7fe010ff9700 1 --2- 172.21.15.46:0/154981760 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe000067170 0x7fe000069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.020+0000 7fe010ff9700 1 -- 172.21.15.46:0/154981760 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7811+0+0 (secure 0 0 0) 0x7fe0080844d0 con 0x7fe0140b4180 2024-01-27T01:08:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.020+0000 7fe018b20700 1 -- 172.21.15.46:0/154981760 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe000067170 msgr2=0x7fe000069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.021+0000 7fe018b20700 1 --2- 172.21.15.46:0/154981760 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe000067170 0x7fe000069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:01.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.026+0000 7fe010ff9700 1 -- 172.21.15.46:0/154981760 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe008050c70 con 0x7fe0140b4180 2024-01-27T01:08:01.053 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.054+0000 7f9b4c19d1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:08:01.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.075+0000 7fe010ff9700 1 -- 172.21.15.46:0/154981760 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 141) v1 ==== 84453+0+0 (secure 0 0 0) 0x7fe008051040 con 0x7fe0140b4180 2024-01-27T01:08:01.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.076+0000 7fe010ff9700 1 -- 172.21.15.46:0/154981760 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe000067170 msgr2=0x7fe000069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:01.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.076+0000 7fe010ff9700 1 --2- 172.21.15.46:0/154981760 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe000067170 0x7fe000069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.138 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.139+0000 7f9acaa8b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.138 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.140+0000 7f9acaa8b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.138 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.140+0000 7f9acaa8b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.139 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.140+0000 7f9acaa8b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.139 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.140+0000 7f9acaa8b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.139 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.140+0000 7f9ac8286700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.139 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.140+0000 7f9ac8286700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.139 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.140+0000 7f9ac8286700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.139 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.140+0000 7f9ac8286700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.139 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.140+0000 7f9ac8286700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.148 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.149+0000 7f9acaa8b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.148 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.149+0000 7f9acaa8b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.148 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.149+0000 7f9acaa8b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.148 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.149+0000 7f9acaa8b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.148 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.149+0000 7f9acaa8b700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.148 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.150+0000 7f9ac8286700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.148 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.150+0000 7f9ac8286700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.148 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.150+0000 7f9ac8286700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.149 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.150+0000 7f9ac8286700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.149 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:01.150+0000 7f9ac8286700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:01.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.247+0000 7fe01ad84700 1 -- 172.21.15.46:0/154981760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0140c6310 con 0x7fe0140b4180 2024-01-27T01:08:01.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.248+0000 7fe010ff9700 1 -- 172.21.15.46:0/154981760 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v249) v1 ==== 74+0+20684 (secure 0 0 0) 0x7fe008017070 con 0x7fe0140b4180 2024-01-27T01:08:01.249 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:01.249 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":249,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:08:01.068218+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:07:25.886791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-01-27T01:07:26.340041+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:07:39.209497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:07:40.386606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/558414078":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/2701940868":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/3540935782":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/3966339865":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/3958137863":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/2317042118":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/1816027959":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/2241845238":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1948052707":"2024-01-28T01:07:53.996892+0000","172.21.15.106:6836/62434":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1918093439":"2024-01-28T01:07:53.996892+0000","172.21.15.106:6837/62434":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/469270357":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1388668891":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1199239718":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/3351808187":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/2250157080":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/2626066195":"2024-01-28T01:07:53.996892+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:08:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.251+0000 7fe01ad84700 1 -- 172.21.15.46:0/154981760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0140b4180 msgr2=0x7fe0140b23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.251+0000 7fe01ad84700 1 --2- 172.21.15.46:0/154981760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0140b4180 0x7fe0140b23c0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fe00800c4e0 tx=0x7fe00800c7f0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.251+0000 7fe01ad84700 1 -- 172.21.15.46:0/154981760 shutdown_connections 2024-01-27T01:08:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.251+0000 7fe01ad84700 1 --2- 172.21.15.46:0/154981760 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe000067170 0x7fe000069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.251+0000 7fe01ad84700 1 --2- 172.21.15.46:0/154981760 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe014157340 0x7fe0140c2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.251+0000 7fe01ad84700 1 --2- 172.21.15.46:0/154981760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe0140ae520 0x7fe014155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.251+0000 7fe01ad84700 1 --2- 172.21.15.46:0/154981760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0140b4180 0x7fe0140b23c0 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.251+0000 7fe01ad84700 1 -- 172.21.15.46:0/154981760 >> 172.21.15.46:0/154981760 conn(0x7fe0140ce530 msgr2=0x7fe0141566f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.251+0000 7fe01ad84700 1 -- 172.21.15.46:0/154981760 shutdown_connections 2024-01-27T01:08:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.251+0000 7fe01ad84700 1 -- 172.21.15.46:0/154981760 wait complete. 2024-01-27T01:08:01.265 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-27T01:08:01.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.399+0000 7f2520763700 1 Processor -- start 2024-01-27T01:08:01.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.399+0000 7f2520763700 1 -- start start 2024-01-27T01:08:01.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.400+0000 7f2520763700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25180b2530 0x7f251815a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:01.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.400+0000 7f2520763700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f251815da10 0x7f251815de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:01.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.400+0000 7f2520763700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25180a9b90 con 0x7f25180b2160 2024-01-27T01:08:01.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.400+0000 7f2520763700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25180aa210 con 0x7f251815da10 2024-01-27T01:08:01.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.400+0000 7f2520763700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f251815d580 con 0x7f25180b2530 2024-01-27T01:08:01.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.400+0000 7f251ed00700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f251815da10 0x7f251815de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:01.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.400+0000 7f251e4ff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f25180b2160 0x7f25180abe50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57168/0 (socket says 172.21.15.46:57168) 2024-01-27T01:08:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.400+0000 7f251e4ff700 1 -- 172.21.15.46:0/3851452040 learned_addr learned my addr 172.21.15.46:0/3851452040 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.401+0000 7f251dcfe700 1 --2- 172.21.15.46:0/3851452040 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25180b2530 0x7f251815a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.401+0000 7f251ed00700 1 -- 172.21.15.46:0/3851452040 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25180b2530 msgr2=0x7f251815a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.401+0000 7f251ed00700 1 --2- 172.21.15.46:0/3851452040 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25180b2530 0x7f251815a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.401+0000 7f251ed00700 1 -- 172.21.15.46:0/3851452040 >> v1:172.21.15.46:6789/0 conn(0x7f25180b2160 legacy=0x7f25180abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.401+0000 7f251ed00700 1 -- 172.21.15.46:0/3851452040 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25180aa460 con 0x7f251815da10 2024-01-27T01:08:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.401+0000 7f251ed00700 1 --2- 172.21.15.46:0/3851452040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f251815da10 0x7f251815de90 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f2510009e60 tx=0x7f2510009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0a904fb8e6bec31 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.401+0000 7f251d4fd700 1 -- 172.21.15.46:0/3851452040 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251000e070 con 0x7f251815da10 2024-01-27T01:08:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.401+0000 7f251d4fd700 1 -- 172.21.15.46:0/3851452040 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251000bb60 con 0x7f251815da10 2024-01-27T01:08:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.401+0000 7f251d4fd700 1 -- 172.21.15.46:0/3851452040 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2510012430 con 0x7f251815da10 2024-01-27T01:08:01.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.402+0000 7f2520763700 1 -- 172.21.15.46:0/3851452040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f251815da10 msgr2=0x7f251815de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:01.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.402+0000 7f2520763700 1 --2- 172.21.15.46:0/3851452040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f251815da10 0x7f251815de90 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f2510009e60 tx=0x7f2510009780 comp rx=0 tx=0).stop 2024-01-27T01:08:01.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.402+0000 7f2520763700 1 -- 172.21.15.46:0/3851452040 shutdown_connections 2024-01-27T01:08:01.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.402+0000 7f2520763700 1 --2- 172.21.15.46:0/3851452040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f251815da10 0x7f251815de90 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.402+0000 7f2520763700 1 --2- 172.21.15.46:0/3851452040 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f25180b2530 0x7f251815a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.402+0000 7f2520763700 1 -- 172.21.15.46:0/3851452040 >> 172.21.15.46:0/3851452040 conn(0x7f25180ccbc0 msgr2=0x7f25180ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:01.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.402+0000 7f2520763700 1 -- 172.21.15.46:0/3851452040 shutdown_connections 2024-01-27T01:08:01.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.402+0000 7f2520763700 1 -- 172.21.15.46:0/3851452040 wait complete. 2024-01-27T01:08:01.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.403+0000 7f2520763700 1 Processor -- start 2024-01-27T01:08:01.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.403+0000 7f2520763700 1 -- start start 2024-01-27T01:08:01.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.404+0000 7f2520763700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25180b2160 0x7f2518169870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:01.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.404+0000 7f2520763700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25180b2530 0x7f25181426b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:01.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.404+0000 7f2520763700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f251815da10 0x7f2518152e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:01.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.404+0000 7f2520763700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2518142bf0 con 0x7f25180b2160 2024-01-27T01:08:01.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.404+0000 7f2520763700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25180ae240 con 0x7f25180b2530 2024-01-27T01:08:01.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.404+0000 7f2520763700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25180ad950 con 0x7f251815da10 2024-01-27T01:08:01.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.404+0000 7f251e4ff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25180b2160 0x7f2518169870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:01.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.404+0000 7f251e4ff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25180b2160 0x7f2518169870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55464/0 (socket says 172.21.15.46:55464) 2024-01-27T01:08:01.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.404+0000 7f251e4ff700 1 -- 172.21.15.46:0/2724809219 learned_addr learned my addr 172.21.15.46:0/2724809219 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:01.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.404+0000 7f251dcfe700 1 --2- 172.21.15.46:0/2724809219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25180b2530 0x7f25181426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:01.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.404+0000 7f251ed00700 1 --2- 172.21.15.46:0/2724809219 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f251815da10 0x7f2518152e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:01.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.404+0000 7f251e4ff700 1 -- 172.21.15.46:0/2724809219 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f251815da10 msgr2=0x7f2518152e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:01.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.404+0000 7f251e4ff700 1 --2- 172.21.15.46:0/2724809219 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f251815da10 0x7f2518152e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.405+0000 7f251e4ff700 1 -- 172.21.15.46:0/2724809219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25180b2530 msgr2=0x7f25181426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:01.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.405+0000 7f251e4ff700 1 --2- 172.21.15.46:0/2724809219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25180b2530 0x7f25181426b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.405+0000 7f251e4ff700 1 -- 172.21.15.46:0/2724809219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2510009a20 con 0x7f25180b2160 2024-01-27T01:08:01.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.405+0000 7f251e4ff700 1 --2- 172.21.15.46:0/2724809219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25180b2160 0x7f2518169870 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f250c00c7f0 tx=0x7f250c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:01.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.405+0000 7f250a7fc700 1 -- 172.21.15.46:0/2724809219 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c005b30 con 0x7f25180b2160 2024-01-27T01:08:01.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.405+0000 7f250a7fc700 1 -- 172.21.15.46:0/2724809219 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250c005cb0 con 0x7f25180b2160 2024-01-27T01:08:01.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.405+0000 7f2520763700 1 -- 172.21.15.46:0/2724809219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25180adbf0 con 0x7f25180b2160 2024-01-27T01:08:01.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.405+0000 7f250a7fc700 1 -- 172.21.15.46:0/2724809219 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c004a80 con 0x7f25180b2160 2024-01-27T01:08:01.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.405+0000 7f2520763700 1 -- 172.21.15.46:0/2724809219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2518152180 con 0x7f25180b2160 2024-01-27T01:08:01.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.406+0000 7f2520763700 1 -- 172.21.15.46:0/2724809219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251804c710 con 0x7f25180b2160 2024-01-27T01:08:01.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.407+0000 7f250a7fc700 1 -- 172.21.15.46:0/2724809219 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 141) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f250c0070d0 con 0x7f25180b2160 2024-01-27T01:08:01.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.408+0000 7f250a7fc700 1 -- 172.21.15.46:0/2724809219 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 8155+0+0 (secure 0 0 0) 0x7f250c083d90 con 0x7f25180b2160 2024-01-27T01:08:01.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.412+0000 7f250a7fc700 1 -- 172.21.15.46:0/2724809219 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f250c04b5a0 con 0x7f25180b2160 2024-01-27T01:08:01.473 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:01.473+0000 7f6ad5cab1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:08:01.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.668+0000 7f2520763700 1 -- 172.21.15.46:0/2724809219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f25180aa740 con 0x7f25180b2160 2024-01-27T01:08:01.668 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-27T01:08:01.669+0000 7f33ffe86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-27T01:08:01.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.673+0000 7f250a7fc700 1 -- 172.21.15.46:0/2724809219 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v139) v1 ==== 161+0+0 (secure 0 0 0) 0x7f250c050940 con 0x7f25180b2160 2024-01-27T01:08:01.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.677+0000 7f2520763700 1 -- 172.21.15.46:0/2724809219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25180b2160 msgr2=0x7f2518169870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:01.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.677+0000 7f2520763700 1 --2- 172.21.15.46:0/2724809219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25180b2160 0x7f2518169870 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f250c00c7f0 tx=0x7f250c007f70 comp rx=0 tx=0).stop 2024-01-27T01:08:01.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.677+0000 7f2520763700 1 -- 172.21.15.46:0/2724809219 shutdown_connections 2024-01-27T01:08:01.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.677+0000 7f2520763700 1 --2- 172.21.15.46:0/2724809219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25180b2160 0x7f2518169870 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.677+0000 7f2520763700 1 --2- 172.21.15.46:0/2724809219 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f251815da10 0x7f2518152e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.677+0000 7f2520763700 1 --2- 172.21.15.46:0/2724809219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25180b2530 0x7f25181426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.677 INFO:tasks.ceph.mds.a.smithi046.stderr: -3145> 2024-01-27T01:07:29.920+0000 7f95e4017700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-27T01:08:01.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.677+0000 7f2520763700 1 -- 172.21.15.46:0/2724809219 >> 172.21.15.46:0/2724809219 conn(0x7f25180ccbc0 msgr2=0x7f25180b10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:01.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.677+0000 7f2520763700 1 -- 172.21.15.46:0/2724809219 shutdown_connections 2024-01-27T01:08:01.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.677+0000 7f2520763700 1 -- 172.21.15.46:0/2724809219 wait complete. 2024-01-27T01:08:01.678 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-27T01:08:01.692 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-27T01:08:01.704 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-27T01:08:01.736 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:01.736+0000 7f6ad5cab1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:08:01.823 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:01.823+0000 7f6ad5cab1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:08:01.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.828+0000 7fccce51d700 1 Processor -- start 2024-01-27T01:08:01.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.828+0000 7fccce51d700 1 -- start start 2024-01-27T01:08:01.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.829+0000 7fccce51d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fccc815a520 0x7fccc8156970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:01.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.829+0000 7fccce51d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc80ae2f0 0x7fccc80ae750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:01.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.829+0000 7fccce51d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc80b0830 con 0x7fccc81565a0 2024-01-27T01:08:01.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.829+0000 7fccce51d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc80ae130 con 0x7fccc815a520 2024-01-27T01:08:01.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.829+0000 7fccce51d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc80aec90 con 0x7fccc80ae2f0 2024-01-27T01:08:01.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.830+0000 7fccccaba700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc80ae2f0 0x7fccc80ae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:01.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.830+0000 7fccccaba700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc80ae2f0 0x7fccc80ae750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:38562/0 (socket says 172.21.15.46:38562) 2024-01-27T01:08:01.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.830+0000 7fccc77fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fccc815a520 0x7fccc8156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:01.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.830+0000 7fccccaba700 1 -- 172.21.15.46:0/1607363145 learned_addr learned my addr 172.21.15.46:0/1607363145 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:01.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.830+0000 7fccc7fff700 1 --1- 172.21.15.46:0/1607363145 >> v1:172.21.15.106:6789/0 conn(0x7fccc81565a0 0x7fccc80acc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:38156/0 (socket says 172.21.15.46:38156) 2024-01-27T01:08:01.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.830+0000 7fccccaba700 1 -- 172.21.15.46:0/1607363145 >> v2:172.21.15.106:3300/0 conn(0x7fccc815a520 msgr2=0x7fccc8156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:01.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.830+0000 7fccccaba700 1 --2- 172.21.15.46:0/1607363145 >> v2:172.21.15.106:3300/0 conn(0x7fccc815a520 0x7fccc8156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.830+0000 7fccccaba700 1 -- 172.21.15.46:0/1607363145 >> v1:172.21.15.106:6789/0 conn(0x7fccc81565a0 legacy=0x7fccc80acc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:01.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.830+0000 7fccccaba700 1 -- 172.21.15.46:0/1607363145 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc80ad350 con 0x7fccc80ae2f0 2024-01-27T01:08:01.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.830+0000 7fccc77fe700 1 --2- 172.21.15.46:0/1607363145 >> v2:172.21.15.106:3300/0 conn(0x7fccc815a520 0x7fccc8156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:01.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.830+0000 7fccccaba700 1 --2- 172.21.15.46:0/1607363145 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc80ae2f0 0x7fccc80ae750 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fccc80ad4e0 tx=0x7fccc000ba50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8625f9d0b5d23cea server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:01.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.830+0000 7fccc6ffd700 1 -- 172.21.15.46:0/1607363145 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc0010070 con 0x7fccc80ae2f0 2024-01-27T01:08:01.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.830+0000 7fccc6ffd700 1 -- 172.21.15.46:0/1607363145 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc000dde0 con 0x7fccc80ae2f0 2024-01-27T01:08:01.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.830+0000 7fccc6ffd700 1 -- 172.21.15.46:0/1607363145 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc000b320 con 0x7fccc80ae2f0 2024-01-27T01:08:01.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.831+0000 7fccce51d700 1 -- 172.21.15.46:0/1607363145 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc80ae2f0 msgr2=0x7fccc80ae750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:01.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.831+0000 7fccce51d700 1 --2- 172.21.15.46:0/1607363145 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc80ae2f0 0x7fccc80ae750 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fccc80ad4e0 tx=0x7fccc000ba50 comp rx=0 tx=0).stop 2024-01-27T01:08:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.831+0000 7fccce51d700 1 -- 172.21.15.46:0/1607363145 shutdown_connections 2024-01-27T01:08:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.831+0000 7fccce51d700 1 --2- 172.21.15.46:0/1607363145 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc80ae2f0 0x7fccc80ae750 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.831+0000 7fccce51d700 1 --2- 172.21.15.46:0/1607363145 >> v2:172.21.15.106:3300/0 conn(0x7fccc815a520 0x7fccc8156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.831+0000 7fccce51d700 1 -- 172.21.15.46:0/1607363145 >> 172.21.15.46:0/1607363145 conn(0x7fccc80cdf80 msgr2=0x7fccc806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.831+0000 7fccce51d700 1 -- 172.21.15.46:0/1607363145 shutdown_connections 2024-01-27T01:08:01.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.831+0000 7fccce51d700 1 -- 172.21.15.46:0/1607363145 wait complete. 2024-01-27T01:08:01.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.832+0000 7fccce51d700 1 Processor -- start 2024-01-27T01:08:01.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.833+0000 7fccce51d700 1 -- start start 2024-01-27T01:08:01.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.833+0000 7fccce51d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80ae2f0 0x7fccc80abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:01.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.833+0000 7fccce51d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc81565a0 0x7fccc81589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:01.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.833+0000 7fccce51d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccc815a520 0x7fccc8169bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:01.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.833+0000 7fccce51d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc8158f10 con 0x7fccc80ae2f0 2024-01-27T01:08:01.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.833+0000 7fccce51d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc816a100 con 0x7fccc815a520 2024-01-27T01:08:01.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.833+0000 7fccce51d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc816a3a0 con 0x7fccc81565a0 2024-01-27T01:08:01.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.834+0000 7fccc7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80ae2f0 0x7fccc80abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:01.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.834+0000 7fccc7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80ae2f0 0x7fccc80abdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55484/0 (socket says 172.21.15.46:55484) 2024-01-27T01:08:01.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.834+0000 7fccc7fff700 1 -- 172.21.15.46:0/1707960200 learned_addr learned my addr 172.21.15.46:0/1707960200 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:01.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.834+0000 7fccc77fe700 1 --2- 172.21.15.46:0/1707960200 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc81565a0 0x7fccc81589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:01.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.834+0000 7fccccaba700 1 --2- 172.21.15.46:0/1707960200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccc815a520 0x7fccc8169bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:01.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.834+0000 7fccc77fe700 1 -- 172.21.15.46:0/1707960200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccc815a520 msgr2=0x7fccc8169bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:01.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.834+0000 7fccc77fe700 1 --2- 172.21.15.46:0/1707960200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccc815a520 0x7fccc8169bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.834+0000 7fccc77fe700 1 -- 172.21.15.46:0/1707960200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80ae2f0 msgr2=0x7fccc80abdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:01.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.834+0000 7fccc77fe700 1 --2- 172.21.15.46:0/1707960200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80ae2f0 0x7fccc80abdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:01.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.834+0000 7fccc77fe700 1 -- 172.21.15.46:0/1707960200 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb0009420 con 0x7fccc81565a0 2024-01-27T01:08:01.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.834+0000 7fccc7fff700 1 --2- 172.21.15.46:0/1707960200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80ae2f0 0x7fccc80abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:01.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.834+0000 7fccc77fe700 1 --2- 172.21.15.46:0/1707960200 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc81565a0 0x7fccc81589d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fccb000ea00 tx=0x7fccb00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:01.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.834+0000 7fccbffff700 1 -- 172.21.15.46:0/1707960200 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb000c6a0 con 0x7fccc81565a0 2024-01-27T01:08:01.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.834+0000 7fccbffff700 1 -- 172.21.15.46:0/1707960200 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb000c800 con 0x7fccc81565a0 2024-01-27T01:08:01.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.834+0000 7fccce51d700 1 -- 172.21.15.46:0/1707960200 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc000bcf0 con 0x7fccc81565a0 2024-01-27T01:08:01.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.834+0000 7fccbffff700 1 -- 172.21.15.46:0/1707960200 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb000cba0 con 0x7fccc81565a0 2024-01-27T01:08:01.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.835+0000 7fccce51d700 1 -- 172.21.15.46:0/1707960200 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc816a960 con 0x7fccc81565a0 2024-01-27T01:08:01.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.836+0000 7fccce51d700 1 -- 172.21.15.46:0/1707960200 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc816ab50 con 0x7fccc81565a0 2024-01-27T01:08:01.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.836+0000 7fccbffff700 1 -- 172.21.15.46:0/1707960200 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 141) v1 ==== 84453+0+0 (secure 0 0 0) 0x7fccb0013470 con 0x7fccc81565a0 2024-01-27T01:08:01.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.837+0000 7fccbffff700 1 -- 172.21.15.46:0/1707960200 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 8241+0+0 (secure 0 0 0) 0x7fccb0084c30 con 0x7fccc81565a0 2024-01-27T01:08:01.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:01.841+0000 7fccbffff700 1 -- 172.21.15.46:0/1707960200 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fccb00517e0 con 0x7fccc81565a0 2024-01-27T01:08:02.010 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:02.010+0000 7f6ad5cab1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:08:02.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.085+0000 7fccbffff700 1 -- 172.21.15.46:0/1707960200 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mgrmap(e 142) v1 ==== 84580+0+0 (secure 0 0 0) 0x7fccb0051290 con 0x7fccc81565a0 2024-01-27T01:08:02.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.085+0000 7fccbffff700 1 --2- 172.21.15.46:0/1707960200 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcca806fa00 0x7fcca8071de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:02.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.086+0000 7fccc7fff700 1 --2- 172.21.15.46:0/1707960200 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcca806fa00 0x7fcca8071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:02.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.087+0000 7fccc7fff700 1 --2- 172.21.15.46:0/1707960200 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcca806fa00 0x7fcca8071de0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fccb8004900 tx=0x7fccb8007f70 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.101+0000 7fccce51d700 1 -- 172.21.15.46:0/1707960200 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fccc804c710 con 0x7fccc81565a0 2024-01-27T01:08:02.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.676+0000 7fccbffff700 1 -- 172.21.15.46:0/1707960200 <== mon.2 v2:172.21.15.106:3301/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v140) v1 ==== 104+0+0 (secure 0 0 0) 0x7fccb0017070 con 0x7fccc81565a0 2024-01-27T01:08:02.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.679+0000 7fccce51d700 1 -- 172.21.15.46:0/1707960200 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcca806fa00 msgr2=0x7fcca8071de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:02.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.679+0000 7fccce51d700 1 --2- 172.21.15.46:0/1707960200 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcca806fa00 0x7fcca8071de0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fccb8004900 tx=0x7fccb8007f70 comp rx=0 tx=0).stop 2024-01-27T01:08:02.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.679+0000 7fccce51d700 1 -- 172.21.15.46:0/1707960200 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc81565a0 msgr2=0x7fccc81589d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:02.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.679+0000 7fccce51d700 1 --2- 172.21.15.46:0/1707960200 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc81565a0 0x7fccc81589d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fccb000ea00 tx=0x7fccb00091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:02.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.679+0000 7fccce51d700 1 -- 172.21.15.46:0/1707960200 shutdown_connections 2024-01-27T01:08:02.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.679+0000 7fccce51d700 1 --2- 172.21.15.46:0/1707960200 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fcca806fa00 0x7fcca8071de0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:02.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.679+0000 7fccce51d700 1 --2- 172.21.15.46:0/1707960200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccc80ae2f0 0x7fccc80abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:02.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.679+0000 7fccce51d700 1 --2- 172.21.15.46:0/1707960200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccc815a520 0x7fccc8169bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:02.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.679+0000 7fccce51d700 1 --2- 172.21.15.46:0/1707960200 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fccc81565a0 0x7fccc81589d0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:02.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.679+0000 7fccce51d700 1 -- 172.21.15.46:0/1707960200 >> 172.21.15.46:0/1707960200 conn(0x7fccc80cdf80 msgr2=0x7fccc806c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:02.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.679+0000 7fccce51d700 1 -- 172.21.15.46:0/1707960200 shutdown_connections 2024-01-27T01:08:02.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.679+0000 7fccce51d700 1 -- 172.21.15.46:0/1707960200 wait complete. 2024-01-27T01:08:02.694 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-27T01:08:02.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.880+0000 7f8afa256700 1 Processor -- start 2024-01-27T01:08:02.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.880+0000 7f8afa256700 1 -- start start 2024-01-27T01:08:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.881+0000 7f8afa256700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8af415d210 0x7f8af40cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.881+0000 7f8afa256700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af40cff70 con 0x7f8af40cbff0 2024-01-27T01:08:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.881+0000 7f8afa256700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af40cf400 con 0x7f8af415ce40 2024-01-27T01:08:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.881+0000 7f8afa256700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af40cc490 con 0x7f8af415d210 2024-01-27T01:08:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.881+0000 7f8af3fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8af40cbff0 0x7f8af40ade30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57188/0 (socket says 172.21.15.46:57188) 2024-01-27T01:08:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.881+0000 7f8af3fff700 1 -- 172.21.15.46:0/3511161814 learned_addr learned my addr 172.21.15.46:0/3511161814 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.881+0000 7f8af2ffd700 1 --2- 172.21.15.46:0/3511161814 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8af415d210 0x7f8af40cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.881+0000 7f8af27fc700 1 -- 172.21.15.46:0/3511161814 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289815182 0 0) 0x7f8af40cff70 con 0x7f8af40cbff0 2024-01-27T01:08:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.882+0000 7f8af27fc700 1 -- 172.21.15.46:0/3511161814 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ae0003570 con 0x7f8af40cbff0 2024-01-27T01:08:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.882+0000 7f8af27fc700 1 -- 172.21.15.46:0/3511161814 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832243843 0 0) 0x7f8af40cf400 con 0x7f8af415ce40 2024-01-27T01:08:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.882+0000 7f8af27fc700 1 -- 172.21.15.46:0/3511161814 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af40cff70 con 0x7f8af415ce40 2024-01-27T01:08:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.882+0000 7f8af2ffd700 1 -- 172.21.15.46:0/3511161814 >> v1:172.21.15.106:6789/0 conn(0x7f8af415ce40 legacy=0x7f8af40d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.882+0000 7f8af2ffd700 1 -- 172.21.15.46:0/3511161814 >> v1:172.21.15.46:6789/0 conn(0x7f8af40cbff0 legacy=0x7f8af40ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.882+0000 7f8af2ffd700 1 -- 172.21.15.46:0/3511161814 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af40cf660 con 0x7f8af415d210 2024-01-27T01:08:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.882+0000 7f8af2ffd700 1 --2- 172.21.15.46:0/3511161814 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8af415d210 0x7f8af40cbab0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8adc009910 tx=0x7f8adc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4962d4e8c9c1136 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.882+0000 7f8af27fc700 1 -- 172.21.15.46:0/3511161814 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc011630 con 0x7f8af415d210 2024-01-27T01:08:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.882+0000 7f8af27fc700 1 -- 172.21.15.46:0/3511161814 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc011790 con 0x7f8af415d210 2024-01-27T01:08:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.882+0000 7f8af27fc700 1 -- 172.21.15.46:0/3511161814 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc011b30 con 0x7f8af415d210 2024-01-27T01:08:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.883+0000 7f8afa256700 1 -- 172.21.15.46:0/3511161814 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8af415d210 msgr2=0x7f8af40cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.883+0000 7f8afa256700 1 --2- 172.21.15.46:0/3511161814 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8af415d210 0x7f8af40cbab0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8adc009910 tx=0x7f8adc0091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.883+0000 7f8afa256700 1 -- 172.21.15.46:0/3511161814 shutdown_connections 2024-01-27T01:08:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.883+0000 7f8afa256700 1 --2- 172.21.15.46:0/3511161814 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8af415d210 0x7f8af40cbab0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.883+0000 7f8afa256700 1 -- 172.21.15.46:0/3511161814 >> 172.21.15.46:0/3511161814 conn(0x7f8af406f840 msgr2=0x7f8af40d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:02.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.883+0000 7f8afa256700 1 -- 172.21.15.46:0/3511161814 shutdown_connections 2024-01-27T01:08:02.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.883+0000 7f8afa256700 1 -- 172.21.15.46:0/3511161814 wait complete. 2024-01-27T01:08:02.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.885+0000 7f8afa256700 1 Processor -- start 2024-01-27T01:08:02.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.885+0000 7f8afa256700 1 -- start start 2024-01-27T01:08:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.886+0000 7f8afa256700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8af40cbff0 0x7f8af409da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.886+0000 7f8afa256700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8af415ce40 0x7f8af415b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.886+0000 7f8afa256700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8af415d210 0x7f8af40c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.886+0000 7f8afa256700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af415c090 con 0x7f8af415d210 2024-01-27T01:08:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.886+0000 7f8afa256700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af40c8b00 con 0x7f8af415ce40 2024-01-27T01:08:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.886+0000 7f8afa256700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af409b710 con 0x7f8af40cbff0 2024-01-27T01:08:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.886+0000 7f8af3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8af415d210 0x7f8af40c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.886+0000 7f8af3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8af415d210 0x7f8af40c9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55498/0 (socket says 172.21.15.46:55498) 2024-01-27T01:08:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.886+0000 7f8af3fff700 1 -- 172.21.15.46:0/3780370491 learned_addr learned my addr 172.21.15.46:0/3780370491 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.886+0000 7f8af37fe700 1 --2- 172.21.15.46:0/3780370491 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8af40cbff0 0x7f8af409da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.886+0000 7f8af2ffd700 1 --2- 172.21.15.46:0/3780370491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8af415ce40 0x7f8af415b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.886+0000 7f8af3fff700 1 -- 172.21.15.46:0/3780370491 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8af40cbff0 msgr2=0x7f8af409da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.886+0000 7f8af3fff700 1 --2- 172.21.15.46:0/3780370491 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8af40cbff0 0x7f8af409da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.886+0000 7f8af3fff700 1 -- 172.21.15.46:0/3780370491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8af415ce40 msgr2=0x7f8af415b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.886+0000 7f8af3fff700 1 --2- 172.21.15.46:0/3780370491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8af415ce40 0x7f8af415b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.886+0000 7f8af3fff700 1 -- 172.21.15.46:0/3780370491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8adc009420 con 0x7f8af415d210 2024-01-27T01:08:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.886+0000 7f8af2ffd700 1 --2- 172.21.15.46:0/3780370491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8af415ce40 0x7f8af415b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:08:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.887+0000 7f8af3fff700 1 --2- 172.21.15.46:0/3780370491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8af415d210 0x7f8af40c9510 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00cc60 tx=0x7f8aec001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.887+0000 7f8aeb7fe700 1 -- 172.21.15.46:0/3780370491 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec00ce70 con 0x7f8af415d210 2024-01-27T01:08:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.887+0000 7f8aeb7fe700 1 -- 172.21.15.46:0/3780370491 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aec00ab70 con 0x7f8af415d210 2024-01-27T01:08:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.887+0000 7f8afa256700 1 -- 172.21.15.46:0/3780370491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af40c8540 con 0x7f8af415d210 2024-01-27T01:08:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.887+0000 7f8aeb7fe700 1 -- 172.21.15.46:0/3780370491 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec0070f0 con 0x7f8af415d210 2024-01-27T01:08:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.887+0000 7f8afa256700 1 -- 172.21.15.46:0/3780370491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af409b3a0 con 0x7f8af415d210 2024-01-27T01:08:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.888+0000 7f8afa256700 1 -- 172.21.15.46:0/3780370491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af404c710 con 0x7f8af415d210 2024-01-27T01:08:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.888+0000 7f8aeb7fe700 1 -- 172.21.15.46:0/3780370491 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 142) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f8aec0074a0 con 0x7f8af415d210 2024-01-27T01:08:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.889+0000 7f8aeb7fe700 1 --2- 172.21.15.46:0/3780370491 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8ad4066d80 0x7f8ad4069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.889+0000 7f8aeb7fe700 1 -- 172.21.15.46:0/3780370491 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 8241+0+0 (secure 0 0 0) 0x7f8aec085100 con 0x7f8af415d210 2024-01-27T01:08:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.889+0000 7f8af37fe700 1 --2- 172.21.15.46:0/3780370491 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8ad4066d80 0x7f8ad4069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.889+0000 7f8af37fe700 1 --2- 172.21.15.46:0/3780370491 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8ad4066d80 0x7f8ad4069230 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8af40b5d90 tx=0x7f8ae4009040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:02.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:02.893+0000 7f8aeb7fe700 1 -- 172.21.15.46:0/3780370491 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8aec0556b0 con 0x7f8af415d210 2024-01-27T01:08:03.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:03.099+0000 7f8afa256700 1 -- 172.21.15.46:0/3780370491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8af4068fd0 con 0x7f8af415d210 2024-01-27T01:08:03.125 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:03.125+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:04.039 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:04.039+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:04.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.085+0000 7f8aeb7fe700 1 -- 172.21.15.46:0/3780370491 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f8aec051ff0 con 0x7f8af415d210 2024-01-27T01:08:04.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.096+0000 7f8aeb7fe700 1 -- 172.21.15.46:0/3780370491 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v251) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8aec0556b0 con 0x7f8af415d210 2024-01-27T01:08:04.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.100+0000 7f8afa256700 1 -- 172.21.15.46:0/3780370491 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8ad4066d80 msgr2=0x7f8ad4069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:04.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.100+0000 7f8afa256700 1 --2- 172.21.15.46:0/3780370491 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8ad4066d80 0x7f8ad4069230 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8af40b5d90 tx=0x7f8ae4009040 comp rx=0 tx=0).stop 2024-01-27T01:08:04.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.100+0000 7f8afa256700 1 -- 172.21.15.46:0/3780370491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8af415d210 msgr2=0x7f8af40c9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:04.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.100+0000 7f8afa256700 1 --2- 172.21.15.46:0/3780370491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8af415d210 0x7f8af40c9510 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00cc60 tx=0x7f8aec001ae0 comp rx=0 tx=0).stop 2024-01-27T01:08:04.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.100+0000 7f8afa256700 1 -- 172.21.15.46:0/3780370491 shutdown_connections 2024-01-27T01:08:04.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.100+0000 7f8afa256700 1 --2- 172.21.15.46:0/3780370491 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8ad4066d80 0x7f8ad4069230 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:04.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.100+0000 7f8afa256700 1 --2- 172.21.15.46:0/3780370491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8af415d210 0x7f8af40c9510 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:04.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.100+0000 7f8afa256700 1 --2- 172.21.15.46:0/3780370491 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8af40cbff0 0x7f8af409da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:04.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.100+0000 7f8afa256700 1 --2- 172.21.15.46:0/3780370491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8af415ce40 0x7f8af415b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:04.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.101+0000 7f8afa256700 1 -- 172.21.15.46:0/3780370491 >> 172.21.15.46:0/3780370491 conn(0x7f8af406f840 msgr2=0x7f8af40b5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:04.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.101+0000 7f8afa256700 1 -- 172.21.15.46:0/3780370491 shutdown_connections 2024-01-27T01:08:04.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.101+0000 7f8afa256700 1 -- 172.21.15.46:0/3780370491 wait complete. 2024-01-27T01:08:04.102 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-01-27T01:08:04.118 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-27T01:08:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.261+0000 7f502e0bd700 1 Processor -- start 2024-01-27T01:08:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.261+0000 7f502e0bd700 1 -- start start 2024-01-27T01:08:04.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.262+0000 7f502e0bd700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f50280ca0e0 0x7f50280ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:04.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.262+0000 7f502e0bd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f502806c0b0 0x7f50280afe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:04.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.262+0000 7f502e0bd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f502806c520 0x7f50280b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:04.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.262+0000 7f502e0bd700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50280cada0 con 0x7f502806c520 2024-01-27T01:08:04.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.262+0000 7f502e0bd700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5028158010 con 0x7f50280ca0e0 2024-01-27T01:08:04.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.262+0000 7f502e0bd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50281582f0 con 0x7f502806c0b0 2024-01-27T01:08:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.262+0000 7f5027fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f502806c520 0x7f50280b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.262+0000 7f5026ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f502806c0b0 0x7f50280afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.262+0000 7f5027fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f502806c520 0x7f50280b3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55510/0 (socket says 172.21.15.46:55510) 2024-01-27T01:08:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.262+0000 7f5027fff700 1 -- 172.21.15.46:0/4145456808 learned_addr learned my addr 172.21.15.46:0/4145456808 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.262+0000 7f50277fe700 1 --2- 172.21.15.46:0/4145456808 >> v2:172.21.15.106:3300/0 conn(0x7f50280ca0e0 0x7f50280ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.262+0000 7f5027fff700 1 -- 172.21.15.46:0/4145456808 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f502806c0b0 msgr2=0x7f50280afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.262+0000 7f5027fff700 1 --2- 172.21.15.46:0/4145456808 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f502806c0b0 0x7f50280afe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.263+0000 7f5027fff700 1 -- 172.21.15.46:0/4145456808 >> v2:172.21.15.106:3300/0 conn(0x7f50280ca0e0 msgr2=0x7f50280ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.263+0000 7f5027fff700 1 --2- 172.21.15.46:0/4145456808 >> v2:172.21.15.106:3300/0 conn(0x7f50280ca0e0 0x7f50280ca4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.263+0000 7f5027fff700 1 -- 172.21.15.46:0/4145456808 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50281585d0 con 0x7f502806c520 2024-01-27T01:08:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.263+0000 7f5027fff700 1 --2- 172.21.15.46:0/4145456808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f502806c520 0x7f50280b3a50 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f502806b850 tx=0x7f5020009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9596b05359cc179f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.263+0000 7f50267fc700 1 -- 172.21.15.46:0/4145456808 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502000e070 con 0x7f502806c520 2024-01-27T01:08:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.263+0000 7f50267fc700 1 -- 172.21.15.46:0/4145456808 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502000b970 con 0x7f502806c520 2024-01-27T01:08:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.263+0000 7f50267fc700 1 -- 172.21.15.46:0/4145456808 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50200092e0 con 0x7f502806c520 2024-01-27T01:08:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.263+0000 7f502e0bd700 1 -- 172.21.15.46:0/4145456808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f502806c520 msgr2=0x7f50280b3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.263+0000 7f502e0bd700 1 --2- 172.21.15.46:0/4145456808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f502806c520 0x7f50280b3a50 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f502806b850 tx=0x7f5020009840 comp rx=0 tx=0).stop 2024-01-27T01:08:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.264+0000 7f502e0bd700 1 -- 172.21.15.46:0/4145456808 shutdown_connections 2024-01-27T01:08:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.264+0000 7f502e0bd700 1 --2- 172.21.15.46:0/4145456808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f502806c520 0x7f50280b3a50 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.264+0000 7f502e0bd700 1 --2- 172.21.15.46:0/4145456808 >> v2:172.21.15.106:3300/0 conn(0x7f50280ca0e0 0x7f50280ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.264+0000 7f502e0bd700 1 --2- 172.21.15.46:0/4145456808 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f502806c0b0 0x7f50280afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.264+0000 7f502e0bd700 1 -- 172.21.15.46:0/4145456808 >> 172.21.15.46:0/4145456808 conn(0x7f50280ce530 msgr2=0x7f50280ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.264+0000 7f502e0bd700 1 -- 172.21.15.46:0/4145456808 shutdown_connections 2024-01-27T01:08:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.264+0000 7f502e0bd700 1 -- 172.21.15.46:0/4145456808 wait complete. 2024-01-27T01:08:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.265+0000 7f502e0bd700 1 Processor -- start 2024-01-27T01:08:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.265+0000 7f502e0bd700 1 -- start start 2024-01-27T01:08:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.266+0000 7f502e0bd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f502806c0b0 0x7f5028150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.266+0000 7f502e0bd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f502806c520 0x7f50280c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.266+0000 7f502e0bd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50280ca0e0 0x7f50280c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.266+0000 7f502e0bd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50280cd6b0 con 0x7f50280ca0e0 2024-01-27T01:08:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.266+0000 7f502e0bd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50280c81e0 con 0x7f502806c0b0 2024-01-27T01:08:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.266+0000 7f502e0bd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50280ccb10 con 0x7f502806c520 2024-01-27T01:08:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.266+0000 7f5027fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50280ca0e0 0x7f50280c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.266+0000 7f5027fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50280ca0e0 0x7f50280c7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55516/0 (socket says 172.21.15.46:55516) 2024-01-27T01:08:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.266+0000 7f5027fff700 1 -- 172.21.15.46:0/1546948799 learned_addr learned my addr 172.21.15.46:0/1546948799 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.266+0000 7f5026ffd700 1 --2- 172.21.15.46:0/1546948799 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f502806c520 0x7f50280c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.266+0000 7f50277fe700 1 --2- 172.21.15.46:0/1546948799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f502806c0b0 0x7f5028150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.266+0000 7f5027fff700 1 -- 172.21.15.46:0/1546948799 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f502806c520 msgr2=0x7f50280c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.266+0000 7f5027fff700 1 --2- 172.21.15.46:0/1546948799 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f502806c520 0x7f50280c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.266+0000 7f5027fff700 1 -- 172.21.15.46:0/1546948799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f502806c0b0 msgr2=0x7f5028150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.266+0000 7f5027fff700 1 --2- 172.21.15.46:0/1546948799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f502806c0b0 0x7f5028150c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.266+0000 7f5027fff700 1 -- 172.21.15.46:0/1546948799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5020009ae0 con 0x7f50280ca0e0 2024-01-27T01:08:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.267+0000 7f5027fff700 1 --2- 172.21.15.46:0/1546948799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50280ca0e0 0x7f50280c7ca0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f502806f390 tx=0x7f5020018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.267+0000 7f501f7fe700 1 -- 172.21.15.46:0/1546948799 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5020008ad0 con 0x7f50280ca0e0 2024-01-27T01:08:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.267+0000 7f501f7fe700 1 -- 172.21.15.46:0/1546948799 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5020008c50 con 0x7f50280ca0e0 2024-01-27T01:08:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.267+0000 7f501f7fe700 1 -- 172.21.15.46:0/1546948799 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5020026460 con 0x7f50280ca0e0 2024-01-27T01:08:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.267+0000 7f502e0bd700 1 -- 172.21.15.46:0/1546948799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50280ccd80 con 0x7f50280ca0e0 2024-01-27T01:08:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.267+0000 7f502e0bd700 1 -- 172.21.15.46:0/1546948799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50280cd340 con 0x7f50280ca0e0 2024-01-27T01:08:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.268+0000 7f501d7fa700 1 -- 172.21.15.46:0/1546948799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50000052b0 con 0x7f50280ca0e0 2024-01-27T01:08:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.268+0000 7f501f7fe700 1 -- 172.21.15.46:0/1546948799 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f5020003040 con 0x7f50280ca0e0 2024-01-27T01:08:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.269+0000 7f501f7fe700 1 --2- 172.21.15.46:0/1546948799 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f50080670e0 0x7f5008069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.269+0000 7f501f7fe700 1 -- 172.21.15.46:0/1546948799 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7803+0+0 (secure 0 0 0) 0x7f5020086e70 con 0x7f50280ca0e0 2024-01-27T01:08:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.269+0000 7f50277fe700 1 --2- 172.21.15.46:0/1546948799 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f50080670e0 0x7f5008069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.270+0000 7f50277fe700 1 --2- 172.21.15.46:0/1546948799 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f50080670e0 0x7f5008069590 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5018009d10 tx=0x7f50180091a0 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:04.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.273+0000 7f501f7fe700 1 -- 172.21.15.46:0/1546948799 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5020053520 con 0x7f50280ca0e0 2024-01-27T01:08:04.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:04.482+0000 7f501d7fa700 1 -- 172.21.15.46:0/1546948799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5000005560 con 0x7f50280ca0e0 2024-01-27T01:08:04.964 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:04.964+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:05.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.101+0000 7f501f7fe700 1 -- 172.21.15.46:0/1546948799 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v252) v1 ==== 169+0+0 (secure 0 0 0) 0x7f502005c070 con 0x7f50280ca0e0 2024-01-27T01:08:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.106+0000 7f501d7fa700 1 -- 172.21.15.46:0/1546948799 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f50080670e0 msgr2=0x7f5008069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.106+0000 7f501d7fa700 1 --2- 172.21.15.46:0/1546948799 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f50080670e0 0x7f5008069590 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5018009d10 tx=0x7f50180091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.106+0000 7f501d7fa700 1 -- 172.21.15.46:0/1546948799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50280ca0e0 msgr2=0x7f50280c7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.106+0000 7f501d7fa700 1 --2- 172.21.15.46:0/1546948799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50280ca0e0 0x7f50280c7ca0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f502806f390 tx=0x7f5020018000 comp rx=0 tx=0).stop 2024-01-27T01:08:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.106+0000 7f501d7fa700 1 -- 172.21.15.46:0/1546948799 shutdown_connections 2024-01-27T01:08:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.106+0000 7f501d7fa700 1 --2- 172.21.15.46:0/1546948799 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f50080670e0 0x7f5008069590 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.106+0000 7f501d7fa700 1 --2- 172.21.15.46:0/1546948799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50280ca0e0 0x7f50280c7ca0 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.106+0000 7f501d7fa700 1 --2- 172.21.15.46:0/1546948799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f502806c0b0 0x7f5028150c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.106+0000 7f501d7fa700 1 --2- 172.21.15.46:0/1546948799 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f502806c520 0x7f50280c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.106+0000 7f501d7fa700 1 -- 172.21.15.46:0/1546948799 >> 172.21.15.46:0/1546948799 conn(0x7f50280ce530 msgr2=0x7f50280afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:05.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.106+0000 7f501d7fa700 1 -- 172.21.15.46:0/1546948799 shutdown_connections 2024-01-27T01:08:05.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.108+0000 7f501d7fa700 1 -- 172.21.15.46:0/1546948799 wait complete. 2024-01-27T01:08:05.109 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-01-27T01:08:05.128 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:08:05.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.265+0000 7f2da669e700 1 Processor -- start 2024-01-27T01:08:05.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.265+0000 7f2da669e700 1 -- start start 2024-01-27T01:08:05.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.266+0000 7f2da669e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2da01573f0 0x7f2da01577c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:05.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.266+0000 7f2da669e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2da00b2570 0x7f2da01612e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:05.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.266+0000 7f2da669e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da00aedf0 con 0x7f2da00b40a0 2024-01-27T01:08:05.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.266+0000 7f2da669e700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2da00b3510 con 0x7f2da00b2570 2024-01-27T01:08:05.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.266+0000 7f2da669e700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2da00af0d0 con 0x7f2da01573f0 2024-01-27T01:08:05.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.266+0000 7f2da4c3b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2da00b2570 0x7f2da01612e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:05.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.266+0000 7f2da4c3b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2da00b2570 0x7f2da01612e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55522/0 (socket says 172.21.15.46:55522) 2024-01-27T01:08:05.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.266+0000 7f2da4c3b700 1 -- 172.21.15.46:0/3695094687 learned_addr learned my addr 172.21.15.46:0/3695094687 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:05.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.267+0000 7f2d9ffff700 1 --2- 172.21.15.46:0/3695094687 >> v2:172.21.15.106:3300/0 conn(0x7f2da01573f0 0x7f2da01577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:05.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.267+0000 7f2da4c3b700 1 -- 172.21.15.46:0/3695094687 >> v2:172.21.15.106:3300/0 conn(0x7f2da01573f0 msgr2=0x7f2da01577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:05.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.267+0000 7f2da4c3b700 1 --2- 172.21.15.46:0/3695094687 >> v2:172.21.15.106:3300/0 conn(0x7f2da01573f0 0x7f2da01577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.267+0000 7f2da4c3b700 1 -- 172.21.15.46:0/3695094687 >> v1:172.21.15.106:6789/0 conn(0x7f2da00b40a0 legacy=0x7f2da00b1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:05.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.267+0000 7f2da4c3b700 1 -- 172.21.15.46:0/3695094687 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da00af3b0 con 0x7f2da00b2570 2024-01-27T01:08:05.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.267+0000 7f2d9ffff700 1 --2- 172.21.15.46:0/3695094687 >> v2:172.21.15.106:3300/0 conn(0x7f2da01573f0 0x7f2da01577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:05.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.267+0000 7f2da4c3b700 1 --2- 172.21.15.46:0/3695094687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da00b2570 0x7f2da01612e0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f2da00649a0 tx=0x7f2d98009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91c47b8aae05f699 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:05.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.267+0000 7f2d9effd700 1 -- 172.21.15.46:0/3695094687 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9800e070 con 0x7f2da00b2570 2024-01-27T01:08:05.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.267+0000 7f2d9effd700 1 -- 172.21.15.46:0/3695094687 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9800ba50 con 0x7f2da00b2570 2024-01-27T01:08:05.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.267+0000 7f2d9effd700 1 -- 172.21.15.46:0/3695094687 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d980092e0 con 0x7f2da00b2570 2024-01-27T01:08:05.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.268+0000 7f2da669e700 1 -- 172.21.15.46:0/3695094687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da00b2570 msgr2=0x7f2da01612e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:05.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.268+0000 7f2da669e700 1 --2- 172.21.15.46:0/3695094687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da00b2570 0x7f2da01612e0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f2da00649a0 tx=0x7f2d98009840 comp rx=0 tx=0).stop 2024-01-27T01:08:05.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.268+0000 7f2da669e700 1 -- 172.21.15.46:0/3695094687 shutdown_connections 2024-01-27T01:08:05.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.268+0000 7f2da669e700 1 --2- 172.21.15.46:0/3695094687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da00b2570 0x7f2da01612e0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.268+0000 7f2da669e700 1 --2- 172.21.15.46:0/3695094687 >> v2:172.21.15.106:3300/0 conn(0x7f2da01573f0 0x7f2da01577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.268+0000 7f2da669e700 1 -- 172.21.15.46:0/3695094687 >> 172.21.15.46:0/3695094687 conn(0x7f2da0087f60 msgr2=0x7f2da0088330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:05.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.268+0000 7f2da669e700 1 -- 172.21.15.46:0/3695094687 shutdown_connections 2024-01-27T01:08:05.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.268+0000 7f2da669e700 1 -- 172.21.15.46:0/3695094687 wait complete. 2024-01-27T01:08:05.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.269+0000 7f2da669e700 1 Processor -- start 2024-01-27T01:08:05.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.269+0000 7f2da669e700 1 -- start start 2024-01-27T01:08:05.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.270+0000 7f2da669e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2da00b2570 0x7f2da015abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:05.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.270+0000 7f2da669e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2da00b40a0 0x7f2da0069cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:05.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.270+0000 7f2da669e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da01573f0 0x7f2da006a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:05.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.270+0000 7f2da669e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da016e090 con 0x7f2da01573f0 2024-01-27T01:08:05.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.270+0000 7f2da669e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da0157e70 con 0x7f2da00b40a0 2024-01-27T01:08:05.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.270+0000 7f2da669e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da0158150 con 0x7f2da00b2570 2024-01-27T01:08:05.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.270+0000 7f2da4c3b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da01573f0 0x7f2da006a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:05.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.270+0000 7f2da4c3b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da01573f0 0x7f2da006a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55530/0 (socket says 172.21.15.46:55530) 2024-01-27T01:08:05.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.270+0000 7f2da4c3b700 1 -- 172.21.15.46:0/381630662 learned_addr learned my addr 172.21.15.46:0/381630662 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:05.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.270+0000 7f2d9f7fe700 1 --2- 172.21.15.46:0/381630662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2da00b40a0 0x7f2da0069cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:05.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.270+0000 7f2d9ffff700 1 --2- 172.21.15.46:0/381630662 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2da00b2570 0x7f2da015abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:05.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.271+0000 7f2da4c3b700 1 -- 172.21.15.46:0/381630662 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2da00b2570 msgr2=0x7f2da015abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:05.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.271+0000 7f2da4c3b700 1 --2- 172.21.15.46:0/381630662 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2da00b2570 0x7f2da015abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.271+0000 7f2da4c3b700 1 -- 172.21.15.46:0/381630662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2da00b40a0 msgr2=0x7f2da0069cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:05.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.271+0000 7f2da4c3b700 1 --2- 172.21.15.46:0/381630662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2da00b40a0 0x7f2da0069cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.271+0000 7f2da4c3b700 1 -- 172.21.15.46:0/381630662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d98009ae0 con 0x7f2da01573f0 2024-01-27T01:08:05.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.271+0000 7f2d9f7fe700 1 --2- 172.21.15.46:0/381630662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2da00b40a0 0x7f2da0069cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:05.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.271+0000 7f2da4c3b700 1 --2- 172.21.15.46:0/381630662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da01573f0 0x7f2da006a1f0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f2da006e9e0 tx=0x7f2d98018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:05.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.271+0000 7f2d97fff700 1 -- 172.21.15.46:0/381630662 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9800be60 con 0x7f2da01573f0 2024-01-27T01:08:05.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.271+0000 7f2d97fff700 1 -- 172.21.15.46:0/381630662 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d98008ce0 con 0x7f2da01573f0 2024-01-27T01:08:05.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.271+0000 7f2da669e700 1 -- 172.21.15.46:0/381630662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da0158430 con 0x7f2da01573f0 2024-01-27T01:08:05.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.271+0000 7f2d97fff700 1 -- 172.21.15.46:0/381630662 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d980266b0 con 0x7f2da01573f0 2024-01-27T01:08:05.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.271+0000 7f2da669e700 1 -- 172.21.15.46:0/381630662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da015b100 con 0x7f2da01573f0 2024-01-27T01:08:05.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.272+0000 7f2da669e700 1 -- 172.21.15.46:0/381630662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da015b290 con 0x7f2da01573f0 2024-01-27T01:08:05.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.273+0000 7f2d97fff700 1 -- 172.21.15.46:0/381630662 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f2d98003040 con 0x7f2da01573f0 2024-01-27T01:08:05.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.274+0000 7f2d97fff700 1 --2- 172.21.15.46:0/381630662 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2d800670e0 0x7f2d80069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:05.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.274+0000 7f2d97fff700 1 -- 172.21.15.46:0/381630662 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7389+0+0 (secure 0 0 0) 0x7f2d98086e80 con 0x7f2da01573f0 2024-01-27T01:08:05.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.274+0000 7f2d9ffff700 1 --2- 172.21.15.46:0/381630662 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2d800670e0 0x7f2d80069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:05.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.275+0000 7f2d9ffff700 1 --2- 172.21.15.46:0/381630662 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2d800670e0 0x7f2d80069590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2d90006f00 tx=0x7f2d90008040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:05.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.278+0000 7f2d97fff700 1 -- 172.21.15.46:0/381630662 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2d980536d0 con 0x7f2da01573f0 2024-01-27T01:08:05.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.488+0000 7f2da669e700 1 -- 172.21.15.46:0/381630662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2da004c710 con 0x7f2da01573f0 2024-01-27T01:08:05.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.489+0000 7f2d97fff700 1 -- 172.21.15.46:0/381630662 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) v1 ==== 74+0+17594 (secure 0 0 0) 0x7f2d98056cf0 con 0x7f2da01573f0 2024-01-27T01:08:05.488 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:05.488 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":252,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:08:05.095424+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:07:39.209497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:07:40.386606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6834/1087205220":"2024-01-28T01:08:01.670110+0000","172.21.15.106:0/558414078":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/2701940868":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/3540935782":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/3966339865":"2024-01-28T01:08:01.068164+0000","172.21.15.46:6832/1087205220":"2024-01-28T01:08:01.670110+0000","172.21.15.106:0/3958137863":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/2317042118":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/1816027959":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/2241845238":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1948052707":"2024-01-28T01:07:53.996892+0000","172.21.15.106:6836/62434":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1918093439":"2024-01-28T01:07:53.996892+0000","172.21.15.106:6837/62434":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/469270357":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1388668891":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1199239718":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/3351808187":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/2250157080":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/2626066195":"2024-01-28T01:07:53.996892+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:08:05.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.491+0000 7f2da669e700 1 -- 172.21.15.46:0/381630662 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2d800670e0 msgr2=0x7f2d80069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:05.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.491+0000 7f2da669e700 1 --2- 172.21.15.46:0/381630662 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2d800670e0 0x7f2d80069590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2d90006f00 tx=0x7f2d90008040 comp rx=0 tx=0).stop 2024-01-27T01:08:05.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.491+0000 7f2da669e700 1 -- 172.21.15.46:0/381630662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da01573f0 msgr2=0x7f2da006a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.491+0000 7f2da669e700 1 --2- 172.21.15.46:0/381630662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da01573f0 0x7f2da006a1f0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f2da006e9e0 tx=0x7f2d98018000 comp rx=0 tx=0).stop 2024-01-27T01:08:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.491+0000 7f2da669e700 1 -- 172.21.15.46:0/381630662 shutdown_connections 2024-01-27T01:08:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.491+0000 7f2da669e700 1 --2- 172.21.15.46:0/381630662 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2d800670e0 0x7f2d80069590 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.491+0000 7f2da669e700 1 --2- 172.21.15.46:0/381630662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da01573f0 0x7f2da006a1f0 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.491+0000 7f2da669e700 1 --2- 172.21.15.46:0/381630662 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2da00b2570 0x7f2da015abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.491+0000 7f2da669e700 1 --2- 172.21.15.46:0/381630662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2da00b40a0 0x7f2da0069cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.491+0000 7f2da669e700 1 -- 172.21.15.46:0/381630662 >> 172.21.15.46:0/381630662 conn(0x7f2da0087f60 msgr2=0x7f2da01567c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.491+0000 7f2da669e700 1 -- 172.21.15.46:0/381630662 shutdown_connections 2024-01-27T01:08:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.491+0000 7f2da669e700 1 -- 172.21.15.46:0/381630662 wait complete. 2024-01-27T01:08:05.505 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-01-27T01:07:39.209497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '246', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2024-01-27T01:07:40.386606+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '247', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}}] 2024-01-27T01:08:05.505 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-27T01:08:05.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.649+0000 7f984eedc700 1 Processor -- start 2024-01-27T01:08:05.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.649+0000 7f984eedc700 1 -- start start 2024-01-27T01:08:05.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.650+0000 7f984eedc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f984806ebe0 0x7f9848067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:05.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.650+0000 7f984eedc700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984806fdb0 con 0x7f98481597e0 2024-01-27T01:08:05.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.650+0000 7f984eedc700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984806f1a0 con 0x7f9848067af0 2024-01-27T01:08:05.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.650+0000 7f984eedc700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f984806f480 con 0x7f984806ebe0 2024-01-27T01:08:05.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.650+0000 7f984d479700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f98481597e0 0x7f9848159bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57200/0 (socket says 172.21.15.46:57200) 2024-01-27T01:08:05.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.650+0000 7f984d479700 1 -- 172.21.15.46:0/1197557154 learned_addr learned my addr 172.21.15.46:0/1197557154 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:05.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.650+0000 7f9847fff700 1 --2- 172.21.15.46:0/1197557154 >> v2:172.21.15.106:3300/0 conn(0x7f984806ebe0 0x7f9848067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:05.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.650+0000 7f98477fe700 1 -- 172.21.15.46:0/1197557154 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572181827 0 0) 0x7f984806fdb0 con 0x7f98481597e0 2024-01-27T01:08:05.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.650+0000 7f98477fe700 1 -- 172.21.15.46:0/1197557154 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9838003570 con 0x7f98481597e0 2024-01-27T01:08:05.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f98477fe700 1 -- 172.21.15.46:0/1197557154 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3009009022 0 0) 0x7f9838003570 con 0x7f98481597e0 2024-01-27T01:08:05.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f98477fe700 1 -- 172.21.15.46:0/1197557154 >> v2:172.21.15.106:3300/0 conn(0x7f984806ebe0 msgr2=0x7f9848067ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:05.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f98477fe700 1 --2- 172.21.15.46:0/1197557154 >> v2:172.21.15.106:3300/0 conn(0x7f984806ebe0 0x7f9848067ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f98477fe700 1 -- 172.21.15.46:0/1197557154 >> v1:172.21.15.106:6789/0 conn(0x7f9848067af0 legacy=0x7f98480ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:05.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f98477fe700 1 -- 172.21.15.46:0/1197557154 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984806f760 con 0x7f98481597e0 2024-01-27T01:08:05.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f98477fe700 1 -- 172.21.15.46:0/1197557154 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f98400017f0 con 0x7f98481597e0 2024-01-27T01:08:05.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f9847fff700 1 --2- 172.21.15.46:0/1197557154 >> v2:172.21.15.106:3300/0 conn(0x7f984806ebe0 0x7f9848067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:05.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f98477fe700 1 -- 172.21.15.46:0/1197557154 >> v1:172.21.15.46:6789/0 conn(0x7f98481597e0 legacy=0x7f9848159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:05.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f98477fe700 1 --2- 172.21.15.46:0/1197557154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9838003a60 0x7f9838003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:05.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f98477fe700 1 --2- 172.21.15.46:0/1197557154 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9838004810 0x7f9838004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:05.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f98477fe700 1 --2- 172.21.15.46:0/1197557154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98380051c0 0x7f983800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:05.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f98477fe700 1 -- 172.21.15.46:0/1197557154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984806f480 con 0x7f98380051c0 2024-01-27T01:08:05.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f98477fe700 1 -- 172.21.15.46:0/1197557154 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983800c1e0 con 0x7f9838003a60 2024-01-27T01:08:05.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f98477fe700 1 -- 172.21.15.46:0/1197557154 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983800c3a0 con 0x7f9838004810 2024-01-27T01:08:05.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f984d479700 1 --2- 172.21.15.46:0/1197557154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98380051c0 0x7f983800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:05.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f984cc78700 1 --2- 172.21.15.46:0/1197557154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9838003a60 0x7f9838003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:05.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f9847fff700 1 --2- 172.21.15.46:0/1197557154 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9838004810 0x7f9838004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:05.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f984d479700 1 -- 172.21.15.46:0/1197557154 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9838004810 msgr2=0x7f9838004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f984d479700 1 --2- 172.21.15.46:0/1197557154 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9838004810 0x7f9838004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f984d479700 1 -- 172.21.15.46:0/1197557154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9838003a60 msgr2=0x7f9838003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f984d479700 1 --2- 172.21.15.46:0/1197557154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9838003a60 0x7f9838003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.651+0000 7f984d479700 1 -- 172.21.15.46:0/1197557154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9838003570 con 0x7f98380051c0 2024-01-27T01:08:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.652+0000 7f984d479700 1 -- 172.21.15.46:0/1197557154 reap_dead start 2024-01-27T01:08:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.652+0000 7f984d479700 1 --2- 172.21.15.46:0/1197557154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98380051c0 0x7f983800ba60 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f984000a920 tx=0x7f98400021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7f3fe2a47d144f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.652+0000 7f98477fe700 1 -- 172.21.15.46:0/1197557154 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98400070d0 con 0x7f98380051c0 2024-01-27T01:08:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.652+0000 7f98477fe700 1 -- 172.21.15.46:0/1197557154 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9840007250 con 0x7f98380051c0 2024-01-27T01:08:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.652+0000 7f98477fe700 1 -- 172.21.15.46:0/1197557154 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9840007610 con 0x7f98380051c0 2024-01-27T01:08:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.652+0000 7f984eedc700 1 -- 172.21.15.46:0/1197557154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98380051c0 msgr2=0x7f983800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.652+0000 7f984eedc700 1 --2- 172.21.15.46:0/1197557154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98380051c0 0x7f983800ba60 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f984000a920 tx=0x7f98400021f0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.652+0000 7f984eedc700 1 -- 172.21.15.46:0/1197557154 shutdown_connections 2024-01-27T01:08:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.652+0000 7f984eedc700 1 --2- 172.21.15.46:0/1197557154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98380051c0 0x7f983800ba60 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.652+0000 7f984eedc700 1 -- 172.21.15.46:0/1197557154 >> 172.21.15.46:0/1197557154 conn(0x7f98480cdf80 msgr2=0x7f984806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.652+0000 7f984eedc700 1 -- 172.21.15.46:0/1197557154 shutdown_connections 2024-01-27T01:08:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.652+0000 7f984eedc700 1 -- 172.21.15.46:0/1197557154 wait complete. 2024-01-27T01:08:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.654+0000 7f984eedc700 1 Processor -- start 2024-01-27T01:08:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.654+0000 7f984eedc700 1 -- start start 2024-01-27T01:08:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f984eedc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98380051c0 0x7f9848142860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f984eedc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f984815cb10 0x7f984815cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f984eedc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f984816a3e0 0x7f984816a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f984eedc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984815bd50 con 0x7f98380051c0 2024-01-27T01:08:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f984eedc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984815c030 con 0x7f984816a3e0 2024-01-27T01:08:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f984eedc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984815c310 con 0x7f984815cb10 2024-01-27T01:08:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f984cc78700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98380051c0 0x7f9848142860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f984cc78700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98380051c0 0x7f9848142860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55550/0 (socket says 172.21.15.46:55550) 2024-01-27T01:08:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f984cc78700 1 -- 172.21.15.46:0/2970305491 learned_addr learned my addr 172.21.15.46:0/2970305491 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f984d479700 1 --2- 172.21.15.46:0/2970305491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f984816a3e0 0x7f984816a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f9847fff700 1 --2- 172.21.15.46:0/2970305491 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f984815cb10 0x7f984815cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f984cc78700 1 -- 172.21.15.46:0/2970305491 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f984815cb10 msgr2=0x7f984815cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f984cc78700 1 --2- 172.21.15.46:0/2970305491 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f984815cb10 0x7f984815cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f984cc78700 1 -- 172.21.15.46:0/2970305491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f984816a3e0 msgr2=0x7f984816a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f984cc78700 1 --2- 172.21.15.46:0/2970305491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f984816a3e0 0x7f984816a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f984cc78700 1 -- 172.21.15.46:0/2970305491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984815c5f0 con 0x7f98380051c0 2024-01-27T01:08:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f984d479700 1 --2- 172.21.15.46:0/2970305491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f984816a3e0 0x7f984816a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.655+0000 7f984cc78700 1 --2- 172.21.15.46:0/2970305491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98380051c0 0x7f9848142860 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f983c002ab0 tx=0x7f983c007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:05.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.656+0000 7f9844ff9700 1 -- 172.21.15.46:0/2970305491 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983c003470 con 0x7f98380051c0 2024-01-27T01:08:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.656+0000 7f9844ff9700 1 -- 172.21.15.46:0/2970305491 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983c00f9a0 con 0x7f98380051c0 2024-01-27T01:08:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.656+0000 7f984eedc700 1 -- 172.21.15.46:0/2970305491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9848142da0 con 0x7f98380051c0 2024-01-27T01:08:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.656+0000 7f9844ff9700 1 -- 172.21.15.46:0/2970305491 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983c00fda0 con 0x7f98380051c0 2024-01-27T01:08:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.656+0000 7f984eedc700 1 -- 172.21.15.46:0/2970305491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98480b23a0 con 0x7f98380051c0 2024-01-27T01:08:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.657+0000 7f984eedc700 1 -- 172.21.15.46:0/2970305491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f984815c830 con 0x7f98380051c0 2024-01-27T01:08:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.657+0000 7f9844ff9700 1 -- 172.21.15.46:0/2970305491 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f983c013600 con 0x7f98380051c0 2024-01-27T01:08:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.658+0000 7f9844ff9700 1 --2- 172.21.15.46:0/2970305491 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f982c0670e0 0x7f982c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.658+0000 7f9844ff9700 1 -- 172.21.15.46:0/2970305491 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7389+0+0 (secure 0 0 0) 0x7f983c085500 con 0x7f98380051c0 2024-01-27T01:08:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.658+0000 7f9847fff700 1 --2- 172.21.15.46:0/2970305491 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f982c0670e0 0x7f982c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:05.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.658+0000 7f9847fff700 1 --2- 172.21.15.46:0/2970305491 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f982c0670e0 0x7f982c069590 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9834006f00 tx=0x7f9834008040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.662+0000 7f9844ff9700 1 -- 172.21.15.46:0/2970305491 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f984815c830 con 0x7f98380051c0 2024-01-27T01:08:05.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.871+0000 7f984eedc700 1 -- 172.21.15.46:0/2970305491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f984815c830 con 0x7f98380051c0 2024-01-27T01:08:05.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.871+0000 7f9844ff9700 1 -- 172.21.15.46:0/2970305491 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v252) v1 ==== 93+0+10 (secure 0 0 0) 0x7f984815c830 con 0x7f98380051c0 2024-01-27T01:08:05.870 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-27T01:08:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.873+0000 7f984eedc700 1 -- 172.21.15.46:0/2970305491 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f982c0670e0 msgr2=0x7f982c069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.873+0000 7f984eedc700 1 --2- 172.21.15.46:0/2970305491 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f982c0670e0 0x7f982c069590 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9834006f00 tx=0x7f9834008040 comp rx=0 tx=0).stop 2024-01-27T01:08:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.873+0000 7f984eedc700 1 -- 172.21.15.46:0/2970305491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98380051c0 msgr2=0x7f9848142860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.873+0000 7f984eedc700 1 --2- 172.21.15.46:0/2970305491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98380051c0 0x7f9848142860 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f983c002ab0 tx=0x7f983c007d10 comp rx=0 tx=0).stop 2024-01-27T01:08:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.873+0000 7f984eedc700 1 -- 172.21.15.46:0/2970305491 shutdown_connections 2024-01-27T01:08:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.873+0000 7f984eedc700 1 --2- 172.21.15.46:0/2970305491 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f982c0670e0 0x7f982c069590 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.874+0000 7f984eedc700 1 --2- 172.21.15.46:0/2970305491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98380051c0 0x7f9848142860 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.874+0000 7f984eedc700 1 --2- 172.21.15.46:0/2970305491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f984816a3e0 0x7f984816a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.874+0000 7f984eedc700 1 --2- 172.21.15.46:0/2970305491 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f984815cb10 0x7f984815cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.874+0000 7f984eedc700 1 -- 172.21.15.46:0/2970305491 >> 172.21.15.46:0/2970305491 conn(0x7f98480cdf80 msgr2=0x7f98480ab850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.874+0000 7f984eedc700 1 -- 172.21.15.46:0/2970305491 shutdown_connections 2024-01-27T01:08:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:05.874+0000 7f984eedc700 1 -- 172.21.15.46:0/2970305491 wait complete. 2024-01-27T01:08:05.886 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-01-27T01:08:05.902 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:05.902+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.034+0000 7f870b83a700 1 Processor -- start 2024-01-27T01:08:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.034+0000 7f870b83a700 1 -- start start 2024-01-27T01:08:06.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.035+0000 7f870b83a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87041599d0 0x7f8704159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.035+0000 7f870b83a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f87040ab1c0 0x7f87040b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.035+0000 7f870b83a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f870415d8c0 0x7f870415dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.035+0000 7f870b83a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f870415e260 con 0x7f87040ab1c0 2024-01-27T01:08:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.035+0000 7f870b83a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87040b1a60 con 0x7f870415d8c0 2024-01-27T01:08:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.035+0000 7f870b83a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87040ab5f0 con 0x7f87041599d0 2024-01-27T01:08:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.035+0000 7f8708dd5700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f87040ab1c0 0x7f87040b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.035+0000 7f8708dd5700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f87040ab1c0 0x7f87040b0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55560/0 (socket says 172.21.15.46:55560) 2024-01-27T01:08:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.035+0000 7f8708dd5700 1 -- 172.21.15.46:0/16775369 learned_addr learned my addr 172.21.15.46:0/16775369 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.035+0000 7f87095d6700 1 --2- 172.21.15.46:0/16775369 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87041599d0 0x7f8704159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.035+0000 7f8709dd7700 1 --2- 172.21.15.46:0/16775369 >> v2:172.21.15.106:3300/0 conn(0x7f870415d8c0 0x7f870415dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.035+0000 7f8708dd5700 1 -- 172.21.15.46:0/16775369 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87041599d0 msgr2=0x7f8704159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.035+0000 7f8708dd5700 1 --2- 172.21.15.46:0/16775369 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87041599d0 0x7f8704159da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.035+0000 7f8708dd5700 1 -- 172.21.15.46:0/16775369 >> v2:172.21.15.106:3300/0 conn(0x7f870415d8c0 msgr2=0x7f870415dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.035+0000 7f8708dd5700 1 --2- 172.21.15.46:0/16775369 >> v2:172.21.15.106:3300/0 conn(0x7f870415d8c0 0x7f870415dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.035+0000 7f8708dd5700 1 -- 172.21.15.46:0/16775369 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87040b0f40 con 0x7f87040ab1c0 2024-01-27T01:08:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.035+0000 7f8708dd5700 1 --2- 172.21.15.46:0/16775369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87040ab1c0 0x7f87040b0a00 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f87040686d0 tx=0x7f86f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a64a2e400741b68 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.036+0000 7f86fbfff700 1 -- 172.21.15.46:0/16775369 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f400b640 con 0x7f87040ab1c0 2024-01-27T01:08:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.036+0000 7f86fbfff700 1 -- 172.21.15.46:0/16775369 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f400b7a0 con 0x7f87040ab1c0 2024-01-27T01:08:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.036+0000 7f86fbfff700 1 -- 172.21.15.46:0/16775369 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f400bb60 con 0x7f87040ab1c0 2024-01-27T01:08:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.036+0000 7f870b83a700 1 -- 172.21.15.46:0/16775369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87040ab1c0 msgr2=0x7f87040b0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.036+0000 7f870b83a700 1 --2- 172.21.15.46:0/16775369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87040ab1c0 0x7f87040b0a00 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f87040686d0 tx=0x7f86f4009250 comp rx=0 tx=0).stop 2024-01-27T01:08:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.036+0000 7f870b83a700 1 -- 172.21.15.46:0/16775369 shutdown_connections 2024-01-27T01:08:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.036+0000 7f870b83a700 1 --2- 172.21.15.46:0/16775369 >> v2:172.21.15.106:3300/0 conn(0x7f870415d8c0 0x7f870415dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.036+0000 7f870b83a700 1 --2- 172.21.15.46:0/16775369 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87041599d0 0x7f8704159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.036+0000 7f870b83a700 1 --2- 172.21.15.46:0/16775369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87040ab1c0 0x7f87040b0a00 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.036+0000 7f870b83a700 1 -- 172.21.15.46:0/16775369 >> 172.21.15.46:0/16775369 conn(0x7f870406f840 msgr2=0x7f8704087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.037+0000 7f870b83a700 1 -- 172.21.15.46:0/16775369 shutdown_connections 2024-01-27T01:08:06.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.037+0000 7f870b83a700 1 -- 172.21.15.46:0/16775369 wait complete. 2024-01-27T01:08:06.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.038+0000 7f870b83a700 1 Processor -- start 2024-01-27T01:08:06.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.038+0000 7f870b83a700 1 -- start start 2024-01-27T01:08:06.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.039+0000 7f870b83a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87040ab1c0 0x7f87041449b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.039+0000 7f870b83a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87041599d0 0x7f87040ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.039+0000 7f870b83a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f870415d8c0 0x7f87040abd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.039+0000 7f870b83a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870409ace0 con 0x7f87040ab1c0 2024-01-27T01:08:06.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.039+0000 7f870b83a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870409e300 con 0x7f870415d8c0 2024-01-27T01:08:06.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.039+0000 7f870b83a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870409a3b0 con 0x7f87041599d0 2024-01-27T01:08:06.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.039+0000 7f87095d6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87040ab1c0 0x7f87041449b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.039+0000 7f87095d6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87040ab1c0 0x7f87041449b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55564/0 (socket says 172.21.15.46:55564) 2024-01-27T01:08:06.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.039+0000 7f87095d6700 1 -- 172.21.15.46:0/3359853358 learned_addr learned my addr 172.21.15.46:0/3359853358 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.039+0000 7f8709dd7700 1 --2- 172.21.15.46:0/3359853358 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f870415d8c0 0x7f87040abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.039+0000 7f8708dd5700 1 --2- 172.21.15.46:0/3359853358 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87041599d0 0x7f87040ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.039+0000 7f87095d6700 1 -- 172.21.15.46:0/3359853358 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87041599d0 msgr2=0x7f87040ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.039+0000 7f87095d6700 1 --2- 172.21.15.46:0/3359853358 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87041599d0 0x7f87040ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.039+0000 7f87095d6700 1 -- 172.21.15.46:0/3359853358 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f870415d8c0 msgr2=0x7f87040abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.039+0000 7f87095d6700 1 --2- 172.21.15.46:0/3359853358 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f870415d8c0 0x7f87040abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.039+0000 7f87095d6700 1 -- 172.21.15.46:0/3359853358 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f40094f0 con 0x7f87040ab1c0 2024-01-27T01:08:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.040+0000 7f87095d6700 1 --2- 172.21.15.46:0/3359853358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87040ab1c0 0x7f87041449b0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f870000ec80 tx=0x7f8700009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.040+0000 7f86f97fa700 1 -- 172.21.15.46:0/3359853358 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870000c850 con 0x7f87040ab1c0 2024-01-27T01:08:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.040+0000 7f86f97fa700 1 -- 172.21.15.46:0/3359853358 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870000c9b0 con 0x7f87040ab1c0 2024-01-27T01:08:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.040+0000 7f86f97fa700 1 -- 172.21.15.46:0/3359853358 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870000cd30 con 0x7f87040ab1c0 2024-01-27T01:08:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.040+0000 7f870b83a700 1 -- 172.21.15.46:0/3359853358 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f870409a690 con 0x7f87040ab1c0 2024-01-27T01:08:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.040+0000 7f870b83a700 1 -- 172.21.15.46:0/3359853358 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87040ad790 con 0x7f87040ab1c0 2024-01-27T01:08:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.041+0000 7f870b83a700 1 -- 172.21.15.46:0/3359853358 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f870404c710 con 0x7f87040ab1c0 2024-01-27T01:08:06.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.041+0000 7f86f97fa700 1 -- 172.21.15.46:0/3359853358 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f8700013680 con 0x7f87040ab1c0 2024-01-27T01:08:06.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.042+0000 7f86f97fa700 1 --2- 172.21.15.46:0/3359853358 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f86e80670e0 0x7f86e8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.042+0000 7f86f97fa700 1 -- 172.21.15.46:0/3359853358 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7389+0+0 (secure 0 0 0) 0x7f8700085450 con 0x7f87040ab1c0 2024-01-27T01:08:06.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.042+0000 7f8708dd5700 1 --2- 172.21.15.46:0/3359853358 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f86e80670e0 0x7f86e8069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.043+0000 7f8708dd5700 1 --2- 172.21.15.46:0/3359853358 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f86e80670e0 0x7f86e8069590 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f87040686d0 tx=0x7f86f4017000 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.047+0000 7f86f97fa700 1 -- 172.21.15.46:0/3359853358 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8700051db0 con 0x7f87040ab1c0 2024-01-27T01:08:06.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.254+0000 7f870b83a700 1 -- 172.21.15.46:0/3359853358 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f87040b11c0 con 0x7f87040ab1c0 2024-01-27T01:08:06.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.255+0000 7f86f97fa700 1 -- 172.21.15.46:0/3359853358 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v252) v1 ==== 107+0+11 (secure 0 0 0) 0x7f87000553d0 con 0x7f87040ab1c0 2024-01-27T01:08:06.254 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-27T01:08:06.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.257+0000 7f870b83a700 1 -- 172.21.15.46:0/3359853358 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f86e80670e0 msgr2=0x7f86e8069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:06.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.257+0000 7f870b83a700 1 --2- 172.21.15.46:0/3359853358 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f86e80670e0 0x7f86e8069590 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f87040686d0 tx=0x7f86f4017000 comp rx=0 tx=0).stop 2024-01-27T01:08:06.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.257+0000 7f870b83a700 1 -- 172.21.15.46:0/3359853358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87040ab1c0 msgr2=0x7f87041449b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:06.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.257+0000 7f870b83a700 1 --2- 172.21.15.46:0/3359853358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87040ab1c0 0x7f87041449b0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f870000ec80 tx=0x7f8700009120 comp rx=0 tx=0).stop 2024-01-27T01:08:06.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.257+0000 7f870b83a700 1 -- 172.21.15.46:0/3359853358 shutdown_connections 2024-01-27T01:08:06.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.257+0000 7f870b83a700 1 --2- 172.21.15.46:0/3359853358 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f86e80670e0 0x7f86e8069590 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.257+0000 7f870b83a700 1 --2- 172.21.15.46:0/3359853358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87040ab1c0 0x7f87041449b0 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.257+0000 7f870b83a700 1 --2- 172.21.15.46:0/3359853358 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f870415d8c0 0x7f87040abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.257+0000 7f870b83a700 1 --2- 172.21.15.46:0/3359853358 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f87041599d0 0x7f87040ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.257+0000 7f870b83a700 1 -- 172.21.15.46:0/3359853358 >> 172.21.15.46:0/3359853358 conn(0x7f870406f840 msgr2=0x7f87040b47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:06.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.257+0000 7f870b83a700 1 -- 172.21.15.46:0/3359853358 shutdown_connections 2024-01-27T01:08:06.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.257+0000 7f870b83a700 1 -- 172.21.15.46:0/3359853358 wait complete. 2024-01-27T01:08:06.270 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-01-27T01:08:06.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.414+0000 7f9e2a3a0700 1 Processor -- start 2024-01-27T01:08:06.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.414+0000 7f9e2a3a0700 1 -- start start 2024-01-27T01:08:06.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.415+0000 7f9e2a3a0700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9e241580a0 0x7f9e240b3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.415+0000 7f9e2a3a0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9e24158470 0x7f9e240b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.415+0000 7f9e2a3a0700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e240b4200 con 0x7f9e240b20e0 2024-01-27T01:08:06.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.415+0000 7f9e2a3a0700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e24069a50 con 0x7f9e24158470 2024-01-27T01:08:06.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.415+0000 7f9e2a3a0700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e240ceeb0 con 0x7f9e241580a0 2024-01-27T01:08:06.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.415+0000 7f9e28b9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9e24158470 0x7f9e240b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.415+0000 7f9e29b9f700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9e240b20e0 0x7f9e240ce180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57212/0 (socket says 172.21.15.46:57212) 2024-01-27T01:08:06.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.415+0000 7f9e29b9f700 1 -- 172.21.15.46:0/3427287964 learned_addr learned my addr 172.21.15.46:0/3427287964 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:06.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.415+0000 7f9e2939e700 1 --2- 172.21.15.46:0/3427287964 >> v2:172.21.15.106:3300/0 conn(0x7f9e241580a0 0x7f9e240b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.415+0000 7f9e28b9d700 1 -- 172.21.15.46:0/3427287964 >> v2:172.21.15.106:3300/0 conn(0x7f9e241580a0 msgr2=0x7f9e240b3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:06.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.415+0000 7f9e28b9d700 1 --2- 172.21.15.46:0/3427287964 >> v2:172.21.15.106:3300/0 conn(0x7f9e241580a0 0x7f9e240b3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.415+0000 7f9e28b9d700 1 -- 172.21.15.46:0/3427287964 >> v1:172.21.15.46:6789/0 conn(0x7f9e240b20e0 legacy=0x7f9e240ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:06.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.415+0000 7f9e28b9d700 1 -- 172.21.15.46:0/3427287964 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e240b1320 con 0x7f9e24158470 2024-01-27T01:08:06.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.415+0000 7f9e1bfff700 1 -- 172.21.15.46:0/3427287964 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 704492174 0 0) 0x7f9e240b4200 con 0x7f9e240b20e0 2024-01-27T01:08:06.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.415+0000 7f9e2939e700 1 --2- 172.21.15.46:0/3427287964 >> v2:172.21.15.106:3300/0 conn(0x7f9e241580a0 0x7f9e240b3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:08:06.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.416+0000 7f9e28b9d700 1 --2- 172.21.15.46:0/3427287964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e24158470 0x7f9e240b3cc0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f9e2406f580 tx=0x7f9e140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cccc3442b1c88548 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:06.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.416+0000 7f9e1bfff700 1 -- 172.21.15.46:0/3427287964 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1400b6b0 con 0x7f9e24158470 2024-01-27T01:08:06.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.416+0000 7f9e1bfff700 1 -- 172.21.15.46:0/3427287964 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1400b810 con 0x7f9e24158470 2024-01-27T01:08:06.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.416+0000 7f9e1bfff700 1 -- 172.21.15.46:0/3427287964 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1400bbb0 con 0x7f9e24158470 2024-01-27T01:08:06.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.416+0000 7f9e2a3a0700 1 -- 172.21.15.46:0/3427287964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e24158470 msgr2=0x7f9e240b3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:06.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.416+0000 7f9e2a3a0700 1 --2- 172.21.15.46:0/3427287964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e24158470 0x7f9e240b3cc0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f9e2406f580 tx=0x7f9e140091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.416+0000 7f9e2a3a0700 1 -- 172.21.15.46:0/3427287964 shutdown_connections 2024-01-27T01:08:06.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.416+0000 7f9e2a3a0700 1 --2- 172.21.15.46:0/3427287964 >> v2:172.21.15.106:3300/0 conn(0x7f9e241580a0 0x7f9e240b3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.416+0000 7f9e2a3a0700 1 --2- 172.21.15.46:0/3427287964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e24158470 0x7f9e240b3cc0 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.416+0000 7f9e2a3a0700 1 -- 172.21.15.46:0/3427287964 >> 172.21.15.46:0/3427287964 conn(0x7f9e2406bc90 msgr2=0x7f9e2406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:06.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.417+0000 7f9e2a3a0700 1 -- 172.21.15.46:0/3427287964 shutdown_connections 2024-01-27T01:08:06.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.417+0000 7f9e2a3a0700 1 -- 172.21.15.46:0/3427287964 wait complete. 2024-01-27T01:08:06.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.418+0000 7f9e2a3a0700 1 Processor -- start 2024-01-27T01:08:06.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.418+0000 7f9e2a3a0700 1 -- start start 2024-01-27T01:08:06.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.419+0000 7f9e2a3a0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9e240b20e0 0x7f9e2414aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.419+0000 7f9e2a3a0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e241580a0 0x7f9e2414fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.419+0000 7f9e2a3a0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9e24158470 0x7f9e2414d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.419+0000 7f9e2a3a0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240aec90 con 0x7f9e241580a0 2024-01-27T01:08:06.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.419+0000 7f9e2a3a0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2414e040 con 0x7f9e24158470 2024-01-27T01:08:06.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.419+0000 7f9e2a3a0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240af500 con 0x7f9e240b20e0 2024-01-27T01:08:06.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.419+0000 7f9e28b9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e241580a0 0x7f9e2414fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.419+0000 7f9e28b9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e241580a0 0x7f9e2414fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55576/0 (socket says 172.21.15.46:55576) 2024-01-27T01:08:06.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.419+0000 7f9e28b9d700 1 -- 172.21.15.46:0/57044627 learned_addr learned my addr 172.21.15.46:0/57044627 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:06.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.419+0000 7f9e29b9f700 1 --2- 172.21.15.46:0/57044627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9e24158470 0x7f9e2414d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.419+0000 7f9e2939e700 1 --2- 172.21.15.46:0/57044627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9e240b20e0 0x7f9e2414aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.419+0000 7f9e28b9d700 1 -- 172.21.15.46:0/57044627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9e240b20e0 msgr2=0x7f9e2414aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:06.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.419+0000 7f9e28b9d700 1 --2- 172.21.15.46:0/57044627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9e240b20e0 0x7f9e2414aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.419+0000 7f9e28b9d700 1 -- 172.21.15.46:0/57044627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9e24158470 msgr2=0x7f9e2414d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:06.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.419+0000 7f9e28b9d700 1 --2- 172.21.15.46:0/57044627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9e24158470 0x7f9e2414d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.419+0000 7f9e28b9d700 1 -- 172.21.15.46:0/57044627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e14009420 con 0x7f9e241580a0 2024-01-27T01:08:06.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.420+0000 7f9e28b9d700 1 --2- 172.21.15.46:0/57044627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e241580a0 0x7f9e2414fc20 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f9e14013000 tx=0x7f9e14017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:06.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.420+0000 7f9e197fa700 1 -- 172.21.15.46:0/57044627 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1400bb90 con 0x7f9e241580a0 2024-01-27T01:08:06.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.420+0000 7f9e197fa700 1 -- 172.21.15.46:0/57044627 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e14011420 con 0x7f9e241580a0 2024-01-27T01:08:06.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.420+0000 7f9e2a3a0700 1 -- 172.21.15.46:0/57044627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e240af7e0 con 0x7f9e241580a0 2024-01-27T01:08:06.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.420+0000 7f9e197fa700 1 -- 172.21.15.46:0/57044627 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e140117a0 con 0x7f9e241580a0 2024-01-27T01:08:06.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.420+0000 7f9e2a3a0700 1 -- 172.21.15.46:0/57044627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e240afda0 con 0x7f9e241580a0 2024-01-27T01:08:06.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.421+0000 7f9e2a3a0700 1 -- 172.21.15.46:0/57044627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e24069f90 con 0x7f9e241580a0 2024-01-27T01:08:06.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.421+0000 7f9e197fa700 1 -- 172.21.15.46:0/57044627 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f9e14011b90 con 0x7f9e241580a0 2024-01-27T01:08:06.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.422+0000 7f9e197fa700 1 --2- 172.21.15.46:0/57044627 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9e080670e0 0x7f9e08069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.422+0000 7f9e197fa700 1 -- 172.21.15.46:0/57044627 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7389+0+0 (secure 0 0 0) 0x7f9e14086430 con 0x7f9e241580a0 2024-01-27T01:08:06.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.422+0000 7f9e2939e700 1 --2- 172.21.15.46:0/57044627 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9e080670e0 0x7f9e08069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.423+0000 7f9e2939e700 1 --2- 172.21.15.46:0/57044627 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9e080670e0 0x7f9e08069590 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000ad30 tx=0x7f9e20006f90 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:06.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.427+0000 7f9e197fa700 1 -- 172.21.15.46:0/57044627 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9e14052d90 con 0x7f9e241580a0 2024-01-27T01:08:06.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.638+0000 7f9e2a3a0700 1 -- 172.21.15.46:0/57044627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f9e240b1600 con 0x7f9e241580a0 2024-01-27T01:08:06.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.639+0000 7f9e197fa700 1 -- 172.21.15.46:0/57044627 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v252) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9e140563b0 con 0x7f9e241580a0 2024-01-27T01:08:06.638 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-27T01:08:06.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.641+0000 7f9e2a3a0700 1 -- 172.21.15.46:0/57044627 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9e080670e0 msgr2=0x7f9e08069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:06.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.641+0000 7f9e2a3a0700 1 --2- 172.21.15.46:0/57044627 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9e080670e0 0x7f9e08069590 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000ad30 tx=0x7f9e20006f90 comp rx=0 tx=0).stop 2024-01-27T01:08:06.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.641+0000 7f9e2a3a0700 1 -- 172.21.15.46:0/57044627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e241580a0 msgr2=0x7f9e2414fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:06.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.641+0000 7f9e2a3a0700 1 --2- 172.21.15.46:0/57044627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e241580a0 0x7f9e2414fc20 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f9e14013000 tx=0x7f9e14017000 comp rx=0 tx=0).stop 2024-01-27T01:08:06.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.641+0000 7f9e2a3a0700 1 -- 172.21.15.46:0/57044627 shutdown_connections 2024-01-27T01:08:06.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.641+0000 7f9e2a3a0700 1 --2- 172.21.15.46:0/57044627 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9e080670e0 0x7f9e08069590 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.641+0000 7f9e2a3a0700 1 --2- 172.21.15.46:0/57044627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9e24158470 0x7f9e2414d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.641+0000 7f9e2a3a0700 1 --2- 172.21.15.46:0/57044627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9e240b20e0 0x7f9e2414aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.641+0000 7f9e2a3a0700 1 --2- 172.21.15.46:0/57044627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e241580a0 0x7f9e2414fc20 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.641+0000 7f9e2a3a0700 1 -- 172.21.15.46:0/57044627 >> 172.21.15.46:0/57044627 conn(0x7f9e2406bc90 msgr2=0x7f9e240cd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:06.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.641+0000 7f9e2a3a0700 1 -- 172.21.15.46:0/57044627 shutdown_connections 2024-01-27T01:08:06.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.641+0000 7f9e2a3a0700 1 -- 172.21.15.46:0/57044627 wait complete. 2024-01-27T01:08:06.654 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:08:06.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.790+0000 7f9b5180e700 1 Processor -- start 2024-01-27T01:08:06.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.790+0000 7f9b5180e700 1 -- start start 2024-01-27T01:08:06.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.790+0000 7f9b5180e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b4c06d310 0x7f9b4c0b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.790+0000 7f9b5180e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9b4c06c290 0x7f9b4c0c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.790+0000 7f9b5180e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b4c06d840 con 0x7f9b4c0b3bf0 2024-01-27T01:08:06.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.791+0000 7f9b5180e700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b4c15b910 con 0x7f9b4c06c290 2024-01-27T01:08:06.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.791+0000 7f9b5180e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c157d00 con 0x7f9b4c06d310 2024-01-27T01:08:06.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.791+0000 7f9b4b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9b4c06c290 0x7f9b4c0c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.791+0000 7f9b4b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9b4c06c290 0x7f9b4c0c9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55578/0 (socket says 172.21.15.46:55578) 2024-01-27T01:08:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.791+0000 7f9b4b7fe700 1 -- 172.21.15.46:0/3828022512 learned_addr learned my addr 172.21.15.46:0/3828022512 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.791+0000 7f9b4a7fc700 1 --2- 172.21.15.46:0/3828022512 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b4c06d310 0x7f9b4c0b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.791+0000 7f9b4b7fe700 1 -- 172.21.15.46:0/3828022512 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b4c06d310 msgr2=0x7f9b4c0b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.791+0000 7f9b4b7fe700 1 --2- 172.21.15.46:0/3828022512 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b4c06d310 0x7f9b4c0b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.791+0000 7f9b4b7fe700 1 -- 172.21.15.46:0/3828022512 >> v1:172.21.15.106:6789/0 conn(0x7f9b4c0b3bf0 legacy=0x7f9b4c15b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.791+0000 7f9b49ffb700 1 -- 172.21.15.46:0/3828022512 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150590588 0 0) 0x7f9b4c06d840 con 0x7f9b4c0b3bf0 2024-01-27T01:08:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.791+0000 7f9b4b7fe700 1 -- 172.21.15.46:0/3828022512 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b4c06db20 con 0x7f9b4c06c290 2024-01-27T01:08:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.791+0000 7f9b4b7fe700 1 --2- 172.21.15.46:0/3828022512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b4c06c290 0x7f9b4c0c9fc0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f9b44009f20 tx=0x7f9b44009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e9c9aa21c89f12 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.791+0000 7f9b49ffb700 1 -- 172.21.15.46:0/3828022512 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4400e070 con 0x7f9b4c06c290 2024-01-27T01:08:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.791+0000 7f9b49ffb700 1 -- 172.21.15.46:0/3828022512 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b4400b9d0 con 0x7f9b4c06c290 2024-01-27T01:08:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.791+0000 7f9b49ffb700 1 -- 172.21.15.46:0/3828022512 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b440092e0 con 0x7f9b4c06c290 2024-01-27T01:08:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.792+0000 7f9b5180e700 1 -- 172.21.15.46:0/3828022512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b4c06c290 msgr2=0x7f9b4c0c9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:06.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.792+0000 7f9b5180e700 1 --2- 172.21.15.46:0/3828022512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b4c06c290 0x7f9b4c0c9fc0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f9b44009f20 tx=0x7f9b44009840 comp rx=0 tx=0).stop 2024-01-27T01:08:06.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.792+0000 7f9b5180e700 1 -- 172.21.15.46:0/3828022512 shutdown_connections 2024-01-27T01:08:06.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.792+0000 7f9b5180e700 1 --2- 172.21.15.46:0/3828022512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b4c06c290 0x7f9b4c0c9fc0 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.792+0000 7f9b5180e700 1 --2- 172.21.15.46:0/3828022512 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b4c06d310 0x7f9b4c0b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.792+0000 7f9b5180e700 1 -- 172.21.15.46:0/3828022512 >> 172.21.15.46:0/3828022512 conn(0x7f9b4c0ce530 msgr2=0x7f9b4c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:06.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.792+0000 7f9b5180e700 1 -- 172.21.15.46:0/3828022512 shutdown_connections 2024-01-27T01:08:06.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.792+0000 7f9b5180e700 1 -- 172.21.15.46:0/3828022512 wait complete. 2024-01-27T01:08:06.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.794+0000 7f9b5180e700 1 Processor -- start 2024-01-27T01:08:06.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.794+0000 7f9b5180e700 1 -- start start 2024-01-27T01:08:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.795+0000 7f9b5180e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b4c06c290 0x7f9b4c151970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.795+0000 7f9b5180e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b4c06d310 0x7f9b4c0cdb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.795+0000 7f9b5180e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b4c0b3bf0 0x7f9b4c15d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.795+0000 7f9b5180e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c0c1f70 con 0x7f9b4c06d310 2024-01-27T01:08:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.795+0000 7f9b5180e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c0ccda0 con 0x7f9b4c06c290 2024-01-27T01:08:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.795+0000 7f9b5180e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c0cc190 con 0x7f9b4c0b3bf0 2024-01-27T01:08:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.795+0000 7f9b4a7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b4c06d310 0x7f9b4c0cdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.795+0000 7f9b4a7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b4c06d310 0x7f9b4c0cdb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55588/0 (socket says 172.21.15.46:55588) 2024-01-27T01:08:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.795+0000 7f9b4a7fc700 1 -- 172.21.15.46:0/4232960721 learned_addr learned my addr 172.21.15.46:0/4232960721 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.795+0000 7f9b4affd700 1 --2- 172.21.15.46:0/4232960721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b4c06c290 0x7f9b4c151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.795+0000 7f9b4b7fe700 1 --2- 172.21.15.46:0/4232960721 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b4c0b3bf0 0x7f9b4c15d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.796+0000 7f9b4a7fc700 1 -- 172.21.15.46:0/4232960721 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b4c0b3bf0 msgr2=0x7f9b4c15d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:06.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.796+0000 7f9b4a7fc700 1 --2- 172.21.15.46:0/4232960721 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b4c0b3bf0 0x7f9b4c15d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.796+0000 7f9b4a7fc700 1 -- 172.21.15.46:0/4232960721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b4c06c290 msgr2=0x7f9b4c151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:06.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.796+0000 7f9b4a7fc700 1 --2- 172.21.15.46:0/4232960721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b4c06c290 0x7f9b4c151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:06.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.796+0000 7f9b4a7fc700 1 -- 172.21.15.46:0/4232960721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b44009ae0 con 0x7f9b4c06d310 2024-01-27T01:08:06.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.796+0000 7f9b4affd700 1 --2- 172.21.15.46:0/4232960721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b4c06c290 0x7f9b4c151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:06.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.796+0000 7f9b4a7fc700 1 --2- 172.21.15.46:0/4232960721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b4c06d310 0x7f9b4c0cdb60 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f9b4000e930 tx=0x7f9b400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:06.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.796+0000 7f9b32ffd700 1 -- 172.21.15.46:0/4232960721 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b40013680 con 0x7f9b4c06d310 2024-01-27T01:08:06.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.796+0000 7f9b32ffd700 1 -- 172.21.15.46:0/4232960721 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b400137e0 con 0x7f9b4c06d310 2024-01-27T01:08:06.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.796+0000 7f9b5180e700 1 -- 172.21.15.46:0/4232960721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b4c0cc470 con 0x7f9b4c06d310 2024-01-27T01:08:06.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.796+0000 7f9b32ffd700 1 -- 172.21.15.46:0/4232960721 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b40013b80 con 0x7f9b4c06d310 2024-01-27T01:08:06.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.796+0000 7f9b5180e700 1 -- 172.21.15.46:0/4232960721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b4c0cca30 con 0x7f9b4c06d310 2024-01-27T01:08:06.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.797+0000 7f9b5180e700 1 -- 172.21.15.46:0/4232960721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b4c06e2b0 con 0x7f9b4c06d310 2024-01-27T01:08:06.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.798+0000 7f9b32ffd700 1 -- 172.21.15.46:0/4232960721 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f9b4000c460 con 0x7f9b4c06d310 2024-01-27T01:08:06.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.798+0000 7f9b32ffd700 1 --2- 172.21.15.46:0/4232960721 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b380670e0 0x7f9b38069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:06.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.798+0000 7f9b32ffd700 1 -- 172.21.15.46:0/4232960721 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7389+0+0 (secure 0 0 0) 0x7f9b40085040 con 0x7f9b4c06d310 2024-01-27T01:08:06.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.798+0000 7f9b4affd700 1 --2- 172.21.15.46:0/4232960721 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b380670e0 0x7f9b38069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:06.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.799+0000 7f9b4affd700 1 --2- 172.21.15.46:0/4232960721 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b380670e0 0x7f9b38069590 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c007fd0 tx=0x7f9b3c009040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:06.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:06.802+0000 7f9b32ffd700 1 -- 172.21.15.46:0/4232960721 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9b400519a0 con 0x7f9b4c06d310 2024-01-27T01:08:06.840 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:06.840+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.063+0000 7f9b5180e700 1 -- 172.21.15.46:0/4232960721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b4c0ccf10 con 0x7f9b4c06d310 2024-01-27T01:08:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.064+0000 7f9b32ffd700 1 -- 172.21.15.46:0/4232960721 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+4611 (secure 0 0 0) 0x7f9b40054fc0 con 0x7f9b4c06d310 2024-01-27T01:08:07.063 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:07.063 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":140,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1114164873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1114164873},{"type":"v1","addr":"172.21.15.106:6834","nonce":1114164873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128},{"gid":9821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/702094428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":702094428},{"type":"v1","addr":"172.21.15.106:6835","nonce":702094428}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":11770,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4159722763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4159722763},{"type":"v1","addr":"172.21.15.46:6834","nonce":4159722763}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:07:41.411084+0000","modified":"2024-01-27T01:07:45.002753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9827":{"gid":9827,"name":"c","rank":0,"incarnation":135,"state":"up:active","state_seq":6,"addr":"172.21.15.46:6835/408985798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":408985798},{"type":"v1","addr":"172.21.15.46:6835","nonce":408985798}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-27T01:08:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.066+0000 7f9b5180e700 1 -- 172.21.15.46:0/4232960721 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b380670e0 msgr2=0x7f9b38069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.066+0000 7f9b5180e700 1 --2- 172.21.15.46:0/4232960721 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b380670e0 0x7f9b38069590 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c007fd0 tx=0x7f9b3c009040 comp rx=0 tx=0).stop 2024-01-27T01:08:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.066+0000 7f9b5180e700 1 -- 172.21.15.46:0/4232960721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b4c06d310 msgr2=0x7f9b4c0cdb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.066+0000 7f9b5180e700 1 --2- 172.21.15.46:0/4232960721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b4c06d310 0x7f9b4c0cdb60 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f9b4000e930 tx=0x7f9b400091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.066+0000 7f9b5180e700 1 -- 172.21.15.46:0/4232960721 shutdown_connections 2024-01-27T01:08:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.066+0000 7f9b5180e700 1 --2- 172.21.15.46:0/4232960721 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9b380670e0 0x7f9b38069590 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.066+0000 7f9b5180e700 1 --2- 172.21.15.46:0/4232960721 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9b4c0b3bf0 0x7f9b4c15d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.066+0000 7f9b5180e700 1 --2- 172.21.15.46:0/4232960721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b4c06c290 0x7f9b4c151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.066+0000 7f9b5180e700 1 --2- 172.21.15.46:0/4232960721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b4c06d310 0x7f9b4c0cdb60 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.066+0000 7f9b5180e700 1 -- 172.21.15.46:0/4232960721 >> 172.21.15.46:0/4232960721 conn(0x7f9b4c0ce530 msgr2=0x7f9b4c1560b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.066+0000 7f9b5180e700 1 -- 172.21.15.46:0/4232960721 shutdown_connections 2024-01-27T01:08:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.066+0000 7f9b5180e700 1 -- 172.21.15.46:0/4232960721 wait complete. 2024-01-27T01:08:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 140 2024-01-27T01:08:07.079 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:08:07.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.218+0000 7f99e1fa4700 1 Processor -- start 2024-01-27T01:08:07.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.218+0000 7f99e1fa4700 1 -- start start 2024-01-27T01:08:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.219+0000 7f99e1fa4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99dc0ae520 0x7f99dc0b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.219+0000 7f99e1fa4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f99dc0b3fb0 0x7f99dc157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.219+0000 7f99e1fa4700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99dc158ae0 con 0x7f99dc157340 2024-01-27T01:08:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.219+0000 7f99e1fa4700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99dc0b4430 con 0x7f99dc0b3fb0 2024-01-27T01:08:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.219+0000 7f99e1fa4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc15b650 con 0x7f99dc0ae520 2024-01-27T01:08:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.219+0000 7f99dbfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f99dc0b3fb0 0x7f99dc157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.219+0000 7f99dbfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f99dc0b3fb0 0x7f99dc157710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55592/0 (socket says 172.21.15.46:55592) 2024-01-27T01:08:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.219+0000 7f99dbfff700 1 -- 172.21.15.46:0/3202869688 learned_addr learned my addr 172.21.15.46:0/3202869688 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.219+0000 7f99db7fe700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f99dc157340 0x7f99dc15bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:38218/0 (socket says 172.21.15.46:38218) 2024-01-27T01:08:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.219+0000 7f99daffd700 1 --2- 172.21.15.46:0/3202869688 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99dc0ae520 0x7f99dc0b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.220+0000 7f99dbfff700 1 -- 172.21.15.46:0/3202869688 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99dc0ae520 msgr2=0x7f99dc0b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.220+0000 7f99dbfff700 1 --2- 172.21.15.46:0/3202869688 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99dc0ae520 0x7f99dc0b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.220+0000 7f99da7fc700 1 -- 172.21.15.46:0/3202869688 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644505484 0 0) 0x7f99dc158ae0 con 0x7f99dc157340 2024-01-27T01:08:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.220+0000 7f99dbfff700 1 -- 172.21.15.46:0/3202869688 >> v1:172.21.15.106:6789/0 conn(0x7f99dc157340 legacy=0x7f99dc15bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.220+0000 7f99dbfff700 1 -- 172.21.15.46:0/3202869688 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99dc158520 con 0x7f99dc0b3fb0 2024-01-27T01:08:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.220+0000 7f99dbfff700 1 --2- 172.21.15.46:0/3202869688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99dc0b3fb0 0x7f99dc157710 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f99d40099b0 tx=0x7f99d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3b0e903789a90c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.220+0000 7f99da7fc700 1 -- 172.21.15.46:0/3202869688 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d4009ea0 con 0x7f99dc0b3fb0 2024-01-27T01:08:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.220+0000 7f99da7fc700 1 -- 172.21.15.46:0/3202869688 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d400b860 con 0x7f99dc0b3fb0 2024-01-27T01:08:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.220+0000 7f99da7fc700 1 -- 172.21.15.46:0/3202869688 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d400bc20 con 0x7f99dc0b3fb0 2024-01-27T01:08:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.221+0000 7f99e1fa4700 1 -- 172.21.15.46:0/3202869688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99dc0b3fb0 msgr2=0x7f99dc157710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.221+0000 7f99e1fa4700 1 --2- 172.21.15.46:0/3202869688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99dc0b3fb0 0x7f99dc157710 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f99d40099b0 tx=0x7f99d4009250 comp rx=0 tx=0).stop 2024-01-27T01:08:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.221+0000 7f99e1fa4700 1 -- 172.21.15.46:0/3202869688 shutdown_connections 2024-01-27T01:08:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.221+0000 7f99e1fa4700 1 --2- 172.21.15.46:0/3202869688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99dc0b3fb0 0x7f99dc157710 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.221+0000 7f99e1fa4700 1 --2- 172.21.15.46:0/3202869688 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99dc0ae520 0x7f99dc0b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.221+0000 7f99e1fa4700 1 -- 172.21.15.46:0/3202869688 >> 172.21.15.46:0/3202869688 conn(0x7f99dc0ce530 msgr2=0x7f99dc0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:07.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.221+0000 7f99e1fa4700 1 -- 172.21.15.46:0/3202869688 shutdown_connections 2024-01-27T01:08:07.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.221+0000 7f99e1fa4700 1 -- 172.21.15.46:0/3202869688 wait complete. 2024-01-27T01:08:07.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.222+0000 7f99e1fa4700 1 Processor -- start 2024-01-27T01:08:07.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.222+0000 7f99e1fa4700 1 -- start start 2024-01-27T01:08:07.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.223+0000 7f99e1fa4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99dc0ae520 0x7f99dc0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:07.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.223+0000 7f99e1fa4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99dc0b3fb0 0x7f99dc152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:07.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.223+0000 7f99e1fa4700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99dc157340 0x7f99dc152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.223+0000 7f99e1fa4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc153d60 con 0x7f99dc0b3fb0 2024-01-27T01:08:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.223+0000 7f99e1fa4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc069010 con 0x7f99dc0ae520 2024-01-27T01:08:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.223+0000 7f99e1fa4700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc1534c0 con 0x7f99dc157340 2024-01-27T01:08:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.223+0000 7f99daffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99dc0b3fb0 0x7f99dc152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.223+0000 7f99db7fe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99dc0ae520 0x7f99dc0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.223+0000 7f99daffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99dc0b3fb0 0x7f99dc152970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55600/0 (socket says 172.21.15.46:55600) 2024-01-27T01:08:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.223+0000 7f99daffd700 1 -- 172.21.15.46:0/463568102 learned_addr learned my addr 172.21.15.46:0/463568102 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.223+0000 7f99dbfff700 1 --2- 172.21.15.46:0/463568102 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99dc157340 0x7f99dc152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.223+0000 7f99db7fe700 1 -- 172.21.15.46:0/463568102 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99dc157340 msgr2=0x7f99dc152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.223+0000 7f99db7fe700 1 --2- 172.21.15.46:0/463568102 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99dc157340 0x7f99dc152eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.223+0000 7f99db7fe700 1 -- 172.21.15.46:0/463568102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99dc0b3fb0 msgr2=0x7f99dc152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.223+0000 7f99db7fe700 1 --2- 172.21.15.46:0/463568102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99dc0b3fb0 0x7f99dc152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.224+0000 7f99db7fe700 1 -- 172.21.15.46:0/463568102 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d40094f0 con 0x7f99dc0ae520 2024-01-27T01:08:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.224+0000 7f99db7fe700 1 --2- 172.21.15.46:0/463568102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99dc0ae520 0x7f99dc0ccbf0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00cd50 tx=0x7f99cc001b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:07.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.224+0000 7f99c37fe700 1 -- 172.21.15.46:0/463568102 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc005df0 con 0x7f99dc0ae520 2024-01-27T01:08:07.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.224+0000 7f99c37fe700 1 -- 172.21.15.46:0/463568102 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99cc00a870 con 0x7f99dc0ae520 2024-01-27T01:08:07.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.224+0000 7f99c37fe700 1 -- 172.21.15.46:0/463568102 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc00ac30 con 0x7f99dc0ae520 2024-01-27T01:08:07.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.224+0000 7f99e1fa4700 1 -- 172.21.15.46:0/463568102 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99dc1537a0 con 0x7f99dc0ae520 2024-01-27T01:08:07.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.224+0000 7f99e1fa4700 1 -- 172.21.15.46:0/463568102 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99dc0cd130 con 0x7f99dc0ae520 2024-01-27T01:08:07.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.225+0000 7f99e1fa4700 1 -- 172.21.15.46:0/463568102 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99dc04c710 con 0x7f99dc0ae520 2024-01-27T01:08:07.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.225+0000 7f99c37fe700 1 -- 172.21.15.46:0/463568102 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f99cc0070f0 con 0x7f99dc0ae520 2024-01-27T01:08:07.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.226+0000 7f99c37fe700 1 --2- 172.21.15.46:0/463568102 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f99c80670e0 0x7f99c8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:07.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.226+0000 7f99c37fe700 1 -- 172.21.15.46:0/463568102 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7389+0+0 (secure 0 0 0) 0x7f99cc084f40 con 0x7f99dc0ae520 2024-01-27T01:08:07.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.226+0000 7f99daffd700 1 --2- 172.21.15.46:0/463568102 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f99c80670e0 0x7f99c8069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:07.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.226+0000 7f99daffd700 1 --2- 172.21.15.46:0/463568102 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f99c80670e0 0x7f99c8069590 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f99dc16ca10 tx=0x7f99d00091a0 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:07.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.231+0000 7f99c37fe700 1 -- 172.21.15.46:0/463568102 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f99cc0518a0 con 0x7f99dc0ae520 2024-01-27T01:08:07.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.457+0000 7f99e1fa4700 1 -- 172.21.15.46:0/463568102 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99dc158800 con 0x7f99dc0ae520 2024-01-27T01:08:07.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.458+0000 7f99c37fe700 1 -- 172.21.15.46:0/463568102 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) v1 ==== 74+0+17594 (secure 0 0 0) 0x7f99cc054ec0 con 0x7f99dc0ae520 2024-01-27T01:08:07.458 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:07.458 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":252,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:08:05.095424+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:07:39.209497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:07:40.386606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6834/1087205220":"2024-01-28T01:08:01.670110+0000","172.21.15.106:0/558414078":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/2701940868":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/3540935782":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/3966339865":"2024-01-28T01:08:01.068164+0000","172.21.15.46:6832/1087205220":"2024-01-28T01:08:01.670110+0000","172.21.15.106:0/3958137863":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/2317042118":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/1816027959":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/2241845238":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1948052707":"2024-01-28T01:07:53.996892+0000","172.21.15.106:6836/62434":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1918093439":"2024-01-28T01:07:53.996892+0000","172.21.15.106:6837/62434":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/469270357":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1388668891":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1199239718":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/3351808187":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/2250157080":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/2626066195":"2024-01-28T01:07:53.996892+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:08:07.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.461+0000 7f99e1fa4700 1 -- 172.21.15.46:0/463568102 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f99c80670e0 msgr2=0x7f99c8069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:07.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.461+0000 7f99e1fa4700 1 --2- 172.21.15.46:0/463568102 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f99c80670e0 0x7f99c8069590 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f99dc16ca10 tx=0x7f99d00091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.461+0000 7f99e1fa4700 1 -- 172.21.15.46:0/463568102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99dc0ae520 msgr2=0x7f99dc0ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:07.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.461+0000 7f99e1fa4700 1 --2- 172.21.15.46:0/463568102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99dc0ae520 0x7f99dc0ccbf0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00cd50 tx=0x7f99cc001b00 comp rx=0 tx=0).stop 2024-01-27T01:08:07.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.461+0000 7f99e1fa4700 1 -- 172.21.15.46:0/463568102 shutdown_connections 2024-01-27T01:08:07.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.461+0000 7f99e1fa4700 1 --2- 172.21.15.46:0/463568102 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f99c80670e0 0x7f99c8069590 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.461+0000 7f99e1fa4700 1 --2- 172.21.15.46:0/463568102 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f99dc157340 0x7f99dc152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.461+0000 7f99e1fa4700 1 --2- 172.21.15.46:0/463568102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99dc0ae520 0x7f99dc0ccbf0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.461+0000 7f99e1fa4700 1 --2- 172.21.15.46:0/463568102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99dc0b3fb0 0x7f99dc152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.461+0000 7f99e1fa4700 1 -- 172.21.15.46:0/463568102 >> 172.21.15.46:0/463568102 conn(0x7f99dc0ce530 msgr2=0x7f99dc159a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:07.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.461+0000 7f99e1fa4700 1 -- 172.21.15.46:0/463568102 shutdown_connections 2024-01-27T01:08:07.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.461+0000 7f99e1fa4700 1 -- 172.21.15.46:0/463568102 wait complete. 2024-01-27T01:08:07.474 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-01-27T01:08:07.474 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:08:07.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.608+0000 7f2c33e5e700 1 Processor -- start 2024-01-27T01:08:07.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.608+0000 7f2c33e5e700 1 -- start start 2024-01-27T01:08:07.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.609+0000 7f2c33e5e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c2c0cc1c0 0x7f2c2c164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:07.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.609+0000 7f2c33e5e700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2c0cff70 con 0x7f2c2c15d210 2024-01-27T01:08:07.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.609+0000 7f2c33e5e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2c0cf400 con 0x7f2c2c15ce40 2024-01-27T01:08:07.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.609+0000 7f2c33e5e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c0cf660 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.609+0000 7f2c313f9700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2c2c15d210 0x7f2c2c0cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57224/0 (socket says 172.21.15.46:57224) 2024-01-27T01:08:07.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.609+0000 7f2c313f9700 1 -- 172.21.15.46:0/244697728 learned_addr learned my addr 172.21.15.46:0/244697728 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:07.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.609+0000 7f2c323fb700 1 --2- 172.21.15.46:0/244697728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c2c0cc1c0 0x7f2c2c164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:07.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.610+0000 7f2c30bf8700 1 -- 172.21.15.46:0/244697728 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427337382 0 0) 0x7f2c2c0cff70 con 0x7f2c2c15d210 2024-01-27T01:08:07.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.610+0000 7f2c30bf8700 1 -- 172.21.15.46:0/244697728 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c18003570 con 0x7f2c2c15d210 2024-01-27T01:08:07.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.610+0000 7f2c30bf8700 1 -- 172.21.15.46:0/244697728 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658986874 0 0) 0x7f2c2c0cf400 con 0x7f2c2c15ce40 2024-01-27T01:08:07.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.610+0000 7f2c30bf8700 1 -- 172.21.15.46:0/244697728 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c2c0cff70 con 0x7f2c2c15ce40 2024-01-27T01:08:07.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.610+0000 7f2c323fb700 1 -- 172.21.15.46:0/244697728 >> v1:172.21.15.106:6789/0 conn(0x7f2c2c15ce40 legacy=0x7f2c2c0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:07.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.610+0000 7f2c323fb700 1 -- 172.21.15.46:0/244697728 >> v1:172.21.15.46:6789/0 conn(0x7f2c2c15d210 legacy=0x7f2c2c0cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.610+0000 7f2c323fb700 1 -- 172.21.15.46:0/244697728 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2c0d0500 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.610+0000 7f2c323fb700 1 --2- 172.21.15.46:0/244697728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c2c0cc1c0 0x7f2c2c164700 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2c24009e90 tx=0x7f2c24009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddd9630d5e89fe74 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.610+0000 7f2c30bf8700 1 -- 172.21.15.46:0/244697728 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2400e070 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.610+0000 7f2c30bf8700 1 -- 172.21.15.46:0/244697728 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2400bb80 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.610+0000 7f2c30bf8700 1 -- 172.21.15.46:0/244697728 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c24012430 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.611+0000 7f2c33e5e700 1 -- 172.21.15.46:0/244697728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c2c0cc1c0 msgr2=0x7f2c2c164700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.611+0000 7f2c33e5e700 1 --2- 172.21.15.46:0/244697728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c2c0cc1c0 0x7f2c2c164700 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2c24009e90 tx=0x7f2c24009780 comp rx=0 tx=0).stop 2024-01-27T01:08:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.611+0000 7f2c33e5e700 1 -- 172.21.15.46:0/244697728 shutdown_connections 2024-01-27T01:08:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.611+0000 7f2c33e5e700 1 --2- 172.21.15.46:0/244697728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c2c0cc1c0 0x7f2c2c164700 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.611+0000 7f2c33e5e700 1 -- 172.21.15.46:0/244697728 >> 172.21.15.46:0/244697728 conn(0x7f2c2c06f840 msgr2=0x7f2c2c0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.611+0000 7f2c33e5e700 1 -- 172.21.15.46:0/244697728 shutdown_connections 2024-01-27T01:08:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.611+0000 7f2c33e5e700 1 -- 172.21.15.46:0/244697728 wait complete. 2024-01-27T01:08:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.612+0000 7f2c33e5e700 1 Processor -- start 2024-01-27T01:08:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.612+0000 7f2c33e5e700 1 -- start start 2024-01-27T01:08:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.613+0000 7f2c33e5e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2c0cc1c0 0x7f2c2c15e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.613+0000 7f2c33e5e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2c2c15ce40 0x7f2c2c15e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.613+0000 7f2c33e5e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c2c15d210 0x7f2c2c0c5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.613+0000 7f2c33e5e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c155840 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.613+0000 7f2c33e5e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c1560e0 con 0x7f2c2c15ce40 2024-01-27T01:08:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.613+0000 7f2c33e5e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c15ee40 con 0x7f2c2c15d210 2024-01-27T01:08:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.613+0000 7f2c31bfa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2c0cc1c0 0x7f2c2c15e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.613+0000 7f2c31bfa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2c0cc1c0 0x7f2c2c15e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55610/0 (socket says 172.21.15.46:55610) 2024-01-27T01:08:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.613+0000 7f2c31bfa700 1 -- 172.21.15.46:0/753747412 learned_addr learned my addr 172.21.15.46:0/753747412 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.613+0000 7f2c323fb700 1 --2- 172.21.15.46:0/753747412 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c2c15d210 0x7f2c2c0c5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.613+0000 7f2c313f9700 1 --2- 172.21.15.46:0/753747412 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2c2c15ce40 0x7f2c2c15e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.614+0000 7f2c31bfa700 1 -- 172.21.15.46:0/753747412 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c2c15d210 msgr2=0x7f2c2c0c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.614+0000 7f2c31bfa700 1 --2- 172.21.15.46:0/753747412 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c2c15d210 0x7f2c2c0c5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.614+0000 7f2c31bfa700 1 -- 172.21.15.46:0/753747412 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2c2c15ce40 msgr2=0x7f2c2c15e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.614+0000 7f2c31bfa700 1 --2- 172.21.15.46:0/753747412 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2c2c15ce40 0x7f2c2c15e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.614+0000 7f2c31bfa700 1 -- 172.21.15.46:0/753747412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c24009a20 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.614+0000 7f2c323fb700 1 --2- 172.21.15.46:0/753747412 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c2c15d210 0x7f2c2c0c5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.614+0000 7f2c313f9700 1 --2- 172.21.15.46:0/753747412 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2c2c15ce40 0x7f2c2c15e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.614+0000 7f2c31bfa700 1 --2- 172.21.15.46:0/753747412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2c0cc1c0 0x7f2c2c15e0a0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f2c2800c7c0 tx=0x7f2c28007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.614+0000 7f2c21ffb700 1 -- 172.21.15.46:0/753747412 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c28001570 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.614+0000 7f2c21ffb700 1 -- 172.21.15.46:0/753747412 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c280016f0 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.614+0000 7f2c33e5e700 1 -- 172.21.15.46:0/753747412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2c155a90 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.614+0000 7f2c21ffb700 1 -- 172.21.15.46:0/753747412 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c28001ab0 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.614+0000 7f2c33e5e700 1 -- 172.21.15.46:0/753747412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2c15eb20 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.616+0000 7f2c33e5e700 1 -- 172.21.15.46:0/753747412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c2c0cd760 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.616+0000 7f2c21ffb700 1 -- 172.21.15.46:0/753747412 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f2c2801b3b0 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.616+0000 7f2c21ffb700 1 --2- 172.21.15.46:0/753747412 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2c100670e0 0x7f2c10069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:07.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.616+0000 7f2c21ffb700 1 -- 172.21.15.46:0/753747412 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7389+0+0 (secure 0 0 0) 0x7f2c280846d0 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.616+0000 7f2c313f9700 1 --2- 172.21.15.46:0/753747412 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2c100670e0 0x7f2c10069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:07.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.617+0000 7f2c313f9700 1 --2- 172.21.15.46:0/753747412 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2c100670e0 0x7f2c10069590 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c161600 tx=0x7f2c1c009040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:07.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.621+0000 7f2c21ffb700 1 -- 172.21.15.46:0/753747412 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2c28051030 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.784 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:07.784+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:07.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.882+0000 7f2c33e5e700 1 -- 172.21.15.46:0/753747412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c2c04c710 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.883+0000 7f2c21ffb700 1 -- 172.21.15.46:0/753747412 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+4611 (secure 0 0 0) 0x7f2c28054650 con 0x7f2c2c0cc1c0 2024-01-27T01:08:07.882 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:07.882 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":140,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1114164873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1114164873},{"type":"v1","addr":"172.21.15.106:6834","nonce":1114164873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128},{"gid":9821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/702094428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":702094428},{"type":"v1","addr":"172.21.15.106:6835","nonce":702094428}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":11770,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4159722763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4159722763},{"type":"v1","addr":"172.21.15.46:6834","nonce":4159722763}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:07:41.411084+0000","modified":"2024-01-27T01:07:45.002753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9827":{"gid":9827,"name":"c","rank":0,"incarnation":135,"state":"up:active","state_seq":6,"addr":"172.21.15.46:6835/408985798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":408985798},{"type":"v1","addr":"172.21.15.46:6835","nonce":408985798}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-27T01:08:07.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.885+0000 7f2c33e5e700 1 -- 172.21.15.46:0/753747412 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2c100670e0 msgr2=0x7f2c10069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:07.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.885+0000 7f2c33e5e700 1 --2- 172.21.15.46:0/753747412 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2c100670e0 0x7f2c10069590 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c161600 tx=0x7f2c1c009040 comp rx=0 tx=0).stop 2024-01-27T01:08:07.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.885+0000 7f2c33e5e700 1 -- 172.21.15.46:0/753747412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2c0cc1c0 msgr2=0x7f2c2c15e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:07.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.885+0000 7f2c33e5e700 1 --2- 172.21.15.46:0/753747412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2c0cc1c0 0x7f2c2c15e0a0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f2c2800c7c0 tx=0x7f2c28007dc0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.885+0000 7f2c33e5e700 1 -- 172.21.15.46:0/753747412 shutdown_connections 2024-01-27T01:08:07.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.885+0000 7f2c33e5e700 1 --2- 172.21.15.46:0/753747412 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2c100670e0 0x7f2c10069590 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.885+0000 7f2c33e5e700 1 --2- 172.21.15.46:0/753747412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2c0cc1c0 0x7f2c2c15e0a0 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.885+0000 7f2c33e5e700 1 --2- 172.21.15.46:0/753747412 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2c2c15d210 0x7f2c2c0c5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.885+0000 7f2c33e5e700 1 --2- 172.21.15.46:0/753747412 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2c2c15ce40 0x7f2c2c15e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:07.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.885+0000 7f2c33e5e700 1 -- 172.21.15.46:0/753747412 >> 172.21.15.46:0/753747412 conn(0x7f2c2c06f840 msgr2=0x7f2c2c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:07.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.885+0000 7f2c33e5e700 1 -- 172.21.15.46:0/753747412 shutdown_connections 2024-01-27T01:08:07.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:07.885+0000 7f2c33e5e700 1 -- 172.21.15.46:0/753747412 wait complete. 2024-01-27T01:08:07.886 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 140 2024-01-27T01:08:07.899 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:08:08.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.035+0000 7f287569b700 1 Processor -- start 2024-01-27T01:08:08.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.035+0000 7f287569b700 1 -- start start 2024-01-27T01:08:08.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.036+0000 7f287569b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f28700c6520 0x7f28700ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:08.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.036+0000 7f287569b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f287015a0b0 con 0x7f28700ced50 2024-01-27T01:08:08.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.036+0000 7f287569b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28700ae4d0 con 0x7f28700b2b40 2024-01-27T01:08:08.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.036+0000 7f287569b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28700af070 con 0x7f28700c6520 2024-01-27T01:08:08.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.036+0000 7f286f7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f28700ced50 0x7f28700cb5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57240/0 (socket says 172.21.15.46:57240) 2024-01-27T01:08:08.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.036+0000 7f286f7fe700 1 -- 172.21.15.46:0/4258493791 learned_addr learned my addr 172.21.15.46:0/4258493791 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:08.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.036+0000 7f286e7fc700 1 --2- 172.21.15.46:0/4258493791 >> v2:172.21.15.106:3300/0 conn(0x7f28700c6520 0x7f28700ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:08.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.036+0000 7f286dffb700 1 -- 172.21.15.46:0/4258493791 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461981419 0 0) 0x7f287015a0b0 con 0x7f28700ced50 2024-01-27T01:08:08.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.036+0000 7f286dffb700 1 -- 172.21.15.46:0/4258493791 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2858003570 con 0x7f28700ced50 2024-01-27T01:08:08.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.036+0000 7f286e7fc700 1 -- 172.21.15.46:0/4258493791 >> v1:172.21.15.106:6789/0 conn(0x7f28700b2b40 legacy=0x7f28700ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.036+0000 7f286dffb700 1 -- 172.21.15.46:0/4258493791 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713424217 0 0) 0x7f28700ae4d0 con 0x7f28700b2b40 2024-01-27T01:08:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.036+0000 7f286e7fc700 1 -- 172.21.15.46:0/4258493791 >> v1:172.21.15.46:6789/0 conn(0x7f28700ced50 legacy=0x7f28700cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.036+0000 7f286e7fc700 1 -- 172.21.15.46:0/4258493791 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28700ad8c0 con 0x7f28700c6520 2024-01-27T01:08:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.037+0000 7f286e7fc700 1 --2- 172.21.15.46:0/4258493791 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28700c6520 0x7f28700ad1c0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f28640098e0 tx=0x7f28640091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d548eba9f7af8868 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.037+0000 7f286dffb700 1 -- 172.21.15.46:0/4258493791 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286400b640 con 0x7f28700c6520 2024-01-27T01:08:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.037+0000 7f286dffb700 1 -- 172.21.15.46:0/4258493791 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286400b7a0 con 0x7f28700c6520 2024-01-27T01:08:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.037+0000 7f286dffb700 1 -- 172.21.15.46:0/4258493791 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286400bb40 con 0x7f28700c6520 2024-01-27T01:08:08.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.037+0000 7f287569b700 1 -- 172.21.15.46:0/4258493791 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28700c6520 msgr2=0x7f28700ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.037+0000 7f287569b700 1 --2- 172.21.15.46:0/4258493791 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28700c6520 0x7f28700ad1c0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f28640098e0 tx=0x7f28640091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.037+0000 7f287569b700 1 -- 172.21.15.46:0/4258493791 shutdown_connections 2024-01-27T01:08:08.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.037+0000 7f287569b700 1 --2- 172.21.15.46:0/4258493791 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28700c6520 0x7f28700ad1c0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.037+0000 7f287569b700 1 -- 172.21.15.46:0/4258493791 >> 172.21.15.46:0/4258493791 conn(0x7f2870099a30 msgr2=0x7f287006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:08.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.037+0000 7f287569b700 1 -- 172.21.15.46:0/4258493791 shutdown_connections 2024-01-27T01:08:08.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.037+0000 7f287569b700 1 -- 172.21.15.46:0/4258493791 wait complete. 2024-01-27T01:08:08.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.039+0000 7f287569b700 1 Processor -- start 2024-01-27T01:08:08.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.039+0000 7f287569b700 1 -- start start 2024-01-27T01:08:08.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f287569b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28700b2b40 0x7f2870156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:08.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f287569b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28700c6520 0x7f28700b5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:08.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f287569b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f28700ced50 0x7f28700b5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:08.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f287569b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28700c8740 con 0x7f28700c6520 2024-01-27T01:08:08.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f287569b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28700c8990 con 0x7f28700b2b40 2024-01-27T01:08:08.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f287569b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28700c8c70 con 0x7f28700ced50 2024-01-27T01:08:08.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f286e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28700c6520 0x7f28700b5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:08.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f286e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28700c6520 0x7f28700b5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55624/0 (socket says 172.21.15.46:55624) 2024-01-27T01:08:08.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f286e7fc700 1 -- 172.21.15.46:0/1460339690 learned_addr learned my addr 172.21.15.46:0/1460339690 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f286effd700 1 --2- 172.21.15.46:0/1460339690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28700b2b40 0x7f2870156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f286f7fe700 1 --2- 172.21.15.46:0/1460339690 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f28700ced50 0x7f28700b5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f286e7fc700 1 -- 172.21.15.46:0/1460339690 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f28700ced50 msgr2=0x7f28700b5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f286e7fc700 1 --2- 172.21.15.46:0/1460339690 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f28700ced50 0x7f28700b5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f286e7fc700 1 -- 172.21.15.46:0/1460339690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28700b2b40 msgr2=0x7f2870156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f286e7fc700 1 --2- 172.21.15.46:0/1460339690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28700b2b40 0x7f2870156240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f286e7fc700 1 -- 172.21.15.46:0/1460339690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2864009420 con 0x7f28700c6520 2024-01-27T01:08:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f286effd700 1 --2- 172.21.15.46:0/1460339690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28700b2b40 0x7f2870156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.040+0000 7f286e7fc700 1 --2- 172.21.15.46:0/1460339690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28700c6520 0x7f28700b5880 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f2870088e10 tx=0x7f2864017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.041+0000 7f2856ffd700 1 -- 172.21.15.46:0/1460339690 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286401f070 con 0x7f28700c6520 2024-01-27T01:08:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.041+0000 7f2856ffd700 1 -- 172.21.15.46:0/1460339690 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2864002f70 con 0x7f28700c6520 2024-01-27T01:08:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.041+0000 7f287569b700 1 -- 172.21.15.46:0/1460339690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28700c8f50 con 0x7f28700c6520 2024-01-27T01:08:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.041+0000 7f2856ffd700 1 -- 172.21.15.46:0/1460339690 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286400b910 con 0x7f28700c6520 2024-01-27T01:08:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.041+0000 7f287569b700 1 -- 172.21.15.46:0/1460339690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28700b4200 con 0x7f28700c6520 2024-01-27T01:08:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.042+0000 7f287569b700 1 -- 172.21.15.46:0/1460339690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f287004c710 con 0x7f28700c6520 2024-01-27T01:08:08.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.042+0000 7f2856ffd700 1 -- 172.21.15.46:0/1460339690 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f2864023070 con 0x7f28700c6520 2024-01-27T01:08:08.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.043+0000 7f2856ffd700 1 --2- 172.21.15.46:0/1460339690 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f285c0670e0 0x7f285c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:08.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.043+0000 7f2856ffd700 1 -- 172.21.15.46:0/1460339690 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7389+0+0 (secure 0 0 0) 0x7f2864086210 con 0x7f28700c6520 2024-01-27T01:08:08.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.043+0000 7f286effd700 1 --2- 172.21.15.46:0/1460339690 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f285c0670e0 0x7f285c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:08.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.043+0000 7f286effd700 1 --2- 172.21.15.46:0/1460339690 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f285c0670e0 0x7f285c069590 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2860007fd0 tx=0x7f2860009040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:08.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.047+0000 7f2856ffd700 1 -- 172.21.15.46:0/1460339690 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2864052b50 con 0x7f28700c6520 2024-01-27T01:08:08.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.328+0000 7f287569b700 1 -- 172.21.15.46:0/1460339690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28700cac20 con 0x7f28700c6520 2024-01-27T01:08:08.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.329+0000 7f2856ffd700 1 -- 172.21.15.46:0/1460339690 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+4611 (secure 0 0 0) 0x7f2864056170 con 0x7f28700c6520 2024-01-27T01:08:08.328 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:08.328 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":140,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1114164873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1114164873},{"type":"v1","addr":"172.21.15.106:6834","nonce":1114164873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128},{"gid":9821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/702094428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":702094428},{"type":"v1","addr":"172.21.15.106:6835","nonce":702094428}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":11770,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4159722763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4159722763},{"type":"v1","addr":"172.21.15.46:6834","nonce":4159722763}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:07:41.411084+0000","modified":"2024-01-27T01:07:45.002753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9827":{"gid":9827,"name":"c","rank":0,"incarnation":135,"state":"up:active","state_seq":6,"addr":"172.21.15.46:6835/408985798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":408985798},{"type":"v1","addr":"172.21.15.46:6835","nonce":408985798}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-27T01:08:08.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.331+0000 7f287569b700 1 -- 172.21.15.46:0/1460339690 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f285c0670e0 msgr2=0x7f285c069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.331+0000 7f287569b700 1 --2- 172.21.15.46:0/1460339690 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f285c0670e0 0x7f285c069590 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2860007fd0 tx=0x7f2860009040 comp rx=0 tx=0).stop 2024-01-27T01:08:08.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.331+0000 7f287569b700 1 -- 172.21.15.46:0/1460339690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28700c6520 msgr2=0x7f28700b5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.331+0000 7f287569b700 1 --2- 172.21.15.46:0/1460339690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28700c6520 0x7f28700b5880 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f2870088e10 tx=0x7f2864017000 comp rx=0 tx=0).stop 2024-01-27T01:08:08.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.331+0000 7f287569b700 1 -- 172.21.15.46:0/1460339690 shutdown_connections 2024-01-27T01:08:08.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.331+0000 7f287569b700 1 --2- 172.21.15.46:0/1460339690 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f285c0670e0 0x7f285c069590 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.331+0000 7f287569b700 1 --2- 172.21.15.46:0/1460339690 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f28700ced50 0x7f28700b5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.331+0000 7f287569b700 1 --2- 172.21.15.46:0/1460339690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28700b2b40 0x7f2870156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.331+0000 7f287569b700 1 --2- 172.21.15.46:0/1460339690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28700c6520 0x7f28700b5880 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.332+0000 7f287569b700 1 -- 172.21.15.46:0/1460339690 >> 172.21.15.46:0/1460339690 conn(0x7f2870099a30 msgr2=0x7f287006a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:08.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.332+0000 7f287569b700 1 -- 172.21.15.46:0/1460339690 shutdown_connections 2024-01-27T01:08:08.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.332+0000 7f287569b700 1 -- 172.21.15.46:0/1460339690 wait complete. 2024-01-27T01:08:08.332 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 140 2024-01-27T01:08:08.344 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:08:08.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.480+0000 7fab1dc56700 1 Processor -- start 2024-01-27T01:08:08.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.480+0000 7fab1dc56700 1 -- start start 2024-01-27T01:08:08.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.481+0000 7fab1dc56700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab1815d210 0x7fab180cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:08.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.481+0000 7fab1dc56700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab180cff70 con 0x7fab180cbff0 2024-01-27T01:08:08.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.481+0000 7fab1dc56700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab180cf400 con 0x7fab1815ce40 2024-01-27T01:08:08.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.481+0000 7fab1dc56700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab180cc490 con 0x7fab1815d210 2024-01-27T01:08:08.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.481+0000 7fab17fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fab180cbff0 0x7fab180ade30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57252/0 (socket says 172.21.15.46:57252) 2024-01-27T01:08:08.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.481+0000 7fab17fff700 1 -- 172.21.15.46:0/3799990345 learned_addr learned my addr 172.21.15.46:0/3799990345 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:08.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.481+0000 7fab16ffd700 1 --2- 172.21.15.46:0/3799990345 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab1815d210 0x7fab180cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:08.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.482+0000 7fab167fc700 1 -- 172.21.15.46:0/3799990345 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816715551 0 0) 0x7fab180cff70 con 0x7fab180cbff0 2024-01-27T01:08:08.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.482+0000 7fab167fc700 1 -- 172.21.15.46:0/3799990345 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab00003570 con 0x7fab180cbff0 2024-01-27T01:08:08.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.482+0000 7fab167fc700 1 -- 172.21.15.46:0/3799990345 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065819862 0 0) 0x7fab180cf400 con 0x7fab1815ce40 2024-01-27T01:08:08.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.482+0000 7fab167fc700 1 -- 172.21.15.46:0/3799990345 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab180cff70 con 0x7fab1815ce40 2024-01-27T01:08:08.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.482+0000 7fab16ffd700 1 -- 172.21.15.46:0/3799990345 >> v1:172.21.15.106:6789/0 conn(0x7fab1815ce40 legacy=0x7fab180d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.482+0000 7fab16ffd700 1 -- 172.21.15.46:0/3799990345 >> v1:172.21.15.46:6789/0 conn(0x7fab180cbff0 legacy=0x7fab180ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.482+0000 7fab16ffd700 1 -- 172.21.15.46:0/3799990345 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab180cf660 con 0x7fab1815d210 2024-01-27T01:08:08.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.482+0000 7fab16ffd700 1 --2- 172.21.15.46:0/3799990345 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab1815d210 0x7fab180cbab0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fab0c009910 tx=0x7fab0c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37e02e0fde603d3d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:08.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.482+0000 7fab167fc700 1 -- 172.21.15.46:0/3799990345 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0c011630 con 0x7fab1815d210 2024-01-27T01:08:08.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.482+0000 7fab167fc700 1 -- 172.21.15.46:0/3799990345 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab0c011790 con 0x7fab1815d210 2024-01-27T01:08:08.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.482+0000 7fab167fc700 1 -- 172.21.15.46:0/3799990345 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0c011b10 con 0x7fab1815d210 2024-01-27T01:08:08.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.483+0000 7fab1dc56700 1 -- 172.21.15.46:0/3799990345 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab1815d210 msgr2=0x7fab180cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.483+0000 7fab1dc56700 1 --2- 172.21.15.46:0/3799990345 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab1815d210 0x7fab180cbab0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fab0c009910 tx=0x7fab0c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.483+0000 7fab1dc56700 1 -- 172.21.15.46:0/3799990345 shutdown_connections 2024-01-27T01:08:08.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.483+0000 7fab1dc56700 1 --2- 172.21.15.46:0/3799990345 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab1815d210 0x7fab180cbab0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.483+0000 7fab1dc56700 1 -- 172.21.15.46:0/3799990345 >> 172.21.15.46:0/3799990345 conn(0x7fab1806f840 msgr2=0x7fab180d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:08.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.483+0000 7fab1dc56700 1 -- 172.21.15.46:0/3799990345 shutdown_connections 2024-01-27T01:08:08.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.483+0000 7fab1dc56700 1 -- 172.21.15.46:0/3799990345 wait complete. 2024-01-27T01:08:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.485+0000 7fab1dc56700 1 Processor -- start 2024-01-27T01:08:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.485+0000 7fab1dc56700 1 -- start start 2024-01-27T01:08:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab1dc56700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab180cbff0 0x7fab18158d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab1dc56700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab1815ce40 0x7fab1816d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab1dc56700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab1815d210 0x7fab1816dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab1dc56700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab18159280 con 0x7fab180cbff0 2024-01-27T01:08:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab1dc56700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1816b3c0 con 0x7fab1815d210 2024-01-27T01:08:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab1dc56700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab180c5910 con 0x7fab1815ce40 2024-01-27T01:08:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab177fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab180cbff0 0x7fab18158d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab177fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab180cbff0 0x7fab18158d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55636/0 (socket says 172.21.15.46:55636) 2024-01-27T01:08:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab177fe700 1 -- 172.21.15.46:0/940830933 learned_addr learned my addr 172.21.15.46:0/940830933 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab17fff700 1 --2- 172.21.15.46:0/940830933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab1815d210 0x7fab1816dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab16ffd700 1 --2- 172.21.15.46:0/940830933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab1815ce40 0x7fab1816d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab177fe700 1 -- 172.21.15.46:0/940830933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab1815ce40 msgr2=0x7fab1816d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab177fe700 1 --2- 172.21.15.46:0/940830933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab1815ce40 0x7fab1816d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab177fe700 1 -- 172.21.15.46:0/940830933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab1815d210 msgr2=0x7fab1816dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab177fe700 1 --2- 172.21.15.46:0/940830933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab1815d210 0x7fab1816dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab177fe700 1 -- 172.21.15.46:0/940830933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0c009420 con 0x7fab180cbff0 2024-01-27T01:08:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab17fff700 1 --2- 172.21.15.46:0/940830933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab1815d210 0x7fab1816dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.486+0000 7fab177fe700 1 --2- 172.21.15.46:0/940830933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab180cbff0 0x7fab18158d40 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fab0800c7c0 tx=0x7fab08007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:08.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.487+0000 7faaff7fe700 1 -- 172.21.15.46:0/940830933 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab08001530 con 0x7fab180cbff0 2024-01-27T01:08:08.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.487+0000 7faaff7fe700 1 -- 172.21.15.46:0/940830933 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab08001690 con 0x7fab180cbff0 2024-01-27T01:08:08.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.487+0000 7fab1dc56700 1 -- 172.21.15.46:0/940830933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1816b610 con 0x7fab180cbff0 2024-01-27T01:08:08.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.487+0000 7faaff7fe700 1 -- 172.21.15.46:0/940830933 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab08001a50 con 0x7fab180cbff0 2024-01-27T01:08:08.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.487+0000 7fab1dc56700 1 -- 172.21.15.46:0/940830933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1816bbd0 con 0x7fab180cbff0 2024-01-27T01:08:08.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.488+0000 7fab1dc56700 1 -- 172.21.15.46:0/940830933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab1804c710 con 0x7fab180cbff0 2024-01-27T01:08:08.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.488+0000 7faaff7fe700 1 -- 172.21.15.46:0/940830933 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7fab0801b400 con 0x7fab180cbff0 2024-01-27T01:08:08.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.489+0000 7faaff7fe700 1 --2- 172.21.15.46:0/940830933 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fab040670e0 0x7fab04069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:08.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.489+0000 7faaff7fe700 1 -- 172.21.15.46:0/940830933 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7389+0+0 (secure 0 0 0) 0x7fab08084b00 con 0x7fab180cbff0 2024-01-27T01:08:08.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.489+0000 7fab16ffd700 1 --2- 172.21.15.46:0/940830933 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fab040670e0 0x7fab04069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:08.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.489+0000 7fab16ffd700 1 --2- 172.21.15.46:0/940830933 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fab040670e0 0x7fab04069590 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fab0c009910 tx=0x7fab0c017000 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:08.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.493+0000 7faaff7fe700 1 -- 172.21.15.46:0/940830933 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fab08051520 con 0x7fab180cbff0 2024-01-27T01:08:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.725+0000 7fab1dc56700 1 -- 172.21.15.46:0/940830933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab180d0500 con 0x7fab180cbff0 2024-01-27T01:08:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.726+0000 7faaff7fe700 1 -- 172.21.15.46:0/940830933 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) v1 ==== 74+0+17594 (secure 0 0 0) 0x7fab0805a020 con 0x7fab180cbff0 2024-01-27T01:08:08.725 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:08.725 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":252,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:08:05.095424+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-01-27T01:07:39.209497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-01-27T01:07:40.386606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6834/1087205220":"2024-01-28T01:08:01.670110+0000","172.21.15.106:0/558414078":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/2701940868":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/3540935782":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/3966339865":"2024-01-28T01:08:01.068164+0000","172.21.15.46:6832/1087205220":"2024-01-28T01:08:01.670110+0000","172.21.15.106:0/3958137863":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/2317042118":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/1816027959":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/2241845238":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1948052707":"2024-01-28T01:07:53.996892+0000","172.21.15.106:6836/62434":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1918093439":"2024-01-28T01:07:53.996892+0000","172.21.15.106:6837/62434":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/469270357":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1388668891":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/1199239718":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/3351808187":"2024-01-28T01:08:01.068164+0000","172.21.15.106:0/2250157080":"2024-01-28T01:07:53.996892+0000","172.21.15.106:0/2626066195":"2024-01-28T01:07:53.996892+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:08:08.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.728+0000 7fab1dc56700 1 -- 172.21.15.46:0/940830933 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fab040670e0 msgr2=0x7fab04069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.728+0000 7fab1dc56700 1 --2- 172.21.15.46:0/940830933 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fab040670e0 0x7fab04069590 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fab0c009910 tx=0x7fab0c017000 comp rx=0 tx=0).stop 2024-01-27T01:08:08.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.728+0000 7fab1dc56700 1 -- 172.21.15.46:0/940830933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab180cbff0 msgr2=0x7fab18158d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.728+0000 7fab1dc56700 1 --2- 172.21.15.46:0/940830933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab180cbff0 0x7fab18158d40 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fab0800c7c0 tx=0x7fab08007db0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.728+0000 7fab1dc56700 1 -- 172.21.15.46:0/940830933 shutdown_connections 2024-01-27T01:08:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.728+0000 7fab1dc56700 1 --2- 172.21.15.46:0/940830933 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fab040670e0 0x7fab04069590 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.728+0000 7fab1dc56700 1 --2- 172.21.15.46:0/940830933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab180cbff0 0x7fab18158d40 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.728+0000 7fab1dc56700 1 --2- 172.21.15.46:0/940830933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab1815d210 0x7fab1816dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.728+0000 7fab1dc56700 1 --2- 172.21.15.46:0/940830933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fab1815ce40 0x7fab1816d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.728+0000 7fab1dc56700 1 -- 172.21.15.46:0/940830933 >> 172.21.15.46:0/940830933 conn(0x7fab1806f840 msgr2=0x7fab1815aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.728+0000 7fab1dc56700 1 -- 172.21.15.46:0/940830933 shutdown_connections 2024-01-27T01:08:08.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.728+0000 7fab1dc56700 1 -- 172.21.15.46:0/940830933 wait complete. 2024-01-27T01:08:08.742 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-01-27T01:08:08.747 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:08.748+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:08.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.883+0000 7f5806510700 1 Processor -- start 2024-01-27T01:08:08.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.883+0000 7f5806510700 1 -- start start 2024-01-27T01:08:08.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.884+0000 7f5806510700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5800159960 0x7f580006bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:08.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.884+0000 7f5806510700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f580006c1f0 0x7f5800069370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:08.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.884+0000 7f5806510700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580006c600 con 0x7f5800159590 2024-01-27T01:08:08.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.884+0000 7f5806510700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f580015e350 con 0x7f580006c1f0 2024-01-27T01:08:08.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.884+0000 7f5806510700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58000c97c0 con 0x7f5800159960 2024-01-27T01:08:08.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.884+0000 7f5804aad700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f580006c1f0 0x7f5800069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:08.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.884+0000 7f57fffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5800159590 0x7f580015d3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57266/0 (socket says 172.21.15.46:57266) 2024-01-27T01:08:08.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.884+0000 7f57fffff700 1 -- 172.21.15.46:0/3706401979 learned_addr learned my addr 172.21.15.46:0/3706401979 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:08.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.884+0000 7f57ff7fe700 1 --2- 172.21.15.46:0/3706401979 >> v2:172.21.15.106:3300/0 conn(0x7f5800159960 0x7f580006bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:08.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.884+0000 7f5804aad700 1 -- 172.21.15.46:0/3706401979 >> v2:172.21.15.106:3300/0 conn(0x7f5800159960 msgr2=0x7f580006bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:08.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.884+0000 7f5804aad700 1 --2- 172.21.15.46:0/3706401979 >> v2:172.21.15.106:3300/0 conn(0x7f5800159960 0x7f580006bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.884+0000 7f5804aad700 1 -- 172.21.15.46:0/3706401979 >> v1:172.21.15.46:6789/0 conn(0x7f5800159590 legacy=0x7f580015d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.884+0000 7f5804aad700 1 -- 172.21.15.46:0/3706401979 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580015dac0 con 0x7f580006c1f0 2024-01-27T01:08:08.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.884+0000 7f57feffd700 1 -- 172.21.15.46:0/3706401979 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812659556 0 0) 0x7f580006c600 con 0x7f5800159590 2024-01-27T01:08:08.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.885+0000 7f5804aad700 1 --2- 172.21.15.46:0/3706401979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f580006c1f0 0x7f5800069370 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f58000caf70 tx=0x7f57f8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94642b7ba73727f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:08.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.885+0000 7f57feffd700 1 -- 172.21.15.46:0/3706401979 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f800e070 con 0x7f580006c1f0 2024-01-27T01:08:08.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.885+0000 7f57feffd700 1 -- 172.21.15.46:0/3706401979 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f800bb20 con 0x7f580006c1f0 2024-01-27T01:08:08.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.885+0000 7f57feffd700 1 -- 172.21.15.46:0/3706401979 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f8012430 con 0x7f580006c1f0 2024-01-27T01:08:08.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.885+0000 7f5806510700 1 -- 172.21.15.46:0/3706401979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f580006c1f0 msgr2=0x7f5800069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.885+0000 7f5806510700 1 --2- 172.21.15.46:0/3706401979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f580006c1f0 0x7f5800069370 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f58000caf70 tx=0x7f57f8009780 comp rx=0 tx=0).stop 2024-01-27T01:08:08.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.885+0000 7f5806510700 1 -- 172.21.15.46:0/3706401979 shutdown_connections 2024-01-27T01:08:08.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.885+0000 7f5806510700 1 --2- 172.21.15.46:0/3706401979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f580006c1f0 0x7f5800069370 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.885+0000 7f5806510700 1 --2- 172.21.15.46:0/3706401979 >> v2:172.21.15.106:3300/0 conn(0x7f5800159960 0x7f580006bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.885+0000 7f5806510700 1 -- 172.21.15.46:0/3706401979 >> 172.21.15.46:0/3706401979 conn(0x7f58000ce530 msgr2=0x7f58000ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:08.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.886+0000 7f5806510700 1 -- 172.21.15.46:0/3706401979 shutdown_connections 2024-01-27T01:08:08.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.886+0000 7f5806510700 1 -- 172.21.15.46:0/3706401979 wait complete. 2024-01-27T01:08:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.887+0000 7f5806510700 1 Processor -- start 2024-01-27T01:08:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.887+0000 7f5806510700 1 -- start start 2024-01-27T01:08:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.888+0000 7f5806510700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f580006c1f0 0x7f580016a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.888+0000 7f5806510700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5800159590 0x7f5800143040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.888+0000 7f5806510700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5800159960 0x7f58000c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.888+0000 7f5806510700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58000c36c0 con 0x7f5800159590 2024-01-27T01:08:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.888+0000 7f5806510700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58000c8850 con 0x7f580006c1f0 2024-01-27T01:08:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.888+0000 7f5806510700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58000c8570 con 0x7f5800159960 2024-01-27T01:08:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.888+0000 7f57ff7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5800159590 0x7f5800143040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.888+0000 7f57ff7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5800159590 0x7f5800143040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55660/0 (socket says 172.21.15.46:55660) 2024-01-27T01:08:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.888+0000 7f57ff7fe700 1 -- 172.21.15.46:0/3182057956 learned_addr learned my addr 172.21.15.46:0/3182057956 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.888+0000 7f57fffff700 1 --2- 172.21.15.46:0/3182057956 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f580006c1f0 0x7f580016a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.888+0000 7f5804aad700 1 --2- 172.21.15.46:0/3182057956 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5800159960 0x7f58000c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.888+0000 7f57ff7fe700 1 -- 172.21.15.46:0/3182057956 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5800159960 msgr2=0x7f58000c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.888+0000 7f57ff7fe700 1 --2- 172.21.15.46:0/3182057956 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5800159960 0x7f58000c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.888+0000 7f57ff7fe700 1 -- 172.21.15.46:0/3182057956 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f580006c1f0 msgr2=0x7f580016a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:08.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.888+0000 7f57ff7fe700 1 --2- 172.21.15.46:0/3182057956 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f580006c1f0 0x7f580016a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:08.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.888+0000 7f57ff7fe700 1 -- 172.21.15.46:0/3182057956 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f8009a20 con 0x7f5800159590 2024-01-27T01:08:08.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.888+0000 7f57fffff700 1 --2- 172.21.15.46:0/3182057956 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f580006c1f0 0x7f580016a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:08.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.889+0000 7f57ff7fe700 1 --2- 172.21.15.46:0/3182057956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5800159590 0x7f5800143040 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f57f400e960 tx=0x7f57f40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:08.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.889+0000 7f57e7fff700 1 -- 172.21.15.46:0/3182057956 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f4013680 con 0x7f5800159590 2024-01-27T01:08:08.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.889+0000 7f57e7fff700 1 -- 172.21.15.46:0/3182057956 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f40137e0 con 0x7f5800159590 2024-01-27T01:08:08.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.889+0000 7f57e7fff700 1 -- 172.21.15.46:0/3182057956 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f4013b80 con 0x7f5800159590 2024-01-27T01:08:08.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.889+0000 7f5806510700 1 -- 172.21.15.46:0/3182057956 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58000c3070 con 0x7f5800159590 2024-01-27T01:08:08.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.889+0000 7f5806510700 1 -- 172.21.15.46:0/3182057956 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5800143580 con 0x7f5800159590 2024-01-27T01:08:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.890+0000 7f5806510700 1 -- 172.21.15.46:0/3182057956 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58000c3870 con 0x7f5800159590 2024-01-27T01:08:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.890+0000 7f57e7fff700 1 -- 172.21.15.46:0/3182057956 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f57f40123d0 con 0x7f5800159590 2024-01-27T01:08:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.891+0000 7f57e7fff700 1 --2- 172.21.15.46:0/3182057956 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f57ec0670e0 0x7f57ec069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:08.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.891+0000 7f57e7fff700 1 -- 172.21.15.46:0/3182057956 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7389+0+0 (secure 0 0 0) 0x7f57f4084f50 con 0x7f5800159590 2024-01-27T01:08:08.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.891+0000 7f57fffff700 1 --2- 172.21.15.46:0/3182057956 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f57ec0670e0 0x7f57ec069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:08.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.891+0000 7f57fffff700 1 --2- 172.21.15.46:0/3182057956 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f57ec0670e0 0x7f57ec069590 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5800166c90 tx=0x7f57f0007d30 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:08.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:08.895+0000 7f57e7fff700 1 -- 172.21.15.46:0/3182057956 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f58000c3870 con 0x7f5800159590 2024-01-27T01:08:09.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.152+0000 7f5806510700 1 -- 172.21.15.46:0/3182057956 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7f580004c710 con 0x7f5800159590 2024-01-27T01:08:09.151 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-27T01:08:09.153+0000 7f33ffe86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-27T01:08:09.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.156+0000 7f57e7fff700 1 -- 172.21.15.46:0/3182057956 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v141) v1 ==== 167+0+0 (secure 0 0 0) 0x7f57f4055340 con 0x7f5800159590 2024-01-27T01:08:09.158 INFO:tasks.ceph.mds.c.smithi046.stderr: -1535> 2024-01-27T01:07:44.922+0000 7f45e261c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-27T01:08:09.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.161+0000 7f5806510700 1 -- 172.21.15.46:0/3182057956 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f57ec0670e0 msgr2=0x7f57ec069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:09.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.161+0000 7f5806510700 1 --2- 172.21.15.46:0/3182057956 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f57ec0670e0 0x7f57ec069590 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5800166c90 tx=0x7f57f0007d30 comp rx=0 tx=0).stop 2024-01-27T01:08:09.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.161+0000 7f5806510700 1 -- 172.21.15.46:0/3182057956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5800159590 msgr2=0x7f5800143040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:09.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.161+0000 7f5806510700 1 --2- 172.21.15.46:0/3182057956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5800159590 0x7f5800143040 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f57f400e960 tx=0x7f57f40091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:09.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.161+0000 7f5806510700 1 -- 172.21.15.46:0/3182057956 shutdown_connections 2024-01-27T01:08:09.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.161+0000 7f5806510700 1 --2- 172.21.15.46:0/3182057956 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f57ec0670e0 0x7f57ec069590 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:09.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.161+0000 7f5806510700 1 --2- 172.21.15.46:0/3182057956 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5800159960 0x7f58000c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:09.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.161+0000 7f5806510700 1 --2- 172.21.15.46:0/3182057956 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f580006c1f0 0x7f580016a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:09.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.161+0000 7f5806510700 1 --2- 172.21.15.46:0/3182057956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5800159590 0x7f5800143040 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:09.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.161+0000 7f5806510700 1 -- 172.21.15.46:0/3182057956 >> 172.21.15.46:0/3182057956 conn(0x7f58000ce530 msgr2=0x7f580006a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:09.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.161+0000 7f5806510700 1 -- 172.21.15.46:0/3182057956 shutdown_connections 2024-01-27T01:08:09.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.161+0000 7f5806510700 1 -- 172.21.15.46:0/3182057956 wait complete. 2024-01-27T01:08:09.161 INFO:teuthology.orchestra.run.smithi046.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-27T01:08:09.175 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-01-27T01:08:09.183 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-27T01:08:09.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.315+0000 7f0ec0828700 1 Processor -- start 2024-01-27T01:08:09.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.315+0000 7f0ec0828700 1 -- start start 2024-01-27T01:08:09.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.316+0000 7f0ec0828700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0ebc1599d0 0x7f0ebc159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:09.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.316+0000 7f0ec0828700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc0ab620 con 0x7f0ebc0ab1c0 2024-01-27T01:08:09.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.316+0000 7f0ec0828700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc0b1a60 con 0x7f0ebc15d8c0 2024-01-27T01:08:09.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.316+0000 7f0ec0828700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ebc0b11c0 con 0x7f0ebc1599d0 2024-01-27T01:08:09.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.316+0000 7f0eb9d9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0ebc0ab1c0 0x7f0ebc0b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57294/0 (socket says 172.21.15.46:57294) 2024-01-27T01:08:09.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.316+0000 7f0eb9d9b700 1 -- 172.21.15.46:0/3098299700 learned_addr learned my addr 172.21.15.46:0/3098299700 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:09.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.316+0000 7f0eba59c700 1 --2- 172.21.15.46:0/3098299700 >> v2:172.21.15.106:3300/0 conn(0x7f0ebc1599d0 0x7f0ebc159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:09.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.316+0000 7f0eb959a700 1 -- 172.21.15.46:0/3098299700 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224922654 0 0) 0x7f0eb0001110 con 0x7f0ebc0ab1c0 2024-01-27T01:08:09.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.316+0000 7f0eb959a700 1 -- 172.21.15.46:0/3098299700 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ea4003570 con 0x7f0ebc0ab1c0 2024-01-27T01:08:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.316+0000 7f0eb959a700 1 -- 172.21.15.46:0/3098299700 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463915023 0 0) 0x7f0ebc0b1a60 con 0x7f0ebc15d8c0 2024-01-27T01:08:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.316+0000 7f0eb959a700 1 -- 172.21.15.46:0/3098299700 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0eb0001110 con 0x7f0ebc15d8c0 2024-01-27T01:08:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.316+0000 7f0eb959a700 1 -- 172.21.15.46:0/3098299700 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3771334554 0 0) 0x7f0ea4003570 con 0x7f0ebc0ab1c0 2024-01-27T01:08:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.317+0000 7f0eba59c700 1 -- 172.21.15.46:0/3098299700 >> v1:172.21.15.106:6789/0 conn(0x7f0ebc15d8c0 legacy=0x7f0ebc0b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.317+0000 7f0eba59c700 1 -- 172.21.15.46:0/3098299700 >> v1:172.21.15.46:6789/0 conn(0x7f0ebc0ab1c0 legacy=0x7f0ebc0b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.317+0000 7f0eba59c700 1 -- 172.21.15.46:0/3098299700 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ebc0b14a0 con 0x7f0ebc1599d0 2024-01-27T01:08:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.317+0000 7f0eba59c700 1 --2- 172.21.15.46:0/3098299700 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ebc1599d0 0x7f0ebc159da0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f0eac0098d0 tx=0x7f0eac009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a87cac57003d102a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.317+0000 7f0eb959a700 1 -- 172.21.15.46:0/3098299700 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0001600 con 0x7f0ebc1599d0 2024-01-27T01:08:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.317+0000 7f0eb959a700 1 -- 172.21.15.46:0/3098299700 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eac011420 con 0x7f0ebc1599d0 2024-01-27T01:08:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.317+0000 7f0eb959a700 1 -- 172.21.15.46:0/3098299700 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac0117e0 con 0x7f0ebc1599d0 2024-01-27T01:08:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.317+0000 7f0ec0828700 1 -- 172.21.15.46:0/3098299700 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ebc1599d0 msgr2=0x7f0ebc159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.317+0000 7f0ec0828700 1 --2- 172.21.15.46:0/3098299700 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ebc1599d0 0x7f0ebc159da0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f0eac0098d0 tx=0x7f0eac009120 comp rx=0 tx=0).stop 2024-01-27T01:08:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.318+0000 7f0ec0828700 1 -- 172.21.15.46:0/3098299700 shutdown_connections 2024-01-27T01:08:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.318+0000 7f0ec0828700 1 --2- 172.21.15.46:0/3098299700 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ebc1599d0 0x7f0ebc159da0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.318+0000 7f0ec0828700 1 -- 172.21.15.46:0/3098299700 >> 172.21.15.46:0/3098299700 conn(0x7f0ebc06f840 msgr2=0x7f0ebc087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.318+0000 7f0ec0828700 1 -- 172.21.15.46:0/3098299700 shutdown_connections 2024-01-27T01:08:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.318+0000 7f0ec0828700 1 -- 172.21.15.46:0/3098299700 wait complete. 2024-01-27T01:08:09.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.319+0000 7f0ec0828700 1 Processor -- start 2024-01-27T01:08:09.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.319+0000 7f0ec0828700 1 -- start start 2024-01-27T01:08:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0ec0828700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ebc0ab1c0 0x7f0ebc09d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0ec0828700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ebc1599d0 0x7f0ebc149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0ec0828700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ebc15d8c0 0x7f0ebc15f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0ec0828700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc09daf0 con 0x7f0ebc0ab1c0 2024-01-27T01:08:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0ec0828700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc0ab7c0 con 0x7f0ebc1599d0 2024-01-27T01:08:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0ec0828700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc0abd80 con 0x7f0ebc15d8c0 2024-01-27T01:08:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0eba59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ebc0ab1c0 0x7f0ebc09d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0eba59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ebc0ab1c0 0x7f0ebc09d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55690/0 (socket says 172.21.15.46:55690) 2024-01-27T01:08:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0eba59c700 1 -- 172.21.15.46:0/3358802190 learned_addr learned my addr 172.21.15.46:0/3358802190 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0eb9d9b700 1 --2- 172.21.15.46:0/3358802190 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ebc1599d0 0x7f0ebc149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0ebad9d700 1 --2- 172.21.15.46:0/3358802190 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ebc15d8c0 0x7f0ebc15f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0eba59c700 1 -- 172.21.15.46:0/3358802190 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ebc15d8c0 msgr2=0x7f0ebc15f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0eba59c700 1 --2- 172.21.15.46:0/3358802190 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ebc15d8c0 0x7f0ebc15f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0eba59c700 1 -- 172.21.15.46:0/3358802190 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ebc1599d0 msgr2=0x7f0ebc149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:09.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0eba59c700 1 --2- 172.21.15.46:0/3358802190 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ebc1599d0 0x7f0ebc149d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:09.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0eba59c700 1 -- 172.21.15.46:0/3358802190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac0093c0 con 0x7f0ebc0ab1c0 2024-01-27T01:08:09.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0eba59c700 1 --2- 172.21.15.46:0/3358802190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ebc0ab1c0 0x7f0ebc09d5b0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc089010 tx=0x7f0eac016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:09.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.320+0000 7f0eaa7fc700 1 -- 172.21.15.46:0/3358802190 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac01caf0 con 0x7f0ebc0ab1c0 2024-01-27T01:08:09.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.321+0000 7f0eaa7fc700 1 -- 172.21.15.46:0/3358802190 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eac01cc50 con 0x7f0ebc0ab1c0 2024-01-27T01:08:09.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.321+0000 7f0ec0828700 1 -- 172.21.15.46:0/3358802190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ebc09dd10 con 0x7f0ebc0ab1c0 2024-01-27T01:08:09.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.321+0000 7f0eaa7fc700 1 -- 172.21.15.46:0/3358802190 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac01a450 con 0x7f0ebc0ab1c0 2024-01-27T01:08:09.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.321+0000 7f0ec0828700 1 -- 172.21.15.46:0/3358802190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ebc0aba10 con 0x7f0ebc0ab1c0 2024-01-27T01:08:09.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.322+0000 7f0ec0828700 1 -- 172.21.15.46:0/3358802190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ebc14a330 con 0x7f0ebc0ab1c0 2024-01-27T01:08:09.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.322+0000 7f0eaa7fc700 1 -- 172.21.15.46:0/3358802190 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f0eac01a7e0 con 0x7f0ebc0ab1c0 2024-01-27T01:08:09.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.323+0000 7f0eaa7fc700 1 --2- 172.21.15.46:0/3358802190 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0e9c0670e0 0x7f0e9c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:09.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.323+0000 7f0eaa7fc700 1 -- 172.21.15.46:0/3358802190 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7475+0+0 (secure 0 0 0) 0x7f0eac086710 con 0x7f0ebc0ab1c0 2024-01-27T01:08:09.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.323+0000 7f0eb9d9b700 1 --2- 172.21.15.46:0/3358802190 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0e9c0670e0 0x7f0e9c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:09.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.323+0000 7f0eb9d9b700 1 --2- 172.21.15.46:0/3358802190 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0e9c0670e0 0x7f0e9c069590 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc160900 tx=0x7f0eb0001330 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:09.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.327+0000 7f0eaa7fc700 1 -- 172.21.15.46:0/3358802190 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0eac052f70 con 0x7f0ebc0ab1c0 2024-01-27T01:08:09.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:09.614+0000 7f0ec0828700 1 -- 172.21.15.46:0/3358802190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0ebc0afc50 con 0x7f0ebc0ab1c0 2024-01-27T01:08:09.706 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:09.706+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:10.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.158+0000 7f0eaa7fc700 1 -- 172.21.15.46:0/3358802190 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v142) v1 ==== 107+0+0 (secure 0 0 0) 0x7f0eac056590 con 0x7f0ebc0ab1c0 2024-01-27T01:08:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.161+0000 7f0ec0828700 1 -- 172.21.15.46:0/3358802190 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0e9c0670e0 msgr2=0x7f0e9c069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.161+0000 7f0ec0828700 1 --2- 172.21.15.46:0/3358802190 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0e9c0670e0 0x7f0e9c069590 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc160900 tx=0x7f0eb0001330 comp rx=0 tx=0).stop 2024-01-27T01:08:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.161+0000 7f0ec0828700 1 -- 172.21.15.46:0/3358802190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ebc0ab1c0 msgr2=0x7f0ebc09d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.161+0000 7f0ec0828700 1 --2- 172.21.15.46:0/3358802190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ebc0ab1c0 0x7f0ebc09d5b0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc089010 tx=0x7f0eac016000 comp rx=0 tx=0).stop 2024-01-27T01:08:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.161+0000 7f0ec0828700 1 -- 172.21.15.46:0/3358802190 shutdown_connections 2024-01-27T01:08:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.161+0000 7f0ec0828700 1 --2- 172.21.15.46:0/3358802190 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0e9c0670e0 0x7f0e9c069590 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.161+0000 7f0ec0828700 1 --2- 172.21.15.46:0/3358802190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ebc0ab1c0 0x7f0ebc09d5b0 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.161+0000 7f0ec0828700 1 --2- 172.21.15.46:0/3358802190 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0ebc15d8c0 0x7f0ebc15f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.161+0000 7f0ec0828700 1 --2- 172.21.15.46:0/3358802190 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ebc1599d0 0x7f0ebc149d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.161+0000 7f0ec0828700 1 -- 172.21.15.46:0/3358802190 >> 172.21.15.46:0/3358802190 conn(0x7f0ebc06f840 msgr2=0x7f0ebc068b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.161+0000 7f0ec0828700 1 -- 172.21.15.46:0/3358802190 shutdown_connections 2024-01-27T01:08:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.161+0000 7f0ec0828700 1 -- 172.21.15.46:0/3358802190 wait complete. 2024-01-27T01:08:10.176 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-01-27T01:08:10.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.368+0000 7f947b2de700 1 Processor -- start 2024-01-27T01:08:10.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.368+0000 7f947b2de700 1 -- start start 2024-01-27T01:08:10.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.369+0000 7f947b2de700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9474159b20 0x7f94740ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:10.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.369+0000 7f947b2de700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94740691c0 con 0x7f94740ae340 2024-01-27T01:08:10.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.369+0000 7f947b2de700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947406a5e0 con 0x7f9474159750 2024-01-27T01:08:10.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.369+0000 7f947b2de700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f947415dfd0 con 0x7f9474159b20 2024-01-27T01:08:10.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.369+0000 7f947987b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f94740ae340 0x7f947415d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44928/0 (socket says 172.21.15.46:44928) 2024-01-27T01:08:10.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.369+0000 7f947987b700 1 -- 172.21.15.46:0/122205215 learned_addr learned my addr 172.21.15.46:0/122205215 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:10.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.369+0000 7f9478879700 1 --2- 172.21.15.46:0/122205215 >> v2:172.21.15.106:3300/0 conn(0x7f9474159b20 0x7f94740ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:10.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.370+0000 7f946bfff700 1 -- 172.21.15.46:0/122205215 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148149451 0 0) 0x7f94740691c0 con 0x7f94740ae340 2024-01-27T01:08:10.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.370+0000 7f946bfff700 1 -- 172.21.15.46:0/122205215 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9460003570 con 0x7f94740ae340 2024-01-27T01:08:10.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.370+0000 7f946bfff700 1 -- 172.21.15.46:0/122205215 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954862746 0 0) 0x7f947406a5e0 con 0x7f9474159750 2024-01-27T01:08:10.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.370+0000 7f946bfff700 1 -- 172.21.15.46:0/122205215 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94740691c0 con 0x7f9474159750 2024-01-27T01:08:10.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.370+0000 7f9478879700 1 -- 172.21.15.46:0/122205215 >> v1:172.21.15.106:6789/0 conn(0x7f9474159750 legacy=0x7f9474069650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:10.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.370+0000 7f9478879700 1 -- 172.21.15.46:0/122205215 >> v1:172.21.15.46:6789/0 conn(0x7f94740ae340 legacy=0x7f947415d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:10.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.370+0000 7f9478879700 1 -- 172.21.15.46:0/122205215 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9474068b70 con 0x7f9474159b20 2024-01-27T01:08:10.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.370+0000 7f9478879700 1 --2- 172.21.15.46:0/122205215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9474159b20 0x7f94740ade00 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f94640098e0 tx=0x7f94640091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a27ea67747cdb744 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:10.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.370+0000 7f946bfff700 1 -- 172.21.15.46:0/122205215 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464011630 con 0x7f9474159b20 2024-01-27T01:08:10.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.370+0000 7f946bfff700 1 -- 172.21.15.46:0/122205215 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9464011790 con 0x7f9474159b20 2024-01-27T01:08:10.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.370+0000 7f946bfff700 1 -- 172.21.15.46:0/122205215 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464011b30 con 0x7f9474159b20 2024-01-27T01:08:10.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.371+0000 7f947b2de700 1 -- 172.21.15.46:0/122205215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9474159b20 msgr2=0x7f94740ade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:10.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.371+0000 7f947b2de700 1 --2- 172.21.15.46:0/122205215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9474159b20 0x7f94740ade00 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f94640098e0 tx=0x7f94640091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:10.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.371+0000 7f947b2de700 1 -- 172.21.15.46:0/122205215 shutdown_connections 2024-01-27T01:08:10.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.371+0000 7f947b2de700 1 --2- 172.21.15.46:0/122205215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9474159b20 0x7f94740ade00 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:10.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.371+0000 7f947b2de700 1 -- 172.21.15.46:0/122205215 >> 172.21.15.46:0/122205215 conn(0x7f94740ce530 msgr2=0x7f94740ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:10.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.371+0000 7f947b2de700 1 -- 172.21.15.46:0/122205215 shutdown_connections 2024-01-27T01:08:10.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.371+0000 7f947b2de700 1 -- 172.21.15.46:0/122205215 wait complete. 2024-01-27T01:08:10.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.373+0000 7f947b2de700 1 Processor -- start 2024-01-27T01:08:10.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.373+0000 7f947b2de700 1 -- start start 2024-01-27T01:08:10.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.374+0000 7f947b2de700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94740ae340 0x7f94740b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:10.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.374+0000 7f947b2de700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9474159750 0x7f947416d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:10.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.374+0000 7f947b2de700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9474159b20 0x7f947416db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:10.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.374+0000 7f947b2de700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94740b3c20 con 0x7f9474159b20 2024-01-27T01:08:10.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.374+0000 7f947b2de700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94740c7e20 con 0x7f9474159750 2024-01-27T01:08:10.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.374+0000 7f947b2de700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94740c8100 con 0x7f94740ae340 2024-01-27T01:08:10.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.374+0000 7f947987b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9474159b20 0x7f947416db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:10.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.374+0000 7f9478879700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9474159750 0x7f947416d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:10.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.374+0000 7f947987b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9474159b20 0x7f947416db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49380/0 (socket says 172.21.15.46:49380) 2024-01-27T01:08:10.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.374+0000 7f947907a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94740ae340 0x7f94740b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:10.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.374+0000 7f947987b700 1 -- 172.21.15.46:0/736109431 learned_addr learned my addr 172.21.15.46:0/736109431 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:10.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.374+0000 7f947987b700 1 -- 172.21.15.46:0/736109431 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94740ae340 msgr2=0x7f94740b36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:10.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.374+0000 7f947987b700 1 --2- 172.21.15.46:0/736109431 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94740ae340 0x7f94740b36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:10.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.374+0000 7f947987b700 1 -- 172.21.15.46:0/736109431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9474159750 msgr2=0x7f947416d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:10.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.374+0000 7f947987b700 1 --2- 172.21.15.46:0/736109431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9474159750 0x7f947416d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:10.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.374+0000 7f947987b700 1 -- 172.21.15.46:0/736109431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9464009420 con 0x7f9474159b20 2024-01-27T01:08:10.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.374+0000 7f9478879700 1 --2- 172.21.15.46:0/736109431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9474159750 0x7f947416d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:10.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.375+0000 7f947987b700 1 --2- 172.21.15.46:0/736109431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9474159b20 0x7f947416db00 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f946c00cd00 tx=0x7f946c001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:10.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.375+0000 7f94697fa700 1 -- 172.21.15.46:0/736109431 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c00ce00 con 0x7f9474159b20 2024-01-27T01:08:10.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.375+0000 7f94697fa700 1 -- 172.21.15.46:0/736109431 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946c00aac0 con 0x7f9474159b20 2024-01-27T01:08:10.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.375+0000 7f947b2de700 1 -- 172.21.15.46:0/736109431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94740c83e0 con 0x7f9474159b20 2024-01-27T01:08:10.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.375+0000 7f94697fa700 1 -- 172.21.15.46:0/736109431 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c00ae80 con 0x7f9474159b20 2024-01-27T01:08:10.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.375+0000 7f947b2de700 1 -- 172.21.15.46:0/736109431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94740b1590 con 0x7f9474159b20 2024-01-27T01:08:10.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.376+0000 7f947b2de700 1 -- 172.21.15.46:0/736109431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94740b17b0 con 0x7f9474159b20 2024-01-27T01:08:10.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.376+0000 7f94697fa700 1 -- 172.21.15.46:0/736109431 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f946c007300 con 0x7f9474159b20 2024-01-27T01:08:10.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.377+0000 7f94697fa700 1 --2- 172.21.15.46:0/736109431 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f94580670e0 0x7f9458069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:10.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.377+0000 7f94697fa700 1 -- 172.21.15.46:0/736109431 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7475+0+0 (secure 0 0 0) 0x7f946c085260 con 0x7f9474159b20 2024-01-27T01:08:10.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.377+0000 7f947907a700 1 --2- 172.21.15.46:0/736109431 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f94580670e0 0x7f9458069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:10.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.378+0000 7f947907a700 1 --2- 172.21.15.46:0/736109431 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f94580670e0 0x7f9458069590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9470007fd0 tx=0x7f9470007d50 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:10.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.381+0000 7f94697fa700 1 -- 172.21.15.46:0/736109431 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f946c051be0 con 0x7f9474159b20 2024-01-27T01:08:10.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:10.590+0000 7f947b2de700 1 -- 172.21.15.46:0/736109431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f947404c710 con 0x7f9474159b20 2024-01-27T01:08:10.646 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:10.646+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:11.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.165+0000 7f94697fa700 1 -- 172.21.15.46:0/736109431 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v254) v1 ==== 190+0+0 (secure 0 0 0) 0x7f946c055200 con 0x7f9474159b20 2024-01-27T01:08:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.169+0000 7f947b2de700 1 -- 172.21.15.46:0/736109431 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f94580670e0 msgr2=0x7f9458069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.169+0000 7f947b2de700 1 --2- 172.21.15.46:0/736109431 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f94580670e0 0x7f9458069590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9470007fd0 tx=0x7f9470007d50 comp rx=0 tx=0).stop 2024-01-27T01:08:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.169+0000 7f947b2de700 1 -- 172.21.15.46:0/736109431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9474159b20 msgr2=0x7f947416db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.169+0000 7f947b2de700 1 --2- 172.21.15.46:0/736109431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9474159b20 0x7f947416db00 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f946c00cd00 tx=0x7f946c001ac0 comp rx=0 tx=0).stop 2024-01-27T01:08:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.170+0000 7f947b2de700 1 -- 172.21.15.46:0/736109431 shutdown_connections 2024-01-27T01:08:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.170+0000 7f947b2de700 1 --2- 172.21.15.46:0/736109431 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f94580670e0 0x7f9458069590 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.170+0000 7f947b2de700 1 --2- 172.21.15.46:0/736109431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9474159b20 0x7f947416db00 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.170+0000 7f947b2de700 1 --2- 172.21.15.46:0/736109431 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94740ae340 0x7f94740b36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.170+0000 7f947b2de700 1 --2- 172.21.15.46:0/736109431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9474159750 0x7f947416d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.170+0000 7f947b2de700 1 -- 172.21.15.46:0/736109431 >> 172.21.15.46:0/736109431 conn(0x7f94740ce530 msgr2=0x7f947406dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:11.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.171+0000 7f947b2de700 1 -- 172.21.15.46:0/736109431 shutdown_connections 2024-01-27T01:08:11.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.172+0000 7f947b2de700 1 -- 172.21.15.46:0/736109431 wait complete. 2024-01-27T01:08:11.171 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'backup_fs_metadata' removed 2024-01-27T01:08:11.187 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-01-27T01:08:11.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.336+0000 7fa60566a700 1 Processor -- start 2024-01-27T01:08:11.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.336+0000 7fa60566a700 1 -- start start 2024-01-27T01:08:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.337+0000 7fa60566a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa6000c9b20 0x7fa60006d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.337+0000 7fa60566a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa600068040 0x7fa60006ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.337+0000 7fa60566a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa60015add0 con 0x7fa6000c7cd0 2024-01-27T01:08:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.337+0000 7fa60566a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa600159ff0 con 0x7fa6000c9b20 2024-01-27T01:08:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.337+0000 7fa60566a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6000c9ef0 con 0x7fa600068040 2024-01-27T01:08:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.337+0000 7fa5feffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa6000c9b20 0x7fa60006d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.337+0000 7fa5feffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa6000c9b20 0x7fa60006d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49384/0 (socket says 172.21.15.46:49384) 2024-01-27T01:08:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.337+0000 7fa5feffd700 1 -- 172.21.15.46:0/1927453488 learned_addr learned my addr 172.21.15.46:0/1927453488 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.337+0000 7fa5ff7fe700 1 --2- 172.21.15.46:0/1927453488 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa600068040 0x7fa60006ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.338+0000 7fa5fdffb700 1 -- 172.21.15.46:0/1927453488 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3449810864 0 0) 0x7fa60015add0 con 0x7fa6000c7cd0 2024-01-27T01:08:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.338+0000 7fa5fdffb700 1 -- 172.21.15.46:0/1927453488 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ec003570 con 0x7fa6000c7cd0 2024-01-27T01:08:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.338+0000 7fa5feffd700 1 -- 172.21.15.46:0/1927453488 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa600068040 msgr2=0x7fa60006ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.338+0000 7fa5feffd700 1 --2- 172.21.15.46:0/1927453488 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa600068040 0x7fa60006ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.338+0000 7fa5feffd700 1 -- 172.21.15.46:0/1927453488 >> v1:172.21.15.106:6789/0 conn(0x7fa6000c7cd0 legacy=0x7fa600067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.338+0000 7fa5feffd700 1 -- 172.21.15.46:0/1927453488 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60015a200 con 0x7fa6000c9b20 2024-01-27T01:08:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.338+0000 7fa5ff7fe700 1 --2- 172.21.15.46:0/1927453488 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa600068040 0x7fa60006ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.338+0000 7fa5feffd700 1 --2- 172.21.15.46:0/1927453488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6000c9b20 0x7fa60006d080 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fa5e80099b0 tx=0x7fa5e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a3e35272b6bd80 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.338+0000 7fa5fdffb700 1 -- 172.21.15.46:0/1927453488 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e800b680 con 0x7fa6000c9b20 2024-01-27T01:08:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.338+0000 7fa5fdffb700 1 -- 172.21.15.46:0/1927453488 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e800b7e0 con 0x7fa6000c9b20 2024-01-27T01:08:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.338+0000 7fa5fdffb700 1 -- 172.21.15.46:0/1927453488 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e800bb80 con 0x7fa6000c9b20 2024-01-27T01:08:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.339+0000 7fa60566a700 1 -- 172.21.15.46:0/1927453488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6000c9b20 msgr2=0x7fa60006d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.339+0000 7fa60566a700 1 --2- 172.21.15.46:0/1927453488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6000c9b20 0x7fa60006d080 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fa5e80099b0 tx=0x7fa5e8009250 comp rx=0 tx=0).stop 2024-01-27T01:08:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.339+0000 7fa60566a700 1 -- 172.21.15.46:0/1927453488 shutdown_connections 2024-01-27T01:08:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.339+0000 7fa60566a700 1 --2- 172.21.15.46:0/1927453488 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa600068040 0x7fa60006ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.339+0000 7fa60566a700 1 --2- 172.21.15.46:0/1927453488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6000c9b20 0x7fa60006d080 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:11.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.339+0000 7fa60566a700 1 -- 172.21.15.46:0/1927453488 >> 172.21.15.46:0/1927453488 conn(0x7fa60006b820 msgr2=0x7fa6000cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.339+0000 7fa60566a700 1 -- 172.21.15.46:0/1927453488 shutdown_connections 2024-01-27T01:08:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.339+0000 7fa60566a700 1 -- 172.21.15.46:0/1927453488 wait complete. 2024-01-27T01:08:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.341+0000 7fa60566a700 1 Processor -- start 2024-01-27T01:08:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.341+0000 7fa60566a700 1 -- start start 2024-01-27T01:08:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa60566a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa600068040 0x7fa60015d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa60566a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6000c7cd0 0x7fa600155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa60566a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6000c9b20 0x7fa600150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa60566a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60015e730 con 0x7fa6000c9b20 2024-01-27T01:08:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa60566a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6000b1d30 con 0x7fa6000c7cd0 2024-01-27T01:08:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa60566a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa600156320 con 0x7fa600068040 2024-01-27T01:08:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa5feffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa600068040 0x7fa60015d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa5ff7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6000c9b20 0x7fa600150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa5feffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa600068040 0x7fa60015d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:39340/0 (socket says 172.21.15.46:39340) 2024-01-27T01:08:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa5feffd700 1 -- 172.21.15.46:0/3681940791 learned_addr learned my addr 172.21.15.46:0/3681940791 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa5fe7fc700 1 --2- 172.21.15.46:0/3681940791 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6000c7cd0 0x7fa600155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa5ff7fe700 1 -- 172.21.15.46:0/3681940791 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa600068040 msgr2=0x7fa60015d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa5ff7fe700 1 --2- 172.21.15.46:0/3681940791 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa600068040 0x7fa60015d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa5ff7fe700 1 -- 172.21.15.46:0/3681940791 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6000c7cd0 msgr2=0x7fa600155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa5ff7fe700 1 --2- 172.21.15.46:0/3681940791 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6000c7cd0 0x7fa600155de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa5ff7fe700 1 -- 172.21.15.46:0/3681940791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f8009a20 con 0x7fa6000c9b20 2024-01-27T01:08:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa5feffd700 1 --2- 172.21.15.46:0/3681940791 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa600068040 0x7fa60015d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.342+0000 7fa5ff7fe700 1 --2- 172.21.15.46:0/3681940791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6000c9b20 0x7fa600150110 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009f10 tx=0x7fa5f800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.343+0000 7fa5f6ffd700 1 -- 172.21.15.46:0/3681940791 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f800cb70 con 0x7fa6000c9b20 2024-01-27T01:08:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.343+0000 7fa5f6ffd700 1 -- 172.21.15.46:0/3681940791 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f800f040 con 0x7fa6000c9b20 2024-01-27T01:08:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.343+0000 7fa60566a700 1 -- 172.21.15.46:0/3681940791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e80094f0 con 0x7fa6000c9b20 2024-01-27T01:08:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.343+0000 7fa5f6ffd700 1 -- 172.21.15.46:0/3681940791 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8014480 con 0x7fa6000c9b20 2024-01-27T01:08:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.343+0000 7fa60566a700 1 -- 172.21.15.46:0/3681940791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa60015de90 con 0x7fa6000c9b20 2024-01-27T01:08:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.344+0000 7fa60566a700 1 -- 172.21.15.46:0/3681940791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa60004c710 con 0x7fa6000c9b20 2024-01-27T01:08:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.344+0000 7fa5f6ffd700 1 -- 172.21.15.46:0/3681940791 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7fa5f8003020 con 0x7fa6000c9b20 2024-01-27T01:08:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.345+0000 7fa5f6ffd700 1 --2- 172.21.15.46:0/3681940791 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa5e00670e0 0x7fa5e0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.345+0000 7fa5f6ffd700 1 -- 172.21.15.46:0/3681940791 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7031+0+0 (secure 0 0 0) 0x7fa5f80856f0 con 0x7fa6000c9b20 2024-01-27T01:08:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.345+0000 7fa5feffd700 1 --2- 172.21.15.46:0/3681940791 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa5e00670e0 0x7fa5e0069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.345+0000 7fa5feffd700 1 --2- 172.21.15.46:0/3681940791 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa5e00670e0 0x7fa5e0069590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8009fd0 tx=0x7fa5e8017000 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:11.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.349+0000 7fa5f6ffd700 1 -- 172.21.15.46:0/3681940791 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa5f80520a0 con 0x7fa6000c9b20 2024-01-27T01:08:11.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:11.559+0000 7fa60566a700 1 -- 172.21.15.46:0/3681940791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa6000ce570 con 0x7fa6000c9b20 2024-01-27T01:08:11.599 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:11.600+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:12.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.165+0000 7fa5f6ffd700 1 -- 172.21.15.46:0/3681940791 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v255) v1 ==== 178+0+0 (secure 0 0 0) 0x7fa5f80556c0 con 0x7fa6000c9b20 2024-01-27T01:08:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.169+0000 7fa60566a700 1 -- 172.21.15.46:0/3681940791 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa5e00670e0 msgr2=0x7fa5e0069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.169+0000 7fa60566a700 1 --2- 172.21.15.46:0/3681940791 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa5e00670e0 0x7fa5e0069590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8009fd0 tx=0x7fa5e8017000 comp rx=0 tx=0).stop 2024-01-27T01:08:12.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.169+0000 7fa60566a700 1 -- 172.21.15.46:0/3681940791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6000c9b20 msgr2=0x7fa600150110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:12.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.169+0000 7fa60566a700 1 --2- 172.21.15.46:0/3681940791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6000c9b20 0x7fa600150110 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009f10 tx=0x7fa5f800b040 comp rx=0 tx=0).stop 2024-01-27T01:08:12.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.169+0000 7fa60566a700 1 -- 172.21.15.46:0/3681940791 shutdown_connections 2024-01-27T01:08:12.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.169+0000 7fa60566a700 1 --2- 172.21.15.46:0/3681940791 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa5e00670e0 0x7fa5e0069590 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.169+0000 7fa60566a700 1 --2- 172.21.15.46:0/3681940791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6000c9b20 0x7fa600150110 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.169+0000 7fa60566a700 1 --2- 172.21.15.46:0/3681940791 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa600068040 0x7fa60015d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.169+0000 7fa60566a700 1 --2- 172.21.15.46:0/3681940791 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6000c7cd0 0x7fa600155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.169+0000 7fa60566a700 1 -- 172.21.15.46:0/3681940791 >> 172.21.15.46:0/3681940791 conn(0x7fa60006b820 msgr2=0x7fa60006f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:12.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.169+0000 7fa60566a700 1 -- 172.21.15.46:0/3681940791 shutdown_connections 2024-01-27T01:08:12.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.169+0000 7fa60566a700 1 -- 172.21.15.46:0/3681940791 wait complete. 2024-01-27T01:08:12.170 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'backup_fs_data' removed 2024-01-27T01:08:12.188 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-27T01:08:12.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.330+0000 7f2295a17700 1 Processor -- start 2024-01-27T01:08:12.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.330+0000 7f2295a17700 1 -- start start 2024-01-27T01:08:12.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.330+0000 7f2295a17700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f22901599d0 0x7f2290159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:12.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.330+0000 7f2295a17700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22900b1a60 con 0x7f22900ab1c0 2024-01-27T01:08:12.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.331+0000 7f2295a17700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22900ab5f0 con 0x7f229015d8c0 2024-01-27T01:08:12.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.331+0000 7f2295a17700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22900b11c0 con 0x7f22901599d0 2024-01-27T01:08:12.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.331+0000 7f22867fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f22900ab1c0 0x7f22900b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44936/0 (socket says 172.21.15.46:44936) 2024-01-27T01:08:12.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.331+0000 7f22867fc700 1 -- 172.21.15.46:0/2289289762 learned_addr learned my addr 172.21.15.46:0/2289289762 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:12.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.331+0000 7f228effd700 1 --2- 172.21.15.46:0/2289289762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f22901599d0 0x7f2290159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:12.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.331+0000 7f228e7fc700 1 -- 172.21.15.46:0/2289289762 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436878938 0 0) 0x7f22900b1a60 con 0x7f22900ab1c0 2024-01-27T01:08:12.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.331+0000 7f228e7fc700 1 -- 172.21.15.46:0/2289289762 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f227c003570 con 0x7f22900ab1c0 2024-01-27T01:08:12.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.331+0000 7f228e7fc700 1 -- 172.21.15.46:0/2289289762 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474442757 0 0) 0x7f22900ab5f0 con 0x7f229015d8c0 2024-01-27T01:08:12.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.331+0000 7f228e7fc700 1 -- 172.21.15.46:0/2289289762 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22900b1a60 con 0x7f229015d8c0 2024-01-27T01:08:12.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.331+0000 7f228effd700 1 -- 172.21.15.46:0/2289289762 >> v1:172.21.15.106:6789/0 conn(0x7f229015d8c0 legacy=0x7f22900b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:12.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.331+0000 7f228effd700 1 -- 172.21.15.46:0/2289289762 >> v1:172.21.15.46:6789/0 conn(0x7f22900ab1c0 legacy=0x7f22900b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:12.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.331+0000 7f228effd700 1 -- 172.21.15.46:0/2289289762 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22900b14a0 con 0x7f22901599d0 2024-01-27T01:08:12.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.331+0000 7f228effd700 1 --2- 172.21.15.46:0/2289289762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f22901599d0 0x7f2290159da0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2280009900 tx=0x7f2280009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a154ee0a985ee136 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:12.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.332+0000 7f228e7fc700 1 -- 172.21.15.46:0/2289289762 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2280005c10 con 0x7f22901599d0 2024-01-27T01:08:12.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.332+0000 7f228e7fc700 1 -- 172.21.15.46:0/2289289762 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2280005d70 con 0x7f22901599d0 2024-01-27T01:08:12.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.332+0000 7f228e7fc700 1 -- 172.21.15.46:0/2289289762 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2280006260 con 0x7f22901599d0 2024-01-27T01:08:12.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.332+0000 7f2295a17700 1 -- 172.21.15.46:0/2289289762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f22901599d0 msgr2=0x7f2290159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:12.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.332+0000 7f2295a17700 1 --2- 172.21.15.46:0/2289289762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f22901599d0 0x7f2290159da0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2280009900 tx=0x7f2280009120 comp rx=0 tx=0).stop 2024-01-27T01:08:12.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.332+0000 7f2295a17700 1 -- 172.21.15.46:0/2289289762 shutdown_connections 2024-01-27T01:08:12.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.332+0000 7f2295a17700 1 --2- 172.21.15.46:0/2289289762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f22901599d0 0x7f2290159da0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.332+0000 7f2295a17700 1 -- 172.21.15.46:0/2289289762 >> 172.21.15.46:0/2289289762 conn(0x7f229006f840 msgr2=0x7f2290087f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:12.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.332+0000 7f2295a17700 1 -- 172.21.15.46:0/2289289762 shutdown_connections 2024-01-27T01:08:12.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.332+0000 7f2295a17700 1 -- 172.21.15.46:0/2289289762 wait complete. 2024-01-27T01:08:12.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.334+0000 7f2295a17700 1 Processor -- start 2024-01-27T01:08:12.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.334+0000 7f2295a17700 1 -- start start 2024-01-27T01:08:12.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.335+0000 7f2295a17700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22900ab1c0 0x7f22900aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:12.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.335+0000 7f2295a17700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f22901599d0 0x7f2290158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:12.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.335+0000 7f2295a17700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f229015d8c0 0x7f22900ac000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:12.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.335+0000 7f2295a17700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22900aeb10 con 0x7f22900ab1c0 2024-01-27T01:08:12.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.335+0000 7f2295a17700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22900adf40 con 0x7f229015d8c0 2024-01-27T01:08:12.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.335+0000 7f2295a17700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22900ae1e0 con 0x7f22901599d0 2024-01-27T01:08:12.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.335+0000 7f228effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22900ab1c0 0x7f22900aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:12.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.335+0000 7f228effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22900ab1c0 0x7f22900aefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49398/0 (socket says 172.21.15.46:49398) 2024-01-27T01:08:12.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.335+0000 7f228effd700 1 -- 172.21.15.46:0/1508801171 learned_addr learned my addr 172.21.15.46:0/1508801171 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:12.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.335+0000 7f22867fc700 1 --2- 172.21.15.46:0/1508801171 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f22901599d0 0x7f2290158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:12.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.335+0000 7f228f7fe700 1 --2- 172.21.15.46:0/1508801171 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f229015d8c0 0x7f22900ac000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:12.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.336+0000 7f228effd700 1 -- 172.21.15.46:0/1508801171 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f22901599d0 msgr2=0x7f2290158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:12.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.336+0000 7f228effd700 1 --2- 172.21.15.46:0/1508801171 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f22901599d0 0x7f2290158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.336+0000 7f228effd700 1 -- 172.21.15.46:0/1508801171 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f229015d8c0 msgr2=0x7f22900ac000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:12.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.336+0000 7f228effd700 1 --2- 172.21.15.46:0/1508801171 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f229015d8c0 0x7f22900ac000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.336+0000 7f228effd700 1 -- 172.21.15.46:0/1508801171 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22800093c0 con 0x7f22900ab1c0 2024-01-27T01:08:12.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.336+0000 7f228effd700 1 --2- 172.21.15.46:0/1508801171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22900ab1c0 0x7f22900aefa0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f2280005d40 tx=0x7f2280016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:12.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.336+0000 7f22877fe700 1 -- 172.21.15.46:0/1508801171 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228001d800 con 0x7f22900ab1c0 2024-01-27T01:08:12.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.336+0000 7f22877fe700 1 -- 172.21.15.46:0/1508801171 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228001d960 con 0x7f22900ab1c0 2024-01-27T01:08:12.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.336+0000 7f2295a17700 1 -- 172.21.15.46:0/1508801171 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22900ae4c0 con 0x7f22900ab1c0 2024-01-27T01:08:12.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.336+0000 7f22877fe700 1 -- 172.21.15.46:0/1508801171 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228001dd20 con 0x7f22900ab1c0 2024-01-27T01:08:12.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.336+0000 7f2295a17700 1 -- 172.21.15.46:0/1508801171 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22900af4e0 con 0x7f22900ab1c0 2024-01-27T01:08:12.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.338+0000 7f2295a17700 1 -- 172.21.15.46:0/1508801171 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229004c710 con 0x7f22900ab1c0 2024-01-27T01:08:12.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.338+0000 7f22877fe700 1 -- 172.21.15.46:0/1508801171 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f228001b690 con 0x7f22900ab1c0 2024-01-27T01:08:12.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.338+0000 7f22877fe700 1 --2- 172.21.15.46:0/1508801171 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f22700670e0 0x7f2270069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:12.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.338+0000 7f22877fe700 1 -- 172.21.15.46:0/1508801171 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6611+0+0 (secure 0 0 0) 0x7f2280086760 con 0x7f22900ab1c0 2024-01-27T01:08:12.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.338+0000 7f22867fc700 1 --2- 172.21.15.46:0/1508801171 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f22700670e0 0x7f2270069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:12.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.339+0000 7f22867fc700 1 --2- 172.21.15.46:0/1508801171 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f22700670e0 0x7f2270069590 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2278007fd0 tx=0x7f2278009040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:12.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.343+0000 7f22877fe700 1 -- 172.21.15.46:0/1508801171 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f22800533c0 con 0x7f22900ab1c0 2024-01-27T01:08:12.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.569+0000 7f2295a17700 1 -- 172.21.15.46:0/1508801171 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f22900b1780 con 0x7f22900ab1c0 2024-01-27T01:08:12.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.573+0000 7f22877fe700 1 -- 172.21.15.46:0/1508801171 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v85) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2280009220 con 0x7f22900ab1c0 2024-01-27T01:08:12.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.575+0000 7f2295a17700 1 -- 172.21.15.46:0/1508801171 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f22700670e0 msgr2=0x7f2270069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:12.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.575+0000 7f2295a17700 1 --2- 172.21.15.46:0/1508801171 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f22700670e0 0x7f2270069590 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2278007fd0 tx=0x7f2278009040 comp rx=0 tx=0).stop 2024-01-27T01:08:12.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.575+0000 7f2295a17700 1 -- 172.21.15.46:0/1508801171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22900ab1c0 msgr2=0x7f22900aefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.575+0000 7f2295a17700 1 --2- 172.21.15.46:0/1508801171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22900ab1c0 0x7f22900aefa0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f2280005d40 tx=0x7f2280016000 comp rx=0 tx=0).stop 2024-01-27T01:08:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.575+0000 7f2295a17700 1 -- 172.21.15.46:0/1508801171 shutdown_connections 2024-01-27T01:08:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.575+0000 7f2295a17700 1 --2- 172.21.15.46:0/1508801171 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f22700670e0 0x7f2270069590 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.575+0000 7f2295a17700 1 --2- 172.21.15.46:0/1508801171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22900ab1c0 0x7f22900aefa0 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.575+0000 7f2295a17700 1 --2- 172.21.15.46:0/1508801171 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f229015d8c0 0x7f22900ac000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.575+0000 7f2295a17700 1 --2- 172.21.15.46:0/1508801171 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f22901599d0 0x7f2290158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.575+0000 7f2295a17700 1 -- 172.21.15.46:0/1508801171 >> 172.21.15.46:0/1508801171 conn(0x7f229006f840 msgr2=0x7f229015a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.575+0000 7f2295a17700 1 -- 172.21.15.46:0/1508801171 shutdown_connections 2024-01-27T01:08:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.575+0000 7f2295a17700 1 -- 172.21.15.46:0/1508801171 wait complete. 2024-01-27T01:08:12.579 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:12.579+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:12.588 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-27T01:08:12.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.733+0000 7f03da50c700 1 Processor -- start 2024-01-27T01:08:12.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.733+0000 7f03da50c700 1 -- start start 2024-01-27T01:08:12.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.734+0000 7f03da50c700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f03d4068b00 0x7f03d4068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:12.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.734+0000 7f03da50c700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d40a9a80 con 0x7f03d406a660 2024-01-27T01:08:12.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.734+0000 7f03da50c700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d406aab0 con 0x7f03d406add0 2024-01-27T01:08:12.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.734+0000 7f03da50c700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d415a680 con 0x7f03d4068b00 2024-01-27T01:08:12.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.734+0000 7f03d37fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f03d406a660 0x7f03d4159ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44952/0 (socket says 172.21.15.46:44952) 2024-01-27T01:08:12.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.734+0000 7f03d37fe700 1 -- 172.21.15.46:0/4096213869 learned_addr learned my addr 172.21.15.46:0/4096213869 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:12.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.735+0000 7f03d3fff700 1 --2- 172.21.15.46:0/4096213869 >> v2:172.21.15.106:3300/0 conn(0x7f03d4068b00 0x7f03d4068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:12.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.735+0000 7f03d2ffd700 1 -- 172.21.15.46:0/4096213869 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419299001 0 0) 0x7f03d406aab0 con 0x7f03d406add0 2024-01-27T01:08:12.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.735+0000 7f03d2ffd700 1 -- 172.21.15.46:0/4096213869 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03c0003570 con 0x7f03d406add0 2024-01-27T01:08:12.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.735+0000 7f03d2ffd700 1 -- 172.21.15.46:0/4096213869 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909089872 0 0) 0x7f03d40a9a80 con 0x7f03d406a660 2024-01-27T01:08:12.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.735+0000 7f03d2ffd700 1 -- 172.21.15.46:0/4096213869 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03d406aab0 con 0x7f03d406a660 2024-01-27T01:08:12.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.735+0000 7f03d3fff700 1 -- 172.21.15.46:0/4096213869 >> v1:172.21.15.106:6789/0 conn(0x7f03d406add0 legacy=0x7f03d40a9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:12.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.735+0000 7f03d3fff700 1 -- 172.21.15.46:0/4096213869 >> v1:172.21.15.46:6789/0 conn(0x7f03d406a660 legacy=0x7f03d4159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:12.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.735+0000 7f03d3fff700 1 -- 172.21.15.46:0/4096213869 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d415a960 con 0x7f03d4068b00 2024-01-27T01:08:12.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.735+0000 7f03d3fff700 1 --2- 172.21.15.46:0/4096213869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03d4068b00 0x7f03d4068ed0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f03c40098d0 tx=0x7f03c4009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bd1bc1a5de24d59 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:12.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.735+0000 7f03d2ffd700 1 -- 172.21.15.46:0/4096213869 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4005c10 con 0x7f03d4068b00 2024-01-27T01:08:12.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.735+0000 7f03d2ffd700 1 -- 172.21.15.46:0/4096213869 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c4005d70 con 0x7f03d4068b00 2024-01-27T01:08:12.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.735+0000 7f03d2ffd700 1 -- 172.21.15.46:0/4096213869 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4006280 con 0x7f03d4068b00 2024-01-27T01:08:12.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.736+0000 7f03da50c700 1 -- 172.21.15.46:0/4096213869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03d4068b00 msgr2=0x7f03d4068ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:12.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.736+0000 7f03da50c700 1 --2- 172.21.15.46:0/4096213869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03d4068b00 0x7f03d4068ed0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f03c40098d0 tx=0x7f03c4009120 comp rx=0 tx=0).stop 2024-01-27T01:08:12.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.736+0000 7f03da50c700 1 -- 172.21.15.46:0/4096213869 shutdown_connections 2024-01-27T01:08:12.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.736+0000 7f03da50c700 1 --2- 172.21.15.46:0/4096213869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03d4068b00 0x7f03d4068ed0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.736+0000 7f03da50c700 1 -- 172.21.15.46:0/4096213869 >> 172.21.15.46:0/4096213869 conn(0x7f03d40ca100 msgr2=0x7f03d406e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:12.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.736+0000 7f03da50c700 1 -- 172.21.15.46:0/4096213869 shutdown_connections 2024-01-27T01:08:12.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.736+0000 7f03da50c700 1 -- 172.21.15.46:0/4096213869 wait complete. 2024-01-27T01:08:12.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.738+0000 7f03da50c700 1 Processor -- start 2024-01-27T01:08:12.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.738+0000 7f03da50c700 1 -- start start 2024-01-27T01:08:12.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.739+0000 7f03da50c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f03d4068b00 0x7f03d40c6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:12.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.739+0000 7f03da50c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03d406a660 0x7f03d414cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:12.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.739+0000 7f03da50c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d406add0 0x7f03d40c47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:12.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.739+0000 7f03da50c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d40ad940 con 0x7f03d406add0 2024-01-27T01:08:12.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.739+0000 7f03da50c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d40c6d40 con 0x7f03d406a660 2024-01-27T01:08:12.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.739+0000 7f03da50c700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d40ac860 con 0x7f03d4068b00 2024-01-27T01:08:12.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.739+0000 7f03d8aa9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d406add0 0x7f03d40c47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:12.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.739+0000 7f03d8aa9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d406add0 0x7f03d40c47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49408/0 (socket says 172.21.15.46:49408) 2024-01-27T01:08:12.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.739+0000 7f03d8aa9700 1 -- 172.21.15.46:0/4147711761 learned_addr learned my addr 172.21.15.46:0/4147711761 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:12.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.739+0000 7f03d37fe700 1 --2- 172.21.15.46:0/4147711761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03d406a660 0x7f03d414cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.739+0000 7f03d3fff700 1 --2- 172.21.15.46:0/4147711761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f03d4068b00 0x7f03d40c6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.740+0000 7f03d8aa9700 1 -- 172.21.15.46:0/4147711761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f03d4068b00 msgr2=0x7f03d40c6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.740+0000 7f03d8aa9700 1 --2- 172.21.15.46:0/4147711761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f03d4068b00 0x7f03d40c6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.740+0000 7f03d8aa9700 1 -- 172.21.15.46:0/4147711761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03d406a660 msgr2=0x7f03d414cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.740+0000 7f03d8aa9700 1 --2- 172.21.15.46:0/4147711761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03d406a660 0x7f03d414cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.740+0000 7f03d8aa9700 1 -- 172.21.15.46:0/4147711761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c40093c0 con 0x7f03d406add0 2024-01-27T01:08:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.740+0000 7f03d37fe700 1 --2- 172.21.15.46:0/4147711761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03d406a660 0x7f03d414cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.740+0000 7f03d3fff700 1 --2- 172.21.15.46:0/4147711761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f03d4068b00 0x7f03d40c6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.740+0000 7f03d8aa9700 1 --2- 172.21.15.46:0/4147711761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d406add0 0x7f03d40c47a0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00c910 tx=0x7f03cc0027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.740+0000 7f03cbfff700 1 -- 172.21.15.46:0/4147711761 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc012630 con 0x7f03d406add0 2024-01-27T01:08:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.740+0000 7f03cbfff700 1 -- 172.21.15.46:0/4147711761 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03cc0127b0 con 0x7f03d406add0 2024-01-27T01:08:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.740+0000 7f03da50c700 1 -- 172.21.15.46:0/4147711761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d40ad010 con 0x7f03d406add0 2024-01-27T01:08:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.740+0000 7f03cbfff700 1 -- 172.21.15.46:0/4147711761 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc012b70 con 0x7f03d406add0 2024-01-27T01:08:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.740+0000 7f03da50c700 1 -- 172.21.15.46:0/4147711761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d40ad5d0 con 0x7f03d406add0 2024-01-27T01:08:12.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.741+0000 7f03da50c700 1 -- 172.21.15.46:0/4147711761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d40c53d0 con 0x7f03d406add0 2024-01-27T01:08:12.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.742+0000 7f03cbfff700 1 -- 172.21.15.46:0/4147711761 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f03cc01b400 con 0x7f03d406add0 2024-01-27T01:08:12.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.742+0000 7f03cbfff700 1 --2- 172.21.15.46:0/4147711761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f03b40670e0 0x7f03b4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:12.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.742+0000 7f03cbfff700 1 -- 172.21.15.46:0/4147711761 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6611+0+0 (secure 0 0 0) 0x7f03cc084fd0 con 0x7f03d406add0 2024-01-27T01:08:12.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.742+0000 7f03d3fff700 1 --2- 172.21.15.46:0/4147711761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f03b40670e0 0x7f03b4069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:12.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.743+0000 7f03d3fff700 1 --2- 172.21.15.46:0/4147711761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f03b40670e0 0x7f03b4069590 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009390 tx=0x7f03c4016000 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:12.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.747+0000 7f03cbfff700 1 -- 172.21.15.46:0/4147711761 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f03cc051b90 con 0x7f03d406add0 2024-01-27T01:08:12.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.957+0000 7f03da50c700 1 -- 172.21.15.46:0/4147711761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f03d404c710 con 0x7f03d406add0 2024-01-27T01:08:12.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.961+0000 7f03cbfff700 1 -- 172.21.15.46:0/4147711761 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v86) v1 ==== 120+0+0 (secure 0 0 0) 0x7f03cc0551b0 con 0x7f03d406add0 2024-01-27T01:08:12.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.964+0000 7f03da50c700 1 -- 172.21.15.46:0/4147711761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f03b40670e0 msgr2=0x7f03b4069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:12.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.964+0000 7f03da50c700 1 --2- 172.21.15.46:0/4147711761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f03b40670e0 0x7f03b4069590 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009390 tx=0x7f03c4016000 comp rx=0 tx=0).stop 2024-01-27T01:08:12.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.964+0000 7f03da50c700 1 -- 172.21.15.46:0/4147711761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d406add0 msgr2=0x7f03d40c47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:12.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.964+0000 7f03da50c700 1 --2- 172.21.15.46:0/4147711761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d406add0 0x7f03d40c47a0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00c910 tx=0x7f03cc0027a0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.964+0000 7f03da50c700 1 -- 172.21.15.46:0/4147711761 shutdown_connections 2024-01-27T01:08:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.964+0000 7f03da50c700 1 --2- 172.21.15.46:0/4147711761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f03b40670e0 0x7f03b4069590 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.964+0000 7f03da50c700 1 --2- 172.21.15.46:0/4147711761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d406add0 0x7f03d40c47a0 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.964+0000 7f03da50c700 1 --2- 172.21.15.46:0/4147711761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f03d4068b00 0x7f03d40c6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.964+0000 7f03da50c700 1 --2- 172.21.15.46:0/4147711761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03d406a660 0x7f03d414cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.964+0000 7f03da50c700 1 -- 172.21.15.46:0/4147711761 >> 172.21.15.46:0/4147711761 conn(0x7f03d40ca100 msgr2=0x7f03d40aee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.964+0000 7f03da50c700 1 -- 172.21.15.46:0/4147711761 shutdown_connections 2024-01-27T01:08:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:12.964+0000 7f03da50c700 1 -- 172.21.15.46:0/4147711761 wait complete. 2024-01-27T01:08:12.977 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-27T01:08:13.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.114+0000 7f9635b6b700 1 Processor -- start 2024-01-27T01:08:13.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.114+0000 7f9635b6b700 1 -- start start 2024-01-27T01:08:13.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.114+0000 7f9635b6b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f96300c8da0 0x7f96300c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:13.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.114+0000 7f9635b6b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96300ccc00 con 0x7f9630140af0 2024-01-27T01:08:13.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.114+0000 7f9635b6b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96300cc310 con 0x7f96300caf30 2024-01-27T01:08:13.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.114+0000 7f9635b6b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96300cbbf0 con 0x7f96300c8da0 2024-01-27T01:08:13.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.115+0000 7f9626ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9630140af0 0x7f9630140f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44956/0 (socket says 172.21.15.46:44956) 2024-01-27T01:08:13.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.115+0000 7f9626ffd700 1 -- 172.21.15.46:0/1973622019 learned_addr learned my addr 172.21.15.46:0/1973622019 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:13.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.115+0000 7f962f7fe700 1 --2- 172.21.15.46:0/1973622019 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f96300c8da0 0x7f96300c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:13.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.115+0000 7f962effd700 1 -- 172.21.15.46:0/1973622019 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646759222 0 0) 0x7f96300ccc00 con 0x7f9630140af0 2024-01-27T01:08:13.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.115+0000 7f962effd700 1 -- 172.21.15.46:0/1973622019 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f961c003570 con 0x7f9630140af0 2024-01-27T01:08:13.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.115+0000 7f962effd700 1 -- 172.21.15.46:0/1973622019 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347495016 0 0) 0x7f96300cc310 con 0x7f96300caf30 2024-01-27T01:08:13.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.115+0000 7f962effd700 1 -- 172.21.15.46:0/1973622019 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96300ccc00 con 0x7f96300caf30 2024-01-27T01:08:13.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.115+0000 7f962f7fe700 1 -- 172.21.15.46:0/1973622019 >> v1:172.21.15.106:6789/0 conn(0x7f96300caf30 legacy=0x7f96300cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:13.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.115+0000 7f962f7fe700 1 -- 172.21.15.46:0/1973622019 >> v1:172.21.15.46:6789/0 conn(0x7f9630140af0 legacy=0x7f9630140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:13.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.115+0000 7f962f7fe700 1 -- 172.21.15.46:0/1973622019 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96300cc5b0 con 0x7f96300c8da0 2024-01-27T01:08:13.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.115+0000 7f962f7fe700 1 --2- 172.21.15.46:0/1973622019 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f96300c8da0 0x7f96300c9170 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9620009910 tx=0x7f96200091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92c9f800c84c9567 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:13.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.115+0000 7f962effd700 1 -- 172.21.15.46:0/1973622019 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9620011630 con 0x7f96300c8da0 2024-01-27T01:08:13.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.116+0000 7f962effd700 1 -- 172.21.15.46:0/1973622019 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9620011790 con 0x7f96300c8da0 2024-01-27T01:08:13.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.116+0000 7f962effd700 1 -- 172.21.15.46:0/1973622019 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9620011b30 con 0x7f96300c8da0 2024-01-27T01:08:13.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.116+0000 7f9635b6b700 1 -- 172.21.15.46:0/1973622019 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f96300c8da0 msgr2=0x7f96300c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:13.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.116+0000 7f9635b6b700 1 --2- 172.21.15.46:0/1973622019 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f96300c8da0 0x7f96300c9170 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9620009910 tx=0x7f96200091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:13.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.116+0000 7f9635b6b700 1 -- 172.21.15.46:0/1973622019 shutdown_connections 2024-01-27T01:08:13.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.116+0000 7f9635b6b700 1 --2- 172.21.15.46:0/1973622019 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f96300c8da0 0x7f96300c9170 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:13.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.116+0000 7f9635b6b700 1 -- 172.21.15.46:0/1973622019 >> 172.21.15.46:0/1973622019 conn(0x7f963006bd90 msgr2=0x7f963006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:13.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.116+0000 7f9635b6b700 1 -- 172.21.15.46:0/1973622019 shutdown_connections 2024-01-27T01:08:13.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.116+0000 7f9635b6b700 1 -- 172.21.15.46:0/1973622019 wait complete. 2024-01-27T01:08:13.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.117+0000 7f9635b6b700 1 Processor -- start 2024-01-27T01:08:13.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.118+0000 7f9635b6b700 1 -- start start 2024-01-27T01:08:13.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.118+0000 7f9635b6b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96300c8da0 0x7f9630069930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:13.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.118+0000 7f9635b6b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f96300caf30 0x7f9630156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:13.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.118+0000 7f9635b6b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9630140af0 0x7f96300c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:13.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.118+0000 7f9635b6b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300c6950 con 0x7f96300c8da0 2024-01-27T01:08:13.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.118+0000 7f9635b6b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300b2bc0 con 0x7f96300caf30 2024-01-27T01:08:13.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.118+0000 7f9635b6b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963006a870 con 0x7f9630140af0 2024-01-27T01:08:13.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.119+0000 7f962f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96300c8da0 0x7f9630069930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:13.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.119+0000 7f9626ffd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f96300caf30 0x7f9630156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:13.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.119+0000 7f962f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96300c8da0 0x7f9630069930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49412/0 (socket says 172.21.15.46:49412) 2024-01-27T01:08:13.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.119+0000 7f962f7fe700 1 -- 172.21.15.46:0/2675868663 learned_addr learned my addr 172.21.15.46:0/2675868663 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:13.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.119+0000 7f962ffff700 1 --2- 172.21.15.46:0/2675868663 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9630140af0 0x7f96300c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.119+0000 7f9626ffd700 1 -- 172.21.15.46:0/2675868663 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9630140af0 msgr2=0x7f96300c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.119+0000 7f9626ffd700 1 --2- 172.21.15.46:0/2675868663 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9630140af0 0x7f96300c6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.119+0000 7f9626ffd700 1 -- 172.21.15.46:0/2675868663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96300c8da0 msgr2=0x7f9630069930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.119+0000 7f9626ffd700 1 --2- 172.21.15.46:0/2675868663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96300c8da0 0x7f9630069930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.119+0000 7f9626ffd700 1 -- 172.21.15.46:0/2675868663 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9620009420 con 0x7f96300caf30 2024-01-27T01:08:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.119+0000 7f962f7fe700 1 --2- 172.21.15.46:0/2675868663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96300c8da0 0x7f9630069930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.119+0000 7f9626ffd700 1 --2- 172.21.15.46:0/2675868663 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f96300caf30 0x7f9630156fa0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f961800c560 tx=0x7f9618007d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.119+0000 7f9627fff700 1 -- 172.21.15.46:0/2675868663 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618005b30 con 0x7f96300caf30 2024-01-27T01:08:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.119+0000 7f9627fff700 1 -- 172.21.15.46:0/2675868663 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9618005cb0 con 0x7f96300caf30 2024-01-27T01:08:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.119+0000 7f9627fff700 1 -- 172.21.15.46:0/2675868663 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618007320 con 0x7f96300caf30 2024-01-27T01:08:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.119+0000 7f9635b6b700 1 -- 172.21.15.46:0/2675868663 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96300c6390 con 0x7f96300caf30 2024-01-27T01:08:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.120+0000 7f9635b6b700 1 -- 172.21.15.46:0/2675868663 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96300b2850 con 0x7f96300caf30 2024-01-27T01:08:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.120+0000 7f9635b6b700 1 -- 172.21.15.46:0/2675868663 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96300b26b0 con 0x7f96300caf30 2024-01-27T01:08:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.121+0000 7f9627fff700 1 -- 172.21.15.46:0/2675868663 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f961801b3b0 con 0x7f96300caf30 2024-01-27T01:08:13.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.121+0000 7f9627fff700 1 --2- 172.21.15.46:0/2675868663 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f96100670e0 0x7f9610069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:13.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.121+0000 7f9627fff700 1 -- 172.21.15.46:0/2675868663 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6611+0+0 (secure 0 0 0) 0x7f9618083f40 con 0x7f96300caf30 2024-01-27T01:08:13.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.121+0000 7f962f7fe700 1 --2- 172.21.15.46:0/2675868663 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f96100670e0 0x7f9610069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:13.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.122+0000 7f962f7fe700 1 --2- 172.21.15.46:0/2675868663 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f96100670e0 0x7f9610069590 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f96200093a0 tx=0x7f9620017000 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:13.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.126+0000 7f9627fff700 1 -- 172.21.15.46:0/2675868663 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f96300b26b0 con 0x7f96300caf30 2024-01-27T01:08:13.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.333+0000 7f9635b6b700 1 -- 172.21.15.46:0/2675868663 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f96300b26b0 con 0x7f96300caf30 2024-01-27T01:08:13.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.338+0000 7f9627fff700 1 -- 172.21.15.46:0/2675868663 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v87) v1 ==== 115+0+0 (secure 0 0 0) 0x7f96300b26b0 con 0x7f96300caf30 2024-01-27T01:08:13.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.340+0000 7f9635b6b700 1 -- 172.21.15.46:0/2675868663 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f96100670e0 msgr2=0x7f9610069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.340+0000 7f9635b6b700 1 --2- 172.21.15.46:0/2675868663 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f96100670e0 0x7f9610069590 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f96200093a0 tx=0x7f9620017000 comp rx=0 tx=0).stop 2024-01-27T01:08:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.340+0000 7f9635b6b700 1 -- 172.21.15.46:0/2675868663 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f96300caf30 msgr2=0x7f9630156fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.340+0000 7f9635b6b700 1 --2- 172.21.15.46:0/2675868663 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f96300caf30 0x7f9630156fa0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f961800c560 tx=0x7f9618007d70 comp rx=0 tx=0).stop 2024-01-27T01:08:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.340+0000 7f9635b6b700 1 -- 172.21.15.46:0/2675868663 shutdown_connections 2024-01-27T01:08:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.340+0000 7f9635b6b700 1 --2- 172.21.15.46:0/2675868663 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f96100670e0 0x7f9610069590 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.340+0000 7f9635b6b700 1 --2- 172.21.15.46:0/2675868663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96300c8da0 0x7f9630069930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.340+0000 7f9635b6b700 1 --2- 172.21.15.46:0/2675868663 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9630140af0 0x7f96300c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.340+0000 7f9635b6b700 1 --2- 172.21.15.46:0/2675868663 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f96300caf30 0x7f9630156fa0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.340+0000 7f9635b6b700 1 -- 172.21.15.46:0/2675868663 >> 172.21.15.46:0/2675868663 conn(0x7f963006bd90 msgr2=0x7f963013f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.340+0000 7f9635b6b700 1 -- 172.21.15.46:0/2675868663 shutdown_connections 2024-01-27T01:08:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.340+0000 7f9635b6b700 1 -- 172.21.15.46:0/2675868663 wait complete. 2024-01-27T01:08:13.355 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-01-27T01:08:13.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.490+0000 7f47cd3e9700 1 Processor -- start 2024-01-27T01:08:13.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.490+0000 7f47cd3e9700 1 -- start start 2024-01-27T01:08:13.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.491+0000 7f47cd3e9700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f47c80ad810 0x7f47c80ae800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:13.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.491+0000 7f47cd3e9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f47c80adbe0 0x7f47c80c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:13.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.491+0000 7f47cd3e9700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c80cbf30 con 0x7f47c80b1670 2024-01-27T01:08:13.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.491+0000 7f47cd3e9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c80aeea0 con 0x7f47c80adbe0 2024-01-27T01:08:13.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.491+0000 7f47cd3e9700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c80cb400 con 0x7f47c80ad810 2024-01-27T01:08:13.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.491+0000 7f47bffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f47c80adbe0 0x7f47c80c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:13.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.491+0000 7f47bffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f47c80adbe0 0x7f47c80c8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49422/0 (socket says 172.21.15.46:49422) 2024-01-27T01:08:13.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.491+0000 7f47bffff700 1 -- 172.21.15.46:0/3392868692 learned_addr learned my addr 172.21.15.46:0/3392868692 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:13.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.491+0000 7f47c6ffd700 1 --2- 172.21.15.46:0/3392868692 >> v2:172.21.15.106:3300/0 conn(0x7f47c80ad810 0x7f47c80ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:13.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.492+0000 7f47bffff700 1 -- 172.21.15.46:0/3392868692 >> v2:172.21.15.106:3300/0 conn(0x7f47c80ad810 msgr2=0x7f47c80ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:13.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.492+0000 7f47bffff700 1 --2- 172.21.15.46:0/3392868692 >> v2:172.21.15.106:3300/0 conn(0x7f47c80ad810 0x7f47c80ae800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:13.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.492+0000 7f47bffff700 1 -- 172.21.15.46:0/3392868692 >> v1:172.21.15.106:6789/0 conn(0x7f47c80b1670 legacy=0x7f47c815b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:13.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.492+0000 7f47bffff700 1 -- 172.21.15.46:0/3392868692 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c80c9520 con 0x7f47c80adbe0 2024-01-27T01:08:13.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.492+0000 7f47bffff700 1 --2- 172.21.15.46:0/3392868692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c80adbe0 0x7f47c80c8fe0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f47b00099b0 tx=0x7f47b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ff07e1fac4bedf server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:13.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.492+0000 7f47c67fc700 1 -- 172.21.15.46:0/3392868692 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b0009ea0 con 0x7f47c80adbe0 2024-01-27T01:08:13.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.492+0000 7f47c67fc700 1 -- 172.21.15.46:0/3392868692 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b000b820 con 0x7f47c80adbe0 2024-01-27T01:08:13.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.492+0000 7f47c67fc700 1 -- 172.21.15.46:0/3392868692 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b000bbe0 con 0x7f47c80adbe0 2024-01-27T01:08:13.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.493+0000 7f47cd3e9700 1 -- 172.21.15.46:0/3392868692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c80adbe0 msgr2=0x7f47c80c8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:13.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.493+0000 7f47cd3e9700 1 --2- 172.21.15.46:0/3392868692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c80adbe0 0x7f47c80c8fe0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f47b00099b0 tx=0x7f47b0009250 comp rx=0 tx=0).stop 2024-01-27T01:08:13.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.493+0000 7f47cd3e9700 1 -- 172.21.15.46:0/3392868692 shutdown_connections 2024-01-27T01:08:13.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.493+0000 7f47cd3e9700 1 --2- 172.21.15.46:0/3392868692 >> v2:172.21.15.106:3300/0 conn(0x7f47c80ad810 0x7f47c80ae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:13.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.493+0000 7f47cd3e9700 1 --2- 172.21.15.46:0/3392868692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c80adbe0 0x7f47c80c8fe0 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:13.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.493+0000 7f47cd3e9700 1 -- 172.21.15.46:0/3392868692 >> 172.21.15.46:0/3392868692 conn(0x7f47c8068400 msgr2=0x7f47c80cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:13.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.493+0000 7f47cd3e9700 1 -- 172.21.15.46:0/3392868692 shutdown_connections 2024-01-27T01:08:13.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.493+0000 7f47cd3e9700 1 -- 172.21.15.46:0/3392868692 wait complete. 2024-01-27T01:08:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.494+0000 7f47cd3e9700 1 Processor -- start 2024-01-27T01:08:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.494+0000 7f47cd3e9700 1 -- start start 2024-01-27T01:08:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.495+0000 7f47cd3e9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c80ad810 0x7f47c8146e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.495+0000 7f47cd3e9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47c80adbe0 0x7f47c81439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.495+0000 7f47cd3e9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47c80b1670 0x7f47c80c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.495+0000 7f47cd3e9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c81469f0 con 0x7f47c80ad810 2024-01-27T01:08:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.495+0000 7f47cd3e9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c8145de0 con 0x7f47c80b1670 2024-01-27T01:08:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.495+0000 7f47cd3e9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c81460c0 con 0x7f47c80adbe0 2024-01-27T01:08:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.496+0000 7f47c6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c80ad810 0x7f47c8146e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.496+0000 7f47c6ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c80ad810 0x7f47c8146e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49434/0 (socket says 172.21.15.46:49434) 2024-01-27T01:08:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.496+0000 7f47c6ffd700 1 -- 172.21.15.46:0/731815907 learned_addr learned my addr 172.21.15.46:0/731815907 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.496+0000 7f47bffff700 1 --2- 172.21.15.46:0/731815907 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47c80adbe0 0x7f47c81439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.496+0000 7f47c77fe700 1 --2- 172.21.15.46:0/731815907 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47c80b1670 0x7f47c80c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.496+0000 7f47c6ffd700 1 -- 172.21.15.46:0/731815907 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47c80adbe0 msgr2=0x7f47c81439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.496+0000 7f47c6ffd700 1 --2- 172.21.15.46:0/731815907 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47c80adbe0 0x7f47c81439b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.496+0000 7f47c6ffd700 1 -- 172.21.15.46:0/731815907 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47c80b1670 msgr2=0x7f47c80c76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.496+0000 7f47c6ffd700 1 --2- 172.21.15.46:0/731815907 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47c80b1670 0x7f47c80c76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.496+0000 7f47c6ffd700 1 -- 172.21.15.46:0/731815907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b00094f0 con 0x7f47c80ad810 2024-01-27T01:08:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.496+0000 7f47c6ffd700 1 --2- 172.21.15.46:0/731815907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c80ad810 0x7f47c8146e80 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f47b800e930 tx=0x7f47b80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.496+0000 7f47beffd700 1 -- 172.21.15.46:0/731815907 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b8013680 con 0x7f47c80ad810 2024-01-27T01:08:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.496+0000 7f47beffd700 1 -- 172.21.15.46:0/731815907 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b80137e0 con 0x7f47c80ad810 2024-01-27T01:08:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.496+0000 7f47cd3e9700 1 -- 172.21.15.46:0/731815907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c81463a0 con 0x7f47c80ad810 2024-01-27T01:08:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.496+0000 7f47beffd700 1 -- 172.21.15.46:0/731815907 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b8013b80 con 0x7f47c80ad810 2024-01-27T01:08:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.496+0000 7f47cd3e9700 1 -- 172.21.15.46:0/731815907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c8143ef0 con 0x7f47c80ad810 2024-01-27T01:08:13.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.498+0000 7f47cd3e9700 1 -- 172.21.15.46:0/731815907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c804c710 con 0x7f47c80ad810 2024-01-27T01:08:13.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.498+0000 7f47beffd700 1 -- 172.21.15.46:0/731815907 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f47b800c3e0 con 0x7f47c80ad810 2024-01-27T01:08:13.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.498+0000 7f47beffd700 1 --2- 172.21.15.46:0/731815907 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f47a80670e0 0x7f47a8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:13.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.498+0000 7f47beffd700 1 -- 172.21.15.46:0/731815907 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6611+0+0 (secure 0 0 0) 0x7f47b8084c80 con 0x7f47c80ad810 2024-01-27T01:08:13.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.499+0000 7f47bffff700 1 --2- 172.21.15.46:0/731815907 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f47a80670e0 0x7f47a8069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:13.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.499+0000 7f47bffff700 1 --2- 172.21.15.46:0/731815907 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f47a80670e0 0x7f47a8069590 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f47b00099b0 tx=0x7f47b0017000 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:13.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.503+0000 7f47beffd700 1 -- 172.21.15.46:0/731815907 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f47b80518e0 con 0x7f47c80ad810 2024-01-27T01:08:13.586 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:13.554+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:13.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:13.794+0000 7f47cd3e9700 1 -- 172.21.15.46:0/731815907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7f47c8146cd0 con 0x7f47c80ad810 2024-01-27T01:08:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.173+0000 7f47beffd700 1 -- 172.21.15.46:0/731815907 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3244) v1 ==== 187+0+0 (secure 0 0 0) 0x7f47b8054f00 con 0x7f47c80ad810 2024-01-27T01:08:14.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.176+0000 7f47cd3e9700 1 -- 172.21.15.46:0/731815907 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f47a80670e0 msgr2=0x7f47a8069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:14.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.176+0000 7f47cd3e9700 1 --2- 172.21.15.46:0/731815907 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f47a80670e0 0x7f47a8069590 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f47b00099b0 tx=0x7f47b0017000 comp rx=0 tx=0).stop 2024-01-27T01:08:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.176+0000 7f47cd3e9700 1 -- 172.21.15.46:0/731815907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c80ad810 msgr2=0x7f47c8146e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.176+0000 7f47cd3e9700 1 --2- 172.21.15.46:0/731815907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c80ad810 0x7f47c8146e80 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f47b800e930 tx=0x7f47b80091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.176+0000 7f47cd3e9700 1 -- 172.21.15.46:0/731815907 shutdown_connections 2024-01-27T01:08:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.176+0000 7f47cd3e9700 1 --2- 172.21.15.46:0/731815907 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f47a80670e0 0x7f47a8069590 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.176+0000 7f47cd3e9700 1 --2- 172.21.15.46:0/731815907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c80ad810 0x7f47c8146e80 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.176+0000 7f47cd3e9700 1 --2- 172.21.15.46:0/731815907 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47c80b1670 0x7f47c80c76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.176+0000 7f47cd3e9700 1 --2- 172.21.15.46:0/731815907 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47c80adbe0 0x7f47c81439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.176+0000 7f47cd3e9700 1 -- 172.21.15.46:0/731815907 >> 172.21.15.46:0/731815907 conn(0x7f47c8068400 msgr2=0x7f47c8156f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.176+0000 7f47cd3e9700 1 -- 172.21.15.46:0/731815907 shutdown_connections 2024-01-27T01:08:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.176+0000 7f47cd3e9700 1 -- 172.21.15.46:0/731815907 wait complete. 2024-01-27T01:08:14.191 INFO:tasks.cephfs_test_runner:Test command fails without --fs argument in presence of multiple fs ... ok 2024-01-27T01:08:14.192 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2024-01-27T01:08:14.192 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2024-01-27T01:08:14.192 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-01-27T01:08:14.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.334+0000 7fba224fa700 1 Processor -- start 2024-01-27T01:08:14.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.334+0000 7fba224fa700 1 -- start start 2024-01-27T01:08:14.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.335+0000 7fba224fa700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fba1c1561d0 0x7fba1c0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:14.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.335+0000 7fba224fa700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fba1c0b0280 0x7fba1c1565a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:14.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.335+0000 7fba224fa700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba1c0b0760 con 0x7fba1c06eb70 2024-01-27T01:08:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.335+0000 7fba224fa700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba1c0b0a40 con 0x7fba1c1561d0 2024-01-27T01:08:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.335+0000 7fba224fa700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba1c0b0d20 con 0x7fba1c0b0280 2024-01-27T01:08:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.335+0000 7fba1bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fba1c1561d0 0x7fba1c0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.335+0000 7fba1bfff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fba1c1561d0 0x7fba1c0ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49440/0 (socket says 172.21.15.46:49440) 2024-01-27T01:08:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.335+0000 7fba1bfff700 1 -- 172.21.15.46:0/1763681765 learned_addr learned my addr 172.21.15.46:0/1763681765 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.335+0000 7fba1b7fe700 1 --2- 172.21.15.46:0/1763681765 >> v2:172.21.15.106:3300/0 conn(0x7fba1c0b0280 0x7fba1c1565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.335+0000 7fba1bfff700 1 -- 172.21.15.46:0/1763681765 >> v2:172.21.15.106:3300/0 conn(0x7fba1c0b0280 msgr2=0x7fba1c1565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.335+0000 7fba1bfff700 1 --2- 172.21.15.46:0/1763681765 >> v2:172.21.15.106:3300/0 conn(0x7fba1c0b0280 0x7fba1c1565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.335+0000 7fba1bfff700 1 -- 172.21.15.46:0/1763681765 >> v1:172.21.15.106:6789/0 conn(0x7fba1c06eb70 legacy=0x7fba1c06c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.335+0000 7fba1bfff700 1 -- 172.21.15.46:0/1763681765 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c0b1000 con 0x7fba1c1561d0 2024-01-27T01:08:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.336+0000 7fba1bfff700 1 --2- 172.21.15.46:0/1763681765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c1561d0 0x7fba1c0ca5f0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7fba1c06d720 tx=0x7fba0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97500d14b3d7edd3 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.336+0000 7fba1affd700 1 -- 172.21.15.46:0/1763681765 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c009e70 con 0x7fba1c1561d0 2024-01-27T01:08:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.336+0000 7fba1affd700 1 -- 172.21.15.46:0/1763681765 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0c00b860 con 0x7fba1c1561d0 2024-01-27T01:08:14.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.336+0000 7fba1affd700 1 -- 172.21.15.46:0/1763681765 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c00bc20 con 0x7fba1c1561d0 2024-01-27T01:08:14.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.336+0000 7fba224fa700 1 -- 172.21.15.46:0/1763681765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c1561d0 msgr2=0x7fba1c0ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:14.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.336+0000 7fba224fa700 1 --2- 172.21.15.46:0/1763681765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c1561d0 0x7fba1c0ca5f0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7fba1c06d720 tx=0x7fba0c009250 comp rx=0 tx=0).stop 2024-01-27T01:08:14.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.336+0000 7fba224fa700 1 -- 172.21.15.46:0/1763681765 shutdown_connections 2024-01-27T01:08:14.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.336+0000 7fba224fa700 1 --2- 172.21.15.46:0/1763681765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c1561d0 0x7fba1c0ca5f0 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:14.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.336+0000 7fba224fa700 1 --2- 172.21.15.46:0/1763681765 >> v2:172.21.15.106:3300/0 conn(0x7fba1c0b0280 0x7fba1c1565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:14.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.336+0000 7fba224fa700 1 -- 172.21.15.46:0/1763681765 >> 172.21.15.46:0/1763681765 conn(0x7fba1c06b630 msgr2=0x7fba1c0aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:14.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.336+0000 7fba224fa700 1 -- 172.21.15.46:0/1763681765 shutdown_connections 2024-01-27T01:08:14.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.336+0000 7fba224fa700 1 -- 172.21.15.46:0/1763681765 wait complete. 2024-01-27T01:08:14.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.338+0000 7fba224fa700 1 Processor -- start 2024-01-27T01:08:14.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.338+0000 7fba224fa700 1 -- start start 2024-01-27T01:08:14.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.339+0000 7fba224fa700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba1c06eb70 0x7fba1c1309a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:14.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.339+0000 7fba224fa700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c0b0280 0x7fba1c0adf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:14.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.339+0000 7fba224fa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c1561d0 0x7fba1c0ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:14.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.339+0000 7fba224fa700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c12e030 con 0x7fba1c1561d0 2024-01-27T01:08:14.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.339+0000 7fba224fa700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c1478d0 con 0x7fba1c06eb70 2024-01-27T01:08:14.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.339+0000 7fba224fa700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c146d50 con 0x7fba1c0b0280 2024-01-27T01:08:14.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.339+0000 7fba20a97700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c1561d0 0x7fba1c0ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:14.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.339+0000 7fba20a97700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c1561d0 0x7fba1c0ae4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49444/0 (socket says 172.21.15.46:49444) 2024-01-27T01:08:14.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.339+0000 7fba20a97700 1 -- 172.21.15.46:0/2154125007 learned_addr learned my addr 172.21.15.46:0/2154125007 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:14.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.339+0000 7fba1bfff700 1 --2- 172.21.15.46:0/2154125007 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba1c06eb70 0x7fba1c1309a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:14.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.339+0000 7fba1b7fe700 1 --2- 172.21.15.46:0/2154125007 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c0b0280 0x7fba1c0adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:14.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.339+0000 7fba20a97700 1 -- 172.21.15.46:0/2154125007 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c0b0280 msgr2=0x7fba1c0adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:14.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.339+0000 7fba20a97700 1 --2- 172.21.15.46:0/2154125007 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c0b0280 0x7fba1c0adf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:14.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.339+0000 7fba20a97700 1 -- 172.21.15.46:0/2154125007 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba1c06eb70 msgr2=0x7fba1c1309a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:14.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.339+0000 7fba20a97700 1 --2- 172.21.15.46:0/2154125007 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba1c06eb70 0x7fba1c1309a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:14.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.339+0000 7fba20a97700 1 -- 172.21.15.46:0/2154125007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0c0094f0 con 0x7fba1c1561d0 2024-01-27T01:08:14.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.340+0000 7fba20a97700 1 --2- 172.21.15.46:0/2154125007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c1561d0 0x7fba1c0ae4a0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7fba1400c8e0 tx=0x7fba14001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:14.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.340+0000 7fba03fff700 1 -- 172.21.15.46:0/2154125007 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba14012630 con 0x7fba1c1561d0 2024-01-27T01:08:14.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.340+0000 7fba03fff700 1 -- 172.21.15.46:0/2154125007 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba140127b0 con 0x7fba1c1561d0 2024-01-27T01:08:14.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.340+0000 7fba224fa700 1 -- 172.21.15.46:0/2154125007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba1c147030 con 0x7fba1c1561d0 2024-01-27T01:08:14.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.340+0000 7fba03fff700 1 -- 172.21.15.46:0/2154125007 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba14012b70 con 0x7fba1c1561d0 2024-01-27T01:08:14.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.340+0000 7fba224fa700 1 -- 172.21.15.46:0/2154125007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1c1475f0 con 0x7fba1c1561d0 2024-01-27T01:08:14.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.341+0000 7fba224fa700 1 -- 172.21.15.46:0/2154125007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba1c14c130 con 0x7fba1c1561d0 2024-01-27T01:08:14.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.342+0000 7fba03fff700 1 -- 172.21.15.46:0/2154125007 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7fba14003030 con 0x7fba1c1561d0 2024-01-27T01:08:14.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.342+0000 7fba03fff700 1 --2- 172.21.15.46:0/2154125007 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fba080670e0 0x7fba08069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:14.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.342+0000 7fba03fff700 1 -- 172.21.15.46:0/2154125007 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6611+0+0 (secure 0 0 0) 0x7fba14084ae0 con 0x7fba1c1561d0 2024-01-27T01:08:14.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.342+0000 7fba1bfff700 1 --2- 172.21.15.46:0/2154125007 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fba080670e0 0x7fba08069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:14.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.343+0000 7fba1bfff700 1 --2- 172.21.15.46:0/2154125007 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fba080670e0 0x7fba08069590 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fba1c06d720 tx=0x7fba0c017000 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:14.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.346+0000 7fba03fff700 1 -- 172.21.15.46:0/2154125007 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fba1c14c130 con 0x7fba1c1561d0 2024-01-27T01:08:14.499 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:14.499+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:14.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:14.640+0000 7fba224fa700 1 -- 172.21.15.46:0/2154125007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7fba1c14c130 con 0x7fba1c1561d0 2024-01-27T01:08:15.057 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:08:15.060 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:08:15.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.179+0000 7fba03fff700 1 -- 172.21.15.46:0/2154125007 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3245) v1 ==== 158+0+0 (secure 0 0 0) 0x7fba1c14c130 con 0x7fba1c1561d0 2024-01-27T01:08:15.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.182+0000 7fba224fa700 1 -- 172.21.15.46:0/2154125007 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fba080670e0 msgr2=0x7fba08069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:15.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.182+0000 7fba224fa700 1 --2- 172.21.15.46:0/2154125007 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fba080670e0 0x7fba08069590 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fba1c06d720 tx=0x7fba0c017000 comp rx=0 tx=0).stop 2024-01-27T01:08:15.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.182+0000 7fba224fa700 1 -- 172.21.15.46:0/2154125007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c1561d0 msgr2=0x7fba1c0ae4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:15.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.182+0000 7fba224fa700 1 --2- 172.21.15.46:0/2154125007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c1561d0 0x7fba1c0ae4a0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7fba1400c8e0 tx=0x7fba14001710 comp rx=0 tx=0).stop 2024-01-27T01:08:15.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.182+0000 7fba224fa700 1 -- 172.21.15.46:0/2154125007 shutdown_connections 2024-01-27T01:08:15.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.182+0000 7fba224fa700 1 --2- 172.21.15.46:0/2154125007 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fba080670e0 0x7fba08069590 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:15.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.182+0000 7fba224fa700 1 --2- 172.21.15.46:0/2154125007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba1c1561d0 0x7fba1c0ae4a0 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:15.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.182+0000 7fba224fa700 1 --2- 172.21.15.46:0/2154125007 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba1c06eb70 0x7fba1c1309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:15.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.182+0000 7fba224fa700 1 --2- 172.21.15.46:0/2154125007 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fba1c0b0280 0x7fba1c0adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:15.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.182+0000 7fba224fa700 1 -- 172.21.15.46:0/2154125007 >> 172.21.15.46:0/2154125007 conn(0x7fba1c06b630 msgr2=0x7fba1c06c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:15.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.182+0000 7fba224fa700 1 -- 172.21.15.46:0/2154125007 shutdown_connections 2024-01-27T01:08:15.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.182+0000 7fba224fa700 1 -- 172.21.15.46:0/2154125007 wait complete. 2024-01-27T01:08:15.197 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-27T01:08:15.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.339+0000 7fde51255700 1 Processor -- start 2024-01-27T01:08:15.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.340+0000 7fde51255700 1 -- start start 2024-01-27T01:08:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.340+0000 7fde51255700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde4c0cb640 0x7fde4c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.340+0000 7fde51255700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c0ccc00 con 0x7fde4c141200 2024-01-27T01:08:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.340+0000 7fde51255700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c0cc310 con 0x7fde4c0c8da0 2024-01-27T01:08:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.340+0000 7fde51255700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde4c0cbbf0 con 0x7fde4c0cb640 2024-01-27T01:08:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.341+0000 7fde4b59e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde4c0cb640 0x7fde4c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.341+0000 7fde43fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fde4c141200 0x7fde4c0caf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44958/0 (socket says 172.21.15.46:44958) 2024-01-27T01:08:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.341+0000 7fde43fff700 1 -- 172.21.15.46:0/3482028408 learned_addr learned my addr 172.21.15.46:0/3482028408 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.341+0000 7fde4b59e700 1 -- 172.21.15.46:0/3482028408 >> v1:172.21.15.106:6789/0 conn(0x7fde4c0c8da0 legacy=0x7fde4c140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.341+0000 7fde4b59e700 1 -- 172.21.15.46:0/3482028408 >> v1:172.21.15.46:6789/0 conn(0x7fde4c141200 legacy=0x7fde4c0caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.341+0000 7fde4b59e700 1 -- 172.21.15.46:0/3482028408 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4c0cc5b0 con 0x7fde4c0cb640 2024-01-27T01:08:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.341+0000 7fde4a59c700 1 -- 172.21.15.46:0/3482028408 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218475432 0 0) 0x7fde4c0ccc00 con 0x7fde4c141200 2024-01-27T01:08:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.341+0000 7fde4b59e700 1 --2- 172.21.15.46:0/3482028408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde4c0cb640 0x7fde4c0c9170 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fde44009e60 tx=0x7fde44009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c184b1d9d7610c5c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.341+0000 7fde4a59c700 1 -- 172.21.15.46:0/3482028408 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4400e070 con 0x7fde4c0cb640 2024-01-27T01:08:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.341+0000 7fde4a59c700 1 -- 172.21.15.46:0/3482028408 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4400bc30 con 0x7fde4c0cb640 2024-01-27T01:08:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.341+0000 7fde4a59c700 1 -- 172.21.15.46:0/3482028408 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde44012430 con 0x7fde4c0cb640 2024-01-27T01:08:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.342+0000 7fde51255700 1 -- 172.21.15.46:0/3482028408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde4c0cb640 msgr2=0x7fde4c0c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.342+0000 7fde51255700 1 --2- 172.21.15.46:0/3482028408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde4c0cb640 0x7fde4c0c9170 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fde44009e60 tx=0x7fde44009780 comp rx=0 tx=0).stop 2024-01-27T01:08:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.342+0000 7fde51255700 1 -- 172.21.15.46:0/3482028408 shutdown_connections 2024-01-27T01:08:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.342+0000 7fde51255700 1 --2- 172.21.15.46:0/3482028408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde4c0cb640 0x7fde4c0c9170 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.342+0000 7fde51255700 1 -- 172.21.15.46:0/3482028408 >> 172.21.15.46:0/3482028408 conn(0x7fde4c06bd90 msgr2=0x7fde4c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.342+0000 7fde51255700 1 -- 172.21.15.46:0/3482028408 shutdown_connections 2024-01-27T01:08:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.342+0000 7fde51255700 1 -- 172.21.15.46:0/3482028408 wait complete. 2024-01-27T01:08:15.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.343+0000 7fde51255700 1 Processor -- start 2024-01-27T01:08:15.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.343+0000 7fde51255700 1 -- start start 2024-01-27T01:08:15.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.344+0000 7fde51255700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde4c0c8da0 0x7fde4c157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:15.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.344+0000 7fde51255700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde4c0cb640 0x7fde4c158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:15.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.344+0000 7fde51255700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde4c141200 0x7fde4c159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:15.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.344+0000 7fde51255700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c15a2d0 con 0x7fde4c0cb640 2024-01-27T01:08:15.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.344+0000 7fde51255700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c0c8560 con 0x7fde4c0c8da0 2024-01-27T01:08:15.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.344+0000 7fde51255700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c157680 con 0x7fde4c141200 2024-01-27T01:08:15.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.344+0000 7fde43fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde4c0cb640 0x7fde4c158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:15.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.344+0000 7fde43fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde4c0cb640 0x7fde4c158b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49448/0 (socket says 172.21.15.46:49448) 2024-01-27T01:08:15.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.344+0000 7fde43fff700 1 -- 172.21.15.46:0/2140134604 learned_addr learned my addr 172.21.15.46:0/2140134604 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:15.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.344+0000 7fde4ad9d700 1 --2- 172.21.15.46:0/2140134604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde4c0c8da0 0x7fde4c157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:15.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.344+0000 7fde4b59e700 1 --2- 172.21.15.46:0/2140134604 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde4c141200 0x7fde4c159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:15.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.345+0000 7fde43fff700 1 -- 172.21.15.46:0/2140134604 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde4c141200 msgr2=0x7fde4c159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:15.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.345+0000 7fde43fff700 1 --2- 172.21.15.46:0/2140134604 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde4c141200 0x7fde4c159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:15.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.345+0000 7fde43fff700 1 -- 172.21.15.46:0/2140134604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde4c0c8da0 msgr2=0x7fde4c157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:15.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.345+0000 7fde43fff700 1 --2- 172.21.15.46:0/2140134604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde4c0c8da0 0x7fde4c157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:15.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.345+0000 7fde43fff700 1 -- 172.21.15.46:0/2140134604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde44009a20 con 0x7fde4c0cb640 2024-01-27T01:08:15.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.345+0000 7fde43fff700 1 --2- 172.21.15.46:0/2140134604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde4c0cb640 0x7fde4c158b20 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fde3400c5b0 tx=0x7fde34001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:15.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.345+0000 7fde42ffd700 1 -- 172.21.15.46:0/2140134604 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde34005b30 con 0x7fde4c0cb640 2024-01-27T01:08:15.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.345+0000 7fde42ffd700 1 -- 172.21.15.46:0/2140134604 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde34005c90 con 0x7fde4c0cb640 2024-01-27T01:08:15.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.345+0000 7fde51255700 1 -- 172.21.15.46:0/2140134604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4c1596c0 con 0x7fde4c0cb640 2024-01-27T01:08:15.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.345+0000 7fde42ffd700 1 -- 172.21.15.46:0/2140134604 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde34007300 con 0x7fde4c0cb640 2024-01-27T01:08:15.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.345+0000 7fde51255700 1 -- 172.21.15.46:0/2140134604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde4c159c80 con 0x7fde4c0cb640 2024-01-27T01:08:15.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.346+0000 7fde51255700 1 -- 172.21.15.46:0/2140134604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde4c15a440 con 0x7fde4c0cb640 2024-01-27T01:08:15.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.347+0000 7fde42ffd700 1 -- 172.21.15.46:0/2140134604 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7fde3401b3b0 con 0x7fde4c0cb640 2024-01-27T01:08:15.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.347+0000 7fde42ffd700 1 --2- 172.21.15.46:0/2140134604 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fde2c0670e0 0x7fde2c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:15.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.347+0000 7fde42ffd700 1 -- 172.21.15.46:0/2140134604 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6611+0+0 (secure 0 0 0) 0x7fde34084000 con 0x7fde4c0cb640 2024-01-27T01:08:15.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.347+0000 7fde4ad9d700 1 --2- 172.21.15.46:0/2140134604 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fde2c0670e0 0x7fde2c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:15.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.348+0000 7fde4ad9d700 1 --2- 172.21.15.46:0/2140134604 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fde2c0670e0 0x7fde2c069590 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fde3c003a80 tx=0x7fde3c007ec0 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:15.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.352+0000 7fde42ffd700 1 -- 172.21.15.46:0/2140134604 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fde34050c60 con 0x7fde4c0cb640 2024-01-27T01:08:15.464 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:15.465+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.558+0000 7fde51255700 1 -- 172.21.15.46:0/2140134604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fde4c04c710 con 0x7fde4c0cb640 2024-01-27T01:08:15.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.561+0000 7fde42ffd700 1 -- 172.21.15.46:0/2140134604 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v88)=0 v88) v1 ==== 122+0+0 (secure 0 0 0) 0x7fde34054280 con 0x7fde4c0cb640 2024-01-27T01:08:15.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.564+0000 7fde51255700 1 -- 172.21.15.46:0/2140134604 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fde2c0670e0 msgr2=0x7fde2c069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:15.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.564+0000 7fde51255700 1 --2- 172.21.15.46:0/2140134604 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fde2c0670e0 0x7fde2c069590 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fde3c003a80 tx=0x7fde3c007ec0 comp rx=0 tx=0).stop 2024-01-27T01:08:15.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.564+0000 7fde51255700 1 -- 172.21.15.46:0/2140134604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde4c0cb640 msgr2=0x7fde4c158b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:15.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.564+0000 7fde51255700 1 --2- 172.21.15.46:0/2140134604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde4c0cb640 0x7fde4c158b20 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fde3400c5b0 tx=0x7fde34001070 comp rx=0 tx=0).stop 2024-01-27T01:08:15.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.564+0000 7fde51255700 1 -- 172.21.15.46:0/2140134604 shutdown_connections 2024-01-27T01:08:15.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.564+0000 7fde51255700 1 --2- 172.21.15.46:0/2140134604 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fde2c0670e0 0x7fde2c069590 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:15.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.564+0000 7fde51255700 1 --2- 172.21.15.46:0/2140134604 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde4c141200 0x7fde4c159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:15.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.564+0000 7fde51255700 1 --2- 172.21.15.46:0/2140134604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde4c0c8da0 0x7fde4c157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:15.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.564+0000 7fde51255700 1 --2- 172.21.15.46:0/2140134604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde4c0cb640 0x7fde4c158b20 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:15.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.564+0000 7fde51255700 1 -- 172.21.15.46:0/2140134604 >> 172.21.15.46:0/2140134604 conn(0x7fde4c06bd90 msgr2=0x7fde4c0c3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:15.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.564+0000 7fde51255700 1 -- 172.21.15.46:0/2140134604 shutdown_connections 2024-01-27T01:08:15.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.564+0000 7fde51255700 1 -- 172.21.15.46:0/2140134604 wait complete. 2024-01-27T01:08:15.577 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-27T01:08:15.577 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:08:15.627 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:08:15.665 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-27T01:08:15.665 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-27T01:08:15.665 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:08:15.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.805+0000 7f04c1f6e700 1 Processor -- start 2024-01-27T01:08:15.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.805+0000 7f04c1f6e700 1 -- start start 2024-01-27T01:08:15.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.806+0000 7f04c1f6e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f04bc06c3c0 0x7f04bc164750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:15.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.806+0000 7f04c1f6e700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04bc15dac0 con 0x7f04bc159960 2024-01-27T01:08:15.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.806+0000 7f04c1f6e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04bc15dd90 con 0x7f04bc159590 2024-01-27T01:08:15.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.806+0000 7f04c1f6e700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04bc15e350 con 0x7f04bc06c3c0 2024-01-27T01:08:15.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.806+0000 7f04baffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f04bc159960 0x7f04bc06bcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44962/0 (socket says 172.21.15.46:44962) 2024-01-27T01:08:15.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.806+0000 7f04baffd700 1 -- 172.21.15.46:0/3737514175 learned_addr learned my addr 172.21.15.46:0/3737514175 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:15.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.806+0000 7f04bbfff700 1 --2- 172.21.15.46:0/3737514175 >> v2:172.21.15.106:3300/0 conn(0x7f04bc06c3c0 0x7f04bc164750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:15.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.806+0000 7f04ba7fc700 1 -- 172.21.15.46:0/3737514175 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515632472 0 0) 0x7f04bc15dac0 con 0x7f04bc159960 2024-01-27T01:08:15.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.806+0000 7f04ba7fc700 1 -- 172.21.15.46:0/3737514175 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a4003570 con 0x7f04bc159960 2024-01-27T01:08:15.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.806+0000 7f04ba7fc700 1 -- 172.21.15.46:0/3737514175 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606064649 0 0) 0x7f04bc15dd90 con 0x7f04bc159590 2024-01-27T01:08:15.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.806+0000 7f04ba7fc700 1 -- 172.21.15.46:0/3737514175 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04bc15dac0 con 0x7f04bc159590 2024-01-27T01:08:15.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.806+0000 7f04bbfff700 1 -- 172.21.15.46:0/3737514175 >> v1:172.21.15.106:6789/0 conn(0x7f04bc159590 legacy=0x7f04bc15d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:15.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.806+0000 7f04bbfff700 1 -- 172.21.15.46:0/3737514175 >> v1:172.21.15.46:6789/0 conn(0x7f04bc159960 legacy=0x7f04bc06bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:15.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.806+0000 7f04bbfff700 1 -- 172.21.15.46:0/3737514175 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04bc15e070 con 0x7f04bc06c3c0 2024-01-27T01:08:15.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.807+0000 7f04bbfff700 1 --2- 172.21.15.46:0/3737514175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04bc06c3c0 0x7f04bc164750 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009e60 tx=0x7f04b4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df0573d7c1de79b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:15.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.807+0000 7f04ba7fc700 1 -- 172.21.15.46:0/3737514175 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b400e070 con 0x7f04bc06c3c0 2024-01-27T01:08:15.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.807+0000 7f04ba7fc700 1 -- 172.21.15.46:0/3737514175 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b400bab0 con 0x7f04bc06c3c0 2024-01-27T01:08:15.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.807+0000 7f04ba7fc700 1 -- 172.21.15.46:0/3737514175 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b400be30 con 0x7f04bc06c3c0 2024-01-27T01:08:15.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.807+0000 7f04c1f6e700 1 -- 172.21.15.46:0/3737514175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04bc06c3c0 msgr2=0x7f04bc164750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:15.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.807+0000 7f04c1f6e700 1 --2- 172.21.15.46:0/3737514175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04bc06c3c0 0x7f04bc164750 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009e60 tx=0x7f04b4009780 comp rx=0 tx=0).stop 2024-01-27T01:08:15.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.807+0000 7f04c1f6e700 1 -- 172.21.15.46:0/3737514175 shutdown_connections 2024-01-27T01:08:15.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.807+0000 7f04c1f6e700 1 --2- 172.21.15.46:0/3737514175 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04bc06c3c0 0x7f04bc164750 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:15.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.807+0000 7f04c1f6e700 1 -- 172.21.15.46:0/3737514175 >> 172.21.15.46:0/3737514175 conn(0x7f04bc0ce530 msgr2=0x7f04bc0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:15.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.807+0000 7f04c1f6e700 1 -- 172.21.15.46:0/3737514175 shutdown_connections 2024-01-27T01:08:15.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.807+0000 7f04c1f6e700 1 -- 172.21.15.46:0/3737514175 wait complete. 2024-01-27T01:08:15.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.809+0000 7f04c1f6e700 1 Processor -- start 2024-01-27T01:08:15.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.809+0000 7f04c1f6e700 1 -- start start 2024-01-27T01:08:15.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.809+0000 7f04c1f6e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc06c3c0 0x7f04bc1579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:15.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.809+0000 7f04c1f6e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04bc159590 0x7f04bc16cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:15.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.809+0000 7f04c1f6e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04bc159960 0x7f04bc16d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:15.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04c1f6e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc0b2d10 con 0x7f04bc06c3c0 2024-01-27T01:08:15.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04c1f6e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc0b2ff0 con 0x7f04bc159590 2024-01-27T01:08:15.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04c1f6e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc0b32d0 con 0x7f04bc159960 2024-01-27T01:08:15.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04bb7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc06c3c0 0x7f04bc1579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:15.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04bb7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc06c3c0 0x7f04bc1579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49458/0 (socket says 172.21.15.46:49458) 2024-01-27T01:08:15.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04bb7fe700 1 -- 172.21.15.46:0/1928801431 learned_addr learned my addr 172.21.15.46:0/1928801431 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:15.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04baffd700 1 --2- 172.21.15.46:0/1928801431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04bc159590 0x7f04bc16cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:15.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04bbfff700 1 --2- 172.21.15.46:0/1928801431 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04bc159960 0x7f04bc16d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:15.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04bb7fe700 1 -- 172.21.15.46:0/1928801431 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04bc159960 msgr2=0x7f04bc16d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:15.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04bb7fe700 1 --2- 172.21.15.46:0/1928801431 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04bc159960 0x7f04bc16d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:15.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04bb7fe700 1 -- 172.21.15.46:0/1928801431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04bc159590 msgr2=0x7f04bc16cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:15.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04bb7fe700 1 --2- 172.21.15.46:0/1928801431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04bc159590 0x7f04bc16cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:15.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04bb7fe700 1 -- 172.21.15.46:0/1928801431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4009a20 con 0x7f04bc06c3c0 2024-01-27T01:08:15.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04baffd700 1 --2- 172.21.15.46:0/1928801431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04bc159590 0x7f04bc16cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:15.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04bb7fe700 1 --2- 172.21.15.46:0/1928801431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc06c3c0 0x7f04bc1579d0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00c790 tx=0x7f04ac007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:15.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04a37fe700 1 -- 172.21.15.46:0/1928801431 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac010510 con 0x7f04bc06c3c0 2024-01-27T01:08:15.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04a37fe700 1 -- 172.21.15.46:0/1928801431 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac010670 con 0x7f04bc06c3c0 2024-01-27T01:08:15.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04c1f6e700 1 -- 172.21.15.46:0/1928801431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04bc0b35b0 con 0x7f04bc06c3c0 2024-01-27T01:08:15.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.810+0000 7f04a37fe700 1 -- 172.21.15.46:0/1928801431 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac010a10 con 0x7f04bc06c3c0 2024-01-27T01:08:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.811+0000 7f04c1f6e700 1 -- 172.21.15.46:0/1928801431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04bc0b3b70 con 0x7f04bc06c3c0 2024-01-27T01:08:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.811+0000 7f04c1f6e700 1 -- 172.21.15.46:0/1928801431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04bc04c710 con 0x7f04bc06c3c0 2024-01-27T01:08:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.812+0000 7f04a37fe700 1 -- 172.21.15.46:0/1928801431 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f04ac01b3b0 con 0x7f04bc06c3c0 2024-01-27T01:08:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.812+0000 7f04a37fe700 1 --2- 172.21.15.46:0/1928801431 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f04a80670e0 0x7f04a8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.812+0000 7f04a37fe700 1 -- 172.21.15.46:0/1928801431 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6611+0+0 (secure 0 0 0) 0x7f04ac084430 con 0x7f04bc06c3c0 2024-01-27T01:08:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.813+0000 7f04baffd700 1 --2- 172.21.15.46:0/1928801431 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f04a80670e0 0x7f04a8069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:15.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.813+0000 7f04baffd700 1 --2- 172.21.15.46:0/1928801431 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f04a80670e0 0x7f04a8069590 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f04bc15a7f0 tx=0x7f04b0009040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:15.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:15.817+0000 7f04a37fe700 1 -- 172.21.15.46:0/1928801431 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f04ac051090 con 0x7f04bc06c3c0 2024-01-27T01:08:16.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.069+0000 7f04c1f6e700 1 -- 172.21.15.46:0/1928801431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04bc06c9a0 con 0x7f04bc06c3c0 2024-01-27T01:08:16.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.069+0000 7f04a37fe700 1 -- 172.21.15.46:0/1928801431 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+3539 (secure 0 0 0) 0x7f04ac017df0 con 0x7f04bc06c3c0 2024-01-27T01:08:16.068 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:16.069 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":142,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6834/1114164873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":1114164873},{"type":"v1","addr":"172.21.15.106:6834","nonce":1114164873}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128},{"gid":9821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/702094428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6833","nonce":702094428},{"type":"v1","addr":"172.21.15.106:6835","nonce":702094428}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":11770,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4159722763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4159722763},{"type":"v1","addr":"172.21.15.46:6834","nonce":4159722763}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140},{"gid":12264,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2379838208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2379838208},{"type":"v1","addr":"172.21.15.46:6835","nonce":2379838208}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":142}],"filesystems":[]} 2024-01-27T01:08:16.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.071+0000 7f04c1f6e700 1 -- 172.21.15.46:0/1928801431 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f04a80670e0 msgr2=0x7f04a8069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:16.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.071+0000 7f04c1f6e700 1 --2- 172.21.15.46:0/1928801431 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f04a80670e0 0x7f04a8069590 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f04bc15a7f0 tx=0x7f04b0009040 comp rx=0 tx=0).stop 2024-01-27T01:08:16.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.071+0000 7f04c1f6e700 1 -- 172.21.15.46:0/1928801431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc06c3c0 msgr2=0x7f04bc1579d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:16.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.071+0000 7f04c1f6e700 1 --2- 172.21.15.46:0/1928801431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc06c3c0 0x7f04bc1579d0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00c790 tx=0x7f04ac007e10 comp rx=0 tx=0).stop 2024-01-27T01:08:16.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.071+0000 7f04c1f6e700 1 -- 172.21.15.46:0/1928801431 shutdown_connections 2024-01-27T01:08:16.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.071+0000 7f04c1f6e700 1 --2- 172.21.15.46:0/1928801431 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f04a80670e0 0x7f04a8069590 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:16.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.071+0000 7f04c1f6e700 1 --2- 172.21.15.46:0/1928801431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc06c3c0 0x7f04bc1579d0 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:16.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.071+0000 7f04c1f6e700 1 --2- 172.21.15.46:0/1928801431 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04bc159960 0x7f04bc16d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:16.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.071+0000 7f04c1f6e700 1 --2- 172.21.15.46:0/1928801431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04bc159590 0x7f04bc16cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:16.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.071+0000 7f04c1f6e700 1 -- 172.21.15.46:0/1928801431 >> 172.21.15.46:0/1928801431 conn(0x7f04bc0ce530 msgr2=0x7f04bc0cc620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:16.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.072+0000 7f04c1f6e700 1 -- 172.21.15.46:0/1928801431 shutdown_connections 2024-01-27T01:08:16.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.072+0000 7f04c1f6e700 1 -- 172.21.15.46:0/1928801431 wait complete. 2024-01-27T01:08:16.072 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 142 2024-01-27T01:08:16.085 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-27T01:08:16.085 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-27T01:08:16.085 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-27T01:08:16.085 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:08:16.085 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-27T01:08:16.086 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-27T01:08:16.086 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-27T01:08:16.086 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:08:16.086 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-27T01:08:16.086 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-27T01:08:16.086 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-27T01:08:16.086 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:08:16.086 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-27T01:08:16.087 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-27T01:08:16.087 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-27T01:08:16.087 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:08:16.095 INFO:tasks.ceph.mds.b:Stopped 2024-01-27T01:08:16.095 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-27T01:08:16.096 INFO:tasks.ceph.mds.d:Stopped 2024-01-27T01:08:16.096 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-27T01:08:16.097 INFO:tasks.ceph.mds.a:Stopped 2024-01-27T01:08:16.097 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-27T01:08:16.098 INFO:tasks.ceph.mds.c:Stopped 2024-01-27T01:08:16.098 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-27T01:08:16.138 INFO:tasks.ceph.mds.b:Started 2024-01-27T01:08:16.139 DEBUG:teuthology.parallel:result is None 2024-01-27T01:08:16.140 INFO:tasks.ceph.mds.d:Started 2024-01-27T01:08:16.140 DEBUG:teuthology.parallel:result is None 2024-01-27T01:08:16.141 INFO:tasks.ceph.mds.a:Started 2024-01-27T01:08:16.141 DEBUG:teuthology.parallel:result is None 2024-01-27T01:08:16.142 INFO:tasks.ceph.mds.c:Started 2024-01-27T01:08:16.142 DEBUG:teuthology.parallel:result is None 2024-01-27T01:08:16.142 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-27T01:08:16.215 INFO:tasks.ceph.mds.b.smithi106.stdout:starting mds.b at 2024-01-27T01:08:16.216 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-01-27T01:08:16.231 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-27T01:08:16.238 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-01-27T01:08:16.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.294+0000 7fb831574700 1 Processor -- start 2024-01-27T01:08:16.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.294+0000 7fb831574700 1 -- start start 2024-01-27T01:08:16.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.295+0000 7fb831574700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb82c0c6520 0x7fb82c0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:16.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.295+0000 7fb831574700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb82c0ced50 0x7fb82c0cf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:16.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.295+0000 7fb831574700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c15a0b0 con 0x7fb82c0b2b40 2024-01-27T01:08:16.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.295+0000 7fb831574700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82c0af070 con 0x7fb82c0ced50 2024-01-27T01:08:16.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.295+0000 7fb831574700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82c099fc0 con 0x7fb82c0c6520 2024-01-27T01:08:16.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.295+0000 7fb82b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb82c0ced50 0x7fb82c0cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:16.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.295+0000 7fb82b7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb82c0ced50 0x7fb82c0cf190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49506/0 (socket says 172.21.15.46:49506) 2024-01-27T01:08:16.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.295+0000 7fb82b7fe700 1 -- 172.21.15.46:0/2095669098 learned_addr learned my addr 172.21.15.46:0/2095669098 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:16.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.295+0000 7fb82a7fc700 1 --2- 172.21.15.46:0/2095669098 >> v2:172.21.15.106:3300/0 conn(0x7fb82c0c6520 0x7fb82c0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:16.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.296+0000 7fb82b7fe700 1 -- 172.21.15.46:0/2095669098 >> v2:172.21.15.106:3300/0 conn(0x7fb82c0c6520 msgr2=0x7fb82c0ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:16.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.296+0000 7fb82b7fe700 1 --2- 172.21.15.46:0/2095669098 >> v2:172.21.15.106:3300/0 conn(0x7fb82c0c6520 0x7fb82c0ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:16.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.296+0000 7fb82b7fe700 1 -- 172.21.15.46:0/2095669098 >> v1:172.21.15.106:6789/0 conn(0x7fb82c0b2b40 legacy=0x7fb82c0ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:16.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.296+0000 7fb82b7fe700 1 -- 172.21.15.46:0/2095669098 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c0ae4d0 con 0x7fb82c0ced50 2024-01-27T01:08:16.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.296+0000 7fb82b7fe700 1 --2- 172.21.15.46:0/2095669098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c0ced50 0x7fb82c0cf190 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0ce2f0 tx=0x7fb824009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ad75d79b73bdeb server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:16.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.296+0000 7fb829ffb700 1 -- 172.21.15.46:0/2095669098 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82400e070 con 0x7fb82c0ced50 2024-01-27T01:08:16.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.296+0000 7fb829ffb700 1 -- 172.21.15.46:0/2095669098 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82400ba20 con 0x7fb82c0ced50 2024-01-27T01:08:16.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.296+0000 7fb829ffb700 1 -- 172.21.15.46:0/2095669098 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8240092e0 con 0x7fb82c0ced50 2024-01-27T01:08:16.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.297+0000 7fb831574700 1 -- 172.21.15.46:0/2095669098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c0ced50 msgr2=0x7fb82c0cf190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:16.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.297+0000 7fb831574700 1 --2- 172.21.15.46:0/2095669098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c0ced50 0x7fb82c0cf190 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0ce2f0 tx=0x7fb824009840 comp rx=0 tx=0).stop 2024-01-27T01:08:16.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.297+0000 7fb831574700 1 -- 172.21.15.46:0/2095669098 shutdown_connections 2024-01-27T01:08:16.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.297+0000 7fb831574700 1 --2- 172.21.15.46:0/2095669098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c0ced50 0x7fb82c0cf190 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:16.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.297+0000 7fb831574700 1 --2- 172.21.15.46:0/2095669098 >> v2:172.21.15.106:3300/0 conn(0x7fb82c0c6520 0x7fb82c0ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:16.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.297+0000 7fb831574700 1 -- 172.21.15.46:0/2095669098 >> 172.21.15.46:0/2095669098 conn(0x7fb82c099a30 msgr2=0x7fb82c06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:16.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.297+0000 7fb831574700 1 -- 172.21.15.46:0/2095669098 shutdown_connections 2024-01-27T01:08:16.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.297+0000 7fb831574700 1 -- 172.21.15.46:0/2095669098 wait complete. 2024-01-27T01:08:16.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.298+0000 7fb831574700 1 Processor -- start 2024-01-27T01:08:16.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.298+0000 7fb831574700 1 -- start start 2024-01-27T01:08:16.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.299+0000 7fb831574700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb82c0b2b40 0x7fb82c0b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:16.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.299+0000 7fb831574700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb82c0c6520 0x7fb82c1514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:16.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.299+0000 7fb831574700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c0ced50 0x7fb82c0b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:16.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.299+0000 7fb831574700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c1551a0 con 0x7fb82c0ced50 2024-01-27T01:08:16.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.299+0000 7fb831574700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c0b59d0 con 0x7fb82c0b2b40 2024-01-27T01:08:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.299+0000 7fb831574700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c0b3740 con 0x7fb82c0c6520 2024-01-27T01:08:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.299+0000 7fb82b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c0ced50 0x7fb82c0b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.299+0000 7fb82b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c0ced50 0x7fb82c0b3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49514/0 (socket says 172.21.15.46:49514) 2024-01-27T01:08:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.299+0000 7fb82b7fe700 1 -- 172.21.15.46:0/1592678453 learned_addr learned my addr 172.21.15.46:0/1592678453 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.299+0000 7fb82affd700 1 --2- 172.21.15.46:0/1592678453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb82c0b2b40 0x7fb82c0b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.299+0000 7fb82a7fc700 1 --2- 172.21.15.46:0/1592678453 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb82c0c6520 0x7fb82c1514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.299+0000 7fb82affd700 1 -- 172.21.15.46:0/1592678453 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb82c0c6520 msgr2=0x7fb82c1514a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.299+0000 7fb82affd700 1 --2- 172.21.15.46:0/1592678453 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb82c0c6520 0x7fb82c1514a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.299+0000 7fb82affd700 1 -- 172.21.15.46:0/1592678453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c0ced50 msgr2=0x7fb82c0b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.299+0000 7fb82affd700 1 --2- 172.21.15.46:0/1592678453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c0ced50 0x7fb82c0b3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.299+0000 7fb82affd700 1 -- 172.21.15.46:0/1592678453 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb824009ae0 con 0x7fb82c0b2b40 2024-01-27T01:08:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.300+0000 7fb82b7fe700 1 --2- 172.21.15.46:0/1592678453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c0ced50 0x7fb82c0b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.300+0000 7fb82affd700 1 --2- 172.21.15.46:0/1592678453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb82c0b2b40 0x7fb82c0b4dc0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00c560 tx=0x7fb81c007eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:16.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.300+0000 7fb812ffd700 1 -- 172.21.15.46:0/1592678453 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c011590 con 0x7fb82c0b2b40 2024-01-27T01:08:16.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.300+0000 7fb812ffd700 1 -- 172.21.15.46:0/1592678453 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81c011710 con 0x7fb82c0b2b40 2024-01-27T01:08:16.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.300+0000 7fb831574700 1 -- 172.21.15.46:0/1592678453 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb82c0b5300 con 0x7fb82c0b2b40 2024-01-27T01:08:16.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.300+0000 7fb812ffd700 1 -- 172.21.15.46:0/1592678453 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c011ad0 con 0x7fb82c0b2b40 2024-01-27T01:08:16.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.300+0000 7fb831574700 1 -- 172.21.15.46:0/1592678453 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb82c154ec0 con 0x7fb82c0b2b40 2024-01-27T01:08:16.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.301+0000 7fb831574700 1 -- 172.21.15.46:0/1592678453 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb82c0b3a20 con 0x7fb82c0b2b40 2024-01-27T01:08:16.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.301+0000 7fb812ffd700 1 -- 172.21.15.46:0/1592678453 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7fb81c01b3b0 con 0x7fb82c0b2b40 2024-01-27T01:08:16.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.302+0000 7fb812ffd700 1 --2- 172.21.15.46:0/1592678453 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb8180670e0 0x7fb818069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:16.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.302+0000 7fb812ffd700 1 -- 172.21.15.46:0/1592678453 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6611+0+0 (secure 0 0 0) 0x7fb81c083fb0 con 0x7fb82c0b2b40 2024-01-27T01:08:16.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.302+0000 7fb82a7fc700 1 --2- 172.21.15.46:0/1592678453 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb8180670e0 0x7fb818069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:16.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.302+0000 7fb82a7fc700 1 --2- 172.21.15.46:0/1592678453 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb8180670e0 0x7fb818069590 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0b6390 tx=0x7fb8200091a0 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:16.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.306+0000 7fb812ffd700 1 -- 172.21.15.46:0/1592678453 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb81c050c20 con 0x7fb82c0b2b40 2024-01-27T01:08:16.479 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:16.479+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:16.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:16.554+0000 7fb831574700 1 -- 172.21.15.46:0/1592678453 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb82c0b54f0 con 0x7fb82c0b2b40 2024-01-27T01:08:17.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.191+0000 7fb812ffd700 1 -- 172.21.15.46:0/1592678453 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v257) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb81c017070 con 0x7fb82c0b2b40 2024-01-27T01:08:17.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.194+0000 7fb831574700 1 -- 172.21.15.46:0/1592678453 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb8180670e0 msgr2=0x7fb818069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:17.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.194+0000 7fb831574700 1 --2- 172.21.15.46:0/1592678453 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb8180670e0 0x7fb818069590 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0b6390 tx=0x7fb8200091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.194+0000 7fb831574700 1 -- 172.21.15.46:0/1592678453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb82c0b2b40 msgr2=0x7fb82c0b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:17.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.194+0000 7fb831574700 1 --2- 172.21.15.46:0/1592678453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb82c0b2b40 0x7fb82c0b4dc0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00c560 tx=0x7fb81c007eb0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.194+0000 7fb831574700 1 -- 172.21.15.46:0/1592678453 shutdown_connections 2024-01-27T01:08:17.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.194+0000 7fb831574700 1 --2- 172.21.15.46:0/1592678453 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb8180670e0 0x7fb818069590 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.194+0000 7fb831574700 1 --2- 172.21.15.46:0/1592678453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb82c0ced50 0x7fb82c0b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.194+0000 7fb831574700 1 --2- 172.21.15.46:0/1592678453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb82c0b2b40 0x7fb82c0b4dc0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.194+0000 7fb831574700 1 --2- 172.21.15.46:0/1592678453 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb82c0c6520 0x7fb82c1514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.194+0000 7fb831574700 1 -- 172.21.15.46:0/1592678453 >> 172.21.15.46:0/1592678453 conn(0x7fb82c099a30 msgr2=0x7fb82c068c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:17.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.194+0000 7fb831574700 1 -- 172.21.15.46:0/1592678453 shutdown_connections 2024-01-27T01:08:17.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.194+0000 7fb831574700 1 -- 172.21.15.46:0/1592678453 wait complete. 2024-01-27T01:08:17.195 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-01-27T01:08:17.210 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-27T01:08:17.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.350+0000 7fe044838700 1 Processor -- start 2024-01-27T01:08:17.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.350+0000 7fe044838700 1 -- start start 2024-01-27T01:08:17.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.351+0000 7fe044838700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe040159590 0x7fe04015d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:17.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.351+0000 7fe044838700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe04015d8f0 0x7fe04006bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:17.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.351+0000 7fe044838700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe040159960 0x7fe04006c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:17.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.351+0000 7fe044838700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe04015e350 con 0x7fe04015d8f0 2024-01-27T01:08:17.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.351+0000 7fe044838700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0400c97c0 con 0x7fe040159590 2024-01-27T01:08:17.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.351+0000 7fe044838700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe04015dd90 con 0x7fe040159960 2024-01-27T01:08:17.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.351+0000 7fe03dd9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe04015d8f0 0x7fe04006bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:17.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.351+0000 7fe03dd9b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe04015d8f0 0x7fe04006bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49522/0 (socket says 172.21.15.46:49522) 2024-01-27T01:08:17.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.351+0000 7fe03dd9b700 1 -- 172.21.15.46:0/2167601109 learned_addr learned my addr 172.21.15.46:0/2167601109 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:17.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.351+0000 7fe03e59c700 1 --2- 172.21.15.46:0/2167601109 >> v2:172.21.15.106:3300/0 conn(0x7fe040159590 0x7fe04015d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:17.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.351+0000 7fe03ed9d700 1 --2- 172.21.15.46:0/2167601109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe040159960 0x7fe04006c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:17.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.351+0000 7fe03dd9b700 1 -- 172.21.15.46:0/2167601109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe040159960 msgr2=0x7fe04006c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.351+0000 7fe03dd9b700 1 --2- 172.21.15.46:0/2167601109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe040159960 0x7fe04006c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.351+0000 7fe03dd9b700 1 -- 172.21.15.46:0/2167601109 >> v2:172.21.15.106:3300/0 conn(0x7fe040159590 msgr2=0x7fe04015d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.351+0000 7fe03dd9b700 1 --2- 172.21.15.46:0/2167601109 >> v2:172.21.15.106:3300/0 conn(0x7fe040159590 0x7fe04015d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.351+0000 7fe03dd9b700 1 -- 172.21.15.46:0/2167601109 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04015e070 con 0x7fe04015d8f0 2024-01-27T01:08:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.352+0000 7fe03dd9b700 1 --2- 172.21.15.46:0/2167601109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe04015d8f0 0x7fe04006bcb0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fe0400c1f60 tx=0x7fe034009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a0e13a07c2123e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.352+0000 7fe03d59a700 1 -- 172.21.15.46:0/2167601109 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03400b860 con 0x7fe04015d8f0 2024-01-27T01:08:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.352+0000 7fe03d59a700 1 -- 172.21.15.46:0/2167601109 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe03400b9c0 con 0x7fe04015d8f0 2024-01-27T01:08:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.352+0000 7fe03d59a700 1 -- 172.21.15.46:0/2167601109 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03400bd80 con 0x7fe04015d8f0 2024-01-27T01:08:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.352+0000 7fe044838700 1 -- 172.21.15.46:0/2167601109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe04015d8f0 msgr2=0x7fe04006bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.352+0000 7fe044838700 1 --2- 172.21.15.46:0/2167601109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe04015d8f0 0x7fe04006bcb0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fe0400c1f60 tx=0x7fe034009250 comp rx=0 tx=0).stop 2024-01-27T01:08:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.352+0000 7fe044838700 1 -- 172.21.15.46:0/2167601109 shutdown_connections 2024-01-27T01:08:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.352+0000 7fe044838700 1 --2- 172.21.15.46:0/2167601109 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe040159960 0x7fe04006c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.352+0000 7fe044838700 1 --2- 172.21.15.46:0/2167601109 >> v2:172.21.15.106:3300/0 conn(0x7fe040159590 0x7fe04015d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.352+0000 7fe044838700 1 --2- 172.21.15.46:0/2167601109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe04015d8f0 0x7fe04006bcb0 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.352+0000 7fe044838700 1 -- 172.21.15.46:0/2167601109 >> 172.21.15.46:0/2167601109 conn(0x7fe0400ce530 msgr2=0x7fe0400ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:17.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.352+0000 7fe044838700 1 -- 172.21.15.46:0/2167601109 shutdown_connections 2024-01-27T01:08:17.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.353+0000 7fe044838700 1 -- 172.21.15.46:0/2167601109 wait complete. 2024-01-27T01:08:17.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.354+0000 7fe044838700 1 Processor -- start 2024-01-27T01:08:17.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.354+0000 7fe044838700 1 -- start start 2024-01-27T01:08:17.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.354+0000 7fe044838700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe040159590 0x7fe04015b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:17.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.354+0000 7fe044838700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe040159960 0x7fe0400c2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:17.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.354+0000 7fe044838700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe04015d8f0 0x7fe0400c2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:17.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.354+0000 7fe044838700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0400c3f00 con 0x7fe04015d8f0 2024-01-27T01:08:17.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.354+0000 7fe044838700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04016a840 con 0x7fe040159590 2024-01-27T01:08:17.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.355+0000 7fe044838700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe04016aae0 con 0x7fe040159960 2024-01-27T01:08:17.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.355+0000 7fe03ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe04015d8f0 0x7fe0400c2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:17.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.355+0000 7fe03ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe04015d8f0 0x7fe0400c2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49530/0 (socket says 172.21.15.46:49530) 2024-01-27T01:08:17.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.355+0000 7fe03ed9d700 1 -- 172.21.15.46:0/95773041 learned_addr learned my addr 172.21.15.46:0/95773041 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:17.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.355+0000 7fe03dd9b700 1 --2- 172.21.15.46:0/95773041 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe040159960 0x7fe0400c2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:17.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.355+0000 7fe03e59c700 1 --2- 172.21.15.46:0/95773041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe040159590 0x7fe04015b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:17.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.355+0000 7fe03ed9d700 1 -- 172.21.15.46:0/95773041 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe040159960 msgr2=0x7fe0400c2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:17.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.355+0000 7fe03ed9d700 1 --2- 172.21.15.46:0/95773041 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe040159960 0x7fe0400c2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.355+0000 7fe03ed9d700 1 -- 172.21.15.46:0/95773041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe040159590 msgr2=0x7fe04015b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:17.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.355+0000 7fe03ed9d700 1 --2- 172.21.15.46:0/95773041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe040159590 0x7fe04015b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.355+0000 7fe03ed9d700 1 -- 172.21.15.46:0/95773041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0340094f0 con 0x7fe04015d8f0 2024-01-27T01:08:17.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.355+0000 7fe03ed9d700 1 --2- 172.21.15.46:0/95773041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe04015d8f0 0x7fe0400c2cd0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fe038009fa0 tx=0x7fe038009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:17.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.355+0000 7fe02e7fc700 1 -- 172.21.15.46:0/95773041 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03800ebc0 con 0x7fe04015d8f0 2024-01-27T01:08:17.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.355+0000 7fe02e7fc700 1 -- 172.21.15.46:0/95773041 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe03800ed20 con 0x7fe04015d8f0 2024-01-27T01:08:17.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.355+0000 7fe044838700 1 -- 172.21.15.46:0/95773041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0400c3b90 con 0x7fe04015d8f0 2024-01-27T01:08:17.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.355+0000 7fe02e7fc700 1 -- 172.21.15.46:0/95773041 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe038014670 con 0x7fe04015d8f0 2024-01-27T01:08:17.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.355+0000 7fe044838700 1 -- 172.21.15.46:0/95773041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0400c3650 con 0x7fe04015d8f0 2024-01-27T01:08:17.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.357+0000 7fe044838700 1 -- 172.21.15.46:0/95773041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0400c39f0 con 0x7fe04015d8f0 2024-01-27T01:08:17.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.357+0000 7fe02e7fc700 1 -- 172.21.15.46:0/95773041 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7fe0380149e0 con 0x7fe04015d8f0 2024-01-27T01:08:17.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.357+0000 7fe02e7fc700 1 --2- 172.21.15.46:0/95773041 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe0200670e0 0x7fe020069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:17.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.357+0000 7fe02e7fc700 1 -- 172.21.15.46:0/95773041 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 5665+0+0 (secure 0 0 0) 0x7fe038084ec0 con 0x7fe04015d8f0 2024-01-27T01:08:17.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.357+0000 7fe03e59c700 1 --2- 172.21.15.46:0/95773041 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe0200670e0 0x7fe020069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:17.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.358+0000 7fe03e59c700 1 --2- 172.21.15.46:0/95773041 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe0200670e0 0x7fe020069590 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe030006fd0 tx=0x7fe030008040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:17.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.362+0000 7fe02e7fc700 1 -- 172.21.15.46:0/95773041 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe0400c39f0 con 0x7fe04015d8f0 2024-01-27T01:08:17.451 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:17.451+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:17.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.655+0000 7fe044838700 1 -- 172.21.15.46:0/95773041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe0400c39f0 con 0x7fe04015d8f0 2024-01-27T01:08:17.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.656+0000 7fe02e7fc700 1 -- 172.21.15.46:0/95773041 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v55) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fe0400c39f0 con 0x7fe04015d8f0 2024-01-27T01:08:17.655 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:17.655 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:08:17.655 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-01-27T01:08:17.655 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.655 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-01-27T01:08:17.655 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlM6dKKBAAPpeWeDZWrFgc/4zuxKCvHw==", 2024-01-27T01:08:17.655 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.655 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:08:17.655 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:08:17.655 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:08:17.656 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:08:17.656 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.656 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.656 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.656 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-01-27T01:08:17.656 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlmzHRNRAAflWYZ/p4AXeMy54tvLpugA==", 2024-01-27T01:08:17.656 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.656 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:08:17.656 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:08:17.656 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:08:17.656 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:08:17.656 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.656 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.657 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.657 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-01-27T01:08:17.657 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlMZxoMBAAG5kyfm+LLG86y/p2A8BfDQ==", 2024-01-27T01:08:17.657 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.657 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:08:17.657 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:08:17.657 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:08:17.657 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:08:17.657 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.657 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.657 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.657 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-01-27T01:08:17.657 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl5jx/ARAA3cwtLAOWQ/n2NG+xwEZ3QQ==", 2024-01-27T01:08:17.657 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.658 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:08:17.658 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:08:17.658 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:08:17.658 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:08:17.658 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.658 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.658 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.658 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-01-27T01:08:17.658 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4QLRlJ2OKJRAAWVhxEdQkUW20294w/KLL6g==", 2024-01-27T01:08:17.658 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.658 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:08:17.658 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:08:17.658 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:08:17.658 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.658 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.658 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.659 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-01-27T01:08:17.659 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7QLRlE2tAJRAAzNlTiCr6a2eWiabRy1SqSw==", 2024-01-27T01:08:17.659 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.659 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:08:17.659 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:08:17.659 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:08:17.659 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.659 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.659 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.659 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-01-27T01:08:17.659 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+QLRlyySaJBAA5072ba5HUNi2dYE3D02M/Q==", 2024-01-27T01:08:17.659 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.659 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:08:17.659 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:08:17.660 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:08:17.660 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.660 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.660 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.660 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-01-27T01:08:17.660 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBQLRl9+tAIxAApZk0btXIpQPWSb3BYUYsFQ==", 2024-01-27T01:08:17.660 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.660 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:08:17.660 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:08:17.660 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:08:17.660 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.660 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.660 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.661 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-01-27T01:08:17.661 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHQLRlurUtJxAAT9fwKrAWl62VOXwShg7NJg==", 2024-01-27T01:08:17.661 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.661 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:08:17.661 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:08:17.661 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:08:17.661 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.661 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.661 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.661 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-01-27T01:08:17.661 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKQLRl7aqoJRAAXu9fykWELsW3SwnahoIOLA==", 2024-01-27T01:08:17.661 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.661 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:08:17.661 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:08:17.661 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:08:17.661 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.662 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.662 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.662 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-01-27T01:08:17.662 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNQLRlWf7pJBAAopmFLIKb8s32YZs46VwbGg==", 2024-01-27T01:08:17.662 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.662 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:08:17.662 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:08:17.662 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:08:17.662 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.662 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.662 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.662 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-01-27T01:08:17.662 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQQLRlNGfhIxAAmCgmOB3QAGfSIYHBb5BUZw==", 2024-01-27T01:08:17.662 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.663 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-01-27T01:08:17.663 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-01-27T01:08:17.663 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:08:17.663 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.663 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.663 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.663 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-01-27T01:08:17.663 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRleO8wBhAA1018AcO3KcyOZxgB0Itxaw==", 2024-01-27T01:08:17.663 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.663 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:08:17.663 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-01-27T01:08:17.663 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-27T01:08:17.663 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.664 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.664 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.664 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-01-27T01:08:17.664 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlsiZEDRAAjXSamHe2bZcDqVsqXqIcGg==", 2024-01-27T01:08:17.664 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.664 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:08:17.664 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T01:08:17.664 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T01:08:17.664 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T01:08:17.664 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.664 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.664 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.664 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-01-27T01:08:17.664 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRl6YT1CRAApXlN4j20xzwda0tDMy94Eg==", 2024-01-27T01:08:17.665 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.665 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:08:17.665 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T01:08:17.665 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T01:08:17.665 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T01:08:17.665 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.665 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.665 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.665 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-01-27T01:08:17.665 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA3QLRlHvDQEhAAxMhgWJsZC5/6Af5zFV6XIA==", 2024-01-27T01:08:17.665 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.665 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-01-27T01:08:17.665 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-01-27T01:08:17.665 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-01-27T01:08:17.665 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-01-27T01:08:17.666 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.666 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.666 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.666 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-01-27T01:08:17.666 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlvMwDCBAAXnFyMduz7PfWdLWbjytIiQ==", 2024-01-27T01:08:17.666 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.666 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T01:08:17.666 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-01-27T01:08:17.666 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-01-27T01:08:17.666 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:08:17.666 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.666 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.666 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.666 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-01-27T01:08:17.666 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlhMNSJBAA38QU6DrCvFjO+u9zQ8bkEA==", 2024-01-27T01:08:17.667 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.667 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T01:08:17.667 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-27T01:08:17.667 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:08:17.667 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.667 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:17.667 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:17.667 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-01-27T01:08:17.667 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2QLRlI9EbIBAACTpzAO+8A/VFJZcT/WWBng==", 2024-01-27T01:08:17.667 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-01-27T01:08:17.667 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-01-27T01:08:17.667 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-01-27T01:08:17.667 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-01-27T01:08:17.667 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.668 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:17.668 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-01-27T01:08:17.668 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:08:17.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.658+0000 7fe044838700 1 -- 172.21.15.46:0/95773041 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe0200670e0 msgr2=0x7fe020069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.658+0000 7fe044838700 1 --2- 172.21.15.46:0/95773041 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe0200670e0 0x7fe020069590 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe030006fd0 tx=0x7fe030008040 comp rx=0 tx=0).stop 2024-01-27T01:08:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.658+0000 7fe044838700 1 -- 172.21.15.46:0/95773041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe04015d8f0 msgr2=0x7fe0400c2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.658+0000 7fe044838700 1 --2- 172.21.15.46:0/95773041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe04015d8f0 0x7fe0400c2cd0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fe038009fa0 tx=0x7fe038009900 comp rx=0 tx=0).stop 2024-01-27T01:08:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.658+0000 7fe044838700 1 -- 172.21.15.46:0/95773041 shutdown_connections 2024-01-27T01:08:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.658+0000 7fe044838700 1 --2- 172.21.15.46:0/95773041 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe0200670e0 0x7fe020069590 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.658+0000 7fe044838700 1 --2- 172.21.15.46:0/95773041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe04015d8f0 0x7fe0400c2cd0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.658+0000 7fe044838700 1 --2- 172.21.15.46:0/95773041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe040159590 0x7fe04015b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.658+0000 7fe044838700 1 --2- 172.21.15.46:0/95773041 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe040159960 0x7fe0400c2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.658+0000 7fe044838700 1 -- 172.21.15.46:0/95773041 >> 172.21.15.46:0/95773041 conn(0x7fe0400ce530 msgr2=0x7fe0400cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.658+0000 7fe044838700 1 -- 172.21.15.46:0/95773041 shutdown_connections 2024-01-27T01:08:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.658+0000 7fe044838700 1 -- 172.21.15.46:0/95773041 wait complete. 2024-01-27T01:08:17.671 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:08:17.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.808+0000 7f52a1251700 1 Processor -- start 2024-01-27T01:08:17.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.809+0000 7f52a1251700 1 -- start start 2024-01-27T01:08:17.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.809+0000 7f52a1251700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f529c0acf30 0x7f529c0c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:17.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.809+0000 7f52a1251700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f529c0ad300 0x7f529c0ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:17.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.809+0000 7f52a1251700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f529c0c4230 con 0x7f529c0ad780 2024-01-27T01:08:17.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.809+0000 7f52a1251700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f529c157be0 con 0x7f529c0acf30 2024-01-27T01:08:17.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.809+0000 7f52a1251700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f529c0cbf10 con 0x7f529c0ad300 2024-01-27T01:08:17.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.810+0000 7f529ad9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f529c0acf30 0x7f529c0c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:17.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.810+0000 7f529ad9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f529c0acf30 0x7f529c0c3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49544/0 (socket says 172.21.15.46:49544) 2024-01-27T01:08:17.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.810+0000 7f529ad9d700 1 -- 172.21.15.46:0/821992836 learned_addr learned my addr 172.21.15.46:0/821992836 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:17.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.810+0000 7f529a59c700 1 --2- 172.21.15.46:0/821992836 >> v2:172.21.15.106:3300/0 conn(0x7f529c0ad300 0x7f529c0ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:17.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.810+0000 7f529ad9d700 1 -- 172.21.15.46:0/821992836 >> v2:172.21.15.106:3300/0 conn(0x7f529c0ad300 msgr2=0x7f529c0ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:17.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.810+0000 7f529ad9d700 1 --2- 172.21.15.46:0/821992836 >> v2:172.21.15.106:3300/0 conn(0x7f529c0ad300 0x7f529c0ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.810+0000 7f529ad9d700 1 -- 172.21.15.46:0/821992836 >> v1:172.21.15.106:6789/0 conn(0x7f529c0ad780 legacy=0x7f529c15b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:17.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.810+0000 7f529ad9d700 1 -- 172.21.15.46:0/821992836 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529c0c4460 con 0x7f529c0acf30 2024-01-27T01:08:17.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.810+0000 7f529ad9d700 1 --2- 172.21.15.46:0/821992836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f529c0acf30 0x7f529c0c3cf0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f529c0b1060 tx=0x7f5290009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef7233312738a6e9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:17.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.810+0000 7f5299d9b700 1 -- 172.21.15.46:0/821992836 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5290009e70 con 0x7f529c0acf30 2024-01-27T01:08:17.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.810+0000 7f5299d9b700 1 -- 172.21.15.46:0/821992836 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529000b860 con 0x7f529c0acf30 2024-01-27T01:08:17.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.810+0000 7f5299d9b700 1 -- 172.21.15.46:0/821992836 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529000bc00 con 0x7f529c0acf30 2024-01-27T01:08:17.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.811+0000 7f52a1251700 1 -- 172.21.15.46:0/821992836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f529c0acf30 msgr2=0x7f529c0c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:17.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.811+0000 7f52a1251700 1 --2- 172.21.15.46:0/821992836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f529c0acf30 0x7f529c0c3cf0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f529c0b1060 tx=0x7f5290009250 comp rx=0 tx=0).stop 2024-01-27T01:08:17.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.811+0000 7f52a1251700 1 -- 172.21.15.46:0/821992836 shutdown_connections 2024-01-27T01:08:17.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.811+0000 7f52a1251700 1 --2- 172.21.15.46:0/821992836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f529c0acf30 0x7f529c0c3cf0 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.811+0000 7f52a1251700 1 --2- 172.21.15.46:0/821992836 >> v2:172.21.15.106:3300/0 conn(0x7f529c0ad300 0x7f529c0ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.811+0000 7f52a1251700 1 -- 172.21.15.46:0/821992836 >> 172.21.15.46:0/821992836 conn(0x7f529c097820 msgr2=0x7f529c068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:17.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.811+0000 7f52a1251700 1 -- 172.21.15.46:0/821992836 shutdown_connections 2024-01-27T01:08:17.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.811+0000 7f52a1251700 1 -- 172.21.15.46:0/821992836 wait complete. 2024-01-27T01:08:17.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.813+0000 7f52a1251700 1 Processor -- start 2024-01-27T01:08:17.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.813+0000 7f52a1251700 1 -- start start 2024-01-27T01:08:17.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.813+0000 7f52a1251700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f529c0acf30 0x7f529c0b0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:17.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.813+0000 7f52a1251700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f529c0ad300 0x7f529c0b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:17.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.813+0000 7f52a1251700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f529c0ad780 0x7f529c0ab850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:17.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.813+0000 7f52a1251700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c0b08d0 con 0x7f529c0acf30 2024-01-27T01:08:17.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.813+0000 7f52a1251700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c069930 con 0x7f529c0ad780 2024-01-27T01:08:17.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.813+0000 7f52a1251700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f529c068a40 con 0x7f529c0ad300 2024-01-27T01:08:17.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.814+0000 7f529ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f529c0acf30 0x7f529c0b0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:17.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.814+0000 7f529ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f529c0acf30 0x7f529c0b0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49558/0 (socket says 172.21.15.46:49558) 2024-01-27T01:08:17.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.814+0000 7f529ad9d700 1 -- 172.21.15.46:0/2907068541 learned_addr learned my addr 172.21.15.46:0/2907068541 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:17.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.814+0000 7f529a59c700 1 --2- 172.21.15.46:0/2907068541 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f529c0ad300 0x7f529c0b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:17.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.814+0000 7f529b59e700 1 --2- 172.21.15.46:0/2907068541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f529c0ad780 0x7f529c0ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:17.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.814+0000 7f529ad9d700 1 -- 172.21.15.46:0/2907068541 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f529c0ad300 msgr2=0x7f529c0b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:17.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.814+0000 7f529ad9d700 1 --2- 172.21.15.46:0/2907068541 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f529c0ad300 0x7f529c0b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.814+0000 7f529ad9d700 1 -- 172.21.15.46:0/2907068541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f529c0ad780 msgr2=0x7f529c0ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:17.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.814+0000 7f529ad9d700 1 --2- 172.21.15.46:0/2907068541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f529c0ad780 0x7f529c0ab850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:17.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.814+0000 7f529ad9d700 1 -- 172.21.15.46:0/2907068541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52900094f0 con 0x7f529c0acf30 2024-01-27T01:08:17.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.814+0000 7f529b59e700 1 --2- 172.21.15.46:0/2907068541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f529c0ad780 0x7f529c0ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:17.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.814+0000 7f529ad9d700 1 --2- 172.21.15.46:0/2907068541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f529c0acf30 0x7f529c0b0390 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f529c06ef50 tx=0x7f5290017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:17.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.814+0000 7f5282ffd700 1 -- 172.21.15.46:0/2907068541 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529000b9d0 con 0x7f529c0acf30 2024-01-27T01:08:17.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.814+0000 7f5282ffd700 1 -- 172.21.15.46:0/2907068541 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529000bb30 con 0x7f529c0acf30 2024-01-27T01:08:17.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.814+0000 7f52a1251700 1 -- 172.21.15.46:0/2907068541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529c068d20 con 0x7f529c0acf30 2024-01-27T01:08:17.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.814+0000 7f5282ffd700 1 -- 172.21.15.46:0/2907068541 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5290025670 con 0x7f529c0acf30 2024-01-27T01:08:17.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.814+0000 7f52a1251700 1 -- 172.21.15.46:0/2907068541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f529c0692e0 con 0x7f529c0acf30 2024-01-27T01:08:17.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.815+0000 7f52a1251700 1 -- 172.21.15.46:0/2907068541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f529c04c710 con 0x7f529c0acf30 2024-01-27T01:08:17.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.816+0000 7f5282ffd700 1 -- 172.21.15.46:0/2907068541 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f5290023070 con 0x7f529c0acf30 2024-01-27T01:08:17.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.817+0000 7f5282ffd700 1 --2- 172.21.15.46:0/2907068541 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f52880670e0 0x7f5288069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:17.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.817+0000 7f5282ffd700 1 -- 172.21.15.46:0/2907068541 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 5665+0+0 (secure 0 0 0) 0x7f5290086190 con 0x7f529c0acf30 2024-01-27T01:08:17.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.817+0000 7f529a59c700 1 --2- 172.21.15.46:0/2907068541 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f52880670e0 0x7f5288069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:17.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.818+0000 7f529a59c700 1 --2- 172.21.15.46:0/2907068541 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f52880670e0 0x7f5288069590 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f529c06bbc0 tx=0x7f528c0091a0 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:17.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:17.821+0000 7f5282ffd700 1 -- 172.21.15.46:0/2907068541 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5290053220 con 0x7f529c0acf30 2024-01-27T01:08:18.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.033+0000 7f52a1251700 1 -- 172.21.15.46:0/2907068541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f529c0c4740 con 0x7f529c0acf30 2024-01-27T01:08:18.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.034+0000 7f5282ffd700 1 -- 172.21.15.46:0/2907068541 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v259) v1 ==== 74+0+13105 (secure 0 0 0) 0x7f5290056840 con 0x7f529c0acf30 2024-01-27T01:08:18.033 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:18.033 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":259,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:08:17.195652+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:08:18.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.037+0000 7f52a1251700 1 -- 172.21.15.46:0/2907068541 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f52880670e0 msgr2=0x7f5288069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:18.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.037+0000 7f52a1251700 1 --2- 172.21.15.46:0/2907068541 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f52880670e0 0x7f5288069590 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f529c06bbc0 tx=0x7f528c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.037+0000 7f52a1251700 1 -- 172.21.15.46:0/2907068541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f529c0acf30 msgr2=0x7f529c0b0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:18.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.037+0000 7f52a1251700 1 --2- 172.21.15.46:0/2907068541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f529c0acf30 0x7f529c0b0390 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f529c06ef50 tx=0x7f5290017000 comp rx=0 tx=0).stop 2024-01-27T01:08:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.037+0000 7f52a1251700 1 -- 172.21.15.46:0/2907068541 shutdown_connections 2024-01-27T01:08:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.037+0000 7f52a1251700 1 --2- 172.21.15.46:0/2907068541 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f52880670e0 0x7f5288069590 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.037+0000 7f52a1251700 1 --2- 172.21.15.46:0/2907068541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f529c0acf30 0x7f529c0b0390 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.037+0000 7f52a1251700 1 --2- 172.21.15.46:0/2907068541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f529c0ad780 0x7f529c0ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.037+0000 7f52a1251700 1 --2- 172.21.15.46:0/2907068541 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f529c0ad300 0x7f529c0b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.037+0000 7f52a1251700 1 -- 172.21.15.46:0/2907068541 >> 172.21.15.46:0/2907068541 conn(0x7f529c097820 msgr2=0x7f529c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.037+0000 7f52a1251700 1 -- 172.21.15.46:0/2907068541 shutdown_connections 2024-01-27T01:08:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.037+0000 7f52a1251700 1 -- 172.21.15.46:0/2907068541 wait complete. 2024-01-27T01:08:18.050 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-27T01:08:18.051 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-27T01:08:18.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.191+0000 7f065a9e2700 1 Processor -- start 2024-01-27T01:08:18.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.191+0000 7f065a9e2700 1 -- start start 2024-01-27T01:08:18.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.192+0000 7f065a9e2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0654155c00 0x7f065406bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.192+0000 7f065a9e2700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0654155fd0 0x7f065415a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.192+0000 7f065a9e2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f065406a3c0 0x7f0654068940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.192+0000 7f065a9e2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065406b740 con 0x7f065406a3c0 2024-01-27T01:08:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.192+0000 7f065a9e2700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065406ab30 con 0x7f0654155fd0 2024-01-27T01:08:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.192+0000 7f065a9e2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065406ae10 con 0x7f0654155c00 2024-01-27T01:08:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.192+0000 7f0658f7f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f065406a3c0 0x7f0654068940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.192+0000 7f064bfff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0654155fd0 0x7f065415a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.192+0000 7f0658f7f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f065406a3c0 0x7f0654068940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49574/0 (socket says 172.21.15.46:49574) 2024-01-27T01:08:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.192+0000 7f0653fff700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0654155c00 0x7f065406bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.192+0000 7f0658f7f700 1 -- 172.21.15.46:0/4077405912 learned_addr learned my addr 172.21.15.46:0/4077405912 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.193+0000 7f0658f7f700 1 -- 172.21.15.46:0/4077405912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0654155c00 msgr2=0x7f065406bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.193+0000 7f0658f7f700 1 --2- 172.21.15.46:0/4077405912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0654155c00 0x7f065406bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.193+0000 7f0658f7f700 1 -- 172.21.15.46:0/4077405912 >> v2:172.21.15.106:3300/0 conn(0x7f0654155fd0 msgr2=0x7f065415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.193+0000 7f0658f7f700 1 --2- 172.21.15.46:0/4077405912 >> v2:172.21.15.106:3300/0 conn(0x7f0654155fd0 0x7f065415a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.193+0000 7f0658f7f700 1 -- 172.21.15.46:0/4077405912 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065406b0f0 con 0x7f065406a3c0 2024-01-27T01:08:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.193+0000 7f064bfff700 1 --2- 172.21.15.46:0/4077405912 >> v2:172.21.15.106:3300/0 conn(0x7f0654155fd0 0x7f065415a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.193+0000 7f0658f7f700 1 --2- 172.21.15.46:0/4077405912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f065406a3c0 0x7f0654068940 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f065406dc80 tx=0x7f064c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a949554f75f6f8d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.193+0000 7f06537fe700 1 -- 172.21.15.46:0/4077405912 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c00e070 con 0x7f065406a3c0 2024-01-27T01:08:18.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.193+0000 7f06537fe700 1 -- 172.21.15.46:0/4077405912 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064c00b970 con 0x7f065406a3c0 2024-01-27T01:08:18.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.193+0000 7f06537fe700 1 -- 172.21.15.46:0/4077405912 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c0092e0 con 0x7f065406a3c0 2024-01-27T01:08:18.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.194+0000 7f065a9e2700 1 -- 172.21.15.46:0/4077405912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f065406a3c0 msgr2=0x7f0654068940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:18.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.194+0000 7f065a9e2700 1 --2- 172.21.15.46:0/4077405912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f065406a3c0 0x7f0654068940 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f065406dc80 tx=0x7f064c009840 comp rx=0 tx=0).stop 2024-01-27T01:08:18.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.194+0000 7f065a9e2700 1 -- 172.21.15.46:0/4077405912 shutdown_connections 2024-01-27T01:08:18.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.194+0000 7f065a9e2700 1 --2- 172.21.15.46:0/4077405912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f065406a3c0 0x7f0654068940 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.194+0000 7f065a9e2700 1 --2- 172.21.15.46:0/4077405912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0654155c00 0x7f065406bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.194+0000 7f065a9e2700 1 --2- 172.21.15.46:0/4077405912 >> v2:172.21.15.106:3300/0 conn(0x7f0654155fd0 0x7f065415a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.194+0000 7f065a9e2700 1 -- 172.21.15.46:0/4077405912 >> 172.21.15.46:0/4077405912 conn(0x7f06540ca100 msgr2=0x7f065406e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:18.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.194+0000 7f065a9e2700 1 -- 172.21.15.46:0/4077405912 shutdown_connections 2024-01-27T01:08:18.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.194+0000 7f065a9e2700 1 -- 172.21.15.46:0/4077405912 wait complete. 2024-01-27T01:08:18.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.195+0000 7f065a9e2700 1 Processor -- start 2024-01-27T01:08:18.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.195+0000 7f065a9e2700 1 -- start start 2024-01-27T01:08:18.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.196+0000 7f065a9e2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f065406a3c0 0x7f0654151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.196+0000 7f065a9e2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0654155c00 0x7f06540b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.196+0000 7f065a9e2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0654155fd0 0x7f06540c2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.196+0000 7f065a9e2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06540b0700 con 0x7f065406a3c0 2024-01-27T01:08:18.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.196+0000 7f065a9e2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06540b09e0 con 0x7f0654155c00 2024-01-27T01:08:18.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.196+0000 7f065a9e2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06540b0cc0 con 0x7f0654155fd0 2024-01-27T01:08:18.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.196+0000 7f0653fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f065406a3c0 0x7f0654151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.196+0000 7f0653fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f065406a3c0 0x7f0654151340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49576/0 (socket says 172.21.15.46:49576) 2024-01-27T01:08:18.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.196+0000 7f0653fff700 1 -- 172.21.15.46:0/741918353 learned_addr learned my addr 172.21.15.46:0/741918353 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:18.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.196+0000 7f064bfff700 1 --2- 172.21.15.46:0/741918353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0654155c00 0x7f06540b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.196+0000 7f0658f7f700 1 --2- 172.21.15.46:0/741918353 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0654155fd0 0x7f06540c2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.197+0000 7f0653fff700 1 -- 172.21.15.46:0/741918353 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0654155fd0 msgr2=0x7f06540c2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:18.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.197+0000 7f0653fff700 1 --2- 172.21.15.46:0/741918353 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0654155fd0 0x7f06540c2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.197+0000 7f0653fff700 1 -- 172.21.15.46:0/741918353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0654155c00 msgr2=0x7f06540b2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:18.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.197+0000 7f0653fff700 1 --2- 172.21.15.46:0/741918353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0654155c00 0x7f06540b2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.197+0000 7f0653fff700 1 -- 172.21.15.46:0/741918353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063c009420 con 0x7f065406a3c0 2024-01-27T01:08:18.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.197+0000 7f0653fff700 1 --2- 172.21.15.46:0/741918353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f065406a3c0 0x7f0654151340 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f064400ec80 tx=0x7f0644009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:18.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.197+0000 7f0650ff9700 1 -- 172.21.15.46:0/741918353 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644009680 con 0x7f065406a3c0 2024-01-27T01:08:18.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.197+0000 7f0650ff9700 1 -- 172.21.15.46:0/741918353 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064400c850 con 0x7f065406a3c0 2024-01-27T01:08:18.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.197+0000 7f065a9e2700 1 -- 172.21.15.46:0/741918353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f064c009ae0 con 0x7f065406a3c0 2024-01-27T01:08:18.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.197+0000 7f0650ff9700 1 -- 172.21.15.46:0/741918353 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064400cbd0 con 0x7f065406a3c0 2024-01-27T01:08:18.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.197+0000 7f065a9e2700 1 -- 172.21.15.46:0/741918353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06540b1280 con 0x7f065406a3c0 2024-01-27T01:08:18.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.198+0000 7f065a9e2700 1 -- 172.21.15.46:0/741918353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06540b15f0 con 0x7f065406a3c0 2024-01-27T01:08:18.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.199+0000 7f0650ff9700 1 -- 172.21.15.46:0/741918353 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f0644013470 con 0x7f065406a3c0 2024-01-27T01:08:18.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.199+0000 7f0650ff9700 1 --2- 172.21.15.46:0/741918353 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f06340670e0 0x7f0634069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.199+0000 7f0650ff9700 1 -- 172.21.15.46:0/741918353 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 5665+0+0 (secure 0 0 0) 0x7f0644084de0 con 0x7f065406a3c0 2024-01-27T01:08:18.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.200+0000 7f064bfff700 1 --2- 172.21.15.46:0/741918353 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f06340670e0 0x7f0634069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.200+0000 7f064bfff700 1 --2- 172.21.15.46:0/741918353 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f06340670e0 0x7f0634069590 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f065416d9e0 tx=0x7f063c0091a0 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:18.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.204+0000 7f0650ff9700 1 -- 172.21.15.46:0/741918353 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0644051d50 con 0x7f065406a3c0 2024-01-27T01:08:18.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.410+0000 7f065a9e2700 1 -- 172.21.15.46:0/741918353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f06540b1410 con 0x7f065406a3c0 2024-01-27T01:08:18.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.410+0000 7f0650ff9700 1 -- 172.21.15.46:0/741918353 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v259) v1 ==== 93+0+10 (secure 0 0 0) 0x7f06440210a0 con 0x7f065406a3c0 2024-01-27T01:08:18.410 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-27T01:08:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.413+0000 7f065a9e2700 1 -- 172.21.15.46:0/741918353 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f06340670e0 msgr2=0x7f0634069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.413+0000 7f065a9e2700 1 --2- 172.21.15.46:0/741918353 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f06340670e0 0x7f0634069590 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f065416d9e0 tx=0x7f063c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.413+0000 7f065a9e2700 1 -- 172.21.15.46:0/741918353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f065406a3c0 msgr2=0x7f0654151340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.413+0000 7f065a9e2700 1 --2- 172.21.15.46:0/741918353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f065406a3c0 0x7f0654151340 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f064400ec80 tx=0x7f0644009120 comp rx=0 tx=0).stop 2024-01-27T01:08:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.413+0000 7f065a9e2700 1 -- 172.21.15.46:0/741918353 shutdown_connections 2024-01-27T01:08:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.413+0000 7f065a9e2700 1 --2- 172.21.15.46:0/741918353 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f06340670e0 0x7f0634069590 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.413+0000 7f065a9e2700 1 --2- 172.21.15.46:0/741918353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f065406a3c0 0x7f0654151340 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.413+0000 7f065a9e2700 1 --2- 172.21.15.46:0/741918353 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0654155fd0 0x7f06540c2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.413+0000 7f065a9e2700 1 --2- 172.21.15.46:0/741918353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0654155c00 0x7f06540b2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.413+0000 7f065a9e2700 1 -- 172.21.15.46:0/741918353 >> 172.21.15.46:0/741918353 conn(0x7f06540ca100 msgr2=0x7f06540ae310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:18.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.413+0000 7f065a9e2700 1 -- 172.21.15.46:0/741918353 shutdown_connections 2024-01-27T01:08:18.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.413+0000 7f065a9e2700 1 -- 172.21.15.46:0/741918353 wait complete. 2024-01-27T01:08:18.426 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-27T01:08:18.426 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:18.426+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:18.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.561+0000 7f1c42628700 1 Processor -- start 2024-01-27T01:08:18.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.561+0000 7f1c42628700 1 -- start start 2024-01-27T01:08:18.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.562+0000 7f1c42628700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c3c15ce40 0x7f1c3c0d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.562+0000 7f1c42628700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1c3c0cc1c0 0x7f1c3c0ade30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.562+0000 7f1c42628700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3c0cff70 con 0x7f1c3c15d210 2024-01-27T01:08:18.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.562+0000 7f1c42628700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c3c0cf400 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.562+0000 7f1c42628700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c0cf660 con 0x7f1c3c15ce40 2024-01-27T01:08:18.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.562+0000 7f1c3b7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1c3c15d210 0x7f1c3c0cbab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44976/0 (socket says 172.21.15.46:44976) 2024-01-27T01:08:18.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.562+0000 7f1c3b7fe700 1 -- 172.21.15.46:0/2126329783 learned_addr learned my addr 172.21.15.46:0/2126329783 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:18.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.562+0000 7f1c3bfff700 1 --2- 172.21.15.46:0/2126329783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c3c15ce40 0x7f1c3c0d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.562+0000 7f1c40bc5700 1 --2- 172.21.15.46:0/2126329783 >> v2:172.21.15.106:3300/0 conn(0x7f1c3c0cc1c0 0x7f1c3c0ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.562+0000 7f1c3affd700 1 -- 172.21.15.46:0/2126329783 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910375141 0 0) 0x7f1c3c0cff70 con 0x7f1c3c15d210 2024-01-27T01:08:18.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.563+0000 7f1c3affd700 1 -- 172.21.15.46:0/2126329783 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c24003570 con 0x7f1c3c15d210 2024-01-27T01:08:18.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.563+0000 7f1c40bc5700 1 -- 172.21.15.46:0/2126329783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c3c15ce40 msgr2=0x7f1c3c0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:18.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.563+0000 7f1c40bc5700 1 --2- 172.21.15.46:0/2126329783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c3c15ce40 0x7f1c3c0d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.563+0000 7f1c40bc5700 1 -- 172.21.15.46:0/2126329783 >> v1:172.21.15.46:6789/0 conn(0x7f1c3c15d210 legacy=0x7f1c3c0cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:18.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.563+0000 7f1c40bc5700 1 -- 172.21.15.46:0/2126329783 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3c0d0500 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.563+0000 7f1c3bfff700 1 --2- 172.21.15.46:0/2126329783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c3c15ce40 0x7f1c3c0d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:18.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.563+0000 7f1c40bc5700 1 --2- 172.21.15.46:0/2126329783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c3c0cc1c0 0x7f1c3c0ade30 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f1c34009e60 tx=0x7f1c34009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a92acc5dccf5a63d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:18.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.563+0000 7f1c3affd700 1 -- 172.21.15.46:0/2126329783 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3400e070 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.563+0000 7f1c3affd700 1 -- 172.21.15.46:0/2126329783 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3400bab0 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.563+0000 7f1c3affd700 1 -- 172.21.15.46:0/2126329783 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3400be70 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.564+0000 7f1c42628700 1 -- 172.21.15.46:0/2126329783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c3c0cc1c0 msgr2=0x7f1c3c0ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:18.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.564+0000 7f1c42628700 1 --2- 172.21.15.46:0/2126329783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c3c0cc1c0 0x7f1c3c0ade30 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f1c34009e60 tx=0x7f1c34009780 comp rx=0 tx=0).stop 2024-01-27T01:08:18.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.564+0000 7f1c42628700 1 -- 172.21.15.46:0/2126329783 shutdown_connections 2024-01-27T01:08:18.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.564+0000 7f1c42628700 1 --2- 172.21.15.46:0/2126329783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c3c0cc1c0 0x7f1c3c0ade30 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.564+0000 7f1c42628700 1 --2- 172.21.15.46:0/2126329783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c3c15ce40 0x7f1c3c0d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.564+0000 7f1c42628700 1 -- 172.21.15.46:0/2126329783 >> 172.21.15.46:0/2126329783 conn(0x7f1c3c06f840 msgr2=0x7f1c3c0d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:18.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.564+0000 7f1c42628700 1 -- 172.21.15.46:0/2126329783 shutdown_connections 2024-01-27T01:08:18.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.564+0000 7f1c42628700 1 -- 172.21.15.46:0/2126329783 wait complete. 2024-01-27T01:08:18.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.565+0000 7f1c42628700 1 Processor -- start 2024-01-27T01:08:18.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.565+0000 7f1c42628700 1 -- start start 2024-01-27T01:08:18.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.566+0000 7f1c42628700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3c0cc1c0 0x7f1c3c0c40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.566+0000 7f1c42628700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c3c15ce40 0x7f1c3c0c6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.566+0000 7f1c42628700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c3c15d210 0x7f1c3c159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.566+0000 7f1c42628700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c0c52b0 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.566+0000 7f1c42628700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c0c7490 con 0x7f1c3c15ce40 2024-01-27T01:08:18.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.566+0000 7f1c42628700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c0c46a0 con 0x7f1c3c15d210 2024-01-27T01:08:18.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.566+0000 7f1c3bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3c0cc1c0 0x7f1c3c0c40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.566+0000 7f1c3bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3c0cc1c0 0x7f1c3c0c40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49584/0 (socket says 172.21.15.46:49584) 2024-01-27T01:08:18.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.566+0000 7f1c3bfff700 1 -- 172.21.15.46:0/473642267 learned_addr learned my addr 172.21.15.46:0/473642267 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:18.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.566+0000 7f1c3b7fe700 1 --2- 172.21.15.46:0/473642267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c3c15ce40 0x7f1c3c0c6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.566+0000 7f1c40bc5700 1 --2- 172.21.15.46:0/473642267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c3c15d210 0x7f1c3c159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.566+0000 7f1c3bfff700 1 -- 172.21.15.46:0/473642267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c3c15d210 msgr2=0x7f1c3c159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:18.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.566+0000 7f1c3bfff700 1 --2- 172.21.15.46:0/473642267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c3c15d210 0x7f1c3c159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.566+0000 7f1c3bfff700 1 -- 172.21.15.46:0/473642267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c3c15ce40 msgr2=0x7f1c3c0c6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:18.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.566+0000 7f1c3bfff700 1 --2- 172.21.15.46:0/473642267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c3c15ce40 0x7f1c3c0c6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.566+0000 7f1c3bfff700 1 -- 172.21.15.46:0/473642267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c0093c0 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.566+0000 7f1c3b7fe700 1 --2- 172.21.15.46:0/473642267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c3c15ce40 0x7f1c3c0c6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:18.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.567+0000 7f1c3bfff700 1 --2- 172.21.15.46:0/473642267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3c0cc1c0 0x7f1c3c0c40c0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c00ec80 tx=0x7f1c2c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:18.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.567+0000 7f1c23fff700 1 -- 172.21.15.46:0/473642267 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c005a00 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.567+0000 7f1c23fff700 1 -- 172.21.15.46:0/473642267 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c2c005b60 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.567+0000 7f1c42628700 1 -- 172.21.15.46:0/473642267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c34009a20 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.567+0000 7f1c23fff700 1 -- 172.21.15.46:0/473642267 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c00c9d0 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.567+0000 7f1c42628700 1 -- 172.21.15.46:0/473642267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c3c0c4c60 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.568+0000 7f1c42628700 1 -- 172.21.15.46:0/473642267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c3c0c5590 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.569+0000 7f1c23fff700 1 -- 172.21.15.46:0/473642267 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f1c2c00cd80 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.569+0000 7f1c23fff700 1 --2- 172.21.15.46:0/473642267 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c280670e0 0x7f1c28069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.569+0000 7f1c23fff700 1 -- 172.21.15.46:0/473642267 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 5665+0+0 (secure 0 0 0) 0x7f1c2c084e50 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.569+0000 7f1c3b7fe700 1 --2- 172.21.15.46:0/473642267 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c280670e0 0x7f1c28069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.570+0000 7f1c3b7fe700 1 --2- 172.21.15.46:0/473642267 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c280670e0 0x7f1c28069590 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c0c3e60 tx=0x7f1c30009040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:18.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.574+0000 7f1c23fff700 1 -- 172.21.15.46:0/473642267 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1c2c051e60 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.785+0000 7f1c42628700 1 -- 172.21.15.46:0/473642267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1c3c0cf920 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.785+0000 7f1c23fff700 1 -- 172.21.15.46:0/473642267 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v259) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1c2c020020 con 0x7f1c3c0cc1c0 2024-01-27T01:08:18.784 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:18.784 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-27T01:08:18.784 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:18.785 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-01-27T01:08:18.785 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-01-27T01:08:18.785 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:18.785 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-27T01:08:18.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.787+0000 7f1c42628700 1 -- 172.21.15.46:0/473642267 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c280670e0 msgr2=0x7f1c28069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.787+0000 7f1c42628700 1 --2- 172.21.15.46:0/473642267 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c280670e0 0x7f1c28069590 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c0c3e60 tx=0x7f1c30009040 comp rx=0 tx=0).stop 2024-01-27T01:08:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.787+0000 7f1c42628700 1 -- 172.21.15.46:0/473642267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3c0cc1c0 msgr2=0x7f1c3c0c40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.787+0000 7f1c42628700 1 --2- 172.21.15.46:0/473642267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3c0cc1c0 0x7f1c3c0c40c0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c00ec80 tx=0x7f1c2c009120 comp rx=0 tx=0).stop 2024-01-27T01:08:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.787+0000 7f1c42628700 1 -- 172.21.15.46:0/473642267 shutdown_connections 2024-01-27T01:08:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.787+0000 7f1c42628700 1 --2- 172.21.15.46:0/473642267 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c280670e0 0x7f1c28069590 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.787+0000 7f1c42628700 1 --2- 172.21.15.46:0/473642267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3c0cc1c0 0x7f1c3c0c40c0 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.787+0000 7f1c42628700 1 --2- 172.21.15.46:0/473642267 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c3c15d210 0x7f1c3c159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.787+0000 7f1c42628700 1 --2- 172.21.15.46:0/473642267 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c3c15ce40 0x7f1c3c0c6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.787+0000 7f1c42628700 1 -- 172.21.15.46:0/473642267 >> 172.21.15.46:0/473642267 conn(0x7f1c3c06f840 msgr2=0x7f1c3c0b44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.788+0000 7f1c42628700 1 -- 172.21.15.46:0/473642267 shutdown_connections 2024-01-27T01:08:18.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.788+0000 7f1c42628700 1 -- 172.21.15.46:0/473642267 wait complete. 2024-01-27T01:08:18.800 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-27T01:08:18.801 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-27T01:08:18.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.939+0000 7f9d3c534700 1 Processor -- start 2024-01-27T01:08:18.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.939+0000 7f9d3c534700 1 -- start start 2024-01-27T01:08:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.940+0000 7f9d3c534700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9d340ca4b0 0x7f9d340afe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.940+0000 7f9d3c534700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d340b3a50 0x7f9d340b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.940+0000 7f9d3c534700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d340b43d0 con 0x7f9d340ca0e0 2024-01-27T01:08:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.940+0000 7f9d3c534700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d3406c7c0 con 0x7f9d340b3a50 2024-01-27T01:08:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.940+0000 7f9d3c534700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d340cada0 con 0x7f9d340ca4b0 2024-01-27T01:08:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.940+0000 7f9d3aad1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d340b3a50 0x7f9d340b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.940+0000 7f9d3a2d0700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9d340ca0e0 0x7f9d3406c0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44982/0 (socket says 172.21.15.46:44982) 2024-01-27T01:08:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.940+0000 7f9d3a2d0700 1 -- 172.21.15.46:0/212104899 learned_addr learned my addr 172.21.15.46:0/212104899 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.940+0000 7f9d39acf700 1 --2- 172.21.15.46:0/212104899 >> v2:172.21.15.106:3300/0 conn(0x7f9d340ca4b0 0x7f9d340afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.940+0000 7f9d3aad1700 1 -- 172.21.15.46:0/212104899 >> v2:172.21.15.106:3300/0 conn(0x7f9d340ca4b0 msgr2=0x7f9d340afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.940+0000 7f9d3aad1700 1 --2- 172.21.15.46:0/212104899 >> v2:172.21.15.106:3300/0 conn(0x7f9d340ca4b0 0x7f9d340afe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.940+0000 7f9d3aad1700 1 -- 172.21.15.46:0/212104899 >> v1:172.21.15.46:6789/0 conn(0x7f9d340ca0e0 legacy=0x7f9d3406c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.940+0000 7f9d3aad1700 1 -- 172.21.15.46:0/212104899 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d34158010 con 0x7f9d340b3a50 2024-01-27T01:08:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.941+0000 7f9d3aad1700 1 --2- 172.21.15.46:0/212104899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d340b3a50 0x7f9d340b3e90 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f9d34155790 tx=0x7f9d2c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e0dc02b5950819d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.941+0000 7f9d392ce700 1 -- 172.21.15.46:0/212104899 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c00e070 con 0x7f9d340b3a50 2024-01-27T01:08:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.941+0000 7f9d392ce700 1 -- 172.21.15.46:0/212104899 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2c00bb60 con 0x7f9d340b3a50 2024-01-27T01:08:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.941+0000 7f9d392ce700 1 -- 172.21.15.46:0/212104899 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c012430 con 0x7f9d340b3a50 2024-01-27T01:08:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.941+0000 7f9d3c534700 1 -- 172.21.15.46:0/212104899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d340b3a50 msgr2=0x7f9d340b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.941+0000 7f9d3c534700 1 --2- 172.21.15.46:0/212104899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d340b3a50 0x7f9d340b3e90 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f9d34155790 tx=0x7f9d2c009780 comp rx=0 tx=0).stop 2024-01-27T01:08:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.941+0000 7f9d3c534700 1 -- 172.21.15.46:0/212104899 shutdown_connections 2024-01-27T01:08:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.941+0000 7f9d3c534700 1 --2- 172.21.15.46:0/212104899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d340b3a50 0x7f9d340b3e90 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.941+0000 7f9d3c534700 1 --2- 172.21.15.46:0/212104899 >> v2:172.21.15.106:3300/0 conn(0x7f9d340ca4b0 0x7f9d340afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.941+0000 7f9d3c534700 1 -- 172.21.15.46:0/212104899 >> 172.21.15.46:0/212104899 conn(0x7f9d340ce530 msgr2=0x7f9d340ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.941+0000 7f9d3c534700 1 -- 172.21.15.46:0/212104899 shutdown_connections 2024-01-27T01:08:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.941+0000 7f9d3c534700 1 -- 172.21.15.46:0/212104899 wait complete. 2024-01-27T01:08:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.943+0000 7f9d3c534700 1 Processor -- start 2024-01-27T01:08:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.943+0000 7f9d3c534700 1 -- start start 2024-01-27T01:08:18.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.943+0000 7f9d3c534700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d340b3a50 0x7f9d34151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.943+0000 7f9d3c534700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d340ca0e0 0x7f9d340692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.943+0000 7f9d3c534700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d340ca4b0 0x7f9d340b2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d3c534700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3415af70 con 0x7f9d340b3a50 2024-01-27T01:08:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d3c534700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3415a6d0 con 0x7f9d340ca4b0 2024-01-27T01:08:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d3c534700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3415a9b0 con 0x7f9d340ca0e0 2024-01-27T01:08:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d3a2d0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d340b3a50 0x7f9d34151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d3a2d0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d340b3a50 0x7f9d34151640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49590/0 (socket says 172.21.15.46:49590) 2024-01-27T01:08:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d3a2d0700 1 -- 172.21.15.46:0/4277758994 learned_addr learned my addr 172.21.15.46:0/4277758994 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d39acf700 1 --2- 172.21.15.46:0/4277758994 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d340ca0e0 0x7f9d340692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d3aad1700 1 --2- 172.21.15.46:0/4277758994 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d340ca4b0 0x7f9d340b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d3a2d0700 1 -- 172.21.15.46:0/4277758994 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d340ca0e0 msgr2=0x7f9d340692b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d3a2d0700 1 --2- 172.21.15.46:0/4277758994 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d340ca0e0 0x7f9d340692b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d3a2d0700 1 -- 172.21.15.46:0/4277758994 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d340ca4b0 msgr2=0x7f9d340b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d3a2d0700 1 --2- 172.21.15.46:0/4277758994 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d340ca4b0 0x7f9d340b2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d3a2d0700 1 -- 172.21.15.46:0/4277758994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2c009a20 con 0x7f9d340b3a50 2024-01-27T01:08:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d3a2d0700 1 --2- 172.21.15.46:0/4277758994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d340b3a50 0x7f9d34151640 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000c560 tx=0x7f9d30001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d2a7fc700 1 -- 172.21.15.46:0/4277758994 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d30005b30 con 0x7f9d340b3a50 2024-01-27T01:08:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d2a7fc700 1 -- 172.21.15.46:0/4277758994 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d30005c90 con 0x7f9d340b3a50 2024-01-27T01:08:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d3c534700 1 -- 172.21.15.46:0/4277758994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3415ac90 con 0x7f9d340b3a50 2024-01-27T01:08:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.944+0000 7f9d2a7fc700 1 -- 172.21.15.46:0/4277758994 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d30007320 con 0x7f9d340b3a50 2024-01-27T01:08:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.945+0000 7f9d3c534700 1 -- 172.21.15.46:0/4277758994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3415a380 con 0x7f9d340b3a50 2024-01-27T01:08:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.945+0000 7f9d3c534700 1 -- 172.21.15.46:0/4277758994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3404c710 con 0x7f9d340b3a50 2024-01-27T01:08:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.946+0000 7f9d2a7fc700 1 -- 172.21.15.46:0/4277758994 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f9d3001b3b0 con 0x7f9d340b3a50 2024-01-27T01:08:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.946+0000 7f9d2a7fc700 1 --2- 172.21.15.46:0/4277758994 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9d180670e0 0x7f9d18069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.947+0000 7f9d2a7fc700 1 -- 172.21.15.46:0/4277758994 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 5665+0+0 (secure 0 0 0) 0x7f9d30083bc0 con 0x7f9d340b3a50 2024-01-27T01:08:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.947+0000 7f9d39acf700 1 --2- 172.21.15.46:0/4277758994 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9d180670e0 0x7f9d18069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.947+0000 7f9d39acf700 1 --2- 172.21.15.46:0/4277758994 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9d180670e0 0x7f9d18069590 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9d3406ac20 tx=0x7f9d240091a0 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:18.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:18.951+0000 7f9d2a7fc700 1 -- 172.21.15.46:0/4277758994 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9d30050bd0 con 0x7f9d340b3a50 2024-01-27T01:08:19.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.161+0000 7f9d3c534700 1 -- 172.21.15.46:0/4277758994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9d340c88b0 con 0x7f9d340b3a50 2024-01-27T01:08:19.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.251+0000 7f9d2a7fc700 1 -- 172.21.15.46:0/4277758994 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v260) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9d300170b0 con 0x7f9d340b3a50 2024-01-27T01:08:19.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.254+0000 7f9d3c534700 1 -- 172.21.15.46:0/4277758994 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9d180670e0 msgr2=0x7f9d18069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:19.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.254+0000 7f9d3c534700 1 --2- 172.21.15.46:0/4277758994 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9d180670e0 0x7f9d18069590 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9d3406ac20 tx=0x7f9d240091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:19.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.254+0000 7f9d3c534700 1 -- 172.21.15.46:0/4277758994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d340b3a50 msgr2=0x7f9d34151640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:19.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.254+0000 7f9d3c534700 1 --2- 172.21.15.46:0/4277758994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d340b3a50 0x7f9d34151640 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000c560 tx=0x7f9d30001020 comp rx=0 tx=0).stop 2024-01-27T01:08:19.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.254+0000 7f9d3c534700 1 -- 172.21.15.46:0/4277758994 shutdown_connections 2024-01-27T01:08:19.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.254+0000 7f9d3c534700 1 --2- 172.21.15.46:0/4277758994 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9d180670e0 0x7f9d18069590 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:19.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.254+0000 7f9d3c534700 1 --2- 172.21.15.46:0/4277758994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d340b3a50 0x7f9d34151640 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:19.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.254+0000 7f9d3c534700 1 --2- 172.21.15.46:0/4277758994 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d340ca4b0 0x7f9d340b2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:19.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.254+0000 7f9d3c534700 1 --2- 172.21.15.46:0/4277758994 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d340ca0e0 0x7f9d340692b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:19.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.254+0000 7f9d3c534700 1 -- 172.21.15.46:0/4277758994 >> 172.21.15.46:0/4277758994 conn(0x7f9d340ce530 msgr2=0x7f9d340cea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:19.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.254+0000 7f9d3c534700 1 -- 172.21.15.46:0/4277758994 shutdown_connections 2024-01-27T01:08:19.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.254+0000 7f9d3c534700 1 -- 172.21.15.46:0/4277758994 wait complete. 2024-01-27T01:08:19.255 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-01-27T01:08:19.271 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-27T01:08:19.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.418+0000 7f1cfe7bf700 1 Processor -- start 2024-01-27T01:08:19.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.418+0000 7f1cfe7bf700 1 -- start start 2024-01-27T01:08:19.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.419+0000 7f1cfe7bf700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1cf8159cb0 0x7f1cf80cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:19.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.419+0000 7f1cfe7bf700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf80cc100 con 0x7f1cf80afb40 2024-01-27T01:08:19.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.419+0000 7f1cfe7bf700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf80cb850 con 0x7f1cf80cf050 2024-01-27T01:08:19.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.419+0000 7f1cfe7bf700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf80cbaf0 con 0x7f1cf8159cb0 2024-01-27T01:08:19.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.419+0000 7f1cfcd5c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1cf80afb40 0x7f1cf80c5530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44990/0 (socket says 172.21.15.46:44990) 2024-01-27T01:08:19.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.419+0000 7f1cfcd5c700 1 -- 172.21.15.46:0/734478742 learned_addr learned my addr 172.21.15.46:0/734478742 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:19.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.419+0000 7f1cf7fff700 1 --2- 172.21.15.46:0/734478742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1cf8159cb0 0x7f1cf80cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:19.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.419+0000 7f1cf6ffd700 1 -- 172.21.15.46:0/734478742 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330369750 0 0) 0x7f1cf80cc100 con 0x7f1cf80afb40 2024-01-27T01:08:19.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.419+0000 7f1cf6ffd700 1 -- 172.21.15.46:0/734478742 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce0003570 con 0x7f1cf80afb40 2024-01-27T01:08:19.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.419+0000 7f1cf6ffd700 1 -- 172.21.15.46:0/734478742 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884116277 0 0) 0x7f1cf80cb850 con 0x7f1cf80cf050 2024-01-27T01:08:19.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.419+0000 7f1cf6ffd700 1 -- 172.21.15.46:0/734478742 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf80cc100 con 0x7f1cf80cf050 2024-01-27T01:08:19.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.419+0000 7f1cf7fff700 1 -- 172.21.15.46:0/734478742 >> v1:172.21.15.106:6789/0 conn(0x7f1cf80cf050 legacy=0x7f1cf80af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:19.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.419+0000 7f1cf7fff700 1 -- 172.21.15.46:0/734478742 >> v1:172.21.15.46:6789/0 conn(0x7f1cf80afb40 legacy=0x7f1cf80c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:19.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.419+0000 7f1cf7fff700 1 -- 172.21.15.46:0/734478742 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf80cfc80 con 0x7f1cf8159cb0 2024-01-27T01:08:19.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.420+0000 7f1cf7fff700 1 --2- 172.21.15.46:0/734478742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1cf8159cb0 0x7f1cf80cc590 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8009900 tx=0x7f1ce8009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20da30d0260d311f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:19.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.420+0000 7f1cf6ffd700 1 -- 172.21.15.46:0/734478742 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8005c10 con 0x7f1cf8159cb0 2024-01-27T01:08:19.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.420+0000 7f1cf6ffd700 1 -- 172.21.15.46:0/734478742 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce8005d70 con 0x7f1cf8159cb0 2024-01-27T01:08:19.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.420+0000 7f1cf6ffd700 1 -- 172.21.15.46:0/734478742 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8006260 con 0x7f1cf8159cb0 2024-01-27T01:08:19.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.420+0000 7f1cfe7bf700 1 -- 172.21.15.46:0/734478742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1cf8159cb0 msgr2=0x7f1cf80cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:19.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.420+0000 7f1cfe7bf700 1 --2- 172.21.15.46:0/734478742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1cf8159cb0 0x7f1cf80cc590 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8009900 tx=0x7f1ce8009120 comp rx=0 tx=0).stop 2024-01-27T01:08:19.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.420+0000 7f1cfe7bf700 1 -- 172.21.15.46:0/734478742 shutdown_connections 2024-01-27T01:08:19.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.420+0000 7f1cfe7bf700 1 --2- 172.21.15.46:0/734478742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1cf8159cb0 0x7f1cf80cc590 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:19.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.420+0000 7f1cfe7bf700 1 -- 172.21.15.46:0/734478742 >> 172.21.15.46:0/734478742 conn(0x7f1cf8087f60 msgr2=0x7f1cf806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:19.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.421+0000 7f1cfe7bf700 1 -- 172.21.15.46:0/734478742 shutdown_connections 2024-01-27T01:08:19.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.421+0000 7f1cfe7bf700 1 -- 172.21.15.46:0/734478742 wait complete. 2024-01-27T01:08:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.422+0000 7f1cfe7bf700 1 Processor -- start 2024-01-27T01:08:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.422+0000 7f1cfe7bf700 1 -- start start 2024-01-27T01:08:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cfe7bf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1cf80afb40 0x7f1cf80b5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cfe7bf700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1cf80cf050 0x7f1cf81581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cfe7bf700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1cf8159cb0 0x7f1cf816d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cfe7bf700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf80c9f10 con 0x7f1cf80afb40 2024-01-27T01:08:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cfe7bf700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf8159220 con 0x7f1cf8159cb0 2024-01-27T01:08:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cfe7bf700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf8159500 con 0x7f1cf80cf050 2024-01-27T01:08:19.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cf7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1cf80afb40 0x7f1cf80b5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cf7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1cf80afb40 0x7f1cf80b5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49596/0 (socket says 172.21.15.46:49596) 2024-01-27T01:08:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cf7fff700 1 -- 172.21.15.46:0/2014896852 learned_addr learned my addr 172.21.15.46:0/2014896852 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cfcd5c700 1 --2- 172.21.15.46:0/2014896852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1cf8159cb0 0x7f1cf816d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cf77fe700 1 --2- 172.21.15.46:0/2014896852 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1cf80cf050 0x7f1cf81581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cf77fe700 1 -- 172.21.15.46:0/2014896852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1cf8159cb0 msgr2=0x7f1cf816d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cf77fe700 1 --2- 172.21.15.46:0/2014896852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1cf8159cb0 0x7f1cf816d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cf77fe700 1 -- 172.21.15.46:0/2014896852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1cf80afb40 msgr2=0x7f1cf80b5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cf77fe700 1 --2- 172.21.15.46:0/2014896852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1cf80afb40 0x7f1cf80b5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cf77fe700 1 -- 172.21.15.46:0/2014896852 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce80093c0 con 0x7f1cf80cf050 2024-01-27T01:08:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cf7fff700 1 --2- 172.21.15.46:0/2014896852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1cf80afb40 0x7f1cf80b5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.423+0000 7f1cf77fe700 1 --2- 172.21.15.46:0/2014896852 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1cf80cf050 0x7f1cf81581e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00c560 tx=0x7f1cec007d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.424+0000 7f1cdffff700 1 -- 172.21.15.46:0/2014896852 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec011590 con 0x7f1cf80cf050 2024-01-27T01:08:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.424+0000 7f1cdffff700 1 -- 172.21.15.46:0/2014896852 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec0116f0 con 0x7f1cf80cf050 2024-01-27T01:08:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.424+0000 7f1cfe7bf700 1 -- 172.21.15.46:0/2014896852 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf80c98c0 con 0x7f1cf80cf050 2024-01-27T01:08:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.424+0000 7f1cdffff700 1 -- 172.21.15.46:0/2014896852 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec011ab0 con 0x7f1cf80cf050 2024-01-27T01:08:19.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.424+0000 7f1cfe7bf700 1 -- 172.21.15.46:0/2014896852 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf80b6480 con 0x7f1cf80cf050 2024-01-27T01:08:19.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.425+0000 7f1cfe7bf700 1 -- 172.21.15.46:0/2014896852 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf80ca0c0 con 0x7f1cf80cf050 2024-01-27T01:08:19.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.425+0000 7f1cdffff700 1 -- 172.21.15.46:0/2014896852 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f1cec01c460 con 0x7f1cf80cf050 2024-01-27T01:08:19.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.426+0000 7f1cdffff700 1 --2- 172.21.15.46:0/2014896852 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ce40670e0 0x7f1ce4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:19.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.426+0000 7f1cdffff700 1 -- 172.21.15.46:0/2014896852 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 6035+0+0 (secure 0 0 0) 0x7f1cec0845f0 con 0x7f1cf80cf050 2024-01-27T01:08:19.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.426+0000 7f1cf7fff700 1 --2- 172.21.15.46:0/2014896852 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ce40670e0 0x7f1ce4069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:19.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.426+0000 7f1cf7fff700 1 --2- 172.21.15.46:0/2014896852 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ce40670e0 0x7f1ce4069590 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8009360 tx=0x7f1ce8016000 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:19.427 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:19.423+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:19.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.430+0000 7f1cdffff700 1 -- 172.21.15.46:0/2014896852 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1cf80ca0c0 con 0x7f1cf80cf050 2024-01-27T01:08:19.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:19.637+0000 7f1cfe7bf700 1 -- 172.21.15.46:0/2014896852 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1cf80ca080 con 0x7f1cf80cf050 2024-01-27T01:08:20.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.256+0000 7f1cdffff700 1 -- 172.21.15.46:0/2014896852 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v261) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1cec0513f0 con 0x7f1cf80cf050 2024-01-27T01:08:20.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.264+0000 7f1cfe7bf700 1 -- 172.21.15.46:0/2014896852 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ce40670e0 msgr2=0x7f1ce4069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:20.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.264+0000 7f1cfe7bf700 1 --2- 172.21.15.46:0/2014896852 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ce40670e0 0x7f1ce4069590 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8009360 tx=0x7f1ce8016000 comp rx=0 tx=0).stop 2024-01-27T01:08:20.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.264+0000 7f1cfe7bf700 1 -- 172.21.15.46:0/2014896852 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1cf80cf050 msgr2=0x7f1cf81581e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:20.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.264+0000 7f1cfe7bf700 1 --2- 172.21.15.46:0/2014896852 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1cf80cf050 0x7f1cf81581e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00c560 tx=0x7f1cec007d50 comp rx=0 tx=0).stop 2024-01-27T01:08:20.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.265+0000 7f1cfe7bf700 1 -- 172.21.15.46:0/2014896852 shutdown_connections 2024-01-27T01:08:20.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.265+0000 7f1cfe7bf700 1 --2- 172.21.15.46:0/2014896852 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1ce40670e0 0x7f1ce4069590 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:20.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.265+0000 7f1cfe7bf700 1 --2- 172.21.15.46:0/2014896852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1cf80afb40 0x7f1cf80b5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:20.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.265+0000 7f1cfe7bf700 1 --2- 172.21.15.46:0/2014896852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1cf8159cb0 0x7f1cf816d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:20.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.265+0000 7f1cfe7bf700 1 --2- 172.21.15.46:0/2014896852 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1cf80cf050 0x7f1cf81581e0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:20.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.265+0000 7f1cfe7bf700 1 -- 172.21.15.46:0/2014896852 >> 172.21.15.46:0/2014896852 conn(0x7f1cf8087f60 msgr2=0x7f1cf80c6260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:20.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.265+0000 7f1cfe7bf700 1 -- 172.21.15.46:0/2014896852 shutdown_connections 2024-01-27T01:08:20.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.266+0000 7f1cfe7bf700 1 -- 172.21.15.46:0/2014896852 wait complete. 2024-01-27T01:08:20.268 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-01-27T01:08:20.288 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-27T01:08:20.424 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:20.424+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:20.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.428+0000 7f125589d700 1 Processor -- start 2024-01-27T01:08:20.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.429+0000 7f125589d700 1 -- start start 2024-01-27T01:08:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.429+0000 7f125589d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f125006b7f0 0x7f125006a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.429+0000 7f125589d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f125006db60 con 0x7f125006a570 2024-01-27T01:08:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.429+0000 7f125589d700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12500a9e60 con 0x7f12500a9980 2024-01-27T01:08:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.429+0000 7f125589d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12500aa140 con 0x7f125006b7f0 2024-01-27T01:08:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.430+0000 7f124e7fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f125006b7f0 0x7f125006a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.430+0000 7f124effd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f125006a570 0x7f125006e2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51678/0 (socket says 172.21.15.46:51678) 2024-01-27T01:08:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.430+0000 7f124effd700 1 -- 172.21.15.46:0/1130892366 learned_addr learned my addr 172.21.15.46:0/1130892366 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.430+0000 7f124e7fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f125006b7f0 0x7f125006a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38918/0 (socket says 172.21.15.46:38918) 2024-01-27T01:08:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.430+0000 7f124e7fc700 1 -- 172.21.15.46:0/1130892366 >> v1:172.21.15.106:6789/0 conn(0x7f12500a9980 legacy=0x7f12500aba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.430+0000 7f124e7fc700 1 -- 172.21.15.46:0/1130892366 >> v1:172.21.15.46:6789/0 conn(0x7f125006a570 legacy=0x7f125006e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.430+0000 7f124dffb700 1 -- 172.21.15.46:0/1130892366 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623431639 0 0) 0x7f12500a9e60 con 0x7f12500a9980 2024-01-27T01:08:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.430+0000 7f124e7fc700 1 -- 172.21.15.46:0/1130892366 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12500aa420 con 0x7f125006b7f0 2024-01-27T01:08:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.430+0000 7f124e7fc700 1 --2- 172.21.15.46:0/1130892366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f125006b7f0 0x7f125006a940 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f12440098e0 tx=0x7f12440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d504e0d9927bb65d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.430+0000 7f124dffb700 1 -- 172.21.15.46:0/1130892366 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1244009e80 con 0x7f125006b7f0 2024-01-27T01:08:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.430+0000 7f124dffb700 1 -- 172.21.15.46:0/1130892366 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f124400b760 con 0x7f125006b7f0 2024-01-27T01:08:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.430+0000 7f124dffb700 1 -- 172.21.15.46:0/1130892366 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124400bae0 con 0x7f125006b7f0 2024-01-27T01:08:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.431+0000 7f125589d700 1 -- 172.21.15.46:0/1130892366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f125006b7f0 msgr2=0x7f125006a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.431+0000 7f125589d700 1 --2- 172.21.15.46:0/1130892366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f125006b7f0 0x7f125006a940 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f12440098e0 tx=0x7f12440091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.431+0000 7f125589d700 1 -- 172.21.15.46:0/1130892366 shutdown_connections 2024-01-27T01:08:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.431+0000 7f125589d700 1 --2- 172.21.15.46:0/1130892366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f125006b7f0 0x7f125006a940 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.431+0000 7f125589d700 1 -- 172.21.15.46:0/1130892366 >> 172.21.15.46:0/1130892366 conn(0x7f1250067930 msgr2=0x7f1250067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.431+0000 7f125589d700 1 -- 172.21.15.46:0/1130892366 shutdown_connections 2024-01-27T01:08:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.431+0000 7f125589d700 1 -- 172.21.15.46:0/1130892366 wait complete. 2024-01-27T01:08:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.432+0000 7f125589d700 1 Processor -- start 2024-01-27T01:08:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.432+0000 7f125589d700 1 -- start start 2024-01-27T01:08:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.433+0000 7f125589d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f125006a570 0x7f125016a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.433+0000 7f125589d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f125006b7f0 0x7f125014c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.433+0000 7f125589d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12500a9980 0x7f12500cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.433+0000 7f125589d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125016adf0 con 0x7f125006b7f0 2024-01-27T01:08:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.433+0000 7f125589d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12500ac180 con 0x7f125006a570 2024-01-27T01:08:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.433+0000 7f125589d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12500cb550 con 0x7f12500a9980 2024-01-27T01:08:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.433+0000 7f124e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f125006b7f0 0x7f125014c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.433+0000 7f124e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f125006b7f0 0x7f125014c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38920/0 (socket says 172.21.15.46:38920) 2024-01-27T01:08:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.433+0000 7f124e7fc700 1 -- 172.21.15.46:0/2827504526 learned_addr learned my addr 172.21.15.46:0/2827504526 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.434+0000 7f124effd700 1 --2- 172.21.15.46:0/2827504526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f125006a570 0x7f125016a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.434+0000 7f124f7fe700 1 --2- 172.21.15.46:0/2827504526 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12500a9980 0x7f12500cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.434+0000 7f124e7fc700 1 -- 172.21.15.46:0/2827504526 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12500a9980 msgr2=0x7f12500cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.434+0000 7f124e7fc700 1 --2- 172.21.15.46:0/2827504526 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12500a9980 0x7f12500cb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.434+0000 7f124e7fc700 1 -- 172.21.15.46:0/2827504526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f125006a570 msgr2=0x7f125016a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.434+0000 7f124e7fc700 1 --2- 172.21.15.46:0/2827504526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f125006a570 0x7f125016a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.434+0000 7f124e7fc700 1 -- 172.21.15.46:0/2827504526 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1244009420 con 0x7f125006b7f0 2024-01-27T01:08:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.434+0000 7f124f7fe700 1 --2- 172.21.15.46:0/2827504526 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12500a9980 0x7f12500cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.434+0000 7f124e7fc700 1 --2- 172.21.15.46:0/2827504526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f125006b7f0 0x7f125014c220 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f12440030c0 tx=0x7f1244016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.434+0000 7f1236ffd700 1 -- 172.21.15.46:0/2827504526 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124400bae0 con 0x7f125006b7f0 2024-01-27T01:08:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.434+0000 7f1236ffd700 1 -- 172.21.15.46:0/2827504526 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f124401c460 con 0x7f125006b7f0 2024-01-27T01:08:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.434+0000 7f125589d700 1 -- 172.21.15.46:0/2827504526 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12500cb810 con 0x7f125006b7f0 2024-01-27T01:08:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.434+0000 7f1236ffd700 1 -- 172.21.15.46:0/2827504526 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124401c820 con 0x7f125006b7f0 2024-01-27T01:08:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.434+0000 7f125589d700 1 -- 172.21.15.46:0/2827504526 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12500abb30 con 0x7f125006b7f0 2024-01-27T01:08:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.436+0000 7f1236ffd700 1 -- 172.21.15.46:0/2827504526 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f124401cbd0 con 0x7f125006b7f0 2024-01-27T01:08:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.436+0000 7f1236ffd700 1 --2- 172.21.15.46:0/2827504526 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f123c0670e0 0x7f123c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.436+0000 7f1236ffd700 1 -- 172.21.15.46:0/2827504526 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f1244087020 con 0x7f125006b7f0 2024-01-27T01:08:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.436+0000 7f125589d700 1 -- 172.21.15.46:0/2827504526 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f125004c710 con 0x7f125006b7f0 2024-01-27T01:08:20.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.436+0000 7f124effd700 1 --2- 172.21.15.46:0/2827504526 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f123c0670e0 0x7f123c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:20.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.437+0000 7f124effd700 1 --2- 172.21.15.46:0/2827504526 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f123c0670e0 0x7f123c069590 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1240000ef0 tx=0x7f1240009040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:20.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.442+0000 7f1236ffd700 1 -- 172.21.15.46:0/2827504526 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1244053d20 con 0x7f125006b7f0 2024-01-27T01:08:20.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:20.702+0000 7f125589d700 1 -- 172.21.15.46:0/2827504526 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f125004c710 con 0x7f125006b7f0 2024-01-27T01:08:21.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.267+0000 7f1236ffd700 1 -- 172.21.15.46:0/2827504526 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 35 and data pool 36 v145) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1244057340 con 0x7f125006b7f0 2024-01-27T01:08:21.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.282+0000 7f125589d700 1 -- 172.21.15.46:0/2827504526 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f123c0670e0 msgr2=0x7f123c069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:21.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.282+0000 7f125589d700 1 --2- 172.21.15.46:0/2827504526 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f123c0670e0 0x7f123c069590 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1240000ef0 tx=0x7f1240009040 comp rx=0 tx=0).stop 2024-01-27T01:08:21.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.282+0000 7f125589d700 1 -- 172.21.15.46:0/2827504526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f125006b7f0 msgr2=0x7f125014c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:21.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.282+0000 7f125589d700 1 --2- 172.21.15.46:0/2827504526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f125006b7f0 0x7f125014c220 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f12440030c0 tx=0x7f1244016000 comp rx=0 tx=0).stop 2024-01-27T01:08:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.283+0000 7f125589d700 1 -- 172.21.15.46:0/2827504526 shutdown_connections 2024-01-27T01:08:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.283+0000 7f125589d700 1 --2- 172.21.15.46:0/2827504526 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f123c0670e0 0x7f123c069590 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.283+0000 7f125589d700 1 --2- 172.21.15.46:0/2827504526 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12500a9980 0x7f12500cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.283+0000 7f125589d700 1 --2- 172.21.15.46:0/2827504526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f125006a570 0x7f125016a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.283+0000 7f125589d700 1 --2- 172.21.15.46:0/2827504526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f125006b7f0 0x7f125014c220 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.283+0000 7f125589d700 1 -- 172.21.15.46:0/2827504526 >> 172.21.15.46:0/2827504526 conn(0x7f1250067930 msgr2=0x7f1250158d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.283+0000 7f125589d700 1 -- 172.21.15.46:0/2827504526 shutdown_connections 2024-01-27T01:08:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.283+0000 7f125589d700 1 -- 172.21.15.46:0/2827504526 wait complete. 2024-01-27T01:08:21.290 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 35 and data pool 36 2024-01-27T01:08:21.304 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:08:21.431 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:21.431+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:21.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.442+0000 7f5d6b620700 1 Processor -- start 2024-01-27T01:08:21.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.442+0000 7f5d6b620700 1 -- start start 2024-01-27T01:08:21.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.443+0000 7f5d6b620700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5d640b2530 0x7f5d6415a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:21.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.443+0000 7f5d6b620700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5d6415da10 0x7f5d6415de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.443+0000 7f5d6b620700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d640a9b90 con 0x7f5d640b2160 2024-01-27T01:08:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.443+0000 7f5d6b620700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d640aa210 con 0x7f5d6415da10 2024-01-27T01:08:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.443+0000 7f5d6b620700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d6415d580 con 0x7f5d640b2530 2024-01-27T01:08:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.443+0000 7f5d69bbd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5d6415da10 0x7f5d6415de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.443+0000 7f5d693bc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5d640b2160 0x7f5d640abe50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51686/0 (socket says 172.21.15.46:51686) 2024-01-27T01:08:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.444+0000 7f5d693bc700 1 -- 172.21.15.46:0/747650011 learned_addr learned my addr 172.21.15.46:0/747650011 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.444+0000 7f5d68bbb700 1 --2- 172.21.15.46:0/747650011 >> v2:172.21.15.106:3300/0 conn(0x7f5d640b2530 0x7f5d6415a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.444+0000 7f5d69bbd700 1 -- 172.21.15.46:0/747650011 >> v2:172.21.15.106:3300/0 conn(0x7f5d640b2530 msgr2=0x7f5d6415a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.444+0000 7f5d69bbd700 1 --2- 172.21.15.46:0/747650011 >> v2:172.21.15.106:3300/0 conn(0x7f5d640b2530 0x7f5d6415a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.444+0000 7f5d69bbd700 1 -- 172.21.15.46:0/747650011 >> v1:172.21.15.46:6789/0 conn(0x7f5d640b2160 legacy=0x7f5d640abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.444+0000 7f5d5bfff700 1 -- 172.21.15.46:0/747650011 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308995227 0 0) 0x7f5d640a9b90 con 0x7f5d640b2160 2024-01-27T01:08:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.444+0000 7f5d69bbd700 1 -- 172.21.15.46:0/747650011 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d640aa460 con 0x7f5d6415da10 2024-01-27T01:08:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.444+0000 7f5d69bbd700 1 --2- 172.21.15.46:0/747650011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d6415da10 0x7f5d6415de30 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f5d64063c70 tx=0x7f5d5c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f515b7acfa34e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.444+0000 7f5d5bfff700 1 -- 172.21.15.46:0/747650011 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c00e070 con 0x7f5d6415da10 2024-01-27T01:08:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.444+0000 7f5d5bfff700 1 -- 172.21.15.46:0/747650011 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d5c00bc30 con 0x7f5d6415da10 2024-01-27T01:08:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.444+0000 7f5d5bfff700 1 -- 172.21.15.46:0/747650011 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c012430 con 0x7f5d6415da10 2024-01-27T01:08:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.445+0000 7f5d6b620700 1 -- 172.21.15.46:0/747650011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d6415da10 msgr2=0x7f5d6415de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.445+0000 7f5d6b620700 1 --2- 172.21.15.46:0/747650011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d6415da10 0x7f5d6415de30 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f5d64063c70 tx=0x7f5d5c009780 comp rx=0 tx=0).stop 2024-01-27T01:08:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.445+0000 7f5d6b620700 1 -- 172.21.15.46:0/747650011 shutdown_connections 2024-01-27T01:08:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.445+0000 7f5d6b620700 1 --2- 172.21.15.46:0/747650011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d6415da10 0x7f5d6415de30 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.445+0000 7f5d6b620700 1 --2- 172.21.15.46:0/747650011 >> v2:172.21.15.106:3300/0 conn(0x7f5d640b2530 0x7f5d6415a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.445+0000 7f5d6b620700 1 -- 172.21.15.46:0/747650011 >> 172.21.15.46:0/747650011 conn(0x7f5d640ccbc0 msgr2=0x7f5d640ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.445+0000 7f5d6b620700 1 -- 172.21.15.46:0/747650011 shutdown_connections 2024-01-27T01:08:21.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.445+0000 7f5d6b620700 1 -- 172.21.15.46:0/747650011 wait complete. 2024-01-27T01:08:21.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.446+0000 7f5d6b620700 1 Processor -- start 2024-01-27T01:08:21.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.446+0000 7f5d6b620700 1 -- start start 2024-01-27T01:08:21.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.447+0000 7f5d6b620700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5d640b2160 0x7f5d6414b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:21.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.447+0000 7f5d6b620700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d640b2530 0x7f5d64147830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:21.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.447+0000 7f5d6b620700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d6415da10 0x7f5d6415be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:21.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.447+0000 7f5d6b620700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d64169df0 con 0x7f5d640b2530 2024-01-27T01:08:21.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.447+0000 7f5d6b620700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d64149510 con 0x7f5d6415da10 2024-01-27T01:08:21.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.447+0000 7f5d6b620700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d641697a0 con 0x7f5d640b2160 2024-01-27T01:08:21.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.447+0000 7f5d68bbb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d640b2530 0x7f5d64147830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:21.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.448+0000 7f5d68bbb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d640b2530 0x7f5d64147830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38946/0 (socket says 172.21.15.46:38946) 2024-01-27T01:08:21.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.448+0000 7f5d68bbb700 1 -- 172.21.15.46:0/3592200112 learned_addr learned my addr 172.21.15.46:0/3592200112 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:21.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.448+0000 7f5d693bc700 1 --2- 172.21.15.46:0/3592200112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5d640b2160 0x7f5d6414b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.448+0000 7f5d69bbd700 1 --2- 172.21.15.46:0/3592200112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d6415da10 0x7f5d6415be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.448+0000 7f5d68bbb700 1 -- 172.21.15.46:0/3592200112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5d640b2160 msgr2=0x7f5d6414b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.448+0000 7f5d68bbb700 1 --2- 172.21.15.46:0/3592200112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5d640b2160 0x7f5d6414b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.448+0000 7f5d68bbb700 1 -- 172.21.15.46:0/3592200112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d6415da10 msgr2=0x7f5d6415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.448+0000 7f5d68bbb700 1 --2- 172.21.15.46:0/3592200112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d6415da10 0x7f5d6415be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.448+0000 7f5d68bbb700 1 -- 172.21.15.46:0/3592200112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5c009a20 con 0x7f5d640b2530 2024-01-27T01:08:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.448+0000 7f5d693bc700 1 --2- 172.21.15.46:0/3592200112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5d640b2160 0x7f5d6414b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.448+0000 7f5d69bbd700 1 --2- 172.21.15.46:0/3592200112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d6415da10 0x7f5d6415be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.448+0000 7f5d68bbb700 1 --2- 172.21.15.46:0/3592200112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d640b2530 0x7f5d64147830 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f5d5400d890 tx=0x7f5d54008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.448+0000 7f5d597fa700 1 -- 172.21.15.46:0/3592200112 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d540125f0 con 0x7f5d640b2530 2024-01-27T01:08:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.448+0000 7f5d597fa700 1 -- 172.21.15.46:0/3592200112 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d54012750 con 0x7f5d640b2530 2024-01-27T01:08:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.448+0000 7f5d6b620700 1 -- 172.21.15.46:0/3592200112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d64169a80 con 0x7f5d640b2530 2024-01-27T01:08:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.448+0000 7f5d597fa700 1 -- 172.21.15.46:0/3592200112 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d54012af0 con 0x7f5d640b2530 2024-01-27T01:08:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.448+0000 7f5d6b620700 1 -- 172.21.15.46:0/3592200112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d64147d70 con 0x7f5d640b2530 2024-01-27T01:08:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.449+0000 7f5d6b620700 1 -- 172.21.15.46:0/3592200112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d6404c710 con 0x7f5d640b2530 2024-01-27T01:08:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.450+0000 7f5d597fa700 1 -- 172.21.15.46:0/3592200112 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f5d5401e460 con 0x7f5d640b2530 2024-01-27T01:08:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.450+0000 7f5d597fa700 1 --2- 172.21.15.46:0/3592200112 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5d480670e0 0x7f5d48069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.450+0000 7f5d597fa700 1 -- 172.21.15.46:0/3592200112 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f5d540850e0 con 0x7f5d640b2530 2024-01-27T01:08:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.450+0000 7f5d693bc700 1 --2- 172.21.15.46:0/3592200112 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5d480670e0 0x7f5d48069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.451+0000 7f5d693bc700 1 --2- 172.21.15.46:0/3592200112 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5d480670e0 0x7f5d48069590 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5d6416c9b0 tx=0x7f5d60007f50 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:21.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.455+0000 7f5d597fa700 1 -- 172.21.15.46:0/3592200112 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5d54051da0 con 0x7f5d640b2530 2024-01-27T01:08:21.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.662+0000 7f5d6b620700 1 -- 172.21.15.46:0/3592200112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d640aa740 con 0x7f5d640b2530 2024-01-27T01:08:21.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.663+0000 7f5d597fa700 1 -- 172.21.15.46:0/3592200112 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v262) v1 ==== 74+0+16338 (secure 0 0 0) 0x7f5d540553c0 con 0x7f5d640b2530 2024-01-27T01:08:21.662 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:21.662 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":262,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:08:20.703308+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:08:19.198347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-01-27T01:08:19.674810+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:08:21.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.665+0000 7f5d6b620700 1 -- 172.21.15.46:0/3592200112 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5d480670e0 msgr2=0x7f5d48069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:21.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.665+0000 7f5d6b620700 1 --2- 172.21.15.46:0/3592200112 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5d480670e0 0x7f5d48069590 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5d6416c9b0 tx=0x7f5d60007f50 comp rx=0 tx=0).stop 2024-01-27T01:08:21.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.665+0000 7f5d6b620700 1 -- 172.21.15.46:0/3592200112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d640b2530 msgr2=0x7f5d64147830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:21.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.665+0000 7f5d6b620700 1 --2- 172.21.15.46:0/3592200112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d640b2530 0x7f5d64147830 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f5d5400d890 tx=0x7f5d54008040 comp rx=0 tx=0).stop 2024-01-27T01:08:21.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.665+0000 7f5d6b620700 1 -- 172.21.15.46:0/3592200112 shutdown_connections 2024-01-27T01:08:21.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.665+0000 7f5d6b620700 1 --2- 172.21.15.46:0/3592200112 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5d480670e0 0x7f5d48069590 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.665+0000 7f5d6b620700 1 --2- 172.21.15.46:0/3592200112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d6415da10 0x7f5d6415be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.665+0000 7f5d6b620700 1 --2- 172.21.15.46:0/3592200112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5d640b2160 0x7f5d6414b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.665+0000 7f5d6b620700 1 --2- 172.21.15.46:0/3592200112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d640b2530 0x7f5d64147830 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.665+0000 7f5d6b620700 1 -- 172.21.15.46:0/3592200112 >> 172.21.15.46:0/3592200112 conn(0x7f5d640ccbc0 msgr2=0x7f5d640b0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:21.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.665+0000 7f5d6b620700 1 -- 172.21.15.46:0/3592200112 shutdown_connections 2024-01-27T01:08:21.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.665+0000 7f5d6b620700 1 -- 172.21.15.46:0/3592200112 wait complete. 2024-01-27T01:08:21.679 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:08:21.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.813+0000 7f03a4762700 1 Processor -- start 2024-01-27T01:08:21.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.813+0000 7f03a4762700 1 -- start start 2024-01-27T01:08:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.814+0000 7f03a4762700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f039c159590 0x7f039c15d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.814+0000 7f03a4762700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f039c06c3c0 0x7f039c069370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.814+0000 7f03a4762700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039c15dd90 con 0x7f039c15d8f0 2024-01-27T01:08:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.814+0000 7f03a4762700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f039c15e350 con 0x7f039c159590 2024-01-27T01:08:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.814+0000 7f03a4762700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f039c0c97c0 con 0x7f039c06c3c0 2024-01-27T01:08:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.814+0000 7f03a24fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f039c159590 0x7f039c15d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.814+0000 7f03a24fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f039c159590 0x7f039c15d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38958/0 (socket says 172.21.15.46:38958) 2024-01-27T01:08:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.814+0000 7f03a24fe700 1 -- 172.21.15.46:0/2094985676 learned_addr learned my addr 172.21.15.46:0/2094985676 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.814+0000 7f03a2cff700 1 --2- 172.21.15.46:0/2094985676 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f039c06c3c0 0x7f039c069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:21.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.815+0000 7f03a24fe700 1 -- 172.21.15.46:0/2094985676 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f039c06c3c0 msgr2=0x7f039c069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:21.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.815+0000 7f03a24fe700 1 --2- 172.21.15.46:0/2094985676 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f039c06c3c0 0x7f039c069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.815+0000 7f03a24fe700 1 -- 172.21.15.46:0/2094985676 >> v1:172.21.15.106:6789/0 conn(0x7f039c15d8f0 legacy=0x7f039c06bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:21.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.815+0000 7f03a24fe700 1 -- 172.21.15.46:0/2094985676 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c15e070 con 0x7f039c159590 2024-01-27T01:08:21.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.815+0000 7f03a14fc700 1 -- 172.21.15.46:0/2094985676 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157291208 0 0) 0x7f039c15dd90 con 0x7f039c15d8f0 2024-01-27T01:08:21.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.815+0000 7f03a24fe700 1 --2- 172.21.15.46:0/2094985676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f039c159590 0x7f039c15d3b0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f03980099e0 tx=0x7f0398009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e2862bb6234ea79 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:21.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.815+0000 7f03a14fc700 1 -- 172.21.15.46:0/2094985676 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039800b820 con 0x7f039c159590 2024-01-27T01:08:21.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.815+0000 7f03a14fc700 1 -- 172.21.15.46:0/2094985676 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039800b980 con 0x7f039c159590 2024-01-27T01:08:21.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.815+0000 7f03a14fc700 1 -- 172.21.15.46:0/2094985676 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039800bd40 con 0x7f039c159590 2024-01-27T01:08:21.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.815+0000 7f03a4762700 1 -- 172.21.15.46:0/2094985676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f039c159590 msgr2=0x7f039c15d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:21.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.815+0000 7f03a4762700 1 --2- 172.21.15.46:0/2094985676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f039c159590 0x7f039c15d3b0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f03980099e0 tx=0x7f0398009250 comp rx=0 tx=0).stop 2024-01-27T01:08:21.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.816+0000 7f03a4762700 1 -- 172.21.15.46:0/2094985676 shutdown_connections 2024-01-27T01:08:21.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.816+0000 7f03a4762700 1 --2- 172.21.15.46:0/2094985676 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f039c06c3c0 0x7f039c069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.816+0000 7f03a4762700 1 --2- 172.21.15.46:0/2094985676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f039c159590 0x7f039c15d3b0 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.816+0000 7f03a4762700 1 -- 172.21.15.46:0/2094985676 >> 172.21.15.46:0/2094985676 conn(0x7f039c0ce530 msgr2=0x7f039c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:21.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.816+0000 7f03a4762700 1 -- 172.21.15.46:0/2094985676 shutdown_connections 2024-01-27T01:08:21.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.816+0000 7f03a4762700 1 -- 172.21.15.46:0/2094985676 wait complete. 2024-01-27T01:08:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.817+0000 7f03a4762700 1 Processor -- start 2024-01-27T01:08:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.817+0000 7f03a4762700 1 -- start start 2024-01-27T01:08:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.818+0000 7f03a4762700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f039c06c3c0 0x7f039c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.818+0000 7f03a4762700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f039c159590 0x7f039c06dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.818+0000 7f03a4762700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f039c15d8f0 0x7f039c0c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.818+0000 7f03a4762700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c15aaf0 con 0x7f039c06c3c0 2024-01-27T01:08:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.818+0000 7f03a4762700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c0c81d0 con 0x7f039c159590 2024-01-27T01:08:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.818+0000 7f03a4762700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f039c0c84b0 con 0x7f039c15d8f0 2024-01-27T01:08:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.818+0000 7f03a24fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f039c06c3c0 0x7f039c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.818+0000 7f03a24fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f039c06c3c0 0x7f039c06d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38960/0 (socket says 172.21.15.46:38960) 2024-01-27T01:08:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.818+0000 7f03a24fe700 1 -- 172.21.15.46:0/802900459 learned_addr learned my addr 172.21.15.46:0/802900459 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:21.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.818+0000 7f03a1cfd700 1 --2- 172.21.15.46:0/802900459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f039c159590 0x7f039c06dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:21.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.818+0000 7f03a2cff700 1 --2- 172.21.15.46:0/802900459 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f039c15d8f0 0x7f039c0c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:21.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.818+0000 7f03a24fe700 1 -- 172.21.15.46:0/802900459 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f039c15d8f0 msgr2=0x7f039c0c7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:21.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.818+0000 7f03a24fe700 1 --2- 172.21.15.46:0/802900459 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f039c15d8f0 0x7f039c0c7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.818+0000 7f03a24fe700 1 -- 172.21.15.46:0/802900459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f039c159590 msgr2=0x7f039c06dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:21.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.818+0000 7f03a24fe700 1 --2- 172.21.15.46:0/802900459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f039c159590 0x7f039c06dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:21.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.818+0000 7f03a24fe700 1 -- 172.21.15.46:0/802900459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03980094f0 con 0x7f039c06c3c0 2024-01-27T01:08:21.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.818+0000 7f03a24fe700 1 --2- 172.21.15.46:0/802900459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f039c06c3c0 0x7f039c06d570 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f039c06f390 tx=0x7f0398017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:21.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.819+0000 7f038e7fc700 1 -- 172.21.15.46:0/802900459 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039801f070 con 0x7f039c06c3c0 2024-01-27T01:08:21.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.819+0000 7f038e7fc700 1 -- 172.21.15.46:0/802900459 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0398002f30 con 0x7f039c06c3c0 2024-01-27T01:08:21.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.819+0000 7f03a4762700 1 -- 172.21.15.46:0/802900459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f039c0c8790 con 0x7f039c06c3c0 2024-01-27T01:08:21.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.819+0000 7f038e7fc700 1 -- 172.21.15.46:0/802900459 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039800b820 con 0x7f039c06c3c0 2024-01-27T01:08:21.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.819+0000 7f03a4762700 1 -- 172.21.15.46:0/802900459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f039c1462c0 con 0x7f039c06c3c0 2024-01-27T01:08:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.820+0000 7f03a4762700 1 -- 172.21.15.46:0/802900459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f039c15aff0 con 0x7f039c06c3c0 2024-01-27T01:08:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.820+0000 7f038e7fc700 1 -- 172.21.15.46:0/802900459 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f0398023070 con 0x7f039c06c3c0 2024-01-27T01:08:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.821+0000 7f038e7fc700 1 --2- 172.21.15.46:0/802900459 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f03800670e0 0x7f0380069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.821+0000 7f038e7fc700 1 -- 172.21.15.46:0/802900459 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f0398086080 con 0x7f039c06c3c0 2024-01-27T01:08:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.821+0000 7f03a1cfd700 1 --2- 172.21.15.46:0/802900459 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f03800670e0 0x7f0380069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.821+0000 7f03a1cfd700 1 --2- 172.21.15.46:0/802900459 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f03800670e0 0x7f0380069590 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f039c158260 tx=0x7f0390009040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:21.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:21.825+0000 7f038e7fc700 1 -- 172.21.15.46:0/802900459 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f03980b75a0 con 0x7f039c06c3c0 2024-01-27T01:08:22.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.037+0000 7f03a4762700 1 -- 172.21.15.46:0/802900459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f039c146450 con 0x7f039c06c3c0 2024-01-27T01:08:22.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.038+0000 7f038e7fc700 1 -- 172.21.15.46:0/802900459 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v262) v1 ==== 74+0+16338 (secure 0 0 0) 0x7f039802a090 con 0x7f039c06c3c0 2024-01-27T01:08:22.037 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:22.037 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":262,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:08:20.703308+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:08:19.198347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-01-27T01:08:19.674810+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:08:22.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.040+0000 7f03a4762700 1 -- 172.21.15.46:0/802900459 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f03800670e0 msgr2=0x7f0380069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:22.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.040+0000 7f03a4762700 1 --2- 172.21.15.46:0/802900459 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f03800670e0 0x7f0380069590 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f039c158260 tx=0x7f0390009040 comp rx=0 tx=0).stop 2024-01-27T01:08:22.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.040+0000 7f03a4762700 1 -- 172.21.15.46:0/802900459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f039c06c3c0 msgr2=0x7f039c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:22.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.040+0000 7f03a4762700 1 --2- 172.21.15.46:0/802900459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f039c06c3c0 0x7f039c06d570 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f039c06f390 tx=0x7f0398017000 comp rx=0 tx=0).stop 2024-01-27T01:08:22.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.040+0000 7f03a4762700 1 -- 172.21.15.46:0/802900459 shutdown_connections 2024-01-27T01:08:22.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.040+0000 7f03a4762700 1 --2- 172.21.15.46:0/802900459 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f03800670e0 0x7f0380069590 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:22.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.040+0000 7f03a4762700 1 --2- 172.21.15.46:0/802900459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f039c06c3c0 0x7f039c06d570 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:22.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.040+0000 7f03a4762700 1 --2- 172.21.15.46:0/802900459 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f039c15d8f0 0x7f039c0c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:22.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.040+0000 7f03a4762700 1 --2- 172.21.15.46:0/802900459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f039c159590 0x7f039c06dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:22.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.040+0000 7f03a4762700 1 -- 172.21.15.46:0/802900459 >> 172.21.15.46:0/802900459 conn(0x7f039c0ce530 msgr2=0x7f039c158920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:22.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.040+0000 7f03a4762700 1 -- 172.21.15.46:0/802900459 shutdown_connections 2024-01-27T01:08:22.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.040+0000 7f03a4762700 1 -- 172.21.15.46:0/802900459 wait complete. 2024-01-27T01:08:22.054 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-27T01:08:22.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.195+0000 7f516e9d6700 1 Processor -- start 2024-01-27T01:08:22.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.195+0000 7f516e9d6700 1 -- start start 2024-01-27T01:08:22.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.196+0000 7f516e9d6700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f51680b25b0 0x7f51680b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:22.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.196+0000 7f516e9d6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51680cdbf0 0x7f51680ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:22.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.196+0000 7f516e9d6700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51680abb30 con 0x7f51680ae610 2024-01-27T01:08:22.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.196+0000 7f516e9d6700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51680b2980 con 0x7f51680b25b0 2024-01-27T01:08:22.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.196+0000 7f516e9d6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516815a560 con 0x7f51680cdbf0 2024-01-27T01:08:22.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.197+0000 7f5167fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f51680b25b0 0x7f51680b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:22.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.197+0000 7f51677fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51680cdbf0 0x7f51680ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:22.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.197+0000 7f51677fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51680cdbf0 0x7f51680ae0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:49206/0 (socket says 172.21.15.46:49206) 2024-01-27T01:08:22.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.197+0000 7f516cf73700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f51680ae610 0x7f51680abfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:47666/0 (socket says 172.21.15.46:47666) 2024-01-27T01:08:22.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.197+0000 7f5167fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f51680b25b0 0x7f51680b04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38968/0 (socket says 172.21.15.46:38968) 2024-01-27T01:08:22.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.197+0000 7f51677fe700 1 -- 172.21.15.46:0/71338314 learned_addr learned my addr 172.21.15.46:0/71338314 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:22.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.197+0000 7f5166ffd700 1 -- 172.21.15.46:0/71338314 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920614273 0 0) 0x7f51680abb30 con 0x7f51680ae610 2024-01-27T01:08:22.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.197+0000 7f5166ffd700 1 -- 172.21.15.46:0/71338314 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5150003570 con 0x7f51680ae610 2024-01-27T01:08:22.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.197+0000 7f51677fe700 1 -- 172.21.15.46:0/71338314 >> v2:172.21.15.46:3300/0 conn(0x7f51680b25b0 msgr2=0x7f51680b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:22.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.197+0000 7f51677fe700 1 --2- 172.21.15.46:0/71338314 >> v2:172.21.15.46:3300/0 conn(0x7f51680b25b0 0x7f51680b04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:22.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.197+0000 7f51677fe700 1 -- 172.21.15.46:0/71338314 >> v1:172.21.15.106:6789/0 conn(0x7f51680ae610 legacy=0x7f51680abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:22.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.197+0000 7f51677fe700 1 -- 172.21.15.46:0/71338314 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51680b3210 con 0x7f51680cdbf0 2024-01-27T01:08:22.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.197+0000 7f5167fff700 1 --2- 172.21.15.46:0/71338314 >> v2:172.21.15.46:3300/0 conn(0x7f51680b25b0 0x7f51680b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:22.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.197+0000 7f51677fe700 1 --2- 172.21.15.46:0/71338314 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51680cdbf0 0x7f51680ae0d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f515c0099b0 tx=0x7f515c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b39e49e6eb955735 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:22.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.197+0000 7f5166ffd700 1 -- 172.21.15.46:0/71338314 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c00b680 con 0x7f51680cdbf0 2024-01-27T01:08:22.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.197+0000 7f5166ffd700 1 -- 172.21.15.46:0/71338314 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c00b7e0 con 0x7f51680cdbf0 2024-01-27T01:08:22.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.197+0000 7f5166ffd700 1 -- 172.21.15.46:0/71338314 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c00bba0 con 0x7f51680cdbf0 2024-01-27T01:08:22.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.198+0000 7f516e9d6700 1 -- 172.21.15.46:0/71338314 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51680cdbf0 msgr2=0x7f51680ae0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:22.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.198+0000 7f516e9d6700 1 --2- 172.21.15.46:0/71338314 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51680cdbf0 0x7f51680ae0d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f515c0099b0 tx=0x7f515c009250 comp rx=0 tx=0).stop 2024-01-27T01:08:22.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.198+0000 7f516e9d6700 1 -- 172.21.15.46:0/71338314 shutdown_connections 2024-01-27T01:08:22.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.198+0000 7f516e9d6700 1 --2- 172.21.15.46:0/71338314 >> v2:172.21.15.46:3300/0 conn(0x7f51680b25b0 0x7f51680b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:22.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.198+0000 7f516e9d6700 1 --2- 172.21.15.46:0/71338314 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51680cdbf0 0x7f51680ae0d0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:22.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.198+0000 7f516e9d6700 1 -- 172.21.15.46:0/71338314 >> 172.21.15.46:0/71338314 conn(0x7f5168069220 msgr2=0x7f51680cd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:22.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.198+0000 7f516e9d6700 1 -- 172.21.15.46:0/71338314 shutdown_connections 2024-01-27T01:08:22.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.198+0000 7f516e9d6700 1 -- 172.21.15.46:0/71338314 wait complete. 2024-01-27T01:08:22.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.199+0000 7f516e9d6700 1 Processor -- start 2024-01-27T01:08:22.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.200+0000 7f516e9d6700 1 -- start start 2024-01-27T01:08:22.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.200+0000 7f516e9d6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51680ae610 0x7f5168157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:22.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.200+0000 7f516e9d6700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51680b25b0 0x7f51680ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:22.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.200+0000 7f516e9d6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51680cdbf0 0x7f51680c8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:22.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.200+0000 7f516e9d6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5168157e70 con 0x7f51680ae610 2024-01-27T01:08:22.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.200+0000 7f516e9d6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51680ca3a0 con 0x7f51680cdbf0 2024-01-27T01:08:22.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.200+0000 7f516e9d6700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51680ca680 con 0x7f51680b25b0 2024-01-27T01:08:22.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.200+0000 7f5167fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51680ae610 0x7f5168157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:22.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.200+0000 7f51677fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51680b25b0 0x7f51680ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:22.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.201+0000 7f5167fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51680ae610 0x7f5168157930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38980/0 (socket says 172.21.15.46:38980) 2024-01-27T01:08:22.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.201+0000 7f5167fff700 1 -- 172.21.15.46:0/2102848487 learned_addr learned my addr 172.21.15.46:0/2102848487 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:22.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.201+0000 7f516cf73700 1 --2- 172.21.15.46:0/2102848487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51680cdbf0 0x7f51680c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:22.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.201+0000 7f51677fe700 1 -- 172.21.15.46:0/2102848487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51680cdbf0 msgr2=0x7f51680c8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:22.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.201+0000 7f51677fe700 1 --2- 172.21.15.46:0/2102848487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51680cdbf0 0x7f51680c8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:22.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.201+0000 7f51677fe700 1 -- 172.21.15.46:0/2102848487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51680ae610 msgr2=0x7f5168157930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:22.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.201+0000 7f51677fe700 1 --2- 172.21.15.46:0/2102848487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51680ae610 0x7f5168157930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:22.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.201+0000 7f51677fe700 1 -- 172.21.15.46:0/2102848487 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5158009420 con 0x7f51680b25b0 2024-01-27T01:08:22.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.201+0000 7f51677fe700 1 --2- 172.21.15.46:0/2102848487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51680b25b0 0x7f51680ad2c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f515c013000 tx=0x7f515c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:22.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.201+0000 7f514ffff700 1 -- 172.21.15.46:0/2102848487 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c006a90 con 0x7f51680b25b0 2024-01-27T01:08:22.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.201+0000 7f514ffff700 1 -- 172.21.15.46:0/2102848487 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c00b950 con 0x7f51680b25b0 2024-01-27T01:08:22.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.201+0000 7f516e9d6700 1 -- 172.21.15.46:0/2102848487 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515c0094f0 con 0x7f51680b25b0 2024-01-27T01:08:22.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.201+0000 7f514ffff700 1 -- 172.21.15.46:0/2102848487 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c01a450 con 0x7f51680b25b0 2024-01-27T01:08:22.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.201+0000 7f516e9d6700 1 -- 172.21.15.46:0/2102848487 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51680cac40 con 0x7f51680b25b0 2024-01-27T01:08:22.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.202+0000 7f516e9d6700 1 -- 172.21.15.46:0/2102848487 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51680c6740 con 0x7f51680b25b0 2024-01-27T01:08:22.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.203+0000 7f514ffff700 1 -- 172.21.15.46:0/2102848487 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f515c01a820 con 0x7f51680b25b0 2024-01-27T01:08:22.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.203+0000 7f514ffff700 1 --2- 172.21.15.46:0/2102848487 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f51540670e0 0x7f5154069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:22.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.203+0000 7f514ffff700 1 -- 172.21.15.46:0/2102848487 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f515c0867f0 con 0x7f51680b25b0 2024-01-27T01:08:22.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.203+0000 7f5167fff700 1 --2- 172.21.15.46:0/2102848487 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f51540670e0 0x7f5154069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:22.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.204+0000 7f5167fff700 1 --2- 172.21.15.46:0/2102848487 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f51540670e0 0x7f5154069590 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5158011310 tx=0x7f51580091a0 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:22.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.208+0000 7f514ffff700 1 -- 172.21.15.46:0/2102848487 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f515c053530 con 0x7f51680b25b0 2024-01-27T01:08:22.451 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:22.451+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:22.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:22.479+0000 7f516e9d6700 1 -- 172.21.15.46:0/2102848487 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f51680cae30 con 0x7f51680b25b0 2024-01-27T01:08:23.191 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-01-27T01:08:23.192+0000 7fcbf9662700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-27T01:08:23.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.274+0000 7f514ffff700 1 -- 172.21.15.46:0/2102848487 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v148) v1 ==== 118+0+0 (secure 0 0 0) 0x7f515c056b50 con 0x7f51680b25b0 2024-01-27T01:08:23.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.277+0000 7f516e9d6700 1 -- 172.21.15.46:0/2102848487 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f51540670e0 msgr2=0x7f5154069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:23.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.277+0000 7f516e9d6700 1 --2- 172.21.15.46:0/2102848487 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f51540670e0 0x7f5154069590 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5158011310 tx=0x7f51580091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:23.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.277+0000 7f516e9d6700 1 -- 172.21.15.46:0/2102848487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51680b25b0 msgr2=0x7f51680ad2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:23.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.277+0000 7f516e9d6700 1 --2- 172.21.15.46:0/2102848487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51680b25b0 0x7f51680ad2c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f515c013000 tx=0x7f515c017000 comp rx=0 tx=0).stop 2024-01-27T01:08:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.277+0000 7f516e9d6700 1 -- 172.21.15.46:0/2102848487 shutdown_connections 2024-01-27T01:08:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.277+0000 7f516e9d6700 1 --2- 172.21.15.46:0/2102848487 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f51540670e0 0x7f5154069590 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.277+0000 7f516e9d6700 1 --2- 172.21.15.46:0/2102848487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51680ae610 0x7f5168157930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.277+0000 7f516e9d6700 1 --2- 172.21.15.46:0/2102848487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51680cdbf0 0x7f51680c8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.277+0000 7f516e9d6700 1 --2- 172.21.15.46:0/2102848487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51680b25b0 0x7f51680ad2c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.277+0000 7f516e9d6700 1 -- 172.21.15.46:0/2102848487 >> 172.21.15.46:0/2102848487 conn(0x7f5168069220 msgr2=0x7f51680b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.277+0000 7f516e9d6700 1 -- 172.21.15.46:0/2102848487 shutdown_connections 2024-01-27T01:08:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.277+0000 7f516e9d6700 1 -- 172.21.15.46:0/2102848487 wait complete. 2024-01-27T01:08:23.293 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-27T01:08:23.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.437+0000 7fdcbfa44700 1 Processor -- start 2024-01-27T01:08:23.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.437+0000 7fdcbfa44700 1 -- start start 2024-01-27T01:08:23.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.438+0000 7fdcbfa44700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb8067ec0 0x7fdcb815df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:23.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.438+0000 7fdcbfa44700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb806fe90 con 0x7fdcb8067af0 2024-01-27T01:08:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.438+0000 7fdcbfa44700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb806edf0 con 0x7fdcb80afa80 2024-01-27T01:08:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.438+0000 7fdcbfa44700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb815e8a0 con 0x7fdcb8067ec0 2024-01-27T01:08:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.438+0000 7fdcbd7e0700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdcb8067af0 0x7fdcb80af370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51696/0 (socket says 172.21.15.46:51696) 2024-01-27T01:08:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.438+0000 7fdcbd7e0700 1 -- 172.21.15.46:0/2311186618 learned_addr learned my addr 172.21.15.46:0/2311186618 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.438+0000 7fdcbdfe1700 1 --2- 172.21.15.46:0/2311186618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb8067ec0 0x7fdcb815df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.438+0000 7fdcabfff700 1 -- 172.21.15.46:0/2311186618 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492124099 0 0) 0x7fdcb806fe90 con 0x7fdcb8067af0 2024-01-27T01:08:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.438+0000 7fdcabfff700 1 -- 172.21.15.46:0/2311186618 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca4003570 con 0x7fdcb8067af0 2024-01-27T01:08:23.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcabfff700 1 -- 172.21.15.46:0/2311186618 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475605316 0 0) 0x7fdcb806edf0 con 0x7fdcb80afa80 2024-01-27T01:08:23.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcabfff700 1 -- 172.21.15.46:0/2311186618 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcb806fe90 con 0x7fdcb80afa80 2024-01-27T01:08:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcabfff700 1 -- 172.21.15.46:0/2311186618 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1257371744 0 0) 0x7fdca4003570 con 0x7fdcb8067af0 2024-01-27T01:08:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcabfff700 1 -- 172.21.15.46:0/2311186618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb8067ec0 msgr2=0x7fdcb815df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcabfff700 1 --2- 172.21.15.46:0/2311186618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb8067ec0 0x7fdcb815df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcabfff700 1 -- 172.21.15.46:0/2311186618 >> v1:172.21.15.106:6789/0 conn(0x7fdcb80afa80 legacy=0x7fdcb815d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcabfff700 1 -- 172.21.15.46:0/2311186618 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb806f0d0 con 0x7fdcb8067af0 2024-01-27T01:08:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcabfff700 1 -- 172.21.15.46:0/2311186618 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fdcac001470 con 0x7fdcb8067af0 2024-01-27T01:08:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcabfff700 1 -- 172.21.15.46:0/2311186618 >> v1:172.21.15.46:6789/0 conn(0x7fdcb8067af0 legacy=0x7fdcb80af370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcabfff700 1 --2- 172.21.15.46:0/2311186618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca4003c20 0x7fdca4004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcabfff700 1 --2- 172.21.15.46:0/2311186618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb8067ec0 0x7fdca4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcabfff700 1 --2- 172.21.15.46:0/2311186618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdca4004d50 0x7fdca40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcabfff700 1 -- 172.21.15.46:0/2311186618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb815e8a0 con 0x7fdca4003c20 2024-01-27T01:08:23.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcabfff700 1 -- 172.21.15.46:0/2311186618 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca400bc40 con 0x7fdca4004d50 2024-01-27T01:08:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcabfff700 1 -- 172.21.15.46:0/2311186618 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca400be00 con 0x7fdcb8067ec0 2024-01-27T01:08:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcbd7e0700 1 --2- 172.21.15.46:0/2311186618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca4003c20 0x7fdca4004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcbdfe1700 1 --2- 172.21.15.46:0/2311186618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdca4004d50 0x7fdca40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcbcfdf700 1 --2- 172.21.15.46:0/2311186618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb8067ec0 0x7fdca4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcbd7e0700 1 -- 172.21.15.46:0/2311186618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb8067ec0 msgr2=0x7fdca4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcbd7e0700 1 --2- 172.21.15.46:0/2311186618 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb8067ec0 0x7fdca4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcbd7e0700 1 -- 172.21.15.46:0/2311186618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdca4004d50 msgr2=0x7fdca40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcbd7e0700 1 --2- 172.21.15.46:0/2311186618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdca4004d50 0x7fdca40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcbd7e0700 1 -- 172.21.15.46:0/2311186618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca4003570 con 0x7fdca4003c20 2024-01-27T01:08:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.439+0000 7fdcbd7e0700 1 --2- 172.21.15.46:0/2311186618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca4003c20 0x7fdca4004010 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00c210 tx=0x7fdcac001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb134e0b7487c24 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.440+0000 7fdcabfff700 1 -- 172.21.15.46:0/2311186618 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac00cc10 con 0x7fdca4003c20 2024-01-27T01:08:23.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.440+0000 7fdcabfff700 1 -- 172.21.15.46:0/2311186618 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcac00cd70 con 0x7fdca4003c20 2024-01-27T01:08:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.440+0000 7fdcabfff700 1 -- 172.21.15.46:0/2311186618 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac005b70 con 0x7fdca4003c20 2024-01-27T01:08:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.440+0000 7fdcbfa44700 1 -- 172.21.15.46:0/2311186618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca4003c20 msgr2=0x7fdca4004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.440+0000 7fdcbfa44700 1 --2- 172.21.15.46:0/2311186618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca4003c20 0x7fdca4004010 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00c210 tx=0x7fdcac001d40 comp rx=0 tx=0).stop 2024-01-27T01:08:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.440+0000 7fdcbdfe1700 1 -- 172.21.15.46:0/2311186618 reap_dead start 2024-01-27T01:08:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.440+0000 7fdcbfa44700 1 -- 172.21.15.46:0/2311186618 shutdown_connections 2024-01-27T01:08:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.440+0000 7fdcbfa44700 1 -- 172.21.15.46:0/2311186618 >> 172.21.15.46:0/2311186618 conn(0x7fdcb80cdf80 msgr2=0x7fdcb806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.441+0000 7fdcbfa44700 1 -- 172.21.15.46:0/2311186618 shutdown_connections 2024-01-27T01:08:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.441+0000 7fdcbfa44700 1 -- 172.21.15.46:0/2311186618 wait complete. 2024-01-27T01:08:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.442+0000 7fdcbfa44700 1 Processor -- start 2024-01-27T01:08:23.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.442+0000 7fdcbfa44700 1 -- start start 2024-01-27T01:08:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.443+0000 7fdcbfa44700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca4003c20 0x7fdcb8148060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.443+0000 7fdcbfa44700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcb8146b30 0x7fdcb81519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.443+0000 7fdcbfa44700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb8145ce0 0x7fdcb814ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.443+0000 7fdcbfa44700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb8146fc0 con 0x7fdca4003c20 2024-01-27T01:08:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.443+0000 7fdcbfa44700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb81472a0 con 0x7fdcb8146b30 2024-01-27T01:08:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.443+0000 7fdcbfa44700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb8147580 con 0x7fdcb8145ce0 2024-01-27T01:08:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.443+0000 7fdcbd7e0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca4003c20 0x7fdcb8148060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.443+0000 7fdcbd7e0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca4003c20 0x7fdcb8148060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38994/0 (socket says 172.21.15.46:38994) 2024-01-27T01:08:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.443+0000 7fdcbd7e0700 1 -- 172.21.15.46:0/2414454294 learned_addr learned my addr 172.21.15.46:0/2414454294 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:23.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.443+0000 7fdcbcfdf700 1 --2- 172.21.15.46:0/2414454294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcb8146b30 0x7fdcb81519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.443+0000 7fdcbdfe1700 1 --2- 172.21.15.46:0/2414454294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb8145ce0 0x7fdcb814ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.443+0000 7fdcbd7e0700 1 -- 172.21.15.46:0/2414454294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb8145ce0 msgr2=0x7fdcb814ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.443+0000 7fdcbd7e0700 1 --2- 172.21.15.46:0/2414454294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb8145ce0 0x7fdcb814ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.443+0000 7fdcbd7e0700 1 -- 172.21.15.46:0/2414454294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcb8146b30 msgr2=0x7fdcb81519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.443+0000 7fdcbd7e0700 1 --2- 172.21.15.46:0/2414454294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcb8146b30 0x7fdcb81519c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.443+0000 7fdcbd7e0700 1 -- 172.21.15.46:0/2414454294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb8147860 con 0x7fdca4003c20 2024-01-27T01:08:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.443+0000 7fdcbd7e0700 1 --2- 172.21.15.46:0/2414454294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca4003c20 0x7fdcb8148060 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00c800 tx=0x7fdcac015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.444+0000 7fdca97fa700 1 -- 172.21.15.46:0/2414454294 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac00cc10 con 0x7fdca4003c20 2024-01-27T01:08:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.444+0000 7fdca97fa700 1 -- 172.21.15.46:0/2414454294 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcac007a30 con 0x7fdca4003c20 2024-01-27T01:08:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.444+0000 7fdcbfa44700 1 -- 172.21.15.46:0/2414454294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb8147b40 con 0x7fdca4003c20 2024-01-27T01:08:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.444+0000 7fdca97fa700 1 -- 172.21.15.46:0/2414454294 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac00cee0 con 0x7fdca4003c20 2024-01-27T01:08:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.444+0000 7fdcbfa44700 1 -- 172.21.15.46:0/2414454294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb814b2d0 con 0x7fdca4003c20 2024-01-27T01:08:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.445+0000 7fdcbfa44700 1 -- 172.21.15.46:0/2414454294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb8147eb0 con 0x7fdca4003c20 2024-01-27T01:08:23.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.445+0000 7fdca97fa700 1 -- 172.21.15.46:0/2414454294 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7fdcac019030 con 0x7fdca4003c20 2024-01-27T01:08:23.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.446+0000 7fdca97fa700 1 --2- 172.21.15.46:0/2414454294 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdc9c0670e0 0x7fdc9c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:23.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.446+0000 7fdca97fa700 1 -- 172.21.15.46:0/2414454294 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fdcac0861c0 con 0x7fdca4003c20 2024-01-27T01:08:23.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.446+0000 7fdcbcfdf700 1 --2- 172.21.15.46:0/2414454294 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdc9c0670e0 0x7fdc9c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:23.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.446+0000 7fdcbcfdf700 1 --2- 172.21.15.46:0/2414454294 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdc9c0670e0 0x7fdc9c069590 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4007a80 tx=0x7fdcb4009040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:23.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.450+0000 7fdca97fa700 1 -- 172.21.15.46:0/2414454294 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdcac052e80 con 0x7fdca4003c20 2024-01-27T01:08:23.450 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:23.451+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:23.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:23.710+0000 7fdcbfa44700 1 -- 172.21.15.46:0/2414454294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fdcb8152140 con 0x7fdca4003c20 2024-01-27T01:08:24.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.275+0000 7fdca97fa700 1 -- 172.21.15.46:0/2414454294 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v149) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdcac0564a0 con 0x7fdca4003c20 2024-01-27T01:08:24.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.280+0000 7fdcbfa44700 1 -- 172.21.15.46:0/2414454294 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdc9c0670e0 msgr2=0x7fdc9c069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:24.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.280+0000 7fdcbfa44700 1 --2- 172.21.15.46:0/2414454294 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdc9c0670e0 0x7fdc9c069590 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4007a80 tx=0x7fdcb4009040 comp rx=0 tx=0).stop 2024-01-27T01:08:24.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.280+0000 7fdcbfa44700 1 -- 172.21.15.46:0/2414454294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca4003c20 msgr2=0x7fdcb8148060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:24.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.280+0000 7fdcbfa44700 1 --2- 172.21.15.46:0/2414454294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca4003c20 0x7fdcb8148060 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00c800 tx=0x7fdcac015000 comp rx=0 tx=0).stop 2024-01-27T01:08:24.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.280+0000 7fdcbfa44700 1 -- 172.21.15.46:0/2414454294 shutdown_connections 2024-01-27T01:08:24.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.280+0000 7fdcbfa44700 1 --2- 172.21.15.46:0/2414454294 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdc9c0670e0 0x7fdc9c069590 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.280+0000 7fdcbfa44700 1 --2- 172.21.15.46:0/2414454294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca4003c20 0x7fdcb8148060 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.280+0000 7fdcbfa44700 1 --2- 172.21.15.46:0/2414454294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb8145ce0 0x7fdcb814ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.280+0000 7fdcbfa44700 1 --2- 172.21.15.46:0/2414454294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcb8146b30 0x7fdcb81519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.280+0000 7fdcbfa44700 1 -- 172.21.15.46:0/2414454294 >> 172.21.15.46:0/2414454294 conn(0x7fdcb80cdf80 msgr2=0x7fdcb80cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:24.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.280+0000 7fdcbfa44700 1 -- 172.21.15.46:0/2414454294 shutdown_connections 2024-01-27T01:08:24.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.280+0000 7fdcbfa44700 1 -- 172.21.15.46:0/2414454294 wait complete. 2024-01-27T01:08:24.295 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:08:24.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.440+0000 7f0bc3f8e700 1 Processor -- start 2024-01-27T01:08:24.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.440+0000 7f0bc3f8e700 1 -- start start 2024-01-27T01:08:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.441+0000 7f0bc3f8e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0bbc1575a0 0x7f0bbc150af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.441+0000 7f0bc3f8e700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc068d20 con 0x7f0bbc150720 2024-01-27T01:08:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.441+0000 7f0bc3f8e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc068110 con 0x7f0bbc06e2a0 2024-01-27T01:08:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.441+0000 7f0bc3f8e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc0683f0 con 0x7f0bbc1575a0 2024-01-27T01:08:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.441+0000 7f0bc1d2a700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0bbc150720 0x7f0bbc067930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51712/0 (socket says 172.21.15.46:51712) 2024-01-27T01:08:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.441+0000 7f0bc1d2a700 1 -- 172.21.15.46:0/4063142919 learned_addr learned my addr 172.21.15.46:0/4063142919 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.441+0000 7f0bc1529700 1 --2- 172.21.15.46:0/4063142919 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0bbc1575a0 0x7f0bbc150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.441+0000 7f0bc0d28700 1 -- 172.21.15.46:0/4063142919 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57966040 0 0) 0x7f0bbc068d20 con 0x7f0bbc150720 2024-01-27T01:08:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.441+0000 7f0bc0d28700 1 -- 172.21.15.46:0/4063142919 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba8003570 con 0x7f0bbc150720 2024-01-27T01:08:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.441+0000 7f0bc0d28700 1 -- 172.21.15.46:0/4063142919 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590373111 0 0) 0x7f0bbc068110 con 0x7f0bbc06e2a0 2024-01-27T01:08:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.441+0000 7f0bc0d28700 1 -- 172.21.15.46:0/4063142919 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bbc068d20 con 0x7f0bbc06e2a0 2024-01-27T01:08:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.442+0000 7f0bc1529700 1 -- 172.21.15.46:0/4063142919 >> v1:172.21.15.106:6789/0 conn(0x7f0bbc06e2a0 legacy=0x7f0bbc157b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.442+0000 7f0bc1529700 1 -- 172.21.15.46:0/4063142919 >> v1:172.21.15.46:6789/0 conn(0x7f0bbc150720 legacy=0x7f0bbc067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.442+0000 7f0bc1529700 1 -- 172.21.15.46:0/4063142919 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc0686d0 con 0x7f0bbc1575a0 2024-01-27T01:08:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.442+0000 7f0bc1529700 1 --2- 172.21.15.46:0/4063142919 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0bbc1575a0 0x7f0bbc150af0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0bac009910 tx=0x7f0bac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab460323d5db1148 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.442+0000 7f0bc0d28700 1 -- 172.21.15.46:0/4063142919 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac011630 con 0x7f0bbc1575a0 2024-01-27T01:08:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.442+0000 7f0bc0d28700 1 -- 172.21.15.46:0/4063142919 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bac011790 con 0x7f0bbc1575a0 2024-01-27T01:08:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.442+0000 7f0bc0d28700 1 -- 172.21.15.46:0/4063142919 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac011b30 con 0x7f0bbc1575a0 2024-01-27T01:08:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.443+0000 7f0bc3f8e700 1 -- 172.21.15.46:0/4063142919 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0bbc1575a0 msgr2=0x7f0bbc150af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.443+0000 7f0bc3f8e700 1 --2- 172.21.15.46:0/4063142919 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0bbc1575a0 0x7f0bbc150af0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0bac009910 tx=0x7f0bac0091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.443+0000 7f0bc3f8e700 1 -- 172.21.15.46:0/4063142919 shutdown_connections 2024-01-27T01:08:24.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.443+0000 7f0bc3f8e700 1 --2- 172.21.15.46:0/4063142919 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0bbc1575a0 0x7f0bbc150af0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.443+0000 7f0bc3f8e700 1 -- 172.21.15.46:0/4063142919 >> 172.21.15.46:0/4063142919 conn(0x7f0bbc0ccbc0 msgr2=0x7f0bbc0ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:24.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.443+0000 7f0bc3f8e700 1 -- 172.21.15.46:0/4063142919 shutdown_connections 2024-01-27T01:08:24.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.443+0000 7f0bc3f8e700 1 -- 172.21.15.46:0/4063142919 wait complete. 2024-01-27T01:08:24.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.444+0000 7f0bc3f8e700 1 Processor -- start 2024-01-27T01:08:24.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.444+0000 7f0bc3f8e700 1 -- start start 2024-01-27T01:08:24.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.445+0000 7f0bc3f8e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0bbc06e2a0 0x7f0bbc150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:24.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.445+0000 7f0bc3f8e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0bbc150720 0x7f0bbc14f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:24.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.445+0000 7f0bc3f8e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bbc1575a0 0x7f0bbc0c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:24.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.445+0000 7f0bc3f8e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc1545a0 con 0x7f0bbc1575a0 2024-01-27T01:08:24.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.445+0000 7f0bc3f8e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc151a90 con 0x7f0bbc06e2a0 2024-01-27T01:08:24.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.445+0000 7f0bc3f8e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc159320 con 0x7f0bbc150720 2024-01-27T01:08:24.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.446+0000 7f0bc252b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bbc1575a0 0x7f0bbc0c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:24.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.446+0000 7f0bc1529700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0bbc150720 0x7f0bbc14f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:24.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.446+0000 7f0bc1d2a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0bbc06e2a0 0x7f0bbc150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:24.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.446+0000 7f0bc252b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bbc1575a0 0x7f0bbc0c9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38998/0 (socket says 172.21.15.46:38998) 2024-01-27T01:08:24.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.446+0000 7f0bc252b700 1 -- 172.21.15.46:0/217718253 learned_addr learned my addr 172.21.15.46:0/217718253 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:24.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.446+0000 7f0bc1529700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0bbc150720 0x7f0bbc14f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:49260/0 (socket says 172.21.15.46:49260) 2024-01-27T01:08:24.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.446+0000 7f0bc252b700 1 -- 172.21.15.46:0/217718253 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0bbc150720 msgr2=0x7f0bbc14f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:24.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.446+0000 7f0bc252b700 1 --2- 172.21.15.46:0/217718253 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0bbc150720 0x7f0bbc14f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.446+0000 7f0bc252b700 1 -- 172.21.15.46:0/217718253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0bbc06e2a0 msgr2=0x7f0bbc150af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:24.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.446+0000 7f0bc252b700 1 --2- 172.21.15.46:0/217718253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0bbc06e2a0 0x7f0bbc150af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.446+0000 7f0bc252b700 1 -- 172.21.15.46:0/217718253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bac009420 con 0x7f0bbc1575a0 2024-01-27T01:08:24.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.446+0000 7f0bc1d2a700 1 --2- 172.21.15.46:0/217718253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0bbc06e2a0 0x7f0bbc150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:24.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.446+0000 7f0bc252b700 1 --2- 172.21.15.46:0/217718253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bbc1575a0 0x7f0bbc0c9f60 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f0bb400c860 tx=0x7f0bb4002700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:24.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.446+0000 7f0bb1ffb700 1 -- 172.21.15.46:0/217718253 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb4002030 con 0x7f0bbc1575a0 2024-01-27T01:08:24.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.446+0000 7f0bb1ffb700 1 -- 172.21.15.46:0/217718253 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb40021b0 con 0x7f0bbc1575a0 2024-01-27T01:08:24.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.446+0000 7f0bc3f8e700 1 -- 172.21.15.46:0/217718253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bbc154270 con 0x7f0bbc1575a0 2024-01-27T01:08:24.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.446+0000 7f0bb1ffb700 1 -- 172.21.15.46:0/217718253 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb40106b0 con 0x7f0bbc1575a0 2024-01-27T01:08:24.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.447+0000 7f0bc3f8e700 1 -- 172.21.15.46:0/217718253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bbc1514d0 con 0x7f0bbc1575a0 2024-01-27T01:08:24.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.448+0000 7f0bc3f8e700 1 -- 172.21.15.46:0/217718253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bbc04c710 con 0x7f0bbc1575a0 2024-01-27T01:08:24.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.448+0000 7f0bb1ffb700 1 -- 172.21.15.46:0/217718253 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f0bb4010aa0 con 0x7f0bbc1575a0 2024-01-27T01:08:24.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.448+0000 7f0bb1ffb700 1 --2- 172.21.15.46:0/217718253 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0ba00670e0 0x7f0ba0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:24.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.449+0000 7f0bb1ffb700 1 -- 172.21.15.46:0/217718253 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f0bb4015070 con 0x7f0bbc1575a0 2024-01-27T01:08:24.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.449+0000 7f0bc1d2a700 1 --2- 172.21.15.46:0/217718253 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0ba00670e0 0x7f0ba0069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:24.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.449+0000 7f0bc1d2a700 1 --2- 172.21.15.46:0/217718253 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0ba00670e0 0x7f0ba0069590 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8007fd0 tx=0x7f0bb8009040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:24.452 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:24.452+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:24.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.454+0000 7f0bb1ffb700 1 -- 172.21.15.46:0/217718253 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0bb4053770 con 0x7f0bbc1575a0 2024-01-27T01:08:24.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.708+0000 7f0bc3f8e700 1 -- 172.21.15.46:0/217718253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bbc064be0 con 0x7f0bbc1575a0 2024-01-27T01:08:24.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.709+0000 7f0bb1ffb700 1 -- 172.21.15.46:0/217718253 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+4616 (secure 0 0 0) 0x7f0bb4056d90 con 0x7f0bbc1575a0 2024-01-27T01:08:24.708 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:24.708 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":149,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3826743114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3826743114},{"type":"v1","addr":"172.21.15.46:6835","nonce":3826743114}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":144},{"gid":11947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2514069949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2514069949},{"type":"v1","addr":"172.21.15.106:6833","nonce":2514069949}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":144},{"gid":11950,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/251184858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":251184858},{"type":"v1","addr":"172.21.15.106:6835","nonce":251184858}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:08:20.703665+0000","modified":"2024-01-27T01:08:24.272153+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12348},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12348":{"gid":12348,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6833/3798781145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3798781145},{"type":"v1","addr":"172.21.15.46:6833","nonce":3798781145}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2024-01-27T01:08:24.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.711+0000 7f0bc3f8e700 1 -- 172.21.15.46:0/217718253 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0ba00670e0 msgr2=0x7f0ba0069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:24.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.711+0000 7f0bc3f8e700 1 --2- 172.21.15.46:0/217718253 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0ba00670e0 0x7f0ba0069590 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8007fd0 tx=0x7f0bb8009040 comp rx=0 tx=0).stop 2024-01-27T01:08:24.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.711+0000 7f0bc3f8e700 1 -- 172.21.15.46:0/217718253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bbc1575a0 msgr2=0x7f0bbc0c9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:24.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.711+0000 7f0bc3f8e700 1 --2- 172.21.15.46:0/217718253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bbc1575a0 0x7f0bbc0c9f60 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f0bb400c860 tx=0x7f0bb4002700 comp rx=0 tx=0).stop 2024-01-27T01:08:24.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.711+0000 7f0bc3f8e700 1 -- 172.21.15.46:0/217718253 shutdown_connections 2024-01-27T01:08:24.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.711+0000 7f0bc3f8e700 1 --2- 172.21.15.46:0/217718253 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f0ba00670e0 0x7f0ba0069590 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.711+0000 7f0bc3f8e700 1 --2- 172.21.15.46:0/217718253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bbc1575a0 0x7f0bbc0c9f60 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.711+0000 7f0bc3f8e700 1 --2- 172.21.15.46:0/217718253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0bbc06e2a0 0x7f0bbc150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.711+0000 7f0bc3f8e700 1 --2- 172.21.15.46:0/217718253 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0bbc150720 0x7f0bbc14f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.711+0000 7f0bc3f8e700 1 -- 172.21.15.46:0/217718253 >> 172.21.15.46:0/217718253 conn(0x7f0bbc0ccbc0 msgr2=0x7f0bbc0cbf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:24.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.711+0000 7f0bc3f8e700 1 -- 172.21.15.46:0/217718253 shutdown_connections 2024-01-27T01:08:24.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.711+0000 7f0bc3f8e700 1 -- 172.21.15.46:0/217718253 wait complete. 2024-01-27T01:08:24.712 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 149 2024-01-27T01:08:24.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:08:24.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.858+0000 7ff0b783e700 1 Processor -- start 2024-01-27T01:08:24.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.858+0000 7ff0b783e700 1 -- start start 2024-01-27T01:08:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.859+0000 7ff0b783e700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff0b0067af0 0x7ff0b00cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.859+0000 7ff0b783e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0b0159d30 0x7ff0b015e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.859+0000 7ff0b783e700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b015e690 con 0x7ff0b0067ec0 2024-01-27T01:08:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.859+0000 7ff0b783e700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b00ce0a0 con 0x7ff0b0067af0 2024-01-27T01:08:24.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.859+0000 7ff0b783e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b00ce340 con 0x7ff0b0159d30 2024-01-27T01:08:24.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.859+0000 7ff0b55da700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff0b0067af0 0x7ff0b00cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:24.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.859+0000 7ff0b4dd9700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff0b0067ec0 0x7ff0b0159620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51718/0 (socket says 172.21.15.46:51718) 2024-01-27T01:08:24.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.859+0000 7ff0b4dd9700 1 -- 172.21.15.46:0/2117236130 learned_addr learned my addr 172.21.15.46:0/2117236130 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:24.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.859+0000 7ff0b5ddb700 1 --2- 172.21.15.46:0/2117236130 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0b0159d30 0x7ff0b015e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:24.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.860+0000 7ff0b55da700 1 -- 172.21.15.46:0/2117236130 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0b0159d30 msgr2=0x7ff0b015e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:24.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.860+0000 7ff0b55da700 1 --2- 172.21.15.46:0/2117236130 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0b0159d30 0x7ff0b015e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.860+0000 7ff0b55da700 1 -- 172.21.15.46:0/2117236130 >> v1:172.21.15.46:6789/0 conn(0x7ff0b0067ec0 legacy=0x7ff0b0159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:24.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.860+0000 7ff0b55da700 1 -- 172.21.15.46:0/2117236130 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b006c630 con 0x7ff0b0067af0 2024-01-27T01:08:24.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.860+0000 7ff0b55da700 1 --2- 172.21.15.46:0/2117236130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b0067af0 0x7ff0b00cbbc0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0098e0 tx=0x7ff0ac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9acec0dc8c76ffa server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:24.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.860+0000 7ff0a7fff700 1 -- 172.21.15.46:0/2117236130 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac00b640 con 0x7ff0b0067af0 2024-01-27T01:08:24.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.860+0000 7ff0a7fff700 1 -- 172.21.15.46:0/2117236130 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac00b7a0 con 0x7ff0b0067af0 2024-01-27T01:08:24.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.860+0000 7ff0a7fff700 1 -- 172.21.15.46:0/2117236130 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac00bb40 con 0x7ff0b0067af0 2024-01-27T01:08:24.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.861+0000 7ff0b783e700 1 -- 172.21.15.46:0/2117236130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b0067af0 msgr2=0x7ff0b00cbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:24.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.861+0000 7ff0b783e700 1 --2- 172.21.15.46:0/2117236130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b0067af0 0x7ff0b00cbbc0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0098e0 tx=0x7ff0ac0091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.861+0000 7ff0b783e700 1 -- 172.21.15.46:0/2117236130 shutdown_connections 2024-01-27T01:08:24.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.861+0000 7ff0b783e700 1 --2- 172.21.15.46:0/2117236130 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0b0159d30 0x7ff0b015e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.861+0000 7ff0b783e700 1 --2- 172.21.15.46:0/2117236130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b0067af0 0x7ff0b00cbbc0 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.861+0000 7ff0b783e700 1 -- 172.21.15.46:0/2117236130 >> 172.21.15.46:0/2117236130 conn(0x7ff0b006b820 msgr2=0x7ff0b00cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:24.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.861+0000 7ff0b783e700 1 -- 172.21.15.46:0/2117236130 shutdown_connections 2024-01-27T01:08:24.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.861+0000 7ff0b783e700 1 -- 172.21.15.46:0/2117236130 wait complete. 2024-01-27T01:08:24.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.862+0000 7ff0b783e700 1 Processor -- start 2024-01-27T01:08:24.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.862+0000 7ff0b783e700 1 -- start start 2024-01-27T01:08:24.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.863+0000 7ff0b783e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0b0067af0 0x7ff0b00c2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:24.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.863+0000 7ff0b783e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b0067ec0 0x7ff0b0156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:24.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.863+0000 7ff0b783e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0b0159d30 0x7ff0b01522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:24.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.863+0000 7ff0b783e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b0157400 con 0x7ff0b0067ec0 2024-01-27T01:08:24.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.863+0000 7ff0b783e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b01576e0 con 0x7ff0b0159d30 2024-01-27T01:08:24.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.863+0000 7ff0b783e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b01579c0 con 0x7ff0b0067af0 2024-01-27T01:08:24.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.863+0000 7ff0b4dd9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b0067ec0 0x7ff0b0156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:24.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.864+0000 7ff0b55da700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0b0067af0 0x7ff0b00c2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:24.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.864+0000 7ff0b5ddb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0b0159d30 0x7ff0b01522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:24.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.864+0000 7ff0b4dd9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b0067ec0 0x7ff0b0156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39022/0 (socket says 172.21.15.46:39022) 2024-01-27T01:08:24.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.864+0000 7ff0b4dd9700 1 -- 172.21.15.46:0/1273760059 learned_addr learned my addr 172.21.15.46:0/1273760059 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:24.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.864+0000 7ff0b5ddb700 1 -- 172.21.15.46:0/1273760059 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0b0067af0 msgr2=0x7ff0b00c2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:24.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.864+0000 7ff0b5ddb700 1 --2- 172.21.15.46:0/1273760059 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0b0067af0 0x7ff0b00c2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.864+0000 7ff0b5ddb700 1 -- 172.21.15.46:0/1273760059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b0067ec0 msgr2=0x7ff0b0156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:24.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.864+0000 7ff0b5ddb700 1 --2- 172.21.15.46:0/1273760059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b0067ec0 0x7ff0b0156d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:24.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.864+0000 7ff0b5ddb700 1 -- 172.21.15.46:0/1273760059 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ac009420 con 0x7ff0b0159d30 2024-01-27T01:08:24.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.864+0000 7ff0b55da700 1 --2- 172.21.15.46:0/1273760059 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0b0067af0 0x7ff0b00c2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:24.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.864+0000 7ff0b4dd9700 1 --2- 172.21.15.46:0/1273760059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b0067ec0 0x7ff0b0156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:24.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.864+0000 7ff0b5ddb700 1 --2- 172.21.15.46:0/1273760059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0b0159d30 0x7ff0b01522a0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009e50 tx=0x7ff0a800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:24.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.864+0000 7ff0a57fa700 1 -- 172.21.15.46:0/1273760059 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a800caa0 con 0x7ff0b0159d30 2024-01-27T01:08:24.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.864+0000 7ff0a57fa700 1 -- 172.21.15.46:0/1273760059 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a800f040 con 0x7ff0b0159d30 2024-01-27T01:08:24.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.864+0000 7ff0b783e700 1 -- 172.21.15.46:0/1273760059 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b0157ca0 con 0x7ff0b0159d30 2024-01-27T01:08:24.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.864+0000 7ff0a57fa700 1 -- 172.21.15.46:0/1273760059 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a800ce50 con 0x7ff0b0159d30 2024-01-27T01:08:24.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.864+0000 7ff0b783e700 1 -- 172.21.15.46:0/1273760059 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b01527e0 con 0x7ff0b0159d30 2024-01-27T01:08:24.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.865+0000 7ff0b783e700 1 -- 172.21.15.46:0/1273760059 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b004c710 con 0x7ff0b0159d30 2024-01-27T01:08:24.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.866+0000 7ff0a57fa700 1 -- 172.21.15.46:0/1273760059 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7ff0a8014690 con 0x7ff0b0159d30 2024-01-27T01:08:24.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.866+0000 7ff0a57fa700 1 --2- 172.21.15.46:0/1273760059 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff0940670e0 0x7ff094069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:24.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.866+0000 7ff0a57fa700 1 -- 172.21.15.46:0/1273760059 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6517+0+0 (secure 0 0 0) 0x7ff0a8085850 con 0x7ff0b0159d30 2024-01-27T01:08:24.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.866+0000 7ff0b55da700 1 --2- 172.21.15.46:0/1273760059 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff0940670e0 0x7ff094069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:24.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.867+0000 7ff0b55da700 1 --2- 172.21.15.46:0/1273760059 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff0940670e0 0x7ff094069590 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0093f0 tx=0x7ff0ac017000 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:24.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:24.871+0000 7ff0a57fa700 1 -- 172.21.15.46:0/1273760059 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff0a8052400 con 0x7ff0b0159d30 2024-01-27T01:08:25.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.083+0000 7ff0b783e700 1 -- 172.21.15.46:0/1273760059 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0b0157820 con 0x7ff0b0159d30 2024-01-27T01:08:25.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.084+0000 7ff0a57fa700 1 -- 172.21.15.46:0/1273760059 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v263) v1 ==== 74+0+16325 (secure 0 0 0) 0x7ff0b0157820 con 0x7ff0b0159d30 2024-01-27T01:08:25.083 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:25.083 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":263,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:08:22.260381+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:08:19.198347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-01-27T01:08:19.674810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"263","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:08:25.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.086+0000 7ff0b783e700 1 -- 172.21.15.46:0/1273760059 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff0940670e0 msgr2=0x7ff094069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:25.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.086+0000 7ff0b783e700 1 --2- 172.21.15.46:0/1273760059 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff0940670e0 0x7ff094069590 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0093f0 tx=0x7ff0ac017000 comp rx=0 tx=0).stop 2024-01-27T01:08:25.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.086+0000 7ff0b783e700 1 -- 172.21.15.46:0/1273760059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0b0159d30 msgr2=0x7ff0b01522a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:25.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.086+0000 7ff0b783e700 1 --2- 172.21.15.46:0/1273760059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0b0159d30 0x7ff0b01522a0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009e50 tx=0x7ff0a800b040 comp rx=0 tx=0).stop 2024-01-27T01:08:25.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.086+0000 7ff0b783e700 1 -- 172.21.15.46:0/1273760059 shutdown_connections 2024-01-27T01:08:25.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.086+0000 7ff0b783e700 1 --2- 172.21.15.46:0/1273760059 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff0940670e0 0x7ff094069590 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.086+0000 7ff0b783e700 1 --2- 172.21.15.46:0/1273760059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff0b0159d30 0x7ff0b01522a0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.086+0000 7ff0b783e700 1 --2- 172.21.15.46:0/1273760059 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff0b0067af0 0x7ff0b00c2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.086+0000 7ff0b783e700 1 --2- 172.21.15.46:0/1273760059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0b0067ec0 0x7ff0b0156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.086+0000 7ff0b783e700 1 -- 172.21.15.46:0/1273760059 >> 172.21.15.46:0/1273760059 conn(0x7ff0b006b820 msgr2=0x7ff0b0154120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:25.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.086+0000 7ff0b783e700 1 -- 172.21.15.46:0/1273760059 shutdown_connections 2024-01-27T01:08:25.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.086+0000 7ff0b783e700 1 -- 172.21.15.46:0/1273760059 wait complete. 2024-01-27T01:08:25.099 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-27T01:08:25.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:08:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.234+0000 7fe4f39e2700 1 Processor -- start 2024-01-27T01:08:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.234+0000 7fe4f39e2700 1 -- start start 2024-01-27T01:08:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.235+0000 7fe4f39e2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4ec159590 0x7fe4ec15d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.235+0000 7fe4f39e2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ec15dd90 con 0x7fe4ec06c3c0 2024-01-27T01:08:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.235+0000 7fe4f39e2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ec15e070 con 0x7fe4ec15d8f0 2024-01-27T01:08:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.235+0000 7fe4f39e2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec15e350 con 0x7fe4ec159590 2024-01-27T01:08:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.235+0000 7fe4f1f7f700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe4ec06c3c0 0x7fe4ec069370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51732/0 (socket says 172.21.15.46:51732) 2024-01-27T01:08:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.235+0000 7fe4f1f7f700 1 -- 172.21.15.46:0/4074729216 learned_addr learned my addr 172.21.15.46:0/4074729216 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.235+0000 7fe4f177e700 1 --2- 172.21.15.46:0/4074729216 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4ec159590 0x7fe4ec15d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.235+0000 7fe4e3fff700 1 -- 172.21.15.46:0/4074729216 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185651333 0 0) 0x7fe4ec15dd90 con 0x7fe4ec06c3c0 2024-01-27T01:08:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.235+0000 7fe4e3fff700 1 -- 172.21.15.46:0/4074729216 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d8003570 con 0x7fe4ec06c3c0 2024-01-27T01:08:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.235+0000 7fe4e3fff700 1 -- 172.21.15.46:0/4074729216 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350208379 0 0) 0x7fe4ec15e070 con 0x7fe4ec15d8f0 2024-01-27T01:08:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.235+0000 7fe4e3fff700 1 -- 172.21.15.46:0/4074729216 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4ec15dd90 con 0x7fe4ec15d8f0 2024-01-27T01:08:25.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.235+0000 7fe4f177e700 1 -- 172.21.15.46:0/4074729216 >> v1:172.21.15.106:6789/0 conn(0x7fe4ec15d8f0 legacy=0x7fe4ec06bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:25.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.235+0000 7fe4f177e700 1 -- 172.21.15.46:0/4074729216 >> v1:172.21.15.46:6789/0 conn(0x7fe4ec06c3c0 legacy=0x7fe4ec069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:25.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.235+0000 7fe4f177e700 1 -- 172.21.15.46:0/4074729216 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec068b70 con 0x7fe4ec159590 2024-01-27T01:08:25.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.236+0000 7fe4f177e700 1 --2- 172.21.15.46:0/4074729216 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4ec159590 0x7fe4ec15d3b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8009910 tx=0x7fe4e80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1d4a41277704c92 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:25.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.236+0000 7fe4e3fff700 1 -- 172.21.15.46:0/4074729216 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8011630 con 0x7fe4ec159590 2024-01-27T01:08:25.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.236+0000 7fe4e3fff700 1 -- 172.21.15.46:0/4074729216 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e8011790 con 0x7fe4ec159590 2024-01-27T01:08:25.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.236+0000 7fe4e3fff700 1 -- 172.21.15.46:0/4074729216 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8011b30 con 0x7fe4ec159590 2024-01-27T01:08:25.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.236+0000 7fe4f39e2700 1 -- 172.21.15.46:0/4074729216 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4ec159590 msgr2=0x7fe4ec15d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:25.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.236+0000 7fe4f39e2700 1 --2- 172.21.15.46:0/4074729216 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4ec159590 0x7fe4ec15d3b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8009910 tx=0x7fe4e80091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.236+0000 7fe4f39e2700 1 -- 172.21.15.46:0/4074729216 shutdown_connections 2024-01-27T01:08:25.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.236+0000 7fe4f39e2700 1 --2- 172.21.15.46:0/4074729216 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4ec159590 0x7fe4ec15d3b0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.236+0000 7fe4f39e2700 1 -- 172.21.15.46:0/4074729216 >> 172.21.15.46:0/4074729216 conn(0x7fe4ec0ce530 msgr2=0x7fe4ec0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:25.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.237+0000 7fe4f39e2700 1 -- 172.21.15.46:0/4074729216 shutdown_connections 2024-01-27T01:08:25.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.237+0000 7fe4f39e2700 1 -- 172.21.15.46:0/4074729216 wait complete. 2024-01-27T01:08:25.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.238+0000 7fe4f39e2700 1 Processor -- start 2024-01-27T01:08:25.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.238+0000 7fe4f39e2700 1 -- start start 2024-01-27T01:08:25.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4f39e2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4ec06c3c0 0x7fe4ec155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:25.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4f39e2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4ec159590 0x7fe4ec169c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:25.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4f39e2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4ec15d8f0 0x7fe4ec0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:25.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4f39e2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec156220 con 0x7fe4ec06c3c0 2024-01-27T01:08:25.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4f39e2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec0b1380 con 0x7fe4ec159590 2024-01-27T01:08:25.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4f39e2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec16a620 con 0x7fe4ec15d8f0 2024-01-27T01:08:25.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4f177e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4ec06c3c0 0x7fe4ec155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4f177e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4ec06c3c0 0x7fe4ec155ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39038/0 (socket says 172.21.15.46:39038) 2024-01-27T01:08:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4f177e700 1 -- 172.21.15.46:0/1817585450 learned_addr learned my addr 172.21.15.46:0/1817585450 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4f0f7d700 1 --2- 172.21.15.46:0/1817585450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4ec159590 0x7fe4ec169c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4f1f7f700 1 --2- 172.21.15.46:0/1817585450 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4ec15d8f0 0x7fe4ec0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4f177e700 1 -- 172.21.15.46:0/1817585450 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4ec15d8f0 msgr2=0x7fe4ec0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4f177e700 1 --2- 172.21.15.46:0/1817585450 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4ec15d8f0 0x7fe4ec0b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4f177e700 1 -- 172.21.15.46:0/1817585450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4ec159590 msgr2=0x7fe4ec169c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4f177e700 1 --2- 172.21.15.46:0/1817585450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4ec159590 0x7fe4ec169c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4f177e700 1 -- 172.21.15.46:0/1817585450 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e8009420 con 0x7fe4ec06c3c0 2024-01-27T01:08:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4f177e700 1 --2- 172.21.15.46:0/1817585450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4ec06c3c0 0x7fe4ec155ce0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8009f00 tx=0x7fe4e8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.239+0000 7fe4e17fa700 1 -- 172.21.15.46:0/1817585450 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e801f070 con 0x7fe4ec06c3c0 2024-01-27T01:08:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.240+0000 7fe4e17fa700 1 -- 172.21.15.46:0/1817585450 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e8002f70 con 0x7fe4ec06c3c0 2024-01-27T01:08:25.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.240+0000 7fe4f39e2700 1 -- 172.21.15.46:0/1817585450 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ec16a900 con 0x7fe4ec06c3c0 2024-01-27T01:08:25.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.240+0000 7fe4e17fa700 1 -- 172.21.15.46:0/1817585450 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8011900 con 0x7fe4ec06c3c0 2024-01-27T01:08:25.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.240+0000 7fe4f39e2700 1 -- 172.21.15.46:0/1817585450 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ec16abe0 con 0x7fe4ec06c3c0 2024-01-27T01:08:25.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.241+0000 7fe4f39e2700 1 -- 172.21.15.46:0/1817585450 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4ec04c710 con 0x7fe4ec06c3c0 2024-01-27T01:08:25.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.241+0000 7fe4e17fa700 1 -- 172.21.15.46:0/1817585450 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7fe4e8023070 con 0x7fe4ec06c3c0 2024-01-27T01:08:25.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.242+0000 7fe4e17fa700 1 --2- 172.21.15.46:0/1817585450 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe4d00670e0 0x7fe4d0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:25.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.242+0000 7fe4e17fa700 1 -- 172.21.15.46:0/1817585450 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fe4e8085e10 con 0x7fe4ec06c3c0 2024-01-27T01:08:25.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.242+0000 7fe4f0f7d700 1 --2- 172.21.15.46:0/1817585450 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe4d00670e0 0x7fe4d0069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:25.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.242+0000 7fe4f0f7d700 1 --2- 172.21.15.46:0/1817585450 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe4d00670e0 0x7fe4d0069590 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec0b3a50 tx=0x7fe4dc009040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:25.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.246+0000 7fe4e17fa700 1 -- 172.21.15.46:0/1817585450 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe4e8052ad0 con 0x7fe4ec06c3c0 2024-01-27T01:08:25.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.455+0000 7fe4f39e2700 1 -- 172.21.15.46:0/1817585450 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe4ec06c9a0 con 0x7fe4d00670e0 2024-01-27T01:08:25.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.463+0000 7fe4e17fa700 1 -- 172.21.15.46:0/1817585450 <== mgr.11614 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290112 (secure 0 0 0) 0x7fe4ec06c9a0 con 0x7fe4d00670e0 2024-01-27T01:08:25.466 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:25.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.467+0000 7fe4f39e2700 1 -- 172.21.15.46:0/1817585450 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe4d00670e0 msgr2=0x7fe4d0069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.467+0000 7fe4f39e2700 1 --2- 172.21.15.46:0/1817585450 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe4d00670e0 0x7fe4d0069590 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec0b3a50 tx=0x7fe4dc009040 comp rx=0 tx=0).stop 2024-01-27T01:08:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.467+0000 7fe4f39e2700 1 -- 172.21.15.46:0/1817585450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4ec06c3c0 msgr2=0x7fe4ec155ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.467+0000 7fe4f39e2700 1 --2- 172.21.15.46:0/1817585450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4ec06c3c0 0x7fe4ec155ce0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8009f00 tx=0x7fe4e8017000 comp rx=0 tx=0).stop 2024-01-27T01:08:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.467+0000 7fe4f39e2700 1 -- 172.21.15.46:0/1817585450 shutdown_connections 2024-01-27T01:08:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.467+0000 7fe4f39e2700 1 --2- 172.21.15.46:0/1817585450 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fe4d00670e0 0x7fe4d0069590 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.467+0000 7fe4f39e2700 1 --2- 172.21.15.46:0/1817585450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4ec06c3c0 0x7fe4ec155ce0 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.467+0000 7fe4f39e2700 1 --2- 172.21.15.46:0/1817585450 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4ec15d8f0 0x7fe4ec0b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.467+0000 7fe4f39e2700 1 --2- 172.21.15.46:0/1817585450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4ec159590 0x7fe4ec169c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.467+0000 7fe4f39e2700 1 -- 172.21.15.46:0/1817585450 >> 172.21.15.46:0/1817585450 conn(0x7fe4ec0ce530 msgr2=0x7fe4ec156a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.467+0000 7fe4f39e2700 1 -- 172.21.15.46:0/1817585450 shutdown_connections 2024-01-27T01:08:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.467+0000 7fe4f39e2700 1 -- 172.21.15.46:0/1817585450 wait complete. 2024-01-27T01:08:25.469 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:08:25.472 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:25.472+0000 7f9adfb35700 -1 client.11722: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.11557:d0e77da6-cfcd-420e-a110-0a62fbf79c65:v1:172.21.15.106:0/3540935782] 2024-01-27T01:08:25.480 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-01-27T01:08:25.093393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2528,"num_read_kb":5347,"num_write":724,"num_write_kb":3475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8554,"ondisk_log_size":8554,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":72,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2483228,"kb_used_data":27100,"kb_used_omap":0,"kb_used_meta":2456064,"kb_avail":752491492,"statfs":{"total":773094113280,"available":770551287808,"internally_reserved":0,"allocated":27750400,"data_stored":15292070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2515009536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.838356"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.276679+0000","last_change":"2024-01-27T01:08:21.276679+0000","last_active":"2024-01-27T01:08:21.276679+0000","last_peered":"2024-01-27T01:08:21.276679+0000","last_clean":"2024-01-27T01:08:21.276679+0000","last_became_active":"2024-01-27T01:08:21.276413+0000","last_became_peered":"2024-01-27T01:08:21.276413+0000","last_unstale":"2024-01-27T01:08:21.276679+0000","last_undegraded":"2024-01-27T01:08:21.276679+0000","last_fullsized":"2024-01-27T01:08:21.276679+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:16:52.898998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.286323+0000","last_change":"2024-01-27T01:08:21.286323+0000","last_active":"2024-01-27T01:08:21.286323+0000","last_peered":"2024-01-27T01:08:21.286323+0000","last_clean":"2024-01-27T01:08:21.286323+0000","last_became_active":"2024-01-27T01:08:21.286073+0000","last_became_peered":"2024-01-27T01:08:21.286073+0000","last_unstale":"2024-01-27T01:08:21.286323+0000","last_undegraded":"2024-01-27T01:08:21.286323+0000","last_fullsized":"2024-01-27T01:08:21.286323+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:48:57.898191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.288314+0000","last_change":"2024-01-27T01:08:21.288314+0000","last_active":"2024-01-27T01:08:21.288314+0000","last_peered":"2024-01-27T01:08:21.288314+0000","last_clean":"2024-01-27T01:08:21.288314+0000","last_became_active":"2024-01-27T01:08:21.288027+0000","last_became_peered":"2024-01-27T01:08:21.288027+0000","last_unstale":"2024-01-27T01:08:21.288314+0000","last_undegraded":"2024-01-27T01:08:21.288314+0000","last_fullsized":"2024-01-27T01:08:21.288314+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:08:59.412251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.272715+0000","last_change":"2024-01-27T01:08:21.272715+0000","last_active":"2024-01-27T01:08:21.272715+0000","last_peered":"2024-01-27T01:08:21.272715+0000","last_clean":"2024-01-27T01:08:21.272715+0000","last_became_active":"2024-01-27T01:08:21.272411+0000","last_became_peered":"2024-01-27T01:08:21.272411+0000","last_unstale":"2024-01-27T01:08:21.272715+0000","last_undegraded":"2024-01-27T01:08:21.272715+0000","last_fullsized":"2024-01-27T01:08:21.272715+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:07:52.525416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.261376+0000","last_change":"2024-01-27T01:08:21.261376+0000","last_active":"2024-01-27T01:08:21.261376+0000","last_peered":"2024-01-27T01:08:21.261376+0000","last_clean":"2024-01-27T01:08:21.261376+0000","last_became_active":"2024-01-27T01:08:21.261073+0000","last_became_peered":"2024-01-27T01:08:21.261073+0000","last_unstale":"2024-01-27T01:08:21.261376+0000","last_undegraded":"2024-01-27T01:08:21.261376+0000","last_fullsized":"2024-01-27T01:08:21.261376+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:47.554300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.284818+0000","last_change":"2024-01-27T01:08:21.284818+0000","last_active":"2024-01-27T01:08:21.284818+0000","last_peered":"2024-01-27T01:08:21.284818+0000","last_clean":"2024-01-27T01:08:21.284818+0000","last_became_active":"2024-01-27T01:08:21.284500+0000","last_became_peered":"2024-01-27T01:08:21.284500+0000","last_unstale":"2024-01-27T01:08:21.284818+0000","last_undegraded":"2024-01-27T01:08:21.284818+0000","last_fullsized":"2024-01-27T01:08:21.284818+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:58.708489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.274959+0000","last_change":"2024-01-27T01:08:21.274959+0000","last_active":"2024-01-27T01:08:21.274959+0000","last_peered":"2024-01-27T01:08:21.274959+0000","last_clean":"2024-01-27T01:08:21.274959+0000","last_became_active":"2024-01-27T01:08:21.274713+0000","last_became_peered":"2024-01-27T01:08:21.274713+0000","last_unstale":"2024-01-27T01:08:21.274959+0000","last_undegraded":"2024-01-27T01:08:21.274959+0000","last_fullsized":"2024-01-27T01:08:21.274959+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:49:53.469375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.278371+0000","last_change":"2024-01-27T01:08:21.278371+0000","last_active":"2024-01-27T01:08:21.278371+0000","last_peered":"2024-01-27T01:08:21.278371+0000","last_clean":"2024-01-27T01:08:21.278371+0000","last_became_active":"2024-01-27T01:08:21.278126+0000","last_became_peered":"2024-01-27T01:08:21.278126+0000","last_unstale":"2024-01-27T01:08:21.278371+0000","last_undegraded":"2024-01-27T01:08:21.278371+0000","last_fullsized":"2024-01-27T01:08:21.278371+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:39:38.359802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.289308+0000","last_change":"2024-01-27T01:08:21.289308+0000","last_active":"2024-01-27T01:08:21.289308+0000","last_peered":"2024-01-27T01:08:21.289308+0000","last_clean":"2024-01-27T01:08:21.289308+0000","last_became_active":"2024-01-27T01:08:21.289059+0000","last_became_peered":"2024-01-27T01:08:21.289059+0000","last_unstale":"2024-01-27T01:08:21.289308+0000","last_undegraded":"2024-01-27T01:08:21.289308+0000","last_fullsized":"2024-01-27T01:08:21.289308+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:30:08.847499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.270204+0000","last_change":"2024-01-27T01:08:21.270204+0000","last_active":"2024-01-27T01:08:21.270204+0000","last_peered":"2024-01-27T01:08:21.270204+0000","last_clean":"2024-01-27T01:08:21.270204+0000","last_became_active":"2024-01-27T01:08:21.269959+0000","last_became_peered":"2024-01-27T01:08:21.269959+0000","last_unstale":"2024-01-27T01:08:21.270204+0000","last_undegraded":"2024-01-27T01:08:21.270204+0000","last_fullsized":"2024-01-27T01:08:21.270204+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:03.783822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.276079+0000","last_change":"2024-01-27T01:08:21.276079+0000","last_active":"2024-01-27T01:08:21.276079+0000","last_peered":"2024-01-27T01:08:21.276079+0000","last_clean":"2024-01-27T01:08:21.276079+0000","last_became_active":"2024-01-27T01:08:21.275642+0000","last_became_peered":"2024-01-27T01:08:21.275642+0000","last_unstale":"2024-01-27T01:08:21.276079+0000","last_undegraded":"2024-01-27T01:08:21.276079+0000","last_fullsized":"2024-01-27T01:08:21.276079+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:18:01.913878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.276250+0000","last_change":"2024-01-27T01:08:21.276250+0000","last_active":"2024-01-27T01:08:21.276250+0000","last_peered":"2024-01-27T01:08:21.276250+0000","last_clean":"2024-01-27T01:08:21.276250+0000","last_became_active":"2024-01-27T01:08:21.276000+0000","last_became_peered":"2024-01-27T01:08:21.276000+0000","last_unstale":"2024-01-27T01:08:21.276250+0000","last_undegraded":"2024-01-27T01:08:21.276250+0000","last_fullsized":"2024-01-27T01:08:21.276250+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:52:12.651940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.277100+0000","last_change":"2024-01-27T01:08:21.277100+0000","last_active":"2024-01-27T01:08:21.277100+0000","last_peered":"2024-01-27T01:08:21.277100+0000","last_clean":"2024-01-27T01:08:21.277100+0000","last_became_active":"2024-01-27T01:08:21.276785+0000","last_became_peered":"2024-01-27T01:08:21.276785+0000","last_unstale":"2024-01-27T01:08:21.277100+0000","last_undegraded":"2024-01-27T01:08:21.277100+0000","last_fullsized":"2024-01-27T01:08:21.277100+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:54:14.121153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.269660+0000","last_change":"2024-01-27T01:08:21.269660+0000","last_active":"2024-01-27T01:08:21.269660+0000","last_peered":"2024-01-27T01:08:21.269660+0000","last_clean":"2024-01-27T01:08:21.269660+0000","last_became_active":"2024-01-27T01:08:21.269401+0000","last_became_peered":"2024-01-27T01:08:21.269401+0000","last_unstale":"2024-01-27T01:08:21.269660+0000","last_undegraded":"2024-01-27T01:08:21.269660+0000","last_fullsized":"2024-01-27T01:08:21.269660+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:32:20.263478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.278476+0000","last_change":"2024-01-27T01:08:21.278476+0000","last_active":"2024-01-27T01:08:21.278476+0000","last_peered":"2024-01-27T01:08:21.278476+0000","last_clean":"2024-01-27T01:08:21.278476+0000","last_became_active":"2024-01-27T01:08:21.278177+0000","last_became_peered":"2024-01-27T01:08:21.278177+0000","last_unstale":"2024-01-27T01:08:21.278476+0000","last_undegraded":"2024-01-27T01:08:21.278476+0000","last_fullsized":"2024-01-27T01:08:21.278476+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:22:44.405022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.270324+0000","last_change":"2024-01-27T01:08:21.270324+0000","last_active":"2024-01-27T01:08:21.270324+0000","last_peered":"2024-01-27T01:08:21.270324+0000","last_clean":"2024-01-27T01:08:21.270324+0000","last_became_active":"2024-01-27T01:08:21.270120+0000","last_became_peered":"2024-01-27T01:08:21.270120+0000","last_unstale":"2024-01-27T01:08:21.270324+0000","last_undegraded":"2024-01-27T01:08:21.270324+0000","last_fullsized":"2024-01-27T01:08:21.270324+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:21:05.876052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.282682+0000","last_change":"2024-01-27T01:08:21.282682+0000","last_active":"2024-01-27T01:08:21.282682+0000","last_peered":"2024-01-27T01:08:21.282682+0000","last_clean":"2024-01-27T01:08:21.282682+0000","last_became_active":"2024-01-27T01:08:21.282458+0000","last_became_peered":"2024-01-27T01:08:21.282458+0000","last_unstale":"2024-01-27T01:08:21.282682+0000","last_undegraded":"2024-01-27T01:08:21.282682+0000","last_fullsized":"2024-01-27T01:08:21.282682+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:41:09.975158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.294910+0000","last_change":"2024-01-27T01:08:21.294910+0000","last_active":"2024-01-27T01:08:21.294910+0000","last_peered":"2024-01-27T01:08:21.294910+0000","last_clean":"2024-01-27T01:08:21.294910+0000","last_became_active":"2024-01-27T01:08:21.294637+0000","last_became_peered":"2024-01-27T01:08:21.294637+0000","last_unstale":"2024-01-27T01:08:21.294910+0000","last_undegraded":"2024-01-27T01:08:21.294910+0000","last_fullsized":"2024-01-27T01:08:21.294910+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:19.239747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.274066+0000","last_change":"2024-01-27T01:08:21.274066+0000","last_active":"2024-01-27T01:08:21.274066+0000","last_peered":"2024-01-27T01:08:21.274066+0000","last_clean":"2024-01-27T01:08:21.274066+0000","last_became_active":"2024-01-27T01:08:21.273831+0000","last_became_peered":"2024-01-27T01:08:21.273831+0000","last_unstale":"2024-01-27T01:08:21.274066+0000","last_undegraded":"2024-01-27T01:08:21.274066+0000","last_fullsized":"2024-01-27T01:08:21.274066+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:49:55.152011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.275813+0000","last_change":"2024-01-27T01:08:21.275813+0000","last_active":"2024-01-27T01:08:21.275813+0000","last_peered":"2024-01-27T01:08:21.275813+0000","last_clean":"2024-01-27T01:08:21.275813+0000","last_became_active":"2024-01-27T01:08:21.275526+0000","last_became_peered":"2024-01-27T01:08:21.275526+0000","last_unstale":"2024-01-27T01:08:21.275813+0000","last_undegraded":"2024-01-27T01:08:21.275813+0000","last_fullsized":"2024-01-27T01:08:21.275813+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:37:04.315404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.290800+0000","last_change":"2024-01-27T01:08:21.290800+0000","last_active":"2024-01-27T01:08:21.290800+0000","last_peered":"2024-01-27T01:08:21.290800+0000","last_clean":"2024-01-27T01:08:21.290800+0000","last_became_active":"2024-01-27T01:08:21.290562+0000","last_became_peered":"2024-01-27T01:08:21.290562+0000","last_unstale":"2024-01-27T01:08:21.290800+0000","last_undegraded":"2024-01-27T01:08:21.290800+0000","last_fullsized":"2024-01-27T01:08:21.290800+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:02:41.866282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.284607+0000","last_change":"2024-01-27T01:08:21.284607+0000","last_active":"2024-01-27T01:08:21.284607+0000","last_peered":"2024-01-27T01:08:21.284607+0000","last_clean":"2024-01-27T01:08:21.284607+0000","last_became_active":"2024-01-27T01:08:21.284279+0000","last_became_peered":"2024-01-27T01:08:21.284279+0000","last_unstale":"2024-01-27T01:08:21.284607+0000","last_undegraded":"2024-01-27T01:08:21.284607+0000","last_fullsized":"2024-01-27T01:08:21.284607+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:38:00.882525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.282751+0000","last_change":"2024-01-27T01:08:21.282751+0000","last_active":"2024-01-27T01:08:21.282751+0000","last_peered":"2024-01-27T01:08:21.282751+0000","last_clean":"2024-01-27T01:08:21.282751+0000","last_became_active":"2024-01-27T01:08:21.282483+0000","last_became_peered":"2024-01-27T01:08:21.282483+0000","last_unstale":"2024-01-27T01:08:21.282751+0000","last_undegraded":"2024-01-27T01:08:21.282751+0000","last_fullsized":"2024-01-27T01:08:21.282751+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:59:19.338283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.286437+0000","last_change":"2024-01-27T01:08:21.286437+0000","last_active":"2024-01-27T01:08:21.286437+0000","last_peered":"2024-01-27T01:08:21.286437+0000","last_clean":"2024-01-27T01:08:21.286437+0000","last_became_active":"2024-01-27T01:08:21.285799+0000","last_became_peered":"2024-01-27T01:08:21.285799+0000","last_unstale":"2024-01-27T01:08:21.286437+0000","last_undegraded":"2024-01-27T01:08:21.286437+0000","last_fullsized":"2024-01-27T01:08:21.286437+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:20:40.823053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.260813+0000","last_change":"2024-01-27T01:08:21.260813+0000","last_active":"2024-01-27T01:08:21.260813+0000","last_peered":"2024-01-27T01:08:21.260813+0000","last_clean":"2024-01-27T01:08:21.260813+0000","last_became_active":"2024-01-27T01:08:21.260543+0000","last_became_peered":"2024-01-27T01:08:21.260543+0000","last_unstale":"2024-01-27T01:08:21.260813+0000","last_undegraded":"2024-01-27T01:08:21.260813+0000","last_fullsized":"2024-01-27T01:08:21.260813+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:49:57.871691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.263075+0000","last_change":"2024-01-27T01:08:21.263075+0000","last_active":"2024-01-27T01:08:21.263075+0000","last_peered":"2024-01-27T01:08:21.263075+0000","last_clean":"2024-01-27T01:08:21.263075+0000","last_became_active":"2024-01-27T01:08:21.262800+0000","last_became_peered":"2024-01-27T01:08:21.262800+0000","last_unstale":"2024-01-27T01:08:21.263075+0000","last_undegraded":"2024-01-27T01:08:21.263075+0000","last_fullsized":"2024-01-27T01:08:21.263075+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:02:12.652310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.272775+0000","last_change":"2024-01-27T01:08:21.272775+0000","last_active":"2024-01-27T01:08:21.272775+0000","last_peered":"2024-01-27T01:08:21.272775+0000","last_clean":"2024-01-27T01:08:21.272775+0000","last_became_active":"2024-01-27T01:08:21.272170+0000","last_became_peered":"2024-01-27T01:08:21.272170+0000","last_unstale":"2024-01-27T01:08:21.272775+0000","last_undegraded":"2024-01-27T01:08:21.272775+0000","last_fullsized":"2024-01-27T01:08:21.272775+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:55.133025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.287706+0000","last_change":"2024-01-27T01:08:21.287706+0000","last_active":"2024-01-27T01:08:21.287706+0000","last_peered":"2024-01-27T01:08:21.287706+0000","last_clean":"2024-01-27T01:08:21.287706+0000","last_became_active":"2024-01-27T01:08:21.287381+0000","last_became_peered":"2024-01-27T01:08:21.287381+0000","last_unstale":"2024-01-27T01:08:21.287706+0000","last_undegraded":"2024-01-27T01:08:21.287706+0000","last_fullsized":"2024-01-27T01:08:21.287706+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:48:26.555763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.270701+0000","last_change":"2024-01-27T01:08:21.270701+0000","last_active":"2024-01-27T01:08:21.270701+0000","last_peered":"2024-01-27T01:08:21.270701+0000","last_clean":"2024-01-27T01:08:21.270701+0000","last_became_active":"2024-01-27T01:08:21.270483+0000","last_became_peered":"2024-01-27T01:08:21.270483+0000","last_unstale":"2024-01-27T01:08:21.270701+0000","last_undegraded":"2024-01-27T01:08:21.270701+0000","last_fullsized":"2024-01-27T01:08:21.270701+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:26:15.721480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.287910+0000","last_change":"2024-01-27T01:08:21.287910+0000","last_active":"2024-01-27T01:08:21.287910+0000","last_peered":"2024-01-27T01:08:21.287910+0000","last_clean":"2024-01-27T01:08:21.287910+0000","last_became_active":"2024-01-27T01:08:21.286874+0000","last_became_peered":"2024-01-27T01:08:21.286874+0000","last_unstale":"2024-01-27T01:08:21.287910+0000","last_undegraded":"2024-01-27T01:08:21.287910+0000","last_fullsized":"2024-01-27T01:08:21.287910+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:46:54.670266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.284650+0000","last_change":"2024-01-27T01:08:21.284650+0000","last_active":"2024-01-27T01:08:21.284650+0000","last_peered":"2024-01-27T01:08:21.284650+0000","last_clean":"2024-01-27T01:08:21.284650+0000","last_became_active":"2024-01-27T01:08:21.284356+0000","last_became_peered":"2024-01-27T01:08:21.284356+0000","last_unstale":"2024-01-27T01:08:21.284650+0000","last_undegraded":"2024-01-27T01:08:21.284650+0000","last_fullsized":"2024-01-27T01:08:21.284650+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:29:07.694697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.287297+0000","last_change":"2024-01-27T01:08:21.287297+0000","last_active":"2024-01-27T01:08:21.287297+0000","last_peered":"2024-01-27T01:08:21.287297+0000","last_clean":"2024-01-27T01:08:21.287297+0000","last_became_active":"2024-01-27T01:08:21.286118+0000","last_became_peered":"2024-01-27T01:08:21.286118+0000","last_unstale":"2024-01-27T01:08:21.287297+0000","last_undegraded":"2024-01-27T01:08:21.287297+0000","last_fullsized":"2024-01-27T01:08:21.287297+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:59:30.434717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"262'4","reported_seq":13,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.294426+0000","last_change":"2024-01-27T01:08:20.270085+0000","last_active":"2024-01-27T01:08:21.294426+0000","last_peered":"2024-01-27T01:08:21.294426+0000","last_clean":"2024-01-27T01:08:21.294426+0000","last_became_active":"2024-01-27T01:08:20.269801+0000","last_became_peered":"2024-01-27T01:08:20.269801+0000","last_unstale":"2024-01-27T01:08:21.294426+0000","last_undegraded":"2024-01-27T01:08:21.294426+0000","last_fullsized":"2024-01-27T01:08:21.294426+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:03:09.844911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.270768+0000","last_change":"2024-01-27T01:08:21.270768+0000","last_active":"2024-01-27T01:08:21.270768+0000","last_peered":"2024-01-27T01:08:21.270768+0000","last_clean":"2024-01-27T01:08:21.270768+0000","last_became_active":"2024-01-27T01:08:21.270557+0000","last_became_peered":"2024-01-27T01:08:21.270557+0000","last_unstale":"2024-01-27T01:08:21.270768+0000","last_undegraded":"2024-01-27T01:08:21.270768+0000","last_fullsized":"2024-01-27T01:08:21.270768+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:21:11.579156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"262'1","reported_seq":10,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.262539+0000","last_change":"2024-01-27T01:08:20.271567+0000","last_active":"2024-01-27T01:08:21.262539+0000","last_peered":"2024-01-27T01:08:21.262539+0000","last_clean":"2024-01-27T01:08:21.262539+0000","last_became_active":"2024-01-27T01:08:20.271259+0000","last_became_peered":"2024-01-27T01:08:20.271259+0000","last_unstale":"2024-01-27T01:08:21.262539+0000","last_undegraded":"2024-01-27T01:08:21.262539+0000","last_fullsized":"2024-01-27T01:08:21.262539+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:18:00.912354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.270146+0000","last_change":"2024-01-27T01:08:21.270146+0000","last_active":"2024-01-27T01:08:21.270146+0000","last_peered":"2024-01-27T01:08:21.270146+0000","last_clean":"2024-01-27T01:08:21.270146+0000","last_became_active":"2024-01-27T01:08:21.269888+0000","last_became_peered":"2024-01-27T01:08:21.269888+0000","last_unstale":"2024-01-27T01:08:21.270146+0000","last_undegraded":"2024-01-27T01:08:21.270146+0000","last_fullsized":"2024-01-27T01:08:21.270146+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:46:36.752556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.290660+0000","last_change":"2024-01-27T01:08:20.267577+0000","last_active":"2024-01-27T01:08:21.290660+0000","last_peered":"2024-01-27T01:08:21.290660+0000","last_clean":"2024-01-27T01:08:21.290660+0000","last_became_active":"2024-01-27T01:08:20.267292+0000","last_became_peered":"2024-01-27T01:08:20.267292+0000","last_unstale":"2024-01-27T01:08:21.290660+0000","last_undegraded":"2024-01-27T01:08:21.290660+0000","last_fullsized":"2024-01-27T01:08:21.290660+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:09.128990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.288860+0000","last_change":"2024-01-27T01:08:21.288860+0000","last_active":"2024-01-27T01:08:21.288860+0000","last_peered":"2024-01-27T01:08:21.288860+0000","last_clean":"2024-01-27T01:08:21.288860+0000","last_became_active":"2024-01-27T01:08:21.288591+0000","last_became_peered":"2024-01-27T01:08:21.288591+0000","last_unstale":"2024-01-27T01:08:21.288860+0000","last_undegraded":"2024-01-27T01:08:21.288860+0000","last_fullsized":"2024-01-27T01:08:21.288860+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:08:26.343440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.291156+0000","last_change":"2024-01-27T01:08:20.268588+0000","last_active":"2024-01-27T01:08:21.291156+0000","last_peered":"2024-01-27T01:08:21.291156+0000","last_clean":"2024-01-27T01:08:21.291156+0000","last_became_active":"2024-01-27T01:08:20.268304+0000","last_became_peered":"2024-01-27T01:08:20.268304+0000","last_unstale":"2024-01-27T01:08:21.291156+0000","last_undegraded":"2024-01-27T01:08:21.291156+0000","last_fullsized":"2024-01-27T01:08:21.291156+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:24.871704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.287969+0000","last_change":"2024-01-27T01:08:21.287969+0000","last_active":"2024-01-27T01:08:21.287969+0000","last_peered":"2024-01-27T01:08:21.287969+0000","last_clean":"2024-01-27T01:08:21.287969+0000","last_became_active":"2024-01-27T01:08:21.287531+0000","last_became_peered":"2024-01-27T01:08:21.287531+0000","last_unstale":"2024-01-27T01:08:21.287969+0000","last_undegraded":"2024-01-27T01:08:21.287969+0000","last_fullsized":"2024-01-27T01:08:21.287969+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:58.192628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"262'1","reported_seq":10,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.257183+0000","last_change":"2024-01-27T01:08:20.279122+0000","last_active":"2024-01-27T01:08:21.257183+0000","last_peered":"2024-01-27T01:08:21.257183+0000","last_clean":"2024-01-27T01:08:21.257183+0000","last_became_active":"2024-01-27T01:08:20.278897+0000","last_became_peered":"2024-01-27T01:08:20.278897+0000","last_unstale":"2024-01-27T01:08:21.257183+0000","last_undegraded":"2024-01-27T01:08:21.257183+0000","last_fullsized":"2024-01-27T01:08:21.257183+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:29:21.925629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.286633+0000","last_change":"2024-01-27T01:08:21.286633+0000","last_active":"2024-01-27T01:08:21.286633+0000","last_peered":"2024-01-27T01:08:21.286633+0000","last_clean":"2024-01-27T01:08:21.286633+0000","last_became_active":"2024-01-27T01:08:21.286350+0000","last_became_peered":"2024-01-27T01:08:21.286350+0000","last_unstale":"2024-01-27T01:08:21.286633+0000","last_undegraded":"2024-01-27T01:08:21.286633+0000","last_fullsized":"2024-01-27T01:08:21.286633+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:51:03.411287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"262'1","reported_seq":10,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.256457+0000","last_change":"2024-01-27T01:08:20.273820+0000","last_active":"2024-01-27T01:08:21.256457+0000","last_peered":"2024-01-27T01:08:21.256457+0000","last_clean":"2024-01-27T01:08:21.256457+0000","last_became_active":"2024-01-27T01:08:20.273530+0000","last_became_peered":"2024-01-27T01:08:20.273530+0000","last_unstale":"2024-01-27T01:08:21.256457+0000","last_undegraded":"2024-01-27T01:08:21.256457+0000","last_fullsized":"2024-01-27T01:08:21.256457+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:59:18.336759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.288786+0000","last_change":"2024-01-27T01:08:21.288786+0000","last_active":"2024-01-27T01:08:21.288786+0000","last_peered":"2024-01-27T01:08:21.288786+0000","last_clean":"2024-01-27T01:08:21.288786+0000","last_became_active":"2024-01-27T01:08:21.288491+0000","last_became_peered":"2024-01-27T01:08:21.288491+0000","last_unstale":"2024-01-27T01:08:21.288786+0000","last_undegraded":"2024-01-27T01:08:21.288786+0000","last_fullsized":"2024-01-27T01:08:21.288786+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:57:01.151572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.269369+0000","last_change":"2024-01-27T01:08:20.264894+0000","last_active":"2024-01-27T01:08:20.269369+0000","last_peered":"2024-01-27T01:08:20.269369+0000","last_clean":"2024-01-27T01:08:20.269369+0000","last_became_active":"2024-01-27T01:08:20.264621+0000","last_became_peered":"2024-01-27T01:08:20.264621+0000","last_unstale":"2024-01-27T01:08:20.269369+0000","last_undegraded":"2024-01-27T01:08:20.269369+0000","last_fullsized":"2024-01-27T01:08:20.269369+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:55:32.689927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.282182+0000","last_change":"2024-01-27T01:08:21.282182+0000","last_active":"2024-01-27T01:08:21.282182+0000","last_peered":"2024-01-27T01:08:21.282182+0000","last_clean":"2024-01-27T01:08:21.282182+0000","last_became_active":"2024-01-27T01:08:21.281906+0000","last_became_peered":"2024-01-27T01:08:21.281906+0000","last_unstale":"2024-01-27T01:08:21.282182+0000","last_undegraded":"2024-01-27T01:08:21.282182+0000","last_fullsized":"2024-01-27T01:08:21.282182+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:37.460673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.278673+0000","last_change":"2024-01-27T01:08:20.275547+0000","last_active":"2024-01-27T01:08:20.278673+0000","last_peered":"2024-01-27T01:08:20.278673+0000","last_clean":"2024-01-27T01:08:20.278673+0000","last_became_active":"2024-01-27T01:08:20.274939+0000","last_became_peered":"2024-01-27T01:08:20.274939+0000","last_unstale":"2024-01-27T01:08:20.278673+0000","last_undegraded":"2024-01-27T01:08:20.278673+0000","last_fullsized":"2024-01-27T01:08:20.278673+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:59:26.043654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.277745+0000","last_change":"2024-01-27T01:08:21.277745+0000","last_active":"2024-01-27T01:08:21.277745+0000","last_peered":"2024-01-27T01:08:21.277745+0000","last_clean":"2024-01-27T01:08:21.277745+0000","last_became_active":"2024-01-27T01:08:21.277241+0000","last_became_peered":"2024-01-27T01:08:21.277241+0000","last_unstale":"2024-01-27T01:08:21.277745+0000","last_undegraded":"2024-01-27T01:08:21.277745+0000","last_fullsized":"2024-01-27T01:08:21.277745+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:35:24.518760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.275441+0000","last_change":"2024-01-27T01:08:20.269170+0000","last_active":"2024-01-27T01:08:20.275441+0000","last_peered":"2024-01-27T01:08:20.275441+0000","last_clean":"2024-01-27T01:08:20.275441+0000","last_became_active":"2024-01-27T01:08:20.268931+0000","last_became_peered":"2024-01-27T01:08:20.268931+0000","last_unstale":"2024-01-27T01:08:20.275441+0000","last_undegraded":"2024-01-27T01:08:20.275441+0000","last_fullsized":"2024-01-27T01:08:20.275441+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:35:52.543816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.266510+0000","last_change":"2024-01-27T01:08:21.266510+0000","last_active":"2024-01-27T01:08:21.266510+0000","last_peered":"2024-01-27T01:08:21.266510+0000","last_clean":"2024-01-27T01:08:21.266510+0000","last_became_active":"2024-01-27T01:08:21.266275+0000","last_became_peered":"2024-01-27T01:08:21.266275+0000","last_unstale":"2024-01-27T01:08:21.266510+0000","last_undegraded":"2024-01-27T01:08:21.266510+0000","last_fullsized":"2024-01-27T01:08:21.266510+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:11:48.902241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.275490+0000","last_change":"2024-01-27T01:08:20.273129+0000","last_active":"2024-01-27T01:08:20.275490+0000","last_peered":"2024-01-27T01:08:20.275490+0000","last_clean":"2024-01-27T01:08:20.275490+0000","last_became_active":"2024-01-27T01:08:20.272882+0000","last_became_peered":"2024-01-27T01:08:20.272882+0000","last_unstale":"2024-01-27T01:08:20.275490+0000","last_undegraded":"2024-01-27T01:08:20.275490+0000","last_fullsized":"2024-01-27T01:08:20.275490+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:18.238223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.273804+0000","last_change":"2024-01-27T01:08:21.273804+0000","last_active":"2024-01-27T01:08:21.273804+0000","last_peered":"2024-01-27T01:08:21.273804+0000","last_clean":"2024-01-27T01:08:21.273804+0000","last_became_active":"2024-01-27T01:08:21.273451+0000","last_became_peered":"2024-01-27T01:08:21.273451+0000","last_unstale":"2024-01-27T01:08:21.273804+0000","last_undegraded":"2024-01-27T01:08:21.273804+0000","last_fullsized":"2024-01-27T01:08:21.273804+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:10:31.665980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"262'3","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.289274+0000","last_change":"2024-01-27T01:08:20.271233+0000","last_active":"2024-01-27T01:08:21.289274+0000","last_peered":"2024-01-27T01:08:21.289274+0000","last_clean":"2024-01-27T01:08:21.289274+0000","last_became_active":"2024-01-27T01:08:20.270986+0000","last_became_peered":"2024-01-27T01:08:20.270986+0000","last_unstale":"2024-01-27T01:08:21.289274+0000","last_undegraded":"2024-01-27T01:08:21.289274+0000","last_fullsized":"2024-01-27T01:08:21.289274+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:59:18.502011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.278752+0000","last_change":"2024-01-27T01:08:21.278752+0000","last_active":"2024-01-27T01:08:21.278752+0000","last_peered":"2024-01-27T01:08:21.278752+0000","last_clean":"2024-01-27T01:08:21.278752+0000","last_became_active":"2024-01-27T01:08:21.278441+0000","last_became_peered":"2024-01-27T01:08:21.278441+0000","last_unstale":"2024-01-27T01:08:21.278752+0000","last_undegraded":"2024-01-27T01:08:21.278752+0000","last_fullsized":"2024-01-27T01:08:21.278752+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:33:42.773892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"262'6","reported_seq":15,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.287362+0000","last_change":"2024-01-27T01:08:20.266857+0000","last_active":"2024-01-27T01:08:21.287362+0000","last_peered":"2024-01-27T01:08:21.287362+0000","last_clean":"2024-01-27T01:08:21.287362+0000","last_became_active":"2024-01-27T01:08:20.266595+0000","last_became_peered":"2024-01-27T01:08:20.266595+0000","last_unstale":"2024-01-27T01:08:21.287362+0000","last_undegraded":"2024-01-27T01:08:21.287362+0000","last_fullsized":"2024-01-27T01:08:21.287362+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:58.590323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.260864+0000","last_change":"2024-01-27T01:08:21.260864+0000","last_active":"2024-01-27T01:08:21.260864+0000","last_peered":"2024-01-27T01:08:21.260864+0000","last_clean":"2024-01-27T01:08:21.260864+0000","last_became_active":"2024-01-27T01:08:21.260534+0000","last_became_peered":"2024-01-27T01:08:21.260534+0000","last_unstale":"2024-01-27T01:08:21.260864+0000","last_undegraded":"2024-01-27T01:08:21.260864+0000","last_fullsized":"2024-01-27T01:08:21.260864+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:05:36.113124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.287841+0000","last_change":"2024-01-27T01:08:20.271637+0000","last_active":"2024-01-27T01:08:21.287841+0000","last_peered":"2024-01-27T01:08:21.287841+0000","last_clean":"2024-01-27T01:08:21.287841+0000","last_became_active":"2024-01-27T01:08:20.271391+0000","last_became_peered":"2024-01-27T01:08:20.271391+0000","last_unstale":"2024-01-27T01:08:21.287841+0000","last_undegraded":"2024-01-27T01:08:21.287841+0000","last_fullsized":"2024-01-27T01:08:21.287841+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:08:58.410726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.274144+0000","last_change":"2024-01-27T01:08:21.274144+0000","last_active":"2024-01-27T01:08:21.274144+0000","last_peered":"2024-01-27T01:08:21.274144+0000","last_clean":"2024-01-27T01:08:21.274144+0000","last_became_active":"2024-01-27T01:08:21.273628+0000","last_became_peered":"2024-01-27T01:08:21.273628+0000","last_unstale":"2024-01-27T01:08:21.274144+0000","last_undegraded":"2024-01-27T01:08:21.274144+0000","last_fullsized":"2024-01-27T01:08:21.274144+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:00:06.207428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.273174+0000","last_change":"2024-01-27T01:08:20.272261+0000","last_active":"2024-01-27T01:08:20.273174+0000","last_peered":"2024-01-27T01:08:20.273174+0000","last_clean":"2024-01-27T01:08:20.273174+0000","last_became_active":"2024-01-27T01:08:20.272017+0000","last_became_peered":"2024-01-27T01:08:20.272017+0000","last_unstale":"2024-01-27T01:08:20.273174+0000","last_undegraded":"2024-01-27T01:08:20.273174+0000","last_fullsized":"2024-01-27T01:08:20.273174+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:08:25.341916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.282334+0000","last_change":"2024-01-27T01:08:21.282334+0000","last_active":"2024-01-27T01:08:21.282334+0000","last_peered":"2024-01-27T01:08:21.282334+0000","last_clean":"2024-01-27T01:08:21.282334+0000","last_became_active":"2024-01-27T01:08:21.282102+0000","last_became_peered":"2024-01-27T01:08:21.282102+0000","last_unstale":"2024-01-27T01:08:21.282334+0000","last_undegraded":"2024-01-27T01:08:21.282334+0000","last_fullsized":"2024-01-27T01:08:21.282334+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:23:03.223553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.267869+0000","last_change":"2024-01-27T01:08:20.265907+0000","last_active":"2024-01-27T01:08:20.267869+0000","last_peered":"2024-01-27T01:08:20.267869+0000","last_clean":"2024-01-27T01:08:20.267869+0000","last_became_active":"2024-01-27T01:08:20.265654+0000","last_became_peered":"2024-01-27T01:08:20.265654+0000","last_unstale":"2024-01-27T01:08:20.267869+0000","last_undegraded":"2024-01-27T01:08:20.267869+0000","last_fullsized":"2024-01-27T01:08:20.267869+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:12:08.505429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.270384+0000","last_change":"2024-01-27T01:08:21.270384+0000","last_active":"2024-01-27T01:08:21.270384+0000","last_peered":"2024-01-27T01:08:21.270384+0000","last_clean":"2024-01-27T01:08:21.270384+0000","last_became_active":"2024-01-27T01:08:21.270082+0000","last_became_peered":"2024-01-27T01:08:21.270082+0000","last_unstale":"2024-01-27T01:08:21.270384+0000","last_undegraded":"2024-01-27T01:08:21.270384+0000","last_fullsized":"2024-01-27T01:08:21.270384+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:03:10.846435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"262'8","reported_seq":17,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.286865+0000","last_change":"2024-01-27T01:08:20.266827+0000","last_active":"2024-01-27T01:08:21.286865+0000","last_peered":"2024-01-27T01:08:21.286865+0000","last_clean":"2024-01-27T01:08:21.286865+0000","last_became_active":"2024-01-27T01:08:20.266540+0000","last_became_peered":"2024-01-27T01:08:20.266540+0000","last_unstale":"2024-01-27T01:08:21.286865+0000","last_undegraded":"2024-01-27T01:08:21.286865+0000","last_fullsized":"2024-01-27T01:08:21.286865+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:35:19.719786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.261153+0000","last_change":"2024-01-27T01:08:21.261153+0000","last_active":"2024-01-27T01:08:21.261153+0000","last_peered":"2024-01-27T01:08:21.261153+0000","last_clean":"2024-01-27T01:08:21.261153+0000","last_became_active":"2024-01-27T01:08:21.260202+0000","last_became_peered":"2024-01-27T01:08:21.260202+0000","last_unstale":"2024-01-27T01:08:21.261153+0000","last_undegraded":"2024-01-27T01:08:21.261153+0000","last_fullsized":"2024-01-27T01:08:21.261153+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:19:45.729439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"262'1","reported_seq":10,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.263305+0000","last_change":"2024-01-27T01:08:20.266837+0000","last_active":"2024-01-27T01:08:21.263305+0000","last_peered":"2024-01-27T01:08:21.263305+0000","last_clean":"2024-01-27T01:08:21.263305+0000","last_became_active":"2024-01-27T01:08:20.266460+0000","last_became_peered":"2024-01-27T01:08:20.266460+0000","last_unstale":"2024-01-27T01:08:21.263305+0000","last_undegraded":"2024-01-27T01:08:21.263305+0000","last_fullsized":"2024-01-27T01:08:21.263305+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:20:55.709983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.288079+0000","last_change":"2024-01-27T01:08:21.288079+0000","last_active":"2024-01-27T01:08:21.288079+0000","last_peered":"2024-01-27T01:08:21.288079+0000","last_clean":"2024-01-27T01:08:21.288079+0000","last_became_active":"2024-01-27T01:08:21.286007+0000","last_became_peered":"2024-01-27T01:08:21.286007+0000","last_unstale":"2024-01-27T01:08:21.288079+0000","last_undegraded":"2024-01-27T01:08:21.288079+0000","last_fullsized":"2024-01-27T01:08:21.288079+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:05:14.483844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.286686+0000","last_change":"2024-01-27T01:08:20.272835+0000","last_active":"2024-01-27T01:08:21.286686+0000","last_peered":"2024-01-27T01:08:21.286686+0000","last_clean":"2024-01-27T01:08:21.286686+0000","last_became_active":"2024-01-27T01:08:20.272533+0000","last_became_peered":"2024-01-27T01:08:20.272533+0000","last_unstale":"2024-01-27T01:08:21.286686+0000","last_undegraded":"2024-01-27T01:08:21.286686+0000","last_fullsized":"2024-01-27T01:08:21.286686+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:28:37.662641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.261100+0000","last_change":"2024-01-27T01:08:21.261100+0000","last_active":"2024-01-27T01:08:21.261100+0000","last_peered":"2024-01-27T01:08:21.261100+0000","last_clean":"2024-01-27T01:08:21.261100+0000","last_became_active":"2024-01-27T01:08:21.260780+0000","last_became_peered":"2024-01-27T01:08:21.260780+0000","last_unstale":"2024-01-27T01:08:21.261100+0000","last_undegraded":"2024-01-27T01:08:21.261100+0000","last_fullsized":"2024-01-27T01:08:21.261100+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:12:22.115718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"262'1","reported_seq":10,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.262384+0000","last_change":"2024-01-27T01:08:20.273675+0000","last_active":"2024-01-27T01:08:21.262384+0000","last_peered":"2024-01-27T01:08:21.262384+0000","last_clean":"2024-01-27T01:08:21.262384+0000","last_became_active":"2024-01-27T01:08:20.273330+0000","last_became_peered":"2024-01-27T01:08:20.273330+0000","last_unstale":"2024-01-27T01:08:21.262384+0000","last_undegraded":"2024-01-27T01:08:21.262384+0000","last_fullsized":"2024-01-27T01:08:21.262384+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:36:52.647293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.280321+0000","last_change":"2024-01-27T01:08:21.280321+0000","last_active":"2024-01-27T01:08:21.280321+0000","last_peered":"2024-01-27T01:08:21.280321+0000","last_clean":"2024-01-27T01:08:21.280321+0000","last_became_active":"2024-01-27T01:08:21.280058+0000","last_became_peered":"2024-01-27T01:08:21.280058+0000","last_unstale":"2024-01-27T01:08:21.280321+0000","last_undegraded":"2024-01-27T01:08:21.280321+0000","last_fullsized":"2024-01-27T01:08:21.280321+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:55:33.691451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.272767+0000","last_change":"2024-01-27T01:08:20.271089+0000","last_active":"2024-01-27T01:08:20.272767+0000","last_peered":"2024-01-27T01:08:20.272767+0000","last_clean":"2024-01-27T01:08:20.272767+0000","last_became_active":"2024-01-27T01:08:20.270860+0000","last_became_peered":"2024-01-27T01:08:20.270860+0000","last_unstale":"2024-01-27T01:08:20.272767+0000","last_undegraded":"2024-01-27T01:08:20.272767+0000","last_fullsized":"2024-01-27T01:08:20.272767+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:07:51.523892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.286060+0000","last_change":"2024-01-27T01:08:21.286060+0000","last_active":"2024-01-27T01:08:21.286060+0000","last_peered":"2024-01-27T01:08:21.286060+0000","last_clean":"2024-01-27T01:08:21.286060+0000","last_became_active":"2024-01-27T01:08:21.285769+0000","last_became_peered":"2024-01-27T01:08:21.285769+0000","last_unstale":"2024-01-27T01:08:21.286060+0000","last_undegraded":"2024-01-27T01:08:21.286060+0000","last_fullsized":"2024-01-27T01:08:21.286060+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:56:16.351337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.289176+0000","last_change":"2024-01-27T01:08:20.270955+0000","last_active":"2024-01-27T01:08:21.289176+0000","last_peered":"2024-01-27T01:08:21.289176+0000","last_clean":"2024-01-27T01:08:21.289176+0000","last_became_active":"2024-01-27T01:08:20.270707+0000","last_became_peered":"2024-01-27T01:08:20.270707+0000","last_unstale":"2024-01-27T01:08:21.289176+0000","last_undegraded":"2024-01-27T01:08:21.289176+0000","last_fullsized":"2024-01-27T01:08:21.289176+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:57.191103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.286642+0000","last_change":"2024-01-27T01:08:21.286642+0000","last_active":"2024-01-27T01:08:21.286642+0000","last_peered":"2024-01-27T01:08:21.286642+0000","last_clean":"2024-01-27T01:08:21.286642+0000","last_became_active":"2024-01-27T01:08:21.286160+0000","last_became_peered":"2024-01-27T01:08:21.286160+0000","last_unstale":"2024-01-27T01:08:21.286642+0000","last_undegraded":"2024-01-27T01:08:21.286642+0000","last_fullsized":"2024-01-27T01:08:21.286642+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:13:00.293473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.279719+0000","last_change":"2024-01-27T01:08:20.277598+0000","last_active":"2024-01-27T01:08:20.279719+0000","last_peered":"2024-01-27T01:08:20.279719+0000","last_clean":"2024-01-27T01:08:20.279719+0000","last_became_active":"2024-01-27T01:08:20.277279+0000","last_became_peered":"2024-01-27T01:08:20.277279+0000","last_unstale":"2024-01-27T01:08:20.279719+0000","last_undegraded":"2024-01-27T01:08:20.279719+0000","last_fullsized":"2024-01-27T01:08:20.279719+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:44:52.534165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.277018+0000","last_change":"2024-01-27T01:08:21.277018+0000","last_active":"2024-01-27T01:08:21.277018+0000","last_peered":"2024-01-27T01:08:21.277018+0000","last_clean":"2024-01-27T01:08:21.277018+0000","last_became_active":"2024-01-27T01:08:21.276764+0000","last_became_peered":"2024-01-27T01:08:21.276764+0000","last_unstale":"2024-01-27T01:08:21.277018+0000","last_undegraded":"2024-01-27T01:08:21.277018+0000","last_fullsized":"2024-01-27T01:08:21.277018+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:29:16.833734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.269195+0000","last_change":"2024-01-27T01:08:20.268816+0000","last_active":"2024-01-27T01:08:20.269195+0000","last_peered":"2024-01-27T01:08:20.269195+0000","last_clean":"2024-01-27T01:08:20.269195+0000","last_became_active":"2024-01-27T01:08:20.268420+0000","last_became_peered":"2024-01-27T01:08:20.268420+0000","last_unstale":"2024-01-27T01:08:20.269195+0000","last_undegraded":"2024-01-27T01:08:20.269195+0000","last_fullsized":"2024-01-27T01:08:20.269195+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:02:40.864757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.289379+0000","last_change":"2024-01-27T01:08:21.289379+0000","last_active":"2024-01-27T01:08:21.289379+0000","last_peered":"2024-01-27T01:08:21.289379+0000","last_clean":"2024-01-27T01:08:21.289379+0000","last_became_active":"2024-01-27T01:08:21.289016+0000","last_became_peered":"2024-01-27T01:08:21.289016+0000","last_unstale":"2024-01-27T01:08:21.289379+0000","last_undegraded":"2024-01-27T01:08:21.289379+0000","last_fullsized":"2024-01-27T01:08:21.289379+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:12:09.506953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.273287+0000","last_change":"2024-01-27T01:08:20.271769+0000","last_active":"2024-01-27T01:08:20.273287+0000","last_peered":"2024-01-27T01:08:20.273287+0000","last_clean":"2024-01-27T01:08:20.273287+0000","last_became_active":"2024-01-27T01:08:20.271541+0000","last_became_peered":"2024-01-27T01:08:20.271541+0000","last_unstale":"2024-01-27T01:08:20.273287+0000","last_undegraded":"2024-01-27T01:08:20.273287+0000","last_fullsized":"2024-01-27T01:08:20.273287+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:35:52.543816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.277535+0000","last_change":"2024-01-27T01:08:21.277535+0000","last_active":"2024-01-27T01:08:21.277535+0000","last_peered":"2024-01-27T01:08:21.277535+0000","last_clean":"2024-01-27T01:08:21.277535+0000","last_became_active":"2024-01-27T01:08:21.276783+0000","last_became_peered":"2024-01-27T01:08:21.276783+0000","last_unstale":"2024-01-27T01:08:21.277535+0000","last_undegraded":"2024-01-27T01:08:21.277535+0000","last_fullsized":"2024-01-27T01:08:21.277535+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:53:45.819320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.281135+0000","last_change":"2024-01-27T01:08:20.267924+0000","last_active":"2024-01-27T01:08:21.281135+0000","last_peered":"2024-01-27T01:08:21.281135+0000","last_clean":"2024-01-27T01:08:21.281135+0000","last_became_active":"2024-01-27T01:08:20.267642+0000","last_became_peered":"2024-01-27T01:08:20.267642+0000","last_unstale":"2024-01-27T01:08:21.281135+0000","last_undegraded":"2024-01-27T01:08:21.281135+0000","last_fullsized":"2024-01-27T01:08:21.281135+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:58:53.379588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.263670+0000","last_change":"2024-01-27T01:08:21.263670+0000","last_active":"2024-01-27T01:08:21.263670+0000","last_peered":"2024-01-27T01:08:21.263670+0000","last_clean":"2024-01-27T01:08:21.263670+0000","last_became_active":"2024-01-27T01:08:21.263437+0000","last_became_peered":"2024-01-27T01:08:21.263437+0000","last_unstale":"2024-01-27T01:08:21.263670+0000","last_undegraded":"2024-01-27T01:08:21.263670+0000","last_fullsized":"2024-01-27T01:08:21.263670+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:26:15.721480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.272953+0000","last_change":"2024-01-27T01:08:20.269421+0000","last_active":"2024-01-27T01:08:20.272953+0000","last_peered":"2024-01-27T01:08:20.272953+0000","last_clean":"2024-01-27T01:08:20.272953+0000","last_became_active":"2024-01-27T01:08:20.269182+0000","last_became_peered":"2024-01-27T01:08:20.269182+0000","last_unstale":"2024-01-27T01:08:20.272953+0000","last_undegraded":"2024-01-27T01:08:20.272953+0000","last_fullsized":"2024-01-27T01:08:20.272953+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:50:14.733412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.280968+0000","last_change":"2024-01-27T01:08:21.280968+0000","last_active":"2024-01-27T01:08:21.280968+0000","last_peered":"2024-01-27T01:08:21.280968+0000","last_clean":"2024-01-27T01:08:21.280968+0000","last_became_active":"2024-01-27T01:08:21.280700+0000","last_became_peered":"2024-01-27T01:08:21.280700+0000","last_unstale":"2024-01-27T01:08:21.280968+0000","last_undegraded":"2024-01-27T01:08:21.280968+0000","last_fullsized":"2024-01-27T01:08:21.280968+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:50:25.220364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.272618+0000","last_change":"2024-01-27T01:08:20.267520+0000","last_active":"2024-01-27T01:08:20.272618+0000","last_peered":"2024-01-27T01:08:20.272618+0000","last_clean":"2024-01-27T01:08:20.272618+0000","last_became_active":"2024-01-27T01:08:20.267185+0000","last_became_peered":"2024-01-27T01:08:20.267185+0000","last_unstale":"2024-01-27T01:08:20.272618+0000","last_undegraded":"2024-01-27T01:08:20.272618+0000","last_fullsized":"2024-01-27T01:08:20.272618+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:06:42.199152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.282327+0000","last_change":"2024-01-27T01:08:21.282327+0000","last_active":"2024-01-27T01:08:21.282327+0000","last_peered":"2024-01-27T01:08:21.282327+0000","last_clean":"2024-01-27T01:08:21.282327+0000","last_became_active":"2024-01-27T01:08:21.282083+0000","last_became_peered":"2024-01-27T01:08:21.282083+0000","last_unstale":"2024-01-27T01:08:21.282327+0000","last_undegraded":"2024-01-27T01:08:21.282327+0000","last_fullsized":"2024-01-27T01:08:21.282327+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:29:17.826415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.278079+0000","last_change":"2024-01-27T01:08:20.273913+0000","last_active":"2024-01-27T01:08:20.278079+0000","last_peered":"2024-01-27T01:08:20.278079+0000","last_clean":"2024-01-27T01:08:20.278079+0000","last_became_active":"2024-01-27T01:08:20.273631+0000","last_became_peered":"2024-01-27T01:08:20.273631+0000","last_unstale":"2024-01-27T01:08:20.278079+0000","last_undegraded":"2024-01-27T01:08:20.278079+0000","last_fullsized":"2024-01-27T01:08:20.278079+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:57.706964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.282599+0000","last_change":"2024-01-27T01:08:21.282599+0000","last_active":"2024-01-27T01:08:21.282599+0000","last_peered":"2024-01-27T01:08:21.282599+0000","last_clean":"2024-01-27T01:08:21.282599+0000","last_became_active":"2024-01-27T01:08:21.282342+0000","last_became_peered":"2024-01-27T01:08:21.282342+0000","last_unstale":"2024-01-27T01:08:21.282599+0000","last_undegraded":"2024-01-27T01:08:21.282599+0000","last_fullsized":"2024-01-27T01:08:21.282599+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:30:36.572083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.280783+0000","last_change":"2024-01-27T01:08:20.276248+0000","last_active":"2024-01-27T01:08:20.280783+0000","last_peered":"2024-01-27T01:08:20.280783+0000","last_clean":"2024-01-27T01:08:20.280783+0000","last_became_active":"2024-01-27T01:08:20.276033+0000","last_became_peered":"2024-01-27T01:08:20.276033+0000","last_unstale":"2024-01-27T01:08:20.280783+0000","last_undegraded":"2024-01-27T01:08:20.280783+0000","last_fullsized":"2024-01-27T01:08:20.280783+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:54:13.119628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.273912+0000","last_change":"2024-01-27T01:08:21.273912+0000","last_active":"2024-01-27T01:08:21.273912+0000","last_peered":"2024-01-27T01:08:21.273912+0000","last_clean":"2024-01-27T01:08:21.273912+0000","last_became_active":"2024-01-27T01:08:21.273230+0000","last_became_peered":"2024-01-27T01:08:21.273230+0000","last_unstale":"2024-01-27T01:08:21.273912+0000","last_undegraded":"2024-01-27T01:08:21.273912+0000","last_fullsized":"2024-01-27T01:08:21.273912+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:22:21.341852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.280914+0000","last_change":"2024-01-27T01:08:20.279070+0000","last_active":"2024-01-27T01:08:20.280914+0000","last_peered":"2024-01-27T01:08:20.280914+0000","last_clean":"2024-01-27T01:08:20.280914+0000","last_became_active":"2024-01-27T01:08:20.277648+0000","last_became_peered":"2024-01-27T01:08:20.277648+0000","last_unstale":"2024-01-27T01:08:20.280914+0000","last_undegraded":"2024-01-27T01:08:20.280914+0000","last_fullsized":"2024-01-27T01:08:20.280914+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:10:15.436962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.272841+0000","last_change":"2024-01-27T01:08:21.272841+0000","last_active":"2024-01-27T01:08:21.272841+0000","last_peered":"2024-01-27T01:08:21.272841+0000","last_clean":"2024-01-27T01:08:21.272841+0000","last_became_active":"2024-01-27T01:08:21.272505+0000","last_became_peered":"2024-01-27T01:08:21.272505+0000","last_unstale":"2024-01-27T01:08:21.272841+0000","last_undegraded":"2024-01-27T01:08:21.272841+0000","last_fullsized":"2024-01-27T01:08:21.272841+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:29:22.927153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.274303+0000","last_change":"2024-01-27T01:08:20.268435+0000","last_active":"2024-01-27T01:08:20.274303+0000","last_peered":"2024-01-27T01:08:20.274303+0000","last_clean":"2024-01-27T01:08:20.274303+0000","last_became_active":"2024-01-27T01:08:20.268162+0000","last_became_peered":"2024-01-27T01:08:20.268162+0000","last_unstale":"2024-01-27T01:08:20.274303+0000","last_undegraded":"2024-01-27T01:08:20.274303+0000","last_fullsized":"2024-01-27T01:08:20.274303+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:54:13.119628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.290660+0000","last_change":"2024-01-27T01:08:21.290660+0000","last_active":"2024-01-27T01:08:21.290660+0000","last_peered":"2024-01-27T01:08:21.290660+0000","last_clean":"2024-01-27T01:08:21.290660+0000","last_became_active":"2024-01-27T01:08:21.290351+0000","last_became_peered":"2024-01-27T01:08:21.290351+0000","last_unstale":"2024-01-27T01:08:21.290660+0000","last_undegraded":"2024-01-27T01:08:21.290660+0000","last_fullsized":"2024-01-27T01:08:21.290660+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:58:54.381113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.282904+0000","last_change":"2024-01-27T01:08:20.282544+0000","last_active":"2024-01-27T01:08:20.282904+0000","last_peered":"2024-01-27T01:08:20.282904+0000","last_clean":"2024-01-27T01:08:20.282904+0000","last_became_active":"2024-01-27T01:08:20.282312+0000","last_became_peered":"2024-01-27T01:08:20.282312+0000","last_unstale":"2024-01-27T01:08:20.282904+0000","last_undegraded":"2024-01-27T01:08:20.282904+0000","last_fullsized":"2024-01-27T01:08:20.282904+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:18.083351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.269615+0000","last_change":"2024-01-27T01:08:21.269615+0000","last_active":"2024-01-27T01:08:21.269615+0000","last_peered":"2024-01-27T01:08:21.269615+0000","last_clean":"2024-01-27T01:08:21.269615+0000","last_became_active":"2024-01-27T01:08:21.269364+0000","last_became_peered":"2024-01-27T01:08:21.269364+0000","last_unstale":"2024-01-27T01:08:21.269615+0000","last_undegraded":"2024-01-27T01:08:21.269615+0000","last_fullsized":"2024-01-27T01:08:21.269615+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:00.967147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"262'10745","reported_seq":11784,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:22.075592+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:08:22.075592+0000","last_peered":"2024-01-27T01:08:22.075592+0000","last_clean":"2024-01-27T01:08:22.075592+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:08:22.075592+0000","last_undegraded":"2024-01-27T01:08:22.075592+0000","last_fullsized":"2024-01-27T01:08:22.075592+0000","mapping_epoch":14,"log_start":"98'2229","ondisk_log_start":"98'2229","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8516,"log_dups_size":0,"ondisk_log_size":8516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2528,"num_read_kb":5347,"num_write":680,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2528,"num_read_kb":5347,"num_write":680,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8516,"ondisk_log_size":8516,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576022,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309288,"kb_used_data":3808,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062552,"statfs":{"total":96636764160,"available":96320053248,"internally_reserved":0,"allocated":3899392,"data_stored":2348297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.58199999999999996,"15min":0.60199999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.312},"max":{"1min":0.745,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61699999999999999,"15min":0.61799999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33000000000000002},"max":{"1min":0.76400000000000001,"5min":0.879,"15min":0.879},"last":0.35199999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.58599999999999997,"15min":0.60399999999999998},"min":{"1min":0.40300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73799999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.34300000000000003},"max":{"1min":0.81499999999999995,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.40200000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.59499999999999997,"15min":0.60399999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.753,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.39200000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85799999999999998,"5min":0.872,"15min":0.88900000000000001},"last":0.42799999999999999}]},{"osd":3,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58799999999999997,"15min":0.60799999999999998},"min":{"1min":0.374,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79100000000000004,"5min":0.85399999999999998,"15min":0.876},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64300000000000002,"15min":0.64000000000000001},"min":{"1min":0.40600000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.80600000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.48099999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.496,"15min":0.48699999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76100000000000001,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.52600000000000002,"15min":0.51500000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67800000000000005,"5min":0.80300000000000005,"15min":0.86599999999999999},"last":0.32100000000000001}]},{"osd":5,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.54100000000000004,"15min":0.52400000000000002},"min":{"1min":0.33600000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.81399999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.377}]},{"osd":6,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59299999999999997,"15min":0.58399999999999996},"min":{"1min":0.33700000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84499999999999997,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58799999999999997,"15min":0.58199999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77900000000000003,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.36099999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576022,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1757929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.59199999999999997,"15min":0.59899999999999998},"min":{"1min":0.40999999999999998,"5min":0.40799999999999997,"15min":0.36399999999999999},"max":{"1min":0.65400000000000003,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60199999999999998,"15min":0.61199999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.376},"max":{"1min":0.67600000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.41399999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.59199999999999997,"15min":0.60299999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.65600000000000003,"5min":0.80300000000000005,"15min":0.84499999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60199999999999998,"15min":0.61699999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81200000000000006,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.42199999999999999}]},{"osd":2,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.58299999999999996,"15min":0.61099999999999999},"min":{"1min":0.39200000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70999999999999996,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.436},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69999999999999996,"5min":0.80400000000000005,"15min":0.878},"last":0.40500000000000003}]},{"osd":3,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61399999999999999,"15min":0.623},"min":{"1min":0.379,"5min":0.378,"15min":0.378},"max":{"1min":0.72599999999999998,"5min":0.83699999999999997,"15min":0.90100000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64500000000000002,"15min":0.65400000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.754,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.38600000000000001}]},{"osd":4,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51300000000000001,"15min":0.502},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.71399999999999997,"5min":0.77600000000000002,"15min":0.86599999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.36599999999999999,"5min":0.33800000000000002,"15min":0.29799999999999999},"max":{"1min":0.69299999999999995,"5min":0.78500000000000003,"15min":0.88900000000000001},"last":0.311}]},{"osd":5,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.69199999999999995,"5min":0.75900000000000001,"15min":0.79300000000000004},"last":0.372},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.58299999999999996,"15min":0.57399999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.28899999999999998},"max":{"1min":0.84199999999999997,"5min":0.87,"15min":0.874},"last":0.36199999999999999}]},{"osd":7,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57799999999999996,"15min":0.57399999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82099999999999995,"5min":0.84899999999999998,"15min":0.92900000000000005},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58999999999999997,"15min":0.58899999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.34699999999999998},"max":{"1min":0.76500000000000001,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.27900000000000003}]}]},{"osd":1,"up_from":13,"seq":55834576022,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304628,"kb_used_data":3244,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067212,"statfs":{"total":96636764160,"available":96324825088,"internally_reserved":0,"allocated":3321856,"data_stored":1765594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.28299999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.502,"5min":0.66100000000000003,"15min":0.69599999999999995},"last":0.502},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.432,"15min":0.45100000000000001},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.76600000000000001},"last":0.33700000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.40799999999999997,"15min":0.432},"min":{"1min":0.29699999999999999,"5min":0.28599999999999998,"15min":0.26200000000000001},"max":{"1min":0.47299999999999998,"5min":0.748,"15min":0.95799999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46899999999999997,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.57199999999999995,"5min":0.67400000000000004,"15min":0.79000000000000004},"last":0.35199999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.501,"15min":0.52600000000000002},"min":{"1min":0.35999999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59099999999999997,"5min":0.73699999999999999,"15min":1.2130000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.48599999999999999,"15min":0.51100000000000001},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.32300000000000001},"max":{"1min":0.628,"5min":0.71799999999999997,"15min":1.22},"last":0.57999999999999996}]},{"osd":4,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.65200000000000002,"15min":0.66600000000000004},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78600000000000003,"5min":0.998,"15min":1.242},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.66700000000000004,"15min":0.68899999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.35699999999999998},"max":{"1min":0.93600000000000005,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.66100000000000003}]},{"osd":5,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.70999999999999996,"15min":0.70299999999999996},"min":{"1min":0.42299999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81299999999999994,"5min":5.444,"15min":5.444},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.69299999999999995,"15min":0.69999999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82899999999999996,"5min":2.6669999999999998,"15min":2.6669999999999998},"last":0.53000000000000003}]},{"osd":6,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.72199999999999998,"15min":0.71899999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47099999999999997},"max":{"1min":0.85499999999999998,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.79100000000000004,"15min":0.73899999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48499999999999999},"max":{"1min":0.82199999999999995,"5min":11.223000000000001,"15min":11.223000000000001},"last":0.56699999999999995}]},{"osd":7,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.72799999999999998,"15min":0.748},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97799999999999998,"5min":2.9660000000000002,"15min":2.9660000000000002},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.751,"15min":0.76300000000000001},"min":{"1min":0.55400000000000005,"5min":0.53400000000000003,"15min":0.52000000000000002},"max":{"1min":0.83199999999999996,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.747}]}]},{"osd":0,"up_from":13,"seq":55834576021,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309308,"kb_used_data":3828,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062532,"statfs":{"total":96636764160,"available":96320032768,"internally_reserved":0,"allocated":3919872,"data_stored":2356460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.434,"15min":0.42499999999999999},"min":{"1min":0.28899999999999998,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.45100000000000001,"15min":0.46000000000000002},"min":{"1min":0.33200000000000002,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":0.64300000000000002,"5min":0.65600000000000003,"15min":0.73399999999999999},"last":0.72399999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.41899999999999998,"15min":0.433},"min":{"1min":0.27400000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.56499999999999995,"5min":0.66900000000000004,"15min":0.76800000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.47599999999999998,"15min":0.48599999999999999},"min":{"1min":0.32900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60299999999999998,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.5}]},{"osd":3,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.34599999999999997,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.83699999999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.503,"15min":0.504},"min":{"1min":0.35899999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.65900000000000003,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.63}]},{"osd":4,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.67600000000000005,"15min":0.67200000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.42199999999999999},"max":{"1min":0.91200000000000003,"5min":0.91500000000000004,"15min":1.0720000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.65900000000000003,"15min":0.67600000000000005},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.90100000000000002,"5min":1.002,"15min":1.002},"last":0.55900000000000005}]},{"osd":5,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.68700000000000006,"15min":0.69799999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95799999999999996,"5min":1.0209999999999999,"15min":2.165},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69499999999999995,"15min":0.70999999999999996},"min":{"1min":0.50700000000000001,"5min":0.48799999999999999,"15min":0.45600000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.1399999999999999},"last":0.59599999999999997}]},{"osd":6,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68799999999999994,"15min":0.70999999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.71399999999999997,"15min":0.73699999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.88700000000000001,"5min":1.0109999999999999,"15min":1.425},"last":0.67300000000000004}]},{"osd":7,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.73099999999999998,"15min":0.752},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.99199999999999999,"5min":1.0169999999999999,"15min":1.1259999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.755,"15min":0.76600000000000001},"min":{"1min":0.55000000000000004,"5min":0.51100000000000001,"15min":0.48499999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.1910000000000001},"last":0.70799999999999996}]}]},{"osd":2,"up_from":13,"seq":55834576021,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312816,"kb_used_data":3240,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059024,"statfs":{"total":96636764160,"available":96316440576,"internally_reserved":0,"allocated":3317760,"data_stored":1765504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38500000000000001,"15min":0.40200000000000002},"min":{"1min":0.29299999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.56200000000000006,"5min":0.60199999999999998,"15min":0.746},"last":0.44},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.42299999999999999,"15min":0.44},"min":{"1min":0.30399999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.48599999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.36899999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.435,"15min":0.443},"min":{"1min":0.316,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":0.52400000000000002,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.497,"15min":0.51600000000000001},"min":{"1min":0.36799999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.65000000000000002,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.47799999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.47899999999999998,"15min":0.49099999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58599999999999997,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.46200000000000002,"15min":0.48099999999999998},"min":{"1min":0.30499999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56699999999999995,"5min":0.63900000000000001,"15min":0.68999999999999995},"last":0.66100000000000003}]},{"osd":4,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.65300000000000002,"15min":0.66700000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89700000000000002,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.66200000000000003,"15min":0.68200000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.378},"max":{"1min":0.77200000000000002,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.59499999999999997}]},{"osd":5,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.68000000000000005,"15min":0.69499999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.876,"5min":1.075,"15min":1.075},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.67700000000000005,"15min":0.69799999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.39800000000000002},"max":{"1min":0.79600000000000004,"5min":0.97499999999999998,"15min":1.0229999999999999},"last":0.60199999999999998}]},{"osd":6,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.68000000000000005,"15min":0.70199999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.73699999999999999,"5min":0.97899999999999998,"15min":1.0549999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67300000000000004,"15min":0.70599999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.39200000000000002},"max":{"1min":0.83199999999999996,"5min":0.91200000000000003,"15min":1.0269999999999999},"last":0.79700000000000004}]},{"osd":7,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.71199999999999997,"15min":0.73599999999999999},"min":{"1min":0.48899999999999999,"5min":0.442,"15min":0.442},"max":{"1min":0.77600000000000002,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.748},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.72499999999999998,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86099999999999999,"5min":0.94599999999999995,"15min":1.0880000000000001},"last":0.78000000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576020,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304632,"kb_used_data":3248,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067208,"statfs":{"total":96636764160,"available":96324820992,"internally_reserved":0,"allocated":3325952,"data_stored":1765596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.439,"15min":0.42199999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63400000000000001,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44,"15min":0.45400000000000001},"min":{"1min":0.30599999999999999,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.66400000000000003,"5min":0.66700000000000004,"15min":0.84699999999999998},"last":0.496}]},{"osd":1,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.61699999999999999,"15min":0.53200000000000003},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.28899999999999998},"max":{"1min":0.76100000000000001,"5min":15.391,"15min":15.391},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.52400000000000002,"15min":0.52300000000000002},"min":{"1min":0.35799999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.628,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.57099999999999995}]},{"osd":2,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.53600000000000003,"15min":0.51300000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70299999999999996,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.50900000000000001},"min":{"1min":0.308,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67600000000000005,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":0.59099999999999997}]},{"osd":4,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.72899999999999998,"15min":0.71499999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.439},"max":{"1min":0.89200000000000002,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.71499999999999997,"15min":0.70299999999999996},"min":{"1min":0.47499999999999998,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":0.96699999999999997,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.624}]},{"osd":5,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.73099999999999998,"15min":0.70899999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.94499999999999995,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.72699999999999998,"15min":0.71499999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.435},"max":{"1min":0.872,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.67300000000000004}]},{"osd":6,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.73799999999999999,"15min":0.71999999999999997},"min":{"1min":0.49199999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.96699999999999997,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92900000000000005,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.70299999999999996}]},{"osd":7,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.745,"15min":0.73199999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.91400000000000003,"5min":4.569,"15min":4.569},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.78900000000000003,"15min":0.77800000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97999999999999998,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.76800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834576022,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316928,"kb_used_data":3256,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94054912,"statfs":{"total":96636764160,"available":96312229888,"internally_reserved":0,"allocated":3334144,"data_stored":1767130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68500000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59699999999999998,"15min":0.60899999999999999},"min":{"1min":0.35499999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.85699999999999998},"last":0.39100000000000001}]},{"osd":1,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.59899999999999998,"15min":0.622},"min":{"1min":0.41899999999999998,"5min":0.39100000000000001,"15min":0.38800000000000001},"max":{"1min":0.80700000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60999999999999999,"15min":0.62},"min":{"1min":0.38800000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79100000000000004,"5min":0.874,"15min":0.874},"last":0.42999999999999999}]},{"osd":2,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.34200000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80500000000000005,"5min":0.85599999999999998,"15min":0.88700000000000001},"last":0.37},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.626,"15min":0.63300000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.80300000000000005,"5min":0.81499999999999995,"15min":0.89200000000000002},"last":0.41899999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59699999999999998,"15min":0.61699999999999999},"min":{"1min":0.377,"5min":0.373,"15min":0.373},"max":{"1min":0.747,"5min":0.83299999999999996,"15min":0.84199999999999997},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61099999999999999,"15min":0.625},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.39900000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.49299999999999999,"15min":0.48599999999999999},"min":{"1min":0.35599999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.85899999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.30399999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.878},"last":0.317}]},{"osd":6,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.50800000000000001,"15min":0.5},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72599999999999998,"5min":0.82499999999999996,"15min":0.88700000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35199999999999998,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.79400000000000004,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.34399999999999997}]},{"osd":7,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.57199999999999995,"15min":0.56499999999999995},"min":{"1min":0.33800000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.90200000000000002},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.42199999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.76800000000000002,"5min":0.85399999999999998,"15min":0.92600000000000005},"last":0.33400000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576022,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312824,"kb_used_data":3248,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059016,"statfs":{"total":96636764160,"available":96316432384,"internally_reserved":0,"allocated":3325952,"data_stored":1765560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.58799999999999997,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.27300000000000002},"max":{"1min":0.73999999999999999,"5min":0.82799999999999996,"15min":1.006},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.57099999999999995,"15min":0.59799999999999998},"min":{"1min":0.37,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73499999999999999,"5min":0.81499999999999995,"15min":1.1719999999999999},"last":0.51300000000000001}]},{"osd":1,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.61599999999999999,"15min":0.64300000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69699999999999995,"5min":0.89000000000000001,"15min":0.95899999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.65500000000000003,"15min":0.66400000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.377},"max":{"1min":0.77900000000000003,"5min":0.86899999999999999,"15min":1.212},"last":0.63100000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56200000000000006,"15min":0.58699999999999997},"min":{"1min":0.34699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70999999999999996,"5min":0.82099999999999995,"15min":1.631},"last":0.62},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.60299999999999998,"15min":0.61299999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34000000000000002},"max":{"1min":0.67600000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.505}]},{"osd":3,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.373,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71099999999999997,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.61199999999999999,"15min":0.624},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.754,"5min":0.879,"15min":1.4690000000000001},"last":0.65100000000000002}]},{"osd":4,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.5,"15min":0.48699999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69199999999999995,"5min":0.90200000000000002,"15min":0.94399999999999995},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.52300000000000002,"15min":0.51200000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66100000000000003,"5min":0.92500000000000004,"15min":0.93300000000000005},"last":0.316}]},{"osd":6,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51100000000000001,"15min":0.504},"min":{"1min":0.312,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.629,"5min":0.96799999999999997,"15min":1.732},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56399999999999995,"15min":0.54700000000000004},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78300000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.32400000000000001}]},{"osd":7,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59099999999999997,"15min":0.57499999999999996},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.79700000000000004,"5min":0.95399999999999996,"15min":1.0740000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.621,"15min":0.61299999999999999},"min":{"1min":0.41299999999999998,"5min":0.40799999999999997,"15min":0.39900000000000002},"max":{"1min":0.76500000000000001,"5min":0.996,"15min":1.6919999999999999},"last":0.38300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:08:25.485 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:08:25.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.663+0000 7f840de0a700 1 Processor -- start 2024-01-27T01:08:25.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.663+0000 7f840de0a700 1 -- start start 2024-01-27T01:08:25.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.664+0000 7f840de0a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8408157340 0x7f840815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:25.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.664+0000 7f840de0a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84080ae520 0x7f84080b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:25.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.664+0000 7f840de0a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f84080b3fb0 0x7f8408157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:25.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.664+0000 7f840de0a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84080b4430 con 0x7f8408157340 2024-01-27T01:08:25.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.664+0000 7f840de0a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8408158ae0 con 0x7f84080b3fb0 2024-01-27T01:08:25.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.664+0000 7f840de0a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f840815b650 con 0x7f84080ae520 2024-01-27T01:08:25.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.664+0000 7f84077fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8408157340 0x7f840815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:25.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.664+0000 7f84077fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8408157340 0x7f840815bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39044/0 (socket says 172.21.15.46:39044) 2024-01-27T01:08:25.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.664+0000 7f84077fe700 1 -- 172.21.15.46:0/413851004 learned_addr learned my addr 172.21.15.46:0/413851004 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:25.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.664+0000 7f8407fff700 1 --2- 172.21.15.46:0/413851004 >> v2:172.21.15.106:3300/0 conn(0x7f84080b3fb0 0x7f8408157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:25.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.664+0000 7f8406ffd700 1 --2- 172.21.15.46:0/413851004 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84080ae520 0x7f84080b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:25.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.664+0000 7f84077fe700 1 -- 172.21.15.46:0/413851004 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84080ae520 msgr2=0x7f84080b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:25.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.664+0000 7f84077fe700 1 --2- 172.21.15.46:0/413851004 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84080ae520 0x7f84080b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.664+0000 7f84077fe700 1 -- 172.21.15.46:0/413851004 >> v2:172.21.15.106:3300/0 conn(0x7f84080b3fb0 msgr2=0x7f8408157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:25.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.664+0000 7f84077fe700 1 --2- 172.21.15.46:0/413851004 >> v2:172.21.15.106:3300/0 conn(0x7f84080b3fb0 0x7f8408157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.665+0000 7f84077fe700 1 -- 172.21.15.46:0/413851004 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840815c020 con 0x7f8408157340 2024-01-27T01:08:25.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.665+0000 7f84077fe700 1 --2- 172.21.15.46:0/413851004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8408157340 0x7f840815bae0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f84080caee0 tx=0x7f83f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f29e0cf082529a05 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:25.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.665+0000 7f84067fc700 1 -- 172.21.15.46:0/413851004 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f800b640 con 0x7f8408157340 2024-01-27T01:08:25.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.665+0000 7f84067fc700 1 -- 172.21.15.46:0/413851004 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f800b7a0 con 0x7f8408157340 2024-01-27T01:08:25.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.665+0000 7f84067fc700 1 -- 172.21.15.46:0/413851004 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f800bb40 con 0x7f8408157340 2024-01-27T01:08:25.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.665+0000 7f840de0a700 1 -- 172.21.15.46:0/413851004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8408157340 msgr2=0x7f840815bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:25.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.665+0000 7f840de0a700 1 --2- 172.21.15.46:0/413851004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8408157340 0x7f840815bae0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f84080caee0 tx=0x7f83f8009250 comp rx=0 tx=0).stop 2024-01-27T01:08:25.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.666+0000 7f840de0a700 1 -- 172.21.15.46:0/413851004 shutdown_connections 2024-01-27T01:08:25.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.666+0000 7f840de0a700 1 --2- 172.21.15.46:0/413851004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8408157340 0x7f840815bae0 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.666+0000 7f840de0a700 1 --2- 172.21.15.46:0/413851004 >> v2:172.21.15.106:3300/0 conn(0x7f84080b3fb0 0x7f8408157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.666+0000 7f840de0a700 1 --2- 172.21.15.46:0/413851004 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84080ae520 0x7f84080b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.666+0000 7f840de0a700 1 -- 172.21.15.46:0/413851004 >> 172.21.15.46:0/413851004 conn(0x7f84080ce530 msgr2=0x7f84080ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:25.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.666+0000 7f840de0a700 1 -- 172.21.15.46:0/413851004 shutdown_connections 2024-01-27T01:08:25.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.666+0000 7f840de0a700 1 -- 172.21.15.46:0/413851004 wait complete. 2024-01-27T01:08:25.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.667+0000 7f840de0a700 1 Processor -- start 2024-01-27T01:08:25.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.667+0000 7f840de0a700 1 -- start start 2024-01-27T01:08:25.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.668+0000 7f840de0a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84080ae520 0x7f84080b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:25.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.668+0000 7f840de0a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84080b3fb0 0x7f84081569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.668+0000 7f840de0a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8408157340 0x7f84080c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.668+0000 7f840de0a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84080b1c10 con 0x7f8408157340 2024-01-27T01:08:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.668+0000 7f840de0a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8408156f10 con 0x7f84080ae520 2024-01-27T01:08:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.668+0000 7f840de0a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84080c7760 con 0x7f84080b3fb0 2024-01-27T01:08:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.668+0000 7f8407fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8408157340 0x7f84080c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.668+0000 7f8407fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8408157340 0x7f84080c6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39048/0 (socket says 172.21.15.46:39048) 2024-01-27T01:08:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.668+0000 7f8407fff700 1 -- 172.21.15.46:0/4094176375 learned_addr learned my addr 172.21.15.46:0/4094176375 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.668+0000 7f84077fe700 1 --2- 172.21.15.46:0/4094176375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84080ae520 0x7f84080b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.668+0000 7f8406ffd700 1 --2- 172.21.15.46:0/4094176375 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84080b3fb0 0x7f84081569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.668+0000 7f8407fff700 1 -- 172.21.15.46:0/4094176375 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84080b3fb0 msgr2=0x7f84081569d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.668+0000 7f8407fff700 1 --2- 172.21.15.46:0/4094176375 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84080b3fb0 0x7f84081569d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.668+0000 7f8407fff700 1 -- 172.21.15.46:0/4094176375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84080ae520 msgr2=0x7f84080b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.668+0000 7f8407fff700 1 --2- 172.21.15.46:0/4094176375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84080ae520 0x7f84080b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.668+0000 7f8407fff700 1 -- 172.21.15.46:0/4094176375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f80094f0 con 0x7f8408157340 2024-01-27T01:08:25.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.668+0000 7f8406ffd700 1 --2- 172.21.15.46:0/4094176375 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84080b3fb0 0x7f84081569d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:25.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.669+0000 7f8407fff700 1 --2- 172.21.15.46:0/4094176375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8408157340 0x7f84080c6d80 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f8400009e90 tx=0x7f840000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:25.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.669+0000 7f83ff7fe700 1 -- 172.21.15.46:0/4094176375 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840000cc20 con 0x7f8408157340 2024-01-27T01:08:25.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.669+0000 7f83ff7fe700 1 -- 172.21.15.46:0/4094176375 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f840000f040 con 0x7f8408157340 2024-01-27T01:08:25.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.669+0000 7f840de0a700 1 -- 172.21.15.46:0/4094176375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84080b12e0 con 0x7f8408157340 2024-01-27T01:08:25.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.669+0000 7f83ff7fe700 1 -- 172.21.15.46:0/4094176375 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8400014480 con 0x7f8408157340 2024-01-27T01:08:25.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.669+0000 7f840de0a700 1 -- 172.21.15.46:0/4094176375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84080b18a0 con 0x7f8408157340 2024-01-27T01:08:25.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.670+0000 7f840de0a700 1 -- 172.21.15.46:0/4094176375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f840804c710 con 0x7f8408157340 2024-01-27T01:08:25.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.671+0000 7f83ff7fe700 1 -- 172.21.15.46:0/4094176375 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f8400014830 con 0x7f8408157340 2024-01-27T01:08:25.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.671+0000 7f83ff7fe700 1 --2- 172.21.15.46:0/4094176375 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f83e80670e0 0x7f83e8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:25.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.671+0000 7f83ff7fe700 1 -- 172.21.15.46:0/4094176375 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f84000861d0 con 0x7f8408157340 2024-01-27T01:08:25.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.671+0000 7f84077fe700 1 --2- 172.21.15.46:0/4094176375 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f83e80670e0 0x7f83e8069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:25.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.672+0000 7f84077fe700 1 --2- 172.21.15.46:0/4094176375 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f83e80670e0 0x7f83e8069590 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f84080caee0 tx=0x7f83f8017000 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:25.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.675+0000 7f83ff7fe700 1 -- 172.21.15.46:0/4094176375 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8400056890 con 0x7f8408157340 2024-01-27T01:08:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.876+0000 7f840de0a700 1 -- 172.21.15.46:0/4094176375 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f84080b1d50 con 0x7f83e80670e0 2024-01-27T01:08:25.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.884+0000 7f83ff7fe700 1 -- 172.21.15.46:0/4094176375 <== mgr.11614 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290112 (secure 0 0 0) 0x7f84080b1d50 con 0x7f83e80670e0 2024-01-27T01:08:25.887 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:25.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.887+0000 7f840de0a700 1 -- 172.21.15.46:0/4094176375 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f83e80670e0 msgr2=0x7f83e8069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:25.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.887+0000 7f840de0a700 1 --2- 172.21.15.46:0/4094176375 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f83e80670e0 0x7f83e8069590 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f84080caee0 tx=0x7f83f8017000 comp rx=0 tx=0).stop 2024-01-27T01:08:25.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.887+0000 7f840de0a700 1 -- 172.21.15.46:0/4094176375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8408157340 msgr2=0x7f84080c6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:25.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.887+0000 7f840de0a700 1 --2- 172.21.15.46:0/4094176375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8408157340 0x7f84080c6d80 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f8400009e90 tx=0x7f840000b040 comp rx=0 tx=0).stop 2024-01-27T01:08:25.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.888+0000 7f840de0a700 1 -- 172.21.15.46:0/4094176375 shutdown_connections 2024-01-27T01:08:25.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.888+0000 7f840de0a700 1 --2- 172.21.15.46:0/4094176375 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f83e80670e0 0x7f83e8069590 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.888+0000 7f840de0a700 1 --2- 172.21.15.46:0/4094176375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8408157340 0x7f84080c6d80 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.888+0000 7f840de0a700 1 --2- 172.21.15.46:0/4094176375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84080ae520 0x7f84080b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.888+0000 7f840de0a700 1 --2- 172.21.15.46:0/4094176375 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f84080b3fb0 0x7f84081569d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:25.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.888+0000 7f840de0a700 1 -- 172.21.15.46:0/4094176375 >> 172.21.15.46:0/4094176375 conn(0x7f84080ce530 msgr2=0x7f840806d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:25.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.888+0000 7f840de0a700 1 -- 172.21.15.46:0/4094176375 shutdown_connections 2024-01-27T01:08:25.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:25.888+0000 7f840de0a700 1 -- 172.21.15.46:0/4094176375 wait complete. 2024-01-27T01:08:25.890 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:08:25.900 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-01-27T01:08:25.093393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2528,"num_read_kb":5347,"num_write":724,"num_write_kb":3475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8554,"ondisk_log_size":8554,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":72,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2483228,"kb_used_data":27100,"kb_used_omap":0,"kb_used_meta":2456064,"kb_avail":752491492,"statfs":{"total":773094113280,"available":770551287808,"internally_reserved":0,"allocated":27750400,"data_stored":15292070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2515009536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.838356"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.276679+0000","last_change":"2024-01-27T01:08:21.276679+0000","last_active":"2024-01-27T01:08:21.276679+0000","last_peered":"2024-01-27T01:08:21.276679+0000","last_clean":"2024-01-27T01:08:21.276679+0000","last_became_active":"2024-01-27T01:08:21.276413+0000","last_became_peered":"2024-01-27T01:08:21.276413+0000","last_unstale":"2024-01-27T01:08:21.276679+0000","last_undegraded":"2024-01-27T01:08:21.276679+0000","last_fullsized":"2024-01-27T01:08:21.276679+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:16:52.898998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.286323+0000","last_change":"2024-01-27T01:08:21.286323+0000","last_active":"2024-01-27T01:08:21.286323+0000","last_peered":"2024-01-27T01:08:21.286323+0000","last_clean":"2024-01-27T01:08:21.286323+0000","last_became_active":"2024-01-27T01:08:21.286073+0000","last_became_peered":"2024-01-27T01:08:21.286073+0000","last_unstale":"2024-01-27T01:08:21.286323+0000","last_undegraded":"2024-01-27T01:08:21.286323+0000","last_fullsized":"2024-01-27T01:08:21.286323+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:48:57.898191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.288314+0000","last_change":"2024-01-27T01:08:21.288314+0000","last_active":"2024-01-27T01:08:21.288314+0000","last_peered":"2024-01-27T01:08:21.288314+0000","last_clean":"2024-01-27T01:08:21.288314+0000","last_became_active":"2024-01-27T01:08:21.288027+0000","last_became_peered":"2024-01-27T01:08:21.288027+0000","last_unstale":"2024-01-27T01:08:21.288314+0000","last_undegraded":"2024-01-27T01:08:21.288314+0000","last_fullsized":"2024-01-27T01:08:21.288314+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:08:59.412251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.272715+0000","last_change":"2024-01-27T01:08:21.272715+0000","last_active":"2024-01-27T01:08:21.272715+0000","last_peered":"2024-01-27T01:08:21.272715+0000","last_clean":"2024-01-27T01:08:21.272715+0000","last_became_active":"2024-01-27T01:08:21.272411+0000","last_became_peered":"2024-01-27T01:08:21.272411+0000","last_unstale":"2024-01-27T01:08:21.272715+0000","last_undegraded":"2024-01-27T01:08:21.272715+0000","last_fullsized":"2024-01-27T01:08:21.272715+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:07:52.525416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.261376+0000","last_change":"2024-01-27T01:08:21.261376+0000","last_active":"2024-01-27T01:08:21.261376+0000","last_peered":"2024-01-27T01:08:21.261376+0000","last_clean":"2024-01-27T01:08:21.261376+0000","last_became_active":"2024-01-27T01:08:21.261073+0000","last_became_peered":"2024-01-27T01:08:21.261073+0000","last_unstale":"2024-01-27T01:08:21.261376+0000","last_undegraded":"2024-01-27T01:08:21.261376+0000","last_fullsized":"2024-01-27T01:08:21.261376+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:47.554300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.284818+0000","last_change":"2024-01-27T01:08:21.284818+0000","last_active":"2024-01-27T01:08:21.284818+0000","last_peered":"2024-01-27T01:08:21.284818+0000","last_clean":"2024-01-27T01:08:21.284818+0000","last_became_active":"2024-01-27T01:08:21.284500+0000","last_became_peered":"2024-01-27T01:08:21.284500+0000","last_unstale":"2024-01-27T01:08:21.284818+0000","last_undegraded":"2024-01-27T01:08:21.284818+0000","last_fullsized":"2024-01-27T01:08:21.284818+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:58.708489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.274959+0000","last_change":"2024-01-27T01:08:21.274959+0000","last_active":"2024-01-27T01:08:21.274959+0000","last_peered":"2024-01-27T01:08:21.274959+0000","last_clean":"2024-01-27T01:08:21.274959+0000","last_became_active":"2024-01-27T01:08:21.274713+0000","last_became_peered":"2024-01-27T01:08:21.274713+0000","last_unstale":"2024-01-27T01:08:21.274959+0000","last_undegraded":"2024-01-27T01:08:21.274959+0000","last_fullsized":"2024-01-27T01:08:21.274959+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:49:53.469375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.278371+0000","last_change":"2024-01-27T01:08:21.278371+0000","last_active":"2024-01-27T01:08:21.278371+0000","last_peered":"2024-01-27T01:08:21.278371+0000","last_clean":"2024-01-27T01:08:21.278371+0000","last_became_active":"2024-01-27T01:08:21.278126+0000","last_became_peered":"2024-01-27T01:08:21.278126+0000","last_unstale":"2024-01-27T01:08:21.278371+0000","last_undegraded":"2024-01-27T01:08:21.278371+0000","last_fullsized":"2024-01-27T01:08:21.278371+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:39:38.359802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.289308+0000","last_change":"2024-01-27T01:08:21.289308+0000","last_active":"2024-01-27T01:08:21.289308+0000","last_peered":"2024-01-27T01:08:21.289308+0000","last_clean":"2024-01-27T01:08:21.289308+0000","last_became_active":"2024-01-27T01:08:21.289059+0000","last_became_peered":"2024-01-27T01:08:21.289059+0000","last_unstale":"2024-01-27T01:08:21.289308+0000","last_undegraded":"2024-01-27T01:08:21.289308+0000","last_fullsized":"2024-01-27T01:08:21.289308+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:30:08.847499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.270204+0000","last_change":"2024-01-27T01:08:21.270204+0000","last_active":"2024-01-27T01:08:21.270204+0000","last_peered":"2024-01-27T01:08:21.270204+0000","last_clean":"2024-01-27T01:08:21.270204+0000","last_became_active":"2024-01-27T01:08:21.269959+0000","last_became_peered":"2024-01-27T01:08:21.269959+0000","last_unstale":"2024-01-27T01:08:21.270204+0000","last_undegraded":"2024-01-27T01:08:21.270204+0000","last_fullsized":"2024-01-27T01:08:21.270204+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:03.783822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.276079+0000","last_change":"2024-01-27T01:08:21.276079+0000","last_active":"2024-01-27T01:08:21.276079+0000","last_peered":"2024-01-27T01:08:21.276079+0000","last_clean":"2024-01-27T01:08:21.276079+0000","last_became_active":"2024-01-27T01:08:21.275642+0000","last_became_peered":"2024-01-27T01:08:21.275642+0000","last_unstale":"2024-01-27T01:08:21.276079+0000","last_undegraded":"2024-01-27T01:08:21.276079+0000","last_fullsized":"2024-01-27T01:08:21.276079+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:18:01.913878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.276250+0000","last_change":"2024-01-27T01:08:21.276250+0000","last_active":"2024-01-27T01:08:21.276250+0000","last_peered":"2024-01-27T01:08:21.276250+0000","last_clean":"2024-01-27T01:08:21.276250+0000","last_became_active":"2024-01-27T01:08:21.276000+0000","last_became_peered":"2024-01-27T01:08:21.276000+0000","last_unstale":"2024-01-27T01:08:21.276250+0000","last_undegraded":"2024-01-27T01:08:21.276250+0000","last_fullsized":"2024-01-27T01:08:21.276250+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:52:12.651940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.277100+0000","last_change":"2024-01-27T01:08:21.277100+0000","last_active":"2024-01-27T01:08:21.277100+0000","last_peered":"2024-01-27T01:08:21.277100+0000","last_clean":"2024-01-27T01:08:21.277100+0000","last_became_active":"2024-01-27T01:08:21.276785+0000","last_became_peered":"2024-01-27T01:08:21.276785+0000","last_unstale":"2024-01-27T01:08:21.277100+0000","last_undegraded":"2024-01-27T01:08:21.277100+0000","last_fullsized":"2024-01-27T01:08:21.277100+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:54:14.121153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.269660+0000","last_change":"2024-01-27T01:08:21.269660+0000","last_active":"2024-01-27T01:08:21.269660+0000","last_peered":"2024-01-27T01:08:21.269660+0000","last_clean":"2024-01-27T01:08:21.269660+0000","last_became_active":"2024-01-27T01:08:21.269401+0000","last_became_peered":"2024-01-27T01:08:21.269401+0000","last_unstale":"2024-01-27T01:08:21.269660+0000","last_undegraded":"2024-01-27T01:08:21.269660+0000","last_fullsized":"2024-01-27T01:08:21.269660+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:32:20.263478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.278476+0000","last_change":"2024-01-27T01:08:21.278476+0000","last_active":"2024-01-27T01:08:21.278476+0000","last_peered":"2024-01-27T01:08:21.278476+0000","last_clean":"2024-01-27T01:08:21.278476+0000","last_became_active":"2024-01-27T01:08:21.278177+0000","last_became_peered":"2024-01-27T01:08:21.278177+0000","last_unstale":"2024-01-27T01:08:21.278476+0000","last_undegraded":"2024-01-27T01:08:21.278476+0000","last_fullsized":"2024-01-27T01:08:21.278476+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:22:44.405022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.270324+0000","last_change":"2024-01-27T01:08:21.270324+0000","last_active":"2024-01-27T01:08:21.270324+0000","last_peered":"2024-01-27T01:08:21.270324+0000","last_clean":"2024-01-27T01:08:21.270324+0000","last_became_active":"2024-01-27T01:08:21.270120+0000","last_became_peered":"2024-01-27T01:08:21.270120+0000","last_unstale":"2024-01-27T01:08:21.270324+0000","last_undegraded":"2024-01-27T01:08:21.270324+0000","last_fullsized":"2024-01-27T01:08:21.270324+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:21:05.876052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.282682+0000","last_change":"2024-01-27T01:08:21.282682+0000","last_active":"2024-01-27T01:08:21.282682+0000","last_peered":"2024-01-27T01:08:21.282682+0000","last_clean":"2024-01-27T01:08:21.282682+0000","last_became_active":"2024-01-27T01:08:21.282458+0000","last_became_peered":"2024-01-27T01:08:21.282458+0000","last_unstale":"2024-01-27T01:08:21.282682+0000","last_undegraded":"2024-01-27T01:08:21.282682+0000","last_fullsized":"2024-01-27T01:08:21.282682+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:41:09.975158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.294910+0000","last_change":"2024-01-27T01:08:21.294910+0000","last_active":"2024-01-27T01:08:21.294910+0000","last_peered":"2024-01-27T01:08:21.294910+0000","last_clean":"2024-01-27T01:08:21.294910+0000","last_became_active":"2024-01-27T01:08:21.294637+0000","last_became_peered":"2024-01-27T01:08:21.294637+0000","last_unstale":"2024-01-27T01:08:21.294910+0000","last_undegraded":"2024-01-27T01:08:21.294910+0000","last_fullsized":"2024-01-27T01:08:21.294910+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:19.239747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.274066+0000","last_change":"2024-01-27T01:08:21.274066+0000","last_active":"2024-01-27T01:08:21.274066+0000","last_peered":"2024-01-27T01:08:21.274066+0000","last_clean":"2024-01-27T01:08:21.274066+0000","last_became_active":"2024-01-27T01:08:21.273831+0000","last_became_peered":"2024-01-27T01:08:21.273831+0000","last_unstale":"2024-01-27T01:08:21.274066+0000","last_undegraded":"2024-01-27T01:08:21.274066+0000","last_fullsized":"2024-01-27T01:08:21.274066+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:49:55.152011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.275813+0000","last_change":"2024-01-27T01:08:21.275813+0000","last_active":"2024-01-27T01:08:21.275813+0000","last_peered":"2024-01-27T01:08:21.275813+0000","last_clean":"2024-01-27T01:08:21.275813+0000","last_became_active":"2024-01-27T01:08:21.275526+0000","last_became_peered":"2024-01-27T01:08:21.275526+0000","last_unstale":"2024-01-27T01:08:21.275813+0000","last_undegraded":"2024-01-27T01:08:21.275813+0000","last_fullsized":"2024-01-27T01:08:21.275813+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:37:04.315404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.290800+0000","last_change":"2024-01-27T01:08:21.290800+0000","last_active":"2024-01-27T01:08:21.290800+0000","last_peered":"2024-01-27T01:08:21.290800+0000","last_clean":"2024-01-27T01:08:21.290800+0000","last_became_active":"2024-01-27T01:08:21.290562+0000","last_became_peered":"2024-01-27T01:08:21.290562+0000","last_unstale":"2024-01-27T01:08:21.290800+0000","last_undegraded":"2024-01-27T01:08:21.290800+0000","last_fullsized":"2024-01-27T01:08:21.290800+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:02:41.866282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.284607+0000","last_change":"2024-01-27T01:08:21.284607+0000","last_active":"2024-01-27T01:08:21.284607+0000","last_peered":"2024-01-27T01:08:21.284607+0000","last_clean":"2024-01-27T01:08:21.284607+0000","last_became_active":"2024-01-27T01:08:21.284279+0000","last_became_peered":"2024-01-27T01:08:21.284279+0000","last_unstale":"2024-01-27T01:08:21.284607+0000","last_undegraded":"2024-01-27T01:08:21.284607+0000","last_fullsized":"2024-01-27T01:08:21.284607+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:38:00.882525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.282751+0000","last_change":"2024-01-27T01:08:21.282751+0000","last_active":"2024-01-27T01:08:21.282751+0000","last_peered":"2024-01-27T01:08:21.282751+0000","last_clean":"2024-01-27T01:08:21.282751+0000","last_became_active":"2024-01-27T01:08:21.282483+0000","last_became_peered":"2024-01-27T01:08:21.282483+0000","last_unstale":"2024-01-27T01:08:21.282751+0000","last_undegraded":"2024-01-27T01:08:21.282751+0000","last_fullsized":"2024-01-27T01:08:21.282751+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:59:19.338283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.286437+0000","last_change":"2024-01-27T01:08:21.286437+0000","last_active":"2024-01-27T01:08:21.286437+0000","last_peered":"2024-01-27T01:08:21.286437+0000","last_clean":"2024-01-27T01:08:21.286437+0000","last_became_active":"2024-01-27T01:08:21.285799+0000","last_became_peered":"2024-01-27T01:08:21.285799+0000","last_unstale":"2024-01-27T01:08:21.286437+0000","last_undegraded":"2024-01-27T01:08:21.286437+0000","last_fullsized":"2024-01-27T01:08:21.286437+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:20:40.823053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.260813+0000","last_change":"2024-01-27T01:08:21.260813+0000","last_active":"2024-01-27T01:08:21.260813+0000","last_peered":"2024-01-27T01:08:21.260813+0000","last_clean":"2024-01-27T01:08:21.260813+0000","last_became_active":"2024-01-27T01:08:21.260543+0000","last_became_peered":"2024-01-27T01:08:21.260543+0000","last_unstale":"2024-01-27T01:08:21.260813+0000","last_undegraded":"2024-01-27T01:08:21.260813+0000","last_fullsized":"2024-01-27T01:08:21.260813+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:49:57.871691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.263075+0000","last_change":"2024-01-27T01:08:21.263075+0000","last_active":"2024-01-27T01:08:21.263075+0000","last_peered":"2024-01-27T01:08:21.263075+0000","last_clean":"2024-01-27T01:08:21.263075+0000","last_became_active":"2024-01-27T01:08:21.262800+0000","last_became_peered":"2024-01-27T01:08:21.262800+0000","last_unstale":"2024-01-27T01:08:21.263075+0000","last_undegraded":"2024-01-27T01:08:21.263075+0000","last_fullsized":"2024-01-27T01:08:21.263075+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:02:12.652310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.272775+0000","last_change":"2024-01-27T01:08:21.272775+0000","last_active":"2024-01-27T01:08:21.272775+0000","last_peered":"2024-01-27T01:08:21.272775+0000","last_clean":"2024-01-27T01:08:21.272775+0000","last_became_active":"2024-01-27T01:08:21.272170+0000","last_became_peered":"2024-01-27T01:08:21.272170+0000","last_unstale":"2024-01-27T01:08:21.272775+0000","last_undegraded":"2024-01-27T01:08:21.272775+0000","last_fullsized":"2024-01-27T01:08:21.272775+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:24:55.133025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.287706+0000","last_change":"2024-01-27T01:08:21.287706+0000","last_active":"2024-01-27T01:08:21.287706+0000","last_peered":"2024-01-27T01:08:21.287706+0000","last_clean":"2024-01-27T01:08:21.287706+0000","last_became_active":"2024-01-27T01:08:21.287381+0000","last_became_peered":"2024-01-27T01:08:21.287381+0000","last_unstale":"2024-01-27T01:08:21.287706+0000","last_undegraded":"2024-01-27T01:08:21.287706+0000","last_fullsized":"2024-01-27T01:08:21.287706+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:48:26.555763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.270701+0000","last_change":"2024-01-27T01:08:21.270701+0000","last_active":"2024-01-27T01:08:21.270701+0000","last_peered":"2024-01-27T01:08:21.270701+0000","last_clean":"2024-01-27T01:08:21.270701+0000","last_became_active":"2024-01-27T01:08:21.270483+0000","last_became_peered":"2024-01-27T01:08:21.270483+0000","last_unstale":"2024-01-27T01:08:21.270701+0000","last_undegraded":"2024-01-27T01:08:21.270701+0000","last_fullsized":"2024-01-27T01:08:21.270701+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:26:15.721480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.287910+0000","last_change":"2024-01-27T01:08:21.287910+0000","last_active":"2024-01-27T01:08:21.287910+0000","last_peered":"2024-01-27T01:08:21.287910+0000","last_clean":"2024-01-27T01:08:21.287910+0000","last_became_active":"2024-01-27T01:08:21.286874+0000","last_became_peered":"2024-01-27T01:08:21.286874+0000","last_unstale":"2024-01-27T01:08:21.287910+0000","last_undegraded":"2024-01-27T01:08:21.287910+0000","last_fullsized":"2024-01-27T01:08:21.287910+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:46:54.670266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.284650+0000","last_change":"2024-01-27T01:08:21.284650+0000","last_active":"2024-01-27T01:08:21.284650+0000","last_peered":"2024-01-27T01:08:21.284650+0000","last_clean":"2024-01-27T01:08:21.284650+0000","last_became_active":"2024-01-27T01:08:21.284356+0000","last_became_peered":"2024-01-27T01:08:21.284356+0000","last_unstale":"2024-01-27T01:08:21.284650+0000","last_undegraded":"2024-01-27T01:08:21.284650+0000","last_fullsized":"2024-01-27T01:08:21.284650+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:29:07.694697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.287297+0000","last_change":"2024-01-27T01:08:21.287297+0000","last_active":"2024-01-27T01:08:21.287297+0000","last_peered":"2024-01-27T01:08:21.287297+0000","last_clean":"2024-01-27T01:08:21.287297+0000","last_became_active":"2024-01-27T01:08:21.286118+0000","last_became_peered":"2024-01-27T01:08:21.286118+0000","last_unstale":"2024-01-27T01:08:21.287297+0000","last_undegraded":"2024-01-27T01:08:21.287297+0000","last_fullsized":"2024-01-27T01:08:21.287297+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:59:30.434717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"262'4","reported_seq":13,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.294426+0000","last_change":"2024-01-27T01:08:20.270085+0000","last_active":"2024-01-27T01:08:21.294426+0000","last_peered":"2024-01-27T01:08:21.294426+0000","last_clean":"2024-01-27T01:08:21.294426+0000","last_became_active":"2024-01-27T01:08:20.269801+0000","last_became_peered":"2024-01-27T01:08:20.269801+0000","last_unstale":"2024-01-27T01:08:21.294426+0000","last_undegraded":"2024-01-27T01:08:21.294426+0000","last_fullsized":"2024-01-27T01:08:21.294426+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:03:09.844911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.270768+0000","last_change":"2024-01-27T01:08:21.270768+0000","last_active":"2024-01-27T01:08:21.270768+0000","last_peered":"2024-01-27T01:08:21.270768+0000","last_clean":"2024-01-27T01:08:21.270768+0000","last_became_active":"2024-01-27T01:08:21.270557+0000","last_became_peered":"2024-01-27T01:08:21.270557+0000","last_unstale":"2024-01-27T01:08:21.270768+0000","last_undegraded":"2024-01-27T01:08:21.270768+0000","last_fullsized":"2024-01-27T01:08:21.270768+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:21:11.579156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"262'1","reported_seq":10,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.262539+0000","last_change":"2024-01-27T01:08:20.271567+0000","last_active":"2024-01-27T01:08:21.262539+0000","last_peered":"2024-01-27T01:08:21.262539+0000","last_clean":"2024-01-27T01:08:21.262539+0000","last_became_active":"2024-01-27T01:08:20.271259+0000","last_became_peered":"2024-01-27T01:08:20.271259+0000","last_unstale":"2024-01-27T01:08:21.262539+0000","last_undegraded":"2024-01-27T01:08:21.262539+0000","last_fullsized":"2024-01-27T01:08:21.262539+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:18:00.912354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.270146+0000","last_change":"2024-01-27T01:08:21.270146+0000","last_active":"2024-01-27T01:08:21.270146+0000","last_peered":"2024-01-27T01:08:21.270146+0000","last_clean":"2024-01-27T01:08:21.270146+0000","last_became_active":"2024-01-27T01:08:21.269888+0000","last_became_peered":"2024-01-27T01:08:21.269888+0000","last_unstale":"2024-01-27T01:08:21.270146+0000","last_undegraded":"2024-01-27T01:08:21.270146+0000","last_fullsized":"2024-01-27T01:08:21.270146+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:46:36.752556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.290660+0000","last_change":"2024-01-27T01:08:20.267577+0000","last_active":"2024-01-27T01:08:21.290660+0000","last_peered":"2024-01-27T01:08:21.290660+0000","last_clean":"2024-01-27T01:08:21.290660+0000","last_became_active":"2024-01-27T01:08:20.267292+0000","last_became_peered":"2024-01-27T01:08:20.267292+0000","last_unstale":"2024-01-27T01:08:21.290660+0000","last_undegraded":"2024-01-27T01:08:21.290660+0000","last_fullsized":"2024-01-27T01:08:21.290660+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:33:09.128990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.288860+0000","last_change":"2024-01-27T01:08:21.288860+0000","last_active":"2024-01-27T01:08:21.288860+0000","last_peered":"2024-01-27T01:08:21.288860+0000","last_clean":"2024-01-27T01:08:21.288860+0000","last_became_active":"2024-01-27T01:08:21.288591+0000","last_became_peered":"2024-01-27T01:08:21.288591+0000","last_unstale":"2024-01-27T01:08:21.288860+0000","last_undegraded":"2024-01-27T01:08:21.288860+0000","last_fullsized":"2024-01-27T01:08:21.288860+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:08:26.343440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.291156+0000","last_change":"2024-01-27T01:08:20.268588+0000","last_active":"2024-01-27T01:08:21.291156+0000","last_peered":"2024-01-27T01:08:21.291156+0000","last_clean":"2024-01-27T01:08:21.291156+0000","last_became_active":"2024-01-27T01:08:20.268304+0000","last_became_peered":"2024-01-27T01:08:20.268304+0000","last_unstale":"2024-01-27T01:08:21.291156+0000","last_undegraded":"2024-01-27T01:08:21.291156+0000","last_fullsized":"2024-01-27T01:08:21.291156+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:34:24.871704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.287969+0000","last_change":"2024-01-27T01:08:21.287969+0000","last_active":"2024-01-27T01:08:21.287969+0000","last_peered":"2024-01-27T01:08:21.287969+0000","last_clean":"2024-01-27T01:08:21.287969+0000","last_became_active":"2024-01-27T01:08:21.287531+0000","last_became_peered":"2024-01-27T01:08:21.287531+0000","last_unstale":"2024-01-27T01:08:21.287969+0000","last_undegraded":"2024-01-27T01:08:21.287969+0000","last_fullsized":"2024-01-27T01:08:21.287969+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:58.192628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"262'1","reported_seq":10,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.257183+0000","last_change":"2024-01-27T01:08:20.279122+0000","last_active":"2024-01-27T01:08:21.257183+0000","last_peered":"2024-01-27T01:08:21.257183+0000","last_clean":"2024-01-27T01:08:21.257183+0000","last_became_active":"2024-01-27T01:08:20.278897+0000","last_became_peered":"2024-01-27T01:08:20.278897+0000","last_unstale":"2024-01-27T01:08:21.257183+0000","last_undegraded":"2024-01-27T01:08:21.257183+0000","last_fullsized":"2024-01-27T01:08:21.257183+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:29:21.925629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.286633+0000","last_change":"2024-01-27T01:08:21.286633+0000","last_active":"2024-01-27T01:08:21.286633+0000","last_peered":"2024-01-27T01:08:21.286633+0000","last_clean":"2024-01-27T01:08:21.286633+0000","last_became_active":"2024-01-27T01:08:21.286350+0000","last_became_peered":"2024-01-27T01:08:21.286350+0000","last_unstale":"2024-01-27T01:08:21.286633+0000","last_undegraded":"2024-01-27T01:08:21.286633+0000","last_fullsized":"2024-01-27T01:08:21.286633+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:51:03.411287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"262'1","reported_seq":10,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.256457+0000","last_change":"2024-01-27T01:08:20.273820+0000","last_active":"2024-01-27T01:08:21.256457+0000","last_peered":"2024-01-27T01:08:21.256457+0000","last_clean":"2024-01-27T01:08:21.256457+0000","last_became_active":"2024-01-27T01:08:20.273530+0000","last_became_peered":"2024-01-27T01:08:20.273530+0000","last_unstale":"2024-01-27T01:08:21.256457+0000","last_undegraded":"2024-01-27T01:08:21.256457+0000","last_fullsized":"2024-01-27T01:08:21.256457+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:59:18.336759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.288786+0000","last_change":"2024-01-27T01:08:21.288786+0000","last_active":"2024-01-27T01:08:21.288786+0000","last_peered":"2024-01-27T01:08:21.288786+0000","last_clean":"2024-01-27T01:08:21.288786+0000","last_became_active":"2024-01-27T01:08:21.288491+0000","last_became_peered":"2024-01-27T01:08:21.288491+0000","last_unstale":"2024-01-27T01:08:21.288786+0000","last_undegraded":"2024-01-27T01:08:21.288786+0000","last_fullsized":"2024-01-27T01:08:21.288786+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:57:01.151572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.269369+0000","last_change":"2024-01-27T01:08:20.264894+0000","last_active":"2024-01-27T01:08:20.269369+0000","last_peered":"2024-01-27T01:08:20.269369+0000","last_clean":"2024-01-27T01:08:20.269369+0000","last_became_active":"2024-01-27T01:08:20.264621+0000","last_became_peered":"2024-01-27T01:08:20.264621+0000","last_unstale":"2024-01-27T01:08:20.269369+0000","last_undegraded":"2024-01-27T01:08:20.269369+0000","last_fullsized":"2024-01-27T01:08:20.269369+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:55:32.689927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.282182+0000","last_change":"2024-01-27T01:08:21.282182+0000","last_active":"2024-01-27T01:08:21.282182+0000","last_peered":"2024-01-27T01:08:21.282182+0000","last_clean":"2024-01-27T01:08:21.282182+0000","last_became_active":"2024-01-27T01:08:21.281906+0000","last_became_peered":"2024-01-27T01:08:21.281906+0000","last_unstale":"2024-01-27T01:08:21.282182+0000","last_undegraded":"2024-01-27T01:08:21.282182+0000","last_fullsized":"2024-01-27T01:08:21.282182+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:11:37.460673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.278673+0000","last_change":"2024-01-27T01:08:20.275547+0000","last_active":"2024-01-27T01:08:20.278673+0000","last_peered":"2024-01-27T01:08:20.278673+0000","last_clean":"2024-01-27T01:08:20.278673+0000","last_became_active":"2024-01-27T01:08:20.274939+0000","last_became_peered":"2024-01-27T01:08:20.274939+0000","last_unstale":"2024-01-27T01:08:20.278673+0000","last_undegraded":"2024-01-27T01:08:20.278673+0000","last_fullsized":"2024-01-27T01:08:20.278673+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:59:26.043654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.277745+0000","last_change":"2024-01-27T01:08:21.277745+0000","last_active":"2024-01-27T01:08:21.277745+0000","last_peered":"2024-01-27T01:08:21.277745+0000","last_clean":"2024-01-27T01:08:21.277745+0000","last_became_active":"2024-01-27T01:08:21.277241+0000","last_became_peered":"2024-01-27T01:08:21.277241+0000","last_unstale":"2024-01-27T01:08:21.277745+0000","last_undegraded":"2024-01-27T01:08:21.277745+0000","last_fullsized":"2024-01-27T01:08:21.277745+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:35:24.518760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.275441+0000","last_change":"2024-01-27T01:08:20.269170+0000","last_active":"2024-01-27T01:08:20.275441+0000","last_peered":"2024-01-27T01:08:20.275441+0000","last_clean":"2024-01-27T01:08:20.275441+0000","last_became_active":"2024-01-27T01:08:20.268931+0000","last_became_peered":"2024-01-27T01:08:20.268931+0000","last_unstale":"2024-01-27T01:08:20.275441+0000","last_undegraded":"2024-01-27T01:08:20.275441+0000","last_fullsized":"2024-01-27T01:08:20.275441+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:35:52.543816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.266510+0000","last_change":"2024-01-27T01:08:21.266510+0000","last_active":"2024-01-27T01:08:21.266510+0000","last_peered":"2024-01-27T01:08:21.266510+0000","last_clean":"2024-01-27T01:08:21.266510+0000","last_became_active":"2024-01-27T01:08:21.266275+0000","last_became_peered":"2024-01-27T01:08:21.266275+0000","last_unstale":"2024-01-27T01:08:21.266510+0000","last_undegraded":"2024-01-27T01:08:21.266510+0000","last_fullsized":"2024-01-27T01:08:21.266510+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:11:48.902241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.275490+0000","last_change":"2024-01-27T01:08:20.273129+0000","last_active":"2024-01-27T01:08:20.275490+0000","last_peered":"2024-01-27T01:08:20.275490+0000","last_clean":"2024-01-27T01:08:20.275490+0000","last_became_active":"2024-01-27T01:08:20.272882+0000","last_became_peered":"2024-01-27T01:08:20.272882+0000","last_unstale":"2024-01-27T01:08:20.275490+0000","last_undegraded":"2024-01-27T01:08:20.275490+0000","last_fullsized":"2024-01-27T01:08:20.275490+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:31:18.238223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.273804+0000","last_change":"2024-01-27T01:08:21.273804+0000","last_active":"2024-01-27T01:08:21.273804+0000","last_peered":"2024-01-27T01:08:21.273804+0000","last_clean":"2024-01-27T01:08:21.273804+0000","last_became_active":"2024-01-27T01:08:21.273451+0000","last_became_peered":"2024-01-27T01:08:21.273451+0000","last_unstale":"2024-01-27T01:08:21.273804+0000","last_undegraded":"2024-01-27T01:08:21.273804+0000","last_fullsized":"2024-01-27T01:08:21.273804+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:10:31.665980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"262'3","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.289274+0000","last_change":"2024-01-27T01:08:20.271233+0000","last_active":"2024-01-27T01:08:21.289274+0000","last_peered":"2024-01-27T01:08:21.289274+0000","last_clean":"2024-01-27T01:08:21.289274+0000","last_became_active":"2024-01-27T01:08:20.270986+0000","last_became_peered":"2024-01-27T01:08:20.270986+0000","last_unstale":"2024-01-27T01:08:21.289274+0000","last_undegraded":"2024-01-27T01:08:21.289274+0000","last_fullsized":"2024-01-27T01:08:21.289274+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:59:18.502011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.278752+0000","last_change":"2024-01-27T01:08:21.278752+0000","last_active":"2024-01-27T01:08:21.278752+0000","last_peered":"2024-01-27T01:08:21.278752+0000","last_clean":"2024-01-27T01:08:21.278752+0000","last_became_active":"2024-01-27T01:08:21.278441+0000","last_became_peered":"2024-01-27T01:08:21.278441+0000","last_unstale":"2024-01-27T01:08:21.278752+0000","last_undegraded":"2024-01-27T01:08:21.278752+0000","last_fullsized":"2024-01-27T01:08:21.278752+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:33:42.773892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"262'6","reported_seq":15,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.287362+0000","last_change":"2024-01-27T01:08:20.266857+0000","last_active":"2024-01-27T01:08:21.287362+0000","last_peered":"2024-01-27T01:08:21.287362+0000","last_clean":"2024-01-27T01:08:21.287362+0000","last_became_active":"2024-01-27T01:08:20.266595+0000","last_became_peered":"2024-01-27T01:08:20.266595+0000","last_unstale":"2024-01-27T01:08:21.287362+0000","last_undegraded":"2024-01-27T01:08:21.287362+0000","last_fullsized":"2024-01-27T01:08:21.287362+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:46:58.590323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.260864+0000","last_change":"2024-01-27T01:08:21.260864+0000","last_active":"2024-01-27T01:08:21.260864+0000","last_peered":"2024-01-27T01:08:21.260864+0000","last_clean":"2024-01-27T01:08:21.260864+0000","last_became_active":"2024-01-27T01:08:21.260534+0000","last_became_peered":"2024-01-27T01:08:21.260534+0000","last_unstale":"2024-01-27T01:08:21.260864+0000","last_undegraded":"2024-01-27T01:08:21.260864+0000","last_fullsized":"2024-01-27T01:08:21.260864+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:05:36.113124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.287841+0000","last_change":"2024-01-27T01:08:20.271637+0000","last_active":"2024-01-27T01:08:21.287841+0000","last_peered":"2024-01-27T01:08:21.287841+0000","last_clean":"2024-01-27T01:08:21.287841+0000","last_became_active":"2024-01-27T01:08:20.271391+0000","last_became_peered":"2024-01-27T01:08:20.271391+0000","last_unstale":"2024-01-27T01:08:21.287841+0000","last_undegraded":"2024-01-27T01:08:21.287841+0000","last_fullsized":"2024-01-27T01:08:21.287841+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:08:58.410726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.274144+0000","last_change":"2024-01-27T01:08:21.274144+0000","last_active":"2024-01-27T01:08:21.274144+0000","last_peered":"2024-01-27T01:08:21.274144+0000","last_clean":"2024-01-27T01:08:21.274144+0000","last_became_active":"2024-01-27T01:08:21.273628+0000","last_became_peered":"2024-01-27T01:08:21.273628+0000","last_unstale":"2024-01-27T01:08:21.274144+0000","last_undegraded":"2024-01-27T01:08:21.274144+0000","last_fullsized":"2024-01-27T01:08:21.274144+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:00:06.207428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.273174+0000","last_change":"2024-01-27T01:08:20.272261+0000","last_active":"2024-01-27T01:08:20.273174+0000","last_peered":"2024-01-27T01:08:20.273174+0000","last_clean":"2024-01-27T01:08:20.273174+0000","last_became_active":"2024-01-27T01:08:20.272017+0000","last_became_peered":"2024-01-27T01:08:20.272017+0000","last_unstale":"2024-01-27T01:08:20.273174+0000","last_undegraded":"2024-01-27T01:08:20.273174+0000","last_fullsized":"2024-01-27T01:08:20.273174+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:08:25.341916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.282334+0000","last_change":"2024-01-27T01:08:21.282334+0000","last_active":"2024-01-27T01:08:21.282334+0000","last_peered":"2024-01-27T01:08:21.282334+0000","last_clean":"2024-01-27T01:08:21.282334+0000","last_became_active":"2024-01-27T01:08:21.282102+0000","last_became_peered":"2024-01-27T01:08:21.282102+0000","last_unstale":"2024-01-27T01:08:21.282334+0000","last_undegraded":"2024-01-27T01:08:21.282334+0000","last_fullsized":"2024-01-27T01:08:21.282334+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:23:03.223553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.267869+0000","last_change":"2024-01-27T01:08:20.265907+0000","last_active":"2024-01-27T01:08:20.267869+0000","last_peered":"2024-01-27T01:08:20.267869+0000","last_clean":"2024-01-27T01:08:20.267869+0000","last_became_active":"2024-01-27T01:08:20.265654+0000","last_became_peered":"2024-01-27T01:08:20.265654+0000","last_unstale":"2024-01-27T01:08:20.267869+0000","last_undegraded":"2024-01-27T01:08:20.267869+0000","last_fullsized":"2024-01-27T01:08:20.267869+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:12:08.505429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.270384+0000","last_change":"2024-01-27T01:08:21.270384+0000","last_active":"2024-01-27T01:08:21.270384+0000","last_peered":"2024-01-27T01:08:21.270384+0000","last_clean":"2024-01-27T01:08:21.270384+0000","last_became_active":"2024-01-27T01:08:21.270082+0000","last_became_peered":"2024-01-27T01:08:21.270082+0000","last_unstale":"2024-01-27T01:08:21.270384+0000","last_undegraded":"2024-01-27T01:08:21.270384+0000","last_fullsized":"2024-01-27T01:08:21.270384+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:03:10.846435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"262'8","reported_seq":17,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.286865+0000","last_change":"2024-01-27T01:08:20.266827+0000","last_active":"2024-01-27T01:08:21.286865+0000","last_peered":"2024-01-27T01:08:21.286865+0000","last_clean":"2024-01-27T01:08:21.286865+0000","last_became_active":"2024-01-27T01:08:20.266540+0000","last_became_peered":"2024-01-27T01:08:20.266540+0000","last_unstale":"2024-01-27T01:08:21.286865+0000","last_undegraded":"2024-01-27T01:08:21.286865+0000","last_fullsized":"2024-01-27T01:08:21.286865+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:35:19.719786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.261153+0000","last_change":"2024-01-27T01:08:21.261153+0000","last_active":"2024-01-27T01:08:21.261153+0000","last_peered":"2024-01-27T01:08:21.261153+0000","last_clean":"2024-01-27T01:08:21.261153+0000","last_became_active":"2024-01-27T01:08:21.260202+0000","last_became_peered":"2024-01-27T01:08:21.260202+0000","last_unstale":"2024-01-27T01:08:21.261153+0000","last_undegraded":"2024-01-27T01:08:21.261153+0000","last_fullsized":"2024-01-27T01:08:21.261153+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:19:45.729439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"262'1","reported_seq":10,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.263305+0000","last_change":"2024-01-27T01:08:20.266837+0000","last_active":"2024-01-27T01:08:21.263305+0000","last_peered":"2024-01-27T01:08:21.263305+0000","last_clean":"2024-01-27T01:08:21.263305+0000","last_became_active":"2024-01-27T01:08:20.266460+0000","last_became_peered":"2024-01-27T01:08:20.266460+0000","last_unstale":"2024-01-27T01:08:21.263305+0000","last_undegraded":"2024-01-27T01:08:21.263305+0000","last_fullsized":"2024-01-27T01:08:21.263305+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:20:55.709983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.288079+0000","last_change":"2024-01-27T01:08:21.288079+0000","last_active":"2024-01-27T01:08:21.288079+0000","last_peered":"2024-01-27T01:08:21.288079+0000","last_clean":"2024-01-27T01:08:21.288079+0000","last_became_active":"2024-01-27T01:08:21.286007+0000","last_became_peered":"2024-01-27T01:08:21.286007+0000","last_unstale":"2024-01-27T01:08:21.288079+0000","last_undegraded":"2024-01-27T01:08:21.288079+0000","last_fullsized":"2024-01-27T01:08:21.288079+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:05:14.483844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.286686+0000","last_change":"2024-01-27T01:08:20.272835+0000","last_active":"2024-01-27T01:08:21.286686+0000","last_peered":"2024-01-27T01:08:21.286686+0000","last_clean":"2024-01-27T01:08:21.286686+0000","last_became_active":"2024-01-27T01:08:20.272533+0000","last_became_peered":"2024-01-27T01:08:20.272533+0000","last_unstale":"2024-01-27T01:08:21.286686+0000","last_undegraded":"2024-01-27T01:08:21.286686+0000","last_fullsized":"2024-01-27T01:08:21.286686+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:28:37.662641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.261100+0000","last_change":"2024-01-27T01:08:21.261100+0000","last_active":"2024-01-27T01:08:21.261100+0000","last_peered":"2024-01-27T01:08:21.261100+0000","last_clean":"2024-01-27T01:08:21.261100+0000","last_became_active":"2024-01-27T01:08:21.260780+0000","last_became_peered":"2024-01-27T01:08:21.260780+0000","last_unstale":"2024-01-27T01:08:21.261100+0000","last_undegraded":"2024-01-27T01:08:21.261100+0000","last_fullsized":"2024-01-27T01:08:21.261100+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:12:22.115718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"262'1","reported_seq":10,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.262384+0000","last_change":"2024-01-27T01:08:20.273675+0000","last_active":"2024-01-27T01:08:21.262384+0000","last_peered":"2024-01-27T01:08:21.262384+0000","last_clean":"2024-01-27T01:08:21.262384+0000","last_became_active":"2024-01-27T01:08:20.273330+0000","last_became_peered":"2024-01-27T01:08:20.273330+0000","last_unstale":"2024-01-27T01:08:21.262384+0000","last_undegraded":"2024-01-27T01:08:21.262384+0000","last_fullsized":"2024-01-27T01:08:21.262384+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:36:52.647293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.280321+0000","last_change":"2024-01-27T01:08:21.280321+0000","last_active":"2024-01-27T01:08:21.280321+0000","last_peered":"2024-01-27T01:08:21.280321+0000","last_clean":"2024-01-27T01:08:21.280321+0000","last_became_active":"2024-01-27T01:08:21.280058+0000","last_became_peered":"2024-01-27T01:08:21.280058+0000","last_unstale":"2024-01-27T01:08:21.280321+0000","last_undegraded":"2024-01-27T01:08:21.280321+0000","last_fullsized":"2024-01-27T01:08:21.280321+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:55:33.691451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.272767+0000","last_change":"2024-01-27T01:08:20.271089+0000","last_active":"2024-01-27T01:08:20.272767+0000","last_peered":"2024-01-27T01:08:20.272767+0000","last_clean":"2024-01-27T01:08:20.272767+0000","last_became_active":"2024-01-27T01:08:20.270860+0000","last_became_peered":"2024-01-27T01:08:20.270860+0000","last_unstale":"2024-01-27T01:08:20.272767+0000","last_undegraded":"2024-01-27T01:08:20.272767+0000","last_fullsized":"2024-01-27T01:08:20.272767+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:07:51.523892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.286060+0000","last_change":"2024-01-27T01:08:21.286060+0000","last_active":"2024-01-27T01:08:21.286060+0000","last_peered":"2024-01-27T01:08:21.286060+0000","last_clean":"2024-01-27T01:08:21.286060+0000","last_became_active":"2024-01-27T01:08:21.285769+0000","last_became_peered":"2024-01-27T01:08:21.285769+0000","last_unstale":"2024-01-27T01:08:21.286060+0000","last_undegraded":"2024-01-27T01:08:21.286060+0000","last_fullsized":"2024-01-27T01:08:21.286060+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:56:16.351337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.289176+0000","last_change":"2024-01-27T01:08:20.270955+0000","last_active":"2024-01-27T01:08:21.289176+0000","last_peered":"2024-01-27T01:08:21.289176+0000","last_clean":"2024-01-27T01:08:21.289176+0000","last_became_active":"2024-01-27T01:08:20.270707+0000","last_became_peered":"2024-01-27T01:08:20.270707+0000","last_unstale":"2024-01-27T01:08:21.289176+0000","last_undegraded":"2024-01-27T01:08:21.289176+0000","last_fullsized":"2024-01-27T01:08:21.289176+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:56:57.191103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.286642+0000","last_change":"2024-01-27T01:08:21.286642+0000","last_active":"2024-01-27T01:08:21.286642+0000","last_peered":"2024-01-27T01:08:21.286642+0000","last_clean":"2024-01-27T01:08:21.286642+0000","last_became_active":"2024-01-27T01:08:21.286160+0000","last_became_peered":"2024-01-27T01:08:21.286160+0000","last_unstale":"2024-01-27T01:08:21.286642+0000","last_undegraded":"2024-01-27T01:08:21.286642+0000","last_fullsized":"2024-01-27T01:08:21.286642+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:13:00.293473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.279719+0000","last_change":"2024-01-27T01:08:20.277598+0000","last_active":"2024-01-27T01:08:20.279719+0000","last_peered":"2024-01-27T01:08:20.279719+0000","last_clean":"2024-01-27T01:08:20.279719+0000","last_became_active":"2024-01-27T01:08:20.277279+0000","last_became_peered":"2024-01-27T01:08:20.277279+0000","last_unstale":"2024-01-27T01:08:20.279719+0000","last_undegraded":"2024-01-27T01:08:20.279719+0000","last_fullsized":"2024-01-27T01:08:20.279719+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:44:52.534165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.277018+0000","last_change":"2024-01-27T01:08:21.277018+0000","last_active":"2024-01-27T01:08:21.277018+0000","last_peered":"2024-01-27T01:08:21.277018+0000","last_clean":"2024-01-27T01:08:21.277018+0000","last_became_active":"2024-01-27T01:08:21.276764+0000","last_became_peered":"2024-01-27T01:08:21.276764+0000","last_unstale":"2024-01-27T01:08:21.277018+0000","last_undegraded":"2024-01-27T01:08:21.277018+0000","last_fullsized":"2024-01-27T01:08:21.277018+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:29:16.833734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.269195+0000","last_change":"2024-01-27T01:08:20.268816+0000","last_active":"2024-01-27T01:08:20.269195+0000","last_peered":"2024-01-27T01:08:20.269195+0000","last_clean":"2024-01-27T01:08:20.269195+0000","last_became_active":"2024-01-27T01:08:20.268420+0000","last_became_peered":"2024-01-27T01:08:20.268420+0000","last_unstale":"2024-01-27T01:08:20.269195+0000","last_undegraded":"2024-01-27T01:08:20.269195+0000","last_fullsized":"2024-01-27T01:08:20.269195+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:02:40.864757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.289379+0000","last_change":"2024-01-27T01:08:21.289379+0000","last_active":"2024-01-27T01:08:21.289379+0000","last_peered":"2024-01-27T01:08:21.289379+0000","last_clean":"2024-01-27T01:08:21.289379+0000","last_became_active":"2024-01-27T01:08:21.289016+0000","last_became_peered":"2024-01-27T01:08:21.289016+0000","last_unstale":"2024-01-27T01:08:21.289379+0000","last_undegraded":"2024-01-27T01:08:21.289379+0000","last_fullsized":"2024-01-27T01:08:21.289379+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:12:09.506953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.273287+0000","last_change":"2024-01-27T01:08:20.271769+0000","last_active":"2024-01-27T01:08:20.273287+0000","last_peered":"2024-01-27T01:08:20.273287+0000","last_clean":"2024-01-27T01:08:20.273287+0000","last_became_active":"2024-01-27T01:08:20.271541+0000","last_became_peered":"2024-01-27T01:08:20.271541+0000","last_unstale":"2024-01-27T01:08:20.273287+0000","last_undegraded":"2024-01-27T01:08:20.273287+0000","last_fullsized":"2024-01-27T01:08:20.273287+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:35:52.543816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.277535+0000","last_change":"2024-01-27T01:08:21.277535+0000","last_active":"2024-01-27T01:08:21.277535+0000","last_peered":"2024-01-27T01:08:21.277535+0000","last_clean":"2024-01-27T01:08:21.277535+0000","last_became_active":"2024-01-27T01:08:21.276783+0000","last_became_peered":"2024-01-27T01:08:21.276783+0000","last_unstale":"2024-01-27T01:08:21.277535+0000","last_undegraded":"2024-01-27T01:08:21.277535+0000","last_fullsized":"2024-01-27T01:08:21.277535+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:53:45.819320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.281135+0000","last_change":"2024-01-27T01:08:20.267924+0000","last_active":"2024-01-27T01:08:21.281135+0000","last_peered":"2024-01-27T01:08:21.281135+0000","last_clean":"2024-01-27T01:08:21.281135+0000","last_became_active":"2024-01-27T01:08:20.267642+0000","last_became_peered":"2024-01-27T01:08:20.267642+0000","last_unstale":"2024-01-27T01:08:21.281135+0000","last_undegraded":"2024-01-27T01:08:21.281135+0000","last_fullsized":"2024-01-27T01:08:21.281135+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:58:53.379588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.263670+0000","last_change":"2024-01-27T01:08:21.263670+0000","last_active":"2024-01-27T01:08:21.263670+0000","last_peered":"2024-01-27T01:08:21.263670+0000","last_clean":"2024-01-27T01:08:21.263670+0000","last_became_active":"2024-01-27T01:08:21.263437+0000","last_became_peered":"2024-01-27T01:08:21.263437+0000","last_unstale":"2024-01-27T01:08:21.263670+0000","last_undegraded":"2024-01-27T01:08:21.263670+0000","last_fullsized":"2024-01-27T01:08:21.263670+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:26:15.721480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.272953+0000","last_change":"2024-01-27T01:08:20.269421+0000","last_active":"2024-01-27T01:08:20.272953+0000","last_peered":"2024-01-27T01:08:20.272953+0000","last_clean":"2024-01-27T01:08:20.272953+0000","last_became_active":"2024-01-27T01:08:20.269182+0000","last_became_peered":"2024-01-27T01:08:20.269182+0000","last_unstale":"2024-01-27T01:08:20.272953+0000","last_undegraded":"2024-01-27T01:08:20.272953+0000","last_fullsized":"2024-01-27T01:08:20.272953+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:50:14.733412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.280968+0000","last_change":"2024-01-27T01:08:21.280968+0000","last_active":"2024-01-27T01:08:21.280968+0000","last_peered":"2024-01-27T01:08:21.280968+0000","last_clean":"2024-01-27T01:08:21.280968+0000","last_became_active":"2024-01-27T01:08:21.280700+0000","last_became_peered":"2024-01-27T01:08:21.280700+0000","last_unstale":"2024-01-27T01:08:21.280968+0000","last_undegraded":"2024-01-27T01:08:21.280968+0000","last_fullsized":"2024-01-27T01:08:21.280968+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:50:25.220364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.272618+0000","last_change":"2024-01-27T01:08:20.267520+0000","last_active":"2024-01-27T01:08:20.272618+0000","last_peered":"2024-01-27T01:08:20.272618+0000","last_clean":"2024-01-27T01:08:20.272618+0000","last_became_active":"2024-01-27T01:08:20.267185+0000","last_became_peered":"2024-01-27T01:08:20.267185+0000","last_unstale":"2024-01-27T01:08:20.272618+0000","last_undegraded":"2024-01-27T01:08:20.272618+0000","last_fullsized":"2024-01-27T01:08:20.272618+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:06:42.199152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.282327+0000","last_change":"2024-01-27T01:08:21.282327+0000","last_active":"2024-01-27T01:08:21.282327+0000","last_peered":"2024-01-27T01:08:21.282327+0000","last_clean":"2024-01-27T01:08:21.282327+0000","last_became_active":"2024-01-27T01:08:21.282083+0000","last_became_peered":"2024-01-27T01:08:21.282083+0000","last_unstale":"2024-01-27T01:08:21.282327+0000","last_undegraded":"2024-01-27T01:08:21.282327+0000","last_fullsized":"2024-01-27T01:08:21.282327+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:29:17.826415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.278079+0000","last_change":"2024-01-27T01:08:20.273913+0000","last_active":"2024-01-27T01:08:20.278079+0000","last_peered":"2024-01-27T01:08:20.278079+0000","last_clean":"2024-01-27T01:08:20.278079+0000","last_became_active":"2024-01-27T01:08:20.273631+0000","last_became_peered":"2024-01-27T01:08:20.273631+0000","last_unstale":"2024-01-27T01:08:20.278079+0000","last_undegraded":"2024-01-27T01:08:20.278079+0000","last_fullsized":"2024-01-27T01:08:20.278079+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:04:57.706964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.282599+0000","last_change":"2024-01-27T01:08:21.282599+0000","last_active":"2024-01-27T01:08:21.282599+0000","last_peered":"2024-01-27T01:08:21.282599+0000","last_clean":"2024-01-27T01:08:21.282599+0000","last_became_active":"2024-01-27T01:08:21.282342+0000","last_became_peered":"2024-01-27T01:08:21.282342+0000","last_unstale":"2024-01-27T01:08:21.282599+0000","last_undegraded":"2024-01-27T01:08:21.282599+0000","last_fullsized":"2024-01-27T01:08:21.282599+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T01:30:36.572083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.280783+0000","last_change":"2024-01-27T01:08:20.276248+0000","last_active":"2024-01-27T01:08:20.280783+0000","last_peered":"2024-01-27T01:08:20.280783+0000","last_clean":"2024-01-27T01:08:20.280783+0000","last_became_active":"2024-01-27T01:08:20.276033+0000","last_became_peered":"2024-01-27T01:08:20.276033+0000","last_unstale":"2024-01-27T01:08:20.280783+0000","last_undegraded":"2024-01-27T01:08:20.280783+0000","last_fullsized":"2024-01-27T01:08:20.280783+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:54:13.119628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.273912+0000","last_change":"2024-01-27T01:08:21.273912+0000","last_active":"2024-01-27T01:08:21.273912+0000","last_peered":"2024-01-27T01:08:21.273912+0000","last_clean":"2024-01-27T01:08:21.273912+0000","last_became_active":"2024-01-27T01:08:21.273230+0000","last_became_peered":"2024-01-27T01:08:21.273230+0000","last_unstale":"2024-01-27T01:08:21.273912+0000","last_undegraded":"2024-01-27T01:08:21.273912+0000","last_fullsized":"2024-01-27T01:08:21.273912+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:22:21.341852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.280914+0000","last_change":"2024-01-27T01:08:20.279070+0000","last_active":"2024-01-27T01:08:20.280914+0000","last_peered":"2024-01-27T01:08:20.280914+0000","last_clean":"2024-01-27T01:08:20.280914+0000","last_became_active":"2024-01-27T01:08:20.277648+0000","last_became_peered":"2024-01-27T01:08:20.277648+0000","last_unstale":"2024-01-27T01:08:20.280914+0000","last_undegraded":"2024-01-27T01:08:20.280914+0000","last_fullsized":"2024-01-27T01:08:20.280914+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:10:15.436962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.272841+0000","last_change":"2024-01-27T01:08:21.272841+0000","last_active":"2024-01-27T01:08:21.272841+0000","last_peered":"2024-01-27T01:08:21.272841+0000","last_clean":"2024-01-27T01:08:21.272841+0000","last_became_active":"2024-01-27T01:08:21.272505+0000","last_became_peered":"2024-01-27T01:08:21.272505+0000","last_unstale":"2024-01-27T01:08:21.272841+0000","last_undegraded":"2024-01-27T01:08:21.272841+0000","last_fullsized":"2024-01-27T01:08:21.272841+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:29:22.927153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.274303+0000","last_change":"2024-01-27T01:08:20.268435+0000","last_active":"2024-01-27T01:08:20.274303+0000","last_peered":"2024-01-27T01:08:20.274303+0000","last_clean":"2024-01-27T01:08:20.274303+0000","last_became_active":"2024-01-27T01:08:20.268162+0000","last_became_peered":"2024-01-27T01:08:20.268162+0000","last_unstale":"2024-01-27T01:08:20.274303+0000","last_undegraded":"2024-01-27T01:08:20.274303+0000","last_fullsized":"2024-01-27T01:08:20.274303+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T02:54:13.119628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.290660+0000","last_change":"2024-01-27T01:08:21.290660+0000","last_active":"2024-01-27T01:08:21.290660+0000","last_peered":"2024-01-27T01:08:21.290660+0000","last_clean":"2024-01-27T01:08:21.290660+0000","last_became_active":"2024-01-27T01:08:21.290351+0000","last_became_peered":"2024-01-27T01:08:21.290351+0000","last_unstale":"2024-01-27T01:08:21.290660+0000","last_undegraded":"2024-01-27T01:08:21.290660+0000","last_fullsized":"2024-01-27T01:08:21.290660+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:58:54.381113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":9,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-27T01:08:20.282904+0000","last_change":"2024-01-27T01:08:20.282544+0000","last_active":"2024-01-27T01:08:20.282904+0000","last_peered":"2024-01-27T01:08:20.282904+0000","last_clean":"2024-01-27T01:08:20.282904+0000","last_became_active":"2024-01-27T01:08:20.282312+0000","last_became_peered":"2024-01-27T01:08:20.282312+0000","last_unstale":"2024-01-27T01:08:20.282904+0000","last_undegraded":"2024-01-27T01:08:20.282904+0000","last_fullsized":"2024-01-27T01:08:20.282904+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:19.248477+0000","last_clean_scrub_stamp":"2024-01-27T01:08:19.248477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:18.083351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:21.269615+0000","last_change":"2024-01-27T01:08:21.269615+0000","last_active":"2024-01-27T01:08:21.269615+0000","last_peered":"2024-01-27T01:08:21.269615+0000","last_clean":"2024-01-27T01:08:21.269615+0000","last_became_active":"2024-01-27T01:08:21.269364+0000","last_became_peered":"2024-01-27T01:08:21.269364+0000","last_unstale":"2024-01-27T01:08:21.269615+0000","last_undegraded":"2024-01-27T01:08:21.269615+0000","last_fullsized":"2024-01-27T01:08:21.269615+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T01:08:20.250001+0000","last_clean_scrub_stamp":"2024-01-27T01:08:20.250001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:56:00.967147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"262'10745","reported_seq":11784,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-01-27T01:08:22.075592+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:08:22.075592+0000","last_peered":"2024-01-27T01:08:22.075592+0000","last_clean":"2024-01-27T01:08:22.075592+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:08:22.075592+0000","last_undegraded":"2024-01-27T01:08:22.075592+0000","last_fullsized":"2024-01-27T01:08:22.075592+0000","mapping_epoch":14,"log_start":"98'2229","ondisk_log_start":"98'2229","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8516,"log_dups_size":0,"ondisk_log_size":8516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2528,"num_read_kb":5347,"num_write":680,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2528,"num_read_kb":5347,"num_write":680,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8516,"ondisk_log_size":8516,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576022,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309288,"kb_used_data":3808,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062552,"statfs":{"total":96636764160,"available":96320053248,"internally_reserved":0,"allocated":3899392,"data_stored":2348297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.58199999999999996,"15min":0.60199999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.312},"max":{"1min":0.745,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61699999999999999,"15min":0.61799999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33000000000000002},"max":{"1min":0.76400000000000001,"5min":0.879,"15min":0.879},"last":0.35199999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.58599999999999997,"15min":0.60399999999999998},"min":{"1min":0.40300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73799999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.34300000000000003},"max":{"1min":0.81499999999999995,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.40200000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.59499999999999997,"15min":0.60399999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.753,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.39200000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85799999999999998,"5min":0.872,"15min":0.88900000000000001},"last":0.42799999999999999}]},{"osd":3,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58799999999999997,"15min":0.60799999999999998},"min":{"1min":0.374,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79100000000000004,"5min":0.85399999999999998,"15min":0.876},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64300000000000002,"15min":0.64000000000000001},"min":{"1min":0.40600000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.80600000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.48099999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.496,"15min":0.48699999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76100000000000001,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.52600000000000002,"15min":0.51500000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67800000000000005,"5min":0.80300000000000005,"15min":0.86599999999999999},"last":0.32100000000000001}]},{"osd":5,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.54100000000000004,"15min":0.52400000000000002},"min":{"1min":0.33600000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.81399999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.377}]},{"osd":6,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59299999999999997,"15min":0.58399999999999996},"min":{"1min":0.33700000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84499999999999997,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58799999999999997,"15min":0.58199999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77900000000000003,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.36099999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576022,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1757929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.59199999999999997,"15min":0.59899999999999998},"min":{"1min":0.40999999999999998,"5min":0.40799999999999997,"15min":0.36399999999999999},"max":{"1min":0.65400000000000003,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60199999999999998,"15min":0.61199999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.376},"max":{"1min":0.67600000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.41399999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.59199999999999997,"15min":0.60299999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.65600000000000003,"5min":0.80300000000000005,"15min":0.84499999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60199999999999998,"15min":0.61699999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81200000000000006,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.42199999999999999}]},{"osd":2,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.58299999999999996,"15min":0.61099999999999999},"min":{"1min":0.39200000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70999999999999996,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.436},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69999999999999996,"5min":0.80400000000000005,"15min":0.878},"last":0.40500000000000003}]},{"osd":3,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61399999999999999,"15min":0.623},"min":{"1min":0.379,"5min":0.378,"15min":0.378},"max":{"1min":0.72599999999999998,"5min":0.83699999999999997,"15min":0.90100000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64500000000000002,"15min":0.65400000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.754,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.38600000000000001}]},{"osd":4,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51300000000000001,"15min":0.502},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.71399999999999997,"5min":0.77600000000000002,"15min":0.86599999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.36599999999999999,"5min":0.33800000000000002,"15min":0.29799999999999999},"max":{"1min":0.69299999999999995,"5min":0.78500000000000003,"15min":0.88900000000000001},"last":0.311}]},{"osd":5,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.69199999999999995,"5min":0.75900000000000001,"15min":0.79300000000000004},"last":0.372},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.58299999999999996,"15min":0.57399999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.28899999999999998},"max":{"1min":0.84199999999999997,"5min":0.87,"15min":0.874},"last":0.36199999999999999}]},{"osd":7,"last update":"Sat Jan 27 01:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57799999999999996,"15min":0.57399999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82099999999999995,"5min":0.84899999999999998,"15min":0.92900000000000005},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58999999999999997,"15min":0.58899999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.34699999999999998},"max":{"1min":0.76500000000000001,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.27900000000000003}]}]},{"osd":1,"up_from":13,"seq":55834576022,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304628,"kb_used_data":3244,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067212,"statfs":{"total":96636764160,"available":96324825088,"internally_reserved":0,"allocated":3321856,"data_stored":1765594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.28299999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.502,"5min":0.66100000000000003,"15min":0.69599999999999995},"last":0.502},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.432,"15min":0.45100000000000001},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.76600000000000001},"last":0.33700000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.40799999999999997,"15min":0.432},"min":{"1min":0.29699999999999999,"5min":0.28599999999999998,"15min":0.26200000000000001},"max":{"1min":0.47299999999999998,"5min":0.748,"15min":0.95799999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46899999999999997,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.57199999999999995,"5min":0.67400000000000004,"15min":0.79000000000000004},"last":0.35199999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.501,"15min":0.52600000000000002},"min":{"1min":0.35999999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59099999999999997,"5min":0.73699999999999999,"15min":1.2130000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.48599999999999999,"15min":0.51100000000000001},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.32300000000000001},"max":{"1min":0.628,"5min":0.71799999999999997,"15min":1.22},"last":0.57999999999999996}]},{"osd":4,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.65200000000000002,"15min":0.66600000000000004},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78600000000000003,"5min":0.998,"15min":1.242},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.66700000000000004,"15min":0.68899999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.35699999999999998},"max":{"1min":0.93600000000000005,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.66100000000000003}]},{"osd":5,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.70999999999999996,"15min":0.70299999999999996},"min":{"1min":0.42299999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81299999999999994,"5min":5.444,"15min":5.444},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.69299999999999995,"15min":0.69999999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82899999999999996,"5min":2.6669999999999998,"15min":2.6669999999999998},"last":0.53000000000000003}]},{"osd":6,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.72199999999999998,"15min":0.71899999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47099999999999997},"max":{"1min":0.85499999999999998,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.79100000000000004,"15min":0.73899999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48499999999999999},"max":{"1min":0.82199999999999995,"5min":11.223000000000001,"15min":11.223000000000001},"last":0.56699999999999995}]},{"osd":7,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.72799999999999998,"15min":0.748},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97799999999999998,"5min":2.9660000000000002,"15min":2.9660000000000002},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.751,"15min":0.76300000000000001},"min":{"1min":0.55400000000000005,"5min":0.53400000000000003,"15min":0.52000000000000002},"max":{"1min":0.83199999999999996,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.747}]}]},{"osd":0,"up_from":13,"seq":55834576021,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309308,"kb_used_data":3828,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062532,"statfs":{"total":96636764160,"available":96320032768,"internally_reserved":0,"allocated":3919872,"data_stored":2356460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.434,"15min":0.42499999999999999},"min":{"1min":0.28899999999999998,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.45100000000000001,"15min":0.46000000000000002},"min":{"1min":0.33200000000000002,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":0.64300000000000002,"5min":0.65600000000000003,"15min":0.73399999999999999},"last":0.72399999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.41899999999999998,"15min":0.433},"min":{"1min":0.27400000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.56499999999999995,"5min":0.66900000000000004,"15min":0.76800000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.47599999999999998,"15min":0.48599999999999999},"min":{"1min":0.32900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60299999999999998,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.5}]},{"osd":3,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.34599999999999997,"5min":0.316,"15min":0.30499999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.83699999999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.503,"15min":0.504},"min":{"1min":0.35899999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.65900000000000003,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.63}]},{"osd":4,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.67600000000000005,"15min":0.67200000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.42199999999999999},"max":{"1min":0.91200000000000003,"5min":0.91500000000000004,"15min":1.0720000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.65900000000000003,"15min":0.67600000000000005},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.90100000000000002,"5min":1.002,"15min":1.002},"last":0.55900000000000005}]},{"osd":5,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.68700000000000006,"15min":0.69799999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95799999999999996,"5min":1.0209999999999999,"15min":2.165},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69499999999999995,"15min":0.70999999999999996},"min":{"1min":0.50700000000000001,"5min":0.48799999999999999,"15min":0.45600000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.1399999999999999},"last":0.59599999999999997}]},{"osd":6,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68799999999999994,"15min":0.70999999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.71399999999999997,"15min":0.73699999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.88700000000000001,"5min":1.0109999999999999,"15min":1.425},"last":0.67300000000000004}]},{"osd":7,"last update":"Sat Jan 27 01:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.73099999999999998,"15min":0.752},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.99199999999999999,"5min":1.0169999999999999,"15min":1.1259999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.755,"15min":0.76600000000000001},"min":{"1min":0.55000000000000004,"5min":0.51100000000000001,"15min":0.48499999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.1910000000000001},"last":0.70799999999999996}]}]},{"osd":2,"up_from":13,"seq":55834576021,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312816,"kb_used_data":3240,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059024,"statfs":{"total":96636764160,"available":96316440576,"internally_reserved":0,"allocated":3317760,"data_stored":1765504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38500000000000001,"15min":0.40200000000000002},"min":{"1min":0.29299999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.56200000000000006,"5min":0.60199999999999998,"15min":0.746},"last":0.44},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.42299999999999999,"15min":0.44},"min":{"1min":0.30399999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.48599999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.36899999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.435,"15min":0.443},"min":{"1min":0.316,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":0.52400000000000002,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.497,"15min":0.51600000000000001},"min":{"1min":0.36799999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.65000000000000002,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.47799999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.47899999999999998,"15min":0.49099999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58599999999999997,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.46200000000000002,"15min":0.48099999999999998},"min":{"1min":0.30499999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56699999999999995,"5min":0.63900000000000001,"15min":0.68999999999999995},"last":0.66100000000000003}]},{"osd":4,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.65300000000000002,"15min":0.66700000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89700000000000002,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.66200000000000003,"15min":0.68200000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.378},"max":{"1min":0.77200000000000002,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.59499999999999997}]},{"osd":5,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.68000000000000005,"15min":0.69499999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.876,"5min":1.075,"15min":1.075},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.67700000000000005,"15min":0.69799999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.39800000000000002},"max":{"1min":0.79600000000000004,"5min":0.97499999999999998,"15min":1.0229999999999999},"last":0.60199999999999998}]},{"osd":6,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.68000000000000005,"15min":0.70199999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.73699999999999999,"5min":0.97899999999999998,"15min":1.0549999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67300000000000004,"15min":0.70599999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.39200000000000002},"max":{"1min":0.83199999999999996,"5min":0.91200000000000003,"15min":1.0269999999999999},"last":0.79700000000000004}]},{"osd":7,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.71199999999999997,"15min":0.73599999999999999},"min":{"1min":0.48899999999999999,"5min":0.442,"15min":0.442},"max":{"1min":0.77600000000000002,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.748},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.72499999999999998,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86099999999999999,"5min":0.94599999999999995,"15min":1.0880000000000001},"last":0.78000000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576020,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304632,"kb_used_data":3248,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067208,"statfs":{"total":96636764160,"available":96324820992,"internally_reserved":0,"allocated":3325952,"data_stored":1765596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.439,"15min":0.42199999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63400000000000001,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44,"15min":0.45400000000000001},"min":{"1min":0.30599999999999999,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.66400000000000003,"5min":0.66700000000000004,"15min":0.84699999999999998},"last":0.496}]},{"osd":1,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.61699999999999999,"15min":0.53200000000000003},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.28899999999999998},"max":{"1min":0.76100000000000001,"5min":15.391,"15min":15.391},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.52400000000000002,"15min":0.52300000000000002},"min":{"1min":0.35799999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.628,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.57099999999999995}]},{"osd":2,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.53600000000000003,"15min":0.51300000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70299999999999996,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.50900000000000001},"min":{"1min":0.308,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67600000000000005,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":0.59099999999999997}]},{"osd":4,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.72899999999999998,"15min":0.71499999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.439},"max":{"1min":0.89200000000000002,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.71499999999999997,"15min":0.70299999999999996},"min":{"1min":0.47499999999999998,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":0.96699999999999997,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.624}]},{"osd":5,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.73099999999999998,"15min":0.70899999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.94499999999999995,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.72699999999999998,"15min":0.71499999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.435},"max":{"1min":0.872,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.67300000000000004}]},{"osd":6,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.73799999999999999,"15min":0.71999999999999997},"min":{"1min":0.49199999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.96699999999999997,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92900000000000005,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.70299999999999996}]},{"osd":7,"last update":"Sat Jan 27 01:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.745,"15min":0.73199999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.91400000000000003,"5min":4.569,"15min":4.569},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.78900000000000003,"15min":0.77800000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97999999999999998,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.76800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834576022,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316928,"kb_used_data":3256,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94054912,"statfs":{"total":96636764160,"available":96312229888,"internally_reserved":0,"allocated":3334144,"data_stored":1767130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68500000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59699999999999998,"15min":0.60899999999999999},"min":{"1min":0.35499999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.85699999999999998},"last":0.39100000000000001}]},{"osd":1,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.59899999999999998,"15min":0.622},"min":{"1min":0.41899999999999998,"5min":0.39100000000000001,"15min":0.38800000000000001},"max":{"1min":0.80700000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60999999999999999,"15min":0.62},"min":{"1min":0.38800000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79100000000000004,"5min":0.874,"15min":0.874},"last":0.42999999999999999}]},{"osd":2,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.34200000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80500000000000005,"5min":0.85599999999999998,"15min":0.88700000000000001},"last":0.37},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.626,"15min":0.63300000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.80300000000000005,"5min":0.81499999999999995,"15min":0.89200000000000002},"last":0.41899999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59699999999999998,"15min":0.61699999999999999},"min":{"1min":0.377,"5min":0.373,"15min":0.373},"max":{"1min":0.747,"5min":0.83299999999999996,"15min":0.84199999999999997},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61099999999999999,"15min":0.625},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.39900000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.49299999999999999,"15min":0.48599999999999999},"min":{"1min":0.35599999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.85899999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.30399999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.878},"last":0.317}]},{"osd":6,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.50800000000000001,"15min":0.5},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72599999999999998,"5min":0.82499999999999996,"15min":0.88700000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35199999999999998,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.79400000000000004,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.34399999999999997}]},{"osd":7,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.57199999999999995,"15min":0.56499999999999995},"min":{"1min":0.33800000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.90200000000000002},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.42199999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.76800000000000002,"5min":0.85399999999999998,"15min":0.92600000000000005},"last":0.33400000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576022,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312824,"kb_used_data":3248,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059016,"statfs":{"total":96636764160,"available":96316432384,"internally_reserved":0,"allocated":3325952,"data_stored":1765560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.58799999999999997,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.27300000000000002},"max":{"1min":0.73999999999999999,"5min":0.82799999999999996,"15min":1.006},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.57099999999999995,"15min":0.59799999999999998},"min":{"1min":0.37,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73499999999999999,"5min":0.81499999999999995,"15min":1.1719999999999999},"last":0.51300000000000001}]},{"osd":1,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.61599999999999999,"15min":0.64300000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69699999999999995,"5min":0.89000000000000001,"15min":0.95899999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.65500000000000003,"15min":0.66400000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.377},"max":{"1min":0.77900000000000003,"5min":0.86899999999999999,"15min":1.212},"last":0.63100000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56200000000000006,"15min":0.58699999999999997},"min":{"1min":0.34699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70999999999999996,"5min":0.82099999999999995,"15min":1.631},"last":0.62},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.60299999999999998,"15min":0.61299999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34000000000000002},"max":{"1min":0.67600000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.505}]},{"osd":3,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.373,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71099999999999997,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.61199999999999999,"15min":0.624},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.754,"5min":0.879,"15min":1.4690000000000001},"last":0.65100000000000002}]},{"osd":4,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.5,"15min":0.48699999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69199999999999995,"5min":0.90200000000000002,"15min":0.94399999999999995},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.52300000000000002,"15min":0.51200000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66100000000000003,"5min":0.92500000000000004,"15min":0.93300000000000005},"last":0.316}]},{"osd":6,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51100000000000001,"15min":0.504},"min":{"1min":0.312,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.629,"5min":0.96799999999999997,"15min":1.732},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56399999999999995,"15min":0.54700000000000004},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78300000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.32400000000000001}]},{"osd":7,"last update":"Sat Jan 27 01:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59099999999999997,"15min":0.57499999999999996},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.79700000000000004,"5min":0.95399999999999996,"15min":1.0740000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.621,"15min":0.61299999999999999},"min":{"1min":0.41299999999999998,"5min":0.40799999999999997,"15min":0.39900000000000002},"max":{"1min":0.76500000000000001,"5min":0.996,"15min":1.6919999999999999},"last":0.38300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:08:25.906 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-27T01:08:25.906 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-27T01:08:26.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.086+0000 7fbfa2cbe700 1 Processor -- start 2024-01-27T01:08:26.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.086+0000 7fbfa2cbe700 1 -- start start 2024-01-27T01:08:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.087+0000 7fbfa2cbe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf9c159510 0x7fbf9c1598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.087+0000 7fbfa2cbe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbf9c0ca690 0x7fbf9c0cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.087+0000 7fbfa2cbe700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c069260 con 0x7fbf9c0ca220 2024-01-27T01:08:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.087+0000 7fbfa2cbe700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9c15d670 con 0x7fbf9c0ca690 2024-01-27T01:08:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.087+0000 7fbfa2cbe700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c0c9750 con 0x7fbf9c159510 2024-01-27T01:08:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.087+0000 7fbf9bfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbf9c0ca220 0x7fbf9c0c8be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51742/0 (socket says 172.21.15.46:51742) 2024-01-27T01:08:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.087+0000 7fbf9bfff700 1 -- 172.21.15.46:0/4169818036 learned_addr learned my addr 172.21.15.46:0/4169818036 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.087+0000 7fbfa125b700 1 --2- 172.21.15.46:0/4169818036 >> v2:172.21.15.106:3300/0 conn(0x7fbf9c0ca690 0x7fbf9c0cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.087+0000 7fbfa0a5a700 1 --2- 172.21.15.46:0/4169818036 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf9c159510 0x7fbf9c1598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.087+0000 7fbf9b7fe700 1 -- 172.21.15.46:0/4169818036 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436262817 0 0) 0x7fbf9c069260 con 0x7fbf9c0ca220 2024-01-27T01:08:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.088+0000 7fbf9b7fe700 1 -- 172.21.15.46:0/4169818036 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf88003570 con 0x7fbf9c0ca220 2024-01-27T01:08:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.088+0000 7fbfa125b700 1 -- 172.21.15.46:0/4169818036 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf9c159510 msgr2=0x7fbf9c1598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.088+0000 7fbfa125b700 1 --2- 172.21.15.46:0/4169818036 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf9c159510 0x7fbf9c1598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.088+0000 7fbfa125b700 1 -- 172.21.15.46:0/4169818036 >> v1:172.21.15.46:6789/0 conn(0x7fbf9c0ca220 legacy=0x7fbf9c0c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.088+0000 7fbfa125b700 1 -- 172.21.15.46:0/4169818036 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c0c9a30 con 0x7fbf9c0ca690 2024-01-27T01:08:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.088+0000 7fbfa0a5a700 1 --2- 172.21.15.46:0/4169818036 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf9c159510 0x7fbf9c1598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.088+0000 7fbfa125b700 1 --2- 172.21.15.46:0/4169818036 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf9c0ca690 0x7fbf9c0cc380 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009e90 tx=0x7fbf94009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a33cc9accac845fe server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.088+0000 7fbf9b7fe700 1 -- 172.21.15.46:0/4169818036 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9400e070 con 0x7fbf9c0ca690 2024-01-27T01:08:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.088+0000 7fbf9b7fe700 1 -- 172.21.15.46:0/4169818036 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9400bb40 con 0x7fbf9c0ca690 2024-01-27T01:08:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.088+0000 7fbf9b7fe700 1 -- 172.21.15.46:0/4169818036 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94012430 con 0x7fbf9c0ca690 2024-01-27T01:08:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.089+0000 7fbfa2cbe700 1 -- 172.21.15.46:0/4169818036 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf9c0ca690 msgr2=0x7fbf9c0cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.089+0000 7fbfa2cbe700 1 --2- 172.21.15.46:0/4169818036 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf9c0ca690 0x7fbf9c0cc380 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009e90 tx=0x7fbf94009780 comp rx=0 tx=0).stop 2024-01-27T01:08:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.089+0000 7fbfa2cbe700 1 -- 172.21.15.46:0/4169818036 shutdown_connections 2024-01-27T01:08:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.089+0000 7fbfa2cbe700 1 --2- 172.21.15.46:0/4169818036 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf9c0ca690 0x7fbf9c0cc380 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.089+0000 7fbfa2cbe700 1 --2- 172.21.15.46:0/4169818036 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf9c159510 0x7fbf9c1598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.089+0000 7fbfa2cbe700 1 -- 172.21.15.46:0/4169818036 >> 172.21.15.46:0/4169818036 conn(0x7fbf9c06bd90 msgr2=0x7fbf9c06c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.089+0000 7fbfa2cbe700 1 -- 172.21.15.46:0/4169818036 shutdown_connections 2024-01-27T01:08:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.089+0000 7fbfa2cbe700 1 -- 172.21.15.46:0/4169818036 wait complete. 2024-01-27T01:08:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.090+0000 7fbfa2cbe700 1 Processor -- start 2024-01-27T01:08:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.090+0000 7fbfa2cbe700 1 -- start start 2024-01-27T01:08:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.091+0000 7fbfa2cbe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf9c0ca220 0x7fbf9c15c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.091+0000 7fbfa2cbe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf9c159510 0x7fbf9c1547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.091+0000 7fbfa2cbe700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf9c151380 0x7fbf9c0b2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.091+0000 7fbfa2cbe700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c153710 con 0x7fbf9c159510 2024-01-27T01:08:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.091+0000 7fbfa2cbe700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c1539f0 con 0x7fbf9c151380 2024-01-27T01:08:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.091+0000 7fbfa2cbe700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c153cd0 con 0x7fbf9c0ca220 2024-01-27T01:08:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.091+0000 7fbf9bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf9c159510 0x7fbf9c1547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.091+0000 7fbf9bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf9c159510 0x7fbf9c1547b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39052/0 (socket says 172.21.15.46:39052) 2024-01-27T01:08:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.091+0000 7fbf9bfff700 1 -- 172.21.15.46:0/1066774338 learned_addr learned my addr 172.21.15.46:0/1066774338 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.091+0000 7fbfa0a5a700 1 --2- 172.21.15.46:0/1066774338 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf9c0ca220 0x7fbf9c15c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.091+0000 7fbfa125b700 1 --2- 172.21.15.46:0/1066774338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf9c151380 0x7fbf9c0b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.092+0000 7fbf9bfff700 1 -- 172.21.15.46:0/1066774338 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf9c0ca220 msgr2=0x7fbf9c15c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.092+0000 7fbf9bfff700 1 --2- 172.21.15.46:0/1066774338 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf9c0ca220 0x7fbf9c15c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.092+0000 7fbf9bfff700 1 -- 172.21.15.46:0/1066774338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf9c151380 msgr2=0x7fbf9c0b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.092+0000 7fbf9bfff700 1 --2- 172.21.15.46:0/1066774338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf9c151380 0x7fbf9c0b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.092+0000 7fbf9bfff700 1 -- 172.21.15.46:0/1066774338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c009420 con 0x7fbf9c159510 2024-01-27T01:08:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.092+0000 7fbf9bfff700 1 --2- 172.21.15.46:0/1066774338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf9c159510 0x7fbf9c1547b0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fbf8400c7c0 tx=0x7fbf84007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.092+0000 7fbf98ff9700 1 -- 172.21.15.46:0/1066774338 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84005b30 con 0x7fbf9c159510 2024-01-27T01:08:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.092+0000 7fbf98ff9700 1 -- 172.21.15.46:0/1066774338 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf84005c90 con 0x7fbf9c159510 2024-01-27T01:08:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.092+0000 7fbf98ff9700 1 -- 172.21.15.46:0/1066774338 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84007340 con 0x7fbf9c159510 2024-01-27T01:08:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.092+0000 7fbfa2cbe700 1 -- 172.21.15.46:0/1066774338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf94009a20 con 0x7fbf9c159510 2024-01-27T01:08:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.092+0000 7fbfa2cbe700 1 -- 172.21.15.46:0/1066774338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9c154290 con 0x7fbf9c159510 2024-01-27T01:08:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.093+0000 7fbfa2cbe700 1 -- 172.21.15.46:0/1066774338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9c1544b0 con 0x7fbf9c159510 2024-01-27T01:08:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.093+0000 7fbf98ff9700 1 -- 172.21.15.46:0/1066774338 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7fbf8401b400 con 0x7fbf9c159510 2024-01-27T01:08:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.094+0000 7fbf98ff9700 1 --2- 172.21.15.46:0/1066774338 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbf7c0670e0 0x7fbf7c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.094+0000 7fbf98ff9700 1 -- 172.21.15.46:0/1066774338 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fbf840846b0 con 0x7fbf9c159510 2024-01-27T01:08:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.094+0000 7fbfa0a5a700 1 --2- 172.21.15.46:0/1066774338 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbf7c0670e0 0x7fbf7c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.095+0000 7fbfa0a5a700 1 --2- 172.21.15.46:0/1066774338 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbf7c0670e0 0x7fbf7c069590 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0093f0 tx=0x7fbf8c008040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.098+0000 7fbf98ff9700 1 -- 172.21.15.46:0/1066774338 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbf84051350 con 0x7fbf9c159510 2024-01-27T01:08:26.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.394+0000 7fbfa2cbe700 1 -- 172.21.15.46:0/1066774338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fbf9c04c710 con 0x7fbf9c159510 2024-01-27T01:08:26.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.398+0000 7fbf98ff9700 1 -- 172.21.15.46:0/1066774338 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v56) v1 ==== 190+0+0 (secure 0 0 0) 0x7fbf84054970 con 0x7fbf9c159510 2024-01-27T01:08:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.400+0000 7fbfa2cbe700 1 -- 172.21.15.46:0/1066774338 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbf7c0670e0 msgr2=0x7fbf7c069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.400+0000 7fbfa2cbe700 1 --2- 172.21.15.46:0/1066774338 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbf7c0670e0 0x7fbf7c069590 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0093f0 tx=0x7fbf8c008040 comp rx=0 tx=0).stop 2024-01-27T01:08:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.400+0000 7fbfa2cbe700 1 -- 172.21.15.46:0/1066774338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf9c159510 msgr2=0x7fbf9c1547b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.400+0000 7fbfa2cbe700 1 --2- 172.21.15.46:0/1066774338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf9c159510 0x7fbf9c1547b0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fbf8400c7c0 tx=0x7fbf84007e00 comp rx=0 tx=0).stop 2024-01-27T01:08:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.400+0000 7fbfa2cbe700 1 -- 172.21.15.46:0/1066774338 shutdown_connections 2024-01-27T01:08:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.400+0000 7fbfa2cbe700 1 --2- 172.21.15.46:0/1066774338 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fbf7c0670e0 0x7fbf7c069590 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.400+0000 7fbfa2cbe700 1 --2- 172.21.15.46:0/1066774338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbf9c151380 0x7fbf9c0b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.400+0000 7fbfa2cbe700 1 --2- 172.21.15.46:0/1066774338 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbf9c0ca220 0x7fbf9c15c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.400+0000 7fbfa2cbe700 1 --2- 172.21.15.46:0/1066774338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf9c159510 0x7fbf9c1547b0 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.400+0000 7fbfa2cbe700 1 -- 172.21.15.46:0/1066774338 >> 172.21.15.46:0/1066774338 conn(0x7fbf9c06bd90 msgr2=0x7fbf9c06c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.400+0000 7fbfa2cbe700 1 -- 172.21.15.46:0/1066774338 shutdown_connections 2024-01-27T01:08:26.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.400+0000 7fbfa2cbe700 1 -- 172.21.15.46:0/1066774338 wait complete. 2024-01-27T01:08:26.401 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-01-27T01:08:26.413 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:08:26.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.545+0000 7f1a20a08700 1 Processor -- start 2024-01-27T01:08:26.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.545+0000 7f1a20a08700 1 -- start start 2024-01-27T01:08:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.546+0000 7f1a20a08700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1a1c159960 0x7f1a1c06bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.546+0000 7f1a20a08700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c06c600 con 0x7f1a1c06c1f0 2024-01-27T01:08:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.547+0000 7f1a20a08700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c15dac0 con 0x7f1a1c159590 2024-01-27T01:08:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.547+0000 7f1a20a08700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a1c15e350 con 0x7f1a1c159960 2024-01-27T01:08:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.547+0000 7f1a1ad9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1a1c06c1f0 0x7f1a1c069370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51752/0 (socket says 172.21.15.46:51752) 2024-01-27T01:08:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.547+0000 7f1a1ad9d700 1 -- 172.21.15.46:0/2452000161 learned_addr learned my addr 172.21.15.46:0/2452000161 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.547+0000 7f1a19d9b700 1 --2- 172.21.15.46:0/2452000161 >> v2:172.21.15.106:3300/0 conn(0x7f1a1c159960 0x7f1a1c06bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.547+0000 7f1a1959a700 1 -- 172.21.15.46:0/2452000161 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073836514 0 0) 0x7f1a1c06c600 con 0x7f1a1c06c1f0 2024-01-27T01:08:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.547+0000 7f1a1959a700 1 -- 172.21.15.46:0/2452000161 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a04003570 con 0x7f1a1c06c1f0 2024-01-27T01:08:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.547+0000 7f1a19d9b700 1 -- 172.21.15.46:0/2452000161 >> v1:172.21.15.106:6789/0 conn(0x7f1a1c159590 legacy=0x7f1a1c15d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.547+0000 7f1a19d9b700 1 -- 172.21.15.46:0/2452000161 >> v1:172.21.15.46:6789/0 conn(0x7f1a1c06c1f0 legacy=0x7f1a1c069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.547+0000 7f1a1959a700 1 -- 172.21.15.46:0/2452000161 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874648672 0 0) 0x7f1a1c15dac0 con 0x7f1a1c159590 2024-01-27T01:08:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.547+0000 7f1a19d9b700 1 -- 172.21.15.46:0/2452000161 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c15dd90 con 0x7f1a1c159960 2024-01-27T01:08:26.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.547+0000 7f1a19d9b700 1 --2- 172.21.15.46:0/2452000161 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1a1c159960 0x7f1a1c06bcb0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f1a100098e0 tx=0x7f1a100091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5437947149085a4b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:26.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.548+0000 7f1a1959a700 1 -- 172.21.15.46:0/2452000161 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1000b640 con 0x7f1a1c159960 2024-01-27T01:08:26.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.548+0000 7f1a1959a700 1 -- 172.21.15.46:0/2452000161 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1000b7a0 con 0x7f1a1c159960 2024-01-27T01:08:26.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.548+0000 7f1a1959a700 1 -- 172.21.15.46:0/2452000161 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1000bb40 con 0x7f1a1c159960 2024-01-27T01:08:26.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.548+0000 7f1a20a08700 1 -- 172.21.15.46:0/2452000161 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1a1c159960 msgr2=0x7f1a1c06bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:26.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.548+0000 7f1a20a08700 1 --2- 172.21.15.46:0/2452000161 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1a1c159960 0x7f1a1c06bcb0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f1a100098e0 tx=0x7f1a100091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.548+0000 7f1a20a08700 1 -- 172.21.15.46:0/2452000161 shutdown_connections 2024-01-27T01:08:26.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.548+0000 7f1a20a08700 1 --2- 172.21.15.46:0/2452000161 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1a1c159960 0x7f1a1c06bcb0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.548+0000 7f1a20a08700 1 -- 172.21.15.46:0/2452000161 >> 172.21.15.46:0/2452000161 conn(0x7f1a1c0ce530 msgr2=0x7f1a1c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:26.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.548+0000 7f1a20a08700 1 -- 172.21.15.46:0/2452000161 shutdown_connections 2024-01-27T01:08:26.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.548+0000 7f1a20a08700 1 -- 172.21.15.46:0/2452000161 wait complete. 2024-01-27T01:08:26.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.550+0000 7f1a20a08700 1 Processor -- start 2024-01-27T01:08:26.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.550+0000 7f1a20a08700 1 -- start start 2024-01-27T01:08:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.551+0000 7f1a20a08700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1a1c06c1f0 0x7f1a1c16dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.551+0000 7f1a20a08700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1a1c159590 0x7f1a1c158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.551+0000 7f1a20a08700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a1c159960 0x7f1a1c0b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.551+0000 7f1a20a08700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c0b1fe0 con 0x7f1a1c159960 2024-01-27T01:08:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.551+0000 7f1a20a08700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c0b1410 con 0x7f1a1c06c1f0 2024-01-27T01:08:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.551+0000 7f1a20a08700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c16e020 con 0x7f1a1c159590 2024-01-27T01:08:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.551+0000 7f1a1ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a1c159960 0x7f1a1c0b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.551+0000 7f1a1ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a1c159960 0x7f1a1c0b2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39056/0 (socket says 172.21.15.46:39056) 2024-01-27T01:08:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.551+0000 7f1a1ad9d700 1 -- 172.21.15.46:0/302894879 learned_addr learned my addr 172.21.15.46:0/302894879 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.551+0000 7f1a19d9b700 1 --2- 172.21.15.46:0/302894879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1a1c159590 0x7f1a1c158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.551+0000 7f1a1a59c700 1 --2- 172.21.15.46:0/302894879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1a1c06c1f0 0x7f1a1c16dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.552+0000 7f1a1ad9d700 1 -- 172.21.15.46:0/302894879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1a1c159590 msgr2=0x7f1a1c158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.552+0000 7f1a1ad9d700 1 --2- 172.21.15.46:0/302894879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1a1c159590 0x7f1a1c158a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.552+0000 7f1a1ad9d700 1 -- 172.21.15.46:0/302894879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1a1c06c1f0 msgr2=0x7f1a1c16dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.552+0000 7f1a1ad9d700 1 --2- 172.21.15.46:0/302894879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1a1c06c1f0 0x7f1a1c16dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.552+0000 7f1a1ad9d700 1 -- 172.21.15.46:0/302894879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a10009420 con 0x7f1a1c159960 2024-01-27T01:08:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.552+0000 7f1a19d9b700 1 --2- 172.21.15.46:0/302894879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1a1c159590 0x7f1a1c158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.552+0000 7f1a1ad9d700 1 --2- 172.21.15.46:0/302894879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a1c159960 0x7f1a1c0b2470 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f1a14001460 tx=0x7f1a14001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.552+0000 7f1a0a7fc700 1 -- 172.21.15.46:0/302894879 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1400ce80 con 0x7f1a1c159960 2024-01-27T01:08:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.552+0000 7f1a0a7fc700 1 -- 172.21.15.46:0/302894879 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1400ab70 con 0x7f1a1c159960 2024-01-27T01:08:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.552+0000 7f1a20a08700 1 -- 172.21.15.46:0/302894879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a1c158fa0 con 0x7f1a1c159960 2024-01-27T01:08:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.552+0000 7f1a0a7fc700 1 -- 172.21.15.46:0/302894879 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a140070f0 con 0x7f1a1c159960 2024-01-27T01:08:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.552+0000 7f1a20a08700 1 -- 172.21.15.46:0/302894879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a1c0b1990 con 0x7f1a1c159960 2024-01-27T01:08:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.553+0000 7f1a20a08700 1 -- 172.21.15.46:0/302894879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a1c04c710 con 0x7f1a1c159960 2024-01-27T01:08:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.554+0000 7f1a0a7fc700 1 -- 172.21.15.46:0/302894879 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f1a14007520 con 0x7f1a1c159960 2024-01-27T01:08:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.554+0000 7f1a0a7fc700 1 --2- 172.21.15.46:0/302894879 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f19fc0670e0 0x7f19fc069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.554+0000 7f1a0a7fc700 1 -- 172.21.15.46:0/302894879 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f1a14085020 con 0x7f1a1c159960 2024-01-27T01:08:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.554+0000 7f1a1a59c700 1 --2- 172.21.15.46:0/302894879 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f19fc0670e0 0x7f19fc069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.555+0000 7f1a1a59c700 1 --2- 172.21.15.46:0/302894879 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f19fc0670e0 0x7f19fc069590 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c0011e0 tx=0x7f1a0c009040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:26.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.558+0000 7f1a0a7fc700 1 -- 172.21.15.46:0/302894879 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1a14051d40 con 0x7f1a1c159960 2024-01-27T01:08:26.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.811+0000 7f1a20a08700 1 -- 172.21.15.46:0/302894879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a1c0b4410 con 0x7f1a1c159960 2024-01-27T01:08:26.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.811+0000 7f1a0a7fc700 1 -- 172.21.15.46:0/302894879 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+4616 (secure 0 0 0) 0x7f1a14055360 con 0x7f1a1c159960 2024-01-27T01:08:26.810 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:26.810 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":149,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3826743114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3826743114},{"type":"v1","addr":"172.21.15.46:6835","nonce":3826743114}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":144},{"gid":11947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2514069949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2514069949},{"type":"v1","addr":"172.21.15.106:6833","nonce":2514069949}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":144},{"gid":11950,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/251184858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":251184858},{"type":"v1","addr":"172.21.15.106:6835","nonce":251184858}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:08:20.703665+0000","modified":"2024-01-27T01:08:24.272153+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12348},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12348":{"gid":12348,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6833/3798781145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3798781145},{"type":"v1","addr":"172.21.15.46:6833","nonce":3798781145}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2024-01-27T01:08:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.813+0000 7f1a20a08700 1 -- 172.21.15.46:0/302894879 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f19fc0670e0 msgr2=0x7f19fc069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.813+0000 7f1a20a08700 1 --2- 172.21.15.46:0/302894879 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f19fc0670e0 0x7f19fc069590 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c0011e0 tx=0x7f1a0c009040 comp rx=0 tx=0).stop 2024-01-27T01:08:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.813+0000 7f1a20a08700 1 -- 172.21.15.46:0/302894879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a1c159960 msgr2=0x7f1a1c0b2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.813+0000 7f1a20a08700 1 --2- 172.21.15.46:0/302894879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a1c159960 0x7f1a1c0b2470 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f1a14001460 tx=0x7f1a14001ae0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.813+0000 7f1a20a08700 1 -- 172.21.15.46:0/302894879 shutdown_connections 2024-01-27T01:08:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.813+0000 7f1a20a08700 1 --2- 172.21.15.46:0/302894879 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f19fc0670e0 0x7f19fc069590 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.813+0000 7f1a20a08700 1 --2- 172.21.15.46:0/302894879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a1c159960 0x7f1a1c0b2470 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.813+0000 7f1a20a08700 1 --2- 172.21.15.46:0/302894879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1a1c06c1f0 0x7f1a1c16dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.813+0000 7f1a20a08700 1 --2- 172.21.15.46:0/302894879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1a1c159590 0x7f1a1c158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.813+0000 7f1a20a08700 1 -- 172.21.15.46:0/302894879 >> 172.21.15.46:0/302894879 conn(0x7f1a1c0ce530 msgr2=0x7f1a1c069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.813+0000 7f1a20a08700 1 -- 172.21.15.46:0/302894879 shutdown_connections 2024-01-27T01:08:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:26.813+0000 7f1a20a08700 1 -- 172.21.15.46:0/302894879 wait complete. 2024-01-27T01:08:26.814 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 149 2024-01-27T01:08:27.229 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 149, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-27T01:08:20.703665+0000', 'modified': '2024-01-27T01:08:24.272153+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12348}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12348': {'gid': 12348, 'name': 'c', 'rank': 0, 'incarnation': 145, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6833/3798781145', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 3798781145}, {'type': 'v1', 'addr': '172.21.15.46:6833', 'nonce': 3798781145}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [36], 'metadata_pool': 35, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-27T01:08:27.229 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-27T01:08:27.229 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-27T01:08:27.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.373+0000 7f5e8880d700 1 Processor -- start 2024-01-27T01:08:27.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.373+0000 7f5e8880d700 1 -- start start 2024-01-27T01:08:27.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.374+0000 7f5e8880d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5e840c3cf0 0x7f5e840ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.374+0000 7f5e8880d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5e840cd080 0x7f5e840ab1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.374+0000 7f5e8880d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e840cbf10 con 0x7f5e840acf30 2024-01-27T01:08:27.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.374+0000 7f5e8880d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e840ada10 con 0x7f5e840c3cf0 2024-01-27T01:08:27.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.374+0000 7f5e8880d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e84157be0 con 0x7f5e840cd080 2024-01-27T01:08:27.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.374+0000 7f5e8259c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5e840acf30 0x7f5e840ad300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51754/0 (socket says 172.21.15.46:51754) 2024-01-27T01:08:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.374+0000 7f5e8259c700 1 -- 172.21.15.46:0/442418409 learned_addr learned my addr 172.21.15.46:0/442418409 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.375+0000 7f5e81d9b700 1 --2- 172.21.15.46:0/442418409 >> v2:172.21.15.106:3300/0 conn(0x7f5e840c3cf0 0x7f5e840ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.375+0000 7f5e82d9d700 1 --2- 172.21.15.46:0/442418409 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5e840cd080 0x7f5e840ab1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.375+0000 7f5e8159a700 1 -- 172.21.15.46:0/442418409 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074727410 0 0) 0x7f5e840cbf10 con 0x7f5e840acf30 2024-01-27T01:08:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.375+0000 7f5e8159a700 1 -- 172.21.15.46:0/442418409 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e6c003570 con 0x7f5e840acf30 2024-01-27T01:08:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.375+0000 7f5e8159a700 1 -- 172.21.15.46:0/442418409 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3714063131 0 0) 0x7f5e6c003570 con 0x7f5e840acf30 2024-01-27T01:08:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.375+0000 7f5e81d9b700 1 -- 172.21.15.46:0/442418409 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5e840cd080 msgr2=0x7f5e840ab1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.375+0000 7f5e81d9b700 1 --2- 172.21.15.46:0/442418409 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5e840cd080 0x7f5e840ab1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.375+0000 7f5e81d9b700 1 -- 172.21.15.46:0/442418409 >> v1:172.21.15.46:6789/0 conn(0x7f5e840acf30 legacy=0x7f5e840ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.375+0000 7f5e81d9b700 1 -- 172.21.15.46:0/442418409 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e840cd480 con 0x7f5e840c3cf0 2024-01-27T01:08:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.375+0000 7f5e82d9d700 1 --2- 172.21.15.46:0/442418409 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5e840cd080 0x7f5e840ab1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.375+0000 7f5e81d9b700 1 --2- 172.21.15.46:0/442418409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5e840c3cf0 0x7f5e840ccb40 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f5e780098e0 tx=0x7f5e780091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdfe7482412f975d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.375+0000 7f5e8159a700 1 -- 172.21.15.46:0/442418409 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e74001470 con 0x7f5e840c3cf0 2024-01-27T01:08:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.375+0000 7f5e8159a700 1 -- 172.21.15.46:0/442418409 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e78011420 con 0x7f5e840c3cf0 2024-01-27T01:08:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.375+0000 7f5e8159a700 1 -- 172.21.15.46:0/442418409 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e780117c0 con 0x7f5e840c3cf0 2024-01-27T01:08:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.376+0000 7f5e8880d700 1 -- 172.21.15.46:0/442418409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5e840c3cf0 msgr2=0x7f5e840ccb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.376+0000 7f5e8880d700 1 --2- 172.21.15.46:0/442418409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5e840c3cf0 0x7f5e840ccb40 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f5e780098e0 tx=0x7f5e780091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.376+0000 7f5e8880d700 1 -- 172.21.15.46:0/442418409 shutdown_connections 2024-01-27T01:08:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.376+0000 7f5e8880d700 1 --2- 172.21.15.46:0/442418409 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5e840cd080 0x7f5e840ab1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.376+0000 7f5e8880d700 1 --2- 172.21.15.46:0/442418409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5e840c3cf0 0x7f5e840ccb40 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.376+0000 7f5e8880d700 1 -- 172.21.15.46:0/442418409 >> 172.21.15.46:0/442418409 conn(0x7f5e84097820 msgr2=0x7f5e84068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.376+0000 7f5e8880d700 1 -- 172.21.15.46:0/442418409 shutdown_connections 2024-01-27T01:08:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.376+0000 7f5e8880d700 1 -- 172.21.15.46:0/442418409 wait complete. 2024-01-27T01:08:27.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.377+0000 7f5e8880d700 1 Processor -- start 2024-01-27T01:08:27.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.377+0000 7f5e8880d700 1 -- start start 2024-01-27T01:08:27.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.378+0000 7f5e8880d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e840acf30 0x7f5e840ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.378+0000 7f5e8880d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5e840c3cf0 0x7f5e84159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.378+0000 7f5e8880d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5e840cd080 0x7f5e84153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.378+0000 7f5e8880d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e840ae6b0 con 0x7f5e840acf30 2024-01-27T01:08:27.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.378+0000 7f5e8880d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e84158c10 con 0x7f5e840cd080 2024-01-27T01:08:27.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.378+0000 7f5e8880d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e84158090 con 0x7f5e840c3cf0 2024-01-27T01:08:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.378+0000 7f5e8259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e840acf30 0x7f5e840ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.378+0000 7f5e8259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e840acf30 0x7f5e840ae170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39070/0 (socket says 172.21.15.46:39070) 2024-01-27T01:08:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.378+0000 7f5e8259c700 1 -- 172.21.15.46:0/72743525 learned_addr learned my addr 172.21.15.46:0/72743525 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.378+0000 7f5e81d9b700 1 --2- 172.21.15.46:0/72743525 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5e840c3cf0 0x7f5e84159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.378+0000 7f5e82d9d700 1 --2- 172.21.15.46:0/72743525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5e840cd080 0x7f5e84153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.379+0000 7f5e8259c700 1 -- 172.21.15.46:0/72743525 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5e840c3cf0 msgr2=0x7f5e84159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.379+0000 7f5e8259c700 1 --2- 172.21.15.46:0/72743525 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5e840c3cf0 0x7f5e84159790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.379+0000 7f5e8259c700 1 -- 172.21.15.46:0/72743525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5e840cd080 msgr2=0x7f5e84153600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.379+0000 7f5e8259c700 1 --2- 172.21.15.46:0/72743525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5e840cd080 0x7f5e84153600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.379+0000 7f5e8259c700 1 -- 172.21.15.46:0/72743525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7c009a20 con 0x7f5e840acf30 2024-01-27T01:08:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.379+0000 7f5e8259c700 1 --2- 172.21.15.46:0/72743525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e840acf30 0x7f5e840ae170 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400cb90 tx=0x7f5e74001ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.379+0000 7f5e727fc700 1 -- 172.21.15.46:0/72743525 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e74001d60 con 0x7f5e840acf30 2024-01-27T01:08:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.379+0000 7f5e727fc700 1 -- 172.21.15.46:0/72743525 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e74007360 con 0x7f5e840acf30 2024-01-27T01:08:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.379+0000 7f5e8880d700 1 -- 172.21.15.46:0/72743525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e78009420 con 0x7f5e840acf30 2024-01-27T01:08:27.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.379+0000 7f5e727fc700 1 -- 172.21.15.46:0/72743525 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e74007700 con 0x7f5e840acf30 2024-01-27T01:08:27.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.379+0000 7f5e8880d700 1 -- 172.21.15.46:0/72743525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e84158930 con 0x7f5e840acf30 2024-01-27T01:08:27.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.381+0000 7f5e727fc700 1 -- 172.21.15.46:0/72743525 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 143) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f5e74004a40 con 0x7f5e840acf30 2024-01-27T01:08:27.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.381+0000 7f5e727fc700 1 --2- 172.21.15.46:0/72743525 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5e64067110 0x7f5e640695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.381+0000 7f5e727fc700 1 -- 172.21.15.46:0/72743525 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f5e74084500 con 0x7f5e840acf30 2024-01-27T01:08:27.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.381+0000 7f5e81d9b700 1 --2- 172.21.15.46:0/72743525 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5e64067110 0x7f5e640695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.382+0000 7f5e81d9b700 1 --2- 172.21.15.46:0/72743525 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5e64067110 0x7f5e640695c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5e780091e0 tx=0x7f5e78016000 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:27.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.383+0000 7f5e91b10b80 1 Processor -- start 2024-01-27T01:08:27.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.383+0000 7f5e91b10b80 1 -- start start 2024-01-27T01:08:27.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.384+0000 7f5e91b10b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037eda010 0x55e037eda3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.384+0000 7f5e91b10b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037f22360 0x55e037f04840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.384+0000 7f5e91b10b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037ed95d0 0x55e037f04d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.384+0000 7f5e91b10b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e037ddd9f0 con 0x55e037ed95d0 2024-01-27T01:08:27.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.384+0000 7f5e91b10b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e037e2f130 con 0x55e037f22360 2024-01-27T01:08:27.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.384+0000 7f5e91b10b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e037e2c930 con 0x55e037eda010 2024-01-27T01:08:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.384+0000 7f5e82d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037ed95d0 0x55e037f04d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.384+0000 7f5e82d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037ed95d0 0x55e037f04d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39082/0 (socket says 172.21.15.46:39082) 2024-01-27T01:08:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.384+0000 7f5e82d9d700 1 -- 172.21.15.46:0/1258122931 learned_addr learned my addr 172.21.15.46:0/1258122931 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.384+0000 7f5e81d9b700 1 --2- 172.21.15.46:0/1258122931 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037f22360 0x55e037f04840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.384+0000 7f5e8259c700 1 --2- 172.21.15.46:0/1258122931 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037eda010 0x55e037eda3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.384+0000 7f5e82d9d700 1 -- 172.21.15.46:0/1258122931 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037eda010 msgr2=0x55e037eda3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.384+0000 7f5e82d9d700 1 --2- 172.21.15.46:0/1258122931 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037eda010 0x55e037eda3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.384+0000 7f5e82d9d700 1 -- 172.21.15.46:0/1258122931 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037f22360 msgr2=0x55e037f04840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.384+0000 7f5e82d9d700 1 --2- 172.21.15.46:0/1258122931 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037f22360 0x55e037f04840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.384+0000 7f5e82d9d700 1 -- 172.21.15.46:0/1258122931 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e037c67b10 con 0x55e037ed95d0 2024-01-27T01:08:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.385+0000 7f5e82d9d700 1 --2- 172.21.15.46:0/1258122931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037ed95d0 0x55e037f04d80 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c007720 tx=0x7f5e7c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b52e2bbaffcd08f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.385+0000 7f5e637fe700 1 -- 172.21.15.46:0/1258122931 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c00e070 con 0x55e037ed95d0 2024-01-27T01:08:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.385+0000 7f5e637fe700 1 -- 172.21.15.46:0/1258122931 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7c00f040 con 0x55e037ed95d0 2024-01-27T01:08:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.385+0000 7f5e637fe700 1 -- 172.21.15.46:0/1258122931 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c015880 con 0x55e037ed95d0 2024-01-27T01:08:27.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.386+0000 7f5e91b10b80 1 -- 172.21.15.46:0/1258122931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037ed95d0 msgr2=0x55e037f04d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:27.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.386+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/1258122931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037ed95d0 0x55e037f04d80 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c007720 tx=0x7f5e7c00b040 comp rx=0 tx=0).stop 2024-01-27T01:08:27.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.386+0000 7f5e91b10b80 1 -- 172.21.15.46:0/1258122931 shutdown_connections 2024-01-27T01:08:27.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.386+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/1258122931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037ed95d0 0x55e037f04d80 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.386+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/1258122931 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037eda010 0x55e037eda3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.386+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/1258122931 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037f22360 0x55e037f04840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.386+0000 7f5e91b10b80 1 -- 172.21.15.46:0/1258122931 >> 172.21.15.46:0/1258122931 conn(0x55e037ed9a70 msgr2=0x55e037f22820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:27.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.386+0000 7f5e91b10b80 1 -- 172.21.15.46:0/1258122931 shutdown_connections 2024-01-27T01:08:27.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.386+0000 7f5e91b10b80 1 -- 172.21.15.46:0/1258122931 wait complete. 2024-01-27T01:08:27.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.386+0000 7f5e91b10b80 1 -- start start 2024-01-27T01:08:27.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.386+0000 7f5e91b10b80 1 Processor -- start 2024-01-27T01:08:27.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.387+0000 7f5e91b10b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037ed95d0 0x55e037f07e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.387+0000 7f5e91b10b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037eda010 0x55e037f05ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.387+0000 7f5e91b10b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037f22360 0x55e037f06410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.387+0000 7f5e91b10b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e037e28720 con 0x55e037ed95d0 2024-01-27T01:08:27.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.387+0000 7f5e91b10b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e037dd41b0 con 0x55e037f22360 2024-01-27T01:08:27.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.387+0000 7f5e91b10b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e037dd3e50 con 0x55e037eda010 2024-01-27T01:08:27.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.387+0000 7f5e91b10b80 10 client.0 _ll_register_callbacks cb 0x55e037c367b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-27T01:08:27.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e8259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037ed95d0 0x55e037f07e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e8259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037ed95d0 0x55e037f07e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39092/0 (socket says 172.21.15.46:39092) 2024-01-27T01:08:27.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e8259c700 1 -- 172.21.15.46:0/2277921918 learned_addr learned my addr 172.21.15.46:0/2277921918 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:27.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e81d9b700 1 --2- 172.21.15.46:0/2277921918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037eda010 0x55e037f05ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e82d9d700 1 --2- 172.21.15.46:0/2277921918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037f22360 0x55e037f06410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e8259c700 1 -- 172.21.15.46:0/2277921918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037eda010 msgr2=0x55e037f05ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:27.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e8259c700 1 --2- 172.21.15.46:0/2277921918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037eda010 0x55e037f05ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e8259c700 1 -- 172.21.15.46:0/2277921918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037f22360 msgr2=0x55e037f06410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e8259c700 1 --2- 172.21.15.46:0/2277921918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037f22360 0x55e037f06410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e8259c700 1 -- 172.21.15.46:0/2277921918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e7c007420 con 0x55e037ed95d0 2024-01-27T01:08:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e82d9d700 1 --2- 172.21.15.46:0/2277921918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037f22360 0x55e037f06410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e8259c700 1 --2- 172.21.15.46:0/2277921918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037ed95d0 0x55e037f07e20 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f5e74051360 tx=0x7f5e74009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fac6a608247dc30 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e60ff9700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-27T01:08:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e60ff9700 1 -- 172.21.15.46:0/2277921918 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7405be00 con 0x55e037ed95d0 2024-01-27T01:08:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e60ff9700 1 -- 172.21.15.46:0/2277921918 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(264..264 src has 1..264) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f5e7406b180 con 0x55e037ed95d0 2024-01-27T01:08:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e91b10b80 1 -- 172.21.15.46:0/2277921918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e037ecd190 con 0x55e037ed95d0 2024-01-27T01:08:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e60ff9700 1 -- 172.21.15.46:0/2277921918 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=149) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e74054560 con 0x55e037ed95d0 2024-01-27T01:08:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e91b10b80 10 client.12495 fetch_fsmap learned FSMap version 149 2024-01-27T01:08:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.388+0000 7f5e91b10b80 1 -- 172.21.15.46:0/2277921918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=149,monmap=2+}) v3 -- 0x55e037df6290 con 0x55e037ed95d0 2024-01-27T01:08:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.389+0000 7f5e60ff9700 1 -- 172.21.15.46:0/2277921918 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e74080070 con 0x55e037ed95d0 2024-01-27T01:08:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.389+0000 7f5e60ff9700 1 -- 172.21.15.46:0/2277921918 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 149) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f5e7405b970 con 0x55e037ed95d0 2024-01-27T01:08:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.389+0000 7f5e91b10b80 10 client.12495 fetch_fsmap finished waiting for FSMap version 149 2024-01-27T01:08:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.389+0000 7f5e91b10b80 10 client.12495 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-27T01:08:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.389+0000 7f5e91b10b80 20 client.12495 populate_metadata read hostname 'smithi046' 2024-01-27T01:08:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.389+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/2277921918 >> [v2:172.21.15.46:6832/3798781145,v1:172.21.15.46:6833/3798781145] conn(0x55e037f21400 0x55e037f217d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.389+0000 7f5e91b10b80 4 client.12495 mds_command: new command op to 12348 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-27T01:08:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.389+0000 7f5e91b10b80 1 -- 172.21.15.46:0/2277921918 --> [v2:172.21.15.46:6832/3798781145,v1:172.21.15.46:6833/3798781145] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e037dcaee0 con 0x55e037f21400 2024-01-27T01:08:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.389+0000 7f5e81d9b700 1 --2- 172.21.15.46:0/2277921918 >> [v2:172.21.15.46:6832/3798781145,v1:172.21.15.46:6833/3798781145] conn(0x55e037f21400 0x55e037f217d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.389+0000 7f5e81d9b700 1 --2- 172.21.15.46:0/2277921918 >> [v2:172.21.15.46:6832/3798781145,v1:172.21.15.46:6833/3798781145] conn(0x55e037f21400 0x55e037f217d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a4ab0f83227627d server_cookie=831e550fec4dd704 in_seq=0 out_seq=0 2024-01-27T01:08:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.389+0000 7f5e60ff9700 10 client.12495 ms_handle_connect on v2:172.21.15.46:6832/3798781145 2024-01-27T01:08:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.390+0000 7f5e60ff9700 1 -- 172.21.15.46:0/2277921918 <== mds.0 v2:172.21.15.46:6832/3798781145 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17146 (crc 0 0 0) 0x55e037dcaee0 con 0x55e037f21400 2024-01-27T01:08:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.390+0000 7f5e60ff9700 10 client.12495 handle_command_reply: tid=0 2024-01-27T01:08:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.390+0000 7f5e91b10b80 1 client.12495 shutdown 2024-01-27T01:08:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.390+0000 7f5e91b10b80 1 -- 172.21.15.46:0/2277921918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037ed95d0 msgr2=0x55e037f07e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.390+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/2277921918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037ed95d0 0x55e037f07e20 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f5e74051360 tx=0x7f5e74009040 comp rx=0 tx=0).stop 2024-01-27T01:08:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.390+0000 7f5e91b10b80 1 -- 172.21.15.46:0/2277921918 shutdown_connections 2024-01-27T01:08:27.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.390+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/2277921918 >> [v2:172.21.15.46:6832/3798781145,v1:172.21.15.46:6833/3798781145] conn(0x55e037f21400 0x55e037f217d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.390+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/2277921918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037ed95d0 0x55e037f07e20 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.390+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/2277921918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037f22360 0x55e037f06410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.390+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/2277921918 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037eda010 0x55e037f05ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.390+0000 7f5e60ff9700 0 client.12495 ms_handle_reset on v2:172.21.15.46:6832/3798781145 2024-01-27T01:08:27.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.390+0000 7f5e91b10b80 1 -- 172.21.15.46:0/2277921918 >> 172.21.15.46:0/2277921918 conn(0x55e037ed9a70 msgr2=0x55e037ed4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:27.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.390+0000 7f5e91b10b80 1 -- 172.21.15.46:0/2277921918 shutdown_connections 2024-01-27T01:08:27.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.390+0000 7f5e91b10b80 1 -- 172.21.15.46:0/2277921918 wait complete. 2024-01-27T01:08:27.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.391+0000 7f5e91b10b80 20 client.12495 trim_cache size 0 max 16384 2024-01-27T01:08:27.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.407+0000 7f5e727fc700 1 -- 172.21.15.46:0/72743525 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mgrmap(e 144) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f5e7405b970 con 0x7f5e840acf30 2024-01-27T01:08:27.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.411+0000 7f5e91b10b80 1 Processor -- start 2024-01-27T01:08:27.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.411+0000 7f5e91b10b80 1 -- start start 2024-01-27T01:08:27.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.411+0000 7f5e91b10b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037ed95d0 0x55e037f1f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.411+0000 7f5e91b10b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037eda010 0x55e037ed4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.411+0000 7f5e91b10b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037f22360 0x55e037f309b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.411+0000 7f5e91b10b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e037d20e40 con 0x55e037f22360 2024-01-27T01:08:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.411+0000 7f5e91b10b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e037f1fae0 con 0x55e037eda010 2024-01-27T01:08:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.411+0000 7f5e91b10b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e037ed2bc0 con 0x55e037ed95d0 2024-01-27T01:08:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.412+0000 7f5e82d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037f22360 0x55e037f309b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.412+0000 7f5e82d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037f22360 0x55e037f309b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39102/0 (socket says 172.21.15.46:39102) 2024-01-27T01:08:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.412+0000 7f5e82d9d700 1 -- 172.21.15.46:0/3940092509 learned_addr learned my addr 172.21.15.46:0/3940092509 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.412+0000 7f5e81d9b700 1 --2- 172.21.15.46:0/3940092509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037eda010 0x55e037ed4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.412+0000 7f5e8259c700 1 --2- 172.21.15.46:0/3940092509 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037ed95d0 0x55e037f1f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.412+0000 7f5e82d9d700 1 -- 172.21.15.46:0/3940092509 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037ed95d0 msgr2=0x55e037f1f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.412+0000 7f5e82d9d700 1 --2- 172.21.15.46:0/3940092509 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037ed95d0 0x55e037f1f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.412+0000 7f5e82d9d700 1 -- 172.21.15.46:0/3940092509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037eda010 msgr2=0x55e037ed4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.412+0000 7f5e82d9d700 1 --2- 172.21.15.46:0/3940092509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037eda010 0x55e037ed4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.412+0000 7f5e82d9d700 1 -- 172.21.15.46:0/3940092509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e037ed62e0 con 0x55e037f22360 2024-01-27T01:08:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.412+0000 7f5e8259c700 1 --2- 172.21.15.46:0/3940092509 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037ed95d0 0x55e037f1f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:08:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.412+0000 7f5e82d9d700 1 --2- 172.21.15.46:0/3940092509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037f22360 0x55e037f309b0 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c017000 tx=0x7f5e7c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c5ad42b0cf748f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.412+0000 7f5e63fff700 1 -- 172.21.15.46:0/3940092509 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c024070 con 0x55e037f22360 2024-01-27T01:08:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.412+0000 7f5e63fff700 1 -- 172.21.15.46:0/3940092509 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7c001010 con 0x55e037f22360 2024-01-27T01:08:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.412+0000 7f5e63fff700 1 -- 172.21.15.46:0/3940092509 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c015e80 con 0x55e037f22360 2024-01-27T01:08:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.413+0000 7f5e91b10b80 1 -- 172.21.15.46:0/3940092509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037f22360 msgr2=0x55e037f309b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.413+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/3940092509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037f22360 0x55e037f309b0 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c017000 tx=0x7f5e7c01b000 comp rx=0 tx=0).stop 2024-01-27T01:08:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.413+0000 7f5e91b10b80 1 -- 172.21.15.46:0/3940092509 shutdown_connections 2024-01-27T01:08:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.413+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/3940092509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037f22360 0x55e037f309b0 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.413+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/3940092509 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037ed95d0 0x55e037f1f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.413+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/3940092509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037eda010 0x55e037ed4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.413+0000 7f5e91b10b80 1 -- 172.21.15.46:0/3940092509 >> 172.21.15.46:0/3940092509 conn(0x55e037ed9a70 msgr2=0x55e037f206a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.413+0000 7f5e91b10b80 1 -- 172.21.15.46:0/3940092509 shutdown_connections 2024-01-27T01:08:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.413+0000 7f5e91b10b80 1 -- 172.21.15.46:0/3940092509 wait complete. 2024-01-27T01:08:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.414+0000 7f5e91b10b80 1 -- start start 2024-01-27T01:08:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.414+0000 7f5e91b10b80 1 Processor -- start 2024-01-27T01:08:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.415+0000 7f5e91b10b80 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037ed95d0 0x55e037eff960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.415+0000 7f5e91b10b80 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037eda010 0x55e037f81c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.415+0000 7f5e91b10b80 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037f22360 0x55e037f03f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.415+0000 7f5e91b10b80 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e037f30d20 con 0x55e037eda010 2024-01-27T01:08:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.415+0000 7f5e91b10b80 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e037f88af0 con 0x55e037f22360 2024-01-27T01:08:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.415+0000 7f5e91b10b80 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e037f89020 con 0x55e037ed95d0 2024-01-27T01:08:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.415+0000 7f5e91b10b80 10 client.0 _ll_register_callbacks cb 0x55e037c367b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-27T01:08:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.415+0000 7f5e81d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037eda010 0x55e037f81c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.415+0000 7f5e81d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037eda010 0x55e037f81c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39114/0 (socket says 172.21.15.46:39114) 2024-01-27T01:08:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.415+0000 7f5e81d9b700 1 -- 172.21.15.46:0/874244949 learned_addr learned my addr 172.21.15.46:0/874244949 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.416+0000 7f5e8259c700 1 --2- 172.21.15.46:0/874244949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037ed95d0 0x55e037eff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.416+0000 7f5e82d9d700 1 --2- 172.21.15.46:0/874244949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037f22360 0x55e037f03f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.416+0000 7f5e81d9b700 1 -- 172.21.15.46:0/874244949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037ed95d0 msgr2=0x55e037eff960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.416+0000 7f5e81d9b700 1 --2- 172.21.15.46:0/874244949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037ed95d0 0x55e037eff960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.416+0000 7f5e81d9b700 1 -- 172.21.15.46:0/874244949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037f22360 msgr2=0x55e037f03f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.416+0000 7f5e81d9b700 1 --2- 172.21.15.46:0/874244949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037f22360 0x55e037f03f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.416+0000 7f5e81d9b700 1 -- 172.21.15.46:0/874244949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e037c67b10 con 0x55e037eda010 2024-01-27T01:08:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.416+0000 7f5e81d9b700 1 --2- 172.21.15.46:0/874244949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037eda010 0x55e037f81c90 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f5e78002000 tx=0x7f5e7802b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d465805ad60915a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.416+0000 7f5e60ff9700 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-01-27T01:08:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.416+0000 7f5e60ff9700 1 -- 172.21.15.46:0/874244949 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e78021070 con 0x55e037eda010 2024-01-27T01:08:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.416+0000 7f5e60ff9700 1 -- 172.21.15.46:0/874244949 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(264..264 src has 1..264) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f5e78034950 con 0x55e037eda010 2024-01-27T01:08:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.416+0000 7f5e91b10b80 1 -- 172.21.15.46:0/874244949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e840b1910 con 0x55e037eda010 2024-01-27T01:08:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.416+0000 7f5e60ff9700 1 -- 172.21.15.46:0/874244949 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=149) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e78034d20 con 0x55e037eda010 2024-01-27T01:08:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.416+0000 7f5e91b10b80 10 client.12501 fetch_fsmap learned FSMap version 149 2024-01-27T01:08:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.416+0000 7f5e91b10b80 1 -- 172.21.15.46:0/874244949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=149,monmap=2+}) v3 -- 0x55e037cbfc80 con 0x55e037eda010 2024-01-27T01:08:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.417+0000 7f5e60ff9700 1 -- 172.21.15.46:0/874244949 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e840b1910 con 0x55e037eda010 2024-01-27T01:08:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.417+0000 7f5e60ff9700 1 -- 172.21.15.46:0/874244949 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 149) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f5e7803e430 con 0x55e037eda010 2024-01-27T01:08:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.417+0000 7f5e91b10b80 10 client.12501 fetch_fsmap finished waiting for FSMap version 149 2024-01-27T01:08:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.417+0000 7f5e91b10b80 10 client.12501 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-27T01:08:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.417+0000 7f5e91b10b80 20 client.12501 populate_metadata read hostname 'smithi046' 2024-01-27T01:08:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.417+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/874244949 >> [v2:172.21.15.46:6832/3798781145,v1:172.21.15.46:6833/3798781145] conn(0x55e037f1b790 0x55e037f1bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.417+0000 7f5e91b10b80 4 client.12501 mds_command: new command op to 12348 tid=0[{"prefix": "status"}] 2024-01-27T01:08:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.417+0000 7f5e91b10b80 1 -- 172.21.15.46:0/874244949 --> [v2:172.21.15.46:6832/3798781145,v1:172.21.15.46:6833/3798781145] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f5e7c00f040 con 0x55e037f1b790 2024-01-27T01:08:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.417+0000 7f5e8259c700 1 --2- 172.21.15.46:0/874244949 >> [v2:172.21.15.46:6832/3798781145,v1:172.21.15.46:6833/3798781145] conn(0x55e037f1b790 0x55e037f1bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.417+0000 7f5e60ff9700 10 client.12501 ms_handle_connect on v2:172.21.15.46:6832/3798781145 2024-01-27T01:08:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.417+0000 7f5e8259c700 1 --2- 172.21.15.46:0/874244949 >> [v2:172.21.15.46:6832/3798781145,v1:172.21.15.46:6833/3798781145] conn(0x55e037f1b790 0x55e037f1bb60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc111d8d1f14c181 server_cookie=d986c74fd8f6fcbe in_seq=0 out_seq=0 2024-01-27T01:08:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.418+0000 7f5e60ff9700 1 -- 172.21.15.46:0/874244949 <== mds.0 v2:172.21.15.46:6832/3798781145 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x7f5e7c00f040 con 0x55e037f1b790 2024-01-27T01:08:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.418+0000 7f5e60ff9700 10 client.12501 handle_command_reply: tid=0 2024-01-27T01:08:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.418+0000 7f5e91b10b80 1 client.12501 shutdown 2024-01-27T01:08:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.418+0000 7f5e91b10b80 1 -- 172.21.15.46:0/874244949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037eda010 msgr2=0x55e037f81c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.418+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/874244949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037eda010 0x55e037f81c90 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f5e78002000 tx=0x7f5e7802b040 comp rx=0 tx=0).stop 2024-01-27T01:08:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.418+0000 7f5e91b10b80 1 -- 172.21.15.46:0/874244949 shutdown_connections 2024-01-27T01:08:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.418+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/874244949 >> [v2:172.21.15.46:6832/3798781145,v1:172.21.15.46:6833/3798781145] conn(0x55e037f1b790 0x55e037f1bb60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.418+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/874244949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e037f22360 0x55e037f03f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.418+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/874244949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e037ed95d0 0x55e037eff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.418+0000 7f5e60ff9700 0 client.12501 ms_handle_reset on v2:172.21.15.46:6832/3798781145 2024-01-27T01:08:27.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.418+0000 7f5e91b10b80 1 --2- 172.21.15.46:0/874244949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e037eda010 0x55e037f81c90 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.418+0000 7f5e91b10b80 1 -- 172.21.15.46:0/874244949 >> 172.21.15.46:0/874244949 conn(0x55e037ed9a70 msgr2=0x55e037ed3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:27.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.418+0000 7f5e91b10b80 1 -- 172.21.15.46:0/874244949 shutdown_connections 2024-01-27T01:08:27.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.418+0000 7f5e91b10b80 1 -- 172.21.15.46:0/874244949 wait complete. 2024-01-27T01:08:27.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.418+0000 7f5e91b10b80 20 client.12501 trim_cache size 0 max 16384 2024-01-27T01:08:27.422 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:08:27.422 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "62f24da2-f257-4a3f-ab5a-77a1100d6f1e", 2024-01-27T01:08:27.422 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-01-27T01:08:27.422 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 12348, 2024-01-27T01:08:27.422 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-01-27T01:08:27.422 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-01-27T01:08:27.423 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-01-27T01:08:27.423 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.1558165120000004, 2024-01-27T01:08:27.423 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 149, 2024-01-27T01:08:27.423 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 264, 2024-01-27T01:08:27.423 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 263, 2024-01-27T01:08:27.423 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 11.184222686 2024-01-27T01:08:27.423 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:08:27.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.419+0000 7f5e8880d700 1 -- 172.21.15.46:0/72743525 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5e64067110 msgr2=0x7f5e640695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:27.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.419+0000 7f5e8880d700 1 --2- 172.21.15.46:0/72743525 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5e64067110 0x7f5e640695c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5e780091e0 tx=0x7f5e78016000 comp rx=0 tx=0).stop 2024-01-27T01:08:27.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.419+0000 7f5e8880d700 1 -- 172.21.15.46:0/72743525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e840acf30 msgr2=0x7f5e840ae170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:27.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.419+0000 7f5e8880d700 1 --2- 172.21.15.46:0/72743525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e840acf30 0x7f5e840ae170 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400cb90 tx=0x7f5e74001ad0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.419+0000 7f5e8880d700 1 -- 172.21.15.46:0/72743525 shutdown_connections 2024-01-27T01:08:27.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.419+0000 7f5e8880d700 1 --2- 172.21.15.46:0/72743525 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5e64067110 0x7f5e640695c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.419+0000 7f5e8880d700 1 --2- 172.21.15.46:0/72743525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e840acf30 0x7f5e840ae170 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.419+0000 7f5e8880d700 1 --2- 172.21.15.46:0/72743525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5e840cd080 0x7f5e84153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.419+0000 7f5e8880d700 1 --2- 172.21.15.46:0/72743525 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5e840c3cf0 0x7f5e84159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.419+0000 7f5e8880d700 1 -- 172.21.15.46:0/72743525 >> 172.21.15.46:0/72743525 conn(0x7f5e84097820 msgr2=0x7f5e84154760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:27.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.419+0000 7f5e8880d700 1 -- 172.21.15.46:0/72743525 shutdown_connections 2024-01-27T01:08:27.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.419+0000 7f5e8880d700 1 -- 172.21.15.46:0/72743525 wait complete. 2024-01-27T01:08:27.431 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-27T01:08:27.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.564+0000 7fa59a0ba700 1 Processor -- start 2024-01-27T01:08:27.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.564+0000 7fa59a0ba700 1 -- start start 2024-01-27T01:08:27.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.565+0000 7fa59a0ba700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa5940b2160 0x7fa59415a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.565+0000 7fa59a0ba700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa5940b2530 0x7fa5940abe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.565+0000 7fa59a0ba700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa59415da10 0x7fa59415de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.565+0000 7fa59a0ba700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5940a9b90 con 0x7fa5940b2160 2024-01-27T01:08:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.565+0000 7fa59a0ba700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5940aa210 con 0x7fa5940b2530 2024-01-27T01:08:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.565+0000 7fa59a0ba700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59415d580 con 0x7fa59415da10 2024-01-27T01:08:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.565+0000 7fa5937fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa5940b2160 0x7fa59415a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.565+0000 7fa5937fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa5940b2160 0x7fa59415a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39128/0 (socket says 172.21.15.46:39128) 2024-01-27T01:08:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.565+0000 7fa5937fe700 1 -- 172.21.15.46:0/1986919714 learned_addr learned my addr 172.21.15.46:0/1986919714 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.565+0000 7fa593fff700 1 --2- 172.21.15.46:0/1986919714 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa59415da10 0x7fa59415de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.565+0000 7fa592ffd700 1 --2- 172.21.15.46:0/1986919714 >> v2:172.21.15.106:3300/0 conn(0x7fa5940b2530 0x7fa5940abe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.565+0000 7fa5937fe700 1 -- 172.21.15.46:0/1986919714 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa59415da10 msgr2=0x7fa59415de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.565+0000 7fa5937fe700 1 --2- 172.21.15.46:0/1986919714 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa59415da10 0x7fa59415de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.565+0000 7fa5937fe700 1 -- 172.21.15.46:0/1986919714 >> v2:172.21.15.106:3300/0 conn(0x7fa5940b2530 msgr2=0x7fa5940abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.565+0000 7fa5937fe700 1 --2- 172.21.15.46:0/1986919714 >> v2:172.21.15.106:3300/0 conn(0x7fa5940b2530 0x7fa5940abe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.565+0000 7fa5937fe700 1 -- 172.21.15.46:0/1986919714 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5940ac390 con 0x7fa5940b2160 2024-01-27T01:08:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.565+0000 7fa593fff700 1 --2- 172.21.15.46:0/1986919714 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa59415da10 0x7fa59415de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.566+0000 7fa5937fe700 1 --2- 172.21.15.46:0/1986919714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940b2160 0x7fa59415a7a0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7fa594063c70 tx=0x7fa588009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a0059ac3215ea5 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.566+0000 7fa5927fc700 1 -- 172.21.15.46:0/1986919714 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa588009e70 con 0x7fa5940b2160 2024-01-27T01:08:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.566+0000 7fa5927fc700 1 -- 172.21.15.46:0/1986919714 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa58800b860 con 0x7fa5940b2160 2024-01-27T01:08:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.566+0000 7fa5927fc700 1 -- 172.21.15.46:0/1986919714 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58800bc20 con 0x7fa5940b2160 2024-01-27T01:08:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.566+0000 7fa59a0ba700 1 -- 172.21.15.46:0/1986919714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940b2160 msgr2=0x7fa59415a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.566+0000 7fa59a0ba700 1 --2- 172.21.15.46:0/1986919714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940b2160 0x7fa59415a7a0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7fa594063c70 tx=0x7fa588009250 comp rx=0 tx=0).stop 2024-01-27T01:08:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.566+0000 7fa59a0ba700 1 -- 172.21.15.46:0/1986919714 shutdown_connections 2024-01-27T01:08:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.566+0000 7fa59a0ba700 1 --2- 172.21.15.46:0/1986919714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940b2160 0x7fa59415a7a0 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.566+0000 7fa59a0ba700 1 --2- 172.21.15.46:0/1986919714 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa59415da10 0x7fa59415de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.566+0000 7fa59a0ba700 1 --2- 172.21.15.46:0/1986919714 >> v2:172.21.15.106:3300/0 conn(0x7fa5940b2530 0x7fa5940abe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.566+0000 7fa59a0ba700 1 -- 172.21.15.46:0/1986919714 >> 172.21.15.46:0/1986919714 conn(0x7fa5940ccbc0 msgr2=0x7fa5940ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.566+0000 7fa59a0ba700 1 -- 172.21.15.46:0/1986919714 shutdown_connections 2024-01-27T01:08:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.566+0000 7fa59a0ba700 1 -- 172.21.15.46:0/1986919714 wait complete. 2024-01-27T01:08:27.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.568+0000 7fa59a0ba700 1 Processor -- start 2024-01-27T01:08:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.568+0000 7fa59a0ba700 1 -- start start 2024-01-27T01:08:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa59a0ba700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5940b2160 0x7fa594168d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa59a0ba700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5940b2530 0x7fa59415c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa59a0ba700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa59415da10 0x7fa59415cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa59a0ba700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa594169680 con 0x7fa59415da10 2024-01-27T01:08:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa59a0ba700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5940ae340 con 0x7fa5940b2530 2024-01-27T01:08:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa59a0ba700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5940ae620 con 0x7fa5940b2160 2024-01-27T01:08:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa593fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa59415da10 0x7fa59415cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa593fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa59415da10 0x7fa59415cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39134/0 (socket says 172.21.15.46:39134) 2024-01-27T01:08:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa593fff700 1 -- 172.21.15.46:0/4286482186 learned_addr learned my addr 172.21.15.46:0/4286482186 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa5937fe700 1 --2- 172.21.15.46:0/4286482186 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5940b2160 0x7fa594168d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa592ffd700 1 --2- 172.21.15.46:0/4286482186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5940b2530 0x7fa59415c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa593fff700 1 -- 172.21.15.46:0/4286482186 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5940b2160 msgr2=0x7fa594168d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa593fff700 1 --2- 172.21.15.46:0/4286482186 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5940b2160 0x7fa594168d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa593fff700 1 -- 172.21.15.46:0/4286482186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5940b2530 msgr2=0x7fa59415c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:27.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa593fff700 1 --2- 172.21.15.46:0/4286482186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5940b2530 0x7fa59415c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa593fff700 1 -- 172.21.15.46:0/4286482186 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5880094f0 con 0x7fa59415da10 2024-01-27T01:08:27.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa593fff700 1 --2- 172.21.15.46:0/4286482186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa59415da10 0x7fa59415cc20 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7fa58c009fa0 tx=0x7fa58c0099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:27.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.569+0000 7fa57b7fe700 1 -- 172.21.15.46:0/4286482186 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c00dbb0 con 0x7fa59415da10 2024-01-27T01:08:27.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.570+0000 7fa57b7fe700 1 -- 172.21.15.46:0/4286482186 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa58c00f040 con 0x7fa59415da10 2024-01-27T01:08:27.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.570+0000 7fa59a0ba700 1 -- 172.21.15.46:0/4286482186 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5940ae900 con 0x7fa59415da10 2024-01-27T01:08:27.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.570+0000 7fa57b7fe700 1 -- 172.21.15.46:0/4286482186 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c015400 con 0x7fa59415da10 2024-01-27T01:08:27.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.570+0000 7fa59a0ba700 1 -- 172.21.15.46:0/4286482186 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa594169260 con 0x7fa59415da10 2024-01-27T01:08:27.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.571+0000 7fa59a0ba700 1 -- 172.21.15.46:0/4286482186 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa59404c710 con 0x7fa59415da10 2024-01-27T01:08:27.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.571+0000 7fa57b7fe700 1 -- 172.21.15.46:0/4286482186 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 144) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fa58c0157f0 con 0x7fa59415da10 2024-01-27T01:08:27.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.572+0000 7fa57b7fe700 1 --2- 172.21.15.46:0/4286482186 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa580067170 0x7fa580069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:27.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.572+0000 7fa57b7fe700 1 -- 172.21.15.46:0/4286482186 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6560+0+0 (secure 0 0 0) 0x7fa58c0854c0 con 0x7fa59415da10 2024-01-27T01:08:27.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.572+0000 7fa5937fe700 1 --2- 172.21.15.46:0/4286482186 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa580067170 0x7fa580069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:27.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.572+0000 7fa5937fe700 1 --2- 172.21.15.46:0/4286482186 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa580067170 0x7fa580069620 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa594063c70 tx=0x7fa588017000 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:27.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.576+0000 7fa57b7fe700 1 -- 172.21.15.46:0/4286482186 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa58c051fc0 con 0x7fa59415da10 2024-01-27T01:08:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.829+0000 7fa59a0ba700 1 -- 172.21.15.46:0/4286482186 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa5940aa460 con 0x7fa59415da10 2024-01-27T01:08:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.830+0000 7fa57b7fe700 1 -- 172.21.15.46:0/4286482186 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v149) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa58c0555e0 con 0x7fa59415da10 2024-01-27T01:08:27.829 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-27T01:08:27.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.832+0000 7fa59a0ba700 1 -- 172.21.15.46:0/4286482186 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa580067170 msgr2=0x7fa580069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.832+0000 7fa59a0ba700 1 --2- 172.21.15.46:0/4286482186 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa580067170 0x7fa580069620 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa594063c70 tx=0x7fa588017000 comp rx=0 tx=0).stop 2024-01-27T01:08:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.832+0000 7fa59a0ba700 1 -- 172.21.15.46:0/4286482186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa59415da10 msgr2=0x7fa59415cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.832+0000 7fa59a0ba700 1 --2- 172.21.15.46:0/4286482186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa59415da10 0x7fa59415cc20 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7fa58c009fa0 tx=0x7fa58c0099b0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.832+0000 7fa59a0ba700 1 -- 172.21.15.46:0/4286482186 shutdown_connections 2024-01-27T01:08:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.832+0000 7fa59a0ba700 1 --2- 172.21.15.46:0/4286482186 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa580067170 0x7fa580069620 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.832+0000 7fa59a0ba700 1 --2- 172.21.15.46:0/4286482186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa59415da10 0x7fa59415cc20 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.832+0000 7fa59a0ba700 1 --2- 172.21.15.46:0/4286482186 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5940b2160 0x7fa594168d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.832+0000 7fa59a0ba700 1 --2- 172.21.15.46:0/4286482186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5940b2530 0x7fa59415c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.832+0000 7fa59a0ba700 1 -- 172.21.15.46:0/4286482186 >> 172.21.15.46:0/4286482186 conn(0x7fa5940ccbc0 msgr2=0x7fa5940afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.832+0000 7fa59a0ba700 1 -- 172.21.15.46:0/4286482186 shutdown_connections 2024-01-27T01:08:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:27.832+0000 7fa59a0ba700 1 -- 172.21.15.46:0/4286482186 wait complete. 2024-01-27T01:08:27.845 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-27T01:08:27.845 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-27T01:08:27.845 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-27T01:08:27.845 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-27T01:08:27.845 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-27T01:08:27.845 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-27T01:08:27.845 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-27T01:08:27.845 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-27T01:08:27.845 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-27T01:08:27.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:27.846 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-27T01:08:27.865 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-27T01:08:27.865 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-27T01:08:27.865 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-27T01:08:27.865 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:08:27.865 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-27T01:08:27.865 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:08:27.865 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T01:08:27.865 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:08:27.865 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-27T01:08:27.866 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T01:08:27.866 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:08:27.866 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-27T01:08:27.866 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-27T01:08:27.866 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:08:27.866 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-27T01:08:27.866 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-27T01:08:27.866 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 36423sec preferred_lft 36423sec 2024-01-27T01:08:27.866 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-27T01:08:27.866 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:08:27.866 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T01:08:27.866 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:08:27.866 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-27T01:08:27.866 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-27T01:08:27.866 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:08:27.867 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-27T01:08:27.867 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:08:27.867 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::cce6:a7ff:fe4f:38d/64 scope link 2024-01-27T01:08:27.867 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:08:27.867 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-27T01:08:27.867 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 76:ef:90:4c:9d:6f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-27T01:08:27.867 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::74ef:90ff:fe4c:9d6f/64 scope link 2024-01-27T01:08:27.867 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:08:27.867 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-27T01:08:27.867 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-27T01:08:27.867 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::2c73:afff:fef0:6f39/64 scope link 2024-01-27T01:08:27.867 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:08:27.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:27.868 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-27T01:08:27.936 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-27T01:08:27.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:27.938 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-27T01:08:28.002 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-27T01:08:28.002 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-27T01:08:28.002 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-27T01:08:28.002 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-27T01:08:28.002 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-27T01:08:28.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:28.003 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-27T01:08:28.003 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-27T01:08:28.003 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-27T01:08:28.003 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-27T01:08:28.003 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-27T01:08:28.003 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-27T01:08:28.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:28.127 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-27T01:08:28.147 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-27T01:08:28.147 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-27T01:08:28.147 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-27T01:08:28.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:28.147 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-01-27T01:08:28.223 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-27T01:08:28.223 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-27T01:08:28.223 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-27T01:08:28.285 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-27T01:08:28.285 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-27T01:08:28.285 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-27T01:08:28.346 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-27T01:08:28.421 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-27T01:08:28.421 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-27T01:08:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-27T01:08:28.497 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-27T01:08:28.497 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:08:28.498 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-27T01:08:28.559 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-27T01:08:28.560 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-27T01:08:28.603 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-27T01:08:28.637 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-27T01:08:28.637 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-27T01:08:28.706 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-01-27T01:08:28.707+0000 7f6388f3f540 -1 init, newargv = 0x5615ff5dd5b0 newargc=15 2024-01-27T01:08:28.706 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[122008]: starting ceph client 2024-01-27T01:08:28.710 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-27T01:08:28.711 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-27T01:08:28.711 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:08:28.712 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-27T01:08:28.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[122008]: starting fuse 2024-01-27T01:08:28.771 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-27T01:08:28.771 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-27T01:08:28.771 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-27T01:08:28.771 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-27T01:08:28.771 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-27T01:08:28.771 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-27T01:08:28.771 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-27T01:08:28.771 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:08:28.771 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-27T01:08:28.771 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-27T01:08:28.772 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-27T01:08:28.772 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-27T01:08:28.772 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:08:28.772 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-27T01:08:28.772 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-27T01:08:28.772 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-27T01:08:28.772 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-27T01:08:28.772 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:08:28.772 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-27T01:08:28.772 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-27T01:08:28.772 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-27T01:08:28.772 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-27T01:08:28.772 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-27T01:08:28.772 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-27T01:08:28.772 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-27T01:08:28.772 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-27T01:08:28.773 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:08:28.773 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-01-27T01:08:28.773 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-27T01:08:28.910 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.122008.asok 2024-01-27T01:08:29.091 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.122008.asok 2024-01-27T01:08:29.091 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:08:29.091 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.122008.asok status 2024-01-27T01:08:29.266 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:08:29.266 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-27T01:08:29.266 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-27T01:08:29.266 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-27T01:08:29.266 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-01-27T01:08:29.266 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-27T01:08:29.266 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-27T01:08:29.267 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "122008", 2024-01-27T01:08:29.267 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-27T01:08:29.267 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:29.267 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-27T01:08:29.267 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-27T01:08:29.267 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 12513, 2024-01-27T01:08:29.267 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-27T01:08:29.267 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-27T01:08:29.267 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-27T01:08:29.267 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 12513 2024-01-27T01:08:29.267 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:29.267 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-27T01:08:29.267 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-27T01:08:29.267 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-27T01:08:29.267 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2520756984 2024-01-27T01:08:29.267 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:29.267 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:29.268 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-27T01:08:29.268 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-27T01:08:29.268 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-27T01:08:29.268 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2520756984 2024-01-27T01:08:29.268 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:29.268 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.12513 192.168.0.1:0/2520756984", 2024-01-27T01:08:29.268 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/2520756984", 2024-01-27T01:08:29.268 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-27T01:08:29.268 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 149, 2024-01-27T01:08:29.268 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 264, 2024-01-27T01:08:29.268 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-27T01:08:29.268 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-27T01:08:29.268 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-27T01:08:29.268 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:08:29.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:08:29.278 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-27T01:08:29.278 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-27T01:08:29.298 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-27T01:08:29.298 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-27T01:08:29.298 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:08:29.298 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-27T01:08:29.371 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-27T01:08:29.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.553+0000 7fdcf318d700 1 Processor -- start 2024-01-27T01:08:29.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.553+0000 7fdcf318d700 1 -- start start 2024-01-27T01:08:29.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdcf318d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcec15cd60 0x7fdcec0c3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:29.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdcf318d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdcec0c2240 0x7fdcec0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdcf318d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcec068220 con 0x7fdcec15d130 2024-01-27T01:08:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdcf318d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcec0c8af0 con 0x7fdcec0c2240 2024-01-27T01:08:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdcf318d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec067930 con 0x7fdcec15cd60 2024-01-27T01:08:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdcf172a700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdcec15d130 0x7fdcec0cb3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51770/0 (socket says 172.21.15.46:51770) 2024-01-27T01:08:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdcf172a700 1 -- 172.21.15.46:0/827810407 learned_addr learned my addr 172.21.15.46:0/827810407 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdcebfff700 1 --2- 172.21.15.46:0/827810407 >> v2:172.21.15.106:3300/0 conn(0x7fdcec0c2240 0x7fdcec0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdcf0f29700 1 --2- 172.21.15.46:0/827810407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcec15cd60 0x7fdcec0c3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdceb7fe700 1 -- 172.21.15.46:0/827810407 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251405715 0 0) 0x7fdcec068220 con 0x7fdcec15d130 2024-01-27T01:08:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdceb7fe700 1 -- 172.21.15.46:0/827810407 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd4003570 con 0x7fdcec15d130 2024-01-27T01:08:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdceb7fe700 1 -- 172.21.15.46:0/827810407 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3182019308 0 0) 0x7fdcd4003570 con 0x7fdcec15d130 2024-01-27T01:08:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdceb7fe700 1 -- 172.21.15.46:0/827810407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcec15cd60 msgr2=0x7fdcec0c3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdceb7fe700 1 --2- 172.21.15.46:0/827810407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcec15cd60 0x7fdcec0c3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdceb7fe700 1 -- 172.21.15.46:0/827810407 >> v2:172.21.15.106:3300/0 conn(0x7fdcec0c2240 msgr2=0x7fdcec0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdceb7fe700 1 --2- 172.21.15.46:0/827810407 >> v2:172.21.15.106:3300/0 conn(0x7fdcec0c2240 0x7fdcec0cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdceb7fe700 1 -- 172.21.15.46:0/827810407 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcec0c4210 con 0x7fdcec15d130 2024-01-27T01:08:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdceb7fe700 1 -- 172.21.15.46:0/827810407 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7fdce40017f0 con 0x7fdcec15d130 2024-01-27T01:08:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdceb7fe700 1 -- 172.21.15.46:0/827810407 >> v1:172.21.15.46:6789/0 conn(0x7fdcec15d130 legacy=0x7fdcec0cb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.554+0000 7fdceb7fe700 1 --2- 172.21.15.46:0/827810407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcec15cd60 0x7fdcd4003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdceb7fe700 1 --2- 172.21.15.46:0/827810407 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcd4003fe0 0x7fdcd4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdceb7fe700 1 --2- 172.21.15.46:0/827810407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd4004d50 0x7fdcd40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdceb7fe700 1 -- 172.21.15.46:0/827810407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec0c8af0 con 0x7fdcd4004d50 2024-01-27T01:08:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdceb7fe700 1 -- 172.21.15.46:0/827810407 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec067930 con 0x7fdcd4003fe0 2024-01-27T01:08:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdceb7fe700 1 -- 172.21.15.46:0/827810407 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd400ba90 con 0x7fdcec15cd60 2024-01-27T01:08:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdcf172a700 1 --2- 172.21.15.46:0/827810407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd4004d50 0x7fdcd40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdcf0f29700 1 --2- 172.21.15.46:0/827810407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcec15cd60 0x7fdcd4003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdcebfff700 1 --2- 172.21.15.46:0/827810407 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcd4003fe0 0x7fdcd4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdcf172a700 1 -- 172.21.15.46:0/827810407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcec15cd60 msgr2=0x7fdcd4003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdcf172a700 1 --2- 172.21.15.46:0/827810407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcec15cd60 0x7fdcd4003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdcf172a700 1 -- 172.21.15.46:0/827810407 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcd4003fe0 msgr2=0x7fdcd4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdcf172a700 1 --2- 172.21.15.46:0/827810407 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcd4003fe0 0x7fdcd4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdcf172a700 1 -- 172.21.15.46:0/827810407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd4003570 con 0x7fdcd4004d50 2024-01-27T01:08:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdcf0f29700 1 --2- 172.21.15.46:0/827810407 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcec15cd60 0x7fdcd4003a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdcebfff700 1 --2- 172.21.15.46:0/827810407 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcd4003fe0 0x7fdcd4004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdcf172a700 1 --2- 172.21.15.46:0/827810407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd4004d50 0x7fdcd40051c0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fdce400a930 tx=0x7fdce40021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc74b9c234f6c7c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdceb7fe700 1 -- 172.21.15.46:0/827810407 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce40070d0 con 0x7fdcd4004d50 2024-01-27T01:08:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdceb7fe700 1 -- 172.21.15.46:0/827810407 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce4007230 con 0x7fdcd4004d50 2024-01-27T01:08:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.555+0000 7fdceb7fe700 1 -- 172.21.15.46:0/827810407 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce40075d0 con 0x7fdcd4004d50 2024-01-27T01:08:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.556+0000 7fdcf318d700 1 -- 172.21.15.46:0/827810407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd4004d50 msgr2=0x7fdcd40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.556+0000 7fdcf318d700 1 --2- 172.21.15.46:0/827810407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd4004d50 0x7fdcd40051c0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fdce400a930 tx=0x7fdce40021d0 comp rx=0 tx=0).stop 2024-01-27T01:08:29.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.556+0000 7fdcf172a700 1 -- 172.21.15.46:0/827810407 reap_dead start 2024-01-27T01:08:29.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.556+0000 7fdcf318d700 1 -- 172.21.15.46:0/827810407 shutdown_connections 2024-01-27T01:08:29.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.556+0000 7fdcf318d700 1 -- 172.21.15.46:0/827810407 >> 172.21.15.46:0/827810407 conn(0x7fdcec069220 msgr2=0x7fdcec0cd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:29.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.556+0000 7fdcf318d700 1 -- 172.21.15.46:0/827810407 shutdown_connections 2024-01-27T01:08:29.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.556+0000 7fdcf318d700 1 -- 172.21.15.46:0/827810407 wait complete. 2024-01-27T01:08:29.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.557+0000 7fdcf318d700 1 Processor -- start 2024-01-27T01:08:29.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.557+0000 7fdcf318d700 1 -- start start 2024-01-27T01:08:29.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.558+0000 7fdcf318d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd4004d50 0x7fdcec157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:29.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.558+0000 7fdcf318d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcec0987b0 0x7fdcec15d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:29.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.558+0000 7fdcf318d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcec15b250 0x7fdcec15b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.558+0000 7fdcf318d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec15bc30 con 0x7fdcd4004d50 2024-01-27T01:08:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.558+0000 7fdcf318d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec15dfb0 con 0x7fdcec0987b0 2024-01-27T01:08:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.558+0000 7fdcf318d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec15e290 con 0x7fdcec15b250 2024-01-27T01:08:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.558+0000 7fdcf0f29700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd4004d50 0x7fdcec157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.558+0000 7fdcf0f29700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd4004d50 0x7fdcec157d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39154/0 (socket says 172.21.15.46:39154) 2024-01-27T01:08:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.558+0000 7fdcf0f29700 1 -- 172.21.15.46:0/3161677932 learned_addr learned my addr 172.21.15.46:0/3161677932 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.559+0000 7fdcebfff700 1 --2- 172.21.15.46:0/3161677932 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcec0987b0 0x7fdcec15d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.559+0000 7fdcf172a700 1 --2- 172.21.15.46:0/3161677932 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcec15b250 0x7fdcec15b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.559+0000 7fdcf0f29700 1 -- 172.21.15.46:0/3161677932 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcec15b250 msgr2=0x7fdcec15b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.559+0000 7fdcf0f29700 1 --2- 172.21.15.46:0/3161677932 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcec15b250 0x7fdcec15b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.559+0000 7fdcf0f29700 1 -- 172.21.15.46:0/3161677932 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcec0987b0 msgr2=0x7fdcec15d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.559+0000 7fdcf0f29700 1 --2- 172.21.15.46:0/3161677932 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcec0987b0 0x7fdcec15d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.559+0000 7fdcf0f29700 1 -- 172.21.15.46:0/3161677932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcec15e570 con 0x7fdcd4004d50 2024-01-27T01:08:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.559+0000 7fdcebfff700 1 --2- 172.21.15.46:0/3161677932 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcec0987b0 0x7fdcec15d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:08:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.559+0000 7fdcf0f29700 1 --2- 172.21.15.46:0/3161677932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd4004d50 0x7fdcec157d80 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc006e50 tx=0x7fdcdc009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.559+0000 7fdce8ff9700 1 -- 172.21.15.46:0/3161677932 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc011ec0 con 0x7fdcd4004d50 2024-01-27T01:08:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.559+0000 7fdcf318d700 1 -- 172.21.15.46:0/3161677932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcec169a80 con 0x7fdcd4004d50 2024-01-27T01:08:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.559+0000 7fdce8ff9700 1 -- 172.21.15.46:0/3161677932 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcdc00fcd0 con 0x7fdcd4004d50 2024-01-27T01:08:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.559+0000 7fdce8ff9700 1 -- 172.21.15.46:0/3161677932 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc005c30 con 0x7fdcd4004d50 2024-01-27T01:08:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.559+0000 7fdcf318d700 1 -- 172.21.15.46:0/3161677932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcec1582c0 con 0x7fdcd4004d50 2024-01-27T01:08:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.560+0000 7fdcf318d700 1 -- 172.21.15.46:0/3161677932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcec04c710 con 0x7fdcd4004d50 2024-01-27T01:08:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.561+0000 7fdce8ff9700 1 -- 172.21.15.46:0/3161677932 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 144) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fdcdc006050 con 0x7fdcd4004d50 2024-01-27T01:08:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.561+0000 7fdce8ff9700 1 --2- 172.21.15.46:0/3161677932 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdcd8067170 0x7fdcd8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.561+0000 7fdce8ff9700 1 -- 172.21.15.46:0/3161677932 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6560+0+0 (secure 0 0 0) 0x7fdcdc085540 con 0x7fdcd4004d50 2024-01-27T01:08:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.561+0000 7fdcebfff700 1 --2- 172.21.15.46:0/3161677932 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdcd8067170 0x7fdcd8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.562+0000 7fdcebfff700 1 --2- 172.21.15.46:0/3161677932 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdcd8067170 0x7fdcd8069620 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdcec0696e0 tx=0x7fdce0008040 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:29.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.566+0000 7fdce8ff9700 1 -- 172.21.15.46:0/3161677932 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdcdc0521d0 con 0x7fdcd4004d50 2024-01-27T01:08:29.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.805+0000 7fdcf318d700 1 -- 172.21.15.46:0/3161677932 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fdcec0c3280 con 0x7fdcd8067170 2024-01-27T01:08:29.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.807+0000 7fdce8ff9700 1 -- 172.21.15.46:0/3161677932 <== mgr.11614 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fdcec0c3280 con 0x7fdcd8067170 2024-01-27T01:08:29.806 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-01-27T01:08:29.806 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-01-27T01:08:29.806 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephfs" 2024-01-27T01:08:29.807 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:29.807 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-01-27T01:08:29.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.809+0000 7fdcf318d700 1 -- 172.21.15.46:0/3161677932 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdcd8067170 msgr2=0x7fdcd8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:29.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.809+0000 7fdcf318d700 1 --2- 172.21.15.46:0/3161677932 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdcd8067170 0x7fdcd8069620 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdcec0696e0 tx=0x7fdce0008040 comp rx=0 tx=0).stop 2024-01-27T01:08:29.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.809+0000 7fdcf318d700 1 -- 172.21.15.46:0/3161677932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd4004d50 msgr2=0x7fdcec157d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:29.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.809+0000 7fdcf318d700 1 --2- 172.21.15.46:0/3161677932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd4004d50 0x7fdcec157d80 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc006e50 tx=0x7fdcdc009120 comp rx=0 tx=0).stop 2024-01-27T01:08:29.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.809+0000 7fdcf318d700 1 -- 172.21.15.46:0/3161677932 shutdown_connections 2024-01-27T01:08:29.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.809+0000 7fdcf318d700 1 --2- 172.21.15.46:0/3161677932 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdcd8067170 0x7fdcd8069620 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:29.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.809+0000 7fdcf318d700 1 --2- 172.21.15.46:0/3161677932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd4004d50 0x7fdcec157d80 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:29.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.809+0000 7fdcf318d700 1 --2- 172.21.15.46:0/3161677932 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcec15b250 0x7fdcec15b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:29.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.809+0000 7fdcf318d700 1 --2- 172.21.15.46:0/3161677932 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcec0987b0 0x7fdcec15d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:29.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.809+0000 7fdcf318d700 1 -- 172.21.15.46:0/3161677932 >> 172.21.15.46:0/3161677932 conn(0x7fdcec069220 msgr2=0x7fdcec0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:29.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.810+0000 7fdcf318d700 1 -- 172.21.15.46:0/3161677932 shutdown_connections 2024-01-27T01:08:29.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.810+0000 7fdcf318d700 1 -- 172.21.15.46:0/3161677932 wait complete. 2024-01-27T01:08:29.822 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-27T01:08:29.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.956+0000 7fee510b8700 1 Processor -- start 2024-01-27T01:08:29.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.956+0000 7fee510b8700 1 -- start start 2024-01-27T01:08:29.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.957+0000 7fee510b8700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fee4c159cb0 0x7fee4c0cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.957+0000 7fee510b8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee4c0cf050 0x7fee4c0af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.957+0000 7fee510b8700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4c0cc100 con 0x7fee4c0cf4a0 2024-01-27T01:08:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.957+0000 7fee510b8700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee4c0cb850 con 0x7fee4c159cb0 2024-01-27T01:08:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.957+0000 7fee510b8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c0cbaf0 con 0x7fee4c0cf050 2024-01-27T01:08:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.957+0000 7fee4ad9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fee4c159cb0 0x7fee4c0cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.957+0000 7fee4ad9d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fee4c159cb0 0x7fee4c0cc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39162/0 (socket says 172.21.15.46:39162) 2024-01-27T01:08:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.957+0000 7fee4b59e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fee4c0cf4a0 0x7fee4c0c5530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51774/0 (socket says 172.21.15.46:51774) 2024-01-27T01:08:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.957+0000 7fee4ad9d700 1 -- 172.21.15.46:0/2788953821 learned_addr learned my addr 172.21.15.46:0/2788953821 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.957+0000 7fee4a59c700 1 --2- 172.21.15.46:0/2788953821 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee4c0cf050 0x7fee4c0af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.958+0000 7fee4ad9d700 1 -- 172.21.15.46:0/2788953821 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee4c0cf050 msgr2=0x7fee4c0af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.958+0000 7fee4ad9d700 1 --2- 172.21.15.46:0/2788953821 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee4c0cf050 0x7fee4c0af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.958+0000 7fee4ad9d700 1 -- 172.21.15.46:0/2788953821 >> v1:172.21.15.46:6789/0 conn(0x7fee4c0cf4a0 legacy=0x7fee4c0c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.958+0000 7fee4ad9d700 1 -- 172.21.15.46:0/2788953821 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c0cfc80 con 0x7fee4c159cb0 2024-01-27T01:08:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.958+0000 7fee49d9b700 1 -- 172.21.15.46:0/2788953821 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548144766 0 0) 0x7fee4c0cc100 con 0x7fee4c0cf4a0 2024-01-27T01:08:29.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.958+0000 7fee4ad9d700 1 --2- 172.21.15.46:0/2788953821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee4c159cb0 0x7fee4c0cc590 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009990 tx=0x7fee3c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75cfbfdb5d3c890a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:29.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.958+0000 7fee49d9b700 1 -- 172.21.15.46:0/2788953821 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3c009e80 con 0x7fee4c159cb0 2024-01-27T01:08:29.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.958+0000 7fee49d9b700 1 -- 172.21.15.46:0/2788953821 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee3c005c10 con 0x7fee4c159cb0 2024-01-27T01:08:29.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.958+0000 7fee49d9b700 1 -- 172.21.15.46:0/2788953821 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3c00b650 con 0x7fee4c159cb0 2024-01-27T01:08:29.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.958+0000 7fee510b8700 1 -- 172.21.15.46:0/2788953821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee4c159cb0 msgr2=0x7fee4c0cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:29.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.958+0000 7fee510b8700 1 --2- 172.21.15.46:0/2788953821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee4c159cb0 0x7fee4c0cc590 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009990 tx=0x7fee3c009220 comp rx=0 tx=0).stop 2024-01-27T01:08:29.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.959+0000 7fee510b8700 1 -- 172.21.15.46:0/2788953821 shutdown_connections 2024-01-27T01:08:29.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.959+0000 7fee510b8700 1 --2- 172.21.15.46:0/2788953821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee4c159cb0 0x7fee4c0cc590 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:29.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.959+0000 7fee510b8700 1 --2- 172.21.15.46:0/2788953821 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee4c0cf050 0x7fee4c0af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:29.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.959+0000 7fee510b8700 1 -- 172.21.15.46:0/2788953821 >> 172.21.15.46:0/2788953821 conn(0x7fee4c087f60 msgr2=0x7fee4c06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:29.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.959+0000 7fee510b8700 1 -- 172.21.15.46:0/2788953821 shutdown_connections 2024-01-27T01:08:29.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.959+0000 7fee510b8700 1 -- 172.21.15.46:0/2788953821 wait complete. 2024-01-27T01:08:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.960+0000 7fee510b8700 1 Processor -- start 2024-01-27T01:08:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.960+0000 7fee510b8700 1 -- start start 2024-01-27T01:08:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.961+0000 7fee510b8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee4c0cf050 0x7fee4c15c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.961+0000 7fee510b8700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee4c0cf4a0 0x7fee4c147fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.961+0000 7fee510b8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee4c159cb0 0x7fee4c148500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.961+0000 7fee510b8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c15fa80 con 0x7fee4c159cb0 2024-01-27T01:08:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.961+0000 7fee510b8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c15eeb0 con 0x7fee4c0cf050 2024-01-27T01:08:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.961+0000 7fee510b8700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c15f150 con 0x7fee4c0cf4a0 2024-01-27T01:08:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.961+0000 7fee4b59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee4c159cb0 0x7fee4c148500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.961+0000 7fee4b59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee4c159cb0 0x7fee4c148500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39168/0 (socket says 172.21.15.46:39168) 2024-01-27T01:08:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.961+0000 7fee4b59e700 1 -- 172.21.15.46:0/1987065584 learned_addr learned my addr 172.21.15.46:0/1987065584 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.961+0000 7fee4a59c700 1 --2- 172.21.15.46:0/1987065584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee4c0cf4a0 0x7fee4c147fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.961+0000 7fee4ad9d700 1 --2- 172.21.15.46:0/1987065584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee4c0cf050 0x7fee4c15c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.961+0000 7fee4b59e700 1 -- 172.21.15.46:0/1987065584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee4c0cf4a0 msgr2=0x7fee4c147fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.961+0000 7fee4b59e700 1 --2- 172.21.15.46:0/1987065584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee4c0cf4a0 0x7fee4c147fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.961+0000 7fee4b59e700 1 -- 172.21.15.46:0/1987065584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee4c0cf050 msgr2=0x7fee4c15c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.961+0000 7fee4b59e700 1 --2- 172.21.15.46:0/1987065584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee4c0cf050 0x7fee4c15c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.961+0000 7fee4b59e700 1 -- 172.21.15.46:0/1987065584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3c009480 con 0x7fee4c159cb0 2024-01-27T01:08:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.961+0000 7fee4b59e700 1 --2- 172.21.15.46:0/1987065584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee4c159cb0 0x7fee4c148500 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7fee4400cd00 tx=0x7fee44001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.962+0000 7fee32ffd700 1 -- 172.21.15.46:0/1987065584 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4400ce60 con 0x7fee4c159cb0 2024-01-27T01:08:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.962+0000 7fee32ffd700 1 -- 172.21.15.46:0/1987065584 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee44005df0 con 0x7fee4c159cb0 2024-01-27T01:08:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.962+0000 7fee510b8700 1 -- 172.21.15.46:0/1987065584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee4c15f430 con 0x7fee4c159cb0 2024-01-27T01:08:29.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.962+0000 7fee32ffd700 1 -- 172.21.15.46:0/1987065584 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4400aae0 con 0x7fee4c159cb0 2024-01-27T01:08:29.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.962+0000 7fee510b8700 1 -- 172.21.15.46:0/1987065584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4c156190 con 0x7fee4c159cb0 2024-01-27T01:08:29.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.963+0000 7fee510b8700 1 -- 172.21.15.46:0/1987065584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4c15fbf0 con 0x7fee4c159cb0 2024-01-27T01:08:29.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.963+0000 7fee32ffd700 1 -- 172.21.15.46:0/1987065584 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 144) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fee440070f0 con 0x7fee4c159cb0 2024-01-27T01:08:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.964+0000 7fee32ffd700 1 --2- 172.21.15.46:0/1987065584 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee38067170 0x7fee38069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.964+0000 7fee32ffd700 1 -- 172.21.15.46:0/1987065584 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6560+0+0 (secure 0 0 0) 0x7fee44084c20 con 0x7fee4c159cb0 2024-01-27T01:08:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.964+0000 7fee4ad9d700 1 --2- 172.21.15.46:0/1987065584 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee38067170 0x7fee38069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.965+0000 7fee4ad9d700 1 --2- 172.21.15.46:0/1987065584 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee38067170 0x7fee38069620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009260 tx=0x7fee3c016000 comp rx=0 tx=0).ready entity=mgr.11614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:29.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:29.969+0000 7fee32ffd700 1 -- 172.21.15.46:0/1987065584 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fee44051930 con 0x7fee4c159cb0 2024-01-27T01:08:30.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.232+0000 7fee510b8700 1 -- 172.21.15.46:0/1987065584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fee4c04c710 con 0x7fee4c159cb0 2024-01-27T01:08:30.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.418+0000 7fee32ffd700 1 -- 172.21.15.46:0/1987065584 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 145) v1 ==== 84963+0+0 (secure 0 0 0) 0x7fee44051d00 con 0x7fee4c159cb0 2024-01-27T01:08:30.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.420+0000 7fee32ffd700 1 -- 172.21.15.46:0/1987065584 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v145) v1 ==== 92+0+0 (secure 0 0 0) 0x7fee44051930 con 0x7fee4c159cb0 2024-01-27T01:08:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.423+0000 7fee510b8700 1 -- 172.21.15.46:0/1987065584 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee38067170 msgr2=0x7fee38069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.423+0000 7fee510b8700 1 --2- 172.21.15.46:0/1987065584 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee38067170 0x7fee38069620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009260 tx=0x7fee3c016000 comp rx=0 tx=0).stop 2024-01-27T01:08:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.423+0000 7fee510b8700 1 -- 172.21.15.46:0/1987065584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee4c159cb0 msgr2=0x7fee4c148500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.423+0000 7fee510b8700 1 --2- 172.21.15.46:0/1987065584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee4c159cb0 0x7fee4c148500 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7fee4400cd00 tx=0x7fee44001ac0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.423+0000 7fee510b8700 1 -- 172.21.15.46:0/1987065584 shutdown_connections 2024-01-27T01:08:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.423+0000 7fee510b8700 1 --2- 172.21.15.46:0/1987065584 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fee38067170 0x7fee38069620 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.423+0000 7fee510b8700 1 --2- 172.21.15.46:0/1987065584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee4c159cb0 0x7fee4c148500 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.423+0000 7fee510b8700 1 --2- 172.21.15.46:0/1987065584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee4c0cf050 0x7fee4c15c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.423+0000 7fee510b8700 1 --2- 172.21.15.46:0/1987065584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fee4c0cf4a0 0x7fee4c147fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.423+0000 7fee510b8700 1 -- 172.21.15.46:0/1987065584 >> 172.21.15.46:0/1987065584 conn(0x7fee4c087f60 msgr2=0x7fee4c0c5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:30.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.423+0000 7fee510b8700 1 -- 172.21.15.46:0/1987065584 shutdown_connections 2024-01-27T01:08:30.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.423+0000 7fee510b8700 1 -- 172.21.15.46:0/1987065584 wait complete. 2024-01-27T01:08:30.438 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-27T01:08:30.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.579+0000 7f8ea8735700 1 Processor -- start 2024-01-27T01:08:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.579+0000 7f8ea8735700 1 -- start start 2024-01-27T01:08:30.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.580+0000 7f8ea8735700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8ea00cf050 0x7f8ea00af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:30.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.581+0000 7f8ea8735700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea00cc100 con 0x7f8ea00cf4a0 2024-01-27T01:08:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.581+0000 7f8ea8735700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea00cb850 con 0x7f8ea0159cb0 2024-01-27T01:08:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.581+0000 7f8ea8735700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea00ccca0 con 0x7f8ea00cf050 2024-01-27T01:08:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.581+0000 7f8ea6cd2700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8ea00cf4a0 0x7f8ea00c5530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43450/0 (socket says 172.21.15.46:43450) 2024-01-27T01:08:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.581+0000 7f8ea6cd2700 1 -- 172.21.15.46:0/990874212 learned_addr learned my addr 172.21.15.46:0/990874212 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.581+0000 7f8ea5cd0700 1 --2- 172.21.15.46:0/990874212 >> v2:172.21.15.106:3300/0 conn(0x7f8ea00cf050 0x7f8ea00af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.581+0000 7f8ea54cf700 1 -- 172.21.15.46:0/990874212 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510273102 0 0) 0x7f8ea00cc100 con 0x7f8ea00cf4a0 2024-01-27T01:08:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.581+0000 7f8ea54cf700 1 -- 172.21.15.46:0/990874212 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e88003570 con 0x7f8ea00cf4a0 2024-01-27T01:08:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.581+0000 7f8ea5cd0700 1 -- 172.21.15.46:0/990874212 >> v1:172.21.15.106:6789/0 conn(0x7f8ea0159cb0 legacy=0x7f8ea00cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.581+0000 7f8ea5cd0700 1 -- 172.21.15.46:0/990874212 >> v1:172.21.15.46:6789/0 conn(0x7f8ea00cf4a0 legacy=0x7f8ea00c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.581+0000 7f8ea5cd0700 1 -- 172.21.15.46:0/990874212 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea00cbaf0 con 0x7f8ea00cf050 2024-01-27T01:08:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.581+0000 7f8ea5cd0700 1 --2- 172.21.15.46:0/990874212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ea00cf050 0x7f8ea00af430 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f8e940098e0 tx=0x7f8e940091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d37de4de8e9825a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.582+0000 7f8ea54cf700 1 -- 172.21.15.46:0/990874212 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9400b640 con 0x7f8ea00cf050 2024-01-27T01:08:30.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.582+0000 7f8ea54cf700 1 -- 172.21.15.46:0/990874212 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9400b7a0 con 0x7f8ea00cf050 2024-01-27T01:08:30.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.582+0000 7f8ea54cf700 1 -- 172.21.15.46:0/990874212 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9400bb40 con 0x7f8ea00cf050 2024-01-27T01:08:30.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.582+0000 7f8ea8735700 1 -- 172.21.15.46:0/990874212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ea00cf050 msgr2=0x7f8ea00af430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:30.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.582+0000 7f8ea8735700 1 --2- 172.21.15.46:0/990874212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ea00cf050 0x7f8ea00af430 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f8e940098e0 tx=0x7f8e940091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.582+0000 7f8ea8735700 1 -- 172.21.15.46:0/990874212 shutdown_connections 2024-01-27T01:08:30.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.582+0000 7f8ea8735700 1 --2- 172.21.15.46:0/990874212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ea00cf050 0x7f8ea00af430 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.582+0000 7f8ea8735700 1 -- 172.21.15.46:0/990874212 >> 172.21.15.46:0/990874212 conn(0x7f8ea0087f60 msgr2=0x7f8ea006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:30.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.582+0000 7f8ea8735700 1 -- 172.21.15.46:0/990874212 shutdown_connections 2024-01-27T01:08:30.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.582+0000 7f8ea8735700 1 -- 172.21.15.46:0/990874212 wait complete. 2024-01-27T01:08:30.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.584+0000 7f8ea8735700 1 Processor -- start 2024-01-27T01:08:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.584+0000 7f8ea8735700 1 -- start start 2024-01-27T01:08:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.584+0000 7f8ea8735700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ea00cf050 0x7f8ea00c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.584+0000 7f8ea8735700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ea00cf4a0 0x7f8ea00c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.584+0000 7f8ea8735700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ea0159cb0 0x7f8ea015d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.584+0000 7f8ea8735700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00ca480 con 0x7f8ea00cf4a0 2024-01-27T01:08:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.584+0000 7f8ea8735700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea015dae0 con 0x7f8ea0159cb0 2024-01-27T01:08:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.584+0000 7f8ea8735700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00c9ab0 con 0x7f8ea00cf050 2024-01-27T01:08:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.585+0000 7f8ea5cd0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ea00cf4a0 0x7f8ea00c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.585+0000 7f8ea5cd0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ea00cf4a0 0x7f8ea00c9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34784/0 (socket says 172.21.15.46:34784) 2024-01-27T01:08:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.585+0000 7f8ea5cd0700 1 -- 172.21.15.46:0/1621983582 learned_addr learned my addr 172.21.15.46:0/1621983582 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.585+0000 7f8ea6cd2700 1 --2- 172.21.15.46:0/1621983582 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ea0159cb0 0x7f8ea015d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.585+0000 7f8ea64d1700 1 --2- 172.21.15.46:0/1621983582 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ea00cf050 0x7f8ea00c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.585+0000 7f8ea64d1700 1 -- 172.21.15.46:0/1621983582 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ea0159cb0 msgr2=0x7f8ea015d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.585+0000 7f8ea64d1700 1 --2- 172.21.15.46:0/1621983582 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ea0159cb0 0x7f8ea015d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.585+0000 7f8ea64d1700 1 -- 172.21.15.46:0/1621983582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ea00cf4a0 msgr2=0x7f8ea00c9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.585+0000 7f8ea64d1700 1 --2- 172.21.15.46:0/1621983582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ea00cf4a0 0x7f8ea00c9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.585+0000 7f8ea64d1700 1 -- 172.21.15.46:0/1621983582 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e94009420 con 0x7f8ea00cf050 2024-01-27T01:08:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.585+0000 7f8ea64d1700 1 --2- 172.21.15.46:0/1621983582 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ea00cf050 0x7f8ea00c8810 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8e9000cca0 tx=0x7f8e90001ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.585+0000 7f8e9e7fc700 1 -- 172.21.15.46:0/1621983582 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9000a890 con 0x7f8ea00cf050 2024-01-27T01:08:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.586+0000 7f8e9e7fc700 1 -- 172.21.15.46:0/1621983582 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9000aa10 con 0x7f8ea00cf050 2024-01-27T01:08:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.586+0000 7f8ea8735700 1 -- 172.21.15.46:0/1621983582 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea00c8ea0 con 0x7f8ea00cf050 2024-01-27T01:08:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.586+0000 7f8e9e7fc700 1 -- 172.21.15.46:0/1621983582 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9000add0 con 0x7f8ea00cf050 2024-01-27T01:08:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.586+0000 7f8ea8735700 1 -- 172.21.15.46:0/1621983582 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea00c9460 con 0x7f8ea00cf050 2024-01-27T01:08:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.586+0000 7f8ea8735700 1 -- 172.21.15.46:0/1621983582 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea00c9d90 con 0x7f8ea00cf050 2024-01-27T01:08:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.587+0000 7f8e9e7fc700 1 -- 172.21.15.46:0/1621983582 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 145) v1 ==== 84963+0+0 (secure 0 0 0) 0x7f8e90012630 con 0x7f8ea00cf050 2024-01-27T01:08:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.588+0000 7f8e9e7fc700 1 --2- 172.21.15.46:0/1621983582 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8e8c0671c0 0x7f8e8c069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.588+0000 7f8ea5cd0700 1 -- 172.21.15.46:0/1621983582 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8e8c0671c0 msgr2=0x7f8e8c069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.588+0000 7f8e9e7fc700 1 -- 172.21.15.46:0/1621983582 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f8e90085170 con 0x7f8ea00cf050 2024-01-27T01:08:30.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.588+0000 7f8ea5cd0700 1 --2- 172.21.15.46:0/1621983582 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8e8c0671c0 0x7f8e8c069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:30.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.592+0000 7f8e9e7fc700 1 -- 172.21.15.46:0/1621983582 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8e90051e00 con 0x7f8ea00cf050 2024-01-27T01:08:30.757 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:30.757+0000 7fbef6a741c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:08:30.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.789+0000 7f8ea5cd0700 1 -- 172.21.15.46:0/1621983582 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8e8c0671c0 msgr2=0x7f8e8c069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:30.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.789+0000 7f8ea5cd0700 1 --2- 172.21.15.46:0/1621983582 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8e8c0671c0 0x7f8e8c069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:08:30.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.800+0000 7f8ea8735700 1 -- 172.21.15.46:0/1621983582 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f8ea0063980 con 0x7f8ea00cf050 2024-01-27T01:08:30.802 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:30.803+0000 7f58747b21c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:08:30.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.806+0000 7f8e9e7fc700 1 -- 172.21.15.46:0/1621983582 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v89)=0 v89) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8e90055420 con 0x7f8ea00cf050 2024-01-27T01:08:30.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.808+0000 7f8ea8735700 1 -- 172.21.15.46:0/1621983582 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8e8c0671c0 msgr2=0x7f8e8c069670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:30.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.808+0000 7f8ea8735700 1 --2- 172.21.15.46:0/1621983582 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8e8c0671c0 0x7f8e8c069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.808+0000 7f8ea8735700 1 -- 172.21.15.46:0/1621983582 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ea00cf050 msgr2=0x7f8ea00c8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:30.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.808+0000 7f8ea8735700 1 --2- 172.21.15.46:0/1621983582 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ea00cf050 0x7f8ea00c8810 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8e9000cca0 tx=0x7f8e90001ab0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.808+0000 7f8ea8735700 1 -- 172.21.15.46:0/1621983582 shutdown_connections 2024-01-27T01:08:30.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.808+0000 7f8ea8735700 1 --2- 172.21.15.46:0/1621983582 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8e8c0671c0 0x7f8e8c069670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.808+0000 7f8ea8735700 1 --2- 172.21.15.46:0/1621983582 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ea0159cb0 0x7f8ea015d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.808+0000 7f8ea8735700 1 --2- 172.21.15.46:0/1621983582 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ea00cf050 0x7f8ea00c8810 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.808+0000 7f8ea8735700 1 --2- 172.21.15.46:0/1621983582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ea00cf4a0 0x7f8ea00c9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.808+0000 7f8ea8735700 1 -- 172.21.15.46:0/1621983582 >> 172.21.15.46:0/1621983582 conn(0x7f8ea0087f60 msgr2=0x7f8ea006a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:30.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.808+0000 7f8ea8735700 1 -- 172.21.15.46:0/1621983582 shutdown_connections 2024-01-27T01:08:30.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.808+0000 7f8ea8735700 1 -- 172.21.15.46:0/1621983582 wait complete. 2024-01-27T01:08:30.822 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-27T01:08:30.835 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:30.835+0000 7fbef6a741c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:08:30.893 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:30.894+0000 7f58747b21c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:08:30.905 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:30.906+0000 7fbef6a741c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:08:30.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.958+0000 7f648602f700 1 Processor -- start 2024-01-27T01:08:30.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.958+0000 7f648602f700 1 -- start start 2024-01-27T01:08:30.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.959+0000 7f648602f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64800aef60 0x7f64800c8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:30.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.959+0000 7f648602f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64800c87c0 con 0x7f64800c9190 2024-01-27T01:08:30.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.959+0000 7f648602f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64800c7bb0 con 0x7f64800af330 2024-01-27T01:08:30.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.959+0000 7f648602f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64800c7e90 con 0x7f64800aef60 2024-01-27T01:08:30.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.959+0000 7f6476ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f64800c9190 0x7f64800c7160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43466/0 (socket says 172.21.15.46:43466) 2024-01-27T01:08:30.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.959+0000 7f6476ffd700 1 -- 172.21.15.46:0/2324943391 learned_addr learned my addr 172.21.15.46:0/2324943391 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:30.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.959+0000 7f647f7fe700 1 --2- 172.21.15.46:0/2324943391 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64800aef60 0x7f64800c8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:30.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.960+0000 7f647effd700 1 -- 172.21.15.46:0/2324943391 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139249775 0 0) 0x7f64800c87c0 con 0x7f64800c9190 2024-01-27T01:08:30.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.960+0000 7f647effd700 1 -- 172.21.15.46:0/2324943391 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f646c003570 con 0x7f64800c9190 2024-01-27T01:08:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.960+0000 7f647effd700 1 -- 172.21.15.46:0/2324943391 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349535442 0 0) 0x7f64800c7bb0 con 0x7f64800af330 2024-01-27T01:08:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.960+0000 7f647effd700 1 -- 172.21.15.46:0/2324943391 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64800c87c0 con 0x7f64800af330 2024-01-27T01:08:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.960+0000 7f647f7fe700 1 -- 172.21.15.46:0/2324943391 >> v1:172.21.15.106:6789/0 conn(0x7f64800af330 legacy=0x7f6480068940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.960+0000 7f647f7fe700 1 -- 172.21.15.46:0/2324943391 >> v1:172.21.15.46:6789/0 conn(0x7f64800c9190 legacy=0x7f64800c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.960+0000 7f647f7fe700 1 -- 172.21.15.46:0/2324943391 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64800c8170 con 0x7f64800aef60 2024-01-27T01:08:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.960+0000 7f647f7fe700 1 --2- 172.21.15.46:0/2324943391 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64800aef60 0x7f64800c8c50 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f6470009900 tx=0x7f6470009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e538c640cfe00d67 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.960+0000 7f647effd700 1 -- 172.21.15.46:0/2324943391 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470005c10 con 0x7f64800aef60 2024-01-27T01:08:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.960+0000 7f647effd700 1 -- 172.21.15.46:0/2324943391 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6470005d70 con 0x7f64800aef60 2024-01-27T01:08:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.960+0000 7f647effd700 1 -- 172.21.15.46:0/2324943391 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470006260 con 0x7f64800aef60 2024-01-27T01:08:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.961+0000 7f648602f700 1 -- 172.21.15.46:0/2324943391 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64800aef60 msgr2=0x7f64800c8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.961+0000 7f648602f700 1 --2- 172.21.15.46:0/2324943391 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64800aef60 0x7f64800c8c50 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f6470009900 tx=0x7f6470009120 comp rx=0 tx=0).stop 2024-01-27T01:08:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.961+0000 7f648602f700 1 -- 172.21.15.46:0/2324943391 shutdown_connections 2024-01-27T01:08:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.961+0000 7f648602f700 1 --2- 172.21.15.46:0/2324943391 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64800aef60 0x7f64800c8c50 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.961+0000 7f648602f700 1 -- 172.21.15.46:0/2324943391 >> 172.21.15.46:0/2324943391 conn(0x7f64800ca100 msgr2=0x7f648006e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.961+0000 7f648602f700 1 -- 172.21.15.46:0/2324943391 shutdown_connections 2024-01-27T01:08:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.961+0000 7f648602f700 1 -- 172.21.15.46:0/2324943391 wait complete. 2024-01-27T01:08:30.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.963+0000 7f648602f700 1 Processor -- start 2024-01-27T01:08:30.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.963+0000 7f648602f700 1 -- start start 2024-01-27T01:08:30.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.964+0000 7f648602f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64800aef60 0x7f64800c6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.964+0000 7f648602f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64800af330 0x7f64800aa2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.964+0000 7f648602f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64800c9190 0x7f6480154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.964+0000 7f648602f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648006b2b0 con 0x7f64800aef60 2024-01-27T01:08:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.964+0000 7f648602f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648006a3c0 con 0x7f64800c9190 2024-01-27T01:08:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.964+0000 7f648602f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648006a6a0 con 0x7f64800af330 2024-01-27T01:08:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.964+0000 7f647f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64800aef60 0x7f64800c6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.964+0000 7f6476ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64800af330 0x7f64800aa2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.964+0000 7f647f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64800aef60 0x7f64800c6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34790/0 (socket says 172.21.15.46:34790) 2024-01-27T01:08:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.964+0000 7f647f7fe700 1 -- 172.21.15.46:0/3238697790 learned_addr learned my addr 172.21.15.46:0/3238697790 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.964+0000 7f647ffff700 1 --2- 172.21.15.46:0/3238697790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64800c9190 0x7f6480154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.965+0000 7f647f7fe700 1 -- 172.21.15.46:0/3238697790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64800af330 msgr2=0x7f64800aa2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.965+0000 7f647f7fe700 1 --2- 172.21.15.46:0/3238697790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64800af330 0x7f64800aa2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.965+0000 7f647f7fe700 1 -- 172.21.15.46:0/3238697790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64800c9190 msgr2=0x7f6480154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.965+0000 7f647f7fe700 1 --2- 172.21.15.46:0/3238697790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64800c9190 0x7f6480154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.965+0000 7f647f7fe700 1 -- 172.21.15.46:0/3238697790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64700093c0 con 0x7f64800aef60 2024-01-27T01:08:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.965+0000 7f6476ffd700 1 --2- 172.21.15.46:0/3238697790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64800af330 0x7f64800aa2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:30.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.965+0000 7f647f7fe700 1 --2- 172.21.15.46:0/3238697790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64800aef60 0x7f64800c6280 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f6470009390 tx=0x7f6470016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:30.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.965+0000 7f6477fff700 1 -- 172.21.15.46:0/3238697790 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647001d800 con 0x7f64800aef60 2024-01-27T01:08:30.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.965+0000 7f6477fff700 1 -- 172.21.15.46:0/3238697790 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647001d960 con 0x7f64800aef60 2024-01-27T01:08:30.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.965+0000 7f648602f700 1 -- 172.21.15.46:0/3238697790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648006a980 con 0x7f64800aef60 2024-01-27T01:08:30.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.965+0000 7f6477fff700 1 -- 172.21.15.46:0/3238697790 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647001dd20 con 0x7f64800aef60 2024-01-27T01:08:30.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.965+0000 7f648602f700 1 -- 172.21.15.46:0/3238697790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648006af40 con 0x7f64800aef60 2024-01-27T01:08:30.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.966+0000 7f648602f700 1 -- 172.21.15.46:0/3238697790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648004c710 con 0x7f64800aef60 2024-01-27T01:08:30.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.967+0000 7f6477fff700 1 -- 172.21.15.46:0/3238697790 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 145) v1 ==== 84963+0+0 (secure 0 0 0) 0x7f647001b690 con 0x7f64800aef60 2024-01-27T01:08:30.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.967+0000 7f6477fff700 1 --2- 172.21.15.46:0/3238697790 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f64600671c0 0x7f6460069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:30.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.967+0000 7f6477fff700 1 -- 172.21.15.46:0/3238697790 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f6470086830 con 0x7f64800aef60 2024-01-27T01:08:30.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.967+0000 7f6476ffd700 1 -- 172.21.15.46:0/3238697790 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f64600671c0 msgr2=0x7f6460069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:30.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.967+0000 7f6476ffd700 1 --2- 172.21.15.46:0/3238697790 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f64600671c0 0x7f6460069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:30.970 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:30.971+0000 7fbef6a741c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:08:30.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:30.972+0000 7f6477fff700 1 -- 172.21.15.46:0/3238697790 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f64700534c0 con 0x7f64800aef60 2024-01-27T01:08:30.977 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:30.978+0000 7f58747b21c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:08:31.053 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:31.054+0000 7f58747b21c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:08:31.126 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:31.126+0000 7fbef6a741c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:08:31.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.168+0000 7f6476ffd700 1 -- 172.21.15.46:0/3238697790 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f64600671c0 msgr2=0x7f6460069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:31.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.168+0000 7f6476ffd700 1 --2- 172.21.15.46:0/3238697790 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f64600671c0 0x7f6460069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:08:31.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.183+0000 7f648602f700 1 -- 172.21.15.46:0/3238697790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f64800c8450 con 0x7f64800aef60 2024-01-27T01:08:31.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.186+0000 7f6477fff700 1 -- 172.21.15.46:0/3238697790 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v90)=0 v90) v1 ==== 133+0+0 (secure 0 0 0) 0x7f647002a020 con 0x7f64800aef60 2024-01-27T01:08:31.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.189+0000 7f648602f700 1 -- 172.21.15.46:0/3238697790 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f64600671c0 msgr2=0x7f6460069670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:31.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.189+0000 7f648602f700 1 --2- 172.21.15.46:0/3238697790 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f64600671c0 0x7f6460069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:31.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.189+0000 7f648602f700 1 -- 172.21.15.46:0/3238697790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64800aef60 msgr2=0x7f64800c6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:31.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.189+0000 7f648602f700 1 --2- 172.21.15.46:0/3238697790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64800aef60 0x7f64800c6280 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f6470009390 tx=0x7f6470016000 comp rx=0 tx=0).stop 2024-01-27T01:08:31.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.189+0000 7f648602f700 1 -- 172.21.15.46:0/3238697790 shutdown_connections 2024-01-27T01:08:31.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.189+0000 7f648602f700 1 --2- 172.21.15.46:0/3238697790 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f64600671c0 0x7f6460069670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:31.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.189+0000 7f648602f700 1 --2- 172.21.15.46:0/3238697790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64800aef60 0x7f64800c6280 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:31.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.189+0000 7f648602f700 1 --2- 172.21.15.46:0/3238697790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64800c9190 0x7f6480154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:31.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.189+0000 7f648602f700 1 --2- 172.21.15.46:0/3238697790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f64800af330 0x7f64800aa2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:31.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.189+0000 7f648602f700 1 -- 172.21.15.46:0/3238697790 >> 172.21.15.46:0/3238697790 conn(0x7f64800ca100 msgr2=0x7f64800c7140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:31.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.189+0000 7f648602f700 1 -- 172.21.15.46:0/3238697790 shutdown_connections 2024-01-27T01:08:31.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.189+0000 7f648602f700 1 -- 172.21.15.46:0/3238697790 wait complete. 2024-01-27T01:08:31.202 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:08:31.202 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-27T01:08:31.226 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-01-27T01:08:31.232 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:31.233+0000 7f58747b21c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:08:31.264 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:31.265+0000 7fbef6a741c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:08:31.331 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:31.331+0000 7fbef6a741c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:08:31.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.369+0000 7efced149700 1 Processor -- start 2024-01-27T01:08:31.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.369+0000 7efced149700 1 -- start start 2024-01-27T01:08:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.370+0000 7efced149700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efce81597d0 0x7efce806ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.370+0000 7efced149700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce80cbfb0 con 0x7efce815ad40 2024-01-27T01:08:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.370+0000 7efced149700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce815a8b0 con 0x7efce80cb740 2024-01-27T01:08:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.370+0000 7efced149700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce80cbcd0 con 0x7efce81597d0 2024-01-27T01:08:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.370+0000 7efce659c700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efce815ad40 0x7efce80cb030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43476/0 (socket says 172.21.15.46:43476) 2024-01-27T01:08:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.370+0000 7efce659c700 1 -- 172.21.15.46:0/2059262298 learned_addr learned my addr 172.21.15.46:0/2059262298 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.370+0000 7efce6d9d700 1 --2- 172.21.15.46:0/2059262298 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efce81597d0 0x7efce806ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.370+0000 7efce5d9b700 1 -- 172.21.15.46:0/2059262298 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170465650 0 0) 0x7efce80cbfb0 con 0x7efce815ad40 2024-01-27T01:08:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.370+0000 7efce5d9b700 1 -- 172.21.15.46:0/2059262298 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcd4003570 con 0x7efce815ad40 2024-01-27T01:08:31.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.370+0000 7efce5d9b700 1 -- 172.21.15.46:0/2059262298 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711183833 0 0) 0x7efce815a8b0 con 0x7efce80cb740 2024-01-27T01:08:31.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.371+0000 7efce6d9d700 1 -- 172.21.15.46:0/2059262298 >> v1:172.21.15.106:6789/0 conn(0x7efce80cb740 legacy=0x7efce80c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:31.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.371+0000 7efce6d9d700 1 -- 172.21.15.46:0/2059262298 >> v1:172.21.15.46:6789/0 conn(0x7efce815ad40 legacy=0x7efce80cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:31.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.371+0000 7efce6d9d700 1 -- 172.21.15.46:0/2059262298 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce815a260 con 0x7efce81597d0 2024-01-27T01:08:31.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.371+0000 7efce6d9d700 1 --2- 172.21.15.46:0/2059262298 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efce81597d0 0x7efce806ec50 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7efcd8009e90 tx=0x7efcd8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76829fff99a3c029 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:31.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.371+0000 7efce5d9b700 1 -- 172.21.15.46:0/2059262298 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd800e070 con 0x7efce81597d0 2024-01-27T01:08:31.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.371+0000 7efce5d9b700 1 -- 172.21.15.46:0/2059262298 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd800bc30 con 0x7efce81597d0 2024-01-27T01:08:31.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.371+0000 7efce5d9b700 1 -- 172.21.15.46:0/2059262298 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd8012430 con 0x7efce81597d0 2024-01-27T01:08:31.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.372+0000 7efced149700 1 -- 172.21.15.46:0/2059262298 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efce81597d0 msgr2=0x7efce806ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:31.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.372+0000 7efced149700 1 --2- 172.21.15.46:0/2059262298 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efce81597d0 0x7efce806ec50 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7efcd8009e90 tx=0x7efcd8009780 comp rx=0 tx=0).stop 2024-01-27T01:08:31.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.372+0000 7efced149700 1 -- 172.21.15.46:0/2059262298 shutdown_connections 2024-01-27T01:08:31.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.372+0000 7efced149700 1 --2- 172.21.15.46:0/2059262298 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efce81597d0 0x7efce806ec50 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:31.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.372+0000 7efced149700 1 -- 172.21.15.46:0/2059262298 >> 172.21.15.46:0/2059262298 conn(0x7efce80ca9a0 msgr2=0x7efce8068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:31.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.372+0000 7efced149700 1 -- 172.21.15.46:0/2059262298 shutdown_connections 2024-01-27T01:08:31.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.372+0000 7efced149700 1 -- 172.21.15.46:0/2059262298 wait complete. 2024-01-27T01:08:31.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.373+0000 7efced149700 1 Processor -- start 2024-01-27T01:08:31.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.373+0000 7efced149700 1 -- start start 2024-01-27T01:08:31.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.374+0000 7efced149700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efce80cb740 0x7efce80c4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:31.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.374+0000 7efced149700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efce81597d0 0x7efce81532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:31.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.374+0000 7efced149700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efce815ad40 0x7efce80cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:31.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.374+0000 7efced149700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce80c4f40 con 0x7efce81597d0 2024-01-27T01:08:31.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.374+0000 7efced149700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce80c5da0 con 0x7efce815ad40 2024-01-27T01:08:31.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.374+0000 7efced149700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce80c5190 con 0x7efce80cb740 2024-01-27T01:08:31.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.374+0000 7efce659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efce81597d0 0x7efce81532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:31.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.374+0000 7efce6d9d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efce80cb740 0x7efce80c4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:31.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.374+0000 7efce659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efce81597d0 0x7efce81532a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34794/0 (socket says 172.21.15.46:34794) 2024-01-27T01:08:31.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.374+0000 7efce659c700 1 -- 172.21.15.46:0/1402658761 learned_addr learned my addr 172.21.15.46:0/1402658761 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:31.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.374+0000 7efce759e700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efce815ad40 0x7efce80cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:31.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.374+0000 7efce659c700 1 -- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efce80cb740 msgr2=0x7efce80c4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:31.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.374+0000 7efce659c700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efce80cb740 0x7efce80c4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:31.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.374+0000 7efce659c700 1 -- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efce815ad40 msgr2=0x7efce80cc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:31.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.374+0000 7efce659c700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efce815ad40 0x7efce80cc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:31.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.374+0000 7efce659c700 1 -- 172.21.15.46:0/1402658761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd8009a20 con 0x7efce81597d0 2024-01-27T01:08:31.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.375+0000 7efce659c700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efce81597d0 0x7efce81532a0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7efcd000c580 tx=0x7efcd0007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:31.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.375+0000 7efcdeffd700 1 -- 172.21.15.46:0/1402658761 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd0005b30 con 0x7efce81597d0 2024-01-27T01:08:31.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.375+0000 7efcdeffd700 1 -- 172.21.15.46:0/1402658761 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd0005c90 con 0x7efce81597d0 2024-01-27T01:08:31.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.375+0000 7efced149700 1 -- 172.21.15.46:0/1402658761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce80c5470 con 0x7efce81597d0 2024-01-27T01:08:31.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.375+0000 7efcdeffd700 1 -- 172.21.15.46:0/1402658761 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd0007320 con 0x7efce81597d0 2024-01-27T01:08:31.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.375+0000 7efced149700 1 -- 172.21.15.46:0/1402658761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce80c5a30 con 0x7efce81597d0 2024-01-27T01:08:31.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.376+0000 7efced149700 1 -- 172.21.15.46:0/1402658761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce804c710 con 0x7efce81597d0 2024-01-27T01:08:31.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.376+0000 7efcdeffd700 1 -- 172.21.15.46:0/1402658761 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 145) v1 ==== 84963+0+0 (secure 0 0 0) 0x7efcd0007690 con 0x7efce81597d0 2024-01-27T01:08:31.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.377+0000 7efcdeffd700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 0x7efcc8069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:31.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.377+0000 7efce6d9d700 1 -- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 msgr2=0x7efcc8069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:31.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.377+0000 7efcdeffd700 1 -- 172.21.15.46:0/1402658761 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6560+0+0 (secure 0 0 0) 0x7efcd00840c0 con 0x7efce81597d0 2024-01-27T01:08:31.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.377+0000 7efce6d9d700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 0x7efcc8069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:31.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.386+0000 7efcdeffd700 1 -- 172.21.15.46:0/1402658761 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efcd0050c80 con 0x7efce81597d0 2024-01-27T01:08:31.407 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:31.407+0000 7f58747b21c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:08:31.452 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:31.452+0000 7fbef6a741c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:08:31.491 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:31.493+0000 7f58747b21c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:08:31.524 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:31.525+0000 7fbef6a741c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:08:31.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.578+0000 7efce6d9d700 1 -- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 msgr2=0x7efcc8069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:31.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.578+0000 7efce6d9d700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 0x7efcc8069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:08:31.631 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:31.632+0000 7f58747b21c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:08:31.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.640+0000 7efced149700 1 -- 172.21.15.46:0/1402658761 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7efce815a540 con 0x7efcc80671c0 2024-01-27T01:08:31.663 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:31.664+0000 7fbef6a741c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:08:31.714 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:31.714+0000 7f58747b21c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:08:31.789 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:31.790+0000 7fbef6a741c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:08:31.889 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:31.890+0000 7f58747b21c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:08:31.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.979+0000 7efce6d9d700 1 -- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 msgr2=0x7efcc8069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:31.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:31.979+0000 7efce6d9d700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 0x7efcc8069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-27T01:08:32.032 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:32.033+0000 7f58747b21c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:08:32.104 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:32.105+0000 7fbef6a741c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:08:32.315 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:32.315+0000 7fbef6a741c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:08:32.393 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:32.394+0000 7f58747b21c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:08:32.538 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:32.539+0000 7fbef6a741c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:08:32.608 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:32.609+0000 7fbef6a741c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:08:32.632 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:32.633+0000 7f58747b21c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:08:32.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:32.780+0000 7efce6d9d700 1 -- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 msgr2=0x7efcc8069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:32.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:32.780+0000 7efce6d9d700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 0x7efcc8069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-27T01:08:32.895 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:32.896+0000 7f58747b21c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:08:32.976 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:32.977+0000 7f58747b21c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:08:33.215 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:33.216+0000 7fbef6a741c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:08:33.291 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:33.291+0000 7fbef6a741c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:08:33.413 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:33.414+0000 7fbef6a741c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:08:33.672 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:33.673+0000 7f58747b21c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:08:33.757 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:33.758+0000 7f58747b21c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:08:33.897 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:33.898+0000 7f58747b21c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:08:33.912 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:33.912+0000 7fbef6a741c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:08:34.061 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:34.061+0000 7fbef6a741c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:08:34.213 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:34.213+0000 7fbef6a741c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:08:34.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:34.382+0000 7efce6d9d700 1 -- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 msgr2=0x7efcc8069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:34.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:34.382+0000 7efce6d9d700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 0x7efcc8069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-27T01:08:34.483 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:34.484+0000 7f58747b21c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:08:34.654 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:34.655+0000 7f58747b21c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:08:34.826 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:34.827+0000 7f58747b21c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:08:34.914 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:34.915+0000 7fbef6a741c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:08:35.137 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:35.137+0000 7fbef6a741c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:08:35.211 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:35.211+0000 7fbef6a741c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:08:35.369 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:35.369+0000 7fbef6a741c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:08:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:35.392+0000 7efcdeffd700 1 -- 172.21.15.46:0/1402658761 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 146) v1 ==== 84470+0+0 (secure 0 0 0) 0x7efcd0051050 con 0x7efce81597d0 2024-01-27T01:08:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:35.393+0000 7efcdeffd700 1 -- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 msgr2=0x7efcc8069670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:35.393+0000 7efcdeffd700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 0x7efcc8069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:35.493 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:35.493+0000 7fbe6bb4f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:35.493 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:35.493+0000 7fbe6bb4f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:35.493 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:35.493+0000 7fbe6bb4f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:35.493 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:35.493+0000 7fbe6bb4f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:35.493 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:35.493+0000 7fbe6bb4f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:35.494 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:35.494+0000 7fbe70358700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:35.494 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:35.494+0000 7fbe70358700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:35.494 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:35.494+0000 7fbe70358700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:35.494 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:35.494+0000 7fbe70358700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:35.494 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:35.494+0000 7fbe70358700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:35.656 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:35.657+0000 7f58747b21c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:08:35.909 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:35.910+0000 7f58747b21c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:08:35.993 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:35.994+0000 7f58747b21c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:08:36.172 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:36.173+0000 7f58747b21c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:08:36.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.397+0000 7efcdeffd700 1 -- 172.21.15.46:0/1402658761 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 147) v1 ==== 84597+0+0 (secure 0 0 0) 0x7efcd0054ba0 con 0x7efce81597d0 2024-01-27T01:08:36.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.398+0000 7efcdeffd700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 0x7efcc8069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:36.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.398+0000 7efcdeffd700 1 -- 172.21.15.46:0/1402658761 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7efce815a540 con 0x7efcc80671c0 2024-01-27T01:08:36.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.399+0000 7efce6d9d700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 0x7efcc8069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:36.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.400+0000 7efce6d9d700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 0x7efcc8069670 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7efcd8009de0 tx=0x7efcd8019000 comp rx=0 tx=0).ready entity=mgr.10475 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.455+0000 7efcdeffd700 1 -- 172.21.15.46:0/1402658761 <== mgr.10475 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efce815a540 con 0x7efcc80671c0 2024-01-27T01:08:36.456 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path snap_test_dir1 2024-01-27T01:08:36.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.458+0000 7efced149700 1 -- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 msgr2=0x7efcc8069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:36.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.458+0000 7efced149700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 0x7efcc8069670 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7efcd8009de0 tx=0x7efcd8019000 comp rx=0 tx=0).stop 2024-01-27T01:08:36.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.458+0000 7efced149700 1 -- 172.21.15.46:0/1402658761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efce81597d0 msgr2=0x7efce81532a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:36.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.458+0000 7efced149700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efce81597d0 0x7efce81532a0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7efcd000c580 tx=0x7efcd0007df0 comp rx=0 tx=0).stop 2024-01-27T01:08:36.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.458+0000 7efced149700 1 -- 172.21.15.46:0/1402658761 shutdown_connections 2024-01-27T01:08:36.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.458+0000 7efced149700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efcc80671c0 0x7efcc8069670 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:36.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.458+0000 7efced149700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efce815ad40 0x7efce80cc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:36.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.458+0000 7efced149700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efce80cb740 0x7efce80c4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:36.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.458+0000 7efced149700 1 --2- 172.21.15.46:0/1402658761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efce81597d0 0x7efce81532a0 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:36.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.458+0000 7efced149700 1 -- 172.21.15.46:0/1402658761 >> 172.21.15.46:0/1402658761 conn(0x7efce80ca9a0 msgr2=0x7efce806b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:36.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.458+0000 7efced149700 1 -- 172.21.15.46:0/1402658761 shutdown_connections 2024-01-27T01:08:36.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.458+0000 7efced149700 1 -- 172.21.15.46:0/1402658761 wait complete. 2024-01-27T01:08:36.473 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-01-27T01:08:36.473 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-01-27T01:08:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.614+0000 7ffbbd66d700 1 Processor -- start 2024-01-27T01:08:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.614+0000 7ffbbd66d700 1 -- start start 2024-01-27T01:08:36.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.615+0000 7ffbbd66d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbb80ca4b0 0x7ffbb80afe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:36.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.615+0000 7ffbbd66d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffbb80b3a50 0x7ffbb80b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:36.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.615+0000 7ffbbd66d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb8158010 con 0x7ffbb80ca0e0 2024-01-27T01:08:36.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.615+0000 7ffbbd66d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbb806c7c0 con 0x7ffbb80b3a50 2024-01-27T01:08:36.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.615+0000 7ffbbd66d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb80cada0 con 0x7ffbb80ca4b0 2024-01-27T01:08:36.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.615+0000 7ffbb77fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffbb80b3a50 0x7ffbb80b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:36.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.615+0000 7ffbb77fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffbb80b3a50 0x7ffbb80b3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34806/0 (socket says 172.21.15.46:34806) 2024-01-27T01:08:36.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.615+0000 7ffbb6ffd700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ffbb80ca0e0 0x7ffbb806c0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43484/0 (socket says 172.21.15.46:43484) 2024-01-27T01:08:36.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.615+0000 7ffbb77fe700 1 -- 172.21.15.46:0/3467857231 learned_addr learned my addr 172.21.15.46:0/3467857231 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:36.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.616+0000 7ffbb67fc700 1 --2- 172.21.15.46:0/3467857231 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbb80ca4b0 0x7ffbb80afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:36.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.616+0000 7ffbb77fe700 1 -- 172.21.15.46:0/3467857231 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbb80ca4b0 msgr2=0x7ffbb80afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:36.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.616+0000 7ffbb77fe700 1 --2- 172.21.15.46:0/3467857231 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbb80ca4b0 0x7ffbb80afe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:36.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.616+0000 7ffbb77fe700 1 -- 172.21.15.46:0/3467857231 >> v1:172.21.15.46:6789/0 conn(0x7ffbb80ca0e0 legacy=0x7ffbb806c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:36.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.616+0000 7ffbb77fe700 1 -- 172.21.15.46:0/3467857231 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb81582f0 con 0x7ffbb80b3a50 2024-01-27T01:08:36.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.616+0000 7ffbb5ffb700 1 -- 172.21.15.46:0/3467857231 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180527383 0 0) 0x7ffbb8158010 con 0x7ffbb80ca0e0 2024-01-27T01:08:36.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.616+0000 7ffbb77fe700 1 --2- 172.21.15.46:0/3467857231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbb80b3a50 0x7ffbb80b3ed0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0009f20 tx=0x7ffbb0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ae8cc9ae808501 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:36.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.616+0000 7ffbb5ffb700 1 -- 172.21.15.46:0/3467857231 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb000e070 con 0x7ffbb80b3a50 2024-01-27T01:08:36.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.616+0000 7ffbb5ffb700 1 -- 172.21.15.46:0/3467857231 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb000b9d0 con 0x7ffbb80b3a50 2024-01-27T01:08:36.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.616+0000 7ffbb5ffb700 1 -- 172.21.15.46:0/3467857231 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00092e0 con 0x7ffbb80b3a50 2024-01-27T01:08:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.617+0000 7ffbbd66d700 1 -- 172.21.15.46:0/3467857231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbb80b3a50 msgr2=0x7ffbb80b3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.617+0000 7ffbbd66d700 1 --2- 172.21.15.46:0/3467857231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbb80b3a50 0x7ffbb80b3ed0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0009f20 tx=0x7ffbb0009840 comp rx=0 tx=0).stop 2024-01-27T01:08:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.617+0000 7ffbbd66d700 1 -- 172.21.15.46:0/3467857231 shutdown_connections 2024-01-27T01:08:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.617+0000 7ffbbd66d700 1 --2- 172.21.15.46:0/3467857231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbb80b3a50 0x7ffbb80b3ed0 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.617+0000 7ffbbd66d700 1 --2- 172.21.15.46:0/3467857231 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbb80ca4b0 0x7ffbb80afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.617+0000 7ffbbd66d700 1 -- 172.21.15.46:0/3467857231 >> 172.21.15.46:0/3467857231 conn(0x7ffbb80ce530 msgr2=0x7ffbb80ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.617+0000 7ffbbd66d700 1 -- 172.21.15.46:0/3467857231 shutdown_connections 2024-01-27T01:08:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.617+0000 7ffbbd66d700 1 -- 172.21.15.46:0/3467857231 wait complete. 2024-01-27T01:08:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.619+0000 7ffbbd66d700 1 Processor -- start 2024-01-27T01:08:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.619+0000 7ffbbd66d700 1 -- start start 2024-01-27T01:08:36.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.620+0000 7ffbbd66d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbb80b3a50 0x7ffbb8151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.620+0000 7ffbbd66d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbb80ca0e0 0x7ffbb80b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.620+0000 7ffbbd66d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbb80ca4b0 0x7ffbb815a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.620+0000 7ffbbd66d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb8151fc0 con 0x7ffbb80ca0e0 2024-01-27T01:08:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.620+0000 7ffbbd66d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb80c6750 con 0x7ffbb80ca4b0 2024-01-27T01:08:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.620+0000 7ffbbd66d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb80c5b40 con 0x7ffbb80b3a50 2024-01-27T01:08:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.620+0000 7ffbb67fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbb80ca0e0 0x7ffbb80b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.620+0000 7ffbb67fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbb80ca0e0 0x7ffbb80b30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34812/0 (socket says 172.21.15.46:34812) 2024-01-27T01:08:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.620+0000 7ffbb67fc700 1 -- 172.21.15.46:0/2242987038 learned_addr learned my addr 172.21.15.46:0/2242987038 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.620+0000 7ffbb6ffd700 1 --2- 172.21.15.46:0/2242987038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbb80b3a50 0x7ffbb8151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.620+0000 7ffbb77fe700 1 --2- 172.21.15.46:0/2242987038 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbb80ca4b0 0x7ffbb815a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.620+0000 7ffbb67fc700 1 -- 172.21.15.46:0/2242987038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbb80b3a50 msgr2=0x7ffbb8151a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.620+0000 7ffbb67fc700 1 --2- 172.21.15.46:0/2242987038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbb80b3a50 0x7ffbb8151a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.620+0000 7ffbb67fc700 1 -- 172.21.15.46:0/2242987038 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbb80ca4b0 msgr2=0x7ffbb815a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.620+0000 7ffbb67fc700 1 --2- 172.21.15.46:0/2242987038 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbb80ca4b0 0x7ffbb815a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:36.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.620+0000 7ffbb67fc700 1 -- 172.21.15.46:0/2242987038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb0009ae0 con 0x7ffbb80ca0e0 2024-01-27T01:08:36.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.620+0000 7ffbb67fc700 1 --2- 172.21.15.46:0/2242987038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbb80ca0e0 0x7ffbb80b30e0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7ffbac00e950 tx=0x7ffbac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:36.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.621+0000 7ffb9effd700 1 -- 172.21.15.46:0/2242987038 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac013680 con 0x7ffbb80ca0e0 2024-01-27T01:08:36.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.621+0000 7ffb9effd700 1 -- 172.21.15.46:0/2242987038 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbac0137e0 con 0x7ffbb80ca0e0 2024-01-27T01:08:36.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.621+0000 7ffbbd66d700 1 -- 172.21.15.46:0/2242987038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb80c5e20 con 0x7ffbb80ca0e0 2024-01-27T01:08:36.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.621+0000 7ffb9effd700 1 -- 172.21.15.46:0/2242987038 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac013ba0 con 0x7ffbb80ca0e0 2024-01-27T01:08:36.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.621+0000 7ffbbd66d700 1 -- 172.21.15.46:0/2242987038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb80c63e0 con 0x7ffbb80ca0e0 2024-01-27T01:08:36.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.622+0000 7ffbbd66d700 1 -- 172.21.15.46:0/2242987038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb804c710 con 0x7ffbb80ca0e0 2024-01-27T01:08:36.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.622+0000 7ffb9effd700 1 -- 172.21.15.46:0/2242987038 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 147) v1 ==== 84597+0+0 (secure 0 0 0) 0x7ffbac0123d0 con 0x7ffbb80ca0e0 2024-01-27T01:08:36.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.623+0000 7ffb9effd700 1 --2- 172.21.15.46:0/2242987038 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ffba4066dd0 0x7ffba4069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:36.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.623+0000 7ffb9effd700 1 -- 172.21.15.46:0/2242987038 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6990+0+0 (secure 0 0 0) 0x7ffbac084840 con 0x7ffbb80ca0e0 2024-01-27T01:08:36.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.623+0000 7ffbb6ffd700 1 --2- 172.21.15.46:0/2242987038 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ffba4066dd0 0x7ffba4069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:36.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.623+0000 7ffbb6ffd700 1 --2- 172.21.15.46:0/2242987038 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ffba4066dd0 0x7ffba4069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ffba8007fd0 tx=0x7ffba8007d50 comp rx=0 tx=0).ready entity=mgr.10475 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.627+0000 7ffb9effd700 1 -- 172.21.15.46:0/2242987038 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffbac051860 con 0x7ffbb80ca0e0 2024-01-27T01:08:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.861+0000 7ffbbd66d700 1 -- 172.21.15.46:0/2242987038 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ffbb8153220 con 0x7ffba4066dd0 2024-01-27T01:08:36.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.911+0000 7ffb9effd700 1 -- 172.21.15.46:0/2242987038 <== mgr.10475 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7ffbb8153220 con 0x7ffba4066dd0 2024-01-27T01:08:36.911 INFO:teuthology.orchestra.run.smithi046.stdout:Retention added to path snap_test_dir1 2024-01-27T01:08:36.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.914+0000 7ffbbd66d700 1 -- 172.21.15.46:0/2242987038 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ffba4066dd0 msgr2=0x7ffba4069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:36.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.914+0000 7ffbbd66d700 1 --2- 172.21.15.46:0/2242987038 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ffba4066dd0 0x7ffba4069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ffba8007fd0 tx=0x7ffba8007d50 comp rx=0 tx=0).stop 2024-01-27T01:08:36.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.914+0000 7ffbbd66d700 1 -- 172.21.15.46:0/2242987038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbb80ca0e0 msgr2=0x7ffbb80b30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:36.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.914+0000 7ffbbd66d700 1 --2- 172.21.15.46:0/2242987038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbb80ca0e0 0x7ffbb80b30e0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7ffbac00e950 tx=0x7ffbac0091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:36.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.914+0000 7ffbbd66d700 1 -- 172.21.15.46:0/2242987038 shutdown_connections 2024-01-27T01:08:36.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.914+0000 7ffbbd66d700 1 --2- 172.21.15.46:0/2242987038 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ffba4066dd0 0x7ffba4069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:36.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.914+0000 7ffbbd66d700 1 --2- 172.21.15.46:0/2242987038 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbb80ca4b0 0x7ffbb815a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:36.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.914+0000 7ffbbd66d700 1 --2- 172.21.15.46:0/2242987038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbb80b3a50 0x7ffbb8151a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:36.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.914+0000 7ffbbd66d700 1 --2- 172.21.15.46:0/2242987038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbb80ca0e0 0x7ffbb80b30e0 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:36.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.914+0000 7ffbbd66d700 1 -- 172.21.15.46:0/2242987038 >> 172.21.15.46:0/2242987038 conn(0x7ffbb80ce530 msgr2=0x7ffbb81557c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:36.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.914+0000 7ffbbd66d700 1 -- 172.21.15.46:0/2242987038 shutdown_connections 2024-01-27T01:08:36.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:36.914+0000 7ffbbd66d700 1 -- 172.21.15.46:0/2242987038 wait complete. 2024-01-27T01:08:36.928 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-01-27T01:08:36.933 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-27T01:08:36.935 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-27T01:08:36.987 INFO:teuthology.orchestra.run.smithi106.stdout:[global] 2024-01-27T01:08:36.987 INFO:teuthology.orchestra.run.smithi106.stdout: chdir = "" 2024-01-27T01:08:36.987 INFO:teuthology.orchestra.run.smithi106.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-27T01:08:36.987 INFO:teuthology.orchestra.run.smithi106.stdout: auth supported = cephx 2024-01-27T01:08:36.987 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.987 INFO:teuthology.orchestra.run.smithi106.stdout: filestore xattr use omap = true 2024-01-27T01:08:36.987 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.987 INFO:teuthology.orchestra.run.smithi106.stdout: mon clock drift allowed = 1.000 2024-01-27T01:08:36.987 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.987 INFO:teuthology.orchestra.run.smithi106.stdout: osd crush chooseleaf type = 0 2024-01-27T01:08:36.987 INFO:teuthology.orchestra.run.smithi106.stdout: auth debug = true 2024-01-27T01:08:36.987 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.988 INFO:teuthology.orchestra.run.smithi106.stdout: ms die on old message = true 2024-01-27T01:08:36.988 INFO:teuthology.orchestra.run.smithi106.stdout: ms die on bug = true 2024-01-27T01:08:36.988 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.988 INFO:teuthology.orchestra.run.smithi106.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-27T01:08:36.988 INFO:teuthology.orchestra.run.smithi106.stdout: mon pg warn max object skew = 0 2024-01-27T01:08:36.988 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.988 INFO:teuthology.orchestra.run.smithi106.stdout: # disable pg_autoscaler by default for new pools 2024-01-27T01:08:36.988 INFO:teuthology.orchestra.run.smithi106.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-27T01:08:36.988 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.988 INFO:teuthology.orchestra.run.smithi106.stdout: osd pool default size = 2 2024-01-27T01:08:36.988 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.988 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd allow primary affinity = true 2024-01-27T01:08:36.988 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd allow pg remap = true 2024-01-27T01:08:36.988 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on legacy crush tunables = false 2024-01-27T01:08:36.988 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on crush straw calc version zero = false 2024-01-27T01:08:36.989 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on no sortbitwise = false 2024-01-27T01:08:36.989 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on osd down out interval zero = false 2024-01-27T01:08:36.989 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on too few osds = false 2024-01-27T01:08:36.989 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-27T01:08:36.989 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-27T01:08:36.989 INFO:teuthology.orchestra.run.smithi106.stdout: mon_allow_pool_size_one = true 2024-01-27T01:08:36.989 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.989 INFO:teuthology.orchestra.run.smithi106.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-27T01:08:36.989 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.989 INFO:teuthology.orchestra.run.smithi106.stdout: osd default data pool replay window = 5 2024-01-27T01:08:36.989 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.989 INFO:teuthology.orchestra.run.smithi106.stdout: mon allow pool delete = true 2024-01-27T01:08:36.989 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.989 INFO:teuthology.orchestra.run.smithi106.stdout: mon cluster log file level = debug 2024-01-27T01:08:36.990 INFO:teuthology.orchestra.run.smithi106.stdout: debug asserts on shutdown = true 2024-01-27T01:08:36.990 INFO:teuthology.orchestra.run.smithi106.stdout: mon health detail to clog = false 2024-01-27T01:08:36.990 INFO:teuthology.orchestra.run.smithi106.stdout: mon host = "172.21.15.46,172.21.15.106,[v2:172.21.15.106:3301,v1:172.21.15.106:6790]" 2024-01-27T01:08:36.990 INFO:teuthology.orchestra.run.smithi106.stdout: fsid = 62f24da2-f257-4a3f-ab5a-77a1100d6f1e 2024-01-27T01:08:36.990 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.990 INFO:teuthology.orchestra.run.smithi106.stdout:[osd] 2024-01-27T01:08:36.990 INFO:teuthology.orchestra.run.smithi106.stdout: osd journal size = 100 2024-01-27T01:08:36.990 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.990 INFO:teuthology.orchestra.run.smithi106.stdout: osd scrub load threshold = 5.0 2024-01-27T01:08:36.990 INFO:teuthology.orchestra.run.smithi106.stdout: osd scrub max interval = 600 2024-01-27T01:08:36.990 INFO:teuthology.orchestra.run.smithi106.stdout: osd mclock profile = high_recovery_ops 2024-01-27T01:08:36.990 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.990 INFO:teuthology.orchestra.run.smithi106.stdout: osd recover clone overlap = true 2024-01-27T01:08:36.990 INFO:teuthology.orchestra.run.smithi106.stdout: osd recovery max chunk = 1048576 2024-01-27T01:08:36.990 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.991 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug shutdown = true 2024-01-27T01:08:36.991 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug op order = true 2024-01-27T01:08:36.991 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug verify stray on activate = true 2024-01-27T01:08:36.991 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.991 INFO:teuthology.orchestra.run.smithi106.stdout: osd open classes on start = true 2024-01-27T01:08:36.991 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug pg log writeout = true 2024-01-27T01:08:36.991 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.991 INFO:teuthology.orchestra.run.smithi106.stdout: osd deep scrub update digest min age = 30 2024-01-27T01:08:36.991 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.991 INFO:teuthology.orchestra.run.smithi106.stdout: osd map max advance = 10 2024-01-27T01:08:36.991 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.991 INFO:teuthology.orchestra.run.smithi106.stdout: journal zero on create = true 2024-01-27T01:08:36.991 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.991 INFO:teuthology.orchestra.run.smithi106.stdout: filestore ondisk finisher threads = 3 2024-01-27T01:08:36.991 INFO:teuthology.orchestra.run.smithi106.stdout: filestore apply finisher threads = 3 2024-01-27T01:08:36.992 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.992 INFO:teuthology.orchestra.run.smithi106.stdout: bdev debug aio = true 2024-01-27T01:08:36.992 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug misdirected ops = true 2024-01-27T01:08:36.992 INFO:teuthology.orchestra.run.smithi106.stdout: bluestore block size = 96636764160 2024-01-27T01:08:36.992 INFO:teuthology.orchestra.run.smithi106.stdout: bluestore fsck on mount = True 2024-01-27T01:08:36.992 INFO:teuthology.orchestra.run.smithi106.stdout: debug bluefs = 20 2024-01-27T01:08:36.992 INFO:teuthology.orchestra.run.smithi106.stdout: debug bluestore = 20 2024-01-27T01:08:36.992 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-01-27T01:08:36.992 INFO:teuthology.orchestra.run.smithi106.stdout: debug osd = 20 2024-01-27T01:08:36.992 INFO:teuthology.orchestra.run.smithi106.stdout: debug rocksdb = 10 2024-01-27T01:08:36.992 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd backfillfull_ratio = 0.85 2024-01-27T01:08:36.992 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd full ratio = 0.9 2024-01-27T01:08:36.992 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd nearfull ratio = 0.8 2024-01-27T01:08:36.992 INFO:teuthology.orchestra.run.smithi106.stdout: osd failsafe full ratio = 0.95 2024-01-27T01:08:36.992 INFO:teuthology.orchestra.run.smithi106.stdout: osd objectstore = bluestore 2024-01-27T01:08:36.993 INFO:teuthology.orchestra.run.smithi106.stdout: osd op complaint time = 180 2024-01-27T01:08:36.993 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.993 INFO:teuthology.orchestra.run.smithi106.stdout:[mgr] 2024-01-27T01:08:36.993 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-01-27T01:08:36.993 INFO:teuthology.orchestra.run.smithi106.stdout: debug mgr = 20 2024-01-27T01:08:36.993 INFO:teuthology.orchestra.run.smithi106.stdout: debug mon = 20 2024-01-27T01:08:36.993 INFO:teuthology.orchestra.run.smithi106.stdout: debug auth = 20 2024-01-27T01:08:36.993 INFO:teuthology.orchestra.run.smithi106.stdout: mon reweight min pgs per osd = 4 2024-01-27T01:08:36.993 INFO:teuthology.orchestra.run.smithi106.stdout: mon reweight min bytes per osd = 10 2024-01-27T01:08:36.993 INFO:teuthology.orchestra.run.smithi106.stdout: mgr/telemetry/nag = false 2024-01-27T01:08:36.993 INFO:teuthology.orchestra.run.smithi106.stdout: debug client = 20 2024-01-27T01:08:36.993 INFO:teuthology.orchestra.run.smithi106.stdout: debug finisher = 20 2024-01-27T01:08:36.993 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.993 INFO:teuthology.orchestra.run.smithi106.stdout:[mon] 2024-01-27T01:08:36.993 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-01-27T01:08:36.994 INFO:teuthology.orchestra.run.smithi106.stdout: debug mon = 20 2024-01-27T01:08:36.994 INFO:teuthology.orchestra.run.smithi106.stdout: debug paxos = 20 2024-01-27T01:08:36.994 INFO:teuthology.orchestra.run.smithi106.stdout: debug auth = 20 2024-01-27T01:08:36.994 INFO:teuthology.orchestra.run.smithi106.stdout: mon data avail warn = 5 2024-01-27T01:08:36.994 INFO:teuthology.orchestra.run.smithi106.stdout: mon mgr mkfs grace = 240 2024-01-27T01:08:36.994 INFO:teuthology.orchestra.run.smithi106.stdout: mon reweight min pgs per osd = 4 2024-01-27T01:08:36.994 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd reporter subtree level = osd 2024-01-27T01:08:36.994 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd prime pg temp = true 2024-01-27T01:08:36.994 INFO:teuthology.orchestra.run.smithi106.stdout: mon reweight min bytes per osd = 10 2024-01-27T01:08:36.994 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.994 INFO:teuthology.orchestra.run.smithi106.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-27T01:08:36.994 INFO:teuthology.orchestra.run.smithi106.stdout: auth mon ticket ttl = 660 # 11m 2024-01-27T01:08:36.994 INFO:teuthology.orchestra.run.smithi106.stdout: auth service ticket ttl = 240 # 4m 2024-01-27T01:08:36.994 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.994 INFO:teuthology.orchestra.run.smithi106.stdout: # don't complain about insecure global_id in the test suite 2024-01-27T01:08:36.995 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-27T01:08:36.995 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-27T01:08:36.995 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.995 INFO:teuthology.orchestra.run.smithi106.stdout: # 1m isn't quite enough 2024-01-27T01:08:36.995 INFO:teuthology.orchestra.run.smithi106.stdout: mon_down_mkfs_grace = 2m 2024-01-27T01:08:36.995 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.995 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_filestore_osds = false 2024-01-27T01:08:36.995 INFO:teuthology.orchestra.run.smithi106.stdout: mon op complaint time = 120 2024-01-27T01:08:36.995 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-27T01:08:36.995 INFO:teuthology.orchestra.run.smithi106.stdout:[client] 2024-01-27T01:08:36.995 INFO:teuthology.orchestra.run.smithi106.stdout: rgw cache enabled = true 2024-01-27T01:08:36.995 INFO:teuthology.orchestra.run.smithi106.stdout: rgw enable ops log = true 2024-01-27T01:08:36.995 INFO:teuthology.orchestra.run.smithi106.stdout: rgw enable usage log = true 2024-01-27T01:08:36.995 INFO:teuthology.orchestra.run.smithi106.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-27T01:08:36.995 INFO:teuthology.orchestra.run.smithi106.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-27T01:08:36.996 INFO:teuthology.orchestra.run.smithi106.stdout: client mount timeout = 600 2024-01-27T01:08:36.996 INFO:teuthology.orchestra.run.smithi106.stdout: debug client = 20 2024-01-27T01:08:36.996 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-01-27T01:08:36.996 INFO:teuthology.orchestra.run.smithi106.stdout: pid file = "" 2024-01-27T01:08:36.996 INFO:teuthology.orchestra.run.smithi106.stdout: rados mon op timeout = 900 2024-01-27T01:08:36.996 INFO:teuthology.orchestra.run.smithi106.stdout: rados osd op timeout = 900 2024-01-27T01:08:36.996 INFO:teuthology.orchestra.run.smithi106.stdout: client snapdir = .snap 2024-01-27T01:08:36.996 INFO:teuthology.orchestra.run.smithi106.stdout:[mon.a] 2024-01-27T01:08:36.996 INFO:teuthology.orchestra.run.smithi106.stdout:[mon.b] 2024-01-27T01:08:36.996 INFO:teuthology.orchestra.run.smithi106.stdout:[mon.c] 2024-01-27T01:08:36.996 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.a] 2024-01-27T01:08:36.996 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.c] 2024-01-27T01:08:36.996 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.b] 2024-01-27T01:08:36.996 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.d] 2024-01-27T01:08:36.996 INFO:teuthology.orchestra.run.smithi106.stdout:[mds] 2024-01-27T01:08:36.997 INFO:teuthology.orchestra.run.smithi106.stdout: debug mds = 20 2024-01-27T01:08:36.997 INFO:teuthology.orchestra.run.smithi106.stdout: debug mds balancer = 20 2024-01-27T01:08:36.997 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-01-27T01:08:36.997 INFO:teuthology.orchestra.run.smithi106.stdout: mds debug frag = True 2024-01-27T01:08:36.997 INFO:teuthology.orchestra.run.smithi106.stdout: mds debug scatterstat = True 2024-01-27T01:08:36.997 INFO:teuthology.orchestra.run.smithi106.stdout: mds op complaint time = 180 2024-01-27T01:08:36.997 INFO:teuthology.orchestra.run.smithi106.stdout: mds verify scatter = True 2024-01-27T01:08:36.997 INFO:teuthology.orchestra.run.smithi106.stdout: osd op complaint time = 180 2024-01-27T01:08:36.997 INFO:teuthology.orchestra.run.smithi106.stdout: rados mon op timeout = 900 2024-01-27T01:08:36.997 INFO:teuthology.orchestra.run.smithi106.stdout: rados osd op timeout = 900 2024-01-27T01:08:36.998 INFO:teuthology.orchestra.run.smithi046.stdout:[global] 2024-01-27T01:08:36.998 INFO:teuthology.orchestra.run.smithi046.stdout: chdir = "" 2024-01-27T01:08:36.998 INFO:teuthology.orchestra.run.smithi046.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-27T01:08:36.998 INFO:teuthology.orchestra.run.smithi046.stdout: auth supported = cephx 2024-01-27T01:08:36.998 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:36.998 INFO:teuthology.orchestra.run.smithi046.stdout: filestore xattr use omap = true 2024-01-27T01:08:36.998 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:36.998 INFO:teuthology.orchestra.run.smithi046.stdout: mon clock drift allowed = 1.000 2024-01-27T01:08:36.998 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:36.998 INFO:teuthology.orchestra.run.smithi046.stdout: osd crush chooseleaf type = 0 2024-01-27T01:08:36.998 INFO:teuthology.orchestra.run.smithi046.stdout: auth debug = true 2024-01-27T01:08:36.998 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:36.998 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on old message = true 2024-01-27T01:08:36.999 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on bug = true 2024-01-27T01:08:36.999 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:36.999 INFO:teuthology.orchestra.run.smithi046.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-27T01:08:36.999 INFO:teuthology.orchestra.run.smithi046.stdout: mon pg warn max object skew = 0 2024-01-27T01:08:36.999 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:36.999 INFO:teuthology.orchestra.run.smithi046.stdout: # disable pg_autoscaler by default for new pools 2024-01-27T01:08:36.999 INFO:teuthology.orchestra.run.smithi046.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-27T01:08:36.999 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:36.999 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default size = 2 2024-01-27T01:08:36.999 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:36.999 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow primary affinity = true 2024-01-27T01:08:36.999 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow pg remap = true 2024-01-27T01:08:36.999 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on legacy crush tunables = false 2024-01-27T01:08:36.999 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on crush straw calc version zero = false 2024-01-27T01:08:36.999 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on no sortbitwise = false 2024-01-27T01:08:37.000 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on osd down out interval zero = false 2024-01-27T01:08:37.000 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on too few osds = false 2024-01-27T01:08:37.000 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-27T01:08:37.000 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-27T01:08:37.000 INFO:teuthology.orchestra.run.smithi046.stdout: mon_allow_pool_size_one = true 2024-01-27T01:08:37.000 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.000 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-27T01:08:37.000 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.000 INFO:teuthology.orchestra.run.smithi046.stdout: osd default data pool replay window = 5 2024-01-27T01:08:37.000 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.000 INFO:teuthology.orchestra.run.smithi046.stdout: mon allow pool delete = true 2024-01-27T01:08:37.000 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.000 INFO:teuthology.orchestra.run.smithi046.stdout: mon cluster log file level = debug 2024-01-27T01:08:37.000 INFO:teuthology.orchestra.run.smithi046.stdout: debug asserts on shutdown = true 2024-01-27T01:08:37.001 INFO:teuthology.orchestra.run.smithi046.stdout: mon health detail to clog = false 2024-01-27T01:08:37.001 INFO:teuthology.orchestra.run.smithi046.stdout: mon host = "172.21.15.46,172.21.15.106,[v2:172.21.15.106:3301,v1:172.21.15.106:6790]" 2024-01-27T01:08:37.001 INFO:teuthology.orchestra.run.smithi046.stdout: fsid = 62f24da2-f257-4a3f-ab5a-77a1100d6f1e 2024-01-27T01:08:37.001 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.001 INFO:teuthology.orchestra.run.smithi046.stdout:[osd] 2024-01-27T01:08:37.001 INFO:teuthology.orchestra.run.smithi046.stdout: osd journal size = 100 2024-01-27T01:08:37.001 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.001 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub load threshold = 5.0 2024-01-27T01:08:37.001 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub max interval = 600 2024-01-27T01:08:37.001 INFO:teuthology.orchestra.run.smithi046.stdout: osd mclock profile = high_recovery_ops 2024-01-27T01:08:37.001 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.001 INFO:teuthology.orchestra.run.smithi046.stdout: osd recover clone overlap = true 2024-01-27T01:08:37.001 INFO:teuthology.orchestra.run.smithi046.stdout: osd recovery max chunk = 1048576 2024-01-27T01:08:37.001 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.001 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug shutdown = true 2024-01-27T01:08:37.002 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug op order = true 2024-01-27T01:08:37.002 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify stray on activate = true 2024-01-27T01:08:37.002 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.002 INFO:teuthology.orchestra.run.smithi046.stdout: osd open classes on start = true 2024-01-27T01:08:37.002 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug pg log writeout = true 2024-01-27T01:08:37.002 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.002 INFO:teuthology.orchestra.run.smithi046.stdout: osd deep scrub update digest min age = 30 2024-01-27T01:08:37.002 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.002 INFO:teuthology.orchestra.run.smithi046.stdout: osd map max advance = 10 2024-01-27T01:08:37.002 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.002 INFO:teuthology.orchestra.run.smithi046.stdout: journal zero on create = true 2024-01-27T01:08:37.002 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.002 INFO:teuthology.orchestra.run.smithi046.stdout: filestore ondisk finisher threads = 3 2024-01-27T01:08:37.002 INFO:teuthology.orchestra.run.smithi046.stdout: filestore apply finisher threads = 3 2024-01-27T01:08:37.002 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.003 INFO:teuthology.orchestra.run.smithi046.stdout: bdev debug aio = true 2024-01-27T01:08:37.003 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug misdirected ops = true 2024-01-27T01:08:37.003 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore block size = 96636764160 2024-01-27T01:08:37.003 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore fsck on mount = True 2024-01-27T01:08:37.003 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluefs = 20 2024-01-27T01:08:37.003 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluestore = 20 2024-01-27T01:08:37.003 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-01-27T01:08:37.003 INFO:teuthology.orchestra.run.smithi046.stdout: debug osd = 20 2024-01-27T01:08:37.003 INFO:teuthology.orchestra.run.smithi046.stdout: debug rocksdb = 10 2024-01-27T01:08:37.003 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd backfillfull_ratio = 0.85 2024-01-27T01:08:37.003 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd full ratio = 0.9 2024-01-27T01:08:37.003 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd nearfull ratio = 0.8 2024-01-27T01:08:37.003 INFO:teuthology.orchestra.run.smithi046.stdout: osd failsafe full ratio = 0.95 2024-01-27T01:08:37.003 INFO:teuthology.orchestra.run.smithi046.stdout: osd objectstore = bluestore 2024-01-27T01:08:37.004 INFO:teuthology.orchestra.run.smithi046.stdout: osd op complaint time = 180 2024-01-27T01:08:37.004 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.004 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr] 2024-01-27T01:08:37.004 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-01-27T01:08:37.004 INFO:teuthology.orchestra.run.smithi046.stdout: debug mgr = 20 2024-01-27T01:08:37.004 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2024-01-27T01:08:37.004 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2024-01-27T01:08:37.004 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2024-01-27T01:08:37.004 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2024-01-27T01:08:37.004 INFO:teuthology.orchestra.run.smithi046.stdout: mgr/telemetry/nag = false 2024-01-27T01:08:37.004 INFO:teuthology.orchestra.run.smithi046.stdout: debug client = 20 2024-01-27T01:08:37.004 INFO:teuthology.orchestra.run.smithi046.stdout: debug finisher = 20 2024-01-27T01:08:37.004 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.004 INFO:teuthology.orchestra.run.smithi046.stdout:[mon] 2024-01-27T01:08:37.004 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-01-27T01:08:37.005 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2024-01-27T01:08:37.005 INFO:teuthology.orchestra.run.smithi046.stdout: debug paxos = 20 2024-01-27T01:08:37.005 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2024-01-27T01:08:37.005 INFO:teuthology.orchestra.run.smithi046.stdout: mon data avail warn = 5 2024-01-27T01:08:37.005 INFO:teuthology.orchestra.run.smithi046.stdout: mon mgr mkfs grace = 240 2024-01-27T01:08:37.005 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2024-01-27T01:08:37.005 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd reporter subtree level = osd 2024-01-27T01:08:37.005 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd prime pg temp = true 2024-01-27T01:08:37.005 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2024-01-27T01:08:37.005 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.005 INFO:teuthology.orchestra.run.smithi046.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-27T01:08:37.005 INFO:teuthology.orchestra.run.smithi046.stdout: auth mon ticket ttl = 660 # 11m 2024-01-27T01:08:37.005 INFO:teuthology.orchestra.run.smithi046.stdout: auth service ticket ttl = 240 # 4m 2024-01-27T01:08:37.005 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.005 INFO:teuthology.orchestra.run.smithi046.stdout: # don't complain about insecure global_id in the test suite 2024-01-27T01:08:37.006 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-27T01:08:37.006 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-27T01:08:37.006 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.006 INFO:teuthology.orchestra.run.smithi046.stdout: # 1m isn't quite enough 2024-01-27T01:08:37.006 INFO:teuthology.orchestra.run.smithi046.stdout: mon_down_mkfs_grace = 2m 2024-01-27T01:08:37.006 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.006 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_filestore_osds = false 2024-01-27T01:08:37.006 INFO:teuthology.orchestra.run.smithi046.stdout: mon op complaint time = 120 2024-01-27T01:08:37.006 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:37.006 INFO:teuthology.orchestra.run.smithi046.stdout:[client] 2024-01-27T01:08:37.006 INFO:teuthology.orchestra.run.smithi046.stdout: rgw cache enabled = true 2024-01-27T01:08:37.006 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable ops log = true 2024-01-27T01:08:37.006 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable usage log = true 2024-01-27T01:08:37.006 INFO:teuthology.orchestra.run.smithi046.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-27T01:08:37.006 INFO:teuthology.orchestra.run.smithi046.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-27T01:08:37.007 INFO:teuthology.orchestra.run.smithi046.stdout: client mount timeout = 600 2024-01-27T01:08:37.007 INFO:teuthology.orchestra.run.smithi046.stdout: debug client = 20 2024-01-27T01:08:37.007 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-01-27T01:08:37.007 INFO:teuthology.orchestra.run.smithi046.stdout: pid file = "" 2024-01-27T01:08:37.007 INFO:teuthology.orchestra.run.smithi046.stdout: rados mon op timeout = 900 2024-01-27T01:08:37.007 INFO:teuthology.orchestra.run.smithi046.stdout: rados osd op timeout = 900 2024-01-27T01:08:37.007 INFO:teuthology.orchestra.run.smithi046.stdout: client snapdir = .snap 2024-01-27T01:08:37.007 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.a] 2024-01-27T01:08:37.007 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.b] 2024-01-27T01:08:37.007 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.c] 2024-01-27T01:08:37.007 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.a] 2024-01-27T01:08:37.007 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.c] 2024-01-27T01:08:37.007 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.b] 2024-01-27T01:08:37.007 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.d] 2024-01-27T01:08:37.007 INFO:teuthology.orchestra.run.smithi046.stdout:[mds] 2024-01-27T01:08:37.008 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds = 20 2024-01-27T01:08:37.008 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds balancer = 20 2024-01-27T01:08:37.008 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-01-27T01:08:37.008 INFO:teuthology.orchestra.run.smithi046.stdout: mds debug frag = True 2024-01-27T01:08:37.008 INFO:teuthology.orchestra.run.smithi046.stdout: mds debug scatterstat = True 2024-01-27T01:08:37.008 INFO:teuthology.orchestra.run.smithi046.stdout: mds op complaint time = 180 2024-01-27T01:08:37.008 INFO:teuthology.orchestra.run.smithi046.stdout: mds verify scatter = True 2024-01-27T01:08:37.008 INFO:teuthology.orchestra.run.smithi046.stdout: osd op complaint time = 180 2024-01-27T01:08:37.008 INFO:teuthology.orchestra.run.smithi046.stdout: rados mon op timeout = 900 2024-01-27T01:08:37.008 INFO:teuthology.orchestra.run.smithi046.stdout: rados osd op timeout = 900 2024-01-27T01:08:37.009 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-27T01:08:37.010 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-27T01:08:37.010 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-27T01:08:37.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[122008]: fuse finished with error 0 and tester_r 0 2024-01-27T01:08:37.048 INFO:teuthology.orchestra.run:waiting for 900 2024-01-27T01:08:37.108 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T01:08:37.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:37.109 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-27T01:08:37.130 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-27T01:08:37.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.316+0000 7efd8b131700 1 Processor -- start 2024-01-27T01:08:37.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.316+0000 7efd8b131700 1 -- start start 2024-01-27T01:08:37.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.318+0000 7efd8b131700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efd841574b0 0x7efd8415bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:37.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.318+0000 7efd8b131700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7efd840b2ce0 0x7efd8406c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:37.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.318+0000 7efd8b131700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd8406f5c0 con 0x7efd84157880 2024-01-27T01:08:37.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.318+0000 7efd8b131700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd840cd180 con 0x7efd841574b0 2024-01-27T01:08:37.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.318+0000 7efd8b131700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd8415c140 con 0x7efd840b2ce0 2024-01-27T01:08:37.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.318+0000 7efd88ecd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efd841574b0 0x7efd8415bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:37.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.318+0000 7efd88ecd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efd841574b0 0x7efd8415bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34816/0 (socket says 172.21.15.46:34816) 2024-01-27T01:08:37.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.318+0000 7efd88ecd700 1 -- 172.21.15.46:0/3691095054 learned_addr learned my addr 172.21.15.46:0/3691095054 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:37.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.318+0000 7efd896ce700 1 --2- 172.21.15.46:0/3691095054 >> v2:172.21.15.106:3300/0 conn(0x7efd840b2ce0 0x7efd8406c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:37.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.318+0000 7efd88ecd700 1 -- 172.21.15.46:0/3691095054 >> v2:172.21.15.106:3300/0 conn(0x7efd840b2ce0 msgr2=0x7efd8406c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:37.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.318+0000 7efd88ecd700 1 --2- 172.21.15.46:0/3691095054 >> v2:172.21.15.106:3300/0 conn(0x7efd840b2ce0 0x7efd8406c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:37.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.318+0000 7efd88ecd700 1 -- 172.21.15.46:0/3691095054 >> v1:172.21.15.106:6789/0 conn(0x7efd84157880 legacy=0x7efd840b25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:37.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.318+0000 7efd88ecd700 1 -- 172.21.15.46:0/3691095054 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd840ca140 con 0x7efd841574b0 2024-01-27T01:08:37.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.318+0000 7efd896ce700 1 --2- 172.21.15.46:0/3691095054 >> v2:172.21.15.106:3300/0 conn(0x7efd840b2ce0 0x7efd8406c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:08:37.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.318+0000 7efd88ecd700 1 --2- 172.21.15.46:0/3691095054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd841574b0 0x7efd8415bc00 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7efd840cce00 tx=0x7efd78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c2e534a3a044621 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:37.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.318+0000 7efd837fe700 1 -- 172.21.15.46:0/3691095054 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7800b820 con 0x7efd841574b0 2024-01-27T01:08:37.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.319+0000 7efd837fe700 1 -- 172.21.15.46:0/3691095054 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7800b980 con 0x7efd841574b0 2024-01-27T01:08:37.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.319+0000 7efd837fe700 1 -- 172.21.15.46:0/3691095054 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7800bd20 con 0x7efd841574b0 2024-01-27T01:08:37.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.319+0000 7efd8b131700 1 -- 172.21.15.46:0/3691095054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd841574b0 msgr2=0x7efd8415bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:37.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.319+0000 7efd8b131700 1 --2- 172.21.15.46:0/3691095054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd841574b0 0x7efd8415bc00 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7efd840cce00 tx=0x7efd78009250 comp rx=0 tx=0).stop 2024-01-27T01:08:37.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.319+0000 7efd8b131700 1 -- 172.21.15.46:0/3691095054 shutdown_connections 2024-01-27T01:08:37.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.319+0000 7efd8b131700 1 --2- 172.21.15.46:0/3691095054 >> v2:172.21.15.106:3300/0 conn(0x7efd840b2ce0 0x7efd8406c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:37.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.319+0000 7efd8b131700 1 --2- 172.21.15.46:0/3691095054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd841574b0 0x7efd8415bc00 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:37.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.319+0000 7efd8b131700 1 -- 172.21.15.46:0/3691095054 >> 172.21.15.46:0/3691095054 conn(0x7efd8406b6c0 msgr2=0x7efd840aef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:37.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.319+0000 7efd8b131700 1 -- 172.21.15.46:0/3691095054 shutdown_connections 2024-01-27T01:08:37.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.319+0000 7efd8b131700 1 -- 172.21.15.46:0/3691095054 wait complete. 2024-01-27T01:08:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.320+0000 7efd8b131700 1 Processor -- start 2024-01-27T01:08:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.321+0000 7efd8b131700 1 -- start start 2024-01-27T01:08:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.321+0000 7efd8b131700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd840b2ce0 0x7efd84167d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.321+0000 7efd8b131700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd841574b0 0x7efd8414d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.321+0000 7efd8b131700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd84157880 0x7efd8414d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.321+0000 7efd8b131700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd840c2370 con 0x7efd841574b0 2024-01-27T01:08:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.321+0000 7efd8b131700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8414deb0 con 0x7efd84157880 2024-01-27T01:08:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.321+0000 7efd8b131700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd84154630 con 0x7efd840b2ce0 2024-01-27T01:08:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.321+0000 7efd83fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd841574b0 0x7efd8414d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.321+0000 7efd83fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd841574b0 0x7efd8414d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34824/0 (socket says 172.21.15.46:34824) 2024-01-27T01:08:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.321+0000 7efd83fff700 1 -- 172.21.15.46:0/3141553193 learned_addr learned my addr 172.21.15.46:0/3141553193 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.322+0000 7efd896ce700 1 --2- 172.21.15.46:0/3141553193 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd84157880 0x7efd8414d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.322+0000 7efd88ecd700 1 --2- 172.21.15.46:0/3141553193 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd840b2ce0 0x7efd84167d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.322+0000 7efd83fff700 1 -- 172.21.15.46:0/3141553193 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd840b2ce0 msgr2=0x7efd84167d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.322+0000 7efd83fff700 1 --2- 172.21.15.46:0/3141553193 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd840b2ce0 0x7efd84167d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.322+0000 7efd83fff700 1 -- 172.21.15.46:0/3141553193 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd84157880 msgr2=0x7efd8414d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.322+0000 7efd83fff700 1 --2- 172.21.15.46:0/3141553193 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd84157880 0x7efd8414d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.322+0000 7efd83fff700 1 -- 172.21.15.46:0/3141553193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd780094f0 con 0x7efd841574b0 2024-01-27T01:08:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.322+0000 7efd83fff700 1 --2- 172.21.15.46:0/3141553193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd841574b0 0x7efd8414d430 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7efd7000b610 tx=0x7efd70009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.322+0000 7efd80ff9700 1 -- 172.21.15.46:0/3141553193 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7000f510 con 0x7efd841574b0 2024-01-27T01:08:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.322+0000 7efd80ff9700 1 -- 172.21.15.46:0/3141553193 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7000f670 con 0x7efd841574b0 2024-01-27T01:08:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.322+0000 7efd8b131700 1 -- 172.21.15.46:0/3141553193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd84153a20 con 0x7efd841574b0 2024-01-27T01:08:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.322+0000 7efd80ff9700 1 -- 172.21.15.46:0/3141553193 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7000fa30 con 0x7efd841574b0 2024-01-27T01:08:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.322+0000 7efd8b131700 1 -- 172.21.15.46:0/3141553193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd841542c0 con 0x7efd841574b0 2024-01-27T01:08:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.323+0000 7efd8b131700 1 -- 172.21.15.46:0/3141553193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd84153e20 con 0x7efd841574b0 2024-01-27T01:08:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.324+0000 7efd80ff9700 1 -- 172.21.15.46:0/3141553193 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 147) v1 ==== 84597+0+0 (secure 0 0 0) 0x7efd7001a400 con 0x7efd841574b0 2024-01-27T01:08:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.324+0000 7efd80ff9700 1 --2- 172.21.15.46:0/3141553193 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efd68066dd0 0x7efd68069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.325+0000 7efd80ff9700 1 -- 172.21.15.46:0/3141553193 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6990+0+0 (secure 0 0 0) 0x7efd70084090 con 0x7efd841574b0 2024-01-27T01:08:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.325+0000 7efd88ecd700 1 --2- 172.21.15.46:0/3141553193 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efd68066dd0 0x7efd68069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.325+0000 7efd88ecd700 1 --2- 172.21.15.46:0/3141553193 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efd68066dd0 0x7efd68069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efd840cce00 tx=0x7efd78016000 comp rx=0 tx=0).ready entity=mgr.10475 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:37.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.329+0000 7efd80ff9700 1 -- 172.21.15.46:0/3141553193 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efd70051130 con 0x7efd841574b0 2024-01-27T01:08:37.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.596+0000 7efd8b131700 1 -- 172.21.15.46:0/3141553193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7efd8404c710 con 0x7efd841574b0 2024-01-27T01:08:37.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.597+0000 7efd80ff9700 1 -- 172.21.15.46:0/3141553193 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v149) v1 ==== 53+0+73 (secure 0 0 0) 0x7efd70054750 con 0x7efd841574b0 2024-01-27T01:08:37.596 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-27T01:08:37.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.599+0000 7efd8b131700 1 -- 172.21.15.46:0/3141553193 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efd68066dd0 msgr2=0x7efd68069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:37.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.599+0000 7efd8b131700 1 --2- 172.21.15.46:0/3141553193 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efd68066dd0 0x7efd68069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efd840cce00 tx=0x7efd78016000 comp rx=0 tx=0).stop 2024-01-27T01:08:37.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.599+0000 7efd8b131700 1 -- 172.21.15.46:0/3141553193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd841574b0 msgr2=0x7efd8414d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:37.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.599+0000 7efd8b131700 1 --2- 172.21.15.46:0/3141553193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd841574b0 0x7efd8414d430 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7efd7000b610 tx=0x7efd70009120 comp rx=0 tx=0).stop 2024-01-27T01:08:37.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.599+0000 7efd8b131700 1 -- 172.21.15.46:0/3141553193 shutdown_connections 2024-01-27T01:08:37.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.599+0000 7efd8b131700 1 --2- 172.21.15.46:0/3141553193 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7efd68066dd0 0x7efd68069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:37.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.599+0000 7efd8b131700 1 --2- 172.21.15.46:0/3141553193 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd84157880 0x7efd8414d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:37.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.599+0000 7efd8b131700 1 --2- 172.21.15.46:0/3141553193 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd840b2ce0 0x7efd84167d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:37.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.599+0000 7efd8b131700 1 --2- 172.21.15.46:0/3141553193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd841574b0 0x7efd8414d430 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:37.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.599+0000 7efd8b131700 1 -- 172.21.15.46:0/3141553193 >> 172.21.15.46:0/3141553193 conn(0x7efd8406b6c0 msgr2=0x7efd8415b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:37.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.599+0000 7efd8b131700 1 -- 172.21.15.46:0/3141553193 shutdown_connections 2024-01-27T01:08:37.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:37.599+0000 7efd8b131700 1 -- 172.21.15.46:0/3141553193 wait complete. 2024-01-27T01:08:37.612 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-27T01:08:37.612 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-27T01:08:37.612 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-01-27T01:08:37.612 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-27T01:08:37.612 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-27T01:08:37.613 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-27T01:08:37.613 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-27T01:08:37.613 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-27T01:08:37.613 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-27T01:08:37.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:37.613 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-01-27T01:08:37.632 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-27T01:08:37.632 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-27T01:08:37.632 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-01-27T01:08:37.633 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:08:37.633 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-01-27T01:08:37.633 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:08:37.633 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T01:08:37.633 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:08:37.633 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-01-27T01:08:37.633 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T01:08:37.633 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:08:37.633 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-01-27T01:08:37.633 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-27T01:08:37.633 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:08:37.633 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-01-27T01:08:37.633 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-27T01:08:37.633 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 36413sec preferred_lft 36413sec 2024-01-27T01:08:37.634 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2024-01-27T01:08:37.634 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:08:37.634 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-27T01:08:37.634 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:08:37.634 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-01-27T01:08:37.634 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-27T01:08:37.634 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff 2024-01-27T01:08:37.634 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-27T01:08:37.634 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:08:37.634 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::cce6:a7ff:fe4f:38d/64 scope link 2024-01-27T01:08:37.634 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:08:37.634 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-27T01:08:37.634 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 76:ef:90:4c:9d:6f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-27T01:08:37.634 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::74ef:90ff:fe4c:9d6f/64 scope link 2024-01-27T01:08:37.634 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:08:37.635 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-27T01:08:37.635 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 2e:73:af:f0:6f:39 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-27T01:08:37.635 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::2c73:afff:fef0:6f39/64 scope link 2024-01-27T01:08:37.635 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-01-27T01:08:37.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:37.635 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-27T01:08:37.703 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-27T01:08:37.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:37.705 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-01-27T01:08:37.769 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-01-27T01:08:37.769 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-27T01:08:37.769 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-27T01:08:37.769 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-27T01:08:37.769 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-27T01:08:37.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:37.770 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-01-27T01:08:37.770 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-01-27T01:08:37.770 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-27T01:08:37.770 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-27T01:08:37.770 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-27T01:08:37.770 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-01-27T01:08:37.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:37.894 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-27T01:08:37.914 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-27T01:08:37.914 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-27T01:08:37.914 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-27T01:08:37.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:37.915 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-01-27T01:08:37.990 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-27T01:08:37.991 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-27T01:08:37.991 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-27T01:08:38.052 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-27T01:08:38.053 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-27T01:08:38.053 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-27T01:08:38.115 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-27T01:08:38.190 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-27T01:08:38.190 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-27T01:08:38.265 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-27T01:08:38.267 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-27T01:08:38.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:08:38.267 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-27T01:08:38.333 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-27T01:08:38.333 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-27T01:08:38.375 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-01-27T01:08:38.414 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-27T01:08:38.414 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-27T01:08:38.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-01-27T01:08:38.481+0000 7f2926aa0540 -1 init, newargv = 0x557c54bdd390 newargc=15 2024-01-27T01:08:38.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[122957]: starting ceph client 2024-01-27T01:08:38.485 INFO:teuthology.orchestra.run.smithi046.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-27T01:08:38.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[122957]: starting fuse 2024-01-27T01:08:38.487 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-27T01:08:38.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:08:38.487 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-01-27T01:08:38.547 INFO:teuthology.orchestra.run.smithi046.stdout:47 2024-01-27T01:08:38.547 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-01-27T01:08:38.548 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-27T01:08:38.548 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-01-27T01:08:38.548 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-01-27T01:08:38.548 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-01-27T01:08:38.548 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-01-27T01:08:38.548 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:08:38.548 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-01-27T01:08:38.548 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-27T01:08:38.548 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-01-27T01:08:38.548 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-27T01:08:38.548 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:08:38.548 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-01-27T01:08:38.548 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-01-27T01:08:38.549 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-01-27T01:08:38.549 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-01-27T01:08:38.549 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:08:38.549 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-01-27T01:08:38.549 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-27T01:08:38.549 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-27T01:08:38.549 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-27T01:08:38.549 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-01-27T01:08:38.549 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-01-27T01:08:38.549 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-01-27T01:08:38.549 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-27T01:08:38.549 DEBUG:teuthology.orchestra.run.smithi046:> 2024-01-27T01:08:38.549 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-01-27T01:08:38.549 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-01-27T01:08:38.687 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.122957.asok 2024-01-27T01:08:38.868 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.122957.asok 2024-01-27T01:08:38.869 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:08:38.869 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.122957.asok status 2024-01-27T01:08:39.043 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:08:39.044 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-01-27T01:08:39.044 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7495007658df82356749387b98425d4a7fcef9b6", 2024-01-27T01:08:39.044 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.7-472-g74950076 (7495007658df82356749387b98425d4a7fcef9b6) quincy (stable)", 2024-01-27T01:08:39.044 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-01-27T01:08:39.044 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-01-27T01:08:39.044 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-27T01:08:39.044 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "122957", 2024-01-27T01:08:39.044 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-01-27T01:08:39.044 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:39.044 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-01-27T01:08:39.045 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-01-27T01:08:39.045 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 12621, 2024-01-27T01:08:39.045 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-01-27T01:08:39.045 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-01-27T01:08:39.045 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-01-27T01:08:39.045 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 12621 2024-01-27T01:08:39.045 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:39.045 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-27T01:08:39.045 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-27T01:08:39.045 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-27T01:08:39.045 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 856139789 2024-01-27T01:08:39.045 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-01-27T01:08:39.045 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:39.045 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-01-27T01:08:39.045 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-01-27T01:08:39.046 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-01-27T01:08:39.046 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 856139789 2024-01-27T01:08:39.046 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-01-27T01:08:39.046 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.12621 192.168.0.1:0/856139789", 2024-01-27T01:08:39.046 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/856139789", 2024-01-27T01:08:39.046 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-01-27T01:08:39.046 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 149, 2024-01-27T01:08:39.046 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 265, 2024-01-27T01:08:39.046 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-01-27T01:08:39.046 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-01-27T01:08:39.046 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-01-27T01:08:39.046 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:08:39.055 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:08:39.055 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-01-27T01:08:39.055 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-27T01:08:39.076 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-01-27T01:08:39.076 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-27T01:08:39.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:08:39.077 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-27T01:08:39.150 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-01-27-01_09_00 in ~24s... 2024-01-27T01:08:39.150 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-01-27T01:08:39.150 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-01-27T01:08:39.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.334+0000 7fea8b03f700 1 Processor -- start 2024-01-27T01:08:39.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.334+0000 7fea8b03f700 1 -- start start 2024-01-27T01:08:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.336+0000 7fea8b03f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea841565a0 0x7fea84156970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.336+0000 7fea8b03f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fea8406e9d0 0x7fea84156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.336+0000 7fea8b03f700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea840c8410 con 0x7fea840c7280 2024-01-27T01:08:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.336+0000 7fea8b03f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea840c7800 con 0x7fea8406e9d0 2024-01-27T01:08:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.336+0000 7fea8b03f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea840c7ae0 con 0x7fea841565a0 2024-01-27T01:08:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.336+0000 7fea83fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fea840c7280 0x7fea8406e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43498/0 (socket says 172.21.15.46:43498) 2024-01-27T01:08:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.336+0000 7fea83fff700 1 -- 172.21.15.46:0/2603004334 learned_addr learned my addr 172.21.15.46:0/2603004334 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.336+0000 7fea895dc700 1 --2- 172.21.15.46:0/2603004334 >> v2:172.21.15.106:3300/0 conn(0x7fea8406e9d0 0x7fea84156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.336+0000 7fea88ddb700 1 --2- 172.21.15.46:0/2603004334 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea841565a0 0x7fea84156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.336+0000 7fea837fe700 1 -- 172.21.15.46:0/2603004334 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512057364 0 0) 0x7fea840c8410 con 0x7fea840c7280 2024-01-27T01:08:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.336+0000 7fea837fe700 1 -- 172.21.15.46:0/2603004334 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea6c003570 con 0x7fea840c7280 2024-01-27T01:08:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.336+0000 7fea895dc700 1 -- 172.21.15.46:0/2603004334 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea841565a0 msgr2=0x7fea84156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.336+0000 7fea837fe700 1 -- 172.21.15.46:0/2603004334 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1866379158 0 0) 0x7fea6c003570 con 0x7fea840c7280 2024-01-27T01:08:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.336+0000 7fea895dc700 1 --2- 172.21.15.46:0/2603004334 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea841565a0 0x7fea84156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:39.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.336+0000 7fea895dc700 1 -- 172.21.15.46:0/2603004334 >> v1:172.21.15.46:6789/0 conn(0x7fea840c7280 legacy=0x7fea8406e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:39.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.336+0000 7fea895dc700 1 -- 172.21.15.46:0/2603004334 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea840c7dc0 con 0x7fea8406e9d0 2024-01-27T01:08:39.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.337+0000 7fea895dc700 1 --2- 172.21.15.46:0/2603004334 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea8406e9d0 0x7fea84156eb0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fea7c009e60 tx=0x7fea7c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bae64a38de8f6c61 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:39.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.337+0000 7fea837fe700 1 -- 172.21.15.46:0/2603004334 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c00e070 con 0x7fea8406e9d0 2024-01-27T01:08:39.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.337+0000 7fea837fe700 1 -- 172.21.15.46:0/2603004334 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7c00b7a0 con 0x7fea8406e9d0 2024-01-27T01:08:39.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.337+0000 7fea837fe700 1 -- 172.21.15.46:0/2603004334 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78001470 con 0x7fea8406e9d0 2024-01-27T01:08:39.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.337+0000 7fea8b03f700 1 -- 172.21.15.46:0/2603004334 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea8406e9d0 msgr2=0x7fea84156eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:39.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.337+0000 7fea8b03f700 1 --2- 172.21.15.46:0/2603004334 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea8406e9d0 0x7fea84156eb0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fea7c009e60 tx=0x7fea7c009780 comp rx=0 tx=0).stop 2024-01-27T01:08:39.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.337+0000 7fea8b03f700 1 -- 172.21.15.46:0/2603004334 shutdown_connections 2024-01-27T01:08:39.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.337+0000 7fea8b03f700 1 --2- 172.21.15.46:0/2603004334 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea8406e9d0 0x7fea84156eb0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:39.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.337+0000 7fea8b03f700 1 --2- 172.21.15.46:0/2603004334 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea841565a0 0x7fea84156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:39.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.338+0000 7fea8b03f700 1 -- 172.21.15.46:0/2603004334 >> 172.21.15.46:0/2603004334 conn(0x7fea840ccc40 msgr2=0x7fea840cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.338+0000 7fea8b03f700 1 -- 172.21.15.46:0/2603004334 shutdown_connections 2024-01-27T01:08:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.338+0000 7fea8b03f700 1 -- 172.21.15.46:0/2603004334 wait complete. 2024-01-27T01:08:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.339+0000 7fea8b03f700 1 Processor -- start 2024-01-27T01:08:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.339+0000 7fea8b03f700 1 -- start start 2024-01-27T01:08:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea8b03f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea8406e9d0 0x7fea840c4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea8b03f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea840c7280 0x7fea84165270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea8b03f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea841565a0 0x7fea84152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea8b03f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea84166440 con 0x7fea840c7280 2024-01-27T01:08:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea8b03f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea840c1bf0 con 0x7fea841565a0 2024-01-27T01:08:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea8b03f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea840c1ed0 con 0x7fea8406e9d0 2024-01-27T01:08:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea83fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea840c7280 0x7fea84165270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea83fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea840c7280 0x7fea84165270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34826/0 (socket says 172.21.15.46:34826) 2024-01-27T01:08:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea83fff700 1 -- 172.21.15.46:0/1908390072 learned_addr learned my addr 172.21.15.46:0/1908390072 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea88ddb700 1 --2- 172.21.15.46:0/1908390072 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea8406e9d0 0x7fea840c4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea895dc700 1 --2- 172.21.15.46:0/1908390072 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea841565a0 0x7fea84152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea83fff700 1 -- 172.21.15.46:0/1908390072 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea8406e9d0 msgr2=0x7fea840c4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea83fff700 1 --2- 172.21.15.46:0/1908390072 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea8406e9d0 0x7fea840c4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea83fff700 1 -- 172.21.15.46:0/1908390072 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea841565a0 msgr2=0x7fea84152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea83fff700 1 --2- 172.21.15.46:0/1908390072 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea841565a0 0x7fea84152c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea83fff700 1 -- 172.21.15.46:0/1908390072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea7c009a20 con 0x7fea840c7280 2024-01-27T01:08:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea88ddb700 1 --2- 172.21.15.46:0/1908390072 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea8406e9d0 0x7fea840c4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.340+0000 7fea895dc700 1 --2- 172.21.15.46:0/1908390072 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea841565a0 0x7fea84152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.341+0000 7fea83fff700 1 --2- 172.21.15.46:0/1908390072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea840c7280 0x7fea84165270 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fea7800cb90 tx=0x7fea78001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.341+0000 7fea80ff9700 1 -- 172.21.15.46:0/1908390072 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78007360 con 0x7fea840c7280 2024-01-27T01:08:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.341+0000 7fea80ff9700 1 -- 172.21.15.46:0/1908390072 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea780074e0 con 0x7fea840c7280 2024-01-27T01:08:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.341+0000 7fea80ff9700 1 -- 172.21.15.46:0/1908390072 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78004830 con 0x7fea840c7280 2024-01-27T01:08:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.341+0000 7fea8b03f700 1 -- 172.21.15.46:0/1908390072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea84165ac0 con 0x7fea840c7280 2024-01-27T01:08:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.341+0000 7fea8b03f700 1 -- 172.21.15.46:0/1908390072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea84166080 con 0x7fea840c7280 2024-01-27T01:08:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.342+0000 7fea8b03f700 1 -- 172.21.15.46:0/1908390072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea840c2040 con 0x7fea840c7280 2024-01-27T01:08:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.343+0000 7fea80ff9700 1 -- 172.21.15.46:0/1908390072 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 148) v1 ==== 84862+0+0 (secure 0 0 0) 0x7fea78004ba0 con 0x7fea840c7280 2024-01-27T01:08:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.343+0000 7fea80ff9700 1 --2- 172.21.15.46:0/1908390072 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea700670a0 0x7fea70069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.343+0000 7fea80ff9700 1 -- 172.21.15.46:0/1908390072 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6990+0+0 (secure 0 0 0) 0x7fea78084a00 con 0x7fea840c7280 2024-01-27T01:08:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.343+0000 7fea88ddb700 1 --2- 172.21.15.46:0/1908390072 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea700670a0 0x7fea70069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.344+0000 7fea88ddb700 1 --2- 172.21.15.46:0/1908390072 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea700670a0 0x7fea70069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fea7400ae00 tx=0x7fea74008040 comp rx=0 tx=0).ready entity=mgr.10475 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.347+0000 7fea80ff9700 1 -- 172.21.15.46:0/1908390072 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fea780516c0 con 0x7fea840c7280 2024-01-27T01:08:39.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.449+0000 7fea80ff9700 1 -- 172.21.15.46:0/1908390072 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 149) v1 ==== 84918+0+0 (secure 0 0 0) 0x7fea78051a90 con 0x7fea840c7280 2024-01-27T01:08:39.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.579+0000 7fea8b03f700 1 -- 172.21.15.46:0/1908390072 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fea840c80a0 con 0x7fea700670a0 2024-01-27T01:08:39.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.587+0000 7fea80ff9700 1 -- 172.21.15.46:0/1908390072 <== mgr.10475 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7fea840c80a0 con 0x7fea700670a0 2024-01-27T01:08:39.586 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:39.586 INFO:teuthology.orchestra.run.smithi046.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-01-27T01:08:39.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.589+0000 7fea8b03f700 1 -- 172.21.15.46:0/1908390072 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea700670a0 msgr2=0x7fea70069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.589+0000 7fea8b03f700 1 --2- 172.21.15.46:0/1908390072 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea700670a0 0x7fea70069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fea7400ae00 tx=0x7fea74008040 comp rx=0 tx=0).stop 2024-01-27T01:08:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.589+0000 7fea8b03f700 1 -- 172.21.15.46:0/1908390072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea840c7280 msgr2=0x7fea84165270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.589+0000 7fea8b03f700 1 --2- 172.21.15.46:0/1908390072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea840c7280 0x7fea84165270 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fea7800cb90 tx=0x7fea78001b50 comp rx=0 tx=0).stop 2024-01-27T01:08:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.589+0000 7fea8b03f700 1 -- 172.21.15.46:0/1908390072 shutdown_connections 2024-01-27T01:08:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.589+0000 7fea8b03f700 1 --2- 172.21.15.46:0/1908390072 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fea700670a0 0x7fea70069550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.589+0000 7fea8b03f700 1 --2- 172.21.15.46:0/1908390072 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea841565a0 0x7fea84152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.589+0000 7fea8b03f700 1 --2- 172.21.15.46:0/1908390072 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea8406e9d0 0x7fea840c4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.589+0000 7fea8b03f700 1 --2- 172.21.15.46:0/1908390072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea840c7280 0x7fea84165270 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.589+0000 7fea8b03f700 1 -- 172.21.15.46:0/1908390072 >> 172.21.15.46:0/1908390072 conn(0x7fea840ccc40 msgr2=0x7fea84070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.589+0000 7fea8b03f700 1 -- 172.21.15.46:0/1908390072 shutdown_connections 2024-01-27T01:08:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.590+0000 7fea8b03f700 1 -- 172.21.15.46:0/1908390072 wait complete. 2024-01-27T01:08:39.602 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-01-27T01:08:39.603 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-01-27T01:08:39.603 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-01-27T01:08:39.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.738+0000 7f3febf0f700 1 Processor -- start 2024-01-27T01:08:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.738+0000 7f3febf0f700 1 -- start start 2024-01-27T01:08:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.739+0000 7f3febf0f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3fe4156980 0x7f3fe40ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.739+0000 7f3febf0f700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3fe40cb7b0 0x7f3fe40c29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.739+0000 7f3febf0f700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe40c2270 con 0x7f3fe40cdf20 2024-01-27T01:08:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.739+0000 7f3febf0f700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe40c2550 con 0x7f3fe4156980 2024-01-27T01:08:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.739+0000 7f3febf0f700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe40ac9c0 con 0x7f3fe40cb7b0 2024-01-27T01:08:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.739+0000 7f3fe9cab700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3fe4156980 0x7f3fe40ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.739+0000 7f3fe9cab700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3fe4156980 0x7f3fe40ac0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34838/0 (socket says 172.21.15.46:34838) 2024-01-27T01:08:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.739+0000 7f3fe9cab700 1 -- 172.21.15.46:0/1370681587 learned_addr learned my addr 172.21.15.46:0/1370681587 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.740+0000 7f3fea4ac700 1 --2- 172.21.15.46:0/1370681587 >> v2:172.21.15.106:3300/0 conn(0x7f3fe40cb7b0 0x7f3fe40c29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.740+0000 7f3fe9cab700 1 -- 172.21.15.46:0/1370681587 >> v2:172.21.15.106:3300/0 conn(0x7f3fe40cb7b0 msgr2=0x7f3fe40c29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.740+0000 7f3fe9cab700 1 --2- 172.21.15.46:0/1370681587 >> v2:172.21.15.106:3300/0 conn(0x7f3fe40cb7b0 0x7f3fe40c29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.740+0000 7f3fe9cab700 1 -- 172.21.15.46:0/1370681587 >> v1:172.21.15.106:6789/0 conn(0x7f3fe40cdf20 legacy=0x7f3fe40cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.740+0000 7f3fe9cab700 1 -- 172.21.15.46:0/1370681587 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe40cbc80 con 0x7f3fe4156980 2024-01-27T01:08:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.740+0000 7f3fe9cab700 1 --2- 172.21.15.46:0/1370681587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fe4156980 0x7f3fe40ac0e0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00099b0 tx=0x7f3fe0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb8b698f0d5d7e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.740+0000 7f3fe8ca9700 1 -- 172.21.15.46:0/1370681587 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe000b820 con 0x7f3fe4156980 2024-01-27T01:08:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.740+0000 7f3fe8ca9700 1 -- 172.21.15.46:0/1370681587 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe000b980 con 0x7f3fe4156980 2024-01-27T01:08:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.740+0000 7f3fe8ca9700 1 -- 172.21.15.46:0/1370681587 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe000bd20 con 0x7f3fe4156980 2024-01-27T01:08:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.741+0000 7f3febf0f700 1 -- 172.21.15.46:0/1370681587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fe4156980 msgr2=0x7f3fe40ac0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.741+0000 7f3febf0f700 1 --2- 172.21.15.46:0/1370681587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fe4156980 0x7f3fe40ac0e0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00099b0 tx=0x7f3fe0009250 comp rx=0 tx=0).stop 2024-01-27T01:08:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.741+0000 7f3febf0f700 1 -- 172.21.15.46:0/1370681587 shutdown_connections 2024-01-27T01:08:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.741+0000 7f3febf0f700 1 --2- 172.21.15.46:0/1370681587 >> v2:172.21.15.106:3300/0 conn(0x7f3fe40cb7b0 0x7f3fe40c29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.741+0000 7f3febf0f700 1 --2- 172.21.15.46:0/1370681587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fe4156980 0x7f3fe40ac0e0 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.741+0000 7f3febf0f700 1 -- 172.21.15.46:0/1370681587 >> 172.21.15.46:0/1370681587 conn(0x7f3fe40692a0 msgr2=0x7f3fe40cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.741+0000 7f3febf0f700 1 -- 172.21.15.46:0/1370681587 shutdown_connections 2024-01-27T01:08:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.741+0000 7f3febf0f700 1 -- 172.21.15.46:0/1370681587 wait complete. 2024-01-27T01:08:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.742+0000 7f3febf0f700 1 Processor -- start 2024-01-27T01:08:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.742+0000 7f3febf0f700 1 -- start start 2024-01-27T01:08:39.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.743+0000 7f3febf0f700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fe40cb7b0 0x7f3fe4150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:39.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.743+0000 7f3febf0f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fe40cdf20 0x7f3fe4152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:39.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.743+0000 7f3febf0f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fe4156980 0x7f3fe4152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:39.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.743+0000 7f3febf0f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe40c52e0 con 0x7f3fe40cdf20 2024-01-27T01:08:39.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.743+0000 7f3febf0f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe40c5470 con 0x7f3fe4156980 2024-01-27T01:08:39.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.743+0000 7f3febf0f700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe4150ac0 con 0x7f3fe40cb7b0 2024-01-27T01:08:39.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.743+0000 7f3fe94aa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fe40cdf20 0x7f3fe4152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:39.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.743+0000 7f3fe94aa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fe40cdf20 0x7f3fe4152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34854/0 (socket says 172.21.15.46:34854) 2024-01-27T01:08:39.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.743+0000 7f3fe94aa700 1 -- 172.21.15.46:0/3946762788 learned_addr learned my addr 172.21.15.46:0/3946762788 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:39.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.743+0000 7f3fea4ac700 1 --2- 172.21.15.46:0/3946762788 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fe4156980 0x7f3fe4152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.743+0000 7f3fe9cab700 1 --2- 172.21.15.46:0/3946762788 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fe40cb7b0 0x7f3fe4150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.743+0000 7f3fe94aa700 1 -- 172.21.15.46:0/3946762788 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fe40cb7b0 msgr2=0x7f3fe4150580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.743+0000 7f3fe94aa700 1 --2- 172.21.15.46:0/3946762788 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fe40cb7b0 0x7f3fe4150580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.743+0000 7f3fe94aa700 1 -- 172.21.15.46:0/3946762788 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fe4156980 msgr2=0x7f3fe4152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.743+0000 7f3fe94aa700 1 --2- 172.21.15.46:0/3946762788 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fe4156980 0x7f3fe4152e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.743+0000 7f3fe94aa700 1 -- 172.21.15.46:0/3946762788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe00094f0 con 0x7f3fe40cdf20 2024-01-27T01:08:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.744+0000 7f3fe94aa700 1 --2- 172.21.15.46:0/3946762788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fe40cdf20 0x7f3fe4152900 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800c590 tx=0x7f3fd800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.744+0000 7f3fd5ffb700 1 -- 172.21.15.46:0/3946762788 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8011590 con 0x7f3fe40cdf20 2024-01-27T01:08:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.744+0000 7f3fd5ffb700 1 -- 172.21.15.46:0/3946762788 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd80116f0 con 0x7f3fe40cdf20 2024-01-27T01:08:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.744+0000 7f3fd5ffb700 1 -- 172.21.15.46:0/3946762788 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8011ab0 con 0x7f3fe40cdf20 2024-01-27T01:08:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.744+0000 7f3febf0f700 1 -- 172.21.15.46:0/3946762788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe4154860 con 0x7f3fe40cdf20 2024-01-27T01:08:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.744+0000 7f3febf0f700 1 -- 172.21.15.46:0/3946762788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe4154e20 con 0x7f3fe40cdf20 2024-01-27T01:08:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.745+0000 7f3febf0f700 1 -- 172.21.15.46:0/3946762788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe40c55e0 con 0x7f3fe40cdf20 2024-01-27T01:08:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.745+0000 7f3fd5ffb700 1 -- 172.21.15.46:0/3946762788 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 149) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f3fd801b3b0 con 0x7f3fe40cdf20 2024-01-27T01:08:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.746+0000 7f3fd5ffb700 1 --2- 172.21.15.46:0/3946762788 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fc8067130 0x7f3fc80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.746+0000 7f3fd5ffb700 1 -- 172.21.15.46:0/3946762788 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6990+0+0 (secure 0 0 0) 0x7f3fd8084360 con 0x7f3fe40cdf20 2024-01-27T01:08:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.746+0000 7f3fe9cab700 1 --2- 172.21.15.46:0/3946762788 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fc8067130 0x7f3fc80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.747+0000 7f3fe9cab700 1 --2- 172.21.15.46:0/3946762788 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fc8067130 0x7f3fc80695e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00099b0 tx=0x7f3fe0017000 comp rx=0 tx=0).ready entity=mgr.10475 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.750+0000 7f3fd5ffb700 1 -- 172.21.15.46:0/3946762788 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3fd8050e50 con 0x7f3fe40cdf20 2024-01-27T01:08:39.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:39.983+0000 7f3febf0f700 1 -- 172.21.15.46:0/3946762788 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fe40cbf60 con 0x7f3fc8067130 2024-01-27T01:08:40.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.043+0000 7f3fd5ffb700 1 -- 172.21.15.46:0/3946762788 <== mgr.10475 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f3fe40cbf60 con 0x7f3fc8067130 2024-01-27T01:08:40.042 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule removed for path snap_test_dir1 2024-01-27T01:08:40.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.046+0000 7f3febf0f700 1 -- 172.21.15.46:0/3946762788 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fc8067130 msgr2=0x7f3fc80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:40.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.046+0000 7f3febf0f700 1 --2- 172.21.15.46:0/3946762788 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fc8067130 0x7f3fc80695e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00099b0 tx=0x7f3fe0017000 comp rx=0 tx=0).stop 2024-01-27T01:08:40.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.046+0000 7f3febf0f700 1 -- 172.21.15.46:0/3946762788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fe40cdf20 msgr2=0x7f3fe4152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:40.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.046+0000 7f3febf0f700 1 --2- 172.21.15.46:0/3946762788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fe40cdf20 0x7f3fe4152900 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800c590 tx=0x7f3fd800c950 comp rx=0 tx=0).stop 2024-01-27T01:08:40.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.046+0000 7f3febf0f700 1 -- 172.21.15.46:0/3946762788 shutdown_connections 2024-01-27T01:08:40.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.046+0000 7f3febf0f700 1 --2- 172.21.15.46:0/3946762788 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3fc8067130 0x7f3fc80695e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:40.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.046+0000 7f3febf0f700 1 --2- 172.21.15.46:0/3946762788 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fe4156980 0x7f3fe4152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:40.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.046+0000 7f3febf0f700 1 --2- 172.21.15.46:0/3946762788 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fe40cb7b0 0x7f3fe4150580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:40.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.046+0000 7f3febf0f700 1 --2- 172.21.15.46:0/3946762788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fe40cdf20 0x7f3fe4152900 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:40.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.046+0000 7f3febf0f700 1 -- 172.21.15.46:0/3946762788 >> 172.21.15.46:0/3946762788 conn(0x7f3fe40692a0 msgr2=0x7f3fe406bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:40.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.046+0000 7f3febf0f700 1 -- 172.21.15.46:0/3946762788 shutdown_connections 2024-01-27T01:08:40.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.046+0000 7f3febf0f700 1 -- 172.21.15.46:0/3946762788 wait complete. 2024-01-27T01:08:40.060 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-01-27T01:08:40.060 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:08:40.060 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-27T01:08:40.086 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-27T01:08:40.086 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-01-27T01:08:40.154 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-27T01:08:40.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.339+0000 7f704f710700 1 Processor -- start 2024-01-27T01:08:40.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.339+0000 7f704f710700 1 -- start start 2024-01-27T01:08:40.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.340+0000 7f704f710700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70480c8160 0x7f7048164a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:40.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.340+0000 7f704f710700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704815dec0 con 0x7f7048159a60 2024-01-27T01:08:40.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.340+0000 7f704f710700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704806ba90 con 0x7f704815e350 2024-01-27T01:08:40.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.340+0000 7f704f710700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70480afc00 con 0x7f70480c8160 2024-01-27T01:08:40.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.340+0000 7f704d4ac700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7048159a60 0x7f704806aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59286/0 (socket says 172.21.15.46:59286) 2024-01-27T01:08:40.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.340+0000 7f704d4ac700 1 -- 172.21.15.46:0/3626606820 learned_addr learned my addr 172.21.15.46:0/3626606820 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:40.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.340+0000 7f704dcad700 1 --2- 172.21.15.46:0/3626606820 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70480c8160 0x7f7048164a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:40.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.340+0000 7f703ffff700 1 -- 172.21.15.46:0/3626606820 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043402798 0 0) 0x7f704815dec0 con 0x7f7048159a60 2024-01-27T01:08:40.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.340+0000 7f703ffff700 1 -- 172.21.15.46:0/3626606820 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7034003570 con 0x7f7048159a60 2024-01-27T01:08:40.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.340+0000 7f703ffff700 1 -- 172.21.15.46:0/3626606820 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341322906 0 0) 0x7f704806ba90 con 0x7f704815e350 2024-01-27T01:08:40.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.340+0000 7f703ffff700 1 -- 172.21.15.46:0/3626606820 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f704815dec0 con 0x7f704815e350 2024-01-27T01:08:40.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.340+0000 7f704dcad700 1 -- 172.21.15.46:0/3626606820 >> v1:172.21.15.106:6789/0 conn(0x7f704815e350 legacy=0x7f70480c7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:40.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.340+0000 7f704dcad700 1 -- 172.21.15.46:0/3626606820 >> v1:172.21.15.46:6789/0 conn(0x7f7048159a60 legacy=0x7f704806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:40.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.340+0000 7f703ffff700 1 -- 172.21.15.46:0/3626606820 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3343326934 0 0) 0x7f7034003570 con 0x7f7048159a60 2024-01-27T01:08:40.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.340+0000 7f704dcad700 1 -- 172.21.15.46:0/3626606820 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704815d870 con 0x7f70480c8160 2024-01-27T01:08:40.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.341+0000 7f704dcad700 1 --2- 172.21.15.46:0/3626606820 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70480c8160 0x7f7048164a30 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7040009e90 tx=0x7f7040009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf0512d8d01f4edb server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:40.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.341+0000 7f703ffff700 1 -- 172.21.15.46:0/3626606820 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704000e070 con 0x7f70480c8160 2024-01-27T01:08:40.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.341+0000 7f703ffff700 1 -- 172.21.15.46:0/3626606820 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704000b870 con 0x7f70480c8160 2024-01-27T01:08:40.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.341+0000 7f703ffff700 1 -- 172.21.15.46:0/3626606820 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7044001470 con 0x7f70480c8160 2024-01-27T01:08:40.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.341+0000 7f704f710700 1 -- 172.21.15.46:0/3626606820 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70480c8160 msgr2=0x7f7048164a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:40.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.341+0000 7f704f710700 1 --2- 172.21.15.46:0/3626606820 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70480c8160 0x7f7048164a30 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7040009e90 tx=0x7f7040009780 comp rx=0 tx=0).stop 2024-01-27T01:08:40.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.342+0000 7f704f710700 1 -- 172.21.15.46:0/3626606820 shutdown_connections 2024-01-27T01:08:40.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.342+0000 7f704f710700 1 --2- 172.21.15.46:0/3626606820 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f70480c8160 0x7f7048164a30 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:40.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.342+0000 7f704f710700 1 -- 172.21.15.46:0/3626606820 >> 172.21.15.46:0/3626606820 conn(0x7f7048068ac0 msgr2=0x7f7048068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:40.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.342+0000 7f704f710700 1 -- 172.21.15.46:0/3626606820 shutdown_connections 2024-01-27T01:08:40.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.342+0000 7f704f710700 1 -- 172.21.15.46:0/3626606820 wait complete. 2024-01-27T01:08:40.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.343+0000 7f704f710700 1 Processor -- start 2024-01-27T01:08:40.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.343+0000 7f704f710700 1 -- start start 2024-01-27T01:08:40.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.344+0000 7f704f710700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70480c8160 0x7f70480aaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:40.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.344+0000 7f704f710700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7048159a60 0x7f70480c33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:40.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704f710700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f704815e350 0x7f70480c0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:40.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704f710700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70480ac500 con 0x7f70480c8160 2024-01-27T01:08:40.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704f710700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70480aaa20 con 0x7f7048159a60 2024-01-27T01:08:40.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704f710700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70480ac220 con 0x7f704815e350 2024-01-27T01:08:40.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704d4ac700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70480c8160 0x7f70480aaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:40.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704d4ac700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70480c8160 0x7f70480aaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48138/0 (socket says 172.21.15.46:48138) 2024-01-27T01:08:40.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704d4ac700 1 -- 172.21.15.46:0/3459853248 learned_addr learned my addr 172.21.15.46:0/3459853248 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:40.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704dcad700 1 --2- 172.21.15.46:0/3459853248 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f704815e350 0x7f70480c0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:40.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704ccab700 1 --2- 172.21.15.46:0/3459853248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7048159a60 0x7f70480c33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:40.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704d4ac700 1 -- 172.21.15.46:0/3459853248 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f704815e350 msgr2=0x7f70480c0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:40.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704d4ac700 1 --2- 172.21.15.46:0/3459853248 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f704815e350 0x7f70480c0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:40.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704d4ac700 1 -- 172.21.15.46:0/3459853248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7048159a60 msgr2=0x7f70480c33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:40.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704d4ac700 1 --2- 172.21.15.46:0/3459853248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7048159a60 0x7f70480c33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:40.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704d4ac700 1 -- 172.21.15.46:0/3459853248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7040009a20 con 0x7f70480c8160 2024-01-27T01:08:40.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704ccab700 1 --2- 172.21.15.46:0/3459853248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7048159a60 0x7f70480c33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:40.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704d4ac700 1 --2- 172.21.15.46:0/3459853248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70480c8160 0x7f70480aaeb0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f704400c810 tx=0x7f7044001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:40.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f703d7fa700 1 -- 172.21.15.46:0/3459853248 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7044007360 con 0x7f70480c8160 2024-01-27T01:08:40.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f703d7fa700 1 -- 172.21.15.46:0/3459853248 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70440074e0 con 0x7f70480c8160 2024-01-27T01:08:40.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704f710700 1 -- 172.21.15.46:0/3459853248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70480aa3d0 con 0x7f70480c8160 2024-01-27T01:08:40.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f703d7fa700 1 -- 172.21.15.46:0/3459853248 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7044004850 con 0x7f70480c8160 2024-01-27T01:08:40.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.345+0000 7f704f710700 1 -- 172.21.15.46:0/3459853248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70480c38e0 con 0x7f70480c8160 2024-01-27T01:08:40.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.346+0000 7f704f710700 1 -- 172.21.15.46:0/3459853248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70480aa240 con 0x7f70480c8160 2024-01-27T01:08:40.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.347+0000 7f703d7fa700 1 -- 172.21.15.46:0/3459853248 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 149) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f7044004bc0 con 0x7f70480c8160 2024-01-27T01:08:40.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.347+0000 7f703d7fa700 1 --2- 172.21.15.46:0/3459853248 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f702c067130 0x7f702c0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:40.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.347+0000 7f703d7fa700 1 -- 172.21.15.46:0/3459853248 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6990+0+0 (secure 0 0 0) 0x7f7044084390 con 0x7f70480c8160 2024-01-27T01:08:40.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.348+0000 7f704ccab700 1 --2- 172.21.15.46:0/3459853248 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f702c067130 0x7f702c0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:40.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.348+0000 7f704ccab700 1 --2- 172.21.15.46:0/3459853248 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f702c067130 0x7f702c0695e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7048157490 tx=0x7f7038009040 comp rx=0 tx=0).ready entity=mgr.10475 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:40.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.352+0000 7f703d7fa700 1 -- 172.21.15.46:0/3459853248 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7044050f00 con 0x7f70480c8160 2024-01-27T01:08:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:40.617+0000 7f704f710700 1 -- 172.21.15.46:0/3459853248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f704804bed0 con 0x7f70480c8160 2024-01-27T01:08:41.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.455+0000 7f703d7fa700 1 -- 172.21.15.46:0/3459853248 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f70440512d0 con 0x7f70480c8160 2024-01-27T01:08:41.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.457+0000 7f703d7fa700 1 -- 172.21.15.46:0/3459853248 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v150) v1 ==== 93+0+0 (secure 0 0 0) 0x7f7044050f00 con 0x7f70480c8160 2024-01-27T01:08:41.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.459+0000 7f704f710700 1 -- 172.21.15.46:0/3459853248 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f702c067130 msgr2=0x7f702c0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:41.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.459+0000 7f704f710700 1 --2- 172.21.15.46:0/3459853248 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f702c067130 0x7f702c0695e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7048157490 tx=0x7f7038009040 comp rx=0 tx=0).stop 2024-01-27T01:08:41.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.459+0000 7f704f710700 1 -- 172.21.15.46:0/3459853248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70480c8160 msgr2=0x7f70480aaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:41.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.459+0000 7f704f710700 1 --2- 172.21.15.46:0/3459853248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70480c8160 0x7f70480aaeb0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f704400c810 tx=0x7f7044001a30 comp rx=0 tx=0).stop 2024-01-27T01:08:41.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.460+0000 7f704f710700 1 -- 172.21.15.46:0/3459853248 shutdown_connections 2024-01-27T01:08:41.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.460+0000 7f704f710700 1 --2- 172.21.15.46:0/3459853248 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f702c067130 0x7f702c0695e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:41.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.460+0000 7f704f710700 1 --2- 172.21.15.46:0/3459853248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70480c8160 0x7f70480aaeb0 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:41.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.460+0000 7f704f710700 1 --2- 172.21.15.46:0/3459853248 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f704815e350 0x7f70480c0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:41.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.460+0000 7f704f710700 1 --2- 172.21.15.46:0/3459853248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7048159a60 0x7f70480c33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:41.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.460+0000 7f704f710700 1 -- 172.21.15.46:0/3459853248 >> 172.21.15.46:0/3459853248 conn(0x7f7048068ac0 msgr2=0x7f704806aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:41.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.460+0000 7f704f710700 1 -- 172.21.15.46:0/3459853248 shutdown_connections 2024-01-27T01:08:41.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.460+0000 7f704f710700 1 -- 172.21.15.46:0/3459853248 wait complete. 2024-01-27T01:08:41.475 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-27T01:08:41.476 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:08:41.525 DEBUG:teuthology.orchestra.run.smithi106:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-27T01:08:41.566 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-27T01:08:41.566 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-01-27T01:08:41.566 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-27T01:08:41.566 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-27T01:08:41.603 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[122957]: fuse finished with error 0 and tester_r 0 2024-01-27T01:08:41.605 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T01:08:41.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:41.605 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-27T01:08:41.674 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T01:08:41.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:41.674 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-27T01:08:41.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:08:41.691 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-27T01:08:41.691 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T01:08:41.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:41.691 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-27T01:08:41.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:08:41.732 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-27T01:08:41.732 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T01:08:41.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:41.733 DEBUG:teuthology.orchestra.run.smithi106:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-27T01:08:41.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:08:41.749 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-27T01:08:41.750 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:08:41.800 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:41.801+0000 7f4965bd41c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:08:41.843 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:41.843+0000 7f66656d31c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T01:08:41.876 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:41.878+0000 7f4965bd41c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:08:41.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.907+0000 7fa6f3071700 1 Processor -- start 2024-01-27T01:08:41.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.907+0000 7fa6f3071700 1 -- start start 2024-01-27T01:08:41.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.908+0000 7fa6f3071700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa6ec157710 0x7fa6ec0aed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:41.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.908+0000 7fa6f3071700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ec06eab0 con 0x7fa6ec0b0260 2024-01-27T01:08:41.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.908+0000 7fa6f3071700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ec0af430 con 0x7fa6ec157ae0 2024-01-27T01:08:41.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.908+0000 7fa6f3071700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6ec0af710 con 0x7fa6ec157710 2024-01-27T01:08:41.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.908+0000 7fa6f0e0d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa6ec157710 0x7fa6ec0aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:41.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.908+0000 7fa6f160e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa6ec0b0260 0x7fa6ec0c9eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59296/0 (socket says 172.21.15.46:59296) 2024-01-27T01:08:41.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.908+0000 7fa6f160e700 1 -- 172.21.15.46:0/1537755160 learned_addr learned my addr 172.21.15.46:0/1537755160 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:41.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.908+0000 7fa6f0e0d700 1 -- 172.21.15.46:0/1537755160 >> v1:172.21.15.106:6789/0 conn(0x7fa6ec157ae0 legacy=0x7fa6ec06b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:41.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.908+0000 7fa6f0e0d700 1 -- 172.21.15.46:0/1537755160 >> v1:172.21.15.46:6789/0 conn(0x7fa6ec0b0260 legacy=0x7fa6ec0c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:41.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.908+0000 7fa6eb7fe700 1 -- 172.21.15.46:0/1537755160 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460698668 0 0) 0x7fa6ec06eab0 con 0x7fa6ec0b0260 2024-01-27T01:08:41.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.908+0000 7fa6f0e0d700 1 -- 172.21.15.46:0/1537755160 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ec0af9f0 con 0x7fa6ec157710 2024-01-27T01:08:41.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.909+0000 7fa6f0e0d700 1 --2- 172.21.15.46:0/1537755160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec157710 0x7fa6ec0aed90 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0009860 tx=0x7fa6e00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db640a259c2369a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:41.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.909+0000 7fa6eb7fe700 1 -- 172.21.15.46:0/1537755160 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0011630 con 0x7fa6ec157710 2024-01-27T01:08:41.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.909+0000 7fa6eb7fe700 1 -- 172.21.15.46:0/1537755160 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e0011790 con 0x7fa6ec157710 2024-01-27T01:08:41.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.909+0000 7fa6eb7fe700 1 -- 172.21.15.46:0/1537755160 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0011b50 con 0x7fa6ec157710 2024-01-27T01:08:41.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.909+0000 7fa6f3071700 1 -- 172.21.15.46:0/1537755160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec157710 msgr2=0x7fa6ec0aed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:41.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.909+0000 7fa6f3071700 1 --2- 172.21.15.46:0/1537755160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec157710 0x7fa6ec0aed90 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0009860 tx=0x7fa6e00091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:41.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.910+0000 7fa6f3071700 1 -- 172.21.15.46:0/1537755160 shutdown_connections 2024-01-27T01:08:41.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.910+0000 7fa6f3071700 1 --2- 172.21.15.46:0/1537755160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec157710 0x7fa6ec0aed90 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:41.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.910+0000 7fa6f3071700 1 -- 172.21.15.46:0/1537755160 >> 172.21.15.46:0/1537755160 conn(0x7fa6ec068ac0 msgr2=0x7fa6ec068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:41.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.910+0000 7fa6f3071700 1 -- 172.21.15.46:0/1537755160 shutdown_connections 2024-01-27T01:08:41.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.910+0000 7fa6f3071700 1 -- 172.21.15.46:0/1537755160 wait complete. 2024-01-27T01:08:41.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.911+0000 7fa6f3071700 1 Processor -- start 2024-01-27T01:08:41.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.911+0000 7fa6f3071700 1 -- start start 2024-01-27T01:08:41.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.912+0000 7fa6f3071700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6ec0b0260 0x7fa6ec0c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:41.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.912+0000 7fa6f3071700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6ec157710 0x7fa6ec0c0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:41.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.912+0000 7fa6f3071700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec157ae0 0x7fa6ec14ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:41.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.912+0000 7fa6f3071700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec0c5210 con 0x7fa6ec157ae0 2024-01-27T01:08:41.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.912+0000 7fa6f3071700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec0c1cb0 con 0x7fa6ec157710 2024-01-27T01:08:41.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.912+0000 7fa6f3071700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec0c1450 con 0x7fa6ec0b0260 2024-01-27T01:08:41.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.912+0000 7fa6f160e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec157ae0 0x7fa6ec14ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:41.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.912+0000 7fa6f160e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec157ae0 0x7fa6ec14ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48160/0 (socket says 172.21.15.46:48160) 2024-01-27T01:08:41.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.912+0000 7fa6f160e700 1 -- 172.21.15.46:0/4050393986 learned_addr learned my addr 172.21.15.46:0/4050393986 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:41.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.912+0000 7fa6f0e0d700 1 --2- 172.21.15.46:0/4050393986 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6ec0b0260 0x7fa6ec0c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:41.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.912+0000 7fa6ebfff700 1 --2- 172.21.15.46:0/4050393986 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6ec157710 0x7fa6ec0c0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:41.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.912+0000 7fa6f160e700 1 -- 172.21.15.46:0/4050393986 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6ec0b0260 msgr2=0x7fa6ec0c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:41.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.912+0000 7fa6f160e700 1 --2- 172.21.15.46:0/4050393986 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6ec0b0260 0x7fa6ec0c6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:41.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.912+0000 7fa6f160e700 1 -- 172.21.15.46:0/4050393986 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6ec157710 msgr2=0x7fa6ec0c0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:41.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.912+0000 7fa6f160e700 1 --2- 172.21.15.46:0/4050393986 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6ec157710 0x7fa6ec0c0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:41.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.912+0000 7fa6f160e700 1 -- 172.21.15.46:0/4050393986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e0009420 con 0x7fa6ec157ae0 2024-01-27T01:08:41.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.913+0000 7fa6f160e700 1 --2- 172.21.15.46:0/4050393986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec157ae0 0x7fa6ec14ed10 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7fa6e400cd70 tx=0x7fa6e4001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:41.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.913+0000 7fa6e8ff9700 1 -- 172.21.15.46:0/4050393986 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e400ce70 con 0x7fa6ec157ae0 2024-01-27T01:08:41.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.913+0000 7fa6e8ff9700 1 -- 172.21.15.46:0/4050393986 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e400aac0 con 0x7fa6ec157ae0 2024-01-27T01:08:41.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.913+0000 7fa6f3071700 1 -- 172.21.15.46:0/4050393986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ec0c16f0 con 0x7fa6ec157ae0 2024-01-27T01:08:41.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.913+0000 7fa6e8ff9700 1 -- 172.21.15.46:0/4050393986 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e400aea0 con 0x7fa6ec157ae0 2024-01-27T01:08:41.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.913+0000 7fa6f3071700 1 -- 172.21.15.46:0/4050393986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ec0c4ea0 con 0x7fa6ec157ae0 2024-01-27T01:08:41.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.914+0000 7fa6f3071700 1 -- 172.21.15.46:0/4050393986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ec0c53c0 con 0x7fa6ec157ae0 2024-01-27T01:08:41.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.915+0000 7fa6e8ff9700 1 -- 172.21.15.46:0/4050393986 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7fa6e4007300 con 0x7fa6ec157ae0 2024-01-27T01:08:41.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.916+0000 7fa6e8ff9700 1 --2- 172.21.15.46:0/4050393986 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa6d00670e0 0x7fa6d0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:41.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.916+0000 7fa6f0e0d700 1 -- 172.21.15.46:0/4050393986 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa6d00670e0 msgr2=0x7fa6d0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:41.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.916+0000 7fa6e8ff9700 1 -- 172.21.15.46:0/4050393986 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6990+0+0 (secure 0 0 0) 0x7fa6e4085ac0 con 0x7fa6ec157ae0 2024-01-27T01:08:41.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.916+0000 7fa6f0e0d700 1 --2- 172.21.15.46:0/4050393986 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa6d00670e0 0x7fa6d0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:41.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:41.920+0000 7fa6e8ff9700 1 -- 172.21.15.46:0/4050393986 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa6e4052610 con 0x7fa6ec157ae0 2024-01-27T01:08:41.935 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:41.935+0000 7f66656d31c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T01:08:41.948 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:41.949+0000 7f4965bd41c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:08:42.012 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:42.014+0000 7f4965bd41c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:08:42.018 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:42.018+0000 7f66656d31c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T01:08:42.093 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:42.093+0000 7f66656d31c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T01:08:42.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.117+0000 7fa6f0e0d700 1 -- 172.21.15.46:0/4050393986 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa6d00670e0 msgr2=0x7fa6d0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:42.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.117+0000 7fa6f0e0d700 1 --2- 172.21.15.46:0/4050393986 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa6d00670e0 0x7fa6d0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:08:42.168 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:42.169+0000 7f4965bd41c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:08:42.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.174+0000 7fa6f3071700 1 -- 172.21.15.46:0/4050393986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6ec0b3270 con 0x7fa6ec157ae0 2024-01-27T01:08:42.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.175+0000 7fa6e8ff9700 1 -- 172.21.15.46:0/4050393986 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+4616 (secure 0 0 0) 0x7fa6e4055c30 con 0x7fa6ec157ae0 2024-01-27T01:08:42.175 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:42.175 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":149,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3826743114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3826743114},{"type":"v1","addr":"172.21.15.46:6835","nonce":3826743114}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":144},{"gid":11947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2514069949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2514069949},{"type":"v1","addr":"172.21.15.106:6833","nonce":2514069949}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":144},{"gid":11950,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/251184858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":251184858},{"type":"v1","addr":"172.21.15.106:6835","nonce":251184858}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:08:20.703665+0000","modified":"2024-01-27T01:08:24.272153+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12348},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12348":{"gid":12348,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6833/3798781145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3798781145},{"type":"v1","addr":"172.21.15.46:6833","nonce":3798781145}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2024-01-27T01:08:42.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.179+0000 7fa6f3071700 1 -- 172.21.15.46:0/4050393986 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa6d00670e0 msgr2=0x7fa6d0069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:42.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.179+0000 7fa6f3071700 1 --2- 172.21.15.46:0/4050393986 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa6d00670e0 0x7fa6d0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.179+0000 7fa6f3071700 1 -- 172.21.15.46:0/4050393986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec157ae0 msgr2=0x7fa6ec14ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:42.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.179+0000 7fa6f3071700 1 --2- 172.21.15.46:0/4050393986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec157ae0 0x7fa6ec14ed10 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7fa6e400cd70 tx=0x7fa6e4001aa0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.179+0000 7fa6f3071700 1 -- 172.21.15.46:0/4050393986 shutdown_connections 2024-01-27T01:08:42.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.179+0000 7fa6f3071700 1 --2- 172.21.15.46:0/4050393986 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa6d00670e0 0x7fa6d0069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.179+0000 7fa6f3071700 1 --2- 172.21.15.46:0/4050393986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec157ae0 0x7fa6ec14ed10 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.179+0000 7fa6f3071700 1 --2- 172.21.15.46:0/4050393986 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6ec0b0260 0x7fa6ec0c6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.179+0000 7fa6f3071700 1 --2- 172.21.15.46:0/4050393986 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6ec157710 0x7fa6ec0c0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.179+0000 7fa6f3071700 1 -- 172.21.15.46:0/4050393986 >> 172.21.15.46:0/4050393986 conn(0x7fa6ec068ac0 msgr2=0x7fa6ec14fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:42.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.179+0000 7fa6f3071700 1 -- 172.21.15.46:0/4050393986 shutdown_connections 2024-01-27T01:08:42.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.179+0000 7fa6f3071700 1 -- 172.21.15.46:0/4050393986 wait complete. 2024-01-27T01:08:42.181 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 149 2024-01-27T01:08:42.199 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:08:42.285 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:42.285+0000 7f66656d31c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:08:42.307 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:42.307+0000 7f4965bd41c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:08:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.336+0000 7f573a5ca700 1 Processor -- start 2024-01-27T01:08:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.336+0000 7f573a5ca700 1 -- start start 2024-01-27T01:08:42.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.337+0000 7f573a5ca700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5734159770 0x7f57340cb100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:42.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.337+0000 7f573a5ca700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5734159b40 0x7f5734159f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:42.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.337+0000 7f573a5ca700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57340ce380 con 0x7f573415a4d0 2024-01-27T01:08:42.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.337+0000 7f573a5ca700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573415a8f0 con 0x7f5734159770 2024-01-27T01:08:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.337+0000 7f573a5ca700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57340cbaa0 con 0x7f5734159b40 2024-01-27T01:08:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.337+0000 7f5733fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5734159770 0x7f57340cb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.338+0000 7f5733fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5734159770 0x7f57340cb100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48162/0 (socket says 172.21.15.46:48162) 2024-01-27T01:08:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.338+0000 7f5733fff700 1 -- 172.21.15.46:0/4004991632 learned_addr learned my addr 172.21.15.46:0/4004991632 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.338+0000 7f57337fe700 1 --2- 172.21.15.46:0/4004991632 >> v2:172.21.15.106:3300/0 conn(0x7f5734159b40 0x7f5734159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.338+0000 7f5733fff700 1 -- 172.21.15.46:0/4004991632 >> v2:172.21.15.106:3300/0 conn(0x7f5734159b40 msgr2=0x7f5734159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.338+0000 7f5733fff700 1 --2- 172.21.15.46:0/4004991632 >> v2:172.21.15.106:3300/0 conn(0x7f5734159b40 0x7f5734159f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.338+0000 7f5733fff700 1 -- 172.21.15.46:0/4004991632 >> v1:172.21.15.106:6789/0 conn(0x7f573415a4d0 legacy=0x7f57340ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.338+0000 7f5733fff700 1 -- 172.21.15.46:0/4004991632 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57340cbd80 con 0x7f5734159770 2024-01-27T01:08:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.338+0000 7f5733fff700 1 --2- 172.21.15.46:0/4004991632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5734159770 0x7f57340cb100 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f57340651e0 tx=0x7f5724009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b26f399e7ebeeac server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.338+0000 7f5732ffd700 1 -- 172.21.15.46:0/4004991632 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5724009e70 con 0x7f5734159770 2024-01-27T01:08:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.338+0000 7f5732ffd700 1 -- 172.21.15.46:0/4004991632 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572400b860 con 0x7f5734159770 2024-01-27T01:08:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.338+0000 7f5732ffd700 1 -- 172.21.15.46:0/4004991632 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572400bc00 con 0x7f5734159770 2024-01-27T01:08:42.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.339+0000 7f573a5ca700 1 -- 172.21.15.46:0/4004991632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5734159770 msgr2=0x7f57340cb100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:42.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.339+0000 7f573a5ca700 1 --2- 172.21.15.46:0/4004991632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5734159770 0x7f57340cb100 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f57340651e0 tx=0x7f5724009250 comp rx=0 tx=0).stop 2024-01-27T01:08:42.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.339+0000 7f573a5ca700 1 -- 172.21.15.46:0/4004991632 shutdown_connections 2024-01-27T01:08:42.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.339+0000 7f573a5ca700 1 --2- 172.21.15.46:0/4004991632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5734159770 0x7f57340cb100 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.339+0000 7f573a5ca700 1 --2- 172.21.15.46:0/4004991632 >> v2:172.21.15.106:3300/0 conn(0x7f5734159b40 0x7f5734159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.339+0000 7f573a5ca700 1 -- 172.21.15.46:0/4004991632 >> 172.21.15.46:0/4004991632 conn(0x7f5734099a40 msgr2=0x7f573406f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:42.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.339+0000 7f573a5ca700 1 -- 172.21.15.46:0/4004991632 shutdown_connections 2024-01-27T01:08:42.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.339+0000 7f573a5ca700 1 -- 172.21.15.46:0/4004991632 wait complete. 2024-01-27T01:08:42.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.341+0000 7f573a5ca700 1 Processor -- start 2024-01-27T01:08:42.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.341+0000 7f573a5ca700 1 -- start start 2024-01-27T01:08:42.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.342+0000 7f573a5ca700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5734159770 0x7f5734153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:42.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.342+0000 7f573a5ca700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5734159b40 0x7f57340c8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:42.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.342+0000 7f573a5ca700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573415a4d0 0x7f57340ce810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:42.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.342+0000 7f573a5ca700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57340ca1d0 con 0x7f573415a4d0 2024-01-27T01:08:42.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.342+0000 7f573a5ca700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5734153fe0 con 0x7f5734159770 2024-01-27T01:08:42.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.342+0000 7f573a5ca700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57340c95c0 con 0x7f5734159b40 2024-01-27T01:08:42.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.342+0000 7f5738b67700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573415a4d0 0x7f57340ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:42.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.342+0000 7f5733fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5734159770 0x7f5734153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:42.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.342+0000 7f5738b67700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573415a4d0 0x7f57340ce810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48170/0 (socket says 172.21.15.46:48170) 2024-01-27T01:08:42.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.342+0000 7f5738b67700 1 -- 172.21.15.46:0/1954477283 learned_addr learned my addr 172.21.15.46:0/1954477283 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:42.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.342+0000 7f57337fe700 1 --2- 172.21.15.46:0/1954477283 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5734159b40 0x7f57340c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:42.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.343+0000 7f5738b67700 1 -- 172.21.15.46:0/1954477283 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5734159b40 msgr2=0x7f57340c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:42.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.343+0000 7f5738b67700 1 --2- 172.21.15.46:0/1954477283 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5734159b40 0x7f57340c8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.343+0000 7f5738b67700 1 -- 172.21.15.46:0/1954477283 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5734159770 msgr2=0x7f5734153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:42.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.343+0000 7f5738b67700 1 --2- 172.21.15.46:0/1954477283 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5734159770 0x7f5734153aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.343+0000 7f5738b67700 1 -- 172.21.15.46:0/1954477283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57240094f0 con 0x7f573415a4d0 2024-01-27T01:08:42.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.343+0000 7f5733fff700 1 --2- 172.21.15.46:0/1954477283 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5734159770 0x7f5734153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:42.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.343+0000 7f5738b67700 1 --2- 172.21.15.46:0/1954477283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573415a4d0 0x7f57340ce810 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f572c00c8e0 tx=0x7f572c001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.343+0000 7f571bfff700 1 -- 172.21.15.46:0/1954477283 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c012630 con 0x7f573415a4d0 2024-01-27T01:08:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.343+0000 7f571bfff700 1 -- 172.21.15.46:0/1954477283 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572c012790 con 0x7f573415a4d0 2024-01-27T01:08:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.343+0000 7f573a5ca700 1 -- 172.21.15.46:0/1954477283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57340c98a0 con 0x7f573415a4d0 2024-01-27T01:08:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.343+0000 7f571bfff700 1 -- 172.21.15.46:0/1954477283 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c012b50 con 0x7f573415a4d0 2024-01-27T01:08:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.343+0000 7f573a5ca700 1 -- 172.21.15.46:0/1954477283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57340c9e60 con 0x7f573415a4d0 2024-01-27T01:08:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.344+0000 7f573a5ca700 1 -- 172.21.15.46:0/1954477283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573404c710 con 0x7f573415a4d0 2024-01-27T01:08:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.345+0000 7f571bfff700 1 -- 172.21.15.46:0/1954477283 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f572c01a3a0 con 0x7f573415a4d0 2024-01-27T01:08:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.345+0000 7f571bfff700 1 --2- 172.21.15.46:0/1954477283 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f57200670e0 0x7f5720069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.345+0000 7f5733fff700 1 -- 172.21.15.46:0/1954477283 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f57200670e0 msgr2=0x7f5720069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.345+0000 7f571bfff700 1 -- 172.21.15.46:0/1954477283 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6990+0+0 (secure 0 0 0) 0x7f572c084ef0 con 0x7f573415a4d0 2024-01-27T01:08:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.345+0000 7f5733fff700 1 --2- 172.21.15.46:0/1954477283 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f57200670e0 0x7f5720069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:42.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.350+0000 7f571bfff700 1 -- 172.21.15.46:0/1954477283 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f572c055470 con 0x7f573415a4d0 2024-01-27T01:08:42.374 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:42.374+0000 7f4965bd41c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:08:42.446 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:42.446+0000 7f66656d31c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T01:08:42.495 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:42.495+0000 7f4965bd41c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:08:42.523 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:42.523+0000 7f66656d31c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T01:08:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.546+0000 7f5733fff700 1 -- 172.21.15.46:0/1954477283 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f57200670e0 msgr2=0x7f5720069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.546+0000 7f5733fff700 1 --2- 172.21.15.46:0/1954477283 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f57200670e0 0x7f5720069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:08:42.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.561+0000 7f573a5ca700 1 -- 172.21.15.46:0/1954477283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57340cc060 con 0x7f573415a4d0 2024-01-27T01:08:42.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.562+0000 7f571bfff700 1 -- 172.21.15.46:0/1954477283 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v265) v1 ==== 74+0+17013 (secure 0 0 0) 0x7f572c055000 con 0x7f573415a4d0 2024-01-27T01:08:42.563 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:42.563 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":265,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:08:35.385018+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:08:19.198347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-01-27T01:08:19.674810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"263","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:6837/62434":"2024-01-28T01:08:35.384966+0000","172.21.15.106:6836/62434":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3497341404":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3894462583":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/1144438896":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3166472261":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/1405196355":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/1752514247":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/2487246395":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3477459747":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3540935782":"2024-01-27T02:08:26.076460+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:08:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.565+0000 7f573a5ca700 1 -- 172.21.15.46:0/1954477283 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f57200670e0 msgr2=0x7f5720069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.565+0000 7f573a5ca700 1 --2- 172.21.15.46:0/1954477283 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f57200670e0 0x7f5720069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.565+0000 7f573a5ca700 1 -- 172.21.15.46:0/1954477283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573415a4d0 msgr2=0x7f57340ce810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.565+0000 7f573a5ca700 1 --2- 172.21.15.46:0/1954477283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573415a4d0 0x7f57340ce810 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f572c00c8e0 tx=0x7f572c001710 comp rx=0 tx=0).stop 2024-01-27T01:08:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.565+0000 7f573a5ca700 1 -- 172.21.15.46:0/1954477283 shutdown_connections 2024-01-27T01:08:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.565+0000 7f573a5ca700 1 --2- 172.21.15.46:0/1954477283 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f57200670e0 0x7f5720069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.565+0000 7f573a5ca700 1 --2- 172.21.15.46:0/1954477283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f573415a4d0 0x7f57340ce810 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.565+0000 7f573a5ca700 1 --2- 172.21.15.46:0/1954477283 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5734159770 0x7f5734153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.565+0000 7f573a5ca700 1 --2- 172.21.15.46:0/1954477283 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5734159b40 0x7f57340c8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.565+0000 7f573a5ca700 1 -- 172.21.15.46:0/1954477283 >> 172.21.15.46:0/1954477283 conn(0x7f5734099a40 msgr2=0x7f57340cade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.565+0000 7f573a5ca700 1 -- 172.21.15.46:0/1954477283 shutdown_connections 2024-01-27T01:08:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.565+0000 7f573a5ca700 1 -- 172.21.15.46:0/1954477283 wait complete. 2024-01-27T01:08:42.567 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:42.567+0000 7f4965bd41c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:08:42.580 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-26T23:29:38.279409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 35, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-27T01:08:19.198347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '262', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 36, 'pool_name': 'cephfs_data', 'create_time': '2024-01-27T01:08:19.674810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '263', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-27T01:08:42.581 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-27T01:08:42.670 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:42.670+0000 7f66656d31c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T01:08:42.707 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:42.708+0000 7f4965bd41c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:08:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.717+0000 7f23d7482700 1 Processor -- start 2024-01-27T01:08:42.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.718+0000 7f23d7482700 1 -- start start 2024-01-27T01:08:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.719+0000 7f23d7482700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f23d00cb7b0 0x7f23d00ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.719+0000 7f23d7482700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f23d00cdf20 0x7f23d00c29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.719+0000 7f23d7482700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d00c2270 con 0x7f23d0156980 2024-01-27T01:08:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.719+0000 7f23d7482700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d00c2550 con 0x7f23d00cdf20 2024-01-27T01:08:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.719+0000 7f23d7482700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d00ac9c0 con 0x7f23d00cb7b0 2024-01-27T01:08:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.719+0000 7f23d5a1f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f23d00cdf20 0x7f23d00c29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.719+0000 7f23d521e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f23d0156980 0x7f23d00cb0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59308/0 (socket says 172.21.15.46:59308) 2024-01-27T01:08:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.719+0000 7f23d521e700 1 -- 172.21.15.46:0/30283078 learned_addr learned my addr 172.21.15.46:0/30283078 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.719+0000 7f23d4a1d700 1 --2- 172.21.15.46:0/30283078 >> v2:172.21.15.106:3300/0 conn(0x7f23d00cb7b0 0x7f23d00ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.719+0000 7f23d5a1f700 1 -- 172.21.15.46:0/30283078 >> v2:172.21.15.106:3300/0 conn(0x7f23d00cb7b0 msgr2=0x7f23d00ac0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.719+0000 7f23d5a1f700 1 --2- 172.21.15.46:0/30283078 >> v2:172.21.15.106:3300/0 conn(0x7f23d00cb7b0 0x7f23d00ac0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.719+0000 7f23d5a1f700 1 -- 172.21.15.46:0/30283078 >> v1:172.21.15.46:6789/0 conn(0x7f23d0156980 legacy=0x7f23d00cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.719+0000 7f23d5a1f700 1 -- 172.21.15.46:0/30283078 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d00cbc80 con 0x7f23d00cdf20 2024-01-27T01:08:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.719+0000 7f23c7fff700 1 -- 172.21.15.46:0/30283078 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26393319 0 0) 0x7f23d00c2270 con 0x7f23d0156980 2024-01-27T01:08:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.720+0000 7f23d5a1f700 1 --2- 172.21.15.46:0/30283078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23d00cdf20 0x7f23d00c29e0 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f23d006fa70 tx=0x7f23c8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=790776434dae7e67 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.720+0000 7f23c7fff700 1 -- 172.21.15.46:0/30283078 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c800e070 con 0x7f23d00cdf20 2024-01-27T01:08:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.720+0000 7f23c7fff700 1 -- 172.21.15.46:0/30283078 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c800bb60 con 0x7f23d00cdf20 2024-01-27T01:08:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.720+0000 7f23c7fff700 1 -- 172.21.15.46:0/30283078 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c8012430 con 0x7f23d00cdf20 2024-01-27T01:08:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.720+0000 7f23d7482700 1 -- 172.21.15.46:0/30283078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23d00cdf20 msgr2=0x7f23d00c29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.720+0000 7f23d7482700 1 --2- 172.21.15.46:0/30283078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23d00cdf20 0x7f23d00c29e0 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f23d006fa70 tx=0x7f23c8009780 comp rx=0 tx=0).stop 2024-01-27T01:08:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.720+0000 7f23d7482700 1 -- 172.21.15.46:0/30283078 shutdown_connections 2024-01-27T01:08:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.720+0000 7f23d7482700 1 --2- 172.21.15.46:0/30283078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23d00cdf20 0x7f23d00c29e0 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.720+0000 7f23d7482700 1 --2- 172.21.15.46:0/30283078 >> v2:172.21.15.106:3300/0 conn(0x7f23d00cb7b0 0x7f23d00ac0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.720+0000 7f23d7482700 1 -- 172.21.15.46:0/30283078 >> 172.21.15.46:0/30283078 conn(0x7f23d00692a0 msgr2=0x7f23d00cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.721+0000 7f23d7482700 1 -- 172.21.15.46:0/30283078 shutdown_connections 2024-01-27T01:08:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.721+0000 7f23d7482700 1 -- 172.21.15.46:0/30283078 wait complete. 2024-01-27T01:08:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.722+0000 7f23d7482700 1 Processor -- start 2024-01-27T01:08:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.722+0000 7f23d7482700 1 -- start start 2024-01-27T01:08:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.723+0000 7f23d7482700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23d00cb7b0 0x7f23d01531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.723+0000 7f23d7482700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f23d00cdf20 0x7f23d00acb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.723+0000 7f23d7482700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f23d0156980 0x7f23d00ad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.723+0000 7f23d7482700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d00add80 con 0x7f23d00cb7b0 2024-01-27T01:08:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.723+0000 7f23d7482700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d00aebe0 con 0x7f23d0156980 2024-01-27T01:08:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.723+0000 7f23d7482700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d00b2e30 con 0x7f23d00cdf20 2024-01-27T01:08:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.723+0000 7f23d521e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23d00cb7b0 0x7f23d01531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.723+0000 7f23d521e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23d00cb7b0 0x7f23d01531c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48184/0 (socket says 172.21.15.46:48184) 2024-01-27T01:08:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.723+0000 7f23d521e700 1 -- 172.21.15.46:0/758478568 learned_addr learned my addr 172.21.15.46:0/758478568 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.723+0000 7f23d5a1f700 1 --2- 172.21.15.46:0/758478568 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f23d0156980 0x7f23d00ad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.723+0000 7f23d4a1d700 1 --2- 172.21.15.46:0/758478568 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f23d00cdf20 0x7f23d00acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.723+0000 7f23d521e700 1 -- 172.21.15.46:0/758478568 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f23d00cdf20 msgr2=0x7f23d00acb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.723+0000 7f23d521e700 1 --2- 172.21.15.46:0/758478568 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f23d00cdf20 0x7f23d00acb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.723+0000 7f23d521e700 1 -- 172.21.15.46:0/758478568 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f23d0156980 msgr2=0x7f23d00ad0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.723+0000 7f23d521e700 1 --2- 172.21.15.46:0/758478568 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f23d0156980 0x7f23d00ad0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.723+0000 7f23d521e700 1 -- 172.21.15.46:0/758478568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c8009a20 con 0x7f23d00cb7b0 2024-01-27T01:08:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.723+0000 7f23d521e700 1 --2- 172.21.15.46:0/758478568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23d00cb7b0 0x7f23d01531c0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f23cc00c5b0 tx=0x7f23cc001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.724+0000 7f23c57fa700 1 -- 172.21.15.46:0/758478568 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23cc005b30 con 0x7f23d00cb7b0 2024-01-27T01:08:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.724+0000 7f23c57fa700 1 -- 172.21.15.46:0/758478568 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23cc005c90 con 0x7f23d00cb7b0 2024-01-27T01:08:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.724+0000 7f23d7482700 1 -- 172.21.15.46:0/758478568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d00adfd0 con 0x7f23d00cb7b0 2024-01-27T01:08:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.724+0000 7f23c57fa700 1 -- 172.21.15.46:0/758478568 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23cc007300 con 0x7f23d00cb7b0 2024-01-27T01:08:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.724+0000 7f23d7482700 1 -- 172.21.15.46:0/758478568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d00ae590 con 0x7f23d00cb7b0 2024-01-27T01:08:42.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.725+0000 7f23d7482700 1 -- 172.21.15.46:0/758478568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d006ced0 con 0x7f23d00cb7b0 2024-01-27T01:08:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.726+0000 7f23c57fa700 1 -- 172.21.15.46:0/758478568 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f23cc01b3b0 con 0x7f23d00cb7b0 2024-01-27T01:08:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.726+0000 7f23c57fa700 1 --2- 172.21.15.46:0/758478568 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23b40670e0 0x7f23b4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.726+0000 7f23c57fa700 1 -- 172.21.15.46:0/758478568 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6990+0+0 (secure 0 0 0) 0x7f23cc084200 con 0x7f23d00cb7b0 2024-01-27T01:08:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.726+0000 7f23d4a1d700 1 -- 172.21.15.46:0/758478568 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23b40670e0 msgr2=0x7f23b4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.726+0000 7f23d4a1d700 1 --2- 172.21.15.46:0/758478568 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23b40670e0 0x7f23b4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:42.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.731+0000 7f23c57fa700 1 -- 172.21.15.46:0/758478568 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f23cc050cf0 con 0x7f23d00cb7b0 2024-01-27T01:08:42.760 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:42.760+0000 7f66656d31c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T01:08:42.835 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:42.836+0000 7f4965bd41c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:08:42.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.927+0000 7f23d4a1d700 1 -- 172.21.15.46:0/758478568 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23b40670e0 msgr2=0x7f23b4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:42.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.927+0000 7f23d4a1d700 1 --2- 172.21.15.46:0/758478568 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23b40670e0 0x7f23b4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:08:42.933 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:42.933+0000 7f66656d31c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T01:08:42.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.981+0000 7f23d7482700 1 -- 172.21.15.46:0/758478568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f23d00aef70 con 0x7f23d00cb7b0 2024-01-27T01:08:42.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.982+0000 7f23c57fa700 1 -- 172.21.15.46:0/758478568 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v265) v1 ==== 93+0+10 (secure 0 0 0) 0x7f23cc054310 con 0x7f23d00cb7b0 2024-01-27T01:08:42.982 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-01-27T01:08:42.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.984+0000 7f23d7482700 1 -- 172.21.15.46:0/758478568 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23b40670e0 msgr2=0x7f23b4069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:42.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.984+0000 7f23d7482700 1 --2- 172.21.15.46:0/758478568 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23b40670e0 0x7f23b4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.984+0000 7f23d7482700 1 -- 172.21.15.46:0/758478568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23d00cb7b0 msgr2=0x7f23d01531c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:42.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.984+0000 7f23d7482700 1 --2- 172.21.15.46:0/758478568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23d00cb7b0 0x7f23d01531c0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f23cc00c5b0 tx=0x7f23cc001070 comp rx=0 tx=0).stop 2024-01-27T01:08:42.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.984+0000 7f23d7482700 1 -- 172.21.15.46:0/758478568 shutdown_connections 2024-01-27T01:08:42.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.984+0000 7f23d7482700 1 --2- 172.21.15.46:0/758478568 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23b40670e0 0x7f23b4069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.984+0000 7f23d7482700 1 --2- 172.21.15.46:0/758478568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23d00cb7b0 0x7f23d01531c0 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.984+0000 7f23d7482700 1 --2- 172.21.15.46:0/758478568 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f23d0156980 0x7f23d00ad0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.984+0000 7f23d7482700 1 --2- 172.21.15.46:0/758478568 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f23d00cdf20 0x7f23d00acb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:42.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.984+0000 7f23d7482700 1 -- 172.21.15.46:0/758478568 >> 172.21.15.46:0/758478568 conn(0x7f23d00692a0 msgr2=0x7f23d006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:42.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.984+0000 7f23d7482700 1 -- 172.21.15.46:0/758478568 shutdown_connections 2024-01-27T01:08:42.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:42.984+0000 7f23d7482700 1 -- 172.21.15.46:0/758478568 wait complete. 2024-01-27T01:08:42.999 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-27T01:08:43.082 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:43.083+0000 7f66656d31c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T01:08:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.136+0000 7f238ad5a700 1 Processor -- start 2024-01-27T01:08:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.136+0000 7f238ad5a700 1 -- start start 2024-01-27T01:08:43.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.137+0000 7f238ad5a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f238415a290 0x7f23840c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.137+0000 7f238ad5a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f238406a630 0x7f238415a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.137+0000 7f238ad5a700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23840c8be0 con 0x7f238415aba0 2024-01-27T01:08:43.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.137+0000 7f238ad5a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f238406e200 con 0x7f238406a630 2024-01-27T01:08:43.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.137+0000 7f238ad5a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238406aaa0 con 0x7f238415a290 2024-01-27T01:08:43.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.137+0000 7f23892f7700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f238415aba0 0x7f238406d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59318/0 (socket says 172.21.15.46:59318) 2024-01-27T01:08:43.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.137+0000 7f23892f7700 1 -- 172.21.15.46:0/2858272687 learned_addr learned my addr 172.21.15.46:0/2858272687 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:43.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.137+0000 7f2388af6700 1 --2- 172.21.15.46:0/2858272687 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f238415a290 0x7f23840c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:43.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.137+0000 7f2383fff700 1 --2- 172.21.15.46:0/2858272687 >> v2:172.21.15.106:3300/0 conn(0x7f238406a630 0x7f238415a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:43.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.138+0000 7f23837fe700 1 -- 172.21.15.46:0/2858272687 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415838127 0 0) 0x7f23840c8be0 con 0x7f238415aba0 2024-01-27T01:08:43.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.138+0000 7f23837fe700 1 -- 172.21.15.46:0/2858272687 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f236c003570 con 0x7f238415aba0 2024-01-27T01:08:43.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.138+0000 7f2383fff700 1 -- 172.21.15.46:0/2858272687 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f238415a290 msgr2=0x7f23840c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:43.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.138+0000 7f2383fff700 1 --2- 172.21.15.46:0/2858272687 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f238415a290 0x7f23840c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.138+0000 7f2383fff700 1 -- 172.21.15.46:0/2858272687 >> v1:172.21.15.46:6789/0 conn(0x7f238415aba0 legacy=0x7f238406d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:43.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.138+0000 7f2383fff700 1 -- 172.21.15.46:0/2858272687 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23840c8e80 con 0x7f238406a630 2024-01-27T01:08:43.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.138+0000 7f2388af6700 1 --2- 172.21.15.46:0/2858272687 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f238415a290 0x7f23840c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:43.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.138+0000 7f2383fff700 1 --2- 172.21.15.46:0/2858272687 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f238406a630 0x7f238415a660 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f23780098e0 tx=0x7f23780091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de3e439c1a9c1532 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:43.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.138+0000 7f23837fe700 1 -- 172.21.15.46:0/2858272687 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2378011630 con 0x7f238406a630 2024-01-27T01:08:43.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.138+0000 7f23837fe700 1 -- 172.21.15.46:0/2858272687 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2378011790 con 0x7f238406a630 2024-01-27T01:08:43.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.138+0000 7f23837fe700 1 -- 172.21.15.46:0/2858272687 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2378011b10 con 0x7f238406a630 2024-01-27T01:08:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.139+0000 7f238ad5a700 1 -- 172.21.15.46:0/2858272687 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f238406a630 msgr2=0x7f238415a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.139+0000 7f238ad5a700 1 --2- 172.21.15.46:0/2858272687 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f238406a630 0x7f238415a660 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f23780098e0 tx=0x7f23780091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.139+0000 7f238ad5a700 1 -- 172.21.15.46:0/2858272687 shutdown_connections 2024-01-27T01:08:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.139+0000 7f238ad5a700 1 --2- 172.21.15.46:0/2858272687 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f238415a290 0x7f23840c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.139+0000 7f238ad5a700 1 --2- 172.21.15.46:0/2858272687 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f238406a630 0x7f238415a660 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.139+0000 7f238ad5a700 1 -- 172.21.15.46:0/2858272687 >> 172.21.15.46:0/2858272687 conn(0x7f238406bd70 msgr2=0x7f238406c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:43.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.139+0000 7f238ad5a700 1 -- 172.21.15.46:0/2858272687 shutdown_connections 2024-01-27T01:08:43.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.139+0000 7f238ad5a700 1 -- 172.21.15.46:0/2858272687 wait complete. 2024-01-27T01:08:43.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.140+0000 7f238ad5a700 1 Processor -- start 2024-01-27T01:08:43.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.140+0000 7f238ad5a700 1 -- start start 2024-01-27T01:08:43.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.141+0000 7f238ad5a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f238406a630 0x7f2384154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.141+0000 7f238ad5a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f238415a290 0x7f23840c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.141+0000 7f238ad5a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f238415aba0 0x7f23840c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.141+0000 7f238ad5a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23840b1340 con 0x7f238406a630 2024-01-27T01:08:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.141+0000 7f238ad5a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23840b1620 con 0x7f238415a290 2024-01-27T01:08:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.141+0000 7f238ad5a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23840b1900 con 0x7f238415aba0 2024-01-27T01:08:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.141+0000 7f2388af6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f238406a630 0x7f2384154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.141+0000 7f2388af6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f238406a630 0x7f2384154f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48192/0 (socket says 172.21.15.46:48192) 2024-01-27T01:08:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.141+0000 7f2388af6700 1 -- 172.21.15.46:0/532186249 learned_addr learned my addr 172.21.15.46:0/532186249 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.141+0000 7f2383fff700 1 --2- 172.21.15.46:0/532186249 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f238415a290 0x7f23840c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.141+0000 7f23892f7700 1 --2- 172.21.15.46:0/532186249 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f238415aba0 0x7f23840c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.141+0000 7f2388af6700 1 -- 172.21.15.46:0/532186249 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f238415aba0 msgr2=0x7f23840c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.141+0000 7f2388af6700 1 --2- 172.21.15.46:0/532186249 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f238415aba0 0x7f23840c7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.141+0000 7f2388af6700 1 -- 172.21.15.46:0/532186249 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f238415a290 msgr2=0x7f23840c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.141+0000 7f2388af6700 1 --2- 172.21.15.46:0/532186249 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f238415a290 0x7f23840c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.142+0000 7f2388af6700 1 -- 172.21.15.46:0/532186249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2374009420 con 0x7f238406a630 2024-01-27T01:08:43.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.142+0000 7f2388af6700 1 --2- 172.21.15.46:0/532186249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f238406a630 0x7f2384154f70 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f237400ec80 tx=0x7f23740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:43.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.142+0000 7f2380ff9700 1 -- 172.21.15.46:0/532186249 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237400c920 con 0x7f238406a630 2024-01-27T01:08:43.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.142+0000 7f2380ff9700 1 -- 172.21.15.46:0/532186249 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237400ca80 con 0x7f238406a630 2024-01-27T01:08:43.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.142+0000 7f238ad5a700 1 -- 172.21.15.46:0/532186249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2378009420 con 0x7f238406a630 2024-01-27T01:08:43.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.142+0000 7f2380ff9700 1 -- 172.21.15.46:0/532186249 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237400ce00 con 0x7f238406a630 2024-01-27T01:08:43.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.142+0000 7f238ad5a700 1 -- 172.21.15.46:0/532186249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23840b1ec0 con 0x7f238406a630 2024-01-27T01:08:43.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.143+0000 7f238ad5a700 1 -- 172.21.15.46:0/532186249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23840b2510 con 0x7f238406a630 2024-01-27T01:08:43.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.143+0000 7f2380ff9700 1 -- 172.21.15.46:0/532186249 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f2374013680 con 0x7f238406a630 2024-01-27T01:08:43.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.144+0000 7f2380ff9700 1 --2- 172.21.15.46:0/532186249 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23700670e0 0x7f2370069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.144+0000 7f2383fff700 1 -- 172.21.15.46:0/532186249 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23700670e0 msgr2=0x7f2370069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:43.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.144+0000 7f2380ff9700 1 -- 172.21.15.46:0/532186249 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6990+0+0 (secure 0 0 0) 0x7f2374085440 con 0x7f238406a630 2024-01-27T01:08:43.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.144+0000 7f2383fff700 1 --2- 172.21.15.46:0/532186249 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23700670e0 0x7f2370069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:43.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.148+0000 7f2380ff9700 1 -- 172.21.15.46:0/532186249 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2374051f30 con 0x7f238406a630 2024-01-27T01:08:43.149 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:43.149+0000 7f4965bd41c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:08:43.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.345+0000 7f2383fff700 1 -- 172.21.15.46:0/532186249 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23700670e0 msgr2=0x7f2370069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:43.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.345+0000 7f2383fff700 1 --2- 172.21.15.46:0/532186249 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23700670e0 0x7f2370069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:08:43.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.353+0000 7f238ad5a700 1 -- 172.21.15.46:0/532186249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f23840c9160 con 0x7f238406a630 2024-01-27T01:08:43.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.354+0000 7f2380ff9700 1 -- 172.21.15.46:0/532186249 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v265) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2374055550 con 0x7f238406a630 2024-01-27T01:08:43.354 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-01-27T01:08:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.356+0000 7f238ad5a700 1 -- 172.21.15.46:0/532186249 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23700670e0 msgr2=0x7f2370069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.356+0000 7f238ad5a700 1 --2- 172.21.15.46:0/532186249 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23700670e0 0x7f2370069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.356+0000 7f238ad5a700 1 -- 172.21.15.46:0/532186249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f238406a630 msgr2=0x7f2384154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.356+0000 7f238ad5a700 1 --2- 172.21.15.46:0/532186249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f238406a630 0x7f2384154f70 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f237400ec80 tx=0x7f23740091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.356+0000 7f238ad5a700 1 -- 172.21.15.46:0/532186249 shutdown_connections 2024-01-27T01:08:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.356+0000 7f238ad5a700 1 --2- 172.21.15.46:0/532186249 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f23700670e0 0x7f2370069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.356+0000 7f238ad5a700 1 --2- 172.21.15.46:0/532186249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f238406a630 0x7f2384154f70 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.356+0000 7f238ad5a700 1 --2- 172.21.15.46:0/532186249 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f238415aba0 0x7f23840c7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.356+0000 7f238ad5a700 1 --2- 172.21.15.46:0/532186249 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f238415a290 0x7f23840c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.356+0000 7f238ad5a700 1 -- 172.21.15.46:0/532186249 >> 172.21.15.46:0/532186249 conn(0x7f238406bd70 msgr2=0x7f2384155a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.356+0000 7f238ad5a700 1 -- 172.21.15.46:0/532186249 shutdown_connections 2024-01-27T01:08:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.356+0000 7f238ad5a700 1 -- 172.21.15.46:0/532186249 wait complete. 2024-01-27T01:08:43.361 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:43.361+0000 7f4965bd41c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:08:43.371 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-27T01:08:43.458 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:43.458+0000 7f66656d31c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T01:08:43.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.509+0000 7f795373d700 1 Processor -- start 2024-01-27T01:08:43.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.510+0000 7f795373d700 1 -- start start 2024-01-27T01:08:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.511+0000 7f795373d700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f794c0cb7b0 0x7f794c0ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.511+0000 7f795373d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f794c0cdf20 0x7f794c0c29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.511+0000 7f795373d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794c0c2270 con 0x7f794c156980 2024-01-27T01:08:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.511+0000 7f795373d700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f794c0c2550 con 0x7f794c0cb7b0 2024-01-27T01:08:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.511+0000 7f795373d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794c0ac9c0 con 0x7f794c0cdf20 2024-01-27T01:08:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.511+0000 7f79514d9700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f794c156980 0x7f794c0cb0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59322/0 (socket says 172.21.15.46:59322) 2024-01-27T01:08:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.511+0000 7f79514d9700 1 -- 172.21.15.46:0/981888199 learned_addr learned my addr 172.21.15.46:0/981888199 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.511+0000 7f7950cd8700 1 --2- 172.21.15.46:0/981888199 >> v2:172.21.15.106:3300/0 conn(0x7f794c0cb7b0 0x7f794c0ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.511+0000 7f7951cda700 1 --2- 172.21.15.46:0/981888199 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f794c0cdf20 0x7f794c0c29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.511+0000 7f7943fff700 1 -- 172.21.15.46:0/981888199 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194065314 0 0) 0x7f794c0c2270 con 0x7f794c156980 2024-01-27T01:08:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.511+0000 7f7943fff700 1 -- 172.21.15.46:0/981888199 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7938003570 con 0x7f794c156980 2024-01-27T01:08:43.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.511+0000 7f7950cd8700 1 -- 172.21.15.46:0/981888199 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f794c0cdf20 msgr2=0x7f794c0c29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:43.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.511+0000 7f7950cd8700 1 --2- 172.21.15.46:0/981888199 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f794c0cdf20 0x7f794c0c29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.511+0000 7f7950cd8700 1 -- 172.21.15.46:0/981888199 >> v1:172.21.15.46:6789/0 conn(0x7f794c156980 legacy=0x7f794c0cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:43.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.511+0000 7f7950cd8700 1 -- 172.21.15.46:0/981888199 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794c0cbc80 con 0x7f794c0cb7b0 2024-01-27T01:08:43.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.511+0000 7f7951cda700 1 --2- 172.21.15.46:0/981888199 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f794c0cdf20 0x7f794c0c29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:43.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.512+0000 7f7950cd8700 1 --2- 172.21.15.46:0/981888199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f794c0cb7b0 0x7f794c0ac0e0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f793c0098e0 tx=0x7f793c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17155bed1a3732d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:43.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.512+0000 7f7943fff700 1 -- 172.21.15.46:0/981888199 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c011630 con 0x7f794c0cb7b0 2024-01-27T01:08:43.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.512+0000 7f7943fff700 1 -- 172.21.15.46:0/981888199 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793c011790 con 0x7f794c0cb7b0 2024-01-27T01:08:43.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.512+0000 7f7943fff700 1 -- 172.21.15.46:0/981888199 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c011b30 con 0x7f794c0cb7b0 2024-01-27T01:08:43.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.512+0000 7f795373d700 1 -- 172.21.15.46:0/981888199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f794c0cb7b0 msgr2=0x7f794c0ac0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:43.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.512+0000 7f795373d700 1 --2- 172.21.15.46:0/981888199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f794c0cb7b0 0x7f794c0ac0e0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f793c0098e0 tx=0x7f793c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.512+0000 7f795373d700 1 -- 172.21.15.46:0/981888199 shutdown_connections 2024-01-27T01:08:43.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.512+0000 7f795373d700 1 --2- 172.21.15.46:0/981888199 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f794c0cdf20 0x7f794c0c29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.512+0000 7f795373d700 1 --2- 172.21.15.46:0/981888199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f794c0cb7b0 0x7f794c0ac0e0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.512+0000 7f795373d700 1 -- 172.21.15.46:0/981888199 >> 172.21.15.46:0/981888199 conn(0x7f794c0692a0 msgr2=0x7f794c0cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:43.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.513+0000 7f795373d700 1 -- 172.21.15.46:0/981888199 shutdown_connections 2024-01-27T01:08:43.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.513+0000 7f795373d700 1 -- 172.21.15.46:0/981888199 wait complete. 2024-01-27T01:08:43.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.514+0000 7f795373d700 1 Processor -- start 2024-01-27T01:08:43.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.514+0000 7f795373d700 1 -- start start 2024-01-27T01:08:43.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.515+0000 7f795373d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f794c0cb7b0 0x7f794c16b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.515+0000 7f795373d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f794c0cdf20 0x7f794c0b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.516+0000 7f795373d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f794c156980 0x7f794c149cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.516+0000 7f795373d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794c14a9f0 con 0x7f794c0cb7b0 2024-01-27T01:08:43.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.516+0000 7f795373d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794c16b9a0 con 0x7f794c0cdf20 2024-01-27T01:08:43.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.516+0000 7f795373d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794c16bc80 con 0x7f794c156980 2024-01-27T01:08:43.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.516+0000 7f79514d9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f794c0cb7b0 0x7f794c16b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:43.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.516+0000 7f79514d9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f794c0cb7b0 0x7f794c16b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48204/0 (socket says 172.21.15.46:48204) 2024-01-27T01:08:43.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.516+0000 7f7950cd8700 1 --2- 172.21.15.46:0/829833620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f794c0cdf20 0x7f794c0b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:43.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.516+0000 7f79514d9700 1 -- 172.21.15.46:0/829833620 learned_addr learned my addr 172.21.15.46:0/829833620 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:43.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.516+0000 7f7951cda700 1 --2- 172.21.15.46:0/829833620 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f794c156980 0x7f794c149cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:43.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.516+0000 7f7950cd8700 1 -- 172.21.15.46:0/829833620 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f794c156980 msgr2=0x7f794c149cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:43.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.516+0000 7f7950cd8700 1 --2- 172.21.15.46:0/829833620 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f794c156980 0x7f794c149cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.516+0000 7f7950cd8700 1 -- 172.21.15.46:0/829833620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f794c0cb7b0 msgr2=0x7f794c16b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:43.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.516+0000 7f7950cd8700 1 --2- 172.21.15.46:0/829833620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f794c0cb7b0 0x7f794c16b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.516+0000 7f7950cd8700 1 -- 172.21.15.46:0/829833620 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7944009a20 con 0x7f794c0cdf20 2024-01-27T01:08:43.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.516+0000 7f79514d9700 1 --2- 172.21.15.46:0/829833620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f794c0cb7b0 0x7f794c16b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:43.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.516+0000 7f7950cd8700 1 --2- 172.21.15.46:0/829833620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f794c0cdf20 0x7f794c0b26d0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f793c016000 tx=0x7f793c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:43.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.517+0000 7f79417fa700 1 -- 172.21.15.46:0/829833620 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c011b10 con 0x7f794c0cdf20 2024-01-27T01:08:43.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.517+0000 7f79417fa700 1 -- 172.21.15.46:0/829833620 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793c0253f0 con 0x7f794c0cdf20 2024-01-27T01:08:43.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.517+0000 7f795373d700 1 -- 172.21.15.46:0/829833620 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f793c009420 con 0x7f794c0cdf20 2024-01-27T01:08:43.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.517+0000 7f79417fa700 1 -- 172.21.15.46:0/829833620 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c0257d0 con 0x7f794c0cdf20 2024-01-27T01:08:43.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.517+0000 7f795373d700 1 -- 172.21.15.46:0/829833620 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794c0c5a40 con 0x7f794c0cdf20 2024-01-27T01:08:43.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.518+0000 7f795373d700 1 -- 172.21.15.46:0/829833620 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794c06ced0 con 0x7f794c0cdf20 2024-01-27T01:08:43.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.518+0000 7f79417fa700 1 -- 172.21.15.46:0/829833620 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f793c0193b0 con 0x7f794c0cdf20 2024-01-27T01:08:43.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.519+0000 7f79417fa700 1 --2- 172.21.15.46:0/829833620 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f79300670e0 0x7f7930069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.519+0000 7f79514d9700 1 -- 172.21.15.46:0/829833620 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f79300670e0 msgr2=0x7f7930069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:43.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.519+0000 7f79417fa700 1 -- 172.21.15.46:0/829833620 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6990+0+0 (secure 0 0 0) 0x7f793c086950 con 0x7f794c0cdf20 2024-01-27T01:08:43.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.519+0000 7f79514d9700 1 --2- 172.21.15.46:0/829833620 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f79300670e0 0x7f7930069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:43.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.523+0000 7f79417fa700 1 -- 172.21.15.46:0/829833620 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f793c053500 con 0x7f794c0cdf20 2024-01-27T01:08:43.588 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:43.589+0000 7f4965bd41c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:08:43.659 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:43.659+0000 7f4965bd41c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:08:43.705 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:43.705+0000 7f66656d31c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T01:08:43.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.720+0000 7f79514d9700 1 -- 172.21.15.46:0/829833620 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f79300670e0 msgr2=0x7f7930069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.720+0000 7f79514d9700 1 --2- 172.21.15.46:0/829833620 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f79300670e0 0x7f7930069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:08:43.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.726+0000 7f795373d700 1 -- 172.21.15.46:0/829833620 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f794c0cbf60 con 0x7f794c0cdf20 2024-01-27T01:08:43.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.727+0000 7f79417fa700 1 -- 172.21.15.46:0/829833620 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v265) v1 ==== 100+0+11 (secure 0 0 0) 0x7f793c056b20 con 0x7f794c0cdf20 2024-01-27T01:08:43.727 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-01-27T01:08:43.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.729+0000 7f795373d700 1 -- 172.21.15.46:0/829833620 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f79300670e0 msgr2=0x7f7930069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:43.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.729+0000 7f795373d700 1 --2- 172.21.15.46:0/829833620 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f79300670e0 0x7f7930069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.729+0000 7f795373d700 1 -- 172.21.15.46:0/829833620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f794c0cdf20 msgr2=0x7f794c0b26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:43.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.729+0000 7f795373d700 1 --2- 172.21.15.46:0/829833620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f794c0cdf20 0x7f794c0b26d0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f793c016000 tx=0x7f793c017000 comp rx=0 tx=0).stop 2024-01-27T01:08:43.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.729+0000 7f795373d700 1 -- 172.21.15.46:0/829833620 shutdown_connections 2024-01-27T01:08:43.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.729+0000 7f795373d700 1 --2- 172.21.15.46:0/829833620 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f79300670e0 0x7f7930069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.729+0000 7f795373d700 1 --2- 172.21.15.46:0/829833620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f794c0cb7b0 0x7f794c16b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.729+0000 7f795373d700 1 --2- 172.21.15.46:0/829833620 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f794c156980 0x7f794c149cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.729+0000 7f795373d700 1 --2- 172.21.15.46:0/829833620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f794c0cdf20 0x7f794c0b26d0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.729+0000 7f795373d700 1 -- 172.21.15.46:0/829833620 >> 172.21.15.46:0/829833620 conn(0x7f794c0692a0 msgr2=0x7f794c0ac030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:43.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.729+0000 7f795373d700 1 -- 172.21.15.46:0/829833620 shutdown_connections 2024-01-27T01:08:43.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.729+0000 7f795373d700 1 -- 172.21.15.46:0/829833620 wait complete. 2024-01-27T01:08:43.743 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:08:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.879+0000 7fdf1bf95700 1 Processor -- start 2024-01-27T01:08:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.879+0000 7fdf1bf95700 1 -- start start 2024-01-27T01:08:43.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.880+0000 7fdf1bf95700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf140b2c60 0x7fdf140b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.880+0000 7fdf1bf95700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdf140ce3a0 0x7fdf140ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.880+0000 7fdf1bf95700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf140d1490 con 0x7fdf140b3ad0 2024-01-27T01:08:43.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.880+0000 7fdf1bf95700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf140c6390 con 0x7fdf140ce3a0 2024-01-27T01:08:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.880+0000 7fdf1bf95700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf140ccd80 con 0x7fdf140b2c60 2024-01-27T01:08:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.881+0000 7fdf19d31700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf140b2c60 0x7fdf140b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.881+0000 7fdf1a532700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdf140ce3a0 0x7fdf140ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.881+0000 7fdf19d31700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf140b2c60 0x7fdf140b3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:34024/0 (socket says 172.21.15.46:34024) 2024-01-27T01:08:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.881+0000 7fdf1a532700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdf140ce3a0 0x7fdf140ce800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:57462/0 (socket says 172.21.15.46:57462) 2024-01-27T01:08:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.881+0000 7fdf19d31700 1 -- 172.21.15.46:0/3155687409 learned_addr learned my addr 172.21.15.46:0/3155687409 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.881+0000 7fdf19530700 1 --1- 172.21.15.46:0/3155687409 >> v1:172.21.15.106:6789/0 conn(0x7fdf140b3ad0 0x7fdf140c6820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:34562/0 (socket says 172.21.15.46:34562) 2024-01-27T01:08:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.881+0000 7fdf19d31700 1 -- 172.21.15.46:0/3155687409 >> v2:172.21.15.106:3300/0 conn(0x7fdf140ce3a0 msgr2=0x7fdf140ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.881+0000 7fdf19d31700 1 --2- 172.21.15.46:0/3155687409 >> v2:172.21.15.106:3300/0 conn(0x7fdf140ce3a0 0x7fdf140ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.881+0000 7fdf19d31700 1 -- 172.21.15.46:0/3155687409 >> v1:172.21.15.106:6789/0 conn(0x7fdf140b3ad0 legacy=0x7fdf140c6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.881+0000 7fdf19d31700 1 -- 172.21.15.46:0/3155687409 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf140cc7c0 con 0x7fdf140b2c60 2024-01-27T01:08:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.881+0000 7fdf18d2f700 1 -- 172.21.15.46:0/3155687409 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831950970 0 0) 0x7fdf140d1490 con 0x7fdf140b3ad0 2024-01-27T01:08:43.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.881+0000 7fdf1a532700 1 --2- 172.21.15.46:0/3155687409 >> v2:172.21.15.106:3300/0 conn(0x7fdf140ce3a0 0x7fdf140ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.881+0000 7fdf19d31700 1 --2- 172.21.15.46:0/3155687409 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf140b2c60 0x7fdf140b3590 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fdf100099b0 tx=0x7fdf10009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3903500b2d19aab server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.881+0000 7fdf18d2f700 1 -- 172.21.15.46:0/3155687409 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf10009ea0 con 0x7fdf140b2c60 2024-01-27T01:08:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.882+0000 7fdf18d2f700 1 -- 172.21.15.46:0/3155687409 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1000b860 con 0x7fdf140b2c60 2024-01-27T01:08:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.882+0000 7fdf18d2f700 1 -- 172.21.15.46:0/3155687409 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1000bc00 con 0x7fdf140b2c60 2024-01-27T01:08:43.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.882+0000 7fdf1bf95700 1 -- 172.21.15.46:0/3155687409 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf140b2c60 msgr2=0x7fdf140b3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.882+0000 7fdf1bf95700 1 --2- 172.21.15.46:0/3155687409 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf140b2c60 0x7fdf140b3590 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fdf100099b0 tx=0x7fdf10009250 comp rx=0 tx=0).stop 2024-01-27T01:08:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.882+0000 7fdf1bf95700 1 -- 172.21.15.46:0/3155687409 shutdown_connections 2024-01-27T01:08:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.882+0000 7fdf1bf95700 1 --2- 172.21.15.46:0/3155687409 >> v2:172.21.15.106:3300/0 conn(0x7fdf140ce3a0 0x7fdf140ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.882+0000 7fdf1bf95700 1 --2- 172.21.15.46:0/3155687409 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf140b2c60 0x7fdf140b3590 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.882+0000 7fdf1bf95700 1 -- 172.21.15.46:0/3155687409 >> 172.21.15.46:0/3155687409 conn(0x7fdf1406f8c0 msgr2=0x7fdf140d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.882+0000 7fdf1bf95700 1 -- 172.21.15.46:0/3155687409 shutdown_connections 2024-01-27T01:08:43.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.882+0000 7fdf1bf95700 1 -- 172.21.15.46:0/3155687409 wait complete. 2024-01-27T01:08:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.884+0000 7fdf1bf95700 1 Processor -- start 2024-01-27T01:08:43.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.884+0000 7fdf1bf95700 1 -- start start 2024-01-27T01:08:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.885+0000 7fdf1bf95700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf140b2c60 0x7fdf14152370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.885+0000 7fdf1bf95700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf140b3ad0 0x7fdf141581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.885+0000 7fdf1bf95700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf140ce3a0 0x7fdf140cd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.885+0000 7fdf1bf95700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf141528b0 con 0x7fdf140b2c60 2024-01-27T01:08:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.885+0000 7fdf1bf95700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf14158720 con 0x7fdf140b3ad0 2024-01-27T01:08:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.885+0000 7fdf1bf95700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf14157260 con 0x7fdf140ce3a0 2024-01-27T01:08:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.885+0000 7fdf19d31700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf140b2c60 0x7fdf14152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.885+0000 7fdf19d31700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf140b2c60 0x7fdf14152370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48214/0 (socket says 172.21.15.46:48214) 2024-01-27T01:08:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.885+0000 7fdf19d31700 1 -- 172.21.15.46:0/193648845 learned_addr learned my addr 172.21.15.46:0/193648845 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.885+0000 7fdf19530700 1 --2- 172.21.15.46:0/193648845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf140b3ad0 0x7fdf141581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.885+0000 7fdf1a532700 1 --2- 172.21.15.46:0/193648845 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf140ce3a0 0x7fdf140cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.886+0000 7fdf19d31700 1 -- 172.21.15.46:0/193648845 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf140ce3a0 msgr2=0x7fdf140cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.886+0000 7fdf19d31700 1 --2- 172.21.15.46:0/193648845 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf140ce3a0 0x7fdf140cd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.886+0000 7fdf19d31700 1 -- 172.21.15.46:0/193648845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf140b3ad0 msgr2=0x7fdf141581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.886+0000 7fdf19d31700 1 --2- 172.21.15.46:0/193648845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf140b3ad0 0x7fdf141581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.886+0000 7fdf19d31700 1 -- 172.21.15.46:0/193648845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c009a20 con 0x7fdf140b2c60 2024-01-27T01:08:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.886+0000 7fdf19530700 1 --2- 172.21.15.46:0/193648845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf140b3ad0 0x7fdf141581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.886+0000 7fdf19d31700 1 --2- 172.21.15.46:0/193648845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf140b2c60 0x7fdf14152370 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7fdf140ccec0 tx=0x7fdf10017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.886+0000 7fdf09ffb700 1 -- 172.21.15.46:0/193648845 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1000bbe0 con 0x7fdf140b2c60 2024-01-27T01:08:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.886+0000 7fdf09ffb700 1 -- 172.21.15.46:0/193648845 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf10011420 con 0x7fdf140b2c60 2024-01-27T01:08:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.886+0000 7fdf1bf95700 1 -- 172.21.15.46:0/193648845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf100094f0 con 0x7fdf140b2c60 2024-01-27T01:08:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.886+0000 7fdf09ffb700 1 -- 172.21.15.46:0/193648845 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf100117e0 con 0x7fdf140b2c60 2024-01-27T01:08:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.886+0000 7fdf1bf95700 1 -- 172.21.15.46:0/193648845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf14156930 con 0x7fdf140b2c60 2024-01-27T01:08:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.887+0000 7fdf1bf95700 1 -- 172.21.15.46:0/193648845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf1404c710 con 0x7fdf140b2c60 2024-01-27T01:08:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.888+0000 7fdf09ffb700 1 -- 172.21.15.46:0/193648845 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7fdf10011b90 con 0x7fdf140b2c60 2024-01-27T01:08:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.889+0000 7fdf09ffb700 1 --2- 172.21.15.46:0/193648845 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdef80670e0 0x7fdef8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.889+0000 7fdf19530700 1 -- 172.21.15.46:0/193648845 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdef80670e0 msgr2=0x7fdef8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.889+0000 7fdf09ffb700 1 -- 172.21.15.46:0/193648845 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6990+0+0 (secure 0 0 0) 0x7fdf100865a0 con 0x7fdf140b2c60 2024-01-27T01:08:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.889+0000 7fdf19530700 1 --2- 172.21.15.46:0/193648845 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdef80670e0 0x7fdef8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:43.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:43.898+0000 7fdf09ffb700 1 -- 172.21.15.46:0/193648845 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdf100530b0 con 0x7fdf140b2c60 2024-01-27T01:08:43.987 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:43.987+0000 7f66656d31c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T01:08:44.069 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:44.069+0000 7f66656d31c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T01:08:44.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.090+0000 7fdf19530700 1 -- 172.21.15.46:0/193648845 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdef80670e0 msgr2=0x7fdef8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:44.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.090+0000 7fdf19530700 1 --2- 172.21.15.46:0/193648845 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdef80670e0 0x7fdef8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:08:44.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.163+0000 7fdf1bf95700 1 -- 172.21.15.46:0/193648845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf140cf910 con 0x7fdf140b2c60 2024-01-27T01:08:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.164+0000 7fdf09ffb700 1 -- 172.21.15.46:0/193648845 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+4616 (secure 0 0 0) 0x7fdf100566d0 con 0x7fdf140b2c60 2024-01-27T01:08:44.164 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:44.164 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":149,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3826743114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3826743114},{"type":"v1","addr":"172.21.15.46:6835","nonce":3826743114}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":144},{"gid":11947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2514069949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2514069949},{"type":"v1","addr":"172.21.15.106:6833","nonce":2514069949}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":144},{"gid":11950,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/251184858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":251184858},{"type":"v1","addr":"172.21.15.106:6835","nonce":251184858}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:08:20.703665+0000","modified":"2024-01-27T01:08:24.272153+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12348},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12348":{"gid":12348,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6833/3798781145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3798781145},{"type":"v1","addr":"172.21.15.46:6833","nonce":3798781145}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2024-01-27T01:08:44.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.166+0000 7fdf1bf95700 1 -- 172.21.15.46:0/193648845 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdef80670e0 msgr2=0x7fdef8069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:44.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.166+0000 7fdf1bf95700 1 --2- 172.21.15.46:0/193648845 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdef80670e0 0x7fdef8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.166+0000 7fdf1bf95700 1 -- 172.21.15.46:0/193648845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf140b2c60 msgr2=0x7fdf14152370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:44.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.166+0000 7fdf1bf95700 1 --2- 172.21.15.46:0/193648845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf140b2c60 0x7fdf14152370 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7fdf140ccec0 tx=0x7fdf10017000 comp rx=0 tx=0).stop 2024-01-27T01:08:44.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.166+0000 7fdf1bf95700 1 -- 172.21.15.46:0/193648845 shutdown_connections 2024-01-27T01:08:44.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.166+0000 7fdf1bf95700 1 --2- 172.21.15.46:0/193648845 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdef80670e0 0x7fdef8069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.166+0000 7fdf1bf95700 1 --2- 172.21.15.46:0/193648845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf140b2c60 0x7fdf14152370 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.166+0000 7fdf1bf95700 1 --2- 172.21.15.46:0/193648845 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf140ce3a0 0x7fdf140cd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.166+0000 7fdf1bf95700 1 --2- 172.21.15.46:0/193648845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf140b3ad0 0x7fdf141581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.166+0000 7fdf1bf95700 1 -- 172.21.15.46:0/193648845 >> 172.21.15.46:0/193648845 conn(0x7fdf1406f8c0 msgr2=0x7fdf14068f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:44.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.167+0000 7fdf1bf95700 1 -- 172.21.15.46:0/193648845 shutdown_connections 2024-01-27T01:08:44.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.167+0000 7fdf1bf95700 1 -- 172.21.15.46:0/193648845 wait complete. 2024-01-27T01:08:44.168 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 149 2024-01-27T01:08:44.185 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:08:44.278 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:44.279+0000 7f4965bd41c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:08:44.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.319+0000 7fb73e293700 1 Processor -- start 2024-01-27T01:08:44.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.319+0000 7fb73e293700 1 -- start start 2024-01-27T01:08:44.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.320+0000 7fb73e293700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb7380c6820 0x7fb7380c6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:44.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.320+0000 7fb73e293700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7380ce3a0 0x7fb7380ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:44.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.320+0000 7fb73e293700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7380d1490 con 0x7fb7380b2c60 2024-01-27T01:08:44.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.320+0000 7fb73e293700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7380c71b0 con 0x7fb7380c6820 2024-01-27T01:08:44.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.320+0000 7fb73e293700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380c6390 con 0x7fb7380ce3a0 2024-01-27T01:08:44.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.320+0000 7fb7377fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb7380c6820 0x7fb7380c6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:44.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.320+0000 7fb73c830700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7380ce3a0 0x7fb7380ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.320+0000 7fb73c830700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7380ce3a0 0x7fb7380ce7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:34042/0 (socket says 172.21.15.46:34042) 2024-01-27T01:08:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.320+0000 7fb7377fe700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb7380c6820 0x7fb7380c6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:57476/0 (socket says 172.21.15.46:57476) 2024-01-27T01:08:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.320+0000 7fb73c830700 1 -- 172.21.15.46:0/122885181 learned_addr learned my addr 172.21.15.46:0/122885181 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.320+0000 7fb737fff700 1 --1- 172.21.15.46:0/122885181 >> v1:172.21.15.106:6789/0 conn(0x7fb7380b2c60 0x7fb7380b3590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:34578/0 (socket says 172.21.15.46:34578) 2024-01-27T01:08:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.320+0000 7fb736ffd700 1 -- 172.21.15.46:0/122885181 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284879785 0 0) 0x7fb7380d1490 con 0x7fb7380b2c60 2024-01-27T01:08:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.320+0000 7fb736ffd700 1 -- 172.21.15.46:0/122885181 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb720003570 con 0x7fb7380b2c60 2024-01-27T01:08:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.320+0000 7fb73c830700 1 -- 172.21.15.46:0/122885181 >> v2:172.21.15.106:3300/0 conn(0x7fb7380c6820 msgr2=0x7fb7380c6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.320+0000 7fb73c830700 1 --2- 172.21.15.46:0/122885181 >> v2:172.21.15.106:3300/0 conn(0x7fb7380c6820 0x7fb7380c6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.321+0000 7fb73c830700 1 -- 172.21.15.46:0/122885181 >> v1:172.21.15.106:6789/0 conn(0x7fb7380b2c60 legacy=0x7fb7380b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.321+0000 7fb73c830700 1 -- 172.21.15.46:0/122885181 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7380ccd80 con 0x7fb7380ce3a0 2024-01-27T01:08:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.321+0000 7fb7377fe700 1 --2- 172.21.15.46:0/122885181 >> v2:172.21.15.106:3300/0 conn(0x7fb7380c6820 0x7fb7380c6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.321+0000 7fb73c830700 1 --2- 172.21.15.46:0/122885181 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7380ce3a0 0x7fb7380ce7e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fb730009f20 tx=0x7fb730009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b91fb0c1ba9f2929 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.321+0000 7fb736ffd700 1 -- 172.21.15.46:0/122885181 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73000e070 con 0x7fb7380ce3a0 2024-01-27T01:08:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.321+0000 7fb736ffd700 1 -- 172.21.15.46:0/122885181 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73000b970 con 0x7fb7380ce3a0 2024-01-27T01:08:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.321+0000 7fb736ffd700 1 -- 172.21.15.46:0/122885181 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7300092e0 con 0x7fb7380ce3a0 2024-01-27T01:08:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.321+0000 7fb73e293700 1 -- 172.21.15.46:0/122885181 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7380ce3a0 msgr2=0x7fb7380ce7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.321+0000 7fb73e293700 1 --2- 172.21.15.46:0/122885181 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7380ce3a0 0x7fb7380ce7e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fb730009f20 tx=0x7fb730009840 comp rx=0 tx=0).stop 2024-01-27T01:08:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.322+0000 7fb73e293700 1 -- 172.21.15.46:0/122885181 shutdown_connections 2024-01-27T01:08:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.322+0000 7fb73e293700 1 --2- 172.21.15.46:0/122885181 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7380ce3a0 0x7fb7380ce7e0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.322+0000 7fb73e293700 1 --2- 172.21.15.46:0/122885181 >> v2:172.21.15.106:3300/0 conn(0x7fb7380c6820 0x7fb7380c6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.322+0000 7fb73e293700 1 -- 172.21.15.46:0/122885181 >> 172.21.15.46:0/122885181 conn(0x7fb73806f8c0 msgr2=0x7fb7380d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.322+0000 7fb73e293700 1 -- 172.21.15.46:0/122885181 shutdown_connections 2024-01-27T01:08:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.322+0000 7fb73e293700 1 -- 172.21.15.46:0/122885181 wait complete. 2024-01-27T01:08:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.323+0000 7fb73e293700 1 Processor -- start 2024-01-27T01:08:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.323+0000 7fb73e293700 1 -- start start 2024-01-27T01:08:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.324+0000 7fb73e293700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7380b2c60 0x7fb7380b6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.324+0000 7fb73e293700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7380c6820 0x7fb738152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.324+0000 7fb73e293700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7380ce3a0 0x7fb738158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.324+0000 7fb73e293700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb738153020 con 0x7fb7380c6820 2024-01-27T01:08:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.324+0000 7fb73e293700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb738169d20 con 0x7fb7380b2c60 2024-01-27T01:08:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.324+0000 7fb73e293700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380b36a0 con 0x7fb7380ce3a0 2024-01-27T01:08:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.324+0000 7fb7377fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7380c6820 0x7fb738152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.324+0000 7fb737fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7380b2c60 0x7fb7380b6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.324+0000 7fb7377fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7380c6820 0x7fb738152ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48222/0 (socket says 172.21.15.46:48222) 2024-01-27T01:08:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.324+0000 7fb7377fe700 1 -- 172.21.15.46:0/3166014321 learned_addr learned my addr 172.21.15.46:0/3166014321 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.324+0000 7fb73c830700 1 --2- 172.21.15.46:0/3166014321 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7380ce3a0 0x7fb738158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.324+0000 7fb737fff700 1 -- 172.21.15.46:0/3166014321 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7380ce3a0 msgr2=0x7fb738158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.324+0000 7fb737fff700 1 --2- 172.21.15.46:0/3166014321 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7380ce3a0 0x7fb738158160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.324+0000 7fb737fff700 1 -- 172.21.15.46:0/3166014321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7380c6820 msgr2=0x7fb738152ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.324+0000 7fb737fff700 1 --2- 172.21.15.46:0/3166014321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7380c6820 0x7fb738152ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.324+0000 7fb737fff700 1 -- 172.21.15.46:0/3166014321 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c009420 con 0x7fb7380b2c60 2024-01-27T01:08:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.324+0000 7fb7377fe700 1 --2- 172.21.15.46:0/3166014321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7380c6820 0x7fb738152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.325+0000 7fb737fff700 1 --2- 172.21.15.46:0/3166014321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7380b2c60 0x7fb7380b6100 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb72800c4c0 tx=0x7fb728007d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.325+0000 7fb71ffff700 1 -- 172.21.15.46:0/3166014321 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72800ce70 con 0x7fb7380b2c60 2024-01-27T01:08:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.325+0000 7fb71ffff700 1 -- 172.21.15.46:0/3166014321 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb728005a70 con 0x7fb7380b2c60 2024-01-27T01:08:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.325+0000 7fb71ffff700 1 -- 172.21.15.46:0/3166014321 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb728005e50 con 0x7fb7380b2c60 2024-01-27T01:08:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.325+0000 7fb73e293700 1 -- 172.21.15.46:0/3166014321 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb730009ae0 con 0x7fb7380b2c60 2024-01-27T01:08:44.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.325+0000 7fb73e293700 1 -- 172.21.15.46:0/3166014321 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7380b3f40 con 0x7fb7380b2c60 2024-01-27T01:08:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.326+0000 7fb73e293700 1 -- 172.21.15.46:0/3166014321 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7380b3a90 con 0x7fb7380b2c60 2024-01-27T01:08:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.326+0000 7fb71ffff700 1 -- 172.21.15.46:0/3166014321 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7fb7280072e0 con 0x7fb7380b2c60 2024-01-27T01:08:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.327+0000 7fb71ffff700 1 --2- 172.21.15.46:0/3166014321 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb7240670e0 0x7fb724069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.327+0000 7fb7377fe700 1 -- 172.21.15.46:0/3166014321 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb7240670e0 msgr2=0x7fb724069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.327+0000 7fb71ffff700 1 -- 172.21.15.46:0/3166014321 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6990+0+0 (secure 0 0 0) 0x7fb728084090 con 0x7fb7380b2c60 2024-01-27T01:08:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.327+0000 7fb7377fe700 1 --2- 172.21.15.46:0/3166014321 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb7240670e0 0x7fb724069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.331+0000 7fb71ffff700 1 -- 172.21.15.46:0/3166014321 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb728050b80 con 0x7fb7380b2c60 2024-01-27T01:08:44.355 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:44.355+0000 7f4965bd41c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:08:44.478 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:44.479+0000 7f4965bd41c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:08:44.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.528+0000 7fb7377fe700 1 -- 172.21.15.46:0/3166014321 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb7240670e0 msgr2=0x7fb724069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:44.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.528+0000 7fb7377fe700 1 --2- 172.21.15.46:0/3166014321 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb7240670e0 0x7fb724069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:08:44.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.539+0000 7fb73e293700 1 -- 172.21.15.46:0/3166014321 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb73804c710 con 0x7fb7380b2c60 2024-01-27T01:08:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.540+0000 7fb71ffff700 1 -- 172.21.15.46:0/3166014321 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v265) v1 ==== 74+0+17013 (secure 0 0 0) 0x7fb72806ae60 con 0x7fb7380b2c60 2024-01-27T01:08:44.540 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:44.540 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":265,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:08:35.385018+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:08:19.198347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-01-27T01:08:19.674810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"263","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:6837/62434":"2024-01-28T01:08:35.384966+0000","172.21.15.106:6836/62434":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3497341404":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3894462583":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/1144438896":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3166472261":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/1405196355":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/1752514247":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/2487246395":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3477459747":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3540935782":"2024-01-27T02:08:26.076460+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:08:44.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.542+0000 7fb73e293700 1 -- 172.21.15.46:0/3166014321 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb7240670e0 msgr2=0x7fb724069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.542+0000 7fb73e293700 1 --2- 172.21.15.46:0/3166014321 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb7240670e0 0x7fb724069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.542+0000 7fb73e293700 1 -- 172.21.15.46:0/3166014321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7380b2c60 msgr2=0x7fb7380b6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.542+0000 7fb73e293700 1 --2- 172.21.15.46:0/3166014321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7380b2c60 0x7fb7380b6100 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb72800c4c0 tx=0x7fb728007d40 comp rx=0 tx=0).stop 2024-01-27T01:08:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.542+0000 7fb73e293700 1 -- 172.21.15.46:0/3166014321 shutdown_connections 2024-01-27T01:08:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.542+0000 7fb73e293700 1 --2- 172.21.15.46:0/3166014321 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb7240670e0 0x7fb724069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.542+0000 7fb73e293700 1 --2- 172.21.15.46:0/3166014321 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7380ce3a0 0x7fb738158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.542+0000 7fb73e293700 1 --2- 172.21.15.46:0/3166014321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7380b2c60 0x7fb7380b6100 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.542+0000 7fb73e293700 1 --2- 172.21.15.46:0/3166014321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7380c6820 0x7fb738152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.542+0000 7fb73e293700 1 -- 172.21.15.46:0/3166014321 >> 172.21.15.46:0/3166014321 conn(0x7fb73806f8c0 msgr2=0x7fb7381590a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.542+0000 7fb73e293700 1 -- 172.21.15.46:0/3166014321 shutdown_connections 2024-01-27T01:08:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.542+0000 7fb73e293700 1 -- 172.21.15.46:0/3166014321 wait complete. 2024-01-27T01:08:44.557 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-27T01:08:44.557 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:08:44.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.693+0000 7ff1e9adf700 1 Processor -- start 2024-01-27T01:08:44.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.693+0000 7ff1e9adf700 1 -- start start 2024-01-27T01:08:44.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.694+0000 7ff1e9adf700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff1e40c8c30 0x7ff1e4159d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:44.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.694+0000 7ff1e9adf700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff1e40cf0b0 0x7ff1e4069d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:44.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.694+0000 7ff1e9adf700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e40cf530 con 0x7ff1e41599b0 2024-01-27T01:08:44.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.694+0000 7ff1e9adf700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1e40c43a0 con 0x7ff1e40cf0b0 2024-01-27T01:08:44.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.694+0000 7ff1e9adf700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e40c45b0 con 0x7ff1e40c8c30 2024-01-27T01:08:44.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.694+0000 7ff1e3fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff1e40cf0b0 0x7ff1e4069d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:44.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.694+0000 7ff1e3fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff1e40cf0b0 0x7ff1e4069d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48224/0 (socket says 172.21.15.46:48224) 2024-01-27T01:08:44.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.694+0000 7ff1e3fff700 1 -- 172.21.15.46:0/3925589122 learned_addr learned my addr 172.21.15.46:0/3925589122 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.694+0000 7ff1e2ffd700 1 --2- 172.21.15.46:0/3925589122 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff1e40c8c30 0x7ff1e4159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.695+0000 7ff1e3fff700 1 -- 172.21.15.46:0/3925589122 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff1e40c8c30 msgr2=0x7ff1e4159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.695+0000 7ff1e3fff700 1 --2- 172.21.15.46:0/3925589122 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff1e40c8c30 0x7ff1e4159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.695+0000 7ff1e27fc700 1 -- 172.21.15.46:0/3925589122 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147640108 0 0) 0x7ff1e40cf530 con 0x7ff1e41599b0 2024-01-27T01:08:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.695+0000 7ff1e3fff700 1 -- 172.21.15.46:0/3925589122 >> v1:172.21.15.106:6789/0 conn(0x7ff1e41599b0 legacy=0x7ff1e4069640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.695+0000 7ff1e3fff700 1 -- 172.21.15.46:0/3925589122 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e40c4890 con 0x7ff1e40cf0b0 2024-01-27T01:08:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.695+0000 7ff1e3fff700 1 --2- 172.21.15.46:0/3925589122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e40cf0b0 0x7ff1e4069d50 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc009f20 tx=0x7ff1dc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59519858128c73a7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.695+0000 7ff1e27fc700 1 -- 172.21.15.46:0/3925589122 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc00e070 con 0x7ff1e40cf0b0 2024-01-27T01:08:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.695+0000 7ff1e27fc700 1 -- 172.21.15.46:0/3925589122 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1dc00b9d0 con 0x7ff1e40cf0b0 2024-01-27T01:08:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.695+0000 7ff1e27fc700 1 -- 172.21.15.46:0/3925589122 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc0092e0 con 0x7ff1e40cf0b0 2024-01-27T01:08:44.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.696+0000 7ff1e9adf700 1 -- 172.21.15.46:0/3925589122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e40cf0b0 msgr2=0x7ff1e4069d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:44.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.696+0000 7ff1e9adf700 1 --2- 172.21.15.46:0/3925589122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e40cf0b0 0x7ff1e4069d50 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc009f20 tx=0x7ff1dc009840 comp rx=0 tx=0).stop 2024-01-27T01:08:44.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.696+0000 7ff1e9adf700 1 -- 172.21.15.46:0/3925589122 shutdown_connections 2024-01-27T01:08:44.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.696+0000 7ff1e9adf700 1 --2- 172.21.15.46:0/3925589122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e40cf0b0 0x7ff1e4069d50 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.696+0000 7ff1e9adf700 1 --2- 172.21.15.46:0/3925589122 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff1e40c8c30 0x7ff1e4159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.696+0000 7ff1e9adf700 1 -- 172.21.15.46:0/3925589122 >> 172.21.15.46:0/3925589122 conn(0x7ff1e4087fa0 msgr2=0x7ff1e40ce5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:44.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.696+0000 7ff1e9adf700 1 -- 172.21.15.46:0/3925589122 shutdown_connections 2024-01-27T01:08:44.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.696+0000 7ff1e9adf700 1 -- 172.21.15.46:0/3925589122 wait complete. 2024-01-27T01:08:44.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.697+0000 7ff1e9adf700 1 Processor -- start 2024-01-27T01:08:44.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.697+0000 7ff1e9adf700 1 -- start start 2024-01-27T01:08:44.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.698+0000 7ff1e9adf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e40c8c30 0x7ff1e40b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:44.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.698+0000 7ff1e9adf700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff1e40cf0b0 0x7ff1e40c6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:44.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.698+0000 7ff1e9adf700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff1e41599b0 0x7ff1e4154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:44.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.698+0000 7ff1e9adf700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e40c6900 con 0x7ff1e40c8c30 2024-01-27T01:08:44.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.698+0000 7ff1e9adf700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e40c6be0 con 0x7ff1e40cf0b0 2024-01-27T01:08:44.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.698+0000 7ff1e9adf700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e40c6ec0 con 0x7ff1e41599b0 2024-01-27T01:08:44.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.698+0000 7ff1e37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e40c8c30 0x7ff1e40b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:44.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.698+0000 7ff1e37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e40c8c30 0x7ff1e40b30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48228/0 (socket says 172.21.15.46:48228) 2024-01-27T01:08:44.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.698+0000 7ff1e3fff700 1 --2- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff1e41599b0 0x7ff1e4154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.698+0000 7ff1e37fe700 1 -- 172.21.15.46:0/2147586230 learned_addr learned my addr 172.21.15.46:0/2147586230 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.698+0000 7ff1e2ffd700 1 --2- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff1e40cf0b0 0x7ff1e40c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.699+0000 7ff1e37fe700 1 -- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff1e41599b0 msgr2=0x7ff1e4154630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.699+0000 7ff1e37fe700 1 --2- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff1e41599b0 0x7ff1e4154630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.699+0000 7ff1e37fe700 1 -- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff1e40cf0b0 msgr2=0x7ff1e40c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.699+0000 7ff1e37fe700 1 --2- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff1e40cf0b0 0x7ff1e40c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.699+0000 7ff1e37fe700 1 -- 172.21.15.46:0/2147586230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1dc009ae0 con 0x7ff1e40c8c30 2024-01-27T01:08:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.699+0000 7ff1e2ffd700 1 --2- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff1e40cf0b0 0x7ff1e40c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.699+0000 7ff1e3fff700 1 --2- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff1e41599b0 0x7ff1e4154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.699+0000 7ff1e37fe700 1 --2- 172.21.15.46:0/2147586230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e40c8c30 0x7ff1e40b30c0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4001f00 tx=0x7ff1d4001060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.699+0000 7ff1cb7fe700 1 -- 172.21.15.46:0/2147586230 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4011590 con 0x7ff1e40c8c30 2024-01-27T01:08:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.699+0000 7ff1cb7fe700 1 -- 172.21.15.46:0/2147586230 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d40116f0 con 0x7ff1e40c8c30 2024-01-27T01:08:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.699+0000 7ff1e9adf700 1 -- 172.21.15.46:0/2147586230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e40c71a0 con 0x7ff1e40c8c30 2024-01-27T01:08:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.699+0000 7ff1cb7fe700 1 -- 172.21.15.46:0/2147586230 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4011ab0 con 0x7ff1e40c8c30 2024-01-27T01:08:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.699+0000 7ff1e9adf700 1 -- 172.21.15.46:0/2147586230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e40c65c0 con 0x7ff1e40c8c30 2024-01-27T01:08:44.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.700+0000 7ff1e9adf700 1 -- 172.21.15.46:0/2147586230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e404c710 con 0x7ff1e40c8c30 2024-01-27T01:08:44.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.701+0000 7ff1cb7fe700 1 -- 172.21.15.46:0/2147586230 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7ff1d401b3b0 con 0x7ff1e40c8c30 2024-01-27T01:08:44.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.701+0000 7ff1cb7fe700 1 --2- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff1d00670e0 0x7ff1d0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:44.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.701+0000 7ff1cb7fe700 1 -- 172.21.15.46:0/2147586230 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6990+0+0 (secure 0 0 0) 0x7ff1d4084180 con 0x7ff1e40c8c30 2024-01-27T01:08:44.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.701+0000 7ff1e2ffd700 1 -- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff1d00670e0 msgr2=0x7ff1d0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:44.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.701+0000 7ff1e2ffd700 1 --2- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff1d00670e0 0x7ff1d0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:44.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.706+0000 7ff1cb7fe700 1 -- 172.21.15.46:0/2147586230 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff1d4050d20 con 0x7ff1e40c8c30 2024-01-27T01:08:44.797 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:44.797+0000 7f66656d31c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T01:08:44.884 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:44.884+0000 7f66656d31c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T01:08:44.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.902+0000 7ff1e2ffd700 1 -- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff1d00670e0 msgr2=0x7ff1d0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:44.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.902+0000 7ff1e2ffd700 1 --2- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff1d00670e0 0x7ff1d0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:08:44.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.962+0000 7ff1e9adf700 1 -- 172.21.15.46:0/2147586230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1e40b43e0 con 0x7ff1e40c8c30 2024-01-27T01:08:44.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.963+0000 7ff1cb7fe700 1 -- 172.21.15.46:0/2147586230 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+4616 (secure 0 0 0) 0x7ff1d4017090 con 0x7ff1e40c8c30 2024-01-27T01:08:44.963 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:44.963 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":149,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3826743114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3826743114},{"type":"v1","addr":"172.21.15.46:6835","nonce":3826743114}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":144},{"gid":11947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2514069949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2514069949},{"type":"v1","addr":"172.21.15.106:6833","nonce":2514069949}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":144},{"gid":11950,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/251184858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":251184858},{"type":"v1","addr":"172.21.15.106:6835","nonce":251184858}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:08:20.703665+0000","modified":"2024-01-27T01:08:24.272153+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12348},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12348":{"gid":12348,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6833/3798781145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3798781145},{"type":"v1","addr":"172.21.15.46:6833","nonce":3798781145}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2024-01-27T01:08:44.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.965+0000 7ff1e9adf700 1 -- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff1d00670e0 msgr2=0x7ff1d0069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:44.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.965+0000 7ff1e9adf700 1 --2- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff1d00670e0 0x7ff1d0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.965+0000 7ff1e9adf700 1 -- 172.21.15.46:0/2147586230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e40c8c30 msgr2=0x7ff1e40b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:44.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.965+0000 7ff1e9adf700 1 --2- 172.21.15.46:0/2147586230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e40c8c30 0x7ff1e40b30c0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4001f00 tx=0x7ff1d4001060 comp rx=0 tx=0).stop 2024-01-27T01:08:44.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.965+0000 7ff1e9adf700 1 -- 172.21.15.46:0/2147586230 shutdown_connections 2024-01-27T01:08:44.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.965+0000 7ff1e9adf700 1 --2- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff1d00670e0 0x7ff1d0069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.965+0000 7ff1e9adf700 1 --2- 172.21.15.46:0/2147586230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e40c8c30 0x7ff1e40b30c0 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.965+0000 7ff1e9adf700 1 --2- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff1e41599b0 0x7ff1e4154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.965+0000 7ff1e9adf700 1 --2- 172.21.15.46:0/2147586230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff1e40cf0b0 0x7ff1e40c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:44.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.965+0000 7ff1e9adf700 1 -- 172.21.15.46:0/2147586230 >> 172.21.15.46:0/2147586230 conn(0x7ff1e4087fa0 msgr2=0x7ff1e40ccd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:44.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.966+0000 7ff1e9adf700 1 -- 172.21.15.46:0/2147586230 shutdown_connections 2024-01-27T01:08:44.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:44.966+0000 7ff1e9adf700 1 -- 172.21.15.46:0/2147586230 wait complete. 2024-01-27T01:08:44.967 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 149 2024-01-27T01:08:44.980 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:08:44.981 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:44.982+0000 7f4965bd41c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:08:45.031 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:45.031+0000 7f66656d31c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T01:08:45.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.114+0000 7f055e772700 1 Processor -- start 2024-01-27T01:08:45.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.114+0000 7f055e772700 1 -- start start 2024-01-27T01:08:45.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.115+0000 7f055e772700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0558159770 0x7f05580cb100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.115+0000 7f055e772700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05580cf820 0x7f05581634a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.115+0000 7f055e772700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05580ce380 con 0x7f0558159b40 2024-01-27T01:08:45.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.115+0000 7f055e772700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05580cbaa0 con 0x7f0558159770 2024-01-27T01:08:45.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.115+0000 7f055e772700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05580cbd80 con 0x7f05580cf820 2024-01-27T01:08:45.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.115+0000 7f0557fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0558159770 0x7f05580cb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:45.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.115+0000 7f0557fff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0558159770 0x7f05580cb100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48238/0 (socket says 172.21.15.46:48238) 2024-01-27T01:08:45.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.115+0000 7f0557fff700 1 -- 172.21.15.46:0/3983557327 learned_addr learned my addr 172.21.15.46:0/3983557327 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:45.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.115+0000 7f055cd0f700 1 --2- 172.21.15.46:0/3983557327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05580cf820 0x7f05581634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:45.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.116+0000 7f0557fff700 1 -- 172.21.15.46:0/3983557327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05580cf820 msgr2=0x7f05581634a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:45.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.116+0000 7f0557fff700 1 --2- 172.21.15.46:0/3983557327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05580cf820 0x7f05581634a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.116+0000 7f0557fff700 1 -- 172.21.15.46:0/3983557327 >> v1:172.21.15.106:6789/0 conn(0x7f0558159b40 legacy=0x7f0558159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:45.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.116+0000 7f0556ffd700 1 -- 172.21.15.46:0/3983557327 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702873413 0 0) 0x7f05580ce380 con 0x7f0558159b40 2024-01-27T01:08:45.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.116+0000 7f0557fff700 1 -- 172.21.15.46:0/3983557327 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05580cc060 con 0x7f0558159770 2024-01-27T01:08:45.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.116+0000 7f0557fff700 1 --2- 172.21.15.46:0/3983557327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0558159770 0x7f05580cb100 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f05480099b0 tx=0x7f0548009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=273f9a38644e2964 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:45.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.116+0000 7f0556ffd700 1 -- 172.21.15.46:0/3983557327 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0548009ea0 con 0x7f0558159770 2024-01-27T01:08:45.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.116+0000 7f0556ffd700 1 -- 172.21.15.46:0/3983557327 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054800b860 con 0x7f0558159770 2024-01-27T01:08:45.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.116+0000 7f0556ffd700 1 -- 172.21.15.46:0/3983557327 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054800bc00 con 0x7f0558159770 2024-01-27T01:08:45.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.117+0000 7f055e772700 1 -- 172.21.15.46:0/3983557327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0558159770 msgr2=0x7f05580cb100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:45.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.117+0000 7f055e772700 1 --2- 172.21.15.46:0/3983557327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0558159770 0x7f05580cb100 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f05480099b0 tx=0x7f0548009250 comp rx=0 tx=0).stop 2024-01-27T01:08:45.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.117+0000 7f055e772700 1 -- 172.21.15.46:0/3983557327 shutdown_connections 2024-01-27T01:08:45.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.117+0000 7f055e772700 1 --2- 172.21.15.46:0/3983557327 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f05580cf820 0x7f05581634a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.117+0000 7f055e772700 1 --2- 172.21.15.46:0/3983557327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0558159770 0x7f05580cb100 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.117+0000 7f055e772700 1 -- 172.21.15.46:0/3983557327 >> 172.21.15.46:0/3983557327 conn(0x7f0558099a40 msgr2=0x7f055806f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:45.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.117+0000 7f055e772700 1 -- 172.21.15.46:0/3983557327 shutdown_connections 2024-01-27T01:08:45.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.117+0000 7f055e772700 1 -- 172.21.15.46:0/3983557327 wait complete. 2024-01-27T01:08:45.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.118+0000 7f055e772700 1 Processor -- start 2024-01-27T01:08:45.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.118+0000 7f055e772700 1 -- start start 2024-01-27T01:08:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.119+0000 7f055e772700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05580cf820 0x7f05580c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.119+0000 7f055e772700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0558159770 0x7f055809a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.119+0000 7f055e772700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0558159b40 0x7f055809dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.119+0000 7f055e772700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055809a660 con 0x7f05580cf820 2024-01-27T01:08:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.119+0000 7f055e772700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05580c7e30 con 0x7f0558159b40 2024-01-27T01:08:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.119+0000 7f055e772700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05580c8110 con 0x7f0558159770 2024-01-27T01:08:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.119+0000 7f0557fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05580cf820 0x7f05580c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.119+0000 7f0557fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05580cf820 0x7f05580c77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48254/0 (socket says 172.21.15.46:48254) 2024-01-27T01:08:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.119+0000 7f0557fff700 1 -- 172.21.15.46:0/464158204 learned_addr learned my addr 172.21.15.46:0/464158204 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.119+0000 7f05577fe700 1 --2- 172.21.15.46:0/464158204 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0558159770 0x7f055809a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.119+0000 7f055cd0f700 1 --2- 172.21.15.46:0/464158204 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0558159b40 0x7f055809dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.119+0000 7f0557fff700 1 -- 172.21.15.46:0/464158204 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0558159770 msgr2=0x7f055809a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.119+0000 7f0557fff700 1 --2- 172.21.15.46:0/464158204 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0558159770 0x7f055809a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.119+0000 7f0557fff700 1 -- 172.21.15.46:0/464158204 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0558159b40 msgr2=0x7f055809dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.119+0000 7f0557fff700 1 --2- 172.21.15.46:0/464158204 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0558159b40 0x7f055809dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.119+0000 7f0557fff700 1 -- 172.21.15.46:0/464158204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05480094f0 con 0x7f05580cf820 2024-01-27T01:08:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.120+0000 7f0557fff700 1 --2- 172.21.15.46:0/464158204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05580cf820 0x7f05580c77a0 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f05580712b0 tx=0x7f0548017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.120+0000 7f053ffff700 1 -- 172.21.15.46:0/464158204 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0548000d10 con 0x7f05580cf820 2024-01-27T01:08:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.120+0000 7f053ffff700 1 -- 172.21.15.46:0/464158204 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054800bbe0 con 0x7f05580cf820 2024-01-27T01:08:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.120+0000 7f055e772700 1 -- 172.21.15.46:0/464158204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05580c83f0 con 0x7f05580cf820 2024-01-27T01:08:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.120+0000 7f053ffff700 1 -- 172.21.15.46:0/464158204 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0548011670 con 0x7f05580cf820 2024-01-27T01:08:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.120+0000 7f055e772700 1 -- 172.21.15.46:0/464158204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05580c89b0 con 0x7f05580cf820 2024-01-27T01:08:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.121+0000 7f055e772700 1 -- 172.21.15.46:0/464158204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05580c8ba0 con 0x7f05580cf820 2024-01-27T01:08:45.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.122+0000 7f053ffff700 1 -- 172.21.15.46:0/464158204 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f0548011aa0 con 0x7f05580cf820 2024-01-27T01:08:45.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.122+0000 7f053ffff700 1 --2- 172.21.15.46:0/464158204 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f05440670e0 0x7f0544069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.122+0000 7f05577fe700 1 -- 172.21.15.46:0/464158204 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f05440670e0 msgr2=0x7f0544069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:45.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.122+0000 7f053ffff700 1 -- 172.21.15.46:0/464158204 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6990+0+0 (secure 0 0 0) 0x7f05480865f0 con 0x7f05580cf820 2024-01-27T01:08:45.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.122+0000 7f05577fe700 1 --2- 172.21.15.46:0/464158204 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f05440670e0 0x7f0544069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:45.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.127+0000 7f053ffff700 1 -- 172.21.15.46:0/464158204 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f05480530e0 con 0x7f05580cf820 2024-01-27T01:08:45.132 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:45.132+0000 7f4965bd41c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:08:45.193 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:08:45.196 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:08:45.289 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:45.289+0000 7f4965bd41c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:08:45.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.323+0000 7f05577fe700 1 -- 172.21.15.46:0/464158204 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f05440670e0 msgr2=0x7f0544069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:45.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.323+0000 7f05577fe700 1 --2- 172.21.15.46:0/464158204 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f05440670e0 0x7f0544069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:08:45.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.386+0000 7f055e772700 1 -- 172.21.15.46:0/464158204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f055804c710 con 0x7f05580cf820 2024-01-27T01:08:45.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.387+0000 7f053ffff700 1 -- 172.21.15.46:0/464158204 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+4616 (secure 0 0 0) 0x7f0548056700 con 0x7f05580cf820 2024-01-27T01:08:45.387 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:45.387 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":149,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3826743114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3826743114},{"type":"v1","addr":"172.21.15.46:6835","nonce":3826743114}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":144},{"gid":11947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2514069949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2514069949},{"type":"v1","addr":"172.21.15.106:6833","nonce":2514069949}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":144},{"gid":11950,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/251184858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":251184858},{"type":"v1","addr":"172.21.15.106:6835","nonce":251184858}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-27T01:08:20.703665+0000","modified":"2024-01-27T01:08:24.272153+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12348},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12348":{"gid":12348,"name":"c","rank":0,"incarnation":145,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6833/3798781145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3798781145},{"type":"v1","addr":"172.21.15.46:6833","nonce":3798781145}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2024-01-27T01:08:45.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.389+0000 7f055e772700 1 -- 172.21.15.46:0/464158204 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f05440670e0 msgr2=0x7f0544069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:45.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.389+0000 7f055e772700 1 --2- 172.21.15.46:0/464158204 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f05440670e0 0x7f0544069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.389+0000 7f055e772700 1 -- 172.21.15.46:0/464158204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05580cf820 msgr2=0x7f05580c77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:45.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.389+0000 7f055e772700 1 --2- 172.21.15.46:0/464158204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05580cf820 0x7f05580c77a0 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f05580712b0 tx=0x7f0548017000 comp rx=0 tx=0).stop 2024-01-27T01:08:45.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.389+0000 7f055e772700 1 -- 172.21.15.46:0/464158204 shutdown_connections 2024-01-27T01:08:45.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.389+0000 7f055e772700 1 --2- 172.21.15.46:0/464158204 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f05440670e0 0x7f0544069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.389+0000 7f055e772700 1 --2- 172.21.15.46:0/464158204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05580cf820 0x7f05580c77a0 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.389+0000 7f055e772700 1 --2- 172.21.15.46:0/464158204 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0558159b40 0x7f055809dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.389+0000 7f055e772700 1 --2- 172.21.15.46:0/464158204 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0558159770 0x7f055809a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.389+0000 7f055e772700 1 -- 172.21.15.46:0/464158204 >> 172.21.15.46:0/464158204 conn(0x7f0558099a40 msgr2=0x7f0558155050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:45.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.389+0000 7f055e772700 1 -- 172.21.15.46:0/464158204 shutdown_connections 2024-01-27T01:08:45.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.389+0000 7f055e772700 1 -- 172.21.15.46:0/464158204 wait complete. 2024-01-27T01:08:45.390 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 149 2024-01-27T01:08:45.404 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:08:45.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.541+0000 7ff29f5d3700 1 Processor -- start 2024-01-27T01:08:45.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.541+0000 7ff29f5d3700 1 -- start start 2024-01-27T01:08:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.542+0000 7ff29f5d3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2981599b0 0x7ff2980c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.542+0000 7ff29f5d3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff2980cf0b0 0x7ff298069d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.542+0000 7ff29f5d3700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2980cf530 con 0x7ff298159d80 2024-01-27T01:08:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.542+0000 7ff29f5d3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2980c43a0 con 0x7ff2980cf0b0 2024-01-27T01:08:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.542+0000 7ff29f5d3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2980c45b0 con 0x7ff2981599b0 2024-01-27T01:08:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.542+0000 7ff29db70700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff2980cf0b0 0x7ff298069d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.542+0000 7ff29db70700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff2980cf0b0 0x7ff298069d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48270/0 (socket says 172.21.15.46:48270) 2024-01-27T01:08:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.542+0000 7ff29cb6e700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff298159d80 0x7ff298069640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59336/0 (socket says 172.21.15.46:59336) 2024-01-27T01:08:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.542+0000 7ff29db70700 1 -- 172.21.15.46:0/3477974002 learned_addr learned my addr 172.21.15.46:0/3477974002 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.542+0000 7ff29d36f700 1 --2- 172.21.15.46:0/3477974002 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2981599b0 0x7ff2980c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.542+0000 7ff29db70700 1 -- 172.21.15.46:0/3477974002 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2981599b0 msgr2=0x7ff2980c8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.542+0000 7ff29db70700 1 --2- 172.21.15.46:0/3477974002 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2981599b0 0x7ff2980c8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.542+0000 7ff29db70700 1 -- 172.21.15.46:0/3477974002 >> v1:172.21.15.46:6789/0 conn(0x7ff298159d80 legacy=0x7ff298069640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.542+0000 7ff28ffff700 1 -- 172.21.15.46:0/3477974002 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274489383 0 0) 0x7ff2980cf530 con 0x7ff298159d80 2024-01-27T01:08:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.542+0000 7ff29db70700 1 -- 172.21.15.46:0/3477974002 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2980c4890 con 0x7ff2980cf0b0 2024-01-27T01:08:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.543+0000 7ff29db70700 1 --2- 172.21.15.46:0/3477974002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2980cf0b0 0x7ff298069d50 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7ff290009f20 tx=0x7ff290009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bda4d72c2bc4c815 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.543+0000 7ff28ffff700 1 -- 172.21.15.46:0/3477974002 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29000e070 con 0x7ff2980cf0b0 2024-01-27T01:08:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.543+0000 7ff28ffff700 1 -- 172.21.15.46:0/3477974002 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29000b9d0 con 0x7ff2980cf0b0 2024-01-27T01:08:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.543+0000 7ff28ffff700 1 -- 172.21.15.46:0/3477974002 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2900092e0 con 0x7ff2980cf0b0 2024-01-27T01:08:45.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.543+0000 7ff29f5d3700 1 -- 172.21.15.46:0/3477974002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2980cf0b0 msgr2=0x7ff298069d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:45.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.543+0000 7ff29f5d3700 1 --2- 172.21.15.46:0/3477974002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2980cf0b0 0x7ff298069d50 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7ff290009f20 tx=0x7ff290009840 comp rx=0 tx=0).stop 2024-01-27T01:08:45.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.544+0000 7ff29f5d3700 1 -- 172.21.15.46:0/3477974002 shutdown_connections 2024-01-27T01:08:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.544+0000 7ff29f5d3700 1 --2- 172.21.15.46:0/3477974002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2980cf0b0 0x7ff298069d50 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.544+0000 7ff29f5d3700 1 --2- 172.21.15.46:0/3477974002 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2981599b0 0x7ff2980c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.544+0000 7ff29f5d3700 1 -- 172.21.15.46:0/3477974002 >> 172.21.15.46:0/3477974002 conn(0x7ff298087fa0 msgr2=0x7ff2980ce5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.544+0000 7ff29f5d3700 1 -- 172.21.15.46:0/3477974002 shutdown_connections 2024-01-27T01:08:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.544+0000 7ff29f5d3700 1 -- 172.21.15.46:0/3477974002 wait complete. 2024-01-27T01:08:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.545+0000 7ff29f5d3700 1 Processor -- start 2024-01-27T01:08:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.545+0000 7ff29f5d3700 1 -- start start 2024-01-27T01:08:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.547+0000 7ff29f5d3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2980cf0b0 0x7ff2981420c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.547+0000 7ff29f5d3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2981599b0 0x7ff2980c6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.547+0000 7ff29f5d3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff298159d80 0x7ff2980c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.547+0000 7ff29f5d3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2980b1430 con 0x7ff2980cf0b0 2024-01-27T01:08:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.547+0000 7ff29f5d3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2980b1710 con 0x7ff298159d80 2024-01-27T01:08:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.547+0000 7ff29f5d3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2980b19f0 con 0x7ff2981599b0 2024-01-27T01:08:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.547+0000 7ff29d36f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2980cf0b0 0x7ff2981420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.547+0000 7ff29d36f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2980cf0b0 0x7ff2981420c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48276/0 (socket says 172.21.15.46:48276) 2024-01-27T01:08:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.547+0000 7ff29d36f700 1 -- 172.21.15.46:0/71037268 learned_addr learned my addr 172.21.15.46:0/71037268 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.547+0000 7ff29db70700 1 --2- 172.21.15.46:0/71037268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff298159d80 0x7ff2980c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.547+0000 7ff29cb6e700 1 --2- 172.21.15.46:0/71037268 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2981599b0 0x7ff2980c6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.547+0000 7ff29d36f700 1 -- 172.21.15.46:0/71037268 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2981599b0 msgr2=0x7ff2980c6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.547+0000 7ff29d36f700 1 --2- 172.21.15.46:0/71037268 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2981599b0 0x7ff2980c6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.547+0000 7ff29d36f700 1 -- 172.21.15.46:0/71037268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff298159d80 msgr2=0x7ff2980c6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:45.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.547+0000 7ff29d36f700 1 --2- 172.21.15.46:0/71037268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff298159d80 0x7ff2980c6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.547+0000 7ff29d36f700 1 -- 172.21.15.46:0/71037268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff290009ae0 con 0x7ff2980cf0b0 2024-01-27T01:08:45.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.547+0000 7ff29d36f700 1 --2- 172.21.15.46:0/71037268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2980cf0b0 0x7ff2981420c0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7ff29400ec80 tx=0x7ff294009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:45.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.548+0000 7ff28d7fa700 1 -- 172.21.15.46:0/71037268 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29400c850 con 0x7ff2980cf0b0 2024-01-27T01:08:45.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.548+0000 7ff28d7fa700 1 -- 172.21.15.46:0/71037268 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29400c9b0 con 0x7ff2980cf0b0 2024-01-27T01:08:45.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.548+0000 7ff29f5d3700 1 -- 172.21.15.46:0/71037268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2980b1cd0 con 0x7ff2980cf0b0 2024-01-27T01:08:45.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.548+0000 7ff28d7fa700 1 -- 172.21.15.46:0/71037268 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29400cd30 con 0x7ff2980cf0b0 2024-01-27T01:08:45.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.548+0000 7ff29f5d3700 1 -- 172.21.15.46:0/71037268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29815a650 con 0x7ff2980cf0b0 2024-01-27T01:08:45.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.549+0000 7ff29f5d3700 1 -- 172.21.15.46:0/71037268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff29804c710 con 0x7ff2980cf0b0 2024-01-27T01:08:45.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.549+0000 7ff28d7fa700 1 -- 172.21.15.46:0/71037268 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7ff294013680 con 0x7ff2980cf0b0 2024-01-27T01:08:45.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.550+0000 7ff28d7fa700 1 --2- 172.21.15.46:0/71037268 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff27c0670e0 0x7ff27c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.550+0000 7ff29cb6e700 1 -- 172.21.15.46:0/71037268 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff27c0670e0 msgr2=0x7ff27c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:45.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.550+0000 7ff28d7fa700 1 -- 172.21.15.46:0/71037268 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6990+0+0 (secure 0 0 0) 0x7ff294085720 con 0x7ff2980cf0b0 2024-01-27T01:08:45.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.550+0000 7ff29cb6e700 1 --2- 172.21.15.46:0/71037268 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff27c0670e0 0x7ff27c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:45.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.554+0000 7ff28d7fa700 1 -- 172.21.15.46:0/71037268 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff294052290 con 0x7ff2980cf0b0 2024-01-27T01:08:45.642 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:45.642+0000 7f66656d31c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T01:08:45.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.751+0000 7ff29cb6e700 1 -- 172.21.15.46:0/71037268 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff27c0670e0 msgr2=0x7ff27c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:45.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.751+0000 7ff29cb6e700 1 --2- 172.21.15.46:0/71037268 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff27c0670e0 0x7ff27c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:08:45.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.761+0000 7ff29f5d3700 1 -- 172.21.15.46:0/71037268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2980b43e0 con 0x7ff2980cf0b0 2024-01-27T01:08:45.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.762+0000 7ff28d7fa700 1 -- 172.21.15.46:0/71037268 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v265) v1 ==== 74+0+17013 (secure 0 0 0) 0x7ff2940558b0 con 0x7ff2980cf0b0 2024-01-27T01:08:45.762 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:45.763 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":265,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:08:35.385018+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-01-27T01:08:19.198347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-01-27T01:08:19.674810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"263","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:6837/62434":"2024-01-28T01:08:35.384966+0000","172.21.15.106:6836/62434":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3497341404":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3894462583":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/1144438896":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3166472261":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/1405196355":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/1752514247":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/2487246395":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3477459747":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3540935782":"2024-01-27T02:08:26.076460+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:08:45.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.764+0000 7ff29f5d3700 1 -- 172.21.15.46:0/71037268 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff27c0670e0 msgr2=0x7ff27c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:45.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.764+0000 7ff29f5d3700 1 --2- 172.21.15.46:0/71037268 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff27c0670e0 0x7ff27c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.764+0000 7ff29f5d3700 1 -- 172.21.15.46:0/71037268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2980cf0b0 msgr2=0x7ff2981420c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:45.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.764+0000 7ff29f5d3700 1 --2- 172.21.15.46:0/71037268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2980cf0b0 0x7ff2981420c0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7ff29400ec80 tx=0x7ff294009120 comp rx=0 tx=0).stop 2024-01-27T01:08:45.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.764+0000 7ff29f5d3700 1 -- 172.21.15.46:0/71037268 shutdown_connections 2024-01-27T01:08:45.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.764+0000 7ff29f5d3700 1 --2- 172.21.15.46:0/71037268 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7ff27c0670e0 0x7ff27c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.764+0000 7ff29f5d3700 1 --2- 172.21.15.46:0/71037268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2980cf0b0 0x7ff2981420c0 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.764+0000 7ff29f5d3700 1 --2- 172.21.15.46:0/71037268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff298159d80 0x7ff2980c6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.764+0000 7ff29f5d3700 1 --2- 172.21.15.46:0/71037268 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2981599b0 0x7ff2980c6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.764+0000 7ff29f5d3700 1 -- 172.21.15.46:0/71037268 >> 172.21.15.46:0/71037268 conn(0x7ff298087fa0 msgr2=0x7ff2980cb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:45.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.764+0000 7ff29f5d3700 1 -- 172.21.15.46:0/71037268 shutdown_connections 2024-01-27T01:08:45.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.764+0000 7ff29f5d3700 1 -- 172.21.15.46:0/71037268 wait complete. 2024-01-27T01:08:45.779 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-27T01:08:45.820 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:45.820+0000 7f66656d31c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T01:08:45.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.913+0000 7f354ce3a700 1 Processor -- start 2024-01-27T01:08:45.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.913+0000 7f354ce3a700 1 -- start start 2024-01-27T01:08:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.914+0000 7f354ce3a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35480b2c60 0x7f35480b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.914+0000 7f354ce3a700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f35480b3ad0 0x7f35480c6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.914+0000 7f354ce3a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35480d1490 con 0x7f35480c6d60 2024-01-27T01:08:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.914+0000 7f354ce3a700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35480c6390 con 0x7f35480b3ad0 2024-01-27T01:08:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.914+0000 7f354ce3a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35480ccd80 con 0x7f35480b2c60 2024-01-27T01:08:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.915+0000 7f354659c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35480b2c60 0x7f35480b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.915+0000 7f3545d9b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f35480b3ad0 0x7f35480c6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.915+0000 7f3545d9b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f35480b3ad0 0x7f35480c6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:57520/0 (socket says 172.21.15.46:57520) 2024-01-27T01:08:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.915+0000 7f3546d9d700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f35480c6d60 0x7f35480cb5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:34608/0 (socket says 172.21.15.46:34608) 2024-01-27T01:08:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.915+0000 7f354659c700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35480b2c60 0x7f35480b3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:34116/0 (socket says 172.21.15.46:34116) 2024-01-27T01:08:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.915+0000 7f3545d9b700 1 -- 172.21.15.46:0/1255260382 learned_addr learned my addr 172.21.15.46:0/1255260382 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.915+0000 7f3545d9b700 1 -- 172.21.15.46:0/1255260382 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35480b2c60 msgr2=0x7f35480b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.915+0000 7f3545d9b700 1 --2- 172.21.15.46:0/1255260382 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35480b2c60 0x7f35480b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.915+0000 7f3545d9b700 1 -- 172.21.15.46:0/1255260382 >> v1:172.21.15.106:6789/0 conn(0x7f35480c6d60 legacy=0x7f35480cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.915+0000 7f3545d9b700 1 -- 172.21.15.46:0/1255260382 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35480cc7c0 con 0x7f35480b3ad0 2024-01-27T01:08:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.915+0000 7f354659c700 1 --2- 172.21.15.46:0/1255260382 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35480b2c60 0x7f35480b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.915+0000 7f3545d9b700 1 --2- 172.21.15.46:0/1255260382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35480b3ad0 0x7f35480c6820 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f353c0099b0 tx=0x7f353c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11b20b8a395f8957 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:45.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.915+0000 7f354559a700 1 -- 172.21.15.46:0/1255260382 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c009ea0 con 0x7f35480b3ad0 2024-01-27T01:08:45.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.915+0000 7f354559a700 1 -- 172.21.15.46:0/1255260382 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353c00b860 con 0x7f35480b3ad0 2024-01-27T01:08:45.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.915+0000 7f354559a700 1 -- 172.21.15.46:0/1255260382 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c00bc00 con 0x7f35480b3ad0 2024-01-27T01:08:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.916+0000 7f354ce3a700 1 -- 172.21.15.46:0/1255260382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35480b3ad0 msgr2=0x7f35480c6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.916+0000 7f354ce3a700 1 --2- 172.21.15.46:0/1255260382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35480b3ad0 0x7f35480c6820 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f353c0099b0 tx=0x7f353c009250 comp rx=0 tx=0).stop 2024-01-27T01:08:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.916+0000 7f354ce3a700 1 -- 172.21.15.46:0/1255260382 shutdown_connections 2024-01-27T01:08:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.916+0000 7f354ce3a700 1 --2- 172.21.15.46:0/1255260382 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35480b2c60 0x7f35480b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.916+0000 7f354ce3a700 1 --2- 172.21.15.46:0/1255260382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35480b3ad0 0x7f35480c6820 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.916+0000 7f354ce3a700 1 -- 172.21.15.46:0/1255260382 >> 172.21.15.46:0/1255260382 conn(0x7f354806f8c0 msgr2=0x7f35480d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.916+0000 7f354ce3a700 1 -- 172.21.15.46:0/1255260382 shutdown_connections 2024-01-27T01:08:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.916+0000 7f354ce3a700 1 -- 172.21.15.46:0/1255260382 wait complete. 2024-01-27T01:08:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.917+0000 7f354ce3a700 1 Processor -- start 2024-01-27T01:08:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.917+0000 7f354ce3a700 1 -- start start 2024-01-27T01:08:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.918+0000 7f354ce3a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35480b2c60 0x7f3548158360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.918+0000 7f354ce3a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35480b3ad0 0x7f354815cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.918+0000 7f354ce3a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35480c6d60 0x7f3548068fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.918+0000 7f354ce3a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354815eb50 con 0x7f35480b2c60 2024-01-27T01:08:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.918+0000 7f354ce3a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354815df40 con 0x7f35480b3ad0 2024-01-27T01:08:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.918+0000 7f354ce3a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f354815e220 con 0x7f35480c6d60 2024-01-27T01:08:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.918+0000 7f354659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35480b2c60 0x7f3548158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.918+0000 7f3545d9b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35480b3ad0 0x7f354815cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.918+0000 7f354659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35480b2c60 0x7f3548158360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48288/0 (socket says 172.21.15.46:48288) 2024-01-27T01:08:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.918+0000 7f354659c700 1 -- 172.21.15.46:0/1865218068 learned_addr learned my addr 172.21.15.46:0/1865218068 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.918+0000 7f3546d9d700 1 --2- 172.21.15.46:0/1865218068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35480c6d60 0x7f3548068fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.919+0000 7f3545d9b700 1 -- 172.21.15.46:0/1865218068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35480c6d60 msgr2=0x7f3548068fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.919+0000 7f3545d9b700 1 --2- 172.21.15.46:0/1865218068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35480c6d60 0x7f3548068fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.919+0000 7f3545d9b700 1 -- 172.21.15.46:0/1865218068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35480b2c60 msgr2=0x7f3548158360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.919+0000 7f3545d9b700 1 --2- 172.21.15.46:0/1865218068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35480b2c60 0x7f3548158360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.919+0000 7f3545d9b700 1 -- 172.21.15.46:0/1865218068 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3538009420 con 0x7f35480b3ad0 2024-01-27T01:08:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.919+0000 7f3545d9b700 1 --2- 172.21.15.46:0/1865218068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35480b3ad0 0x7f354815cf80 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f3548089b90 tx=0x7f353c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.919+0000 7f35367fc700 1 -- 172.21.15.46:0/1865218068 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c000d10 con 0x7f35480b3ad0 2024-01-27T01:08:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.919+0000 7f35367fc700 1 -- 172.21.15.46:0/1865218068 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353c00bbe0 con 0x7f35480b3ad0 2024-01-27T01:08:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.919+0000 7f35367fc700 1 -- 172.21.15.46:0/1865218068 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c011670 con 0x7f35480b3ad0 2024-01-27T01:08:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.919+0000 7f354ce3a700 1 -- 172.21.15.46:0/1865218068 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f353c0094f0 con 0x7f35480b3ad0 2024-01-27T01:08:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.919+0000 7f354ce3a700 1 -- 172.21.15.46:0/1865218068 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f354815e7e0 con 0x7f35480b3ad0 2024-01-27T01:08:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.920+0000 7f354ce3a700 1 -- 172.21.15.46:0/1865218068 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f354815e360 con 0x7f35480b3ad0 2024-01-27T01:08:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.921+0000 7f35367fc700 1 -- 172.21.15.46:0/1865218068 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f353c011aa0 con 0x7f35480b3ad0 2024-01-27T01:08:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.921+0000 7f35367fc700 1 --2- 172.21.15.46:0/1865218068 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3528067110 0x7f35280695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:45.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.922+0000 7f354659c700 1 -- 172.21.15.46:0/1865218068 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3528067110 msgr2=0x7f35280695c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:45.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.922+0000 7f35367fc700 1 -- 172.21.15.46:0/1865218068 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6990+0+0 (secure 0 0 0) 0x7f353c086180 con 0x7f35480b3ad0 2024-01-27T01:08:45.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.922+0000 7f354659c700 1 --2- 172.21.15.46:0/1865218068 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3528067110 0x7f35280695c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:45.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:45.926+0000 7f35367fc700 1 -- 172.21.15.46:0/1865218068 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f354815e360 con 0x7f35480b3ad0 2024-01-27T01:08:46.002 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:46.003+0000 7f4965bd41c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:08:46.003 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:46.003+0000 7f66656d31c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T01:08:46.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.123+0000 7f354659c700 1 -- 172.21.15.46:0/1865218068 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3528067110 msgr2=0x7f35280695c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:46.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.123+0000 7f354659c700 1 --2- 172.21.15.46:0/1865218068 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3528067110 0x7f35280695c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T01:08:46.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.186+0000 7f354ce3a700 1 -- 172.21.15.46:0/1865218068 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f354815e360 con 0x7f35480b3ad0 2024-01-27T01:08:46.187 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-27T01:08:46.187+0000 7f33ffe86700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-27T01:08:46.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.192+0000 7f35367fc700 1 -- 172.21.15.46:0/1865218068 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v150) v1 ==== 161+0+0 (secure 0 0 0) 0x7f354815e360 con 0x7f35480b3ad0 2024-01-27T01:08:46.193 INFO:tasks.ceph.mds.c.smithi046.stderr: -3278> 2024-01-27T01:08:23.192+0000 7fcbf9662700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-27T01:08:46.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.194+0000 7f354ce3a700 1 -- 172.21.15.46:0/1865218068 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3528067110 msgr2=0x7f35280695c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:46.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.194+0000 7f354ce3a700 1 --2- 172.21.15.46:0/1865218068 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3528067110 0x7f35280695c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.194+0000 7f354ce3a700 1 -- 172.21.15.46:0/1865218068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35480b3ad0 msgr2=0x7f354815cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.194+0000 7f354ce3a700 1 --2- 172.21.15.46:0/1865218068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35480b3ad0 0x7f354815cf80 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f3548089b90 tx=0x7f353c017000 comp rx=0 tx=0).stop 2024-01-27T01:08:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.194+0000 7f354ce3a700 1 -- 172.21.15.46:0/1865218068 shutdown_connections 2024-01-27T01:08:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.194+0000 7f354ce3a700 1 --2- 172.21.15.46:0/1865218068 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3528067110 0x7f35280695c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.194+0000 7f354ce3a700 1 --2- 172.21.15.46:0/1865218068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35480b2c60 0x7f3548158360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.194+0000 7f354ce3a700 1 --2- 172.21.15.46:0/1865218068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35480c6d60 0x7f3548068fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.194+0000 7f354ce3a700 1 --2- 172.21.15.46:0/1865218068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35480b3ad0 0x7f354815cf80 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.194+0000 7f354ce3a700 1 -- 172.21.15.46:0/1865218068 >> 172.21.15.46:0/1865218068 conn(0x7f354806f8c0 msgr2=0x7f3548159090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.194+0000 7f354ce3a700 1 -- 172.21.15.46:0/1865218068 shutdown_connections 2024-01-27T01:08:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.194+0000 7f354ce3a700 1 -- 172.21.15.46:0/1865218068 wait complete. 2024-01-27T01:08:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-27T01:08:46.209 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-27T01:08:46.220 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-01-27T01:08:46.226 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:46.227+0000 7f4965bd41c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:08:46.302 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:46.302+0000 7f4965bd41c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:08:46.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.356+0000 7f1c9da3a700 1 Processor -- start 2024-01-27T01:08:46.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.356+0000 7f1c9da3a700 1 -- start start 2024-01-27T01:08:46.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.358+0000 7f1c9da3a700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c9815a8e0 0x7f1c9815acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.358+0000 7f1c9da3a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c9806a340 0x7f1c980afad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.358+0000 7f1c9da3a700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c9806cd00 con 0x7f1c980ae800 2024-01-27T01:08:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.358+0000 7f1c9da3a700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c980add50 con 0x7f1c9815a8e0 2024-01-27T01:08:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.358+0000 7f1c9da3a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c980aeca0 con 0x7f1c9806a340 2024-01-27T01:08:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.358+0000 7f1c96ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c9815a8e0 0x7f1c9815acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.358+0000 7f1c96ffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c9815a8e0 0x7f1c9815acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48320/0 (socket says 172.21.15.46:48320) 2024-01-27T01:08:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.358+0000 7f1c96ffd700 1 -- 172.21.15.46:0/2756220051 learned_addr learned my addr 172.21.15.46:0/2756220051 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.358+0000 7f1c977fe700 1 --2- 172.21.15.46:0/2756220051 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c9806a340 0x7f1c980afad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.358+0000 7f1c96ffd700 1 -- 172.21.15.46:0/2756220051 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c9806a340 msgr2=0x7f1c980afad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.358+0000 7f1c96ffd700 1 --2- 172.21.15.46:0/2756220051 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c9806a340 0x7f1c980afad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.358+0000 7f1c96ffd700 1 -- 172.21.15.46:0/2756220051 >> v1:172.21.15.106:6789/0 conn(0x7f1c980ae800 legacy=0x7f1c9806bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.358+0000 7f1c96ffd700 1 -- 172.21.15.46:0/2756220051 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9806b6a0 con 0x7f1c9815a8e0 2024-01-27T01:08:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.358+0000 7f1c96ffd700 1 --2- 172.21.15.46:0/2756220051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c9815a8e0 0x7f1c9815acb0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f1c880099b0 tx=0x7f1c88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eaf4502cefc460f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:46.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.358+0000 7f1c95ffb700 1 -- 172.21.15.46:0/2756220051 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c88009ea0 con 0x7f1c9815a8e0 2024-01-27T01:08:46.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.359+0000 7f1c95ffb700 1 -- 172.21.15.46:0/2756220051 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c8800b860 con 0x7f1c9815a8e0 2024-01-27T01:08:46.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.359+0000 7f1c95ffb700 1 -- 172.21.15.46:0/2756220051 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8800bc00 con 0x7f1c9815a8e0 2024-01-27T01:08:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.359+0000 7f1c9da3a700 1 -- 172.21.15.46:0/2756220051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c9815a8e0 msgr2=0x7f1c9815acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.359+0000 7f1c9da3a700 1 --2- 172.21.15.46:0/2756220051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c9815a8e0 0x7f1c9815acb0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f1c880099b0 tx=0x7f1c88009250 comp rx=0 tx=0).stop 2024-01-27T01:08:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.359+0000 7f1c9da3a700 1 -- 172.21.15.46:0/2756220051 shutdown_connections 2024-01-27T01:08:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.359+0000 7f1c9da3a700 1 --2- 172.21.15.46:0/2756220051 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c9806a340 0x7f1c980afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.359+0000 7f1c9da3a700 1 --2- 172.21.15.46:0/2756220051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c9815a8e0 0x7f1c9815acb0 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.359+0000 7f1c9da3a700 1 -- 172.21.15.46:0/2756220051 >> 172.21.15.46:0/2756220051 conn(0x7f1c980ccc40 msgr2=0x7f1c980cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.359+0000 7f1c9da3a700 1 -- 172.21.15.46:0/2756220051 shutdown_connections 2024-01-27T01:08:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.359+0000 7f1c9da3a700 1 -- 172.21.15.46:0/2756220051 wait complete. 2024-01-27T01:08:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.361+0000 7f1c9da3a700 1 Processor -- start 2024-01-27T01:08:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.361+0000 7f1c9da3a700 1 -- start start 2024-01-27T01:08:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c9da3a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c9806a340 0x7f1c9814b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c9da3a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c980ae800 0x7f1c980aadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c9da3a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c9815a8e0 0x7f1c980ab2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c9da3a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c98159a10 con 0x7f1c9815a8e0 2024-01-27T01:08:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c9da3a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9814bd70 con 0x7f1c980ae800 2024-01-27T01:08:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c9da3a700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c980ab970 con 0x7f1c9806a340 2024-01-27T01:08:46.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c977fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c9815a8e0 0x7f1c980ab2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:46.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c977fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c9815a8e0 0x7f1c980ab2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48332/0 (socket says 172.21.15.46:48332) 2024-01-27T01:08:46.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c977fe700 1 -- 172.21.15.46:0/2978621376 learned_addr learned my addr 172.21.15.46:0/2978621376 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:46.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c96ffd700 1 --2- 172.21.15.46:0/2978621376 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c9806a340 0x7f1c9814b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:46.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c967fc700 1 --2- 172.21.15.46:0/2978621376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c980ae800 0x7f1c980aadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:46.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c96ffd700 1 -- 172.21.15.46:0/2978621376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c980ae800 msgr2=0x7f1c980aadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:46.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c96ffd700 1 --2- 172.21.15.46:0/2978621376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c980ae800 0x7f1c980aadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:46.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c96ffd700 1 -- 172.21.15.46:0/2978621376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c9815a8e0 msgr2=0x7f1c980ab2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:46.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c96ffd700 1 --2- 172.21.15.46:0/2978621376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c9815a8e0 0x7f1c980ab2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:46.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c96ffd700 1 -- 172.21.15.46:0/2978621376 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c880094f0 con 0x7f1c9806a340 2024-01-27T01:08:46.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c977fe700 1 --2- 172.21.15.46:0/2978621376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c9815a8e0 0x7f1c980ab2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:46.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.362+0000 7f1c96ffd700 1 --2- 172.21.15.46:0/2978621376 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c9806a340 0x7f1c9814b830 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1c980ca720 tx=0x7f1c88017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:46.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.363+0000 7f1c7effd700 1 -- 172.21.15.46:0/2978621376 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8800bbe0 con 0x7f1c9806a340 2024-01-27T01:08:46.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.363+0000 7f1c7effd700 1 -- 172.21.15.46:0/2978621376 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c8801be00 con 0x7f1c9806a340 2024-01-27T01:08:46.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.363+0000 7f1c9da3a700 1 -- 172.21.15.46:0/2978621376 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c980abc50 con 0x7f1c9806a340 2024-01-27T01:08:46.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.363+0000 7f1c7effd700 1 -- 172.21.15.46:0/2978621376 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c88011650 con 0x7f1c9806a340 2024-01-27T01:08:46.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.363+0000 7f1c9da3a700 1 -- 172.21.15.46:0/2978621376 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c980ac210 con 0x7f1c9806a340 2024-01-27T01:08:46.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.364+0000 7f1c9da3a700 1 -- 172.21.15.46:0/2978621376 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c9806d220 con 0x7f1c9806a340 2024-01-27T01:08:46.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.364+0000 7f1c7effd700 1 -- 172.21.15.46:0/2978621376 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f1c88011a80 con 0x7f1c9806a340 2024-01-27T01:08:46.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.365+0000 7f1c7effd700 1 --2- 172.21.15.46:0/2978621376 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c840670e0 0x7f1c84069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:46.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.365+0000 7f1c967fc700 1 -- 172.21.15.46:0/2978621376 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c840670e0 msgr2=0x7f1c84069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.106:6836/62434 2024-01-27T01:08:46.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.365+0000 7f1c7effd700 1 -- 172.21.15.46:0/2978621376 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 7076+0+0 (secure 0 0 0) 0x7f1c88086780 con 0x7f1c9806a340 2024-01-27T01:08:46.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.365+0000 7f1c967fc700 1 --2- 172.21.15.46:0/2978621376 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c840670e0 0x7f1c84069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T01:08:46.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.369+0000 7f1c7effd700 1 -- 172.21.15.46:0/2978621376 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1c88053210 con 0x7f1c9806a340 2024-01-27T01:08:46.461 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:46.462+0000 7f4965bd41c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:08:46.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.483+0000 7f1c7effd700 1 -- 172.21.15.46:0/2978621376 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mgrmap(e 151) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f1c8800fdc0 con 0x7f1c9806a340 2024-01-27T01:08:46.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.484+0000 7f1c7effd700 1 -- 172.21.15.46:0/2978621376 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c840670e0 msgr2=0x7f1c84069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T01:08:46.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.484+0000 7f1c7effd700 1 --2- 172.21.15.46:0/2978621376 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c840670e0 0x7f1c84069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:46.540 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:46.540+0000 7f48e1cbd700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:46.540 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:46.540+0000 7f48e1cbd700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:46.540 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:46.540+0000 7f48e1cbd700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:46.540 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:46.540+0000 7f48e1cbd700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:46.540 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:46.540+0000 7f48e1cbd700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:46.540 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:46.540+0000 7f48e4cc3700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:46.540 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:46.540+0000 7f48e4cc3700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:46.540 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:46.540+0000 7f48e4cc3700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:46.540 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:46.540+0000 7f48e4cc3700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:46.540 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-01-27T01:08:46.540+0000 7f48e4cc3700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-27T01:08:46.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:46.629+0000 7f1c9da3a700 1 -- 172.21.15.46:0/2978621376 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1c980c00b0 con 0x7f1c9806a340 2024-01-27T01:08:46.839 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:46.839+0000 7f66656d31c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T01:08:47.096 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:47.096+0000 7f66656d31c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T01:08:47.181 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:47.181+0000 7f66656d31c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T01:08:47.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.191+0000 7f1c7effd700 1 -- 172.21.15.46:0/2978621376 <== mon.2 v2:172.21.15.106:3301/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v151) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1c8805ae60 con 0x7f1c9806a340 2024-01-27T01:08:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.194+0000 7f1c9da3a700 1 -- 172.21.15.46:0/2978621376 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c9806a340 msgr2=0x7f1c9814b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.194+0000 7f1c9da3a700 1 --2- 172.21.15.46:0/2978621376 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c9806a340 0x7f1c9814b830 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1c980ca720 tx=0x7f1c88017000 comp rx=0 tx=0).stop 2024-01-27T01:08:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.194+0000 7f1c9da3a700 1 -- 172.21.15.46:0/2978621376 shutdown_connections 2024-01-27T01:08:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.194+0000 7f1c9da3a700 1 --2- 172.21.15.46:0/2978621376 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c840670e0 0x7f1c84069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.194+0000 7f1c9da3a700 1 --2- 172.21.15.46:0/2978621376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c9815a8e0 0x7f1c980ab2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.194+0000 7f1c9da3a700 1 --2- 172.21.15.46:0/2978621376 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c9806a340 0x7f1c9814b830 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.194+0000 7f1c9da3a700 1 --2- 172.21.15.46:0/2978621376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c980ae800 0x7f1c980aadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.194+0000 7f1c9da3a700 1 -- 172.21.15.46:0/2978621376 >> 172.21.15.46:0/2978621376 conn(0x7f1c980ccc40 msgr2=0x7f1c9806e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:47.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.194+0000 7f1c9da3a700 1 -- 172.21.15.46:0/2978621376 shutdown_connections 2024-01-27T01:08:47.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.194+0000 7f1c9da3a700 1 -- 172.21.15.46:0/2978621376 wait complete. 2024-01-27T01:08:47.208 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-27T01:08:47.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.346+0000 7fb6c685d700 1 Processor -- start 2024-01-27T01:08:47.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.346+0000 7fb6c685d700 1 -- start start 2024-01-27T01:08:47.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.347+0000 7fb6c685d700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb6c00cf3c0 0x7fb6c00cb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:47.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.347+0000 7fb6c685d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6c00cbb70 0x7fb6c00c6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:47.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.347+0000 7fb6c685d700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c00c5580 con 0x7fb6c015d110 2024-01-27T01:08:47.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.347+0000 7fb6c685d700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6c00c63e0 con 0x7fb6c00cf3c0 2024-01-27T01:08:47.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.347+0000 7fb6c685d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c00c57d0 con 0x7fb6c00cbb70 2024-01-27T01:08:47.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.347+0000 7fb6bf7fe700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb6c00cf3c0 0x7fb6c00cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:47.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.347+0000 7fb6bffff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb6c015d110 0x7fb6c00c7890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59350/0 (socket says 172.21.15.46:59350) 2024-01-27T01:08:47.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.347+0000 7fb6bffff700 1 -- 172.21.15.46:0/3045782626 learned_addr learned my addr 172.21.15.46:0/3045782626 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:47.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.347+0000 7fb6c4dfa700 1 --2- 172.21.15.46:0/3045782626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6c00cbb70 0x7fb6c00c6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:47.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.348+0000 7fb6bf7fe700 1 -- 172.21.15.46:0/3045782626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6c00cbb70 msgr2=0x7fb6c00c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:47.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.348+0000 7fb6bf7fe700 1 --2- 172.21.15.46:0/3045782626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6c00cbb70 0x7fb6c00c6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:47.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.348+0000 7fb6bf7fe700 1 -- 172.21.15.46:0/3045782626 >> v1:172.21.15.46:6789/0 conn(0x7fb6c015d110 legacy=0x7fb6c00c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:47.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.348+0000 7fb6bf7fe700 1 -- 172.21.15.46:0/3045782626 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c00c5ab0 con 0x7fb6c00cf3c0 2024-01-27T01:08:47.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.348+0000 7fb6beffd700 1 -- 172.21.15.46:0/3045782626 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696748302 0 0) 0x7fb6c00c5580 con 0x7fb6c015d110 2024-01-27T01:08:47.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.348+0000 7fb6bf7fe700 1 --2- 172.21.15.46:0/3045782626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c00cf3c0 0x7fb6c00cb630 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40098e0 tx=0x7fb6b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710281fcf340080 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:47.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.348+0000 7fb6beffd700 1 -- 172.21.15.46:0/3045782626 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b400b640 con 0x7fb6c00cf3c0 2024-01-27T01:08:47.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.348+0000 7fb6beffd700 1 -- 172.21.15.46:0/3045782626 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b400b7a0 con 0x7fb6c00cf3c0 2024-01-27T01:08:47.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.348+0000 7fb6beffd700 1 -- 172.21.15.46:0/3045782626 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b400bb40 con 0x7fb6c00cf3c0 2024-01-27T01:08:47.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.349+0000 7fb6c685d700 1 -- 172.21.15.46:0/3045782626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c00cf3c0 msgr2=0x7fb6c00cb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:47.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.349+0000 7fb6c685d700 1 --2- 172.21.15.46:0/3045782626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c00cf3c0 0x7fb6c00cb630 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40098e0 tx=0x7fb6b40091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:47.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.349+0000 7fb6c685d700 1 -- 172.21.15.46:0/3045782626 shutdown_connections 2024-01-27T01:08:47.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.349+0000 7fb6c685d700 1 --2- 172.21.15.46:0/3045782626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6c00cbb70 0x7fb6c00c6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:47.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.349+0000 7fb6c685d700 1 --2- 172.21.15.46:0/3045782626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c00cf3c0 0x7fb6c00cb630 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:47.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.349+0000 7fb6c685d700 1 -- 172.21.15.46:0/3045782626 >> 172.21.15.46:0/3045782626 conn(0x7fb6c006f8c0 msgr2=0x7fb6c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:47.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.349+0000 7fb6c685d700 1 -- 172.21.15.46:0/3045782626 shutdown_connections 2024-01-27T01:08:47.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.349+0000 7fb6c685d700 1 -- 172.21.15.46:0/3045782626 wait complete. 2024-01-27T01:08:47.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.350+0000 7fb6c685d700 1 Processor -- start 2024-01-27T01:08:47.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.350+0000 7fb6c685d700 1 -- start start 2024-01-27T01:08:47.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.351+0000 7fb6c685d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6c00cbb70 0x7fb6c015ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:47.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.351+0000 7fb6c685d700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6c00cf3c0 0x7fb6c00b5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:47.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.351+0000 7fb6c685d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c015d110 0x7fb6c015fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:47.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.351+0000 7fb6c685d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c00b5940 con 0x7fb6c015d110 2024-01-27T01:08:47.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.351+0000 7fb6c685d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c00b4d70 con 0x7fb6c00cbb70 2024-01-27T01:08:47.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.351+0000 7fb6c685d700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c00b5010 con 0x7fb6c00cf3c0 2024-01-27T01:08:47.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.351+0000 7fb6c4dfa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c015d110 0x7fb6c015fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:47.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.351+0000 7fb6c4dfa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c015d110 0x7fb6c015fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48348/0 (socket says 172.21.15.46:48348) 2024-01-27T01:08:47.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.351+0000 7fb6c4dfa700 1 -- 172.21.15.46:0/2026266238 learned_addr learned my addr 172.21.15.46:0/2026266238 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:47.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.351+0000 7fb6bffff700 1 --2- 172.21.15.46:0/2026266238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6c00cbb70 0x7fb6c015ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:47.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.351+0000 7fb6bf7fe700 1 --2- 172.21.15.46:0/2026266238 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6c00cf3c0 0x7fb6c00b5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:47.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.352+0000 7fb6c4dfa700 1 -- 172.21.15.46:0/2026266238 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6c00cf3c0 msgr2=0x7fb6c00b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:47.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.352+0000 7fb6c4dfa700 1 --2- 172.21.15.46:0/2026266238 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6c00cf3c0 0x7fb6c00b5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:47.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.352+0000 7fb6c4dfa700 1 -- 172.21.15.46:0/2026266238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6c00cbb70 msgr2=0x7fb6c015ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:47.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.352+0000 7fb6c4dfa700 1 --2- 172.21.15.46:0/2026266238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6c00cbb70 0x7fb6c015ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:47.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.352+0000 7fb6c4dfa700 1 -- 172.21.15.46:0/2026266238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b4009420 con 0x7fb6c015d110 2024-01-27T01:08:47.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.352+0000 7fb6c4dfa700 1 --2- 172.21.15.46:0/2026266238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c015d110 0x7fb6c015fa10 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009df0 tx=0x7fb6b800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:47.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.352+0000 7fb6a7fff700 1 -- 172.21.15.46:0/2026266238 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b800cb50 con 0x7fb6c015d110 2024-01-27T01:08:47.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.352+0000 7fb6a7fff700 1 -- 172.21.15.46:0/2026266238 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b800f040 con 0x7fb6c015d110 2024-01-27T01:08:47.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.352+0000 7fb6c685d700 1 -- 172.21.15.46:0/2026266238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c00b52f0 con 0x7fb6c015d110 2024-01-27T01:08:47.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.352+0000 7fb6a7fff700 1 -- 172.21.15.46:0/2026266238 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8014480 con 0x7fb6c015d110 2024-01-27T01:08:47.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.352+0000 7fb6c685d700 1 -- 172.21.15.46:0/2026266238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c00b6310 con 0x7fb6c015d110 2024-01-27T01:08:47.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.353+0000 7fb6c685d700 1 -- 172.21.15.46:0/2026266238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c00b5ab0 con 0x7fb6c015d110 2024-01-27T01:08:47.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.354+0000 7fb6a7fff700 1 -- 172.21.15.46:0/2026266238 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 151) v1 ==== 84453+0+0 (secure 0 0 0) 0x7fb6b8014830 con 0x7fb6c015d110 2024-01-27T01:08:47.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.354+0000 7fb6a7fff700 1 -- 172.21.15.46:0/2026266238 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 7377+0+0 (secure 0 0 0) 0x7fb6b8085060 con 0x7fb6c015d110 2024-01-27T01:08:47.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.359+0000 7fb6a7fff700 1 -- 172.21.15.46:0/2026266238 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb6b8052000 con 0x7fb6c015d110 2024-01-27T01:08:47.375 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-01-27T01:08:47.375+0000 7f66656d31c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T01:08:47.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.486+0000 7fb6a7fff700 1 -- 172.21.15.46:0/2026266238 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 152) v1 ==== 84580+0+0 (secure 0 0 0) 0x7fb6b8051ab0 con 0x7fb6c015d110 2024-01-27T01:08:47.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.486+0000 7fb6a7fff700 1 --2- 172.21.15.46:0/2026266238 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6ac06d330 0x7fb6ac06f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:47.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.487+0000 7fb6bffff700 1 --2- 172.21.15.46:0/2026266238 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6ac06d330 0x7fb6ac06f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:47.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.489+0000 7fb6bffff700 1 --2- 172.21.15.46:0/2026266238 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6ac06d330 0x7fb6ac06f710 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000c8f0 tx=0x7fb6b0009040 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:47.579+0000 7fb6c685d700 1 -- 172.21.15.46:0/2026266238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb6c004c710 con 0x7fb6c015d110 2024-01-27T01:08:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.492+0000 7fb6a7fff700 1 -- 172.21.15.46:0/2026266238 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v268) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb6b806fa60 con 0x7fb6c015d110 2024-01-27T01:08:48.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.498+0000 7fb6c685d700 1 -- 172.21.15.46:0/2026266238 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6ac06d330 msgr2=0x7fb6ac06f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:48.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.498+0000 7fb6c685d700 1 --2- 172.21.15.46:0/2026266238 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6ac06d330 0x7fb6ac06f710 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000c8f0 tx=0x7fb6b0009040 comp rx=0 tx=0).stop 2024-01-27T01:08:48.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.498+0000 7fb6c685d700 1 -- 172.21.15.46:0/2026266238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c015d110 msgr2=0x7fb6c015fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:48.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.498+0000 7fb6c685d700 1 --2- 172.21.15.46:0/2026266238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c015d110 0x7fb6c015fa10 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009df0 tx=0x7fb6b800b040 comp rx=0 tx=0).stop 2024-01-27T01:08:48.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.499+0000 7fb6c685d700 1 -- 172.21.15.46:0/2026266238 shutdown_connections 2024-01-27T01:08:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.499+0000 7fb6c685d700 1 --2- 172.21.15.46:0/2026266238 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6ac06d330 0x7fb6ac06f710 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.499+0000 7fb6c685d700 1 --2- 172.21.15.46:0/2026266238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c015d110 0x7fb6c015fa10 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.499+0000 7fb6c685d700 1 --2- 172.21.15.46:0/2026266238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6c00cbb70 0x7fb6c015ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.499+0000 7fb6c685d700 1 --2- 172.21.15.46:0/2026266238 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6c00cf3c0 0x7fb6c00b5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.499+0000 7fb6c685d700 1 -- 172.21.15.46:0/2026266238 >> 172.21.15.46:0/2026266238 conn(0x7fb6c006f8c0 msgr2=0x7fb6c00cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.499+0000 7fb6c685d700 1 -- 172.21.15.46:0/2026266238 shutdown_connections 2024-01-27T01:08:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.499+0000 7fb6c685d700 1 -- 172.21.15.46:0/2026266238 wait complete. 2024-01-27T01:08:48.501 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-01-27T01:08:48.516 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-27T01:08:48.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.659+0000 7fd299f36700 1 Processor -- start 2024-01-27T01:08:48.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.659+0000 7fd299f36700 1 -- start start 2024-01-27T01:08:48.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.660+0000 7fd299f36700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd2940ce120 0x7fd2940afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:48.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.660+0000 7fd299f36700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2940cb0a0 con 0x7fd2940af6c0 2024-01-27T01:08:48.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.660+0000 7fd299f36700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2940cb5d0 con 0x7fd2940680c0 2024-01-27T01:08:48.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.660+0000 7fd299f36700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2940c97f0 con 0x7fd2940ce120 2024-01-27T01:08:48.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.660+0000 7fd28affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd2940ce120 0x7fd2940afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:48.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.661+0000 7fd2937fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd2940af6c0 0x7fd2940679b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59352/0 (socket says 172.21.15.46:59352) 2024-01-27T01:08:48.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.661+0000 7fd2937fe700 1 -- 172.21.15.46:0/2385145312 learned_addr learned my addr 172.21.15.46:0/2385145312 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:48.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.661+0000 7fd28affd700 1 -- 172.21.15.46:0/2385145312 >> v1:172.21.15.106:6789/0 conn(0x7fd2940680c0 legacy=0x7fd2940c2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:48.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.661+0000 7fd28affd700 1 -- 172.21.15.46:0/2385145312 >> v1:172.21.15.46:6789/0 conn(0x7fd2940af6c0 legacy=0x7fd2940679b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:48.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.661+0000 7fd292ffd700 1 -- 172.21.15.46:0/2385145312 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783238767 0 0) 0x7fd2940cb0a0 con 0x7fd2940af6c0 2024-01-27T01:08:48.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.661+0000 7fd28affd700 1 -- 172.21.15.46:0/2385145312 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2940cd310 con 0x7fd2940ce120 2024-01-27T01:08:48.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.661+0000 7fd28affd700 1 --2- 172.21.15.46:0/2385145312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2940ce120 0x7fd2940afa90 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fd27c0098e0 tx=0x7fd27c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea239e9568761a05 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:48.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.661+0000 7fd292ffd700 1 -- 172.21.15.46:0/2385145312 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c009e80 con 0x7fd2940ce120 2024-01-27T01:08:48.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.661+0000 7fd292ffd700 1 -- 172.21.15.46:0/2385145312 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27c00b860 con 0x7fd2940ce120 2024-01-27T01:08:48.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.661+0000 7fd292ffd700 1 -- 172.21.15.46:0/2385145312 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c00bbe0 con 0x7fd2940ce120 2024-01-27T01:08:48.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.662+0000 7fd299f36700 1 -- 172.21.15.46:0/2385145312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2940ce120 msgr2=0x7fd2940afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:48.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.662+0000 7fd299f36700 1 --2- 172.21.15.46:0/2385145312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2940ce120 0x7fd2940afa90 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fd27c0098e0 tx=0x7fd27c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:48.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.662+0000 7fd299f36700 1 -- 172.21.15.46:0/2385145312 shutdown_connections 2024-01-27T01:08:48.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.662+0000 7fd299f36700 1 --2- 172.21.15.46:0/2385145312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2940ce120 0x7fd2940afa90 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:48.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.662+0000 7fd299f36700 1 -- 172.21.15.46:0/2385145312 >> 172.21.15.46:0/2385145312 conn(0x7fd2940692a0 msgr2=0x7fd2940cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:48.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.662+0000 7fd299f36700 1 -- 172.21.15.46:0/2385145312 shutdown_connections 2024-01-27T01:08:48.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.662+0000 7fd299f36700 1 -- 172.21.15.46:0/2385145312 wait complete. 2024-01-27T01:08:48.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.663+0000 7fd299f36700 1 Processor -- start 2024-01-27T01:08:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.663+0000 7fd299f36700 1 -- start start 2024-01-27T01:08:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.664+0000 7fd299f36700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2940680c0 0x7fd2940aaff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.664+0000 7fd299f36700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2940af6c0 0x7fd2941567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.664+0000 7fd299f36700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2940ce120 0x7fd2940ae320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.664+0000 7fd299f36700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2940ade00 con 0x7fd2940ce120 2024-01-27T01:08:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.664+0000 7fd299f36700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2940aab60 con 0x7fd2940af6c0 2024-01-27T01:08:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.664+0000 7fd299f36700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd294156d20 con 0x7fd2940680c0 2024-01-27T01:08:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.664+0000 7fd293fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2940ce120 0x7fd2940ae320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.664+0000 7fd293fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2940ce120 0x7fd2940ae320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48360/0 (socket says 172.21.15.46:48360) 2024-01-27T01:08:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.664+0000 7fd293fff700 1 -- 172.21.15.46:0/2957640208 learned_addr learned my addr 172.21.15.46:0/2957640208 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.664+0000 7fd28affd700 1 --2- 172.21.15.46:0/2957640208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2940af6c0 0x7fd2941567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:48.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.664+0000 7fd2937fe700 1 --2- 172.21.15.46:0/2957640208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2940680c0 0x7fd2940aaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:48.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.665+0000 7fd293fff700 1 -- 172.21.15.46:0/2957640208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2940680c0 msgr2=0x7fd2940aaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:48.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.665+0000 7fd293fff700 1 --2- 172.21.15.46:0/2957640208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2940680c0 0x7fd2940aaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:48.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.665+0000 7fd293fff700 1 -- 172.21.15.46:0/2957640208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2940af6c0 msgr2=0x7fd2941567e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:48.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.665+0000 7fd293fff700 1 --2- 172.21.15.46:0/2957640208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2940af6c0 0x7fd2941567e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:48.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.665+0000 7fd293fff700 1 -- 172.21.15.46:0/2957640208 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27c009420 con 0x7fd2940ce120 2024-01-27T01:08:48.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.665+0000 7fd293fff700 1 --2- 172.21.15.46:0/2957640208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2940ce120 0x7fd2940ae320 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00c870 tx=0x7fd28c001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:48.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.665+0000 7fd28bfff700 1 -- 172.21.15.46:0/2957640208 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c012630 con 0x7fd2940ce120 2024-01-27T01:08:48.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.665+0000 7fd28bfff700 1 -- 172.21.15.46:0/2957640208 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c0127b0 con 0x7fd2940ce120 2024-01-27T01:08:48.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.665+0000 7fd299f36700 1 -- 172.21.15.46:0/2957640208 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2940aa2a0 con 0x7fd2940ce120 2024-01-27T01:08:48.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.665+0000 7fd28bfff700 1 -- 172.21.15.46:0/2957640208 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c012b70 con 0x7fd2940ce120 2024-01-27T01:08:48.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.665+0000 7fd299f36700 1 -- 172.21.15.46:0/2957640208 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29409b020 con 0x7fd2940ce120 2024-01-27T01:08:48.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.666+0000 7fd299f36700 1 -- 172.21.15.46:0/2957640208 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29404c710 con 0x7fd2940ce120 2024-01-27T01:08:48.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.667+0000 7fd28bfff700 1 -- 172.21.15.46:0/2957640208 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 152) v1 ==== 84580+0+0 (secure 0 0 0) 0x7fd28c003030 con 0x7fd2940ce120 2024-01-27T01:08:48.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.667+0000 7fd28bfff700 1 --2- 172.21.15.46:0/2957640208 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd274066d80 0x7fd274069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:48.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.667+0000 7fd28bfff700 1 -- 172.21.15.46:0/2957640208 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fd28c084460 con 0x7fd2940ce120 2024-01-27T01:08:48.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.667+0000 7fd2937fe700 1 --2- 172.21.15.46:0/2957640208 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd274066d80 0x7fd274069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:48.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.668+0000 7fd2937fe700 1 --2- 172.21.15.46:0/2957640208 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd274066d80 0x7fd274069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd2940c3f90 tx=0x7fd284009040 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.672+0000 7fd28bfff700 1 -- 172.21.15.46:0/2957640208 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd28c0514b0 con 0x7fd2940ce120 2024-01-27T01:08:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:48.876+0000 7fd299f36700 1 -- 172.21.15.46:0/2957640208 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd2940cb380 con 0x7fd2940ce120 2024-01-27T01:08:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.498+0000 7fd28bfff700 1 -- 172.21.15.46:0/2957640208 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 153) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fd28c084170 con 0x7fd2940ce120 2024-01-27T01:08:49.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.499+0000 7fd28bfff700 1 -- 172.21.15.46:0/2957640208 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v269) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd28c054f40 con 0x7fd2940ce120 2024-01-27T01:08:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.506+0000 7fd299f36700 1 -- 172.21.15.46:0/2957640208 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd274066d80 msgr2=0x7fd274069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.506+0000 7fd299f36700 1 --2- 172.21.15.46:0/2957640208 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd274066d80 0x7fd274069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd2940c3f90 tx=0x7fd284009040 comp rx=0 tx=0).stop 2024-01-27T01:08:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.506+0000 7fd299f36700 1 -- 172.21.15.46:0/2957640208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2940ce120 msgr2=0x7fd2940ae320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:49.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.506+0000 7fd299f36700 1 --2- 172.21.15.46:0/2957640208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2940ce120 0x7fd2940ae320 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00c870 tx=0x7fd28c001670 comp rx=0 tx=0).stop 2024-01-27T01:08:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.508+0000 7fd299f36700 1 -- 172.21.15.46:0/2957640208 shutdown_connections 2024-01-27T01:08:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.508+0000 7fd299f36700 1 --2- 172.21.15.46:0/2957640208 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd274066d80 0x7fd274069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.508+0000 7fd299f36700 1 --2- 172.21.15.46:0/2957640208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2940ce120 0x7fd2940ae320 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.508+0000 7fd299f36700 1 --2- 172.21.15.46:0/2957640208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2940680c0 0x7fd2940aaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.508+0000 7fd299f36700 1 --2- 172.21.15.46:0/2957640208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2940af6c0 0x7fd2941567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.508+0000 7fd299f36700 1 -- 172.21.15.46:0/2957640208 >> 172.21.15.46:0/2957640208 conn(0x7fd2940692a0 msgr2=0x7fd294159070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:49.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.510+0000 7fd299f36700 1 -- 172.21.15.46:0/2957640208 shutdown_connections 2024-01-27T01:08:49.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.510+0000 7fd299f36700 1 -- 172.21.15.46:0/2957640208 wait complete. 2024-01-27T01:08:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-01-27T01:08:49.529 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-27T01:08:49.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.669+0000 7f9d843f7700 1 Processor -- start 2024-01-27T01:08:49.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.669+0000 7f9d843f7700 1 -- start start 2024-01-27T01:08:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.670+0000 7f9d843f7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d7c0af870 0x7f9d7c0679b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.670+0000 7f9d843f7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d7c0afc40 0x7f9d7c067ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.670+0000 7f9d843f7700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9d7c0cb0a0 0x7f9d7c0c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.670+0000 7f9d843f7700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d7c0ce420 con 0x7f9d7c0af870 2024-01-27T01:08:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.670+0000 7f9d843f7700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d7c0c94e0 con 0x7f9d7c0cb0a0 2024-01-27T01:08:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.670+0000 7f9d843f7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c0c2270 con 0x7f9d7c0afc40 2024-01-27T01:08:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.670+0000 7f9d82193700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d7c0af870 0x7f9d7c0679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.670+0000 7f9d82193700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d7c0af870 0x7f9d7c0679b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48374/0 (socket says 172.21.15.46:48374) 2024-01-27T01:08:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.670+0000 7f9d82193700 1 -- 172.21.15.46:0/3921524141 learned_addr learned my addr 172.21.15.46:0/3921524141 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.670+0000 7f9d82994700 1 --2- 172.21.15.46:0/3921524141 >> v2:172.21.15.106:3300/0 conn(0x7f9d7c0cb0a0 0x7f9d7c0c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.670+0000 7f9d82193700 1 -- 172.21.15.46:0/3921524141 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d7c0afc40 msgr2=0x7f9d7c067ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.670+0000 7f9d82193700 1 --2- 172.21.15.46:0/3921524141 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d7c0afc40 0x7f9d7c067ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.670+0000 7f9d82193700 1 -- 172.21.15.46:0/3921524141 >> v2:172.21.15.106:3300/0 conn(0x7f9d7c0cb0a0 msgr2=0x7f9d7c0c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:49.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.670+0000 7f9d82193700 1 --2- 172.21.15.46:0/3921524141 >> v2:172.21.15.106:3300/0 conn(0x7f9d7c0cb0a0 0x7f9d7c0c7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:49.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.670+0000 7f9d82193700 1 -- 172.21.15.46:0/3921524141 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c0c9780 con 0x7f9d7c0af870 2024-01-27T01:08:49.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.671+0000 7f9d82193700 1 --2- 172.21.15.46:0/3921524141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d7c0af870 0x7f9d7c0679b0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c06eb00 tx=0x7f9d78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c71a49ba0156da7c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:49.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.671+0000 7f9d81191700 1 -- 172.21.15.46:0/3921524141 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d78009e70 con 0x7f9d7c0af870 2024-01-27T01:08:49.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.671+0000 7f9d81191700 1 -- 172.21.15.46:0/3921524141 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7800b860 con 0x7f9d7c0af870 2024-01-27T01:08:49.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.671+0000 7f9d81191700 1 -- 172.21.15.46:0/3921524141 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7800bc00 con 0x7f9d7c0af870 2024-01-27T01:08:49.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.671+0000 7f9d843f7700 1 -- 172.21.15.46:0/3921524141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d7c0af870 msgr2=0x7f9d7c0679b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:49.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.671+0000 7f9d843f7700 1 --2- 172.21.15.46:0/3921524141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d7c0af870 0x7f9d7c0679b0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c06eb00 tx=0x7f9d78009250 comp rx=0 tx=0).stop 2024-01-27T01:08:49.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.672+0000 7f9d843f7700 1 -- 172.21.15.46:0/3921524141 shutdown_connections 2024-01-27T01:08:49.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.672+0000 7f9d843f7700 1 --2- 172.21.15.46:0/3921524141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d7c0af870 0x7f9d7c0679b0 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:49.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.672+0000 7f9d843f7700 1 --2- 172.21.15.46:0/3921524141 >> v2:172.21.15.106:3300/0 conn(0x7f9d7c0cb0a0 0x7f9d7c0c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:49.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.672+0000 7f9d843f7700 1 --2- 172.21.15.46:0/3921524141 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d7c0afc40 0x7f9d7c067ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:49.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.672+0000 7f9d843f7700 1 -- 172.21.15.46:0/3921524141 >> 172.21.15.46:0/3921524141 conn(0x7f9d7c0692a0 msgr2=0x7f9d7c0cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:49.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.672+0000 7f9d843f7700 1 -- 172.21.15.46:0/3921524141 shutdown_connections 2024-01-27T01:08:49.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.672+0000 7f9d843f7700 1 -- 172.21.15.46:0/3921524141 wait complete. 2024-01-27T01:08:49.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.673+0000 7f9d843f7700 1 Processor -- start 2024-01-27T01:08:49.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.673+0000 7f9d843f7700 1 -- start start 2024-01-27T01:08:49.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.674+0000 7f9d843f7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d7c0af870 0x7f9d7c157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:49.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.674+0000 7f9d843f7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d7c0afc40 0x7f9d7c15c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:49.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.674+0000 7f9d843f7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d7c0cb0a0 0x7f9d7c15ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:49.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.674+0000 7f9d843f7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c0c5f70 con 0x7f9d7c0afc40 2024-01-27T01:08:49.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.674+0000 7f9d843f7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c158520 con 0x7f9d7c0af870 2024-01-27T01:08:49.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.674+0000 7f9d843f7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c15b5a0 con 0x7f9d7c0cb0a0 2024-01-27T01:08:49.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.674+0000 7f9d81992700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d7c0afc40 0x7f9d7c15c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:49.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.674+0000 7f9d81992700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d7c0afc40 0x7f9d7c15c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48376/0 (socket says 172.21.15.46:48376) 2024-01-27T01:08:49.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.674+0000 7f9d81992700 1 -- 172.21.15.46:0/3697593357 learned_addr learned my addr 172.21.15.46:0/3697593357 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:49.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.674+0000 7f9d82193700 1 --2- 172.21.15.46:0/3697593357 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d7c0af870 0x7f9d7c157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:49.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.674+0000 7f9d82994700 1 --2- 172.21.15.46:0/3697593357 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d7c0cb0a0 0x7f9d7c15ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:49.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.674+0000 7f9d81992700 1 -- 172.21.15.46:0/3697593357 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d7c0cb0a0 msgr2=0x7f9d7c15ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:49.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.674+0000 7f9d81992700 1 --2- 172.21.15.46:0/3697593357 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d7c0cb0a0 0x7f9d7c15ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:49.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.674+0000 7f9d81992700 1 -- 172.21.15.46:0/3697593357 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d7c0af870 msgr2=0x7f9d7c157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:49.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.674+0000 7f9d81992700 1 --2- 172.21.15.46:0/3697593357 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d7c0af870 0x7f9d7c157fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:49.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.674+0000 7f9d81992700 1 -- 172.21.15.46:0/3697593357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d780094f0 con 0x7f9d7c0afc40 2024-01-27T01:08:49.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.674+0000 7f9d81992700 1 --2- 172.21.15.46:0/3697593357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d7c0afc40 0x7f9d7c15c360 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00b7e0 tx=0x7f9d6c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:49.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.675+0000 7f9d727fc700 1 -- 172.21.15.46:0/3697593357 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c00f510 con 0x7f9d7c0afc40 2024-01-27T01:08:49.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.675+0000 7f9d727fc700 1 -- 172.21.15.46:0/3697593357 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6c00f670 con 0x7f9d7c0afc40 2024-01-27T01:08:49.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.675+0000 7f9d843f7700 1 -- 172.21.15.46:0/3697593357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d7c15b880 con 0x7f9d7c0afc40 2024-01-27T01:08:49.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.675+0000 7f9d727fc700 1 -- 172.21.15.46:0/3697593357 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c00fa30 con 0x7f9d7c0afc40 2024-01-27T01:08:49.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.675+0000 7f9d843f7700 1 -- 172.21.15.46:0/3697593357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7c15be40 con 0x7f9d7c0afc40 2024-01-27T01:08:49.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.676+0000 7f9d843f7700 1 -- 172.21.15.46:0/3697593357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d7c15c060 con 0x7f9d7c0afc40 2024-01-27T01:08:49.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.676+0000 7f9d727fc700 1 -- 172.21.15.46:0/3697593357 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 153) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f9d6c01a400 con 0x7f9d7c0afc40 2024-01-27T01:08:49.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.677+0000 7f9d727fc700 1 --2- 172.21.15.46:0/3697593357 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9d60066ea0 0x7f9d60069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:49.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.677+0000 7f9d727fc700 1 -- 172.21.15.46:0/3697593357 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f9d6c084220 con 0x7f9d7c0afc40 2024-01-27T01:08:49.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.677+0000 7f9d82193700 1 --2- 172.21.15.46:0/3697593357 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9d60066ea0 0x7f9d60069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:49.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.677+0000 7f9d82193700 1 --2- 172.21.15.46:0/3697593357 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9d60066ea0 0x7f9d60069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c06eb00 tx=0x7f9d78017000 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:49.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.681+0000 7f9d727fc700 1 -- 172.21.15.46:0/3697593357 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9d6c051220 con 0x7f9d7c0afc40 2024-01-27T01:08:49.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.889+0000 7f9d843f7700 1 -- 172.21.15.46:0/3697593357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9d7c06ced0 con 0x7f9d7c0afc40 2024-01-27T01:08:49.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.893+0000 7f9d727fc700 1 -- 172.21.15.46:0/3697593357 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v91) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9d6c054840 con 0x7f9d7c0afc40 2024-01-27T01:08:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.895+0000 7f9d843f7700 1 -- 172.21.15.46:0/3697593357 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9d60066ea0 msgr2=0x7f9d60069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.895+0000 7f9d843f7700 1 --2- 172.21.15.46:0/3697593357 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9d60066ea0 0x7f9d60069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c06eb00 tx=0x7f9d78017000 comp rx=0 tx=0).stop 2024-01-27T01:08:49.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.895+0000 7f9d843f7700 1 -- 172.21.15.46:0/3697593357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d7c0afc40 msgr2=0x7f9d7c15c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:49.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.895+0000 7f9d843f7700 1 --2- 172.21.15.46:0/3697593357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d7c0afc40 0x7f9d7c15c360 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00b7e0 tx=0x7f9d6c009220 comp rx=0 tx=0).stop 2024-01-27T01:08:49.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.895+0000 7f9d843f7700 1 -- 172.21.15.46:0/3697593357 shutdown_connections 2024-01-27T01:08:49.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.895+0000 7f9d843f7700 1 --2- 172.21.15.46:0/3697593357 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9d60066ea0 0x7f9d60069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:49.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.895+0000 7f9d843f7700 1 --2- 172.21.15.46:0/3697593357 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d7c0cb0a0 0x7f9d7c15ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:49.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.895+0000 7f9d843f7700 1 --2- 172.21.15.46:0/3697593357 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d7c0af870 0x7f9d7c157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:49.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.895+0000 7f9d843f7700 1 --2- 172.21.15.46:0/3697593357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d7c0afc40 0x7f9d7c15c360 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:49.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.895+0000 7f9d843f7700 1 -- 172.21.15.46:0/3697593357 >> 172.21.15.46:0/3697593357 conn(0x7f9d7c0692a0 msgr2=0x7f9d7c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:49.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.895+0000 7f9d843f7700 1 -- 172.21.15.46:0/3697593357 shutdown_connections 2024-01-27T01:08:49.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:49.895+0000 7f9d843f7700 1 -- 172.21.15.46:0/3697593357 wait complete. 2024-01-27T01:08:49.909 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-27T01:08:50.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.043+0000 7fb49ce0b700 1 Processor -- start 2024-01-27T01:08:50.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.043+0000 7fb49ce0b700 1 -- start start 2024-01-27T01:08:50.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.044+0000 7fb49ce0b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb4980cb6e0 0x7fb4980cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:50.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.044+0000 7fb49ce0b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4980c6500 con 0x7fb4980cf3e0 2024-01-27T01:08:50.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.044+0000 7fb49ce0b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4980d1680 con 0x7fb498159f70 2024-01-27T01:08:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.044+0000 7fb49ce0b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4980ad9a0 con 0x7fb4980cb6e0 2024-01-27T01:08:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.045+0000 7fb496d9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb4980cf3e0 0x7fb4980c6990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59366/0 (socket says 172.21.15.46:59366) 2024-01-27T01:08:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.045+0000 7fb496d9d700 1 -- 172.21.15.46:0/2022917061 learned_addr learned my addr 172.21.15.46:0/2022917061 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.045+0000 7fb49659c700 1 --2- 172.21.15.46:0/2022917061 >> v2:172.21.15.106:3300/0 conn(0x7fb4980cb6e0 0x7fb4980cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.045+0000 7fb49559a700 1 -- 172.21.15.46:0/2022917061 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051536732 0 0) 0x7fb4980c6500 con 0x7fb4980cf3e0 2024-01-27T01:08:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.045+0000 7fb49559a700 1 -- 172.21.15.46:0/2022917061 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb480003570 con 0x7fb4980cf3e0 2024-01-27T01:08:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.045+0000 7fb49659c700 1 -- 172.21.15.46:0/2022917061 >> v1:172.21.15.106:6789/0 conn(0x7fb498159f70 legacy=0x7fb4980ad290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.045+0000 7fb49659c700 1 -- 172.21.15.46:0/2022917061 >> v1:172.21.15.46:6789/0 conn(0x7fb4980cf3e0 legacy=0x7fb4980c6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.045+0000 7fb49659c700 1 -- 172.21.15.46:0/2022917061 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4980cbff0 con 0x7fb4980cb6e0 2024-01-27T01:08:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.045+0000 7fb49659c700 1 --2- 172.21.15.46:0/2022917061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4980cb6e0 0x7fb4980cbab0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fb4880098e0 tx=0x7fb4880091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f43677d17712a3b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.045+0000 7fb49559a700 1 -- 172.21.15.46:0/2022917061 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48800b640 con 0x7fb4980cb6e0 2024-01-27T01:08:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.046+0000 7fb49559a700 1 -- 172.21.15.46:0/2022917061 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48800b7a0 con 0x7fb4980cb6e0 2024-01-27T01:08:50.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.046+0000 7fb49559a700 1 -- 172.21.15.46:0/2022917061 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48800bb40 con 0x7fb4980cb6e0 2024-01-27T01:08:50.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.046+0000 7fb49ce0b700 1 -- 172.21.15.46:0/2022917061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4980cb6e0 msgr2=0x7fb4980cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:50.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.046+0000 7fb49ce0b700 1 --2- 172.21.15.46:0/2022917061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4980cb6e0 0x7fb4980cbab0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fb4880098e0 tx=0x7fb4880091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.046+0000 7fb49ce0b700 1 -- 172.21.15.46:0/2022917061 shutdown_connections 2024-01-27T01:08:50.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.046+0000 7fb49ce0b700 1 --2- 172.21.15.46:0/2022917061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4980cb6e0 0x7fb4980cbab0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.046+0000 7fb49ce0b700 1 -- 172.21.15.46:0/2022917061 >> 172.21.15.46:0/2022917061 conn(0x7fb49806f8c0 msgr2=0x7fb4980d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:50.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.046+0000 7fb49ce0b700 1 -- 172.21.15.46:0/2022917061 shutdown_connections 2024-01-27T01:08:50.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.046+0000 7fb49ce0b700 1 -- 172.21.15.46:0/2022917061 wait complete. 2024-01-27T01:08:50.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.047+0000 7fb49ce0b700 1 Processor -- start 2024-01-27T01:08:50.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.047+0000 7fb49ce0b700 1 -- start start 2024-01-27T01:08:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.048+0000 7fb49ce0b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4980cb6e0 0x7fb4980b1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.048+0000 7fb49ce0b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4980cf3e0 0x7fb498151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.048+0000 7fb49ce0b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb498159f70 0x7fb49815d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.048+0000 7fb49ce0b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4980b02f0 con 0x7fb498159f70 2024-01-27T01:08:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.048+0000 7fb49ce0b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49815e9e0 con 0x7fb4980cb6e0 2024-01-27T01:08:50.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.048+0000 7fb49ce0b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49815e140 con 0x7fb4980cf3e0 2024-01-27T01:08:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.048+0000 7fb496d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb498159f70 0x7fb49815d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.048+0000 7fb496d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb498159f70 0x7fb49815d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48392/0 (socket says 172.21.15.46:48392) 2024-01-27T01:08:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.048+0000 7fb496d9d700 1 -- 172.21.15.46:0/2875644125 learned_addr learned my addr 172.21.15.46:0/2875644125 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.048+0000 7fb49659c700 1 --2- 172.21.15.46:0/2875644125 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4980cb6e0 0x7fb4980b1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.049+0000 7fb495d9b700 1 --2- 172.21.15.46:0/2875644125 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4980cf3e0 0x7fb498151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.049+0000 7fb496d9d700 1 -- 172.21.15.46:0/2875644125 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4980cf3e0 msgr2=0x7fb498151170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.049+0000 7fb496d9d700 1 --2- 172.21.15.46:0/2875644125 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4980cf3e0 0x7fb498151170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.049+0000 7fb496d9d700 1 -- 172.21.15.46:0/2875644125 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4980cb6e0 msgr2=0x7fb4980b1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.049+0000 7fb496d9d700 1 --2- 172.21.15.46:0/2875644125 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4980cb6e0 0x7fb4980b1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.049+0000 7fb496d9d700 1 -- 172.21.15.46:0/2875644125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb488009420 con 0x7fb498159f70 2024-01-27T01:08:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.049+0000 7fb496d9d700 1 --2- 172.21.15.46:0/2875644125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb498159f70 0x7fb49815d840 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7fb49000cca0 tx=0x7fb490001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.049+0000 7fb4867fc700 1 -- 172.21.15.46:0/2875644125 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49000ab30 con 0x7fb498159f70 2024-01-27T01:08:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.049+0000 7fb4867fc700 1 -- 172.21.15.46:0/2875644125 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49000ac90 con 0x7fb498159f70 2024-01-27T01:08:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.049+0000 7fb49ce0b700 1 -- 172.21.15.46:0/2875644125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49815e420 con 0x7fb498159f70 2024-01-27T01:08:50.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.049+0000 7fb4867fc700 1 -- 172.21.15.46:0/2875644125 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb490007300 con 0x7fb498159f70 2024-01-27T01:08:50.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.049+0000 7fb49ce0b700 1 -- 172.21.15.46:0/2875644125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4981516b0 con 0x7fb498159f70 2024-01-27T01:08:50.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.050+0000 7fb49ce0b700 1 -- 172.21.15.46:0/2875644125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4980b0460 con 0x7fb498159f70 2024-01-27T01:08:50.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.051+0000 7fb4867fc700 1 -- 172.21.15.46:0/2875644125 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 153) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fb490007730 con 0x7fb498159f70 2024-01-27T01:08:50.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.051+0000 7fb4867fc700 1 --2- 172.21.15.46:0/2875644125 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb478066ed0 0x7fb478069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:50.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.051+0000 7fb4867fc700 1 -- 172.21.15.46:0/2875644125 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7fb490084bf0 con 0x7fb498159f70 2024-01-27T01:08:50.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.051+0000 7fb49659c700 1 --2- 172.21.15.46:0/2875644125 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb478066ed0 0x7fb478069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:50.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.052+0000 7fb49659c700 1 --2- 172.21.15.46:0/2875644125 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb478066ed0 0x7fb478069380 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb4880093f0 tx=0x7fb488017000 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:50.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.056+0000 7fb4867fc700 1 -- 172.21.15.46:0/2875644125 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb4980b0460 con 0x7fb498159f70 2024-01-27T01:08:50.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.262+0000 7fb49ce0b700 1 -- 172.21.15.46:0/2875644125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fb4980b0430 con 0x7fb498159f70 2024-01-27T01:08:50.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.266+0000 7fb4867fc700 1 -- 172.21.15.46:0/2875644125 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v92) v1 ==== 120+0+0 (secure 0 0 0) 0x7fb490051c00 con 0x7fb498159f70 2024-01-27T01:08:50.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.268+0000 7fb49ce0b700 1 -- 172.21.15.46:0/2875644125 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb478066ed0 msgr2=0x7fb478069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:50.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.268+0000 7fb49ce0b700 1 --2- 172.21.15.46:0/2875644125 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb478066ed0 0x7fb478069380 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb4880093f0 tx=0x7fb488017000 comp rx=0 tx=0).stop 2024-01-27T01:08:50.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.268+0000 7fb49ce0b700 1 -- 172.21.15.46:0/2875644125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb498159f70 msgr2=0x7fb49815d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:50.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.268+0000 7fb49ce0b700 1 --2- 172.21.15.46:0/2875644125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb498159f70 0x7fb49815d840 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7fb49000cca0 tx=0x7fb490001ae0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.268+0000 7fb49ce0b700 1 -- 172.21.15.46:0/2875644125 shutdown_connections 2024-01-27T01:08:50.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.268+0000 7fb49ce0b700 1 --2- 172.21.15.46:0/2875644125 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb478066ed0 0x7fb478069380 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.268+0000 7fb49ce0b700 1 --2- 172.21.15.46:0/2875644125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb498159f70 0x7fb49815d840 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.268+0000 7fb49ce0b700 1 --2- 172.21.15.46:0/2875644125 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4980cb6e0 0x7fb4980b1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.268+0000 7fb49ce0b700 1 --2- 172.21.15.46:0/2875644125 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4980cf3e0 0x7fb498151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.268+0000 7fb49ce0b700 1 -- 172.21.15.46:0/2875644125 >> 172.21.15.46:0/2875644125 conn(0x7fb49806f8c0 msgr2=0x7fb4980ad290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:50.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.268+0000 7fb49ce0b700 1 -- 172.21.15.46:0/2875644125 shutdown_connections 2024-01-27T01:08:50.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.268+0000 7fb49ce0b700 1 -- 172.21.15.46:0/2875644125 wait complete. 2024-01-27T01:08:50.282 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-27T01:08:50.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.415+0000 7f2aa4427700 1 Processor -- start 2024-01-27T01:08:50.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.415+0000 7f2aa4427700 1 -- start start 2024-01-27T01:08:50.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.416+0000 7f2aa4427700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2a9c0cbbe0 0x7f2a9c1411b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:50.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.416+0000 7f2aa4427700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a9c0b1380 0x7f2a9c0c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:50.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.416+0000 7f2aa4427700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9c0cc190 con 0x7f2a9c140de0 2024-01-27T01:08:50.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.416+0000 7f2aa4427700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a9c15c1a0 con 0x7f2a9c0cbbe0 2024-01-27T01:08:50.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.416+0000 7f2aa4427700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c0cc3e0 con 0x7f2a9c0b1380 2024-01-27T01:08:50.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.416+0000 7f2aa21c3700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2a9c140de0 0x7f2a9c15ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60946/0 (socket says 172.21.15.46:60946) 2024-01-27T01:08:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.416+0000 7f2aa21c3700 1 -- 172.21.15.46:0/1558526211 learned_addr learned my addr 172.21.15.46:0/1558526211 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.416+0000 7f2aa19c2700 1 --2- 172.21.15.46:0/1558526211 >> v2:172.21.15.106:3300/0 conn(0x7f2a9c0cbbe0 0x7f2a9c1411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.416+0000 7f2aa29c4700 1 --2- 172.21.15.46:0/1558526211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a9c0b1380 0x7f2a9c0c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.416+0000 7f2aa11c1700 1 -- 172.21.15.46:0/1558526211 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769420850 0 0) 0x7f2a9c0cc190 con 0x7f2a9c140de0 2024-01-27T01:08:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.416+0000 7f2aa11c1700 1 -- 172.21.15.46:0/1558526211 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a88003570 con 0x7f2a9c140de0 2024-01-27T01:08:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.417+0000 7f2aa29c4700 1 -- 172.21.15.46:0/1558526211 >> v2:172.21.15.106:3300/0 conn(0x7f2a9c0cbbe0 msgr2=0x7f2a9c1411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.417+0000 7f2aa11c1700 1 -- 172.21.15.46:0/1558526211 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 963422232 0 0) 0x7f2a88003570 con 0x7f2a9c140de0 2024-01-27T01:08:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.417+0000 7f2aa29c4700 1 --2- 172.21.15.46:0/1558526211 >> v2:172.21.15.106:3300/0 conn(0x7f2a9c0cbbe0 0x7f2a9c1411b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.417+0000 7f2aa29c4700 1 -- 172.21.15.46:0/1558526211 >> v1:172.21.15.46:6789/0 conn(0x7f2a9c140de0 legacy=0x7f2a9c15ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.417+0000 7f2aa29c4700 1 -- 172.21.15.46:0/1558526211 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c0cc6c0 con 0x7f2a9c0b1380 2024-01-27T01:08:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.417+0000 7f2aa19c2700 1 --2- 172.21.15.46:0/1558526211 >> v2:172.21.15.106:3300/0 conn(0x7f2a9c0cbbe0 0x7f2a9c1411b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.417+0000 7f2aa29c4700 1 --2- 172.21.15.46:0/1558526211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a9c0b1380 0x7f2a9c0c8c00 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2a94009e60 tx=0x7f2a94009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc865ba1bfbd3166 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.417+0000 7f2aa11c1700 1 -- 172.21.15.46:0/1558526211 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9400e070 con 0x7f2a9c0b1380 2024-01-27T01:08:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.417+0000 7f2aa11c1700 1 -- 172.21.15.46:0/1558526211 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9400b7a0 con 0x7f2a9c0b1380 2024-01-27T01:08:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.417+0000 7f2aa11c1700 1 -- 172.21.15.46:0/1558526211 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98001470 con 0x7f2a9c0b1380 2024-01-27T01:08:50.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.418+0000 7f2aa4427700 1 -- 172.21.15.46:0/1558526211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a9c0b1380 msgr2=0x7f2a9c0c8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.418+0000 7f2aa4427700 1 --2- 172.21.15.46:0/1558526211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a9c0b1380 0x7f2a9c0c8c00 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2a94009e60 tx=0x7f2a94009780 comp rx=0 tx=0).stop 2024-01-27T01:08:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.418+0000 7f2aa4427700 1 -- 172.21.15.46:0/1558526211 shutdown_connections 2024-01-27T01:08:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.418+0000 7f2aa4427700 1 --2- 172.21.15.46:0/1558526211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a9c0b1380 0x7f2a9c0c8c00 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.418+0000 7f2aa4427700 1 --2- 172.21.15.46:0/1558526211 >> v2:172.21.15.106:3300/0 conn(0x7f2a9c0cbbe0 0x7f2a9c1411b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.418+0000 7f2aa4427700 1 -- 172.21.15.46:0/1558526211 >> 172.21.15.46:0/1558526211 conn(0x7f2a9c0679b0 msgr2=0x7f2a9c067db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.418+0000 7f2aa4427700 1 -- 172.21.15.46:0/1558526211 shutdown_connections 2024-01-27T01:08:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.418+0000 7f2aa4427700 1 -- 172.21.15.46:0/1558526211 wait complete. 2024-01-27T01:08:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.419+0000 7f2aa4427700 1 Processor -- start 2024-01-27T01:08:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.419+0000 7f2aa4427700 1 -- start start 2024-01-27T01:08:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.420+0000 7f2aa4427700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a9c0b1380 0x7f2a9c138af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.420+0000 7f2aa4427700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0cbbe0 0x7f2a9c139750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.420+0000 7f2aa4427700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a9c140de0 0x7f2a9c13b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.420+0000 7f2aa4427700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c0699e0 con 0x7f2a9c0cbbe0 2024-01-27T01:08:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.420+0000 7f2aa4427700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c0696e0 con 0x7f2a9c140de0 2024-01-27T01:08:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.420+0000 7f2aa4427700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c139c90 con 0x7f2a9c0b1380 2024-01-27T01:08:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.420+0000 7f2aa19c2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0cbbe0 0x7f2a9c139750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.420+0000 7f2aa19c2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0cbbe0 0x7f2a9c139750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55302/0 (socket says 172.21.15.46:55302) 2024-01-27T01:08:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.420+0000 7f2aa19c2700 1 -- 172.21.15.46:0/1725269183 learned_addr learned my addr 172.21.15.46:0/1725269183 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.420+0000 7f2aa29c4700 1 --2- 172.21.15.46:0/1725269183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a9c140de0 0x7f2a9c13b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.420+0000 7f2aa21c3700 1 --2- 172.21.15.46:0/1725269183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a9c0b1380 0x7f2a9c138af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.421+0000 7f2aa19c2700 1 -- 172.21.15.46:0/1725269183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a9c0b1380 msgr2=0x7f2a9c138af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.421+0000 7f2aa19c2700 1 --2- 172.21.15.46:0/1725269183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a9c0b1380 0x7f2a9c138af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.421+0000 7f2aa19c2700 1 -- 172.21.15.46:0/1725269183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a9c140de0 msgr2=0x7f2a9c13b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.421+0000 7f2aa19c2700 1 --2- 172.21.15.46:0/1725269183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a9c140de0 0x7f2a9c13b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.421+0000 7f2aa19c2700 1 -- 172.21.15.46:0/1725269183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c009420 con 0x7f2a9c0cbbe0 2024-01-27T01:08:50.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.421+0000 7f2aa19c2700 1 --2- 172.21.15.46:0/1725269183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0cbbe0 0x7f2a9c139750 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00ec80 tx=0x7f2a8c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:50.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.421+0000 7f2a927fc700 1 -- 172.21.15.46:0/1725269183 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c00c850 con 0x7f2a9c0cbbe0 2024-01-27T01:08:50.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.421+0000 7f2a927fc700 1 -- 172.21.15.46:0/1725269183 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8c00c9b0 con 0x7f2a9c0cbbe0 2024-01-27T01:08:50.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.421+0000 7f2a927fc700 1 -- 172.21.15.46:0/1725269183 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c00cd50 con 0x7f2a9c0cbbe0 2024-01-27T01:08:50.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.421+0000 7f2aa4427700 1 -- 172.21.15.46:0/1725269183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a94009a20 con 0x7f2a9c0cbbe0 2024-01-27T01:08:50.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.421+0000 7f2aa4427700 1 -- 172.21.15.46:0/1725269183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a9c069030 con 0x7f2a9c0cbbe0 2024-01-27T01:08:50.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.422+0000 7f2aa4427700 1 -- 172.21.15.46:0/1725269183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a9c06eea0 con 0x7f2a9c0cbbe0 2024-01-27T01:08:50.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.422+0000 7f2a927fc700 1 -- 172.21.15.46:0/1725269183 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 153) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f2a8c013680 con 0x7f2a9c0cbbe0 2024-01-27T01:08:50.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.423+0000 7f2a927fc700 1 --2- 172.21.15.46:0/1725269183 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2a80066ea0 0x7f2a80069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:50.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.423+0000 7f2a927fc700 1 -- 172.21.15.46:0/1725269183 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f2a8c084d30 con 0x7f2a9c0cbbe0 2024-01-27T01:08:50.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.423+0000 7f2aa21c3700 1 --2- 172.21.15.46:0/1725269183 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2a80066ea0 0x7f2a80069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:50.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.424+0000 7f2aa21c3700 1 --2- 172.21.15.46:0/1725269183 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2a80066ea0 0x7f2a80069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c16d130 tx=0x7f2a98001a00 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:50.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.428+0000 7f2a927fc700 1 -- 172.21.15.46:0/1725269183 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2a8c089070 con 0x7f2a9c0cbbe0 2024-01-27T01:08:50.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.632+0000 7f2aa4427700 1 -- 172.21.15.46:0/1725269183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f2a9c0cc9a0 con 0x7f2a9c0cbbe0 2024-01-27T01:08:50.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.636+0000 7f2a927fc700 1 -- 172.21.15.46:0/1725269183 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v93) v1 ==== 115+0+0 (secure 0 0 0) 0x7f2a8c0170b0 con 0x7f2a9c0cbbe0 2024-01-27T01:08:50.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.638+0000 7f2aa4427700 1 -- 172.21.15.46:0/1725269183 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2a80066ea0 msgr2=0x7f2a80069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:50.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.638+0000 7f2aa4427700 1 --2- 172.21.15.46:0/1725269183 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2a80066ea0 0x7f2a80069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c16d130 tx=0x7f2a98001a00 comp rx=0 tx=0).stop 2024-01-27T01:08:50.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.638+0000 7f2aa4427700 1 -- 172.21.15.46:0/1725269183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0cbbe0 msgr2=0x7f2a9c139750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:50.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.638+0000 7f2aa4427700 1 --2- 172.21.15.46:0/1725269183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0cbbe0 0x7f2a9c139750 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00ec80 tx=0x7f2a8c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.638+0000 7f2aa4427700 1 -- 172.21.15.46:0/1725269183 shutdown_connections 2024-01-27T01:08:50.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.638+0000 7f2aa4427700 1 --2- 172.21.15.46:0/1725269183 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2a80066ea0 0x7f2a80069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.638+0000 7f2aa4427700 1 --2- 172.21.15.46:0/1725269183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a9c140de0 0x7f2a9c13b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.638+0000 7f2aa4427700 1 --2- 172.21.15.46:0/1725269183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a9c0b1380 0x7f2a9c138af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.638+0000 7f2aa4427700 1 --2- 172.21.15.46:0/1725269183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9c0cbbe0 0x7f2a9c139750 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.638+0000 7f2aa4427700 1 -- 172.21.15.46:0/1725269183 >> 172.21.15.46:0/1725269183 conn(0x7f2a9c0679b0 msgr2=0x7f2a9c0c3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:50.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.638+0000 7f2aa4427700 1 -- 172.21.15.46:0/1725269183 shutdown_connections 2024-01-27T01:08:50.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.638+0000 7f2aa4427700 1 -- 172.21.15.46:0/1725269183 wait complete. 2024-01-27T01:08:50.652 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-01-27T01:08:50.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.784+0000 7fb2df937700 1 Processor -- start 2024-01-27T01:08:50.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.785+0000 7fb2df937700 1 -- start start 2024-01-27T01:08:50.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.785+0000 7fb2df937700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d8156770 0x7fb2d806ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:50.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.785+0000 7fb2df937700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb2d806f130 0x7fb2d80b1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.785+0000 7fb2df937700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d806f590 con 0x7fb2d81563a0 2024-01-27T01:08:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.785+0000 7fb2df937700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d80b02c0 con 0x7fb2d806f130 2024-01-27T01:08:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.785+0000 7fb2df937700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d80b05a0 con 0x7fb2d8156770 2024-01-27T01:08:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.786+0000 7fb2dd6d3700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb2d81563a0 0x7fb2d815aaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60948/0 (socket says 172.21.15.46:60948) 2024-01-27T01:08:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.786+0000 7fb2dd6d3700 1 -- 172.21.15.46:0/755996465 learned_addr learned my addr 172.21.15.46:0/755996465 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.786+0000 7fb2dded4700 1 --2- 172.21.15.46:0/755996465 >> v2:172.21.15.106:3300/0 conn(0x7fb2d806f130 0x7fb2d80b1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.786+0000 7fb2dced2700 1 --2- 172.21.15.46:0/755996465 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d8156770 0x7fb2d806ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.786+0000 7fb2cbfff700 1 -- 172.21.15.46:0/755996465 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862380409 0 0) 0x7fb2cc001110 con 0x7fb2d81563a0 2024-01-27T01:08:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.786+0000 7fb2cbfff700 1 -- 172.21.15.46:0/755996465 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c4003570 con 0x7fb2d81563a0 2024-01-27T01:08:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.786+0000 7fb2cbfff700 1 -- 172.21.15.46:0/755996465 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2668087972 0 0) 0x7fb2c4003570 con 0x7fb2d81563a0 2024-01-27T01:08:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.786+0000 7fb2dced2700 1 -- 172.21.15.46:0/755996465 >> v2:172.21.15.106:3300/0 conn(0x7fb2d806f130 msgr2=0x7fb2d80b1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.786+0000 7fb2dced2700 1 --2- 172.21.15.46:0/755996465 >> v2:172.21.15.106:3300/0 conn(0x7fb2d806f130 0x7fb2d80b1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.786+0000 7fb2dced2700 1 -- 172.21.15.46:0/755996465 >> v1:172.21.15.46:6789/0 conn(0x7fb2d81563a0 legacy=0x7fb2d815aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.786+0000 7fb2dced2700 1 -- 172.21.15.46:0/755996465 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d80b0880 con 0x7fb2d8156770 2024-01-27T01:08:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.786+0000 7fb2dded4700 1 --2- 172.21.15.46:0/755996465 >> v2:172.21.15.106:3300/0 conn(0x7fb2d806f130 0x7fb2d80b1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.786+0000 7fb2dced2700 1 --2- 172.21.15.46:0/755996465 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d8156770 0x7fb2d806ebf0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb2d40098e0 tx=0x7fb2d40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7661c748417449c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.787+0000 7fb2cbfff700 1 -- 172.21.15.46:0/755996465 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc001600 con 0x7fb2d8156770 2024-01-27T01:08:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.787+0000 7fb2cbfff700 1 -- 172.21.15.46:0/755996465 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d4011420 con 0x7fb2d8156770 2024-01-27T01:08:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.787+0000 7fb2cbfff700 1 -- 172.21.15.46:0/755996465 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d40117c0 con 0x7fb2d8156770 2024-01-27T01:08:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.787+0000 7fb2df937700 1 -- 172.21.15.46:0/755996465 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d8156770 msgr2=0x7fb2d806ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.787+0000 7fb2df937700 1 --2- 172.21.15.46:0/755996465 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d8156770 0x7fb2d806ebf0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb2d40098e0 tx=0x7fb2d40091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.787+0000 7fb2df937700 1 -- 172.21.15.46:0/755996465 shutdown_connections 2024-01-27T01:08:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.787+0000 7fb2df937700 1 --2- 172.21.15.46:0/755996465 >> v2:172.21.15.106:3300/0 conn(0x7fb2d806f130 0x7fb2d80b1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.787+0000 7fb2df937700 1 --2- 172.21.15.46:0/755996465 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d8156770 0x7fb2d806ebf0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.787+0000 7fb2df937700 1 -- 172.21.15.46:0/755996465 >> 172.21.15.46:0/755996465 conn(0x7fb2d806b6c0 msgr2=0x7fb2d80aef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.787+0000 7fb2df937700 1 -- 172.21.15.46:0/755996465 shutdown_connections 2024-01-27T01:08:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.787+0000 7fb2df937700 1 -- 172.21.15.46:0/755996465 wait complete. 2024-01-27T01:08:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.789+0000 7fb2df937700 1 Processor -- start 2024-01-27T01:08:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.789+0000 7fb2df937700 1 -- start start 2024-01-27T01:08:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2df937700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2d806f130 0x7fb2d81593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2df937700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d81563a0 0x7fb2d8169960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2df937700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d8156770 0x7fb2d8150740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2df937700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d8153e60 con 0x7fb2d81563a0 2024-01-27T01:08:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2df937700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d8154140 con 0x7fb2d806f130 2024-01-27T01:08:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2df937700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d8154420 con 0x7fb2d8156770 2024-01-27T01:08:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2dced2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d81563a0 0x7fb2d8169960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2dced2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d81563a0 0x7fb2d8169960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55318/0 (socket says 172.21.15.46:55318) 2024-01-27T01:08:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2dced2700 1 -- 172.21.15.46:0/4206014101 learned_addr learned my addr 172.21.15.46:0/4206014101 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2dd6d3700 1 --2- 172.21.15.46:0/4206014101 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2d806f130 0x7fb2d81593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2dded4700 1 --2- 172.21.15.46:0/4206014101 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d8156770 0x7fb2d8150740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2dced2700 1 -- 172.21.15.46:0/4206014101 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d8156770 msgr2=0x7fb2d8150740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2dced2700 1 --2- 172.21.15.46:0/4206014101 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d8156770 0x7fb2d8150740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2dced2700 1 -- 172.21.15.46:0/4206014101 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2d806f130 msgr2=0x7fb2d81593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2dced2700 1 --2- 172.21.15.46:0/4206014101 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2d806f130 0x7fb2d81593a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2dced2700 1 -- 172.21.15.46:0/4206014101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d0009a20 con 0x7fb2d81563a0 2024-01-27T01:08:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2dced2700 1 --2- 172.21.15.46:0/4206014101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d81563a0 0x7fb2d8169960 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7fb2d40030c0 tx=0x7fb2d4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2c97fa700 1 -- 172.21.15.46:0/4206014101 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d401cbe0 con 0x7fb2d81563a0 2024-01-27T01:08:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2c97fa700 1 -- 172.21.15.46:0/4206014101 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d401cd40 con 0x7fb2d81563a0 2024-01-27T01:08:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.790+0000 7fb2df937700 1 -- 172.21.15.46:0/4206014101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d4009420 con 0x7fb2d81563a0 2024-01-27T01:08:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.791+0000 7fb2c97fa700 1 -- 172.21.15.46:0/4206014101 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d401a6a0 con 0x7fb2d81563a0 2024-01-27T01:08:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.791+0000 7fb2df937700 1 -- 172.21.15.46:0/4206014101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d81549e0 con 0x7fb2d81563a0 2024-01-27T01:08:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.792+0000 7fb2df937700 1 -- 172.21.15.46:0/4206014101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d8154bd0 con 0x7fb2d81563a0 2024-01-27T01:08:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.792+0000 7fb2c97fa700 1 -- 172.21.15.46:0/4206014101 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 153) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fb2d401aad0 con 0x7fb2d81563a0 2024-01-27T01:08:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.792+0000 7fb2c97fa700 1 --2- 172.21.15.46:0/4206014101 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb2bc066ea0 0x7fb2bc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.792+0000 7fb2c97fa700 1 -- 172.21.15.46:0/4206014101 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7fb2d4085f80 con 0x7fb2d81563a0 2024-01-27T01:08:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.793+0000 7fb2dd6d3700 1 --2- 172.21.15.46:0/4206014101 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb2bc066ea0 0x7fb2bc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:50.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.793+0000 7fb2dd6d3700 1 --2- 172.21.15.46:0/4206014101 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb2bc066ea0 0x7fb2bc069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc004e10 tx=0x7fb2cc001310 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:50.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:50.797+0000 7fb2c97fa700 1 -- 172.21.15.46:0/4206014101 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb2d4053060 con 0x7fb2d81563a0 2024-01-27T01:08:51.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:51.084+0000 7fb2df937700 1 -- 172.21.15.46:0/4206014101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7fb2d804c710 con 0x7fb2d81563a0 2024-01-27T01:08:51.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:51.497+0000 7fb2c97fa700 1 -- 172.21.15.46:0/4206014101 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3277) v1 ==== 155+0+0 (secure 0 0 0) 0x7fb2d40290e0 con 0x7fb2d81563a0 2024-01-27T01:08:51.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:51.501+0000 7fb2df937700 1 -- 172.21.15.46:0/4206014101 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb2bc066ea0 msgr2=0x7fb2bc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:51.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:51.501+0000 7fb2df937700 1 --2- 172.21.15.46:0/4206014101 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb2bc066ea0 0x7fb2bc069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc004e10 tx=0x7fb2cc001310 comp rx=0 tx=0).stop 2024-01-27T01:08:51.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:51.501+0000 7fb2df937700 1 -- 172.21.15.46:0/4206014101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d81563a0 msgr2=0x7fb2d8169960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:51.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:51.501+0000 7fb2df937700 1 --2- 172.21.15.46:0/4206014101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d81563a0 0x7fb2d8169960 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7fb2d40030c0 tx=0x7fb2d4016000 comp rx=0 tx=0).stop 2024-01-27T01:08:51.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:51.501+0000 7fb2df937700 1 -- 172.21.15.46:0/4206014101 shutdown_connections 2024-01-27T01:08:51.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:51.501+0000 7fb2df937700 1 --2- 172.21.15.46:0/4206014101 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb2bc066ea0 0x7fb2bc069350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:51.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:51.501+0000 7fb2df937700 1 --2- 172.21.15.46:0/4206014101 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2d8156770 0x7fb2d8150740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:51.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:51.501+0000 7fb2df937700 1 --2- 172.21.15.46:0/4206014101 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2d806f130 0x7fb2d81593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:51.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:51.501+0000 7fb2df937700 1 --2- 172.21.15.46:0/4206014101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d81563a0 0x7fb2d8169960 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:51.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:51.501+0000 7fb2df937700 1 -- 172.21.15.46:0/4206014101 >> 172.21.15.46:0/4206014101 conn(0x7fb2d806b6c0 msgr2=0x7fb2d80b1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:51.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:51.501+0000 7fb2df937700 1 -- 172.21.15.46:0/4206014101 shutdown_connections 2024-01-27T01:08:51.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:51.501+0000 7fb2df937700 1 -- 172.21.15.46:0/4206014101 wait complete. 2024-01-27T01:08:51.519 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2024-01-27T01:08:51.520 INFO:tasks.cephfs_test_runner: 2024-01-27T01:08:51.520 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-27T01:08:51.520 INFO:tasks.cephfs_test_runner:Ran 14 tests in 5897.461s 2024-01-27T01:08:51.520 INFO:tasks.cephfs_test_runner: 2024-01-27T01:08:51.520 INFO:tasks.cephfs_test_runner:OK 2024-01-27T01:08:51.521 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-01-27T01:08:51.534 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-01-27T01:08:51.547 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-01-27T01:08:51.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:51.547 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-01-27T01:08:51.569 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-27T01:08:51.569 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-27T01:08:51.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:51.570 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-27T01:08:51.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:51.654 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-27T01:08:51.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:51.741 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link delete ceph-brx 2024-01-27T01:08:51.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:51.821 DEBUG:teuthology.orchestra.run.smithi106:> ip netns list 2024-01-27T01:08:51.843 INFO:teuthology.orchestra.run.smithi106.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-27T01:08:51.843 INFO:teuthology.orchestra.run.smithi106.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-27T01:08:51.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:51.844 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-27T01:08:51.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:51.932 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-27T01:08:51.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-27T01:08:51.989 DEBUG:teuthology.orchestra.run.smithi106:> sudo ip link delete ceph-brx 2024-01-27T01:08:52.075 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-27T01:08:52.084 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-01-27T01:08:52.084 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:08:52.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.228+0000 7f274f124700 1 Processor -- start 2024-01-27T01:08:52.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.228+0000 7f274f124700 1 -- start start 2024-01-27T01:08:52.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.229+0000 7f274f124700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f27480b2c60 0x7f27480b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:52.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.229+0000 7f274f124700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27480d1490 con 0x7f27480b3ad0 2024-01-27T01:08:52.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.229+0000 7f274f124700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27480c6390 con 0x7f27480ce3a0 2024-01-27T01:08:52.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.229+0000 7f274f124700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27480ccd80 con 0x7f27480b2c60 2024-01-27T01:08:52.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.229+0000 7f274cec0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f27480b2c60 0x7f27480b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:52.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.229+0000 7f2747fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f27480b3ad0 0x7f27480c6820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60954/0 (socket says 172.21.15.46:60954) 2024-01-27T01:08:52.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.229+0000 7f2747fff700 1 -- 172.21.15.46:0/3947964479 learned_addr learned my addr 172.21.15.46:0/3947964479 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:52.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.230+0000 7f274cec0700 1 -- 172.21.15.46:0/3947964479 >> v1:172.21.15.106:6789/0 conn(0x7f27480ce3a0 legacy=0x7f27480ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.230+0000 7f274cec0700 1 -- 172.21.15.46:0/3947964479 >> v1:172.21.15.46:6789/0 conn(0x7f27480b3ad0 legacy=0x7f27480c6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.230+0000 7f274cec0700 1 -- 172.21.15.46:0/3947964479 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27480cc7c0 con 0x7f27480b2c60 2024-01-27T01:08:52.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.230+0000 7f27477fe700 1 -- 172.21.15.46:0/3947964479 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632640574 0 0) 0x7f27480d1490 con 0x7f27480b3ad0 2024-01-27T01:08:52.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.230+0000 7f274cec0700 1 --2- 172.21.15.46:0/3947964479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27480b2c60 0x7f27480b3590 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f27380098e0 tx=0x7f27380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e05ab6d6c5cc55c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:52.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.230+0000 7f27477fe700 1 -- 172.21.15.46:0/3947964479 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738009e80 con 0x7f27480b2c60 2024-01-27T01:08:52.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.230+0000 7f27477fe700 1 -- 172.21.15.46:0/3947964479 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273800b860 con 0x7f27480b2c60 2024-01-27T01:08:52.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.230+0000 7f27477fe700 1 -- 172.21.15.46:0/3947964479 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273800bbe0 con 0x7f27480b2c60 2024-01-27T01:08:52.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.231+0000 7f274f124700 1 -- 172.21.15.46:0/3947964479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27480b2c60 msgr2=0x7f27480b3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.231+0000 7f274f124700 1 --2- 172.21.15.46:0/3947964479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27480b2c60 0x7f27480b3590 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f27380098e0 tx=0x7f27380091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.231+0000 7f274f124700 1 -- 172.21.15.46:0/3947964479 shutdown_connections 2024-01-27T01:08:52.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.231+0000 7f274f124700 1 --2- 172.21.15.46:0/3947964479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27480b2c60 0x7f27480b3590 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.231+0000 7f274f124700 1 -- 172.21.15.46:0/3947964479 >> 172.21.15.46:0/3947964479 conn(0x7f274806f8c0 msgr2=0x7f27480d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:52.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.231+0000 7f274f124700 1 -- 172.21.15.46:0/3947964479 shutdown_connections 2024-01-27T01:08:52.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.231+0000 7f274f124700 1 -- 172.21.15.46:0/3947964479 wait complete. 2024-01-27T01:08:52.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.232+0000 7f274f124700 1 Processor -- start 2024-01-27T01:08:52.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.232+0000 7f274f124700 1 -- start start 2024-01-27T01:08:52.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.233+0000 7f274f124700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27480b2c60 0x7f27480b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:52.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.233+0000 7f274f124700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27480b3ad0 0x7f2748151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:52.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.233+0000 7f274f124700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f27480ce3a0 0x7f2748152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:52.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.233+0000 7f274f124700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2748152980 con 0x7f27480b2c60 2024-01-27T01:08:52.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.233+0000 7f274f124700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27480c7920 con 0x7f27480b3ad0 2024-01-27T01:08:52.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.233+0000 7f274f124700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27480c7c00 con 0x7f27480ce3a0 2024-01-27T01:08:52.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.233+0000 7f274cec0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27480b2c60 0x7f27480b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:52.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.233+0000 7f274cec0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27480b2c60 0x7f27480b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55322/0 (socket says 172.21.15.46:55322) 2024-01-27T01:08:52.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.233+0000 7f274cec0700 1 -- 172.21.15.46:0/3989343762 learned_addr learned my addr 172.21.15.46:0/3989343762 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:52.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.233+0000 7f2747fff700 1 --2- 172.21.15.46:0/3989343762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27480b3ad0 0x7f2748151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:52.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.233+0000 7f274d6c1700 1 --2- 172.21.15.46:0/3989343762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f27480ce3a0 0x7f2748152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:52.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.233+0000 7f274cec0700 1 -- 172.21.15.46:0/3989343762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f27480ce3a0 msgr2=0x7f2748152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.233+0000 7f274cec0700 1 --2- 172.21.15.46:0/3989343762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f27480ce3a0 0x7f2748152440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.233+0000 7f274cec0700 1 -- 172.21.15.46:0/3989343762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27480b3ad0 msgr2=0x7f2748151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.233+0000 7f274cec0700 1 --2- 172.21.15.46:0/3989343762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27480b3ad0 0x7f2748151f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.234+0000 7f274cec0700 1 -- 172.21.15.46:0/3989343762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2738009420 con 0x7f27480b2c60 2024-01-27T01:08:52.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.234+0000 7f274d6c1700 1 --2- 172.21.15.46:0/3989343762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f27480ce3a0 0x7f2748152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:52.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.234+0000 7f274cec0700 1 --2- 172.21.15.46:0/3989343762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27480b2c60 0x7f27480b52f0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f2748088990 tx=0x7f2738017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:52.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.234+0000 7f2744ff9700 1 -- 172.21.15.46:0/3989343762 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273801f070 con 0x7f27480b2c60 2024-01-27T01:08:52.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.234+0000 7f2744ff9700 1 -- 172.21.15.46:0/3989343762 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2738000c90 con 0x7f27480b2c60 2024-01-27T01:08:52.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.234+0000 7f274f124700 1 -- 172.21.15.46:0/3989343762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27480c7ee0 con 0x7f27480b2c60 2024-01-27T01:08:52.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.234+0000 7f2744ff9700 1 -- 172.21.15.46:0/3989343762 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738025420 con 0x7f27480b2c60 2024-01-27T01:08:52.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.234+0000 7f274f124700 1 -- 172.21.15.46:0/3989343762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27480c84a0 con 0x7f27480b2c60 2024-01-27T01:08:52.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.235+0000 7f274f124700 1 -- 172.21.15.46:0/3989343762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27480c86c0 con 0x7f27480b2c60 2024-01-27T01:08:52.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.235+0000 7f2744ff9700 1 -- 172.21.15.46:0/3989343762 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2738023070 con 0x7f27480b2c60 2024-01-27T01:08:52.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.236+0000 7f2744ff9700 1 --2- 172.21.15.46:0/3989343762 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2728066f30 0x7f27280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:52.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.236+0000 7f2744ff9700 1 -- 172.21.15.46:0/3989343762 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f2738085de0 con 0x7f27480b2c60 2024-01-27T01:08:52.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.236+0000 7f2747fff700 1 --2- 172.21.15.46:0/3989343762 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2728066f30 0x7f27280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:52.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.237+0000 7f2747fff700 1 --2- 172.21.15.46:0/3989343762 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2728066f30 0x7f27280693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f27480b5cd0 tx=0x7f2730007f30 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:52.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.240+0000 7f2744ff9700 1 -- 172.21.15.46:0/3989343762 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2738052d20 con 0x7f27480b2c60 2024-01-27T01:08:52.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.447+0000 7f274f124700 1 -- 172.21.15.46:0/3989343762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f274804c710 con 0x7f27480b2c60 2024-01-27T01:08:52.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.447+0000 7f2744ff9700 1 -- 172.21.15.46:0/3989343762 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v269) v1 ==== 74+0+14363 (secure 0 0 0) 0x7f273802a090 con 0x7f27480b2c60 2024-01-27T01:08:52.448 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:52.448 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":269,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:08:49.489481+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/1756359338":"2024-01-28T01:08:46.477249+0000","172.21.15.106:0/1457405865":"2024-01-28T01:08:46.477249+0000","172.21.15.106:0/1421527645":"2024-01-28T01:08:46.477249+0000","172.21.15.106:0/1106084814":"2024-01-28T01:08:46.477249+0000","172.21.15.46:6833/3798781145":"2024-01-28T01:08:46.188351+0000","172.21.15.46:6832/3798781145":"2024-01-28T01:08:46.188351+0000","172.21.15.106:6837/62434":"2024-01-28T01:08:35.384966+0000","172.21.15.106:6836/62434":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3497341404":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3894462583":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/1144438896":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3124086304":"2024-01-28T01:08:46.477249+0000","172.21.15.106:0/3166472261":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/1405196355":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/1502373513":"2024-01-28T01:08:46.477249+0000","172.21.15.106:0/1752514247":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/2487246395":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/860101723":"2024-01-28T01:08:46.477249+0000","172.21.15.106:0/3477459747":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3540935782":"2024-01-27T02:08:26.076460+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:08:52.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.450+0000 7f274f124700 1 -- 172.21.15.46:0/3989343762 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2728066f30 msgr2=0x7f27280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.450+0000 7f274f124700 1 --2- 172.21.15.46:0/3989343762 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2728066f30 0x7f27280693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f27480b5cd0 tx=0x7f2730007f30 comp rx=0 tx=0).stop 2024-01-27T01:08:52.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.450+0000 7f274f124700 1 -- 172.21.15.46:0/3989343762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27480b2c60 msgr2=0x7f27480b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.450+0000 7f274f124700 1 --2- 172.21.15.46:0/3989343762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27480b2c60 0x7f27480b52f0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f2748088990 tx=0x7f2738017000 comp rx=0 tx=0).stop 2024-01-27T01:08:52.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.450+0000 7f274f124700 1 -- 172.21.15.46:0/3989343762 shutdown_connections 2024-01-27T01:08:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.450+0000 7f274f124700 1 --2- 172.21.15.46:0/3989343762 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2728066f30 0x7f27280693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.450+0000 7f274f124700 1 --2- 172.21.15.46:0/3989343762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27480b2c60 0x7f27480b52f0 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.450+0000 7f274f124700 1 --2- 172.21.15.46:0/3989343762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f27480ce3a0 0x7f2748152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.450+0000 7f274f124700 1 --2- 172.21.15.46:0/3989343762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27480b3ad0 0x7f2748151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.450+0000 7f274f124700 1 -- 172.21.15.46:0/3989343762 >> 172.21.15.46:0/3989343762 conn(0x7f274806f8c0 msgr2=0x7f27480cdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.450+0000 7f274f124700 1 -- 172.21.15.46:0/3989343762 shutdown_connections 2024-01-27T01:08:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.450+0000 7f274f124700 1 -- 172.21.15.46:0/3989343762 wait complete. 2024-01-27T01:08:52.464 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-27T01:08:52.464 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:08:52.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.597+0000 7fdb517f0700 1 Processor -- start 2024-01-27T01:08:52.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.597+0000 7fdb517f0700 1 -- start start 2024-01-27T01:08:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.598+0000 7fdb517f0700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdb4c0b13b0 0x7fdb4c0b1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.598+0000 7fdb517f0700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4c069200 con 0x7fdb4c06dae0 2024-01-27T01:08:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.598+0000 7fdb517f0700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4c0b2450 con 0x7fdb4c06d710 2024-01-27T01:08:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.598+0000 7fdb517f0700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb4c069450 con 0x7fdb4c0b13b0 2024-01-27T01:08:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.598+0000 7fdb427fc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdb4c0b13b0 0x7fdb4c0b1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.598+0000 7fdb4b7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdb4c06dae0 0x7fdb4c1593a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60960/0 (socket says 172.21.15.46:60960) 2024-01-27T01:08:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.598+0000 7fdb4b7fe700 1 -- 172.21.15.46:0/3321645209 learned_addr learned my addr 172.21.15.46:0/3321645209 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.598+0000 7fdb427fc700 1 -- 172.21.15.46:0/3321645209 >> v1:172.21.15.106:6789/0 conn(0x7fdb4c06d710 legacy=0x7fdb4c069f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.598+0000 7fdb427fc700 1 -- 172.21.15.46:0/3321645209 >> v1:172.21.15.46:6789/0 conn(0x7fdb4c06dae0 legacy=0x7fdb4c1593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.599+0000 7fdb4a7fc700 1 -- 172.21.15.46:0/3321645209 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135494744 0 0) 0x7fdb4c069200 con 0x7fdb4c06dae0 2024-01-27T01:08:52.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.599+0000 7fdb427fc700 1 -- 172.21.15.46:0/3321645209 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4c069730 con 0x7fdb4c0b13b0 2024-01-27T01:08:52.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.599+0000 7fdb427fc700 1 --2- 172.21.15.46:0/3321645209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb4c0b13b0 0x7fdb4c0b1e70 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7fdb340098e0 tx=0x7fdb340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94080cb7e8d83893 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:52.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.599+0000 7fdb4a7fc700 1 -- 172.21.15.46:0/3321645209 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34009e80 con 0x7fdb4c0b13b0 2024-01-27T01:08:52.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.599+0000 7fdb4a7fc700 1 -- 172.21.15.46:0/3321645209 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3400b860 con 0x7fdb4c0b13b0 2024-01-27T01:08:52.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.599+0000 7fdb4a7fc700 1 -- 172.21.15.46:0/3321645209 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3400bbe0 con 0x7fdb4c0b13b0 2024-01-27T01:08:52.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.599+0000 7fdb517f0700 1 -- 172.21.15.46:0/3321645209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb4c0b13b0 msgr2=0x7fdb4c0b1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.599+0000 7fdb517f0700 1 --2- 172.21.15.46:0/3321645209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb4c0b13b0 0x7fdb4c0b1e70 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7fdb340098e0 tx=0x7fdb340091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.599+0000 7fdb517f0700 1 -- 172.21.15.46:0/3321645209 shutdown_connections 2024-01-27T01:08:52.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.599+0000 7fdb517f0700 1 --2- 172.21.15.46:0/3321645209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb4c0b13b0 0x7fdb4c0b1e70 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.599+0000 7fdb517f0700 1 -- 172.21.15.46:0/3321645209 >> 172.21.15.46:0/3321645209 conn(0x7fdb4c06bd40 msgr2=0x7fdb4c06c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:52.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.600+0000 7fdb517f0700 1 -- 172.21.15.46:0/3321645209 shutdown_connections 2024-01-27T01:08:52.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.600+0000 7fdb517f0700 1 -- 172.21.15.46:0/3321645209 wait complete. 2024-01-27T01:08:52.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.601+0000 7fdb517f0700 1 Processor -- start 2024-01-27T01:08:52.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.601+0000 7fdb517f0700 1 -- start start 2024-01-27T01:08:52.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.602+0000 7fdb517f0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb4c06d710 0x7fdb4c16cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:52.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.602+0000 7fdb517f0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb4c06dae0 0x7fdb4c15b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:52.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.602+0000 7fdb517f0700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb4c0b13b0 0x7fdb4c0aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:52.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.602+0000 7fdb517f0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c15c930 con 0x7fdb4c06dae0 2024-01-27T01:08:52.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.602+0000 7fdb517f0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c16d070 con 0x7fdb4c06d710 2024-01-27T01:08:52.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.602+0000 7fdb517f0700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c15bd20 con 0x7fdb4c0b13b0 2024-01-27T01:08:52.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb427fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb4c06dae0 0x7fdb4c15b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:52.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb427fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb4c06dae0 0x7fdb4c15b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55340/0 (socket says 172.21.15.46:55340) 2024-01-27T01:08:52.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb427fc700 1 -- 172.21.15.46:0/3288288167 learned_addr learned my addr 172.21.15.46:0/3288288167 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:52.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb4affd700 1 --2- 172.21.15.46:0/3288288167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb4c06d710 0x7fdb4c16cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:52.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb4b7fe700 1 --2- 172.21.15.46:0/3288288167 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb4c0b13b0 0x7fdb4c0aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:52.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb427fc700 1 -- 172.21.15.46:0/3288288167 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb4c0b13b0 msgr2=0x7fdb4c0aebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb427fc700 1 --2- 172.21.15.46:0/3288288167 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb4c0b13b0 0x7fdb4c0aebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb427fc700 1 -- 172.21.15.46:0/3288288167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb4c06d710 msgr2=0x7fdb4c16cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb427fc700 1 --2- 172.21.15.46:0/3288288167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb4c06d710 0x7fdb4c16cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb427fc700 1 -- 172.21.15.46:0/3288288167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb34009420 con 0x7fdb4c06dae0 2024-01-27T01:08:52.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb4b7fe700 1 --2- 172.21.15.46:0/3288288167 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb4c0b13b0 0x7fdb4c0aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:08:52.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb4affd700 1 --2- 172.21.15.46:0/3288288167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb4c06d710 0x7fdb4c16cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:52.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb427fc700 1 --2- 172.21.15.46:0/3288288167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb4c06dae0 0x7fdb4c15b2e0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c06f990 tx=0x7fdb34017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:52.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb437fe700 1 -- 172.21.15.46:0/3288288167 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3401f070 con 0x7fdb4c06dae0 2024-01-27T01:08:52.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb437fe700 1 -- 172.21.15.46:0/3288288167 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3400b9d0 con 0x7fdb4c06dae0 2024-01-27T01:08:52.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb437fe700 1 -- 172.21.15.46:0/3288288167 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34025670 con 0x7fdb4c06dae0 2024-01-27T01:08:52.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb517f0700 1 -- 172.21.15.46:0/3288288167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb4c15c000 con 0x7fdb4c06dae0 2024-01-27T01:08:52.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.603+0000 7fdb517f0700 1 -- 172.21.15.46:0/3288288167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4c15c5c0 con 0x7fdb4c06dae0 2024-01-27T01:08:52.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.604+0000 7fdb517f0700 1 -- 172.21.15.46:0/3288288167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb4c04c710 con 0x7fdb4c06dae0 2024-01-27T01:08:52.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.605+0000 7fdb437fe700 1 -- 172.21.15.46:0/3288288167 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fdb34023070 con 0x7fdb4c06dae0 2024-01-27T01:08:52.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.605+0000 7fdb437fe700 1 --2- 172.21.15.46:0/3288288167 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdb2c066f30 0x7fdb2c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:52.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.605+0000 7fdb437fe700 1 -- 172.21.15.46:0/3288288167 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7fdb34085eb0 con 0x7fdb4c06dae0 2024-01-27T01:08:52.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.606+0000 7fdb4affd700 1 --2- 172.21.15.46:0/3288288167 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdb2c066f30 0x7fdb2c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:52.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.606+0000 7fdb4affd700 1 --2- 172.21.15.46:0/3288288167 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdb2c066f30 0x7fdb2c0693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c007fd0 tx=0x7fdb3c009040 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:52.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.610+0000 7fdb437fe700 1 -- 172.21.15.46:0/3288288167 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdb34052df0 con 0x7fdb4c06dae0 2024-01-27T01:08:52.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.814+0000 7fdb517f0700 1 -- 172.21.15.46:0/3288288167 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdb4c15c750 con 0x7fdb2c066f30 2024-01-27T01:08:52.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.817+0000 7fdb437fe700 1 -- 172.21.15.46:0/3288288167 <== mgr.10565 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51218 (secure 0 0 0) 0x7fdb4c15c750 con 0x7fdb2c066f30 2024-01-27T01:08:52.817 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:52.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.819+0000 7fdb517f0700 1 -- 172.21.15.46:0/3288288167 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdb2c066f30 msgr2=0x7fdb2c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.819+0000 7fdb517f0700 1 --2- 172.21.15.46:0/3288288167 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdb2c066f30 0x7fdb2c0693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c007fd0 tx=0x7fdb3c009040 comp rx=0 tx=0).stop 2024-01-27T01:08:52.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.819+0000 7fdb517f0700 1 -- 172.21.15.46:0/3288288167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb4c06dae0 msgr2=0x7fdb4c15b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.819+0000 7fdb517f0700 1 --2- 172.21.15.46:0/3288288167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb4c06dae0 0x7fdb4c15b2e0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c06f990 tx=0x7fdb34017000 comp rx=0 tx=0).stop 2024-01-27T01:08:52.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.819+0000 7fdb517f0700 1 -- 172.21.15.46:0/3288288167 shutdown_connections 2024-01-27T01:08:52.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.819+0000 7fdb517f0700 1 --2- 172.21.15.46:0/3288288167 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fdb2c066f30 0x7fdb2c0693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.819+0000 7fdb517f0700 1 --2- 172.21.15.46:0/3288288167 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdb4c0b13b0 0x7fdb4c0aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.819+0000 7fdb517f0700 1 --2- 172.21.15.46:0/3288288167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdb4c06d710 0x7fdb4c16cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.819+0000 7fdb517f0700 1 --2- 172.21.15.46:0/3288288167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb4c06dae0 0x7fdb4c15b2e0 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.819+0000 7fdb517f0700 1 -- 172.21.15.46:0/3288288167 >> 172.21.15.46:0/3288288167 conn(0x7fdb4c06bd40 msgr2=0x7fdb4c0ae010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:52.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.819+0000 7fdb517f0700 1 -- 172.21.15.46:0/3288288167 shutdown_connections 2024-01-27T01:08:52.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.819+0000 7fdb517f0700 1 -- 172.21.15.46:0/3288288167 wait complete. 2024-01-27T01:08:52.824 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:08:52.832 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-01-27T01:08:52.492281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2719,"num_read_kb":5760,"num_write":720,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8930,"ondisk_log_size":8930,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2488844,"kb_used_data":28560,"kb_used_omap":0,"kb_used_meta":2460160,"kb_avail":752485876,"statfs":{"total":773094113280,"available":770545537024,"internally_reserved":0,"allocated":29245440,"data_stored":16564239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2519203840},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-665943,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-280,"num_write_kb":-2051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.996864"},"pg_stats":[{"pgid":"1.0","version":"265'11159","reported_seq":12260,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-01-27T01:08:37.481193+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:08:37.481193+0000","last_peered":"2024-01-27T01:08:37.481193+0000","last_clean":"2024-01-27T01:08:37.481193+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:08:37.481193+0000","last_undegraded":"2024-01-27T01:08:37.481193+0000","last_fullsized":"2024-01-27T01:08:37.481193+0000","mapping_epoch":14,"log_start":"98'2229","ondisk_log_start":"98'2229","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8930,"log_dups_size":0,"ondisk_log_size":8930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2719,"num_read_kb":5760,"num_write":720,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2719,"num_read_kb":5760,"num_write":720,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8930,"ondisk_log_size":8930,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576026,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313844,"kb_used_data":4208,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94057996,"statfs":{"total":96636764160,"available":96315387904,"internally_reserved":0,"allocated":4308992,"data_stored":2715395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.58199999999999996,"15min":0.60199999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.312},"max":{"1min":0.745,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61699999999999999,"15min":0.61799999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33000000000000002},"max":{"1min":0.76400000000000001,"5min":0.879,"15min":0.879},"last":0.57099999999999995}]},{"osd":1,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.58599999999999997,"15min":0.60399999999999998},"min":{"1min":0.40300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73799999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.432},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.34300000000000003},"max":{"1min":0.81499999999999995,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.51000000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.59499999999999997,"15min":0.60399999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.753,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.39200000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85799999999999998,"5min":0.872,"15min":0.88900000000000001},"last":0.53000000000000003}]},{"osd":3,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58799999999999997,"15min":0.60799999999999998},"min":{"1min":0.374,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79100000000000004,"5min":0.85399999999999998,"15min":0.876},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64300000000000002,"15min":0.64000000000000001},"min":{"1min":0.40600000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.80600000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.49199999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.496,"15min":0.48699999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76100000000000001,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.52600000000000002,"15min":0.51500000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67800000000000005,"5min":0.80300000000000005,"15min":0.86599999999999999},"last":0.42199999999999999}]},{"osd":5,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.54100000000000004,"15min":0.52400000000000002},"min":{"1min":0.33600000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.81399999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.54300000000000004}]},{"osd":6,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59299999999999997,"15min":0.58399999999999996},"min":{"1min":0.33700000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84499999999999997,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58799999999999997,"15min":0.58199999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77900000000000003,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.501}]}]},{"osd":6,"up_from":13,"seq":55834576027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312908,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058932,"statfs":{"total":96636764160,"available":96316346368,"internally_reserved":0,"allocated":3411968,"data_stored":1817037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.56499999999999995,"15min":0.59199999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.83499999999999996,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.57499999999999996,"15min":0.60799999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73499999999999999,"5min":0.82499999999999996,"15min":0.82899999999999996},"last":0.42299999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.55800000000000005,"15min":0.59599999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70399999999999996,"5min":0.80300000000000005,"15min":0.84499999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57899999999999996,"15min":0.61099999999999999},"min":{"1min":0.39600000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76400000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.45000000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.55300000000000005,"15min":0.60099999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78100000000000003,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.57699999999999996,"15min":0.61099999999999999},"min":{"1min":0.379,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.754,"5min":0.80400000000000005,"15min":0.878},"last":0.49299999999999999}]},{"osd":3,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.58699999999999997,"15min":0.61499999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80600000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.62,"15min":0.64900000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.877,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.503}]},{"osd":4,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.495,"15min":0.497},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.80300000000000005},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.441,"5min":0.51600000000000001,"15min":0.53100000000000003},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.46800000000000003}]},{"osd":5,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.50600000000000001,"15min":0.51700000000000002},"min":{"1min":0.309,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70899999999999996,"5min":0.73999999999999999,"15min":0.79300000000000004},"last":0.437},{"interface":"front","average":{"1min":0.504,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.28899999999999998},"max":{"1min":0.752,"5min":0.87,"15min":0.87},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56699999999999995,"15min":0.56999999999999995},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.57499999999999996,"15min":0.58499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.79300000000000004,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.56000000000000005}]}]},{"osd":1,"up_from":13,"seq":55834576026,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304688,"kb_used_data":3304,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067152,"statfs":{"total":96636764160,"available":96324763648,"internally_reserved":0,"allocated":3383296,"data_stored":1802411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.28299999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.502,"5min":0.66100000000000003,"15min":0.69599999999999995},"last":0.379},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.432,"15min":0.45100000000000001},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.76600000000000001},"last":0.34899999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.40799999999999997,"15min":0.432},"min":{"1min":0.29699999999999999,"5min":0.28599999999999998,"15min":0.26200000000000001},"max":{"1min":0.47299999999999998,"5min":0.748,"15min":0.95799999999999996},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46899999999999997,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.57199999999999995,"5min":0.67400000000000004,"15min":0.79000000000000004},"last":0.438}]},{"osd":3,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.501,"15min":0.52600000000000002},"min":{"1min":0.35999999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59099999999999997,"5min":0.73699999999999999,"15min":1.2130000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.48599999999999999,"15min":0.51100000000000001},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.32300000000000001},"max":{"1min":0.628,"5min":0.71799999999999997,"15min":1.22},"last":0.40400000000000003}]},{"osd":4,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.65200000000000002,"15min":0.66600000000000004},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78600000000000003,"5min":0.998,"15min":1.242},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.66700000000000004,"15min":0.68899999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.35699999999999998},"max":{"1min":0.93600000000000005,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.56699999999999995}]},{"osd":5,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.70999999999999996,"15min":0.70299999999999996},"min":{"1min":0.42299999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81299999999999994,"5min":5.444,"15min":5.444},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.69299999999999995,"15min":0.69999999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82899999999999996,"5min":2.6669999999999998,"15min":2.6669999999999998},"last":0.54800000000000004}]},{"osd":6,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.72199999999999998,"15min":0.71899999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47099999999999997},"max":{"1min":0.85499999999999998,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.79100000000000004,"15min":0.73899999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48499999999999999},"max":{"1min":0.82199999999999995,"5min":11.223000000000001,"15min":11.223000000000001},"last":0.45700000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.72799999999999998,"15min":0.748},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97799999999999998,"5min":2.9660000000000002,"15min":2.9660000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.751,"15min":0.76300000000000001},"min":{"1min":0.55400000000000005,"5min":0.53400000000000003,"15min":0.52000000000000002},"max":{"1min":0.83199999999999996,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.58199999999999996}]}]},{"osd":0,"up_from":13,"seq":55834576025,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309676,"kb_used_data":4196,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062164,"statfs":{"total":96636764160,"available":96319655936,"internally_reserved":0,"allocated":4296704,"data_stored":2713001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.42499999999999999,"15min":0.42199999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.53600000000000003,"5min":1.276,"15min":1.276},"last":0.373},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.46100000000000002},"min":{"1min":0.29999999999999999,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.44}]},{"osd":2,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.29199999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.443,"5min":0.46400000000000002,"15min":0.48199999999999998},"min":{"1min":0.32500000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.52700000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48399999999999999,"15min":0.48499999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65800000000000003,"5min":0.80000000000000004,"15min":0.83699999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.498,"15min":0.503},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.65100000000000002,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.61099999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.42199999999999999},"max":{"1min":0.90200000000000002,"5min":0.91200000000000003,"15min":1.0720000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.64600000000000002,"15min":0.66800000000000004},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.81299999999999994,"5min":1.002,"15min":1.002},"last":0.56000000000000005}]},{"osd":5,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.67600000000000005,"15min":0.69199999999999995},"min":{"1min":0.47699999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.92400000000000004,"5min":1.0209999999999999,"15min":2.165},"last":0.629},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.93700000000000006,"5min":1.0700000000000001,"15min":1.1399999999999999},"last":0.50800000000000001}]},{"osd":6,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67900000000000005,"15min":0.70399999999999996},"min":{"1min":0.46800000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82799999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70699999999999996,"15min":0.73199999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.425},"last":0.68200000000000005}]},{"osd":7,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.71899999999999997,"15min":0.745},"min":{"1min":0.48199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.98899999999999999,"5min":1.0169999999999999,"15min":1.1259999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.747,"15min":0.76100000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.96299999999999997,"5min":1.042,"15min":1.1910000000000001},"last":0.54600000000000004}]}]},{"osd":2,"up_from":13,"seq":55834576025,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312852,"kb_used_data":3276,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058988,"statfs":{"total":96636764160,"available":96316403712,"internally_reserved":0,"allocated":3354624,"data_stored":1787011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38500000000000001,"15min":0.40200000000000002},"min":{"1min":0.29299999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.56200000000000006,"5min":0.60199999999999998,"15min":0.746},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.42299999999999999,"15min":0.44},"min":{"1min":0.30399999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.48599999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.48099999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.435,"15min":0.443},"min":{"1min":0.316,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":0.52400000000000002,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.497,"15min":0.51600000000000001},"min":{"1min":0.36799999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.65000000000000002,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.54200000000000004}]},{"osd":3,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.47899999999999998,"15min":0.49099999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58599999999999997,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.46200000000000002,"15min":0.48099999999999998},"min":{"1min":0.30499999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56699999999999995,"5min":0.63900000000000001,"15min":0.68999999999999995},"last":0.5}]},{"osd":4,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.65300000000000002,"15min":0.66700000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89700000000000002,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.66200000000000003,"15min":0.68200000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.378},"max":{"1min":0.77200000000000002,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.58899999999999997}]},{"osd":5,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.68000000000000005,"15min":0.69499999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.876,"5min":1.075,"15min":1.075},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.67700000000000005,"15min":0.69799999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.39800000000000002},"max":{"1min":0.79600000000000004,"5min":0.97499999999999998,"15min":1.0229999999999999},"last":0.63900000000000001}]},{"osd":6,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.68000000000000005,"15min":0.70199999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.73699999999999999,"5min":0.97899999999999998,"15min":1.0549999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67300000000000004,"15min":0.70599999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.39200000000000002},"max":{"1min":0.83199999999999996,"5min":0.91200000000000003,"15min":1.0269999999999999},"last":0.72899999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.71199999999999997,"15min":0.73599999999999999},"min":{"1min":0.48899999999999999,"5min":0.442,"15min":0.442},"max":{"1min":0.77600000000000002,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.623},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.72499999999999998,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86099999999999999,"5min":0.94599999999999995,"15min":1.0880000000000001},"last":0.59999999999999998}]}]},{"osd":3,"up_from":13,"seq":55834576024,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305016,"kb_used_data":3632,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94066824,"statfs":{"total":96636764160,"available":96324427776,"internally_reserved":0,"allocated":3719168,"data_stored":2130645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.436,"15min":0.41899999999999998},"min":{"1min":0.27400000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.45000000000000001,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.371},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.44900000000000001,"15min":0.45400000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.28299999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.34999999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.60999999999999999,"15min":0.52900000000000003},"min":{"1min":0.314,"5min":0.30099999999999999,"15min":0.28899999999999998},"max":{"1min":0.754,"5min":15.391,"15min":15.391},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.57099999999999995,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.46500000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52800000000000002,"15min":0.51100000000000001},"min":{"1min":0.36699999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64800000000000002,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.53600000000000003,"15min":0.50900000000000001},"min":{"1min":0.32900000000000001,"5min":0.308,"15min":0.29799999999999999},"max":{"1min":0.82299999999999995,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":0.45000000000000001}]},{"osd":4,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.71799999999999997,"15min":0.70999999999999996},"min":{"1min":0.46600000000000003,"5min":0.46000000000000002,"15min":0.439},"max":{"1min":0.76400000000000001,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.70199999999999996,"15min":0.69599999999999995},"min":{"1min":0.442,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":0.76900000000000002,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.51100000000000001}]},{"osd":5,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.71599999999999997,"15min":0.70299999999999996},"min":{"1min":0.47099999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76100000000000001,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.621},{"interface":"front","average":{"1min":0.628,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76600000000000001,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.61099999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.72799999999999998,"15min":0.71399999999999997},"min":{"1min":0.54100000000000004,"5min":0.49199999999999999,"15min":0.47499999999999998},"max":{"1min":0.83499999999999996,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.71399999999999997,"15min":0.72699999999999998},"min":{"1min":0.46300000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81200000000000006,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.67600000000000005}]},{"osd":7,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.73899999999999999,"15min":0.72699999999999998},"min":{"1min":0.52500000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1779999999999999,"5min":4.569,"15min":4.569},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.77100000000000002,"15min":0.77200000000000002},"min":{"1min":0.56499999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.80700000000000005,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.69499999999999995}]}]},{"osd":4,"up_from":13,"seq":55834576026,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316976,"kb_used_data":3304,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94054864,"statfs":{"total":96636764160,"available":96312180736,"internally_reserved":0,"allocated":3383296,"data_stored":1796362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68500000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59699999999999998,"15min":0.60899999999999999},"min":{"1min":0.35499999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.85699999999999998},"last":0.621}]},{"osd":1,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.59899999999999998,"15min":0.622},"min":{"1min":0.41899999999999998,"5min":0.39100000000000001,"15min":0.38800000000000001},"max":{"1min":0.80700000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60999999999999999,"15min":0.62},"min":{"1min":0.38800000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79100000000000004,"5min":0.874,"15min":0.874},"last":0.56299999999999994}]},{"osd":2,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.34200000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80500000000000005,"5min":0.85599999999999998,"15min":0.88700000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.626,"15min":0.63300000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.80300000000000005,"5min":0.81499999999999995,"15min":0.89200000000000002},"last":0.501}]},{"osd":3,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59699999999999998,"15min":0.61699999999999999},"min":{"1min":0.377,"5min":0.373,"15min":0.373},"max":{"1min":0.747,"5min":0.83299999999999996,"15min":0.84199999999999997},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61099999999999999,"15min":0.625},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.52000000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.49299999999999999,"15min":0.48599999999999999},"min":{"1min":0.35599999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.85899999999999999},"last":0.379},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.30399999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.878},"last":0.42899999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.50800000000000001,"15min":0.5},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72599999999999998,"5min":0.82499999999999996,"15min":0.88700000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35199999999999998,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.79400000000000004,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.45700000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.57199999999999995,"15min":0.56499999999999995},"min":{"1min":0.33800000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.90200000000000002},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.42199999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.76800000000000002,"5min":0.85399999999999998,"15min":0.92600000000000005},"last":0.46999999999999997}]}]},{"osd":5,"up_from":13,"seq":55834576026,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312884,"kb_used_data":3308,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058956,"statfs":{"total":96636764160,"available":96316370944,"internally_reserved":0,"allocated":3387392,"data_stored":1802377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.56699999999999995,"15min":0.59099999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.27300000000000002},"max":{"1min":0.73599999999999999,"5min":0.82799999999999996,"15min":1.006},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55500000000000005,"15min":0.59099999999999997},"min":{"1min":0.38600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.80000000000000004,"5min":0.81499999999999995,"15min":1.1719999999999999},"last":0.39000000000000001}]},{"osd":1,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59499999999999997,"15min":0.63500000000000001},"min":{"1min":0.42399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.752,"5min":0.89000000000000001,"15min":0.95899999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.63100000000000001,"15min":0.65600000000000003},"min":{"1min":0.434,"5min":0.434,"15min":0.377},"max":{"1min":0.68899999999999995,"5min":0.86899999999999999,"15min":1.212},"last":0.441}]},{"osd":2,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54800000000000004,"15min":0.58099999999999996},"min":{"1min":0.35399999999999998,"5min":0.34699999999999998,"15min":0.33700000000000002},"max":{"1min":0.68200000000000005,"5min":0.82099999999999995,"15min":1.631},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.57699999999999996,"15min":0.60399999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.34000000000000002},"max":{"1min":0.65600000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.42799999999999999}]},{"osd":3,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.58999999999999997},"min":{"1min":0.40300000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.63300000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.58399999999999996,"15min":0.61599999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73399999999999999,"5min":0.879,"15min":1.4690000000000001},"last":0.41399999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.47899999999999998,"15min":0.48299999999999998},"min":{"1min":0.28899999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.60399999999999998,"5min":0.90200000000000002,"15min":0.94399999999999995},"last":0.377},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.71299999999999997,"5min":0.92500000000000004,"15min":0.93300000000000005},"last":0.505}]},{"osd":6,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.48799999999999999,"15min":0.5},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64100000000000001,"5min":0.96799999999999997,"15min":1.732},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54600000000000004,"15min":0.54500000000000004},"min":{"1min":0.32400000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66800000000000004,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.47599999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.66200000000000003,"5min":0.95399999999999996,"15min":1.0740000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.59499999999999997,"15min":0.60799999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67800000000000005,"5min":0.996,"15min":1.6919999999999999},"last":0.45300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:08:52.834 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:08:52.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.966+0000 7f20a4811700 1 Processor -- start 2024-01-27T01:08:52.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.966+0000 7f20a4811700 1 -- start start 2024-01-27T01:08:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.967+0000 7f20a4811700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f20a0159980 0x7f20a00af170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.967+0000 7f20a4811700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a00ad080 con 0x7f20a00adad0 2024-01-27T01:08:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.967+0000 7f20a4811700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a00ad640 con 0x7f20a0159d50 2024-01-27T01:08:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.967+0000 7f20a4811700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a00abeb0 con 0x7f20a0159980 2024-01-27T01:08:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.967+0000 7f209ed9d700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f20a00adad0 0x7f20a00ab7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60976/0 (socket says 172.21.15.46:60976) 2024-01-27T01:08:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.967+0000 7f209ed9d700 1 -- 172.21.15.46:0/2725151309 learned_addr learned my addr 172.21.15.46:0/2725151309 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.967+0000 7f209e59c700 1 --2- 172.21.15.46:0/2725151309 >> v2:172.21.15.106:3300/0 conn(0x7f20a0159980 0x7f20a00af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.967+0000 7f209d59a700 1 -- 172.21.15.46:0/2725151309 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314155840 0 0) 0x7f20a00ad080 con 0x7f20a00adad0 2024-01-27T01:08:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.967+0000 7f209d59a700 1 -- 172.21.15.46:0/2725151309 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2088003570 con 0x7f20a00adad0 2024-01-27T01:08:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.967+0000 7f209d59a700 1 -- 172.21.15.46:0/2725151309 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614684471 0 0) 0x7f20a00ad640 con 0x7f20a0159d50 2024-01-27T01:08:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.968+0000 7f209d59a700 1 -- 172.21.15.46:0/2725151309 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20a00ad080 con 0x7f20a0159d50 2024-01-27T01:08:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.968+0000 7f209d59a700 1 -- 172.21.15.46:0/2725151309 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 569979250 0 0) 0x7f2088003570 con 0x7f20a00adad0 2024-01-27T01:08:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.968+0000 7f209e59c700 1 -- 172.21.15.46:0/2725151309 >> v1:172.21.15.106:6789/0 conn(0x7f20a0159d50 legacy=0x7f20a00ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.968+0000 7f209e59c700 1 -- 172.21.15.46:0/2725151309 >> v1:172.21.15.46:6789/0 conn(0x7f20a00adad0 legacy=0x7f20a00ab7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.968+0000 7f209e59c700 1 -- 172.21.15.46:0/2725151309 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a00cdfa0 con 0x7f20a0159980 2024-01-27T01:08:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.968+0000 7f209e59c700 1 --2- 172.21.15.46:0/2725151309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20a0159980 0x7f20a00af170 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f20900098d0 tx=0x7f2090009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bda4a26410c8772 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.968+0000 7f209d59a700 1 -- 172.21.15.46:0/2725151309 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20980017f0 con 0x7f20a0159980 2024-01-27T01:08:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.968+0000 7f209d59a700 1 -- 172.21.15.46:0/2725151309 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2090011420 con 0x7f20a0159980 2024-01-27T01:08:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.968+0000 7f209d59a700 1 -- 172.21.15.46:0/2725151309 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20900117c0 con 0x7f20a0159980 2024-01-27T01:08:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.969+0000 7f20a4811700 1 -- 172.21.15.46:0/2725151309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20a0159980 msgr2=0x7f20a00af170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.969+0000 7f20a4811700 1 --2- 172.21.15.46:0/2725151309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20a0159980 0x7f20a00af170 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f20900098d0 tx=0x7f2090009120 comp rx=0 tx=0).stop 2024-01-27T01:08:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.969+0000 7f20a4811700 1 -- 172.21.15.46:0/2725151309 shutdown_connections 2024-01-27T01:08:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.969+0000 7f20a4811700 1 --2- 172.21.15.46:0/2725151309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20a0159980 0x7f20a00af170 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.969+0000 7f20a4811700 1 -- 172.21.15.46:0/2725151309 >> 172.21.15.46:0/2725151309 conn(0x7f20a00692a0 msgr2=0x7f20a00cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.969+0000 7f20a4811700 1 -- 172.21.15.46:0/2725151309 shutdown_connections 2024-01-27T01:08:52.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.969+0000 7f20a4811700 1 -- 172.21.15.46:0/2725151309 wait complete. 2024-01-27T01:08:52.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.970+0000 7f20a4811700 1 Processor -- start 2024-01-27T01:08:52.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.970+0000 7f20a4811700 1 -- start start 2024-01-27T01:08:52.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.971+0000 7f20a4811700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a00adad0 0x7f20a0154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:52.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.971+0000 7f20a4811700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20a0159980 0x7f20a015f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:52.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.971+0000 7f20a4811700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20a0159d50 0x7f20a015a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:52.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.971+0000 7f20a4811700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a01532a0 con 0x7f20a00adad0 2024-01-27T01:08:52.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.971+0000 7f20a4811700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a0154100 con 0x7f20a0159d50 2024-01-27T01:08:52.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.971+0000 7f20a4811700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a015afc0 con 0x7f20a0159980 2024-01-27T01:08:52.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.971+0000 7f209e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a00adad0 0x7f20a0154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:52.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.971+0000 7f209e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a00adad0 0x7f20a0154590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55356/0 (socket says 172.21.15.46:55356) 2024-01-27T01:08:52.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.971+0000 7f209e59c700 1 -- 172.21.15.46:0/668100066 learned_addr learned my addr 172.21.15.46:0/668100066 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:52.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.971+0000 7f209ed9d700 1 --2- 172.21.15.46:0/668100066 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20a0159d50 0x7f20a015a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:52.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.971+0000 7f209dd9b700 1 --2- 172.21.15.46:0/668100066 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20a0159980 0x7f20a015f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:52.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.971+0000 7f209e59c700 1 -- 172.21.15.46:0/668100066 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20a0159980 msgr2=0x7f20a015f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.971+0000 7f209e59c700 1 --2- 172.21.15.46:0/668100066 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20a0159980 0x7f20a015f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.971+0000 7f209e59c700 1 -- 172.21.15.46:0/668100066 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20a0159d50 msgr2=0x7f20a015a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:52.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.971+0000 7f209e59c700 1 --2- 172.21.15.46:0/668100066 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20a0159d50 0x7f20a015a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:52.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.971+0000 7f209e59c700 1 -- 172.21.15.46:0/668100066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20900093c0 con 0x7f20a00adad0 2024-01-27T01:08:52.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.971+0000 7f209dd9b700 1 --2- 172.21.15.46:0/668100066 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20a0159980 0x7f20a015f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:52.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.972+0000 7f209e59c700 1 --2- 172.21.15.46:0/668100066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a00adad0 0x7f20a0154590 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f2090009390 tx=0x7f2090016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:52.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.972+0000 7f208e7fc700 1 -- 172.21.15.46:0/668100066 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209001cc20 con 0x7f20a00adad0 2024-01-27T01:08:52.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.972+0000 7f208e7fc700 1 -- 172.21.15.46:0/668100066 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209001cda0 con 0x7f20a00adad0 2024-01-27T01:08:52.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.972+0000 7f20a4811700 1 -- 172.21.15.46:0/668100066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a01534f0 con 0x7f20a00adad0 2024-01-27T01:08:52.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.972+0000 7f208e7fc700 1 -- 172.21.15.46:0/668100066 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209001a6a0 con 0x7f20a00adad0 2024-01-27T01:08:52.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.972+0000 7f20a4811700 1 -- 172.21.15.46:0/668100066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a0153ab0 con 0x7f20a00adad0 2024-01-27T01:08:52.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.973+0000 7f20a4811700 1 -- 172.21.15.46:0/668100066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a004c710 con 0x7f20a00adad0 2024-01-27T01:08:52.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.973+0000 7f208e7fc700 1 -- 172.21.15.46:0/668100066 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f209001aa50 con 0x7f20a00adad0 2024-01-27T01:08:52.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.974+0000 7f208e7fc700 1 --2- 172.21.15.46:0/668100066 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2080066f30 0x7f20800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:52.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.974+0000 7f208e7fc700 1 -- 172.21.15.46:0/668100066 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f2090086530 con 0x7f20a00adad0 2024-01-27T01:08:52.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.974+0000 7f209dd9b700 1 --2- 172.21.15.46:0/668100066 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2080066f30 0x7f20800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:52.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.974+0000 7f209dd9b700 1 --2- 172.21.15.46:0/668100066 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2080066f30 0x7f20800693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f20940038f0 tx=0x7f2094009040 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:52.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:52.979+0000 7f208e7fc700 1 -- 172.21.15.46:0/668100066 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2090053470 con 0x7f20a00adad0 2024-01-27T01:08:53.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.179+0000 7f20a4811700 1 -- 172.21.15.46:0/668100066 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f20a00ce280 con 0x7f2080066f30 2024-01-27T01:08:53.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.181+0000 7f208e7fc700 1 -- 172.21.15.46:0/668100066 <== mgr.10565 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51218 (secure 0 0 0) 0x7f20a00ce280 con 0x7f2080066f30 2024-01-27T01:08:53.182 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:53.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.184+0000 7f20a4811700 1 -- 172.21.15.46:0/668100066 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2080066f30 msgr2=0x7f20800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:53.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.184+0000 7f20a4811700 1 --2- 172.21.15.46:0/668100066 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2080066f30 0x7f20800693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f20940038f0 tx=0x7f2094009040 comp rx=0 tx=0).stop 2024-01-27T01:08:53.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.184+0000 7f20a4811700 1 -- 172.21.15.46:0/668100066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a00adad0 msgr2=0x7f20a0154590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:53.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.184+0000 7f20a4811700 1 --2- 172.21.15.46:0/668100066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a00adad0 0x7f20a0154590 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f2090009390 tx=0x7f2090016000 comp rx=0 tx=0).stop 2024-01-27T01:08:53.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.184+0000 7f20a4811700 1 -- 172.21.15.46:0/668100066 shutdown_connections 2024-01-27T01:08:53.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.184+0000 7f20a4811700 1 --2- 172.21.15.46:0/668100066 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2080066f30 0x7f20800693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.184+0000 7f20a4811700 1 --2- 172.21.15.46:0/668100066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20a00adad0 0x7f20a0154590 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.184+0000 7f20a4811700 1 --2- 172.21.15.46:0/668100066 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20a0159d50 0x7f20a015a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.184+0000 7f20a4811700 1 --2- 172.21.15.46:0/668100066 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20a0159980 0x7f20a015f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.184+0000 7f20a4811700 1 -- 172.21.15.46:0/668100066 >> 172.21.15.46:0/668100066 conn(0x7f20a00692a0 msgr2=0x7f20a00aafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:53.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.184+0000 7f20a4811700 1 -- 172.21.15.46:0/668100066 shutdown_connections 2024-01-27T01:08:53.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.184+0000 7f20a4811700 1 -- 172.21.15.46:0/668100066 wait complete. 2024-01-27T01:08:53.185 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:08:53.197 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-01-27T01:08:52.492281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2719,"num_read_kb":5760,"num_write":720,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8930,"ondisk_log_size":8930,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2488844,"kb_used_data":28560,"kb_used_omap":0,"kb_used_meta":2460160,"kb_avail":752485876,"statfs":{"total":773094113280,"available":770545537024,"internally_reserved":0,"allocated":29245440,"data_stored":16564239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2519203840},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-665943,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-280,"num_write_kb":-2051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.996864"},"pg_stats":[{"pgid":"1.0","version":"265'11159","reported_seq":12260,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-01-27T01:08:37.481193+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:08:37.481193+0000","last_peered":"2024-01-27T01:08:37.481193+0000","last_clean":"2024-01-27T01:08:37.481193+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:08:37.481193+0000","last_undegraded":"2024-01-27T01:08:37.481193+0000","last_fullsized":"2024-01-27T01:08:37.481193+0000","mapping_epoch":14,"log_start":"98'2229","ondisk_log_start":"98'2229","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8930,"log_dups_size":0,"ondisk_log_size":8930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2719,"num_read_kb":5760,"num_write":720,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2719,"num_read_kb":5760,"num_write":720,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8930,"ondisk_log_size":8930,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576026,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313844,"kb_used_data":4208,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94057996,"statfs":{"total":96636764160,"available":96315387904,"internally_reserved":0,"allocated":4308992,"data_stored":2715395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.58199999999999996,"15min":0.60199999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.312},"max":{"1min":0.745,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61699999999999999,"15min":0.61799999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33000000000000002},"max":{"1min":0.76400000000000001,"5min":0.879,"15min":0.879},"last":0.57099999999999995}]},{"osd":1,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.58599999999999997,"15min":0.60399999999999998},"min":{"1min":0.40300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73799999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.432},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.34300000000000003},"max":{"1min":0.81499999999999995,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.51000000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.59499999999999997,"15min":0.60399999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.753,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.39200000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85799999999999998,"5min":0.872,"15min":0.88900000000000001},"last":0.53000000000000003}]},{"osd":3,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58799999999999997,"15min":0.60799999999999998},"min":{"1min":0.374,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79100000000000004,"5min":0.85399999999999998,"15min":0.876},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64300000000000002,"15min":0.64000000000000001},"min":{"1min":0.40600000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.80600000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.49199999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.496,"15min":0.48699999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76100000000000001,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.52600000000000002,"15min":0.51500000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67800000000000005,"5min":0.80300000000000005,"15min":0.86599999999999999},"last":0.42199999999999999}]},{"osd":5,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.54100000000000004,"15min":0.52400000000000002},"min":{"1min":0.33600000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.81399999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.54300000000000004}]},{"osd":6,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59299999999999997,"15min":0.58399999999999996},"min":{"1min":0.33700000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84499999999999997,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58799999999999997,"15min":0.58199999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77900000000000003,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.501}]}]},{"osd":6,"up_from":13,"seq":55834576027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312908,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058932,"statfs":{"total":96636764160,"available":96316346368,"internally_reserved":0,"allocated":3411968,"data_stored":1817037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.56499999999999995,"15min":0.59199999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.83499999999999996,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.57499999999999996,"15min":0.60799999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73499999999999999,"5min":0.82499999999999996,"15min":0.82899999999999996},"last":0.42299999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.55800000000000005,"15min":0.59599999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70399999999999996,"5min":0.80300000000000005,"15min":0.84499999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57899999999999996,"15min":0.61099999999999999},"min":{"1min":0.39600000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76400000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.45000000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.55300000000000005,"15min":0.60099999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78100000000000003,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.57699999999999996,"15min":0.61099999999999999},"min":{"1min":0.379,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.754,"5min":0.80400000000000005,"15min":0.878},"last":0.49299999999999999}]},{"osd":3,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.58699999999999997,"15min":0.61499999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80600000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.62,"15min":0.64900000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.877,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.503}]},{"osd":4,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.495,"15min":0.497},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.80300000000000005},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.441,"5min":0.51600000000000001,"15min":0.53100000000000003},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.46800000000000003}]},{"osd":5,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.50600000000000001,"15min":0.51700000000000002},"min":{"1min":0.309,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70899999999999996,"5min":0.73999999999999999,"15min":0.79300000000000004},"last":0.437},{"interface":"front","average":{"1min":0.504,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.28899999999999998},"max":{"1min":0.752,"5min":0.87,"15min":0.87},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56699999999999995,"15min":0.56999999999999995},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.57499999999999996,"15min":0.58499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.79300000000000004,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.56000000000000005}]}]},{"osd":1,"up_from":13,"seq":55834576026,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304688,"kb_used_data":3304,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067152,"statfs":{"total":96636764160,"available":96324763648,"internally_reserved":0,"allocated":3383296,"data_stored":1802411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.28299999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.502,"5min":0.66100000000000003,"15min":0.69599999999999995},"last":0.379},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.432,"15min":0.45100000000000001},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.76600000000000001},"last":0.34899999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.40799999999999997,"15min":0.432},"min":{"1min":0.29699999999999999,"5min":0.28599999999999998,"15min":0.26200000000000001},"max":{"1min":0.47299999999999998,"5min":0.748,"15min":0.95799999999999996},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46899999999999997,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.57199999999999995,"5min":0.67400000000000004,"15min":0.79000000000000004},"last":0.438}]},{"osd":3,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.501,"15min":0.52600000000000002},"min":{"1min":0.35999999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59099999999999997,"5min":0.73699999999999999,"15min":1.2130000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.48599999999999999,"15min":0.51100000000000001},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.32300000000000001},"max":{"1min":0.628,"5min":0.71799999999999997,"15min":1.22},"last":0.40400000000000003}]},{"osd":4,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.65200000000000002,"15min":0.66600000000000004},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78600000000000003,"5min":0.998,"15min":1.242},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.66700000000000004,"15min":0.68899999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.35699999999999998},"max":{"1min":0.93600000000000005,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.56699999999999995}]},{"osd":5,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.70999999999999996,"15min":0.70299999999999996},"min":{"1min":0.42299999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81299999999999994,"5min":5.444,"15min":5.444},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.69299999999999995,"15min":0.69999999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82899999999999996,"5min":2.6669999999999998,"15min":2.6669999999999998},"last":0.54800000000000004}]},{"osd":6,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.72199999999999998,"15min":0.71899999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47099999999999997},"max":{"1min":0.85499999999999998,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.79100000000000004,"15min":0.73899999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48499999999999999},"max":{"1min":0.82199999999999995,"5min":11.223000000000001,"15min":11.223000000000001},"last":0.45700000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.72799999999999998,"15min":0.748},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97799999999999998,"5min":2.9660000000000002,"15min":2.9660000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.751,"15min":0.76300000000000001},"min":{"1min":0.55400000000000005,"5min":0.53400000000000003,"15min":0.52000000000000002},"max":{"1min":0.83199999999999996,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.58199999999999996}]}]},{"osd":0,"up_from":13,"seq":55834576025,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309676,"kb_used_data":4196,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062164,"statfs":{"total":96636764160,"available":96319655936,"internally_reserved":0,"allocated":4296704,"data_stored":2713001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.42499999999999999,"15min":0.42199999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.53600000000000003,"5min":1.276,"15min":1.276},"last":0.373},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.46100000000000002},"min":{"1min":0.29999999999999999,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.44}]},{"osd":2,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.29199999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.443,"5min":0.46400000000000002,"15min":0.48199999999999998},"min":{"1min":0.32500000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.52700000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48399999999999999,"15min":0.48499999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65800000000000003,"5min":0.80000000000000004,"15min":0.83699999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.498,"15min":0.503},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.65100000000000002,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.61099999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.42199999999999999},"max":{"1min":0.90200000000000002,"5min":0.91200000000000003,"15min":1.0720000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.64600000000000002,"15min":0.66800000000000004},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.81299999999999994,"5min":1.002,"15min":1.002},"last":0.56000000000000005}]},{"osd":5,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.67600000000000005,"15min":0.69199999999999995},"min":{"1min":0.47699999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.92400000000000004,"5min":1.0209999999999999,"15min":2.165},"last":0.629},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.93700000000000006,"5min":1.0700000000000001,"15min":1.1399999999999999},"last":0.50800000000000001}]},{"osd":6,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67900000000000005,"15min":0.70399999999999996},"min":{"1min":0.46800000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82799999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70699999999999996,"15min":0.73199999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.425},"last":0.68200000000000005}]},{"osd":7,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.71899999999999997,"15min":0.745},"min":{"1min":0.48199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.98899999999999999,"5min":1.0169999999999999,"15min":1.1259999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.747,"15min":0.76100000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.96299999999999997,"5min":1.042,"15min":1.1910000000000001},"last":0.54600000000000004}]}]},{"osd":2,"up_from":13,"seq":55834576025,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312852,"kb_used_data":3276,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058988,"statfs":{"total":96636764160,"available":96316403712,"internally_reserved":0,"allocated":3354624,"data_stored":1787011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38500000000000001,"15min":0.40200000000000002},"min":{"1min":0.29299999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.56200000000000006,"5min":0.60199999999999998,"15min":0.746},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.42299999999999999,"15min":0.44},"min":{"1min":0.30399999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.48599999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.48099999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.435,"15min":0.443},"min":{"1min":0.316,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":0.52400000000000002,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.497,"15min":0.51600000000000001},"min":{"1min":0.36799999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.65000000000000002,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.54200000000000004}]},{"osd":3,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.47899999999999998,"15min":0.49099999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58599999999999997,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.46200000000000002,"15min":0.48099999999999998},"min":{"1min":0.30499999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56699999999999995,"5min":0.63900000000000001,"15min":0.68999999999999995},"last":0.5}]},{"osd":4,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.65300000000000002,"15min":0.66700000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89700000000000002,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.66200000000000003,"15min":0.68200000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.378},"max":{"1min":0.77200000000000002,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.58899999999999997}]},{"osd":5,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.68000000000000005,"15min":0.69499999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.876,"5min":1.075,"15min":1.075},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.67700000000000005,"15min":0.69799999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.39800000000000002},"max":{"1min":0.79600000000000004,"5min":0.97499999999999998,"15min":1.0229999999999999},"last":0.63900000000000001}]},{"osd":6,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.68000000000000005,"15min":0.70199999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.73699999999999999,"5min":0.97899999999999998,"15min":1.0549999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67300000000000004,"15min":0.70599999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.39200000000000002},"max":{"1min":0.83199999999999996,"5min":0.91200000000000003,"15min":1.0269999999999999},"last":0.72899999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.71199999999999997,"15min":0.73599999999999999},"min":{"1min":0.48899999999999999,"5min":0.442,"15min":0.442},"max":{"1min":0.77600000000000002,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.623},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.72499999999999998,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86099999999999999,"5min":0.94599999999999995,"15min":1.0880000000000001},"last":0.59999999999999998}]}]},{"osd":3,"up_from":13,"seq":55834576024,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305016,"kb_used_data":3632,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94066824,"statfs":{"total":96636764160,"available":96324427776,"internally_reserved":0,"allocated":3719168,"data_stored":2130645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.436,"15min":0.41899999999999998},"min":{"1min":0.27400000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.45000000000000001,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.371},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.44900000000000001,"15min":0.45400000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.28299999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.34999999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.60999999999999999,"15min":0.52900000000000003},"min":{"1min":0.314,"5min":0.30099999999999999,"15min":0.28899999999999998},"max":{"1min":0.754,"5min":15.391,"15min":15.391},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.57099999999999995,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.46500000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52800000000000002,"15min":0.51100000000000001},"min":{"1min":0.36699999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64800000000000002,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.53600000000000003,"15min":0.50900000000000001},"min":{"1min":0.32900000000000001,"5min":0.308,"15min":0.29799999999999999},"max":{"1min":0.82299999999999995,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":0.45000000000000001}]},{"osd":4,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.71799999999999997,"15min":0.70999999999999996},"min":{"1min":0.46600000000000003,"5min":0.46000000000000002,"15min":0.439},"max":{"1min":0.76400000000000001,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.70199999999999996,"15min":0.69599999999999995},"min":{"1min":0.442,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":0.76900000000000002,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.51100000000000001}]},{"osd":5,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.71599999999999997,"15min":0.70299999999999996},"min":{"1min":0.47099999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76100000000000001,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.621},{"interface":"front","average":{"1min":0.628,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76600000000000001,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.61099999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.72799999999999998,"15min":0.71399999999999997},"min":{"1min":0.54100000000000004,"5min":0.49199999999999999,"15min":0.47499999999999998},"max":{"1min":0.83499999999999996,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.71399999999999997,"15min":0.72699999999999998},"min":{"1min":0.46300000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81200000000000006,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.67600000000000005}]},{"osd":7,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.73899999999999999,"15min":0.72699999999999998},"min":{"1min":0.52500000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1779999999999999,"5min":4.569,"15min":4.569},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.77100000000000002,"15min":0.77200000000000002},"min":{"1min":0.56499999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.80700000000000005,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.69499999999999995}]}]},{"osd":4,"up_from":13,"seq":55834576026,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316976,"kb_used_data":3304,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94054864,"statfs":{"total":96636764160,"available":96312180736,"internally_reserved":0,"allocated":3383296,"data_stored":1796362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68500000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59699999999999998,"15min":0.60899999999999999},"min":{"1min":0.35499999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.85699999999999998},"last":0.621}]},{"osd":1,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.59899999999999998,"15min":0.622},"min":{"1min":0.41899999999999998,"5min":0.39100000000000001,"15min":0.38800000000000001},"max":{"1min":0.80700000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60999999999999999,"15min":0.62},"min":{"1min":0.38800000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79100000000000004,"5min":0.874,"15min":0.874},"last":0.56299999999999994}]},{"osd":2,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.34200000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80500000000000005,"5min":0.85599999999999998,"15min":0.88700000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.626,"15min":0.63300000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.80300000000000005,"5min":0.81499999999999995,"15min":0.89200000000000002},"last":0.501}]},{"osd":3,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59699999999999998,"15min":0.61699999999999999},"min":{"1min":0.377,"5min":0.373,"15min":0.373},"max":{"1min":0.747,"5min":0.83299999999999996,"15min":0.84199999999999997},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61099999999999999,"15min":0.625},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.52000000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.49299999999999999,"15min":0.48599999999999999},"min":{"1min":0.35599999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.85899999999999999},"last":0.379},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.30399999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.878},"last":0.42899999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.50800000000000001,"15min":0.5},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72599999999999998,"5min":0.82499999999999996,"15min":0.88700000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35199999999999998,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.79400000000000004,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.45700000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.57199999999999995,"15min":0.56499999999999995},"min":{"1min":0.33800000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.90200000000000002},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.42199999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.76800000000000002,"5min":0.85399999999999998,"15min":0.92600000000000005},"last":0.46999999999999997}]}]},{"osd":5,"up_from":13,"seq":55834576026,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312884,"kb_used_data":3308,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058956,"statfs":{"total":96636764160,"available":96316370944,"internally_reserved":0,"allocated":3387392,"data_stored":1802377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.56699999999999995,"15min":0.59099999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.27300000000000002},"max":{"1min":0.73599999999999999,"5min":0.82799999999999996,"15min":1.006},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55500000000000005,"15min":0.59099999999999997},"min":{"1min":0.38600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.80000000000000004,"5min":0.81499999999999995,"15min":1.1719999999999999},"last":0.39000000000000001}]},{"osd":1,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59499999999999997,"15min":0.63500000000000001},"min":{"1min":0.42399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.752,"5min":0.89000000000000001,"15min":0.95899999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.63100000000000001,"15min":0.65600000000000003},"min":{"1min":0.434,"5min":0.434,"15min":0.377},"max":{"1min":0.68899999999999995,"5min":0.86899999999999999,"15min":1.212},"last":0.441}]},{"osd":2,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54800000000000004,"15min":0.58099999999999996},"min":{"1min":0.35399999999999998,"5min":0.34699999999999998,"15min":0.33700000000000002},"max":{"1min":0.68200000000000005,"5min":0.82099999999999995,"15min":1.631},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.57699999999999996,"15min":0.60399999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.34000000000000002},"max":{"1min":0.65600000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.42799999999999999}]},{"osd":3,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.58999999999999997},"min":{"1min":0.40300000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.63300000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.58399999999999996,"15min":0.61599999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73399999999999999,"5min":0.879,"15min":1.4690000000000001},"last":0.41399999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.47899999999999998,"15min":0.48299999999999998},"min":{"1min":0.28899999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.60399999999999998,"5min":0.90200000000000002,"15min":0.94399999999999995},"last":0.377},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.71299999999999997,"5min":0.92500000000000004,"15min":0.93300000000000005},"last":0.505}]},{"osd":6,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.48799999999999999,"15min":0.5},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64100000000000001,"5min":0.96799999999999997,"15min":1.732},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54600000000000004,"15min":0.54500000000000004},"min":{"1min":0.32400000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66800000000000004,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.47599999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.66200000000000003,"5min":0.95399999999999996,"15min":1.0740000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.59499999999999997,"15min":0.60799999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67800000000000005,"5min":0.996,"15min":1.6919999999999999},"last":0.45300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:08:53.198 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-27T01:08:53.199 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:08:53.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.331+0000 7fa329492700 1 Processor -- start 2024-01-27T01:08:53.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.331+0000 7fa329492700 1 -- start start 2024-01-27T01:08:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.332+0000 7fa329492700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa3240c6820 0x7fa3240c6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.332+0000 7fa329492700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3240ce3a0 0x7fa3240ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.332+0000 7fa329492700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3240d1490 con 0x7fa3240b2c60 2024-01-27T01:08:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.332+0000 7fa329492700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3240c71b0 con 0x7fa3240c6820 2024-01-27T01:08:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.332+0000 7fa329492700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3240c6390 con 0x7fa3240ce3a0 2024-01-27T01:08:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.332+0000 7fa3227fc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa3240c6820 0x7fa3240c6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.332+0000 7fa3237fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3240ce3a0 0x7fa3240ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.332+0000 7fa322ffd700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fa3240b2c60 0x7fa3240b3590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:47810/0 (socket says 172.21.15.46:47810) 2024-01-27T01:08:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.332+0000 7fa3227fc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa3240c6820 0x7fa3240c6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:59648/0 (socket says 172.21.15.46:59648) 2024-01-27T01:08:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.332+0000 7fa322ffd700 1 -- 172.21.15.46:0/2894367145 learned_addr learned my addr 172.21.15.46:0/2894367145 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.332+0000 7fa3237fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3240ce3a0 0x7fa3240ce7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:44660/0 (socket says 172.21.15.46:44660) 2024-01-27T01:08:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.333+0000 7fa3227fc700 1 -- 172.21.15.46:0/2894367145 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3240ce3a0 msgr2=0x7fa3240ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.333+0000 7fa3227fc700 1 --2- 172.21.15.46:0/2894367145 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3240ce3a0 0x7fa3240ce7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.333+0000 7fa3227fc700 1 -- 172.21.15.46:0/2894367145 >> v1:172.21.15.106:6789/0 conn(0x7fa3240b2c60 legacy=0x7fa3240b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.333+0000 7fa3227fc700 1 -- 172.21.15.46:0/2894367145 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3240ccd80 con 0x7fa3240c6820 2024-01-27T01:08:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.333+0000 7fa321ffb700 1 -- 172.21.15.46:0/2894367145 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416031543 0 0) 0x7fa3240d1490 con 0x7fa3240b2c60 2024-01-27T01:08:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.333+0000 7fa3237fe700 1 --2- 172.21.15.46:0/2894367145 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3240ce3a0 0x7fa3240ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.333+0000 7fa3227fc700 1 --2- 172.21.15.46:0/2894367145 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3240c6820 0x7fa3240c6c70 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fa30c0098e0 tx=0x7fa30c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=998845829cf889e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.333+0000 7fa321ffb700 1 -- 172.21.15.46:0/2894367145 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c00b640 con 0x7fa3240c6820 2024-01-27T01:08:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.333+0000 7fa321ffb700 1 -- 172.21.15.46:0/2894367145 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30c00b7a0 con 0x7fa3240c6820 2024-01-27T01:08:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.333+0000 7fa321ffb700 1 -- 172.21.15.46:0/2894367145 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c00bb40 con 0x7fa3240c6820 2024-01-27T01:08:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.334+0000 7fa329492700 1 -- 172.21.15.46:0/2894367145 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3240c6820 msgr2=0x7fa3240c6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:53.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.334+0000 7fa329492700 1 --2- 172.21.15.46:0/2894367145 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3240c6820 0x7fa3240c6c70 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fa30c0098e0 tx=0x7fa30c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.334+0000 7fa329492700 1 -- 172.21.15.46:0/2894367145 shutdown_connections 2024-01-27T01:08:53.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.334+0000 7fa329492700 1 --2- 172.21.15.46:0/2894367145 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3240ce3a0 0x7fa3240ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.334+0000 7fa329492700 1 --2- 172.21.15.46:0/2894367145 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3240c6820 0x7fa3240c6c70 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.334+0000 7fa329492700 1 -- 172.21.15.46:0/2894367145 >> 172.21.15.46:0/2894367145 conn(0x7fa32406f8c0 msgr2=0x7fa3240d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:53.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.334+0000 7fa329492700 1 -- 172.21.15.46:0/2894367145 shutdown_connections 2024-01-27T01:08:53.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.334+0000 7fa329492700 1 -- 172.21.15.46:0/2894367145 wait complete. 2024-01-27T01:08:53.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.335+0000 7fa329492700 1 Processor -- start 2024-01-27T01:08:53.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.335+0000 7fa329492700 1 -- start start 2024-01-27T01:08:53.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.336+0000 7fa329492700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3240b2c60 0x7fa32415b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:53.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.336+0000 7fa329492700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3240c6820 0x7fa32415b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:53.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.336+0000 7fa329492700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3240ce3a0 0x7fa324156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:53.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.336+0000 7fa329492700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32416abb0 con 0x7fa3240c6820 2024-01-27T01:08:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.336+0000 7fa329492700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa324169fa0 con 0x7fa3240b2c60 2024-01-27T01:08:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.336+0000 7fa329492700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32416a280 con 0x7fa3240ce3a0 2024-01-27T01:08:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.336+0000 7fa3227fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3240c6820 0x7fa32415b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.336+0000 7fa3227fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3240c6820 0x7fa32415b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55366/0 (socket says 172.21.15.46:55366) 2024-01-27T01:08:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.336+0000 7fa3227fc700 1 -- 172.21.15.46:0/3231942633 learned_addr learned my addr 172.21.15.46:0/3231942633 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.336+0000 7fa3237fe700 1 --2- 172.21.15.46:0/3231942633 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3240ce3a0 0x7fa324156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.336+0000 7fa322ffd700 1 --2- 172.21.15.46:0/3231942633 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3240b2c60 0x7fa32415b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.336+0000 7fa3227fc700 1 -- 172.21.15.46:0/3231942633 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3240ce3a0 msgr2=0x7fa324156e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.336+0000 7fa3227fc700 1 --2- 172.21.15.46:0/3231942633 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3240ce3a0 0x7fa324156e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.336+0000 7fa3227fc700 1 -- 172.21.15.46:0/3231942633 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3240b2c60 msgr2=0x7fa32415b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.336+0000 7fa3227fc700 1 --2- 172.21.15.46:0/3231942633 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3240b2c60 0x7fa32415b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.337+0000 7fa3227fc700 1 -- 172.21.15.46:0/3231942633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa31c009a20 con 0x7fa3240c6820 2024-01-27T01:08:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.337+0000 7fa3227fc700 1 --2- 172.21.15.46:0/3231942633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3240c6820 0x7fa32415b630 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7fa324088990 tx=0x7fa30c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:53.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.337+0000 7fa31affd700 1 -- 172.21.15.46:0/3231942633 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c000d10 con 0x7fa3240c6820 2024-01-27T01:08:53.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.337+0000 7fa31affd700 1 -- 172.21.15.46:0/3231942633 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30c00bb20 con 0x7fa3240c6820 2024-01-27T01:08:53.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.337+0000 7fa31affd700 1 -- 172.21.15.46:0/3231942633 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c011650 con 0x7fa3240c6820 2024-01-27T01:08:53.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.337+0000 7fa329492700 1 -- 172.21.15.46:0/3231942633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa30c009420 con 0x7fa3240c6820 2024-01-27T01:08:53.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.337+0000 7fa329492700 1 -- 172.21.15.46:0/3231942633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa32416a840 con 0x7fa3240c6820 2024-01-27T01:08:53.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.338+0000 7fa329492700 1 -- 172.21.15.46:0/3231942633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa32404c710 con 0x7fa3240c6820 2024-01-27T01:08:53.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.338+0000 7fa31affd700 1 -- 172.21.15.46:0/3231942633 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa30c0119c0 con 0x7fa3240c6820 2024-01-27T01:08:53.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.339+0000 7fa31affd700 1 --2- 172.21.15.46:0/3231942633 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa304066f30 0x7fa3040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:53.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.339+0000 7fa31affd700 1 -- 172.21.15.46:0/3231942633 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7fa30c085a30 con 0x7fa3240c6820 2024-01-27T01:08:53.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.339+0000 7fa322ffd700 1 --2- 172.21.15.46:0/3231942633 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa304066f30 0x7fa3040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:53.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.340+0000 7fa322ffd700 1 --2- 172.21.15.46:0/3231942633 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa304066f30 0x7fa3040693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa3240cd8b0 tx=0x7fa314007f50 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:53.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.344+0000 7fa31affd700 1 -- 172.21.15.46:0/3231942633 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa30c052ab0 con 0x7fa3240c6820 2024-01-27T01:08:53.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.547+0000 7fa329492700 1 -- 172.21.15.46:0/3231942633 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa3240b2a40 con 0x7fa304066f30 2024-01-27T01:08:53.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.550+0000 7fa31affd700 1 -- 172.21.15.46:0/3231942633 <== mgr.10565 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51218 (secure 0 0 0) 0x7fa3240b2a40 con 0x7fa304066f30 2024-01-27T01:08:53.550 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:53.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.552+0000 7fa329492700 1 -- 172.21.15.46:0/3231942633 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa304066f30 msgr2=0x7fa3040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:53.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.552+0000 7fa329492700 1 --2- 172.21.15.46:0/3231942633 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa304066f30 0x7fa3040693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa3240cd8b0 tx=0x7fa314007f50 comp rx=0 tx=0).stop 2024-01-27T01:08:53.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.552+0000 7fa329492700 1 -- 172.21.15.46:0/3231942633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3240c6820 msgr2=0x7fa32415b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:53.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.552+0000 7fa329492700 1 --2- 172.21.15.46:0/3231942633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3240c6820 0x7fa32415b630 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7fa324088990 tx=0x7fa30c017000 comp rx=0 tx=0).stop 2024-01-27T01:08:53.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.552+0000 7fa329492700 1 -- 172.21.15.46:0/3231942633 shutdown_connections 2024-01-27T01:08:53.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.552+0000 7fa329492700 1 --2- 172.21.15.46:0/3231942633 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fa304066f30 0x7fa3040693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.552+0000 7fa329492700 1 --2- 172.21.15.46:0/3231942633 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3240ce3a0 0x7fa324156e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.552+0000 7fa329492700 1 --2- 172.21.15.46:0/3231942633 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3240b2c60 0x7fa32415b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.552+0000 7fa329492700 1 --2- 172.21.15.46:0/3231942633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3240c6820 0x7fa32415b630 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.552+0000 7fa329492700 1 -- 172.21.15.46:0/3231942633 >> 172.21.15.46:0/3231942633 conn(0x7fa32406f8c0 msgr2=0x7fa324159230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:53.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.552+0000 7fa329492700 1 -- 172.21.15.46:0/3231942633 shutdown_connections 2024-01-27T01:08:53.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.552+0000 7fa329492700 1 -- 172.21.15.46:0/3231942633 wait complete. 2024-01-27T01:08:53.554 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:08:53.565 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-01-27T01:08:52.492281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2719,"num_read_kb":5760,"num_write":720,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8930,"ondisk_log_size":8930,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2488844,"kb_used_data":28560,"kb_used_omap":0,"kb_used_meta":2460160,"kb_avail":752485876,"statfs":{"total":773094113280,"available":770545537024,"internally_reserved":0,"allocated":29245440,"data_stored":16564239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2519203840},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-665943,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-280,"num_write_kb":-2051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.996864"},"pg_stats":[{"pgid":"1.0","version":"265'11159","reported_seq":12260,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-01-27T01:08:37.481193+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:08:37.481193+0000","last_peered":"2024-01-27T01:08:37.481193+0000","last_clean":"2024-01-27T01:08:37.481193+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:08:37.481193+0000","last_undegraded":"2024-01-27T01:08:37.481193+0000","last_fullsized":"2024-01-27T01:08:37.481193+0000","mapping_epoch":14,"log_start":"98'2229","ondisk_log_start":"98'2229","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8930,"log_dups_size":0,"ondisk_log_size":8930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2719,"num_read_kb":5760,"num_write":720,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2719,"num_read_kb":5760,"num_write":720,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8930,"ondisk_log_size":8930,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576026,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313844,"kb_used_data":4208,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94057996,"statfs":{"total":96636764160,"available":96315387904,"internally_reserved":0,"allocated":4308992,"data_stored":2715395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.58199999999999996,"15min":0.60199999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.312},"max":{"1min":0.745,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61699999999999999,"15min":0.61799999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33000000000000002},"max":{"1min":0.76400000000000001,"5min":0.879,"15min":0.879},"last":0.57099999999999995}]},{"osd":1,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.58599999999999997,"15min":0.60399999999999998},"min":{"1min":0.40300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73799999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.432},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.34300000000000003},"max":{"1min":0.81499999999999995,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.51000000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.59499999999999997,"15min":0.60399999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.753,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.39200000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85799999999999998,"5min":0.872,"15min":0.88900000000000001},"last":0.53000000000000003}]},{"osd":3,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58799999999999997,"15min":0.60799999999999998},"min":{"1min":0.374,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79100000000000004,"5min":0.85399999999999998,"15min":0.876},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64300000000000002,"15min":0.64000000000000001},"min":{"1min":0.40600000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.80600000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.49199999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.496,"15min":0.48699999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76100000000000001,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.52600000000000002,"15min":0.51500000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67800000000000005,"5min":0.80300000000000005,"15min":0.86599999999999999},"last":0.42199999999999999}]},{"osd":5,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.54100000000000004,"15min":0.52400000000000002},"min":{"1min":0.33600000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.81399999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.54300000000000004}]},{"osd":6,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59299999999999997,"15min":0.58399999999999996},"min":{"1min":0.33700000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84499999999999997,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58799999999999997,"15min":0.58199999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77900000000000003,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.501}]}]},{"osd":6,"up_from":13,"seq":55834576027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312908,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058932,"statfs":{"total":96636764160,"available":96316346368,"internally_reserved":0,"allocated":3411968,"data_stored":1817037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.56499999999999995,"15min":0.59199999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.83499999999999996,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.57499999999999996,"15min":0.60799999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73499999999999999,"5min":0.82499999999999996,"15min":0.82899999999999996},"last":0.42299999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.55800000000000005,"15min":0.59599999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70399999999999996,"5min":0.80300000000000005,"15min":0.84499999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57899999999999996,"15min":0.61099999999999999},"min":{"1min":0.39600000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76400000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.45000000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.55300000000000005,"15min":0.60099999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78100000000000003,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.57699999999999996,"15min":0.61099999999999999},"min":{"1min":0.379,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.754,"5min":0.80400000000000005,"15min":0.878},"last":0.49299999999999999}]},{"osd":3,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.58699999999999997,"15min":0.61499999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80600000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.62,"15min":0.64900000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.877,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.503}]},{"osd":4,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.495,"15min":0.497},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.80300000000000005},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.441,"5min":0.51600000000000001,"15min":0.53100000000000003},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.46800000000000003}]},{"osd":5,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.50600000000000001,"15min":0.51700000000000002},"min":{"1min":0.309,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70899999999999996,"5min":0.73999999999999999,"15min":0.79300000000000004},"last":0.437},{"interface":"front","average":{"1min":0.504,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.28899999999999998},"max":{"1min":0.752,"5min":0.87,"15min":0.87},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56699999999999995,"15min":0.56999999999999995},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.57499999999999996,"15min":0.58499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.79300000000000004,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.56000000000000005}]}]},{"osd":1,"up_from":13,"seq":55834576026,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304688,"kb_used_data":3304,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067152,"statfs":{"total":96636764160,"available":96324763648,"internally_reserved":0,"allocated":3383296,"data_stored":1802411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.28299999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.502,"5min":0.66100000000000003,"15min":0.69599999999999995},"last":0.379},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.432,"15min":0.45100000000000001},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.76600000000000001},"last":0.34899999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.40799999999999997,"15min":0.432},"min":{"1min":0.29699999999999999,"5min":0.28599999999999998,"15min":0.26200000000000001},"max":{"1min":0.47299999999999998,"5min":0.748,"15min":0.95799999999999996},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46899999999999997,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.57199999999999995,"5min":0.67400000000000004,"15min":0.79000000000000004},"last":0.438}]},{"osd":3,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.501,"15min":0.52600000000000002},"min":{"1min":0.35999999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59099999999999997,"5min":0.73699999999999999,"15min":1.2130000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.48599999999999999,"15min":0.51100000000000001},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.32300000000000001},"max":{"1min":0.628,"5min":0.71799999999999997,"15min":1.22},"last":0.40400000000000003}]},{"osd":4,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.65200000000000002,"15min":0.66600000000000004},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78600000000000003,"5min":0.998,"15min":1.242},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.66700000000000004,"15min":0.68899999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.35699999999999998},"max":{"1min":0.93600000000000005,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.56699999999999995}]},{"osd":5,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.70999999999999996,"15min":0.70299999999999996},"min":{"1min":0.42299999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81299999999999994,"5min":5.444,"15min":5.444},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.69299999999999995,"15min":0.69999999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82899999999999996,"5min":2.6669999999999998,"15min":2.6669999999999998},"last":0.54800000000000004}]},{"osd":6,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.72199999999999998,"15min":0.71899999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47099999999999997},"max":{"1min":0.85499999999999998,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.79100000000000004,"15min":0.73899999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48499999999999999},"max":{"1min":0.82199999999999995,"5min":11.223000000000001,"15min":11.223000000000001},"last":0.45700000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.72799999999999998,"15min":0.748},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97799999999999998,"5min":2.9660000000000002,"15min":2.9660000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.751,"15min":0.76300000000000001},"min":{"1min":0.55400000000000005,"5min":0.53400000000000003,"15min":0.52000000000000002},"max":{"1min":0.83199999999999996,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.58199999999999996}]}]},{"osd":0,"up_from":13,"seq":55834576025,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309676,"kb_used_data":4196,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062164,"statfs":{"total":96636764160,"available":96319655936,"internally_reserved":0,"allocated":4296704,"data_stored":2713001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.42499999999999999,"15min":0.42199999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.53600000000000003,"5min":1.276,"15min":1.276},"last":0.373},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.46100000000000002},"min":{"1min":0.29999999999999999,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.44}]},{"osd":2,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.29199999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.443,"5min":0.46400000000000002,"15min":0.48199999999999998},"min":{"1min":0.32500000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.52700000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48399999999999999,"15min":0.48499999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65800000000000003,"5min":0.80000000000000004,"15min":0.83699999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.498,"15min":0.503},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.65100000000000002,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.61099999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.42199999999999999},"max":{"1min":0.90200000000000002,"5min":0.91200000000000003,"15min":1.0720000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.64600000000000002,"15min":0.66800000000000004},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.81299999999999994,"5min":1.002,"15min":1.002},"last":0.56000000000000005}]},{"osd":5,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.67600000000000005,"15min":0.69199999999999995},"min":{"1min":0.47699999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.92400000000000004,"5min":1.0209999999999999,"15min":2.165},"last":0.629},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.93700000000000006,"5min":1.0700000000000001,"15min":1.1399999999999999},"last":0.50800000000000001}]},{"osd":6,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67900000000000005,"15min":0.70399999999999996},"min":{"1min":0.46800000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82799999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70699999999999996,"15min":0.73199999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.425},"last":0.68200000000000005}]},{"osd":7,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.71899999999999997,"15min":0.745},"min":{"1min":0.48199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.98899999999999999,"5min":1.0169999999999999,"15min":1.1259999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.747,"15min":0.76100000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.96299999999999997,"5min":1.042,"15min":1.1910000000000001},"last":0.54600000000000004}]}]},{"osd":2,"up_from":13,"seq":55834576025,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312852,"kb_used_data":3276,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058988,"statfs":{"total":96636764160,"available":96316403712,"internally_reserved":0,"allocated":3354624,"data_stored":1787011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38500000000000001,"15min":0.40200000000000002},"min":{"1min":0.29299999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.56200000000000006,"5min":0.60199999999999998,"15min":0.746},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.42299999999999999,"15min":0.44},"min":{"1min":0.30399999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.48599999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.48099999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.435,"15min":0.443},"min":{"1min":0.316,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":0.52400000000000002,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.497,"15min":0.51600000000000001},"min":{"1min":0.36799999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.65000000000000002,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.54200000000000004}]},{"osd":3,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.47899999999999998,"15min":0.49099999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58599999999999997,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.46200000000000002,"15min":0.48099999999999998},"min":{"1min":0.30499999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56699999999999995,"5min":0.63900000000000001,"15min":0.68999999999999995},"last":0.5}]},{"osd":4,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.65300000000000002,"15min":0.66700000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89700000000000002,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.66200000000000003,"15min":0.68200000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.378},"max":{"1min":0.77200000000000002,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.58899999999999997}]},{"osd":5,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.68000000000000005,"15min":0.69499999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.876,"5min":1.075,"15min":1.075},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.67700000000000005,"15min":0.69799999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.39800000000000002},"max":{"1min":0.79600000000000004,"5min":0.97499999999999998,"15min":1.0229999999999999},"last":0.63900000000000001}]},{"osd":6,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.68000000000000005,"15min":0.70199999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.73699999999999999,"5min":0.97899999999999998,"15min":1.0549999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67300000000000004,"15min":0.70599999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.39200000000000002},"max":{"1min":0.83199999999999996,"5min":0.91200000000000003,"15min":1.0269999999999999},"last":0.72899999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.71199999999999997,"15min":0.73599999999999999},"min":{"1min":0.48899999999999999,"5min":0.442,"15min":0.442},"max":{"1min":0.77600000000000002,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.623},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.72499999999999998,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86099999999999999,"5min":0.94599999999999995,"15min":1.0880000000000001},"last":0.59999999999999998}]}]},{"osd":3,"up_from":13,"seq":55834576024,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305016,"kb_used_data":3632,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94066824,"statfs":{"total":96636764160,"available":96324427776,"internally_reserved":0,"allocated":3719168,"data_stored":2130645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.436,"15min":0.41899999999999998},"min":{"1min":0.27400000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.45000000000000001,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.371},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.44900000000000001,"15min":0.45400000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.28299999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.34999999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.60999999999999999,"15min":0.52900000000000003},"min":{"1min":0.314,"5min":0.30099999999999999,"15min":0.28899999999999998},"max":{"1min":0.754,"5min":15.391,"15min":15.391},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.57099999999999995,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.46500000000000002}]},{"osd":2,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52800000000000002,"15min":0.51100000000000001},"min":{"1min":0.36699999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64800000000000002,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.53600000000000003,"15min":0.50900000000000001},"min":{"1min":0.32900000000000001,"5min":0.308,"15min":0.29799999999999999},"max":{"1min":0.82299999999999995,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":0.45000000000000001}]},{"osd":4,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.71799999999999997,"15min":0.70999999999999996},"min":{"1min":0.46600000000000003,"5min":0.46000000000000002,"15min":0.439},"max":{"1min":0.76400000000000001,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.70199999999999996,"15min":0.69599999999999995},"min":{"1min":0.442,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":0.76900000000000002,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.51100000000000001}]},{"osd":5,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.71599999999999997,"15min":0.70299999999999996},"min":{"1min":0.47099999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76100000000000001,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.621},{"interface":"front","average":{"1min":0.628,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76600000000000001,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.61099999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.72799999999999998,"15min":0.71399999999999997},"min":{"1min":0.54100000000000004,"5min":0.49199999999999999,"15min":0.47499999999999998},"max":{"1min":0.83499999999999996,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.71399999999999997,"15min":0.72699999999999998},"min":{"1min":0.46300000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81200000000000006,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.67600000000000005}]},{"osd":7,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.73899999999999999,"15min":0.72699999999999998},"min":{"1min":0.52500000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1779999999999999,"5min":4.569,"15min":4.569},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.77100000000000002,"15min":0.77200000000000002},"min":{"1min":0.56499999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.80700000000000005,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.69499999999999995}]}]},{"osd":4,"up_from":13,"seq":55834576026,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316976,"kb_used_data":3304,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94054864,"statfs":{"total":96636764160,"available":96312180736,"internally_reserved":0,"allocated":3383296,"data_stored":1796362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68500000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59699999999999998,"15min":0.60899999999999999},"min":{"1min":0.35499999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.85699999999999998},"last":0.621}]},{"osd":1,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.59899999999999998,"15min":0.622},"min":{"1min":0.41899999999999998,"5min":0.39100000000000001,"15min":0.38800000000000001},"max":{"1min":0.80700000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60999999999999999,"15min":0.62},"min":{"1min":0.38800000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79100000000000004,"5min":0.874,"15min":0.874},"last":0.56299999999999994}]},{"osd":2,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.34200000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80500000000000005,"5min":0.85599999999999998,"15min":0.88700000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.626,"15min":0.63300000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.80300000000000005,"5min":0.81499999999999995,"15min":0.89200000000000002},"last":0.501}]},{"osd":3,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59699999999999998,"15min":0.61699999999999999},"min":{"1min":0.377,"5min":0.373,"15min":0.373},"max":{"1min":0.747,"5min":0.83299999999999996,"15min":0.84199999999999997},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61099999999999999,"15min":0.625},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.52000000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.49299999999999999,"15min":0.48599999999999999},"min":{"1min":0.35599999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.85899999999999999},"last":0.379},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.30399999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.878},"last":0.42899999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.50800000000000001,"15min":0.5},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72599999999999998,"5min":0.82499999999999996,"15min":0.88700000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35199999999999998,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.79400000000000004,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.45700000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.57199999999999995,"15min":0.56499999999999995},"min":{"1min":0.33800000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.90200000000000002},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.42199999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.76800000000000002,"5min":0.85399999999999998,"15min":0.92600000000000005},"last":0.46999999999999997}]}]},{"osd":5,"up_from":13,"seq":55834576026,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312884,"kb_used_data":3308,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058956,"statfs":{"total":96636764160,"available":96316370944,"internally_reserved":0,"allocated":3387392,"data_stored":1802377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.56699999999999995,"15min":0.59099999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.27300000000000002},"max":{"1min":0.73599999999999999,"5min":0.82799999999999996,"15min":1.006},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55500000000000005,"15min":0.59099999999999997},"min":{"1min":0.38600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.80000000000000004,"5min":0.81499999999999995,"15min":1.1719999999999999},"last":0.39000000000000001}]},{"osd":1,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59499999999999997,"15min":0.63500000000000001},"min":{"1min":0.42399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.752,"5min":0.89000000000000001,"15min":0.95899999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.63100000000000001,"15min":0.65600000000000003},"min":{"1min":0.434,"5min":0.434,"15min":0.377},"max":{"1min":0.68899999999999995,"5min":0.86899999999999999,"15min":1.212},"last":0.441}]},{"osd":2,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54800000000000004,"15min":0.58099999999999996},"min":{"1min":0.35399999999999998,"5min":0.34699999999999998,"15min":0.33700000000000002},"max":{"1min":0.68200000000000005,"5min":0.82099999999999995,"15min":1.631},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.57699999999999996,"15min":0.60399999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.34000000000000002},"max":{"1min":0.65600000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.42799999999999999}]},{"osd":3,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.58999999999999997},"min":{"1min":0.40300000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.63300000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.58399999999999996,"15min":0.61599999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73399999999999999,"5min":0.879,"15min":1.4690000000000001},"last":0.41399999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.47899999999999998,"15min":0.48299999999999998},"min":{"1min":0.28899999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.60399999999999998,"5min":0.90200000000000002,"15min":0.94399999999999995},"last":0.377},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.71299999999999997,"5min":0.92500000000000004,"15min":0.93300000000000005},"last":0.505}]},{"osd":6,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.48799999999999999,"15min":0.5},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64100000000000001,"5min":0.96799999999999997,"15min":1.732},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54600000000000004,"15min":0.54500000000000004},"min":{"1min":0.32400000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66800000000000004,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.47599999999999998}]},{"osd":7,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.66200000000000003,"5min":0.95399999999999996,"15min":1.0740000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.59499999999999997,"15min":0.60799999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67800000000000005,"5min":0.996,"15min":1.6919999999999999},"last":0.45300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:08:53.567 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T01:08:53.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.698+0000 7f01b2efd700 1 Processor -- start 2024-01-27T01:08:53.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.698+0000 7f01b2efd700 1 -- start start 2024-01-27T01:08:53.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.699+0000 7f01b2efd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f01ac0cced0 0x7f01ac15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:53.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.699+0000 7f01b2efd700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ac068ac0 con 0x7f01ac0cd2a0 2024-01-27T01:08:53.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.699+0000 7f01b2efd700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ac0c9470 con 0x7f01ac0cf0b0 2024-01-27T01:08:53.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.699+0000 7f01b2efd700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ac0c4c40 con 0x7f01ac0cced0 2024-01-27T01:08:53.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.699+0000 7f01b0c99700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f01ac0cced0 0x7f01ac15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:53.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.699+0000 7f01b149a700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f01ac0cd2a0 0x7f01ac0c3370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60982/0 (socket says 172.21.15.46:60982) 2024-01-27T01:08:53.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.699+0000 7f01b149a700 1 -- 172.21.15.46:0/332238366 learned_addr learned my addr 172.21.15.46:0/332238366 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:53.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.700+0000 7f01b0c99700 1 -- 172.21.15.46:0/332238366 >> v1:172.21.15.106:6789/0 conn(0x7f01ac0cf0b0 legacy=0x7f01ac15dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:53.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.700+0000 7f01b0c99700 1 -- 172.21.15.46:0/332238366 >> v1:172.21.15.46:6789/0 conn(0x7f01ac0cd2a0 legacy=0x7f01ac0c3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:53.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.700+0000 7f01b0c99700 1 -- 172.21.15.46:0/332238366 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ac0c4960 con 0x7f01ac0cced0 2024-01-27T01:08:53.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.700+0000 7f01b0c99700 1 --2- 172.21.15.46:0/332238366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01ac0cced0 0x7f01ac15d6f0 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f019c0098e0 tx=0x7f019c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23101d5de1affcca server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:53.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.700+0000 7f01ab7fe700 1 -- 172.21.15.46:0/332238366 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c009e80 con 0x7f01ac0cced0 2024-01-27T01:08:53.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.700+0000 7f01ab7fe700 1 -- 172.21.15.46:0/332238366 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c00b860 con 0x7f01ac0cced0 2024-01-27T01:08:53.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.700+0000 7f01ab7fe700 1 -- 172.21.15.46:0/332238366 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c00bbe0 con 0x7f01ac0cced0 2024-01-27T01:08:53.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.701+0000 7f01b2efd700 1 -- 172.21.15.46:0/332238366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01ac0cced0 msgr2=0x7f01ac15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:53.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.701+0000 7f01b2efd700 1 --2- 172.21.15.46:0/332238366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01ac0cced0 0x7f01ac15d6f0 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f019c0098e0 tx=0x7f019c0091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.701+0000 7f01b2efd700 1 -- 172.21.15.46:0/332238366 shutdown_connections 2024-01-27T01:08:53.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.701+0000 7f01b2efd700 1 --2- 172.21.15.46:0/332238366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01ac0cced0 0x7f01ac15d6f0 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.701+0000 7f01b2efd700 1 -- 172.21.15.46:0/332238366 >> 172.21.15.46:0/332238366 conn(0x7f01ac068190 msgr2=0x7f01ac0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:53.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.701+0000 7f01b2efd700 1 -- 172.21.15.46:0/332238366 shutdown_connections 2024-01-27T01:08:53.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.701+0000 7f01b2efd700 1 -- 172.21.15.46:0/332238366 wait complete. 2024-01-27T01:08:53.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.702+0000 7f01b2efd700 1 Processor -- start 2024-01-27T01:08:53.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.702+0000 7f01b2efd700 1 -- start start 2024-01-27T01:08:53.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.703+0000 7f01b2efd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ac0cced0 0x7f01ac157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:53.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.703+0000 7f01b2efd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01ac0cd2a0 0x7f01ac14f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:53.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.703+0000 7f01b2efd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f01ac0cf0b0 0x7f01ac15e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:53.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.703+0000 7f01b2efd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac16e770 con 0x7f01ac0cd2a0 2024-01-27T01:08:53.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.703+0000 7f01b2efd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac1576b0 con 0x7f01ac0cced0 2024-01-27T01:08:53.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.703+0000 7f01b2efd700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac16d880 con 0x7f01ac0cf0b0 2024-01-27T01:08:53.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.703+0000 7f01abfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01ac0cd2a0 0x7f01ac14f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:53.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.703+0000 7f01b0c99700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ac0cced0 0x7f01ac157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:53.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.703+0000 7f01b149a700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f01ac0cf0b0 0x7f01ac15e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:53.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.703+0000 7f01abfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01ac0cd2a0 0x7f01ac14f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55374/0 (socket says 172.21.15.46:55374) 2024-01-27T01:08:53.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.703+0000 7f01b0c99700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ac0cced0 0x7f01ac157170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:59658/0 (socket says 172.21.15.46:59658) 2024-01-27T01:08:53.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.703+0000 7f01abfff700 1 -- 172.21.15.46:0/18268357 learned_addr learned my addr 172.21.15.46:0/18268357 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:53.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.704+0000 7f01abfff700 1 -- 172.21.15.46:0/18268357 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f01ac0cf0b0 msgr2=0x7f01ac15e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:53.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.704+0000 7f01abfff700 1 --2- 172.21.15.46:0/18268357 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f01ac0cf0b0 0x7f01ac15e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.704+0000 7f01abfff700 1 -- 172.21.15.46:0/18268357 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ac0cced0 msgr2=0x7f01ac157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:53.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.704+0000 7f01abfff700 1 --2- 172.21.15.46:0/18268357 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ac0cced0 0x7f01ac157170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.704+0000 7f01abfff700 1 -- 172.21.15.46:0/18268357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c009420 con 0x7f01ac0cd2a0 2024-01-27T01:08:53.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.704+0000 7f01abfff700 1 --2- 172.21.15.46:0/18268357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01ac0cd2a0 0x7f01ac14f390 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f019400c4e0 tx=0x7f019400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:53.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.704+0000 7f01a8ff9700 1 -- 172.21.15.46:0/18268357 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0194011590 con 0x7f01ac0cd2a0 2024-01-27T01:08:53.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.704+0000 7f01a8ff9700 1 -- 172.21.15.46:0/18268357 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0194011710 con 0x7f01ac0cd2a0 2024-01-27T01:08:53.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.704+0000 7f01b2efd700 1 -- 172.21.15.46:0/18268357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ac16db60 con 0x7f01ac0cd2a0 2024-01-27T01:08:53.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.704+0000 7f01a8ff9700 1 -- 172.21.15.46:0/18268357 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0194011ad0 con 0x7f01ac0cd2a0 2024-01-27T01:08:53.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.704+0000 7f01b2efd700 1 -- 172.21.15.46:0/18268357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ac16e1a0 con 0x7f01ac0cd2a0 2024-01-27T01:08:53.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.705+0000 7f01b2efd700 1 -- 172.21.15.46:0/18268357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ac04c710 con 0x7f01ac0cd2a0 2024-01-27T01:08:53.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.706+0000 7f01a8ff9700 1 -- 172.21.15.46:0/18268357 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f019401b3b0 con 0x7f01ac0cd2a0 2024-01-27T01:08:53.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.706+0000 7f01a8ff9700 1 --2- 172.21.15.46:0/18268357 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f018c066f30 0x7f018c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:53.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.706+0000 7f01a8ff9700 1 -- 172.21.15.46:0/18268357 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f0194083bf0 con 0x7f01ac0cd2a0 2024-01-27T01:08:53.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.706+0000 7f01b0c99700 1 --2- 172.21.15.46:0/18268357 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f018c066f30 0x7f018c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:53.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.707+0000 7f01b0c99700 1 --2- 172.21.15.46:0/18268357 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f018c066f30 0x7f018c0693e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f019c0093f0 tx=0x7f019c017000 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:53.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.711+0000 7f01a8ff9700 1 -- 172.21.15.46:0/18268357 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0194050b30 con 0x7f01ac0cd2a0 2024-01-27T01:08:53.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.916+0000 7f01b2efd700 1 -- 172.21.15.46:0/18268357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01ac088ba0 con 0x7f01ac0cd2a0 2024-01-27T01:08:53.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.917+0000 7f01a8ff9700 1 -- 172.21.15.46:0/18268357 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v269) v1 ==== 74+0+14363 (secure 0 0 0) 0x7f0194017070 con 0x7f01ac0cd2a0 2024-01-27T01:08:53.917 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:53.917 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":269,"fsid":"62f24da2-f257-4a3f-ab5a-77a1100d6f1e","created":"2024-01-26T23:29:26.673550+0000","modified":"2024-01-27T01:08:49.489481+0000","last_up_change":"2024-01-26T23:29:37.506695+0000","last_in_change":"2024-01-26T23:29:29.296944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-26T23:29:38.279409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66210791-18e9-4245-bbc8-2071c29bc2cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6819","nonce":43682},{"type":"v1","addr":"172.21.15.46:6821","nonce":43682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":43682},{"type":"v1","addr":"172.21.15.46:6824","nonce":43682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":43682},{"type":"v1","addr":"172.21.15.46:6831","nonce":43682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":43682},{"type":"v1","addr":"172.21.15.46:6828","nonce":43682}]},"public_addr":"172.21.15.46:6821/43682","cluster_addr":"172.21.15.46:6824/43682","heartbeat_back_addr":"172.21.15.46:6831/43682","heartbeat_front_addr":"172.21.15.46:6828/43682","state":["exists","up"]},{"osd":1,"uuid":"a6bdcf0b-37dd-44d3-bff9-6e7639924cd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":43676},{"type":"v1","addr":"172.21.15.46:6801","nonce":43676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":43676},{"type":"v1","addr":"172.21.15.46:6803","nonce":43676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":43676},{"type":"v1","addr":"172.21.15.46:6807","nonce":43676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":43676},{"type":"v1","addr":"172.21.15.46:6805","nonce":43676}]},"public_addr":"172.21.15.46:6801/43676","cluster_addr":"172.21.15.46:6803/43676","heartbeat_back_addr":"172.21.15.46:6807/43676","heartbeat_front_addr":"172.21.15.46:6805/43676","state":["exists","up"]},{"osd":2,"uuid":"c27617ee-7d44-4220-9fb0-c2739304afc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":43680},{"type":"v1","addr":"172.21.15.46:6809","nonce":43680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":43680},{"type":"v1","addr":"172.21.15.46:6811","nonce":43680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":43680},{"type":"v1","addr":"172.21.15.46:6816","nonce":43680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":43680},{"type":"v1","addr":"172.21.15.46:6813","nonce":43680}]},"public_addr":"172.21.15.46:6809/43680","cluster_addr":"172.21.15.46:6811/43680","heartbeat_back_addr":"172.21.15.46:6816/43680","heartbeat_front_addr":"172.21.15.46:6813/43680","state":["exists","up"]},{"osd":3,"uuid":"83b0aff3-46e9-4da0-9b6f-4e1088814137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6815","nonce":43691},{"type":"v1","addr":"172.21.15.46:6817","nonce":43691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":43691},{"type":"v1","addr":"172.21.15.46:6820","nonce":43691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":43691},{"type":"v1","addr":"172.21.15.46:6829","nonce":43691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":43691},{"type":"v1","addr":"172.21.15.46:6825","nonce":43691}]},"public_addr":"172.21.15.46:6817/43691","cluster_addr":"172.21.15.46:6820/43691","heartbeat_back_addr":"172.21.15.46:6829/43691","heartbeat_front_addr":"172.21.15.46:6825/43691","state":["exists","up"]},{"osd":4,"uuid":"372407c3-89ba-4f74-ab8a-b15a924b188d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":44050},{"type":"v1","addr":"172.21.15.106:6809","nonce":44050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":44050},{"type":"v1","addr":"172.21.15.106:6811","nonce":44050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":44050},{"type":"v1","addr":"172.21.15.106:6815","nonce":44050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":44050},{"type":"v1","addr":"172.21.15.106:6813","nonce":44050}]},"public_addr":"172.21.15.106:6809/44050","cluster_addr":"172.21.15.106:6811/44050","heartbeat_back_addr":"172.21.15.106:6815/44050","heartbeat_front_addr":"172.21.15.106:6813/44050","state":["exists","up"]},{"osd":5,"uuid":"647f327a-33e0-4b2c-a84b-d2636d18a18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":44051},{"type":"v1","addr":"172.21.15.106:6801","nonce":44051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":44051},{"type":"v1","addr":"172.21.15.106:6803","nonce":44051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":44051},{"type":"v1","addr":"172.21.15.106:6807","nonce":44051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":44051},{"type":"v1","addr":"172.21.15.106:6805","nonce":44051}]},"public_addr":"172.21.15.106:6801/44051","cluster_addr":"172.21.15.106:6803/44051","heartbeat_back_addr":"172.21.15.106:6807/44051","heartbeat_front_addr":"172.21.15.106:6805/44051","state":["exists","up"]},{"osd":6,"uuid":"d11ca954-7718-40e9-8579-083963a480f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":44080},{"type":"v1","addr":"172.21.15.106:6817","nonce":44080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":44080},{"type":"v1","addr":"172.21.15.106:6819","nonce":44080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":44080},{"type":"v1","addr":"172.21.15.106:6823","nonce":44080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":44080},{"type":"v1","addr":"172.21.15.106:6821","nonce":44080}]},"public_addr":"172.21.15.106:6817/44080","cluster_addr":"172.21.15.106:6819/44080","heartbeat_back_addr":"172.21.15.106:6823/44080","heartbeat_front_addr":"172.21.15.106:6821/44080","state":["exists","up"]},{"osd":7,"uuid":"dbd8c1e0-cbb4-435b-8fb9-6a2baa2be724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":44081},{"type":"v1","addr":"172.21.15.106:6825","nonce":44081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":44081},{"type":"v1","addr":"172.21.15.106:6827","nonce":44081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":44081},{"type":"v1","addr":"172.21.15.106:6831","nonce":44081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":44081},{"type":"v1","addr":"172.21.15.106:6829","nonce":44081}]},"public_addr":"172.21.15.106:6825/44081","cluster_addr":"172.21.15.106:6827/44081","heartbeat_back_addr":"172.21.15.106:6831/44081","heartbeat_front_addr":"172.21.15.106:6829/44081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.493408+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.355178+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.520024+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.500790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.487946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.535795+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.539529+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-26T23:29:34.497237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/1756359338":"2024-01-28T01:08:46.477249+0000","172.21.15.106:0/1457405865":"2024-01-28T01:08:46.477249+0000","172.21.15.106:0/1421527645":"2024-01-28T01:08:46.477249+0000","172.21.15.106:0/1106084814":"2024-01-28T01:08:46.477249+0000","172.21.15.46:6833/3798781145":"2024-01-28T01:08:46.188351+0000","172.21.15.46:6832/3798781145":"2024-01-28T01:08:46.188351+0000","172.21.15.106:6837/62434":"2024-01-28T01:08:35.384966+0000","172.21.15.106:6836/62434":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3497341404":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3894462583":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/1144438896":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3124086304":"2024-01-28T01:08:46.477249+0000","172.21.15.106:0/3166472261":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/1405196355":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/1502373513":"2024-01-28T01:08:46.477249+0000","172.21.15.106:0/1752514247":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/2487246395":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/860101723":"2024-01-28T01:08:46.477249+0000","172.21.15.106:0/3477459747":"2024-01-28T01:08:35.384966+0000","172.21.15.106:0/3540935782":"2024-01-27T02:08:26.076460+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":8,"length":1},{"begin":14,"length":1},{"begin":19,"length":1},{"begin":24,"length":1},{"begin":29,"length":1},{"begin":35,"length":1},{"begin":40,"length":1},{"begin":45,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":61,"length":2},{"begin":67,"length":1},{"begin":73,"length":1},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":88,"length":1},{"begin":94,"length":1},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":1},{"begin":199,"length":23}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T01:08:53.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.919+0000 7f01b2efd700 1 -- 172.21.15.46:0/18268357 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f018c066f30 msgr2=0x7f018c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:53.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.919+0000 7f01b2efd700 1 --2- 172.21.15.46:0/18268357 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f018c066f30 0x7f018c0693e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f019c0093f0 tx=0x7f019c017000 comp rx=0 tx=0).stop 2024-01-27T01:08:53.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.919+0000 7f01b2efd700 1 -- 172.21.15.46:0/18268357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01ac0cd2a0 msgr2=0x7f01ac14f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:53.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.919+0000 7f01b2efd700 1 --2- 172.21.15.46:0/18268357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01ac0cd2a0 0x7f01ac14f390 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f019400c4e0 tx=0x7f019400c7f0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.919+0000 7f01b2efd700 1 -- 172.21.15.46:0/18268357 shutdown_connections 2024-01-27T01:08:53.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.919+0000 7f01b2efd700 1 --2- 172.21.15.46:0/18268357 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f018c066f30 0x7f018c0693e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.919+0000 7f01b2efd700 1 --2- 172.21.15.46:0/18268357 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f01ac0cf0b0 0x7f01ac15e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.919+0000 7f01b2efd700 1 --2- 172.21.15.46:0/18268357 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ac0cced0 0x7f01ac157170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.919+0000 7f01b2efd700 1 --2- 172.21.15.46:0/18268357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01ac0cd2a0 0x7f01ac14f390 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:53.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.919+0000 7f01b2efd700 1 -- 172.21.15.46:0/18268357 >> 172.21.15.46:0/18268357 conn(0x7f01ac068190 msgr2=0x7f01ac0ce340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:53.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.919+0000 7f01b2efd700 1 -- 172.21.15.46:0/18268357 shutdown_connections 2024-01-27T01:08:53.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:53.919+0000 7f01b2efd700 1 -- 172.21.15.46:0/18268357 wait complete. 2024-01-27T01:08:54.934 INFO:tasks.ceph:Scrubbing osd.0 2024-01-27T01:08:54.934 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-01-27T01:08:55.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.080+0000 7f52aa23e700 1 Processor -- start 2024-01-27T01:08:55.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.080+0000 7f52aa23e700 1 -- start start 2024-01-27T01:08:55.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.081+0000 7f52aa23e700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f52a41572e0 0x7f52a40b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.081+0000 7f52aa23e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52a41576b0 0x7f52a4159f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.081+0000 7f52aa23e700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a406ebd0 con 0x7f52a4157bc0 2024-01-27T01:08:55.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.081+0000 7f52aa23e700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a415a7b0 con 0x7f52a41572e0 2024-01-27T01:08:55.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.081+0000 7f52aa23e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a415aa90 con 0x7f52a41576b0 2024-01-27T01:08:55.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.081+0000 7f52a2ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52a41576b0 0x7f52a4159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.081+0000 7f52a2ffd700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52a41576b0 0x7f52a4159f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:44674/0 (socket says 172.21.15.46:44674) 2024-01-27T01:08:55.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.081+0000 7f52a2ffd700 1 -- 172.21.15.46:0/4171280325 learned_addr learned my addr 172.21.15.46:0/4171280325 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.081+0000 7f52a37fe700 1 --2- 172.21.15.46:0/4171280325 >> v2:172.21.15.106:3300/0 conn(0x7f52a41572e0 0x7f52a40b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.081+0000 7f52a2ffd700 1 -- 172.21.15.46:0/4171280325 >> v2:172.21.15.106:3300/0 conn(0x7f52a41572e0 msgr2=0x7f52a40b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.081+0000 7f52a2ffd700 1 --2- 172.21.15.46:0/4171280325 >> v2:172.21.15.106:3300/0 conn(0x7f52a41572e0 0x7f52a40b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.081+0000 7f52a2ffd700 1 -- 172.21.15.46:0/4171280325 >> v1:172.21.15.106:6789/0 conn(0x7f52a4157bc0 legacy=0x7f52a40679b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.082+0000 7f52a2ffd700 1 -- 172.21.15.46:0/4171280325 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a415ad70 con 0x7f52a41576b0 2024-01-27T01:08:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.082+0000 7f52a2ffd700 1 --2- 172.21.15.46:0/4171280325 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52a41576b0 0x7f52a4159f80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f52980099b0 tx=0x7f5298009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bee94b1c10e21cc2 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.082+0000 7f52a27fc700 1 -- 172.21.15.46:0/4171280325 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298009ea0 con 0x7f52a41576b0 2024-01-27T01:08:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.082+0000 7f52a27fc700 1 -- 172.21.15.46:0/4171280325 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529800b860 con 0x7f52a41576b0 2024-01-27T01:08:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.082+0000 7f52a27fc700 1 -- 172.21.15.46:0/4171280325 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529800bc00 con 0x7f52a41576b0 2024-01-27T01:08:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.082+0000 7f52aa23e700 1 -- 172.21.15.46:0/4171280325 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52a41576b0 msgr2=0x7f52a4159f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.082+0000 7f52aa23e700 1 --2- 172.21.15.46:0/4171280325 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52a41576b0 0x7f52a4159f80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f52980099b0 tx=0x7f5298009250 comp rx=0 tx=0).stop 2024-01-27T01:08:55.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.083+0000 7f52aa23e700 1 -- 172.21.15.46:0/4171280325 shutdown_connections 2024-01-27T01:08:55.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.083+0000 7f52aa23e700 1 --2- 172.21.15.46:0/4171280325 >> v2:172.21.15.106:3300/0 conn(0x7f52a41572e0 0x7f52a40b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.083+0000 7f52aa23e700 1 --2- 172.21.15.46:0/4171280325 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52a41576b0 0x7f52a4159f80 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.083+0000 7f52aa23e700 1 -- 172.21.15.46:0/4171280325 >> 172.21.15.46:0/4171280325 conn(0x7f52a4069590 msgr2=0x7f52a406aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:55.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.083+0000 7f52aa23e700 1 -- 172.21.15.46:0/4171280325 shutdown_connections 2024-01-27T01:08:55.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.083+0000 7f52aa23e700 1 -- 172.21.15.46:0/4171280325 wait complete. 2024-01-27T01:08:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.084+0000 7f52aa23e700 1 Processor -- start 2024-01-27T01:08:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.084+0000 7f52aa23e700 1 -- start start 2024-01-27T01:08:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.085+0000 7f52aa23e700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52a41572e0 0x7f52a4150540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.085+0000 7f52aa23e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52a41576b0 0x7f52a40aa290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.085+0000 7f52aa23e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52a4157bc0 0x7f52a40ad020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.085+0000 7f52aa23e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a40abc30 con 0x7f52a41576b0 2024-01-27T01:08:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.085+0000 7f52aa23e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a40ab7a0 con 0x7f52a4157bc0 2024-01-27T01:08:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.085+0000 7f52aa23e700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a40aab90 con 0x7f52a41572e0 2024-01-27T01:08:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.085+0000 7f52a2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52a41576b0 0x7f52a40aa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.085+0000 7f52a2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52a41576b0 0x7f52a40aa290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55382/0 (socket says 172.21.15.46:55382) 2024-01-27T01:08:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.085+0000 7f52a2ffd700 1 -- 172.21.15.46:0/3491313142 learned_addr learned my addr 172.21.15.46:0/3491313142 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.085+0000 7f52a3fff700 1 --2- 172.21.15.46:0/3491313142 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52a4157bc0 0x7f52a40ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.085+0000 7f52a37fe700 1 --2- 172.21.15.46:0/3491313142 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52a41572e0 0x7f52a4150540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.085+0000 7f52a2ffd700 1 -- 172.21.15.46:0/3491313142 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52a41572e0 msgr2=0x7f52a4150540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.085+0000 7f52a2ffd700 1 --2- 172.21.15.46:0/3491313142 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52a41572e0 0x7f52a4150540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.085+0000 7f52a2ffd700 1 -- 172.21.15.46:0/3491313142 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52a4157bc0 msgr2=0x7f52a40ad020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.085+0000 7f52a2ffd700 1 --2- 172.21.15.46:0/3491313142 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52a4157bc0 0x7f52a40ad020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.085+0000 7f52a2ffd700 1 -- 172.21.15.46:0/3491313142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52980094f0 con 0x7f52a41576b0 2024-01-27T01:08:55.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.086+0000 7f52a2ffd700 1 --2- 172.21.15.46:0/3491313142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52a41576b0 0x7f52a40aa290 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f52a415abd0 tx=0x7f5298017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:55.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.086+0000 7f528b7fe700 1 -- 172.21.15.46:0/3491313142 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298000d10 con 0x7f52a41576b0 2024-01-27T01:08:55.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.086+0000 7f528b7fe700 1 -- 172.21.15.46:0/3491313142 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529800bbe0 con 0x7f52a41576b0 2024-01-27T01:08:55.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.086+0000 7f52aa23e700 1 -- 172.21.15.46:0/3491313142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a40aae70 con 0x7f52a41576b0 2024-01-27T01:08:55.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.086+0000 7f528b7fe700 1 -- 172.21.15.46:0/3491313142 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298011670 con 0x7f52a41576b0 2024-01-27T01:08:55.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.086+0000 7f52aa23e700 1 -- 172.21.15.46:0/3491313142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a40ab430 con 0x7f52a41576b0 2024-01-27T01:08:55.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.087+0000 7f52aa23e700 1 -- 172.21.15.46:0/3491313142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f52a404c710 con 0x7f52a41576b0 2024-01-27T01:08:55.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.087+0000 7f528b7fe700 1 -- 172.21.15.46:0/3491313142 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5298011aa0 con 0x7f52a41576b0 2024-01-27T01:08:55.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.088+0000 7f528b7fe700 1 --2- 172.21.15.46:0/3491313142 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5290066f30 0x7f52900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.088+0000 7f528b7fe700 1 -- 172.21.15.46:0/3491313142 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f5298086920 con 0x7f52a41576b0 2024-01-27T01:08:55.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.088+0000 7f528b7fe700 1 --2- 172.21.15.46:0/3491313142 >> [v2:172.21.15.46:6819/43682,v1:172.21.15.46:6821/43682] conn(0x7f529006dbd0 0x7f5290070060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.088+0000 7f528b7fe700 1 -- 172.21.15.46:0/3491313142 --> [v2:172.21.15.46:6819/43682,v1:172.21.15.46:6821/43682] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5290070690 con 0x7f529006dbd0 2024-01-27T01:08:55.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.088+0000 7f528b7fe700 1 -- 172.21.15.46:0/3491313142 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=269) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5298086cf0 con 0x7f52a41576b0 2024-01-27T01:08:55.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.088+0000 7f52a37fe700 1 --2- 172.21.15.46:0/3491313142 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5290066f30 0x7f52900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.088+0000 7f52a3fff700 1 --2- 172.21.15.46:0/3491313142 >> [v2:172.21.15.46:6819/43682,v1:172.21.15.46:6821/43682] conn(0x7f529006dbd0 0x7f5290070060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.088+0000 7f52a3fff700 1 --2- 172.21.15.46:0/3491313142 >> [v2:172.21.15.46:6819/43682,v1:172.21.15.46:6821/43682] conn(0x7f529006dbd0 0x7f5290070060 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:55.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.088+0000 7f52a37fe700 1 --2- 172.21.15.46:0/3491313142 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5290066f30 0x7f52900693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5294009d10 tx=0x7f52940091a0 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:55.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.089+0000 7f528b7fe700 1 -- 172.21.15.46:0/3491313142 <== osd.0 v2:172.21.15.46:6819/43682 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f5290070690 con 0x7f529006dbd0 2024-01-27T01:08:55.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.104+0000 7f52aa23e700 1 -- 172.21.15.46:0/3491313142 --> [v2:172.21.15.46:6819/43682,v1:172.21.15.46:6821/43682] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f52a4069c60 con 0x7f529006dbd0 2024-01-27T01:08:55.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.104+0000 7f528b7fe700 1 -- 172.21.15.46:0/3491313142 <== osd.0 v2:172.21.15.46:6819/43682 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f52a4069c60 con 0x7f529006dbd0 2024-01-27T01:08:55.104 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:08:55.105 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-27T01:08:55.105 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:08:55.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.104+0000 7f52aa23e700 1 -- 172.21.15.46:0/3491313142 >> [v2:172.21.15.46:6819/43682,v1:172.21.15.46:6821/43682] conn(0x7f529006dbd0 msgr2=0x7f5290070060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.104+0000 7f52aa23e700 1 --2- 172.21.15.46:0/3491313142 >> [v2:172.21.15.46:6819/43682,v1:172.21.15.46:6821/43682] conn(0x7f529006dbd0 0x7f5290070060 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.105+0000 7f52aa23e700 1 -- 172.21.15.46:0/3491313142 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5290066f30 msgr2=0x7f52900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.105+0000 7f52aa23e700 1 --2- 172.21.15.46:0/3491313142 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5290066f30 0x7f52900693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5294009d10 tx=0x7f52940091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.105+0000 7f52aa23e700 1 -- 172.21.15.46:0/3491313142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52a41576b0 msgr2=0x7f52a40aa290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.105+0000 7f52aa23e700 1 --2- 172.21.15.46:0/3491313142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52a41576b0 0x7f52a40aa290 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f52a415abd0 tx=0x7f5298017000 comp rx=0 tx=0).stop 2024-01-27T01:08:55.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.105+0000 7f52a3fff700 1 -- 172.21.15.46:0/3491313142 reap_dead start 2024-01-27T01:08:55.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.105+0000 7f52aa23e700 1 -- 172.21.15.46:0/3491313142 shutdown_connections 2024-01-27T01:08:55.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.105+0000 7f52aa23e700 1 -- 172.21.15.46:0/3491313142 >> 172.21.15.46:0/3491313142 conn(0x7f52a4069590 msgr2=0x7f52a40679b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:55.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.105+0000 7f52aa23e700 1 -- 172.21.15.46:0/3491313142 shutdown_connections 2024-01-27T01:08:55.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.105+0000 7f52aa23e700 1 -- 172.21.15.46:0/3491313142 wait complete. 2024-01-27T01:08:55.118 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-01-27T01:08:55.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.250+0000 7f3467037700 1 Processor -- start 2024-01-27T01:08:55.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.250+0000 7f3467037700 1 -- start start 2024-01-27T01:08:55.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.251+0000 7f3467037700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f346006af20 0x7f346006b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.251+0000 7f3467037700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f346015a090 0x7f346015a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.251+0000 7f3467037700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346015d620 con 0x7f3460157570 2024-01-27T01:08:55.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.251+0000 7f3467037700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f346015dec0 con 0x7f346015a090 2024-01-27T01:08:55.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.251+0000 7f3467037700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f346006b8d0 con 0x7f346006af20 2024-01-27T01:08:55.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.251+0000 7f34655d4700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f346015a090 0x7f346015a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.251+0000 7f3464dd3700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3460157570 0x7f34600c7a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60986/0 (socket says 172.21.15.46:60986) 2024-01-27T01:08:55.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.251+0000 7f3464dd3700 1 -- 172.21.15.46:0/2307459513 learned_addr learned my addr 172.21.15.46:0/2307459513 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:55.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.251+0000 7f345ffff700 1 --2- 172.21.15.46:0/2307459513 >> v2:172.21.15.106:3300/0 conn(0x7f346006af20 0x7f346006b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.252+0000 7f34655d4700 1 -- 172.21.15.46:0/2307459513 >> v2:172.21.15.106:3300/0 conn(0x7f346006af20 msgr2=0x7f346006b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:55.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.252+0000 7f34655d4700 1 --2- 172.21.15.46:0/2307459513 >> v2:172.21.15.106:3300/0 conn(0x7f346006af20 0x7f346006b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.252+0000 7f34655d4700 1 -- 172.21.15.46:0/2307459513 >> v1:172.21.15.46:6789/0 conn(0x7f3460157570 legacy=0x7f34600c7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.252+0000 7f345f7fe700 1 -- 172.21.15.46:0/2307459513 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59320383 0 0) 0x7f346015d620 con 0x7f3460157570 2024-01-27T01:08:55.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.252+0000 7f34655d4700 1 -- 172.21.15.46:0/2307459513 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34600af900 con 0x7f346015a090 2024-01-27T01:08:55.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.252+0000 7f34655d4700 1 --2- 172.21.15.46:0/2307459513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f346015a090 0x7f346015a4f0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f346006ffb0 tx=0x7f3458009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1619a16b81c0c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:55.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.252+0000 7f345f7fe700 1 -- 172.21.15.46:0/2307459513 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345800e070 con 0x7f346015a090 2024-01-27T01:08:55.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.252+0000 7f345f7fe700 1 -- 172.21.15.46:0/2307459513 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345800bb60 con 0x7f346015a090 2024-01-27T01:08:55.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.252+0000 7f345f7fe700 1 -- 172.21.15.46:0/2307459513 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3458012430 con 0x7f346015a090 2024-01-27T01:08:55.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.252+0000 7f3467037700 1 -- 172.21.15.46:0/2307459513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f346015a090 msgr2=0x7f346015a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.252+0000 7f3467037700 1 --2- 172.21.15.46:0/2307459513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f346015a090 0x7f346015a4f0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f346006ffb0 tx=0x7f3458009780 comp rx=0 tx=0).stop 2024-01-27T01:08:55.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.252+0000 7f3467037700 1 -- 172.21.15.46:0/2307459513 shutdown_connections 2024-01-27T01:08:55.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.252+0000 7f3467037700 1 --2- 172.21.15.46:0/2307459513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f346015a090 0x7f346015a4f0 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.252+0000 7f3467037700 1 --2- 172.21.15.46:0/2307459513 >> v2:172.21.15.106:3300/0 conn(0x7f346006af20 0x7f346006b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.252+0000 7f3467037700 1 -- 172.21.15.46:0/2307459513 >> 172.21.15.46:0/2307459513 conn(0x7f3460068ac0 msgr2=0x7f3460068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:55.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.253+0000 7f3467037700 1 -- 172.21.15.46:0/2307459513 shutdown_connections 2024-01-27T01:08:55.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.253+0000 7f3467037700 1 -- 172.21.15.46:0/2307459513 wait complete. 2024-01-27T01:08:55.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.254+0000 7f3467037700 1 Processor -- start 2024-01-27T01:08:55.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.254+0000 7f3467037700 1 -- start start 2024-01-27T01:08:55.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.255+0000 7f3467037700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f346006af20 0x7f34600abad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.255+0000 7f3467037700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3460157570 0x7f34600ac010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.255+0000 7f3467037700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f346015a090 0x7f3460169a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.255+0000 7f3467037700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3460153b30 con 0x7f3460157570 2024-01-27T01:08:55.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.255+0000 7f3467037700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34600c4aa0 con 0x7f346015a090 2024-01-27T01:08:55.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.255+0000 7f3467037700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34600ac6f0 con 0x7f346006af20 2024-01-27T01:08:55.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.255+0000 7f345ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3460157570 0x7f34600ac010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.255+0000 7f345ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3460157570 0x7f34600ac010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55388/0 (socket says 172.21.15.46:55388) 2024-01-27T01:08:55.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.255+0000 7f345ffff700 1 -- 172.21.15.46:0/1085646397 learned_addr learned my addr 172.21.15.46:0/1085646397 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:55.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.255+0000 7f34655d4700 1 --2- 172.21.15.46:0/1085646397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f346015a090 0x7f3460169a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.255+0000 7f3464dd3700 1 --2- 172.21.15.46:0/1085646397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f346006af20 0x7f34600abad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.255+0000 7f345ffff700 1 -- 172.21.15.46:0/1085646397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f346006af20 msgr2=0x7f34600abad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.255+0000 7f345ffff700 1 --2- 172.21.15.46:0/1085646397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f346006af20 0x7f34600abad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.255+0000 7f345ffff700 1 -- 172.21.15.46:0/1085646397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f346015a090 msgr2=0x7f3460169a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.255+0000 7f345ffff700 1 --2- 172.21.15.46:0/1085646397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f346015a090 0x7f3460169a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.255+0000 7f345ffff700 1 -- 172.21.15.46:0/1085646397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3458009a20 con 0x7f3460157570 2024-01-27T01:08:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.255+0000 7f345ffff700 1 --2- 172.21.15.46:0/1085646397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3460157570 0x7f34600ac010 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f344800e960 tx=0x7f34480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.256+0000 7f345cff9700 1 -- 172.21.15.46:0/1085646397 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3448013680 con 0x7f3460157570 2024-01-27T01:08:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.256+0000 7f345cff9700 1 -- 172.21.15.46:0/1085646397 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34480137e0 con 0x7f3460157570 2024-01-27T01:08:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.256+0000 7f3467037700 1 -- 172.21.15.46:0/1085646397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34600a99a0 con 0x7f3460157570 2024-01-27T01:08:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.256+0000 7f345cff9700 1 -- 172.21.15.46:0/1085646397 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3448013b80 con 0x7f3460157570 2024-01-27T01:08:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.256+0000 7f3467037700 1 -- 172.21.15.46:0/1085646397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34600a9f60 con 0x7f3460157570 2024-01-27T01:08:55.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.257+0000 7f3467037700 1 -- 172.21.15.46:0/1085646397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34600c4c10 con 0x7f3460157570 2024-01-27T01:08:55.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.257+0000 7f345cff9700 1 -- 172.21.15.46:0/1085646397 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f344800c490 con 0x7f3460157570 2024-01-27T01:08:55.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.258+0000 7f345cff9700 1 --2- 172.21.15.46:0/1085646397 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3440066f30 0x7f34400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.258+0000 7f345cff9700 1 -- 172.21.15.46:0/1085646397 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f34480849a0 con 0x7f3460157570 2024-01-27T01:08:55.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.258+0000 7f3464dd3700 1 --2- 172.21.15.46:0/1085646397 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3440066f30 0x7f34400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.258+0000 7f3464dd3700 1 --2- 172.21.15.46:0/1085646397 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3440066f30 0x7f34400693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f34500011e0 tx=0x7f3450009040 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:55.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.262+0000 7f345cff9700 1 -- 172.21.15.46:0/1085646397 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f34480518e0 con 0x7f3460157570 2024-01-27T01:08:55.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.464+0000 7f3467037700 1 -- 172.21.15.46:0/1085646397 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f346004c710 con 0x7f3440066f30 2024-01-27T01:08:55.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.465+0000 7f345cff9700 1 -- 172.21.15.46:0/1085646397 <== mgr.10565 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f346004c710 con 0x7f3440066f30 2024-01-27T01:08:55.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.467+0000 7f3467037700 1 -- 172.21.15.46:0/1085646397 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3440066f30 msgr2=0x7f34400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.467+0000 7f3467037700 1 --2- 172.21.15.46:0/1085646397 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3440066f30 0x7f34400693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f34500011e0 tx=0x7f3450009040 comp rx=0 tx=0).stop 2024-01-27T01:08:55.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.467+0000 7f3467037700 1 -- 172.21.15.46:0/1085646397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3460157570 msgr2=0x7f34600ac010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.467+0000 7f3467037700 1 --2- 172.21.15.46:0/1085646397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3460157570 0x7f34600ac010 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f344800e960 tx=0x7f34480091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.467+0000 7f3467037700 1 -- 172.21.15.46:0/1085646397 shutdown_connections 2024-01-27T01:08:55.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.467+0000 7f3467037700 1 --2- 172.21.15.46:0/1085646397 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3440066f30 0x7f34400693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.467+0000 7f3467037700 1 --2- 172.21.15.46:0/1085646397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f346015a090 0x7f3460169a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.467+0000 7f3467037700 1 --2- 172.21.15.46:0/1085646397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f346006af20 0x7f34600abad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.467+0000 7f3467037700 1 --2- 172.21.15.46:0/1085646397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3460157570 0x7f34600ac010 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.467+0000 7f3467037700 1 -- 172.21.15.46:0/1085646397 >> 172.21.15.46:0/1085646397 conn(0x7f3460068ac0 msgr2=0x7f3460154630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:55.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.467+0000 7f3467037700 1 -- 172.21.15.46:0/1085646397 shutdown_connections 2024-01-27T01:08:55.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.467+0000 7f3467037700 1 -- 172.21.15.46:0/1085646397 wait complete. 2024-01-27T01:08:55.469 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 0 to deep-scrub 2024-01-27T01:08:55.481 INFO:tasks.ceph:Scrubbing osd.1 2024-01-27T01:08:55.481 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-01-27T01:08:55.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.617+0000 7fda583f7700 1 Processor -- start 2024-01-27T01:08:55.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.617+0000 7fda583f7700 1 -- start start 2024-01-27T01:08:55.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.618+0000 7fda583f7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda500af6c0 0x7fda500afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.618+0000 7fda583f7700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fda500ce120 0x7fda500679b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.618+0000 7fda583f7700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda500cb0a0 con 0x7fda50067ef0 2024-01-27T01:08:55.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.618+0000 7fda583f7700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda500c97f0 con 0x7fda500ce120 2024-01-27T01:08:55.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.618+0000 7fda583f7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda500cb5d0 con 0x7fda500af6c0 2024-01-27T01:08:55.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.619+0000 7fda55992700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fda500ce120 0x7fda500679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.619+0000 7fda56193700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda500af6c0 0x7fda500afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.619+0000 7fda55992700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fda500ce120 0x7fda500679b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:59700/0 (socket says 172.21.15.46:59700) 2024-01-27T01:08:55.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.619+0000 7fda56994700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fda50067ef0 0x7fda500c7b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:47838/0 (socket says 172.21.15.46:47838) 2024-01-27T01:08:55.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.619+0000 7fda55992700 1 -- 172.21.15.46:0/2462569435 learned_addr learned my addr 172.21.15.46:0/2462569435 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:55.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.619+0000 7fda55992700 1 -- 172.21.15.46:0/2462569435 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda500af6c0 msgr2=0x7fda500afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:55.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.619+0000 7fda55992700 1 --2- 172.21.15.46:0/2462569435 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda500af6c0 0x7fda500afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.619+0000 7fda55992700 1 -- 172.21.15.46:0/2462569435 >> v1:172.21.15.106:6789/0 conn(0x7fda50067ef0 legacy=0x7fda500c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.619+0000 7fda55992700 1 -- 172.21.15.46:0/2462569435 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda500cd310 con 0x7fda500ce120 2024-01-27T01:08:55.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.619+0000 7fda55992700 1 --2- 172.21.15.46:0/2462569435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda500ce120 0x7fda500679b0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fda400099b0 tx=0x7fda40009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0f0ddad47124d48 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:55.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.620+0000 7fda55191700 1 -- 172.21.15.46:0/2462569435 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda40009ea0 con 0x7fda500ce120 2024-01-27T01:08:55.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.620+0000 7fda55191700 1 -- 172.21.15.46:0/2462569435 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4000b860 con 0x7fda500ce120 2024-01-27T01:08:55.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.620+0000 7fda55191700 1 -- 172.21.15.46:0/2462569435 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4000bc00 con 0x7fda500ce120 2024-01-27T01:08:55.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.620+0000 7fda583f7700 1 -- 172.21.15.46:0/2462569435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda500ce120 msgr2=0x7fda500679b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.620+0000 7fda583f7700 1 --2- 172.21.15.46:0/2462569435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda500ce120 0x7fda500679b0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fda400099b0 tx=0x7fda40009250 comp rx=0 tx=0).stop 2024-01-27T01:08:55.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.620+0000 7fda583f7700 1 -- 172.21.15.46:0/2462569435 shutdown_connections 2024-01-27T01:08:55.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.620+0000 7fda583f7700 1 --2- 172.21.15.46:0/2462569435 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda500af6c0 0x7fda500afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.620+0000 7fda583f7700 1 --2- 172.21.15.46:0/2462569435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda500ce120 0x7fda500679b0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.620+0000 7fda583f7700 1 -- 172.21.15.46:0/2462569435 >> 172.21.15.46:0/2462569435 conn(0x7fda500692a0 msgr2=0x7fda500cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:55.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.620+0000 7fda583f7700 1 -- 172.21.15.46:0/2462569435 shutdown_connections 2024-01-27T01:08:55.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.620+0000 7fda583f7700 1 -- 172.21.15.46:0/2462569435 wait complete. 2024-01-27T01:08:55.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.622+0000 7fda583f7700 1 Processor -- start 2024-01-27T01:08:55.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.622+0000 7fda583f7700 1 -- start start 2024-01-27T01:08:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.623+0000 7fda583f7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda50067ef0 0x7fda500b0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.623+0000 7fda583f7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda500af6c0 0x7fda5016de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.623+0000 7fda583f7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda500ce120 0x7fda50154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.623+0000 7fda583f7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5016d650 con 0x7fda50067ef0 2024-01-27T01:08:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.623+0000 7fda583f7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda50155d80 con 0x7fda500af6c0 2024-01-27T01:08:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.623+0000 7fda583f7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda500b0920 con 0x7fda500ce120 2024-01-27T01:08:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.623+0000 7fda56193700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda50067ef0 0x7fda500b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.623+0000 7fda56193700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda50067ef0 0x7fda500b0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55398/0 (socket says 172.21.15.46:55398) 2024-01-27T01:08:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.623+0000 7fda56193700 1 -- 172.21.15.46:0/2343389481 learned_addr learned my addr 172.21.15.46:0/2343389481 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.623+0000 7fda55992700 1 --2- 172.21.15.46:0/2343389481 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda500af6c0 0x7fda5016de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.623+0000 7fda56994700 1 --2- 172.21.15.46:0/2343389481 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda500ce120 0x7fda50154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.624+0000 7fda56193700 1 -- 172.21.15.46:0/2343389481 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda500ce120 msgr2=0x7fda50154b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.624+0000 7fda56193700 1 --2- 172.21.15.46:0/2343389481 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda500ce120 0x7fda50154b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.624+0000 7fda56193700 1 -- 172.21.15.46:0/2343389481 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda500af6c0 msgr2=0x7fda5016de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.624+0000 7fda56193700 1 --2- 172.21.15.46:0/2343389481 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda500af6c0 0x7fda5016de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.624+0000 7fda56193700 1 -- 172.21.15.46:0/2343389481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda400094f0 con 0x7fda50067ef0 2024-01-27T01:08:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.624+0000 7fda56994700 1 --2- 172.21.15.46:0/2343389481 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda500ce120 0x7fda50154b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.624+0000 7fda56193700 1 --2- 172.21.15.46:0/2343389481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda50067ef0 0x7fda500b0290 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00ec80 tx=0x7fda4c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.624+0000 7fda467fc700 1 -- 172.21.15.46:0/2343389481 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c00c850 con 0x7fda50067ef0 2024-01-27T01:08:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.624+0000 7fda467fc700 1 -- 172.21.15.46:0/2343389481 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4c00c9b0 con 0x7fda50067ef0 2024-01-27T01:08:55.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.624+0000 7fda467fc700 1 -- 172.21.15.46:0/2343389481 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c00cd30 con 0x7fda50067ef0 2024-01-27T01:08:55.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.624+0000 7fda583f7700 1 -- 172.21.15.46:0/2343389481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda500b0c00 con 0x7fda50067ef0 2024-01-27T01:08:55.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.624+0000 7fda583f7700 1 -- 172.21.15.46:0/2343389481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda5016d8f0 con 0x7fda50067ef0 2024-01-27T01:08:55.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.625+0000 7fda583f7700 1 -- 172.21.15.46:0/2343389481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fda500b1120 con 0x7fda50067ef0 2024-01-27T01:08:55.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.626+0000 7fda467fc700 1 -- 172.21.15.46:0/2343389481 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fda4c013680 con 0x7fda50067ef0 2024-01-27T01:08:55.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.626+0000 7fda467fc700 1 --2- 172.21.15.46:0/2343389481 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fda34066f00 0x7fda340693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.626+0000 7fda467fc700 1 -- 172.21.15.46:0/2343389481 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7fda4c085540 con 0x7fda50067ef0 2024-01-27T01:08:55.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.626+0000 7fda467fc700 1 --2- 172.21.15.46:0/2343389481 >> [v2:172.21.15.46:6800/43676,v1:172.21.15.46:6801/43676] conn(0x7fda3406dd10 0x7fda340701a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.626+0000 7fda467fc700 1 -- 172.21.15.46:0/2343389481 --> [v2:172.21.15.46:6800/43676,v1:172.21.15.46:6801/43676] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda340707d0 con 0x7fda3406dd10 2024-01-27T01:08:55.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.626+0000 7fda467fc700 1 -- 172.21.15.46:0/2343389481 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=269) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda4c085910 con 0x7fda50067ef0 2024-01-27T01:08:55.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.626+0000 7fda55992700 1 --2- 172.21.15.46:0/2343389481 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fda34066f00 0x7fda340693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.626+0000 7fda56994700 1 --2- 172.21.15.46:0/2343389481 >> [v2:172.21.15.46:6800/43676,v1:172.21.15.46:6801/43676] conn(0x7fda3406dd10 0x7fda340701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.627+0000 7fda56994700 1 --2- 172.21.15.46:0/2343389481 >> [v2:172.21.15.46:6800/43676,v1:172.21.15.46:6801/43676] conn(0x7fda3406dd10 0x7fda340701a0 crc :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:55.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.627+0000 7fda55992700 1 --2- 172.21.15.46:0/2343389481 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fda34066f00 0x7fda340693b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fda400099b0 tx=0x7fda40017000 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:55.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.627+0000 7fda467fc700 1 -- 172.21.15.46:0/2343389481 <== osd.1 v2:172.21.15.46:6800/43676 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fda340707d0 con 0x7fda3406dd10 2024-01-27T01:08:55.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.642+0000 7fda583f7700 1 -- 172.21.15.46:0/2343389481 --> [v2:172.21.15.46:6800/43676,v1:172.21.15.46:6801/43676] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fda500c78e0 con 0x7fda3406dd10 2024-01-27T01:08:55.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.642+0000 7fda467fc700 1 -- 172.21.15.46:0/2343389481 <== osd.1 v2:172.21.15.46:6800/43676 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fda500c78e0 con 0x7fda3406dd10 2024-01-27T01:08:55.643 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:08:55.643 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-27T01:08:55.643 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:08:55.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.643+0000 7fda583f7700 1 -- 172.21.15.46:0/2343389481 >> [v2:172.21.15.46:6800/43676,v1:172.21.15.46:6801/43676] conn(0x7fda3406dd10 msgr2=0x7fda340701a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.643+0000 7fda583f7700 1 --2- 172.21.15.46:0/2343389481 >> [v2:172.21.15.46:6800/43676,v1:172.21.15.46:6801/43676] conn(0x7fda3406dd10 0x7fda340701a0 crc :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.643+0000 7fda583f7700 1 -- 172.21.15.46:0/2343389481 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fda34066f00 msgr2=0x7fda340693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.643+0000 7fda583f7700 1 --2- 172.21.15.46:0/2343389481 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fda34066f00 0x7fda340693b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fda400099b0 tx=0x7fda40017000 comp rx=0 tx=0).stop 2024-01-27T01:08:55.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.643+0000 7fda583f7700 1 -- 172.21.15.46:0/2343389481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda50067ef0 msgr2=0x7fda500b0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.643+0000 7fda583f7700 1 --2- 172.21.15.46:0/2343389481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda50067ef0 0x7fda500b0290 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00ec80 tx=0x7fda4c009120 comp rx=0 tx=0).stop 2024-01-27T01:08:55.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.643+0000 7fda56994700 1 -- 172.21.15.46:0/2343389481 reap_dead start 2024-01-27T01:08:55.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.643+0000 7fda583f7700 1 -- 172.21.15.46:0/2343389481 shutdown_connections 2024-01-27T01:08:55.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.643+0000 7fda583f7700 1 -- 172.21.15.46:0/2343389481 >> 172.21.15.46:0/2343389481 conn(0x7fda500692a0 msgr2=0x7fda50158d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:55.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.643+0000 7fda583f7700 1 -- 172.21.15.46:0/2343389481 shutdown_connections 2024-01-27T01:08:55.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.643+0000 7fda583f7700 1 -- 172.21.15.46:0/2343389481 wait complete. 2024-01-27T01:08:55.656 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-01-27T01:08:55.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.789+0000 7f7cd22d2700 1 Processor -- start 2024-01-27T01:08:55.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.789+0000 7f7cd22d2700 1 -- start start 2024-01-27T01:08:55.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.790+0000 7f7cd22d2700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7ccc0cd2a0 0x7f7ccc0c4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.790+0000 7f7cd22d2700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ccc068ac0 con 0x7f7ccc0cced0 2024-01-27T01:08:55.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.790+0000 7f7cd22d2700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ccc0c9470 con 0x7f7ccc0cf0b0 2024-01-27T01:08:55.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.790+0000 7f7cd22d2700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ccc0c4c40 con 0x7f7ccc0cd2a0 2024-01-27T01:08:55.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.790+0000 7f7cd086f700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7ccc0cd2a0 0x7f7ccc0c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.790+0000 7f7ccbfff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7ccc0cced0 0x7f7ccc15d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32768/0 (socket says 172.21.15.46:32768) 2024-01-27T01:08:55.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.790+0000 7f7ccbfff700 1 -- 172.21.15.46:0/2267012885 learned_addr learned my addr 172.21.15.46:0/2267012885 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:55.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.791+0000 7f7cd086f700 1 -- 172.21.15.46:0/2267012885 >> v1:172.21.15.106:6789/0 conn(0x7f7ccc0cf0b0 legacy=0x7f7ccc15de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.791+0000 7f7cd086f700 1 -- 172.21.15.46:0/2267012885 >> v1:172.21.15.46:6789/0 conn(0x7f7ccc0cced0 legacy=0x7f7ccc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.791+0000 7f7ccaffd700 1 -- 172.21.15.46:0/2267012885 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251071784 0 0) 0x7f7ccc068ac0 con 0x7f7ccc0cced0 2024-01-27T01:08:55.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.791+0000 7f7cd086f700 1 -- 172.21.15.46:0/2267012885 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc15e510 con 0x7f7ccc0cd2a0 2024-01-27T01:08:55.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.791+0000 7f7cd086f700 1 --2- 172.21.15.46:0/2267012885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc0cd2a0 0x7f7ccc0c4380 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009e60 tx=0x7f7cc4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad14d6c9b20eebc server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:55.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.791+0000 7f7ccaffd700 1 -- 172.21.15.46:0/2267012885 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc400e070 con 0x7f7ccc0cd2a0 2024-01-27T01:08:55.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.791+0000 7f7ccaffd700 1 -- 172.21.15.46:0/2267012885 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc400bc30 con 0x7f7ccc0cd2a0 2024-01-27T01:08:55.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.791+0000 7f7ccaffd700 1 -- 172.21.15.46:0/2267012885 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4012430 con 0x7f7ccc0cd2a0 2024-01-27T01:08:55.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.792+0000 7f7cd22d2700 1 -- 172.21.15.46:0/2267012885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc0cd2a0 msgr2=0x7f7ccc0c4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.792+0000 7f7cd22d2700 1 --2- 172.21.15.46:0/2267012885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc0cd2a0 0x7f7ccc0c4380 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009e60 tx=0x7f7cc4009780 comp rx=0 tx=0).stop 2024-01-27T01:08:55.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.792+0000 7f7cd22d2700 1 -- 172.21.15.46:0/2267012885 shutdown_connections 2024-01-27T01:08:55.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.792+0000 7f7cd22d2700 1 --2- 172.21.15.46:0/2267012885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc0cd2a0 0x7f7ccc0c4380 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.792+0000 7f7cd22d2700 1 -- 172.21.15.46:0/2267012885 >> 172.21.15.46:0/2267012885 conn(0x7f7ccc068190 msgr2=0x7f7ccc0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:55.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.792+0000 7f7cd22d2700 1 -- 172.21.15.46:0/2267012885 shutdown_connections 2024-01-27T01:08:55.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.792+0000 7f7cd22d2700 1 -- 172.21.15.46:0/2267012885 wait complete. 2024-01-27T01:08:55.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.793+0000 7f7cd22d2700 1 Processor -- start 2024-01-27T01:08:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.793+0000 7f7cd22d2700 1 -- start start 2024-01-27T01:08:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.794+0000 7f7cd22d2700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ccc0cced0 0x7f7ccc14cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.794+0000 7f7cd22d2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ccc0cd2a0 0x7f7ccc0c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.794+0000 7f7cd22d2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc0cf0b0 0x7f7ccc0c7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.794+0000 7f7cd22d2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc14d1d0 con 0x7f7ccc0cf0b0 2024-01-27T01:08:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.794+0000 7f7cd22d2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc0b1db0 con 0x7f7ccc0cd2a0 2024-01-27T01:08:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.794+0000 7f7cd22d2700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc0c7a60 con 0x7f7ccc0cced0 2024-01-27T01:08:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.794+0000 7f7cd086f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc0cf0b0 0x7f7ccc0c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.794+0000 7f7cd086f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc0cf0b0 0x7f7ccc0c7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55412/0 (socket says 172.21.15.46:55412) 2024-01-27T01:08:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.794+0000 7f7cd086f700 1 -- 172.21.15.46:0/1888725112 learned_addr learned my addr 172.21.15.46:0/1888725112 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.794+0000 7f7ccb7fe700 1 --2- 172.21.15.46:0/1888725112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ccc0cd2a0 0x7f7ccc0c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.794+0000 7f7ccbfff700 1 --2- 172.21.15.46:0/1888725112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ccc0cced0 0x7f7ccc14cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.794+0000 7f7cd086f700 1 -- 172.21.15.46:0/1888725112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ccc0cced0 msgr2=0x7f7ccc14cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.794+0000 7f7cd086f700 1 --2- 172.21.15.46:0/1888725112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ccc0cced0 0x7f7ccc14cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.794+0000 7f7cd086f700 1 -- 172.21.15.46:0/1888725112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ccc0cd2a0 msgr2=0x7f7ccc0c6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:55.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.794+0000 7f7cd086f700 1 --2- 172.21.15.46:0/1888725112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ccc0cd2a0 0x7f7ccc0c6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:55.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.795+0000 7f7cd086f700 1 -- 172.21.15.46:0/1888725112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc4009a20 con 0x7f7ccc0cf0b0 2024-01-27T01:08:55.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.795+0000 7f7cd086f700 1 --2- 172.21.15.46:0/1888725112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc0cf0b0 0x7f7ccc0c7520 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4013000 tx=0x7f7cc4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:55.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.795+0000 7f7cb3fff700 1 -- 172.21.15.46:0/1888725112 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc401b420 con 0x7f7ccc0cf0b0 2024-01-27T01:08:55.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.795+0000 7f7cb3fff700 1 -- 172.21.15.46:0/1888725112 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc401b580 con 0x7f7ccc0cf0b0 2024-01-27T01:08:55.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.795+0000 7f7cd22d2700 1 -- 172.21.15.46:0/1888725112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ccc1592b0 con 0x7f7ccc0cf0b0 2024-01-27T01:08:55.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.795+0000 7f7cb3fff700 1 -- 172.21.15.46:0/1888725112 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc401b900 con 0x7f7ccc0cf0b0 2024-01-27T01:08:55.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.795+0000 7f7cd22d2700 1 -- 172.21.15.46:0/1888725112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ccc0b1760 con 0x7f7ccc0cf0b0 2024-01-27T01:08:55.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.796+0000 7f7cd22d2700 1 -- 172.21.15.46:0/1888725112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ccc0b1f60 con 0x7f7ccc0cf0b0 2024-01-27T01:08:55.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.796+0000 7f7cb3fff700 1 -- 172.21.15.46:0/1888725112 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7cc401bc70 con 0x7f7ccc0cf0b0 2024-01-27T01:08:55.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.797+0000 7f7cb3fff700 1 --2- 172.21.15.46:0/1888725112 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7cb8066f30 0x7f7cb80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:55.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.797+0000 7f7cb3fff700 1 -- 172.21.15.46:0/1888725112 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f7cc4086610 con 0x7f7ccc0cf0b0 2024-01-27T01:08:55.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.797+0000 7f7ccbfff700 1 --2- 172.21.15.46:0/1888725112 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7cb8066f30 0x7f7cb80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:55.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.798+0000 7f7ccbfff700 1 --2- 172.21.15.46:0/1888725112 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7cb8066f30 0x7f7cb80693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc007160 tx=0x7f7cbc007f50 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:55.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:55.801+0000 7f7cb3fff700 1 -- 172.21.15.46:0/1888725112 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7ccc0b1f60 con 0x7f7ccc0cf0b0 2024-01-27T01:08:56.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.004+0000 7f7cd22d2700 1 -- 172.21.15.46:0/1888725112 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ccc0b1f20 con 0x7f7cb8066f30 2024-01-27T01:08:56.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.005+0000 7f7cb3fff700 1 -- 172.21.15.46:0/1888725112 <== mgr.10565 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7ccc0b1f20 con 0x7f7cb8066f30 2024-01-27T01:08:56.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.007+0000 7f7cd22d2700 1 -- 172.21.15.46:0/1888725112 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7cb8066f30 msgr2=0x7f7cb80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.007+0000 7f7cd22d2700 1 --2- 172.21.15.46:0/1888725112 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7cb8066f30 0x7f7cb80693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc007160 tx=0x7f7cbc007f50 comp rx=0 tx=0).stop 2024-01-27T01:08:56.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.007+0000 7f7cd22d2700 1 -- 172.21.15.46:0/1888725112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc0cf0b0 msgr2=0x7f7ccc0c7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.007+0000 7f7cd22d2700 1 --2- 172.21.15.46:0/1888725112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc0cf0b0 0x7f7ccc0c7520 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4013000 tx=0x7f7cc4019000 comp rx=0 tx=0).stop 2024-01-27T01:08:56.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.007+0000 7f7cd22d2700 1 -- 172.21.15.46:0/1888725112 shutdown_connections 2024-01-27T01:08:56.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.007+0000 7f7cd22d2700 1 --2- 172.21.15.46:0/1888725112 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7cb8066f30 0x7f7cb80693e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.007+0000 7f7cd22d2700 1 --2- 172.21.15.46:0/1888725112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc0cf0b0 0x7f7ccc0c7520 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.007+0000 7f7cd22d2700 1 --2- 172.21.15.46:0/1888725112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ccc0cced0 0x7f7ccc14cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.007+0000 7f7cd22d2700 1 --2- 172.21.15.46:0/1888725112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ccc0cd2a0 0x7f7ccc0c6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.007+0000 7f7cd22d2700 1 -- 172.21.15.46:0/1888725112 >> 172.21.15.46:0/1888725112 conn(0x7f7ccc068190 msgr2=0x7f7ccc0cae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:56.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.007+0000 7f7cd22d2700 1 -- 172.21.15.46:0/1888725112 shutdown_connections 2024-01-27T01:08:56.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.007+0000 7f7cd22d2700 1 -- 172.21.15.46:0/1888725112 wait complete. 2024-01-27T01:08:56.008 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 1 to deep-scrub 2024-01-27T01:08:56.020 INFO:tasks.ceph:Scrubbing osd.2 2024-01-27T01:08:56.020 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-01-27T01:08:56.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.156+0000 7f118afb9700 1 Processor -- start 2024-01-27T01:08:56.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.156+0000 7f118afb9700 1 -- start start 2024-01-27T01:08:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.157+0000 7f118afb9700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1184157670 0x7f118415b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.157+0000 7f118afb9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11840c7b80 0x7f11840679b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.157+0000 7f118afb9700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11840cccf0 con 0x7f1184157a40 2024-01-27T01:08:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.157+0000 7f118afb9700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11840cb720 con 0x7f1184157670 2024-01-27T01:08:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.157+0000 7f118afb9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11840cb140 con 0x7f11840c7b80 2024-01-27T01:08:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.157+0000 7f1188d55700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1184157670 0x7f118415b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.157+0000 7f1188d55700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1184157670 0x7f118415b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55418/0 (socket says 172.21.15.46:55418) 2024-01-27T01:08:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.157+0000 7f1188d55700 1 -- 172.21.15.46:0/4025036402 learned_addr learned my addr 172.21.15.46:0/4025036402 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.158+0000 7f1189556700 1 --2- 172.21.15.46:0/4025036402 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11840c7b80 0x7f11840679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.158+0000 7f1188d55700 1 -- 172.21.15.46:0/4025036402 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11840c7b80 msgr2=0x7f11840679b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.158+0000 7f1188d55700 1 --2- 172.21.15.46:0/4025036402 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11840c7b80 0x7f11840679b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.158+0000 7f1188d55700 1 -- 172.21.15.46:0/4025036402 >> v1:172.21.15.106:6789/0 conn(0x7f1184157a40 legacy=0x7f118415b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.158+0000 7f1188d55700 1 -- 172.21.15.46:0/4025036402 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11840ccf40 con 0x7f1184157670 2024-01-27T01:08:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.158+0000 7f1188d55700 1 --2- 172.21.15.46:0/4025036402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1184157670 0x7f118415b3a0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7f1174009990 tx=0x7f1174009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85a039dc9deee3d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.158+0000 7f11837fe700 1 -- 172.21.15.46:0/4025036402 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174005c10 con 0x7f1184157670 2024-01-27T01:08:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.158+0000 7f11837fe700 1 -- 172.21.15.46:0/4025036402 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1174005d70 con 0x7f1184157670 2024-01-27T01:08:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.158+0000 7f11837fe700 1 -- 172.21.15.46:0/4025036402 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117400b880 con 0x7f1184157670 2024-01-27T01:08:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.159+0000 7f118afb9700 1 -- 172.21.15.46:0/4025036402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1184157670 msgr2=0x7f118415b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.159+0000 7f118afb9700 1 --2- 172.21.15.46:0/4025036402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1184157670 0x7f118415b3a0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7f1174009990 tx=0x7f1174009220 comp rx=0 tx=0).stop 2024-01-27T01:08:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.159+0000 7f118afb9700 1 -- 172.21.15.46:0/4025036402 shutdown_connections 2024-01-27T01:08:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.159+0000 7f118afb9700 1 --2- 172.21.15.46:0/4025036402 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11840c7b80 0x7f11840679b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.159+0000 7f118afb9700 1 --2- 172.21.15.46:0/4025036402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1184157670 0x7f118415b3a0 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.159+0000 7f118afb9700 1 -- 172.21.15.46:0/4025036402 >> 172.21.15.46:0/4025036402 conn(0x7f11840692a0 msgr2=0x7f11840cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:56.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.159+0000 7f118afb9700 1 -- 172.21.15.46:0/4025036402 shutdown_connections 2024-01-27T01:08:56.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.159+0000 7f118afb9700 1 -- 172.21.15.46:0/4025036402 wait complete. 2024-01-27T01:08:56.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.160+0000 7f118afb9700 1 Processor -- start 2024-01-27T01:08:56.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.160+0000 7f118afb9700 1 -- start start 2024-01-27T01:08:56.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.161+0000 7f118afb9700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11840c7b80 0x7f11840b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.161+0000 7f118afb9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1184157670 0x7f11840c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.161+0000 7f118afb9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1184157a40 0x7f11840abe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.161+0000 7f118afb9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1184169fd0 con 0x7f1184157670 2024-01-27T01:08:56.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.161+0000 7f118afb9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11840b1c60 con 0x7f1184157a40 2024-01-27T01:08:56.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.161+0000 7f118afb9700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11840b1f40 con 0x7f11840c7b80 2024-01-27T01:08:56.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.161+0000 7f1183fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1184157670 0x7f11840c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.161+0000 7f1183fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1184157670 0x7f11840c6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55426/0 (socket says 172.21.15.46:55426) 2024-01-27T01:08:56.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.161+0000 7f1183fff700 1 -- 172.21.15.46:0/146648951 learned_addr learned my addr 172.21.15.46:0/146648951 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:56.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.161+0000 7f1189556700 1 --2- 172.21.15.46:0/146648951 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1184157a40 0x7f11840abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.161+0000 7f1188d55700 1 --2- 172.21.15.46:0/146648951 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11840c7b80 0x7f11840b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.162+0000 7f1183fff700 1 -- 172.21.15.46:0/146648951 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11840c7b80 msgr2=0x7f11840b2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.162+0000 7f1183fff700 1 --2- 172.21.15.46:0/146648951 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11840c7b80 0x7f11840b2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.162+0000 7f1183fff700 1 -- 172.21.15.46:0/146648951 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1184157a40 msgr2=0x7f11840abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.162+0000 7f1183fff700 1 --2- 172.21.15.46:0/146648951 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1184157a40 0x7f11840abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.162+0000 7f1183fff700 1 -- 172.21.15.46:0/146648951 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1174009480 con 0x7f1184157670 2024-01-27T01:08:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.162+0000 7f1183fff700 1 --2- 172.21.15.46:0/146648951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1184157670 0x7f11840c6de0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f117800c590 tx=0x7f117800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.162+0000 7f1180ff9700 1 -- 172.21.15.46:0/146648951 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1178011590 con 0x7f1184157670 2024-01-27T01:08:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.162+0000 7f1180ff9700 1 -- 172.21.15.46:0/146648951 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1178011710 con 0x7f1184157670 2024-01-27T01:08:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.162+0000 7f1180ff9700 1 -- 172.21.15.46:0/146648951 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1178011ad0 con 0x7f1184157670 2024-01-27T01:08:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.162+0000 7f118afb9700 1 -- 172.21.15.46:0/146648951 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11840b2220 con 0x7f1184157670 2024-01-27T01:08:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.162+0000 7f118afb9700 1 -- 172.21.15.46:0/146648951 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11840b27e0 con 0x7f1184157670 2024-01-27T01:08:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.163+0000 7f118afb9700 1 -- 172.21.15.46:0/146648951 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f118404c710 con 0x7f1184157670 2024-01-27T01:08:56.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.164+0000 7f1180ff9700 1 -- 172.21.15.46:0/146648951 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f117801b3b0 con 0x7f1184157670 2024-01-27T01:08:56.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.164+0000 7f1180ff9700 1 --2- 172.21.15.46:0/146648951 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1170066f30 0x7f11700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.164+0000 7f1180ff9700 1 -- 172.21.15.46:0/146648951 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f1178084430 con 0x7f1184157670 2024-01-27T01:08:56.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.164+0000 7f1180ff9700 1 --2- 172.21.15.46:0/146648951 >> [v2:172.21.15.46:6808/43680,v1:172.21.15.46:6809/43680] conn(0x7f117006dd40 0x7f11700701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.164+0000 7f1180ff9700 1 -- 172.21.15.46:0/146648951 --> [v2:172.21.15.46:6808/43680,v1:172.21.15.46:6809/43680] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1170070800 con 0x7f117006dd40 2024-01-27T01:08:56.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.164+0000 7f1180ff9700 1 -- 172.21.15.46:0/146648951 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=269) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1178084800 con 0x7f1184157670 2024-01-27T01:08:56.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.164+0000 7f1188d55700 1 --2- 172.21.15.46:0/146648951 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1170066f30 0x7f11700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.164+0000 7f1189556700 1 --2- 172.21.15.46:0/146648951 >> [v2:172.21.15.46:6808/43680,v1:172.21.15.46:6809/43680] conn(0x7f117006dd40 0x7f11700701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.165+0000 7f1188d55700 1 --2- 172.21.15.46:0/146648951 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1170066f30 0x7f11700693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1174009260 tx=0x7f1174016000 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.165+0000 7f1189556700 1 --2- 172.21.15.46:0/146648951 >> [v2:172.21.15.46:6808/43680,v1:172.21.15.46:6809/43680] conn(0x7f117006dd40 0x7f11700701d0 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.166+0000 7f1180ff9700 1 -- 172.21.15.46:0/146648951 <== osd.2 v2:172.21.15.46:6808/43680 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f1170070800 con 0x7f117006dd40 2024-01-27T01:08:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.180+0000 7f118afb9700 1 -- 172.21.15.46:0/146648951 --> [v2:172.21.15.46:6808/43680,v1:172.21.15.46:6809/43680] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f118406fd10 con 0x7f117006dd40 2024-01-27T01:08:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.180+0000 7f1180ff9700 1 -- 172.21.15.46:0/146648951 <== osd.2 v2:172.21.15.46:6808/43680 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f118406fd10 con 0x7f117006dd40 2024-01-27T01:08:56.181 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:08:56.181 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-27T01:08:56.181 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:08:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.181+0000 7f118afb9700 1 -- 172.21.15.46:0/146648951 >> [v2:172.21.15.46:6808/43680,v1:172.21.15.46:6809/43680] conn(0x7f117006dd40 msgr2=0x7f11700701d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.181+0000 7f118afb9700 1 --2- 172.21.15.46:0/146648951 >> [v2:172.21.15.46:6808/43680,v1:172.21.15.46:6809/43680] conn(0x7f117006dd40 0x7f11700701d0 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.181+0000 7f118afb9700 1 -- 172.21.15.46:0/146648951 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1170066f30 msgr2=0x7f11700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.181+0000 7f118afb9700 1 --2- 172.21.15.46:0/146648951 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1170066f30 0x7f11700693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1174009260 tx=0x7f1174016000 comp rx=0 tx=0).stop 2024-01-27T01:08:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.181+0000 7f118afb9700 1 -- 172.21.15.46:0/146648951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1184157670 msgr2=0x7f11840c6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.181+0000 7f118afb9700 1 --2- 172.21.15.46:0/146648951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1184157670 0x7f11840c6de0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f117800c590 tx=0x7f117800c8a0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.181+0000 7f1189556700 1 -- 172.21.15.46:0/146648951 reap_dead start 2024-01-27T01:08:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.181+0000 7f118afb9700 1 -- 172.21.15.46:0/146648951 shutdown_connections 2024-01-27T01:08:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.181+0000 7f118afb9700 1 -- 172.21.15.46:0/146648951 >> 172.21.15.46:0/146648951 conn(0x7f11840692a0 msgr2=0x7f118406bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.181+0000 7f118afb9700 1 -- 172.21.15.46:0/146648951 shutdown_connections 2024-01-27T01:08:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.181+0000 7f118afb9700 1 -- 172.21.15.46:0/146648951 wait complete. 2024-01-27T01:08:56.194 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-01-27T01:08:56.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.327+0000 7fb6bab77700 1 Processor -- start 2024-01-27T01:08:56.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.328+0000 7fb6bab77700 1 -- start start 2024-01-27T01:08:56.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.329+0000 7fb6bab77700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6b40cced0 0x7fb6b415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.329+0000 7fb6bab77700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb6b40cd2a0 0x7fb6b40c3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.329+0000 7fb6bab77700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b4068ac0 con 0x7fb6b40cf0b0 2024-01-27T01:08:56.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.329+0000 7fb6bab77700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6b40c4c40 con 0x7fb6b40cd2a0 2024-01-27T01:08:56.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.329+0000 7fb6bab77700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b40c8c10 con 0x7fb6b40cced0 2024-01-27T01:08:56.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.329+0000 7fb6b9114700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb6b40cd2a0 0x7fb6b40c3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.329+0000 7fb6b8913700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6b40cced0 0x7fb6b415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.329+0000 7fb6b3fff700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fb6b40cf0b0 0x7fb6b415dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:47854/0 (socket says 172.21.15.46:47854) 2024-01-27T01:08:56.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.329+0000 7fb6b3fff700 1 -- 172.21.15.46:0/1756593867 learned_addr learned my addr 172.21.15.46:0/1756593867 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:56.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.329+0000 7fb6b8913700 1 -- 172.21.15.46:0/1756593867 >> v2:172.21.15.106:3300/0 conn(0x7fb6b40cd2a0 msgr2=0x7fb6b40c3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:56.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.330+0000 7fb6b8913700 1 --2- 172.21.15.46:0/1756593867 >> v2:172.21.15.106:3300/0 conn(0x7fb6b40cd2a0 0x7fb6b40c3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.330+0000 7fb6b8913700 1 -- 172.21.15.46:0/1756593867 >> v1:172.21.15.106:6789/0 conn(0x7fb6b40cf0b0 legacy=0x7fb6b415dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.330+0000 7fb6b8913700 1 -- 172.21.15.46:0/1756593867 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b40c9470 con 0x7fb6b40cced0 2024-01-27T01:08:56.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.330+0000 7fb6b8913700 1 --2- 172.21.15.46:0/1756593867 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6b40cced0 0x7fb6b415d6f0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb6a40098e0 tx=0x7fb6a40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=364d26a5ad14d8fc server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:56.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.330+0000 7fb6b37fe700 1 -- 172.21.15.46:0/1756593867 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a400b640 con 0x7fb6b40cced0 2024-01-27T01:08:56.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.330+0000 7fb6b37fe700 1 -- 172.21.15.46:0/1756593867 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a400b7a0 con 0x7fb6b40cced0 2024-01-27T01:08:56.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.330+0000 7fb6b37fe700 1 -- 172.21.15.46:0/1756593867 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a400bb40 con 0x7fb6b40cced0 2024-01-27T01:08:56.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.331+0000 7fb6bab77700 1 -- 172.21.15.46:0/1756593867 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6b40cced0 msgr2=0x7fb6b415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.331+0000 7fb6bab77700 1 --2- 172.21.15.46:0/1756593867 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6b40cced0 0x7fb6b415d6f0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb6a40098e0 tx=0x7fb6a40091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.331+0000 7fb6bab77700 1 -- 172.21.15.46:0/1756593867 shutdown_connections 2024-01-27T01:08:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.331+0000 7fb6bab77700 1 --2- 172.21.15.46:0/1756593867 >> v2:172.21.15.106:3300/0 conn(0x7fb6b40cd2a0 0x7fb6b40c3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.331+0000 7fb6bab77700 1 --2- 172.21.15.46:0/1756593867 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6b40cced0 0x7fb6b415d6f0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.331+0000 7fb6bab77700 1 -- 172.21.15.46:0/1756593867 >> 172.21.15.46:0/1756593867 conn(0x7fb6b4068190 msgr2=0x7fb6b40ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.331+0000 7fb6bab77700 1 -- 172.21.15.46:0/1756593867 shutdown_connections 2024-01-27T01:08:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.331+0000 7fb6bab77700 1 -- 172.21.15.46:0/1756593867 wait complete. 2024-01-27T01:08:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.332+0000 7fb6bab77700 1 Processor -- start 2024-01-27T01:08:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.333+0000 7fb6bab77700 1 -- start start 2024-01-27T01:08:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.333+0000 7fb6bab77700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6b40cced0 0x7fb6b40c7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6bab77700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6b40cd2a0 0x7fb6b4156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6bab77700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6b40cf0b0 0x7fb6b4068c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6bab77700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b40c6fc0 con 0x7fb6b40cced0 2024-01-27T01:08:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6bab77700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b40c63b0 con 0x7fb6b40cd2a0 2024-01-27T01:08:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6bab77700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b40c6690 con 0x7fb6b40cf0b0 2024-01-27T01:08:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6b8913700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6b40cced0 0x7fb6b40c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6b8913700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6b40cced0 0x7fb6b40c7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55436/0 (socket says 172.21.15.46:55436) 2024-01-27T01:08:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6b8913700 1 -- 172.21.15.46:0/610532026 learned_addr learned my addr 172.21.15.46:0/610532026 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6b3fff700 1 --2- 172.21.15.46:0/610532026 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6b40cd2a0 0x7fb6b4156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6b9114700 1 --2- 172.21.15.46:0/610532026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6b40cf0b0 0x7fb6b4068c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6b8913700 1 -- 172.21.15.46:0/610532026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6b40cf0b0 msgr2=0x7fb6b4068c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6b8913700 1 --2- 172.21.15.46:0/610532026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6b40cf0b0 0x7fb6b4068c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6b8913700 1 -- 172.21.15.46:0/610532026 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6b40cd2a0 msgr2=0x7fb6b4156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6b8913700 1 --2- 172.21.15.46:0/610532026 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6b40cd2a0 0x7fb6b4156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6b8913700 1 -- 172.21.15.46:0/610532026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a4009420 con 0x7fb6b40cced0 2024-01-27T01:08:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6b8913700 1 --2- 172.21.15.46:0/610532026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6b40cced0 0x7fb6b40c7450 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40c8d50 tx=0x7fb6a4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6b0ff9700 1 -- 172.21.15.46:0/610532026 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a401bc40 con 0x7fb6b40cced0 2024-01-27T01:08:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.334+0000 7fb6b0ff9700 1 -- 172.21.15.46:0/610532026 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a401bde0 con 0x7fb6b40cced0 2024-01-27T01:08:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.335+0000 7fb6bab77700 1 -- 172.21.15.46:0/610532026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b40c6970 con 0x7fb6b40cced0 2024-01-27T01:08:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.335+0000 7fb6b0ff9700 1 -- 172.21.15.46:0/610532026 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4011670 con 0x7fb6b40cced0 2024-01-27T01:08:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.335+0000 7fb6bab77700 1 -- 172.21.15.46:0/610532026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b4156640 con 0x7fb6b40cced0 2024-01-27T01:08:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.336+0000 7fb6bab77700 1 -- 172.21.15.46:0/610532026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b404c710 con 0x7fb6b40cced0 2024-01-27T01:08:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.336+0000 7fb6b0ff9700 1 -- 172.21.15.46:0/610532026 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb6a40119e0 con 0x7fb6b40cced0 2024-01-27T01:08:56.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.336+0000 7fb6b0ff9700 1 --2- 172.21.15.46:0/610532026 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6a0066f30 0x7fb6a00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.337+0000 7fb6b0ff9700 1 -- 172.21.15.46:0/610532026 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7fb6a4085c90 con 0x7fb6b40cced0 2024-01-27T01:08:56.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.337+0000 7fb6b3fff700 1 --2- 172.21.15.46:0/610532026 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6a0066f30 0x7fb6a00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.337+0000 7fb6b3fff700 1 --2- 172.21.15.46:0/610532026 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6a0066f30 0x7fb6a00693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb6b415d5b0 tx=0x7fb6a8007eb0 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:56.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.341+0000 7fb6b0ff9700 1 -- 172.21.15.46:0/610532026 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb6a4052b30 con 0x7fb6b40cced0 2024-01-27T01:08:56.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.550+0000 7fb6bab77700 1 -- 172.21.15.46:0/610532026 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6b40c7100 con 0x7fb6a0066f30 2024-01-27T01:08:56.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.550+0000 7fb6b0ff9700 1 -- 172.21.15.46:0/610532026 <== mgr.10565 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb6a8004ce0 con 0x7fb6a0066f30 2024-01-27T01:08:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.553+0000 7fb6bab77700 1 -- 172.21.15.46:0/610532026 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6a0066f30 msgr2=0x7fb6a00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.553+0000 7fb6bab77700 1 --2- 172.21.15.46:0/610532026 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6a0066f30 0x7fb6a00693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb6b415d5b0 tx=0x7fb6a8007eb0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.553+0000 7fb6bab77700 1 -- 172.21.15.46:0/610532026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6b40cced0 msgr2=0x7fb6b40c7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.553+0000 7fb6bab77700 1 --2- 172.21.15.46:0/610532026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6b40cced0 0x7fb6b40c7450 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40c8d50 tx=0x7fb6a4017000 comp rx=0 tx=0).stop 2024-01-27T01:08:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.553+0000 7fb6bab77700 1 -- 172.21.15.46:0/610532026 shutdown_connections 2024-01-27T01:08:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.553+0000 7fb6bab77700 1 --2- 172.21.15.46:0/610532026 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb6a0066f30 0x7fb6a00693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.553+0000 7fb6bab77700 1 --2- 172.21.15.46:0/610532026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6b40cced0 0x7fb6b40c7450 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.553+0000 7fb6bab77700 1 --2- 172.21.15.46:0/610532026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6b40cf0b0 0x7fb6b4068c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.553+0000 7fb6bab77700 1 --2- 172.21.15.46:0/610532026 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6b40cd2a0 0x7fb6b4156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.553+0000 7fb6bab77700 1 -- 172.21.15.46:0/610532026 >> 172.21.15.46:0/610532026 conn(0x7fb6b4068190 msgr2=0x7fb6b406bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.553+0000 7fb6bab77700 1 -- 172.21.15.46:0/610532026 shutdown_connections 2024-01-27T01:08:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.553+0000 7fb6bab77700 1 -- 172.21.15.46:0/610532026 wait complete. 2024-01-27T01:08:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 2 to deep-scrub 2024-01-27T01:08:56.566 INFO:tasks.ceph:Scrubbing osd.3 2024-01-27T01:08:56.566 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-01-27T01:08:56.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.703+0000 7f98d6edc700 1 Processor -- start 2024-01-27T01:08:56.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.703+0000 7f98d6edc700 1 -- start start 2024-01-27T01:08:56.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.703+0000 7f98d6edc700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f98d01599c0 0x7f98d00c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.703+0000 7f98d6edc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98d00c9bf0 0x7f98d00cb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.704+0000 7f98d6edc700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d00cb0b0 con 0x7f98d01595f0 2024-01-27T01:08:56.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.704+0000 7f98d6edc700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d00ca070 con 0x7f98d01599c0 2024-01-27T01:08:56.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.704+0000 7f98d6edc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d0159e10 con 0x7f98d00c9bf0 2024-01-27T01:08:56.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.704+0000 7f98cffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f98d01599c0 0x7f98d00c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.704+0000 7f98cffff700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f98d01599c0 0x7f98d00c96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55450/0 (socket says 172.21.15.46:55450) 2024-01-27T01:08:56.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.704+0000 7f98cffff700 1 -- 172.21.15.46:0/2945721068 learned_addr learned my addr 172.21.15.46:0/2945721068 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:56.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.704+0000 7f98d5479700 1 --2- 172.21.15.46:0/2945721068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98d00c9bf0 0x7f98d00cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.704+0000 7f98cffff700 1 -- 172.21.15.46:0/2945721068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98d00c9bf0 msgr2=0x7f98d00cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:56.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.704+0000 7f98cffff700 1 --2- 172.21.15.46:0/2945721068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98d00c9bf0 0x7f98d00cb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.704+0000 7f98cffff700 1 -- 172.21.15.46:0/2945721068 >> v1:172.21.15.106:6789/0 conn(0x7f98d01595f0 legacy=0x7f98d0069000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.704+0000 7f98cffff700 1 -- 172.21.15.46:0/2945721068 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d00ca4a0 con 0x7f98d01599c0 2024-01-27T01:08:56.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.704+0000 7f98cffff700 1 --2- 172.21.15.46:0/2945721068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98d01599c0 0x7f98d00c96b0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f98bc009990 tx=0x7f98bc009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28ccac6107602430 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:56.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.704+0000 7f98cf7fe700 1 -- 172.21.15.46:0/2945721068 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc009e80 con 0x7f98d01599c0 2024-01-27T01:08:56.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.705+0000 7f98cf7fe700 1 -- 172.21.15.46:0/2945721068 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98bc005c10 con 0x7f98d01599c0 2024-01-27T01:08:56.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.705+0000 7f98cf7fe700 1 -- 172.21.15.46:0/2945721068 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc00b6e0 con 0x7f98d01599c0 2024-01-27T01:08:56.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.705+0000 7f98d6edc700 1 -- 172.21.15.46:0/2945721068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98d01599c0 msgr2=0x7f98d00c96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.705+0000 7f98d6edc700 1 --2- 172.21.15.46:0/2945721068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98d01599c0 0x7f98d00c96b0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f98bc009990 tx=0x7f98bc009220 comp rx=0 tx=0).stop 2024-01-27T01:08:56.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.705+0000 7f98d6edc700 1 -- 172.21.15.46:0/2945721068 shutdown_connections 2024-01-27T01:08:56.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.705+0000 7f98d6edc700 1 --2- 172.21.15.46:0/2945721068 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98d00c9bf0 0x7f98d00cb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.705+0000 7f98d6edc700 1 --2- 172.21.15.46:0/2945721068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98d01599c0 0x7f98d00c96b0 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.705+0000 7f98d6edc700 1 -- 172.21.15.46:0/2945721068 >> 172.21.15.46:0/2945721068 conn(0x7f98d00679b0 msgr2=0x7f98d0067db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:56.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.705+0000 7f98d6edc700 1 -- 172.21.15.46:0/2945721068 shutdown_connections 2024-01-27T01:08:56.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.705+0000 7f98d6edc700 1 -- 172.21.15.46:0/2945721068 wait complete. 2024-01-27T01:08:56.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.706+0000 7f98d6edc700 1 Processor -- start 2024-01-27T01:08:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.707+0000 7f98d6edc700 1 -- start start 2024-01-27T01:08:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.707+0000 7f98d6edc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98d00c9bf0 0x7f98d0153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.707+0000 7f98d6edc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98d01595f0 0x7f98d00b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.707+0000 7f98d6edc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98d01599c0 0x7f98d00b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.707+0000 7f98d6edc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d00b3fd0 con 0x7f98d00c9bf0 2024-01-27T01:08:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.707+0000 7f98d6edc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d015bde0 con 0x7f98d01595f0 2024-01-27T01:08:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.707+0000 7f98d6edc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d015c0c0 con 0x7f98d01599c0 2024-01-27T01:08:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.707+0000 7f98d4c78700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98d00c9bf0 0x7f98d0153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.707+0000 7f98d4c78700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98d00c9bf0 0x7f98d0153a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55456/0 (socket says 172.21.15.46:55456) 2024-01-27T01:08:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.707+0000 7f98d4c78700 1 -- 172.21.15.46:0/845249839 learned_addr learned my addr 172.21.15.46:0/845249839 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.708+0000 7f98cffff700 1 --2- 172.21.15.46:0/845249839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98d01595f0 0x7f98d00b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.708+0000 7f98d5479700 1 --2- 172.21.15.46:0/845249839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98d01599c0 0x7f98d00b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.708+0000 7f98d4c78700 1 -- 172.21.15.46:0/845249839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98d01599c0 msgr2=0x7f98d00b32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.708+0000 7f98d4c78700 1 --2- 172.21.15.46:0/845249839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98d01599c0 0x7f98d00b32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.708+0000 7f98d4c78700 1 -- 172.21.15.46:0/845249839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98d01595f0 msgr2=0x7f98d00b2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.708+0000 7f98d4c78700 1 --2- 172.21.15.46:0/845249839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98d01595f0 0x7f98d00b2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.708+0000 7f98d4c78700 1 -- 172.21.15.46:0/845249839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc009480 con 0x7f98d00c9bf0 2024-01-27T01:08:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.708+0000 7f98d4c78700 1 --2- 172.21.15.46:0/845249839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98d00c9bf0 0x7f98d0153a50 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c590 tx=0x7f98c400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.708+0000 7f98ccff9700 1 -- 172.21.15.46:0/845249839 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4011590 con 0x7f98d00c9bf0 2024-01-27T01:08:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.708+0000 7f98ccff9700 1 -- 172.21.15.46:0/845249839 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c4011710 con 0x7f98d00c9bf0 2024-01-27T01:08:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.708+0000 7f98d6edc700 1 -- 172.21.15.46:0/845249839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d015c3a0 con 0x7f98d00c9bf0 2024-01-27T01:08:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.708+0000 7f98ccff9700 1 -- 172.21.15.46:0/845249839 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4011ad0 con 0x7f98d00c9bf0 2024-01-27T01:08:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.708+0000 7f98d6edc700 1 -- 172.21.15.46:0/845249839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d0153f90 con 0x7f98d00c9bf0 2024-01-27T01:08:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.709+0000 7f98d6edc700 1 -- 172.21.15.46:0/845249839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f98d00c5690 con 0x7f98d00c9bf0 2024-01-27T01:08:56.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.710+0000 7f98ccff9700 1 -- 172.21.15.46:0/845249839 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f98c401b3b0 con 0x7f98d00c9bf0 2024-01-27T01:08:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.710+0000 7f98ccff9700 1 --2- 172.21.15.46:0/845249839 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f98b4066f60 0x7f98b4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.710+0000 7f98ccff9700 1 -- 172.21.15.46:0/845249839 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f98c4084430 con 0x7f98d00c9bf0 2024-01-27T01:08:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.710+0000 7f98cffff700 1 --2- 172.21.15.46:0/845249839 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f98b4066f60 0x7f98b4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.710+0000 7f98ccff9700 1 --2- 172.21.15.46:0/845249839 >> [v2:172.21.15.46:6815/43691,v1:172.21.15.46:6817/43691] conn(0x7f98b406dd70 0x7f98b4070200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.710+0000 7f98ccff9700 1 -- 172.21.15.46:0/845249839 --> [v2:172.21.15.46:6815/43691,v1:172.21.15.46:6817/43691] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f98b4070830 con 0x7f98b406dd70 2024-01-27T01:08:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.711+0000 7f98ccff9700 1 -- 172.21.15.46:0/845249839 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=269) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98c4084800 con 0x7f98d00c9bf0 2024-01-27T01:08:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.711+0000 7f98d5479700 1 --2- 172.21.15.46:0/845249839 >> [v2:172.21.15.46:6815/43691,v1:172.21.15.46:6817/43691] conn(0x7f98b406dd70 0x7f98b4070200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.711+0000 7f98cffff700 1 --2- 172.21.15.46:0/845249839 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f98b4066f60 0x7f98b4069410 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f98bc009260 tx=0x7f98bc016000 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.711+0000 7f98d5479700 1 --2- 172.21.15.46:0/845249839 >> [v2:172.21.15.46:6815/43691,v1:172.21.15.46:6817/43691] conn(0x7f98b406dd70 0x7f98b4070200 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:56.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.712+0000 7f98ccff9700 1 -- 172.21.15.46:0/845249839 <== osd.3 v2:172.21.15.46:6815/43691 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f98b4070830 con 0x7f98b406dd70 2024-01-27T01:08:56.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.726+0000 7f98d6edc700 1 -- 172.21.15.46:0/845249839 --> [v2:172.21.15.46:6815/43691,v1:172.21.15.46:6817/43691] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f98d004c710 con 0x7f98b406dd70 2024-01-27T01:08:56.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.727+0000 7f98ccff9700 1 -- 172.21.15.46:0/845249839 <== osd.3 v2:172.21.15.46:6815/43691 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f98d004c710 con 0x7f98b406dd70 2024-01-27T01:08:56.727 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:08:56.727 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-27T01:08:56.727 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:08:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.727+0000 7f98d6edc700 1 -- 172.21.15.46:0/845249839 >> [v2:172.21.15.46:6815/43691,v1:172.21.15.46:6817/43691] conn(0x7f98b406dd70 msgr2=0x7f98b4070200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.727+0000 7f98d6edc700 1 --2- 172.21.15.46:0/845249839 >> [v2:172.21.15.46:6815/43691,v1:172.21.15.46:6817/43691] conn(0x7f98b406dd70 0x7f98b4070200 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.727+0000 7f98d6edc700 1 -- 172.21.15.46:0/845249839 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f98b4066f60 msgr2=0x7f98b4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.727+0000 7f98d6edc700 1 --2- 172.21.15.46:0/845249839 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f98b4066f60 0x7f98b4069410 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f98bc009260 tx=0x7f98bc016000 comp rx=0 tx=0).stop 2024-01-27T01:08:56.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.727+0000 7f98d6edc700 1 -- 172.21.15.46:0/845249839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98d00c9bf0 msgr2=0x7f98d0153a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.727+0000 7f98d6edc700 1 --2- 172.21.15.46:0/845249839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98d00c9bf0 0x7f98d0153a50 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c590 tx=0x7f98c400c8a0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.727+0000 7f98d5479700 1 -- 172.21.15.46:0/845249839 reap_dead start 2024-01-27T01:08:56.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.727+0000 7f98d6edc700 1 -- 172.21.15.46:0/845249839 shutdown_connections 2024-01-27T01:08:56.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.727+0000 7f98d6edc700 1 -- 172.21.15.46:0/845249839 >> 172.21.15.46:0/845249839 conn(0x7f98d00679b0 msgr2=0x7f98d00c37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:56.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.727+0000 7f98d6edc700 1 -- 172.21.15.46:0/845249839 shutdown_connections 2024-01-27T01:08:56.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.727+0000 7f98d6edc700 1 -- 172.21.15.46:0/845249839 wait complete. 2024-01-27T01:08:56.740 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-01-27T01:08:56.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.873+0000 7f1c7ecc3700 1 Processor -- start 2024-01-27T01:08:56.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.873+0000 7f1c7ecc3700 1 -- start start 2024-01-27T01:08:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.874+0000 7f1c7ecc3700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1c781599b0 0x7f1c780c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.874+0000 7f1c7ecc3700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c780cf510 con 0x7f1c78159d80 2024-01-27T01:08:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.875+0000 7f1c7ecc3700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c780c45b0 con 0x7f1c780cf0b0 2024-01-27T01:08:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.875+0000 7f1c7ecc3700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c780c43a0 con 0x7f1c781599b0 2024-01-27T01:08:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.875+0000 7f1c77fff700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1c78159d80 0x7f1c78069640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32772/0 (socket says 172.21.15.46:32772) 2024-01-27T01:08:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.875+0000 7f1c77fff700 1 -- 172.21.15.46:0/137240882 learned_addr learned my addr 172.21.15.46:0/137240882 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.875+0000 7f1c7ca5f700 1 --2- 172.21.15.46:0/137240882 >> v2:172.21.15.106:3300/0 conn(0x7f1c781599b0 0x7f1c780c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.875+0000 7f1c777fe700 1 -- 172.21.15.46:0/137240882 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351589138 0 0) 0x7f1c780cf510 con 0x7f1c78159d80 2024-01-27T01:08:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.875+0000 7f1c777fe700 1 -- 172.21.15.46:0/137240882 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c64003570 con 0x7f1c78159d80 2024-01-27T01:08:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.875+0000 7f1c777fe700 1 -- 172.21.15.46:0/137240882 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759143282 0 0) 0x7f1c780c45b0 con 0x7f1c780cf0b0 2024-01-27T01:08:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.875+0000 7f1c777fe700 1 -- 172.21.15.46:0/137240882 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c780cf510 con 0x7f1c780cf0b0 2024-01-27T01:08:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.875+0000 7f1c7ca5f700 1 -- 172.21.15.46:0/137240882 >> v1:172.21.15.106:6789/0 conn(0x7f1c780cf0b0 legacy=0x7f1c78069d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.875+0000 7f1c7ca5f700 1 -- 172.21.15.46:0/137240882 >> v1:172.21.15.46:6789/0 conn(0x7f1c78159d80 legacy=0x7f1c78069640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.875+0000 7f1c777fe700 1 -- 172.21.15.46:0/137240882 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 913129439 0 0) 0x7f1c64003570 con 0x7f1c78159d80 2024-01-27T01:08:56.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.875+0000 7f1c7ca5f700 1 -- 172.21.15.46:0/137240882 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c780c4890 con 0x7f1c781599b0 2024-01-27T01:08:56.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.875+0000 7f1c7ca5f700 1 --2- 172.21.15.46:0/137240882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c781599b0 0x7f1c780c8c30 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f1c680098d0 tx=0x7f1c68009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92f1fb254d792b9d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:56.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.876+0000 7f1c777fe700 1 -- 172.21.15.46:0/137240882 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c68005c10 con 0x7f1c781599b0 2024-01-27T01:08:56.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.876+0000 7f1c777fe700 1 -- 172.21.15.46:0/137240882 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c68005d70 con 0x7f1c781599b0 2024-01-27T01:08:56.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.876+0000 7f1c777fe700 1 -- 172.21.15.46:0/137240882 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c68006260 con 0x7f1c781599b0 2024-01-27T01:08:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.876+0000 7f1c7ecc3700 1 -- 172.21.15.46:0/137240882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c781599b0 msgr2=0x7f1c780c8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.876+0000 7f1c7ecc3700 1 --2- 172.21.15.46:0/137240882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c781599b0 0x7f1c780c8c30 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f1c680098d0 tx=0x7f1c68009120 comp rx=0 tx=0).stop 2024-01-27T01:08:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.876+0000 7f1c7ecc3700 1 -- 172.21.15.46:0/137240882 shutdown_connections 2024-01-27T01:08:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.876+0000 7f1c7ecc3700 1 --2- 172.21.15.46:0/137240882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c781599b0 0x7f1c780c8c30 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.876+0000 7f1c7ecc3700 1 -- 172.21.15.46:0/137240882 >> 172.21.15.46:0/137240882 conn(0x7f1c78087fa0 msgr2=0x7f1c780ce5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.876+0000 7f1c7ecc3700 1 -- 172.21.15.46:0/137240882 shutdown_connections 2024-01-27T01:08:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.876+0000 7f1c7ecc3700 1 -- 172.21.15.46:0/137240882 wait complete. 2024-01-27T01:08:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.878+0000 7f1c7ecc3700 1 Processor -- start 2024-01-27T01:08:56.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.878+0000 7f1c7ecc3700 1 -- start start 2024-01-27T01:08:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.879+0000 7f1c7ecc3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c780cf0b0 0x7f1c7815a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.879+0000 7f1c7ecc3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c781599b0 0x7f1c78154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.879+0000 7f1c7ecc3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c78159d80 0x7f1c78154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.879+0000 7f1c7ecc3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c781553a0 con 0x7f1c78159d80 2024-01-27T01:08:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.879+0000 7f1c7ecc3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c780b2ed0 con 0x7f1c780cf0b0 2024-01-27T01:08:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.879+0000 7f1c7ecc3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c780b22c0 con 0x7f1c781599b0 2024-01-27T01:08:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.879+0000 7f1c7d260700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c78159d80 0x7f1c78154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.879+0000 7f1c7d260700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c78159d80 0x7f1c78154e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55460/0 (socket says 172.21.15.46:55460) 2024-01-27T01:08:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.879+0000 7f1c7d260700 1 -- 172.21.15.46:0/753186720 learned_addr learned my addr 172.21.15.46:0/753186720 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.880+0000 7f1c7ca5f700 1 --2- 172.21.15.46:0/753186720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c780cf0b0 0x7f1c7815a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.880+0000 7f1c77fff700 1 --2- 172.21.15.46:0/753186720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c781599b0 0x7f1c78154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.880+0000 7f1c7d260700 1 -- 172.21.15.46:0/753186720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c781599b0 msgr2=0x7f1c78154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.880+0000 7f1c7d260700 1 --2- 172.21.15.46:0/753186720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c781599b0 0x7f1c78154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.880+0000 7f1c7d260700 1 -- 172.21.15.46:0/753186720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c780cf0b0 msgr2=0x7f1c7815a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.880+0000 7f1c7d260700 1 --2- 172.21.15.46:0/753186720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c780cf0b0 0x7f1c7815a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.880+0000 7f1c7d260700 1 -- 172.21.15.46:0/753186720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c680093c0 con 0x7f1c78159d80 2024-01-27T01:08:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.880+0000 7f1c7ca5f700 1 --2- 172.21.15.46:0/753186720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c780cf0b0 0x7f1c7815a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.880+0000 7f1c77fff700 1 --2- 172.21.15.46:0/753186720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c781599b0 0x7f1c78154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.880+0000 7f1c7d260700 1 --2- 172.21.15.46:0/753186720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c78159d80 0x7f1c78154e60 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000c890 tx=0x7f1c70002700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.880+0000 7f1c74ff9700 1 -- 172.21.15.46:0/753186720 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c70002030 con 0x7f1c78159d80 2024-01-27T01:08:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.880+0000 7f1c74ff9700 1 -- 172.21.15.46:0/753186720 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c700021b0 con 0x7f1c78159d80 2024-01-27T01:08:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.880+0000 7f1c7ecc3700 1 -- 172.21.15.46:0/753186720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c780b25a0 con 0x7f1c78159d80 2024-01-27T01:08:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.880+0000 7f1c74ff9700 1 -- 172.21.15.46:0/753186720 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c70012630 con 0x7f1c78159d80 2024-01-27T01:08:56.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.880+0000 7f1c7ecc3700 1 -- 172.21.15.46:0/753186720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c780b2b60 con 0x7f1c78159d80 2024-01-27T01:08:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.881+0000 7f1c7ecc3700 1 -- 172.21.15.46:0/753186720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c7804c710 con 0x7f1c78159d80 2024-01-27T01:08:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.882+0000 7f1c74ff9700 1 -- 172.21.15.46:0/753186720 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1c70012a20 con 0x7f1c78159d80 2024-01-27T01:08:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.882+0000 7f1c74ff9700 1 --2- 172.21.15.46:0/753186720 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c58066f30 0x7f1c580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.882+0000 7f1c74ff9700 1 -- 172.21.15.46:0/753186720 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f1c700848c0 con 0x7f1c78159d80 2024-01-27T01:08:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.882+0000 7f1c7ca5f700 1 --2- 172.21.15.46:0/753186720 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c58066f30 0x7f1c580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.883+0000 7f1c7ca5f700 1 --2- 172.21.15.46:0/753186720 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c58066f30 0x7f1c580693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009160 tx=0x7f1c68016000 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:56.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:56.887+0000 7f1c74ff9700 1 -- 172.21.15.46:0/753186720 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1c70051800 con 0x7f1c78159d80 2024-01-27T01:08:57.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.096+0000 7f1c7ecc3700 1 -- 172.21.15.46:0/753186720 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c7806dbc0 con 0x7f1c58066f30 2024-01-27T01:08:57.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.097+0000 7f1c74ff9700 1 -- 172.21.15.46:0/753186720 <== mgr.10565 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1c7806dbc0 con 0x7f1c58066f30 2024-01-27T01:08:57.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.099+0000 7f1c7ecc3700 1 -- 172.21.15.46:0/753186720 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c58066f30 msgr2=0x7f1c580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.099+0000 7f1c7ecc3700 1 --2- 172.21.15.46:0/753186720 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c58066f30 0x7f1c580693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009160 tx=0x7f1c68016000 comp rx=0 tx=0).stop 2024-01-27T01:08:57.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.099+0000 7f1c7ecc3700 1 -- 172.21.15.46:0/753186720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c78159d80 msgr2=0x7f1c78154e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.099+0000 7f1c7ecc3700 1 --2- 172.21.15.46:0/753186720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c78159d80 0x7f1c78154e60 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000c890 tx=0x7f1c70002700 comp rx=0 tx=0).stop 2024-01-27T01:08:57.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.099+0000 7f1c7ecc3700 1 -- 172.21.15.46:0/753186720 shutdown_connections 2024-01-27T01:08:57.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.099+0000 7f1c7ecc3700 1 --2- 172.21.15.46:0/753186720 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f1c58066f30 0x7f1c580693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.099+0000 7f1c7ecc3700 1 --2- 172.21.15.46:0/753186720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c78159d80 0x7f1c78154e60 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.099+0000 7f1c7ecc3700 1 --2- 172.21.15.46:0/753186720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c780cf0b0 0x7f1c7815a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.099+0000 7f1c7ecc3700 1 --2- 172.21.15.46:0/753186720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c781599b0 0x7f1c78154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.099+0000 7f1c7ecc3700 1 -- 172.21.15.46:0/753186720 >> 172.21.15.46:0/753186720 conn(0x7f1c78087fa0 msgr2=0x7f1c780cc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:57.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.099+0000 7f1c7ecc3700 1 -- 172.21.15.46:0/753186720 shutdown_connections 2024-01-27T01:08:57.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.099+0000 7f1c7ecc3700 1 -- 172.21.15.46:0/753186720 wait complete. 2024-01-27T01:08:57.102 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 3 to deep-scrub 2024-01-27T01:08:57.113 INFO:tasks.ceph:Scrubbing osd.4 2024-01-27T01:08:57.113 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-01-27T01:08:57.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.249+0000 7f2a6fb0b700 1 Processor -- start 2024-01-27T01:08:57.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.249+0000 7f2a6fb0b700 1 -- start start 2024-01-27T01:08:57.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.250+0000 7f2a6fb0b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2a68140f20 0x7f2a680caf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.250+0000 7f2a6fb0b700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a680ccc60 con 0x7f2a681412f0 2024-01-27T01:08:57.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.250+0000 7f2a6fb0b700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a680cc330 con 0x7f2a680b1360 2024-01-27T01:08:57.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.250+0000 7f2a6fb0b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a680cc610 con 0x7f2a68140f20 2024-01-27T01:08:57.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.250+0000 7f2a6e0a8700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2a681412f0 0x7f2a680cd0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32788/0 (socket says 172.21.15.46:32788) 2024-01-27T01:08:57.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.250+0000 7f2a6e0a8700 1 -- 172.21.15.46:0/670643123 learned_addr learned my addr 172.21.15.46:0/670643123 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:57.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.250+0000 7f2a6d8a7700 1 --2- 172.21.15.46:0/670643123 >> v2:172.21.15.106:3300/0 conn(0x7f2a68140f20 0x7f2a680caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.250+0000 7f2a6c8a5700 1 -- 172.21.15.46:0/670643123 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283913961 0 0) 0x7f2a680ccc60 con 0x7f2a681412f0 2024-01-27T01:08:57.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.250+0000 7f2a6c8a5700 1 -- 172.21.15.46:0/670643123 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a54003570 con 0x7f2a681412f0 2024-01-27T01:08:57.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6c8a5700 1 -- 172.21.15.46:0/670643123 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3002841361 0 0) 0x7f2a54003570 con 0x7f2a681412f0 2024-01-27T01:08:57.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6c8a5700 1 -- 172.21.15.46:0/670643123 >> v2:172.21.15.106:3300/0 conn(0x7f2a68140f20 msgr2=0x7f2a680caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:57.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6c8a5700 1 --2- 172.21.15.46:0/670643123 >> v2:172.21.15.106:3300/0 conn(0x7f2a68140f20 0x7f2a680caf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6c8a5700 1 -- 172.21.15.46:0/670643123 >> v1:172.21.15.106:6789/0 conn(0x7f2a680b1360 legacy=0x7f2a680c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6c8a5700 1 -- 172.21.15.46:0/670643123 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a680cc8f0 con 0x7f2a681412f0 2024-01-27T01:08:57.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6c8a5700 1 -- 172.21.15.46:0/670643123 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4232861929 0 0) 0x7f2a600017f0 con 0x7f2a681412f0 2024-01-27T01:08:57.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6c8a5700 1 -- 172.21.15.46:0/670643123 >> v1:172.21.15.46:6789/0 conn(0x7f2a681412f0 legacy=0x7f2a680cd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6c8a5700 1 --2- 172.21.15.46:0/670643123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a54003a60 0x7f2a54003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6c8a5700 1 --2- 172.21.15.46:0/670643123 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a54004810 0x7f2a54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6c8a5700 1 --2- 172.21.15.46:0/670643123 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a540051c0 0x7f2a5400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6c8a5700 1 -- 172.21.15.46:0/670643123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a680cc610 con 0x7f2a54003a60 2024-01-27T01:08:57.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6c8a5700 1 -- 172.21.15.46:0/670643123 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5400c0b0 con 0x7f2a54004810 2024-01-27T01:08:57.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6c8a5700 1 -- 172.21.15.46:0/670643123 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5400c270 con 0x7f2a540051c0 2024-01-27T01:08:57.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6d8a7700 1 --2- 172.21.15.46:0/670643123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a54003a60 0x7f2a54003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6d0a6700 1 --2- 172.21.15.46:0/670643123 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a54004810 0x7f2a54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6e0a8700 1 --2- 172.21.15.46:0/670643123 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a540051c0 0x7f2a5400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6d8a7700 1 -- 172.21.15.46:0/670643123 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a540051c0 msgr2=0x7f2a5400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:57.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6d8a7700 1 --2- 172.21.15.46:0/670643123 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a540051c0 0x7f2a5400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6d8a7700 1 -- 172.21.15.46:0/670643123 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a54004810 msgr2=0x7f2a54004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:57.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6d8a7700 1 --2- 172.21.15.46:0/670643123 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a54004810 0x7f2a54004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6d8a7700 1 -- 172.21.15.46:0/670643123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a54003570 con 0x7f2a54003a60 2024-01-27T01:08:57.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6e0a8700 1 --2- 172.21.15.46:0/670643123 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a540051c0 0x7f2a5400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:57.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.251+0000 7f2a6e0a8700 1 -- 172.21.15.46:0/670643123 reap_dead start 2024-01-27T01:08:57.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.252+0000 7f2a6d8a7700 1 --2- 172.21.15.46:0/670643123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a54003a60 0x7f2a54003e30 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f2a64009c00 tx=0x7f2a640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d5a51a0d3fb93e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:57.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.252+0000 7f2a6c8a5700 1 -- 172.21.15.46:0/670643123 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6400e710 con 0x7f2a54003a60 2024-01-27T01:08:57.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.252+0000 7f2a6c8a5700 1 -- 172.21.15.46:0/670643123 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6400e870 con 0x7f2a54003a60 2024-01-27T01:08:57.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.252+0000 7f2a6c8a5700 1 -- 172.21.15.46:0/670643123 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6400ec30 con 0x7f2a54003a60 2024-01-27T01:08:57.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.252+0000 7f2a6fb0b700 1 -- 172.21.15.46:0/670643123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a54003a60 msgr2=0x7f2a54003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.252+0000 7f2a6fb0b700 1 --2- 172.21.15.46:0/670643123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a54003a60 0x7f2a54003e30 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f2a64009c00 tx=0x7f2a640091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.252+0000 7f2a6fb0b700 1 -- 172.21.15.46:0/670643123 shutdown_connections 2024-01-27T01:08:57.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.252+0000 7f2a6fb0b700 1 --2- 172.21.15.46:0/670643123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a54003a60 0x7f2a54003e30 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.252+0000 7f2a6fb0b700 1 -- 172.21.15.46:0/670643123 >> 172.21.15.46:0/670643123 conn(0x7f2a6806bc50 msgr2=0x7f2a6806c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:57.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.252+0000 7f2a6fb0b700 1 -- 172.21.15.46:0/670643123 shutdown_connections 2024-01-27T01:08:57.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.252+0000 7f2a6fb0b700 1 -- 172.21.15.46:0/670643123 wait complete. 2024-01-27T01:08:57.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.254+0000 7f2a6fb0b700 1 Processor -- start 2024-01-27T01:08:57.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.254+0000 7f2a6fb0b700 1 -- start start 2024-01-27T01:08:57.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.255+0000 7f2a6fb0b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a54003a60 0x7f2a681570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.255+0000 7f2a6fb0b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a6813c8b0 0x7f2a68159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.255+0000 7f2a6fb0b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a68159dd0 0x7f2a6813d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.255+0000 7f2a6fb0b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6815a240 con 0x7f2a68159dd0 2024-01-27T01:08:57.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.255+0000 7f2a6fb0b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a68158880 con 0x7f2a6813c8b0 2024-01-27T01:08:57.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.255+0000 7f2a6fb0b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a68158d10 con 0x7f2a54003a60 2024-01-27T01:08:57.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.255+0000 7f2a6e0a8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a68159dd0 0x7f2a6813d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.255+0000 7f2a6e0a8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a68159dd0 0x7f2a6813d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55474/0 (socket says 172.21.15.46:55474) 2024-01-27T01:08:57.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.255+0000 7f2a6e0a8700 1 -- 172.21.15.46:0/3170212165 learned_addr learned my addr 172.21.15.46:0/3170212165 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:57.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.255+0000 7f2a6d0a6700 1 --2- 172.21.15.46:0/3170212165 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a6813c8b0 0x7f2a68159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.255+0000 7f2a6d8a7700 1 --2- 172.21.15.46:0/3170212165 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a54003a60 0x7f2a681570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.256+0000 7f2a6e0a8700 1 -- 172.21.15.46:0/3170212165 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a54003a60 msgr2=0x7f2a681570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.256+0000 7f2a6e0a8700 1 --2- 172.21.15.46:0/3170212165 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2a54003a60 0x7f2a681570e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.256+0000 7f2a6e0a8700 1 -- 172.21.15.46:0/3170212165 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a6813c8b0 msgr2=0x7f2a68159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.256+0000 7f2a6e0a8700 1 --2- 172.21.15.46:0/3170212165 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a6813c8b0 0x7f2a68159890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.256+0000 7f2a6e0a8700 1 -- 172.21.15.46:0/3170212165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a681579d0 con 0x7f2a68159dd0 2024-01-27T01:08:57.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.256+0000 7f2a6e0a8700 1 --2- 172.21.15.46:0/3170212165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a68159dd0 0x7f2a6813d7a0 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f2a680c5620 tx=0x7f2a60001820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:57.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.256+0000 7f2a5dffb700 1 -- 172.21.15.46:0/3170212165 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6000ebc0 con 0x7f2a68159dd0 2024-01-27T01:08:57.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.256+0000 7f2a5dffb700 1 -- 172.21.15.46:0/3170212165 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6000ed20 con 0x7f2a68159dd0 2024-01-27T01:08:57.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.256+0000 7f2a5dffb700 1 -- 172.21.15.46:0/3170212165 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a60013670 con 0x7f2a68159dd0 2024-01-27T01:08:57.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.256+0000 7f2a6fb0b700 1 -- 172.21.15.46:0/3170212165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a68140740 con 0x7f2a68159dd0 2024-01-27T01:08:57.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.256+0000 7f2a6fb0b700 1 -- 172.21.15.46:0/3170212165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a68157f50 con 0x7f2a68159dd0 2024-01-27T01:08:57.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.257+0000 7f2a6fb0b700 1 -- 172.21.15.46:0/3170212165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2a6804c710 con 0x7f2a68159dd0 2024-01-27T01:08:57.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.258+0000 7f2a5dffb700 1 -- 172.21.15.46:0/3170212165 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2a60013a60 con 0x7f2a68159dd0 2024-01-27T01:08:57.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.258+0000 7f2a5dffb700 1 --2- 172.21.15.46:0/3170212165 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2a4c066f30 0x7f2a4c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.258+0000 7f2a5dffb700 1 -- 172.21.15.46:0/3170212165 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f2a60084e00 con 0x7f2a68159dd0 2024-01-27T01:08:57.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.258+0000 7f2a6d8a7700 1 --2- 172.21.15.46:0/3170212165 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2a4c066f30 0x7f2a4c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.258+0000 7f2a5dffb700 1 --2- 172.21.15.46:0/3170212165 >> [v2:172.21.15.106:6808/44050,v1:172.21.15.106:6809/44050] conn(0x7f2a4c06dd40 0x7f2a4c0701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.258+0000 7f2a5dffb700 1 -- 172.21.15.46:0/3170212165 --> [v2:172.21.15.106:6808/44050,v1:172.21.15.106:6809/44050] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a4c070800 con 0x7f2a4c06dd40 2024-01-27T01:08:57.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.258+0000 7f2a5dffb700 1 -- 172.21.15.46:0/3170212165 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=269) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a60087440 con 0x7f2a68159dd0 2024-01-27T01:08:57.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.259+0000 7f2a6d0a6700 1 --2- 172.21.15.46:0/3170212165 >> [v2:172.21.15.106:6808/44050,v1:172.21.15.106:6809/44050] conn(0x7f2a4c06dd40 0x7f2a4c0701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.259+0000 7f2a6d8a7700 1 --2- 172.21.15.46:0/3170212165 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2a4c066f30 0x7f2a4c0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2a640091e0 tx=0x7f2a6401a000 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:57.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.259+0000 7f2a6d0a6700 1 --2- 172.21.15.46:0/3170212165 >> [v2:172.21.15.106:6808/44050,v1:172.21.15.106:6809/44050] conn(0x7f2a4c06dd40 0x7f2a4c0701d0 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:57.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.260+0000 7f2a5dffb700 1 -- 172.21.15.46:0/3170212165 <== osd.4 v2:172.21.15.106:6808/44050 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f2a4c070800 con 0x7f2a4c06dd40 2024-01-27T01:08:57.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.275+0000 7f2a6fb0b700 1 -- 172.21.15.46:0/3170212165 --> [v2:172.21.15.106:6808/44050,v1:172.21.15.106:6809/44050] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2a6806ca70 con 0x7f2a4c06dd40 2024-01-27T01:08:57.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.275+0000 7f2a5dffb700 1 -- 172.21.15.46:0/3170212165 <== osd.4 v2:172.21.15.106:6808/44050 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f2a6806ca70 con 0x7f2a4c06dd40 2024-01-27T01:08:57.275 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:08:57.275 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-27T01:08:57.276 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:08:57.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.275+0000 7f2a6fb0b700 1 -- 172.21.15.46:0/3170212165 >> [v2:172.21.15.106:6808/44050,v1:172.21.15.106:6809/44050] conn(0x7f2a4c06dd40 msgr2=0x7f2a4c0701d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.275+0000 7f2a6fb0b700 1 --2- 172.21.15.46:0/3170212165 >> [v2:172.21.15.106:6808/44050,v1:172.21.15.106:6809/44050] conn(0x7f2a4c06dd40 0x7f2a4c0701d0 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.276+0000 7f2a6fb0b700 1 -- 172.21.15.46:0/3170212165 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2a4c066f30 msgr2=0x7f2a4c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.276+0000 7f2a6fb0b700 1 --2- 172.21.15.46:0/3170212165 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f2a4c066f30 0x7f2a4c0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2a640091e0 tx=0x7f2a6401a000 comp rx=0 tx=0).stop 2024-01-27T01:08:57.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.276+0000 7f2a6fb0b700 1 -- 172.21.15.46:0/3170212165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a68159dd0 msgr2=0x7f2a6813d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.276+0000 7f2a6fb0b700 1 --2- 172.21.15.46:0/3170212165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a68159dd0 0x7f2a6813d7a0 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f2a680c5620 tx=0x7f2a60001820 comp rx=0 tx=0).stop 2024-01-27T01:08:57.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.276+0000 7f2a6e0a8700 1 -- 172.21.15.46:0/3170212165 reap_dead start 2024-01-27T01:08:57.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.276+0000 7f2a6fb0b700 1 -- 172.21.15.46:0/3170212165 shutdown_connections 2024-01-27T01:08:57.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.276+0000 7f2a6fb0b700 1 -- 172.21.15.46:0/3170212165 >> 172.21.15.46:0/3170212165 conn(0x7f2a6806bc50 msgr2=0x7f2a6806da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:57.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.276+0000 7f2a6fb0b700 1 -- 172.21.15.46:0/3170212165 shutdown_connections 2024-01-27T01:08:57.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.276+0000 7f2a6fb0b700 1 -- 172.21.15.46:0/3170212165 wait complete. 2024-01-27T01:08:57.289 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-01-27T01:08:57.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.422+0000 7f3af73ca700 1 Processor -- start 2024-01-27T01:08:57.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.422+0000 7f3af73ca700 1 -- start start 2024-01-27T01:08:57.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.423+0000 7f3af73ca700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3af00cb5c0 0x7f3af006a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.423+0000 7f3af73ca700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3af0068fd0 0x7f3af00c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.423+0000 7f3af73ca700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af0069660 con 0x7f3af00cb1f0 2024-01-27T01:08:57.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.423+0000 7f3af73ca700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af0069450 con 0x7f3af0068fd0 2024-01-27T01:08:57.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.423+0000 7f3af73ca700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af0069940 con 0x7f3af00cb5c0 2024-01-27T01:08:57.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.423+0000 7f3af4965700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3af00cb5c0 0x7f3af006a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.423+0000 7f3af5967700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3af0068fd0 0x7f3af00c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.423+0000 7f3af5967700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3af0068fd0 0x7f3af00c92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.46:59790/0 (socket says 172.21.15.46:59790) 2024-01-27T01:08:57.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.423+0000 7f3af5166700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f3af00cb1f0 0x7f3af00c8be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:47878/0 (socket says 172.21.15.46:47878) 2024-01-27T01:08:57.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.423+0000 7f3af4965700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3af00cb5c0 0x7f3af006a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:44814/0 (socket says 172.21.15.46:44814) 2024-01-27T01:08:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.423+0000 7f3af5967700 1 -- 172.21.15.46:0/3318897176 learned_addr learned my addr 172.21.15.46:0/3318897176 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.423+0000 7f3af5967700 1 -- 172.21.15.46:0/3318897176 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3af00cb5c0 msgr2=0x7f3af006a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.423+0000 7f3af5967700 1 --2- 172.21.15.46:0/3318897176 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3af00cb5c0 0x7f3af006a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.423+0000 7f3af5967700 1 -- 172.21.15.46:0/3318897176 >> v1:172.21.15.106:6789/0 conn(0x7f3af00cb1f0 legacy=0x7f3af00c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.424+0000 7f3ae7fff700 1 -- 172.21.15.46:0/3318897176 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97370047 0 0) 0x7f3af0069660 con 0x7f3af00cb1f0 2024-01-27T01:08:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.424+0000 7f3af5967700 1 -- 172.21.15.46:0/3318897176 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af0069c20 con 0x7f3af0068fd0 2024-01-27T01:08:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.424+0000 7f3af4965700 1 --2- 172.21.15.46:0/3318897176 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3af00cb5c0 0x7f3af006a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:08:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.424+0000 7f3af5967700 1 --2- 172.21.15.46:0/3318897176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3af0068fd0 0x7f3af00c92f0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8009f20 tx=0x7f3ae8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edec14121486c963 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.424+0000 7f3ae7fff700 1 -- 172.21.15.46:0/3318897176 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae800e070 con 0x7f3af0068fd0 2024-01-27T01:08:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.424+0000 7f3ae7fff700 1 -- 172.21.15.46:0/3318897176 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae800b9d0 con 0x7f3af0068fd0 2024-01-27T01:08:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.424+0000 7f3ae7fff700 1 -- 172.21.15.46:0/3318897176 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae80092e0 con 0x7f3af0068fd0 2024-01-27T01:08:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.424+0000 7f3af73ca700 1 -- 172.21.15.46:0/3318897176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3af0068fd0 msgr2=0x7f3af00c92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.424+0000 7f3af73ca700 1 --2- 172.21.15.46:0/3318897176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3af0068fd0 0x7f3af00c92f0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8009f20 tx=0x7f3ae8009840 comp rx=0 tx=0).stop 2024-01-27T01:08:57.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.425+0000 7f3af73ca700 1 -- 172.21.15.46:0/3318897176 shutdown_connections 2024-01-27T01:08:57.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.425+0000 7f3af73ca700 1 --2- 172.21.15.46:0/3318897176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3af0068fd0 0x7f3af00c92f0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.425+0000 7f3af73ca700 1 --2- 172.21.15.46:0/3318897176 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3af00cb5c0 0x7f3af006a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.425+0000 7f3af73ca700 1 -- 172.21.15.46:0/3318897176 >> 172.21.15.46:0/3318897176 conn(0x7f3af006bd70 msgr2=0x7f3af006c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:57.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.425+0000 7f3af73ca700 1 -- 172.21.15.46:0/3318897176 shutdown_connections 2024-01-27T01:08:57.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.425+0000 7f3af73ca700 1 -- 172.21.15.46:0/3318897176 wait complete. 2024-01-27T01:08:57.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.426+0000 7f3af73ca700 1 Processor -- start 2024-01-27T01:08:57.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.426+0000 7f3af73ca700 1 -- start start 2024-01-27T01:08:57.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af73ca700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3af0068fd0 0x7f3af00af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af73ca700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af00cb1f0 0x7f3af0153770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af73ca700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3af00cb5c0 0x7f3af016e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af73ca700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af00afbf0 con 0x7f3af00cb1f0 2024-01-27T01:08:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af73ca700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af00afed0 con 0x7f3af0068fd0 2024-01-27T01:08:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af73ca700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af00b01b0 con 0x7f3af00cb5c0 2024-01-27T01:08:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af4965700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af00cb1f0 0x7f3af0153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af4965700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af00cb1f0 0x7f3af0153770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55476/0 (socket says 172.21.15.46:55476) 2024-01-27T01:08:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af4965700 1 -- 172.21.15.46:0/2010908054 learned_addr learned my addr 172.21.15.46:0/2010908054 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af5967700 1 --2- 172.21.15.46:0/2010908054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3af00cb5c0 0x7f3af016e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af5166700 1 --2- 172.21.15.46:0/2010908054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3af0068fd0 0x7f3af00af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af4965700 1 -- 172.21.15.46:0/2010908054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3af00cb5c0 msgr2=0x7f3af016e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af4965700 1 --2- 172.21.15.46:0/2010908054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3af00cb5c0 0x7f3af016e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af4965700 1 -- 172.21.15.46:0/2010908054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3af0068fd0 msgr2=0x7f3af00af3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af4965700 1 --2- 172.21.15.46:0/2010908054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3af0068fd0 0x7f3af00af3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af4965700 1 -- 172.21.15.46:0/2010908054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae0009420 con 0x7f3af00cb1f0 2024-01-27T01:08:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af5166700 1 --2- 172.21.15.46:0/2010908054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3af0068fd0 0x7f3af00af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.427+0000 7f3af5967700 1 --2- 172.21.15.46:0/2010908054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3af00cb5c0 0x7f3af016e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.428+0000 7f3af4965700 1 --2- 172.21.15.46:0/2010908054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af00cb1f0 0x7f3af0153770 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f3ae000ec80 tx=0x7f3ae00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.428+0000 7f3ae57fa700 1 -- 172.21.15.46:0/2010908054 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae000c9d0 con 0x7f3af00cb1f0 2024-01-27T01:08:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.428+0000 7f3ae57fa700 1 -- 172.21.15.46:0/2010908054 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae000cb30 con 0x7f3af00cb1f0 2024-01-27T01:08:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.428+0000 7f3af73ca700 1 -- 172.21.15.46:0/2010908054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae8009ae0 con 0x7f3af00cb1f0 2024-01-27T01:08:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.428+0000 7f3ae57fa700 1 -- 172.21.15.46:0/2010908054 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae000ceb0 con 0x7f3af00cb1f0 2024-01-27T01:08:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.428+0000 7f3af73ca700 1 -- 172.21.15.46:0/2010908054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af00b0770 con 0x7f3af00cb1f0 2024-01-27T01:08:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.429+0000 7f3af73ca700 1 -- 172.21.15.46:0/2010908054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af004c710 con 0x7f3af00cb1f0 2024-01-27T01:08:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.430+0000 7f3ae57fa700 1 -- 172.21.15.46:0/2010908054 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3ae0013680 con 0x7f3af00cb1f0 2024-01-27T01:08:57.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.430+0000 7f3ae57fa700 1 --2- 172.21.15.46:0/2010908054 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3ad4066f30 0x7f3ad40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.430+0000 7f3ae57fa700 1 -- 172.21.15.46:0/2010908054 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f3ae0084c00 con 0x7f3af00cb1f0 2024-01-27T01:08:57.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.430+0000 7f3af5166700 1 --2- 172.21.15.46:0/2010908054 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3ad4066f30 0x7f3ad40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.431+0000 7f3af5166700 1 --2- 172.21.15.46:0/2010908054 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3ad4066f30 0x7f3ad40693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3aec0011e0 tx=0x7f3aec007c90 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:57.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.435+0000 7f3ae57fa700 1 -- 172.21.15.46:0/2010908054 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3ae008b080 con 0x7f3af00cb1f0 2024-01-27T01:08:57.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.639+0000 7f3af73ca700 1 -- 172.21.15.46:0/2010908054 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3af0069f00 con 0x7f3ad4066f30 2024-01-27T01:08:57.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.640+0000 7f3ae57fa700 1 -- 172.21.15.46:0/2010908054 <== mgr.10565 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3af0069f00 con 0x7f3ad4066f30 2024-01-27T01:08:57.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.642+0000 7f3af73ca700 1 -- 172.21.15.46:0/2010908054 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3ad4066f30 msgr2=0x7f3ad40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.642+0000 7f3af73ca700 1 --2- 172.21.15.46:0/2010908054 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3ad4066f30 0x7f3ad40693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3aec0011e0 tx=0x7f3aec007c90 comp rx=0 tx=0).stop 2024-01-27T01:08:57.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.642+0000 7f3af73ca700 1 -- 172.21.15.46:0/2010908054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af00cb1f0 msgr2=0x7f3af0153770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.642+0000 7f3af73ca700 1 --2- 172.21.15.46:0/2010908054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af00cb1f0 0x7f3af0153770 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f3ae000ec80 tx=0x7f3ae00091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.642+0000 7f3af73ca700 1 -- 172.21.15.46:0/2010908054 shutdown_connections 2024-01-27T01:08:57.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.642+0000 7f3af73ca700 1 --2- 172.21.15.46:0/2010908054 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3ad4066f30 0x7f3ad40693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.642+0000 7f3af73ca700 1 --2- 172.21.15.46:0/2010908054 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3af00cb5c0 0x7f3af016e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.642+0000 7f3af73ca700 1 --2- 172.21.15.46:0/2010908054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3af0068fd0 0x7f3af00af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.642+0000 7f3af73ca700 1 --2- 172.21.15.46:0/2010908054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af00cb1f0 0x7f3af0153770 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.642+0000 7f3af73ca700 1 -- 172.21.15.46:0/2010908054 >> 172.21.15.46:0/2010908054 conn(0x7f3af006bd70 msgr2=0x7f3af00c3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:57.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.642+0000 7f3af73ca700 1 -- 172.21.15.46:0/2010908054 shutdown_connections 2024-01-27T01:08:57.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.642+0000 7f3af73ca700 1 -- 172.21.15.46:0/2010908054 wait complete. 2024-01-27T01:08:57.643 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 4 to deep-scrub 2024-01-27T01:08:57.655 INFO:tasks.ceph:Scrubbing osd.5 2024-01-27T01:08:57.656 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-01-27T01:08:57.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.791+0000 7f7de8734700 1 Processor -- start 2024-01-27T01:08:57.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.791+0000 7f7de8734700 1 -- start start 2024-01-27T01:08:57.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.792+0000 7f7de8734700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7de0159a50 0x7f7de00cd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.792+0000 7f7de8734700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de006f4a0 con 0x7f7de0159680 2024-01-27T01:08:57.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.792+0000 7f7de8734700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de00a9df0 con 0x7f7de00a9880 2024-01-27T01:08:57.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.792+0000 7f7de8734700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de00aa0d0 con 0x7f7de0159a50 2024-01-27T01:08:57.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.792+0000 7f7de6cd1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7de0159a50 0x7f7de00cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.792+0000 7f7de6cd1700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7de0159a50 0x7f7de00cd340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55480/0 (socket says 172.21.15.46:55480) 2024-01-27T01:08:57.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.792+0000 7f7de64d0700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7de0159680 0x7f7de00aae90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32800/0 (socket says 172.21.15.46:32800) 2024-01-27T01:08:57.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.792+0000 7f7de6cd1700 1 -- 172.21.15.46:0/2429059184 learned_addr learned my addr 172.21.15.46:0/2429059184 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:57.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.793+0000 7f7de6cd1700 1 -- 172.21.15.46:0/2429059184 >> v1:172.21.15.106:6789/0 conn(0x7f7de00a9880 legacy=0x7f7de00c19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.793+0000 7f7de6cd1700 1 -- 172.21.15.46:0/2429059184 >> v1:172.21.15.46:6789/0 conn(0x7f7de0159680 legacy=0x7f7de00aae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.793+0000 7f7de54ce700 1 -- 172.21.15.46:0/2429059184 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652666912 0 0) 0x7f7de006f4a0 con 0x7f7de0159680 2024-01-27T01:08:57.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.793+0000 7f7de6cd1700 1 -- 172.21.15.46:0/2429059184 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de00aa3b0 con 0x7f7de0159a50 2024-01-27T01:08:57.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.793+0000 7f7de6cd1700 1 --2- 172.21.15.46:0/2429059184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de0159a50 0x7f7de00cd340 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8009f20 tx=0x7f7dd8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c80a0f1178a1ad3d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:57.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.793+0000 7f7de54ce700 1 -- 172.21.15.46:0/2429059184 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd800e070 con 0x7f7de0159a50 2024-01-27T01:08:57.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.793+0000 7f7de54ce700 1 -- 172.21.15.46:0/2429059184 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd800ba80 con 0x7f7de0159a50 2024-01-27T01:08:57.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.793+0000 7f7de54ce700 1 -- 172.21.15.46:0/2429059184 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd80092e0 con 0x7f7de0159a50 2024-01-27T01:08:57.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.794+0000 7f7de8734700 1 -- 172.21.15.46:0/2429059184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de0159a50 msgr2=0x7f7de00cd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.794+0000 7f7de8734700 1 --2- 172.21.15.46:0/2429059184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de0159a50 0x7f7de00cd340 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8009f20 tx=0x7f7dd8009840 comp rx=0 tx=0).stop 2024-01-27T01:08:57.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.794+0000 7f7de8734700 1 -- 172.21.15.46:0/2429059184 shutdown_connections 2024-01-27T01:08:57.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.794+0000 7f7de8734700 1 --2- 172.21.15.46:0/2429059184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de0159a50 0x7f7de00cd340 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.794+0000 7f7de8734700 1 -- 172.21.15.46:0/2429059184 >> 172.21.15.46:0/2429059184 conn(0x7f7de0068430 msgr2=0x7f7de00ccc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:57.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.794+0000 7f7de8734700 1 -- 172.21.15.46:0/2429059184 shutdown_connections 2024-01-27T01:08:57.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.794+0000 7f7de8734700 1 -- 172.21.15.46:0/2429059184 wait complete. 2024-01-27T01:08:57.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.795+0000 7f7de8734700 1 Processor -- start 2024-01-27T01:08:57.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.795+0000 7f7de8734700 1 -- start start 2024-01-27T01:08:57.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.796+0000 7f7de8734700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7de00a9880 0x7f7de014a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.796+0000 7f7de8734700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7de0159680 0x7f7de00c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.796+0000 7f7de8734700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de0159a50 0x7f7de006e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.796+0000 7f7de8734700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de00c5840 con 0x7f7de0159a50 2024-01-27T01:08:57.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.796+0000 7f7de8734700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de00c4c30 con 0x7f7de00a9880 2024-01-27T01:08:57.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.796+0000 7f7de8734700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de00c4f10 con 0x7f7de0159680 2024-01-27T01:08:57.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.796+0000 7f7de6cd1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de0159a50 0x7f7de006e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.796+0000 7f7de6cd1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de0159a50 0x7f7de006e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55482/0 (socket says 172.21.15.46:55482) 2024-01-27T01:08:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.796+0000 7f7de6cd1700 1 -- 172.21.15.46:0/3964069826 learned_addr learned my addr 172.21.15.46:0/3964069826 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.796+0000 7f7de64d0700 1 --2- 172.21.15.46:0/3964069826 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7de00a9880 0x7f7de014a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.796+0000 7f7de5ccf700 1 --2- 172.21.15.46:0/3964069826 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7de0159680 0x7f7de00c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.796+0000 7f7de6cd1700 1 -- 172.21.15.46:0/3964069826 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7de0159680 msgr2=0x7f7de00c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.796+0000 7f7de6cd1700 1 --2- 172.21.15.46:0/3964069826 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7de0159680 0x7f7de00c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.796+0000 7f7de6cd1700 1 -- 172.21.15.46:0/3964069826 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7de00a9880 msgr2=0x7f7de014a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.796+0000 7f7de6cd1700 1 --2- 172.21.15.46:0/3964069826 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7de00a9880 0x7f7de014a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.796+0000 7f7de6cd1700 1 -- 172.21.15.46:0/3964069826 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd8009ae0 con 0x7f7de0159a50 2024-01-27T01:08:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.796+0000 7f7de6cd1700 1 --2- 172.21.15.46:0/3964069826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de0159a50 0x7f7de006e1e0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f7de006fab0 tx=0x7f7dd8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.797+0000 7f7dd27fc700 1 -- 172.21.15.46:0/3964069826 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8007cd0 con 0x7f7de0159a50 2024-01-27T01:08:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.797+0000 7f7dd27fc700 1 -- 172.21.15.46:0/3964069826 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd8007e70 con 0x7f7de0159a50 2024-01-27T01:08:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.797+0000 7f7de8734700 1 -- 172.21.15.46:0/3964069826 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de00c51f0 con 0x7f7de0159a50 2024-01-27T01:08:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.797+0000 7f7dd27fc700 1 -- 172.21.15.46:0/3964069826 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8026690 con 0x7f7de0159a50 2024-01-27T01:08:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.797+0000 7f7de8734700 1 -- 172.21.15.46:0/3964069826 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de00b2710 con 0x7f7de0159a50 2024-01-27T01:08:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.798+0000 7f7de8734700 1 -- 172.21.15.46:0/3964069826 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7de00acbd0 con 0x7f7de0159a50 2024-01-27T01:08:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.798+0000 7f7dd27fc700 1 -- 172.21.15.46:0/3964069826 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7dd801d420 con 0x7f7de0159a50 2024-01-27T01:08:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.799+0000 7f7dd27fc700 1 --2- 172.21.15.46:0/3964069826 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7dc4066f30 0x7f7dc40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.799+0000 7f7dd27fc700 1 -- 172.21.15.46:0/3964069826 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f7dd80862e0 con 0x7f7de0159a50 2024-01-27T01:08:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.799+0000 7f7dd27fc700 1 --2- 172.21.15.46:0/3964069826 >> [v2:172.21.15.106:6800/44051,v1:172.21.15.106:6801/44051] conn(0x7f7dc406dd40 0x7f7dc40701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.799+0000 7f7dd27fc700 1 -- 172.21.15.46:0/3964069826 --> [v2:172.21.15.106:6800/44051,v1:172.21.15.106:6801/44051] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7dc4070800 con 0x7f7dc406dd40 2024-01-27T01:08:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.799+0000 7f7dd27fc700 1 -- 172.21.15.46:0/3964069826 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=269) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7dd8088920 con 0x7f7de0159a50 2024-01-27T01:08:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.799+0000 7f7de64d0700 1 --2- 172.21.15.46:0/3964069826 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7dc4066f30 0x7f7dc40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.799+0000 7f7de5ccf700 1 --2- 172.21.15.46:0/3964069826 >> [v2:172.21.15.106:6800/44051,v1:172.21.15.106:6801/44051] conn(0x7f7dc406dd40 0x7f7dc40701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.799+0000 7f7de64d0700 1 --2- 172.21.15.46:0/3964069826 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7dc4066f30 0x7f7dc40693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7de016d2d0 tx=0x7f7dd4009040 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.800+0000 7f7de5ccf700 1 --2- 172.21.15.46:0/3964069826 >> [v2:172.21.15.106:6800/44051,v1:172.21.15.106:6801/44051] conn(0x7f7dc406dd40 0x7f7dc40701d0 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.800+0000 7f7dd27fc700 1 -- 172.21.15.46:0/3964069826 <== osd.5 v2:172.21.15.106:6800/44051 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f7dc4070800 con 0x7f7dc406dd40 2024-01-27T01:08:57.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.815+0000 7f7de8734700 1 -- 172.21.15.46:0/3964069826 --> [v2:172.21.15.106:6800/44051,v1:172.21.15.106:6801/44051] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7de004c710 con 0x7f7dc406dd40 2024-01-27T01:08:57.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.816+0000 7f7dd27fc700 1 -- 172.21.15.46:0/3964069826 <== osd.5 v2:172.21.15.106:6800/44051 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f7de004c710 con 0x7f7dc406dd40 2024-01-27T01:08:57.816 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:08:57.816 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-27T01:08:57.816 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:08:57.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.816+0000 7f7de8734700 1 -- 172.21.15.46:0/3964069826 >> [v2:172.21.15.106:6800/44051,v1:172.21.15.106:6801/44051] conn(0x7f7dc406dd40 msgr2=0x7f7dc40701d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.816+0000 7f7de8734700 1 --2- 172.21.15.46:0/3964069826 >> [v2:172.21.15.106:6800/44051,v1:172.21.15.106:6801/44051] conn(0x7f7dc406dd40 0x7f7dc40701d0 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.816+0000 7f7de8734700 1 -- 172.21.15.46:0/3964069826 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7dc4066f30 msgr2=0x7f7dc40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.816+0000 7f7de8734700 1 --2- 172.21.15.46:0/3964069826 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f7dc4066f30 0x7f7dc40693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7de016d2d0 tx=0x7f7dd4009040 comp rx=0 tx=0).stop 2024-01-27T01:08:57.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.816+0000 7f7de8734700 1 -- 172.21.15.46:0/3964069826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de0159a50 msgr2=0x7f7de006e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.816+0000 7f7de8734700 1 --2- 172.21.15.46:0/3964069826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de0159a50 0x7f7de006e1e0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f7de006fab0 tx=0x7f7dd8018000 comp rx=0 tx=0).stop 2024-01-27T01:08:57.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.816+0000 7f7de6cd1700 1 -- 172.21.15.46:0/3964069826 reap_dead start 2024-01-27T01:08:57.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.816+0000 7f7de8734700 1 -- 172.21.15.46:0/3964069826 shutdown_connections 2024-01-27T01:08:57.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.816+0000 7f7de8734700 1 -- 172.21.15.46:0/3964069826 >> 172.21.15.46:0/3964069826 conn(0x7f7de0068430 msgr2=0x7f7de00c7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:57.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.816+0000 7f7de8734700 1 -- 172.21.15.46:0/3964069826 shutdown_connections 2024-01-27T01:08:57.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.816+0000 7f7de8734700 1 -- 172.21.15.46:0/3964069826 wait complete. 2024-01-27T01:08:57.829 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-01-27T01:08:57.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.961+0000 7f5432172700 1 Processor -- start 2024-01-27T01:08:57.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.961+0000 7f5432172700 1 -- start start 2024-01-27T01:08:57.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.962+0000 7f5432172700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f542c15af00 0x7f542c0ab040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.962+0000 7f5432172700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f542c0c9520 0x7f542c0c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.962+0000 7f5432172700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542c0c42d0 con 0x7f542c0c4760 2024-01-27T01:08:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.962+0000 7f5432172700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f542c0c9f00 con 0x7f542c0c9520 2024-01-27T01:08:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.962+0000 7f5432172700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f542c0c36c0 con 0x7f542c15af00 2024-01-27T01:08:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.962+0000 7f542affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f542c0c9520 0x7f542c0c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.963+0000 7f542affd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f542c0c9520 0x7f542c0c99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55486/0 (socket says 172.21.15.46:55486) 2024-01-27T01:08:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.963+0000 7f542affd700 1 -- 172.21.15.46:0/1071478287 learned_addr learned my addr 172.21.15.46:0/1071478287 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.963+0000 7f542b7fe700 1 --2- 172.21.15.46:0/1071478287 >> v2:172.21.15.106:3300/0 conn(0x7f542c15af00 0x7f542c0ab040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.963+0000 7f542affd700 1 -- 172.21.15.46:0/1071478287 >> v2:172.21.15.106:3300/0 conn(0x7f542c15af00 msgr2=0x7f542c0ab040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.963+0000 7f542affd700 1 --2- 172.21.15.46:0/1071478287 >> v2:172.21.15.106:3300/0 conn(0x7f542c15af00 0x7f542c0ab040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.963+0000 7f542affd700 1 -- 172.21.15.46:0/1071478287 >> v1:172.21.15.106:6789/0 conn(0x7f542c0c4760 legacy=0x7f542c0c4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.963+0000 7f542affd700 1 -- 172.21.15.46:0/1071478287 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542c0c39a0 con 0x7f542c0c9520 2024-01-27T01:08:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.963+0000 7f542affd700 1 --2- 172.21.15.46:0/1071478287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f542c0c9520 0x7f542c0c99c0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f542c06e7b0 tx=0x7f5420009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40bdf9e86dcd3e11 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:57.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.963+0000 7f542a7fc700 1 -- 172.21.15.46:0/1071478287 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5420009e70 con 0x7f542c0c9520 2024-01-27T01:08:57.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.963+0000 7f542a7fc700 1 -- 172.21.15.46:0/1071478287 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f542000b860 con 0x7f542c0c9520 2024-01-27T01:08:57.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.963+0000 7f542a7fc700 1 -- 172.21.15.46:0/1071478287 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542000bc00 con 0x7f542c0c9520 2024-01-27T01:08:57.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.964+0000 7f5432172700 1 -- 172.21.15.46:0/1071478287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f542c0c9520 msgr2=0x7f542c0c99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.964+0000 7f5432172700 1 --2- 172.21.15.46:0/1071478287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f542c0c9520 0x7f542c0c99c0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f542c06e7b0 tx=0x7f5420009250 comp rx=0 tx=0).stop 2024-01-27T01:08:57.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.964+0000 7f5432172700 1 -- 172.21.15.46:0/1071478287 shutdown_connections 2024-01-27T01:08:57.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.964+0000 7f5432172700 1 --2- 172.21.15.46:0/1071478287 >> v2:172.21.15.106:3300/0 conn(0x7f542c15af00 0x7f542c0ab040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.964+0000 7f5432172700 1 --2- 172.21.15.46:0/1071478287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f542c0c9520 0x7f542c0c99c0 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.964+0000 7f5432172700 1 -- 172.21.15.46:0/1071478287 >> 172.21.15.46:0/1071478287 conn(0x7f542c068190 msgr2=0x7f542c0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:57.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.964+0000 7f5432172700 1 -- 172.21.15.46:0/1071478287 shutdown_connections 2024-01-27T01:08:57.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.964+0000 7f5432172700 1 -- 172.21.15.46:0/1071478287 wait complete. 2024-01-27T01:08:57.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.965+0000 7f5432172700 1 Processor -- start 2024-01-27T01:08:57.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.965+0000 7f5432172700 1 -- start start 2024-01-27T01:08:57.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.966+0000 7f5432172700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f542c0c4760 0x7f542c152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.966+0000 7f5432172700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f542c0c9520 0x7f542c068c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.966+0000 7f5432172700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f542c15af00 0x7f542c155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.966+0000 7f5432172700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542c06a270 con 0x7f542c0c9520 2024-01-27T01:08:57.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.966+0000 7f5432172700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542c152c50 con 0x7f542c0c4760 2024-01-27T01:08:57.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.966+0000 7f5432172700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f542c069660 con 0x7f542c15af00 2024-01-27T01:08:57.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.966+0000 7f542affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f542c0c9520 0x7f542c068c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.966+0000 7f542affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f542c0c9520 0x7f542c068c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55488/0 (socket says 172.21.15.46:55488) 2024-01-27T01:08:57.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.966+0000 7f542affd700 1 -- 172.21.15.46:0/1492807745 learned_addr learned my addr 172.21.15.46:0/1492807745 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:57.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.966+0000 7f542b7fe700 1 --2- 172.21.15.46:0/1492807745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f542c0c4760 0x7f542c152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.966+0000 7f542bfff700 1 --2- 172.21.15.46:0/1492807745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f542c15af00 0x7f542c155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.967+0000 7f542affd700 1 -- 172.21.15.46:0/1492807745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f542c15af00 msgr2=0x7f542c155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.967+0000 7f542affd700 1 --2- 172.21.15.46:0/1492807745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f542c15af00 0x7f542c155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.967+0000 7f542affd700 1 -- 172.21.15.46:0/1492807745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f542c0c4760 msgr2=0x7f542c152710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:57.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.967+0000 7f542affd700 1 --2- 172.21.15.46:0/1492807745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f542c0c4760 0x7f542c152710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:57.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.967+0000 7f542affd700 1 -- 172.21.15.46:0/1492807745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54200094f0 con 0x7f542c0c9520 2024-01-27T01:08:57.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.967+0000 7f542bfff700 1 --2- 172.21.15.46:0/1492807745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f542c15af00 0x7f542c155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:57.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.967+0000 7f542affd700 1 --2- 172.21.15.46:0/1492807745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f542c0c9520 0x7f542c068c80 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f5420006fd0 tx=0x7f5420017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:57.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.967+0000 7f54137fe700 1 -- 172.21.15.46:0/1492807745 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542000b9d0 con 0x7f542c0c9520 2024-01-27T01:08:57.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.967+0000 7f54137fe700 1 -- 172.21.15.46:0/1492807745 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f542000bb30 con 0x7f542c0c9520 2024-01-27T01:08:57.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.967+0000 7f54137fe700 1 -- 172.21.15.46:0/1492807745 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5420025650 con 0x7f542c0c9520 2024-01-27T01:08:57.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.967+0000 7f5432172700 1 -- 172.21.15.46:0/1492807745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f542c069940 con 0x7f542c0c9520 2024-01-27T01:08:57.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.967+0000 7f5432172700 1 -- 172.21.15.46:0/1492807745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f542c069f00 con 0x7f542c0c9520 2024-01-27T01:08:57.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.968+0000 7f5432172700 1 -- 172.21.15.46:0/1492807745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f542c04c710 con 0x7f542c0c9520 2024-01-27T01:08:57.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.969+0000 7f54137fe700 1 -- 172.21.15.46:0/1492807745 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5420023070 con 0x7f542c0c9520 2024-01-27T01:08:57.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.969+0000 7f54137fe700 1 --2- 172.21.15.46:0/1492807745 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5418066f30 0x7f54180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:57.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.969+0000 7f54137fe700 1 -- 172.21.15.46:0/1492807745 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f5420085e60 con 0x7f542c0c9520 2024-01-27T01:08:57.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.970+0000 7f542b7fe700 1 --2- 172.21.15.46:0/1492807745 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5418066f30 0x7f54180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:57.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.970+0000 7f542b7fe700 1 --2- 172.21.15.46:0/1492807745 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5418066f30 0x7f54180693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f541c00f570 tx=0x7f541c00b040 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:57.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:57.974+0000 7f54137fe700 1 -- 172.21.15.46:0/1492807745 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5420052d80 con 0x7f542c0c9520 2024-01-27T01:08:58.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.177+0000 7f5432172700 1 -- 172.21.15.46:0/1492807745 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f542c0cf3c0 con 0x7f5418066f30 2024-01-27T01:08:58.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.177+0000 7f54137fe700 1 -- 172.21.15.46:0/1492807745 <== mgr.10565 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f542c0cf3c0 con 0x7f5418066f30 2024-01-27T01:08:58.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.179+0000 7f5432172700 1 -- 172.21.15.46:0/1492807745 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5418066f30 msgr2=0x7f54180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.179+0000 7f5432172700 1 --2- 172.21.15.46:0/1492807745 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5418066f30 0x7f54180693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f541c00f570 tx=0x7f541c00b040 comp rx=0 tx=0).stop 2024-01-27T01:08:58.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.179+0000 7f5432172700 1 -- 172.21.15.46:0/1492807745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f542c0c9520 msgr2=0x7f542c068c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.179+0000 7f5432172700 1 --2- 172.21.15.46:0/1492807745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f542c0c9520 0x7f542c068c80 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f5420006fd0 tx=0x7f5420017000 comp rx=0 tx=0).stop 2024-01-27T01:08:58.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.179+0000 7f5432172700 1 -- 172.21.15.46:0/1492807745 shutdown_connections 2024-01-27T01:08:58.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.179+0000 7f5432172700 1 --2- 172.21.15.46:0/1492807745 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f5418066f30 0x7f54180693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.179+0000 7f5432172700 1 --2- 172.21.15.46:0/1492807745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f542c15af00 0x7f542c155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.179+0000 7f5432172700 1 --2- 172.21.15.46:0/1492807745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f542c0c4760 0x7f542c152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.179+0000 7f5432172700 1 --2- 172.21.15.46:0/1492807745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f542c0c9520 0x7f542c068c80 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.179+0000 7f5432172700 1 -- 172.21.15.46:0/1492807745 >> 172.21.15.46:0/1492807745 conn(0x7f542c068190 msgr2=0x7f542c1544c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:58.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.179+0000 7f5432172700 1 -- 172.21.15.46:0/1492807745 shutdown_connections 2024-01-27T01:08:58.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.179+0000 7f5432172700 1 -- 172.21.15.46:0/1492807745 wait complete. 2024-01-27T01:08:58.181 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 5 to deep-scrub 2024-01-27T01:08:58.193 INFO:tasks.ceph:Scrubbing osd.6 2024-01-27T01:08:58.193 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-01-27T01:08:58.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.328+0000 7fc39490b700 1 Processor -- start 2024-01-27T01:08:58.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.328+0000 7fc39490b700 1 -- start start 2024-01-27T01:08:58.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.329+0000 7fc39490b700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3900c97f0 0x7fc3900b1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc39490b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc39006b4b0 0x7fc3900b2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc39490b700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc3900c9bc0 0x7fc39015a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc39490b700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc390156d60 con 0x7fc3900c97f0 2024-01-27T01:08:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc39490b700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc390156ec0 con 0x7fc3900c9bc0 2024-01-27T01:08:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc39490b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc39006f120 con 0x7fc39006b4b0 2024-01-27T01:08:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc38e59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3900c97f0 0x7fc3900b1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc38e59c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3900c97f0 0x7fc3900b1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55500/0 (socket says 172.21.15.46:55500) 2024-01-27T01:08:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc38e59c700 1 -- 172.21.15.46:0/522342472 learned_addr learned my addr 172.21.15.46:0/522342472 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc38ed9d700 1 --2- 172.21.15.46:0/522342472 >> v2:172.21.15.106:3300/0 conn(0x7fc3900c9bc0 0x7fc39015a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc38dd9b700 1 --2- 172.21.15.46:0/522342472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc39006b4b0 0x7fc3900b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc38e59c700 1 -- 172.21.15.46:0/522342472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc39006b4b0 msgr2=0x7fc3900b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc38e59c700 1 --2- 172.21.15.46:0/522342472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc39006b4b0 0x7fc3900b2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc38e59c700 1 -- 172.21.15.46:0/522342472 >> v2:172.21.15.106:3300/0 conn(0x7fc3900c9bc0 msgr2=0x7fc39015a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc38e59c700 1 --2- 172.21.15.46:0/522342472 >> v2:172.21.15.106:3300/0 conn(0x7fc3900c9bc0 0x7fc39015a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc38e59c700 1 -- 172.21.15.46:0/522342472 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39006f400 con 0x7fc3900c97f0 2024-01-27T01:08:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc38dd9b700 1 --2- 172.21.15.46:0/522342472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc39006b4b0 0x7fc3900b2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc38ed9d700 1 --2- 172.21.15.46:0/522342472 >> v2:172.21.15.106:3300/0 conn(0x7fc3900c9bc0 0x7fc39015a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:58.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc38e59c700 1 --2- 172.21.15.46:0/522342472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3900c97f0 0x7fc3900b1b30 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7fc39015a280 tx=0x7fc378009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a6214dcd10988c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:58.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.330+0000 7fc38d59a700 1 -- 172.21.15.46:0/522342472 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37800b860 con 0x7fc3900c97f0 2024-01-27T01:08:58.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.331+0000 7fc38d59a700 1 -- 172.21.15.46:0/522342472 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37800b9c0 con 0x7fc3900c97f0 2024-01-27T01:08:58.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.331+0000 7fc38d59a700 1 -- 172.21.15.46:0/522342472 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37800bd60 con 0x7fc3900c97f0 2024-01-27T01:08:58.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.331+0000 7fc39490b700 1 -- 172.21.15.46:0/522342472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3900c97f0 msgr2=0x7fc3900b1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.331+0000 7fc39490b700 1 --2- 172.21.15.46:0/522342472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3900c97f0 0x7fc3900b1b30 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7fc39015a280 tx=0x7fc378009250 comp rx=0 tx=0).stop 2024-01-27T01:08:58.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.331+0000 7fc39490b700 1 -- 172.21.15.46:0/522342472 shutdown_connections 2024-01-27T01:08:58.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.331+0000 7fc39490b700 1 --2- 172.21.15.46:0/522342472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3900c97f0 0x7fc3900b1b30 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.331+0000 7fc39490b700 1 --2- 172.21.15.46:0/522342472 >> v2:172.21.15.106:3300/0 conn(0x7fc3900c9bc0 0x7fc39015a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.331+0000 7fc39490b700 1 --2- 172.21.15.46:0/522342472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc39006b4b0 0x7fc3900b2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.331+0000 7fc39490b700 1 -- 172.21.15.46:0/522342472 >> 172.21.15.46:0/522342472 conn(0x7fc390069590 msgr2=0x7fc39006aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:58.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.331+0000 7fc39490b700 1 -- 172.21.15.46:0/522342472 shutdown_connections 2024-01-27T01:08:58.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.331+0000 7fc39490b700 1 -- 172.21.15.46:0/522342472 wait complete. 2024-01-27T01:08:58.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.332+0000 7fc39490b700 1 Processor -- start 2024-01-27T01:08:58.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.333+0000 7fc39490b700 1 -- start start 2024-01-27T01:08:58.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.333+0000 7fc39490b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc39006b4b0 0x7fc3901446b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.333+0000 7fc39490b700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3900c97f0 0x7fc390145ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.333+0000 7fc39490b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3900c9bc0 0x7fc3901463e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.333+0000 7fc39490b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc390147270 con 0x7fc3900c9bc0 2024-01-27T01:08:58.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.333+0000 7fc39490b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39014d050 con 0x7fc39006b4b0 2024-01-27T01:08:58.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.333+0000 7fc39490b700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3901469d0 con 0x7fc3900c97f0 2024-01-27T01:08:58.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.333+0000 7fc38ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3900c9bc0 0x7fc3901463e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.333+0000 7fc38ed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3900c9bc0 0x7fc3901463e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55508/0 (socket says 172.21.15.46:55508) 2024-01-27T01:08:58.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.333+0000 7fc38ed9d700 1 -- 172.21.15.46:0/2353867148 learned_addr learned my addr 172.21.15.46:0/2353867148 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:58.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.334+0000 7fc38dd9b700 1 --2- 172.21.15.46:0/2353867148 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3900c97f0 0x7fc390145ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.334+0000 7fc38e59c700 1 --2- 172.21.15.46:0/2353867148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc39006b4b0 0x7fc3901446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.334+0000 7fc38ed9d700 1 -- 172.21.15.46:0/2353867148 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3900c97f0 msgr2=0x7fc390145ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.334+0000 7fc38ed9d700 1 --2- 172.21.15.46:0/2353867148 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc3900c97f0 0x7fc390145ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.334+0000 7fc38ed9d700 1 -- 172.21.15.46:0/2353867148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc39006b4b0 msgr2=0x7fc3901446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.334+0000 7fc38ed9d700 1 --2- 172.21.15.46:0/2353867148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc39006b4b0 0x7fc3901446b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.334+0000 7fc38ed9d700 1 -- 172.21.15.46:0/2353867148 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3780094f0 con 0x7fc3900c9bc0 2024-01-27T01:08:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.334+0000 7fc38ed9d700 1 --2- 172.21.15.46:0/2353867148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3900c9bc0 0x7fc3901463e0 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7fc38800d0d0 tx=0x7fc3880099d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.334+0000 7fc3867fc700 1 -- 172.21.15.46:0/2353867148 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38800dc10 con 0x7fc3900c9bc0 2024-01-27T01:08:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.334+0000 7fc3867fc700 1 -- 172.21.15.46:0/2353867148 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38800f040 con 0x7fc3900c9bc0 2024-01-27T01:08:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.334+0000 7fc3867fc700 1 -- 172.21.15.46:0/2353867148 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc388015440 con 0x7fc3900c9bc0 2024-01-27T01:08:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.334+0000 7fc39490b700 1 -- 172.21.15.46:0/2353867148 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc390146cb0 con 0x7fc3900c9bc0 2024-01-27T01:08:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.334+0000 7fc39490b700 1 -- 172.21.15.46:0/2353867148 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc390144bf0 con 0x7fc3900c9bc0 2024-01-27T01:08:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.335+0000 7fc39490b700 1 -- 172.21.15.46:0/2353867148 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc39004c710 con 0x7fc3900c9bc0 2024-01-27T01:08:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.336+0000 7fc3867fc700 1 -- 172.21.15.46:0/2353867148 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc38801e3b0 con 0x7fc3900c9bc0 2024-01-27T01:08:58.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.336+0000 7fc3867fc700 1 --2- 172.21.15.46:0/2353867148 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc370066f60 0x7fc370069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.336+0000 7fc3867fc700 1 -- 172.21.15.46:0/2353867148 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7fc3880851c0 con 0x7fc3900c9bc0 2024-01-27T01:08:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.337+0000 7fc3867fc700 1 --2- 172.21.15.46:0/2353867148 >> [v2:172.21.15.106:6816/44080,v1:172.21.15.106:6817/44080] conn(0x7fc37006dd70 0x7fc370070200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.337+0000 7fc3867fc700 1 -- 172.21.15.46:0/2353867148 --> [v2:172.21.15.106:6816/44080,v1:172.21.15.106:6817/44080] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc370070830 con 0x7fc37006dd70 2024-01-27T01:08:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.337+0000 7fc3867fc700 1 -- 172.21.15.46:0/2353867148 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=269) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc388085590 con 0x7fc3900c9bc0 2024-01-27T01:08:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.337+0000 7fc38e59c700 1 --2- 172.21.15.46:0/2353867148 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc370066f60 0x7fc370069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.337+0000 7fc38dd9b700 1 --2- 172.21.15.46:0/2353867148 >> [v2:172.21.15.106:6816/44080,v1:172.21.15.106:6817/44080] conn(0x7fc37006dd70 0x7fc370070200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.337+0000 7fc38e59c700 1 --2- 172.21.15.46:0/2353867148 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc370066f60 0x7fc370069410 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc39015a280 tx=0x7fc378017000 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.337+0000 7fc38dd9b700 1 --2- 172.21.15.46:0/2353867148 >> [v2:172.21.15.106:6816/44080,v1:172.21.15.106:6817/44080] conn(0x7fc37006dd70 0x7fc370070200 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.338+0000 7fc3867fc700 1 -- 172.21.15.46:0/2353867148 <== osd.6 v2:172.21.15.106:6816/44080 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fc370070830 con 0x7fc37006dd70 2024-01-27T01:08:58.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.353+0000 7fc39490b700 1 -- 172.21.15.46:0/2353867148 --> [v2:172.21.15.106:6816/44080,v1:172.21.15.106:6817/44080] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc39006bad0 con 0x7fc37006dd70 2024-01-27T01:08:58.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.353+0000 7fc3867fc700 1 -- 172.21.15.46:0/2353867148 <== osd.6 v2:172.21.15.106:6816/44080 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fc39006bad0 con 0x7fc37006dd70 2024-01-27T01:08:58.353 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:08:58.354 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-27T01:08:58.354 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:08:58.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.354+0000 7fc39490b700 1 -- 172.21.15.46:0/2353867148 >> [v2:172.21.15.106:6816/44080,v1:172.21.15.106:6817/44080] conn(0x7fc37006dd70 msgr2=0x7fc370070200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.354+0000 7fc39490b700 1 --2- 172.21.15.46:0/2353867148 >> [v2:172.21.15.106:6816/44080,v1:172.21.15.106:6817/44080] conn(0x7fc37006dd70 0x7fc370070200 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.354+0000 7fc39490b700 1 -- 172.21.15.46:0/2353867148 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc370066f60 msgr2=0x7fc370069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.354+0000 7fc39490b700 1 --2- 172.21.15.46:0/2353867148 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fc370066f60 0x7fc370069410 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc39015a280 tx=0x7fc378017000 comp rx=0 tx=0).stop 2024-01-27T01:08:58.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.354+0000 7fc39490b700 1 -- 172.21.15.46:0/2353867148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3900c9bc0 msgr2=0x7fc3901463e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.354+0000 7fc39490b700 1 --2- 172.21.15.46:0/2353867148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3900c9bc0 0x7fc3901463e0 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7fc38800d0d0 tx=0x7fc3880099d0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.354+0000 7fc38ed9d700 1 -- 172.21.15.46:0/2353867148 reap_dead start 2024-01-27T01:08:58.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.354+0000 7fc39490b700 1 -- 172.21.15.46:0/2353867148 shutdown_connections 2024-01-27T01:08:58.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.354+0000 7fc39490b700 1 -- 172.21.15.46:0/2353867148 >> 172.21.15.46:0/2353867148 conn(0x7fc390069590 msgr2=0x7fc3900adfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:58.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.354+0000 7fc39490b700 1 -- 172.21.15.46:0/2353867148 shutdown_connections 2024-01-27T01:08:58.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.354+0000 7fc39490b700 1 -- 172.21.15.46:0/2353867148 wait complete. 2024-01-27T01:08:58.367 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-01-27T01:08:58.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.500+0000 7f33fb200700 1 Processor -- start 2024-01-27T01:08:58.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.500+0000 7f33fb200700 1 -- start start 2024-01-27T01:08:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33fb200700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33f4157570 0x7f33f40c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33fb200700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33f40c7f90 0x7f33f406af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33fb200700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f33f406b460 0x7f33f40679b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33fb200700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33f415d620 con 0x7f33f4157570 2024-01-27T01:08:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33fb200700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33f415dec0 con 0x7f33f406b460 2024-01-27T01:08:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33fb200700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f406b8e0 con 0x7f33f40c7f90 2024-01-27T01:08:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33f8f9c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33f4157570 0x7f33f40c7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33f8f9c700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33f4157570 0x7f33f40c7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55524/0 (socket says 172.21.15.46:55524) 2024-01-27T01:08:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33f8f9c700 1 -- 172.21.15.46:0/2971516813 learned_addr learned my addr 172.21.15.46:0/2971516813 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33f979d700 1 --2- 172.21.15.46:0/2971516813 >> v2:172.21.15.106:3300/0 conn(0x7f33f406b460 0x7f33f40679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33f3fff700 1 --2- 172.21.15.46:0/2971516813 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33f40c7f90 0x7f33f406af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33f8f9c700 1 -- 172.21.15.46:0/2971516813 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33f40c7f90 msgr2=0x7f33f406af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33f8f9c700 1 --2- 172.21.15.46:0/2971516813 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33f40c7f90 0x7f33f406af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33f8f9c700 1 -- 172.21.15.46:0/2971516813 >> v2:172.21.15.106:3300/0 conn(0x7f33f406b460 msgr2=0x7f33f40679b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33f8f9c700 1 --2- 172.21.15.46:0/2971516813 >> v2:172.21.15.106:3300/0 conn(0x7f33f406b460 0x7f33f40679b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33f8f9c700 1 -- 172.21.15.46:0/2971516813 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f40af900 con 0x7f33f4157570 2024-01-27T01:08:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33f979d700 1 --2- 172.21.15.46:0/2971516813 >> v2:172.21.15.106:3300/0 conn(0x7f33f406b460 0x7f33f40679b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.501+0000 7f33f8f9c700 1 --2- 172.21.15.46:0/2971516813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f4157570 0x7f33f40c7a50 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f33f406e8f0 tx=0x7f33e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e08d4406b2a8b5d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.502+0000 7f33f37fe700 1 -- 172.21.15.46:0/2971516813 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e800b860 con 0x7f33f4157570 2024-01-27T01:08:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.502+0000 7f33f37fe700 1 -- 172.21.15.46:0/2971516813 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e800b9c0 con 0x7f33f4157570 2024-01-27T01:08:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.502+0000 7f33f37fe700 1 -- 172.21.15.46:0/2971516813 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e800bd60 con 0x7f33f4157570 2024-01-27T01:08:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.502+0000 7f33fb200700 1 -- 172.21.15.46:0/2971516813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f4157570 msgr2=0x7f33f40c7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.502+0000 7f33fb200700 1 --2- 172.21.15.46:0/2971516813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f4157570 0x7f33f40c7a50 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f33f406e8f0 tx=0x7f33e8009250 comp rx=0 tx=0).stop 2024-01-27T01:08:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.502+0000 7f33fb200700 1 -- 172.21.15.46:0/2971516813 shutdown_connections 2024-01-27T01:08:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.502+0000 7f33fb200700 1 --2- 172.21.15.46:0/2971516813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f4157570 0x7f33f40c7a50 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.502+0000 7f33fb200700 1 --2- 172.21.15.46:0/2971516813 >> v2:172.21.15.106:3300/0 conn(0x7f33f406b460 0x7f33f40679b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.502+0000 7f33fb200700 1 --2- 172.21.15.46:0/2971516813 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33f40c7f90 0x7f33f406af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.502+0000 7f33fb200700 1 -- 172.21.15.46:0/2971516813 >> 172.21.15.46:0/2971516813 conn(0x7f33f4068ac0 msgr2=0x7f33f4068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:58.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.502+0000 7f33fb200700 1 -- 172.21.15.46:0/2971516813 shutdown_connections 2024-01-27T01:08:58.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.502+0000 7f33fb200700 1 -- 172.21.15.46:0/2971516813 wait complete. 2024-01-27T01:08:58.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.504+0000 7f33fb200700 1 Processor -- start 2024-01-27T01:08:58.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.504+0000 7f33fb200700 1 -- start start 2024-01-27T01:08:58.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33fb200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f406b460 0x7f33f4155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33fb200700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33f40c7f90 0x7f33f4169a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33fb200700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33f4157570 0x7f33f40ae4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33fb200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f40cb3f0 con 0x7f33f406b460 2024-01-27T01:08:58.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33fb200700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f416a350 con 0x7f33f40c7f90 2024-01-27T01:08:58.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33fb200700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f416a630 con 0x7f33f4157570 2024-01-27T01:08:58.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33f8f9c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f406b460 0x7f33f4155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33f8f9c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f406b460 0x7f33f4155ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55526/0 (socket says 172.21.15.46:55526) 2024-01-27T01:08:58.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33f8f9c700 1 -- 172.21.15.46:0/1252336779 learned_addr learned my addr 172.21.15.46:0/1252336779 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:58.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33f3fff700 1 --2- 172.21.15.46:0/1252336779 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33f40c7f90 0x7f33f4169a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33f979d700 1 --2- 172.21.15.46:0/1252336779 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33f4157570 0x7f33f40ae4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33f8f9c700 1 -- 172.21.15.46:0/1252336779 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33f4157570 msgr2=0x7f33f40ae4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33f8f9c700 1 --2- 172.21.15.46:0/1252336779 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33f4157570 0x7f33f40ae4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33f8f9c700 1 -- 172.21.15.46:0/1252336779 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33f40c7f90 msgr2=0x7f33f4169a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33f8f9c700 1 --2- 172.21.15.46:0/1252336779 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33f40c7f90 0x7f33f4169a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33f8f9c700 1 -- 172.21.15.46:0/1252336779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e80094f0 con 0x7f33f406b460 2024-01-27T01:08:58.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33f3fff700 1 --2- 172.21.15.46:0/1252336779 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33f40c7f90 0x7f33f4169a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:58.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33f8f9c700 1 --2- 172.21.15.46:0/1252336779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f406b460 0x7f33f4155ce0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f33e8009290 tx=0x7f33e8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:58.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.505+0000 7f33f0ff9700 1 -- 172.21.15.46:0/1252336779 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e8026480 con 0x7f33f406b460 2024-01-27T01:08:58.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.506+0000 7f33f0ff9700 1 -- 172.21.15.46:0/1252336779 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e801f070 con 0x7f33f406b460 2024-01-27T01:08:58.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.506+0000 7f33fb200700 1 -- 172.21.15.46:0/1252336779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f416a910 con 0x7f33f406b460 2024-01-27T01:08:58.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.506+0000 7f33f0ff9700 1 -- 172.21.15.46:0/1252336779 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e8026830 con 0x7f33f406b460 2024-01-27T01:08:58.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.506+0000 7f33fb200700 1 -- 172.21.15.46:0/1252336779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f4169f70 con 0x7f33f406b460 2024-01-27T01:08:58.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.507+0000 7f33de7fc700 1 -- 172.21.15.46:0/1252336779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33cc0052b0 con 0x7f33f406b460 2024-01-27T01:08:58.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.507+0000 7f33f0ff9700 1 -- 172.21.15.46:0/1252336779 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f33e8026ba0 con 0x7f33f406b460 2024-01-27T01:08:58.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.507+0000 7f33f0ff9700 1 --2- 172.21.15.46:0/1252336779 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f33d8066f30 0x7f33d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.508+0000 7f33f0ff9700 1 -- 172.21.15.46:0/1252336779 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f33e8021070 con 0x7f33f406b460 2024-01-27T01:08:58.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.508+0000 7f33f3fff700 1 --2- 172.21.15.46:0/1252336779 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f33d8066f30 0x7f33d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.508+0000 7f33f3fff700 1 --2- 172.21.15.46:0/1252336779 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f33d8066f30 0x7f33d80693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f33f4156be0 tx=0x7f33e0008040 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:58.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.512+0000 7f33f0ff9700 1 -- 172.21.15.46:0/1252336779 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f33e80535d0 con 0x7f33f406b460 2024-01-27T01:08:58.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.719+0000 7f33de7fc700 1 -- 172.21.15.46:0/1252336779 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f33cc000bc0 con 0x7f33d8066f30 2024-01-27T01:08:58.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.720+0000 7f33f0ff9700 1 -- 172.21.15.46:0/1252336779 <== mgr.10565 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f33cc000bc0 con 0x7f33d8066f30 2024-01-27T01:08:58.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.722+0000 7f33de7fc700 1 -- 172.21.15.46:0/1252336779 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f33d8066f30 msgr2=0x7f33d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.722+0000 7f33de7fc700 1 --2- 172.21.15.46:0/1252336779 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f33d8066f30 0x7f33d80693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f33f4156be0 tx=0x7f33e0008040 comp rx=0 tx=0).stop 2024-01-27T01:08:58.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.722+0000 7f33de7fc700 1 -- 172.21.15.46:0/1252336779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f406b460 msgr2=0x7f33f4155ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.722+0000 7f33de7fc700 1 --2- 172.21.15.46:0/1252336779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f406b460 0x7f33f4155ce0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f33e8009290 tx=0x7f33e8017000 comp rx=0 tx=0).stop 2024-01-27T01:08:58.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.722+0000 7f33de7fc700 1 -- 172.21.15.46:0/1252336779 shutdown_connections 2024-01-27T01:08:58.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.722+0000 7f33de7fc700 1 --2- 172.21.15.46:0/1252336779 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f33d8066f30 0x7f33d80693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.722+0000 7f33de7fc700 1 --2- 172.21.15.46:0/1252336779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f406b460 0x7f33f4155ce0 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.722+0000 7f33de7fc700 1 --2- 172.21.15.46:0/1252336779 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f33f4157570 0x7f33f40ae4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.722+0000 7f33de7fc700 1 --2- 172.21.15.46:0/1252336779 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33f40c7f90 0x7f33f4169a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.722+0000 7f33de7fc700 1 -- 172.21.15.46:0/1252336779 >> 172.21.15.46:0/1252336779 conn(0x7f33f4068ac0 msgr2=0x7f33f40b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.722+0000 7f33de7fc700 1 -- 172.21.15.46:0/1252336779 shutdown_connections 2024-01-27T01:08:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.722+0000 7f33de7fc700 1 -- 172.21.15.46:0/1252336779 wait complete. 2024-01-27T01:08:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 6 to deep-scrub 2024-01-27T01:08:58.735 INFO:tasks.ceph:Scrubbing osd.7 2024-01-27T01:08:58.735 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-01-27T01:08:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.871+0000 7fd2f5ad3700 1 Processor -- start 2024-01-27T01:08:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.871+0000 7fd2f5ad3700 1 -- start start 2024-01-27T01:08:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.872+0000 7fd2f5ad3700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd2f006b4b0 0x7fd2f0157350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.872+0000 7fd2f5ad3700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd2f00b2240 0x7fd2f0157890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.872+0000 7fd2f5ad3700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f0156ec0 con 0x7fd2f00c97f0 2024-01-27T01:08:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.872+0000 7fd2f5ad3700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f0156d60 con 0x7fd2f006b4b0 2024-01-27T01:08:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.872+0000 7fd2f5ad3700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f006f120 con 0x7fd2f00b2240 2024-01-27T01:08:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.873+0000 7fd2eeffd700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd2f006b4b0 0x7fd2f0157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.873+0000 7fd2ef7fe700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd2f00c97f0 0x7fd2f00b1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32808/0 (socket says 172.21.15.46:32808) 2024-01-27T01:08:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.873+0000 7fd2ef7fe700 1 -- 172.21.15.46:0/2646303483 learned_addr learned my addr 172.21.15.46:0/2646303483 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.873+0000 7fd2effff700 1 --2- 172.21.15.46:0/2646303483 >> v2:172.21.15.106:3300/0 conn(0x7fd2f00b2240 0x7fd2f0157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.873+0000 7fd2eeffd700 1 -- 172.21.15.46:0/2646303483 >> v2:172.21.15.106:3300/0 conn(0x7fd2f00b2240 msgr2=0x7fd2f0157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:58.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.873+0000 7fd2eeffd700 1 --2- 172.21.15.46:0/2646303483 >> v2:172.21.15.106:3300/0 conn(0x7fd2f00b2240 0x7fd2f0157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.873+0000 7fd2eeffd700 1 -- 172.21.15.46:0/2646303483 >> v1:172.21.15.46:6789/0 conn(0x7fd2f00c97f0 legacy=0x7fd2f00b1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.873+0000 7fd2ee7fc700 1 -- 172.21.15.46:0/2646303483 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568480182 0 0) 0x7fd2f0156ec0 con 0x7fd2f00c97f0 2024-01-27T01:08:58.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.873+0000 7fd2eeffd700 1 -- 172.21.15.46:0/2646303483 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f006f400 con 0x7fd2f006b4b0 2024-01-27T01:08:58.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.873+0000 7fd2eeffd700 1 --2- 172.21.15.46:0/2646303483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f006b4b0 0x7fd2f0157350 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7fd2f006c440 tx=0x7fd2e40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aacd39a563d516bb server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:58.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.873+0000 7fd2ee7fc700 1 -- 172.21.15.46:0/2646303483 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4011630 con 0x7fd2f006b4b0 2024-01-27T01:08:58.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.873+0000 7fd2ee7fc700 1 -- 172.21.15.46:0/2646303483 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e4011790 con 0x7fd2f006b4b0 2024-01-27T01:08:58.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.873+0000 7fd2ee7fc700 1 -- 172.21.15.46:0/2646303483 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4011b30 con 0x7fd2f006b4b0 2024-01-27T01:08:58.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.874+0000 7fd2f5ad3700 1 -- 172.21.15.46:0/2646303483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f006b4b0 msgr2=0x7fd2f0157350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.874+0000 7fd2f5ad3700 1 --2- 172.21.15.46:0/2646303483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f006b4b0 0x7fd2f0157350 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7fd2f006c440 tx=0x7fd2e40091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.874+0000 7fd2f5ad3700 1 -- 172.21.15.46:0/2646303483 shutdown_connections 2024-01-27T01:08:58.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.874+0000 7fd2f5ad3700 1 --2- 172.21.15.46:0/2646303483 >> v2:172.21.15.106:3300/0 conn(0x7fd2f00b2240 0x7fd2f0157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.874+0000 7fd2f5ad3700 1 --2- 172.21.15.46:0/2646303483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f006b4b0 0x7fd2f0157350 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.874+0000 7fd2f5ad3700 1 -- 172.21.15.46:0/2646303483 >> 172.21.15.46:0/2646303483 conn(0x7fd2f0069590 msgr2=0x7fd2f006aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:58.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.874+0000 7fd2f5ad3700 1 -- 172.21.15.46:0/2646303483 shutdown_connections 2024-01-27T01:08:58.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.874+0000 7fd2f5ad3700 1 -- 172.21.15.46:0/2646303483 wait complete. 2024-01-27T01:08:58.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.876+0000 7fd2f5ad3700 1 Processor -- start 2024-01-27T01:08:58.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.876+0000 7fd2f5ad3700 1 -- start start 2024-01-27T01:08:58.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2f5ad3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f006b4b0 0x7fd2f00abd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2f5ad3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f00b2240 0x7fd2f00cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2f5ad3700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f00c97f0 0x7fd2f00a79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2f5ad3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f00a5e40 con 0x7fd2f00b2240 2024-01-27T01:08:58.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2f5ad3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f00cc120 con 0x7fd2f006b4b0 2024-01-27T01:08:58.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2f5ad3700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f00a5510 con 0x7fd2f00c97f0 2024-01-27T01:08:58.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2eeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f00b2240 0x7fd2f00cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2eeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f00b2240 0x7fd2f00cb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55546/0 (socket says 172.21.15.46:55546) 2024-01-27T01:08:58.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2eeffd700 1 -- 172.21.15.46:0/3406757520 learned_addr learned my addr 172.21.15.46:0/3406757520 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:58.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2effff700 1 --2- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f00c97f0 0x7fd2f00a79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2ef7fe700 1 --2- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f006b4b0 0x7fd2f00abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2eeffd700 1 -- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f00c97f0 msgr2=0x7fd2f00a79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2eeffd700 1 --2- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f00c97f0 0x7fd2f00a79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2eeffd700 1 -- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f006b4b0 msgr2=0x7fd2f00abd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2eeffd700 1 --2- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f006b4b0 0x7fd2f00abd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2eeffd700 1 -- 172.21.15.46:0/3406757520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e4009420 con 0x7fd2f00b2240 2024-01-27T01:08:58.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2effff700 1 --2- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd2f00c97f0 0x7fd2f00a79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:08:58.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.877+0000 7fd2ef7fe700 1 --2- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd2f006b4b0 0x7fd2f00abd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T01:08:58.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.878+0000 7fd2eeffd700 1 --2- 172.21.15.46:0/3406757520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f00b2240 0x7fd2f00cb650 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4013000 tx=0x7fd2e4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:58.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.878+0000 7fd2d77fe700 1 -- 172.21.15.46:0/3406757520 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4011b10 con 0x7fd2f00b2240 2024-01-27T01:08:58.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.878+0000 7fd2d77fe700 1 -- 172.21.15.46:0/3406757520 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e40233f0 con 0x7fd2f00b2240 2024-01-27T01:08:58.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.878+0000 7fd2f5ad3700 1 -- 172.21.15.46:0/3406757520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f00a57f0 con 0x7fd2f00b2240 2024-01-27T01:08:58.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.878+0000 7fd2d77fe700 1 -- 172.21.15.46:0/3406757520 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e40237b0 con 0x7fd2f00b2240 2024-01-27T01:08:58.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.878+0000 7fd2f5ad3700 1 -- 172.21.15.46:0/3406757520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f00ac2d0 con 0x7fd2f00b2240 2024-01-27T01:08:58.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.879+0000 7fd2f5ad3700 1 -- 172.21.15.46:0/3406757520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd2f004c710 con 0x7fd2f00b2240 2024-01-27T01:08:58.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.880+0000 7fd2d77fe700 1 -- 172.21.15.46:0/3406757520 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd2e40193b0 con 0x7fd2f00b2240 2024-01-27T01:08:58.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.880+0000 7fd2d77fe700 1 --2- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd2dc066f30 0x7fd2dc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.880+0000 7fd2d77fe700 1 -- 172.21.15.46:0/3406757520 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7fd2e4086470 con 0x7fd2f00b2240 2024-01-27T01:08:58.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.880+0000 7fd2d77fe700 1 --2- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:6824/44081,v1:172.21.15.106:6825/44081] conn(0x7fd2dc06dd40 0x7fd2dc0701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:58.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.880+0000 7fd2ef7fe700 1 --2- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd2dc066f30 0x7fd2dc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.880+0000 7fd2d77fe700 1 -- 172.21.15.46:0/3406757520 --> [v2:172.21.15.106:6824/44081,v1:172.21.15.106:6825/44081] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2dc070800 con 0x7fd2dc06dd40 2024-01-27T01:08:58.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.880+0000 7fd2d77fe700 1 -- 172.21.15.46:0/3406757520 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=269) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2e4086840 con 0x7fd2f00b2240 2024-01-27T01:08:58.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.880+0000 7fd2effff700 1 --2- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:6824/44081,v1:172.21.15.106:6825/44081] conn(0x7fd2dc06dd40 0x7fd2dc0701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:58.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.881+0000 7fd2ef7fe700 1 --2- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd2dc066f30 0x7fd2dc0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0001840 tx=0x7fd2e0009040 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:58.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.881+0000 7fd2effff700 1 --2- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:6824/44081,v1:172.21.15.106:6825/44081] conn(0x7fd2dc06dd40 0x7fd2dc0701d0 crc :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:58.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.882+0000 7fd2d77fe700 1 -- 172.21.15.46:0/3406757520 <== osd.7 v2:172.21.15.106:6824/44081 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fd2dc070800 con 0x7fd2dc06dd40 2024-01-27T01:08:58.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.896+0000 7fd2f5ad3700 1 -- 172.21.15.46:0/3406757520 --> [v2:172.21.15.106:6824/44081,v1:172.21.15.106:6825/44081] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd2f006bad0 con 0x7fd2dc06dd40 2024-01-27T01:08:58.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.897+0000 7fd2d77fe700 1 -- 172.21.15.46:0/3406757520 <== osd.7 v2:172.21.15.106:6824/44081 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fd2f006bad0 con 0x7fd2dc06dd40 2024-01-27T01:08:58.897 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-01-27T01:08:58.897 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-27T01:08:58.897 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-01-27T01:08:58.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.897+0000 7fd2f5ad3700 1 -- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:6824/44081,v1:172.21.15.106:6825/44081] conn(0x7fd2dc06dd40 msgr2=0x7fd2dc0701d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.897+0000 7fd2f5ad3700 1 --2- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:6824/44081,v1:172.21.15.106:6825/44081] conn(0x7fd2dc06dd40 0x7fd2dc0701d0 crc :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:58.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.897+0000 7fd2f5ad3700 1 -- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd2dc066f30 msgr2=0x7fd2dc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.897+0000 7fd2f5ad3700 1 --2- 172.21.15.46:0/3406757520 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fd2dc066f30 0x7fd2dc0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0001840 tx=0x7fd2e0009040 comp rx=0 tx=0).stop 2024-01-27T01:08:58.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.897+0000 7fd2f5ad3700 1 -- 172.21.15.46:0/3406757520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f00b2240 msgr2=0x7fd2f00cb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.897+0000 7fd2f5ad3700 1 --2- 172.21.15.46:0/3406757520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2f00b2240 0x7fd2f00cb650 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4013000 tx=0x7fd2e4017000 comp rx=0 tx=0).stop 2024-01-27T01:08:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.897+0000 7fd2effff700 1 -- 172.21.15.46:0/3406757520 reap_dead start 2024-01-27T01:08:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.897+0000 7fd2f5ad3700 1 -- 172.21.15.46:0/3406757520 shutdown_connections 2024-01-27T01:08:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.897+0000 7fd2f5ad3700 1 -- 172.21.15.46:0/3406757520 >> 172.21.15.46:0/3406757520 conn(0x7fd2f0069590 msgr2=0x7fd2f01539e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.898+0000 7fd2f5ad3700 1 -- 172.21.15.46:0/3406757520 shutdown_connections 2024-01-27T01:08:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:58.898+0000 7fd2f5ad3700 1 -- 172.21.15.46:0/3406757520 wait complete. 2024-01-27T01:08:58.911 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-01-27T01:08:59.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.045+0000 7f12a55aa700 1 Processor -- start 2024-01-27T01:08:59.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.045+0000 7f12a55aa700 1 -- start start 2024-01-27T01:08:59.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.045+0000 7f12a55aa700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f12a0156490 0x7f12a0157530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:59.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.046+0000 7f12a55aa700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12a00c8320 0x7f12a00c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:59.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.046+0000 7f12a55aa700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a006ebb0 con 0x7f12a00c1150 2024-01-27T01:08:59.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.046+0000 7f12a55aa700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a006f9f0 con 0x7f12a0156490 2024-01-27T01:08:59.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.046+0000 7f12a55aa700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0157a70 con 0x7f12a00c8320 2024-01-27T01:08:59.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.046+0000 7f129e7fc700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f12a00c1150 0x7f12a00c7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32816/0 (socket says 172.21.15.46:32816) 2024-01-27T01:08:59.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.046+0000 7f129e7fc700 1 -- 172.21.15.46:0/1091884126 learned_addr learned my addr 172.21.15.46:0/1091884126 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:59.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.046+0000 7f129effd700 1 --2- 172.21.15.46:0/1091884126 >> v2:172.21.15.106:3300/0 conn(0x7f12a0156490 0x7f12a0157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:59.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.046+0000 7f129f7fe700 1 --2- 172.21.15.46:0/1091884126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12a00c8320 0x7f12a00c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:59.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.046+0000 7f129dffb700 1 -- 172.21.15.46:0/1091884126 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447599574 0 0) 0x7f12a006ebb0 con 0x7f12a00c1150 2024-01-27T01:08:59.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.046+0000 7f129dffb700 1 -- 172.21.15.46:0/1091884126 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1288003570 con 0x7f12a00c1150 2024-01-27T01:08:59.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.046+0000 7f129effd700 1 -- 172.21.15.46:0/1091884126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12a00c8320 msgr2=0x7f12a00c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:08:59.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.046+0000 7f129effd700 1 --2- 172.21.15.46:0/1091884126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12a00c8320 0x7f12a00c9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.046+0000 7f129effd700 1 -- 172.21.15.46:0/1091884126 >> v1:172.21.15.46:6789/0 conn(0x7f12a00c1150 legacy=0x7f12a00c7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:59.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.046+0000 7f129effd700 1 -- 172.21.15.46:0/1091884126 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a006ede0 con 0x7f12a0156490 2024-01-27T01:08:59.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.046+0000 7f129effd700 1 --2- 172.21.15.46:0/1091884126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12a0156490 0x7f12a0157530 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f12900098e0 tx=0x7f12900091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af0f620c79a17ddf server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.047+0000 7f129dffb700 1 -- 172.21.15.46:0/1091884126 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1290011630 con 0x7f12a0156490 2024-01-27T01:08:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.047+0000 7f129dffb700 1 -- 172.21.15.46:0/1091884126 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1290011790 con 0x7f12a0156490 2024-01-27T01:08:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.047+0000 7f129dffb700 1 -- 172.21.15.46:0/1091884126 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1290011b30 con 0x7f12a0156490 2024-01-27T01:08:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.047+0000 7f12a55aa700 1 -- 172.21.15.46:0/1091884126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12a0156490 msgr2=0x7f12a0157530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.047+0000 7f12a55aa700 1 --2- 172.21.15.46:0/1091884126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12a0156490 0x7f12a0157530 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f12900098e0 tx=0x7f12900091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.047+0000 7f12a55aa700 1 -- 172.21.15.46:0/1091884126 shutdown_connections 2024-01-27T01:08:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.047+0000 7f12a55aa700 1 --2- 172.21.15.46:0/1091884126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12a00c8320 0x7f12a00c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.047+0000 7f12a55aa700 1 --2- 172.21.15.46:0/1091884126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12a0156490 0x7f12a0157530 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.047+0000 7f12a55aa700 1 -- 172.21.15.46:0/1091884126 >> 172.21.15.46:0/1091884126 conn(0x7f12a00cc3b0 msgr2=0x7f12a00cc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.047+0000 7f12a55aa700 1 -- 172.21.15.46:0/1091884126 shutdown_connections 2024-01-27T01:08:59.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.047+0000 7f12a55aa700 1 -- 172.21.15.46:0/1091884126 wait complete. 2024-01-27T01:08:59.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.049+0000 7f12a55aa700 1 Processor -- start 2024-01-27T01:08:59.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.049+0000 7f12a55aa700 1 -- start start 2024-01-27T01:08:59.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f12a55aa700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12a00c1150 0x7f12a0153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:59.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f12a55aa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12a00c8320 0x7f12a00b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:59.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f12a55aa700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12a0156490 0x7f12a00c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:59.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f12a55aa700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a00af940 con 0x7f12a00c8320 2024-01-27T01:08:59.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f12a55aa700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a00c7340 con 0x7f12a0156490 2024-01-27T01:08:59.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f12a55aa700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a00c75c0 con 0x7f12a00c1150 2024-01-27T01:08:59.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f129e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12a00c8320 0x7f12a00b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:59.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f129e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12a00c8320 0x7f12a00b0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55556/0 (socket says 172.21.15.46:55556) 2024-01-27T01:08:59.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f129e7fc700 1 -- 172.21.15.46:0/3348047891 learned_addr learned my addr 172.21.15.46:0/3348047891 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:59.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f129effd700 1 --2- 172.21.15.46:0/3348047891 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12a00c1150 0x7f12a0153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:59.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f129f7fe700 1 --2- 172.21.15.46:0/3348047891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12a0156490 0x7f12a00c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:59.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f129e7fc700 1 -- 172.21.15.46:0/3348047891 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12a00c1150 msgr2=0x7f12a0153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:59.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f129e7fc700 1 --2- 172.21.15.46:0/3348047891 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12a00c1150 0x7f12a0153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f129e7fc700 1 -- 172.21.15.46:0/3348047891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12a0156490 msgr2=0x7f12a00c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:59.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f129e7fc700 1 --2- 172.21.15.46:0/3348047891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12a0156490 0x7f12a00c6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f129e7fc700 1 -- 172.21.15.46:0/3348047891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1290009420 con 0x7f12a00c8320 2024-01-27T01:08:59.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f129effd700 1 --2- 172.21.15.46:0/3348047891 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12a00c1150 0x7f12a0153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T01:08:59.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f129e7fc700 1 --2- 172.21.15.46:0/3348047891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12a00c8320 0x7f12a00b0cb0 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f129400c7f0 tx=0x7f1294007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:59.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f1286ffd700 1 -- 172.21.15.46:0/3348047891 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1294005b30 con 0x7f12a00c8320 2024-01-27T01:08:59.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f1286ffd700 1 -- 172.21.15.46:0/3348047891 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1294005c90 con 0x7f12a00c8320 2024-01-27T01:08:59.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.050+0000 7f12a55aa700 1 -- 172.21.15.46:0/3348047891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a00af2f0 con 0x7f12a00c8320 2024-01-27T01:08:59.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.051+0000 7f1286ffd700 1 -- 172.21.15.46:0/3348047891 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1294007340 con 0x7f12a00c8320 2024-01-27T01:08:59.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.051+0000 7f12a55aa700 1 -- 172.21.15.46:0/3348047891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a00afdd0 con 0x7f12a00c8320 2024-01-27T01:08:59.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.052+0000 7f12a55aa700 1 -- 172.21.15.46:0/3348047891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a00af150 con 0x7f12a00c8320 2024-01-27T01:08:59.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.052+0000 7f1286ffd700 1 -- 172.21.15.46:0/3348047891 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f129401b400 con 0x7f12a00c8320 2024-01-27T01:08:59.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.052+0000 7f1286ffd700 1 --2- 172.21.15.46:0/3348047891 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f128c066f30 0x7f128c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:59.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.053+0000 7f1286ffd700 1 -- 172.21.15.46:0/3348047891 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f1294084400 con 0x7f12a00c8320 2024-01-27T01:08:59.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.053+0000 7f129effd700 1 --2- 172.21.15.46:0/3348047891 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f128c066f30 0x7f128c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:59.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.053+0000 7f129effd700 1 --2- 172.21.15.46:0/3348047891 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f128c066f30 0x7f128c0693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f12900091e0 tx=0x7f1290017000 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:59.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.057+0000 7f1286ffd700 1 -- 172.21.15.46:0/3348047891 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f12a00af150 con 0x7f12a00c8320 2024-01-27T01:08:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.261+0000 7f12a55aa700 1 -- 172.21.15.46:0/3348047891 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f12a00af150 con 0x7f128c066f30 2024-01-27T01:08:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.261+0000 7f1286ffd700 1 -- 172.21.15.46:0/3348047891 <== mgr.10565 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f12a00af150 con 0x7f128c066f30 2024-01-27T01:08:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.264+0000 7f12a55aa700 1 -- 172.21.15.46:0/3348047891 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f128c066f30 msgr2=0x7f128c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.264+0000 7f12a55aa700 1 --2- 172.21.15.46:0/3348047891 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f128c066f30 0x7f128c0693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f12900091e0 tx=0x7f1290017000 comp rx=0 tx=0).stop 2024-01-27T01:08:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.264+0000 7f12a55aa700 1 -- 172.21.15.46:0/3348047891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12a00c8320 msgr2=0x7f12a00b0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.264+0000 7f12a55aa700 1 --2- 172.21.15.46:0/3348047891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12a00c8320 0x7f12a00b0cb0 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f129400c7f0 tx=0x7f1294007e00 comp rx=0 tx=0).stop 2024-01-27T01:08:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.264+0000 7f12a55aa700 1 -- 172.21.15.46:0/3348047891 shutdown_connections 2024-01-27T01:08:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.264+0000 7f12a55aa700 1 --2- 172.21.15.46:0/3348047891 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f128c066f30 0x7f128c0693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.264+0000 7f12a55aa700 1 --2- 172.21.15.46:0/3348047891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12a0156490 0x7f12a00c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.264+0000 7f12a55aa700 1 --2- 172.21.15.46:0/3348047891 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f12a00c1150 0x7f12a0153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.264+0000 7f12a55aa700 1 --2- 172.21.15.46:0/3348047891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12a00c8320 0x7f12a00b0cb0 unknown :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.264+0000 7f12a55aa700 1 -- 172.21.15.46:0/3348047891 >> 172.21.15.46:0/3348047891 conn(0x7f12a00cc3b0 msgr2=0x7f12a0155710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.264+0000 7f12a55aa700 1 -- 172.21.15.46:0/3348047891 shutdown_connections 2024-01-27T01:08:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.264+0000 7f12a55aa700 1 -- 172.21.15.46:0/3348047891 wait complete. 2024-01-27T01:08:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 7 to deep-scrub 2024-01-27T01:08:59.277 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:08:59.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.409+0000 7fb89f595700 1 Processor -- start 2024-01-27T01:08:59.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.409+0000 7fb89f595700 1 -- start start 2024-01-27T01:08:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.410+0000 7fb89f595700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb8980c3390 0x7fb8980c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.410+0000 7fb89f595700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8980c4a40 con 0x7fb898069920 2024-01-27T01:08:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.410+0000 7fb89f595700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb898088620 con 0x7fb89806a4b0 2024-01-27T01:08:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.410+0000 7fb89f595700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb898068b60 con 0x7fb8980c3390 2024-01-27T01:08:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.410+0000 7fb89cb30700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb898069920 0x7fb898069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32824/0 (socket says 172.21.15.46:32824) 2024-01-27T01:08:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.410+0000 7fb89d331700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb8980c3390 0x7fb8980c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.410+0000 7fb89cb30700 1 -- 172.21.15.46:0/864355120 learned_addr learned my addr 172.21.15.46:0/864355120 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:08:59.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.410+0000 7fb89d331700 1 -- 172.21.15.46:0/864355120 >> v1:172.21.15.106:6789/0 conn(0x7fb89806a4b0 legacy=0x7fb8980c3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:59.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.410+0000 7fb89d331700 1 -- 172.21.15.46:0/864355120 >> v1:172.21.15.46:6789/0 conn(0x7fb898069920 legacy=0x7fb898069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:59.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.410+0000 7fb89d331700 1 -- 172.21.15.46:0/864355120 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb898068e40 con 0x7fb8980c3390 2024-01-27T01:08:59.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.410+0000 7fb88ffff700 1 -- 172.21.15.46:0/864355120 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841027273 0 0) 0x7fb8980c4a40 con 0x7fb898069920 2024-01-27T01:08:59.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.411+0000 7fb89d331700 1 --2- 172.21.15.46:0/864355120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8980c3390 0x7fb8980c8c30 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fb8940098e0 tx=0x7fb8940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52bda2d4f5256c0a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:59.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.411+0000 7fb88ffff700 1 -- 172.21.15.46:0/864355120 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb894009e80 con 0x7fb8980c3390 2024-01-27T01:08:59.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.411+0000 7fb88ffff700 1 -- 172.21.15.46:0/864355120 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89400b860 con 0x7fb8980c3390 2024-01-27T01:08:59.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.411+0000 7fb88ffff700 1 -- 172.21.15.46:0/864355120 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89400bbe0 con 0x7fb8980c3390 2024-01-27T01:08:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.411+0000 7fb89f595700 1 -- 172.21.15.46:0/864355120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8980c3390 msgr2=0x7fb8980c8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.411+0000 7fb89f595700 1 --2- 172.21.15.46:0/864355120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8980c3390 0x7fb8980c8c30 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fb8940098e0 tx=0x7fb8940091a0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.411+0000 7fb89f595700 1 -- 172.21.15.46:0/864355120 shutdown_connections 2024-01-27T01:08:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.411+0000 7fb89f595700 1 --2- 172.21.15.46:0/864355120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8980c3390 0x7fb8980c8c30 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.411+0000 7fb89f595700 1 -- 172.21.15.46:0/864355120 >> 172.21.15.46:0/864355120 conn(0x7fb898087fa0 msgr2=0x7fb8980ce5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:59.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.412+0000 7fb89f595700 1 -- 172.21.15.46:0/864355120 shutdown_connections 2024-01-27T01:08:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.412+0000 7fb89f595700 1 -- 172.21.15.46:0/864355120 wait complete. 2024-01-27T01:08:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.413+0000 7fb89f595700 1 Processor -- start 2024-01-27T01:08:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.413+0000 7fb89f595700 1 -- start start 2024-01-27T01:08:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.414+0000 7fb89f595700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb898069920 0x7fb89816ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.414+0000 7fb89f595700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb89806a4b0 0x7fb89809b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.414+0000 7fb89f595700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8980c3390 0x7fb89809b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.414+0000 7fb89f595700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb898157700 con 0x7fb8980c3390 2024-01-27T01:08:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.414+0000 7fb89f595700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8980b2f50 con 0x7fb898069920 2024-01-27T01:08:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.414+0000 7fb89f595700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8980b2660 con 0x7fb89806a4b0 2024-01-27T01:08:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.414+0000 7fb89db32700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8980c3390 0x7fb89809b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:59.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.414+0000 7fb89db32700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8980c3390 0x7fb89809b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55570/0 (socket says 172.21.15.46:55570) 2024-01-27T01:08:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.414+0000 7fb89db32700 1 -- 172.21.15.46:0/773205993 learned_addr learned my addr 172.21.15.46:0/773205993 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:08:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.414+0000 7fb89d331700 1 --2- 172.21.15.46:0/773205993 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb898069920 0x7fb89816ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.414+0000 7fb89cb30700 1 --2- 172.21.15.46:0/773205993 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb89806a4b0 0x7fb89809b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.414+0000 7fb89db32700 1 -- 172.21.15.46:0/773205993 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb89806a4b0 msgr2=0x7fb89809b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.414+0000 7fb89db32700 1 --2- 172.21.15.46:0/773205993 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb89806a4b0 0x7fb89809b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.414+0000 7fb89db32700 1 -- 172.21.15.46:0/773205993 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb898069920 msgr2=0x7fb89816ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.414+0000 7fb89db32700 1 --2- 172.21.15.46:0/773205993 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb898069920 0x7fb89816ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.414+0000 7fb89db32700 1 -- 172.21.15.46:0/773205993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb894009420 con 0x7fb8980c3390 2024-01-27T01:08:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.414+0000 7fb89db32700 1 --2- 172.21.15.46:0/773205993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8980c3390 0x7fb89809b850 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7fb89000c930 tx=0x7fb890001750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.415+0000 7fb88d7fa700 1 -- 172.21.15.46:0/773205993 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb890012630 con 0x7fb8980c3390 2024-01-27T01:08:59.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.415+0000 7fb88d7fa700 1 -- 172.21.15.46:0/773205993 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb890012790 con 0x7fb8980c3390 2024-01-27T01:08:59.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.415+0000 7fb89f595700 1 -- 172.21.15.46:0/773205993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb89809bd90 con 0x7fb8980c3390 2024-01-27T01:08:59.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.415+0000 7fb88d7fa700 1 -- 172.21.15.46:0/773205993 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb890012b50 con 0x7fb8980c3390 2024-01-27T01:08:59.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.415+0000 7fb89f595700 1 -- 172.21.15.46:0/773205993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8980b2be0 con 0x7fb8980c3390 2024-01-27T01:08:59.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.416+0000 7fb89f595700 1 -- 172.21.15.46:0/773205993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8980b30e0 con 0x7fb8980c3390 2024-01-27T01:08:59.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.416+0000 7fb88d7fa700 1 -- 172.21.15.46:0/773205993 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb89000a3f0 con 0x7fb8980c3390 2024-01-27T01:08:59.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.416+0000 7fb88d7fa700 1 --2- 172.21.15.46:0/773205993 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb87c066f30 0x7fb87c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:08:59.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.417+0000 7fb88d7fa700 1 -- 172.21.15.46:0/773205993 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7fb890084b10 con 0x7fb8980c3390 2024-01-27T01:08:59.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.417+0000 7fb89d331700 1 --2- 172.21.15.46:0/773205993 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb87c066f30 0x7fb87c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:08:59.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.417+0000 7fb89d331700 1 --2- 172.21.15.46:0/773205993 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb87c066f30 0x7fb87c0693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb8940093f0 tx=0x7fb894017000 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:08:59.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.421+0000 7fb88d7fa700 1 -- 172.21.15.46:0/773205993 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb890051a50 con 0x7fb8980c3390 2024-01-27T01:08:59.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.621+0000 7fb89f595700 1 -- 172.21.15.46:0/773205993 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb89804c710 con 0x7fb87c066f30 2024-01-27T01:08:59.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.624+0000 7fb88d7fa700 1 -- 172.21.15.46:0/773205993 <== mgr.10565 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51234 (secure 0 0 0) 0x7fb89804c710 con 0x7fb87c066f30 2024-01-27T01:08:59.655 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:08:59.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.626+0000 7fb89f595700 1 -- 172.21.15.46:0/773205993 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb87c066f30 msgr2=0x7fb87c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:59.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.626+0000 7fb89f595700 1 --2- 172.21.15.46:0/773205993 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb87c066f30 0x7fb87c0693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb8940093f0 tx=0x7fb894017000 comp rx=0 tx=0).stop 2024-01-27T01:08:59.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.626+0000 7fb89f595700 1 -- 172.21.15.46:0/773205993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8980c3390 msgr2=0x7fb89809b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:08:59.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.626+0000 7fb89f595700 1 --2- 172.21.15.46:0/773205993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8980c3390 0x7fb89809b850 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7fb89000c930 tx=0x7fb890001750 comp rx=0 tx=0).stop 2024-01-27T01:08:59.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.626+0000 7fb89f595700 1 -- 172.21.15.46:0/773205993 shutdown_connections 2024-01-27T01:08:59.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.626+0000 7fb89f595700 1 --2- 172.21.15.46:0/773205993 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7fb87c066f30 0x7fb87c0693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.626+0000 7fb89f595700 1 --2- 172.21.15.46:0/773205993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8980c3390 0x7fb89809b850 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.626+0000 7fb89f595700 1 --2- 172.21.15.46:0/773205993 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb898069920 0x7fb89816ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.626+0000 7fb89f595700 1 --2- 172.21.15.46:0/773205993 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb89806a4b0 0x7fb89809b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:08:59.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.626+0000 7fb89f595700 1 -- 172.21.15.46:0/773205993 >> 172.21.15.46:0/773205993 conn(0x7fb898087fa0 msgr2=0x7fb898159180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:08:59.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.626+0000 7fb89f595700 1 -- 172.21.15.46:0/773205993 shutdown_connections 2024-01-27T01:08:59.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:08:59.626+0000 7fb89f595700 1 -- 172.21.15.46:0/773205993 wait complete. 2024-01-27T01:08:59.658 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:08:59.659 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-01-27T01:08:58.493701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2815,"num_read_kb":5966,"num_write":740,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8941,"ondisk_log_size":8941,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":2488040,"kb_used_data":27816,"kb_used_omap":0,"kb_used_meta":2460160,"kb_avail":752486680,"statfs":{"total":773094113280,"available":770546360320,"internally_reserved":0,"allocated":28483584,"data_stored":15717032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2519203840},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002326"},"pg_stats":[{"pgid":"1.0","version":"268'11170","reported_seq":12302,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-01-27T01:08:48.598795+0000","last_change":"2024-01-26T23:29:40.378321+0000","last_active":"2024-01-27T01:08:48.598795+0000","last_peered":"2024-01-27T01:08:48.598795+0000","last_clean":"2024-01-27T01:08:48.598795+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:08:48.598795+0000","last_undegraded":"2024-01-27T01:08:48.598795+0000","last_fullsized":"2024-01-27T01:08:48.598795+0000","mapping_epoch":14,"log_start":"98'2229","ondisk_log_start":"98'2229","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-26T23:29:38.508092+0000","last_clean_scrub_stamp":"2024-01-26T23:29:38.508092+0000","objects_scrubbed":0,"log_size":8941,"log_dups_size":0,"ondisk_log_size":8941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:43:22.556873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2815,"num_read_kb":5966,"num_write":740,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2815,"num_read_kb":5966,"num_write":740,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8941,"ondisk_log_size":8941,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576028,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313488,"kb_used_data":3912,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058352,"statfs":{"total":96636764160,"available":96315752448,"internally_reserved":0,"allocated":4005888,"data_stored":2407405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.58199999999999996,"15min":0.60199999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.312},"max":{"1min":0.745,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61699999999999999,"15min":0.61799999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33000000000000002},"max":{"1min":0.76400000000000001,"5min":0.879,"15min":0.879},"last":0.629}]},{"osd":1,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.58599999999999997,"15min":0.60399999999999998},"min":{"1min":0.40300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73799999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.34300000000000003},"max":{"1min":0.81499999999999995,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.48599999999999999}]},{"osd":2,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.59499999999999997,"15min":0.60399999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.753,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.39200000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85799999999999998,"5min":0.872,"15min":0.88900000000000001},"last":0.64300000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58799999999999997,"15min":0.60799999999999998},"min":{"1min":0.374,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79100000000000004,"5min":0.85399999999999998,"15min":0.876},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64300000000000002,"15min":0.64000000000000001},"min":{"1min":0.40600000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.80600000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.60399999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.496,"15min":0.48699999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76100000000000001,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.52600000000000002,"15min":0.51500000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67800000000000005,"5min":0.80300000000000005,"15min":0.86599999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.54100000000000004,"15min":0.52400000000000002},"min":{"1min":0.33600000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.499},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.81399999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.52500000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59299999999999997,"15min":0.58399999999999996},"min":{"1min":0.33700000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84499999999999997,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58799999999999997,"15min":0.58199999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77900000000000003,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.61499999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312908,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058932,"statfs":{"total":96636764160,"available":96316346368,"internally_reserved":0,"allocated":3411968,"data_stored":1817037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.56499999999999995,"15min":0.59199999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.83499999999999996,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.745},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.57499999999999996,"15min":0.60799999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73499999999999999,"5min":0.82499999999999996,"15min":0.82899999999999996},"last":0.54500000000000004}]},{"osd":1,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.55800000000000005,"15min":0.59599999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70399999999999996,"5min":0.80300000000000005,"15min":0.84499999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57899999999999996,"15min":0.61099999999999999},"min":{"1min":0.39600000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76400000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69799999999999995}]},{"osd":2,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.55300000000000005,"15min":0.60099999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78100000000000003,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.57699999999999996,"15min":0.61099999999999999},"min":{"1min":0.379,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.754,"5min":0.80400000000000005,"15min":0.878},"last":0.57999999999999996}]},{"osd":3,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.58699999999999997,"15min":0.61499999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80600000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.62,"15min":0.64900000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.877,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.72699999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.495,"15min":0.497},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.80300000000000005},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.441,"5min":0.51600000000000001,"15min":0.53100000000000003},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.61799999999999999}]},{"osd":5,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.50600000000000001,"15min":0.51700000000000002},"min":{"1min":0.309,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70899999999999996,"5min":0.73999999999999999,"15min":0.79300000000000004},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.28899999999999998},"max":{"1min":0.752,"5min":0.87,"15min":0.87},"last":0.77400000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56699999999999995,"15min":0.56999999999999995},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.57499999999999996,"15min":0.58499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.79300000000000004,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.71499999999999997}]}]},{"osd":1,"up_from":13,"seq":55834576028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":304716,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067124,"statfs":{"total":96636764160,"available":96324734976,"internally_reserved":0,"allocated":3411968,"data_stored":1817037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.28299999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.502,"5min":0.66100000000000003,"15min":0.69599999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.432,"15min":0.45100000000000001},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.76600000000000001},"last":0.42599999999999999}]},{"osd":2,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.40799999999999997,"15min":0.432},"min":{"1min":0.29699999999999999,"5min":0.28599999999999998,"15min":0.26200000000000001},"max":{"1min":0.47299999999999998,"5min":0.748,"15min":0.95799999999999996},"last":0.312},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46899999999999997,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.57199999999999995,"5min":0.67400000000000004,"15min":0.79000000000000004},"last":0.41299999999999998}]},{"osd":3,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.501,"15min":0.52600000000000002},"min":{"1min":0.35999999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59099999999999997,"5min":0.73699999999999999,"15min":1.2130000000000001},"last":0.441},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.48599999999999999,"15min":0.51100000000000001},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.32300000000000001},"max":{"1min":0.628,"5min":0.71799999999999997,"15min":1.22},"last":0.36899999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.65200000000000002,"15min":0.66600000000000004},"min":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78600000000000003,"5min":0.998,"15min":1.242},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.66700000000000004,"15min":0.68899999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.35699999999999998},"max":{"1min":0.93600000000000005,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.74199999999999999}]},{"osd":5,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.70999999999999996,"15min":0.70299999999999996},"min":{"1min":0.42299999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81299999999999994,"5min":5.444,"15min":5.444},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.69299999999999995,"15min":0.69999999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82899999999999996,"5min":2.6669999999999998,"15min":2.6669999999999998},"last":0.61599999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.72199999999999998,"15min":0.71899999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47099999999999997},"max":{"1min":0.85499999999999998,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.79100000000000004,"15min":0.73899999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48499999999999999},"max":{"1min":0.82199999999999995,"5min":11.223000000000001,"15min":11.223000000000001},"last":0.59099999999999997}]},{"osd":7,"last update":"Sat Jan 27 01:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.72799999999999998,"15min":0.748},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97799999999999998,"5min":2.9660000000000002,"15min":2.9660000000000002},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.751,"15min":0.76300000000000001},"min":{"1min":0.55400000000000005,"5min":0.53400000000000003,"15min":0.52000000000000002},"max":{"1min":0.83199999999999996,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.59999999999999998}]}]},{"osd":0,"up_from":13,"seq":55834576027,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309392,"kb_used_data":3912,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062448,"statfs":{"total":96636764160,"available":96319946752,"internally_reserved":0,"allocated":4005888,"data_stored":2407405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.42499999999999999,"15min":0.42199999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.53600000000000003,"5min":1.276,"15min":1.276},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.46100000000000002},"min":{"1min":0.29999999999999999,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.34699999999999998}]},{"osd":2,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.29199999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.308},{"interface":"front","average":{"1min":0.443,"5min":0.46400000000000002,"15min":0.48199999999999998},"min":{"1min":0.32500000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.40799999999999997}]},{"osd":3,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48399999999999999,"15min":0.48499999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65800000000000003,"5min":0.80000000000000004,"15min":0.83699999999999997},"last":0.437},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.498,"15min":0.503},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.65100000000000002,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.42699999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.42199999999999999},"max":{"1min":0.90200000000000002,"5min":0.91200000000000003,"15min":1.0720000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.64600000000000002,"15min":0.66800000000000004},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.81299999999999994,"5min":1.002,"15min":1.002},"last":0.49299999999999999}]},{"osd":5,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.67600000000000005,"15min":0.69199999999999995},"min":{"1min":0.47699999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.92400000000000004,"5min":1.0209999999999999,"15min":2.165},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.93700000000000006,"5min":1.0700000000000001,"15min":1.1399999999999999},"last":0.69799999999999995}]},{"osd":6,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67900000000000005,"15min":0.70399999999999996},"min":{"1min":0.46800000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82799999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70699999999999996,"15min":0.73199999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.425},"last":0.65200000000000002}]},{"osd":7,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.71899999999999997,"15min":0.745},"min":{"1min":0.48199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.98899999999999999,"5min":1.0169999999999999,"15min":1.1259999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.747,"15min":0.76100000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.96299999999999997,"5min":1.042,"15min":1.1910000000000001},"last":0.79100000000000004}]}]},{"osd":2,"up_from":13,"seq":55834576027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312908,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058932,"statfs":{"total":96636764160,"available":96316346368,"internally_reserved":0,"allocated":3411968,"data_stored":1817037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38300000000000001,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.251},"max":{"1min":0.56000000000000005,"5min":0.60199999999999998,"15min":0.746},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.438},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.57999999999999996,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.57999999999999996}]},{"osd":1,"last update":"Sat Jan 27 01:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42599999999999999,"15min":0.44},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61299999999999999,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.49099999999999999,"15min":0.51100000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.65000000000000002,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.34100000000000003}]},{"osd":3,"last update":"Sat Jan 27 01:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46600000000000003,"15min":0.48699999999999999},"min":{"1min":0.30099999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.61099999999999999,"5min":0.65400000000000003,"15min":0.78100000000000003},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46100000000000002,"15min":0.47999999999999998},"min":{"1min":0.34300000000000003,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.35999999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.64500000000000002,"15min":0.65900000000000003},"min":{"1min":0.434,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.82599999999999996,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.64800000000000002,"15min":0.67600000000000005},"min":{"1min":0.495,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.73899999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.625}]},{"osd":5,"last update":"Sat Jan 27 01:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.48399999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.75600000000000001,"5min":1.075,"15min":1.075},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.65800000000000003,"15min":0.69099999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39800000000000002},"max":{"1min":0.80300000000000005,"5min":0.89600000000000002,"15min":1.0229999999999999},"last":0.53600000000000003}]},{"osd":6,"last update":"Sat Jan 27 01:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.66600000000000004,"15min":0.69299999999999995},"min":{"1min":0.5,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.877,"5min":0.90200000000000002,"15min":1.0549999999999999},"last":0.5},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.66700000000000004,"15min":0.69899999999999995},"min":{"1min":0.47699999999999998,"5min":0.44500000000000001,"15min":0.39200000000000002},"max":{"1min":0.879,"5min":0.91200000000000003,"15min":1.0269999999999999},"last":0.59599999999999997}]},{"osd":7,"last update":"Sat Jan 27 01:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69899999999999995,"15min":0.72899999999999998},"min":{"1min":0.52800000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.88700000000000001,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.71299999999999997,"15min":0.747},"min":{"1min":0.54400000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85099999999999998,"5min":0.94599999999999995,"15min":1.0880000000000001},"last":0.63800000000000001}]}]},{"osd":3,"up_from":13,"seq":55834576026,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":304716,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067124,"statfs":{"total":96636764160,"available":96324734976,"internally_reserved":0,"allocated":3411968,"data_stored":1817037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.436,"15min":0.41899999999999998},"min":{"1min":0.27400000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.45000000000000001,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.246},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.44900000000000001,"15min":0.45400000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.28299999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.36399999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.60999999999999999,"15min":0.52900000000000003},"min":{"1min":0.314,"5min":0.30099999999999999,"15min":0.28899999999999998},"max":{"1min":0.754,"5min":15.391,"15min":15.391},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.57099999999999995,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.32600000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52800000000000002,"15min":0.51100000000000001},"min":{"1min":0.36699999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64800000000000002,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.53600000000000003,"15min":0.50900000000000001},"min":{"1min":0.32900000000000001,"5min":0.308,"15min":0.29799999999999999},"max":{"1min":0.82299999999999995,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":0.51900000000000002}]},{"osd":4,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.71799999999999997,"15min":0.70999999999999996},"min":{"1min":0.46600000000000003,"5min":0.46000000000000002,"15min":0.439},"max":{"1min":0.76400000000000001,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.70199999999999996,"15min":0.69599999999999995},"min":{"1min":0.442,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":0.76900000000000002,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.56399999999999995}]},{"osd":5,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.71599999999999997,"15min":0.70299999999999996},"min":{"1min":0.47099999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76100000000000001,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76600000000000001,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.79300000000000004}]},{"osd":6,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.72799999999999998,"15min":0.71399999999999997},"min":{"1min":0.54100000000000004,"5min":0.49199999999999999,"15min":0.47499999999999998},"max":{"1min":0.83499999999999996,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.71399999999999997,"15min":0.72699999999999998},"min":{"1min":0.46300000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81200000000000006,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.61699999999999999}]},{"osd":7,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.73899999999999999,"15min":0.72699999999999998},"min":{"1min":0.52500000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1779999999999999,"5min":4.569,"15min":4.569},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.77100000000000002,"15min":0.77200000000000002},"min":{"1min":0.56499999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.80700000000000005,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.83599999999999997}]}]},{"osd":4,"up_from":13,"seq":55834576028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":317004,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94054836,"statfs":{"total":96636764160,"available":96312152064,"internally_reserved":0,"allocated":3411968,"data_stored":1817037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68500000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59699999999999998,"15min":0.60899999999999999},"min":{"1min":0.35499999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.85699999999999998},"last":0.49199999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.59899999999999998,"15min":0.622},"min":{"1min":0.41899999999999998,"5min":0.39100000000000001,"15min":0.38800000000000001},"max":{"1min":0.80700000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60999999999999999,"15min":0.62},"min":{"1min":0.38800000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79100000000000004,"5min":0.874,"15min":0.874},"last":0.63300000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.59399999999999997},"min":{"1min":0.34200000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80500000000000005,"5min":0.85599999999999998,"15min":0.88700000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.626,"15min":0.63300000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.80300000000000005,"5min":0.81499999999999995,"15min":0.89200000000000002},"last":0.70199999999999996}]},{"osd":3,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59699999999999998,"15min":0.61699999999999999},"min":{"1min":0.377,"5min":0.373,"15min":0.373},"max":{"1min":0.747,"5min":0.83299999999999996,"15min":0.84199999999999997},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61099999999999999,"15min":0.625},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.61899999999999999}]},{"osd":5,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.49299999999999999,"15min":0.48599999999999999},"min":{"1min":0.35599999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.85899999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.30399999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.878},"last":0.73599999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.50800000000000001,"15min":0.5},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72599999999999998,"5min":0.82499999999999996,"15min":0.88700000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35199999999999998,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.79400000000000004,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.75}]},{"osd":7,"last update":"Sat Jan 27 01:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.57199999999999995,"15min":0.56499999999999995},"min":{"1min":0.33800000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.90200000000000002},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.42199999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.76800000000000002,"5min":0.85399999999999998,"15min":0.92600000000000005},"last":0.76700000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312908,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058932,"statfs":{"total":96636764160,"available":96316346368,"internally_reserved":0,"allocated":3411968,"data_stored":1817037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.56699999999999995,"15min":0.59099999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.27300000000000002},"max":{"1min":0.73599999999999999,"5min":0.82799999999999996,"15min":1.006},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55500000000000005,"15min":0.59099999999999997},"min":{"1min":0.38600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.80000000000000004,"5min":0.81499999999999995,"15min":1.1719999999999999},"last":0.67000000000000004}]},{"osd":1,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59499999999999997,"15min":0.63500000000000001},"min":{"1min":0.42399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.752,"5min":0.89000000000000001,"15min":0.95899999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.63100000000000001,"15min":0.65600000000000003},"min":{"1min":0.434,"5min":0.434,"15min":0.377},"max":{"1min":0.68899999999999995,"5min":0.86899999999999999,"15min":1.212},"last":0.70499999999999996}]},{"osd":2,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54800000000000004,"15min":0.58099999999999996},"min":{"1min":0.35399999999999998,"5min":0.34699999999999998,"15min":0.33700000000000002},"max":{"1min":0.68200000000000005,"5min":0.82099999999999995,"15min":1.631},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.57699999999999996,"15min":0.60399999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.34000000000000002},"max":{"1min":0.65600000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.64100000000000001}]},{"osd":3,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.58999999999999997},"min":{"1min":0.40300000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.63300000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.58399999999999996,"15min":0.61599999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73399999999999999,"5min":0.879,"15min":1.4690000000000001},"last":0.51100000000000001}]},{"osd":4,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.47899999999999998,"15min":0.48299999999999998},"min":{"1min":0.28899999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.60399999999999998,"5min":0.90200000000000002,"15min":0.94399999999999995},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.71299999999999997,"5min":0.92500000000000004,"15min":0.93300000000000005},"last":0.72899999999999998}]},{"osd":6,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.48799999999999999,"15min":0.5},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64100000000000001,"5min":0.96799999999999997,"15min":1.732},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54600000000000004,"15min":0.54500000000000004},"min":{"1min":0.32400000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66800000000000004,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.68400000000000005}]},{"osd":7,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.66200000000000003,"5min":0.95399999999999996,"15min":1.0740000000000001},"last":0.753},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.59499999999999997,"15min":0.60799999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67800000000000005,"5min":0.996,"15min":1.6919999999999999},"last":0.58799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:08:59.662 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-26T23:29:38.508092+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=26, tm_hour=23, tm_min=29, tm_sec=38, tm_wday=4, tm_yday=26, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=1, tm_min=8, tm_sec=53, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T01:08:59.663 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-27T01:09:15.333 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:09:15.336 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:09:19.663 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T01:09:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.805+0000 7f912a1ae700 1 Processor -- start 2024-01-27T01:09:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.805+0000 7f912a1ae700 1 -- start start 2024-01-27T01:09:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.806+0000 7f912a1ae700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91241576b0 0x7f912406f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.806+0000 7f912a1ae700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f91240ac8b0 0x7f912406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.806+0000 7f912a1ae700 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91240b0d50 con 0x7f9124157a80 2024-01-27T01:09:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.806+0000 7f912a1ae700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f912406fe70 con 0x7f91240ac8b0 2024-01-27T01:09:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.806+0000 7f912a1ae700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91240b11e0 con 0x7f91241576b0 2024-01-27T01:09:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.806+0000 7f91237fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91241576b0 0x7f912406f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.806+0000 7f9123fff700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f91240ac8b0 0x7f912406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.806+0000 7f9122ffd700 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f9124157a80 0x7f91240ac1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.46:54562/0 (socket says 172.21.15.46:54562) 2024-01-27T01:09:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.806+0000 7f9122ffd700 1 -- 172.21.15.46:0/3862131727 learned_addr learned my addr 172.21.15.46:0/3862131727 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:09:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.806+0000 7f91237fe700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91241576b0 0x7f912406f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.46:54728/0 (socket says 172.21.15.46:54728) 2024-01-27T01:09:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.807+0000 7f9123fff700 1 -- 172.21.15.46:0/3862131727 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91241576b0 msgr2=0x7f912406f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:09:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.807+0000 7f9123fff700 1 --2- 172.21.15.46:0/3862131727 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91241576b0 0x7f912406f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.807+0000 7f9123fff700 1 -- 172.21.15.46:0/3862131727 >> v1:172.21.15.106:6789/0 conn(0x7f9124157a80 legacy=0x7f91240ac1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.807+0000 7f9123fff700 1 -- 172.21.15.46:0/3862131727 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91240acfe0 con 0x7f91240ac8b0 2024-01-27T01:09:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.807+0000 7f91227fc700 1 -- 172.21.15.46:0/3862131727 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008846772 0 0) 0x7f91240b0d50 con 0x7f9124157a80 2024-01-27T01:09:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.807+0000 7f91237fe700 1 --2- 172.21.15.46:0/3862131727 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91241576b0 0x7f912406f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:09:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.807+0000 7f9123fff700 1 --2- 172.21.15.46:0/3862131727 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91240ac8b0 0x7f912406f840 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f911c009e90 tx=0x7f911c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c51bd30e372180c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:09:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.807+0000 7f91227fc700 1 -- 172.21.15.46:0/3862131727 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c00e070 con 0x7f91240ac8b0 2024-01-27T01:09:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.807+0000 7f91227fc700 1 -- 172.21.15.46:0/3862131727 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911c00bbf0 con 0x7f91240ac8b0 2024-01-27T01:09:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.807+0000 7f91227fc700 1 -- 172.21.15.46:0/3862131727 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c012430 con 0x7f91240ac8b0 2024-01-27T01:09:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.808+0000 7f912a1ae700 1 -- 172.21.15.46:0/3862131727 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91240ac8b0 msgr2=0x7f912406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.808+0000 7f912a1ae700 1 --2- 172.21.15.46:0/3862131727 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91240ac8b0 0x7f912406f840 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f911c009e90 tx=0x7f911c009780 comp rx=0 tx=0).stop 2024-01-27T01:09:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.808+0000 7f912a1ae700 1 -- 172.21.15.46:0/3862131727 shutdown_connections 2024-01-27T01:09:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.808+0000 7f912a1ae700 1 --2- 172.21.15.46:0/3862131727 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91240ac8b0 0x7f912406f840 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.808+0000 7f912a1ae700 1 --2- 172.21.15.46:0/3862131727 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91241576b0 0x7f912406f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.808+0000 7f912a1ae700 1 -- 172.21.15.46:0/3862131727 >> 172.21.15.46:0/3862131727 conn(0x7f9124068ac0 msgr2=0x7f9124068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:09:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.808+0000 7f912a1ae700 1 -- 172.21.15.46:0/3862131727 shutdown_connections 2024-01-27T01:09:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.808+0000 7f912a1ae700 1 -- 172.21.15.46:0/3862131727 wait complete. 2024-01-27T01:09:19.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.809+0000 7f912a1ae700 1 Processor -- start 2024-01-27T01:09:19.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.809+0000 7f912a1ae700 1 -- start start 2024-01-27T01:09:19.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.810+0000 7f912a1ae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91240ac8b0 0x7f9124159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:19.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.810+0000 7f912a1ae700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91241576b0 0x7f91240ae620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:19.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.810+0000 7f912a1ae700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9124157a80 0x7f91240aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:19.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.810+0000 7f912a1ae700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91241686c0 con 0x7f91240ac8b0 2024-01-27T01:09:19.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.810+0000 7f912a1ae700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9124152920 con 0x7f9124157a80 2024-01-27T01:09:19.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.810+0000 7f912a1ae700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9124167d90 con 0x7f91241576b0 2024-01-27T01:09:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.810+0000 7f91237fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91240ac8b0 0x7f9124159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.810+0000 7f91237fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91240ac8b0 0x7f9124159630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46666/0 (socket says 172.21.15.46:46666) 2024-01-27T01:09:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.810+0000 7f91237fe700 1 -- 172.21.15.46:0/1348268765 learned_addr learned my addr 172.21.15.46:0/1348268765 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:09:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.810+0000 7f9123fff700 1 --2- 172.21.15.46:0/1348268765 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9124157a80 0x7f91240aeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.810+0000 7f9122ffd700 1 --2- 172.21.15.46:0/1348268765 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91241576b0 0x7f91240ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.810+0000 7f91237fe700 1 -- 172.21.15.46:0/1348268765 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91241576b0 msgr2=0x7f91240ae620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.810+0000 7f91237fe700 1 --2- 172.21.15.46:0/1348268765 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91241576b0 0x7f91240ae620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.810+0000 7f91237fe700 1 -- 172.21.15.46:0/1348268765 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9124157a80 msgr2=0x7f91240aeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.810+0000 7f91237fe700 1 --2- 172.21.15.46:0/1348268765 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9124157a80 0x7f91240aeb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.810+0000 7f91237fe700 1 -- 172.21.15.46:0/1348268765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9114009420 con 0x7f91240ac8b0 2024-01-27T01:09:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.811+0000 7f91237fe700 1 --2- 172.21.15.46:0/1348268765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91240ac8b0 0x7f9124159630 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f911400ec80 tx=0x7f91140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:09:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.811+0000 7f910b7fe700 1 -- 172.21.15.46:0/1348268765 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911400c920 con 0x7f91240ac8b0 2024-01-27T01:09:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.811+0000 7f910b7fe700 1 -- 172.21.15.46:0/1348268765 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911400ca80 con 0x7f91240ac8b0 2024-01-27T01:09:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.811+0000 7f910b7fe700 1 -- 172.21.15.46:0/1348268765 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911400ce00 con 0x7f91240ac8b0 2024-01-27T01:09:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.811+0000 7f912a1ae700 1 -- 172.21.15.46:0/1348268765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f911c009a20 con 0x7f91240ac8b0 2024-01-27T01:09:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.811+0000 7f912a1ae700 1 -- 172.21.15.46:0/1348268765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9124168350 con 0x7f91240ac8b0 2024-01-27T01:09:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.812+0000 7f912a1ae700 1 -- 172.21.15.46:0/1348268765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912404c710 con 0x7f91240ac8b0 2024-01-27T01:09:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.812+0000 7f910b7fe700 1 -- 172.21.15.46:0/1348268765 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9114013680 con 0x7f91240ac8b0 2024-01-27T01:09:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.813+0000 7f910b7fe700 1 --2- 172.21.15.46:0/1348268765 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9110066f30 0x7f91100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.813+0000 7f910b7fe700 1 -- 172.21.15.46:0/1348268765 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f9114084fd0 con 0x7f91240ac8b0 2024-01-27T01:09:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.813+0000 7f9122ffd700 1 --2- 172.21.15.46:0/1348268765 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9110066f30 0x7f91100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.814+0000 7f9122ffd700 1 --2- 172.21.15.46:0/1348268765 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9110066f30 0x7f91100693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f912415a010 tx=0x7f9118007f50 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:09:19.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:19.817+0000 7f910b7fe700 1 -- 172.21.15.46:0/1348268765 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9114051f10 con 0x7f91240ac8b0 2024-01-27T01:09:20.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.016+0000 7f912a1ae700 1 -- 172.21.15.46:0/1348268765 --> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9124167f00 con 0x7f9110066f30 2024-01-27T01:09:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.019+0000 7f910b7fe700 1 -- 172.21.15.46:0/1348268765 <== mgr.10565 v2:172.21.15.106:6836/62434 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51245 (secure 0 0 0) 0x7f9124167f00 con 0x7f9110066f30 2024-01-27T01:09:20.019 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:09:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.021+0000 7f912a1ae700 1 -- 172.21.15.46:0/1348268765 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9110066f30 msgr2=0x7f91100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.021+0000 7f912a1ae700 1 --2- 172.21.15.46:0/1348268765 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9110066f30 0x7f91100693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f912415a010 tx=0x7f9118007f50 comp rx=0 tx=0).stop 2024-01-27T01:09:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.021+0000 7f912a1ae700 1 -- 172.21.15.46:0/1348268765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91240ac8b0 msgr2=0x7f9124159630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.021+0000 7f912a1ae700 1 --2- 172.21.15.46:0/1348268765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91240ac8b0 0x7f9124159630 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f911400ec80 tx=0x7f91140091a0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.021+0000 7f912a1ae700 1 -- 172.21.15.46:0/1348268765 shutdown_connections 2024-01-27T01:09:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.021+0000 7f912a1ae700 1 --2- 172.21.15.46:0/1348268765 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f9110066f30 0x7f91100693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.021+0000 7f912a1ae700 1 --2- 172.21.15.46:0/1348268765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91240ac8b0 0x7f9124159630 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.021+0000 7f912a1ae700 1 --2- 172.21.15.46:0/1348268765 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9124157a80 0x7f91240aeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.021+0000 7f912a1ae700 1 --2- 172.21.15.46:0/1348268765 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91241576b0 0x7f91240ae620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.021+0000 7f912a1ae700 1 -- 172.21.15.46:0/1348268765 >> 172.21.15.46:0/1348268765 conn(0x7f9124068ac0 msgr2=0x7f91240cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:09:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.021+0000 7f912a1ae700 1 -- 172.21.15.46:0/1348268765 shutdown_connections 2024-01-27T01:09:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.021+0000 7f912a1ae700 1 -- 172.21.15.46:0/1348268765 wait complete. 2024-01-27T01:09:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-27T01:09:20.035 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-27T01:09:18.497947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2815,"num_read_kb":5966,"num_write":740,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8942,"ondisk_log_size":8942,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":2488040,"kb_used_data":27816,"kb_used_omap":0,"kb_used_meta":2460160,"kb_avail":752486680,"statfs":{"total":773094113280,"available":770546360320,"internally_reserved":0,"allocated":28483584,"data_stored":15717032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2519203840},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002621"},"pg_stats":[{"pgid":"1.0","version":"269'11171","reported_seq":12314,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-01-27T01:09:00.128018+0000","last_change":"2024-01-27T01:09:00.127990+0000","last_active":"2024-01-27T01:09:00.128018+0000","last_peered":"2024-01-27T01:09:00.128018+0000","last_clean":"2024-01-27T01:09:00.128018+0000","last_became_active":"2024-01-26T23:29:40.377251+0000","last_became_peered":"2024-01-26T23:29:40.377251+0000","last_unstale":"2024-01-27T01:09:00.128018+0000","last_undegraded":"2024-01-27T01:09:00.128018+0000","last_fullsized":"2024-01-27T01:09:00.128018+0000","mapping_epoch":14,"log_start":"98'2229","ondisk_log_start":"98'2229","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'11171","last_scrub_stamp":"2024-01-27T01:09:00.127894+0000","last_deep_scrub":"269'11171","last_deep_scrub_stamp":"2024-01-27T01:09:00.127894+0000","last_clean_scrub_stamp":"2024-01-27T01:09:00.127894+0000","objects_scrubbed":2,"log_size":8942,"log_dups_size":0,"ondisk_log_size":8942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:48:49.927135+0000","scrub_duration":0.0087971100000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2815,"num_read_kb":5966,"num_write":740,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2815,"num_read_kb":5966,"num_write":740,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8942,"ondisk_log_size":8942,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576032,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313488,"kb_used_data":3912,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058352,"statfs":{"total":96636764160,"available":96315752448,"internally_reserved":0,"allocated":4005888,"data_stored":2407405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56100000000000005,"15min":0.59499999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.76900000000000002,"5min":0.82099999999999995,"15min":0.86299999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.59399999999999997,"15min":0.60999999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33000000000000002},"max":{"1min":0.63200000000000001,"5min":0.879,"15min":0.879},"last":0.72799999999999998}]},{"osd":1,"last update":"Sat Jan 27 01:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.57499999999999996,"15min":0.60099999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76500000000000001,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.59499999999999997,"15min":0.62},"min":{"1min":0.40100000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.69499999999999995,"5min":0.83499999999999996,"15min":0.84099999999999997},"last":0.76300000000000001}]},{"osd":2,"last update":"Sat Jan 27 01:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57799999999999996,"15min":0.59799999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69399999999999995,"5min":0.82099999999999995,"15min":0.89500000000000002},"last":0.752},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.59699999999999998,"15min":0.621},"min":{"1min":0.39900000000000002,"5min":0.372,"15min":0.36599999999999999},"max":{"1min":0.74399999999999999,"5min":0.85799999999999998,"15min":0.88900000000000001},"last":0.68999999999999995}]},{"osd":3,"last update":"Sat Jan 27 01:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57499999999999996,"15min":0.60399999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.36199999999999999},"max":{"1min":0.70099999999999996,"5min":0.79900000000000004,"15min":0.876},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.42999999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.72999999999999998,"5min":0.87,"15min":0.90200000000000002},"last":0.80800000000000005}]},{"osd":4,"last update":"Sat Jan 27 01:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.30299999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.50700000000000001,"15min":0.51400000000000001},"min":{"1min":0.29299999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65900000000000003,"5min":0.77500000000000002,"15min":0.86599999999999999},"last":0.59199999999999997}]},{"osd":5,"last update":"Sat Jan 27 01:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.52300000000000002,"15min":0.52100000000000002},"min":{"1min":0.29799999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79600000000000004,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.623},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55300000000000005,"15min":0.55800000000000005},"min":{"1min":0.33600000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80800000000000005,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Jan 27 01:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.58199999999999996,"15min":0.58399999999999996},"min":{"1min":0.372,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85699999999999998,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56899999999999995,"15min":0.57999999999999996},"min":{"1min":0.36099999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77200000000000002,"5min":0.80700000000000005,"15min":0.84899999999999998},"last":0.71399999999999997}]}]},{"osd":6,"up_from":13,"seq":55834576032,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312908,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058932,"statfs":{"total":96636764160,"available":96316346368,"internally_reserved":0,"allocated":3411968,"data_stored":1817037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.56499999999999995,"15min":0.59199999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.83499999999999996,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.57499999999999996,"15min":0.60799999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73499999999999999,"5min":0.82499999999999996,"15min":0.82899999999999996},"last":0.46600000000000003}]},{"osd":1,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.55800000000000005,"15min":0.59599999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70399999999999996,"5min":0.80300000000000005,"15min":0.84499999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57899999999999996,"15min":0.61099999999999999},"min":{"1min":0.39600000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76400000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.53700000000000003}]},{"osd":2,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.55300000000000005,"15min":0.60099999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78100000000000003,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.57699999999999996,"15min":0.61099999999999999},"min":{"1min":0.379,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.754,"5min":0.80400000000000005,"15min":0.878},"last":0.57999999999999996}]},{"osd":3,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.58699999999999997,"15min":0.61499999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80600000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.62,"15min":0.64900000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.877,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.56000000000000005}]},{"osd":4,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.495,"15min":0.497},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.80300000000000005},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.51600000000000001,"15min":0.53100000000000003},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.59499999999999997}]},{"osd":5,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.50600000000000001,"15min":0.51700000000000002},"min":{"1min":0.309,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70899999999999996,"5min":0.73999999999999999,"15min":0.79300000000000004},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.28899999999999998},"max":{"1min":0.752,"5min":0.87,"15min":0.87},"last":0.57199999999999995}]},{"osd":7,"last update":"Sat Jan 27 01:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56699999999999995,"15min":0.56999999999999995},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.57499999999999996,"15min":0.58499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.79300000000000004,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.70899999999999996}]}]},{"osd":1,"up_from":13,"seq":55834576032,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":304716,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067124,"statfs":{"total":96636764160,"available":96324734976,"internally_reserved":0,"allocated":3411968,"data_stored":1817037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:09:06 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40999999999999998,"15min":0.41599999999999998},"min":{"1min":0.26500000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.434,"5min":0.43099999999999999,"15min":0.44900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.68200000000000005,"5min":0.69099999999999995,"15min":0.76600000000000001},"last":0.54500000000000004}]},{"osd":2,"last update":"Sat Jan 27 01:09:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.40600000000000003,"15min":0.42899999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.73199999999999998,"5min":0.748,"15min":0.95799999999999996},"last":0.441},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.47499999999999998,"15min":0.48499999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.28399999999999997},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.79000000000000004},"last":0.45700000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:09:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.50600000000000001,"15min":0.52500000000000002},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":1.2130000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48499999999999999,"15min":0.50900000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":1.22},"last":0.41699999999999998}]},{"osd":4,"last update":"Sat Jan 27 01:09:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.63500000000000001,"15min":0.66100000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.242},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66000000000000003,"15min":0.68700000000000006},"min":{"1min":0.46500000000000002,"5min":0.41799999999999998,"15min":0.35699999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":1.1519999999999999},"last":0.629}]},{"osd":5,"last update":"Sat Jan 27 01:09:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.66500000000000004,"15min":0.70099999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.044,"5min":1.044,"15min":5.444},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.65800000000000003,"15min":0.69299999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93000000000000005,"5min":0.97799999999999998,"15min":2.6669999999999998},"last":0.73799999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:09:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66700000000000004,"15min":0.71599999999999997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":5.3529999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67800000000000005,"15min":0.73499999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.077,"5min":1.077,"15min":11.223000000000001},"last":0.70399999999999996}]},{"osd":7,"last update":"Sat Jan 27 01:09:06 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.70399999999999996,"15min":0.74299999999999999},"min":{"1min":0.53100000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":2.9660000000000002},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.71399999999999997,"15min":0.75700000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":2.9860000000000002},"last":0.71599999999999997}]}]},{"osd":0,"up_from":13,"seq":55834576031,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309392,"kb_used_data":3912,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062448,"statfs":{"total":96636764160,"available":96319946752,"internally_reserved":0,"allocated":4005888,"data_stored":2407405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.42499999999999999,"15min":0.42199999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.53600000000000003,"5min":1.276,"15min":1.276},"last":0.374},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.46100000000000002},"min":{"1min":0.29999999999999999,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.47099999999999997}]},{"osd":2,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.29199999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.443,"5min":0.46400000000000002,"15min":0.48199999999999998},"min":{"1min":0.32500000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.53400000000000003}]},{"osd":3,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48399999999999999,"15min":0.48499999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65800000000000003,"5min":0.80000000000000004,"15min":0.83699999999999997},"last":0.495},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.498,"15min":0.503},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.65100000000000002,"5min":0.68799999999999994,"15min":0.75700000000000001},"last":0.432}]},{"osd":4,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.42199999999999999},"max":{"1min":0.90200000000000002,"5min":0.91200000000000003,"15min":1.0720000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.64600000000000002,"15min":0.66800000000000004},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.81299999999999994,"5min":1.002,"15min":1.002},"last":0.58299999999999996}]},{"osd":5,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.67600000000000005,"15min":0.69199999999999995},"min":{"1min":0.47699999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.92400000000000004,"5min":1.0209999999999999,"15min":2.165},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.93700000000000006,"5min":1.0700000000000001,"15min":1.1399999999999999},"last":0.64800000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67900000000000005,"15min":0.70399999999999996},"min":{"1min":0.46800000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82799999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70699999999999996,"15min":0.73199999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.425},"last":0.68300000000000005}]},{"osd":7,"last update":"Sat Jan 27 01:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.71899999999999997,"15min":0.745},"min":{"1min":0.48199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.98899999999999999,"5min":1.0169999999999999,"15min":1.1259999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.747,"15min":0.76100000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.96299999999999997,"5min":1.042,"15min":1.1910000000000001},"last":0.70099999999999996}]}]},{"osd":2,"up_from":13,"seq":55834576031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312908,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058932,"statfs":{"total":96636764160,"available":96316346368,"internally_reserved":0,"allocated":3411968,"data_stored":1817037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38300000000000001,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.251},"max":{"1min":0.56000000000000005,"5min":0.60199999999999998,"15min":0.746},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.438},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.57999999999999996,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.45100000000000001}]},{"osd":1,"last update":"Sat Jan 27 01:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42599999999999999,"15min":0.44},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61299999999999999,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.49099999999999999,"15min":0.51100000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.65000000000000002,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.52700000000000002}]},{"osd":3,"last update":"Sat Jan 27 01:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46600000000000003,"15min":0.48699999999999999},"min":{"1min":0.30099999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.61099999999999999,"5min":0.65400000000000003,"15min":0.78100000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46100000000000002,"15min":0.47999999999999998},"min":{"1min":0.34300000000000003,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.61499999999999999}]},{"osd":4,"last update":"Sat Jan 27 01:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.64500000000000002,"15min":0.65900000000000003},"min":{"1min":0.434,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.82599999999999996,"5min":0.95399999999999996,"15min":0.97599999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.64800000000000002,"15min":0.67600000000000005},"min":{"1min":0.495,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.73899999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.65300000000000002}]},{"osd":5,"last update":"Sat Jan 27 01:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.48399999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.75600000000000001,"5min":1.075,"15min":1.075},"last":0.754},{"interface":"front","average":{"1min":0.627,"5min":0.65800000000000003,"15min":0.69099999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39800000000000002},"max":{"1min":0.80300000000000005,"5min":0.89600000000000002,"15min":1.0229999999999999},"last":0.68400000000000005}]},{"osd":6,"last update":"Sat Jan 27 01:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.66600000000000004,"15min":0.69299999999999995},"min":{"1min":0.5,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.877,"5min":0.90200000000000002,"15min":1.0549999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.66700000000000004,"15min":0.69899999999999995},"min":{"1min":0.47699999999999998,"5min":0.44500000000000001,"15min":0.39200000000000002},"max":{"1min":0.879,"5min":0.91200000000000003,"15min":1.0269999999999999},"last":0.73699999999999999}]},{"osd":7,"last update":"Sat Jan 27 01:08:56 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69899999999999995,"15min":0.72899999999999998},"min":{"1min":0.52800000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.88700000000000001,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.71299999999999997,"15min":0.747},"min":{"1min":0.54400000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85099999999999998,"5min":0.94599999999999995,"15min":1.0880000000000001},"last":0.72499999999999998}]}]},{"osd":3,"up_from":13,"seq":55834576030,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":304716,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067124,"statfs":{"total":96636764160,"available":96324734976,"internally_reserved":0,"allocated":3411968,"data_stored":1817037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.436,"15min":0.41899999999999998},"min":{"1min":0.27400000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.45000000000000001,"5min":4.2949999999999999,"15min":4.2949999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.44900000000000001,"15min":0.45400000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.28299999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.45900000000000002}]},{"osd":1,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.60999999999999999,"15min":0.52900000000000003},"min":{"1min":0.314,"5min":0.30099999999999999,"15min":0.28899999999999998},"max":{"1min":0.754,"5min":15.391,"15min":15.391},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.57099999999999995,"5min":3.6619999999999999,"15min":3.6619999999999999},"last":0.53500000000000003}]},{"osd":2,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52800000000000002,"15min":0.51100000000000001},"min":{"1min":0.36699999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64800000000000002,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.53600000000000003,"15min":0.50900000000000001},"min":{"1min":0.32900000000000001,"5min":0.308,"15min":0.29799999999999999},"max":{"1min":0.82299999999999995,"5min":4.6260000000000003,"15min":4.6260000000000003},"last":0.59199999999999997}]},{"osd":4,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.71799999999999997,"15min":0.70999999999999996},"min":{"1min":0.46600000000000003,"5min":0.46000000000000002,"15min":0.439},"max":{"1min":0.76400000000000001,"5min":6.0010000000000003,"15min":6.0010000000000003},"last":0.754},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.70199999999999996,"15min":0.69599999999999995},"min":{"1min":0.442,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":0.76900000000000002,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.68200000000000005}]},{"osd":5,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.71599999999999997,"15min":0.70299999999999996},"min":{"1min":0.47099999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76100000000000001,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.628,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76600000000000001,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.77400000000000002}]},{"osd":6,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.72799999999999998,"15min":0.71399999999999997},"min":{"1min":0.54100000000000004,"5min":0.49199999999999999,"15min":0.47499999999999998},"max":{"1min":0.83499999999999996,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.71399999999999997,"15min":0.72699999999999998},"min":{"1min":0.46300000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81200000000000006,"5min":3.8849999999999998,"15min":3.8849999999999998},"last":0.82699999999999996}]},{"osd":7,"last update":"Sat Jan 27 01:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.73899999999999999,"15min":0.72699999999999998},"min":{"1min":0.52500000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1779999999999999,"5min":4.569,"15min":4.569},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.77100000000000002,"15min":0.77200000000000002},"min":{"1min":0.56499999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.80700000000000005,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.79500000000000004}]}]},{"osd":4,"up_from":13,"seq":55834576032,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":317004,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94054836,"statfs":{"total":96636764160,"available":96312152064,"internally_reserved":0,"allocated":3411968,"data_stored":1817037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.57599999999999996,"15min":0.60199999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82499999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.58199999999999996,"15min":0.60399999999999998},"min":{"1min":0.376,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.70599999999999996,"5min":0.83799999999999997,"15min":0.85699999999999998},"last":0.50700000000000001}]},{"osd":1,"last update":"Sat Jan 27 01:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.58799999999999997,"15min":0.61399999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76900000000000002,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.60099999999999998,"15min":0.61499999999999999},"min":{"1min":0.35699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.65500000000000003}]},{"osd":2,"last update":"Sat Jan 27 01:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55900000000000005,"15min":0.58499999999999996},"min":{"1min":0.37,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73299999999999998,"5min":0.85599999999999998,"15min":0.872},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.55300000000000005}]},{"osd":3,"last update":"Sat Jan 27 01:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58599999999999997,"15min":0.61099999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.69899999999999995,"5min":0.83299999999999996,"15min":0.84199999999999997},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.59699999999999998,"15min":0.61799999999999999},"min":{"1min":0.39900000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79200000000000004,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.54200000000000004}]},{"osd":5,"last update":"Sat Jan 27 01:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.47699999999999998,"15min":0.48199999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69099999999999995,"5min":0.83899999999999997,"15min":0.85899999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.504},"min":{"1min":0.317,"5min":0.317,"15min":0.30399999999999999},"max":{"1min":0.73599999999999999,"5min":0.82499999999999996,"15min":0.878},"last":0.39000000000000001}]},{"osd":6,"last update":"Sat Jan 27 01:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.503,"15min":0.497},"min":{"1min":0.30499999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.76900000000000002,"5min":0.82499999999999996,"15min":0.88700000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54500000000000004,"15min":0.54400000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34000000000000002},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.40899999999999997}]},{"osd":7,"last update":"Sat Jan 27 01:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.56200000000000006},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.77900000000000003,"5min":0.85899999999999999,"15min":0.90200000000000002},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.46500000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576032,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312908,"kb_used_data":3332,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058932,"statfs":{"total":96636764160,"available":96316346368,"internally_reserved":0,"allocated":3411968,"data_stored":1817037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.56699999999999995,"15min":0.59099999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.27300000000000002},"max":{"1min":0.73599999999999999,"5min":0.82799999999999996,"15min":1.006},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55500000000000005,"15min":0.59099999999999997},"min":{"1min":0.38600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.80000000000000004,"5min":0.81499999999999995,"15min":1.1719999999999999},"last":0.61399999999999999}]},{"osd":1,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59499999999999997,"15min":0.63500000000000001},"min":{"1min":0.42399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.752,"5min":0.89000000000000001,"15min":0.95899999999999996},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.63100000000000001,"15min":0.65600000000000003},"min":{"1min":0.434,"5min":0.434,"15min":0.377},"max":{"1min":0.68899999999999995,"5min":0.86899999999999999,"15min":1.212},"last":0.53600000000000003}]},{"osd":2,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54800000000000004,"15min":0.58099999999999996},"min":{"1min":0.35399999999999998,"5min":0.34699999999999998,"15min":0.33700000000000002},"max":{"1min":0.68200000000000005,"5min":0.82099999999999995,"15min":1.631},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.57699999999999996,"15min":0.60399999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.34000000000000002},"max":{"1min":0.65600000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.56799999999999995}]},{"osd":3,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.58999999999999997},"min":{"1min":0.40300000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.63300000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.58399999999999996,"15min":0.61599999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73399999999999999,"5min":0.879,"15min":1.4690000000000001},"last":0.50900000000000001}]},{"osd":4,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.47899999999999998,"15min":0.48299999999999998},"min":{"1min":0.28899999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.60399999999999998,"5min":0.90200000000000002,"15min":0.94399999999999995},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.71299999999999997,"5min":0.92500000000000004,"15min":0.93300000000000005},"last":0.36399999999999999}]},{"osd":6,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.48799999999999999,"15min":0.5},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64100000000000001,"5min":0.96799999999999997,"15min":1.732},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54600000000000004,"15min":0.54500000000000004},"min":{"1min":0.32400000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66800000000000004,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.497}]},{"osd":7,"last update":"Sat Jan 27 01:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.66200000000000003,"5min":0.95399999999999996,"15min":1.0740000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.59499999999999997,"15min":0.60799999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67800000000000005,"5min":0.996,"15min":1.6919999999999999},"last":0.59399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T01:09:20.037 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-27T01:09:20.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.168+0000 7f3aecebc700 1 Processor -- start 2024-01-27T01:09:20.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.168+0000 7f3aecebc700 1 -- start start 2024-01-27T01:09:20.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.169+0000 7f3aecebc700 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3ae80ca950 0x7f3ae80cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:20.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.169+0000 7f3aecebc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae80cb1c0 0x7f3ae80acca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:20.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.169+0000 7f3aecebc700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae80abf70 con 0x7f3ae80cad20 2024-01-27T01:09:20.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.169+0000 7f3aecebc700 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae815b280 con 0x7f3ae80ca950 2024-01-27T01:09:20.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.169+0000 7f3aecebc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae80cc030 con 0x7f3ae80cb1c0 2024-01-27T01:09:20.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.170+0000 7f3ae5d9b700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3ae80cad20 0x7f3ae80b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53178/0 (socket says 172.21.15.46:53178) 2024-01-27T01:09:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.170+0000 7f3ae5d9b700 1 -- 172.21.15.46:0/1777524707 learned_addr learned my addr 172.21.15.46:0/1777524707 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:09:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.170+0000 7f3ae659c700 1 --2- 172.21.15.46:0/1777524707 >> v2:172.21.15.106:3300/0 conn(0x7f3ae80ca950 0x7f3ae80cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.170+0000 7f3ae6d9d700 1 --2- 172.21.15.46:0/1777524707 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae80cb1c0 0x7f3ae80acca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.170+0000 7f3ae559a700 1 -- 172.21.15.46:0/1777524707 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639974708 0 0) 0x7f3ae80abf70 con 0x7f3ae80cad20 2024-01-27T01:09:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.170+0000 7f3ae559a700 1 -- 172.21.15.46:0/1777524707 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ad4003570 con 0x7f3ae80cad20 2024-01-27T01:09:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.170+0000 7f3ae6d9d700 1 -- 172.21.15.46:0/1777524707 >> v2:172.21.15.106:3300/0 conn(0x7f3ae80ca950 msgr2=0x7f3ae80cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:09:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.170+0000 7f3ae6d9d700 1 --2- 172.21.15.46:0/1777524707 >> v2:172.21.15.106:3300/0 conn(0x7f3ae80ca950 0x7f3ae80cb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.170+0000 7f3ae6d9d700 1 -- 172.21.15.46:0/1777524707 >> v1:172.21.15.46:6789/0 conn(0x7f3ae80cad20 legacy=0x7f3ae80b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.170+0000 7f3ae6d9d700 1 -- 172.21.15.46:0/1777524707 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae80ac1c0 con 0x7f3ae80cb1c0 2024-01-27T01:09:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.170+0000 7f3ae659c700 1 --2- 172.21.15.46:0/1777524707 >> v2:172.21.15.106:3300/0 conn(0x7f3ae80ca950 0x7f3ae80cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T01:09:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.171+0000 7f3ae6d9d700 1 --2- 172.21.15.46:0/1777524707 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae80cb1c0 0x7f3ae80acca0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0009e60 tx=0x7f3ae0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66a3cb73fc76dd0f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:09:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.171+0000 7f3ae559a700 1 -- 172.21.15.46:0/1777524707 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae000e070 con 0x7f3ae80cb1c0 2024-01-27T01:09:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.171+0000 7f3ae559a700 1 -- 172.21.15.46:0/1777524707 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae000bab0 con 0x7f3ae80cb1c0 2024-01-27T01:09:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.171+0000 7f3ae559a700 1 -- 172.21.15.46:0/1777524707 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae000be30 con 0x7f3ae80cb1c0 2024-01-27T01:09:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.171+0000 7f3aecebc700 1 -- 172.21.15.46:0/1777524707 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae80cb1c0 msgr2=0x7f3ae80acca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.171+0000 7f3aecebc700 1 --2- 172.21.15.46:0/1777524707 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae80cb1c0 0x7f3ae80acca0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0009e60 tx=0x7f3ae0009780 comp rx=0 tx=0).stop 2024-01-27T01:09:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.171+0000 7f3aecebc700 1 -- 172.21.15.46:0/1777524707 shutdown_connections 2024-01-27T01:09:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.171+0000 7f3aecebc700 1 --2- 172.21.15.46:0/1777524707 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae80cb1c0 0x7f3ae80acca0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.171+0000 7f3aecebc700 1 --2- 172.21.15.46:0/1777524707 >> v2:172.21.15.106:3300/0 conn(0x7f3ae80ca950 0x7f3ae80cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.172+0000 7f3aecebc700 1 -- 172.21.15.46:0/1777524707 >> 172.21.15.46:0/1777524707 conn(0x7f3ae80cc3a0 msgr2=0x7f3ae80cc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:09:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.172+0000 7f3aecebc700 1 -- 172.21.15.46:0/1777524707 shutdown_connections 2024-01-27T01:09:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.172+0000 7f3aecebc700 1 -- 172.21.15.46:0/1777524707 wait complete. 2024-01-27T01:09:20.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.173+0000 7f3aecebc700 1 Processor -- start 2024-01-27T01:09:20.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.173+0000 7f3aecebc700 1 -- start start 2024-01-27T01:09:20.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.173+0000 7f3aecebc700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae80ca950 0x7f3ae80983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:20.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.173+0000 7f3aecebc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ae80cad20 0x7f3ae815c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:20.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3aecebc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ae8156de0 0x7f3ae815d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:20.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3aecebc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae8098ad0 con 0x7f3ae80cad20 2024-01-27T01:09:20.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3aecebc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae8098db0 con 0x7f3ae8156de0 2024-01-27T01:09:20.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3aecebc700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae8099090 con 0x7f3ae80ca950 2024-01-27T01:09:20.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3ae5d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ae80cad20 0x7f3ae815c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:20.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3ae5d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ae80cad20 0x7f3ae815c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46674/0 (socket says 172.21.15.46:46674) 2024-01-27T01:09:20.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3ae5d9b700 1 -- 172.21.15.46:0/401067731 learned_addr learned my addr 172.21.15.46:0/401067731 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:09:20.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3ae659c700 1 --2- 172.21.15.46:0/401067731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae80ca950 0x7f3ae80983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:20.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3ae6d9d700 1 --2- 172.21.15.46:0/401067731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ae8156de0 0x7f3ae815d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:20.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3ae5d9b700 1 -- 172.21.15.46:0/401067731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae80ca950 msgr2=0x7f3ae80983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:20.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3ae5d9b700 1 --2- 172.21.15.46:0/401067731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae80ca950 0x7f3ae80983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3ae5d9b700 1 -- 172.21.15.46:0/401067731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ae8156de0 msgr2=0x7f3ae815d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:20.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3ae5d9b700 1 --2- 172.21.15.46:0/401067731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ae8156de0 0x7f3ae815d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3ae5d9b700 1 -- 172.21.15.46:0/401067731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad8009420 con 0x7f3ae80cad20 2024-01-27T01:09:20.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3ae5d9b700 1 --2- 172.21.15.46:0/401067731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ae80cad20 0x7f3ae815c450 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f3ad00011e0 tx=0x7f3ad0007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:09:20.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3ade7fc700 1 -- 172.21.15.46:0/401067731 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad0005b30 con 0x7f3ae80cad20 2024-01-27T01:09:20.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3ade7fc700 1 -- 172.21.15.46:0/401067731 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad0005c90 con 0x7f3ae80cad20 2024-01-27T01:09:20.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3aecebc700 1 -- 172.21.15.46:0/401067731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae0009a20 con 0x7f3ae80cad20 2024-01-27T01:09:20.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3ade7fc700 1 -- 172.21.15.46:0/401067731 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad0007300 con 0x7f3ae80cad20 2024-01-27T01:09:20.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.174+0000 7f3aecebc700 1 -- 172.21.15.46:0/401067731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae80a8550 con 0x7f3ae80cad20 2024-01-27T01:09:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.176+0000 7f3aecebc700 1 -- 172.21.15.46:0/401067731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae80a87d0 con 0x7f3ae80cad20 2024-01-27T01:09:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.176+0000 7f3ade7fc700 1 -- 172.21.15.46:0/401067731 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3ad0007670 con 0x7f3ae80cad20 2024-01-27T01:09:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.177+0000 7f3ade7fc700 1 --2- 172.21.15.46:0/401067731 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3ac8066f30 0x7f3ac80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.177+0000 7f3ade7fc700 1 -- 172.21.15.46:0/401067731 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f3ad0083da0 con 0x7f3ae80cad20 2024-01-27T01:09:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.177+0000 7f3ae659c700 1 --2- 172.21.15.46:0/401067731 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3ac8066f30 0x7f3ac80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.177+0000 7f3ae659c700 1 --2- 172.21.15.46:0/401067731 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3ac8066f30 0x7f3ac80693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009f40 tx=0x7f3ad80091a0 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:09:20.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.181+0000 7f3ade7fc700 1 -- 172.21.15.46:0/401067731 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3ad0050c40 con 0x7f3ae80cad20 2024-01-27T01:09:20.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.387+0000 7f3aecebc700 1 -- 172.21.15.46:0/401067731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f3ae806c6e0 con 0x7f3ae80cad20 2024-01-27T01:09:20.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.390+0000 7f3ade7fc700 1 -- 172.21.15.46:0/401067731 <== mon.0 v2:172.21.15.46:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f3ad0050c40 con 0x7f3ae80cad20 2024-01-27T01:09:20.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.391+0000 7f3ade7fc700 1 -- 172.21.15.46:0/401067731 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v94)=0 v94) v1 ==== 123+0+0 (secure 0 0 0) 0x7f3ad0017020 con 0x7f3ae80cad20 2024-01-27T01:09:20.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.393+0000 7f3aecebc700 1 -- 172.21.15.46:0/401067731 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3ac8066f30 msgr2=0x7f3ac80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:20.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.393+0000 7f3aecebc700 1 --2- 172.21.15.46:0/401067731 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3ac8066f30 0x7f3ac80693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009f40 tx=0x7f3ad80091a0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.393+0000 7f3aecebc700 1 -- 172.21.15.46:0/401067731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ae80cad20 msgr2=0x7f3ae815c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:20.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.393+0000 7f3aecebc700 1 --2- 172.21.15.46:0/401067731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ae80cad20 0x7f3ae815c450 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f3ad00011e0 tx=0x7f3ad0007df0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.393+0000 7f3aecebc700 1 -- 172.21.15.46:0/401067731 shutdown_connections 2024-01-27T01:09:20.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.393+0000 7f3aecebc700 1 --2- 172.21.15.46:0/401067731 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f3ac8066f30 0x7f3ac80693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.393+0000 7f3aecebc700 1 --2- 172.21.15.46:0/401067731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ae8156de0 0x7f3ae815d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.393+0000 7f3aecebc700 1 --2- 172.21.15.46:0/401067731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae80ca950 0x7f3ae80983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.393+0000 7f3aecebc700 1 --2- 172.21.15.46:0/401067731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ae80cad20 0x7f3ae815c450 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.393+0000 7f3aecebc700 1 -- 172.21.15.46:0/401067731 >> 172.21.15.46:0/401067731 conn(0x7f3ae80cc3a0 msgr2=0x7f3ae80b1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:09:20.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.394+0000 7f3aecebc700 1 -- 172.21.15.46:0/401067731 shutdown_connections 2024-01-27T01:09:20.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.394+0000 7f3aecebc700 1 -- 172.21.15.46:0/401067731 wait complete. 2024-01-27T01:09:20.407 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-27T01:09:20.407 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-27T01:09:20.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.539+0000 7f873afe7700 1 Processor -- start 2024-01-27T01:09:20.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.540+0000 7f873afe7700 1 -- start start 2024-01-27T01:09:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.541+0000 7f873afe7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8734159c80 0x7f87340c9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.541+0000 7f873afe7700 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f873406c920 0x7f87340c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.541+0000 7f873afe7700 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87340c9f90 con 0x7f87341598b0 2024-01-27T01:09:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.541+0000 7f873afe7700 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f873406b1e0 con 0x7f873406c920 2024-01-27T01:09:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.541+0000 7f873afe7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873406aa30 con 0x7f8734159c80 2024-01-27T01:09:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.541+0000 7f8738d83700 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f87341598b0 0x7f873406a320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47520/0 (socket says 172.21.15.46:47520) 2024-01-27T01:09:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.541+0000 7f8738d83700 1 -- 172.21.15.46:0/1418862757 learned_addr learned my addr 172.21.15.46:0/1418862757 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-01-27T01:09:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.541+0000 7f8739584700 1 --2- 172.21.15.46:0/1418862757 >> v2:172.21.15.46:3300/0 conn(0x7f873406c920 0x7f87340c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.541+0000 7f87337fe700 1 -- 172.21.15.46:0/1418862757 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162737485 0 0) 0x7f8724001110 con 0x7f87341598b0 2024-01-27T01:09:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.541+0000 7f87337fe700 1 -- 172.21.15.46:0/1418862757 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8720003570 con 0x7f87341598b0 2024-01-27T01:09:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.541+0000 7f8733fff700 1 --2- 172.21.15.46:0/1418862757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8734159c80 0x7f87340c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.541+0000 7f8739584700 1 -- 172.21.15.46:0/1418862757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8734159c80 msgr2=0x7f87340c9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T01:09:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.541+0000 7f8739584700 1 --2- 172.21.15.46:0/1418862757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8734159c80 0x7f87340c9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.541+0000 7f8739584700 1 -- 172.21.15.46:0/1418862757 >> v1:172.21.15.46:6789/0 conn(0x7f87341598b0 legacy=0x7f873406a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.542+0000 7f8739584700 1 -- 172.21.15.46:0/1418862757 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87340affe0 con 0x7f873406c920 2024-01-27T01:09:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.542+0000 7f8739584700 1 --2- 172.21.15.46:0/1418862757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f873406c920 0x7f87340c9a50 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f872c009e90 tx=0x7f872c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc7b48a791dd027 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:09:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.542+0000 7f87337fe700 1 -- 172.21.15.46:0/1418862757 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c00e070 con 0x7f873406c920 2024-01-27T01:09:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.542+0000 7f87337fe700 1 -- 172.21.15.46:0/1418862757 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f872c00bbf0 con 0x7f873406c920 2024-01-27T01:09:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.542+0000 7f87337fe700 1 -- 172.21.15.46:0/1418862757 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c012430 con 0x7f873406c920 2024-01-27T01:09:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.542+0000 7f873afe7700 1 -- 172.21.15.46:0/1418862757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f873406c920 msgr2=0x7f87340c9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.542+0000 7f873afe7700 1 --2- 172.21.15.46:0/1418862757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f873406c920 0x7f87340c9a50 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f872c009e90 tx=0x7f872c009780 comp rx=0 tx=0).stop 2024-01-27T01:09:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.542+0000 7f873afe7700 1 -- 172.21.15.46:0/1418862757 shutdown_connections 2024-01-27T01:09:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.542+0000 7f873afe7700 1 --2- 172.21.15.46:0/1418862757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f873406c920 0x7f87340c9a50 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.542+0000 7f873afe7700 1 --2- 172.21.15.46:0/1418862757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8734159c80 0x7f87340c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.542+0000 7f873afe7700 1 -- 172.21.15.46:0/1418862757 >> 172.21.15.46:0/1418862757 conn(0x7f873406bda0 msgr2=0x7f873406c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:09:20.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.543+0000 7f873afe7700 1 -- 172.21.15.46:0/1418862757 shutdown_connections 2024-01-27T01:09:20.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.543+0000 7f873afe7700 1 -- 172.21.15.46:0/1418862757 wait complete. 2024-01-27T01:09:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.544+0000 7f873afe7700 1 Processor -- start 2024-01-27T01:09:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.544+0000 7f873afe7700 1 -- start start 2024-01-27T01:09:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.545+0000 7f873afe7700 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f873406c920 0x7f87341583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.545+0000 7f873afe7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87341598b0 0x7f873415b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.545+0000 7f873afe7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8734159c80 0x7f873415e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.545+0000 7f873afe7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87340c6dc0 con 0x7f87341598b0 2024-01-27T01:09:20.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.545+0000 7f873afe7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873415a050 con 0x7f8734159c80 2024-01-27T01:09:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.545+0000 7f873afe7700 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873415aeb0 con 0x7f873406c920 2024-01-27T01:09:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.545+0000 7f8733fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87341598b0 0x7f873415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.545+0000 7f8733fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87341598b0 0x7f873415b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45794/0 (socket says 172.21.15.46:45794) 2024-01-27T01:09:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.545+0000 7f8733fff700 1 -- 172.21.15.46:0/209304312 learned_addr learned my addr 172.21.15.46:0/209304312 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-01-27T01:09:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.545+0000 7f8738d83700 1 --2- 172.21.15.46:0/209304312 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f873406c920 0x7f87341583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.545+0000 7f8739584700 1 --2- 172.21.15.46:0/209304312 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8734159c80 0x7f873415e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.545+0000 7f8733fff700 1 -- 172.21.15.46:0/209304312 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f873406c920 msgr2=0x7f87341583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.545+0000 7f8733fff700 1 --2- 172.21.15.46:0/209304312 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f873406c920 0x7f87341583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.545+0000 7f8733fff700 1 -- 172.21.15.46:0/209304312 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8734159c80 msgr2=0x7f873415e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.545+0000 7f8733fff700 1 --2- 172.21.15.46:0/209304312 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8734159c80 0x7f873415e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.545+0000 7f8733fff700 1 -- 172.21.15.46:0/209304312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872c009a20 con 0x7f87341598b0 2024-01-27T01:09:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.546+0000 7f8733fff700 1 --2- 172.21.15.46:0/209304312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87341598b0 0x7f873415b340 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f871c00d8c0 tx=0x7f871c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:09:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.546+0000 7f8730ff9700 1 -- 172.21.15.46:0/209304312 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c0125f0 con 0x7f87341598b0 2024-01-27T01:09:20.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.546+0000 7f8730ff9700 1 -- 172.21.15.46:0/209304312 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f871c012750 con 0x7f87341598b0 2024-01-27T01:09:20.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.546+0000 7f873afe7700 1 -- 172.21.15.46:0/209304312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f873415a2a0 con 0x7f87341598b0 2024-01-27T01:09:20.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.546+0000 7f8730ff9700 1 -- 172.21.15.46:0/209304312 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c012b10 con 0x7f87341598b0 2024-01-27T01:09:20.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.546+0000 7f873afe7700 1 -- 172.21.15.46:0/209304312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f873415a8e0 con 0x7f87341598b0 2024-01-27T01:09:20.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.547+0000 7f873afe7700 1 -- 172.21.15.46:0/209304312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f873415b040 con 0x7f87341598b0 2024-01-27T01:09:20.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.547+0000 7f8730ff9700 1 -- 172.21.15.46:0/209304312 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f871c01e460 con 0x7f87341598b0 2024-01-27T01:09:20.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.548+0000 7f8730ff9700 1 --2- 172.21.15.46:0/209304312 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8714066f30 0x7f87140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T01:09:20.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.548+0000 7f8730ff9700 1 -- 172.21.15.46:0/209304312 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6525+0+0 (secure 0 0 0) 0x7f871c084dd0 con 0x7f87341598b0 2024-01-27T01:09:20.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.548+0000 7f8738d83700 1 --2- 172.21.15.46:0/209304312 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8714066f30 0x7f87140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T01:09:20.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.549+0000 7f8738d83700 1 --2- 172.21.15.46:0/209304312 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8714066f30 0x7f87140693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8724001270 tx=0x7f8724007bf0 comp rx=0 tx=0).ready entity=mgr.10565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T01:09:20.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.552+0000 7f8730ff9700 1 -- 172.21.15.46:0/209304312 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f871c055820 con 0x7f87341598b0 2024-01-27T01:09:20.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.806+0000 7f873afe7700 1 -- 172.21.15.46:0/209304312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f873415acf0 con 0x7f87341598b0 2024-01-27T01:09:20.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.807+0000 7f8730ff9700 1 -- 172.21.15.46:0/209304312 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+3541 (secure 0 0 0) 0x7f873415acf0 con 0x7f87341598b0 2024-01-27T01:09:20.807 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-27T01:09:20.807 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":151,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3826743114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3826743114},{"type":"v1","addr":"172.21.15.46:6835","nonce":3826743114}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":144},{"gid":11947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/2514069949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2514069949},{"type":"v1","addr":"172.21.15.106:6833","nonce":2514069949}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":144},{"gid":11950,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/251184858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":251184858},{"type":"v1","addr":"172.21.15.106:6835","nonce":251184858}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":143},{"gid":12729,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3100207851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3100207851},{"type":"v1","addr":"172.21.15.46:6833","nonce":3100207851}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":151}],"filesystems":[]} 2024-01-27T01:09:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.809+0000 7f873afe7700 1 -- 172.21.15.46:0/209304312 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8714066f30 msgr2=0x7f87140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.809+0000 7f873afe7700 1 --2- 172.21.15.46:0/209304312 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8714066f30 0x7f87140693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8724001270 tx=0x7f8724007bf0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.809+0000 7f873afe7700 1 -- 172.21.15.46:0/209304312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87341598b0 msgr2=0x7f873415b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T01:09:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.809+0000 7f873afe7700 1 --2- 172.21.15.46:0/209304312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87341598b0 0x7f873415b340 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f871c00d8c0 tx=0x7f871c008040 comp rx=0 tx=0).stop 2024-01-27T01:09:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.809+0000 7f873afe7700 1 -- 172.21.15.46:0/209304312 shutdown_connections 2024-01-27T01:09:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.809+0000 7f873afe7700 1 --2- 172.21.15.46:0/209304312 >> [v2:172.21.15.106:6836/62434,v1:172.21.15.106:6837/62434] conn(0x7f8714066f30 0x7f87140693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.809+0000 7f873afe7700 1 --2- 172.21.15.46:0/209304312 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8734159c80 0x7f873415e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.809+0000 7f873afe7700 1 --2- 172.21.15.46:0/209304312 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f873406c920 0x7f87341583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.809+0000 7f873afe7700 1 --2- 172.21.15.46:0/209304312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87341598b0 0x7f873415b340 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T01:09:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.809+0000 7f873afe7700 1 -- 172.21.15.46:0/209304312 >> 172.21.15.46:0/209304312 conn(0x7f873406bda0 msgr2=0x7f87340b0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T01:09:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.809+0000 7f873afe7700 1 -- 172.21.15.46:0/209304312 shutdown_connections 2024-01-27T01:09:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-27T01:09:20.809+0000 7f873afe7700 1 -- 172.21.15.46:0/209304312 wait complete. 2024-01-27T01:09:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 151 2024-01-27T01:09:20.822 DEBUG:tasks.cephfs.filesystem:already dead... 2024-01-27T01:09:20.822 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-27T01:09:20.823 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-27T01:09:20.823 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:09:20.832 INFO:tasks.ceph.mds.a:Stopped 2024-01-27T01:09:20.832 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-27T01:09:20.832 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:09:20.881 INFO:tasks.ceph.mds.c:Stopped 2024-01-27T01:09:20.882 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-27T01:09:20.882 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:09:20.892 INFO:tasks.ceph.mds.b:Stopped 2024-01-27T01:09:20.892 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-27T01:09:20.892 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:09:20.943 INFO:tasks.ceph.mds.d:Stopped 2024-01-27T01:09:20.943 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-27T01:09:20.943 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-27T01:09:20.944 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:09:20.986 INFO:tasks.ceph.osd.0:Stopped 2024-01-27T01:09:20.987 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-27T01:09:20.987 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:09:21.025 INFO:tasks.ceph.osd.1:Stopped 2024-01-27T01:09:21.025 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-27T01:09:21.025 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:09:21.099 INFO:tasks.ceph.osd.2:Stopped 2024-01-27T01:09:21.099 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-01-27T01:09:21.099 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:09:21.144 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-01-27T01:09:21.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-01-27T01:09:21.171 INFO:tasks.ceph.osd.3:Stopped 2024-01-27T01:09:21.171 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-01-27T01:09:21.171 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:09:21.213 INFO:tasks.ceph.osd.4:Stopped 2024-01-27T01:09:21.213 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-01-27T01:09:21.213 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:09:21.259 INFO:tasks.ceph.osd.5:Stopped 2024-01-27T01:09:21.259 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-01-27T01:09:21.259 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:09:21.305 INFO:tasks.ceph.osd.6:Stopped 2024-01-27T01:09:21.305 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-01-27T01:09:21.305 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:09:21.348 INFO:tasks.ceph.osd.7:Stopped 2024-01-27T01:09:21.348 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-27T01:09:21.349 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-01-27T01:09:21.349 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:09:21.382 INFO:tasks.ceph.mgr.y:Stopped 2024-01-27T01:09:21.382 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-27T01:09:21.382 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:09:21.413 INFO:tasks.ceph.mgr.x:Stopped 2024-01-27T01:09:21.413 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-27T01:09:21.413 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-27T01:09:21.413 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:09:21.441 INFO:tasks.ceph.mon.a:Stopped 2024-01-27T01:09:21.441 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-01-27T01:09:21.441 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:09:21.466 INFO:tasks.ceph.mon.b:Stopped 2024-01-27T01:09:21.466 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-01-27T01:09:21.466 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T01:09:21.520 INFO:tasks.ceph.mon.c:Stopped 2024-01-27T01:09:21.520 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-27T01:09:21.520 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-27T01:09:21.563 INFO:teuthology.orchestra.run.smithi046.stdout:1706312662.6613069 mon.a (mon.0) 512 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-01-27T01:09:21.563 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-01-27T01:09:21.563 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-27T01:09:21.644 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-27T01:09:21.725 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-27T01:09:21.805 INFO:teuthology.orchestra.run.smithi046.stdout:1706312662.6613069 mon.a (mon.0) 512 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-01-27T01:09:21.805 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T01:09:21.806 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-27T01:09:22.049 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T01:09:22.049 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-27T01:09:22.138 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T01:09:22.138 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-27T01:09:22.215 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T01:09:22.216 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-01-27T01:09:22.306 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T01:09:22.307 DEBUG:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-01-27T01:09:22.578 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T01:09:22.578 DEBUG:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-01-27T01:09:22.644 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T01:09:22.645 DEBUG:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-01-27T01:09:22.713 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T01:09:22.713 DEBUG:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-01-27T01:09:22.779 INFO:tasks.ceph:Archiving mon data... 2024-01-27T01:09:22.780 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi/7533360/data/mon.a.tgz 2024-01-27T01:09:22.780 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-01-27T01:09:22.803 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.IWOCEPORYN 2024-01-27T01:09:22.803 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.IWOCEPORYN 2024-01-27T01:09:23.467 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.IWOCEPORYN 2024-01-27T01:09:23.560 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.IWOCEPORYN is 4MB 2024-01-27T01:09:23.715 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.IWOCEPORYN 2024-01-27T01:09:23.737 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi/7533360/data/mon.b.tgz 2024-01-27T01:09:23.737 DEBUG:teuthology.orchestra.run.smithi106:> mktemp 2024-01-27T01:09:23.758 INFO:teuthology.orchestra.run.smithi106.stdout:/tmp/tmp.0sWyxgtBHv 2024-01-27T01:09:23.758 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.0sWyxgtBHv 2024-01-27T01:09:24.392 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0666 /tmp/tmp.0sWyxgtBHv 2024-01-27T01:09:24.479 DEBUG:teuthology.orchestra.remote:smithi106:/tmp/tmp.0sWyxgtBHv is 4MB 2024-01-27T01:09:24.631 DEBUG:teuthology.orchestra.run.smithi106:> rm -fr /tmp/tmp.0sWyxgtBHv 2024-01-27T01:09:24.648 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi/7533360/data/mon.c.tgz 2024-01-27T01:09:24.649 DEBUG:teuthology.orchestra.run.smithi106:> mktemp 2024-01-27T01:09:24.705 INFO:teuthology.orchestra.run.smithi106.stdout:/tmp/tmp.N94DUGy3xr 2024-01-27T01:09:24.705 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.N94DUGy3xr 2024-01-27T01:09:25.324 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0666 /tmp/tmp.N94DUGy3xr 2024-01-27T01:09:25.410 DEBUG:teuthology.orchestra.remote:smithi106:/tmp/tmp.N94DUGy3xr is 4MB 2024-01-27T01:09:25.563 DEBUG:teuthology.orchestra.run.smithi106:> rm -fr /tmp/tmp.N94DUGy3xr 2024-01-27T01:09:25.580 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-27T01:09:25.580 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-27T01:09:25.583 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-27T01:09:25.645 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-01-27T01:09:25.646 DEBUG:teuthology.orchestra.run.smithi046:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-01-27T01:09:25.657 DEBUG:teuthology.orchestra.run.smithi106:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-01-27T01:09:25.691 INFO:teuthology.orchestra.run.smithi046.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-01-27T01:09:25.713 INFO:teuthology.orchestra.run.smithi106.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-01-27T01:09:25.719 INFO:tasks.ceph:Archiving crash dumps... 2024-01-27T01:09:25.721 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi/7533360/remote/smithi046/crash 2024-01-27T01:09:25.721 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-27T01:09:25.773 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi/7533360/remote/smithi106/crash 2024-01-27T01:09:25.774 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-27T01:09:25.800 INFO:tasks.ceph:Shutting down logrotate 2024-01-27T01:09:25.800 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:09:25.818 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-27T01:09:25.890 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-27T01:09:25.924 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-27T01:09:25.951 INFO:tasks.ceph:Compressing logs... 2024-01-27T01:09:25.951 DEBUG:teuthology.orchestra.run.smithi046:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-27T01:09:25.967 DEBUG:teuthology.orchestra.run.smithi106:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-27T01:09:26.000 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53512.log 2024-01-27T01:09:26.001 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97261.log 2024-01-27T01:09:26.001 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98522.log 2024-01-27T01:09:26.003 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97261.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53512.log.gz 2024-01-27T01:09:26.003 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60028.log 2024-01-27T01:09:26.003 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.97261.log.gz 2024-01-27T01:09:26.003 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.98522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119385.log 2024-01-27T01:09:26.004 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.98522.log.gz 2024-01-27T01:09:26.004 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70548.log 2024-01-27T01:09:26.004 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60028.log: /var/log/ceph/ceph-client.admin.119385.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60028.log.gz 2024-01-27T01:09:26.004 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105500.log 2024-01-27T01:09:26.004 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56671.log 2024-01-27T01:09:26.004 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.119385.log.gz 2024-01-27T01:09:26.004 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70548.log.gz 2024-01-27T01:09:26.004 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55873.log 2024-01-27T01:09:26.004 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.105500.log.gz 2024-01-27T01:09:26.004 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75275.log 2024-01-27T01:09:26.008 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56671.log.gz 2024-01-27T01:09:26.008 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116070.log 2024-01-27T01:09:26.008 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55873.log.gz 2024-01-27T01:09:26.008 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87981.log 2024-01-27T01:09:26.008 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75275.log.gz 2024-01-27T01:09:26.008 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.116070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72026.log 2024-01-27T01:09:26.008 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116070.log.gz 2024-01-27T01:09:26.008 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59218.log 2024-01-27T01:09:26.008 INFO:teuthology.orchestra.run.smithi046.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.87981.log.gz 2024-01-27T01:09:26.009 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123780.log 2024-01-27T01:09:26.009 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59218.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72026.log.gz 2024-01-27T01:09:26.009 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78673.log 2024-01-27T01:09:26.009 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59218.log.gz 2024-01-27T01:09:26.009 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.123780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51021.log 2024-01-27T01:09:26.009 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.123780.log.gz 2024-01-27T01:09:26.009 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55136.log 2024-01-27T01:09:26.009 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.78673.log.gz 2024-01-27T01:09:26.009 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122180.log 2024-01-27T01:09:26.009 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51021.log.gz 2024-01-27T01:09:26.009 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81270.log 2024-01-27T01:09:26.009 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55136.log.gz 2024-01-27T01:09:26.009 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.122180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99132.log 2024-01-27T01:09:26.009 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124859.log 89.3% 2024-01-27T01:09:26.009 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.122180.log.gz 2024-01-27T01:09:26.010 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.81270.log.gz 2024-01-27T01:09:26.010 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118934.log 2024-01-27T01:09:26.015 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124859.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99132.log.gz 2024-01-27T01:09:26.015 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112183.log 2024-01-27T01:09:26.016 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.124859.log.gz 2024-01-27T01:09:26.016 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.118934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66311.log 2024-01-27T01:09:26.016 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.118934.log.gz 2024-01-27T01:09:26.016 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65619.log 2024-01-27T01:09:26.016 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112183.log.gz 2024-01-27T01:09:26.016 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64887.log 2024-01-27T01:09:26.016 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66311.log.gz 2024-01-27T01:09:26.016 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57681.log 2024-01-27T01:09:26.016 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65619.log.gz 2024-01-27T01:09:26.016 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.106264.log 2024-01-27T01:09:26.016 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57681.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64887.log.gz 2024-01-27T01:09:26.016 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111199.log 2024-01-27T01:09:26.016 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57681.log.gz 2024-01-27T01:09:26.016 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.106264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89971.log 2024-01-27T01:09:26.017 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99636.log 2024-01-27T01:09:26.017 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89971.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.111199.log.gz 2024-01-27T01:09:26.017 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109998.log 2024-01-27T01:09:26.017 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89971.log.gz 2024-01-27T01:09:26.017 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75574.log 2024-01-27T01:09:26.017 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99636.log: /var/log/ceph/ceph-client.admin.109998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86225.log 2024-01-27T01:09:26.017 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.99636.log.gz 2024-01-27T01:09:26.017 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.109998.log.gz 2024-01-27T01:09:26.017 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81700.log 2024-01-27T01:09:26.017 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75574.log.gz 2024-01-27T01:09:26.017 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109949.log 2024-01-27T01:09:26.017 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86225.log: /var/log/ceph/ceph-client.admin.81700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55229.log 2024-01-27T01:09:26.017 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.86225.log.gz 2024-01-27T01:09:26.017 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81700.log.gz 2024-01-27T01:09:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75920.log 2024-01-27T01:09:26.018 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.109949.log.gz 2024-01-27T01:09:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74281.log 2024-01-27T01:09:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55229.log: /var/log/ceph/ceph-client.admin.75920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121243.log 2024-01-27T01:09:26.018 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55229.log.gz 2024-01-27T01:09:26.018 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.75920.log.gz 2024-01-27T01:09:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114950.log 2024-01-27T01:09:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.121243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72180.log 2024-01-27T01:09:26.018 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74281.log.gz 2024-01-27T01:09:26.018 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.121243.log.gz 2024-01-27T01:09:26.018 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.114950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105594.log 2024-01-27T01:09:26.025 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-01-27T01:09:26.026 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-01-27T01:09:26.026 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43600.log 2024-01-27T01:09:26.026 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43695.log 2024-01-27T01:09:26.026 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.43600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43649.log 2024-01-27T01:09:26.026 INFO:teuthology.orchestra.run.smithi106.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43600.log.gz 2024-01-27T01:09:26.026 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.43695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43934.log 2024-01-27T01:09:26.026 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-27T01:09:26.026 INFO:teuthology.orchestra.run.smithi106.stderr: 88.1%/var/log/ceph/ceph-client.admin.43649.log: -- replaced with /var/log/ceph/ceph-client.admin.43695.log.gz 2024-01-27T01:09:26.026 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.43934.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-01-27T01:09:26.026 INFO:teuthology.orchestra.run.smithi106.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43649.log.gz 2024-01-27T01:09:26.026 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43934.log.gz 2024-01-27T01:09:26.026 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43887.log 2024-01-27T01:09:26.026 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-27T01:09:26.027 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.43887.log: /var/log/ceph/ceph.audit.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43887.log.gz 2024-01-27T01:09:26.027 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.51047.log 2024-01-27T01:09:26.027 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.51656.log 2024-01-27T01:09:26.027 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.1.51047.log: /var/log/ceph/ceph-client.3.51656.log: 89.8% -- replaced with /var/log/ceph/ceph-client.1.51047.log.gz 2024-01-27T01:09:26.027 INFO:teuthology.orchestra.run.smithi106.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.3.51656.log.gz 2024-01-27T01:09:26.027 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-01-27T01:09:26.027 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-01-27T01:09:26.038 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-osd.6.log: 93.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-27T01:09:26.038 INFO:teuthology.orchestra.run.smithi106.stderr: 93.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-27T01:09:26.038 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43794.log 2024-01-27T01:09:26.038 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-01-27T01:09:26.038 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.43794.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43794.log.gz 2024-01-27T01:09:26.043 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43839.log 2024-01-27T01:09:26.043 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.43839.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43839.log.gz 2024-01-27T01:09:26.043 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-01-27T01:09:26.047 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43748.log 2024-01-27T01:09:26.047 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.43748.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43748.log.gz 2024-01-27T01:09:26.048 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-27T01:09:26.053 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72180.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.114950.log.gz 2024-01-27T01:09:26.053 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120947.log 2024-01-27T01:09:26.053 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72180.log.gz 2024-01-27T01:09:26.053 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118533.log 2024-01-27T01:09:26.053 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105594.log: /var/log/ceph/ceph-client.admin.120947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85362.log 2024-01-27T01:09:26.053 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.105594.log.gz 2024-01-27T01:09:26.054 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.120947.log.gz 2024-01-27T01:09:26.054 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.118533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99390.log 2024-01-27T01:09:26.054 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.118533.log.gz 2024-01-27T01:09:26.054 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123264.log 2024-01-27T01:09:26.054 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99390.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85362.log.gz 2024-01-27T01:09:26.054 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77248.log 2024-01-27T01:09:26.054 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.99390.log.gz 2024-01-27T01:09:26.054 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.123264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81430.log 2024-01-27T01:09:26.054 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110267.log 2024-01-27T01:09:26.054 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.123264.log.gz 2024-01-27T01:09:26.054 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77248.log.gz 2024-01-27T01:09:26.054 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.117260.log 2024-01-27T01:09:26.054 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81430.log.gz 2024-01-27T01:09:26.055 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117955.log 2024-01-27T01:09:26.055 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.110267.log.gz 2024-01-27T01:09:26.055 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.117260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50075.log 2024-01-27T01:09:26.055 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.117955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97909.log 2024-01-27T01:09:26.055 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.117955.log.gz 2024-01-27T01:09:26.055 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87669.log 2024-01-27T01:09:26.055 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71223.log 2024-01-27T01:09:26.055 INFO:teuthology.orchestra.run.smithi046.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.50075.log.gz 2024-01-27T01:09:26.055 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123525.log 2024-01-27T01:09:26.055 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88026.log 2024-01-27T01:09:26.055 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71223.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87669.log.gz 2024-01-27T01:09:26.055 INFO:teuthology.orchestra.run.smithi046.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.97909.log.gz 2024-01-27T01:09:26.055 INFO:teuthology.orchestra.run.smithi046.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.0.117260.log.gz 2024-01-27T01:09:26.056 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71223.log.gz 2024-01-27T01:09:26.056 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.123525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124997.log 2024-01-27T01:09:26.056 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.123525.log.gz 2024-01-27T01:09:26.056 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83427.log 2024-01-27T01:09:26.056 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124997.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.88026.log.gz 2024-01-27T01:09:26.056 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-27T01:09:26.056 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.124997.log.gz 2024-01-27T01:09:26.056 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.83427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109798.log 2024-01-27T01:09:26.056 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mon.a.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.83427.log.gz 2024-01-27T01:09:26.056 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120563.log 2024-01-27T01:09:26.056 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66357.log 2024-01-27T01:09:26.056 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.109798.log.gz 2024-01-27T01:09:26.056 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.120563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68203.log 2024-01-27T01:09:26.056 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.120563.log.gz 2024-01-27T01:09:26.057 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71361.log 2024-01-27T01:09:26.057 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66357.log: /var/log/ceph/ceph-client.admin.68203.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66357.log.gz 2024-01-27T01:09:26.057 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90111.log 2024-01-27T01:09:26.057 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71361.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68203.log.gz 2024-01-27T01:09:26.057 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112702.log 2024-01-27T01:09:26.057 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71361.log.gz 2024-01-27T01:09:26.057 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88422.log 2024-01-27T01:09:26.057 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.90111.log.gz 2024-01-27T01:09:26.057 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85199.log 2024-01-27T01:09:26.057 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86066.log 2024-01-27T01:09:26.057 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.112702.log.gz 2024-01-27T01:09:26.057 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85199.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.88422.log.gz 2024-01-27T01:09:26.057 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74987.log 2024-01-27T01:09:26.057 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85199.log.gz 2024-01-27T01:09:26.058 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124119.log 2024-01-27T01:09:26.058 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74987.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86066.log.gz 2024-01-27T01:09:26.058 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98641.log 2024-01-27T01:09:26.058 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74987.log.gz 2024-01-27T01:09:26.058 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79654.log 2024-01-27T01:09:26.058 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.124119.log.gz 2024-01-27T01:09:26.058 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.98641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116730.log 2024-01-27T01:09:26.058 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115272.log 2024-01-27T01:09:26.058 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.98641.log.gz 2024-01-27T01:09:26.058 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79654.log: /var/log/ceph/ceph-client.admin.116730.log: gzip -5 88.3%--verbose -- /var/log/ceph/ceph-client.admin.112556.log -- replaced with /var/log/ceph/ceph-client.admin.79654.log.gz 2024-01-27T01:09:26.058 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-27T01:09:26.058 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.116730.log.gz 2024-01-27T01:09:26.058 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97403.log 2024-01-27T01:09:26.058 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123177.log 2024-01-27T01:09:26.058 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.115272.log.gz 2024-01-27T01:09:26.059 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112556.log: /var/log/ceph/ceph-client.admin.97403.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.112556.log.gz 2024-01-27T01:09:26.059 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76994.log 2024-01-27T01:09:26.059 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97403.log.gz 2024-01-27T01:09:26.059 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.123177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102810.log 2024-01-27T01:09:26.059 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76994.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.123177.log.gz 2024-01-27T01:09:26.059 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54063.log 2024-01-27T01:09:26.059 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76994.log.gz 2024-01-27T01:09:26.059 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65502.log 2024-01-27T01:09:26.059 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102810.log: /var/log/ceph/ceph-client.0.54063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79969.log 2024-01-27T01:09:26.059 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.102810.log.gz 2024-01-27T01:09:26.059 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119239.log 2024-01-27T01:09:26.059 INFO:teuthology.orchestra.run.smithi046.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.0.54063.log.gz 2024-01-27T01:09:26.059 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65502.log.gz 2024-01-27T01:09:26.059 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74419.log 2024-01-27T01:09:26.059 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.79969.log.gzgzip 2024-01-27T01:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.88473.log 2024-01-27T01:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119239.log: /var/log/ceph/ceph-client.admin.74419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52333.log 2024-01-27T01:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.119239.log.gz 2024-01-27T01:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88473.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74419.log.gz 2024-01-27T01:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78949.log 2024-01-27T01:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52333.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.88473.log.gz 2024-01-27T01:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87801.log 2024-01-27T01:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52333.log.gz 2024-01-27T01:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49212.log 2024-01-27T01:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.78949.log.gz 2024-01-27T01:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77620.log 2024-01-27T01:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87801.log: /var/log/ceph/ceph-client.admin.49212.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.87801.log.gz 2024-01-27T01:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55827.log 2024-01-27T01:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2%/var/log/ceph/ceph-client.admin.77620.log: -- replaced with /var/log/ceph/ceph-client.admin.49212.log.gz 2024-01-27T01:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74009.log 2024-01-27T01:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77620.log.gz 2024-01-27T01:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98713.log 2024-01-27T01:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55827.log.gz 2024-01-27T01:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:gzip /var/log/ceph/ceph-client.admin.74009.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.55781.log 2024-01-27T01:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.98713.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74009.log.gz 2024-01-27T01:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57418.log 2024-01-27T01:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.98713.log.gz 2024-01-27T01:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80368.log 2024-01-27T01:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55781.log.gz 2024-01-27T01:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121567.log 2024-01-27T01:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57418.log.gz 2024-01-27T01:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84969.log 2024-01-27T01:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.121567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55044.log 2024-01-27T01:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80368.log.gz 2024-01-27T01:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84969.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.121567.log.gz 2024-01-27T01:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49271.log 2024-01-27T01:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.84969.log.gz 2024-01-27T01:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49340.log 2024-01-27T01:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49271.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55044.log.gz 2024-01-27T01:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117907.log 2024-01-27T01:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49271.log.gz 2024-01-27T01:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77099.log 2024-01-27T01:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.117907.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49340.log.gz 2024-01-27T01:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119140.log 2024-01-27T01:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.117907.log.gz 2024-01-27T01:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73811.log 2024-01-27T01:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77099.log: /var/log/ceph/ceph-client.admin.119140.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77099.log.gz 2024-01-27T01:09:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59614.log 2024-01-27T01:09:26.063 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.119140.log.gz 2024-01-27T01:09:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98476.log 2024-01-27T01:09:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66693.log 2024-01-27T01:09:26.063 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73811.log.gz 2024-01-27T01:09:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.98476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88825.log 2024-01-27T01:09:26.063 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59614.log.gz 2024-01-27T01:09:26.063 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.98476.log.gz 2024-01-27T01:09:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75229.log 2024-01-27T01:09:26.063 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.66693.log.gz 2024-01-27T01:09:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112955.log 2024-01-27T01:09:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88825.log: /var/log/ceph/ceph-client.admin.75229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119288.log 2024-01-27T01:09:26.063 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.88825.log.gz 2024-01-27T01:09:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112955.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.75229.log.gz 2024-01-27T01:09:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70707.log 2024-01-27T01:09:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119288.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.112955.log.gz 2024-01-27T01:09:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105099.log 2024-01-27T01:09:26.064 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.119288.log.gz 2024-01-27T01:09:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73704.log 2024-01-27T01:09:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70707.log: /var/log/ceph/ceph-client.admin.105099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80744.log 2024-01-27T01:09:26.064 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70707.log.gz 2024-01-27T01:09:26.064 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.105099.log.gz 2024-01-27T01:09:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81990.log 2024-01-27T01:09:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97541.log 2024-01-27T01:09:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80744.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73704.log.gz 2024-01-27T01:09:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81990.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.80744.log.gz 2024-01-27T01:09:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100624.log 2024-01-27T01:09:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97541.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81990.log.gz 2024-01-27T01:09:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.122957.log 2024-01-27T01:09:26.065 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.97541.log.gz 2024-01-27T01:09:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116498.log 2024-01-27T01:09:26.065 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.100624.log.gz 2024-01-27T01:09:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.122957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81796.log 2024-01-27T01:09:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.116498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60778.log 2024-01-27T01:09:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81796.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.116498.log.gz 2024-01-27T01:09:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49128.log 2024-01-27T01:09:26.065 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81796.log.gz 2024-01-27T01:09:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78349.log 2024-01-27T01:09:26.065 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60778.log.gz 2024-01-27T01:09:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49128.log: /var/log/ceph/ceph-client.admin.78349.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49128.log.gz 2024-01-27T01:09:26.065 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78349.log.gz 2024-01-27T01:09:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97215.log 2024-01-27T01:09:26.065 INFO:teuthology.orchestra.run.smithi046.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.0.122957.log.gz 2024-01-27T01:09:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56487.log 2024-01-27T01:09:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72641.log 2024-01-27T01:09:26.066 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.97215.log.gz 2024-01-27T01:09:26.066 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91000.log 2024-01-27T01:09:26.066 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56487.log.gz 2024-01-27T01:09:26.066 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67247.log 2024-01-27T01:09:26.066 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72641.log: /var/log/ceph/ceph-client.admin.91000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65048.log 2024-01-27T01:09:26.066 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72641.log.gz 2024-01-27T01:09:26.066 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.91000.log.gz 2024-01-27T01:09:26.066 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77666.log 2024-01-27T01:09:26.066 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97495.log 2024-01-27T01:09:26.066 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65048.log: /var/log/ceph/ceph-client.admin.77666.log: gzip -5 --verbose -- 87.5% 91.4%/var/log/ceph/ceph-client.admin.85245.log 2024-01-27T01:09:26.066 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67247.log.gz -- replaced with /var/log/ceph/ceph-client.admin.65048.log.gz 2024-01-27T01:09:26.066 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-27T01:09:26.066 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77666.log.gz 2024-01-27T01:09:26.066 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.79483.log 2024-01-27T01:09:26.066 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.97495.log.gz 2024-01-27T01:09:26.066 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61827.log 2024-01-27T01:09:26.067 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85245.log.gz 2024-01-27T01:09:26.067 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.79483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120063.log 2024-01-27T01:09:26.067 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59482.log 2024-01-27T01:09:26.068 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-01-27T01:09:26.068 INFO:teuthology.orchestra.run.smithi106.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-01-27T01:09:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61827.log: /var/log/ceph/ceph-client.admin.120063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59982.log 2024-01-27T01:09:26.084 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61827.log.gz 2024-01-27T01:09:26.084 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.120063.log.gz 2024-01-27T01:09:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80647.log 2024-01-27T01:09:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59982.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59482.log.gz 2024-01-27T01:09:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79087.log 2024-01-27T01:09:26.084 INFO:teuthology.orchestra.run.smithi046.stderr: 94.2% -- replaced with /var/log/ceph/ceph-client.0.79483.log.gz 2024-01-27T01:09:26.084 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59982.log.gz 2024-01-27T01:09:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116210.log 2024-01-27T01:09:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120723.log 2024-01-27T01:09:26.084 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80647.log.gz/var/log/ceph/ceph-client.admin.79087.log: 2024-01-27T01:09:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.116210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119482.log 2024-01-27T01:09:26.085 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116210.log.gz 2024-01-27T01:09:26.085 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.120723.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.79087.log.gz 2024-01-27T01:09:26.085 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115780.log 2024-01-27T01:09:26.085 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.120723.log.gz 2024-01-27T01:09:26.085 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53048.log 2024-01-27T01:09:26.085 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.119482.log.gz 2024-01-27T01:09:26.085 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120017.log 2024-01-27T01:09:26.085 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111245.log 2024-01-27T01:09:26.085 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.115780.log.gz 2024-01-27T01:09:26.085 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53048.log.gz 2024-01-27T01:09:26.085 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.120017.log: /var/log/ceph/ceph-client.admin.111245.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.120017.log.gz 2024-01-27T01:09:26.085 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111245.log.gz 2024-01-27T01:09:26.085 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113001.log 2024-01-27T01:09:26.086 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112796.log 2024-01-27T01:09:26.086 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110865.log 2024-01-27T01:09:26.086 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.113001.log.gz 2024-01-27T01:09:26.086 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59936.log 2024-01-27T01:09:26.086 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49394.log 2024-01-27T01:09:26.086 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.112796.log.gz 2024-01-27T01:09:26.086 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110865.log: /var/log/ceph/ceph-client.admin.59936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.86941.log 2024-01-27T01:09:26.086 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.110865.log.gz 2024-01-27T01:09:26.086 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49394.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59936.log.gz 2024-01-27T01:09:26.086 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116776.log 2024-01-27T01:09:26.086 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49394.log.gz 2024-01-27T01:09:26.086 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.86941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110359.log 2024-01-27T01:09:26.086 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.116776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89505.log 2024-01-27T01:09:26.086 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.116776.log.gz 2024-01-27T01:09:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111385.log 2024-01-27T01:09:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89505.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110359.log.gz 2024-01-27T01:09:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96408.log 2024-01-27T01:09:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111385.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89505.log.gz 2024-01-27T01:09:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99805.log 2024-01-27T01:09:26.087 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.111385.log.gz 2024-01-27T01:09:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104715.log 2024-01-27T01:09:26.087 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96408.log.gz 2024-01-27T01:09:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124721.log 2024-01-27T01:09:26.087 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6%/var/log/ceph/ceph-client.admin.104715.log: -- replaced with /var/log/ceph/ceph-client.admin.99805.log.gz 2024-01-27T01:09:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116592.log 2024-01-27T01:09:26.087 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2%/var/log/ceph/ceph-client.admin.124721.log: -- replaced with /var/log/ceph/ceph-client.admin.104715.log.gz 2024-01-27T01:09:26.087 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.124721.log.gz 2024-01-27T01:09:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.116592.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.116592.log.gz 2024-01-27T01:09:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96476.log 2024-01-27T01:09:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124905.log 2024-01-27T01:09:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99488.log 2024-01-27T01:09:26.088 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.96476.log.gz 2024-01-27T01:09:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100948.log 2024-01-27T01:09:26.088 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.124905.log.gz 2024-01-27T01:09:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117759.log 2024-01-27T01:09:26.088 INFO:teuthology.orchestra.run.smithi046.stderr: 93.5% -- replaced with /var/log/ceph/ceph-client.0.86941.log.gz 2024-01-27T01:09:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100948.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.99488.log.gz 2024-01-27T01:09:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77871.log 2024-01-27T01:09:26.088 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.100948.log.gz 2024-01-27T01:09:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.117759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110095.log 2024-01-27T01:09:26.088 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.117759.log.gz 2024-01-27T01:09:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55505.log 2024-01-27T01:09:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105548.log 2024-01-27T01:09:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110095.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77871.log.gz 2024-01-27T01:09:26.089 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.110095.log.gz 2024-01-27T01:09:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67930.log 2024-01-27T01:09:26.089 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8%/var/log/ceph/ceph-client.admin.105548.log: gzip -- replaced with /var/log/ceph/ceph-client.admin.55505.log.gz-5 2024-01-27T01:09:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:--verbose -- /var/log/ceph/ceph-client.admin.105408.log 2024-01-27T01:09:26.089 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105548.log.gz 2024-01-27T01:09:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113866.log 2024-01-27T01:09:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 /var/log/ceph/ceph-client.admin.105408.log: --verbose -- /var/log/ceph/ceph-client.admin.100856.log 2024-01-27T01:09:26.089 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67930.log.gz 2024-01-27T01:09:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113866.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105408.log.gz 2024-01-27T01:09:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55367.log 2024-01-27T01:09:26.089 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.113866.log.gz 2024-01-27T01:09:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95744.log 2024-01-27T01:09:26.090 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100856.log.gz 2024-01-27T01:09:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86363.log 2024-01-27T01:09:26.090 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55367.log.gz 2024-01-27T01:09:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87161.log 2024-01-27T01:09:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86363.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.95744.log.gz 2024-01-27T01:09:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96310.log 2024-01-27T01:09:26.090 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.86363.log.gz 2024-01-27T01:09:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87115.log 2024-01-27T01:09:26.090 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87161.log.gz 2024-01-27T01:09:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99585.log 2024-01-27T01:09:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87115.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96310.log.gz 2024-01-27T01:09:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110497.log 2024-01-27T01:09:26.090 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87115.log.gz 2024-01-27T01:09:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119971.log 2024-01-27T01:09:26.091 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.99585.log.gz 2024-01-27T01:09:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52219.log 2024-01-27T01:09:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119971.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.110497.log.gz 2024-01-27T01:09:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-01-27T01:09:26.091 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.119971.log.gz 2024-01-27T01:09:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57140.log 2024-01-27T01:09:26.091 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52219.log.gz 2024-01-27T01:09:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81039.log 2024-01-27T01:09:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115042.log 2024-01-27T01:09:26.091 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57140.log.gz 2024-01-27T01:09:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72088.log 2024-01-27T01:09:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115042.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.81039.log.gz 2024-01-27T01:09:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104807.log 2024-01-27T01:09:26.091 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.115042.log.gz 2024-01-27T01:09:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58756.log 2024-01-27T01:09:26.092 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2%/var/log/ceph/ceph-client.admin.104807.log: -- replaced with /var/log/ceph/ceph-client.admin.72088.log.gz 2024-01-27T01:09:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58756.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.104807.log.gz 2024-01-27T01:09:26.092 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58756.log.gz 2024-01-27T01:09:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70600.log 2024-01-27T01:09:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74557.log 2024-01-27T01:09:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85684.log 2024-01-27T01:09:26.092 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70600.log.gz 2024-01-27T01:09:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.67642.log 2024-01-27T01:09:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:gzip 88.5%-5 -- replaced with /var/log/ceph/ceph-client.admin.74557.log.gz--verbose 2024-01-27T01:09:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:-- /var/log/ceph/ceph-client.0.72965.log 2024-01-27T01:09:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85684.log: /var/log/ceph/ceph-client.0.67642.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.85684.log.gz 2024-01-27T01:09:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.72965.log: 91.6% -- replaced with /var/log/ceph/ceph-client.0.72965.log.gz 2024-01-27T01:09:26.092 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115339.log 2024-01-27T01:09:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88777.log 2024-01-27T01:09:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115339.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.115339.log.gz/var/log/ceph/ceph-client.admin.88777.log: 2024-01-27T01:09:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76634.log 2024-01-27T01:09:26.093 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.88777.log.gz 2024-01-27T01:09:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59890.log 2024-01-27T01:09:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123681.log 2024-01-27T01:09:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59890.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76634.log.gz 2024-01-27T01:09:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.123681.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59890.log.gz 2024-01-27T01:09:26.093 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.123681.log.gz 2024-01-27T01:09:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115542.log 2024-01-27T01:09:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105191.log 2024-01-27T01:09:26.094 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-osd.4.log: 92.4% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-01-27T01:09:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110045.log 2024-01-27T01:09:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105191.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.115542.log.gz 2024-01-27T01:09:26.096 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7%/var/log/ceph/ceph-client.admin.110045.log: -- replaced with /var/log/ceph/ceph-client.admin.105191.log.gz 2024-01-27T01:09:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82128.log 2024-01-27T01:09:26.096 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.110045.log.gz 2024-01-27T01:09:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89682.log 2024-01-27T01:09:26.096 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82128.log.gz 2024-01-27T01:09:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58143.log 2024-01-27T01:09:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90954.log 2024-01-27T01:09:26.097 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89682.log.gz 2024-01-27T01:09:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58143.log: /var/log/ceph/ceph-client.admin.90954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118298.log 2024-01-27T01:09:26.097 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90954.log.gz 2024-01-27T01:09:26.097 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.58143.log.gz 2024-01-27T01:09:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.118298.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.118298.log.gz 2024-01-27T01:09:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59752.log 2024-01-27T01:09:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56211.log 2024-01-27T01:09:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65676.log 2024-01-27T01:09:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56211.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59752.log.gz 2024-01-27T01:09:26.097 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56211.log.gz 2024-01-27T01:09:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65676.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65676.log.gz 2024-01-27T01:09:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.58537.log 2024-01-27T01:09:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.90782.log 2024-01-27T01:09:26.099 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.58537.log: /var/log/ceph/ceph-client.0.90782.log: 94.4% -- replaced with /var/log/ceph/ceph-client.0.106264.log.gz 2024-01-27T01:09:26.099 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60732.log 2024-01-27T01:09:26.099 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75758.log 2024-01-27T01:09:26.100 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60732.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60732.log.gz 2024-01-27T01:09:26.101 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75758.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.75758.log.gz 2024-01-27T01:09:26.101 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73652.log 2024-01-27T01:09:26.101 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49732.log 2024-01-27T01:09:26.102 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81131.log 2024-01-27T01:09:26.102 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73652.log.gz 2024-01-27T01:09:26.102 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49732.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49732.log.gz 2024-01-27T01:09:26.103 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81131.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81131.log.gz 2024-01-27T01:09:26.103 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51999.log 2024-01-27T01:09:26.103 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84181.log 2024-01-27T01:09:26.104 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51999.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51999.log.gz 2024-01-27T01:09:26.104 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84181.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84181.log.gz 2024-01-27T01:09:26.104 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75620.log 2024-01-27T01:09:26.105 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77574.log 2024-01-27T01:09:26.105 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75620.log: /var/log/ceph/ceph-client.admin.77574.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75620.log.gz 2024-01-27T01:09:26.106 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77574.log.gz 2024-01-27T01:09:26.106 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57959.log 2024-01-27T01:09:26.106 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59086.log 2024-01-27T01:09:26.107 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57959.log: /var/log/ceph/ceph-client.admin.59086.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57959.log.gz 2024-01-27T01:09:26.107 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59086.log.gz 2024-01-27T01:09:26.107 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49778.log 2024-01-27T01:09:26.108 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52818.log 2024-01-27T01:09:26.108 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49778.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49778.log.gz 2024-01-27T01:09:26.109 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52818.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52818.log.gz 2024-01-27T01:09:26.109 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50145.log 2024-01-27T01:09:26.109 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96809.log 2024-01-27T01:09:26.110 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50145.log: /var/log/ceph/ceph-client.admin.96809.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50145.log.gz 2024-01-27T01:09:26.111 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.96809.log.gz 2024-01-27T01:09:26.111 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85452.log 2024-01-27T01:09:26.111 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71745.log 2024-01-27T01:09:26.112 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85452.log: /var/log/ceph/ceph-client.admin.71745.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.85452.log.gz 2024-01-27T01:09:26.112 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.71745.log.gz 2024-01-27T01:09:26.112 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55551.log 2024-01-27T01:09:26.112 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78581.log 2024-01-27T01:09:26.113 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55551.log: /var/log/ceph/ceph-client.admin.78581.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55551.log.gz 2024-01-27T01:09:26.114 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78581.log.gz 2024-01-27T01:09:26.114 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78115.log 2024-01-27T01:09:26.114 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104669.log 2024-01-27T01:09:26.115 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78115.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78115.log.gz 2024-01-27T01:09:26.115 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104669.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.104669.log.gz 2024-01-27T01:09:26.115 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118882.log 2024-01-27T01:09:26.116 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.118882.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.118882.log.gz 2024-01-27T01:09:26.116 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70262.log 2024-01-27T01:09:26.117 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78719.log 2024-01-27T01:09:26.118 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70262.log: /var/log/ceph/ceph-client.admin.78719.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70262.log.gz 2024-01-27T01:09:26.118 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78719.log.gz 2024-01-27T01:09:26.118 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99439.log 2024-01-27T01:09:26.118 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115088.log 2024-01-27T01:09:26.119 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56533.log 2024-01-27T01:09:26.119 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.99439.log.gz 2024-01-27T01:09:26.119 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115088.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.115088.log.gz 2024-01-27T01:09:26.120 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56533.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56533.log.gz 2024-01-27T01:09:26.120 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56735.log 2024-01-27T01:09:26.120 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78903.log 2024-01-27T01:09:26.121 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56735.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56735.log.gz 2024-01-27T01:09:26.121 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78903.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.78903.log.gz 2024-01-27T01:09:26.121 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51160.log 2024-01-27T01:09:26.122 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67109.log 2024-01-27T01:09:26.122 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51160.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51160.log.gz 2024-01-27T01:09:26.123 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67109.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67109.log.gz 2024-01-27T01:09:26.123 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122515.log 2024-01-27T01:09:26.123 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65722.log 2024-01-27T01:09:26.124 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.122515.log: /var/log/ceph/ceph-client.admin.65722.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.122515.log.gz 2024-01-27T01:09:26.124 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49962.log 2024-01-27T01:09:26.124 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65722.log.gz 2024-01-27T01:09:26.125 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115826.log 2024-01-27T01:09:26.125 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49962.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49962.log.gz 2024-01-27T01:09:26.126 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115826.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.115826.log.gz 2024-01-27T01:09:26.126 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52726.log 2024-01-27T01:09:26.126 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121197.log 2024-01-27T01:09:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52726.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52726.log.gz 2024-01-27T01:09:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.121197.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.121197.log.gz 2024-01-27T01:09:26.128 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68157.log 2024-01-27T01:09:26.128 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77941.log 2024-01-27T01:09:26.128 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68157.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68157.log.gz 2024-01-27T01:09:26.129 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77941.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77941.log.gz 2024-01-27T01:09:26.129 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115180.log 2024-01-27T01:09:26.129 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78443.log 2024-01-27T01:09:26.130 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115180.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.115180.log.gz 2024-01-27T01:09:26.131 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78443.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78443.log.gz 2024-01-27T01:09:26.131 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89729.log 2024-01-27T01:09:26.131 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61008.log 2024-01-27T01:09:26.132 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89729.log: /var/log/ceph/ceph-client.admin.61008.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.89729.log.gz 2024-01-27T01:09:26.132 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61008.log.gz 2024-01-27T01:09:26.132 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119649.log 2024-01-27T01:09:26.133 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71407.log 2024-01-27T01:09:26.133 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119649.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.119649.log.gz 2024-01-27T01:09:26.134 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71407.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71407.log.gz 2024-01-27T01:09:26.134 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96671.log 2024-01-27T01:09:26.134 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60074.log 2024-01-27T01:09:26.135 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96671.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96671.log.gz 2024-01-27T01:09:26.135 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60074.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60074.log.gz 2024-01-27T01:09:26.135 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73254.log 2024-01-27T01:09:26.136 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101953.log 2024-01-27T01:09:26.136 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73254.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.73254.log.gz 2024-01-27T01:09:26.137 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101953.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.101953.log.gz 2024-01-27T01:09:26.137 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106818.log 2024-01-27T01:09:26.137 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70755.log 2024-01-27T01:09:26.139 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.106818.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.106818.log.gz 2024-01-27T01:09:26.139 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70755.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70755.log.gz 2024-01-27T01:09:26.139 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124071.log 2024-01-27T01:09:26.139 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117665.log 2024-01-27T01:09:26.140 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124071.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.124071.log.gz 2024-01-27T01:09:26.140 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.117665.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.117665.log.gz 2024-01-27T01:09:26.140 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84795.log 2024-01-27T01:09:26.141 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66741.log 2024-01-27T01:09:26.141 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84795.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84795.log.gz 2024-01-27T01:09:26.142 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66741.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66741.log.gz 2024-01-27T01:09:26.142 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52772.log 2024-01-27T01:09:26.142 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50008.log 2024-01-27T01:09:26.143 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52772.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52772.log.gz 2024-01-27T01:09:26.143 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50008.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50008.log.gz 2024-01-27T01:09:26.144 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65906.log 2024-01-27T01:09:26.144 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120496.log 2024-01-27T01:09:26.145 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65906.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65906.log.gz 2024-01-27T01:09:26.145 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.120496.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.120496.log.gz 2024-01-27T01:09:26.145 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57867.log 2024-01-27T01:09:26.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-01-27T01:09:26.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-01-27T01:09:26.146 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56257.log 2024-01-27T01:09:26.146 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57867.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57867.log.gz 2024-01-27T01:09:26.147 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56257.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56257.log.gz 2024-01-27T01:09:26.147 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80694.log 2024-01-27T01:09:26.147 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99288.log 2024-01-27T01:09:26.147 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80694.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80694.log.gz 2024-01-27T01:09:26.148 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99288.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.99288.log.gz 2024-01-27T01:09:26.148 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65551.log 2024-01-27T01:09:26.149 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65551.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65551.log.gz 2024-01-27T01:09:26.149 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85291.log 2024-01-27T01:09:26.150 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95956.log 2024-01-27T01:09:26.150 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85291.log: 88.1%/var/log/ceph/ceph-client.admin.95956.log: -- replaced with /var/log/ceph/ceph-client.admin.85291.log.gz 2024-01-27T01:09:26.150 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114904.log 2024-01-27T01:09:26.151 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95956.log.gz 2024-01-27T01:09:26.151 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85538.log 2024-01-27T01:09:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.114904.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.114904.log.gz 2024-01-27T01:09:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122345.log 2024-01-27T01:09:26.152 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85538.log.gz 2024-01-27T01:09:26.153 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.122345.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.122345.log.gz 2024-01-27T01:09:26.153 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57606.log 2024-01-27T01:09:26.153 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74465.log 2024-01-27T01:09:26.154 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57606.log: /var/log/ceph/ceph-client.admin.74465.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57606.log.gz 2024-01-27T01:09:26.154 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74465.log.gz 2024-01-27T01:09:26.154 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51206.log 2024-01-27T01:09:26.155 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61054.log 2024-01-27T01:09:26.155 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51206.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51206.log.gz 2024-01-27T01:09:26.159 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61054.log: 94.3% -- replaced with /var/log/ceph/ceph-client.0.67642.log.gz 2024-01-27T01:09:26.159 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61054.log.gz 2024-01-27T01:09:26.159 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105237.log 2024-01-27T01:09:26.160 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78489.log 2024-01-27T01:09:26.160 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85880.log 2024-01-27T01:09:26.160 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105237.log.gz 2024-01-27T01:09:26.161 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78489.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78489.log.gz 2024-01-27T01:09:26.161 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85880.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-27T01:09:26.161 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85880.log.gz 2024-01-27T01:09:26.163 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80316.log 2024-01-27T01:09:26.164 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74673.log 2024-01-27T01:09:26.165 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80316.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80316.log.gz 2024-01-27T01:09:26.165 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74673.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74673.log.gz 2024-01-27T01:09:26.165 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79041.log 2024-01-27T01:09:26.165 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71570.log 2024-01-27T01:09:26.166 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111003.log 2024-01-27T01:09:26.166 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.79041.log.gz 2024-01-27T01:09:26.166 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52384.log 2024-01-27T01:09:26.166 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111003.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71570.log.gz 2024-01-27T01:09:26.167 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.111003.log.gz 2024-01-27T01:09:26.167 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52384.log: gzip -5 87.3%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.52384.log.gz/var/log/ceph/ceph-client.admin.103688.log 2024-01-27T01:09:26.167 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-27T01:09:26.168 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57727.log 2024-01-27T01:09:26.168 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103688.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.103688.log.gz 2024-01-27T01:09:26.172 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105285.log 2024-01-27T01:09:26.172 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57727.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57727.log.gz 2024-01-27T01:09:26.173 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105285.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.105285.log.gz 2024-01-27T01:09:26.173 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104311.log 2024-01-27T01:09:26.175 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90295.log 2024-01-27T01:09:26.176 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104311.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.104311.log.gz 2024-01-27T01:09:26.176 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90295.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90295.log.gz 2024-01-27T01:09:26.176 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116406.log 2024-01-27T01:09:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52353.log 2024-01-27T01:09:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.116406.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.116406.log.gz/var/log/ceph/ceph-client.admin.52353.log: 2024-01-27T01:09:26.177 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52353.log.gz 2024-01-27T01:09:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105778.log 2024-01-27T01:09:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80993.log 2024-01-27T01:09:26.178 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105778.log: /var/log/ceph/ceph-client.admin.80993.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.105778.log.gz 2024-01-27T01:09:26.178 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80993.log.gz 2024-01-27T01:09:26.178 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110405.log 2024-01-27T01:09:26.178 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78765.log 2024-01-27T01:09:26.178 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110405.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110405.log.gz 2024-01-27T01:09:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78765.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.78765.log.gz 2024-01-27T01:09:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125319.log 2024-01-27T01:09:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73138.log 2024-01-27T01:09:26.180 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.125319.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.125319.log.gz 2024-01-27T01:09:26.180 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84340.log 2024-01-27T01:09:26.180 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73138.log.gz 2024-01-27T01:09:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59152.log 2024-01-27T01:09:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84340.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84340.log.gz 2024-01-27T01:09:26.182 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59152.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59152.log.gz 2024-01-27T01:09:26.182 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100035.log 2024-01-27T01:09:26.182 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71003.log 2024-01-27T01:09:26.183 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118414.log 2024-01-27T01:09:26.183 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.100035.log.gz 2024-01-27T01:09:26.183 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71003.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71003.log.gz 2024-01-27T01:09:26.184 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.118414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73962.log 2024-01-27T01:09:26.184 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.118414.log.gz 2024-01-27T01:09:26.185 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78397.log 2024-01-27T01:09:26.185 INFO:teuthology.orchestra.run.smithi046.stderr: 95.6% -- replaced with /var/log/ceph/ceph-client.0.90782.log.gz 2024-01-27T01:09:26.185 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73962.log.gz 2024-01-27T01:09:26.185 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109694.log 2024-01-27T01:09:26.185 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120404.log 2024-01-27T01:09:26.186 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78397.log: /var/log/ceph/ceph-client.admin.109694.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78397.log.gz 2024-01-27T01:09:26.186 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.120404.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.109694.log.gz 2024-01-27T01:09:26.186 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.120404.log.gz 2024-01-27T01:09:26.186 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73859.log 2024-01-27T01:09:26.187 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58097.log 2024-01-27T01:09:26.187 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105824.log 2024-01-27T01:09:26.188 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73859.log.gz 2024-01-27T01:09:26.188 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115226.log 2024-01-27T01:09:26.188 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58097.log.gz 2024-01-27T01:09:26.188 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112275.log 2024-01-27T01:09:26.188 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.105824.log.gz 2024-01-27T01:09:26.188 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125577.log 2024-01-27T01:09:26.189 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.115226.log.gz 2024-01-27T01:09:26.189 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112275.log: /var/log/ceph/ceph-client.admin.125577.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112275.log.gz 2024-01-27T01:09:26.189 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.125577.log.gz 2024-01-27T01:09:26.190 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121291.log 2024-01-27T01:09:26.190 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121097.log 2024-01-27T01:09:26.191 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.121291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80947.log 2024-01-27T01:09:26.191 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.121097.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.121291.log.gz 2024-01-27T01:09:26.191 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53324.log 2024-01-27T01:09:26.191 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.121097.log.gz 2024-01-27T01:09:26.191 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66450.log 2024-01-27T01:09:26.191 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.80947.log.gz 2024-01-27T01:09:26.192 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84650.log 2024-01-27T01:09:26.193 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66450.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53324.log.gz 2024-01-27T01:09:26.193 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84650.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66450.log.gz 2024-01-27T01:09:26.193 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84650.log.gz 2024-01-27T01:09:26.193 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90157.log 2024-01-27T01:09:26.193 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66265.log 2024-01-27T01:09:26.193 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121429.log 2024-01-27T01:09:26.194 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5%/var/log/ceph/ceph-client.admin.66265.log: -- replaced with /var/log/ceph/ceph-client.admin.90157.log.gz 2024-01-27T01:09:26.194 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59416.log 2024-01-27T01:09:26.194 INFO:teuthology.orchestra.run.smithi046.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.66265.log.gz 2024-01-27T01:09:26.195 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.121429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66019.log 2024-01-27T01:09:26.195 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.121429.log.gz 2024-01-27T01:09:26.195 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112610.log 2024-01-27T01:09:26.195 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59416.log.gz 2024-01-27T01:09:26.196 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53186.log 2024-01-27T01:09:26.196 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112610.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66019.log.gz 2024-01-27T01:09:26.196 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53186.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.112610.log.gz 2024-01-27T01:09:26.196 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53186.log.gz 2024-01-27T01:09:26.196 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111570.log 2024-01-27T01:09:26.196 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109850.log 2024-01-27T01:09:26.197 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89775.log 2024-01-27T01:09:26.197 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.111570.log.gz 2024-01-27T01:09:26.198 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77051.log 2024-01-27T01:09:26.198 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109850.log.gz 2024-01-27T01:09:26.198 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65768.log 2024-01-27T01:09:26.198 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89775.log.gz 2024-01-27T01:09:26.198 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61385.log 2024-01-27T01:09:26.199 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77051.log.gz 2024-01-27T01:09:26.199 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65768.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65768.log.gz 2024-01-27T01:09:26.200 INFO:teuthology.orchestra.run.smithi046.stderr: 93.9% -- replaced with /var/log/ceph/ceph-client.0.58537.log.gz 2024-01-27T01:09:26.200 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61385.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61385.log.gz 2024-01-27T01:09:26.200 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51436.log 2024-01-27T01:09:26.201 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84912.log 2024-01-27T01:09:26.201 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75344.log 2024-01-27T01:09:26.201 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56782.log 2024-01-27T01:09:26.201 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51436.log.gz 2024-01-27T01:09:26.202 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84912.log.gz 2024-01-27T01:09:26.202 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56441.log 2024-01-27T01:09:26.202 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75344.log.gz 2024-01-27T01:09:26.202 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113978.log 2024-01-27T01:09:26.202 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.56782.log.gz 2024-01-27T01:09:26.203 INFO:teuthology.orchestra.run.smithi046.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.100716.log 2024-01-27T01:09:26.204 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56441.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56441.log.gz 2024-01-27T01:09:26.204 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113978.log: /var/log/ceph/ceph-client.admin.100716.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113978.log.gz 2024-01-27T01:09:26.205 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100716.log.gz 2024-01-27T01:09:26.205 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123473.log 2024-01-27T01:09:26.205 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56625.log 2024-01-27T01:09:26.205 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.123473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51641.log 2024-01-27T01:09:26.205 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.123473.log.gz 2024-01-27T01:09:26.205 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111051.log 2024-01-27T01:09:26.205 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-01-27T01:09:26.205 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51641.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56625.log.gz 2024-01-27T01:09:26.206 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51641.log.gz 2024-01-27T01:09:26.206 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111051.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.111051.log.gz 2024-01-27T01:09:26.208 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116304.log 2024-01-27T01:09:26.208 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57326.log 2024-01-27T01:09:26.209 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.116304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116352.log 2024-01-27T01:09:26.209 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57326.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.116304.log.gz 2024-01-27T01:09:26.209 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57326.log.gz 2024-01-27T01:09:26.210 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.116352.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.116352.log.gz 2024-01-27T01:09:26.210 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66404.log 2024-01-27T01:09:26.210 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72458.log 2024-01-27T01:09:26.210 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66404.log: 88.8%/var/log/ceph/ceph-client.admin.72458.log: -- replaced with /var/log/ceph/ceph-client.admin.66404.log.gz 2024-01-27T01:09:26.211 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72458.log.gz 2024-01-27T01:09:26.211 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96717.log 2024-01-27T01:09:26.211 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79157.log 2024-01-27T01:09:26.212 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112460.log 2024-01-27T01:09:26.212 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.96717.log.gz 2024-01-27T01:09:26.212 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72525.log 2024-01-27T01:09:26.212 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79157.log.gz 2024-01-27T01:09:26.213 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112460.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.112460.log.gz 2024-01-27T01:09:26.213 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112322.log 2024-01-27T01:09:26.213 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72525.log.gz 2024-01-27T01:09:26.213 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104877.log 2024-01-27T01:09:26.213 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77471.log 2024-01-27T01:09:26.214 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112322.log: /var/log/ceph/ceph-client.admin.104877.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.112322.log.gz 2024-01-27T01:09:26.214 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.104877.log.gz 2024-01-27T01:09:26.214 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77471.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77471.log.gz 2024-01-27T01:09:26.214 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57048.log 2024-01-27T01:09:26.215 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96214.log 2024-01-27T01:09:26.215 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57048.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57048.log.gz 2024-01-27T01:09:26.216 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96214.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96214.log.gz 2024-01-27T01:09:26.216 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90065.log 2024-01-27T01:09:26.216 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124257.log 2024-01-27T01:09:26.217 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90065.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90065.log.gz 2024-01-27T01:09:26.217 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124257.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.124257.log.gz 2024-01-27T01:09:26.217 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105454.log 2024-01-27T01:09:26.217 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120899.log 2024-01-27T01:09:26.218 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105454.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-27T01:09:26.228 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105454.log.gz/var/log/ceph/ceph-client.admin.120899.log: 2024-01-27T01:09:26.228 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.120899.log.gz 2024-01-27T01:09:26.229 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102856.log 2024-01-27T01:09:26.230 INFO:teuthology.orchestra.run.smithi046.stderr:gzip 88.3%-5 -- replaced with /var/log/ceph/ceph.log.gz--verbose 2024-01-27T01:09:26.230 INFO:teuthology.orchestra.run.smithi046.stderr:-- /var/log/ceph/ceph-client.admin.74847.log 2024-01-27T01:09:26.230 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85630.log 2024-01-27T01:09:26.230 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74847.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.102856.log.gz 2024-01-27T01:09:26.230 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.74847.log.gz 2024-01-27T01:09:26.231 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85630.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85630.log.gz 2024-01-27T01:09:26.231 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59798.log 2024-01-27T01:09:26.231 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81340.log 2024-01-27T01:09:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59798.log: /var/log/ceph/ceph-client.admin.81340.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59798.log.gz 2024-01-27T01:09:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116452.log 2024-01-27T01:09:26.232 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.81340.log.gz 2024-01-27T01:09:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114691.log 2024-01-27T01:09:26.233 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99082.log 2024-01-27T01:09:26.233 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.116452.log: /var/log/ceph/ceph-client.admin.114691.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.116452.log.gz 2024-01-27T01:09:26.233 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99082.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.114691.log.gz 2024-01-27T01:09:26.233 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51643.log 2024-01-27T01:09:26.233 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.99082.log.gz 2024-01-27T01:09:26.234 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88211.log 2024-01-27T01:09:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80065.log 2024-01-27T01:09:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51643.log: /var/log/ceph/ceph-client.admin.88211.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51643.log.gz 2024-01-27T01:09:26.235 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0%/var/log/ceph/ceph-client.admin.80065.log: -- replaced with /var/log/ceph/ceph-client.admin.88211.log.gz 2024-01-27T01:09:26.235 INFO:teuthology.orchestra.run.smithi046.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.80065.log.gz 2024-01-27T01:09:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78811.log 2024-01-27T01:09:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71930.log 2024-01-27T01:09:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105732.log 2024-01-27T01:09:26.236 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78811.log.gz 2024-01-27T01:09:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71930.log: /var/log/ceph/ceph-client.admin.105732.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71930.log.gz 2024-01-27T01:09:26.237 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.105732.log.gz 2024-01-27T01:09:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84697.log 2024-01-27T01:09:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101132.log 2024-01-27T01:09:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99897.log 2024-01-27T01:09:26.238 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84697.log.gz 2024-01-27T01:09:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56165.log 2024-01-27T01:09:26.238 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.101132.log.gz 2024-01-27T01:09:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99897.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.99897.log.gz 2024-01-27T01:09:26.245 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106483.log 2024-01-27T01:09:26.245 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113466.log 2024-01-27T01:09:26.246 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.106483.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56165.log.gz 2024-01-27T01:09:26.246 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106483.log.gz 2024-01-27T01:09:26.246 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113466.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113466.log.gz 2024-01-27T01:09:26.246 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60962.log 2024-01-27T01:09:26.247 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53278.log 2024-01-27T01:09:26.247 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60962.log: /var/log/ceph/ceph-client.admin.53278.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60962.log.gz 2024-01-27T01:09:26.248 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53278.log.gz 2024-01-27T01:09:26.248 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49916.log 2024-01-27T01:09:26.248 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86409.log 2024-01-27T01:09:26.249 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49916.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49916.log.gz 2024-01-27T01:09:26.249 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86409.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86409.log.gz 2024-01-27T01:09:26.249 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87537.log 2024-01-27T01:09:26.249 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52011.log 2024-01-27T01:09:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87537.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.87537.log.gz 2024-01-27T01:09:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52011.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52011.log.gz 2024-01-27T01:09:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62745.log 2024-01-27T01:09:26.251 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117573.log 2024-01-27T01:09:26.251 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62745.log: /var/log/ceph/ceph-client.admin.117573.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62745.log.gz 2024-01-27T01:09:26.252 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.117573.log.gz 2024-01-27T01:09:26.252 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115588.log 2024-01-27T01:09:26.252 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77712.log 2024-01-27T01:09:26.253 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115588.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.115588.log.gz 2024-01-27T01:09:26.253 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77712.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.77712.log.gz 2024-01-27T01:09:26.253 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118252.log 2024-01-27T01:09:26.254 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65973.log 2024-01-27T01:09:26.254 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.118252.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.118252.log.gz 2024-01-27T01:09:26.255 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65973.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65973.log.gz 2024-01-27T01:09:26.255 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101224.log 2024-01-27T01:09:26.255 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-01-27T01:09:26.256 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101224.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101224.log.gz 2024-01-27T01:09:26.266 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63663.log 2024-01-27T01:09:26.267 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51390.log 2024-01-27T01:09:26.268 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63663.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63663.log.gz 2024-01-27T01:09:26.268 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51390.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51390.log.gz 2024-01-27T01:09:26.268 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86271.log 2024-01-27T01:09:26.269 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106437.log 2024-01-27T01:09:26.269 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86271.log: /var/log/ceph/ceph-client.admin.106437.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.86271.log.gz 2024-01-27T01:09:26.269 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.106437.log.gz 2024-01-27T01:09:26.270 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87891.log 2024-01-27T01:09:26.271 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87231.log 2024-01-27T01:09:26.271 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87891.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.87891.log.gz 2024-01-27T01:09:26.271 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87231.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87231.log.gz 2024-01-27T01:09:26.271 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97587.log 2024-01-27T01:09:26.272 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114295.log 2024-01-27T01:09:26.272 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97587.log: /var/log/ceph/ceph-client.admin.114295.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97587.log.gz 2024-01-27T01:09:26.272 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.114295.log.gz 2024-01-27T01:09:26.273 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101178.log 2024-01-27T01:09:26.273 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91298.log 2024-01-27T01:09:26.273 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101178.log: /var/log/ceph/ceph-client.admin.91298.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.101178.log.gz 2024-01-27T01:09:26.274 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91298.log.gz 2024-01-27T01:09:26.274 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75850.log 2024-01-27T01:09:26.274 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116822.log 2024-01-27T01:09:26.275 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75850.log: /var/log/ceph/ceph-client.admin.116822.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.75850.log.gz 2024-01-27T01:09:26.275 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.116822.log.gz 2024-01-27T01:09:26.275 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76533.log 2024-01-27T01:09:26.276 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118068.log 2024-01-27T01:09:26.276 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76533.log: /var/log/ceph/ceph-client.admin.118068.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76533.log.gz 2024-01-27T01:09:26.276 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.118068.log.gz 2024-01-27T01:09:26.277 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77758.log 2024-01-27T01:09:26.277 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75804.log 2024-01-27T01:09:26.278 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77758.log: /var/log/ceph/ceph-client.admin.75804.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77758.log.gz 2024-01-27T01:09:26.278 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75804.log.gz 2024-01-27T01:09:26.278 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89045.log 2024-01-27T01:09:26.278 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106772.log 2024-01-27T01:09:26.279 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89045.log: /var/log/ceph/ceph-client.admin.106772.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.89045.log.gz 2024-01-27T01:09:26.279 INFO:teuthology.orchestra.run.smithi046.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.106772.log.gz 2024-01-27T01:09:26.279 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119833.log 2024-01-27T01:09:26.280 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63969.log 2024-01-27T01:09:26.281 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119833.log: /var/log/ceph/ceph-client.admin.63969.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.119833.log.gz 2024-01-27T01:09:26.281 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63969.log.gz 2024-01-27T01:09:26.281 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86547.log 2024-01-27T01:09:26.281 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63357.log 2024-01-27T01:09:26.282 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86547.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-27T01:09:26.282 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63357.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.86547.log.gz 2024-01-27T01:09:26.282 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63357.log.gz 2024-01-27T01:09:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74603.log 2024-01-27T01:09:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78255.log 2024-01-27T01:09:26.285 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74603.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74603.log.gz 2024-01-27T01:09:26.286 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78255.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78255.log.gz 2024-01-27T01:09:26.286 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104063.log 2024-01-27T01:09:26.287 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104063.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.104063.log.gz 2024-01-27T01:09:26.287 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74511.log 2024-01-27T01:09:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74511.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.74511.log.gz 2024-01-27T01:09:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65100.log 2024-01-27T01:09:26.290 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65100.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65100.log.gz 2024-01-27T01:09:26.290 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114445.log 2024-01-27T01:09:26.291 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59660.log 2024-01-27T01:09:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.114445.log: /var/log/ceph/ceph-client.admin.59660.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.114445.log.gz 2024-01-27T01:09:26.292 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59660.log.gz 2024-01-27T01:09:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72412.log 2024-01-27T01:09:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72412.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72412.log.gz 2024-01-27T01:09:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111431.log 2024-01-27T01:09:26.294 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111431.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.111431.log.gz 2024-01-27T01:09:26.294 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112368.log 2024-01-27T01:09:26.296 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112368.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.112368.log.gz 2024-01-27T01:09:26.296 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70651.log 2024-01-27T01:09:26.296 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70651.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70651.log.gz 2024-01-27T01:09:26.297 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100106.log 2024-01-27T01:09:26.298 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100106.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.100106.log.gz 2024-01-27T01:09:26.298 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96855.log 2024-01-27T01:09:26.299 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96855.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.96855.log.gz 2024-01-27T01:09:26.299 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118160.log 2024-01-27T01:09:26.300 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.118160.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.118160.log.gz 2024-01-27T01:09:26.300 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120312.log 2024-01-27T01:09:26.302 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.120312.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.120312.log.gz 2024-01-27T01:09:26.302 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76891.log 2024-01-27T01:09:26.304 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76891.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76891.log.gz 2024-01-27T01:09:26.304 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89636.log 2024-01-27T01:09:26.306 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89636.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.89636.log.gz 2024-01-27T01:09:26.306 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79770.log 2024-01-27T01:09:26.309 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79770.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.79770.log.gz 2024-01-27T01:09:26.309 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54196.log 2024-01-27T01:09:26.310 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54196.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.54196.log.gz 2024-01-27T01:09:26.311 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121683.log 2024-01-27T01:09:26.311 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.121683.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.121683.log.gz 2024-01-27T01:09:26.312 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85926.log 2024-01-27T01:09:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85926.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85926.log.gz 2024-01-27T01:09:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77403.log 2024-01-27T01:09:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77403.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77403.log.gz 2024-01-27T01:09:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120358.log 2024-01-27T01:09:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.120358.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.120358.log.gz 2024-01-27T01:09:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76582.log 2024-01-27T01:09:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76582.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76582.log.gz 2024-01-27T01:09:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94550.log 2024-01-27T01:09:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.94550.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.94550.log.gz 2024-01-27T01:09:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101270.log 2024-01-27T01:09:26.319 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101270.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.101270.log.gz 2024-01-27T01:09:26.319 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85778.log 2024-01-27T01:09:26.320 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85778.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.85778.log.gz 2024-01-27T01:09:26.320 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57464.log 2024-01-27T01:09:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57464.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57464.log.gz 2024-01-27T01:09:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114343.log 2024-01-27T01:09:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.114343.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.114343.log.gz 2024-01-27T01:09:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97771.log 2024-01-27T01:09:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97771.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.97771.log.gz 2024-01-27T01:09:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103170.log 2024-01-27T01:09:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103170.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.103170.log.gz 2024-01-27T01:09:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105870.log 2024-01-27T01:09:26.326 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105870.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.105870.log.gz 2024-01-27T01:09:26.326 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88680.log 2024-01-27T01:09:26.327 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88680.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88680.log.gz 2024-01-27T01:09:26.327 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117527.log 2024-01-27T01:09:26.328 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.117527.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.117527.log.gz 2024-01-27T01:09:26.328 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52338.log 2024-01-27T01:09:26.329 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52338.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52338.log.gz 2024-01-27T01:09:26.329 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117435.log 2024-01-27T01:09:26.330 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.117435.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.117435.log.gz 2024-01-27T01:09:26.330 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65207.log 2024-01-27T01:09:26.331 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65207.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65207.log.gz 2024-01-27T01:09:26.332 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65305.log 2024-01-27T01:09:26.333 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65305.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65305.log.gz 2024-01-27T01:09:26.333 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.98304.log 2024-01-27T01:09:26.335 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83071.log 2024-01-27T01:09:26.335 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.98304.log: /var/log/ceph/ceph-client.admin.83071.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83071.log.gz 2024-01-27T01:09:26.335 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57821.log 2024-01-27T01:09:26.336 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57821.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57821.log.gz 2024-01-27T01:09:26.336 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61319.log 2024-01-27T01:09:26.336 INFO:teuthology.orchestra.run.smithi046.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.0.98304.log.gz 2024-01-27T01:09:26.337 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61319.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61319.log.gz 2024-01-27T01:09:26.337 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111477.log 2024-01-27T01:09:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100670.log 2024-01-27T01:09:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111477.log: /var/log/ceph/ceph-client.admin.100670.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.111477.log.gz 2024-01-27T01:09:26.339 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.100670.log.gz 2024-01-27T01:09:26.339 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81562.log 2024-01-27T01:09:26.340 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81562.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81562.log.gz 2024-01-27T01:09:26.340 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67201.log 2024-01-27T01:09:26.341 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67201.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67201.log.gz 2024-01-27T01:09:26.341 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97449.log 2024-01-27T01:09:26.342 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110957.log 2024-01-27T01:09:26.343 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97449.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.97449.log.gz 2024-01-27T01:09:26.343 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110957.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.110957.log.gz 2024-01-27T01:09:26.343 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121041.log 2024-01-27T01:09:26.344 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.121041.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.121041.log.gz 2024-01-27T01:09:26.344 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53232.log 2024-01-27T01:09:26.345 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53232.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53232.log.gz 2024-01-27T01:09:26.345 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110911.log 2024-01-27T01:09:26.347 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110911.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110911.log.gz 2024-01-27T01:09:26.347 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80595.log 2024-01-27T01:09:26.348 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80595.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80595.log.gz 2024-01-27T01:09:26.348 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103959.log 2024-01-27T01:09:26.349 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103959.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.103959.log.gz 2024-01-27T01:09:26.350 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115634.log 2024-01-27T01:09:26.351 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80891.log 2024-01-27T01:09:26.351 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115634.log: /var/log/ceph/ceph-client.admin.80891.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.115634.log.gz 2024-01-27T01:09:26.351 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80891.log.gz 2024-01-27T01:09:26.351 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120993.log 2024-01-27T01:09:26.352 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.120993.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.120993.log.gz 2024-01-27T01:09:26.352 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96062.log 2024-01-27T01:09:26.353 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96062.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96062.log.gz 2024-01-27T01:09:26.353 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114737.log 2024-01-27T01:09:26.355 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.114737.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.114737.log.gz 2024-01-27T01:09:26.355 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105145.log 2024-01-27T01:09:26.356 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105145.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105145.log.gz 2024-01-27T01:09:26.356 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119741.log 2024-01-27T01:09:26.357 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119741.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.119741.log.gz 2024-01-27T01:09:26.358 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59844.log 2024-01-27T01:09:26.359 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59844.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59844.log.gz 2024-01-27T01:09:26.359 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122296.log 2024-01-27T01:09:26.360 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.122296.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.122296.log.gz 2024-01-27T01:09:26.360 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49870.log 2024-01-27T01:09:26.361 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49870.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49870.log.gz 2024-01-27T01:09:26.361 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104761.log 2024-01-27T01:09:26.362 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104761.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.104761.log.gz 2024-01-27T01:09:26.362 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112750.log 2024-01-27T01:09:26.363 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112750.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112750.log.gz 2024-01-27T01:09:26.363 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.101665.log 2024-01-27T01:09:26.368 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.101665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49684.log 2024-01-27T01:09:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49684.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49684.log.gz 2024-01-27T01:09:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114137.log 2024-01-27T01:09:26.389 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.114137.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.114137.log.gz 2024-01-27T01:09:26.389 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114494.log 2024-01-27T01:09:26.390 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.114494.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.114494.log.gz 2024-01-27T01:09:26.390 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67063.log 2024-01-27T01:09:26.391 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67063.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67063.log.gz 2024-01-27T01:09:26.391 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70905.log 2024-01-27T01:09:26.393 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70905.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70905.log.gz 2024-01-27T01:09:26.393 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111153.log 2024-01-27T01:09:26.394 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111153.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.111153.log.gz 2024-01-27T01:09:26.394 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82366.log 2024-01-27T01:09:26.395 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82366.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82366.log.gz 2024-01-27T01:09:26.395 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72274.log 2024-01-27T01:09:26.397 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72274.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72274.log.gz 2024-01-27T01:09:26.397 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109326.log 2024-01-27T01:09:26.398 INFO:teuthology.orchestra.run.smithi046.stderr: 95.1% -- replaced with /var/log/ceph/ceph-client.0.101665.log.gz 2024-01-27T01:09:26.398 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109326.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.109326.log.gz 2024-01-27T01:09:26.398 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123131.log 2024-01-27T01:09:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.123131.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.123131.log.gz 2024-01-27T01:09:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94504.log 2024-01-27T01:09:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.94504.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.94504.log.gz 2024-01-27T01:09:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77201.log 2024-01-27T01:09:26.401 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77201.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77201.log.gz 2024-01-27T01:09:26.401 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124583.log 2024-01-27T01:09:26.403 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124583.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.124583.log.gz 2024-01-27T01:09:26.403 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85730.log 2024-01-27T01:09:26.403 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86020.log 2024-01-27T01:09:26.405 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85730.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85730.log.gz 2024-01-27T01:09:26.405 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86020.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86020.log.gz 2024-01-27T01:09:26.405 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50720.log 2024-01-27T01:09:26.405 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96625.log 2024-01-27T01:09:26.406 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50720.log: /var/log/ceph/ceph-client.admin.96625.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96625.log.gz 2024-01-27T01:09:26.406 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52910.log 2024-01-27T01:09:26.408 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52910.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52910.log.gz 2024-01-27T01:09:26.408 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53370.log 2024-01-27T01:09:26.408 INFO:teuthology.orchestra.run.smithi046.stderr: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.50720.log.gz 2024-01-27T01:09:26.409 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53370.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.53370.log.gz 2024-01-27T01:09:26.409 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-27T01:09:26.410 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66925.log 2024-01-27T01:09:26.411 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.66925.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.66925.log.gz 2024-01-27T01:09:26.411 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102003.log 2024-01-27T01:09:26.412 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102003.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.102003.log.gz 2024-01-27T01:09:26.413 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57510.log 2024-01-27T01:09:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57510.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57510.log.gz 2024-01-27T01:09:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95904.log 2024-01-27T01:09:26.415 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95904.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.95904.log.gz 2024-01-27T01:09:26.415 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67980.log 2024-01-27T01:09:26.416 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67980.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67980.log.gz 2024-01-27T01:09:26.416 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84845.log 2024-01-27T01:09:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84845.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84845.log.gz 2024-01-27T01:09:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96261.log 2024-01-27T01:09:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96261.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.96261.log.gz 2024-01-27T01:09:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87735.log 2024-01-27T01:09:26.420 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87735.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87735.log.gz 2024-01-27T01:09:26.420 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71315.log 2024-01-27T01:09:26.422 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71315.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71315.log.gz 2024-01-27T01:09:26.422 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85107.log 2024-01-27T01:09:26.423 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85107.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.85107.log.gz 2024-01-27T01:09:26.423 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89590.log 2024-01-27T01:09:26.424 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89590.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.89590.log.gz 2024-01-27T01:09:26.424 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78161.log 2024-01-27T01:09:26.425 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78161.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78161.log.gz 2024-01-27T01:09:26.426 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81654.log 2024-01-27T01:09:26.427 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81654.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81654.log.gz 2024-01-27T01:09:26.427 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78301.log 2024-01-27T01:09:26.428 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78301.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.78301.log.gz 2024-01-27T01:09:26.428 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124016.log 2024-01-27T01:09:26.429 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124016.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.124016.log.gz 2024-01-27T01:09:26.429 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84500.log 2024-01-27T01:09:26.431 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84500.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.84500.log.gz 2024-01-27T01:09:26.431 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125043.log 2024-01-27T01:09:26.432 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.125043.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.125043.log.gz 2024-01-27T01:09:26.432 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118484.log 2024-01-27T01:09:26.433 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.118484.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.118484.log.gz 2024-01-27T01:09:26.433 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68052.log 2024-01-27T01:09:26.435 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68052.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68052.log.gz 2024-01-27T01:09:26.435 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104485.log 2024-01-27T01:09:26.436 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104485.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.104485.log.gz 2024-01-27T01:09:26.436 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123897.log 2024-01-27T01:09:26.437 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.123897.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.123897.log.gz 2024-01-27T01:09:26.437 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98811.log 2024-01-27T01:09:26.439 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.98811.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.98811.log.gz 2024-01-27T01:09:26.439 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75666.log 2024-01-27T01:09:26.440 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75666.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75666.log.gz 2024-01-27T01:09:26.440 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105053.log 2024-01-27T01:09:26.441 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105053.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.105053.log.gz 2024-01-27T01:09:26.441 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51344.log 2024-01-27T01:09:26.442 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51344.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51344.log.gz 2024-01-27T01:09:26.443 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51992.log 2024-01-27T01:09:26.444 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51992.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51992.log.gz 2024-01-27T01:09:26.444 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96359.log 2024-01-27T01:09:26.445 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96359.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96359.log.gz 2024-01-27T01:09:26.445 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96110.log 2024-01-27T01:09:26.446 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96110.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96110.log.gz 2024-01-27T01:09:26.447 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64581.log 2024-01-27T01:09:26.448 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64581.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64581.log.gz 2024-01-27T01:09:26.448 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116938.log 2024-01-27T01:09:26.449 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.116938.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.116938.log.gz 2024-01-27T01:09:26.449 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111686.log 2024-01-27T01:09:26.451 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111686.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.111686.log.gz 2024-01-27T01:09:26.451 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84392.log 2024-01-27T01:09:26.452 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84392.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.84392.log.gz 2024-01-27T01:09:26.452 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114590.log 2024-01-27T01:09:26.453 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.114590.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.114590.log.gz 2024-01-27T01:09:26.453 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77149.log 2024-01-27T01:09:26.454 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77149.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.77149.log.gz 2024-01-27T01:09:26.455 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103908.log 2024-01-27T01:09:26.456 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103908.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.103908.log.gz 2024-01-27T01:09:26.456 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112656.log 2024-01-27T01:09:26.457 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112656.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112656.log.gz 2024-01-27T01:09:26.457 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67814.log 2024-01-27T01:09:26.459 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67814.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67814.log.gz 2024-01-27T01:09:26.459 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60686.log 2024-01-27T01:09:26.460 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60686.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60686.log.gz 2024-01-27T01:09:26.460 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115986.log 2024-01-27T01:09:26.461 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115986.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.115986.log.gz 2024-01-27T01:09:26.461 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55919.log 2024-01-27T01:09:26.462 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55919.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55919.log.gz 2024-01-27T01:09:26.462 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96926.log 2024-01-27T01:09:26.464 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96926.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96926.log.gz 2024-01-27T01:09:26.464 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82574.log 2024-01-27T01:09:26.465 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82574.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82574.log.gz 2024-01-27T01:09:26.465 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.112010.log 2024-01-27T01:09:26.469 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.112010.log: 92.0% -- replaced with /var/log/ceph/ceph-client.0.112010.log.gz 2024-01-27T01:09:26.469 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80111.log 2024-01-27T01:09:26.470 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80111.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.80111.log.gz 2024-01-27T01:09:26.470 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86179.log 2024-01-27T01:09:26.471 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86179.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86179.log.gz 2024-01-27T01:09:26.471 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119553.log 2024-01-27T01:09:26.472 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119553.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.119553.log.gz 2024-01-27T01:09:26.472 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91119.log 2024-01-27T01:09:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91119.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.91119.log.gz 2024-01-27T01:09:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76943.log 2024-01-27T01:09:26.475 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76943.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76943.log.gz 2024-01-27T01:09:26.475 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59350.log 2024-01-27T01:09:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59350.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59350.log.gz 2024-01-27T01:09:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72571.log 2024-01-27T01:09:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72571.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.72571.log.gz 2024-01-27T01:09:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91344.log 2024-01-27T01:09:26.480 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91344.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91344.log.gz 2024-01-27T01:09:26.480 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100466.log 2024-01-27T01:09:26.481 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100466.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.100466.log.gz 2024-01-27T01:09:26.481 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55459.log 2024-01-27T01:09:26.482 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55459.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55459.log.gz 2024-01-27T01:09:26.482 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104015.log 2024-01-27T01:09:26.484 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104015.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.104015.log.gz 2024-01-27T01:09:26.484 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90203.log 2024-01-27T01:09:26.485 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90203.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90203.log.gz 2024-01-27T01:09:26.485 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58051.log 2024-01-27T01:09:26.486 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58051.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58051.log.gz 2024-01-27T01:09:26.486 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113347.log 2024-01-27T01:09:26.487 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113347.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113347.log.gz 2024-01-27T01:09:26.488 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51642.log 2024-01-27T01:09:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51642.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51642.log.gz 2024-01-27T01:09:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119925.log 2024-01-27T01:09:26.490 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119925.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.119925.log.gz 2024-01-27T01:09:26.490 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51640.log 2024-01-27T01:09:26.491 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51640.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51640.log.gz 2024-01-27T01:09:26.491 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99738.log 2024-01-27T01:09:26.493 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99738.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.99738.log.gz 2024-01-27T01:09:26.493 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81177.log 2024-01-27T01:09:26.494 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81177.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81177.log.gz 2024-01-27T01:09:26.494 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75129.log 2024-01-27T01:09:26.496 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75129.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.75129.log.gz 2024-01-27T01:09:26.496 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102214.log 2024-01-27T01:09:26.497 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102214.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.102214.log.gz 2024-01-27T01:09:26.497 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99340.log 2024-01-27T01:09:26.498 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99340.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.99340.log.gz 2024-01-27T01:09:26.498 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111293.log 2024-01-27T01:09:26.500 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111293.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.111293.log.gz 2024-01-27T01:09:26.500 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120633.log 2024-01-27T01:09:26.501 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.120633.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.120633.log.gz 2024-01-27T01:09:26.501 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90387.log 2024-01-27T01:09:26.503 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74059.log 2024-01-27T01:09:26.503 INFO:teuthology.orchestra.run.smithi046.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.90387.log.gz 2024-01-27T01:09:26.504 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74059.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74059.log.gz 2024-01-27T01:09:26.504 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85826.log 2024-01-27T01:09:26.506 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85826.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85826.log.gz 2024-01-27T01:09:26.506 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125523.log 2024-01-27T01:09:26.507 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.125523.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.125523.log.gz 2024-01-27T01:09:26.507 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73492.log 2024-01-27T01:09:26.508 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73492.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73492.log.gz 2024-01-27T01:09:26.508 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-27T01:09:26.512 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71791.log 2024-01-27T01:09:26.517 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-27T01:09:26.518 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71791.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71791.log.gz 2024-01-27T01:09:26.518 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49019.log 2024-01-27T01:09:26.520 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49019.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49019.log.gz 2024-01-27T01:09:26.520 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100994.log 2024-01-27T01:09:26.521 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100994.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.100994.log.gz 2024-01-27T01:09:26.521 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100810.log 2024-01-27T01:09:26.522 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100810.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.100810.log.gz 2024-01-27T01:09:26.522 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82458.log 2024-01-27T01:09:26.523 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82458.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.82458.log.gz 2024-01-27T01:09:26.524 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118605.log 2024-01-27T01:09:26.525 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.118605.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.118605.log.gz 2024-01-27T01:09:26.525 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74107.log 2024-01-27T01:09:26.526 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74107.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74107.log.gz 2024-01-27T01:09:26.526 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88577.log 2024-01-27T01:09:26.527 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88577.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88577.log.gz 2024-01-27T01:09:26.527 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114858.log 2024-01-27T01:09:26.529 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.114858.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.114858.log.gz 2024-01-27T01:09:26.529 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115896.log 2024-01-27T01:09:26.530 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115896.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.115896.log.gz 2024-01-27T01:09:26.530 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52336.log 2024-01-27T01:09:26.531 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52336.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52336.log.gz 2024-01-27T01:09:26.531 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90017.log 2024-01-27T01:09:26.532 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90017.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90017.log.gz 2024-01-27T01:09:26.533 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78207.log 2024-01-27T01:09:26.534 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78207.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78207.log.gz 2024-01-27T01:09:26.534 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121521.log 2024-01-27T01:09:26.535 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.121521.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.121521.log.gz 2024-01-27T01:09:26.535 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116162.log 2024-01-27T01:09:26.536 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.116162.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.116162.log.gz 2024-01-27T01:09:26.537 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61583.log 2024-01-27T01:09:26.538 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61583.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61583.log.gz 2024-01-27T01:09:26.538 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88874.log 2024-01-27T01:09:26.539 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88874.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.88874.log.gz 2024-01-27T01:09:26.539 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79700.log 2024-01-27T01:09:26.540 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79700.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79700.log.gz 2024-01-27T01:09:26.540 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66879.log 2024-01-27T01:09:26.542 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66879.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66879.log.gz 2024-01-27T01:09:26.542 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55413.log 2024-01-27T01:09:26.543 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55413.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55413.log.gz 2024-01-27T01:09:26.543 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49534.log 2024-01-27T01:09:26.544 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49534.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49534.log.gz 2024-01-27T01:09:26.544 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114395.log 2024-01-27T01:09:26.545 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.114395.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.114395.log.gz 2024-01-27T01:09:26.546 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51991.log 2024-01-27T01:09:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51991.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51991.log.gz 2024-01-27T01:09:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113774.log 2024-01-27T01:09:26.548 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113774.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.113774.log.gz 2024-01-27T01:09:26.548 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105333.log 2024-01-27T01:09:26.549 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105333.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.105333.log.gz 2024-01-27T01:09:26.549 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87846.log 2024-01-27T01:09:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87846.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87846.log.gz 2024-01-27T01:09:26.551 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82412.log 2024-01-27T01:09:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82412.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82412.log.gz 2024-01-27T01:09:26.552 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123731.log 2024-01-27T01:09:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.123731.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.123731.log.gz 2024-01-27T01:09:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113231.log 2024-01-27T01:09:26.555 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113231.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.113231.log.gz 2024-01-27T01:09:26.555 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55321.log 2024-01-27T01:09:26.557 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55321.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55321.log.gz 2024-01-27T01:09:26.557 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125479.log 2024-01-27T01:09:26.558 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.125479.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.125479.log.gz 2024-01-27T01:09:26.558 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118368.log 2024-01-27T01:09:26.559 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.118368.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.118368.log.gz 2024-01-27T01:09:26.559 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96579.log 2024-01-27T01:09:26.561 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96579.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.96579.log.gz 2024-01-27T01:09:26.561 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116546.log 2024-01-27T01:09:26.562 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.116546.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116546.log.gz 2024-01-27T01:09:26.562 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59706.log 2024-01-27T01:09:26.563 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59706.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59706.log.gz 2024-01-27T01:09:26.564 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109902.log 2024-01-27T01:09:26.565 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109902.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109902.log.gz 2024-01-27T01:09:26.565 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96007.log 2024-01-27T01:09:26.566 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96007.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.96007.log.gz 2024-01-27T01:09:26.566 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113417.log 2024-01-27T01:09:26.567 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113417.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113417.log.gz 2024-01-27T01:09:26.567 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114996.log 2024-01-27T01:09:26.569 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.114996.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.114996.log.gz 2024-01-27T01:09:26.569 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52634.log 2024-01-27T01:09:26.570 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52634.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52634.log.gz 2024-01-27T01:09:26.570 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80157.log 2024-01-27T01:09:26.571 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80157.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80157.log.gz 2024-01-27T01:09:26.571 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120266.log 2024-01-27T01:09:26.573 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.120266.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.120266.log.gz 2024-01-27T01:09:26.573 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57372.log 2024-01-27T01:09:26.574 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57372.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57372.log.gz 2024-01-27T01:09:26.574 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66090.log 2024-01-27T01:09:26.575 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66090.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66090.log.gz 2024-01-27T01:09:26.575 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77298.log 2024-01-27T01:09:26.577 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77298.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77298.log.gz 2024-01-27T01:09:26.577 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85061.log 2024-01-27T01:09:26.578 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85061.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.85061.log.gz 2024-01-27T01:09:26.578 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82266.log 2024-01-27T01:09:26.579 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82266.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.82266.log.gz 2024-01-27T01:09:26.579 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116638.log 2024-01-27T01:09:26.581 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.116638.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.116638.log.gz 2024-01-27T01:09:26.581 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71978.log 2024-01-27T01:09:26.582 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71978.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71978.log.gz 2024-01-27T01:09:26.582 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66647.log 2024-01-27T01:09:26.583 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66647.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66647.log.gz 2024-01-27T01:09:26.583 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103856.log 2024-01-27T01:09:26.585 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103856.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.103856.log.gz 2024-01-27T01:09:26.585 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113538.log 2024-01-27T01:09:26.586 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113538.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.113538.log.gz 2024-01-27T01:09:26.586 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80792.log 2024-01-27T01:09:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80792.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.80792.log.gz 2024-01-27T01:09:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113912.log 2024-01-27T01:09:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113912.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113912.log.gz 2024-01-27T01:09:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74156.log 2024-01-27T01:09:26.590 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74156.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74156.log.gz 2024-01-27T01:09:26.590 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89112.log 2024-01-27T01:09:26.591 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89112.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89112.log.gz 2024-01-27T01:09:26.591 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66546.log 2024-01-27T01:09:26.592 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66546.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66546.log.gz 2024-01-27T01:09:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87603.log 2024-01-27T01:09:26.594 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87603.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87603.log.gz 2024-01-27T01:09:26.594 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52680.log 2024-01-27T01:09:26.595 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52680.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52680.log.gz 2024-01-27T01:09:26.595 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97056.log 2024-01-27T01:09:26.597 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97056.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.97056.log.gz 2024-01-27T01:09:26.597 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99851.log 2024-01-27T01:09:26.598 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99851.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.99851.log.gz 2024-01-27T01:09:26.598 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124211.log 2024-01-27T01:09:26.599 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124211.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.124211.log.gz 2024-01-27T01:09:26.599 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124629.log 2024-01-27T01:09:26.601 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124629.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.124629.log.gz 2024-01-27T01:09:26.601 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89158.log 2024-01-27T01:09:26.602 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89158.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.89158.log.gz 2024-01-27T01:09:26.602 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121151.log 2024-01-27T01:09:26.603 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.121151.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.121151.log.gz 2024-01-27T01:09:26.603 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56011.log 2024-01-27T01:09:26.604 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56011.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56011.log.gz 2024-01-27T01:09:26.605 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51968.log 2024-01-27T01:09:26.606 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51968.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51968.log.gz 2024-01-27T01:09:26.606 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124813.log 2024-01-27T01:09:26.607 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124813.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.124813.log.gz 2024-01-27T01:09:26.607 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116116.log 2024-01-27T01:09:26.608 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.116116.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.116116.log.gz 2024-01-27T01:09:26.608 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65151.log 2024-01-27T01:09:26.610 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65151.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65151.log.gz 2024-01-27T01:09:26.610 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109592.log 2024-01-27T01:09:26.611 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109592.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.109592.log.gz 2024-01-27T01:09:26.611 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96763.log 2024-01-27T01:09:26.612 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96763.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96763.log.gz 2024-01-27T01:09:26.612 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97633.log 2024-01-27T01:09:26.614 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97633.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97633.log.gz 2024-01-27T01:09:26.614 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113301.log 2024-01-27T01:09:26.615 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113301.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113301.log.gz 2024-01-27T01:09:26.615 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82312.log 2024-01-27T01:09:26.616 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82312.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82312.log.gz 2024-01-27T01:09:26.616 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84443.log 2024-01-27T01:09:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84443.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84443.log.gz 2024-01-27T01:09:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89823.log 2024-01-27T01:09:26.619 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89823.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.89823.log.gz 2024-01-27T01:09:26.619 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100326.log 2024-01-27T01:09:26.620 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100326.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.100326.log.gz 2024-01-27T01:09:26.620 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.82900.log 2024-01-27T01:09:26.626 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.82900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110164.log 2024-01-27T01:09:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110164.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.110164.log.gz 2024-01-27T01:09:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58213.log 2024-01-27T01:09:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58213.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58213.log.gz 2024-01-27T01:09:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73184.log 2024-01-27T01:09:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73184.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.73184.log.gz 2024-01-27T01:09:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81748.log 2024-01-27T01:09:26.649 INFO:teuthology.orchestra.run.smithi046.stderr: 94.4% -- replaced with /var/log/ceph/ceph-client.0.82900.log.gz 2024-01-27T01:09:26.651 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81748.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81748.log.gz 2024-01-27T01:09:26.651 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85153.log 2024-01-27T01:09:26.652 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85153.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.85153.log.gz 2024-01-27T01:09:26.652 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91070.log 2024-01-27T01:09:26.653 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91070.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91070.log.gz 2024-01-27T01:09:26.653 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75436.log 2024-01-27T01:09:26.655 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75436.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75436.log.gz 2024-01-27T01:09:26.655 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97863.log 2024-01-27T01:09:26.656 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97863.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.97863.log.gz 2024-01-27T01:09:26.656 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101883.log 2024-01-27T01:09:26.657 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101883.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.101883.log.gz 2024-01-27T01:09:26.657 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89871.log 2024-01-27T01:09:26.658 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89871.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.89871.log.gz 2024-01-27T01:09:26.659 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84075.log 2024-01-27T01:09:26.660 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84075.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84075.log.gz 2024-01-27T01:09:26.660 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97679.log 2024-01-27T01:09:26.661 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97679.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.97679.log.gz 2024-01-27T01:09:26.661 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102098.log 2024-01-27T01:09:26.662 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102098.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.102098.log.gz 2024-01-27T01:09:26.663 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97307.log 2024-01-27T01:09:26.664 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97307.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.97307.log.gz 2024-01-27T01:09:26.664 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62133.log 2024-01-27T01:09:26.665 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62133.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62133.log.gz 2024-01-27T01:09:26.665 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.37079.log 2024-01-27T01:09:26.666 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.tmp-client.admin.37079.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.37079.log.gz 2024-01-27T01:09:26.666 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80476.log 2024-01-27T01:09:26.667 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80476.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80476.log.gz 2024-01-27T01:09:26.668 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124675.log 2024-01-27T01:09:26.669 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124675.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.124675.log.gz 2024-01-27T01:09:26.669 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82220.log 2024-01-27T01:09:26.670 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82220.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82220.log.gz 2024-01-27T01:09:26.670 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51972.log 2024-01-27T01:09:26.671 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51972.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51972.log.gz 2024-01-27T01:09:26.671 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51113.log 2024-01-27T01:09:26.673 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51113.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51113.log.gz 2024-01-27T01:09:26.673 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49582.log 2024-01-27T01:09:26.674 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49582.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49582.log.gz 2024-01-27T01:09:26.675 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66498.log 2024-01-27T01:09:26.675 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66498.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66498.log.gz 2024-01-27T01:09:26.676 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53140.log 2024-01-27T01:09:26.677 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53140.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53140.log.gz 2024-01-27T01:09:26.677 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71177.log 2024-01-27T01:09:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71177.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71177.log.gz 2024-01-27T01:09:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97817.log 2024-01-27T01:09:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97817.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97817.log.gz 2024-01-27T01:09:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51988.log 2024-01-27T01:09:26.681 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51988.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51988.log.gz 2024-01-27T01:09:26.681 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65357.log 2024-01-27T01:09:26.682 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65357.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65357.log.gz 2024-01-27T01:09:26.682 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104263.log 2024-01-27T01:09:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104263.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.104263.log.gz 2024-01-27T01:09:26.684 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78627.log 2024-01-27T01:09:26.685 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78627.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.78627.log.gz 2024-01-27T01:09:26.685 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52331.log 2024-01-27T01:09:26.686 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52331.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52331.log.gz 2024-01-27T01:09:26.686 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100764.log 2024-01-27T01:09:26.687 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100764.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100764.log.gz 2024-01-27T01:09:26.687 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66600.log 2024-01-27T01:09:26.689 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66600.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66600.log.gz 2024-01-27T01:09:26.689 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120220.log 2024-01-27T01:09:26.690 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.120220.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.120220.log.gz 2024-01-27T01:09:26.690 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76463.log 2024-01-27T01:09:26.691 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76463.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76463.log.gz 2024-01-27T01:09:26.692 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114543.log 2024-01-27T01:09:26.693 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.114543.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.114543.log.gz 2024-01-27T01:09:26.693 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79871.log 2024-01-27T01:09:26.694 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79871.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79871.log.gz 2024-01-27T01:09:26.694 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72134.log 2024-01-27T01:09:26.695 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72134.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72134.log.gz 2024-01-27T01:09:26.696 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79819.log 2024-01-27T01:09:26.697 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79819.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79819.log.gz 2024-01-27T01:09:26.697 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104360.log 2024-01-27T01:09:26.698 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104360.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.104360.log.gz 2024-01-27T01:09:26.698 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100902.log 2024-01-27T01:09:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100902.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.100902.log.gz 2024-01-27T01:09:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73755.log 2024-01-27T01:09:26.701 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73755.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73755.log.gz 2024-01-27T01:09:26.701 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63051.log 2024-01-27T01:09:26.702 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63051.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63051.log.gz 2024-01-27T01:09:26.702 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51638.log 2024-01-27T01:09:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51638.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51638.log.gz 2024-01-27T01:09:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71453.log 2024-01-27T01:09:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71453.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71453.log.gz 2024-01-27T01:09:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89342.log 2024-01-27T01:09:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89342.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.89342.log.gz 2024-01-27T01:09:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57558.log 2024-01-27T01:09:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57558.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57558.log.gz 2024-01-27T01:09:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105640.log 2024-01-27T01:09:26.709 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105640.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.105640.log.gz 2024-01-27T01:09:26.709 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120173.log 2024-01-27T01:09:26.710 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.120173.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.120173.log.gz 2024-01-27T01:09:26.710 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75390.log 2024-01-27T01:09:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75390.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75390.log.gz 2024-01-27T01:09:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56874.log 2024-01-27T01:09:26.713 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56874.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56874.log.gz 2024-01-27T01:09:26.713 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88942.log 2024-01-27T01:09:26.714 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88942.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.88942.log.gz 2024-01-27T01:09:26.714 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122634.log 2024-01-27T01:09:26.715 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.122634.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.122634.log.gz 2024-01-27T01:09:26.715 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103282.log 2024-01-27T01:09:26.716 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103282.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.103282.log.gz 2024-01-27T01:09:26.716 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72228.log 2024-01-27T01:09:26.718 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72228.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72228.log.gz 2024-01-27T01:09:26.718 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125089.log 2024-01-27T01:09:26.719 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.125089.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.125089.log.gz 2024-01-27T01:09:26.719 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112414.log 2024-01-27T01:09:26.720 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112414.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.112414.log.gz 2024-01-27T01:09:26.720 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89925.log 2024-01-27T01:09:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89925.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.89925.log.gz 2024-01-27T01:09:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113047.log 2024-01-27T01:09:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113047.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.113047.log.gz 2024-01-27T01:09:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100514.log 2024-01-27T01:09:26.724 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100514.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.100514.log.gz 2024-01-27T01:09:26.724 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65255.log 2024-01-27T01:09:26.725 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65255.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65255.log.gz 2024-01-27T01:09:26.726 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83329.log 2024-01-27T01:09:26.727 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.83329.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83329.log.gz 2024-01-27T01:09:26.727 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51636.log 2024-01-27T01:09:26.728 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51636.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51636.log.gz 2024-01-27T01:09:26.728 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61649.log 2024-01-27T01:09:26.729 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61649.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61649.log.gz 2024-01-27T01:09:26.730 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119603.log 2024-01-27T01:09:26.731 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119603.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.119603.log.gz 2024-01-27T01:09:26.731 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67017.log 2024-01-27T01:09:26.732 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67017.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67017.log.gz 2024-01-27T01:09:26.732 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113682.log 2024-01-27T01:09:26.733 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113682.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113682.log.gz 2024-01-27T01:09:26.733 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56395.log 2024-01-27T01:09:26.735 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56395.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56395.log.gz 2024-01-27T01:09:26.735 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100280.log 2024-01-27T01:09:26.736 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100280.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100280.log.gz 2024-01-27T01:09:26.736 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100196.log 2024-01-27T01:09:26.737 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100196.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.100196.log.gz 2024-01-27T01:09:26.737 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66180.log 2024-01-27T01:09:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66180.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66180.log.gz 2024-01-27T01:09:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67155.log 2024-01-27T01:09:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67155.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67155.log.gz 2024-01-27T01:09:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52380.log 2024-01-27T01:09:26.741 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52380.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52380.log.gz 2024-01-27T01:09:26.741 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106602.log 2024-01-27T01:09:26.742 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.106602.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.106602.log.gz 2024-01-27T01:09:26.743 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58005.log 2024-01-27T01:09:26.744 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58005.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58005.log.gz 2024-01-27T01:09:26.744 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118206.log 2024-01-27T01:09:26.745 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.118206.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.118206.log.gz 2024-01-27T01:09:26.745 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106674.log 2024-01-27T01:09:26.747 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.106674.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.106674.log.gz 2024-01-27T01:09:26.747 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76731.log 2024-01-27T01:09:26.748 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76731.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.76731.log.gz 2024-01-27T01:09:26.748 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110451.log 2024-01-27T01:09:26.749 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110451.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110451.log.gz 2024-01-27T01:09:26.749 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123947.log 2024-01-27T01:09:26.751 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.123947.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.123947.log.gz 2024-01-27T01:09:26.751 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117807.log 2024-01-27T01:09:26.752 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.117807.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.117807.log.gz 2024-01-27T01:09:26.752 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104623.log 2024-01-27T01:09:26.753 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104623.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104623.log.gz 2024-01-27T01:09:26.753 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113093.log 2024-01-27T01:09:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113093.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.113093.log.gz 2024-01-27T01:09:26.755 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75712.log 2024-01-27T01:09:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75712.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75712.log.gz 2024-01-27T01:09:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.54719.log 2024-01-27T01:09:26.758 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.2.54719.log: 90.3% -- replaced with /var/log/ceph/ceph-client.2.54719.log.gz 2024-01-27T01:09:26.758 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119695.log 2024-01-27T01:09:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119695.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.119695.log.gz 2024-01-27T01:09:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124445.log 2024-01-27T01:09:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124445.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.124445.log.gz 2024-01-27T01:09:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71269.log 2024-01-27T01:09:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71269.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71269.log.gz 2024-01-27T01:09:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.76245.log 2024-01-27T01:09:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.76245.log: 89.8% -- replaced with /var/log/ceph/ceph-client.0.76245.log.gz 2024-01-27T01:09:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121337.log 2024-01-27T01:09:26.764 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.121337.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.121337.log.gz 2024-01-27T01:09:26.765 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50975.log 2024-01-27T01:09:26.766 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50975.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50975.log.gz 2024-01-27T01:09:26.766 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75081.log 2024-01-27T01:09:26.767 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75081.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75081.log.gz 2024-01-27T01:09:26.767 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111524.log 2024-01-27T01:09:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111524.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.111524.log.gz 2024-01-27T01:09:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70858.log 2024-01-27T01:09:26.770 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70858.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70858.log.gz 2024-01-27T01:09:26.770 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62439.log 2024-01-27T01:09:26.771 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62439.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.62439.log.gz 2024-01-27T01:09:26.771 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104531.log 2024-01-27T01:09:26.772 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104531.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.104531.log.gz 2024-01-27T01:09:26.772 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109642.log 2024-01-27T01:09:26.774 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109642.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.109642.log.gz 2024-01-27T01:09:26.774 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89415.log 2024-01-27T01:09:26.775 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89415.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.89415.log.gz 2024-01-27T01:09:26.775 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72320.log 2024-01-27T01:09:26.776 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72320.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72320.log.gz 2024-01-27T01:09:26.776 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124951.log 2024-01-27T01:09:26.778 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124951.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.124951.log.gz 2024-01-27T01:09:26.778 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58710.log 2024-01-27T01:09:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58710.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58710.log.gz 2024-01-27T01:09:26.779 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117861.log 2024-01-27T01:09:26.780 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.117861.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.117861.log.gz 2024-01-27T01:09:26.780 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65404.log 2024-01-27T01:09:26.781 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65404.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65404.log.gz 2024-01-27T01:09:26.782 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100420.log 2024-01-27T01:09:26.783 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100420.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.100420.log.gz 2024-01-27T01:09:26.783 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110635.log 2024-01-27T01:09:26.784 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110635.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.110635.log.gz 2024-01-27T01:09:26.784 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101086.log 2024-01-27T01:09:26.785 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101086.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.101086.log.gz 2024-01-27T01:09:26.786 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77346.log 2024-01-27T01:09:26.787 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77346.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77346.log.gz 2024-01-27T01:09:26.787 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56349.log 2024-01-27T01:09:26.788 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56349.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56349.log.gz 2024-01-27T01:09:26.788 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51298.log 2024-01-27T01:09:26.789 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51298.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51298.log.gz 2024-01-27T01:09:26.789 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116868.log 2024-01-27T01:09:26.791 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.116868.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.116868.log.gz 2024-01-27T01:09:26.791 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117619.log 2024-01-27T01:09:26.792 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.117619.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.117619.log.gz 2024-01-27T01:09:26.792 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71052.log 2024-01-27T01:09:26.794 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71052.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71052.log.gz 2024-01-27T01:09:26.794 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85015.log 2024-01-27T01:09:26.795 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85015.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85015.log.gz 2024-01-27T01:09:26.795 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97123.log 2024-01-27T01:09:26.796 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97123.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.97123.log.gz 2024-01-27T01:09:26.796 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73303.log 2024-01-27T01:09:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73303.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73303.log.gz 2024-01-27T01:09:26.798 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121383.log 2024-01-27T01:09:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.121383.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.121383.log.gz 2024-01-27T01:09:26.799 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49486.log 2024-01-27T01:09:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49486.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49486.log.gz 2024-01-27T01:09:26.800 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96162.log 2024-01-27T01:09:26.801 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96162.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96162.log.gz 2024-01-27T01:09:26.802 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91192.log 2024-01-27T01:09:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91192.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.91192.log.gz 2024-01-27T01:09:26.803 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56828.log 2024-01-27T01:09:26.804 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56828.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56828.log.gz 2024-01-27T01:09:26.804 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117711.log 2024-01-27T01:09:26.805 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.117711.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.117711.log.gz 2024-01-27T01:09:26.805 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51971.log 2024-01-27T01:09:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51971.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51971.log.gz 2024-01-27T01:09:26.807 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104577.log 2024-01-27T01:09:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104577.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.104577.log.gz 2024-01-27T01:09:26.808 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103236.log 2024-01-27T01:09:26.809 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103236.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.103236.log.gz 2024-01-27T01:09:26.809 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57280.log 2024-01-27T01:09:26.811 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57280.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57280.log.gz 2024-01-27T01:09:26.811 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99184.log 2024-01-27T01:09:26.812 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99184.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.99184.log.gz 2024-01-27T01:09:26.812 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114640.log 2024-01-27T01:09:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.114640.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.114640.log.gz 2024-01-27T01:09:26.813 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80841.log 2024-01-27T01:09:26.814 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80841.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80841.log.gz 2024-01-27T01:09:26.815 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115449.log 2024-01-27T01:09:26.816 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115449.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.115449.log.gz 2024-01-27T01:09:26.816 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120109.log 2024-01-27T01:09:26.817 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.120109.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.120109.log.gz 2024-01-27T01:09:26.817 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51067.log 2024-01-27T01:09:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51067.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51067.log.gz 2024-01-27T01:09:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119089.log 2024-01-27T01:09:26.820 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119089.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.119089.log.gz 2024-01-27T01:09:26.820 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98592.log 2024-01-27T01:09:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.98592.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.98592.log.gz 2024-01-27T01:09:26.821 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86455.log 2024-01-27T01:09:26.822 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86455.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86455.log.gz 2024-01-27T01:09:26.822 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61715.log 2024-01-27T01:09:26.824 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61715.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61715.log.gz 2024-01-27T01:09:26.824 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52956.log 2024-01-27T01:09:26.825 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52956.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52956.log.gz 2024-01-27T01:09:26.825 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51633.log 2024-01-27T01:09:26.826 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51633.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51633.log.gz 2024-01-27T01:09:26.826 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74327.log 2024-01-27T01:09:26.828 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74327.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74327.log.gz 2024-01-27T01:09:26.828 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70388.log 2024-01-27T01:09:26.829 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70388.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70388.log.gz 2024-01-27T01:09:26.829 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123581.log 2024-01-27T01:09:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.123581.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.123581.log.gz 2024-01-27T01:09:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49824.log 2024-01-27T01:09:26.832 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49824.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49824.log.gz 2024-01-27T01:09:26.832 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87936.log 2024-01-27T01:09:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87936.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.87936.log.gz 2024-01-27T01:09:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71837.log 2024-01-27T01:09:26.834 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71837.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71837.log.gz 2024-01-27T01:09:26.834 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55965.log 2024-01-27T01:09:26.835 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55965.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55965.log.gz 2024-01-27T01:09:26.836 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104213.log 2024-01-27T01:09:26.837 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104213.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.104213.log.gz 2024-01-27T01:09:26.837 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80545.log 2024-01-27T01:09:26.838 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80545.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80545.log.gz 2024-01-27T01:09:26.838 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87354.log 2024-01-27T01:09:26.839 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87354.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.87354.log.gz 2024-01-27T01:09:26.839 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66787.log 2024-01-27T01:09:26.841 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66787.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66787.log.gz 2024-01-27T01:09:26.841 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70955.log 2024-01-27T01:09:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70955.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70955.log.gz 2024-01-27T01:09:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73375.log 2024-01-27T01:09:26.843 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73375.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73375.log.gz 2024-01-27T01:09:26.843 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75528.log 2024-01-27T01:09:26.844 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75528.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75528.log.gz 2024-01-27T01:09:26.845 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86112.log 2024-01-27T01:09:26.846 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86112.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86112.log.gz 2024-01-27T01:09:26.846 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90457.log 2024-01-27T01:09:26.847 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90457.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.90457.log.gz 2024-01-27T01:09:26.847 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75482.log 2024-01-27T01:09:26.848 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75482.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75482.log.gz 2024-01-27T01:09:26.848 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65814.log 2024-01-27T01:09:26.850 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65814.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65814.log.gz 2024-01-27T01:09:26.850 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120807.log 2024-01-27T01:09:26.851 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.120807.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.120807.log.gz 2024-01-27T01:09:26.851 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43681.log 2024-01-27T01:09:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43681.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43681.log.gz 2024-01-27T01:09:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88628.log 2024-01-27T01:09:26.854 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88628.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.88628.log.gz 2024-01-27T01:09:26.854 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55735.log 2024-01-27T01:09:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55735.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55735.log.gz 2024-01-27T01:09:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56579.log 2024-01-27T01:09:26.856 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56579.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56579.log.gz 2024-01-27T01:09:26.856 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97355.log 2024-01-27T01:09:26.857 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97355.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.97355.log.gz 2024-01-27T01:09:26.858 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70806.log 2024-01-27T01:09:26.859 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70806.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70806.log.gz 2024-01-27T01:09:26.859 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57188.log 2024-01-27T01:09:26.860 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57188.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57188.log.gz 2024-01-27T01:09:26.860 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85972.log 2024-01-27T01:09:26.861 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85972.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85972.log.gz 2024-01-27T01:09:26.861 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43806.log 2024-01-27T01:09:26.863 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43806.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43806.log.gz 2024-01-27T01:09:26.863 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74939.log 2024-01-27T01:09:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74939.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74939.log.gz 2024-01-27T01:09:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55183.log 2024-01-27T01:09:26.865 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55183.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55183.log.gz 2024-01-27T01:09:26.866 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59284.log 2024-01-27T01:09:26.867 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59284.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59284.log.gz 2024-01-27T01:09:26.867 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118722.log 2024-01-27T01:09:26.868 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.118722.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.118722.log.gz 2024-01-27T01:09:26.868 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84598.log 2024-01-27T01:09:26.869 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84598.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84598.log.gz 2024-01-27T01:09:26.869 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99692.log 2024-01-27T01:09:26.871 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99692.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.99692.log.gz 2024-01-27T01:09:26.871 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89296.log 2024-01-27T01:09:26.872 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89296.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.89296.log.gz 2024-01-27T01:09:26.872 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117481.log 2024-01-27T01:09:26.873 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.117481.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.117481.log.gz 2024-01-27T01:09:26.873 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78995.log 2024-01-27T01:09:26.875 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78995.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78995.log.gz 2024-01-27T01:09:26.875 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54918.log 2024-01-27T01:09:26.876 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54918.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54918.log.gz 2024-01-27T01:09:26.876 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113636.log 2024-01-27T01:09:26.877 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113636.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113636.log.gz 2024-01-27T01:09:26.878 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86501.log 2024-01-27T01:09:26.879 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86501.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86501.log.gz 2024-01-27T01:09:26.879 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124165.log 2024-01-27T01:09:26.880 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124165.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.124165.log.gz 2024-01-27T01:09:26.880 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87280.log 2024-01-27T01:09:26.881 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87280.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87280.log.gz 2024-01-27T01:09:26.881 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81085.log 2024-01-27T01:09:26.882 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81085.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81085.log.gz 2024-01-27T01:09:26.882 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119041.log 2024-01-27T01:09:26.884 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119041.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.119041.log.gz 2024-01-27T01:09:26.884 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112842.log 2024-01-27T01:09:26.885 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112842.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.112842.log.gz 2024-01-27T01:09:26.885 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118985.log 2024-01-27T01:09:26.886 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.118985.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.118985.log.gz 2024-01-27T01:09:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115680.log 2024-01-27T01:09:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115680.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.115680.log.gz 2024-01-27T01:09:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80419.log 2024-01-27T01:09:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80419.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.80419.log.gz 2024-01-27T01:09:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121475.log 2024-01-27T01:09:26.890 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.121475.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.121475.log.gz 2024-01-27T01:09:26.890 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96533.log 2024-01-27T01:09:26.891 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96533.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.96533.log.gz 2024-01-27T01:09:26.892 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114808.log 2024-01-27T01:09:26.893 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.114808.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.114808.log.gz 2024-01-27T01:09:26.893 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52864.log 2024-01-27T01:09:26.894 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52864.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52864.log.gz 2024-01-27T01:09:26.894 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113820.log 2024-01-27T01:09:26.907 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113820.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113820.log.gz 2024-01-27T01:09:26.907 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119879.log 2024-01-27T01:09:26.908 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119879.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.119879.log.gz 2024-01-27T01:09:26.908 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125227.log 2024-01-27T01:09:26.909 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.125227.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.125227.log.gz 2024-01-27T01:09:26.910 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121613.log 2024-01-27T01:09:26.911 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.121613.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.121613.log.gz 2024-01-27T01:09:26.911 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111616.log 2024-01-27T01:09:26.913 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111616.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.111616.log.gz 2024-01-27T01:09:26.913 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105007.log 2024-01-27T01:09:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105007.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.105007.log.gz 2024-01-27T01:09:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99240.log 2024-01-27T01:09:26.916 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99240.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.99240.log.gz 2024-01-27T01:09:26.916 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43363.log 2024-01-27T01:09:26.917 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43363.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43363.log.gz 2024-01-27T01:09:26.917 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110753.log 2024-01-27T01:09:26.919 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110753.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.110753.log.gz 2024-01-27T01:09:26.919 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97169.log 2024-01-27T01:09:26.920 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97169.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.97169.log.gz 2024-01-27T01:09:26.920 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-01-27T01:09:26.924 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87471.log 2024-01-27T01:09:26.939 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87471.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87471.log.gz 2024-01-27T01:09:26.939 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78857.log 2024-01-27T01:09:26.951 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78857.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78857.log.gz 2024-01-27T01:09:26.951 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105686.log 2024-01-27T01:09:26.965 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105686.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.105686.log.gz 2024-01-27T01:09:26.965 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65860.log 2024-01-27T01:09:26.978 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65860.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65860.log.gz 2024-01-27T01:09:26.979 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55090.log 2024-01-27T01:09:27.002 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56964.log 2024-01-27T01:09:27.002 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55090.log.gz 2024-01-27T01:09:27.015 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56964.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56964.log.gz 2024-01-27T01:09:27.015 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115385.log 2024-01-27T01:09:27.027 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82174.log 2024-01-27T01:09:27.027 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.115385.log.gz 2024-01-27T01:09:27.040 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82174.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82174.log.gz 2024-01-27T01:09:27.041 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115134.log 2024-01-27T01:09:27.062 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115134.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.115134.log.gz 2024-01-27T01:09:27.063 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101837.log 2024-01-27T01:09:27.064 INFO:teuthology.orchestra.run.smithi046.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-01-27T01:09:27.067 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60870.log 2024-01-27T01:09:27.067 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.101837.log.gz 2024-01-27T01:09:27.068 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60870.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60870.log.gz 2024-01-27T01:09:27.080 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73910.log 2024-01-27T01:09:27.080 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73910.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.73910.log.gz 2024-01-27T01:09:27.080 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61253.log 2024-01-27T01:09:27.080 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61253.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61253.log.gz 2024-01-27T01:09:27.080 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98922.log 2024-01-27T01:09:27.085 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.98922.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.98922.log.gz 2024-01-27T01:09:27.085 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123629.log 2024-01-27T01:09:27.085 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.123629.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.123629.log.gz 2024-01-27T01:09:27.085 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84548.log 2024-01-27T01:09:27.085 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84548.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.84548.log.gz 2024-01-27T01:09:27.085 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113728.log 2024-01-27T01:09:27.085 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113728.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.113728.log.gz 2024-01-27T01:09:27.085 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119787.log 2024-01-27T01:09:27.089 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119787.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.119787.log.gz 2024-01-27T01:09:27.089 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54851.log 2024-01-27T01:09:27.089 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54851.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54851.log.gz 2024-01-27T01:09:27.089 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124491.log 2024-01-27T01:09:27.089 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124491.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.124491.log.gz 2024-01-27T01:09:27.089 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74224.log 2024-01-27T01:09:27.089 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74224.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74224.log.gz 2024-01-27T01:09:27.089 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118114.log 2024-01-27T01:09:27.094 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.118114.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.118114.log.gz 2024-01-27T01:09:27.094 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123423.log 2024-01-27T01:09:27.094 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.123423.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.123423.log.gz 2024-01-27T01:09:27.094 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111099.log 2024-01-27T01:09:27.095 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111099.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.111099.log.gz 2024-01-27T01:09:27.095 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55643.log 2024-01-27T01:09:27.095 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55643.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55643.log.gz 2024-01-27T01:09:27.095 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49440.log 2024-01-27T01:09:27.095 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49440.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49440.log.gz 2024-01-27T01:09:27.095 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110313.log 2024-01-27T01:09:27.105 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110313.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.110313.log.gz 2024-01-27T01:09:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124303.log 2024-01-27T01:09:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124303.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.124303.log.gz 2024-01-27T01:09:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57913.log 2024-01-27T01:09:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57913.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57913.log.gz 2024-01-27T01:09:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120450.log 2024-01-27T01:09:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.120450.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.120450.log.gz 2024-01-27T01:09:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113139.log 2024-01-27T01:09:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113139.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.113139.log.gz 2024-01-27T01:09:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51632.log 2024-01-27T01:09:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51632.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51632.log.gz 2024-01-27T01:09:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86317.log 2024-01-27T01:09:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86317.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86317.log.gz 2024-01-27T01:09:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109750.log 2024-01-27T01:09:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109750.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.109750.log.gz 2024-01-27T01:09:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97725.log 2024-01-27T01:09:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97725.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.97725.log.gz 2024-01-27T01:09:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67317.log 2024-01-27T01:09:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67317.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67317.log.gz 2024-01-27T01:09:27.108 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111339.log 2024-01-27T01:09:27.108 INFO:teuthology.orchestra.run.smithi046.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-01-27T01:09:27.108 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111339.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.111339.log.gz 2024-01-27T01:09:27.108 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125135.log 2024-01-27T01:09:27.108 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.125135.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.125135.log.gz 2024-01-27T01:09:27.108 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71499.log 2024-01-27T01:09:27.109 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71499.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71499.log.gz 2024-01-27T01:09:27.109 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125181.log 2024-01-27T01:09:27.110 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.125181.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.125181.log.gz 2024-01-27T01:09:27.110 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115496.log 2024-01-27T01:09:27.111 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115496.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.115496.log.gz 2024-01-27T01:09:27.111 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118001.log 2024-01-27T01:09:27.112 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.118001.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.118001.log.gz 2024-01-27T01:09:27.112 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61517.log 2024-01-27T01:09:27.115 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61517.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61517.log.gz 2024-01-27T01:09:27.116 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57773.log 2024-01-27T01:09:27.117 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57773.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57773.log.gz 2024-01-27T01:09:27.117 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61781.log 2024-01-27T01:09:27.118 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61781.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61781.log.gz 2024-01-27T01:09:27.118 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97979.log 2024-01-27T01:09:27.119 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97979.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.97979.log.gz 2024-01-27T01:09:27.119 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88999.log 2024-01-27T01:09:27.120 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88999.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88999.log.gz 2024-01-27T01:09:27.120 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58969.log 2024-01-27T01:09:27.121 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58969.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58969.log.gz 2024-01-27T01:09:27.121 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110564.log 2024-01-27T01:09:27.122 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110564.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.110564.log.gz 2024-01-27T01:09:27.122 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61451.log 2024-01-27T01:09:27.123 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61451.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61451.log.gz 2024-01-27T01:09:27.123 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110819.log 2024-01-27T01:09:27.124 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110819.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.110819.log.gz 2024-01-27T01:09:27.125 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101040.log 2024-01-27T01:09:27.126 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84747.log 2024-01-27T01:09:27.126 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101040.log: /var/log/ceph/ceph-client.admin.84747.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.101040.log.gz 2024-01-27T01:09:27.126 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.84747.log.gz 2024-01-27T01:09:27.126 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125365.log 2024-01-27T01:09:27.127 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.125365.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.125365.log.gz 2024-01-27T01:09:27.127 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78535.log 2024-01-27T01:09:27.128 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78535.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78535.log.gz 2024-01-27T01:09:27.128 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113185.log 2024-01-27T01:09:27.130 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113185.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.113185.log.gz 2024-01-27T01:09:27.130 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81516.log 2024-01-27T01:09:27.131 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81516.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.81516.log.gz 2024-01-27T01:09:27.131 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55275.log 2024-01-27T01:09:27.132 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55275.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55275.log.gz 2024-01-27T01:09:27.132 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122226.log 2024-01-27T01:09:27.133 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.122226.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.122226.log.gz 2024-01-27T01:09:27.133 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99535.log 2024-01-27T01:09:27.134 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99535.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.99535.log.gz 2024-01-27T01:09:27.134 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71660.log 2024-01-27T01:09:27.135 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71660.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71660.log.gz 2024-01-27T01:09:27.135 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116256.log 2024-01-27T01:09:27.136 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.116256.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116256.log.gz 2024-01-27T01:09:27.136 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114240.log 2024-01-27T01:09:27.137 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.114240.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.114240.log.gz 2024-01-27T01:09:27.137 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85584.log 2024-01-27T01:09:27.138 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85584.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85584.log.gz 2024-01-27T01:09:27.139 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67860.log 2024-01-27T01:09:27.140 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67860.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67860.log.gz 2024-01-27T01:09:27.140 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58826.log 2024-01-27T01:09:27.141 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74373.log 2024-01-27T01:09:27.141 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58826.log: /var/log/ceph/ceph-client.admin.74373.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58826.log.gz 2024-01-27T01:09:27.141 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74373.log.gz 2024-01-27T01:09:27.141 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104428.log 2024-01-27T01:09:27.142 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104428.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.104428.log.gz 2024-01-27T01:09:27.142 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125411.log 2024-01-27T01:09:27.144 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.125411.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.125411.log.gz 2024-01-27T01:09:27.144 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66833.log 2024-01-27T01:09:27.145 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66833.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66833.log.gz 2024-01-27T01:09:27.145 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58875.log 2024-01-27T01:09:27.146 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58875.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58875.log.gz 2024-01-27T01:09:27.147 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75183.log 2024-01-27T01:09:27.148 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75183.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75183.log.gz 2024-01-27T01:09:27.148 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112229.log 2024-01-27T01:09:27.149 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112229.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.112229.log.gz 2024-01-27T01:09:27.149 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90249.log 2024-01-27T01:09:27.150 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57094.log 2024-01-27T01:09:27.150 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90249.log: /var/log/ceph/ceph-client.admin.57094.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.90249.log.gz 2024-01-27T01:09:27.150 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57094.log.gz 2024-01-27T01:09:27.151 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119192.log 2024-01-27T01:09:27.152 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119192.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.119192.log.gz 2024-01-27T01:09:27.152 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72366.log 2024-01-27T01:09:27.153 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72366.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72366.log.gz 2024-01-27T01:09:27.153 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60916.log 2024-01-27T01:09:27.155 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60916.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60916.log.gz 2024-01-27T01:09:27.155 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99943.log 2024-01-27T01:09:27.156 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81850.log 2024-01-27T01:09:27.156 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99943.log: /var/log/ceph/ceph-client.admin.81850.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.99943.log.gz 2024-01-27T01:09:27.156 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81850.log.gz 2024-01-27T01:09:27.156 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56303.log 2024-01-27T01:09:27.157 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56303.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56303.log.gz 2024-01-27T01:09:27.157 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86617.log 2024-01-27T01:09:27.159 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86617.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86617.log.gz 2024-01-27T01:09:27.159 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115734.log 2024-01-27T01:09:27.160 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.115734.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.115734.log.gz 2024-01-27T01:09:27.160 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112508.log 2024-01-27T01:09:27.161 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112508.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.112508.log.gz 2024-01-27T01:09:27.162 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116684.log 2024-01-27T01:09:27.163 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100562.log 2024-01-27T01:09:27.163 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.116684.log: /var/log/ceph/ceph-client.admin.100562.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.116684.log.gz 2024-01-27T01:09:27.163 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.100562.log.gz 2024-01-27T01:09:27.163 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82036.log 2024-01-27T01:09:27.164 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82036.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82036.log.gz 2024-01-27T01:09:27.164 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88370.log 2024-01-27T01:09:27.165 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88370.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88370.log.gz 2024-01-27T01:09:27.165 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53002.log 2024-01-27T01:09:27.166 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53002.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53002.log.gz 2024-01-27T01:09:27.167 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81224.log 2024-01-27T01:09:27.168 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61100.log 2024-01-27T01:09:27.168 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81224.log: /var/log/ceph/ceph-client.admin.61100.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.81224.log.gz 2024-01-27T01:09:27.168 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61100.log.gz 2024-01-27T01:09:27.168 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119436.log 2024-01-27T01:09:27.169 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119436.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.119436.log.gz 2024-01-27T01:09:27.169 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100372.log 2024-01-27T01:09:27.170 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100372.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100372.log.gz 2024-01-27T01:09:27.170 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90341.log 2024-01-27T01:09:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90341.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.90341.log.gz 2024-01-27T01:09:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60824.log 2024-01-27T01:09:27.173 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60824.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60824.log.gz 2024-01-27T01:09:27.174 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82504.log 2024-01-27T01:09:27.176 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82504.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.82504.log.gz 2024-01-27T01:09:27.176 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76417.log 2024-01-27T01:09:27.177 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76417.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.76417.log.gz 2024-01-27T01:09:27.177 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82082.log 2024-01-27T01:09:27.178 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82082.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.82082.log.gz 2024-01-27T01:09:27.178 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89204.log 2024-01-27T01:09:27.179 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89204.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.89204.log.gz 2024-01-27T01:09:27.179 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125273.log 2024-01-27T01:09:27.180 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.125273.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.125273.log.gz 2024-01-27T01:09:27.180 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55597.log 2024-01-27T01:09:27.181 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55597.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55597.log.gz 2024-01-27T01:09:27.181 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106553.log 2024-01-27T01:09:27.182 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.106553.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106553.log.gz 2024-01-27T01:09:27.183 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101340.log 2024-01-27T01:09:27.184 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81608.log 2024-01-27T01:09:27.184 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101340.log: /var/log/ceph/ceph-client.admin.81608.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.101340.log.gz 2024-01-27T01:09:27.184 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81608.log.gz 2024-01-27T01:09:27.184 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71884.log 2024-01-27T01:09:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71884.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.71884.log.gz 2024-01-27T01:09:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88727.log 2024-01-27T01:09:27.186 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88727.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.88727.log.gz 2024-01-27T01:09:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77804.log 2024-01-27T01:09:27.188 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77804.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77804.log.gz 2024-01-27T01:09:27.188 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64275.log 2024-01-27T01:09:27.189 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64275.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64275.log.gz 2024-01-27T01:09:27.189 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66971.log 2024-01-27T01:09:27.190 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66971.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.66971.log.gz 2024-01-27T01:09:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109432.log 2024-01-27T01:09:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123829.log 2024-01-27T01:09:27.192 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.109432.log.gz 2024-01-27T01:09:27.196 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.123829.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.123829.log.gz 2024-01-27T01:09:27.196 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112909.log 2024-01-27T01:09:27.198 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99989.log 2024-01-27T01:09:27.198 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112909.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.112909.log.gz 2024-01-27T01:09:27.198 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99989.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.99989.log.gz 2024-01-27T01:09:27.198 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71120.log 2024-01-27T01:09:27.200 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.122008.log 2024-01-27T01:09:27.200 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71120.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71120.log.gz 2024-01-27T01:09:27.201 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.122008.log: 91.4% -- replaced with /var/log/ceph/ceph-client.0.122008.log.gz 2024-01-27T01:09:27.202 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78031.log 2024-01-27T01:09:27.203 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55689.log 2024-01-27T01:09:27.203 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78031.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78031.log.gz 2024-01-27T01:09:27.203 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55689.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55689.log.gz 2024-01-27T01:09:27.204 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53094.log 2024-01-27T01:09:27.205 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122417.log 2024-01-27T01:09:27.205 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53094.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53094.log.gz 2024-01-27T01:09:27.205 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.122417.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.122417.log.gz 2024-01-27T01:09:27.206 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120853.log 2024-01-27T01:09:27.207 INFO:teuthology.orchestra.run.smithi046.stderr:gzip /var/log/ceph/ceph-client.admin.120853.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.74893.log 2024-01-27T01:09:27.207 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.120853.log.gz 2024-01-27T01:09:27.211 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74893.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74893.log.gz 2024-01-27T01:09:27.211 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75033.log 2024-01-27T01:09:27.213 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75033.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75033.log.gz 2024-01-27T01:09:27.213 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81896.log 2024-01-27T01:09:27.214 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81896.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81896.log.gz 2024-01-27T01:09:27.214 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77528.log 2024-01-27T01:09:27.214 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110221.log 2024-01-27T01:09:27.216 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77528.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77528.log.gz 2024-01-27T01:09:27.216 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110221.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110221.log.gz 2024-01-27T01:09:27.216 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59548.log 2024-01-27T01:09:27.217 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59548.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59548.log.gz 2024-01-27T01:09:27.217 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114187.log 2024-01-27T01:09:27.218 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.114187.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.114187.log.gz 2024-01-27T01:09:27.218 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52588.log 2024-01-27T01:09:27.219 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52588.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52588.log.gz 2024-01-27T01:09:27.219 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104166.log 2024-01-27T01:09:27.220 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89250.log 2024-01-27T01:09:27.221 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104166.log: /var/log/ceph/ceph-client.admin.89250.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104166.log.gz 2024-01-27T01:09:27.221 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89250.log.gz 2024-01-27T01:09:27.221 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119335.log 2024-01-27T01:09:27.222 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.119335.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.119335.log.gz 2024-01-27T01:09:27.222 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104114.log 2024-01-27T01:09:27.222 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74763.log 2024-01-27T01:09:27.223 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104114.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.104114.log.gz 2024-01-27T01:09:27.224 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74763.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74763.log.gz 2024-01-27T01:09:27.224 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83117.log 2024-01-27T01:09:27.225 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.83117.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.83117.log.gz 2024-01-27T01:09:27.225 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83208.log 2024-01-27T01:09:27.225 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83257.log 2024-01-27T01:09:27.226 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.83208.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83208.log.gz 2024-01-27T01:09:27.226 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.83257.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83257.log.gz 2024-01-27T01:09:27.226 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124537.log 2024-01-27T01:09:27.228 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124537.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.124537.log.gz 2024-01-27T01:09:27.228 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98857.log 2024-01-27T01:09:27.228 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81942.log 2024-01-27T01:09:27.229 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.98857.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.98857.log.gz 2024-01-27T01:09:27.229 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81942.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81942.log.gz 2024-01-27T01:09:27.229 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124767.log 2024-01-27T01:09:27.230 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.124767.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.124767.log.gz 2024-01-27T01:09:27.230 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61187.log 2024-01-27T01:09:27.231 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65454.log 2024-01-27T01:09:27.232 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61187.log: /var/log/ceph/ceph-client.admin.65454.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61187.log.gz 2024-01-27T01:09:27.232 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65454.log.gz 2024-01-27T01:09:27.232 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105940.log 2024-01-27T01:09:27.233 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105940.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.105940.log.gz 2024-01-27T01:09:27.233 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51252.log 2024-01-27T01:09:27.234 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57234.log 2024-01-27T01:09:27.234 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51252.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51252.log.gz 2024-01-27T01:09:27.236 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57234.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57234.log.gz 2024-01-27T01:09:27.236 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88529.log 2024-01-27T01:09:27.237 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88529.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88529.log.gz 2024-01-27T01:09:27.237 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49636.log 2024-01-27T01:09:27.239 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49636.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49636.log.gz 2024-01-27T01:09:27.500 INFO:teuthology.orchestra.run.smithi106.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-27T01:09:27.587 INFO:teuthology.orchestra.run.smithi046.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-01-27T01:09:27.870 INFO:teuthology.orchestra.run.smithi106.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-01-27T01:09:27.935 INFO:teuthology.orchestra.run.smithi106.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-01-27T01:09:29.774 INFO:teuthology.orchestra.run.smithi046.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-27T01:09:30.454 INFO:teuthology.orchestra.run.smithi106.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-01-27T01:09:30.781 INFO:teuthology.orchestra.run.smithi106.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-01-27T01:09:30.814 INFO:teuthology.orchestra.run.smithi106.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-01-27T01:09:30.945 INFO:teuthology.orchestra.run.smithi046.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-27T01:09:31.187 INFO:teuthology.orchestra.run.smithi046.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-01-27T01:09:33.028 INFO:teuthology.orchestra.run.smithi046.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-27T01:09:33.353 INFO:teuthology.orchestra.run.smithi046.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-27T01:09:33.354 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-27T01:09:33.355 INFO:teuthology.orchestra.run.smithi046.stderr:real 0m7.369s 2024-01-27T01:09:33.355 INFO:teuthology.orchestra.run.smithi046.stderr:user 0m29.557s 2024-01-27T01:09:33.355 INFO:teuthology.orchestra.run.smithi046.stderr:sys 0m1.696s 2024-01-27T01:09:33.532 INFO:teuthology.orchestra.run.smithi106.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-01-27T01:09:33.534 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-01-27T01:09:33.534 INFO:teuthology.orchestra.run.smithi106.stderr:real 0m7.526s 2024-01-27T01:09:33.534 INFO:teuthology.orchestra.run.smithi106.stderr:user 0m25.865s 2024-01-27T01:09:33.534 INFO:teuthology.orchestra.run.smithi106.stderr:sys 0m0.893s 2024-01-27T01:09:33.534 INFO:tasks.ceph:Archiving logs... 2024-01-27T01:09:33.534 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi/7533360/remote/smithi046/log 2024-01-27T01:09:33.535 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-27T01:09:38.001 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi/7533360/remote/smithi106/log 2024-01-27T01:09:38.002 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-27T01:09:41.592 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-27T01:09:41.604 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-27T01:09:41.605 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-27T01:09:41.608 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-27T01:09:41.700 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-27T01:09:41.708 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-27T01:09:41.709 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-27T01:09:41.711 DEBUG:teuthology.orchestra.run.smithi106:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-27T01:09:41.730 INFO:teuthology.orchestra.run.smithi046.stderr:bash: ntpq: command not found 2024-01-27T01:09:41.730 INFO:teuthology.orchestra.run.smithi106.stderr:bash: ntpq: command not found 2024-01-27T01:09:41.734 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-27T01:09:41.735 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-01-27T01:09:41.735 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 +338us[ +338us] +/- 84ms 2024-01-27T01:09:41.735 INFO:teuthology.orchestra.run.smithi046.stdout:^* hv02.front.sepia.ceph.com 3 7 377 38 +19us[ +15us] +/- 70ms 2024-01-27T01:09:41.735 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 40 -331us[ -335us] +/- 90ms 2024-01-27T01:09:41.735 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T01:09:41.736 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-27T01:09:41.737 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-01-27T01:09:41.737 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 +292us[ +292us] +/- 84ms 2024-01-27T01:09:41.737 INFO:teuthology.orchestra.run.smithi106.stdout:^* hv02.front.sepia.ceph.com 3 7 377 34 -14us[ -22us] +/- 70ms 2024-01-27T01:09:41.737 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 29 -362us[ -362us] +/- 90ms 2024-01-27T01:09:41.737 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T01:09:41.737 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-27T01:09:41.746 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-27T01:09:41.747 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-27T01:09:41.761 DEBUG:teuthology.orchestra.run.smithi046:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-27T01:09:41.897 DEBUG:teuthology.orchestra.run.smithi106:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-27T01:09:41.979 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-27T01:09:42.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:09:42.018 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-27T01:09:42.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:09:42.050 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-27T01:09:42.060 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-27T01:09:42.069 INFO:teuthology.task.internal:Duration was 6808.478515 seconds 2024-01-27T01:09:42.070 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-27T01:09:42.078 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-27T01:09:42.078 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-27T01:09:42.081 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-27T01:09:42.137 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-27T01:09:42.140 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-27T01:09:42.328 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-27T01:09:42.328 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2024-01-27T01:09:42.328 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-27T01:09:42.356 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi106.front.sepia.ceph.com 2024-01-27T01:09:42.356 DEBUG:teuthology.orchestra.run.smithi106:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-27T01:09:42.378 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-27T01:09:42.379 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-27T01:09:42.399 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-27T01:09:42.465 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-27T01:09:42.465 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-27T01:09:42.467 DEBUG:teuthology.orchestra.run.smithi106:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-27T01:09:42.593 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-27T01:09:42.604 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-27T01:09:42.604 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-27T01:09:42.635 DEBUG:teuthology.orchestra.run.smithi106:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-27T01:09:42.662 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-27T01:09:42.674 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-27T01:09:42.679 DEBUG:teuthology.orchestra.run.smithi106:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-27T01:09:42.708 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2024-01-27T01:09:42.728 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = core 2024-01-27T01:09:42.743 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-27T01:09:42.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:09:42.787 DEBUG:teuthology.orchestra.run.smithi106:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-27T01:09:42.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T01:09:42.805 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-27T01:09:42.815 INFO:teuthology.task.internal:Transferring archived files... 2024-01-27T01:09:42.815 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi/7533360/remote/smithi046 2024-01-27T01:09:42.816 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-27T01:09:42.879 DEBUG:teuthology.misc:Transferring archived files from smithi106:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi/7533360/remote/smithi106 2024-01-27T01:09:42.879 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-27T01:09:42.918 INFO:teuthology.task.internal:Removing archive directory... 2024-01-27T01:09:42.918 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-27T01:09:42.921 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-27T01:09:42.970 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-27T01:09:42.981 INFO:teuthology.task.internal:Not uploading archives. 2024-01-27T01:09:42.981 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-27T01:09:42.991 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-27T01:09:42.991 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-27T01:09:42.993 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-27T01:09:43.012 INFO:teuthology.orchestra.run.smithi046.stdout: 262175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 27 01:09 /home/ubuntu/cephtest 2024-01-27T01:09:43.025 INFO:teuthology.orchestra.run.smithi106.stdout: 262175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 27 01:09 /home/ubuntu/cephtest 2024-01-27T01:09:43.027 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-27T01:09:43.037 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-27T01:09:43.058 INFO:teuthology.nuke:Checking targets against current locks 2024-01-27T01:09:43.083 DEBUG:teuthology.nuke:shortname: smithi046 2024-01-27T01:09:43.083 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-27T01:09:43.106 DEBUG:teuthology.nuke:shortname: smithi106 2024-01-27T01:09:43.106 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-27T01:09:43.109 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi/7533360', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-26 23:10:49.696299', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH0jCpB2oSBYuBJFz8r7h5gvoxbVdD5G78u+0MHqsdwmoDh9tqhFT/LoUCId1h189cSnuoFeCa4UY2kBN3lkbp0='} 2024-01-27T01:09:43.135 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-26_01:07:29-fs-wip-yuri4-testing-2024-01-25-1331-quincy-distro-default-smithi/7533360', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-26 23:10:49.697695', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGNrzIpyQQzXce3S622+VCGLhT0dDQ6JzjjnSWBTu1pGvGlS9KyptTAX9xnumt/OLUGHnhziceBhmNU9cWNCBCE='} 2024-01-27T01:09:43.164 INFO:teuthology.orchestra.console.smithi046:Power off 2024-01-27T01:09:43.164 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-27T01:09:43.189 DEBUG:teuthology.orchestra.console.smithi046:power off output: Chassis Power Control: Down/Off 2024-01-27T01:09:43.190 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T01:09:43.204 INFO:teuthology.orchestra.console.smithi106:Power off 2024-01-27T01:09:43.204 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-27T01:09:43.213 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-01-27T01:09:43.238 DEBUG:teuthology.orchestra.console.smithi106:power off output: Chassis Power Control: Down/Off 2024-01-27T01:09:43.238 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T01:09:43.262 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-01-27T01:09:47.214 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T01:09:47.263 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T01:09:47.324 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-01-27T01:09:47.373 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-01-27T01:09:51.326 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T01:09:51.374 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T01:09:51.436 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is off 2024-01-27T01:09:51.483 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-01-27T01:09:51.538 INFO:teuthology.orchestra.console.smithi046:Power off completed 2024-01-27T01:09:55.488 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T01:09:55.597 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is off 2024-01-27T01:09:55.698 INFO:teuthology.orchestra.console.smithi106:Power off completed 2024-01-27T01:09:55.802 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} duration: 6808.478514671326 failure_reason: '"1706312662.6613069 mon.a (mon.0) 512 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2024-01-27T01:09:55.802 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-27T01:09:55.897 INFO:teuthology.run:FAIL